2024-05-27T07:16:35.390 INFO:root:teuthology version: 0.0.1.dev291+ge7ae758 2024-05-27T07:16:35.390 DEBUG:teuthology.run:Teuthology command: teuthology --name vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi --archive /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664 --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/no 6-workunit/suites/dbench}} --owner scheduled_vshankar@teuthology -- /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664/orig.config.yaml 2024-05-27T07:16:35.461 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T07:16:35.559 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664 branch: wip-vshankar-testing-20240521.063558-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/no 6-workunit/suites/dbench}} email: vshankar@redhat.com first_in_suite: false job_id: '7727664' kernel: client: sha1: distro flavor: default kdb: 1 sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240521.063558-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240521.063558-debug sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 owner: scheduled_vshankar@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 2499 sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 sleep_before_teardown: 0 subset: 573/1024 suite: fs suite_branch: wip-vshankar-testing-20240521.063558-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFAoN9uEGt6uE4fKw0EjLDbyLmoa/NdQJnm03S3Jupj0WPsgFp4cadl0B9iaWFejTbbTaEnqazjLX6vd0ayX7/g= smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCfuq6F6scO9oScrmecE7cP918MNe7KI6xTBSQgIQS2DiprKt2kxGszP6d5xVDLBn9sDeMdWLbwbTI83cLbqXN4= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPYJwcg/DDpYib3mydxQQsCdBUC6KeLpDErcK3hVnz7v1ING2X0MrHJcykpjgJM58pJ6/yDgLkUws+UqEINIAC8= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e7ae7580c417d6ab27fda98fecadca5d9f85d63e timestamp: 2024-05-27_05:52:49 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-27T07:16:35.559 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa; will attempt to use it 2024-05-27T07:16:35.560 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks 2024-05-27T07:16:35.560 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-27T07:16:35.562 INFO:teuthology.task.internal:Checking packages... 2024-05-27T07:16:35.582 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '38f9d900c83c63799fbdbe61acc9a11b0d3554a6' 2024-05-27T07:16:35.582 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-27T07:16:35.582 INFO:teuthology.packaging:ref: None 2024-05-27T07:16:35.582 INFO:teuthology.packaging:tag: None 2024-05-27T07:16:35.582 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240521.063558-debug 2024-05-27T07:16:35.583 INFO:teuthology.packaging:sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:16:35.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240521.063558-debug 2024-05-27T07:16:35.815 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3872.g38f9d900 2024-05-27T07:16:35.817 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-27T07:16:35.824 INFO:teuthology.task.internal:no buildpackages task found 2024-05-27T07:16:35.824 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-27T07:16:35.845 INFO:teuthology.task.internal:Saving configuration 2024-05-27T07:16:35.865 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-27T07:16:35.873 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-27T07:16:35.899 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 07:10:48.914136', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFAoN9uEGt6uE4fKw0EjLDbyLmoa/NdQJnm03S3Jupj0WPsgFp4cadl0B9iaWFejTbbTaEnqazjLX6vd0ayX7/g='} 2024-05-27T07:16:35.920 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 07:10:48.915756', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCfuq6F6scO9oScrmecE7cP918MNe7KI6xTBSQgIQS2DiprKt2kxGszP6d5xVDLBn9sDeMdWLbwbTI83cLbqXN4='} 2024-05-27T07:16:35.940 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 07:10:48.917147', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPYJwcg/DDpYib3mydxQQsCdBUC6KeLpDErcK3hVnz7v1ING2X0MrHJcykpjgJM58pJ6/yDgLkUws+UqEINIAC8='} 2024-05-27T07:16:35.940 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-27T07:16:35.947 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-27T07:16:35.947 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-27T07:16:35.947 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-27T07:16:35.947 INFO:teuthology.run_tasks:Running task console_log... 2024-05-27T07:16:36.033 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4a7b4bbac0>, signals=[15]) 2024-05-27T07:16:36.033 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-27T07:16:36.041 INFO:teuthology.task.internal:Opening connections... 2024-05-27T07:16:36.041 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-05-27T07:16:36.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:16:36.121 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-05-27T07:16:36.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:16:36.198 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-05-27T07:16:36.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:16:36.271 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-27T07:16:36.279 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-05-27T07:16:36.297 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-05-27T07:16:36.298 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-05-27T07:16:36.355 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-05-27T07:16:36.355 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-05-27T07:16:36.355 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-05-27T07:16:36.355 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-05-27T07:16:36.355 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-05-27T07:16:36.356 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-05-27T07:16:36.356 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T07:16:36.356 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-05-27T07:16:36.356 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-05-27T07:16:36.356 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T07:16:36.356 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-05-27T07:16:36.356 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T07:16:36.356 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T07:16:36.356 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T07:16:36.357 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-05-27T07:16:36.457 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-05-27T07:16:36.476 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-05-27T07:16:36.476 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-05-27T07:16:36.533 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-05-27T07:16:36.533 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="9" 2024-05-27T07:16:36.533 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-05-27T07:16:36.533 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-05-27T07:16:36.533 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="9" 2024-05-27T07:16:36.534 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el9" 2024-05-27T07:16:36.534 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T07:16:36.534 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-05-27T07:16:36.534 INFO:teuthology.orchestra.run.smithi163.stdout:LOGO="fedora-logo-icon" 2024-05-27T07:16:36.534 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T07:16:36.534 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-05-27T07:16:36.534 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T07:16:36.534 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T07:16:36.534 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T07:16:36.535 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-05-27T07:16:36.560 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-05-27T07:16:36.577 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-05-27T07:16:36.578 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-05-27T07:16:36.635 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-05-27T07:16:36.635 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-05-27T07:16:36.635 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-05-27T07:16:36.635 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-05-27T07:16:36.635 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-05-27T07:16:36.635 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-05-27T07:16:36.635 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T07:16:36.635 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-05-27T07:16:36.636 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-05-27T07:16:36.636 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T07:16:36.636 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-05-27T07:16:36.636 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T07:16:36.636 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T07:16:36.636 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T07:16:36.636 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-05-27T07:16:36.668 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-27T07:16:36.680 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-27T07:16:36.687 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-27T07:16:36.687 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-05-27T07:16:36.689 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-05-27T07:16:36.692 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-05-27T07:16:36.708 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-27T07:16:36.716 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-27T07:16:36.716 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-05-27T07:16:36.746 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-05-27T07:16:36.749 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-05-27T07:16:36.847 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-27T07:16:36.927 INFO:teuthology.run_tasks:Running task kernel... 2024-05-27T07:16:36.946 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'} 2024-05-27T07:16:36.946 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-05-27T07:16:36.946 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-05-27T07:16:36.947 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T07:16:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:16:36.963 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-05-27T07:16:37.021 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-437.el9.x86_64 2024-05-27T07:16:37.021 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-437.el9.x86_64 2024-05-27T07:16:37.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-05-27T07:16:39.409 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.1 MB 00:00 2024-05-27T07:16:41.318 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-05-27T07:16:45.319 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-05-27T07:16:46.861 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 17 kB 00:00 2024-05-27T07:16:48.473 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-27T07:16:53.671 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-05-27T07:16:55.535 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-27T07:16:55.536 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-27T07:16:55.630 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-05-27T07:16:55.638 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-05-27T07:16:55.638 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-05-27T07:16:55.638 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-05-27T07:16:55.638 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-05-27T07:16:55.638 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-447.el9 baseos 576 k 2024-05-27T07:16:55.639 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-05-27T07:16:55.639 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 5.14.0-447.el9 baseos 16 M 2024-05-27T07:16:55.639 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 5.14.0-447.el9 baseos 34 M 2024-05-27T07:16:55.639 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core x86_64 5.14.0-447.el9 baseos 28 M 2024-05-27T07:16:55.639 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:16:55.639 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-05-27T07:16:55.640 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-05-27T07:16:55.640 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-05-27T07:16:55.640 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:16:55.640 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 78 M 2024-05-27T07:16:55.640 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 123 M 2024-05-27T07:16:55.641 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-05-27T07:16:56.106 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): kernel-5.14.0-447.el9.x86_64.rpm 2.8 MB/s | 576 kB 00:00 2024-05-27T07:16:58.681 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): kernel-core-5.14.0-447.el9.x86_64.rpm 5.6 MB/s | 16 MB 00:02 2024-05-27T07:16:59.823 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): kernel-modules-core-5.14.0-447.el9.x86_6 7.5 MB/s | 28 MB 00:03 2024-05-27T07:17:00.248 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): kernel-modules-5.14.0-447.el9.x86_64.rpm 7.8 MB/s | 34 MB 00:04 2024-05-27T07:17:00.248 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:17:00.248 INFO:teuthology.orchestra.run.smithi114.stdout:Total 17 MB/s | 78 MB 00:04 2024-05-27T07:17:00.936 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-05-27T07:17:01.280 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-05-27T07:17:01.280 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-05-27T07:17:02.176 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-05-27T07:17:02.176 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-05-27T07:17:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-05-27T07:17:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-core-5.14.0-447.el9.x86_64 1/4 2024-05-27T07:17:05.267 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-27T07:17:07.923 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-27T07:17:08.060 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-27T07:17:12.789 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-27T07:17:12.839 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-5.14.0-447.el9.x86_64 4/4 2024-05-27T07:17:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-core-5.14.0-447.el9.x86_64 4/4 2024-05-27T07:17:47.083 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-447.el9.x86_64 4/4 2024-05-27T07:17:47.120 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-447.el9.x86_64 4/4 2024-05-27T07:17:47.407 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-447.el9.x86_64 4/4 2024-05-27T07:17:47.407 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-447.el9.x86_64 1/4 2024-05-27T07:17:47.407 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-27T07:17:47.407 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-27T07:17:48.168 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-core-5.14.0-447.el9.x86_64 4/4 2024-05-27T07:17:48.168 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:17:48.168 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-05-27T07:17:48.168 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-447.el9.x86_64 2024-05-27T07:17:48.168 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core-5.14.0-447.el9.x86_64 2024-05-27T07:17:48.168 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-5.14.0-447.el9.x86_64 2024-05-27T07:17:48.169 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core-5.14.0-447.el9.x86_64 2024-05-27T07:17:48.169 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:17:48.169 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-05-27T07:17:48.517 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-05-27T07:17:48.595 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-447.el9.x86_64 2024-05-27T07:17:48.596 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-447.el9.x86_64 2024-05-27T07:17:48.596 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-447.el9.x86_64 2024-05-27T07:17:48.596 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-27T07:17:48.596 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-27T07:17:48.596 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-05-27T07:17:48.596 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-05-27T07:17:48.596 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-05-27T07:17:48.646 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-447.el9.x86_64 2024-05-27T07:17:48.647 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-447.el9.x86_64 2024-05-27T07:17:48.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-05-27T07:17:48.684 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-05-27T07:17:48.685 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-05-27T07:17:48.685 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-05-27T07:17:48.685 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 68.el9 2024-05-27T07:17:48.685 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-05-27T07:17:48.685 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-27T07:17:48.686 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-05-27T07:17:48.686 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8264254 2024-05-27T07:17:48.686 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-05-27T07:17:48.686 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-27T07:17:48.686 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-27T07:17:48.686 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-27T07:17:48.687 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-27T07:17:48.687 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-05-27T07:17:48.687 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-05-27T07:17:48.687 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-27T07:17:48.687 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-05-27T07:17:48.687 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-05-27T07:17:48.687 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:17:48.687 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-27T07:17:48.688 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-27T07:17:48.688 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-27T07:17:48.688 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-05-27T07:17:48.688 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:17:48.688 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-05-27T07:17:48.689 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-27T07:17:48.689 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 5.14.0-447.el9.x86_64 2024-05-27T07:17:48.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-27T07:17:49.298 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-05-27T07:17:50.691 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-27T07:17:50.724 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-05-27T07:17:50.727 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-05-27T07:17:50.742 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.NDqfIN0YUV 2024-05-27T07:17:50.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NDqfIN0YUV 2024-05-27T07:17:50.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.NDqfIN0YUV 2024-05-27T07:17:50.924 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.NDqfIN0YUV is 6KB 2024-05-27T07:17:50.972 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.NDqfIN0YUV 2024-05-27T07:17:50.986 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-05-27T07:17:51.047 INFO:teuthology.orchestra.run.smithi114.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-27T07:17:51.047 INFO:teuthology.orchestra.run.smithi114.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-27T07:17:51.047 INFO:teuthology.orchestra.run.smithi114.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-27T07:17:51.047 INFO:teuthology.orchestra.run.smithi114.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-447.el9.x86_64.conf 2024-05-27T07:17:51.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-447.el9.x86_64 2024-05-27T07:17:51.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-05-27T07:18:21.212 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-27T07:18:21.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-27T07:18:21.213 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:19:21.216 DEBUG:teuthology.orchestra.remote:timed out 2024-05-27T07:19:30.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-27T07:19:30.218 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:19:37.791 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-05-27T07:19:38.224 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-27T07:19:38.225 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-447.el9.x86_64"... 2024-05-27T07:19:38.225 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-05-27T07:19:38.239 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-447.el9.x86_64 2024-05-27T07:19:38.240 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-447.el9.x86_64 vs 5.14.0-447.el9.x86_64 2024-05-27T07:19:38.240 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-27T07:19:38.240 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-27T07:19:39.241 DEBUG:teuthology.parallel:result is None 2024-05-27T07:19:39.241 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-27T07:19:39.250 INFO:teuthology.task.internal:Creating test directory... 2024-05-27T07:19:39.250 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T07:19:39.253 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T07:19:39.255 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T07:19:39.273 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-27T07:19:39.284 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-27T07:19:39.298 INFO:teuthology.task.internal:Creating archive directory... 2024-05-27T07:19:39.298 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T07:19:39.308 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T07:19:39.313 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T07:19:39.352 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-27T07:19:39.358 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-27T07:19:39.358 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T07:19:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:19:39.395 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T07:19:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:19:39.411 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T07:19:39.413 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T07:19:39.437 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T07:19:39.470 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T07:19:39.485 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T07:19:39.490 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T07:19:39.501 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T07:19:39.505 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T07:19:39.550 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T07:19:39.552 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-27T07:19:39.559 INFO:teuthology.task.internal:Configuring sudo... 2024-05-27T07:19:39.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T07:19:39.562 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T07:19:39.564 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T07:19:39.590 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-27T07:19:39.599 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-27T07:19:39.599 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T07:19:39.626 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T07:19:39.631 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T07:19:39.647 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T07:19:39.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T07:19:39.791 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T07:19:39.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T07:19:39.883 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:19:39.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T07:19:39.949 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T07:19:39.995 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T07:19:40.053 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T07:19:40.099 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T07:19:40.138 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-27T07:19:40.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T07:19:40.203 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T07:19:40.247 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T07:19:40.303 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T07:19:40.387 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T07:19:40.428 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-27T07:19:40.428 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T07:19:40.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-05-27T07:19:40.496 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-05-27T07:19:40.498 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-05-27T07:19:40.533 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T07:19:40.572 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T07:19:40.573 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T07:19:41.005 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-27T07:19:41.013 INFO:teuthology.task.internal:Starting timer... 2024-05-27T07:19:41.013 INFO:teuthology.run_tasks:Running task pcp... 2024-05-27T07:19:41.024 INFO:teuthology.run_tasks:Running task selinux... 2024-05-27T07:19:41.033 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-27T07:19:41.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-05-27T07:19:41.136 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-05-27T07:19:41.141 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-05-27T07:19:41.196 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-05-27T07:19:41.198 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-05-27T07:19:41.248 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-05-27T07:19:41.250 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-27T07:19:41.250 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-05-27T07:19:41.279 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-05-27T07:19:41.279 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-05-27T07:19:41.317 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-05-27T07:19:41.317 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-05-27T07:19:41.343 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-05-27T07:19:41.343 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi163.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-05-27T07:19:41.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T07:19:41.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:19:41.393 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T07:19:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:19:41.420 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T07:19:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:19:41.447 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-27T07:19:41.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-05-27T07:19:41.797 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-05-27T07:19:41.827 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-05-27T07:19:41.856 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-27T07:19:41.870 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-27T07:19:41.882 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-27T07:19:41.883 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi163.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-27T07:24:36.796 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi163.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-05-27T07:24:36.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-27T07:24:36.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:24:36.879 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-05-27T07:24:36.963 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-27T07:24:36.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-05-27T07:24:36.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:24:37.037 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-05-27T07:24:37.113 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-05-27T07:24:37.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-27T07:24:37.114 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:24:37.185 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-05-27T07:24:37.230 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-27T07:24:37.231 INFO:teuthology.run_tasks:Running task clock... 2024-05-27T07:24:37.241 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-27T07:24:37.241 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T07:24:37.242 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:24:37.245 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T07:24:37.245 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:24:37.247 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T07:24:37.247 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:24:37.274 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T07:24:37.274 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T07:24:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T07:24:37.290 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T07:24:37.298 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T07:24:37.311 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2024-05-27T07:24:37.312 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T07:24:37.316 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-05-27T07:24:37.321 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2024-05-27T07:24:37.327 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-05-27T07:24:37.334 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-05-27T07:24:37.335 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T07:24:37.342 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T07:24:37.344 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-05-27T07:24:37.348 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T07:24:37.356 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T07:24:37.357 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T07:24:37.370 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T07:24:37.402 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:24:37.405 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:24:37.406 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-05-27T07:24:37.406 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:24:37.406 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:24:37.406 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:24:37.406 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:24:37.410 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:24:37.414 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:24:37.414 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-05-27T07:24:37.414 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:24:37.414 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:24:37.414 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:24:37.414 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:24:37.432 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:24:37.434 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:24:37.434 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-05-27T07:24:37.434 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:24:37.434 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:24:37.434 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:24:37.434 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:24:37.435 INFO:teuthology.run_tasks:Running task install... 2024-05-27T07:24:37.443 DEBUG:teuthology.task.install:project ceph 2024-05-27T07:24:37.444 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'}} 2024-05-27T07:24:37.444 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'} 2024-05-27T07:24:37.444 INFO:teuthology.task.install:Using flavor: default 2024-05-27T07:24:37.449 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-27T07:24:37.449 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-27T07:24:37.450 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'tag': None, 'wait_for_package': False} 2024-05-27T07:24:37.450 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:24:37.452 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'tag': None, 'wait_for_package': False} 2024-05-27T07:24:37.452 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:24:37.453 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'tag': None, 'wait_for_package': False} 2024-05-27T07:24:37.453 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:24:37.687 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/ 2024-05-27T07:24:37.687 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3872.g38f9d900 2024-05-27T07:24:37.691 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/ 2024-05-27T07:24:37.691 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3872.g38f9d900 2024-05-27T07:24:37.692 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/ 2024-05-27T07:24:37.692 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3872.g38f9d900 2024-05-27T07:24:37.802 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T07:24:37.802 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-27T07:24:37.802 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T07:24:37.811 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T07:24:37.811 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-27T07:24:37.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T07:24:37.827 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T07:24:37.827 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T07:24:37.836 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T07:24:37.836 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T07:24:37.862 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T07:24:37.862 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:24:37.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T07:24:37.888 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T07:24:37.888 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T07:24:37.893 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T07:24:37.900 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T07:24:37.953 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T07:24:37.965 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T07:24:37.973 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T07:24:38.027 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-05-27T07:24:38.029 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-05-27T07:24:38.030 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T07:24:38.036 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2024-05-27T07:24:38.038 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2024-05-27T07:24:38.094 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-05-27T07:24:38.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-05-27T07:24:38.252 INFO:teuthology.orchestra.run.smithi186.stdout:45 files removed 2024-05-27T07:24:38.260 INFO:teuthology.orchestra.run.smithi163.stdout:45 files removed 2024-05-27T07:24:38.280 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T07:24:38.286 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T07:24:38.321 INFO:teuthology.orchestra.run.smithi114.stdout:45 files removed 2024-05-27T07:24:38.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T07:24:38.870 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 261 kB/s | 84 kB 00:00 2024-05-27T07:24:39.000 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 183 kB/s | 84 kB 00:00 2024-05-27T07:24:39.130 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 164 kB/s | 84 kB 00:00 2024-05-27T07:24:39.298 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-05-27T07:24:39.518 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 25 kB/s | 12 kB 00:00 2024-05-27T07:24:39.656 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 25 kB/s | 12 kB 00:00 2024-05-27T07:24:39.812 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-05-27T07:24:40.176 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 3.0 kB/s | 1.9 kB 00:00 2024-05-27T07:24:40.283 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 3.3 kB/s | 1.9 kB 00:00 2024-05-27T07:24:41.987 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.1 MB 00:02 2024-05-27T07:24:42.310 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.1 MB 00:02 2024-05-27T07:24:42.334 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.1 MB 00:02 2024-05-27T07:24:43.889 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-05-27T07:24:44.178 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-05-27T07:24:44.910 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-05-27T07:24:49.123 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.2 MB 00:01 2024-05-27T07:24:50.465 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 17 kB 00:00 2024-05-27T07:24:50.520 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.2 MB 00:02 2024-05-27T07:24:51.853 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 17 kB 00:00 2024-05-27T07:24:53.217 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-05-27T07:24:53.344 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 1.3 MB/s | 6.2 MB 00:04 2024-05-27T07:24:54.344 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 5.6 MB/s | 21 MB 00:03 2024-05-27T07:24:54.961 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 17 kB 00:00 2024-05-27T07:24:58.348 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-05-27T07:24:59.575 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-05-27T07:25:00.499 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:25:00.499 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:25:00.511 INFO:teuthology.orchestra.run.smithi186.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T07:25:00.512 INFO:teuthology.orchestra.run.smithi186.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T07:25:00.513 INFO:teuthology.orchestra.run.smithi186.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T07:25:00.513 INFO:teuthology.orchestra.run.smithi186.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T07:25:00.515 INFO:teuthology.orchestra.run.smithi186.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T07:25:00.517 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T07:25:00.518 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T07:25:00.581 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-27T07:25:00.591 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-05-27T07:25:00.591 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-27T07:25:00.591 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-05-27T07:25:00.592 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-27T07:25:00.592 INFO:teuthology.orchestra.run.smithi186.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T07:25:00.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 6.6 k 2024-05-27T07:25:00.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.4 M 2024-05-27T07:25:00.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 863 k 2024-05-27T07:25:00.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 147 k 2024-05-27T07:25:00.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 1.5 M 2024-05-27T07:25:00.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 143 k 2024-05-27T07:25:00.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 3.5 M 2024-05-27T07:25:00.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 7.4 M 2024-05-27T07:25:00.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 49 k 2024-05-27T07:25:00.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 10 M 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 43 M 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 273 k 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 764 k 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 224 k 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 27 k 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 18 k 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 32 k 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 755 k 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 127 k 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T07:25:00.593 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T07:25:00.594 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T07:25:00.594 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T07:25:00.594 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T07:25:00.594 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T07:25:00.594 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 161 k 2024-05-27T07:25:00.594 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 320 k 2024-05-27T07:25:00.594 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 300 k 2024-05-27T07:25:00.594 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 99 k 2024-05-27T07:25:00.594 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 86 k 2024-05-27T07:25:00.594 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.0 M 2024-05-27T07:25:00.594 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 169 k 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.3 M 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.1 M 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T07:25:00.595 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-27T07:25:00.596 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T07:25:00.596 INFO:teuthology.orchestra.run.smithi186.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T07:25:00.596 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T07:25:00.596 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 20 M 2024-05-27T07:25:00.596 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 34 k 2024-05-27T07:25:00.596 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 2.3 M 2024-05-27T07:25:00.596 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 274 k 2024-05-27T07:25:00.596 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 4.9 M 2024-05-27T07:25:00.596 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 18 M 2024-05-27T07:25:00.596 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 16 k 2024-05-27T07:25:00.596 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 25 k 2024-05-27T07:25:00.596 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T07:25:00.596 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T07:25:00.597 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T07:25:00.597 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T07:25:00.597 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T07:25:00.597 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T07:25:00.597 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T07:25:00.598 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T07:25:00.598 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 157 k 2024-05-27T07:25:00.598 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T07:25:00.598 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T07:25:00.598 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T07:25:00.598 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T07:25:00.598 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T07:25:00.598 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 498 k 2024-05-27T07:25:00.598 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T07:25:00.598 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.2 M 2024-05-27T07:25:00.598 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T07:25:00.599 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T07:25:00.599 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T07:25:00.599 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T07:25:00.599 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T07:25:00.599 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T07:25:00.599 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T07:25:00.599 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T07:25:00.599 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T07:25:00.599 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T07:25:00.599 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T07:25:00.599 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T07:25:00.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T07:25:00.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T07:25:00.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T07:25:00.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T07:25:00.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T07:25:00.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 45 k 2024-05-27T07:25:00.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 136 k 2024-05-27T07:25:00.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T07:25:00.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T07:25:00.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T07:25:00.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T07:25:00.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T07:25:00.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T07:25:00.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T07:25:00.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T07:25:00.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T07:25:00.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T07:25:00.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T07:25:00.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T07:25:00.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T07:25:00.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T07:25:00.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T07:25:00.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T07:25:00.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T07:25:00.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T07:25:00.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T07:25:00.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T07:25:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T07:25:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T07:25:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T07:25:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T07:25:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T07:25:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T07:25:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T07:25:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T07:25:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T07:25:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T07:25:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T07:25:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T07:25:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T07:25:00.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T07:25:00.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T07:25:00.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T07:25:00.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T07:25:00.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T07:25:00.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T07:25:00.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T07:25:00.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T07:25:00.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T07:25:00.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T07:25:00.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T07:25:00.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T07:25:00.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T07:25:00.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T07:25:00.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T07:25:00.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T07:25:00.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T07:25:00.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T07:25:00.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T07:25:00.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T07:25:00.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T07:25:00.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T07:25:00.604 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T07:25:00.604 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T07:25:00.604 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T07:25:00.604 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T07:25:00.605 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T07:25:00.605 INFO:teuthology.orchestra.run.smithi186.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T07:25:00.605 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T07:25:00.605 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T07:25:00.605 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-05-27T07:25:00.605 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T07:25:00.605 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:25:00.605 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-27T07:25:00.605 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-05-27T07:25:00.605 INFO:teuthology.orchestra.run.smithi186.stdout:Install 148 Packages 2024-05-27T07:25:00.605 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 11 Packages 2024-05-27T07:25:00.605 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:25:00.606 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 219 M 2024-05-27T07:25:00.606 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-27T07:25:01.357 INFO:teuthology.orchestra.run.smithi186.stdout:(1/159): ceph-19.0.0-3872.g38f9d900.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-05-27T07:25:01.582 INFO:teuthology.orchestra.run.smithi186.stdout:(2/159): ceph-fuse-19.0.0-3872.g38f9d900.el9.x8 3.7 MB/s | 863 kB 00:00 2024-05-27T07:25:01.641 INFO:teuthology.orchestra.run.smithi186.stdout:(3/159): ceph-immutable-object-cache-19.0.0-387 2.5 MB/s | 147 kB 00:00 2024-05-27T07:25:01.726 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:25:01.727 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:25:01.737 INFO:teuthology.orchestra.run.smithi163.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T07:25:01.738 INFO:teuthology.orchestra.run.smithi163.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T07:25:01.739 INFO:teuthology.orchestra.run.smithi163.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T07:25:01.740 INFO:teuthology.orchestra.run.smithi163.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T07:25:01.741 INFO:teuthology.orchestra.run.smithi163.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T07:25:01.744 INFO:teuthology.orchestra.run.smithi163.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T07:25:01.745 INFO:teuthology.orchestra.run.smithi163.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T07:25:01.809 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-27T07:25:01.823 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-05-27T07:25:01.823 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-05-27T07:25:01.823 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-05-27T07:25:01.823 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-27T07:25:01.823 INFO:teuthology.orchestra.run.smithi163.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T07:25:01.823 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 6.6 k 2024-05-27T07:25:01.823 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.4 M 2024-05-27T07:25:01.823 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 863 k 2024-05-27T07:25:01.823 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 147 k 2024-05-27T07:25:01.823 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 1.5 M 2024-05-27T07:25:01.823 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 143 k 2024-05-27T07:25:01.824 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 3.5 M 2024-05-27T07:25:01.824 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 7.4 M 2024-05-27T07:25:01.824 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 49 k 2024-05-27T07:25:01.824 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 10 M 2024-05-27T07:25:01.824 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 43 M 2024-05-27T07:25:01.824 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 273 k 2024-05-27T07:25:01.824 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 764 k 2024-05-27T07:25:01.824 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 224 k 2024-05-27T07:25:01.824 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-shell x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 27 k 2024-05-27T07:25:01.824 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-top noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 18 k 2024-05-27T07:25:01.824 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T07:25:01.824 INFO:teuthology.orchestra.run.smithi163.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T07:25:01.824 INFO:teuthology.orchestra.run.smithi163.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T07:25:01.825 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 32 k 2024-05-27T07:25:01.825 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 755 k 2024-05-27T07:25:01.825 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 127 k 2024-05-27T07:25:01.825 INFO:teuthology.orchestra.run.smithi163.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T07:25:01.825 INFO:teuthology.orchestra.run.smithi163.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T07:25:01.825 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T07:25:01.825 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T07:25:01.826 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T07:25:01.826 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T07:25:01.826 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 161 k 2024-05-27T07:25:01.826 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 320 k 2024-05-27T07:25:01.826 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 300 k 2024-05-27T07:25:01.826 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 99 k 2024-05-27T07:25:01.826 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 86 k 2024-05-27T07:25:01.826 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.0 M 2024-05-27T07:25:01.826 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 169 k 2024-05-27T07:25:01.826 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T07:25:01.827 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2024-05-27T07:25:01.827 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T07:25:01.827 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T07:25:01.827 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T07:25:01.827 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T07:25:01.827 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T07:25:01.827 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T07:25:01.827 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T07:25:01.827 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.3 M 2024-05-27T07:25:01.827 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.1 M 2024-05-27T07:25:01.827 INFO:teuthology.orchestra.run.smithi163.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T07:25:01.827 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T07:25:01.827 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-05-27T07:25:01.828 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T07:25:01.828 INFO:teuthology.orchestra.run.smithi163.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T07:25:01.828 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T07:25:01.828 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 20 M 2024-05-27T07:25:01.828 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 34 k 2024-05-27T07:25:01.828 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 2.3 M 2024-05-27T07:25:01.828 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 274 k 2024-05-27T07:25:01.828 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 4.9 M 2024-05-27T07:25:01.828 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 18 M 2024-05-27T07:25:01.828 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 16 k 2024-05-27T07:25:01.829 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 25 k 2024-05-27T07:25:01.829 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T07:25:01.829 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T07:25:01.829 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T07:25:01.830 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T07:25:01.830 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T07:25:01.830 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T07:25:01.830 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T07:25:01.830 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T07:25:01.830 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 157 k 2024-05-27T07:25:01.831 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T07:25:01.831 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T07:25:01.831 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T07:25:01.831 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T07:25:01.831 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T07:25:01.832 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 498 k 2024-05-27T07:25:01.832 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T07:25:01.832 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.2 M 2024-05-27T07:25:01.833 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T07:25:01.833 INFO:teuthology.orchestra.run.smithi163.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T07:25:01.833 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T07:25:01.833 INFO:teuthology.orchestra.run.smithi163.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T07:25:01.833 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T07:25:01.833 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T07:25:01.834 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T07:25:01.834 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T07:25:01.834 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T07:25:01.834 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T07:25:01.834 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T07:25:01.835 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T07:25:01.835 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T07:25:01.835 INFO:teuthology.orchestra.run.smithi163.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T07:25:01.835 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T07:25:01.835 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T07:25:01.835 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T07:25:01.836 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 45 k 2024-05-27T07:25:01.836 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 136 k 2024-05-27T07:25:01.836 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T07:25:01.836 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T07:25:01.837 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T07:25:01.837 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T07:25:01.837 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T07:25:01.837 INFO:teuthology.orchestra.run.smithi163.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T07:25:01.837 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T07:25:01.837 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T07:25:01.838 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T07:25:01.838 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T07:25:01.838 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T07:25:01.838 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T07:25:01.838 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T07:25:01.838 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T07:25:01.838 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T07:25:01.838 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T07:25:01.838 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T07:25:01.838 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T07:25:01.838 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T07:25:01.838 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T07:25:01.839 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T07:25:01.839 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T07:25:01.839 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T07:25:01.839 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T07:25:01.839 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T07:25:01.839 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T07:25:01.839 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T07:25:01.839 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T07:25:01.839 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T07:25:01.839 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T07:25:01.839 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T07:25:01.839 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T07:25:01.839 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T07:25:01.840 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T07:25:01.840 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T07:25:01.840 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T07:25:01.840 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T07:25:01.840 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T07:25:01.840 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T07:25:01.841 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T07:25:01.841 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T07:25:01.841 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T07:25:01.841 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T07:25:01.841 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T07:25:01.841 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T07:25:01.842 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T07:25:01.842 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T07:25:01.842 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T07:25:01.842 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T07:25:01.842 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T07:25:01.842 INFO:teuthology.orchestra.run.smithi163.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T07:25:01.842 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T07:25:01.842 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T07:25:01.843 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T07:25:01.843 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T07:25:01.843 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T07:25:01.843 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T07:25:01.843 INFO:teuthology.orchestra.run.smithi163.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T07:25:01.843 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T07:25:01.844 INFO:teuthology.orchestra.run.smithi163.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T07:25:01.844 INFO:teuthology.orchestra.run.smithi163.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T07:25:01.844 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T07:25:01.844 INFO:teuthology.orchestra.run.smithi163.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T07:25:01.844 INFO:teuthology.orchestra.run.smithi163.stdout:Installing weak dependencies: 2024-05-27T07:25:01.844 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T07:25:01.845 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:25:01.845 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-27T07:25:01.845 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-05-27T07:25:01.845 INFO:teuthology.orchestra.run.smithi163.stdout:Install 148 Packages 2024-05-27T07:25:01.845 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 11 Packages 2024-05-27T07:25:01.845 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:25:01.846 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 219 M 2024-05-27T07:25:01.846 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-27T07:25:01.983 INFO:teuthology.orchestra.run.smithi186.stdout:(4/159): ceph-mds-19.0.0-3872.g38f9d900.el9.x86 6.6 MB/s | 2.3 MB 00:00 2024-05-27T07:25:02.092 INFO:teuthology.orchestra.run.smithi186.stdout:(5/159): ceph-base-19.0.0-3872.g38f9d900.el9.x8 5.9 MB/s | 5.4 MB 00:00 2024-05-27T07:25:02.141 INFO:teuthology.orchestra.run.smithi186.stdout:(6/159): ceph-mgr-19.0.0-3872.g38f9d900.el9.x86 9.5 MB/s | 1.5 MB 00:00 2024-05-27T07:25:02.675 INFO:teuthology.orchestra.run.smithi186.stdout:(7/159): ceph-mon-19.0.0-3872.g38f9d900.el9.x86 8.4 MB/s | 4.9 MB 00:00 2024-05-27T07:25:02.692 INFO:teuthology.orchestra.run.smithi163.stdout:(1/159): ceph-19.0.0-3872.g38f9d900.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-05-27T07:25:03.676 INFO:teuthology.orchestra.run.smithi163.stdout:(2/159): ceph-fuse-19.0.0-3872.g38f9d900.el9.x8 878 kB/s | 863 kB 00:00 2024-05-27T07:25:03.809 INFO:teuthology.orchestra.run.smithi163.stdout:(3/159): ceph-immutable-object-cache-19.0.0-387 1.1 MB/s | 147 kB 00:00 2024-05-27T07:25:04.075 INFO:teuthology.orchestra.run.smithi186.stdout:(8/159): ceph-radosgw-19.0.0-3872.g38f9d900.el9 7.2 MB/s | 10 MB 00:01 2024-05-27T07:25:04.267 INFO:teuthology.orchestra.run.smithi186.stdout:(9/159): ceph-common-19.0.0-3872.g38f9d900.el9. 6.3 MB/s | 20 MB 00:03 2024-05-27T07:25:04.284 INFO:teuthology.orchestra.run.smithi186.stdout:(10/159): ceph-selinux-19.0.0-3872.g38f9d900.el 121 kB/s | 25 kB 00:00 2024-05-27T07:25:04.326 INFO:teuthology.orchestra.run.smithi163.stdout:(4/159): ceph-base-19.0.0-3872.g38f9d900.el9.x8 2.8 MB/s | 5.4 MB 00:01 2024-05-27T07:25:04.359 INFO:teuthology.orchestra.run.smithi186.stdout:(11/159): cephfs-mirror-19.0.0-3872.g38f9d900.e 2.9 MB/s | 224 kB 00:00 2024-05-27T07:25:04.402 INFO:teuthology.orchestra.run.smithi163.stdout:(5/159): ceph-mds-19.0.0-3872.g38f9d900.el9.x86 3.8 MB/s | 2.3 MB 00:00 2024-05-27T07:25:04.434 INFO:teuthology.orchestra.run.smithi186.stdout:(12/159): cephfs-shell-19.0.0-3872.g38f9d900.el 361 kB/s | 27 kB 00:00 2024-05-27T07:25:04.502 INFO:teuthology.orchestra.run.smithi186.stdout:(13/159): libcephfs-devel-19.0.0-3872.g38f9d900 475 kB/s | 32 kB 00:00 2024-05-27T07:25:04.652 INFO:teuthology.orchestra.run.smithi163.stdout:(6/159): ceph-mgr-19.0.0-3872.g38f9d900.el9.x86 4.6 MB/s | 1.5 MB 00:00 2024-05-27T07:25:04.670 INFO:teuthology.orchestra.run.smithi186.stdout:(14/159): ceph-osd-19.0.0-3872.g38f9d900.el9.x8 7.0 MB/s | 18 MB 00:02 2024-05-27T07:25:04.710 INFO:teuthology.orchestra.run.smithi186.stdout:(15/159): libcephfs2-19.0.0-3872.g38f9d900.el9. 3.5 MB/s | 755 kB 00:00 2024-05-27T07:25:04.736 INFO:teuthology.orchestra.run.smithi186.stdout:(16/159): libcephsqlite-19.0.0-3872.g38f9d900.e 2.3 MB/s | 157 kB 00:00 2024-05-27T07:25:04.787 INFO:teuthology.orchestra.run.smithi186.stdout:(17/159): librados-devel-19.0.0-3872.g38f9d900. 1.6 MB/s | 127 kB 00:00 2024-05-27T07:25:04.853 INFO:teuthology.orchestra.run.smithi186.stdout:(18/159): libradosstriper1-19.0.0-3872.g38f9d90 4.1 MB/s | 498 kB 00:00 2024-05-27T07:25:04.945 INFO:teuthology.orchestra.run.smithi186.stdout:(19/159): python3-ceph-argparse-19.0.0-3872.g38 494 kB/s | 45 kB 00:00 2024-05-27T07:25:05.045 INFO:teuthology.orchestra.run.smithi186.stdout:(20/159): python3-ceph-common-19.0.0-3872.g38f9 1.3 MB/s | 136 kB 00:00 2024-05-27T07:25:05.146 INFO:teuthology.orchestra.run.smithi186.stdout:(21/159): python3-cephfs-19.0.0-3872.g38f9d900. 1.6 MB/s | 161 kB 00:00 2024-05-27T07:25:05.277 INFO:teuthology.orchestra.run.smithi163.stdout:(7/159): ceph-mon-19.0.0-3872.g38f9d900.el9.x86 5.6 MB/s | 4.9 MB 00:00 2024-05-27T07:25:05.314 INFO:teuthology.orchestra.run.smithi186.stdout:(22/159): python3-rados-19.0.0-3872.g38f9d900.e 1.9 MB/s | 320 kB 00:00 2024-05-27T07:25:05.413 INFO:teuthology.orchestra.run.smithi186.stdout:(23/159): librgw2-19.0.0-3872.g38f9d900.el9.x86 8.2 MB/s | 5.2 MB 00:00 2024-05-27T07:25:05.438 INFO:teuthology.orchestra.run.smithi186.stdout:(24/159): python3-rbd-19.0.0-3872.g38f9d900.el9 2.4 MB/s | 300 kB 00:00 2024-05-27T07:25:05.497 INFO:teuthology.orchestra.run.smithi186.stdout:(25/159): python3-rgw-19.0.0-3872.g38f9d900.el9 1.2 MB/s | 99 kB 00:00 2024-05-27T07:25:05.522 INFO:teuthology.orchestra.run.smithi186.stdout:(26/159): rbd-fuse-19.0.0-3872.g38f9d900.el9.x8 1.0 MB/s | 86 kB 00:00 2024-05-27T07:25:05.864 INFO:teuthology.orchestra.run.smithi186.stdout:(27/159): rbd-mirror-19.0.0-3872.g38f9d900.el9. 8.3 MB/s | 3.0 MB 00:00 2024-05-27T07:25:05.940 INFO:teuthology.orchestra.run.smithi186.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3872.g 456 kB/s | 34 kB 00:00 2024-05-27T07:25:06.023 INFO:teuthology.orchestra.run.smithi186.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3872.g38f9d90 1.7 MB/s | 143 kB 00:00 2024-05-27T07:25:06.048 INFO:teuthology.orchestra.run.smithi186.stdout:(30/159): rbd-nbd-19.0.0-3872.g38f9d900.el9.x86 322 kB/s | 169 kB 00:00 2024-05-27T07:25:06.415 INFO:teuthology.orchestra.run.smithi186.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3872.g38f9d 9.0 MB/s | 3.5 MB 00:00 2024-05-27T07:25:06.482 INFO:teuthology.orchestra.run.smithi186.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3872.g38 4.0 MB/s | 274 kB 00:00 2024-05-27T07:25:06.485 INFO:teuthology.orchestra.run.smithi163.stdout:(8/159): ceph-radosgw-19.0.0-3872.g38f9d900.el9 8.4 MB/s | 10 MB 00:01 2024-05-27T07:25:06.544 INFO:teuthology.orchestra.run.smithi163.stdout:(9/159): ceph-selinux-19.0.0-3872.g38f9d900.el9 431 kB/s | 25 kB 00:00 2024-05-27T07:25:06.549 INFO:teuthology.orchestra.run.smithi186.stdout:(33/159): ceph-mgr-rook-19.0.0-3872.g38f9d900.e 742 kB/s | 49 kB 00:00 2024-05-27T07:25:06.624 INFO:teuthology.orchestra.run.smithi186.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3872.g3 220 kB/s | 16 kB 00:00 2024-05-27T07:25:06.741 INFO:teuthology.orchestra.run.smithi186.stdout:(35/159): ceph-volume-19.0.0-3872.g38f9d900.el9 2.3 MB/s | 273 kB 00:00 2024-05-27T07:25:06.858 INFO:teuthology.orchestra.run.smithi186.stdout:(36/159): cephadm-19.0.0-3872.g38f9d900.el9.noa 6.4 MB/s | 764 kB 00:00 2024-05-27T07:25:06.934 INFO:teuthology.orchestra.run.smithi186.stdout:(37/159): cephfs-top-19.0.0-3872.g38f9d900.el9. 244 kB/s | 18 kB 00:00 2024-05-27T07:25:07.392 INFO:teuthology.orchestra.run.smithi186.stdout:(38/159): fuse-2.9.9-15.el9.x86_64.rpm 175 kB/s | 80 kB 00:00 2024-05-27T07:25:07.709 INFO:teuthology.orchestra.run.smithi186.stdout:(39/159): ceph-mgr-diskprediction-local-19.0.0- 4.5 MB/s | 7.4 MB 00:01 2024-05-27T07:25:08.376 INFO:teuthology.orchestra.run.smithi186.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 293 kB/s | 195 kB 00:00 2024-05-27T07:25:08.985 INFO:teuthology.orchestra.run.smithi186.stdout:(41/159): libgfortran-11.4.1-3.el9.x86_64.rpm 504 kB/s | 803 kB 00:01 2024-05-27T07:25:08.986 INFO:teuthology.orchestra.run.smithi163.stdout:(10/159): ceph-osd-19.0.0-3872.g38f9d900.el9.x8 4.1 MB/s | 18 MB 00:04 2024-05-27T07:25:09.110 INFO:teuthology.orchestra.run.smithi186.stdout:(42/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 345 kB/s | 253 kB 00:00 2024-05-27T07:25:09.111 INFO:teuthology.orchestra.run.smithi163.stdout:(11/159): cephfs-mirror-19.0.0-3872.g38f9d900.e 1.8 MB/s | 224 kB 00:00 2024-05-27T07:25:09.178 INFO:teuthology.orchestra.run.smithi163.stdout:(12/159): cephfs-shell-19.0.0-3872.g38f9d900.el 407 kB/s | 27 kB 00:00 2024-05-27T07:25:09.237 INFO:teuthology.orchestra.run.smithi163.stdout:(13/159): libcephfs-devel-19.0.0-3872.g38f9d900 545 kB/s | 32 kB 00:00 2024-05-27T07:25:09.454 INFO:teuthology.orchestra.run.smithi163.stdout:(14/159): libcephfs2-19.0.0-3872.g38f9d900.el9. 3.4 MB/s | 755 kB 00:00 2024-05-27T07:25:09.527 INFO:teuthology.orchestra.run.smithi186.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 256 kB/s | 106 kB 00:00 2024-05-27T07:25:09.529 INFO:teuthology.orchestra.run.smithi163.stdout:(15/159): libcephsqlite-19.0.0-3872.g38f9d900.e 2.1 MB/s | 157 kB 00:00 2024-05-27T07:25:09.629 INFO:teuthology.orchestra.run.smithi163.stdout:(16/159): librados-devel-19.0.0-3872.g38f9d900. 1.2 MB/s | 127 kB 00:00 2024-05-27T07:25:09.821 INFO:teuthology.orchestra.run.smithi163.stdout:(17/159): libradosstriper1-19.0.0-3872.g38f9d90 2.5 MB/s | 498 kB 00:00 2024-05-27T07:25:10.036 INFO:teuthology.orchestra.run.smithi186.stdout:(44/159): python3-pycparser-2.20-6.el9.noarch.r 266 kB/s | 135 kB 00:00 2024-05-27T07:25:10.155 INFO:teuthology.orchestra.run.smithi163.stdout:(18/159): ceph-common-19.0.0-3872.g38f9d900.el9 2.5 MB/s | 20 MB 00:07 2024-05-27T07:25:10.213 INFO:teuthology.orchestra.run.smithi163.stdout:(19/159): python3-ceph-argparse-19.0.0-3872.g38 777 kB/s | 45 kB 00:00 2024-05-27T07:25:10.314 INFO:teuthology.orchestra.run.smithi163.stdout:(20/159): python3-ceph-common-19.0.0-3872.g38f9 1.3 MB/s | 136 kB 00:00 2024-05-27T07:25:10.503 INFO:teuthology.orchestra.run.smithi186.stdout:(45/159): ceph-test-19.0.0-3872.g38f9d900.el9.x 6.9 MB/s | 43 MB 00:06 2024-05-27T07:25:10.731 INFO:teuthology.orchestra.run.smithi163.stdout:(21/159): ceph-test-19.0.0-3872.g38f9d900.el9.x 10 MB/s | 43 MB 00:04 2024-05-27T07:25:10.756 INFO:teuthology.orchestra.run.smithi163.stdout:(22/159): python3-cephfs-19.0.0-3872.g38f9d900. 365 kB/s | 161 kB 00:00 2024-05-27T07:25:10.798 INFO:teuthology.orchestra.run.smithi163.stdout:(23/159): python3-rados-19.0.0-3872.g38f9d900.e 4.7 MB/s | 320 kB 00:00 2024-05-27T07:25:10.848 INFO:teuthology.orchestra.run.smithi163.stdout:(24/159): python3-rgw-19.0.0-3872.g38f9d900.el9 1.9 MB/s | 99 kB 00:00 2024-05-27T07:25:10.898 INFO:teuthology.orchestra.run.smithi163.stdout:(25/159): rbd-fuse-19.0.0-3872.g38f9d900.el9.x8 1.7 MB/s | 86 kB 00:00 2024-05-27T07:25:10.911 INFO:teuthology.orchestra.run.smithi186.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 143 kB/s | 125 kB 00:00 2024-05-27T07:25:10.923 INFO:teuthology.orchestra.run.smithi163.stdout:(26/159): python3-rbd-19.0.0-3872.g38f9d900.el9 1.8 MB/s | 300 kB 00:00 2024-05-27T07:25:10.991 INFO:teuthology.orchestra.run.smithi163.stdout:(27/159): rbd-nbd-19.0.0-3872.g38f9d900.el9.x86 2.5 MB/s | 169 kB 00:00 2024-05-27T07:25:11.053 INFO:teuthology.orchestra.run.smithi186.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 4.8 MB/s | 689 kB 00:00 2024-05-27T07:25:11.095 INFO:teuthology.orchestra.run.smithi186.stdout:(48/159): python3-cryptography-36.0.1-4.el9.x86 595 kB/s | 1.2 MB 00:02 2024-05-27T07:25:11.099 INFO:teuthology.orchestra.run.smithi163.stdout:(28/159): librgw2-19.0.0-3872.g38f9d900.el9.x86 4.0 MB/s | 5.2 MB 00:01 2024-05-27T07:25:11.124 INFO:teuthology.orchestra.run.smithi163.stdout:(29/159): ceph-grafana-dashboards-19.0.0-3872.g 256 kB/s | 34 kB 00:00 2024-05-27T07:25:11.128 INFO:teuthology.orchestra.run.smithi186.stdout:(49/159): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 943 kB 00:00 2024-05-27T07:25:11.162 INFO:teuthology.orchestra.run.smithi186.stdout:(50/159): python3-urllib3-1.26.5-5.el9.noarch.r 326 kB/s | 215 kB 00:00 2024-05-27T07:25:11.166 INFO:teuthology.orchestra.run.smithi163.stdout:(30/159): ceph-mgr-cephadm-19.0.0-3872.g38f9d90 2.1 MB/s | 143 kB 00:00 2024-05-27T07:25:11.187 INFO:teuthology.orchestra.run.smithi186.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 372 kB/s | 22 kB 00:00 2024-05-27T07:25:11.225 INFO:teuthology.orchestra.run.smithi163.stdout:(31/159): rbd-mirror-19.0.0-3872.g38f9d900.el9. 9.3 MB/s | 3.0 MB 00:00 2024-05-27T07:25:11.229 INFO:teuthology.orchestra.run.smithi186.stdout:(52/159): boost-program-options-1.75.0-8.el9.x8 797 kB/s | 107 kB 00:00 2024-05-27T07:25:11.255 INFO:teuthology.orchestra.run.smithi186.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 489 kB/s | 33 kB 00:00 2024-05-27T07:25:11.284 INFO:teuthology.orchestra.run.smithi163.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3872.g38 4.6 MB/s | 274 kB 00:00 2024-05-27T07:25:11.296 INFO:teuthology.orchestra.run.smithi186.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 439 kB/s | 18 kB 00:00 2024-05-27T07:25:11.322 INFO:teuthology.orchestra.run.smithi186.stdout:(55/159): flex-2.6.4-9.el9.x86_64.rpm 1.9 MB/s | 313 kB 00:00 2024-05-27T07:25:11.334 INFO:teuthology.orchestra.run.smithi163.stdout:(33/159): ceph-mgr-rook-19.0.0-3872.g38f9d900.e 990 kB/s | 49 kB 00:00 2024-05-27T07:25:11.347 INFO:teuthology.orchestra.run.smithi186.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-05-27T07:25:11.372 INFO:teuthology.orchestra.run.smithi186.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-05-27T07:25:11.384 INFO:teuthology.orchestra.run.smithi163.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3872.g3 329 kB/s | 16 kB 00:00 2024-05-27T07:25:11.389 INFO:teuthology.orchestra.run.smithi186.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-05-27T07:25:11.423 INFO:teuthology.orchestra.run.smithi186.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-05-27T07:25:11.443 INFO:teuthology.orchestra.run.smithi163.stdout:(35/159): ceph-volume-19.0.0-3872.g38f9d900.el9 4.6 MB/s | 273 kB 00:00 2024-05-27T07:25:11.456 INFO:teuthology.orchestra.run.smithi186.stdout:(60/159): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-05-27T07:25:11.473 INFO:teuthology.orchestra.run.smithi186.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 507 kB/s | 25 kB 00:00 2024-05-27T07:25:11.490 INFO:teuthology.orchestra.run.smithi186.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-05-27T07:25:11.524 INFO:teuthology.orchestra.run.smithi186.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-05-27T07:25:11.535 INFO:teuthology.orchestra.run.smithi163.stdout:(36/159): cephadm-19.0.0-3872.g38f9d900.el9.noa 8.2 MB/s | 764 kB 00:00 2024-05-27T07:25:11.549 INFO:teuthology.orchestra.run.smithi186.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-05-27T07:25:11.574 INFO:teuthology.orchestra.run.smithi186.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-05-27T07:25:11.577 INFO:teuthology.orchestra.run.smithi163.stdout:(37/159): cephfs-top-19.0.0-3872.g38f9d900.el9. 435 kB/s | 18 kB 00:00 2024-05-27T07:25:11.625 INFO:teuthology.orchestra.run.smithi186.stdout:(66/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.6 MB/s | 3.0 MB 00:00 2024-05-27T07:25:11.709 INFO:teuthology.orchestra.run.smithi186.stdout:(67/159): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-05-27T07:25:11.750 INFO:teuthology.orchestra.run.smithi186.stdout:(68/159): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-05-27T07:25:11.784 INFO:teuthology.orchestra.run.smithi186.stdout:(69/159): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-05-27T07:25:11.843 INFO:teuthology.orchestra.run.smithi186.stdout:(70/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-05-27T07:25:11.868 INFO:teuthology.orchestra.run.smithi186.stdout:(71/159): postgresql-private-libs-13.14-1.el9.x 1.1 MB/s | 133 kB 00:00 2024-05-27T07:25:11.901 INFO:teuthology.orchestra.run.smithi186.stdout:(72/159): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-05-27T07:25:11.943 INFO:teuthology.orchestra.run.smithi186.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-05-27T07:25:11.985 INFO:teuthology.orchestra.run.smithi186.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-05-27T07:25:12.011 INFO:teuthology.orchestra.run.smithi163.stdout:(38/159): ceph-mgr-dashboard-19.0.0-3872.g38f9d 4.0 MB/s | 3.5 MB 00:00 2024-05-27T07:25:12.078 INFO:teuthology.orchestra.run.smithi186.stdout:(75/159): postgresql-server-13.14-1.el9.x86_64. 20 MB/s | 5.8 MB 00:00 2024-05-27T07:25:12.144 INFO:teuthology.orchestra.run.smithi186.stdout:(76/159): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 6.0 MB 00:00 2024-05-27T07:25:12.170 INFO:teuthology.orchestra.run.smithi186.stdout:(77/159): python3-devel-3.9.18-3.el9.x86_64.rpm 1.3 MB/s | 244 kB 00:00 2024-05-27T07:25:12.195 INFO:teuthology.orchestra.run.smithi186.stdout:(78/159): python3-jsonpatch-1.21-16.el9.noarch. 516 kB/s | 26 kB 00:00 2024-05-27T07:25:12.220 INFO:teuthology.orchestra.run.smithi186.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-05-27T07:25:12.245 INFO:teuthology.orchestra.run.smithi186.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-05-27T07:25:12.270 INFO:teuthology.orchestra.run.smithi186.stdout:(81/159): python3-jinja2-2.11.3-6.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-05-27T07:25:12.286 INFO:teuthology.orchestra.run.smithi163.stdout:(39/159): fuse-2.9.9-15.el9.x86_64.rpm 113 kB/s | 80 kB 00:00 2024-05-27T07:25:12.296 INFO:teuthology.orchestra.run.smithi186.stdout:(82/159): python3-markupsafe-1.1.1-12.el9.x86_6 462 kB/s | 35 kB 00:00 2024-05-27T07:25:12.321 INFO:teuthology.orchestra.run.smithi186.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-05-27T07:25:12.349 INFO:teuthology.orchestra.run.smithi186.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-05-27T07:25:12.372 INFO:teuthology.orchestra.run.smithi186.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-27T07:25:12.397 INFO:teuthology.orchestra.run.smithi186.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 845 kB/s | 42 kB 00:00 2024-05-27T07:25:12.422 INFO:teuthology.orchestra.run.smithi186.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-05-27T07:25:12.447 INFO:teuthology.orchestra.run.smithi186.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-05-27T07:25:12.473 INFO:teuthology.orchestra.run.smithi186.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-05-27T07:25:12.498 INFO:teuthology.orchestra.run.smithi186.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-27T07:25:12.523 INFO:teuthology.orchestra.run.smithi186.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-05-27T07:25:12.553 INFO:teuthology.orchestra.run.smithi163.stdout:(40/159): ceph-mgr-diskprediction-local-19.0.0- 5.3 MB/s | 7.4 MB 00:01 2024-05-27T07:25:12.565 INFO:teuthology.orchestra.run.smithi186.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-05-27T07:25:12.632 INFO:teuthology.orchestra.run.smithi186.stdout:(93/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-05-27T07:25:12.658 INFO:teuthology.orchestra.run.smithi186.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 481 kB/s | 44 kB 00:00 2024-05-27T07:25:12.683 INFO:teuthology.orchestra.run.smithi186.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-27T07:25:12.708 INFO:teuthology.orchestra.run.smithi186.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 716 kB/s | 36 kB 00:00 2024-05-27T07:25:12.734 INFO:teuthology.orchestra.run.smithi186.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-05-27T07:25:12.759 INFO:teuthology.orchestra.run.smithi186.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-05-27T07:25:12.784 INFO:teuthology.orchestra.run.smithi186.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-27T07:25:12.809 INFO:teuthology.orchestra.run.smithi186.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-05-27T07:25:12.826 INFO:teuthology.orchestra.run.smithi186.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-05-27T07:25:12.843 INFO:teuthology.orchestra.run.smithi186.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-05-27T07:25:12.860 INFO:teuthology.orchestra.run.smithi186.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 667 kB/s | 22 kB 00:00 2024-05-27T07:25:12.902 INFO:teuthology.orchestra.run.smithi186.stdout:(104/159): protobuf-compiler-3.14.0-13.el9.x86_ 14 MB/s | 863 kB 00:00 2024-05-27T07:25:13.020 INFO:teuthology.orchestra.run.smithi163.stdout:(41/159): libgfortran-11.4.1-3.el9.x86_64.rpm 796 kB/s | 803 kB 00:01 2024-05-27T07:25:13.128 INFO:teuthology.orchestra.run.smithi163.stdout:(42/159): libquadmath-11.4.1-3.el9.x86_64.rpm 232 kB/s | 195 kB 00:00 2024-05-27T07:25:13.178 INFO:teuthology.orchestra.run.smithi186.stdout:(105/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-05-27T07:25:13.311 INFO:teuthology.orchestra.run.smithi186.stdout:(106/159): gperftools-libs-2.9.1-2.el9.x86_64.r 809 kB/s | 309 kB 00:00 2024-05-27T07:25:13.336 INFO:teuthology.orchestra.run.smithi186.stdout:(107/159): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2024-05-27T07:25:13.361 INFO:teuthology.orchestra.run.smithi186.stdout:(108/159): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-05-27T07:25:13.387 INFO:teuthology.orchestra.run.smithi186.stdout:(109/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 517 kB/s | 26 kB 00:00 2024-05-27T07:25:13.404 INFO:teuthology.orchestra.run.smithi163.stdout:(43/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 307 kB/s | 253 kB 00:00 2024-05-27T07:25:13.412 INFO:teuthology.orchestra.run.smithi186.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 965 kB/s | 49 kB 00:00 2024-05-27T07:25:13.454 INFO:teuthology.orchestra.run.smithi186.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-05-27T07:25:13.479 INFO:teuthology.orchestra.run.smithi186.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-05-27T07:25:13.613 INFO:teuthology.orchestra.run.smithi186.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-05-27T07:25:13.646 INFO:teuthology.orchestra.run.smithi186.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.3 MB/s | 839 kB 00:00 2024-05-27T07:25:13.672 INFO:teuthology.orchestra.run.smithi186.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 744 kB/s | 43 kB 00:00 2024-05-27T07:25:13.697 INFO:teuthology.orchestra.run.smithi186.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 640 kB/s | 32 kB 00:00 2024-05-27T07:25:13.723 INFO:teuthology.orchestra.run.smithi186.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-05-27T07:25:13.764 INFO:teuthology.orchestra.run.smithi186.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-05-27T07:25:13.840 INFO:teuthology.orchestra.run.smithi186.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-05-27T07:25:13.862 INFO:teuthology.orchestra.run.smithi163.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 145 kB/s | 106 kB 00:00 2024-05-27T07:25:13.882 INFO:teuthology.orchestra.run.smithi186.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-05-27T07:25:13.931 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:18 2024-05-27T07:25:13.940 INFO:teuthology.orchestra.run.smithi186.stdout:(121/159): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-05-27T07:25:14.012 INFO:teuthology.orchestra.run.smithi163.stdout:(45/159): python3-cryptography-36.0.1-4.el9.x86 1.2 MB/s | 1.2 MB 00:00 2024-05-27T07:25:14.032 INFO:teuthology.orchestra.run.smithi186.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-05-27T07:25:14.074 INFO:teuthology.orchestra.run.smithi186.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 257 kB/s | 11 kB 00:00 2024-05-27T07:25:14.116 INFO:teuthology.orchestra.run.smithi186.stdout:(124/159): python3-jaraco-classes-3.2.1-5.el9.n 425 kB/s | 18 kB 00:00 2024-05-27T07:25:14.154 INFO:teuthology.orchestra.run.smithi163.stdout:(46/159): python3-pycparser-2.20-6.el9.noarch.r 180 kB/s | 135 kB 00:00 2024-05-27T07:25:14.158 INFO:teuthology.orchestra.run.smithi186.stdout:(125/159): python3-jaraco-collections-3.0.0-8.e 557 kB/s | 23 kB 00:00 2024-05-27T07:25:14.200 INFO:teuthology.orchestra.run.smithi186.stdout:(126/159): python3-jaraco-functools-3.5.0-2.el9 466 kB/s | 19 kB 00:00 2024-05-27T07:25:14.242 INFO:teuthology.orchestra.run.smithi186.stdout:(127/159): python3-jaraco-text-3.2.0-6.el9.noar 475 kB/s | 20 kB 00:00 2024-05-27T07:25:14.334 INFO:teuthology.orchestra.run.smithi186.stdout:(128/159): libarrow-9.0.0-11.el9.x86_64.rpm 4.3 MB/s | 4.4 MB 00:01 2024-05-27T07:25:14.376 INFO:teuthology.orchestra.run.smithi186.stdout:(129/159): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-05-27T07:25:14.405 INFO:teuthology.orchestra.run.smithi163.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 689 kB 00:00 2024-05-27T07:25:14.418 INFO:teuthology.orchestra.run.smithi186.stdout:(130/159): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-05-27T07:25:14.460 INFO:teuthology.orchestra.run.smithi186.stdout:(131/159): python3-grpcio-1.46.7-10.el9.x86_64. 3.5 MB/s | 2.0 MB 00:00 2024-05-27T07:25:14.494 INFO:teuthology.orchestra.run.smithi186.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 768 kB/s | 58 kB 00:00 2024-05-27T07:25:14.535 INFO:teuthology.orchestra.run.smithi186.stdout:(133/159): python3-portend-3.1.0-2.el9.noarch.r 396 kB/s | 16 kB 00:00 2024-05-27T07:25:14.570 INFO:teuthology.orchestra.run.smithi186.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-05-27T07:25:14.603 INFO:teuthology.orchestra.run.smithi186.stdout:(135/159): python3-kubernetes-26.1.0-2.el9.noar 2.8 MB/s | 1.0 MB 00:00 2024-05-27T07:25:14.604 INFO:teuthology.orchestra.run.smithi163.stdout:(48/159): python3-requests-2.25.1-8.el9.noarch. 168 kB/s | 125 kB 00:00 2024-05-27T07:25:14.645 INFO:teuthology.orchestra.run.smithi186.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 823 kB/s | 90 kB 00:00 2024-05-27T07:25:14.663 INFO:teuthology.orchestra.run.smithi163.stdout:(49/159): bison-3.7.4-5.el9.x86_64.rpm 3.6 MB/s | 943 kB 00:00 2024-05-27T07:25:14.670 INFO:teuthology.orchestra.run.smithi186.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-05-27T07:25:14.695 INFO:teuthology.orchestra.run.smithi186.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-05-27T07:25:14.721 INFO:teuthology.orchestra.run.smithi186.stdout:(139/159): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-05-27T07:25:14.746 INFO:teuthology.orchestra.run.smithi186.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-05-27T07:25:14.771 INFO:teuthology.orchestra.run.smithi186.stdout:(141/159): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-05-27T07:25:14.772 INFO:teuthology.orchestra.run.smithi163.stdout:(50/159): boost-program-options-1.75.0-8.el9.x8 642 kB/s | 107 kB 00:00 2024-05-27T07:25:14.788 INFO:teuthology.orchestra.run.smithi186.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 763 kB/s | 51 kB 00:00 2024-05-27T07:25:14.789 INFO:teuthology.orchestra.run.smithi163.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 173 kB/s | 22 kB 00:00 2024-05-27T07:25:14.822 INFO:teuthology.orchestra.run.smithi163.stdout:(52/159): python3-urllib3-1.26.5-5.el9.noarch.r 265 kB/s | 215 kB 00:00 2024-05-27T07:25:14.830 INFO:teuthology.orchestra.run.smithi186.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-05-27T07:25:14.855 INFO:teuthology.orchestra.run.smithi186.stdout:(144/159): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-05-27T07:25:14.872 INFO:teuthology.orchestra.run.smithi186.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 477 kB/s | 20 kB 00:00 2024-05-27T07:25:14.906 INFO:teuthology.orchestra.run.smithi186.stdout:(146/159): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-05-27T07:25:14.906 INFO:teuthology.orchestra.run.smithi163.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 279 kB/s | 33 kB 00:00 2024-05-27T07:25:14.931 INFO:teuthology.orchestra.run.smithi186.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-05-27T07:25:15.015 INFO:teuthology.orchestra.run.smithi163.stdout:(54/159): flex-2.6.4-9.el9.x86_64.rpm 1.3 MB/s | 313 kB 00:00 2024-05-27T07:25:15.032 INFO:teuthology.orchestra.run.smithi163.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 146 kB/s | 18 kB 00:00 2024-05-27T07:25:15.090 INFO:teuthology.orchestra.run.smithi163.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 149 kB/s | 11 kB 00:00 2024-05-27T07:25:15.199 INFO:teuthology.orchestra.run.smithi163.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 959 kB/s | 160 kB 00:00 2024-05-27T07:25:15.206 INFO:teuthology.orchestra.run.smithi186.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 4.7 MB/s | 1.6 MB 00:00 2024-05-27T07:25:15.216 INFO:teuthology.orchestra.run.smithi163.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 362 kB/s | 45 kB 00:00 2024-05-27T07:25:15.333 INFO:teuthology.orchestra.run.smithi163.stdout:(59/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.9 MB/s | 3.0 MB 00:00 2024-05-27T07:25:15.348 INFO:teuthology.orchestra.run.smithi186.stdout:(149/159): librbd1-19.0.0-3872.g38f9d900.el9.x8 7.4 MB/s | 3.1 MB 00:00 2024-05-27T07:25:15.440 INFO:teuthology.orchestra.run.smithi186.stdout:(150/159): librados2-19.0.0-3872.g38f9d900.el9. 6.2 MB/s | 3.3 MB 00:00 2024-05-27T07:25:15.441 INFO:teuthology.orchestra.run.smithi163.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 235 kB/s | 25 kB 00:00 2024-05-27T07:25:15.466 INFO:teuthology.orchestra.run.smithi163.stdout:(61/159): libtool-2.4.6-45.el9.x86_64.rpm 2.3 MB/s | 582 kB 00:00 2024-05-27T07:25:15.500 INFO:teuthology.orchestra.run.smithi163.stdout:(62/159): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 662 kB 00:00 2024-05-27T07:25:15.609 INFO:teuthology.orchestra.run.smithi163.stdout:(63/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 284 kB/s | 47 kB 00:00 2024-05-27T07:25:15.668 INFO:teuthology.orchestra.run.smithi163.stdout:(64/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-05-27T07:25:15.699 INFO:teuthology.orchestra.run.smithi186.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 161 kB/s | 41 kB 00:00 2024-05-27T07:25:15.701 INFO:teuthology.orchestra.run.smithi163.stdout:(65/159): lua-5.4.4-4.el9.x86_64.rpm 938 kB/s | 188 kB 00:00 2024-05-27T07:25:15.727 INFO:teuthology.orchestra.run.smithi163.stdout:(66/159): openblas-0.3.21-2.el9.x86_64.rpm 300 kB/s | 35 kB 00:00 2024-05-27T07:25:16.069 INFO:teuthology.orchestra.run.smithi163.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 4.6 MB/s | 1.6 MB 00:00 2024-05-27T07:25:16.277 INFO:teuthology.orchestra.run.smithi163.stdout:(68/159): openblas-openmp-0.3.21-2.el9.x86_64.r 7.8 MB/s | 4.7 MB 00:00 2024-05-27T07:25:16.336 INFO:teuthology.orchestra.run.smithi163.stdout:(69/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.0 MB/s | 4.4 MB 00:00 2024-05-27T07:25:16.386 INFO:teuthology.orchestra.run.smithi163.stdout:(70/159): postgresql-contrib-13.14-1.el9.x86_64 2.8 MB/s | 911 kB 00:00 2024-05-27T07:25:16.478 INFO:teuthology.orchestra.run.smithi163.stdout:(71/159): postgresql-private-libs-13.14-1.el9.x 665 kB/s | 133 kB 00:00 2024-05-27T07:25:16.516 INFO:teuthology.orchestra.run.smithi186.stdout:(152/159): NetworkManager-tui-1.47.90-1.el9.x86 304 kB/s | 248 kB 00:00 2024-05-27T07:25:16.611 INFO:teuthology.orchestra.run.smithi163.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 657 kB/s | 87 kB 00:00 2024-05-27T07:25:16.678 INFO:teuthology.orchestra.run.smithi163.stdout:(73/159): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-05-27T07:25:16.774 INFO:teuthology.orchestra.run.smithi186.stdout:(153/159): elfutils-debuginfod-client-0.191-4.e 142 kB/s | 37 kB 00:00 2024-05-27T07:25:16.804 INFO:teuthology.orchestra.run.smithi163.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 490 kB/s | 61 kB 00:00 2024-05-27T07:25:16.979 INFO:teuthology.orchestra.run.smithi163.stdout:(75/159): postgresql-server-13.14-1.el9.x86_64. 9.0 MB/s | 5.8 MB 00:00 2024-05-27T07:25:17.088 INFO:teuthology.orchestra.run.smithi163.stdout:(76/159): python3-devel-3.9.18-3.el9.x86_64.rpm 861 kB/s | 244 kB 00:00 2024-05-27T07:25:17.171 INFO:teuthology.orchestra.run.smithi163.stdout:(77/159): python3-jinja2-2.11.3-6.el9.noarch.rp 1.3 MB/s | 248 kB 00:00 2024-05-27T07:25:17.188 INFO:teuthology.orchestra.run.smithi163.stdout:(78/159): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-05-27T07:25:17.255 INFO:teuthology.orchestra.run.smithi163.stdout:(79/159): python3-babel-2.9.1-2.el9.noarch.rpm 9.3 MB/s | 6.0 MB 00:00 2024-05-27T07:25:17.274 INFO:teuthology.orchestra.run.smithi186.stdout:(154/159): elfutils-libelf-0.191-4.el9.x86_64.r 414 kB/s | 207 kB 00:00 2024-05-27T07:25:17.330 INFO:teuthology.orchestra.run.smithi163.stdout:(80/159): python3-jsonpointer-2.0-4.el9.noarch. 121 kB/s | 19 kB 00:00 2024-05-27T07:25:17.356 INFO:teuthology.orchestra.run.smithi163.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 347 kB/s | 35 kB 00:00 2024-05-27T07:25:17.414 INFO:teuthology.orchestra.run.smithi163.stdout:(82/159): python3-mako-1.1.4-6.el9.noarch.rpm 762 kB/s | 172 kB 00:00 2024-05-27T07:25:17.548 INFO:teuthology.orchestra.run.smithi163.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 250 kB 00:00 2024-05-27T07:25:17.606 INFO:teuthology.orchestra.run.smithi163.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 222 kB 00:00 2024-05-27T07:25:17.715 INFO:teuthology.orchestra.run.smithi163.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 462 kB/s | 77 kB 00:00 2024-05-27T07:25:17.732 INFO:teuthology.orchestra.run.smithi163.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 337 kB/s | 42 kB 00:00 2024-05-27T07:25:17.858 INFO:teuthology.orchestra.run.smithi186.stdout:(155/159): NetworkManager-1.47.90-1.el9.x86_64. 892 kB/s | 2.3 MB 00:02 2024-05-27T07:25:17.907 INFO:teuthology.orchestra.run.smithi163.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 1.4 MB/s | 269 kB 00:00 2024-05-27T07:25:17.932 INFO:teuthology.orchestra.run.smithi163.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 792 kB/s | 159 kB 00:00 2024-05-27T07:25:18.000 INFO:teuthology.orchestra.run.smithi163.stdout:(89/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.6 MB/s | 5.1 MB 00:00 2024-05-27T07:25:18.025 INFO:teuthology.orchestra.run.smithi186.stdout:(156/159): NetworkManager-libnm-1.47.90-1.el9.x 690 kB/s | 1.8 MB 00:02 2024-05-27T07:25:18.050 INFO:teuthology.orchestra.run.smithi186.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 335 kB/s | 260 kB 00:00 2024-05-27T07:25:18.108 INFO:teuthology.orchestra.run.smithi163.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 291 kB/s | 51 kB 00:00 2024-05-27T07:25:18.125 INFO:teuthology.orchestra.run.smithi163.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 429 kB/s | 54 kB 00:00 2024-05-27T07:25:18.192 INFO:teuthology.orchestra.run.smithi163.stdout:(92/159): python3-pyasn1-modules-0.4.8-6.el9.no 981 kB/s | 279 kB 00:00 2024-05-27T07:25:18.250 INFO:teuthology.orchestra.run.smithi163.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 332 kB/s | 42 kB 00:00 2024-05-27T07:25:18.318 INFO:teuthology.orchestra.run.smithi163.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 353 kB/s | 44 kB 00:00 2024-05-27T07:25:18.426 INFO:teuthology.orchestra.run.smithi163.stdout:(95/159): qatlib-service-24.02.0-1.el9.x86_64.r 331 kB/s | 36 kB 00:00 2024-05-27T07:25:18.452 INFO:teuthology.orchestra.run.smithi163.stdout:(96/159): qatlib-24.02.0-1.el9.x86_64.rpm 1.1 MB/s | 221 kB 00:00 2024-05-27T07:25:18.602 INFO:teuthology.orchestra.run.smithi163.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 270 kB/s | 47 kB 00:00 2024-05-27T07:25:18.802 INFO:teuthology.orchestra.run.smithi163.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 870 kB/s | 305 kB 00:00 2024-05-27T07:25:18.903 INFO:teuthology.orchestra.run.smithi163.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 192 kB/s | 58 kB 00:00 2024-05-27T07:25:19.011 INFO:teuthology.orchestra.run.smithi163.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 449 kB/s | 94 kB 00:00 2024-05-27T07:25:19.036 INFO:teuthology.orchestra.run.smithi163.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 477 kB/s | 64 kB 00:00 2024-05-27T07:25:19.162 INFO:teuthology.orchestra.run.smithi163.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 299 kB/s | 45 kB 00:00 2024-05-27T07:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-05-27T07:25:19.304 INFO:teuthology.orchestra.run.smithi163.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 83 kB/s | 22 kB 00:00 2024-05-27T07:25:19.462 INFO:teuthology.orchestra.run.smithi163.stdout:(104/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 12 MB/s | 16 MB 00:01 2024-05-27T07:25:19.904 INFO:teuthology.orchestra.run.smithi163.stdout:(105/159): abseil-cpp-20211102.0-3.el9.x86_64.r 913 kB/s | 548 kB 00:00 2024-05-27T07:25:19.929 INFO:teuthology.orchestra.run.smithi163.stdout:(106/159): gperftools-libs-2.9.1-2.el9.x86_64.r 662 kB/s | 309 kB 00:00 2024-05-27T07:25:19.996 INFO:teuthology.orchestra.run.smithi163.stdout:(107/159): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-05-27T07:25:20.075 INFO:teuthology.orchestra.run.smithi186.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 623 kB/s | 1.3 MB 00:02 2024-05-27T07:25:20.096 INFO:teuthology.orchestra.run.smithi163.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-05-27T07:25:20.197 INFO:teuthology.orchestra.run.smithi163.stdout:(109/159): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-05-27T07:25:20.330 INFO:teuthology.orchestra.run.smithi163.stdout:(110/159): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-05-27T07:25:20.356 INFO:teuthology.orchestra.run.smithi163.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 424 kB/s | 67 kB 00:00 2024-05-27T07:25:20.427 INFO:teuthology.orchestra.run.smithi163.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-05-27T07:25:20.531 INFO:teuthology.orchestra.run.smithi163.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-05-27T07:25:20.556 INFO:teuthology.orchestra.run.smithi163.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.1 MB/s | 839 kB 00:00 2024-05-27T07:25:20.623 INFO:teuthology.orchestra.run.smithi163.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 474 kB/s | 43 kB 00:00 2024-05-27T07:25:20.649 INFO:teuthology.orchestra.run.smithi163.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 350 kB/s | 32 kB 00:00 2024-05-27T07:25:20.716 INFO:teuthology.orchestra.run.smithi163.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-05-27T07:25:20.741 INFO:teuthology.orchestra.run.smithi163.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-05-27T07:25:20.766 INFO:teuthology.orchestra.run.smithi163.stdout:(119/159): protobuf-compiler-3.14.0-13.el9.x86_ 538 kB/s | 863 kB 00:01 2024-05-27T07:25:20.816 INFO:teuthology.orchestra.run.smithi163.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-05-27T07:25:20.834 INFO:teuthology.orchestra.run.smithi186.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 873 kB/s | 2.4 MB 00:02 2024-05-27T07:25:20.841 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:25:20.841 INFO:teuthology.orchestra.run.smithi186.stdout:Total 11 MB/s | 219 MB 00:20 2024-05-27T07:25:20.842 INFO:teuthology.orchestra.run.smithi163.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-05-27T07:25:20.942 INFO:teuthology.orchestra.run.smithi163.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-05-27T07:25:21.043 INFO:teuthology.orchestra.run.smithi163.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-05-27T07:25:21.135 INFO:teuthology.orchestra.run.smithi163.stdout:(124/159): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2024-05-27T07:25:21.226 INFO:teuthology.orchestra.run.smithi163.stdout:(125/159): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-05-27T07:25:21.252 INFO:teuthology.orchestra.run.smithi163.stdout:(126/159): python3-google-auth-2.29.0-1.el9.noa 424 kB/s | 206 kB 00:00 2024-05-27T07:25:21.319 INFO:teuthology.orchestra.run.smithi163.stdout:(127/159): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-05-27T07:25:21.352 INFO:teuthology.orchestra.run.smithi163.stdout:(128/159): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-05-27T07:25:21.421 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:25:21.422 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:25:21.428 INFO:teuthology.orchestra.run.smithi163.stdout:(129/159): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2024-05-27T07:25:21.433 INFO:teuthology.orchestra.run.smithi114.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T07:25:21.433 INFO:teuthology.orchestra.run.smithi114.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T07:25:21.434 INFO:teuthology.orchestra.run.smithi114.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T07:25:21.435 INFO:teuthology.orchestra.run.smithi114.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T07:25:21.437 INFO:teuthology.orchestra.run.smithi114.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T07:25:21.439 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T07:25:21.440 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T07:25:21.453 INFO:teuthology.orchestra.run.smithi163.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-05-27T07:25:21.502 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-05-27T07:25:21.520 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-05-27T07:25:21.520 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-05-27T07:25:21.521 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-05-27T07:25:21.521 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-05-27T07:25:21.521 INFO:teuthology.orchestra.run.smithi114.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T07:25:21.521 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 6.6 k 2024-05-27T07:25:21.521 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.4 M 2024-05-27T07:25:21.521 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 863 k 2024-05-27T07:25:21.521 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 147 k 2024-05-27T07:25:21.521 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 1.5 M 2024-05-27T07:25:21.521 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 143 k 2024-05-27T07:25:21.521 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 3.5 M 2024-05-27T07:25:21.522 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 7.4 M 2024-05-27T07:25:21.522 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 49 k 2024-05-27T07:25:21.522 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 10 M 2024-05-27T07:25:21.522 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 43 M 2024-05-27T07:25:21.522 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 273 k 2024-05-27T07:25:21.522 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 764 k 2024-05-27T07:25:21.522 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 224 k 2024-05-27T07:25:21.522 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 27 k 2024-05-27T07:25:21.522 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 18 k 2024-05-27T07:25:21.522 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T07:25:21.522 INFO:teuthology.orchestra.run.smithi114.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T07:25:21.523 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T07:25:21.523 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 32 k 2024-05-27T07:25:21.523 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 755 k 2024-05-27T07:25:21.523 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 127 k 2024-05-27T07:25:21.523 INFO:teuthology.orchestra.run.smithi114.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T07:25:21.523 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T07:25:21.523 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T07:25:21.523 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T07:25:21.523 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T07:25:21.523 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T07:25:21.523 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 161 k 2024-05-27T07:25:21.523 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 320 k 2024-05-27T07:25:21.523 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 300 k 2024-05-27T07:25:21.524 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 99 k 2024-05-27T07:25:21.524 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 86 k 2024-05-27T07:25:21.524 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.0 M 2024-05-27T07:25:21.524 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 169 k 2024-05-27T07:25:21.524 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T07:25:21.524 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-05-27T07:25:21.524 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T07:25:21.524 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T07:25:21.524 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T07:25:21.524 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T07:25:21.524 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T07:25:21.524 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T07:25:21.524 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T07:25:21.525 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.3 M 2024-05-27T07:25:21.525 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.1 M 2024-05-27T07:25:21.525 INFO:teuthology.orchestra.run.smithi114.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T07:25:21.525 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T07:25:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-05-27T07:25:21.525 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T07:25:21.525 INFO:teuthology.orchestra.run.smithi114.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T07:25:21.525 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T07:25:21.525 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 20 M 2024-05-27T07:25:21.525 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 34 k 2024-05-27T07:25:21.525 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 2.3 M 2024-05-27T07:25:21.525 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 274 k 2024-05-27T07:25:21.526 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 4.9 M 2024-05-27T07:25:21.526 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 18 M 2024-05-27T07:25:21.526 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 16 k 2024-05-27T07:25:21.526 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 25 k 2024-05-27T07:25:21.526 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T07:25:21.526 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T07:25:21.526 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T07:25:21.526 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T07:25:21.526 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T07:25:21.526 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T07:25:21.526 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T07:25:21.526 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T07:25:21.527 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 157 k 2024-05-27T07:25:21.527 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T07:25:21.527 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T07:25:21.527 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T07:25:21.527 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T07:25:21.527 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T07:25:21.527 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 498 k 2024-05-27T07:25:21.527 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T07:25:21.527 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.2 M 2024-05-27T07:25:21.527 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T07:25:21.527 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T07:25:21.527 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T07:25:21.527 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T07:25:21.528 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T07:25:21.528 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T07:25:21.528 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T07:25:21.528 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T07:25:21.528 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T07:25:21.528 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T07:25:21.528 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T07:25:21.528 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T07:25:21.528 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T07:25:21.528 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T07:25:21.528 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T07:25:21.528 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T07:25:21.529 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T07:25:21.529 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 45 k 2024-05-27T07:25:21.529 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 136 k 2024-05-27T07:25:21.529 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T07:25:21.529 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T07:25:21.529 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T07:25:21.529 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T07:25:21.529 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T07:25:21.529 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T07:25:21.529 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T07:25:21.529 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T07:25:21.529 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T07:25:21.529 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T07:25:21.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T07:25:21.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T07:25:21.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T07:25:21.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T07:25:21.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T07:25:21.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T07:25:21.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T07:25:21.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T07:25:21.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T07:25:21.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T07:25:21.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T07:25:21.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T07:25:21.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T07:25:21.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T07:25:21.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T07:25:21.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T07:25:21.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T07:25:21.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T07:25:21.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T07:25:21.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T07:25:21.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T07:25:21.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T07:25:21.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T07:25:21.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T07:25:21.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T07:25:21.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T07:25:21.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T07:25:21.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T07:25:21.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T07:25:21.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T07:25:21.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T07:25:21.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T07:25:21.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T07:25:21.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T07:25:21.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T07:25:21.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T07:25:21.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T07:25:21.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T07:25:21.533 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T07:25:21.533 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T07:25:21.533 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T07:25:21.533 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T07:25:21.533 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T07:25:21.533 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T07:25:21.533 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T07:25:21.533 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T07:25:21.533 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T07:25:21.533 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T07:25:21.533 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T07:25:21.533 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T07:25:21.534 INFO:teuthology.orchestra.run.smithi114.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T07:25:21.534 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T07:25:21.534 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T07:25:21.534 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-05-27T07:25:21.534 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T07:25:21.534 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:25:21.534 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-05-27T07:25:21.534 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-05-27T07:25:21.534 INFO:teuthology.orchestra.run.smithi114.stdout:Install 148 Packages 2024-05-27T07:25:21.534 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 11 Packages 2024-05-27T07:25:21.534 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:25:21.535 INFO:teuthology.orchestra.run.smithi163.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 858 kB/s | 79 kB 00:00 2024-05-27T07:25:21.535 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 219 M 2024-05-27T07:25:21.535 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-05-27T07:25:21.553 INFO:teuthology.orchestra.run.smithi163.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-05-27T07:25:21.612 INFO:teuthology.orchestra.run.smithi163.stdout:(133/159): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-05-27T07:25:21.646 INFO:teuthology.orchestra.run.smithi163.stdout:(134/159): python3-portend-3.1.0-2.el9.noarch.r 178 kB/s | 16 kB 00:00 2024-05-27T07:25:21.704 INFO:teuthology.orchestra.run.smithi163.stdout:(135/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 978 kB/s | 90 kB 00:00 2024-05-27T07:25:21.746 INFO:teuthology.orchestra.run.smithi163.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 282 kB/s | 28 kB 00:00 2024-05-27T07:25:21.797 INFO:teuthology.orchestra.run.smithi163.stdout:(137/159): python3-repoze-lru-0.7-16.el9.noarch 335 kB/s | 31 kB 00:00 2024-05-27T07:25:21.847 INFO:teuthology.orchestra.run.smithi163.stdout:(138/159): python3-grpcio-1.46.7-10.el9.x86_64. 2.0 MB/s | 2.0 MB 00:01 2024-05-27T07:25:21.880 INFO:teuthology.orchestra.run.smithi163.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-05-27T07:25:21.906 INFO:teuthology.orchestra.run.smithi163.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 543 kB/s | 59 kB 00:00 2024-05-27T07:25:21.939 INFO:teuthology.orchestra.run.smithi163.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-05-27T07:25:21.981 INFO:teuthology.orchestra.run.smithi163.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-05-27T07:25:22.007 INFO:teuthology.orchestra.run.smithi163.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-05-27T07:25:22.032 INFO:teuthology.orchestra.run.smithi163.stdout:(144/159): python3-websocket-client-1.2.3-2.el9 972 kB/s | 90 kB 00:00 2024-05-27T07:25:22.075 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-27T07:25:22.099 INFO:teuthology.orchestra.run.smithi163.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-05-27T07:25:22.124 INFO:teuthology.orchestra.run.smithi163.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-05-27T07:25:22.149 INFO:teuthology.orchestra.run.smithi163.stdout:(147/159): python3-werkzeug-2.0.3-3.el9.1.noarc 2.5 MB/s | 427 kB 00:00 2024-05-27T07:25:22.154 INFO:teuthology.orchestra.run.smithi114.stdout:(1/159): ceph-19.0.0-3872.g38f9d900.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-05-27T07:25:22.237 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-27T07:25:22.237 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-27T07:25:22.413 INFO:teuthology.orchestra.run.smithi114.stdout:(2/159): ceph-fuse-19.0.0-3872.g38f9d900.el9.x8 3.3 MB/s | 863 kB 00:00 2024-05-27T07:25:22.488 INFO:teuthology.orchestra.run.smithi114.stdout:(3/159): ceph-immutable-object-cache-19.0.0-387 1.9 MB/s | 147 kB 00:00 2024-05-27T07:25:22.566 INFO:teuthology.orchestra.run.smithi163.stdout:(148/159): librados2-19.0.0-3872.g38f9d900.el9. 7.5 MB/s | 3.3 MB 00:00 2024-05-27T07:25:22.808 INFO:teuthology.orchestra.run.smithi163.stdout:(149/159): thrift-0.15.0-2.el9.x86_64.rpm 2.2 MB/s | 1.6 MB 00:00 2024-05-27T07:25:22.813 INFO:teuthology.orchestra.run.smithi114.stdout:(4/159): ceph-mds-19.0.0-3872.g38f9d900.el9.x86 7.0 MB/s | 2.3 MB 00:00 2024-05-27T07:25:23.080 INFO:teuthology.orchestra.run.smithi114.stdout:(5/159): ceph-mgr-19.0.0-3872.g38f9d900.el9.x86 5.6 MB/s | 1.5 MB 00:00 2024-05-27T07:25:23.108 INFO:teuthology.orchestra.run.smithi163.stdout:(150/159): librbd1-19.0.0-3872.g38f9d900.el9.x8 3.2 MB/s | 3.1 MB 00:00 2024-05-27T07:25:23.539 INFO:teuthology.orchestra.run.smithi114.stdout:(6/159): ceph-base-19.0.0-3872.g38f9d900.el9.x8 3.5 MB/s | 5.4 MB 00:01 2024-05-27T07:25:23.708 INFO:teuthology.orchestra.run.smithi163.stdout:(151/159): NetworkManager-1.47.90-1.el9.x86_64. 2.0 MB/s | 2.3 MB 00:01 2024-05-27T07:25:23.742 INFO:teuthology.orchestra.run.smithi163.stdout:(152/159): NetworkManager-team-1.47.90-1.el9.x8 68 kB/s | 41 kB 00:00 2024-05-27T07:25:23.864 INFO:teuthology.orchestra.run.smithi114.stdout:(7/159): ceph-common-19.0.0-3872.g38f9d900.el9. 10 MB/s | 20 MB 00:01 2024-05-27T07:25:23.931 INFO:teuthology.orchestra.run.smithi114.stdout:(8/159): ceph-mon-19.0.0-3872.g38f9d900.el9.x86 5.7 MB/s | 4.9 MB 00:00 2024-05-27T07:25:23.998 INFO:teuthology.orchestra.run.smithi114.stdout:(9/159): ceph-selinux-19.0.0-3872.g38f9d900.el9 378 kB/s | 25 kB 00:00 2024-05-27T07:25:24.010 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-27T07:25:24.011 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-27T07:25:24.018 INFO:teuthology.orchestra.run.smithi163.stdout:(153/159): NetworkManager-libnm-1.47.90-1.el9.x 1.5 MB/s | 1.8 MB 00:01 2024-05-27T07:25:24.426 INFO:teuthology.orchestra.run.smithi163.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 54 kB/s | 37 kB 00:00 2024-05-27T07:25:24.560 INFO:teuthology.orchestra.run.smithi163.stdout:(155/159): NetworkManager-tui-1.47.90-1.el9.x86 291 kB/s | 248 kB 00:00 2024-05-27T07:25:24.648 INFO:teuthology.orchestra.run.smithi114.stdout:(10/159): ceph-radosgw-19.0.0-3872.g38f9d900.el 13 MB/s | 10 MB 00:00 2024-05-27T07:25:24.715 INFO:teuthology.orchestra.run.smithi114.stdout:(11/159): cephfs-mirror-19.0.0-3872.g38f9d900.e 3.3 MB/s | 224 kB 00:00 2024-05-27T07:25:24.774 INFO:teuthology.orchestra.run.smithi114.stdout:(12/159): cephfs-shell-19.0.0-3872.g38f9d900.el 464 kB/s | 27 kB 00:00 2024-05-27T07:25:24.841 INFO:teuthology.orchestra.run.smithi114.stdout:(13/159): libcephfs-devel-19.0.0-3872.g38f9d900 476 kB/s | 32 kB 00:00 2024-05-27T07:25:24.843 INFO:teuthology.orchestra.run.smithi163.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 259 kB/s | 207 kB 00:00 2024-05-27T07:25:24.933 INFO:teuthology.orchestra.run.smithi114.stdout:(14/159): libcephfs2-19.0.0-3872.g38f9d900.el9. 8.1 MB/s | 755 kB 00:00 2024-05-27T07:25:25.000 INFO:teuthology.orchestra.run.smithi114.stdout:(15/159): libcephsqlite-19.0.0-3872.g38f9d900.e 2.3 MB/s | 157 kB 00:00 2024-05-27T07:25:25.075 INFO:teuthology.orchestra.run.smithi114.stdout:(16/159): librados-devel-19.0.0-3872.g38f9d900. 1.6 MB/s | 127 kB 00:00 2024-05-27T07:25:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:(17/159): libradosstriper1-19.0.0-3872.g38f9d90 6.5 MB/s | 498 kB 00:00 2024-05-27T07:25:25.269 INFO:teuthology.orchestra.run.smithi163.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 308 kB/s | 260 kB 00:00 2024-05-27T07:25:25.576 INFO:teuthology.orchestra.run.smithi114.stdout:(18/159): librgw2-19.0.0-3872.g38f9d900.el9.x86 12 MB/s | 5.2 MB 00:00 2024-05-27T07:25:25.585 INFO:teuthology.orchestra.run.smithi163.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 1.3 MB/s | 1.3 MB 00:01 2024-05-27T07:25:25.642 INFO:teuthology.orchestra.run.smithi114.stdout:(19/159): python3-ceph-argparse-19.0.0-3872.g38 679 kB/s | 45 kB 00:00 2024-05-27T07:25:25.709 INFO:teuthology.orchestra.run.smithi114.stdout:(20/159): python3-ceph-common-19.0.0-3872.g38f9 2.0 MB/s | 136 kB 00:00 2024-05-27T07:25:25.776 INFO:teuthology.orchestra.run.smithi114.stdout:(21/159): python3-cephfs-19.0.0-3872.g38f9d900. 2.4 MB/s | 161 kB 00:00 2024-05-27T07:25:25.852 INFO:teuthology.orchestra.run.smithi114.stdout:(22/159): python3-rados-19.0.0-3872.g38f9d900.e 4.2 MB/s | 320 kB 00:00 2024-05-27T07:25:25.927 INFO:teuthology.orchestra.run.smithi114.stdout:(23/159): python3-rbd-19.0.0-3872.g38f9d900.el9 3.9 MB/s | 300 kB 00:00 2024-05-27T07:25:25.969 INFO:teuthology.orchestra.run.smithi163.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.1 MB/s | 2.4 MB 00:01 2024-05-27T07:25:25.973 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:25:25.973 INFO:teuthology.orchestra.run.smithi163.stdout:Total 9.1 MB/s | 219 MB 00:24 2024-05-27T07:25:25.994 INFO:teuthology.orchestra.run.smithi114.stdout:(24/159): python3-rgw-19.0.0-3872.g38f9d900.el9 1.5 MB/s | 99 kB 00:00 2024-05-27T07:25:26.061 INFO:teuthology.orchestra.run.smithi114.stdout:(25/159): rbd-fuse-19.0.0-3872.g38f9d900.el9.x8 1.3 MB/s | 86 kB 00:00 2024-05-27T07:25:26.387 INFO:teuthology.orchestra.run.smithi114.stdout:(26/159): ceph-osd-19.0.0-3872.g38f9d900.el9.x8 6.2 MB/s | 18 MB 00:02 2024-05-27T07:25:26.437 INFO:teuthology.orchestra.run.smithi114.stdout:(27/159): rbd-mirror-19.0.0-3872.g38f9d900.el9. 8.1 MB/s | 3.0 MB 00:00 2024-05-27T07:25:26.462 INFO:teuthology.orchestra.run.smithi114.stdout:(28/159): rbd-nbd-19.0.0-3872.g38f9d900.el9.x86 2.2 MB/s | 169 kB 00:00 2024-05-27T07:25:26.487 INFO:teuthology.orchestra.run.smithi114.stdout:(29/159): ceph-grafana-dashboards-19.0.0-3872.g 683 kB/s | 34 kB 00:00 2024-05-27T07:25:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:(30/159): ceph-mgr-cephadm-19.0.0-3872.g38f9d90 1.9 MB/s | 143 kB 00:00 2024-05-27T07:25:26.796 INFO:teuthology.orchestra.run.smithi114.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3872.g38f9d 11 MB/s | 3.5 MB 00:00 2024-05-27T07:25:26.888 INFO:teuthology.orchestra.run.smithi114.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3872.g38 2.9 MB/s | 274 kB 00:00 2024-05-27T07:25:26.955 INFO:teuthology.orchestra.run.smithi114.stdout:(33/159): ceph-mgr-rook-19.0.0-3872.g38f9d900.e 740 kB/s | 49 kB 00:00 2024-05-27T07:25:27.039 INFO:teuthology.orchestra.run.smithi114.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3872.g3 197 kB/s | 16 kB 00:00 2024-05-27T07:25:27.122 INFO:teuthology.orchestra.run.smithi114.stdout:(35/159): ceph-volume-19.0.0-3872.g38f9d900.el9 3.2 MB/s | 273 kB 00:00 2024-05-27T07:25:27.159 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-27T07:25:27.223 INFO:teuthology.orchestra.run.smithi114.stdout:(36/159): cephadm-19.0.0-3872.g38f9d900.el9.noa 7.5 MB/s | 764 kB 00:00 2024-05-27T07:25:27.290 INFO:teuthology.orchestra.run.smithi114.stdout:(37/159): cephfs-top-19.0.0-3872.g38f9d900.el9. 274 kB/s | 18 kB 00:00 2024-05-27T07:25:27.315 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-27T07:25:27.316 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-27T07:25:27.665 INFO:teuthology.orchestra.run.smithi114.stdout:(38/159): ceph-test-19.0.0-3872.g38f9d900.el9.x 12 MB/s | 43 MB 00:03 2024-05-27T07:25:27.757 INFO:teuthology.orchestra.run.smithi114.stdout:(39/159): ceph-mgr-diskprediction-local-19.0.0- 6.1 MB/s | 7.4 MB 00:01 2024-05-27T07:25:28.007 INFO:teuthology.orchestra.run.smithi114.stdout:(40/159): fuse-2.9.9-15.el9.x86_64.rpm 112 kB/s | 80 kB 00:00 2024-05-27T07:25:28.416 INFO:teuthology.orchestra.run.smithi114.stdout:(41/159): libquadmath-11.4.1-3.el9.x86_64.rpm 297 kB/s | 195 kB 00:00 2024-05-27T07:25:28.733 INFO:teuthology.orchestra.run.smithi114.stdout:(42/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 349 kB/s | 253 kB 00:00 2024-05-27T07:25:29.099 INFO:teuthology.orchestra.run.smithi114.stdout:(43/159): libgfortran-11.4.1-3.el9.x86_64.rpm 560 kB/s | 803 kB 00:01 2024-05-27T07:25:29.124 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-27T07:25:29.126 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-27T07:25:29.225 INFO:teuthology.orchestra.run.smithi114.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 217 kB/s | 106 kB 00:00 2024-05-27T07:25:29.666 INFO:teuthology.orchestra.run.smithi114.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 238 kB/s | 135 kB 00:00 2024-05-27T07:25:29.717 INFO:teuthology.orchestra.run.smithi114.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 254 kB/s | 125 kB 00:00 2024-05-27T07:25:29.917 INFO:teuthology.orchestra.run.smithi114.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 689 kB 00:00 2024-05-27T07:25:29.992 INFO:teuthology.orchestra.run.smithi114.stdout:(48/159): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 943 kB 00:00 2024-05-27T07:25:30.059 INFO:teuthology.orchestra.run.smithi114.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-05-27T07:25:30.101 INFO:teuthology.orchestra.run.smithi114.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 522 kB/s | 22 kB 00:00 2024-05-27T07:25:30.160 INFO:teuthology.orchestra.run.smithi114.stdout:(51/159): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 313 kB 00:00 2024-05-27T07:25:30.202 INFO:teuthology.orchestra.run.smithi114.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 788 kB/s | 33 kB 00:00 2024-05-27T07:25:30.288 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-27T07:25:30.310 INFO:teuthology.orchestra.run.smithi114.stdout:(53/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-05-27T07:25:30.336 INFO:teuthology.orchestra.run.smithi114.stdout:(54/159): python3-urllib3-1.26.5-5.el9.noarch.r 321 kB/s | 215 kB 00:00 2024-05-27T07:25:30.361 INFO:teuthology.orchestra.run.smithi114.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 362 kB/s | 18 kB 00:00 2024-05-27T07:25:30.363 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T07:25:30.401 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T07:25:30.403 INFO:teuthology.orchestra.run.smithi114.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 166 kB/s | 11 kB 00:00 2024-05-27T07:25:30.429 INFO:teuthology.orchestra.run.smithi114.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-05-27T07:25:30.455 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T07:25:30.462 INFO:teuthology.orchestra.run.smithi114.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 771 kB/s | 45 kB 00:00 2024-05-27T07:25:30.489 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T07:25:30.496 INFO:teuthology.orchestra.run.smithi114.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-05-27T07:25:30.514 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T07:25:30.529 INFO:teuthology.orchestra.run.smithi114.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 765 kB/s | 25 kB 00:00 2024-05-27T07:25:30.556 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T07:25:30.563 INFO:teuthology.orchestra.run.smithi114.stdout:(61/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-05-27T07:25:30.589 INFO:teuthology.orchestra.run.smithi114.stdout:(62/159): libtool-2.4.6-45.el9.x86_64.rpm 4.5 MB/s | 582 kB 00:00 2024-05-27T07:25:30.622 INFO:teuthology.orchestra.run.smithi114.stdout:(63/159): python3-cryptography-36.0.1-4.el9.x86 569 kB/s | 1.2 MB 00:02 2024-05-27T07:25:30.647 INFO:teuthology.orchestra.run.smithi114.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-05-27T07:25:30.673 INFO:teuthology.orchestra.run.smithi114.stdout:(65/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-05-27T07:25:30.715 INFO:teuthology.orchestra.run.smithi114.stdout:(66/159): openblas-0.3.21-2.el9.x86_64.rpm 382 kB/s | 35 kB 00:00 2024-05-27T07:25:30.807 INFO:teuthology.orchestra.run.smithi114.stdout:(67/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 33 MB/s | 4.4 MB 00:00 2024-05-27T07:25:30.873 INFO:teuthology.orchestra.run.smithi114.stdout:(68/159): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-05-27T07:25:30.884 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T07:25:30.907 INFO:teuthology.orchestra.run.smithi114.stdout:(69/159): postgresql-contrib-13.14-1.el9.x86_64 8.9 MB/s | 911 kB 00:00 2024-05-27T07:25:30.932 INFO:teuthology.orchestra.run.smithi114.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-05-27T07:25:30.966 INFO:teuthology.orchestra.run.smithi114.stdout:(71/159): postgresql-13.14-1.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-05-27T07:25:30.998 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:25:31.016 INFO:teuthology.orchestra.run.smithi114.stdout:(72/159): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-05-27T07:25:31.041 INFO:teuthology.orchestra.run.smithi114.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-05-27T07:25:31.115 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:25:31.134 INFO:teuthology.orchestra.run.smithi114.stdout:(74/159): postgresql-server-13.14-1.el9.x86_64. 26 MB/s | 5.8 MB 00:00 2024-05-27T07:25:31.149 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:25:31.159 INFO:teuthology.orchestra.run.smithi114.stdout:(75/159): python3-colorama-0.4.6-3.el9.noarch.r 524 kB/s | 61 kB 00:00 2024-05-27T07:25:31.240 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:25:31.270 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:25:31.284 INFO:teuthology.orchestra.run.smithi114.stdout:(76/159): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 6.0 MB 00:00 2024-05-27T07:25:31.311 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T07:25:31.334 INFO:teuthology.orchestra.run.smithi114.stdout:(77/159): python3-devel-3.9.18-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2024-05-27T07:25:31.360 INFO:teuthology.orchestra.run.smithi114.stdout:(78/159): python3-jsonpatch-1.21-16.el9.noarch. 341 kB/s | 26 kB 00:00 2024-05-27T07:25:31.385 INFO:teuthology.orchestra.run.smithi114.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-05-27T07:25:31.410 INFO:teuthology.orchestra.run.smithi114.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-05-27T07:25:31.435 INFO:teuthology.orchestra.run.smithi114.stdout:(81/159): python3-jinja2-2.11.3-6.el9.noarch.rp 899 kB/s | 248 kB 00:00 2024-05-27T07:25:31.461 INFO:teuthology.orchestra.run.smithi114.stdout:(82/159): python3-markupsafe-1.1.1-12.el9.x86_6 461 kB/s | 35 kB 00:00 2024-05-27T07:25:31.494 INFO:teuthology.orchestra.run.smithi114.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-05-27T07:25:31.497 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T07:25:31.519 INFO:teuthology.orchestra.run.smithi114.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-05-27T07:25:31.544 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T07:25:31.545 INFO:teuthology.orchestra.run.smithi114.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-27T07:25:31.570 INFO:teuthology.orchestra.run.smithi114.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 842 kB/s | 42 kB 00:00 2024-05-27T07:25:31.571 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:25:31.595 INFO:teuthology.orchestra.run.smithi114.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-05-27T07:25:31.629 INFO:teuthology.orchestra.run.smithi114.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-05-27T07:25:31.654 INFO:teuthology.orchestra.run.smithi114.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-05-27T07:25:31.659 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:25:31.679 INFO:teuthology.orchestra.run.smithi114.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-27T07:25:31.689 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:25:31.705 INFO:teuthology.orchestra.run.smithi114.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-05-27T07:25:31.735 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:25:31.756 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:25:31.781 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:25:31.789 INFO:teuthology.orchestra.run.smithi114.stdout:(92/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-05-27T07:25:31.814 INFO:teuthology.orchestra.run.smithi114.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-05-27T07:25:31.864 INFO:teuthology.orchestra.run.smithi114.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 588 kB/s | 44 kB 00:00 2024-05-27T07:25:31.890 INFO:teuthology.orchestra.run.smithi114.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-05-27T07:25:31.915 INFO:teuthology.orchestra.run.smithi114.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-05-27T07:25:31.932 INFO:teuthology.orchestra.run.smithi114.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-05-27T07:25:31.965 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:25:31.965 INFO:teuthology.orchestra.run.smithi114.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-05-27T07:25:31.982 INFO:teuthology.orchestra.run.smithi114.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-27T07:25:32.008 INFO:teuthology.orchestra.run.smithi114.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-05-27T07:25:32.024 INFO:teuthology.orchestra.run.smithi114.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-05-27T07:25:32.042 INFO:teuthology.orchestra.run.smithi114.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-05-27T07:25:32.091 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:25:32.201 INFO:teuthology.orchestra.run.smithi114.stdout:(103/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-05-27T07:25:32.217 INFO:teuthology.orchestra.run.smithi114.stdout:(104/159): lua-devel-5.4.4-4.el9.x86_64.rpm 116 kB/s | 22 kB 00:00 2024-05-27T07:25:32.262 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:25:32.314 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:25:32.351 INFO:teuthology.orchestra.run.smithi114.stdout:(105/159): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-05-27T07:25:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:(106/159): gperftools-libs-2.9.1-2.el9.x86_64.r 1.6 MB/s | 309 kB 00:00 2024-05-27T07:25:32.423 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:25:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:(107/159): abseil-cpp-20211102.0-3.el9.x86_64.r 2.3 MB/s | 548 kB 00:00 2024-05-27T07:25:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:(108/159): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-05-27T07:25:32.507 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:25:32.527 INFO:teuthology.orchestra.run.smithi114.stdout:(109/159): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2024-05-27T07:25:32.586 INFO:teuthology.orchestra.run.smithi114.stdout:(110/159): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-05-27T07:25:32.636 INFO:teuthology.orchestra.run.smithi114.stdout:(111/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 129 kB/s | 26 kB 00:00 2024-05-27T07:25:32.661 INFO:teuthology.orchestra.run.smithi114.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-05-27T07:25:32.721 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:25:32.728 INFO:teuthology.orchestra.run.smithi114.stdout:(113/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-05-27T07:25:32.761 INFO:teuthology.orchestra.run.smithi114.stdout:(114/159): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-05-27T07:25:32.829 INFO:teuthology.orchestra.run.smithi114.stdout:(115/159): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-05-27T07:25:32.850 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T07:25:32.854 INFO:teuthology.orchestra.run.smithi114.stdout:(116/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 347 kB/s | 43 kB 00:00 2024-05-27T07:25:32.879 INFO:teuthology.orchestra.run.smithi114.stdout:(117/159): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-05-27T07:25:32.886 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T07:25:32.904 INFO:teuthology.orchestra.run.smithi114.stdout:(118/159): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-05-27T07:25:32.921 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T07:25:32.929 INFO:teuthology.orchestra.run.smithi114.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-05-27T07:25:32.955 INFO:teuthology.orchestra.run.smithi114.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-05-27T07:25:32.980 INFO:teuthology.orchestra.run.smithi114.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-05-27T07:25:33.004 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T07:25:33.006 INFO:teuthology.orchestra.run.smithi114.stdout:(122/159): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-05-27T07:25:33.031 INFO:teuthology.orchestra.run.smithi114.stdout:(123/159): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-05-27T07:25:33.046 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T07:25:33.048 INFO:teuthology.orchestra.run.smithi114.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-05-27T07:25:33.065 INFO:teuthology.orchestra.run.smithi114.stdout:(125/159): python3-jaraco-classes-3.2.1-5.el9.n 528 kB/s | 18 kB 00:00 2024-05-27T07:25:33.081 INFO:teuthology.orchestra.run.smithi114.stdout:(126/159): python3-jaraco-collections-3.0.0-8.e 691 kB/s | 23 kB 00:00 2024-05-27T07:25:33.099 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T07:25:33.123 INFO:teuthology.orchestra.run.smithi114.stdout:(127/159): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-05-27T07:25:33.132 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T07:25:33.140 INFO:teuthology.orchestra.run.smithi114.stdout:(128/159): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-05-27T07:25:33.157 INFO:teuthology.orchestra.run.smithi114.stdout:(129/159): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-05-27T07:25:33.159 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T07:25:33.199 INFO:teuthology.orchestra.run.smithi114.stdout:(130/159): python3-kubernetes-26.1.0-2.el9.noar 14 MB/s | 1.0 MB 00:00 2024-05-27T07:25:33.216 INFO:teuthology.orchestra.run.smithi114.stdout:(131/159): python3-logutils-0.3.5-21.el9.noarch 793 kB/s | 46 kB 00:00 2024-05-27T07:25:33.230 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T07:25:33.242 INFO:teuthology.orchestra.run.smithi114.stdout:(132/159): python3-more-itertools-8.12.0-2.el9. 943 kB/s | 79 kB 00:00 2024-05-27T07:25:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:(133/159): python3-natsort-7.1.1-5.el9.noarch.r 983 kB/s | 58 kB 00:00 2024-05-27T07:25:33.283 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T07:25:33.284 INFO:teuthology.orchestra.run.smithi114.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-05-27T07:25:33.301 INFO:teuthology.orchestra.run.smithi114.stdout:(135/159): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-05-27T07:25:33.313 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T07:25:33.326 INFO:teuthology.orchestra.run.smithi114.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-05-27T07:25:33.343 INFO:teuthology.orchestra.run.smithi114.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 481 kB/s | 28 kB 00:00 2024-05-27T07:25:33.360 INFO:teuthology.orchestra.run.smithi114.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 523 kB/s | 31 kB 00:00 2024-05-27T07:25:33.366 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T07:25:33.385 INFO:teuthology.orchestra.run.smithi114.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-05-27T07:25:33.410 INFO:teuthology.orchestra.run.smithi114.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 717 kB/s | 36 kB 00:00 2024-05-27T07:25:33.427 INFO:teuthology.orchestra.run.smithi114.stdout:(141/159): python3-rsa-4.9-2.el9.noarch.rpm 882 kB/s | 59 kB 00:00 2024-05-27T07:25:33.442 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T07:25:33.444 INFO:teuthology.orchestra.run.smithi114.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 870 kB/s | 51 kB 00:00 2024-05-27T07:25:33.470 INFO:teuthology.orchestra.run.smithi114.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-05-27T07:25:33.476 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 33/170 2024-05-27T07:25:33.495 INFO:teuthology.orchestra.run.smithi114.stdout:(144/159): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-05-27T07:25:33.511 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:25:33.511 INFO:teuthology.orchestra.run.smithi114.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 480 kB/s | 20 kB 00:00 2024-05-27T07:25:33.537 INFO:teuthology.orchestra.run.smithi114.stdout:(146/159): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-05-27T07:25:33.540 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 35/170 2024-05-27T07:25:33.562 INFO:teuthology.orchestra.run.smithi114.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-05-27T07:25:33.604 INFO:teuthology.orchestra.run.smithi114.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-05-27T07:25:33.612 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T07:25:33.655 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T07:25:33.730 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T07:25:33.821 INFO:teuthology.orchestra.run.smithi114.stdout:(149/159): librados2-19.0.0-3872.g38f9d900.el9. 12 MB/s | 3.3 MB 00:00 2024-05-27T07:25:33.863 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T07:25:33.917 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T07:25:34.055 INFO:teuthology.orchestra.run.smithi114.stdout:(150/159): librbd1-19.0.0-3872.g38f9d900.el9.x8 6.3 MB/s | 3.1 MB 00:00 2024-05-27T07:25:34.321 INFO:teuthology.orchestra.run.smithi114.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 156 kB/s | 41 kB 00:00 2024-05-27T07:25:34.434 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T07:25:34.513 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T07:25:34.542 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T07:25:34.578 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T07:25:34.619 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T07:25:34.666 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 46/170 2024-05-27T07:25:34.707 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T07:25:34.863 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T07:25:34.889 INFO:teuthology.orchestra.run.smithi114.stdout:(152/159): NetworkManager-tui-1.47.90-1.el9.x86 437 kB/s | 248 kB 00:00 2024-05-27T07:25:34.918 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T07:25:35.011 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T07:25:35.075 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T07:25:35.147 INFO:teuthology.orchestra.run.smithi114.stdout:(153/159): elfutils-debuginfod-client-0.191-4.e 142 kB/s | 37 kB 00:00 2024-05-27T07:25:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T07:25:35.178 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T07:25:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T07:25:35.262 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T07:25:35.297 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T07:25:35.341 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T07:25:35.392 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T07:25:35.450 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-27T07:25:35.523 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T07:25:35.528 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T07:25:35.560 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T07:25:35.568 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T07:25:35.609 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T07:25:35.614 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T07:25:35.648 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T07:25:35.674 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T07:25:35.706 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T07:25:35.716 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T07:25:35.730 INFO:teuthology.orchestra.run.smithi114.stdout:(154/159): elfutils-libelf-0.191-4.el9.x86_64.r 355 kB/s | 207 kB 00:00 2024-05-27T07:25:35.743 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T07:25:35.783 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T07:25:35.808 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T07:25:35.833 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T07:25:35.899 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T07:25:36.027 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T07:25:36.042 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T07:25:36.074 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T07:25:36.102 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T07:25:36.123 INFO:teuthology.orchestra.run.smithi114.stdout:(155/159): NetworkManager-libnm-1.47.90-1.el9.x 802 kB/s | 1.8 MB 00:02 2024-05-27T07:25:36.153 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T07:25:36.167 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:25:36.192 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T07:25:36.224 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T07:25:36.254 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T07:25:36.273 INFO:teuthology.orchestra.run.smithi114.stdout:(156/159): NetworkManager-1.47.90-1.el9.x86_64. 886 kB/s | 2.3 MB 00:02 2024-05-27T07:25:36.274 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:25:36.288 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T07:25:36.301 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:25:36.321 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T07:25:36.363 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T07:25:36.389 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T07:25:36.393 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:25:36.398 INFO:teuthology.orchestra.run.smithi114.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 389 kB/s | 260 kB 00:00 2024-05-27T07:25:36.422 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:25:36.454 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T07:25:36.478 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T07:25:36.528 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T07:25:36.559 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T07:25:36.607 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T07:25:36.635 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T07:25:36.695 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T07:25:36.728 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:25:36.753 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T07:25:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:25:36.868 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:25:36.922 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:25:36.944 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:25:36.971 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:25:37.136 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:25:37.269 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:25:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:25:37.468 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:25:37.474 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T07:25:37.533 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T07:25:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:25:37.566 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T07:25:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 921 kB/s | 1.3 MB 00:01 2024-05-27T07:25:37.665 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T07:25:37.677 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:25:37.721 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T07:25:37.767 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T07:25:37.855 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T07:25:37.878 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:25:37.902 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T07:25:37.991 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T07:25:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T07:25:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T07:25:38.080 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T07:25:38.109 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T07:25:38.130 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T07:25:38.166 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T07:25:38.187 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T07:25:38.209 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T07:25:38.215 INFO:teuthology.orchestra.run.smithi114.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 1.2 MB/s | 2.4 MB 00:01 2024-05-27T07:25:38.217 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:25:38.218 INFO:teuthology.orchestra.run.smithi114.stdout:Total 13 MB/s | 219 MB 00:16 2024-05-27T07:25:38.241 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T07:25:38.278 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T07:25:38.346 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T07:25:38.399 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T07:25:38.430 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T07:25:38.466 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T07:25:38.541 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T07:25:38.577 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 33/170 2024-05-27T07:25:38.620 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:25:38.649 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 35/170 2024-05-27T07:25:38.717 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T07:25:38.763 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T07:25:38.837 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T07:25:38.841 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T07:25:38.957 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T07:25:38.969 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T07:25:39.023 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T07:25:39.448 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-05-27T07:25:39.550 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T07:25:39.602 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-05-27T07:25:39.602 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-05-27T07:25:39.638 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T07:25:39.666 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T07:25:39.701 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T07:25:39.750 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T07:25:39.788 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 46/170 2024-05-27T07:25:39.829 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T07:25:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T07:25:40.040 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T07:25:40.131 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T07:25:40.186 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T07:25:40.196 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T07:25:40.263 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T07:25:40.269 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T07:25:40.291 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T07:25:40.308 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T07:25:40.323 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T07:25:40.345 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T07:25:40.392 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T07:25:40.436 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T07:25:40.479 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T07:25:40.521 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T07:25:40.640 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T07:25:40.656 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T07:25:40.678 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:25:40.706 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T07:25:40.747 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T07:25:40.770 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:25:40.800 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 104/170 2024-05-27T07:25:40.836 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T07:25:40.841 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T07:25:40.881 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T07:25:40.921 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T07:25:40.954 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T07:25:40.979 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T07:25:41.053 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T07:25:41.181 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T07:25:41.211 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T07:25:41.239 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T07:25:41.273 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T07:25:41.312 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T07:25:41.325 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T07:25:41.343 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T07:25:41.365 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:25:41.374 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T07:25:41.408 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T07:25:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:25:41.442 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T07:25:41.458 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 108/170 2024-05-27T07:25:41.483 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T07:25:41.517 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T07:25:41.604 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T07:25:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T07:25:41.679 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T07:25:41.725 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T07:25:41.786 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-05-27T07:25:41.786 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-05-27T07:25:41.843 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T07:25:41.905 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T07:25:42.624 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T07:25:42.669 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T07:25:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T07:25:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T07:25:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T07:25:42.911 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T07:25:42.998 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T07:25:43.029 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T07:25:43.209 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T07:25:43.236 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T07:25:43.314 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T07:25:43.620 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:25:43.663 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:25:43.950 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T07:25:44.067 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T07:25:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:25:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:25:44.355 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:25:44.356 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T07:25:44.356 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T07:25:44.356 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:25:44.384 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:25:45.288 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T07:25:45.372 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T07:25:45.391 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T07:25:45.416 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T07:25:45.733 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T07:25:45.779 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:25:45.854 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:25:45.884 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 104/170 2024-05-27T07:25:45.920 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T07:25:46.408 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T07:25:46.440 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:25:46.511 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:25:46.533 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 108/170 2024-05-27T07:25:48.067 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-05-27T07:25:48.153 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T07:25:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T07:25:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T07:25:48.279 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T07:25:48.304 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T07:25:48.357 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T07:25:48.755 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T07:25:48.792 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:25:48.814 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:25:48.830 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:25:48.931 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:25:48.965 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:25:49.057 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:25:49.086 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:25:49.138 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T07:25:49.339 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T07:25:49.386 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:25:49.392 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T07:25:49.429 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:25:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:25:49.525 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:25:49.526 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T07:25:49.526 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T07:25:49.526 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:25:49.534 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:25:49.550 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:25:49.563 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:25:49.626 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:25:49.647 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:25:49.670 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:25:49.838 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:25:50.007 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:25:50.144 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:25:50.197 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:25:50.302 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:25:50.373 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:25:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:25:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T07:25:50.752 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T07:25:50.787 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T07:25:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T07:25:50.887 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T07:25:50.930 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T07:25:50.971 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T07:25:50.999 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T07:25:51.067 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T07:25:51.121 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T07:25:51.152 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T07:25:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T07:25:51.271 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T07:25:51.306 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 33/170 2024-05-27T07:25:51.341 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:25:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 35/170 2024-05-27T07:25:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T07:25:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T07:25:51.558 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T07:25:51.690 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T07:25:51.745 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T07:25:52.265 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T07:25:52.349 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T07:25:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T07:25:52.423 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T07:25:52.473 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T07:25:52.510 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 46/170 2024-05-27T07:25:52.552 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T07:25:52.714 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T07:25:52.805 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T07:25:52.895 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T07:25:52.952 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T07:25:53.025 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T07:25:53.065 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T07:25:53.093 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T07:25:53.140 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T07:25:53.176 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T07:25:53.219 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T07:25:53.261 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T07:25:53.395 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T07:25:53.430 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T07:25:53.471 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T07:25:53.566 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T07:25:53.613 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T07:25:53.654 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T07:25:53.687 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T07:25:53.711 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T07:25:53.794 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T07:25:53.920 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T07:25:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T07:25:53.980 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T07:25:54.015 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T07:25:54.070 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T07:25:54.093 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T07:25:54.123 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T07:25:54.158 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T07:25:54.191 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T07:25:54.232 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T07:25:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T07:25:54.574 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T07:25:54.624 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T07:25:54.679 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T07:25:54.727 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T07:25:54.842 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T07:25:54.897 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T07:25:55.616 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T07:25:55.669 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T07:25:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T07:25:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T07:25:55.858 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T07:25:55.903 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T07:25:55.992 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T07:25:56.030 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T07:25:56.209 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T07:25:56.238 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:25:56.238 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-05-27T07:25:56.238 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-05-27T07:25:56.238 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-05-27T07:25:56.238 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-05-27T07:25:56.238 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-05-27T07:25:56.238 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-05-27T07:25:56.238 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-05-27T07:25:56.238 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:25:56.245 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T07:25:56.324 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T07:25:56.425 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:25:56.441 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:25:56.441 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:25:56.441 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T07:25:56.441 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:25:56.976 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T07:25:57.092 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T07:25:58.033 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:25:58.059 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:25:58.059 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:25:58.059 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T07:25:58.059 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:25:58.059 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:25:58.059 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:25:58.284 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:25:58.306 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:25:58.307 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:25:58.307 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T07:25:58.307 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:25:58.307 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:25:58.307 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:25:58.356 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T07:25:58.459 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T07:25:58.487 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T07:25:58.520 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T07:25:58.768 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:25:58.791 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:25:58.791 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:25:58.791 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T07:25:58.791 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:25:58.791 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:25:58.791 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:25:58.843 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T07:25:58.892 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:25:58.898 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T07:25:58.956 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T07:25:58.965 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:25:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 104/170 2024-05-27T07:25:59.011 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T07:25:59.041 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T07:25:59.197 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T07:25:59.244 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T07:25:59.322 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T07:25:59.411 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T07:25:59.536 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T07:25:59.549 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T07:25:59.570 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:25:59.594 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T07:25:59.618 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T07:25:59.640 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:25:59.646 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T07:25:59.663 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 108/170 2024-05-27T07:26:00.024 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:26:00.053 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:26:00.085 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 128/170 2024-05-27T07:26:00.147 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 129/170 2024-05-27T07:26:00.176 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:26:01.216 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:26:01.252 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:26:01.395 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:26:01.395 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2024-05-27T07:26:01.395 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2024-05-27T07:26:01.395 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2024-05-27T07:26:01.395 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2024-05-27T07:26:01.395 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2024-05-27T07:26:01.396 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /root 2024-05-27T07:26:01.396 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2024-05-27T07:26:01.396 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:26:01.571 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:26:01.586 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:26:01.586 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:01.586 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T07:26:01.586 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:26:01.827 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:26:01.867 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:26:02.214 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:26:02.244 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:26:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:26:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:26:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:26:02.521 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 133/170 2024-05-27T07:26:02.547 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:26:02.547 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T07:26:02.547 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T07:26:02.547 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:26:02.547 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:26:02.571 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:26:02.571 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:02.571 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T07:26:02.571 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:26:02.571 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:26:02.571 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:26:02.580 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:26:02.623 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:26:02.634 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:26:02.732 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 136/170 2024-05-27T07:26:02.779 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 137/170 2024-05-27T07:26:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:26:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:26:03.212 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:03.212 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T07:26:03.212 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:26:03.212 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:26:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:26:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:26:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:26:03.459 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:03.459 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T07:26:03.459 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:26:03.460 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:26:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:26:03.716 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T07:26:03.750 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:26:03.774 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:26:03.774 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:03.774 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T07:26:03.774 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:26:03.777 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:26:03.777 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:26:03.839 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:26:03.862 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:26:03.862 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:03.862 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T07:26:03.862 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:26:03.910 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:26:03.921 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:26:03.932 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:26:03.932 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:03.932 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T07:26:03.932 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:26:03.932 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:26:03.932 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:26:03.944 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:26:03.944 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:03.944 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T07:26:03.944 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:26:03.944 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:26:03.944 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:26:04.051 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T07:26:04.099 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T07:26:04.163 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T07:26:04.252 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:26:04.275 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:26:04.275 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:04.275 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T07:26:04.275 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:26:04.275 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:26:04.275 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:26:04.349 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T07:26:04.397 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T07:26:04.474 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T07:26:04.562 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T07:26:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T07:26:04.747 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T07:26:04.779 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T07:26:04.816 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T07:26:05.268 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:26:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:26:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 128/170 2024-05-27T07:26:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 129/170 2024-05-27T07:26:05.446 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:26:06.488 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:26:06.523 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:26:07.485 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:26:07.507 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:26:07.617 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:26:07.790 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 133/170 2024-05-27T07:26:07.818 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:26:07.841 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:26:07.841 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:07.842 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T07:26:07.842 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:26:07.842 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:26:07.842 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:26:07.885 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:26:07.896 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:26:07.996 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 136/170 2024-05-27T07:26:08.050 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 137/170 2024-05-27T07:26:08.499 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 143/170 2024-05-27T07:26:08.539 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 144/170 2024-05-27T07:26:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 145/170 2024-05-27T07:26:08.612 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T07:26:08.639 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T07:26:08.824 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 148/170 2024-05-27T07:26:08.877 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T07:26:08.910 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 150/170 2024-05-27T07:26:08.950 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T07:26:08.980 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T07:26:08.983 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T07:26:09.013 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:26:09.026 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:26:09.036 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:26:09.036 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:09.036 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T07:26:09.036 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:26:09.036 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:26:09.036 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:26:09.080 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:26:09.102 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:26:09.102 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:09.102 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T07:26:09.102 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:26:09.148 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:26:09.170 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:26:09.170 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:09.171 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T07:26:09.171 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:26:09.171 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:26:09.171 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:26:09.490 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:26:09.512 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:26:09.512 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:09.513 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T07:26:09.513 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:26:09.513 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:26:09.513 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:26:09.648 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:26:09.682 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:26:09.703 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:26:10.551 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 155/170 2024-05-27T07:26:10.646 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T07:26:10.687 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T07:26:10.770 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T07:26:10.808 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T07:26:10.841 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:26:10.867 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T07:26:10.893 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T07:26:10.893 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:26:10.978 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:26:11.011 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T07:26:11.011 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:26:11.095 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:26:11.120 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T07:26:11.125 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:26:11.125 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:26:11.313 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:26:11.314 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:26:11.398 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:26:11.422 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T07:26:11.487 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:26:13.761 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 143/170 2024-05-27T07:26:13.802 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 144/170 2024-05-27T07:26:13.843 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 145/170 2024-05-27T07:26:13.875 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T07:26:13.902 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T07:26:14.056 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:26:14.057 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-05-27T07:26:14.057 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-05-27T07:26:14.057 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-05-27T07:26:14.057 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-05-27T07:26:14.057 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-05-27T07:26:14.057 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-05-27T07:26:14.057 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-05-27T07:26:14.057 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:26:14.085 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 148/170 2024-05-27T07:26:14.132 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T07:26:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 150/170 2024-05-27T07:26:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T07:26:14.229 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:26:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T07:26:14.245 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:26:14.245 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:14.245 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T07:26:14.245 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:26:14.273 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:26:14.930 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:26:14.972 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:26:14.992 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:26:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:26:15.831 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 155/170 2024-05-27T07:26:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:26:15.842 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:15.842 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T07:26:15.843 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:26:15.843 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:26:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:26:15.912 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T07:26:15.953 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T07:26:16.034 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T07:26:16.067 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:26:16.082 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T07:26:16.090 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:26:16.090 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:16.090 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T07:26:16.090 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:26:16.090 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:26:16.090 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:26:16.106 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:26:16.132 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T07:26:16.159 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T07:26:16.159 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:26:16.246 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:26:16.284 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T07:26:16.285 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:26:16.369 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:26:16.394 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T07:26:16.399 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:26:16.400 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:26:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:26:16.565 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:26:16.566 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:16.566 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T07:26:16.566 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:26:16.566 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:26:16.566 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:26:16.588 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:26:16.588 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:26:16.672 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:26:16.681 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T07:26:16.697 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T07:26:16.738 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T07:26:16.760 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:26:16.793 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T07:26:16.979 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T07:26:17.026 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T07:26:17.029 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:26:17.029 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 1/170 2024-05-27T07:26:17.029 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2/170 2024-05-27T07:26:17.029 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 3/170 2024-05-27T07:26:17.029 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 4/170 2024-05-27T07:26:17.029 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 5/170 2024-05-27T07:26:17.029 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 6/170 2024-05-27T07:26:17.029 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 7/170 2024-05-27T07:26:17.029 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:26:17.029 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:26:17.029 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:26:17.029 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 11/170 2024-05-27T07:26:17.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 12/170 2024-05-27T07:26:17.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 13/170 2024-05-27T07:26:17.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:26:17.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 15/170 2024-05-27T07:26:17.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 16/170 2024-05-27T07:26:17.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 17/170 2024-05-27T07:26:17.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 18/170 2024-05-27T07:26:17.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 19/170 2024-05-27T07:26:17.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 20/170 2024-05-27T07:26:17.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 21/170 2024-05-27T07:26:17.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 22/170 2024-05-27T07:26:17.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 23/170 2024-05-27T07:26:17.030 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 24/170 2024-05-27T07:26:17.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 25/170 2024-05-27T07:26:17.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 26/170 2024-05-27T07:26:17.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 27/170 2024-05-27T07:26:17.031 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 28/170 2024-05-27T07:26:17.042 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 29/170 2024-05-27T07:26:17.042 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 30/170 2024-05-27T07:26:17.042 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 31/170 2024-05-27T07:26:17.042 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 32/170 2024-05-27T07:26:17.042 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 33/170 2024-05-27T07:26:17.043 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:26:17.043 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 35/170 2024-05-27T07:26:17.043 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 36/170 2024-05-27T07:26:17.043 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 37/170 2024-05-27T07:26:17.043 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 38/170 2024-05-27T07:26:17.043 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 39/170 2024-05-27T07:26:17.043 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T07:26:17.043 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T07:26:17.043 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T07:26:17.043 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T07:26:17.043 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T07:26:17.043 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T07:26:17.043 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T07:26:17.044 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T07:26:17.044 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T07:26:17.044 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T07:26:17.044 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T07:26:17.044 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T07:26:17.044 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T07:26:17.044 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T07:26:17.044 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T07:26:17.044 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T07:26:17.044 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T07:26:17.044 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T07:26:17.044 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T07:26:17.044 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T07:26:17.045 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T07:26:17.045 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T07:26:17.045 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T07:26:17.045 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T07:26:17.045 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T07:26:17.045 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T07:26:17.045 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T07:26:17.045 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T07:26:17.045 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T07:26:17.045 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T07:26:17.045 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T07:26:17.045 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T07:26:17.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T07:26:17.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T07:26:17.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T07:26:17.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T07:26:17.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T07:26:17.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T07:26:17.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T07:26:17.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T07:26:17.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T07:26:17.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T07:26:17.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T07:26:17.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T07:26:17.046 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T07:26:17.047 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T07:26:17.048 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T07:26:17.049 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T07:26:17.050 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T07:26:17.050 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T07:26:17.050 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T07:26:17.050 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T07:26:17.050 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T07:26:17.050 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T07:26:17.050 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T07:26:17.050 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T07:26:17.050 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T07:26:17.050 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T07:26:17.050 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T07:26:17.050 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T07:26:17.051 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T07:26:17.051 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T07:26:17.051 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T07:26:17.051 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T07:26:17.051 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T07:26:17.051 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T07:26:17.051 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T07:26:17.051 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T07:26:17.051 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T07:26:17.051 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T07:26:17.051 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 149/170 2024-05-27T07:26:17.051 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T07:26:17.051 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 151/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T07:26:17.052 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T07:26:17.053 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T07:26:17.053 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T07:26:17.053 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T07:26:17.095 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T07:26:17.185 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T07:26:17.324 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T07:26:17.368 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T07:26:17.400 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T07:26:17.428 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T07:26:17.822 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:26:17.851 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:26:17.883 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 128/170 2024-05-27T07:26:17.945 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 129/170 2024-05-27T07:26:17.975 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:26:19.013 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:26:19.043 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:26:19.593 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T07:26:19.593 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:26:19.593 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-05-27T07:26:19.593 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T07:26:19.593 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T07:26:19.593 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T07:26:19.593 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T07:26:19.593 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T07:26:19.593 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T07:26:19.594 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T07:26:19.594 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.594 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.594 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T07:26:19.594 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T07:26:19.594 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-27T07:26:19.594 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T07:26:19.594 INFO:teuthology.orchestra.run.smithi186.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T07:26:19.594 INFO:teuthology.orchestra.run.smithi186.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T07:26:19.594 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T07:26:19.594 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.594 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.595 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T07:26:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T07:26:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T07:26:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T07:26:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T07:26:19.597 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T07:26:19.598 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.598 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.598 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T07:26:19.598 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.598 INFO:teuthology.orchestra.run.smithi186.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T07:26:19.598 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T07:26:19.598 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T07:26:19.598 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T07:26:19.598 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T07:26:19.598 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T07:26:19.598 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T07:26:19.598 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T07:26:19.598 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T07:26:19.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T07:26:19.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T07:26:19.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T07:26:19.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T07:26:19.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T07:26:19.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T07:26:19.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T07:26:19.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T07:26:19.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T07:26:19.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T07:26:19.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T07:26:19.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T07:26:19.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T07:26:19.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T07:26:19.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T07:26:19.602 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T07:26:19.603 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T07:26:19.604 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T07:26:19.605 INFO:teuthology.orchestra.run.smithi186.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T07:26:19.605 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T07:26:19.605 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T07:26:19.605 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T07:26:19.605 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:26:19.605 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-27T07:26:20.004 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:26:20.026 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:26:20.031 DEBUG:teuthology.parallel:result is None 2024-05-27T07:26:20.137 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:26:20.304 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 133/170 2024-05-27T07:26:20.333 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:26:20.357 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:26:20.357 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:20.358 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T07:26:20.358 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:26:20.358 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:26:20.358 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:26:20.409 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:26:20.420 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:26:20.518 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 136/170 2024-05-27T07:26:20.572 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 137/170 2024-05-27T07:26:21.502 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T07:26:21.536 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:26:21.560 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:26:21.560 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:21.560 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T07:26:21.560 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:26:21.560 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:26:21.561 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:26:21.603 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:26:21.626 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:26:21.626 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:21.627 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T07:26:21.627 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:26:21.679 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:26:21.702 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:26:21.702 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:21.702 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T07:26:21.702 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:26:21.702 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:26:21.702 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:26:22.013 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:26:22.036 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:26:22.036 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:26:22.036 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T07:26:22.036 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:26:22.036 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:26:22.036 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:26:22.439 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:26:22.440 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 1/170 2024-05-27T07:26:22.440 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2/170 2024-05-27T07:26:22.440 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 3/170 2024-05-27T07:26:22.440 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 4/170 2024-05-27T07:26:22.440 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 5/170 2024-05-27T07:26:22.440 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 6/170 2024-05-27T07:26:22.440 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 7/170 2024-05-27T07:26:22.454 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:26:22.454 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:26:22.454 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:26:22.454 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 11/170 2024-05-27T07:26:22.454 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 12/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 13/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 15/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 16/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 17/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 18/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 19/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 20/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 21/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 22/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 23/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 24/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 25/170 2024-05-27T07:26:22.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 26/170 2024-05-27T07:26:22.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 27/170 2024-05-27T07:26:22.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 28/170 2024-05-27T07:26:22.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 29/170 2024-05-27T07:26:22.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 30/170 2024-05-27T07:26:22.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 31/170 2024-05-27T07:26:22.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 32/170 2024-05-27T07:26:22.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 33/170 2024-05-27T07:26:22.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:26:22.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 35/170 2024-05-27T07:26:22.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 36/170 2024-05-27T07:26:22.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 37/170 2024-05-27T07:26:22.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 38/170 2024-05-27T07:26:22.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 39/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T07:26:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T07:26:22.458 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T07:26:22.458 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T07:26:22.458 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T07:26:22.458 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T07:26:22.458 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T07:26:22.458 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T07:26:22.458 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T07:26:22.458 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T07:26:22.458 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T07:26:22.458 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T07:26:22.458 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T07:26:22.458 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T07:26:22.458 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T07:26:22.459 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T07:26:22.460 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T07:26:22.460 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T07:26:22.460 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T07:26:22.460 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T07:26:22.460 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T07:26:22.460 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T07:26:22.460 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T07:26:22.460 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T07:26:22.460 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T07:26:22.460 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T07:26:22.460 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T07:26:22.460 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T07:26:22.460 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T07:26:22.461 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T07:26:22.461 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T07:26:22.461 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T07:26:22.461 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T07:26:22.461 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T07:26:22.461 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T07:26:22.461 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T07:26:22.461 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T07:26:22.461 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T07:26:22.461 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T07:26:22.462 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T07:26:22.462 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T07:26:22.462 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T07:26:22.462 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T07:26:22.462 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T07:26:22.462 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T07:26:22.462 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T07:26:22.462 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T07:26:22.462 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T07:26:22.462 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T07:26:22.462 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T07:26:22.462 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T07:26:22.462 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T07:26:22.463 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T07:26:22.463 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T07:26:22.463 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T07:26:22.463 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T07:26:22.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T07:26:22.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T07:26:22.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T07:26:22.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T07:26:22.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T07:26:22.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T07:26:22.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T07:26:22.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T07:26:22.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T07:26:22.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T07:26:22.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T07:26:22.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T07:26:22.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T07:26:22.465 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T07:26:22.466 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T07:26:22.466 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 149/170 2024-05-27T07:26:22.466 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T07:26:22.466 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 151/170 2024-05-27T07:26:22.466 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T07:26:22.466 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:26:22.466 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T07:26:22.466 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T07:26:22.466 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T07:26:22.466 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T07:26:22.466 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T07:26:22.466 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T07:26:22.466 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:26:22.467 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T07:26:22.467 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T07:26:22.467 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T07:26:22.467 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T07:26:22.467 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T07:26:22.467 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T07:26:22.467 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T07:26:22.467 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T07:26:22.467 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T07:26:25.391 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T07:26:25.392 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:26:25.392 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2024-05-27T07:26:25.392 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T07:26:25.392 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T07:26:25.392 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T07:26:25.392 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T07:26:25.392 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T07:26:25.392 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T07:26:25.392 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T07:26:25.393 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.393 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.393 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T07:26:25.393 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T07:26:25.393 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-27T07:26:25.393 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T07:26:25.393 INFO:teuthology.orchestra.run.smithi163.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T07:26:25.393 INFO:teuthology.orchestra.run.smithi163.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T07:26:25.393 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T07:26:25.393 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.393 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.393 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:25.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:25.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:25.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:25.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:25.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:25.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.395 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.395 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:25.395 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.395 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.395 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.395 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:25.395 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:25.395 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.395 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.395 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:25.395 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T07:26:25.395 INFO:teuthology.orchestra.run.smithi163.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T07:26:25.396 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T07:26:25.396 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T07:26:25.396 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T07:26:25.396 INFO:teuthology.orchestra.run.smithi163.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T07:26:25.396 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T07:26:25.396 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T07:26:25.396 INFO:teuthology.orchestra.run.smithi163.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T07:26:25.396 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T07:26:25.396 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T07:26:25.397 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T07:26:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T07:26:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T07:26:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T07:26:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T07:26:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T07:26:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T07:26:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T07:26:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T07:26:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T07:26:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T07:26:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T07:26:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T07:26:25.398 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T07:26:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T07:26:25.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T07:26:25.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T07:26:25.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T07:26:25.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T07:26:25.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T07:26:25.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T07:26:25.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T07:26:25.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T07:26:25.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T07:26:25.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T07:26:25.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T07:26:25.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T07:26:25.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T07:26:25.401 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T07:26:25.401 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T07:26:25.401 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T07:26:25.401 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T07:26:25.401 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T07:26:25.401 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T07:26:25.401 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T07:26:25.401 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T07:26:25.401 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T07:26:25.401 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T07:26:25.401 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T07:26:25.401 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T07:26:25.401 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T07:26:25.402 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T07:26:25.402 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T07:26:25.402 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T07:26:25.402 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T07:26:25.402 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T07:26:25.402 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T07:26:25.402 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T07:26:25.402 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T07:26:25.402 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T07:26:25.403 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T07:26:25.403 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T07:26:25.403 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.403 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.403 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T07:26:25.403 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T07:26:25.403 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T07:26:25.403 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.403 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T07:26:25.403 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T07:26:25.403 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T07:26:25.403 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T07:26:25.403 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T07:26:25.404 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T07:26:25.404 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T07:26:25.404 INFO:teuthology.orchestra.run.smithi163.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T07:26:25.404 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T07:26:25.404 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T07:26:25.404 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T07:26:25.404 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T07:26:25.404 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T07:26:25.404 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T07:26:25.404 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T07:26:25.404 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.404 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.404 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:25.405 INFO:teuthology.orchestra.run.smithi163.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T07:26:25.405 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T07:26:25.405 INFO:teuthology.orchestra.run.smithi163.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T07:26:25.405 INFO:teuthology.orchestra.run.smithi163.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T07:26:25.405 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T07:26:25.405 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T07:26:25.405 INFO:teuthology.orchestra.run.smithi163.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T07:26:25.406 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-27T07:26:25.406 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-27T07:26:25.824 DEBUG:teuthology.parallel:result is None 2024-05-27T07:26:26.309 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 143/170 2024-05-27T07:26:26.350 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 144/170 2024-05-27T07:26:26.390 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 145/170 2024-05-27T07:26:26.422 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T07:26:26.450 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T07:26:26.641 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 148/170 2024-05-27T07:26:26.690 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T07:26:26.720 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 150/170 2024-05-27T07:26:26.752 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T07:26:26.781 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T07:26:26.828 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:26:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:26:27.392 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:26:27.413 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:26:28.241 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 155/170 2024-05-27T07:26:28.350 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T07:26:28.397 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T07:26:28.482 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T07:26:28.517 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T07:26:28.542 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:26:28.567 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T07:26:28.594 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T07:26:28.594 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:26:28.678 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:26:28.719 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T07:26:28.720 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:26:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:26:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T07:26:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:26:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:26:29.014 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:26:29.014 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:26:29.098 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:26:29.123 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T07:26:29.188 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:26:34.983 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:26:34.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 1/170 2024-05-27T07:26:34.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2/170 2024-05-27T07:26:34.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 3/170 2024-05-27T07:26:34.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 4/170 2024-05-27T07:26:34.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 5/170 2024-05-27T07:26:34.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 6/170 2024-05-27T07:26:34.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 7/170 2024-05-27T07:26:34.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:26:34.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:26:34.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:26:34.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 11/170 2024-05-27T07:26:34.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 12/170 2024-05-27T07:26:34.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 13/170 2024-05-27T07:26:34.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:26:34.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 15/170 2024-05-27T07:26:34.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 16/170 2024-05-27T07:26:34.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 17/170 2024-05-27T07:26:34.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 18/170 2024-05-27T07:26:34.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 19/170 2024-05-27T07:26:34.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 20/170 2024-05-27T07:26:34.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 21/170 2024-05-27T07:26:34.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 22/170 2024-05-27T07:26:34.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 23/170 2024-05-27T07:26:34.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 24/170 2024-05-27T07:26:34.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 25/170 2024-05-27T07:26:34.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 26/170 2024-05-27T07:26:34.986 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 27/170 2024-05-27T07:26:34.986 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 28/170 2024-05-27T07:26:34.986 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 29/170 2024-05-27T07:26:34.986 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 30/170 2024-05-27T07:26:34.986 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 31/170 2024-05-27T07:26:34.986 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 32/170 2024-05-27T07:26:34.986 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 33/170 2024-05-27T07:26:34.986 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:26:34.986 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 35/170 2024-05-27T07:26:34.986 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 36/170 2024-05-27T07:26:34.986 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 37/170 2024-05-27T07:26:34.986 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 38/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 39/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T07:26:34.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T07:26:34.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T07:26:34.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T07:26:34.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T07:26:34.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T07:26:34.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T07:26:34.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T07:26:34.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T07:26:34.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T07:26:34.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T07:26:34.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T07:26:34.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T07:26:34.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T07:26:34.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T07:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T07:26:34.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T07:26:34.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T07:26:34.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T07:26:34.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T07:26:34.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T07:26:34.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T07:26:34.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T07:26:34.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T07:26:34.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T07:26:34.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T07:26:34.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T07:26:34.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T07:26:34.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T07:26:34.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T07:26:34.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T07:26:34.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T07:26:34.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 149/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 151/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T07:26:34.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T07:26:34.996 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T07:26:34.996 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T07:26:34.996 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T07:26:34.996 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T07:26:34.996 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T07:26:34.996 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T07:26:34.996 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T07:26:37.563 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T07:26:37.563 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:26:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-05-27T07:26:37.563 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T07:26:37.563 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T07:26:37.563 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T07:26:37.563 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T07:26:37.563 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T07:26:37.564 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T07:26:37.564 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T07:26:37.564 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.564 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.564 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T07:26:37.564 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T07:26:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-05-27T07:26:37.564 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T07:26:37.564 INFO:teuthology.orchestra.run.smithi114.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T07:26:37.564 INFO:teuthology.orchestra.run.smithi114.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T07:26:37.564 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T07:26:37.564 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.564 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.565 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.565 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.565 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:37.565 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.565 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.565 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.566 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:37.566 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:37.566 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:37.566 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:37.566 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:37.566 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.566 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.566 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:37.566 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.566 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.566 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T07:26:37.567 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T07:26:37.568 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.569 INFO:teuthology.orchestra.run.smithi114.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T07:26:37.569 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T07:26:37.569 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T07:26:37.569 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T07:26:37.569 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T07:26:37.569 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T07:26:37.569 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T07:26:37.569 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T07:26:37.569 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T07:26:37.569 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T07:26:37.569 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T07:26:37.569 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T07:26:37.569 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T07:26:37.570 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T07:26:37.570 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T07:26:37.570 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T07:26:37.570 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T07:26:37.570 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T07:26:37.570 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T07:26:37.570 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T07:26:37.570 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T07:26:37.570 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T07:26:37.570 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T07:26:37.570 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T07:26:37.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T07:26:37.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T07:26:37.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T07:26:37.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T07:26:37.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T07:26:37.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T07:26:37.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T07:26:37.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T07:26:37.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T07:26:37.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T07:26:37.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T07:26:37.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T07:26:37.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T07:26:37.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T07:26:37.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T07:26:37.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T07:26:37.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T07:26:37.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T07:26:37.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T07:26:37.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T07:26:37.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T07:26:37.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T07:26:37.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T07:26:37.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T07:26:37.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T07:26:37.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T07:26:37.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T07:26:37.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T07:26:37.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T07:26:37.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T07:26:37.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T07:26:37.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T07:26:37.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T07:26:37.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T07:26:37.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T07:26:37.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T07:26:37.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T07:26:37.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T07:26:37.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T07:26:37.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.577 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T07:26:37.577 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T07:26:37.577 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T07:26:37.577 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.577 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T07:26:37.578 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T07:26:37.578 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T07:26:37.578 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T07:26:37.578 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T07:26:37.578 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T07:26:37.578 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T07:26:37.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T07:26:37.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T07:26:37.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T07:26:37.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T07:26:37.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T07:26:37.579 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T07:26:37.579 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T07:26:37.580 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T07:26:37.580 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.580 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.580 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:26:37.580 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T07:26:37.581 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T07:26:37.581 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T07:26:37.581 INFO:teuthology.orchestra.run.smithi114.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T07:26:37.581 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T07:26:37.581 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T07:26:37.581 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T07:26:37.582 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:26:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-05-27T07:26:37.993 DEBUG:teuthology.parallel:result is None 2024-05-27T07:26:37.993 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:26:38.144 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T07:26:38.167 INFO:teuthology.orchestra.run.smithi114.stdout:19.0.0-3872.g38f9d900.el9 2024-05-27T07:26:38.168 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872.g38f9d900.el9 2024-05-27T07:26:38.168 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872.g38f9d900 is installed. 2024-05-27T07:26:38.171 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:26:38.396 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T07:26:38.422 INFO:teuthology.orchestra.run.smithi163.stdout:19.0.0-3872.g38f9d900.el9 2024-05-27T07:26:38.422 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872.g38f9d900.el9 2024-05-27T07:26:38.422 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872.g38f9d900 is installed. 2024-05-27T07:26:38.424 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:26:38.568 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T07:26:38.595 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-3872.g38f9d900.el9 2024-05-27T07:26:38.595 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872.g38f9d900.el9 2024-05-27T07:26:38.595 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872.g38f9d900 is installed. 2024-05-27T07:26:38.597 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-27T07:26:38.597 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:26:38.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T07:26:38.626 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-27T07:26:38.626 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T07:26:38.658 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-27T07:26:38.658 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T07:26:38.688 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-27T07:26:38.688 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:26:38.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T07:26:38.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T07:26:38.783 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-27T07:26:38.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T07:26:38.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T07:26:38.881 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-27T07:26:38.881 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T07:26:38.908 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T07:26:38.983 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-27T07:26:38.983 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:26:38.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T07:26:39.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T07:26:39.079 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-27T07:26:39.080 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T07:26:39.104 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T07:26:39.169 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-27T07:26:39.169 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T07:26:39.193 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T07:26:39.257 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-27T07:26:39.258 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:26:39.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T07:26:39.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T07:26:39.358 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-27T07:26:39.358 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T07:26:39.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T07:26:39.446 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-27T07:26:39.446 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T07:26:39.470 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T07:26:39.534 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-27T07:26:39.645 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-05-27T07:26:39.645 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:26:39.646 INFO:tasks.cephadm:Cluster fsid is 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:26:39.646 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-27T07:26:39.646 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.114', 'mon.b': '172.21.15.163', 'mon.c': '172.21.15.186'} 2024-05-27T07:26:39.646 INFO:tasks.cephadm:First mon is mon.a on smithi114 2024-05-27T07:26:39.646 INFO:tasks.cephadm:First mgr is x 2024-05-27T07:26:39.646 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-27T07:26:39.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-05-27T07:26:39.677 DEBUG:teuthology.orchestra.run.smithi163:> sudo hostname $(hostname -s) 2024-05-27T07:26:39.700 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname $(hostname -s) 2024-05-27T07:26:39.724 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-27T07:26:39.724 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:26:39.862 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79587/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3872-g38f9d900', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3872.g38f9d900'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-21 08:08:15.155937', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240521.063558-debug', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79583/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3872-g38f9d900', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3872.g38f9d900'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-21 07:54:08.730060', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240521.063558-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-27T07:26:40.000 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-vshankar-testing-20240521.063558-debug, sha1 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:26:40.002 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm 2024-05-27T07:26:40.002 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm 2024-05-27T07:26:40.002 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T07:26:40.633 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-r--r--. 1 ubuntu ubuntu 782786 May 27 07:26 /home/ubuntu/cephtest/cephadm 2024-05-27T07:26:40.634 DEBUG:teuthology.orchestra.run.smithi163:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T07:26:41.245 INFO:teuthology.orchestra.run.smithi163.stdout:-rw-r--r--. 1 ubuntu ubuntu 782786 May 27 07:26 /home/ubuntu/cephtest/cephadm 2024-05-27T07:26:41.246 DEBUG:teuthology.orchestra.run.smithi186:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T07:26:41.719 INFO:teuthology.orchestra.run.smithi186.stdout:-rw-r--r--. 1 ubuntu ubuntu 782786 May 27 07:26 /home/ubuntu/cephtest/cephadm 2024-05-27T07:26:41.720 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T07:26:41.740 DEBUG:teuthology.orchestra.run.smithi163:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T07:26:41.759 DEBUG:teuthology.orchestra.run.smithi186:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T07:26:41.802 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 on all hosts... 2024-05-27T07:26:41.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 pull 2024-05-27T07:26:41.805 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 pull 2024-05-27T07:26:41.807 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 pull 2024-05-27T07:26:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:26:42.707 INFO:teuthology.orchestra.run.smithi186.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:26:42.793 INFO:teuthology.orchestra.run.smithi163.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:27:15.314 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-05-27T07:27:15.314 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:27:15.314 INFO:teuthology.orchestra.run.smithi186.stdout: "image_id": "e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799", 2024-05-27T07:27:15.314 INFO:teuthology.orchestra.run.smithi186.stdout: "repo_digests": [ 2024-05-27T07:27:15.314 INFO:teuthology.orchestra.run.smithi186.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6" 2024-05-27T07:27:15.314 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-05-27T07:27:15.314 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-05-27T07:27:22.124 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:27:22.124 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:27:22.124 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799", 2024-05-27T07:27:22.124 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-05-27T07:27:22.124 INFO:teuthology.orchestra.run.smithi114.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6" 2024-05-27T07:27:22.124 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-05-27T07:27:22.125 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:27:25.907 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-05-27T07:27:25.907 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:27:25.907 INFO:teuthology.orchestra.run.smithi163.stdout: "image_id": "e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799", 2024-05-27T07:27:25.908 INFO:teuthology.orchestra.run.smithi163.stdout: "repo_digests": [ 2024-05-27T07:27:25.908 INFO:teuthology.orchestra.run.smithi163.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6" 2024-05-27T07:27:25.908 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-05-27T07:27:25.908 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-05-27T07:27:25.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-05-27T07:27:25.965 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph 2024-05-27T07:27:25.992 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph 2024-05-27T07:27:26.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-05-27T07:27:26.053 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /etc/ceph 2024-05-27T07:27:26.077 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /etc/ceph 2024-05-27T07:27:26.103 INFO:tasks.cephadm:Writing seed config... 2024-05-27T07:27:26.105 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-27T07:27:26.105 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-27T07:27:26.105 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-27T07:27:26.105 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-27T07:27:26.105 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-27T07:27:26.105 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-05-27T07:27:26.105 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-27T07:27:26.105 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-27T07:27:26.106 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-27T07:27:26.106 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-27T07:27:26.106 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-27T07:27:26.106 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-27T07:27:26.106 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-27T07:27:26.106 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-27T07:27:26.106 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-27T07:27:26.106 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-27T07:27:26.107 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-27T07:27:26.107 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-27T07:27:26.107 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-27T07:27:26.107 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-27T07:27:26.107 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-27T07:27:26.107 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-05-27T07:27:26.107 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-27T07:27:26.107 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-27T07:27:26.107 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-27T07:27:26.107 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-27T07:27:26.108 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-27T07:27:26.108 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-27T07:27:26.108 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-27T07:27:26.108 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-27T07:27:26.108 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-27T07:27:26.108 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-05-27T07:27:26.108 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-27T07:27:26.108 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-05-27T07:27:26.108 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-05-27T07:27:26.108 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-27T07:27:26.108 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-27T07:27:26.108 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-05-27T07:27:26.108 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-27T07:27:26.109 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-27T07:27:26.109 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-27T07:27:26.109 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-27T07:27:26.109 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-27T07:27:26.109 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-27T07:27:26.109 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-27T07:27:26.109 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-05-27T07:27:26.110 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:27:26.110 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-27T07:27:26.127 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 755b3728-1bfa-11ef-bc9b-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-27T07:27:26.128 DEBUG:teuthology.orchestra.run.smithi114:mon.a> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.a.service 2024-05-27T07:27:26.174 DEBUG:teuthology.orchestra.run.smithi114:mgr.x> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.x.service 2024-05-27T07:27:26.217 INFO:tasks.cephadm:Bootstrapping... 2024-05-27T07:27:26.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 -v bootstrap --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.114 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:27:26.410 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:27:26.410 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6', '-v', 'bootstrap', '--fsid', '755b3728-1bfa-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.114', '--skip-admin-label'] 2024-05-27T07:27:26.411 INFO:teuthology.orchestra.run.smithi114.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-27T07:27:26.411 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying podman|docker is present... 2024-05-27T07:27:26.436 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 5.0.2 2024-05-27T07:27:26.436 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying lvm2 is present... 2024-05-27T07:27:26.436 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying time synchronization is in place... 2024-05-27T07:27:26.444 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-27T07:27:26.444 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-27T07:27:26.450 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-27T07:27:26.450 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-05-27T07:27:26.457 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-05-27T07:27:26.463 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-05-27T07:27:26.464 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-05-27T07:27:26.464 INFO:teuthology.orchestra.run.smithi114.stdout:Repeating the final host check... 2024-05-27T07:27:26.487 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 5.0.2 2024-05-27T07:27:26.487 INFO:teuthology.orchestra.run.smithi114.stdout:podman (/bin/podman) version 5.0.2 is present 2024-05-27T07:27:26.488 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl is present 2024-05-27T07:27:26.488 INFO:teuthology.orchestra.run.smithi114.stdout:lvcreate is present 2024-05-27T07:27:26.494 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-27T07:27:26.494 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-27T07:27:26.500 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-27T07:27:26.500 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-05-27T07:27:26.506 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-05-27T07:27:26.513 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-05-27T07:27:26.513 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-05-27T07:27:26.513 INFO:teuthology.orchestra.run.smithi114.stdout:Host looks OK 2024-05-27T07:27:26.513 INFO:teuthology.orchestra.run.smithi114.stdout:Cluster fsid: 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:27:26.513 INFO:teuthology.orchestra.run.smithi114.stdout:Acquiring lock 140472725901072 on /run/cephadm/755b3728-1bfa-11ef-bc9b-c7b262605968.lock 2024-05-27T07:27:26.513 INFO:teuthology.orchestra.run.smithi114.stdout:Lock 140472725901072 acquired on /run/cephadm/755b3728-1bfa-11ef-bc9b-c7b262605968.lock 2024-05-27T07:27:26.514 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 3300 ... 2024-05-27T07:27:26.514 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 6789 ... 2024-05-27T07:27:26.514 INFO:teuthology.orchestra.run.smithi114.stdout:Base mon IP(s) is [172.21.15.114:3300, 172.21.15.114:6789], mon addrv is [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-05-27T07:27:26.517 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.114 metric 100 2024-05-27T07:27:26.517 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.114 metric 100 2024-05-27T07:27:26.519 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-27T07:27:26.520 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-05-27T07:27:26.520 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1785sec hoplimit 64 pref medium 2024-05-27T07:27:26.522 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-27T07:27:26.522 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-27T07:27:26.522 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-27T07:27:26.522 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-27T07:27:26.522 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link 2024-05-27T07:27:26.522 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-27T07:27:26.523 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-05-27T07:27:26.523 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-05-27T07:27:26.523 INFO:teuthology.orchestra.run.smithi114.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-27T07:27:26.523 INFO:teuthology.orchestra.run.smithi114.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-27T07:27:26.524 INFO:teuthology.orchestra.run.smithi114.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:27:27.001 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 2024-05-27T07:27:27.002 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:27:27.002 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Getting image source signatures 2024-05-27T07:27:27.002 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:cf46015854c6cdc81d3273f73f8f001fe4a1edc6ed77e464da8c9d5e01e9b108 2024-05-27T07:27:27.002 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:6ac36e7c57f227110df5cd65cfdca1cc05e0b17df6748637a6aa04c5c80f89c6 2024-05-27T07:27:27.002 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying config sha256:e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 2024-05-27T07:27:27.003 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-27T07:27:27.855 INFO:teuthology.orchestra.run.smithi114.stdout:ceph: stdout ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:27.856 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph version: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:27.856 INFO:teuthology.orchestra.run.smithi114.stdout:Extracting ceph user uid/gid from container image... 2024-05-27T07:27:28.853 INFO:teuthology.orchestra.run.smithi114.stdout:stat: stdout 167 167 2024-05-27T07:27:28.853 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial keys... 2024-05-27T07:27:29.674 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQDhNVRmPBCqChAAyYzEAlJC58ZAhqhXBWai0g== 2024-05-27T07:27:30.564 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQDiNVRm+6XwARAA49wMo32VnYmRbo51DoaIHg== 2024-05-27T07:27:35.485 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQDiNVRmTsZ5NBAA5/HX4m3X07him40fg2h3Xw== 2024-05-27T07:27:35.486 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial monmap... 2024-05-27T07:27:44.831 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-27T07:27:44.831 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-27T07:27:44.831 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:27:44.831 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-27T07:27:44.831 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool for a [v2:172.21.15.114:3300,v1:172.21.15.114:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-27T07:27:44.831 INFO:teuthology.orchestra.run.smithi114.stdout:setting min_mon_release = quincy 2024-05-27T07:27:44.831 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: set fsid to 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:27:44.831 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-27T07:27:44.831 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:27:44.831 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mon... 2024-05-27T07:27:46.297 INFO:teuthology.orchestra.run.smithi114.stdout:create mon.a on 2024-05-27T07:27:46.495 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-27T07:27:46.636 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-27T07:27:46.815 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-755b3728-1bfa-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-755b3728-1bfa-11ef-bc9b-c7b262605968.target. 2024-05-27T07:27:46.815 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-755b3728-1bfa-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-755b3728-1bfa-11ef-bc9b-c7b262605968.target. 2024-05-27T07:27:47.050 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.a 2024-05-27T07:27:47.050 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.a.service: Unit ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.a.service not loaded. 2024-05-27T07:27:47.212 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-755b3728-1bfa-11ef-bc9b-c7b262605968.target.wants/ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.a.service → /etc/systemd/system/ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@.service. 2024-05-27T07:27:47.705 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T07:27:47.705 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-05-27T07:27:47.711 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T07:27:47.712 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-05-27T07:27:47.712 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-05-27T07:27:47.712 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-05-27T07:27:47.713 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon to start... 2024-05-27T07:27:47.713 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon... 2024-05-27T07:27:48.772 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:48 smithi114 ceph-mon[31665]: from='client.? 172.21.15.114:0/1436038586' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-27T07:27:48.775 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout cluster: 2024-05-27T07:27:48.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout id: 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:27:48.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-27T07:27:48.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-05-27T07:27:48.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout services: 2024-05-27T07:27:48.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.542003s) 2024-05-27T07:27:48.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-27T07:27:48.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-27T07:27:48.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-05-27T07:27:48.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout data: 2024-05-27T07:27:48.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-27T07:27:48.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-27T07:27:48.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-27T07:27:48.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pgs: 2024-05-27T07:27:48.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-05-27T07:27:48.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.155+0000 7fb005b9f700 1 Processor -- start 2024-05-27T07:27:48.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.155+0000 7fb005b9f700 1 -- start start 2024-05-27T07:27:48.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.155+0000 7fb005b9f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 0x7fb00010c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.156+0000 7fb005b9f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb00010cb70 con 0x7fb00007d580 2024-05-27T07:27:48.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.156+0000 7fb004b9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 0x7fb00010c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.156+0000 7fb004b9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 0x7fb00010c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34624/0 (socket says 172.21.15.114:34624) 2024-05-27T07:27:48.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.156+0000 7fb004b9d700 1 -- 172.21.15.114:0/4067438310 learned_addr learned my addr 172.21.15.114:0/4067438310 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:27:48.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.156+0000 7fb004b9d700 1 -- 172.21.15.114:0/4067438310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00010ccf0 con 0x7fb00007d580 2024-05-27T07:27:48.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.156+0000 7fb004b9d700 1 --2- 172.21.15.114:0/4067438310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 0x7fb00010c560 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7faff80098a0 tx=0x7faff8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69076707a5de52a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:48.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.156+0000 7fafff7fe700 1 -- 172.21.15.114:0/4067438310 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faff8004160 con 0x7fb00007d580 2024-05-27T07:27:48.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.157+0000 7fafff7fe700 1 -- 172.21.15.114:0/4067438310 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faff8004300 con 0x7fb00007d580 2024-05-27T07:27:48.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.157+0000 7fafff7fe700 1 -- 172.21.15.114:0/4067438310 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faff80036a0 con 0x7fb00007d580 2024-05-27T07:27:48.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.157+0000 7fb005b9f700 1 -- 172.21.15.114:0/4067438310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 msgr2=0x7fb00010c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:48.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.157+0000 7fb005b9f700 1 --2- 172.21.15.114:0/4067438310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 0x7fb00010c560 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7faff80098a0 tx=0x7faff8009bb0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.157+0000 7fb005b9f700 1 -- 172.21.15.114:0/4067438310 shutdown_connections 2024-05-27T07:27:48.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.157+0000 7fb005b9f700 1 --2- 172.21.15.114:0/4067438310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 0x7fb00010c560 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.157+0000 7fb005b9f700 1 -- 172.21.15.114:0/4067438310 >> 172.21.15.114:0/4067438310 conn(0x7fb000103ff0 msgr2=0x7fb000106440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:48.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.157+0000 7fb005b9f700 1 -- 172.21.15.114:0/4067438310 shutdown_connections 2024-05-27T07:27:48.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.157+0000 7fb005b9f700 1 -- 172.21.15.114:0/4067438310 wait complete. 2024-05-27T07:27:48.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.158+0000 7fb005b9f700 1 Processor -- start 2024-05-27T07:27:48.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.158+0000 7fb005b9f700 1 -- start start 2024-05-27T07:27:48.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.158+0000 7fb005b9f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 0x7fb00019d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.158+0000 7fb005b9f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb00010d220 con 0x7fb00007d580 2024-05-27T07:27:48.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.159+0000 7fb004b9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 0x7fb00019d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.159+0000 7fb004b9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 0x7fb00019d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34626/0 (socket says 172.21.15.114:34626) 2024-05-27T07:27:48.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.159+0000 7fb004b9d700 1 -- 172.21.15.114:0/1436038586 learned_addr learned my addr 172.21.15.114:0/1436038586 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:27:48.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.159+0000 7fb004b9d700 1 -- 172.21.15.114:0/1436038586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00019d9e0 con 0x7fb00007d580 2024-05-27T07:27:48.785 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.159+0000 7fb004b9d700 1 --2- 172.21.15.114:0/1436038586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 0x7fb00019d4a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb00007cac0 tx=0x7faff8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:48.785 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.159+0000 7faffdffb700 1 -- 172.21.15.114:0/1436038586 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faff8003db0 con 0x7fb00007d580 2024-05-27T07:27:48.785 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.159+0000 7faffdffb700 1 -- 172.21.15.114:0/1436038586 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faff8011630 con 0x7fb00007d580 2024-05-27T07:27:48.785 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.159+0000 7fb005b9f700 1 -- 172.21.15.114:0/1436038586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb00019dc70 con 0x7fb00007d580 2024-05-27T07:27:48.785 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.159+0000 7faffdffb700 1 -- 172.21.15.114:0/1436038586 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faff8011930 con 0x7fb00007d580 2024-05-27T07:27:48.785 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.159+0000 7fb005b9f700 1 -- 172.21.15.114:0/1436038586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0001a0880 con 0x7fb00007d580 2024-05-27T07:27:48.785 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.160+0000 7faffdffb700 1 -- 172.21.15.114:0/1436038586 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7faff8011ad0 con 0x7fb00007d580 2024-05-27T07:27:48.785 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.160+0000 7faffdffb700 1 -- 172.21.15.114:0/1436038586 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7faff801b450 con 0x7fb00007d580 2024-05-27T07:27:48.785 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.161+0000 7fb005b9f700 1 -- 172.21.15.114:0/1436038586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb000064e10 con 0x7fb00007d580 2024-05-27T07:27:48.785 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.163+0000 7faffdffb700 1 -- 172.21.15.114:0/1436038586 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7faff8033080 con 0x7fb00007d580 2024-05-27T07:27:48.786 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.232+0000 7fb005b9f700 1 -- 172.21.15.114:0/1436038586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fb00002e9a0 con 0x7fb00007d580 2024-05-27T07:27:48.786 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.233+0000 7faffdffb700 1 -- 172.21.15.114:0/1436038586 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7faff8004260 con 0x7fb00007d580 2024-05-27T07:27:48.786 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.233+0000 7fb005b9f700 1 -- 172.21.15.114:0/1436038586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 msgr2=0x7fb00019d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:48.786 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.233+0000 7fb005b9f700 1 --2- 172.21.15.114:0/1436038586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 0x7fb00019d4a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb00007cac0 tx=0x7faff8004750 comp rx=0 tx=0).stop 2024-05-27T07:27:48.786 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.234+0000 7fb005b9f700 1 -- 172.21.15.114:0/1436038586 shutdown_connections 2024-05-27T07:27:48.786 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.234+0000 7fb005b9f700 1 --2- 172.21.15.114:0/1436038586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb00007d580 0x7fb00019d4a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.786 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.234+0000 7fb005b9f700 1 -- 172.21.15.114:0/1436038586 >> 172.21.15.114:0/1436038586 conn(0x7fb000103ff0 msgr2=0x7fb000105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:48.786 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.234+0000 7fb005b9f700 1 -- 172.21.15.114:0/1436038586 shutdown_connections 2024-05-27T07:27:48.786 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:48.234+0000 7fb005b9f700 1 -- 172.21.15.114:0/1436038586 wait complete. 2024-05-27T07:27:48.786 INFO:teuthology.orchestra.run.smithi114.stdout:mon is available 2024-05-27T07:27:48.786 INFO:teuthology.orchestra.run.smithi114.stdout:Assimilating anything we can from ceph.conf... 2024-05-27T07:27:49.879 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:49 smithi114 ceph-mon[31665]: from='client.? 172.21.15.114:0/1628628710' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-27T07:27:49.879 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:49 smithi114 ceph-mon[31665]: from='client.? 172.21.15.114:0/1628628710' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-27T07:27:49.886 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-05-27T07:27:49.886 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-27T07:27:49.887 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-27T07:27:49.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-27T07:27:49.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-27T07:27:49.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-27T07:27:49.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.253+0000 7f927f90e700 1 Processor -- start 2024-05-27T07:27:49.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.253+0000 7f927f90e700 1 -- start start 2024-05-27T07:27:49.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.253+0000 7f927f90e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 0x7f927810a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:49.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.253+0000 7f927f90e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f927810a630 con 0x7f9278109c30 2024-05-27T07:27:49.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.253+0000 7f927d6aa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 0x7f927810a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:49.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.253+0000 7f927d6aa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 0x7f927810a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34628/0 (socket says 172.21.15.114:34628) 2024-05-27T07:27:49.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.253+0000 7f927d6aa700 1 -- 172.21.15.114:0/925924656 learned_addr learned my addr 172.21.15.114:0/925924656 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:27:49.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.254+0000 7f927d6aa700 1 -- 172.21.15.114:0/925924656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f927810ad60 con 0x7f9278109c30 2024-05-27T07:27:49.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.254+0000 7f927d6aa700 1 --2- 172.21.15.114:0/925924656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 0x7f927810a040 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f92740098a0 tx=0x7f9274009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e668d3343e89969 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:49.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.254+0000 7f926ffff700 1 -- 172.21.15.114:0/925924656 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9274004160 con 0x7f9278109c30 2024-05-27T07:27:49.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.254+0000 7f926ffff700 1 -- 172.21.15.114:0/925924656 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9274004300 con 0x7f9278109c30 2024-05-27T07:27:49.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.254+0000 7f926ffff700 1 -- 172.21.15.114:0/925924656 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92740036a0 con 0x7f9278109c30 2024-05-27T07:27:49.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.255+0000 7f927f90e700 1 -- 172.21.15.114:0/925924656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 msgr2=0x7f927810a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:49.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.255+0000 7f927f90e700 1 --2- 172.21.15.114:0/925924656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 0x7f927810a040 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f92740098a0 tx=0x7f9274009bb0 comp rx=0 tx=0).stop 2024-05-27T07:27:49.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.255+0000 7f927f90e700 1 -- 172.21.15.114:0/925924656 shutdown_connections 2024-05-27T07:27:49.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.255+0000 7f927f90e700 1 --2- 172.21.15.114:0/925924656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 0x7f927810a040 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:49.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.255+0000 7f927f90e700 1 -- 172.21.15.114:0/925924656 >> 172.21.15.114:0/925924656 conn(0x7f92781052b0 msgr2=0x7f9278107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:49.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.255+0000 7f927f90e700 1 -- 172.21.15.114:0/925924656 shutdown_connections 2024-05-27T07:27:49.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.255+0000 7f927f90e700 1 -- 172.21.15.114:0/925924656 wait complete. 2024-05-27T07:27:49.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.256+0000 7f927f90e700 1 Processor -- start 2024-05-27T07:27:49.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.256+0000 7f927f90e700 1 -- start start 2024-05-27T07:27:49.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.256+0000 7f927f90e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 0x7f927819d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:49.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.256+0000 7f927f90e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f927810b290 con 0x7f9278109c30 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.256+0000 7f927d6aa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 0x7f927819d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.256+0000 7f927d6aa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 0x7f927819d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34636/0 (socket says 172.21.15.114:34636) 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.256+0000 7f927d6aa700 1 -- 172.21.15.114:0/1628628710 learned_addr learned my addr 172.21.15.114:0/1628628710 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.256+0000 7f927d6aa700 1 -- 172.21.15.114:0/1628628710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f927819dbd0 con 0x7f9278109c30 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.256+0000 7f927d6aa700 1 --2- 172.21.15.114:0/1628628710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 0x7f927819d690 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9274000c00 tx=0x7f9274004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.256+0000 7f926e7fc700 1 -- 172.21.15.114:0/1628628710 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9274003db0 con 0x7f9278109c30 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.256+0000 7f926e7fc700 1 -- 172.21.15.114:0/1628628710 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9274011630 con 0x7f9278109c30 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.256+0000 7f927f90e700 1 -- 172.21.15.114:0/1628628710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f927819de60 con 0x7f9278109c30 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.256+0000 7f926e7fc700 1 -- 172.21.15.114:0/1628628710 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9274011950 con 0x7f9278109c30 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.257+0000 7f927f90e700 1 -- 172.21.15.114:0/1628628710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92781a0a70 con 0x7f9278109c30 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.257+0000 7f926e7fc700 1 -- 172.21.15.114:0/1628628710 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9274011af0 con 0x7f9278109c30 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.257+0000 7f926e7fc700 1 -- 172.21.15.114:0/1628628710 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f927401c450 con 0x7f9278109c30 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.257+0000 7f927f90e700 1 -- 172.21.15.114:0/1628628710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9278064e10 con 0x7f9278109c30 2024-05-27T07:27:49.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.260+0000 7f926e7fc700 1 -- 172.21.15.114:0/1628628710 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f9274032020 con 0x7f9278109c30 2024-05-27T07:27:49.891 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.328+0000 7f927f90e700 1 -- 172.21.15.114:0/1628628710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f927802dc30 con 0x7f9278109c30 2024-05-27T07:27:49.891 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.332+0000 7f926e7fc700 1 -- 172.21.15.114:0/1628628710 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+467 (secure 0 0 0) 0x7f9274004260 con 0x7f9278109c30 2024-05-27T07:27:49.891 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.333+0000 7f927f90e700 1 -- 172.21.15.114:0/1628628710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 msgr2=0x7f927819d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:49.891 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.333+0000 7f927f90e700 1 --2- 172.21.15.114:0/1628628710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 0x7f927819d690 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9274000c00 tx=0x7f9274004750 comp rx=0 tx=0).stop 2024-05-27T07:27:49.891 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.333+0000 7f927f90e700 1 -- 172.21.15.114:0/1628628710 shutdown_connections 2024-05-27T07:27:49.891 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.333+0000 7f927f90e700 1 --2- 172.21.15.114:0/1628628710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9278109c30 0x7f927819d690 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:49.891 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.333+0000 7f927f90e700 1 -- 172.21.15.114:0/1628628710 >> 172.21.15.114:0/1628628710 conn(0x7f92781052b0 msgr2=0x7f9278193d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:49.891 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.333+0000 7f927f90e700 1 -- 172.21.15.114:0/1628628710 shutdown_connections 2024-05-27T07:27:49.891 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:49.333+0000 7f927f90e700 1 -- 172.21.15.114:0/1628628710 wait complete. 2024-05-27T07:27:49.891 INFO:teuthology.orchestra.run.smithi114.stdout:Generating new minimal ceph.conf... 2024-05-27T07:27:52.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:51 smithi114 ceph-mon[31665]: from='client.? 172.21.15.114:0/3121903109' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.753+0000 7f6f98317700 1 Processor -- start 2024-05-27T07:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.753+0000 7f6f98317700 1 -- start start 2024-05-27T07:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.753+0000 7f6f98317700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 0x7f6f9010c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.753+0000 7f6f98317700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9010cd40 con 0x7f6f9007d580 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.753+0000 7f6f960b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 0x7f6f9010c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.754+0000 7f6f960b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 0x7f6f9010c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34644/0 (socket says 172.21.15.114:34644) 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.754+0000 7f6f960b3700 1 -- 172.21.15.114:0/1900596909 learned_addr learned my addr 172.21.15.114:0/1900596909 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.754+0000 7f6f960b3700 1 -- 172.21.15.114:0/1900596909 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9010cec0 con 0x7f6f9007d580 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.754+0000 7f6f960b3700 1 --2- 172.21.15.114:0/1900596909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 0x7f6f9010c730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c0098a0 tx=0x7f6f8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5491dfdb17f211a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.754+0000 7f6f950b1700 1 -- 172.21.15.114:0/1900596909 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f8c004160 con 0x7f6f9007d580 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.754+0000 7f6f950b1700 1 -- 172.21.15.114:0/1900596909 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6f8c004300 con 0x7f6f9007d580 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.754+0000 7f6f950b1700 1 -- 172.21.15.114:0/1900596909 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f8c011af0 con 0x7f6f9007d580 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.755+0000 7f6f98317700 1 -- 172.21.15.114:0/1900596909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 msgr2=0x7f6f9010c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.755+0000 7f6f98317700 1 --2- 172.21.15.114:0/1900596909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 0x7f6f9010c730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c0098a0 tx=0x7f6f8c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.755+0000 7f6f98317700 1 -- 172.21.15.114:0/1900596909 shutdown_connections 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.755+0000 7f6f98317700 1 --2- 172.21.15.114:0/1900596909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 0x7f6f9010c730 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.755+0000 7f6f98317700 1 -- 172.21.15.114:0/1900596909 >> 172.21.15.114:0/1900596909 conn(0x7f6f901041c0 msgr2=0x7f6f90106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.755+0000 7f6f98317700 1 -- 172.21.15.114:0/1900596909 shutdown_connections 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.755+0000 7f6f98317700 1 -- 172.21.15.114:0/1900596909 wait complete. 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.756+0000 7f6f98317700 1 Processor -- start 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.756+0000 7f6f98317700 1 -- start start 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.756+0000 7f6f98317700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 0x7f6f9019df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.756+0000 7f6f98317700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9010df40 con 0x7f6f9007d580 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.756+0000 7f6f960b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 0x7f6f9019df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.756+0000 7f6f960b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 0x7f6f9019df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34652/0 (socket says 172.21.15.114:34652) 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.756+0000 7f6f960b3700 1 -- 172.21.15.114:0/3121903109 learned_addr learned my addr 172.21.15.114:0/3121903109 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.756+0000 7f6f960b3700 1 -- 172.21.15.114:0/3121903109 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9019e480 con 0x7f6f9007d580 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.756+0000 7f6f960b3700 1 --2- 172.21.15.114:0/3121903109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 0x7f6f9019df40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009fb0 tx=0x7f6f8c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.757+0000 7f6f877fe700 1 -- 172.21.15.114:0/3121903109 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f8c0036a0 con 0x7f6f9007d580 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.757+0000 7f6f877fe700 1 -- 172.21.15.114:0/3121903109 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6f8c01a430 con 0x7f6f9007d580 2024-05-27T07:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.757+0000 7f6f98317700 1 -- 172.21.15.114:0/3121903109 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f9019e710 con 0x7f6f9007d580 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.757+0000 7f6f877fe700 1 -- 172.21.15.114:0/3121903109 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f8c019550 con 0x7f6f9007d580 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.757+0000 7f6f98317700 1 -- 172.21.15.114:0/3121903109 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f901a1320 con 0x7f6f9007d580 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.757+0000 7f6f877fe700 1 -- 172.21.15.114:0/3121903109 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6f8c019b90 con 0x7f6f9007d580 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.757+0000 7f6f877fe700 1 -- 172.21.15.114:0/3121903109 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6f8c024730 con 0x7f6f9007d580 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.758+0000 7f6f98317700 1 -- 172.21.15.114:0/3121903109 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f90064e10 con 0x7f6f9007d580 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.760+0000 7f6f877fe700 1 -- 172.21.15.114:0/3121903109 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f6f8c043ec0 con 0x7f6f9007d580 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.827+0000 7f6f98317700 1 -- 172.21.15.114:0/3121903109 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f6f901a14f0 con 0x7f6f9007d580 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.827+0000 7f6f877fe700 1 -- 172.21.15.114:0/3121903109 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f6f8c032020 con 0x7f6f9007d580 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.828+0000 7f6f98317700 1 -- 172.21.15.114:0/3121903109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 msgr2=0x7f6f9019df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.828+0000 7f6f98317700 1 --2- 172.21.15.114:0/3121903109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 0x7f6f9019df40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009fb0 tx=0x7f6f8c011940 comp rx=0 tx=0).stop 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.828+0000 7f6f98317700 1 -- 172.21.15.114:0/3121903109 shutdown_connections 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.828+0000 7f6f98317700 1 --2- 172.21.15.114:0/3121903109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9007d580 0x7f6f9019df40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.828+0000 7f6f98317700 1 -- 172.21.15.114:0/3121903109 >> 172.21.15.114:0/3121903109 conn(0x7f6f901041c0 msgr2=0x7f6f90105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.828+0000 7f6f98317700 1 -- 172.21.15.114:0/3121903109 shutdown_connections 2024-05-27T07:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:51.828+0000 7f6f98317700 1 -- 172.21.15.114:0/3121903109 wait complete. 2024-05-27T07:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout:Restarting the monitor... 2024-05-27T07:27:52.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:52 smithi114 systemd[1]: Stopping Ceph mon.a for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:27:52.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:52 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a[31661]: 2024-05-27T07:27:52.620+0000 7fceafb4d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:27:52.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:52 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a[31661]: 2024-05-27T07:27:52.620+0000 7fceafb4d700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-27T07:27:53.101 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:52 smithi114 podman[31913]: 2024-05-27 07:27:52.835204156 +0000 UTC m=+0.290733839 container died 28bafefbfdd59968e33117682ad6cc1ac79e258b90b9bc9a679834ee02306c9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T07:27:53.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 podman[31913]: 2024-05-27 07:27:53.211389407 +0000 UTC m=+0.666919045 container cleanup 28bafefbfdd59968e33117682ad6cc1ac79e258b90b9bc9a679834ee02306c9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:27:53.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 bash[31913]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a 2024-05-27T07:27:53.781 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 podman[31925]: 2024-05-27 07:27:53.453745531 +0000 UTC m=+0.612686600 container remove 28bafefbfdd59968e33117682ad6cc1ac79e258b90b9bc9a679834ee02306c9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True) 2024-05-27T07:27:53.781 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-27T07:27:53.781 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 systemd[1]: Stopped Ceph mon.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:27:53.781 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 systemd[1]: Starting Ceph mon.a for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:27:54.009 INFO:teuthology.orchestra.run.smithi114.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-27T07:27:54.035 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 podman[32019]: 2024-05-27 07:27:53.694316402 +0000 UTC m=+0.016923583 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:27:54.035 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 podman[32019]: 2024-05-27 07:27:53.796486266 +0000 UTC m=+0.119093446 container create b13d09b4408fae57d961203d4b3dc4b83a85fd2fb43b7045e0d030c234ec1f09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True) 2024-05-27T07:27:54.035 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 podman[32019]: 2024-05-27 07:27:53.940521391 +0000 UTC m=+0.263128583 container init b13d09b4408fae57d961203d4b3dc4b83a85fd2fb43b7045e0d030c234ec1f09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True) 2024-05-27T07:27:54.035 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 podman[32019]: 2024-05-27 07:27:53.943846886 +0000 UTC m=+0.266454069 container start b13d09b4408fae57d961203d4b3dc4b83a85fd2fb43b7045e0d030c234ec1f09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-27T07:27:54.035 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: set uid:gid to 167:167 (ceph:ceph) 2024-05-27T07:27:54.036 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev), process ceph-mon, pid 2 2024-05-27T07:27:54.036 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: pidfile_write: ignore empty --pid-file 2024-05-27T07:27:54.036 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: load: jerasure load: lrc 2024-05-27T07:27:54.036 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: RocksDB version: 7.9.2 2024-05-27T07:27:54.036 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Git sha 0 2024-05-27T07:27:54.036 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Compile date 2024-05-21 07:00:12 2024-05-27T07:27:54.036 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: DB SUMMARY 2024-05-27T07:27:54.036 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: DB Session ID: S31U6JTIO45IQ4XMP21B 2024-05-27T07:27:54.036 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: CURRENT file: CURRENT 2024-05-27T07:27:54.036 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: IDENTITY file: IDENTITY 2024-05-27T07:27:54.037 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-27T07:27:54.037 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-27T07:27:54.037 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104562 ; 2024-05-27T07:27:54.038 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.error_if_exists: 0 2024-05-27T07:27:54.038 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.create_if_missing: 0 2024-05-27T07:27:54.038 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.paranoid_checks: 1 2024-05-27T07:27:54.039 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-27T07:27:54.039 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-27T07:27:54.039 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-27T07:27:54.039 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.env: 0x562c6dd68500 2024-05-27T07:27:54.039 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.fs: PosixFileSystem 2024-05-27T07:27:54.039 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.info_log: 0x562c6fdedfe0 2024-05-27T07:27:54.040 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_file_opening_threads: 16 2024-05-27T07:27:54.040 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.statistics: (nil) 2024-05-27T07:27:54.040 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.use_fsync: 0 2024-05-27T07:27:54.040 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_log_file_size: 0 2024-05-27T07:27:54.040 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-27T07:27:54.040 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-27T07:27:54.040 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.keep_log_file_num: 1000 2024-05-27T07:27:54.040 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.recycle_log_file_num: 0 2024-05-27T07:27:54.040 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.allow_fallocate: 1 2024-05-27T07:27:54.040 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.allow_mmap_reads: 0 2024-05-27T07:27:54.040 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.allow_mmap_writes: 0 2024-05-27T07:27:54.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.use_direct_reads: 0 2024-05-27T07:27:54.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-27T07:27:54.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.create_missing_column_families: 0 2024-05-27T07:27:54.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.db_log_dir: 2024-05-27T07:27:54.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.wal_dir: 2024-05-27T07:27:54.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-27T07:27:54.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-27T07:27:54.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-27T07:27:54.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-27T07:27:54.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-27T07:27:54.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-27T07:27:54.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.advise_random_on_open: 1 2024-05-27T07:27:54.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.db_write_buffer_size: 0 2024-05-27T07:27:54.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.write_buffer_manager: 0x562c6fdbe8c0 2024-05-27T07:27:54.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-27T07:27:54.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-27T07:27:54.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-27T07:27:54.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.rate_limiter: (nil) 2024-05-27T07:27:54.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-27T07:27:54.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.wal_recovery_mode: 2 2024-05-27T07:27:54.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.enable_thread_tracking: 0 2024-05-27T07:27:54.043 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.enable_pipelined_write: 0 2024-05-27T07:27:54.043 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.unordered_write: 0 2024-05-27T07:27:54.043 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-27T07:27:54.043 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-27T07:27:54.043 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-27T07:27:54.043 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-27T07:27:54.043 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.row_cache: None 2024-05-27T07:27:54.043 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.wal_filter: None 2024-05-27T07:27:54.043 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-27T07:27:54.043 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.allow_ingest_behind: 0 2024-05-27T07:27:54.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.two_write_queues: 0 2024-05-27T07:27:54.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.manual_wal_flush: 0 2024-05-27T07:27:54.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.wal_compression: 0 2024-05-27T07:27:54.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.atomic_flush: 0 2024-05-27T07:27:54.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-27T07:27:54.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-27T07:27:54.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-27T07:27:54.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.log_readahead_size: 0 2024-05-27T07:27:54.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-27T07:27:54.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.best_efforts_recovery: 0 2024-05-27T07:27:54.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.allow_data_in_errors: 0 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.db_host_id: __hostname__ 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_background_jobs: 2 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_background_compactions: -1 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_subcompactions: 1 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_total_wal_size: 0 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-27T07:27:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-27T07:27:54.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_open_files: -1 2024-05-27T07:27:54.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.bytes_per_sync: 0 2024-05-27T07:27:54.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-27T07:27:54.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-27T07:27:54.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compaction_readahead_size: 0 2024-05-27T07:27:54.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_background_flushes: -1 2024-05-27T07:27:54.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Compression algorithms supported: 2024-05-27T07:27:54.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-27T07:27:54.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: kZSTD supported: 0 2024-05-27T07:27:54.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: kXpressCompression supported: 0 2024-05-27T07:27:54.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: kLZ4HCCompression supported: 1 2024-05-27T07:27:54.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: kZlibCompression supported: 1 2024-05-27T07:27:54.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: kSnappyCompression supported: 1 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: kLZ4Compression supported: 1 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: kBZip2Compression supported: 0 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.merge_operator: 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compaction_filter: None 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compaction_filter_factory: None 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.sst_partitioner_factory: None 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-27T07:27:54.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562c6fef41c0) 2024-05-27T07:27:54.048 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-05-27T07:27:54.048 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-27T07:27:54.048 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-27T07:27:54.049 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-05-27T07:27:54.049 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_type: 0 2024-05-27T07:27:54.049 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_index_type: 0 2024-05-27T07:27:54.049 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_shortening: 1 2024-05-27T07:27:54.049 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-27T07:27:54.049 INFO:journalctl@ceph.mon.a.smithi114.stdout: checksum: 4 2024-05-27T07:27:54.049 INFO:journalctl@ceph.mon.a.smithi114.stdout: no_block_cache: 0 2024-05-27T07:27:54.049 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache: 0x562c6fdf11f0 2024-05-27T07:27:54.049 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-05-27T07:27:54.049 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_options: 2024-05-27T07:27:54.049 INFO:journalctl@ceph.mon.a.smithi114.stdout: capacity : 536870912 2024-05-27T07:27:54.049 INFO:journalctl@ceph.mon.a.smithi114.stdout: num_shard_bits : 4 2024-05-27T07:27:54.049 INFO:journalctl@ceph.mon.a.smithi114.stdout: strict_capacity_limit : 0 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_compressed: (nil) 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: persistent_cache: (nil) 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size: 4096 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size_deviation: 10 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_restart_interval: 16 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_block_restart_interval: 1 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: metadata_block_size: 4096 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: partition_filters: 0 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: use_delta_encoding: 1 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: filter_policy: bloomfilter 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: whole_key_filtering: 1 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: verify_compression: 0 2024-05-27T07:27:54.050 INFO:journalctl@ceph.mon.a.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout: format_version: 5 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_index_compression: 1 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_align: 0 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout: max_auto_readahead_size: 262144 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout: prepopulate_block_cache: 0 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout: initial_auto_readahead_size: 8192 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.write_buffer_size: 33554432 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_write_buffer_number: 2 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compression: NoCompression 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.bottommost_compression: Disabled 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.prefix_extractor: nullptr 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.num_levels: 7 2024-05-27T07:27:54.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compression_opts.level: 32767 2024-05-27T07:27:54.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compression_opts.strategy: 0 2024-05-27T07:27:54.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-27T07:27:54.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:27:54.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:27:54.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-27T07:27:54.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compression_opts.enabled: false 2024-05-27T07:27:54.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:27:54.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-27T07:27:54.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-27T07:27:54.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-27T07:27:54.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.target_file_size_base: 67108864 2024-05-27T07:27:54.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-27T07:27:54.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-27T07:27:54.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.arena_block_size: 1048576 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-27T07:27:54.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.disable_auto_compactions: 0 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.inplace_update_support: 0 2024-05-27T07:27:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.bloom_locality: 0 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.max_successive_merges: 0 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.paranoid_file_checks: 0 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.force_consistency_checks: 1 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.report_bg_io_stats: 0 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.ttl: 2592000 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-27T07:27:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.enable_blob_files: false 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.min_blob_size: 0 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.blob_file_size: 268435456 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.blob_file_starting_level: 0 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 15cac137-509d-44ee-95d0-125aae58d304 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794873984129, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-27T07:27:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794873985456, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99789, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97671, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14028, "raw_average_key_size": 49, "raw_value_size": 90029, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716794873, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "15cac137-509d-44ee-95d0-125aae58d304", "db_session_id": "S31U6JTIO45IQ4XMP21B", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794873985518, "job": 1, "event": "recovery_finished"} 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562c6fdd5c00 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: DB pointer 0x562c74bc8000 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** DB Stats ** 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:27:54.058 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: L0 2/0 99.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: Sum 2/0 99.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-27T07:27:54.059 INFO:journalctl@ceph.mon.a.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 120.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative compaction: 0.00 GB write, 28.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval compaction: 0.00 GB write, 28.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: Block cache BinnedLRUCache@0x562c6fdf11f0#2 capacity: 512.00 MB usage: 1.53 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-05-27T07:27:54.060 INFO:journalctl@ceph.mon.a.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.03 KB,0.000196695%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: starting mon.a rank 0 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: mon.a@-1(???) e1 preinit fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: mon.a@-1(???).mds e1 new map 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: mon.a@-1(???).mds e1 print_map 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout: e1 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout: legacy client fscid: -1 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout: No filesystems configured 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-27T07:27:54.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:53 smithi114 ceph-mon[32038]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:54 smithi114 bash[32019]: b13d09b4408fae57d961203d4b3dc4b83a85fd2fb43b7045e0d030c234ec1f09 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:54 smithi114 systemd[1]: Started Ceph mon.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:54 smithi114 ceph-mon[32038]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:54 smithi114 ceph-mon[32038]: monmap epoch 1 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:54 smithi114 ceph-mon[32038]: fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:54 smithi114 ceph-mon[32038]: last_changed 2024-05-27T07:27:38.185727+0000 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:54 smithi114 ceph-mon[32038]: created 2024-05-27T07:27:38.185727+0000 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:54 smithi114 ceph-mon[32038]: min_mon_release 19 (squid) 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:54 smithi114 ceph-mon[32038]: election_strategy: 1 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:54 smithi114 ceph-mon[32038]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-05-27T07:27:54.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:54 smithi114 ceph-mon[32038]: fsmap 2024-05-27T07:27:54.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:54 smithi114 ceph-mon[32038]: osdmap e1: 0 total, 0 up, 0 in 2024-05-27T07:27:54.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:54 smithi114 ceph-mon[32038]: mgrmap e1: no daemons active 2024-05-27T07:27:55.134 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.523+0000 7fe29249b700 1 Processor -- start 2024-05-27T07:27:55.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.523+0000 7fe29249b700 1 -- start start 2024-05-27T07:27:55.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.524+0000 7fe29249b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 0x7fe28c10c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:55.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.524+0000 7fe29249b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c10cce0 con 0x7fe28c07ccf0 2024-05-27T07:27:55.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.524+0000 7fe28bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 0x7fe28c10c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:55.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.524+0000 7fe28bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 0x7fe28c10c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34662/0 (socket says 172.21.15.114:34662) 2024-05-27T07:27:55.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.524+0000 7fe28bfff700 1 -- 172.21.15.114:0/4278359478 learned_addr learned my addr 172.21.15.114:0/4278359478 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:27:55.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.524+0000 7fe28bfff700 1 -- 172.21.15.114:0/4278359478 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c10ce60 con 0x7fe28c07ccf0 2024-05-27T07:27:55.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.525+0000 7fe28bfff700 1 --2- 172.21.15.114:0/4278359478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 0x7fe28c10c6d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0098a0 tx=0x7fe27c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac77d2252b64460 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:55.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.525+0000 7fe28b7fe700 1 -- 172.21.15.114:0/4278359478 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe27c004160 con 0x7fe28c07ccf0 2024-05-27T07:27:55.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.525+0000 7fe28b7fe700 1 -- 172.21.15.114:0/4278359478 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe27c004300 con 0x7fe28c07ccf0 2024-05-27T07:27:55.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.525+0000 7fe28b7fe700 1 -- 172.21.15.114:0/4278359478 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe27c011af0 con 0x7fe28c07ccf0 2024-05-27T07:27:55.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.526+0000 7fe29249b700 1 -- 172.21.15.114:0/4278359478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 msgr2=0x7fe28c10c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:55.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.526+0000 7fe29249b700 1 --2- 172.21.15.114:0/4278359478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 0x7fe28c10c6d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0098a0 tx=0x7fe27c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:27:55.136 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.526+0000 7fe29249b700 1 -- 172.21.15.114:0/4278359478 shutdown_connections 2024-05-27T07:27:55.136 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.526+0000 7fe29249b700 1 --2- 172.21.15.114:0/4278359478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 0x7fe28c10c6d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:55.136 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.526+0000 7fe29249b700 1 -- 172.21.15.114:0/4278359478 >> 172.21.15.114:0/4278359478 conn(0x7fe28c104120 msgr2=0x7fe28c106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:55.136 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.526+0000 7fe29249b700 1 -- 172.21.15.114:0/4278359478 shutdown_connections 2024-05-27T07:27:55.136 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.526+0000 7fe29249b700 1 -- 172.21.15.114:0/4278359478 wait complete. 2024-05-27T07:27:55.136 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.526+0000 7fe29249b700 1 Processor -- start 2024-05-27T07:27:55.136 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.526+0000 7fe29249b700 1 -- start start 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.527+0000 7fe29249b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 0x7fe28c19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.527+0000 7fe29249b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c10dee0 con 0x7fe28c07ccf0 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.527+0000 7fe28bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 0x7fe28c19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.527+0000 7fe28bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 0x7fe28c19dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34678/0 (socket says 172.21.15.114:34678) 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.527+0000 7fe28bfff700 1 -- 172.21.15.114:0/3904002624 learned_addr learned my addr 172.21.15.114:0/3904002624 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.527+0000 7fe28bfff700 1 -- 172.21.15.114:0/3904002624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c19e4e0 con 0x7fe28c07ccf0 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.527+0000 7fe28bfff700 1 --2- 172.21.15.114:0/3904002624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 0x7fe28c19dfa0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0117b0 tx=0x7fe27c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.527+0000 7fe289ffb700 1 -- 172.21.15.114:0/3904002624 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe27c0039c0 con 0x7fe28c07ccf0 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.527+0000 7fe289ffb700 1 -- 172.21.15.114:0/3904002624 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe27c003b60 con 0x7fe28c07ccf0 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.527+0000 7fe29249b700 1 -- 172.21.15.114:0/3904002624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28c19e770 con 0x7fe28c07ccf0 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.527+0000 7fe289ffb700 1 -- 172.21.15.114:0/3904002624 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe27c0196a0 con 0x7fe28c07ccf0 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.527+0000 7fe29249b700 1 -- 172.21.15.114:0/3904002624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28c1a1380 con 0x7fe28c07ccf0 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.528+0000 7fe289ffb700 1 -- 172.21.15.114:0/3904002624 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe27c019ca0 con 0x7fe28c07ccf0 2024-05-27T07:27:55.137 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.528+0000 7fe289ffb700 1 -- 172.21.15.114:0/3904002624 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe27c024730 con 0x7fe28c07ccf0 2024-05-27T07:27:55.138 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.528+0000 7fe29249b700 1 -- 172.21.15.114:0/3904002624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe28c064e10 con 0x7fe28c07ccf0 2024-05-27T07:27:55.138 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.531+0000 7fe289ffb700 1 -- 172.21.15.114:0/3904002624 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fe27c0042a0 con 0x7fe28c07ccf0 2024-05-27T07:27:55.138 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.597+0000 7fe29249b700 1 -- 172.21.15.114:0/3904002624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fe28c1a1550 con 0x7fe28c07ccf0 2024-05-27T07:27:55.138 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.599+0000 7fe289ffb700 1 -- 172.21.15.114:0/3904002624 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fe27c01ab00 con 0x7fe28c07ccf0 2024-05-27T07:27:55.138 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.600+0000 7fe29249b700 1 -- 172.21.15.114:0/3904002624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 msgr2=0x7fe28c19dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:55.138 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.600+0000 7fe29249b700 1 --2- 172.21.15.114:0/3904002624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 0x7fe28c19dfa0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0117b0 tx=0x7fe27c011b70 comp rx=0 tx=0).stop 2024-05-27T07:27:55.138 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.600+0000 7fe29249b700 1 -- 172.21.15.114:0/3904002624 shutdown_connections 2024-05-27T07:27:55.138 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.600+0000 7fe29249b700 1 --2- 172.21.15.114:0/3904002624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe28c07ccf0 0x7fe28c19dfa0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:55.138 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.600+0000 7fe29249b700 1 -- 172.21.15.114:0/3904002624 >> 172.21.15.114:0/3904002624 conn(0x7fe28c104120 msgr2=0x7fe28c104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:55.138 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.600+0000 7fe29249b700 1 -- 172.21.15.114:0/3904002624 shutdown_connections 2024-05-27T07:27:55.138 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:54.600+0000 7fe29249b700 1 -- 172.21.15.114:0/3904002624 wait complete. 2024-05-27T07:27:55.138 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-27T07:27:55.160 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:27:55.161 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mgr... 2024-05-27T07:27:55.161 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-27T07:27:55.161 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-27T07:27:55.406 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.x 2024-05-27T07:27:55.406 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.x.service: Unit ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.x.service not loaded. 2024-05-27T07:27:55.553 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-755b3728-1bfa-11ef-bc9b-c7b262605968.target.wants/ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@.service. 2024-05-27T07:27:55.863 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:55 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3904002624' entity='client.admin' 2024-05-27T07:27:56.054 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T07:27:56.054 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-05-27T07:27:56.061 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T07:27:56.061 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-05-27T07:27:56.061 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-05-27T07:27:56.062 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-05-27T07:27:56.070 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T07:27:56.070 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-05-27T07:27:56.078 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T07:27:56.078 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-05-27T07:27:56.078 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-05-27T07:27:56.078 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-27T07:27:56.078 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr to start... 2024-05-27T07:27:56.079 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr... 2024-05-27T07:27:57.179 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:27:56 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/219926474' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T07:27:57.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-05-27T07:27:57.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:27:57.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "755b3728-1bfa-11ef-bc9b-c7b262605968", 2024-05-27T07:27:57.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T07:27:57.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T07:27:57.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T07:27:57.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T07:27:57.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:27:57.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T07:27:57.479 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T07:27:57.492 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-05-27T07:27:57.492 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:27:57.492 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T07:27:57.492 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T07:27:57.492 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:27:57.492 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-27T07:27:57.492 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T07:27:57.492 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:27:57.492 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T07:27:57.492 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T07:27:57.492 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:27:57.492 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T07:27:57.493 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T07:27:57.494 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T07:27:47.694541+0000", 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.798+0000 7f2df8911700 1 Processor -- start 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.798+0000 7f2df8911700 1 -- start start 2024-05-27T07:27:57.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.798+0000 7f2df8911700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 0x7f2df410eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.798+0000 7f2df8911700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2df410f070 con 0x7f2df407cc90 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.799+0000 7f2df259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 0x7f2df410eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.799+0000 7f2df259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 0x7f2df410eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48234/0 (socket says 172.21.15.114:48234) 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.799+0000 7f2df259c700 1 -- 172.21.15.114:0/3078989798 learned_addr learned my addr 172.21.15.114:0/3078989798 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.799+0000 7f2df259c700 1 -- 172.21.15.114:0/3078989798 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df410f1f0 con 0x7f2df407cc90 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.799+0000 7f2df259c700 1 --2- 172.21.15.114:0/3078989798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 0x7f2df410eb30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2de40098a0 tx=0x7f2de4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14374d6424c4c86 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.799+0000 7f2df159a700 1 -- 172.21.15.114:0/3078989798 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2de4004160 con 0x7f2df407cc90 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.799+0000 7f2df159a700 1 -- 172.21.15.114:0/3078989798 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2de4004300 con 0x7f2df407cc90 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.799+0000 7f2df159a700 1 -- 172.21.15.114:0/3078989798 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2de4011af0 con 0x7f2df407cc90 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.800+0000 7f2df8911700 1 -- 172.21.15.114:0/3078989798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 msgr2=0x7f2df410eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.800+0000 7f2df8911700 1 --2- 172.21.15.114:0/3078989798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 0x7f2df410eb30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2de40098a0 tx=0x7f2de4009bb0 comp rx=0 tx=0).stop 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.800+0000 7f2df8911700 1 -- 172.21.15.114:0/3078989798 shutdown_connections 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.800+0000 7f2df8911700 1 --2- 172.21.15.114:0/3078989798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 0x7f2df410eb30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.800+0000 7f2df8911700 1 -- 172.21.15.114:0/3078989798 >> 172.21.15.114:0/3078989798 conn(0x7f2df4104120 msgr2=0x7f2df4106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.800+0000 7f2df8911700 1 -- 172.21.15.114:0/3078989798 shutdown_connections 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.800+0000 7f2df8911700 1 -- 172.21.15.114:0/3078989798 wait complete. 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.801+0000 7f2df8911700 1 Processor -- start 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.801+0000 7f2df8911700 1 -- start start 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.801+0000 7f2df8911700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 0x7f2df41a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.801+0000 7f2df8911700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2df4110100 con 0x7f2df407cc90 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.802+0000 7f2df259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 0x7f2df41a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.802+0000 7f2df259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 0x7f2df41a01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48236/0 (socket says 172.21.15.114:48236) 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.802+0000 7f2df259c700 1 -- 172.21.15.114:0/219926474 learned_addr learned my addr 172.21.15.114:0/219926474 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.802+0000 7f2df259c700 1 -- 172.21.15.114:0/219926474 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df41a0730 con 0x7f2df407cc90 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.802+0000 7f2df259c700 1 --2- 172.21.15.114:0/219926474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 0x7f2df41a01f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2de40117b0 tx=0x7f2de4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.802+0000 7f2de37fe700 1 -- 172.21.15.114:0/219926474 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2de40039c0 con 0x7f2df407cc90 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.802+0000 7f2df8911700 1 -- 172.21.15.114:0/219926474 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2df41a09c0 con 0x7f2df407cc90 2024-05-27T07:27:57.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.802+0000 7f2de37fe700 1 -- 172.21.15.114:0/219926474 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2de4003b60 con 0x7f2df407cc90 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.802+0000 7f2de37fe700 1 -- 172.21.15.114:0/219926474 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2de40196a0 con 0x7f2df407cc90 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.802+0000 7f2df8911700 1 -- 172.21.15.114:0/219926474 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2df41a35d0 con 0x7f2df407cc90 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.803+0000 7f2de37fe700 1 -- 172.21.15.114:0/219926474 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2de4019cf0 con 0x7f2df407cc90 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.803+0000 7f2de37fe700 1 -- 172.21.15.114:0/219926474 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2de4024730 con 0x7f2df407cc90 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.803+0000 7f2df8911700 1 -- 172.21.15.114:0/219926474 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2df4064e10 con 0x7f2df407cc90 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.807+0000 7f2de37fe700 1 -- 172.21.15.114:0/219926474 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f2de4043ec0 con 0x7f2df407cc90 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.881+0000 7f2df8911700 1 -- 172.21.15.114:0/219926474 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2df41a37a0 con 0x7f2df407cc90 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.881+0000 7f2de37fe700 1 -- 172.21.15.114:0/219926474 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2de4032030 con 0x7f2df407cc90 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.882+0000 7f2df8911700 1 -- 172.21.15.114:0/219926474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 msgr2=0x7f2df41a01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.882+0000 7f2df8911700 1 --2- 172.21.15.114:0/219926474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 0x7f2df41a01f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2de40117b0 tx=0x7f2de4011b70 comp rx=0 tx=0).stop 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.882+0000 7f2df8911700 1 -- 172.21.15.114:0/219926474 shutdown_connections 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.882+0000 7f2df8911700 1 --2- 172.21.15.114:0/219926474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df407cc90 0x7f2df41a01f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.882+0000 7f2df8911700 1 -- 172.21.15.114:0/219926474 >> 172.21.15.114:0/219926474 conn(0x7f2df4104120 msgr2=0x7f2df4104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:57.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.882+0000 7f2df8911700 1 -- 172.21.15.114:0/219926474 shutdown_connections 2024-05-27T07:27:57.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:56.882+0000 7f2df8911700 1 -- 172.21.15.114:0/219926474 wait complete. 2024-05-27T07:27:57.499 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (1/15)... 2024-05-27T07:27:58.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:27:58 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:27:58.217+0000 7fe032d18200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:27:58.658 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:27:58 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:27:58.384+0000 7fe032d18200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:27:58.659 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:27:58 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:27:58.668+0000 7fe032d18200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:27:58.948 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:27:58 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:27:58.759+0000 7fe032d18200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:27:58.948 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:27:58 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:27:58.957+0000 7fe032d18200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:27:59.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:27:59 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:27:59.148+0000 7fe032d18200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:27:59.932 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:27:59 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:27:59.757+0000 7fe032d18200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:28:00.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:00 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3092398541' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T07:28:00.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:27:59 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:27:59.985+0000 7fe032d18200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:28:00.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-05-27T07:28:00.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:28:00.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "755b3728-1bfa-11ef-bc9b-c7b262605968", 2024-05-27T07:28:00.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T07:28:00.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T07:28:00.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T07:28:00.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T07:28:00.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:00.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T07:28:00.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:28:00.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T07:28:00.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T07:28:00.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T07:28:00.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T07:28:00.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T07:28:00.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T07:28:00.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:00.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T07:28:00.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T07:28:00.635 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T07:28:00.635 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T07:28:00.635 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T07:28:00.635 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T07:28:00.635 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T07:28:00.635 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T07:28:00.635 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T07:28:00.635 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:00.635 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T07:28:00.635 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:28:00.635 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T07:28:00.635 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T07:28:00.635 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:00.636 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T07:28:00.636 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T07:28:00.636 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T07:28:00.636 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T07:28:00.636 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T07:28:00.636 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T07:28:00.636 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T07:28:00.636 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:28:00.636 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:28:00.636 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:00.636 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T07:28:00.636 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:28:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T07:27:47.694541+0000", 2024-05-27T07:28:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:28:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T07:28:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:28:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.968+0000 7fa35ff58700 1 Processor -- start 2024-05-27T07:28:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.968+0000 7fa35ff58700 1 -- start start 2024-05-27T07:28:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.968+0000 7fa35ff58700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 0x7fa35810c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.968+0000 7fa35ff58700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa35810cce0 con 0x7fa35807ccf0 2024-05-27T07:28:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.969+0000 7fa35dcf4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 0x7fa35810c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.969+0000 7fa35dcf4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 0x7fa35810c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48242/0 (socket says 172.21.15.114:48242) 2024-05-27T07:28:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.969+0000 7fa35dcf4700 1 -- 172.21.15.114:0/3244124043 learned_addr learned my addr 172.21.15.114:0/3244124043 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.969+0000 7fa35dcf4700 1 -- 172.21.15.114:0/3244124043 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35810ce60 con 0x7fa35807ccf0 2024-05-27T07:28:00.638 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.969+0000 7fa35dcf4700 1 --2- 172.21.15.114:0/3244124043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 0x7fa35810c6d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa3540098a0 tx=0x7fa354009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab32eb970b491b72 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:00.638 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.969+0000 7fa35ccf2700 1 -- 172.21.15.114:0/3244124043 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa354004160 con 0x7fa35807ccf0 2024-05-27T07:28:00.638 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.970+0000 7fa35ccf2700 1 -- 172.21.15.114:0/3244124043 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa354004300 con 0x7fa35807ccf0 2024-05-27T07:28:00.638 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.970+0000 7fa35ccf2700 1 -- 172.21.15.114:0/3244124043 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa354011af0 con 0x7fa35807ccf0 2024-05-27T07:28:00.638 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.970+0000 7fa35ff58700 1 -- 172.21.15.114:0/3244124043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 msgr2=0x7fa35810c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:00.638 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.970+0000 7fa35ff58700 1 --2- 172.21.15.114:0/3244124043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 0x7fa35810c6d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa3540098a0 tx=0x7fa354009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:00.638 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.970+0000 7fa35ff58700 1 -- 172.21.15.114:0/3244124043 shutdown_connections 2024-05-27T07:28:00.638 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.970+0000 7fa35ff58700 1 --2- 172.21.15.114:0/3244124043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 0x7fa35810c6d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:00.638 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.970+0000 7fa35ff58700 1 -- 172.21.15.114:0/3244124043 >> 172.21.15.114:0/3244124043 conn(0x7fa358104120 msgr2=0x7fa358106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:00.638 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.971+0000 7fa35ff58700 1 -- 172.21.15.114:0/3244124043 shutdown_connections 2024-05-27T07:28:00.638 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.971+0000 7fa35ff58700 1 -- 172.21.15.114:0/3244124043 wait complete. 2024-05-27T07:28:00.638 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.971+0000 7fa35ff58700 1 Processor -- start 2024-05-27T07:28:00.638 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.971+0000 7fa35ff58700 1 -- start start 2024-05-27T07:28:00.639 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.971+0000 7fa35ff58700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 0x7fa358109e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:00.639 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.971+0000 7fa35ff58700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa35810dee0 con 0x7fa35807ccf0 2024-05-27T07:28:00.639 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.972+0000 7fa35dcf4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 0x7fa358109e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:00.639 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.972+0000 7fa35dcf4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 0x7fa358109e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48250/0 (socket says 172.21.15.114:48250) 2024-05-27T07:28:00.639 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.972+0000 7fa35dcf4700 1 -- 172.21.15.114:0/3092398541 learned_addr learned my addr 172.21.15.114:0/3092398541 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:00.639 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.972+0000 7fa35dcf4700 1 -- 172.21.15.114:0/3092398541 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35810a380 con 0x7fa35807ccf0 2024-05-27T07:28:00.639 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.972+0000 7fa35dcf4700 1 --2- 172.21.15.114:0/3092398541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 0x7fa358109e40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa354009fb0 tx=0x7fa354011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:00.639 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.972+0000 7fa34effd700 1 -- 172.21.15.114:0/3092398541 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa354011bc0 con 0x7fa35807ccf0 2024-05-27T07:28:00.639 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.972+0000 7fa35ff58700 1 -- 172.21.15.114:0/3092398541 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3581085d0 con 0x7fa35807ccf0 2024-05-27T07:28:00.639 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.972+0000 7fa35ff58700 1 -- 172.21.15.114:0/3092398541 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa358108ab0 con 0x7fa35807ccf0 2024-05-27T07:28:00.639 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.972+0000 7fa34effd700 1 -- 172.21.15.114:0/3092398541 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa354024440 con 0x7fa35807ccf0 2024-05-27T07:28:00.639 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.972+0000 7fa34effd700 1 -- 172.21.15.114:0/3092398541 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa35401a5c0 con 0x7fa35807ccf0 2024-05-27T07:28:00.639 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.972+0000 7fa34effd700 1 -- 172.21.15.114:0/3092398541 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa35401abc0 con 0x7fa35807ccf0 2024-05-27T07:28:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.973+0000 7fa34effd700 1 -- 172.21.15.114:0/3092398541 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa354019900 con 0x7fa35807ccf0 2024-05-27T07:28:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.973+0000 7fa35ff58700 1 -- 172.21.15.114:0/3092398541 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa358064e10 con 0x7fa35807ccf0 2024-05-27T07:28:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:27:59.976+0000 7fa34effd700 1 -- 172.21.15.114:0/3092398541 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fa3540245f0 con 0x7fa35807ccf0 2024-05-27T07:28:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:00.049+0000 7fa35ff58700 1 -- 172.21.15.114:0/3092398541 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa358108d30 con 0x7fa35807ccf0 2024-05-27T07:28:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:00.050+0000 7fa34effd700 1 -- 172.21.15.114:0/3092398541 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa354024930 con 0x7fa35807ccf0 2024-05-27T07:28:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:00.050+0000 7fa35ff58700 1 -- 172.21.15.114:0/3092398541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 msgr2=0x7fa358109e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:00.050+0000 7fa35ff58700 1 --2- 172.21.15.114:0/3092398541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 0x7fa358109e40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa354009fb0 tx=0x7fa354011890 comp rx=0 tx=0).stop 2024-05-27T07:28:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:00.051+0000 7fa35ff58700 1 -- 172.21.15.114:0/3092398541 shutdown_connections 2024-05-27T07:28:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:00.051+0000 7fa35ff58700 1 --2- 172.21.15.114:0/3092398541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa35807ccf0 0x7fa358109e40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:00.051+0000 7fa35ff58700 1 -- 172.21.15.114:0/3092398541 >> 172.21.15.114:0/3092398541 conn(0x7fa358104120 msgr2=0x7fa358104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:00.051+0000 7fa35ff58700 1 -- 172.21.15.114:0/3092398541 shutdown_connections 2024-05-27T07:28:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:00.051+0000 7fa35ff58700 1 -- 172.21.15.114:0/3092398541 wait complete. 2024-05-27T07:28:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (2/15)... 2024-05-27T07:28:00.796 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:00.424+0000 7fe032d18200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:28:00.796 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:00.522+0000 7fe032d18200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:28:01.237 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:00.894+0000 7fe032d18200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:28:01.237 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:01.247+0000 7fe032d18200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:28:01.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:01.340+0000 7fe032d18200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:28:02.046 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:01.586+0000 7fe032d18200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:28:02.046 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:01.721+0000 7fe032d18200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:28:02.046 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:01.817+0000 7fe032d18200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:28:02.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:02 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:02.154+0000 7fe032d18200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:28:02.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:02 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:02.247+0000 7fe032d18200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:28:03.046 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:02 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:02.860+0000 7fe032d18200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:28:03.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:03 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:03.085+0000 7fe032d18200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:28:03.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:03 smithi114 ceph-mon[32038]: Activating manager daemon x 2024-05-27T07:28:03.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:03 smithi114 ceph-mon[32038]: mgrmap e2: x(active, starting, since 0.00350074s) 2024-05-27T07:28:03.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:03 smithi114 ceph-mon[32038]: from='mgr.14100 172.21.15.114:0/3662655871' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:28:03.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:03 smithi114 ceph-mon[32038]: from='mgr.14100 172.21.15.114:0/3662655871' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:28:03.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:03 smithi114 ceph-mon[32038]: from='mgr.14100 172.21.15.114:0/3662655871' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:28:03.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:03 smithi114 ceph-mon[32038]: from='mgr.14100 172.21.15.114:0/3662655871' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:28:03.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:03 smithi114 ceph-mon[32038]: from='mgr.14100 172.21.15.114:0/3662655871' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:28:03.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:03 smithi114 ceph-mon[32038]: Manager daemon x is now available 2024-05-27T07:28:03.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:03 smithi114 ceph-mon[32038]: from='mgr.14100 172.21.15.114:0/3662655871' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:28:03.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:03 smithi114 ceph-mon[32038]: from='mgr.14100 172.21.15.114:0/3662655871' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:28:03.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:03 smithi114 ceph-mon[32038]: from='mgr.14100 172.21.15.114:0/3662655871' entity='mgr.x' 2024-05-27T07:28:03.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:03 smithi114 ceph-mon[32038]: from='mgr.14100 172.21.15.114:0/3662655871' entity='mgr.x' 2024-05-27T07:28:03.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:03 smithi114 ceph-mon[32038]: from='mgr.14100 172.21.15.114:0/3662655871' entity='mgr.x' 2024-05-27T07:28:03.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-05-27T07:28:03.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:28:03.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "755b3728-1bfa-11ef-bc9b-c7b262605968", 2024-05-27T07:28:03.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T07:28:03.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T07:28:03.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T07:28:03.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T07:28:03.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:03.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T07:28:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T07:28:03.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T07:28:03.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T07:28:03.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T07:28:03.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:03.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T07:28:03.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:28:03.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T07:28:03.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T07:28:03.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:03.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T07:28:03.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T07:28:03.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T07:28:03.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T07:28:03.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T07:28:03.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T07:28:03.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T07:28:03.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:28:03.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:28:03.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:03.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T07:28:03.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:28:03.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T07:27:47.694541+0000", 2024-05-27T07:28:03.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:28:03.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:03.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T07:28:03.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:28:03.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.094+0000 7f74fcec3700 1 Processor -- start 2024-05-27T07:28:03.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.094+0000 7f74fcec3700 1 -- start start 2024-05-27T07:28:03.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.094+0000 7f74fcec3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 0x7f74f810a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:03.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.094+0000 7f74fcec3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f74f810a6a0 con 0x7f74f8109ca0 2024-05-27T07:28:03.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.094+0000 7f74f659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 0x7f74f810a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:03.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.094+0000 7f74f659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 0x7f74f810a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48270/0 (socket says 172.21.15.114:48270) 2024-05-27T07:28:03.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.094+0000 7f74f659c700 1 -- 172.21.15.114:0/4221934673 learned_addr learned my addr 172.21.15.114:0/4221934673 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:03.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.095+0000 7f74f659c700 1 -- 172.21.15.114:0/4221934673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f810add0 con 0x7f74f8109ca0 2024-05-27T07:28:03.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.095+0000 7f74f659c700 1 --2- 172.21.15.114:0/4221934673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 0x7f74f810a0b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f74e80098a0 tx=0x7f74e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9221540acba64d9e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:03.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.095+0000 7f74f559a700 1 -- 172.21.15.114:0/4221934673 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74e8004160 con 0x7f74f8109ca0 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.095+0000 7f74f559a700 1 -- 172.21.15.114:0/4221934673 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f74e8004300 con 0x7f74f8109ca0 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.095+0000 7f74f559a700 1 -- 172.21.15.114:0/4221934673 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74e8011af0 con 0x7f74f8109ca0 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.096+0000 7f74fcec3700 1 -- 172.21.15.114:0/4221934673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 msgr2=0x7f74f810a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.096+0000 7f74fcec3700 1 --2- 172.21.15.114:0/4221934673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 0x7f74f810a0b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f74e80098a0 tx=0x7f74e8009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.096+0000 7f74fcec3700 1 -- 172.21.15.114:0/4221934673 shutdown_connections 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.096+0000 7f74fcec3700 1 --2- 172.21.15.114:0/4221934673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 0x7f74f810a0b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.096+0000 7f74fcec3700 1 -- 172.21.15.114:0/4221934673 >> 172.21.15.114:0/4221934673 conn(0x7f74f81052c0 msgr2=0x7f74f8107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.096+0000 7f74fcec3700 1 -- 172.21.15.114:0/4221934673 shutdown_connections 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.096+0000 7f74fcec3700 1 -- 172.21.15.114:0/4221934673 wait complete. 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.097+0000 7f74fcec3700 1 Processor -- start 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.097+0000 7f74fcec3700 1 -- start start 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.097+0000 7f74fcec3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 0x7f74f807f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.097+0000 7f74fcec3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f74f810e180 con 0x7f74f8109ca0 2024-05-27T07:28:03.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.098+0000 7f74f659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 0x7f74f807f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:03.794 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.098+0000 7f74f659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 0x7f74f807f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48286/0 (socket says 172.21.15.114:48286) 2024-05-27T07:28:03.794 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.098+0000 7f74f659c700 1 -- 172.21.15.114:0/1313682930 learned_addr learned my addr 172.21.15.114:0/1313682930 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:03.794 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.098+0000 7f74f659c700 1 -- 172.21.15.114:0/1313682930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f807fa00 con 0x7f74f8109ca0 2024-05-27T07:28:03.794 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.098+0000 7f74f659c700 1 --2- 172.21.15.114:0/1313682930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 0x7f74f807f4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f74e8009fb0 tx=0x7f74e8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:03.794 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.099+0000 7f74e77fe700 1 -- 172.21.15.114:0/1313682930 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74e80036a0 con 0x7f74f8109ca0 2024-05-27T07:28:03.794 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.099+0000 7f74e77fe700 1 -- 172.21.15.114:0/1313682930 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f74e8003cd0 con 0x7f74f8109ca0 2024-05-27T07:28:03.794 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.099+0000 7f74e77fe700 1 -- 172.21.15.114:0/1313682930 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74e801a430 con 0x7f74f8109ca0 2024-05-27T07:28:03.794 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.099+0000 7f74fcec3700 1 -- 172.21.15.114:0/1313682930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74f807dc50 con 0x7f74f8109ca0 2024-05-27T07:28:03.794 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.099+0000 7f74fcec3700 1 -- 172.21.15.114:0/1313682930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74f807e070 con 0x7f74f8109ca0 2024-05-27T07:28:03.794 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.100+0000 7f74e77fe700 1 -- 172.21.15.114:0/1313682930 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 2) ==== 49978+0+0 (secure 0 0 0) 0x7f74e801a6a0 con 0x7f74f8109ca0 2024-05-27T07:28:03.794 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.100+0000 7f74fcec3700 1 -- 172.21.15.114:0/1313682930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74d8005360 con 0x7f74f8109ca0 2024-05-27T07:28:03.795 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.100+0000 7f74e77fe700 1 -- 172.21.15.114:0/1313682930 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f74e8054180 con 0x7f74f8109ca0 2024-05-27T07:28:03.795 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.102+0000 7f74e77fe700 1 -- 172.21.15.114:0/1313682930 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f74e801a990 con 0x7f74f8109ca0 2024-05-27T07:28:03.795 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.180+0000 7f74fcec3700 1 -- 172.21.15.114:0/1313682930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f74d8005650 con 0x7f74f8109ca0 2024-05-27T07:28:03.795 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.180+0000 7f74e77fe700 1 -- 172.21.15.114:0/1313682930 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f74e8040230 con 0x7f74f8109ca0 2024-05-27T07:28:03.795 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.181+0000 7f74fcec3700 1 -- 172.21.15.114:0/1313682930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 msgr2=0x7f74f807f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:03.795 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.181+0000 7f74fcec3700 1 --2- 172.21.15.114:0/1313682930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 0x7f74f807f4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f74e8009fb0 tx=0x7f74e8011890 comp rx=0 tx=0).stop 2024-05-27T07:28:03.795 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.181+0000 7f74fcec3700 1 -- 172.21.15.114:0/1313682930 shutdown_connections 2024-05-27T07:28:03.795 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.181+0000 7f74fcec3700 1 --2- 172.21.15.114:0/1313682930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74f8109ca0 0x7f74f807f4c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:03.795 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.181+0000 7f74fcec3700 1 -- 172.21.15.114:0/1313682930 >> 172.21.15.114:0/1313682930 conn(0x7f74f81052c0 msgr2=0x7f74f8105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:03.795 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.182+0000 7f74fcec3700 1 -- 172.21.15.114:0/1313682930 shutdown_connections 2024-05-27T07:28:03.795 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:03.182+0000 7f74fcec3700 1 -- 172.21.15.114:0/1313682930 wait complete. 2024-05-27T07:28:03.795 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (3/15)... 2024-05-27T07:28:04.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:04 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1313682930' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T07:28:04.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:04 smithi114 ceph-mon[32038]: mgrmap e3: x(active, since 1.00757s) 2024-05-27T07:28:06.178 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:06 smithi114 ceph-mon[32038]: mgrmap e4: x(active, since 2s) 2024-05-27T07:28:07.071 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-05-27T07:28:07.071 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:28:07.071 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "755b3728-1bfa-11ef-bc9b-c7b262605968", 2024-05-27T07:28:07.071 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T07:28:07.071 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T07:28:07.071 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T07:28:07.072 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T07:28:07.072 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:07.072 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T07:28:07.072 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T07:28:07.072 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-05-27T07:28:07.072 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:28:07.072 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T07:28:07.072 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T07:28:07.072 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:28:07.072 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-05-27T07:28:07.072 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T07:28:07.072 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:28:07.072 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T07:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T07:28:07.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T07:28:07.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T07:28:07.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T07:28:07.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T07:28:07.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T07:28:07.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T07:28:07.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:07.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T07:28:07.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:28:07.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T07:28:07.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T07:28:07.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:07.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T07:27:47.694541+0000", 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:28:07.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.223+0000 7f08516be700 1 Processor -- start 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.223+0000 7f08516be700 1 -- start start 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.223+0000 7f08516be700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 0x7f084c109ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.223+0000 7f08516be700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f084c10a5e0 con 0x7f084c109be0 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.224+0000 7f084affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 0x7f084c109ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.224+0000 7f084affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 0x7f084c109ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55720/0 (socket says 172.21.15.114:55720) 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.224+0000 7f084affd700 1 -- 172.21.15.114:0/2207376036 learned_addr learned my addr 172.21.15.114:0/2207376036 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.224+0000 7f084affd700 1 -- 172.21.15.114:0/2207376036 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084c10ad10 con 0x7f084c109be0 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.224+0000 7f084affd700 1 --2- 172.21.15.114:0/2207376036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 0x7f084c109ff0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f083c009e80 tx=0x7f083c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14413971bb404762 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.224+0000 7f0849ffb700 1 -- 172.21.15.114:0/2207376036 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f083c017070 con 0x7f084c109be0 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.224+0000 7f0849ffb700 1 -- 172.21.15.114:0/2207376036 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f083c00b8c0 con 0x7f084c109be0 2024-05-27T07:28:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.224+0000 7f0849ffb700 1 -- 172.21.15.114:0/2207376036 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f083c004490 con 0x7f084c109be0 2024-05-27T07:28:07.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.225+0000 7f08516be700 1 -- 172.21.15.114:0/2207376036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 msgr2=0x7f084c109ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:07.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.225+0000 7f08516be700 1 --2- 172.21.15.114:0/2207376036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 0x7f084c109ff0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f083c009e80 tx=0x7f083c00b330 comp rx=0 tx=0).stop 2024-05-27T07:28:07.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.225+0000 7f08516be700 1 -- 172.21.15.114:0/2207376036 shutdown_connections 2024-05-27T07:28:07.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.225+0000 7f08516be700 1 --2- 172.21.15.114:0/2207376036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 0x7f084c109ff0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:07.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.225+0000 7f08516be700 1 -- 172.21.15.114:0/2207376036 >> 172.21.15.114:0/2207376036 conn(0x7f084c105280 msgr2=0x7f084c1076d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:07.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.225+0000 7f08516be700 1 -- 172.21.15.114:0/2207376036 shutdown_connections 2024-05-27T07:28:07.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.225+0000 7f08516be700 1 -- 172.21.15.114:0/2207376036 wait complete. 2024-05-27T07:28:07.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.226+0000 7f08516be700 1 Processor -- start 2024-05-27T07:28:07.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.226+0000 7f08516be700 1 -- start start 2024-05-27T07:28:07.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.227+0000 7f08516be700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 0x7f084c19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:07.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.227+0000 7f08516be700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f084c10e0c0 con 0x7f084c109be0 2024-05-27T07:28:07.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.227+0000 7f084affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 0x7f084c19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:07.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.227+0000 7f084affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 0x7f084c19dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55724/0 (socket says 172.21.15.114:55724) 2024-05-27T07:28:07.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.227+0000 7f084affd700 1 -- 172.21.15.114:0/3335676545 learned_addr learned my addr 172.21.15.114:0/3335676545 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:07.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.227+0000 7f084affd700 1 -- 172.21.15.114:0/3335676545 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084c19e500 con 0x7f084c109be0 2024-05-27T07:28:07.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.227+0000 7f084affd700 1 --2- 172.21.15.114:0/3335676545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 0x7f084c19dfc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f083c009fd0 tx=0x7f083c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:07.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.227+0000 7f0843fff700 1 -- 172.21.15.114:0/3335676545 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f083c020020 con 0x7f084c109be0 2024-05-27T07:28:07.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.228+0000 7f0843fff700 1 -- 172.21.15.114:0/3335676545 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f083c012430 con 0x7f084c109be0 2024-05-27T07:28:07.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.228+0000 7f0843fff700 1 -- 172.21.15.114:0/3335676545 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f083c017040 con 0x7f084c109be0 2024-05-27T07:28:07.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.228+0000 7f08516be700 1 -- 172.21.15.114:0/3335676545 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f084c19e790 con 0x7f084c109be0 2024-05-27T07:28:07.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.228+0000 7f08516be700 1 -- 172.21.15.114:0/3335676545 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f084c1a13a0 con 0x7f084c109be0 2024-05-27T07:28:07.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.229+0000 7f08516be700 1 -- 172.21.15.114:0/3335676545 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f084c064e10 con 0x7f084c109be0 2024-05-27T07:28:07.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.229+0000 7f0843fff700 1 -- 172.21.15.114:0/3335676545 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f083c023020 con 0x7f084c109be0 2024-05-27T07:28:07.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.229+0000 7f0843fff700 1 --2- 172.21.15.114:0/3335676545 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083803e420 0x7f0838040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:07.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.229+0000 7f0843fff700 1 -- 172.21.15.114:0/3335676545 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f083c00f050 con 0x7f084c109be0 2024-05-27T07:28:07.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.229+0000 7f084a7fc700 1 --2- 172.21.15.114:0/3335676545 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083803e420 0x7f0838040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:07.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.230+0000 7f084a7fc700 1 --2- 172.21.15.114:0/3335676545 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083803e420 0x7f0838040930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0834006fd0 tx=0x7f0834006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:07.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.235+0000 7f0843fff700 1 -- 172.21.15.114:0/3335676545 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f083c02a020 con 0x7f084c109be0 2024-05-27T07:28:07.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.541+0000 7f08516be700 1 -- 172.21.15.114:0/3335676545 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f084c1a1a70 con 0x7f084c109be0 2024-05-27T07:28:07.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.542+0000 7f0843fff700 1 -- 172.21.15.114:0/3335676545 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f083c02dd60 con 0x7f084c109be0 2024-05-27T07:28:07.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.544+0000 7f08516be700 1 -- 172.21.15.114:0/3335676545 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083803e420 msgr2=0x7f0838040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:07.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.544+0000 7f08516be700 1 --2- 172.21.15.114:0/3335676545 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083803e420 0x7f0838040930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0834006fd0 tx=0x7f0834006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:07.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.544+0000 7f08516be700 1 -- 172.21.15.114:0/3335676545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 msgr2=0x7f084c19dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:07.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.544+0000 7f08516be700 1 --2- 172.21.15.114:0/3335676545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 0x7f084c19dfc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f083c009fd0 tx=0x7f083c00bd50 comp rx=0 tx=0).stop 2024-05-27T07:28:07.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.544+0000 7f08516be700 1 -- 172.21.15.114:0/3335676545 shutdown_connections 2024-05-27T07:28:07.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.544+0000 7f08516be700 1 --2- 172.21.15.114:0/3335676545 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083803e420 0x7f0838040930 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:07.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.544+0000 7f08516be700 1 --2- 172.21.15.114:0/3335676545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c109be0 0x7f084c19dfc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:07.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.544+0000 7f08516be700 1 -- 172.21.15.114:0/3335676545 >> 172.21.15.114:0/3335676545 conn(0x7f084c105280 msgr2=0x7f084c105ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:07.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.545+0000 7f08516be700 1 -- 172.21.15.114:0/3335676545 shutdown_connections 2024-05-27T07:28:07.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:06.545+0000 7f08516be700 1 -- 172.21.15.114:0/3335676545 wait complete. 2024-05-27T07:28:07.080 INFO:teuthology.orchestra.run.smithi114.stdout:mgr is available 2024-05-27T07:28:07.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:07 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3335676545' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T07:28:08.296 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:08 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3638726684' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-27T07:28:08.321 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-05-27T07:28:08.321 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-05-27T07:28:08.321 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:28:08.321 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-27T07:28:08.321 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-27T07:28:08.321 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-27T07:28:08.321 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-27T07:28:08.321 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-27T07:28:08.321 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-05-27T07:28:08.321 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-27T07:28:08.322 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-27T07:28:08.322 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-05-27T07:28:08.322 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-05-27T07:28:08.322 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-27T07:28:08.322 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-27T07:28:08.322 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-27T07:28:08.322 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-27T07:28:08.322 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-27T07:28:08.322 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.513+0000 7f23653c4700 1 Processor -- start 2024-05-27T07:28:08.322 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.513+0000 7f23653c4700 1 -- start start 2024-05-27T07:28:08.322 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.513+0000 7f23653c4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 0x7f23601088e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:08.322 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.513+0000 7f23653c4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f236007d690 con 0x7f23601084d0 2024-05-27T07:28:08.322 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.513+0000 7f235effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 0x7f23601088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:08.323 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.514+0000 7f235effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 0x7f23601088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55740/0 (socket says 172.21.15.114:55740) 2024-05-27T07:28:08.323 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.514+0000 7f235effd700 1 -- 172.21.15.114:0/545047077 learned_addr learned my addr 172.21.15.114:0/545047077 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:08.323 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.514+0000 7f235effd700 1 -- 172.21.15.114:0/545047077 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2360108e20 con 0x7f23601084d0 2024-05-27T07:28:08.323 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.514+0000 7f235effd700 1 --2- 172.21.15.114:0/545047077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 0x7f23601088e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2350009e80 tx=0x7f235000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43206bd47e8e3461 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:08.323 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.514+0000 7f235e7fc700 1 -- 172.21.15.114:0/545047077 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2350017070 con 0x7f23601084d0 2024-05-27T07:28:08.323 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.514+0000 7f235e7fc700 1 -- 172.21.15.114:0/545047077 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f235000b8c0 con 0x7f23601084d0 2024-05-27T07:28:08.323 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.514+0000 7f235e7fc700 1 -- 172.21.15.114:0/545047077 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2350004490 con 0x7f23601084d0 2024-05-27T07:28:08.323 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.515+0000 7f23653c4700 1 -- 172.21.15.114:0/545047077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 msgr2=0x7f23601088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:08.323 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.515+0000 7f23653c4700 1 --2- 172.21.15.114:0/545047077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 0x7f23601088e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2350009e80 tx=0x7f235000b330 comp rx=0 tx=0).stop 2024-05-27T07:28:08.323 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.515+0000 7f23653c4700 1 -- 172.21.15.114:0/545047077 shutdown_connections 2024-05-27T07:28:08.323 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.515+0000 7f23653c4700 1 --2- 172.21.15.114:0/545047077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 0x7f23601088e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:08.323 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.515+0000 7f23653c4700 1 -- 172.21.15.114:0/545047077 >> 172.21.15.114:0/545047077 conn(0x7f23601041c0 msgr2=0x7f2360106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:08.323 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.515+0000 7f23653c4700 1 -- 172.21.15.114:0/545047077 shutdown_connections 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.515+0000 7f23653c4700 1 -- 172.21.15.114:0/545047077 wait complete. 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.516+0000 7f23653c4700 1 Processor -- start 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.516+0000 7f23653c4700 1 -- start start 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.516+0000 7f23653c4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 0x7f23601a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.516+0000 7f23653c4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2360109d20 con 0x7f23601084d0 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.516+0000 7f235effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 0x7f23601a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.516+0000 7f235effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 0x7f23601a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55750/0 (socket says 172.21.15.114:55750) 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.516+0000 7f235effd700 1 -- 172.21.15.114:0/3638726684 learned_addr learned my addr 172.21.15.114:0/3638726684 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.516+0000 7f235effd700 1 -- 172.21.15.114:0/3638726684 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23601a0690 con 0x7f23601084d0 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.516+0000 7f235effd700 1 --2- 172.21.15.114:0/3638726684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 0x7f23601a0150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2350004220 tx=0x7f235000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.516+0000 7f235cff9700 1 -- 172.21.15.114:0/3638726684 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2350020020 con 0x7f23601084d0 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.516+0000 7f235cff9700 1 -- 172.21.15.114:0/3638726684 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2350012430 con 0x7f23601084d0 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.517+0000 7f23653c4700 1 -- 172.21.15.114:0/3638726684 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23601a0920 con 0x7f23601084d0 2024-05-27T07:28:08.324 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.517+0000 7f235cff9700 1 -- 172.21.15.114:0/3638726684 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2350017040 con 0x7f23601084d0 2024-05-27T07:28:08.325 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.517+0000 7f23653c4700 1 -- 172.21.15.114:0/3638726684 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23601a3530 con 0x7f23601084d0 2024-05-27T07:28:08.325 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.518+0000 7f23653c4700 1 -- 172.21.15.114:0/3638726684 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2360064e10 con 0x7f23601084d0 2024-05-27T07:28:08.325 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.518+0000 7f235cff9700 1 -- 172.21.15.114:0/3638726684 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f2350023020 con 0x7f23601084d0 2024-05-27T07:28:08.325 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.518+0000 7f235cff9700 1 --2- 172.21.15.114:0/3638726684 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f234803e420 0x7f2348040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:08.325 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.518+0000 7f235cff9700 1 -- 172.21.15.114:0/3638726684 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f235000f050 con 0x7f23601084d0 2024-05-27T07:28:08.325 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.518+0000 7f234ffff700 1 --2- 172.21.15.114:0/3638726684 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f234803e420 0x7f2348040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:08.325 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.519+0000 7f234ffff700 1 --2- 172.21.15.114:0/3638726684 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f234803e420 0x7f2348040930 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2354006fd0 tx=0x7f2354006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:08.325 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.523+0000 7f235cff9700 1 -- 172.21.15.114:0/3638726684 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f235002a020 con 0x7f23601084d0 2024-05-27T07:28:08.325 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.722+0000 7f23653c4700 1 -- 172.21.15.114:0/3638726684 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f236002e970 con 0x7f23601084d0 2024-05-27T07:28:08.325 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.724+0000 7f235cff9700 1 -- 172.21.15.114:0/3638726684 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f235002a200 con 0x7f23601084d0 2024-05-27T07:28:08.325 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.727+0000 7f23653c4700 1 -- 172.21.15.114:0/3638726684 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f234803e420 msgr2=0x7f2348040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:08.325 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.727+0000 7f23653c4700 1 --2- 172.21.15.114:0/3638726684 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f234803e420 0x7f2348040930 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2354006fd0 tx=0x7f2354006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:08.325 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.727+0000 7f23653c4700 1 -- 172.21.15.114:0/3638726684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 msgr2=0x7f23601a0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:08.326 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.727+0000 7f23653c4700 1 --2- 172.21.15.114:0/3638726684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 0x7f23601a0150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2350004220 tx=0x7f235000bd50 comp rx=0 tx=0).stop 2024-05-27T07:28:08.326 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.727+0000 7f23653c4700 1 -- 172.21.15.114:0/3638726684 shutdown_connections 2024-05-27T07:28:08.326 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.727+0000 7f23653c4700 1 --2- 172.21.15.114:0/3638726684 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f234803e420 0x7f2348040930 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:08.326 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.727+0000 7f23653c4700 1 --2- 172.21.15.114:0/3638726684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23601084d0 0x7f23601a0150 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:08.326 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.727+0000 7f23653c4700 1 -- 172.21.15.114:0/3638726684 >> 172.21.15.114:0/3638726684 conn(0x7f23601041c0 msgr2=0x7f2360194160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:08.326 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.727+0000 7f23653c4700 1 -- 172.21.15.114:0/3638726684 shutdown_connections 2024-05-27T07:28:08.326 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:07.727+0000 7f23653c4700 1 -- 172.21.15.114:0/3638726684 wait complete. 2024-05-27T07:28:08.326 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling cephadm module... 2024-05-27T07:28:09.296 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:09 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3789118292' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-27T07:28:09.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:09 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: ignoring --setuser ceph since I am not root 2024-05-27T07:28:09.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:09 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: ignoring --setgroup ceph since I am not root 2024-05-27T07:28:09.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:09 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:09.343+0000 7f5b7cd2b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:28:09.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:09 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:09.533+0000 7f5b7cd2b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:28:09.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:09 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:09.637+0000 7f5b7cd2b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:28:09.688 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.776+0000 7f70994cf700 1 Processor -- start 2024-05-27T07:28:09.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.776+0000 7f70994cf700 1 -- start start 2024-05-27T07:28:09.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.777+0000 7f70994cf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 0x7f709410caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:09.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.777+0000 7f70994cf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f709407ce00 con 0x7f709410a680 2024-05-27T07:28:09.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.777+0000 7f7092ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 0x7f709410caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:09.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.777+0000 7f7092ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 0x7f709410caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55760/0 (socket says 172.21.15.114:55760) 2024-05-27T07:28:09.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.777+0000 7f7092ffd700 1 -- 172.21.15.114:0/2429794327 learned_addr learned my addr 172.21.15.114:0/2429794327 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:09.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.777+0000 7f7092ffd700 1 -- 172.21.15.114:0/2429794327 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709410cfe0 con 0x7f709410a680 2024-05-27T07:28:09.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.777+0000 7f7092ffd700 1 --2- 172.21.15.114:0/2429794327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 0x7f709410caa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f70840098a0 tx=0x7f7084009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb371a08304361b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:09.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.778+0000 7f70927fc700 1 -- 172.21.15.114:0/2429794327 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7084004160 con 0x7f709410a680 2024-05-27T07:28:09.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.778+0000 7f70927fc700 1 -- 172.21.15.114:0/2429794327 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7084004300 con 0x7f709410a680 2024-05-27T07:28:09.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.778+0000 7f70927fc700 1 -- 172.21.15.114:0/2429794327 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7084011af0 con 0x7f709410a680 2024-05-27T07:28:09.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.778+0000 7f70994cf700 1 -- 172.21.15.114:0/2429794327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 msgr2=0x7f709410caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:09.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.778+0000 7f70994cf700 1 --2- 172.21.15.114:0/2429794327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 0x7f709410caa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f70840098a0 tx=0x7f7084009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:09.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.778+0000 7f70994cf700 1 -- 172.21.15.114:0/2429794327 shutdown_connections 2024-05-27T07:28:09.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.778+0000 7f70994cf700 1 --2- 172.21.15.114:0/2429794327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 0x7f709410caa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:09.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.778+0000 7f70994cf700 1 -- 172.21.15.114:0/2429794327 >> 172.21.15.114:0/2429794327 conn(0x7f7094104120 msgr2=0x7f7094106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:09.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.779+0000 7f70994cf700 1 -- 172.21.15.114:0/2429794327 shutdown_connections 2024-05-27T07:28:09.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.779+0000 7f70994cf700 1 -- 172.21.15.114:0/2429794327 wait complete. 2024-05-27T07:28:09.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.779+0000 7f70994cf700 1 Processor -- start 2024-05-27T07:28:09.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.779+0000 7f70994cf700 1 -- start start 2024-05-27T07:28:09.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.780+0000 7f70994cf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 0x7f709419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:09.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.780+0000 7f70994cf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f709410dee0 con 0x7f709410a680 2024-05-27T07:28:09.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.780+0000 7f7092ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 0x7f709419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:09.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.780+0000 7f7092ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 0x7f709419dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55762/0 (socket says 172.21.15.114:55762) 2024-05-27T07:28:09.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.780+0000 7f7092ffd700 1 -- 172.21.15.114:0/3789118292 learned_addr learned my addr 172.21.15.114:0/3789118292 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:09.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.780+0000 7f7092ffd700 1 -- 172.21.15.114:0/3789118292 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709419e4e0 con 0x7f709410a680 2024-05-27T07:28:09.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.780+0000 7f7092ffd700 1 --2- 172.21.15.114:0/3789118292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 0x7f709419dfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f70840117b0 tx=0x7f7084011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:09.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.780+0000 7f7090ff9700 1 -- 172.21.15.114:0/3789118292 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70840039c0 con 0x7f709410a680 2024-05-27T07:28:09.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.780+0000 7f7090ff9700 1 -- 172.21.15.114:0/3789118292 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7084003b60 con 0x7f709410a680 2024-05-27T07:28:09.696 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.780+0000 7f7090ff9700 1 -- 172.21.15.114:0/3789118292 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70840196a0 con 0x7f709410a680 2024-05-27T07:28:09.696 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.780+0000 7f70994cf700 1 -- 172.21.15.114:0/3789118292 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f709419e770 con 0x7f709410a680 2024-05-27T07:28:09.696 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.780+0000 7f70994cf700 1 -- 172.21.15.114:0/3789118292 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70941a1380 con 0x7f709410a680 2024-05-27T07:28:09.696 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.781+0000 7f7090ff9700 1 -- 172.21.15.114:0/3789118292 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f7084019960 con 0x7f709410a680 2024-05-27T07:28:09.696 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.781+0000 7f70994cf700 1 -- 172.21.15.114:0/3789118292 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7094064e10 con 0x7f709410a680 2024-05-27T07:28:09.696 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.781+0000 7f7090ff9700 1 --2- 172.21.15.114:0/3789118292 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f708003e420 0x7f7080040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:09.696 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.782+0000 7f7090ff9700 1 -- 172.21.15.114:0/3789118292 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7084053a90 con 0x7f709410a680 2024-05-27T07:28:09.696 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.782+0000 7f708a7fc700 1 --2- 172.21.15.114:0/3789118292 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f708003e420 0x7f7080040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:09.696 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.782+0000 7f708a7fc700 1 --2- 172.21.15.114:0/3789118292 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f708003e420 0x7f7080040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f707c006fd0 tx=0x7f707c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:09.696 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:08.787+0000 7f7090ff9700 1 -- 172.21.15.114:0/3789118292 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7084017e90 con 0x7f709410a680 2024-05-27T07:28:09.696 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:09.043+0000 7f70994cf700 1 -- 172.21.15.114:0/3789118292 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f70941a1a00 con 0x7f709410a680 2024-05-27T07:28:09.696 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:09.145+0000 7f7090ff9700 1 -- 172.21.15.114:0/3789118292 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f7084023390 con 0x7f709410a680 2024-05-27T07:28:09.696 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:09.146+0000 7f7090ff9700 1 -- 172.21.15.114:0/3789118292 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f708400fd80 con 0x7f709410a680 2024-05-27T07:28:09.697 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:09.148+0000 7f70994cf700 1 -- 172.21.15.114:0/3789118292 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f708003e420 msgr2=0x7f7080040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:09.697 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:09.148+0000 7f70994cf700 1 --2- 172.21.15.114:0/3789118292 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f708003e420 0x7f7080040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f707c006fd0 tx=0x7f707c006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:09.697 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:09.148+0000 7f70994cf700 1 -- 172.21.15.114:0/3789118292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 msgr2=0x7f709419dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:09.697 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:09.148+0000 7f70994cf700 1 --2- 172.21.15.114:0/3789118292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 0x7f709419dfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f70840117b0 tx=0x7f7084011b70 comp rx=0 tx=0).stop 2024-05-27T07:28:09.697 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:09.148+0000 7f70994cf700 1 -- 172.21.15.114:0/3789118292 shutdown_connections 2024-05-27T07:28:09.697 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:09.148+0000 7f70994cf700 1 --2- 172.21.15.114:0/3789118292 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f708003e420 0x7f7080040930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:09.697 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:09.148+0000 7f70994cf700 1 --2- 172.21.15.114:0/3789118292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f709410a680 0x7f709419dfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:09.697 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:09.148+0000 7f70994cf700 1 -- 172.21.15.114:0/3789118292 >> 172.21.15.114:0/3789118292 conn(0x7f7094104120 msgr2=0x7f7094104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:09.697 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:09.148+0000 7f70994cf700 1 -- 172.21.15.114:0/3789118292 shutdown_connections 2024-05-27T07:28:09.697 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:09.148+0000 7f70994cf700 1 -- 172.21.15.114:0/3789118292 wait complete. 2024-05-27T07:28:10.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:10 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3789118292' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-27T07:28:10.296 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:10 smithi114 ceph-mon[32038]: mgrmap e5: x(active, since 6s) 2024-05-27T07:28:10.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:09 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:09.960+0000 7f5b7cd2b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:28:10.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:10 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:10.049+0000 7f5b7cd2b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:28:11.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:28:11.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-27T07:28:11.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T07:28:11.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-27T07:28:11.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-27T07:28:11.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:28:11.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.176+0000 7f0851542700 1 Processor -- start 2024-05-27T07:28:11.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.176+0000 7f0851542700 1 -- start start 2024-05-27T07:28:11.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.177+0000 7f0851542700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 0x7f084c108f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:11.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.177+0000 7f0851542700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f084c109500 con 0x7f084c108b00 2024-05-27T07:28:11.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.177+0000 7f084affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 0x7f084c108f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:11.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.177+0000 7f084affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 0x7f084c108f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55790/0 (socket says 172.21.15.114:55790) 2024-05-27T07:28:11.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.177+0000 7f084affd700 1 -- 172.21.15.114:0/3028593045 learned_addr learned my addr 172.21.15.114:0/3028593045 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:11.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.177+0000 7f084affd700 1 -- 172.21.15.114:0/3028593045 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084c109c30 con 0x7f084c108b00 2024-05-27T07:28:11.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.177+0000 7f084affd700 1 --2- 172.21.15.114:0/3028593045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 0x7f084c108f10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f083c0098a0 tx=0x7f083c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b8bbb23b72eefb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:11.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.177+0000 7f0849ffb700 1 -- 172.21.15.114:0/3028593045 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f083c004160 con 0x7f084c108b00 2024-05-27T07:28:11.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.177+0000 7f0849ffb700 1 -- 172.21.15.114:0/3028593045 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f083c004300 con 0x7f084c108b00 2024-05-27T07:28:11.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.177+0000 7f0849ffb700 1 -- 172.21.15.114:0/3028593045 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f083c011af0 con 0x7f084c108b00 2024-05-27T07:28:11.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.178+0000 7f0851542700 1 -- 172.21.15.114:0/3028593045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 msgr2=0x7f084c108f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:11.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.178+0000 7f0851542700 1 --2- 172.21.15.114:0/3028593045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 0x7f084c108f10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f083c0098a0 tx=0x7f083c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:11.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.178+0000 7f0851542700 1 -- 172.21.15.114:0/3028593045 shutdown_connections 2024-05-27T07:28:11.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.178+0000 7f0851542700 1 --2- 172.21.15.114:0/3028593045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 0x7f084c108f10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:11.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.178+0000 7f0851542700 1 -- 172.21.15.114:0/3028593045 >> 172.21.15.114:0/3028593045 conn(0x7f084c1041c0 msgr2=0x7f084c106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:11.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.178+0000 7f0851542700 1 -- 172.21.15.114:0/3028593045 shutdown_connections 2024-05-27T07:28:11.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.178+0000 7f0851542700 1 -- 172.21.15.114:0/3028593045 wait complete. 2024-05-27T07:28:11.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.179+0000 7f0851542700 1 Processor -- start 2024-05-27T07:28:11.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.179+0000 7f0851542700 1 -- start start 2024-05-27T07:28:11.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.179+0000 7f0851542700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 0x7f084c07d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:11.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.179+0000 7f0851542700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f084c10cfe0 con 0x7f084c108b00 2024-05-27T07:28:11.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.180+0000 7f084affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 0x7f084c07d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:11.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.180+0000 7f084affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 0x7f084c07d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55800/0 (socket says 172.21.15.114:55800) 2024-05-27T07:28:11.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.180+0000 7f084affd700 1 -- 172.21.15.114:0/2262698559 learned_addr learned my addr 172.21.15.114:0/2262698559 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:11.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.180+0000 7f084affd700 1 -- 172.21.15.114:0/2262698559 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084c07d740 con 0x7f084c108b00 2024-05-27T07:28:11.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.180+0000 7f084affd700 1 --2- 172.21.15.114:0/2262698559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 0x7f084c07d200 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f083c011890 tx=0x7f083c011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:11.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.180+0000 7f0833fff700 1 -- 172.21.15.114:0/2262698559 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f083c0039e0 con 0x7f084c108b00 2024-05-27T07:28:11.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.180+0000 7f0833fff700 1 -- 172.21.15.114:0/2262698559 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f083c003b80 con 0x7f084c108b00 2024-05-27T07:28:11.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.180+0000 7f0851542700 1 -- 172.21.15.114:0/2262698559 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f084c1a0d70 con 0x7f084c108b00 2024-05-27T07:28:11.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.180+0000 7f0833fff700 1 -- 172.21.15.114:0/2262698559 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f083c0246e0 con 0x7f084c108b00 2024-05-27T07:28:11.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.181+0000 7f0851542700 1 -- 172.21.15.114:0/2262698559 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f084c1a11f0 con 0x7f084c108b00 2024-05-27T07:28:11.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.182+0000 7f0833fff700 1 -- 172.21.15.114:0/2262698559 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f083c022020 con 0x7f084c108b00 2024-05-27T07:28:11.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.182+0000 7f0833fff700 1 --2- 172.21.15.114:0/2262698559 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083403e470 0x7f0834040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:11.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.182+0000 7f0851542700 1 -- 172.21.15.114:0/2262698559 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f084c064e10 con 0x7f084c108b00 2024-05-27T07:28:11.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.182+0000 7f084a7fc700 1 -- 172.21.15.114:0/2262698559 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083403e470 msgr2=0x7f0834040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1568684596 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.182+0000 7f0833fff700 1 -- 172.21.15.114:0/2262698559 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f083c053400 con 0x7f084c108b00 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.182+0000 7f084a7fc700 1 --2- 172.21.15.114:0/2262698559 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083403e470 0x7f0834040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.187+0000 7f0833fff700 1 -- 172.21.15.114:0/2262698559 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f083c053160 con 0x7f084c108b00 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.383+0000 7f084a7fc700 1 -- 172.21.15.114:0/2262698559 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083403e470 msgr2=0x7f0834040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1568684596 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.383+0000 7f084a7fc700 1 --2- 172.21.15.114:0/2262698559 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083403e470 0x7f0834040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.468+0000 7f0851542700 1 -- 172.21.15.114:0/2262698559 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f084c1a1ae0 con 0x7f084c108b00 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.469+0000 7f0833fff700 1 -- 172.21.15.114:0/2262698559 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f083c020e00 con 0x7f084c108b00 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.472+0000 7f0851542700 1 -- 172.21.15.114:0/2262698559 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083403e470 msgr2=0x7f0834040980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.472+0000 7f0851542700 1 --2- 172.21.15.114:0/2262698559 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083403e470 0x7f0834040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.472+0000 7f0851542700 1 -- 172.21.15.114:0/2262698559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 msgr2=0x7f084c07d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.472+0000 7f0851542700 1 --2- 172.21.15.114:0/2262698559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 0x7f084c07d200 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f083c011890 tx=0x7f083c011970 comp rx=0 tx=0).stop 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.472+0000 7f0851542700 1 -- 172.21.15.114:0/2262698559 shutdown_connections 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.472+0000 7f0851542700 1 --2- 172.21.15.114:0/2262698559 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f083403e470 0x7f0834040980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:11.020 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.472+0000 7f0851542700 1 --2- 172.21.15.114:0/2262698559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f084c108b00 0x7f084c07d200 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:11.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.472+0000 7f0851542700 1 -- 172.21.15.114:0/2262698559 >> 172.21.15.114:0/2262698559 conn(0x7f084c1041c0 msgr2=0x7f084c105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:11.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.472+0000 7f0851542700 1 -- 172.21.15.114:0/2262698559 shutdown_connections 2024-05-27T07:28:11.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:10.473+0000 7f0851542700 1 -- 172.21.15.114:0/2262698559 wait complete. 2024-05-27T07:28:11.021 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-05-27T07:28:11.021 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 5... 2024-05-27T07:28:11.411 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:11 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2262698559' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-27T07:28:11.411 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:11 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:11.291+0000 7f5b7cd2b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:28:11.796 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:11 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:11.474+0000 7f5b7cd2b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:28:11.796 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:11 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:11.643+0000 7f5b7cd2b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:28:12.223 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:11 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:11.936+0000 7f5b7cd2b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:28:12.223 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:12 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:12.031+0000 7f5b7cd2b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:28:12.224 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:12 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:12.233+0000 7f5b7cd2b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:28:12.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:12 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:12.429+0000 7f5b7cd2b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:28:13.241 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:13 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:13.028+0000 7f5b7cd2b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:28:13.241 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:13 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:13.251+0000 7f5b7cd2b200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:28:14.046 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:13 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:13.682+0000 7f5b7cd2b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:28:14.046 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:13 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:13.780+0000 7f5b7cd2b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:28:14.434 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:14.151+0000 7f5b7cd2b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:28:14.796 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:14.507+0000 7f5b7cd2b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:28:14.796 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:14.603+0000 7f5b7cd2b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:28:15.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:14.850+0000 7f5b7cd2b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:28:15.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:14.984+0000 7f5b7cd2b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:28:15.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:15.082+0000 7f5b7cd2b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:28:15.795 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:15.407+0000 7f5b7cd2b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:28:15.796 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:15.502+0000 7f5b7cd2b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:28:16.374 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:16.108+0000 7f5b7cd2b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:28:16.375 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:16.328+0000 7f5b7cd2b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:28:16.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: Active manager daemon x restarted 2024-05-27T07:28:16.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: Activating manager daemon x 2024-05-27T07:28:16.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: osdmap e2: 0 total, 0 up, 0 in 2024-05-27T07:28:16.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: mgrmap e6: x(active, starting, since 0.00413771s) 2024-05-27T07:28:16.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:28:16.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:28:16.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:28:16.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:28:16.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:28:16.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: Manager daemon x is now available 2024-05-27T07:28:16.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' 2024-05-27T07:28:16.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' 2024-05-27T07:28:16.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:16.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:16 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:17.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:17 smithi114 ceph-mon[32038]: Found migration_current of "None". Setting to last migration. 2024-05-27T07:28:17.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:17 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:28:17.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:17 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:28:17.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:17 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' 2024-05-27T07:28:17.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:17 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' 2024-05-27T07:28:17.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:17 smithi114 ceph-mon[32038]: mgrmap e7: x(active, since 1.00904s) 2024-05-27T07:28:17.917 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:28:17.917 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-27T07:28:17.917 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-27T07:28:17.917 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:28:17.917 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.433+0000 7f225b900700 1 Processor -- start 2024-05-27T07:28:17.918 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.433+0000 7f225b900700 1 -- start start 2024-05-27T07:28:17.918 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.433+0000 7f225b900700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 0x7f225410a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:17.918 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.433+0000 7f225b900700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f225410a9f0 con 0x7f225410a010 2024-05-27T07:28:17.918 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.433+0000 7f225969c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 0x7f225410a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:17.918 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.433+0000 7f225969c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 0x7f225410a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55804/0 (socket says 172.21.15.114:55804) 2024-05-27T07:28:17.918 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.433+0000 7f225969c700 1 -- 172.21.15.114:0/3604453875 learned_addr learned my addr 172.21.15.114:0/3604453875 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:17.918 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.434+0000 7f225969c700 1 -- 172.21.15.114:0/3604453875 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f225410ab70 con 0x7f225410a010 2024-05-27T07:28:17.918 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.434+0000 7f225969c700 1 --2- 172.21.15.114:0/3604453875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 0x7f225410a420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f225000bb40 tx=0x7f225000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d258c1fb38ef2d74 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:17.918 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.434+0000 7f224bfff700 1 -- 172.21.15.114:0/3604453875 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f225000dbe0 con 0x7f225410a010 2024-05-27T07:28:17.918 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.434+0000 7f224bfff700 1 -- 172.21.15.114:0/3604453875 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f22500044c0 con 0x7f225410a010 2024-05-27T07:28:17.919 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.434+0000 7f224bfff700 1 -- 172.21.15.114:0/3604453875 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2250010470 con 0x7f225410a010 2024-05-27T07:28:17.919 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.435+0000 7f225b900700 1 -- 172.21.15.114:0/3604453875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 msgr2=0x7f225410a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:17.919 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.435+0000 7f225b900700 1 --2- 172.21.15.114:0/3604453875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 0x7f225410a420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f225000bb40 tx=0x7f225000be50 comp rx=0 tx=0).stop 2024-05-27T07:28:17.919 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.435+0000 7f225b900700 1 -- 172.21.15.114:0/3604453875 shutdown_connections 2024-05-27T07:28:17.919 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.435+0000 7f225b900700 1 --2- 172.21.15.114:0/3604453875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 0x7f225410a420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.919 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.435+0000 7f225b900700 1 -- 172.21.15.114:0/3604453875 >> 172.21.15.114:0/3604453875 conn(0x7f22541038b0 msgr2=0x7f2254105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:17.919 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.435+0000 7f225b900700 1 -- 172.21.15.114:0/3604453875 shutdown_connections 2024-05-27T07:28:17.919 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.435+0000 7f225b900700 1 -- 172.21.15.114:0/3604453875 wait complete. 2024-05-27T07:28:17.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.436+0000 7f225b900700 1 Processor -- start 2024-05-27T07:28:17.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.436+0000 7f225b900700 1 -- start start 2024-05-27T07:28:17.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.436+0000 7f225b900700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 0x7f225419bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:17.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.436+0000 7f225b900700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2254107c20 con 0x7f225410a010 2024-05-27T07:28:17.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.436+0000 7f225969c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 0x7f225419bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:17.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.436+0000 7f225969c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 0x7f225419bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55820/0 (socket says 172.21.15.114:55820) 2024-05-27T07:28:17.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.436+0000 7f225969c700 1 -- 172.21.15.114:0/622115399 learned_addr learned my addr 172.21.15.114:0/622115399 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:17.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.437+0000 7f225969c700 1 -- 172.21.15.114:0/622115399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f225419c270 con 0x7f225410a010 2024-05-27T07:28:17.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.437+0000 7f225969c700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 0x7f225419bd30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f225000bfd0 tx=0x7f2250003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:17.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.437+0000 7f224a7fc700 1 -- 172.21.15.114:0/622115399 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2250027070 con 0x7f225410a010 2024-05-27T07:28:17.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.437+0000 7f224a7fc700 1 -- 172.21.15.114:0/622115399 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2250010c00 con 0x7f225410a010 2024-05-27T07:28:17.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.437+0000 7f225b900700 1 -- 172.21.15.114:0/622115399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f225419c500 con 0x7f225410a010 2024-05-27T07:28:17.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.437+0000 7f224a7fc700 1 -- 172.21.15.114:0/622115399 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2250022c10 con 0x7f225410a010 2024-05-27T07:28:17.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.437+0000 7f225b900700 1 -- 172.21.15.114:0/622115399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f225419f100 con 0x7f225410a010 2024-05-27T07:28:17.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.438+0000 7f224a7fc700 1 -- 172.21.15.114:0/622115399 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f225002a070 con 0x7f225410a010 2024-05-27T07:28:17.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.439+0000 7f224a7fc700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 0x7f2240040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:17.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.439+0000 7f224a7fc700 1 -- 172.21.15.114:0/622115399 --> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2240041040 con 0x7f224003e420 2024-05-27T07:28:17.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.439+0000 7f2258e9b700 1 -- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 msgr2=0x7f2240040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1568684596 2024-05-27T07:28:17.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.439+0000 7f224a7fc700 1 -- 172.21.15.114:0/622115399 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f225001f070 con 0x7f225410a010 2024-05-27T07:28:17.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.439+0000 7f2258e9b700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 0x7f2240040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:28:17.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.640+0000 7f2258e9b700 1 -- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 msgr2=0x7f2240040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1568684596 2024-05-27T07:28:17.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:11.640+0000 7f2258e9b700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 0x7f2240040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:28:17.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:12.041+0000 7f2258e9b700 1 -- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 msgr2=0x7f2240040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1568684596 2024-05-27T07:28:17.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:12.041+0000 7f2258e9b700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 0x7f2240040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:12.842+0000 7f2258e9b700 1 -- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 msgr2=0x7f2240040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1568684596 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:12.842+0000 7f2258e9b700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 0x7f2240040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:14.443+0000 7f2258e9b700 1 -- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 msgr2=0x7f2240040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1568684596 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:14.443+0000 7f2258e9b700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 0x7f2240040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:16.333+0000 7f224a7fc700 1 -- 172.21.15.114:0/622115399 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mgrmap(e 6) ==== 49989+0+0 (secure 0 0 0) 0x7f225002cce0 con 0x7f225410a010 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:16.333+0000 7f224a7fc700 1 -- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 msgr2=0x7f2240040930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:16.333+0000 7f224a7fc700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 0x7f2240040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.336+0000 7f224a7fc700 1 -- 172.21.15.114:0/622115399 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7f2250011d10 con 0x7f225410a010 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.336+0000 7f224a7fc700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f2240041d60 0x7f2240044180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.336+0000 7f224a7fc700 1 -- 172.21.15.114:0/622115399 --> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2250020ec0 con 0x7f2240041d60 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.337+0000 7f2258e9b700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f2240041d60 0x7f2240044180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.338+0000 7f2258e9b700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f2240041d60 0x7f2240044180 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2244003a10 tx=0x7f2244009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.338+0000 7f224a7fc700 1 -- 172.21.15.114:0/622115399 <== mgr.14120 v2:172.21.15.114:6800/752776783 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f2250020ec0 con 0x7f2240041d60 2024-05-27T07:28:17.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.347+0000 7f225b900700 1 -- 172.21.15.114:0/622115399 --> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f2254064e10 con 0x7f2240041d60 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.347+0000 7f224a7fc700 1 -- 172.21.15.114:0/622115399 <== mgr.14120 v2:172.21.15.114:6800/752776783 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f2254064e10 con 0x7f2240041d60 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.347+0000 7f225b900700 1 -- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f2240041d60 msgr2=0x7f2240044180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.347+0000 7f225b900700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f2240041d60 0x7f2240044180 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2244003a10 tx=0x7f2244009290 comp rx=0 tx=0).stop 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.347+0000 7f225b900700 1 -- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 msgr2=0x7f225419bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.347+0000 7f225b900700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 0x7f225419bd30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f225000bfd0 tx=0x7f2250003ff0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.347+0000 7f225b900700 1 -- 172.21.15.114:0/622115399 shutdown_connections 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.347+0000 7f225b900700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f2240041d60 0x7f2240044180 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.347+0000 7f225b900700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225410a010 0x7f225419bd30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.347+0000 7f225b900700 1 --2- 172.21.15.114:0/622115399 >> [v2:172.21.15.114:6800/1568684596,v1:172.21.15.114:6801/1568684596] conn(0x7f224003e420 0x7f2240040930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.347+0000 7f225b900700 1 -- 172.21.15.114:0/622115399 >> 172.21.15.114:0/622115399 conn(0x7f22541038b0 msgr2=0x7f2254192000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.347+0000 7f225b900700 1 -- 172.21.15.114:0/622115399 shutdown_connections 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:17.347+0000 7f225b900700 1 -- 172.21.15.114:0/622115399 wait complete. 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 5 is available 2024-05-27T07:28:17.923 INFO:teuthology.orchestra.run.smithi114.stdout:Setting orchestrator backend to cephadm... 2024-05-27T07:28:18.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:18 smithi114 ceph-mon[32038]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-27T07:28:18.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:18 smithi114 ceph-mon[32038]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-27T07:28:18.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:18 smithi114 ceph-mon[32038]: [27/May/2024:07:28:17] ENGINE Bus STARTING 2024-05-27T07:28:18.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:18 smithi114 ceph-mon[32038]: [27/May/2024:07:28:17] ENGINE Serving on http://172.21.15.114:8765 2024-05-27T07:28:18.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:18 smithi114 ceph-mon[32038]: [27/May/2024:07:28:17] ENGINE Serving on https://172.21.15.114:7150 2024-05-27T07:28:18.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:18 smithi114 ceph-mon[32038]: [27/May/2024:07:28:17] ENGINE Bus STARTED 2024-05-27T07:28:18.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:18 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:19.184 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.374+0000 7f6dfa15d700 1 Processor -- start 2024-05-27T07:28:19.185 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.374+0000 7f6dfa15d700 1 -- start start 2024-05-27T07:28:19.185 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.375+0000 7f6dfa15d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 0x7f6df410c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:19.185 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.375+0000 7f6dfa15d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6df410cd40 con 0x7f6df407cc50 2024-05-27T07:28:19.185 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.375+0000 7f6df37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 0x7f6df410c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:19.185 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.375+0000 7f6df37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 0x7f6df410c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41070/0 (socket says 172.21.15.114:41070) 2024-05-27T07:28:19.186 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.375+0000 7f6df37fe700 1 -- 172.21.15.114:0/3321232606 learned_addr learned my addr 172.21.15.114:0/3321232606 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:19.186 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.375+0000 7f6df37fe700 1 -- 172.21.15.114:0/3321232606 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df410cec0 con 0x7f6df407cc50 2024-05-27T07:28:19.186 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.375+0000 7f6df37fe700 1 --2- 172.21.15.114:0/3321232606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 0x7f6df410c770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6de40098a0 tx=0x7f6de4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe67de7330df3efb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:19.186 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.375+0000 7f6df27fc700 1 -- 172.21.15.114:0/3321232606 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6de4004160 con 0x7f6df407cc50 2024-05-27T07:28:19.186 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.375+0000 7f6df27fc700 1 -- 172.21.15.114:0/3321232606 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6de4004300 con 0x7f6df407cc50 2024-05-27T07:28:19.186 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.375+0000 7f6df27fc700 1 -- 172.21.15.114:0/3321232606 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6de4011af0 con 0x7f6df407cc50 2024-05-27T07:28:19.187 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.376+0000 7f6dfa15d700 1 -- 172.21.15.114:0/3321232606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 msgr2=0x7f6df410c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:19.187 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.376+0000 7f6dfa15d700 1 --2- 172.21.15.114:0/3321232606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 0x7f6df410c770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6de40098a0 tx=0x7f6de4009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.187 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.376+0000 7f6dfa15d700 1 -- 172.21.15.114:0/3321232606 shutdown_connections 2024-05-27T07:28:19.187 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.376+0000 7f6dfa15d700 1 --2- 172.21.15.114:0/3321232606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 0x7f6df410c770 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.188 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.376+0000 7f6dfa15d700 1 -- 172.21.15.114:0/3321232606 >> 172.21.15.114:0/3321232606 conn(0x7f6df4104120 msgr2=0x7f6df4106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:19.188 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.376+0000 7f6dfa15d700 1 -- 172.21.15.114:0/3321232606 shutdown_connections 2024-05-27T07:28:19.188 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.376+0000 7f6dfa15d700 1 -- 172.21.15.114:0/3321232606 wait complete. 2024-05-27T07:28:19.188 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.377+0000 7f6dfa15d700 1 Processor -- start 2024-05-27T07:28:19.188 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.377+0000 7f6dfa15d700 1 -- start start 2024-05-27T07:28:19.188 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.377+0000 7f6dfa15d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 0x7f6df419dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:19.189 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.377+0000 7f6dfa15d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6df410dee0 con 0x7f6df407cc50 2024-05-27T07:28:19.189 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.378+0000 7f6df37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 0x7f6df419dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:19.189 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.378+0000 7f6df37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 0x7f6df419dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41072/0 (socket says 172.21.15.114:41072) 2024-05-27T07:28:19.189 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.378+0000 7f6df37fe700 1 -- 172.21.15.114:0/63611262 learned_addr learned my addr 172.21.15.114:0/63611262 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:19.189 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.378+0000 7f6df37fe700 1 -- 172.21.15.114:0/63611262 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df419e500 con 0x7f6df407cc50 2024-05-27T07:28:19.189 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.378+0000 7f6df37fe700 1 --2- 172.21.15.114:0/63611262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 0x7f6df419dfc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009fb0 tx=0x7f6de4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:19.190 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.378+0000 7f6df0ff9700 1 -- 172.21.15.114:0/63611262 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6de4011bc0 con 0x7f6df407cc50 2024-05-27T07:28:19.190 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.378+0000 7f6df0ff9700 1 -- 172.21.15.114:0/63611262 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6de4024440 con 0x7f6df407cc50 2024-05-27T07:28:19.190 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.378+0000 7f6dfa15d700 1 -- 172.21.15.114:0/63611262 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df419e790 con 0x7f6df407cc50 2024-05-27T07:28:19.190 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.378+0000 7f6df0ff9700 1 -- 172.21.15.114:0/63611262 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6de401a5c0 con 0x7f6df407cc50 2024-05-27T07:28:19.190 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.378+0000 7f6dfa15d700 1 -- 172.21.15.114:0/63611262 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df41a13a0 con 0x7f6df407cc50 2024-05-27T07:28:19.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.379+0000 7f6df0ff9700 1 -- 172.21.15.114:0/63611262 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7f6de401a880 con 0x7f6df407cc50 2024-05-27T07:28:19.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.379+0000 7f6df0ff9700 1 --2- 172.21.15.114:0/63611262 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f6de003e310 0x7f6de0040820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:19.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.379+0000 7f6dfa15d700 1 -- 172.21.15.114:0/63611262 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6df4064e10 con 0x7f6df407cc50 2024-05-27T07:28:19.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.379+0000 7f6df0ff9700 1 -- 172.21.15.114:0/63611262 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6de40549f0 con 0x7f6df407cc50 2024-05-27T07:28:19.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.379+0000 7f6df2ffd700 1 --2- 172.21.15.114:0/63611262 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f6de003e310 0x7f6de0040820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:19.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.380+0000 7f6df2ffd700 1 --2- 172.21.15.114:0/63611262 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f6de003e310 0x7f6de0040820 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc006fd0 tx=0x7f6ddc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:19.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.385+0000 7f6df0ff9700 1 -- 172.21.15.114:0/63611262 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6de400b3e0 con 0x7f6df407cc50 2024-05-27T07:28:19.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.392+0000 7f6df0ff9700 1 -- 172.21.15.114:0/63611262 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f6de40245f0 con 0x7f6df407cc50 2024-05-27T07:28:19.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.604+0000 7f6dfa15d700 1 -- 172.21.15.114:0/63611262 --> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f6df41052c0 con 0x7f6de003e310 2024-05-27T07:28:19.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.610+0000 7f6df0ff9700 1 -- 172.21.15.114:0/63611262 <== mgr.14120 v2:172.21.15.114:6800/752776783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6df41052c0 con 0x7f6de003e310 2024-05-27T07:28:19.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.612+0000 7f6dfa15d700 1 -- 172.21.15.114:0/63611262 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f6de003e310 msgr2=0x7f6de0040820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:19.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.612+0000 7f6dfa15d700 1 --2- 172.21.15.114:0/63611262 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f6de003e310 0x7f6de0040820 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc006fd0 tx=0x7f6ddc006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:19.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.612+0000 7f6dfa15d700 1 -- 172.21.15.114:0/63611262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 msgr2=0x7f6df419dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:19.194 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.612+0000 7f6dfa15d700 1 --2- 172.21.15.114:0/63611262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 0x7f6df419dfc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009fb0 tx=0x7f6de4011890 comp rx=0 tx=0).stop 2024-05-27T07:28:19.194 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.612+0000 7f6dfa15d700 1 -- 172.21.15.114:0/63611262 shutdown_connections 2024-05-27T07:28:19.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.612+0000 7f6dfa15d700 1 --2- 172.21.15.114:0/63611262 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f6de003e310 0x7f6de0040820 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.612+0000 7f6dfa15d700 1 --2- 172.21.15.114:0/63611262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df407cc50 0x7f6df419dfc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.612+0000 7f6dfa15d700 1 -- 172.21.15.114:0/63611262 >> 172.21.15.114:0/63611262 conn(0x7f6df4104120 msgr2=0x7f6df4104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:19.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.613+0000 7f6dfa15d700 1 -- 172.21.15.114:0/63611262 shutdown_connections 2024-05-27T07:28:19.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:18.613+0000 7f6dfa15d700 1 -- 172.21.15.114:0/63611262 wait complete. 2024-05-27T07:28:19.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:19 smithi114 ceph-mon[32038]: mgrmap e8: x(active, since 2s) 2024-05-27T07:28:19.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:19 smithi114 ceph-mon[32038]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:28:19.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:19 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' 2024-05-27T07:28:19.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:19 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:20.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:20 smithi114 ceph-mon[32038]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:28:20.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-27T07:28:20.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.651+0000 7f41f32dc700 1 Processor -- start 2024-05-27T07:28:20.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.651+0000 7f41f32dc700 1 -- start start 2024-05-27T07:28:20.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.651+0000 7f41f32dc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 0x7f41ec07db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.651+0000 7f41f32dc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec07e070 con 0x7f41ec07f660 2024-05-27T07:28:20.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.652+0000 7f41f1078700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 0x7f41ec07db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.652+0000 7f41f1078700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 0x7f41ec07db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41082/0 (socket says 172.21.15.114:41082) 2024-05-27T07:28:20.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.652+0000 7f41f1078700 1 -- 172.21.15.114:0/1256793671 learned_addr learned my addr 172.21.15.114:0/1256793671 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:20.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.652+0000 7f41f1078700 1 -- 172.21.15.114:0/1256793671 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec07e1f0 con 0x7f41ec07f660 2024-05-27T07:28:20.495 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.652+0000 7f41f1078700 1 --2- 172.21.15.114:0/1256793671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 0x7f41ec07db30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f41e80098a0 tx=0x7f41e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca61f76b21f0a603 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:20.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.652+0000 7f41e3fff700 1 -- 172.21.15.114:0/1256793671 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41e8004160 con 0x7f41ec07f660 2024-05-27T07:28:20.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.652+0000 7f41e3fff700 1 -- 172.21.15.114:0/1256793671 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f41e8004300 con 0x7f41ec07f660 2024-05-27T07:28:20.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.652+0000 7f41e3fff700 1 -- 172.21.15.114:0/1256793671 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41e8011af0 con 0x7f41ec07f660 2024-05-27T07:28:20.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.653+0000 7f41f32dc700 1 -- 172.21.15.114:0/1256793671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 msgr2=0x7f41ec07db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:20.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.653+0000 7f41f32dc700 1 --2- 172.21.15.114:0/1256793671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 0x7f41ec07db30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f41e80098a0 tx=0x7f41e8009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.653+0000 7f41f32dc700 1 -- 172.21.15.114:0/1256793671 shutdown_connections 2024-05-27T07:28:20.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.653+0000 7f41f32dc700 1 --2- 172.21.15.114:0/1256793671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 0x7f41ec07db30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.653+0000 7f41f32dc700 1 -- 172.21.15.114:0/1256793671 >> 172.21.15.114:0/1256793671 conn(0x7f41ec1052b0 msgr2=0x7f41ec107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:20.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.653+0000 7f41f32dc700 1 -- 172.21.15.114:0/1256793671 shutdown_connections 2024-05-27T07:28:20.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.653+0000 7f41f32dc700 1 -- 172.21.15.114:0/1256793671 wait complete. 2024-05-27T07:28:20.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.654+0000 7f41f32dc700 1 Processor -- start 2024-05-27T07:28:20.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.654+0000 7f41f32dc700 1 -- start start 2024-05-27T07:28:20.496 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.654+0000 7f41f32dc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 0x7f41ec1a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.654+0000 7f41f32dc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec110170 con 0x7f41ec07f660 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.654+0000 7f41f1078700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 0x7f41ec1a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.654+0000 7f41f1078700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 0x7f41ec1a01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41094/0 (socket says 172.21.15.114:41094) 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.654+0000 7f41f1078700 1 -- 172.21.15.114:0/2571446197 learned_addr learned my addr 172.21.15.114:0/2571446197 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.654+0000 7f41f1078700 1 -- 172.21.15.114:0/2571446197 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec1a0720 con 0x7f41ec07f660 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.654+0000 7f41f1078700 1 --2- 172.21.15.114:0/2571446197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 0x7f41ec1a01e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009fb0 tx=0x7f41e8011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.655+0000 7f41e27fc700 1 -- 172.21.15.114:0/2571446197 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41e80036a0 con 0x7f41ec07f660 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.655+0000 7f41e27fc700 1 -- 172.21.15.114:0/2571446197 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f41e801a430 con 0x7f41ec07f660 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.655+0000 7f41f32dc700 1 -- 172.21.15.114:0/2571446197 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ec1a09b0 con 0x7f41ec07f660 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.655+0000 7f41e27fc700 1 -- 172.21.15.114:0/2571446197 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41e8019550 con 0x7f41ec07f660 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.655+0000 7f41f32dc700 1 -- 172.21.15.114:0/2571446197 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ec1a35c0 con 0x7f41ec07f660 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.656+0000 7f41e27fc700 1 -- 172.21.15.114:0/2571446197 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f41e8019800 con 0x7f41ec07f660 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.656+0000 7f41f32dc700 1 -- 172.21.15.114:0/2571446197 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41ec064e10 con 0x7f41ec07f660 2024-05-27T07:28:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.656+0000 7f41e27fc700 1 --2- 172.21.15.114:0/2571446197 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f41d803e470 0x7f41d8040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.656+0000 7f41e27fc700 1 -- 172.21.15.114:0/2571446197 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f41e8053990 con 0x7f41ec07f660 2024-05-27T07:28:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.656+0000 7f41f0877700 1 --2- 172.21.15.114:0/2571446197 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f41d803e470 0x7f41d8040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.656+0000 7f41f0877700 1 --2- 172.21.15.114:0/2571446197 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f41d803e470 0x7f41d8040980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f41dc006fd0 tx=0x7f41dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.661+0000 7f41e27fc700 1 -- 172.21.15.114:0/2571446197 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f41e8023390 con 0x7f41ec07f660 2024-05-27T07:28:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.866+0000 7f41f32dc700 1 -- 172.21.15.114:0/2571446197 --> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f41ec1062f0 con 0x7f41d803e470 2024-05-27T07:28:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.867+0000 7f41e27fc700 1 -- 172.21.15.114:0/2571446197 <== mgr.14120 v2:172.21.15.114:6800/752776783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f41ec1062f0 con 0x7f41d803e470 2024-05-27T07:28:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.870+0000 7f41f32dc700 1 -- 172.21.15.114:0/2571446197 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f41d803e470 msgr2=0x7f41d8040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.870+0000 7f41f32dc700 1 --2- 172.21.15.114:0/2571446197 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f41d803e470 0x7f41d8040980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f41dc006fd0 tx=0x7f41dc006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.870+0000 7f41f32dc700 1 -- 172.21.15.114:0/2571446197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 msgr2=0x7f41ec1a01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.870+0000 7f41f32dc700 1 --2- 172.21.15.114:0/2571446197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 0x7f41ec1a01e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009fb0 tx=0x7f41e8011940 comp rx=0 tx=0).stop 2024-05-27T07:28:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.870+0000 7f41f32dc700 1 -- 172.21.15.114:0/2571446197 shutdown_connections 2024-05-27T07:28:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.870+0000 7f41f32dc700 1 --2- 172.21.15.114:0/2571446197 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f41d803e470 0x7f41d8040980 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.870+0000 7f41f32dc700 1 --2- 172.21.15.114:0/2571446197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec07f660 0x7f41ec1a01e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.870+0000 7f41f32dc700 1 -- 172.21.15.114:0/2571446197 >> 172.21.15.114:0/2571446197 conn(0x7f41ec1052b0 msgr2=0x7f41ec105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:20.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.870+0000 7f41f32dc700 1 -- 172.21.15.114:0/2571446197 shutdown_connections 2024-05-27T07:28:20.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:19.870+0000 7f41f32dc700 1 -- 172.21.15.114:0/2571446197 wait complete. 2024-05-27T07:28:20.499 INFO:teuthology.orchestra.run.smithi114.stdout:Generating ssh key... 2024-05-27T07:28:21.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: Generating public/private rsa key pair. 2024-05-27T07:28:21.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: Your identification has been saved in /tmp/tmpnppzxqvn/key. 2024-05-27T07:28:21.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: Your public key has been saved in /tmp/tmpnppzxqvn/key.pub. 2024-05-27T07:28:21.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: The key fingerprint is: 2024-05-27T07:28:21.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: SHA256:zBhpDLMcToXJwfVA1bL8mK475X8jMpZ0gbt8eNkV6Rs ceph-755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:28:21.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: The key's randomart image is: 2024-05-27T07:28:21.547 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: +---[RSA 3072]----+ 2024-05-27T07:28:21.547 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: | o*B=... | 2024-05-27T07:28:21.547 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: | +=* +. . | 2024-05-27T07:28:21.547 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: | + =..+ . | 2024-05-27T07:28:21.547 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: | . =+ . o | 2024-05-27T07:28:21.547 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: | . S= . . . | 2024-05-27T07:28:21.547 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: | * o E | 2024-05-27T07:28:21.547 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: | * = o . o | 2024-05-27T07:28:21.547 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: | . @ = + . | 2024-05-27T07:28:21.547 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: | o= *.o . | 2024-05-27T07:28:21.547 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: +----[SHA256]-----+ 2024-05-27T07:28:21.842 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.914+0000 7ff5bff8e700 1 Processor -- start 2024-05-27T07:28:21.842 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.914+0000 7ff5bff8e700 1 -- start start 2024-05-27T07:28:21.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.914+0000 7ff5bff8e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 0x7ff5b810cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:21.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.914+0000 7ff5bff8e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b807d690 con 0x7ff5b810a6e0 2024-05-27T07:28:21.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.914+0000 7ff5bdd2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 0x7ff5b810cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:21.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.914+0000 7ff5bdd2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 0x7ff5b810cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41100/0 (socket says 172.21.15.114:41100) 2024-05-27T07:28:21.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.914+0000 7ff5bdd2a700 1 -- 172.21.15.114:0/1474993770 learned_addr learned my addr 172.21.15.114:0/1474993770 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:21.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.915+0000 7ff5bdd2a700 1 -- 172.21.15.114:0/1474993770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b810d040 con 0x7ff5b810a6e0 2024-05-27T07:28:21.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.915+0000 7ff5bdd2a700 1 --2- 172.21.15.114:0/1474993770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 0x7ff5b810cb00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff5b40098a0 tx=0x7ff5b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6994803f028f03c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:21.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.915+0000 7ff5bcd28700 1 -- 172.21.15.114:0/1474993770 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5b4004160 con 0x7ff5b810a6e0 2024-05-27T07:28:21.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.915+0000 7ff5bcd28700 1 -- 172.21.15.114:0/1474993770 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff5b4004300 con 0x7ff5b810a6e0 2024-05-27T07:28:21.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.915+0000 7ff5bcd28700 1 -- 172.21.15.114:0/1474993770 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5b4011af0 con 0x7ff5b810a6e0 2024-05-27T07:28:21.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.916+0000 7ff5bff8e700 1 -- 172.21.15.114:0/1474993770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 msgr2=0x7ff5b810cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:21.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.916+0000 7ff5bff8e700 1 --2- 172.21.15.114:0/1474993770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 0x7ff5b810cb00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff5b40098a0 tx=0x7ff5b4009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.916+0000 7ff5bff8e700 1 -- 172.21.15.114:0/1474993770 shutdown_connections 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.916+0000 7ff5bff8e700 1 --2- 172.21.15.114:0/1474993770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 0x7ff5b810cb00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.916+0000 7ff5bff8e700 1 -- 172.21.15.114:0/1474993770 >> 172.21.15.114:0/1474993770 conn(0x7ff5b81041c0 msgr2=0x7ff5b8106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.916+0000 7ff5bff8e700 1 -- 172.21.15.114:0/1474993770 shutdown_connections 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.916+0000 7ff5bff8e700 1 -- 172.21.15.114:0/1474993770 wait complete. 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.917+0000 7ff5bff8e700 1 Processor -- start 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.917+0000 7ff5bff8e700 1 -- start start 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.917+0000 7ff5bff8e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 0x7ff5b819df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.917+0000 7ff5bff8e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b810df40 con 0x7ff5b810a6e0 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.917+0000 7ff5bdd2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 0x7ff5b819df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.917+0000 7ff5bdd2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 0x7ff5b819df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41108/0 (socket says 172.21.15.114:41108) 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.917+0000 7ff5bdd2a700 1 -- 172.21.15.114:0/655639847 learned_addr learned my addr 172.21.15.114:0/655639847 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.917+0000 7ff5bdd2a700 1 -- 172.21.15.114:0/655639847 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b819e4a0 con 0x7ff5b810a6e0 2024-05-27T07:28:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.917+0000 7ff5bdd2a700 1 --2- 172.21.15.114:0/655639847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 0x7ff5b819df60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009fb0 tx=0x7ff5b4011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.917+0000 7ff5aeffd700 1 -- 172.21.15.114:0/655639847 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5b40036a0 con 0x7ff5b810a6e0 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.917+0000 7ff5aeffd700 1 -- 172.21.15.114:0/655639847 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff5b401a430 con 0x7ff5b810a6e0 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.917+0000 7ff5bff8e700 1 -- 172.21.15.114:0/655639847 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b819e730 con 0x7ff5b810a6e0 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.917+0000 7ff5aeffd700 1 -- 172.21.15.114:0/655639847 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5b4019550 con 0x7ff5b810a6e0 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.918+0000 7ff5bff8e700 1 -- 172.21.15.114:0/655639847 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b81a1340 con 0x7ff5b810a6e0 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.918+0000 7ff5bff8e700 1 -- 172.21.15.114:0/655639847 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5b8064e10 con 0x7ff5b810a6e0 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.918+0000 7ff5aeffd700 1 -- 172.21.15.114:0/655639847 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7ff5b4019800 con 0x7ff5b810a6e0 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.919+0000 7ff5aeffd700 1 --2- 172.21.15.114:0/655639847 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7ff5a403e470 0x7ff5a4040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.919+0000 7ff5aeffd700 1 -- 172.21.15.114:0/655639847 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff5b4053820 con 0x7ff5b810a6e0 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.919+0000 7ff5bd529700 1 --2- 172.21.15.114:0/655639847 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7ff5a403e470 0x7ff5a4040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.919+0000 7ff5bd529700 1 --2- 172.21.15.114:0/655639847 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7ff5a403e470 0x7ff5a4040980 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8006fd0 tx=0x7ff5a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:20.924+0000 7ff5aeffd700 1 -- 172.21.15.114:0/655639847 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff5b4017e90 con 0x7ff5b810a6e0 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:21.125+0000 7ff5bff8e700 1 -- 172.21.15.114:0/655639847 --> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7ff5b8106210 con 0x7ff5a403e470 2024-05-27T07:28:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:21.206+0000 7ff5aeffd700 1 -- 172.21.15.114:0/655639847 <== mgr.14120 v2:172.21.15.114:6800/752776783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff5b8106210 con 0x7ff5a403e470 2024-05-27T07:28:21.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:21.208+0000 7ff5bff8e700 1 -- 172.21.15.114:0/655639847 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7ff5a403e470 msgr2=0x7ff5a4040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:21.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:21.208+0000 7ff5bff8e700 1 --2- 172.21.15.114:0/655639847 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7ff5a403e470 0x7ff5a4040980 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8006fd0 tx=0x7ff5a8006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:21.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:21.208+0000 7ff5bff8e700 1 -- 172.21.15.114:0/655639847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 msgr2=0x7ff5b819df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:21.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:21.208+0000 7ff5bff8e700 1 --2- 172.21.15.114:0/655639847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 0x7ff5b819df60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009fb0 tx=0x7ff5b4011940 comp rx=0 tx=0).stop 2024-05-27T07:28:21.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:21.208+0000 7ff5bff8e700 1 -- 172.21.15.114:0/655639847 shutdown_connections 2024-05-27T07:28:21.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:21.208+0000 7ff5bff8e700 1 --2- 172.21.15.114:0/655639847 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7ff5a403e470 0x7ff5a4040980 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:21.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:21.208+0000 7ff5bff8e700 1 --2- 172.21.15.114:0/655639847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b810a6e0 0x7ff5b819df60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:21.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:21.208+0000 7ff5bff8e700 1 -- 172.21.15.114:0/655639847 >> 172.21.15.114:0/655639847 conn(0x7ff5b81041c0 msgr2=0x7ff5b8105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:21.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:21.208+0000 7ff5bff8e700 1 -- 172.21.15.114:0/655639847 shutdown_connections 2024-05-27T07:28:21.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:21.208+0000 7ff5bff8e700 1 -- 172.21.15.114:0/655639847 wait complete. 2024-05-27T07:28:22.230 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:22 smithi114 ceph-mon[32038]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:28:22.230 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:22 smithi114 ceph-mon[32038]: Generating ssh key... 2024-05-27T07:28:22.230 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:22 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' 2024-05-27T07:28:22.230 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:22 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' 2024-05-27T07:28:23.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEgQXBtxItCMdeMK1vaJ2BUmw/u/ATCmqfGUAGvHmapPUrvO2dMkcnps7orIoYJeQpi0DsMfAlnImqOexhZ/WUgw3PXz2vLg22XQAcA4jhdv9LnqsrBhPpWZR2hpr1VOv52BFJ1yuBfFyNK/yE32KgCA48v1NJpABq5EmlE6i3xH4bdqooCMDa/1j+5bU3GxtGK0lN4g+kJeAuX7dY9P5gpw4CPFZC7jVedo4rtzmgTWk7lbaYyY6fZVebxJKFMocDG6x/zlRHLXNhb+EjTITiiBdoZF7m8O4I1eeZk4w0cP611957wJuXSDjEn+H5Obwxgv0NukMXGelBHHjuOB4/V3PjaLjZkpGARbm5BjIpx1epxPDpwN6/ilE3txD/S6QFXaVoFew9a0+k7hO5F3EM1u2EQrAy5r9Up/+/jsj+VHZhGehySKIsUMmFqy2p8ARqvlROZ8LwvguXHmDEngkrx4RXNQniWUnlEvsuRMT2J/tkjp8Pu0uJc95zmoTB8Tk= ceph-755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:28:23.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.281+0000 7ff09d1d2700 1 Processor -- start 2024-05-27T07:28:23.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.281+0000 7ff09d1d2700 1 -- start start 2024-05-27T07:28:23.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.281+0000 7ff09d1d2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 0x7ff09810c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.281+0000 7ff09d1d2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff09810cd80 con 0x7ff09807d500 2024-05-27T07:28:23.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.282+0000 7ff096d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 0x7ff09810c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.282+0000 7ff096d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 0x7ff09810c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41122/0 (socket says 172.21.15.114:41122) 2024-05-27T07:28:23.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.282+0000 7ff096d9d700 1 -- 172.21.15.114:0/1226974094 learned_addr learned my addr 172.21.15.114:0/1226974094 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:23.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.282+0000 7ff096d9d700 1 -- 172.21.15.114:0/1226974094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09810cf00 con 0x7ff09807d500 2024-05-27T07:28:23.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.282+0000 7ff096d9d700 1 --2- 172.21.15.114:0/1226974094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 0x7ff09810c7b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff0880098a0 tx=0x7ff088009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af781e81b057d71c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:23.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.282+0000 7ff09659c700 1 -- 172.21.15.114:0/1226974094 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff088004160 con 0x7ff09807d500 2024-05-27T07:28:23.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.282+0000 7ff09659c700 1 -- 172.21.15.114:0/1226974094 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff088004300 con 0x7ff09807d500 2024-05-27T07:28:23.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.282+0000 7ff09659c700 1 -- 172.21.15.114:0/1226974094 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff088011af0 con 0x7ff09807d500 2024-05-27T07:28:23.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.283+0000 7ff09d1d2700 1 -- 172.21.15.114:0/1226974094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 msgr2=0x7ff09810c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:23.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.283+0000 7ff09d1d2700 1 --2- 172.21.15.114:0/1226974094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 0x7ff09810c7b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff0880098a0 tx=0x7ff088009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.283+0000 7ff09d1d2700 1 -- 172.21.15.114:0/1226974094 shutdown_connections 2024-05-27T07:28:23.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.283+0000 7ff09d1d2700 1 --2- 172.21.15.114:0/1226974094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 0x7ff09810c7b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.283+0000 7ff09d1d2700 1 -- 172.21.15.114:0/1226974094 >> 172.21.15.114:0/1226974094 conn(0x7ff0981041c0 msgr2=0x7ff098106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:23.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.283+0000 7ff09d1d2700 1 -- 172.21.15.114:0/1226974094 shutdown_connections 2024-05-27T07:28:23.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.283+0000 7ff09d1d2700 1 -- 172.21.15.114:0/1226974094 wait complete. 2024-05-27T07:28:23.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.284+0000 7ff09d1d2700 1 Processor -- start 2024-05-27T07:28:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.284+0000 7ff09d1d2700 1 -- start start 2024-05-27T07:28:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.284+0000 7ff09d1d2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 0x7ff09819df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.284+0000 7ff09d1d2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff09810df80 con 0x7ff09807d500 2024-05-27T07:28:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.284+0000 7ff096d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 0x7ff09819df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.284+0000 7ff096d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 0x7ff09819df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41132/0 (socket says 172.21.15.114:41132) 2024-05-27T07:28:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.284+0000 7ff096d9d700 1 -- 172.21.15.114:0/2765420249 learned_addr learned my addr 172.21.15.114:0/2765420249 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.285+0000 7ff096d9d700 1 -- 172.21.15.114:0/2765420249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09819e4a0 con 0x7ff09807d500 2024-05-27T07:28:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.285+0000 7ff096d9d700 1 --2- 172.21.15.114:0/2765420249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 0x7ff09819df60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff088009fb0 tx=0x7ff088011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.285+0000 7ff094d99700 1 -- 172.21.15.114:0/2765420249 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0880036a0 con 0x7ff09807d500 2024-05-27T07:28:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.285+0000 7ff094d99700 1 -- 172.21.15.114:0/2765420249 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff08801a430 con 0x7ff09807d500 2024-05-27T07:28:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.285+0000 7ff09d1d2700 1 -- 172.21.15.114:0/2765420249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff09819e730 con 0x7ff09807d500 2024-05-27T07:28:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.285+0000 7ff094d99700 1 -- 172.21.15.114:0/2765420249 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff088019550 con 0x7ff09807d500 2024-05-27T07:28:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.285+0000 7ff09d1d2700 1 -- 172.21.15.114:0/2765420249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0981a1340 con 0x7ff09807d500 2024-05-27T07:28:23.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.286+0000 7ff094d99700 1 -- 172.21.15.114:0/2765420249 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7ff088019850 con 0x7ff09807d500 2024-05-27T07:28:23.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.286+0000 7ff09d1d2700 1 -- 172.21.15.114:0/2765420249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff098064e10 con 0x7ff09807d500 2024-05-27T07:28:23.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.286+0000 7ff094d99700 1 --2- 172.21.15.114:0/2765420249 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7ff08403e420 0x7ff084040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.286+0000 7ff094d99700 1 -- 172.21.15.114:0/2765420249 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff0880538a0 con 0x7ff09807d500 2024-05-27T07:28:23.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.286+0000 7ff08ffff700 1 --2- 172.21.15.114:0/2765420249 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7ff08403e420 0x7ff084040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.287+0000 7ff08ffff700 1 --2- 172.21.15.114:0/2765420249 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7ff08403e420 0x7ff084040930 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff080006fd0 tx=0x7ff080006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:23.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.292+0000 7ff094d99700 1 -- 172.21.15.114:0/2765420249 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff088017e90 con 0x7ff09807d500 2024-05-27T07:28:23.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.498+0000 7ff09d1d2700 1 -- 172.21.15.114:0/2765420249 --> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7ff098106210 con 0x7ff08403e420 2024-05-27T07:28:23.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.499+0000 7ff094d99700 1 -- 172.21.15.114:0/2765420249 <== mgr.14120 v2:172.21.15.114:6800/752776783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7ff098106210 con 0x7ff08403e420 2024-05-27T07:28:23.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.501+0000 7ff09d1d2700 1 -- 172.21.15.114:0/2765420249 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7ff08403e420 msgr2=0x7ff084040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:23.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.501+0000 7ff09d1d2700 1 --2- 172.21.15.114:0/2765420249 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7ff08403e420 0x7ff084040930 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff080006fd0 tx=0x7ff080006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:23.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.501+0000 7ff09d1d2700 1 -- 172.21.15.114:0/2765420249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 msgr2=0x7ff09819df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:23.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.501+0000 7ff09d1d2700 1 --2- 172.21.15.114:0/2765420249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 0x7ff09819df60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff088009fb0 tx=0x7ff088011940 comp rx=0 tx=0).stop 2024-05-27T07:28:23.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.501+0000 7ff09d1d2700 1 -- 172.21.15.114:0/2765420249 shutdown_connections 2024-05-27T07:28:23.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.501+0000 7ff09d1d2700 1 --2- 172.21.15.114:0/2765420249 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7ff08403e420 0x7ff084040930 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.501+0000 7ff09d1d2700 1 --2- 172.21.15.114:0/2765420249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09807d500 0x7ff09819df60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.501+0000 7ff09d1d2700 1 -- 172.21.15.114:0/2765420249 >> 172.21.15.114:0/2765420249 conn(0x7ff0981041c0 msgr2=0x7ff098105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:23.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.502+0000 7ff09d1d2700 1 -- 172.21.15.114:0/2765420249 shutdown_connections 2024-05-27T07:28:23.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:22.502+0000 7ff09d1d2700 1 -- 172.21.15.114:0/2765420249 wait complete. 2024-05-27T07:28:23.095 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-27T07:28:23.095 INFO:teuthology.orchestra.run.smithi114.stdout:Adding key to root@localhost authorized_keys... 2024-05-27T07:28:23.095 INFO:teuthology.orchestra.run.smithi114.stdout:Adding host smithi114... 2024-05-27T07:28:23.478 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:23 smithi114 ceph-mon[32038]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:28:24.296 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:24 smithi114 ceph-mon[32038]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "addr": "172.21.15.114", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:28:25.545 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:25 smithi114 ceph-mon[32038]: Deploying cephadm binary to smithi114 2024-05-27T07:28:28.390 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Added host 'smithi114' with addr '172.21.15.114' 2024-05-27T07:28:28.390 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.521+0000 7f0c58639700 1 Processor -- start 2024-05-27T07:28:28.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.521+0000 7f0c58639700 1 -- start start 2024-05-27T07:28:28.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.521+0000 7f0c58639700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 0x7f0c5010cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.521+0000 7f0c58639700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5007d690 con 0x7f0c5010a6e0 2024-05-27T07:28:28.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.522+0000 7f0c563d5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 0x7f0c5010cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.522+0000 7f0c563d5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 0x7f0c5010cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41146/0 (socket says 172.21.15.114:41146) 2024-05-27T07:28:28.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.522+0000 7f0c563d5700 1 -- 172.21.15.114:0/3140325756 learned_addr learned my addr 172.21.15.114:0/3140325756 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:28.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.522+0000 7f0c563d5700 1 -- 172.21.15.114:0/3140325756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5010d040 con 0x7f0c5010a6e0 2024-05-27T07:28:28.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.522+0000 7f0c563d5700 1 --2- 172.21.15.114:0/3140325756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 0x7f0c5010cb00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0098a0 tx=0x7f0c4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c51745b3e17882f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:28.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.522+0000 7f0c553d3700 1 -- 172.21.15.114:0/3140325756 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c4c004160 con 0x7f0c5010a6e0 2024-05-27T07:28:28.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.522+0000 7f0c553d3700 1 -- 172.21.15.114:0/3140325756 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0c4c004300 con 0x7f0c5010a6e0 2024-05-27T07:28:28.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.522+0000 7f0c553d3700 1 -- 172.21.15.114:0/3140325756 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c4c011af0 con 0x7f0c5010a6e0 2024-05-27T07:28:28.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.523+0000 7f0c58639700 1 -- 172.21.15.114:0/3140325756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 msgr2=0x7f0c5010cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:28.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.523+0000 7f0c58639700 1 --2- 172.21.15.114:0/3140325756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 0x7f0c5010cb00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0098a0 tx=0x7f0c4c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.523+0000 7f0c58639700 1 -- 172.21.15.114:0/3140325756 shutdown_connections 2024-05-27T07:28:28.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.523+0000 7f0c58639700 1 --2- 172.21.15.114:0/3140325756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 0x7f0c5010cb00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.523+0000 7f0c58639700 1 -- 172.21.15.114:0/3140325756 >> 172.21.15.114:0/3140325756 conn(0x7f0c501041c0 msgr2=0x7f0c50106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:28.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.523+0000 7f0c58639700 1 -- 172.21.15.114:0/3140325756 shutdown_connections 2024-05-27T07:28:28.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.523+0000 7f0c58639700 1 -- 172.21.15.114:0/3140325756 wait complete. 2024-05-27T07:28:28.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.524+0000 7f0c58639700 1 Processor -- start 2024-05-27T07:28:28.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.524+0000 7f0c58639700 1 -- start start 2024-05-27T07:28:28.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.524+0000 7f0c58639700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 0x7f0c5019df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.524+0000 7f0c58639700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5010df40 con 0x7f0c5010a6e0 2024-05-27T07:28:28.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.524+0000 7f0c563d5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 0x7f0c5019df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.524+0000 7f0c563d5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 0x7f0c5019df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41148/0 (socket says 172.21.15.114:41148) 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.524+0000 7f0c563d5700 1 -- 172.21.15.114:0/440652788 learned_addr learned my addr 172.21.15.114:0/440652788 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.524+0000 7f0c563d5700 1 -- 172.21.15.114:0/440652788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5019e4a0 con 0x7f0c5010a6e0 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.525+0000 7f0c563d5700 1 --2- 172.21.15.114:0/440652788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 0x7f0c5019df60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0117b0 tx=0x7f0c4c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.525+0000 7f0c477fe700 1 -- 172.21.15.114:0/440652788 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c4c003af0 con 0x7f0c5010a6e0 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.525+0000 7f0c477fe700 1 -- 172.21.15.114:0/440652788 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0c4c020070 con 0x7f0c5010a6e0 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.525+0000 7f0c58639700 1 -- 172.21.15.114:0/440652788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c5019e730 con 0x7f0c5010a6e0 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.525+0000 7f0c477fe700 1 -- 172.21.15.114:0/440652788 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c4c019630 con 0x7f0c5010a6e0 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.525+0000 7f0c58639700 1 -- 172.21.15.114:0/440652788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c501a1340 con 0x7f0c5010a6e0 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.526+0000 7f0c58639700 1 -- 172.21.15.114:0/440652788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c50064e10 con 0x7f0c5010a6e0 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.526+0000 7f0c477fe700 1 -- 172.21.15.114:0/440652788 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f0c4c0198f0 con 0x7f0c5010a6e0 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.526+0000 7f0c477fe700 1 --2- 172.21.15.114:0/440652788 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f0c3c03e470 0x7f0c3c040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.526+0000 7f0c477fe700 1 -- 172.21.15.114:0/440652788 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0c4c053b40 con 0x7f0c5010a6e0 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.526+0000 7f0c55bd4700 1 --2- 172.21.15.114:0/440652788 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f0c3c03e470 0x7f0c3c040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.527+0000 7f0c55bd4700 1 --2- 172.21.15.114:0/440652788 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f0c3c03e470 0x7f0c3c040980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0c40006fd0 tx=0x7f0c40006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:28.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.531+0000 7f0c477fe700 1 -- 172.21.15.114:0/440652788 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0c4c016260 con 0x7f0c5010a6e0 2024-05-27T07:28:28.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:23.736+0000 7f0c58639700 1 -- 172.21.15.114:0/440652788 --> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi114", "addr": "172.21.15.114", "target": ["mon-mgr", ""]}) -- 0x7f0c50106210 con 0x7f0c3c03e470 2024-05-27T07:28:28.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:27.812+0000 7f0c477fe700 1 -- 172.21.15.114:0/440652788 <== mgr.14120 v2:172.21.15.114:6800/752776783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f0c50106210 con 0x7f0c3c03e470 2024-05-27T07:28:28.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:27.814+0000 7f0c58639700 1 -- 172.21.15.114:0/440652788 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f0c3c03e470 msgr2=0x7f0c3c040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:28.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:27.814+0000 7f0c58639700 1 --2- 172.21.15.114:0/440652788 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f0c3c03e470 0x7f0c3c040980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0c40006fd0 tx=0x7f0c40006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:28.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:27.814+0000 7f0c58639700 1 -- 172.21.15.114:0/440652788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 msgr2=0x7f0c5019df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:28.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:27.814+0000 7f0c58639700 1 --2- 172.21.15.114:0/440652788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 0x7f0c5019df60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0117b0 tx=0x7f0c4c011b70 comp rx=0 tx=0).stop 2024-05-27T07:28:28.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:27.815+0000 7f0c58639700 1 -- 172.21.15.114:0/440652788 shutdown_connections 2024-05-27T07:28:28.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:27.815+0000 7f0c58639700 1 --2- 172.21.15.114:0/440652788 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f0c3c03e470 0x7f0c3c040980 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:27.815+0000 7f0c58639700 1 --2- 172.21.15.114:0/440652788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c5010a6e0 0x7f0c5019df60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:27.815+0000 7f0c58639700 1 -- 172.21.15.114:0/440652788 >> 172.21.15.114:0/440652788 conn(0x7f0c501041c0 msgr2=0x7f0c50105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:28.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:27.815+0000 7f0c58639700 1 -- 172.21.15.114:0/440652788 shutdown_connections 2024-05-27T07:28:28.395 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:27.815+0000 7f0c58639700 1 -- 172.21.15.114:0/440652788 wait complete. 2024-05-27T07:28:28.395 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying unmanaged mon service... 2024-05-27T07:28:29.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:28 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' 2024-05-27T07:28:29.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:28 smithi114 ceph-mon[32038]: Added host smithi114 2024-05-27T07:28:29.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:28 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:29.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-27T07:28:29.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.820+0000 7fbbab257700 1 Processor -- start 2024-05-27T07:28:29.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.820+0000 7fbbab257700 1 -- start start 2024-05-27T07:28:29.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.821+0000 7fbbab257700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 0x7fbba410a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.821+0000 7fbbab257700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbba410a9f0 con 0x7fbba410a010 2024-05-27T07:28:29.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.821+0000 7fbba8ff3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 0x7fbba410a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.821+0000 7fbba8ff3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 0x7fbba410a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44036/0 (socket says 172.21.15.114:44036) 2024-05-27T07:28:29.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.821+0000 7fbba8ff3700 1 -- 172.21.15.114:0/2100687064 learned_addr learned my addr 172.21.15.114:0/2100687064 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:29.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.821+0000 7fbba8ff3700 1 -- 172.21.15.114:0/2100687064 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba4108c90 con 0x7fbba410a010 2024-05-27T07:28:29.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.822+0000 7fbba8ff3700 1 --2- 172.21.15.114:0/2100687064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 0x7fbba410a420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbb980098a0 tx=0x7fbb98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b5add6b479fe84 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:29.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.822+0000 7fbba37fe700 1 -- 172.21.15.114:0/2100687064 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb98004160 con 0x7fbba410a010 2024-05-27T07:28:29.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.822+0000 7fbba37fe700 1 -- 172.21.15.114:0/2100687064 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbb98004300 con 0x7fbba410a010 2024-05-27T07:28:29.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.822+0000 7fbba37fe700 1 -- 172.21.15.114:0/2100687064 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb98011af0 con 0x7fbba410a010 2024-05-27T07:28:29.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.823+0000 7fbbab257700 1 -- 172.21.15.114:0/2100687064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 msgr2=0x7fbba410a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:29.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.823+0000 7fbbab257700 1 --2- 172.21.15.114:0/2100687064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 0x7fbba410a420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbb980098a0 tx=0x7fbb98009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.823+0000 7fbbab257700 1 -- 172.21.15.114:0/2100687064 shutdown_connections 2024-05-27T07:28:29.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.823+0000 7fbbab257700 1 --2- 172.21.15.114:0/2100687064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 0x7fbba410a420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.823+0000 7fbbab257700 1 -- 172.21.15.114:0/2100687064 >> 172.21.15.114:0/2100687064 conn(0x7fbba41038b0 msgr2=0x7fbba4105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:29.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.823+0000 7fbbab257700 1 -- 172.21.15.114:0/2100687064 shutdown_connections 2024-05-27T07:28:29.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.823+0000 7fbbab257700 1 -- 172.21.15.114:0/2100687064 wait complete. 2024-05-27T07:28:29.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.824+0000 7fbbab257700 1 Processor -- start 2024-05-27T07:28:29.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.824+0000 7fbbab257700 1 -- start start 2024-05-27T07:28:29.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.824+0000 7fbbab257700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 0x7fbba4108500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.824+0000 7fbbab257700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbba410d050 con 0x7fbba410a010 2024-05-27T07:28:29.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.824+0000 7fbba8ff3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 0x7fbba4108500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.824+0000 7fbba8ff3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 0x7fbba4108500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44048/0 (socket says 172.21.15.114:44048) 2024-05-27T07:28:29.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.824+0000 7fbba8ff3700 1 -- 172.21.15.114:0/3297291783 learned_addr learned my addr 172.21.15.114:0/3297291783 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:29.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.824+0000 7fbba8ff3700 1 -- 172.21.15.114:0/3297291783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba4108a40 con 0x7fbba410a010 2024-05-27T07:28:29.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.824+0000 7fbba8ff3700 1 --2- 172.21.15.114:0/3297291783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 0x7fbba4108500 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbb980090e0 tx=0x7fbb98011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:29.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.825+0000 7fbba1ffb700 1 -- 172.21.15.114:0/3297291783 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb980039e0 con 0x7fbba410a010 2024-05-27T07:28:29.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.825+0000 7fbba1ffb700 1 -- 172.21.15.114:0/3297291783 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbb98003b80 con 0x7fbba410a010 2024-05-27T07:28:29.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.825+0000 7fbba1ffb700 1 -- 172.21.15.114:0/3297291783 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb980246e0 con 0x7fbba410a010 2024-05-27T07:28:29.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.825+0000 7fbbab257700 1 -- 172.21.15.114:0/3297291783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba41a0e70 con 0x7fbba410a010 2024-05-27T07:28:29.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.825+0000 7fbbab257700 1 -- 172.21.15.114:0/3297291783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba41a12f0 con 0x7fbba410a010 2024-05-27T07:28:29.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.826+0000 7fbbab257700 1 -- 172.21.15.114:0/3297291783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba4064e10 con 0x7fbba410a010 2024-05-27T07:28:29.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.826+0000 7fbba1ffb700 1 -- 172.21.15.114:0/3297291783 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fbb98022020 con 0x7fbba410a010 2024-05-27T07:28:29.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.826+0000 7fbba1ffb700 1 --2- 172.21.15.114:0/3297291783 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fbb9403e4c0 0x7fbb940409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.826+0000 7fbba1ffb700 1 -- 172.21.15.114:0/3297291783 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbb98053a60 con 0x7fbba410a010 2024-05-27T07:28:29.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.827+0000 7fbba3fff700 1 --2- 172.21.15.114:0/3297291783 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fbb9403e4c0 0x7fbb940409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.827+0000 7fbba3fff700 1 --2- 172.21.15.114:0/3297291783 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fbb9403e4c0 0x7fbb940409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbb90006fd0 tx=0x7fbb90006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:29.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:28.832+0000 7fbba1ffb700 1 -- 172.21.15.114:0/3297291783 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbb9800fdd0 con 0x7fbba410a010 2024-05-27T07:28:29.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:29.080+0000 7fbbab257700 1 -- 172.21.15.114:0/3297291783 --> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fbba4063bf0 con 0x7fbb9403e4c0 2024-05-27T07:28:29.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:29.085+0000 7fbba1ffb700 1 -- 172.21.15.114:0/3297291783 <== mgr.14120 v2:172.21.15.114:6800/752776783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fbba4063bf0 con 0x7fbb9403e4c0 2024-05-27T07:28:29.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:29.088+0000 7fbbab257700 1 -- 172.21.15.114:0/3297291783 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fbb9403e4c0 msgr2=0x7fbb940409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:29.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:29.088+0000 7fbbab257700 1 --2- 172.21.15.114:0/3297291783 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fbb9403e4c0 0x7fbb940409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbb90006fd0 tx=0x7fbb90006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:29.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:29.088+0000 7fbbab257700 1 -- 172.21.15.114:0/3297291783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 msgr2=0x7fbba4108500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:29.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:29.088+0000 7fbbab257700 1 --2- 172.21.15.114:0/3297291783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 0x7fbba4108500 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbb980090e0 tx=0x7fbb98011940 comp rx=0 tx=0).stop 2024-05-27T07:28:29.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:29.088+0000 7fbbab257700 1 -- 172.21.15.114:0/3297291783 shutdown_connections 2024-05-27T07:28:29.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:29.088+0000 7fbbab257700 1 --2- 172.21.15.114:0/3297291783 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fbb9403e4c0 0x7fbb940409d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:29.088+0000 7fbbab257700 1 --2- 172.21.15.114:0/3297291783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba410a010 0x7fbba4108500 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:29.088+0000 7fbbab257700 1 -- 172.21.15.114:0/3297291783 >> 172.21.15.114:0/3297291783 conn(0x7fbba41038b0 msgr2=0x7fbba4104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:29.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:29.088+0000 7fbbab257700 1 -- 172.21.15.114:0/3297291783 shutdown_connections 2024-05-27T07:28:29.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:29.088+0000 7fbbab257700 1 -- 172.21.15.114:0/3297291783 wait complete. 2024-05-27T07:28:29.812 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying unmanaged mgr service... 2024-05-27T07:28:30.407 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:30 smithi114 ceph-mon[32038]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:28:30.407 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:30 smithi114 ceph-mon[32038]: Saving service mon spec with placement count:5 2024-05-27T07:28:30.408 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:30 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' 2024-05-27T07:28:31.069 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-27T07:28:31.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.240+0000 7f55c2ee7700 1 Processor -- start 2024-05-27T07:28:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.241+0000 7f55c2ee7700 1 -- start start 2024-05-27T07:28:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.241+0000 7f55c2ee7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 0x7f55bc10c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.241+0000 7f55c2ee7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc10cd40 con 0x7f55bc07d580 2024-05-27T07:28:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.241+0000 7f55c0c83700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 0x7f55bc10c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.241+0000 7f55c0c83700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 0x7f55bc10c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44060/0 (socket says 172.21.15.114:44060) 2024-05-27T07:28:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.241+0000 7f55c0c83700 1 -- 172.21.15.114:0/2695919141 learned_addr learned my addr 172.21.15.114:0/2695919141 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.241+0000 7f55c0c83700 1 -- 172.21.15.114:0/2695919141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc10cec0 con 0x7f55bc07d580 2024-05-27T07:28:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.241+0000 7f55c0c83700 1 --2- 172.21.15.114:0/2695919141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 0x7f55bc10c730 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f55b00098a0 tx=0x7f55b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7e7e7eb54daacc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.242+0000 7f55bb7fe700 1 -- 172.21.15.114:0/2695919141 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55b0004160 con 0x7f55bc07d580 2024-05-27T07:28:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.242+0000 7f55bb7fe700 1 -- 172.21.15.114:0/2695919141 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f55b0004300 con 0x7f55bc07d580 2024-05-27T07:28:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.242+0000 7f55bb7fe700 1 -- 172.21.15.114:0/2695919141 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55b0011af0 con 0x7f55bc07d580 2024-05-27T07:28:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.242+0000 7f55c2ee7700 1 -- 172.21.15.114:0/2695919141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 msgr2=0x7f55bc10c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.242+0000 7f55c2ee7700 1 --2- 172.21.15.114:0/2695919141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 0x7f55bc10c730 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f55b00098a0 tx=0x7f55b0009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.242+0000 7f55c2ee7700 1 -- 172.21.15.114:0/2695919141 shutdown_connections 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.242+0000 7f55c2ee7700 1 --2- 172.21.15.114:0/2695919141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 0x7f55bc10c730 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.242+0000 7f55c2ee7700 1 -- 172.21.15.114:0/2695919141 >> 172.21.15.114:0/2695919141 conn(0x7f55bc1041c0 msgr2=0x7f55bc106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.243+0000 7f55c2ee7700 1 -- 172.21.15.114:0/2695919141 shutdown_connections 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.243+0000 7f55c2ee7700 1 -- 172.21.15.114:0/2695919141 wait complete. 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.243+0000 7f55c2ee7700 1 Processor -- start 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.243+0000 7f55c2ee7700 1 -- start start 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.243+0000 7f55c2ee7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 0x7f55bc19df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.243+0000 7f55c2ee7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc10df40 con 0x7f55bc07d580 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.243+0000 7f55c0c83700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 0x7f55bc19df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.244+0000 7f55c0c83700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 0x7f55bc19df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44076/0 (socket says 172.21.15.114:44076) 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.244+0000 7f55c0c83700 1 -- 172.21.15.114:0/3416665800 learned_addr learned my addr 172.21.15.114:0/3416665800 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.244+0000 7f55c0c83700 1 -- 172.21.15.114:0/3416665800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc19e480 con 0x7f55bc07d580 2024-05-27T07:28:31.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.244+0000 7f55c0c83700 1 --2- 172.21.15.114:0/3416665800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 0x7f55bc19df40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f55b0009fb0 tx=0x7f55b0011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:31.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.244+0000 7f55b9ffb700 1 -- 172.21.15.114:0/3416665800 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55b00036a0 con 0x7f55bc07d580 2024-05-27T07:28:31.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.244+0000 7f55c2ee7700 1 -- 172.21.15.114:0/3416665800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55bc19e710 con 0x7f55bc07d580 2024-05-27T07:28:31.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.244+0000 7f55b9ffb700 1 -- 172.21.15.114:0/3416665800 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f55b001a430 con 0x7f55bc07d580 2024-05-27T07:28:31.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.244+0000 7f55b9ffb700 1 -- 172.21.15.114:0/3416665800 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55b0019550 con 0x7f55bc07d580 2024-05-27T07:28:31.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.244+0000 7f55c2ee7700 1 -- 172.21.15.114:0/3416665800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55bc1a1320 con 0x7f55bc07d580 2024-05-27T07:28:31.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.245+0000 7f55b9ffb700 1 -- 172.21.15.114:0/3416665800 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f55b0019850 con 0x7f55bc07d580 2024-05-27T07:28:31.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.245+0000 7f55c2ee7700 1 -- 172.21.15.114:0/3416665800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55bc064e10 con 0x7f55bc07d580 2024-05-27T07:28:31.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.245+0000 7f55b9ffb700 1 --2- 172.21.15.114:0/3416665800 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f55ac03e420 0x7f55ac040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:31.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.246+0000 7f55b9ffb700 1 -- 172.21.15.114:0/3416665800 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f55b0053890 con 0x7f55bc07d580 2024-05-27T07:28:31.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.246+0000 7f55bbfff700 1 --2- 172.21.15.114:0/3416665800 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f55ac03e420 0x7f55ac040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:31.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.246+0000 7f55bbfff700 1 --2- 172.21.15.114:0/3416665800 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f55ac03e420 0x7f55ac040930 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f55a8006fd0 tx=0x7f55a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:31.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.251+0000 7f55b9ffb700 1 -- 172.21.15.114:0/3416665800 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f55b0017e90 con 0x7f55bc07d580 2024-05-27T07:28:31.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.478+0000 7f55c2ee7700 1 -- 172.21.15.114:0/3416665800 --> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f55bc106210 con 0x7f55ac03e420 2024-05-27T07:28:31.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.486+0000 7f55b9ffb700 1 -- 172.21.15.114:0/3416665800 <== mgr.14120 v2:172.21.15.114:6800/752776783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f55bc106210 con 0x7f55ac03e420 2024-05-27T07:28:31.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.489+0000 7f55c2ee7700 1 -- 172.21.15.114:0/3416665800 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f55ac03e420 msgr2=0x7f55ac040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:31.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.489+0000 7f55c2ee7700 1 --2- 172.21.15.114:0/3416665800 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f55ac03e420 0x7f55ac040930 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f55a8006fd0 tx=0x7f55a8006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:31.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.489+0000 7f55c2ee7700 1 -- 172.21.15.114:0/3416665800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 msgr2=0x7f55bc19df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:31.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.489+0000 7f55c2ee7700 1 --2- 172.21.15.114:0/3416665800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 0x7f55bc19df40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f55b0009fb0 tx=0x7f55b0011940 comp rx=0 tx=0).stop 2024-05-27T07:28:31.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.489+0000 7f55c2ee7700 1 -- 172.21.15.114:0/3416665800 shutdown_connections 2024-05-27T07:28:31.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.489+0000 7f55c2ee7700 1 --2- 172.21.15.114:0/3416665800 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f55ac03e420 0x7f55ac040930 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:31.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.489+0000 7f55c2ee7700 1 --2- 172.21.15.114:0/3416665800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55bc07d580 0x7f55bc19df40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:31.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.489+0000 7f55c2ee7700 1 -- 172.21.15.114:0/3416665800 >> 172.21.15.114:0/3416665800 conn(0x7f55bc1041c0 msgr2=0x7f55bc105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:31.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.489+0000 7f55c2ee7700 1 -- 172.21.15.114:0/3416665800 shutdown_connections 2024-05-27T07:28:31.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:30.489+0000 7f55c2ee7700 1 -- 172.21.15.114:0/3416665800 wait complete. 2024-05-27T07:28:31.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:31 smithi114 ceph-mon[32038]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:28:31.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:31 smithi114 ceph-mon[32038]: Saving service mgr spec with placement count:2 2024-05-27T07:28:31.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:31 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' 2024-05-27T07:28:32.954 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:32 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2762503792' entity='client.admin' 2024-05-27T07:28:32.954 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:32 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' 2024-05-27T07:28:32.961 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.556+0000 7f71aa806700 1 Processor -- start 2024-05-27T07:28:32.961 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.557+0000 7f71aa806700 1 -- start start 2024-05-27T07:28:32.961 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.557+0000 7f71aa806700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 0x7f71a410c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.961 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.557+0000 7f71aa806700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f71a410cd30 con 0x7f71a407d720 2024-05-27T07:28:32.961 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.557+0000 7f71a3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 0x7f71a410c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:32.962 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.557+0000 7f71a3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 0x7f71a410c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44088/0 (socket says 172.21.15.114:44088) 2024-05-27T07:28:32.962 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.557+0000 7f71a3fff700 1 -- 172.21.15.114:0/3237832049 learned_addr learned my addr 172.21.15.114:0/3237832049 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:32.962 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.557+0000 7f71a3fff700 1 -- 172.21.15.114:0/3237832049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a410ceb0 con 0x7f71a407d720 2024-05-27T07:28:32.962 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.557+0000 7f71a3fff700 1 --2- 172.21.15.114:0/3237832049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 0x7f71a410c720 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f71940098a0 tx=0x7f7194009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2879974dee09a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:32.962 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.558+0000 7f71a37fe700 1 -- 172.21.15.114:0/3237832049 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7194004160 con 0x7f71a407d720 2024-05-27T07:28:32.962 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.558+0000 7f71a37fe700 1 -- 172.21.15.114:0/3237832049 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7194004300 con 0x7f71a407d720 2024-05-27T07:28:32.962 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.558+0000 7f71a37fe700 1 -- 172.21.15.114:0/3237832049 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7194011af0 con 0x7f71a407d720 2024-05-27T07:28:32.962 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.558+0000 7f71aa806700 1 -- 172.21.15.114:0/3237832049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 msgr2=0x7f71a410c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:32.962 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.558+0000 7f71aa806700 1 --2- 172.21.15.114:0/3237832049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 0x7f71a410c720 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f71940098a0 tx=0x7f7194009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.962 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.559+0000 7f71aa806700 1 -- 172.21.15.114:0/3237832049 shutdown_connections 2024-05-27T07:28:32.963 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.559+0000 7f71aa806700 1 --2- 172.21.15.114:0/3237832049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 0x7f71a410c720 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.963 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.559+0000 7f71aa806700 1 -- 172.21.15.114:0/3237832049 >> 172.21.15.114:0/3237832049 conn(0x7f71a4104190 msgr2=0x7f71a41065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:32.963 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.559+0000 7f71aa806700 1 -- 172.21.15.114:0/3237832049 shutdown_connections 2024-05-27T07:28:32.963 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.559+0000 7f71aa806700 1 -- 172.21.15.114:0/3237832049 wait complete. 2024-05-27T07:28:32.963 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.559+0000 7f71aa806700 1 Processor -- start 2024-05-27T07:28:32.963 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.559+0000 7f71aa806700 1 -- start start 2024-05-27T07:28:32.963 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.560+0000 7f71aa806700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 0x7f71a419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.963 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.560+0000 7f71aa806700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f71a410df30 con 0x7f71a407d720 2024-05-27T07:28:32.963 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.560+0000 7f71a3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 0x7f71a419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:32.963 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.560+0000 7f71a3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 0x7f71a419dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44092/0 (socket says 172.21.15.114:44092) 2024-05-27T07:28:32.963 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.560+0000 7f71a3fff700 1 -- 172.21.15.114:0/2762503792 learned_addr learned my addr 172.21.15.114:0/2762503792 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:32.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.560+0000 7f71a3fff700 1 -- 172.21.15.114:0/2762503792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a419e520 con 0x7f71a407d720 2024-05-27T07:28:32.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.560+0000 7f71a3fff700 1 --2- 172.21.15.114:0/2762503792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 0x7f71a419dfe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f71940117b0 tx=0x7f7194011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:32.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.560+0000 7f71a1ffb700 1 -- 172.21.15.114:0/2762503792 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71940039c0 con 0x7f71a407d720 2024-05-27T07:28:32.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.560+0000 7f71a1ffb700 1 -- 172.21.15.114:0/2762503792 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7194003b60 con 0x7f71a407d720 2024-05-27T07:28:32.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.560+0000 7f71a1ffb700 1 -- 172.21.15.114:0/2762503792 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71940196a0 con 0x7f71a407d720 2024-05-27T07:28:32.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.560+0000 7f71aa806700 1 -- 172.21.15.114:0/2762503792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a419e7b0 con 0x7f71a407d720 2024-05-27T07:28:32.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.560+0000 7f71aa806700 1 -- 172.21.15.114:0/2762503792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a41a13c0 con 0x7f71a407d720 2024-05-27T07:28:32.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.561+0000 7f71aa806700 1 -- 172.21.15.114:0/2762503792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71a4064e10 con 0x7f71a407d720 2024-05-27T07:28:32.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.561+0000 7f71a1ffb700 1 -- 172.21.15.114:0/2762503792 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f7194019910 con 0x7f71a407d720 2024-05-27T07:28:32.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.562+0000 7f71a1ffb700 1 --2- 172.21.15.114:0/2762503792 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f719003e4c0 0x7f71900409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.562+0000 7f71a1ffb700 1 -- 172.21.15.114:0/2762503792 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7194053b30 con 0x7f71a407d720 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.562+0000 7f719bfff700 1 --2- 172.21.15.114:0/2762503792 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f719003e4c0 0x7f71900409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.562+0000 7f719bfff700 1 --2- 172.21.15.114:0/2762503792 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f719003e4c0 0x7f71900409d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f718c006fd0 tx=0x7f718c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.567+0000 7f71a1ffb700 1 -- 172.21.15.114:0/2762503792 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7194017e90 con 0x7f71a407d720 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.782+0000 7f71aa806700 1 -- 172.21.15.114:0/2762503792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f71a41a1a40 con 0x7f71a407d720 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.788+0000 7f71a1ffb700 1 -- 172.21.15.114:0/2762503792 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7f7194023390 con 0x7f71a407d720 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.790+0000 7f71aa806700 1 -- 172.21.15.114:0/2762503792 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f719003e4c0 msgr2=0x7f71900409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.790+0000 7f71aa806700 1 --2- 172.21.15.114:0/2762503792 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f719003e4c0 0x7f71900409d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f718c006fd0 tx=0x7f718c006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.790+0000 7f71aa806700 1 -- 172.21.15.114:0/2762503792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 msgr2=0x7f71a419dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.790+0000 7f71aa806700 1 --2- 172.21.15.114:0/2762503792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 0x7f71a419dfe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f71940117b0 tx=0x7f7194011b70 comp rx=0 tx=0).stop 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.790+0000 7f71aa806700 1 -- 172.21.15.114:0/2762503792 shutdown_connections 2024-05-27T07:28:32.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.790+0000 7f71aa806700 1 --2- 172.21.15.114:0/2762503792 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f719003e4c0 0x7f71900409d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.790+0000 7f71aa806700 1 --2- 172.21.15.114:0/2762503792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a407d720 0x7f71a419dfe0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.790+0000 7f71aa806700 1 -- 172.21.15.114:0/2762503792 >> 172.21.15.114:0/2762503792 conn(0x7f71a4104190 msgr2=0x7f71a4104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:32.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.790+0000 7f71aa806700 1 -- 172.21.15.114:0/2762503792 shutdown_connections 2024-05-27T07:28:32.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:31.790+0000 7f71aa806700 1 -- 172.21.15.114:0/2762503792 wait complete. 2024-05-27T07:28:34.195 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:34 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' 2024-05-27T07:28:34.195 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:34 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2061784981' entity='client.admin' 2024-05-27T07:28:34.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.416+0000 7fca16e4d700 1 Processor -- start 2024-05-27T07:28:34.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.416+0000 7fca16e4d700 1 -- start start 2024-05-27T07:28:34.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.416+0000 7fca16e4d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 0x7fca1010c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:34.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.416+0000 7fca16e4d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fca1010cce0 con 0x7fca1007ccf0 2024-05-27T07:28:34.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.417+0000 7fca14be9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 0x7fca1010c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:34.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.417+0000 7fca14be9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 0x7fca1010c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44096/0 (socket says 172.21.15.114:44096) 2024-05-27T07:28:34.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.417+0000 7fca14be9700 1 -- 172.21.15.114:0/55807670 learned_addr learned my addr 172.21.15.114:0/55807670 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:34.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.417+0000 7fca14be9700 1 -- 172.21.15.114:0/55807670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1010ce60 con 0x7fca1007ccf0 2024-05-27T07:28:34.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.417+0000 7fca14be9700 1 --2- 172.21.15.114:0/55807670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 0x7fca1010c6d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fca040098a0 tx=0x7fca04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f928f6377bd96441 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.417+0000 7fca0f7fe700 1 -- 172.21.15.114:0/55807670 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca04004160 con 0x7fca1007ccf0 2024-05-27T07:28:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.417+0000 7fca0f7fe700 1 -- 172.21.15.114:0/55807670 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca04004300 con 0x7fca1007ccf0 2024-05-27T07:28:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.417+0000 7fca0f7fe700 1 -- 172.21.15.114:0/55807670 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca04011a20 con 0x7fca1007ccf0 2024-05-27T07:28:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.418+0000 7fca16e4d700 1 -- 172.21.15.114:0/55807670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 msgr2=0x7fca1010c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.418+0000 7fca16e4d700 1 --2- 172.21.15.114:0/55807670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 0x7fca1010c6d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fca040098a0 tx=0x7fca04009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.418+0000 7fca16e4d700 1 -- 172.21.15.114:0/55807670 shutdown_connections 2024-05-27T07:28:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.418+0000 7fca16e4d700 1 --2- 172.21.15.114:0/55807670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 0x7fca1010c6d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.418+0000 7fca16e4d700 1 -- 172.21.15.114:0/55807670 >> 172.21.15.114:0/55807670 conn(0x7fca10104120 msgr2=0x7fca10106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.418+0000 7fca16e4d700 1 -- 172.21.15.114:0/55807670 shutdown_connections 2024-05-27T07:28:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.418+0000 7fca16e4d700 1 -- 172.21.15.114:0/55807670 wait complete. 2024-05-27T07:28:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.419+0000 7fca16e4d700 1 Processor -- start 2024-05-27T07:28:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.419+0000 7fca16e4d700 1 -- start start 2024-05-27T07:28:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.419+0000 7fca16e4d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 0x7fca1019df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:34.463 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.419+0000 7fca16e4d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fca1010df20 con 0x7fca1007ccf0 2024-05-27T07:28:34.463 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.419+0000 7fca14be9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 0x7fca1019df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:34.463 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.419+0000 7fca14be9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 0x7fca1019df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44108/0 (socket says 172.21.15.114:44108) 2024-05-27T07:28:34.463 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.419+0000 7fca14be9700 1 -- 172.21.15.114:0/2061784981 learned_addr learned my addr 172.21.15.114:0/2061784981 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:34.463 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.420+0000 7fca14be9700 1 -- 172.21.15.114:0/2061784981 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1019e490 con 0x7fca1007ccf0 2024-05-27T07:28:34.463 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.420+0000 7fca14be9700 1 --2- 172.21.15.114:0/2061784981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 0x7fca1019df50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fca04004660 tx=0x7fca04011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:34.463 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.420+0000 7fca0dffb700 1 -- 172.21.15.114:0/2061784981 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca04003a60 con 0x7fca1007ccf0 2024-05-27T07:28:34.463 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.420+0000 7fca0dffb700 1 -- 172.21.15.114:0/2061784981 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca0401b8f0 con 0x7fca1007ccf0 2024-05-27T07:28:34.463 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.420+0000 7fca16e4d700 1 -- 172.21.15.114:0/2061784981 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca1019e720 con 0x7fca1007ccf0 2024-05-27T07:28:34.463 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.420+0000 7fca0dffb700 1 -- 172.21.15.114:0/2061784981 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca04023890 con 0x7fca1007ccf0 2024-05-27T07:28:34.463 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.420+0000 7fca16e4d700 1 -- 172.21.15.114:0/2061784981 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca101a1330 con 0x7fca1007ccf0 2024-05-27T07:28:34.463 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.421+0000 7fca0dffb700 1 -- 172.21.15.114:0/2061784981 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fca0402c3f0 con 0x7fca1007ccf0 2024-05-27T07:28:34.463 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.421+0000 7fca0dffb700 1 --2- 172.21.15.114:0/2061784981 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fca0003e470 0x7fca00040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:34.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.421+0000 7fca0dffb700 1 -- 172.21.15.114:0/2061784981 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fca04054e50 con 0x7fca1007ccf0 2024-05-27T07:28:34.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.421+0000 7fca16e4d700 1 -- 172.21.15.114:0/2061784981 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca10064e10 con 0x7fca1007ccf0 2024-05-27T07:28:34.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.421+0000 7fca0ffff700 1 --2- 172.21.15.114:0/2061784981 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fca0003e470 0x7fca00040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:34.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.422+0000 7fca0ffff700 1 --2- 172.21.15.114:0/2061784981 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fca0003e470 0x7fca00040980 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc006fd0 tx=0x7fc9fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:34.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.427+0000 7fca0dffb700 1 -- 172.21.15.114:0/2061784981 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fca0401f030 con 0x7fca1007ccf0 2024-05-27T07:28:34.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.644+0000 7fca16e4d700 1 -- 172.21.15.114:0/2061784981 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fca101a1500 con 0x7fca1007ccf0 2024-05-27T07:28:34.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.647+0000 7fca0dffb700 1 -- 172.21.15.114:0/2061784981 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fca0400f2e0 con 0x7fca1007ccf0 2024-05-27T07:28:34.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.649+0000 7fca16e4d700 1 -- 172.21.15.114:0/2061784981 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fca0003e470 msgr2=0x7fca00040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:34.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.649+0000 7fca16e4d700 1 --2- 172.21.15.114:0/2061784981 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fca0003e470 0x7fca00040980 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc006fd0 tx=0x7fc9fc006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:34.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.649+0000 7fca16e4d700 1 -- 172.21.15.114:0/2061784981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 msgr2=0x7fca1019df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:34.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.649+0000 7fca16e4d700 1 --2- 172.21.15.114:0/2061784981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 0x7fca1019df50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fca04004660 tx=0x7fca04011aa0 comp rx=0 tx=0).stop 2024-05-27T07:28:34.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.649+0000 7fca16e4d700 1 -- 172.21.15.114:0/2061784981 shutdown_connections 2024-05-27T07:28:34.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.649+0000 7fca16e4d700 1 --2- 172.21.15.114:0/2061784981 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fca0003e470 0x7fca00040980 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:34.465 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.649+0000 7fca16e4d700 1 --2- 172.21.15.114:0/2061784981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca1007ccf0 0x7fca1019df50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:34.465 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.649+0000 7fca16e4d700 1 -- 172.21.15.114:0/2061784981 >> 172.21.15.114:0/2061784981 conn(0x7fca10104120 msgr2=0x7fca10104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:34.465 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.649+0000 7fca16e4d700 1 -- 172.21.15.114:0/2061784981 shutdown_connections 2024-05-27T07:28:34.465 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:33.649+0000 7fca16e4d700 1 -- 172.21.15.114:0/2061784981 wait complete. 2024-05-27T07:28:34.465 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling the dashboard module... 2024-05-27T07:28:36.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:36 smithi114 ceph-mon[32038]: from='mgr.14120 172.21.15.114:0/1398961930' entity='mgr.x' 2024-05-27T07:28:36.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:36 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2850073714' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-27T07:28:36.268 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: ignoring --setuser ceph since I am not root 2024-05-27T07:28:36.268 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: ignoring --setgroup ceph since I am not root 2024-05-27T07:28:36.268 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:36.219+0000 7f315657b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:28:36.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:36.413+0000 7f315657b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:28:36.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:36.517+0000 7f315657b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:28:36.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.192+0000 7f3220c2b700 1 Processor -- start 2024-05-27T07:28:36.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.192+0000 7f3220c2b700 1 -- start start 2024-05-27T07:28:36.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.192+0000 7f3220c2b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 0x7f321c108eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:36.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.192+0000 7f3220c2b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f321c1094a0 con 0x7f321c108aa0 2024-05-27T07:28:36.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.193+0000 7f321a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 0x7f321c108eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:36.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.193+0000 7f321a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 0x7f321c108eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37594/0 (socket says 172.21.15.114:37594) 2024-05-27T07:28:36.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.193+0000 7f321a59c700 1 -- 172.21.15.114:0/1061140903 learned_addr learned my addr 172.21.15.114:0/1061140903 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:36.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.193+0000 7f321a59c700 1 -- 172.21.15.114:0/1061140903 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321c109bd0 con 0x7f321c108aa0 2024-05-27T07:28:36.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.193+0000 7f321a59c700 1 --2- 172.21.15.114:0/1061140903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 0x7f321c108eb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f32040098a0 tx=0x7f3204009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=956ff2097b393813 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:36.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.193+0000 7f321959a700 1 -- 172.21.15.114:0/1061140903 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3204004160 con 0x7f321c108aa0 2024-05-27T07:28:36.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.193+0000 7f321959a700 1 -- 172.21.15.114:0/1061140903 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3204004300 con 0x7f321c108aa0 2024-05-27T07:28:36.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.193+0000 7f321959a700 1 -- 172.21.15.114:0/1061140903 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3204011a20 con 0x7f321c108aa0 2024-05-27T07:28:36.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.194+0000 7f3220c2b700 1 -- 172.21.15.114:0/1061140903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 msgr2=0x7f321c108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:36.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.194+0000 7f3220c2b700 1 --2- 172.21.15.114:0/1061140903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 0x7f321c108eb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f32040098a0 tx=0x7f3204009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:36.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.194+0000 7f3220c2b700 1 -- 172.21.15.114:0/1061140903 shutdown_connections 2024-05-27T07:28:36.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.194+0000 7f3220c2b700 1 --2- 172.21.15.114:0/1061140903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 0x7f321c108eb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:36.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.194+0000 7f3220c2b700 1 -- 172.21.15.114:0/1061140903 >> 172.21.15.114:0/1061140903 conn(0x7f321c104120 msgr2=0x7f321c106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:36.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.194+0000 7f3220c2b700 1 -- 172.21.15.114:0/1061140903 shutdown_connections 2024-05-27T07:28:36.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.194+0000 7f3220c2b700 1 -- 172.21.15.114:0/1061140903 wait complete. 2024-05-27T07:28:36.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.195+0000 7f3220c2b700 1 Processor -- start 2024-05-27T07:28:36.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.195+0000 7f3220c2b700 1 -- start start 2024-05-27T07:28:36.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.195+0000 7f3220c2b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 0x7f321c07c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:36.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.195+0000 7f3220c2b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f321c10cf80 con 0x7f321c108aa0 2024-05-27T07:28:36.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.195+0000 7f321a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 0x7f321c07c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:36.601 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.195+0000 7f321a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 0x7f321c07c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37608/0 (socket says 172.21.15.114:37608) 2024-05-27T07:28:36.601 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.195+0000 7f321a59c700 1 -- 172.21.15.114:0/2850073714 learned_addr learned my addr 172.21.15.114:0/2850073714 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:36.601 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.196+0000 7f321a59c700 1 -- 172.21.15.114:0/2850073714 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321c07ced0 con 0x7f321c108aa0 2024-05-27T07:28:36.601 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.196+0000 7f321a59c700 1 --2- 172.21.15.114:0/2850073714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 0x7f321c07c990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3204011a20 tx=0x7f3204011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:36.601 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.196+0000 7f32137fe700 1 -- 172.21.15.114:0/2850073714 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32040039c0 con 0x7f321c108aa0 2024-05-27T07:28:36.601 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.196+0000 7f3220c2b700 1 -- 172.21.15.114:0/2850073714 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f321c1a0d50 con 0x7f321c108aa0 2024-05-27T07:28:36.601 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.196+0000 7f32137fe700 1 -- 172.21.15.114:0/2850073714 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3204003b60 con 0x7f321c108aa0 2024-05-27T07:28:36.601 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.196+0000 7f32137fe700 1 -- 172.21.15.114:0/2850073714 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3204019700 con 0x7f321c108aa0 2024-05-27T07:28:36.601 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.196+0000 7f3220c2b700 1 -- 172.21.15.114:0/2850073714 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f321c1a11d0 con 0x7f321c108aa0 2024-05-27T07:28:36.601 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.197+0000 7f32137fe700 1 -- 172.21.15.114:0/2850073714 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f3204019a00 con 0x7f321c108aa0 2024-05-27T07:28:36.602 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.197+0000 7f32137fe700 1 --2- 172.21.15.114:0/2850073714 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f320803e420 0x7f3208040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:36.602 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.197+0000 7f32137fe700 1 -- 172.21.15.114:0/2850073714 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f32040548a0 con 0x7f321c108aa0 2024-05-27T07:28:36.602 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.198+0000 7f3220c2b700 1 -- 172.21.15.114:0/2850073714 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f321c064e10 con 0x7f321c108aa0 2024-05-27T07:28:36.602 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.198+0000 7f3219d9b700 1 --2- 172.21.15.114:0/2850073714 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f320803e420 0x7f3208040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:36.602 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.198+0000 7f3219d9b700 1 --2- 172.21.15.114:0/2850073714 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f320803e420 0x7f3208040930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f320c006fd0 tx=0x7f320c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:36.602 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.203+0000 7f32137fe700 1 -- 172.21.15.114:0/2850073714 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3204020e20 con 0x7f321c108aa0 2024-05-27T07:28:36.602 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:35.481+0000 7f3220c2b700 1 -- 172.21.15.114:0/2850073714 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f321c1a1890 con 0x7f321c108aa0 2024-05-27T07:28:36.602 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:36.011+0000 7f32137fe700 1 -- 172.21.15.114:0/2850073714 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f3204018d60 con 0x7f321c108aa0 2024-05-27T07:28:36.602 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:36.013+0000 7f32137fe700 1 -- 172.21.15.114:0/2850073714 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f3204040d10 con 0x7f321c108aa0 2024-05-27T07:28:36.603 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:36.015+0000 7f3220c2b700 1 -- 172.21.15.114:0/2850073714 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f320803e420 msgr2=0x7f3208040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:36.603 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:36.015+0000 7f3220c2b700 1 --2- 172.21.15.114:0/2850073714 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f320803e420 0x7f3208040930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f320c006fd0 tx=0x7f320c006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:36.603 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:36.015+0000 7f3220c2b700 1 -- 172.21.15.114:0/2850073714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 msgr2=0x7f321c07c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:36.603 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:36.015+0000 7f3220c2b700 1 --2- 172.21.15.114:0/2850073714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 0x7f321c07c990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3204011a20 tx=0x7f3204011b00 comp rx=0 tx=0).stop 2024-05-27T07:28:36.603 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:36.015+0000 7f3220c2b700 1 -- 172.21.15.114:0/2850073714 shutdown_connections 2024-05-27T07:28:36.603 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:36.015+0000 7f3220c2b700 1 --2- 172.21.15.114:0/2850073714 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7f320803e420 0x7f3208040930 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:36.603 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:36.015+0000 7f3220c2b700 1 --2- 172.21.15.114:0/2850073714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c108aa0 0x7f321c07c990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:36.603 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:36.015+0000 7f3220c2b700 1 -- 172.21.15.114:0/2850073714 >> 172.21.15.114:0/2850073714 conn(0x7f321c104120 msgr2=0x7f321c104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:36.603 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:36.015+0000 7f3220c2b700 1 -- 172.21.15.114:0/2850073714 shutdown_connections 2024-05-27T07:28:36.603 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:36.015+0000 7f3220c2b700 1 -- 172.21.15.114:0/2850073714 wait complete. 2024-05-27T07:28:36.848 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:36.833+0000 7f315657b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:28:37.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:37 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2850073714' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-27T07:28:37.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:37 smithi114 ceph-mon[32038]: mgrmap e9: x(active, since 19s) 2024-05-27T07:28:37.288 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:36.924+0000 7f315657b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:28:37.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:28:37.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-27T07:28:37.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T07:28:37.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-27T07:28:37.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-27T07:28:37.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:28:37.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.091+0000 7fa20b23f700 1 Processor -- start 2024-05-27T07:28:37.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.092+0000 7fa20b23f700 1 -- start start 2024-05-27T07:28:37.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.092+0000 7fa20b23f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 0x7fa20410c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.092+0000 7fa20b23f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa20410ccf0 con 0x7fa20407d720 2024-05-27T07:28:37.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.092+0000 7fa208fdb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 0x7fa20410c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:37.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.092+0000 7fa208fdb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 0x7fa20410c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37628/0 (socket says 172.21.15.114:37628) 2024-05-27T07:28:37.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.092+0000 7fa208fdb700 1 -- 172.21.15.114:0/1336935343 learned_addr learned my addr 172.21.15.114:0/1336935343 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:37.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.092+0000 7fa208fdb700 1 -- 172.21.15.114:0/1336935343 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20410ce70 con 0x7fa20407d720 2024-05-27T07:28:37.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.093+0000 7fa208fdb700 1 --2- 172.21.15.114:0/1336935343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 0x7fa20410c6e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80098a0 tx=0x7fa1f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abdb78e67c51a39a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:37.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.093+0000 7fa2037fe700 1 -- 172.21.15.114:0/1336935343 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1f8004160 con 0x7fa20407d720 2024-05-27T07:28:37.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.093+0000 7fa2037fe700 1 -- 172.21.15.114:0/1336935343 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1f8004300 con 0x7fa20407d720 2024-05-27T07:28:37.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.093+0000 7fa2037fe700 1 -- 172.21.15.114:0/1336935343 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1f8011a20 con 0x7fa20407d720 2024-05-27T07:28:37.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.094+0000 7fa20b23f700 1 -- 172.21.15.114:0/1336935343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 msgr2=0x7fa20410c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:37.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.094+0000 7fa20b23f700 1 --2- 172.21.15.114:0/1336935343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 0x7fa20410c6e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80098a0 tx=0x7fa1f8009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.094+0000 7fa20b23f700 1 -- 172.21.15.114:0/1336935343 shutdown_connections 2024-05-27T07:28:37.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.094+0000 7fa20b23f700 1 --2- 172.21.15.114:0/1336935343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 0x7fa20410c6e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.094+0000 7fa20b23f700 1 -- 172.21.15.114:0/1336935343 >> 172.21.15.114:0/1336935343 conn(0x7fa204104170 msgr2=0x7fa2041065c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:37.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.094+0000 7fa20b23f700 1 -- 172.21.15.114:0/1336935343 shutdown_connections 2024-05-27T07:28:37.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.094+0000 7fa20b23f700 1 -- 172.21.15.114:0/1336935343 wait complete. 2024-05-27T07:28:37.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.095+0000 7fa20b23f700 1 Processor -- start 2024-05-27T07:28:37.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.095+0000 7fa20b23f700 1 -- start start 2024-05-27T07:28:37.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.095+0000 7fa20b23f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 0x7fa20419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.095+0000 7fa20b23f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa20410df30 con 0x7fa20407d720 2024-05-27T07:28:37.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.095+0000 7fa208fdb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 0x7fa20419df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:37.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.095+0000 7fa208fdb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 0x7fa20419df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37638/0 (socket says 172.21.15.114:37638) 2024-05-27T07:28:37.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.095+0000 7fa208fdb700 1 -- 172.21.15.114:0/574156450 learned_addr learned my addr 172.21.15.114:0/574156450 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:37.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.095+0000 7fa208fdb700 1 -- 172.21.15.114:0/574156450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20419e4a0 con 0x7fa20407d720 2024-05-27T07:28:37.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.095+0000 7fa208fdb700 1 --2- 172.21.15.114:0/574156450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 0x7fa20419df60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009520 tx=0x7fa1f8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:37.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.096+0000 7fa201ffb700 1 -- 172.21.15.114:0/574156450 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1f80039e0 con 0x7fa20407d720 2024-05-27T07:28:37.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.096+0000 7fa20b23f700 1 -- 172.21.15.114:0/574156450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa20419e730 con 0x7fa20407d720 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.096+0000 7fa201ffb700 1 -- 172.21.15.114:0/574156450 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1f801b8f0 con 0x7fa20407d720 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.096+0000 7fa201ffb700 1 -- 172.21.15.114:0/574156450 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1f80197d0 con 0x7fa20407d720 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.096+0000 7fa20b23f700 1 -- 172.21.15.114:0/574156450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2041a1340 con 0x7fa20407d720 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.097+0000 7fa20b23f700 1 -- 172.21.15.114:0/574156450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa204064e10 con 0x7fa20407d720 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.097+0000 7fa201ffb700 1 -- 172.21.15.114:0/574156450 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7fa1f8019a90 con 0x7fa20407d720 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.097+0000 7fa201ffb700 1 --2- 172.21.15.114:0/574156450 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fa1f403e4c0 0x7fa1f40409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.097+0000 7fa201ffb700 1 -- 172.21.15.114:0/574156450 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa1f8053fe0 con 0x7fa20407d720 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.097+0000 7fa203fff700 1 -- 172.21.15.114:0/574156450 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fa1f403e4c0 msgr2=0x7fa1f40409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/752776783 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.097+0000 7fa203fff700 1 --2- 172.21.15.114:0/574156450 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fa1f403e4c0 0x7fa1f40409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.103+0000 7fa201ffb700 1 -- 172.21.15.114:0/574156450 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa1f801f030 con 0x7fa20407d720 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.298+0000 7fa203fff700 1 -- 172.21.15.114:0/574156450 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fa1f403e4c0 msgr2=0x7fa1f40409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/752776783 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.298+0000 7fa203fff700 1 --2- 172.21.15.114:0/574156450 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fa1f403e4c0 0x7fa1f40409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.404+0000 7fa20b23f700 1 -- 172.21.15.114:0/574156450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fa2041a1510 con 0x7fa20407d720 2024-05-27T07:28:37.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.405+0000 7fa201ffb700 1 -- 172.21.15.114:0/574156450 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fa1f800b3e0 con 0x7fa20407d720 2024-05-27T07:28:37.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.407+0000 7fa20b23f700 1 -- 172.21.15.114:0/574156450 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fa1f403e4c0 msgr2=0x7fa1f40409d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:28:37.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.407+0000 7fa20b23f700 1 --2- 172.21.15.114:0/574156450 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fa1f403e4c0 0x7fa1f40409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.407+0000 7fa20b23f700 1 -- 172.21.15.114:0/574156450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 msgr2=0x7fa20419df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:37.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.407+0000 7fa20b23f700 1 --2- 172.21.15.114:0/574156450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 0x7fa20419df60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009520 tx=0x7fa1f8011aa0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.407+0000 7fa20b23f700 1 -- 172.21.15.114:0/574156450 shutdown_connections 2024-05-27T07:28:37.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.407+0000 7fa20b23f700 1 --2- 172.21.15.114:0/574156450 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fa1f403e4c0 0x7fa1f40409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.407+0000 7fa20b23f700 1 --2- 172.21.15.114:0/574156450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa20407d720 0x7fa20419df60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.407+0000 7fa20b23f700 1 -- 172.21.15.114:0/574156450 >> 172.21.15.114:0/574156450 conn(0x7fa204104170 msgr2=0x7fa204105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:37.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.407+0000 7fa20b23f700 1 -- 172.21.15.114:0/574156450 shutdown_connections 2024-05-27T07:28:37.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:37.408+0000 7fa20b23f700 1 -- 172.21.15.114:0/574156450 wait complete. 2024-05-27T07:28:37.924 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-05-27T07:28:37.924 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 9... 2024-05-27T07:28:38.259 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:38 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/574156450' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-27T07:28:38.259 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:38 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:38.148+0000 7f315657b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:28:38.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:38 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:38.334+0000 7f315657b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:28:38.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:38 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:38.516+0000 7f315657b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:28:38.913 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:38 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:38.826+0000 7f315657b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:28:38.913 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:38 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:38.924+0000 7f315657b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:28:39.295 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:39.132+0000 7f315657b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:28:39.781 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:39.335+0000 7f315657b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:28:40.046 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:39.940+0000 7f315657b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:28:40.546 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:40 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:40.166+0000 7f315657b200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:28:41.046 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:40 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:40.599+0000 7f315657b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:28:41.046 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:40 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:40.698+0000 7f315657b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:28:41.383 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:41 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:41.074+0000 7f315657b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:28:41.770 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:41 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:41.435+0000 7f315657b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:28:41.770 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:41 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:41.530+0000 7f315657b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:28:42.046 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:41 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:41.780+0000 7f315657b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:28:42.046 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:41 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:41.915+0000 7f315657b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:28:42.046 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:42 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:42.014+0000 7f315657b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:28:42.796 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:42 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:42.341+0000 7f315657b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:28:42.796 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:42 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:42.436+0000 7f315657b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:28:43.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:43 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:43.045+0000 7f315657b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:28:43.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:28:43 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:28:43.268+0000 7f315657b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:28:43.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:43 smithi114 ceph-mon[32038]: Active manager daemon x restarted 2024-05-27T07:28:43.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:43 smithi114 ceph-mon[32038]: Activating manager daemon x 2024-05-27T07:28:43.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:43 smithi114 ceph-mon[32038]: osdmap e3: 0 total, 0 up, 0 in 2024-05-27T07:28:43.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:43 smithi114 ceph-mon[32038]: mgrmap e10: x(active, starting, since 0.00455358s) 2024-05-27T07:28:43.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:28:43.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:28:43.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:28:43.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:28:43.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:28:43.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:43 smithi114 ceph-mon[32038]: Manager daemon x is now available 2024-05-27T07:28:43.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:44.545 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:28:44.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:28:44.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:44 smithi114 ceph-mon[32038]: mgrmap e11: x(active, since 1.00635s) 2024-05-27T07:28:44.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:28:44.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-27T07:28:44.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-27T07:28:44.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:28:44.920 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.383+0000 7fdf1100a700 1 Processor -- start 2024-05-27T07:28:44.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.383+0000 7fdf1100a700 1 -- start start 2024-05-27T07:28:44.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.383+0000 7fdf1100a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 0x7fdf0c10a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:44.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.383+0000 7fdf1100a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c10aa80 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.383+0000 7fdf0ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 0x7fdf0c10a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:44.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.383+0000 7fdf0ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 0x7fdf0c10a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37640/0 (socket says 172.21.15.114:37640) 2024-05-27T07:28:44.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.383+0000 7fdf0ad9d700 1 -- 172.21.15.114:0/3095134072 learned_addr learned my addr 172.21.15.114:0/3095134072 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:44.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.383+0000 7fdf0ad9d700 1 -- 172.21.15.114:0/3095134072 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf0c10ac00 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.384+0000 7fdf0ad9d700 1 --2- 172.21.15.114:0/3095134072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 0x7fdf0c10a4b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0098a0 tx=0x7fdefc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f4128b540cd119 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:44.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.384+0000 7fdf09d9b700 1 -- 172.21.15.114:0/3095134072 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdefc004160 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.384+0000 7fdf09d9b700 1 -- 172.21.15.114:0/3095134072 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdefc004300 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.384+0000 7fdf09d9b700 1 -- 172.21.15.114:0/3095134072 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdefc011a20 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.385+0000 7fdf1100a700 1 -- 172.21.15.114:0/3095134072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 msgr2=0x7fdf0c10a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:44.921 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.385+0000 7fdf1100a700 1 --2- 172.21.15.114:0/3095134072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 0x7fdf0c10a4b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0098a0 tx=0x7fdefc009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:44.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.385+0000 7fdf1100a700 1 -- 172.21.15.114:0/3095134072 shutdown_connections 2024-05-27T07:28:44.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.385+0000 7fdf1100a700 1 --2- 172.21.15.114:0/3095134072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 0x7fdf0c10a4b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:44.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.385+0000 7fdf1100a700 1 -- 172.21.15.114:0/3095134072 >> 172.21.15.114:0/3095134072 conn(0x7fdf0c1038b0 msgr2=0x7fdf0c105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:44.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.385+0000 7fdf1100a700 1 -- 172.21.15.114:0/3095134072 shutdown_connections 2024-05-27T07:28:44.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.385+0000 7fdf1100a700 1 -- 172.21.15.114:0/3095134072 wait complete. 2024-05-27T07:28:44.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.386+0000 7fdf1100a700 1 Processor -- start 2024-05-27T07:28:44.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.386+0000 7fdf1100a700 1 -- start start 2024-05-27T07:28:44.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.386+0000 7fdf1100a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 0x7fdf0c1a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:44.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.386+0000 7fdf1100a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c110190 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.387+0000 7fdf0ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 0x7fdf0c1a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:44.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.387+0000 7fdf0ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 0x7fdf0c1a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37644/0 (socket says 172.21.15.114:37644) 2024-05-27T07:28:44.922 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.387+0000 7fdf0ad9d700 1 -- 172.21.15.114:0/3003742563 learned_addr learned my addr 172.21.15.114:0/3003742563 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.387+0000 7fdf0ad9d700 1 -- 172.21.15.114:0/3003742563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf0c1a06d0 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.387+0000 7fdf0ad9d700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 0x7fdf0c1a0190 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009520 tx=0x7fdefc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.387+0000 7fdef3fff700 1 -- 172.21.15.114:0/3003742563 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdefc0039e0 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.387+0000 7fdef3fff700 1 -- 172.21.15.114:0/3003742563 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdefc01b8f0 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.387+0000 7fdef3fff700 1 -- 172.21.15.114:0/3003742563 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdefc0197d0 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.387+0000 7fdf1100a700 1 -- 172.21.15.114:0/3003742563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf0c1a0960 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.387+0000 7fdf1100a700 1 -- 172.21.15.114:0/3003742563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf0c1a3570 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.389+0000 7fdef3fff700 1 -- 172.21.15.114:0/3003742563 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7fdefc019a40 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.389+0000 7fdef3fff700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 0x7fdef4040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.389+0000 7fdef3fff700 1 -- 172.21.15.114:0/3003742563 --> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdef4041140 con 0x7fdef403e520 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.389+0000 7fdef3fff700 1 -- 172.21.15.114:0/3003742563 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdefc053f40 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.389+0000 7fdf0a59c700 1 -- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 msgr2=0x7fdef4040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/752776783 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.389+0000 7fdf0a59c700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 0x7fdef4040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:28:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.590+0000 7fdf0a59c700 1 -- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 msgr2=0x7fdef4040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/752776783 2024-05-27T07:28:44.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.590+0000 7fdf0a59c700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 0x7fdef4040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:28:44.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.991+0000 7fdf0a59c700 1 -- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 msgr2=0x7fdef4040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/752776783 2024-05-27T07:28:44.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:38.991+0000 7fdf0a59c700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 0x7fdef4040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:28:44.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:39.792+0000 7fdf0a59c700 1 -- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 msgr2=0x7fdef4040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/752776783 2024-05-27T07:28:44.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:39.792+0000 7fdf0a59c700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 0x7fdef4040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:28:44.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:41.392+0000 7fdf0a59c700 1 -- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 msgr2=0x7fdef4040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/752776783 2024-05-27T07:28:44.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:41.392+0000 7fdf0a59c700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 0x7fdef4040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-27T07:28:44.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:43.273+0000 7fdef3fff700 1 -- 172.21.15.114:0/3003742563 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mgrmap(e 10) ==== 50002+0+0 (secure 0 0 0) 0x7fdefc024cc0 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:43.273+0000 7fdef3fff700 1 -- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 msgr2=0x7fdef4040a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:28:44.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:43.273+0000 7fdef3fff700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 0x7fdef4040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:44.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.274+0000 7fdef3fff700 1 -- 172.21.15.114:0/3003742563 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 11) ==== 50129+0+0 (secure 0 0 0) 0x7fdefc055eb0 con 0x7fdf0c10a0a0 2024-05-27T07:28:44.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.274+0000 7fdef3fff700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fdefc053ad0 0x7fdef4044000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.274+0000 7fdef3fff700 1 -- 172.21.15.114:0/3003742563 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdefc012ea0 con 0x7fdefc053ad0 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.274+0000 7fdf0a59c700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fdefc053ad0 0x7fdef4044000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.275+0000 7fdf0a59c700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fdefc053ad0 0x7fdef4044000 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdf00003a10 tx=0x7fdf00009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.276+0000 7fdef3fff700 1 -- 172.21.15.114:0/3003742563 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fdefc012ea0 con 0x7fdefc053ad0 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.284+0000 7fdf1100a700 1 -- 172.21.15.114:0/3003742563 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fdf0c064e10 con 0x7fdefc053ad0 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.285+0000 7fdef3fff700 1 -- 172.21.15.114:0/3003742563 <== mgr.14152 v2:172.21.15.114:6800/2291864634 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fdf0c064e10 con 0x7fdefc053ad0 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.285+0000 7fdf1100a700 1 -- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fdefc053ad0 msgr2=0x7fdef4044000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.285+0000 7fdf1100a700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fdefc053ad0 0x7fdef4044000 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdf00003a10 tx=0x7fdf00009270 comp rx=0 tx=0).stop 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.285+0000 7fdf1100a700 1 -- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 msgr2=0x7fdf0c1a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.285+0000 7fdf1100a700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 0x7fdf0c1a0190 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009520 tx=0x7fdefc011aa0 comp rx=0 tx=0).stop 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.285+0000 7fdf1100a700 1 -- 172.21.15.114:0/3003742563 shutdown_connections 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.285+0000 7fdf1100a700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fdefc053ad0 0x7fdef4044000 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.285+0000 7fdf1100a700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c10a0a0 0x7fdf0c1a0190 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.285+0000 7fdf1100a700 1 --2- 172.21.15.114:0/3003742563 >> [v2:172.21.15.114:6800/752776783,v1:172.21.15.114:6801/752776783] conn(0x7fdef403e520 0x7fdef4040a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:44.926 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.285+0000 7fdf1100a700 1 -- 172.21.15.114:0/3003742563 >> 172.21.15.114:0/3003742563 conn(0x7fdf0c1038b0 msgr2=0x7fdf0c104220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:44.926 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.285+0000 7fdf1100a700 1 -- 172.21.15.114:0/3003742563 shutdown_connections 2024-05-27T07:28:44.926 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:44.285+0000 7fdf1100a700 1 -- 172.21.15.114:0/3003742563 wait complete. 2024-05-27T07:28:44.926 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 9 is available 2024-05-27T07:28:44.926 INFO:teuthology.orchestra.run.smithi114.stdout:Generating a dashboard self-signed certificate... 2024-05-27T07:28:45.574 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:45 smithi114 ceph-mon[32038]: [27/May/2024:07:28:44] ENGINE Bus STARTING 2024-05-27T07:28:45.574 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:45 smithi114 ceph-mon[32038]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-27T07:28:45.574 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:45 smithi114 ceph-mon[32038]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-27T07:28:45.574 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:45 smithi114 ceph-mon[32038]: [27/May/2024:07:28:44] ENGINE Serving on https://172.21.15.114:7150 2024-05-27T07:28:45.574 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:45 smithi114 ceph-mon[32038]: [27/May/2024:07:28:44] ENGINE Serving on http://172.21.15.114:8765 2024-05-27T07:28:45.574 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:45 smithi114 ceph-mon[32038]: [27/May/2024:07:28:44] ENGINE Bus STARTED 2024-05-27T07:28:45.574 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:28:46.644 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:46 smithi114 ceph-mon[32038]: mgrmap e12: x(active, since 2s) 2024-05-27T07:28:46.644 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:46 smithi114 ceph-mon[32038]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:28:46.644 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:28:46.644 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:28:46.651 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-27T07:28:46.651 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.697+0000 7f3cbc1c8700 1 Processor -- start 2024-05-27T07:28:46.651 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.697+0000 7f3cbc1c8700 1 -- start start 2024-05-27T07:28:46.651 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.698+0000 7f3cbc1c8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 0x7f3cb410c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:46.651 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.698+0000 7f3cbc1c8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb410cd20 con 0x7f3cb407d720 2024-05-27T07:28:46.651 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.698+0000 7f3cb9f64700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 0x7f3cb410c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:46.652 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.698+0000 7f3cb9f64700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 0x7f3cb410c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34902/0 (socket says 172.21.15.114:34902) 2024-05-27T07:28:46.652 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.698+0000 7f3cb9f64700 1 -- 172.21.15.114:0/4170388207 learned_addr learned my addr 172.21.15.114:0/4170388207 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:46.652 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.698+0000 7f3cb9f64700 1 -- 172.21.15.114:0/4170388207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb410cea0 con 0x7f3cb407d720 2024-05-27T07:28:46.652 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.698+0000 7f3cb9f64700 1 --2- 172.21.15.114:0/4170388207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 0x7f3cb410c710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00098a0 tx=0x7f3cb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e24be9a4429f1cc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:46.652 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.699+0000 7f3cb8f62700 1 -- 172.21.15.114:0/4170388207 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cb0004160 con 0x7f3cb407d720 2024-05-27T07:28:46.652 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.699+0000 7f3cb8f62700 1 -- 172.21.15.114:0/4170388207 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cb0004300 con 0x7f3cb407d720 2024-05-27T07:28:46.652 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.699+0000 7f3cb8f62700 1 -- 172.21.15.114:0/4170388207 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cb0011a20 con 0x7f3cb407d720 2024-05-27T07:28:46.652 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.700+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/4170388207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 msgr2=0x7f3cb410c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:46.652 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.700+0000 7f3cbc1c8700 1 --2- 172.21.15.114:0/4170388207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 0x7f3cb410c710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00098a0 tx=0x7f3cb0009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:46.652 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.700+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/4170388207 shutdown_connections 2024-05-27T07:28:46.652 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.700+0000 7f3cbc1c8700 1 --2- 172.21.15.114:0/4170388207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 0x7f3cb410c710 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:46.652 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.700+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/4170388207 >> 172.21.15.114:0/4170388207 conn(0x7f3cb4104160 msgr2=0x7f3cb41065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:46.652 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.700+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/4170388207 shutdown_connections 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.700+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/4170388207 wait complete. 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.700+0000 7f3cbc1c8700 1 Processor -- start 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.701+0000 7f3cbc1c8700 1 -- start start 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.701+0000 7f3cbc1c8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 0x7f3cb4109e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.701+0000 7f3cbc1c8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb410df60 con 0x7f3cb407d720 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.701+0000 7f3cb9f64700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 0x7f3cb4109e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.701+0000 7f3cb9f64700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 0x7f3cb4109e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34912/0 (socket says 172.21.15.114:34912) 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.701+0000 7f3cb9f64700 1 -- 172.21.15.114:0/1683110396 learned_addr learned my addr 172.21.15.114:0/1683110396 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.701+0000 7f3cb9f64700 1 -- 172.21.15.114:0/1683110396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb410a3c0 con 0x7f3cb407d720 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.701+0000 7f3cb9f64700 1 --2- 172.21.15.114:0/1683110396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 0x7f3cb4109e80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0004660 tx=0x7f3cb0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.701+0000 7f3caaffd700 1 -- 172.21.15.114:0/1683110396 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cb0003a60 con 0x7f3cb407d720 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.702+0000 7f3caaffd700 1 -- 172.21.15.114:0/1683110396 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cb001b8f0 con 0x7f3cb407d720 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.702+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/1683110396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cb4108610 con 0x7f3cb407d720 2024-05-27T07:28:46.653 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.702+0000 7f3caaffd700 1 -- 172.21.15.114:0/1683110396 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cb0023890 con 0x7f3cb407d720 2024-05-27T07:28:46.654 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.702+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/1683110396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cb4108af0 con 0x7f3cb407d720 2024-05-27T07:28:46.654 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.703+0000 7f3caaffd700 1 -- 172.21.15.114:0/1683110396 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f3cb002c3f0 con 0x7f3cb407d720 2024-05-27T07:28:46.654 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.703+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/1683110396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cb4064e10 con 0x7f3cb407d720 2024-05-27T07:28:46.654 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.703+0000 7f3caaffd700 1 --2- 172.21.15.114:0/1683110396 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3ca003e470 0x7f3ca0040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:46.654 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.703+0000 7f3caaffd700 1 -- 172.21.15.114:0/1683110396 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3cb0054090 con 0x7f3cb407d720 2024-05-27T07:28:46.654 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.703+0000 7f3cb9763700 1 --2- 172.21.15.114:0/1683110396 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3ca003e470 0x7f3ca0040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:46.654 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.704+0000 7f3cb9763700 1 --2- 172.21.15.114:0/1683110396 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3ca003e470 0x7f3ca0040980 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4006fd0 tx=0x7f3ca4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:46.654 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.709+0000 7f3caaffd700 1 -- 172.21.15.114:0/1683110396 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3cb001f030 con 0x7f3cb407d720 2024-05-27T07:28:46.654 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:45.930+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/1683110396 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f3cb4109220 con 0x7f3ca003e470 2024-05-27T07:28:46.654 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:46.037+0000 7f3caaffd700 1 -- 172.21.15.114:0/1683110396 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f3cb4109220 con 0x7f3ca003e470 2024-05-27T07:28:46.654 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:46.039+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/1683110396 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3ca003e470 msgr2=0x7f3ca0040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:46.654 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:46.039+0000 7f3cbc1c8700 1 --2- 172.21.15.114:0/1683110396 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3ca003e470 0x7f3ca0040980 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4006fd0 tx=0x7f3ca4006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:46.655 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:46.039+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/1683110396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 msgr2=0x7f3cb4109e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:46.655 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:46.039+0000 7f3cbc1c8700 1 --2- 172.21.15.114:0/1683110396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 0x7f3cb4109e80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0004660 tx=0x7f3cb0011aa0 comp rx=0 tx=0).stop 2024-05-27T07:28:46.655 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:46.039+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/1683110396 shutdown_connections 2024-05-27T07:28:46.655 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:46.039+0000 7f3cbc1c8700 1 --2- 172.21.15.114:0/1683110396 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3ca003e470 0x7f3ca0040980 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:46.655 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:46.039+0000 7f3cbc1c8700 1 --2- 172.21.15.114:0/1683110396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb407d720 0x7f3cb4109e80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:46.655 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:46.039+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/1683110396 >> 172.21.15.114:0/1683110396 conn(0x7f3cb4104160 msgr2=0x7f3cb4104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:46.655 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:46.039+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/1683110396 shutdown_connections 2024-05-27T07:28:46.655 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:46.039+0000 7f3cbc1c8700 1 -- 172.21.15.114:0/1683110396 wait complete. 2024-05-27T07:28:46.655 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial admin user... 2024-05-27T07:28:48.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:28:48.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:28:48.270 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$op3DxnSmRClx3eMyriz0vehSQWCnEm4PB5YLungqeIajhIY9FTz7y", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1716794927, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-27T07:28:48.271 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.130+0000 7fbd0f154700 1 Processor -- start 2024-05-27T07:28:48.271 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.130+0000 7fbd0f154700 1 -- start start 2024-05-27T07:28:48.271 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.131+0000 7fbd0f154700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 0x7fbd08108eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:48.271 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.131+0000 7fbd0f154700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbd081094a0 con 0x7fbd08108aa0 2024-05-27T07:28:48.271 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.131+0000 7fbd0cef0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 0x7fbd08108eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:48.271 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.131+0000 7fbd0cef0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 0x7fbd08108eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34918/0 (socket says 172.21.15.114:34918) 2024-05-27T07:28:48.272 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.131+0000 7fbd0cef0700 1 -- 172.21.15.114:0/880902708 learned_addr learned my addr 172.21.15.114:0/880902708 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:48.272 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.131+0000 7fbd0cef0700 1 -- 172.21.15.114:0/880902708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd08109bd0 con 0x7fbd08108aa0 2024-05-27T07:28:48.272 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.131+0000 7fbd0cef0700 1 --2- 172.21.15.114:0/880902708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 0x7fbd08108eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0098a0 tx=0x7fbcfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcecc07a82f1c603 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:48.272 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.131+0000 7fbd077fe700 1 -- 172.21.15.114:0/880902708 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcfc004160 con 0x7fbd08108aa0 2024-05-27T07:28:48.272 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.131+0000 7fbd077fe700 1 -- 172.21.15.114:0/880902708 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcfc004300 con 0x7fbd08108aa0 2024-05-27T07:28:48.272 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.131+0000 7fbd077fe700 1 -- 172.21.15.114:0/880902708 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcfc011a20 con 0x7fbd08108aa0 2024-05-27T07:28:48.273 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.132+0000 7fbd0f154700 1 -- 172.21.15.114:0/880902708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 msgr2=0x7fbd08108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:48.273 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.132+0000 7fbd0f154700 1 --2- 172.21.15.114:0/880902708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 0x7fbd08108eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0098a0 tx=0x7fbcfc009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:48.273 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.132+0000 7fbd0f154700 1 -- 172.21.15.114:0/880902708 shutdown_connections 2024-05-27T07:28:48.273 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.132+0000 7fbd0f154700 1 --2- 172.21.15.114:0/880902708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 0x7fbd08108eb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:48.273 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.132+0000 7fbd0f154700 1 -- 172.21.15.114:0/880902708 >> 172.21.15.114:0/880902708 conn(0x7fbd08104120 msgr2=0x7fbd08106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:48.273 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.132+0000 7fbd0f154700 1 -- 172.21.15.114:0/880902708 shutdown_connections 2024-05-27T07:28:48.274 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.132+0000 7fbd0f154700 1 -- 172.21.15.114:0/880902708 wait complete. 2024-05-27T07:28:48.274 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.133+0000 7fbd0f154700 1 Processor -- start 2024-05-27T07:28:48.274 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.133+0000 7fbd0f154700 1 -- start start 2024-05-27T07:28:48.274 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.133+0000 7fbd0f154700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 0x7fbd08106300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:48.274 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.133+0000 7fbd0f154700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0810cf80 con 0x7fbd08108aa0 2024-05-27T07:28:48.274 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.133+0000 7fbd0cef0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 0x7fbd08106300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:48.275 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.133+0000 7fbd0cef0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 0x7fbd08106300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34920/0 (socket says 172.21.15.114:34920) 2024-05-27T07:28:48.275 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.133+0000 7fbd0cef0700 1 -- 172.21.15.114:0/3375322612 learned_addr learned my addr 172.21.15.114:0/3375322612 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:48.275 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.134+0000 7fbd0cef0700 1 -- 172.21.15.114:0/3375322612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd08106840 con 0x7fbd08108aa0 2024-05-27T07:28:48.275 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.134+0000 7fbd0cef0700 1 --2- 172.21.15.114:0/3375322612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 0x7fbd08106300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc011a20 tx=0x7fbcfc011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:48.275 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.134+0000 7fbd05ffb700 1 -- 172.21.15.114:0/3375322612 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcfc0039c0 con 0x7fbd08108aa0 2024-05-27T07:28:48.275 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.134+0000 7fbd05ffb700 1 -- 172.21.15.114:0/3375322612 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcfc003b60 con 0x7fbd08108aa0 2024-05-27T07:28:48.276 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.134+0000 7fbd0f154700 1 -- 172.21.15.114:0/3375322612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd08104a90 con 0x7fbd08108aa0 2024-05-27T07:28:48.276 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.134+0000 7fbd05ffb700 1 -- 172.21.15.114:0/3375322612 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcfc019700 con 0x7fbd08108aa0 2024-05-27T07:28:48.276 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.134+0000 7fbd0f154700 1 -- 172.21.15.114:0/3375322612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd08104f70 con 0x7fbd08108aa0 2024-05-27T07:28:48.276 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.136+0000 7fbd05ffb700 1 -- 172.21.15.114:0/3375322612 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7fbcfc0199b0 con 0x7fbd08108aa0 2024-05-27T07:28:48.276 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.136+0000 7fbd0f154700 1 -- 172.21.15.114:0/3375322612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd08064e10 con 0x7fbd08108aa0 2024-05-27T07:28:48.276 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.136+0000 7fbd05ffb700 1 --2- 172.21.15.114:0/3375322612 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fbcf803e4d0 0x7fbcf80409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:48.277 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.136+0000 7fbd05ffb700 1 -- 172.21.15.114:0/3375322612 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbcfc053e70 con 0x7fbd08108aa0 2024-05-27T07:28:48.277 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.136+0000 7fbd07fff700 1 --2- 172.21.15.114:0/3375322612 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fbcf803e4d0 0x7fbcf80409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:48.277 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.136+0000 7fbd07fff700 1 --2- 172.21.15.114:0/3375322612 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fbcf803e4d0 0x7fbcf80409e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4006fd0 tx=0x7fbcf4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:48.277 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.141+0000 7fbd05ffb700 1 -- 172.21.15.114:0/3375322612 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbcfc020e20 con 0x7fbd08108aa0 2024-05-27T07:28:48.277 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.379+0000 7fbd0f154700 1 -- 172.21.15.114:0/3375322612 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fbd081970b0 con 0x7fbcf803e4d0 2024-05-27T07:28:48.277 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.604+0000 7fbd05ffb700 1 -- 172.21.15.114:0/3375322612 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fbd081970b0 con 0x7fbcf803e4d0 2024-05-27T07:28:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.606+0000 7fbd0f154700 1 -- 172.21.15.114:0/3375322612 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fbcf803e4d0 msgr2=0x7fbcf80409e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.606+0000 7fbd0f154700 1 --2- 172.21.15.114:0/3375322612 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fbcf803e4d0 0x7fbcf80409e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4006fd0 tx=0x7fbcf4006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.606+0000 7fbd0f154700 1 -- 172.21.15.114:0/3375322612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 msgr2=0x7fbd08106300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.606+0000 7fbd0f154700 1 --2- 172.21.15.114:0/3375322612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 0x7fbd08106300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc011a20 tx=0x7fbcfc011b00 comp rx=0 tx=0).stop 2024-05-27T07:28:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.607+0000 7fbd0f154700 1 -- 172.21.15.114:0/3375322612 shutdown_connections 2024-05-27T07:28:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.607+0000 7fbd0f154700 1 --2- 172.21.15.114:0/3375322612 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fbcf803e4d0 0x7fbcf80409e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:48.279 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.607+0000 7fbd0f154700 1 --2- 172.21.15.114:0/3375322612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd08108aa0 0x7fbd08106300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:48.279 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.607+0000 7fbd0f154700 1 -- 172.21.15.114:0/3375322612 >> 172.21.15.114:0/3375322612 conn(0x7fbd08104120 msgr2=0x7fbd08194520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:48.279 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.607+0000 7fbd0f154700 1 -- 172.21.15.114:0/3375322612 shutdown_connections 2024-05-27T07:28:48.279 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:47.607+0000 7fbd0f154700 1 -- 172.21.15.114:0/3375322612 wait complete. 2024-05-27T07:28:48.279 INFO:teuthology.orchestra.run.smithi114.stdout:Fetching dashboard port number... 2024-05-27T07:28:48.795 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:48 smithi114 ceph-mon[32038]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:28:50.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:49 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/399559049' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-27T07:28:50.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 8443 2024-05-27T07:28:50.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.174+0000 7f98eb7e1700 1 Processor -- start 2024-05-27T07:28:50.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.174+0000 7f98eb7e1700 1 -- start start 2024-05-27T07:28:50.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.174+0000 7f98eb7e1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 0x7f98e4108f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.174+0000 7f98eb7e1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f98e41094f0 con 0x7f98e4108af0 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.174+0000 7f98e957d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 0x7f98e4108f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.174+0000 7f98e957d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 0x7f98e4108f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34936/0 (socket says 172.21.15.114:34936) 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.174+0000 7f98e957d700 1 -- 172.21.15.114:0/712558726 learned_addr learned my addr 172.21.15.114:0/712558726 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.175+0000 7f98e957d700 1 -- 172.21.15.114:0/712558726 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e4109c20 con 0x7f98e4108af0 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.175+0000 7f98e957d700 1 --2- 172.21.15.114:0/712558726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 0x7f98e4108f00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f98e00098a0 tx=0x7f98e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9fb24d8c6778448 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.175+0000 7f98dbfff700 1 -- 172.21.15.114:0/712558726 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e0004160 con 0x7f98e4108af0 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.175+0000 7f98dbfff700 1 -- 172.21.15.114:0/712558726 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98e0004300 con 0x7f98e4108af0 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.175+0000 7f98dbfff700 1 -- 172.21.15.114:0/712558726 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e0011a20 con 0x7f98e4108af0 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.176+0000 7f98eb7e1700 1 -- 172.21.15.114:0/712558726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 msgr2=0x7f98e4108f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.176+0000 7f98eb7e1700 1 --2- 172.21.15.114:0/712558726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 0x7f98e4108f00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f98e00098a0 tx=0x7f98e0009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.176+0000 7f98eb7e1700 1 -- 172.21.15.114:0/712558726 shutdown_connections 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.176+0000 7f98eb7e1700 1 --2- 172.21.15.114:0/712558726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 0x7f98e4108f00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.176+0000 7f98eb7e1700 1 -- 172.21.15.114:0/712558726 >> 172.21.15.114:0/712558726 conn(0x7f98e41041b0 msgr2=0x7f98e4106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:50.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.176+0000 7f98eb7e1700 1 -- 172.21.15.114:0/712558726 shutdown_connections 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.176+0000 7f98eb7e1700 1 -- 172.21.15.114:0/712558726 wait complete. 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.177+0000 7f98eb7e1700 1 Processor -- start 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.177+0000 7f98eb7e1700 1 -- start start 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.177+0000 7f98eb7e1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 0x7f98e419df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.177+0000 7f98eb7e1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f98e410cfd0 con 0x7f98e4108af0 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.177+0000 7f98e957d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 0x7f98e419df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.177+0000 7f98e957d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 0x7f98e419df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34938/0 (socket says 172.21.15.114:34938) 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.177+0000 7f98e957d700 1 -- 172.21.15.114:0/399559049 learned_addr learned my addr 172.21.15.114:0/399559049 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.177+0000 7f98e957d700 1 -- 172.21.15.114:0/399559049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e419e490 con 0x7f98e4108af0 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.177+0000 7f98e957d700 1 --2- 172.21.15.114:0/399559049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 0x7f98e419df50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f98e0003e70 tx=0x7f98e0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.178+0000 7f98da7fc700 1 -- 172.21.15.114:0/399559049 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e0003af0 con 0x7f98e4108af0 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.178+0000 7f98eb7e1700 1 -- 172.21.15.114:0/399559049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e419e720 con 0x7f98e4108af0 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.178+0000 7f98da7fc700 1 -- 172.21.15.114:0/399559049 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98e001a8e0 con 0x7f98e4108af0 2024-05-27T07:28:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.178+0000 7f98da7fc700 1 -- 172.21.15.114:0/399559049 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e0024870 con 0x7f98e4108af0 2024-05-27T07:28:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.178+0000 7f98eb7e1700 1 -- 172.21.15.114:0/399559049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e41a1330 con 0x7f98e4108af0 2024-05-27T07:28:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.178+0000 7f98eb7e1700 1 -- 172.21.15.114:0/399559049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98e4064e10 con 0x7f98e4108af0 2024-05-27T07:28:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.179+0000 7f98da7fc700 1 -- 172.21.15.114:0/399559049 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f98e0022020 con 0x7f98e4108af0 2024-05-27T07:28:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.179+0000 7f98da7fc700 1 --2- 172.21.15.114:0/399559049 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f98d003e470 0x7f98d0040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.179+0000 7f98da7fc700 1 -- 172.21.15.114:0/399559049 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f98e0053a50 con 0x7f98e4108af0 2024-05-27T07:28:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.179+0000 7f98e8d7c700 1 --2- 172.21.15.114:0/399559049 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f98d003e470 0x7f98d0040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.179+0000 7f98e8d7c700 1 --2- 172.21.15.114:0/399559049 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f98d003e470 0x7f98d0040980 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f98d4006fd0 tx=0x7f98d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.184+0000 7f98da7fc700 1 -- 172.21.15.114:0/399559049 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f98e000e260 con 0x7f98e4108af0 2024-05-27T07:28:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.388+0000 7f98eb7e1700 1 -- 172.21.15.114:0/399559049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f98e402e8b0 con 0x7f98e4108af0 2024-05-27T07:28:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.389+0000 7f98da7fc700 1 -- 172.21.15.114:0/399559049 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f98e001ad90 con 0x7f98e4108af0 2024-05-27T07:28:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.391+0000 7f98eb7e1700 1 -- 172.21.15.114:0/399559049 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f98d003e470 msgr2=0x7f98d0040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.391+0000 7f98eb7e1700 1 --2- 172.21.15.114:0/399559049 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f98d003e470 0x7f98d0040980 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f98d4006fd0 tx=0x7f98d4006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.391+0000 7f98eb7e1700 1 -- 172.21.15.114:0/399559049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 msgr2=0x7f98e419df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:50.159 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.391+0000 7f98eb7e1700 1 --2- 172.21.15.114:0/399559049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 0x7f98e419df50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f98e0003e70 tx=0x7f98e0011aa0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.159 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.391+0000 7f98eb7e1700 1 -- 172.21.15.114:0/399559049 shutdown_connections 2024-05-27T07:28:50.159 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.391+0000 7f98eb7e1700 1 --2- 172.21.15.114:0/399559049 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f98d003e470 0x7f98d0040980 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.159 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.391+0000 7f98eb7e1700 1 --2- 172.21.15.114:0/399559049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98e4108af0 0x7f98e419df50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.159 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.391+0000 7f98eb7e1700 1 -- 172.21.15.114:0/399559049 >> 172.21.15.114:0/399559049 conn(0x7f98e41041b0 msgr2=0x7f98e4194500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:50.159 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.391+0000 7f98eb7e1700 1 -- 172.21.15.114:0/399559049 shutdown_connections 2024-05-27T07:28:50.159 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:49.391+0000 7f98eb7e1700 1 -- 172.21.15.114:0/399559049 wait complete. 2024-05-27T07:28:50.162 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T07:28:50.162 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-05-27T07:28:50.170 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T07:28:50.170 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-05-27T07:28:50.170 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-05-27T07:28:50.170 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-27T07:28:50.172 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph Dashboard is now available at: 2024-05-27T07:28:50.172 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:28:50.172 INFO:teuthology.orchestra.run.smithi114.stdout: URL: https://smithi114.front.sepia.ceph.com:8443/ 2024-05-27T07:28:50.172 INFO:teuthology.orchestra.run.smithi114.stdout: User: admin 2024-05-27T07:28:50.172 INFO:teuthology.orchestra.run.smithi114.stdout: Password: pon0axi9c5 2024-05-27T07:28:50.172 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:28:50.173 INFO:teuthology.orchestra.run.smithi114.stdout:Saving cluster configuration to /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config directory 2024-05-27T07:28:50.173 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling autotune for osd_memory_target 2024-05-27T07:28:51.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:50 smithi114 ceph-mon[32038]: mgrmap e13: x(active, since 6s) 2024-05-27T07:28:51.398 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.630+0000 7f1b75cc5700 1 Processor -- start 2024-05-27T07:28:51.404 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.630+0000 7f1b75cc5700 1 -- start start 2024-05-27T07:28:51.404 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.631+0000 7f1b75cc5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 0x7f1b70108eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:51.404 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.631+0000 7f1b75cc5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1b701094a0 con 0x7f1b70108aa0 2024-05-27T07:28:51.404 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.631+0000 7f1b6f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 0x7f1b70108eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:51.404 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.631+0000 7f1b6f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 0x7f1b70108eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34948/0 (socket says 172.21.15.114:34948) 2024-05-27T07:28:51.404 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.631+0000 7f1b6f7fe700 1 -- 172.21.15.114:0/1231800557 learned_addr learned my addr 172.21.15.114:0/1231800557 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:51.404 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.631+0000 7f1b6f7fe700 1 -- 172.21.15.114:0/1231800557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b70109bd0 con 0x7f1b70108aa0 2024-05-27T07:28:51.404 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.631+0000 7f1b6f7fe700 1 --2- 172.21.15.114:0/1231800557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 0x7f1b70108eb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1b600098a0 tx=0x7f1b60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbf382e09082336 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.632+0000 7f1b6e7fc700 1 -- 172.21.15.114:0/1231800557 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b60004160 con 0x7f1b70108aa0 2024-05-27T07:28:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.632+0000 7f1b6e7fc700 1 -- 172.21.15.114:0/1231800557 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b60004300 con 0x7f1b70108aa0 2024-05-27T07:28:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.632+0000 7f1b6e7fc700 1 -- 172.21.15.114:0/1231800557 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b60011a20 con 0x7f1b70108aa0 2024-05-27T07:28:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.632+0000 7f1b75cc5700 1 -- 172.21.15.114:0/1231800557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 msgr2=0x7f1b70108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.632+0000 7f1b75cc5700 1 --2- 172.21.15.114:0/1231800557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 0x7f1b70108eb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1b600098a0 tx=0x7f1b60009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.632+0000 7f1b75cc5700 1 -- 172.21.15.114:0/1231800557 shutdown_connections 2024-05-27T07:28:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.632+0000 7f1b75cc5700 1 --2- 172.21.15.114:0/1231800557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 0x7f1b70108eb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.632+0000 7f1b75cc5700 1 -- 172.21.15.114:0/1231800557 >> 172.21.15.114:0/1231800557 conn(0x7f1b70104120 msgr2=0x7f1b70106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.633+0000 7f1b75cc5700 1 -- 172.21.15.114:0/1231800557 shutdown_connections 2024-05-27T07:28:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.633+0000 7f1b75cc5700 1 -- 172.21.15.114:0/1231800557 wait complete. 2024-05-27T07:28:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.633+0000 7f1b75cc5700 1 Processor -- start 2024-05-27T07:28:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.633+0000 7f1b75cc5700 1 -- start start 2024-05-27T07:28:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.634+0000 7f1b75cc5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 0x7f1b701062c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:51.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.634+0000 7f1b75cc5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7010cf80 con 0x7f1b70108aa0 2024-05-27T07:28:51.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.634+0000 7f1b6f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 0x7f1b701062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:51.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.634+0000 7f1b6f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 0x7f1b701062c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34964/0 (socket says 172.21.15.114:34964) 2024-05-27T07:28:51.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.634+0000 7f1b6f7fe700 1 -- 172.21.15.114:0/4190939954 learned_addr learned my addr 172.21.15.114:0/4190939954 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:51.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.634+0000 7f1b6f7fe700 1 -- 172.21.15.114:0/4190939954 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b70106800 con 0x7f1b70108aa0 2024-05-27T07:28:51.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.634+0000 7f1b6f7fe700 1 --2- 172.21.15.114:0/4190939954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 0x7f1b701062c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b60011a20 tx=0x7f1b60011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:51.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.634+0000 7f1b6cff9700 1 -- 172.21.15.114:0/4190939954 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b600039c0 con 0x7f1b70108aa0 2024-05-27T07:28:51.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.634+0000 7f1b75cc5700 1 -- 172.21.15.114:0/4190939954 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b70104a50 con 0x7f1b70108aa0 2024-05-27T07:28:51.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.634+0000 7f1b6cff9700 1 -- 172.21.15.114:0/4190939954 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b60003b60 con 0x7f1b70108aa0 2024-05-27T07:28:51.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.634+0000 7f1b75cc5700 1 -- 172.21.15.114:0/4190939954 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b70104f30 con 0x7f1b70108aa0 2024-05-27T07:28:51.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.634+0000 7f1b6cff9700 1 -- 172.21.15.114:0/4190939954 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b60019700 con 0x7f1b70108aa0 2024-05-27T07:28:51.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.635+0000 7f1b75cc5700 1 -- 172.21.15.114:0/4190939954 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b70064e10 con 0x7f1b70108aa0 2024-05-27T07:28:51.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.635+0000 7f1b6cff9700 1 -- 172.21.15.114:0/4190939954 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f1b60019a00 con 0x7f1b70108aa0 2024-05-27T07:28:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.636+0000 7f1b6cff9700 1 --2- 172.21.15.114:0/4190939954 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f1b5803e510 0x7f1b58040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.636+0000 7f1b6cff9700 1 -- 172.21.15.114:0/4190939954 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1b60053f30 con 0x7f1b70108aa0 2024-05-27T07:28:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.636+0000 7f1b6effd700 1 --2- 172.21.15.114:0/4190939954 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f1b5803e510 0x7f1b58040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.636+0000 7f1b6effd700 1 --2- 172.21.15.114:0/4190939954 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f1b5803e510 0x7f1b58040a20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1b64006fd0 tx=0x7f1b64006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.641+0000 7f1b6cff9700 1 -- 172.21.15.114:0/4190939954 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1b60017e90 con 0x7f1b70108aa0 2024-05-27T07:28:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.846+0000 7f1b75cc5700 1 -- 172.21.15.114:0/4190939954 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f1b70197130 con 0x7f1b70108aa0 2024-05-27T07:28:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.847+0000 7f1b6cff9700 1 -- 172.21.15.114:0/4190939954 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f1b60023390 con 0x7f1b70108aa0 2024-05-27T07:28:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.849+0000 7f1b75cc5700 1 -- 172.21.15.114:0/4190939954 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f1b5803e510 msgr2=0x7f1b58040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.849+0000 7f1b75cc5700 1 --2- 172.21.15.114:0/4190939954 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f1b5803e510 0x7f1b58040a20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1b64006fd0 tx=0x7f1b64006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.849+0000 7f1b75cc5700 1 -- 172.21.15.114:0/4190939954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 msgr2=0x7f1b701062c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.849+0000 7f1b75cc5700 1 --2- 172.21.15.114:0/4190939954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 0x7f1b701062c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b60011a20 tx=0x7f1b60011b00 comp rx=0 tx=0).stop 2024-05-27T07:28:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.849+0000 7f1b75cc5700 1 -- 172.21.15.114:0/4190939954 shutdown_connections 2024-05-27T07:28:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.849+0000 7f1b75cc5700 1 --2- 172.21.15.114:0/4190939954 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f1b5803e510 0x7f1b58040a20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:51.408 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.849+0000 7f1b75cc5700 1 --2- 172.21.15.114:0/4190939954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b70108aa0 0x7f1b701062c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:51.408 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.849+0000 7f1b75cc5700 1 -- 172.21.15.114:0/4190939954 >> 172.21.15.114:0/4190939954 conn(0x7f1b70104120 msgr2=0x7f1b70194520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:51.408 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.849+0000 7f1b75cc5700 1 -- 172.21.15.114:0/4190939954 shutdown_connections 2024-05-27T07:28:51.408 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:50.849+0000 7f1b75cc5700 1 -- 172.21.15.114:0/4190939954 wait complete. 2024-05-27T07:28:52.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.855+0000 7f10961e0700 1 Processor -- start 2024-05-27T07:28:52.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.855+0000 7f10961e0700 1 -- start start 2024-05-27T07:28:52.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.856+0000 7f10961e0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 0x7f109010b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.856+0000 7f10961e0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f109010bb10 con 0x7f109010b130 2024-05-27T07:28:52.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.856+0000 7f108f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 0x7f109010b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.856+0000 7f108f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 0x7f109010b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34978/0 (socket says 172.21.15.114:34978) 2024-05-27T07:28:52.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.856+0000 7f108f7fe700 1 -- 172.21.15.114:0/2643569329 learned_addr learned my addr 172.21.15.114:0/2643569329 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:52.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.856+0000 7f108f7fe700 1 -- 172.21.15.114:0/2643569329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109010bc90 con 0x7f109010b130 2024-05-27T07:28:52.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.856+0000 7f108f7fe700 1 --2- 172.21.15.114:0/2643569329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 0x7f109010b540 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f10780098a0 tx=0x7f1078009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c9bd40fa28fec09 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:52.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.856+0000 7f108e7fc700 1 -- 172.21.15.114:0/2643569329 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1078004160 con 0x7f109010b130 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.856+0000 7f108e7fc700 1 -- 172.21.15.114:0/2643569329 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1078004300 con 0x7f109010b130 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.856+0000 7f108e7fc700 1 -- 172.21.15.114:0/2643569329 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1078011a20 con 0x7f109010b130 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.857+0000 7f10961e0700 1 -- 172.21.15.114:0/2643569329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 msgr2=0x7f109010b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.857+0000 7f10961e0700 1 --2- 172.21.15.114:0/2643569329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 0x7f109010b540 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f10780098a0 tx=0x7f1078009bb0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.857+0000 7f10961e0700 1 -- 172.21.15.114:0/2643569329 shutdown_connections 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.857+0000 7f10961e0700 1 --2- 172.21.15.114:0/2643569329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 0x7f109010b540 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.857+0000 7f10961e0700 1 -- 172.21.15.114:0/2643569329 >> 172.21.15.114:0/2643569329 conn(0x7f10901038b0 msgr2=0x7f1090105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.857+0000 7f10961e0700 1 -- 172.21.15.114:0/2643569329 shutdown_connections 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.857+0000 7f10961e0700 1 -- 172.21.15.114:0/2643569329 wait complete. 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.858+0000 7f10961e0700 1 Processor -- start 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.858+0000 7f10961e0700 1 -- start start 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.858+0000 7f10961e0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 0x7f109019df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.858+0000 7f10961e0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f109010e0d0 con 0x7f109010b130 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.858+0000 7f108f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 0x7f109019df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.858+0000 7f108f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 0x7f109019df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34984/0 (socket says 172.21.15.114:34984) 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.858+0000 7f108f7fe700 1 -- 172.21.15.114:0/2159248180 learned_addr learned my addr 172.21.15.114:0/2159248180 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.859+0000 7f108f7fe700 1 -- 172.21.15.114:0/2159248180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109019e4a0 con 0x7f109010b130 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.859+0000 7f108f7fe700 1 --2- 172.21.15.114:0/2159248180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 0x7f109019df60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1078004660 tx=0x7f1078011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.859+0000 7f108cff9700 1 -- 172.21.15.114:0/2159248180 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1078003a60 con 0x7f109010b130 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.859+0000 7f108cff9700 1 -- 172.21.15.114:0/2159248180 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f107801b8f0 con 0x7f109010b130 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.859+0000 7f10961e0700 1 -- 172.21.15.114:0/2159248180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f109019e730 con 0x7f109010b130 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.859+0000 7f108cff9700 1 -- 172.21.15.114:0/2159248180 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1078023890 con 0x7f109010b130 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.859+0000 7f10961e0700 1 -- 172.21.15.114:0/2159248180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10901a1340 con 0x7f109010b130 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.860+0000 7f10961e0700 1 -- 172.21.15.114:0/2159248180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1090064e10 con 0x7f109010b130 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.860+0000 7f108cff9700 1 -- 172.21.15.114:0/2159248180 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f107802c3f0 con 0x7f109010b130 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.860+0000 7f108cff9700 1 --2- 172.21.15.114:0/2159248180 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f107c03e570 0x7f107c040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.860+0000 7f108cff9700 1 -- 172.21.15.114:0/2159248180 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1078054130 con 0x7f109010b130 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.860+0000 7f108effd700 1 --2- 172.21.15.114:0/2159248180 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f107c03e570 0x7f107c040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.861+0000 7f108effd700 1 --2- 172.21.15.114:0/2159248180 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f107c03e570 0x7f107c040a80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1080006fd0 tx=0x7f1080006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:52.737 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:51.866+0000 7f108cff9700 1 -- 172.21.15.114:0/2159248180 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f107801f030 con 0x7f109010b130 2024-05-27T07:28:52.738 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:52.177+0000 7f10961e0700 1 -- 172.21.15.114:0/2159248180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f10901a1650 con 0x7f109010b130 2024-05-27T07:28:52.738 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:52.179+0000 7f108cff9700 1 -- 172.21.15.114:0/2159248180 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f10780202b0 con 0x7f109010b130 2024-05-27T07:28:52.738 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:52.182+0000 7f10961e0700 1 -- 172.21.15.114:0/2159248180 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f107c03e570 msgr2=0x7f107c040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.738 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:52.182+0000 7f10961e0700 1 --2- 172.21.15.114:0/2159248180 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f107c03e570 0x7f107c040a80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1080006fd0 tx=0x7f1080006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:52.738 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:52.182+0000 7f10961e0700 1 -- 172.21.15.114:0/2159248180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 msgr2=0x7f109019df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.738 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:52.182+0000 7f10961e0700 1 --2- 172.21.15.114:0/2159248180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 0x7f109019df60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1078004660 tx=0x7f1078011aa0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.738 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:52.182+0000 7f10961e0700 1 -- 172.21.15.114:0/2159248180 shutdown_connections 2024-05-27T07:28:52.738 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:52.182+0000 7f10961e0700 1 --2- 172.21.15.114:0/2159248180 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f107c03e570 0x7f107c040a80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.738 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:52.182+0000 7f10961e0700 1 --2- 172.21.15.114:0/2159248180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f109010b130 0x7f109019df60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.738 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:52.182+0000 7f10961e0700 1 -- 172.21.15.114:0/2159248180 >> 172.21.15.114:0/2159248180 conn(0x7f10901038b0 msgr2=0x7f1090104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:52.738 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:52.182+0000 7f10961e0700 1 -- 172.21.15.114:0/2159248180 shutdown_connections 2024-05-27T07:28:52.738 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-05-27T07:28:52.182+0000 7f10961e0700 1 -- 172.21.15.114:0/2159248180 wait complete. 2024-05-27T07:28:52.738 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout:Or, if you are only running a single cluster on this host: 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout: ceph telemetry on 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout:For more information see: 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-27T07:28:52.739 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:28:52.740 INFO:teuthology.orchestra.run.smithi114.stdout:Bootstrap complete. 2024-05-27T07:28:52.771 INFO:tasks.cephadm:Fetching config... 2024-05-27T07:28:52.772 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:28:52.772 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-27T07:28:52.788 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-27T07:28:52.789 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:28:52.789 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-27T07:28:52.844 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-27T07:28:52.844 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:28:52.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-27T07:28:52.909 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-27T07:28:52.910 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:28:52.910 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-27T07:28:52.965 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-27T07:28:52.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEgQXBtxItCMdeMK1vaJ2BUmw/u/ATCmqfGUAGvHmapPUrvO2dMkcnps7orIoYJeQpi0DsMfAlnImqOexhZ/WUgw3PXz2vLg22XQAcA4jhdv9LnqsrBhPpWZR2hpr1VOv52BFJ1yuBfFyNK/yE32KgCA48v1NJpABq5EmlE6i3xH4bdqooCMDa/1j+5bU3GxtGK0lN4g+kJeAuX7dY9P5gpw4CPFZC7jVedo4rtzmgTWk7lbaYyY6fZVebxJKFMocDG6x/zlRHLXNhb+EjTITiiBdoZF7m8O4I1eeZk4w0cP611957wJuXSDjEn+H5Obwxgv0NukMXGelBHHjuOB4/V3PjaLjZkpGARbm5BjIpx1epxPDpwN6/ilE3txD/S6QFXaVoFew9a0+k7hO5F3EM1u2EQrAy5r9Up/+/jsj+VHZhGehySKIsUMmFqy2p8ARqvlROZ8LwvguXHmDEngkrx4RXNQniWUnlEvsuRMT2J/tkjp8Pu0uJc95zmoTB8Tk= ceph-755b3728-1bfa-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T07:28:53.039 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEgQXBtxItCMdeMK1vaJ2BUmw/u/ATCmqfGUAGvHmapPUrvO2dMkcnps7orIoYJeQpi0DsMfAlnImqOexhZ/WUgw3PXz2vLg22XQAcA4jhdv9LnqsrBhPpWZR2hpr1VOv52BFJ1yuBfFyNK/yE32KgCA48v1NJpABq5EmlE6i3xH4bdqooCMDa/1j+5bU3GxtGK0lN4g+kJeAuX7dY9P5gpw4CPFZC7jVedo4rtzmgTWk7lbaYyY6fZVebxJKFMocDG6x/zlRHLXNhb+EjTITiiBdoZF7m8O4I1eeZk4w0cP611957wJuXSDjEn+H5Obwxgv0NukMXGelBHHjuOB4/V3PjaLjZkpGARbm5BjIpx1epxPDpwN6/ilE3txD/S6QFXaVoFew9a0+k7hO5F3EM1u2EQrAy5r9Up/+/jsj+VHZhGehySKIsUMmFqy2p8ARqvlROZ8LwvguXHmDEngkrx4RXNQniWUnlEvsuRMT2J/tkjp8Pu0uJc95zmoTB8Tk= ceph-755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:28:53.051 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEgQXBtxItCMdeMK1vaJ2BUmw/u/ATCmqfGUAGvHmapPUrvO2dMkcnps7orIoYJeQpi0DsMfAlnImqOexhZ/WUgw3PXz2vLg22XQAcA4jhdv9LnqsrBhPpWZR2hpr1VOv52BFJ1yuBfFyNK/yE32KgCA48v1NJpABq5EmlE6i3xH4bdqooCMDa/1j+5bU3GxtGK0lN4g+kJeAuX7dY9P5gpw4CPFZC7jVedo4rtzmgTWk7lbaYyY6fZVebxJKFMocDG6x/zlRHLXNhb+EjTITiiBdoZF7m8O4I1eeZk4w0cP611957wJuXSDjEn+H5Obwxgv0NukMXGelBHHjuOB4/V3PjaLjZkpGARbm5BjIpx1epxPDpwN6/ilE3txD/S6QFXaVoFew9a0+k7hO5F3EM1u2EQrAy5r9Up/+/jsj+VHZhGehySKIsUMmFqy2p8ARqvlROZ8LwvguXHmDEngkrx4RXNQniWUnlEvsuRMT2J/tkjp8Pu0uJc95zmoTB8Tk= ceph-755b3728-1bfa-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T07:28:53.091 INFO:teuthology.orchestra.run.smithi163.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEgQXBtxItCMdeMK1vaJ2BUmw/u/ATCmqfGUAGvHmapPUrvO2dMkcnps7orIoYJeQpi0DsMfAlnImqOexhZ/WUgw3PXz2vLg22XQAcA4jhdv9LnqsrBhPpWZR2hpr1VOv52BFJ1yuBfFyNK/yE32KgCA48v1NJpABq5EmlE6i3xH4bdqooCMDa/1j+5bU3GxtGK0lN4g+kJeAuX7dY9P5gpw4CPFZC7jVedo4rtzmgTWk7lbaYyY6fZVebxJKFMocDG6x/zlRHLXNhb+EjTITiiBdoZF7m8O4I1eeZk4w0cP611957wJuXSDjEn+H5Obwxgv0NukMXGelBHHjuOB4/V3PjaLjZkpGARbm5BjIpx1epxPDpwN6/ilE3txD/S6QFXaVoFew9a0+k7hO5F3EM1u2EQrAy5r9Up/+/jsj+VHZhGehySKIsUMmFqy2p8ARqvlROZ8LwvguXHmDEngkrx4RXNQniWUnlEvsuRMT2J/tkjp8Pu0uJc95zmoTB8Tk= ceph-755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:28:53.102 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEgQXBtxItCMdeMK1vaJ2BUmw/u/ATCmqfGUAGvHmapPUrvO2dMkcnps7orIoYJeQpi0DsMfAlnImqOexhZ/WUgw3PXz2vLg22XQAcA4jhdv9LnqsrBhPpWZR2hpr1VOv52BFJ1yuBfFyNK/yE32KgCA48v1NJpABq5EmlE6i3xH4bdqooCMDa/1j+5bU3GxtGK0lN4g+kJeAuX7dY9P5gpw4CPFZC7jVedo4rtzmgTWk7lbaYyY6fZVebxJKFMocDG6x/zlRHLXNhb+EjTITiiBdoZF7m8O4I1eeZk4w0cP611957wJuXSDjEn+H5Obwxgv0NukMXGelBHHjuOB4/V3PjaLjZkpGARbm5BjIpx1epxPDpwN6/ilE3txD/S6QFXaVoFew9a0+k7hO5F3EM1u2EQrAy5r9Up/+/jsj+VHZhGehySKIsUMmFqy2p8ARqvlROZ8LwvguXHmDEngkrx4RXNQniWUnlEvsuRMT2J/tkjp8Pu0uJc95zmoTB8Tk= ceph-755b3728-1bfa-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T07:28:53.142 INFO:teuthology.orchestra.run.smithi186.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEgQXBtxItCMdeMK1vaJ2BUmw/u/ATCmqfGUAGvHmapPUrvO2dMkcnps7orIoYJeQpi0DsMfAlnImqOexhZ/WUgw3PXz2vLg22XQAcA4jhdv9LnqsrBhPpWZR2hpr1VOv52BFJ1yuBfFyNK/yE32KgCA48v1NJpABq5EmlE6i3xH4bdqooCMDa/1j+5bU3GxtGK0lN4g+kJeAuX7dY9P5gpw4CPFZC7jVedo4rtzmgTWk7lbaYyY6fZVebxJKFMocDG6x/zlRHLXNhb+EjTITiiBdoZF7m8O4I1eeZk4w0cP611957wJuXSDjEn+H5Obwxgv0NukMXGelBHHjuOB4/V3PjaLjZkpGARbm5BjIpx1epxPDpwN6/ilE3txD/S6QFXaVoFew9a0+k7hO5F3EM1u2EQrAy5r9Up/+/jsj+VHZhGehySKIsUMmFqy2p8ARqvlROZ8LwvguXHmDEngkrx4RXNQniWUnlEvsuRMT2J/tkjp8Pu0uJc95zmoTB8Tk= ceph-755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:28:53.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-27T07:28:53.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:53 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2159248180' entity='client.admin' 2024-05-27T07:28:53.402 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:55.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.603+0000 7f799bfdf700 1 -- 172.21.15.114:0/139241477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79941046c0 msgr2=0x7f7994106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:55.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.603+0000 7f799bfdf700 1 --2- 172.21.15.114:0/139241477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79941046c0 0x7f7994106ae0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f799000b0a0 tx=0x7f799000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:28:55.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.603+0000 7f799bfdf700 1 -- 172.21.15.114:0/139241477 shutdown_connections 2024-05-27T07:28:55.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.603+0000 7f799bfdf700 1 --2- 172.21.15.114:0/139241477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79941046c0 0x7f7994106ae0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:55.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.603+0000 7f799bfdf700 1 -- 172.21.15.114:0/139241477 >> 172.21.15.114:0/139241477 conn(0x7f79940fe120 msgr2=0x7f79941005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:55.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.603+0000 7f799bfdf700 1 -- 172.21.15.114:0/139241477 shutdown_connections 2024-05-27T07:28:55.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.603+0000 7f799bfdf700 1 -- 172.21.15.114:0/139241477 wait complete. 2024-05-27T07:28:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.604+0000 7f799bfdf700 1 Processor -- start 2024-05-27T07:28:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.604+0000 7f799bfdf700 1 -- start start 2024-05-27T07:28:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.604+0000 7f799bfdf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79941046c0 0x7f7994103e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.604+0000 7f799bfdf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7994109b80 con 0x7f79941046c0 2024-05-27T07:28:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.604+0000 7f7999d7b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79941046c0 0x7f7994103e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.604+0000 7f7999d7b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79941046c0 0x7f7994103e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41886/0 (socket says 172.21.15.114:41886) 2024-05-27T07:28:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.604+0000 7f7999d7b700 1 -- 172.21.15.114:0/2868806207 learned_addr learned my addr 172.21.15.114:0/2868806207 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.605+0000 7f7999d7b700 1 -- 172.21.15.114:0/2868806207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7994104380 con 0x7f79941046c0 2024-05-27T07:28:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.605+0000 7f7999d7b700 1 --2- 172.21.15.114:0/2868806207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79941046c0 0x7f7994103e40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7990015040 tx=0x7f7990003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.605+0000 7f798affd700 1 -- 172.21.15.114:0/2868806207 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7990021020 con 0x7f79941046c0 2024-05-27T07:28:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.605+0000 7f799bfdf700 1 -- 172.21.15.114:0/2868806207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79941025d0 con 0x7f79941046c0 2024-05-27T07:28:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.605+0000 7f798affd700 1 -- 172.21.15.114:0/2868806207 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79900044b0 con 0x7f79941046c0 2024-05-27T07:28:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.605+0000 7f798affd700 1 -- 172.21.15.114:0/2868806207 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7990017040 con 0x7f79941046c0 2024-05-27T07:28:55.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.605+0000 7f799bfdf700 1 -- 172.21.15.114:0/2868806207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7994102a90 con 0x7f79941046c0 2024-05-27T07:28:55.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.606+0000 7f799bfdf700 1 -- 172.21.15.114:0/2868806207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f799410a300 con 0x7f79941046c0 2024-05-27T07:28:55.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.606+0000 7f798affd700 1 -- 172.21.15.114:0/2868806207 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f7990024020 con 0x7f79941046c0 2024-05-27T07:28:55.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.606+0000 7f798affd700 1 --2- 172.21.15.114:0/2868806207 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f798003e510 0x7f7980040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:55.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.606+0000 7f798affd700 1 -- 172.21.15.114:0/2868806207 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f799000f050 con 0x7f79941046c0 2024-05-27T07:28:55.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.606+0000 7f799957a700 1 --2- 172.21.15.114:0/2868806207 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f798003e510 0x7f7980040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:55.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.607+0000 7f799957a700 1 --2- 172.21.15.114:0/2868806207 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f798003e510 0x7f7980040a20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7984006fd0 tx=0x7f7984006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:55.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.611+0000 7f798affd700 1 -- 172.21.15.114:0/2868806207 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f799002b020 con 0x7f79941046c0 2024-05-27T07:28:55.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.815+0000 7f799bfdf700 1 -- 172.21.15.114:0/2868806207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f79941924c0 con 0x7f79941046c0 2024-05-27T07:28:55.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.820+0000 7f798affd700 1 -- 172.21.15.114:0/2868806207 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f799002e9e0 con 0x7f79941046c0 2024-05-27T07:28:55.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.823+0000 7f799bfdf700 1 -- 172.21.15.114:0/2868806207 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f798003e510 msgr2=0x7f7980040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:55.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.823+0000 7f799bfdf700 1 --2- 172.21.15.114:0/2868806207 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f798003e510 0x7f7980040a20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7984006fd0 tx=0x7f7984006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:55.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.823+0000 7f799bfdf700 1 -- 172.21.15.114:0/2868806207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79941046c0 msgr2=0x7f7994103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:55.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.823+0000 7f799bfdf700 1 --2- 172.21.15.114:0/2868806207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79941046c0 0x7f7994103e40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7990015040 tx=0x7f7990003ce0 comp rx=0 tx=0).stop 2024-05-27T07:28:55.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.823+0000 7f799bfdf700 1 -- 172.21.15.114:0/2868806207 shutdown_connections 2024-05-27T07:28:55.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.823+0000 7f799bfdf700 1 --2- 172.21.15.114:0/2868806207 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f798003e510 0x7f7980040a20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:55.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.823+0000 7f799bfdf700 1 --2- 172.21.15.114:0/2868806207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79941046c0 0x7f7994103e40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:55.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.823+0000 7f799bfdf700 1 -- 172.21.15.114:0/2868806207 >> 172.21.15.114:0/2868806207 conn(0x7f79940fe120 msgr2=0x7f79940fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:55.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.823+0000 7f799bfdf700 1 -- 172.21.15.114:0/2868806207 shutdown_connections 2024-05-27T07:28:55.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:55.823+0000 7f799bfdf700 1 -- 172.21.15.114:0/2868806207 wait complete. 2024-05-27T07:28:56.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:55 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:28:56.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:55 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:28:56.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:55 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:28:56.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:55 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:28:56.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:55 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:56.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:55 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:28:56.425 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-27T07:28:56.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-27T07:28:56.658 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:57.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:56 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2868806207' entity='client.admin' 2024-05-27T07:28:57.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:57.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:28:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.118+0000 7f5eea3f5700 1 -- 172.21.15.114:0/1226250268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee4101ea0 msgr2=0x7f5ee41022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.118+0000 7f5eea3f5700 1 --2- 172.21.15.114:0/1226250268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee4101ea0 0x7f5ee41022b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009910 tx=0x7f5ed4009c20 comp rx=0 tx=0).stop 2024-05-27T07:28:58.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.118+0000 7f5eea3f5700 1 -- 172.21.15.114:0/1226250268 shutdown_connections 2024-05-27T07:28:58.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.118+0000 7f5eea3f5700 1 --2- 172.21.15.114:0/1226250268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee4101ea0 0x7f5ee41022b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.118+0000 7f5eea3f5700 1 -- 172.21.15.114:0/1226250268 >> 172.21.15.114:0/1226250268 conn(0x7f5ee40fd8e0 msgr2=0x7f5ee40ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:58.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.118+0000 7f5eea3f5700 1 -- 172.21.15.114:0/1226250268 shutdown_connections 2024-05-27T07:28:58.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.118+0000 7f5eea3f5700 1 -- 172.21.15.114:0/1226250268 wait complete. 2024-05-27T07:28:58.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.118+0000 7f5eea3f5700 1 Processor -- start 2024-05-27T07:28:58.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.118+0000 7f5eea3f5700 1 -- start start 2024-05-27T07:28:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.119+0000 7f5eea3f5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee4101ea0 0x7f5ee4104670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.119+0000 7f5eea3f5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee4109ba0 con 0x7f5ee4101ea0 2024-05-27T07:28:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.119+0000 7f5ee3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee4101ea0 0x7f5ee4104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.119+0000 7f5ee3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee4101ea0 0x7f5ee4104670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41908/0 (socket says 172.21.15.114:41908) 2024-05-27T07:28:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.119+0000 7f5ee3fff700 1 -- 172.21.15.114:0/141431633 learned_addr learned my addr 172.21.15.114:0/141431633 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:28:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.119+0000 7f5ee3fff700 1 -- 172.21.15.114:0/141431633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee4104bb0 con 0x7f5ee4101ea0 2024-05-27T07:28:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.119+0000 7f5ee3fff700 1 --2- 172.21.15.114:0/141431633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee4101ea0 0x7f5ee4104670 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4006010 tx=0x7f5ed4004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.119+0000 7f5ee17fa700 1 -- 172.21.15.114:0/141431633 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ed4026070 con 0x7f5ee4101ea0 2024-05-27T07:28:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.119+0000 7f5ee17fa700 1 -- 172.21.15.114:0/141431633 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ed400bc00 con 0x7f5ee4101ea0 2024-05-27T07:28:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.119+0000 7f5ee17fa700 1 -- 172.21.15.114:0/141431633 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ed4003970 con 0x7f5ee4101ea0 2024-05-27T07:28:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.119+0000 7f5eea3f5700 1 -- 172.21.15.114:0/141431633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ee4102e00 con 0x7f5ee4101ea0 2024-05-27T07:28:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.120+0000 7f5eea3f5700 1 -- 172.21.15.114:0/141431633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ee41032e0 con 0x7f5ee4101ea0 2024-05-27T07:28:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.120+0000 7f5eea3f5700 1 -- 172.21.15.114:0/141431633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ee410a320 con 0x7f5ee4101ea0 2024-05-27T07:28:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.121+0000 7f5ee17fa700 1 -- 172.21.15.114:0/141431633 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f5ed4029070 con 0x7f5ee4101ea0 2024-05-27T07:28:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.121+0000 7f5ee17fa700 1 --2- 172.21.15.114:0/141431633 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5ecc03e570 0x7f5ecc040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.121+0000 7f5ee17fa700 1 -- 172.21.15.114:0/141431633 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5ed401e070 con 0x7f5ee4101ea0 2024-05-27T07:28:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.121+0000 7f5ee37fe700 1 --2- 172.21.15.114:0/141431633 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5ecc03e570 0x7f5ecc040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.121+0000 7f5ee37fe700 1 --2- 172.21.15.114:0/141431633 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5ecc03e570 0x7f5ecc040a80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8006fd0 tx=0x7f5ed8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:58.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.126+0000 7f5ee17fa700 1 -- 172.21.15.114:0/141431633 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5ed4013070 con 0x7f5ee4101ea0 2024-05-27T07:28:58.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.330+0000 7f5eea3f5700 1 -- 172.21.15.114:0/141431633 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f5ee4192540 con 0x7f5ecc03e570 2024-05-27T07:28:58.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.334+0000 7f5ee17fa700 1 -- 172.21.15.114:0/141431633 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5ee4192540 con 0x7f5ecc03e570 2024-05-27T07:28:58.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.336+0000 7f5eea3f5700 1 -- 172.21.15.114:0/141431633 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5ecc03e570 msgr2=0x7f5ecc040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:58.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.336+0000 7f5eea3f5700 1 --2- 172.21.15.114:0/141431633 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5ecc03e570 0x7f5ecc040a80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8006fd0 tx=0x7f5ed8006e40 comp rx=0 tx=0).stop 2024-05-27T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.336+0000 7f5eea3f5700 1 -- 172.21.15.114:0/141431633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee4101ea0 msgr2=0x7f5ee4104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.336+0000 7f5eea3f5700 1 --2- 172.21.15.114:0/141431633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee4101ea0 0x7f5ee4104670 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4006010 tx=0x7f5ed4004fa0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.336+0000 7f5eea3f5700 1 -- 172.21.15.114:0/141431633 shutdown_connections 2024-05-27T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.336+0000 7f5eea3f5700 1 --2- 172.21.15.114:0/141431633 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5ecc03e570 0x7f5ecc040a80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.336+0000 7f5eea3f5700 1 --2- 172.21.15.114:0/141431633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee4101ea0 0x7f5ee4104670 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.336+0000 7f5eea3f5700 1 -- 172.21.15.114:0/141431633 >> 172.21.15.114:0/141431633 conn(0x7f5ee40fd8e0 msgr2=0x7f5ee40fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.336+0000 7f5eea3f5700 1 -- 172.21.15.114:0/141431633 shutdown_connections 2024-05-27T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:28:58.336+0000 7f5eea3f5700 1 -- 172.21.15.114:0/141431633 wait complete. 2024-05-27T07:28:59.059 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi163 2024-05-27T07:28:59.060 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-27T07:28:59.060 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/etc/ceph/ceph.conf 2024-05-27T07:28:59.079 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-27T07:28:59.079 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:28:59.137 INFO:tasks.cephadm:Adding host smithi163 to orchestrator... 2024-05-27T07:28:59.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch host add smithi163 2024-05-27T07:28:59.379 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:59.585 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:59 smithi114 ceph-mon[32038]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:28:59.585 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:28:59.585 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:59.585 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:28:59.585 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:28:59.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:59 smithi114 ceph-mon[32038]: Updating smithi114:/etc/ceph/ceph.conf 2024-05-27T07:28:59.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:59 smithi114 ceph-mon[32038]: Updating smithi114:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:28:59.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:59 smithi114 ceph-mon[32038]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:28:59.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:59 smithi114 ceph-mon[32038]: Updating smithi114:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:28:59.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:28:59.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:28:59.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:28:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:00.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.729+0000 7fa4e5356700 1 -- 172.21.15.114:0/1991001979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e0108a70 msgr2=0x7fa4e0108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:00.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.729+0000 7fa4e5356700 1 --2- 172.21.15.114:0/1991001979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e0108a70 0x7fa4e0108e80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0009910 tx=0x7fa4d0009c20 comp rx=0 tx=0).stop 2024-05-27T07:29:00.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.729+0000 7fa4e5356700 1 -- 172.21.15.114:0/1991001979 shutdown_connections 2024-05-27T07:29:00.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.730+0000 7fa4e5356700 1 --2- 172.21.15.114:0/1991001979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e0108a70 0x7fa4e0108e80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:00.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.730+0000 7fa4e5356700 1 -- 172.21.15.114:0/1991001979 >> 172.21.15.114:0/1991001979 conn(0x7fa4e00785a0 msgr2=0x7fa4e00789e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:00.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.730+0000 7fa4e5356700 1 -- 172.21.15.114:0/1991001979 shutdown_connections 2024-05-27T07:29:00.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.730+0000 7fa4e5356700 1 -- 172.21.15.114:0/1991001979 wait complete. 2024-05-27T07:29:00.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.730+0000 7fa4e5356700 1 Processor -- start 2024-05-27T07:29:00.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.730+0000 7fa4e5356700 1 -- start start 2024-05-27T07:29:00.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.731+0000 7fa4e5356700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e0108a70 0x7fa4e019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:00.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.731+0000 7fa4e5356700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e010cfc0 con 0x7fa4e0108a70 2024-05-27T07:29:00.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.731+0000 7fa4deffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e0108a70 0x7fa4e019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:00.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.731+0000 7fa4deffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e0108a70 0x7fa4e019a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41920/0 (socket says 172.21.15.114:41920) 2024-05-27T07:29:00.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.731+0000 7fa4deffd700 1 -- 172.21.15.114:0/1798277753 learned_addr learned my addr 172.21.15.114:0/1798277753 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:29:00.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.731+0000 7fa4deffd700 1 -- 172.21.15.114:0/1798277753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e006bcd0 con 0x7fa4e0108a70 2024-05-27T07:29:00.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.731+0000 7fa4deffd700 1 --2- 172.21.15.114:0/1798277753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e0108a70 0x7fa4e019a7d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00056d0 tx=0x7fa4d00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:00.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.731+0000 7fa4c7fff700 1 -- 172.21.15.114:0/1798277753 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4d0027070 con 0x7fa4e0108a70 2024-05-27T07:29:00.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.731+0000 7fa4c7fff700 1 -- 172.21.15.114:0/1798277753 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4d000be10 con 0x7fa4e0108a70 2024-05-27T07:29:00.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.731+0000 7fa4e5356700 1 -- 172.21.15.114:0/1798277753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4e019ad70 con 0x7fa4e0108a70 2024-05-27T07:29:00.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.731+0000 7fa4c7fff700 1 -- 172.21.15.114:0/1798277753 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4d0022d20 con 0x7fa4e0108a70 2024-05-27T07:29:00.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.731+0000 7fa4e5356700 1 -- 172.21.15.114:0/1798277753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4e019b230 con 0x7fa4e0108a70 2024-05-27T07:29:00.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.732+0000 7fa4c7fff700 1 -- 172.21.15.114:0/1798277753 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7fa4d002a070 con 0x7fa4e0108a70 2024-05-27T07:29:00.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.732+0000 7fa4e5356700 1 -- 172.21.15.114:0/1798277753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4e0104180 con 0x7fa4e0108a70 2024-05-27T07:29:00.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.733+0000 7fa4c7fff700 1 --2- 172.21.15.114:0/1798277753 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa4c803e510 0x7fa4c8040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:00.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.733+0000 7fa4c7fff700 1 -- 172.21.15.114:0/1798277753 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa4d001e030 con 0x7fa4e0108a70 2024-05-27T07:29:00.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.733+0000 7fa4de7fc700 1 --2- 172.21.15.114:0/1798277753 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa4c803e510 0x7fa4c8040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:00.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.733+0000 7fa4de7fc700 1 --2- 172.21.15.114:0/1798277753 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa4c803e510 0x7fa4c8040a20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4006fd0 tx=0x7fa4d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:00.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.738+0000 7fa4c7fff700 1 -- 172.21.15.114:0/1798277753 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa4d0013070 con 0x7fa4e0108a70 2024-05-27T07:29:00.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:00.947+0000 7fa4e5356700 1 -- 172.21.15.114:0/1798277753 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi163", "target": ["mon-mgr", ""]}) -- 0x7fa4e006b420 con 0x7fa4c803e510 2024-05-27T07:29:01.296 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:01 smithi114 ceph-mon[32038]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi163", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:29:02.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:02 smithi114 ceph-mon[32038]: Deploying cephadm binary to smithi163 2024-05-27T07:29:04.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:04 smithi114 ceph-mon[32038]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:05.047 INFO:teuthology.orchestra.run.smithi114.stdout:Added host 'smithi163' with addr '172.21.15.163' 2024-05-27T07:29:05.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:05.056+0000 7fa4c7fff700 1 -- 172.21.15.114:0/1798277753 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fa4e006b420 con 0x7fa4c803e510 2024-05-27T07:29:05.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:05.059+0000 7fa4e5356700 1 -- 172.21.15.114:0/1798277753 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa4c803e510 msgr2=0x7fa4c8040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:05.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:05.059+0000 7fa4e5356700 1 --2- 172.21.15.114:0/1798277753 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa4c803e510 0x7fa4c8040a20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4006fd0 tx=0x7fa4d4006e40 comp rx=0 tx=0).stop 2024-05-27T07:29:05.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:05.059+0000 7fa4e5356700 1 -- 172.21.15.114:0/1798277753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e0108a70 msgr2=0x7fa4e019a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:05.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:05.059+0000 7fa4e5356700 1 --2- 172.21.15.114:0/1798277753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e0108a70 0x7fa4e019a7d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00056d0 tx=0x7fa4d00057b0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:05.059+0000 7fa4e5356700 1 -- 172.21.15.114:0/1798277753 shutdown_connections 2024-05-27T07:29:05.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:05.059+0000 7fa4e5356700 1 --2- 172.21.15.114:0/1798277753 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa4c803e510 0x7fa4c8040a20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:05.059+0000 7fa4e5356700 1 --2- 172.21.15.114:0/1798277753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e0108a70 0x7fa4e019a7d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:05.059+0000 7fa4e5356700 1 -- 172.21.15.114:0/1798277753 >> 172.21.15.114:0/1798277753 conn(0x7fa4e00785a0 msgr2=0x7fa4e0101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:05.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:05.059+0000 7fa4e5356700 1 -- 172.21.15.114:0/1798277753 shutdown_connections 2024-05-27T07:29:05.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:05.059+0000 7fa4e5356700 1 -- 172.21.15.114:0/1798277753 wait complete. 2024-05-27T07:29:05.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-05-27T07:29:05.849 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:06.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:06 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:06.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:06 smithi114 ceph-mon[32038]: Added host smithi163 2024-05-27T07:29:06.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:06 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:06.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:06 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:07.160 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:07 smithi114 ceph-mon[32038]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:07.160 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:07 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:07.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.284+0000 7fe3298d6700 1 -- 172.21.15.114:0/1773400782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324108a70 msgr2=0x7fe324108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:07.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.284+0000 7fe3298d6700 1 --2- 172.21.15.114:0/1773400782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324108a70 0x7fe324108e80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009910 tx=0x7fe30c009c20 comp rx=0 tx=0).stop 2024-05-27T07:29:07.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.284+0000 7fe3298d6700 1 -- 172.21.15.114:0/1773400782 shutdown_connections 2024-05-27T07:29:07.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.284+0000 7fe3298d6700 1 --2- 172.21.15.114:0/1773400782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324108a70 0x7fe324108e80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:07.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.284+0000 7fe3298d6700 1 -- 172.21.15.114:0/1773400782 >> 172.21.15.114:0/1773400782 conn(0x7fe3240785a0 msgr2=0x7fe3240789e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:07.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.284+0000 7fe3298d6700 1 -- 172.21.15.114:0/1773400782 shutdown_connections 2024-05-27T07:29:07.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.284+0000 7fe3298d6700 1 -- 172.21.15.114:0/1773400782 wait complete. 2024-05-27T07:29:07.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.285+0000 7fe3298d6700 1 Processor -- start 2024-05-27T07:29:07.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.285+0000 7fe3298d6700 1 -- start start 2024-05-27T07:29:07.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.285+0000 7fe3298d6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324108a70 0x7fe3241985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:07.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.285+0000 7fe3298d6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe32410cfc0 con 0x7fe324108a70 2024-05-27T07:29:07.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.286+0000 7fe322ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324108a70 0x7fe3241985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:07.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.286+0000 7fe322ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324108a70 0x7fe3241985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38106/0 (socket says 172.21.15.114:38106) 2024-05-27T07:29:07.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.286+0000 7fe322ffd700 1 -- 172.21.15.114:0/1267618677 learned_addr learned my addr 172.21.15.114:0/1267618677 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:29:07.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.286+0000 7fe322ffd700 1 -- 172.21.15.114:0/1267618677 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32406bb00 con 0x7fe324108a70 2024-05-27T07:29:07.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.286+0000 7fe322ffd700 1 --2- 172.21.15.114:0/1267618677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324108a70 0x7fe3241985e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0056d0 tx=0x7fe30c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:07.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.286+0000 7fe3288d4700 1 -- 172.21.15.114:0/1267618677 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe30c027070 con 0x7fe324108a70 2024-05-27T07:29:07.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.286+0000 7fe3298d6700 1 -- 172.21.15.114:0/1267618677 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe324198b20 con 0x7fe324108a70 2024-05-27T07:29:07.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.286+0000 7fe3288d4700 1 -- 172.21.15.114:0/1267618677 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe30c00be10 con 0x7fe324108a70 2024-05-27T07:29:07.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.286+0000 7fe3288d4700 1 -- 172.21.15.114:0/1267618677 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe30c022d20 con 0x7fe324108a70 2024-05-27T07:29:07.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.286+0000 7fe3298d6700 1 -- 172.21.15.114:0/1267618677 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe324198fe0 con 0x7fe324108a70 2024-05-27T07:29:07.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.287+0000 7fe3298d6700 1 -- 172.21.15.114:0/1267618677 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe324104180 con 0x7fe324108a70 2024-05-27T07:29:07.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.287+0000 7fe3288d4700 1 -- 172.21.15.114:0/1267618677 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7fe30c02a070 con 0x7fe324108a70 2024-05-27T07:29:07.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.287+0000 7fe3288d4700 1 --2- 172.21.15.114:0/1267618677 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe31003e510 0x7fe310040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:07.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.287+0000 7fe3288d4700 1 -- 172.21.15.114:0/1267618677 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe30c01e030 con 0x7fe324108a70 2024-05-27T07:29:07.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.288+0000 7fe3227fc700 1 --2- 172.21.15.114:0/1267618677 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe31003e510 0x7fe310040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:07.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.288+0000 7fe3227fc700 1 --2- 172.21.15.114:0/1267618677 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe31003e510 0x7fe310040a20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe314006fd0 tx=0x7fe314006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:07.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.293+0000 7fe3288d4700 1 -- 172.21.15.114:0/1267618677 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe30c013070 con 0x7fe324108a70 2024-05-27T07:29:07.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.502+0000 7fe3298d6700 1 -- 172.21.15.114:0/1267618677 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe32406b430 con 0x7fe31003e510 2024-05-27T07:29:07.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.504+0000 7fe3288d4700 1 -- 172.21.15.114:0/1267618677 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7fe32406b430 con 0x7fe31003e510 2024-05-27T07:29:07.494 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:29:07.494 INFO:teuthology.orchestra.run.smithi114.stdout:[{"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.163", "hostname": "smithi163", "labels": [], "status": ""}] 2024-05-27T07:29:07.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.506+0000 7fe3298d6700 1 -- 172.21.15.114:0/1267618677 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe31003e510 msgr2=0x7fe310040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:07.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.506+0000 7fe3298d6700 1 --2- 172.21.15.114:0/1267618677 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe31003e510 0x7fe310040a20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe314006fd0 tx=0x7fe314006e40 comp rx=0 tx=0).stop 2024-05-27T07:29:07.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.506+0000 7fe3298d6700 1 -- 172.21.15.114:0/1267618677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324108a70 msgr2=0x7fe3241985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:07.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.506+0000 7fe3298d6700 1 --2- 172.21.15.114:0/1267618677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324108a70 0x7fe3241985e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0056d0 tx=0x7fe30c0057b0 comp rx=0 tx=0).stop 2024-05-27T07:29:07.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.506+0000 7fe3298d6700 1 -- 172.21.15.114:0/1267618677 shutdown_connections 2024-05-27T07:29:07.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.506+0000 7fe3298d6700 1 --2- 172.21.15.114:0/1267618677 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe31003e510 0x7fe310040a20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:07.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.506+0000 7fe3298d6700 1 --2- 172.21.15.114:0/1267618677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324108a70 0x7fe3241985e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:07.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.506+0000 7fe3298d6700 1 -- 172.21.15.114:0/1267618677 >> 172.21.15.114:0/1267618677 conn(0x7fe3240785a0 msgr2=0x7fe324101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:07.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.506+0000 7fe3298d6700 1 -- 172.21.15.114:0/1267618677 shutdown_connections 2024-05-27T07:29:07.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:07.506+0000 7fe3298d6700 1 -- 172.21.15.114:0/1267618677 wait complete. 2024-05-27T07:29:08.073 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi186 2024-05-27T07:29:08.073 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-27T07:29:08.073 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.conf 2024-05-27T07:29:08.091 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-27T07:29:08.091 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:29:08.148 INFO:tasks.cephadm:Adding host smithi186 to orchestrator... 2024-05-27T07:29:08.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch host add smithi186 2024-05-27T07:29:08.384 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:09.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:08 smithi114 ceph-mon[32038]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:09.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:08 smithi114 ceph-mon[32038]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:29:09.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:09.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.837+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3127569250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e98107a80 msgr2=0x7f9e98107e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:09.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.837+0000 7f9e9ee72700 1 --2- 172.21.15.114:0/3127569250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e98107a80 0x7f9e98107e90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009910 tx=0x7f9e8c009c20 comp rx=0 tx=0).stop 2024-05-27T07:29:09.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.837+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3127569250 shutdown_connections 2024-05-27T07:29:09.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.837+0000 7f9e9ee72700 1 --2- 172.21.15.114:0/3127569250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e98107a80 0x7f9e98107e90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.837+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3127569250 >> 172.21.15.114:0/3127569250 conn(0x7f9e98078d50 msgr2=0x7f9e98079190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:09.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.837+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3127569250 shutdown_connections 2024-05-27T07:29:09.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.837+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3127569250 wait complete. 2024-05-27T07:29:09.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.837+0000 7f9e9ee72700 1 Processor -- start 2024-05-27T07:29:09.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.837+0000 7f9e9ee72700 1 -- start start 2024-05-27T07:29:09.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.838+0000 7f9e9ee72700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e98107a80 0x7f9e9819f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.838+0000 7f9e9ee72700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9810d120 con 0x7f9e98107a80 2024-05-27T07:29:09.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.838+0000 7f9e9cc0e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e98107a80 0x7f9e9819f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.838+0000 7f9e9cc0e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e98107a80 0x7f9e9819f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38122/0 (socket says 172.21.15.114:38122) 2024-05-27T07:29:09.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.838+0000 7f9e9cc0e700 1 -- 172.21.15.114:0/3889322966 learned_addr learned my addr 172.21.15.114:0/3889322966 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:29:09.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.838+0000 7f9e9cc0e700 1 -- 172.21.15.114:0/3889322966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9819faa0 con 0x7f9e98107a80 2024-05-27T07:29:09.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.838+0000 7f9e9cc0e700 1 --2- 172.21.15.114:0/3889322966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e98107a80 0x7f9e9819f560 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c006010 tx=0x7f9e8c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:09.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.838+0000 7f9e95ffb700 1 -- 172.21.15.114:0/3889322966 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e8c026070 con 0x7f9e98107a80 2024-05-27T07:29:09.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.838+0000 7f9e95ffb700 1 -- 172.21.15.114:0/3889322966 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e8c00bc00 con 0x7f9e98107a80 2024-05-27T07:29:09.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.838+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3889322966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e9819fd30 con 0x7f9e98107a80 2024-05-27T07:29:09.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.838+0000 7f9e95ffb700 1 -- 172.21.15.114:0/3889322966 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e8c003910 con 0x7f9e98107a80 2024-05-27T07:29:09.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.839+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3889322966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e981997a0 con 0x7f9e98107a80 2024-05-27T07:29:09.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.840+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3889322966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e98104210 con 0x7f9e98107a80 2024-05-27T07:29:09.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.840+0000 7f9e95ffb700 1 -- 172.21.15.114:0/3889322966 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f9e8c029070 con 0x7f9e98107a80 2024-05-27T07:29:09.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.840+0000 7f9e95ffb700 1 --2- 172.21.15.114:0/3889322966 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9e8803e570 0x7f9e88040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.840+0000 7f9e95ffb700 1 -- 172.21.15.114:0/3889322966 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9e8c01e070 con 0x7f9e98107a80 2024-05-27T07:29:09.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.840+0000 7f9e97fff700 1 --2- 172.21.15.114:0/3889322966 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9e8803e570 0x7f9e88040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.840+0000 7f9e97fff700 1 --2- 172.21.15.114:0/3889322966 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9e8803e570 0x7f9e88040a80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9e84006fd0 tx=0x7f9e84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:09.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:09.845+0000 7f9e95ffb700 1 -- 172.21.15.114:0/3889322966 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9e8c013070 con 0x7f9e98107a80 2024-05-27T07:29:10.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:10.052+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3889322966 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}) -- 0x7f9e9819acb0 con 0x7f9e8803e570 2024-05-27T07:29:11.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:10 smithi114 ceph-mon[32038]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:11.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:10 smithi114 ceph-mon[32038]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:29:12.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:11 smithi114 ceph-mon[32038]: Deploying cephadm binary to smithi186 2024-05-27T07:29:12.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:12.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:12.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:12.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:12.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi163", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:29:12.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:12.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:12.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:11 smithi114 ceph-mon[32038]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-27T07:29:12.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:11 smithi114 ceph-mon[32038]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:13.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:12 smithi114 ceph-mon[32038]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:14.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:14.099+0000 7f9e95ffb700 1 -- 172.21.15.114:0/3889322966 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f9e9819acb0 con 0x7f9e8803e570 2024-05-27T07:29:14.090 INFO:teuthology.orchestra.run.smithi114.stdout:Added host 'smithi186' with addr '172.21.15.186' 2024-05-27T07:29:14.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:14.101+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3889322966 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9e8803e570 msgr2=0x7f9e88040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:14.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:14.101+0000 7f9e9ee72700 1 --2- 172.21.15.114:0/3889322966 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9e8803e570 0x7f9e88040a80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9e84006fd0 tx=0x7f9e84006e40 comp rx=0 tx=0).stop 2024-05-27T07:29:14.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:14.101+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3889322966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e98107a80 msgr2=0x7f9e9819f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:14.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:14.101+0000 7f9e9ee72700 1 --2- 172.21.15.114:0/3889322966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e98107a80 0x7f9e9819f560 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c006010 tx=0x7f9e8c005780 comp rx=0 tx=0).stop 2024-05-27T07:29:14.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:14.101+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3889322966 shutdown_connections 2024-05-27T07:29:14.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:14.101+0000 7f9e9ee72700 1 --2- 172.21.15.114:0/3889322966 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9e8803e570 0x7f9e88040a80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:14.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:14.101+0000 7f9e9ee72700 1 --2- 172.21.15.114:0/3889322966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e98107a80 0x7f9e9819f560 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:14.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:14.101+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3889322966 >> 172.21.15.114:0/3889322966 conn(0x7f9e98078d50 msgr2=0x7f9e981020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:14.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:14.102+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3889322966 shutdown_connections 2024-05-27T07:29:14.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:14.102+0000 7f9e9ee72700 1 -- 172.21.15.114:0/3889322966 wait complete. 2024-05-27T07:29:14.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-05-27T07:29:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:15.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:14 smithi114 ceph-mon[32038]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:15.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:14 smithi114 ceph-mon[32038]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:29:15.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:14 smithi114 ceph-mon[32038]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:29:15.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:15.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:15.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:15.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:15.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:14 smithi114 ceph-mon[32038]: Added host smithi186 2024-05-27T07:29:15.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:15.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:16.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:16 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:16.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:16 smithi114 ceph-mon[32038]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:16.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.490+0000 7f5e2194e700 1 -- 172.21.15.114:0/1280710397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e1c104d30 msgr2=0x7f5e1c105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:16.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.490+0000 7f5e2194e700 1 --2- 172.21.15.114:0/1280710397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e1c104d30 0x7f5e1c105140 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009910 tx=0x7f5e0c009c20 comp rx=0 tx=0).stop 2024-05-27T07:29:16.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.490+0000 7f5e2194e700 1 -- 172.21.15.114:0/1280710397 shutdown_connections 2024-05-27T07:29:16.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.490+0000 7f5e2194e700 1 --2- 172.21.15.114:0/1280710397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e1c104d30 0x7f5e1c105140 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:16.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.490+0000 7f5e2194e700 1 -- 172.21.15.114:0/1280710397 >> 172.21.15.114:0/1280710397 conn(0x7f5e1c0785a0 msgr2=0x7f5e1c0789e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:16.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.490+0000 7f5e2194e700 1 -- 172.21.15.114:0/1280710397 shutdown_connections 2024-05-27T07:29:16.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.490+0000 7f5e2194e700 1 -- 172.21.15.114:0/1280710397 wait complete. 2024-05-27T07:29:16.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.491+0000 7f5e2194e700 1 Processor -- start 2024-05-27T07:29:16.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.491+0000 7f5e2194e700 1 -- start start 2024-05-27T07:29:16.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.491+0000 7f5e2194e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e1c104d30 0x7f5e1c19f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:16.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.491+0000 7f5e2194e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1c10e120 con 0x7f5e1c104d30 2024-05-27T07:29:16.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.491+0000 7f5e1affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e1c104d30 0x7f5e1c19f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:16.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.492+0000 7f5e1affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e1c104d30 0x7f5e1c19f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37930/0 (socket says 172.21.15.114:37930) 2024-05-27T07:29:16.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.492+0000 7f5e1affd700 1 -- 172.21.15.114:0/4073557334 learned_addr learned my addr 172.21.15.114:0/4073557334 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:29:16.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.492+0000 7f5e1affd700 1 -- 172.21.15.114:0/4073557334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e1c19faa0 con 0x7f5e1c104d30 2024-05-27T07:29:16.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.492+0000 7f5e1affd700 1 --2- 172.21.15.114:0/4073557334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e1c104d30 0x7f5e1c19f560 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c0056d0 tx=0x7f5e0c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:16.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.492+0000 7f5e2094c700 1 -- 172.21.15.114:0/4073557334 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e0c026070 con 0x7f5e1c104d30 2024-05-27T07:29:16.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.492+0000 7f5e2094c700 1 -- 172.21.15.114:0/4073557334 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e0c004370 con 0x7f5e1c104d30 2024-05-27T07:29:16.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.492+0000 7f5e2094c700 1 -- 172.21.15.114:0/4073557334 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e0c0039b0 con 0x7f5e1c104d30 2024-05-27T07:29:16.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.492+0000 7f5e2194e700 1 -- 172.21.15.114:0/4073557334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e1c19fd30 con 0x7f5e1c104d30 2024-05-27T07:29:16.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.492+0000 7f5e2194e700 1 -- 172.21.15.114:0/4073557334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e1c1997a0 con 0x7f5e1c104d30 2024-05-27T07:29:16.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.493+0000 7f5e2194e700 1 -- 172.21.15.114:0/4073557334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e1c06b6d0 con 0x7f5e1c104d30 2024-05-27T07:29:16.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.493+0000 7f5e2094c700 1 -- 172.21.15.114:0/4073557334 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f5e0c029070 con 0x7f5e1c104d30 2024-05-27T07:29:16.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.493+0000 7f5e2094c700 1 --2- 172.21.15.114:0/4073557334 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5e0803e510 0x7f5e08040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:16.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.494+0000 7f5e2094c700 1 -- 172.21.15.114:0/4073557334 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5e0c01e070 con 0x7f5e1c104d30 2024-05-27T07:29:16.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.494+0000 7f5e127fc700 1 --2- 172.21.15.114:0/4073557334 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5e0803e510 0x7f5e08040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:16.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.494+0000 7f5e127fc700 1 --2- 172.21.15.114:0/4073557334 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5e0803e510 0x7f5e08040a20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5e04006fd0 tx=0x7f5e04006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:16.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.499+0000 7f5e2094c700 1 -- 172.21.15.114:0/4073557334 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5e0c013070 con 0x7f5e1c104d30 2024-05-27T07:29:16.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.708+0000 7f5e2194e700 1 -- 172.21.15.114:0/4073557334 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5e1c19acb0 con 0x7f5e0803e510 2024-05-27T07:29:16.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.709+0000 7f5e2094c700 1 -- 172.21.15.114:0/4073557334 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f5e1c19acb0 con 0x7f5e0803e510 2024-05-27T07:29:16.699 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:29:16.699 INFO:teuthology.orchestra.run.smithi114.stdout:[{"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.163", "hostname": "smithi163", "labels": [], "status": ""}, {"addr": "172.21.15.186", "hostname": "smithi186", "labels": [], "status": ""}] 2024-05-27T07:29:16.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.712+0000 7f5e2194e700 1 -- 172.21.15.114:0/4073557334 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5e0803e510 msgr2=0x7f5e08040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:16.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.712+0000 7f5e2194e700 1 --2- 172.21.15.114:0/4073557334 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5e0803e510 0x7f5e08040a20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5e04006fd0 tx=0x7f5e04006e40 comp rx=0 tx=0).stop 2024-05-27T07:29:16.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.712+0000 7f5e2194e700 1 -- 172.21.15.114:0/4073557334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e1c104d30 msgr2=0x7f5e1c19f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:16.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.712+0000 7f5e2194e700 1 --2- 172.21.15.114:0/4073557334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e1c104d30 0x7f5e1c19f560 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c0056d0 tx=0x7f5e0c0057b0 comp rx=0 tx=0).stop 2024-05-27T07:29:16.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.712+0000 7f5e2194e700 1 -- 172.21.15.114:0/4073557334 shutdown_connections 2024-05-27T07:29:16.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.712+0000 7f5e2194e700 1 --2- 172.21.15.114:0/4073557334 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5e0803e510 0x7f5e08040a20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:16.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.712+0000 7f5e2194e700 1 --2- 172.21.15.114:0/4073557334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e1c104d30 0x7f5e1c19f560 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:16.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.712+0000 7f5e2194e700 1 -- 172.21.15.114:0/4073557334 >> 172.21.15.114:0/4073557334 conn(0x7f5e1c0785a0 msgr2=0x7f5e1c105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:16.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.712+0000 7f5e2194e700 1 -- 172.21.15.114:0/4073557334 shutdown_connections 2024-05-27T07:29:16.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:16.712+0000 7f5e2194e700 1 -- 172.21.15.114:0/4073557334 wait complete. 2024-05-27T07:29:17.305 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-27T07:29:17.305 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-05-27T07:29:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:17.955 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:17 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:17.955 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:17 smithi114 ceph-mon[32038]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:29:18.786 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:18 smithi114 ceph-mon[32038]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.036+0000 7f4a93180700 1 -- 172.21.15.114:0/432998846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c108a70 msgr2=0x7f4a8c108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.036+0000 7f4a93180700 1 --2- 172.21.15.114:0/432998846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c108a70 0x7f4a8c108e80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000b0a0 tx=0x7f4a8000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:29:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.036+0000 7f4a93180700 1 -- 172.21.15.114:0/432998846 shutdown_connections 2024-05-27T07:29:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.036+0000 7f4a93180700 1 --2- 172.21.15.114:0/432998846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c108a70 0x7f4a8c108e80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.036+0000 7f4a93180700 1 -- 172.21.15.114:0/432998846 >> 172.21.15.114:0/432998846 conn(0x7f4a8c0785a0 msgr2=0x7f4a8c0789e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.036+0000 7f4a93180700 1 -- 172.21.15.114:0/432998846 shutdown_connections 2024-05-27T07:29:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.036+0000 7f4a93180700 1 -- 172.21.15.114:0/432998846 wait complete. 2024-05-27T07:29:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.037+0000 7f4a93180700 1 Processor -- start 2024-05-27T07:29:19.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.037+0000 7f4a93180700 1 -- start start 2024-05-27T07:29:19.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.037+0000 7f4a93180700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c108a70 0x7f4a8c0ff8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:19.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.037+0000 7f4a93180700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c10cfc0 con 0x7f4a8c108a70 2024-05-27T07:29:19.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.037+0000 7f4a90f1c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c108a70 0x7f4a8c0ff8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:19.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.037+0000 7f4a90f1c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c108a70 0x7f4a8c0ff8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37940/0 (socket says 172.21.15.114:37940) 2024-05-27T07:29:19.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.037+0000 7f4a90f1c700 1 -- 172.21.15.114:0/1640421888 learned_addr learned my addr 172.21.15.114:0/1640421888 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:29:19.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.038+0000 7f4a90f1c700 1 -- 172.21.15.114:0/1640421888 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c101710 con 0x7f4a8c108a70 2024-05-27T07:29:19.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.038+0000 7f4a90f1c700 1 --2- 172.21.15.114:0/1640421888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c108a70 0x7f4a8c0ff8c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4a80015040 tx=0x7f4a80009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:19.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.038+0000 7f4a89ffb700 1 -- 172.21.15.114:0/1640421888 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a80021020 con 0x7f4a8c108a70 2024-05-27T07:29:19.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.038+0000 7f4a89ffb700 1 -- 172.21.15.114:0/1640421888 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a80007970 con 0x7f4a8c108a70 2024-05-27T07:29:19.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.038+0000 7f4a93180700 1 -- 172.21.15.114:0/1640421888 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8c0ffe00 con 0x7f4a8c108a70 2024-05-27T07:29:19.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.038+0000 7f4a89ffb700 1 -- 172.21.15.114:0/1640421888 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a80017040 con 0x7f4a8c108a70 2024-05-27T07:29:19.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.038+0000 7f4a93180700 1 -- 172.21.15.114:0/1640421888 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a8c100260 con 0x7f4a8c108a70 2024-05-27T07:29:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.039+0000 7f4a93180700 1 -- 172.21.15.114:0/1640421888 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a8c104180 con 0x7f4a8c108a70 2024-05-27T07:29:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.039+0000 7f4a89ffb700 1 -- 172.21.15.114:0/1640421888 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f4a80024020 con 0x7f4a8c108a70 2024-05-27T07:29:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.039+0000 7f4a89ffb700 1 --2- 172.21.15.114:0/1640421888 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f4a7c03e510 0x7f4a7c040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.039+0000 7f4a89ffb700 1 -- 172.21.15.114:0/1640421888 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4a8000f050 con 0x7f4a8c108a70 2024-05-27T07:29:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.040+0000 7f4a8bfff700 1 --2- 172.21.15.114:0/1640421888 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f4a7c03e510 0x7f4a7c040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.040+0000 7f4a8bfff700 1 --2- 172.21.15.114:0/1640421888 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f4a7c03e510 0x7f4a7c040a20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4a78006fd0 tx=0x7f4a78006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:19.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.045+0000 7f4a89ffb700 1 -- 172.21.15.114:0/1640421888 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4a8002b020 con 0x7f4a8c108a70 2024-05-27T07:29:19.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.253+0000 7f4a93180700 1 -- 172.21.15.114:0/1640421888 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f4a8c100e20 con 0x7f4a8c108a70 2024-05-27T07:29:19.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.686+0000 7f4a89ffb700 1 -- 172.21.15.114:0/1640421888 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f4a8002e9e0 con 0x7f4a8c108a70 2024-05-27T07:29:19.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.689+0000 7f4a93180700 1 -- 172.21.15.114:0/1640421888 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f4a7c03e510 msgr2=0x7f4a7c040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:19.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.689+0000 7f4a93180700 1 --2- 172.21.15.114:0/1640421888 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f4a7c03e510 0x7f4a7c040a20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4a78006fd0 tx=0x7f4a78006e40 comp rx=0 tx=0).stop 2024-05-27T07:29:19.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.689+0000 7f4a93180700 1 -- 172.21.15.114:0/1640421888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c108a70 msgr2=0x7f4a8c0ff8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:19.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.689+0000 7f4a93180700 1 --2- 172.21.15.114:0/1640421888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c108a70 0x7f4a8c0ff8c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4a80015040 tx=0x7f4a80009790 comp rx=0 tx=0).stop 2024-05-27T07:29:19.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.689+0000 7f4a93180700 1 -- 172.21.15.114:0/1640421888 shutdown_connections 2024-05-27T07:29:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.689+0000 7f4a93180700 1 --2- 172.21.15.114:0/1640421888 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f4a7c03e510 0x7f4a7c040a20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.689+0000 7f4a93180700 1 --2- 172.21.15.114:0/1640421888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c108a70 0x7f4a8c0ff8c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.689+0000 7f4a93180700 1 -- 172.21.15.114:0/1640421888 >> 172.21.15.114:0/1640421888 conn(0x7f4a8c0785a0 msgr2=0x7f4a8c06adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.689+0000 7f4a93180700 1 -- 172.21.15.114:0/1640421888 shutdown_connections 2024-05-27T07:29:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:19.689+0000 7f4a93180700 1 -- 172.21.15.114:0/1640421888 wait complete. 2024-05-27T07:29:19.681 INFO:teuthology.orchestra.run.smithi114.stderr:adjusted tunables profile to default 2024-05-27T07:29:20.039 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:19 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1640421888' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-27T07:29:20.458 INFO:tasks.cephadm:Adding mon.a on smithi114 2024-05-27T07:29:20.458 INFO:tasks.cephadm:Adding mon.b on smithi163 2024-05-27T07:29:20.458 INFO:tasks.cephadm:Adding mon.c on smithi186 2024-05-27T07:29:20.458 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch apply mon '3;smithi114:172.21.15.114=a;smithi163:172.21.15.163=b;smithi186:172.21.15.186=c' 2024-05-27T07:29:20.692 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:20.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:20.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1640421888' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-27T07:29:20.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:29:20.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:20.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:20.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:20.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:20.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:29:20.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:20.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:20.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-27T07:29:20.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:20.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:20.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:20.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:22.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:21 smithi114 ceph-mon[32038]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:29:22.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:21 smithi114 ceph-mon[32038]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:29:22.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.190+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1666739184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bbc102470 msgr2=0x7f8bbc102880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:22.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.190+0000 7f8bc1efe700 1 --2- 172.21.15.186:0/1666739184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bbc102470 0x7f8bbc102880 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009910 tx=0x7f8bac009c20 comp rx=0 tx=0).stop 2024-05-27T07:29:22.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.190+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1666739184 shutdown_connections 2024-05-27T07:29:22.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.190+0000 7f8bc1efe700 1 --2- 172.21.15.186:0/1666739184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bbc102470 0x7f8bbc102880 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:22.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.190+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1666739184 >> 172.21.15.186:0/1666739184 conn(0x7f8bbc078d50 msgr2=0x7f8bbc079190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:22.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.190+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1666739184 shutdown_connections 2024-05-27T07:29:22.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.190+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1666739184 wait complete. 2024-05-27T07:29:22.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.191+0000 7f8bc1efe700 1 Processor -- start 2024-05-27T07:29:22.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.191+0000 7f8bc1efe700 1 -- start start 2024-05-27T07:29:22.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.191+0000 7f8bc1efe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bbc102470 0x7f8bbc19a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:22.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.191+0000 7f8bc1efe700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc10c760 con 0x7f8bbc102470 2024-05-27T07:29:22.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.192+0000 7f8bbb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bbc102470 0x7f8bbc19a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:22.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.192+0000 7f8bbb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bbc102470 0x7f8bbc19a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.186:54794/0 (socket says 172.21.15.186:54794) 2024-05-27T07:29:22.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.192+0000 7f8bbb7fe700 1 -- 172.21.15.186:0/1305186524 learned_addr learned my addr 172.21.15.186:0/1305186524 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-27T07:29:22.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.192+0000 7f8bbb7fe700 1 -- 172.21.15.186:0/1305186524 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bbc104d40 con 0x7f8bbc102470 2024-05-27T07:29:22.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.192+0000 7f8bbb7fe700 1 --2- 172.21.15.186:0/1305186524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bbc102470 0x7f8bbc19a880 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0056d0 tx=0x7f8bac0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:22.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.192+0000 7f8bb8ff9700 1 -- 172.21.15.186:0/1305186524 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bac026070 con 0x7f8bbc102470 2024-05-27T07:29:22.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.192+0000 7f8bb8ff9700 1 -- 172.21.15.186:0/1305186524 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bac00bc00 con 0x7f8bbc102470 2024-05-27T07:29:22.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.192+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1305186524 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bbc19ae80 con 0x7f8bbc102470 2024-05-27T07:29:22.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.192+0000 7f8bb8ff9700 1 -- 172.21.15.186:0/1305186524 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bac003920 con 0x7f8bbc102470 2024-05-27T07:29:22.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.192+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1305186524 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bbc19b340 con 0x7f8bbc102470 2024-05-27T07:29:22.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.193+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1305186524 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bbc103920 con 0x7f8bbc102470 2024-05-27T07:29:22.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.193+0000 7f8bb8ff9700 1 -- 172.21.15.186:0/1305186524 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f8bac029070 con 0x7f8bbc102470 2024-05-27T07:29:22.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.194+0000 7f8bb8ff9700 1 --2- 172.21.15.186:0/1305186524 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8ba403e510 0x7f8ba4040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:22.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.194+0000 7f8bb8ff9700 1 -- 172.21.15.186:0/1305186524 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8bac01e070 con 0x7f8bbc102470 2024-05-27T07:29:22.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.194+0000 7f8bbaffd700 1 --2- 172.21.15.186:0/1305186524 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8ba403e510 0x7f8ba4040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:22.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.194+0000 7f8bbaffd700 1 --2- 172.21.15.186:0/1305186524 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8ba403e510 0x7f8ba4040a20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0006fd0 tx=0x7f8bb0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:22.190 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.199+0000 7f8bb8ff9700 1 -- 172.21.15.186:0/1305186524 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8bbc104d40 con 0x7f8bbc102470 2024-05-27T07:29:22.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.415+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1305186524 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi163:172.21.15.163=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}) -- 0x7f8bbc19b620 con 0x7f8ba403e510 2024-05-27T07:29:22.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.423+0000 7f8bb8ff9700 1 -- 172.21.15.186:0/1305186524 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8bbc19b620 con 0x7f8ba403e510 2024-05-27T07:29:22.413 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mon update... 2024-05-27T07:29:22.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.425+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1305186524 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8ba403e510 msgr2=0x7f8ba4040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:22.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.425+0000 7f8bc1efe700 1 --2- 172.21.15.186:0/1305186524 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8ba403e510 0x7f8ba4040a20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0006fd0 tx=0x7f8bb0006e40 comp rx=0 tx=0).stop 2024-05-27T07:29:22.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.425+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1305186524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bbc102470 msgr2=0x7f8bbc19a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:22.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.425+0000 7f8bc1efe700 1 --2- 172.21.15.186:0/1305186524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bbc102470 0x7f8bbc19a880 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0056d0 tx=0x7f8bac0057b0 comp rx=0 tx=0).stop 2024-05-27T07:29:22.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.425+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1305186524 shutdown_connections 2024-05-27T07:29:22.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.425+0000 7f8bc1efe700 1 --2- 172.21.15.186:0/1305186524 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8ba403e510 0x7f8ba4040a20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:22.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.425+0000 7f8bc1efe700 1 --2- 172.21.15.186:0/1305186524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bbc102470 0x7f8bbc19a880 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:22.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.425+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1305186524 >> 172.21.15.186:0/1305186524 conn(0x7f8bbc078d50 msgr2=0x7f8bbc109fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:22.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.425+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1305186524 shutdown_connections 2024-05-27T07:29:22.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:22.425+0000 7f8bc1efe700 1 -- 172.21.15.186:0/1305186524 wait complete. 2024-05-27T07:29:22.990 DEBUG:teuthology.orchestra.run.smithi163:mon.b> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.b.service 2024-05-27T07:29:22.993 DEBUG:teuthology.orchestra.run.smithi186:mon.c> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.c.service 2024-05-27T07:29:22.996 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T07:29:22.996 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T07:29:23.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:22 smithi114 ceph-mon[32038]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:23.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:22 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:23.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:22 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:23.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:22 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:23.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:22 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:23.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:22 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:23.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:22 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:23.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:22 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:24.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:23 smithi114 ceph-mon[32038]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi163:172.21.15.163=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:29:24.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:23 smithi114 ceph-mon[32038]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi163:172.21.15.163=b;smithi186:172.21.15.186=c;count:3 2024-05-27T07:29:24.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:23 smithi114 ceph-mon[32038]: Deploying daemon mon.c on smithi186 2024-05-27T07:29:25.020 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:29:25.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:24 smithi114 ceph-mon[32038]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:26.719 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:26 smithi186 podman[36601]: 2024-05-27 07:29:26.632661866 +0000 UTC m=+0.254609918 container create 497b6738db89c9e8eb4b680467c2da43ef047f64336f6763c68bbac0501fe068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-c, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-27T07:29:26.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.930+0000 7fde4600f700 1 -- 172.21.15.186:0/999279397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde40103ec0 msgr2=0x7fde401042d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.930+0000 7fde4600f700 1 --2- 172.21.15.186:0/999279397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde40103ec0 0x7fde401042d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fde30009910 tx=0x7fde30009c20 comp rx=0 tx=0).stop 2024-05-27T07:29:26.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.930+0000 7fde4600f700 1 -- 172.21.15.186:0/999279397 shutdown_connections 2024-05-27T07:29:26.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.930+0000 7fde4600f700 1 --2- 172.21.15.186:0/999279397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde40103ec0 0x7fde401042d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.930+0000 7fde4600f700 1 -- 172.21.15.186:0/999279397 >> 172.21.15.186:0/999279397 conn(0x7fde400ff790 msgr2=0x7fde40101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:26.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.930+0000 7fde4600f700 1 -- 172.21.15.186:0/999279397 shutdown_connections 2024-05-27T07:29:26.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.930+0000 7fde4600f700 1 -- 172.21.15.186:0/999279397 wait complete. 2024-05-27T07:29:26.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.931+0000 7fde4600f700 1 Processor -- start 2024-05-27T07:29:26.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.931+0000 7fde4600f700 1 -- start start 2024-05-27T07:29:26.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.932+0000 7fde4600f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde40103ec0 0x7fde4019d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.932+0000 7fde4600f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fde40077070 con 0x7fde40103ec0 2024-05-27T07:29:26.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.932+0000 7fde3f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde40103ec0 0x7fde4019d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.932+0000 7fde3f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde40103ec0 0x7fde4019d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.186:54810/0 (socket says 172.21.15.186:54810) 2024-05-27T07:29:26.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.932+0000 7fde3f7fe700 1 -- 172.21.15.186:0/4102068147 learned_addr learned my addr 172.21.15.186:0/4102068147 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-27T07:29:26.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.932+0000 7fde3f7fe700 1 -- 172.21.15.186:0/4102068147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde4019da10 con 0x7fde40103ec0 2024-05-27T07:29:26.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.933+0000 7fde3f7fe700 1 --2- 172.21.15.186:0/4102068147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde40103ec0 0x7fde4019d4d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fde300056d0 tx=0x7fde300057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.933+0000 7fde3d7fa700 1 -- 172.21.15.186:0/4102068147 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde30026070 con 0x7fde40103ec0 2024-05-27T07:29:26.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.933+0000 7fde4600f700 1 -- 172.21.15.186:0/4102068147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde4019dd00 con 0x7fde40103ec0 2024-05-27T07:29:26.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.933+0000 7fde3d7fa700 1 -- 172.21.15.186:0/4102068147 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde3000bba0 con 0x7fde40103ec0 2024-05-27T07:29:26.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.933+0000 7fde3d7fa700 1 -- 172.21.15.186:0/4102068147 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde30003920 con 0x7fde40103ec0 2024-05-27T07:29:26.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.933+0000 7fde4600f700 1 -- 172.21.15.186:0/4102068147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde401976b0 con 0x7fde40103ec0 2024-05-27T07:29:26.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.934+0000 7fde3d7fa700 1 -- 172.21.15.186:0/4102068147 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7fde30029070 con 0x7fde40103ec0 2024-05-27T07:29:26.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.934+0000 7fde4600f700 1 -- 172.21.15.186:0/4102068147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde40105370 con 0x7fde40103ec0 2024-05-27T07:29:26.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.934+0000 7fde3d7fa700 1 --2- 172.21.15.186:0/4102068147 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fde2c042970 0x7fde2c044e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.934+0000 7fde3d7fa700 1 -- 172.21.15.186:0/4102068147 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fde3001e070 con 0x7fde40103ec0 2024-05-27T07:29:26.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.934+0000 7fde36ffd700 1 --2- 172.21.15.186:0/4102068147 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fde2c042970 0x7fde2c044e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.935+0000 7fde36ffd700 1 --2- 172.21.15.186:0/4102068147 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fde2c042970 0x7fde2c044e80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fde28006fd0 tx=0x7fde28006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:26.940+0000 7fde3d7fa700 1 -- 172.21.15.186:0/4102068147 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fde30013070 con 0x7fde40103ec0 2024-05-27T07:29:27.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:26 smithi114 ceph-mon[32038]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:27.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:27.090+0000 7fde3d7fa700 1 -- 172.21.15.186:0/4102068147 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fde30013250 con 0x7fde40103ec0 2024-05-27T07:29:27.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:27.237+0000 7fde4600f700 1 -- 172.21.15.186:0/4102068147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fde40198bc0 con 0x7fde40103ec0 2024-05-27T07:29:27.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 podman[36601]: 2024-05-27 07:29:27.020507349 +0000 UTC m=+0.642455361 container init 497b6738db89c9e8eb4b680467c2da43ef047f64336f6763c68bbac0501fe068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-05-27T07:29:27.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 podman[36601]: 2024-05-27 07:29:27.023857855 +0000 UTC m=+0.645805865 container start 497b6738db89c9e8eb4b680467c2da43ef047f64336f6763c68bbac0501fe068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T07:29:27.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: set uid:gid to 167:167 (ceph:ceph) 2024-05-27T07:29:27.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev), process ceph-mon, pid 2 2024-05-27T07:29:27.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: pidfile_write: ignore empty --pid-file 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: load: jerasure load: lrc 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: RocksDB version: 7.9.2 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Git sha 0 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Compile date 2024-05-21 07:00:12 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: DB SUMMARY 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: DB Session ID: ITRMKIM3VEZHRY32YBCX 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: CURRENT file: CURRENT 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: IDENTITY file: IDENTITY 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.error_if_exists: 0 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.create_if_missing: 0 2024-05-27T07:29:27.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.paranoid_checks: 1 2024-05-27T07:29:27.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-27T07:29:27.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-27T07:29:27.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-27T07:29:27.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.env: 0x55fce20a1500 2024-05-27T07:29:27.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.fs: PosixFileSystem 2024-05-27T07:29:27.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.info_log: 0x55fce489ffe0 2024-05-27T07:29:27.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_file_opening_threads: 16 2024-05-27T07:29:27.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.statistics: (nil) 2024-05-27T07:29:27.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.use_fsync: 0 2024-05-27T07:29:27.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_log_file_size: 0 2024-05-27T07:29:27.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-27T07:29:27.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-27T07:29:27.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.keep_log_file_num: 1000 2024-05-27T07:29:27.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.recycle_log_file_num: 0 2024-05-27T07:29:27.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.allow_fallocate: 1 2024-05-27T07:29:27.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.allow_mmap_reads: 0 2024-05-27T07:29:27.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.allow_mmap_writes: 0 2024-05-27T07:29:27.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.use_direct_reads: 0 2024-05-27T07:29:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-27T07:29:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.create_missing_column_families: 0 2024-05-27T07:29:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.db_log_dir: 2024-05-27T07:29:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.wal_dir: 2024-05-27T07:29:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-27T07:29:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-27T07:29:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-27T07:29:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-27T07:29:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-27T07:29:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-27T07:29:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.advise_random_on_open: 1 2024-05-27T07:29:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.db_write_buffer_size: 0 2024-05-27T07:29:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.write_buffer_manager: 0x55fce48708c0 2024-05-27T07:29:27.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-27T07:29:27.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-27T07:29:27.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-27T07:29:27.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.rate_limiter: (nil) 2024-05-27T07:29:27.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-27T07:29:27.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.wal_recovery_mode: 2 2024-05-27T07:29:27.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.enable_thread_tracking: 0 2024-05-27T07:29:27.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.enable_pipelined_write: 0 2024-05-27T07:29:27.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.unordered_write: 0 2024-05-27T07:29:27.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-27T07:29:27.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-27T07:29:27.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.row_cache: None 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.wal_filter: None 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.allow_ingest_behind: 0 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.two_write_queues: 0 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.manual_wal_flush: 0 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.wal_compression: 0 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.atomic_flush: 0 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.log_readahead_size: 0 2024-05-27T07:29:27.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.best_efforts_recovery: 0 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.allow_data_in_errors: 0 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.db_host_id: __hostname__ 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_background_jobs: 2 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_background_compactions: -1 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_subcompactions: 1 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_total_wal_size: 0 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-27T07:29:27.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_open_files: -1 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.bytes_per_sync: 0 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compaction_readahead_size: 0 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_background_flushes: -1 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Compression algorithms supported: 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: kZSTD supported: 0 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: kXpressCompression supported: 0 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: kLZ4HCCompression supported: 1 2024-05-27T07:29:27.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: kZlibCompression supported: 1 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: kSnappyCompression supported: 1 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: kLZ4Compression supported: 1 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: kBZip2Compression supported: 0 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.merge_operator: 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compaction_filter: None 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compaction_filter_factory: None 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.sst_partitioner_factory: None 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fce49a61c0) 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout: cache_index_and_filter_blocks: 1 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout: pin_top_level_index_and_filter: 1 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout: index_type: 0 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout: data_block_index_type: 0 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout: index_shortening: 1 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout: checksum: 4 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout: no_block_cache: 0 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_cache: 0x55fce48a31f0 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_cache_name: BinnedLRUCache 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_cache_options: 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: capacity : 536870912 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: num_shard_bits : 4 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: strict_capacity_limit : 0 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: high_pri_pool_ratio: 0.000 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_cache_compressed: (nil) 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: persistent_cache: (nil) 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_size: 4096 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_size_deviation: 10 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_restart_interval: 16 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: index_block_restart_interval: 1 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: metadata_block_size: 4096 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: partition_filters: 0 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: use_delta_encoding: 1 2024-05-27T07:29:27.481 INFO:journalctl@ceph.mon.c.smithi186.stdout: filter_policy: bloomfilter 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout: whole_key_filtering: 1 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout: verify_compression: 0 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout: read_amp_bytes_per_bit: 0 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout: format_version: 5 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout: enable_index_compression: 1 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_align: 0 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout: max_auto_readahead_size: 262144 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout: prepopulate_block_cache: 0 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout: initial_auto_readahead_size: 8192 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout: num_file_reads_for_auto_readahead: 2 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.write_buffer_size: 33554432 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_write_buffer_number: 2 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compression: NoCompression 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.bottommost_compression: Disabled 2024-05-27T07:29:27.482 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.prefix_extractor: nullptr 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.num_levels: 7 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:29:27.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compression_opts.level: 32767 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compression_opts.strategy: 0 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compression_opts.enabled: false 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.target_file_size_base: 67108864 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-27T07:29:27.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.arena_block_size: 1048576 2024-05-27T07:29:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-27T07:29:27.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-27T07:29:27.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.disable_auto_compactions: 0 2024-05-27T07:29:27.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-27T07:29:27.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-27T07:29:27.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-27T07:29:27.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-27T07:29:27.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-27T07:29:27.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-27T07:29:27.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-27T07:29:27.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-27T07:29:27.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-27T07:29:27.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-27T07:29:27.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.inplace_update_support: 0 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.bloom_locality: 0 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.max_successive_merges: 0 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.paranoid_file_checks: 0 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.force_consistency_checks: 1 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.report_bg_io_stats: 0 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.ttl: 2592000 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-27T07:29:27.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.enable_blob_files: false 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.min_blob_size: 0 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.blob_file_size: 268435456 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.blob_file_starting_level: 0 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2af278bc-ff6a-457c-9654-19c4a5e81cee 2024-05-27T07:29:27.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794967064321, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794967064750, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716794967, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2af278bc-ff6a-457c-9654-19c4a5e81cee", "db_session_id": "ITRMKIM3VEZHRY32YBCX", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794967064896, "job": 1, "event": "recovery_finished"} 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fce4887c00 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: DB pointer 0x55fce967a000 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout: ** DB Stats ** 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:29:27.489 INFO:journalctl@ceph.mon.c.smithi186.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:29:27.490 INFO:journalctl@ceph.mon.c.smithi186.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-27T07:29:27.490 INFO:journalctl@ceph.mon.c.smithi186.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:29:27.490 INFO:journalctl@ceph.mon.c.smithi186.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:29:27.490 INFO:journalctl@ceph.mon.c.smithi186.stdout: 2024-05-27T07:29:27.490 INFO:journalctl@ceph.mon.c.smithi186.stdout: ** Compaction Stats [default] ** 2024-05-27T07:29:27.490 INFO:journalctl@ceph.mon.c.smithi186.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:29:27.490 INFO:journalctl@ceph.mon.c.smithi186.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-27T07:29:27.490 INFO:journalctl@ceph.mon.c.smithi186.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:29:27.490 INFO:journalctl@ceph.mon.c.smithi186.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:29:27.490 INFO:journalctl@ceph.mon.c.smithi186.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:29:27.490 INFO:journalctl@ceph.mon.c.smithi186.stdout: 2024-05-27T07:29:27.490 INFO:journalctl@ceph.mon.c.smithi186.stdout: ** Compaction Stats [default] ** 2024-05-27T07:29:27.490 INFO:journalctl@ceph.mon.c.smithi186.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:29:27.491 INFO:journalctl@ceph.mon.c.smithi186.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout: Block cache BinnedLRUCache@0x55fce48a31f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout: 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: using public_addr v2:172.21.15.186:0/0 -> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: starting mon.c rank -1 at public addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] at bind addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(???) e0 preinit fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(synchronizing).mds e1 new map 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout: e1 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:29:27.492 INFO:journalctl@ceph.mon.c.smithi186.stdout: legacy client fscid: -1 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout: 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout: No filesystems configured 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:29:27.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.494 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:27.494 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:29:27.494 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: Deploying cephadm binary to smithi186 2024-05-27T07:29:27.494 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.494 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.494 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.494 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.494 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi163", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:29:27.494 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:27.494 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:27.494 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: Added host smithi186 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1640421888' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1640421888' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:27.496 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:27.497 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-27T07:29:27.497 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:27.497 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.497 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.497 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.497 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:29:27.497 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:29:27.497 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:27.497 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.497 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:27.497 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:27.497 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:27.497 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:27.498 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:27.498 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:27.498 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi163:172.21.15.163=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:29:27.498 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi163:172.21.15.163=b;smithi186:172.21.15.186=c;count:3 2024-05-27T07:29:27.498 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: Deploying daemon mon.c on smithi186 2024-05-27T07:29:27.498 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:27.498 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:27.498 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 ceph-mon[36647]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-27T07:29:27.498 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 bash[36601]: 497b6738db89c9e8eb4b680467c2da43ef047f64336f6763c68bbac0501fe068 2024-05-27T07:29:27.498 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:27 smithi186 systemd[1]: Started Ceph mon.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:29:32.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:32.102+0000 7fde3d7fa700 1 -- 172.21.15.186:0/4102068147 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7fde3002ada0 con 0x7fde40103ec0 2024-05-27T07:29:32.093 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:29:32.093 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":2,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","modified":"2024-05-27T07:29:27.091200Z","created":"2024-05-27T07:27:38.185727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-27T07:29:32.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:32.105+0000 7fde4600f700 1 -- 172.21.15.186:0/4102068147 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fde2c042970 msgr2=0x7fde2c044e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:32.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:32.105+0000 7fde4600f700 1 --2- 172.21.15.186:0/4102068147 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fde2c042970 0x7fde2c044e80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fde28006fd0 tx=0x7fde28006e40 comp rx=0 tx=0).stop 2024-05-27T07:29:32.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:32.105+0000 7fde4600f700 1 -- 172.21.15.186:0/4102068147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde40103ec0 msgr2=0x7fde4019d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:32.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:32.105+0000 7fde4600f700 1 --2- 172.21.15.186:0/4102068147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde40103ec0 0x7fde4019d4d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fde300056d0 tx=0x7fde300057b0 comp rx=0 tx=0).stop 2024-05-27T07:29:32.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:32.106+0000 7fde4600f700 1 -- 172.21.15.186:0/4102068147 shutdown_connections 2024-05-27T07:29:32.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:32.106+0000 7fde4600f700 1 --2- 172.21.15.186:0/4102068147 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fde2c042970 0x7fde2c044e80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:32.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:32.106+0000 7fde4600f700 1 --2- 172.21.15.186:0/4102068147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde40103ec0 0x7fde4019d4d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:32.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:32.106+0000 7fde4600f700 1 -- 172.21.15.186:0/4102068147 >> 172.21.15.186:0/4102068147 conn(0x7fde400ff790 msgr2=0x7fde40102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:32.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:32.106+0000 7fde4600f700 1 -- 172.21.15.186:0/4102068147 shutdown_connections 2024-05-27T07:29:32.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:32.106+0000 7fde4600f700 1 -- 172.21.15.186:0/4102068147 wait complete. 2024-05-27T07:29:32.097 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 2 2024-05-27T07:29:32.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:29:32.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: mon.a calling monitor election 2024-05-27T07:29:32.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:32.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: from='client.? 172.21.15.186:0/4102068147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:29:32.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:32.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:32.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:32.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: mon.c calling monitor election 2024-05-27T07:29:32.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:32.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:32.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:32.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: monmap epoch 2 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: last_changed 2024-05-27T07:29:27.091200+0000 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: created 2024-05-27T07:27:38.185727+0000 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: min_mon_release 19 (squid) 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: election_strategy: 1 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: fsmap 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: mgrmap e13: x(active, since 48s) 2024-05-27T07:29:32.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: overall HEALTH_OK 2024-05-27T07:29:32.357 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:32.357 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:32 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:32.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:29:32.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: mon.a calling monitor election 2024-05-27T07:29:32.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:32.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: from='client.? 172.21.15.186:0/4102068147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:29:32.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:32.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:32.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:32.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: mon.c calling monitor election 2024-05-27T07:29:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T07:29:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: monmap epoch 2 2024-05-27T07:29:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:29:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: last_changed 2024-05-27T07:29:27.091200+0000 2024-05-27T07:29:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: created 2024-05-27T07:27:38.185727+0000 2024-05-27T07:29:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: min_mon_release 19 (squid) 2024-05-27T07:29:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: election_strategy: 1 2024-05-27T07:29:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-05-27T07:29:32.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-27T07:29:32.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: fsmap 2024-05-27T07:29:32.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:29:32.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: mgrmap e13: x(active, since 48s) 2024-05-27T07:29:32.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: overall HEALTH_OK 2024-05-27T07:29:32.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:32.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:32 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:33.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:33 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:33.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:33 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:33.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:33 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:33.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:33 smithi186 ceph-mon[36647]: Deploying daemon mon.b on smithi163 2024-05-27T07:29:33.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:33 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:33.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:33 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:33.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:33 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:33.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:33 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:33.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:33 smithi114 ceph-mon[32038]: Deploying daemon mon.b on smithi163 2024-05-27T07:29:33.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:33 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:33.630 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T07:29:33.631 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T07:29:33.868 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:29:34.308 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:34 smithi186 ceph-mon[36647]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:34.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:34 smithi114 ceph-mon[32038]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:35.310 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.319+0000 7f7edd863700 1 -- 172.21.15.186:0/1063394020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ed8103ec0 msgr2=0x7f7ed81042d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.319+0000 7f7edd863700 1 --2- 172.21.15.186:0/1063394020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ed8103ec0 0x7f7ed81042d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009910 tx=0x7f7ec8009c20 comp rx=0 tx=0).stop 2024-05-27T07:29:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.319+0000 7f7edd863700 1 -- 172.21.15.186:0/1063394020 shutdown_connections 2024-05-27T07:29:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.319+0000 7f7edd863700 1 --2- 172.21.15.186:0/1063394020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ed8103ec0 0x7f7ed81042d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.319+0000 7f7edd863700 1 -- 172.21.15.186:0/1063394020 >> 172.21.15.186:0/1063394020 conn(0x7f7ed80ff790 msgr2=0x7f7ed8101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.319+0000 7f7edd863700 1 -- 172.21.15.186:0/1063394020 shutdown_connections 2024-05-27T07:29:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.319+0000 7f7edd863700 1 -- 172.21.15.186:0/1063394020 wait complete. 2024-05-27T07:29:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.320+0000 7f7edd863700 1 Processor -- start 2024-05-27T07:29:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.320+0000 7f7edd863700 1 -- start start 2024-05-27T07:29:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.320+0000 7f7edd863700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ed8103ec0 0x7f7ed819f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.320+0000 7f7edd863700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ed819fd60 0x7f7ed8199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.320+0000 7f7edd863700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed810e1b0 con 0x7f7ed8103ec0 2024-05-27T07:29:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.320+0000 7f7edd863700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed810e030 con 0x7f7ed819fd60 2024-05-27T07:29:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.321+0000 7f7ece7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ed819fd60 0x7f7ed8199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:35.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.321+0000 7f7ece7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ed819fd60 0x7f7ed8199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:55454/0 (socket says 172.21.15.186:55454) 2024-05-27T07:29:35.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.321+0000 7f7ece7fc700 1 -- 172.21.15.186:0/3416992151 learned_addr learned my addr 172.21.15.186:0/3416992151 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-27T07:29:35.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.321+0000 7f7ece7fc700 1 -- 172.21.15.186:0/3416992151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ed819fd60 msgr2=0x7f7ed8199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-27T07:29:35.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.321+0000 7f7ece7fc700 1 -- 172.21.15.186:0/3416992151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ed819fd60 msgr2=0x7f7ed8199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-27T07:29:35.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.321+0000 7f7ece7fc700 1 --2- 172.21.15.186:0/3416992151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ed819fd60 0x7f7ed8199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T07:29:35.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.321+0000 7f7ed6ffd700 1 --2- 172.21.15.186:0/3416992151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ed8103ec0 0x7f7ed819f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:35.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.321+0000 7f7ece7fc700 1 --2- 172.21.15.186:0/3416992151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ed819fd60 0x7f7ed8199a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:29:35.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.321+0000 7f7ed6ffd700 1 -- 172.21.15.186:0/3416992151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ed819fd60 msgr2=0x7f7ed8199a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:29:35.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.321+0000 7f7ed6ffd700 1 --2- 172.21.15.186:0/3416992151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ed819fd60 0x7f7ed8199a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.321+0000 7f7ed6ffd700 1 -- 172.21.15.186:0/3416992151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed81a0330 con 0x7f7ed8103ec0 2024-05-27T07:29:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.321+0000 7f7ed6ffd700 1 --2- 172.21.15.186:0/3416992151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ed8103ec0 0x7f7ed819f820 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8014000 tx=0x7f7ec8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.322+0000 7f7edc861700 1 -- 172.21.15.186:0/3416992151 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7ec8005dc0 con 0x7f7ed8103ec0 2024-05-27T07:29:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.322+0000 7f7edc861700 1 -- 172.21.15.186:0/3416992151 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ec800b750 con 0x7f7ed8103ec0 2024-05-27T07:29:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.322+0000 7f7edc861700 1 -- 172.21.15.186:0/3416992151 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7ec80039a0 con 0x7f7ed8103ec0 2024-05-27T07:29:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.322+0000 7f7edd863700 1 -- 172.21.15.186:0/3416992151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed8199fe0 con 0x7f7ed8103ec0 2024-05-27T07:29:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.322+0000 7f7edd863700 1 -- 172.21.15.186:0/3416992151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed819a570 con 0x7f7ed8103ec0 2024-05-27T07:29:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.322+0000 7f7edd863700 1 -- 172.21.15.186:0/3416992151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed819a880 con 0x7f7ed8103ec0 2024-05-27T07:29:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.323+0000 7f7edc861700 1 -- 172.21.15.186:0/3416992151 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f7ec8029070 con 0x7f7ed8103ec0 2024-05-27T07:29:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.323+0000 7f7edc861700 1 --2- 172.21.15.186:0/3416992151 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7ec403e300 0x7f7ec4040810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.323+0000 7f7edc861700 1 -- 172.21.15.186:0/3416992151 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7ec801e070 con 0x7f7ed8103ec0 2024-05-27T07:29:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.323+0000 7f7ece7fc700 1 --2- 172.21.15.186:0/3416992151 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7ec403e300 0x7f7ec4040810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.324+0000 7f7ece7fc700 1 --2- 172.21.15.186:0/3416992151 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7ec403e300 0x7f7ec4040810 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8103360 tx=0x7f7ec0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:35.318 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.329+0000 7f7edc861700 1 -- 172.21.15.186:0/3416992151 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7ec8013070 con 0x7f7ed8103ec0 2024-05-27T07:29:35.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.614+0000 7f7edd863700 1 -- 172.21.15.186:0/3416992151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7ed819b010 con 0x7f7ed8103ec0 2024-05-27T07:29:35.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.615+0000 7f7edc861700 1 -- 172.21.15.186:0/3416992151 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7f7ec8030410 con 0x7f7ed8103ec0 2024-05-27T07:29:35.605 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:29:35.605 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":2,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","modified":"2024-05-27T07:29:27.091200Z","created":"2024-05-27T07:27:38.185727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-27T07:29:35.607 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.617+0000 7f7edd863700 1 -- 172.21.15.186:0/3416992151 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7ec403e300 msgr2=0x7f7ec4040810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:35.607 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.617+0000 7f7edd863700 1 --2- 172.21.15.186:0/3416992151 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7ec403e300 0x7f7ec4040810 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8103360 tx=0x7f7ec0008040 comp rx=0 tx=0).stop 2024-05-27T07:29:35.608 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.617+0000 7f7edd863700 1 -- 172.21.15.186:0/3416992151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ed8103ec0 msgr2=0x7f7ed819f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:35.608 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.617+0000 7f7edd863700 1 --2- 172.21.15.186:0/3416992151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ed8103ec0 0x7f7ed819f820 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8014000 tx=0x7f7ec8005030 comp rx=0 tx=0).stop 2024-05-27T07:29:35.608 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.618+0000 7f7edd863700 1 -- 172.21.15.186:0/3416992151 shutdown_connections 2024-05-27T07:29:35.608 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.618+0000 7f7edd863700 1 --2- 172.21.15.186:0/3416992151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ed8103ec0 0x7f7ed819f820 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:35.608 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.618+0000 7f7edd863700 1 --2- 172.21.15.186:0/3416992151 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7ec403e300 0x7f7ec4040810 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:35.608 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.618+0000 7f7edd863700 1 --2- 172.21.15.186:0/3416992151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ed819fd60 0x7f7ed8199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:35.608 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.618+0000 7f7edd863700 1 -- 172.21.15.186:0/3416992151 >> 172.21.15.186:0/3416992151 conn(0x7f7ed80ff790 msgr2=0x7f7ed81099f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:35.608 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.618+0000 7f7edd863700 1 -- 172.21.15.186:0/3416992151 shutdown_connections 2024-05-27T07:29:35.608 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:35.618+0000 7f7edd863700 1 -- 172.21.15.186:0/3416992151 wait complete. 2024-05-27T07:29:35.608 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 2 2024-05-27T07:29:35.961 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:35 smithi163 podman[36432]: 2024-05-27 07:29:35.970817235 +0000 UTC m=+0.290138758 container init 61a1565b03aed0cadf900de1da0cfe8ea0dddc3559430fac53d3abf409abab75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-27T07:29:36.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:35 smithi163 podman[36432]: 2024-05-27 07:29:35.973882493 +0000 UTC m=+0.293204023 container start 61a1565b03aed0cadf900de1da0cfe8ea0dddc3559430fac53d3abf409abab75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-b, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-27T07:29:36.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: set uid:gid to 167:167 (ceph:ceph) 2024-05-27T07:29:36.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev), process ceph-mon, pid 2 2024-05-27T07:29:36.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pidfile_write: ignore empty --pid-file 2024-05-27T07:29:36.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: load: jerasure load: lrc 2024-05-27T07:29:36.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: RocksDB version: 7.9.2 2024-05-27T07:29:36.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Git sha 0 2024-05-27T07:29:36.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Compile date 2024-05-21 07:00:12 2024-05-27T07:29:36.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: DB SUMMARY 2024-05-27T07:29:36.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: DB Session ID: FCVKHWK3CIBYINYIFHBS 2024-05-27T07:29:36.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: CURRENT file: CURRENT 2024-05-27T07:29:36.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: IDENTITY file: IDENTITY 2024-05-27T07:29:36.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-27T07:29:36.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-05-27T07:29:36.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-05-27T07:29:36.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.error_if_exists: 0 2024-05-27T07:29:36.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.create_if_missing: 0 2024-05-27T07:29:36.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.paranoid_checks: 1 2024-05-27T07:29:36.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-27T07:29:36.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-27T07:29:36.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-27T07:29:36.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.env: 0x56509ba19500 2024-05-27T07:29:36.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.fs: PosixFileSystem 2024-05-27T07:29:36.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.info_log: 0x56509cb19fe0 2024-05-27T07:29:36.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_file_opening_threads: 16 2024-05-27T07:29:36.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.statistics: (nil) 2024-05-27T07:29:36.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.use_fsync: 0 2024-05-27T07:29:36.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_log_file_size: 0 2024-05-27T07:29:36.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-27T07:29:36.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-27T07:29:36.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.keep_log_file_num: 1000 2024-05-27T07:29:36.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.recycle_log_file_num: 0 2024-05-27T07:29:36.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.allow_fallocate: 1 2024-05-27T07:29:36.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.allow_mmap_reads: 0 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.allow_mmap_writes: 0 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.use_direct_reads: 0 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.create_missing_column_families: 0 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.db_log_dir: 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.wal_dir: 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.advise_random_on_open: 1 2024-05-27T07:29:36.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.db_write_buffer_size: 0 2024-05-27T07:29:36.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.write_buffer_manager: 0x56509caea8c0 2024-05-27T07:29:36.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-27T07:29:36.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-27T07:29:36.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-27T07:29:36.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.rate_limiter: (nil) 2024-05-27T07:29:36.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-27T07:29:36.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.wal_recovery_mode: 2 2024-05-27T07:29:36.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.enable_thread_tracking: 0 2024-05-27T07:29:36.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.enable_pipelined_write: 0 2024-05-27T07:29:36.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.unordered_write: 0 2024-05-27T07:29:36.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-27T07:29:36.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-27T07:29:36.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-27T07:29:36.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-27T07:29:36.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.row_cache: None 2024-05-27T07:29:36.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.wal_filter: None 2024-05-27T07:29:36.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-27T07:29:36.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.allow_ingest_behind: 0 2024-05-27T07:29:36.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.two_write_queues: 0 2024-05-27T07:29:36.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.manual_wal_flush: 0 2024-05-27T07:29:36.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.wal_compression: 0 2024-05-27T07:29:36.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.atomic_flush: 0 2024-05-27T07:29:36.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-27T07:29:36.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-27T07:29:36.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-27T07:29:36.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.log_readahead_size: 0 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.best_efforts_recovery: 0 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.allow_data_in_errors: 0 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.db_host_id: __hostname__ 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_background_jobs: 2 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_background_compactions: -1 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_subcompactions: 1 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_total_wal_size: 0 2024-05-27T07:29:36.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_open_files: -1 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.bytes_per_sync: 0 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compaction_readahead_size: 0 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_background_flushes: -1 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Compression algorithms supported: 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: kZSTD supported: 0 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: kXpressCompression supported: 0 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: kLZ4HCCompression supported: 1 2024-05-27T07:29:36.225 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: kZlibCompression supported: 1 2024-05-27T07:29:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: kSnappyCompression supported: 1 2024-05-27T07:29:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: kLZ4Compression supported: 1 2024-05-27T07:29:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: kBZip2Compression supported: 0 2024-05-27T07:29:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-27T07:29:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-27T07:29:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-05-27T07:29:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-27T07:29:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-27T07:29:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.merge_operator: 2024-05-27T07:29:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compaction_filter: None 2024-05-27T07:29:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compaction_filter_factory: None 2024-05-27T07:29:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.sst_partitioner_factory: None 2024-05-27T07:29:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56509cc201c0) 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout: cache_index_and_filter_blocks: 1 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout: pin_top_level_index_and_filter: 1 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout: index_type: 0 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout: data_block_index_type: 0 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout: index_shortening: 1 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout: checksum: 4 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout: no_block_cache: 0 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_cache: 0x56509cb1d1f0 2024-05-27T07:29:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_cache_name: BinnedLRUCache 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_cache_options: 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: capacity : 536870912 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: num_shard_bits : 4 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: strict_capacity_limit : 0 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: high_pri_pool_ratio: 0.000 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_cache_compressed: (nil) 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: persistent_cache: (nil) 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_size: 4096 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_size_deviation: 10 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_restart_interval: 16 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: index_block_restart_interval: 1 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: metadata_block_size: 4096 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: partition_filters: 0 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: use_delta_encoding: 1 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: filter_policy: bloomfilter 2024-05-27T07:29:36.228 INFO:journalctl@ceph.mon.b.smithi163.stdout: whole_key_filtering: 1 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout: verify_compression: 0 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout: read_amp_bytes_per_bit: 0 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout: format_version: 5 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout: enable_index_compression: 1 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_align: 0 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout: max_auto_readahead_size: 262144 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout: prepopulate_block_cache: 0 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout: initial_auto_readahead_size: 8192 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout: num_file_reads_for_auto_readahead: 2 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.write_buffer_size: 33554432 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_write_buffer_number: 2 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compression: NoCompression 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.bottommost_compression: Disabled 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.prefix_extractor: nullptr 2024-05-27T07:29:36.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-27T07:29:36.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.num_levels: 7 2024-05-27T07:29:36.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-27T07:29:36.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-27T07:29:36.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-27T07:29:36.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-27T07:29:36.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-27T07:29:36.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-27T07:29:36.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-27T07:29:36.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:29:36.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-27T07:29:36.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-27T07:29:36.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:29:36.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compression_opts.level: 32767 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compression_opts.strategy: 0 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compression_opts.enabled: false 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.target_file_size_base: 67108864 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-27T07:29:36.231 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.arena_block_size: 1048576 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-27T07:29:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.disable_auto_compactions: 0 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.inplace_update_support: 0 2024-05-27T07:29:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.bloom_locality: 0 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.max_successive_merges: 0 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.paranoid_file_checks: 0 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.force_consistency_checks: 1 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.report_bg_io_stats: 0 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.ttl: 2592000 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-27T07:29:36.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.enable_blob_files: false 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.min_blob_size: 0 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.blob_file_size: 268435456 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.blob_file_starting_level: 0 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 857a4f0a-ebfc-4b91-bb78-492caff4bdc3 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794976014239, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-27T07:29:36.235 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794976014678, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716794976, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "857a4f0a-ebfc-4b91-bb78-492caff4bdc3", "db_session_id": "FCVKHWK3CIBYINYIFHBS", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794976014760, "job": 1, "event": "recovery_finished"} 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56509cb01c00 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: DB pointer 0x5650a18f4000 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout: ** DB Stats ** 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:29:36.236 INFO:journalctl@ceph.mon.b.smithi163.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: ** Compaction Stats [default] ** 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: ** Compaction Stats [default] ** 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:29:36.237 INFO:journalctl@ceph.mon.b.smithi163.stdout: 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: Cumulative compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: Interval compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: Block cache BinnedLRUCache@0x56509cb1d1f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-27T07:29:36.238 INFO:journalctl@ceph.mon.b.smithi163.stdout: 2024-05-27T07:29:36.239 INFO:journalctl@ceph.mon.b.smithi163.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-27T07:29:36.239 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-05-27T07:29:36.239 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: using public_addr v2:172.21.15.163:0/0 -> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-05-27T07:29:36.239 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: starting mon.b rank -1 at public addrs [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] at bind addrs [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:29:36.239 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(???) e0 preinit fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:29:36.239 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(synchronizing).mds e1 new map 2024-05-27T07:29:36.239 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(synchronizing).mds e1 print_map 2024-05-27T07:29:36.239 INFO:journalctl@ceph.mon.b.smithi163.stdout: e1 2024-05-27T07:29:36.239 INFO:journalctl@ceph.mon.b.smithi163.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T07:29:36.239 INFO:journalctl@ceph.mon.b.smithi163.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout: legacy client fscid: -1 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout: 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout: No filesystems configured 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:29:36.240 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: Deploying cephadm binary to smithi186 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi163", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-27T07:29:36.241 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: Added host smithi186 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.242 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1640421888' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1640421888' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:36.243 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:36.244 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi163:172.21.15.163=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi163:172.21.15.163=b;smithi186:172.21.15.186=c;count:3 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: Deploying daemon mon.c on smithi186 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.a calling monitor election 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='client.? 172.21.15.186:0/4102068147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.c calling monitor election 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.245 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: monmap epoch 2 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: last_changed 2024-05-27T07:29:27.091200+0000 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: created 2024-05-27T07:27:38.185727+0000 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: min_mon_release 19 (squid) 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: election_strategy: 1 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: fsmap 2024-05-27T07:29:36.246 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:29:36.247 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mgrmap e13: x(active, since 48s) 2024-05-27T07:29:36.247 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: overall HEALTH_OK 2024-05-27T07:29:36.247 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.247 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.247 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:36.247 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:36.247 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:36.247 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: Deploying daemon mon.b on smithi163 2024-05-27T07:29:36.247 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:36.247 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:36.247 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 ceph-mon[36451]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-27T07:29:36.247 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 bash[36432]: 61a1565b03aed0cadf900de1da0cfe8ea0dddc3559430fac53d3abf409abab75 2024-05-27T07:29:36.247 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:36 smithi163 systemd[1]: Started Ceph mon.b for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:29:37.189 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T07:29:37.190 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T07:29:37.432 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:29:41.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:41.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:29:41.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: mon.a calling monitor election 2024-05-27T07:29:41.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:41.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: mon.c calling monitor election 2024-05-27T07:29:41.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:41.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: mon.b calling monitor election 2024-05-27T07:29:41.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:41.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T07:29:41.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: monmap epoch 3 2024-05-27T07:29:41.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:29:41.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: last_changed 2024-05-27T07:29:36.040477+0000 2024-05-27T07:29:41.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: created 2024-05-27T07:27:38.185727+0000 2024-05-27T07:29:41.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: min_mon_release 19 (squid) 2024-05-27T07:29:41.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: election_strategy: 1 2024-05-27T07:29:41.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-05-27T07:29:41.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-27T07:29:41.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: 2: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-05-27T07:29:41.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: fsmap 2024-05-27T07:29:41.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:29:41.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: mgrmap e13: x(active, since 57s) 2024-05-27T07:29:41.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: overall HEALTH_OK 2024-05-27T07:29:41.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:41.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:41.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:41.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:41.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:29:41.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: mon.a calling monitor election 2024-05-27T07:29:41.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:41.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: mon.c calling monitor election 2024-05-27T07:29:41.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:41.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: mon.b calling monitor election 2024-05-27T07:29:41.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:41.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T07:29:41.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: monmap epoch 3 2024-05-27T07:29:41.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:29:41.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: last_changed 2024-05-27T07:29:36.040477+0000 2024-05-27T07:29:41.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: created 2024-05-27T07:27:38.185727+0000 2024-05-27T07:29:41.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: min_mon_release 19 (squid) 2024-05-27T07:29:41.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: election_strategy: 1 2024-05-27T07:29:41.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-05-27T07:29:41.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-27T07:29:41.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: 2: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-05-27T07:29:41.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: fsmap 2024-05-27T07:29:41.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:29:41.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: mgrmap e13: x(active, since 57s) 2024-05-27T07:29:41.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: overall HEALTH_OK 2024-05-27T07:29:41.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:41.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:41.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:41.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:41.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:29:41.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: mon.a calling monitor election 2024-05-27T07:29:41.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:29:41.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: mon.c calling monitor election 2024-05-27T07:29:41.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:41.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: mon.b calling monitor election 2024-05-27T07:29:41.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:41.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:41.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T07:29:41.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: monmap epoch 3 2024-05-27T07:29:41.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:29:41.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: last_changed 2024-05-27T07:29:36.040477+0000 2024-05-27T07:29:41.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: created 2024-05-27T07:27:38.185727+0000 2024-05-27T07:29:41.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: min_mon_release 19 (squid) 2024-05-27T07:29:41.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: election_strategy: 1 2024-05-27T07:29:41.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-05-27T07:29:41.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-27T07:29:41.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: 2: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-05-27T07:29:41.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: fsmap 2024-05-27T07:29:41.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:29:41.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: mgrmap e13: x(active, since 57s) 2024-05-27T07:29:41.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: overall HEALTH_OK 2024-05-27T07:29:41.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:41.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:41.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:41.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.839+0000 7f2749c71700 1 -- 172.21.15.186:0/3089556722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2744102c10 msgr2=0x7f272c005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:41.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.839+0000 7f2749c71700 1 --2- 172.21.15.186:0/3089556722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2744102c10 0x7f272c005610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2734005fa0 tx=0x7f273400ffc0 comp rx=0 tx=0).stop 2024-05-27T07:29:41.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.840+0000 7f2749c71700 1 -- 172.21.15.186:0/3089556722 shutdown_connections 2024-05-27T07:29:41.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.840+0000 7f2749c71700 1 --2- 172.21.15.186:0/3089556722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2744102c10 0x7f272c005610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:41.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.840+0000 7f2749c71700 1 -- 172.21.15.186:0/3089556722 >> 172.21.15.186:0/3089556722 conn(0x7f27440797d0 msgr2=0x7f2744079c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:41.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.840+0000 7f2749c71700 1 -- 172.21.15.186:0/3089556722 shutdown_connections 2024-05-27T07:29:41.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.840+0000 7f2749c71700 1 -- 172.21.15.186:0/3089556722 wait complete. 2024-05-27T07:29:41.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.841+0000 7f2749c71700 1 Processor -- start 2024-05-27T07:29:41.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.841+0000 7f2749c71700 1 -- start start 2024-05-27T07:29:41.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.841+0000 7f2749c71700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2744102c10 0x7f27441a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:41.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.841+0000 7f2749c71700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27441a06a0 0x7f27441a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:41.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.841+0000 7f2749c71700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27441a4ed0 0x7f27441a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:41.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.841+0000 7f2749c71700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f27441121d0 con 0x7f2744102c10 2024-05-27T07:29:41.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.841+0000 7f2749c71700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2744112050 con 0x7f27441a4ed0 2024-05-27T07:29:41.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.841+0000 7f2749c71700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2744112350 con 0x7f27441a06a0 2024-05-27T07:29:41.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.841+0000 7f27437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2744102c10 0x7f27441a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:41.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.841+0000 7f27437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2744102c10 0x7f27441a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.186:47606/0 (socket says 172.21.15.186:47606) 2024-05-27T07:29:41.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.841+0000 7f27437fe700 1 -- 172.21.15.186:0/214371996 learned_addr learned my addr 172.21.15.186:0/214371996 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-27T07:29:41.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.841+0000 7f2743fff700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27441a4ed0 0x7f27441a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:41.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2742ffd700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27441a06a0 0x7f27441a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:41.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2743fff700 1 -- 172.21.15.186:0/214371996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27441a4ed0 msgr2=0x7f27441a52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-05-27T07:29:41.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2743fff700 1 -- 172.21.15.186:0/214371996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27441a4ed0 msgr2=0x7f27441a52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-27T07:29:41.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2743fff700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27441a4ed0 0x7f27441a52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T07:29:41.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2742ffd700 1 -- 172.21.15.186:0/214371996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27441a06a0 msgr2=0x7f27441a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-27T07:29:41.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2742ffd700 1 -- 172.21.15.186:0/214371996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27441a06a0 msgr2=0x7f27441a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-27T07:29:41.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2742ffd700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27441a06a0 0x7f27441a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T07:29:41.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2743fff700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27441a4ed0 0x7f27441a52e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:29:41.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f27437fe700 1 -- 172.21.15.186:0/214371996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27441a06a0 msgr2=0x7f27441a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:41.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2742ffd700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27441a06a0 0x7f27441a4990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:29:41.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f27437fe700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27441a06a0 0x7f27441a4990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:41.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f27437fe700 1 -- 172.21.15.186:0/214371996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27441a4ed0 msgr2=0x7f27441a52e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:29:41.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f27437fe700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27441a4ed0 0x7f27441a52e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:41.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f27437fe700 1 -- 172.21.15.186:0/214371996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27441acc40 con 0x7f2744102c10 2024-05-27T07:29:41.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f27437fe700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2744102c10 0x7f27441a0160 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f273400f990 tx=0x7f2734013d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:41.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2740ff9700 1 -- 172.21.15.186:0/214371996 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2734005530 con 0x7f2744102c10 2024-05-27T07:29:41.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2740ff9700 1 -- 172.21.15.186:0/214371996 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27340056d0 con 0x7f2744102c10 2024-05-27T07:29:41.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2749c71700 1 -- 172.21.15.186:0/214371996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27441ace70 con 0x7f2744102c10 2024-05-27T07:29:41.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2740ff9700 1 -- 172.21.15.186:0/214371996 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2734011870 con 0x7f2744102c10 2024-05-27T07:29:41.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.842+0000 7f2749c71700 1 -- 172.21.15.186:0/214371996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27441ad2d0 con 0x7f2744102c10 2024-05-27T07:29:41.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.843+0000 7f2740ff9700 1 -- 172.21.15.186:0/214371996 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f2734011a10 con 0x7f2744102c10 2024-05-27T07:29:41.837 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.843+0000 7f2749c71700 1 -- 172.21.15.186:0/214371996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2744064e10 con 0x7f2744102c10 2024-05-27T07:29:41.837 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.844+0000 7f2740ff9700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f273003e9a0 0x7f2730040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:41.837 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.844+0000 7f2740ff9700 1 -- 172.21.15.186:0/214371996 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2734054f00 con 0x7f2744102c10 2024-05-27T07:29:41.837 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.844+0000 7f2742ffd700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f273003e9a0 0x7f2730040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:41.837 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.844+0000 7f2742ffd700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f273003e9a0 0x7f2730040eb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f27441078d0 tx=0x7f2738008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:41.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:41.849+0000 7f2740ff9700 1 -- 172.21.15.186:0/214371996 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2734022070 con 0x7f2744102c10 2024-05-27T07:29:42.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.146+0000 7f2749c71700 1 -- 172.21.15.186:0/214371996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f27441ad5c0 con 0x7f2744102c10 2024-05-27T07:29:42.137 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.146+0000 7f2740ff9700 1 -- 172.21.15.186:0/214371996 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1289 (secure 0 0 0) 0x7f273401ba90 con 0x7f2744102c10 2024-05-27T07:29:42.137 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-27T07:29:42.137 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":3,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","modified":"2024-05-27T07:29:36.040477Z","created":"2024-05-27T07:27:38.185727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-05-27T07:29:42.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.148+0000 7f2749c71700 1 -- 172.21.15.186:0/214371996 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f273003e9a0 msgr2=0x7f2730040eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:42.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.148+0000 7f2749c71700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f273003e9a0 0x7f2730040eb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f27441078d0 tx=0x7f2738008040 comp rx=0 tx=0).stop 2024-05-27T07:29:42.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.148+0000 7f2749c71700 1 -- 172.21.15.186:0/214371996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2744102c10 msgr2=0x7f27441a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:42.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.149+0000 7f2749c71700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2744102c10 0x7f27441a0160 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f273400f990 tx=0x7f2734013d80 comp rx=0 tx=0).stop 2024-05-27T07:29:42.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.149+0000 7f2749c71700 1 -- 172.21.15.186:0/214371996 shutdown_connections 2024-05-27T07:29:42.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.149+0000 7f2749c71700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27441a4ed0 0x7f27441a52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:42.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.149+0000 7f2749c71700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2744102c10 0x7f27441a0160 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:42.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.149+0000 7f2749c71700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f273003e9a0 0x7f2730040eb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:42.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.149+0000 7f2749c71700 1 --2- 172.21.15.186:0/214371996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27441a06a0 0x7f27441a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:42.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.149+0000 7f2749c71700 1 -- 172.21.15.186:0/214371996 >> 172.21.15.186:0/214371996 conn(0x7f27440797d0 msgr2=0x7f2744101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:42.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.149+0000 7f2749c71700 1 -- 172.21.15.186:0/214371996 shutdown_connections 2024-05-27T07:29:42.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:42.149+0000 7f2749c71700 1 -- 172.21.15.186:0/214371996 wait complete. 2024-05-27T07:29:42.141 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 3 2024-05-27T07:29:42.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:42 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:42.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:42 smithi163 ceph-mon[36451]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:42.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:42 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:42.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:42 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:42.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:42 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:42.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:42 smithi186 ceph-mon[36647]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:42.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:42 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:42.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:42 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:42.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:42 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:42.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:42 smithi114 ceph-mon[32038]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:42.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:42 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:42.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:42 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:29:42.737 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-27T07:29:42.737 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-05-27T07:29:42.970 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:43.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:43 smithi114 ceph-mon[32038]: from='client.? 172.21.15.186:0/214371996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:29:43.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:43.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:43.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:43.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:43.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:43.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:43.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:43 smithi114 ceph-mon[32038]: Updating smithi114:/etc/ceph/ceph.conf 2024-05-27T07:29:43.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:43 smithi114 ceph-mon[32038]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-27T07:29:43.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:43 smithi114 ceph-mon[32038]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-27T07:29:43.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:43 smithi163 ceph-mon[36451]: from='client.? 172.21.15.186:0/214371996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:29:43.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:43.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:43.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:43.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:43.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:43.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:43.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:43 smithi163 ceph-mon[36451]: Updating smithi114:/etc/ceph/ceph.conf 2024-05-27T07:29:43.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:43 smithi163 ceph-mon[36451]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-27T07:29:43.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:43 smithi163 ceph-mon[36451]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-27T07:29:43.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:43 smithi186 ceph-mon[36647]: from='client.? 172.21.15.186:0/214371996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:29:43.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:43.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:43.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:43.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:43.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:43.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:43.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:43 smithi186 ceph-mon[36647]: Updating smithi114:/etc/ceph/ceph.conf 2024-05-27T07:29:43.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:43 smithi186 ceph-mon[36647]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-27T07:29:43.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:43 smithi186 ceph-mon[36647]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-27T07:29:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: Updating smithi114:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T07:29:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:29:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:44 smithi114 ceph-mon[32038]: Reconfiguring daemon mon.a on smithi114 2024-05-27T07:29:44.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:44.466 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: Updating smithi114:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T07:29:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:29:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:44 smithi163 ceph-mon[36451]: Reconfiguring daemon mon.a on smithi114 2024-05-27T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.628+0000 7fea2441f700 1 -- 172.21.15.114:0/729592873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea1c103de0 msgr2=0x7fea1c1041f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.628+0000 7fea2441f700 1 --2- 172.21.15.114:0/729592873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea1c103de0 0x7fea1c1041f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fea10009910 tx=0x7fea10009c20 comp rx=0 tx=0).stop 2024-05-27T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.628+0000 7fea2441f700 1 -- 172.21.15.114:0/729592873 shutdown_connections 2024-05-27T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.628+0000 7fea2441f700 1 --2- 172.21.15.114:0/729592873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea1c103de0 0x7fea1c1041f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.628+0000 7fea2441f700 1 -- 172.21.15.114:0/729592873 >> 172.21.15.114:0/729592873 conn(0x7fea1c0ff690 msgr2=0x7fea1c101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.628+0000 7fea2441f700 1 -- 172.21.15.114:0/729592873 shutdown_connections 2024-05-27T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.628+0000 7fea2441f700 1 -- 172.21.15.114:0/729592873 wait complete. 2024-05-27T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.629+0000 7fea2441f700 1 Processor -- start 2024-05-27T07:29:44.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.629+0000 7fea2441f700 1 -- start start 2024-05-27T07:29:44.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.629+0000 7fea2441f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea1c103de0 0x7fea1c075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.629+0000 7fea2441f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea1c075ff0 0x7fea1c076400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.629+0000 7fea2441f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea1c1a5e20 0x7fea1c1a8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.629+0000 7fea2441f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fea1c10e0d0 con 0x7fea1c1a5e20 2024-05-27T07:29:44.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.629+0000 7fea2441f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fea1c10df50 con 0x7fea1c103de0 2024-05-27T07:29:44.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.629+0000 7fea2441f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fea1c10e250 con 0x7fea1c075ff0 2024-05-27T07:29:44.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea229bc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea1c1a5e20 0x7fea1c1a8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea229bc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea1c1a5e20 0x7fea1c1a8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42154/0 (socket says 172.21.15.114:42154) 2024-05-27T07:29:44.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea229bc700 1 -- 172.21.15.114:0/779867961 learned_addr learned my addr 172.21.15.114:0/779867961 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:29:44.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea221bb700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea1c103de0 0x7fea1c075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea219ba700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea1c075ff0 0x7fea1c076400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea229bc700 1 -- 172.21.15.114:0/779867961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea1c075ff0 msgr2=0x7fea1c076400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea229bc700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea1c075ff0 0x7fea1c076400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea229bc700 1 -- 172.21.15.114:0/779867961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea1c103de0 msgr2=0x7fea1c075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea229bc700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea1c103de0 0x7fea1c075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea229bc700 1 -- 172.21.15.114:0/779867961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1c1a88c0 con 0x7fea1c1a5e20 2024-05-27T07:29:44.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea229bc700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea1c1a5e20 0x7fea1c1a8240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fea1400ba50 tx=0x7fea1400bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea0f7fe700 1 -- 172.21.15.114:0/779867961 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea14019070 con 0x7fea1c1a5e20 2024-05-27T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea0f7fe700 1 -- 172.21.15.114:0/779867961 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea14014440 con 0x7fea1c1a5e20 2024-05-27T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea0f7fe700 1 -- 172.21.15.114:0/779867961 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1400ce60 con 0x7fea1c1a5e20 2024-05-27T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.630+0000 7fea2441f700 1 -- 172.21.15.114:0/779867961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea1c1a8bb0 con 0x7fea1c1a5e20 2024-05-27T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.631+0000 7fea2441f700 1 -- 172.21.15.114:0/779867961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea1c1a90f0 con 0x7fea1c1a5e20 2024-05-27T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.631+0000 7fea2441f700 1 -- 172.21.15.114:0/779867961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea1c064e10 con 0x7fea1c1a5e20 2024-05-27T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.632+0000 7fea0f7fe700 1 -- 172.21.15.114:0/779867961 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7fea1401c070 con 0x7fea1c1a5e20 2024-05-27T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.632+0000 7fea0f7fe700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fea0803e9a0 0x7fea08040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.632+0000 7fea0f7fe700 1 -- 172.21.15.114:0/779867961 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fea14051d90 con 0x7fea1c1a5e20 2024-05-27T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.632+0000 7fea221bb700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fea0803e9a0 0x7fea08040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.632+0000 7fea221bb700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fea0803e9a0 0x7fea08040eb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fea10006010 tx=0x7fea1001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.637+0000 7fea0f7fe700 1 -- 172.21.15.114:0/779867961 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fea14022170 con 0x7fea1c1a5e20 2024-05-27T07:29:44.719 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:44.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: Updating smithi114:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:44.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:44.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:29:44.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:44.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T07:29:44.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:44.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:29:44.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:44.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:44 smithi186 ceph-mon[36647]: Reconfiguring daemon mon.a on smithi114 2024-05-27T07:29:44.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.842+0000 7fea2441f700 1 -- 172.21.15.114:0/779867961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fea1c1a9440 con 0x7fea1c1a5e20 2024-05-27T07:29:44.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.842+0000 7fea0f7fe700 1 -- 172.21.15.114:0/779867961 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7fea14012c40 con 0x7fea1c1a5e20 2024-05-27T07:29:44.834 INFO:teuthology.orchestra.run.smithi114.stdout:# minimal ceph.conf for 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:29:44.834 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-05-27T07:29:44.834 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:29:44.834 INFO:teuthology.orchestra.run.smithi114.stdout: mon_host = [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-27T07:29:44.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.845+0000 7fea2441f700 1 -- 172.21.15.114:0/779867961 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fea0803e9a0 msgr2=0x7fea08040eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.845+0000 7fea2441f700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fea0803e9a0 0x7fea08040eb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fea10006010 tx=0x7fea1001a040 comp rx=0 tx=0).stop 2024-05-27T07:29:44.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.845+0000 7fea2441f700 1 -- 172.21.15.114:0/779867961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea1c1a5e20 msgr2=0x7fea1c1a8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.845+0000 7fea2441f700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea1c1a5e20 0x7fea1c1a8240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fea1400ba50 tx=0x7fea1400bd60 comp rx=0 tx=0).stop 2024-05-27T07:29:44.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.845+0000 7fea2441f700 1 -- 172.21.15.114:0/779867961 shutdown_connections 2024-05-27T07:29:44.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.845+0000 7fea2441f700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea1c1a5e20 0x7fea1c1a8240 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.845+0000 7fea2441f700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea1c103de0 0x7fea1c075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.845+0000 7fea2441f700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fea0803e9a0 0x7fea08040eb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.845+0000 7fea2441f700 1 --2- 172.21.15.114:0/779867961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea1c075ff0 0x7fea1c076400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.845+0000 7fea2441f700 1 -- 172.21.15.114:0/779867961 >> 172.21.15.114:0/779867961 conn(0x7fea1c0ff690 msgr2=0x7fea1c109910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:44.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.845+0000 7fea2441f700 1 -- 172.21.15.114:0/779867961 shutdown_connections 2024-05-27T07:29:44.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:44.845+0000 7fea2441f700 1 -- 172.21.15.114:0/779867961 wait complete. 2024-05-27T07:29:45.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:45 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/779867961' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:45.716 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:45 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/779867961' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:45.719 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:45 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/779867961' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:45.866 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-27T07:29:45.866 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:29:45.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T07:29:45.895 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:29:45.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:29:45.960 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-27T07:29:45.960 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T07:29:45.989 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-27T07:29:45.989 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:29:46.055 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-27T07:29:46.055 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T07:29:46.086 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-27T07:29:46.086 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:29:46.154 INFO:tasks.cephadm:Adding mgr.x on smithi114 2024-05-27T07:29:46.155 INFO:tasks.cephadm:Adding mgr.y on smithi163 2024-05-27T07:29:46.155 INFO:tasks.cephadm:Adding mgr.z on smithi186 2024-05-27T07:29:46.155 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch apply mgr '3;smithi114=x;smithi163=y;smithi186=z' 2024-05-27T07:29:46.427 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:29:46.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:46 smithi186 ceph-mon[36647]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:46.716 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:46 smithi163 ceph-mon[36451]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:46.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:46 smithi114 ceph-mon[32038]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:47.678 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:47.678 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:47.678 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:47 smithi186 ceph-mon[36647]: Reconfiguring mon.b (monmap changed)... 2024-05-27T07:29:47.678 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:47.678 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:29:47.678 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:47.678 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:47 smithi186 ceph-mon[36647]: Reconfiguring daemon mon.b on smithi163 2024-05-27T07:29:47.716 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:47.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:47.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:47 smithi163 ceph-mon[36451]: Reconfiguring mon.b (monmap changed)... 2024-05-27T07:29:47.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:47.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:29:47.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:47.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:47 smithi163 ceph-mon[36451]: Reconfiguring daemon mon.b on smithi163 2024-05-27T07:29:47.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:47.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:47.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:47 smithi114 ceph-mon[32038]: Reconfiguring mon.b (monmap changed)... 2024-05-27T07:29:47.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:47.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:29:47.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:47.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:47 smithi114 ceph-mon[32038]: Reconfiguring daemon mon.b on smithi163 2024-05-27T07:29:47.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.874+0000 7fceedded700 1 -- 172.21.15.186:0/2255845324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcee8104840 msgr2=0x7fcee8106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:47.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.874+0000 7fceedded700 1 --2- 172.21.15.186:0/2255845324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcee8104840 0x7fcee8106c60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fced8009910 tx=0x7fced8009c20 comp rx=0 tx=0).stop 2024-05-27T07:29:47.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.874+0000 7fceedded700 1 -- 172.21.15.186:0/2255845324 shutdown_connections 2024-05-27T07:29:47.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.874+0000 7fceedded700 1 --2- 172.21.15.186:0/2255845324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcee8104840 0x7fcee8106c60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.874+0000 7fceedded700 1 -- 172.21.15.186:0/2255845324 >> 172.21.15.186:0/2255845324 conn(0x7fcee80fe240 msgr2=0x7fcee81006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:47.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.874+0000 7fceedded700 1 -- 172.21.15.186:0/2255845324 shutdown_connections 2024-05-27T07:29:47.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.874+0000 7fceedded700 1 -- 172.21.15.186:0/2255845324 wait complete. 2024-05-27T07:29:47.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.875+0000 7fceedded700 1 Processor -- start 2024-05-27T07:29:47.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.875+0000 7fceedded700 1 -- start start 2024-05-27T07:29:47.868 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.875+0000 7fceedded700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcee8104840 0x7fcee8199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.868 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.875+0000 7fceedded700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee8199ed0 0x7fcee819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.875+0000 7fceedded700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcee819f3a0 0x7fcee81a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.875+0000 7fceedded700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcee8109ca0 con 0x7fcee819f3a0 2024-05-27T07:29:47.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.875+0000 7fceedded700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcee8109b20 con 0x7fcee8104840 2024-05-27T07:29:47.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.875+0000 7fceedded700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcee8109e20 con 0x7fcee8199ed0 2024-05-27T07:29:47.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.875+0000 7fcee6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee8199ed0 0x7fcee819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.875+0000 7fcee6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee8199ed0 0x7fcee819ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49334/0 (socket says 172.21.15.186:49334) 2024-05-27T07:29:47.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.875+0000 7fcee6ffd700 1 -- 172.21.15.186:0/2807031501 learned_addr learned my addr 172.21.15.186:0/2807031501 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-27T07:29:47.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.876+0000 7fcee77fe700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcee8104840 0x7fcee8199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.876+0000 7fcee7fff700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcee819f3a0 0x7fcee81a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.876+0000 7fcee6ffd700 1 -- 172.21.15.186:0/2807031501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcee8104840 msgr2=0x7fcee8199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:47.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.876+0000 7fcee6ffd700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcee8104840 0x7fcee8199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.876+0000 7fcee6ffd700 1 -- 172.21.15.186:0/2807031501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcee819f3a0 msgr2=0x7fcee81a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:47.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.876+0000 7fcee6ffd700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcee819f3a0 0x7fcee81a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.876+0000 7fcee6ffd700 1 -- 172.21.15.186:0/2807031501 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee81a1d30 con 0x7fcee8199ed0 2024-05-27T07:29:47.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.876+0000 7fcee6ffd700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee8199ed0 0x7fcee819ee60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcedc00b780 tx=0x7fcedc00ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:47.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.876+0000 7fcee4ff9700 1 -- 172.21.15.186:0/2807031501 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc004230 con 0x7fcee8199ed0 2024-05-27T07:29:47.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.876+0000 7fcee4ff9700 1 -- 172.21.15.186:0/2807031501 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcedc0043d0 con 0x7fcee8199ed0 2024-05-27T07:29:47.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.876+0000 7fceedded700 1 -- 172.21.15.186:0/2807031501 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcee81a1fc0 con 0x7fcee8199ed0 2024-05-27T07:29:47.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.876+0000 7fcee4ff9700 1 -- 172.21.15.186:0/2807031501 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc011b20 con 0x7fcee8199ed0 2024-05-27T07:29:47.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.876+0000 7fceedded700 1 -- 172.21.15.186:0/2807031501 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcee81a2570 con 0x7fcee8199ed0 2024-05-27T07:29:47.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.877+0000 7fcee4ff9700 1 -- 172.21.15.186:0/2807031501 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7fcedc011cc0 con 0x7fcee8199ed0 2024-05-27T07:29:47.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.877+0000 7fceedded700 1 -- 172.21.15.186:0/2807031501 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcee8064e10 con 0x7fcee8199ed0 2024-05-27T07:29:47.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.877+0000 7fcee4ff9700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fced003e9a0 0x7fced0040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.877+0000 7fcee4ff9700 1 -- 172.21.15.186:0/2807031501 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fcedc0522e0 con 0x7fcee8199ed0 2024-05-27T07:29:47.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.878+0000 7fcee77fe700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fced003e9a0 0x7fced0040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.878+0000 7fcee77fe700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fced003e9a0 0x7fced0040eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fced800b5c0 tx=0x7fced801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:47.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:47.883+0000 7fcee4ff9700 1 -- 172.21.15.186:0/2807031501 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcedc020740 con 0x7fcee8199ed0 2024-05-27T07:29:48.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.103+0000 7fceedded700 1 -- 172.21.15.186:0/2807031501 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi114=x;smithi163=y;smithi186=z", "target": ["mon-mgr", ""]}) -- 0x7fcee8100010 con 0x7fced003e9a0 2024-05-27T07:29:48.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.109+0000 7fcee4ff9700 1 -- 172.21.15.186:0/2807031501 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fcee8100010 con 0x7fced003e9a0 2024-05-27T07:29:48.100 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mgr update... 2024-05-27T07:29:48.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.112+0000 7fceedded700 1 -- 172.21.15.186:0/2807031501 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fced003e9a0 msgr2=0x7fced0040eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:48.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.112+0000 7fceedded700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fced003e9a0 0x7fced0040eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fced800b5c0 tx=0x7fced801a040 comp rx=0 tx=0).stop 2024-05-27T07:29:48.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.112+0000 7fceedded700 1 -- 172.21.15.186:0/2807031501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee8199ed0 msgr2=0x7fcee819ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:48.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.112+0000 7fceedded700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee8199ed0 0x7fcee819ee60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcedc00b780 tx=0x7fcedc00ba90 comp rx=0 tx=0).stop 2024-05-27T07:29:48.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.112+0000 7fceedded700 1 -- 172.21.15.186:0/2807031501 shutdown_connections 2024-05-27T07:29:48.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.112+0000 7fceedded700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcee819f3a0 0x7fcee81a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:48.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.112+0000 7fceedded700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcee8104840 0x7fcee8199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:48.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.112+0000 7fceedded700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fced003e9a0 0x7fced0040eb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:48.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.112+0000 7fceedded700 1 --2- 172.21.15.186:0/2807031501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee8199ed0 0x7fcee819ee60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:48.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.112+0000 7fceedded700 1 -- 172.21.15.186:0/2807031501 >> 172.21.15.186:0/2807031501 conn(0x7fcee80fe240 msgr2=0x7fcee80ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:48.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.112+0000 7fceedded700 1 -- 172.21.15.186:0/2807031501 shutdown_connections 2024-05-27T07:29:48.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:29:48.112+0000 7fceedded700 1 -- 172.21.15.186:0/2807031501 wait complete. 2024-05-27T07:29:48.751 DEBUG:teuthology.orchestra.run.smithi163:mgr.y> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.y.service 2024-05-27T07:29:48.754 DEBUG:teuthology.orchestra.run.smithi186:mgr.z> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.z.service 2024-05-27T07:29:48.756 INFO:tasks.cephadm:Deploying OSDs... 2024-05-27T07:29:48.756 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:29:48.756 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T07:29:48.776 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T07:29:48.776 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-05-27T07:29:48.834 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T07:29:48.834 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:29:48.834 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-27T07:29:48.834 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:29:48.834 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:29:48.834 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-05-27 07:28:53.789701448 +0000 2024-05-27T07:29:48.834 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-05-27 07:23:16.845331493 +0000 2024-05-27T07:29:48.835 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-05-27 07:23:16.845331493 +0000 2024-05-27T07:29:48.835 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-05-27 07:23:16.845331493 +0000 2024-05-27T07:29:48.835 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T07:29:48.903 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-05-27T07:29:48.903 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-05-27T07:29:48.903 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000235061 s, 2.2 MB/s 2024-05-27T07:29:48.904 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T07:29:48.963 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-05-27T07:29:49.021 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T07:29:49.021 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:29:49.021 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-27T07:29:49.021 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:29:49.021 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:29:49.021 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-05-27 07:28:53.815700876 +0000 2024-05-27T07:29:49.021 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-05-27 07:23:17.219323281 +0000 2024-05-27T07:29:49.021 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-05-27 07:23:17.219323281 +0000 2024-05-27T07:29:49.021 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-05-27 07:23:17.219323281 +0000 2024-05-27T07:29:49.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T07:29:49.086 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-05-27T07:29:49.086 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-05-27T07:29:49.086 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000252191 s, 2.0 MB/s 2024-05-27T07:29:49.087 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T07:29:49.145 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-05-27T07:29:49.201 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T07:29:49.201 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:29:49.201 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-27T07:29:49.201 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:29:49.201 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:29:49.201 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-05-27 07:28:53.841700303 +0000 2024-05-27T07:29:49.201 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-05-27 07:23:17.565315684 +0000 2024-05-27T07:29:49.201 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-05-27 07:23:17.565315684 +0000 2024-05-27T07:29:49.202 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-05-27 07:23:17.565315684 +0000 2024-05-27T07:29:49.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T07:29:49.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:48 smithi163 ceph-mon[36451]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:49.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:48 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:49.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:48 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:49.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:48 smithi163 ceph-mon[36451]: Reconfiguring mon.c (monmap changed)... 2024-05-27T07:29:49.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:48 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:49.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:48 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:29:49.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:48 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:49.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:48 smithi163 ceph-mon[36451]: Reconfiguring daemon mon.c on smithi186 2024-05-27T07:29:49.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:48 smithi163 ceph-mon[36451]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi114=x;smithi163=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:29:49.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:48 smithi163 ceph-mon[36451]: Saving service mgr spec with placement smithi114=x;smithi163=y;smithi186=z;count:3 2024-05-27T07:29:49.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:48 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:49.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:48 smithi186 ceph-mon[36647]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:49.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:48 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:49.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:48 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:49.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:48 smithi186 ceph-mon[36647]: Reconfiguring mon.c (monmap changed)... 2024-05-27T07:29:49.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:48 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:49.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:48 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:29:49.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:48 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:49.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:48 smithi186 ceph-mon[36647]: Reconfiguring daemon mon.c on smithi186 2024-05-27T07:29:49.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:48 smithi186 ceph-mon[36647]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi114=x;smithi163=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:29:49.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:48 smithi186 ceph-mon[36647]: Saving service mgr spec with placement smithi114=x;smithi163=y;smithi186=z;count:3 2024-05-27T07:29:49.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:48 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:49.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:48 smithi114 ceph-mon[32038]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:49.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:48 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:49.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:48 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:49.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:48 smithi114 ceph-mon[32038]: Reconfiguring mon.c (monmap changed)... 2024-05-27T07:29:49.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:48 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:29:49.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:48 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:29:49.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:48 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:49.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:48 smithi114 ceph-mon[32038]: Reconfiguring daemon mon.c on smithi186 2024-05-27T07:29:49.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:48 smithi114 ceph-mon[32038]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi114=x;smithi163=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:29:49.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:48 smithi114 ceph-mon[32038]: Saving service mgr spec with placement smithi114=x;smithi163=y;smithi186=z;count:3 2024-05-27T07:29:49.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:48 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:49.265 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-05-27T07:29:49.266 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-05-27T07:29:49.266 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000217879 s, 2.3 MB/s 2024-05-27T07:29:49.267 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T07:29:49.323 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-05-27T07:29:49.379 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T07:29:49.380 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:29:49.380 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-27T07:29:49.380 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:29:49.380 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:29:49.380 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-05-27 07:28:53.867699731 +0000 2024-05-27T07:29:49.380 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-05-27 07:23:17.889308571 +0000 2024-05-27T07:29:49.380 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-05-27 07:23:17.889308571 +0000 2024-05-27T07:29:49.380 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-05-27 07:23:17.889308571 +0000 2024-05-27T07:29:49.380 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T07:29:49.443 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-05-27T07:29:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-05-27T07:29:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000194372 s, 2.6 MB/s 2024-05-27T07:29:49.445 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T07:29:49.502 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-27T07:29:49.503 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T07:29:49.521 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T07:29:49.522 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_1 2024-05-27T07:29:49.579 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T07:29:49.580 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:29:49.580 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-27T07:29:49.580 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:29:49.580 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:29:49.580 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-05-27 07:29:35.476361386 +0000 2024-05-27T07:29:49.580 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-05-27 07:23:58.343122802 +0000 2024-05-27T07:29:49.580 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-05-27 07:23:58.343122802 +0000 2024-05-27T07:29:49.580 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-05-27 07:23:58.343122802 +0000 2024-05-27T07:29:49.580 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T07:29:49.646 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-05-27T07:29:49.646 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-05-27T07:29:49.646 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000235456 s, 2.2 MB/s 2024-05-27T07:29:49.648 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T07:29:49.706 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_2 2024-05-27T07:29:49.763 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T07:29:49.763 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:29:49.763 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-05-27T07:29:49.763 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:29:49.763 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:29:49.764 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-05-27 07:29:35.476361386 +0000 2024-05-27T07:29:49.764 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-05-27 07:23:58.673115396 +0000 2024-05-27T07:29:49.764 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-05-27 07:23:58.673115396 +0000 2024-05-27T07:29:49.764 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-05-27 07:23:58.673115396 +0000 2024-05-27T07:29:49.764 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T07:29:49.827 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-05-27T07:29:49.827 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-05-27T07:29:49.827 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000306894 s, 1.7 MB/s 2024-05-27T07:29:49.828 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T07:29:49.886 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_3 2024-05-27T07:29:49.942 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T07:29:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:29:49.943 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-27T07:29:49.943 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:29:49.943 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:29:49.943 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-05-27 07:29:35.477361364 +0000 2024-05-27T07:29:49.943 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-05-27 07:23:59.004107967 +0000 2024-05-27T07:29:49.943 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-05-27 07:23:59.004107967 +0000 2024-05-27T07:29:49.943 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-05-27 07:23:59.004107967 +0000 2024-05-27T07:29:49.943 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T07:29:50.005 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-05-27T07:29:50.006 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-05-27T07:29:50.006 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000250213 s, 2.0 MB/s 2024-05-27T07:29:50.007 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T07:29:50.064 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_4 2024-05-27T07:29:50.119 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T07:29:50.119 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:29:50.120 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-05-27T07:29:50.120 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:29:50.120 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:29:50.120 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-05-27 07:29:35.477361364 +0000 2024-05-27T07:29:50.120 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-05-27 07:23:59.332100605 +0000 2024-05-27T07:29:50.120 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-05-27 07:23:59.332100605 +0000 2024-05-27T07:29:50.120 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-05-27 07:23:59.332100605 +0000 2024-05-27T07:29:50.120 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T07:29:50.182 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-05-27T07:29:50.182 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-05-27T07:29:50.182 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000219528 s, 2.3 MB/s 2024-05-27T07:29:50.183 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T07:29:50.239 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-27T07:29:50.239 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T07:29:50.257 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T07:29:50.258 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-05-27T07:29:50.315 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T07:29:50.315 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:29:50.315 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-27T07:29:50.315 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:29:50.315 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:29:50.315 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-27 07:29:26.113189882 +0000 2024-05-27T07:29:50.315 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-27 07:23:33.285012796 +0000 2024-05-27T07:29:50.315 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-27 07:23:33.285012796 +0000 2024-05-27T07:29:50.315 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-27 07:23:33.285012796 +0000 2024-05-27T07:29:50.316 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T07:29:50.379 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-27T07:29:50.380 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-27T07:29:50.380 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000252965 s, 2.0 MB/s 2024-05-27T07:29:50.381 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T07:29:50.436 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-05-27T07:29:50.491 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T07:29:50.491 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:29:50.491 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-05-27T07:29:50.492 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:29:50.492 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:29:50.492 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-27 07:29:26.113189882 +0000 2024-05-27T07:29:50.492 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-27 07:23:33.616006689 +0000 2024-05-27T07:29:50.492 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-27 07:23:33.616006689 +0000 2024-05-27T07:29:50.492 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-27 07:23:33.616006689 +0000 2024-05-27T07:29:50.492 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T07:29:50.554 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-27T07:29:50.554 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-27T07:29:50.554 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000222934 s, 2.3 MB/s 2024-05-27T07:29:50.555 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T07:29:50.611 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-05-27T07:29:50.666 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T07:29:50.666 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:29:50.666 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-05-27T07:29:50.666 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:29:50.666 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:29:50.666 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-27 07:29:26.114189863 +0000 2024-05-27T07:29:50.666 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-27 07:23:33.955000433 +0000 2024-05-27T07:29:50.666 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-27 07:23:33.955000433 +0000 2024-05-27T07:29:50.666 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-27 07:23:33.955000433 +0000 2024-05-27T07:29:50.667 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T07:29:50.729 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-27T07:29:50.729 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-27T07:29:50.730 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000203234 s, 2.5 MB/s 2024-05-27T07:29:50.731 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T07:29:50.786 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-05-27T07:29:50.841 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T07:29:50.841 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:29:50.842 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-05-27T07:29:50.842 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:29:50.842 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:29:50.842 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-27 07:29:26.114189863 +0000 2024-05-27T07:29:50.842 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-27 07:23:34.282994381 +0000 2024-05-27T07:29:50.842 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-27 07:23:34.282994381 +0000 2024-05-27T07:29:50.842 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-27 07:23:34.282994381 +0000 2024-05-27T07:29:50.842 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T07:29:50.903 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-27T07:29:50.903 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-27T07:29:50.903 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000171397 s, 3.0 MB/s 2024-05-27T07:29:50.904 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T07:29:50.915 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:50 smithi186 ceph-mon[36647]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:50.915 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:50.915 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:50.915 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:50.915 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:50.915 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:50.915 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:50.915 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:29:50.915 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:29:50.915 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:29:50.915 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:50.929 INFO:tasks.cephadm:Deploying osd.0 on smithi114 with /dev/vg_nvme/lv_4... 2024-05-27T07:29:50.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:50 smithi114 ceph-mon[32038]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:29:50.954 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:29:50.954 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:50.966 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:50 smithi163 ceph-mon[36451]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:50.966 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:50.966 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:50.966 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:50.966 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:50.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:50.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:50.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:29:50.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:29:50.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:29:50.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:51.162 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:51.916 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:51 smithi186 ceph-mon[36647]: Deploying daemon mgr.z on smithi186 2024-05-27T07:29:51.966 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:51 smithi163 ceph-mon[36451]: Deploying daemon mgr.z on smithi186 2024-05-27T07:29:52.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:51 smithi114 ceph-mon[32038]: Deploying daemon mgr.z on smithi186 2024-05-27T07:29:52.219 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:52 smithi186 systemd[1]: Starting Ceph mgr.z for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:29:52.617 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:52 smithi186 podman[38481]: 2024-05-27 07:29:52.258808343 +0000 UTC m=+0.017290536 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:52.617 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:52 smithi186 podman[38481]: 2024-05-27 07:29:52.357264749 +0000 UTC m=+0.115746934 container create 32c8c30f71f14326c96fddcba2848a053d17e18843c4bf1c11d9868d958bc019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:52.617 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:52 smithi186 podman[38481]: 2024-05-27 07:29:52.543251129 +0000 UTC m=+0.301733325 container init 32c8c30f71f14326c96fddcba2848a053d17e18843c4bf1c11d9868d958bc019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-27T07:29:52.617 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:52 smithi186 podman[38481]: 2024-05-27 07:29:52.545767127 +0000 UTC m=+0.304249322 container start 32c8c30f71f14326c96fddcba2848a053d17e18843c4bf1c11d9868d958bc019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-27T07:29:52.617 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:52 smithi186 ceph-mgr[38500]: -- 172.21.15.186:0/514471665 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560ad44725a0 con 0x560ad91fc000 2024-05-27T07:29:52.618 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:52 smithi186 bash[38481]: 32c8c30f71f14326c96fddcba2848a053d17e18843c4bf1c11d9868d958bc019 2024-05-27T07:29:52.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:52 smithi114 ceph-mon[32038]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:52.885 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:52 smithi186 ceph-mon[36647]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:52.886 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:52 smithi186 systemd[1]: Started Ceph mgr.z for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:29:52.886 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:52.711+0000 7f19ff1dd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:29:52.966 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:52 smithi163 ceph-mon[36451]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:53.002 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T07:29:53.002 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T07:29:53.002 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-05-27T07:29:53.002 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-05-27T07:29:53.002 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011246 s, 932 MB/s 2024-05-27T07:29:53.003 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-05-27T07:29:53.219 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:52.894+0000 7f19ff1dd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:29:53.219 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:52.996+0000 7f19ff1dd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:29:53.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_4 2024-05-27T07:29:53.687 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:53 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:53.301+0000 7f19ff1dd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:29:53.687 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:53 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:53.386+0000 7f19ff1dd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:29:53.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:53 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:53.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:53 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:53.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:53 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:53.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:53 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:29:53.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:53 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:29:53.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:53 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:29:53.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:53 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:53.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:53 smithi114 ceph-mon[32038]: Deploying daemon mgr.y on smithi163 2024-05-27T07:29:53.823 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:53.966 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:53 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:53.966 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:53 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:53.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:53 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:53.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:53 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:29:53.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:53 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:29:53.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:53 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:29:53.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:53 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:53.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:53 smithi163 ceph-mon[36451]: Deploying daemon mgr.y on smithi163 2024-05-27T07:29:53.969 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:53 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:53.969 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:53 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:53.969 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:53 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:53.969 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:53 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:29:53.969 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:53 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:29:53.969 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:53 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:29:53.969 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:53 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:53.969 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:53 smithi186 ceph-mon[36647]: Deploying daemon mgr.y on smithi163 2024-05-27T07:29:54.852 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:54 smithi163 ceph-mon[36451]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:54.919 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:54 smithi186 ceph-mon[36647]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:54.919 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:54 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:54.574+0000 7f19ff1dd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:29:54.919 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:54 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:54.749+0000 7f19ff1dd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:29:54.920 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:54 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:54.928+0000 7f19ff1dd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:29:54.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:54 smithi114 ceph-mon[32038]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:55.107 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:55 smithi163 ceph-mgr[37822]: -- 172.21.15.163:0/3965657587 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cb846325a0 con 0x55cb893bc000 2024-05-27T07:29:55.391 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:55 smithi163 bash[37803]: a3f682a56a5e28985ed87b9e022ceb5c44bcb26c5cf7bd834cb4b5293457893b 2024-05-27T07:29:55.391 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:55 smithi163 systemd[1]: Started Ceph mgr.y for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:29:55.392 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:55 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:55.219+0000 7f0cce1b0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:29:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.520+0000 7f936a3e4700 1 -- 172.21.15.114:0/154550688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93641029d0 msgr2=0x7f9364102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.520+0000 7f936a3e4700 1 --2- 172.21.15.114:0/154550688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93641029d0 0x7f9364102de0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9354009990 tx=0x7f9354009ca0 comp rx=0 tx=0).stop 2024-05-27T07:29:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.520+0000 7f936a3e4700 1 -- 172.21.15.114:0/154550688 shutdown_connections 2024-05-27T07:29:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.520+0000 7f936a3e4700 1 --2- 172.21.15.114:0/154550688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93641044a0 0x7f936410ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.520+0000 7f936a3e4700 1 --2- 172.21.15.114:0/154550688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93641029d0 0x7f9364102de0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.520+0000 7f936a3e4700 1 --2- 172.21.15.114:0/154550688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9364103b00 0x7f9364103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.520+0000 7f936a3e4700 1 -- 172.21.15.114:0/154550688 >> 172.21.15.114:0/154550688 conn(0x7f93640fe010 msgr2=0x7f93641004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.520+0000 7f936a3e4700 1 -- 172.21.15.114:0/154550688 shutdown_connections 2024-05-27T07:29:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.520+0000 7f936a3e4700 1 -- 172.21.15.114:0/154550688 wait complete. 2024-05-27T07:29:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.521+0000 7f936a3e4700 1 Processor -- start 2024-05-27T07:29:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.521+0000 7f936a3e4700 1 -- start start 2024-05-27T07:29:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.521+0000 7f936a3e4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93641029d0 0x7f936410c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.521+0000 7f936a3e4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9364103b00 0x7f936419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.521+0000 7f936a3e4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93641044a0 0x7f93641a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.521+0000 7f936a3e4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f936410ec90 con 0x7f93641044a0 2024-05-27T07:29:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.521+0000 7f936a3e4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f936410eb10 con 0x7f9364103b00 2024-05-27T07:29:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.521+0000 7f936a3e4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f936410ee10 con 0x7f93641029d0 2024-05-27T07:29:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.521+0000 7f9368981700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93641044a0 0x7f93641a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:55.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.521+0000 7f9368981700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93641044a0 0x7f93641a1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58046/0 (socket says 172.21.15.114:58046) 2024-05-27T07:29:55.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.521+0000 7f9368981700 1 -- 172.21.15.114:0/1339337119 learned_addr learned my addr 172.21.15.114:0/1339337119 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:29:55.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.521+0000 7f935bfff700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9364103b00 0x7f936419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:55.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.521+0000 7f9363fff700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93641029d0 0x7f936410c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:55.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.522+0000 7f9368981700 1 -- 172.21.15.114:0/1339337119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93641029d0 msgr2=0x7f936410c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:55.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.522+0000 7f9368981700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93641029d0 0x7f936410c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:55.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.522+0000 7f9368981700 1 -- 172.21.15.114:0/1339337119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9364103b00 msgr2=0x7f936419f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:55.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.522+0000 7f9368981700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9364103b00 0x7f936419f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:55.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.522+0000 7f9368981700 1 -- 172.21.15.114:0/1339337119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93641a2460 con 0x7f93641044a0 2024-05-27T07:29:55.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.522+0000 7f9368981700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93641044a0 0x7f93641a1c30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f935c00ef30 tx=0x7f935c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.522+0000 7f9361ffb700 1 -- 172.21.15.114:0/1339337119 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c0092e0 con 0x7f93641044a0 2024-05-27T07:29:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.522+0000 7f936a3e4700 1 -- 172.21.15.114:0/1339337119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93641a26f0 con 0x7f93641044a0 2024-05-27T07:29:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.522+0000 7f9361ffb700 1 -- 172.21.15.114:0/1339337119 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f935c004bb0 con 0x7f93641044a0 2024-05-27T07:29:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.522+0000 7f9361ffb700 1 -- 172.21.15.114:0/1339337119 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c004020 con 0x7f93641044a0 2024-05-27T07:29:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.522+0000 7f936a3e4700 1 -- 172.21.15.114:0/1339337119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93641a2c30 con 0x7f93641044a0 2024-05-27T07:29:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.523+0000 7f936a3e4700 1 -- 172.21.15.114:0/1339337119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93640512b0 con 0x7f93641044a0 2024-05-27T07:29:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.523+0000 7f9361ffb700 1 -- 172.21.15.114:0/1339337119 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f935c01b030 con 0x7f93641044a0 2024-05-27T07:29:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.523+0000 7f9361ffb700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f935003e750 0x7f9350040c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.523+0000 7f9361ffb700 1 -- 172.21.15.114:0/1339337119 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f935c052350 con 0x7f93641044a0 2024-05-27T07:29:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.523+0000 7f9363fff700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f935003e750 0x7f9350040c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.524+0000 7f9363fff700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f935003e750 0x7f9350040c60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f935400b6c0 tx=0x7f935400b720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:55.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.529+0000 7f9361ffb700 1 -- 172.21.15.114:0/1339337119 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f935c022020 con 0x7f93641044a0 2024-05-27T07:29:55.540 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:55 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:55.237+0000 7f19ff1dd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:29:55.540 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:55 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:55.333+0000 7f19ff1dd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:29:55.540 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:55 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:55.549+0000 7f19ff1dd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:29:55.717 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:55 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:55.413+0000 7f0cce1b0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:29:55.717 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:55 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:55.527+0000 7f0cce1b0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:29:55.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:55.743+0000 7f936a3e4700 1 -- 172.21.15.114:0/1339337119 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9364063bf0 con 0x7f935003e750 2024-05-27T07:29:55.932 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:55 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:55.754+0000 7f19ff1dd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:29:56.201 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:55 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:55.879+0000 7f0cce1b0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:29:56.201 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:55 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:55.978+0000 7f0cce1b0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:29:56.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:56.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:56.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:56.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:56.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:56 smithi114 ceph-mon[32038]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:56.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:56.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:56 smithi114 ceph-mon[32038]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:29:56.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:29:56.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:29:56.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:56.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:56 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:56.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:56 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:56.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:56 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:56.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:56 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:56.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:56 smithi163 ceph-mon[36451]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:56.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:56 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:56.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:56 smithi163 ceph-mon[36451]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:29:56.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:56 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:29:56.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:56 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:29:56.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:56 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:56.469 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:56 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:56.399+0000 7f19ff1dd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:29:56.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:56 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:56.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:56 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:56.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:56 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:56.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:56 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:56.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:56 smithi186 ceph-mon[36647]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:56.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:56 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:56.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:56 smithi186 ceph-mon[36647]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:29:56.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:56 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:29:56.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:56 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:29:56.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:56 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:56.969 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:56 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:56.627+0000 7f19ff1dd200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:29:57.469 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:57.057+0000 7f19ff1dd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:29:57.469 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:57.154+0000 7f19ff1dd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:29:57.558 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:57 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:57.226+0000 7f0cce1b0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:29:57.558 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:57 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:57.395+0000 7f0cce1b0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:29:57.558 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:57 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:57.566+0000 7f0cce1b0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:29:57.847 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:57.847 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:57.847 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:57.847 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:57.847 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:57.847 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:57.847 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:57.847 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:57 smithi163 ceph-mon[36451]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T07:29:57.847 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:29:57.848 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:29:57.848 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:57.848 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:57 smithi163 ceph-mon[36451]: Reconfiguring daemon mgr.x on smithi114 2024-05-27T07:29:57.871 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:57.525+0000 7f19ff1dd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:29:57.872 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:57.872 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:57.872 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:57.872 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:57.872 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:57.872 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:57.872 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:57.872 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:57 smithi186 ceph-mon[36647]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T07:29:57.872 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:29:57.872 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:29:57.872 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:57.873 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:57 smithi186 ceph-mon[36647]: Reconfiguring daemon mgr.x on smithi114 2024-05-27T07:29:58.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:58.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:58.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:58.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:58.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:58.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:58.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:58.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:57 smithi114 ceph-mon[32038]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T07:29:58.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:29:58.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:29:58.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:58.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:57 smithi114 ceph-mon[32038]: Reconfiguring daemon mgr.x on smithi114 2024-05-27T07:29:58.146 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:57 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:57.856+0000 7f0cce1b0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:29:58.146 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:57 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:57.948+0000 7f0cce1b0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:29:58.146 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:58 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:58.154+0000 7f0cce1b0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:29:58.205 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:57.880+0000 7f19ff1dd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:29:58.205 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:57.973+0000 7f19ff1dd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:29:58.466 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:58 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:58.347+0000 7f0cce1b0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:29:58.469 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:58 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:58.214+0000 7f19ff1dd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:29:58.469 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:58 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:58.347+0000 7f19ff1dd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:29:58.469 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:58 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:58.443+0000 7f19ff1dd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:29:59.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:58 smithi114 ceph-mon[32038]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:59.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:58 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:59.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:58 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3624541632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24622a6e-31e4-46ce-89d3-163303255c27"}]: dispatch 2024-05-27T07:29:59.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:58 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3624541632' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24622a6e-31e4-46ce-89d3-163303255c27"}]': finished 2024-05-27T07:29:59.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:58 smithi114 ceph-mon[32038]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T07:29:59.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:58 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:29:59.168 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:58 smithi163 ceph-mon[36451]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:59.168 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:58 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:59.168 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:58 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3624541632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24622a6e-31e4-46ce-89d3-163303255c27"}]: dispatch 2024-05-27T07:29:59.168 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:58 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3624541632' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24622a6e-31e4-46ce-89d3-163303255c27"}]': finished 2024-05-27T07:29:59.168 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:58 smithi163 ceph-mon[36451]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T07:29:59.168 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:58 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:29:59.168 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:58 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:58.956+0000 7f0cce1b0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:29:59.169 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:59.176+0000 7f0cce1b0200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:29:59.219 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:58 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:58.769+0000 7f19ff1dd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:29:59.219 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:58 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:58.860+0000 7f19ff1dd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:29:59.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:58 smithi186 ceph-mon[36647]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:29:59.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:58 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:29:59.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:58 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3624541632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24622a6e-31e4-46ce-89d3-163303255c27"}]: dispatch 2024-05-27T07:29:59.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:58 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3624541632' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24622a6e-31e4-46ce-89d3-163303255c27"}]': finished 2024-05-27T07:29:59.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:58 smithi186 ceph-mon[36647]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T07:29:59.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:58 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:59 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:59.455+0000 7f19ff1dd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:29:59 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:29:59.671+0000 7f19ff1dd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:29:59.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:29:59.747+0000 7f9361ffb700 1 -- 172.21.15.114:0/1339337119 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 14) ==== 99938+0+0 (secure 0 0 0) 0x7f935c012050 con 0x7f93641044a0 2024-05-27T07:29:59.966 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:59 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/836435627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:29:59.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:59 smithi163 ceph-mon[36451]: Standby manager daemon z started 2024-05-27T07:29:59.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:59 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.186:0/3466179683' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:29:59.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:59 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.186:0/3466179683' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:29:59.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:59 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.186:0/3466179683' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:29:59.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:29:59 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.186:0/3466179683' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:29:59.967 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:59.603+0000 7f0cce1b0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:29:59.967 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:29:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:29:59.699+0000 7f0cce1b0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:30:00.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:59 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/836435627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:30:00.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:59 smithi114 ceph-mon[32038]: Standby manager daemon z started 2024-05-27T07:30:00.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:59 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.186:0/3466179683' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:30:00.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:59 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.186:0/3466179683' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:30:00.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:59 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.186:0/3466179683' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:30:00.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:29:59 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.186:0/3466179683' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:30:00.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:59 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/836435627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:30:00.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:59 smithi186 ceph-mon[36647]: Standby manager daemon z started 2024-05-27T07:30:00.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:59 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.186:0/3466179683' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:30:00.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:59 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.186:0/3466179683' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:30:00.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:59 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.186:0/3466179683' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:30:00.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:29:59 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.186:0/3466179683' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:30:00.417 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:30:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:30:00.072+0000 7f0cce1b0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:30:00.417 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:30:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:30:00.425+0000 7f0cce1b0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:30:00.716 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:30:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:30:00.518+0000 7f0cce1b0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:30:00.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:00 smithi163 ceph-mon[36451]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:00.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:00 smithi163 ceph-mon[36451]: mgrmap e14: x(active, since 76s), standbys: z 2024-05-27T07:30:00.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:00 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:30:00.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:00 smithi163 ceph-mon[36451]: overall HEALTH_OK 2024-05-27T07:30:00.988 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:30:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:30:00.765+0000 7f0cce1b0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:30:00.988 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:30:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:30:00.899+0000 7f0cce1b0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:30:00.988 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:30:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:30:00.996+0000 7f0cce1b0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:30:01.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:00 smithi114 ceph-mon[32038]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:01.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:00 smithi114 ceph-mon[32038]: mgrmap e14: x(active, since 76s), standbys: z 2024-05-27T07:30:01.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:00 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:30:01.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:00 smithi114 ceph-mon[32038]: overall HEALTH_OK 2024-05-27T07:30:01.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:00 smithi186 ceph-mon[36647]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:01.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:00 smithi186 ceph-mon[36647]: mgrmap e14: x(active, since 76s), standbys: z 2024-05-27T07:30:01.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:00 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:30:01.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:00 smithi186 ceph-mon[36647]: overall HEALTH_OK 2024-05-27T07:30:01.716 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:30:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:30:01.323+0000 7f0cce1b0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:30:01.717 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:30:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:30:01.417+0000 7f0cce1b0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:30:02.466 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:30:02 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:30:02.020+0000 7f0cce1b0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:30:02.467 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:30:02 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:30:02.235+0000 7f0cce1b0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:30:02.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:02.756+0000 7f9361ffb700 1 -- 172.21.15.114:0/1339337119 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mgrmap(e 15) ==== 149597+0+0 (secure 0 0 0) 0x7f935c052550 con 0x7f93641044a0 2024-05-27T07:30:03.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:02 smithi114 ceph-mon[32038]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:03.023 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:02 smithi114 ceph-mon[32038]: Standby manager daemon y started 2024-05-27T07:30:03.023 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:02 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.163:0/347545548' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:30:03.023 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:02 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.163:0/347545548' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:30:03.023 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:02 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.163:0/347545548' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:30:03.023 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:02 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.163:0/347545548' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:30:03.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:02 smithi163 ceph-mon[36451]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:03.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:02 smithi163 ceph-mon[36451]: Standby manager daemon y started 2024-05-27T07:30:03.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:02 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.163:0/347545548' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:30:03.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:02 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.163:0/347545548' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:30:03.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:02 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.163:0/347545548' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:30:03.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:02 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.163:0/347545548' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:30:03.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:02 smithi186 ceph-mon[36647]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:03.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:02 smithi186 ceph-mon[36647]: Standby manager daemon y started 2024-05-27T07:30:03.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:02 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.163:0/347545548' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:30:03.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:02 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.163:0/347545548' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:30:03.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:02 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.163:0/347545548' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:30:03.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:02 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.163:0/347545548' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:30:03.874 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:03 smithi114 ceph-mon[32038]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-05-27T07:30:03.874 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:03 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:30:04.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:03 smithi163 ceph-mon[36451]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-05-27T07:30:04.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:03 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:30:04.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:03 smithi186 ceph-mon[36647]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-05-27T07:30:04.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:03 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:30:05.149 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:04 smithi114 ceph-mon[32038]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:05.149 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:05.149 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:05.149 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:05.150 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:05.150 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:05.150 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:05.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:04 smithi163 ceph-mon[36451]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:05.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:05.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:05.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:05.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:05.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:05.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:05.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:04 smithi186 ceph-mon[36647]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:05.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:05.221 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:05.221 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:05.221 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:05.221 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:05.221 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:07.204 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:06 smithi114 ceph-mon[32038]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:07.204 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:06 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T07:30:07.204 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:06 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:07.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:06 smithi163 ceph-mon[36451]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:07.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:06 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T07:30:07.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:06 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:07.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:06 smithi186 ceph-mon[36647]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:07.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:06 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T07:30:07.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:06 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:08.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:07 smithi114 ceph-mon[32038]: Deploying daemon osd.0 on smithi114 2024-05-27T07:30:08.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:07 smithi163 ceph-mon[36451]: Deploying daemon osd.0 on smithi114 2024-05-27T07:30:08.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:07 smithi186 ceph-mon[36647]: Deploying daemon osd.0 on smithi114 2024-05-27T07:30:09.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:08 smithi163 ceph-mon[36451]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:09.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:08 smithi186 ceph-mon[36647]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:09.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:08 smithi114 ceph-mon[32038]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:11.141 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:10 smithi114 ceph-mon[32038]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:11.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:10 smithi163 ceph-mon[36451]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:11.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:10 smithi186 ceph-mon[36647]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:13.297 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:13 smithi114 ceph-mon[32038]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:13.297 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:13 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:13.297 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:13 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:13.297 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:13 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:13.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:13 smithi163 ceph-mon[36451]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:13.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:13 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:13.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:13 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:13.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:13 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:13.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:13 smithi186 ceph-mon[36647]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:13.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:13 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:13.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:13 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:13.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:13 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:15.297 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:15 smithi114 ceph-mon[32038]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:15.297 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:15 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:15.297 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:15 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:15.297 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:15 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:15.297 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:15 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:15.297 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:15 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:15.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:15 smithi163 ceph-mon[36451]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:15.485 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:15 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:15.485 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:15 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:15.485 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:15 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:15.485 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:15 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:15.485 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:15 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:15.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:15 smithi186 ceph-mon[36647]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:15.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:15 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:15.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:15 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:15.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:15 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:15.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:15 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:15.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:15 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:15.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:15.712+0000 7f9361ffb700 1 -- 172.21.15.114:0/1339337119 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9364063bf0 con 0x7f935003e750 2024-05-27T07:30:15.704 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 0 on host 'smithi114' 2024-05-27T07:30:15.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:15.715+0000 7f936a3e4700 1 -- 172.21.15.114:0/1339337119 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f935003e750 msgr2=0x7f9350040c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:15.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:15.715+0000 7f936a3e4700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f935003e750 0x7f9350040c60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f935400b6c0 tx=0x7f935400b720 comp rx=0 tx=0).stop 2024-05-27T07:30:15.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:15.715+0000 7f936a3e4700 1 -- 172.21.15.114:0/1339337119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93641044a0 msgr2=0x7f93641a1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:15.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:15.715+0000 7f936a3e4700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93641044a0 0x7f93641a1c30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f935c00ef30 tx=0x7f935c00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:30:15.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:15.715+0000 7f936a3e4700 1 -- 172.21.15.114:0/1339337119 shutdown_connections 2024-05-27T07:30:15.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:15.715+0000 7f936a3e4700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93641044a0 0x7f93641a1c30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:15.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:15.715+0000 7f936a3e4700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f935003e750 0x7f9350040c60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:15.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:15.715+0000 7f936a3e4700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93641029d0 0x7f936410c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:15.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:15.715+0000 7f936a3e4700 1 --2- 172.21.15.114:0/1339337119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9364103b00 0x7f936419f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:15.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:15.715+0000 7f936a3e4700 1 -- 172.21.15.114:0/1339337119 >> 172.21.15.114:0/1339337119 conn(0x7f93640fe010 msgr2=0x7f9364106d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:15.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:15.715+0000 7f936a3e4700 1 -- 172.21.15.114:0/1339337119 shutdown_connections 2024-05-27T07:30:15.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:30:15.715+0000 7f936a3e4700 1 -- 172.21.15.114:0/1339337119 wait complete. 2024-05-27T07:30:16.406 DEBUG:teuthology.orchestra.run.smithi114:osd.0> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.0.service 2024-05-27T07:30:16.408 INFO:tasks.cephadm:Deploying osd.1 on smithi163 with /dev/vg_nvme/lv_4... 2024-05-27T07:30:16.409 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T07:30:16.653 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:30:16.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:16 smithi163 ceph-mon[36451]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:16.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:16 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:16.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:16 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:16.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:16 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:16.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:16 smithi163 ceph-mon[36451]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T07:30:16.908 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:16 smithi114 ceph-mon[32038]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:16.908 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:16 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:16.908 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:16 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:16.908 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:16 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:16.908 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:16 smithi114 ceph-mon[32038]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T07:30:16.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:16 smithi186 ceph-mon[36647]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:16.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:16 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:16.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:16 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:16.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:16 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:16.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:16 smithi186 ceph-mon[36647]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T07:30:17.893 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:17 smithi114 ceph-mon[32038]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T07:30:17.893 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:17 smithi114 ceph-mon[32038]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T07:30:17.893 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:17 smithi114 ceph-mon[32038]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-05-27T07:30:17.893 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:17 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:17.944 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:17 smithi163 ceph-mon[36451]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T07:30:17.945 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:17 smithi163 ceph-mon[36451]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T07:30:17.945 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:17 smithi163 ceph-mon[36451]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-05-27T07:30:17.945 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:17 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:17.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:17 smithi186 ceph-mon[36647]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T07:30:17.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:17 smithi186 ceph-mon[36647]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T07:30:17.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:17 smithi186 ceph-mon[36647]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-05-27T07:30:17.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:17 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:18.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 07:30:18 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0[44060]: 2024-05-27T07:30:18.132+0000 7ff89b7ca700 -1 osd.0 0 waiting for initial osdmap 2024-05-27T07:30:18.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 07:30:18 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0[44060]: 2024-05-27T07:30:18.137+0000 7ff893577700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:30:18.498 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T07:30:18.499 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T07:30:18.499 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-05-27T07:30:18.499 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-05-27T07:30:18.499 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114032 s, 920 MB/s 2024-05-27T07:30:18.499 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-05-27T07:30:18.849 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:18 smithi163 ceph-mon[36451]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:18.849 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:18 smithi163 ceph-mon[36451]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-05-27T07:30:18.849 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:18 smithi163 ceph-mon[36451]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T07:30:18.849 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:18.849 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:18.849 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:18 smithi163 ceph-mon[36451]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' 2024-05-27T07:30:18.969 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:18 smithi186 ceph-mon[36647]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:18.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:18 smithi186 ceph-mon[36647]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-05-27T07:30:18.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:18 smithi186 ceph-mon[36647]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T07:30:18.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:18.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:18.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:18 smithi186 ceph-mon[36647]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' 2024-05-27T07:30:19.019 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:18 smithi114 ceph-mon[32038]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:19.019 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:18 smithi114 ceph-mon[32038]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-05-27T07:30:19.019 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:18 smithi114 ceph-mon[32038]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T07:30:19.019 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:19.019 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:19.019 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:18 smithi114 ceph-mon[32038]: from='osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464]' entity='osd.0' 2024-05-27T07:30:19.163 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_4 2024-05-27T07:30:19.406 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:30:19.699 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:19 smithi163 ceph-mon[36451]: purged_snaps scrub starts 2024-05-27T07:30:19.700 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:19 smithi163 ceph-mon[36451]: purged_snaps scrub ok 2024-05-27T07:30:19.700 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:19 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:19.700 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:19 smithi163 ceph-mon[36451]: osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464] boot 2024-05-27T07:30:19.700 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:19 smithi163 ceph-mon[36451]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T07:30:19.700 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:19 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:19.700 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:19 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:19.700 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:19 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:19.700 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:19 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:30:19.700 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:19 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:19.700 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:19 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:19.700 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:19 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:19.700 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:19 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:19.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:19 smithi186 ceph-mon[36647]: purged_snaps scrub starts 2024-05-27T07:30:19.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:19 smithi186 ceph-mon[36647]: purged_snaps scrub ok 2024-05-27T07:30:19.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:19 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:19.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:19 smithi186 ceph-mon[36647]: osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464] boot 2024-05-27T07:30:19.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:19 smithi186 ceph-mon[36647]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T07:30:19.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:19 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:19.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:19 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:19.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:19 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:19.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:19 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:30:19.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:19 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:19.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:19 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:19.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:19 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:19.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:19 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:20.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:19 smithi114 ceph-mon[32038]: purged_snaps scrub starts 2024-05-27T07:30:20.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:19 smithi114 ceph-mon[32038]: purged_snaps scrub ok 2024-05-27T07:30:20.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:19 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:20.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:19 smithi114 ceph-mon[32038]: osd.0 [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464] boot 2024-05-27T07:30:20.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:19 smithi114 ceph-mon[32038]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T07:30:20.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:19 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:30:20.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:19 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:20.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:19 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:20.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:19 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:30:20.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:19 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:20.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:19 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:20.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:19 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:20.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:19 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:20.800 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:20 smithi163 ceph-mon[36451]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:20.800 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:20 smithi163 ceph-mon[36451]: Detected new or changed devices on smithi114 2024-05-27T07:30:20.800 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:20 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi114 to 17088M 2024-05-27T07:30:20.969 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:20 smithi186 ceph-mon[36647]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:20.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:20 smithi186 ceph-mon[36647]: Detected new or changed devices on smithi114 2024-05-27T07:30:20.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:20 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi114 to 17088M 2024-05-27T07:30:20.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.005+0000 7f0460eb4700 1 -- 172.21.15.163:0/2397585052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f045c1029d0 msgr2=0x7f045c102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:20.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.005+0000 7f0460eb4700 1 --2- 172.21.15.163:0/2397585052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f045c1029d0 0x7f045c102de0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f044c00aaa0 tx=0x7f044c00adb0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.005+0000 7f0460eb4700 1 -- 172.21.15.163:0/2397585052 shutdown_connections 2024-05-27T07:30:20.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.005+0000 7f0460eb4700 1 --2- 172.21.15.163:0/2397585052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c1044a0 0x7f045c10ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.005+0000 7f0460eb4700 1 --2- 172.21.15.163:0/2397585052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f045c1029d0 0x7f045c102de0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.005+0000 7f0460eb4700 1 --2- 172.21.15.163:0/2397585052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f045c103b00 0x7f045c103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.005+0000 7f0460eb4700 1 -- 172.21.15.163:0/2397585052 >> 172.21.15.163:0/2397585052 conn(0x7f045c0fe010 msgr2=0x7f045c1004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:20.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.005+0000 7f0460eb4700 1 -- 172.21.15.163:0/2397585052 shutdown_connections 2024-05-27T07:30:20.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.005+0000 7f0460eb4700 1 -- 172.21.15.163:0/2397585052 wait complete. 2024-05-27T07:30:21.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0460eb4700 1 Processor -- start 2024-05-27T07:30:21.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0460eb4700 1 -- start start 2024-05-27T07:30:21.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0460eb4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c1029d0 0x7f045c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:21.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0460eb4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f045c103b00 0x7f045c198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:21.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0460eb4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f045c1044a0 0x7f045c19fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:21.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0460eb4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f045c10dba0 con 0x7f045c1029d0 2024-05-27T07:30:21.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0460eb4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f045c10da20 con 0x7f045c103b00 2024-05-27T07:30:21.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0460eb4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f045c10dd20 con 0x7f045c1044a0 2024-05-27T07:30:21.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0459d9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f045c103b00 0x7f045c198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:21.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0459d9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f045c103b00 0x7f045c198cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55460/0 (socket says 172.21.15.163:55460) 2024-05-27T07:30:21.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0459d9b700 1 -- 172.21.15.163:0/3485888941 learned_addr learned my addr 172.21.15.163:0/3485888941 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-27T07:30:21.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f045ad9d700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f045c1044a0 0x7f045c19fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:21.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0459d9b700 1 -- 172.21.15.163:0/3485888941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f045c1044a0 msgr2=0x7f045c19fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:21.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0459d9b700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f045c1044a0 0x7f045c19fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:21.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0459d9b700 1 -- 172.21.15.163:0/3485888941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c1029d0 msgr2=0x7f045c198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:21.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0459d9b700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c1029d0 0x7f045c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:21.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f0459d9b700 1 -- 172.21.15.163:0/3485888941 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045c1a0240 con 0x7f045c103b00 2024-05-27T07:30:21.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.006+0000 7f045ad9d700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f045c1044a0 0x7f045c19fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:21.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.007+0000 7f0459d9b700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f045c103b00 0x7f045c198cb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f045000d950 tx=0x7f045000dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:21.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.007+0000 7f044b7fe700 1 -- 172.21.15.163:0/3485888941 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450016070 con 0x7f045c103b00 2024-05-27T07:30:21.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.007+0000 7f044b7fe700 1 -- 172.21.15.163:0/3485888941 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0450010460 con 0x7f045c103b00 2024-05-27T07:30:21.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.007+0000 7f0460eb4700 1 -- 172.21.15.163:0/3485888941 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f045c1a0530 con 0x7f045c103b00 2024-05-27T07:30:21.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.007+0000 7f044b7fe700 1 -- 172.21.15.163:0/3485888941 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045000f620 con 0x7f045c103b00 2024-05-27T07:30:21.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.007+0000 7f0460eb4700 1 -- 172.21.15.163:0/3485888941 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f045c1a0ac0 con 0x7f045c103b00 2024-05-27T07:30:21.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.008+0000 7f0460eb4700 1 -- 172.21.15.163:0/3485888941 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f045c0512b0 con 0x7f045c103b00 2024-05-27T07:30:21.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.009+0000 7f044b7fe700 1 -- 172.21.15.163:0/3485888941 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 15) ==== 149597+0+0 (secure 0 0 0) 0x7f0450019070 con 0x7f045c103b00 2024-05-27T07:30:21.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.010+0000 7f044b7fe700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f04440b2870 0x7f04440b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:21.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.010+0000 7f044b7fe700 1 -- 172.21.15.163:0/3485888941 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f04500e0310 con 0x7f045c103b00 2024-05-27T07:30:21.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.010+0000 7f045a59c700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f04440b2870 0x7f04440b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:21.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.011+0000 7f045a59c700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f04440b2870 0x7f04440b4d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f044c00c6d0 tx=0x7f044c003cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:21.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.014+0000 7f044b7fe700 1 -- 172.21.15.163:0/3485888941 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f04500a9aa0 con 0x7f045c103b00 2024-05-27T07:30:21.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:20 smithi114 ceph-mon[32038]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:30:21.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:20 smithi114 ceph-mon[32038]: Detected new or changed devices on smithi114 2024-05-27T07:30:21.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:20 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi114 to 17088M 2024-05-27T07:30:21.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:21.224+0000 7f0460eb4700 1 -- 172.21.15.163:0/3485888941 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f045c063cb0 con 0x7f04440b2870 2024-05-27T07:30:21.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:21 smithi163 ceph-mon[36451]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T07:30:21.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:21 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:30:21.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:21 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:30:21.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:21 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:21.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:21 smithi186 ceph-mon[36647]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T07:30:21.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:21 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:30:21.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:21 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:30:21.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:21 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:22.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:21 smithi114 ceph-mon[32038]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T07:30:22.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:21 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:30:22.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:21 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:30:22.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:21 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:22.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:22 smithi163 ceph-mon[36451]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:30:22.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:22 smithi163 ceph-mon[36451]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:22.969 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:22 smithi186 ceph-mon[36647]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:30:22.969 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:22 smithi186 ceph-mon[36647]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:23.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:22 smithi114 ceph-mon[32038]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:30:23.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:22 smithi114 ceph-mon[32038]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:24.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:24 smithi163 ceph-mon[36451]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:24.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:24 smithi163 ceph-mon[36451]: from='client.? 172.21.15.163:0/2004914083' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e291f805-5a90-4c10-b6d6-3481a96f3f10"}]: dispatch 2024-05-27T07:30:24.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:24 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e291f805-5a90-4c10-b6d6-3481a96f3f10"}]: dispatch 2024-05-27T07:30:24.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:24 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e291f805-5a90-4c10-b6d6-3481a96f3f10"}]': finished 2024-05-27T07:30:24.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:24 smithi163 ceph-mon[36451]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T07:30:24.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:24 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:24.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:24 smithi163 ceph-mon[36451]: from='client.? 172.21.15.163:0/1906606724' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:30:24.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:24 smithi186 ceph-mon[36647]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:24.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:24 smithi186 ceph-mon[36647]: from='client.? 172.21.15.163:0/2004914083' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e291f805-5a90-4c10-b6d6-3481a96f3f10"}]: dispatch 2024-05-27T07:30:24.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:24 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e291f805-5a90-4c10-b6d6-3481a96f3f10"}]: dispatch 2024-05-27T07:30:24.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:24 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e291f805-5a90-4c10-b6d6-3481a96f3f10"}]': finished 2024-05-27T07:30:24.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:24 smithi186 ceph-mon[36647]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T07:30:24.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:24 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:24.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:24 smithi186 ceph-mon[36647]: from='client.? 172.21.15.163:0/1906606724' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:30:25.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:24 smithi114 ceph-mon[32038]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:25.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:24 smithi114 ceph-mon[32038]: from='client.? 172.21.15.163:0/2004914083' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e291f805-5a90-4c10-b6d6-3481a96f3f10"}]: dispatch 2024-05-27T07:30:25.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:24 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e291f805-5a90-4c10-b6d6-3481a96f3f10"}]: dispatch 2024-05-27T07:30:25.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:24 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e291f805-5a90-4c10-b6d6-3481a96f3f10"}]': finished 2024-05-27T07:30:25.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:24 smithi114 ceph-mon[32038]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T07:30:25.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:24 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:25.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:24 smithi114 ceph-mon[32038]: from='client.? 172.21.15.163:0/1906606724' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:30:26.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:26 smithi163 ceph-mon[36451]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:26.969 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:26 smithi186 ceph-mon[36647]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:27.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:26 smithi114 ceph-mon[32038]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:28.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:28 smithi186 ceph-mon[36647]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:28.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:28 smithi163 ceph-mon[36451]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:29.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:28 smithi114 ceph-mon[32038]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:30.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:30 smithi163 ceph-mon[36451]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:31.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:30 smithi114 ceph-mon[32038]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:31.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:30 smithi186 ceph-mon[36647]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:32.036 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:31 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T07:30:32.036 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:31 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:32.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:31 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T07:30:32.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:31 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:32.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:31 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T07:30:32.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:31 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:32.772 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:32 smithi163 ceph-mon[36451]: Deploying daemon osd.1 on smithi163 2024-05-27T07:30:32.773 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:32 smithi163 ceph-mon[36451]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:33.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:32 smithi114 ceph-mon[32038]: Deploying daemon osd.1 on smithi163 2024-05-27T07:30:33.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:32 smithi114 ceph-mon[32038]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:33.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:32 smithi186 ceph-mon[36647]: Deploying daemon osd.1 on smithi163 2024-05-27T07:30:33.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:32 smithi186 ceph-mon[36647]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:34.815 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:34 smithi163 ceph-mon[36451]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:35.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:34 smithi114 ceph-mon[32038]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:35.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:34 smithi186 ceph-mon[36647]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:36.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:36 smithi163 ceph-mon[36451]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:36.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:36.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:36.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:37.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:36 smithi186 ceph-mon[36647]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:37.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:37.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:37.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:37.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:36 smithi114 ceph-mon[32038]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:37.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:37.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:37.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:39.012 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:38 smithi163 ceph-mon[36451]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:39.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:38 smithi114 ceph-mon[32038]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:39.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:38 smithi186 ceph-mon[36647]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:40.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:40.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:40.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:40.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:40.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:40.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:40.347+0000 7f044b7fe700 1 -- 172.21.15.163:0/3485888941 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f045c063cb0 con 0x7f04440b2870 2024-05-27T07:30:40.340 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 1 on host 'smithi163' 2024-05-27T07:30:40.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:40.349+0000 7f0460eb4700 1 -- 172.21.15.163:0/3485888941 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f04440b2870 msgr2=0x7f04440b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:40.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:40.349+0000 7f0460eb4700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f04440b2870 0x7f04440b4d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f044c00c6d0 tx=0x7f044c003cc0 comp rx=0 tx=0).stop 2024-05-27T07:30:40.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:40.349+0000 7f0460eb4700 1 -- 172.21.15.163:0/3485888941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f045c103b00 msgr2=0x7f045c198cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:40.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:40.349+0000 7f0460eb4700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f045c103b00 0x7f045c198cb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f045000d950 tx=0x7f045000dd10 comp rx=0 tx=0).stop 2024-05-27T07:30:40.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:40.349+0000 7f0460eb4700 1 -- 172.21.15.163:0/3485888941 shutdown_connections 2024-05-27T07:30:40.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:40.349+0000 7f0460eb4700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f04440b2870 0x7f04440b4d80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:40.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:40.349+0000 7f0460eb4700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f045c1044a0 0x7f045c19fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:40.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:40.349+0000 7f0460eb4700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c1029d0 0x7f045c198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:40.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:40.349+0000 7f0460eb4700 1 --2- 172.21.15.163:0/3485888941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f045c103b00 0x7f045c198cb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:40.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:40.349+0000 7f0460eb4700 1 -- 172.21.15.163:0/3485888941 >> 172.21.15.163:0/3485888941 conn(0x7f045c0fe010 msgr2=0x7f045c106d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:40.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:40.349+0000 7f0460eb4700 1 -- 172.21.15.163:0/3485888941 shutdown_connections 2024-05-27T07:30:40.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:30:40.349+0000 7f0460eb4700 1 -- 172.21.15.163:0/3485888941 wait complete. 2024-05-27T07:30:40.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:40.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:40.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:40.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:40.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:40.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:40.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:40.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:40.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:40.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:41.025 DEBUG:teuthology.orchestra.run.smithi163:osd.1> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.1.service 2024-05-27T07:30:41.028 INFO:tasks.cephadm:Deploying osd.2 on smithi186 with /dev/vg_nvme/lv_4... 2024-05-27T07:30:41.028 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T07:30:41.271 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:30:41.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:41 smithi163 ceph-mon[36451]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:41.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:41.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:41.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:41.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:41 smithi163 ceph-mon[36451]: from='osd.1 [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:30:41.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:41 smithi163 ceph-mon[36451]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:30:41.393 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:41 smithi186 ceph-mon[36647]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:41.393 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:41.393 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:41.393 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:41.394 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:41 smithi186 ceph-mon[36647]: from='osd.1 [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:30:41.394 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:41 smithi186 ceph-mon[36647]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:30:41.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:41 smithi114 ceph-mon[32038]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:41.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:41.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:41.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:41.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:41 smithi114 ceph-mon[32038]: from='osd.1 [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:30:41.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:41 smithi114 ceph-mon[32038]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:30:42.586 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:42 smithi186 ceph-mon[36647]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:42.586 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:42 smithi186 ceph-mon[36647]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T07:30:42.586 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:42 smithi186 ceph-mon[36647]: from='osd.1 [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:30:42.586 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:42 smithi186 ceph-mon[36647]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T07:30:42.586 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:42 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:42.586 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:42 smithi186 ceph-mon[36647]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:30:42.652 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:42 smithi163 ceph-mon[36451]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:42.652 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:42 smithi163 ceph-mon[36451]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T07:30:42.652 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:42 smithi163 ceph-mon[36451]: from='osd.1 [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:30:42.652 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:42 smithi163 ceph-mon[36451]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T07:30:42.653 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:42 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:42.653 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:42 smithi163 ceph-mon[36451]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:30:42.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:42 smithi114 ceph-mon[32038]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:42.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:42 smithi114 ceph-mon[32038]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T07:30:42.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:42 smithi114 ceph-mon[32038]: from='osd.1 [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:30:42.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:42 smithi114 ceph-mon[32038]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T07:30:42.729 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:42 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:42.729 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:42 smithi114 ceph-mon[32038]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:30:42.968 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 07:30:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1[40993]: 2024-05-27T07:30:42.800+0000 7f08e93d8700 -1 osd.1 0 waiting for initial osdmap 2024-05-27T07:30:42.968 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 07:30:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1[40993]: 2024-05-27T07:30:42.803+0000 7f08e0984700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:30:43.034 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T07:30:43.035 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T07:30:43.035 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-05-27T07:30:43.035 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-05-27T07:30:43.035 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129526 s, 810 MB/s 2024-05-27T07:30:43.035 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-05-27T07:30:43.640 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:43 smithi186 ceph-mon[36647]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-27T07:30:43.640 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:43 smithi186 ceph-mon[36647]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T07:30:43.640 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:43.640 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:43.640 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:43 smithi186 ceph-mon[36647]: from='osd.1 ' entity='osd.1' 2024-05-27T07:30:43.664 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:43 smithi114 ceph-mon[32038]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-27T07:30:43.664 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:43 smithi114 ceph-mon[32038]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T07:30:43.664 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:43.664 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:43.664 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:43 smithi114 ceph-mon[32038]: from='osd.1 ' entity='osd.1' 2024-05-27T07:30:43.666 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_4 2024-05-27T07:30:43.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:43 smithi163 ceph-mon[36451]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-27T07:30:43.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:43 smithi163 ceph-mon[36451]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T07:30:43.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:43.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:43.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:43 smithi163 ceph-mon[36451]: from='osd.1 ' entity='osd.1' 2024-05-27T07:30:43.904 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:30:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: purged_snaps scrub starts 2024-05-27T07:30:44.467 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: purged_snaps scrub ok 2024-05-27T07:30:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: osd.1 [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241] boot 2024-05-27T07:30:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T07:30:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:30:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:44.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:44.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: purged_snaps scrub starts 2024-05-27T07:30:44.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: purged_snaps scrub ok 2024-05-27T07:30:44.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:44.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:44.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: osd.1 [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241] boot 2024-05-27T07:30:44.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T07:30:44.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:44.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:44.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:44.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:30:44.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:44.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:44.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:44.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:44.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: purged_snaps scrub starts 2024-05-27T07:30:44.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: purged_snaps scrub ok 2024-05-27T07:30:44.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:30:44.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:44.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: osd.1 [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241] boot 2024-05-27T07:30:44.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T07:30:44.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:30:44.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:44.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:44.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:30:44.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:44.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:44.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:44.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:30:45.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.437+0000 7f05e045f700 1 -- 172.21.15.186:0/1421507105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05d8105e30 msgr2=0x7f05d810c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:45.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.437+0000 7f05e045f700 1 --2- 172.21.15.186:0/1421507105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05d8105e30 0x7f05d810c6e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f05d000b0a0 tx=0x7f05d000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:30:45.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.437+0000 7f05e045f700 1 -- 172.21.15.186:0/1421507105 shutdown_connections 2024-05-27T07:30:45.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.437+0000 7f05e045f700 1 --2- 172.21.15.186:0/1421507105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05d8105e30 0x7f05d810c6e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:45.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.437+0000 7f05e045f700 1 --2- 172.21.15.186:0/1421507105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05d8104360 0x7f05d8104770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:45.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.437+0000 7f05e045f700 1 --2- 172.21.15.186:0/1421507105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05d8105490 0x7f05d81058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:45.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.437+0000 7f05e045f700 1 -- 172.21.15.186:0/1421507105 >> 172.21.15.186:0/1421507105 conn(0x7f05d80ff9c0 msgr2=0x7f05d8101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:45.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.437+0000 7f05e045f700 1 -- 172.21.15.186:0/1421507105 shutdown_connections 2024-05-27T07:30:45.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.437+0000 7f05e045f700 1 -- 172.21.15.186:0/1421507105 wait complete. 2024-05-27T07:30:45.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.437+0000 7f05e045f700 1 Processor -- start 2024-05-27T07:30:45.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05e045f700 1 -- start start 2024-05-27T07:30:45.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05e045f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05d8104360 0x7f05d819a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:45.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05e045f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05d8105490 0x7f05d819ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:45.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05e045f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05d8105e30 0x7f05d81a1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:45.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05e045f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f05d810fea0 con 0x7f05d8105e30 2024-05-27T07:30:45.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05e045f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f05d810fd20 con 0x7f05d8105490 2024-05-27T07:30:45.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05e045f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f05d8110020 con 0x7f05d8104360 2024-05-27T07:30:45.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05de1fb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05d8104360 0x7f05d819a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:45.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05de1fb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05d8104360 0x7f05d819a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36282/0 (socket says 172.21.15.186:36282) 2024-05-27T07:30:45.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05de1fb700 1 -- 172.21.15.186:0/2787492439 learned_addr learned my addr 172.21.15.186:0/2787492439 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-27T07:30:45.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05de9fc700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05d8105e30 0x7f05d81a1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:45.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05dd9fa700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05d8105490 0x7f05d819ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:45.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05de1fb700 1 -- 172.21.15.186:0/2787492439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05d8105490 msgr2=0x7f05d819ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:45.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05de1fb700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05d8105490 0x7f05d819ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:45.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05de1fb700 1 -- 172.21.15.186:0/2787492439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05d8105e30 msgr2=0x7f05d81a1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:45.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05de1fb700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05d8105e30 0x7f05d81a1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:45.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05de1fb700 1 -- 172.21.15.186:0/2787492439 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d81a2400 con 0x7f05d8104360 2024-05-27T07:30:45.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05dd9fa700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05d8105490 0x7f05d819ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:45.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.438+0000 7f05de1fb700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05d8104360 0x7f05d819a930 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f05c800b4d0 tx=0x7f05c800b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:45.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.439+0000 7f05cf7fe700 1 -- 172.21.15.186:0/2787492439 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8015020 con 0x7f05d8104360 2024-05-27T07:30:45.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.439+0000 7f05e045f700 1 -- 172.21.15.186:0/2787492439 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d81a26f0 con 0x7f05d8104360 2024-05-27T07:30:45.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.439+0000 7f05cf7fe700 1 -- 172.21.15.186:0/2787492439 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05c8011ce0 con 0x7f05d8104360 2024-05-27T07:30:45.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.439+0000 7f05cf7fe700 1 -- 172.21.15.186:0/2787492439 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8010cf0 con 0x7f05d8104360 2024-05-27T07:30:45.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.439+0000 7f05e045f700 1 -- 172.21.15.186:0/2787492439 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d81a2cb0 con 0x7f05d8104360 2024-05-27T07:30:45.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.440+0000 7f05e045f700 1 -- 172.21.15.186:0/2787492439 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05d80512b0 con 0x7f05d8104360 2024-05-27T07:30:45.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.441+0000 7f05cf7fe700 1 -- 172.21.15.186:0/2787492439 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) ==== 149597+0+0 (secure 0 0 0) 0x7f05c8018020 con 0x7f05d8104360 2024-05-27T07:30:45.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.442+0000 7f05cf7fe700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f05c40b27f0 0x7f05c40b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:45.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.442+0000 7f05dd9fa700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f05c40b27f0 0x7f05c40b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:45.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.442+0000 7f05cf7fe700 1 -- 172.21.15.186:0/2787492439 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f05c80e0680 con 0x7f05d8104360 2024-05-27T07:30:45.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.443+0000 7f05dd9fa700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f05c40b27f0 0x7f05c40b4d00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f05d4005fd0 tx=0x7f05d4005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:45.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.446+0000 7f05cf7fe700 1 -- 172.21.15.186:0/2787492439 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f05c801f020 con 0x7f05d8104360 2024-05-27T07:30:45.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:45 smithi186 ceph-mon[36647]: Detected new or changed devices on smithi163 2024-05-27T07:30:45.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:45 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi163 to 17088M 2024-05-27T07:30:45.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:30:45.659+0000 7f05e045f700 1 -- 172.21.15.186:0/2787492439 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f05d8063bf0 con 0x7f05c40b27f0 2024-05-27T07:30:45.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:45 smithi163 ceph-mon[36451]: Detected new or changed devices on smithi163 2024-05-27T07:30:45.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:45 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi163 to 17088M 2024-05-27T07:30:45.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:45 smithi114 ceph-mon[32038]: Detected new or changed devices on smithi163 2024-05-27T07:30:45.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:45 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi163 to 17088M 2024-05-27T07:30:46.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:46 smithi186 ceph-mon[36647]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:46.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:46 smithi186 ceph-mon[36647]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T07:30:46.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:46 smithi186 ceph-mon[36647]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:30:46.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:30:46.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:30:46.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:46.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:46 smithi163 ceph-mon[36451]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:46.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:46 smithi163 ceph-mon[36451]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T07:30:46.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:46 smithi163 ceph-mon[36451]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:30:46.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:30:46.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:30:46.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:46.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:46 smithi114 ceph-mon[32038]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:46.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:46 smithi114 ceph-mon[32038]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T07:30:46.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:46 smithi114 ceph-mon[32038]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:30:46.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:30:46.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:30:46.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:48.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:48 smithi163 ceph-mon[36451]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:48.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:48 smithi163 ceph-mon[36451]: from='client.? 172.21.15.186:0/1807083899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bde05b7-5b12-46fe-a8f3-6992ed96f603"}]: dispatch 2024-05-27T07:30:48.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:48 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bde05b7-5b12-46fe-a8f3-6992ed96f603"}]: dispatch 2024-05-27T07:30:48.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:48 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bde05b7-5b12-46fe-a8f3-6992ed96f603"}]': finished 2024-05-27T07:30:48.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:48 smithi163 ceph-mon[36451]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T07:30:48.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:48 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:30:48.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:48 smithi186 ceph-mon[36647]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:48.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:48 smithi186 ceph-mon[36647]: from='client.? 172.21.15.186:0/1807083899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bde05b7-5b12-46fe-a8f3-6992ed96f603"}]: dispatch 2024-05-27T07:30:48.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:48 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bde05b7-5b12-46fe-a8f3-6992ed96f603"}]: dispatch 2024-05-27T07:30:48.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:48 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bde05b7-5b12-46fe-a8f3-6992ed96f603"}]': finished 2024-05-27T07:30:48.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:48 smithi186 ceph-mon[36647]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T07:30:48.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:48 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:30:48.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:48 smithi114 ceph-mon[32038]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:48.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:48 smithi114 ceph-mon[32038]: from='client.? 172.21.15.186:0/1807083899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bde05b7-5b12-46fe-a8f3-6992ed96f603"}]: dispatch 2024-05-27T07:30:48.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:48 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bde05b7-5b12-46fe-a8f3-6992ed96f603"}]: dispatch 2024-05-27T07:30:48.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:48 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bde05b7-5b12-46fe-a8f3-6992ed96f603"}]': finished 2024-05-27T07:30:48.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:48 smithi114 ceph-mon[32038]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T07:30:48.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:48 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:30:49.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:49 smithi163 ceph-mon[36451]: from='client.? 172.21.15.186:0/3837915117' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:30:49.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:49 smithi186 ceph-mon[36647]: from='client.? 172.21.15.186:0/3837915117' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:30:49.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:49 smithi114 ceph-mon[32038]: from='client.? 172.21.15.186:0/3837915117' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:30:50.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:50 smithi163 ceph-mon[36451]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:50.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:50 smithi186 ceph-mon[36647]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:50.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:50 smithi114 ceph-mon[32038]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:52.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:52 smithi186 ceph-mon[36647]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:52.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:52 smithi163 ceph-mon[36451]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:52.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:52 smithi114 ceph-mon[32038]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:54 smithi186 ceph-mon[36647]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:54.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:54 smithi163 ceph-mon[36451]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:54.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:54 smithi114 ceph-mon[32038]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:56.610 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:56 smithi186 ceph-mon[36647]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:56.610 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:56 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T07:30:56.610 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:56 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:56.610 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:56 smithi186 ceph-mon[36647]: Deploying daemon osd.2 on smithi186 2024-05-27T07:30:56.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:56 smithi163 ceph-mon[36451]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:56.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:56 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T07:30:56.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:56 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:56.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:56 smithi163 ceph-mon[36451]: Deploying daemon osd.2 on smithi186 2024-05-27T07:30:56.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:56 smithi114 ceph-mon[32038]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:56.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T07:30:56.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:56.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:56 smithi114 ceph-mon[32038]: Deploying daemon osd.2 on smithi186 2024-05-27T07:30:58.469 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:30:58 smithi186 ceph-mon[36647]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:58.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:30:58 smithi163 ceph-mon[36451]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:30:58.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:30:58 smithi114 ceph-mon[32038]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:00.588 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:00 smithi186 ceph-mon[36647]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:00.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:00 smithi163 ceph-mon[36451]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:00.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:00 smithi114 ceph-mon[32038]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:02.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:01 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:02.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:01 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:02.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:01 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:02.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:01 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:02.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:01 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:02.221 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:01 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:02.298 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:01 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:02.298 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:01 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:02.298 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:01 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:02.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:02 smithi186 ceph-mon[36647]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:03.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:02 smithi163 ceph-mon[36451]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:03.298 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:02 smithi114 ceph-mon[32038]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:04.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:31:04.178+0000 7f05cf7fe700 1 -- 172.21.15.186:0/2787492439 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f05d8063bf0 con 0x7f05c40b27f0 2024-05-27T07:31:04.173 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 2 on host 'smithi186' 2024-05-27T07:31:04.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:31:04.180+0000 7f05e045f700 1 -- 172.21.15.186:0/2787492439 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f05c40b27f0 msgr2=0x7f05c40b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:04.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:31:04.180+0000 7f05e045f700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f05c40b27f0 0x7f05c40b4d00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f05d4005fd0 tx=0x7f05d4005e40 comp rx=0 tx=0).stop 2024-05-27T07:31:04.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:31:04.180+0000 7f05e045f700 1 -- 172.21.15.186:0/2787492439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05d8104360 msgr2=0x7f05d819a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:04.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:31:04.180+0000 7f05e045f700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05d8104360 0x7f05d819a930 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f05c800b4d0 tx=0x7f05c800b7e0 comp rx=0 tx=0).stop 2024-05-27T07:31:04.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:31:04.180+0000 7f05e045f700 1 -- 172.21.15.186:0/2787492439 shutdown_connections 2024-05-27T07:31:04.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:31:04.180+0000 7f05e045f700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05d8105e30 0x7f05d81a1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:04.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:31:04.180+0000 7f05e045f700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f05c40b27f0 0x7f05c40b4d00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:04.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:31:04.180+0000 7f05e045f700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05d8104360 0x7f05d819a930 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:04.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:31:04.180+0000 7f05e045f700 1 --2- 172.21.15.186:0/2787492439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05d8105490 0x7f05d819ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:04.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:31:04.180+0000 7f05e045f700 1 -- 172.21.15.186:0/2787492439 >> 172.21.15.186:0/2787492439 conn(0x7f05d80ff9c0 msgr2=0x7f05d81086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:04.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:31:04.181+0000 7f05e045f700 1 -- 172.21.15.186:0/2787492439 shutdown_connections 2024-05-27T07:31:04.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:31:04.181+0000 7f05e045f700 1 -- 172.21.15.186:0/2787492439 wait complete. 2024-05-27T07:31:04.724 DEBUG:teuthology.orchestra.run.smithi186:osd.2> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.2.service 2024-05-27T07:31:04.726 INFO:tasks.cephadm:Deploying osd.3 on smithi114 with /dev/vg_nvme/lv_3... 2024-05-27T07:31:04.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T07:31:04.973 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:05.029 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 07:31:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2[41791]: 2024-05-27T07:31:04.762+0000 7f875dd4b7c0 -1 osd.2 0 log_to_monitors true 2024-05-27T07:31:05.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:04 smithi186 ceph-mon[36647]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:05.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:05.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:05.030 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:05.030 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:05.030 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:04 smithi186 ceph-mon[36647]: from='osd.2 [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:31:05.030 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:04 smithi186 ceph-mon[36647]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:31:05.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:04 smithi114 ceph-mon[32038]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:05.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:05.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:05.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:05.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:05.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:04 smithi114 ceph-mon[32038]: from='osd.2 [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:31:05.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:04 smithi114 ceph-mon[32038]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:31:05.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:04 smithi163 ceph-mon[36451]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:05.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:05.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:05.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:05.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:05.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:04 smithi163 ceph-mon[36451]: from='osd.2 [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:31:05.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:04 smithi163 ceph-mon[36451]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:31:06.703 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:06 smithi186 ceph-mon[36647]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:06.703 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:06 smithi186 ceph-mon[36647]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T07:31:06.703 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:06 smithi186 ceph-mon[36647]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T07:31:06.703 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:06 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:06.703 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:06 smithi186 ceph-mon[36647]: from='osd.2 [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:31:06.703 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:06 smithi186 ceph-mon[36647]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:31:06.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:06 smithi163 ceph-mon[36451]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:06.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:06 smithi163 ceph-mon[36451]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T07:31:06.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:06 smithi163 ceph-mon[36451]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T07:31:06.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:06 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:06.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:06 smithi163 ceph-mon[36451]: from='osd.2 [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:31:06.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:06 smithi163 ceph-mon[36451]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:31:06.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:06 smithi114 ceph-mon[32038]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:06.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:06 smithi114 ceph-mon[32038]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T07:31:06.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:06 smithi114 ceph-mon[32038]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T07:31:06.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:06 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:06.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:06 smithi114 ceph-mon[32038]: from='osd.2 [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:31:06.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:06 smithi114 ceph-mon[32038]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:31:06.971 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 07:31:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2[41791]: 2024-05-27T07:31:06.956+0000 7f87533af700 -1 osd.2 0 waiting for initial osdmap 2024-05-27T07:31:06.971 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 07:31:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2[41791]: 2024-05-27T07:31:06.960+0000 7f874b15c700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:31:06.992 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T07:31:06.993 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T07:31:06.993 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-05-27T07:31:06.993 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-05-27T07:31:06.993 INFO:teuthology.orchestra.run.smithi114.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114666 s, 914 MB/s 2024-05-27T07:31:06.993 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-05-27T07:31:07.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:07 smithi114 ceph-mon[32038]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-27T07:31:07.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:07 smithi114 ceph-mon[32038]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T07:31:07.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:07 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:07.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:07 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:07.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:07 smithi114 ceph-mon[32038]: from='osd.2 ' entity='osd.2' 2024-05-27T07:31:07.666 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:07 smithi186 ceph-mon[36647]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-27T07:31:07.667 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:07 smithi186 ceph-mon[36647]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T07:31:07.667 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:07 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:07.667 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:07 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:07.667 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:07 smithi186 ceph-mon[36647]: from='osd.2 ' entity='osd.2' 2024-05-27T07:31:07.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_3 2024-05-27T07:31:07.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:07 smithi163 ceph-mon[36451]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-27T07:31:07.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:07 smithi163 ceph-mon[36451]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T07:31:07.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:07 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:07.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:07 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:07.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:07 smithi163 ceph-mon[36451]: from='osd.2 ' entity='osd.2' 2024-05-27T07:31:07.920 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:08.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: purged_snaps scrub starts 2024-05-27T07:31:08.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: purged_snaps scrub ok 2024-05-27T07:31:08.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:08.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:08.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: osd.2 [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108] boot 2024-05-27T07:31:08.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T07:31:08.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:08.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: Detected new or changed devices on smithi186 2024-05-27T07:31:08.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:08.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi186 to 17088M 2024-05-27T07:31:08.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:08.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:08.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:08.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:08.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:08.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: purged_snaps scrub starts 2024-05-27T07:31:08.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: purged_snaps scrub ok 2024-05-27T07:31:08.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: osd.2 [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108] boot 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: Detected new or changed devices on smithi186 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi186 to 17088M 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:08.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:08.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:08.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.758 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: purged_snaps scrub starts 2024-05-27T07:31:08.758 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: purged_snaps scrub ok 2024-05-27T07:31:08.758 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:31:08.758 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:08.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: osd.2 [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108] boot 2024-05-27T07:31:08.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T07:31:08.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:31:08.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: Detected new or changed devices on smithi186 2024-05-27T07:31:08.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:08.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi186 to 17088M 2024-05-27T07:31:08.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:08.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:08.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:08.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:08.760 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:08.760 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:08.760 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:09.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.618+0000 7f9646213700 1 -- 172.21.15.114:0/3869823429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9640105e30 msgr2=0x7f964010c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:09.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.618+0000 7f9646213700 1 --2- 172.21.15.114:0/3869823429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9640105e30 0x7f964010c6e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f963800b0a0 tx=0x7f963800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:31:09.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.618+0000 7f9646213700 1 -- 172.21.15.114:0/3869823429 shutdown_connections 2024-05-27T07:31:09.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.618+0000 7f9646213700 1 --2- 172.21.15.114:0/3869823429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9640105e30 0x7f964010c6e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.618+0000 7f9646213700 1 --2- 172.21.15.114:0/3869823429 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9640104360 0x7f9640104770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.618+0000 7f9646213700 1 --2- 172.21.15.114:0/3869823429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640105490 0x7f96401058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.618+0000 7f9646213700 1 -- 172.21.15.114:0/3869823429 >> 172.21.15.114:0/3869823429 conn(0x7f96400ff9c0 msgr2=0x7f9640101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.618+0000 7f9646213700 1 -- 172.21.15.114:0/3869823429 shutdown_connections 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.618+0000 7f9646213700 1 -- 172.21.15.114:0/3869823429 wait complete. 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.619+0000 7f9646213700 1 Processor -- start 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.619+0000 7f9646213700 1 -- start start 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.619+0000 7f9646213700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9640104360 0x7f9640078560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.619+0000 7f9646213700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640105490 0x7f9640076c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.619+0000 7f9646213700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9640105e30 0x7f9640077150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.619+0000 7f9646213700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f964010fec0 con 0x7f9640104360 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.619+0000 7f9646213700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f964010fd40 con 0x7f9640105e30 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.619+0000 7f9646213700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9640110040 con 0x7f9640105490 2024-05-27T07:31:09.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f963f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9640104360 0x7f9640078560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:09.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f963f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9640104360 0x7f9640078560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38974/0 (socket says 172.21.15.114:38974) 2024-05-27T07:31:09.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f963f7fe700 1 -- 172.21.15.114:0/1852399451 learned_addr learned my addr 172.21.15.114:0/1852399451 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:31:09.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f963f7fe700 1 -- 172.21.15.114:0/1852399451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640105490 msgr2=0x7f9640076c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:09.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f963f7fe700 1 --2- 172.21.15.114:0/1852399451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640105490 0x7f9640076c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:09.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f963f7fe700 1 -- 172.21.15.114:0/1852399451 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9640105e30 msgr2=0x7f9640077150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:09.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f963f7fe700 1 --2- 172.21.15.114:0/1852399451 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9640105e30 0x7f9640077150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:09.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f963f7fe700 1 -- 172.21.15.114:0/1852399451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96400778b0 con 0x7f9640104360 2024-05-27T07:31:09.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f963f7fe700 1 --2- 172.21.15.114:0/1852399451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9640104360 0x7f9640078560 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f963000b4d0 tx=0x7f963000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:09.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f963d7fa700 1 -- 172.21.15.114:0/1852399451 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9630015020 con 0x7f9640104360 2024-05-27T07:31:09.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f963d7fa700 1 -- 172.21.15.114:0/1852399451 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9630011ce0 con 0x7f9640104360 2024-05-27T07:31:09.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f9646213700 1 -- 172.21.15.114:0/1852399451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96401a4a80 con 0x7f9640104360 2024-05-27T07:31:09.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f963d7fa700 1 -- 172.21.15.114:0/1852399451 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9630010cf0 con 0x7f9640104360 2024-05-27T07:31:09.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.620+0000 7f9646213700 1 -- 172.21.15.114:0/1852399451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96401a4fe0 con 0x7f9640104360 2024-05-27T07:31:09.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.621+0000 7f9646213700 1 -- 172.21.15.114:0/1852399451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96400512b0 con 0x7f9640104360 2024-05-27T07:31:09.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.623+0000 7f963d7fa700 1 -- 172.21.15.114:0/1852399451 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 149597+0+0 (secure 0 0 0) 0x7f9630018020 con 0x7f9640104360 2024-05-27T07:31:09.616 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:09 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:31:09.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.623+0000 7f963d7fa700 1 --2- 172.21.15.114:0/1852399451 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f962c0b27f0 0x7f962c0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:09.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.624+0000 7f9637fff700 1 --2- 172.21.15.114:0/1852399451 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f962c0b27f0 0x7f962c0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:09.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.624+0000 7f963d7fa700 1 -- 172.21.15.114:0/1852399451 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7f96300e05a0 con 0x7f9640104360 2024-05-27T07:31:09.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.624+0000 7f9637fff700 1 --2- 172.21.15.114:0/1852399451 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f962c0b27f0 0x7f962c0b4d00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9640077f10 tx=0x7f9628006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:09.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.629+0000 7f963d7fa700 1 -- 172.21.15.114:0/1852399451 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f963001f020 con 0x7f9640104360 2024-05-27T07:31:09.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:09 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:31:09.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:09 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:31:09.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:09.840+0000 7f9646213700 1 -- 172.21.15.114:0/1852399451 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f9640063bf0 con 0x7f962c0b27f0 2024-05-27T07:31:10.698 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 ceph-mon[32038]: pgmap v81: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:10.698 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T07:31:10.698 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 ceph-mon[32038]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T07:31:10.698 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:31:10.698 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 ceph-mon[32038]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:31:10.698 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:31:10.698 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:31:10.698 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:10.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48540]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:31:10.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48540]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:31:10.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48540]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:31:10.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48540]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:31:10.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48544]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:31:10.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48544]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:31:10.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48544]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:31:10.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48544]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:31:10.717 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44679]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:31:10.718 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44679]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:31:10.718 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44679]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:31:10.718 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44679]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:31:10.718 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44682]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:31:10.718 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44682]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:31:10.718 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44682]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:31:10.718 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44682]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:31:10.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 ceph-mon[36451]: pgmap v81: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:10.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T07:31:10.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 ceph-mon[36451]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T07:31:10.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:31:10.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 ceph-mon[36451]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:31:10.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:31:10.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:31:10.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:10.719 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45822]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:31:10.719 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45822]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:31:10.719 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45822]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:31:10.720 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45822]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:31:10.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 ceph-mon[36647]: pgmap v81: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:10.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T07:31:10.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 ceph-mon[36647]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T07:31:10.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:31:10.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 ceph-mon[36647]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:31:10.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:31:10.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:31:10.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:10.970 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45825]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:31:10.971 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45825]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:31:10.971 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45825]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:31:10.971 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45825]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:31:10.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45828]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:31:10.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45828]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:31:10.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45828]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:31:10.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45828]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:31:10.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45831]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:31:10.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45831]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:31:10.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45831]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:31:10.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:10 smithi186 sudo[45831]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:31:11.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48563]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:31:11.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48563]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:31:11.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48563]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:31:11.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48563]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:31:11.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48566]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:31:11.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48566]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:31:11.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48566]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:31:11.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:10 smithi114 sudo[48566]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:31:11.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44685]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:31:11.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44685]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:31:11.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44685]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:31:11.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44685]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:31:11.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44688]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:31:11.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44688]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:31:11.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44688]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:31:11.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:10 smithi163 sudo[44688]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:31:11.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:31:11.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T07:31:11.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:11.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:11.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:11.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:31:11.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:31:11.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:31:11.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:31:11.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:31:11.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:31:11.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:31:11.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:31:11.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:31:11.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:31:11.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:31:11.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:31:11.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:31:11.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:31:11.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:11 smithi186 ceph-mon[36647]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:31:11.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:11.768+0000 7f963d7fa700 1 -- 172.21.15.114:0/1852399451 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f96300a5310 con 0x7f9640104360 2024-05-27T07:31:11.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:31:11.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T07:31:11.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:11.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:11.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:11.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:31:11.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:31:11.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:31:11.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:31:11.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:31:11.763 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:31:11.763 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:31:11.763 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:31:11.763 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:31:11.763 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:31:11.763 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:31:11.763 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:31:11.763 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:31:11.763 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:31:11.763 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:11 smithi114 ceph-mon[32038]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:31:11.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:31:11.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T07:31:11.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:11.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:11.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:11.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:31:11.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:11 smithi163 ceph-mon[36451]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:31:12.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:12 smithi186 ceph-mon[36647]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:12.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:12 smithi186 ceph-mon[36647]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T07:31:12.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:12 smithi186 ceph-mon[36647]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-27T07:31:12.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:12 smithi114 ceph-mon[32038]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:12.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:12 smithi114 ceph-mon[32038]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T07:31:12.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:12 smithi114 ceph-mon[32038]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-27T07:31:12.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:12 smithi163 ceph-mon[36451]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:12.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:12 smithi163 ceph-mon[36451]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T07:31:12.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:12 smithi163 ceph-mon[36451]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-27T07:31:13.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:13 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/412715231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d810071c-e2dd-442a-b5c5-080b81828549"}]: dispatch 2024-05-27T07:31:13.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:13 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/412715231' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d810071c-e2dd-442a-b5c5-080b81828549"}]': finished 2024-05-27T07:31:13.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:13 smithi114 ceph-mon[32038]: osdmap e22: 4 total, 3 up, 4 in 2024-05-27T07:31:13.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:13 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:13.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:13 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/670251594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:31:13.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:13 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/412715231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d810071c-e2dd-442a-b5c5-080b81828549"}]: dispatch 2024-05-27T07:31:13.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:13 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/412715231' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d810071c-e2dd-442a-b5c5-080b81828549"}]': finished 2024-05-27T07:31:13.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:13 smithi163 ceph-mon[36451]: osdmap e22: 4 total, 3 up, 4 in 2024-05-27T07:31:13.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:13 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:13.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:13 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/670251594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:31:13.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:13 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/412715231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d810071c-e2dd-442a-b5c5-080b81828549"}]: dispatch 2024-05-27T07:31:13.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:13 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/412715231' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d810071c-e2dd-442a-b5c5-080b81828549"}]': finished 2024-05-27T07:31:13.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:13 smithi186 ceph-mon[36647]: osdmap e22: 4 total, 3 up, 4 in 2024-05-27T07:31:13.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:13 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:13.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:13 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/670251594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:31:14.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:14 smithi163 ceph-mon[36451]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:14.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:14 smithi186 ceph-mon[36647]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:15.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:14 smithi114 ceph-mon[32038]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:16.967 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:16 smithi163 ceph-mon[36451]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:16.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:16 smithi186 ceph-mon[36647]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:17.019 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:16 smithi114 ceph-mon[32038]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:18.945 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:18 smithi114 ceph-mon[32038]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:18.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:18 smithi163 ceph-mon[36451]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:18.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:18 smithi186 ceph-mon[36647]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:20.926 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:20 smithi114 ceph-mon[32038]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:20.926 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T07:31:20.926 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:20.927 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:20 smithi114 ceph-mon[32038]: Deploying daemon osd.3 on smithi114 2024-05-27T07:31:20.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:20 smithi163 ceph-mon[36451]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:20.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T07:31:20.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:20.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:20 smithi163 ceph-mon[36451]: Deploying daemon osd.3 on smithi114 2024-05-27T07:31:20.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:20 smithi186 ceph-mon[36647]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:20.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T07:31:20.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:20.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:20 smithi186 ceph-mon[36647]: Deploying daemon osd.3 on smithi114 2024-05-27T07:31:22.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:22 smithi163 ceph-mon[36451]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:22.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:22 smithi186 ceph-mon[36647]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:23.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:22 smithi114 ceph-mon[32038]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:24.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:24 smithi114 ceph-mon[32038]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:24.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:24 smithi163 ceph-mon[36451]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:24.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:24 smithi186 ceph-mon[36647]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:26.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:26 smithi163 ceph-mon[36451]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:26.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:26 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:26.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:26 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:26.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:26 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:26.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:26 smithi186 ceph-mon[36647]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:26.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:26 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:26.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:26 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:26.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:26 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:27.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:26 smithi114 ceph-mon[32038]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:27.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:26 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:27.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:26 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:27.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:26 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:28.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:28 smithi163 ceph-mon[36451]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:28.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:28.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:28.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:28.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:28.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:28.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:28 smithi186 ceph-mon[36647]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:28.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:28.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:28.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:28.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:28.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:29.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:28 smithi114 ceph-mon[32038]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:29.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:29.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:29.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:29.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:29.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:29.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:29.604+0000 7f963d7fa700 1 -- 172.21.15.114:0/1852399451 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9640063bf0 con 0x7f962c0b27f0 2024-05-27T07:31:29.597 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 3 on host 'smithi114' 2024-05-27T07:31:29.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:29.606+0000 7f9646213700 1 -- 172.21.15.114:0/1852399451 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f962c0b27f0 msgr2=0x7f962c0b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:29.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:29.606+0000 7f9646213700 1 --2- 172.21.15.114:0/1852399451 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f962c0b27f0 0x7f962c0b4d00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9640077f10 tx=0x7f9628006d20 comp rx=0 tx=0).stop 2024-05-27T07:31:29.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:29.606+0000 7f9646213700 1 -- 172.21.15.114:0/1852399451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9640104360 msgr2=0x7f9640078560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:29.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:29.606+0000 7f9646213700 1 --2- 172.21.15.114:0/1852399451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9640104360 0x7f9640078560 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f963000b4d0 tx=0x7f963000b890 comp rx=0 tx=0).stop 2024-05-27T07:31:29.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:29.606+0000 7f9646213700 1 -- 172.21.15.114:0/1852399451 shutdown_connections 2024-05-27T07:31:29.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:29.606+0000 7f9646213700 1 --2- 172.21.15.114:0/1852399451 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9640105e30 0x7f9640077150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:29.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:29.606+0000 7f9646213700 1 --2- 172.21.15.114:0/1852399451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9640104360 0x7f9640078560 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:29.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:29.606+0000 7f9646213700 1 --2- 172.21.15.114:0/1852399451 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f962c0b27f0 0x7f962c0b4d00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:29.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:29.607+0000 7f9646213700 1 --2- 172.21.15.114:0/1852399451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640105490 0x7f9640076c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:29.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:29.607+0000 7f9646213700 1 -- 172.21.15.114:0/1852399451 >> 172.21.15.114:0/1852399451 conn(0x7f96400ff9c0 msgr2=0x7f96401086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:29.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:29.607+0000 7f9646213700 1 -- 172.21.15.114:0/1852399451 shutdown_connections 2024-05-27T07:31:29.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:31:29.607+0000 7f9646213700 1 -- 172.21.15.114:0/1852399451 wait complete. 2024-05-27T07:31:29.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:29 smithi114 ceph-mon[32038]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T07:31:29.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:29 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:29.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:29 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:29.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:29 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:29.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:29 smithi163 ceph-mon[36451]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T07:31:29.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:29 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:29.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:29 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:29.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:29 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:29.970 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:29 smithi186 ceph-mon[36647]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T07:31:29.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:29 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:29.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:29 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:29.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:29 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:30.332 DEBUG:teuthology.orchestra.run.smithi114:osd.3> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.3.service 2024-05-27T07:31:30.334 INFO:tasks.cephadm:Deploying osd.4 on smithi163 with /dev/vg_nvme/lv_3... 2024-05-27T07:31:30.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T07:31:30.577 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:31:30.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:30 smithi163 ceph-mon[36451]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:30.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:30 smithi163 ceph-mon[36451]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T07:31:30.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:30 smithi163 ceph-mon[36451]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T07:31:30.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:30 smithi163 ceph-mon[36451]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-05-27T07:31:30.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:30 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:30.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:30 smithi186 ceph-mon[36647]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:30.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:30 smithi186 ceph-mon[36647]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T07:31:30.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:30 smithi186 ceph-mon[36647]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T07:31:30.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:30 smithi186 ceph-mon[36647]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-05-27T07:31:30.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:30 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:31.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:30 smithi114 ceph-mon[32038]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:31.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:30 smithi114 ceph-mon[32038]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T07:31:31.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:30 smithi114 ceph-mon[32038]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T07:31:31.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:30 smithi114 ceph-mon[32038]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-05-27T07:31:31.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:30 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:31.461 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 07:31:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3[50652]: 2024-05-27T07:31:31.113+0000 7fde5723e700 -1 osd.3 0 waiting for initial osdmap 2024-05-27T07:31:31.461 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 07:31:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3[50652]: 2024-05-27T07:31:31.118+0000 7fde4efeb700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:31:31.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:31 smithi114 ceph-mon[32038]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-05-27T07:31:31.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:31 smithi114 ceph-mon[32038]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T07:31:31.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:31 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:31.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:31 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:31.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:31 smithi114 ceph-mon[32038]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' 2024-05-27T07:31:31.936 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:31 smithi163 ceph-mon[36451]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-05-27T07:31:31.936 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:31 smithi163 ceph-mon[36451]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T07:31:31.936 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:31 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:31.936 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:31 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:31.937 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:31 smithi163 ceph-mon[36451]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' 2024-05-27T07:31:31.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:31 smithi186 ceph-mon[36647]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-05-27T07:31:31.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:31 smithi186 ceph-mon[36647]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T07:31:31.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:31 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:31.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:31 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:31.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:31 smithi186 ceph-mon[36647]: from='osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170]' entity='osd.3' 2024-05-27T07:31:32.509 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T07:31:32.510 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T07:31:32.510 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-05-27T07:31:32.510 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-05-27T07:31:32.510 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0158396 s, 662 MB/s 2024-05-27T07:31:32.510 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-05-27T07:31:32.856 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:32 smithi163 ceph-mon[36451]: purged_snaps scrub starts 2024-05-27T07:31:32.856 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:32 smithi163 ceph-mon[36451]: purged_snaps scrub ok 2024-05-27T07:31:32.856 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:32 smithi163 ceph-mon[36451]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:32.856 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:32 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:32.856 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:32 smithi163 ceph-mon[36451]: osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170] boot 2024-05-27T07:31:32.856 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:32 smithi163 ceph-mon[36451]: osdmap e25: 4 total, 4 up, 4 in 2024-05-27T07:31:32.857 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:32 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:32.900 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:32 smithi114 ceph-mon[32038]: purged_snaps scrub starts 2024-05-27T07:31:32.900 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:32 smithi114 ceph-mon[32038]: purged_snaps scrub ok 2024-05-27T07:31:32.900 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:32 smithi114 ceph-mon[32038]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:32.900 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:32 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:32.901 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:32 smithi114 ceph-mon[32038]: osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170] boot 2024-05-27T07:31:32.901 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:32 smithi114 ceph-mon[32038]: osdmap e25: 4 total, 4 up, 4 in 2024-05-27T07:31:32.901 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:32 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:32.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:32 smithi186 ceph-mon[36647]: purged_snaps scrub starts 2024-05-27T07:31:32.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:32 smithi186 ceph-mon[36647]: purged_snaps scrub ok 2024-05-27T07:31:32.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:32 smithi186 ceph-mon[36647]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:31:32.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:32 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:32.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:32 smithi186 ceph-mon[36647]: osd.3 [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170] boot 2024-05-27T07:31:32.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:32 smithi186 ceph-mon[36647]: osdmap e25: 4 total, 4 up, 4 in 2024-05-27T07:31:32.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:32 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:31:33.191 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_3 2024-05-27T07:31:33.429 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:31:33.883 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:33 smithi163 ceph-mon[36451]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T07:31:33.883 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:33 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:33.883 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:33 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:33.883 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:33 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:33.884 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:33 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:33.884 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:33 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:33.884 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:33 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:33.884 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:33 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:33.884 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:33 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:33.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:33 smithi186 ceph-mon[36647]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T07:31:33.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:33 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:33.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:33 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:33.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:33 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:33.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:33 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:33.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:33 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:33.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:33 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:33.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:33 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:33.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:33 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:34.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:33 smithi114 ceph-mon[32038]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T07:31:34.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:33 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:34.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:33 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:34.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:33 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:34.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:33 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:34.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:33 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:34.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:33 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:34.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:33 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:34.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:33 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:34.816 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:34 smithi163 ceph-mon[36451]: Detected new or changed devices on smithi114 2024-05-27T07:31:34.816 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:34 smithi163 ceph-mon[36451]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:34.816 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:34 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi114 to 8544M 2024-05-27T07:31:34.816 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:34 smithi163 ceph-mon[36451]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T07:31:34.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:34 smithi186 ceph-mon[36647]: Detected new or changed devices on smithi114 2024-05-27T07:31:34.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:34 smithi186 ceph-mon[36647]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:34.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:34 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi114 to 8544M 2024-05-27T07:31:34.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:34 smithi186 ceph-mon[36647]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T07:31:35.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.035+0000 7ff9d4824700 1 -- 172.21.15.163:0/2793337581 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d0105490 msgr2=0x7ff9d01058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:35.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.035+0000 7ff9d4824700 1 --2- 172.21.15.163:0/2793337581 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d0105490 0x7ff9d01058a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009990 tx=0x7ff9b8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:31:35.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.035+0000 7ff9d4824700 1 -- 172.21.15.163:0/2793337581 shutdown_connections 2024-05-27T07:31:35.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.035+0000 7ff9d4824700 1 --2- 172.21.15.163:0/2793337581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d0105e30 0x7ff9d010c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:35.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.035+0000 7ff9d4824700 1 --2- 172.21.15.163:0/2793337581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d0104360 0x7ff9d0104770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:35.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.035+0000 7ff9d4824700 1 --2- 172.21.15.163:0/2793337581 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d0105490 0x7ff9d01058a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:35.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.035+0000 7ff9d4824700 1 -- 172.21.15.163:0/2793337581 >> 172.21.15.163:0/2793337581 conn(0x7ff9d00ff9c0 msgr2=0x7ff9d0101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:35.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.035+0000 7ff9d4824700 1 -- 172.21.15.163:0/2793337581 shutdown_connections 2024-05-27T07:31:35.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.035+0000 7ff9d4824700 1 -- 172.21.15.163:0/2793337581 wait complete. 2024-05-27T07:31:35.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.036+0000 7ff9d4824700 1 Processor -- start 2024-05-27T07:31:35.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.036+0000 7ff9d4824700 1 -- start start 2024-05-27T07:31:35.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.036+0000 7ff9d4824700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d0104360 0x7ff9d019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:35.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.036+0000 7ff9d4824700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d0105490 0x7ff9d019ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:35.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.036+0000 7ff9d4824700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d0105e30 0x7ff9d01a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:35.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.036+0000 7ff9d4824700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d010fe40 con 0x7ff9d0105490 2024-05-27T07:31:35.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.036+0000 7ff9d4824700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d010fcc0 con 0x7ff9d0105e30 2024-05-27T07:31:35.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.036+0000 7ff9d4824700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d010ffc0 con 0x7ff9d0104360 2024-05-27T07:31:35.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.036+0000 7ff9ced9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d0105e30 0x7ff9d01a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:35.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.036+0000 7ff9ced9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d0105e30 0x7ff9d01a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:52528/0 (socket says 172.21.15.163:52528) 2024-05-27T07:31:35.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.036+0000 7ff9ced9d700 1 -- 172.21.15.163:0/3439758722 learned_addr learned my addr 172.21.15.163:0/3439758722 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-27T07:31:35.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.037+0000 7ff9c5d9b700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d0105490 0x7ff9d019ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:35.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.037+0000 7ff9ce59c700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d0104360 0x7ff9d019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:35.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.037+0000 7ff9ced9d700 1 -- 172.21.15.163:0/3439758722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d0104360 msgr2=0x7ff9d019a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:35.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.037+0000 7ff9ced9d700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d0104360 0x7ff9d019a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:35.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.037+0000 7ff9ced9d700 1 -- 172.21.15.163:0/3439758722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d0105490 msgr2=0x7ff9d019ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:35.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.037+0000 7ff9ced9d700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d0105490 0x7ff9d019ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:35.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.037+0000 7ff9ced9d700 1 -- 172.21.15.163:0/3439758722 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d01a24b0 con 0x7ff9d0105e30 2024-05-27T07:31:35.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.037+0000 7ff9ced9d700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d0105e30 0x7ff9d01a1e30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800b970 tx=0x7ff9c800bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:35.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.037+0000 7ff9c7fff700 1 -- 172.21.15.163:0/3439758722 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8019070 con 0x7ff9d0105e30 2024-05-27T07:31:35.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.037+0000 7ff9c7fff700 1 -- 172.21.15.163:0/3439758722 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9c8014440 con 0x7ff9d0105e30 2024-05-27T07:31:35.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.037+0000 7ff9d4824700 1 -- 172.21.15.163:0/3439758722 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d01a27a0 con 0x7ff9d0105e30 2024-05-27T07:31:35.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.037+0000 7ff9c7fff700 1 -- 172.21.15.163:0/3439758722 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c800ce60 con 0x7ff9d0105e30 2024-05-27T07:31:35.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.037+0000 7ff9d4824700 1 -- 172.21.15.163:0/3439758722 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d01a2d60 con 0x7ff9d0105e30 2024-05-27T07:31:35.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.038+0000 7ff9d4824700 1 -- 172.21.15.163:0/3439758722 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9d00512b0 con 0x7ff9d0105e30 2024-05-27T07:31:35.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.041+0000 7ff9c7fff700 1 -- 172.21.15.163:0/3439758722 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7ff9c801c070 con 0x7ff9d0105e30 2024-05-27T07:31:35.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.041+0000 7ff9c7fff700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff9bc0b2900 0x7ff9bc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:35.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.041+0000 7ff9c7fff700 1 -- 172.21.15.163:0/3439758722 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3669+0+0 (secure 0 0 0) 0x7ff9c80e0860 con 0x7ff9d0105e30 2024-05-27T07:31:35.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.041+0000 7ff9ce59c700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff9bc0b2900 0x7ff9bc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:35.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.042+0000 7ff9ce59c700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff9bc0b2900 0x7ff9bc0b4e10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0005950 tx=0x7ff9c00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:35.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.045+0000 7ff9c7fff700 1 -- 172.21.15.163:0/3439758722 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff9c80a97a0 con 0x7ff9d0105e30 2024-05-27T07:31:35.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:34 smithi114 ceph-mon[32038]: Detected new or changed devices on smithi114 2024-05-27T07:31:35.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:34 smithi114 ceph-mon[32038]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:35.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:34 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi114 to 8544M 2024-05-27T07:31:35.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:34 smithi114 ceph-mon[32038]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T07:31:35.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:35.256+0000 7ff9d4824700 1 -- 172.21.15.163:0/3439758722 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff9d0063bf0 con 0x7ff9bc0b2900 2024-05-27T07:31:35.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:35 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:31:35.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:35 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:31:35.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:35 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:35.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:35 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:31:35.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:35 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:31:35.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:35 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:36.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:35 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:31:36.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:35 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:31:36.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:35 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:36.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:36 smithi163 ceph-mon[36451]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:31:36.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:36 smithi163 ceph-mon[36451]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-05-27T07:31:36.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:36 smithi186 ceph-mon[36647]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:31:36.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:36 smithi186 ceph-mon[36647]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-05-27T07:31:37.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:36 smithi114 ceph-mon[32038]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:31:37.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:36 smithi114 ceph-mon[32038]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-05-27T07:31:38.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:38 smithi163 ceph-mon[36451]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:31:38.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:38 smithi163 ceph-mon[36451]: from='client.? 172.21.15.163:0/1495363259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f013c785-92e2-493f-884a-3c71bb47c24c"}]: dispatch 2024-05-27T07:31:38.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:38 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f013c785-92e2-493f-884a-3c71bb47c24c"}]: dispatch 2024-05-27T07:31:38.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:38 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f013c785-92e2-493f-884a-3c71bb47c24c"}]': finished 2024-05-27T07:31:38.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:38 smithi163 ceph-mon[36451]: osdmap e28: 5 total, 4 up, 5 in 2024-05-27T07:31:38.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:38 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:38.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:38 smithi186 ceph-mon[36647]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:31:38.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:38 smithi186 ceph-mon[36647]: from='client.? 172.21.15.163:0/1495363259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f013c785-92e2-493f-884a-3c71bb47c24c"}]: dispatch 2024-05-27T07:31:38.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:38 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f013c785-92e2-493f-884a-3c71bb47c24c"}]: dispatch 2024-05-27T07:31:38.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:38 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f013c785-92e2-493f-884a-3c71bb47c24c"}]': finished 2024-05-27T07:31:38.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:38 smithi186 ceph-mon[36647]: osdmap e28: 5 total, 4 up, 5 in 2024-05-27T07:31:38.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:38 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:39.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:38 smithi114 ceph-mon[32038]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:31:39.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:38 smithi114 ceph-mon[32038]: from='client.? 172.21.15.163:0/1495363259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f013c785-92e2-493f-884a-3c71bb47c24c"}]: dispatch 2024-05-27T07:31:39.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:38 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f013c785-92e2-493f-884a-3c71bb47c24c"}]: dispatch 2024-05-27T07:31:39.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:38 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f013c785-92e2-493f-884a-3c71bb47c24c"}]': finished 2024-05-27T07:31:39.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:38 smithi114 ceph-mon[32038]: osdmap e28: 5 total, 4 up, 5 in 2024-05-27T07:31:39.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:38 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:39.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:39 smithi163 ceph-mon[36451]: from='client.? 172.21.15.163:0/2398253176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:31:39.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:39 smithi186 ceph-mon[36647]: from='client.? 172.21.15.163:0/2398253176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:31:40.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:39 smithi114 ceph-mon[32038]: from='client.? 172.21.15.163:0/2398253176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:31:40.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:40 smithi163 ceph-mon[36451]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-27T07:31:40.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:40 smithi186 ceph-mon[36647]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-27T07:31:41.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:40 smithi114 ceph-mon[32038]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-27T07:31:42.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:42 smithi163 ceph-mon[36451]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:31:42.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:42 smithi186 ceph-mon[36647]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:31:43.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:42 smithi114 ceph-mon[32038]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:31:44.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:44 smithi163 ceph-mon[36451]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-27T07:31:44.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:44 smithi186 ceph-mon[36647]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-27T07:31:45.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:44 smithi114 ceph-mon[32038]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-27T07:31:45.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T07:31:45.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:45.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T07:31:45.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:46.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T07:31:46.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:46.968 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:46 smithi163 ceph-mon[36451]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:46.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:46 smithi163 ceph-mon[36451]: Deploying daemon osd.4 on smithi163 2024-05-27T07:31:46.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:46 smithi186 ceph-mon[36647]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:46.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:46 smithi186 ceph-mon[36647]: Deploying daemon osd.4 on smithi163 2024-05-27T07:31:47.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:46 smithi114 ceph-mon[32038]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:47.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:46 smithi114 ceph-mon[32038]: Deploying daemon osd.4 on smithi163 2024-05-27T07:31:48.899 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:48 smithi163 ceph-mon[36451]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:48.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:48 smithi186 ceph-mon[36647]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:49.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:48 smithi114 ceph-mon[32038]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:50.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:50 smithi186 ceph-mon[36647]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:51.013 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:50 smithi163 ceph-mon[36451]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:51.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:50 smithi114 ceph-mon[32038]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:52.468 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:52 smithi163 ceph-mon[36451]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:52.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:52.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:52.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:52.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:52 smithi186 ceph-mon[36647]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:52.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:52.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:52.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:52.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:52 smithi114 ceph-mon[32038]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:52.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:52.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:52.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:54.692 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:54 smithi163 ceph-mon[36451]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:54.692 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:54 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:54.692 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:54 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:54.692 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:54 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:54.692 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:54 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:54.693 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:54 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:54.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:54 smithi186 ceph-mon[36647]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:54.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:54 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:54.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:54 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:54.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:54 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:54.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:54 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:54.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:54 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:54.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:54 smithi114 ceph-mon[32038]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:54.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:54 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:54.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:54 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:54.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:54 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:54.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:54 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:54.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:54 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:55.267 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 4 on host 'smithi163' 2024-05-27T07:31:55.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:55.274+0000 7ff9c7fff700 1 -- 172.21.15.163:0/3439758722 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff9d0063bf0 con 0x7ff9bc0b2900 2024-05-27T07:31:55.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:55.276+0000 7ff9d4824700 1 -- 172.21.15.163:0/3439758722 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff9bc0b2900 msgr2=0x7ff9bc0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:55.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:55.276+0000 7ff9d4824700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff9bc0b2900 0x7ff9bc0b4e10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0005950 tx=0x7ff9c00058e0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:55.277+0000 7ff9d4824700 1 -- 172.21.15.163:0/3439758722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d0105e30 msgr2=0x7ff9d01a1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:55.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:55.277+0000 7ff9d4824700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d0105e30 0x7ff9d01a1e30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800b970 tx=0x7ff9c800bd30 comp rx=0 tx=0).stop 2024-05-27T07:31:55.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:55.277+0000 7ff9d4824700 1 -- 172.21.15.163:0/3439758722 shutdown_connections 2024-05-27T07:31:55.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:55.277+0000 7ff9d4824700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d0105e30 0x7ff9d01a1e30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:55.277+0000 7ff9d4824700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff9bc0b2900 0x7ff9bc0b4e10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:55.277+0000 7ff9d4824700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d0104360 0x7ff9d019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:55.277+0000 7ff9d4824700 1 --2- 172.21.15.163:0/3439758722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d0105490 0x7ff9d019ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:55.277+0000 7ff9d4824700 1 -- 172.21.15.163:0/3439758722 >> 172.21.15.163:0/3439758722 conn(0x7ff9d00ff9c0 msgr2=0x7ff9d01086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:55.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:55.277+0000 7ff9d4824700 1 -- 172.21.15.163:0/3439758722 shutdown_connections 2024-05-27T07:31:55.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:31:55.277+0000 7ff9d4824700 1 -- 172.21.15.163:0/3439758722 wait complete. 2024-05-27T07:31:56.114 DEBUG:teuthology.orchestra.run.smithi163:osd.4> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.4.service 2024-05-27T07:31:56.116 INFO:tasks.cephadm:Deploying osd.5 on smithi186 with /dev/vg_nvme/lv_3... 2024-05-27T07:31:56.117 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T07:31:56.346 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:56 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:56.346 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:56 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:56.346 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:56 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:56.346 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:56 smithi163 ceph-mon[36451]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:56.346 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:56 smithi163 ceph-mon[36451]: from='osd.4 [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:31:56.346 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:56 smithi163 ceph-mon[36451]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:31:56.356 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:31:56.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:56 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:56.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:56 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:56.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:56 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:56.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:56 smithi186 ceph-mon[36647]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:56.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:56 smithi186 ceph-mon[36647]: from='osd.4 [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:31:56.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:56 smithi186 ceph-mon[36647]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:31:56.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:56.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:56.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:56 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:56.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:56 smithi114 ceph-mon[32038]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:56.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:56 smithi114 ceph-mon[32038]: from='osd.4 [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:31:56.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:56 smithi114 ceph-mon[32038]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:31:57.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:57 smithi114 ceph-mon[32038]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T07:31:57.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:57 smithi114 ceph-mon[32038]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T07:31:57.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:57.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:57 smithi114 ceph-mon[32038]: from='osd.4 [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:31:57.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:57 smithi114 ceph-mon[32038]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:31:57.577 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:57 smithi186 ceph-mon[36647]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T07:31:57.577 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:57 smithi186 ceph-mon[36647]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T07:31:57.577 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:57.578 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:57 smithi186 ceph-mon[36647]: from='osd.4 [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:31:57.578 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:57 smithi186 ceph-mon[36647]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:31:57.589 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:57 smithi163 ceph-mon[36451]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T07:31:57.589 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:57 smithi163 ceph-mon[36451]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T07:31:57.589 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:57.589 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:57 smithi163 ceph-mon[36451]: from='osd.4 [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:31:57.589 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:57 smithi163 ceph-mon[36451]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:31:57.590 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 07:31:57 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4[47519]: 2024-05-27T07:31:57.271+0000 7fc771bd1700 -1 osd.4 0 waiting for initial osdmap 2024-05-27T07:31:57.590 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 07:31:57 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4[47519]: 2024-05-27T07:31:57.278+0000 7fc76d986700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:31:58.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:58 smithi186 ceph-mon[36647]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-27T07:31:58.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:58 smithi186 ceph-mon[36647]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T07:31:58.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:58 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:58.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:58 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:58.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:58 smithi186 ceph-mon[36647]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:58.540 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T07:31:58.540 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T07:31:58.540 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-05-27T07:31:58.540 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-05-27T07:31:58.540 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128082 s, 819 MB/s 2024-05-27T07:31:58.540 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-05-27T07:31:58.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:58 smithi114 ceph-mon[32038]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-27T07:31:58.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:58 smithi114 ceph-mon[32038]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T07:31:58.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:58 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:58.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:58 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:58.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:58 smithi114 ceph-mon[32038]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:58.708 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:58 smithi163 ceph-mon[36451]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-27T07:31:58.708 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:58 smithi163 ceph-mon[36451]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T07:31:58.709 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:58 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:58.709 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:58 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:58.709 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:58 smithi163 ceph-mon[36451]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:31:59.226 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_3 2024-05-27T07:31:59.467 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:31:59.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: purged_snaps scrub starts 2024-05-27T07:31:59.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: purged_snaps scrub ok 2024-05-27T07:31:59.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: osd.4 [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927] boot 2024-05-27T07:31:59.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: osdmap e31: 5 total, 5 up, 5 in 2024-05-27T07:31:59.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:59.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: Detected new or changed devices on smithi163 2024-05-27T07:31:59.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:59.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:59.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:59.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:59.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi163 to 8544M 2024-05-27T07:31:59.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:59.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:59.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:59.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:31:59 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:59.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: purged_snaps scrub starts 2024-05-27T07:31:59.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: purged_snaps scrub ok 2024-05-27T07:31:59.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: osd.4 [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927] boot 2024-05-27T07:31:59.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: osdmap e31: 5 total, 5 up, 5 in 2024-05-27T07:31:59.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:59.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: Detected new or changed devices on smithi163 2024-05-27T07:31:59.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:59.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:59.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:59.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:59.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi163 to 8544M 2024-05-27T07:31:59.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:59.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:59.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:59.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:31:59 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:59.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: purged_snaps scrub starts 2024-05-27T07:31:59.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: purged_snaps scrub ok 2024-05-27T07:31:59.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: osd.4 [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927] boot 2024-05-27T07:31:59.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: osdmap e31: 5 total, 5 up, 5 in 2024-05-27T07:31:59.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:31:59.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: Detected new or changed devices on smithi163 2024-05-27T07:31:59.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:59.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:59.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:59.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:31:59.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi163 to 8544M 2024-05-27T07:31:59.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:31:59.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:59.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:59.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:31:59 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:00.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:00 smithi114 ceph-mon[32038]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T07:32:00.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:00 smithi114 ceph-mon[32038]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:00.553 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:00 smithi186 ceph-mon[36647]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T07:32:00.553 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:00 smithi186 ceph-mon[36647]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:00.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:00 smithi163 ceph-mon[36451]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T07:32:00.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:00 smithi163 ceph-mon[36451]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:01.131 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.134+0000 7f484dbd0700 1 -- 172.21.15.186:0/2032025083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48481029d0 msgr2=0x7f4848102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:01.131 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.134+0000 7f484dbd0700 1 --2- 172.21.15.186:0/2032025083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48481029d0 0x7f4848102de0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4838009990 tx=0x7f4838009ca0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.131 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.134+0000 7f484dbd0700 1 -- 172.21.15.186:0/2032025083 shutdown_connections 2024-05-27T07:32:01.131 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.134+0000 7f484dbd0700 1 --2- 172.21.15.186:0/2032025083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48481044a0 0x7f484810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.132 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.134+0000 7f484dbd0700 1 --2- 172.21.15.186:0/2032025083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48481029d0 0x7f4848102de0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.132 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.134+0000 7f484dbd0700 1 --2- 172.21.15.186:0/2032025083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4848103b00 0x7f4848103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.132 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.134+0000 7f484dbd0700 1 -- 172.21.15.186:0/2032025083 >> 172.21.15.186:0/2032025083 conn(0x7f48480fe010 msgr2=0x7f48481004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:01.132 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.134+0000 7f484dbd0700 1 -- 172.21.15.186:0/2032025083 shutdown_connections 2024-05-27T07:32:01.132 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.134+0000 7f484dbd0700 1 -- 172.21.15.186:0/2032025083 wait complete. 2024-05-27T07:32:01.132 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.135+0000 7f484dbd0700 1 Processor -- start 2024-05-27T07:32:01.133 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.135+0000 7f484dbd0700 1 -- start start 2024-05-27T07:32:01.133 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.135+0000 7f484dbd0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48481029d0 0x7f484819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.133 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.135+0000 7f484dbd0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4848103b00 0x7f484819aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.133 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.135+0000 7f484dbd0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48481044a0 0x7f48481a1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.133 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.135+0000 7f484dbd0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f484810ec90 con 0x7f48481044a0 2024-05-27T07:32:01.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.135+0000 7f484dbd0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f484810eb10 con 0x7f48481029d0 2024-05-27T07:32:01.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.135+0000 7f484dbd0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f484810ee10 con 0x7f4848103b00 2024-05-27T07:32:01.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.135+0000 7f483ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4848103b00 0x7f484819aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.135+0000 7f483ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4848103b00 0x7f484819aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:33656/0 (socket says 172.21.15.186:33656) 2024-05-27T07:32:01.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.135+0000 7f483ffff700 1 -- 172.21.15.186:0/2608171089 learned_addr learned my addr 172.21.15.186:0/2608171089 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-27T07:32:01.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f4847fff700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48481044a0 0x7f48481a1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.135 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f48477fe700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48481029d0 0x7f484819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.135 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f483ffff700 1 -- 172.21.15.186:0/2608171089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48481029d0 msgr2=0x7f484819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:01.135 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f483ffff700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48481029d0 0x7f484819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.135 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f483ffff700 1 -- 172.21.15.186:0/2608171089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48481044a0 msgr2=0x7f48481a1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:01.135 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f483ffff700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48481044a0 0x7f48481a1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f483ffff700 1 -- 172.21.15.186:0/2608171089 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48481a2510 con 0x7f4848103b00 2024-05-27T07:32:01.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f4847fff700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48481044a0 0x7f48481a1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:01.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f483ffff700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4848103b00 0x7f484819aef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f483000d950 tx=0x7f483000dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:01.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f48457fa700 1 -- 172.21.15.186:0/2608171089 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830016070 con 0x7f4848103b00 2024-05-27T07:32:01.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f484dbd0700 1 -- 172.21.15.186:0/2608171089 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48481a2800 con 0x7f4848103b00 2024-05-27T07:32:01.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f48457fa700 1 -- 172.21.15.186:0/2608171089 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4830010460 con 0x7f4848103b00 2024-05-27T07:32:01.137 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f48457fa700 1 -- 172.21.15.186:0/2608171089 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483000f620 con 0x7f4848103b00 2024-05-27T07:32:01.137 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.136+0000 7f484dbd0700 1 -- 172.21.15.186:0/2608171089 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48481a2d90 con 0x7f4848103b00 2024-05-27T07:32:01.137 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.137+0000 7f484dbd0700 1 -- 172.21.15.186:0/2608171089 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48480512b0 con 0x7f4848103b00 2024-05-27T07:32:01.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.139+0000 7f48457fa700 1 -- 172.21.15.186:0/2608171089 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f4830019070 con 0x7f4848103b00 2024-05-27T07:32:01.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.140+0000 7f48457fa700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f48340b2880 0x7f48340b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.140+0000 7f48457fa700 1 -- 172.21.15.186:0/2608171089 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4101+0+0 (secure 0 0 0) 0x7f48300e11c0 con 0x7f4848103b00 2024-05-27T07:32:01.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.140+0000 7f48477fe700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f48340b2880 0x7f48340b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.140+0000 7f48477fe700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f48340b2880 0x7f48340b4d90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f48380038a0 tx=0x7f483801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:01.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.144+0000 7f48457fa700 1 -- 172.21.15.186:0/2608171089 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f48300a9f30 con 0x7f4848103b00 2024-05-27T07:32:01.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:01.353+0000 7f484dbd0700 1 -- 172.21.15.186:0/2608171089 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f48481000e0 con 0x7f48340b2880 2024-05-27T07:32:01.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:01 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:32:01.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:01 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:32:01.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:01 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:01.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:01 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:32:01.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:01 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:32:01.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:01 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:01.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:01 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:32:01.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:01 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:32:01.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:01 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:02.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:02 smithi163 ceph-mon[36451]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:02.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:02 smithi163 ceph-mon[36451]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:32:02.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:02 smithi186 ceph-mon[36647]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:02.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:02 smithi186 ceph-mon[36647]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:32:02.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:02 smithi114 ceph-mon[32038]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:02.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:02 smithi114 ceph-mon[32038]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:32:04.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:04 smithi163 ceph-mon[36451]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:04.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:04 smithi163 ceph-mon[36451]: from='client.? 172.21.15.186:0/3951875257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf"}]: dispatch 2024-05-27T07:32:04.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:04 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf"}]: dispatch 2024-05-27T07:32:04.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:04 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf"}]': finished 2024-05-27T07:32:04.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:04 smithi163 ceph-mon[36451]: osdmap e33: 6 total, 5 up, 6 in 2024-05-27T07:32:04.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:04.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:04 smithi186 ceph-mon[36647]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:04.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:04 smithi186 ceph-mon[36647]: from='client.? 172.21.15.186:0/3951875257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf"}]: dispatch 2024-05-27T07:32:04.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:04 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf"}]: dispatch 2024-05-27T07:32:04.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:04 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf"}]': finished 2024-05-27T07:32:04.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:04 smithi186 ceph-mon[36647]: osdmap e33: 6 total, 5 up, 6 in 2024-05-27T07:32:04.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:04.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:04 smithi114 ceph-mon[32038]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:04.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:04 smithi114 ceph-mon[32038]: from='client.? 172.21.15.186:0/3951875257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf"}]: dispatch 2024-05-27T07:32:04.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:04 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf"}]: dispatch 2024-05-27T07:32:04.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:04 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf"}]': finished 2024-05-27T07:32:04.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:04 smithi114 ceph-mon[32038]: osdmap e33: 6 total, 5 up, 6 in 2024-05-27T07:32:04.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:05.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:05 smithi163 ceph-mon[36451]: from='client.? 172.21.15.186:0/390089878' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:32:05.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:05 smithi186 ceph-mon[36647]: from='client.? 172.21.15.186:0/390089878' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:32:05.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:05 smithi114 ceph-mon[32038]: from='client.? 172.21.15.186:0/390089878' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:32:06.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:06 smithi163 ceph-mon[36451]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:06.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:06 smithi186 ceph-mon[36647]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:06.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:06 smithi114 ceph-mon[32038]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:08.639 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:08 smithi186 ceph-mon[36647]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:08.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:08 smithi163 ceph-mon[36451]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:08.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:08 smithi114 ceph-mon[32038]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:10.498 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:10 smithi186 ceph-mon[36647]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:10.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:10 smithi163 ceph-mon[36451]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:10.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:10 smithi114 ceph-mon[32038]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:12.581 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:12 smithi186 ceph-mon[36647]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:12.581 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:12 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T07:32:12.581 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:12 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:12.581 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:12 smithi186 ceph-mon[36647]: Deploying daemon osd.5 on smithi186 2024-05-27T07:32:12.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:12 smithi163 ceph-mon[36451]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:12.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:12 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T07:32:12.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:12 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:12.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:12 smithi163 ceph-mon[36451]: Deploying daemon osd.5 on smithi186 2024-05-27T07:32:12.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:12 smithi114 ceph-mon[32038]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:12.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:12 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T07:32:12.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:12 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:12.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:12 smithi114 ceph-mon[32038]: Deploying daemon osd.5 on smithi186 2024-05-27T07:32:14.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:14 smithi163 ceph-mon[36451]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:14.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:14 smithi186 ceph-mon[36647]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:14.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:14 smithi114 ceph-mon[32038]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:16.658 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:16 smithi186 ceph-mon[36647]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:16.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:16 smithi163 ceph-mon[36451]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:16.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:16 smithi114 ceph-mon[32038]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:18.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:18 smithi163 ceph-mon[36451]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:18.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:18.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:18.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:18.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:18 smithi186 ceph-mon[36647]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:18.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:18.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:18.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:19.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:18 smithi114 ceph-mon[32038]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:19.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:19.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:19.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:20.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:20 smithi163 ceph-mon[36451]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:20.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:20.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:20.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:20.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:20.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:20.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:20 smithi186 ceph-mon[36647]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:20.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:20.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:20.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:20.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:20.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:21.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:20 smithi114 ceph-mon[32038]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:21.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:21.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:21.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:21.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:21.050 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:21.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:21.523+0000 7f48457fa700 1 -- 172.21.15.186:0/2608171089 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f48481000e0 con 0x7f48340b2880 2024-05-27T07:32:21.516 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 5 on host 'smithi186' 2024-05-27T07:32:21.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:21.525+0000 7f484dbd0700 1 -- 172.21.15.186:0/2608171089 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f48340b2880 msgr2=0x7f48340b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:21.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:21.525+0000 7f484dbd0700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f48340b2880 0x7f48340b4d90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f48380038a0 tx=0x7f483801a040 comp rx=0 tx=0).stop 2024-05-27T07:32:21.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:21.525+0000 7f484dbd0700 1 -- 172.21.15.186:0/2608171089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4848103b00 msgr2=0x7f484819aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:21.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:21.525+0000 7f484dbd0700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4848103b00 0x7f484819aef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f483000d950 tx=0x7f483000dd10 comp rx=0 tx=0).stop 2024-05-27T07:32:21.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:21.525+0000 7f484dbd0700 1 -- 172.21.15.186:0/2608171089 shutdown_connections 2024-05-27T07:32:21.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:21.525+0000 7f484dbd0700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48481044a0 0x7f48481a1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:21.525+0000 7f484dbd0700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48481029d0 0x7f484819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:21.525+0000 7f484dbd0700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f48340b2880 0x7f48340b4d90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:21.525+0000 7f484dbd0700 1 --2- 172.21.15.186:0/2608171089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4848103b00 0x7f484819aef0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:21.525+0000 7f484dbd0700 1 -- 172.21.15.186:0/2608171089 >> 172.21.15.186:0/2608171089 conn(0x7f48480fe010 msgr2=0x7f48480ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:21.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:21.525+0000 7f484dbd0700 1 -- 172.21.15.186:0/2608171089 shutdown_connections 2024-05-27T07:32:21.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:32:21.525+0000 7f484dbd0700 1 -- 172.21.15.186:0/2608171089 wait complete. 2024-05-27T07:32:22.145 DEBUG:teuthology.orchestra.run.smithi186:osd.5> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.5.service 2024-05-27T07:32:22.147 INFO:tasks.cephadm:Deploying osd.6 on smithi114 with /dev/vg_nvme/lv_2... 2024-05-27T07:32:22.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T07:32:22.400 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:22.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:22 smithi163 ceph-mon[36451]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:22.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:22 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:22.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:22 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:22.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:22 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:22.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:22 smithi163 ceph-mon[36451]: from='osd.5 [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:32:22.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:22 smithi163 ceph-mon[36451]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:32:22.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:22 smithi186 ceph-mon[36647]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:22.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:22 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:22.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:22 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:22.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:22 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:22.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:22 smithi186 ceph-mon[36647]: from='osd.5 [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:32:22.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:22 smithi186 ceph-mon[36647]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:32:22.787 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:22 smithi114 ceph-mon[32038]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:22.787 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:22 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:22.787 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:22 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:22.787 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:22 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:22.787 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:22 smithi114 ceph-mon[32038]: from='osd.5 [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:32:22.787 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:22 smithi114 ceph-mon[32038]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:32:23.574 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:23 smithi186 ceph-mon[36647]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T07:32:23.574 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:23 smithi186 ceph-mon[36647]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T07:32:23.574 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:23 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:23.574 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:23 smithi186 ceph-mon[36647]: from='osd.5 [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:32:23.574 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:23 smithi186 ceph-mon[36647]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:32:23.575 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 07:32:23 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T07:32:23.494+0000 7f1e8ed17700 -1 osd.5 0 waiting for initial osdmap 2024-05-27T07:32:23.575 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 07:32:23 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T07:32:23.501+0000 7f1e8aacc700 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:32:23.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:23 smithi114 ceph-mon[32038]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T07:32:23.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:23 smithi114 ceph-mon[32038]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T07:32:23.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:23 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:23.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:23 smithi114 ceph-mon[32038]: from='osd.5 [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:32:23.659 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:23 smithi114 ceph-mon[32038]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:32:23.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:23 smithi163 ceph-mon[36451]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T07:32:23.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:23 smithi163 ceph-mon[36451]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T07:32:23.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:23 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:23.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:23 smithi163 ceph-mon[36451]: from='osd.5 [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:32:23.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:23 smithi163 ceph-mon[36451]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:32:24.576 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:24 smithi114 ceph-mon[32038]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:24.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:24 smithi114 ceph-mon[32038]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-27T07:32:24.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:24 smithi114 ceph-mon[32038]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T07:32:24.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:24 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:24.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:24 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:24.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:24 smithi114 ceph-mon[32038]: osd.5 [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241] boot 2024-05-27T07:32:24.578 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:24 smithi114 ceph-mon[32038]: osdmap e36: 6 total, 6 up, 6 in 2024-05-27T07:32:24.578 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:24 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:24.606 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T07:32:24.606 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T07:32:24.606 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-05-27T07:32:24.606 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-05-27T07:32:24.606 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117514 s, 892 MB/s 2024-05-27T07:32:24.607 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-05-27T07:32:24.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:24 smithi186 ceph-mon[36647]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:24.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:24 smithi186 ceph-mon[36647]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-27T07:32:24.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:24 smithi186 ceph-mon[36647]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T07:32:24.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:24 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:24.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:24 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:24.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:24 smithi186 ceph-mon[36647]: osd.5 [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241] boot 2024-05-27T07:32:24.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:24 smithi186 ceph-mon[36647]: osdmap e36: 6 total, 6 up, 6 in 2024-05-27T07:32:24.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:24 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:24.870 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:24 smithi163 ceph-mon[36451]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:32:24.870 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:24 smithi163 ceph-mon[36451]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-27T07:32:24.870 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:24 smithi163 ceph-mon[36451]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T07:32:24.871 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:24 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:24.871 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:24 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:24.871 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:24 smithi163 ceph-mon[36451]: osd.5 [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241] boot 2024-05-27T07:32:24.871 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:24 smithi163 ceph-mon[36451]: osdmap e36: 6 total, 6 up, 6 in 2024-05-27T07:32:24.871 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:24 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:32:25.284 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_2 2024-05-27T07:32:25.529 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:25.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:25 smithi114 ceph-mon[32038]: purged_snaps scrub starts 2024-05-27T07:32:25.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:25 smithi114 ceph-mon[32038]: purged_snaps scrub ok 2024-05-27T07:32:25.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:25 smithi114 ceph-mon[32038]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T07:32:25.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:25 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:25.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:25 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:25.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:25 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:25.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:25 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:25.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:25 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:25.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:25 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:25.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:25 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:25.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:25 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:25.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:25 smithi163 ceph-mon[36451]: purged_snaps scrub starts 2024-05-27T07:32:25.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:25 smithi163 ceph-mon[36451]: purged_snaps scrub ok 2024-05-27T07:32:25.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:25 smithi163 ceph-mon[36451]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T07:32:25.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:25 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:25.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:25 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:25.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:25 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:25.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:25 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:25.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:25 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:25.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:25 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:25.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:25 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:25.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:25 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:25.971 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:25 smithi186 ceph-mon[36647]: purged_snaps scrub starts 2024-05-27T07:32:25.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:25 smithi186 ceph-mon[36647]: purged_snaps scrub ok 2024-05-27T07:32:25.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:25 smithi186 ceph-mon[36647]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T07:32:25.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:25 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:25.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:25 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:25.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:25 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:25.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:25 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:25.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:25 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:25.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:25 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:25.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:25 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:25.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:25 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:27.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.233+0000 7f09fbc7a700 1 -- 172.21.15.114:0/2880458085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09f41037c0 msgr2=0x7f09f4103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:27.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.233+0000 7f09fbc7a700 1 --2- 172.21.15.114:0/2880458085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09f41037c0 0x7f09f4103bd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009990 tx=0x7f09f0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:32:27.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.233+0000 7f09fbc7a700 1 -- 172.21.15.114:0/2880458085 shutdown_connections 2024-05-27T07:32:27.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.233+0000 7f09fbc7a700 1 --2- 172.21.15.114:0/2880458085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f4104d00 0x7f09f410b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:27.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.233+0000 7f09fbc7a700 1 --2- 172.21.15.114:0/2880458085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09f41037c0 0x7f09f4103bd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:27.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.233+0000 7f09fbc7a700 1 --2- 172.21.15.114:0/2880458085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09f41041a0 0x7f09f4104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:27.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.233+0000 7f09fbc7a700 1 -- 172.21.15.114:0/2880458085 >> 172.21.15.114:0/2880458085 conn(0x7f09f40fe010 msgr2=0x7f09f41004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:27.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.233+0000 7f09fbc7a700 1 -- 172.21.15.114:0/2880458085 shutdown_connections 2024-05-27T07:32:27.227 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:27 smithi114 ceph-mon[32038]: Detected new or changed devices on smithi186 2024-05-27T07:32:27.227 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:27 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi186 to 8544M 2024-05-27T07:32:27.227 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:27 smithi114 ceph-mon[32038]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:27.227 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:27 smithi114 ceph-mon[32038]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T07:32:27.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.233+0000 7f09fbc7a700 1 -- 172.21.15.114:0/2880458085 wait complete. 2024-05-27T07:32:27.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.234+0000 7f09fbc7a700 1 Processor -- start 2024-05-27T07:32:27.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.234+0000 7f09fbc7a700 1 -- start start 2024-05-27T07:32:27.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.234+0000 7f09fbc7a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09f41037c0 0x7f09f419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:27.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.234+0000 7f09fbc7a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09f41041a0 0x7f09f419aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:27.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.234+0000 7f09fbc7a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f4104d00 0x7f09f41a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:27.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.234+0000 7f09fbc7a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f09f410fe30 con 0x7f09f41037c0 2024-05-27T07:32:27.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.234+0000 7f09fbc7a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f09f410fcb0 con 0x7f09f41041a0 2024-05-27T07:32:27.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.234+0000 7f09fbc7a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09f410ffb0 con 0x7f09f4104d00 2024-05-27T07:32:27.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.235+0000 7f09fa217700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f4104d00 0x7f09f41a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:27.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.235+0000 7f09fa217700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f4104d00 0x7f09f41a1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:51012/0 (socket says 172.21.15.114:51012) 2024-05-27T07:32:27.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.235+0000 7f09fa217700 1 -- 172.21.15.114:0/159031614 learned_addr learned my addr 172.21.15.114:0/159031614 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:32:27.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.235+0000 7f09f9a16700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09f41037c0 0x7f09f419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:27.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.235+0000 7f09f9215700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09f41041a0 0x7f09f419aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:27.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.235+0000 7f09fa217700 1 -- 172.21.15.114:0/159031614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09f41041a0 msgr2=0x7f09f419aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:27.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.235+0000 7f09fa217700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09f41041a0 0x7f09f419aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:27.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.235+0000 7f09fa217700 1 -- 172.21.15.114:0/159031614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09f41037c0 msgr2=0x7f09f419a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:27.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.235+0000 7f09fa217700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09f41037c0 0x7f09f419a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:27.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.235+0000 7f09fa217700 1 -- 172.21.15.114:0/159031614 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f41a24d0 con 0x7f09f4104d00 2024-05-27T07:32:27.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.235+0000 7f09f9a16700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09f41037c0 0x7f09f419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:27.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.235+0000 7f09fa217700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f4104d00 0x7f09f41a1e50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00ef30 tx=0x7f09ec00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:27.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.236+0000 7f09eaffd700 1 -- 172.21.15.114:0/159031614 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ec0092e0 con 0x7f09f4104d00 2024-05-27T07:32:27.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.236+0000 7f09fbc7a700 1 -- 172.21.15.114:0/159031614 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f41a2760 con 0x7f09f4104d00 2024-05-27T07:32:27.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.236+0000 7f09eaffd700 1 -- 172.21.15.114:0/159031614 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09ec004bb0 con 0x7f09f4104d00 2024-05-27T07:32:27.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.236+0000 7f09eaffd700 1 -- 172.21.15.114:0/159031614 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ec004280 con 0x7f09f4104d00 2024-05-27T07:32:27.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.236+0000 7f09fbc7a700 1 -- 172.21.15.114:0/159031614 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f41a2cd0 con 0x7f09f4104d00 2024-05-27T07:32:27.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.237+0000 7f09fbc7a700 1 -- 172.21.15.114:0/159031614 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f40512b0 con 0x7f09f4104d00 2024-05-27T07:32:27.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.240+0000 7f09eaffd700 1 -- 172.21.15.114:0/159031614 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f09ec01b030 con 0x7f09f4104d00 2024-05-27T07:32:27.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.241+0000 7f09eaffd700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f09e00b2880 0x7f09e00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:27.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.241+0000 7f09f9a16700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f09e00b2880 0x7f09e00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:27.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.241+0000 7f09eaffd700 1 -- 172.21.15.114:0/159031614 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4533+0+0 (secure 0 0 0) 0x7f09ec0e1940 con 0x7f09f4104d00 2024-05-27T07:32:27.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.241+0000 7f09f9a16700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f09e00b2880 0x7f09e00b4d90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f09f0006010 tx=0x7f09f000b780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:27.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.242+0000 7f09eaffd700 1 -- 172.21.15.114:0/159031614 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f09ec022020 con 0x7f09f4104d00 2024-05-27T07:32:27.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:27.456+0000 7f09fbc7a700 1 -- 172.21.15.114:0/159031614 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f09f4063bf0 con 0x7f09e00b2880 2024-05-27T07:32:27.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:27 smithi163 ceph-mon[36451]: Detected new or changed devices on smithi186 2024-05-27T07:32:27.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:27 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi186 to 8544M 2024-05-27T07:32:27.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:27 smithi163 ceph-mon[36451]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:27.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:27 smithi163 ceph-mon[36451]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T07:32:27.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:27 smithi186 ceph-mon[36647]: Detected new or changed devices on smithi186 2024-05-27T07:32:27.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:27 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi186 to 8544M 2024-05-27T07:32:27.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:27 smithi186 ceph-mon[36647]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:27.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:27 smithi186 ceph-mon[36647]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T07:32:28.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:32:28.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:32:28.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:28.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:32:28.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:32:28.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:28.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:32:28.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:32:28.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:29.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:29 smithi114 ceph-mon[32038]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:29.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:29 smithi114 ceph-mon[32038]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:32:29.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:29 smithi163 ceph-mon[36451]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:29.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:29 smithi163 ceph-mon[36451]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:32:29.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:29 smithi186 ceph-mon[36647]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:29.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:29 smithi186 ceph-mon[36647]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:32:31.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:31 smithi114 ceph-mon[32038]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:31.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:31 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/670254639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21fe0537-2b7a-4588-82fc-a0df295b078c"}]: dispatch 2024-05-27T07:32:31.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:31 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/670254639' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21fe0537-2b7a-4588-82fc-a0df295b078c"}]': finished 2024-05-27T07:32:31.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:31 smithi114 ceph-mon[32038]: osdmap e39: 7 total, 6 up, 7 in 2024-05-27T07:32:31.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:31 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:31.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:31 smithi163 ceph-mon[36451]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:31.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:31 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/670254639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21fe0537-2b7a-4588-82fc-a0df295b078c"}]: dispatch 2024-05-27T07:32:31.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:31 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/670254639' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21fe0537-2b7a-4588-82fc-a0df295b078c"}]': finished 2024-05-27T07:32:31.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:31 smithi163 ceph-mon[36451]: osdmap e39: 7 total, 6 up, 7 in 2024-05-27T07:32:31.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:31 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:31.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:31 smithi186 ceph-mon[36647]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:31.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:31 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/670254639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21fe0537-2b7a-4588-82fc-a0df295b078c"}]: dispatch 2024-05-27T07:32:31.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:31 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/670254639' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21fe0537-2b7a-4588-82fc-a0df295b078c"}]': finished 2024-05-27T07:32:31.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:31 smithi186 ceph-mon[36647]: osdmap e39: 7 total, 6 up, 7 in 2024-05-27T07:32:31.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:31 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:32.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:32 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3970953523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:32:32.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:32 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3970953523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:32:32.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:32 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3970953523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:32:33.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:33 smithi163 ceph-mon[36451]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-05-27T07:32:33.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:33 smithi186 ceph-mon[36647]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-05-27T07:32:33.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:33 smithi114 ceph-mon[32038]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-05-27T07:32:35.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:35 smithi114 ceph-mon[32038]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:32:35.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:35 smithi163 ceph-mon[36451]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:32:35.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:35 smithi186 ceph-mon[36647]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:32:37.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:37 smithi114 ceph-mon[32038]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-27T07:32:37.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:37 smithi163 ceph-mon[36451]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-27T07:32:37.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:37 smithi186 ceph-mon[36647]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-27T07:32:38.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:38 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T07:32:38.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:38 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:38.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:38 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T07:32:38.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:38 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:38.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:38 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T07:32:38.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:38 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:39.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:39 smithi114 ceph-mon[32038]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:32:39.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:39 smithi114 ceph-mon[32038]: Deploying daemon osd.6 on smithi114 2024-05-27T07:32:39.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:39 smithi163 ceph-mon[36451]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:32:39.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:39 smithi163 ceph-mon[36451]: Deploying daemon osd.6 on smithi114 2024-05-27T07:32:39.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:39 smithi186 ceph-mon[36647]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:32:39.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:39 smithi186 ceph-mon[36647]: Deploying daemon osd.6 on smithi114 2024-05-27T07:32:41.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:41 smithi114 ceph-mon[32038]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:32:41.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:41 smithi163 ceph-mon[36451]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:32:41.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:41 smithi186 ceph-mon[36647]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:32:43.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:43 smithi114 ceph-mon[32038]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:43.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:43 smithi163 ceph-mon[36451]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:43.471 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:43 smithi186 ceph-mon[36647]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:45.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:44 smithi163 ceph-mon[36451]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:45.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:45.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:45.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:45.221 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:44 smithi186 ceph-mon[36647]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:45.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:45.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:45.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:45.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:44 smithi114 ceph-mon[32038]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:45.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:45.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:45.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:47.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:46 smithi114 ceph-mon[32038]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:47.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:46 smithi163 ceph-mon[36451]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:47.221 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:46 smithi186 ceph-mon[36647]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:47.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:47.914+0000 7f09eaffd700 1 -- 172.21.15.114:0/159031614 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f09f4063bf0 con 0x7f09e00b2880 2024-05-27T07:32:47.909 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 6 on host 'smithi114' 2024-05-27T07:32:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:47.917+0000 7f09fbc7a700 1 -- 172.21.15.114:0/159031614 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f09e00b2880 msgr2=0x7f09e00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:47.917+0000 7f09fbc7a700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f09e00b2880 0x7f09e00b4d90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f09f0006010 tx=0x7f09f000b780 comp rx=0 tx=0).stop 2024-05-27T07:32:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:47.917+0000 7f09fbc7a700 1 -- 172.21.15.114:0/159031614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f4104d00 msgr2=0x7f09f41a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:47.917+0000 7f09fbc7a700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f4104d00 0x7f09f41a1e50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00ef30 tx=0x7f09ec00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:32:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:47.917+0000 7f09fbc7a700 1 -- 172.21.15.114:0/159031614 shutdown_connections 2024-05-27T07:32:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:47.917+0000 7f09fbc7a700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f09e00b2880 0x7f09e00b4d90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:47.917+0000 7f09fbc7a700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f4104d00 0x7f09f41a1e50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:47.917+0000 7f09fbc7a700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09f41037c0 0x7f09f419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:47.917+0000 7f09fbc7a700 1 --2- 172.21.15.114:0/159031614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09f41041a0 0x7f09f419aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:47.917+0000 7f09fbc7a700 1 -- 172.21.15.114:0/159031614 >> 172.21.15.114:0/159031614 conn(0x7f09f40fe010 msgr2=0x7f09f4107770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:47.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:47.917+0000 7f09fbc7a700 1 -- 172.21.15.114:0/159031614 shutdown_connections 2024-05-27T07:32:47.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:32:47.917+0000 7f09fbc7a700 1 -- 172.21.15.114:0/159031614 wait complete. 2024-05-27T07:32:48.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:48.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:48.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:47 smithi163 ceph-mon[36451]: from='osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T07:32:48.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:48.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:48.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:48.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:47 smithi186 ceph-mon[36647]: from='osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T07:32:48.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:48.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.285 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:48.285 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:48.285 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.285 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:47 smithi114 ceph-mon[32038]: from='osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T07:32:48.285 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.285 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:48.285 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:48.736 DEBUG:teuthology.orchestra.run.smithi114:osd.6> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.6.service 2024-05-27T07:32:48.738 INFO:tasks.cephadm:Deploying osd.7 on smithi163 with /dev/vg_nvme/lv_2... 2024-05-27T07:32:48.739 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T07:32:48.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:48 smithi114 ceph-mon[32038]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:48.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:48 smithi114 ceph-mon[32038]: from='osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T07:32:48.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:48 smithi114 ceph-mon[32038]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T07:32:48.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:48 smithi114 ceph-mon[32038]: from='osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-05-27T07:32:48.986 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:32:49.108 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:48 smithi163 ceph-mon[36451]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:49.108 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:48 smithi163 ceph-mon[36451]: from='osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T07:32:49.108 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:48 smithi163 ceph-mon[36451]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T07:32:49.108 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:48 smithi163 ceph-mon[36451]: from='osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-05-27T07:32:49.108 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:48 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:49.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:48 smithi186 ceph-mon[36647]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:49.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:48 smithi186 ceph-mon[36647]: from='osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T07:32:49.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:48 smithi186 ceph-mon[36647]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T07:32:49.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:48 smithi186 ceph-mon[36647]: from='osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-05-27T07:32:49.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:48 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:49.230 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:48 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:49.231 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 07:32:48 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T07:32:48.967+0000 7fa82e885700 -1 osd.6 0 waiting for initial osdmap 2024-05-27T07:32:49.231 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 07:32:48 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T07:32:48.977+0000 7fa82a63a700 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:32:50.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:49 smithi163 ceph-mon[36451]: from='osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-05-27T07:32:50.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:49 smithi163 ceph-mon[36451]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T07:32:50.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:49 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:50.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:49 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:50.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:49 smithi186 ceph-mon[36647]: from='osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-05-27T07:32:50.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:49 smithi186 ceph-mon[36647]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T07:32:50.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:49 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:50.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:49 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:50.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:49 smithi114 ceph-mon[32038]: from='osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-05-27T07:32:50.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:49 smithi114 ceph-mon[32038]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T07:32:50.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:49 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:50.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:49 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:51.120 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T07:32:51.121 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T07:32:51.121 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-05-27T07:32:51.121 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-05-27T07:32:51.121 INFO:teuthology.orchestra.run.smithi163.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114104 s, 919 MB/s 2024-05-27T07:32:51.121 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 2024-05-27T07:32:51.121 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-05-27T07:32:51.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:50 smithi163 ceph-mon[36451]: purged_snaps scrub starts 2024-05-27T07:32:51.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:50 smithi163 ceph-mon[36451]: purged_snaps scrub ok 2024-05-27T07:32:51.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:50 smithi163 ceph-mon[36451]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:51.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:51.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:50 smithi163 ceph-mon[36451]: osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644] boot 2024-05-27T07:32:51.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:50 smithi163 ceph-mon[36451]: osdmap e42: 7 total, 7 up, 7 in 2024-05-27T07:32:51.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:51.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:50 smithi186 ceph-mon[36647]: purged_snaps scrub starts 2024-05-27T07:32:51.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:50 smithi186 ceph-mon[36647]: purged_snaps scrub ok 2024-05-27T07:32:51.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:50 smithi186 ceph-mon[36647]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:51.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:51.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:50 smithi186 ceph-mon[36647]: osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644] boot 2024-05-27T07:32:51.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:50 smithi186 ceph-mon[36647]: osdmap e42: 7 total, 7 up, 7 in 2024-05-27T07:32:51.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:51.282 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:50 smithi114 ceph-mon[32038]: purged_snaps scrub starts 2024-05-27T07:32:51.282 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:50 smithi114 ceph-mon[32038]: purged_snaps scrub ok 2024-05-27T07:32:51.282 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:50 smithi114 ceph-mon[32038]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:32:51.282 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:51.282 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:50 smithi114 ceph-mon[32038]: osd.6 [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644] boot 2024-05-27T07:32:51.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:50 smithi114 ceph-mon[32038]: osdmap e42: 7 total, 7 up, 7 in 2024-05-27T07:32:51.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:32:51.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_2 2024-05-27T07:32:52.034 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:32:52.156 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:51 smithi163 ceph-mon[36451]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T07:32:52.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:51 smithi114 ceph-mon[32038]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T07:32:52.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:51 smithi186 ceph-mon[36647]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T07:32:53.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:52 smithi114 ceph-mon[32038]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:32:53.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:53.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:53.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:53.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:53.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:53.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:53.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:53.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:53.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:53.358 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:52 smithi163 ceph-mon[36451]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:32:53.358 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:53.358 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:53.358 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:53.358 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:53.358 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:53.358 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:53.358 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:53.358 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:53.359 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:53.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:52 smithi186 ceph-mon[36647]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:32:53.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:53.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:53.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:53.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:53.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:32:53.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:53.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:53.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:53.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:32:53.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.767+0000 7fd532c46700 1 -- 172.21.15.163:0/1075790858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd52c104470 msgr2=0x7fd52c10ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:53.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.767+0000 7fd532c46700 1 --2- 172.21.15.163:0/1075790858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd52c104470 0x7fd52c10ad20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd524009990 tx=0x7fd524009ca0 comp rx=0 tx=0).stop 2024-05-27T07:32:53.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.767+0000 7fd532c46700 1 -- 172.21.15.163:0/1075790858 shutdown_connections 2024-05-27T07:32:53.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.767+0000 7fd532c46700 1 --2- 172.21.15.163:0/1075790858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd52c104470 0x7fd52c10ad20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:53.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.767+0000 7fd532c46700 1 --2- 172.21.15.163:0/1075790858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd52c1029a0 0x7fd52c102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:53.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.767+0000 7fd532c46700 1 --2- 172.21.15.163:0/1075790858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd52c103ad0 0x7fd52c103ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:53.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.767+0000 7fd532c46700 1 -- 172.21.15.163:0/1075790858 >> 172.21.15.163:0/1075790858 conn(0x7fd52c0fe000 msgr2=0x7fd52c100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:53.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.767+0000 7fd532c46700 1 -- 172.21.15.163:0/1075790858 shutdown_connections 2024-05-27T07:32:53.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.767+0000 7fd532c46700 1 -- 172.21.15.163:0/1075790858 wait complete. 2024-05-27T07:32:53.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.768+0000 7fd532c46700 1 Processor -- start 2024-05-27T07:32:53.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.768+0000 7fd532c46700 1 -- start start 2024-05-27T07:32:53.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.768+0000 7fd532c46700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd52c1029a0 0x7fd52c19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:53.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.768+0000 7fd532c46700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd52c103ad0 0x7fd52c19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:53.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.768+0000 7fd532c46700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd52c104470 0x7fd52c1a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:53.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.768+0000 7fd532c46700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c10e480 con 0x7fd52c1029a0 2024-05-27T07:32:53.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.768+0000 7fd532c46700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c10e300 con 0x7fd52c104470 2024-05-27T07:32:53.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.768+0000 7fd532c46700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c10e600 con 0x7fd52c103ad0 2024-05-27T07:32:53.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.768+0000 7fd5311e3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd52c104470 0x7fd52c1a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:53.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.768+0000 7fd5311e3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd52c104470 0x7fd52c1a1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53916/0 (socket says 172.21.15.163:53916) 2024-05-27T07:32:53.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.768+0000 7fd5311e3700 1 -- 172.21.15.163:0/3450459201 learned_addr learned my addr 172.21.15.163:0/3450459201 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-27T07:32:53.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.769+0000 7fd523fff700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd52c103ad0 0x7fd52c19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:53.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.769+0000 7fd5311e3700 1 -- 172.21.15.163:0/3450459201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd52c103ad0 msgr2=0x7fd52c19ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:53.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.769+0000 7fd5311e3700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd52c103ad0 0x7fd52c19ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:53.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.769+0000 7fd5311e3700 1 -- 172.21.15.163:0/3450459201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd52c1029a0 msgr2=0x7fd52c19a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:53.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.769+0000 7fd5311e3700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd52c1029a0 0x7fd52c19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:53.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.769+0000 7fd5311e3700 1 -- 172.21.15.163:0/3450459201 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52c1a2420 con 0x7fd52c104470 2024-05-27T07:32:53.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.769+0000 7fd523fff700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd52c103ad0 0x7fd52c19ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:53.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.769+0000 7fd5311e3700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd52c104470 0x7fd52c1a1da0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd5240092e0 tx=0x7fd5240047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:53.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.769+0000 7fd521ffb700 1 -- 172.21.15.163:0/3450459201 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524027070 con 0x7fd52c104470 2024-05-27T07:32:53.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.769+0000 7fd532c46700 1 -- 172.21.15.163:0/3450459201 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52c1a26b0 con 0x7fd52c104470 2024-05-27T07:32:53.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.769+0000 7fd521ffb700 1 -- 172.21.15.163:0/3450459201 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5240040e0 con 0x7fd52c104470 2024-05-27T07:32:53.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.769+0000 7fd521ffb700 1 -- 172.21.15.163:0/3450459201 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524022de0 con 0x7fd52c104470 2024-05-27T07:32:53.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.769+0000 7fd532c46700 1 -- 172.21.15.163:0/3450459201 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd52c1a2b90 con 0x7fd52c104470 2024-05-27T07:32:53.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.771+0000 7fd532c46700 1 -- 172.21.15.163:0/3450459201 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd52c0512b0 con 0x7fd52c104470 2024-05-27T07:32:53.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.772+0000 7fd521ffb700 1 -- 172.21.15.163:0/3450459201 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fd52402a070 con 0x7fd52c104470 2024-05-27T07:32:53.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.772+0000 7fd521ffb700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fd5140b2880 0x7fd5140b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:53.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.773+0000 7fd521ffb700 1 -- 172.21.15.163:0/3450459201 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4965+0+0 (secure 0 0 0) 0x7fd5240e24c0 con 0x7fd52c104470 2024-05-27T07:32:53.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.773+0000 7fd5309e2700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fd5140b2880 0x7fd5140b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:53.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.775+0000 7fd5309e2700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fd5140b2880 0x7fd5140b4d90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd52c1039d0 tx=0x7fd528009250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:53.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.777+0000 7fd521ffb700 1 -- 172.21.15.163:0/3450459201 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd524013070 con 0x7fd52c104470 2024-05-27T07:32:53.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:32:53.991+0000 7fd532c46700 1 -- 172.21.15.163:0/3450459201 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fd52c109f20 con 0x7fd5140b2880 2024-05-27T07:32:54.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:53 smithi114 ceph-mon[32038]: Detected new or changed devices on smithi114 2024-05-27T07:32:54.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:53 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi114 to 5696M 2024-05-27T07:32:54.373 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:53 smithi186 ceph-mon[36647]: Detected new or changed devices on smithi114 2024-05-27T07:32:54.373 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:53 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi114 to 5696M 2024-05-27T07:32:54.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:53 smithi163 ceph-mon[36451]: Detected new or changed devices on smithi114 2024-05-27T07:32:54.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:53 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi114 to 5696M 2024-05-27T07:32:55.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:54 smithi114 ceph-mon[32038]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:32:55.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:54 smithi114 ceph-mon[32038]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:32:55.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:54 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:32:55.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:54 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:32:55.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:54 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:55.303 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:55 smithi163 ceph-mon[36451]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:32:55.303 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:55 smithi163 ceph-mon[36451]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:32:55.303 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:55 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:32:55.304 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:55 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:32:55.304 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:55 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:55.392 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:54 smithi186 ceph-mon[36647]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:32:55.393 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:54 smithi186 ceph-mon[36647]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:32:55.393 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:54 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:32:55.393 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:55 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:32:55.393 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:55 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:57.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:57 smithi114 ceph-mon[32038]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:32:57.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:57 smithi114 ceph-mon[32038]: from='client.? 172.21.15.163:0/1965732447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d"}]: dispatch 2024-05-27T07:32:57.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:57 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d"}]: dispatch 2024-05-27T07:32:57.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:57 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d"}]': finished 2024-05-27T07:32:57.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:57 smithi114 ceph-mon[32038]: osdmap e44: 8 total, 7 up, 8 in 2024-05-27T07:32:57.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:32:57.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:57 smithi163 ceph-mon[36451]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:32:57.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:57 smithi163 ceph-mon[36451]: from='client.? 172.21.15.163:0/1965732447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d"}]: dispatch 2024-05-27T07:32:57.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:57 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d"}]: dispatch 2024-05-27T07:32:57.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:57 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d"}]': finished 2024-05-27T07:32:57.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:57 smithi163 ceph-mon[36451]: osdmap e44: 8 total, 7 up, 8 in 2024-05-27T07:32:57.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:32:57.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:57 smithi186 ceph-mon[36647]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:32:57.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:57 smithi186 ceph-mon[36647]: from='client.? 172.21.15.163:0/1965732447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d"}]: dispatch 2024-05-27T07:32:57.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:57 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d"}]: dispatch 2024-05-27T07:32:57.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:57 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d"}]': finished 2024-05-27T07:32:57.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:57 smithi186 ceph-mon[36647]: osdmap e44: 8 total, 7 up, 8 in 2024-05-27T07:32:57.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:32:58.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:58 smithi114 ceph-mon[32038]: from='client.? 172.21.15.163:0/2555186622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:32:58.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:58 smithi163 ceph-mon[36451]: from='client.? 172.21.15.163:0/2555186622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:32:58.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:58 smithi186 ceph-mon[36647]: from='client.? 172.21.15.163:0/2555186622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:32:59.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:32:59 smithi114 ceph-mon[32038]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:32:59.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:32:59 smithi163 ceph-mon[36451]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:32:59.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:32:59 smithi186 ceph-mon[36647]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:01.206 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:01 smithi163 ceph-mon[36451]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:01.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:01 smithi114 ceph-mon[32038]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:01.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:01 smithi186 ceph-mon[36647]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:03.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:03 smithi114 ceph-mon[32038]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:03.308 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:03 smithi163 ceph-mon[36451]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:03.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:03 smithi186 ceph-mon[36647]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:05.292 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:05 smithi163 ceph-mon[36451]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:05.292 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:05 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T07:33:05.292 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:05 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:05.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:05 smithi114 ceph-mon[32038]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:05.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:05 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T07:33:05.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:05 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:05.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:05 smithi186 ceph-mon[36647]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:05.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:05 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T07:33:05.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:05 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:06.288 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:06 smithi163 ceph-mon[36451]: Deploying daemon osd.7 on smithi163 2024-05-27T07:33:06.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:06 smithi114 ceph-mon[32038]: Deploying daemon osd.7 on smithi163 2024-05-27T07:33:06.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:06 smithi186 ceph-mon[36647]: Deploying daemon osd.7 on smithi163 2024-05-27T07:33:07.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:07 smithi114 ceph-mon[32038]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:07.370 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:07 smithi163 ceph-mon[36451]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:07.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:07 smithi186 ceph-mon[36647]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:09.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:09 smithi114 ceph-mon[32038]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:09.379 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:09 smithi163 ceph-mon[36451]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:09.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:09 smithi186 ceph-mon[36647]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:11.210 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:11 smithi163 ceph-mon[36451]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:11.211 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:11.211 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:11.211 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:11.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:11 smithi114 ceph-mon[32038]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:11.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:11.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:11.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:11.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:11 smithi186 ceph-mon[36647]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:11.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:11.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:11.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:13.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:13 smithi163 ceph-mon[36451]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:13.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:13 smithi114 ceph-mon[32038]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:13.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:13 smithi186 ceph-mon[36647]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:14.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:33:14.059+0000 7fd521ffb700 1 -- 172.21.15.163:0/3450459201 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd52c109f20 con 0x7fd5140b2880 2024-05-27T07:33:14.053 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 7 on host 'smithi163' 2024-05-27T07:33:14.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:33:14.061+0000 7fd532c46700 1 -- 172.21.15.163:0/3450459201 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fd5140b2880 msgr2=0x7fd5140b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:14.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:33:14.061+0000 7fd532c46700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fd5140b2880 0x7fd5140b4d90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd52c1039d0 tx=0x7fd528009250 comp rx=0 tx=0).stop 2024-05-27T07:33:14.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:33:14.061+0000 7fd532c46700 1 -- 172.21.15.163:0/3450459201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd52c104470 msgr2=0x7fd52c1a1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:14.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:33:14.061+0000 7fd532c46700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd52c104470 0x7fd52c1a1da0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd5240092e0 tx=0x7fd5240047b0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:33:14.061+0000 7fd532c46700 1 -- 172.21.15.163:0/3450459201 shutdown_connections 2024-05-27T07:33:14.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:33:14.061+0000 7fd532c46700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd52c104470 0x7fd52c1a1da0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:33:14.061+0000 7fd532c46700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd52c1029a0 0x7fd52c19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:33:14.061+0000 7fd532c46700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fd5140b2880 0x7fd5140b4d90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:33:14.061+0000 7fd532c46700 1 --2- 172.21.15.163:0/3450459201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd52c103ad0 0x7fd52c19ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:33:14.061+0000 7fd532c46700 1 -- 172.21.15.163:0/3450459201 >> 172.21.15.163:0/3450459201 conn(0x7fd52c0fe000 msgr2=0x7fd52c101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:14.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:33:14.062+0000 7fd532c46700 1 -- 172.21.15.163:0/3450459201 shutdown_connections 2024-05-27T07:33:14.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:33:14.062+0000 7fd532c46700 1 -- 172.21.15.163:0/3450459201 wait complete. 2024-05-27T07:33:14.763 DEBUG:teuthology.orchestra.run.smithi163:osd.7> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.7.service 2024-05-27T07:33:14.766 INFO:tasks.cephadm:Deploying osd.8 on smithi186 with /dev/vg_nvme/lv_2... 2024-05-27T07:33:14.766 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T07:33:15.012 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:33:15.133 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:15 smithi186 ceph-mon[36647]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:15.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:15 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:15.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:15 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:15.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:15 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:15.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:15 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:15.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:15 smithi186 ceph-mon[36647]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:33:15.298 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:15 smithi163 ceph-mon[36451]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:15.298 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:15 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:15.298 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:15 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:15.298 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:15 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:15.298 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:15 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:15.298 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:15 smithi163 ceph-mon[36451]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:33:15.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:15 smithi114 ceph-mon[32038]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:15.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:15 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:15.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:15 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:15.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:15 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:15.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:15 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:15.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:15 smithi114 ceph-mon[32038]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:33:16.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:16 smithi163 ceph-mon[36451]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T07:33:16.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:16 smithi163 ceph-mon[36451]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T07:33:16.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:16 smithi163 ceph-mon[36451]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:33:16.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:16 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:16.298 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:16 smithi186 ceph-mon[36647]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T07:33:16.298 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:16 smithi186 ceph-mon[36647]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T07:33:16.298 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:16 smithi186 ceph-mon[36647]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:33:16.298 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:16 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:16.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:16 smithi114 ceph-mon[32038]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T07:33:16.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:16 smithi114 ceph-mon[32038]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T07:33:16.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:16 smithi114 ceph-mon[32038]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:33:16.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:16 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:16.969 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 07:33:16 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T07:33:16.596+0000 7f21a1250700 -1 osd.7 0 waiting for initial osdmap 2024-05-27T07:33:16.970 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 07:33:16 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T07:33:16.606+0000 7f2198ffd700 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:33:17.184 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T07:33:17.184 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T07:33:17.184 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-05-27T07:33:17.184 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-05-27T07:33:17.184 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131482 s, 798 MB/s 2024-05-27T07:33:17.184 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-05-27T07:33:17.241 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:17 smithi186 ceph-mon[36647]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:17.241 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:17 smithi186 ceph-mon[36647]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-27T07:33:17.241 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:17 smithi186 ceph-mon[36647]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T07:33:17.241 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:17 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:17.241 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:17 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:17.241 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:17 smithi186 ceph-mon[36647]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' 2024-05-27T07:33:17.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:17 smithi163 ceph-mon[36451]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:17.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:17 smithi163 ceph-mon[36451]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-27T07:33:17.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:17 smithi163 ceph-mon[36451]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T07:33:17.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:17 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:17.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:17 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:17.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:17 smithi163 ceph-mon[36451]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' 2024-05-27T07:33:17.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:17 smithi114 ceph-mon[32038]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:17.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:17 smithi114 ceph-mon[32038]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-27T07:33:17.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:17 smithi114 ceph-mon[32038]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T07:33:17.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:17 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:17.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:17 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:17.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:17 smithi114 ceph-mon[32038]: from='osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962]' entity='osd.7' 2024-05-27T07:33:17.913 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_2 2024-05-27T07:33:18.159 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:33:18.219 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: purged_snaps scrub starts 2024-05-27T07:33:18.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: purged_snaps scrub ok 2024-05-27T07:33:18.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:18.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962] boot 2024-05-27T07:33:18.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: osdmap e47: 8 total, 8 up, 8 in 2024-05-27T07:33:18.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:18.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:18.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:18.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:18.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:18.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:18.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:18.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:18.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:18.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:18 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:18.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: purged_snaps scrub starts 2024-05-27T07:33:18.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: purged_snaps scrub ok 2024-05-27T07:33:18.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:18.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962] boot 2024-05-27T07:33:18.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: osdmap e47: 8 total, 8 up, 8 in 2024-05-27T07:33:18.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:18.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:18.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:18.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:18.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:18.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:18.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:18.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:18.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:18.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:18 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:18.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: purged_snaps scrub starts 2024-05-27T07:33:18.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: purged_snaps scrub ok 2024-05-27T07:33:18.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:18.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: osd.7 [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962] boot 2024-05-27T07:33:18.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: osdmap e47: 8 total, 8 up, 8 in 2024-05-27T07:33:18.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:33:18.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:18.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:18.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:18.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:18.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:18.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:18.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:18.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:18.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:18 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:19.288 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:19 smithi186 ceph-mon[36647]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:19.288 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:19 smithi186 ceph-mon[36647]: Detected new or changed devices on smithi163 2024-05-27T07:33:19.288 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:19 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi163 to 5696M 2024-05-27T07:33:19.288 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:19 smithi186 ceph-mon[36647]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T07:33:19.288 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:19 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:19.288 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:19 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:19.288 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:19 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:19.288 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:19 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:19.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:19 smithi163 ceph-mon[36451]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:19.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:19 smithi163 ceph-mon[36451]: Detected new or changed devices on smithi163 2024-05-27T07:33:19.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:19 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi163 to 5696M 2024-05-27T07:33:19.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:19 smithi163 ceph-mon[36451]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T07:33:19.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:19 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:19.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:19 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:19.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:19 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:19.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:19 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:19.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:19 smithi114 ceph-mon[32038]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:33:19.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:19 smithi114 ceph-mon[32038]: Detected new or changed devices on smithi163 2024-05-27T07:33:19.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:19 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi163 to 5696M 2024-05-27T07:33:19.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:19 smithi114 ceph-mon[32038]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T07:33:19.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:19 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:19.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:19 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:19.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:19 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:19.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:19 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:19.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.886+0000 7fcde7d30700 1 -- 172.21.15.186:0/389787791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde0103230 msgr2=0x7fcde0103640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:19.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.887+0000 7fcde7d30700 1 --2- 172.21.15.186:0/389787791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde0103230 0x7fcde0103640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009990 tx=0x7fcddc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.887+0000 7fcde7d30700 1 -- 172.21.15.186:0/389787791 shutdown_connections 2024-05-27T07:33:19.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.887+0000 7fcde7d30700 1 --2- 172.21.15.186:0/389787791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcde0104d00 0x7fcde010b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.887+0000 7fcde7d30700 1 --2- 172.21.15.186:0/389787791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde0103230 0x7fcde0103640 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.887+0000 7fcde7d30700 1 --2- 172.21.15.186:0/389787791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcde0104360 0x7fcde0104770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.887+0000 7fcde7d30700 1 -- 172.21.15.186:0/389787791 >> 172.21.15.186:0/389787791 conn(0x7fcde00fe870 msgr2=0x7fcde0100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:19.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.887+0000 7fcde7d30700 1 -- 172.21.15.186:0/389787791 shutdown_connections 2024-05-27T07:33:19.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.887+0000 7fcde7d30700 1 -- 172.21.15.186:0/389787791 wait complete. 2024-05-27T07:33:19.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.887+0000 7fcde7d30700 1 Processor -- start 2024-05-27T07:33:19.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.887+0000 7fcde7d30700 1 -- start start 2024-05-27T07:33:19.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde7d30700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde0103230 0x7fcde006c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde7d30700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcde0104360 0x7fcde019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde7d30700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcde0104d00 0x7fcde01a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde7d30700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcde010ec80 con 0x7fcde0104d00 2024-05-27T07:33:19.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde7d30700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcde010eb00 con 0x7fcde0104360 2024-05-27T07:33:19.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde7d30700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcde010ee00 con 0x7fcde0103230 2024-05-27T07:33:19.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde5acc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde0103230 0x7fcde006c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde5acc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde0103230 0x7fcde006c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:37176/0 (socket says 172.21.15.186:37176) 2024-05-27T07:33:19.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde5acc700 1 -- 172.21.15.186:0/1397238664 learned_addr learned my addr 172.21.15.186:0/1397238664 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-27T07:33:19.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde52cb700 1 --2- 172.21.15.186:0/1397238664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcde0104360 0x7fcde019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde5acc700 1 -- 172.21.15.186:0/1397238664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcde0104360 msgr2=0x7fcde019f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:19.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde5acc700 1 --2- 172.21.15.186:0/1397238664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcde0104360 0x7fcde019f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde5acc700 1 -- 172.21.15.186:0/1397238664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcde0104d00 msgr2=0x7fcde01a1c60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:33:19.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde5acc700 1 --2- 172.21.15.186:0/1397238664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcde0104d00 0x7fcde01a1c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde5acc700 1 -- 172.21.15.186:0/1397238664 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde01994c0 con 0x7fcde0103230 2024-05-27T07:33:19.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde5acc700 1 --2- 172.21.15.186:0/1397238664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde0103230 0x7fcde006c060 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcddc006010 tx=0x7fcddc0057a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:19.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcdd6ffd700 1 -- 172.21.15.186:0/1397238664 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc027070 con 0x7fcde0103230 2024-05-27T07:33:19.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcdd6ffd700 1 -- 172.21.15.186:0/1397238664 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcddc004000 con 0x7fcde0103230 2024-05-27T07:33:19.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcdd6ffd700 1 -- 172.21.15.186:0/1397238664 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc022ca0 con 0x7fcde0103230 2024-05-27T07:33:19.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde7d30700 1 -- 172.21.15.186:0/1397238664 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcde0199750 con 0x7fcde0103230 2024-05-27T07:33:19.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.888+0000 7fcde7d30700 1 -- 172.21.15.186:0/1397238664 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcde0199c80 con 0x7fcde0103230 2024-05-27T07:33:19.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.890+0000 7fcde7d30700 1 -- 172.21.15.186:0/1397238664 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcde00512b0 con 0x7fcde0103230 2024-05-27T07:33:19.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.891+0000 7fcdd6ffd700 1 -- 172.21.15.186:0/1397238664 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fcddc02a070 con 0x7fcde0103230 2024-05-27T07:33:19.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.892+0000 7fcdd6ffd700 1 --2- 172.21.15.186:0/1397238664 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fcdcc0b2900 0x7fcdcc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.892+0000 7fcdd6ffd700 1 -- 172.21.15.186:0/1397238664 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5397+0+0 (secure 0 0 0) 0x7fcddc0e2cb0 con 0x7fcde0103230 2024-05-27T07:33:19.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.893+0000 7fcde52cb700 1 --2- 172.21.15.186:0/1397238664 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fcdcc0b2900 0x7fcdcc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.888 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.895+0000 7fcdd6ffd700 1 -- 172.21.15.186:0/1397238664 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcddc013070 con 0x7fcde0103230 2024-05-27T07:33:19.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:19.897+0000 7fcde52cb700 1 --2- 172.21.15.186:0/1397238664 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fcdcc0b2900 0x7fcdcc0b4e10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcde006ba10 tx=0x7fcdd000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:20.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:20.107+0000 7fcde7d30700 1 -- 172.21.15.186:0/1397238664 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fcde0109ea0 con 0x7fcdcc0b2900 2024-05-27T07:33:20.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:20 smithi163 ceph-mon[36451]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T07:33:20.469 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:20 smithi163 ceph-mon[36451]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:20.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:20 smithi163 ceph-mon[36451]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:33:20.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:33:20.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:33:20.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:20.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:20 smithi186 ceph-mon[36647]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T07:33:20.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:20 smithi186 ceph-mon[36647]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:20.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:20 smithi186 ceph-mon[36647]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:33:20.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:33:20.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:33:20.472 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:20.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:20 smithi114 ceph-mon[32038]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T07:33:20.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:20 smithi114 ceph-mon[32038]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:20.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:20 smithi114 ceph-mon[32038]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:33:20.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:33:20.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:33:20.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:22.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:22 smithi163 ceph-mon[36451]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:22.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:22 smithi186 ceph-mon[36647]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:22.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:22 smithi114 ceph-mon[32038]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:23.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:23 smithi163 ceph-mon[36451]: from='client.? 172.21.15.186:0/1298407590' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5acc04eb-36af-4a65-9840-dba989b888a5"}]: dispatch 2024-05-27T07:33:23.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:23 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5acc04eb-36af-4a65-9840-dba989b888a5"}]: dispatch 2024-05-27T07:33:23.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:23 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5acc04eb-36af-4a65-9840-dba989b888a5"}]': finished 2024-05-27T07:33:23.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:23 smithi163 ceph-mon[36451]: osdmap e50: 9 total, 8 up, 9 in 2024-05-27T07:33:23.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:23 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:23.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:23 smithi186 ceph-mon[36647]: from='client.? 172.21.15.186:0/1298407590' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5acc04eb-36af-4a65-9840-dba989b888a5"}]: dispatch 2024-05-27T07:33:23.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:23 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5acc04eb-36af-4a65-9840-dba989b888a5"}]: dispatch 2024-05-27T07:33:23.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:23 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5acc04eb-36af-4a65-9840-dba989b888a5"}]': finished 2024-05-27T07:33:23.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:23 smithi186 ceph-mon[36647]: osdmap e50: 9 total, 8 up, 9 in 2024-05-27T07:33:23.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:23 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:23.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:23 smithi114 ceph-mon[32038]: from='client.? 172.21.15.186:0/1298407590' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5acc04eb-36af-4a65-9840-dba989b888a5"}]: dispatch 2024-05-27T07:33:23.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:23 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5acc04eb-36af-4a65-9840-dba989b888a5"}]: dispatch 2024-05-27T07:33:23.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:23 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5acc04eb-36af-4a65-9840-dba989b888a5"}]': finished 2024-05-27T07:33:23.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:23 smithi114 ceph-mon[32038]: osdmap e50: 9 total, 8 up, 9 in 2024-05-27T07:33:23.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:23 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:24.635 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:24 smithi163 ceph-mon[36451]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:33:24.635 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:24 smithi163 ceph-mon[36451]: from='client.? 172.21.15.186:0/1192954644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:33:24.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:24 smithi186 ceph-mon[36647]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:33:24.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:24 smithi186 ceph-mon[36647]: from='client.? 172.21.15.186:0/1192954644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:33:24.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:24 smithi114 ceph-mon[32038]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:33:24.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:24 smithi114 ceph-mon[32038]: from='client.? 172.21.15.186:0/1192954644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:33:26.661 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:26 smithi186 ceph-mon[36647]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-05-27T07:33:26.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:26 smithi163 ceph-mon[36451]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-05-27T07:33:26.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:26 smithi114 ceph-mon[32038]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-05-27T07:33:28.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:28 smithi163 ceph-mon[36451]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-05-27T07:33:28.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:28 smithi186 ceph-mon[36647]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-05-27T07:33:28.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:28 smithi114 ceph-mon[32038]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-05-27T07:33:30.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:30 smithi163 ceph-mon[36451]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:33:30.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:30 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T07:33:30.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:30 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:30.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:30 smithi186 ceph-mon[36647]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:33:30.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:30 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T07:33:30.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:30 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:30.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:30 smithi114 ceph-mon[32038]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:33:30.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:30 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T07:33:30.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:30 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:31.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:31 smithi163 ceph-mon[36451]: Deploying daemon osd.8 on smithi186 2024-05-27T07:33:31.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:31 smithi186 ceph-mon[36647]: Deploying daemon osd.8 on smithi186 2024-05-27T07:33:31.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:31 smithi114 ceph-mon[32038]: Deploying daemon osd.8 on smithi186 2024-05-27T07:33:32.643 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:32 smithi186 ceph-mon[36647]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:33:32.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:32 smithi163 ceph-mon[36451]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:33:32.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:32 smithi114 ceph-mon[32038]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:33:34.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:34 smithi163 ceph-mon[36451]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-05-27T07:33:34.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:34 smithi186 ceph-mon[36647]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-05-27T07:33:34.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:34 smithi114 ceph-mon[32038]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-05-27T07:33:36.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:36 smithi186 ceph-mon[36647]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:36.713 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:36.713 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:36.713 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:36.719 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:36 smithi163 ceph-mon[36451]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:36.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:36.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:36.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:36.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:36 smithi114 ceph-mon[32038]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:36.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:36.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:36.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:38.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:38 smithi163 ceph-mon[36451]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:38.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:38 smithi186 ceph-mon[36647]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:38.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:38 smithi114 ceph-mon[32038]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:40.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:40.421+0000 7fcdd6ffd700 1 -- 172.21.15.186:0/1397238664 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcde0109ea0 con 0x7fcdcc0b2900 2024-05-27T07:33:40.416 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 8 on host 'smithi186' 2024-05-27T07:33:40.418 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:40.423+0000 7fcde7d30700 1 -- 172.21.15.186:0/1397238664 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fcdcc0b2900 msgr2=0x7fcdcc0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:40.418 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:40.424+0000 7fcde7d30700 1 --2- 172.21.15.186:0/1397238664 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fcdcc0b2900 0x7fcdcc0b4e10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcde006ba10 tx=0x7fcdd000b410 comp rx=0 tx=0).stop 2024-05-27T07:33:40.418 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:40.424+0000 7fcde7d30700 1 -- 172.21.15.186:0/1397238664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde0103230 msgr2=0x7fcde006c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:40.418 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:40.424+0000 7fcde7d30700 1 --2- 172.21.15.186:0/1397238664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde0103230 0x7fcde006c060 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcddc006010 tx=0x7fcddc0057a0 comp rx=0 tx=0).stop 2024-05-27T07:33:40.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:40.424+0000 7fcde7d30700 1 -- 172.21.15.186:0/1397238664 shutdown_connections 2024-05-27T07:33:40.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:40.424+0000 7fcde7d30700 1 --2- 172.21.15.186:0/1397238664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcde0104d00 0x7fcde01a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:40.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:40.424+0000 7fcde7d30700 1 --2- 172.21.15.186:0/1397238664 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fcdcc0b2900 0x7fcdcc0b4e10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:40.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:40.424+0000 7fcde7d30700 1 --2- 172.21.15.186:0/1397238664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde0103230 0x7fcde006c060 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:40.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:40.424+0000 7fcde7d30700 1 --2- 172.21.15.186:0/1397238664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcde0104360 0x7fcde019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:40.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:40.424+0000 7fcde7d30700 1 -- 172.21.15.186:0/1397238664 >> 172.21.15.186:0/1397238664 conn(0x7fcde00fe870 msgr2=0x7fcde0107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:40.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:40.424+0000 7fcde7d30700 1 -- 172.21.15.186:0/1397238664 shutdown_connections 2024-05-27T07:33:40.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:33:40.424+0000 7fcde7d30700 1 -- 172.21.15.186:0/1397238664 wait complete. 2024-05-27T07:33:40.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:40 smithi186 ceph-mon[36647]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:40.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:40.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:40.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:40.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:40.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:40.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:40 smithi186 ceph-mon[36647]: from='osd.8 [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:33:40.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:40 smithi186 ceph-mon[36647]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:33:40.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:40.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:40.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:40.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:40 smithi163 ceph-mon[36451]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:40.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:40.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:40.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:40.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:40.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:40.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:40 smithi163 ceph-mon[36451]: from='osd.8 [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:33:40.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:40 smithi163 ceph-mon[36451]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:33:40.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:40.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:40.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:41.050 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:40 smithi114 ceph-mon[32038]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:41.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:41.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:41.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:41.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:41.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:41.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:40 smithi114 ceph-mon[32038]: from='osd.8 [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:33:41.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:40 smithi114 ceph-mon[32038]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:33:41.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:41.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:41.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:41.093 DEBUG:teuthology.orchestra.run.smithi186:osd.8> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.8.service 2024-05-27T07:33:41.095 INFO:tasks.cephadm:Deploying osd.9 on smithi114 with /dev/vg_nvme/lv_1... 2024-05-27T07:33:41.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T07:33:41.343 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:41.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:41 smithi186 ceph-mon[36647]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T07:33:41.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:41 smithi186 ceph-mon[36647]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T07:33:41.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:41.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:41 smithi186 ceph-mon[36647]: from='osd.8 [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:33:41.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:41 smithi186 ceph-mon[36647]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:33:41.724 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 07:33:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T07:33:41.596+0000 7f8bf1f7f700 -1 osd.8 0 waiting for initial osdmap 2024-05-27T07:33:41.724 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 07:33:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T07:33:41.605+0000 7f8bedd34700 -1 osd.8 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:33:41.748 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:41 smithi114 ceph-mon[32038]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T07:33:41.748 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:41 smithi114 ceph-mon[32038]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T07:33:41.748 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:41.748 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:41 smithi114 ceph-mon[32038]: from='osd.8 [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:33:41.749 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:41 smithi114 ceph-mon[32038]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:33:41.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:41 smithi163 ceph-mon[36451]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T07:33:41.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:41 smithi163 ceph-mon[36451]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T07:33:41.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:41.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:41 smithi163 ceph-mon[36451]: from='osd.8 [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:33:41.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:41 smithi163 ceph-mon[36451]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:33:42.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:42 smithi114 ceph-mon[32038]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:42.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:42 smithi114 ceph-mon[32038]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-27T07:33:42.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:42 smithi114 ceph-mon[32038]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T07:33:42.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:42 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:42.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:42 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:42.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:42 smithi114 ceph-mon[32038]: osd.8 [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353] boot 2024-05-27T07:33:42.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:42 smithi114 ceph-mon[32038]: osdmap e53: 9 total, 9 up, 9 in 2024-05-27T07:33:42.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:42 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:42.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:42 smithi163 ceph-mon[36451]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:42.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:42 smithi163 ceph-mon[36451]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-27T07:33:42.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:42 smithi163 ceph-mon[36451]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T07:33:42.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:42 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:42.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:42 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:42.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:42 smithi163 ceph-mon[36451]: osd.8 [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353] boot 2024-05-27T07:33:42.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:42 smithi163 ceph-mon[36451]: osdmap e53: 9 total, 9 up, 9 in 2024-05-27T07:33:42.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:42 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:42.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:42 smithi186 ceph-mon[36647]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:33:42.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:42 smithi186 ceph-mon[36647]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-27T07:33:42.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:42 smithi186 ceph-mon[36647]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T07:33:42.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:42 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:42.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:42 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:42.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:42 smithi186 ceph-mon[36647]: osd.8 [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353] boot 2024-05-27T07:33:42.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:42 smithi186 ceph-mon[36647]: osdmap e53: 9 total, 9 up, 9 in 2024-05-27T07:33:42.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:42 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:33:43.548 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T07:33:43.548 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T07:33:43.548 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-05-27T07:33:43.548 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-05-27T07:33:43.548 INFO:teuthology.orchestra.run.smithi114.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0150232 s, 698 MB/s 2024-05-27T07:33:43.549 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-05-27T07:33:43.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:43 smithi114 ceph-mon[32038]: purged_snaps scrub starts 2024-05-27T07:33:43.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:43 smithi114 ceph-mon[32038]: purged_snaps scrub ok 2024-05-27T07:33:43.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:43 smithi163 ceph-mon[36451]: purged_snaps scrub starts 2024-05-27T07:33:43.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:43 smithi163 ceph-mon[36451]: purged_snaps scrub ok 2024-05-27T07:33:43.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:43 smithi186 ceph-mon[36647]: purged_snaps scrub starts 2024-05-27T07:33:43.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:43 smithi186 ceph-mon[36647]: purged_snaps scrub ok 2024-05-27T07:33:44.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_1 2024-05-27T07:33:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:44.882 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:33:44.882 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T07:33:44.882 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: Detected new or changed devices on smithi186 2024-05-27T07:33:44.882 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:44.882 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:44.882 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:44.882 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:44.882 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:44.882 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi186 to 5696M 2024-05-27T07:33:44.882 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:44.883 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:44.883 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:44.883 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:44.883 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:44 smithi114 ceph-mon[32038]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T07:33:44.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:33:44.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T07:33:44.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: Detected new or changed devices on smithi186 2024-05-27T07:33:44.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:44.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:44.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:44.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:44.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:44.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi186 to 5696M 2024-05-27T07:33:44.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:44.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:44.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:44.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:44.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:44 smithi163 ceph-mon[36451]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T07:33:44.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:33:44.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T07:33:44.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: Detected new or changed devices on smithi186 2024-05-27T07:33:44.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:44.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:44.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:44.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:44.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:44.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi186 to 5696M 2024-05-27T07:33:44.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:44.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:44.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:44.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:33:44.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:44 smithi186 ceph-mon[36647]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T07:33:46.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.381+0000 7fa2a046b700 1 -- 172.21.15.114:0/1362609040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa298104e20 msgr2=0x7fa298107240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:46.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.381+0000 7fa2a046b700 1 --2- 172.21.15.114:0/1362609040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa298104e20 0x7fa298107240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa294009990 tx=0x7fa294009ca0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.381+0000 7fa2a046b700 1 -- 172.21.15.114:0/1362609040 shutdown_connections 2024-05-27T07:33:46.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.381+0000 7fa2a046b700 1 --2- 172.21.15.114:0/1362609040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa29810a0e0 0x7fa29810c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.381+0000 7fa2a046b700 1 --2- 172.21.15.114:0/1362609040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa298104e20 0x7fa298107240 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.381+0000 7fa2a046b700 1 --2- 172.21.15.114:0/1362609040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa298107780 0x7fa298109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.381+0000 7fa2a046b700 1 -- 172.21.15.114:0/1362609040 >> 172.21.15.114:0/1362609040 conn(0x7fa2980fe860 msgr2=0x7fa298100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.381+0000 7fa2a046b700 1 -- 172.21.15.114:0/1362609040 shutdown_connections 2024-05-27T07:33:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.381+0000 7fa2a046b700 1 -- 172.21.15.114:0/1362609040 wait complete. 2024-05-27T07:33:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.382+0000 7fa2a046b700 1 Processor -- start 2024-05-27T07:33:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.382+0000 7fa2a046b700 1 -- start start 2024-05-27T07:33:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.382+0000 7fa2a046b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa298104e20 0x7fa29819a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.382+0000 7fa2a046b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa298107780 0x7fa29819ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.382+0000 7fa2a046b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa29810a0e0 0x7fa2981a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa2a046b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa29810fd60 con 0x7fa298104e20 2024-05-27T07:33:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa2a046b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa29810fbe0 con 0x7fa298107780 2024-05-27T07:33:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa2a046b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa29810fee0 con 0x7fa29810a0e0 2024-05-27T07:33:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa29e207700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa298104e20 0x7fa29819a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa29e207700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa298104e20 0x7fa29819a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36908/0 (socket says 172.21.15.114:36908) 2024-05-27T07:33:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa29e207700 1 -- 172.21.15.114:0/2970842577 learned_addr learned my addr 172.21.15.114:0/2970842577 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa29ea08700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa29810a0e0 0x7fa2981a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa29da06700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa298107780 0x7fa29819ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa29e207700 1 -- 172.21.15.114:0/2970842577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa29810a0e0 msgr2=0x7fa2981a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa29e207700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa29810a0e0 0x7fa2981a1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa29e207700 1 -- 172.21.15.114:0/2970842577 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa298107780 msgr2=0x7fa29819ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa29e207700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa298107780 0x7fa29819ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa29e207700 1 -- 172.21.15.114:0/2970842577 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2981a2460 con 0x7fa298104e20 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa29e207700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa298104e20 0x7fa29819a900 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa294006010 tx=0x7fa2940057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa28f7fe700 1 -- 172.21.15.114:0/2970842577 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa294027070 con 0x7fa298104e20 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa2a046b700 1 -- 172.21.15.114:0/2970842577 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2981a26f0 con 0x7fa298104e20 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa28f7fe700 1 -- 172.21.15.114:0/2970842577 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2940040c0 con 0x7fa298104e20 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.383+0000 7fa28f7fe700 1 -- 172.21.15.114:0/2970842577 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa294022d60 con 0x7fa298104e20 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.384+0000 7fa2a046b700 1 -- 172.21.15.114:0/2970842577 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2981a2b40 con 0x7fa298104e20 2024-05-27T07:33:46.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.385+0000 7fa2a046b700 1 -- 172.21.15.114:0/2970842577 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2980512b0 con 0x7fa298104e20 2024-05-27T07:33:46.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.386+0000 7fa28f7fe700 1 -- 172.21.15.114:0/2970842577 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fa29402a070 con 0x7fa298104e20 2024-05-27T07:33:46.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.387+0000 7fa28f7fe700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa2840b2880 0x7fa2840b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.387+0000 7fa29da06700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa2840b2880 0x7fa2840b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.387+0000 7fa28f7fe700 1 -- 172.21.15.114:0/2970842577 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5829+0+0 (secure 0 0 0) 0x7fa2940e3070 con 0x7fa298104e20 2024-05-27T07:33:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.388+0000 7fa29da06700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa2840b2880 0x7fa2840b4d90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa288005950 tx=0x7fa288005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.391+0000 7fa28f7fe700 1 -- 172.21.15.114:0/2970842577 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa294031080 con 0x7fa298104e20 2024-05-27T07:33:46.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:33:46.606+0000 7fa2a046b700 1 -- 172.21.15.114:0/2970842577 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fa298063bf0 con 0x7fa2840b2880 2024-05-27T07:33:46.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:46 smithi114 ceph-mon[32038]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:33:46.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:46 smithi163 ceph-mon[36451]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:33:46.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:46 smithi186 ceph-mon[36647]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:33:47.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:47 smithi114 ceph-mon[32038]: from='client.14397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:33:47.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:33:47.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:33:47.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:47 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:47.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:47 smithi163 ceph-mon[36451]: from='client.14397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:33:47.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:33:47.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:33:47.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:47 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:47.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:47 smithi186 ceph-mon[36647]: from='client.14397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:33:47.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:33:47.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:33:47.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:47 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:48.908 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:48 smithi114 ceph-mon[32038]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:33:48.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:48 smithi163 ceph-mon[36451]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:33:48.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:48 smithi186 ceph-mon[36647]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:33:49.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:49 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3318221286' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56bba69d-42d5-4bc9-8e80-1761b4fb12cd"}]: dispatch 2024-05-27T07:33:49.945 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:49 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3318221286' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56bba69d-42d5-4bc9-8e80-1761b4fb12cd"}]': finished 2024-05-27T07:33:49.945 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:49 smithi114 ceph-mon[32038]: osdmap e56: 10 total, 9 up, 10 in 2024-05-27T07:33:49.945 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:49 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:33:49.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:49 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3318221286' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56bba69d-42d5-4bc9-8e80-1761b4fb12cd"}]: dispatch 2024-05-27T07:33:49.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:49 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3318221286' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56bba69d-42d5-4bc9-8e80-1761b4fb12cd"}]': finished 2024-05-27T07:33:49.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:49 smithi163 ceph-mon[36451]: osdmap e56: 10 total, 9 up, 10 in 2024-05-27T07:33:49.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:49 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:33:49.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:49 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3318221286' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56bba69d-42d5-4bc9-8e80-1761b4fb12cd"}]: dispatch 2024-05-27T07:33:49.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:49 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3318221286' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56bba69d-42d5-4bc9-8e80-1761b4fb12cd"}]': finished 2024-05-27T07:33:49.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:49 smithi186 ceph-mon[36647]: osdmap e56: 10 total, 9 up, 10 in 2024-05-27T07:33:49.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:49 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:33:50.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:50 smithi163 ceph-mon[36451]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-27T07:33:50.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:50 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2050838521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:33:50.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:50 smithi186 ceph-mon[36647]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-27T07:33:50.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:50 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2050838521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:33:51.050 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:50 smithi114 ceph-mon[32038]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-27T07:33:51.050 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:50 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2050838521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:33:52.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:52 smithi163 ceph-mon[36451]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:33:52.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:52 smithi186 ceph-mon[36647]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:33:53.050 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:52 smithi114 ceph-mon[32038]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:33:54.896 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:54 smithi114 ceph-mon[32038]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-27T07:33:54.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:54 smithi163 ceph-mon[36451]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-27T07:33:54.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:54 smithi186 ceph-mon[36647]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-27T07:33:56.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:56 smithi114 ceph-mon[32038]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:33:56.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:56 smithi163 ceph-mon[36451]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:33:56.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:56 smithi186 ceph-mon[36647]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:33:57.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T07:33:57.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:57.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T07:33:57.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:57.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T07:33:57.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:58.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:58 smithi163 ceph-mon[36451]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:33:58.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:33:58 smithi163 ceph-mon[36451]: Deploying daemon osd.9 on smithi114 2024-05-27T07:33:58.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:58 smithi186 ceph-mon[36647]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:33:58.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:33:58 smithi186 ceph-mon[36647]: Deploying daemon osd.9 on smithi114 2024-05-27T07:33:59.050 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:58 smithi114 ceph-mon[32038]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:33:59.050 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:33:58 smithi114 ceph-mon[32038]: Deploying daemon osd.9 on smithi114 2024-05-27T07:34:00.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:00 smithi163 ceph-mon[36451]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:00.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:00 smithi186 ceph-mon[36647]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:01.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:00 smithi114 ceph-mon[32038]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:02.955 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:02 smithi114 ceph-mon[32038]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:02.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:02 smithi163 ceph-mon[36451]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:02.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:02 smithi186 ceph-mon[36647]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:04.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:04 smithi163 ceph-mon[36451]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:04.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:04.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:04.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:04.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:04 smithi186 ceph-mon[36647]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:04.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:04.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:04.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:05.050 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:04 smithi114 ceph-mon[32038]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:05.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:05.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:05.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:06.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:06 smithi114 ceph-mon[32038]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:06.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:06 smithi163 ceph-mon[36451]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:06.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:06 smithi186 ceph-mon[36647]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:07.416 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 9 on host 'smithi114' 2024-05-27T07:34:07.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:34:07.421+0000 7fa28f7fe700 1 -- 172.21.15.114:0/2970842577 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa298063bf0 con 0x7fa2840b2880 2024-05-27T07:34:07.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:34:07.424+0000 7fa2a046b700 1 -- 172.21.15.114:0/2970842577 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa2840b2880 msgr2=0x7fa2840b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:07.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:34:07.424+0000 7fa2a046b700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa2840b2880 0x7fa2840b4d90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa288005950 tx=0x7fa288005ed0 comp rx=0 tx=0).stop 2024-05-27T07:34:07.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:34:07.424+0000 7fa2a046b700 1 -- 172.21.15.114:0/2970842577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa298104e20 msgr2=0x7fa29819a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:07.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:34:07.424+0000 7fa2a046b700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa298104e20 0x7fa29819a900 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa294006010 tx=0x7fa2940057c0 comp rx=0 tx=0).stop 2024-05-27T07:34:07.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:34:07.424+0000 7fa2a046b700 1 -- 172.21.15.114:0/2970842577 shutdown_connections 2024-05-27T07:34:07.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:34:07.424+0000 7fa2a046b700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa2840b2880 0x7fa2840b4d90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:07.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:34:07.424+0000 7fa2a046b700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa29810a0e0 0x7fa2981a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:07.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:34:07.424+0000 7fa2a046b700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa298104e20 0x7fa29819a900 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:07.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:34:07.424+0000 7fa2a046b700 1 --2- 172.21.15.114:0/2970842577 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa298107780 0x7fa29819ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:07.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:34:07.424+0000 7fa2a046b700 1 -- 172.21.15.114:0/2970842577 >> 172.21.15.114:0/2970842577 conn(0x7fa2980fe860 msgr2=0x7fa298106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:07.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:34:07.424+0000 7fa2a046b700 1 -- 172.21.15.114:0/2970842577 shutdown_connections 2024-05-27T07:34:07.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:34:07.424+0000 7fa2a046b700 1 -- 172.21.15.114:0/2970842577 wait complete. 2024-05-27T07:34:08.175 DEBUG:teuthology.orchestra.run.smithi114:osd.9> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.9.service 2024-05-27T07:34:08.178 INFO:tasks.cephadm:Deploying osd.10 on smithi163 with /dev/vg_nvme/lv_1... 2024-05-27T07:34:08.178 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T07:34:08.425 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:34:08.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:08 smithi114 ceph-mon[32038]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:08.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:08.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:08.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:08.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:08.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:08 smithi114 ceph-mon[32038]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T07:34:08.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:08 smithi163 ceph-mon[36451]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:08.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:08.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:08.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:08.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:08.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:08 smithi163 ceph-mon[36451]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T07:34:08.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:08 smithi186 ceph-mon[36647]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:08.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:08.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:08.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:08.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:08.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:08 smithi186 ceph-mon[36647]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T07:34:09.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:09 smithi163 ceph-mon[36451]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T07:34:09.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:09 smithi163 ceph-mon[36451]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T07:34:09.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:09 smithi163 ceph-mon[36451]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-05-27T07:34:09.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:09 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:09.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:09 smithi186 ceph-mon[36647]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T07:34:09.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:09 smithi186 ceph-mon[36647]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T07:34:09.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:09 smithi186 ceph-mon[36647]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-05-27T07:34:09.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:09 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:09.751 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:09 smithi114 ceph-mon[32038]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T07:34:09.751 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:09 smithi114 ceph-mon[32038]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T07:34:09.751 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:09 smithi114 ceph-mon[32038]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-05-27T07:34:09.751 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:09 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:10.050 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 07:34:09 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T07:34:09.943+0000 7f0d8dff3700 -1 osd.9 0 waiting for initial osdmap 2024-05-27T07:34:10.051 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 07:34:09 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T07:34:09.958+0000 7f0d85da0700 -1 osd.9 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:34:10.520 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T07:34:10.520 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T07:34:10.520 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-05-27T07:34:10.520 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-05-27T07:34:10.520 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115781 s, 906 MB/s 2024-05-27T07:34:10.520 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-05-27T07:34:10.713 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:10 smithi114 ceph-mon[32038]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:10.713 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:10 smithi114 ceph-mon[32038]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-05-27T07:34:10.713 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:10 smithi114 ceph-mon[32038]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T07:34:10.713 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:10 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:10.713 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:10 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:10.714 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:10 smithi114 ceph-mon[32038]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' 2024-05-27T07:34:10.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:10 smithi163 ceph-mon[36451]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:10.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:10 smithi163 ceph-mon[36451]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-05-27T07:34:10.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:10 smithi163 ceph-mon[36451]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T07:34:10.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:10 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:10.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:10 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:10.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:10 smithi163 ceph-mon[36451]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' 2024-05-27T07:34:10.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:10 smithi186 ceph-mon[36647]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:10.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:10 smithi186 ceph-mon[36647]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-05-27T07:34:10.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:10 smithi186 ceph-mon[36647]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T07:34:10.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:10 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:10.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:10 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:10.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:10 smithi186 ceph-mon[36647]: from='osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915]' entity='osd.9' 2024-05-27T07:34:11.317 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_1 2024-05-27T07:34:11.568 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:34:11.690 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: purged_snaps scrub starts 2024-05-27T07:34:11.690 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: purged_snaps scrub ok 2024-05-27T07:34:11.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:11.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915] boot 2024-05-27T07:34:11.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: osdmap e59: 10 total, 10 up, 10 in 2024-05-27T07:34:11.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:11.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:11.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:11.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:11.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:11.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:11.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:11.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:11.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:11.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:11.692 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:11 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:11.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: purged_snaps scrub starts 2024-05-27T07:34:11.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: purged_snaps scrub ok 2024-05-27T07:34:11.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:11.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915] boot 2024-05-27T07:34:11.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: osdmap e59: 10 total, 10 up, 10 in 2024-05-27T07:34:11.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:11.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:11.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:11.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:11.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:11.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:11.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:11.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:11.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:11.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:11.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:11 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:11.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: purged_snaps scrub starts 2024-05-27T07:34:11.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: purged_snaps scrub ok 2024-05-27T07:34:11.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:11.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: osd.9 [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915] boot 2024-05-27T07:34:11.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: osdmap e59: 10 total, 10 up, 10 in 2024-05-27T07:34:11.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:34:11.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:11.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:11.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:11.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:11.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:11.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:11.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:11.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:11.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:11.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:11 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:12.714 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:12 smithi163 ceph-mon[36451]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:12.714 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:12 smithi163 ceph-mon[36451]: Detected new or changed devices on smithi114 2024-05-27T07:34:12.714 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:12 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi114 to 4272M 2024-05-27T07:34:12.714 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:12 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:12.714 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:12 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:12.714 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:12 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:12.715 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:12 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:12.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:12 smithi186 ceph-mon[36647]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:12.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:12 smithi186 ceph-mon[36647]: Detected new or changed devices on smithi114 2024-05-27T07:34:12.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:12 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi114 to 4272M 2024-05-27T07:34:12.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:12 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:12.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:12 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:12.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:12 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:12.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:12 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:12.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:12 smithi114 ceph-mon[32038]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:34:12.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:12 smithi114 ceph-mon[32038]: Detected new or changed devices on smithi114 2024-05-27T07:34:12.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:12 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi114 to 4272M 2024-05-27T07:34:12.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:12 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:12.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:12 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:12.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:12 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:12.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:12 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:13.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.280+0000 7f374170c700 1 -- 172.21.15.163:0/711356505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c107780 msgr2=0x7f373c109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:13.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.280+0000 7f374170c700 1 --2- 172.21.15.163:0/711356505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c107780 0x7f373c109ba0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3724009990 tx=0x7f3724009ca0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.280+0000 7f374170c700 1 -- 172.21.15.163:0/711356505 shutdown_connections 2024-05-27T07:34:13.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.280+0000 7f374170c700 1 --2- 172.21.15.163:0/711356505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f373c10a0e0 0x7f373c10c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.280+0000 7f374170c700 1 --2- 172.21.15.163:0/711356505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c104e20 0x7f373c107240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.280+0000 7f374170c700 1 --2- 172.21.15.163:0/711356505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c107780 0x7f373c109ba0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.280+0000 7f374170c700 1 -- 172.21.15.163:0/711356505 >> 172.21.15.163:0/711356505 conn(0x7f373c0fe860 msgr2=0x7f373c100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:13.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.280+0000 7f374170c700 1 -- 172.21.15.163:0/711356505 shutdown_connections 2024-05-27T07:34:13.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.280+0000 7f374170c700 1 -- 172.21.15.163:0/711356505 wait complete. 2024-05-27T07:34:13.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.281+0000 7f374170c700 1 Processor -- start 2024-05-27T07:34:13.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.281+0000 7f374170c700 1 -- start start 2024-05-27T07:34:13.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.281+0000 7f374170c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c104e20 0x7f373c19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:13.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.281+0000 7f374170c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f373c107780 0x7f373c19ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:13.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.281+0000 7f374170c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c10a0e0 0x7f373c1a1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:13.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.281+0000 7f374170c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f373c10fd80 con 0x7f373c107780 2024-05-27T07:34:13.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.281+0000 7f374170c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f373c10fc00 con 0x7f373c10a0e0 2024-05-27T07:34:13.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.281+0000 7f374170c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f373c10ff00 con 0x7f373c104e20 2024-05-27T07:34:13.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.281+0000 7f373b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c10a0e0 0x7f373c1a1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:13.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f373b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c10a0e0 0x7f373c1a1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:32988/0 (socket says 172.21.15.163:32988) 2024-05-27T07:34:13.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f373b7fe700 1 -- 172.21.15.163:0/1909632234 learned_addr learned my addr 172.21.15.163:0/1909632234 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-27T07:34:13.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f373affd700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c104e20 0x7f373c19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:13.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f37327fc700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f373c107780 0x7f373c19ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:13.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f373b7fe700 1 -- 172.21.15.163:0/1909632234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c104e20 msgr2=0x7f373c19a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:13.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f373b7fe700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c104e20 0x7f373c19a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f373b7fe700 1 -- 172.21.15.163:0/1909632234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f373c107780 msgr2=0x7f373c19ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:13.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f373b7fe700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f373c107780 0x7f373c19ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f373b7fe700 1 -- 172.21.15.163:0/1909632234 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373c1a2450 con 0x7f373c10a0e0 2024-05-27T07:34:13.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f373b7fe700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c10a0e0 0x7f373c1a1dd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3734007ee0 tx=0x7f373400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:13.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f3738ff9700 1 -- 172.21.15.163:0/1909632234 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373400dbd0 con 0x7f373c10a0e0 2024-05-27T07:34:13.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f3738ff9700 1 -- 172.21.15.163:0/1909632234 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f373400dd70 con 0x7f373c10a0e0 2024-05-27T07:34:13.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f374170c700 1 -- 172.21.15.163:0/1909632234 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f373c1a2740 con 0x7f373c10a0e0 2024-05-27T07:34:13.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f3738ff9700 1 -- 172.21.15.163:0/1909632234 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3734005990 con 0x7f373c10a0e0 2024-05-27T07:34:13.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.282+0000 7f374170c700 1 -- 172.21.15.163:0/1909632234 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f373c1a2c30 con 0x7f373c10a0e0 2024-05-27T07:34:13.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.283+0000 7f374170c700 1 -- 172.21.15.163:0/1909632234 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f373c0512b0 con 0x7f373c10a0e0 2024-05-27T07:34:13.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.285+0000 7f3738ff9700 1 -- 172.21.15.163:0/1909632234 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f373401a020 con 0x7f373c10a0e0 2024-05-27T07:34:13.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.285+0000 7f3738ff9700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f37280b2880 0x7f37280b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:13.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.285+0000 7f3738ff9700 1 -- 172.21.15.163:0/1909632234 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6261+0+0 (secure 0 0 0) 0x7f37340e1850 con 0x7f373c10a0e0 2024-05-27T07:34:13.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.286+0000 7f373affd700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f37280b2880 0x7f37280b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:13.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.286+0000 7f373affd700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f37280b2880 0x7f37280b4d90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f372c005950 tx=0x7f372c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:13.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.289+0000 7f3738ff9700 1 -- 172.21.15.163:0/1909632234 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3734010070 con 0x7f373c10a0e0 2024-05-27T07:34:13.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:13.501+0000 7f374170c700 1 -- 172.21.15.163:0/1909632234 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f373c063bf0 con 0x7f37280b2880 2024-05-27T07:34:13.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:13 smithi163 ceph-mon[36451]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T07:34:13.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:13 smithi114 ceph-mon[32038]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T07:34:13.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:13 smithi186 ceph-mon[36647]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T07:34:14.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:14 smithi163 ceph-mon[36451]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:14.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:14 smithi163 ceph-mon[36451]: from='client.24311 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:34:14.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:14 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:34:14.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:14 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:34:14.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:14 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:14.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:14 smithi114 ceph-mon[32038]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:14.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:14 smithi114 ceph-mon[32038]: from='client.24311 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:34:14.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:34:14.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:34:14.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:14.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:14 smithi186 ceph-mon[36647]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:14.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:14 smithi186 ceph-mon[36647]: from='client.24311 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:34:14.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:14 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:34:14.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:14 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:34:14.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:14 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:16.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:16 smithi114 ceph-mon[32038]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:16.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:16 smithi114 ceph-mon[32038]: from='client.? 172.21.15.163:0/2908645070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "825ee9ad-c4c8-4199-98df-76444163633d"}]: dispatch 2024-05-27T07:34:16.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:16 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "825ee9ad-c4c8-4199-98df-76444163633d"}]: dispatch 2024-05-27T07:34:16.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:16 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "825ee9ad-c4c8-4199-98df-76444163633d"}]': finished 2024-05-27T07:34:16.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:16 smithi114 ceph-mon[32038]: osdmap e61: 11 total, 10 up, 11 in 2024-05-27T07:34:16.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:16 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:16.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:16 smithi163 ceph-mon[36451]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:16.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:16 smithi163 ceph-mon[36451]: from='client.? 172.21.15.163:0/2908645070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "825ee9ad-c4c8-4199-98df-76444163633d"}]: dispatch 2024-05-27T07:34:16.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:16 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "825ee9ad-c4c8-4199-98df-76444163633d"}]: dispatch 2024-05-27T07:34:16.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:16 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "825ee9ad-c4c8-4199-98df-76444163633d"}]': finished 2024-05-27T07:34:16.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:16 smithi163 ceph-mon[36451]: osdmap e61: 11 total, 10 up, 11 in 2024-05-27T07:34:16.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:16 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:16.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:16 smithi186 ceph-mon[36647]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:16.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:16 smithi186 ceph-mon[36647]: from='client.? 172.21.15.163:0/2908645070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "825ee9ad-c4c8-4199-98df-76444163633d"}]: dispatch 2024-05-27T07:34:16.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:16 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "825ee9ad-c4c8-4199-98df-76444163633d"}]: dispatch 2024-05-27T07:34:16.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:16 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "825ee9ad-c4c8-4199-98df-76444163633d"}]': finished 2024-05-27T07:34:16.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:16 smithi186 ceph-mon[36647]: osdmap e61: 11 total, 10 up, 11 in 2024-05-27T07:34:16.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:16 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:17.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:17 smithi114 ceph-mon[32038]: from='client.? 172.21.15.163:0/2228895345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:34:17.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:17 smithi163 ceph-mon[36451]: from='client.? 172.21.15.163:0/2228895345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:34:17.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:17 smithi186 ceph-mon[36647]: from='client.? 172.21.15.163:0/2228895345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:34:18.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:18 smithi114 ceph-mon[32038]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:18.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:18 smithi163 ceph-mon[36451]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:18.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:18 smithi186 ceph-mon[36647]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:20.711 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:20 smithi163 ceph-mon[36451]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:20.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:20 smithi114 ceph-mon[32038]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:20.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:20 smithi186 ceph-mon[36647]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:22.697 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:22 smithi163 ceph-mon[36451]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:22.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:22 smithi114 ceph-mon[32038]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:22.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:22 smithi186 ceph-mon[36647]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:24.746 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:24 smithi163 ceph-mon[36451]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:24.746 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:24 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T07:34:24.746 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:24 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:24.746 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:24 smithi163 ceph-mon[36451]: Deploying daemon osd.10 on smithi163 2024-05-27T07:34:24.775 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:24 smithi186 ceph-mon[36647]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:24.775 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:24 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T07:34:24.775 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:24 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:24.775 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:24 smithi186 ceph-mon[36647]: Deploying daemon osd.10 on smithi163 2024-05-27T07:34:24.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:24 smithi114 ceph-mon[32038]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:24.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:24 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T07:34:24.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:24 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:24.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:24 smithi114 ceph-mon[32038]: Deploying daemon osd.10 on smithi163 2024-05-27T07:34:26.608 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:26 smithi163 ceph-mon[36451]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:26.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:26 smithi114 ceph-mon[32038]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:26.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:26 smithi186 ceph-mon[36647]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:28.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:28 smithi114 ceph-mon[32038]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:28.899 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:28 smithi163 ceph-mon[36451]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:28.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:28 smithi186 ceph-mon[36647]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:30.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:30 smithi163 ceph-mon[36451]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:30.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:30 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:30.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:30 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:30.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:30 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:31.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:30 smithi186 ceph-mon[36647]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:31.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:30 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:31.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:30 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:31.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:30 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:31.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:30 smithi114 ceph-mon[32038]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:31.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:30 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:31.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:30 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:31.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:30 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:33.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:32 smithi163 ceph-mon[36451]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:33.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:32 smithi186 ceph-mon[36647]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:33.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:32 smithi114 ceph-mon[32038]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:34.272 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 10 on host 'smithi163' 2024-05-27T07:34:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:34.275+0000 7f3738ff9700 1 -- 172.21.15.163:0/1909632234 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f373c063bf0 con 0x7f37280b2880 2024-05-27T07:34:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:34.278+0000 7f374170c700 1 -- 172.21.15.163:0/1909632234 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f37280b2880 msgr2=0x7f37280b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:34.278+0000 7f374170c700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f37280b2880 0x7f37280b4d90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f372c005950 tx=0x7f372c00b410 comp rx=0 tx=0).stop 2024-05-27T07:34:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:34.278+0000 7f374170c700 1 -- 172.21.15.163:0/1909632234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c10a0e0 msgr2=0x7f373c1a1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:34.278+0000 7f374170c700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c10a0e0 0x7f373c1a1dd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3734007ee0 tx=0x7f373400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:34:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:34.278+0000 7f374170c700 1 -- 172.21.15.163:0/1909632234 shutdown_connections 2024-05-27T07:34:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:34.278+0000 7f374170c700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c10a0e0 0x7f373c1a1dd0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:34.278+0000 7f374170c700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f37280b2880 0x7f37280b4d90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:34.278+0000 7f374170c700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c104e20 0x7f373c19a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:34.278+0000 7f374170c700 1 --2- 172.21.15.163:0/1909632234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f373c107780 0x7f373c19ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:34.278+0000 7f374170c700 1 -- 172.21.15.163:0/1909632234 >> 172.21.15.163:0/1909632234 conn(0x7f373c0fe860 msgr2=0x7f373c105990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:34.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:34.278+0000 7f374170c700 1 -- 172.21.15.163:0/1909632234 shutdown_connections 2024-05-27T07:34:34.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-27T07:34:34.278+0000 7f374170c700 1 -- 172.21.15.163:0/1909632234 wait complete. 2024-05-27T07:34:34.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:34 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:34.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:34 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:34.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:34 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:34.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:34 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:34.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:34 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:34.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:34 smithi163 ceph-mon[36451]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:34.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:34 smithi163 ceph-mon[36451]: from='osd.10 [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:34:34.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:34 smithi163 ceph-mon[36451]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:34:34.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:34 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:34.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:34 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:34.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:34 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:34.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:34 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:34.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:34 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:34.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:34 smithi114 ceph-mon[32038]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:34.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:34 smithi114 ceph-mon[32038]: from='osd.10 [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:34:34.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:34 smithi114 ceph-mon[32038]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:34:34.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:34 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:34.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:34 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:34.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:34 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:34.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:34 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:34.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:34 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:34.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:34 smithi186 ceph-mon[36647]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:34.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:34 smithi186 ceph-mon[36647]: from='osd.10 [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:34:34.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:34 smithi186 ceph-mon[36647]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:34:35.077 DEBUG:teuthology.orchestra.run.smithi163:osd.10> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.10.service 2024-05-27T07:34:35.079 INFO:tasks.cephadm:Deploying osd.11 on smithi186 with /dev/vg_nvme/lv_1... 2024-05-27T07:34:35.080 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T07:34:35.309 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:35 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:35.309 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:35 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:35.309 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:35 smithi163 ceph-mon[36451]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T07:34:35.309 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:35 smithi163 ceph-mon[36451]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T07:34:35.309 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:35 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:35.309 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:35 smithi163 ceph-mon[36451]: from='osd.10 [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:34:35.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:35 smithi163 ceph-mon[36451]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:34:35.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:35 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:35.310 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 07:34:35 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T07:34:35.263+0000 7f39194fc700 -1 osd.10 0 waiting for initial osdmap 2024-05-27T07:34:35.310 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 07:34:35 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T07:34:35.275+0000 7f39152b1700 -1 osd.10 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:34:35.328 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:34:35.450 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:35 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:35.450 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:35 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:35.451 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:35 smithi186 ceph-mon[36647]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T07:34:35.451 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:35 smithi186 ceph-mon[36647]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T07:34:35.451 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:35 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:35.451 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:35 smithi186 ceph-mon[36647]: from='osd.10 [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:34:35.451 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:35 smithi186 ceph-mon[36647]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:34:35.451 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:35 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:35.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:35 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:35.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:35 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:35.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:35 smithi114 ceph-mon[32038]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T07:34:35.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:35 smithi114 ceph-mon[32038]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T07:34:35.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:35 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:35.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:35 smithi114 ceph-mon[32038]: from='osd.10 [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:34:35.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:35 smithi114 ceph-mon[32038]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-27T07:34:35.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:35 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:36.543 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:36 smithi163 ceph-mon[36451]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-27T07:34:36.543 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:36 smithi163 ceph-mon[36451]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T07:34:36.543 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:36.543 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:36.543 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:36 smithi163 ceph-mon[36451]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:36.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:36 smithi114 ceph-mon[32038]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-27T07:34:36.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:36 smithi114 ceph-mon[32038]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T07:34:36.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:36.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:36.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:36 smithi114 ceph-mon[32038]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:36.616 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:36 smithi186 ceph-mon[36647]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-27T07:34:36.616 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:36 smithi186 ceph-mon[36647]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T07:34:36.616 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:36.616 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:36.616 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:36 smithi186 ceph-mon[36647]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:34:37.502 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:37 smithi186 ceph-mon[36647]: purged_snaps scrub starts 2024-05-27T07:34:37.502 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:37 smithi186 ceph-mon[36647]: purged_snaps scrub ok 2024-05-27T07:34:37.502 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:37 smithi186 ceph-mon[36647]: osd.10 [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938] boot 2024-05-27T07:34:37.502 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:37 smithi186 ceph-mon[36647]: osdmap e64: 11 total, 11 up, 11 in 2024-05-27T07:34:37.502 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:37 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:37.517 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:37 smithi163 ceph-mon[36451]: purged_snaps scrub starts 2024-05-27T07:34:37.518 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:37 smithi163 ceph-mon[36451]: purged_snaps scrub ok 2024-05-27T07:34:37.518 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:37 smithi163 ceph-mon[36451]: osd.10 [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938] boot 2024-05-27T07:34:37.518 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:37 smithi163 ceph-mon[36451]: osdmap e64: 11 total, 11 up, 11 in 2024-05-27T07:34:37.518 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:37 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:37.539 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T07:34:37.539 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T07:34:37.539 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-05-27T07:34:37.539 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-05-27T07:34:37.539 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134339 s, 781 MB/s 2024-05-27T07:34:37.539 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-05-27T07:34:37.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:37 smithi114 ceph-mon[32038]: purged_snaps scrub starts 2024-05-27T07:34:37.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:37 smithi114 ceph-mon[32038]: purged_snaps scrub ok 2024-05-27T07:34:37.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:37 smithi114 ceph-mon[32038]: osd.10 [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938] boot 2024-05-27T07:34:37.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:37 smithi114 ceph-mon[32038]: osdmap e64: 11 total, 11 up, 11 in 2024-05-27T07:34:37.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:37 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:34:38.361 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_1 2024-05-27T07:34:38.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:38 smithi114 ceph-mon[32038]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T07:34:38.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:38 smithi114 ceph-mon[32038]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:38.612 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:34:38.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:38 smithi163 ceph-mon[36451]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T07:34:38.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:38 smithi163 ceph-mon[36451]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:38.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:38 smithi186 ceph-mon[36647]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T07:34:38.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:38 smithi186 ceph-mon[36647]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:39.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:39 smithi114 ceph-mon[32038]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T07:34:39.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:39 smithi114 ceph-mon[32038]: Detected new or changed devices on smithi163 2024-05-27T07:34:39.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:39 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:39.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:39 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:39.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:39 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:39.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:39 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:39.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:39 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:39.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:39 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:39.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:39 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi163 to 4272M 2024-05-27T07:34:39.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:39 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:39.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:39 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:39.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:39 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:39.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:39 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:39.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:39 smithi163 ceph-mon[36451]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T07:34:39.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:39 smithi163 ceph-mon[36451]: Detected new or changed devices on smithi163 2024-05-27T07:34:39.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:39 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:39.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:39 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:39.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:39 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:39.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:39 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:39.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:39 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:39.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:39 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:39.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:39 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi163 to 4272M 2024-05-27T07:34:39.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:39 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:39.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:39 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:39.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:39 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:39.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:39 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:39.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:39 smithi186 ceph-mon[36647]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T07:34:39.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:39 smithi186 ceph-mon[36647]: Detected new or changed devices on smithi163 2024-05-27T07:34:39.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:39 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:39.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:39 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:39.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:39 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:39.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:39 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:39.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:39 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:39.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:39 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:34:39.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:39 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi163 to 4272M 2024-05-27T07:34:39.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:39 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:39.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:39 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:39.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:39 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:39.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:39 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:40.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.383+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/104145750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e78102690 msgr2=0x7f6e78102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:40.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.383+0000 7f6e7e8e5700 1 --2- 172.21.15.186:0/104145750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e78102690 0x7f6e78102aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6e68009990 tx=0x7f6e68009ca0 comp rx=0 tx=0).stop 2024-05-27T07:34:40.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.383+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/104145750 shutdown_connections 2024-05-27T07:34:40.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.383+0000 7f6e7e8e5700 1 --2- 172.21.15.186:0/104145750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e7810a170 0x7f6e7810c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:40.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.383+0000 7f6e7e8e5700 1 --2- 172.21.15.186:0/104145750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e78102690 0x7f6e78102aa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:40.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.383+0000 7f6e7e8e5700 1 --2- 172.21.15.186:0/104145750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e78107810 0x7f6e78109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:40.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.383+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/104145750 >> 172.21.15.186:0/104145750 conn(0x7f6e780fe010 msgr2=0x7f6e781004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:40.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.383+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/104145750 shutdown_connections 2024-05-27T07:34:40.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.383+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/104145750 wait complete. 2024-05-27T07:34:40.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.383+0000 7f6e7e8e5700 1 Processor -- start 2024-05-27T07:34:40.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.383+0000 7f6e7e8e5700 1 -- start start 2024-05-27T07:34:40.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e7e8e5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e78102690 0x7f6e7819a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:40.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e7e8e5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e78107810 0x7f6e7819ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:40.380 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:40 smithi186 ceph-mon[36647]: pgmap v234: 1 pgs: 1 peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:40.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e7e8e5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e7810a170 0x7f6e781a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:40.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e7e8e5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7810fdd0 con 0x7f6e78107810 2024-05-27T07:34:40.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e7e8e5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7810fc50 con 0x7f6e78102690 2024-05-27T07:34:40.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e7e8e5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7810ff50 con 0x7f6e7810a170 2024-05-27T07:34:40.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e7ce82700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e7810a170 0x7f6e781a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:40.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e7ce82700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e7810a170 0x7f6e781a1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:57418/0 (socket says 172.21.15.186:57418) 2024-05-27T07:34:40.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e7ce82700 1 -- 172.21.15.186:0/3066478150 learned_addr learned my addr 172.21.15.186:0/3066478150 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-27T07:34:40.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e77fff700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e78102690 0x7f6e7819a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:40.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e777fe700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e78107810 0x7f6e7819ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:40.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e7ce82700 1 -- 172.21.15.186:0/3066478150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e78102690 msgr2=0x7f6e7819a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:40.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e7ce82700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e78102690 0x7f6e7819a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:40.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e7ce82700 1 -- 172.21.15.186:0/3066478150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e78107810 msgr2=0x7f6e7819ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:40.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.384+0000 7f6e7ce82700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e78107810 0x7f6e7819ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:40.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.385+0000 7f6e7ce82700 1 -- 172.21.15.186:0/3066478150 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e781a23d0 con 0x7f6e7810a170 2024-05-27T07:34:40.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.385+0000 7f6e777fe700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e78107810 0x7f6e7819ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:40.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.385+0000 7f6e7ce82700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e7810a170 0x7f6e781a1de0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e70007ee0 tx=0x7f6e7000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:40.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.385+0000 7f6e757fa700 1 -- 172.21.15.186:0/3066478150 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7000dbd0 con 0x7f6e7810a170 2024-05-27T07:34:40.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.385+0000 7f6e757fa700 1 -- 172.21.15.186:0/3066478150 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e7000dd70 con 0x7f6e7810a170 2024-05-27T07:34:40.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.385+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/3066478150 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e781a26c0 con 0x7f6e7810a170 2024-05-27T07:34:40.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.385+0000 7f6e757fa700 1 -- 172.21.15.186:0/3066478150 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e70005990 con 0x7f6e7810a170 2024-05-27T07:34:40.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.385+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/3066478150 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e781a2bb0 con 0x7f6e7810a170 2024-05-27T07:34:40.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.386+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/3066478150 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e780512b0 con 0x7f6e7810a170 2024-05-27T07:34:40.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.388+0000 7f6e757fa700 1 -- 172.21.15.186:0/3066478150 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f6e7001a020 con 0x7f6e7810a170 2024-05-27T07:34:40.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.389+0000 7f6e757fa700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f6e600b2880 0x7f6e600b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:40.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.389+0000 7f6e757fa700 1 -- 172.21.15.186:0/3066478150 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 6694+0+0 (secure 0 0 0) 0x7f6e700e1a70 con 0x7f6e7810a170 2024-05-27T07:34:40.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.389+0000 7f6e77fff700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f6e600b2880 0x7f6e600b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:40.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.389+0000 7f6e77fff700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f6e600b2880 0x7f6e600b4d90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6e680038a0 tx=0x7f6e6801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:40.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.393+0000 7f6e757fa700 1 -- 172.21.15.186:0/3066478150 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6e70010070 con 0x7f6e7810a170 2024-05-27T07:34:40.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:40 smithi114 ceph-mon[32038]: pgmap v234: 1 pgs: 1 peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:34:40.600+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/3066478150 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6e78063bf0 con 0x7f6e600b2880 2024-05-27T07:34:40.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:40 smithi163 ceph-mon[36451]: pgmap v234: 1 pgs: 1 peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:41.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:41 smithi186 ceph-mon[36647]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:34:41.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:34:41.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:34:41.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:41 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:41.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:41 smithi114 ceph-mon[32038]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:34:41.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:34:41.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:34:41.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:41 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:41.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:41 smithi163 ceph-mon[36451]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:34:41.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:34:41.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:34:41.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:41 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:42.489 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:42 smithi186 ceph-mon[36647]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:42.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:42 smithi114 ceph-mon[32038]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:42.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:42 smithi163 ceph-mon[36451]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:44.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:44 smithi163 ceph-mon[36451]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-05-27T07:34:44.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:44 smithi163 ceph-mon[36451]: from='client.? 172.21.15.186:0/1765445672' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "920e5b32-7159-48c0-9957-d7cb7db2ac38"}]: dispatch 2024-05-27T07:34:44.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:44 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "920e5b32-7159-48c0-9957-d7cb7db2ac38"}]: dispatch 2024-05-27T07:34:44.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:44 smithi163 ceph-mon[36451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "920e5b32-7159-48c0-9957-d7cb7db2ac38"}]': finished 2024-05-27T07:34:44.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:44 smithi163 ceph-mon[36451]: osdmap e67: 12 total, 11 up, 12 in 2024-05-27T07:34:44.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:34:44.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:44 smithi163 ceph-mon[36451]: from='client.? 172.21.15.186:0/954235868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:34:44.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:44 smithi186 ceph-mon[36647]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-05-27T07:34:44.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:44 smithi186 ceph-mon[36647]: from='client.? 172.21.15.186:0/1765445672' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "920e5b32-7159-48c0-9957-d7cb7db2ac38"}]: dispatch 2024-05-27T07:34:44.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:44 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "920e5b32-7159-48c0-9957-d7cb7db2ac38"}]: dispatch 2024-05-27T07:34:44.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:44 smithi186 ceph-mon[36647]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "920e5b32-7159-48c0-9957-d7cb7db2ac38"}]': finished 2024-05-27T07:34:44.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:44 smithi186 ceph-mon[36647]: osdmap e67: 12 total, 11 up, 12 in 2024-05-27T07:34:44.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:34:44.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:44 smithi186 ceph-mon[36647]: from='client.? 172.21.15.186:0/954235868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:34:44.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:44 smithi114 ceph-mon[32038]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-05-27T07:34:44.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:44 smithi114 ceph-mon[32038]: from='client.? 172.21.15.186:0/1765445672' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "920e5b32-7159-48c0-9957-d7cb7db2ac38"}]: dispatch 2024-05-27T07:34:44.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:44 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "920e5b32-7159-48c0-9957-d7cb7db2ac38"}]: dispatch 2024-05-27T07:34:44.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:44 smithi114 ceph-mon[32038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "920e5b32-7159-48c0-9957-d7cb7db2ac38"}]': finished 2024-05-27T07:34:44.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:44 smithi114 ceph-mon[32038]: osdmap e67: 12 total, 11 up, 12 in 2024-05-27T07:34:44.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:34:44.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:44 smithi114 ceph-mon[32038]: from='client.? 172.21.15.186:0/954235868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:34:46.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:46 smithi163 ceph-mon[36451]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:34:46.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:46 smithi186 ceph-mon[36647]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:34:46.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:46 smithi114 ceph-mon[32038]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:34:48.691 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:48 smithi186 ceph-mon[36647]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-27T07:34:48.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:48 smithi163 ceph-mon[36451]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-27T07:34:48.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:48 smithi114 ceph-mon[32038]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-27T07:34:50.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:50 smithi163 ceph-mon[36451]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:34:50.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:50 smithi186 ceph-mon[36647]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:34:50.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:50 smithi114 ceph-mon[32038]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:34:51.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:51 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T07:34:51.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:51 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:51.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:51 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T07:34:51.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:51 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:51.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:51 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T07:34:51.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:51 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:52.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:52 smithi163 ceph-mon[36451]: Deploying daemon osd.11 on smithi186 2024-05-27T07:34:52.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:52 smithi163 ceph-mon[36451]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:34:52.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:52 smithi186 ceph-mon[36647]: Deploying daemon osd.11 on smithi186 2024-05-27T07:34:52.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:52 smithi186 ceph-mon[36647]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:34:52.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:52 smithi114 ceph-mon[32038]: Deploying daemon osd.11 on smithi186 2024-05-27T07:34:52.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:52 smithi114 ceph-mon[32038]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:34:54.622 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:54 smithi186 ceph-mon[36647]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:54.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:54 smithi163 ceph-mon[36451]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:54.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:54 smithi114 ceph-mon[32038]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:56.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:56 smithi186 ceph-mon[36647]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:56.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:56 smithi163 ceph-mon[36451]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:56.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:56 smithi114 ceph-mon[32038]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:58.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:58 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:58.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:58 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:58.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:58 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:58.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:34:58 smithi163 ceph-mon[36451]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:58.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:58 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:58.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:58 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:58.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:58 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:58.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:34:58 smithi186 ceph-mon[36647]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:34:58.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:58 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:58.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:58 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:34:58.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:58 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:58.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:34:58 smithi114 ceph-mon[32038]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:35:00.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:00 smithi163 ceph-mon[36451]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:35:00.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:00 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:00.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:00 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:00.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:00 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:00.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:00 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:00.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:00 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:00.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:00 smithi186 ceph-mon[36647]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:35:00.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:00 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:00.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:00 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:00.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:00 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:00.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:00 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:00.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:00 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:00.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:00 smithi114 ceph-mon[32038]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:35:00.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:00 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:00.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:00 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:00.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:00 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:00.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:00 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:00.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:00 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:01.124+0000 7f6e757fa700 1 -- 172.21.15.186:0/3066478150 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f6e78063bf0 con 0x7f6e600b2880 2024-05-27T07:35:01.119 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 11 on host 'smithi186' 2024-05-27T07:35:01.121 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:01.126+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/3066478150 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f6e600b2880 msgr2=0x7f6e600b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:01.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:01.126+0000 7f6e7e8e5700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f6e600b2880 0x7f6e600b4d90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6e680038a0 tx=0x7f6e6801a040 comp rx=0 tx=0).stop 2024-05-27T07:35:01.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:01.126+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/3066478150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e7810a170 msgr2=0x7f6e781a1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:01.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:01.126+0000 7f6e7e8e5700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e7810a170 0x7f6e781a1de0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e70007ee0 tx=0x7f6e7000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:35:01.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:01.126+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/3066478150 shutdown_connections 2024-05-27T07:35:01.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:01.126+0000 7f6e7e8e5700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f6e600b2880 0x7f6e600b4d90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:01.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:01.126+0000 7f6e7e8e5700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e7810a170 0x7f6e781a1de0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:01.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:01.126+0000 7f6e7e8e5700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e78102690 0x7f6e7819a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:01.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:01.126+0000 7f6e7e8e5700 1 --2- 172.21.15.186:0/3066478150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e78107810 0x7f6e7819ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:01.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:01.126+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/3066478150 >> 172.21.15.186:0/3066478150 conn(0x7f6e780fe010 msgr2=0x7f6e78108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:01.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:01.126+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/3066478150 shutdown_connections 2024-05-27T07:35:01.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:01.127+0000 7f6e7e8e5700 1 -- 172.21.15.186:0/3066478150 wait complete. 2024-05-27T07:35:01.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:01 smithi163 ceph-mon[36451]: from='osd.11 [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:35:01.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:01 smithi163 ceph-mon[36451]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:35:01.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:01 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:01.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:01 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:01.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:01 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:01.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:01 smithi186 ceph-mon[36647]: from='osd.11 [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:35:01.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:01 smithi186 ceph-mon[36647]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:35:01.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:01 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:01.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:01 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:01.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:01 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:01.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:01 smithi114 ceph-mon[32038]: from='osd.11 [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:35:01.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:01 smithi114 ceph-mon[32038]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:35:01.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:01 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:01.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:01 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:01.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:01 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:01.811 DEBUG:teuthology.orchestra.run.smithi186:osd.11> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.11.service 2024-05-27T07:35:01.813 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-27T07:35:01.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-05-27T07:35:02.061 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:02.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:02 smithi114 ceph-mon[32038]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:35:02.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:02 smithi114 ceph-mon[32038]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T07:35:02.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:02 smithi114 ceph-mon[32038]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T07:35:02.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:02 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:02.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:02 smithi114 ceph-mon[32038]: from='osd.11 [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:35:02.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:02 smithi114 ceph-mon[32038]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:35:02.718 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:02 smithi186 ceph-mon[36647]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:35:02.719 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:02 smithi186 ceph-mon[36647]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T07:35:02.719 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:02 smithi186 ceph-mon[36647]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T07:35:02.719 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:02 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:02.719 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:02 smithi186 ceph-mon[36647]: from='osd.11 [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:35:02.719 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:02 smithi186 ceph-mon[36647]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:35:02.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:02 smithi163 ceph-mon[36451]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:35:02.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:02 smithi163 ceph-mon[36451]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T07:35:02.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:02 smithi163 ceph-mon[36451]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T07:35:02.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:02 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:02.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:02 smithi163 ceph-mon[36451]: from='osd.11 [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:35:02.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:02 smithi163 ceph-mon[36451]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-27T07:35:02.973 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 07:35:02 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T07:35:02.877+0000 7f827c148700 -1 osd.11 0 waiting for initial osdmap 2024-05-27T07:35:02.974 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 07:35:02 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T07:35:02.892+0000 7f8273ef5700 -1 osd.11 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:35:03.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:03 smithi114 ceph-mon[32038]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-27T07:35:03.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:03 smithi114 ceph-mon[32038]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T07:35:03.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:03 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:03.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:03 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:03.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:03 smithi114 ceph-mon[32038]: from='osd.11 ' entity='osd.11' 2024-05-27T07:35:03.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:03 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:03.720 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:03 smithi163 ceph-mon[36451]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-27T07:35:03.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:03 smithi163 ceph-mon[36451]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T07:35:03.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:03 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:03.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:03 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:03.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:03 smithi163 ceph-mon[36451]: from='osd.11 ' entity='osd.11' 2024-05-27T07:35:03.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:03 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:03.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:03 smithi186 ceph-mon[36647]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-27T07:35:03.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:03 smithi186 ceph-mon[36647]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T07:35:03.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:03 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:03.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:03 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:03.723 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:03 smithi186 ceph-mon[36647]: from='osd.11 ' entity='osd.11' 2024-05-27T07:35:03.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:03 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:03.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.861+0000 7f124405b700 1 -- 172.21.15.114:0/1250459729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123c105460 msgr2=0x7f123c105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:03.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.861+0000 7f124405b700 1 --2- 172.21.15.114:0/1250459729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123c105460 0x7f123c105870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f123000b0a0 tx=0x7f123000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.861+0000 7f124405b700 1 -- 172.21.15.114:0/1250459729 shutdown_connections 2024-05-27T07:35:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.861+0000 7f124405b700 1 --2- 172.21.15.114:0/1250459729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f123c105e00 0x7f123c10c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.861+0000 7f124405b700 1 --2- 172.21.15.114:0/1250459729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f123c104330 0x7f123c104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.861+0000 7f124405b700 1 --2- 172.21.15.114:0/1250459729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123c105460 0x7f123c105870 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.861+0000 7f124405b700 1 -- 172.21.15.114:0/1250459729 >> 172.21.15.114:0/1250459729 conn(0x7f123c0ff9b0 msgr2=0x7f123c101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.861+0000 7f124405b700 1 -- 172.21.15.114:0/1250459729 shutdown_connections 2024-05-27T07:35:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.861+0000 7f124405b700 1 -- 172.21.15.114:0/1250459729 wait complete. 2024-05-27T07:35:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.862+0000 7f124405b700 1 Processor -- start 2024-05-27T07:35:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.862+0000 7f124405b700 1 -- start start 2024-05-27T07:35:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.862+0000 7f124405b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f123c104330 0x7f123c19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.862+0000 7f124405b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123c105460 0x7f123c19ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.862+0000 7f124405b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f123c105e00 0x7f123c1a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.862+0000 7f124405b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f123c10fda0 con 0x7f123c105460 2024-05-27T07:35:03.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.862+0000 7f124405b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f123c10fc20 con 0x7f123c104330 2024-05-27T07:35:03.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.862+0000 7f124405b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f123c10ff20 con 0x7f123c105e00 2024-05-27T07:35:03.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.862+0000 7f12415f6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123c105460 0x7f123c19ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.862+0000 7f12415f6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123c105460 0x7f123c19ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48490/0 (socket says 172.21.15.114:48490) 2024-05-27T07:35:03.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.862+0000 7f12415f6700 1 -- 172.21.15.114:0/3532465273 learned_addr learned my addr 172.21.15.114:0/3532465273 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:35:03.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.862+0000 7f12425f8700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f123c105e00 0x7f123c1a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.862+0000 7f1241df7700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f123c104330 0x7f123c19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.863+0000 7f12415f6700 1 -- 172.21.15.114:0/3532465273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f123c105e00 msgr2=0x7f123c1a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:03.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.863+0000 7f12415f6700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f123c105e00 0x7f123c1a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.863+0000 7f12415f6700 1 -- 172.21.15.114:0/3532465273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f123c104330 msgr2=0x7f123c19a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:03.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.863+0000 7f12415f6700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f123c104330 0x7f123c19a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.863+0000 7f12415f6700 1 -- 172.21.15.114:0/3532465273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123c1a2470 con 0x7f123c105460 2024-05-27T07:35:03.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.863+0000 7f1241df7700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f123c104330 0x7f123c19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:03.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.863+0000 7f12415f6700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123c105460 0x7f123c19ae50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1230006070 tx=0x7f1230004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:03.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.863+0000 7f123affd700 1 -- 172.21.15.114:0/3532465273 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1230021070 con 0x7f123c105460 2024-05-27T07:35:03.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.863+0000 7f123affd700 1 -- 172.21.15.114:0/3532465273 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1230007c80 con 0x7f123c105460 2024-05-27T07:35:03.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.863+0000 7f123affd700 1 -- 172.21.15.114:0/3532465273 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1230004880 con 0x7f123c105460 2024-05-27T07:35:03.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.863+0000 7f124405b700 1 -- 172.21.15.114:0/3532465273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f123c1a2700 con 0x7f123c105460 2024-05-27T07:35:03.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.863+0000 7f124405b700 1 -- 172.21.15.114:0/3532465273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f123c1a2bf0 con 0x7f123c105460 2024-05-27T07:35:03.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.864+0000 7f124405b700 1 -- 172.21.15.114:0/3532465273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f123c0512b0 con 0x7f123c105460 2024-05-27T07:35:03.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.867+0000 7f123affd700 1 -- 172.21.15.114:0/3532465273 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f1230003f40 con 0x7f123c105460 2024-05-27T07:35:03.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.867+0000 7f123affd700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f12240b2880 0x7f12240b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.868+0000 7f1241df7700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f12240b2880 0x7f12240b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.868+0000 7f123affd700 1 -- 172.21.15.114:0/3532465273 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6847+0+0 (secure 0 0 0) 0x7f123000f020 con 0x7f123c105460 2024-05-27T07:35:03.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.868+0000 7f1241df7700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f12240b2880 0x7f12240b4d90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1234005950 tx=0x7f12340058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:03.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:03.871+0000 7f123affd700 1 -- 172.21.15.114:0/3532465273 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f123002a020 con 0x7f123c105460 2024-05-27T07:35:04.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.079+0000 7f124405b700 1 -- 172.21.15.114:0/3532465273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f123c02e5e0 con 0x7f123c105460 2024-05-27T07:35:04.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.079+0000 7f123affd700 1 -- 172.21.15.114:0/3532465273 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v70) ==== 74+0+133 (secure 0 0 0) 0x7f12300ab960 con 0x7f123c105460 2024-05-27T07:35:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:35:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.081+0000 7f124405b700 1 -- 172.21.15.114:0/3532465273 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f12240b2880 msgr2=0x7f12240b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.081+0000 7f124405b700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f12240b2880 0x7f12240b4d90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1234005950 tx=0x7f12340058e0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.081+0000 7f124405b700 1 -- 172.21.15.114:0/3532465273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123c105460 msgr2=0x7f123c19ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.081+0000 7f124405b700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123c105460 0x7f123c19ae50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1230006070 tx=0x7f1230004e60 comp rx=0 tx=0).stop 2024-05-27T07:35:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.081+0000 7f124405b700 1 -- 172.21.15.114:0/3532465273 shutdown_connections 2024-05-27T07:35:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.081+0000 7f124405b700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f12240b2880 0x7f12240b4d90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.081+0000 7f124405b700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f123c105e00 0x7f123c1a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.081+0000 7f124405b700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f123c104330 0x7f123c19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.081+0000 7f124405b700 1 --2- 172.21.15.114:0/3532465273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123c105460 0x7f123c19ae50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.081+0000 7f124405b700 1 -- 172.21.15.114:0/3532465273 >> 172.21.15.114:0/3532465273 conn(0x7f123c0ff9b0 msgr2=0x7f123c108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.082+0000 7f124405b700 1 -- 172.21.15.114:0/3532465273 shutdown_connections 2024-05-27T07:35:04.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:04.082+0000 7f124405b700 1 -- 172.21.15.114:0/3532465273 wait complete. 2024-05-27T07:35:04.845 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"num_osds":12,"num_up_osds":12,"osd_up_since":1716795303,"num_in_osds":12,"osd_in_since":1716795283,"num_remapped_pgs":0} 2024-05-27T07:35:04.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:35:05.097 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:05.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:04 smithi114 ceph-mon[32038]: purged_snaps scrub starts 2024-05-27T07:35:05.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:04 smithi114 ceph-mon[32038]: purged_snaps scrub ok 2024-05-27T07:35:05.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:04 smithi114 ceph-mon[32038]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:35:05.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:04 smithi114 ceph-mon[32038]: osd.11 [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501] boot 2024-05-27T07:35:05.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:04 smithi114 ceph-mon[32038]: osdmap e70: 12 total, 12 up, 12 in 2024-05-27T07:35:05.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:04 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:05.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:04 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3532465273' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:35:05.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:04 smithi163 ceph-mon[36451]: purged_snaps scrub starts 2024-05-27T07:35:05.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:04 smithi163 ceph-mon[36451]: purged_snaps scrub ok 2024-05-27T07:35:05.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:04 smithi163 ceph-mon[36451]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:35:05.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:04 smithi163 ceph-mon[36451]: osd.11 [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501] boot 2024-05-27T07:35:05.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:04 smithi163 ceph-mon[36451]: osdmap e70: 12 total, 12 up, 12 in 2024-05-27T07:35:05.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:04 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:05.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:04 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3532465273' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:35:05.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:04 smithi186 ceph-mon[36647]: purged_snaps scrub starts 2024-05-27T07:35:05.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:04 smithi186 ceph-mon[36647]: purged_snaps scrub ok 2024-05-27T07:35:05.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:04 smithi186 ceph-mon[36647]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:35:05.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:04 smithi186 ceph-mon[36647]: osd.11 [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501] boot 2024-05-27T07:35:05.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:04 smithi186 ceph-mon[36647]: osdmap e70: 12 total, 12 up, 12 in 2024-05-27T07:35:05.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:04 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:35:05.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:04 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3532465273' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:35:06.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:05 smithi163 ceph-mon[36451]: Detected new or changed devices on smithi186 2024-05-27T07:35:06.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:05 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:06.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:05 smithi163 ceph-mon[36451]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T07:35:06.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:05 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:06.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:05 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:05 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:05 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:05 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:05 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi186 to 4272M 2024-05-27T07:35:06.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:05 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:06.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:05 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:06.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:05 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:06.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:05 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:06.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:05 smithi186 ceph-mon[36647]: Detected new or changed devices on smithi186 2024-05-27T07:35:06.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:05 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:06.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:05 smithi186 ceph-mon[36647]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T07:35:06.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:05 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:06.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:05 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:05 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:05 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:05 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:05 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi186 to 4272M 2024-05-27T07:35:06.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:05 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:06.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:05 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:06.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:05 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:06.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:05 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:06.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:05 smithi114 ceph-mon[32038]: Detected new or changed devices on smithi186 2024-05-27T07:35:06.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:05 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:06.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:05 smithi114 ceph-mon[32038]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T07:35:06.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:05 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:06.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:05 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:05 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:05 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:05 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:05 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi186 to 4272M 2024-05-27T07:35:06.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:05 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:06.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:05 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:06.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:05 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:06.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:05 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:06.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.898+0000 7fe2a10dd700 1 -- 172.21.15.114:0/3141416082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe29c108f00 msgr2=0x7fe29c109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:06.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.898+0000 7fe2a10dd700 1 --2- 172.21.15.114:0/3141416082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe29c108f00 0x7fe29c109310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0099c0 tx=0x7fe28c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:35:06.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.898+0000 7fe2a10dd700 1 -- 172.21.15.114:0/3141416082 shutdown_connections 2024-05-27T07:35:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.898+0000 7fe2a10dd700 1 --2- 172.21.15.114:0/3141416082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe29c109850 0x7fe29c1111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.898+0000 7fe2a10dd700 1 --2- 172.21.15.114:0/3141416082 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe29c1045d0 0x7fe29c1049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.898+0000 7fe2a10dd700 1 --2- 172.21.15.114:0/3141416082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe29c108f00 0x7fe29c109310 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.898+0000 7fe2a10dd700 1 -- 172.21.15.114:0/3141416082 >> 172.21.15.114:0/3141416082 conn(0x7fe29c0ffec0 msgr2=0x7fe29c102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.898+0000 7fe2a10dd700 1 -- 172.21.15.114:0/3141416082 shutdown_connections 2024-05-27T07:35:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.898+0000 7fe2a10dd700 1 -- 172.21.15.114:0/3141416082 wait complete. 2024-05-27T07:35:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.899+0000 7fe2a10dd700 1 Processor -- start 2024-05-27T07:35:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe2a10dd700 1 -- start start 2024-05-27T07:35:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe2a10dd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe29c1045d0 0x7fe29c19ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe2a10dd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe29c108f00 0x7fe29c19b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe2a10dd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe29c109850 0x7fe29c19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe2a10dd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c1144d0 con 0x7fe29c109850 2024-05-27T07:35:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe2a10dd700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c114350 con 0x7fe29c1045d0 2024-05-27T07:35:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe2a10dd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c114650 con 0x7fe29c108f00 2024-05-27T07:35:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe29b59e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe29c109850 0x7fe29c19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe29b59e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe29c109850 0x7fe29c19f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44322/0 (socket says 172.21.15.114:44322) 2024-05-27T07:35:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe29b59e700 1 -- 172.21.15.114:0/2742461775 learned_addr learned my addr 172.21.15.114:0/2742461775 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:35:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe29ad9d700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe29c1045d0 0x7fe29c19ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe29a59c700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe29c108f00 0x7fe29c19b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe29b59e700 1 -- 172.21.15.114:0/2742461775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe29c108f00 msgr2=0x7fe29c19b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe29b59e700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe29c108f00 0x7fe29c19b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe29b59e700 1 -- 172.21.15.114:0/2742461775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe29c1045d0 msgr2=0x7fe29c19ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe29b59e700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe29c1045d0 0x7fe29c19ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.900+0000 7fe29b59e700 1 -- 172.21.15.114:0/2742461775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c19fd30 con 0x7fe29c109850 2024-05-27T07:35:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.901+0000 7fe29b59e700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe29c109850 0x7fe29c19f690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe29400b970 tx=0x7fe29400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.901+0000 7fe283fff700 1 -- 172.21.15.114:0/2742461775 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29400d860 con 0x7fe29c109850 2024-05-27T07:35:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.901+0000 7fe2a10dd700 1 -- 172.21.15.114:0/2742461775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29c1a0020 con 0x7fe29c109850 2024-05-27T07:35:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.901+0000 7fe283fff700 1 -- 172.21.15.114:0/2742461775 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe29400e430 con 0x7fe29c109850 2024-05-27T07:35:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.901+0000 7fe283fff700 1 -- 172.21.15.114:0/2742461775 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe294017660 con 0x7fe29c109850 2024-05-27T07:35:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.901+0000 7fe2a10dd700 1 -- 172.21.15.114:0/2742461775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe29c1a7940 con 0x7fe29c109850 2024-05-27T07:35:06.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.902+0000 7fe2a10dd700 1 -- 172.21.15.114:0/2742461775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe29c0512b0 con 0x7fe29c109850 2024-05-27T07:35:06.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.904+0000 7fe283fff700 1 -- 172.21.15.114:0/2742461775 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fe29400da00 con 0x7fe29c109850 2024-05-27T07:35:06.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.905+0000 7fe283fff700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe2840b2880 0x7fe2840b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:06.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.905+0000 7fe29ad9d700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe2840b2880 0x7fe2840b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:06.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.905+0000 7fe283fff700 1 -- 172.21.15.114:0/2742461775 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fe2940e1b30 con 0x7fe29c109850 2024-05-27T07:35:06.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.906+0000 7fe29ad9d700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe2840b2880 0x7fe2840b4d90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe290005950 tx=0x7fe29000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:06.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:06.909+0000 7fe283fff700 1 -- 172.21.15.114:0/2742461775 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe294053df0 con 0x7fe29c109850 2024-05-27T07:35:07.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.117+0000 7fe2a10dd700 1 -- 172.21.15.114:0/2742461775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe29c19c1c0 con 0x7fe29c109850 2024-05-27T07:35:07.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.118+0000 7fe283fff700 1 -- 172.21.15.114:0/2742461775 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+18538 (secure 0 0 0) 0x7fe294013030 con 0x7fe29c109850 2024-05-27T07:35:07.114 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:35:07.114 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":71,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T07:35:04.893084+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:35:07.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.121+0000 7fe2a10dd700 1 -- 172.21.15.114:0/2742461775 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe2840b2880 msgr2=0x7fe2840b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:07.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.121+0000 7fe2a10dd700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe2840b2880 0x7fe2840b4d90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe290005950 tx=0x7fe29000b410 comp rx=0 tx=0).stop 2024-05-27T07:35:07.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.121+0000 7fe2a10dd700 1 -- 172.21.15.114:0/2742461775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe29c109850 msgr2=0x7fe29c19f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:07.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.121+0000 7fe2a10dd700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe29c109850 0x7fe29c19f690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe29400b970 tx=0x7fe29400bc80 comp rx=0 tx=0).stop 2024-05-27T07:35:07.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.121+0000 7fe2a10dd700 1 -- 172.21.15.114:0/2742461775 shutdown_connections 2024-05-27T07:35:07.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.121+0000 7fe2a10dd700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe29c109850 0x7fe29c19f690 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:07.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.121+0000 7fe2a10dd700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe29c1045d0 0x7fe29c19ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:07.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.121+0000 7fe2a10dd700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe2840b2880 0x7fe2840b4d90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:07.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.121+0000 7fe2a10dd700 1 --2- 172.21.15.114:0/2742461775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe29c108f00 0x7fe29c19b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:07.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.121+0000 7fe2a10dd700 1 -- 172.21.15.114:0/2742461775 >> 172.21.15.114:0/2742461775 conn(0x7fe29c0ffec0 msgr2=0x7fe29c1086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:07.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.121+0000 7fe2a10dd700 1 -- 172.21.15.114:0/2742461775 shutdown_connections 2024-05-27T07:35:07.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:07.121+0000 7fe2a10dd700 1 -- 172.21.15.114:0/2742461775 wait complete. 2024-05-27T07:35:07.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:06 smithi163 ceph-mon[36451]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:07.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:06 smithi186 ceph-mon[36647]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:07.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:06 smithi114 ceph-mon[32038]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:07.857 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T07:31:09.365695+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T07:35:07.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-27T07:35:08.101 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:08.195 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:07 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2742461775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:35:08.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:07 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2742461775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:35:08.223 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:07 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2742461775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:35:09.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:08 smithi163 ceph-mon[36451]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:09.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:08 smithi186 ceph-mon[36647]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:09.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:08 smithi114 ceph-mon[32038]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:09.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.843+0000 7efd1f1db700 1 -- 172.21.15.114:0/478024627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd18108ef0 msgr2=0x7efd18109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:09.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.843+0000 7efd1f1db700 1 --2- 172.21.15.114:0/478024627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd18108ef0 0x7efd18109300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efd040099c0 tx=0x7efd04009cd0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.844+0000 7efd1f1db700 1 -- 172.21.15.114:0/478024627 shutdown_connections 2024-05-27T07:35:09.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.844+0000 7efd1f1db700 1 --2- 172.21.15.114:0/478024627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd18109840 0x7efd181111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.844+0000 7efd1f1db700 1 --2- 172.21.15.114:0/478024627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd181045c0 0x7efd181049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.844+0000 7efd1f1db700 1 --2- 172.21.15.114:0/478024627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd18108ef0 0x7efd18109300 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.844+0000 7efd1f1db700 1 -- 172.21.15.114:0/478024627 >> 172.21.15.114:0/478024627 conn(0x7efd180ffe90 msgr2=0x7efd181022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:09.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.844+0000 7efd1f1db700 1 -- 172.21.15.114:0/478024627 shutdown_connections 2024-05-27T07:35:09.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.844+0000 7efd1f1db700 1 -- 172.21.15.114:0/478024627 wait complete. 2024-05-27T07:35:09.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.844+0000 7efd1f1db700 1 Processor -- start 2024-05-27T07:35:09.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd1f1db700 1 -- start start 2024-05-27T07:35:09.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd1f1db700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd181045c0 0x7efd1819ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd1f1db700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd18108ef0 0x7efd1819b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd1f1db700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd18109840 0x7efd1819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd1f1db700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd18114450 con 0x7efd18108ef0 2024-05-27T07:35:09.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd1f1db700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd181142d0 con 0x7efd181045c0 2024-05-27T07:35:09.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd1f1db700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efd181145d0 con 0x7efd18109840 2024-05-27T07:35:09.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd17fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd18108ef0 0x7efd1819b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd17fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd18108ef0 0x7efd1819b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44330/0 (socket says 172.21.15.114:44330) 2024-05-27T07:35:09.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd17fff700 1 -- 172.21.15.114:0/3223475360 learned_addr learned my addr 172.21.15.114:0/3223475360 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:35:09.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd1cf77700 1 --2- 172.21.15.114:0/3223475360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd181045c0 0x7efd1819ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd17fff700 1 -- 172.21.15.114:0/3223475360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd18109840 msgr2=0x7efd1819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:09.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd17fff700 1 --2- 172.21.15.114:0/3223475360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd18109840 0x7efd1819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd17fff700 1 -- 172.21.15.114:0/3223475360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd181045c0 msgr2=0x7efd1819ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:09.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd17fff700 1 --2- 172.21.15.114:0/3223475360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd181045c0 0x7efd1819ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.845+0000 7efd17fff700 1 -- 172.21.15.114:0/3223475360 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd1819fcf0 con 0x7efd18108ef0 2024-05-27T07:35:09.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.846+0000 7efd17fff700 1 --2- 172.21.15.114:0/3223475360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd18108ef0 0x7efd1819b3c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efd04006010 tx=0x7efd040047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:09.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.846+0000 7efd15ffb700 1 -- 172.21.15.114:0/3223475360 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd04027070 con 0x7efd18108ef0 2024-05-27T07:35:09.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.846+0000 7efd1f1db700 1 -- 172.21.15.114:0/3223475360 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd1819ff80 con 0x7efd18108ef0 2024-05-27T07:35:09.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.846+0000 7efd15ffb700 1 -- 172.21.15.114:0/3223475360 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd04003fa0 con 0x7efd18108ef0 2024-05-27T07:35:09.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.846+0000 7efd15ffb700 1 -- 172.21.15.114:0/3223475360 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd04022dd0 con 0x7efd18108ef0 2024-05-27T07:35:09.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.846+0000 7efd1f1db700 1 -- 172.21.15.114:0/3223475360 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd181a0460 con 0x7efd18108ef0 2024-05-27T07:35:09.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.847+0000 7efd1f1db700 1 -- 172.21.15.114:0/3223475360 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd180512b0 con 0x7efd18108ef0 2024-05-27T07:35:09.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.848+0000 7efd15ffb700 1 -- 172.21.15.114:0/3223475360 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7efd0402a070 con 0x7efd18108ef0 2024-05-27T07:35:09.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.849+0000 7efd15ffb700 1 --2- 172.21.15.114:0/3223475360 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7efd080b2880 0x7efd080b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.849+0000 7efd1cf77700 1 --2- 172.21.15.114:0/3223475360 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7efd080b2880 0x7efd080b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.849+0000 7efd15ffb700 1 -- 172.21.15.114:0/3223475360 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7efd040e2ef0 con 0x7efd18108ef0 2024-05-27T07:35:09.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.850+0000 7efd1cf77700 1 --2- 172.21.15.114:0/3223475360 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7efd080b2880 0x7efd080b4d90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efd0c005950 tx=0x7efd0c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:09.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:09.853+0000 7efd15ffb700 1 -- 172.21.15.114:0/3223475360 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7efd04031080 con 0x7efd18108ef0 2024-05-27T07:35:10.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.056+0000 7efd1f1db700 1 -- 172.21.15.114:0/3223475360 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efd1802e5e0 con 0x7efd18108ef0 2024-05-27T07:35:10.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.056+0000 7efd15ffb700 1 -- 172.21.15.114:0/3223475360 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) ==== 93+0+10 (secure 0 0 0) 0x7efd04013070 con 0x7efd18108ef0 2024-05-27T07:35:10.052 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-05-27T07:35:10.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.060+0000 7efd1f1db700 1 -- 172.21.15.114:0/3223475360 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7efd080b2880 msgr2=0x7efd080b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:10.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.060+0000 7efd1f1db700 1 --2- 172.21.15.114:0/3223475360 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7efd080b2880 0x7efd080b4d90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efd0c005950 tx=0x7efd0c0058e0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.060+0000 7efd1f1db700 1 -- 172.21.15.114:0/3223475360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd18108ef0 msgr2=0x7efd1819b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:10.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.060+0000 7efd1f1db700 1 --2- 172.21.15.114:0/3223475360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd18108ef0 0x7efd1819b3c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efd04006010 tx=0x7efd040047b0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.060+0000 7efd1f1db700 1 -- 172.21.15.114:0/3223475360 shutdown_connections 2024-05-27T07:35:10.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.060+0000 7efd1f1db700 1 --2- 172.21.15.114:0/3223475360 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7efd080b2880 0x7efd080b4d90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.060+0000 7efd1f1db700 1 --2- 172.21.15.114:0/3223475360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd18109840 0x7efd1819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.060+0000 7efd1f1db700 1 --2- 172.21.15.114:0/3223475360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd181045c0 0x7efd1819ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.060+0000 7efd1f1db700 1 --2- 172.21.15.114:0/3223475360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd18108ef0 0x7efd1819b3c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.060+0000 7efd1f1db700 1 -- 172.21.15.114:0/3223475360 >> 172.21.15.114:0/3223475360 conn(0x7efd180ffe90 msgr2=0x7efd1810a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:10.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.060+0000 7efd1f1db700 1 -- 172.21.15.114:0/3223475360 shutdown_connections 2024-05-27T07:35:10.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:10.060+0000 7efd1f1db700 1 -- 172.21.15.114:0/3223475360 wait complete. 2024-05-27T07:35:10.855 INFO:tasks.cephadm:Adding mds.a on smithi114 2024-05-27T07:35:10.855 INFO:tasks.cephadm:Adding mds.d on smithi114 2024-05-27T07:35:10.855 INFO:tasks.cephadm:Adding mds.g on smithi114 2024-05-27T07:35:10.855 INFO:tasks.cephadm:Adding mds.j on smithi114 2024-05-27T07:35:10.855 INFO:tasks.cephadm:Adding mds.b on smithi163 2024-05-27T07:35:10.855 INFO:tasks.cephadm:Adding mds.e on smithi163 2024-05-27T07:35:10.855 INFO:tasks.cephadm:Adding mds.h on smithi163 2024-05-27T07:35:10.855 INFO:tasks.cephadm:Adding mds.k on smithi163 2024-05-27T07:35:10.855 INFO:tasks.cephadm:Adding mds.c on smithi186 2024-05-27T07:35:10.855 INFO:tasks.cephadm:Adding mds.f on smithi186 2024-05-27T07:35:10.855 INFO:tasks.cephadm:Adding mds.i on smithi186 2024-05-27T07:35:10.855 INFO:tasks.cephadm:Adding mds.l on smithi186 2024-05-27T07:35:10.856 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph orch apply mds all '12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l' 2024-05-27T07:35:11.107 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:35:11.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:10 smithi163 ceph-mon[36451]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:11.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:10 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3223475360' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:35:11.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:10 smithi186 ceph-mon[36647]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:11.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:10 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3223475360' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:35:11.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:10 smithi114 ceph-mon[32038]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:11.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:10 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3223475360' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:35:12.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.809+0000 7fe150cc6700 1 -- 172.21.15.186:0/996484693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe14c102c60 msgr2=0x7fe14c10c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:12.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.809+0000 7fe150cc6700 1 --2- 172.21.15.186:0/996484693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe14c102c60 0x7fe14c10c880 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe14400b0a0 tx=0x7fe14400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.809+0000 7fe150cc6700 1 -- 172.21.15.186:0/996484693 shutdown_connections 2024-05-27T07:35:12.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.809+0000 7fe150cc6700 1 --2- 172.21.15.186:0/996484693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe14c102c60 0x7fe14c10c880 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.809+0000 7fe150cc6700 1 --2- 172.21.15.186:0/996484693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe14c1058c0 0x7fe14c105cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.809+0000 7fe150cc6700 1 --2- 172.21.15.186:0/996484693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe14c102310 0x7fe14c102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.809+0000 7fe150cc6700 1 -- 172.21.15.186:0/996484693 >> 172.21.15.186:0/996484693 conn(0x7fe14c0fe000 msgr2=0x7fe14c100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:12.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.809+0000 7fe150cc6700 1 -- 172.21.15.186:0/996484693 shutdown_connections 2024-05-27T07:35:12.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.809+0000 7fe150cc6700 1 -- 172.21.15.186:0/996484693 wait complete. 2024-05-27T07:35:12.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.810+0000 7fe150cc6700 1 Processor -- start 2024-05-27T07:35:12.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.810+0000 7fe150cc6700 1 -- start start 2024-05-27T07:35:12.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.810+0000 7fe150cc6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe14c102310 0x7fe14c19a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:12.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.810+0000 7fe150cc6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe14c102c60 0x7fe14c19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:12.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.810+0000 7fe150cc6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe14c1058c0 0x7fe14c1a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:12.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.810+0000 7fe150cc6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe14c10fdd0 con 0x7fe14c102c60 2024-05-27T07:35:12.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.810+0000 7fe150cc6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe14c10fc50 con 0x7fe14c102310 2024-05-27T07:35:12.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.810+0000 7fe150cc6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe14c10ff50 con 0x7fe14c1058c0 2024-05-27T07:35:12.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe14ad9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe14c1058c0 0x7fe14c1a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:12.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe14ad9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe14c1058c0 0x7fe14c1a1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52906/0 (socket says 172.21.15.186:52906) 2024-05-27T07:35:12.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe14ad9d700 1 -- 172.21.15.186:0/3685588472 learned_addr learned my addr 172.21.15.186:0/3685588472 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-27T07:35:12.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe14a59c700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe14c102310 0x7fe14c19a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:12.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe141d9b700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe14c102c60 0x7fe14c19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:12.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe14ad9d700 1 -- 172.21.15.186:0/3685588472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe14c102310 msgr2=0x7fe14c19a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:12.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe14ad9d700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe14c102310 0x7fe14c19a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe14ad9d700 1 -- 172.21.15.186:0/3685588472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe14c102c60 msgr2=0x7fe14c19ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:12.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe14ad9d700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe14c102c60 0x7fe14c19ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe14ad9d700 1 -- 172.21.15.186:0/3685588472 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe14c1a23b0 con 0x7fe14c1058c0 2024-05-27T07:35:12.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe14ad9d700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe14c1058c0 0x7fe14c1a1dc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe14400b070 tx=0x7fe14400ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:12.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe143fff700 1 -- 172.21.15.186:0/3685588472 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe144021020 con 0x7fe14c1058c0 2024-05-27T07:35:12.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe143fff700 1 -- 172.21.15.186:0/3685588472 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe144007cc0 con 0x7fe14c1058c0 2024-05-27T07:35:12.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe150cc6700 1 -- 172.21.15.186:0/3685588472 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe14c1a2640 con 0x7fe14c1058c0 2024-05-27T07:35:12.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe143fff700 1 -- 172.21.15.186:0/3685588472 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1440053b0 con 0x7fe14c1058c0 2024-05-27T07:35:12.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.811+0000 7fe150cc6700 1 -- 172.21.15.186:0/3685588472 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe14c1a2b80 con 0x7fe14c1058c0 2024-05-27T07:35:12.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.812+0000 7fe150cc6700 1 -- 172.21.15.186:0/3685588472 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe14c0512b0 con 0x7fe14c1058c0 2024-05-27T07:35:12.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.814+0000 7fe143fff700 1 -- 172.21.15.186:0/3685588472 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fe144024020 con 0x7fe14c1058c0 2024-05-27T07:35:12.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.814+0000 7fe143fff700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe1380b2880 0x7fe1380b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:12.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.814+0000 7fe143fff700 1 -- 172.21.15.186:0/3685588472 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fe14400f020 con 0x7fe14c1058c0 2024-05-27T07:35:12.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.815+0000 7fe14a59c700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe1380b2880 0x7fe1380b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:12.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.815+0000 7fe14a59c700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe1380b2880 0x7fe1380b4d90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe14c06b360 tx=0x7fe13c00a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:12.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:12.818+0000 7fe143fff700 1 -- 172.21.15.186:0/3685588472 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe1440abd10 con 0x7fe14c1058c0 2024-05-27T07:35:13.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.036+0000 7fe150cc6700 1 -- 172.21.15.186:0/3685588472 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}) -- 0x7fe14c063df0 con 0x7fe1380b2880 2024-05-27T07:35:13.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.046+0000 7fe143fff700 1 -- 172.21.15.186:0/3685588472 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fe14c063df0 con 0x7fe1380b2880 2024-05-27T07:35:13.041 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mds.all update... 2024-05-27T07:35:13.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.049+0000 7fe150cc6700 1 -- 172.21.15.186:0/3685588472 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe1380b2880 msgr2=0x7fe1380b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:13.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.049+0000 7fe150cc6700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe1380b2880 0x7fe1380b4d90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe14c06b360 tx=0x7fe13c00a3b0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.049+0000 7fe150cc6700 1 -- 172.21.15.186:0/3685588472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe14c1058c0 msgr2=0x7fe14c1a1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:13.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.049+0000 7fe150cc6700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe14c1058c0 0x7fe14c1a1dc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe14400b070 tx=0x7fe14400ced0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.049+0000 7fe150cc6700 1 -- 172.21.15.186:0/3685588472 shutdown_connections 2024-05-27T07:35:13.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.049+0000 7fe150cc6700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe1380b2880 0x7fe1380b4d90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.049+0000 7fe150cc6700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe14c1058c0 0x7fe14c1a1dc0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.049+0000 7fe150cc6700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe14c102310 0x7fe14c19a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.049+0000 7fe150cc6700 1 --2- 172.21.15.186:0/3685588472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe14c102c60 0x7fe14c19ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.049+0000 7fe150cc6700 1 -- 172.21.15.186:0/3685588472 >> 172.21.15.186:0/3685588472 conn(0x7fe14c0fe000 msgr2=0x7fe14c108860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:13.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.049+0000 7fe150cc6700 1 -- 172.21.15.186:0/3685588472 shutdown_connections 2024-05-27T07:35:13.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-27T07:35:13.049+0000 7fe150cc6700 1 -- 172.21.15.186:0/3685588472 wait complete. 2024-05-27T07:35:13.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:12 smithi163 ceph-mon[36451]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:13.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:12 smithi186 ceph-mon[36647]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:13.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:12 smithi114 ceph-mon[32038]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:13.733 DEBUG:teuthology.orchestra.run.smithi114:mds.a> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service 2024-05-27T07:35:13.736 DEBUG:teuthology.orchestra.run.smithi114:mds.d> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service 2024-05-27T07:35:13.779 DEBUG:teuthology.orchestra.run.smithi114:mds.g> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service 2024-05-27T07:35:13.822 DEBUG:teuthology.orchestra.run.smithi114:mds.j> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.j.service 2024-05-27T07:35:13.865 DEBUG:teuthology.orchestra.run.smithi163:mds.b> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service 2024-05-27T07:35:13.867 DEBUG:teuthology.orchestra.run.smithi163:mds.e> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service 2024-05-27T07:35:13.911 DEBUG:teuthology.orchestra.run.smithi163:mds.h> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service 2024-05-27T07:35:13.954 DEBUG:teuthology.orchestra.run.smithi163:mds.k> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service 2024-05-27T07:35:13.997 DEBUG:teuthology.orchestra.run.smithi186:mds.c> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service 2024-05-27T07:35:13.999 DEBUG:teuthology.orchestra.run.smithi186:mds.f> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service 2024-05-27T07:35:14.042 DEBUG:teuthology.orchestra.run.smithi186:mds.i> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.i.service 2024-05-27T07:35:14.084 DEBUG:teuthology.orchestra.run.smithi186:mds.l> sudo journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service 2024-05-27T07:35:14.128 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-27T07:35:14.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:35:14.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:14 smithi163 ceph-mon[36451]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:35:14.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:14 smithi163 ceph-mon[36451]: Saving service mds.all spec with placement smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-05-27T07:35:14.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:14 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:14.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:14 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:14.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:14 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:14.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:14 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:14.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:14 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:14.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:14 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:14.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:14 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:14.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:14 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:14.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:14 smithi114 ceph-mon[32038]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:35:14.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:14 smithi114 ceph-mon[32038]: Saving service mds.all spec with placement smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-05-27T07:35:14.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:14.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:14.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:14.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:14.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:14.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:14.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:14.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:14 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:14.315 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:14 smithi186 ceph-mon[36647]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:35:14.316 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:14 smithi186 ceph-mon[36647]: Saving service mds.all spec with placement smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-05-27T07:35:14.316 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:14 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:14.316 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:14 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:14.316 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:14 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:14.316 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:14 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:14.316 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:14 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:14.316 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:14 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:14.316 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:14 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:14.316 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:14 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:14.362 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:15.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:15 smithi114 ceph-mon[32038]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:15.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:15 smithi114 ceph-mon[32038]: Deploying daemon mds.l on smithi186 2024-05-27T07:35:15.410 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:15 smithi186 ceph-mon[36647]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:15.410 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:15 smithi186 ceph-mon[36647]: Deploying daemon mds.l on smithi186 2024-05-27T07:35:15.470 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:15 smithi163 ceph-mon[36451]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:15.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:15 smithi163 ceph-mon[36451]: Deploying daemon mds.l on smithi186 2024-05-27T07:35:15.974 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:35:15 smithi186 systemd[1]: Starting Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:35:16.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.201+0000 7f45c709d700 1 -- 172.21.15.114:0/3850453612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c0103300 msgr2=0x7f45c010f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:16.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.201+0000 7f45c709d700 1 --2- 172.21.15.114:0/3850453612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c0103300 0x7f45c010f880 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f45b800b0a0 tx=0x7f45b800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.201+0000 7f45c709d700 1 -- 172.21.15.114:0/3850453612 shutdown_connections 2024-05-27T07:35:16.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.201+0000 7f45c709d700 1 --2- 172.21.15.114:0/3850453612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c0103300 0x7f45c010f880 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.201+0000 7f45c709d700 1 --2- 172.21.15.114:0/3850453612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45c01089e0 0x7f45c0108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.201+0000 7f45c709d700 1 --2- 172.21.15.114:0/3850453612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45c01029b0 0x7f45c0102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.201+0000 7f45c709d700 1 -- 172.21.15.114:0/3850453612 >> 172.21.15.114:0/3850453612 conn(0x7f45c00fe510 msgr2=0x7f45c0100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:16.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.201+0000 7f45c709d700 1 -- 172.21.15.114:0/3850453612 shutdown_connections 2024-05-27T07:35:16.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.201+0000 7f45c709d700 1 -- 172.21.15.114:0/3850453612 wait complete. 2024-05-27T07:35:16.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c709d700 1 Processor -- start 2024-05-27T07:35:16.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c709d700 1 -- start start 2024-05-27T07:35:16.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c709d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45c01029b0 0x7f45c019e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c709d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45c0103300 0x7f45c019ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c709d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c01089e0 0x7f45c0198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c709d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f45c01122d0 con 0x7f45c01089e0 2024-05-27T07:35:16.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c709d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f45c0112150 con 0x7f45c01029b0 2024-05-27T07:35:16.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c709d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45c0112450 con 0x7f45c0103300 2024-05-27T07:35:16.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c563a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c01089e0 0x7f45c0198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c563a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c01089e0 0x7f45c0198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52444/0 (socket says 172.21.15.114:52444) 2024-05-27T07:35:16.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c563a700 1 -- 172.21.15.114:0/971403679 learned_addr learned my addr 172.21.15.114:0/971403679 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:35:16.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45bffff700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45c0103300 0x7f45c019ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c4e39700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45c01029b0 0x7f45c019e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c563a700 1 -- 172.21.15.114:0/971403679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45c0103300 msgr2=0x7f45c019ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:16.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c563a700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45c0103300 0x7f45c019ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c563a700 1 -- 172.21.15.114:0/971403679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45c01029b0 msgr2=0x7f45c019e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:16.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c563a700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45c01029b0 0x7f45c019e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c563a700 1 -- 172.21.15.114:0/971403679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c01990c0 con 0x7f45c01089e0 2024-05-27T07:35:16.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45bffff700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45c0103300 0x7f45c019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:16.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c4e39700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45c01029b0 0x7f45c019e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:16.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.202+0000 7f45c563a700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c01089e0 0x7f45c0198a10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f45b80062a0 tx=0x7f45b8004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:16.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.203+0000 7f45bdffb700 1 -- 172.21.15.114:0/971403679 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8021070 con 0x7f45c01089e0 2024-05-27T07:35:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.203+0000 7f45bdffb700 1 -- 172.21.15.114:0/971403679 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45b8003c00 con 0x7f45c01089e0 2024-05-27T07:35:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.203+0000 7f45c709d700 1 -- 172.21.15.114:0/971403679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c0199350 con 0x7f45c01089e0 2024-05-27T07:35:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.203+0000 7f45bdffb700 1 -- 172.21.15.114:0/971403679 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8004830 con 0x7f45c01089e0 2024-05-27T07:35:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.203+0000 7f45c709d700 1 -- 172.21.15.114:0/971403679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c01a5630 con 0x7f45c01089e0 2024-05-27T07:35:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.204+0000 7f45c709d700 1 -- 172.21.15.114:0/971403679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45c01040f0 con 0x7f45c01089e0 2024-05-27T07:35:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.206+0000 7f45bdffb700 1 -- 172.21.15.114:0/971403679 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f45b8007a00 con 0x7f45c01089e0 2024-05-27T07:35:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.206+0000 7f45bdffb700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f45b00b2880 0x7f45b00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.206+0000 7f45c4e39700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f45b00b2880 0x7f45b00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.206+0000 7f45bdffb700 1 -- 172.21.15.114:0/971403679 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f45b800f020 con 0x7f45c01089e0 2024-05-27T07:35:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.207+0000 7f45c4e39700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f45b00b2880 0x7f45b00b4d90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f45b4005e10 tx=0x7f45b4005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:16.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.210+0000 7f45bdffb700 1 -- 172.21.15.114:0/971403679 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f45b802a020 con 0x7f45c01089e0 2024-05-27T07:35:16.383 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:35:16 smithi186 podman[67069]: 2024-05-27 07:35:16.002193404 +0000 UTC m=+0.017784000 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:35:16.383 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:35:16 smithi186 podman[67069]: 2024-05-27 07:35:16.126747287 +0000 UTC m=+0.142337862 container create f84b3ce82512db54743ac267d2a282adc7bb82f1858b228e461af939e666bbec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-27T07:35:16.383 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:35:16 smithi186 podman[67069]: 2024-05-27 07:35:16.38869339 +0000 UTC m=+0.404283989 container init f84b3ce82512db54743ac267d2a282adc7bb82f1858b228e461af939e666bbec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-27T07:35:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.471+0000 7f45c709d700 1 -- 172.21.15.114:0/971403679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45c00512b0 con 0x7f45c01089e0 2024-05-27T07:35:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.471+0000 7f45bdffb700 1 -- 172.21.15.114:0/971403679 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f45b80ab980 con 0x7f45c01089e0 2024-05-27T07:35:16.467 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:35:16.467 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-27T07:35:16.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.474+0000 7f45c709d700 1 -- 172.21.15.114:0/971403679 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f45b00b2880 msgr2=0x7f45b00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:16.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.474+0000 7f45c709d700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f45b00b2880 0x7f45b00b4d90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f45b4005e10 tx=0x7f45b4005d60 comp rx=0 tx=0).stop 2024-05-27T07:35:16.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.474+0000 7f45c709d700 1 -- 172.21.15.114:0/971403679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c01089e0 msgr2=0x7f45c0198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:16.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.474+0000 7f45c709d700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c01089e0 0x7f45c0198a10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f45b80062a0 tx=0x7f45b8004d30 comp rx=0 tx=0).stop 2024-05-27T07:35:16.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.474+0000 7f45c709d700 1 -- 172.21.15.114:0/971403679 shutdown_connections 2024-05-27T07:35:16.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.474+0000 7f45c709d700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c01089e0 0x7f45c0198a10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.474+0000 7f45c709d700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45c01029b0 0x7f45c019e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.474+0000 7f45c709d700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f45b00b2880 0x7f45b00b4d90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.474+0000 7f45c709d700 1 --2- 172.21.15.114:0/971403679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45c0103300 0x7f45c019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.474+0000 7f45c709d700 1 -- 172.21.15.114:0/971403679 >> 172.21.15.114:0/971403679 conn(0x7f45c00fe510 msgr2=0x7f45c0100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:16.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.474+0000 7f45c709d700 1 -- 172.21.15.114:0/971403679 shutdown_connections 2024-05-27T07:35:16.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:16.474+0000 7f45c709d700 1 -- 172.21.15.114:0/971403679 wait complete. 2024-05-27T07:35:16.471 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 2 2024-05-27T07:35:16.724 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:35:16 smithi186 podman[67069]: 2024-05-27 07:35:16.391578045 +0000 UTC m=+0.407168635 container start f84b3ce82512db54743ac267d2a282adc7bb82f1858b228e461af939e666bbec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-27T07:35:16.724 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:35:16 smithi186 ceph-mds[67088]: -- 172.21.15.186:0/2060861931 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562886f15a40 con 0x562886f5d180 2024-05-27T07:35:16.724 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:35:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: starting mds.l at 2024-05-27T07:35:16.724 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:35:16 smithi186 bash[67069]: f84b3ce82512db54743ac267d2a282adc7bb82f1858b228e461af939e666bbec 2024-05-27T07:35:16.724 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:35:16 smithi186 systemd[1]: Started Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:35:17.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:17 smithi114 ceph-mon[32038]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:17.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:17 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6832/1801956554,v1:172.21.15.186:6833/1801956554] up:boot 2024-05-27T07:35:17.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:17 smithi114 ceph-mon[32038]: fsmap 1 up:standby 2024-05-27T07:35:17.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:17 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:35:17.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:17 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/971403679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:17.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:17 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:17.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:17 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:17.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:17 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:17.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:17 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:17.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:17 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:17.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:17 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:17.385 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:17 smithi163 ceph-mon[36451]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:17.386 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:17 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6832/1801956554,v1:172.21.15.186:6833/1801956554] up:boot 2024-05-27T07:35:17.386 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:17 smithi163 ceph-mon[36451]: fsmap 1 up:standby 2024-05-27T07:35:17.386 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:17 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:35:17.386 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:17 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/971403679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:17.386 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:17 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:17.386 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:17 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:17.386 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:17 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:17.386 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:17 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:17.386 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:17 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:17.386 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:17 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:17 smithi186 ceph-mon[36647]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:17 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6832/1801956554,v1:172.21.15.186:6833/1801956554] up:boot 2024-05-27T07:35:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:17 smithi186 ceph-mon[36647]: fsmap 1 up:standby 2024-05-27T07:35:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:17 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:35:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:17 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/971403679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:17 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:17 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:17 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:17 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:17 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:17.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:17 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:18.410 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:18 smithi163 ceph-mon[36451]: Deploying daemon mds.k on smithi163 2024-05-27T07:35:18.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:18 smithi186 ceph-mon[36647]: Deploying daemon mds.k on smithi163 2024-05-27T07:35:18.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:18 smithi114 ceph-mon[32038]: Deploying daemon mds.k on smithi163 2024-05-27T07:35:18.971 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:35:18 smithi163 systemd[1]: Starting Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:35:19.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:35:19.373 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:19 smithi163 ceph-mon[36451]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:19.374 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:35:19 smithi163 podman[65380]: 2024-05-27 07:35:18.995972127 +0000 UTC m=+0.017050360 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:35:19.374 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:35:19 smithi163 podman[65380]: 2024-05-27 07:35:19.118212906 +0000 UTC m=+0.139291117 container create aa7f7804209788d8785ca4a1416d69cb401668ef25110f2865cc272871660b1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:35:19.374 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:35:19 smithi163 podman[65380]: 2024-05-27 07:35:19.379424555 +0000 UTC m=+0.400502774 container init aa7f7804209788d8785ca4a1416d69cb401668ef25110f2865cc272871660b1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:35:19.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:19 smithi186 ceph-mon[36647]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:19.502 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:19.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:19 smithi114 ceph-mon[32038]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:19.721 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:35:19 smithi163 podman[65380]: 2024-05-27 07:35:19.382071244 +0000 UTC m=+0.403149473 container start aa7f7804209788d8785ca4a1416d69cb401668ef25110f2865cc272871660b1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-27T07:35:19.721 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:35:19 smithi163 ceph-mds[65401]: -- 172.21.15.163:0/1174337597 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f0e1cfba40 con 0x55f0e1d43600 2024-05-27T07:35:19.721 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:35:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: starting mds.k at 2024-05-27T07:35:19.721 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:35:19 smithi163 bash[65380]: aa7f7804209788d8785ca4a1416d69cb401668ef25110f2865cc272871660b1a 2024-05-27T07:35:19.721 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:35:19 smithi163 systemd[1]: Started Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:35:20.793 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:20 smithi163 ceph-mon[36451]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:20.793 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:20.793 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:20.793 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:20.794 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:20.794 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:20.794 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:20.794 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:20 smithi163 ceph-mon[36451]: Deploying daemon mds.j on smithi114 2024-05-27T07:35:20.794 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:20 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6832/1125403618,v1:172.21.15.163:6833/1125403618] up:boot 2024-05-27T07:35:20.794 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:20 smithi163 ceph-mon[36451]: fsmap 2 up:standby 2024-05-27T07:35:20.794 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:20 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:35:20.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:20 smithi114 ceph-mon[32038]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:20.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:20.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:20.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:20.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:20.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:20.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:20.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:20 smithi114 ceph-mon[32038]: Deploying daemon mds.j on smithi114 2024-05-27T07:35:20.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:20 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6832/1125403618,v1:172.21.15.163:6833/1125403618] up:boot 2024-05-27T07:35:20.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:20 smithi114 ceph-mon[32038]: fsmap 2 up:standby 2024-05-27T07:35:20.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:20 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:35:20.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:20 smithi186 ceph-mon[36647]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:20.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:20.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:20.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:20.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:20.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:20.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:20.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:20 smithi186 ceph-mon[36647]: Deploying daemon mds.j on smithi114 2024-05-27T07:35:20.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:20 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6832/1125403618,v1:172.21.15.163:6833/1125403618] up:boot 2024-05-27T07:35:20.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:20 smithi186 ceph-mon[36647]: fsmap 2 up:standby 2024-05-27T07:35:20.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:20 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:35:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.440+0000 7f102fa47700 1 -- 172.21.15.114:0/4011044221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1028108ef0 msgr2=0x7f1028109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.440+0000 7f102fa47700 1 --2- 172.21.15.114:0/4011044221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1028108ef0 0x7f1028109300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f10240099c0 tx=0x7f1024009cd0 comp rx=0 tx=0).stop 2024-05-27T07:35:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.440+0000 7f102fa47700 1 -- 172.21.15.114:0/4011044221 shutdown_connections 2024-05-27T07:35:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.440+0000 7f102fa47700 1 --2- 172.21.15.114:0/4011044221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1028109840 0x7f10281111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.440+0000 7f102fa47700 1 --2- 172.21.15.114:0/4011044221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10281045c0 0x7f10281049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.440+0000 7f102fa47700 1 --2- 172.21.15.114:0/4011044221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1028108ef0 0x7f1028109300 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.440+0000 7f102fa47700 1 -- 172.21.15.114:0/4011044221 >> 172.21.15.114:0/4011044221 conn(0x7f10280ffe90 msgr2=0x7f10281022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.440+0000 7f102fa47700 1 -- 172.21.15.114:0/4011044221 shutdown_connections 2024-05-27T07:35:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.440+0000 7f102fa47700 1 -- 172.21.15.114:0/4011044221 wait complete. 2024-05-27T07:35:22.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.441+0000 7f102fa47700 1 Processor -- start 2024-05-27T07:35:22.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.441+0000 7f102fa47700 1 -- start start 2024-05-27T07:35:22.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.441+0000 7f102fa47700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10281045c0 0x7f102819aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.441+0000 7f102fa47700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1028108ef0 0x7f102819b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102fa47700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1028109840 0x7f102819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102fa47700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1028114450 con 0x7f10281045c0 2024-05-27T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102fa47700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f10281142d0 con 0x7f1028108ef0 2024-05-27T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102fa47700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f10281145d0 con 0x7f1028109840 2024-05-27T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102d7e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10281045c0 0x7f102819aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102d7e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10281045c0 0x7f102819aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52466/0 (socket says 172.21.15.114:52466) 2024-05-27T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102d7e3700 1 -- 172.21.15.114:0/2987531673 learned_addr learned my addr 172.21.15.114:0/2987531673 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102dfe4700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1028109840 0x7f102819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102cfe2700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1028108ef0 0x7f102819b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102d7e3700 1 -- 172.21.15.114:0/2987531673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1028109840 msgr2=0x7f102819f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102d7e3700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1028109840 0x7f102819f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102d7e3700 1 -- 172.21.15.114:0/2987531673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1028108ef0 msgr2=0x7f102819b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102d7e3700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1028108ef0 0x7f102819b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102d7e3700 1 -- 172.21.15.114:0/2987531673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102819fd90 con 0x7f10281045c0 2024-05-27T07:35:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.442+0000 7f102d7e3700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10281045c0 0x7f102819aec0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f101800d950 tx=0x7f101800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.443+0000 7f101e7fc700 1 -- 172.21.15.114:0/2987531673 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1018016070 con 0x7f10281045c0 2024-05-27T07:35:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.443+0000 7f102fa47700 1 -- 172.21.15.114:0/2987531673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10281a0080 con 0x7f10281045c0 2024-05-27T07:35:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.443+0000 7f101e7fc700 1 -- 172.21.15.114:0/2987531673 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f101800fd40 con 0x7f10281045c0 2024-05-27T07:35:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.443+0000 7f101e7fc700 1 -- 172.21.15.114:0/2987531673 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101800e3b0 con 0x7f10281045c0 2024-05-27T07:35:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.443+0000 7f102fa47700 1 -- 172.21.15.114:0/2987531673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10281a79a0 con 0x7f10281045c0 2024-05-27T07:35:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.444+0000 7f102fa47700 1 -- 172.21.15.114:0/2987531673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10280512b0 con 0x7f10281045c0 2024-05-27T07:35:22.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.445+0000 7f101e7fc700 1 -- 172.21.15.114:0/2987531673 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f1018019070 con 0x7f10281045c0 2024-05-27T07:35:22.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.446+0000 7f101e7fc700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f10140b2900 0x7f10140b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:22.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.446+0000 7f102cfe2700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f10140b2900 0x7f10140b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:22.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.446+0000 7f101e7fc700 1 -- 172.21.15.114:0/2987531673 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f10180e1740 con 0x7f10281045c0 2024-05-27T07:35:22.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.447+0000 7f102cfe2700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f10140b2900 0x7f10140b4e10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1024006010 tx=0x7f1024000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:22.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.450+0000 7f101e7fc700 1 -- 172.21.15.114:0/2987531673 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f10180a9900 con 0x7f10281045c0 2024-05-27T07:35:22.689 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:22 smithi114 ceph-mon[32038]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:22.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.731+0000 7f102fa47700 1 -- 172.21.15.114:0/2987531673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f102819c220 con 0x7f10281045c0 2024-05-27T07:35:22.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.731+0000 7f101e7fc700 1 -- 172.21.15.114:0/2987531673 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f10180a9200 con 0x7f10281045c0 2024-05-27T07:35:22.730 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:35:22.730 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-27T07:35:22.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.734+0000 7f102fa47700 1 -- 172.21.15.114:0/2987531673 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f10140b2900 msgr2=0x7f10140b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:22.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.734+0000 7f102fa47700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f10140b2900 0x7f10140b4e10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1024006010 tx=0x7f1024000bc0 comp rx=0 tx=0).stop 2024-05-27T07:35:22.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.734+0000 7f102fa47700 1 -- 172.21.15.114:0/2987531673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10281045c0 msgr2=0x7f102819aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:22.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.734+0000 7f102fa47700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10281045c0 0x7f102819aec0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f101800d950 tx=0x7f101800dc60 comp rx=0 tx=0).stop 2024-05-27T07:35:22.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.734+0000 7f102fa47700 1 -- 172.21.15.114:0/2987531673 shutdown_connections 2024-05-27T07:35:22.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.734+0000 7f102fa47700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f10140b2900 0x7f10140b4e10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:22.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.734+0000 7f102fa47700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1028109840 0x7f102819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:22.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.734+0000 7f102fa47700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10281045c0 0x7f102819aec0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:22.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.734+0000 7f102fa47700 1 --2- 172.21.15.114:0/2987531673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1028108ef0 0x7f102819b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:22.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.734+0000 7f102fa47700 1 -- 172.21.15.114:0/2987531673 >> 172.21.15.114:0/2987531673 conn(0x7f10280ffe90 msgr2=0x7f102810a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:22.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.734+0000 7f102fa47700 1 -- 172.21.15.114:0/2987531673 shutdown_connections 2024-05-27T07:35:22.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:22.734+0000 7f102fa47700 1 -- 172.21.15.114:0/2987531673 wait complete. 2024-05-27T07:35:22.732 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 3 2024-05-27T07:35:22.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:22 smithi163 ceph-mon[36451]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:22.973 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:22 smithi186 ceph-mon[36647]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:23.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:23 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2987531673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:23.788 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:23 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2987531673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:23.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:23 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2987531673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:24.051 INFO:journalctl@ceph.mds.j.smithi114.stdout:May 27 07:35:23 smithi114 systemd[1]: Started Ceph mds.j for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:35:25.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:24 smithi163 ceph-mon[36451]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:25.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:24 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:25.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:24 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:25.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:24 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:25.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:24 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:25.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:24 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:25.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:24 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:25.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:24 smithi163 ceph-mon[36451]: Deploying daemon mds.d on smithi114 2024-05-27T07:35:25.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:24 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6834/807487035,v1:172.21.15.114:6835/807487035] up:boot 2024-05-27T07:35:25.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:24 smithi163 ceph-mon[36451]: fsmap 3 up:standby 2024-05-27T07:35:25.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:24 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:35:25.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:24 smithi186 ceph-mon[36647]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:25.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:24 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:25.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:24 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:25.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:24 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:25.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:24 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:25.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:24 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:25.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:24 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:25.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:24 smithi186 ceph-mon[36647]: Deploying daemon mds.d on smithi114 2024-05-27T07:35:25.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:24 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6834/807487035,v1:172.21.15.114:6835/807487035] up:boot 2024-05-27T07:35:25.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:24 smithi186 ceph-mon[36647]: fsmap 3 up:standby 2024-05-27T07:35:25.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:24 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:35:25.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:24 smithi114 ceph-mon[32038]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:25.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:24 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:25.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:24 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:25.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:24 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:25.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:24 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:25.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:24 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:25.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:24 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:25.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:24 smithi114 ceph-mon[32038]: Deploying daemon mds.d on smithi114 2024-05-27T07:35:25.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:24 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6834/807487035,v1:172.21.15.114:6835/807487035] up:boot 2024-05-27T07:35:25.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:24 smithi114 ceph-mon[32038]: fsmap 3 up:standby 2024-05-27T07:35:25.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:24 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:35:25.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:35:25.689 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:27.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:26 smithi163 ceph-mon[36451]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:27.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:26 smithi186 ceph-mon[36647]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:26 smithi114 ceph-mon[32038]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:27.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.786+0000 7f722cc7b700 1 -- 172.21.15.114:0/912819651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7228107e10 msgr2=0x7f7228108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:27.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.786+0000 7f722cc7b700 1 --2- 172.21.15.114:0/912819651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7228107e10 0x7f7228108220 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f72180099c0 tx=0x7f7218009cd0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.786+0000 7f722cc7b700 1 -- 172.21.15.114:0/912819651 shutdown_connections 2024-05-27T07:35:27.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.786+0000 7f722cc7b700 1 --2- 172.21.15.114:0/912819651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7228108760 0x7f72281100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.786+0000 7f722cc7b700 1 --2- 172.21.15.114:0/912819651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7228103530 0x7f7228103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.786+0000 7f722cc7b700 1 --2- 172.21.15.114:0/912819651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7228107e10 0x7f7228108220 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.786+0000 7f722cc7b700 1 -- 172.21.15.114:0/912819651 >> 172.21.15.114:0/912819651 conn(0x7f7228078e40 msgr2=0x7f7228079280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:27.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.786+0000 7f722cc7b700 1 -- 172.21.15.114:0/912819651 shutdown_connections 2024-05-27T07:35:27.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.786+0000 7f722cc7b700 1 -- 172.21.15.114:0/912819651 wait complete. 2024-05-27T07:35:27.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.787+0000 7f722cc7b700 1 Processor -- start 2024-05-27T07:35:27.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.787+0000 7f722cc7b700 1 -- start start 2024-05-27T07:35:27.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.787+0000 7f722cc7b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7228103530 0x7f722819bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:27.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.787+0000 7f722cc7b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7228107e10 0x7f722819c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:27.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.787+0000 7f722cc7b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7228108760 0x7f722819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:27.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.787+0000 7f722cc7b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f72281133e0 con 0x7f7228103530 2024-05-27T07:35:27.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.787+0000 7f722cc7b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7228113260 con 0x7f7228108760 2024-05-27T07:35:27.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.787+0000 7f722cc7b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7228113560 con 0x7f7228107e10 2024-05-27T07:35:27.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.787+0000 7f722659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7228103530 0x7f722819bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:27.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.787+0000 7f722659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7228103530 0x7f722819bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44348/0 (socket says 172.21.15.114:44348) 2024-05-27T07:35:27.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.787+0000 7f722659c700 1 -- 172.21.15.114:0/500356870 learned_addr learned my addr 172.21.15.114:0/500356870 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:35:27.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.787+0000 7f7226d9d700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7228108760 0x7f722819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:27.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.787+0000 7f7225d9b700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7228107e10 0x7f722819c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:27.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.788+0000 7f722659c700 1 -- 172.21.15.114:0/500356870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7228107e10 msgr2=0x7f722819c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:27.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.788+0000 7f722659c700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7228107e10 0x7f722819c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.788+0000 7f722659c700 1 -- 172.21.15.114:0/500356870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7228108760 msgr2=0x7f722819c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:27.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.788+0000 7f722659c700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7228108760 0x7f722819c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.788+0000 7f722659c700 1 -- 172.21.15.114:0/500356870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72281a7010 con 0x7f7228103530 2024-05-27T07:35:27.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.788+0000 7f722659c700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7228103530 0x7f722819bca0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f721000d950 tx=0x7f721000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:27.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.788+0000 7f721f7fe700 1 -- 172.21.15.114:0/500356870 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7210016070 con 0x7f7228103530 2024-05-27T07:35:27.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.788+0000 7f722cc7b700 1 -- 172.21.15.114:0/500356870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72281a7300 con 0x7f7228103530 2024-05-27T07:35:27.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.788+0000 7f721f7fe700 1 -- 172.21.15.114:0/500356870 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7210009df0 con 0x7f7228103530 2024-05-27T07:35:27.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.788+0000 7f721f7fe700 1 -- 172.21.15.114:0/500356870 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721000f3c0 con 0x7f7228103530 2024-05-27T07:35:27.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.788+0000 7f722cc7b700 1 -- 172.21.15.114:0/500356870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72281a7860 con 0x7f7228103530 2024-05-27T07:35:27.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.789+0000 7f722cc7b700 1 -- 172.21.15.114:0/500356870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72280512b0 con 0x7f7228103530 2024-05-27T07:35:27.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.791+0000 7f721f7fe700 1 -- 172.21.15.114:0/500356870 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f7210019070 con 0x7f7228103530 2024-05-27T07:35:27.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.791+0000 7f721f7fe700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f72140b2880 0x7f72140b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:27.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.792+0000 7f7225d9b700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f72140b2880 0x7f72140b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:27.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.792+0000 7f721f7fe700 1 -- 172.21.15.114:0/500356870 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f72100e1980 con 0x7f7228103530 2024-05-27T07:35:27.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.792+0000 7f7225d9b700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f72140b2880 0x7f72140b4d90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7218000c00 tx=0x7f721801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:27.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:27.795+0000 7f721f7fe700 1 -- 172.21.15.114:0/500356870 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f72100a9b40 con 0x7f7228103530 2024-05-27T07:35:27.969 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 07:35:27 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: starting mds.d at 2024-05-27T07:35:27.969 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 07:35:27 smithi114 bash[70100]: b20fb979dcf785fe8c4694ba1c525a1f2a180548501c24dd0755f2b028cf205a 2024-05-27T07:35:28.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.059+0000 7f722cc7b700 1 -- 172.21.15.114:0/500356870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f722802e5e0 con 0x7f7228103530 2024-05-27T07:35:28.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.060+0000 7f721f7fe700 1 -- 172.21.15.114:0/500356870 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3877 (secure 0 0 0) 0x7f72100a9440 con 0x7f7228103530 2024-05-27T07:35:28.055 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:35:28.055 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-27T07:35:28.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.062+0000 7f722cc7b700 1 -- 172.21.15.114:0/500356870 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f72140b2880 msgr2=0x7f72140b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:28.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.062+0000 7f722cc7b700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f72140b2880 0x7f72140b4d90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7218000c00 tx=0x7f721801a040 comp rx=0 tx=0).stop 2024-05-27T07:35:28.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.062+0000 7f722cc7b700 1 -- 172.21.15.114:0/500356870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7228103530 msgr2=0x7f722819bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:28.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.062+0000 7f722cc7b700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7228103530 0x7f722819bca0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f721000d950 tx=0x7f721000dc60 comp rx=0 tx=0).stop 2024-05-27T07:35:28.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.063+0000 7f722cc7b700 1 -- 172.21.15.114:0/500356870 shutdown_connections 2024-05-27T07:35:28.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.063+0000 7f722cc7b700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7228108760 0x7f722819c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:28.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.063+0000 7f722cc7b700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7228103530 0x7f722819bca0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:28.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.063+0000 7f722cc7b700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f72140b2880 0x7f72140b4d90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:28.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.063+0000 7f722cc7b700 1 --2- 172.21.15.114:0/500356870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7228107e10 0x7f722819c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:28.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.063+0000 7f722cc7b700 1 -- 172.21.15.114:0/500356870 >> 172.21.15.114:0/500356870 conn(0x7f7228078e40 msgr2=0x7f72281007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:28.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.063+0000 7f722cc7b700 1 -- 172.21.15.114:0/500356870 shutdown_connections 2024-05-27T07:35:28.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:28.063+0000 7f722cc7b700 1 -- 172.21.15.114:0/500356870 wait complete. 2024-05-27T07:35:28.058 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 5 2024-05-27T07:35:28.301 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 07:35:27 smithi114 systemd[1]: Started Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:35:28.960 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:28 smithi163 ceph-mon[36451]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:28.960 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:28 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6836/1221779710,v1:172.21.15.114:6837/1221779710] up:boot 2024-05-27T07:35:28.960 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:28 smithi163 ceph-mon[36451]: fsmap 4 up:standby 2024-05-27T07:35:28.960 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:35:28.960 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:28.960 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:28.961 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:28 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/500356870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:28.961 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:28.961 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:28.961 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:28.961 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:28 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:28.961 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:28 smithi163 ceph-mon[36451]: Deploying daemon mds.e on smithi163 2024-05-27T07:35:29.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:28 smithi186 ceph-mon[36647]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:29.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:28 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6836/1221779710,v1:172.21.15.114:6837/1221779710] up:boot 2024-05-27T07:35:29.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:28 smithi186 ceph-mon[36647]: fsmap 4 up:standby 2024-05-27T07:35:29.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:35:29.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:29.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:29.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:28 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/500356870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:29.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:29.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:29.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:29.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:28 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:29.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:28 smithi186 ceph-mon[36647]: Deploying daemon mds.e on smithi163 2024-05-27T07:35:29.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:28 smithi114 ceph-mon[32038]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:29.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:28 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6836/1221779710,v1:172.21.15.114:6837/1221779710] up:boot 2024-05-27T07:35:29.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:28 smithi114 ceph-mon[32038]: fsmap 4 up:standby 2024-05-27T07:35:29.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:35:29.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:29.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:29.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:28 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/500356870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:29.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:29.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:29.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:29.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:28 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:29.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:28 smithi114 ceph-mon[32038]: Deploying daemon mds.e on smithi163 2024-05-27T07:35:30.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:35:30 smithi163 systemd[1]: Starting Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:35:30.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:35:30.771 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:35:30 smithi163 podman[65701]: 2024-05-27 07:35:30.399759714 +0000 UTC m=+0.017592984 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:35:30.771 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:35:30 smithi163 podman[65701]: 2024-05-27 07:35:30.514237847 +0000 UTC m=+0.132071109 container create 4e04f6774a3a1923d89bb5c20563d34f2fb5a54932f7537f27b6b8aa6378e890 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-27T07:35:30.962 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:31.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:30 smithi114 ceph-mon[32038]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:31.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:30 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:31.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:30 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:31.213 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:30 smithi163 ceph-mon[36451]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:31.214 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:30 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:31.214 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:30 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:31.214 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:35:30 smithi163 podman[65701]: 2024-05-27 07:35:30.775243607 +0000 UTC m=+0.393076867 container init 4e04f6774a3a1923d89bb5c20563d34f2fb5a54932f7537f27b6b8aa6378e890 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T07:35:31.214 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:35:30 smithi163 podman[65701]: 2024-05-27 07:35:30.778506966 +0000 UTC m=+0.396340232 container start 4e04f6774a3a1923d89bb5c20563d34f2fb5a54932f7537f27b6b8aa6378e890 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-27T07:35:31.214 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:35:30 smithi163 ceph-mds[65721]: -- 172.21.15.163:0/3673621159 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55deb1cc9a40 con 0x55deb1d11600 2024-05-27T07:35:31.214 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:35:30 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: starting mds.e at 2024-05-27T07:35:31.214 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:35:30 smithi163 bash[65701]: 4e04f6774a3a1923d89bb5c20563d34f2fb5a54932f7537f27b6b8aa6378e890 2024-05-27T07:35:31.214 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:35:30 smithi163 systemd[1]: Started Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:35:31.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:30 smithi186 ceph-mon[36647]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:31.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:30 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:31.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:30 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:32.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:31 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6834/2110370477,v1:172.21.15.163:6835/2110370477] up:boot 2024-05-27T07:35:32.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:31 smithi163 ceph-mon[36451]: fsmap 5 up:standby 2024-05-27T07:35:32.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:31 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:35:32.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:31 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:32.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:31 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:32.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:31 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:32.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:31 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:32.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:31 smithi163 ceph-mon[36451]: Deploying daemon mds.f on smithi186 2024-05-27T07:35:32.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:31 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6834/2110370477,v1:172.21.15.163:6835/2110370477] up:boot 2024-05-27T07:35:32.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:31 smithi186 ceph-mon[36647]: fsmap 5 up:standby 2024-05-27T07:35:32.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:31 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:35:32.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:31 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:32.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:31 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:32.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:31 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:32.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:31 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:32.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:31 smithi186 ceph-mon[36647]: Deploying daemon mds.f on smithi186 2024-05-27T07:35:32.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:31 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6834/2110370477,v1:172.21.15.163:6835/2110370477] up:boot 2024-05-27T07:35:32.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:31 smithi114 ceph-mon[32038]: fsmap 5 up:standby 2024-05-27T07:35:32.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:31 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:35:32.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:31 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:32.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:31 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:32.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:31 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:32.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:31 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:32.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:31 smithi114 ceph-mon[32038]: Deploying daemon mds.f on smithi186 2024-05-27T07:35:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.572+0000 7f694fd38700 1 -- 172.21.15.114:0/1898550039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6948107de0 msgr2=0x7f69481081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.572+0000 7f694fd38700 1 --2- 172.21.15.114:0/1898550039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6948107de0 0x7f69481081f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f693c0099c0 tx=0x7f693c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:35:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.572+0000 7f694fd38700 1 -- 172.21.15.114:0/1898550039 shutdown_connections 2024-05-27T07:35:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.572+0000 7f694fd38700 1 --2- 172.21.15.114:0/1898550039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6948108730 0x7f6948110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.572+0000 7f694fd38700 1 --2- 172.21.15.114:0/1898550039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69481034b0 0x7f69481038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.572+0000 7f694fd38700 1 --2- 172.21.15.114:0/1898550039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6948107de0 0x7f69481081f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.572+0000 7f694fd38700 1 -- 172.21.15.114:0/1898550039 >> 172.21.15.114:0/1898550039 conn(0x7f69480fede0 msgr2=0x7f6948101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.572+0000 7f694fd38700 1 -- 172.21.15.114:0/1898550039 shutdown_connections 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.572+0000 7f694fd38700 1 -- 172.21.15.114:0/1898550039 wait complete. 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.573+0000 7f694fd38700 1 Processor -- start 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.573+0000 7f694fd38700 1 -- start start 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.573+0000 7f694fd38700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69481034b0 0x7f694819bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.573+0000 7f694fd38700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6948107de0 0x7f694819c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.573+0000 7f694fd38700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6948108730 0x7f694819c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.573+0000 7f694fd38700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f69481133d0 con 0x7f6948107de0 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.573+0000 7f694fd38700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6948113250 con 0x7f69481034b0 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.573+0000 7f694fd38700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6948113550 con 0x7f6948108730 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.573+0000 7f694d2d3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6948107de0 0x7f694819c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.573+0000 7f694d2d3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6948107de0 0x7f694819c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44384/0 (socket says 172.21.15.114:44384) 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.573+0000 7f694d2d3700 1 -- 172.21.15.114:0/785838740 learned_addr learned my addr 172.21.15.114:0/785838740 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:35:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.573+0000 7f694e2d5700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6948108730 0x7f694819c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:32.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.573+0000 7f694dad4700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69481034b0 0x7f694819bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:32.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.574+0000 7f694d2d3700 1 -- 172.21.15.114:0/785838740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6948108730 msgr2=0x7f694819c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:32.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.574+0000 7f694d2d3700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6948108730 0x7f694819c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:32.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.574+0000 7f694d2d3700 1 -- 172.21.15.114:0/785838740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69481034b0 msgr2=0x7f694819bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:32.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.574+0000 7f694d2d3700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69481034b0 0x7f694819bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:32.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.574+0000 7f694d2d3700 1 -- 172.21.15.114:0/785838740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69481a7020 con 0x7f6948107de0 2024-05-27T07:35:32.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.574+0000 7f694d2d3700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6948107de0 0x7f694819c1f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f693c006010 tx=0x7f693c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:32.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.574+0000 7f693affd700 1 -- 172.21.15.114:0/785838740 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c027070 con 0x7f6948107de0 2024-05-27T07:35:32.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.574+0000 7f693affd700 1 -- 172.21.15.114:0/785838740 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f693c003f20 con 0x7f6948107de0 2024-05-27T07:35:32.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.574+0000 7f694fd38700 1 -- 172.21.15.114:0/785838740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69481a72b0 con 0x7f6948107de0 2024-05-27T07:35:32.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.574+0000 7f693affd700 1 -- 172.21.15.114:0/785838740 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c022cd0 con 0x7f6948107de0 2024-05-27T07:35:32.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.574+0000 7f694fd38700 1 -- 172.21.15.114:0/785838740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69481a7760 con 0x7f6948107de0 2024-05-27T07:35:32.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.575+0000 7f694fd38700 1 -- 172.21.15.114:0/785838740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69480512b0 con 0x7f6948107de0 2024-05-27T07:35:32.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.576+0000 7f693affd700 1 -- 172.21.15.114:0/785838740 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f693c02a070 con 0x7f6948107de0 2024-05-27T07:35:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.577+0000 7f693affd700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f69340b2900 0x7f69340b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.577+0000 7f694dad4700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f69340b2900 0x7f69340b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.577+0000 7f693affd700 1 -- 172.21.15.114:0/785838740 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f693c0e2f50 con 0x7f6948107de0 2024-05-27T07:35:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.578+0000 7f694dad4700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f69340b2900 0x7f69340b4e10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6944006fd0 tx=0x7f6944008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:32.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.581+0000 7f693affd700 1 -- 172.21.15.114:0/785838740 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f693c031080 con 0x7f6948107de0 2024-05-27T07:35:32.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.832+0000 7f694fd38700 1 -- 172.21.15.114:0/785838740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f694802e5e0 con 0x7f6948107de0 2024-05-27T07:35:32.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.832+0000 7f693affd700 1 -- 172.21.15.114:0/785838740 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4705 (secure 0 0 0) 0x7f693c013070 con 0x7f6948107de0 2024-05-27T07:35:32.828 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:35:32.828 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-27T07:35:32.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.835+0000 7f694fd38700 1 -- 172.21.15.114:0/785838740 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f69340b2900 msgr2=0x7f69340b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:32.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.835+0000 7f694fd38700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f69340b2900 0x7f69340b4e10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6944006fd0 tx=0x7f6944008040 comp rx=0 tx=0).stop 2024-05-27T07:35:32.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.835+0000 7f694fd38700 1 -- 172.21.15.114:0/785838740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6948107de0 msgr2=0x7f694819c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:32.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.835+0000 7f694fd38700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6948107de0 0x7f694819c1f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f693c006010 tx=0x7f693c0047b0 comp rx=0 tx=0).stop 2024-05-27T07:35:32.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.835+0000 7f694fd38700 1 -- 172.21.15.114:0/785838740 shutdown_connections 2024-05-27T07:35:32.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.835+0000 7f694fd38700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f69340b2900 0x7f69340b4e10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:32.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.835+0000 7f694fd38700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6948108730 0x7f694819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:32.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.835+0000 7f694fd38700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69481034b0 0x7f694819bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:32.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.835+0000 7f694fd38700 1 --2- 172.21.15.114:0/785838740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6948107de0 0x7f694819c1f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:32.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.835+0000 7f694fd38700 1 -- 172.21.15.114:0/785838740 >> 172.21.15.114:0/785838740 conn(0x7f69480fede0 msgr2=0x7f6948109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:32.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.835+0000 7f694fd38700 1 -- 172.21.15.114:0/785838740 shutdown_connections 2024-05-27T07:35:32.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:32.835+0000 7f694fd38700 1 -- 172.21.15.114:0/785838740 wait complete. 2024-05-27T07:35:32.832 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 6 2024-05-27T07:35:33.193 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:32 smithi114 ceph-mon[32038]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:33.193 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:32 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/785838740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:33.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:32 smithi163 ceph-mon[36451]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:33.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:32 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/785838740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:33.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:32 smithi186 ceph-mon[36647]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:33.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:32 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/785838740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:33.224 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:35:33 smithi186 systemd[1]: Starting Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:35:33.505 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:35:33 smithi186 podman[67374]: 2024-05-27 07:35:33.138500135 +0000 UTC m=+0.017401105 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:35:33.505 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:35:33 smithi186 podman[67374]: 2024-05-27 07:35:33.239609832 +0000 UTC m=+0.118510813 container create 65fabfafe5eab5db11d7a13dfbf51d877e354d13d6c287d4c1b5e55f3318c679 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:35:33.505 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:35:33 smithi186 podman[67374]: 2024-05-27 07:35:33.510498409 +0000 UTC m=+0.389399347 container init 65fabfafe5eab5db11d7a13dfbf51d877e354d13d6c287d4c1b5e55f3318c679 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-27T07:35:33.974 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:35:33 smithi186 podman[67374]: 2024-05-27 07:35:33.513178123 +0000 UTC m=+0.392079062 container start 65fabfafe5eab5db11d7a13dfbf51d877e354d13d6c287d4c1b5e55f3318c679 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:35:33.974 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:35:33 smithi186 ceph-mds[67393]: -- 172.21.15.186:0/2707409922 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e4ad19ba40 con 0x55e4ad1e3a80 2024-05-27T07:35:33.974 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:35:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: starting mds.f at 2024-05-27T07:35:33.974 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:35:33 smithi186 bash[67374]: 65fabfafe5eab5db11d7a13dfbf51d877e354d13d6c287d4c1b5e55f3318c679 2024-05-27T07:35:33.974 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:35:33 smithi186 systemd[1]: Started Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:35:34.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:34 smithi163 ceph-mon[36451]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:34.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:34 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:34.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:34 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:34.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:34 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:34.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:34 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:34.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:34 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:34.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:34 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:34.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:34 smithi163 ceph-mon[36451]: Deploying daemon mds.c on smithi186 2024-05-27T07:35:34.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:34 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6834/199200469,v1:172.21.15.186:6835/199200469] up:boot 2024-05-27T07:35:34.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:34 smithi163 ceph-mon[36451]: fsmap 6 up:standby 2024-05-27T07:35:34.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:34 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:35:34.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:34 smithi186 ceph-mon[36647]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:34.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:34 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:34.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:34 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:34.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:34 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:34.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:34 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:34.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:34 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:34.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:34 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:34.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:34 smithi186 ceph-mon[36647]: Deploying daemon mds.c on smithi186 2024-05-27T07:35:34.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:34 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6834/199200469,v1:172.21.15.186:6835/199200469] up:boot 2024-05-27T07:35:34.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:34 smithi186 ceph-mon[36647]: fsmap 6 up:standby 2024-05-27T07:35:34.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:34 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:35:35.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:34 smithi114 ceph-mon[32038]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:35.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:34 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:35.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:34 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:35.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:34 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:35.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:34 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:35.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:34 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:35.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:34 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:35.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:34 smithi114 ceph-mon[32038]: Deploying daemon mds.c on smithi186 2024-05-27T07:35:35.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:34 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6834/199200469,v1:172.21.15.186:6835/199200469] up:boot 2024-05-27T07:35:35.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:34 smithi114 ceph-mon[32038]: fsmap 6 up:standby 2024-05-27T07:35:35.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:34 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:35:35.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:35:35.749 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:35.974 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:35:35 smithi186 systemd[1]: Starting Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:35:36.261 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:35:36 smithi186 podman[67676]: 2024-05-27 07:35:36.014103314 +0000 UTC m=+0.109929050 container create a03d8aed320f71d84db154665a48737696953dec88b4e314ecf575f6616dc1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-05-27T07:35:36.261 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:35:36 smithi186 podman[67676]: 2024-05-27 07:35:35.921414061 +0000 UTC m=+0.017239815 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:35:36.261 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:35:36 smithi186 podman[67676]: 2024-05-27 07:35:36.266904534 +0000 UTC m=+0.362730303 container init a03d8aed320f71d84db154665a48737696953dec88b4e314ecf575f6616dc1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:35:36.664 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:35:36 smithi186 podman[67676]: 2024-05-27 07:35:36.269531494 +0000 UTC m=+0.365357239 container start a03d8aed320f71d84db154665a48737696953dec88b4e314ecf575f6616dc1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0) 2024-05-27T07:35:36.664 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:35:36 smithi186 ceph-mds[67695]: -- 172.21.15.186:0/4129171805 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565059dbba40 con 0x565059e03180 2024-05-27T07:35:36.664 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:35:36 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: starting mds.c at 2024-05-27T07:35:36.664 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:35:36 smithi186 bash[67676]: a03d8aed320f71d84db154665a48737696953dec88b4e314ecf575f6616dc1d0 2024-05-27T07:35:36.664 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:35:36 smithi186 systemd[1]: Started Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:35:36.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:36 smithi163 ceph-mon[36451]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:36.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:36.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:36.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:36.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:36.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:36.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:36 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:36.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:36 smithi186 ceph-mon[36647]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:36.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:36.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:36.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:36.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:36.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:36.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:36 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:37.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:36 smithi114 ceph-mon[32038]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:37.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:37.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:37.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:37.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:37.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:37.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:36 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:37.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.404+0000 7f58f87c9700 1 -- 172.21.15.114:0/2078846928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f0108ef0 msgr2=0x7f58f0109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:37.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.404+0000 7f58f87c9700 1 --2- 172.21.15.114:0/2078846928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f0108ef0 0x7f58f0109300 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f58e40099c0 tx=0x7f58e4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:35:37.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.404+0000 7f58f87c9700 1 -- 172.21.15.114:0/2078846928 shutdown_connections 2024-05-27T07:35:37.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.404+0000 7f58f87c9700 1 --2- 172.21.15.114:0/2078846928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58f0109840 0x7f58f01111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:37.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.404+0000 7f58f87c9700 1 --2- 172.21.15.114:0/2078846928 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58f01045c0 0x7f58f01049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:37.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.404+0000 7f58f87c9700 1 --2- 172.21.15.114:0/2078846928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f0108ef0 0x7f58f0109300 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:37.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.404+0000 7f58f87c9700 1 -- 172.21.15.114:0/2078846928 >> 172.21.15.114:0/2078846928 conn(0x7f58f00ffe90 msgr2=0x7f58f01022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:37.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.404+0000 7f58f87c9700 1 -- 172.21.15.114:0/2078846928 shutdown_connections 2024-05-27T07:35:37.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.404+0000 7f58f87c9700 1 -- 172.21.15.114:0/2078846928 wait complete. 2024-05-27T07:35:37.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f87c9700 1 Processor -- start 2024-05-27T07:35:37.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f87c9700 1 -- start start 2024-05-27T07:35:37.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f87c9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58f01045c0 0x7f58f019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:37.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f87c9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58f0108ef0 0x7f58f019b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:37.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f87c9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f0109840 0x7f58f019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:37.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f87c9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f58f0114450 con 0x7f58f0109840 2024-05-27T07:35:37.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f87c9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f58f01142d0 con 0x7f58f01045c0 2024-05-27T07:35:37.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f87c9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f58f01145d0 con 0x7f58f0108ef0 2024-05-27T07:35:37.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f6d66700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f0109840 0x7f58f019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:37.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f6d66700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f0109840 0x7f58f019f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39872/0 (socket says 172.21.15.114:39872) 2024-05-27T07:35:37.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f6d66700 1 -- 172.21.15.114:0/3299303235 learned_addr learned my addr 172.21.15.114:0/3299303235 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:35:37.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f6565700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58f01045c0 0x7f58f019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:37.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f5d64700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58f0108ef0 0x7f58f019b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:37.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f6d66700 1 -- 172.21.15.114:0/3299303235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58f0108ef0 msgr2=0x7f58f019b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:37.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f6d66700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58f0108ef0 0x7f58f019b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:37.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f6d66700 1 -- 172.21.15.114:0/3299303235 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58f01045c0 msgr2=0x7f58f019ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:37.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f6d66700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58f01045c0 0x7f58f019ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:37.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f6d66700 1 -- 172.21.15.114:0/3299303235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f019fcb0 con 0x7f58f0109840 2024-05-27T07:35:37.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.405+0000 7f58f6565700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58f01045c0 0x7f58f019ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:37.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.406+0000 7f58f6d66700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f0109840 0x7f58f019f610 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f58e000c970 tx=0x7f58e000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:37.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.406+0000 7f58ef7fe700 1 -- 172.21.15.114:0/3299303235 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e000ce60 con 0x7f58f0109840 2024-05-27T07:35:37.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.406+0000 7f58ef7fe700 1 -- 172.21.15.114:0/3299303235 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58e0004830 con 0x7f58f0109840 2024-05-27T07:35:37.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.406+0000 7f58f87c9700 1 -- 172.21.15.114:0/3299303235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f019ffa0 con 0x7f58f0109840 2024-05-27T07:35:37.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.406+0000 7f58f87c9700 1 -- 172.21.15.114:0/3299303235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f01a78c0 con 0x7f58f0109840 2024-05-27T07:35:37.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.406+0000 7f58ef7fe700 1 -- 172.21.15.114:0/3299303235 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e00054c0 con 0x7f58f0109840 2024-05-27T07:35:37.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.407+0000 7f58f87c9700 1 -- 172.21.15.114:0/3299303235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f00512b0 con 0x7f58f0109840 2024-05-27T07:35:37.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.408+0000 7f58ef7fe700 1 -- 172.21.15.114:0/3299303235 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f58e00049e0 con 0x7f58f0109840 2024-05-27T07:35:37.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.410+0000 7f58ef7fe700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f58d80b2900 0x7f58d80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:37.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.410+0000 7f58f6565700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f58d80b2900 0x7f58d80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:37.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.410+0000 7f58ef7fe700 1 -- 172.21.15.114:0/3299303235 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f58e00e1cb0 con 0x7f58f0109840 2024-05-27T07:35:37.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.410+0000 7f58f6565700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f58d80b2900 0x7f58d80b4e10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009dd0 tx=0x7f58e8009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:37.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.413+0000 7f58ef7fe700 1 -- 172.21.15.114:0/3299303235 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f58e0014030 con 0x7f58f0109840 2024-05-27T07:35:37.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.661+0000 7f58f87c9700 1 -- 172.21.15.114:0/3299303235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58f019c140 con 0x7f58f0109840 2024-05-27T07:35:37.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.662+0000 7f58ef7fe700 1 -- 172.21.15.114:0/3299303235 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6358 (secure 0 0 0) 0x7f58e00a9e50 con 0x7f58f0109840 2024-05-27T07:35:37.658 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:35:37.658 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-05-27T07:35:37.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.664+0000 7f58f87c9700 1 -- 172.21.15.114:0/3299303235 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f58d80b2900 msgr2=0x7f58d80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:37.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.664+0000 7f58f87c9700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f58d80b2900 0x7f58d80b4e10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009dd0 tx=0x7f58e8009040 comp rx=0 tx=0).stop 2024-05-27T07:35:37.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.664+0000 7f58f87c9700 1 -- 172.21.15.114:0/3299303235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f0109840 msgr2=0x7f58f019f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:37.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.664+0000 7f58f87c9700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f0109840 0x7f58f019f610 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f58e000c970 tx=0x7f58e000cc80 comp rx=0 tx=0).stop 2024-05-27T07:35:37.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.664+0000 7f58f87c9700 1 -- 172.21.15.114:0/3299303235 shutdown_connections 2024-05-27T07:35:37.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.664+0000 7f58f87c9700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f0109840 0x7f58f019f610 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:37.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.664+0000 7f58f87c9700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58f01045c0 0x7f58f019ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:37.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.664+0000 7f58f87c9700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f58d80b2900 0x7f58d80b4e10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:37.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.664+0000 7f58f87c9700 1 --2- 172.21.15.114:0/3299303235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58f0108ef0 0x7f58f019b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:37.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.664+0000 7f58f87c9700 1 -- 172.21.15.114:0/3299303235 >> 172.21.15.114:0/3299303235 conn(0x7f58f00ffe90 msgr2=0x7f58f01086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:37.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.665+0000 7f58f87c9700 1 -- 172.21.15.114:0/3299303235 shutdown_connections 2024-05-27T07:35:37.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:37.665+0000 7f58f87c9700 1 -- 172.21.15.114:0/3299303235 wait complete. 2024-05-27T07:35:37.661 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 8 2024-05-27T07:35:37.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:37 smithi163 ceph-mon[36451]: Deploying daemon mds.h on smithi163 2024-05-27T07:35:37.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:37 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6836/2251439141,v1:172.21.15.186:6837/2251439141] up:boot 2024-05-27T07:35:37.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:37 smithi163 ceph-mon[36451]: fsmap 7 up:standby 2024-05-27T07:35:37.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:37 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:35:37.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:37 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3299303235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:37.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:37 smithi186 ceph-mon[36647]: Deploying daemon mds.h on smithi163 2024-05-27T07:35:37.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:37 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6836/2251439141,v1:172.21.15.186:6837/2251439141] up:boot 2024-05-27T07:35:37.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:37 smithi186 ceph-mon[36647]: fsmap 7 up:standby 2024-05-27T07:35:37.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:37 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:35:37.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:37 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3299303235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:37.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:37 smithi114 ceph-mon[32038]: Deploying daemon mds.h on smithi163 2024-05-27T07:35:37.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:37 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6836/2251439141,v1:172.21.15.186:6837/2251439141] up:boot 2024-05-27T07:35:37.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:37 smithi114 ceph-mon[32038]: fsmap 7 up:standby 2024-05-27T07:35:37.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:37 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:35:37.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:37 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3299303235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:38.920 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:38 smithi163 ceph-mon[36451]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:38.920 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:35:38 smithi163 systemd[1]: Starting Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:35:38.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:38 smithi186 ceph-mon[36647]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:39.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:38 smithi114 ceph-mon[32038]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:39.208 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:35:38 smithi163 podman[66009]: 2024-05-27 07:35:38.82816156 +0000 UTC m=+0.017487555 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:35:39.208 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:35:38 smithi163 podman[66009]: 2024-05-27 07:35:38.944388676 +0000 UTC m=+0.133714648 container create ea2b072a7ecad398048b1bbed426087e7c26f0cf040561a2993df10d8bad7424 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-27T07:35:39.208 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:35:39 smithi163 podman[66009]: 2024-05-27 07:35:39.213696114 +0000 UTC m=+0.403022093 container init ea2b072a7ecad398048b1bbed426087e7c26f0cf040561a2993df10d8bad7424 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True) 2024-05-27T07:35:39.471 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:35:39 smithi163 podman[66009]: 2024-05-27 07:35:39.216436537 +0000 UTC m=+0.405762522 container start ea2b072a7ecad398048b1bbed426087e7c26f0cf040561a2993df10d8bad7424 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-05-27T07:35:39.471 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:35:39 smithi163 ceph-mds[66028]: -- 172.21.15.163:0/1875797517 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56223e549a40 con 0x56223e591a80 2024-05-27T07:35:39.471 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:35:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: starting mds.h at 2024-05-27T07:35:39.471 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:35:39 smithi163 bash[66009]: ea2b072a7ecad398048b1bbed426087e7c26f0cf040561a2993df10d8bad7424 2024-05-27T07:35:39.471 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:35:39 smithi163 systemd[1]: Started Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:35:40.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:35:40.552 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:40.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:40 smithi114 ceph-mon[32038]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:40.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:40.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:40.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:40.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:40.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:40.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:40.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:40 smithi114 ceph-mon[32038]: Deploying daemon mds.i on smithi186 2024-05-27T07:35:40.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:40 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6836/869688838,v1:172.21.15.163:6837/869688838] up:boot 2024-05-27T07:35:40.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:40 smithi114 ceph-mon[32038]: fsmap 8 up:standby 2024-05-27T07:35:40.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:40 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:35:40.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:40 smithi163 ceph-mon[36451]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:40.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:40.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:40.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:40.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:40.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:40.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:40.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:40 smithi163 ceph-mon[36451]: Deploying daemon mds.i on smithi186 2024-05-27T07:35:40.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:40 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6836/869688838,v1:172.21.15.163:6837/869688838] up:boot 2024-05-27T07:35:40.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:40 smithi163 ceph-mon[36451]: fsmap 8 up:standby 2024-05-27T07:35:40.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:40 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:35:40.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:40 smithi186 ceph-mon[36647]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:40.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:40.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:40.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:40.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:40.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:40.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:40.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:40 smithi186 ceph-mon[36647]: Deploying daemon mds.i on smithi186 2024-05-27T07:35:40.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:40 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6836/869688838,v1:172.21.15.163:6837/869688838] up:boot 2024-05-27T07:35:40.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:40 smithi186 ceph-mon[36647]: fsmap 8 up:standby 2024-05-27T07:35:40.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:40 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:35:41.724 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 07:35:41 smithi186 systemd[1]: Starting Ceph mds.i for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:35:41.994 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 07:35:41 smithi186 podman[67978]: 2024-05-27 07:35:41.633837377 +0000 UTC m=+0.017062073 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:35:41.994 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 07:35:41 smithi186 podman[67978]: 2024-05-27 07:35:41.746681086 +0000 UTC m=+0.129905771 container create a502f02b07474e87e2c9b1a76eaa4c55a6a6caa72b9fc9185d0e9f645df63997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-i, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-27T07:35:41.994 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 07:35:41 smithi186 podman[67978]: 2024-05-27 07:35:41.999462966 +0000 UTC m=+0.382687651 container init a502f02b07474e87e2c9b1a76eaa4c55a6a6caa72b9fc9185d0e9f645df63997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0) 2024-05-27T07:35:42.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.224+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/2350701562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a410e8c0 msgr2=0x7fb3a4110da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.224+0000 7fb3aa2cb700 1 --2- 172.21.15.114:0/2350701562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a410e8c0 0x7fb3a4110da0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00b0a0 tx=0x7fb39c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.224+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/2350701562 shutdown_connections 2024-05-27T07:35:42.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.224+0000 7fb3aa2cb700 1 --2- 172.21.15.114:0/2350701562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a410e8c0 0x7fb3a4110da0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.224+0000 7fb3aa2cb700 1 --2- 172.21.15.114:0/2350701562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3a4105450 0x7fb3a4105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.224+0000 7fb3aa2cb700 1 --2- 172.21.15.114:0/2350701562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3a4105da0 0x7fb3a410e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.224+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/2350701562 >> 172.21.15.114:0/2350701562 conn(0x7fb3a40fedb0 msgr2=0x7fb3a41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:42.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.224+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/2350701562 shutdown_connections 2024-05-27T07:35:42.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.224+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/2350701562 wait complete. 2024-05-27T07:35:42.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.224+0000 7fb3aa2cb700 1 Processor -- start 2024-05-27T07:35:42.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.224+0000 7fb3aa2cb700 1 -- start start 2024-05-27T07:35:42.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3aa2cb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a4105450 0x7fb3a419ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3aa2cb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3a4105da0 0x7fb3a419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3aa2cb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3a410e8c0 0x7fb3a419f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3aa2cb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a4114420 con 0x7fb3a4105450 2024-05-27T07:35:42.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3aa2cb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a41142a0 con 0x7fb3a4105da0 2024-05-27T07:35:42.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3aa2cb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a41145a0 con 0x7fb3a410e8c0 2024-05-27T07:35:42.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3a3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a4105450 0x7fb3a419ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3a3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a4105450 0x7fb3a419ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39894/0 (socket says 172.21.15.114:39894) 2024-05-27T07:35:42.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3a3fff700 1 -- 172.21.15.114:0/1204530806 learned_addr learned my addr 172.21.15.114:0/1204530806 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:35:42.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3a8868700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3a410e8c0 0x7fb3a419f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb39bfff700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3a4105da0 0x7fb3a419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3a3fff700 1 -- 172.21.15.114:0/1204530806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3a410e8c0 msgr2=0x7fb3a419f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3a3fff700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3a410e8c0 0x7fb3a419f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3a3fff700 1 -- 172.21.15.114:0/1204530806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3a4105da0 msgr2=0x7fb3a419b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3a3fff700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3a4105da0 0x7fb3a419b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb3a3fff700 1 -- 172.21.15.114:0/1204530806 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a419fc40 con 0x7fb3a4105450 2024-05-27T07:35:42.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.225+0000 7fb39bfff700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3a4105da0 0x7fb3a419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:42.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.226+0000 7fb3a3fff700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a4105450 0x7fb3a419ad70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb39400d950 tx=0x7fb39400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:42.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.226+0000 7fb3a1ffb700 1 -- 172.21.15.114:0/1204530806 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb394016070 con 0x7fb3a4105450 2024-05-27T07:35:42.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.226+0000 7fb3a1ffb700 1 -- 172.21.15.114:0/1204530806 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb394009df0 con 0x7fb3a4105450 2024-05-27T07:35:42.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.226+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/1204530806 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a419ff30 con 0x7fb3a4105450 2024-05-27T07:35:42.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.226+0000 7fb3a1ffb700 1 -- 172.21.15.114:0/1204530806 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39400f3c0 con 0x7fb3a4105450 2024-05-27T07:35:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.226+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/1204530806 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a41a7850 con 0x7fb3a4105450 2024-05-27T07:35:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.227+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/1204530806 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3a40512b0 con 0x7fb3a4105450 2024-05-27T07:35:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.229+0000 7fb3a1ffb700 1 -- 172.21.15.114:0/1204530806 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fb394019070 con 0x7fb3a4105450 2024-05-27T07:35:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.230+0000 7fb3a1ffb700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb3900b2880 0x7fb3900b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.230+0000 7fb39bfff700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb3900b2880 0x7fb3900b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.230+0000 7fb3a1ffb700 1 -- 172.21.15.114:0/1204530806 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fb3940e1980 con 0x7fb3a4105450 2024-05-27T07:35:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.231+0000 7fb39bfff700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb3900b2880 0x7fb3900b4d90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb3a41022a0 tx=0x7fb38c005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:42.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.234+0000 7fb3a1ffb700 1 -- 172.21.15.114:0/1204530806 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb3940a9b40 con 0x7fb3a4105450 2024-05-27T07:35:42.474 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 07:35:42 smithi186 podman[67978]: 2024-05-27 07:35:42.002383494 +0000 UTC m=+0.385608191 container start a502f02b07474e87e2c9b1a76eaa4c55a6a6caa72b9fc9185d0e9f645df63997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T07:35:42.474 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 07:35:42 smithi186 ceph-mds[67998]: -- 172.21.15.186:0/2678002925 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556b49f97a40 con 0x556b49fdf180 2024-05-27T07:35:42.474 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 07:35:42 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-i[67994]: starting mds.i at 2024-05-27T07:35:42.474 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 07:35:42 smithi186 bash[67978]: a502f02b07474e87e2c9b1a76eaa4c55a6a6caa72b9fc9185d0e9f645df63997 2024-05-27T07:35:42.474 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 07:35:42 smithi186 systemd[1]: Started Ceph mds.i for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:35:42.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.495+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/1204530806 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3a402e5e0 con 0x7fb3a4105450 2024-05-27T07:35:42.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.496+0000 7fb3a1ffb700 1 -- 172.21.15.114:0/1204530806 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7fb3940a9440 con 0x7fb3a4105450 2024-05-27T07:35:42.491 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:35:42.492 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24463,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-27T07:35:42.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.498+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/1204530806 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb3900b2880 msgr2=0x7fb3900b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.498+0000 7fb3aa2cb700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb3900b2880 0x7fb3900b4d90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb3a41022a0 tx=0x7fb38c005d80 comp rx=0 tx=0).stop 2024-05-27T07:35:42.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.498+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/1204530806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a4105450 msgr2=0x7fb3a419ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.498+0000 7fb3aa2cb700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a4105450 0x7fb3a419ad70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb39400d950 tx=0x7fb39400dc60 comp rx=0 tx=0).stop 2024-05-27T07:35:42.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.498+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/1204530806 shutdown_connections 2024-05-27T07:35:42.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.498+0000 7fb3aa2cb700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb3900b2880 0x7fb3900b4d90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.498+0000 7fb3aa2cb700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3a410e8c0 0x7fb3a419f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.498+0000 7fb3aa2cb700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a4105450 0x7fb3a419ad70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.498+0000 7fb3aa2cb700 1 --2- 172.21.15.114:0/1204530806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3a4105da0 0x7fb3a419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.498+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/1204530806 >> 172.21.15.114:0/1204530806 conn(0x7fb3a40fedb0 msgr2=0x7fb3a40ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:42.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.498+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/1204530806 shutdown_connections 2024-05-27T07:35:42.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:42.498+0000 7fb3aa2cb700 1 -- 172.21.15.114:0/1204530806 wait complete. 2024-05-27T07:35:42.496 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 10 2024-05-27T07:35:43.419 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:43 smithi186 ceph-mon[36647]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:43.419 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:43.419 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:43.419 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:43.419 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:43.420 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:43.420 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:43.420 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:43 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] up:boot 2024-05-27T07:35:43.420 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:43 smithi186 ceph-mon[36647]: fsmap 9 up:standby 2024-05-27T07:35:43.420 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:43 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:35:43.420 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:43 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1204530806' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:43.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:43 smithi163 ceph-mon[36451]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:43.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:43.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:43.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:43.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:43.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:43.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:43.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:43 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] up:boot 2024-05-27T07:35:43.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:43 smithi163 ceph-mon[36451]: fsmap 9 up:standby 2024-05-27T07:35:43.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:43 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:35:43.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:43 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1204530806' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:43.508 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:43 smithi114 ceph-mon[32038]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:43.508 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:43.508 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:43.508 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:43.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:43.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:43.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:43.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:43 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] up:boot 2024-05-27T07:35:43.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:43 smithi114 ceph-mon[32038]: fsmap 9 up:standby 2024-05-27T07:35:43.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:43 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:35:43.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:43 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1204530806' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:44.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:44 smithi114 ceph-mon[32038]: Deploying daemon mds.g on smithi114 2024-05-27T07:35:44.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:44 smithi163 ceph-mon[36451]: Deploying daemon mds.g on smithi114 2024-05-27T07:35:44.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:44 smithi186 ceph-mon[36647]: Deploying daemon mds.g on smithi114 2024-05-27T07:35:45.051 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:35:44 smithi114 systemd[1]: Starting Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:35:45.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:35:45.335 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:45 smithi114 ceph-mon[32038]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:45.335 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:35:45 smithi114 podman[70937]: 2024-05-27 07:35:44.959773009 +0000 UTC m=+0.017379980 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:35:45.335 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:35:45 smithi114 podman[70937]: 2024-05-27 07:35:45.057312172 +0000 UTC m=+0.114919144 container create 5542c738fdbb55c1542dbd44aab6cfe2e3d865fbeb89bfc73ab14b6138196adb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True) 2024-05-27T07:35:45.335 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:35:45 smithi114 podman[70937]: 2024-05-27 07:35:45.302656171 +0000 UTC m=+0.360263155 container init 5542c738fdbb55c1542dbd44aab6cfe2e3d865fbeb89bfc73ab14b6138196adb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-27T07:35:45.335 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:35:45 smithi114 podman[70937]: 2024-05-27 07:35:45.306130997 +0000 UTC m=+0.363737972 container start 5542c738fdbb55c1542dbd44aab6cfe2e3d865fbeb89bfc73ab14b6138196adb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:35:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:45.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:45 smithi163 ceph-mon[36451]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:45.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:45 smithi186 ceph-mon[36647]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:45.590 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:35:45 smithi114 ceph-mds[70974]: -- 172.21.15.114:0/4257869382 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b42c631a40 con 0x55b42c679a80 2024-05-27T07:35:45.590 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:35:45 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: starting mds.g at 2024-05-27T07:35:45.590 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:35:45 smithi114 bash[70937]: 5542c738fdbb55c1542dbd44aab6cfe2e3d865fbeb89bfc73ab14b6138196adb 2024-05-27T07:35:45.590 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:35:45 smithi114 systemd[1]: Started Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:35:46.681 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:46 smithi114 ceph-mon[32038]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:46.681 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:46.681 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:46.681 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:46.681 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:46.681 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:46.681 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:46.681 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:46 smithi114 ceph-mon[32038]: Deploying daemon mds.a on smithi114 2024-05-27T07:35:46.681 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:46 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6838/2637229588,v1:172.21.15.114:6839/2637229588] up:boot 2024-05-27T07:35:46.682 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:46 smithi114 ceph-mon[32038]: fsmap 10 up:standby 2024-05-27T07:35:46.682 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:35:46.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:46 smithi163 ceph-mon[36451]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:46.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:46.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:46.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:46.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:46.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:46.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:46.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:46 smithi163 ceph-mon[36451]: Deploying daemon mds.a on smithi114 2024-05-27T07:35:46.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:46 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6838/2637229588,v1:172.21.15.114:6839/2637229588] up:boot 2024-05-27T07:35:46.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:46 smithi163 ceph-mon[36451]: fsmap 10 up:standby 2024-05-27T07:35:46.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:35:46.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:46 smithi186 ceph-mon[36647]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:46.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:46.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:46.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:46.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:46.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:46.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:46.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:46 smithi186 ceph-mon[36647]: Deploying daemon mds.a on smithi114 2024-05-27T07:35:46.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:46 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6838/2637229588,v1:172.21.15.114:6839/2637229588] up:boot 2024-05-27T07:35:46.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:46 smithi186 ceph-mon[36647]: fsmap 10 up:standby 2024-05-27T07:35:46.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:35:47.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.757+0000 7efda8ea4700 1 -- 172.21.15.114:0/3456474244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efda4103300 msgr2=0x7efda410f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:47.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.757+0000 7efda8ea4700 1 --2- 172.21.15.114:0/3456474244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efda4103300 0x7efda410f880 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efd9400b0a0 tx=0x7efd9400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.757+0000 7efda8ea4700 1 -- 172.21.15.114:0/3456474244 shutdown_connections 2024-05-27T07:35:47.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.757+0000 7efda8ea4700 1 --2- 172.21.15.114:0/3456474244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efda4103300 0x7efda410f880 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.757+0000 7efda8ea4700 1 --2- 172.21.15.114:0/3456474244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efda41089e0 0x7efda4108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.757+0000 7efda8ea4700 1 --2- 172.21.15.114:0/3456474244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efda41029b0 0x7efda4102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.757+0000 7efda8ea4700 1 -- 172.21.15.114:0/3456474244 >> 172.21.15.114:0/3456474244 conn(0x7efda40fe4d0 msgr2=0x7efda4100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:47.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.757+0000 7efda8ea4700 1 -- 172.21.15.114:0/3456474244 shutdown_connections 2024-05-27T07:35:47.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.757+0000 7efda8ea4700 1 -- 172.21.15.114:0/3456474244 wait complete. 2024-05-27T07:35:47.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.758+0000 7efda8ea4700 1 Processor -- start 2024-05-27T07:35:47.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.758+0000 7efda8ea4700 1 -- start start 2024-05-27T07:35:47.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.758+0000 7efda8ea4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efda41029b0 0x7efda419add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.758+0000 7efda8ea4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efda4103300 0x7efda419b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.758+0000 7efda8ea4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efda41089e0 0x7efda419f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.758+0000 7efda8ea4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efda41144e0 con 0x7efda4103300 2024-05-27T07:35:47.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.758+0000 7efda8ea4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efda4114360 con 0x7efda41089e0 2024-05-27T07:35:47.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.758+0000 7efda8ea4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efda4114660 con 0x7efda41029b0 2024-05-27T07:35:47.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.758+0000 7efda1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efda4103300 0x7efda419b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.758+0000 7efda1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efda4103300 0x7efda419b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52370/0 (socket says 172.21.15.114:52370) 2024-05-27T07:35:47.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.758+0000 7efda1d9b700 1 -- 172.21.15.114:0/1517774324 learned_addr learned my addr 172.21.15.114:0/1517774324 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:35:47.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.758+0000 7efda259c700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efda41029b0 0x7efda419add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.758+0000 7efda2d9d700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efda41089e0 0x7efda419f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.759+0000 7efda1d9b700 1 -- 172.21.15.114:0/1517774324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efda41029b0 msgr2=0x7efda419add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:47.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.759+0000 7efda1d9b700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efda41029b0 0x7efda419add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.759+0000 7efda1d9b700 1 -- 172.21.15.114:0/1517774324 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efda41089e0 msgr2=0x7efda419f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:47.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.759+0000 7efda1d9b700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efda41089e0 0x7efda419f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.759+0000 7efda1d9b700 1 -- 172.21.15.114:0/1517774324 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda419fca0 con 0x7efda4103300 2024-05-27T07:35:47.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.759+0000 7efda1d9b700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efda4103300 0x7efda419b370 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efd8c00c8f0 tx=0x7efd8c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:47.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.759+0000 7efd9b7fe700 1 -- 172.21.15.114:0/1517774324 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8c007940 con 0x7efda4103300 2024-05-27T07:35:47.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.759+0000 7efd9b7fe700 1 -- 172.21.15.114:0/1517774324 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd8c00f450 con 0x7efda4103300 2024-05-27T07:35:47.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.759+0000 7efda8ea4700 1 -- 172.21.15.114:0/1517774324 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efda419ff90 con 0x7efda4103300 2024-05-27T07:35:47.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.759+0000 7efd9b7fe700 1 -- 172.21.15.114:0/1517774324 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8c018660 con 0x7efda4103300 2024-05-27T07:35:47.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.759+0000 7efda8ea4700 1 -- 172.21.15.114:0/1517774324 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efda41a78b0 con 0x7efda4103300 2024-05-27T07:35:47.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.760+0000 7efda8ea4700 1 -- 172.21.15.114:0/1517774324 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efda40512b0 con 0x7efda4103300 2024-05-27T07:35:47.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.762+0000 7efd9b7fe700 1 -- 172.21.15.114:0/1517774324 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7efd8c00f890 con 0x7efda4103300 2024-05-27T07:35:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.762+0000 7efd9b7fe700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7efd900b2880 0x7efd900b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.762+0000 7efda259c700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7efd900b2880 0x7efd900b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.762+0000 7efd9b7fe700 1 -- 172.21.15.114:0/1517774324 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7efd8c0e1d00 con 0x7efda4103300 2024-05-27T07:35:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.763+0000 7efda259c700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7efd900b2880 0x7efd900b4d90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efda4103af0 tx=0x7efd9c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:47.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:47.766+0000 7efd9b7fe700 1 -- 172.21.15.114:0/1517774324 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7efd8c014030 con 0x7efda4103300 2024-05-27T07:35:48.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.023+0000 7efda8ea4700 1 -- 172.21.15.114:0/1517774324 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efda419c130 con 0x7efda4103300 2024-05-27T07:35:48.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.023+0000 7efd9b7fe700 1 -- 172.21.15.114:0/1517774324 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8842 (secure 0 0 0) 0x7efd8c0a9f50 con 0x7efda4103300 2024-05-27T07:35:48.019 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:35:48.020 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24463,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-27T07:35:48.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.026+0000 7efda8ea4700 1 -- 172.21.15.114:0/1517774324 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7efd900b2880 msgr2=0x7efd900b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:48.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.026+0000 7efda8ea4700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7efd900b2880 0x7efd900b4d90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efda4103af0 tx=0x7efd9c009260 comp rx=0 tx=0).stop 2024-05-27T07:35:48.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.026+0000 7efda8ea4700 1 -- 172.21.15.114:0/1517774324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efda4103300 msgr2=0x7efda419b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:48.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.026+0000 7efda8ea4700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efda4103300 0x7efda419b370 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efd8c00c8f0 tx=0x7efd8c00cc00 comp rx=0 tx=0).stop 2024-05-27T07:35:48.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.026+0000 7efda8ea4700 1 -- 172.21.15.114:0/1517774324 shutdown_connections 2024-05-27T07:35:48.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.026+0000 7efda8ea4700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efda41089e0 0x7efda419f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:48.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.026+0000 7efda8ea4700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7efd900b2880 0x7efd900b4d90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:48.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.026+0000 7efda8ea4700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efda41029b0 0x7efda419add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:48.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.026+0000 7efda8ea4700 1 --2- 172.21.15.114:0/1517774324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efda4103300 0x7efda419b370 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:48.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.026+0000 7efda8ea4700 1 -- 172.21.15.114:0/1517774324 >> 172.21.15.114:0/1517774324 conn(0x7efda40fe4d0 msgr2=0x7efda4106cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:48.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.026+0000 7efda8ea4700 1 -- 172.21.15.114:0/1517774324 shutdown_connections 2024-05-27T07:35:48.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:48.026+0000 7efda8ea4700 1 -- 172.21.15.114:0/1517774324 wait complete. 2024-05-27T07:35:48.023 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 11 2024-05-27T07:35:48.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:48 smithi114 ceph-mon[32038]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:48.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:48 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1517774324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:48.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:48 smithi163 ceph-mon[36451]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:48.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:48 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1517774324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:48.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:48 smithi186 ceph-mon[36647]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:48.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:48 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1517774324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:49.801 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:35:49 smithi114 systemd[1]: Started Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:35:50.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:50 smithi163 ceph-mon[36451]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:50.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:50.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:50.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:50.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:50.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:50.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:50.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:50 smithi163 ceph-mon[36451]: Deploying daemon mds.b on smithi163 2024-05-27T07:35:50.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:50 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6840/3861955816,v1:172.21.15.114:6841/3861955816] up:boot 2024-05-27T07:35:50.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:50 smithi163 ceph-mon[36451]: fsmap 11 up:standby 2024-05-27T07:35:50.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:50 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:35:50.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:50 smithi186 ceph-mon[36647]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:50.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:50.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:50.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:50.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:50.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:50.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:50.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:50 smithi186 ceph-mon[36647]: Deploying daemon mds.b on smithi163 2024-05-27T07:35:50.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:50 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6840/3861955816,v1:172.21.15.114:6841/3861955816] up:boot 2024-05-27T07:35:50.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:50 smithi186 ceph-mon[36647]: fsmap 11 up:standby 2024-05-27T07:35:50.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:50 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:35:50.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:50 smithi114 ceph-mon[32038]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:50.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:50.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:50.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:50.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:35:50.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:35:50.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:50.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:50 smithi114 ceph-mon[32038]: Deploying daemon mds.b on smithi163 2024-05-27T07:35:50.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:50 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6840/3861955816,v1:172.21.15.114:6841/3861955816] up:boot 2024-05-27T07:35:50.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:50 smithi114 ceph-mon[32038]: fsmap 11 up:standby 2024-05-27T07:35:50.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:50 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:35:50.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:35:51.130 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:51.941 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:35:51 smithi163 systemd[1]: Starting Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:35:52.221 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:35:51 smithi163 podman[66309]: 2024-05-27 07:35:51.849301667 +0000 UTC m=+0.017499777 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:35:52.221 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:35:51 smithi163 podman[66309]: 2024-05-27 07:35:51.957887281 +0000 UTC m=+0.126085384 container create 9e612e37bd4f6b4935012bee7ba5a2727fb911092a3e708a73ca42252ef53211 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T07:35:52.221 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:35:52 smithi163 podman[66309]: 2024-05-27 07:35:52.193720757 +0000 UTC m=+0.361918859 container init 9e612e37bd4f6b4935012bee7ba5a2727fb911092a3e708a73ca42252ef53211 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, ceph=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:35:52.221 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:35:52 smithi163 podman[66309]: 2024-05-27 07:35:52.196251539 +0000 UTC m=+0.364449644 container start 9e612e37bd4f6b4935012bee7ba5a2727fb911092a3e708a73ca42252ef53211 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T07:35:52.520 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:52 smithi114 ceph-mon[32038]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:52.520 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.520 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.520 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.520 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.520 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:52 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:52 smithi163 ceph-mon[36451]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:52.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:52 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.722 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:35:52 smithi163 ceph-mds[66327]: -- 172.21.15.163:0/4236001469 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c6eec37a40 con 0x55c6eec7f600 2024-05-27T07:35:52.722 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:35:52 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: starting mds.b at 2024-05-27T07:35:52.722 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:35:52 smithi163 bash[66309]: 9e612e37bd4f6b4935012bee7ba5a2727fb911092a3e708a73ca42252ef53211 2024-05-27T07:35:52.722 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:35:52 smithi163 systemd[1]: Started Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:35:52.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:52 smithi186 ceph-mon[36647]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:52.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:52 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:52.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.777+0000 7f282559c700 1 -- 172.21.15.114:0/106298650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2820105e30 msgr2=0x7f282010e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:52.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.777+0000 7f282559c700 1 --2- 172.21.15.114:0/106298650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2820105e30 0x7f282010e380 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2814009990 tx=0x7f2814009ca0 comp rx=0 tx=0).stop 2024-05-27T07:35:52.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.777+0000 7f282559c700 1 -- 172.21.15.114:0/106298650 shutdown_connections 2024-05-27T07:35:52.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.777+0000 7f282559c700 1 --2- 172.21.15.114:0/106298650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f282010e910 0x7f2820110df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:52.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.777+0000 7f282559c700 1 --2- 172.21.15.114:0/106298650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28201054e0 0x7f28201058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:52.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.777+0000 7f282559c700 1 --2- 172.21.15.114:0/106298650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2820105e30 0x7f282010e380 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:52.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.777+0000 7f282559c700 1 -- 172.21.15.114:0/106298650 >> 172.21.15.114:0/106298650 conn(0x7f2820078df0 msgr2=0x7f2820079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:52.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.777+0000 7f282559c700 1 -- 172.21.15.114:0/106298650 shutdown_connections 2024-05-27T07:35:52.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.777+0000 7f282559c700 1 -- 172.21.15.114:0/106298650 wait complete. 2024-05-27T07:35:52.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.778+0000 7f282559c700 1 Processor -- start 2024-05-27T07:35:52.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.778+0000 7f282559c700 1 -- start start 2024-05-27T07:35:52.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.778+0000 7f282559c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28201054e0 0x7f28201031d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:52.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.778+0000 7f282559c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2820105e30 0x7f2820103710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:52.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.778+0000 7f282559c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f282010e910 0x7f2820103c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:52.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.778+0000 7f282559c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2820114470 con 0x7f28201054e0 2024-05-27T07:35:52.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.778+0000 7f282559c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f28201142f0 con 0x7f282010e910 2024-05-27T07:35:52.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.778+0000 7f282559c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28201145f0 con 0x7f2820105e30 2024-05-27T07:35:52.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f281effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28201054e0 0x7f28201031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:52.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f281effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28201054e0 0x7f28201031d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52388/0 (socket says 172.21.15.114:52388) 2024-05-27T07:35:52.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f281effd700 1 -- 172.21.15.114:0/3368061494 learned_addr learned my addr 172.21.15.114:0/3368061494 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:35:52.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f281f7fe700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f282010e910 0x7f2820103c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:52.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f281e7fc700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2820105e30 0x7f2820103710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:52.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f281effd700 1 -- 172.21.15.114:0/3368061494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2820105e30 msgr2=0x7f2820103710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:52.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f281effd700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2820105e30 0x7f2820103710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:52.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f281effd700 1 -- 172.21.15.114:0/3368061494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f282010e910 msgr2=0x7f2820103c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:52.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f281effd700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f282010e910 0x7f2820103c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:52.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f281effd700 1 -- 172.21.15.114:0/3368061494 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28201a9280 con 0x7f28201054e0 2024-05-27T07:35:52.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f281e7fc700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2820105e30 0x7f2820103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:35:52.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f281effd700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28201054e0 0x7f28201031d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f281000d950 tx=0x7f281000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:52.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f2807fff700 1 -- 172.21.15.114:0/3368061494 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2810016070 con 0x7f28201054e0 2024-05-27T07:35:52.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f2807fff700 1 -- 172.21.15.114:0/3368061494 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2810009df0 con 0x7f28201054e0 2024-05-27T07:35:52.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.779+0000 7f282559c700 1 -- 172.21.15.114:0/3368061494 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28201a9570 con 0x7f28201054e0 2024-05-27T07:35:52.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.780+0000 7f2807fff700 1 -- 172.21.15.114:0/3368061494 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281000f3c0 con 0x7f28201054e0 2024-05-27T07:35:52.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.780+0000 7f282559c700 1 -- 172.21.15.114:0/3368061494 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28201a9ab0 con 0x7f28201054e0 2024-05-27T07:35:52.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.781+0000 7f282559c700 1 -- 172.21.15.114:0/3368061494 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28200512b0 con 0x7f28201054e0 2024-05-27T07:35:52.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.783+0000 7f2807fff700 1 -- 172.21.15.114:0/3368061494 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f2810019070 con 0x7f28201054e0 2024-05-27T07:35:52.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.783+0000 7f2807fff700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f28080b2880 0x7f28080b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:52.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.783+0000 7f281e7fc700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f28080b2880 0x7f28080b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:52.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.783+0000 7f2807fff700 1 -- 172.21.15.114:0/3368061494 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f28100e1960 con 0x7f28201054e0 2024-05-27T07:35:52.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.784+0000 7f281e7fc700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f28080b2880 0x7f28080b4d90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2814003800 tx=0x7f2814005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:52.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:52.787+0000 7f2807fff700 1 -- 172.21.15.114:0/3368061494 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f28100a9b20 con 0x7f28201054e0 2024-05-27T07:35:53.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.042+0000 7f282559c700 1 -- 172.21.15.114:0/3368061494 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f282002e5e0 con 0x7f28201054e0 2024-05-27T07:35:53.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.043+0000 7f2807fff700 1 -- 172.21.15.114:0/3368061494 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10500 (secure 0 0 0) 0x7f28100a9420 con 0x7f28201054e0 2024-05-27T07:35:53.039 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:35:53.039 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24463,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-27T07:35:53.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.046+0000 7f282559c700 1 -- 172.21.15.114:0/3368061494 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f28080b2880 msgr2=0x7f28080b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:53.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.046+0000 7f282559c700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f28080b2880 0x7f28080b4d90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2814003800 tx=0x7f2814005fb0 comp rx=0 tx=0).stop 2024-05-27T07:35:53.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.046+0000 7f282559c700 1 -- 172.21.15.114:0/3368061494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28201054e0 msgr2=0x7f28201031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:53.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.046+0000 7f282559c700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28201054e0 0x7f28201031d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f281000d950 tx=0x7f281000dc60 comp rx=0 tx=0).stop 2024-05-27T07:35:53.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.046+0000 7f282559c700 1 -- 172.21.15.114:0/3368061494 shutdown_connections 2024-05-27T07:35:53.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.046+0000 7f282559c700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f282010e910 0x7f2820103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:53.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.046+0000 7f282559c700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28201054e0 0x7f28201031d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:53.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.046+0000 7f282559c700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f28080b2880 0x7f28080b4d90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:53.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.046+0000 7f282559c700 1 --2- 172.21.15.114:0/3368061494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2820105e30 0x7f2820103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:53.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.046+0000 7f282559c700 1 -- 172.21.15.114:0/3368061494 >> 172.21.15.114:0/3368061494 conn(0x7f2820078df0 msgr2=0x7f28201011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:53.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.047+0000 7f282559c700 1 -- 172.21.15.114:0/3368061494 shutdown_connections 2024-05-27T07:35:53.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:53.047+0000 7f282559c700 1 -- 172.21.15.114:0/3368061494 wait complete. 2024-05-27T07:35:53.043 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 13 2024-05-27T07:35:53.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:53 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6838/3774280856,v1:172.21.15.163:6839/3774280856] up:boot 2024-05-27T07:35:53.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:53 smithi163 ceph-mon[36451]: fsmap 12 up:standby 2024-05-27T07:35:53.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:53 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:35:53.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:53 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:53.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:53 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3368061494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:53.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:53 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:53.708 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:53 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6838/3774280856,v1:172.21.15.163:6839/3774280856] up:boot 2024-05-27T07:35:53.708 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:53 smithi114 ceph-mon[32038]: fsmap 12 up:standby 2024-05-27T07:35:53.708 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:53 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:35:53.708 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:53 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:53.709 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:53 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3368061494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:53.709 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:53 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:53.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:53 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6838/3774280856,v1:172.21.15.163:6839/3774280856] up:boot 2024-05-27T07:35:53.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:53 smithi186 ceph-mon[36647]: fsmap 12 up:standby 2024-05-27T07:35:53.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:53 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:35:53.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:53 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:53.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:53 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3368061494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:53.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:53 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:53.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-05-27T07:35:54.522 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:54.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:54 smithi163 ceph-mon[36451]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:54.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:54 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:54.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:54 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:54.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:54 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:54.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:54 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:54.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:54 smithi186 ceph-mon[36647]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:54.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:54 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:54.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:54 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:54.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:54 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:54.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:54 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:54.778 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:54 smithi114 ceph-mon[32038]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:54.778 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:54 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:54.778 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:54 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:54.778 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:54 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:54.778 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:54 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:56.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.204+0000 7f2ed9aab700 1 -- 172.21.15.114:0/4053816913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ed410a3d0 msgr2=0x7f2ed410a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.204+0000 7f2ed9aab700 1 --2- 172.21.15.114:0/4053816913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ed410a3d0 0x7f2ed410a7e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4009990 tx=0x7f2ec4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.204+0000 7f2ed9aab700 1 -- 172.21.15.114:0/4053816913 shutdown_connections 2024-05-27T07:35:56.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.204+0000 7f2ed9aab700 1 --2- 172.21.15.114:0/4053816913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed4103120 0x7f2ed4111110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.204+0000 7f2ed9aab700 1 --2- 172.21.15.114:0/4053816913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ed410a3d0 0x7f2ed410a7e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.204+0000 7f2ed9aab700 1 --2- 172.21.15.114:0/4053816913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ed41027d0 0x7f2ed4102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.204+0000 7f2ed9aab700 1 -- 172.21.15.114:0/4053816913 >> 172.21.15.114:0/4053816913 conn(0x7f2ed40fe6e0 msgr2=0x7f2ed4100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.204+0000 7f2ed9aab700 1 -- 172.21.15.114:0/4053816913 shutdown_connections 2024-05-27T07:35:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.204+0000 7f2ed9aab700 1 -- 172.21.15.114:0/4053816913 wait complete. 2024-05-27T07:35:56.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.205+0000 7f2ed9aab700 1 Processor -- start 2024-05-27T07:35:56.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.205+0000 7f2ed9aab700 1 -- start start 2024-05-27T07:35:56.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.205+0000 7f2ed9aab700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ed41027d0 0x7f2ed41039b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.205+0000 7f2ed9aab700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed4103120 0x7f2ed4103ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.205+0000 7f2ed9aab700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ed410a3d0 0x7f2ed4104430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.205+0000 7f2ed9aab700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed4114470 con 0x7f2ed41027d0 2024-05-27T07:35:56.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.205+0000 7f2ed9aab700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed41142f0 con 0x7f2ed410a3d0 2024-05-27T07:35:56.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.205+0000 7f2ed9aab700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed41145f0 con 0x7f2ed4103120 2024-05-27T07:35:56.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.205+0000 7f2ed37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ed41027d0 0x7f2ed41039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ed41027d0 0x7f2ed41039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51620/0 (socket says 172.21.15.114:51620) 2024-05-27T07:35:56.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed37fe700 1 -- 172.21.15.114:0/1239860054 learned_addr learned my addr 172.21.15.114:0/1239860054 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:35:56.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed2ffd700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed4103120 0x7f2ed4103ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed3fff700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ed410a3d0 0x7f2ed4104430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed37fe700 1 -- 172.21.15.114:0/1239860054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed4103120 msgr2=0x7f2ed4103ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed37fe700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed4103120 0x7f2ed4103ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed37fe700 1 -- 172.21.15.114:0/1239860054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ed410a3d0 msgr2=0x7f2ed4104430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed37fe700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ed410a3d0 0x7f2ed4104430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed37fe700 1 -- 172.21.15.114:0/1239860054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ed41a91a0 con 0x7f2ed41027d0 2024-05-27T07:35:56.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed3fff700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ed410a3d0 0x7f2ed4104430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:35:56.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed37fe700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ed41027d0 0x7f2ed41039b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4000c00 tx=0x7f2ec4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:56.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed0ff9700 1 -- 172.21.15.114:0/1239860054 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec4027070 con 0x7f2ed41027d0 2024-05-27T07:35:56.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed9aab700 1 -- 172.21.15.114:0/1239860054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ed41a9430 con 0x7f2ed41027d0 2024-05-27T07:35:56.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed0ff9700 1 -- 172.21.15.114:0/1239860054 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ec4022470 con 0x7f2ed41027d0 2024-05-27T07:35:56.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed0ff9700 1 -- 172.21.15.114:0/1239860054 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec4017650 con 0x7f2ed41027d0 2024-05-27T07:35:56.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.206+0000 7f2ed9aab700 1 -- 172.21.15.114:0/1239860054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ed41a98e0 con 0x7f2ed41027d0 2024-05-27T07:35:56.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.207+0000 7f2ed9aab700 1 -- 172.21.15.114:0/1239860054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ed410c840 con 0x7f2ed41027d0 2024-05-27T07:35:56.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.209+0000 7f2ed0ff9700 1 -- 172.21.15.114:0/1239860054 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f2ec402a070 con 0x7f2ed41027d0 2024-05-27T07:35:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.210+0000 7f2ed0ff9700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f2ebc0b2900 0x7f2ebc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.210+0000 7f2ed2ffd700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f2ebc0b2900 0x7f2ebc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.210+0000 7f2ed0ff9700 1 -- 172.21.15.114:0/1239860054 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f2ec40e2f30 con 0x7f2ed41027d0 2024-05-27T07:35:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.211+0000 7f2ed2ffd700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f2ebc0b2900 0x7f2ebc0b4e10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2ed41002f0 tx=0x7f2ec8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:56.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.214+0000 7f2ed0ff9700 1 -- 172.21.15.114:0/1239860054 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2ec4013070 con 0x7f2ed41027d0 2024-05-27T07:35:56.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.448+0000 7f2ed9aab700 1 -- 172.21.15.114:0/1239860054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f2ed410c220 con 0x7f2ed41027d0 2024-05-27T07:35:56.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.448+0000 7f2ed0ff9700 1 -- 172.21.15.114:0/1239860054 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v71) ==== 84+0+65 (secure 0 0 0) 0x7f2ec40ab180 con 0x7f2ed41027d0 2024-05-27T07:35:56.444 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:35:56.444 INFO:teuthology.orchestra.run.smithi114.stdout:[ 2024-05-27T07:35:56.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T07:35:56.444 INFO:teuthology.orchestra.run.smithi114.stdout: "poolnum": 1, 2024-05-27T07:35:56.444 INFO:teuthology.orchestra.run.smithi114.stdout: "poolname": ".mgr" 2024-05-27T07:35:56.444 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:35:56.445 INFO:teuthology.orchestra.run.smithi114.stdout:] 2024-05-27T07:35:56.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.450+0000 7f2ed9aab700 1 -- 172.21.15.114:0/1239860054 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f2ebc0b2900 msgr2=0x7f2ebc0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.450+0000 7f2ed9aab700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f2ebc0b2900 0x7f2ebc0b4e10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2ed41002f0 tx=0x7f2ec8009260 comp rx=0 tx=0).stop 2024-05-27T07:35:56.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.450+0000 7f2ed9aab700 1 -- 172.21.15.114:0/1239860054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ed41027d0 msgr2=0x7f2ed41039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.450+0000 7f2ed9aab700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ed41027d0 0x7f2ed41039b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4000c00 tx=0x7f2ec4009580 comp rx=0 tx=0).stop 2024-05-27T07:35:56.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.451+0000 7f2ed9aab700 1 -- 172.21.15.114:0/1239860054 shutdown_connections 2024-05-27T07:35:56.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.451+0000 7f2ed9aab700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ed410a3d0 0x7f2ed4104430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.451+0000 7f2ed9aab700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ed41027d0 0x7f2ed41039b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.451+0000 7f2ed9aab700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f2ebc0b2900 0x7f2ebc0b4e10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.451+0000 7f2ed9aab700 1 --2- 172.21.15.114:0/1239860054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed4103120 0x7f2ed4103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.451+0000 7f2ed9aab700 1 -- 172.21.15.114:0/1239860054 >> 172.21.15.114:0/1239860054 conn(0x7f2ed40fe6e0 msgr2=0x7f2ed4100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:56.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.451+0000 7f2ed9aab700 1 -- 172.21.15.114:0/1239860054 shutdown_connections 2024-05-27T07:35:56.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:56.451+0000 7f2ed9aab700 1 -- 172.21.15.114:0/1239860054 wait complete. 2024-05-27T07:35:56.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:56 smithi163 ceph-mon[36451]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:56.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:56 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1239860054' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T07:35:56.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:56 smithi186 ceph-mon[36647]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:56.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:56 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1239860054' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T07:35:56.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:56 smithi114 ceph-mon[32038]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:56.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:56 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1239860054' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T07:35:57.168 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-27T07:35:57.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-27T07:35:57.770 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:57.770 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:57.771 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:57.771 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:57.771 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:57.771 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:57 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:57.779 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:57.779 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:57.779 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:57.780 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:57.780 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:57.780 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:57 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:57.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:57.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:57.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:57.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:57.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:57.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:57 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:58.085 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:58.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:58 smithi114 ceph-mon[32038]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:58.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:58 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:58.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:58 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:58.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:58 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:58.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:35:58 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:58.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:58 smithi163 ceph-mon[36451]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:58.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:58 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:58.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:58 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:58.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:58 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:58.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:35:58 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:58.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:58 smithi186 ceph-mon[36647]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:35:58.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:58 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:58.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:58 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:58.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:58 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:58.974 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:35:58 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:35:59.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.704+0000 7fcbb8146700 1 -- 172.21.15.114:0/1589618795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb0108730 msgr2=0x7fcbb0110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:59.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.704+0000 7fcbb8146700 1 --2- 172.21.15.114:0/1589618795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb0108730 0x7fcbb0110090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcba80099c0 tx=0x7fcba8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.705+0000 7fcbb8146700 1 -- 172.21.15.114:0/1589618795 shutdown_connections 2024-05-27T07:35:59.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.705+0000 7fcbb8146700 1 --2- 172.21.15.114:0/1589618795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb0108730 0x7fcbb0110090 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.705+0000 7fcbb8146700 1 --2- 172.21.15.114:0/1589618795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb01034b0 0x7fcbb01038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.705+0000 7fcbb8146700 1 --2- 172.21.15.114:0/1589618795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcbb0107de0 0x7fcbb01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.705+0000 7fcbb8146700 1 -- 172.21.15.114:0/1589618795 >> 172.21.15.114:0/1589618795 conn(0x7fcbb00fedb0 msgr2=0x7fcbb01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:59.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.705+0000 7fcbb8146700 1 -- 172.21.15.114:0/1589618795 shutdown_connections 2024-05-27T07:35:59.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.705+0000 7fcbb8146700 1 -- 172.21.15.114:0/1589618795 wait complete. 2024-05-27T07:35:59.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.706+0000 7fcbb8146700 1 Processor -- start 2024-05-27T07:35:59.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.706+0000 7fcbb8146700 1 -- start start 2024-05-27T07:35:59.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.706+0000 7fcbb8146700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb01034b0 0x7fcbb019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.706+0000 7fcbb8146700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcbb0107de0 0x7fcbb019c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.706+0000 7fcbb8146700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb0108730 0x7fcbb019c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.706+0000 7fcbb8146700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb0113450 con 0x7fcbb0108730 2024-05-27T07:35:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.706+0000 7fcbb8146700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb01132d0 con 0x7fcbb0107de0 2024-05-27T07:35:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.706+0000 7fcbb8146700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb01135d0 con 0x7fcbb01034b0 2024-05-27T07:35:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.706+0000 7fcbb66e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb0108730 0x7fcbb019c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.706+0000 7fcbb56e1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcbb0107de0 0x7fcbb019c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.706+0000 7fcbb66e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb0108730 0x7fcbb019c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51638/0 (socket says 172.21.15.114:51638) 2024-05-27T07:35:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.706+0000 7fcbb66e3700 1 -- 172.21.15.114:0/4263246530 learned_addr learned my addr 172.21.15.114:0/4263246530 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:35:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.706+0000 7fcbb5ee2700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb01034b0 0x7fcbb019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.707+0000 7fcbb66e3700 1 -- 172.21.15.114:0/4263246530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb01034b0 msgr2=0x7fcbb019bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.707+0000 7fcbb66e3700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb01034b0 0x7fcbb019bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.707+0000 7fcbb66e3700 1 -- 172.21.15.114:0/4263246530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcbb0107de0 msgr2=0x7fcbb019c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.707+0000 7fcbb66e3700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcbb0107de0 0x7fcbb019c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.707+0000 7fcbb66e3700 1 -- 172.21.15.114:0/4263246530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb01a7080 con 0x7fcbb0108730 2024-05-27T07:35:59.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.707+0000 7fcbb5ee2700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb01034b0 0x7fcbb019bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:59.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.707+0000 7fcbb66e3700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb0108730 0x7fcbb019c7d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009990 tx=0x7fcba80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:59.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.707+0000 7fcbaeffd700 1 -- 172.21.15.114:0/4263246530 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba8027070 con 0x7fcbb0108730 2024-05-27T07:35:59.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.707+0000 7fcbb8146700 1 -- 172.21.15.114:0/4263246530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb01a7310 con 0x7fcbb0108730 2024-05-27T07:35:59.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.707+0000 7fcbaeffd700 1 -- 172.21.15.114:0/4263246530 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcba8004020 con 0x7fcbb0108730 2024-05-27T07:35:59.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.707+0000 7fcbaeffd700 1 -- 172.21.15.114:0/4263246530 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba8022ce0 con 0x7fcbb0108730 2024-05-27T07:35:59.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.707+0000 7fcbb8146700 1 -- 172.21.15.114:0/4263246530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb01a77c0 con 0x7fcbb0108730 2024-05-27T07:35:59.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.708+0000 7fcbb8146700 1 -- 172.21.15.114:0/4263246530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb00512b0 con 0x7fcbb0108730 2024-05-27T07:35:59.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.711+0000 7fcbaeffd700 1 -- 172.21.15.114:0/4263246530 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fcba802a070 con 0x7fcbb0108730 2024-05-27T07:35:59.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.712+0000 7fcbaeffd700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fcb980b2900 0x7fcb980b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.712+0000 7fcbb5ee2700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fcb980b2900 0x7fcb980b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.712+0000 7fcbaeffd700 1 -- 172.21.15.114:0/4263246530 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fcba80e3120 con 0x7fcbb0108730 2024-05-27T07:35:59.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.712+0000 7fcbb5ee2700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fcb980b2900 0x7fcb980b4e10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009fa0 tx=0x7fcba0009570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:59.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.715+0000 7fcbaeffd700 1 -- 172.21.15.114:0/4263246530 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcba8031080 con 0x7fcbb0108730 2024-05-27T07:35:59.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:35:59.957+0000 7fcbb8146700 1 -- 172.21.15.114:0/4263246530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fcbb002e7a0 con 0x7fcbb0108730 2024-05-27T07:36:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:00.932+0000 7fcbaeffd700 1 -- 172.21.15.114:0/4263246530 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v72) ==== 138+0+0 (secure 0 0 0) 0x7fcba80b1070 con 0x7fcbb0108730 2024-05-27T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:00.934+0000 7fcbb8146700 1 -- 172.21.15.114:0/4263246530 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fcb980b2900 msgr2=0x7fcb980b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:00.934+0000 7fcbb8146700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fcb980b2900 0x7fcb980b4e10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009fa0 tx=0x7fcba0009570 comp rx=0 tx=0).stop 2024-05-27T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:00.935+0000 7fcbb8146700 1 -- 172.21.15.114:0/4263246530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb0108730 msgr2=0x7fcbb019c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:00.935+0000 7fcbb8146700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb0108730 0x7fcbb019c7d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009990 tx=0x7fcba80047b0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:00.935+0000 7fcbb8146700 1 -- 172.21.15.114:0/4263246530 shutdown_connections 2024-05-27T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:00.935+0000 7fcbb8146700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb0108730 0x7fcbb019c7d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:00.935+0000 7fcbb8146700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fcb980b2900 0x7fcb980b4e10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:00.935+0000 7fcbb8146700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb01034b0 0x7fcbb019bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:00.935+0000 7fcbb8146700 1 --2- 172.21.15.114:0/4263246530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcbb0107de0 0x7fcbb019c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:00.935+0000 7fcbb8146700 1 -- 172.21.15.114:0/4263246530 >> 172.21.15.114:0/4263246530 conn(0x7fcbb00fedb0 msgr2=0x7fcbb0107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:00.935+0000 7fcbb8146700 1 -- 172.21.15.114:0/4263246530 shutdown_connections 2024-05-27T07:36:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:00.935+0000 7fcbb8146700 1 -- 172.21.15.114:0/4263246530 wait complete. 2024-05-27T07:36:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' created 2024-05-27T07:36:01.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:00 smithi163 ceph-mon[36451]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:01.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:00 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:36:01.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:00 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:36:01.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:00 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/4263246530' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T07:36:01.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:00 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:36:01.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:00 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:36:01.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:00 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:36:01.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:00 smithi186 ceph-mon[36647]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:01.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:00 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:36:01.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:00 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:36:01.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:00 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/4263246530' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T07:36:01.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:00 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:36:01.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:00 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:36:01.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:00 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:36:01.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:00 smithi114 ceph-mon[32038]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:01.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:00 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:36:01.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:00 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:36:01.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:00 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/4263246530' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T07:36:01.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:00 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:36:01.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:00 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:36:01.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:00 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:36:01.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-05-27T07:36:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:02.191 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:01 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/4263246530' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T07:36:02.191 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:01 smithi114 ceph-mon[32038]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T07:36:02.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:01 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/4263246530' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T07:36:02.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:01 smithi163 ceph-mon[36451]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T07:36:02.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:01 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/4263246530' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T07:36:02.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:01 smithi186 ceph-mon[36647]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T07:36:03.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:02 smithi163 ceph-mon[36451]: pgmap v281: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:03.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:02 smithi163 ceph-mon[36451]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:36:03.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:02 smithi163 ceph-mon[36451]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T07:36:03.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:02 smithi186 ceph-mon[36647]: pgmap v281: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:03.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:02 smithi186 ceph-mon[36647]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:36:03.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:02 smithi186 ceph-mon[36647]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T07:36:03.249 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:02 smithi114 ceph-mon[32038]: pgmap v281: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:03.249 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:02 smithi114 ceph-mon[32038]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:36:03.249 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:02 smithi114 ceph-mon[32038]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T07:36:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.491+0000 7f54ca422700 1 -- 172.21.15.114:0/10539972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c41054d0 msgr2=0x7f54c41058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.491+0000 7f54ca422700 1 --2- 172.21.15.114:0/10539972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c41054d0 0x7f54c41058e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f54b4009990 tx=0x7f54b4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:36:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.491+0000 7f54ca422700 1 -- 172.21.15.114:0/10539972 shutdown_connections 2024-05-27T07:36:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.491+0000 7f54ca422700 1 --2- 172.21.15.114:0/10539972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54c410e8b0 0x7f54c4110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.491+0000 7f54ca422700 1 --2- 172.21.15.114:0/10539972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c41054d0 0x7f54c41058e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.491+0000 7f54ca422700 1 --2- 172.21.15.114:0/10539972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c4105e20 0x7f54c410e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.491+0000 7f54ca422700 1 -- 172.21.15.114:0/10539972 >> 172.21.15.114:0/10539972 conn(0x7f54c40fefa0 msgr2=0x7f54c41013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.491+0000 7f54ca422700 1 -- 172.21.15.114:0/10539972 shutdown_connections 2024-05-27T07:36:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.491+0000 7f54ca422700 1 -- 172.21.15.114:0/10539972 wait complete. 2024-05-27T07:36:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.492+0000 7f54ca422700 1 Processor -- start 2024-05-27T07:36:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.492+0000 7f54ca422700 1 -- start start 2024-05-27T07:36:03.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.492+0000 7f54ca422700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c41054d0 0x7f54c419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:03.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.492+0000 7f54ca422700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54c4105e20 0x7f54c419b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:03.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.492+0000 7f54ca422700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c410e8b0 0x7f54c419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:03.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.492+0000 7f54ca422700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54c4114450 con 0x7f54c41054d0 2024-05-27T07:36:03.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.492+0000 7f54ca422700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f54c41142d0 con 0x7f54c4105e20 2024-05-27T07:36:03.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.492+0000 7f54ca422700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f54c41145d0 con 0x7f54c410e8b0 2024-05-27T07:36:03.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.492+0000 7f54c3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c41054d0 0x7f54c419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:03.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.492+0000 7f54c3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c41054d0 0x7f54c419ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51666/0 (socket says 172.21.15.114:51666) 2024-05-27T07:36:03.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.492+0000 7f54c3fff700 1 -- 172.21.15.114:0/2921677536 learned_addr learned my addr 172.21.15.114:0/2921677536 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:03.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.492+0000 7f54c37fe700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54c4105e20 0x7f54c419b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:03.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.493+0000 7f54c89bf700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c410e8b0 0x7f54c419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:03.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.493+0000 7f54c3fff700 1 -- 172.21.15.114:0/2921677536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c410e8b0 msgr2=0x7f54c419f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:03.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.493+0000 7f54c3fff700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c410e8b0 0x7f54c419f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:03.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.493+0000 7f54c3fff700 1 -- 172.21.15.114:0/2921677536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54c4105e20 msgr2=0x7f54c419b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:03.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.493+0000 7f54c3fff700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54c4105e20 0x7f54c419b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:03.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.493+0000 7f54c3fff700 1 -- 172.21.15.114:0/2921677536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c419fcd0 con 0x7f54c41054d0 2024-05-27T07:36:03.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.493+0000 7f54c89bf700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c410e8b0 0x7f54c419f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:03.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.493+0000 7f54c3fff700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c41054d0 0x7f54c419ae00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f54b4000c00 tx=0x7f54b40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:03.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.493+0000 7f54c17fa700 1 -- 172.21.15.114:0/2921677536 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b4027070 con 0x7f54c41054d0 2024-05-27T07:36:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.493+0000 7f54ca422700 1 -- 172.21.15.114:0/2921677536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54c419ff60 con 0x7f54c41054d0 2024-05-27T07:36:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.493+0000 7f54c17fa700 1 -- 172.21.15.114:0/2921677536 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54b4003f20 con 0x7f54c41054d0 2024-05-27T07:36:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.493+0000 7f54ca422700 1 -- 172.21.15.114:0/2921677536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54c41a78e0 con 0x7f54c41054d0 2024-05-27T07:36:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.493+0000 7f54c17fa700 1 -- 172.21.15.114:0/2921677536 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b4022cf0 con 0x7f54c41054d0 2024-05-27T07:36:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.494+0000 7f54ca422700 1 -- 172.21.15.114:0/2921677536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54c40512b0 con 0x7f54c41054d0 2024-05-27T07:36:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.496+0000 7f54c17fa700 1 -- 172.21.15.114:0/2921677536 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f54b402a070 con 0x7f54c41054d0 2024-05-27T07:36:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.496+0000 7f54c17fa700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f54ac0b2880 0x7f54ac0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.496+0000 7f54c37fe700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f54ac0b2880 0x7f54ac0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.497+0000 7f54c17fa700 1 -- 172.21.15.114:0/2921677536 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7497+0+0 (secure 0 0 0) 0x7f54b40e3050 con 0x7f54c41054d0 2024-05-27T07:36:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.497+0000 7f54c37fe700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f54ac0b2880 0x7f54ac0b4d90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f54c4102420 tx=0x7f54b800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:03.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.500+0000 7f54c17fa700 1 -- 172.21.15.114:0/2921677536 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f54b4013070 con 0x7f54c41054d0 2024-05-27T07:36:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.703+0000 7f54ca422700 1 -- 172.21.15.114:0/2921677536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f54c402e5e0 con 0x7f54c41054d0 2024-05-27T07:36:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.956+0000 7f54c17fa700 1 -- 172.21.15.114:0/2921677536 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v75) ==== 171+0+0 (secure 0 0 0) 0x7f54b40ab130 con 0x7f54c41054d0 2024-05-27T07:36:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.959+0000 7f54ca422700 1 -- 172.21.15.114:0/2921677536 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f54ac0b2880 msgr2=0x7f54ac0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.959+0000 7f54ca422700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f54ac0b2880 0x7f54ac0b4d90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f54c4102420 tx=0x7f54b800b410 comp rx=0 tx=0).stop 2024-05-27T07:36:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.959+0000 7f54ca422700 1 -- 172.21.15.114:0/2921677536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c41054d0 msgr2=0x7f54c419ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.959+0000 7f54ca422700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c41054d0 0x7f54c419ae00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f54b4000c00 tx=0x7f54b40047b0 comp rx=0 tx=0).stop 2024-05-27T07:36:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.959+0000 7f54ca422700 1 -- 172.21.15.114:0/2921677536 shutdown_connections 2024-05-27T07:36:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.959+0000 7f54ca422700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f54ac0b2880 0x7f54ac0b4d90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.959+0000 7f54ca422700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c410e8b0 0x7f54c419f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.959+0000 7f54ca422700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c41054d0 0x7f54c419ae00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.959+0000 7f54ca422700 1 --2- 172.21.15.114:0/2921677536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54c4105e20 0x7f54c419b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.959+0000 7f54ca422700 1 -- 172.21.15.114:0/2921677536 >> 172.21.15.114:0/2921677536 conn(0x7f54c40fefa0 msgr2=0x7f54c40ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.959+0000 7f54ca422700 1 -- 172.21.15.114:0/2921677536 shutdown_connections 2024-05-27T07:36:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:03.959+0000 7f54ca422700 1 -- 172.21.15.114:0/2921677536 wait complete. 2024-05-27T07:36:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' created 2024-05-27T07:36:04.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:03 smithi163 ceph-mon[36451]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T07:36:04.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:03 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2921677536' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:36:04.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:03 smithi186 ceph-mon[36647]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T07:36:04.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:03 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2921677536' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:36:04.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:03 smithi114 ceph-mon[32038]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T07:36:04.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:03 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2921677536' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:36:04.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-27T07:36:04.816 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:05.189 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:04 smithi114 ceph-mon[32038]: pgmap v284: 33 pgs: 9 creating+peering, 12 active+clean, 12 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:05.189 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:04 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2921677536' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-27T07:36:05.189 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:04 smithi114 ceph-mon[32038]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T07:36:05.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:04 smithi163 ceph-mon[36451]: pgmap v284: 33 pgs: 9 creating+peering, 12 active+clean, 12 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:05.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:04 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2921677536' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-27T07:36:05.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:04 smithi163 ceph-mon[36451]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T07:36:05.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:04 smithi186 ceph-mon[36647]: pgmap v284: 33 pgs: 9 creating+peering, 12 active+clean, 12 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:05.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:04 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2921677536' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-27T07:36:05.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:04 smithi186 ceph-mon[36647]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T07:36:06.222 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:05 smithi114 ceph-mon[32038]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T07:36:06.224 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:05 smithi186 ceph-mon[36647]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T07:36:06.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:05 smithi163 ceph-mon[36451]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T07:36:06.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.587+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2232559722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaac108ef0 msgr2=0x7fbaac109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:06.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.587+0000 7fbab3d2f700 1 --2- 172.21.15.114:0/2232559722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaac108ef0 0x7fbaac109300 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0099c0 tx=0x7fba9c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:36:06.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.587+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2232559722 shutdown_connections 2024-05-27T07:36:06.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.587+0000 7fbab3d2f700 1 --2- 172.21.15.114:0/2232559722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbaac109840 0x7fbaac1111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:06.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.587+0000 7fbab3d2f700 1 --2- 172.21.15.114:0/2232559722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaac1045c0 0x7fbaac1049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.587+0000 7fbab3d2f700 1 --2- 172.21.15.114:0/2232559722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaac108ef0 0x7fbaac109300 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.587+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2232559722 >> 172.21.15.114:0/2232559722 conn(0x7fbaac0ffe90 msgr2=0x7fbaac1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.587+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2232559722 shutdown_connections 2024-05-27T07:36:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.587+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2232559722 wait complete. 2024-05-27T07:36:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.588+0000 7fbab3d2f700 1 Processor -- start 2024-05-27T07:36:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.588+0000 7fbab3d2f700 1 -- start start 2024-05-27T07:36:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.588+0000 7fbab3d2f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaac1045c0 0x7fbaac19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.588+0000 7fbab3d2f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaac108ef0 0x7fbaac19b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.588+0000 7fbab3d2f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbaac109840 0x7fbaac19f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.588+0000 7fbab3d2f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac114450 con 0x7fbaac108ef0 2024-05-27T07:36:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.588+0000 7fbab3d2f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac1142d0 con 0x7fbaac1045c0 2024-05-27T07:36:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.588+0000 7fbab3d2f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac1145d0 con 0x7fbaac109840 2024-05-27T07:36:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.588+0000 7fbab12ca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaac108ef0 0x7fbaac19b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:06.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.588+0000 7fbab12ca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaac108ef0 0x7fbaac19b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45660/0 (socket says 172.21.15.114:45660) 2024-05-27T07:36:06.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.588+0000 7fbab12ca700 1 -- 172.21.15.114:0/2382056244 learned_addr learned my addr 172.21.15.114:0/2382056244 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:06.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.588+0000 7fbab1acb700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaac1045c0 0x7fbaac19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:06.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.588+0000 7fbab22cc700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbaac109840 0x7fbaac19f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:06.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.589+0000 7fbab12ca700 1 -- 172.21.15.114:0/2382056244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbaac109840 msgr2=0x7fbaac19f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:06.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.589+0000 7fbab12ca700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbaac109840 0x7fbaac19f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:06.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.589+0000 7fbab12ca700 1 -- 172.21.15.114:0/2382056244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaac1045c0 msgr2=0x7fbaac19ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:06.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.589+0000 7fbab12ca700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaac1045c0 0x7fbaac19ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:06.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.589+0000 7fbab12ca700 1 -- 172.21.15.114:0/2382056244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaac19fc70 con 0x7fbaac108ef0 2024-05-27T07:36:06.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.589+0000 7fbab12ca700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaac108ef0 0x7fbaac19b340 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fba9c006010 tx=0x7fba9c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:06.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.589+0000 7fbaa2ffd700 1 -- 172.21.15.114:0/2382056244 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c027070 con 0x7fbaac108ef0 2024-05-27T07:36:06.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.589+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2382056244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaac19ff00 con 0x7fbaac108ef0 2024-05-27T07:36:06.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.589+0000 7fbaa2ffd700 1 -- 172.21.15.114:0/2382056244 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba9c003f20 con 0x7fbaac108ef0 2024-05-27T07:36:06.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.589+0000 7fbaa2ffd700 1 -- 172.21.15.114:0/2382056244 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c022c60 con 0x7fbaac108ef0 2024-05-27T07:36:06.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.589+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2382056244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaac1a7880 con 0x7fbaac108ef0 2024-05-27T07:36:06.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.590+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2382056244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaac105a70 con 0x7fbaac108ef0 2024-05-27T07:36:06.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.592+0000 7fbaa2ffd700 1 -- 172.21.15.114:0/2382056244 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fba9c02a070 con 0x7fbaac108ef0 2024-05-27T07:36:06.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.592+0000 7fbaa2ffd700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fba980b2880 0x7fba980b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:06.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.592+0000 7fbab1acb700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fba980b2880 0x7fba980b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:06.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.592+0000 7fbaa2ffd700 1 -- 172.21.15.114:0/2382056244 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7879+0+0 (secure 0 0 0) 0x7fba9c022e00 con 0x7fbaac108ef0 2024-05-27T07:36:06.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.593+0000 7fbab1acb700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fba980b2880 0x7fba980b4d90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8005950 tx=0x7fbaa800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:06.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.597+0000 7fbaa2ffd700 1 -- 172.21.15.114:0/2382056244 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fba9c013070 con 0x7fbaac108ef0 2024-05-27T07:36:06.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.847+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2382056244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fbaac19c100 con 0x7fbaac108ef0 2024-05-27T07:36:06.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.973+0000 7fbaa2ffd700 1 -- 172.21.15.114:0/2382056244 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fba9c0aade0 con 0x7fbaac108ef0 2024-05-27T07:36:06.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.976+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2382056244 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fba980b2880 msgr2=0x7fba980b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:06.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.976+0000 7fbab3d2f700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fba980b2880 0x7fba980b4d90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8005950 tx=0x7fbaa800a300 comp rx=0 tx=0).stop 2024-05-27T07:36:06.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.976+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2382056244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaac108ef0 msgr2=0x7fbaac19b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:06.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.976+0000 7fbab3d2f700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaac108ef0 0x7fbaac19b340 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fba9c006010 tx=0x7fba9c0047b0 comp rx=0 tx=0).stop 2024-05-27T07:36:06.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.976+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2382056244 shutdown_connections 2024-05-27T07:36:06.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.976+0000 7fbab3d2f700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fba980b2880 0x7fba980b4d90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:06.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.976+0000 7fbab3d2f700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbaac109840 0x7fbaac19f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:06.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.976+0000 7fbab3d2f700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaac1045c0 0x7fbaac19ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:06.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.976+0000 7fbab3d2f700 1 --2- 172.21.15.114:0/2382056244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaac108ef0 0x7fbaac19b340 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:06.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.976+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2382056244 >> 172.21.15.114:0/2382056244 conn(0x7fbaac0ffe90 msgr2=0x7fbaac10a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:06.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.976+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2382056244 shutdown_connections 2024-05-27T07:36:06.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:06.976+0000 7fbab3d2f700 1 -- 172.21.15.114:0/2382056244 wait complete. 2024-05-27T07:36:06.972 INFO:teuthology.orchestra.run.smithi114.stderr:new fs with metadata pool 2 and data pool 3 2024-05-27T07:36:07.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:06 smithi114 ceph-mon[32038]: pgmap v287: 97 pgs: 37 creating+peering, 24 active+clean, 36 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:07.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:06 smithi114 ceph-mon[32038]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T07:36:07.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:06 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2382056244' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T07:36:07.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:06 smithi114 ceph-mon[32038]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-05-27T07:36:07.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:06 smithi163 ceph-mon[36451]: pgmap v287: 97 pgs: 37 creating+peering, 24 active+clean, 36 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:07.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:06 smithi163 ceph-mon[36451]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T07:36:07.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:06 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2382056244' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T07:36:07.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:06 smithi163 ceph-mon[36451]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-05-27T07:36:07.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:06 smithi186 ceph-mon[36647]: pgmap v287: 97 pgs: 37 creating+peering, 24 active+clean, 36 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:07.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:06 smithi186 ceph-mon[36647]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T07:36:07.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:06 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2382056244' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T07:36:07.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:06 smithi186 ceph-mon[36647]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-05-27T07:36:07.749 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-05-27T07:36:07.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-05-27T07:36:07.990 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:08.246 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:07 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2382056244' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T07:36:08.246 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:07 smithi114 ceph-mon[32038]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T07:36:08.246 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:07 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-05-27T07:36:08.246 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:07 smithi114 ceph-mon[32038]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-05-27T07:36:08.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:07 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2382056244' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T07:36:08.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:07 smithi163 ceph-mon[36451]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T07:36:08.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:07 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-05-27T07:36:08.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:07 smithi163 ceph-mon[36451]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-05-27T07:36:08.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:07 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2382056244' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T07:36:08.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:07 smithi186 ceph-mon[36647]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T07:36:08.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:07 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-05-27T07:36:08.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:07 smithi186 ceph-mon[36647]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-05-27T07:36:09.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:08 smithi114 ceph-mon[32038]: pgmap v290: 97 pgs: 28 creating+peering, 45 active+clean, 24 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:09.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:08 smithi114 ceph-mon[32038]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T07:36:09.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:08 smithi114 ceph-mon[32038]: Cluster is now healthy 2024-05-27T07:36:09.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:08 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] up:active 2024-05-27T07:36:09.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:08 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:36:09.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:08 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:36:09.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:08 smithi163 ceph-mon[36451]: pgmap v290: 97 pgs: 28 creating+peering, 45 active+clean, 24 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:09.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:08 smithi163 ceph-mon[36451]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T07:36:09.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:08 smithi163 ceph-mon[36451]: Cluster is now healthy 2024-05-27T07:36:09.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:08 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] up:active 2024-05-27T07:36:09.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:08 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:36:09.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:08 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:36:09.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:08 smithi186 ceph-mon[36647]: pgmap v290: 97 pgs: 28 creating+peering, 45 active+clean, 24 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:09.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:08 smithi186 ceph-mon[36647]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T07:36:09.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:08 smithi186 ceph-mon[36647]: Cluster is now healthy 2024-05-27T07:36:09.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:08 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] up:active 2024-05-27T07:36:09.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:08 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:36:09.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:08 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:36:09.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.675+0000 7f0f15cf6700 1 -- 172.21.15.114:0/2521205276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f101045c0 msgr2=0x7f0f101049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.675+0000 7f0f15cf6700 1 --2- 172.21.15.114:0/2521205276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f101045c0 0x7f0f101049d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0f0000b0d0 tx=0x7f0f0000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:36:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.675+0000 7f0f15cf6700 1 -- 172.21.15.114:0/2521205276 shutdown_connections 2024-05-27T07:36:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.675+0000 7f0f15cf6700 1 --2- 172.21.15.114:0/2521205276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f10109840 0x7f0f101111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.675+0000 7f0f15cf6700 1 --2- 172.21.15.114:0/2521205276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f101045c0 0x7f0f101049d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.675+0000 7f0f15cf6700 1 --2- 172.21.15.114:0/2521205276 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f10108ef0 0x7f0f10109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.675+0000 7f0f15cf6700 1 -- 172.21.15.114:0/2521205276 >> 172.21.15.114:0/2521205276 conn(0x7f0f100ffe90 msgr2=0x7f0f101022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.675+0000 7f0f15cf6700 1 -- 172.21.15.114:0/2521205276 shutdown_connections 2024-05-27T07:36:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.675+0000 7f0f15cf6700 1 -- 172.21.15.114:0/2521205276 wait complete. 2024-05-27T07:36:09.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.676+0000 7f0f15cf6700 1 Processor -- start 2024-05-27T07:36:09.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.676+0000 7f0f15cf6700 1 -- start start 2024-05-27T07:36:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.676+0000 7f0f15cf6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f101045c0 0x7f0f1019ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.676+0000 7f0f15cf6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f10108ef0 0x7f0f1019b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.676+0000 7f0f15cf6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f10109840 0x7f0f1019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.676+0000 7f0f15cf6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0f10114470 con 0x7f0f101045c0 2024-05-27T07:36:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.676+0000 7f0f15cf6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0f101142f0 con 0x7f0f10108ef0 2024-05-27T07:36:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.676+0000 7f0f15cf6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0f101145f0 con 0x7f0f10109840 2024-05-27T07:36:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f0f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f101045c0 0x7f0f1019ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f0f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f101045c0 0x7f0f1019ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45678/0 (socket says 172.21.15.114:45678) 2024-05-27T07:36:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f0f7fe700 1 -- 172.21.15.114:0/1342138001 learned_addr learned my addr 172.21.15.114:0/1342138001 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0efffff700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f10108ef0 0x7f0f1019b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f0ffff700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f10109840 0x7f0f1019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f0f7fe700 1 -- 172.21.15.114:0/1342138001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f10109840 msgr2=0x7f0f1019f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f0f7fe700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f10109840 0x7f0f1019f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f0f7fe700 1 -- 172.21.15.114:0/1342138001 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f10108ef0 msgr2=0x7f0f1019b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f0f7fe700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f10108ef0 0x7f0f1019b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f0f7fe700 1 -- 172.21.15.114:0/1342138001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1019fd30 con 0x7f0f101045c0 2024-05-27T07:36:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0efffff700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f10108ef0 0x7f0f1019b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f0f7fe700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f101045c0 0x7f0f1019ae60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0f0000d010 tx=0x7f0f00007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f0d7fa700 1 -- 172.21.15.114:0/1342138001 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0001f070 con 0x7f0f101045c0 2024-05-27T07:36:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f0d7fa700 1 -- 172.21.15.114:0/1342138001 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f00003c00 con 0x7f0f101045c0 2024-05-27T07:36:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f0d7fa700 1 -- 172.21.15.114:0/1342138001 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0001b6a0 con 0x7f0f101045c0 2024-05-27T07:36:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f15cf6700 1 -- 172.21.15.114:0/1342138001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f1019ffc0 con 0x7f0f101045c0 2024-05-27T07:36:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.677+0000 7f0f15cf6700 1 -- 172.21.15.114:0/1342138001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f101a0450 con 0x7f0f101045c0 2024-05-27T07:36:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.678+0000 7f0f15cf6700 1 -- 172.21.15.114:0/1342138001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f100512b0 con 0x7f0f101045c0 2024-05-27T07:36:09.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.681+0000 7f0f0d7fa700 1 -- 172.21.15.114:0/1342138001 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f0f00022070 con 0x7f0f101045c0 2024-05-27T07:36:09.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.681+0000 7f0f0d7fa700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0ef80b2880 0x7f0ef80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:09.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.681+0000 7f0efffff700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0ef80b2880 0x7f0ef80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:09.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.681+0000 7f0f0d7fa700 1 -- 172.21.15.114:0/1342138001 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7f0f0000f020 con 0x7f0f101045c0 2024-05-27T07:36:09.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.682+0000 7f0efffff700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0ef80b2880 0x7f0ef80b4d90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0f04005950 tx=0x7f0f040058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:09.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.685+0000 7f0f0d7fa700 1 -- 172.21.15.114:0/1342138001 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0f00029020 con 0x7f0f101045c0 2024-05-27T07:36:09.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.901+0000 7f0f15cf6700 1 -- 172.21.15.114:0/1342138001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f0f1002e5e0 con 0x7f0f101045c0 2024-05-27T07:36:09.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.995+0000 7f0f0d7fa700 1 -- 172.21.15.114:0/1342138001 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v79) ==== 157+0+0 (secure 0 0 0) 0x7f0f000ab610 con 0x7f0f101045c0 2024-05-27T07:36:09.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.997+0000 7f0f15cf6700 1 -- 172.21.15.114:0/1342138001 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0ef80b2880 msgr2=0x7f0ef80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:09.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.997+0000 7f0f15cf6700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0ef80b2880 0x7f0ef80b4d90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0f04005950 tx=0x7f0f040058e0 comp rx=0 tx=0).stop 2024-05-27T07:36:09.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.997+0000 7f0f15cf6700 1 -- 172.21.15.114:0/1342138001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f101045c0 msgr2=0x7f0f1019ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:09.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.997+0000 7f0f15cf6700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f101045c0 0x7f0f1019ae60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0f0000d010 tx=0x7f0f00007a20 comp rx=0 tx=0).stop 2024-05-27T07:36:09.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.997+0000 7f0f15cf6700 1 -- 172.21.15.114:0/1342138001 shutdown_connections 2024-05-27T07:36:09.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.997+0000 7f0f15cf6700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0ef80b2880 0x7f0ef80b4d90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:09.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.997+0000 7f0f15cf6700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f10109840 0x7f0f1019f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:09.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.997+0000 7f0f15cf6700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f101045c0 0x7f0f1019ae60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:09.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.997+0000 7f0f15cf6700 1 --2- 172.21.15.114:0/1342138001 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f10108ef0 0x7f0f1019b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:09.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.997+0000 7f0f15cf6700 1 -- 172.21.15.114:0/1342138001 >> 172.21.15.114:0/1342138001 conn(0x7f0f100ffe90 msgr2=0x7f0f1010a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:09.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.998+0000 7f0f15cf6700 1 -- 172.21.15.114:0/1342138001 shutdown_connections 2024-05-27T07:36:09.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:09.998+0000 7f0f15cf6700 1 -- 172.21.15.114:0/1342138001 wait complete. 2024-05-27T07:36:10.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:09 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1342138001' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-27T07:36:10.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:09 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1342138001' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-27T07:36:10.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:09 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1342138001' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-27T07:36:10.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-05-27T07:36:11.025 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:11.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:10 smithi114 ceph-mon[32038]: pgmap v291: 97 pgs: 26 creating+peering, 71 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-05-27T07:36:11.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:10 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1342138001' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-27T07:36:11.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:10 smithi114 ceph-mon[32038]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T07:36:11.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:10 smithi163 ceph-mon[36451]: pgmap v291: 97 pgs: 26 creating+peering, 71 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-05-27T07:36:11.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:10 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1342138001' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-27T07:36:11.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:10 smithi163 ceph-mon[36451]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T07:36:11.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:10 smithi186 ceph-mon[36647]: pgmap v291: 97 pgs: 26 creating+peering, 71 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-05-27T07:36:11.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:10 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1342138001' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-27T07:36:11.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:10 smithi186 ceph-mon[36647]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T07:36:12.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.653+0000 7f1942b83700 1 -- 172.21.15.114:0/3424837336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f193c109880 msgr2=0x7f193c1111e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:12.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.653+0000 7f1942b83700 1 --2- 172.21.15.114:0/3424837336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f193c109880 0x7f193c1111e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f193400b0d0 tx=0x7f193400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:36:12.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.653+0000 7f1942b83700 1 -- 172.21.15.114:0/3424837336 shutdown_connections 2024-05-27T07:36:12.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.653+0000 7f1942b83700 1 --2- 172.21.15.114:0/3424837336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f193c109880 0x7f193c1111e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:12.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.653+0000 7f1942b83700 1 --2- 172.21.15.114:0/3424837336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f193c104600 0x7f193c104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:12.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.653+0000 7f1942b83700 1 --2- 172.21.15.114:0/3424837336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f193c108f30 0x7f193c109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:12.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.654+0000 7f1942b83700 1 -- 172.21.15.114:0/3424837336 >> 172.21.15.114:0/3424837336 conn(0x7f193c07af60 msgr2=0x7f193c07b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:12.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.654+0000 7f1942b83700 1 -- 172.21.15.114:0/3424837336 shutdown_connections 2024-05-27T07:36:12.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.654+0000 7f1942b83700 1 -- 172.21.15.114:0/3424837336 wait complete. 2024-05-27T07:36:12.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.654+0000 7f1942b83700 1 Processor -- start 2024-05-27T07:36:12.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.654+0000 7f1942b83700 1 -- start start 2024-05-27T07:36:12.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1942b83700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f193c104600 0x7f193c075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:12.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1942b83700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f193c108f30 0x7f193c075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:12.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1942b83700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f193c109880 0x7f193c076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:12.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1942b83700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f193c1144b0 con 0x7f193c109880 2024-05-27T07:36:12.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1942b83700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f193c114330 con 0x7f193c108f30 2024-05-27T07:36:12.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1942b83700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f193c114630 con 0x7f193c104600 2024-05-27T07:36:12.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1941120700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f193c109880 0x7f193c076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:12.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1941120700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f193c109880 0x7f193c076530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45704/0 (socket says 172.21.15.114:45704) 2024-05-27T07:36:12.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1941120700 1 -- 172.21.15.114:0/954818519 learned_addr learned my addr 172.21.15.114:0/954818519 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:12.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f194091f700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f193c104600 0x7f193c075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:12.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f193bfff700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f193c108f30 0x7f193c075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:12.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1941120700 1 -- 172.21.15.114:0/954818519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f193c104600 msgr2=0x7f193c075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:12.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1941120700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f193c104600 0x7f193c075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:12.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1941120700 1 -- 172.21.15.114:0/954818519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f193c108f30 msgr2=0x7f193c075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:12.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1941120700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f193c108f30 0x7f193c075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:12.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1941120700 1 -- 172.21.15.114:0/954818519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193c1a90b0 con 0x7f193c109880 2024-05-27T07:36:12.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f194091f700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f193c104600 0x7f193c075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:12.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1941120700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f193c109880 0x7f193c076530 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f19340095c0 tx=0x7f1934007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:12.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1939ffb700 1 -- 172.21.15.114:0/954818519 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193401f070 con 0x7f193c109880 2024-05-27T07:36:12.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1939ffb700 1 -- 172.21.15.114:0/954818519 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1934003c00 con 0x7f193c109880 2024-05-27T07:36:12.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.655+0000 7f1939ffb700 1 -- 172.21.15.114:0/954818519 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193401b6a0 con 0x7f193c109880 2024-05-27T07:36:12.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.656+0000 7f1942b83700 1 -- 172.21.15.114:0/954818519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f193c1a9340 con 0x7f193c109880 2024-05-27T07:36:12.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.656+0000 7f1942b83700 1 -- 172.21.15.114:0/954818519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f193c1a9740 con 0x7f193c109880 2024-05-27T07:36:12.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.657+0000 7f1942b83700 1 -- 172.21.15.114:0/954818519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f193c0512b0 con 0x7f193c109880 2024-05-27T07:36:12.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.658+0000 7f1939ffb700 1 -- 172.21.15.114:0/954818519 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f1934022070 con 0x7f193c109880 2024-05-27T07:36:12.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.659+0000 7f1939ffb700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f192c0b2900 0x7f192c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:12.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.659+0000 7f194091f700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f192c0b2900 0x7f192c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:12.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.659+0000 7f1939ffb700 1 -- 172.21.15.114:0/954818519 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8277+0+0 (secure 0 0 0) 0x7f193400f020 con 0x7f193c109880 2024-05-27T07:36:12.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.659+0000 7f194091f700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f192c0b2900 0x7f192c0b4e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1930005950 tx=0x7f19300058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:12.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.662+0000 7f1939ffb700 1 -- 172.21.15.114:0/954818519 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1934029020 con 0x7f193c109880 2024-05-27T07:36:12.724 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 07:36:12 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-i[67994]: 2024-05-27T07:36:12.384+0000 7fa1976d1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T07:36:12.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:12.865+0000 7f1942b83700 1 -- 172.21.15.114:0/954818519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f193c02e680 con 0x7f193c109880 2024-05-27T07:36:13.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:13 smithi114 ceph-mon[32038]: pgmap v293: 97 pgs: 97 active+clean; 578 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-05-27T07:36:13.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:13 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/954818519' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T07:36:13.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:13 smithi163 ceph-mon[36451]: pgmap v293: 97 pgs: 97 active+clean; 578 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-05-27T07:36:13.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:13 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/954818519' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T07:36:13.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:13 smithi186 ceph-mon[36647]: pgmap v293: 97 pgs: 97 active+clean; 578 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-05-27T07:36:13.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:13 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/954818519' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T07:36:13.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:14.002+0000 7f1939ffb700 1 -- 172.21.15.114:0/954818519 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v81) ==== 228+0+0 (secure 0 0 0) 0x7f19340ab700 con 0x7f193c109880 2024-05-27T07:36:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:14.006+0000 7f1942b83700 1 -- 172.21.15.114:0/954818519 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f192c0b2900 msgr2=0x7f192c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:14.006+0000 7f1942b83700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f192c0b2900 0x7f192c0b4e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1930005950 tx=0x7f19300058e0 comp rx=0 tx=0).stop 2024-05-27T07:36:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:14.006+0000 7f1942b83700 1 -- 172.21.15.114:0/954818519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f193c109880 msgr2=0x7f193c076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:14.006+0000 7f1942b83700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f193c109880 0x7f193c076530 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f19340095c0 tx=0x7f1934007ad0 comp rx=0 tx=0).stop 2024-05-27T07:36:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:14.006+0000 7f1942b83700 1 -- 172.21.15.114:0/954818519 shutdown_connections 2024-05-27T07:36:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:14.006+0000 7f1942b83700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f193c109880 0x7f193c076530 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:14.006+0000 7f1942b83700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f192c0b2900 0x7f192c0b4e10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:14.006+0000 7f1942b83700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f193c104600 0x7f193c075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:14.006+0000 7f1942b83700 1 --2- 172.21.15.114:0/954818519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f193c108f30 0x7f193c075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:14.006+0000 7f1942b83700 1 -- 172.21.15.114:0/954818519 >> 172.21.15.114:0/954818519 conn(0x7f193c07af60 msgr2=0x7f193c101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:14.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:14.007+0000 7f1942b83700 1 -- 172.21.15.114:0/954818519 shutdown_connections 2024-05-27T07:36:14.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:14.007+0000 7f1942b83700 1 -- 172.21.15.114:0/954818519 wait complete. 2024-05-27T07:36:14.003 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data_ec' created 2024-05-27T07:36:14.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:14 smithi114 ceph-mon[32038]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T07:36:14.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:14 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/954818519' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T07:36:14.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:14 smithi114 ceph-mon[32038]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T07:36:14.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:14 smithi163 ceph-mon[36451]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T07:36:14.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:14 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/954818519' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T07:36:14.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:14 smithi163 ceph-mon[36451]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T07:36:14.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:14 smithi186 ceph-mon[36647]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T07:36:14.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:14 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/954818519' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T07:36:14.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:14 smithi186 ceph-mon[36647]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T07:36:14.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-05-27T07:36:14.905 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:15.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:15 smithi114 ceph-mon[32038]: pgmap v295: 97 pgs: 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-05-27T07:36:15.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:15 smithi114 ceph-mon[32038]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T07:36:15.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:15 smithi163 ceph-mon[36451]: pgmap v295: 97 pgs: 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-05-27T07:36:15.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:15 smithi163 ceph-mon[36451]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T07:36:15.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:15 smithi186 ceph-mon[36647]: pgmap v295: 97 pgs: 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-05-27T07:36:15.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:15 smithi186 ceph-mon[36647]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T07:36:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.608+0000 7fb48a3bd700 1 -- 172.21.15.114:0/3255442316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb484107de0 msgr2=0x7fb4841081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.608+0000 7fb48a3bd700 1 --2- 172.21.15.114:0/3255442316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb484107de0 0x7fb4841081f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0099c0 tx=0x7fb46c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:36:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.608+0000 7fb48a3bd700 1 -- 172.21.15.114:0/3255442316 shutdown_connections 2024-05-27T07:36:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.608+0000 7fb48a3bd700 1 --2- 172.21.15.114:0/3255442316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb484108730 0x7fb484110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.608+0000 7fb48a3bd700 1 --2- 172.21.15.114:0/3255442316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4841034b0 0x7fb4841038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.608+0000 7fb48a3bd700 1 --2- 172.21.15.114:0/3255442316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb484107de0 0x7fb4841081f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.608+0000 7fb48a3bd700 1 -- 172.21.15.114:0/3255442316 >> 172.21.15.114:0/3255442316 conn(0x7fb4840fede0 msgr2=0x7fb484101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.608+0000 7fb48a3bd700 1 -- 172.21.15.114:0/3255442316 shutdown_connections 2024-05-27T07:36:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.608+0000 7fb48a3bd700 1 -- 172.21.15.114:0/3255442316 wait complete. 2024-05-27T07:36:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.609+0000 7fb48a3bd700 1 Processor -- start 2024-05-27T07:36:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.609+0000 7fb48a3bd700 1 -- start start 2024-05-27T07:36:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb48a3bd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4841034b0 0x7fb48419d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb48a3bd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb484107de0 0x7fb48419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb48a3bd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb484108730 0x7fb4841a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb48a3bd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb4841133b0 con 0x7fb484107de0 2024-05-27T07:36:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb48a3bd700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb484113230 con 0x7fb484108730 2024-05-27T07:36:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb48a3bd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb484113530 con 0x7fb4841034b0 2024-05-27T07:36:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb47bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb484107de0 0x7fb48419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb47bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb484107de0 0x7fb48419d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54664/0 (socket says 172.21.15.114:54664) 2024-05-27T07:36:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb47bfff700 1 -- 172.21.15.114:0/2708641 learned_addr learned my addr 172.21.15.114:0/2708641 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb483fff700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4841034b0 0x7fb48419d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb47bfff700 1 -- 172.21.15.114:0/2708641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4841034b0 msgr2=0x7fb48419d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb47bfff700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4841034b0 0x7fb48419d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb48895a700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb484108730 0x7fb4841a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb47bfff700 1 -- 172.21.15.114:0/2708641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb484108730 msgr2=0x7fb4841a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb47bfff700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb484108730 0x7fb4841a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb47bfff700 1 -- 172.21.15.114:0/2708641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4841a1e70 con 0x7fb484107de0 2024-05-27T07:36:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb483fff700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4841034b0 0x7fb48419d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb48895a700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb484108730 0x7fb4841a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.610+0000 7fb47bfff700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb484107de0 0x7fb48419d5f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0098e0 tx=0x7fb46c003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:16.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.611+0000 7fb481ffb700 1 -- 172.21.15.114:0/2708641 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c027070 con 0x7fb484107de0 2024-05-27T07:36:16.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.611+0000 7fb48a3bd700 1 -- 172.21.15.114:0/2708641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4841a2100 con 0x7fb484107de0 2024-05-27T07:36:16.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.611+0000 7fb481ffb700 1 -- 172.21.15.114:0/2708641 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb46c0040a0 con 0x7fb484107de0 2024-05-27T07:36:16.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.611+0000 7fb481ffb700 1 -- 172.21.15.114:0/2708641 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c00f460 con 0x7fb484107de0 2024-05-27T07:36:16.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.611+0000 7fb48a3bd700 1 -- 172.21.15.114:0/2708641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4841a25e0 con 0x7fb484107de0 2024-05-27T07:36:16.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.612+0000 7fb48a3bd700 1 -- 172.21.15.114:0/2708641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4840512b0 con 0x7fb484107de0 2024-05-27T07:36:16.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.613+0000 7fb481ffb700 1 -- 172.21.15.114:0/2708641 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fb46c02a070 con 0x7fb484107de0 2024-05-27T07:36:16.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.614+0000 7fb481ffb700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb4700b2880 0x7fb4700b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:16.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.614+0000 7fb483fff700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb4700b2880 0x7fb4700b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:16.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.614+0000 7fb481ffb700 1 -- 172.21.15.114:0/2708641 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8770+0+0 (secure 0 0 0) 0x7fb46c0e3420 con 0x7fb484107de0 2024-05-27T07:36:16.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.614+0000 7fb483fff700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb4700b2880 0x7fb4700b4d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb474005fd0 tx=0x7fb474005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:16.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.619+0000 7fb481ffb700 1 -- 172.21.15.114:0/2708641 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb46c013070 con 0x7fb484107de0 2024-05-27T07:36:16.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:16.826+0000 7fb48a3bd700 1 -- 172.21.15.114:0/2708641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fb4841a9b30 con 0x7fb484107de0 2024-05-27T07:36:17.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:17.031+0000 7fb481ffb700 1 -- 172.21.15.114:0/2708641 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v84) ==== 169+0+0 (secure 0 0 0) 0x7fb46c0aafb0 con 0x7fb484107de0 2024-05-27T07:36:17.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:17.033+0000 7fb48a3bd700 1 -- 172.21.15.114:0/2708641 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb4700b2880 msgr2=0x7fb4700b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:17.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:17.033+0000 7fb48a3bd700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb4700b2880 0x7fb4700b4d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb474005fd0 tx=0x7fb474005e20 comp rx=0 tx=0).stop 2024-05-27T07:36:17.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:17.033+0000 7fb48a3bd700 1 -- 172.21.15.114:0/2708641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb484107de0 msgr2=0x7fb48419d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:17.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:17.033+0000 7fb48a3bd700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb484107de0 0x7fb48419d5f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0098e0 tx=0x7fb46c003680 comp rx=0 tx=0).stop 2024-05-27T07:36:17.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:17.033+0000 7fb48a3bd700 1 -- 172.21.15.114:0/2708641 shutdown_connections 2024-05-27T07:36:17.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:17.033+0000 7fb48a3bd700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb484108730 0x7fb4841a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:17.033+0000 7fb48a3bd700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb4700b2880 0x7fb4700b4d90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:17.033+0000 7fb48a3bd700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4841034b0 0x7fb48419d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:17.033+0000 7fb48a3bd700 1 --2- 172.21.15.114:0/2708641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb484107de0 0x7fb48419d5f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:17.033+0000 7fb48a3bd700 1 -- 172.21.15.114:0/2708641 >> 172.21.15.114:0/2708641 conn(0x7fb4840fede0 msgr2=0x7fb484075bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:17.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:17.034+0000 7fb48a3bd700 1 -- 172.21.15.114:0/2708641 shutdown_connections 2024-05-27T07:36:17.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:17.034+0000 7fb48a3bd700 1 -- 172.21.15.114:0/2708641 wait complete. 2024-05-27T07:36:17.031 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 4 allow_ec_overwrites to true 2024-05-27T07:36:17.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:17 smithi114 ceph-mon[32038]: pgmap v298: 129 pgs: 17 creating+peering, 15 unknown, 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 763 B/s wr, 2 op/s 2024-05-27T07:36:17.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:17 smithi114 ceph-mon[32038]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T07:36:17.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:17 smithi114 ceph-mon[32038]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:36:17.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:17 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2708641' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-27T07:36:17.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:17 smithi163 ceph-mon[36451]: pgmap v298: 129 pgs: 17 creating+peering, 15 unknown, 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 763 B/s wr, 2 op/s 2024-05-27T07:36:17.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:17 smithi163 ceph-mon[36451]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T07:36:17.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:17 smithi163 ceph-mon[36451]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:36:17.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:17 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2708641' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-27T07:36:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:17 smithi186 ceph-mon[36647]: pgmap v298: 129 pgs: 17 creating+peering, 15 unknown, 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 763 B/s wr, 2 op/s 2024-05-27T07:36:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:17 smithi186 ceph-mon[36647]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T07:36:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:17 smithi186 ceph-mon[36647]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:36:17.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:17 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2708641' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-27T07:36:17.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-05-27T07:36:18.095 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:18.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:18 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2708641' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-27T07:36:18.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:18 smithi114 ceph-mon[32038]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T07:36:18.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:18 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2708641' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-27T07:36:18.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:18 smithi163 ceph-mon[36451]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T07:36:18.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:18 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2708641' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-27T07:36:18.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:18 smithi186 ceph-mon[36647]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T07:36:19.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:19 smithi114 ceph-mon[32038]: pgmap v301: 129 pgs: 17 creating+peering, 8 unknown, 104 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:19.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:19 smithi163 ceph-mon[36451]: pgmap v301: 129 pgs: 17 creating+peering, 8 unknown, 104 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:19.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:19 smithi186 ceph-mon[36647]: pgmap v301: 129 pgs: 17 creating+peering, 8 unknown, 104 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:19.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.918+0000 7f66d7ee0700 1 -- 172.21.15.114:0/3948052168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d0108ef0 msgr2=0x7f66d0109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.918+0000 7f66d7ee0700 1 --2- 172.21.15.114:0/3948052168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d0108ef0 0x7f66d0109300 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f66c00099c0 tx=0x7f66c0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:36:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.918+0000 7f66d7ee0700 1 -- 172.21.15.114:0/3948052168 shutdown_connections 2024-05-27T07:36:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.918+0000 7f66d7ee0700 1 --2- 172.21.15.114:0/3948052168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66d0109840 0x7f66d01111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.918+0000 7f66d7ee0700 1 --2- 172.21.15.114:0/3948052168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66d01045c0 0x7f66d01049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.918+0000 7f66d7ee0700 1 --2- 172.21.15.114:0/3948052168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d0108ef0 0x7f66d0109300 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.918+0000 7f66d7ee0700 1 -- 172.21.15.114:0/3948052168 >> 172.21.15.114:0/3948052168 conn(0x7f66d00ffe90 msgr2=0x7f66d01022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.918+0000 7f66d7ee0700 1 -- 172.21.15.114:0/3948052168 shutdown_connections 2024-05-27T07:36:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.918+0000 7f66d7ee0700 1 -- 172.21.15.114:0/3948052168 wait complete. 2024-05-27T07:36:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d7ee0700 1 Processor -- start 2024-05-27T07:36:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d7ee0700 1 -- start start 2024-05-27T07:36:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d7ee0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66d01045c0 0x7f66d019ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d7ee0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d0108ef0 0x7f66d019b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d7ee0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66d0109840 0x7f66d019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d7ee0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f66d0114450 con 0x7f66d0108ef0 2024-05-27T07:36:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d7ee0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f66d01142d0 con 0x7f66d0109840 2024-05-27T07:36:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d7ee0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f66d01145d0 con 0x7f66d01045c0 2024-05-27T07:36:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d547b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d0108ef0 0x7f66d019b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d547b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d0108ef0 0x7f66d019b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54676/0 (socket says 172.21.15.114:54676) 2024-05-27T07:36:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d547b700 1 -- 172.21.15.114:0/2432628484 learned_addr learned my addr 172.21.15.114:0/2432628484 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d5c7c700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66d01045c0 0x7f66d019ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d647d700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66d0109840 0x7f66d019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d547b700 1 -- 172.21.15.114:0/2432628484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66d01045c0 msgr2=0x7f66d019ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d547b700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66d01045c0 0x7f66d019ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d547b700 1 -- 172.21.15.114:0/2432628484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66d0109840 msgr2=0x7f66d019f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.919+0000 7f66d547b700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66d0109840 0x7f66d019f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.920+0000 7f66d547b700 1 -- 172.21.15.114:0/2432628484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d019fd50 con 0x7f66d0108ef0 2024-05-27T07:36:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.920+0000 7f66d647d700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66d0109840 0x7f66d019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.920+0000 7f66d547b700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d0108ef0 0x7f66d019b420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f66c0006010 tx=0x7f66c00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:19.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.920+0000 7f66c6ffd700 1 -- 172.21.15.114:0/2432628484 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0027070 con 0x7f66d0108ef0 2024-05-27T07:36:19.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.920+0000 7f66c6ffd700 1 -- 172.21.15.114:0/2432628484 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66c0003f20 con 0x7f66d0108ef0 2024-05-27T07:36:19.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.920+0000 7f66c6ffd700 1 -- 172.21.15.114:0/2432628484 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0022cd0 con 0x7f66d0108ef0 2024-05-27T07:36:19.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.920+0000 7f66d7ee0700 1 -- 172.21.15.114:0/2432628484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66d019ffe0 con 0x7f66d0108ef0 2024-05-27T07:36:19.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.920+0000 7f66d7ee0700 1 -- 172.21.15.114:0/2432628484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66d01a04c0 con 0x7f66d0108ef0 2024-05-27T07:36:19.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.921+0000 7f66d7ee0700 1 -- 172.21.15.114:0/2432628484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66d00512b0 con 0x7f66d0108ef0 2024-05-27T07:36:19.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.922+0000 7f66c6ffd700 1 -- 172.21.15.114:0/2432628484 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f66c002a070 con 0x7f66d0108ef0 2024-05-27T07:36:19.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.923+0000 7f66c6ffd700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f66bc0b2880 0x7f66bc0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:19.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.923+0000 7f66d5c7c700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f66bc0b2880 0x7f66bc0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:19.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.923+0000 7f66c6ffd700 1 -- 172.21.15.114:0/2432628484 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8770+0+0 (secure 0 0 0) 0x7f66c00e32f0 con 0x7f66d0108ef0 2024-05-27T07:36:19.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.924+0000 7f66d5c7c700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f66bc0b2880 0x7f66bc0b4d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f66cc005950 tx=0x7f66cc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:19.928+0000 7f66c6ffd700 1 -- 172.21.15.114:0/2432628484 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f66c0013070 con 0x7f66d0108ef0 2024-05-27T07:36:20.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:20.178+0000 7f66d7ee0700 1 -- 172.21.15.114:0/2432628484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f66d002e5e0 con 0x7f66d0108ef0 2024-05-27T07:36:21.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:21.041+0000 7f66c6ffd700 1 -- 172.21.15.114:0/2432628484 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f66c00b1e20 con 0x7f66d0108ef0 2024-05-27T07:36:21.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:21.045+0000 7f66d7ee0700 1 -- 172.21.15.114:0/2432628484 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f66bc0b2880 msgr2=0x7f66bc0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:21.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:21.045+0000 7f66d7ee0700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f66bc0b2880 0x7f66bc0b4d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f66cc005950 tx=0x7f66cc00b410 comp rx=0 tx=0).stop 2024-05-27T07:36:21.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:21.045+0000 7f66d7ee0700 1 -- 172.21.15.114:0/2432628484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d0108ef0 msgr2=0x7f66d019b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:21.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:21.045+0000 7f66d7ee0700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d0108ef0 0x7f66d019b420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f66c0006010 tx=0x7f66c00047b0 comp rx=0 tx=0).stop 2024-05-27T07:36:21.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:21.045+0000 7f66d7ee0700 1 -- 172.21.15.114:0/2432628484 shutdown_connections 2024-05-27T07:36:21.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:21.045+0000 7f66d7ee0700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66d0109840 0x7f66d019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:21.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:21.045+0000 7f66d7ee0700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f66bc0b2880 0x7f66bc0b4d90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:21.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:21.045+0000 7f66d7ee0700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66d01045c0 0x7f66d019ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:21.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:21.045+0000 7f66d7ee0700 1 --2- 172.21.15.114:0/2432628484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d0108ef0 0x7f66d019b420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:21.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:21.045+0000 7f66d7ee0700 1 -- 172.21.15.114:0/2432628484 >> 172.21.15.114:0/2432628484 conn(0x7f66d00ffe90 msgr2=0x7f66d010a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:21.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:21.045+0000 7f66d7ee0700 1 -- 172.21.15.114:0/2432628484 shutdown_connections 2024-05-27T07:36:21.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:21.045+0000 7f66d7ee0700 1 -- 172.21.15.114:0/2432628484 wait complete. 2024-05-27T07:36:21.043 INFO:teuthology.orchestra.run.smithi114.stderr:added data pool 4 to fsmap 2024-05-27T07:36:21.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:21 smithi114 ceph-mon[32038]: pgmap v302: 129 pgs: 10 creating+peering, 119 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:21.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:21 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2432628484' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-27T07:36:21.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:21 smithi163 ceph-mon[36451]: pgmap v302: 129 pgs: 10 creating+peering, 119 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:21.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:21 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2432628484' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-27T07:36:21.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:21 smithi186 ceph-mon[36647]: pgmap v302: 129 pgs: 10 creating+peering, 119 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:21.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:21 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2432628484' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-27T07:36:21.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:36:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:22.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:22 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2432628484' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-27T07:36:22.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:22 smithi114 ceph-mon[32038]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T07:36:22.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:22 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:36:22.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:22 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2432628484' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-27T07:36:22.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:22 smithi163 ceph-mon[36451]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T07:36:22.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:22 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:36:22.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:22 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2432628484' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-27T07:36:22.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:22 smithi186 ceph-mon[36647]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T07:36:22.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:22 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:36:23.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:23 smithi114 ceph-mon[32038]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:23.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:23 smithi114 ceph-mon[32038]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-27T07:36:23.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:23 smithi114 ceph-mon[32038]: Cluster is now healthy 2024-05-27T07:36:23.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:23 smithi163 ceph-mon[36451]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:23.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:23 smithi163 ceph-mon[36451]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-27T07:36:23.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:23 smithi163 ceph-mon[36451]: Cluster is now healthy 2024-05-27T07:36:23.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:23 smithi186 ceph-mon[36647]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:23.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:23 smithi186 ceph-mon[36647]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-27T07:36:23.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:23 smithi186 ceph-mon[36647]: Cluster is now healthy 2024-05-27T07:36:23.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.867+0000 7f104451f700 1 -- 172.21.15.114:0/4267282129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c108730 msgr2=0x7f103c110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:23.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.867+0000 7f104451f700 1 --2- 172.21.15.114:0/4267282129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c108730 0x7f103c110090 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f103400b0d0 tx=0x7f103400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:36:23.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.867+0000 7f104451f700 1 -- 172.21.15.114:0/4267282129 shutdown_connections 2024-05-27T07:36:23.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.867+0000 7f104451f700 1 --2- 172.21.15.114:0/4267282129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c108730 0x7f103c110090 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:23.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.867+0000 7f104451f700 1 --2- 172.21.15.114:0/4267282129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f103c1034b0 0x7f103c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:23.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.867+0000 7f104451f700 1 --2- 172.21.15.114:0/4267282129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f103c107de0 0x7f103c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:23.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.867+0000 7f104451f700 1 -- 172.21.15.114:0/4267282129 >> 172.21.15.114:0/4267282129 conn(0x7f103c0fedb0 msgr2=0x7f103c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:23.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.867+0000 7f104451f700 1 -- 172.21.15.114:0/4267282129 shutdown_connections 2024-05-27T07:36:23.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.868+0000 7f104451f700 1 -- 172.21.15.114:0/4267282129 wait complete. 2024-05-27T07:36:23.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.868+0000 7f104451f700 1 Processor -- start 2024-05-27T07:36:23.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.868+0000 7f104451f700 1 -- start start 2024-05-27T07:36:23.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f104451f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f103c1034b0 0x7f103c19f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:23.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f104451f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c107de0 0x7f103c19fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:23.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f104451f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f103c108730 0x7f103c199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:23.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f104451f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f103c113380 con 0x7f103c107de0 2024-05-27T07:36:23.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f104451f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f103c113200 con 0x7f103c108730 2024-05-27T07:36:23.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f104451f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f103c113500 con 0x7f103c1034b0 2024-05-27T07:36:23.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f1041aba700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c107de0 0x7f103c19fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:23.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f1041aba700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c107de0 0x7f103c19fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54684/0 (socket says 172.21.15.114:54684) 2024-05-27T07:36:23.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f1041aba700 1 -- 172.21.15.114:0/1854377961 learned_addr learned my addr 172.21.15.114:0/1854377961 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:23.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f1042abc700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f103c108730 0x7f103c199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:23.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f10422bb700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f103c1034b0 0x7f103c19f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:23.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f1041aba700 1 -- 172.21.15.114:0/1854377961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f103c1034b0 msgr2=0x7f103c19f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:23.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f1041aba700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f103c1034b0 0x7f103c19f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:23.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f1041aba700 1 -- 172.21.15.114:0/1854377961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f103c108730 msgr2=0x7f103c199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:23.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f1041aba700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f103c108730 0x7f103c199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:23.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f1041aba700 1 -- 172.21.15.114:0/1854377961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103c19a1e0 con 0x7f103c107de0 2024-05-27T07:36:23.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f1041aba700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c107de0 0x7f103c19fe20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f103800d950 tx=0x7f103800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:23.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.869+0000 7f10337fe700 1 -- 172.21.15.114:0/1854377961 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038017070 con 0x7f103c107de0 2024-05-27T07:36:23.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.870+0000 7f10337fe700 1 -- 172.21.15.114:0/1854377961 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f103800de30 con 0x7f103c107de0 2024-05-27T07:36:23.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.870+0000 7f104451f700 1 -- 172.21.15.114:0/1854377961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f103c19a4d0 con 0x7f103c107de0 2024-05-27T07:36:23.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.870+0000 7f10337fe700 1 -- 172.21.15.114:0/1854377961 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038005440 con 0x7f103c107de0 2024-05-27T07:36:23.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.870+0000 7f104451f700 1 -- 172.21.15.114:0/1854377961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f103c1a6770 con 0x7f103c107de0 2024-05-27T07:36:23.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.870+0000 7f104451f700 1 -- 172.21.15.114:0/1854377961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f103c0512b0 con 0x7f103c107de0 2024-05-27T07:36:23.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.872+0000 7f10337fe700 1 -- 172.21.15.114:0/1854377961 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f1038004750 con 0x7f103c107de0 2024-05-27T07:36:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.873+0000 7f10337fe700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f10280b2880 0x7f10280b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.873+0000 7f10422bb700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f10280b2880 0x7f10280b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.873+0000 7f10337fe700 1 -- 172.21.15.114:0/1854377961 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8802+0+0 (secure 0 0 0) 0x7f10380e20e0 con 0x7f103c107de0 2024-05-27T07:36:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.873+0000 7f10422bb700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f10280b2880 0x7f10280b4d90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f102c005950 tx=0x7f102c005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:23.876+0000 7f10337fe700 1 -- 172.21.15.114:0/1854377961 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f10380a9bc0 con 0x7f103c107de0 2024-05-27T07:36:24.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.133+0000 7f104451f700 1 -- 172.21.15.114:0/1854377961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f103c02e5e0 con 0x7f103c107de0 2024-05-27T07:36:24.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.134+0000 7f10337fe700 1 -- 172.21.15.114:0/1854377961 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11818 (secure 0 0 0) 0x7f10380a94c0 con 0x7f103c107de0 2024-05-27T07:36:24.131 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:36:24.131 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:36:20.180250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:36:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.136+0000 7f104451f700 1 -- 172.21.15.114:0/1854377961 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f10280b2880 msgr2=0x7f10280b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.136+0000 7f104451f700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f10280b2880 0x7f10280b4d90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f102c005950 tx=0x7f102c005ed0 comp rx=0 tx=0).stop 2024-05-27T07:36:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.136+0000 7f104451f700 1 -- 172.21.15.114:0/1854377961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c107de0 msgr2=0x7f103c19fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.136+0000 7f104451f700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c107de0 0x7f103c19fe20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f103800d950 tx=0x7f103800dd10 comp rx=0 tx=0).stop 2024-05-27T07:36:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.136+0000 7f104451f700 1 -- 172.21.15.114:0/1854377961 shutdown_connections 2024-05-27T07:36:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.136+0000 7f104451f700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f103c108730 0x7f103c199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.136+0000 7f104451f700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f10280b2880 0x7f10280b4d90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.136+0000 7f104451f700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f103c1034b0 0x7f103c19f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.136+0000 7f104451f700 1 --2- 172.21.15.114:0/1854377961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c107de0 0x7f103c19fe20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.136+0000 7f104451f700 1 -- 172.21.15.114:0/1854377961 >> 172.21.15.114:0/1854377961 conn(0x7f103c0fedb0 msgr2=0x7f103c107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.137+0000 7f104451f700 1 -- 172.21.15.114:0/1854377961 shutdown_connections 2024-05-27T07:36:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:24.137+0000 7f104451f700 1 -- 172.21.15.114:0/1854377961 wait complete. 2024-05-27T07:36:24.134 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-05-27T07:36:24.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:36:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:25.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:25 smithi114 ceph-mon[32038]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:25.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:25 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1854377961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:36:25.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:25 smithi163 ceph-mon[36451]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:25.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:25 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1854377961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:36:25.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:25 smithi186 ceph-mon[36647]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:25.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:25 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1854377961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:36:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.121+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2826793757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7ec1045c0 msgr2=0x7fd7ec1049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.121+0000 7fd7f2fa0700 1 --2- 172.21.15.114:0/2826793757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7ec1045c0 0x7fd7ec1049d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40099c0 tx=0x7fd7e4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.121+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2826793757 shutdown_connections 2024-05-27T07:36:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.121+0000 7fd7f2fa0700 1 --2- 172.21.15.114:0/2826793757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7ec109840 0x7fd7ec1111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.121+0000 7fd7f2fa0700 1 --2- 172.21.15.114:0/2826793757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7ec1045c0 0x7fd7ec1049d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.121+0000 7fd7f2fa0700 1 --2- 172.21.15.114:0/2826793757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7ec108ef0 0x7fd7ec109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.121+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2826793757 >> 172.21.15.114:0/2826793757 conn(0x7fd7ec0ffe90 msgr2=0x7fd7ec1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.121+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2826793757 shutdown_connections 2024-05-27T07:36:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.121+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2826793757 wait complete. 2024-05-27T07:36:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.122+0000 7fd7f2fa0700 1 Processor -- start 2024-05-27T07:36:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.122+0000 7fd7f2fa0700 1 -- start start 2024-05-27T07:36:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.122+0000 7fd7f2fa0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7ec1045c0 0x7fd7ec19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.122+0000 7fd7f2fa0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7ec108ef0 0x7fd7ec19b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.122+0000 7fd7f2fa0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7ec109840 0x7fd7ec19f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.122+0000 7fd7f2fa0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec1144f0 con 0x7fd7ec1045c0 2024-05-27T07:36:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.122+0000 7fd7f2fa0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec114370 con 0x7fd7ec108ef0 2024-05-27T07:36:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.122+0000 7fd7f2fa0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec114670 con 0x7fd7ec109840 2024-05-27T07:36:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.122+0000 7fd7f0d3c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7ec1045c0 0x7fd7ec19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.122+0000 7fd7f0d3c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7ec1045c0 0x7fd7ec19ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57986/0 (socket says 172.21.15.114:57986) 2024-05-27T07:36:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.122+0000 7fd7f0d3c700 1 -- 172.21.15.114:0/2679321313 learned_addr learned my addr 172.21.15.114:0/2679321313 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.123+0000 7fd7ebfff700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7ec108ef0 0x7fd7ec19b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.123+0000 7fd7f153d700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7ec109840 0x7fd7ec19f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.123+0000 7fd7f0d3c700 1 -- 172.21.15.114:0/2679321313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7ec109840 msgr2=0x7fd7ec19f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.123+0000 7fd7f0d3c700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7ec109840 0x7fd7ec19f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.123+0000 7fd7f0d3c700 1 -- 172.21.15.114:0/2679321313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7ec108ef0 msgr2=0x7fd7ec19b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.123+0000 7fd7f0d3c700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7ec108ef0 0x7fd7ec19b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.123+0000 7fd7f0d3c700 1 -- 172.21.15.114:0/2679321313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ec19fcf0 con 0x7fd7ec1045c0 2024-05-27T07:36:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.123+0000 7fd7f0d3c700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7ec1045c0 0x7fd7ec19ae20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400b580 tx=0x7fd7e40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.123+0000 7fd7e9ffb700 1 -- 172.21.15.114:0/2679321313 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4027070 con 0x7fd7ec1045c0 2024-05-27T07:36:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.123+0000 7fd7e9ffb700 1 -- 172.21.15.114:0/2679321313 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7e4003f20 con 0x7fd7ec1045c0 2024-05-27T07:36:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.123+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2679321313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ec19ff80 con 0x7fd7ec1045c0 2024-05-27T07:36:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.123+0000 7fd7e9ffb700 1 -- 172.21.15.114:0/2679321313 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4022d70 con 0x7fd7ec1045c0 2024-05-27T07:36:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.123+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2679321313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ec1a0460 con 0x7fd7ec1045c0 2024-05-27T07:36:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.124+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2679321313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7ec0512b0 con 0x7fd7ec1045c0 2024-05-27T07:36:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.126+0000 7fd7e9ffb700 1 -- 172.21.15.114:0/2679321313 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fd7e4022500 con 0x7fd7ec1045c0 2024-05-27T07:36:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.126+0000 7fd7e9ffb700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fd7d80b2880 0x7fd7d80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.127+0000 7fd7ebfff700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fd7d80b2880 0x7fd7d80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.127+0000 7fd7e9ffb700 1 -- 172.21.15.114:0/2679321313 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8802+0+0 (secure 0 0 0) 0x7fd7e40e3570 con 0x7fd7ec1045c0 2024-05-27T07:36:27.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.127+0000 7fd7ebfff700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fd7d80b2880 0x7fd7d80b4d90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009550 tx=0x7fd7dc0091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:27.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.130+0000 7fd7e9ffb700 1 -- 172.21.15.114:0/2679321313 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd7e4013070 con 0x7fd7ec1045c0 2024-05-27T07:36:27.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:27 smithi114 ceph-mon[32038]: pgmap v306: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:27.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.335+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2679321313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7ec02e5e0 con 0x7fd7ec1045c0 2024-05-27T07:36:27.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.337+0000 7fd7e9ffb700 1 -- 172.21.15.114:0/2679321313 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24210 (secure 0 0 0) 0x7fd7e40ab0e0 con 0x7fd7ec1045c0 2024-05-27T07:36:27.333 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:36:27.334 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":85,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T07:36:20.180067+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:36:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.339+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2679321313 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fd7d80b2880 msgr2=0x7fd7d80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.339+0000 7fd7f2fa0700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fd7d80b2880 0x7fd7d80b4d90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009550 tx=0x7fd7dc0091b0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.339+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2679321313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7ec1045c0 msgr2=0x7fd7ec19ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.339+0000 7fd7f2fa0700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7ec1045c0 0x7fd7ec19ae20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400b580 tx=0x7fd7e40047b0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.339+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2679321313 shutdown_connections 2024-05-27T07:36:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.339+0000 7fd7f2fa0700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fd7d80b2880 0x7fd7d80b4d90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.339+0000 7fd7f2fa0700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7ec109840 0x7fd7ec19f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.339+0000 7fd7f2fa0700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7ec1045c0 0x7fd7ec19ae20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.339+0000 7fd7f2fa0700 1 --2- 172.21.15.114:0/2679321313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7ec108ef0 0x7fd7ec19b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.339+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2679321313 >> 172.21.15.114:0/2679321313 conn(0x7fd7ec0ffe90 msgr2=0x7fd7ec10a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.340+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2679321313 shutdown_connections 2024-05-27T07:36:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:27.340+0000 7fd7f2fa0700 1 -- 172.21.15.114:0/2679321313 wait complete. 2024-05-27T07:36:27.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:27 smithi163 ceph-mon[36451]: pgmap v306: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:27.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:27 smithi186 ceph-mon[36647]: pgmap v306: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:28.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:36:28.286 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:28.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:28 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2679321313' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:28.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:28 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2679321313' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:28.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:28 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2679321313' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:29.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:29 smithi163 ceph-mon[36451]: pgmap v307: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:29.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:29 smithi186 ceph-mon[36647]: pgmap v307: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:29.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:29 smithi114 ceph-mon[32038]: pgmap v307: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:30.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.071+0000 7f00ef4ca700 1 -- 172.21.15.114:0/3761540280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00e8107de0 msgr2=0x7f00e81081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.071+0000 7f00ef4ca700 1 --2- 172.21.15.114:0/3761540280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00e8107de0 0x7f00e81081f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f00d80099c0 tx=0x7f00d8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.071+0000 7f00ef4ca700 1 -- 172.21.15.114:0/3761540280 shutdown_connections 2024-05-27T07:36:30.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.071+0000 7f00ef4ca700 1 --2- 172.21.15.114:0/3761540280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8108730 0x7f00e8110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.071+0000 7f00ef4ca700 1 --2- 172.21.15.114:0/3761540280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00e81034b0 0x7f00e81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.071+0000 7f00ef4ca700 1 --2- 172.21.15.114:0/3761540280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00e8107de0 0x7f00e81081f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.071+0000 7f00ef4ca700 1 -- 172.21.15.114:0/3761540280 >> 172.21.15.114:0/3761540280 conn(0x7f00e80fedb0 msgr2=0x7f00e81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.071+0000 7f00ef4ca700 1 -- 172.21.15.114:0/3761540280 shutdown_connections 2024-05-27T07:36:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.071+0000 7f00ef4ca700 1 -- 172.21.15.114:0/3761540280 wait complete. 2024-05-27T07:36:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.071+0000 7f00ef4ca700 1 Processor -- start 2024-05-27T07:36:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.071+0000 7f00ef4ca700 1 -- start start 2024-05-27T07:36:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00ef4ca700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00e81034b0 0x7f00e819ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00ef4ca700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8107de0 0x7f00e819b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00ef4ca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00e8108730 0x7f00e819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00ef4ca700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f00e8113390 con 0x7f00e8108730 2024-05-27T07:36:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00ef4ca700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f00e8113210 con 0x7f00e81034b0 2024-05-27T07:36:30.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00ef4ca700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f00e8113510 con 0x7f00e8107de0 2024-05-27T07:36:30.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00eda67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00e8108730 0x7f00e819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00eda67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00e8108730 0x7f00e819f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58002/0 (socket says 172.21.15.114:58002) 2024-05-27T07:36:30.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00eda67700 1 -- 172.21.15.114:0/2521474206 learned_addr learned my addr 172.21.15.114:0/2521474206 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:30.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00eca65700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8107de0 0x7f00e819b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00ed266700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00e81034b0 0x7f00e819ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00eda67700 1 -- 172.21.15.114:0/2521474206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8107de0 msgr2=0x7f00e819b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00eda67700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8107de0 0x7f00e819b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00eda67700 1 -- 172.21.15.114:0/2521474206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00e81034b0 msgr2=0x7f00e819ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00eda67700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00e81034b0 0x7f00e819ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00eda67700 1 -- 172.21.15.114:0/2521474206 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e819fd20 con 0x7f00e8108730 2024-05-27T07:36:30.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00eda67700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00e8108730 0x7f00e819f680 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f00e0007ee0 tx=0x7f00e000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:30.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00de7fc700 1 -- 172.21.15.114:0/2521474206 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e000dbd0 con 0x7f00e8108730 2024-05-27T07:36:30.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00ef4ca700 1 -- 172.21.15.114:0/2521474206 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e81a0010 con 0x7f00e8108730 2024-05-27T07:36:30.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.072+0000 7f00de7fc700 1 -- 172.21.15.114:0/2521474206 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00e000dd70 con 0x7f00e8108730 2024-05-27T07:36:30.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.073+0000 7f00de7fc700 1 -- 172.21.15.114:0/2521474206 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e0005990 con 0x7f00e8108730 2024-05-27T07:36:30.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.073+0000 7f00ef4ca700 1 -- 172.21.15.114:0/2521474206 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e81a7930 con 0x7f00e8108730 2024-05-27T07:36:30.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.073+0000 7f00ef4ca700 1 -- 172.21.15.114:0/2521474206 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00e80512b0 con 0x7f00e8108730 2024-05-27T07:36:30.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.075+0000 7f00de7fc700 1 -- 172.21.15.114:0/2521474206 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f00e001a020 con 0x7f00e8108730 2024-05-27T07:36:30.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.076+0000 7f00de7fc700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f00d40b2880 0x7f00d40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.076+0000 7f00ed266700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f00d40b2880 0x7f00d40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.076+0000 7f00de7fc700 1 -- 172.21.15.114:0/2521474206 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8802+0+0 (secure 0 0 0) 0x7f00e00e2210 con 0x7f00e8108730 2024-05-27T07:36:30.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.076+0000 7f00ed266700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f00d40b2880 0x7f00d40b4d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f00e4005950 tx=0x7f00e40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:30.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.079+0000 7f00de7fc700 1 -- 172.21.15.114:0/2521474206 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f00e0010070 con 0x7f00e8108730 2024-05-27T07:36:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.285+0000 7f00ef4ca700 1 -- 172.21.15.114:0/2521474206 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f00e802e5e0 con 0x7f00e8108730 2024-05-27T07:36:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.286+0000 7f00de7fc700 1 -- 172.21.15.114:0/2521474206 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24210 (secure 0 0 0) 0x7f00e00a9cf0 con 0x7f00e8108730 2024-05-27T07:36:30.283 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:36:30.283 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":85,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T07:36:20.180067+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:36:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.289+0000 7f00ef4ca700 1 -- 172.21.15.114:0/2521474206 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f00d40b2880 msgr2=0x7f00d40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.289+0000 7f00ef4ca700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f00d40b2880 0x7f00d40b4d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f00e4005950 tx=0x7f00e40058e0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.289+0000 7f00ef4ca700 1 -- 172.21.15.114:0/2521474206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00e8108730 msgr2=0x7f00e819f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.289+0000 7f00ef4ca700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00e8108730 0x7f00e819f680 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f00e0007ee0 tx=0x7f00e000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.289+0000 7f00ef4ca700 1 -- 172.21.15.114:0/2521474206 shutdown_connections 2024-05-27T07:36:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.289+0000 7f00ef4ca700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00e8108730 0x7f00e819f680 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.289+0000 7f00ef4ca700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00e81034b0 0x7f00e819ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.289+0000 7f00ef4ca700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f00d40b2880 0x7f00d40b4d90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.289+0000 7f00ef4ca700 1 --2- 172.21.15.114:0/2521474206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8107de0 0x7f00e819b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.289+0000 7f00ef4ca700 1 -- 172.21.15.114:0/2521474206 >> 172.21.15.114:0/2521474206 conn(0x7f00e80fedb0 msgr2=0x7f00e806b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:30.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.289+0000 7f00ef4ca700 1 -- 172.21.15.114:0/2521474206 shutdown_connections 2024-05-27T07:36:30.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:30.289+0000 7f00ef4ca700 1 -- 172.21.15.114:0/2521474206 wait complete. 2024-05-27T07:36:31.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:36:31.291 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:31.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:31 smithi114 ceph-mon[32038]: pgmap v308: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:31.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:31 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2521474206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:31.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:31 smithi163 ceph-mon[36451]: pgmap v308: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:31.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:31 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2521474206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:31.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:31 smithi186 ceph-mon[36647]: pgmap v308: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:31.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:31 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2521474206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:32.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.949+0000 7f27cd4f2700 1 -- 172.21.15.114:0/1740982595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c8108750 msgr2=0x7f27c8110140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:32.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.949+0000 7f27cd4f2700 1 --2- 172.21.15.114:0/1740982595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c8108750 0x7f27c8110140 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f27c000b0d0 tx=0x7f27c000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:36:32.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.949+0000 7f27cd4f2700 1 -- 172.21.15.114:0/1740982595 shutdown_connections 2024-05-27T07:36:32.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.949+0000 7f27cd4f2700 1 --2- 172.21.15.114:0/1740982595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c8108750 0x7f27c8110140 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:32.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.949+0000 7f27cd4f2700 1 --2- 172.21.15.114:0/1740982595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27c8103c90 0x7f27c81040a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:32.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.949+0000 7f27cd4f2700 1 --2- 172.21.15.114:0/1740982595 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27c8107e00 0x7f27c8108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:32.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.949+0000 7f27cd4f2700 1 -- 172.21.15.114:0/1740982595 >> 172.21.15.114:0/1740982595 conn(0x7f27c80fe6e0 msgr2=0x7f27c8100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:32.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.949+0000 7f27cd4f2700 1 -- 172.21.15.114:0/1740982595 shutdown_connections 2024-05-27T07:36:32.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.949+0000 7f27cd4f2700 1 -- 172.21.15.114:0/1740982595 wait complete. 2024-05-27T07:36:32.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.950+0000 7f27cd4f2700 1 Processor -- start 2024-05-27T07:36:32.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.950+0000 7f27cd4f2700 1 -- start start 2024-05-27T07:36:32.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27cd4f2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27c8103c90 0x7f27c819ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:32.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27cd4f2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c8107e00 0x7f27c819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:32.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27cd4f2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27c8108750 0x7f27c819f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:32.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27cd4f2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f27c8113370 con 0x7f27c8107e00 2024-05-27T07:36:32.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27cd4f2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f27c81131f0 con 0x7f27c8103c90 2024-05-27T07:36:32.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27cd4f2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f27c81134f0 con 0x7f27c8108750 2024-05-27T07:36:32.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27c67fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c8107e00 0x7f27c819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:32.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27c67fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c8107e00 0x7f27c819b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58010/0 (socket says 172.21.15.114:58010) 2024-05-27T07:36:32.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27c67fc700 1 -- 172.21.15.114:0/112625926 learned_addr learned my addr 172.21.15.114:0/112625926 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:32.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27c6ffd700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27c8103c90 0x7f27c819ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:32.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27c77fe700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27c8108750 0x7f27c819f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:32.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27c67fc700 1 -- 172.21.15.114:0/112625926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27c8108750 msgr2=0x7f27c819f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:32.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27c67fc700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27c8108750 0x7f27c819f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:32.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27c67fc700 1 -- 172.21.15.114:0/112625926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27c8103c90 msgr2=0x7f27c819ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:32.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27c67fc700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27c8103c90 0x7f27c819ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:32.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27c67fc700 1 -- 172.21.15.114:0/112625926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27c819fbb0 con 0x7f27c8107e00 2024-05-27T07:36:32.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27c77fe700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27c8108750 0x7f27c819f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:32.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27c67fc700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c8107e00 0x7f27c819b330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00c8f0 tx=0x7f27bc00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:32.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27affff700 1 -- 172.21.15.114:0/112625926 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc007940 con 0x7f27c8107e00 2024-05-27T07:36:32.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27cd4f2700 1 -- 172.21.15.114:0/112625926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27c819fea0 con 0x7f27c8107e00 2024-05-27T07:36:32.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.951+0000 7f27affff700 1 -- 172.21.15.114:0/112625926 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27bc00f450 con 0x7f27c8107e00 2024-05-27T07:36:32.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.952+0000 7f27affff700 1 -- 172.21.15.114:0/112625926 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc018660 con 0x7f27c8107e00 2024-05-27T07:36:32.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.952+0000 7f27cd4f2700 1 -- 172.21.15.114:0/112625926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c81a7870 con 0x7f27c8107e00 2024-05-27T07:36:32.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.953+0000 7f27cd4f2700 1 -- 172.21.15.114:0/112625926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27c80512b0 con 0x7f27c8107e00 2024-05-27T07:36:32.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.954+0000 7f27affff700 1 -- 172.21.15.114:0/112625926 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f27bc00f890 con 0x7f27c8107e00 2024-05-27T07:36:32.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.955+0000 7f27affff700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f27b00b2880 0x7f27b00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:32.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.955+0000 7f27affff700 1 -- 172.21.15.114:0/112625926 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8802+0+0 (secure 0 0 0) 0x7f27bc0e2370 con 0x7f27c8107e00 2024-05-27T07:36:32.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.955+0000 7f27c6ffd700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f27b00b2880 0x7f27b00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:32.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.955+0000 7f27c6ffd700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f27b00b2880 0x7f27b00b4d90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f27b800ac50 tx=0x7f27b800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:32.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:32.958+0000 7f27affff700 1 -- 172.21.15.114:0/112625926 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f27bc014030 con 0x7f27c8107e00 2024-05-27T07:36:33.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.206+0000 7f27cd4f2700 1 -- 172.21.15.114:0/112625926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27c819c0f0 con 0x7f27c8107e00 2024-05-27T07:36:33.203 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:33 smithi114 ceph-mon[32038]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:33.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.207+0000 7f27affff700 1 -- 172.21.15.114:0/112625926 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11818 (secure 0 0 0) 0x7f27bc0a9ee0 con 0x7f27c8107e00 2024-05-27T07:36:33.203 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:36:33.203 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:36:20.180250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:36:33.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.209+0000 7f27cd4f2700 1 -- 172.21.15.114:0/112625926 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f27b00b2880 msgr2=0x7f27b00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:33.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.209+0000 7f27cd4f2700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f27b00b2880 0x7f27b00b4d90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f27b800ac50 tx=0x7f27b800a380 comp rx=0 tx=0).stop 2024-05-27T07:36:33.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.209+0000 7f27cd4f2700 1 -- 172.21.15.114:0/112625926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c8107e00 msgr2=0x7f27c819b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:33.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.209+0000 7f27cd4f2700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c8107e00 0x7f27c819b330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00c8f0 tx=0x7f27bc00cc00 comp rx=0 tx=0).stop 2024-05-27T07:36:33.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.209+0000 7f27cd4f2700 1 -- 172.21.15.114:0/112625926 shutdown_connections 2024-05-27T07:36:33.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.209+0000 7f27cd4f2700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f27b00b2880 0x7f27b00b4d90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:33.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.209+0000 7f27cd4f2700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27c8108750 0x7f27c819f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:33.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.209+0000 7f27cd4f2700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27c8103c90 0x7f27c819ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:33.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.209+0000 7f27cd4f2700 1 --2- 172.21.15.114:0/112625926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c8107e00 0x7f27c819b330 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:33.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.209+0000 7f27cd4f2700 1 -- 172.21.15.114:0/112625926 >> 172.21.15.114:0/112625926 conn(0x7f27c80fe6e0 msgr2=0x7f27c8102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:33.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.209+0000 7f27cd4f2700 1 -- 172.21.15.114:0/112625926 shutdown_connections 2024-05-27T07:36:33.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:33.209+0000 7f27cd4f2700 1 -- 172.21.15.114:0/112625926 wait complete. 2024-05-27T07:36:33.206 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-05-27T07:36:33.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:33 smithi163 ceph-mon[36451]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:33.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:33 smithi186 ceph-mon[36647]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:33.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:36:34.045 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:34.406 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:34 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/112625926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:36:34.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:34 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/112625926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:36:34.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:34 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/112625926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:36:35.145 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:35 smithi114 ceph-mon[32038]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:35.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:35 smithi163 ceph-mon[36451]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:35.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:35 smithi186 ceph-mon[36647]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:35.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.665+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/327112735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf81092f0 msgr2=0x7f1cf8109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.665+0000 7f1cfdb7a700 1 --2- 172.21.15.114:0/327112735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf81092f0 0x7f1cf8109700 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009990 tx=0x7f1ce8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.665+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/327112735 shutdown_connections 2024-05-27T07:36:35.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.665+0000 7f1cfdb7a700 1 --2- 172.21.15.114:0/327112735 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cf81030d0 0x7f1cf8111160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.665+0000 7f1cfdb7a700 1 --2- 172.21.15.114:0/327112735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf81092f0 0x7f1cf8109700 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.665+0000 7f1cfdb7a700 1 --2- 172.21.15.114:0/327112735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cf8102780 0x7f1cf8102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.665+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/327112735 >> 172.21.15.114:0/327112735 conn(0x7f1cf80fe4d0 msgr2=0x7f1cf8100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.665+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/327112735 shutdown_connections 2024-05-27T07:36:35.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.665+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/327112735 wait complete. 2024-05-27T07:36:35.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.665+0000 7f1cfdb7a700 1 Processor -- start 2024-05-27T07:36:35.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cfdb7a700 1 -- start start 2024-05-27T07:36:35.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cfdb7a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8102780 0x7f1cf819af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cfdb7a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cf81030d0 0x7f1cf819b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cfdb7a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cf81092f0 0x7f1cf819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cfdb7a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf81145f0 con 0x7f1cf8102780 2024-05-27T07:36:35.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cfdb7a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf8114470 con 0x7f1cf81030d0 2024-05-27T07:36:35.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cfdb7a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf8114770 con 0x7f1cf81092f0 2024-05-27T07:36:35.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cf77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8102780 0x7f1cf819af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cf77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8102780 0x7f1cf819af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40298/0 (socket says 172.21.15.114:40298) 2024-05-27T07:36:35.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cf77fe700 1 -- 172.21.15.114:0/3797112311 learned_addr learned my addr 172.21.15.114:0/3797112311 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:35.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cf7fff700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cf81092f0 0x7f1cf819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cf6ffd700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cf81030d0 0x7f1cf819b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cf77fe700 1 -- 172.21.15.114:0/3797112311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cf81092f0 msgr2=0x7f1cf819f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cf77fe700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cf81092f0 0x7f1cf819f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cf77fe700 1 -- 172.21.15.114:0/3797112311 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cf81030d0 msgr2=0x7f1cf819b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cf77fe700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cf81030d0 0x7f1cf819b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cf77fe700 1 -- 172.21.15.114:0/3797112311 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf819fe50 con 0x7f1cf8102780 2024-05-27T07:36:35.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.666+0000 7f1cf6ffd700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cf81030d0 0x7f1cf819b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:35.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.667+0000 7f1cf77fe700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8102780 0x7f1cf819af80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8006010 tx=0x7f1ce8003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.667+0000 7f1cf4ff9700 1 -- 172.21.15.114:0/3797112311 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8027070 con 0x7f1cf8102780 2024-05-27T07:36:35.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.667+0000 7f1cf4ff9700 1 -- 172.21.15.114:0/3797112311 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ce8004110 con 0x7f1cf8102780 2024-05-27T07:36:35.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.667+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/3797112311 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf81a00e0 con 0x7f1cf8102780 2024-05-27T07:36:35.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.667+0000 7f1cf4ff9700 1 -- 172.21.15.114:0/3797112311 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8022e60 con 0x7f1cf8102780 2024-05-27T07:36:35.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.667+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/3797112311 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf81a05c0 con 0x7f1cf8102780 2024-05-27T07:36:35.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.668+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/3797112311 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf810c890 con 0x7f1cf8102780 2024-05-27T07:36:35.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.669+0000 7f1cf4ff9700 1 -- 172.21.15.114:0/3797112311 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f1ce802a070 con 0x7f1cf8102780 2024-05-27T07:36:35.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.670+0000 7f1cf4ff9700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f1ce00b2880 0x7f1ce00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.670+0000 7f1cf6ffd700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f1ce00b2880 0x7f1ce00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.670+0000 7f1cf4ff9700 1 -- 172.21.15.114:0/3797112311 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8802+0+0 (secure 0 0 0) 0x7f1ce80e38a0 con 0x7f1cf8102780 2024-05-27T07:36:35.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.670+0000 7f1cf6ffd700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f1ce00b2880 0x7f1ce00b4d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8108970 tx=0x7f1cec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.674+0000 7f1cf4ff9700 1 -- 172.21.15.114:0/3797112311 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1ce8013070 con 0x7f1cf8102780 2024-05-27T07:36:35.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.880+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/3797112311 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1cf819c2e0 con 0x7f1cf8102780 2024-05-27T07:36:35.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.882+0000 7f1cf4ff9700 1 -- 172.21.15.114:0/3797112311 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24210 (secure 0 0 0) 0x7f1ce80ab410 con 0x7f1cf8102780 2024-05-27T07:36:35.879 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:36:35.879 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":85,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T07:36:20.180067+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:36:35.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.884+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/3797112311 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f1ce00b2880 msgr2=0x7f1ce00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.884+0000 7f1cfdb7a700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f1ce00b2880 0x7f1ce00b4d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8108970 tx=0x7f1cec008040 comp rx=0 tx=0).stop 2024-05-27T07:36:35.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.884+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/3797112311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8102780 msgr2=0x7f1cf819af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.884+0000 7f1cfdb7a700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8102780 0x7f1cf819af80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8006010 tx=0x7f1ce8003730 comp rx=0 tx=0).stop 2024-05-27T07:36:35.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.884+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/3797112311 shutdown_connections 2024-05-27T07:36:35.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.884+0000 7f1cfdb7a700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f1ce00b2880 0x7f1ce00b4d90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.884+0000 7f1cfdb7a700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cf81092f0 0x7f1cf819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.884+0000 7f1cfdb7a700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8102780 0x7f1cf819af80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.884+0000 7f1cfdb7a700 1 --2- 172.21.15.114:0/3797112311 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cf81030d0 0x7f1cf819b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.884+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/3797112311 >> 172.21.15.114:0/3797112311 conn(0x7f1cf80fe4d0 msgr2=0x7f1cf8107100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.884+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/3797112311 shutdown_connections 2024-05-27T07:36:35.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:35.884+0000 7f1cfdb7a700 1 -- 172.21.15.114:0/3797112311 wait complete. 2024-05-27T07:36:36.235 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:36 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3797112311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:36.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:36 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3797112311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:36.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:36 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3797112311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:36.586 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-27T07:36:06.848027+0000', 'modified': '2024-05-27T07:36:20.180250+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24463': {'gid': 24463, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.186:6839/4115872608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 4115872608}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 4115872608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24463, 'qdb_cluster': [24463]} 2024-05-27T07:36:36.586 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-27T07:36:36.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-05-27T07:36:36.831 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:37.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:37 smithi114 ceph-mon[32038]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:37.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:37 smithi163 ceph-mon[36451]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:37.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:37 smithi186 ceph-mon[36647]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:38.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.533+0000 7f17f521e700 1 -- 172.21.15.114:0/3869311332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0107de0 msgr2=0x7f17f01081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:38.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.533+0000 7f17f521e700 1 --2- 172.21.15.114:0/3869311332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0107de0 0x7f17f01081f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f17d80099c0 tx=0x7f17d8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.533+0000 7f17f521e700 1 -- 172.21.15.114:0/3869311332 shutdown_connections 2024-05-27T07:36:38.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.533+0000 7f17f521e700 1 --2- 172.21.15.114:0/3869311332 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17f0108730 0x7f17f0110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.533+0000 7f17f521e700 1 --2- 172.21.15.114:0/3869311332 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17f01034b0 0x7f17f01038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.533+0000 7f17f521e700 1 --2- 172.21.15.114:0/3869311332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0107de0 0x7f17f01081f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.533+0000 7f17f521e700 1 -- 172.21.15.114:0/3869311332 >> 172.21.15.114:0/3869311332 conn(0x7f17f00fede0 msgr2=0x7f17f0101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:38.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.533+0000 7f17f521e700 1 -- 172.21.15.114:0/3869311332 shutdown_connections 2024-05-27T07:36:38.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.533+0000 7f17f521e700 1 -- 172.21.15.114:0/3869311332 wait complete. 2024-05-27T07:36:38.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.534+0000 7f17f521e700 1 Processor -- start 2024-05-27T07:36:38.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.534+0000 7f17f521e700 1 -- start start 2024-05-27T07:36:38.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17f521e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17f01034b0 0x7f17f019ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17f521e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17f0107de0 0x7f17f019b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17f521e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0108730 0x7f17f019f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17f521e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0113390 con 0x7f17f0108730 2024-05-27T07:36:38.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17f521e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0113210 con 0x7f17f0107de0 2024-05-27T07:36:38.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17f521e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0113510 con 0x7f17f01034b0 2024-05-27T07:36:38.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17ef59e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0108730 0x7f17f019f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17ef59e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0108730 0x7f17f019f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40318/0 (socket says 172.21.15.114:40318) 2024-05-27T07:36:38.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17ef59e700 1 -- 172.21.15.114:0/3005437570 learned_addr learned my addr 172.21.15.114:0/3005437570 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:38.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17e659c700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17f0107de0 0x7f17f019b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17eed9d700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17f01034b0 0x7f17f019ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17ef59e700 1 -- 172.21.15.114:0/3005437570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17f01034b0 msgr2=0x7f17f019ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:38.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17ef59e700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17f01034b0 0x7f17f019ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17ef59e700 1 -- 172.21.15.114:0/3005437570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17f0107de0 msgr2=0x7f17f019b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:38.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17ef59e700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17f0107de0 0x7f17f019b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17ef59e700 1 -- 172.21.15.114:0/3005437570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f019fd40 con 0x7f17f0108730 2024-05-27T07:36:38.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17ef59e700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0108730 0x7f17f019f6a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f17e800ada0 tx=0x7f17e800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:38.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17ecd99700 1 -- 172.21.15.114:0/3005437570 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8017020 con 0x7f17f0108730 2024-05-27T07:36:38.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17ecd99700 1 -- 172.21.15.114:0/3005437570 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e800cd80 con 0x7f17f0108730 2024-05-27T07:36:38.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17ecd99700 1 -- 172.21.15.114:0/3005437570 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8005260 con 0x7f17f0108730 2024-05-27T07:36:38.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.535+0000 7f17f521e700 1 -- 172.21.15.114:0/3005437570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f01a0030 con 0x7f17f0108730 2024-05-27T07:36:38.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.536+0000 7f17f521e700 1 -- 172.21.15.114:0/3005437570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f01a7950 con 0x7f17f0108730 2024-05-27T07:36:38.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.539+0000 7f17ecd99700 1 -- 172.21.15.114:0/3005437570 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f17e800c8d0 con 0x7f17f0108730 2024-05-27T07:36:38.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.540+0000 7f17ecd99700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f17dc0b2880 0x7f17dc0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.540+0000 7f17eed9d700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f17dc0b2880 0x7f17dc0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.540+0000 7f1803c33b80 1 Processor -- start 2024-05-27T07:36:38.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.540+0000 7f17ecd99700 1 -- 172.21.15.114:0/3005437570 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8802+0+0 (secure 0 0 0) 0x7f17e80e1ac0 con 0x7f17f0108730 2024-05-27T07:36:38.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.540+0000 7f1803c33b80 1 -- start start 2024-05-27T07:36:38.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f17eed9d700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f17dc0b2880 0x7f17dc0b4d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f17e0005950 tx=0x7f17e000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:38.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f1803c33b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf014df0 0x5631cf1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f1803c33b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 0x5631cf1929f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f1803c33b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf192f30 0x5631cf193340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f1803c33b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5631cf0c1c40 con 0x5631cf198c00 2024-05-27T07:36:38.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f1803c33b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5631cf0b17e0 con 0x5631cf192f30 2024-05-27T07:36:38.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f1803c33b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5631cf0b5820 con 0x5631cf014df0 2024-05-27T07:36:38.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f17e659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 0x5631cf1929f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f17e659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 0x5631cf1929f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40326/0 (socket says 172.21.15.114:40326) 2024-05-27T07:36:38.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f17e659c700 1 -- 172.21.15.114:0/3782431493 learned_addr learned my addr 172.21.15.114:0/3782431493 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:38.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f17eed9d700 1 --2- 172.21.15.114:0/3782431493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf014df0 0x5631cf1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f17ef59e700 1 --2- 172.21.15.114:0/3782431493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf192f30 0x5631cf193340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f17e659c700 1 -- 172.21.15.114:0/3782431493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf014df0 msgr2=0x5631cf1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:38.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f17e659c700 1 --2- 172.21.15.114:0/3782431493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf014df0 0x5631cf1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f17e659c700 1 -- 172.21.15.114:0/3782431493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf192f30 msgr2=0x5631cf193340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:38.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f17e659c700 1 --2- 172.21.15.114:0/3782431493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf192f30 0x5631cf193340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f17e659c700 1 -- 172.21.15.114:0/3782431493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631cf001880 con 0x5631cf198c00 2024-05-27T07:36:38.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f17ef59e700 1 --2- 172.21.15.114:0/3782431493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf192f30 0x5631cf193340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:38.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.541+0000 7f17eed9d700 1 --2- 172.21.15.114:0/3782431493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf014df0 0x5631cf1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:38.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.542+0000 7f17e659c700 1 --2- 172.21.15.114:0/3782431493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 0x5631cf1929f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f17d8000c00 tx=0x7f17d8003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a789c023eb26af8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:38.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.542+0000 7f17e559a700 1 -- 172.21.15.114:0/3782431493 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8003c10 con 0x5631cf198c00 2024-05-27T07:36:38.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.542+0000 7f17e559a700 1 -- 172.21.15.114:0/3782431493 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17d8003db0 con 0x5631cf198c00 2024-05-27T07:36:38.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.542+0000 7f17e559a700 1 -- 172.21.15.114:0/3782431493 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d800be90 con 0x5631cf198c00 2024-05-27T07:36:38.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.543+0000 7f1803c33b80 1 -- 172.21.15.114:0/3782431493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 msgr2=0x5631cf1929f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:38.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.543+0000 7f1803c33b80 1 --2- 172.21.15.114:0/3782431493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 0x5631cf1929f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f17d8000c00 tx=0x7f17d8003760 comp rx=0 tx=0).stop 2024-05-27T07:36:38.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.543+0000 7f1803c33b80 1 -- 172.21.15.114:0/3782431493 shutdown_connections 2024-05-27T07:36:38.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.543+0000 7f1803c33b80 1 --2- 172.21.15.114:0/3782431493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf192f30 0x5631cf193340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.543+0000 7f1803c33b80 1 --2- 172.21.15.114:0/3782431493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf014df0 0x5631cf1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.543+0000 7f1803c33b80 1 --2- 172.21.15.114:0/3782431493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 0x5631cf1929f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.543+0000 7f1803c33b80 1 -- 172.21.15.114:0/3782431493 >> 172.21.15.114:0/3782431493 conn(0x5631cf06faa0 msgr2=0x5631cf18d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:38.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.543+0000 7f1803c33b80 1 -- 172.21.15.114:0/3782431493 shutdown_connections 2024-05-27T07:36:38.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.543+0000 7f1803c33b80 1 -- 172.21.15.114:0/3782431493 wait complete. 2024-05-27T07:36:38.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f1803c33b80 1 -- start start 2024-05-27T07:36:38.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f1803c33b80 1 Processor -- start 2024-05-27T07:36:38.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f1803c33b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 0x5631cf19c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f1803c33b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf192f30 0x5631cf19c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f1803c33b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf198c00 0x5631cf19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f1803c33b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5631cefb7f50 con 0x5631cf014df0 2024-05-27T07:36:38.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f1803c33b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5631cf053770 con 0x5631cf198c00 2024-05-27T07:36:38.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f1803c33b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5631cf19f630 con 0x5631cf192f30 2024-05-27T07:36:38.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f1803c33b80 10 client.0 _ll_register_callbacks cb 0x5631cf04c8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:38.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f17eed9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 0x5631cf19c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f17eed9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 0x5631cf19c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40340/0 (socket says 172.21.15.114:40340) 2024-05-27T07:36:38.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f17eed9d700 1 -- 172.21.15.114:0/4040794793 learned_addr learned my addr 172.21.15.114:0/4040794793 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:38.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f17ef59e700 1 --2- 172.21.15.114:0/4040794793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf198c00 0x5631cf19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f17e659c700 1 --2- 172.21.15.114:0/4040794793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf192f30 0x5631cf19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f17eed9d700 1 -- 172.21.15.114:0/4040794793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf192f30 msgr2=0x5631cf19c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:38.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f17eed9d700 1 --2- 172.21.15.114:0/4040794793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf192f30 0x5631cf19c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f17eed9d700 1 -- 172.21.15.114:0/4040794793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf198c00 msgr2=0x5631cf19cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:38.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f17eed9d700 1 --2- 172.21.15.114:0/4040794793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf198c00 0x5631cf19cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.544+0000 7f17eed9d700 1 -- 172.21.15.114:0/4040794793 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631cf050640 con 0x5631cf014df0 2024-05-27T07:36:38.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.545+0000 7f17eed9d700 1 --2- 172.21.15.114:0/4040794793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 0x5631cf19c1c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f17e0011770 tx=0x7f17e00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f716880ff4275436 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:38.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.545+0000 7f17d67fc700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:36:38.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.545+0000 7f17d67fc700 1 -- 172.21.15.114:0/4040794793 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e001f840 con 0x5631cf014df0 2024-05-27T07:36:38.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.545+0000 7f17d67fc700 1 -- 172.21.15.114:0/4040794793 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8802+0+0 (secure 0 0 0) 0x7f17e001f9e0 con 0x5631cf014df0 2024-05-27T07:36:38.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.545+0000 7f1803c33b80 1 -- 172.21.15.114:0/4040794793 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17d8003db0 con 0x5631cf014df0 2024-05-27T07:36:38.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.545+0000 7f17d67fc700 1 -- 172.21.15.114:0/4040794793 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f17e001fbe0 con 0x5631cf014df0 2024-05-27T07:36:38.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.545+0000 7f1803c33b80 10 client.14688 fetch_fsmap learned FSMap version 17 2024-05-27T07:36:38.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.545+0000 7f1803c33b80 1 -- 172.21.15.114:0/4040794793 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5631cf1ae4e0 con 0x5631cf014df0 2024-05-27T07:36:38.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.545+0000 7f17d67fc700 1 -- 172.21.15.114:0/4040794793 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17d8003db0 con 0x5631cf014df0 2024-05-27T07:36:38.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.546+0000 7f17d67fc700 1 -- 172.21.15.114:0/4040794793 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f17e002e420 con 0x5631cf014df0 2024-05-27T07:36:38.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.546+0000 7f1803c33b80 10 client.14688 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T07:36:38.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.546+0000 7f1803c33b80 10 client.14688 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:36:38.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.546+0000 7f1803c33b80 20 client.14688 populate_metadata read hostname 'smithi114' 2024-05-27T07:36:38.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.546+0000 7f1803c33b80 1 --2- 172.21.15.114:0/4040794793 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5631cf1b0ba0 0x5631cf1b2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.546+0000 7f1803c33b80 4 client.14688 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:38.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.546+0000 7f1803c33b80 1 -- 172.21.15.114:0/4040794793 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631cf15e9a0 con 0x5631cf1b0ba0 2024-05-27T07:36:38.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.546+0000 7f17e659c700 1 --2- 172.21.15.114:0/4040794793 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5631cf1b0ba0 0x5631cf1b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.547+0000 7f17e659c700 1 --2- 172.21.15.114:0/4040794793 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5631cf1b0ba0 0x5631cf1b2010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26c0e7b1ea53ba81 server_cookie=8c29157b89e0c383 in_seq=0 out_seq=0 2024-05-27T07:36:38.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.547+0000 7f17d67fc700 10 client.14688 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:36:38.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f17d67fc700 1 -- 172.21.15.114:0/4040794793 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5631cf15e9a0 con 0x5631cf1b0ba0 2024-05-27T07:36:38.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f17d67fc700 10 client.14688 handle_command_reply: tid=0 2024-05-27T07:36:38.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f1803c33b80 1 client.14688 shutdown 2024-05-27T07:36:38.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f1803c33b80 1 -- 172.21.15.114:0/4040794793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 msgr2=0x5631cf19c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:38.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f1803c33b80 1 --2- 172.21.15.114:0/4040794793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 0x5631cf19c1c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f17e0011770 tx=0x7f17e00047c0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f1803c33b80 1 -- 172.21.15.114:0/4040794793 shutdown_connections 2024-05-27T07:36:38.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f1803c33b80 1 --2- 172.21.15.114:0/4040794793 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5631cf1b0ba0 0x5631cf1b2010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f1803c33b80 1 --2- 172.21.15.114:0/4040794793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf198c00 0x5631cf19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f1803c33b80 1 --2- 172.21.15.114:0/4040794793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 0x5631cf19c1c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f1803c33b80 1 --2- 172.21.15.114:0/4040794793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf192f30 0x5631cf19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f1803c33b80 1 -- 172.21.15.114:0/4040794793 >> 172.21.15.114:0/4040794793 conn(0x5631cf06faa0 msgr2=0x5631cf1919e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:38.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f17d67fc700 0 client.14688 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:36:38.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f1803c33b80 1 -- 172.21.15.114:0/4040794793 shutdown_connections 2024-05-27T07:36:38.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f1803c33b80 1 -- 172.21.15.114:0/4040794793 wait complete. 2024-05-27T07:36:38.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.548+0000 7f1803c33b80 20 client.14688 trim_cache size 0 max 16384 2024-05-27T07:36:38.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.569+0000 7f1803c33b80 1 Processor -- start 2024-05-27T07:36:38.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.569+0000 7f1803c33b80 1 -- start start 2024-05-27T07:36:38.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.569+0000 7f1803c33b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 0x5631cf1929f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.569+0000 7f1803c33b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf192f30 0x5631cf1a8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.569+0000 7f1803c33b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf198c00 0x5631cf19ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.569+0000 7f1803c33b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5631cf1b0350 con 0x5631cf014df0 2024-05-27T07:36:38.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.569+0000 7f1803c33b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5631cf1b0a20 con 0x5631cf192f30 2024-05-27T07:36:38.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.569+0000 7f1803c33b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5631cf1b10f0 con 0x5631cf198c00 2024-05-27T07:36:38.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.569+0000 7f17eed9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 0x5631cf1929f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.569+0000 7f17eed9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 0x5631cf1929f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40354/0 (socket says 172.21.15.114:40354) 2024-05-27T07:36:38.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.569+0000 7f17eed9d700 1 -- 172.21.15.114:0/3052677493 learned_addr learned my addr 172.21.15.114:0/3052677493 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:38.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.570+0000 7f17ef59e700 1 --2- 172.21.15.114:0/3052677493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf198c00 0x5631cf19ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.570+0000 7f17eed9d700 1 -- 172.21.15.114:0/3052677493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf198c00 msgr2=0x5631cf19ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:38.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.570+0000 7f17e659c700 1 --2- 172.21.15.114:0/3052677493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf192f30 0x5631cf1a8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.570+0000 7f17eed9d700 1 --2- 172.21.15.114:0/3052677493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf198c00 0x5631cf19ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.570+0000 7f17eed9d700 1 -- 172.21.15.114:0/3052677493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf192f30 msgr2=0x5631cf1a8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:38.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.570+0000 7f17eed9d700 1 --2- 172.21.15.114:0/3052677493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf192f30 0x5631cf1a8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.570+0000 7f17eed9d700 1 -- 172.21.15.114:0/3052677493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631cf18ee80 con 0x5631cf014df0 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.570+0000 7f17ef59e700 1 --2- 172.21.15.114:0/3052677493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf198c00 0x5631cf19ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.570+0000 7f17e659c700 1 --2- 172.21.15.114:0/3052677493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf192f30 0x5631cf1a8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.570+0000 7f17eed9d700 1 --2- 172.21.15.114:0/3052677493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 0x5631cf1929f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f17e002e8c0 tx=0x7f17e0011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba17c8ee76b4ec84 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.570+0000 7f17e4d99700 1 -- 172.21.15.114:0/3052677493 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e002eab0 con 0x5631cf014df0 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.570+0000 7f17e4d99700 1 -- 172.21.15.114:0/3052677493 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e0023070 con 0x5631cf014df0 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.570+0000 7f17e4d99700 1 -- 172.21.15.114:0/3052677493 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0028940 con 0x5631cf014df0 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.571+0000 7f1803c33b80 1 -- 172.21.15.114:0/3052677493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 msgr2=0x5631cf1929f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.571+0000 7f1803c33b80 1 --2- 172.21.15.114:0/3052677493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 0x5631cf1929f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f17e002e8c0 tx=0x7f17e0011ec0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.571+0000 7f1803c33b80 1 -- 172.21.15.114:0/3052677493 shutdown_connections 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.571+0000 7f1803c33b80 1 --2- 172.21.15.114:0/3052677493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf198c00 0x5631cf19ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.571+0000 7f1803c33b80 1 --2- 172.21.15.114:0/3052677493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf014df0 0x5631cf1929f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.571+0000 7f1803c33b80 1 --2- 172.21.15.114:0/3052677493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf192f30 0x5631cf1a8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.571+0000 7f1803c33b80 1 -- 172.21.15.114:0/3052677493 >> 172.21.15.114:0/3052677493 conn(0x5631cf06faa0 msgr2=0x5631cf18d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.571+0000 7f1803c33b80 1 -- 172.21.15.114:0/3052677493 shutdown_connections 2024-05-27T07:36:38.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.571+0000 7f1803c33b80 1 -- 172.21.15.114:0/3052677493 wait complete. 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.571+0000 7f1803c33b80 1 -- start start 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f1803c33b80 1 Processor -- start 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f1803c33b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf014df0 0x5631cf1bb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f1803c33b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf192f30 0x5631cf1bbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f1803c33b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 0x5631cf191af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f1803c33b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5631cf1b1e90 con 0x5631cf198c00 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f1803c33b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5631cf1b17c0 con 0x5631cf014df0 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f1803c33b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5631cf1b2560 con 0x5631cf192f30 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f1803c33b80 10 client.0 _ll_register_callbacks cb 0x5631cf04c8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f17ef59e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 0x5631cf191af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f17ef59e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 0x5631cf191af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40356/0 (socket says 172.21.15.114:40356) 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f17ef59e700 1 -- 172.21.15.114:0/787168808 learned_addr learned my addr 172.21.15.114:0/787168808 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f17eed9d700 1 --2- 172.21.15.114:0/787168808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf014df0 0x5631cf1bb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f17e659c700 1 --2- 172.21.15.114:0/787168808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf192f30 0x5631cf1bbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f17ef59e700 1 -- 172.21.15.114:0/787168808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf192f30 msgr2=0x5631cf1bbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f17ef59e700 1 --2- 172.21.15.114:0/787168808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf192f30 0x5631cf1bbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f17ef59e700 1 -- 172.21.15.114:0/787168808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf014df0 msgr2=0x5631cf1bb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f17ef59e700 1 --2- 172.21.15.114:0/787168808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf014df0 0x5631cf1bb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f17ef59e700 1 -- 172.21.15.114:0/787168808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631cf192270 con 0x5631cf198c00 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f17ef59e700 1 --2- 172.21.15.114:0/787168808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 0x5631cf191af0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f17e80df380 tx=0x7f17e800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce8bd14540b881a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.572+0000 7f17d67fc700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f17d67fc700 1 -- 172.21.15.114:0/787168808 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e80a8f30 con 0x5631cf198c00 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f17d67fc700 1 -- 172.21.15.114:0/787168808 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8802+0+0 (secure 0 0 0) 0x7f17e80a93a0 con 0x5631cf198c00 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f1803c33b80 1 -- 172.21.15.114:0/787168808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17e0023070 con 0x5631cf198c00 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f17d67fc700 1 -- 172.21.15.114:0/787168808 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f17e80ad2e0 con 0x5631cf198c00 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f1803c33b80 10 client.14694 fetch_fsmap learned FSMap version 17 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f1803c33b80 1 -- 172.21.15.114:0/787168808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5631cf1baf30 con 0x5631cf198c00 2024-05-27T07:36:38.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f17d67fc700 1 -- 172.21.15.114:0/787168808 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e0023070 con 0x5631cf198c00 2024-05-27T07:36:38.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f17d67fc700 1 -- 172.21.15.114:0/787168808 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f17e80bf070 con 0x5631cf198c00 2024-05-27T07:36:38.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f1803c33b80 10 client.14694 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T07:36:38.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f1803c33b80 10 client.14694 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:36:38.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f1803c33b80 20 client.14694 populate_metadata read hostname 'smithi114' 2024-05-27T07:36:38.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f1803c33b80 1 --2- 172.21.15.114:0/787168808 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5631cf21e1c0 0x5631cf21f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:38.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f1803c33b80 4 client.14694 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:36:38.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f1803c33b80 1 -- 172.21.15.114:0/787168808 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "status"}) -- 0x5631cf0c3b90 con 0x5631cf21e1c0 2024-05-27T07:36:38.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.573+0000 7f17eed9d700 1 --2- 172.21.15.114:0/787168808 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5631cf21e1c0 0x5631cf21f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:38.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f17eed9d700 1 --2- 172.21.15.114:0/787168808 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5631cf21e1c0 0x5631cf21f630 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=821a9999384dcdd server_cookie=af35f78fc34b008a in_seq=0 out_seq=0 2024-05-27T07:36:38.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f17d67fc700 10 client.14694 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f17d67fc700 1 -- 172.21.15.114:0/787168808 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5631cf0c3b90 con 0x5631cf21e1c0 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f17d67fc700 10 client.14694 handle_command_reply: tid=0 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f1803c33b80 1 client.14694 shutdown 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f1803c33b80 1 -- 172.21.15.114:0/787168808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 msgr2=0x5631cf191af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f1803c33b80 1 --2- 172.21.15.114:0/787168808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 0x5631cf191af0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f17e80df380 tx=0x7f17e800cc30 comp rx=0 tx=0).stop 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f1803c33b80 1 -- 172.21.15.114:0/787168808 shutdown_connections 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f1803c33b80 1 --2- 172.21.15.114:0/787168808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631cf198c00 0x5631cf191af0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f1803c33b80 1 --2- 172.21.15.114:0/787168808 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5631cf21e1c0 0x5631cf21f630 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f1803c33b80 1 --2- 172.21.15.114:0/787168808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5631cf014df0 0x5631cf1bb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f17d67fc700 0 client.14694 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f1803c33b80 1 --2- 172.21.15.114:0/787168808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631cf192f30 0x5631cf1bbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f1803c33b80 1 -- 172.21.15.114:0/787168808 >> 172.21.15.114:0/787168808 conn(0x5631cf06faa0 msgr2=0x5631cf19b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f1803c33b80 1 -- 172.21.15.114:0/787168808 shutdown_connections 2024-05-27T07:36:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f1803c33b80 1 -- 172.21.15.114:0/787168808 wait complete. 2024-05-27T07:36:38.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.574+0000 7f1803c33b80 20 client.14694 trim_cache size 0 max 16384 2024-05-27T07:36:38.580 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:36:38.580 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "755b3728-1bfa-11ef-bc9b-c7b262605968", 2024-05-27T07:36:38.580 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-05-27T07:36:38.580 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24463, 2024-05-27T07:36:38.581 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-05-27T07:36:38.581 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-05-27T07:36:38.581 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-05-27T07:36:38.581 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 31.600087659, 2024-05-27T07:36:38.581 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 17, 2024-05-27T07:36:38.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 85, 2024-05-27T07:36:38.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 84, 2024-05-27T07:36:38.582 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 56.537305713999999 2024-05-27T07:36:38.582 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:36:38.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.575+0000 7f17f521e700 1 -- 172.21.15.114:0/3005437570 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f17dc0b2880 msgr2=0x7f17dc0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:38.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.575+0000 7f17f521e700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f17dc0b2880 0x7f17dc0b4d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f17e0005950 tx=0x7f17e000b410 comp rx=0 tx=0).stop 2024-05-27T07:36:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.575+0000 7f17f521e700 1 -- 172.21.15.114:0/3005437570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0108730 msgr2=0x7f17f019f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.575+0000 7f17f521e700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0108730 0x7f17f019f6a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f17e800ada0 tx=0x7f17e800c330 comp rx=0 tx=0).stop 2024-05-27T07:36:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.575+0000 7f17f521e700 1 -- 172.21.15.114:0/3005437570 shutdown_connections 2024-05-27T07:36:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.575+0000 7f17f521e700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0108730 0x7f17f019f6a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.575+0000 7f17f521e700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f17dc0b2880 0x7f17dc0b4d90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.575+0000 7f17f521e700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17f01034b0 0x7f17f019ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.575+0000 7f17f521e700 1 --2- 172.21.15.114:0/3005437570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17f0107de0 0x7f17f019b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:38.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.575+0000 7f17f521e700 1 -- 172.21.15.114:0/3005437570 >> 172.21.15.114:0/3005437570 conn(0x7f17f00fede0 msgr2=0x7f17f0075cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:38.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.575+0000 7f17f521e700 1 -- 172.21.15.114:0/3005437570 shutdown_connections 2024-05-27T07:36:38.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:38.575+0000 7f17f521e700 1 -- 172.21.15.114:0/3005437570 wait complete. 2024-05-27T07:36:39.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:39 smithi114 ceph-mon[32038]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:39.257 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-27T07:36:39.257 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:36:39.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-05-27T07:36:39.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:39 smithi163 ceph-mon[36451]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:39.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:39 smithi186 ceph-mon[36647]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:39.497 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:41.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.194+0000 7feb0a5d5700 1 -- 172.21.15.114:0/3281988159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb04108730 msgr2=0x7feb04110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:41.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.194+0000 7feb0a5d5700 1 --2- 172.21.15.114:0/3281988159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb04108730 0x7feb04110090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7feafc00b0d0 tx=0x7feafc00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:36:41.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.194+0000 7feb0a5d5700 1 -- 172.21.15.114:0/3281988159 shutdown_connections 2024-05-27T07:36:41.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.194+0000 7feb0a5d5700 1 --2- 172.21.15.114:0/3281988159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb04108730 0x7feb04110090 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:41.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.194+0000 7feb0a5d5700 1 --2- 172.21.15.114:0/3281988159 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb041034b0 0x7feb041038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:41.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.194+0000 7feb0a5d5700 1 --2- 172.21.15.114:0/3281988159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb04107de0 0x7feb041081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:41.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.194+0000 7feb0a5d5700 1 -- 172.21.15.114:0/3281988159 >> 172.21.15.114:0/3281988159 conn(0x7feb040fede0 msgr2=0x7feb04101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:41.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.194+0000 7feb0a5d5700 1 -- 172.21.15.114:0/3281988159 shutdown_connections 2024-05-27T07:36:41.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.194+0000 7feb0a5d5700 1 -- 172.21.15.114:0/3281988159 wait complete. 2024-05-27T07:36:41.191 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:41 smithi114 ceph-mon[32038]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.195+0000 7feb0a5d5700 1 Processor -- start 2024-05-27T07:36:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.195+0000 7feb0a5d5700 1 -- start start 2024-05-27T07:36:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.195+0000 7feb0a5d5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb041034b0 0x7feb0410a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.195+0000 7feb0a5d5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb04107de0 0x7feb0410a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.195+0000 7feb0a5d5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb04108730 0x7feb0410ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.195+0000 7feb0a5d5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feb04113360 con 0x7feb041034b0 2024-05-27T07:36:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.195+0000 7feb0a5d5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7feb041131e0 con 0x7feb04108730 2024-05-27T07:36:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.195+0000 7feb0a5d5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feb041134e0 con 0x7feb04107de0 2024-05-27T07:36:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.195+0000 7feb03fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb041034b0 0x7feb0410a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.195+0000 7feb03fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb041034b0 0x7feb0410a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40374/0 (socket says 172.21.15.114:40374) 2024-05-27T07:36:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.195+0000 7feb03fff700 1 -- 172.21.15.114:0/2168572793 learned_addr learned my addr 172.21.15.114:0/2168572793 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb037fe700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb04107de0 0x7feb0410a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb08b72700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb04108730 0x7feb0410ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb03fff700 1 -- 172.21.15.114:0/2168572793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb04107de0 msgr2=0x7feb0410a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb03fff700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb04107de0 0x7feb0410a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb03fff700 1 -- 172.21.15.114:0/2168572793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb04108730 msgr2=0x7feb0410ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb03fff700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb04108730 0x7feb0410ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb03fff700 1 -- 172.21.15.114:0/2168572793 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb041ad540 con 0x7feb041034b0 2024-05-27T07:36:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb08b72700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb04108730 0x7feb0410ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:41.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb03fff700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb041034b0 0x7feb0410a110 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feaf400d950 tx=0x7feaf400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:41.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb017fa700 1 -- 172.21.15.114:0/2168572793 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf4016070 con 0x7feb041034b0 2024-05-27T07:36:41.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb0a5d5700 1 -- 172.21.15.114:0/2168572793 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb041ad7d0 con 0x7feb041034b0 2024-05-27T07:36:41.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb017fa700 1 -- 172.21.15.114:0/2168572793 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaf4009df0 con 0x7feb041034b0 2024-05-27T07:36:41.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb017fa700 1 -- 172.21.15.114:0/2168572793 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf400f3c0 con 0x7feb041034b0 2024-05-27T07:36:41.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.196+0000 7feb0a5d5700 1 -- 172.21.15.114:0/2168572793 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb041add60 con 0x7feb041034b0 2024-05-27T07:36:41.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.198+0000 7feb0a5d5700 1 -- 172.21.15.114:0/2168572793 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb04104960 con 0x7feb041034b0 2024-05-27T07:36:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.199+0000 7feb017fa700 1 -- 172.21.15.114:0/2168572793 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7feaf4019070 con 0x7feb041034b0 2024-05-27T07:36:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.200+0000 7feb017fa700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7feaec0b2880 0x7feaec0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.200+0000 7feb037fe700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7feaec0b2880 0x7feaec0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.200+0000 7feb017fa700 1 -- 172.21.15.114:0/2168572793 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8802+0+0 (secure 0 0 0) 0x7feaf40e2070 con 0x7feb041034b0 2024-05-27T07:36:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.200+0000 7feb037fe700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7feaec0b2880 0x7feaec0b4d90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feb04109740 tx=0x7feaf800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.205+0000 7feb017fa700 1 -- 172.21.15.114:0/2168572793 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7feaf40a9b50 con 0x7feb041034b0 2024-05-27T07:36:41.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:41 smithi163 ceph-mon[36451]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:41.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:41 smithi186 ceph-mon[36647]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:41.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.479+0000 7feb0a5d5700 1 -- 172.21.15.114:0/2168572793 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7feb0402e5e0 con 0x7feb041034b0 2024-05-27T07:36:41.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.480+0000 7feb017fa700 1 -- 172.21.15.114:0/2168572793 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7feaf40a9450 con 0x7feb041034b0 2024-05-27T07:36:41.476 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-27T07:36:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.482+0000 7feb0a5d5700 1 -- 172.21.15.114:0/2168572793 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7feaec0b2880 msgr2=0x7feaec0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.482+0000 7feb0a5d5700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7feaec0b2880 0x7feaec0b4d90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feb04109740 tx=0x7feaf800b410 comp rx=0 tx=0).stop 2024-05-27T07:36:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.482+0000 7feb0a5d5700 1 -- 172.21.15.114:0/2168572793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb041034b0 msgr2=0x7feb0410a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.482+0000 7feb0a5d5700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb041034b0 0x7feb0410a110 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feaf400d950 tx=0x7feaf400dc60 comp rx=0 tx=0).stop 2024-05-27T07:36:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.482+0000 7feb0a5d5700 1 -- 172.21.15.114:0/2168572793 shutdown_connections 2024-05-27T07:36:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.482+0000 7feb0a5d5700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb04108730 0x7feb0410ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.482+0000 7feb0a5d5700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb041034b0 0x7feb0410a110 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.482+0000 7feb0a5d5700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7feaec0b2880 0x7feaec0b4d90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.482+0000 7feb0a5d5700 1 --2- 172.21.15.114:0/2168572793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb04107de0 0x7feb0410a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.482+0000 7feb0a5d5700 1 -- 172.21.15.114:0/2168572793 >> 172.21.15.114:0/2168572793 conn(0x7feb040fede0 msgr2=0x7feb0410ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.482+0000 7feb0a5d5700 1 -- 172.21.15.114:0/2168572793 shutdown_connections 2024-05-27T07:36:41.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:41.482+0000 7feb0a5d5700 1 -- 172.21.15.114:0/2168572793 wait complete. 2024-05-27T07:36:42.126 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-27T07:36:42.126 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-27T07:36:42.126 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-05-27T07:36:42.126 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-05-27T07:36:42.126 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:36:42.126 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-27T07:36:42.126 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-27T07:36:42.126 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-27T07:36:42.126 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-05-27T07:36:42.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:36:42.127 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-05-27T07:36:42.143 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-27T07:36:42.143 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-27T07:36:42.143 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-05-27T07:36:42.143 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:36:42.143 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-05-27T07:36:42.143 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:36:42.143 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:36:42.143 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:36:42.143 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-05-27T07:36:42.143 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:36:42.143 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:36:42.143 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-05-27T07:36:42.144 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:36:42.144 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:36:42.144 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-27T07:36:42.144 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:36:42.144 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-27T07:36:42.144 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42172sec preferred_lft 42172sec 2024-05-27T07:36:42.144 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link 2024-05-27T07:36:42.144 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:36:42.145 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-27T07:36:42.145 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:36:42.145 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-27T07:36:42.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add name ceph-brx type bridge 2024-05-27T07:36:42.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr flush dev ceph-brx 2024-05-27T07:36:42.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set ceph-brx up 2024-05-27T07:36:42.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-27T07:36:42.145 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-27T07:36:42.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:36:42.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:36:42.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:36:42.339 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-27T07:36:42.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:42 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2168572793' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:36:42.364 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-05-27T07:36:42.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:36:42.366 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-05-27T07:36:42.428 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-05-27T07:36:42.429 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-27T07:36:42.429 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-27T07:36:42.429 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-27T07:36:42.429 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-27T07:36:42.429 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:36:42.430 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-27T07:36:42.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-27T07:36:42.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-27T07:36:42.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-27T07:36:42.430 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-27T07:36:42.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:42 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2168572793' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:36:42.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:42 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2168572793' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:36:42.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:36:42.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:36:42.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:36:42.722 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-27T07:36:42.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:36:42.738 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2024-05-27T07:36:42.793 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:36:42.793 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-27T07:36:42.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:36:42.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-05-27T07:36:42.794 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-27T07:36:42.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:36:42.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:36:42.912 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-05-27T07:36:42.912 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:36:42.912 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-27T07:36:42.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-05-27T07:36:42.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-27T07:36:42.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-05-27T07:36:42.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-05-27T07:36:42.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-05-27T07:36:42.912 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-27T07:36:42.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:36:43.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:36:43.039 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:36:43.039 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-27T07:36:43.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-05-27T07:36:43.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.0 master ceph-brx 2024-05-27T07:36:43.039 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-27T07:36:43.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:36:43.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:36:43.117 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-05-27T07:36:43.117 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:36:43.117 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:36:43.133 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-05-27T07:36:43.409 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:36:43.410 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:36:43.411 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:43 smithi114 ceph-mon[32038]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:43.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2024-05-27T07:36:43.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:43 smithi163 ceph-mon[36451]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:43.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:43 smithi186 ceph-mon[36647]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:43.497 DEBUG:teuthology.orchestra.run.smithi114:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:36:43.556 INFO:teuthology.orchestra.run.smithi114.stdout:/proc 2024-05-27T07:36:43.556 INFO:teuthology.orchestra.run.smithi114.stdout:/sys 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/dev 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/security 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/shm 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/pts 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/run 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/cgroup 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/pstore 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/bpf 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/ 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/selinux 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/hugepages 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/mqueue 2024-05-27T07:36:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/debug 2024-05-27T07:36:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/tracing 2024-05-27T07:36:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/fuse/connections 2024-05-27T07:36:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/config 2024-05-27T07:36:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:36:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:36:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:36:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:36:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:/run/user/1000 2024-05-27T07:36:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/ceph 2024-05-27T07:36:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:36:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/94a76e44ce53f523d4fe8fad40ad2ac27ed4c2cc608ce5fe5b3a2995dc8f8432/merged 2024-05-27T07:36:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/ee480c929f8bd9822f52aae45df3aa587b7cd5ed42aef098965da117c7a916c9/merged 2024-05-27T07:36:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:/run/user/0 2024-05-27T07:36:43.559 INFO:teuthology.orchestra.run.smithi114.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:36:43.559 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/673d5a0837d86807d06484440d3afe658f01c7941f261b9775444d243dfdd2e9/merged 2024-05-27T07:36:43.559 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/d927e3fc95c4122297cd265b8a5a04fb0962353c0313eeff6ec02547a8b2e34e/merged 2024-05-27T07:36:43.559 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/fa72f966b5c746878ccfc28b3538a12da909f4674f6b0a1016c10bfed8bbb628/merged 2024-05-27T07:36:43.559 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/536e7f1f97386e3c5090df963fb0ae0fd0348dd0eadc2c2fcaa2e23cb4e5b53a/merged 2024-05-27T07:36:43.559 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/956275fd9eef98faf21ac8e53dd7b69d02e7d64518f82a276aef72e3fc9ee902/merged 2024-05-27T07:36:43.559 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/27451ed229541f491b56d1f35c098a994e4aa2b24204623184bf15c979a538ab/merged 2024-05-27T07:36:43.559 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/f51d641fe8561718c6b044dde30c45e25d93f33a102294293907ca0f3ef7647f/merged 2024-05-27T07:36:43.559 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/f23b2e860701151648ad10c9afddde5fbf2bfffcbf0618b35649372d85107109/merged 2024-05-27T07:36:43.559 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns 2024-05-27T07:36:43.559 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:36:43.559 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:36:43.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:36:43.560 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2024-05-27T07:36:43.617 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-27T07:36:43.617 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-05-27T07:36:43.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2024-05-27T07:36:43.689 DEBUG:teuthology.orchestra.run.smithi114:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:36:43.748 INFO:teuthology.orchestra.run.smithi114.stdout:/proc 2024-05-27T07:36:43.748 INFO:teuthology.orchestra.run.smithi114.stdout:/sys 2024-05-27T07:36:43.748 INFO:teuthology.orchestra.run.smithi114.stdout:/dev 2024-05-27T07:36:43.748 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/security 2024-05-27T07:36:43.748 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/shm 2024-05-27T07:36:43.749 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/pts 2024-05-27T07:36:43.749 INFO:teuthology.orchestra.run.smithi114.stdout:/run 2024-05-27T07:36:43.749 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/cgroup 2024-05-27T07:36:43.749 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/pstore 2024-05-27T07:36:43.749 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/bpf 2024-05-27T07:36:43.749 INFO:teuthology.orchestra.run.smithi114.stdout:/ 2024-05-27T07:36:43.749 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/selinux 2024-05-27T07:36:43.749 INFO:teuthology.orchestra.run.smithi114.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:36:43.749 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/hugepages 2024-05-27T07:36:43.749 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/mqueue 2024-05-27T07:36:43.749 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/debug 2024-05-27T07:36:43.749 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/tracing 2024-05-27T07:36:43.749 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/fuse/connections 2024-05-27T07:36:43.750 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/config 2024-05-27T07:36:43.750 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:36:43.750 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:36:43.750 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:36:43.750 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:36:43.750 INFO:teuthology.orchestra.run.smithi114.stdout:/run/user/1000 2024-05-27T07:36:43.750 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/ceph 2024-05-27T07:36:43.751 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:36:43.751 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/94a76e44ce53f523d4fe8fad40ad2ac27ed4c2cc608ce5fe5b3a2995dc8f8432/merged 2024-05-27T07:36:43.751 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/ee480c929f8bd9822f52aae45df3aa587b7cd5ed42aef098965da117c7a916c9/merged 2024-05-27T07:36:43.751 INFO:teuthology.orchestra.run.smithi114.stdout:/run/user/0 2024-05-27T07:36:43.751 INFO:teuthology.orchestra.run.smithi114.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:36:43.751 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/673d5a0837d86807d06484440d3afe658f01c7941f261b9775444d243dfdd2e9/merged 2024-05-27T07:36:43.751 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/d927e3fc95c4122297cd265b8a5a04fb0962353c0313eeff6ec02547a8b2e34e/merged 2024-05-27T07:36:43.751 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/fa72f966b5c746878ccfc28b3538a12da909f4674f6b0a1016c10bfed8bbb628/merged 2024-05-27T07:36:43.751 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/536e7f1f97386e3c5090df963fb0ae0fd0348dd0eadc2c2fcaa2e23cb4e5b53a/merged 2024-05-27T07:36:43.751 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/956275fd9eef98faf21ac8e53dd7b69d02e7d64518f82a276aef72e3fc9ee902/merged 2024-05-27T07:36:43.752 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/27451ed229541f491b56d1f35c098a994e4aa2b24204623184bf15c979a538ab/merged 2024-05-27T07:36:43.752 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/f51d641fe8561718c6b044dde30c45e25d93f33a102294293907ca0f3ef7647f/merged 2024-05-27T07:36:43.752 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/f23b2e860701151648ad10c9afddde5fbf2bfffcbf0618b35649372d85107109/merged 2024-05-27T07:36:43.752 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns 2024-05-27T07:36:43.752 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:36:43.752 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:36:43.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:36:43.753 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2024-05-27T07:36:43.818 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi114.stderr:2024-05-27T07:36:43.822+0000 7eff10b27480 -1 init, newargv = 0x7eff040045e0 newargc=15 2024-05-27T07:36:43.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi114.stderr:ceph-fuse[75276]: starting ceph client 2024-05-27T07:36:43.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi114.stderr:ceph-fuse[75276]: starting fuse 2024-05-27T07:36:44.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-05-27T07:36:44.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-05-27T07:36:44.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-05-27T07:36:44.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-27T07:36:44.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-05-27T07:36:44.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]: dispatch 2024-05-27T07:36:44.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]: dispatch 2024-05-27T07:36:44.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-05-27T07:36:44.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-05-27T07:36:44.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:44 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-27T07:36:44.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-05-27T07:36:44.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-05-27T07:36:44.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-05-27T07:36:44.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-27T07:36:44.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-05-27T07:36:44.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]: dispatch 2024-05-27T07:36:44.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]: dispatch 2024-05-27T07:36:44.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-05-27T07:36:44.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-05-27T07:36:44.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:44 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-27T07:36:44.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-05-27T07:36:44.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-05-27T07:36:44.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-05-27T07:36:44.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-27T07:36:44.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-05-27T07:36:44.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]: dispatch 2024-05-27T07:36:44.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]: dispatch 2024-05-27T07:36:44.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-05-27T07:36:44.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-05-27T07:36:44.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:44 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-27T07:36:44.908 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2024-05-27T07:36:44.941 DEBUG:teuthology.orchestra.run.smithi114:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:36:44.999 INFO:teuthology.orchestra.run.smithi114.stdout:/proc 2024-05-27T07:36:44.999 INFO:teuthology.orchestra.run.smithi114.stdout:/sys 2024-05-27T07:36:44.999 INFO:teuthology.orchestra.run.smithi114.stdout:/dev 2024-05-27T07:36:44.999 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/security 2024-05-27T07:36:44.999 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/shm 2024-05-27T07:36:44.999 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/pts 2024-05-27T07:36:44.999 INFO:teuthology.orchestra.run.smithi114.stdout:/run 2024-05-27T07:36:44.999 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/cgroup 2024-05-27T07:36:44.999 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/pstore 2024-05-27T07:36:44.999 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/bpf 2024-05-27T07:36:44.999 INFO:teuthology.orchestra.run.smithi114.stdout:/ 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/selinux 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/hugepages 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/mqueue 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/debug 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/tracing 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/fuse/connections 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/config 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/run/user/1000 2024-05-27T07:36:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/ceph 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/94a76e44ce53f523d4fe8fad40ad2ac27ed4c2cc608ce5fe5b3a2995dc8f8432/merged 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/ee480c929f8bd9822f52aae45df3aa587b7cd5ed42aef098965da117c7a916c9/merged 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/run/user/0 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/673d5a0837d86807d06484440d3afe658f01c7941f261b9775444d243dfdd2e9/merged 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/d927e3fc95c4122297cd265b8a5a04fb0962353c0313eeff6ec02547a8b2e34e/merged 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/fa72f966b5c746878ccfc28b3538a12da909f4674f6b0a1016c10bfed8bbb628/merged 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/536e7f1f97386e3c5090df963fb0ae0fd0348dd0eadc2c2fcaa2e23cb4e5b53a/merged 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/956275fd9eef98faf21ac8e53dd7b69d02e7d64518f82a276aef72e3fc9ee902/merged 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/27451ed229541f491b56d1f35c098a994e4aa2b24204623184bf15c979a538ab/merged 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/f51d641fe8561718c6b044dde30c45e25d93f33a102294293907ca0f3ef7647f/merged 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/f23b2e860701151648ad10c9afddde5fbf2bfffcbf0618b35649372d85107109/merged 2024-05-27T07:36:45.001 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns 2024-05-27T07:36:45.002 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:36:45.002 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:36:45.002 INFO:teuthology.orchestra.run.smithi114.stdout:/home/ubuntu/cephtest/mnt.admin 2024-05-27T07:36:45.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:36:45.002 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2024-05-27T07:36:45.056 INFO:teuthology.orchestra.run.smithi114.stdout:119 2024-05-27T07:36:45.056 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-05-27T07:36:45.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo stdin-killer -- python3 -c ' 2024-05-27T07:36:45.056 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2024-05-27T07:36:45.056 DEBUG:teuthology.orchestra.run.smithi114:> import re 2024-05-27T07:36:45.056 DEBUG:teuthology.orchestra.run.smithi114:> import os 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-27T07:36:45.057 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-27T07:36:45.058 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2024-05-27T07:36:45.058 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2024-05-27T07:36:45.058 DEBUG:teuthology.orchestra.run.smithi114:> return f 2024-05-27T07:36:45.058 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-27T07:36:45.058 DEBUG:teuthology.orchestra.run.smithi114:> 2024-05-27T07:36:45.058 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.admin")) 2024-05-27T07:36:45.058 DEBUG:teuthology.orchestra.run.smithi114:> ' 2024-05-27T07:36:45.183 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.admin.75276.asok 2024-05-27T07:36:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:36:45.192 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:45 smithi114 ceph-mon[32038]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:45.193 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-05-27T07:36:45.193 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-05-27T07:36:45.193 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-05-27T07:36:45.193 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-27T07:36:45.193 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-05-27T07:36:45.193 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]': finished 2024-05-27T07:36:45.193 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]': finished 2024-05-27T07:36:45.193 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-05-27T07:36:45.193 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-05-27T07:36:45.194 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-27T07:36:45.194 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:45 smithi114 ceph-mon[32038]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T07:36:45.195 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75276.asok 2024-05-27T07:36:45.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:36:45.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75276.asok status 2024-05-27T07:36:45.338 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:36:45.339 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2024-05-27T07:36:45.339 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "38f9d900c83c63799fbdbe61acc9a11b0d3554a6", 2024-05-27T07:36:45.339 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:36:45.339 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "admin", 2024-05-27T07:36:45.339 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2024-05-27T07:36:45.339 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-05-27T07:36:45.339 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "75276", 2024-05-27T07:36:45.339 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2024-05-27T07:36:45.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T07:36:45.339 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2024-05-27T07:36:45.339 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2024-05-27T07:36:45.339 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 14706, 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 14706 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 564453723 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2024-05-27T07:36:45.340 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 564453723 2024-05-27T07:36:45.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T07:36:45.341 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.14706 192.168.0.1:0/564453723", 2024-05-27T07:36:45.341 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/564453723", 2024-05-27T07:36:45.341 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2024-05-27T07:36:45.341 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 17, 2024-05-27T07:36:45.341 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 85, 2024-05-27T07:36:45.341 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2024-05-27T07:36:45.341 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2024-05-27T07:36:45.341 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2024-05-27T07:36:45.341 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:36:45.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:36:45.348 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2024-05-27T07:36:45.348 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:36:45.363 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2024-05-27T07:36:45.364 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:36:45.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:36:45.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:36:45.430 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-05-27T07:36:45.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:45 smithi163 ceph-mon[36451]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-05-27T07:36:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-05-27T07:36:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-05-27T07:36:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-27T07:36:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-05-27T07:36:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]': finished 2024-05-27T07:36:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]': finished 2024-05-27T07:36:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-05-27T07:36:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-05-27T07:36:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-27T07:36:45.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:45 smithi163 ceph-mon[36451]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T07:36:45.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:45 smithi186 ceph-mon[36647]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:45.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-05-27T07:36:45.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-05-27T07:36:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-05-27T07:36:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-27T07:36:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-05-27T07:36:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]': finished 2024-05-27T07:36:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]': finished 2024-05-27T07:36:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-05-27T07:36:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-05-27T07:36:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-27T07:36:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:45 smithi186 ceph-mon[36647]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T07:36:45.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T07:36:45.544 INFO:teuthology.orchestra.run.smithi114.stdout:# file: . 2024-05-27T07:36:45.545 INFO:teuthology.orchestra.run.smithi114.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-05-27T07:36:45.545 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:36:45.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:36:45.550 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:36:45.550 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:36:45.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T07:36:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:46.229 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:46 smithi114 ceph-mon[32038]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T07:36:46.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:46 smithi163 ceph-mon[36451]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T07:36:46.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:46 smithi186 ceph-mon[36647]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T07:36:47.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:47 smithi114 ceph-mon[32038]: pgmap v318: 129 pgs: 6 peering, 123 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:47.475 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:47 smithi163 ceph-mon[36451]: pgmap v318: 129 pgs: 6 peering, 123 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:47.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:47 smithi186 ceph-mon[36647]: pgmap v318: 129 pgs: 6 peering, 123 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:47.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.481+0000 7f565a812700 1 -- 172.21.15.114:0/3691290575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5654103300 msgr2=0x7f565410f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:47.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.481+0000 7f565a812700 1 --2- 172.21.15.114:0/3691290575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5654103300 0x7f565410f880 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f564c00b0a0 tx=0x7f564c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.481+0000 7f565a812700 1 -- 172.21.15.114:0/3691290575 shutdown_connections 2024-05-27T07:36:47.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.481+0000 7f565a812700 1 --2- 172.21.15.114:0/3691290575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5654103300 0x7f565410f880 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.481+0000 7f565a812700 1 --2- 172.21.15.114:0/3691290575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56541089e0 0x7f5654108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.481+0000 7f565a812700 1 --2- 172.21.15.114:0/3691290575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f56541029b0 0x7f5654102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.481+0000 7f565a812700 1 -- 172.21.15.114:0/3691290575 >> 172.21.15.114:0/3691290575 conn(0x7f56540fe510 msgr2=0x7f5654100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:47.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.481+0000 7f565a812700 1 -- 172.21.15.114:0/3691290575 shutdown_connections 2024-05-27T07:36:47.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.481+0000 7f565a812700 1 -- 172.21.15.114:0/3691290575 wait complete. 2024-05-27T07:36:47.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.482+0000 7f565a812700 1 Processor -- start 2024-05-27T07:36:47.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.482+0000 7f565a812700 1 -- start start 2024-05-27T07:36:47.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.482+0000 7f565a812700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f56541029b0 0x7f565419ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:47.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.482+0000 7f565a812700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5654103300 0x7f565419b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.482+0000 7f565a812700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56541089e0 0x7f565419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f565a812700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f56541133a0 con 0x7f5654103300 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f565a812700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5654113220 con 0x7f56541029b0 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f565a812700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5654113520 con 0x7f56541089e0 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f564bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5654103300 0x7f565419b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f564bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5654103300 0x7f565419b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46206/0 (socket says 172.21.15.114:46206) 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f564bfff700 1 -- 172.21.15.114:0/3444675098 learned_addr learned my addr 172.21.15.114:0/3444675098 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f5658daf700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56541089e0 0x7f565419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f5653fff700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f56541029b0 0x7f565419ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f564bfff700 1 -- 172.21.15.114:0/3444675098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56541089e0 msgr2=0x7f565419f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f564bfff700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56541089e0 0x7f565419f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f564bfff700 1 -- 172.21.15.114:0/3444675098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f56541029b0 msgr2=0x7f565419ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f564bfff700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f56541029b0 0x7f565419ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f564bfff700 1 -- 172.21.15.114:0/3444675098 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f565419fd10 con 0x7f5654103300 2024-05-27T07:36:47.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f5653fff700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f56541029b0 0x7f565419ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:47.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f5658daf700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56541089e0 0x7f565419f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:47.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f564bfff700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5654103300 0x7f565419b3e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f563c00c8f0 tx=0x7f563c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:47.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f5651ffb700 1 -- 172.21.15.114:0/3444675098 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c007b00 con 0x7f5654103300 2024-05-27T07:36:47.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f5651ffb700 1 -- 172.21.15.114:0/3444675098 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f563c007ca0 con 0x7f5654103300 2024-05-27T07:36:47.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f565a812700 1 -- 172.21.15.114:0/3444675098 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56541a0000 con 0x7f5654103300 2024-05-27T07:36:47.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f5651ffb700 1 -- 172.21.15.114:0/3444675098 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c018660 con 0x7f5654103300 2024-05-27T07:36:47.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.483+0000 7f565a812700 1 -- 172.21.15.114:0/3444675098 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56541a7920 con 0x7f5654103300 2024-05-27T07:36:47.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.484+0000 7f565a812700 1 -- 172.21.15.114:0/3444675098 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56540512b0 con 0x7f5654103300 2024-05-27T07:36:47.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.486+0000 7f5651ffb700 1 -- 172.21.15.114:0/3444675098 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f563c00f450 con 0x7f5654103300 2024-05-27T07:36:47.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.487+0000 7f5651ffb700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f56400b2880 0x7f56400b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:47.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.487+0000 7f5653fff700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f56400b2880 0x7f56400b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:47.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.487+0000 7f5651ffb700 1 -- 172.21.15.114:0/3444675098 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f563c0e2650 con 0x7f5654103300 2024-05-27T07:36:47.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.487+0000 7f5653fff700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f56400b2880 0x7f56400b4d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5644005fd0 tx=0x7f5644005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:47.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.490+0000 7f5651ffb700 1 -- 172.21.15.114:0/3444675098 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f563c014030 con 0x7f5654103300 2024-05-27T07:36:47.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.714+0000 7f565a812700 1 -- 172.21.15.114:0/3444675098 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f565402e5e0 con 0x7f5654103300 2024-05-27T07:36:47.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.714+0000 7f5651ffb700 1 -- 172.21.15.114:0/3444675098 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) ==== 81+0+600 (secure 0 0 0) 0x7f563c0aa0a0 con 0x7f5654103300 2024-05-27T07:36:47.711 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/3782422776 2024-05-28T07:28:43.271481+0000 2024-05-27T07:36:47.711 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/3323915605 2024-05-28T07:28:43.271481+0000 2024-05-27T07:36:47.711 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/752776783 2024-05-28T07:28:43.271481+0000 2024-05-27T07:36:47.711 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2604585922 2024-05-28T07:28:16.331472+0000 2024-05-27T07:36:47.711 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2303561217 2024-05-28T07:28:16.331472+0000 2024-05-27T07:36:47.711 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/1568684596 2024-05-28T07:28:16.331472+0000 2024-05-27T07:36:47.711 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/1305604861 2024-05-28T07:28:43.271481+0000 2024-05-27T07:36:47.711 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/1568684596 2024-05-28T07:28:16.331472+0000 2024-05-27T07:36:47.711 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/752776783 2024-05-28T07:28:43.271481+0000 2024-05-27T07:36:47.711 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2442312517 2024-05-28T07:28:16.331472+0000 2024-05-27T07:36:47.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.716+0000 7f565a812700 1 -- 172.21.15.114:0/3444675098 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f56400b2880 msgr2=0x7f56400b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:47.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.716+0000 7f565a812700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f56400b2880 0x7f56400b4d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5644005fd0 tx=0x7f5644005d40 comp rx=0 tx=0).stop 2024-05-27T07:36:47.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.716+0000 7f565a812700 1 -- 172.21.15.114:0/3444675098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5654103300 msgr2=0x7f565419b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:47.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.716+0000 7f565a812700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5654103300 0x7f565419b3e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f563c00c8f0 tx=0x7f563c00ccb0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.716+0000 7f565a812700 1 -- 172.21.15.114:0/3444675098 shutdown_connections 2024-05-27T07:36:47.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.716+0000 7f565a812700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f56400b2880 0x7f56400b4d90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.716+0000 7f565a812700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56541089e0 0x7f565419f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.716+0000 7f565a812700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f56541029b0 0x7f565419ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.716+0000 7f565a812700 1 --2- 172.21.15.114:0/3444675098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5654103300 0x7f565419b3e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.716+0000 7f565a812700 1 -- 172.21.15.114:0/3444675098 >> 172.21.15.114:0/3444675098 conn(0x7f56540fe510 msgr2=0x7f5654100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:47.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.717+0000 7f565a812700 1 -- 172.21.15.114:0/3444675098 shutdown_connections 2024-05-27T07:36:47.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:47.717+0000 7f565a812700 1 -- 172.21.15.114:0/3444675098 wait complete. 2024-05-27T07:36:47.714 INFO:teuthology.orchestra.run.smithi114.stderr:listed 10 entries 2024-05-27T07:36:48.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:48 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3444675098' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:36:48.354 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:36:48.354 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:36:48.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T07:36:48.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:48 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3444675098' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:36:48.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:48 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3444675098' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:36:48.651 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:49.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:49 smithi114 ceph-mon[32038]: pgmap v319: 129 pgs: 6 peering, 123 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-05-27T07:36:49.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:49 smithi163 ceph-mon[36451]: pgmap v319: 129 pgs: 6 peering, 123 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-05-27T07:36:49.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:49 smithi186 ceph-mon[36647]: pgmap v319: 129 pgs: 6 peering, 123 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-05-27T07:36:50.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.380+0000 7f4b49920700 1 -- 172.21.15.114:0/1535541314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b44107de0 msgr2=0x7f4b441081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:50.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.380+0000 7f4b49920700 1 --2- 172.21.15.114:0/1535541314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b44107de0 0x7f4b441081f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c0099c0 tx=0x7f4b2c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.380+0000 7f4b49920700 1 -- 172.21.15.114:0/1535541314 shutdown_connections 2024-05-27T07:36:50.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.380+0000 7f4b49920700 1 --2- 172.21.15.114:0/1535541314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b44108730 0x7f4b44110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.380+0000 7f4b49920700 1 --2- 172.21.15.114:0/1535541314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b441034b0 0x7f4b441038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.380+0000 7f4b49920700 1 --2- 172.21.15.114:0/1535541314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b44107de0 0x7f4b441081f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.380+0000 7f4b49920700 1 -- 172.21.15.114:0/1535541314 >> 172.21.15.114:0/1535541314 conn(0x7f4b440fedb0 msgr2=0x7f4b441011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:50.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.380+0000 7f4b49920700 1 -- 172.21.15.114:0/1535541314 shutdown_connections 2024-05-27T07:36:50.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.380+0000 7f4b49920700 1 -- 172.21.15.114:0/1535541314 wait complete. 2024-05-27T07:36:50.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.381+0000 7f4b49920700 1 Processor -- start 2024-05-27T07:36:50.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.381+0000 7f4b49920700 1 -- start start 2024-05-27T07:36:50.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b49920700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b441034b0 0x7f4b4419bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:50.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b49920700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b44107de0 0x7f4b4419c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:50.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b49920700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b44108730 0x7f4b4419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:50.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b49920700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44113430 con 0x7f4b44108730 2024-05-27T07:36:50.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b49920700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4b441132b0 con 0x7f4b441034b0 2024-05-27T07:36:50.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b49920700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b441135b0 con 0x7f4b44107de0 2024-05-27T07:36:50.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b44108730 0x7f4b4419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:50.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b44108730 0x7f4b4419c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46218/0 (socket says 172.21.15.114:46218) 2024-05-27T07:36:50.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b437fe700 1 -- 172.21.15.114:0/2189397025 learned_addr learned my addr 172.21.15.114:0/2189397025 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:50.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b3a7fc700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b44107de0 0x7f4b4419c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:50.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b42ffd700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b441034b0 0x7f4b4419bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:50.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b437fe700 1 -- 172.21.15.114:0/2189397025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b44107de0 msgr2=0x7f4b4419c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:50.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b437fe700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b44107de0 0x7f4b4419c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b437fe700 1 -- 172.21.15.114:0/2189397025 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b441034b0 msgr2=0x7f4b4419bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:50.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b437fe700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b441034b0 0x7f4b4419bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b437fe700 1 -- 172.21.15.114:0/2189397025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b441a7040 con 0x7f4b44108730 2024-05-27T07:36:50.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b42ffd700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b441034b0 0x7f4b4419bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:50.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b437fe700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b44108730 0x7f4b4419c790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00b970 tx=0x7f4b3c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:50.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.382+0000 7f4b4891e700 1 -- 172.21.15.114:0/2189397025 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c019070 con 0x7f4b44108730 2024-05-27T07:36:50.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.383+0000 7f4b4891e700 1 -- 172.21.15.114:0/2189397025 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b3c00cde0 con 0x7f4b44108730 2024-05-27T07:36:50.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.383+0000 7f4b4891e700 1 -- 172.21.15.114:0/2189397025 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c007370 con 0x7f4b44108730 2024-05-27T07:36:50.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.383+0000 7f4b49920700 1 -- 172.21.15.114:0/2189397025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b441a7330 con 0x7f4b44108730 2024-05-27T07:36:50.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.383+0000 7f4b49920700 1 -- 172.21.15.114:0/2189397025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b441a7840 con 0x7f4b44108730 2024-05-27T07:36:50.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.383+0000 7f4b49920700 1 -- 172.21.15.114:0/2189397025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b440512b0 con 0x7f4b44108730 2024-05-27T07:36:50.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.385+0000 7f4b4891e700 1 -- 172.21.15.114:0/2189397025 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f4b3c01c070 con 0x7f4b44108730 2024-05-27T07:36:50.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.386+0000 7f4b4891e700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f4b300b2900 0x7f4b300b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:50.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.386+0000 7f4b42ffd700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f4b300b2900 0x7f4b300b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:50.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.386+0000 7f4b4891e700 1 -- 172.21.15.114:0/2189397025 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f4b3c0e2140 con 0x7f4b44108730 2024-05-27T07:36:50.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.386+0000 7f4b42ffd700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f4b300b2900 0x7f4b300b4e10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4b340099d0 tx=0x7f4b34008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:50.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.389+0000 7f4b4891e700 1 -- 172.21.15.114:0/2189397025 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4b3c023070 con 0x7f4b44108730 2024-05-27T07:36:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.613+0000 7f4b49920700 1 -- 172.21.15.114:0/2189397025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4b4406bc10 con 0x7f4b44108730 2024-05-27T07:36:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.613+0000 7f4b4891e700 1 -- 172.21.15.114:0/2189397025 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) ==== 81+0+600 (secure 0 0 0) 0x7f4b3c0a9b80 con 0x7f4b44108730 2024-05-27T07:36:50.609 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/3782422776 2024-05-28T07:28:43.271481+0000 2024-05-27T07:36:50.609 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/3323915605 2024-05-28T07:28:43.271481+0000 2024-05-27T07:36:50.609 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/752776783 2024-05-28T07:28:43.271481+0000 2024-05-27T07:36:50.609 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2604585922 2024-05-28T07:28:16.331472+0000 2024-05-27T07:36:50.609 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2303561217 2024-05-28T07:28:16.331472+0000 2024-05-27T07:36:50.609 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/1568684596 2024-05-28T07:28:16.331472+0000 2024-05-27T07:36:50.609 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/1305604861 2024-05-28T07:28:43.271481+0000 2024-05-27T07:36:50.609 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/1568684596 2024-05-28T07:28:16.331472+0000 2024-05-27T07:36:50.609 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/752776783 2024-05-28T07:28:43.271481+0000 2024-05-27T07:36:50.609 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2442312517 2024-05-28T07:28:16.331472+0000 2024-05-27T07:36:50.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.616+0000 7f4b49920700 1 -- 172.21.15.114:0/2189397025 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f4b300b2900 msgr2=0x7f4b300b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:50.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.616+0000 7f4b49920700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f4b300b2900 0x7f4b300b4e10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4b340099d0 tx=0x7f4b34008040 comp rx=0 tx=0).stop 2024-05-27T07:36:50.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.616+0000 7f4b49920700 1 -- 172.21.15.114:0/2189397025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b44108730 msgr2=0x7f4b4419c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:50.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.616+0000 7f4b49920700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b44108730 0x7f4b4419c790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00b970 tx=0x7f4b3c00bc80 comp rx=0 tx=0).stop 2024-05-27T07:36:50.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.616+0000 7f4b49920700 1 -- 172.21.15.114:0/2189397025 shutdown_connections 2024-05-27T07:36:50.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.616+0000 7f4b49920700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b44108730 0x7f4b4419c790 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.616+0000 7f4b49920700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b441034b0 0x7f4b4419bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.616+0000 7f4b49920700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f4b300b2900 0x7f4b300b4e10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.616+0000 7f4b49920700 1 --2- 172.21.15.114:0/2189397025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b44107de0 0x7f4b4419c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.617+0000 7f4b49920700 1 -- 172.21.15.114:0/2189397025 >> 172.21.15.114:0/2189397025 conn(0x7f4b440fedb0 msgr2=0x7f4b441011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:50.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.617+0000 7f4b49920700 1 -- 172.21.15.114:0/2189397025 shutdown_connections 2024-05-27T07:36:50.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:50.617+0000 7f4b49920700 1 -- 172.21.15.114:0/2189397025 wait complete. 2024-05-27T07:36:50.613 INFO:teuthology.orchestra.run.smithi114.stderr:listed 10 entries 2024-05-27T07:36:51.285 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:51 smithi114 ceph-mon[32038]: pgmap v320: 129 pgs: 6 peering, 123 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-05-27T07:36:51.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:51 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2189397025' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:36:51.313 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2024-05-27T07:36:51.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:36:51.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:36:51.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi114.stderr:ceph-fuse[75276]: fuse finished with error 0 and tester_r 0 2024-05-27T07:36:51.341 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:36:51.381 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-05-27T07:36:51.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:36:51.382 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-05-27T07:36:51.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:36:51.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:51 smithi163 ceph-mon[36451]: pgmap v320: 129 pgs: 6 peering, 123 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-05-27T07:36:51.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:51 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2189397025' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:36:51.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:51 smithi186 ceph-mon[36647]: pgmap v320: 129 pgs: 6 peering, 123 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-05-27T07:36:51.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:51 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2189397025' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:36:51.723 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:53.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.398+0000 7f7c5e923700 1 -- 172.21.15.114:0/378973326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c58103990 msgr2=0x7f7c58103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:53.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.398+0000 7f7c5e923700 1 --2- 172.21.15.114:0/378973326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c58103990 0x7f7c58103da0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009990 tx=0x7f7c4c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:53 smithi114 ceph-mon[32038]: pgmap v321: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:36:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.398+0000 7f7c5e923700 1 -- 172.21.15.114:0/378973326 shutdown_connections 2024-05-27T07:36:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.398+0000 7f7c5e923700 1 --2- 172.21.15.114:0/378973326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c581042e0 0x7f7c58110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.398+0000 7f7c5e923700 1 --2- 172.21.15.114:0/378973326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c581081f0 0x7f7c58108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.398+0000 7f7c5e923700 1 --2- 172.21.15.114:0/378973326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c58103990 0x7f7c58103da0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.398+0000 7f7c5e923700 1 -- 172.21.15.114:0/378973326 >> 172.21.15.114:0/378973326 conn(0x7f7c580fe510 msgr2=0x7f7c58100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.398+0000 7f7c5e923700 1 -- 172.21.15.114:0/378973326 shutdown_connections 2024-05-27T07:36:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.398+0000 7f7c5e923700 1 -- 172.21.15.114:0/378973326 wait complete. 2024-05-27T07:36:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.399+0000 7f7c5e923700 1 Processor -- start 2024-05-27T07:36:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.399+0000 7f7c5e923700 1 -- start start 2024-05-27T07:36:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5e923700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c58103990 0x7f7c5819bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5e923700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c581042e0 0x7f7c5819c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5e923700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c581081f0 0x7f7c5819c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5e923700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7c58113370 con 0x7f7c581081f0 2024-05-27T07:36:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5e923700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7c581131f0 con 0x7f7c58103990 2024-05-27T07:36:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5e923700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c581134f0 con 0x7f7c581042e0 2024-05-27T07:36:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5cec0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c581081f0 0x7f7c5819c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5cec0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c581081f0 0x7f7c5819c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46248/0 (socket says 172.21.15.114:46248) 2024-05-27T07:36:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5cec0700 1 -- 172.21.15.114:0/3257902036 learned_addr learned my addr 172.21.15.114:0/3257902036 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c57fff700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c58103990 0x7f7c5819bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c577fe700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c581042e0 0x7f7c5819c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5cec0700 1 -- 172.21.15.114:0/3257902036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c581042e0 msgr2=0x7f7c5819c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5cec0700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c581042e0 0x7f7c5819c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5cec0700 1 -- 172.21.15.114:0/3257902036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c58103990 msgr2=0x7f7c5819bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5cec0700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c58103990 0x7f7c5819bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5cec0700 1 -- 172.21.15.114:0/3257902036 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c581a6f90 con 0x7f7c581081f0 2024-05-27T07:36:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.400+0000 7f7c5cec0700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c581081f0 0x7f7c5819c770 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7c50007d70 tx=0x7f7c5000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.401+0000 7f7c557fa700 1 -- 172.21.15.114:0/3257902036 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50016020 con 0x7f7c581081f0 2024-05-27T07:36:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.401+0000 7f7c5e923700 1 -- 172.21.15.114:0/3257902036 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c581a7280 con 0x7f7c581081f0 2024-05-27T07:36:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.401+0000 7f7c557fa700 1 -- 172.21.15.114:0/3257902036 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c5000ad40 con 0x7f7c581081f0 2024-05-27T07:36:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.401+0000 7f7c557fa700 1 -- 172.21.15.114:0/3257902036 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c500115e0 con 0x7f7c581081f0 2024-05-27T07:36:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.401+0000 7f7c5e923700 1 -- 172.21.15.114:0/3257902036 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c581a77c0 con 0x7f7c581081f0 2024-05-27T07:36:53.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.402+0000 7f7c5e923700 1 -- 172.21.15.114:0/3257902036 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c5810b6e0 con 0x7f7c581081f0 2024-05-27T07:36:53.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.403+0000 7f7c557fa700 1 -- 172.21.15.114:0/3257902036 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f7c50019020 con 0x7f7c581081f0 2024-05-27T07:36:53.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.404+0000 7f7c557fa700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7c400b2900 0x7f7c400b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:53.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.404+0000 7f7c57fff700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7c400b2900 0x7f7c400b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:53.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.404+0000 7f7c557fa700 1 -- 172.21.15.114:0/3257902036 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f7c500e2a00 con 0x7f7c581081f0 2024-05-27T07:36:53.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.405+0000 7f7c57fff700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7c400b2900 0x7f7c400b4e10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7c5806c2e0 tx=0x7f7c4800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:53.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.408+0000 7f7c557fa700 1 -- 172.21.15.114:0/3257902036 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7c50020020 con 0x7f7c581081f0 2024-05-27T07:36:53.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:53 smithi163 ceph-mon[36451]: pgmap v321: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:36:53.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:53 smithi186 ceph-mon[36647]: pgmap v321: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:36:53.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.614+0000 7f7c5e923700 1 -- 172.21.15.114:0/3257902036 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c58102ea0 con 0x7f7c581081f0 2024-05-27T07:36:53.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.616+0000 7f7c557fa700 1 -- 172.21.15.114:0/3257902036 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+24736 (secure 0 0 0) 0x7f7c58102ea0 con 0x7f7c581081f0 2024-05-27T07:36:53.612 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:36:53.612 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T07:36:45.114309+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:36:53.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.619+0000 7f7c5e923700 1 -- 172.21.15.114:0/3257902036 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7c400b2900 msgr2=0x7f7c400b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:53.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.619+0000 7f7c5e923700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7c400b2900 0x7f7c400b4e10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7c5806c2e0 tx=0x7f7c4800a380 comp rx=0 tx=0).stop 2024-05-27T07:36:53.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.619+0000 7f7c5e923700 1 -- 172.21.15.114:0/3257902036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c581081f0 msgr2=0x7f7c5819c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:53.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.619+0000 7f7c5e923700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c581081f0 0x7f7c5819c770 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7c50007d70 tx=0x7f7c5000a340 comp rx=0 tx=0).stop 2024-05-27T07:36:53.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.619+0000 7f7c5e923700 1 -- 172.21.15.114:0/3257902036 shutdown_connections 2024-05-27T07:36:53.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.619+0000 7f7c5e923700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c581081f0 0x7f7c5819c770 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.619+0000 7f7c5e923700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c58103990 0x7f7c5819bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.619+0000 7f7c5e923700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7c400b2900 0x7f7c400b4e10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.619+0000 7f7c5e923700 1 --2- 172.21.15.114:0/3257902036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c581042e0 0x7f7c5819c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.619+0000 7f7c5e923700 1 -- 172.21.15.114:0/3257902036 >> 172.21.15.114:0/3257902036 conn(0x7f7c580fe510 msgr2=0x7f7c58100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:53.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.619+0000 7f7c5e923700 1 -- 172.21.15.114:0/3257902036 shutdown_connections 2024-05-27T07:36:53.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:53.619+0000 7f7c5e923700 1 -- 172.21.15.114:0/3257902036 wait complete. 2024-05-27T07:36:54.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:36:54.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:54 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3257902036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:54.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:54 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3257902036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:54.533 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:54.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:54 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3257902036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:55.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:55 smithi114 ceph-mon[32038]: pgmap v322: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 442 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:36:55.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:55 smithi163 ceph-mon[36451]: pgmap v322: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 442 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:36:55.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:55 smithi186 ceph-mon[36647]: pgmap v322: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 442 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:36:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.092+0000 7f34592ca700 1 -- 172.21.15.114:0/1933372120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3454108730 msgr2=0x7f3454110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.092+0000 7f34592ca700 1 --2- 172.21.15.114:0/1933372120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3454108730 0x7f3454110090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f344c00b0d0 tx=0x7f344c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:36:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.092+0000 7f34592ca700 1 -- 172.21.15.114:0/1933372120 shutdown_connections 2024-05-27T07:36:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.092+0000 7f34592ca700 1 --2- 172.21.15.114:0/1933372120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3454108730 0x7f3454110090 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.092+0000 7f34592ca700 1 --2- 172.21.15.114:0/1933372120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f34541034b0 0x7f34541038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.092+0000 7f34592ca700 1 --2- 172.21.15.114:0/1933372120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3454107de0 0x7f34541081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.092+0000 7f34592ca700 1 -- 172.21.15.114:0/1933372120 >> 172.21.15.114:0/1933372120 conn(0x7f34540fedb0 msgr2=0x7f34541011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.092+0000 7f34592ca700 1 -- 172.21.15.114:0/1933372120 shutdown_connections 2024-05-27T07:36:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.092+0000 7f34592ca700 1 -- 172.21.15.114:0/1933372120 wait complete. 2024-05-27T07:36:56.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f34592ca700 1 Processor -- start 2024-05-27T07:36:56.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f34592ca700 1 -- start start 2024-05-27T07:36:56.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f34592ca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34541034b0 0x7f3454110870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:56.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f34592ca700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3454107de0 0x7f3454110db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:56.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f34592ca700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3454108730 0x7f34541112f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:56.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f34592ca700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f34541144b0 con 0x7f34541034b0 2024-05-27T07:36:56.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f34592ca700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3454114330 con 0x7f3454107de0 2024-05-27T07:36:56.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f34592ca700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3454114630 con 0x7f3454108730 2024-05-27T07:36:56.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f3452ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34541034b0 0x7f3454110870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:56.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f3452ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34541034b0 0x7f3454110870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57796/0 (socket says 172.21.15.114:57796) 2024-05-27T07:36:56.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f3452ffd700 1 -- 172.21.15.114:0/1144046712 learned_addr learned my addr 172.21.15.114:0/1144046712 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:56.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f34537fe700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3454108730 0x7f34541112f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:56.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f34527fc700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3454107de0 0x7f3454110db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:56.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f3452ffd700 1 -- 172.21.15.114:0/1144046712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3454108730 msgr2=0x7f34541112f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:56.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f3452ffd700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3454108730 0x7f34541112f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:56.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f3452ffd700 1 -- 172.21.15.114:0/1144046712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3454107de0 msgr2=0x7f3454110db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:56.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.093+0000 7f3452ffd700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3454107de0 0x7f3454110db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:56.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.094+0000 7f3452ffd700 1 -- 172.21.15.114:0/1144046712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34541a90b0 con 0x7f34541034b0 2024-05-27T07:36:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.094+0000 7f34537fe700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3454108730 0x7f34541112f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.094+0000 7f34527fc700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3454107de0 0x7f3454110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.094+0000 7f3452ffd700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34541034b0 0x7f3454110870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f344400c9b0 tx=0x7f344400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.094+0000 7f343bfff700 1 -- 172.21.15.114:0/1144046712 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444007b00 con 0x7f34541034b0 2024-05-27T07:36:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.094+0000 7f343bfff700 1 -- 172.21.15.114:0/1144046712 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3444007ca0 con 0x7f34541034b0 2024-05-27T07:36:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.094+0000 7f34592ca700 1 -- 172.21.15.114:0/1144046712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34541a93a0 con 0x7f34541034b0 2024-05-27T07:36:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.094+0000 7f343bfff700 1 -- 172.21.15.114:0/1144046712 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444018730 con 0x7f34541034b0 2024-05-27T07:36:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.094+0000 7f34592ca700 1 -- 172.21.15.114:0/1144046712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34541a98e0 con 0x7f34541034b0 2024-05-27T07:36:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.095+0000 7f34592ca700 1 -- 172.21.15.114:0/1144046712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34540512b0 con 0x7f34541034b0 2024-05-27T07:36:56.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.096+0000 7f343bfff700 1 -- 172.21.15.114:0/1144046712 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f344400f450 con 0x7f34541034b0 2024-05-27T07:36:56.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.097+0000 7f343bfff700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f343c0b2880 0x7f343c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:56.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.097+0000 7f34527fc700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f343c0b2880 0x7f343c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:56.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.097+0000 7f343bfff700 1 -- 172.21.15.114:0/1144046712 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f34440e2900 con 0x7f34541034b0 2024-05-27T07:36:56.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.098+0000 7f34527fc700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f343c0b2880 0x7f343c0b4d90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3448005fd0 tx=0x7f3448005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:56.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.101+0000 7f343bfff700 1 -- 172.21.15.114:0/1144046712 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3444014030 con 0x7f34541034b0 2024-05-27T07:36:56.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.311+0000 7f34592ca700 1 -- 172.21.15.114:0/1144046712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3454111da0 con 0x7f34541034b0 2024-05-27T07:36:56.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.313+0000 7f343bfff700 1 -- 172.21.15.114:0/1144046712 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+24736 (secure 0 0 0) 0x7f34440aa350 con 0x7f34541034b0 2024-05-27T07:36:56.309 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:36:56.309 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T07:36:45.114309+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:36:56.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.315+0000 7f34592ca700 1 -- 172.21.15.114:0/1144046712 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f343c0b2880 msgr2=0x7f343c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:56.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.315+0000 7f34592ca700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f343c0b2880 0x7f343c0b4d90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3448005fd0 tx=0x7f3448005db0 comp rx=0 tx=0).stop 2024-05-27T07:36:56.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.315+0000 7f34592ca700 1 -- 172.21.15.114:0/1144046712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34541034b0 msgr2=0x7f3454110870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:56.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.315+0000 7f34592ca700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34541034b0 0x7f3454110870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f344400c9b0 tx=0x7f344400cd70 comp rx=0 tx=0).stop 2024-05-27T07:36:56.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.316+0000 7f34592ca700 1 -- 172.21.15.114:0/1144046712 shutdown_connections 2024-05-27T07:36:56.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.316+0000 7f34592ca700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f343c0b2880 0x7f343c0b4d90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:56.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.316+0000 7f34592ca700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3454108730 0x7f34541112f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:56.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.316+0000 7f34592ca700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34541034b0 0x7f3454110870 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:56.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.316+0000 7f34592ca700 1 --2- 172.21.15.114:0/1144046712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3454107de0 0x7f3454110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:56.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.316+0000 7f34592ca700 1 -- 172.21.15.114:0/1144046712 >> 172.21.15.114:0/1144046712 conn(0x7f34540fedb0 msgr2=0x7f3454107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:56.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.316+0000 7f34592ca700 1 -- 172.21.15.114:0/1144046712 shutdown_connections 2024-05-27T07:36:56.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:56.316+0000 7f34592ca700 1 -- 172.21.15.114:0/1144046712 wait complete. 2024-05-27T07:36:57.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-05-27T07:36:57.249 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:57.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:57 smithi114 ceph-mon[32038]: pgmap v323: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 399 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:36:57.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:57 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1144046712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:57.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:57 smithi163 ceph-mon[36451]: pgmap v323: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 399 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:36:57.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:57 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1144046712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:57.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:57 smithi186 ceph-mon[36647]: pgmap v323: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 399 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:36:57.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:57 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1144046712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:59.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.046+0000 7f672090c700 1 -- 172.21.15.114:0/1576003212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f671c107de0 msgr2=0x7f671c1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:59.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.046+0000 7f672090c700 1 --2- 172.21.15.114:0/1576003212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f671c107de0 0x7f671c1081f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f67040099c0 tx=0x7f6704009cd0 comp rx=0 tx=0).stop 2024-05-27T07:36:59.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.046+0000 7f672090c700 1 -- 172.21.15.114:0/1576003212 shutdown_connections 2024-05-27T07:36:59.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.046+0000 7f672090c700 1 --2- 172.21.15.114:0/1576003212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f671c108730 0x7f671c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:59.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.046+0000 7f672090c700 1 --2- 172.21.15.114:0/1576003212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f671c1034b0 0x7f671c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:59.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.046+0000 7f672090c700 1 --2- 172.21.15.114:0/1576003212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f671c107de0 0x7f671c1081f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:59.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.046+0000 7f672090c700 1 -- 172.21.15.114:0/1576003212 >> 172.21.15.114:0/1576003212 conn(0x7f671c0fede0 msgr2=0x7f671c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:59.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.046+0000 7f672090c700 1 -- 172.21.15.114:0/1576003212 shutdown_connections 2024-05-27T07:36:59.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.046+0000 7f672090c700 1 -- 172.21.15.114:0/1576003212 wait complete. 2024-05-27T07:36:59.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.047+0000 7f672090c700 1 Processor -- start 2024-05-27T07:36:59.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.047+0000 7f672090c700 1 -- start start 2024-05-27T07:36:59.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.047+0000 7f672090c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f671c1034b0 0x7f671c199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:59.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.047+0000 7f672090c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f671c107de0 0x7f671c19a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:59.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.047+0000 7f672090c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f671c108730 0x7f671c19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:59.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.047+0000 7f672090c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f671c1133b0 con 0x7f671c108730 2024-05-27T07:36:59.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.047+0000 7f672090c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f671c113230 con 0x7f671c107de0 2024-05-27T07:36:59.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.047+0000 7f672090c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f671c113530 con 0x7f671c1034b0 2024-05-27T07:36:59.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.047+0000 7f671ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f671c108730 0x7f671c19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:59.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.047+0000 7f671ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f671c108730 0x7f671c19e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57820/0 (socket says 172.21.15.114:57820) 2024-05-27T07:36:59.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.047+0000 7f671ad9d700 1 -- 172.21.15.114:0/2636281067 learned_addr learned my addr 172.21.15.114:0/2636281067 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:36:59.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.047+0000 7f6719d9b700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f671c107de0 0x7f671c19a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:59.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.047+0000 7f671a59c700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f671c1034b0 0x7f671c199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:59.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.048+0000 7f671ad9d700 1 -- 172.21.15.114:0/2636281067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f671c1034b0 msgr2=0x7f671c199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:59.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.048+0000 7f671ad9d700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f671c1034b0 0x7f671c199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:59.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.048+0000 7f671ad9d700 1 -- 172.21.15.114:0/2636281067 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f671c107de0 msgr2=0x7f671c19a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:59.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.048+0000 7f671ad9d700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f671c107de0 0x7f671c19a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:59.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.048+0000 7f671ad9d700 1 -- 172.21.15.114:0/2636281067 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671c19ec90 con 0x7f671c108730 2024-05-27T07:36:59.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.048+0000 7f671ad9d700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f671c108730 0x7f671c19e5f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f670c00d960 tx=0x7f670c00dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:59.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.048+0000 7f67137fe700 1 -- 172.21.15.114:0/2636281067 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c01b070 con 0x7f671c108730 2024-05-27T07:36:59.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.048+0000 7f672090c700 1 -- 172.21.15.114:0/2636281067 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f671c19ef80 con 0x7f671c108730 2024-05-27T07:36:59.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.048+0000 7f67137fe700 1 -- 172.21.15.114:0/2636281067 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f670c016440 con 0x7f671c108730 2024-05-27T07:36:59.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.048+0000 7f67137fe700 1 -- 172.21.15.114:0/2636281067 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c0145a0 con 0x7f671c108730 2024-05-27T07:36:59.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.048+0000 7f672090c700 1 -- 172.21.15.114:0/2636281067 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f671c0759e0 con 0x7f671c108730 2024-05-27T07:36:59.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.049+0000 7f672090c700 1 -- 172.21.15.114:0/2636281067 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f671c104960 con 0x7f671c108730 2024-05-27T07:36:59.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.050+0000 7f67137fe700 1 -- 172.21.15.114:0/2636281067 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f670c01e070 con 0x7f671c108730 2024-05-27T07:36:59.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.051+0000 7f67137fe700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f67080b2880 0x7f67080b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:59.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.051+0000 7f671a59c700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f67080b2880 0x7f67080b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:59.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.051+0000 7f67137fe700 1 -- 172.21.15.114:0/2636281067 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f670c0e4220 con 0x7f671c108730 2024-05-27T07:36:59.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.052+0000 7f671a59c700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f67080b2880 0x7f67080b4d90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f67140060b0 tx=0x7f671400c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:59.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.055+0000 7f67137fe700 1 -- 172.21.15.114:0/2636281067 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f670c025070 con 0x7f671c108730 2024-05-27T07:36:59.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:36:59.307+0000 7f672090c700 1 -- 172.21.15.114:0/2636281067 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f671c0512b0 con 0x7f671c108730 2024-05-27T07:36:59.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:36:59 smithi163 ceph-mon[36451]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s; 1 B/s, 0 objects/s recovering 2024-05-27T07:36:59.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:36:59 smithi186 ceph-mon[36647]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s; 1 B/s, 0 objects/s recovering 2024-05-27T07:36:59.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:36:59 smithi114 ceph-mon[32038]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s; 1 B/s, 0 objects/s recovering 2024-05-27T07:37:00.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:00.165+0000 7f67137fe700 1 -- 172.21.15.114:0/2636281067 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f671c0512b0 con 0x7f671c108730 2024-05-27T07:37:00.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:00.170+0000 7f672090c700 1 -- 172.21.15.114:0/2636281067 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f67080b2880 msgr2=0x7f67080b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:00.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:00.170+0000 7f672090c700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f67080b2880 0x7f67080b4d90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f67140060b0 tx=0x7f671400c040 comp rx=0 tx=0).stop 2024-05-27T07:37:00.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:00.170+0000 7f672090c700 1 -- 172.21.15.114:0/2636281067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f671c108730 msgr2=0x7f671c19e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:00.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:00.170+0000 7f672090c700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f671c108730 0x7f671c19e5f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f670c00d960 tx=0x7f670c00dc70 comp rx=0 tx=0).stop 2024-05-27T07:37:00.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:00.170+0000 7f672090c700 1 -- 172.21.15.114:0/2636281067 shutdown_connections 2024-05-27T07:37:00.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:00.170+0000 7f672090c700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f671c108730 0x7f671c19e5f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:00.170+0000 7f672090c700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f67080b2880 0x7f67080b4d90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:00.170+0000 7f672090c700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f671c1034b0 0x7f671c199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:00.170+0000 7f672090c700 1 --2- 172.21.15.114:0/2636281067 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f671c107de0 0x7f671c19a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:00.170+0000 7f672090c700 1 -- 172.21.15.114:0/2636281067 >> 172.21.15.114:0/2636281067 conn(0x7f671c0fede0 msgr2=0x7f671c109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:00.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:00.170+0000 7f672090c700 1 -- 172.21.15.114:0/2636281067 shutdown_connections 2024-05-27T07:37:00.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:00.170+0000 7f672090c700 1 -- 172.21.15.114:0/2636281067 wait complete. 2024-05-27T07:37:00.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:00 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2636281067' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T07:37:00.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:00 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2636281067' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T07:37:00.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:00 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2636281067' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T07:37:00.896 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-05-27T07:37:00.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-05-27T07:37:01.150 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:01.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:01 smithi163 ceph-mon[36451]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s; 1 B/s, 0 objects/s recovering 2024-05-27T07:37:01.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:01 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2636281067' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T07:37:01.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:01 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:37:01.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:01 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:37:01.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:01 smithi186 ceph-mon[36647]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s; 1 B/s, 0 objects/s recovering 2024-05-27T07:37:01.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:01 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2636281067' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T07:37:01.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:01 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:37:01.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:01 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:37:01.475 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:01 smithi114 ceph-mon[32038]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s; 1 B/s, 0 objects/s recovering 2024-05-27T07:37:01.475 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:01 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2636281067' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T07:37:01.475 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:01 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:37:01.475 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:01 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:37:02.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.796+0000 7ff8511a0700 1 -- 172.21.15.114:0/3349786852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff84c10e8e0 msgr2=0x7ff84c110dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:02.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.796+0000 7ff8511a0700 1 --2- 172.21.15.114:0/3349786852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff84c10e8e0 0x7ff84c110dc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff84400b0a0 tx=0x7ff84400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:37:02.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.796+0000 7ff8511a0700 1 -- 172.21.15.114:0/3349786852 shutdown_connections 2024-05-27T07:37:02.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.796+0000 7ff8511a0700 1 --2- 172.21.15.114:0/3349786852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff84c10e8e0 0x7ff84c110dc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:02.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.796+0000 7ff8511a0700 1 --2- 172.21.15.114:0/3349786852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c105470 0x7ff84c105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:02.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.796+0000 7ff8511a0700 1 --2- 172.21.15.114:0/3349786852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff84c105dc0 0x7ff84c10e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:02.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.796+0000 7ff8511a0700 1 -- 172.21.15.114:0/3349786852 >> 172.21.15.114:0/3349786852 conn(0x7ff84c0fef20 msgr2=0x7ff84c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:02.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.796+0000 7ff8511a0700 1 -- 172.21.15.114:0/3349786852 shutdown_connections 2024-05-27T07:37:02.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.796+0000 7ff8511a0700 1 -- 172.21.15.114:0/3349786852 wait complete. 2024-05-27T07:37:02.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.797+0000 7ff8511a0700 1 Processor -- start 2024-05-27T07:37:02.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.798+0000 7ff8511a0700 1 -- start start 2024-05-27T07:37:02.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.798+0000 7ff8511a0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff84c105470 0x7ff84c19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:02.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.798+0000 7ff8511a0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff84c105dc0 0x7ff84c19b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:02.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.798+0000 7ff8511a0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c10e8e0 0x7ff84c19f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:02.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.798+0000 7ff8511a0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff84c114440 con 0x7ff84c105dc0 2024-05-27T07:37:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.798+0000 7ff8511a0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff84c1142c0 con 0x7ff84c105470 2024-05-27T07:37:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.798+0000 7ff8511a0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff84c1145c0 con 0x7ff84c10e8e0 2024-05-27T07:37:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.798+0000 7ff84a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff84c105dc0 0x7ff84c19b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.798+0000 7ff84b59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c10e8e0 0x7ff84c19f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.798+0000 7ff84a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff84c105dc0 0x7ff84c19b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57852/0 (socket says 172.21.15.114:57852) 2024-05-27T07:37:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.798+0000 7ff84a59c700 1 -- 172.21.15.114:0/2697381749 learned_addr learned my addr 172.21.15.114:0/2697381749 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.798+0000 7ff84ad9d700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff84c105470 0x7ff84c19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.799+0000 7ff84b59e700 1 -- 172.21.15.114:0/2697381749 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff84c105470 msgr2=0x7ff84c19ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.799+0000 7ff84b59e700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff84c105470 0x7ff84c19ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.799+0000 7ff84b59e700 1 -- 172.21.15.114:0/2697381749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff84c105dc0 msgr2=0x7ff84c19b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.799+0000 7ff84b59e700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff84c105dc0 0x7ff84c19b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.799+0000 7ff84b59e700 1 -- 172.21.15.114:0/2697381749 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff84c19fce0 con 0x7ff84c10e8e0 2024-05-27T07:37:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.799+0000 7ff84ad9d700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff84c105470 0x7ff84c19ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:37:02.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.799+0000 7ff84b59e700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c10e8e0 0x7ff84c19f640 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff84400b070 tx=0x7ff844004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:02.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.799+0000 7ff833fff700 1 -- 172.21.15.114:0/2697381749 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff844021070 con 0x7ff84c10e8e0 2024-05-27T07:37:02.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.799+0000 7ff8511a0700 1 -- 172.21.15.114:0/2697381749 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff84c19ff70 con 0x7ff84c10e8e0 2024-05-27T07:37:02.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.799+0000 7ff833fff700 1 -- 172.21.15.114:0/2697381749 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff844007e30 con 0x7ff84c10e8e0 2024-05-27T07:37:02.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.799+0000 7ff833fff700 1 -- 172.21.15.114:0/2697381749 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8440045d0 con 0x7ff84c10e8e0 2024-05-27T07:37:02.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.799+0000 7ff8511a0700 1 -- 172.21.15.114:0/2697381749 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff84c1a78f0 con 0x7ff84c10e8e0 2024-05-27T07:37:02.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.801+0000 7ff8511a0700 1 -- 172.21.15.114:0/2697381749 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff84c06c910 con 0x7ff84c10e8e0 2024-05-27T07:37:02.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.802+0000 7ff833fff700 1 -- 172.21.15.114:0/2697381749 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7ff844003c00 con 0x7ff84c10e8e0 2024-05-27T07:37:02.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.803+0000 7ff833fff700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff8340b2900 0x7ff8340b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:02.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.803+0000 7ff84ad9d700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff8340b2900 0x7ff8340b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:02.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.803+0000 7ff833fff700 1 -- 172.21.15.114:0/2697381749 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7ff84400f020 con 0x7ff84c10e8e0 2024-05-27T07:37:02.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.803+0000 7ff84ad9d700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff8340b2900 0x7ff8340b4e10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff83c011490 tx=0x7ff83c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:02.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:02.811+0000 7ff833fff700 1 -- 172.21.15.114:0/2697381749 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff84402a020 con 0x7ff84c10e8e0 2024-05-27T07:37:03.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.105+0000 7ff8511a0700 1 -- 172.21.15.114:0/2697381749 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7ff84c19c170 con 0x7ff84c10e8e0 2024-05-27T07:37:03.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:03 smithi114 ceph-mon[32038]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s; 1 B/s, 0 objects/s recovering 2024-05-27T07:37:03.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:03 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:03.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:03 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:03.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:03 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:03.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:03 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:03.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:03 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2697381749' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:37:03.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:03 smithi114 ceph-mon[32038]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:37:03.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.442+0000 7ff833fff700 1 -- 172.21.15.114:0/2697381749 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7ff8440ab540 con 0x7ff84c10e8e0 2024-05-27T07:37:03.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.445+0000 7ff8511a0700 1 -- 172.21.15.114:0/2697381749 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff8340b2900 msgr2=0x7ff8340b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:03.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.445+0000 7ff8511a0700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff8340b2900 0x7ff8340b4e10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff83c011490 tx=0x7ff83c009260 comp rx=0 tx=0).stop 2024-05-27T07:37:03.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.445+0000 7ff8511a0700 1 -- 172.21.15.114:0/2697381749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c10e8e0 msgr2=0x7ff84c19f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:03.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.445+0000 7ff8511a0700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c10e8e0 0x7ff84c19f640 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff84400b070 tx=0x7ff844004d00 comp rx=0 tx=0).stop 2024-05-27T07:37:03.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.445+0000 7ff8511a0700 1 -- 172.21.15.114:0/2697381749 shutdown_connections 2024-05-27T07:37:03.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.445+0000 7ff8511a0700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff8340b2900 0x7ff8340b4e10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:03.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.445+0000 7ff8511a0700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c10e8e0 0x7ff84c19f640 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:03.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.445+0000 7ff8511a0700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff84c105470 0x7ff84c19ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:03.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.445+0000 7ff8511a0700 1 --2- 172.21.15.114:0/2697381749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff84c105dc0 0x7ff84c19b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:03.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.445+0000 7ff8511a0700 1 -- 172.21.15.114:0/2697381749 >> 172.21.15.114:0/2697381749 conn(0x7ff84c0fef20 msgr2=0x7ff84c0ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:03.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.445+0000 7ff8511a0700 1 -- 172.21.15.114:0/2697381749 shutdown_connections 2024-05-27T07:37:03.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:03.445+0000 7ff8511a0700 1 -- 172.21.15.114:0/2697381749 wait complete. 2024-05-27T07:37:03.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:03 smithi163 ceph-mon[36451]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s; 1 B/s, 0 objects/s recovering 2024-05-27T07:37:03.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:03 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:03.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:03 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:03.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:03 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:03.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:03 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:03.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:03 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2697381749' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:37:03.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:03 smithi163 ceph-mon[36451]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:37:03.474 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:03 smithi186 ceph-mon[36647]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s; 1 B/s, 0 objects/s recovering 2024-05-27T07:37:03.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:03 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:03.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:03 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:03.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:03 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:03.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:03 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:03.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:03 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2697381749' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:37:03.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:03 smithi186 ceph-mon[36647]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:37:04.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-05-27T07:37:04.646 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:04.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:04 smithi163 ceph-mon[36451]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:37:04.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:04 smithi163 ceph-mon[36451]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T07:37:04.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:04 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:37:04.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:04 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:37:04.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:04 smithi186 ceph-mon[36647]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:37:04.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:04 smithi186 ceph-mon[36647]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T07:37:04.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:04 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:37:04.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:04 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:37:04.768 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:04 smithi114 ceph-mon[32038]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:37:04.768 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:04 smithi114 ceph-mon[32038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T07:37:04.769 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:04 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:37:04.769 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:04 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:37:06.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.380+0000 7fceb8d20700 1 -- 172.21.15.114:0/4218522411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fceb41030d0 msgr2=0x7fceb4111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:06.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.380+0000 7fceb8d20700 1 --2- 172.21.15.114:0/4218522411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fceb41030d0 0x7fceb4111160 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fceac00b0a0 tx=0x7fceac00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.380+0000 7fceb8d20700 1 -- 172.21.15.114:0/4218522411 shutdown_connections 2024-05-27T07:37:06.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.380+0000 7fceb8d20700 1 --2- 172.21.15.114:0/4218522411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fceb41030d0 0x7fceb4111160 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.380+0000 7fceb8d20700 1 --2- 172.21.15.114:0/4218522411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceb41092f0 0x7fceb4109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.380+0000 7fceb8d20700 1 --2- 172.21.15.114:0/4218522411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceb4102780 0x7fceb4102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.380+0000 7fceb8d20700 1 -- 172.21.15.114:0/4218522411 >> 172.21.15.114:0/4218522411 conn(0x7fceb40fe4d0 msgr2=0x7fceb4100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:06.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.380+0000 7fceb8d20700 1 -- 172.21.15.114:0/4218522411 shutdown_connections 2024-05-27T07:37:06.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.380+0000 7fceb8d20700 1 -- 172.21.15.114:0/4218522411 wait complete. 2024-05-27T07:37:06.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.381+0000 7fceb8d20700 1 Processor -- start 2024-05-27T07:37:06.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.381+0000 7fceb8d20700 1 -- start start 2024-05-27T07:37:06.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.381+0000 7fceb8d20700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceb4102780 0x7fceb419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.381+0000 7fceb8d20700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fceb41030d0 0x7fceb419b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.381+0000 7fceb8d20700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceb41092f0 0x7fceb419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.381+0000 7fceb8d20700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fceb4114590 con 0x7fceb41030d0 2024-05-27T07:37:06.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.381+0000 7fceb8d20700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fceb4114410 con 0x7fceb4102780 2024-05-27T07:37:06.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.381+0000 7fceb8d20700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fceb4114710 con 0x7fceb41092f0 2024-05-27T07:37:06.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.381+0000 7fceb1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fceb41030d0 0x7fceb419b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.381+0000 7fceb1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fceb41030d0 0x7fceb419b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32868/0 (socket says 172.21.15.114:32868) 2024-05-27T07:37:06.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.381+0000 7fceb1d9b700 1 -- 172.21.15.114:0/2730507831 learned_addr learned my addr 172.21.15.114:0/2730507831 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:06.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.381+0000 7fceb259c700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceb4102780 0x7fceb419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.381+0000 7fceb2d9d700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceb41092f0 0x7fceb419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.382+0000 7fceb1d9b700 1 -- 172.21.15.114:0/2730507831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceb41092f0 msgr2=0x7fceb419f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:06.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.382+0000 7fceb1d9b700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceb41092f0 0x7fceb419f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.382+0000 7fceb1d9b700 1 -- 172.21.15.114:0/2730507831 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceb4102780 msgr2=0x7fceb419ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:06.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.382+0000 7fceb1d9b700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceb4102780 0x7fceb419ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.382+0000 7fceb1d9b700 1 -- 172.21.15.114:0/2730507831 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb419fc20 con 0x7fceb41030d0 2024-05-27T07:37:06.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.382+0000 7fceb1d9b700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fceb41030d0 0x7fceb419b3a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcea800c9e0 tx=0x7fcea800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:06.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.382+0000 7fcea37fe700 1 -- 172.21.15.114:0/2730507831 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8007aa0 con 0x7fceb41030d0 2024-05-27T07:37:06.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.382+0000 7fceb8d20700 1 -- 172.21.15.114:0/2730507831 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceb419ff10 con 0x7fceb41030d0 2024-05-27T07:37:06.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.382+0000 7fcea37fe700 1 -- 172.21.15.114:0/2730507831 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcea8007c40 con 0x7fceb41030d0 2024-05-27T07:37:06.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.382+0000 7fcea37fe700 1 -- 172.21.15.114:0/2730507831 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea800fe60 con 0x7fceb41030d0 2024-05-27T07:37:06.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.382+0000 7fceb8d20700 1 -- 172.21.15.114:0/2730507831 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb41a0450 con 0x7fceb41030d0 2024-05-27T07:37:06.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.383+0000 7fceb8d20700 1 -- 172.21.15.114:0/2730507831 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceb40512b0 con 0x7fceb41030d0 2024-05-27T07:37:06.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.385+0000 7fcea37fe700 1 -- 172.21.15.114:0/2730507831 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fcea800f450 con 0x7fceb41030d0 2024-05-27T07:37:06.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.386+0000 7fcea37fe700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fce9c0b2880 0x7fce9c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.386+0000 7fceb259c700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fce9c0b2880 0x7fce9c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.386+0000 7fcea37fe700 1 -- 172.21.15.114:0/2730507831 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7fcea80e2740 con 0x7fceb41030d0 2024-05-27T07:37:06.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.386+0000 7fceb259c700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fce9c0b2880 0x7fce9c0b4d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fceb406ca40 tx=0x7fcea400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:06.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.389+0000 7fcea37fe700 1 -- 172.21.15.114:0/2730507831 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcea8014030 con 0x7fceb41030d0 2024-05-27T07:37:06.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:06.643+0000 7fceb8d20700 1 -- 172.21.15.114:0/2730507831 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fceb402e5e0 con 0x7fceb41030d0 2024-05-27T07:37:06.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:06 smithi114 ceph-mon[32038]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 85 B/s wr, 0 op/s 2024-05-27T07:37:06.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:06 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:06.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:06 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:06.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:06 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:37:06.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:06 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:37:06.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:06 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:06.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:06 smithi163 ceph-mon[36451]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 85 B/s wr, 0 op/s 2024-05-27T07:37:06.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:06 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:06.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:06 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:06.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:06 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:37:06.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:06 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:37:06.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:06 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:06.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:06 smithi186 ceph-mon[36647]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 85 B/s wr, 0 op/s 2024-05-27T07:37:06.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:06 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:06.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:06 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:06.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:06 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:37:06.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:06 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:37:06.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:06 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:37:07.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:07.398+0000 7fcea37fe700 1 -- 172.21.15.114:0/2730507831 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7fcea80a5070 con 0x7fceb41030d0 2024-05-27T07:37:07.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:07.401+0000 7fceb8d20700 1 -- 172.21.15.114:0/2730507831 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fce9c0b2880 msgr2=0x7fce9c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:07.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:07.401+0000 7fceb8d20700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fce9c0b2880 0x7fce9c0b4d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fceb406ca40 tx=0x7fcea400a380 comp rx=0 tx=0).stop 2024-05-27T07:37:07.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:07.401+0000 7fceb8d20700 1 -- 172.21.15.114:0/2730507831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fceb41030d0 msgr2=0x7fceb419b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:07.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:07.401+0000 7fceb8d20700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fceb41030d0 0x7fceb419b3a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcea800c9e0 tx=0x7fcea800ccf0 comp rx=0 tx=0).stop 2024-05-27T07:37:07.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:07.401+0000 7fceb8d20700 1 -- 172.21.15.114:0/2730507831 shutdown_connections 2024-05-27T07:37:07.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:07.401+0000 7fceb8d20700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fce9c0b2880 0x7fce9c0b4d90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:07.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:07.401+0000 7fceb8d20700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceb41092f0 0x7fceb419f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:07.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:07.401+0000 7fceb8d20700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceb4102780 0x7fceb419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:07.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:07.401+0000 7fceb8d20700 1 --2- 172.21.15.114:0/2730507831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fceb41030d0 0x7fceb419b3a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:07.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:07.401+0000 7fceb8d20700 1 -- 172.21.15.114:0/2730507831 >> 172.21.15.114:0/2730507831 conn(0x7fceb40fe4d0 msgr2=0x7fceb4106fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:07.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:07.401+0000 7fceb8d20700 1 -- 172.21.15.114:0/2730507831 shutdown_connections 2024-05-27T07:37:07.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:07.401+0000 7fceb8d20700 1 -- 172.21.15.114:0/2730507831 wait complete. 2024-05-27T07:37:07.713 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:07 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2730507831' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T07:37:07.721 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:07 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2730507831' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T07:37:07.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:07 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2730507831' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T07:37:08.074 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-05-27T07:37:08.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-05-27T07:37:08.320 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:08.701 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:08 smithi114 ceph-mon[32038]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 85 B/s wr, 1 op/s 2024-05-27T07:37:08.701 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:08 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2730507831' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T07:37:08.701 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:08 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:37:08.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:08 smithi163 ceph-mon[36451]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 85 B/s wr, 1 op/s 2024-05-27T07:37:08.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:08 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2730507831' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T07:37:08.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:08 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:37:08.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:08 smithi186 ceph-mon[36647]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 85 B/s wr, 1 op/s 2024-05-27T07:37:08.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:08 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2730507831' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T07:37:08.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:08 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:37:09.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.962+0000 7f03a97f8700 1 -- 172.21.15.114:0/1889112859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03a41034b0 msgr2=0x7f03a41038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:09.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.962+0000 7f03a97f8700 1 --2- 172.21.15.114:0/1889112859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03a41034b0 0x7f03a41038c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f03940099c0 tx=0x7f0394009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.962+0000 7f03a97f8700 1 -- 172.21.15.114:0/1889112859 shutdown_connections 2024-05-27T07:37:09.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.962+0000 7f03a97f8700 1 --2- 172.21.15.114:0/1889112859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03a4108730 0x7f03a4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.962+0000 7f03a97f8700 1 --2- 172.21.15.114:0/1889112859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03a41034b0 0x7f03a41038c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.962+0000 7f03a97f8700 1 --2- 172.21.15.114:0/1889112859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03a4107de0 0x7f03a41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.962+0000 7f03a97f8700 1 -- 172.21.15.114:0/1889112859 >> 172.21.15.114:0/1889112859 conn(0x7f03a40fedb0 msgr2=0x7f03a41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:09.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.962+0000 7f03a97f8700 1 -- 172.21.15.114:0/1889112859 shutdown_connections 2024-05-27T07:37:09.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.962+0000 7f03a97f8700 1 -- 172.21.15.114:0/1889112859 wait complete. 2024-05-27T07:37:09.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.963+0000 7f03a97f8700 1 Processor -- start 2024-05-27T07:37:09.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.963+0000 7f03a97f8700 1 -- start start 2024-05-27T07:37:09.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.963+0000 7f03a97f8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03a41034b0 0x7f03a419bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:09.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.963+0000 7f03a97f8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03a4107de0 0x7f03a419c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:09.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.963+0000 7f03a97f8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03a4108730 0x7f03a419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:09.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.963+0000 7f03a97f8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f03a4113450 con 0x7f03a4108730 2024-05-27T07:37:09.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.963+0000 7f03a97f8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f03a41132d0 con 0x7f03a4107de0 2024-05-27T07:37:09.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.963+0000 7f03a97f8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f03a41135d0 con 0x7f03a41034b0 2024-05-27T07:37:09.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.963+0000 7f03a37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03a4108730 0x7f03a419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:09.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.963+0000 7f03a37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03a4108730 0x7f03a419c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32892/0 (socket says 172.21.15.114:32892) 2024-05-27T07:37:09.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.963+0000 7f03a37fe700 1 -- 172.21.15.114:0/473862477 learned_addr learned my addr 172.21.15.114:0/473862477 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:09.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.963+0000 7f03a27fc700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03a4107de0 0x7f03a419c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:09.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.963+0000 7f03a2ffd700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03a41034b0 0x7f03a419bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:09.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.964+0000 7f03a37fe700 1 -- 172.21.15.114:0/473862477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03a41034b0 msgr2=0x7f03a419bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:09.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.964+0000 7f03a37fe700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03a41034b0 0x7f03a419bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.964+0000 7f03a37fe700 1 -- 172.21.15.114:0/473862477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03a4107de0 msgr2=0x7f03a419c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:09.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.964+0000 7f03a37fe700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03a4107de0 0x7f03a419c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.964+0000 7f03a37fe700 1 -- 172.21.15.114:0/473862477 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a41a7020 con 0x7f03a4108730 2024-05-27T07:37:09.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.964+0000 7f03a37fe700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03a4108730 0x7f03a419c770 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b970 tx=0x7f039c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:09.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.964+0000 7f038bfff700 1 -- 172.21.15.114:0/473862477 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c00d860 con 0x7f03a4108730 2024-05-27T07:37:09.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.964+0000 7f03a97f8700 1 -- 172.21.15.114:0/473862477 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a41a7310 con 0x7f03a4108730 2024-05-27T07:37:09.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.964+0000 7f038bfff700 1 -- 172.21.15.114:0/473862477 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f039c00e430 con 0x7f03a4108730 2024-05-27T07:37:09.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.964+0000 7f03a97f8700 1 -- 172.21.15.114:0/473862477 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a41a7800 con 0x7f03a4108730 2024-05-27T07:37:09.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.964+0000 7f038bfff700 1 -- 172.21.15.114:0/473862477 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c017660 con 0x7f03a4108730 2024-05-27T07:37:09.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.965+0000 7f03a97f8700 1 -- 172.21.15.114:0/473862477 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03a40512b0 con 0x7f03a4108730 2024-05-27T07:37:09.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.966+0000 7f038bfff700 1 -- 172.21.15.114:0/473862477 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f039c00da00 con 0x7f03a4108730 2024-05-27T07:37:09.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.967+0000 7f038bfff700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f038c0b2880 0x7f038c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:09.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.967+0000 7f03a2ffd700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f038c0b2880 0x7f038c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:09.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.967+0000 7f038bfff700 1 -- 172.21.15.114:0/473862477 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f039c0e2330 con 0x7f03a4108730 2024-05-27T07:37:09.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.968+0000 7f03a2ffd700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f038c0b2880 0x7f038c0b4d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0394009990 tx=0x7f0394000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:09.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:09.971+0000 7f038bfff700 1 -- 172.21.15.114:0/473862477 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f039c053df0 con 0x7f03a4108730 2024-05-27T07:37:10.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.205+0000 7f03a97f8700 1 -- 172.21.15.114:0/473862477 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f03a4063bf0 con 0x7f038c0b2880 2024-05-27T07:37:10.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.228+0000 7f038bfff700 1 -- 172.21.15.114:0/473862477 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f03a4063bf0 con 0x7f038c0b2880 2024-05-27T07:37:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.230+0000 7f03a97f8700 1 -- 172.21.15.114:0/473862477 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f038c0b2880 msgr2=0x7f038c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.230+0000 7f03a97f8700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f038c0b2880 0x7f038c0b4d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0394009990 tx=0x7f0394000bc0 comp rx=0 tx=0).stop 2024-05-27T07:37:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.230+0000 7f03a97f8700 1 -- 172.21.15.114:0/473862477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03a4108730 msgr2=0x7f03a419c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.230+0000 7f03a97f8700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03a4108730 0x7f03a419c770 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b970 tx=0x7f039c00bc80 comp rx=0 tx=0).stop 2024-05-27T07:37:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.230+0000 7f03a97f8700 1 -- 172.21.15.114:0/473862477 shutdown_connections 2024-05-27T07:37:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.230+0000 7f03a97f8700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03a4108730 0x7f03a419c770 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.230+0000 7f03a97f8700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f038c0b2880 0x7f038c0b4d90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.230+0000 7f03a97f8700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03a41034b0 0x7f03a419bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:10.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.230+0000 7f03a97f8700 1 --2- 172.21.15.114:0/473862477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03a4107de0 0x7f03a419c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:10.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.230+0000 7f03a97f8700 1 -- 172.21.15.114:0/473862477 >> 172.21.15.114:0/473862477 conn(0x7f03a40fedb0 msgr2=0x7f03a4109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:10.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.230+0000 7f03a97f8700 1 -- 172.21.15.114:0/473862477 shutdown_connections 2024-05-27T07:37:10.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:10.230+0000 7f03a97f8700 1 -- 172.21.15.114:0/473862477 wait complete. 2024-05-27T07:37:10.539 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:10 smithi114 ceph-mon[32038]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-27T07:37:10.539 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:37:10 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:37:10.211+0000 7f30b764a700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:37:10.539 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:37:10 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:37:10.211+0000 7f30b764a700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:37:10.539 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:37:10 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:37:10.211+0000 7f30b764a700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:37:10.540 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:37:10 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:37:10.211+0000 7f30b764a700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:37:10.540 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:37:10 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:37:10.211+0000 7f30b764a700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:37:10.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:10 smithi163 ceph-mon[36451]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-27T07:37:10.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:10 smithi186 ceph-mon[36647]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-27T07:37:10.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-27T07:37:11.136 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:11.535 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:11 smithi114 ceph-mon[32038]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:11.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:11 smithi163 ceph-mon[36451]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:11.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:11 smithi186 ceph-mon[36647]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:12.533 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:12 smithi114 ceph-mon[32038]: pgmap v331: 129 pgs: 129 active+clean; 588 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-05-27T07:37:12.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:12 smithi163 ceph-mon[36451]: pgmap v331: 129 pgs: 129 active+clean; 588 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-05-27T07:37:12.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:12 smithi186 ceph-mon[36647]: pgmap v331: 129 pgs: 129 active+clean; 588 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-05-27T07:37:12.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.780+0000 7f6ae414d700 1 -- 172.21.15.114:0/1904932494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6adc107de0 msgr2=0x7f6adc1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:12.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.780+0000 7f6ae414d700 1 --2- 172.21.15.114:0/1904932494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6adc107de0 0x7f6adc1081f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0099c0 tx=0x7f6acc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.781+0000 7f6ae414d700 1 -- 172.21.15.114:0/1904932494 shutdown_connections 2024-05-27T07:37:12.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.781+0000 7f6ae414d700 1 --2- 172.21.15.114:0/1904932494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6adc108730 0x7f6adc110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.781+0000 7f6ae414d700 1 --2- 172.21.15.114:0/1904932494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6adc1034b0 0x7f6adc1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.781+0000 7f6ae414d700 1 --2- 172.21.15.114:0/1904932494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6adc107de0 0x7f6adc1081f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.781+0000 7f6ae414d700 1 -- 172.21.15.114:0/1904932494 >> 172.21.15.114:0/1904932494 conn(0x7f6adc0fede0 msgr2=0x7f6adc101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:12.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.781+0000 7f6ae414d700 1 -- 172.21.15.114:0/1904932494 shutdown_connections 2024-05-27T07:37:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.781+0000 7f6ae414d700 1 -- 172.21.15.114:0/1904932494 wait complete. 2024-05-27T07:37:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.781+0000 7f6ae414d700 1 Processor -- start 2024-05-27T07:37:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.781+0000 7f6ae414d700 1 -- start start 2024-05-27T07:37:12.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae414d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6adc1034b0 0x7f6adc199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae414d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6adc107de0 0x7f6adc19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae414d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6adc108730 0x7f6adc19e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae414d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc113340 con 0x7f6adc107de0 2024-05-27T07:37:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae414d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc1131c0 con 0x7f6adc1034b0 2024-05-27T07:37:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae414d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc1134c0 con 0x7f6adc108730 2024-05-27T07:37:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae16e8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6adc107de0 0x7f6adc19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae16e8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6adc107de0 0x7f6adc19a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32924/0 (socket says 172.21.15.114:32924) 2024-05-27T07:37:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae16e8700 1 -- 172.21.15.114:0/2958046104 learned_addr learned my addr 172.21.15.114:0/2958046104 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae1ee9700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6adc1034b0 0x7f6adc199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae26ea700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6adc108730 0x7f6adc19e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae16e8700 1 -- 172.21.15.114:0/2958046104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6adc108730 msgr2=0x7f6adc19e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae16e8700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6adc108730 0x7f6adc19e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae16e8700 1 -- 172.21.15.114:0/2958046104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6adc1034b0 msgr2=0x7f6adc199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae16e8700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6adc1034b0 0x7f6adc199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae16e8700 1 -- 172.21.15.114:0/2958046104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6adc19eb50 con 0x7f6adc107de0 2024-05-27T07:37:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae26ea700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6adc108730 0x7f6adc19e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:37:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae16e8700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6adc107de0 0x7f6adc19a2d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6acc006010 tx=0x7f6acc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ad2ffd700 1 -- 172.21.15.114:0/2958046104 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc027070 con 0x7f6adc107de0 2024-05-27T07:37:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ad2ffd700 1 -- 172.21.15.114:0/2958046104 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6acc003f20 con 0x7f6adc107de0 2024-05-27T07:37:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.782+0000 7f6ae414d700 1 -- 172.21.15.114:0/2958046104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6adc19ede0 con 0x7f6adc107de0 2024-05-27T07:37:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.783+0000 7f6ad2ffd700 1 -- 172.21.15.114:0/2958046104 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc022cd0 con 0x7f6adc107de0 2024-05-27T07:37:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.783+0000 7f6ae414d700 1 -- 172.21.15.114:0/2958046104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6adc19f2c0 con 0x7f6adc107de0 2024-05-27T07:37:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.783+0000 7f6ae414d700 1 -- 172.21.15.114:0/2958046104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6adc0512b0 con 0x7f6adc107de0 2024-05-27T07:37:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.786+0000 7f6ad2ffd700 1 -- 172.21.15.114:0/2958046104 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f6acc02a070 con 0x7f6adc107de0 2024-05-27T07:37:12.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.787+0000 7f6ad2ffd700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f6ac80b2910 0x7f6ac80b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.787+0000 7f6ae1ee9700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f6ac80b2910 0x7f6ac80b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.787+0000 7f6ad2ffd700 1 -- 172.21.15.114:0/2958046104 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f6acc0e3990 con 0x7f6adc107de0 2024-05-27T07:37:12.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.787+0000 7f6ae1ee9700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f6ac80b2910 0x7f6ac80b4e20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800ac50 tx=0x7f6ad800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:12.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:12.791+0000 7f6ad2ffd700 1 -- 172.21.15.114:0/2958046104 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6acc031080 con 0x7f6adc107de0 2024-05-27T07:37:13.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.026+0000 7f6ae414d700 1 -- 172.21.15.114:0/2958046104 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f6adc19b090 con 0x7f6ac80b2910 2024-05-27T07:37:13.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.059+0000 7f6ad2ffd700 1 -- 172.21.15.114:0/2958046104 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6adc19b090 con 0x7f6ac80b2910 2024-05-27T07:37:13.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.061+0000 7f6ae414d700 1 -- 172.21.15.114:0/2958046104 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f6ac80b2910 msgr2=0x7f6ac80b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:13.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.061+0000 7f6ae414d700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f6ac80b2910 0x7f6ac80b4e20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800ac50 tx=0x7f6ad800a380 comp rx=0 tx=0).stop 2024-05-27T07:37:13.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.061+0000 7f6ae414d700 1 -- 172.21.15.114:0/2958046104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6adc107de0 msgr2=0x7f6adc19a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:13.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.061+0000 7f6ae414d700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6adc107de0 0x7f6adc19a2d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6acc006010 tx=0x7f6acc0047b0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.061+0000 7f6ae414d700 1 -- 172.21.15.114:0/2958046104 shutdown_connections 2024-05-27T07:37:13.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.061+0000 7f6ae414d700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f6ac80b2910 0x7f6ac80b4e20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.061+0000 7f6ae414d700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6adc108730 0x7f6adc19e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.061+0000 7f6ae414d700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6adc1034b0 0x7f6adc199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.061+0000 7f6ae414d700 1 --2- 172.21.15.114:0/2958046104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6adc107de0 0x7f6adc19a2d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.061+0000 7f6ae414d700 1 -- 172.21.15.114:0/2958046104 >> 172.21.15.114:0/2958046104 conn(0x7f6adc0fede0 msgr2=0x7f6adc101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:13.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.061+0000 7f6ae414d700 1 -- 172.21.15.114:0/2958046104 shutdown_connections 2024-05-27T07:37:13.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:13.061+0000 7f6ae414d700 1 -- 172.21.15.114:0/2958046104 wait complete. 2024-05-27T07:37:13.698 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:13 smithi114 ceph-mon[32038]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-05-27T07:37:13.699 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:13 smithi114 ceph-mon[32038]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:13.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:13 smithi163 ceph-mon[36451]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-05-27T07:37:13.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:13 smithi163 ceph-mon[36451]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:13.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:13 smithi186 ceph-mon[36647]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-05-27T07:37:13.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:13 smithi186 ceph-mon[36647]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:13.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-27T07:37:13.975 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:14.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:14 smithi163 ceph-mon[36451]: pgmap v332: 129 pgs: 129 active+clean; 588 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 938 B/s wr, 2 op/s 2024-05-27T07:37:14.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:14 smithi186 ceph-mon[36647]: pgmap v332: 129 pgs: 129 active+clean; 588 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 938 B/s wr, 2 op/s 2024-05-27T07:37:14.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:14 smithi114 ceph-mon[32038]: pgmap v332: 129 pgs: 129 active+clean; 588 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 938 B/s wr, 2 op/s 2024-05-27T07:37:15.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.666+0000 7f399fb1e700 1 -- 172.21.15.114:0/4228606285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998102b10 msgr2=0x7f3998102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:15.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.666+0000 7f399fb1e700 1 --2- 172.21.15.114:0/4228606285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998102b10 0x7f3998102f20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f39940099c0 tx=0x7f3994009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:15.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.666+0000 7f399fb1e700 1 -- 172.21.15.114:0/4228606285 shutdown_connections 2024-05-27T07:37:15.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.666+0000 7f399fb1e700 1 --2- 172.21.15.114:0/4228606285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3998109880 0x7f39981111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:15.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.666+0000 7f399fb1e700 1 --2- 172.21.15.114:0/4228606285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998102b10 0x7f3998102f20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.666+0000 7f399fb1e700 1 --2- 172.21.15.114:0/4228606285 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3998108f30 0x7f3998109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.666+0000 7f399fb1e700 1 -- 172.21.15.114:0/4228606285 >> 172.21.15.114:0/4228606285 conn(0x7f39980fe4d0 msgr2=0x7f3998100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.666+0000 7f399fb1e700 1 -- 172.21.15.114:0/4228606285 shutdown_connections 2024-05-27T07:37:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.666+0000 7f399fb1e700 1 -- 172.21.15.114:0/4228606285 wait complete. 2024-05-27T07:37:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.667+0000 7f399fb1e700 1 Processor -- start 2024-05-27T07:37:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.667+0000 7f399fb1e700 1 -- start start 2024-05-27T07:37:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.667+0000 7f399fb1e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3998102b10 0x7f399819d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.667+0000 7f399fb1e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3998108f30 0x7f399819d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.667+0000 7f399fb1e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998109880 0x7f39981a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.667+0000 7f399fb1e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f39981144d0 con 0x7f3998109880 2024-05-27T07:37:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.667+0000 7f399fb1e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3998114350 con 0x7f3998102b10 2024-05-27T07:37:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.667+0000 7f399fb1e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3998114650 con 0x7f3998108f30 2024-05-27T07:37:15.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.667+0000 7f399e0bb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998109880 0x7f39981a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:15.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.667+0000 7f399e0bb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998109880 0x7f39981a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38726/0 (socket says 172.21.15.114:38726) 2024-05-27T07:37:15.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f399e0bb700 1 -- 172.21.15.114:0/1790604602 learned_addr learned my addr 172.21.15.114:0/1790604602 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:15.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f399d0b9700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3998108f30 0x7f399819d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:15.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f399d8ba700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3998102b10 0x7f399819d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:15.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f399e0bb700 1 -- 172.21.15.114:0/1790604602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3998108f30 msgr2=0x7f399819d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:15.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f399e0bb700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3998108f30 0x7f399819d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:15.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f399e0bb700 1 -- 172.21.15.114:0/1790604602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3998102b10 msgr2=0x7f399819d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:15.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f399e0bb700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3998102b10 0x7f399819d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:15.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f399e0bb700 1 -- 172.21.15.114:0/1790604602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39981a1ee0 con 0x7f3998109880 2024-05-27T07:37:15.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f399e0bb700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998109880 0x7f39981a18f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3990007d70 tx=0x7f399000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:15.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f398affd700 1 -- 172.21.15.114:0/1790604602 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3990016020 con 0x7f3998109880 2024-05-27T07:37:15.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f399fb1e700 1 -- 172.21.15.114:0/1790604602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39981a21d0 con 0x7f3998109880 2024-05-27T07:37:15.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f398affd700 1 -- 172.21.15.114:0/1790604602 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f399000ac90 con 0x7f3998109880 2024-05-27T07:37:15.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f398affd700 1 -- 172.21.15.114:0/1790604602 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39900115e0 con 0x7f3998109880 2024-05-27T07:37:15.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.668+0000 7f399fb1e700 1 -- 172.21.15.114:0/1790604602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39981a2710 con 0x7f3998109880 2024-05-27T07:37:15.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.669+0000 7f399fb1e700 1 -- 172.21.15.114:0/1790604602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39980512b0 con 0x7f3998109880 2024-05-27T07:37:15.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.671+0000 7f398affd700 1 -- 172.21.15.114:0/1790604602 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f3990019020 con 0x7f3998109880 2024-05-27T07:37:15.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.672+0000 7f398affd700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f39840b2910 0x7f39840b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:15.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.672+0000 7f399d8ba700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f39840b2910 0x7f39840b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:15.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.672+0000 7f398affd700 1 -- 172.21.15.114:0/1790604602 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f39900e29d0 con 0x7f3998109880 2024-05-27T07:37:15.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.672+0000 7f399d8ba700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f39840b2910 0x7f39840b4e20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3994009f50 tx=0x7f3994000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:15.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.676+0000 7f398affd700 1 -- 172.21.15.114:0/1790604602 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3990020020 con 0x7f3998109880 2024-05-27T07:37:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.913+0000 7f399fb1e700 1 -- 172.21.15.114:0/1790604602 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f39981049a0 con 0x7f39840b2910 2024-05-27T07:37:15.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.919+0000 7f398affd700 1 -- 172.21.15.114:0/1790604602 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f39981049a0 con 0x7f39840b2910 2024-05-27T07:37:15.916 INFO:teuthology.orchestra.run.smithi114.stdout:/volumes/qa/sv_0/187a044b-245b-4f95-b189-3a66efbf32de 2024-05-27T07:37:15.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.922+0000 7f399fb1e700 1 -- 172.21.15.114:0/1790604602 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f39840b2910 msgr2=0x7f39840b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:15.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.922+0000 7f399fb1e700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f39840b2910 0x7f39840b4e20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3994009f50 tx=0x7f3994000bc0 comp rx=0 tx=0).stop 2024-05-27T07:37:15.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.922+0000 7f399fb1e700 1 -- 172.21.15.114:0/1790604602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998109880 msgr2=0x7f39981a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:15.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.922+0000 7f399fb1e700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998109880 0x7f39981a18f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3990007d70 tx=0x7f399000a340 comp rx=0 tx=0).stop 2024-05-27T07:37:15.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.922+0000 7f399fb1e700 1 -- 172.21.15.114:0/1790604602 shutdown_connections 2024-05-27T07:37:15.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.922+0000 7f399fb1e700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998109880 0x7f39981a18f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:15.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.922+0000 7f399fb1e700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3998102b10 0x7f399819d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:15.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.922+0000 7f399fb1e700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f39840b2910 0x7f39840b4e20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:15.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.922+0000 7f399fb1e700 1 --2- 172.21.15.114:0/1790604602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3998108f30 0x7f399819d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:15.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.922+0000 7f399fb1e700 1 -- 172.21.15.114:0/1790604602 >> 172.21.15.114:0/1790604602 conn(0x7f39980fe4d0 msgr2=0x7f3998108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:15.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.922+0000 7f399fb1e700 1 -- 172.21.15.114:0/1790604602 shutdown_connections 2024-05-27T07:37:15.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:15.922+0000 7f399fb1e700 1 -- 172.21.15.114:0/1790604602 wait complete. 2024-05-27T07:37:16.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-27T07:37:16.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:16 smithi163 ceph-mon[36451]: pgmap v333: 129 pgs: 129 active+clean; 588 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 938 B/s wr, 2 op/s 2024-05-27T07:37:16.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:16 smithi163 ceph-mon[36451]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:16.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:16 smithi186 ceph-mon[36647]: pgmap v333: 129 pgs: 129 active+clean; 588 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 938 B/s wr, 2 op/s 2024-05-27T07:37:16.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:16 smithi186 ceph-mon[36647]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:16.779 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:16.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:16 smithi114 ceph-mon[32038]: pgmap v333: 129 pgs: 129 active+clean; 588 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 938 B/s wr, 2 op/s 2024-05-27T07:37:16.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:16 smithi114 ceph-mon[32038]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:18.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.450+0000 7f8d2fe87700 1 -- 172.21.15.114:0/4279295432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28109350 msgr2=0x7f8d28109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:18.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.450+0000 7f8d2fe87700 1 --2- 172.21.15.114:0/4279295432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28109350 0x7f8d28109760 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8d24009990 tx=0x7f8d24009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.450+0000 7f8d2fe87700 1 -- 172.21.15.114:0/4279295432 shutdown_connections 2024-05-27T07:37:18.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.450+0000 7f8d2fe87700 1 --2- 172.21.15.114:0/4279295432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d28104280 0x7f8d281100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.450+0000 7f8d2fe87700 1 --2- 172.21.15.114:0/4279295432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28109350 0x7f8d28109760 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.450+0000 7f8d2fe87700 1 --2- 172.21.15.114:0/4279295432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d28103930 0x7f8d28103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.450+0000 7f8d2fe87700 1 -- 172.21.15.114:0/4279295432 >> 172.21.15.114:0/4279295432 conn(0x7f8d280fe4d0 msgr2=0x7f8d28100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:18.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.450+0000 7f8d2fe87700 1 -- 172.21.15.114:0/4279295432 shutdown_connections 2024-05-27T07:37:18.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.450+0000 7f8d2fe87700 1 -- 172.21.15.114:0/4279295432 wait complete. 2024-05-27T07:37:18.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:18 smithi114 ceph-mon[32038]: pgmap v334: 129 pgs: 129 active+clean; 601 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-05-27T07:37:18.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.451+0000 7f8d2fe87700 1 Processor -- start 2024-05-27T07:37:18.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.451+0000 7f8d2fe87700 1 -- start start 2024-05-27T07:37:18.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.451+0000 7f8d2fe87700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d28103930 0x7f8d2819be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:18.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2fe87700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d28104280 0x7f8d2819c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:18.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2fe87700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28109350 0x7f8d2819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:18.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2fe87700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8d28113490 con 0x7f8d28109350 2024-05-27T07:37:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2fe87700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8d28113310 con 0x7f8d28103930 2024-05-27T07:37:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2fe87700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8d28113610 con 0x7f8d28104280 2024-05-27T07:37:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2e424700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28109350 0x7f8d2819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2e424700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28109350 0x7f8d2819c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38758/0 (socket says 172.21.15.114:38758) 2024-05-27T07:37:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2e424700 1 -- 172.21.15.114:0/1104085249 learned_addr learned my addr 172.21.15.114:0/1104085249 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2d422700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d28104280 0x7f8d2819c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2dc23700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d28103930 0x7f8d2819be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2e424700 1 -- 172.21.15.114:0/1104085249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d28104280 msgr2=0x7f8d2819c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2e424700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d28104280 0x7f8d2819c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2e424700 1 -- 172.21.15.114:0/1104085249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d28103930 msgr2=0x7f8d2819be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2e424700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d28103930 0x7f8d2819be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2e424700 1 -- 172.21.15.114:0/1104085249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d281a7180 con 0x7f8d28109350 2024-05-27T07:37:18.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d2e424700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28109350 0x7f8d2819c8d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000ada0 tx=0x7f8d2000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:18.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.452+0000 7f8d1affd700 1 -- 172.21.15.114:0/1104085249 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20017020 con 0x7f8d28109350 2024-05-27T07:37:18.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.453+0000 7f8d1affd700 1 -- 172.21.15.114:0/1104085249 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d2000cd80 con 0x7f8d28109350 2024-05-27T07:37:18.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.453+0000 7f8d2fe87700 1 -- 172.21.15.114:0/1104085249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d281a7470 con 0x7f8d28109350 2024-05-27T07:37:18.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.453+0000 7f8d1affd700 1 -- 172.21.15.114:0/1104085249 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20005260 con 0x7f8d28109350 2024-05-27T07:37:18.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.453+0000 7f8d2fe87700 1 -- 172.21.15.114:0/1104085249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d281a79b0 con 0x7f8d28109350 2024-05-27T07:37:18.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.454+0000 7f8d2fe87700 1 -- 172.21.15.114:0/1104085249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d280512b0 con 0x7f8d28109350 2024-05-27T07:37:18.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.455+0000 7f8d1affd700 1 -- 172.21.15.114:0/1104085249 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f8d2000a090 con 0x7f8d28109350 2024-05-27T07:37:18.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.456+0000 7f8d1affd700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8d140b2990 0x7f8d140b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:18.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.456+0000 7f8d2dc23700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8d140b2990 0x7f8d140b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:18.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.456+0000 7f8d1affd700 1 -- 172.21.15.114:0/1104085249 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f8d200e22c0 con 0x7f8d28109350 2024-05-27T07:37:18.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.457+0000 7f8d2dc23700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8d140b2990 0x7f8d140b4ea0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400b5c0 tx=0x7f8d24005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:18.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.460+0000 7f8d1affd700 1 -- 172.21.15.114:0/1104085249 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8d20021020 con 0x7f8d28109350 2024-05-27T07:37:18.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.697+0000 7f8d2fe87700 1 -- 172.21.15.114:0/1104085249 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f8d28102e30 con 0x7f8d140b2990 2024-05-27T07:37:18.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.725+0000 7f8d1affd700 1 -- 172.21.15.114:0/1104085249 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8d28102e30 con 0x7f8d140b2990 2024-05-27T07:37:18.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:18 smithi163 ceph-mon[36451]: pgmap v334: 129 pgs: 129 active+clean; 601 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-05-27T07:37:18.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.727+0000 7f8d2fe87700 1 -- 172.21.15.114:0/1104085249 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8d140b2990 msgr2=0x7f8d140b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:18.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.727+0000 7f8d2fe87700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8d140b2990 0x7f8d140b4ea0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400b5c0 tx=0x7f8d24005960 comp rx=0 tx=0).stop 2024-05-27T07:37:18.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.727+0000 7f8d2fe87700 1 -- 172.21.15.114:0/1104085249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28109350 msgr2=0x7f8d2819c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:18.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.727+0000 7f8d2fe87700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28109350 0x7f8d2819c8d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000ada0 tx=0x7f8d2000c330 comp rx=0 tx=0).stop 2024-05-27T07:37:18.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.727+0000 7f8d2fe87700 1 -- 172.21.15.114:0/1104085249 shutdown_connections 2024-05-27T07:37:18.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.727+0000 7f8d2fe87700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28109350 0x7f8d2819c8d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.727+0000 7f8d2fe87700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d28103930 0x7f8d2819be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.727+0000 7f8d2fe87700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8d140b2990 0x7f8d140b4ea0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.727+0000 7f8d2fe87700 1 --2- 172.21.15.114:0/1104085249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d28104280 0x7f8d2819c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.727+0000 7f8d2fe87700 1 -- 172.21.15.114:0/1104085249 >> 172.21.15.114:0/1104085249 conn(0x7f8d280fe4d0 msgr2=0x7f8d28100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:18.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.727+0000 7f8d2fe87700 1 -- 172.21.15.114:0/1104085249 shutdown_connections 2024-05-27T07:37:18.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:18.727+0000 7f8d2fe87700 1 -- 172.21.15.114:0/1104085249 wait complete. 2024-05-27T07:37:18.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:18 smithi186 ceph-mon[36647]: pgmap v334: 129 pgs: 129 active+clean; 601 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-05-27T07:37:19.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-27T07:37:19.644 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:19.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:19 smithi163 ceph-mon[36451]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:19.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:19 smithi186 ceph-mon[36647]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:19.766 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:19 smithi114 ceph-mon[32038]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:20.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:20 smithi163 ceph-mon[36451]: pgmap v335: 129 pgs: 129 active+clean; 602 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-05-27T07:37:20.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:20 smithi186 ceph-mon[36647]: pgmap v335: 129 pgs: 129 active+clean; 602 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-05-27T07:37:20.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:20 smithi114 ceph-mon[32038]: pgmap v335: 129 pgs: 129 active+clean; 602 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-05-27T07:37:21.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.283+0000 7f43f6357700 1 -- 172.21.15.114:0/1413312234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f0107de0 msgr2=0x7f43f01081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:21.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.283+0000 7f43f6357700 1 --2- 172.21.15.114:0/1413312234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f0107de0 0x7f43f01081f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f43d80099c0 tx=0x7f43d8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.283+0000 7f43f6357700 1 -- 172.21.15.114:0/1413312234 shutdown_connections 2024-05-27T07:37:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.283+0000 7f43f6357700 1 --2- 172.21.15.114:0/1413312234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f0108730 0x7f43f0110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.283+0000 7f43f6357700 1 --2- 172.21.15.114:0/1413312234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43f01034b0 0x7f43f01038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.283+0000 7f43f6357700 1 --2- 172.21.15.114:0/1413312234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f0107de0 0x7f43f01081f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.283+0000 7f43f6357700 1 -- 172.21.15.114:0/1413312234 >> 172.21.15.114:0/1413312234 conn(0x7f43f00fedb0 msgr2=0x7f43f01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.284+0000 7f43f6357700 1 -- 172.21.15.114:0/1413312234 shutdown_connections 2024-05-27T07:37:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.284+0000 7f43f6357700 1 -- 172.21.15.114:0/1413312234 wait complete. 2024-05-27T07:37:21.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.284+0000 7f43f6357700 1 Processor -- start 2024-05-27T07:37:21.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.284+0000 7f43f6357700 1 -- start start 2024-05-27T07:37:21.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43f6357700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43f01034b0 0x7f43f019bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:21.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43f6357700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f0107de0 0x7f43f019c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:21.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43f6357700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f0108730 0x7f43f019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:21.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43f6357700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f43f0113340 con 0x7f43f0107de0 2024-05-27T07:37:21.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43f6357700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f43f01131c0 con 0x7f43f0108730 2024-05-27T07:37:21.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43f6357700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f43f01134c0 con 0x7f43f01034b0 2024-05-27T07:37:21.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43e7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f0107de0 0x7f43f019c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:21.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43e7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f0107de0 0x7f43f019c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38766/0 (socket says 172.21.15.114:38766) 2024-05-27T07:37:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43e7fff700 1 -- 172.21.15.114:0/1287786591 learned_addr learned my addr 172.21.15.114:0/1287786591 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43f48f4700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f0108730 0x7f43f019c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43effff700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43f01034b0 0x7f43f019bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43e7fff700 1 -- 172.21.15.114:0/1287786591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43f01034b0 msgr2=0x7f43f019bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43e7fff700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43f01034b0 0x7f43f019bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43e7fff700 1 -- 172.21.15.114:0/1287786591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f0108730 msgr2=0x7f43f019c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43e7fff700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f0108730 0x7f43f019c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43e7fff700 1 -- 172.21.15.114:0/1287786591 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f01a6ff0 con 0x7f43f0107de0 2024-05-27T07:37:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43effff700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43f01034b0 0x7f43f019bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:37:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43e7fff700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f0107de0 0x7f43f019c1c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f43d8006010 tx=0x7f43d80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43edffb700 1 -- 172.21.15.114:0/1287786591 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8027070 con 0x7f43f0107de0 2024-05-27T07:37:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43f6357700 1 -- 172.21.15.114:0/1287786591 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f01a7280 con 0x7f43f0107de0 2024-05-27T07:37:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43edffb700 1 -- 172.21.15.114:0/1287786591 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43d8003fa0 con 0x7f43f0107de0 2024-05-27T07:37:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.285+0000 7f43edffb700 1 -- 172.21.15.114:0/1287786591 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8022dd0 con 0x7f43f0107de0 2024-05-27T07:37:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.286+0000 7f43f6357700 1 -- 172.21.15.114:0/1287786591 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f01a7730 con 0x7f43f0107de0 2024-05-27T07:37:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.286+0000 7f43f6357700 1 -- 172.21.15.114:0/1287786591 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43f00512b0 con 0x7f43f0107de0 2024-05-27T07:37:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.288+0000 7f43edffb700 1 -- 172.21.15.114:0/1287786591 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f43d802a070 con 0x7f43f0107de0 2024-05-27T07:37:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.289+0000 7f43edffb700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f43dc0b2990 0x7f43dc0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.289+0000 7f43effff700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f43dc0b2990 0x7f43dc0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:21.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.289+0000 7f43edffb700 1 -- 172.21.15.114:0/1287786591 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f43d80e34f0 con 0x7f43f0107de0 2024-05-27T07:37:21.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.289+0000 7f43effff700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f43dc0b2990 0x7f43dc0b4ea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f43e0005fd0 tx=0x7f43e0005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:21.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.292+0000 7f43edffb700 1 -- 172.21.15.114:0/1287786591 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f43d8013070 con 0x7f43f0107de0 2024-05-27T07:37:21.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.532+0000 7f43f6357700 1 -- 172.21.15.114:0/1287786591 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f43f0063bf0 con 0x7f43dc0b2990 2024-05-27T07:37:21.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.537+0000 7f43edffb700 1 -- 172.21.15.114:0/1287786591 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f43f0063bf0 con 0x7f43dc0b2990 2024-05-27T07:37:21.533 INFO:teuthology.orchestra.run.smithi114.stdout:/volumes/qa/sv_1/ce3cee47-a2d9-4102-a357-fb849184a794 2024-05-27T07:37:21.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.539+0000 7f43f6357700 1 -- 172.21.15.114:0/1287786591 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f43dc0b2990 msgr2=0x7f43dc0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:21.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.539+0000 7f43f6357700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f43dc0b2990 0x7f43dc0b4ea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f43e0005fd0 tx=0x7f43e0005ee0 comp rx=0 tx=0).stop 2024-05-27T07:37:21.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.539+0000 7f43f6357700 1 -- 172.21.15.114:0/1287786591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f0107de0 msgr2=0x7f43f019c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:21.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.539+0000 7f43f6357700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f0107de0 0x7f43f019c1c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f43d8006010 tx=0x7f43d80047b0 comp rx=0 tx=0).stop 2024-05-27T07:37:21.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.539+0000 7f43f6357700 1 -- 172.21.15.114:0/1287786591 shutdown_connections 2024-05-27T07:37:21.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.539+0000 7f43f6357700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f0108730 0x7f43f019c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:21.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.539+0000 7f43f6357700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f43dc0b2990 0x7f43dc0b4ea0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:21.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.539+0000 7f43f6357700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43f01034b0 0x7f43f019bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:21.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.539+0000 7f43f6357700 1 --2- 172.21.15.114:0/1287786591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f0107de0 0x7f43f019c1c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:21.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.539+0000 7f43f6357700 1 -- 172.21.15.114:0/1287786591 >> 172.21.15.114:0/1287786591 conn(0x7f43f00fedb0 msgr2=0x7f43f01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:21.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.539+0000 7f43f6357700 1 -- 172.21.15.114:0/1287786591 shutdown_connections 2024-05-27T07:37:21.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:21.539+0000 7f43f6357700 1 -- 172.21.15.114:0/1287786591 wait complete. 2024-05-27T07:37:22.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:37:22.518 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:22.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:22 smithi114 ceph-mon[32038]: pgmap v336: 129 pgs: 129 active+clean; 621 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-05-27T07:37:22.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:22 smithi114 ceph-mon[32038]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:22.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:22 smithi163 ceph-mon[36451]: pgmap v336: 129 pgs: 129 active+clean; 621 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-05-27T07:37:22.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:22 smithi163 ceph-mon[36451]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:22.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:22 smithi186 ceph-mon[36647]: pgmap v336: 129 pgs: 129 active+clean; 621 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-05-27T07:37:22.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:22 smithi186 ceph-mon[36647]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:37:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.144+0000 7f793d28d700 1 -- 172.21.15.114:0/2953084474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7938107de0 msgr2=0x7f79381081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.144+0000 7f793d28d700 1 --2- 172.21.15.114:0/2953084474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7938107de0 0x7f79381081f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f79200099c0 tx=0x7f7920009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.144+0000 7f793d28d700 1 -- 172.21.15.114:0/2953084474 shutdown_connections 2024-05-27T07:37:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.144+0000 7f793d28d700 1 --2- 172.21.15.114:0/2953084474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7938108730 0x7f7938110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.144+0000 7f793d28d700 1 --2- 172.21.15.114:0/2953084474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79381034b0 0x7f79381038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.144+0000 7f793d28d700 1 --2- 172.21.15.114:0/2953084474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7938107de0 0x7f79381081f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.144+0000 7f793d28d700 1 -- 172.21.15.114:0/2953084474 >> 172.21.15.114:0/2953084474 conn(0x7f79380fedb0 msgr2=0x7f79381011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.144+0000 7f793d28d700 1 -- 172.21.15.114:0/2953084474 shutdown_connections 2024-05-27T07:37:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.144+0000 7f793d28d700 1 -- 172.21.15.114:0/2953084474 wait complete. 2024-05-27T07:37:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.145+0000 7f793d28d700 1 Processor -- start 2024-05-27T07:37:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.145+0000 7f793d28d700 1 -- start start 2024-05-27T07:37:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f793d28d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79381034b0 0x7f793819be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f793d28d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7938107de0 0x7f793819c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f793d28d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7938108730 0x7f793819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f793d28d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7938113430 con 0x7f79381034b0 2024-05-27T07:37:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f793d28d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79381132b0 con 0x7f7938108730 2024-05-27T07:37:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f793d28d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79381135b0 con 0x7f7938107de0 2024-05-27T07:37:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f7936ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79381034b0 0x7f793819be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f7936ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79381034b0 0x7f793819be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38790/0 (socket says 172.21.15.114:38790) 2024-05-27T07:37:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f7936ffd700 1 -- 172.21.15.114:0/336474121 learned_addr learned my addr 172.21.15.114:0/336474121 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f792e7fc700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7938107de0 0x7f793819c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f7936ffd700 1 -- 172.21.15.114:0/336474121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7938107de0 msgr2=0x7f793819c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f79377fe700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7938108730 0x7f793819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f7936ffd700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7938107de0 0x7f793819c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f7936ffd700 1 -- 172.21.15.114:0/336474121 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7938108730 msgr2=0x7f793819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f7936ffd700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7938108730 0x7f793819c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f7936ffd700 1 -- 172.21.15.114:0/336474121 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79381a71a0 con 0x7f79381034b0 2024-05-27T07:37:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f792e7fc700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7938107de0 0x7f793819c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:37:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f7936ffd700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79381034b0 0x7f793819be30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f792800b530 tx=0x7f792800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:24.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f7934ff9700 1 -- 172.21.15.114:0/336474121 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928015020 con 0x7f79381034b0 2024-05-27T07:37:24.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f7934ff9700 1 -- 172.21.15.114:0/336474121 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7928011ce0 con 0x7f79381034b0 2024-05-27T07:37:24.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.146+0000 7f7934ff9700 1 -- 172.21.15.114:0/336474121 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928010d20 con 0x7f79381034b0 2024-05-27T07:37:24.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.147+0000 7f793d28d700 1 -- 172.21.15.114:0/336474121 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79381a7490 con 0x7f79381034b0 2024-05-27T07:37:24.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.147+0000 7f793d28d700 1 -- 172.21.15.114:0/336474121 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79381a7950 con 0x7f79381034b0 2024-05-27T07:37:24.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.147+0000 7f793d28d700 1 -- 172.21.15.114:0/336474121 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79380512b0 con 0x7f79381034b0 2024-05-27T07:37:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.149+0000 7f7934ff9700 1 -- 172.21.15.114:0/336474121 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f7928018020 con 0x7f79381034b0 2024-05-27T07:37:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.150+0000 7f7934ff9700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f79240b2910 0x7f79240b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.150+0000 7f792e7fc700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f79240b2910 0x7f79240b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.150+0000 7f7934ff9700 1 -- 172.21.15.114:0/336474121 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f79280e2300 con 0x7f79381034b0 2024-05-27T07:37:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.150+0000 7f792e7fc700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f79240b2910 0x7f79240b4e20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7920009990 tx=0x7f792001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:24.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.153+0000 7f7934ff9700 1 -- 172.21.15.114:0/336474121 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f792801f020 con 0x7f79381034b0 2024-05-27T07:37:24.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.405+0000 7f793d28d700 1 -- 172.21.15.114:0/336474121 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f793806bc10 con 0x7f79381034b0 2024-05-27T07:37:24.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.406+0000 7f7934ff9700 1 -- 172.21.15.114:0/336474121 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11827 (secure 0 0 0) 0x7f79280a9d10 con 0x7f79381034b0 2024-05-27T07:37:24.403 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:37:24.403 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:37:06.645274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:37:24.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.408+0000 7f793d28d700 1 -- 172.21.15.114:0/336474121 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f79240b2910 msgr2=0x7f79240b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:24.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.408+0000 7f793d28d700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f79240b2910 0x7f79240b4e20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7920009990 tx=0x7f792001a040 comp rx=0 tx=0).stop 2024-05-27T07:37:24.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.408+0000 7f793d28d700 1 -- 172.21.15.114:0/336474121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79381034b0 msgr2=0x7f793819be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:24.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.408+0000 7f793d28d700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79381034b0 0x7f793819be30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f792800b530 tx=0x7f792800b8f0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.409+0000 7f793d28d700 1 -- 172.21.15.114:0/336474121 shutdown_connections 2024-05-27T07:37:24.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.409+0000 7f793d28d700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7938108730 0x7f793819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.409+0000 7f793d28d700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79381034b0 0x7f793819be30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.409+0000 7f793d28d700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f79240b2910 0x7f79240b4e20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.409+0000 7f793d28d700 1 --2- 172.21.15.114:0/336474121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7938107de0 0x7f793819c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.409+0000 7f793d28d700 1 -- 172.21.15.114:0/336474121 >> 172.21.15.114:0/336474121 conn(0x7f79380fedb0 msgr2=0x7f79381011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:24.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.409+0000 7f793d28d700 1 -- 172.21.15.114:0/336474121 shutdown_connections 2024-05-27T07:37:24.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:24.409+0000 7f793d28d700 1 -- 172.21.15.114:0/336474121 wait complete. 2024-05-27T07:37:24.406 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:37:24.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:24 smithi114 ceph-mon[32038]: pgmap v337: 129 pgs: 129 active+clean; 621 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.6 KiB/s wr, 3 op/s 2024-05-27T07:37:24.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:24 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/336474121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:24.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:24 smithi163 ceph-mon[36451]: pgmap v337: 129 pgs: 129 active+clean; 621 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.6 KiB/s wr, 3 op/s 2024-05-27T07:37:24.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:24 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/336474121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:24.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:24 smithi186 ceph-mon[36647]: pgmap v337: 129 pgs: 129 active+clean; 621 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.6 KiB/s wr, 3 op/s 2024-05-27T07:37:24.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:24 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/336474121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:25.141 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:37:25.382 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:26.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:26 smithi163 ceph-mon[36451]: pgmap v338: 129 pgs: 129 active+clean; 621 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.4 KiB/s wr, 3 op/s 2024-05-27T07:37:26.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:26 smithi186 ceph-mon[36647]: pgmap v338: 129 pgs: 129 active+clean; 621 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.4 KiB/s wr, 3 op/s 2024-05-27T07:37:26.760 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:26 smithi114 ceph-mon[32038]: pgmap v338: 129 pgs: 129 active+clean; 621 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.4 KiB/s wr, 3 op/s 2024-05-27T07:37:27.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.155+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/840710543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a68103c00 msgr2=0x7f0a68104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:27.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.155+0000 7f0a6cf3e700 1 --2- 172.21.15.114:0/840710543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a68103c00 0x7f0a68104010 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0a580099c0 tx=0x7f0a58009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:27.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.155+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/840710543 shutdown_connections 2024-05-27T07:37:27.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.155+0000 7f0a6cf3e700 1 --2- 172.21.15.114:0/840710543 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a68108960 0x7f0a681102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:27.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.155+0000 7f0a6cf3e700 1 --2- 172.21.15.114:0/840710543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a68103c00 0x7f0a68104010 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:27.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.155+0000 7f0a6cf3e700 1 --2- 172.21.15.114:0/840710543 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a68108010 0x7f0a68108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:27.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.155+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/840710543 >> 172.21.15.114:0/840710543 conn(0x7f0a680fe510 msgr2=0x7f0a68100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:27.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.155+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/840710543 shutdown_connections 2024-05-27T07:37:27.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.155+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/840710543 wait complete. 2024-05-27T07:37:27.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.156+0000 7f0a6cf3e700 1 Processor -- start 2024-05-27T07:37:27.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.156+0000 7f0a6cf3e700 1 -- start start 2024-05-27T07:37:27.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.156+0000 7f0a6cf3e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a68103c00 0x7f0a6819be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:27.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.156+0000 7f0a6cf3e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a68108010 0x7f0a6819c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:27.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a6cf3e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a68108960 0x7f0a6819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:27.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a6cf3e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0a68113590 con 0x7f0a68103c00 2024-05-27T07:37:27.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a6cf3e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0a68113410 con 0x7f0a68108010 2024-05-27T07:37:27.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a6cf3e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0a68113710 con 0x7f0a68108960 2024-05-27T07:37:27.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a6659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a68103c00 0x7f0a6819be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:27.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a6659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a68103c00 0x7f0a6819be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47182/0 (socket says 172.21.15.114:47182) 2024-05-27T07:37:27.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a6659c700 1 -- 172.21.15.114:0/2491560172 learned_addr learned my addr 172.21.15.114:0/2491560172 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:27.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a65d9b700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a68108010 0x7f0a6819c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:27.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a66d9d700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a68108960 0x7f0a6819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:27.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a6659c700 1 -- 172.21.15.114:0/2491560172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a68108960 msgr2=0x7f0a6819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:27.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a6659c700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a68108960 0x7f0a6819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:27.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a6659c700 1 -- 172.21.15.114:0/2491560172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a68108010 msgr2=0x7f0a6819c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:27.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a6659c700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a68108010 0x7f0a6819c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:27.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a6659c700 1 -- 172.21.15.114:0/2491560172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a681a7110 con 0x7f0a68103c00 2024-05-27T07:37:27.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a6659c700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a68103c00 0x7f0a6819be30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0a580098e0 tx=0x7f0a580057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:27.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a577fe700 1 -- 172.21.15.114:0/2491560172 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a58027070 con 0x7f0a68103c00 2024-05-27T07:37:27.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.157+0000 7f0a577fe700 1 -- 172.21.15.114:0/2491560172 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a58003f20 con 0x7f0a68103c00 2024-05-27T07:37:27.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.158+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/2491560172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a681a73a0 con 0x7f0a68103c00 2024-05-27T07:37:27.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.158+0000 7f0a577fe700 1 -- 172.21.15.114:0/2491560172 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a58022da0 con 0x7f0a68103c00 2024-05-27T07:37:27.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.158+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/2491560172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a681a77f0 con 0x7f0a68103c00 2024-05-27T07:37:27.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.159+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/2491560172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a680512b0 con 0x7f0a68103c00 2024-05-27T07:37:27.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.160+0000 7f0a577fe700 1 -- 172.21.15.114:0/2491560172 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f0a5802a070 con 0x7f0a68103c00 2024-05-27T07:37:27.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.161+0000 7f0a577fe700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0a500b2910 0x7f0a500b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:27.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.161+0000 7f0a65d9b700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0a500b2910 0x7f0a500b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:27.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.161+0000 7f0a577fe700 1 -- 172.21.15.114:0/2491560172 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f0a580e3c90 con 0x7f0a68103c00 2024-05-27T07:37:27.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.161+0000 7f0a65d9b700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0a500b2910 0x7f0a500b4e20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0a68106900 tx=0x7f0a5c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:27.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.164+0000 7f0a577fe700 1 -- 172.21.15.114:0/2491560172 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0a58031080 con 0x7f0a68103c00 2024-05-27T07:37:27.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.370+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/2491560172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a6802e5e0 con 0x7f0a68103c00 2024-05-27T07:37:27.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.371+0000 7f0a577fe700 1 -- 172.21.15.114:0/2491560172 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+24736 (secure 0 0 0) 0x7f0a58013070 con 0x7f0a68103c00 2024-05-27T07:37:27.368 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:37:27.369 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T07:36:45.114309+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:37:27.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.374+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/2491560172 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0a500b2910 msgr2=0x7f0a500b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:27.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.374+0000 7f0a6cf3e700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0a500b2910 0x7f0a500b4e20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0a68106900 tx=0x7f0a5c00b410 comp rx=0 tx=0).stop 2024-05-27T07:37:27.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.374+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/2491560172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a68103c00 msgr2=0x7f0a6819be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:27.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.374+0000 7f0a6cf3e700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a68103c00 0x7f0a6819be30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0a580098e0 tx=0x7f0a580057c0 comp rx=0 tx=0).stop 2024-05-27T07:37:27.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.374+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/2491560172 shutdown_connections 2024-05-27T07:37:27.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.374+0000 7f0a6cf3e700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0a500b2910 0x7f0a500b4e20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:27.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.374+0000 7f0a6cf3e700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a68108960 0x7f0a6819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:27.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.374+0000 7f0a6cf3e700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a68103c00 0x7f0a6819be30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:27.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.374+0000 7f0a6cf3e700 1 --2- 172.21.15.114:0/2491560172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a68108010 0x7f0a6819c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:27.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.374+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/2491560172 >> 172.21.15.114:0/2491560172 conn(0x7f0a680fe510 msgr2=0x7f0a681098c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:27.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.374+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/2491560172 shutdown_connections 2024-05-27T07:37:27.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:27.374+0000 7f0a6cf3e700 1 -- 172.21.15.114:0/2491560172 wait complete. 2024-05-27T07:37:27.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:27 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2491560172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:37:27.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:27 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2491560172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:37:27.735 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:27 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2491560172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:37:28.050 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-27T07:37:28.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:37:28.292 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:28.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:28 smithi163 ceph-mon[36451]: pgmap v339: 129 pgs: 129 active+clean; 635 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.8 KiB/s wr, 5 op/s 2024-05-27T07:37:28.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:28 smithi186 ceph-mon[36647]: pgmap v339: 129 pgs: 129 active+clean; 635 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.8 KiB/s wr, 5 op/s 2024-05-27T07:37:28.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:28 smithi114 ceph-mon[32038]: pgmap v339: 129 pgs: 129 active+clean; 635 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.8 KiB/s wr, 5 op/s 2024-05-27T07:37:30.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.180+0000 7f306ea3b700 1 -- 172.21.15.114:0/2343193407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3068108730 msgr2=0x7f3068110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:30.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.180+0000 7f306ea3b700 1 --2- 172.21.15.114:0/2343193407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3068108730 0x7f3068110090 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f306000b0d0 tx=0x7f306000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.180+0000 7f306ea3b700 1 -- 172.21.15.114:0/2343193407 shutdown_connections 2024-05-27T07:37:30.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.180+0000 7f306ea3b700 1 --2- 172.21.15.114:0/2343193407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3068108730 0x7f3068110090 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.180+0000 7f306ea3b700 1 --2- 172.21.15.114:0/2343193407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30681034b0 0x7f30681038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.180+0000 7f306ea3b700 1 --2- 172.21.15.114:0/2343193407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3068107de0 0x7f30681081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.180+0000 7f306ea3b700 1 -- 172.21.15.114:0/2343193407 >> 172.21.15.114:0/2343193407 conn(0x7f30680fede0 msgr2=0x7f3068101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:30.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.180+0000 7f306ea3b700 1 -- 172.21.15.114:0/2343193407 shutdown_connections 2024-05-27T07:37:30.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.180+0000 7f306ea3b700 1 -- 172.21.15.114:0/2343193407 wait complete. 2024-05-27T07:37:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.180+0000 7f306ea3b700 1 Processor -- start 2024-05-27T07:37:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f306ea3b700 1 -- start start 2024-05-27T07:37:30.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f306ea3b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30681034b0 0x7f306819bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f306ea3b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3068107de0 0x7f306819c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f306ea3b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3068108730 0x7f306819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f306ea3b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3068113360 con 0x7f3068107de0 2024-05-27T07:37:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f306ea3b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f30681131e0 con 0x7f30681034b0 2024-05-27T07:37:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f306ea3b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f30681134e0 con 0x7f3068108730 2024-05-27T07:37:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f30677fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3068107de0 0x7f306819c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f30677fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3068107de0 0x7f306819c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47200/0 (socket says 172.21.15.114:47200) 2024-05-27T07:37:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f30677fe700 1 -- 172.21.15.114:0/2136094820 learned_addr learned my addr 172.21.15.114:0/2136094820 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f3067fff700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30681034b0 0x7f306819bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f306cfd8700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3068108730 0x7f306819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f30677fe700 1 -- 172.21.15.114:0/2136094820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3068108730 msgr2=0x7f306819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f30677fe700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3068108730 0x7f306819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f30677fe700 1 -- 172.21.15.114:0/2136094820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30681034b0 msgr2=0x7f306819bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f30677fe700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30681034b0 0x7f306819bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f30677fe700 1 -- 172.21.15.114:0/2136094820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30681a6f50 con 0x7f3068107de0 2024-05-27T07:37:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.181+0000 7f306cfd8700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3068108730 0x7f306819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:37:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.182+0000 7f30677fe700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3068107de0 0x7f306819c120 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f305c00c8f0 tx=0x7f305c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.182+0000 7f30657fa700 1 -- 172.21.15.114:0/2136094820 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c007b00 con 0x7f3068107de0 2024-05-27T07:37:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.182+0000 7f306ea3b700 1 -- 172.21.15.114:0/2136094820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30681a7240 con 0x7f3068107de0 2024-05-27T07:37:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.182+0000 7f30657fa700 1 -- 172.21.15.114:0/2136094820 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f305c007ca0 con 0x7f3068107de0 2024-05-27T07:37:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.182+0000 7f30657fa700 1 -- 172.21.15.114:0/2136094820 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c018660 con 0x7f3068107de0 2024-05-27T07:37:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.182+0000 7f306ea3b700 1 -- 172.21.15.114:0/2136094820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30681a77d0 con 0x7f3068107de0 2024-05-27T07:37:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.183+0000 7f306ea3b700 1 -- 172.21.15.114:0/2136094820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30680512b0 con 0x7f3068107de0 2024-05-27T07:37:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.184+0000 7f30657fa700 1 -- 172.21.15.114:0/2136094820 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f305c00f450 con 0x7f3068107de0 2024-05-27T07:37:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.185+0000 7f30657fa700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f30500b2910 0x7f30500b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.185+0000 7f3067fff700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f30500b2910 0x7f30500b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.185+0000 7f30657fa700 1 -- 172.21.15.114:0/2136094820 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f305c0e2690 con 0x7f3068107de0 2024-05-27T07:37:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.186+0000 7f3067fff700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f30500b2910 0x7f30500b4e20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3058005fd0 tx=0x7f305800a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.190+0000 7f30657fa700 1 -- 172.21.15.114:0/2136094820 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f305c014030 con 0x7f3068107de0 2024-05-27T07:37:30.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.389+0000 7f306ea3b700 1 -- 172.21.15.114:0/2136094820 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3068063bf0 con 0x7f30500b2910 2024-05-27T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.399+0000 7f30657fa700 1 -- 172.21.15.114:0/2136094820 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340543 (secure 0 0 0) 0x7f3068063bf0 con 0x7f30500b2910 2024-05-27T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:37:30.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.402+0000 7f306ea3b700 1 -- 172.21.15.114:0/2136094820 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f30500b2910 msgr2=0x7f30500b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:30.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.402+0000 7f306ea3b700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f30500b2910 0x7f30500b4e20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3058005fd0 tx=0x7f305800a560 comp rx=0 tx=0).stop 2024-05-27T07:37:30.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.402+0000 7f306ea3b700 1 -- 172.21.15.114:0/2136094820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3068107de0 msgr2=0x7f306819c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:30.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.402+0000 7f306ea3b700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3068107de0 0x7f306819c120 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f305c00c8f0 tx=0x7f305c00ccb0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.402+0000 7f306ea3b700 1 -- 172.21.15.114:0/2136094820 shutdown_connections 2024-05-27T07:37:30.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.402+0000 7f306ea3b700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f30500b2910 0x7f30500b4e20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.402+0000 7f306ea3b700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3068108730 0x7f306819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.402+0000 7f306ea3b700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30681034b0 0x7f306819bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.402+0000 7f306ea3b700 1 --2- 172.21.15.114:0/2136094820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3068107de0 0x7f306819c120 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.402+0000 7f306ea3b700 1 -- 172.21.15.114:0/2136094820 >> 172.21.15.114:0/2136094820 conn(0x7f30680fede0 msgr2=0x7f3068107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:30.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.403+0000 7f306ea3b700 1 -- 172.21.15.114:0/2136094820 shutdown_connections 2024-05-27T07:37:30.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:30.403+0000 7f306ea3b700 1 -- 172.21.15.114:0/2136094820 wait complete. 2024-05-27T07:37:30.422 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-05-27T07:37:30.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:30 smithi163 ceph-mon[36451]: pgmap v340: 129 pgs: 129 active+clean; 635 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.5 KiB/s wr, 3 op/s 2024-05-27T07:37:30.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:30 smithi186 ceph-mon[36647]: pgmap v340: 129 pgs: 129 active+clean; 635 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.5 KiB/s wr, 3 op/s 2024-05-27T07:37:30.769 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:30 smithi114 ceph-mon[32038]: pgmap v340: 129 pgs: 129 active+clean; 635 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.5 KiB/s wr, 3 op/s 2024-05-27T07:37:31.123 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-05-27T07:37:29.389621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":650642,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":140,"num_read_kb":159,"num_write":207,"num_write_kb":2238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162,"ondisk_log_size":162,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364852,"kb_used_data":11528,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576972,"statfs":{"total":1151940427776,"available":1151566819328,"internally_reserved":0,"allocated":11804672,"data_stored":7884916,"data_compressed":424713,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34783,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":45,"num_write":22,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003878"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125405+0000","last_change":"2024-05-27T07:36:04.980329+0000","last_active":"2024-05-27T07:36:45.125405+0000","last_peered":"2024-05-27T07:36:45.125405+0000","last_clean":"2024-05-27T07:36:45.125405+0000","last_became_active":"2024-05-27T07:36:04.980006+0000","last_became_peered":"2024-05-27T07:36:04.980006+0000","last_unstale":"2024-05-27T07:36:45.125405+0000","last_undegraded":"2024-05-27T07:36:45.125405+0000","last_fullsized":"2024-05-27T07:36:45.125405+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:51:56.728465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120442+0000","last_change":"2024-05-27T07:36:04.983429+0000","last_active":"2024-05-27T07:36:45.120442+0000","last_peered":"2024-05-27T07:36:45.120442+0000","last_clean":"2024-05-27T07:36:45.120442+0000","last_became_active":"2024-05-27T07:36:04.983099+0000","last_became_peered":"2024-05-27T07:36:04.983099+0000","last_unstale":"2024-05-27T07:36:45.120442+0000","last_undegraded":"2024-05-27T07:36:45.120442+0000","last_fullsized":"2024-05-27T07:36:45.120442+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:18:51.059717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120202+0000","last_change":"2024-05-27T07:36:04.993319+0000","last_active":"2024-05-27T07:36:45.120202+0000","last_peered":"2024-05-27T07:36:45.120202+0000","last_clean":"2024-05-27T07:36:45.120202+0000","last_became_active":"2024-05-27T07:36:04.992983+0000","last_became_peered":"2024-05-27T07:36:04.992983+0000","last_unstale":"2024-05-27T07:36:45.120202+0000","last_undegraded":"2024-05-27T07:36:45.120202+0000","last_fullsized":"2024-05-27T07:36:45.120202+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:50.800194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.141492+0000","last_change":"2024-05-27T07:36:45.140980+0000","last_active":"2024-05-27T07:36:45.141492+0000","last_peered":"2024-05-27T07:36:45.141492+0000","last_clean":"2024-05-27T07:36:45.141492+0000","last_became_active":"2024-05-27T07:36:45.140551+0000","last_became_peered":"2024-05-27T07:36:45.140551+0000","last_unstale":"2024-05-27T07:36:45.141492+0000","last_undegraded":"2024-05-27T07:36:45.141492+0000","last_fullsized":"2024-05-27T07:36:45.141492+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:59:21.646306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121156+0000","last_change":"2024-05-27T07:36:04.977404+0000","last_active":"2024-05-27T07:36:45.121156+0000","last_peered":"2024-05-27T07:36:45.121156+0000","last_clean":"2024-05-27T07:36:45.121156+0000","last_became_active":"2024-05-27T07:36:04.976991+0000","last_became_peered":"2024-05-27T07:36:04.976991+0000","last_unstale":"2024-05-27T07:36:45.121156+0000","last_undegraded":"2024-05-27T07:36:45.121156+0000","last_fullsized":"2024-05-27T07:36:45.121156+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:56.532586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.124398+0000","last_change":"2024-05-27T07:36:04.982149+0000","last_active":"2024-05-27T07:36:45.124398+0000","last_peered":"2024-05-27T07:36:45.124398+0000","last_clean":"2024-05-27T07:36:45.124398+0000","last_became_active":"2024-05-27T07:36:04.981751+0000","last_became_peered":"2024-05-27T07:36:04.981751+0000","last_unstale":"2024-05-27T07:36:45.124398+0000","last_undegraded":"2024-05-27T07:36:45.124398+0000","last_fullsized":"2024-05-27T07:36:45.124398+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:42.773458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125617+0000","last_change":"2024-05-27T07:36:04.987491+0000","last_active":"2024-05-27T07:36:45.125617+0000","last_peered":"2024-05-27T07:36:45.125617+0000","last_clean":"2024-05-27T07:36:45.125617+0000","last_became_active":"2024-05-27T07:36:04.987059+0000","last_became_peered":"2024-05-27T07:36:04.987059+0000","last_unstale":"2024-05-27T07:36:45.125617+0000","last_undegraded":"2024-05-27T07:36:45.125617+0000","last_fullsized":"2024-05-27T07:36:45.125617+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:57.611253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.127432+0000","last_change":"2024-05-27T07:36:45.126961+0000","last_active":"2024-05-27T07:36:45.127432+0000","last_peered":"2024-05-27T07:36:45.127432+0000","last_clean":"2024-05-27T07:36:45.127432+0000","last_became_active":"2024-05-27T07:36:45.126653+0000","last_became_peered":"2024-05-27T07:36:45.126653+0000","last_unstale":"2024-05-27T07:36:45.127432+0000","last_undegraded":"2024-05-27T07:36:45.127432+0000","last_fullsized":"2024-05-27T07:36:45.127432+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:35.586128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121844+0000","last_change":"2024-05-27T07:36:04.990469+0000","last_active":"2024-05-27T07:36:45.121844+0000","last_peered":"2024-05-27T07:36:45.121844+0000","last_clean":"2024-05-27T07:36:45.121844+0000","last_became_active":"2024-05-27T07:36:04.989771+0000","last_became_peered":"2024-05-27T07:36:04.989771+0000","last_unstale":"2024-05-27T07:36:45.121844+0000","last_undegraded":"2024-05-27T07:36:45.121844+0000","last_fullsized":"2024-05-27T07:36:45.121844+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:53.593566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125310+0000","last_change":"2024-05-27T07:36:04.984746+0000","last_active":"2024-05-27T07:36:45.125310+0000","last_peered":"2024-05-27T07:36:45.125310+0000","last_clean":"2024-05-27T07:36:45.125310+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:36:45.125310+0000","last_undegraded":"2024-05-27T07:36:45.125310+0000","last_fullsized":"2024-05-27T07:36:45.125310+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:18.071981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126865+0000","last_change":"2024-05-27T07:36:04.977167+0000","last_active":"2024-05-27T07:36:45.126865+0000","last_peered":"2024-05-27T07:36:45.126865+0000","last_clean":"2024-05-27T07:36:45.126865+0000","last_became_active":"2024-05-27T07:36:04.976788+0000","last_became_peered":"2024-05-27T07:36:04.976788+0000","last_unstale":"2024-05-27T07:36:45.126865+0000","last_undegraded":"2024-05-27T07:36:45.126865+0000","last_fullsized":"2024-05-27T07:36:45.126865+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:59:44.574191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119713+0000","last_change":"2024-05-27T07:36:04.986594+0000","last_active":"2024-05-27T07:36:45.119713+0000","last_peered":"2024-05-27T07:36:45.119713+0000","last_clean":"2024-05-27T07:36:45.119713+0000","last_became_active":"2024-05-27T07:36:04.985161+0000","last_became_peered":"2024-05-27T07:36:04.985161+0000","last_unstale":"2024-05-27T07:36:45.119713+0000","last_undegraded":"2024-05-27T07:36:45.119713+0000","last_fullsized":"2024-05-27T07:36:45.119713+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.239461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125988+0000","last_change":"2024-05-27T07:36:04.984781+0000","last_active":"2024-05-27T07:36:45.125988+0000","last_peered":"2024-05-27T07:36:45.125988+0000","last_clean":"2024-05-27T07:36:45.125988+0000","last_became_active":"2024-05-27T07:36:04.984462+0000","last_became_peered":"2024-05-27T07:36:04.984462+0000","last_unstale":"2024-05-27T07:36:45.125988+0000","last_undegraded":"2024-05-27T07:36:45.125988+0000","last_fullsized":"2024-05-27T07:36:45.125988+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:40.051692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120044+0000","last_change":"2024-05-27T07:36:04.978360+0000","last_active":"2024-05-27T07:36:45.120044+0000","last_peered":"2024-05-27T07:36:45.120044+0000","last_clean":"2024-05-27T07:36:45.120044+0000","last_became_active":"2024-05-27T07:36:04.978105+0000","last_became_peered":"2024-05-27T07:36:04.978105+0000","last_unstale":"2024-05-27T07:36:45.120044+0000","last_undegraded":"2024-05-27T07:36:45.120044+0000","last_fullsized":"2024-05-27T07:36:45.120044+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:04:47.257529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120064+0000","last_change":"2024-05-27T07:36:04.986510+0000","last_active":"2024-05-27T07:36:45.120064+0000","last_peered":"2024-05-27T07:36:45.120064+0000","last_clean":"2024-05-27T07:36:45.120064+0000","last_became_active":"2024-05-27T07:36:04.986150+0000","last_became_peered":"2024-05-27T07:36:04.986150+0000","last_unstale":"2024-05-27T07:36:45.120064+0000","last_undegraded":"2024-05-27T07:36:45.120064+0000","last_fullsized":"2024-05-27T07:36:45.120064+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:07.349470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121156+0000","last_change":"2024-05-27T07:36:04.979880+0000","last_active":"2024-05-27T07:36:45.121156+0000","last_peered":"2024-05-27T07:36:45.121156+0000","last_clean":"2024-05-27T07:36:45.121156+0000","last_became_active":"2024-05-27T07:36:04.979516+0000","last_became_peered":"2024-05-27T07:36:04.979516+0000","last_unstale":"2024-05-27T07:36:45.121156+0000","last_undegraded":"2024-05-27T07:36:45.121156+0000","last_fullsized":"2024-05-27T07:36:45.121156+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:41:03.964277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125103+0000","last_change":"2024-05-27T07:36:04.980231+0000","last_active":"2024-05-27T07:36:45.125103+0000","last_peered":"2024-05-27T07:36:45.125103+0000","last_clean":"2024-05-27T07:36:45.125103+0000","last_became_active":"2024-05-27T07:36:04.979888+0000","last_became_peered":"2024-05-27T07:36:04.979888+0000","last_unstale":"2024-05-27T07:36:45.125103+0000","last_undegraded":"2024-05-27T07:36:45.125103+0000","last_fullsized":"2024-05-27T07:36:45.125103+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.838119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122786+0000","last_change":"2024-05-27T07:36:04.990890+0000","last_active":"2024-05-27T07:36:45.122786+0000","last_peered":"2024-05-27T07:36:45.122786+0000","last_clean":"2024-05-27T07:36:45.122786+0000","last_became_active":"2024-05-27T07:36:04.990581+0000","last_became_peered":"2024-05-27T07:36:04.990581+0000","last_unstale":"2024-05-27T07:36:45.122786+0000","last_undegraded":"2024-05-27T07:36:45.122786+0000","last_fullsized":"2024-05-27T07:36:45.122786+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:33.054395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139872+0000","last_change":"2024-05-27T07:36:04.981620+0000","last_active":"2024-05-27T07:36:45.139872+0000","last_peered":"2024-05-27T07:36:45.139872+0000","last_clean":"2024-05-27T07:36:45.139872+0000","last_became_active":"2024-05-27T07:36:04.981290+0000","last_became_peered":"2024-05-27T07:36:04.981290+0000","last_unstale":"2024-05-27T07:36:45.139872+0000","last_undegraded":"2024-05-27T07:36:45.139872+0000","last_fullsized":"2024-05-27T07:36:45.139872+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:50.287380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139135+0000","last_change":"2024-05-27T07:36:15.030740+0000","last_active":"2024-05-27T07:36:45.139135+0000","last_peered":"2024-05-27T07:36:45.139135+0000","last_clean":"2024-05-27T07:36:45.139135+0000","last_became_active":"2024-05-27T07:36:15.029738+0000","last_became_peered":"2024-05-27T07:36:15.029738+0000","last_unstale":"2024-05-27T07:36:45.139135+0000","last_undegraded":"2024-05-27T07:36:45.139135+0000","last_fullsized":"2024-05-27T07:36:45.139135+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:18.101941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"78'4","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122593+0000","last_change":"2024-05-27T07:36:01.955530+0000","last_active":"2024-05-27T07:36:45.122593+0000","last_peered":"2024-05-27T07:36:45.122593+0000","last_clean":"2024-05-27T07:36:45.122593+0000","last_became_active":"2024-05-27T07:36:01.955225+0000","last_became_peered":"2024-05-27T07:36:01.955225+0000","last_unstale":"2024-05-27T07:36:45.122593+0000","last_undegraded":"2024-05-27T07:36:45.122593+0000","last_fullsized":"2024-05-27T07:36:45.122593+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:02.067672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139432+0000","last_change":"2024-05-27T07:36:04.984397+0000","last_active":"2024-05-27T07:36:45.139432+0000","last_peered":"2024-05-27T07:36:45.139432+0000","last_clean":"2024-05-27T07:36:45.139432+0000","last_became_active":"2024-05-27T07:36:04.984033+0000","last_became_peered":"2024-05-27T07:36:04.984033+0000","last_unstale":"2024-05-27T07:36:45.139432+0000","last_undegraded":"2024-05-27T07:36:45.139432+0000","last_fullsized":"2024-05-27T07:36:45.139432+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:15.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.118802+0000","last_change":"2024-05-27T07:36:15.028412+0000","last_active":"2024-05-27T07:36:45.118802+0000","last_peered":"2024-05-27T07:36:45.118802+0000","last_clean":"2024-05-27T07:36:45.118802+0000","last_became_active":"2024-05-27T07:36:15.027886+0000","last_became_peered":"2024-05-27T07:36:15.027886+0000","last_unstale":"2024-05-27T07:36:45.118802+0000","last_undegraded":"2024-05-27T07:36:45.118802+0000","last_fullsized":"2024-05-27T07:36:45.118802+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:56:26.383126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"78'1","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139601+0000","last_change":"2024-05-27T07:36:02.243079+0000","last_active":"2024-05-27T07:36:45.139601+0000","last_peered":"2024-05-27T07:36:45.139601+0000","last_clean":"2024-05-27T07:36:45.139601+0000","last_became_active":"2024-05-27T07:36:02.242633+0000","last_became_peered":"2024-05-27T07:36:02.242633+0000","last_unstale":"2024-05-27T07:36:45.139601+0000","last_undegraded":"2024-05-27T07:36:45.139601+0000","last_fullsized":"2024-05-27T07:36:45.139601+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:45:33.980842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120596+0000","last_change":"2024-05-27T07:36:04.989775+0000","last_active":"2024-05-27T07:36:45.120596+0000","last_peered":"2024-05-27T07:36:45.120596+0000","last_clean":"2024-05-27T07:36:45.120596+0000","last_became_active":"2024-05-27T07:36:04.989289+0000","last_became_peered":"2024-05-27T07:36:04.989289+0000","last_unstale":"2024-05-27T07:36:45.120596+0000","last_undegraded":"2024-05-27T07:36:45.120596+0000","last_fullsized":"2024-05-27T07:36:45.120596+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:24:38.355231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119229+0000","last_change":"2024-05-27T07:36:15.031737+0000","last_active":"2024-05-27T07:36:45.119229+0000","last_peered":"2024-05-27T07:36:45.119229+0000","last_clean":"2024-05-27T07:36:45.119229+0000","last_became_active":"2024-05-27T07:36:15.031314+0000","last_became_peered":"2024-05-27T07:36:15.031314+0000","last_unstale":"2024-05-27T07:36:45.119229+0000","last_undegraded":"2024-05-27T07:36:45.119229+0000","last_fullsized":"2024-05-27T07:36:45.119229+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:32.742298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"78'2","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121953+0000","last_change":"2024-05-27T07:36:01.957459+0000","last_active":"2024-05-27T07:36:45.121953+0000","last_peered":"2024-05-27T07:36:45.121953+0000","last_clean":"2024-05-27T07:36:45.121953+0000","last_became_active":"2024-05-27T07:36:01.954985+0000","last_became_peered":"2024-05-27T07:36:01.954985+0000","last_unstale":"2024-05-27T07:36:45.121953+0000","last_undegraded":"2024-05-27T07:36:45.121953+0000","last_fullsized":"2024-05-27T07:36:45.121953+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:27.085486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121538+0000","last_change":"2024-05-27T07:36:04.986447+0000","last_active":"2024-05-27T07:36:45.121538+0000","last_peered":"2024-05-27T07:36:45.121538+0000","last_clean":"2024-05-27T07:36:45.121538+0000","last_became_active":"2024-05-27T07:36:04.982099+0000","last_became_peered":"2024-05-27T07:36:04.982099+0000","last_unstale":"2024-05-27T07:36:45.121538+0000","last_undegraded":"2024-05-27T07:36:45.121538+0000","last_fullsized":"2024-05-27T07:36:45.121538+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:32.637479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.118853+0000","last_change":"2024-05-27T07:36:15.024152+0000","last_active":"2024-05-27T07:36:45.118853+0000","last_peered":"2024-05-27T07:36:45.118853+0000","last_clean":"2024-05-27T07:36:45.118853+0000","last_became_active":"2024-05-27T07:36:15.023711+0000","last_became_peered":"2024-05-27T07:36:15.023711+0000","last_unstale":"2024-05-27T07:36:45.118853+0000","last_undegraded":"2024-05-27T07:36:45.118853+0000","last_fullsized":"2024-05-27T07:36:45.118853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:33.809364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121281+0000","last_change":"2024-05-27T07:36:01.950531+0000","last_active":"2024-05-27T07:36:45.121281+0000","last_peered":"2024-05-27T07:36:45.121281+0000","last_clean":"2024-05-27T07:36:45.121281+0000","last_became_active":"2024-05-27T07:36:01.950015+0000","last_became_peered":"2024-05-27T07:36:01.950015+0000","last_unstale":"2024-05-27T07:36:45.121281+0000","last_undegraded":"2024-05-27T07:36:45.121281+0000","last_fullsized":"2024-05-27T07:36:45.121281+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:22.400415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123907+0000","last_change":"2024-05-27T07:36:04.982792+0000","last_active":"2024-05-27T07:36:45.123907+0000","last_peered":"2024-05-27T07:36:45.123907+0000","last_clean":"2024-05-27T07:36:45.123907+0000","last_became_active":"2024-05-27T07:36:04.982400+0000","last_became_peered":"2024-05-27T07:36:04.982400+0000","last_unstale":"2024-05-27T07:36:45.123907+0000","last_undegraded":"2024-05-27T07:36:45.123907+0000","last_fullsized":"2024-05-27T07:36:45.123907+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:27.823654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139531+0000","last_change":"2024-05-27T07:36:15.027932+0000","last_active":"2024-05-27T07:36:45.139531+0000","last_peered":"2024-05-27T07:36:45.139531+0000","last_clean":"2024-05-27T07:36:45.139531+0000","last_became_active":"2024-05-27T07:36:15.027526+0000","last_became_peered":"2024-05-27T07:36:15.027526+0000","last_unstale":"2024-05-27T07:36:45.139531+0000","last_undegraded":"2024-05-27T07:36:45.139531+0000","last_fullsized":"2024-05-27T07:36:45.139531+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:16.165934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"78'1","reported_seq":46,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.127637+0000","last_change":"2024-05-27T07:36:45.127637+0000","last_active":"2024-05-27T07:36:45.127637+0000","last_peered":"2024-05-27T07:36:45.127637+0000","last_clean":"2024-05-27T07:36:45.127637+0000","last_became_active":"2024-05-27T07:36:45.127249+0000","last_became_peered":"2024-05-27T07:36:45.127249+0000","last_unstale":"2024-05-27T07:36:45.127637+0000","last_undegraded":"2024-05-27T07:36:45.127637+0000","last_fullsized":"2024-05-27T07:36:45.127637+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:52.418069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121167+0000","last_change":"2024-05-27T07:36:04.985135+0000","last_active":"2024-05-27T07:36:45.121167+0000","last_peered":"2024-05-27T07:36:45.121167+0000","last_clean":"2024-05-27T07:36:45.121167+0000","last_became_active":"2024-05-27T07:36:04.983621+0000","last_became_peered":"2024-05-27T07:36:04.983621+0000","last_unstale":"2024-05-27T07:36:45.121167+0000","last_undegraded":"2024-05-27T07:36:45.121167+0000","last_fullsized":"2024-05-27T07:36:45.121167+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:11.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139593+0000","last_change":"2024-05-27T07:36:15.024558+0000","last_active":"2024-05-27T07:36:45.139593+0000","last_peered":"2024-05-27T07:36:45.139593+0000","last_clean":"2024-05-27T07:36:45.139593+0000","last_became_active":"2024-05-27T07:36:15.024148+0000","last_became_peered":"2024-05-27T07:36:15.024148+0000","last_unstale":"2024-05-27T07:36:45.139593+0000","last_undegraded":"2024-05-27T07:36:45.139593+0000","last_fullsized":"2024-05-27T07:36:45.139593+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:03.648451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"78'1","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.129413+0000","last_change":"2024-05-27T07:36:45.129413+0000","last_active":"2024-05-27T07:36:45.129413+0000","last_peered":"2024-05-27T07:36:45.129413+0000","last_clean":"2024-05-27T07:36:45.129413+0000","last_became_active":"2024-05-27T07:36:45.128779+0000","last_became_peered":"2024-05-27T07:36:45.128779+0000","last_unstale":"2024-05-27T07:36:45.129413+0000","last_undegraded":"2024-05-27T07:36:45.129413+0000","last_fullsized":"2024-05-27T07:36:45.129413+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:05.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123709+0000","last_change":"2024-05-27T07:36:04.989878+0000","last_active":"2024-05-27T07:36:45.123709+0000","last_peered":"2024-05-27T07:36:45.123709+0000","last_clean":"2024-05-27T07:36:45.123709+0000","last_became_active":"2024-05-27T07:36:04.989532+0000","last_became_peered":"2024-05-27T07:36:04.989532+0000","last_unstale":"2024-05-27T07:36:45.123709+0000","last_undegraded":"2024-05-27T07:36:45.123709+0000","last_fullsized":"2024-05-27T07:36:45.123709+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:54:53.909592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.124963+0000","last_change":"2024-05-27T07:36:15.023739+0000","last_active":"2024-05-27T07:36:45.124963+0000","last_peered":"2024-05-27T07:36:45.124963+0000","last_clean":"2024-05-27T07:36:45.124963+0000","last_became_active":"2024-05-27T07:36:15.023245+0000","last_became_peered":"2024-05-27T07:36:15.023245+0000","last_unstale":"2024-05-27T07:36:45.124963+0000","last_undegraded":"2024-05-27T07:36:45.124963+0000","last_fullsized":"2024-05-27T07:36:45.124963+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:56:12.585673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125698+0000","last_change":"2024-05-27T07:36:01.949684+0000","last_active":"2024-05-27T07:36:45.125698+0000","last_peered":"2024-05-27T07:36:45.125698+0000","last_clean":"2024-05-27T07:36:45.125698+0000","last_became_active":"2024-05-27T07:36:01.949308+0000","last_became_peered":"2024-05-27T07:36:01.949308+0000","last_unstale":"2024-05-27T07:36:45.125698+0000","last_undegraded":"2024-05-27T07:36:45.125698+0000","last_fullsized":"2024-05-27T07:36:45.125698+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:50:54.859172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139012+0000","last_change":"2024-05-27T07:36:04.971266+0000","last_active":"2024-05-27T07:36:45.139012+0000","last_peered":"2024-05-27T07:36:45.139012+0000","last_clean":"2024-05-27T07:36:45.139012+0000","last_became_active":"2024-05-27T07:36:04.970831+0000","last_became_peered":"2024-05-27T07:36:04.970831+0000","last_unstale":"2024-05-27T07:36:45.139012+0000","last_undegraded":"2024-05-27T07:36:45.139012+0000","last_fullsized":"2024-05-27T07:36:45.139012+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:37.237637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120359+0000","last_change":"2024-05-27T07:36:15.024415+0000","last_active":"2024-05-27T07:36:45.120359+0000","last_peered":"2024-05-27T07:36:45.120359+0000","last_clean":"2024-05-27T07:36:45.120359+0000","last_became_active":"2024-05-27T07:36:15.023935+0000","last_became_peered":"2024-05-27T07:36:15.023935+0000","last_unstale":"2024-05-27T07:36:45.120359+0000","last_undegraded":"2024-05-27T07:36:45.120359+0000","last_fullsized":"2024-05-27T07:36:45.120359+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:37:30.386933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121375+0000","last_change":"2024-05-27T07:36:01.952256+0000","last_active":"2024-05-27T07:36:45.121375+0000","last_peered":"2024-05-27T07:36:45.121375+0000","last_clean":"2024-05-27T07:36:45.121375+0000","last_became_active":"2024-05-27T07:36:01.951876+0000","last_became_peered":"2024-05-27T07:36:01.951876+0000","last_unstale":"2024-05-27T07:36:45.121375+0000","last_undegraded":"2024-05-27T07:36:45.121375+0000","last_fullsized":"2024-05-27T07:36:45.121375+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:05.201780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121791+0000","last_change":"2024-05-27T07:36:04.980300+0000","last_active":"2024-05-27T07:36:45.121791+0000","last_peered":"2024-05-27T07:36:45.121791+0000","last_clean":"2024-05-27T07:36:45.121791+0000","last_became_active":"2024-05-27T07:36:04.979954+0000","last_became_peered":"2024-05-27T07:36:04.979954+0000","last_unstale":"2024-05-27T07:36:45.121791+0000","last_undegraded":"2024-05-27T07:36:45.121791+0000","last_fullsized":"2024-05-27T07:36:45.121791+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:15.712773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139072+0000","last_change":"2024-05-27T07:36:15.026685+0000","last_active":"2024-05-27T07:36:45.139072+0000","last_peered":"2024-05-27T07:36:45.139072+0000","last_clean":"2024-05-27T07:36:45.139072+0000","last_became_active":"2024-05-27T07:36:15.026287+0000","last_became_peered":"2024-05-27T07:36:15.026287+0000","last_unstale":"2024-05-27T07:36:45.139072+0000","last_undegraded":"2024-05-27T07:36:45.139072+0000","last_fullsized":"2024-05-27T07:36:45.139072+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:29:26.394435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125066+0000","last_change":"2024-05-27T07:36:01.956566+0000","last_active":"2024-05-27T07:36:45.125066+0000","last_peered":"2024-05-27T07:36:45.125066+0000","last_clean":"2024-05-27T07:36:45.125066+0000","last_became_active":"2024-05-27T07:36:01.955963+0000","last_became_peered":"2024-05-27T07:36:01.955963+0000","last_unstale":"2024-05-27T07:36:45.125066+0000","last_undegraded":"2024-05-27T07:36:45.125066+0000","last_fullsized":"2024-05-27T07:36:45.125066+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:53.081784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122035+0000","last_change":"2024-05-27T07:36:04.983878+0000","last_active":"2024-05-27T07:36:45.122035+0000","last_peered":"2024-05-27T07:36:45.122035+0000","last_clean":"2024-05-27T07:36:45.122035+0000","last_became_active":"2024-05-27T07:36:04.983339+0000","last_became_peered":"2024-05-27T07:36:04.983339+0000","last_unstale":"2024-05-27T07:36:45.122035+0000","last_undegraded":"2024-05-27T07:36:45.122035+0000","last_fullsized":"2024-05-27T07:36:45.122035+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:57:01.726122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121569+0000","last_change":"2024-05-27T07:36:15.027923+0000","last_active":"2024-05-27T07:36:45.121569+0000","last_peered":"2024-05-27T07:36:45.121569+0000","last_clean":"2024-05-27T07:36:45.121569+0000","last_became_active":"2024-05-27T07:36:15.026012+0000","last_became_peered":"2024-05-27T07:36:15.026012+0000","last_unstale":"2024-05-27T07:36:45.121569+0000","last_undegraded":"2024-05-27T07:36:45.121569+0000","last_fullsized":"2024-05-27T07:36:45.121569+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:27:45.238053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126647+0000","last_change":"2024-05-27T07:36:01.956038+0000","last_active":"2024-05-27T07:36:45.126647+0000","last_peered":"2024-05-27T07:36:45.126647+0000","last_clean":"2024-05-27T07:36:45.126647+0000","last_became_active":"2024-05-27T07:36:01.955639+0000","last_became_peered":"2024-05-27T07:36:01.955639+0000","last_unstale":"2024-05-27T07:36:45.126647+0000","last_undegraded":"2024-05-27T07:36:45.126647+0000","last_fullsized":"2024-05-27T07:36:45.126647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:32.920194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120513+0000","last_change":"2024-05-27T07:36:04.986676+0000","last_active":"2024-05-27T07:36:45.120513+0000","last_peered":"2024-05-27T07:36:45.120513+0000","last_clean":"2024-05-27T07:36:45.120513+0000","last_became_active":"2024-05-27T07:36:04.986316+0000","last_became_peered":"2024-05-27T07:36:04.986316+0000","last_unstale":"2024-05-27T07:36:45.120513+0000","last_undegraded":"2024-05-27T07:36:45.120513+0000","last_fullsized":"2024-05-27T07:36:45.120513+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:03:50.947781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119250+0000","last_change":"2024-05-27T07:36:15.027643+0000","last_active":"2024-05-27T07:36:45.119250+0000","last_peered":"2024-05-27T07:36:45.119250+0000","last_clean":"2024-05-27T07:36:45.119250+0000","last_became_active":"2024-05-27T07:36:15.027175+0000","last_became_peered":"2024-05-27T07:36:15.027175+0000","last_unstale":"2024-05-27T07:36:45.119250+0000","last_undegraded":"2024-05-27T07:36:45.119250+0000","last_fullsized":"2024-05-27T07:36:45.119250+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:46:32.764271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"78'3","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123809+0000","last_change":"2024-05-27T07:36:02.245746+0000","last_active":"2024-05-27T07:36:45.123809+0000","last_peered":"2024-05-27T07:36:45.123809+0000","last_clean":"2024-05-27T07:36:45.123809+0000","last_became_active":"2024-05-27T07:36:02.245269+0000","last_became_peered":"2024-05-27T07:36:02.245269+0000","last_unstale":"2024-05-27T07:36:45.123809+0000","last_undegraded":"2024-05-27T07:36:45.123809+0000","last_fullsized":"2024-05-27T07:36:45.123809+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:28.202540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.129374+0000","last_change":"2024-05-27T07:36:45.129374+0000","last_active":"2024-05-27T07:36:45.129374+0000","last_peered":"2024-05-27T07:36:45.129374+0000","last_clean":"2024-05-27T07:36:45.129374+0000","last_became_active":"2024-05-27T07:36:45.128956+0000","last_became_peered":"2024-05-27T07:36:45.128956+0000","last_unstale":"2024-05-27T07:36:45.129374+0000","last_undegraded":"2024-05-27T07:36:45.129374+0000","last_fullsized":"2024-05-27T07:36:45.129374+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:17.037598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122356+0000","last_change":"2024-05-27T07:36:15.027097+0000","last_active":"2024-05-27T07:36:45.122356+0000","last_peered":"2024-05-27T07:36:45.122356+0000","last_clean":"2024-05-27T07:36:45.122356+0000","last_became_active":"2024-05-27T07:36:15.026704+0000","last_became_peered":"2024-05-27T07:36:15.026704+0000","last_unstale":"2024-05-27T07:36:45.122356+0000","last_undegraded":"2024-05-27T07:36:45.122356+0000","last_fullsized":"2024-05-27T07:36:45.122356+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:35:03.747199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'28","reported_seq":121,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:37:25.494337+0000","last_change":"2024-05-27T07:36:01.956926+0000","last_active":"2024-05-27T07:37:25.494337+0000","last_peered":"2024-05-27T07:37:25.494337+0000","last_clean":"2024-05-27T07:37:25.494337+0000","last_became_active":"2024-05-27T07:36:01.956515+0000","last_became_peered":"2024-05-27T07:36:01.956515+0000","last_unstale":"2024-05-27T07:37:25.494337+0000","last_undegraded":"2024-05-27T07:37:25.494337+0000","last_fullsized":"2024-05-27T07:37:25.494337+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:46.566489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59242,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":53,"num_read_kb":84,"num_write":28,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121635+0000","last_change":"2024-05-27T07:36:04.976498+0000","last_active":"2024-05-27T07:36:45.121635+0000","last_peered":"2024-05-27T07:36:45.121635+0000","last_clean":"2024-05-27T07:36:45.121635+0000","last_became_active":"2024-05-27T07:36:04.976020+0000","last_became_peered":"2024-05-27T07:36:04.976020+0000","last_unstale":"2024-05-27T07:36:45.121635+0000","last_undegraded":"2024-05-27T07:36:45.121635+0000","last_fullsized":"2024-05-27T07:36:45.121635+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:25.909259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.124919+0000","last_change":"2024-05-27T07:36:15.022700+0000","last_active":"2024-05-27T07:36:45.124919+0000","last_peered":"2024-05-27T07:36:45.124919+0000","last_clean":"2024-05-27T07:36:45.124919+0000","last_became_active":"2024-05-27T07:36:15.022109+0000","last_became_peered":"2024-05-27T07:36:15.022109+0000","last_unstale":"2024-05-27T07:36:45.124919+0000","last_undegraded":"2024-05-27T07:36:45.124919+0000","last_fullsized":"2024-05-27T07:36:45.124919+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:02:43.498049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"78'2","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.140024+0000","last_change":"2024-05-27T07:36:01.953442+0000","last_active":"2024-05-27T07:36:45.140024+0000","last_peered":"2024-05-27T07:36:45.140024+0000","last_clean":"2024-05-27T07:36:45.140024+0000","last_became_active":"2024-05-27T07:36:01.953051+0000","last_became_peered":"2024-05-27T07:36:01.953051+0000","last_unstale":"2024-05-27T07:36:45.140024+0000","last_undegraded":"2024-05-27T07:36:45.140024+0000","last_fullsized":"2024-05-27T07:36:45.140024+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:47:53.070596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125820+0000","last_change":"2024-05-27T07:36:04.976365+0000","last_active":"2024-05-27T07:36:45.125820+0000","last_peered":"2024-05-27T07:36:45.125820+0000","last_clean":"2024-05-27T07:36:45.125820+0000","last_became_active":"2024-05-27T07:36:04.976050+0000","last_became_peered":"2024-05-27T07:36:04.976050+0000","last_unstale":"2024-05-27T07:36:45.125820+0000","last_undegraded":"2024-05-27T07:36:45.125820+0000","last_fullsized":"2024-05-27T07:36:45.125820+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:19.765746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123022+0000","last_change":"2024-05-27T07:36:15.022561+0000","last_active":"2024-05-27T07:36:45.123022+0000","last_peered":"2024-05-27T07:36:45.123022+0000","last_clean":"2024-05-27T07:36:45.123022+0000","last_became_active":"2024-05-27T07:36:15.021974+0000","last_became_peered":"2024-05-27T07:36:15.021974+0000","last_unstale":"2024-05-27T07:36:45.123022+0000","last_undegraded":"2024-05-27T07:36:45.123022+0000","last_fullsized":"2024-05-27T07:36:45.123022+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:31.520969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122631+0000","last_change":"2024-05-27T07:36:01.952625+0000","last_active":"2024-05-27T07:36:45.122631+0000","last_peered":"2024-05-27T07:36:45.122631+0000","last_clean":"2024-05-27T07:36:45.122631+0000","last_became_active":"2024-05-27T07:36:01.952280+0000","last_became_peered":"2024-05-27T07:36:01.952280+0000","last_unstale":"2024-05-27T07:36:45.122631+0000","last_undegraded":"2024-05-27T07:36:45.122631+0000","last_fullsized":"2024-05-27T07:36:45.122631+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:17.548061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125589+0000","last_change":"2024-05-27T07:36:04.984795+0000","last_active":"2024-05-27T07:36:45.125589+0000","last_peered":"2024-05-27T07:36:45.125589+0000","last_clean":"2024-05-27T07:36:45.125589+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:36:45.125589+0000","last_undegraded":"2024-05-27T07:36:45.125589+0000","last_fullsized":"2024-05-27T07:36:45.125589+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:07.207460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122161+0000","last_change":"2024-05-27T07:36:15.033565+0000","last_active":"2024-05-27T07:36:45.122161+0000","last_peered":"2024-05-27T07:36:45.122161+0000","last_clean":"2024-05-27T07:36:45.122161+0000","last_became_active":"2024-05-27T07:36:15.033145+0000","last_became_peered":"2024-05-27T07:36:15.033145+0000","last_unstale":"2024-05-27T07:36:45.122161+0000","last_undegraded":"2024-05-27T07:36:45.122161+0000","last_fullsized":"2024-05-27T07:36:45.122161+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:38:02.163816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.118952+0000","last_change":"2024-05-27T07:36:01.952759+0000","last_active":"2024-05-27T07:36:45.118952+0000","last_peered":"2024-05-27T07:36:45.118952+0000","last_clean":"2024-05-27T07:36:45.118952+0000","last_became_active":"2024-05-27T07:36:01.952339+0000","last_became_peered":"2024-05-27T07:36:01.952339+0000","last_unstale":"2024-05-27T07:36:45.118952+0000","last_undegraded":"2024-05-27T07:36:45.118952+0000","last_fullsized":"2024-05-27T07:36:45.118952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:14.518123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125653+0000","last_change":"2024-05-27T07:36:04.986312+0000","last_active":"2024-05-27T07:36:45.125653+0000","last_peered":"2024-05-27T07:36:45.125653+0000","last_clean":"2024-05-27T07:36:45.125653+0000","last_became_active":"2024-05-27T07:36:04.985944+0000","last_became_peered":"2024-05-27T07:36:04.985944+0000","last_unstale":"2024-05-27T07:36:45.125653+0000","last_undegraded":"2024-05-27T07:36:45.125653+0000","last_fullsized":"2024-05-27T07:36:45.125653+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:04.955449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126172+0000","last_change":"2024-05-27T07:36:15.020930+0000","last_active":"2024-05-27T07:36:45.126172+0000","last_peered":"2024-05-27T07:36:45.126172+0000","last_clean":"2024-05-27T07:36:45.126172+0000","last_became_active":"2024-05-27T07:36:15.020556+0000","last_became_peered":"2024-05-27T07:36:15.020556+0000","last_unstale":"2024-05-27T07:36:45.126172+0000","last_undegraded":"2024-05-27T07:36:45.126172+0000","last_fullsized":"2024-05-27T07:36:45.126172+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:02.673065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"78'8","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121970+0000","last_change":"2024-05-27T07:36:02.243402+0000","last_active":"2024-05-27T07:36:45.121970+0000","last_peered":"2024-05-27T07:36:45.121970+0000","last_clean":"2024-05-27T07:36:45.121970+0000","last_became_active":"2024-05-27T07:36:02.242978+0000","last_became_peered":"2024-05-27T07:36:02.242978+0000","last_unstale":"2024-05-27T07:36:45.121970+0000","last_undegraded":"2024-05-27T07:36:45.121970+0000","last_fullsized":"2024-05-27T07:36:45.121970+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:25:04.289506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123057+0000","last_change":"2024-05-27T07:36:04.989885+0000","last_active":"2024-05-27T07:36:45.123057+0000","last_peered":"2024-05-27T07:36:45.123057+0000","last_clean":"2024-05-27T07:36:45.123057+0000","last_became_active":"2024-05-27T07:36:04.989394+0000","last_became_peered":"2024-05-27T07:36:04.989394+0000","last_unstale":"2024-05-27T07:36:45.123057+0000","last_undegraded":"2024-05-27T07:36:45.123057+0000","last_fullsized":"2024-05-27T07:36:45.123057+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:54.138286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122907+0000","last_change":"2024-05-27T07:36:15.021772+0000","last_active":"2024-05-27T07:36:45.122907+0000","last_peered":"2024-05-27T07:36:45.122907+0000","last_clean":"2024-05-27T07:36:45.122907+0000","last_became_active":"2024-05-27T07:36:15.021352+0000","last_became_peered":"2024-05-27T07:36:15.021352+0000","last_unstale":"2024-05-27T07:36:45.122907+0000","last_undegraded":"2024-05-27T07:36:45.122907+0000","last_fullsized":"2024-05-27T07:36:45.122907+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:23.066662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.132394+0000","last_change":"2024-05-27T07:36:45.131900+0000","last_active":"2024-05-27T07:36:45.132394+0000","last_peered":"2024-05-27T07:36:45.132394+0000","last_clean":"2024-05-27T07:36:45.132394+0000","last_became_active":"2024-05-27T07:36:45.127340+0000","last_became_peered":"2024-05-27T07:36:45.127340+0000","last_unstale":"2024-05-27T07:36:45.132394+0000","last_undegraded":"2024-05-27T07:36:45.132394+0000","last_fullsized":"2024-05-27T07:36:45.132394+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:07:57.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123373+0000","last_change":"2024-05-27T07:36:04.987671+0000","last_active":"2024-05-27T07:36:45.123373+0000","last_peered":"2024-05-27T07:36:45.123373+0000","last_clean":"2024-05-27T07:36:45.123373+0000","last_became_active":"2024-05-27T07:36:04.987317+0000","last_became_peered":"2024-05-27T07:36:04.987317+0000","last_unstale":"2024-05-27T07:36:45.123373+0000","last_undegraded":"2024-05-27T07:36:45.123373+0000","last_fullsized":"2024-05-27T07:36:45.123373+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:35.985968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121126+0000","last_change":"2024-05-27T07:36:15.019967+0000","last_active":"2024-05-27T07:36:45.121126+0000","last_peered":"2024-05-27T07:36:45.121126+0000","last_clean":"2024-05-27T07:36:45.121126+0000","last_became_active":"2024-05-27T07:36:15.019509+0000","last_became_peered":"2024-05-27T07:36:15.019509+0000","last_unstale":"2024-05-27T07:36:45.121126+0000","last_undegraded":"2024-05-27T07:36:45.121126+0000","last_fullsized":"2024-05-27T07:36:45.121126+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:01:56.137335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122110+0000","last_change":"2024-05-27T07:36:02.243561+0000","last_active":"2024-05-27T07:36:45.122110+0000","last_peered":"2024-05-27T07:36:45.122110+0000","last_clean":"2024-05-27T07:36:45.122110+0000","last_became_active":"2024-05-27T07:36:02.242911+0000","last_became_peered":"2024-05-27T07:36:02.242911+0000","last_unstale":"2024-05-27T07:36:45.122110+0000","last_undegraded":"2024-05-27T07:36:45.122110+0000","last_fullsized":"2024-05-27T07:36:45.122110+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:01:43.915660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.124610+0000","last_change":"2024-05-27T07:36:04.971108+0000","last_active":"2024-05-27T07:36:45.124610+0000","last_peered":"2024-05-27T07:36:45.124610+0000","last_clean":"2024-05-27T07:36:45.124610+0000","last_became_active":"2024-05-27T07:36:04.970682+0000","last_became_peered":"2024-05-27T07:36:04.970682+0000","last_unstale":"2024-05-27T07:36:45.124610+0000","last_undegraded":"2024-05-27T07:36:45.124610+0000","last_fullsized":"2024-05-27T07:36:45.124610+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:15:34.940449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126090+0000","last_change":"2024-05-27T07:36:15.021292+0000","last_active":"2024-05-27T07:36:45.126090+0000","last_peered":"2024-05-27T07:36:45.126090+0000","last_clean":"2024-05-27T07:36:45.126090+0000","last_became_active":"2024-05-27T07:36:15.019988+0000","last_became_peered":"2024-05-27T07:36:15.019988+0000","last_unstale":"2024-05-27T07:36:45.126090+0000","last_undegraded":"2024-05-27T07:36:45.126090+0000","last_fullsized":"2024-05-27T07:36:45.126090+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:34:14.325368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125715+0000","last_change":"2024-05-27T07:36:01.952231+0000","last_active":"2024-05-27T07:36:45.125715+0000","last_peered":"2024-05-27T07:36:45.125715+0000","last_clean":"2024-05-27T07:36:45.125715+0000","last_became_active":"2024-05-27T07:36:01.951818+0000","last_became_peered":"2024-05-27T07:36:01.951818+0000","last_unstale":"2024-05-27T07:36:45.125715+0000","last_undegraded":"2024-05-27T07:36:45.125715+0000","last_fullsized":"2024-05-27T07:36:45.125715+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:52:16.299204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.124378+0000","last_change":"2024-05-27T07:36:04.982761+0000","last_active":"2024-05-27T07:36:45.124378+0000","last_peered":"2024-05-27T07:36:45.124378+0000","last_clean":"2024-05-27T07:36:45.124378+0000","last_became_active":"2024-05-27T07:36:04.982303+0000","last_became_peered":"2024-05-27T07:36:04.982303+0000","last_unstale":"2024-05-27T07:36:45.124378+0000","last_undegraded":"2024-05-27T07:36:45.124378+0000","last_fullsized":"2024-05-27T07:36:45.124378+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:13:19.099227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126456+0000","last_change":"2024-05-27T07:36:15.021921+0000","last_active":"2024-05-27T07:36:45.126456+0000","last_peered":"2024-05-27T07:36:45.126456+0000","last_clean":"2024-05-27T07:36:45.126456+0000","last_became_active":"2024-05-27T07:36:15.021524+0000","last_became_peered":"2024-05-27T07:36:15.021524+0000","last_unstale":"2024-05-27T07:36:45.126456+0000","last_undegraded":"2024-05-27T07:36:45.126456+0000","last_fullsized":"2024-05-27T07:36:45.126456+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:10:27.303498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.142023+0000","last_change":"2024-05-27T07:36:45.142023+0000","last_active":"2024-05-27T07:36:45.142023+0000","last_peered":"2024-05-27T07:36:45.142023+0000","last_clean":"2024-05-27T07:36:45.142023+0000","last_became_active":"2024-05-27T07:36:45.141278+0000","last_became_peered":"2024-05-27T07:36:45.141278+0000","last_unstale":"2024-05-27T07:36:45.142023+0000","last_undegraded":"2024-05-27T07:36:45.142023+0000","last_fullsized":"2024-05-27T07:36:45.142023+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:15.800380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125472+0000","last_change":"2024-05-27T07:36:02.243041+0000","last_active":"2024-05-27T07:36:45.125472+0000","last_peered":"2024-05-27T07:36:45.125472+0000","last_clean":"2024-05-27T07:36:45.125472+0000","last_became_active":"2024-05-27T07:36:02.242689+0000","last_became_peered":"2024-05-27T07:36:02.242689+0000","last_unstale":"2024-05-27T07:36:45.125472+0000","last_undegraded":"2024-05-27T07:36:45.125472+0000","last_fullsized":"2024-05-27T07:36:45.125472+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:44:19.652954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126874+0000","last_change":"2024-05-27T07:36:15.025595+0000","last_active":"2024-05-27T07:36:45.126874+0000","last_peered":"2024-05-27T07:36:45.126874+0000","last_clean":"2024-05-27T07:36:45.126874+0000","last_became_active":"2024-05-27T07:36:15.025177+0000","last_became_peered":"2024-05-27T07:36:15.025177+0000","last_unstale":"2024-05-27T07:36:45.126874+0000","last_undegraded":"2024-05-27T07:36:45.126874+0000","last_fullsized":"2024-05-27T07:36:45.126874+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:08.920778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139265+0000","last_change":"2024-05-27T07:36:04.981422+0000","last_active":"2024-05-27T07:36:45.139265+0000","last_peered":"2024-05-27T07:36:45.139265+0000","last_clean":"2024-05-27T07:36:45.139265+0000","last_became_active":"2024-05-27T07:36:04.981100+0000","last_became_peered":"2024-05-27T07:36:04.981100+0000","last_unstale":"2024-05-27T07:36:45.139265+0000","last_undegraded":"2024-05-27T07:36:45.139265+0000","last_fullsized":"2024-05-27T07:36:45.139265+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:01.076309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119097+0000","last_change":"2024-05-27T07:36:01.953727+0000","last_active":"2024-05-27T07:36:45.119097+0000","last_peered":"2024-05-27T07:36:45.119097+0000","last_clean":"2024-05-27T07:36:45.119097+0000","last_became_active":"2024-05-27T07:36:01.953323+0000","last_became_peered":"2024-05-27T07:36:01.953323+0000","last_unstale":"2024-05-27T07:36:45.119097+0000","last_undegraded":"2024-05-27T07:36:45.119097+0000","last_fullsized":"2024-05-27T07:36:45.119097+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:17.186854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":115,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125453+0000","last_change":"2024-05-27T07:34:38.289913+0000","last_active":"2024-05-27T07:36:45.125453+0000","last_peered":"2024-05-27T07:36:45.125453+0000","last_clean":"2024-05-27T07:36:45.125453+0000","last_became_active":"2024-05-27T07:34:38.285523+0000","last_became_peered":"2024-05-27T07:34:38.285523+0000","last_unstale":"2024-05-27T07:36:45.125453+0000","last_undegraded":"2024-05-27T07:36:45.125453+0000","last_fullsized":"2024-05-27T07:36:45.125453+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_clean_scrub_stamp":"2024-05-27T07:31:09.455079+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:59.537055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123693+0000","last_change":"2024-05-27T07:36:15.027453+0000","last_active":"2024-05-27T07:36:45.123693+0000","last_peered":"2024-05-27T07:36:45.123693+0000","last_clean":"2024-05-27T07:36:45.123693+0000","last_became_active":"2024-05-27T07:36:15.026872+0000","last_became_peered":"2024-05-27T07:36:15.026872+0000","last_unstale":"2024-05-27T07:36:45.123693+0000","last_undegraded":"2024-05-27T07:36:45.123693+0000","last_fullsized":"2024-05-27T07:36:45.123693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:32.351924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125741+0000","last_change":"2024-05-27T07:36:04.982041+0000","last_active":"2024-05-27T07:36:45.125741+0000","last_peered":"2024-05-27T07:36:45.125741+0000","last_clean":"2024-05-27T07:36:45.125741+0000","last_became_active":"2024-05-27T07:36:04.981719+0000","last_became_peered":"2024-05-27T07:36:04.981719+0000","last_unstale":"2024-05-27T07:36:45.125741+0000","last_undegraded":"2024-05-27T07:36:45.125741+0000","last_fullsized":"2024-05-27T07:36:45.125741+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:40.061892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125526+0000","last_change":"2024-05-27T07:36:01.952238+0000","last_active":"2024-05-27T07:36:45.125526+0000","last_peered":"2024-05-27T07:36:45.125526+0000","last_clean":"2024-05-27T07:36:45.125526+0000","last_became_active":"2024-05-27T07:36:01.951853+0000","last_became_peered":"2024-05-27T07:36:01.951853+0000","last_unstale":"2024-05-27T07:36:45.125526+0000","last_undegraded":"2024-05-27T07:36:45.125526+0000","last_fullsized":"2024-05-27T07:36:45.125526+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:46:41.317531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123012+0000","last_change":"2024-05-27T07:36:15.021848+0000","last_active":"2024-05-27T07:36:45.123012+0000","last_peered":"2024-05-27T07:36:45.123012+0000","last_clean":"2024-05-27T07:36:45.123012+0000","last_became_active":"2024-05-27T07:36:15.021371+0000","last_became_peered":"2024-05-27T07:36:15.021371+0000","last_unstale":"2024-05-27T07:36:45.123012+0000","last_undegraded":"2024-05-27T07:36:45.123012+0000","last_fullsized":"2024-05-27T07:36:45.123012+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:32:21.741229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121786+0000","last_change":"2024-05-27T07:36:02.244283+0000","last_active":"2024-05-27T07:36:45.121786+0000","last_peered":"2024-05-27T07:36:45.121786+0000","last_clean":"2024-05-27T07:36:45.121786+0000","last_became_active":"2024-05-27T07:36:02.243785+0000","last_became_peered":"2024-05-27T07:36:02.243785+0000","last_unstale":"2024-05-27T07:36:45.121786+0000","last_undegraded":"2024-05-27T07:36:45.121786+0000","last_fullsized":"2024-05-27T07:36:45.121786+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:38.596510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125864+0000","last_change":"2024-05-27T07:36:04.979903+0000","last_active":"2024-05-27T07:36:45.125864+0000","last_peered":"2024-05-27T07:36:45.125864+0000","last_clean":"2024-05-27T07:36:45.125864+0000","last_became_active":"2024-05-27T07:36:04.979373+0000","last_became_peered":"2024-05-27T07:36:04.979373+0000","last_unstale":"2024-05-27T07:36:45.125864+0000","last_undegraded":"2024-05-27T07:36:45.125864+0000","last_fullsized":"2024-05-27T07:36:45.125864+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:44:33.763230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139014+0000","last_change":"2024-05-27T07:36:15.023740+0000","last_active":"2024-05-27T07:36:45.139014+0000","last_peered":"2024-05-27T07:36:45.139014+0000","last_clean":"2024-05-27T07:36:45.139014+0000","last_became_active":"2024-05-27T07:36:15.023364+0000","last_became_peered":"2024-05-27T07:36:15.023364+0000","last_unstale":"2024-05-27T07:36:45.139014+0000","last_undegraded":"2024-05-27T07:36:45.139014+0000","last_fullsized":"2024-05-27T07:36:45.139014+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:21.399065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'8","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:37:22.992272+0000","last_change":"2024-05-27T07:36:01.951451+0000","last_active":"2024-05-27T07:37:22.992272+0000","last_peered":"2024-05-27T07:37:22.992272+0000","last_clean":"2024-05-27T07:37:22.992272+0000","last_became_active":"2024-05-27T07:36:01.951037+0000","last_became_peered":"2024-05-27T07:36:01.951037+0000","last_unstale":"2024-05-27T07:37:22.992272+0000","last_undegraded":"2024-05-27T07:37:22.992272+0000","last_fullsized":"2024-05-27T07:37:22.992272+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:44.202474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139579+0000","last_change":"2024-05-27T07:36:04.975392+0000","last_active":"2024-05-27T07:36:45.139579+0000","last_peered":"2024-05-27T07:36:45.139579+0000","last_clean":"2024-05-27T07:36:45.139579+0000","last_became_active":"2024-05-27T07:36:04.975029+0000","last_became_peered":"2024-05-27T07:36:04.975029+0000","last_unstale":"2024-05-27T07:36:45.139579+0000","last_undegraded":"2024-05-27T07:36:45.139579+0000","last_fullsized":"2024-05-27T07:36:45.139579+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:32.022670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139076+0000","last_change":"2024-05-27T07:36:15.026401+0000","last_active":"2024-05-27T07:36:45.139076+0000","last_peered":"2024-05-27T07:36:45.139076+0000","last_clean":"2024-05-27T07:36:45.139076+0000","last_became_active":"2024-05-27T07:36:15.025963+0000","last_became_peered":"2024-05-27T07:36:15.025963+0000","last_unstale":"2024-05-27T07:36:45.139076+0000","last_undegraded":"2024-05-27T07:36:45.139076+0000","last_fullsized":"2024-05-27T07:36:45.139076+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:33:47.616099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"78'2","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119586+0000","last_change":"2024-05-27T07:36:02.228776+0000","last_active":"2024-05-27T07:36:45.119586+0000","last_peered":"2024-05-27T07:36:45.119586+0000","last_clean":"2024-05-27T07:36:45.119586+0000","last_became_active":"2024-05-27T07:36:02.228356+0000","last_became_peered":"2024-05-27T07:36:02.228356+0000","last_unstale":"2024-05-27T07:36:45.119586+0000","last_undegraded":"2024-05-27T07:36:45.119586+0000","last_fullsized":"2024-05-27T07:36:45.119586+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:07.289340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120466+0000","last_change":"2024-05-27T07:36:04.986917+0000","last_active":"2024-05-27T07:36:45.120466+0000","last_peered":"2024-05-27T07:36:45.120466+0000","last_clean":"2024-05-27T07:36:45.120466+0000","last_became_active":"2024-05-27T07:36:04.986432+0000","last_became_peered":"2024-05-27T07:36:04.986432+0000","last_unstale":"2024-05-27T07:36:45.120466+0000","last_undegraded":"2024-05-27T07:36:45.120466+0000","last_fullsized":"2024-05-27T07:36:45.120466+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:39:15.011058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119604+0000","last_change":"2024-05-27T07:36:15.032312+0000","last_active":"2024-05-27T07:36:45.119604+0000","last_peered":"2024-05-27T07:36:45.119604+0000","last_clean":"2024-05-27T07:36:45.119604+0000","last_became_active":"2024-05-27T07:36:15.031475+0000","last_became_peered":"2024-05-27T07:36:15.031475+0000","last_unstale":"2024-05-27T07:36:45.119604+0000","last_undegraded":"2024-05-27T07:36:45.119604+0000","last_fullsized":"2024-05-27T07:36:45.119604+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:40.099421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121220+0000","last_change":"2024-05-27T07:36:02.243456+0000","last_active":"2024-05-27T07:36:45.121220+0000","last_peered":"2024-05-27T07:36:45.121220+0000","last_clean":"2024-05-27T07:36:45.121220+0000","last_became_active":"2024-05-27T07:36:02.243164+0000","last_became_peered":"2024-05-27T07:36:02.243164+0000","last_unstale":"2024-05-27T07:36:45.121220+0000","last_undegraded":"2024-05-27T07:36:45.121220+0000","last_fullsized":"2024-05-27T07:36:45.121220+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:33.765834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120937+0000","last_change":"2024-05-27T07:36:04.980192+0000","last_active":"2024-05-27T07:36:45.120937+0000","last_peered":"2024-05-27T07:36:45.120937+0000","last_clean":"2024-05-27T07:36:45.120937+0000","last_became_active":"2024-05-27T07:36:04.979797+0000","last_became_peered":"2024-05-27T07:36:04.979797+0000","last_unstale":"2024-05-27T07:36:45.120937+0000","last_undegraded":"2024-05-27T07:36:45.120937+0000","last_fullsized":"2024-05-27T07:36:45.120937+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:47:37.006746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122364+0000","last_change":"2024-05-27T07:36:15.033153+0000","last_active":"2024-05-27T07:36:45.122364+0000","last_peered":"2024-05-27T07:36:45.122364+0000","last_clean":"2024-05-27T07:36:45.122364+0000","last_became_active":"2024-05-27T07:36:15.032783+0000","last_became_peered":"2024-05-27T07:36:15.032783+0000","last_unstale":"2024-05-27T07:36:45.122364+0000","last_undegraded":"2024-05-27T07:36:45.122364+0000","last_fullsized":"2024-05-27T07:36:45.122364+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:31:26.237222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139193+0000","last_change":"2024-05-27T07:36:01.952754+0000","last_active":"2024-05-27T07:36:45.139193+0000","last_peered":"2024-05-27T07:36:45.139193+0000","last_clean":"2024-05-27T07:36:45.139193+0000","last_became_active":"2024-05-27T07:36:01.952397+0000","last_became_peered":"2024-05-27T07:36:01.952397+0000","last_unstale":"2024-05-27T07:36:45.139193+0000","last_undegraded":"2024-05-27T07:36:45.139193+0000","last_fullsized":"2024-05-27T07:36:45.139193+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:04.350169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121631+0000","last_change":"2024-05-27T07:36:04.979193+0000","last_active":"2024-05-27T07:36:45.121631+0000","last_peered":"2024-05-27T07:36:45.121631+0000","last_clean":"2024-05-27T07:36:45.121631+0000","last_became_active":"2024-05-27T07:36:04.978679+0000","last_became_peered":"2024-05-27T07:36:04.978679+0000","last_unstale":"2024-05-27T07:36:45.121631+0000","last_undegraded":"2024-05-27T07:36:45.121631+0000","last_fullsized":"2024-05-27T07:36:45.121631+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:59.310284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119389+0000","last_change":"2024-05-27T07:36:15.030150+0000","last_active":"2024-05-27T07:36:45.119389+0000","last_peered":"2024-05-27T07:36:45.119389+0000","last_clean":"2024-05-27T07:36:45.119389+0000","last_became_active":"2024-05-27T07:36:15.029633+0000","last_became_peered":"2024-05-27T07:36:15.029633+0000","last_unstale":"2024-05-27T07:36:45.119389+0000","last_undegraded":"2024-05-27T07:36:45.119389+0000","last_fullsized":"2024-05-27T07:36:45.119389+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:41:39.356585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.140940+0000","last_change":"2024-05-27T07:36:45.140940+0000","last_active":"2024-05-27T07:36:45.140940+0000","last_peered":"2024-05-27T07:36:45.140940+0000","last_clean":"2024-05-27T07:36:45.140940+0000","last_became_active":"2024-05-27T07:36:45.140511+0000","last_became_peered":"2024-05-27T07:36:45.140511+0000","last_unstale":"2024-05-27T07:36:45.140940+0000","last_undegraded":"2024-05-27T07:36:45.140940+0000","last_fullsized":"2024-05-27T07:36:45.140940+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:43:00.507442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139572+0000","last_change":"2024-05-27T07:36:04.974709+0000","last_active":"2024-05-27T07:36:45.139572+0000","last_peered":"2024-05-27T07:36:45.139572+0000","last_clean":"2024-05-27T07:36:45.139572+0000","last_became_active":"2024-05-27T07:36:04.974147+0000","last_became_peered":"2024-05-27T07:36:04.974147+0000","last_unstale":"2024-05-27T07:36:45.139572+0000","last_undegraded":"2024-05-27T07:36:45.139572+0000","last_fullsized":"2024-05-27T07:36:45.139572+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:05:11.251124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119431+0000","last_change":"2024-05-27T07:36:15.023187+0000","last_active":"2024-05-27T07:36:45.119431+0000","last_peered":"2024-05-27T07:36:45.119431+0000","last_clean":"2024-05-27T07:36:45.119431+0000","last_became_active":"2024-05-27T07:36:15.022687+0000","last_became_peered":"2024-05-27T07:36:15.022687+0000","last_unstale":"2024-05-27T07:36:45.119431+0000","last_undegraded":"2024-05-27T07:36:45.119431+0000","last_fullsized":"2024-05-27T07:36:45.119431+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:51.713222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"78'2","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121336+0000","last_change":"2024-05-27T07:36:01.954751+0000","last_active":"2024-05-27T07:36:45.121336+0000","last_peered":"2024-05-27T07:36:45.121336+0000","last_clean":"2024-05-27T07:36:45.121336+0000","last_became_active":"2024-05-27T07:36:01.954173+0000","last_became_peered":"2024-05-27T07:36:01.954173+0000","last_unstale":"2024-05-27T07:36:45.121336+0000","last_undegraded":"2024-05-27T07:36:45.121336+0000","last_fullsized":"2024-05-27T07:36:45.121336+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:45:10.504504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125862+0000","last_change":"2024-05-27T07:36:04.980294+0000","last_active":"2024-05-27T07:36:45.125862+0000","last_peered":"2024-05-27T07:36:45.125862+0000","last_clean":"2024-05-27T07:36:45.125862+0000","last_became_active":"2024-05-27T07:36:04.979853+0000","last_became_peered":"2024-05-27T07:36:04.979853+0000","last_unstale":"2024-05-27T07:36:45.125862+0000","last_undegraded":"2024-05-27T07:36:45.125862+0000","last_fullsized":"2024-05-27T07:36:45.125862+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:43.131216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121293+0000","last_change":"2024-05-27T07:36:15.025854+0000","last_active":"2024-05-27T07:36:45.121293+0000","last_peered":"2024-05-27T07:36:45.121293+0000","last_clean":"2024-05-27T07:36:45.121293+0000","last_became_active":"2024-05-27T07:36:15.025412+0000","last_became_peered":"2024-05-27T07:36:15.025412+0000","last_unstale":"2024-05-27T07:36:45.121293+0000","last_undegraded":"2024-05-27T07:36:45.121293+0000","last_fullsized":"2024-05-27T07:36:45.121293+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:49.904811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139721+0000","last_change":"2024-05-27T07:36:02.243764+0000","last_active":"2024-05-27T07:36:45.139721+0000","last_peered":"2024-05-27T07:36:45.139721+0000","last_clean":"2024-05-27T07:36:45.139721+0000","last_became_active":"2024-05-27T07:36:02.242935+0000","last_became_peered":"2024-05-27T07:36:02.242935+0000","last_unstale":"2024-05-27T07:36:45.139721+0000","last_undegraded":"2024-05-27T07:36:45.139721+0000","last_fullsized":"2024-05-27T07:36:45.139721+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:27.535658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.127072+0000","last_change":"2024-05-27T07:36:04.975291+0000","last_active":"2024-05-27T07:36:45.127072+0000","last_peered":"2024-05-27T07:36:45.127072+0000","last_clean":"2024-05-27T07:36:45.127072+0000","last_became_active":"2024-05-27T07:36:04.974868+0000","last_became_peered":"2024-05-27T07:36:04.974868+0000","last_unstale":"2024-05-27T07:36:45.127072+0000","last_undegraded":"2024-05-27T07:36:45.127072+0000","last_fullsized":"2024-05-27T07:36:45.127072+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:13.890494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120044+0000","last_change":"2024-05-27T07:36:15.031812+0000","last_active":"2024-05-27T07:36:45.120044+0000","last_peered":"2024-05-27T07:36:45.120044+0000","last_clean":"2024-05-27T07:36:45.120044+0000","last_became_active":"2024-05-27T07:36:15.031399+0000","last_became_peered":"2024-05-27T07:36:15.031399+0000","last_unstale":"2024-05-27T07:36:45.120044+0000","last_undegraded":"2024-05-27T07:36:45.120044+0000","last_fullsized":"2024-05-27T07:36:45.120044+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.595579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.124122+0000","last_change":"2024-05-27T07:36:01.954581+0000","last_active":"2024-05-27T07:36:45.124122+0000","last_peered":"2024-05-27T07:36:45.124122+0000","last_clean":"2024-05-27T07:36:45.124122+0000","last_became_active":"2024-05-27T07:36:01.953564+0000","last_became_peered":"2024-05-27T07:36:01.953564+0000","last_unstale":"2024-05-27T07:36:45.124122+0000","last_undegraded":"2024-05-27T07:36:45.124122+0000","last_fullsized":"2024-05-27T07:36:45.124122+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:56.866738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.141160+0000","last_change":"2024-05-27T07:36:45.141160+0000","last_active":"2024-05-27T07:36:45.141160+0000","last_peered":"2024-05-27T07:36:45.141160+0000","last_clean":"2024-05-27T07:36:45.141160+0000","last_became_active":"2024-05-27T07:36:45.140840+0000","last_became_peered":"2024-05-27T07:36:45.140840+0000","last_unstale":"2024-05-27T07:36:45.141160+0000","last_undegraded":"2024-05-27T07:36:45.141160+0000","last_fullsized":"2024-05-27T07:36:45.141160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:31.676808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121020+0000","last_change":"2024-05-27T07:36:15.026483+0000","last_active":"2024-05-27T07:36:45.121020+0000","last_peered":"2024-05-27T07:36:45.121020+0000","last_clean":"2024-05-27T07:36:45.121020+0000","last_became_active":"2024-05-27T07:36:15.025955+0000","last_became_peered":"2024-05-27T07:36:15.025955+0000","last_unstale":"2024-05-27T07:36:45.121020+0000","last_undegraded":"2024-05-27T07:36:45.121020+0000","last_fullsized":"2024-05-27T07:36:45.121020+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:25.468146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126306+0000","last_change":"2024-05-27T07:36:01.952505+0000","last_active":"2024-05-27T07:36:45.126306+0000","last_peered":"2024-05-27T07:36:45.126306+0000","last_clean":"2024-05-27T07:36:45.126306+0000","last_became_active":"2024-05-27T07:36:01.952088+0000","last_became_peered":"2024-05-27T07:36:01.952088+0000","last_unstale":"2024-05-27T07:36:45.126306+0000","last_undegraded":"2024-05-27T07:36:45.126306+0000","last_fullsized":"2024-05-27T07:36:45.126306+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:09.235719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126873+0000","last_change":"2024-05-27T07:36:04.974331+0000","last_active":"2024-05-27T07:36:45.126873+0000","last_peered":"2024-05-27T07:36:45.126873+0000","last_clean":"2024-05-27T07:36:45.126873+0000","last_became_active":"2024-05-27T07:36:04.973825+0000","last_became_peered":"2024-05-27T07:36:04.973825+0000","last_unstale":"2024-05-27T07:36:45.126873+0000","last_undegraded":"2024-05-27T07:36:45.126873+0000","last_fullsized":"2024-05-27T07:36:45.126873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:39:42.917309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.141224+0000","last_change":"2024-05-27T07:36:45.141224+0000","last_active":"2024-05-27T07:36:45.141224+0000","last_peered":"2024-05-27T07:36:45.141224+0000","last_clean":"2024-05-27T07:36:45.141224+0000","last_became_active":"2024-05-27T07:36:45.140776+0000","last_became_peered":"2024-05-27T07:36:45.140776+0000","last_unstale":"2024-05-27T07:36:45.141224+0000","last_undegraded":"2024-05-27T07:36:45.141224+0000","last_fullsized":"2024-05-27T07:36:45.141224+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:36:59.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139425+0000","last_change":"2024-05-27T07:36:04.988247+0000","last_active":"2024-05-27T07:36:45.139425+0000","last_peered":"2024-05-27T07:36:45.139425+0000","last_clean":"2024-05-27T07:36:45.139425+0000","last_became_active":"2024-05-27T07:36:04.987825+0000","last_became_peered":"2024-05-27T07:36:04.987825+0000","last_unstale":"2024-05-27T07:36:45.139425+0000","last_undegraded":"2024-05-27T07:36:45.139425+0000","last_fullsized":"2024-05-27T07:36:45.139425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:32.499260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139200+0000","last_change":"2024-05-27T07:36:04.979395+0000","last_active":"2024-05-27T07:36:45.139200+0000","last_peered":"2024-05-27T07:36:45.139200+0000","last_clean":"2024-05-27T07:36:45.139200+0000","last_became_active":"2024-05-27T07:36:04.978983+0000","last_became_peered":"2024-05-27T07:36:04.978983+0000","last_unstale":"2024-05-27T07:36:45.139200+0000","last_undegraded":"2024-05-27T07:36:45.139200+0000","last_fullsized":"2024-05-27T07:36:45.139200+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:53:54.071958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120393+0000","last_change":"2024-05-27T07:36:04.989442+0000","last_active":"2024-05-27T07:36:45.120393+0000","last_peered":"2024-05-27T07:36:45.120393+0000","last_clean":"2024-05-27T07:36:45.120393+0000","last_became_active":"2024-05-27T07:36:04.989079+0000","last_became_peered":"2024-05-27T07:36:04.989079+0000","last_unstale":"2024-05-27T07:36:45.120393+0000","last_undegraded":"2024-05-27T07:36:45.120393+0000","last_fullsized":"2024-05-27T07:36:45.120393+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:39.411255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125882+0000","last_change":"2024-05-27T07:36:04.981289+0000","last_active":"2024-05-27T07:36:45.125882+0000","last_peered":"2024-05-27T07:36:45.125882+0000","last_clean":"2024-05-27T07:36:45.125882+0000","last_became_active":"2024-05-27T07:36:04.980924+0000","last_became_peered":"2024-05-27T07:36:04.980924+0000","last_unstale":"2024-05-27T07:36:45.125882+0000","last_undegraded":"2024-05-27T07:36:45.125882+0000","last_fullsized":"2024-05-27T07:36:45.125882+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:15:04.741303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123565+0000","last_change":"2024-05-27T07:36:04.985712+0000","last_active":"2024-05-27T07:36:45.123565+0000","last_peered":"2024-05-27T07:36:45.123565+0000","last_clean":"2024-05-27T07:36:45.123565+0000","last_became_active":"2024-05-27T07:36:04.983786+0000","last_became_peered":"2024-05-27T07:36:04.983786+0000","last_unstale":"2024-05-27T07:36:45.123565+0000","last_undegraded":"2024-05-27T07:36:45.123565+0000","last_fullsized":"2024-05-27T07:36:45.123565+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:34.549557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139972+0000","last_change":"2024-05-27T07:36:04.991176+0000","last_active":"2024-05-27T07:36:45.139972+0000","last_peered":"2024-05-27T07:36:45.139972+0000","last_clean":"2024-05-27T07:36:45.139972+0000","last_became_active":"2024-05-27T07:36:04.990774+0000","last_became_peered":"2024-05-27T07:36:04.990774+0000","last_unstale":"2024-05-27T07:36:45.139972+0000","last_undegraded":"2024-05-27T07:36:45.139972+0000","last_fullsized":"2024-05-27T07:36:45.139972+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:37.307581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139739+0000","last_change":"2024-05-27T07:36:04.982337+0000","last_active":"2024-05-27T07:36:45.139739+0000","last_peered":"2024-05-27T07:36:45.139739+0000","last_clean":"2024-05-27T07:36:45.139739+0000","last_became_active":"2024-05-27T07:36:04.981988+0000","last_became_peered":"2024-05-27T07:36:04.981988+0000","last_unstale":"2024-05-27T07:36:45.139739+0000","last_undegraded":"2024-05-27T07:36:45.139739+0000","last_fullsized":"2024-05-27T07:36:45.139739+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:17:39.763459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121522+0000","last_change":"2024-05-27T07:36:04.982321+0000","last_active":"2024-05-27T07:36:45.121522+0000","last_peered":"2024-05-27T07:36:45.121522+0000","last_clean":"2024-05-27T07:36:45.121522+0000","last_became_active":"2024-05-27T07:36:04.981690+0000","last_became_peered":"2024-05-27T07:36:04.981690+0000","last_unstale":"2024-05-27T07:36:45.121522+0000","last_undegraded":"2024-05-27T07:36:45.121522+0000","last_fullsized":"2024-05-27T07:36:45.121522+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:10.181606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139197+0000","last_change":"2024-05-27T07:36:04.979411+0000","last_active":"2024-05-27T07:36:45.139197+0000","last_peered":"2024-05-27T07:36:45.139197+0000","last_clean":"2024-05-27T07:36:45.139197+0000","last_became_active":"2024-05-27T07:36:04.979041+0000","last_became_peered":"2024-05-27T07:36:04.979041+0000","last_unstale":"2024-05-27T07:36:45.139197+0000","last_undegraded":"2024-05-27T07:36:45.139197+0000","last_fullsized":"2024-05-27T07:36:45.139197+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:57:29.165997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121494+0000","last_change":"2024-05-27T07:36:04.988564+0000","last_active":"2024-05-27T07:36:45.121494+0000","last_peered":"2024-05-27T07:36:45.121494+0000","last_clean":"2024-05-27T07:36:45.121494+0000","last_became_active":"2024-05-27T07:36:04.986313+0000","last_became_peered":"2024-05-27T07:36:04.986313+0000","last_unstale":"2024-05-27T07:36:45.121494+0000","last_undegraded":"2024-05-27T07:36:45.121494+0000","last_fullsized":"2024-05-27T07:36:45.121494+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:38.809386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126456+0000","last_change":"2024-05-27T07:36:04.974704+0000","last_active":"2024-05-27T07:36:45.126456+0000","last_peered":"2024-05-27T07:36:45.126456+0000","last_clean":"2024-05-27T07:36:45.126456+0000","last_became_active":"2024-05-27T07:36:04.974311+0000","last_became_peered":"2024-05-27T07:36:04.974311+0000","last_unstale":"2024-05-27T07:36:45.126456+0000","last_undegraded":"2024-05-27T07:36:45.126456+0000","last_fullsized":"2024-05-27T07:36:45.126456+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:43.021149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123205+0000","last_change":"2024-05-27T07:36:04.983951+0000","last_active":"2024-05-27T07:36:45.123205+0000","last_peered":"2024-05-27T07:36:45.123205+0000","last_clean":"2024-05-27T07:36:45.123205+0000","last_became_active":"2024-05-27T07:36:04.983489+0000","last_became_peered":"2024-05-27T07:36:04.983489+0000","last_unstale":"2024-05-27T07:36:45.123205+0000","last_undegraded":"2024-05-27T07:36:45.123205+0000","last_fullsized":"2024-05-27T07:36:45.123205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:04.229523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280976,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":60040,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":59,"num_read_kb":90,"num_write":82,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":182396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710751,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":495672,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906980,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1141358,"data_compressed":62191,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070506,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":500328,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266735,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":496232,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462964,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1090696,"data_compressed":62191,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626490,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":491554,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":551934,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":950272,"data_stored":491644,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":551968,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":491658,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986245,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":495672,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822718,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":1086200,"data_compressed":62191,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34107,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34107,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34107,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":60414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":59436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:37:31.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:37:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:31.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:31 smithi163 ceph-mon[36451]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:37:31.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:31 smithi186 ceph-mon[36647]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:37:31.756 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:31 smithi114 ceph-mon[32038]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:37:32.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:32 smithi163 ceph-mon[36451]: pgmap v341: 129 pgs: 129 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.7 KiB/s wr, 4 op/s 2024-05-27T07:37:32.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:32 smithi186 ceph-mon[36647]: pgmap v341: 129 pgs: 129 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.7 KiB/s wr, 4 op/s 2024-05-27T07:37:32.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:32 smithi114 ceph-mon[32038]: pgmap v341: 129 pgs: 129 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.7 KiB/s wr, 4 op/s 2024-05-27T07:37:33.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.073+0000 7f0b62cef700 1 -- 172.21.15.114:0/100946525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b5c1034b0 msgr2=0x7f0b5c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:33.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.073+0000 7f0b62cef700 1 --2- 172.21.15.114:0/100946525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b5c1034b0 0x7f0b5c1038c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0b540099c0 tx=0x7f0b54009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.073+0000 7f0b62cef700 1 -- 172.21.15.114:0/100946525 shutdown_connections 2024-05-27T07:37:33.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.073+0000 7f0b62cef700 1 --2- 172.21.15.114:0/100946525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b5c108730 0x7f0b5c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.073+0000 7f0b62cef700 1 --2- 172.21.15.114:0/100946525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b5c1034b0 0x7f0b5c1038c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.073+0000 7f0b62cef700 1 --2- 172.21.15.114:0/100946525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b5c107de0 0x7f0b5c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.073+0000 7f0b62cef700 1 -- 172.21.15.114:0/100946525 >> 172.21.15.114:0/100946525 conn(0x7f0b5c0fedb0 msgr2=0x7f0b5c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:33.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.074+0000 7f0b62cef700 1 -- 172.21.15.114:0/100946525 shutdown_connections 2024-05-27T07:37:33.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.074+0000 7f0b62cef700 1 -- 172.21.15.114:0/100946525 wait complete. 2024-05-27T07:37:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.074+0000 7f0b62cef700 1 Processor -- start 2024-05-27T07:37:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.074+0000 7f0b62cef700 1 -- start start 2024-05-27T07:37:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.074+0000 7f0b62cef700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b5c1034b0 0x7f0b5c0ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b62cef700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b5c107de0 0x7f0b5c0ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b62cef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b5c108730 0x7f0b5c0ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b62cef700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5c1133e0 con 0x7f0b5c108730 2024-05-27T07:37:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b62cef700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5c113260 con 0x7f0b5c107de0 2024-05-27T07:37:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b62cef700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5c113560 con 0x7f0b5c1034b0 2024-05-27T07:37:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b5bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b5c107de0 0x7f0b5c0ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b6128c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b5c108730 0x7f0b5c0ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b5bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b5c107de0 0x7f0b5c0ff7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:58768/0 (socket says 172.21.15.114:58768) 2024-05-27T07:37:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b5bfff700 1 -- 172.21.15.114:0/2263458626 learned_addr learned my addr 172.21.15.114:0/2263458626 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b60a8b700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b5c1034b0 0x7f0b5c0ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b6128c700 1 -- 172.21.15.114:0/2263458626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b5c1034b0 msgr2=0x7f0b5c0ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b6128c700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b5c1034b0 0x7f0b5c0ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b6128c700 1 -- 172.21.15.114:0/2263458626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b5c107de0 msgr2=0x7f0b5c0ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b6128c700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b5c107de0 0x7f0b5c0ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b6128c700 1 -- 172.21.15.114:0/2263458626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b5c1a91c0 con 0x7f0b5c108730 2024-05-27T07:37:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b5bfff700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b5c107de0 0x7f0b5c0ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:37:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b6128c700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b5c108730 0x7f0b5c0ffd10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0b4400b3f0 tx=0x7f0b4400b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b59ffb700 1 -- 172.21.15.114:0/2263458626 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b44015020 con 0x7f0b5c108730 2024-05-27T07:37:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.075+0000 7f0b59ffb700 1 -- 172.21.15.114:0/2263458626 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b44011ae0 con 0x7f0b5c108730 2024-05-27T07:37:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.076+0000 7f0b62cef700 1 -- 172.21.15.114:0/2263458626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b5c1a94b0 con 0x7f0b5c108730 2024-05-27T07:37:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.076+0000 7f0b59ffb700 1 -- 172.21.15.114:0/2263458626 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b44010bd0 con 0x7f0b5c108730 2024-05-27T07:37:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.076+0000 7f0b62cef700 1 -- 172.21.15.114:0/2263458626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b5c1a99a0 con 0x7f0b5c108730 2024-05-27T07:37:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.077+0000 7f0b62cef700 1 -- 172.21.15.114:0/2263458626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b5c0512b0 con 0x7f0b5c108730 2024-05-27T07:37:33.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.078+0000 7f0b59ffb700 1 -- 172.21.15.114:0/2263458626 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f0b44018020 con 0x7f0b5c108730 2024-05-27T07:37:33.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.079+0000 7f0b59ffb700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0b480b2910 0x7f0b480b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:33.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.079+0000 7f0b60a8b700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0b480b2910 0x7f0b480b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:33.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.079+0000 7f0b59ffb700 1 -- 172.21.15.114:0/2263458626 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f0b440e2100 con 0x7f0b5c108730 2024-05-27T07:37:33.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.079+0000 7f0b60a8b700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0b480b2910 0x7f0b480b4e20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0b54009990 tx=0x7f0b540036e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:33.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.082+0000 7f0b59ffb700 1 -- 172.21.15.114:0/2263458626 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0b4401f020 con 0x7f0b5c108730 2024-05-27T07:37:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.279+0000 7f0b62cef700 1 -- 172.21.15.114:0/2263458626 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0b5c063bf0 con 0x7f0b480b2910 2024-05-27T07:37:33.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.288+0000 7f0b59ffb700 1 -- 172.21.15.114:0/2263458626 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340543 (secure 0 0 0) 0x7f0b5407b250 con 0x7f0b480b2910 2024-05-27T07:37:33.290 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:37:33.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.293+0000 7f0b62cef700 1 -- 172.21.15.114:0/2263458626 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0b480b2910 msgr2=0x7f0b480b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:33.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.293+0000 7f0b62cef700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0b480b2910 0x7f0b480b4e20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0b54009990 tx=0x7f0b540036e0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.293+0000 7f0b62cef700 1 -- 172.21.15.114:0/2263458626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b5c108730 msgr2=0x7f0b5c0ffd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:33.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.293+0000 7f0b62cef700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b5c108730 0x7f0b5c0ffd10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0b4400b3f0 tx=0x7f0b4400b700 comp rx=0 tx=0).stop 2024-05-27T07:37:33.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.293+0000 7f0b62cef700 1 -- 172.21.15.114:0/2263458626 shutdown_connections 2024-05-27T07:37:33.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.293+0000 7f0b62cef700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b5c108730 0x7f0b5c0ffd10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.293+0000 7f0b62cef700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0b480b2910 0x7f0b480b4e20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.293+0000 7f0b62cef700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b5c1034b0 0x7f0b5c0ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.293+0000 7f0b62cef700 1 --2- 172.21.15.114:0/2263458626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b5c107de0 0x7f0b5c0ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.293+0000 7f0b62cef700 1 -- 172.21.15.114:0/2263458626 >> 172.21.15.114:0/2263458626 conn(0x7f0b5c0fedb0 msgr2=0x7f0b5c06b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:33.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.293+0000 7f0b62cef700 1 -- 172.21.15.114:0/2263458626 shutdown_connections 2024-05-27T07:37:33.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:33.293+0000 7f0b62cef700 1 -- 172.21.15.114:0/2263458626 wait complete. 2024-05-27T07:37:33.296 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-05-27T07:37:34.074 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-05-27T07:37:31.390217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651780,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":151,"num_read_kb":166,"num_write":209,"num_write_kb":2241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164,"ondisk_log_size":164,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364852,"kb_used_data":11528,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576972,"statfs":{"total":1151940427776,"available":1151566819328,"internally_reserved":0,"allocated":11804672,"data_stored":7886054,"data_compressed":424713,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35804,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":52,"num_write":23,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003992"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125405+0000","last_change":"2024-05-27T07:36:04.980329+0000","last_active":"2024-05-27T07:36:45.125405+0000","last_peered":"2024-05-27T07:36:45.125405+0000","last_clean":"2024-05-27T07:36:45.125405+0000","last_became_active":"2024-05-27T07:36:04.980006+0000","last_became_peered":"2024-05-27T07:36:04.980006+0000","last_unstale":"2024-05-27T07:36:45.125405+0000","last_undegraded":"2024-05-27T07:36:45.125405+0000","last_fullsized":"2024-05-27T07:36:45.125405+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:51:56.728465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120442+0000","last_change":"2024-05-27T07:36:04.983429+0000","last_active":"2024-05-27T07:36:45.120442+0000","last_peered":"2024-05-27T07:36:45.120442+0000","last_clean":"2024-05-27T07:36:45.120442+0000","last_became_active":"2024-05-27T07:36:04.983099+0000","last_became_peered":"2024-05-27T07:36:04.983099+0000","last_unstale":"2024-05-27T07:36:45.120442+0000","last_undegraded":"2024-05-27T07:36:45.120442+0000","last_fullsized":"2024-05-27T07:36:45.120442+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:18:51.059717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120202+0000","last_change":"2024-05-27T07:36:04.993319+0000","last_active":"2024-05-27T07:36:45.120202+0000","last_peered":"2024-05-27T07:36:45.120202+0000","last_clean":"2024-05-27T07:36:45.120202+0000","last_became_active":"2024-05-27T07:36:04.992983+0000","last_became_peered":"2024-05-27T07:36:04.992983+0000","last_unstale":"2024-05-27T07:36:45.120202+0000","last_undegraded":"2024-05-27T07:36:45.120202+0000","last_fullsized":"2024-05-27T07:36:45.120202+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:50.800194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.141492+0000","last_change":"2024-05-27T07:36:45.140980+0000","last_active":"2024-05-27T07:36:45.141492+0000","last_peered":"2024-05-27T07:36:45.141492+0000","last_clean":"2024-05-27T07:36:45.141492+0000","last_became_active":"2024-05-27T07:36:45.140551+0000","last_became_peered":"2024-05-27T07:36:45.140551+0000","last_unstale":"2024-05-27T07:36:45.141492+0000","last_undegraded":"2024-05-27T07:36:45.141492+0000","last_fullsized":"2024-05-27T07:36:45.141492+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:59:21.646306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121156+0000","last_change":"2024-05-27T07:36:04.977404+0000","last_active":"2024-05-27T07:36:45.121156+0000","last_peered":"2024-05-27T07:36:45.121156+0000","last_clean":"2024-05-27T07:36:45.121156+0000","last_became_active":"2024-05-27T07:36:04.976991+0000","last_became_peered":"2024-05-27T07:36:04.976991+0000","last_unstale":"2024-05-27T07:36:45.121156+0000","last_undegraded":"2024-05-27T07:36:45.121156+0000","last_fullsized":"2024-05-27T07:36:45.121156+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:56.532586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.124398+0000","last_change":"2024-05-27T07:36:04.982149+0000","last_active":"2024-05-27T07:36:45.124398+0000","last_peered":"2024-05-27T07:36:45.124398+0000","last_clean":"2024-05-27T07:36:45.124398+0000","last_became_active":"2024-05-27T07:36:04.981751+0000","last_became_peered":"2024-05-27T07:36:04.981751+0000","last_unstale":"2024-05-27T07:36:45.124398+0000","last_undegraded":"2024-05-27T07:36:45.124398+0000","last_fullsized":"2024-05-27T07:36:45.124398+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:42.773458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125617+0000","last_change":"2024-05-27T07:36:04.987491+0000","last_active":"2024-05-27T07:36:45.125617+0000","last_peered":"2024-05-27T07:36:45.125617+0000","last_clean":"2024-05-27T07:36:45.125617+0000","last_became_active":"2024-05-27T07:36:04.987059+0000","last_became_peered":"2024-05-27T07:36:04.987059+0000","last_unstale":"2024-05-27T07:36:45.125617+0000","last_undegraded":"2024-05-27T07:36:45.125617+0000","last_fullsized":"2024-05-27T07:36:45.125617+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:57.611253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.127432+0000","last_change":"2024-05-27T07:36:45.126961+0000","last_active":"2024-05-27T07:36:45.127432+0000","last_peered":"2024-05-27T07:36:45.127432+0000","last_clean":"2024-05-27T07:36:45.127432+0000","last_became_active":"2024-05-27T07:36:45.126653+0000","last_became_peered":"2024-05-27T07:36:45.126653+0000","last_unstale":"2024-05-27T07:36:45.127432+0000","last_undegraded":"2024-05-27T07:36:45.127432+0000","last_fullsized":"2024-05-27T07:36:45.127432+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:35.586128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121844+0000","last_change":"2024-05-27T07:36:04.990469+0000","last_active":"2024-05-27T07:36:45.121844+0000","last_peered":"2024-05-27T07:36:45.121844+0000","last_clean":"2024-05-27T07:36:45.121844+0000","last_became_active":"2024-05-27T07:36:04.989771+0000","last_became_peered":"2024-05-27T07:36:04.989771+0000","last_unstale":"2024-05-27T07:36:45.121844+0000","last_undegraded":"2024-05-27T07:36:45.121844+0000","last_fullsized":"2024-05-27T07:36:45.121844+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:53.593566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125310+0000","last_change":"2024-05-27T07:36:04.984746+0000","last_active":"2024-05-27T07:36:45.125310+0000","last_peered":"2024-05-27T07:36:45.125310+0000","last_clean":"2024-05-27T07:36:45.125310+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:36:45.125310+0000","last_undegraded":"2024-05-27T07:36:45.125310+0000","last_fullsized":"2024-05-27T07:36:45.125310+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:18.071981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126865+0000","last_change":"2024-05-27T07:36:04.977167+0000","last_active":"2024-05-27T07:36:45.126865+0000","last_peered":"2024-05-27T07:36:45.126865+0000","last_clean":"2024-05-27T07:36:45.126865+0000","last_became_active":"2024-05-27T07:36:04.976788+0000","last_became_peered":"2024-05-27T07:36:04.976788+0000","last_unstale":"2024-05-27T07:36:45.126865+0000","last_undegraded":"2024-05-27T07:36:45.126865+0000","last_fullsized":"2024-05-27T07:36:45.126865+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:59:44.574191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119713+0000","last_change":"2024-05-27T07:36:04.986594+0000","last_active":"2024-05-27T07:36:45.119713+0000","last_peered":"2024-05-27T07:36:45.119713+0000","last_clean":"2024-05-27T07:36:45.119713+0000","last_became_active":"2024-05-27T07:36:04.985161+0000","last_became_peered":"2024-05-27T07:36:04.985161+0000","last_unstale":"2024-05-27T07:36:45.119713+0000","last_undegraded":"2024-05-27T07:36:45.119713+0000","last_fullsized":"2024-05-27T07:36:45.119713+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.239461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125988+0000","last_change":"2024-05-27T07:36:04.984781+0000","last_active":"2024-05-27T07:36:45.125988+0000","last_peered":"2024-05-27T07:36:45.125988+0000","last_clean":"2024-05-27T07:36:45.125988+0000","last_became_active":"2024-05-27T07:36:04.984462+0000","last_became_peered":"2024-05-27T07:36:04.984462+0000","last_unstale":"2024-05-27T07:36:45.125988+0000","last_undegraded":"2024-05-27T07:36:45.125988+0000","last_fullsized":"2024-05-27T07:36:45.125988+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:40.051692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120044+0000","last_change":"2024-05-27T07:36:04.978360+0000","last_active":"2024-05-27T07:36:45.120044+0000","last_peered":"2024-05-27T07:36:45.120044+0000","last_clean":"2024-05-27T07:36:45.120044+0000","last_became_active":"2024-05-27T07:36:04.978105+0000","last_became_peered":"2024-05-27T07:36:04.978105+0000","last_unstale":"2024-05-27T07:36:45.120044+0000","last_undegraded":"2024-05-27T07:36:45.120044+0000","last_fullsized":"2024-05-27T07:36:45.120044+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:04:47.257529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120064+0000","last_change":"2024-05-27T07:36:04.986510+0000","last_active":"2024-05-27T07:36:45.120064+0000","last_peered":"2024-05-27T07:36:45.120064+0000","last_clean":"2024-05-27T07:36:45.120064+0000","last_became_active":"2024-05-27T07:36:04.986150+0000","last_became_peered":"2024-05-27T07:36:04.986150+0000","last_unstale":"2024-05-27T07:36:45.120064+0000","last_undegraded":"2024-05-27T07:36:45.120064+0000","last_fullsized":"2024-05-27T07:36:45.120064+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:07.349470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121156+0000","last_change":"2024-05-27T07:36:04.979880+0000","last_active":"2024-05-27T07:36:45.121156+0000","last_peered":"2024-05-27T07:36:45.121156+0000","last_clean":"2024-05-27T07:36:45.121156+0000","last_became_active":"2024-05-27T07:36:04.979516+0000","last_became_peered":"2024-05-27T07:36:04.979516+0000","last_unstale":"2024-05-27T07:36:45.121156+0000","last_undegraded":"2024-05-27T07:36:45.121156+0000","last_fullsized":"2024-05-27T07:36:45.121156+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:41:03.964277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125103+0000","last_change":"2024-05-27T07:36:04.980231+0000","last_active":"2024-05-27T07:36:45.125103+0000","last_peered":"2024-05-27T07:36:45.125103+0000","last_clean":"2024-05-27T07:36:45.125103+0000","last_became_active":"2024-05-27T07:36:04.979888+0000","last_became_peered":"2024-05-27T07:36:04.979888+0000","last_unstale":"2024-05-27T07:36:45.125103+0000","last_undegraded":"2024-05-27T07:36:45.125103+0000","last_fullsized":"2024-05-27T07:36:45.125103+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.838119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122786+0000","last_change":"2024-05-27T07:36:04.990890+0000","last_active":"2024-05-27T07:36:45.122786+0000","last_peered":"2024-05-27T07:36:45.122786+0000","last_clean":"2024-05-27T07:36:45.122786+0000","last_became_active":"2024-05-27T07:36:04.990581+0000","last_became_peered":"2024-05-27T07:36:04.990581+0000","last_unstale":"2024-05-27T07:36:45.122786+0000","last_undegraded":"2024-05-27T07:36:45.122786+0000","last_fullsized":"2024-05-27T07:36:45.122786+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:33.054395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139872+0000","last_change":"2024-05-27T07:36:04.981620+0000","last_active":"2024-05-27T07:36:45.139872+0000","last_peered":"2024-05-27T07:36:45.139872+0000","last_clean":"2024-05-27T07:36:45.139872+0000","last_became_active":"2024-05-27T07:36:04.981290+0000","last_became_peered":"2024-05-27T07:36:04.981290+0000","last_unstale":"2024-05-27T07:36:45.139872+0000","last_undegraded":"2024-05-27T07:36:45.139872+0000","last_fullsized":"2024-05-27T07:36:45.139872+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:50.287380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139135+0000","last_change":"2024-05-27T07:36:15.030740+0000","last_active":"2024-05-27T07:36:45.139135+0000","last_peered":"2024-05-27T07:36:45.139135+0000","last_clean":"2024-05-27T07:36:45.139135+0000","last_became_active":"2024-05-27T07:36:15.029738+0000","last_became_peered":"2024-05-27T07:36:15.029738+0000","last_unstale":"2024-05-27T07:36:45.139135+0000","last_undegraded":"2024-05-27T07:36:45.139135+0000","last_fullsized":"2024-05-27T07:36:45.139135+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:18.101941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"78'4","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122593+0000","last_change":"2024-05-27T07:36:01.955530+0000","last_active":"2024-05-27T07:36:45.122593+0000","last_peered":"2024-05-27T07:36:45.122593+0000","last_clean":"2024-05-27T07:36:45.122593+0000","last_became_active":"2024-05-27T07:36:01.955225+0000","last_became_peered":"2024-05-27T07:36:01.955225+0000","last_unstale":"2024-05-27T07:36:45.122593+0000","last_undegraded":"2024-05-27T07:36:45.122593+0000","last_fullsized":"2024-05-27T07:36:45.122593+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:02.067672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139432+0000","last_change":"2024-05-27T07:36:04.984397+0000","last_active":"2024-05-27T07:36:45.139432+0000","last_peered":"2024-05-27T07:36:45.139432+0000","last_clean":"2024-05-27T07:36:45.139432+0000","last_became_active":"2024-05-27T07:36:04.984033+0000","last_became_peered":"2024-05-27T07:36:04.984033+0000","last_unstale":"2024-05-27T07:36:45.139432+0000","last_undegraded":"2024-05-27T07:36:45.139432+0000","last_fullsized":"2024-05-27T07:36:45.139432+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:15.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.118802+0000","last_change":"2024-05-27T07:36:15.028412+0000","last_active":"2024-05-27T07:36:45.118802+0000","last_peered":"2024-05-27T07:36:45.118802+0000","last_clean":"2024-05-27T07:36:45.118802+0000","last_became_active":"2024-05-27T07:36:15.027886+0000","last_became_peered":"2024-05-27T07:36:15.027886+0000","last_unstale":"2024-05-27T07:36:45.118802+0000","last_undegraded":"2024-05-27T07:36:45.118802+0000","last_fullsized":"2024-05-27T07:36:45.118802+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:56:26.383126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"78'1","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139601+0000","last_change":"2024-05-27T07:36:02.243079+0000","last_active":"2024-05-27T07:36:45.139601+0000","last_peered":"2024-05-27T07:36:45.139601+0000","last_clean":"2024-05-27T07:36:45.139601+0000","last_became_active":"2024-05-27T07:36:02.242633+0000","last_became_peered":"2024-05-27T07:36:02.242633+0000","last_unstale":"2024-05-27T07:36:45.139601+0000","last_undegraded":"2024-05-27T07:36:45.139601+0000","last_fullsized":"2024-05-27T07:36:45.139601+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:45:33.980842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120596+0000","last_change":"2024-05-27T07:36:04.989775+0000","last_active":"2024-05-27T07:36:45.120596+0000","last_peered":"2024-05-27T07:36:45.120596+0000","last_clean":"2024-05-27T07:36:45.120596+0000","last_became_active":"2024-05-27T07:36:04.989289+0000","last_became_peered":"2024-05-27T07:36:04.989289+0000","last_unstale":"2024-05-27T07:36:45.120596+0000","last_undegraded":"2024-05-27T07:36:45.120596+0000","last_fullsized":"2024-05-27T07:36:45.120596+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:24:38.355231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119229+0000","last_change":"2024-05-27T07:36:15.031737+0000","last_active":"2024-05-27T07:36:45.119229+0000","last_peered":"2024-05-27T07:36:45.119229+0000","last_clean":"2024-05-27T07:36:45.119229+0000","last_became_active":"2024-05-27T07:36:15.031314+0000","last_became_peered":"2024-05-27T07:36:15.031314+0000","last_unstale":"2024-05-27T07:36:45.119229+0000","last_undegraded":"2024-05-27T07:36:45.119229+0000","last_fullsized":"2024-05-27T07:36:45.119229+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:32.742298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"78'2","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121953+0000","last_change":"2024-05-27T07:36:01.957459+0000","last_active":"2024-05-27T07:36:45.121953+0000","last_peered":"2024-05-27T07:36:45.121953+0000","last_clean":"2024-05-27T07:36:45.121953+0000","last_became_active":"2024-05-27T07:36:01.954985+0000","last_became_peered":"2024-05-27T07:36:01.954985+0000","last_unstale":"2024-05-27T07:36:45.121953+0000","last_undegraded":"2024-05-27T07:36:45.121953+0000","last_fullsized":"2024-05-27T07:36:45.121953+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:27.085486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121538+0000","last_change":"2024-05-27T07:36:04.986447+0000","last_active":"2024-05-27T07:36:45.121538+0000","last_peered":"2024-05-27T07:36:45.121538+0000","last_clean":"2024-05-27T07:36:45.121538+0000","last_became_active":"2024-05-27T07:36:04.982099+0000","last_became_peered":"2024-05-27T07:36:04.982099+0000","last_unstale":"2024-05-27T07:36:45.121538+0000","last_undegraded":"2024-05-27T07:36:45.121538+0000","last_fullsized":"2024-05-27T07:36:45.121538+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:32.637479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.118853+0000","last_change":"2024-05-27T07:36:15.024152+0000","last_active":"2024-05-27T07:36:45.118853+0000","last_peered":"2024-05-27T07:36:45.118853+0000","last_clean":"2024-05-27T07:36:45.118853+0000","last_became_active":"2024-05-27T07:36:15.023711+0000","last_became_peered":"2024-05-27T07:36:15.023711+0000","last_unstale":"2024-05-27T07:36:45.118853+0000","last_undegraded":"2024-05-27T07:36:45.118853+0000","last_fullsized":"2024-05-27T07:36:45.118853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:33.809364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121281+0000","last_change":"2024-05-27T07:36:01.950531+0000","last_active":"2024-05-27T07:36:45.121281+0000","last_peered":"2024-05-27T07:36:45.121281+0000","last_clean":"2024-05-27T07:36:45.121281+0000","last_became_active":"2024-05-27T07:36:01.950015+0000","last_became_peered":"2024-05-27T07:36:01.950015+0000","last_unstale":"2024-05-27T07:36:45.121281+0000","last_undegraded":"2024-05-27T07:36:45.121281+0000","last_fullsized":"2024-05-27T07:36:45.121281+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:22.400415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123907+0000","last_change":"2024-05-27T07:36:04.982792+0000","last_active":"2024-05-27T07:36:45.123907+0000","last_peered":"2024-05-27T07:36:45.123907+0000","last_clean":"2024-05-27T07:36:45.123907+0000","last_became_active":"2024-05-27T07:36:04.982400+0000","last_became_peered":"2024-05-27T07:36:04.982400+0000","last_unstale":"2024-05-27T07:36:45.123907+0000","last_undegraded":"2024-05-27T07:36:45.123907+0000","last_fullsized":"2024-05-27T07:36:45.123907+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:27.823654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139531+0000","last_change":"2024-05-27T07:36:15.027932+0000","last_active":"2024-05-27T07:36:45.139531+0000","last_peered":"2024-05-27T07:36:45.139531+0000","last_clean":"2024-05-27T07:36:45.139531+0000","last_became_active":"2024-05-27T07:36:15.027526+0000","last_became_peered":"2024-05-27T07:36:15.027526+0000","last_unstale":"2024-05-27T07:36:45.139531+0000","last_undegraded":"2024-05-27T07:36:45.139531+0000","last_fullsized":"2024-05-27T07:36:45.139531+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:16.165934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"78'1","reported_seq":46,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.127637+0000","last_change":"2024-05-27T07:36:45.127637+0000","last_active":"2024-05-27T07:36:45.127637+0000","last_peered":"2024-05-27T07:36:45.127637+0000","last_clean":"2024-05-27T07:36:45.127637+0000","last_became_active":"2024-05-27T07:36:45.127249+0000","last_became_peered":"2024-05-27T07:36:45.127249+0000","last_unstale":"2024-05-27T07:36:45.127637+0000","last_undegraded":"2024-05-27T07:36:45.127637+0000","last_fullsized":"2024-05-27T07:36:45.127637+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:52.418069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121167+0000","last_change":"2024-05-27T07:36:04.985135+0000","last_active":"2024-05-27T07:36:45.121167+0000","last_peered":"2024-05-27T07:36:45.121167+0000","last_clean":"2024-05-27T07:36:45.121167+0000","last_became_active":"2024-05-27T07:36:04.983621+0000","last_became_peered":"2024-05-27T07:36:04.983621+0000","last_unstale":"2024-05-27T07:36:45.121167+0000","last_undegraded":"2024-05-27T07:36:45.121167+0000","last_fullsized":"2024-05-27T07:36:45.121167+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:11.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139593+0000","last_change":"2024-05-27T07:36:15.024558+0000","last_active":"2024-05-27T07:36:45.139593+0000","last_peered":"2024-05-27T07:36:45.139593+0000","last_clean":"2024-05-27T07:36:45.139593+0000","last_became_active":"2024-05-27T07:36:15.024148+0000","last_became_peered":"2024-05-27T07:36:15.024148+0000","last_unstale":"2024-05-27T07:36:45.139593+0000","last_undegraded":"2024-05-27T07:36:45.139593+0000","last_fullsized":"2024-05-27T07:36:45.139593+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:03.648451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"78'1","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.129413+0000","last_change":"2024-05-27T07:36:45.129413+0000","last_active":"2024-05-27T07:36:45.129413+0000","last_peered":"2024-05-27T07:36:45.129413+0000","last_clean":"2024-05-27T07:36:45.129413+0000","last_became_active":"2024-05-27T07:36:45.128779+0000","last_became_peered":"2024-05-27T07:36:45.128779+0000","last_unstale":"2024-05-27T07:36:45.129413+0000","last_undegraded":"2024-05-27T07:36:45.129413+0000","last_fullsized":"2024-05-27T07:36:45.129413+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:05.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123709+0000","last_change":"2024-05-27T07:36:04.989878+0000","last_active":"2024-05-27T07:36:45.123709+0000","last_peered":"2024-05-27T07:36:45.123709+0000","last_clean":"2024-05-27T07:36:45.123709+0000","last_became_active":"2024-05-27T07:36:04.989532+0000","last_became_peered":"2024-05-27T07:36:04.989532+0000","last_unstale":"2024-05-27T07:36:45.123709+0000","last_undegraded":"2024-05-27T07:36:45.123709+0000","last_fullsized":"2024-05-27T07:36:45.123709+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:54:53.909592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.124963+0000","last_change":"2024-05-27T07:36:15.023739+0000","last_active":"2024-05-27T07:36:45.124963+0000","last_peered":"2024-05-27T07:36:45.124963+0000","last_clean":"2024-05-27T07:36:45.124963+0000","last_became_active":"2024-05-27T07:36:15.023245+0000","last_became_peered":"2024-05-27T07:36:15.023245+0000","last_unstale":"2024-05-27T07:36:45.124963+0000","last_undegraded":"2024-05-27T07:36:45.124963+0000","last_fullsized":"2024-05-27T07:36:45.124963+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:56:12.585673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125698+0000","last_change":"2024-05-27T07:36:01.949684+0000","last_active":"2024-05-27T07:36:45.125698+0000","last_peered":"2024-05-27T07:36:45.125698+0000","last_clean":"2024-05-27T07:36:45.125698+0000","last_became_active":"2024-05-27T07:36:01.949308+0000","last_became_peered":"2024-05-27T07:36:01.949308+0000","last_unstale":"2024-05-27T07:36:45.125698+0000","last_undegraded":"2024-05-27T07:36:45.125698+0000","last_fullsized":"2024-05-27T07:36:45.125698+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:50:54.859172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139012+0000","last_change":"2024-05-27T07:36:04.971266+0000","last_active":"2024-05-27T07:36:45.139012+0000","last_peered":"2024-05-27T07:36:45.139012+0000","last_clean":"2024-05-27T07:36:45.139012+0000","last_became_active":"2024-05-27T07:36:04.970831+0000","last_became_peered":"2024-05-27T07:36:04.970831+0000","last_unstale":"2024-05-27T07:36:45.139012+0000","last_undegraded":"2024-05-27T07:36:45.139012+0000","last_fullsized":"2024-05-27T07:36:45.139012+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:37.237637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120359+0000","last_change":"2024-05-27T07:36:15.024415+0000","last_active":"2024-05-27T07:36:45.120359+0000","last_peered":"2024-05-27T07:36:45.120359+0000","last_clean":"2024-05-27T07:36:45.120359+0000","last_became_active":"2024-05-27T07:36:15.023935+0000","last_became_peered":"2024-05-27T07:36:15.023935+0000","last_unstale":"2024-05-27T07:36:45.120359+0000","last_undegraded":"2024-05-27T07:36:45.120359+0000","last_fullsized":"2024-05-27T07:36:45.120359+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:37:30.386933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121375+0000","last_change":"2024-05-27T07:36:01.952256+0000","last_active":"2024-05-27T07:36:45.121375+0000","last_peered":"2024-05-27T07:36:45.121375+0000","last_clean":"2024-05-27T07:36:45.121375+0000","last_became_active":"2024-05-27T07:36:01.951876+0000","last_became_peered":"2024-05-27T07:36:01.951876+0000","last_unstale":"2024-05-27T07:36:45.121375+0000","last_undegraded":"2024-05-27T07:36:45.121375+0000","last_fullsized":"2024-05-27T07:36:45.121375+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:05.201780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121791+0000","last_change":"2024-05-27T07:36:04.980300+0000","last_active":"2024-05-27T07:36:45.121791+0000","last_peered":"2024-05-27T07:36:45.121791+0000","last_clean":"2024-05-27T07:36:45.121791+0000","last_became_active":"2024-05-27T07:36:04.979954+0000","last_became_peered":"2024-05-27T07:36:04.979954+0000","last_unstale":"2024-05-27T07:36:45.121791+0000","last_undegraded":"2024-05-27T07:36:45.121791+0000","last_fullsized":"2024-05-27T07:36:45.121791+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:15.712773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139072+0000","last_change":"2024-05-27T07:36:15.026685+0000","last_active":"2024-05-27T07:36:45.139072+0000","last_peered":"2024-05-27T07:36:45.139072+0000","last_clean":"2024-05-27T07:36:45.139072+0000","last_became_active":"2024-05-27T07:36:15.026287+0000","last_became_peered":"2024-05-27T07:36:15.026287+0000","last_unstale":"2024-05-27T07:36:45.139072+0000","last_undegraded":"2024-05-27T07:36:45.139072+0000","last_fullsized":"2024-05-27T07:36:45.139072+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:29:26.394435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125066+0000","last_change":"2024-05-27T07:36:01.956566+0000","last_active":"2024-05-27T07:36:45.125066+0000","last_peered":"2024-05-27T07:36:45.125066+0000","last_clean":"2024-05-27T07:36:45.125066+0000","last_became_active":"2024-05-27T07:36:01.955963+0000","last_became_peered":"2024-05-27T07:36:01.955963+0000","last_unstale":"2024-05-27T07:36:45.125066+0000","last_undegraded":"2024-05-27T07:36:45.125066+0000","last_fullsized":"2024-05-27T07:36:45.125066+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:53.081784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122035+0000","last_change":"2024-05-27T07:36:04.983878+0000","last_active":"2024-05-27T07:36:45.122035+0000","last_peered":"2024-05-27T07:36:45.122035+0000","last_clean":"2024-05-27T07:36:45.122035+0000","last_became_active":"2024-05-27T07:36:04.983339+0000","last_became_peered":"2024-05-27T07:36:04.983339+0000","last_unstale":"2024-05-27T07:36:45.122035+0000","last_undegraded":"2024-05-27T07:36:45.122035+0000","last_fullsized":"2024-05-27T07:36:45.122035+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:57:01.726122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121569+0000","last_change":"2024-05-27T07:36:15.027923+0000","last_active":"2024-05-27T07:36:45.121569+0000","last_peered":"2024-05-27T07:36:45.121569+0000","last_clean":"2024-05-27T07:36:45.121569+0000","last_became_active":"2024-05-27T07:36:15.026012+0000","last_became_peered":"2024-05-27T07:36:15.026012+0000","last_unstale":"2024-05-27T07:36:45.121569+0000","last_undegraded":"2024-05-27T07:36:45.121569+0000","last_fullsized":"2024-05-27T07:36:45.121569+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:27:45.238053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126647+0000","last_change":"2024-05-27T07:36:01.956038+0000","last_active":"2024-05-27T07:36:45.126647+0000","last_peered":"2024-05-27T07:36:45.126647+0000","last_clean":"2024-05-27T07:36:45.126647+0000","last_became_active":"2024-05-27T07:36:01.955639+0000","last_became_peered":"2024-05-27T07:36:01.955639+0000","last_unstale":"2024-05-27T07:36:45.126647+0000","last_undegraded":"2024-05-27T07:36:45.126647+0000","last_fullsized":"2024-05-27T07:36:45.126647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:32.920194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120513+0000","last_change":"2024-05-27T07:36:04.986676+0000","last_active":"2024-05-27T07:36:45.120513+0000","last_peered":"2024-05-27T07:36:45.120513+0000","last_clean":"2024-05-27T07:36:45.120513+0000","last_became_active":"2024-05-27T07:36:04.986316+0000","last_became_peered":"2024-05-27T07:36:04.986316+0000","last_unstale":"2024-05-27T07:36:45.120513+0000","last_undegraded":"2024-05-27T07:36:45.120513+0000","last_fullsized":"2024-05-27T07:36:45.120513+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:03:50.947781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119250+0000","last_change":"2024-05-27T07:36:15.027643+0000","last_active":"2024-05-27T07:36:45.119250+0000","last_peered":"2024-05-27T07:36:45.119250+0000","last_clean":"2024-05-27T07:36:45.119250+0000","last_became_active":"2024-05-27T07:36:15.027175+0000","last_became_peered":"2024-05-27T07:36:15.027175+0000","last_unstale":"2024-05-27T07:36:45.119250+0000","last_undegraded":"2024-05-27T07:36:45.119250+0000","last_fullsized":"2024-05-27T07:36:45.119250+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:46:32.764271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"78'3","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123809+0000","last_change":"2024-05-27T07:36:02.245746+0000","last_active":"2024-05-27T07:36:45.123809+0000","last_peered":"2024-05-27T07:36:45.123809+0000","last_clean":"2024-05-27T07:36:45.123809+0000","last_became_active":"2024-05-27T07:36:02.245269+0000","last_became_peered":"2024-05-27T07:36:02.245269+0000","last_unstale":"2024-05-27T07:36:45.123809+0000","last_undegraded":"2024-05-27T07:36:45.123809+0000","last_fullsized":"2024-05-27T07:36:45.123809+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:28.202540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.129374+0000","last_change":"2024-05-27T07:36:45.129374+0000","last_active":"2024-05-27T07:36:45.129374+0000","last_peered":"2024-05-27T07:36:45.129374+0000","last_clean":"2024-05-27T07:36:45.129374+0000","last_became_active":"2024-05-27T07:36:45.128956+0000","last_became_peered":"2024-05-27T07:36:45.128956+0000","last_unstale":"2024-05-27T07:36:45.129374+0000","last_undegraded":"2024-05-27T07:36:45.129374+0000","last_fullsized":"2024-05-27T07:36:45.129374+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:17.037598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122356+0000","last_change":"2024-05-27T07:36:15.027097+0000","last_active":"2024-05-27T07:36:45.122356+0000","last_peered":"2024-05-27T07:36:45.122356+0000","last_clean":"2024-05-27T07:36:45.122356+0000","last_became_active":"2024-05-27T07:36:15.026704+0000","last_became_peered":"2024-05-27T07:36:15.026704+0000","last_unstale":"2024-05-27T07:36:45.122356+0000","last_undegraded":"2024-05-27T07:36:45.122356+0000","last_fullsized":"2024-05-27T07:36:45.122356+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:35:03.747199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'30","reported_seq":134,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:37:30.503184+0000","last_change":"2024-05-27T07:36:01.956926+0000","last_active":"2024-05-27T07:37:30.503184+0000","last_peered":"2024-05-27T07:37:30.503184+0000","last_clean":"2024-05-27T07:37:30.503184+0000","last_became_active":"2024-05-27T07:36:01.956515+0000","last_became_peered":"2024-05-27T07:36:01.956515+0000","last_unstale":"2024-05-27T07:37:30.503184+0000","last_undegraded":"2024-05-27T07:37:30.503184+0000","last_fullsized":"2024-05-27T07:37:30.503184+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:46.566489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60380,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":64,"num_read_kb":91,"num_write":30,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121635+0000","last_change":"2024-05-27T07:36:04.976498+0000","last_active":"2024-05-27T07:36:45.121635+0000","last_peered":"2024-05-27T07:36:45.121635+0000","last_clean":"2024-05-27T07:36:45.121635+0000","last_became_active":"2024-05-27T07:36:04.976020+0000","last_became_peered":"2024-05-27T07:36:04.976020+0000","last_unstale":"2024-05-27T07:36:45.121635+0000","last_undegraded":"2024-05-27T07:36:45.121635+0000","last_fullsized":"2024-05-27T07:36:45.121635+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:25.909259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.124919+0000","last_change":"2024-05-27T07:36:15.022700+0000","last_active":"2024-05-27T07:36:45.124919+0000","last_peered":"2024-05-27T07:36:45.124919+0000","last_clean":"2024-05-27T07:36:45.124919+0000","last_became_active":"2024-05-27T07:36:15.022109+0000","last_became_peered":"2024-05-27T07:36:15.022109+0000","last_unstale":"2024-05-27T07:36:45.124919+0000","last_undegraded":"2024-05-27T07:36:45.124919+0000","last_fullsized":"2024-05-27T07:36:45.124919+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:02:43.498049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"78'2","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.140024+0000","last_change":"2024-05-27T07:36:01.953442+0000","last_active":"2024-05-27T07:36:45.140024+0000","last_peered":"2024-05-27T07:36:45.140024+0000","last_clean":"2024-05-27T07:36:45.140024+0000","last_became_active":"2024-05-27T07:36:01.953051+0000","last_became_peered":"2024-05-27T07:36:01.953051+0000","last_unstale":"2024-05-27T07:36:45.140024+0000","last_undegraded":"2024-05-27T07:36:45.140024+0000","last_fullsized":"2024-05-27T07:36:45.140024+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:47:53.070596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125820+0000","last_change":"2024-05-27T07:36:04.976365+0000","last_active":"2024-05-27T07:36:45.125820+0000","last_peered":"2024-05-27T07:36:45.125820+0000","last_clean":"2024-05-27T07:36:45.125820+0000","last_became_active":"2024-05-27T07:36:04.976050+0000","last_became_peered":"2024-05-27T07:36:04.976050+0000","last_unstale":"2024-05-27T07:36:45.125820+0000","last_undegraded":"2024-05-27T07:36:45.125820+0000","last_fullsized":"2024-05-27T07:36:45.125820+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:19.765746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123022+0000","last_change":"2024-05-27T07:36:15.022561+0000","last_active":"2024-05-27T07:36:45.123022+0000","last_peered":"2024-05-27T07:36:45.123022+0000","last_clean":"2024-05-27T07:36:45.123022+0000","last_became_active":"2024-05-27T07:36:15.021974+0000","last_became_peered":"2024-05-27T07:36:15.021974+0000","last_unstale":"2024-05-27T07:36:45.123022+0000","last_undegraded":"2024-05-27T07:36:45.123022+0000","last_fullsized":"2024-05-27T07:36:45.123022+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:31.520969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122631+0000","last_change":"2024-05-27T07:36:01.952625+0000","last_active":"2024-05-27T07:36:45.122631+0000","last_peered":"2024-05-27T07:36:45.122631+0000","last_clean":"2024-05-27T07:36:45.122631+0000","last_became_active":"2024-05-27T07:36:01.952280+0000","last_became_peered":"2024-05-27T07:36:01.952280+0000","last_unstale":"2024-05-27T07:36:45.122631+0000","last_undegraded":"2024-05-27T07:36:45.122631+0000","last_fullsized":"2024-05-27T07:36:45.122631+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:17.548061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125589+0000","last_change":"2024-05-27T07:36:04.984795+0000","last_active":"2024-05-27T07:36:45.125589+0000","last_peered":"2024-05-27T07:36:45.125589+0000","last_clean":"2024-05-27T07:36:45.125589+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:36:45.125589+0000","last_undegraded":"2024-05-27T07:36:45.125589+0000","last_fullsized":"2024-05-27T07:36:45.125589+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:07.207460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122161+0000","last_change":"2024-05-27T07:36:15.033565+0000","last_active":"2024-05-27T07:36:45.122161+0000","last_peered":"2024-05-27T07:36:45.122161+0000","last_clean":"2024-05-27T07:36:45.122161+0000","last_became_active":"2024-05-27T07:36:15.033145+0000","last_became_peered":"2024-05-27T07:36:15.033145+0000","last_unstale":"2024-05-27T07:36:45.122161+0000","last_undegraded":"2024-05-27T07:36:45.122161+0000","last_fullsized":"2024-05-27T07:36:45.122161+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:38:02.163816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.118952+0000","last_change":"2024-05-27T07:36:01.952759+0000","last_active":"2024-05-27T07:36:45.118952+0000","last_peered":"2024-05-27T07:36:45.118952+0000","last_clean":"2024-05-27T07:36:45.118952+0000","last_became_active":"2024-05-27T07:36:01.952339+0000","last_became_peered":"2024-05-27T07:36:01.952339+0000","last_unstale":"2024-05-27T07:36:45.118952+0000","last_undegraded":"2024-05-27T07:36:45.118952+0000","last_fullsized":"2024-05-27T07:36:45.118952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:14.518123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125653+0000","last_change":"2024-05-27T07:36:04.986312+0000","last_active":"2024-05-27T07:36:45.125653+0000","last_peered":"2024-05-27T07:36:45.125653+0000","last_clean":"2024-05-27T07:36:45.125653+0000","last_became_active":"2024-05-27T07:36:04.985944+0000","last_became_peered":"2024-05-27T07:36:04.985944+0000","last_unstale":"2024-05-27T07:36:45.125653+0000","last_undegraded":"2024-05-27T07:36:45.125653+0000","last_fullsized":"2024-05-27T07:36:45.125653+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:04.955449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126172+0000","last_change":"2024-05-27T07:36:15.020930+0000","last_active":"2024-05-27T07:36:45.126172+0000","last_peered":"2024-05-27T07:36:45.126172+0000","last_clean":"2024-05-27T07:36:45.126172+0000","last_became_active":"2024-05-27T07:36:15.020556+0000","last_became_peered":"2024-05-27T07:36:15.020556+0000","last_unstale":"2024-05-27T07:36:45.126172+0000","last_undegraded":"2024-05-27T07:36:45.126172+0000","last_fullsized":"2024-05-27T07:36:45.126172+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:02.673065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"78'8","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121970+0000","last_change":"2024-05-27T07:36:02.243402+0000","last_active":"2024-05-27T07:36:45.121970+0000","last_peered":"2024-05-27T07:36:45.121970+0000","last_clean":"2024-05-27T07:36:45.121970+0000","last_became_active":"2024-05-27T07:36:02.242978+0000","last_became_peered":"2024-05-27T07:36:02.242978+0000","last_unstale":"2024-05-27T07:36:45.121970+0000","last_undegraded":"2024-05-27T07:36:45.121970+0000","last_fullsized":"2024-05-27T07:36:45.121970+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:25:04.289506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123057+0000","last_change":"2024-05-27T07:36:04.989885+0000","last_active":"2024-05-27T07:36:45.123057+0000","last_peered":"2024-05-27T07:36:45.123057+0000","last_clean":"2024-05-27T07:36:45.123057+0000","last_became_active":"2024-05-27T07:36:04.989394+0000","last_became_peered":"2024-05-27T07:36:04.989394+0000","last_unstale":"2024-05-27T07:36:45.123057+0000","last_undegraded":"2024-05-27T07:36:45.123057+0000","last_fullsized":"2024-05-27T07:36:45.123057+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:54.138286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122907+0000","last_change":"2024-05-27T07:36:15.021772+0000","last_active":"2024-05-27T07:36:45.122907+0000","last_peered":"2024-05-27T07:36:45.122907+0000","last_clean":"2024-05-27T07:36:45.122907+0000","last_became_active":"2024-05-27T07:36:15.021352+0000","last_became_peered":"2024-05-27T07:36:15.021352+0000","last_unstale":"2024-05-27T07:36:45.122907+0000","last_undegraded":"2024-05-27T07:36:45.122907+0000","last_fullsized":"2024-05-27T07:36:45.122907+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:23.066662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.132394+0000","last_change":"2024-05-27T07:36:45.131900+0000","last_active":"2024-05-27T07:36:45.132394+0000","last_peered":"2024-05-27T07:36:45.132394+0000","last_clean":"2024-05-27T07:36:45.132394+0000","last_became_active":"2024-05-27T07:36:45.127340+0000","last_became_peered":"2024-05-27T07:36:45.127340+0000","last_unstale":"2024-05-27T07:36:45.132394+0000","last_undegraded":"2024-05-27T07:36:45.132394+0000","last_fullsized":"2024-05-27T07:36:45.132394+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:07:57.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123373+0000","last_change":"2024-05-27T07:36:04.987671+0000","last_active":"2024-05-27T07:36:45.123373+0000","last_peered":"2024-05-27T07:36:45.123373+0000","last_clean":"2024-05-27T07:36:45.123373+0000","last_became_active":"2024-05-27T07:36:04.987317+0000","last_became_peered":"2024-05-27T07:36:04.987317+0000","last_unstale":"2024-05-27T07:36:45.123373+0000","last_undegraded":"2024-05-27T07:36:45.123373+0000","last_fullsized":"2024-05-27T07:36:45.123373+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:35.985968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121126+0000","last_change":"2024-05-27T07:36:15.019967+0000","last_active":"2024-05-27T07:36:45.121126+0000","last_peered":"2024-05-27T07:36:45.121126+0000","last_clean":"2024-05-27T07:36:45.121126+0000","last_became_active":"2024-05-27T07:36:15.019509+0000","last_became_peered":"2024-05-27T07:36:15.019509+0000","last_unstale":"2024-05-27T07:36:45.121126+0000","last_undegraded":"2024-05-27T07:36:45.121126+0000","last_fullsized":"2024-05-27T07:36:45.121126+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:01:56.137335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122110+0000","last_change":"2024-05-27T07:36:02.243561+0000","last_active":"2024-05-27T07:36:45.122110+0000","last_peered":"2024-05-27T07:36:45.122110+0000","last_clean":"2024-05-27T07:36:45.122110+0000","last_became_active":"2024-05-27T07:36:02.242911+0000","last_became_peered":"2024-05-27T07:36:02.242911+0000","last_unstale":"2024-05-27T07:36:45.122110+0000","last_undegraded":"2024-05-27T07:36:45.122110+0000","last_fullsized":"2024-05-27T07:36:45.122110+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:01:43.915660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.124610+0000","last_change":"2024-05-27T07:36:04.971108+0000","last_active":"2024-05-27T07:36:45.124610+0000","last_peered":"2024-05-27T07:36:45.124610+0000","last_clean":"2024-05-27T07:36:45.124610+0000","last_became_active":"2024-05-27T07:36:04.970682+0000","last_became_peered":"2024-05-27T07:36:04.970682+0000","last_unstale":"2024-05-27T07:36:45.124610+0000","last_undegraded":"2024-05-27T07:36:45.124610+0000","last_fullsized":"2024-05-27T07:36:45.124610+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:15:34.940449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126090+0000","last_change":"2024-05-27T07:36:15.021292+0000","last_active":"2024-05-27T07:36:45.126090+0000","last_peered":"2024-05-27T07:36:45.126090+0000","last_clean":"2024-05-27T07:36:45.126090+0000","last_became_active":"2024-05-27T07:36:15.019988+0000","last_became_peered":"2024-05-27T07:36:15.019988+0000","last_unstale":"2024-05-27T07:36:45.126090+0000","last_undegraded":"2024-05-27T07:36:45.126090+0000","last_fullsized":"2024-05-27T07:36:45.126090+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:34:14.325368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125715+0000","last_change":"2024-05-27T07:36:01.952231+0000","last_active":"2024-05-27T07:36:45.125715+0000","last_peered":"2024-05-27T07:36:45.125715+0000","last_clean":"2024-05-27T07:36:45.125715+0000","last_became_active":"2024-05-27T07:36:01.951818+0000","last_became_peered":"2024-05-27T07:36:01.951818+0000","last_unstale":"2024-05-27T07:36:45.125715+0000","last_undegraded":"2024-05-27T07:36:45.125715+0000","last_fullsized":"2024-05-27T07:36:45.125715+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:52:16.299204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.124378+0000","last_change":"2024-05-27T07:36:04.982761+0000","last_active":"2024-05-27T07:36:45.124378+0000","last_peered":"2024-05-27T07:36:45.124378+0000","last_clean":"2024-05-27T07:36:45.124378+0000","last_became_active":"2024-05-27T07:36:04.982303+0000","last_became_peered":"2024-05-27T07:36:04.982303+0000","last_unstale":"2024-05-27T07:36:45.124378+0000","last_undegraded":"2024-05-27T07:36:45.124378+0000","last_fullsized":"2024-05-27T07:36:45.124378+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:13:19.099227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126456+0000","last_change":"2024-05-27T07:36:15.021921+0000","last_active":"2024-05-27T07:36:45.126456+0000","last_peered":"2024-05-27T07:36:45.126456+0000","last_clean":"2024-05-27T07:36:45.126456+0000","last_became_active":"2024-05-27T07:36:15.021524+0000","last_became_peered":"2024-05-27T07:36:15.021524+0000","last_unstale":"2024-05-27T07:36:45.126456+0000","last_undegraded":"2024-05-27T07:36:45.126456+0000","last_fullsized":"2024-05-27T07:36:45.126456+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:10:27.303498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.142023+0000","last_change":"2024-05-27T07:36:45.142023+0000","last_active":"2024-05-27T07:36:45.142023+0000","last_peered":"2024-05-27T07:36:45.142023+0000","last_clean":"2024-05-27T07:36:45.142023+0000","last_became_active":"2024-05-27T07:36:45.141278+0000","last_became_peered":"2024-05-27T07:36:45.141278+0000","last_unstale":"2024-05-27T07:36:45.142023+0000","last_undegraded":"2024-05-27T07:36:45.142023+0000","last_fullsized":"2024-05-27T07:36:45.142023+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:15.800380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125472+0000","last_change":"2024-05-27T07:36:02.243041+0000","last_active":"2024-05-27T07:36:45.125472+0000","last_peered":"2024-05-27T07:36:45.125472+0000","last_clean":"2024-05-27T07:36:45.125472+0000","last_became_active":"2024-05-27T07:36:02.242689+0000","last_became_peered":"2024-05-27T07:36:02.242689+0000","last_unstale":"2024-05-27T07:36:45.125472+0000","last_undegraded":"2024-05-27T07:36:45.125472+0000","last_fullsized":"2024-05-27T07:36:45.125472+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:44:19.652954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126874+0000","last_change":"2024-05-27T07:36:15.025595+0000","last_active":"2024-05-27T07:36:45.126874+0000","last_peered":"2024-05-27T07:36:45.126874+0000","last_clean":"2024-05-27T07:36:45.126874+0000","last_became_active":"2024-05-27T07:36:15.025177+0000","last_became_peered":"2024-05-27T07:36:15.025177+0000","last_unstale":"2024-05-27T07:36:45.126874+0000","last_undegraded":"2024-05-27T07:36:45.126874+0000","last_fullsized":"2024-05-27T07:36:45.126874+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:08.920778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139265+0000","last_change":"2024-05-27T07:36:04.981422+0000","last_active":"2024-05-27T07:36:45.139265+0000","last_peered":"2024-05-27T07:36:45.139265+0000","last_clean":"2024-05-27T07:36:45.139265+0000","last_became_active":"2024-05-27T07:36:04.981100+0000","last_became_peered":"2024-05-27T07:36:04.981100+0000","last_unstale":"2024-05-27T07:36:45.139265+0000","last_undegraded":"2024-05-27T07:36:45.139265+0000","last_fullsized":"2024-05-27T07:36:45.139265+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:01.076309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119097+0000","last_change":"2024-05-27T07:36:01.953727+0000","last_active":"2024-05-27T07:36:45.119097+0000","last_peered":"2024-05-27T07:36:45.119097+0000","last_clean":"2024-05-27T07:36:45.119097+0000","last_became_active":"2024-05-27T07:36:01.953323+0000","last_became_peered":"2024-05-27T07:36:01.953323+0000","last_unstale":"2024-05-27T07:36:45.119097+0000","last_undegraded":"2024-05-27T07:36:45.119097+0000","last_fullsized":"2024-05-27T07:36:45.119097+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:17.186854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":115,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125453+0000","last_change":"2024-05-27T07:34:38.289913+0000","last_active":"2024-05-27T07:36:45.125453+0000","last_peered":"2024-05-27T07:36:45.125453+0000","last_clean":"2024-05-27T07:36:45.125453+0000","last_became_active":"2024-05-27T07:34:38.285523+0000","last_became_peered":"2024-05-27T07:34:38.285523+0000","last_unstale":"2024-05-27T07:36:45.125453+0000","last_undegraded":"2024-05-27T07:36:45.125453+0000","last_fullsized":"2024-05-27T07:36:45.125453+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_clean_scrub_stamp":"2024-05-27T07:31:09.455079+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:59.537055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123693+0000","last_change":"2024-05-27T07:36:15.027453+0000","last_active":"2024-05-27T07:36:45.123693+0000","last_peered":"2024-05-27T07:36:45.123693+0000","last_clean":"2024-05-27T07:36:45.123693+0000","last_became_active":"2024-05-27T07:36:15.026872+0000","last_became_peered":"2024-05-27T07:36:15.026872+0000","last_unstale":"2024-05-27T07:36:45.123693+0000","last_undegraded":"2024-05-27T07:36:45.123693+0000","last_fullsized":"2024-05-27T07:36:45.123693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:32.351924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125741+0000","last_change":"2024-05-27T07:36:04.982041+0000","last_active":"2024-05-27T07:36:45.125741+0000","last_peered":"2024-05-27T07:36:45.125741+0000","last_clean":"2024-05-27T07:36:45.125741+0000","last_became_active":"2024-05-27T07:36:04.981719+0000","last_became_peered":"2024-05-27T07:36:04.981719+0000","last_unstale":"2024-05-27T07:36:45.125741+0000","last_undegraded":"2024-05-27T07:36:45.125741+0000","last_fullsized":"2024-05-27T07:36:45.125741+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:40.061892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125526+0000","last_change":"2024-05-27T07:36:01.952238+0000","last_active":"2024-05-27T07:36:45.125526+0000","last_peered":"2024-05-27T07:36:45.125526+0000","last_clean":"2024-05-27T07:36:45.125526+0000","last_became_active":"2024-05-27T07:36:01.951853+0000","last_became_peered":"2024-05-27T07:36:01.951853+0000","last_unstale":"2024-05-27T07:36:45.125526+0000","last_undegraded":"2024-05-27T07:36:45.125526+0000","last_fullsized":"2024-05-27T07:36:45.125526+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:46:41.317531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123012+0000","last_change":"2024-05-27T07:36:15.021848+0000","last_active":"2024-05-27T07:36:45.123012+0000","last_peered":"2024-05-27T07:36:45.123012+0000","last_clean":"2024-05-27T07:36:45.123012+0000","last_became_active":"2024-05-27T07:36:15.021371+0000","last_became_peered":"2024-05-27T07:36:15.021371+0000","last_unstale":"2024-05-27T07:36:45.123012+0000","last_undegraded":"2024-05-27T07:36:45.123012+0000","last_fullsized":"2024-05-27T07:36:45.123012+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:32:21.741229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121786+0000","last_change":"2024-05-27T07:36:02.244283+0000","last_active":"2024-05-27T07:36:45.121786+0000","last_peered":"2024-05-27T07:36:45.121786+0000","last_clean":"2024-05-27T07:36:45.121786+0000","last_became_active":"2024-05-27T07:36:02.243785+0000","last_became_peered":"2024-05-27T07:36:02.243785+0000","last_unstale":"2024-05-27T07:36:45.121786+0000","last_undegraded":"2024-05-27T07:36:45.121786+0000","last_fullsized":"2024-05-27T07:36:45.121786+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:38.596510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125864+0000","last_change":"2024-05-27T07:36:04.979903+0000","last_active":"2024-05-27T07:36:45.125864+0000","last_peered":"2024-05-27T07:36:45.125864+0000","last_clean":"2024-05-27T07:36:45.125864+0000","last_became_active":"2024-05-27T07:36:04.979373+0000","last_became_peered":"2024-05-27T07:36:04.979373+0000","last_unstale":"2024-05-27T07:36:45.125864+0000","last_undegraded":"2024-05-27T07:36:45.125864+0000","last_fullsized":"2024-05-27T07:36:45.125864+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:44:33.763230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139014+0000","last_change":"2024-05-27T07:36:15.023740+0000","last_active":"2024-05-27T07:36:45.139014+0000","last_peered":"2024-05-27T07:36:45.139014+0000","last_clean":"2024-05-27T07:36:45.139014+0000","last_became_active":"2024-05-27T07:36:15.023364+0000","last_became_peered":"2024-05-27T07:36:15.023364+0000","last_unstale":"2024-05-27T07:36:45.139014+0000","last_undegraded":"2024-05-27T07:36:45.139014+0000","last_fullsized":"2024-05-27T07:36:45.139014+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:21.399065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'8","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:37:22.992272+0000","last_change":"2024-05-27T07:36:01.951451+0000","last_active":"2024-05-27T07:37:22.992272+0000","last_peered":"2024-05-27T07:37:22.992272+0000","last_clean":"2024-05-27T07:37:22.992272+0000","last_became_active":"2024-05-27T07:36:01.951037+0000","last_became_peered":"2024-05-27T07:36:01.951037+0000","last_unstale":"2024-05-27T07:37:22.992272+0000","last_undegraded":"2024-05-27T07:37:22.992272+0000","last_fullsized":"2024-05-27T07:37:22.992272+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:44.202474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139579+0000","last_change":"2024-05-27T07:36:04.975392+0000","last_active":"2024-05-27T07:36:45.139579+0000","last_peered":"2024-05-27T07:36:45.139579+0000","last_clean":"2024-05-27T07:36:45.139579+0000","last_became_active":"2024-05-27T07:36:04.975029+0000","last_became_peered":"2024-05-27T07:36:04.975029+0000","last_unstale":"2024-05-27T07:36:45.139579+0000","last_undegraded":"2024-05-27T07:36:45.139579+0000","last_fullsized":"2024-05-27T07:36:45.139579+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:32.022670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139076+0000","last_change":"2024-05-27T07:36:15.026401+0000","last_active":"2024-05-27T07:36:45.139076+0000","last_peered":"2024-05-27T07:36:45.139076+0000","last_clean":"2024-05-27T07:36:45.139076+0000","last_became_active":"2024-05-27T07:36:15.025963+0000","last_became_peered":"2024-05-27T07:36:15.025963+0000","last_unstale":"2024-05-27T07:36:45.139076+0000","last_undegraded":"2024-05-27T07:36:45.139076+0000","last_fullsized":"2024-05-27T07:36:45.139076+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:33:47.616099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"78'2","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119586+0000","last_change":"2024-05-27T07:36:02.228776+0000","last_active":"2024-05-27T07:36:45.119586+0000","last_peered":"2024-05-27T07:36:45.119586+0000","last_clean":"2024-05-27T07:36:45.119586+0000","last_became_active":"2024-05-27T07:36:02.228356+0000","last_became_peered":"2024-05-27T07:36:02.228356+0000","last_unstale":"2024-05-27T07:36:45.119586+0000","last_undegraded":"2024-05-27T07:36:45.119586+0000","last_fullsized":"2024-05-27T07:36:45.119586+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:07.289340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120466+0000","last_change":"2024-05-27T07:36:04.986917+0000","last_active":"2024-05-27T07:36:45.120466+0000","last_peered":"2024-05-27T07:36:45.120466+0000","last_clean":"2024-05-27T07:36:45.120466+0000","last_became_active":"2024-05-27T07:36:04.986432+0000","last_became_peered":"2024-05-27T07:36:04.986432+0000","last_unstale":"2024-05-27T07:36:45.120466+0000","last_undegraded":"2024-05-27T07:36:45.120466+0000","last_fullsized":"2024-05-27T07:36:45.120466+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:39:15.011058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119604+0000","last_change":"2024-05-27T07:36:15.032312+0000","last_active":"2024-05-27T07:36:45.119604+0000","last_peered":"2024-05-27T07:36:45.119604+0000","last_clean":"2024-05-27T07:36:45.119604+0000","last_became_active":"2024-05-27T07:36:15.031475+0000","last_became_peered":"2024-05-27T07:36:15.031475+0000","last_unstale":"2024-05-27T07:36:45.119604+0000","last_undegraded":"2024-05-27T07:36:45.119604+0000","last_fullsized":"2024-05-27T07:36:45.119604+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:40.099421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121220+0000","last_change":"2024-05-27T07:36:02.243456+0000","last_active":"2024-05-27T07:36:45.121220+0000","last_peered":"2024-05-27T07:36:45.121220+0000","last_clean":"2024-05-27T07:36:45.121220+0000","last_became_active":"2024-05-27T07:36:02.243164+0000","last_became_peered":"2024-05-27T07:36:02.243164+0000","last_unstale":"2024-05-27T07:36:45.121220+0000","last_undegraded":"2024-05-27T07:36:45.121220+0000","last_fullsized":"2024-05-27T07:36:45.121220+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:33.765834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120937+0000","last_change":"2024-05-27T07:36:04.980192+0000","last_active":"2024-05-27T07:36:45.120937+0000","last_peered":"2024-05-27T07:36:45.120937+0000","last_clean":"2024-05-27T07:36:45.120937+0000","last_became_active":"2024-05-27T07:36:04.979797+0000","last_became_peered":"2024-05-27T07:36:04.979797+0000","last_unstale":"2024-05-27T07:36:45.120937+0000","last_undegraded":"2024-05-27T07:36:45.120937+0000","last_fullsized":"2024-05-27T07:36:45.120937+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:47:37.006746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.122364+0000","last_change":"2024-05-27T07:36:15.033153+0000","last_active":"2024-05-27T07:36:45.122364+0000","last_peered":"2024-05-27T07:36:45.122364+0000","last_clean":"2024-05-27T07:36:45.122364+0000","last_became_active":"2024-05-27T07:36:15.032783+0000","last_became_peered":"2024-05-27T07:36:15.032783+0000","last_unstale":"2024-05-27T07:36:45.122364+0000","last_undegraded":"2024-05-27T07:36:45.122364+0000","last_fullsized":"2024-05-27T07:36:45.122364+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:31:26.237222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139193+0000","last_change":"2024-05-27T07:36:01.952754+0000","last_active":"2024-05-27T07:36:45.139193+0000","last_peered":"2024-05-27T07:36:45.139193+0000","last_clean":"2024-05-27T07:36:45.139193+0000","last_became_active":"2024-05-27T07:36:01.952397+0000","last_became_peered":"2024-05-27T07:36:01.952397+0000","last_unstale":"2024-05-27T07:36:45.139193+0000","last_undegraded":"2024-05-27T07:36:45.139193+0000","last_fullsized":"2024-05-27T07:36:45.139193+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:04.350169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121631+0000","last_change":"2024-05-27T07:36:04.979193+0000","last_active":"2024-05-27T07:36:45.121631+0000","last_peered":"2024-05-27T07:36:45.121631+0000","last_clean":"2024-05-27T07:36:45.121631+0000","last_became_active":"2024-05-27T07:36:04.978679+0000","last_became_peered":"2024-05-27T07:36:04.978679+0000","last_unstale":"2024-05-27T07:36:45.121631+0000","last_undegraded":"2024-05-27T07:36:45.121631+0000","last_fullsized":"2024-05-27T07:36:45.121631+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:59.310284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119389+0000","last_change":"2024-05-27T07:36:15.030150+0000","last_active":"2024-05-27T07:36:45.119389+0000","last_peered":"2024-05-27T07:36:45.119389+0000","last_clean":"2024-05-27T07:36:45.119389+0000","last_became_active":"2024-05-27T07:36:15.029633+0000","last_became_peered":"2024-05-27T07:36:15.029633+0000","last_unstale":"2024-05-27T07:36:45.119389+0000","last_undegraded":"2024-05-27T07:36:45.119389+0000","last_fullsized":"2024-05-27T07:36:45.119389+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:41:39.356585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.140940+0000","last_change":"2024-05-27T07:36:45.140940+0000","last_active":"2024-05-27T07:36:45.140940+0000","last_peered":"2024-05-27T07:36:45.140940+0000","last_clean":"2024-05-27T07:36:45.140940+0000","last_became_active":"2024-05-27T07:36:45.140511+0000","last_became_peered":"2024-05-27T07:36:45.140511+0000","last_unstale":"2024-05-27T07:36:45.140940+0000","last_undegraded":"2024-05-27T07:36:45.140940+0000","last_fullsized":"2024-05-27T07:36:45.140940+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:43:00.507442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139572+0000","last_change":"2024-05-27T07:36:04.974709+0000","last_active":"2024-05-27T07:36:45.139572+0000","last_peered":"2024-05-27T07:36:45.139572+0000","last_clean":"2024-05-27T07:36:45.139572+0000","last_became_active":"2024-05-27T07:36:04.974147+0000","last_became_peered":"2024-05-27T07:36:04.974147+0000","last_unstale":"2024-05-27T07:36:45.139572+0000","last_undegraded":"2024-05-27T07:36:45.139572+0000","last_fullsized":"2024-05-27T07:36:45.139572+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:05:11.251124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.119431+0000","last_change":"2024-05-27T07:36:15.023187+0000","last_active":"2024-05-27T07:36:45.119431+0000","last_peered":"2024-05-27T07:36:45.119431+0000","last_clean":"2024-05-27T07:36:45.119431+0000","last_became_active":"2024-05-27T07:36:15.022687+0000","last_became_peered":"2024-05-27T07:36:15.022687+0000","last_unstale":"2024-05-27T07:36:45.119431+0000","last_undegraded":"2024-05-27T07:36:45.119431+0000","last_fullsized":"2024-05-27T07:36:45.119431+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:51.713222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"78'2","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121336+0000","last_change":"2024-05-27T07:36:01.954751+0000","last_active":"2024-05-27T07:36:45.121336+0000","last_peered":"2024-05-27T07:36:45.121336+0000","last_clean":"2024-05-27T07:36:45.121336+0000","last_became_active":"2024-05-27T07:36:01.954173+0000","last_became_peered":"2024-05-27T07:36:01.954173+0000","last_unstale":"2024-05-27T07:36:45.121336+0000","last_undegraded":"2024-05-27T07:36:45.121336+0000","last_fullsized":"2024-05-27T07:36:45.121336+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:45:10.504504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125862+0000","last_change":"2024-05-27T07:36:04.980294+0000","last_active":"2024-05-27T07:36:45.125862+0000","last_peered":"2024-05-27T07:36:45.125862+0000","last_clean":"2024-05-27T07:36:45.125862+0000","last_became_active":"2024-05-27T07:36:04.979853+0000","last_became_peered":"2024-05-27T07:36:04.979853+0000","last_unstale":"2024-05-27T07:36:45.125862+0000","last_undegraded":"2024-05-27T07:36:45.125862+0000","last_fullsized":"2024-05-27T07:36:45.125862+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:43.131216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121293+0000","last_change":"2024-05-27T07:36:15.025854+0000","last_active":"2024-05-27T07:36:45.121293+0000","last_peered":"2024-05-27T07:36:45.121293+0000","last_clean":"2024-05-27T07:36:45.121293+0000","last_became_active":"2024-05-27T07:36:15.025412+0000","last_became_peered":"2024-05-27T07:36:15.025412+0000","last_unstale":"2024-05-27T07:36:45.121293+0000","last_undegraded":"2024-05-27T07:36:45.121293+0000","last_fullsized":"2024-05-27T07:36:45.121293+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:49.904811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139721+0000","last_change":"2024-05-27T07:36:02.243764+0000","last_active":"2024-05-27T07:36:45.139721+0000","last_peered":"2024-05-27T07:36:45.139721+0000","last_clean":"2024-05-27T07:36:45.139721+0000","last_became_active":"2024-05-27T07:36:02.242935+0000","last_became_peered":"2024-05-27T07:36:02.242935+0000","last_unstale":"2024-05-27T07:36:45.139721+0000","last_undegraded":"2024-05-27T07:36:45.139721+0000","last_fullsized":"2024-05-27T07:36:45.139721+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:27.535658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.127072+0000","last_change":"2024-05-27T07:36:04.975291+0000","last_active":"2024-05-27T07:36:45.127072+0000","last_peered":"2024-05-27T07:36:45.127072+0000","last_clean":"2024-05-27T07:36:45.127072+0000","last_became_active":"2024-05-27T07:36:04.974868+0000","last_became_peered":"2024-05-27T07:36:04.974868+0000","last_unstale":"2024-05-27T07:36:45.127072+0000","last_undegraded":"2024-05-27T07:36:45.127072+0000","last_fullsized":"2024-05-27T07:36:45.127072+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:13.890494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120044+0000","last_change":"2024-05-27T07:36:15.031812+0000","last_active":"2024-05-27T07:36:45.120044+0000","last_peered":"2024-05-27T07:36:45.120044+0000","last_clean":"2024-05-27T07:36:45.120044+0000","last_became_active":"2024-05-27T07:36:15.031399+0000","last_became_peered":"2024-05-27T07:36:15.031399+0000","last_unstale":"2024-05-27T07:36:45.120044+0000","last_undegraded":"2024-05-27T07:36:45.120044+0000","last_fullsized":"2024-05-27T07:36:45.120044+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.595579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.124122+0000","last_change":"2024-05-27T07:36:01.954581+0000","last_active":"2024-05-27T07:36:45.124122+0000","last_peered":"2024-05-27T07:36:45.124122+0000","last_clean":"2024-05-27T07:36:45.124122+0000","last_became_active":"2024-05-27T07:36:01.953564+0000","last_became_peered":"2024-05-27T07:36:01.953564+0000","last_unstale":"2024-05-27T07:36:45.124122+0000","last_undegraded":"2024-05-27T07:36:45.124122+0000","last_fullsized":"2024-05-27T07:36:45.124122+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:56.866738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.141160+0000","last_change":"2024-05-27T07:36:45.141160+0000","last_active":"2024-05-27T07:36:45.141160+0000","last_peered":"2024-05-27T07:36:45.141160+0000","last_clean":"2024-05-27T07:36:45.141160+0000","last_became_active":"2024-05-27T07:36:45.140840+0000","last_became_peered":"2024-05-27T07:36:45.140840+0000","last_unstale":"2024-05-27T07:36:45.141160+0000","last_undegraded":"2024-05-27T07:36:45.141160+0000","last_fullsized":"2024-05-27T07:36:45.141160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:31.676808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121020+0000","last_change":"2024-05-27T07:36:15.026483+0000","last_active":"2024-05-27T07:36:45.121020+0000","last_peered":"2024-05-27T07:36:45.121020+0000","last_clean":"2024-05-27T07:36:45.121020+0000","last_became_active":"2024-05-27T07:36:15.025955+0000","last_became_peered":"2024-05-27T07:36:15.025955+0000","last_unstale":"2024-05-27T07:36:45.121020+0000","last_undegraded":"2024-05-27T07:36:45.121020+0000","last_fullsized":"2024-05-27T07:36:45.121020+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:25.468146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126306+0000","last_change":"2024-05-27T07:36:01.952505+0000","last_active":"2024-05-27T07:36:45.126306+0000","last_peered":"2024-05-27T07:36:45.126306+0000","last_clean":"2024-05-27T07:36:45.126306+0000","last_became_active":"2024-05-27T07:36:01.952088+0000","last_became_peered":"2024-05-27T07:36:01.952088+0000","last_unstale":"2024-05-27T07:36:45.126306+0000","last_undegraded":"2024-05-27T07:36:45.126306+0000","last_fullsized":"2024-05-27T07:36:45.126306+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:09.235719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126873+0000","last_change":"2024-05-27T07:36:04.974331+0000","last_active":"2024-05-27T07:36:45.126873+0000","last_peered":"2024-05-27T07:36:45.126873+0000","last_clean":"2024-05-27T07:36:45.126873+0000","last_became_active":"2024-05-27T07:36:04.973825+0000","last_became_peered":"2024-05-27T07:36:04.973825+0000","last_unstale":"2024-05-27T07:36:45.126873+0000","last_undegraded":"2024-05-27T07:36:45.126873+0000","last_fullsized":"2024-05-27T07:36:45.126873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:39:42.917309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.141224+0000","last_change":"2024-05-27T07:36:45.141224+0000","last_active":"2024-05-27T07:36:45.141224+0000","last_peered":"2024-05-27T07:36:45.141224+0000","last_clean":"2024-05-27T07:36:45.141224+0000","last_became_active":"2024-05-27T07:36:45.140776+0000","last_became_peered":"2024-05-27T07:36:45.140776+0000","last_unstale":"2024-05-27T07:36:45.141224+0000","last_undegraded":"2024-05-27T07:36:45.141224+0000","last_fullsized":"2024-05-27T07:36:45.141224+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:36:59.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139425+0000","last_change":"2024-05-27T07:36:04.988247+0000","last_active":"2024-05-27T07:36:45.139425+0000","last_peered":"2024-05-27T07:36:45.139425+0000","last_clean":"2024-05-27T07:36:45.139425+0000","last_became_active":"2024-05-27T07:36:04.987825+0000","last_became_peered":"2024-05-27T07:36:04.987825+0000","last_unstale":"2024-05-27T07:36:45.139425+0000","last_undegraded":"2024-05-27T07:36:45.139425+0000","last_fullsized":"2024-05-27T07:36:45.139425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:32.499260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139200+0000","last_change":"2024-05-27T07:36:04.979395+0000","last_active":"2024-05-27T07:36:45.139200+0000","last_peered":"2024-05-27T07:36:45.139200+0000","last_clean":"2024-05-27T07:36:45.139200+0000","last_became_active":"2024-05-27T07:36:04.978983+0000","last_became_peered":"2024-05-27T07:36:04.978983+0000","last_unstale":"2024-05-27T07:36:45.139200+0000","last_undegraded":"2024-05-27T07:36:45.139200+0000","last_fullsized":"2024-05-27T07:36:45.139200+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:53:54.071958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.120393+0000","last_change":"2024-05-27T07:36:04.989442+0000","last_active":"2024-05-27T07:36:45.120393+0000","last_peered":"2024-05-27T07:36:45.120393+0000","last_clean":"2024-05-27T07:36:45.120393+0000","last_became_active":"2024-05-27T07:36:04.989079+0000","last_became_peered":"2024-05-27T07:36:04.989079+0000","last_unstale":"2024-05-27T07:36:45.120393+0000","last_undegraded":"2024-05-27T07:36:45.120393+0000","last_fullsized":"2024-05-27T07:36:45.120393+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:39.411255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.125882+0000","last_change":"2024-05-27T07:36:04.981289+0000","last_active":"2024-05-27T07:36:45.125882+0000","last_peered":"2024-05-27T07:36:45.125882+0000","last_clean":"2024-05-27T07:36:45.125882+0000","last_became_active":"2024-05-27T07:36:04.980924+0000","last_became_peered":"2024-05-27T07:36:04.980924+0000","last_unstale":"2024-05-27T07:36:45.125882+0000","last_undegraded":"2024-05-27T07:36:45.125882+0000","last_fullsized":"2024-05-27T07:36:45.125882+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:15:04.741303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123565+0000","last_change":"2024-05-27T07:36:04.985712+0000","last_active":"2024-05-27T07:36:45.123565+0000","last_peered":"2024-05-27T07:36:45.123565+0000","last_clean":"2024-05-27T07:36:45.123565+0000","last_became_active":"2024-05-27T07:36:04.983786+0000","last_became_peered":"2024-05-27T07:36:04.983786+0000","last_unstale":"2024-05-27T07:36:45.123565+0000","last_undegraded":"2024-05-27T07:36:45.123565+0000","last_fullsized":"2024-05-27T07:36:45.123565+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:34.549557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139972+0000","last_change":"2024-05-27T07:36:04.991176+0000","last_active":"2024-05-27T07:36:45.139972+0000","last_peered":"2024-05-27T07:36:45.139972+0000","last_clean":"2024-05-27T07:36:45.139972+0000","last_became_active":"2024-05-27T07:36:04.990774+0000","last_became_peered":"2024-05-27T07:36:04.990774+0000","last_unstale":"2024-05-27T07:36:45.139972+0000","last_undegraded":"2024-05-27T07:36:45.139972+0000","last_fullsized":"2024-05-27T07:36:45.139972+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:37.307581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139739+0000","last_change":"2024-05-27T07:36:04.982337+0000","last_active":"2024-05-27T07:36:45.139739+0000","last_peered":"2024-05-27T07:36:45.139739+0000","last_clean":"2024-05-27T07:36:45.139739+0000","last_became_active":"2024-05-27T07:36:04.981988+0000","last_became_peered":"2024-05-27T07:36:04.981988+0000","last_unstale":"2024-05-27T07:36:45.139739+0000","last_undegraded":"2024-05-27T07:36:45.139739+0000","last_fullsized":"2024-05-27T07:36:45.139739+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:17:39.763459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121522+0000","last_change":"2024-05-27T07:36:04.982321+0000","last_active":"2024-05-27T07:36:45.121522+0000","last_peered":"2024-05-27T07:36:45.121522+0000","last_clean":"2024-05-27T07:36:45.121522+0000","last_became_active":"2024-05-27T07:36:04.981690+0000","last_became_peered":"2024-05-27T07:36:04.981690+0000","last_unstale":"2024-05-27T07:36:45.121522+0000","last_undegraded":"2024-05-27T07:36:45.121522+0000","last_fullsized":"2024-05-27T07:36:45.121522+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:10.181606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.139197+0000","last_change":"2024-05-27T07:36:04.979411+0000","last_active":"2024-05-27T07:36:45.139197+0000","last_peered":"2024-05-27T07:36:45.139197+0000","last_clean":"2024-05-27T07:36:45.139197+0000","last_became_active":"2024-05-27T07:36:04.979041+0000","last_became_peered":"2024-05-27T07:36:04.979041+0000","last_unstale":"2024-05-27T07:36:45.139197+0000","last_undegraded":"2024-05-27T07:36:45.139197+0000","last_fullsized":"2024-05-27T07:36:45.139197+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:57:29.165997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.121494+0000","last_change":"2024-05-27T07:36:04.988564+0000","last_active":"2024-05-27T07:36:45.121494+0000","last_peered":"2024-05-27T07:36:45.121494+0000","last_clean":"2024-05-27T07:36:45.121494+0000","last_became_active":"2024-05-27T07:36:04.986313+0000","last_became_peered":"2024-05-27T07:36:04.986313+0000","last_unstale":"2024-05-27T07:36:45.121494+0000","last_undegraded":"2024-05-27T07:36:45.121494+0000","last_fullsized":"2024-05-27T07:36:45.121494+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:38.809386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.126456+0000","last_change":"2024-05-27T07:36:04.974704+0000","last_active":"2024-05-27T07:36:45.126456+0000","last_peered":"2024-05-27T07:36:45.126456+0000","last_clean":"2024-05-27T07:36:45.126456+0000","last_became_active":"2024-05-27T07:36:04.974311+0000","last_became_peered":"2024-05-27T07:36:04.974311+0000","last_unstale":"2024-05-27T07:36:45.126456+0000","last_undegraded":"2024-05-27T07:36:45.126456+0000","last_fullsized":"2024-05-27T07:36:45.126456+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:43.021149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-27T07:36:45.123205+0000","last_change":"2024-05-27T07:36:04.983951+0000","last_active":"2024-05-27T07:36:45.123205+0000","last_peered":"2024-05-27T07:36:45.123205+0000","last_clean":"2024-05-27T07:36:45.123205+0000","last_became_active":"2024-05-27T07:36:04.983489+0000","last_became_peered":"2024-05-27T07:36:04.983489+0000","last_unstale":"2024-05-27T07:36:45.123205+0000","last_undegraded":"2024-05-27T07:36:45.123205+0000","last_fullsized":"2024-05-27T07:36:45.123205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:04.229523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280976,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61178,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":70,"num_read_kb":97,"num_write":84,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":183534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710752,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":495672,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906981,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1142496,"data_compressed":62191,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070506,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":500328,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266735,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":496232,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462964,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1090696,"data_compressed":62191,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626491,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":491554,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":551934,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":950272,"data_stored":491644,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":551968,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":491658,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986245,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":495672,"data_compressed":26460,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822719,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":1086200,"data_compressed":62191,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34107,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34107,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34107,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":60414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:37:34.082 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-27T07:37:34.083 INFO:tasks.cephadm:Setting up client nodes... 2024-05-27T07:37:34.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-27T07:37:34.324 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:34.706 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:34 smithi114 ceph-mon[32038]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:37:34.707 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:34 smithi114 ceph-mon[32038]: pgmap v342: 129 pgs: 129 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-05-27T07:37:34.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:34 smithi163 ceph-mon[36451]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:37:34.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:34 smithi163 ceph-mon[36451]: pgmap v342: 129 pgs: 129 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-05-27T07:37:34.724 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:34 smithi186 ceph-mon[36647]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:37:34.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:34 smithi186 ceph-mon[36647]: pgmap v342: 129 pgs: 129 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-05-27T07:37:36.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.010+0000 7faadea3d700 1 -- 172.21.15.114:0/2476371231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faad8104370 msgr2=0x7faad8104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:36.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.010+0000 7faadea3d700 1 --2- 172.21.15.114:0/2476371231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faad8104370 0x7faad8104780 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7faacc009990 tx=0x7faacc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:36.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.010+0000 7faadea3d700 1 -- 172.21.15.114:0/2476371231 shutdown_connections 2024-05-27T07:37:36.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.010+0000 7faadea3d700 1 --2- 172.21.15.114:0/2476371231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad8104d10 0x7faad810b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:36.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.010+0000 7faadea3d700 1 --2- 172.21.15.114:0/2476371231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faad8103240 0x7faad8103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:36.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.010+0000 7faadea3d700 1 --2- 172.21.15.114:0/2476371231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faad8104370 0x7faad8104780 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:36.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.010+0000 7faadea3d700 1 -- 172.21.15.114:0/2476371231 >> 172.21.15.114:0/2476371231 conn(0x7faad80fe880 msgr2=0x7faad8100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:36.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.010+0000 7faadea3d700 1 -- 172.21.15.114:0/2476371231 shutdown_connections 2024-05-27T07:37:36.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.010+0000 7faadea3d700 1 -- 172.21.15.114:0/2476371231 wait complete. 2024-05-27T07:37:36.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.010+0000 7faadea3d700 1 Processor -- start 2024-05-27T07:37:36.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.010+0000 7faadea3d700 1 -- start start 2024-05-27T07:37:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.011+0000 7faadea3d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faad8103240 0x7faad819a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.011+0000 7faadea3d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faad8104370 0x7faad819aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.011+0000 7faadea3d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad8104d10 0x7faad81a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.011+0000 7faadea3d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faad810ed20 con 0x7faad8104370 2024-05-27T07:37:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.011+0000 7faadea3d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faad810eba0 con 0x7faad8103240 2024-05-27T07:37:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.011+0000 7faadea3d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faad810eea0 con 0x7faad8104d10 2024-05-27T07:37:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.011+0000 7faad77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faad8104370 0x7faad819aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.011+0000 7faad77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faad8104370 0x7faad819aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42560/0 (socket says 172.21.15.114:42560) 2024-05-27T07:37:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.011+0000 7faad77fe700 1 -- 172.21.15.114:0/1160041093 learned_addr learned my addr 172.21.15.114:0/1160041093 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.011+0000 7faad7fff700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faad8103240 0x7faad819a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.011+0000 7faadcfda700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad8104d10 0x7faad81a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.011+0000 7faad77fe700 1 -- 172.21.15.114:0/1160041093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad8104d10 msgr2=0x7faad81a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.011+0000 7faad77fe700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad8104d10 0x7faad81a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.012+0000 7faad77fe700 1 -- 172.21.15.114:0/1160041093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faad8103240 msgr2=0x7faad819a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.012+0000 7faad77fe700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faad8103240 0x7faad819a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.012+0000 7faad77fe700 1 -- 172.21.15.114:0/1160041093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad81a24e0 con 0x7faad8104370 2024-05-27T07:37:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.012+0000 7faadcfda700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad8104d10 0x7faad81a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:37:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.012+0000 7faad77fe700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faad8104370 0x7faad819aec0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faacc0038c0 tx=0x7faacc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.012+0000 7faad57fa700 1 -- 172.21.15.114:0/1160041093 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc004830 con 0x7faad8104370 2024-05-27T07:37:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.012+0000 7faad57fa700 1 -- 172.21.15.114:0/1160041093 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faacc005020 con 0x7faad8104370 2024-05-27T07:37:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.012+0000 7faadea3d700 1 -- 172.21.15.114:0/1160041093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad81a2770 con 0x7faad8104370 2024-05-27T07:37:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.012+0000 7faad57fa700 1 -- 172.21.15.114:0/1160041093 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc003ef0 con 0x7faad8104370 2024-05-27T07:37:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.012+0000 7faadea3d700 1 -- 172.21.15.114:0/1160041093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad81a2be0 con 0x7faad8104370 2024-05-27T07:37:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.013+0000 7faadea3d700 1 -- 172.21.15.114:0/1160041093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faad80512b0 con 0x7faad8104370 2024-05-27T07:37:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.015+0000 7faad57fa700 1 -- 172.21.15.114:0/1160041093 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7faacc0054d0 con 0x7faad8104370 2024-05-27T07:37:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.015+0000 7faad57fa700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7faac00b2990 0x7faac00b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.015+0000 7faad7fff700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7faac00b2990 0x7faac00b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.015+0000 7faad57fa700 1 -- 172.21.15.114:0/1160041093 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7faacc0e3ad0 con 0x7faad8104370 2024-05-27T07:37:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.016+0000 7faad7fff700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7faac00b2990 0x7faac00b4ea0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faad8104270 tx=0x7faac8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:36.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.019+0000 7faad57fa700 1 -- 172.21.15.114:0/1160041093 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7faacc013070 con 0x7faad8104370 2024-05-27T07:37:36.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.314+0000 7faadea3d700 1 -- 172.21.15.114:0/1160041093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7faad81a3200 con 0x7faad8104370 2024-05-27T07:37:36.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.316+0000 7faad57fa700 1 -- 172.21.15.114:0/1160041093 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7faad81a3200 con 0x7faad8104370 2024-05-27T07:37:36.318 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-05-27T07:37:36.319 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBAOFRm2V/DEhAAAgluIuzAyFMamaxq3XwmJg== 2024-05-27T07:37:36.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.319+0000 7faadea3d700 1 -- 172.21.15.114:0/1160041093 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7faac00b2990 msgr2=0x7faac00b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:36.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.319+0000 7faadea3d700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7faac00b2990 0x7faac00b4ea0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faad8104270 tx=0x7faac8009190 comp rx=0 tx=0).stop 2024-05-27T07:37:36.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.319+0000 7faadea3d700 1 -- 172.21.15.114:0/1160041093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faad8104370 msgr2=0x7faad819aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:36.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.319+0000 7faadea3d700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faad8104370 0x7faad819aec0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faacc0038c0 tx=0x7faacc0047b0 comp rx=0 tx=0).stop 2024-05-27T07:37:36.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.319+0000 7faadea3d700 1 -- 172.21.15.114:0/1160041093 shutdown_connections 2024-05-27T07:37:36.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.319+0000 7faadea3d700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7faac00b2990 0x7faac00b4ea0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:36.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.319+0000 7faadea3d700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad8104d10 0x7faad81a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:36.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.319+0000 7faadea3d700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faad8103240 0x7faad819a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:36.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.319+0000 7faadea3d700 1 --2- 172.21.15.114:0/1160041093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faad8104370 0x7faad819aec0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:36.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.319+0000 7faadea3d700 1 -- 172.21.15.114:0/1160041093 >> 172.21.15.114:0/1160041093 conn(0x7faad80fe880 msgr2=0x7faad81001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:36.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.319+0000 7faadea3d700 1 -- 172.21.15.114:0/1160041093 shutdown_connections 2024-05-27T07:37:36.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:36.319+0000 7faadea3d700 1 -- 172.21.15.114:0/1160041093 wait complete. 2024-05-27T07:37:36.629 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:36 smithi114 ceph-mon[32038]: pgmap v343: 129 pgs: 129 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-05-27T07:37:36.629 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:36 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1160041093' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T07:37:36.629 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:36 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1160041093' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T07:37:36.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:36 smithi163 ceph-mon[36451]: pgmap v343: 129 pgs: 129 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-05-27T07:37:36.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:36 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1160041093' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T07:37:36.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:36 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1160041093' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T07:37:36.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:36 smithi186 ceph-mon[36647]: pgmap v343: 129 pgs: 129 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-05-27T07:37:36.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:36 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1160041093' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T07:37:36.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:36 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1160041093' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T07:37:36.984 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:37:36.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-27T07:37:36.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-27T07:37:37.022 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-27T07:37:37.023 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-27T07:37:37.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-05-27T07:37:37.302 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:38.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:38 smithi186 ceph-mon[36647]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 3 op/s 2024-05-27T07:37:38.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:38 smithi114 ceph-mon[32038]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 3 op/s 2024-05-27T07:37:38.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:38 smithi163 ceph-mon[36451]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 3 op/s 2024-05-27T07:37:39.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.092+0000 7f83e8b42700 1 -- 172.21.15.114:0/2460126807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e4108ef0 msgr2=0x7f83e4109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:39.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.092+0000 7f83e8b42700 1 --2- 172.21.15.114:0/2460126807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e4108ef0 0x7f83e4109300 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f83d40099c0 tx=0x7f83d4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:39.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.092+0000 7f83e8b42700 1 -- 172.21.15.114:0/2460126807 shutdown_connections 2024-05-27T07:37:39.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.092+0000 7f83e8b42700 1 --2- 172.21.15.114:0/2460126807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83e4109840 0x7f83e41111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:39.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.092+0000 7f83e8b42700 1 --2- 172.21.15.114:0/2460126807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e41045c0 0x7f83e41049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:39.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.092+0000 7f83e8b42700 1 --2- 172.21.15.114:0/2460126807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e4108ef0 0x7f83e4109300 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:39.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.092+0000 7f83e8b42700 1 -- 172.21.15.114:0/2460126807 >> 172.21.15.114:0/2460126807 conn(0x7f83e40ffe90 msgr2=0x7f83e41022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:39.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.092+0000 7f83e8b42700 1 -- 172.21.15.114:0/2460126807 shutdown_connections 2024-05-27T07:37:39.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.092+0000 7f83e8b42700 1 -- 172.21.15.114:0/2460126807 wait complete. 2024-05-27T07:37:39.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e8b42700 1 Processor -- start 2024-05-27T07:37:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e8b42700 1 -- start start 2024-05-27T07:37:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e8b42700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83e41045c0 0x7f83e419aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e8b42700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e4108ef0 0x7f83e419b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e8b42700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e4109840 0x7f83e419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e8b42700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83e41144c0 con 0x7f83e4108ef0 2024-05-27T07:37:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e8b42700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f83e4114340 con 0x7f83e41045c0 2024-05-27T07:37:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e8b42700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83e4114640 con 0x7f83e4109840 2024-05-27T07:37:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e4108ef0 0x7f83e419b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e4108ef0 0x7f83e419b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42574/0 (socket says 172.21.15.114:42574) 2024-05-27T07:37:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e1d9b700 1 -- 172.21.15.114:0/809724515 learned_addr learned my addr 172.21.15.114:0/809724515 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e259c700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83e41045c0 0x7f83e419aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e2d9d700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e4109840 0x7f83e419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e1d9b700 1 -- 172.21.15.114:0/809724515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e4109840 msgr2=0x7f83e419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e1d9b700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e4109840 0x7f83e419f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e1d9b700 1 -- 172.21.15.114:0/809724515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83e41045c0 msgr2=0x7f83e419aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e1d9b700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83e41045c0 0x7f83e419aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e1d9b700 1 -- 172.21.15.114:0/809724515 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e419fdc0 con 0x7f83e4108ef0 2024-05-27T07:37:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.093+0000 7f83e259c700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83e41045c0 0x7f83e419aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:37:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.094+0000 7f83e1d9b700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e4108ef0 0x7f83e419b490 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009f50 tx=0x7f83d40037c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.094+0000 7f83d37fe700 1 -- 172.21.15.114:0/809724515 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4027070 con 0x7f83e4108ef0 2024-05-27T07:37:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.094+0000 7f83d37fe700 1 -- 172.21.15.114:0/809724515 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83d4004140 con 0x7f83e4108ef0 2024-05-27T07:37:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.094+0000 7f83d37fe700 1 -- 172.21.15.114:0/809724515 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d400f460 con 0x7f83e4108ef0 2024-05-27T07:37:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.094+0000 7f83e8b42700 1 -- 172.21.15.114:0/809724515 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e41a0050 con 0x7f83e4108ef0 2024-05-27T07:37:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.094+0000 7f83e8b42700 1 -- 172.21.15.114:0/809724515 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e41a79d0 con 0x7f83e4108ef0 2024-05-27T07:37:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.095+0000 7f83e8b42700 1 -- 172.21.15.114:0/809724515 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83e4105a70 con 0x7f83e4108ef0 2024-05-27T07:37:39.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.096+0000 7f83d37fe700 1 -- 172.21.15.114:0/809724515 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f83d402a070 con 0x7f83e4108ef0 2024-05-27T07:37:39.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.097+0000 7f83d37fe700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f83cc0b2990 0x7f83cc0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:39.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.097+0000 7f83e259c700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f83cc0b2990 0x7f83cc0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:39.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.097+0000 7f83d37fe700 1 -- 172.21.15.114:0/809724515 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f83d40e3610 con 0x7f83e4108ef0 2024-05-27T07:37:39.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.098+0000 7f83e259c700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f83cc0b2990 0x7f83cc0b4ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f83d80099d0 tx=0x7f83d8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:39.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.102+0000 7f83d37fe700 1 -- 172.21.15.114:0/809724515 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f83d4013070 con 0x7f83e4108ef0 2024-05-27T07:37:39.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.361+0000 7f83e8b42700 1 -- 172.21.15.114:0/809724515 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f83e40512b0 con 0x7f83e4108ef0 2024-05-27T07:37:39.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.367+0000 7f83d37fe700 1 -- 172.21.15.114:0/809724515 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+288655 (secure 0 0 0) 0x7f83d40aaf00 con 0x7f83e4108ef0 2024-05-27T07:37:39.366 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:37:39.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.370+0000 7f83e8b42700 1 -- 172.21.15.114:0/809724515 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f83cc0b2990 msgr2=0x7f83cc0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:39.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.370+0000 7f83e8b42700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f83cc0b2990 0x7f83cc0b4ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f83d80099d0 tx=0x7f83d8008040 comp rx=0 tx=0).stop 2024-05-27T07:37:39.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.370+0000 7f83e8b42700 1 -- 172.21.15.114:0/809724515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e4108ef0 msgr2=0x7f83e419b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:39.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.370+0000 7f83e8b42700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e4108ef0 0x7f83e419b490 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009f50 tx=0x7f83d40037c0 comp rx=0 tx=0).stop 2024-05-27T07:37:39.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.370+0000 7f83e8b42700 1 -- 172.21.15.114:0/809724515 shutdown_connections 2024-05-27T07:37:39.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.370+0000 7f83e8b42700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f83cc0b2990 0x7f83cc0b4ea0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:39.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.370+0000 7f83e8b42700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e4109840 0x7f83e419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:39.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.370+0000 7f83e8b42700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83e41045c0 0x7f83e419aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:39.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.370+0000 7f83e8b42700 1 --2- 172.21.15.114:0/809724515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e4108ef0 0x7f83e419b490 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:39.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.370+0000 7f83e8b42700 1 -- 172.21.15.114:0/809724515 >> 172.21.15.114:0/809724515 conn(0x7f83e40ffe90 msgr2=0x7f83e410a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:39.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.370+0000 7f83e8b42700 1 -- 172.21.15.114:0/809724515 shutdown_connections 2024-05-27T07:37:39.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:39.370+0000 7f83e8b42700 1 -- 172.21.15.114:0/809724515 wait complete. 2024-05-27T07:37:40.063 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2291864634},{"type":"v1","addr":"172.21.15.114:6801","nonce":2291864634}]},"active_addr":"172.21.15.114:6801/2291864634","active_change":"2024-05-27T07:28:43.271613+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.114:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":1761274168}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2381281099}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2574023732}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":1628485461}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":100967895}]}]} 2024-05-27T07:37:40.069 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-27T07:37:40.069 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-27T07:37:40.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:37:40.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:40 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/809724515' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T07:37:40.313 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:40.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:40 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/809724515' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T07:37:40.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:40 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/809724515' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T07:37:41.459 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:41 smithi114 ceph-mon[32038]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2024-05-27T07:37:41.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:41 smithi163 ceph-mon[36451]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2024-05-27T07:37:41.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:41 smithi186 ceph-mon[36647]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2024-05-27T07:37:42.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.103+0000 7fb3a47ea700 1 -- 172.21.15.114:0/2465613988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb39c103c90 msgr2=0x7fb39c1040a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:42.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.103+0000 7fb3a47ea700 1 --2- 172.21.15.114:0/2465613988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb39c103c90 0x7fb39c1040a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb3980099c0 tx=0x7fb398009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:42.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.103+0000 7fb3a47ea700 1 -- 172.21.15.114:0/2465613988 shutdown_connections 2024-05-27T07:37:42.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.103+0000 7fb3a47ea700 1 --2- 172.21.15.114:0/2465613988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb39c108750 0x7fb39c110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:42.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.103+0000 7fb3a47ea700 1 --2- 172.21.15.114:0/2465613988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb39c103c90 0x7fb39c1040a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:42.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.103+0000 7fb3a47ea700 1 --2- 172.21.15.114:0/2465613988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb39c107e00 0x7fb39c108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:42.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.103+0000 7fb3a47ea700 1 -- 172.21.15.114:0/2465613988 >> 172.21.15.114:0/2465613988 conn(0x7fb39c0fe6e0 msgr2=0x7fb39c100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:42.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.103+0000 7fb3a47ea700 1 -- 172.21.15.114:0/2465613988 shutdown_connections 2024-05-27T07:37:42.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.103+0000 7fb3a47ea700 1 -- 172.21.15.114:0/2465613988 wait complete. 2024-05-27T07:37:42.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.104+0000 7fb3a47ea700 1 Processor -- start 2024-05-27T07:37:42.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.104+0000 7fb3a47ea700 1 -- start start 2024-05-27T07:37:42.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a47ea700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb39c103c90 0x7fb39c199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:42.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a47ea700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb39c107e00 0x7fb39c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:42.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a47ea700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb39c108750 0x7fb39c19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:42.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a47ea700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c113370 con 0x7fb39c107e00 2024-05-27T07:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a47ea700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1131f0 con 0x7fb39c108750 2024-05-27T07:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a47ea700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1134f0 con 0x7fb39c103c90 2024-05-27T07:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a1d85700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb39c107e00 0x7fb39c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a1d85700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb39c107e00 0x7fb39c19a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42606/0 (socket says 172.21.15.114:42606) 2024-05-27T07:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a1d85700 1 -- 172.21.15.114:0/3172350802 learned_addr learned my addr 172.21.15.114:0/3172350802 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a2d87700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb39c108750 0x7fb39c19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a2586700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb39c103c90 0x7fb39c199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a1d85700 1 -- 172.21.15.114:0/3172350802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb39c103c90 msgr2=0x7fb39c199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a1d85700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb39c103c90 0x7fb39c199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a1d85700 1 -- 172.21.15.114:0/3172350802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb39c108750 msgr2=0x7fb39c19e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a1d85700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb39c108750 0x7fb39c19e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a1d85700 1 -- 172.21.15.114:0/3172350802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39c19ec10 con 0x7fb39c107e00 2024-05-27T07:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.105+0000 7fb3a1d85700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb39c107e00 0x7fb39c19a2e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00d980 tx=0x7fb38c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.106+0000 7fb3937fe700 1 -- 172.21.15.114:0/3172350802 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c017070 con 0x7fb39c107e00 2024-05-27T07:37:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.106+0000 7fb3937fe700 1 -- 172.21.15.114:0/3172350802 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb38c009de0 con 0x7fb39c107e00 2024-05-27T07:37:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.106+0000 7fb3a47ea700 1 -- 172.21.15.114:0/3172350802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39c19ef00 con 0x7fb39c107e00 2024-05-27T07:37:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.106+0000 7fb3a47ea700 1 -- 172.21.15.114:0/3172350802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39c1028f0 con 0x7fb39c107e00 2024-05-27T07:37:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.106+0000 7fb3937fe700 1 -- 172.21.15.114:0/3172350802 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c00f3c0 con 0x7fb39c107e00 2024-05-27T07:37:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.107+0000 7fb3a47ea700 1 -- 172.21.15.114:0/3172350802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb39c104a10 con 0x7fb39c107e00 2024-05-27T07:37:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.108+0000 7fb3937fe700 1 -- 172.21.15.114:0/3172350802 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fb38c009930 con 0x7fb39c107e00 2024-05-27T07:37:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.109+0000 7fb3937fe700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb3880b2910 0x7fb3880b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.109+0000 7fb3a2586700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb3880b2910 0x7fb3880b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.109+0000 7fb3937fe700 1 -- 172.21.15.114:0/3172350802 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7fb38c0e2220 con 0x7fb39c107e00 2024-05-27T07:37:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.109+0000 7fb3a2586700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb3880b2910 0x7fb3880b4e20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb398003800 tx=0x7fb39801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:42.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.114+0000 7fb3937fe700 1 -- 172.21.15.114:0/3172350802 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb38c0a9a80 con 0x7fb39c107e00 2024-05-27T07:37:42.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.322+0000 7fb3a47ea700 1 -- 172.21.15.114:0/3172350802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb39c19b0a0 con 0x7fb39c107e00 2024-05-27T07:37:42.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.324+0000 7fb3937fe700 1 -- 172.21.15.114:0/3172350802 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+24736 (secure 0 0 0) 0x7fb38c0a9380 con 0x7fb39c107e00 2024-05-27T07:37:42.320 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:37:42.321 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T07:36:45.114309+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:37:42.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.327+0000 7fb3a47ea700 1 -- 172.21.15.114:0/3172350802 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb3880b2910 msgr2=0x7fb3880b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:42.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.327+0000 7fb3a47ea700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb3880b2910 0x7fb3880b4e20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb398003800 tx=0x7fb39801a040 comp rx=0 tx=0).stop 2024-05-27T07:37:42.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.327+0000 7fb3a47ea700 1 -- 172.21.15.114:0/3172350802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb39c107e00 msgr2=0x7fb39c19a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:42.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.327+0000 7fb3a47ea700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb39c107e00 0x7fb39c19a2e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00d980 tx=0x7fb38c00dc90 comp rx=0 tx=0).stop 2024-05-27T07:37:42.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.327+0000 7fb3a47ea700 1 -- 172.21.15.114:0/3172350802 shutdown_connections 2024-05-27T07:37:42.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.327+0000 7fb3a47ea700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb39c108750 0x7fb39c19e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:42.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.327+0000 7fb3a47ea700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb3880b2910 0x7fb3880b4e20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:42.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.327+0000 7fb3a47ea700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb39c103c90 0x7fb39c199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:42.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.327+0000 7fb3a47ea700 1 --2- 172.21.15.114:0/3172350802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb39c107e00 0x7fb39c19a2e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:42.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.327+0000 7fb3a47ea700 1 -- 172.21.15.114:0/3172350802 >> 172.21.15.114:0/3172350802 conn(0x7fb39c0fe6e0 msgr2=0x7fb39c1096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:42.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.327+0000 7fb3a47ea700 1 -- 172.21.15.114:0/3172350802 shutdown_connections 2024-05-27T07:37:42.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:42.327+0000 7fb3a47ea700 1 -- 172.21.15.114:0/3172350802 wait complete. 2024-05-27T07:37:43.084 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-27T07:37:43.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:37:43.328 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:43.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:43 smithi114 ceph-mon[32038]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-05-27T07:37:43.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:43 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3172350802' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:37:43.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:43 smithi163 ceph-mon[36451]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-05-27T07:37:43.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:43 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3172350802' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:37:43.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:43 smithi186 ceph-mon[36647]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-05-27T07:37:43.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:43 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3172350802' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:37:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.242+0000 7f9c09de7700 1 -- 172.21.15.114:0/1356779892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c04108750 msgr2=0x7f9c041100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.242+0000 7f9c09de7700 1 --2- 172.21.15.114:0/1356779892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c04108750 0x7f9c041100b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9bf40099c0 tx=0x7f9bf4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.242+0000 7f9c09de7700 1 -- 172.21.15.114:0/1356779892 shutdown_connections 2024-05-27T07:37:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.242+0000 7f9c09de7700 1 --2- 172.21.15.114:0/1356779892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c04108750 0x7f9c041100b0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.242+0000 7f9c09de7700 1 --2- 172.21.15.114:0/1356779892 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c041034d0 0x7f9c041038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.242+0000 7f9c09de7700 1 --2- 172.21.15.114:0/1356779892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c04107e00 0x7f9c04108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.242+0000 7f9c09de7700 1 -- 172.21.15.114:0/1356779892 >> 172.21.15.114:0/1356779892 conn(0x7f9c040fef20 msgr2=0x7f9c04101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.242+0000 7f9c09de7700 1 -- 172.21.15.114:0/1356779892 shutdown_connections 2024-05-27T07:37:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.242+0000 7f9c09de7700 1 -- 172.21.15.114:0/1356779892 wait complete. 2024-05-27T07:37:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.243+0000 7f9c09de7700 1 Processor -- start 2024-05-27T07:37:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.243+0000 7f9c09de7700 1 -- start start 2024-05-27T07:37:45.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c09de7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c041034d0 0x7f9c0419bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c09de7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c04107e00 0x7f9c0419c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c09de7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c04108750 0x7f9c0419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c09de7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9c04113380 con 0x7f9c04108750 2024-05-27T07:37:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c09de7700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9c04113200 con 0x7f9c041034d0 2024-05-27T07:37:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c09de7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9c04113500 con 0x7f9c04107e00 2024-05-27T07:37:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c03fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c04108750 0x7f9c0419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c03fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c04108750 0x7f9c0419c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42442/0 (socket says 172.21.15.114:42442) 2024-05-27T07:37:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c03fff700 1 -- 172.21.15.114:0/1951569691 learned_addr learned my addr 172.21.15.114:0/1951569691 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c02ffd700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c04107e00 0x7f9c0419c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c037fe700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c041034d0 0x7f9c0419bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c03fff700 1 -- 172.21.15.114:0/1951569691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c04107e00 msgr2=0x7f9c0419c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:45.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c03fff700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c04107e00 0x7f9c0419c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:45.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c03fff700 1 -- 172.21.15.114:0/1951569691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c041034d0 msgr2=0x7f9c0419bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:45.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c03fff700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c041034d0 0x7f9c0419bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:45.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c03fff700 1 -- 172.21.15.114:0/1951569691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c041a6fb0 con 0x7f9c04108750 2024-05-27T07:37:45.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c037fe700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c041034d0 0x7f9c0419bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:37:45.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c03fff700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c04108750 0x7f9c0419c700 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4005910 tx=0x7f9bf4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:45.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c00ff9700 1 -- 172.21.15.114:0/1951569691 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4027070 con 0x7f9c04108750 2024-05-27T07:37:45.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c00ff9700 1 -- 172.21.15.114:0/1951569691 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bf4003f20 con 0x7f9c04108750 2024-05-27T07:37:45.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.244+0000 7f9c00ff9700 1 -- 172.21.15.114:0/1951569691 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4022c50 con 0x7f9c04108750 2024-05-27T07:37:45.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.245+0000 7f9c09de7700 1 -- 172.21.15.114:0/1951569691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c041a7240 con 0x7f9c04108750 2024-05-27T07:37:45.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.245+0000 7f9c09de7700 1 -- 172.21.15.114:0/1951569691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c041a7640 con 0x7f9c04108750 2024-05-27T07:37:45.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.246+0000 7f9c09de7700 1 -- 172.21.15.114:0/1951569691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c040512b0 con 0x7f9c04108750 2024-05-27T07:37:45.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.247+0000 7f9c00ff9700 1 -- 172.21.15.114:0/1951569691 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f9bf402a070 con 0x7f9c04108750 2024-05-27T07:37:45.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.248+0000 7f9c00ff9700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9bf00b2910 0x7f9bf00b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:45.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.248+0000 7f9c037fe700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9bf00b2910 0x7f9bf00b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:45.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.248+0000 7f9c00ff9700 1 -- 172.21.15.114:0/1951569691 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9208+0+0 (secure 0 0 0) 0x7f9bf40e3810 con 0x7f9c04108750 2024-05-27T07:37:45.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.248+0000 7f9c037fe700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9bf00b2910 0x7f9bf00b4e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0094f0 tx=0x7f9bfc009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:45.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.252+0000 7f9c00ff9700 1 -- 172.21.15.114:0/1951569691 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9bf4013070 con 0x7f9c04108750 2024-05-27T07:37:45.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.459+0000 7f9c09de7700 1 -- 172.21.15.114:0/1951569691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c0402e7a0 con 0x7f9c04108750 2024-05-27T07:37:45.455 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:45 smithi114 ceph-mon[32038]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 426 B/s wr, 1 op/s 2024-05-27T07:37:45.455 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-05-27T07:37:45.455 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-05-27T07:37:45.455 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-05-27T07:37:45.455 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:45 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-27T07:37:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.460+0000 7f9c00ff9700 1 -- 172.21.15.114:0/1951569691 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24923 (secure 0 0 0) 0x7f9bf40ab090 con 0x7f9c04108750 2024-05-27T07:37:45.457 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:37:45.458 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":88,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T07:37:45.122522+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:37:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.463+0000 7f9c09de7700 1 -- 172.21.15.114:0/1951569691 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9bf00b2910 msgr2=0x7f9bf00b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.463+0000 7f9c09de7700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9bf00b2910 0x7f9bf00b4e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0094f0 tx=0x7f9bfc009040 comp rx=0 tx=0).stop 2024-05-27T07:37:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.463+0000 7f9c09de7700 1 -- 172.21.15.114:0/1951569691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c04108750 msgr2=0x7f9c0419c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.463+0000 7f9c09de7700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c04108750 0x7f9c0419c700 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4005910 tx=0x7f9bf4009580 comp rx=0 tx=0).stop 2024-05-27T07:37:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.463+0000 7f9c09de7700 1 -- 172.21.15.114:0/1951569691 shutdown_connections 2024-05-27T07:37:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.463+0000 7f9c09de7700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c04108750 0x7f9c0419c700 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.463+0000 7f9c09de7700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c041034d0 0x7f9c0419bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.463+0000 7f9c09de7700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9bf00b2910 0x7f9bf00b4e20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.463+0000 7f9c09de7700 1 --2- 172.21.15.114:0/1951569691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c04107e00 0x7f9c0419c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.463+0000 7f9c09de7700 1 -- 172.21.15.114:0/1951569691 >> 172.21.15.114:0/1951569691 conn(0x7f9c040fef20 msgr2=0x7f9c04107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:45.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.463+0000 7f9c09de7700 1 -- 172.21.15.114:0/1951569691 shutdown_connections 2024-05-27T07:37:45.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:45.463+0000 7f9c09de7700 1 -- 172.21.15.114:0/1951569691 wait complete. 2024-05-27T07:37:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:45 smithi163 ceph-mon[36451]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 426 B/s wr, 1 op/s 2024-05-27T07:37:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-05-27T07:37:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-05-27T07:37:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-05-27T07:37:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:45 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-27T07:37:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:45 smithi186 ceph-mon[36647]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 426 B/s wr, 1 op/s 2024-05-27T07:37:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-05-27T07:37:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-05-27T07:37:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-05-27T07:37:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:45 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-27T07:37:46.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-27T07:37:46.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-27T07:37:46.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-27T07:37:46.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-27T07:37:46.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-27T07:37:46.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-27T07:37:46.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-27T07:37:46.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-27T07:37:46.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-27T07:37:46.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-27T07:37:46.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-27T07:37:46.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-27T07:37:46.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-05-27T07:37:46.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-05-27T07:37:46.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-05-27T07:37:46.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:46 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-27T07:37:46.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:46 smithi114 ceph-mon[32038]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T07:37:46.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:46 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1951569691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:37:46.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-05-27T07:37:46.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-05-27T07:37:46.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-05-27T07:37:46.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:46 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-27T07:37:46.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:46 smithi163 ceph-mon[36451]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T07:37:46.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:46 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1951569691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:37:46.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-05-27T07:37:46.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-05-27T07:37:46.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-05-27T07:37:46.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:46 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-27T07:37:46.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:46 smithi186 ceph-mon[36647]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T07:37:46.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:46 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1951569691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:37:46.888 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:46.902 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:46.902 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:46.903 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:46.903 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:46.906 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:47.213 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:47.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:47 smithi114 ceph-mon[32038]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-05-27T07:37:47.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:47 smithi114 ceph-mon[32038]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T07:37:47.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:47 smithi163 ceph-mon[36451]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-05-27T07:37:47.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:47 smithi163 ceph-mon[36451]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T07:37:47.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:47 smithi186 ceph-mon[36647]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-05-27T07:37:47.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:47 smithi186 ceph-mon[36647]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T07:37:48.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:48 smithi114 ceph-mon[32038]: pgmap v351: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:37:48.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:48 smithi114 ceph-mon[32038]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-27T07:37:48.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:48 smithi163 ceph-mon[36451]: pgmap v351: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:37:48.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:48 smithi163 ceph-mon[36451]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-27T07:37:48.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:48 smithi186 ceph-mon[36647]: pgmap v351: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:37:48.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:48 smithi186 ceph-mon[36647]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-27T07:37:50.710 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:50 smithi114 ceph-mon[32038]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-05-27T07:37:50.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:50 smithi163 ceph-mon[36451]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-05-27T07:37:50.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:50 smithi186 ceph-mon[36647]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-05-27T07:37:51.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:51 smithi114 ceph-mon[32038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-27T07:37:51.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:51 smithi114 ceph-mon[32038]: Cluster is now healthy 2024-05-27T07:37:51.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:51 smithi163 ceph-mon[36451]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-27T07:37:51.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:51 smithi163 ceph-mon[36451]: Cluster is now healthy 2024-05-27T07:37:51.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:51 smithi186 ceph-mon[36647]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-27T07:37:51.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:51 smithi186 ceph-mon[36647]: Cluster is now healthy 2024-05-27T07:37:52.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:52 smithi114 ceph-mon[32038]: pgmap v353: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-05-27T07:37:52.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:52 smithi163 ceph-mon[36451]: pgmap v353: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-05-27T07:37:52.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:52 smithi186 ceph-mon[36647]: pgmap v353: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-05-27T07:37:54.511 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:54.511 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:54.511 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:54.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:54 smithi114 ceph-mon[32038]: pgmap v354: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-05-27T07:37:54.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:54 smithi163 ceph-mon[36451]: pgmap v354: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-05-27T07:37:54.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:54 smithi186 ceph-mon[36647]: pgmap v354: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-05-27T07:37:55.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.286+0000 7f881ce06700 1 -- 172.21.15.114:0/264065170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88181087a0 msgr2=0x7f88181100e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:55.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.286+0000 7f881ce06700 1 --2- 172.21.15.114:0/264065170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88181087a0 0x7f88181100e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f881000b0a0 tx=0x7f881000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:37:55.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.286+0000 7f881ce06700 1 -- 172.21.15.114:0/264065170 shutdown_connections 2024-05-27T07:37:55.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.286+0000 7f881ce06700 1 --2- 172.21.15.114:0/264065170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88181087a0 0x7f88181100e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:55.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.286+0000 7f881ce06700 1 --2- 172.21.15.114:0/264065170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8818103ba0 0x7f8818103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:55.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.286+0000 7f881ce06700 1 --2- 172.21.15.114:0/264065170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8818107e50 0x7f8818108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:55.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.286+0000 7f881ce06700 1 -- 172.21.15.114:0/264065170 >> 172.21.15.114:0/264065170 conn(0x7f88180fe510 msgr2=0x7f8818100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:55.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.286+0000 7f881ce06700 1 -- 172.21.15.114:0/264065170 shutdown_connections 2024-05-27T07:37:55.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.286+0000 7f881ce06700 1 -- 172.21.15.114:0/264065170 wait complete. 2024-05-27T07:37:55.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.287+0000 7f881ce06700 1 Processor -- start 2024-05-27T07:37:55.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.287+0000 7f881ce06700 1 -- start start 2024-05-27T07:37:55.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.287+0000 7f881ce06700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8818103ba0 0x7f8818100590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:55.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.287+0000 7f881ce06700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8818107e50 0x7f88180fe9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.287+0000 7f881ce06700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88181087a0 0x7f88180fefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.287+0000 7f881ce06700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f88181133a0 con 0x7f88181087a0 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.287+0000 7f881ce06700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8818113220 con 0x7f8818107e50 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.287+0000 7f881ce06700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8818113520 con 0x7f8818103ba0 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.287+0000 7f8816d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88181087a0 0x7f88180fefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.287+0000 7f8816d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88181087a0 0x7f88180fefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41634/0 (socket says 172.21.15.114:41634) 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.287+0000 7f8816d9d700 1 -- 172.21.15.114:0/745446508 learned_addr learned my addr 172.21.15.114:0/745446508 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.287+0000 7f881659c700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8818103ba0 0x7f8818100590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.287+0000 7f8815d9b700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8818107e50 0x7f88180fe9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.288+0000 7f8816d9d700 1 -- 172.21.15.114:0/745446508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8818103ba0 msgr2=0x7f8818100590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.288+0000 7f8816d9d700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8818103ba0 0x7f8818100590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.288+0000 7f8816d9d700 1 -- 172.21.15.114:0/745446508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8818107e50 msgr2=0x7f88180fe9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.288+0000 7f8816d9d700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8818107e50 0x7f88180fe9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:55.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.288+0000 7f8816d9d700 1 -- 172.21.15.114:0/745446508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88180ff800 con 0x7f88181087a0 2024-05-27T07:37:55.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.288+0000 7f8815d9b700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8818107e50 0x7f88180fe9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:37:55.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.288+0000 7f8816d9d700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88181087a0 0x7f88180fefd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f881000b830 tx=0x7f8810004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:55.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.288+0000 7f88077fe700 1 -- 172.21.15.114:0/745446508 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8810021070 con 0x7f88181087a0 2024-05-27T07:37:55.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.288+0000 7f881ce06700 1 -- 172.21.15.114:0/745446508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88181ad7b0 con 0x7f88181087a0 2024-05-27T07:37:55.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.288+0000 7f881ce06700 1 -- 172.21.15.114:0/745446508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88181adc60 con 0x7f88181087a0 2024-05-27T07:37:55.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.288+0000 7f88077fe700 1 -- 172.21.15.114:0/745446508 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8810003c00 con 0x7f88181087a0 2024-05-27T07:37:55.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.288+0000 7f88077fe700 1 -- 172.21.15.114:0/745446508 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881002a060 con 0x7f88181087a0 2024-05-27T07:37:55.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.289+0000 7f881ce06700 1 -- 172.21.15.114:0/745446508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f87f8001030 con 0x7f88181087a0 2024-05-27T07:37:55.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.291+0000 7f88077fe700 1 -- 172.21.15.114:0/745446508 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f8810003db0 con 0x7f88181087a0 2024-05-27T07:37:55.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.291+0000 7f88077fe700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f88000b2990 0x7f88000b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:55.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.292+0000 7f881659c700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f88000b2990 0x7f88000b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:55.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.292+0000 7f88077fe700 1 -- 172.21.15.114:0/745446508 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f88100e3c90 con 0x7f88181087a0 2024-05-27T07:37:55.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.292+0000 7f88077fe700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938] conn(0x7f88000bdd40 0x7f88000c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:55.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.292+0000 7f88077fe700 1 -- 172.21.15.114:0/745446508 --> [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88100075c0 con 0x7f88000bdd40 2024-05-27T07:37:55.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.292+0000 7f88077fe700 1 -- 172.21.15.114:0/745446508 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f88100e40e0 con 0x7f88181087a0 2024-05-27T07:37:55.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.292+0000 7f881659c700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f88000b2990 0x7f88000b4ea0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8808005950 tx=0x7f88080058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:55.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.292+0000 7f8815d9b700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938] conn(0x7f88000bdd40 0x7f88000c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:55.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.293+0000 7f8815d9b700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938] conn(0x7f88000bdd40 0x7f88000c0230 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:55.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.294+0000 7f88077fe700 1 -- 172.21.15.114:0/745446508 <== osd.10 v2:172.21.15.163:6824/1908338938 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f88100075c0 con 0x7f88000bdd40 2024-05-27T07:37:55.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.312+0000 7f881ce06700 1 -- 172.21.15.114:0/745446508 --> [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f87f8002e60 con 0x7f88000bdd40 2024-05-27T07:37:55.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.312+0000 7f88077fe700 1 -- 172.21.15.114:0/745446508 <== osd.10 v2:172.21.15.163:6824/1908338938 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f87f8002e60 con 0x7f88000bdd40 2024-05-27T07:37:55.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.312+0000 7f881ce06700 1 -- 172.21.15.114:0/745446508 >> [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938] conn(0x7f88000bdd40 msgr2=0x7f88000c0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:55.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.312+0000 7f881ce06700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938] conn(0x7f88000bdd40 0x7f88000c0230 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:55.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.313+0000 7f881ce06700 1 -- 172.21.15.114:0/745446508 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f88000b2990 msgr2=0x7f88000b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:55.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.313+0000 7f881ce06700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f88000b2990 0x7f88000b4ea0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8808005950 tx=0x7f88080058e0 comp rx=0 tx=0).stop 2024-05-27T07:37:55.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.313+0000 7f881ce06700 1 -- 172.21.15.114:0/745446508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88181087a0 msgr2=0x7f88180fefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:55.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.313+0000 7f881ce06700 1 --2- 172.21.15.114:0/745446508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88181087a0 0x7f88180fefd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f881000b830 tx=0x7f8810004d90 comp rx=0 tx=0).stop 2024-05-27T07:37:55.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.313+0000 7f8816d9d700 1 -- 172.21.15.114:0/745446508 reap_dead start 2024-05-27T07:37:55.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.313+0000 7f881ce06700 1 -- 172.21.15.114:0/745446508 shutdown_connections 2024-05-27T07:37:55.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.313+0000 7f881ce06700 1 -- 172.21.15.114:0/745446508 >> 172.21.15.114:0/745446508 conn(0x7f88180fe510 msgr2=0x7f8818102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:55.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.313+0000 7f881ce06700 1 -- 172.21.15.114:0/745446508 shutdown_connections 2024-05-27T07:37:55.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:55.313+0000 7f881ce06700 1 -- 172.21.15.114:0/745446508 wait complete. 2024-05-27T07:37:56.118 INFO:teuthology.orchestra.run.smithi114.stdout:274877906986 2024-05-27T07:37:56.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-27T07:37:56.774 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:56 smithi163 ceph-mon[36451]: pgmap v355: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s; 11 B/s, 0 objects/s recovering 2024-05-27T07:37:56.776 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:56 smithi186 ceph-mon[36647]: pgmap v355: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s; 11 B/s, 0 objects/s recovering 2024-05-27T07:37:56.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:56 smithi114 ceph-mon[32038]: pgmap v355: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s; 11 B/s, 0 objects/s recovering 2024-05-27T07:37:58.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.992+0000 7f9b32c38700 1 -- 172.21.15.114:0/696703545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b2c103540 msgr2=0x7f9b2c103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.992+0000 7f9b32c38700 1 --2- 172.21.15.114:0/696703545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b2c103540 0x7f9b2c103950 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9b240099c0 tx=0x7f9b24009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.992+0000 7f9b32c38700 1 -- 172.21.15.114:0/696703545 shutdown_connections 2024-05-27T07:37:58.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.992+0000 7f9b32c38700 1 --2- 172.21.15.114:0/696703545 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b2c108770 0x7f9b2c1100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.992+0000 7f9b32c38700 1 --2- 172.21.15.114:0/696703545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b2c103540 0x7f9b2c103950 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.992+0000 7f9b32c38700 1 --2- 172.21.15.114:0/696703545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b2c107e20 0x7f9b2c108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.992+0000 7f9b32c38700 1 -- 172.21.15.114:0/696703545 >> 172.21.15.114:0/696703545 conn(0x7f9b2c078df0 msgr2=0x7f9b2c079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:58.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.992+0000 7f9b32c38700 1 -- 172.21.15.114:0/696703545 shutdown_connections 2024-05-27T07:37:58.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.992+0000 7f9b32c38700 1 -- 172.21.15.114:0/696703545 wait complete. 2024-05-27T07:37:58.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.993+0000 7f9b32c38700 1 Processor -- start 2024-05-27T07:37:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.993+0000 7f9b32c38700 1 -- start start 2024-05-27T07:37:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.993+0000 7f9b32c38700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b2c103540 0x7f9b2c19bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.993+0000 7f9b32c38700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b2c107e20 0x7f9b2c19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.993+0000 7f9b32c38700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b2c108770 0x7f9b2c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.993+0000 7f9b32c38700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c113440 con 0x7f9b2c108770 2024-05-27T07:37:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.993+0000 7f9b32c38700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1132c0 con 0x7f9b2c103540 2024-05-27T07:37:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.993+0000 7f9b32c38700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1135c0 con 0x7f9b2c107e20 2024-05-27T07:37:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.993+0000 7f9b311d5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b2c108770 0x7f9b2c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.993+0000 7f9b311d5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b2c108770 0x7f9b2c19c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41644/0 (socket says 172.21.15.114:41644) 2024-05-27T07:37:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.993+0000 7f9b311d5700 1 -- 172.21.15.114:0/1656306134 learned_addr learned my addr 172.21.15.114:0/1656306134 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:37:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.994+0000 7f9b309d4700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b2c103540 0x7f9b2c19bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.994+0000 7f9b2bfff700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b2c107e20 0x7f9b2c19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.994+0000 7f9b311d5700 1 -- 172.21.15.114:0/1656306134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b2c107e20 msgr2=0x7f9b2c19c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.994+0000 7f9b311d5700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b2c107e20 0x7f9b2c19c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.994+0000 7f9b311d5700 1 -- 172.21.15.114:0/1656306134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b2c103540 msgr2=0x7f9b2c19bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.994+0000 7f9b311d5700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b2c103540 0x7f9b2c19bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.994+0000 7f9b311d5700 1 -- 172.21.15.114:0/1656306134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b2c1a7060 con 0x7f9b2c108770 2024-05-27T07:37:58.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.994+0000 7f9b311d5700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b2c108770 0x7f9b2c19c7c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9b1400b550 tx=0x7f9b1400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:58.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.994+0000 7f9b29ffb700 1 -- 172.21.15.114:0/1656306134 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14015020 con 0x7f9b2c108770 2024-05-27T07:37:58.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.994+0000 7f9b29ffb700 1 -- 172.21.15.114:0/1656306134 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b14011ce0 con 0x7f9b2c108770 2024-05-27T07:37:58.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.994+0000 7f9b32c38700 1 -- 172.21.15.114:0/1656306134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b2c1a7350 con 0x7f9b2c108770 2024-05-27T07:37:58.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.994+0000 7f9b29ffb700 1 -- 172.21.15.114:0/1656306134 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14010d40 con 0x7f9b2c108770 2024-05-27T07:37:58.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.994+0000 7f9b32c38700 1 -- 172.21.15.114:0/1656306134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b2c1a7840 con 0x7f9b2c108770 2024-05-27T07:37:58.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.995+0000 7f9b32c38700 1 -- 172.21.15.114:0/1656306134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9b2c063ec0 con 0x7f9b2c108770 2024-05-27T07:37:58.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.997+0000 7f9b29ffb700 1 -- 172.21.15.114:0/1656306134 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f9b14018020 con 0x7f9b2c108770 2024-05-27T07:37:58.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.997+0000 7f9b29ffb700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9b180b2910 0x7f9b180b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.998+0000 7f9b309d4700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9b180b2910 0x7f9b180b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.998+0000 7f9b29ffb700 1 -- 172.21.15.114:0/1656306134 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f9b140e2560 con 0x7f9b2c108770 2024-05-27T07:37:58.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.998+0000 7f9b29ffb700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927] conn(0x7f9b180bdcc0 0x7f9b180c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.998+0000 7f9b29ffb700 1 -- 172.21.15.114:0/1656306134 --> [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b180c0880 con 0x7f9b180bdcc0 2024-05-27T07:37:58.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.998+0000 7f9b29ffb700 1 -- 172.21.15.114:0/1656306134 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f9b140e29b0 con 0x7f9b2c108770 2024-05-27T07:37:58.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.998+0000 7f9b309d4700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9b180b2910 0x7f9b180b4e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9b24009fd0 tx=0x7f9b2401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:58.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.998+0000 7f9b2bfff700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927] conn(0x7f9b180bdcc0 0x7f9b180c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.999+0000 7f9b2bfff700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927] conn(0x7f9b180bdcc0 0x7f9b180c01b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:58.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:57.999+0000 7f9b29ffb700 1 -- 172.21.15.114:0/1656306134 <== osd.4 v2:172.21.15.163:6808/4093855927 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b180c0880 con 0x7f9b180bdcc0 2024-05-27T07:37:58.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:58.016+0000 7f9b32c38700 1 -- 172.21.15.114:0/1656306134 --> [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9b2c064340 con 0x7f9b180bdcc0 2024-05-27T07:37:58.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:58.016+0000 7f9b29ffb700 1 -- 172.21.15.114:0/1656306134 <== osd.4 v2:172.21.15.163:6808/4093855927 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9b2c064340 con 0x7f9b180bdcc0 2024-05-27T07:37:58.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:58.017+0000 7f9b32c38700 1 -- 172.21.15.114:0/1656306134 >> [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927] conn(0x7f9b180bdcc0 msgr2=0x7f9b180c01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:58.017+0000 7f9b32c38700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927] conn(0x7f9b180bdcc0 0x7f9b180c01b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:58.017+0000 7f9b32c38700 1 -- 172.21.15.114:0/1656306134 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9b180b2910 msgr2=0x7f9b180b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:58.017+0000 7f9b32c38700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9b180b2910 0x7f9b180b4e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9b24009fd0 tx=0x7f9b2401a040 comp rx=0 tx=0).stop 2024-05-27T07:37:58.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:58.017+0000 7f9b32c38700 1 -- 172.21.15.114:0/1656306134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b2c108770 msgr2=0x7f9b2c19c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:58.017+0000 7f9b32c38700 1 --2- 172.21.15.114:0/1656306134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b2c108770 0x7f9b2c19c7c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9b1400b550 tx=0x7f9b1400b910 comp rx=0 tx=0).stop 2024-05-27T07:37:58.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:58.017+0000 7f9b311d5700 1 -- 172.21.15.114:0/1656306134 reap_dead start 2024-05-27T07:37:58.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:58.017+0000 7f9b32c38700 1 -- 172.21.15.114:0/1656306134 shutdown_connections 2024-05-27T07:37:58.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:58.017+0000 7f9b32c38700 1 -- 172.21.15.114:0/1656306134 >> 172.21.15.114:0/1656306134 conn(0x7f9b2c078df0 msgr2=0x7f9b2c100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:58.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:58.017+0000 7f9b32c38700 1 -- 172.21.15.114:0/1656306134 shutdown_connections 2024-05-27T07:37:58.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:37:58.017+0000 7f9b32c38700 1 -- 172.21.15.114:0/1656306134 wait complete. 2024-05-27T07:37:58.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:37:58 smithi114 ceph-mon[32038]: pgmap v356: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s; 10 B/s, 0 objects/s recovering 2024-05-27T07:37:58.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:37:58 smithi163 ceph-mon[36451]: pgmap v356: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s; 10 B/s, 0 objects/s recovering 2024-05-27T07:37:58.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:37:58 smithi186 ceph-mon[36647]: pgmap v356: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s; 10 B/s, 0 objects/s recovering 2024-05-27T07:37:59.257 INFO:teuthology.orchestra.run.smithi114.stdout:133143986251 2024-05-27T07:37:59.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-27T07:38:00.563 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:00 smithi114 ceph-mon[32038]: pgmap v357: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-05-27T07:38:00.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.747+0000 7fbdd956c700 1 -- 172.21.15.114:0/1487411298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdd41034b0 msgr2=0x7fbdd41038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:00.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.747+0000 7fbdd956c700 1 --2- 172.21.15.114:0/1487411298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdd41034b0 0x7fbdd41038c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40099c0 tx=0x7fbdc4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:00.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.747+0000 7fbdd956c700 1 -- 172.21.15.114:0/1487411298 shutdown_connections 2024-05-27T07:38:00.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.747+0000 7fbdd956c700 1 --2- 172.21.15.114:0/1487411298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdd4108730 0x7fbdd4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:00.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.747+0000 7fbdd956c700 1 --2- 172.21.15.114:0/1487411298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdd41034b0 0x7fbdd41038c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:00.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.747+0000 7fbdd956c700 1 --2- 172.21.15.114:0/1487411298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbdd4107de0 0x7fbdd41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:00.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.747+0000 7fbdd956c700 1 -- 172.21.15.114:0/1487411298 >> 172.21.15.114:0/1487411298 conn(0x7fbdd40fedb0 msgr2=0x7fbdd41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:00.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.748+0000 7fbdd956c700 1 -- 172.21.15.114:0/1487411298 shutdown_connections 2024-05-27T07:38:00.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.748+0000 7fbdd956c700 1 -- 172.21.15.114:0/1487411298 wait complete. 2024-05-27T07:38:00.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.748+0000 7fbdd956c700 1 Processor -- start 2024-05-27T07:38:00.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.748+0000 7fbdd956c700 1 -- start start 2024-05-27T07:38:00.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd956c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdd41034b0 0x7fbdd419bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:00.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd956c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdd4107de0 0x7fbdd419c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:00.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd956c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbdd4108730 0x7fbdd419c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:00.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd956c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd41133e0 con 0x7fbdd4107de0 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd956c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd4113260 con 0x7fbdd4108730 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd956c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd4113560 con 0x7fbdd41034b0 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd27fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdd4107de0 0x7fbdd419c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd27fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdd4107de0 0x7fbdd419c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41668/0 (socket says 172.21.15.114:41668) 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd27fc700 1 -- 172.21.15.114:0/3281764433 learned_addr learned my addr 172.21.15.114:0/3281764433 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd37fe700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbdd4108730 0x7fbdd419c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd2ffd700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdd41034b0 0x7fbdd419bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd27fc700 1 -- 172.21.15.114:0/3281764433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdd41034b0 msgr2=0x7fbdd419bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd27fc700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdd41034b0 0x7fbdd419bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd27fc700 1 -- 172.21.15.114:0/3281764433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbdd4108730 msgr2=0x7fbdd419c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd27fc700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbdd4108730 0x7fbdd419c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd27fc700 1 -- 172.21.15.114:0/3281764433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd41a7010 con 0x7fbdd4107de0 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdd27fc700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdd4107de0 0x7fbdd419c200 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800d980 tx=0x7fbdc800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.749+0000 7fbdbbfff700 1 -- 172.21.15.114:0/3281764433 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8017070 con 0x7fbdd4107de0 2024-05-27T07:38:00.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.750+0000 7fbdd956c700 1 -- 172.21.15.114:0/3281764433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdd41a7300 con 0x7fbdd4107de0 2024-05-27T07:38:00.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.750+0000 7fbdbbfff700 1 -- 172.21.15.114:0/3281764433 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdc8009de0 con 0x7fbdd4107de0 2024-05-27T07:38:00.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.750+0000 7fbdbbfff700 1 -- 172.21.15.114:0/3281764433 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc800f3c0 con 0x7fbdd4107de0 2024-05-27T07:38:00.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.750+0000 7fbdd956c700 1 -- 172.21.15.114:0/3281764433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdd41a7890 con 0x7fbdd4107de0 2024-05-27T07:38:00.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.750+0000 7fbdd956c700 1 -- 172.21.15.114:0/3281764433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbdd4063ec0 con 0x7fbdd4107de0 2024-05-27T07:38:00.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.752+0000 7fbdbbfff700 1 -- 172.21.15.114:0/3281764433 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fbdc8009930 con 0x7fbdd4107de0 2024-05-27T07:38:00.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.753+0000 7fbdbbfff700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fbdbc0b2910 0x7fbdbc0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:00.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.753+0000 7fbdd2ffd700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fbdbc0b2910 0x7fbdbc0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:00.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.753+0000 7fbdbbfff700 1 -- 172.21.15.114:0/3281764433 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fbdc80e28a0 con 0x7fbdd4107de0 2024-05-27T07:38:00.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.753+0000 7fbdbbfff700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464] conn(0x7fbdbc0bdcc0 0x7fbdbc0c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:00.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.753+0000 7fbdbbfff700 1 -- 172.21.15.114:0/3281764433 --> [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbdc8023e50 con 0x7fbdbc0bdcc0 2024-05-27T07:38:00.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.753+0000 7fbdbbfff700 1 -- 172.21.15.114:0/3281764433 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fbdc80e2cf0 con 0x7fbdd4107de0 2024-05-27T07:38:00.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.753+0000 7fbdd37fe700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464] conn(0x7fbdbc0bdcc0 0x7fbdbc0c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:00.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.753+0000 7fbdd2ffd700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fbdbc0b2910 0x7fbdbc0b4e20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4003800 tx=0x7fbdc401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:00.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.754+0000 7fbdd37fe700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464] conn(0x7fbdbc0bdcc0 0x7fbdbc0c01b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:00.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.754+0000 7fbdbbfff700 1 -- 172.21.15.114:0/3281764433 <== osd.0 v2:172.21.15.114:6802/2663646464 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbdc8023e50 con 0x7fbdbc0bdcc0 2024-05-27T07:38:00.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.771+0000 7fbdd956c700 1 -- 172.21.15.114:0/3281764433 --> [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbdd40642f0 con 0x7fbdbc0bdcc0 2024-05-27T07:38:00.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.772+0000 7fbdbbfff700 1 -- 172.21.15.114:0/3281764433 <== osd.0 v2:172.21.15.114:6802/2663646464 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbdd40642f0 con 0x7fbdbc0bdcc0 2024-05-27T07:38:00.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.772+0000 7fbdd956c700 1 -- 172.21.15.114:0/3281764433 >> [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464] conn(0x7fbdbc0bdcc0 msgr2=0x7fbdbc0c01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:00.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.772+0000 7fbdd956c700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464] conn(0x7fbdbc0bdcc0 0x7fbdbc0c01b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:00.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.772+0000 7fbdd956c700 1 -- 172.21.15.114:0/3281764433 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fbdbc0b2910 msgr2=0x7fbdbc0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:00.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.772+0000 7fbdd956c700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fbdbc0b2910 0x7fbdbc0b4e20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4003800 tx=0x7fbdc401a040 comp rx=0 tx=0).stop 2024-05-27T07:38:00.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.772+0000 7fbdd956c700 1 -- 172.21.15.114:0/3281764433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdd4107de0 msgr2=0x7fbdd419c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:00.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.772+0000 7fbdd956c700 1 --2- 172.21.15.114:0/3281764433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdd4107de0 0x7fbdd419c200 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800d980 tx=0x7fbdc800dc90 comp rx=0 tx=0).stop 2024-05-27T07:38:00.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.772+0000 7fbdd37fe700 1 -- 172.21.15.114:0/3281764433 reap_dead start 2024-05-27T07:38:00.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.772+0000 7fbdd956c700 1 -- 172.21.15.114:0/3281764433 shutdown_connections 2024-05-27T07:38:00.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.772+0000 7fbdd956c700 1 -- 172.21.15.114:0/3281764433 >> 172.21.15.114:0/3281764433 conn(0x7fbdd40fedb0 msgr2=0x7fbdd41075c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:00.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.773+0000 7fbdd956c700 1 -- 172.21.15.114:0/3281764433 shutdown_connections 2024-05-27T07:38:00.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:00.773+0000 7fbdd956c700 1 -- 172.21.15.114:0/3281764433 wait complete. 2024-05-27T07:38:00.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:00 smithi163 ceph-mon[36451]: pgmap v357: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-05-27T07:38:00.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:00 smithi186 ceph-mon[36647]: pgmap v357: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-05-27T07:38:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.109+0000 7fe3eeb98700 1 -- 172.21.15.114:0/2083718074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3e8108f00 msgr2=0x7fe3e8109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.109+0000 7fe3eeb98700 1 --2- 172.21.15.114:0/2083718074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3e8108f00 0x7fe3e8109310 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe3d40099c0 tx=0x7fe3d4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.109+0000 7fe3eeb98700 1 -- 172.21.15.114:0/2083718074 shutdown_connections 2024-05-27T07:38:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.109+0000 7fe3eeb98700 1 --2- 172.21.15.114:0/2083718074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3e8109850 0x7fe3e81111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.109+0000 7fe3eeb98700 1 --2- 172.21.15.114:0/2083718074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe3e81045d0 0x7fe3e81049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.109+0000 7fe3eeb98700 1 --2- 172.21.15.114:0/2083718074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3e8108f00 0x7fe3e8109310 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.109+0000 7fe3eeb98700 1 -- 172.21.15.114:0/2083718074 >> 172.21.15.114:0/2083718074 conn(0x7fe3e80ffec0 msgr2=0x7fe3e8102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.109+0000 7fe3eeb98700 1 -- 172.21.15.114:0/2083718074 shutdown_connections 2024-05-27T07:38:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.109+0000 7fe3eeb98700 1 -- 172.21.15.114:0/2083718074 wait complete. 2024-05-27T07:38:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3eeb98700 1 Processor -- start 2024-05-27T07:38:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3eeb98700 1 -- start start 2024-05-27T07:38:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3eeb98700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3e81045d0 0x7fe3e819aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3eeb98700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3e8108f00 0x7fe3e819b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3eeb98700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe3e8109850 0x7fe3e819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3eeb98700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e8114550 con 0x7fe3e81045d0 2024-05-27T07:38:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3eeb98700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e81143d0 con 0x7fe3e8109850 2024-05-27T07:38:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3eeb98700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e81146d0 con 0x7fe3e8108f00 2024-05-27T07:38:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3ec934700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3e81045d0 0x7fe3e819aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3ec934700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3e81045d0 0x7fe3e819aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41684/0 (socket says 172.21.15.114:41684) 2024-05-27T07:38:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3ec934700 1 -- 172.21.15.114:0/337527403 learned_addr learned my addr 172.21.15.114:0/337527403 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3e7fff700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3e8108f00 0x7fe3e819b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3ec934700 1 -- 172.21.15.114:0/337527403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3e8108f00 msgr2=0x7fe3e819b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3ed135700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe3e8109850 0x7fe3e819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3ec934700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3e8108f00 0x7fe3e819b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3ec934700 1 -- 172.21.15.114:0/337527403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe3e8109850 msgr2=0x7fe3e819f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3ec934700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe3e8109850 0x7fe3e819f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.110+0000 7fe3ec934700 1 -- 172.21.15.114:0/337527403 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e819fdb0 con 0x7fe3e81045d0 2024-05-27T07:38:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.111+0000 7fe3ec934700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3e81045d0 0x7fe3e819aee0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00b530 tx=0x7fe3dc00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.111+0000 7fe3e5ffb700 1 -- 172.21.15.114:0/337527403 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc015020 con 0x7fe3e81045d0 2024-05-27T07:38:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.111+0000 7fe3e5ffb700 1 -- 172.21.15.114:0/337527403 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3dc011ce0 con 0x7fe3e81045d0 2024-05-27T07:38:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.111+0000 7fe3eeb98700 1 -- 172.21.15.114:0/337527403 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e81a00a0 con 0x7fe3e81045d0 2024-05-27T07:38:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.111+0000 7fe3e5ffb700 1 -- 172.21.15.114:0/337527403 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc010d20 con 0x7fe3e81045d0 2024-05-27T07:38:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.111+0000 7fe3eeb98700 1 -- 172.21.15.114:0/337527403 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e81a79c0 con 0x7fe3e81045d0 2024-05-27T07:38:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.112+0000 7fe3eeb98700 1 -- 172.21.15.114:0/337527403 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe3e8063f40 con 0x7fe3e81045d0 2024-05-27T07:38:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.113+0000 7fe3e5ffb700 1 -- 172.21.15.114:0/337527403 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fe3dc018020 con 0x7fe3e81045d0 2024-05-27T07:38:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.114+0000 7fe3e5ffb700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe3d80b2910 0x7fe3d80b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.114+0000 7fe3e7fff700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe3d80b2910 0x7fe3d80b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.114+0000 7fe3e5ffb700 1 -- 172.21.15.114:0/337527403 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fe3dc0e2630 con 0x7fe3e81045d0 2024-05-27T07:38:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.114+0000 7fe3e5ffb700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353] conn(0x7fe3d80bdcc0 0x7fe3d80c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.114+0000 7fe3e5ffb700 1 -- 172.21.15.114:0/337527403 --> [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3dc00fec0 con 0x7fe3d80bdcc0 2024-05-27T07:38:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.114+0000 7fe3e5ffb700 1 -- 172.21.15.114:0/337527403 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fe3dc0e2a80 con 0x7fe3e81045d0 2024-05-27T07:38:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.114+0000 7fe3e7fff700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe3d80b2910 0x7fe3d80b4e20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4006010 tx=0x7fe3d4009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:02.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.115+0000 7fe3ed135700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353] conn(0x7fe3d80bdcc0 0x7fe3d80c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:02.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.115+0000 7fe3ed135700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353] conn(0x7fe3d80bdcc0 0x7fe3d80c01b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:02.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.116+0000 7fe3e5ffb700 1 -- 172.21.15.114:0/337527403 <== osd.8 v2:172.21.15.186:6816/3389603353 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3dc00fec0 con 0x7fe3d80bdcc0 2024-05-27T07:38:02.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.133+0000 7fe3eeb98700 1 -- 172.21.15.114:0/337527403 --> [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe3e80643c0 con 0x7fe3d80bdcc0 2024-05-27T07:38:02.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.133+0000 7fe3e5ffb700 1 -- 172.21.15.114:0/337527403 <== osd.8 v2:172.21.15.186:6816/3389603353 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe3e80643c0 con 0x7fe3d80bdcc0 2024-05-27T07:38:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.133+0000 7fe3eeb98700 1 -- 172.21.15.114:0/337527403 >> [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353] conn(0x7fe3d80bdcc0 msgr2=0x7fe3d80c01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.134+0000 7fe3eeb98700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353] conn(0x7fe3d80bdcc0 0x7fe3d80c01b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.134+0000 7fe3eeb98700 1 -- 172.21.15.114:0/337527403 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe3d80b2910 msgr2=0x7fe3d80b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.134+0000 7fe3eeb98700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe3d80b2910 0x7fe3d80b4e20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4006010 tx=0x7fe3d4009f10 comp rx=0 tx=0).stop 2024-05-27T07:38:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.134+0000 7fe3eeb98700 1 -- 172.21.15.114:0/337527403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3e81045d0 msgr2=0x7fe3e819aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.134+0000 7fe3eeb98700 1 --2- 172.21.15.114:0/337527403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3e81045d0 0x7fe3e819aee0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00b530 tx=0x7fe3dc00b8f0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.134+0000 7fe3ed135700 1 -- 172.21.15.114:0/337527403 reap_dead start 2024-05-27T07:38:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.134+0000 7fe3eeb98700 1 -- 172.21.15.114:0/337527403 shutdown_connections 2024-05-27T07:38:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.134+0000 7fe3eeb98700 1 -- 172.21.15.114:0/337527403 >> 172.21.15.114:0/337527403 conn(0x7fe3e80ffec0 msgr2=0x7fe3e81086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.134+0000 7fe3eeb98700 1 -- 172.21.15.114:0/337527403 shutdown_connections 2024-05-27T07:38:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.134+0000 7fe3eeb98700 1 -- 172.21.15.114:0/337527403 wait complete. 2024-05-27T07:38:02.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.344+0000 7fe5a5570700 1 -- 172.21.15.114:0/1706484559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a0108750 msgr2=0x7fe5a01100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:02.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.344+0000 7fe5a5570700 1 --2- 172.21.15.114:0/1706484559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a0108750 0x7fe5a01100b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe59800b0d0 tx=0x7fe59800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.344+0000 7fe5a5570700 1 -- 172.21.15.114:0/1706484559 shutdown_connections 2024-05-27T07:38:02.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.344+0000 7fe5a5570700 1 --2- 172.21.15.114:0/1706484559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a0108750 0x7fe5a01100b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.344+0000 7fe5a5570700 1 --2- 172.21.15.114:0/1706484559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5a01034d0 0x7fe5a01038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.344+0000 7fe5a5570700 1 --2- 172.21.15.114:0/1706484559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a0107e00 0x7fe5a0108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.344+0000 7fe5a5570700 1 -- 172.21.15.114:0/1706484559 >> 172.21.15.114:0/1706484559 conn(0x7fe5a00fef20 msgr2=0x7fe5a0101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:02.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.344+0000 7fe5a5570700 1 -- 172.21.15.114:0/1706484559 shutdown_connections 2024-05-27T07:38:02.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.344+0000 7fe5a5570700 1 -- 172.21.15.114:0/1706484559 wait complete. 2024-05-27T07:38:02.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.345+0000 7fe5a5570700 1 Processor -- start 2024-05-27T07:38:02.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.345+0000 7fe5a5570700 1 -- start start 2024-05-27T07:38:02.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe5a5570700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5a01034d0 0x7fe5a0199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:02.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe5a5570700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a0107e00 0x7fe5a019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:02.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe5a5570700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a0108750 0x7fe5a019e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:02.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe5a5570700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a0113380 con 0x7fe5a0108750 2024-05-27T07:38:02.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe5a5570700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a0113200 con 0x7fe5a0107e00 2024-05-27T07:38:02.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe5a5570700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a0113500 con 0x7fe5a01034d0 2024-05-27T07:38:02.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe59f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a0108750 0x7fe5a019e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:02.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe59f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a0108750 0x7fe5a019e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41702/0 (socket says 172.21.15.114:41702) 2024-05-27T07:38:02.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe59f7fe700 1 -- 172.21.15.114:0/2627653031 learned_addr learned my addr 172.21.15.114:0/2627653031 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:02.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe59effd700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5a01034d0 0x7fe5a0199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:02.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe59e7fc700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a0107e00 0x7fe5a019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe59f7fe700 1 -- 172.21.15.114:0/2627653031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5a01034d0 msgr2=0x7fe5a0199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe59f7fe700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5a01034d0 0x7fe5a0199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe59f7fe700 1 -- 172.21.15.114:0/2627653031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a0107e00 msgr2=0x7fe5a019a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe59f7fe700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a0107e00 0x7fe5a019a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe59f7fe700 1 -- 172.21.15.114:0/2627653031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5a019ec20 con 0x7fe5a0108750 2024-05-27T07:38:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe59effd700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5a01034d0 0x7fe5a0199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe59f7fe700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a0108750 0x7fe5a019e580 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe598009850 tx=0x7fe598007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.346+0000 7fe597fff700 1 -- 172.21.15.114:0/2627653031 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59801f070 con 0x7fe5a0108750 2024-05-27T07:38:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.347+0000 7fe597fff700 1 -- 172.21.15.114:0/2627653031 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe598005500 con 0x7fe5a0108750 2024-05-27T07:38:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.347+0000 7fe5a5570700 1 -- 172.21.15.114:0/2627653031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5a019eeb0 con 0x7fe5a0108750 2024-05-27T07:38:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.347+0000 7fe597fff700 1 -- 172.21.15.114:0/2627653031 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59801b850 con 0x7fe5a0108750 2024-05-27T07:38:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.347+0000 7fe5a5570700 1 -- 172.21.15.114:0/2627653031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5a019f390 con 0x7fe5a0108750 2024-05-27T07:38:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.347+0000 7fe5a5570700 1 -- 172.21.15.114:0/2627653031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe5a006c300 con 0x7fe5a0108750 2024-05-27T07:38:02.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.349+0000 7fe597fff700 1 -- 172.21.15.114:0/2627653031 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fe598022070 con 0x7fe5a0108750 2024-05-27T07:38:02.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.350+0000 7fe597fff700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe58c0b2910 0x7fe58c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:02.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.350+0000 7fe59effd700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe58c0b2910 0x7fe58c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:02.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.350+0000 7fe597fff700 1 -- 172.21.15.114:0/2627653031 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fe59800f020 con 0x7fe5a0108750 2024-05-27T07:38:02.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.350+0000 7fe597fff700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241] conn(0x7fe58c0bdb90 0x7fe58c0c0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:02.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.350+0000 7fe597fff700 1 -- 172.21.15.114:0/2627653031 --> [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe5980075c0 con 0x7fe58c0bdb90 2024-05-27T07:38:02.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.350+0000 7fe597fff700 1 -- 172.21.15.114:0/2627653031 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fe5980e4210 con 0x7fe5a0108750 2024-05-27T07:38:02.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.350+0000 7fe59effd700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe58c0b2910 0x7fe58c0b4e20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe590005950 tx=0x7fe5900058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:02.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.350+0000 7fe59e7fc700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241] conn(0x7fe58c0bdb90 0x7fe58c0c0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:02.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.351+0000 7fe59e7fc700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241] conn(0x7fe58c0bdb90 0x7fe58c0c0080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:02.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.352+0000 7fe597fff700 1 -- 172.21.15.114:0/2627653031 <== osd.1 v2:172.21.15.163:6800/3356982241 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe5980075c0 con 0x7fe58c0bdb90 2024-05-27T07:38:02.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.368+0000 7fe5a5570700 1 -- 172.21.15.114:0/2627653031 --> [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe5a006c700 con 0x7fe58c0bdb90 2024-05-27T07:38:02.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.369+0000 7fe597fff700 1 -- 172.21.15.114:0/2627653031 <== osd.1 v2:172.21.15.163:6800/3356982241 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe5a006c700 con 0x7fe58c0bdb90 2024-05-27T07:38:02.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.369+0000 7fe5a5570700 1 -- 172.21.15.114:0/2627653031 >> [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241] conn(0x7fe58c0bdb90 msgr2=0x7fe58c0c0080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:02.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.369+0000 7fe5a5570700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241] conn(0x7fe58c0bdb90 0x7fe58c0c0080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.369+0000 7fe5a5570700 1 -- 172.21.15.114:0/2627653031 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe58c0b2910 msgr2=0x7fe58c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:02.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.369+0000 7fe5a5570700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fe58c0b2910 0x7fe58c0b4e20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe590005950 tx=0x7fe5900058e0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.369+0000 7fe5a5570700 1 -- 172.21.15.114:0/2627653031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a0108750 msgr2=0x7fe5a019e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:02.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.369+0000 7fe5a5570700 1 --2- 172.21.15.114:0/2627653031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a0108750 0x7fe5a019e580 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe598009850 tx=0x7fe598007ad0 comp rx=0 tx=0).stop 2024-05-27T07:38:02.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.369+0000 7fe59f7fe700 1 -- 172.21.15.114:0/2627653031 reap_dead start 2024-05-27T07:38:02.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.369+0000 7fe5a5570700 1 -- 172.21.15.114:0/2627653031 shutdown_connections 2024-05-27T07:38:02.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.369+0000 7fe5a5570700 1 -- 172.21.15.114:0/2627653031 >> 172.21.15.114:0/2627653031 conn(0x7fe5a00fef20 msgr2=0x7fe5a0107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:02.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.370+0000 7fe5a5570700 1 -- 172.21.15.114:0/2627653031 shutdown_connections 2024-05-27T07:38:02.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:02.370+0000 7fe5a5570700 1 -- 172.21.15.114:0/2627653031 wait complete. 2024-05-27T07:38:02.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:02 smithi114 ceph-mon[32038]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:02.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:02 smithi163 ceph-mon[36451]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:02.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:02 smithi186 ceph-mon[36647]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.075+0000 7f352154c700 1 -- 172.21.15.114:0/1606937554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351c1045d0 msgr2=0x7f351c1049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.075+0000 7f352154c700 1 --2- 172.21.15.114:0/1606937554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351c1045d0 0x7f351c1049e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f35100099c0 tx=0x7f3510009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.075+0000 7f352154c700 1 -- 172.21.15.114:0/1606937554 shutdown_connections 2024-05-27T07:38:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.075+0000 7f352154c700 1 --2- 172.21.15.114:0/1606937554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351c109850 0x7f351c1111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:03.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.075+0000 7f352154c700 1 --2- 172.21.15.114:0/1606937554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351c1045d0 0x7f351c1049e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:03.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.075+0000 7f352154c700 1 --2- 172.21.15.114:0/1606937554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f351c108f00 0x7f351c109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:03.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.075+0000 7f352154c700 1 -- 172.21.15.114:0/1606937554 >> 172.21.15.114:0/1606937554 conn(0x7f351c0ffe60 msgr2=0x7f351c1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:03.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.075+0000 7f352154c700 1 -- 172.21.15.114:0/1606937554 shutdown_connections 2024-05-27T07:38:03.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.075+0000 7f352154c700 1 -- 172.21.15.114:0/1606937554 wait complete. 2024-05-27T07:38:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.076+0000 7f352154c700 1 Processor -- start 2024-05-27T07:38:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.076+0000 7f352154c700 1 -- start start 2024-05-27T07:38:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.076+0000 7f352154c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351c1045d0 0x7f351c075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.076+0000 7f352154c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351c108f00 0x7f351c075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.076+0000 7f352154c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f351c109850 0x7f351c076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.076+0000 7f352154c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f351c1144f0 con 0x7f351c108f00 2024-05-27T07:38:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.076+0000 7f352154c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f351c114370 con 0x7f351c1045d0 2024-05-27T07:38:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.076+0000 7f352154c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f351c114670 con 0x7f351c109850 2024-05-27T07:38:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.076+0000 7f351a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351c108f00 0x7f351c075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.076+0000 7f351a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351c108f00 0x7f351c075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41714/0 (socket says 172.21.15.114:41714) 2024-05-27T07:38:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.076+0000 7f351affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351c1045d0 0x7f351c075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.076+0000 7f351a7fc700 1 -- 172.21.15.114:0/1415179044 learned_addr learned my addr 172.21.15.114:0/1415179044 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.077+0000 7f351b7fe700 1 --2- 172.21.15.114:0/1415179044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f351c109850 0x7f351c076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.077+0000 7f351a7fc700 1 -- 172.21.15.114:0/1415179044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f351c109850 msgr2=0x7f351c076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.077+0000 7f351a7fc700 1 --2- 172.21.15.114:0/1415179044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f351c109850 0x7f351c076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.077+0000 7f351a7fc700 1 -- 172.21.15.114:0/1415179044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351c1045d0 msgr2=0x7f351c075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.077+0000 7f351a7fc700 1 --2- 172.21.15.114:0/1415179044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351c1045d0 0x7f351c075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.077+0000 7f351a7fc700 1 -- 172.21.15.114:0/1415179044 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f351c1a9200 con 0x7f351c108f00 2024-05-27T07:38:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.077+0000 7f351a7fc700 1 --2- 172.21.15.114:0/1415179044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351c108f00 0x7f351c075ff0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f350c00b500 tx=0x7f350c00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.077+0000 7f3503fff700 1 -- 172.21.15.114:0/1415179044 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c015020 con 0x7f351c108f00 2024-05-27T07:38:03.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.077+0000 7f352154c700 1 -- 172.21.15.114:0/1415179044 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f351c1a94f0 con 0x7f351c108f00 2024-05-27T07:38:03.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.077+0000 7f3503fff700 1 -- 172.21.15.114:0/1415179044 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f350c011ce0 con 0x7f351c108f00 2024-05-27T07:38:03.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.077+0000 7f3503fff700 1 -- 172.21.15.114:0/1415179044 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c010cb0 con 0x7f351c108f00 2024-05-27T07:38:03.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.077+0000 7f352154c700 1 -- 172.21.15.114:0/1415179044 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f351c1a99e0 con 0x7f351c108f00 2024-05-27T07:38:03.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.078+0000 7f352154c700 1 -- 172.21.15.114:0/1415179044 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f351c063ec0 con 0x7f351c108f00 2024-05-27T07:38:03.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.080+0000 7f3503fff700 1 -- 172.21.15.114:0/1415179044 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f350c018020 con 0x7f351c108f00 2024-05-27T07:38:03.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.081+0000 7f3503fff700 1 --2- 172.21.15.114:0/1415179044 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f35040b2910 0x7f35040b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:03.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.081+0000 7f351affd700 1 --2- 172.21.15.114:0/1415179044 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f35040b2910 0x7f35040b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:03.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.081+0000 7f3503fff700 1 -- 172.21.15.114:0/1415179044 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f350c0e2380 con 0x7f351c108f00 2024-05-27T07:38:03.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.081+0000 7f3503fff700 1 --2- 172.21.15.114:0/1415179044 >> [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962] conn(0x7f35040bdcc0 0x7f35040c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:03.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.081+0000 7f3503fff700 1 -- 172.21.15.114:0/1415179044 --> [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f350c0044e0 con 0x7f35040bdcc0 2024-05-27T07:38:03.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.081+0000 7f3503fff700 1 -- 172.21.15.114:0/1415179044 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f350c0e27d0 con 0x7f351c108f00 2024-05-27T07:38:03.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.081+0000 7f351affd700 1 --2- 172.21.15.114:0/1415179044 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f35040b2910 0x7f35040b4e20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3510009990 tx=0x7f35100036e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:03.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.081+0000 7f351b7fe700 1 --2- 172.21.15.114:0/1415179044 >> [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962] conn(0x7f35040bdcc0 0x7f35040c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:03.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.082+0000 7f351b7fe700 1 --2- 172.21.15.114:0/1415179044 >> [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962] conn(0x7f35040bdcc0 0x7f35040c01b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:03.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.083+0000 7f3503fff700 1 -- 172.21.15.114:0/1415179044 <== osd.7 v2:172.21.15.163:6816/1909415962 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f350c0044e0 con 0x7f35040bdcc0 2024-05-27T07:38:03.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.099+0000 7f352154c700 1 -- 172.21.15.114:0/1415179044 --> [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f351c0642c0 con 0x7f35040bdcc0 2024-05-27T07:38:03.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.100+0000 7f3503fff700 1 -- 172.21.15.114:0/1415179044 <== osd.7 v2:172.21.15.163:6816/1909415962 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f351c0642c0 con 0x7f35040bdcc0 2024-05-27T07:38:03.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.101+0000 7f352154c700 1 -- 172.21.15.114:0/1415179044 >> [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962] conn(0x7f35040bdcc0 msgr2=0x7f35040c01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:03.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.101+0000 7f352154c700 1 --2- 172.21.15.114:0/1415179044 >> [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962] conn(0x7f35040bdcc0 0x7f35040c01b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:03.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.101+0000 7f352154c700 1 -- 172.21.15.114:0/1415179044 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f35040b2910 msgr2=0x7f35040b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:03.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.101+0000 7f352154c700 1 --2- 172.21.15.114:0/1415179044 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f35040b2910 0x7f35040b4e20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3510009990 tx=0x7f35100036e0 comp rx=0 tx=0).stop 2024-05-27T07:38:03.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.101+0000 7f352154c700 1 -- 172.21.15.114:0/1415179044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351c108f00 msgr2=0x7f351c075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:03.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.101+0000 7f352154c700 1 --2- 172.21.15.114:0/1415179044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351c108f00 0x7f351c075ff0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f350c00b500 tx=0x7f350c00b810 comp rx=0 tx=0).stop 2024-05-27T07:38:03.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.101+0000 7f351b7fe700 1 -- 172.21.15.114:0/1415179044 reap_dead start 2024-05-27T07:38:03.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.101+0000 7f352154c700 1 -- 172.21.15.114:0/1415179044 shutdown_connections 2024-05-27T07:38:03.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.101+0000 7f352154c700 1 -- 172.21.15.114:0/1415179044 >> 172.21.15.114:0/1415179044 conn(0x7f351c0ffe60 msgr2=0x7f351c100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:03.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.101+0000 7f352154c700 1 -- 172.21.15.114:0/1415179044 shutdown_connections 2024-05-27T07:38:03.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.101+0000 7f352154c700 1 -- 172.21.15.114:0/1415179044 wait complete. 2024-05-27T07:38:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.301+0000 7f5420773700 1 -- 172.21.15.114:0/537267619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5418107de0 msgr2=0x7f54181081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.301+0000 7f5420773700 1 --2- 172.21.15.114:0/537267619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5418107de0 0x7f54181081f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f54080099c0 tx=0x7f5408009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.301+0000 7f5420773700 1 -- 172.21.15.114:0/537267619 shutdown_connections 2024-05-27T07:38:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.301+0000 7f5420773700 1 --2- 172.21.15.114:0/537267619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5418108730 0x7f5418110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.301+0000 7f5420773700 1 --2- 172.21.15.114:0/537267619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54181034b0 0x7f54181038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.301+0000 7f5420773700 1 --2- 172.21.15.114:0/537267619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5418107de0 0x7f54181081f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.301+0000 7f5420773700 1 -- 172.21.15.114:0/537267619 >> 172.21.15.114:0/537267619 conn(0x7f54180fedb0 msgr2=0x7f54181011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.301+0000 7f5420773700 1 -- 172.21.15.114:0/537267619 shutdown_connections 2024-05-27T07:38:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.301+0000 7f5420773700 1 -- 172.21.15.114:0/537267619 wait complete. 2024-05-27T07:38:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f5420773700 1 Processor -- start 2024-05-27T07:38:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f5420773700 1 -- start start 2024-05-27T07:38:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f5420773700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54181034b0 0x7f541819bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f5420773700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5418107de0 0x7f541819c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f5420773700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5418108730 0x7f541819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f5420773700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5418113340 con 0x7f5418107de0 2024-05-27T07:38:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f5420773700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f54181131c0 con 0x7f5418108730 2024-05-27T07:38:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f5420773700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f54181134c0 con 0x7f54181034b0 2024-05-27T07:38:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f541dd0e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5418107de0 0x7f541819c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f541dd0e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5418107de0 0x7f541819c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41738/0 (socket says 172.21.15.114:41738) 2024-05-27T07:38:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f541dd0e700 1 -- 172.21.15.114:0/1995099186 learned_addr learned my addr 172.21.15.114:0/1995099186 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f541e50f700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54181034b0 0x7f541819bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f541ed10700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5418108730 0x7f541819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f541dd0e700 1 -- 172.21.15.114:0/1995099186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54181034b0 msgr2=0x7f541819bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f541dd0e700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54181034b0 0x7f541819bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f541dd0e700 1 -- 172.21.15.114:0/1995099186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5418108730 msgr2=0x7f541819c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f541dd0e700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5418108730 0x7f541819c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.302+0000 7f541dd0e700 1 -- 172.21.15.114:0/1995099186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54181a6fd0 con 0x7f5418107de0 2024-05-27T07:38:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.303+0000 7f541dd0e700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5418107de0 0x7f541819c1c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f54080098e0 tx=0x7f54080057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.303+0000 7f540f7fe700 1 -- 172.21.15.114:0/1995099186 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408027070 con 0x7f5418107de0 2024-05-27T07:38:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.303+0000 7f540f7fe700 1 -- 172.21.15.114:0/1995099186 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5408003f20 con 0x7f5418107de0 2024-05-27T07:38:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.303+0000 7f540f7fe700 1 -- 172.21.15.114:0/1995099186 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408022c70 con 0x7f5418107de0 2024-05-27T07:38:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.303+0000 7f5420773700 1 -- 172.21.15.114:0/1995099186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54181a7260 con 0x7f5418107de0 2024-05-27T07:38:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.303+0000 7f5420773700 1 -- 172.21.15.114:0/1995099186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54181a7660 con 0x7f5418107de0 2024-05-27T07:38:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.304+0000 7f5420773700 1 -- 172.21.15.114:0/1995099186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5418063e80 con 0x7f5418107de0 2024-05-27T07:38:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.306+0000 7f540f7fe700 1 -- 172.21.15.114:0/1995099186 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f540802a070 con 0x7f5418107de0 2024-05-27T07:38:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.307+0000 7f540f7fe700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f54040b2910 0x7f54040b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.307+0000 7f541e50f700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f54040b2910 0x7f54040b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.307+0000 7f540f7fe700 1 -- 172.21.15.114:0/1995099186 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f54080e4060 con 0x7f5418107de0 2024-05-27T07:38:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.307+0000 7f540f7fe700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915] conn(0x7f54040bdcc0 0x7f54040c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.307+0000 7f540f7fe700 1 -- 172.21.15.114:0/1995099186 --> [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f54040c0880 con 0x7f54040bdcc0 2024-05-27T07:38:04.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.307+0000 7f540f7fe700 1 -- 172.21.15.114:0/1995099186 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f54080e44b0 con 0x7f5418107de0 2024-05-27T07:38:04.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.307+0000 7f541ed10700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915] conn(0x7f54040bdcc0 0x7f54040c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.307+0000 7f541e50f700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f54040b2910 0x7f54040b4e20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5414007be0 tx=0x7f54140073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.307+0000 7f541ed10700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915] conn(0x7f54040bdcc0 0x7f54040c01b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.308+0000 7f540f7fe700 1 -- 172.21.15.114:0/1995099186 <== osd.9 v2:172.21.15.114:6826/3900437915 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f54040c0880 con 0x7f54040bdcc0 2024-05-27T07:38:04.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.325+0000 7f5420773700 1 -- 172.21.15.114:0/1995099186 --> [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5418064300 con 0x7f54040bdcc0 2024-05-27T07:38:04.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.325+0000 7f540f7fe700 1 -- 172.21.15.114:0/1995099186 <== osd.9 v2:172.21.15.114:6826/3900437915 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5418064300 con 0x7f54040bdcc0 2024-05-27T07:38:04.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.325+0000 7f5420773700 1 -- 172.21.15.114:0/1995099186 >> [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915] conn(0x7f54040bdcc0 msgr2=0x7f54040c01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.325+0000 7f5420773700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915] conn(0x7f54040bdcc0 0x7f54040c01b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.326+0000 7f5420773700 1 -- 172.21.15.114:0/1995099186 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f54040b2910 msgr2=0x7f54040b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.326+0000 7f5420773700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f54040b2910 0x7f54040b4e20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5414007be0 tx=0x7f54140073d0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.326+0000 7f5420773700 1 -- 172.21.15.114:0/1995099186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5418107de0 msgr2=0x7f541819c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.326+0000 7f5420773700 1 --2- 172.21.15.114:0/1995099186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5418107de0 0x7f541819c1c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f54080098e0 tx=0x7f54080057c0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.326+0000 7f541ed10700 1 -- 172.21.15.114:0/1995099186 reap_dead start 2024-05-27T07:38:04.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.326+0000 7f5420773700 1 -- 172.21.15.114:0/1995099186 shutdown_connections 2024-05-27T07:38:04.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.326+0000 7f5420773700 1 -- 172.21.15.114:0/1995099186 >> 172.21.15.114:0/1995099186 conn(0x7f54180fedb0 msgr2=0x7f5418109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.326+0000 7f5420773700 1 -- 172.21.15.114:0/1995099186 shutdown_connections 2024-05-27T07:38:04.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:03.326+0000 7f5420773700 1 -- 172.21.15.114:0/1995099186 wait complete. 2024-05-27T07:38:04.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:04 smithi114 ceph-mon[32038]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:04.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:04 smithi163 ceph-mon[36451]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:04.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:04 smithi186 ceph-mon[36647]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:05.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.073+0000 7f80c6493700 1 -- 172.21.15.114:0/734170459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c01034d0 msgr2=0x7f80c01038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:05.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.073+0000 7f80c6493700 1 --2- 172.21.15.114:0/734170459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c01034d0 0x7f80c01038e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f80b00099c0 tx=0x7f80b0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:05.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.073+0000 7f80c6493700 1 -- 172.21.15.114:0/734170459 shutdown_connections 2024-05-27T07:38:05.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.073+0000 7f80c6493700 1 --2- 172.21.15.114:0/734170459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80c0108750 0x7f80c01100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:05.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.073+0000 7f80c6493700 1 --2- 172.21.15.114:0/734170459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c01034d0 0x7f80c01038e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:05.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.073+0000 7f80c6493700 1 --2- 172.21.15.114:0/734170459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80c0107e00 0x7f80c0108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.073+0000 7f80c6493700 1 -- 172.21.15.114:0/734170459 >> 172.21.15.114:0/734170459 conn(0x7f80c00fef20 msgr2=0x7f80c0101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.073+0000 7f80c6493700 1 -- 172.21.15.114:0/734170459 shutdown_connections 2024-05-27T07:38:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.073+0000 7f80c6493700 1 -- 172.21.15.114:0/734170459 wait complete. 2024-05-27T07:38:05.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.073+0000 7f80c6493700 1 Processor -- start 2024-05-27T07:38:05.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.073+0000 7f80c6493700 1 -- start start 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80c6493700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80c01034d0 0x7f80c019bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80c6493700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c0107e00 0x7f80c019c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80c6493700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80c0108750 0x7f80c019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80c6493700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f80c0113380 con 0x7f80c0107e00 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80c6493700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f80c0113200 con 0x7f80c01034d0 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80c6493700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f80c0113500 con 0x7f80c0108750 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80bf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c0107e00 0x7f80c019c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80bf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c0107e00 0x7f80c019c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34442/0 (socket says 172.21.15.114:34442) 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80bf7fe700 1 -- 172.21.15.114:0/2997675100 learned_addr learned my addr 172.21.15.114:0/2997675100 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80bffff700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80c01034d0 0x7f80c019bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80c4a30700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80c0108750 0x7f80c019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80bf7fe700 1 -- 172.21.15.114:0/2997675100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80c0108750 msgr2=0x7f80c019c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80bf7fe700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80c0108750 0x7f80c019c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80bf7fe700 1 -- 172.21.15.114:0/2997675100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80c01034d0 msgr2=0x7f80c019bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:05.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80bf7fe700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80c01034d0 0x7f80c019bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:05.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.074+0000 7f80bf7fe700 1 -- 172.21.15.114:0/2997675100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c01a6fb0 con 0x7f80c0107e00 2024-05-27T07:38:05.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.075+0000 7f80bf7fe700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c0107e00 0x7f80c019c1a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f80a800d980 tx=0x7f80a800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:05.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.075+0000 7f80bd7fa700 1 -- 172.21.15.114:0/2997675100 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a8017070 con 0x7f80c0107e00 2024-05-27T07:38:05.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.075+0000 7f80c6493700 1 -- 172.21.15.114:0/2997675100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c01a72a0 con 0x7f80c0107e00 2024-05-27T07:38:05.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.075+0000 7f80bd7fa700 1 -- 172.21.15.114:0/2997675100 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80a800fd40 con 0x7f80c0107e00 2024-05-27T07:38:05.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.075+0000 7f80bd7fa700 1 -- 172.21.15.114:0/2997675100 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a800e3b0 con 0x7f80c0107e00 2024-05-27T07:38:05.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.075+0000 7f80c6493700 1 -- 172.21.15.114:0/2997675100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c01a7830 con 0x7f80c0107e00 2024-05-27T07:38:05.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.076+0000 7f80c6493700 1 -- 172.21.15.114:0/2997675100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f80c0063ec0 con 0x7f80c0107e00 2024-05-27T07:38:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.078+0000 7f80bd7fa700 1 -- 172.21.15.114:0/2997675100 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f80a8010610 con 0x7f80c0107e00 2024-05-27T07:38:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.078+0000 7f80bd7fa700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f80ac0b2910 0x7f80ac0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.079+0000 7f80bffff700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f80ac0b2910 0x7f80ac0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.079+0000 7f80bd7fa700 1 -- 172.21.15.114:0/2997675100 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f80a80e27b0 con 0x7f80c0107e00 2024-05-27T07:38:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.079+0000 7f80bd7fa700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170] conn(0x7f80ac0bdcc0 0x7f80ac0c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.079+0000 7f80bd7fa700 1 -- 172.21.15.114:0/2997675100 --> [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80a8016ec0 con 0x7f80ac0bdcc0 2024-05-27T07:38:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.079+0000 7f80bd7fa700 1 -- 172.21.15.114:0/2997675100 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f80a80e2c00 con 0x7f80c0107e00 2024-05-27T07:38:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.079+0000 7f80c4a30700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170] conn(0x7f80ac0bdcc0 0x7f80ac0c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.079+0000 7f80bffff700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f80ac0b2910 0x7f80ac0b4e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009990 tx=0x7f80b00036e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.080+0000 7f80c4a30700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170] conn(0x7f80ac0bdcc0 0x7f80ac0c01b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.080+0000 7f80bd7fa700 1 -- 172.21.15.114:0/2997675100 <== osd.3 v2:172.21.15.114:6810/1973916170 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80a8016ec0 con 0x7f80ac0bdcc0 2024-05-27T07:38:05.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.097+0000 7f80c6493700 1 -- 172.21.15.114:0/2997675100 --> [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f80c00642f0 con 0x7f80ac0bdcc0 2024-05-27T07:38:05.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.098+0000 7f80bd7fa700 1 -- 172.21.15.114:0/2997675100 <== osd.3 v2:172.21.15.114:6810/1973916170 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f80c00642f0 con 0x7f80ac0bdcc0 2024-05-27T07:38:05.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.098+0000 7f80c6493700 1 -- 172.21.15.114:0/2997675100 >> [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170] conn(0x7f80ac0bdcc0 msgr2=0x7f80ac0c01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:05.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.098+0000 7f80c6493700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170] conn(0x7f80ac0bdcc0 0x7f80ac0c01b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:05.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.098+0000 7f80c6493700 1 -- 172.21.15.114:0/2997675100 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f80ac0b2910 msgr2=0x7f80ac0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:05.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.098+0000 7f80c6493700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f80ac0b2910 0x7f80ac0b4e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009990 tx=0x7f80b00036e0 comp rx=0 tx=0).stop 2024-05-27T07:38:05.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.098+0000 7f80c6493700 1 -- 172.21.15.114:0/2997675100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c0107e00 msgr2=0x7f80c019c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:05.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.098+0000 7f80c6493700 1 --2- 172.21.15.114:0/2997675100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c0107e00 0x7f80c019c1a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f80a800d980 tx=0x7f80a800dc90 comp rx=0 tx=0).stop 2024-05-27T07:38:05.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.098+0000 7f80c4a30700 1 -- 172.21.15.114:0/2997675100 reap_dead start 2024-05-27T07:38:05.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.099+0000 7f80c6493700 1 -- 172.21.15.114:0/2997675100 shutdown_connections 2024-05-27T07:38:05.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.099+0000 7f80c6493700 1 -- 172.21.15.114:0/2997675100 >> 172.21.15.114:0/2997675100 conn(0x7f80c00fef20 msgr2=0x7f80c01096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:05.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.099+0000 7f80c6493700 1 -- 172.21.15.114:0/2997675100 shutdown_connections 2024-05-27T07:38:05.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:05.099+0000 7f80c6493700 1 -- 172.21.15.114:0/2997675100 wait complete. 2024-05-27T07:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout:55834574939 2024-05-27T07:38:06.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-27T07:38:06.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:06 smithi114 ceph-mon[32038]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:06.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:06 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:38:06.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:06 smithi163 ceph-mon[36451]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:06.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:06 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:38:06.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:06 smithi186 ceph-mon[36647]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:06.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:06 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:38:07.482 INFO:teuthology.orchestra.run.smithi114.stdout:227633266742 2024-05-27T07:38:07.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-27T07:38:07.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.685+0000 7f7f1856a700 1 -- 172.21.15.114:0/679829480 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f10108730 msgr2=0x7f7f10110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:07.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.685+0000 7f7f1856a700 1 --2- 172.21.15.114:0/679829480 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f10108730 0x7f7f10110090 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800b0d0 tx=0x7f7f0800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:38:07.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.685+0000 7f7f1856a700 1 -- 172.21.15.114:0/679829480 shutdown_connections 2024-05-27T07:38:07.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.685+0000 7f7f1856a700 1 --2- 172.21.15.114:0/679829480 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f10108730 0x7f7f10110090 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:07.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.685+0000 7f7f1856a700 1 --2- 172.21.15.114:0/679829480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f101034b0 0x7f7f101038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:07.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.685+0000 7f7f1856a700 1 --2- 172.21.15.114:0/679829480 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f10107de0 0x7f7f101081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:07.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.685+0000 7f7f1856a700 1 -- 172.21.15.114:0/679829480 >> 172.21.15.114:0/679829480 conn(0x7f7f100fedb0 msgr2=0x7f7f101011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:07.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.685+0000 7f7f1856a700 1 -- 172.21.15.114:0/679829480 shutdown_connections 2024-05-27T07:38:07.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.685+0000 7f7f1856a700 1 -- 172.21.15.114:0/679829480 wait complete. 2024-05-27T07:38:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.686+0000 7f7f1856a700 1 Processor -- start 2024-05-27T07:38:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.686+0000 7f7f1856a700 1 -- start start 2024-05-27T07:38:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.686+0000 7f7f1856a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f101034b0 0x7f7f1019f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.686+0000 7f7f1856a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f10107de0 0x7f7f1019fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.686+0000 7f7f1856a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f10108730 0x7f7f10199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.686+0000 7f7f1856a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7f10113360 con 0x7f7f10108730 2024-05-27T07:38:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.686+0000 7f7f1856a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7f101131e0 con 0x7f7f10107de0 2024-05-27T07:38:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.686+0000 7f7f1856a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7f101134e0 con 0x7f7f101034b0 2024-05-27T07:38:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.687+0000 7f7f16306700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f101034b0 0x7f7f1019f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.687+0000 7f7f16b07700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f10108730 0x7f7f10199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.687+0000 7f7f16306700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f101034b0 0x7f7f1019f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47550/0 (socket says 172.21.15.114:47550) 2024-05-27T07:38:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.687+0000 7f7f16306700 1 -- 172.21.15.114:0/1111272494 learned_addr learned my addr 172.21.15.114:0/1111272494 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.687+0000 7f7f15b05700 1 --2- 172.21.15.114:0/1111272494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f10107de0 0x7f7f1019fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.687+0000 7f7f16b07700 1 -- 172.21.15.114:0/1111272494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f101034b0 msgr2=0x7f7f1019f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.687+0000 7f7f16b07700 1 --2- 172.21.15.114:0/1111272494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f101034b0 0x7f7f1019f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.687+0000 7f7f16b07700 1 -- 172.21.15.114:0/1111272494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f10107de0 msgr2=0x7f7f1019fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.687+0000 7f7f16b07700 1 --2- 172.21.15.114:0/1111272494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f10107de0 0x7f7f1019fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.687+0000 7f7f16b07700 1 -- 172.21.15.114:0/1111272494 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1019a170 con 0x7f7f10108730 2024-05-27T07:38:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.687+0000 7f7f16b07700 1 --2- 172.21.15.114:0/1111272494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f10108730 0x7f7f10199ac0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800b0d0 tx=0x7f7f08007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.688+0000 7f7f077fe700 1 -- 172.21.15.114:0/1111272494 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08004c50 con 0x7f7f10108730 2024-05-27T07:38:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.688+0000 7f7f077fe700 1 -- 172.21.15.114:0/1111272494 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f08004df0 con 0x7f7f10108730 2024-05-27T07:38:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.688+0000 7f7f1856a700 1 -- 172.21.15.114:0/1111272494 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1019a400 con 0x7f7f10108730 2024-05-27T07:38:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.688+0000 7f7f077fe700 1 -- 172.21.15.114:0/1111272494 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f080054d0 con 0x7f7f10108730 2024-05-27T07:38:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.688+0000 7f7f1856a700 1 -- 172.21.15.114:0/1111272494 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f1019a8e0 con 0x7f7f10108730 2024-05-27T07:38:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.689+0000 7f7f1856a700 1 -- 172.21.15.114:0/1111272494 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7f1006b950 con 0x7f7f10108730 2024-05-27T07:38:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.691+0000 7f7f077fe700 1 -- 172.21.15.114:0/1111272494 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f7f08005670 con 0x7f7f10108730 2024-05-27T07:38:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.691+0000 7f7f077fe700 1 --2- 172.21.15.114:0/1111272494 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7efc0b2910 0x7f7efc0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.692+0000 7f7f077fe700 1 -- 172.21.15.114:0/1111272494 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f7f0800f030 con 0x7f7f10108730 2024-05-27T07:38:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.692+0000 7f7f16306700 1 --2- 172.21.15.114:0/1111272494 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7efc0b2910 0x7f7efc0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.692+0000 7f7f077fe700 1 --2- 172.21.15.114:0/1111272494 >> [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501] conn(0x7f7efc0bdcc0 0x7f7efc0c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.692+0000 7f7f077fe700 1 -- 172.21.15.114:0/1111272494 --> [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f08014e50 con 0x7f7efc0bdcc0 2024-05-27T07:38:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.692+0000 7f7f077fe700 1 -- 172.21.15.114:0/1111272494 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f7f080e7af0 con 0x7f7f10108730 2024-05-27T07:38:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.692+0000 7f7f16306700 1 --2- 172.21.15.114:0/1111272494 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7efc0b2910 0x7f7efc0b4e20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7f1019ac00 tx=0x7f7f0c010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.692+0000 7f7f15b05700 1 --2- 172.21.15.114:0/1111272494 >> [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501] conn(0x7f7efc0bdcc0 0x7f7efc0c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.693+0000 7f7f15b05700 1 --2- 172.21.15.114:0/1111272494 >> [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501] conn(0x7f7efc0bdcc0 0x7f7efc0c01b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.694+0000 7f7f077fe700 1 -- 172.21.15.114:0/1111272494 <== osd.11 v2:172.21.15.186:6824/2406495501 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7f08014e50 con 0x7f7efc0bdcc0 2024-05-27T07:38:07.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.714+0000 7f7f1856a700 1 -- 172.21.15.114:0/1111272494 --> [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7f1006bdd0 con 0x7f7efc0bdcc0 2024-05-27T07:38:07.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.715+0000 7f7f077fe700 1 -- 172.21.15.114:0/1111272494 <== osd.11 v2:172.21.15.186:6824/2406495501 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7f1006bdd0 con 0x7f7efc0bdcc0 2024-05-27T07:38:07.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.715+0000 7f7f1856a700 1 -- 172.21.15.114:0/1111272494 >> [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501] conn(0x7f7efc0bdcc0 msgr2=0x7f7efc0c01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:07.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.715+0000 7f7f1856a700 1 --2- 172.21.15.114:0/1111272494 >> [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501] conn(0x7f7efc0bdcc0 0x7f7efc0c01b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:07.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.715+0000 7f7f1856a700 1 -- 172.21.15.114:0/1111272494 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7efc0b2910 msgr2=0x7f7efc0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:07.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.715+0000 7f7f1856a700 1 --2- 172.21.15.114:0/1111272494 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7efc0b2910 0x7f7efc0b4e20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7f1019ac00 tx=0x7f7f0c010040 comp rx=0 tx=0).stop 2024-05-27T07:38:07.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.715+0000 7f7f1856a700 1 -- 172.21.15.114:0/1111272494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f10108730 msgr2=0x7f7f10199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:07.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.715+0000 7f7f1856a700 1 --2- 172.21.15.114:0/1111272494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f10108730 0x7f7f10199ac0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800b0d0 tx=0x7f7f08007a00 comp rx=0 tx=0).stop 2024-05-27T07:38:07.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.715+0000 7f7f16b07700 1 -- 172.21.15.114:0/1111272494 reap_dead start 2024-05-27T07:38:07.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.715+0000 7f7f1856a700 1 -- 172.21.15.114:0/1111272494 shutdown_connections 2024-05-27T07:38:07.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.715+0000 7f7f1856a700 1 -- 172.21.15.114:0/1111272494 >> 172.21.15.114:0/1111272494 conn(0x7f7f100fedb0 msgr2=0x7f7f10107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:07.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.715+0000 7f7f1856a700 1 -- 172.21.15.114:0/1111272494 shutdown_connections 2024-05-27T07:38:07.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:07.715+0000 7f7f1856a700 1 -- 172.21.15.114:0/1111272494 wait complete. 2024-05-27T07:38:08.109 INFO:teuthology.orchestra.run.smithi114.stdout:107374182482 2024-05-27T07:38:08.110 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-27T07:38:08.537 INFO:teuthology.orchestra.run.smithi114.stdout:201863462971 2024-05-27T07:38:08.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-27T07:38:09.152 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:08 smithi114 ceph-mon[32038]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-05-27T07:38:09.152 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:09.152 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:09.153 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:09.153 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:08 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:09.186 INFO:teuthology.orchestra.run.smithi114.stdout:34359738464 2024-05-27T07:38:09.187 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-27T07:38:09.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:08 smithi163 ceph-mon[36451]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-05-27T07:38:09.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:09.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:09.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:09.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:08 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:09.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:08 smithi186 ceph-mon[36647]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-05-27T07:38:09.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:09.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:09.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:09.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:08 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:09.573 INFO:teuthology.orchestra.run.smithi114.stdout:253403070514 2024-05-27T07:38:09.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-27T07:38:09.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.635+0000 7f5d0767b700 1 -- 172.21.15.114:0/2370939968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d001034b0 msgr2=0x7f5d001038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:09.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.635+0000 7f5d0767b700 1 --2- 172.21.15.114:0/2370939968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d001034b0 0x7f5d001038c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00099c0 tx=0x7f5cf0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:09.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.635+0000 7f5d0767b700 1 -- 172.21.15.114:0/2370939968 shutdown_connections 2024-05-27T07:38:09.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.635+0000 7f5d0767b700 1 --2- 172.21.15.114:0/2370939968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d00108730 0x7f5d00110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:09.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.635+0000 7f5d0767b700 1 --2- 172.21.15.114:0/2370939968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d001034b0 0x7f5d001038c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:09.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.635+0000 7f5d0767b700 1 --2- 172.21.15.114:0/2370939968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d00107de0 0x7f5d001081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:09.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.635+0000 7f5d0767b700 1 -- 172.21.15.114:0/2370939968 >> 172.21.15.114:0/2370939968 conn(0x7f5d000fedb0 msgr2=0x7f5d001011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:09.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.636+0000 7f5d0767b700 1 -- 172.21.15.114:0/2370939968 shutdown_connections 2024-05-27T07:38:09.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.636+0000 7f5d0767b700 1 -- 172.21.15.114:0/2370939968 wait complete. 2024-05-27T07:38:09.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.636+0000 7f5d0767b700 1 Processor -- start 2024-05-27T07:38:09.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.636+0000 7f5d0767b700 1 -- start start 2024-05-27T07:38:09.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d0767b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d001034b0 0x7f5d000ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:09.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d0767b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d00107de0 0x7f5d000ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:09.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d0767b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d00108730 0x7f5d000ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:09.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d0767b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5d00113360 con 0x7f5d00108730 2024-05-27T07:38:09.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d0767b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5d001131e0 con 0x7f5d00107de0 2024-05-27T07:38:09.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d0767b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5d001134e0 con 0x7f5d001034b0 2024-05-27T07:38:09.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d05c18700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d00108730 0x7f5d000ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:09.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d05417700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d001034b0 0x7f5d000ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:09.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d05c18700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d00108730 0x7f5d000ffd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34458/0 (socket says 172.21.15.114:34458) 2024-05-27T07:38:09.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d05c18700 1 -- 172.21.15.114:0/4063231986 learned_addr learned my addr 172.21.15.114:0/4063231986 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:09.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d04c16700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d00107de0 0x7f5d000ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:09.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d05417700 1 -- 172.21.15.114:0/4063231986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d00107de0 msgr2=0x7f5d000ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:09.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d05417700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d00107de0 0x7f5d000ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:09.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d05417700 1 -- 172.21.15.114:0/4063231986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d00108730 msgr2=0x7f5d000ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:09.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d05417700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d00108730 0x7f5d000ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:09.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d05417700 1 -- 172.21.15.114:0/4063231986 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d001a91a0 con 0x7f5d001034b0 2024-05-27T07:38:09.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d05c18700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d00108730 0x7f5d000ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:38:09.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.637+0000 7f5d05417700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d001034b0 0x7f5d000ff290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00098e0 tx=0x7f5cf0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:09.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.638+0000 7f5cf67fc700 1 -- 172.21.15.114:0/4063231986 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0026070 con 0x7f5d001034b0 2024-05-27T07:38:09.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.638+0000 7f5cf67fc700 1 -- 172.21.15.114:0/4063231986 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cf0003c70 con 0x7f5d001034b0 2024-05-27T07:38:09.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.638+0000 7f5d0767b700 1 -- 172.21.15.114:0/4063231986 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d001a9430 con 0x7f5d001034b0 2024-05-27T07:38:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.638+0000 7f5cf67fc700 1 -- 172.21.15.114:0/4063231986 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf000f9c0 con 0x7f5d001034b0 2024-05-27T07:38:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.638+0000 7f5d0767b700 1 -- 172.21.15.114:0/4063231986 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d001a9880 con 0x7f5d001034b0 2024-05-27T07:38:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.638+0000 7f5d0767b700 1 -- 172.21.15.114:0/4063231986 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5d00063ec0 con 0x7f5d001034b0 2024-05-27T07:38:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.641+0000 7f5cf67fc700 1 -- 172.21.15.114:0/4063231986 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f5cf0029070 con 0x7f5d001034b0 2024-05-27T07:38:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.641+0000 7f5cf67fc700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5cec0b2910 0x7f5cec0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.641+0000 7f5d04c16700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5cec0b2910 0x7f5cec0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.641+0000 7f5cf67fc700 1 -- 172.21.15.114:0/4063231986 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f5cf00e3850 con 0x7f5d001034b0 2024-05-27T07:38:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.642+0000 7f5cf67fc700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241] conn(0x7f5cec0bdc40 0x7f5cec0c0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.642+0000 7f5cf67fc700 1 -- 172.21.15.114:0/4063231986 --> [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5cf0030e50 con 0x7f5cec0bdc40 2024-05-27T07:38:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.642+0000 7f5cf67fc700 1 -- 172.21.15.114:0/4063231986 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f5cf00e3c90 con 0x7f5d001034b0 2024-05-27T07:38:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.642+0000 7f5d04c16700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5cec0b2910 0x7f5cec0b4e20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5d00109b20 tx=0x7f5cfc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:09.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.642+0000 7f5d05c18700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241] conn(0x7f5cec0bdc40 0x7f5cec0c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:09.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.642+0000 7f5d05c18700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241] conn(0x7f5cec0bdc40 0x7f5cec0c0130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:09.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.643+0000 7f5cf67fc700 1 -- 172.21.15.114:0/4063231986 <== osd.5 v2:172.21.15.186:6808/3483853241 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5cf0030e50 con 0x7f5cec0bdc40 2024-05-27T07:38:09.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.660+0000 7f5d0767b700 1 -- 172.21.15.114:0/4063231986 --> [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5d000642c0 con 0x7f5cec0bdc40 2024-05-27T07:38:09.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.660+0000 7f5cf67fc700 1 -- 172.21.15.114:0/4063231986 <== osd.5 v2:172.21.15.186:6808/3483853241 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5d000642c0 con 0x7f5cec0bdc40 2024-05-27T07:38:09.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.661+0000 7f5d0767b700 1 -- 172.21.15.114:0/4063231986 >> [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241] conn(0x7f5cec0bdc40 msgr2=0x7f5cec0c0130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:09.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.661+0000 7f5d0767b700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241] conn(0x7f5cec0bdc40 0x7f5cec0c0130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:09.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.661+0000 7f5d0767b700 1 -- 172.21.15.114:0/4063231986 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5cec0b2910 msgr2=0x7f5cec0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:09.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.661+0000 7f5d0767b700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5cec0b2910 0x7f5cec0b4e20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5d00109b20 tx=0x7f5cfc009260 comp rx=0 tx=0).stop 2024-05-27T07:38:09.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.661+0000 7f5d0767b700 1 -- 172.21.15.114:0/4063231986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d001034b0 msgr2=0x7f5d000ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:09.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.661+0000 7f5d0767b700 1 --2- 172.21.15.114:0/4063231986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d001034b0 0x7f5d000ff290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00098e0 tx=0x7f5cf0004270 comp rx=0 tx=0).stop 2024-05-27T07:38:09.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.661+0000 7f5d05c18700 1 -- 172.21.15.114:0/4063231986 reap_dead start 2024-05-27T07:38:09.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.661+0000 7f5d0767b700 1 -- 172.21.15.114:0/4063231986 shutdown_connections 2024-05-27T07:38:09.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.661+0000 7f5d0767b700 1 -- 172.21.15.114:0/4063231986 >> 172.21.15.114:0/4063231986 conn(0x7f5d000fedb0 msgr2=0x7f5d0006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:09.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.661+0000 7f5d0767b700 1 -- 172.21.15.114:0/4063231986 shutdown_connections 2024-05-27T07:38:09.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:06.661+0000 7f5d0767b700 1 -- 172.21.15.114:0/4063231986 wait complete. 2024-05-27T07:38:10.594 INFO:teuthology.orchestra.run.smithi114.stdout:154618822727 2024-05-27T07:38:10.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-27T07:38:11.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:10 smithi163 ceph-mon[36451]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:11.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:10 smithi186 ceph-mon[36647]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:11.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:10 smithi114 ceph-mon[32038]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.654+0000 7f571e463700 1 -- 172.21.15.114:0/1891343102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57181051e0 msgr2=0x7f57181055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.654+0000 7f571e463700 1 --2- 172.21.15.114:0/1891343102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57181051e0 0x7f57181055f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5708009990 tx=0x7f5708009ca0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.654+0000 7f571e463700 1 -- 172.21.15.114:0/1891343102 shutdown_connections 2024-05-27T07:38:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.654+0000 7f571e463700 1 --2- 172.21.15.114:0/1891343102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5718103360 0x7f5718112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.654+0000 7f571e463700 1 --2- 172.21.15.114:0/1891343102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57181051e0 0x7f57181055f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.654+0000 7f571e463700 1 --2- 172.21.15.114:0/1891343102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5718102840 0x7f5718102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.654+0000 7f571e463700 1 -- 172.21.15.114:0/1891343102 >> 172.21.15.114:0/1891343102 conn(0x7f57180fe4d0 msgr2=0x7f5718100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.654+0000 7f571e463700 1 -- 172.21.15.114:0/1891343102 shutdown_connections 2024-05-27T07:38:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.654+0000 7f571e463700 1 -- 172.21.15.114:0/1891343102 wait complete. 2024-05-27T07:38:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.655+0000 7f571e463700 1 Processor -- start 2024-05-27T07:38:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.655+0000 7f571e463700 1 -- start start 2024-05-27T07:38:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.655+0000 7f571e463700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5718102840 0x7f571819d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.655+0000 7f571e463700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5718103360 0x7f571819d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.655+0000 7f571e463700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57181051e0 0x7f57181a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.655+0000 7f571e463700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f57181166e0 con 0x7f5718102840 2024-05-27T07:38:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.655+0000 7f571e463700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5718116560 con 0x7f57181051e0 2024-05-27T07:38:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.655+0000 7f571e463700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5718116860 con 0x7f5718103360 2024-05-27T07:38:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f5717fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5718102840 0x7f571819d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f5717fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5718102840 0x7f571819d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34502/0 (socket says 172.21.15.114:34502) 2024-05-27T07:38:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f5717fff700 1 -- 172.21.15.114:0/2220913364 learned_addr learned my addr 172.21.15.114:0/2220913364 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f57177fe700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5718103360 0x7f571819d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f571ca00700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57181051e0 0x7f57181a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f5717fff700 1 -- 172.21.15.114:0/2220913364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5718103360 msgr2=0x7f571819d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f5717fff700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5718103360 0x7f571819d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f5717fff700 1 -- 172.21.15.114:0/2220913364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57181051e0 msgr2=0x7f57181a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f5717fff700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57181051e0 0x7f57181a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f5717fff700 1 -- 172.21.15.114:0/2220913364 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57181a1f80 con 0x7f5718102840 2024-05-27T07:38:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f5717fff700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5718102840 0x7f571819d0b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f570800b5c0 tx=0x7f57080047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f57157fa700 1 -- 172.21.15.114:0/2220913364 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5708003680 con 0x7f5718102840 2024-05-27T07:38:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f571e463700 1 -- 172.21.15.114:0/2220913364 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57181a2210 con 0x7f5718102840 2024-05-27T07:38:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f57157fa700 1 -- 172.21.15.114:0/2220913364 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5708004ce0 con 0x7f5718102840 2024-05-27T07:38:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f57157fa700 1 -- 172.21.15.114:0/2220913364 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5708003cd0 con 0x7f5718102840 2024-05-27T07:38:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.656+0000 7f571e463700 1 -- 172.21.15.114:0/2220913364 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57181a2690 con 0x7f5718102840 2024-05-27T07:38:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.657+0000 7f571e463700 1 -- 172.21.15.114:0/2220913364 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f57181013f0 con 0x7f5718102840 2024-05-27T07:38:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.659+0000 7f57157fa700 1 -- 172.21.15.114:0/2220913364 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f5708004830 con 0x7f5718102840 2024-05-27T07:38:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.660+0000 7f57157fa700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f57000b2ab0 0x7f57000b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.660+0000 7f57177fe700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f57000b2ab0 0x7f57000b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.660+0000 7f57157fa700 1 -- 172.21.15.114:0/2220913364 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f57080e3c30 con 0x7f5718102840 2024-05-27T07:38:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.660+0000 7f57157fa700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108] conn(0x7f57000bde60 0x7f57000c0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.660+0000 7f57157fa700 1 -- 172.21.15.114:0/2220913364 --> [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5708017e50 con 0x7f57000bde60 2024-05-27T07:38:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.660+0000 7f57157fa700 1 -- 172.21.15.114:0/2220913364 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f57080e4080 con 0x7f5718102840 2024-05-27T07:38:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.661+0000 7f57177fe700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f57000b2ab0 0x7f57000b4fc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5718112cf0 tx=0x7f570c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.661+0000 7f571ca00700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108] conn(0x7f57000bde60 0x7f57000c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.661+0000 7f571ca00700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108] conn(0x7f57000bde60 0x7f57000c0350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:11.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.662+0000 7f57157fa700 1 -- 172.21.15.114:0/2220913364 <== osd.2 v2:172.21.15.186:6800/249211108 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5708017e50 con 0x7f57000bde60 2024-05-27T07:38:11.687 INFO:teuthology.orchestra.run.smithi114.stdout:300647710760 2024-05-27T07:38:11.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-27T07:38:11.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.691+0000 7f571e463700 1 -- 172.21.15.114:0/2220913364 --> [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5718103a60 con 0x7f57000bde60 2024-05-27T07:38:11.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.692+0000 7f57157fa700 1 -- 172.21.15.114:0/2220913364 <== osd.2 v2:172.21.15.186:6800/249211108 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5718103a60 con 0x7f57000bde60 2024-05-27T07:38:11.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.692+0000 7f571e463700 1 -- 172.21.15.114:0/2220913364 >> [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108] conn(0x7f57000bde60 msgr2=0x7f57000c0350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.692+0000 7f571e463700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108] conn(0x7f57000bde60 0x7f57000c0350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.692+0000 7f571e463700 1 -- 172.21.15.114:0/2220913364 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f57000b2ab0 msgr2=0x7f57000b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.692+0000 7f571e463700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f57000b2ab0 0x7f57000b4fc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5718112cf0 tx=0x7f570c0073d0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.692+0000 7f571e463700 1 -- 172.21.15.114:0/2220913364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5718102840 msgr2=0x7f571819d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.692+0000 7f571e463700 1 --2- 172.21.15.114:0/2220913364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5718102840 0x7f571819d0b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f570800b5c0 tx=0x7f57080047b0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.692+0000 7f571ca00700 1 -- 172.21.15.114:0/2220913364 reap_dead start 2024-05-27T07:38:11.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.693+0000 7f571e463700 1 -- 172.21.15.114:0/2220913364 shutdown_connections 2024-05-27T07:38:11.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.693+0000 7f571e463700 1 -- 172.21.15.114:0/2220913364 >> 172.21.15.114:0/2220913364 conn(0x7f57180fe4d0 msgr2=0x7f57181009d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:11.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.693+0000 7f571e463700 1 -- 172.21.15.114:0/2220913364 shutdown_connections 2024-05-27T07:38:11.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:11.693+0000 7f571e463700 1 -- 172.21.15.114:0/2220913364 wait complete. 2024-05-27T07:38:12.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.025+0000 7f40bb16a700 1 -- 172.21.15.114:0/1690863620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b4103120 msgr2=0x7f40b4111110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:12.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.025+0000 7f40bb16a700 1 --2- 172.21.15.114:0/1690863620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b4103120 0x7f40b4111110 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00b0a0 tx=0x7f40ac00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:38:12.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.025+0000 7f40bb16a700 1 -- 172.21.15.114:0/1690863620 shutdown_connections 2024-05-27T07:38:12.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.025+0000 7f40bb16a700 1 --2- 172.21.15.114:0/1690863620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b4103120 0x7f40b4111110 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:12.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.025+0000 7f40bb16a700 1 --2- 172.21.15.114:0/1690863620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b410a3d0 0x7f40b410a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:12.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.025+0000 7f40bb16a700 1 --2- 172.21.15.114:0/1690863620 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40b41027d0 0x7f40b4102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:12.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.025+0000 7f40bb16a700 1 -- 172.21.15.114:0/1690863620 >> 172.21.15.114:0/1690863620 conn(0x7f40b40fe6e0 msgr2=0x7f40b4100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:12.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.026+0000 7f40bb16a700 1 -- 172.21.15.114:0/1690863620 shutdown_connections 2024-05-27T07:38:12.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.026+0000 7f40bb16a700 1 -- 172.21.15.114:0/1690863620 wait complete. 2024-05-27T07:38:12.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.026+0000 7f40bb16a700 1 Processor -- start 2024-05-27T07:38:12.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.027+0000 7f40bb16a700 1 -- start start 2024-05-27T07:38:12.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.027+0000 7f40bb16a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b41027d0 0x7f40b41039b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:12.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.027+0000 7f40bb16a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b4103120 0x7f40b4103ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:12.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.027+0000 7f40bb16a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40b410a3d0 0x7f40b4104430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:12.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.027+0000 7f40bb16a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f40b4114470 con 0x7f40b4103120 2024-05-27T07:38:12.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.027+0000 7f40bb16a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f40b41142f0 con 0x7f40b410a3d0 2024-05-27T07:38:12.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.027+0000 7f40bb16a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f40b41145f0 con 0x7f40b41027d0 2024-05-27T07:38:12.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.027+0000 7f40b3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b4103120 0x7f40b4103ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:12.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.027+0000 7f40b3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b4103120 0x7f40b4103ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34520/0 (socket says 172.21.15.114:34520) 2024-05-27T07:38:12.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.027+0000 7f40b3fff700 1 -- 172.21.15.114:0/4008814213 learned_addr learned my addr 172.21.15.114:0/4008814213 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:12.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.027+0000 7f40b9707700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40b410a3d0 0x7f40b4104430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:12.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.027+0000 7f40b8f06700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b41027d0 0x7f40b41039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:12.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.028+0000 7f40b3fff700 1 -- 172.21.15.114:0/4008814213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b41027d0 msgr2=0x7f40b41039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:12.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.028+0000 7f40b3fff700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b41027d0 0x7f40b41039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:12.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.028+0000 7f40b3fff700 1 -- 172.21.15.114:0/4008814213 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40b410a3d0 msgr2=0x7f40b4104430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:12.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.028+0000 7f40b3fff700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40b410a3d0 0x7f40b4104430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:12.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.028+0000 7f40b3fff700 1 -- 172.21.15.114:0/4008814213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b41a9120 con 0x7f40b4103120 2024-05-27T07:38:12.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.028+0000 7f40b8f06700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b41027d0 0x7f40b41039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:12.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.028+0000 7f40b3fff700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b4103120 0x7f40b4103ef0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f40a000e970 tx=0x7f40a000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:12.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.028+0000 7f40b1ffb700 1 -- 172.21.15.114:0/4008814213 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a000cbd0 con 0x7f40b4103120 2024-05-27T07:38:12.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.028+0000 7f40b1ffb700 1 -- 172.21.15.114:0/4008814213 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40a000cd70 con 0x7f40b4103120 2024-05-27T07:38:12.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.028+0000 7f40bb16a700 1 -- 172.21.15.114:0/4008814213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b41a9410 con 0x7f40b4103120 2024-05-27T07:38:12.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.028+0000 7f40b1ffb700 1 -- 172.21.15.114:0/4008814213 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a0005680 con 0x7f40b4103120 2024-05-27T07:38:12.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.028+0000 7f40bb16a700 1 -- 172.21.15.114:0/4008814213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b41a9950 con 0x7f40b4103120 2024-05-27T07:38:12.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.029+0000 7f40bb16a700 1 -- 172.21.15.114:0/4008814213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f40b410bb30 con 0x7f40b4103120 2024-05-27T07:38:12.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.031+0000 7f40b1ffb700 1 -- 172.21.15.114:0/4008814213 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f40a0004750 con 0x7f40b4103120 2024-05-27T07:38:12.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.032+0000 7f40b1ffb700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f40a40b2910 0x7f40a40b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:12.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.032+0000 7f40b8f06700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f40a40b2910 0x7f40a40b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:12.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.032+0000 7f40b1ffb700 1 -- 172.21.15.114:0/4008814213 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f40a00e2fe0 con 0x7f40b4103120 2024-05-27T07:38:12.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.032+0000 7f40b1ffb700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644] conn(0x7f40a40bdcc0 0x7f40a40c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:12.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.032+0000 7f40b1ffb700 1 -- 172.21.15.114:0/4008814213 --> [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40a40c0880 con 0x7f40a40bdcc0 2024-05-27T07:38:12.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.032+0000 7f40b1ffb700 1 -- 172.21.15.114:0/4008814213 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f40a00e3430 con 0x7f40b4103120 2024-05-27T07:38:12.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.033+0000 7f40b9707700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644] conn(0x7f40a40bdcc0 0x7f40a40c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:12.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.033+0000 7f40b8f06700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f40a40b2910 0x7f40a40b4e20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f40a8005e10 tx=0x7f40a8005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:12.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.033+0000 7f40b9707700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644] conn(0x7f40a40bdcc0 0x7f40a40c01b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:12.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.034+0000 7f40b1ffb700 1 -- 172.21.15.114:0/4008814213 <== osd.6 v2:172.21.15.114:6818/2237458644 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f40a40c0880 con 0x7f40a40bdcc0 2024-05-27T07:38:12.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.051+0000 7f40bb16a700 1 -- 172.21.15.114:0/4008814213 --> [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f40b4105af0 con 0x7f40a40bdcc0 2024-05-27T07:38:12.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.051+0000 7f40b1ffb700 1 -- 172.21.15.114:0/4008814213 <== osd.6 v2:172.21.15.114:6818/2237458644 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f40b4105af0 con 0x7f40a40bdcc0 2024-05-27T07:38:12.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.051+0000 7f40bb16a700 1 -- 172.21.15.114:0/4008814213 >> [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644] conn(0x7f40a40bdcc0 msgr2=0x7f40a40c01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:12.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.051+0000 7f40bb16a700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644] conn(0x7f40a40bdcc0 0x7f40a40c01b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:12.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.051+0000 7f40bb16a700 1 -- 172.21.15.114:0/4008814213 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f40a40b2910 msgr2=0x7f40a40b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:12.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.051+0000 7f40bb16a700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f40a40b2910 0x7f40a40b4e20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f40a8005e10 tx=0x7f40a8005d60 comp rx=0 tx=0).stop 2024-05-27T07:38:12.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.051+0000 7f40bb16a700 1 -- 172.21.15.114:0/4008814213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b4103120 msgr2=0x7f40b4103ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:12.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.051+0000 7f40bb16a700 1 --2- 172.21.15.114:0/4008814213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b4103120 0x7f40b4103ef0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f40a000e970 tx=0x7f40a000ec80 comp rx=0 tx=0).stop 2024-05-27T07:38:12.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.051+0000 7f40b9707700 1 -- 172.21.15.114:0/4008814213 reap_dead start 2024-05-27T07:38:12.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.051+0000 7f40bb16a700 1 -- 172.21.15.114:0/4008814213 shutdown_connections 2024-05-27T07:38:12.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.051+0000 7f40bb16a700 1 -- 172.21.15.114:0/4008814213 >> 172.21.15.114:0/4008814213 conn(0x7f40b40fe6e0 msgr2=0x7f40b410b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:12.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.052+0000 7f40bb16a700 1 -- 172.21.15.114:0/4008814213 shutdown_connections 2024-05-27T07:38:12.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:12.052+0000 7f40bb16a700 1 -- 172.21.15.114:0/4008814213 wait complete. 2024-05-27T07:38:12.769 INFO:teuthology.orchestra.run.smithi114.stdout:77309411416 2024-05-27T07:38:12.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-27T07:38:12.986 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:12.987 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:12.989 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:12.990 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:12.993 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:13.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:12 smithi114 ceph-mon[32038]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:13.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:12 smithi163 ceph-mon[36451]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:13.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:12 smithi186 ceph-mon[36647]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:13.439 INFO:teuthology.orchestra.run.smithi114.stdout:180388626500 2024-05-27T07:38:13.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-27T07:38:15.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:14 smithi163 ceph-mon[36451]: pgmap v364: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:15.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:14 smithi186 ceph-mon[36647]: pgmap v364: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:15.294 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:14 smithi114 ceph-mon[32038]: pgmap v364: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:17.018 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:16 smithi114 ceph-mon[32038]: pgmap v365: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:17.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:16 smithi163 ceph-mon[36451]: pgmap v365: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:17.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:16 smithi186 ceph-mon[36647]: pgmap v365: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:19.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:18 smithi114 ceph-mon[32038]: pgmap v366: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:19.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.203+0000 7f67a9c58700 1 -- 172.21.15.114:0/652409589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a4107de0 msgr2=0x7f67a41081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.203+0000 7f67a9c58700 1 --2- 172.21.15.114:0/652409589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a4107de0 0x7f67a41081f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f67980099c0 tx=0x7f6798009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.203+0000 7f67a9c58700 1 -- 172.21.15.114:0/652409589 shutdown_connections 2024-05-27T07:38:19.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.203+0000 7f67a9c58700 1 --2- 172.21.15.114:0/652409589 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67a4108730 0x7f67a4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.203+0000 7f67a9c58700 1 --2- 172.21.15.114:0/652409589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67a41034b0 0x7f67a41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.203+0000 7f67a9c58700 1 --2- 172.21.15.114:0/652409589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a4107de0 0x7f67a41081f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.203+0000 7f67a9c58700 1 -- 172.21.15.114:0/652409589 >> 172.21.15.114:0/652409589 conn(0x7f67a40fedb0 msgr2=0x7f67a41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:19.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.203+0000 7f67a9c58700 1 -- 172.21.15.114:0/652409589 shutdown_connections 2024-05-27T07:38:19.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.203+0000 7f67a9c58700 1 -- 172.21.15.114:0/652409589 wait complete. 2024-05-27T07:38:19.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.204+0000 7f67a9c58700 1 Processor -- start 2024-05-27T07:38:19.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.204+0000 7f67a9c58700 1 -- start start 2024-05-27T07:38:19.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.204+0000 7f67a9c58700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67a41034b0 0x7f67a419f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.204+0000 7f67a9c58700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a4107de0 0x7f67a419fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.204+0000 7f67a9c58700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67a4108730 0x7f67a4199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.204+0000 7f67a9c58700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f67a4113340 con 0x7f67a4107de0 2024-05-27T07:38:19.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.204+0000 7f67a9c58700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f67a41131c0 con 0x7f67a4108730 2024-05-27T07:38:19.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.204+0000 7f67a9c58700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f67a41134c0 con 0x7f67a41034b0 2024-05-27T07:38:19.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a4107de0 0x7f67a419fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a4107de0 0x7f67a419fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49126/0 (socket says 172.21.15.114:49126) 2024-05-27T07:38:19.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a2ffd700 1 -- 172.21.15.114:0/880977152 learned_addr learned my addr 172.21.15.114:0/880977152 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:19.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a3fff700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67a4108730 0x7f67a4199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a37fe700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67a41034b0 0x7f67a419f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a2ffd700 1 -- 172.21.15.114:0/880977152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67a41034b0 msgr2=0x7f67a419f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a2ffd700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67a41034b0 0x7f67a419f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a2ffd700 1 -- 172.21.15.114:0/880977152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67a4108730 msgr2=0x7f67a4199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a2ffd700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67a4108730 0x7f67a4199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a2ffd700 1 -- 172.21.15.114:0/880977152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a419a270 con 0x7f67a4107de0 2024-05-27T07:38:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a3fff700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67a4108730 0x7f67a4199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a2ffd700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a4107de0 0x7f67a419fef0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6798006010 tx=0x7f67980047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a0ff9700 1 -- 172.21.15.114:0/880977152 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798027070 con 0x7f67a4107de0 2024-05-27T07:38:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a9c58700 1 -- 172.21.15.114:0/880977152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a419a500 con 0x7f67a4107de0 2024-05-27T07:38:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a0ff9700 1 -- 172.21.15.114:0/880977152 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6798003fa0 con 0x7f67a4107de0 2024-05-27T07:38:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a0ff9700 1 -- 172.21.15.114:0/880977152 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798022dd0 con 0x7f67a4107de0 2024-05-27T07:38:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.205+0000 7f67a9c58700 1 -- 172.21.15.114:0/880977152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a406b600 con 0x7f67a4107de0 2024-05-27T07:38:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.206+0000 7f67a9c58700 1 -- 172.21.15.114:0/880977152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a419a9e0 con 0x7f67a4107de0 2024-05-27T07:38:19.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.208+0000 7f67a0ff9700 1 -- 172.21.15.114:0/880977152 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f679802a070 con 0x7f67a4107de0 2024-05-27T07:38:19.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.209+0000 7f67a0ff9700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f678c0b2910 0x7f678c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.209+0000 7f67a0ff9700 1 -- 172.21.15.114:0/880977152 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f67980e38f0 con 0x7f67a4107de0 2024-05-27T07:38:19.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.209+0000 7f67a37fe700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f678c0b2910 0x7f678c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.209+0000 7f67a37fe700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f678c0b2910 0x7f678c0b4e20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6794005950 tx=0x7f67940058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:19.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.212+0000 7f67a0ff9700 1 -- 172.21.15.114:0/880977152 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6798031080 con 0x7f67a4107de0 2024-05-27T07:38:19.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:18 smithi163 ceph-mon[36451]: pgmap v366: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:19.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:18 smithi186 ceph-mon[36647]: pgmap v366: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:19.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.427+0000 7f67a9c58700 1 -- 172.21.15.114:0/880977152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f67a419afe0 con 0x7f67a4107de0 2024-05-27T07:38:19.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.428+0000 7f67a0ff9700 1 -- 172.21.15.114:0/880977152 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6798013070 con 0x7f67a4107de0 2024-05-27T07:38:19.425 INFO:teuthology.orchestra.run.smithi114.stdout:55834574942 2024-05-27T07:38:19.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.430+0000 7f67a9c58700 1 -- 172.21.15.114:0/880977152 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f678c0b2910 msgr2=0x7f678c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.430+0000 7f67a9c58700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f678c0b2910 0x7f678c0b4e20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6794005950 tx=0x7f67940058e0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.430+0000 7f67a9c58700 1 -- 172.21.15.114:0/880977152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a4107de0 msgr2=0x7f67a419fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.430+0000 7f67a9c58700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a4107de0 0x7f67a419fef0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6798006010 tx=0x7f67980047b0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.430+0000 7f67a9c58700 1 -- 172.21.15.114:0/880977152 shutdown_connections 2024-05-27T07:38:19.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.430+0000 7f67a9c58700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67a4108730 0x7f67a4199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.430+0000 7f67a9c58700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f678c0b2910 0x7f678c0b4e20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.430+0000 7f67a9c58700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67a41034b0 0x7f67a419f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.430+0000 7f67a9c58700 1 --2- 172.21.15.114:0/880977152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a4107de0 0x7f67a419fef0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.430+0000 7f67a9c58700 1 -- 172.21.15.114:0/880977152 >> 172.21.15.114:0/880977152 conn(0x7f67a40fedb0 msgr2=0x7f67a4109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:19.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.430+0000 7f67a9c58700 1 -- 172.21.15.114:0/880977152 shutdown_connections 2024-05-27T07:38:19.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:19.430+0000 7f67a9c58700 1 -- 172.21.15.114:0/880977152 wait complete. 2024-05-27T07:38:20.081 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:19 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/880977152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T07:38:20.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:19 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/880977152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T07:38:20.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:19 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/880977152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T07:38:21.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:20 smithi163 ceph-mon[36451]: pgmap v367: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:21.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:20 smithi186 ceph-mon[36647]: pgmap v367: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:21.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:20 smithi114 ceph-mon[32038]: pgmap v367: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:21.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.570+0000 7fa189d94700 1 -- 172.21.15.114:0/3550439729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa184109840 msgr2=0x7fa1841111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:21.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.570+0000 7fa189d94700 1 --2- 172.21.15.114:0/3550439729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa184109840 0x7fa1841111a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00b0d0 tx=0x7fa17c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:38:21.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.570+0000 7fa189d94700 1 -- 172.21.15.114:0/3550439729 shutdown_connections 2024-05-27T07:38:21.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.570+0000 7fa189d94700 1 --2- 172.21.15.114:0/3550439729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa184109840 0x7fa1841111a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:21.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.570+0000 7fa189d94700 1 --2- 172.21.15.114:0/3550439729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa184102b40 0x7fa184102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:21.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.570+0000 7fa189d94700 1 --2- 172.21.15.114:0/3550439729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa184108ef0 0x7fa184109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:21.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.570+0000 7fa189d94700 1 -- 172.21.15.114:0/3550439729 >> 172.21.15.114:0/3550439729 conn(0x7fa1840fe6e0 msgr2=0x7fa184100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:21.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.570+0000 7fa189d94700 1 -- 172.21.15.114:0/3550439729 shutdown_connections 2024-05-27T07:38:21.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.570+0000 7fa189d94700 1 -- 172.21.15.114:0/3550439729 wait complete. 2024-05-27T07:38:21.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.571+0000 7fa189d94700 1 Processor -- start 2024-05-27T07:38:21.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.571+0000 7fa189d94700 1 -- start start 2024-05-27T07:38:21.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.571+0000 7fa189d94700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa184102b40 0x7fa18419ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:21.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.571+0000 7fa189d94700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa184108ef0 0x7fa18419b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:21.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.571+0000 7fa189d94700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa184109840 0x7fa18419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:21.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.571+0000 7fa189d94700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa184114470 con 0x7fa184102b40 2024-05-27T07:38:21.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.571+0000 7fa189d94700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa1841142f0 con 0x7fa184109840 2024-05-27T07:38:21.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa189d94700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa1841145f0 con 0x7fa184108ef0 2024-05-27T07:38:21.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa1837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa184102b40 0x7fa18419ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:21.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa1837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa184102b40 0x7fa18419ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49140/0 (socket says 172.21.15.114:49140) 2024-05-27T07:38:21.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa1837fe700 1 -- 172.21.15.114:0/3836730685 learned_addr learned my addr 172.21.15.114:0/3836730685 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:21.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa17bfff700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa184108ef0 0x7fa18419b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:21.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa183fff700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa184109840 0x7fa18419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:21.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa1837fe700 1 -- 172.21.15.114:0/3836730685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa184108ef0 msgr2=0x7fa18419b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:21.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa1837fe700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa184108ef0 0x7fa18419b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:21.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa1837fe700 1 -- 172.21.15.114:0/3836730685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa184109840 msgr2=0x7fa18419f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:21.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa1837fe700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa184109840 0x7fa18419f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:21.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa1837fe700 1 -- 172.21.15.114:0/3836730685 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa18419fbc0 con 0x7fa184102b40 2024-05-27T07:38:21.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa183fff700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa184109840 0x7fa18419f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:21.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa1837fe700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa184102b40 0x7fa18419ada0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa17400c9b0 tx=0x7fa17400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:21.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa1817fa700 1 -- 172.21.15.114:0/3836730685 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa174007b00 con 0x7fa184102b40 2024-05-27T07:38:21.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa1817fa700 1 -- 172.21.15.114:0/3836730685 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa174007ca0 con 0x7fa184102b40 2024-05-27T07:38:21.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa1817fa700 1 -- 172.21.15.114:0/3836730685 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa174018730 con 0x7fa184102b40 2024-05-27T07:38:21.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa189d94700 1 -- 172.21.15.114:0/3836730685 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa18419feb0 con 0x7fa184102b40 2024-05-27T07:38:21.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.572+0000 7fa189d94700 1 -- 172.21.15.114:0/3836730685 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1841a03f0 con 0x7fa184102b40 2024-05-27T07:38:21.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.574+0000 7fa189d94700 1 -- 172.21.15.114:0/3836730685 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1840512b0 con 0x7fa184102b40 2024-05-27T07:38:21.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.575+0000 7fa1817fa700 1 -- 172.21.15.114:0/3836730685 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fa17400f450 con 0x7fa184102b40 2024-05-27T07:38:21.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.575+0000 7fa1817fa700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa1700b2910 0x7fa1700b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:21.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.576+0000 7fa17bfff700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa1700b2910 0x7fa1700b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:21.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.576+0000 7fa1817fa700 1 -- 172.21.15.114:0/3836730685 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fa1740e27f0 con 0x7fa184102b40 2024-05-27T07:38:21.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.576+0000 7fa17bfff700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa1700b2910 0x7fa1700b4e20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa16c005fd0 tx=0x7fa16c00af20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:21.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.580+0000 7fa1817fa700 1 -- 172.21.15.114:0/3836730685 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa174014030 con 0x7fa184102b40 2024-05-27T07:38:21.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.795+0000 7fa189d94700 1 -- 172.21.15.114:0/3836730685 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fa18419c130 con 0x7fa184102b40 2024-05-27T07:38:21.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.795+0000 7fa1817fa700 1 -- 172.21.15.114:0/3836730685 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fa18419c130 con 0x7fa184102b40 2024-05-27T07:38:21.792 INFO:teuthology.orchestra.run.smithi114.stdout:274877906991 2024-05-27T07:38:21.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.798+0000 7fa189d94700 1 -- 172.21.15.114:0/3836730685 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa1700b2910 msgr2=0x7fa1700b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:21.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.798+0000 7fa189d94700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa1700b2910 0x7fa1700b4e20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa16c005fd0 tx=0x7fa16c00af20 comp rx=0 tx=0).stop 2024-05-27T07:38:21.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.798+0000 7fa189d94700 1 -- 172.21.15.114:0/3836730685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa184102b40 msgr2=0x7fa18419ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:21.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.798+0000 7fa189d94700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa184102b40 0x7fa18419ada0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa17400c9b0 tx=0x7fa17400cd70 comp rx=0 tx=0).stop 2024-05-27T07:38:21.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.798+0000 7fa189d94700 1 -- 172.21.15.114:0/3836730685 shutdown_connections 2024-05-27T07:38:21.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.798+0000 7fa189d94700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa184109840 0x7fa18419f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:21.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.798+0000 7fa189d94700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa184102b40 0x7fa18419ada0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:21.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.798+0000 7fa189d94700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fa1700b2910 0x7fa1700b4e20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:21.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.798+0000 7fa189d94700 1 --2- 172.21.15.114:0/3836730685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa184108ef0 0x7fa18419b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:21.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.798+0000 7fa189d94700 1 -- 172.21.15.114:0/3836730685 >> 172.21.15.114:0/3836730685 conn(0x7fa1840fe6e0 msgr2=0x7fa184108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:21.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.798+0000 7fa189d94700 1 -- 172.21.15.114:0/3836730685 shutdown_connections 2024-05-27T07:38:21.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:21.798+0000 7fa189d94700 1 -- 172.21.15.114:0/3836730685 wait complete. 2024-05-27T07:38:22.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:21 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3836730685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T07:38:22.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:21 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3836730685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T07:38:22.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:21 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3836730685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T07:38:22.996 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574939 got 55834574942 for osd.1 2024-05-27T07:38:22.996 DEBUG:teuthology.parallel:result is None 2024-05-27T07:38:23.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:22 smithi163 ceph-mon[36451]: pgmap v368: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:23.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:22 smithi186 ceph-mon[36647]: pgmap v368: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:23.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:22 smithi114 ceph-mon[32038]: pgmap v368: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:24.523 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:24.523 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:25.130 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:24 smithi114 ceph-mon[32038]: pgmap v369: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:25.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:24 smithi163 ceph-mon[36451]: pgmap v369: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:25.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:24 smithi186 ceph-mon[36647]: pgmap v369: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:25.479 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877906986 got 274877906991 for osd.10 2024-05-27T07:38:25.480 DEBUG:teuthology.parallel:result is None 2024-05-27T07:38:26.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.994+0000 7f22d417f700 1 -- 172.21.15.114:0/3797509238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc103930 msgr2=0x7f22cc103d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:26.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.994+0000 7f22d417f700 1 --2- 172.21.15.114:0/3797509238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc103930 0x7f22cc103d40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009990 tx=0x7f22c8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:38:26.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.994+0000 7f22d417f700 1 -- 172.21.15.114:0/3797509238 shutdown_connections 2024-05-27T07:38:26.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.994+0000 7f22d417f700 1 --2- 172.21.15.114:0/3797509238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22cc104280 0x7f22cc1100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:26.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.994+0000 7f22d417f700 1 --2- 172.21.15.114:0/3797509238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc109350 0x7f22cc109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:26.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.994+0000 7f22d417f700 1 --2- 172.21.15.114:0/3797509238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc103930 0x7f22cc103d40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:26.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.994+0000 7f22d417f700 1 -- 172.21.15.114:0/3797509238 >> 172.21.15.114:0/3797509238 conn(0x7f22cc0fe510 msgr2=0x7f22cc100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:26.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.994+0000 7f22d417f700 1 -- 172.21.15.114:0/3797509238 shutdown_connections 2024-05-27T07:38:26.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.994+0000 7f22d417f700 1 -- 172.21.15.114:0/3797509238 wait complete. 2024-05-27T07:38:26.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d417f700 1 Processor -- start 2024-05-27T07:38:26.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d417f700 1 -- start start 2024-05-27T07:38:26.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d417f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22cc103930 0x7f22cc19be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:26.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d417f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc104280 0x7f22cc19c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d417f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc109350 0x7f22cc19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d417f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc113500 con 0x7f22cc104280 2024-05-27T07:38:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d417f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc113380 con 0x7f22cc103930 2024-05-27T07:38:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d417f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc113680 con 0x7f22cc109350 2024-05-27T07:38:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d171a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc104280 0x7f22cc19c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d171a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc104280 0x7f22cc19c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55310/0 (socket says 172.21.15.114:55310) 2024-05-27T07:38:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d171a700 1 -- 172.21.15.114:0/2636938043 learned_addr learned my addr 172.21.15.114:0/2636938043 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d1f1b700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22cc103930 0x7f22cc19be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d271c700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc109350 0x7f22cc19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d171a700 1 -- 172.21.15.114:0/2636938043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc109350 msgr2=0x7f22cc19c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:26.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d171a700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc109350 0x7f22cc19c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:26.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d171a700 1 -- 172.21.15.114:0/2636938043 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22cc103930 msgr2=0x7f22cc19be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:26.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d171a700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22cc103930 0x7f22cc19be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:26.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d171a700 1 -- 172.21.15.114:0/2636938043 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22cc1a7190 con 0x7f22cc104280 2024-05-27T07:38:26.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.995+0000 7f22d271c700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc109350 0x7f22cc19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:26.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.996+0000 7f22d171a700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc104280 0x7f22cc19c360 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f22c8000c00 tx=0x7f22c80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:26.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.996+0000 7f22beffd700 1 -- 172.21.15.114:0/2636938043 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c8004830 con 0x7f22cc104280 2024-05-27T07:38:26.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.996+0000 7f22beffd700 1 -- 172.21.15.114:0/2636938043 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22c8005020 con 0x7f22cc104280 2024-05-27T07:38:26.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.996+0000 7f22beffd700 1 -- 172.21.15.114:0/2636938043 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c8003d40 con 0x7f22cc104280 2024-05-27T07:38:26.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.996+0000 7f22d417f700 1 -- 172.21.15.114:0/2636938043 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22cc1a7420 con 0x7f22cc104280 2024-05-27T07:38:26.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.996+0000 7f22d417f700 1 -- 172.21.15.114:0/2636938043 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22cc1a7870 con 0x7f22cc104280 2024-05-27T07:38:26.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.997+0000 7f22d417f700 1 -- 172.21.15.114:0/2636938043 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22cc10b790 con 0x7f22cc104280 2024-05-27T07:38:26.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.998+0000 7f22beffd700 1 -- 172.21.15.114:0/2636938043 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f22c80051d0 con 0x7f22cc104280 2024-05-27T07:38:26.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.999+0000 7f22beffd700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f22b80b2990 0x7f22b80b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:26.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.999+0000 7f22d1f1b700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f22b80b2990 0x7f22b80b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:26.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.999+0000 7f22beffd700 1 -- 172.21.15.114:0/2636938043 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f22c80e39c0 con 0x7f22cc104280 2024-05-27T07:38:26.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.000+0000 7f22d1f1b700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f22b80b2990 0x7f22b80b4ea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f22cc06bd00 tx=0x7f22c0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:26.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.003+0000 7f22beffd700 1 -- 172.21.15.114:0/2636938043 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f22c8013070 con 0x7f22cc104280 2024-05-27T07:38:27.205 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:26 smithi114 ceph-mon[32038]: pgmap v370: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:27.205 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:26 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/685659708' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T07:38:27.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.214+0000 7f22d417f700 1 -- 172.21.15.114:0/2636938043 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f22cc102e30 con 0x7f22cc104280 2024-05-27T07:38:27.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.214+0000 7f22beffd700 1 -- 172.21.15.114:0/2636938043 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f22c80ab240 con 0x7f22cc104280 2024-05-27T07:38:27.211 INFO:teuthology.orchestra.run.smithi114.stdout:201863462976 2024-05-27T07:38:27.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.217+0000 7f22d417f700 1 -- 172.21.15.114:0/2636938043 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f22b80b2990 msgr2=0x7f22b80b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:27.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.217+0000 7f22d417f700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f22b80b2990 0x7f22b80b4ea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f22cc06bd00 tx=0x7f22c0009190 comp rx=0 tx=0).stop 2024-05-27T07:38:27.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.217+0000 7f22d417f700 1 -- 172.21.15.114:0/2636938043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc104280 msgr2=0x7f22cc19c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:27.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.217+0000 7f22d417f700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc104280 0x7f22cc19c360 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f22c8000c00 tx=0x7f22c80047b0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.217+0000 7f22d417f700 1 -- 172.21.15.114:0/2636938043 shutdown_connections 2024-05-27T07:38:27.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.217+0000 7f22d417f700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f22b80b2990 0x7f22b80b4ea0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.217+0000 7f22d417f700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc109350 0x7f22cc19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.217+0000 7f22d417f700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22cc103930 0x7f22cc19be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.217+0000 7f22d417f700 1 --2- 172.21.15.114:0/2636938043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc104280 0x7f22cc19c360 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.217+0000 7f22d417f700 1 -- 172.21.15.114:0/2636938043 >> 172.21.15.114:0/2636938043 conn(0x7f22cc0fe510 msgr2=0x7f22cc100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:27.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.217+0000 7f22d417f700 1 -- 172.21.15.114:0/2636938043 shutdown_connections 2024-05-27T07:38:27.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.217+0000 7f22d417f700 1 -- 172.21.15.114:0/2636938043 wait complete. 2024-05-27T07:38:27.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:26 smithi163 ceph-mon[36451]: pgmap v370: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:27.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:26 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/685659708' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T07:38:27.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:26 smithi186 ceph-mon[36647]: pgmap v370: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:27.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:26 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/685659708' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T07:38:27.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.454+0000 7fc779c57700 1 -- 172.21.15.114:0/855301894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc774108750 msgr2=0x7fc7741100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:27.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.454+0000 7fc779c57700 1 --2- 172.21.15.114:0/855301894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc774108750 0x7fc7741100b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc76c00b0d0 tx=0x7fc76c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.454+0000 7fc779c57700 1 -- 172.21.15.114:0/855301894 shutdown_connections 2024-05-27T07:38:27.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.454+0000 7fc779c57700 1 --2- 172.21.15.114:0/855301894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc774108750 0x7fc7741100b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.454+0000 7fc779c57700 1 --2- 172.21.15.114:0/855301894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7741034d0 0x7fc7741038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.454+0000 7fc779c57700 1 --2- 172.21.15.114:0/855301894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc774107e00 0x7fc774108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.454+0000 7fc779c57700 1 -- 172.21.15.114:0/855301894 >> 172.21.15.114:0/855301894 conn(0x7fc7740fef40 msgr2=0x7fc774101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:27.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.454+0000 7fc779c57700 1 -- 172.21.15.114:0/855301894 shutdown_connections 2024-05-27T07:38:27.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.454+0000 7fc779c57700 1 -- 172.21.15.114:0/855301894 wait complete. 2024-05-27T07:38:27.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.455+0000 7fc779c57700 1 Processor -- start 2024-05-27T07:38:27.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.455+0000 7fc779c57700 1 -- start start 2024-05-27T07:38:27.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.455+0000 7fc779c57700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7741034d0 0x7fc7740ff420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:27.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.455+0000 7fc779c57700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc774107e00 0x7fc7740ff960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:27.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.455+0000 7fc779c57700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc774108750 0x7fc7740ffea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:27.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.455+0000 7fc779c57700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc774113380 con 0x7fc774107e00 2024-05-27T07:38:27.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.455+0000 7fc779c57700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc774113200 con 0x7fc7741034d0 2024-05-27T07:38:27.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.455+0000 7fc779c57700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc774113500 con 0x7fc774108750 2024-05-27T07:38:27.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.455+0000 7fc76bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc774107e00 0x7fc7740ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:27.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.455+0000 7fc76bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc774107e00 0x7fc7740ff960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55318/0 (socket says 172.21.15.114:55318) 2024-05-27T07:38:27.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.455+0000 7fc76bfff700 1 -- 172.21.15.114:0/1634309509 learned_addr learned my addr 172.21.15.114:0/1634309509 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:27.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.455+0000 7fc773fff700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc774108750 0x7fc7740ffea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:27.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.456+0000 7fc7737fe700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7741034d0 0x7fc7740ff420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:27.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.456+0000 7fc76bfff700 1 -- 172.21.15.114:0/1634309509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc774108750 msgr2=0x7fc7740ffea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:27.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.456+0000 7fc76bfff700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc774108750 0x7fc7740ffea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.456+0000 7fc76bfff700 1 -- 172.21.15.114:0/1634309509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7741034d0 msgr2=0x7fc7740ff420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:27.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.456+0000 7fc76bfff700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7741034d0 0x7fc7740ff420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.456+0000 7fc76bfff700 1 -- 172.21.15.114:0/1634309509 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7741a9070 con 0x7fc774107e00 2024-05-27T07:38:27.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.456+0000 7fc7737fe700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7741034d0 0x7fc7740ff420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:27.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.456+0000 7fc76bfff700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc774107e00 0x7fc7740ff960 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc75c00c9e0 tx=0x7fc75c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:27.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.456+0000 7fc7717fa700 1 -- 172.21.15.114:0/1634309509 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c007aa0 con 0x7fc774107e00 2024-05-27T07:38:27.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.456+0000 7fc7717fa700 1 -- 172.21.15.114:0/1634309509 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc75c007c40 con 0x7fc774107e00 2024-05-27T07:38:27.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.456+0000 7fc779c57700 1 -- 172.21.15.114:0/1634309509 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7741a9360 con 0x7fc774107e00 2024-05-27T07:38:27.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.456+0000 7fc7717fa700 1 -- 172.21.15.114:0/1634309509 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c018660 con 0x7fc774107e00 2024-05-27T07:38:27.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.456+0000 7fc779c57700 1 -- 172.21.15.114:0/1634309509 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7741a98a0 con 0x7fc774107e00 2024-05-27T07:38:27.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.457+0000 7fc779c57700 1 -- 172.21.15.114:0/1634309509 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7740512b0 con 0x7fc774107e00 2024-05-27T07:38:27.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.459+0000 7fc7717fa700 1 -- 172.21.15.114:0/1634309509 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fc75c018800 con 0x7fc774107e00 2024-05-27T07:38:27.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.460+0000 7fc7717fa700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc7600b2910 0x7fc7600b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:27.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.460+0000 7fc7737fe700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc7600b2910 0x7fc7600b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:27.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.460+0000 7fc7717fa700 1 -- 172.21.15.114:0/1634309509 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fc75c0e2940 con 0x7fc774107e00 2024-05-27T07:38:27.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.460+0000 7fc7737fe700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc7600b2910 0x7fc7600b4e20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc764005950 tx=0x7fc7640058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:27.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.463+0000 7fc7717fa700 1 -- 172.21.15.114:0/1634309509 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc75c014030 con 0x7fc774107e00 2024-05-27T07:38:27.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.674+0000 7fc779c57700 1 -- 172.21.15.114:0/1634309509 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fc77402e5e0 con 0x7fc774107e00 2024-05-27T07:38:27.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.675+0000 7fc7717fa700 1 -- 172.21.15.114:0/1634309509 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc75c0aa1c0 con 0x7fc774107e00 2024-05-27T07:38:27.672 INFO:teuthology.orchestra.run.smithi114.stdout:133143986257 2024-05-27T07:38:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.677+0000 7fc779c57700 1 -- 172.21.15.114:0/1634309509 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc7600b2910 msgr2=0x7fc7600b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.677+0000 7fc779c57700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc7600b2910 0x7fc7600b4e20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc764005950 tx=0x7fc7640058e0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.677+0000 7fc779c57700 1 -- 172.21.15.114:0/1634309509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc774107e00 msgr2=0x7fc7740ff960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.677+0000 7fc779c57700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc774107e00 0x7fc7740ff960 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc75c00c9e0 tx=0x7fc75c00ccf0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.677+0000 7fc779c57700 1 -- 172.21.15.114:0/1634309509 shutdown_connections 2024-05-27T07:38:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.677+0000 7fc779c57700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc7600b2910 0x7fc7600b4e20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.677+0000 7fc779c57700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc774108750 0x7fc7740ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.677+0000 7fc779c57700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7741034d0 0x7fc7740ff420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.677+0000 7fc779c57700 1 --2- 172.21.15.114:0/1634309509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc774107e00 0x7fc7740ff960 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.677+0000 7fc779c57700 1 -- 172.21.15.114:0/1634309509 >> 172.21.15.114:0/1634309509 conn(0x7fc7740fef40 msgr2=0x7fc774075c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.677+0000 7fc779c57700 1 -- 172.21.15.114:0/1634309509 shutdown_connections 2024-05-27T07:38:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.677+0000 7fc779c57700 1 -- 172.21.15.114:0/1634309509 wait complete. 2024-05-27T07:38:28.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:27 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2636938043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T07:38:28.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:27 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1634309509' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T07:38:28.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:27 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2636938043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T07:38:28.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:27 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1634309509' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T07:38:28.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:27 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2636938043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T07:38:28.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:27 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1634309509' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T07:38:28.417 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986251 got 133143986257 for osd.4 2024-05-27T07:38:28.418 DEBUG:teuthology.parallel:result is None 2024-05-27T07:38:28.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.845+0000 7fc85e6e9700 1 -- 172.21.15.114:0/3298692025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc858107de0 msgr2=0x7fc8581081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:28.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.845+0000 7fc85e6e9700 1 --2- 172.21.15.114:0/3298692025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc858107de0 0x7fc8581081f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0099c0 tx=0x7fc84c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:28.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.845+0000 7fc85e6e9700 1 -- 172.21.15.114:0/3298692025 shutdown_connections 2024-05-27T07:38:28.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.845+0000 7fc85e6e9700 1 --2- 172.21.15.114:0/3298692025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc858108730 0x7fc858110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:28.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.845+0000 7fc85e6e9700 1 --2- 172.21.15.114:0/3298692025 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8581034b0 0x7fc8581038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:28.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.845+0000 7fc85e6e9700 1 --2- 172.21.15.114:0/3298692025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc858107de0 0x7fc8581081f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:28.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.845+0000 7fc85e6e9700 1 -- 172.21.15.114:0/3298692025 >> 172.21.15.114:0/3298692025 conn(0x7fc8580fedb0 msgr2=0x7fc8581011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:28.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.845+0000 7fc85e6e9700 1 -- 172.21.15.114:0/3298692025 shutdown_connections 2024-05-27T07:38:28.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.845+0000 7fc85e6e9700 1 -- 172.21.15.114:0/3298692025 wait complete. 2024-05-27T07:38:28.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.846+0000 7fc85e6e9700 1 Processor -- start 2024-05-27T07:38:28.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.846+0000 7fc85e6e9700 1 -- start start 2024-05-27T07:38:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.846+0000 7fc85e6e9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8581034b0 0x7fc8580ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.846+0000 7fc85e6e9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc858107de0 0x7fc8580ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.846+0000 7fc85e6e9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc858108730 0x7fc8580ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.846+0000 7fc85e6e9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc858113340 con 0x7fc858108730 2024-05-27T07:38:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.846+0000 7fc85e6e9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc8581131c0 con 0x7fc8581034b0 2024-05-27T07:38:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.846+0000 7fc85e6e9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc8581134c0 con 0x7fc858107de0 2024-05-27T07:38:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc85cc86700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc858108730 0x7fc8580ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc85cc86700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc858108730 0x7fc8580ffd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55334/0 (socket says 172.21.15.114:55334) 2024-05-27T07:38:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc85cc86700 1 -- 172.21.15.114:0/751383738 learned_addr learned my addr 172.21.15.114:0/751383738 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc8577fe700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc858107de0 0x7fc8580ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc857fff700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8581034b0 0x7fc8580ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc85cc86700 1 -- 172.21.15.114:0/751383738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc858107de0 msgr2=0x7fc8580ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc85cc86700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc858107de0 0x7fc8580ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc85cc86700 1 -- 172.21.15.114:0/751383738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8581034b0 msgr2=0x7fc8580ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc85cc86700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8581034b0 0x7fc8580ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc85cc86700 1 -- 172.21.15.114:0/751383738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8581a9180 con 0x7fc858108730 2024-05-27T07:38:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc857fff700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8581034b0 0x7fc8580ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc85cc86700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc858108730 0x7fc8580ffd10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc85000ef30 tx=0x7fc85000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc8557fa700 1 -- 172.21.15.114:0/751383738 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8500092e0 con 0x7fc858108730 2024-05-27T07:38:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc85e6e9700 1 -- 172.21.15.114:0/751383738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8581a9410 con 0x7fc858108730 2024-05-27T07:38:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc8557fa700 1 -- 172.21.15.114:0/751383738 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc850004bb0 con 0x7fc858108730 2024-05-27T07:38:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc8557fa700 1 -- 172.21.15.114:0/751383738 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc850004280 con 0x7fc858108730 2024-05-27T07:38:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.847+0000 7fc85e6e9700 1 -- 172.21.15.114:0/751383738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8581a9920 con 0x7fc858108730 2024-05-27T07:38:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.848+0000 7fc85e6e9700 1 -- 172.21.15.114:0/751383738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8580512b0 con 0x7fc858108730 2024-05-27T07:38:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.850+0000 7fc8557fa700 1 -- 172.21.15.114:0/751383738 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fc85001b030 con 0x7fc858108730 2024-05-27T07:38:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.850+0000 7fc8557fa700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc8400b2910 0x7fc8400b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.851+0000 7fc857fff700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc8400b2910 0x7fc8400b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.851+0000 7fc8557fa700 1 -- 172.21.15.114:0/751383738 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fc8500e2c30 con 0x7fc858108730 2024-05-27T07:38:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.851+0000 7fc857fff700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc8400b2910 0x7fc8400b4e20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc848005fd0 tx=0x7fc848005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:27.854+0000 7fc8557fa700 1 -- 172.21.15.114:0/751383738 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc850022020 con 0x7fc858108730 2024-05-27T07:38:28.556 INFO:teuthology.orchestra.run.smithi114.stdout:253403070519 2024-05-27T07:38:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.071+0000 7fc85e6e9700 1 -- 172.21.15.114:0/751383738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fc85802e700 con 0x7fc858108730 2024-05-27T07:38:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.071+0000 7fc8557fa700 1 -- 172.21.15.114:0/751383738 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc850010040 con 0x7fc858108730 2024-05-27T07:38:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.073+0000 7fc85e6e9700 1 -- 172.21.15.114:0/751383738 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc8400b2910 msgr2=0x7fc8400b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.073+0000 7fc85e6e9700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc8400b2910 0x7fc8400b4e20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc848005fd0 tx=0x7fc848005ee0 comp rx=0 tx=0).stop 2024-05-27T07:38:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.073+0000 7fc85e6e9700 1 -- 172.21.15.114:0/751383738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc858108730 msgr2=0x7fc8580ffd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.073+0000 7fc85e6e9700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc858108730 0x7fc8580ffd10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc85000ef30 tx=0x7fc85000c3a0 comp rx=0 tx=0).stop 2024-05-27T07:38:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.073+0000 7fc85e6e9700 1 -- 172.21.15.114:0/751383738 shutdown_connections 2024-05-27T07:38:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.073+0000 7fc85e6e9700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc858108730 0x7fc8580ffd10 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.073+0000 7fc85e6e9700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8581034b0 0x7fc8580ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.073+0000 7fc85e6e9700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc8400b2910 0x7fc8400b4e20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.073+0000 7fc85e6e9700 1 --2- 172.21.15.114:0/751383738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc858107de0 0x7fc8580ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.073+0000 7fc85e6e9700 1 -- 172.21.15.114:0/751383738 >> 172.21.15.114:0/751383738 conn(0x7fc8580fedb0 msgr2=0x7fc85806b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.073+0000 7fc85e6e9700 1 -- 172.21.15.114:0/751383738 shutdown_connections 2024-05-27T07:38:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:28.073+0000 7fc85e6e9700 1 -- 172.21.15.114:0/751383738 wait complete. 2024-05-27T07:38:29.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:28 smithi114 ceph-mon[32038]: pgmap v371: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:29.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:28 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/751383738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T07:38:29.200 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070514 got 253403070519 for osd.9 2024-05-27T07:38:29.200 DEBUG:teuthology.parallel:result is None 2024-05-27T07:38:29.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:28 smithi163 ceph-mon[36451]: pgmap v371: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:29.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:28 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/751383738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T07:38:29.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:28 smithi186 ceph-mon[36647]: pgmap v371: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:29.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:28 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/751383738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T07:38:30.191 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462971 got 201863462976 for osd.7 2024-05-27T07:38:30.192 DEBUG:teuthology.parallel:result is None 2024-05-27T07:38:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.460+0000 7f7dd88de700 1 -- 172.21.15.114:0/2016484870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd41031c0 msgr2=0x7f7dd4111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.460+0000 7f7dd88de700 1 --2- 172.21.15.114:0/2016484870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd41031c0 0x7f7dd4111160 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00b0a0 tx=0x7f7dcc00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:38:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.460+0000 7f7dd88de700 1 -- 172.21.15.114:0/2016484870 shutdown_connections 2024-05-27T07:38:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.460+0000 7f7dd88de700 1 --2- 172.21.15.114:0/2016484870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd41031c0 0x7f7dd4111160 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.460+0000 7f7dd88de700 1 --2- 172.21.15.114:0/2016484870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dd41092f0 0x7f7dd4109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.460+0000 7f7dd88de700 1 --2- 172.21.15.114:0/2016484870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd4102870 0x7f7dd4102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.460+0000 7f7dd88de700 1 -- 172.21.15.114:0/2016484870 >> 172.21.15.114:0/2016484870 conn(0x7f7dd40fe6e0 msgr2=0x7f7dd4100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.460+0000 7f7dd88de700 1 -- 172.21.15.114:0/2016484870 shutdown_connections 2024-05-27T07:38:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.460+0000 7f7dd88de700 1 -- 172.21.15.114:0/2016484870 wait complete. 2024-05-27T07:38:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.461+0000 7f7dd88de700 1 Processor -- start 2024-05-27T07:38:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.461+0000 7f7dd88de700 1 -- start start 2024-05-27T07:38:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.461+0000 7f7dd88de700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd4102870 0x7f7dd419aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.461+0000 7f7dd88de700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dd41031c0 0x7f7dd419b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.461+0000 7f7dd88de700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd41092f0 0x7f7dd419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.461+0000 7f7dd88de700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd4114600 con 0x7f7dd41092f0 2024-05-27T07:38:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.461+0000 7f7dd88de700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd4114480 con 0x7f7dd41031c0 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.461+0000 7f7dd88de700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd4114780 con 0x7f7dd4102870 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.461+0000 7f7dd2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd41092f0 0x7f7dd419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dd2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd41092f0 0x7f7dd419f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55290/0 (socket says 172.21.15.114:55290) 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dd2d9d700 1 -- 172.21.15.114:0/685659708 learned_addr learned my addr 172.21.15.114:0/685659708 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dd1d9b700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dd41031c0 0x7f7dd419b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dd259c700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd4102870 0x7f7dd419aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dd2d9d700 1 -- 172.21.15.114:0/685659708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd4102870 msgr2=0x7f7dd419aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dd2d9d700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd4102870 0x7f7dd419aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dd2d9d700 1 -- 172.21.15.114:0/685659708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dd41031c0 msgr2=0x7f7dd419b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dd2d9d700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dd41031c0 0x7f7dd419b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dd2d9d700 1 -- 172.21.15.114:0/685659708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd419fd80 con 0x7f7dd41092f0 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dd259c700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd4102870 0x7f7dd419aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dd2d9d700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd41092f0 0x7f7dd419f6e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc0062a0 tx=0x7f7dcc004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dc37fe700 1 -- 172.21.15.114:0/685659708 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc021070 con 0x7f7dd41092f0 2024-05-27T07:38:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dd88de700 1 -- 172.21.15.114:0/685659708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dd41a0010 con 0x7f7dd41092f0 2024-05-27T07:38:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dc37fe700 1 -- 172.21.15.114:0/685659708 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dcc007e30 con 0x7f7dd41092f0 2024-05-27T07:38:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dc37fe700 1 -- 172.21.15.114:0/685659708 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc004780 con 0x7f7dd41092f0 2024-05-27T07:38:30.284 INFO:teuthology.orchestra.run.smithi114.stdout:154618822731 2024-05-27T07:38:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.462+0000 7f7dd88de700 1 -- 172.21.15.114:0/685659708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dd41a7990 con 0x7f7dd41092f0 2024-05-27T07:38:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.463+0000 7f7dd88de700 1 -- 172.21.15.114:0/685659708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dd40512b0 con 0x7f7dd41092f0 2024-05-27T07:38:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.465+0000 7f7dc37fe700 1 -- 172.21.15.114:0/685659708 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f7dcc007980 con 0x7f7dd41092f0 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.466+0000 7f7dc37fe700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7dbc0b2910 0x7f7dbc0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.466+0000 7f7dd259c700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7dbc0b2910 0x7f7dbc0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.466+0000 7f7dc37fe700 1 -- 172.21.15.114:0/685659708 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f7dcc00f020 con 0x7f7dd41092f0 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.466+0000 7f7dd259c700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7dbc0b2910 0x7f7dbc0b4e20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4076270 tx=0x7f7dc4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.473+0000 7f7dc37fe700 1 -- 172.21.15.114:0/685659708 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7dcc02a020 con 0x7f7dd41092f0 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.684+0000 7f7dd88de700 1 -- 172.21.15.114:0/685659708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f7dd419c210 con 0x7f7dd41092f0 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.685+0000 7f7dc37fe700 1 -- 172.21.15.114:0/685659708 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7dcc01a3c0 con 0x7f7dd41092f0 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.687+0000 7f7dd88de700 1 -- 172.21.15.114:0/685659708 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7dbc0b2910 msgr2=0x7f7dbc0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.687+0000 7f7dd88de700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7dbc0b2910 0x7f7dbc0b4e20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4076270 tx=0x7f7dc4009260 comp rx=0 tx=0).stop 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.687+0000 7f7dd88de700 1 -- 172.21.15.114:0/685659708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd41092f0 msgr2=0x7f7dd419f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.687+0000 7f7dd88de700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd41092f0 0x7f7dd419f6e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc0062a0 tx=0x7f7dcc004c80 comp rx=0 tx=0).stop 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.687+0000 7f7dd88de700 1 -- 172.21.15.114:0/685659708 shutdown_connections 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.687+0000 7f7dd88de700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd41092f0 0x7f7dd419f6e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.687+0000 7f7dd88de700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f7dbc0b2910 0x7f7dbc0b4e20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:30.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.687+0000 7f7dd88de700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd4102870 0x7f7dd419aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:30.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.687+0000 7f7dd88de700 1 --2- 172.21.15.114:0/685659708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dd41031c0 0x7f7dd419b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:30.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.687+0000 7f7dd88de700 1 -- 172.21.15.114:0/685659708 >> 172.21.15.114:0/685659708 conn(0x7f7dd40fe6e0 msgr2=0x7f7dd4108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:30.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.687+0000 7f7dd88de700 1 -- 172.21.15.114:0/685659708 shutdown_connections 2024-05-27T07:38:30.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:26.687+0000 7f7dd88de700 1 -- 172.21.15.114:0/685659708 wait complete. 2024-05-27T07:38:30.464 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:30.888 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822727 got 154618822731 for osd.5 2024-05-27T07:38:30.889 DEBUG:teuthology.parallel:result is None 2024-05-27T07:38:31.179 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:30 smithi114 ceph-mon[32038]: pgmap v372: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.195+0000 7f1006190700 1 -- 172.21.15.114:0/215474424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10001081f0 msgr2=0x7f1000108600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.195+0000 7f1006190700 1 --2- 172.21.15.114:0/215474424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10001081f0 0x7f1000108600 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0ff000b0a0 tx=0x7f0ff000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:38:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.195+0000 7f1006190700 1 -- 172.21.15.114:0/215474424 shutdown_connections 2024-05-27T07:38:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.195+0000 7f1006190700 1 --2- 172.21.15.114:0/215474424 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10001042e0 0x7f1000110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.195+0000 7f1006190700 1 --2- 172.21.15.114:0/215474424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10001081f0 0x7f1000108600 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.195+0000 7f1006190700 1 --2- 172.21.15.114:0/215474424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1000103990 0x7f1000103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.195+0000 7f1006190700 1 -- 172.21.15.114:0/215474424 >> 172.21.15.114:0/215474424 conn(0x7f10000fe4d0 msgr2=0x7f1000100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.195+0000 7f1006190700 1 -- 172.21.15.114:0/215474424 shutdown_connections 2024-05-27T07:38:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.195+0000 7f1006190700 1 -- 172.21.15.114:0/215474424 wait complete. 2024-05-27T07:38:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.196+0000 7f1006190700 1 Processor -- start 2024-05-27T07:38:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.196+0000 7f1006190700 1 -- start start 2024-05-27T07:38:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.196+0000 7f1006190700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1000103990 0x7f100019ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.196+0000 7f1006190700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10001042e0 0x7f100019b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.196+0000 7f1006190700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10001081f0 0x7f100019f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.196+0000 7f1006190700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f10001144d0 con 0x7f1000103990 2024-05-27T07:38:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.196+0000 7f1006190700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1000114350 con 0x7f10001042e0 2024-05-27T07:38:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.196+0000 7f1006190700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1000114650 con 0x7f10001081f0 2024-05-27T07:38:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.196+0000 7f0fff7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1000103990 0x7f100019ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.196+0000 7f0fff7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1000103990 0x7f100019ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55362/0 (socket says 172.21.15.114:55362) 2024-05-27T07:38:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.196+0000 7f0fff7fe700 1 -- 172.21.15.114:0/3754746344 learned_addr learned my addr 172.21.15.114:0/3754746344 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.196+0000 7f0ffffff700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10001081f0 0x7f100019f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.196+0000 7f0ffeffd700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10001042e0 0x7f100019b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.197+0000 7f0fff7fe700 1 -- 172.21.15.114:0/3754746344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10001081f0 msgr2=0x7f100019f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.197+0000 7f0fff7fe700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10001081f0 0x7f100019f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.197+0000 7f0fff7fe700 1 -- 172.21.15.114:0/3754746344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10001042e0 msgr2=0x7f100019b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.197+0000 7f0fff7fe700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10001042e0 0x7f100019b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.197+0000 7f0fff7fe700 1 -- 172.21.15.114:0/3754746344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f100019fc20 con 0x7f1000103990 2024-05-27T07:38:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.197+0000 7f0ffeffd700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10001042e0 0x7f100019b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.197+0000 7f0fff7fe700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1000103990 0x7f100019ae00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0ff000b070 tx=0x7f0ff0004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.197+0000 7f0ffcff9700 1 -- 172.21.15.114:0/3754746344 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0021070 con 0x7f1000103990 2024-05-27T07:38:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.197+0000 7f1006190700 1 -- 172.21.15.114:0/3754746344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f100019feb0 con 0x7f1000103990 2024-05-27T07:38:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.197+0000 7f0ffcff9700 1 -- 172.21.15.114:0/3754746344 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ff0007c80 con 0x7f1000103990 2024-05-27T07:38:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.197+0000 7f0ffcff9700 1 -- 172.21.15.114:0/3754746344 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff00052a0 con 0x7f1000103990 2024-05-27T07:38:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.197+0000 7f1006190700 1 -- 172.21.15.114:0/3754746344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10001a0470 con 0x7f1000103990 2024-05-27T07:38:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.198+0000 7f1006190700 1 -- 172.21.15.114:0/3754746344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f100010b790 con 0x7f1000103990 2024-05-27T07:38:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.200+0000 7f0ffcff9700 1 -- 172.21.15.114:0/3754746344 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f0ff0024070 con 0x7f1000103990 2024-05-27T07:38:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.201+0000 7f0ffcff9700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0fe80b2910 0x7f0fe80b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.201+0000 7f0ffeffd700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0fe80b2910 0x7f0fe80b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.201+0000 7f0ffcff9700 1 -- 172.21.15.114:0/3754746344 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f0ff000f020 con 0x7f1000103990 2024-05-27T07:38:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.201+0000 7f0ffeffd700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0fe80b2910 0x7f0fe80b4e20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4005fd0 tx=0x7f0ff4005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.206+0000 7f0ffcff9700 1 -- 172.21.15.114:0/3754746344 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0ff002b020 con 0x7f1000103990 2024-05-27T07:38:31.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:30 smithi163 ceph-mon[36451]: pgmap v372: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:31.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:30 smithi186 ceph-mon[36647]: pgmap v372: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.419+0000 7f1006190700 1 -- 172.21.15.114:0/3754746344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f100019c160 con 0x7f1000103990 2024-05-27T07:38:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.420+0000 7f0ffcff9700 1 -- 172.21.15.114:0/3754746344 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0ff00ab6f0 con 0x7f1000103990 2024-05-27T07:38:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:107374182487 2024-05-27T07:38:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.422+0000 7f1006190700 1 -- 172.21.15.114:0/3754746344 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0fe80b2910 msgr2=0x7f0fe80b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.422+0000 7f1006190700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0fe80b2910 0x7f0fe80b4e20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4005fd0 tx=0x7f0ff4005e30 comp rx=0 tx=0).stop 2024-05-27T07:38:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.422+0000 7f1006190700 1 -- 172.21.15.114:0/3754746344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1000103990 msgr2=0x7f100019ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.422+0000 7f1006190700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1000103990 0x7f100019ae00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0ff000b070 tx=0x7f0ff0004e60 comp rx=0 tx=0).stop 2024-05-27T07:38:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.422+0000 7f1006190700 1 -- 172.21.15.114:0/3754746344 shutdown_connections 2024-05-27T07:38:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.422+0000 7f1006190700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f0fe80b2910 0x7f0fe80b4e20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.422+0000 7f1006190700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10001081f0 0x7f100019f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.422+0000 7f1006190700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1000103990 0x7f100019ae00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.422+0000 7f1006190700 1 --2- 172.21.15.114:0/3754746344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10001042e0 0x7f100019b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.422+0000 7f1006190700 1 -- 172.21.15.114:0/3754746344 >> 172.21.15.114:0/3754746344 conn(0x7f10000fe4d0 msgr2=0x7f1000107640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.422+0000 7f1006190700 1 -- 172.21.15.114:0/3754746344 shutdown_connections 2024-05-27T07:38:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:31.422+0000 7f1006190700 1 -- 172.21.15.114:0/3754746344 wait complete. 2024-05-27T07:38:32.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:31 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3754746344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T07:38:32.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:31 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3754746344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T07:38:32.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:31 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3754746344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T07:38:32.774 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:33.190 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:32 smithi114 ceph-mon[32038]: pgmap v373: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:33.190 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:32 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3972743959' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T07:38:33.217 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182482 got 107374182487 for osd.3 2024-05-27T07:38:33.217 DEBUG:teuthology.parallel:result is None 2024-05-27T07:38:33.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:32 smithi163 ceph-mon[36451]: pgmap v373: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:33.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:32 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3972743959' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T07:38:33.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:32 smithi186 ceph-mon[36647]: pgmap v373: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:33.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:32 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3972743959' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T07:38:33.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.675+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/1677829123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f581089e0 msgr2=0x7f8f58108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:33.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.675+0000 7f8f5f5d6700 1 --2- 172.21.15.114:0/1677829123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f581089e0 0x7f8f58108df0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009990 tx=0x7f8f54009ca0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.675+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/1677829123 shutdown_connections 2024-05-27T07:38:33.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.675+0000 7f8f5f5d6700 1 --2- 172.21.15.114:0/1677829123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f58103300 0x7f8f5810f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.675+0000 7f8f5f5d6700 1 --2- 172.21.15.114:0/1677829123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f581089e0 0x7f8f58108df0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.675+0000 7f8f5f5d6700 1 --2- 172.21.15.114:0/1677829123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f581029b0 0x7f8f58102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.675+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/1677829123 >> 172.21.15.114:0/1677829123 conn(0x7f8f580fe6e0 msgr2=0x7f8f58100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:33.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.675+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/1677829123 shutdown_connections 2024-05-27T07:38:33.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.675+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/1677829123 wait complete. 2024-05-27T07:38:33.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.676+0000 7f8f5f5d6700 1 Processor -- start 2024-05-27T07:38:33.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.676+0000 7f8f5f5d6700 1 -- start start 2024-05-27T07:38:33.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.676+0000 7f8f5f5d6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f581029b0 0x7f8f58199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:33.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.676+0000 7f8f5f5d6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f58103300 0x7f8f5819a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.676+0000 7f8f5f5d6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f581089e0 0x7f8f5819e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.676+0000 7f8f5f5d6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8f58112b70 con 0x7f8f58103300 2024-05-27T07:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.676+0000 7f8f5f5d6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8f581129f0 con 0x7f8f581029b0 2024-05-27T07:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.676+0000 7f8f5f5d6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8f58112cf0 con 0x7f8f581089e0 2024-05-27T07:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f5cb71700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f58103300 0x7f8f5819a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f5cb71700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f58103300 0x7f8f5819a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55372/0 (socket says 172.21.15.114:55372) 2024-05-27T07:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f5cb71700 1 -- 172.21.15.114:0/3972743959 learned_addr learned my addr 172.21.15.114:0/3972743959 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f5db73700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f581089e0 0x7f8f5819e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f5d372700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f581029b0 0x7f8f58199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f5cb71700 1 -- 172.21.15.114:0/3972743959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f581089e0 msgr2=0x7f8f5819e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f5cb71700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f581089e0 0x7f8f5819e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f5cb71700 1 -- 172.21.15.114:0/3972743959 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f581029b0 msgr2=0x7f8f58199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f5cb71700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f581029b0 0x7f8f58199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f5cb71700 1 -- 172.21.15.114:0/3972743959 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5819ec50 con 0x7f8f58103300 2024-05-27T07:38:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f5cb71700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f58103300 0x7f8f5819a320 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800c8f0 tx=0x7f8f4800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f4e7fc700 1 -- 172.21.15.114:0/3972743959 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f48007b00 con 0x7f8f58103300 2024-05-27T07:38:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f4e7fc700 1 -- 172.21.15.114:0/3972743959 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f48007ca0 con 0x7f8f58103300 2024-05-27T07:38:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/3972743959 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f5819ef40 con 0x7f8f58103300 2024-05-27T07:38:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f4e7fc700 1 -- 172.21.15.114:0/3972743959 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f48018660 con 0x7f8f58103300 2024-05-27T07:38:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.677+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/3972743959 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f581a6860 con 0x7f8f58103300 2024-05-27T07:38:33.277 INFO:teuthology.orchestra.run.smithi114.stdout:227633266748 2024-05-27T07:38:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.678+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/3972743959 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f580512b0 con 0x7f8f58103300 2024-05-27T07:38:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.680+0000 7f8f4e7fc700 1 -- 172.21.15.114:0/3972743959 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f8f4800f450 con 0x7f8f58103300 2024-05-27T07:38:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.681+0000 7f8f4e7fc700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8f440b2910 0x7f8f440b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.681+0000 7f8f5d372700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8f440b2910 0x7f8f440b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.681+0000 7f8f4e7fc700 1 -- 172.21.15.114:0/3972743959 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f8f480e2850 con 0x7f8f58103300 2024-05-27T07:38:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.681+0000 7f8f5d372700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8f440b2910 0x7f8f440b4e20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009750 tx=0x7f8f5401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.684+0000 7f8f4e7fc700 1 -- 172.21.15.114:0/3972743959 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8f48014030 con 0x7f8f58103300 2024-05-27T07:38:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.896+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/3972743959 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f8f581a6a30 con 0x7f8f58103300 2024-05-27T07:38:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.896+0000 7f8f4e7fc700 1 -- 172.21.15.114:0/3972743959 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8f480aa0d0 con 0x7f8f58103300 2024-05-27T07:38:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.899+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/3972743959 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8f440b2910 msgr2=0x7f8f440b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.899+0000 7f8f5f5d6700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8f440b2910 0x7f8f440b4e20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009750 tx=0x7f8f5401a040 comp rx=0 tx=0).stop 2024-05-27T07:38:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.899+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/3972743959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f58103300 msgr2=0x7f8f5819a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.899+0000 7f8f5f5d6700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f58103300 0x7f8f5819a320 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800c8f0 tx=0x7f8f4800ccb0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.899+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/3972743959 shutdown_connections 2024-05-27T07:38:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.899+0000 7f8f5f5d6700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8f440b2910 0x7f8f440b4e20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.899+0000 7f8f5f5d6700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f581089e0 0x7f8f5819e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.899+0000 7f8f5f5d6700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f581029b0 0x7f8f58199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.899+0000 7f8f5f5d6700 1 --2- 172.21.15.114:0/3972743959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f58103300 0x7f8f5819a320 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.899+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/3972743959 >> 172.21.15.114:0/3972743959 conn(0x7f8f580fe6e0 msgr2=0x7f8f58106ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.900+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/3972743959 shutdown_connections 2024-05-27T07:38:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:32.900+0000 7f8f5f5d6700 1 -- 172.21.15.114:0/3972743959 wait complete. 2024-05-27T07:38:33.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.719+0000 7fee56ca7700 1 -- 172.21.15.114:0/3886682111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee50107de0 msgr2=0x7fee501081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:33.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.719+0000 7fee56ca7700 1 --2- 172.21.15.114:0/3886682111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee50107de0 0x7fee501081f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fee3c0099c0 tx=0x7fee3c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.719+0000 7fee56ca7700 1 -- 172.21.15.114:0/3886682111 shutdown_connections 2024-05-27T07:38:33.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.719+0000 7fee56ca7700 1 --2- 172.21.15.114:0/3886682111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee50108730 0x7fee50110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.719+0000 7fee56ca7700 1 --2- 172.21.15.114:0/3886682111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee501034b0 0x7fee501038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.719+0000 7fee56ca7700 1 --2- 172.21.15.114:0/3886682111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee50107de0 0x7fee501081f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.719+0000 7fee56ca7700 1 -- 172.21.15.114:0/3886682111 >> 172.21.15.114:0/3886682111 conn(0x7fee500fedb0 msgr2=0x7fee501011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:33.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.719+0000 7fee56ca7700 1 -- 172.21.15.114:0/3886682111 shutdown_connections 2024-05-27T07:38:33.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.719+0000 7fee56ca7700 1 -- 172.21.15.114:0/3886682111 wait complete. 2024-05-27T07:38:33.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.719+0000 7fee56ca7700 1 Processor -- start 2024-05-27T07:38:33.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee56ca7700 1 -- start start 2024-05-27T07:38:33.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee56ca7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee501034b0 0x7fee500ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:33.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee56ca7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee50107de0 0x7fee500ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:33.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee56ca7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee50108730 0x7fee500ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:33.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee56ca7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fee50113390 con 0x7fee501034b0 2024-05-27T07:38:33.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee56ca7700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fee50113210 con 0x7fee50108730 2024-05-27T07:38:33.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee56ca7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fee50113510 con 0x7fee50107de0 2024-05-27T07:38:33.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee54a43700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee501034b0 0x7fee500ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:33.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee54a43700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee501034b0 0x7fee500ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55394/0 (socket says 172.21.15.114:55394) 2024-05-27T07:38:33.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee54a43700 1 -- 172.21.15.114:0/531281837 learned_addr learned my addr 172.21.15.114:0/531281837 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:33.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee4ffff700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee50107de0 0x7fee500ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:33.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee55244700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee50108730 0x7fee500ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:33.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee54a43700 1 -- 172.21.15.114:0/531281837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee50107de0 msgr2=0x7fee500ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:33.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee54a43700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee50107de0 0x7fee500ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee54a43700 1 -- 172.21.15.114:0/531281837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee50108730 msgr2=0x7fee500ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:33.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee54a43700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee50108730 0x7fee500ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.720+0000 7fee54a43700 1 -- 172.21.15.114:0/531281837 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee501a91d0 con 0x7fee501034b0 2024-05-27T07:38:33.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.721+0000 7fee54a43700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee501034b0 0x7fee500ff290 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fee4400d980 tx=0x7fee4400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:33.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.721+0000 7fee4dffb700 1 -- 172.21.15.114:0/531281837 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee44016070 con 0x7fee501034b0 2024-05-27T07:38:33.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.721+0000 7fee4dffb700 1 -- 172.21.15.114:0/531281837 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee44004830 con 0x7fee501034b0 2024-05-27T07:38:33.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.721+0000 7fee56ca7700 1 -- 172.21.15.114:0/531281837 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee501a94c0 con 0x7fee501034b0 2024-05-27T07:38:33.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.721+0000 7fee4dffb700 1 -- 172.21.15.114:0/531281837 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee440054c0 con 0x7fee501034b0 2024-05-27T07:38:33.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.721+0000 7fee56ca7700 1 -- 172.21.15.114:0/531281837 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee501a99d0 con 0x7fee501034b0 2024-05-27T07:38:33.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.722+0000 7fee56ca7700 1 -- 172.21.15.114:0/531281837 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee500512b0 con 0x7fee501034b0 2024-05-27T07:38:33.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.723+0000 7fee4dffb700 1 -- 172.21.15.114:0/531281837 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fee44004220 con 0x7fee501034b0 2024-05-27T07:38:33.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.724+0000 7fee4dffb700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fee400b2910 0x7fee400b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:33.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.724+0000 7fee4ffff700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fee400b2910 0x7fee400b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:33.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.724+0000 7fee4dffb700 1 -- 172.21.15.114:0/531281837 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fee440e2600 con 0x7fee501034b0 2024-05-27T07:38:33.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.725+0000 7fee4ffff700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fee400b2910 0x7fee400b4e20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fee3c006010 tx=0x7fee3c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:33.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.729+0000 7fee4dffb700 1 -- 172.21.15.114:0/531281837 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fee4401f020 con 0x7fee501034b0 2024-05-27T07:38:33.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.963+0000 7fee56ca7700 1 -- 172.21.15.114:0/531281837 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fee5002e5e0 con 0x7fee501034b0 2024-05-27T07:38:33.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.963+0000 7fee4dffb700 1 -- 172.21.15.114:0/531281837 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fee440b0e20 con 0x7fee501034b0 2024-05-27T07:38:33.960 INFO:teuthology.orchestra.run.smithi114.stdout:34359738470 2024-05-27T07:38:33.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.966+0000 7fee56ca7700 1 -- 172.21.15.114:0/531281837 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fee400b2910 msgr2=0x7fee400b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:33.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.966+0000 7fee56ca7700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fee400b2910 0x7fee400b4e20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fee3c006010 tx=0x7fee3c000bc0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.966+0000 7fee56ca7700 1 -- 172.21.15.114:0/531281837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee501034b0 msgr2=0x7fee500ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:33.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.966+0000 7fee56ca7700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee501034b0 0x7fee500ff290 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fee4400d980 tx=0x7fee4400dc90 comp rx=0 tx=0).stop 2024-05-27T07:38:33.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.966+0000 7fee56ca7700 1 -- 172.21.15.114:0/531281837 shutdown_connections 2024-05-27T07:38:33.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.966+0000 7fee56ca7700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee50108730 0x7fee500ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.966+0000 7fee56ca7700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee501034b0 0x7fee500ff290 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.966+0000 7fee56ca7700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fee400b2910 0x7fee400b4e20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.966+0000 7fee56ca7700 1 --2- 172.21.15.114:0/531281837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee50107de0 0x7fee500ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.966+0000 7fee56ca7700 1 -- 172.21.15.114:0/531281837 >> 172.21.15.114:0/531281837 conn(0x7fee500fedb0 msgr2=0x7fee5006b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.966+0000 7fee56ca7700 1 -- 172.21.15.114:0/531281837 shutdown_connections 2024-05-27T07:38:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:33.966+0000 7fee56ca7700 1 -- 172.21.15.114:0/531281837 wait complete. 2024-05-27T07:38:35.215 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:34 smithi114 ceph-mon[32038]: pgmap v374: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:35.215 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:34 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/531281837' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T07:38:35.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:34 smithi163 ceph-mon[36451]: pgmap v374: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:35.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:34 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/531281837' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T07:38:35.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:34 smithi186 ceph-mon[36647]: pgmap v374: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:35.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:34 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/531281837' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T07:38:35.242 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738464 got 34359738470 for osd.0 2024-05-27T07:38:35.242 DEBUG:teuthology.parallel:result is None 2024-05-27T07:38:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.123+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/3426820993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb4108770 msgr2=0x7f9eb41100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.123+0000 7f9eb8cd0700 1 --2- 172.21.15.114:0/3426820993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb4108770 0x7f9eb41100d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9eac0099c0 tx=0x7f9eac009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.123+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/3426820993 shutdown_connections 2024-05-27T07:38:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.123+0000 7f9eb8cd0700 1 --2- 172.21.15.114:0/3426820993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb4108770 0x7f9eb41100d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.123+0000 7f9eb8cd0700 1 --2- 172.21.15.114:0/3426820993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9eb4103540 0x7f9eb4103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.123+0000 7f9eb8cd0700 1 --2- 172.21.15.114:0/3426820993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9eb4107e20 0x7f9eb4108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.123+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/3426820993 >> 172.21.15.114:0/3426820993 conn(0x7f9eb4078df0 msgr2=0x7f9eb4079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.123+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/3426820993 shutdown_connections 2024-05-27T07:38:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.123+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/3426820993 wait complete. 2024-05-27T07:38:36.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.124+0000 7f9eb8cd0700 1 Processor -- start 2024-05-27T07:38:36.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.124+0000 7f9eb8cd0700 1 -- start start 2024-05-27T07:38:36.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.124+0000 7f9eb8cd0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9eb4103540 0x7f9eb4199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:36.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.124+0000 7f9eb8cd0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9eb4107e20 0x7f9eb419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:36.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.124+0000 7f9eb8cd0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb4108770 0x7f9eb419e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:36.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.124+0000 7f9eb8cd0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4113410 con 0x7f9eb4108770 2024-05-27T07:38:36.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.124+0000 7f9eb8cd0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4113290 con 0x7f9eb4107e20 2024-05-27T07:38:36.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.124+0000 7f9eb8cd0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4113590 con 0x7f9eb4103540 2024-05-27T07:38:36.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb4108770 0x7f9eb419e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:36.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb4108770 0x7f9eb419e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59248/0 (socket says 172.21.15.114:59248) 2024-05-27T07:38:36.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb2d9d700 1 -- 172.21.15.114:0/973249986 learned_addr learned my addr 172.21.15.114:0/973249986 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:36.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb259c700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9eb4103540 0x7f9eb4199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:36.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb1d9b700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9eb4107e20 0x7f9eb419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:36.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb2d9d700 1 -- 172.21.15.114:0/973249986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9eb4103540 msgr2=0x7f9eb4199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:36.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb2d9d700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9eb4103540 0x7f9eb4199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:36.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb2d9d700 1 -- 172.21.15.114:0/973249986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9eb4107e20 msgr2=0x7f9eb419a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:36.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb2d9d700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9eb4107e20 0x7f9eb419a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:36.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb2d9d700 1 -- 172.21.15.114:0/973249986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb419eb40 con 0x7f9eb4108770 2024-05-27T07:38:36.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb1d9b700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9eb4107e20 0x7f9eb419a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:36.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb2d9d700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb4108770 0x7f9eb419e550 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009990 tx=0x7f9eac0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:36.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eab7fe700 1 -- 172.21.15.114:0/973249986 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac027070 con 0x7f9eb4108770 2024-05-27T07:38:36.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eab7fe700 1 -- 172.21.15.114:0/973249986 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9eac004150 con 0x7f9eb4108770 2024-05-27T07:38:36.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/973249986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb419edd0 con 0x7f9eb4108770 2024-05-27T07:38:36.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eab7fe700 1 -- 172.21.15.114:0/973249986 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac00f460 con 0x7f9eb4108770 2024-05-27T07:38:36.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.125+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/973249986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb419f260 con 0x7f9eb4108770 2024-05-27T07:38:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.127+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/973249986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eb40512b0 con 0x7f9eb4108770 2024-05-27T07:38:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.128+0000 7f9eab7fe700 1 -- 172.21.15.114:0/973249986 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f9eac02a070 con 0x7f9eb4108770 2024-05-27T07:38:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.129+0000 7f9eab7fe700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9ea00b2910 0x7f9ea00b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.129+0000 7f9eb259c700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9ea00b2910 0x7f9ea00b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.129+0000 7f9eab7fe700 1 -- 172.21.15.114:0/973249986 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f9eac0e3cf0 con 0x7f9eb4108770 2024-05-27T07:38:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.129+0000 7f9eb259c700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9ea00b2910 0x7f9ea00b4e20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400e270 tx=0x7f9ea4009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:36.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.132+0000 7f9eab7fe700 1 -- 172.21.15.114:0/973249986 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9eac031080 con 0x7f9eb4108770 2024-05-27T07:38:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.341+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/973249986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f9eb41a6800 con 0x7f9eb4108770 2024-05-27T07:38:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.341+0000 7f9eab7fe700 1 -- 172.21.15.114:0/973249986 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9eac013070 con 0x7f9eb4108770 2024-05-27T07:38:36.338 INFO:teuthology.orchestra.run.smithi114.stdout:180388626504 2024-05-27T07:38:36.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.343+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/973249986 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9ea00b2910 msgr2=0x7f9ea00b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:36.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.343+0000 7f9eb8cd0700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9ea00b2910 0x7f9ea00b4e20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400e270 tx=0x7f9ea4009040 comp rx=0 tx=0).stop 2024-05-27T07:38:36.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.343+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/973249986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb4108770 msgr2=0x7f9eb419e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:36.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.343+0000 7f9eb8cd0700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb4108770 0x7f9eb419e550 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009990 tx=0x7f9eac0047b0 comp rx=0 tx=0).stop 2024-05-27T07:38:36.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.343+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/973249986 shutdown_connections 2024-05-27T07:38:36.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.343+0000 7f9eb8cd0700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb4108770 0x7f9eb419e550 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:36.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.343+0000 7f9eb8cd0700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f9ea00b2910 0x7f9ea00b4e20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:36.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.343+0000 7f9eb8cd0700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9eb4103540 0x7f9eb4199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:36.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.343+0000 7f9eb8cd0700 1 --2- 172.21.15.114:0/973249986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9eb4107e20 0x7f9eb419a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:36.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.343+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/973249986 >> 172.21.15.114:0/973249986 conn(0x7f9eb4078df0 msgr2=0x7f9eb4101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:36.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.344+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/973249986 shutdown_connections 2024-05-27T07:38:36.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:36.344+0000 7f9eb8cd0700 1 -- 172.21.15.114:0/973249986 wait complete. 2024-05-27T07:38:37.146 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:36 smithi114 ceph-mon[32038]: pgmap v375: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:37.146 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:36 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/973249986' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T07:38:37.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:36 smithi163 ceph-mon[36451]: pgmap v375: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:37.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:36 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/973249986' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T07:38:37.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:36 smithi186 ceph-mon[36647]: pgmap v375: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:37.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:36 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/973249986' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T07:38:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.265+0000 7f344989f700 1 -- 172.21.15.114:0/2920109930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34441034d0 msgr2=0x7f34441038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.265+0000 7f344989f700 1 --2- 172.21.15.114:0/2920109930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34441034d0 0x7f34441038e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f34380099c0 tx=0x7f3438009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.265+0000 7f344989f700 1 -- 172.21.15.114:0/2920109930 shutdown_connections 2024-05-27T07:38:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.265+0000 7f344989f700 1 --2- 172.21.15.114:0/2920109930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3444108750 0x7f34441100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.265+0000 7f344989f700 1 --2- 172.21.15.114:0/2920109930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34441034d0 0x7f34441038e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.265+0000 7f344989f700 1 --2- 172.21.15.114:0/2920109930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3444107e00 0x7f3444108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.265+0000 7f344989f700 1 -- 172.21.15.114:0/2920109930 >> 172.21.15.114:0/2920109930 conn(0x7f34440fef20 msgr2=0x7f3444101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.265+0000 7f344989f700 1 -- 172.21.15.114:0/2920109930 shutdown_connections 2024-05-27T07:38:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.265+0000 7f344989f700 1 -- 172.21.15.114:0/2920109930 wait complete. 2024-05-27T07:38:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.266+0000 7f344989f700 1 Processor -- start 2024-05-27T07:38:37.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.266+0000 7f344989f700 1 -- start start 2024-05-27T07:38:37.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.266+0000 7f344989f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34441034d0 0x7f344419af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:37.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.266+0000 7f344989f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3444107e00 0x7f344419b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:37.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.266+0000 7f344989f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3444108750 0x7f344419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:37.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.266+0000 7f344989f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3444113470 con 0x7f34441034d0 2024-05-27T07:38:37.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.266+0000 7f344989f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f34441132f0 con 0x7f3444108750 2024-05-27T07:38:37.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.266+0000 7f344989f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f34441135f0 con 0x7f3444107e00 2024-05-27T07:38:37.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.266+0000 7f3442ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34441034d0 0x7f344419af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.266+0000 7f3442ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34441034d0 0x7f344419af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59252/0 (socket says 172.21.15.114:59252) 2024-05-27T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.266+0000 7f3442ffd700 1 -- 172.21.15.114:0/2605603476 learned_addr learned my addr 172.21.15.114:0/2605603476 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.266+0000 7f34437fe700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3444108750 0x7f344419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.266+0000 7f34427fc700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3444107e00 0x7f344419b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.267+0000 7f3442ffd700 1 -- 172.21.15.114:0/2605603476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3444107e00 msgr2=0x7f344419b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.267+0000 7f3442ffd700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3444107e00 0x7f344419b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.267+0000 7f3442ffd700 1 -- 172.21.15.114:0/2605603476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3444108750 msgr2=0x7f344419f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.267+0000 7f3442ffd700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3444108750 0x7f344419f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.267+0000 7f3442ffd700 1 -- 172.21.15.114:0/2605603476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344419fe00 con 0x7f34441034d0 2024-05-27T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.267+0000 7f34427fc700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3444107e00 0x7f344419b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.267+0000 7f34437fe700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3444108750 0x7f344419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.267+0000 7f3442ffd700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34441034d0 0x7f344419af30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f343800b580 tx=0x7f34380037c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.267+0000 7f344889d700 1 -- 172.21.15.114:0/2605603476 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3438027070 con 0x7f34441034d0 2024-05-27T07:38:37.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.267+0000 7f344989f700 1 -- 172.21.15.114:0/2605603476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34441a0090 con 0x7f34441034d0 2024-05-27T07:38:37.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.267+0000 7f344889d700 1 -- 172.21.15.114:0/2605603476 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3438004110 con 0x7f34441034d0 2024-05-27T07:38:37.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.267+0000 7f344889d700 1 -- 172.21.15.114:0/2605603476 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343800f460 con 0x7f34441034d0 2024-05-27T07:38:37.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.267+0000 7f344989f700 1 -- 172.21.15.114:0/2605603476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34440ff330 con 0x7f34441034d0 2024-05-27T07:38:37.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.268+0000 7f342a7fc700 1 -- 172.21.15.114:0/2605603476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3424005330 con 0x7f34441034d0 2024-05-27T07:38:37.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.271+0000 7f344889d700 1 -- 172.21.15.114:0/2605603476 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f343802a070 con 0x7f34441034d0 2024-05-27T07:38:37.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.271+0000 7f344889d700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f342c0b2910 0x7f342c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:37.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.271+0000 7f344889d700 1 -- 172.21.15.114:0/2605603476 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f34380e3a90 con 0x7f34441034d0 2024-05-27T07:38:37.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.272+0000 7f34427fc700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f342c0b2910 0x7f342c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:37.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.272+0000 7f34427fc700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f342c0b2910 0x7f342c0b4e20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3444102240 tx=0x7f3434008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:37.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.274+0000 7f344889d700 1 -- 172.21.15.114:0/2605603476 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3438030070 con 0x7f34441034d0 2024-05-27T07:38:37.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.484+0000 7f342a7fc700 1 -- 172.21.15.114:0/2605603476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f34240058f0 con 0x7f34441034d0 2024-05-27T07:38:37.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.484+0000 7f344889d700 1 -- 172.21.15.114:0/2605603476 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3438013070 con 0x7f34441034d0 2024-05-27T07:38:37.482 INFO:teuthology.orchestra.run.smithi114.stdout:77309411421 2024-05-27T07:38:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.487+0000 7f342a7fc700 1 -- 172.21.15.114:0/2605603476 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f342c0b2910 msgr2=0x7f342c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.487+0000 7f342a7fc700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f342c0b2910 0x7f342c0b4e20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3444102240 tx=0x7f3434008040 comp rx=0 tx=0).stop 2024-05-27T07:38:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.487+0000 7f342a7fc700 1 -- 172.21.15.114:0/2605603476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34441034d0 msgr2=0x7f344419af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.487+0000 7f342a7fc700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34441034d0 0x7f344419af30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f343800b580 tx=0x7f34380037c0 comp rx=0 tx=0).stop 2024-05-27T07:38:37.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.487+0000 7f342a7fc700 1 -- 172.21.15.114:0/2605603476 shutdown_connections 2024-05-27T07:38:37.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.487+0000 7f342a7fc700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3444108750 0x7f344419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:37.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.487+0000 7f342a7fc700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34441034d0 0x7f344419af30 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:37.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.487+0000 7f342a7fc700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f342c0b2910 0x7f342c0b4e20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:37.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.487+0000 7f342a7fc700 1 --2- 172.21.15.114:0/2605603476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3444107e00 0x7f344419b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:37.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.487+0000 7f342a7fc700 1 -- 172.21.15.114:0/2605603476 >> 172.21.15.114:0/2605603476 conn(0x7f34440fef20 msgr2=0x7f34441063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:37.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.487+0000 7f342a7fc700 1 -- 172.21.15.114:0/2605603476 shutdown_connections 2024-05-27T07:38:37.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:37.487+0000 7f342a7fc700 1 -- 172.21.15.114:0/2605603476 wait complete. 2024-05-27T07:38:37.673 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626500 got 180388626504 for osd.6 2024-05-27T07:38:37.674 DEBUG:teuthology.parallel:result is None 2024-05-27T07:38:38.209 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:38 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:38.209 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:38 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:38.210 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:38 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2605603476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T07:38:38.210 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:38 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:38:38.210 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:38 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:38:38.210 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:38 smithi114 ceph-mon[32038]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:38.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:38 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:38.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:38 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:38.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:38 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2605603476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T07:38:38.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:38 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:38:38.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:38 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:38:38.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:38 smithi163 ceph-mon[36451]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:38.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:38 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:38.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:38 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:38.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:38 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2605603476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T07:38:38.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:38 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:38:38.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:38 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:38:38.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:38 smithi186 ceph-mon[36647]: from='mgr.14152 172.21.15.114:0/457860227' entity='mgr.x' 2024-05-27T07:38:38.561 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266742 got 227633266748 for osd.8 2024-05-27T07:38:38.561 DEBUG:teuthology.parallel:result is None 2024-05-27T07:38:39.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.008+0000 7f434add9700 1 -- 172.21.15.114:0/3436293718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4344109850 msgr2=0x7f43441111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:39.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.008+0000 7f434add9700 1 --2- 172.21.15.114:0/3436293718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4344109850 0x7f43441111b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f433c00b0d0 tx=0x7f433c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:38:39.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.008+0000 7f434add9700 1 -- 172.21.15.114:0/3436293718 shutdown_connections 2024-05-27T07:38:39.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.008+0000 7f434add9700 1 --2- 172.21.15.114:0/3436293718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4344109850 0x7f43441111b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:39.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.008+0000 7f434add9700 1 --2- 172.21.15.114:0/3436293718 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43441045d0 0x7f43441049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:39.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.008+0000 7f434add9700 1 --2- 172.21.15.114:0/3436293718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4344108f00 0x7f4344109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:39.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.008+0000 7f434add9700 1 -- 172.21.15.114:0/3436293718 >> 172.21.15.114:0/3436293718 conn(0x7f43440ffec0 msgr2=0x7f4344102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:39.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.008+0000 7f434add9700 1 -- 172.21.15.114:0/3436293718 shutdown_connections 2024-05-27T07:38:39.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.008+0000 7f434add9700 1 -- 172.21.15.114:0/3436293718 wait complete. 2024-05-27T07:38:39.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.009+0000 7f434add9700 1 Processor -- start 2024-05-27T07:38:39.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.009+0000 7f434add9700 1 -- start start 2024-05-27T07:38:39.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.009+0000 7f434add9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43441045d0 0x7f434419adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:39.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.009+0000 7f434add9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4344108f00 0x7f434419b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:39.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.009+0000 7f434add9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4344109850 0x7f434419f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.009+0000 7f434add9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4344114480 con 0x7f4344108f00 2024-05-27T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.009+0000 7f434add9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4344114300 con 0x7f4344109850 2024-05-27T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.009+0000 7f434add9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4344114600 con 0x7f43441045d0 2024-05-27T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4343fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4344108f00 0x7f434419b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4343fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4344108f00 0x7f434419b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59270/0 (socket says 172.21.15.114:59270) 2024-05-27T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4343fff700 1 -- 172.21.15.114:0/1410727965 learned_addr learned my addr 172.21.15.114:0/1410727965 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4348b75700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43441045d0 0x7f434419adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4349376700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4344109850 0x7f434419f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4343fff700 1 -- 172.21.15.114:0/1410727965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43441045d0 msgr2=0x7f434419adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4343fff700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43441045d0 0x7f434419adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4343fff700 1 -- 172.21.15.114:0/1410727965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4344109850 msgr2=0x7f434419f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4343fff700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4344109850 0x7f434419f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:39.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4343fff700 1 -- 172.21.15.114:0/1410727965 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434419fc80 con 0x7f4344108f00 2024-05-27T07:38:39.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4348b75700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43441045d0 0x7f434419adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:39.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4343fff700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4344108f00 0x7f434419b350 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f433000c8f0 tx=0x7f433000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:39.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4341ffb700 1 -- 172.21.15.114:0/1410727965 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4330007b00 con 0x7f4344108f00 2024-05-27T07:38:39.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4341ffb700 1 -- 172.21.15.114:0/1410727965 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4330007ca0 con 0x7f4344108f00 2024-05-27T07:38:39.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f434add9700 1 -- 172.21.15.114:0/1410727965 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f434419ff70 con 0x7f4344108f00 2024-05-27T07:38:39.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.010+0000 7f4341ffb700 1 -- 172.21.15.114:0/1410727965 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4330018660 con 0x7f4344108f00 2024-05-27T07:38:39.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.011+0000 7f434add9700 1 -- 172.21.15.114:0/1410727965 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43441a7890 con 0x7f4344108f00 2024-05-27T07:38:39.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.012+0000 7f434add9700 1 -- 172.21.15.114:0/1410727965 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43440512b0 con 0x7f4344108f00 2024-05-27T07:38:39.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.013+0000 7f4341ffb700 1 -- 172.21.15.114:0/1410727965 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f433000f450 con 0x7f4344108f00 2024-05-27T07:38:39.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.014+0000 7f4341ffb700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f43340b2910 0x7f43340b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:39.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.014+0000 7f4341ffb700 1 -- 172.21.15.114:0/1410727965 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f43300e2700 con 0x7f4344108f00 2024-05-27T07:38:39.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.014+0000 7f4348b75700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f43340b2910 0x7f43340b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:39.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.014+0000 7f4348b75700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f43340b2910 0x7f43340b4e20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4338005950 tx=0x7f43380058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:39.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.018+0000 7f4341ffb700 1 -- 172.21.15.114:0/1410727965 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4330014030 con 0x7f4344108f00 2024-05-27T07:38:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.227+0000 7f434add9700 1 -- 172.21.15.114:0/1410727965 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f434419c110 con 0x7f4344108f00 2024-05-27T07:38:39.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:39 smithi114 ceph-mon[32038]: pgmap v376: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.228+0000 7f4341ffb700 1 -- 172.21.15.114:0/1410727965 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f43300a9ef0 con 0x7f4344108f00 2024-05-27T07:38:39.225 INFO:teuthology.orchestra.run.smithi114.stdout:300647710766 2024-05-27T07:38:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.230+0000 7f434add9700 1 -- 172.21.15.114:0/1410727965 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f43340b2910 msgr2=0x7f43340b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.230+0000 7f434add9700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f43340b2910 0x7f43340b4e20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4338005950 tx=0x7f43380058e0 comp rx=0 tx=0).stop 2024-05-27T07:38:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.230+0000 7f434add9700 1 -- 172.21.15.114:0/1410727965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4344108f00 msgr2=0x7f434419b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.230+0000 7f434add9700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4344108f00 0x7f434419b350 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f433000c8f0 tx=0x7f433000ccb0 comp rx=0 tx=0).stop 2024-05-27T07:38:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.230+0000 7f434add9700 1 -- 172.21.15.114:0/1410727965 shutdown_connections 2024-05-27T07:38:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.230+0000 7f434add9700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4344109850 0x7f434419f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.230+0000 7f434add9700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f43340b2910 0x7f43340b4e20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.230+0000 7f434add9700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43441045d0 0x7f434419adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.230+0000 7f434add9700 1 --2- 172.21.15.114:0/1410727965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4344108f00 0x7f434419b350 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.230+0000 7f434add9700 1 -- 172.21.15.114:0/1410727965 >> 172.21.15.114:0/1410727965 conn(0x7f43440ffec0 msgr2=0x7f4344108670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.230+0000 7f434add9700 1 -- 172.21.15.114:0/1410727965 shutdown_connections 2024-05-27T07:38:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:39.230+0000 7f434add9700 1 -- 172.21.15.114:0/1410727965 wait complete. 2024-05-27T07:38:39.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:39 smithi163 ceph-mon[36451]: pgmap v376: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:39.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:39 smithi186 ceph-mon[36647]: pgmap v376: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:39.895 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411416 got 77309411421 for osd.2 2024-05-27T07:38:39.896 DEBUG:teuthology.parallel:result is None 2024-05-27T07:38:40.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:40 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1410727965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T07:38:40.383 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710760 got 300647710766 for osd.11 2024-05-27T07:38:40.383 DEBUG:teuthology.parallel:result is None 2024-05-27T07:38:40.383 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-27T07:38:40.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:38:40.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:40 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1410727965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T07:38:40.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:40 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1410727965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T07:38:40.627 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:41.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:41 smithi114 ceph-mon[32038]: pgmap v377: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:41.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:41 smithi163 ceph-mon[36451]: pgmap v377: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:41.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:41 smithi186 ceph-mon[36647]: pgmap v377: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:42.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.359+0000 7f896209a700 1 -- 172.21.15.114:0/3451632757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f895c1034d0 msgr2=0x7f895c1038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:42.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.359+0000 7f896209a700 1 --2- 172.21.15.114:0/3451632757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f895c1034d0 0x7f895c1038e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f894c0099c0 tx=0x7f894c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:42.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.359+0000 7f896209a700 1 -- 172.21.15.114:0/3451632757 shutdown_connections 2024-05-27T07:38:42.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.359+0000 7f896209a700 1 --2- 172.21.15.114:0/3451632757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f895c108750 0x7f895c1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:42.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.359+0000 7f896209a700 1 --2- 172.21.15.114:0/3451632757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f895c1034d0 0x7f895c1038e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:42.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.359+0000 7f896209a700 1 --2- 172.21.15.114:0/3451632757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f895c107e00 0x7f895c108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:42.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.359+0000 7f896209a700 1 -- 172.21.15.114:0/3451632757 >> 172.21.15.114:0/3451632757 conn(0x7f895c0fef20 msgr2=0x7f895c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:42.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.359+0000 7f896209a700 1 -- 172.21.15.114:0/3451632757 shutdown_connections 2024-05-27T07:38:42.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.359+0000 7f896209a700 1 -- 172.21.15.114:0/3451632757 wait complete. 2024-05-27T07:38:42.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.359+0000 7f896209a700 1 Processor -- start 2024-05-27T07:38:42.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f896209a700 1 -- start start 2024-05-27T07:38:42.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f896209a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f895c1034d0 0x7f895c19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:42.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f896209a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f895c107e00 0x7f895c19b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:42.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f896209a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f895c108750 0x7f895c19f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:42.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f896209a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f895c113380 con 0x7f895c1034d0 2024-05-27T07:38:42.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f896209a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f895c113200 con 0x7f895c107e00 2024-05-27T07:38:42.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f896209a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f895c113500 con 0x7f895c108750 2024-05-27T07:38:42.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f895b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f895c1034d0 0x7f895c19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:42.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f895b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f895c1034d0 0x7f895c19ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59296/0 (socket says 172.21.15.114:59296) 2024-05-27T07:38:42.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f895b7fe700 1 -- 172.21.15.114:0/2448683885 learned_addr learned my addr 172.21.15.114:0/2448683885 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:42.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f8953fff700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f895c107e00 0x7f895c19b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:42.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f895bfff700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f895c108750 0x7f895c19f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:42.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f895b7fe700 1 -- 172.21.15.114:0/2448683885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f895c108750 msgr2=0x7f895c19f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:42.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f895b7fe700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f895c108750 0x7f895c19f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:42.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f895b7fe700 1 -- 172.21.15.114:0/2448683885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f895c107e00 msgr2=0x7f895c19b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:42.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f895b7fe700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f895c107e00 0x7f895c19b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:42.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.360+0000 7f895b7fe700 1 -- 172.21.15.114:0/2448683885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895c19fcf0 con 0x7f895c1034d0 2024-05-27T07:38:42.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.361+0000 7f895b7fe700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f895c1034d0 0x7f895c19ae20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f894c00b580 tx=0x7f894c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:42.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.361+0000 7f89597fa700 1 -- 172.21.15.114:0/2448683885 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c023ad0 con 0x7f895c1034d0 2024-05-27T07:38:42.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.361+0000 7f89597fa700 1 -- 172.21.15.114:0/2448683885 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f894c023c70 con 0x7f895c1034d0 2024-05-27T07:38:42.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.361+0000 7f89597fa700 1 -- 172.21.15.114:0/2448683885 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c00f6c0 con 0x7f895c1034d0 2024-05-27T07:38:42.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.361+0000 7f896209a700 1 -- 172.21.15.114:0/2448683885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895c19ff80 con 0x7f895c1034d0 2024-05-27T07:38:42.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.361+0000 7f896209a700 1 -- 172.21.15.114:0/2448683885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895c0ff330 con 0x7f895c1034d0 2024-05-27T07:38:42.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.362+0000 7f896209a700 1 -- 172.21.15.114:0/2448683885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f895c104980 con 0x7f895c1034d0 2024-05-27T07:38:42.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.363+0000 7f89597fa700 1 -- 172.21.15.114:0/2448683885 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f894c003da0 con 0x7f895c1034d0 2024-05-27T07:38:42.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.364+0000 7f89597fa700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f89480b2990 0x7f89480b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:42.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.365+0000 7f8953fff700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f89480b2990 0x7f89480b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:42.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.365+0000 7f89597fa700 1 -- 172.21.15.114:0/2448683885 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f894c0ebdb0 con 0x7f895c1034d0 2024-05-27T07:38:42.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.365+0000 7f8953fff700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f89480b2990 0x7f89480b4ea0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f895c1099f0 tx=0x7f894400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:42.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.369+0000 7f89597fa700 1 -- 172.21.15.114:0/2448683885 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f894c013070 con 0x7f895c1034d0 2024-05-27T07:38:42.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.573+0000 7f896209a700 1 -- 172.21.15.114:0/2448683885 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f895c0512b0 con 0x7f89480b2990 2024-05-27T07:38:42.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.582+0000 7f89597fa700 1 -- 172.21.15.114:0/2448683885 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340557 (secure 0 0 0) 0x7f895c0512b0 con 0x7f89480b2990 2024-05-27T07:38:42.585 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:38:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.586+0000 7f896209a700 1 -- 172.21.15.114:0/2448683885 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f89480b2990 msgr2=0x7f89480b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.586+0000 7f896209a700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f89480b2990 0x7f89480b4ea0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f895c1099f0 tx=0x7f894400b410 comp rx=0 tx=0).stop 2024-05-27T07:38:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.586+0000 7f896209a700 1 -- 172.21.15.114:0/2448683885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f895c1034d0 msgr2=0x7f895c19ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.586+0000 7f896209a700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f895c1034d0 0x7f895c19ae20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f894c00b580 tx=0x7f894c00bd70 comp rx=0 tx=0).stop 2024-05-27T07:38:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.586+0000 7f896209a700 1 -- 172.21.15.114:0/2448683885 shutdown_connections 2024-05-27T07:38:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.586+0000 7f896209a700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f89480b2990 0x7f89480b4ea0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.586+0000 7f896209a700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f895c108750 0x7f895c19f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.586+0000 7f896209a700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f895c1034d0 0x7f895c19ae20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.586+0000 7f896209a700 1 --2- 172.21.15.114:0/2448683885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f895c107e00 0x7f895c19b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.586+0000 7f896209a700 1 -- 172.21.15.114:0/2448683885 >> 172.21.15.114:0/2448683885 conn(0x7f895c0fef20 msgr2=0x7f895c06c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.586+0000 7f896209a700 1 -- 172.21.15.114:0/2448683885 shutdown_connections 2024-05-27T07:38:42.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:42.586+0000 7f896209a700 1 -- 172.21.15.114:0/2448683885 wait complete. 2024-05-27T07:38:42.589 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-05-27T07:38:43.349 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:43 smithi114 ceph-mon[32038]: pgmap v378: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:43.398 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-05-27T07:38:41.412318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656045,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":292,"num_read_kb":241,"num_write":212,"num_write_kb":2246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2951979,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166,"ondisk_log_size":166,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365208,"kb_used_data":11828,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576616,"statfs":{"total":1151940427776,"available":1151566454784,"internally_reserved":0,"allocated":12111872,"data_stored":8126321,"data_compressed":496341,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003848"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.146501+0000","last_change":"2024-05-27T07:36:04.980329+0000","last_active":"2024-05-27T07:37:46.146501+0000","last_peered":"2024-05-27T07:37:46.146501+0000","last_clean":"2024-05-27T07:37:46.146501+0000","last_became_active":"2024-05-27T07:36:04.980006+0000","last_became_peered":"2024-05-27T07:36:04.980006+0000","last_unstale":"2024-05-27T07:37:46.146501+0000","last_undegraded":"2024-05-27T07:37:46.146501+0000","last_fullsized":"2024-05-27T07:37:46.146501+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:51:56.728465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.319578+0000","last_change":"2024-05-27T07:36:04.983429+0000","last_active":"2024-05-27T07:37:46.319578+0000","last_peered":"2024-05-27T07:37:46.319578+0000","last_clean":"2024-05-27T07:37:46.319578+0000","last_became_active":"2024-05-27T07:36:04.983099+0000","last_became_peered":"2024-05-27T07:36:04.983099+0000","last_unstale":"2024-05-27T07:37:46.319578+0000","last_undegraded":"2024-05-27T07:37:46.319578+0000","last_fullsized":"2024-05-27T07:37:46.319578+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:18:51.059717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.319889+0000","last_change":"2024-05-27T07:36:04.993319+0000","last_active":"2024-05-27T07:37:46.319889+0000","last_peered":"2024-05-27T07:37:46.319889+0000","last_clean":"2024-05-27T07:37:46.319889+0000","last_became_active":"2024-05-27T07:36:04.992983+0000","last_became_peered":"2024-05-27T07:36:04.992983+0000","last_unstale":"2024-05-27T07:37:46.319889+0000","last_undegraded":"2024-05-27T07:37:46.319889+0000","last_fullsized":"2024-05-27T07:37:46.319889+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:50.800194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.143342+0000","last_change":"2024-05-27T07:36:45.140980+0000","last_active":"2024-05-27T07:37:46.143342+0000","last_peered":"2024-05-27T07:37:46.143342+0000","last_clean":"2024-05-27T07:37:46.143342+0000","last_became_active":"2024-05-27T07:36:45.140551+0000","last_became_peered":"2024-05-27T07:36:45.140551+0000","last_unstale":"2024-05-27T07:37:46.143342+0000","last_undegraded":"2024-05-27T07:37:46.143342+0000","last_fullsized":"2024-05-27T07:37:46.143342+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:59:21.646306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147616+0000","last_change":"2024-05-27T07:36:04.977404+0000","last_active":"2024-05-27T07:37:46.147616+0000","last_peered":"2024-05-27T07:37:46.147616+0000","last_clean":"2024-05-27T07:37:46.147616+0000","last_became_active":"2024-05-27T07:36:04.976991+0000","last_became_peered":"2024-05-27T07:36:04.976991+0000","last_unstale":"2024-05-27T07:37:46.147616+0000","last_undegraded":"2024-05-27T07:37:46.147616+0000","last_fullsized":"2024-05-27T07:37:46.147616+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:56.532586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141779+0000","last_change":"2024-05-27T07:36:04.982149+0000","last_active":"2024-05-27T07:37:46.141779+0000","last_peered":"2024-05-27T07:37:46.141779+0000","last_clean":"2024-05-27T07:37:46.141779+0000","last_became_active":"2024-05-27T07:36:04.981751+0000","last_became_peered":"2024-05-27T07:36:04.981751+0000","last_unstale":"2024-05-27T07:37:46.141779+0000","last_undegraded":"2024-05-27T07:37:46.141779+0000","last_fullsized":"2024-05-27T07:37:46.141779+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:42.773458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147276+0000","last_change":"2024-05-27T07:36:04.987491+0000","last_active":"2024-05-27T07:37:46.147276+0000","last_peered":"2024-05-27T07:37:46.147276+0000","last_clean":"2024-05-27T07:37:46.147276+0000","last_became_active":"2024-05-27T07:36:04.987059+0000","last_became_peered":"2024-05-27T07:36:04.987059+0000","last_unstale":"2024-05-27T07:37:46.147276+0000","last_undegraded":"2024-05-27T07:37:46.147276+0000","last_fullsized":"2024-05-27T07:37:46.147276+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:57.611253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141048+0000","last_change":"2024-05-27T07:36:45.126961+0000","last_active":"2024-05-27T07:37:46.141048+0000","last_peered":"2024-05-27T07:37:46.141048+0000","last_clean":"2024-05-27T07:37:46.141048+0000","last_became_active":"2024-05-27T07:36:45.126653+0000","last_became_peered":"2024-05-27T07:36:45.126653+0000","last_unstale":"2024-05-27T07:37:46.141048+0000","last_undegraded":"2024-05-27T07:37:46.141048+0000","last_fullsized":"2024-05-27T07:37:46.141048+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:35.586128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315689+0000","last_change":"2024-05-27T07:36:04.990469+0000","last_active":"2024-05-27T07:37:46.315689+0000","last_peered":"2024-05-27T07:37:46.315689+0000","last_clean":"2024-05-27T07:37:46.315689+0000","last_became_active":"2024-05-27T07:36:04.989771+0000","last_became_peered":"2024-05-27T07:36:04.989771+0000","last_unstale":"2024-05-27T07:37:46.315689+0000","last_undegraded":"2024-05-27T07:37:46.315689+0000","last_fullsized":"2024-05-27T07:37:46.315689+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:53.593566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142268+0000","last_change":"2024-05-27T07:36:04.984746+0000","last_active":"2024-05-27T07:37:46.142268+0000","last_peered":"2024-05-27T07:37:46.142268+0000","last_clean":"2024-05-27T07:37:46.142268+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:37:46.142268+0000","last_undegraded":"2024-05-27T07:37:46.142268+0000","last_fullsized":"2024-05-27T07:37:46.142268+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:18.071981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147629+0000","last_change":"2024-05-27T07:36:04.977167+0000","last_active":"2024-05-27T07:37:46.147629+0000","last_peered":"2024-05-27T07:37:46.147629+0000","last_clean":"2024-05-27T07:37:46.147629+0000","last_became_active":"2024-05-27T07:36:04.976788+0000","last_became_peered":"2024-05-27T07:36:04.976788+0000","last_unstale":"2024-05-27T07:37:46.147629+0000","last_undegraded":"2024-05-27T07:37:46.147629+0000","last_fullsized":"2024-05-27T07:37:46.147629+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:59:44.574191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145225+0000","last_change":"2024-05-27T07:36:04.986594+0000","last_active":"2024-05-27T07:37:46.145225+0000","last_peered":"2024-05-27T07:37:46.145225+0000","last_clean":"2024-05-27T07:37:46.145225+0000","last_became_active":"2024-05-27T07:36:04.985161+0000","last_became_peered":"2024-05-27T07:36:04.985161+0000","last_unstale":"2024-05-27T07:37:46.145225+0000","last_undegraded":"2024-05-27T07:37:46.145225+0000","last_fullsized":"2024-05-27T07:37:46.145225+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.239461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.146306+0000","last_change":"2024-05-27T07:36:04.984781+0000","last_active":"2024-05-27T07:37:46.146306+0000","last_peered":"2024-05-27T07:37:46.146306+0000","last_clean":"2024-05-27T07:37:46.146306+0000","last_became_active":"2024-05-27T07:36:04.984462+0000","last_became_peered":"2024-05-27T07:36:04.984462+0000","last_unstale":"2024-05-27T07:37:46.146306+0000","last_undegraded":"2024-05-27T07:37:46.146306+0000","last_fullsized":"2024-05-27T07:37:46.146306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:40.051692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.319281+0000","last_change":"2024-05-27T07:36:04.978360+0000","last_active":"2024-05-27T07:37:46.319281+0000","last_peered":"2024-05-27T07:37:46.319281+0000","last_clean":"2024-05-27T07:37:46.319281+0000","last_became_active":"2024-05-27T07:36:04.978105+0000","last_became_peered":"2024-05-27T07:36:04.978105+0000","last_unstale":"2024-05-27T07:37:46.319281+0000","last_undegraded":"2024-05-27T07:37:46.319281+0000","last_fullsized":"2024-05-27T07:37:46.319281+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:04:47.257529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.316030+0000","last_change":"2024-05-27T07:36:04.986510+0000","last_active":"2024-05-27T07:37:46.316030+0000","last_peered":"2024-05-27T07:37:46.316030+0000","last_clean":"2024-05-27T07:37:46.316030+0000","last_became_active":"2024-05-27T07:36:04.986150+0000","last_became_peered":"2024-05-27T07:36:04.986150+0000","last_unstale":"2024-05-27T07:37:46.316030+0000","last_undegraded":"2024-05-27T07:37:46.316030+0000","last_fullsized":"2024-05-27T07:37:46.316030+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:07.349470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315130+0000","last_change":"2024-05-27T07:36:04.979880+0000","last_active":"2024-05-27T07:37:46.315130+0000","last_peered":"2024-05-27T07:37:46.315130+0000","last_clean":"2024-05-27T07:37:46.315130+0000","last_became_active":"2024-05-27T07:36:04.979516+0000","last_became_peered":"2024-05-27T07:36:04.979516+0000","last_unstale":"2024-05-27T07:37:46.315130+0000","last_undegraded":"2024-05-27T07:37:46.315130+0000","last_fullsized":"2024-05-27T07:37:46.315130+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:41:03.964277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.146340+0000","last_change":"2024-05-27T07:36:04.980231+0000","last_active":"2024-05-27T07:37:46.146340+0000","last_peered":"2024-05-27T07:37:46.146340+0000","last_clean":"2024-05-27T07:37:46.146340+0000","last_became_active":"2024-05-27T07:36:04.979888+0000","last_became_peered":"2024-05-27T07:36:04.979888+0000","last_unstale":"2024-05-27T07:37:46.146340+0000","last_undegraded":"2024-05-27T07:37:46.146340+0000","last_fullsized":"2024-05-27T07:37:46.146340+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.838119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142784+0000","last_change":"2024-05-27T07:36:04.990890+0000","last_active":"2024-05-27T07:37:46.142784+0000","last_peered":"2024-05-27T07:37:46.142784+0000","last_clean":"2024-05-27T07:37:46.142784+0000","last_became_active":"2024-05-27T07:36:04.990581+0000","last_became_peered":"2024-05-27T07:36:04.990581+0000","last_unstale":"2024-05-27T07:37:46.142784+0000","last_undegraded":"2024-05-27T07:37:46.142784+0000","last_fullsized":"2024-05-27T07:37:46.142784+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:33.054395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141353+0000","last_change":"2024-05-27T07:36:04.981620+0000","last_active":"2024-05-27T07:37:46.141353+0000","last_peered":"2024-05-27T07:37:46.141353+0000","last_clean":"2024-05-27T07:37:46.141353+0000","last_became_active":"2024-05-27T07:36:04.981290+0000","last_became_peered":"2024-05-27T07:36:04.981290+0000","last_unstale":"2024-05-27T07:37:46.141353+0000","last_undegraded":"2024-05-27T07:37:46.141353+0000","last_fullsized":"2024-05-27T07:37:46.141353+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:50.287380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.140654+0000","last_change":"2024-05-27T07:36:15.030740+0000","last_active":"2024-05-27T07:37:46.140654+0000","last_peered":"2024-05-27T07:37:46.140654+0000","last_clean":"2024-05-27T07:37:46.140654+0000","last_became_active":"2024-05-27T07:36:15.029738+0000","last_became_peered":"2024-05-27T07:36:15.029738+0000","last_unstale":"2024-05-27T07:37:46.140654+0000","last_undegraded":"2024-05-27T07:37:46.140654+0000","last_fullsized":"2024-05-27T07:37:46.140654+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:18.101941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"78'4","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.143016+0000","last_change":"2024-05-27T07:36:01.955530+0000","last_active":"2024-05-27T07:37:46.143016+0000","last_peered":"2024-05-27T07:37:46.143016+0000","last_clean":"2024-05-27T07:37:46.143016+0000","last_became_active":"2024-05-27T07:36:01.955225+0000","last_became_peered":"2024-05-27T07:36:01.955225+0000","last_unstale":"2024-05-27T07:37:46.143016+0000","last_undegraded":"2024-05-27T07:37:46.143016+0000","last_fullsized":"2024-05-27T07:37:46.143016+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:02.067672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.317760+0000","last_change":"2024-05-27T07:36:04.984397+0000","last_active":"2024-05-27T07:37:46.317760+0000","last_peered":"2024-05-27T07:37:46.317760+0000","last_clean":"2024-05-27T07:37:46.317760+0000","last_became_active":"2024-05-27T07:36:04.984033+0000","last_became_peered":"2024-05-27T07:36:04.984033+0000","last_unstale":"2024-05-27T07:37:46.317760+0000","last_undegraded":"2024-05-27T07:37:46.317760+0000","last_fullsized":"2024-05-27T07:37:46.317760+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:15.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.322662+0000","last_change":"2024-05-27T07:37:46.322144+0000","last_active":"2024-05-27T07:37:46.322662+0000","last_peered":"2024-05-27T07:37:46.322662+0000","last_clean":"2024-05-27T07:37:46.322662+0000","last_became_active":"2024-05-27T07:37:46.321610+0000","last_became_peered":"2024-05-27T07:37:46.321610+0000","last_unstale":"2024-05-27T07:37:46.322662+0000","last_undegraded":"2024-05-27T07:37:46.322662+0000","last_fullsized":"2024-05-27T07:37:46.322662+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:47.119281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"78'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.317953+0000","last_change":"2024-05-27T07:36:02.243079+0000","last_active":"2024-05-27T07:37:46.317953+0000","last_peered":"2024-05-27T07:37:46.317953+0000","last_clean":"2024-05-27T07:37:46.317953+0000","last_became_active":"2024-05-27T07:36:02.242633+0000","last_became_peered":"2024-05-27T07:36:02.242633+0000","last_unstale":"2024-05-27T07:37:46.317953+0000","last_undegraded":"2024-05-27T07:37:46.317953+0000","last_fullsized":"2024-05-27T07:37:46.317953+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:45:33.980842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.319390+0000","last_change":"2024-05-27T07:36:04.989775+0000","last_active":"2024-05-27T07:37:46.319390+0000","last_peered":"2024-05-27T07:37:46.319390+0000","last_clean":"2024-05-27T07:37:46.319390+0000","last_became_active":"2024-05-27T07:36:04.989289+0000","last_became_peered":"2024-05-27T07:36:04.989289+0000","last_unstale":"2024-05-27T07:37:46.319390+0000","last_undegraded":"2024-05-27T07:37:46.319390+0000","last_fullsized":"2024-05-27T07:37:46.319390+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:24:38.355231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144241+0000","last_change":"2024-05-27T07:36:15.031737+0000","last_active":"2024-05-27T07:37:46.144241+0000","last_peered":"2024-05-27T07:37:46.144241+0000","last_clean":"2024-05-27T07:37:46.144241+0000","last_became_active":"2024-05-27T07:36:15.031314+0000","last_became_peered":"2024-05-27T07:36:15.031314+0000","last_unstale":"2024-05-27T07:37:46.144241+0000","last_undegraded":"2024-05-27T07:37:46.144241+0000","last_fullsized":"2024-05-27T07:37:46.144241+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:32.742298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"78'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.148103+0000","last_change":"2024-05-27T07:36:01.957459+0000","last_active":"2024-05-27T07:37:46.148103+0000","last_peered":"2024-05-27T07:37:46.148103+0000","last_clean":"2024-05-27T07:37:46.148103+0000","last_became_active":"2024-05-27T07:36:01.954985+0000","last_became_peered":"2024-05-27T07:36:01.954985+0000","last_unstale":"2024-05-27T07:37:46.148103+0000","last_undegraded":"2024-05-27T07:37:46.148103+0000","last_fullsized":"2024-05-27T07:37:46.148103+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:27.085486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315291+0000","last_change":"2024-05-27T07:36:04.986447+0000","last_active":"2024-05-27T07:37:46.315291+0000","last_peered":"2024-05-27T07:37:46.315291+0000","last_clean":"2024-05-27T07:37:46.315291+0000","last_became_active":"2024-05-27T07:36:04.982099+0000","last_became_peered":"2024-05-27T07:36:04.982099+0000","last_unstale":"2024-05-27T07:37:46.315291+0000","last_undegraded":"2024-05-27T07:37:46.315291+0000","last_fullsized":"2024-05-27T07:37:46.315291+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:32.637479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.143967+0000","last_change":"2024-05-27T07:36:15.024152+0000","last_active":"2024-05-27T07:37:46.143967+0000","last_peered":"2024-05-27T07:37:46.143967+0000","last_clean":"2024-05-27T07:37:46.143967+0000","last_became_active":"2024-05-27T07:36:15.023711+0000","last_became_peered":"2024-05-27T07:36:15.023711+0000","last_unstale":"2024-05-27T07:37:46.143967+0000","last_undegraded":"2024-05-27T07:37:46.143967+0000","last_fullsized":"2024-05-27T07:37:46.143967+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:33.809364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147668+0000","last_change":"2024-05-27T07:36:01.950531+0000","last_active":"2024-05-27T07:37:46.147668+0000","last_peered":"2024-05-27T07:37:46.147668+0000","last_clean":"2024-05-27T07:37:46.147668+0000","last_became_active":"2024-05-27T07:36:01.950015+0000","last_became_peered":"2024-05-27T07:36:01.950015+0000","last_unstale":"2024-05-27T07:37:46.147668+0000","last_undegraded":"2024-05-27T07:37:46.147668+0000","last_fullsized":"2024-05-27T07:37:46.147668+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:22.400415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315953+0000","last_change":"2024-05-27T07:36:04.982792+0000","last_active":"2024-05-27T07:37:46.315953+0000","last_peered":"2024-05-27T07:37:46.315953+0000","last_clean":"2024-05-27T07:37:46.315953+0000","last_became_active":"2024-05-27T07:36:04.982400+0000","last_became_peered":"2024-05-27T07:36:04.982400+0000","last_unstale":"2024-05-27T07:37:46.315953+0000","last_undegraded":"2024-05-27T07:37:46.315953+0000","last_fullsized":"2024-05-27T07:37:46.315953+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:27.823654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141113+0000","last_change":"2024-05-27T07:36:15.027932+0000","last_active":"2024-05-27T07:37:46.141113+0000","last_peered":"2024-05-27T07:37:46.141113+0000","last_clean":"2024-05-27T07:37:46.141113+0000","last_became_active":"2024-05-27T07:36:15.027526+0000","last_became_peered":"2024-05-27T07:36:15.027526+0000","last_unstale":"2024-05-27T07:37:46.141113+0000","last_undegraded":"2024-05-27T07:37:46.141113+0000","last_fullsized":"2024-05-27T07:37:46.141113+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:16.165934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"78'1","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315759+0000","last_change":"2024-05-27T07:36:45.127637+0000","last_active":"2024-05-27T07:37:46.315759+0000","last_peered":"2024-05-27T07:37:46.315759+0000","last_clean":"2024-05-27T07:37:46.315759+0000","last_became_active":"2024-05-27T07:36:45.127249+0000","last_became_peered":"2024-05-27T07:36:45.127249+0000","last_unstale":"2024-05-27T07:37:46.315759+0000","last_undegraded":"2024-05-27T07:37:46.315759+0000","last_fullsized":"2024-05-27T07:37:46.315759+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:52.418069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147641+0000","last_change":"2024-05-27T07:36:04.985135+0000","last_active":"2024-05-27T07:37:46.147641+0000","last_peered":"2024-05-27T07:37:46.147641+0000","last_clean":"2024-05-27T07:37:46.147641+0000","last_became_active":"2024-05-27T07:36:04.983621+0000","last_became_peered":"2024-05-27T07:36:04.983621+0000","last_unstale":"2024-05-27T07:37:46.147641+0000","last_undegraded":"2024-05-27T07:37:46.147641+0000","last_fullsized":"2024-05-27T07:37:46.147641+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:11.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.140896+0000","last_change":"2024-05-27T07:36:15.024558+0000","last_active":"2024-05-27T07:37:46.140896+0000","last_peered":"2024-05-27T07:37:46.140896+0000","last_clean":"2024-05-27T07:37:46.140896+0000","last_became_active":"2024-05-27T07:36:15.024148+0000","last_became_peered":"2024-05-27T07:36:15.024148+0000","last_unstale":"2024-05-27T07:37:46.140896+0000","last_undegraded":"2024-05-27T07:37:46.140896+0000","last_fullsized":"2024-05-27T07:37:46.140896+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:03.648451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"78'1","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141941+0000","last_change":"2024-05-27T07:36:45.129413+0000","last_active":"2024-05-27T07:37:46.141941+0000","last_peered":"2024-05-27T07:37:46.141941+0000","last_clean":"2024-05-27T07:37:46.141941+0000","last_became_active":"2024-05-27T07:36:45.128779+0000","last_became_peered":"2024-05-27T07:36:45.128779+0000","last_unstale":"2024-05-27T07:37:46.141941+0000","last_undegraded":"2024-05-27T07:37:46.141941+0000","last_fullsized":"2024-05-27T07:37:46.141941+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:05.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315924+0000","last_change":"2024-05-27T07:36:04.989878+0000","last_active":"2024-05-27T07:37:46.315924+0000","last_peered":"2024-05-27T07:37:46.315924+0000","last_clean":"2024-05-27T07:37:46.315924+0000","last_became_active":"2024-05-27T07:36:04.989532+0000","last_became_peered":"2024-05-27T07:36:04.989532+0000","last_unstale":"2024-05-27T07:37:46.315924+0000","last_undegraded":"2024-05-27T07:37:46.315924+0000","last_fullsized":"2024-05-27T07:37:46.315924+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:54:53.909592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.322124+0000","last_change":"2024-05-27T07:37:46.321582+0000","last_active":"2024-05-27T07:37:46.322124+0000","last_peered":"2024-05-27T07:37:46.322124+0000","last_clean":"2024-05-27T07:37:46.322124+0000","last_became_active":"2024-05-27T07:37:46.321193+0000","last_became_peered":"2024-05-27T07:37:46.321193+0000","last_unstale":"2024-05-27T07:37:46.322124+0000","last_undegraded":"2024-05-27T07:37:46.322124+0000","last_fullsized":"2024-05-27T07:37:46.322124+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:25:25.767300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144205+0000","last_change":"2024-05-27T07:36:01.949684+0000","last_active":"2024-05-27T07:37:46.144205+0000","last_peered":"2024-05-27T07:37:46.144205+0000","last_clean":"2024-05-27T07:37:46.144205+0000","last_became_active":"2024-05-27T07:36:01.949308+0000","last_became_peered":"2024-05-27T07:36:01.949308+0000","last_unstale":"2024-05-27T07:37:46.144205+0000","last_undegraded":"2024-05-27T07:37:46.144205+0000","last_fullsized":"2024-05-27T07:37:46.144205+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:50:54.859172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.317641+0000","last_change":"2024-05-27T07:36:04.971266+0000","last_active":"2024-05-27T07:37:46.317641+0000","last_peered":"2024-05-27T07:37:46.317641+0000","last_clean":"2024-05-27T07:37:46.317641+0000","last_became_active":"2024-05-27T07:36:04.970831+0000","last_became_peered":"2024-05-27T07:36:04.970831+0000","last_unstale":"2024-05-27T07:37:46.317641+0000","last_undegraded":"2024-05-27T07:37:46.317641+0000","last_fullsized":"2024-05-27T07:37:46.317641+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:37.237637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.317455+0000","last_change":"2024-05-27T07:36:15.024415+0000","last_active":"2024-05-27T07:37:46.317455+0000","last_peered":"2024-05-27T07:37:46.317455+0000","last_clean":"2024-05-27T07:37:46.317455+0000","last_became_active":"2024-05-27T07:36:15.023935+0000","last_became_peered":"2024-05-27T07:36:15.023935+0000","last_unstale":"2024-05-27T07:37:46.317455+0000","last_undegraded":"2024-05-27T07:37:46.317455+0000","last_fullsized":"2024-05-27T07:37:46.317455+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:37:30.386933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315347+0000","last_change":"2024-05-27T07:36:01.952256+0000","last_active":"2024-05-27T07:37:46.315347+0000","last_peered":"2024-05-27T07:37:46.315347+0000","last_clean":"2024-05-27T07:37:46.315347+0000","last_became_active":"2024-05-27T07:36:01.951876+0000","last_became_peered":"2024-05-27T07:36:01.951876+0000","last_unstale":"2024-05-27T07:37:46.315347+0000","last_undegraded":"2024-05-27T07:37:46.315347+0000","last_fullsized":"2024-05-27T07:37:46.315347+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:05.201780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141666+0000","last_change":"2024-05-27T07:36:04.980300+0000","last_active":"2024-05-27T07:37:46.141666+0000","last_peered":"2024-05-27T07:37:46.141666+0000","last_clean":"2024-05-27T07:37:46.141666+0000","last_became_active":"2024-05-27T07:36:04.979954+0000","last_became_peered":"2024-05-27T07:36:04.979954+0000","last_unstale":"2024-05-27T07:37:46.141666+0000","last_undegraded":"2024-05-27T07:37:46.141666+0000","last_fullsized":"2024-05-27T07:37:46.141666+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:15.712773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.318738+0000","last_change":"2024-05-27T07:36:15.026685+0000","last_active":"2024-05-27T07:37:46.318738+0000","last_peered":"2024-05-27T07:37:46.318738+0000","last_clean":"2024-05-27T07:37:46.318738+0000","last_became_active":"2024-05-27T07:36:15.026287+0000","last_became_peered":"2024-05-27T07:36:15.026287+0000","last_unstale":"2024-05-27T07:37:46.318738+0000","last_undegraded":"2024-05-27T07:37:46.318738+0000","last_fullsized":"2024-05-27T07:37:46.318738+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:29:26.394435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142109+0000","last_change":"2024-05-27T07:36:01.956566+0000","last_active":"2024-05-27T07:37:46.142109+0000","last_peered":"2024-05-27T07:37:46.142109+0000","last_clean":"2024-05-27T07:37:46.142109+0000","last_became_active":"2024-05-27T07:36:01.955963+0000","last_became_peered":"2024-05-27T07:36:01.955963+0000","last_unstale":"2024-05-27T07:37:46.142109+0000","last_undegraded":"2024-05-27T07:37:46.142109+0000","last_fullsized":"2024-05-27T07:37:46.142109+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:53.081784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315700+0000","last_change":"2024-05-27T07:36:04.983878+0000","last_active":"2024-05-27T07:37:46.315700+0000","last_peered":"2024-05-27T07:37:46.315700+0000","last_clean":"2024-05-27T07:37:46.315700+0000","last_became_active":"2024-05-27T07:36:04.983339+0000","last_became_peered":"2024-05-27T07:36:04.983339+0000","last_unstale":"2024-05-27T07:37:46.315700+0000","last_undegraded":"2024-05-27T07:37:46.315700+0000","last_fullsized":"2024-05-27T07:37:46.315700+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:57:01.726122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141512+0000","last_change":"2024-05-27T07:36:15.027923+0000","last_active":"2024-05-27T07:37:46.141512+0000","last_peered":"2024-05-27T07:37:46.141512+0000","last_clean":"2024-05-27T07:37:46.141512+0000","last_became_active":"2024-05-27T07:36:15.026012+0000","last_became_peered":"2024-05-27T07:36:15.026012+0000","last_unstale":"2024-05-27T07:37:46.141512+0000","last_undegraded":"2024-05-27T07:37:46.141512+0000","last_fullsized":"2024-05-27T07:37:46.141512+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:27:45.238053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147660+0000","last_change":"2024-05-27T07:36:01.956038+0000","last_active":"2024-05-27T07:37:46.147660+0000","last_peered":"2024-05-27T07:37:46.147660+0000","last_clean":"2024-05-27T07:37:46.147660+0000","last_became_active":"2024-05-27T07:36:01.955639+0000","last_became_peered":"2024-05-27T07:36:01.955639+0000","last_unstale":"2024-05-27T07:37:46.147660+0000","last_undegraded":"2024-05-27T07:37:46.147660+0000","last_fullsized":"2024-05-27T07:37:46.147660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:32.920194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.319284+0000","last_change":"2024-05-27T07:36:04.986676+0000","last_active":"2024-05-27T07:37:46.319284+0000","last_peered":"2024-05-27T07:37:46.319284+0000","last_clean":"2024-05-27T07:37:46.319284+0000","last_became_active":"2024-05-27T07:36:04.986316+0000","last_became_peered":"2024-05-27T07:36:04.986316+0000","last_unstale":"2024-05-27T07:37:46.319284+0000","last_undegraded":"2024-05-27T07:37:46.319284+0000","last_fullsized":"2024-05-27T07:37:46.319284+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:03:50.947781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144481+0000","last_change":"2024-05-27T07:36:15.027643+0000","last_active":"2024-05-27T07:37:46.144481+0000","last_peered":"2024-05-27T07:37:46.144481+0000","last_clean":"2024-05-27T07:37:46.144481+0000","last_became_active":"2024-05-27T07:36:15.027175+0000","last_became_peered":"2024-05-27T07:36:15.027175+0000","last_unstale":"2024-05-27T07:37:46.144481+0000","last_undegraded":"2024-05-27T07:37:46.144481+0000","last_fullsized":"2024-05-27T07:37:46.144481+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:46:32.764271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"78'3","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315206+0000","last_change":"2024-05-27T07:36:02.245746+0000","last_active":"2024-05-27T07:37:46.315206+0000","last_peered":"2024-05-27T07:37:46.315206+0000","last_clean":"2024-05-27T07:37:46.315206+0000","last_became_active":"2024-05-27T07:36:02.245269+0000","last_became_peered":"2024-05-27T07:36:02.245269+0000","last_unstale":"2024-05-27T07:37:46.315206+0000","last_undegraded":"2024-05-27T07:37:46.315206+0000","last_fullsized":"2024-05-27T07:37:46.315206+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:28.202540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315355+0000","last_change":"2024-05-27T07:36:45.129374+0000","last_active":"2024-05-27T07:37:46.315355+0000","last_peered":"2024-05-27T07:37:46.315355+0000","last_clean":"2024-05-27T07:37:46.315355+0000","last_became_active":"2024-05-27T07:36:45.128956+0000","last_became_peered":"2024-05-27T07:36:45.128956+0000","last_unstale":"2024-05-27T07:37:46.315355+0000","last_undegraded":"2024-05-27T07:37:46.315355+0000","last_fullsized":"2024-05-27T07:37:46.315355+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:17.037598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142054+0000","last_change":"2024-05-27T07:36:15.027097+0000","last_active":"2024-05-27T07:37:46.142054+0000","last_peered":"2024-05-27T07:37:46.142054+0000","last_clean":"2024-05-27T07:37:46.142054+0000","last_became_active":"2024-05-27T07:36:15.026704+0000","last_became_peered":"2024-05-27T07:36:15.026704+0000","last_unstale":"2024-05-27T07:37:46.142054+0000","last_undegraded":"2024-05-27T07:37:46.142054+0000","last_fullsized":"2024-05-27T07:37:46.142054+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:35:03.747199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'31","reported_seq":281,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:40.613959+0000","last_change":"2024-05-27T07:36:01.956926+0000","last_active":"2024-05-27T07:38:40.613959+0000","last_peered":"2024-05-27T07:38:40.613959+0000","last_clean":"2024-05-27T07:38:40.613959+0000","last_became_active":"2024-05-27T07:36:01.956515+0000","last_became_peered":"2024-05-27T07:36:01.956515+0000","last_unstale":"2024-05-27T07:38:40.613959+0000","last_undegraded":"2024-05-27T07:38:40.613959+0000","last_fullsized":"2024-05-27T07:38:40.613959+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:46.566489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64645,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":205,"num_read_kb":166,"num_write":31,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147928+0000","last_change":"2024-05-27T07:36:04.976498+0000","last_active":"2024-05-27T07:37:46.147928+0000","last_peered":"2024-05-27T07:37:46.147928+0000","last_clean":"2024-05-27T07:37:46.147928+0000","last_became_active":"2024-05-27T07:36:04.976020+0000","last_became_peered":"2024-05-27T07:36:04.976020+0000","last_unstale":"2024-05-27T07:37:46.147928+0000","last_undegraded":"2024-05-27T07:37:46.147928+0000","last_fullsized":"2024-05-27T07:37:46.147928+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:25.909259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.316205+0000","last_change":"2024-05-27T07:36:15.022700+0000","last_active":"2024-05-27T07:37:46.316205+0000","last_peered":"2024-05-27T07:37:46.316205+0000","last_clean":"2024-05-27T07:37:46.316205+0000","last_became_active":"2024-05-27T07:36:15.022109+0000","last_became_peered":"2024-05-27T07:36:15.022109+0000","last_unstale":"2024-05-27T07:37:46.316205+0000","last_undegraded":"2024-05-27T07:37:46.316205+0000","last_fullsized":"2024-05-27T07:37:46.316205+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:02:43.498049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"78'2","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.140996+0000","last_change":"2024-05-27T07:36:01.953442+0000","last_active":"2024-05-27T07:37:46.140996+0000","last_peered":"2024-05-27T07:37:46.140996+0000","last_clean":"2024-05-27T07:37:46.140996+0000","last_became_active":"2024-05-27T07:36:01.953051+0000","last_became_peered":"2024-05-27T07:36:01.953051+0000","last_unstale":"2024-05-27T07:37:46.140996+0000","last_undegraded":"2024-05-27T07:37:46.140996+0000","last_fullsized":"2024-05-27T07:37:46.140996+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:47:53.070596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144460+0000","last_change":"2024-05-27T07:36:04.976365+0000","last_active":"2024-05-27T07:37:46.144460+0000","last_peered":"2024-05-27T07:37:46.144460+0000","last_clean":"2024-05-27T07:37:46.144460+0000","last_became_active":"2024-05-27T07:36:04.976050+0000","last_became_peered":"2024-05-27T07:36:04.976050+0000","last_unstale":"2024-05-27T07:37:46.144460+0000","last_undegraded":"2024-05-27T07:37:46.144460+0000","last_fullsized":"2024-05-27T07:37:46.144460+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:19.765746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145211+0000","last_change":"2024-05-27T07:36:15.022561+0000","last_active":"2024-05-27T07:37:46.145211+0000","last_peered":"2024-05-27T07:37:46.145211+0000","last_clean":"2024-05-27T07:37:46.145211+0000","last_became_active":"2024-05-27T07:36:15.021974+0000","last_became_peered":"2024-05-27T07:36:15.021974+0000","last_unstale":"2024-05-27T07:37:46.145211+0000","last_undegraded":"2024-05-27T07:37:46.145211+0000","last_fullsized":"2024-05-27T07:37:46.145211+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:31.520969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315818+0000","last_change":"2024-05-27T07:36:01.952625+0000","last_active":"2024-05-27T07:37:46.315818+0000","last_peered":"2024-05-27T07:37:46.315818+0000","last_clean":"2024-05-27T07:37:46.315818+0000","last_became_active":"2024-05-27T07:36:01.952280+0000","last_became_peered":"2024-05-27T07:36:01.952280+0000","last_unstale":"2024-05-27T07:37:46.315818+0000","last_undegraded":"2024-05-27T07:37:46.315818+0000","last_fullsized":"2024-05-27T07:37:46.315818+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:17.548061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147276+0000","last_change":"2024-05-27T07:36:04.984795+0000","last_active":"2024-05-27T07:37:46.147276+0000","last_peered":"2024-05-27T07:37:46.147276+0000","last_clean":"2024-05-27T07:37:46.147276+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:37:46.147276+0000","last_undegraded":"2024-05-27T07:37:46.147276+0000","last_fullsized":"2024-05-27T07:37:46.147276+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:07.207460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142584+0000","last_change":"2024-05-27T07:36:15.033565+0000","last_active":"2024-05-27T07:37:46.142584+0000","last_peered":"2024-05-27T07:37:46.142584+0000","last_clean":"2024-05-27T07:37:46.142584+0000","last_became_active":"2024-05-27T07:36:15.033145+0000","last_became_peered":"2024-05-27T07:36:15.033145+0000","last_unstale":"2024-05-27T07:37:46.142584+0000","last_undegraded":"2024-05-27T07:37:46.142584+0000","last_fullsized":"2024-05-27T07:37:46.142584+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:38:02.163816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144084+0000","last_change":"2024-05-27T07:36:01.952759+0000","last_active":"2024-05-27T07:37:46.144084+0000","last_peered":"2024-05-27T07:37:46.144084+0000","last_clean":"2024-05-27T07:37:46.144084+0000","last_became_active":"2024-05-27T07:36:01.952339+0000","last_became_peered":"2024-05-27T07:36:01.952339+0000","last_unstale":"2024-05-27T07:37:46.144084+0000","last_undegraded":"2024-05-27T07:37:46.144084+0000","last_fullsized":"2024-05-27T07:37:46.144084+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:14.518123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147455+0000","last_change":"2024-05-27T07:36:04.986312+0000","last_active":"2024-05-27T07:37:46.147455+0000","last_peered":"2024-05-27T07:37:46.147455+0000","last_clean":"2024-05-27T07:37:46.147455+0000","last_became_active":"2024-05-27T07:36:04.985944+0000","last_became_peered":"2024-05-27T07:36:04.985944+0000","last_unstale":"2024-05-27T07:37:46.147455+0000","last_undegraded":"2024-05-27T07:37:46.147455+0000","last_fullsized":"2024-05-27T07:37:46.147455+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:04.955449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144991+0000","last_change":"2024-05-27T07:36:15.020930+0000","last_active":"2024-05-27T07:37:46.144991+0000","last_peered":"2024-05-27T07:37:46.144991+0000","last_clean":"2024-05-27T07:37:46.144991+0000","last_became_active":"2024-05-27T07:36:15.020556+0000","last_became_peered":"2024-05-27T07:36:15.020556+0000","last_unstale":"2024-05-27T07:37:46.144991+0000","last_undegraded":"2024-05-27T07:37:46.144991+0000","last_fullsized":"2024-05-27T07:37:46.144991+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:02.673065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"78'8","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147426+0000","last_change":"2024-05-27T07:36:02.243402+0000","last_active":"2024-05-27T07:37:46.147426+0000","last_peered":"2024-05-27T07:37:46.147426+0000","last_clean":"2024-05-27T07:37:46.147426+0000","last_became_active":"2024-05-27T07:36:02.242978+0000","last_became_peered":"2024-05-27T07:36:02.242978+0000","last_unstale":"2024-05-27T07:37:46.147426+0000","last_undegraded":"2024-05-27T07:37:46.147426+0000","last_fullsized":"2024-05-27T07:37:46.147426+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:25:04.289506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315693+0000","last_change":"2024-05-27T07:36:04.989885+0000","last_active":"2024-05-27T07:37:46.315693+0000","last_peered":"2024-05-27T07:37:46.315693+0000","last_clean":"2024-05-27T07:37:46.315693+0000","last_became_active":"2024-05-27T07:36:04.989394+0000","last_became_peered":"2024-05-27T07:36:04.989394+0000","last_unstale":"2024-05-27T07:37:46.315693+0000","last_undegraded":"2024-05-27T07:37:46.315693+0000","last_fullsized":"2024-05-27T07:37:46.315693+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:54.138286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315531+0000","last_change":"2024-05-27T07:36:15.021772+0000","last_active":"2024-05-27T07:37:46.315531+0000","last_peered":"2024-05-27T07:37:46.315531+0000","last_clean":"2024-05-27T07:37:46.315531+0000","last_became_active":"2024-05-27T07:36:15.021352+0000","last_became_peered":"2024-05-27T07:36:15.021352+0000","last_unstale":"2024-05-27T07:37:46.315531+0000","last_undegraded":"2024-05-27T07:37:46.315531+0000","last_fullsized":"2024-05-27T07:37:46.315531+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:23.066662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.146222+0000","last_change":"2024-05-27T07:36:45.131900+0000","last_active":"2024-05-27T07:37:46.146222+0000","last_peered":"2024-05-27T07:37:46.146222+0000","last_clean":"2024-05-27T07:37:46.146222+0000","last_became_active":"2024-05-27T07:36:45.127340+0000","last_became_peered":"2024-05-27T07:36:45.127340+0000","last_unstale":"2024-05-27T07:37:46.146222+0000","last_undegraded":"2024-05-27T07:37:46.146222+0000","last_fullsized":"2024-05-27T07:37:46.146222+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:07:57.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145578+0000","last_change":"2024-05-27T07:36:04.987671+0000","last_active":"2024-05-27T07:37:46.145578+0000","last_peered":"2024-05-27T07:37:46.145578+0000","last_clean":"2024-05-27T07:37:46.145578+0000","last_became_active":"2024-05-27T07:36:04.987317+0000","last_became_peered":"2024-05-27T07:36:04.987317+0000","last_unstale":"2024-05-27T07:37:46.145578+0000","last_undegraded":"2024-05-27T07:37:46.145578+0000","last_fullsized":"2024-05-27T07:37:46.145578+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:35.985968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147631+0000","last_change":"2024-05-27T07:36:15.019967+0000","last_active":"2024-05-27T07:37:46.147631+0000","last_peered":"2024-05-27T07:37:46.147631+0000","last_clean":"2024-05-27T07:37:46.147631+0000","last_became_active":"2024-05-27T07:36:15.019509+0000","last_became_peered":"2024-05-27T07:36:15.019509+0000","last_unstale":"2024-05-27T07:37:46.147631+0000","last_undegraded":"2024-05-27T07:37:46.147631+0000","last_fullsized":"2024-05-27T07:37:46.147631+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:01:56.137335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.143286+0000","last_change":"2024-05-27T07:36:02.243561+0000","last_active":"2024-05-27T07:37:46.143286+0000","last_peered":"2024-05-27T07:37:46.143286+0000","last_clean":"2024-05-27T07:37:46.143286+0000","last_became_active":"2024-05-27T07:36:02.242911+0000","last_became_peered":"2024-05-27T07:36:02.242911+0000","last_unstale":"2024-05-27T07:37:46.143286+0000","last_undegraded":"2024-05-27T07:37:46.143286+0000","last_fullsized":"2024-05-27T07:37:46.143286+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:01:43.915660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141914+0000","last_change":"2024-05-27T07:36:04.971108+0000","last_active":"2024-05-27T07:37:46.141914+0000","last_peered":"2024-05-27T07:37:46.141914+0000","last_clean":"2024-05-27T07:37:46.141914+0000","last_became_active":"2024-05-27T07:36:04.970682+0000","last_became_peered":"2024-05-27T07:36:04.970682+0000","last_unstale":"2024-05-27T07:37:46.141914+0000","last_undegraded":"2024-05-27T07:37:46.141914+0000","last_fullsized":"2024-05-27T07:37:46.141914+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:15:34.940449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144342+0000","last_change":"2024-05-27T07:36:15.021292+0000","last_active":"2024-05-27T07:37:46.144342+0000","last_peered":"2024-05-27T07:37:46.144342+0000","last_clean":"2024-05-27T07:37:46.144342+0000","last_became_active":"2024-05-27T07:36:15.019988+0000","last_became_peered":"2024-05-27T07:36:15.019988+0000","last_unstale":"2024-05-27T07:37:46.144342+0000","last_undegraded":"2024-05-27T07:37:46.144342+0000","last_fullsized":"2024-05-27T07:37:46.144342+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:34:14.325368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.146911+0000","last_change":"2024-05-27T07:36:01.952231+0000","last_active":"2024-05-27T07:37:46.146911+0000","last_peered":"2024-05-27T07:37:46.146911+0000","last_clean":"2024-05-27T07:37:46.146911+0000","last_became_active":"2024-05-27T07:36:01.951818+0000","last_became_peered":"2024-05-27T07:36:01.951818+0000","last_unstale":"2024-05-27T07:37:46.146911+0000","last_undegraded":"2024-05-27T07:37:46.146911+0000","last_fullsized":"2024-05-27T07:37:46.146911+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:52:16.299204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141910+0000","last_change":"2024-05-27T07:36:04.982761+0000","last_active":"2024-05-27T07:37:46.141910+0000","last_peered":"2024-05-27T07:37:46.141910+0000","last_clean":"2024-05-27T07:37:46.141910+0000","last_became_active":"2024-05-27T07:36:04.982303+0000","last_became_peered":"2024-05-27T07:36:04.982303+0000","last_unstale":"2024-05-27T07:37:46.141910+0000","last_undegraded":"2024-05-27T07:37:46.141910+0000","last_fullsized":"2024-05-27T07:37:46.141910+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:13:19.099227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.324049+0000","last_change":"2024-05-27T07:37:46.323619+0000","last_active":"2024-05-27T07:37:46.324049+0000","last_peered":"2024-05-27T07:37:46.324049+0000","last_clean":"2024-05-27T07:37:46.324049+0000","last_became_active":"2024-05-27T07:37:46.322114+0000","last_became_peered":"2024-05-27T07:37:46.322114+0000","last_unstale":"2024-05-27T07:37:46.324049+0000","last_undegraded":"2024-05-27T07:37:46.324049+0000","last_fullsized":"2024-05-27T07:37:46.324049+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:59:08.966863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315614+0000","last_change":"2024-05-27T07:36:45.142023+0000","last_active":"2024-05-27T07:37:46.315614+0000","last_peered":"2024-05-27T07:37:46.315614+0000","last_clean":"2024-05-27T07:37:46.315614+0000","last_became_active":"2024-05-27T07:36:45.141278+0000","last_became_peered":"2024-05-27T07:36:45.141278+0000","last_unstale":"2024-05-27T07:37:46.315614+0000","last_undegraded":"2024-05-27T07:37:46.315614+0000","last_fullsized":"2024-05-27T07:37:46.315614+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:15.800380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144108+0000","last_change":"2024-05-27T07:36:02.243041+0000","last_active":"2024-05-27T07:37:46.144108+0000","last_peered":"2024-05-27T07:37:46.144108+0000","last_clean":"2024-05-27T07:37:46.144108+0000","last_became_active":"2024-05-27T07:36:02.242689+0000","last_became_peered":"2024-05-27T07:36:02.242689+0000","last_unstale":"2024-05-27T07:37:46.144108+0000","last_undegraded":"2024-05-27T07:37:46.144108+0000","last_fullsized":"2024-05-27T07:37:46.144108+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:44:19.652954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147487+0000","last_change":"2024-05-27T07:36:15.025595+0000","last_active":"2024-05-27T07:37:46.147487+0000","last_peered":"2024-05-27T07:37:46.147487+0000","last_clean":"2024-05-27T07:37:46.147487+0000","last_became_active":"2024-05-27T07:36:15.025177+0000","last_became_peered":"2024-05-27T07:36:15.025177+0000","last_unstale":"2024-05-27T07:37:46.147487+0000","last_undegraded":"2024-05-27T07:37:46.147487+0000","last_fullsized":"2024-05-27T07:37:46.147487+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:08.920778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315492+0000","last_change":"2024-05-27T07:36:04.981422+0000","last_active":"2024-05-27T07:37:46.315492+0000","last_peered":"2024-05-27T07:37:46.315492+0000","last_clean":"2024-05-27T07:37:46.315492+0000","last_became_active":"2024-05-27T07:36:04.981100+0000","last_became_peered":"2024-05-27T07:36:04.981100+0000","last_unstale":"2024-05-27T07:37:46.315492+0000","last_undegraded":"2024-05-27T07:37:46.315492+0000","last_fullsized":"2024-05-27T07:37:46.315492+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:01.076309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144544+0000","last_change":"2024-05-27T07:36:01.953727+0000","last_active":"2024-05-27T07:37:46.144544+0000","last_peered":"2024-05-27T07:37:46.144544+0000","last_clean":"2024-05-27T07:37:46.144544+0000","last_became_active":"2024-05-27T07:36:01.953323+0000","last_became_peered":"2024-05-27T07:36:01.953323+0000","last_unstale":"2024-05-27T07:37:46.144544+0000","last_undegraded":"2024-05-27T07:37:46.144544+0000","last_fullsized":"2024-05-27T07:37:46.144544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:17.186854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":119,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147156+0000","last_change":"2024-05-27T07:34:38.289913+0000","last_active":"2024-05-27T07:37:46.147156+0000","last_peered":"2024-05-27T07:37:46.147156+0000","last_clean":"2024-05-27T07:37:46.147156+0000","last_became_active":"2024-05-27T07:34:38.285523+0000","last_became_peered":"2024-05-27T07:34:38.285523+0000","last_unstale":"2024-05-27T07:37:46.147156+0000","last_undegraded":"2024-05-27T07:37:46.147156+0000","last_fullsized":"2024-05-27T07:37:46.147156+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_clean_scrub_stamp":"2024-05-27T07:31:09.455079+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:59.537055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.316050+0000","last_change":"2024-05-27T07:36:15.027453+0000","last_active":"2024-05-27T07:37:46.316050+0000","last_peered":"2024-05-27T07:37:46.316050+0000","last_clean":"2024-05-27T07:37:46.316050+0000","last_became_active":"2024-05-27T07:36:15.026872+0000","last_became_peered":"2024-05-27T07:36:15.026872+0000","last_unstale":"2024-05-27T07:37:46.316050+0000","last_undegraded":"2024-05-27T07:37:46.316050+0000","last_fullsized":"2024-05-27T07:37:46.316050+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:32.351924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144005+0000","last_change":"2024-05-27T07:36:04.982041+0000","last_active":"2024-05-27T07:37:46.144005+0000","last_peered":"2024-05-27T07:37:46.144005+0000","last_clean":"2024-05-27T07:37:46.144005+0000","last_became_active":"2024-05-27T07:36:04.981719+0000","last_became_peered":"2024-05-27T07:36:04.981719+0000","last_unstale":"2024-05-27T07:37:46.144005+0000","last_undegraded":"2024-05-27T07:37:46.144005+0000","last_fullsized":"2024-05-27T07:37:46.144005+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:40.061892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147061+0000","last_change":"2024-05-27T07:36:01.952238+0000","last_active":"2024-05-27T07:37:46.147061+0000","last_peered":"2024-05-27T07:37:46.147061+0000","last_clean":"2024-05-27T07:37:46.147061+0000","last_became_active":"2024-05-27T07:36:01.951853+0000","last_became_peered":"2024-05-27T07:36:01.951853+0000","last_unstale":"2024-05-27T07:37:46.147061+0000","last_undegraded":"2024-05-27T07:37:46.147061+0000","last_fullsized":"2024-05-27T07:37:46.147061+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:46:41.317531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145516+0000","last_change":"2024-05-27T07:36:15.021848+0000","last_active":"2024-05-27T07:37:46.145516+0000","last_peered":"2024-05-27T07:37:46.145516+0000","last_clean":"2024-05-27T07:37:46.145516+0000","last_became_active":"2024-05-27T07:36:15.021371+0000","last_became_peered":"2024-05-27T07:36:15.021371+0000","last_unstale":"2024-05-27T07:37:46.145516+0000","last_undegraded":"2024-05-27T07:37:46.145516+0000","last_fullsized":"2024-05-27T07:37:46.145516+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:32:21.741229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147962+0000","last_change":"2024-05-27T07:36:02.244283+0000","last_active":"2024-05-27T07:37:46.147962+0000","last_peered":"2024-05-27T07:37:46.147962+0000","last_clean":"2024-05-27T07:37:46.147962+0000","last_became_active":"2024-05-27T07:36:02.243785+0000","last_became_peered":"2024-05-27T07:36:02.243785+0000","last_unstale":"2024-05-27T07:37:46.147962+0000","last_undegraded":"2024-05-27T07:37:46.147962+0000","last_fullsized":"2024-05-27T07:37:46.147962+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:38.596510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144024+0000","last_change":"2024-05-27T07:36:04.979903+0000","last_active":"2024-05-27T07:37:46.144024+0000","last_peered":"2024-05-27T07:37:46.144024+0000","last_clean":"2024-05-27T07:37:46.144024+0000","last_became_active":"2024-05-27T07:36:04.979373+0000","last_became_peered":"2024-05-27T07:36:04.979373+0000","last_unstale":"2024-05-27T07:37:46.144024+0000","last_undegraded":"2024-05-27T07:37:46.144024+0000","last_fullsized":"2024-05-27T07:37:46.144024+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:44:33.763230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.317538+0000","last_change":"2024-05-27T07:36:15.023740+0000","last_active":"2024-05-27T07:37:46.317538+0000","last_peered":"2024-05-27T07:37:46.317538+0000","last_clean":"2024-05-27T07:37:46.317538+0000","last_became_active":"2024-05-27T07:36:15.023364+0000","last_became_peered":"2024-05-27T07:36:15.023364+0000","last_unstale":"2024-05-27T07:37:46.317538+0000","last_undegraded":"2024-05-27T07:37:46.317538+0000","last_fullsized":"2024-05-27T07:37:46.317538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:21.399065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'9","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142898+0000","last_change":"2024-05-27T07:36:01.951451+0000","last_active":"2024-05-27T07:37:46.142898+0000","last_peered":"2024-05-27T07:37:46.142898+0000","last_clean":"2024-05-27T07:37:46.142898+0000","last_became_active":"2024-05-27T07:36:01.951037+0000","last_became_peered":"2024-05-27T07:36:01.951037+0000","last_unstale":"2024-05-27T07:37:46.142898+0000","last_undegraded":"2024-05-27T07:37:46.142898+0000","last_fullsized":"2024-05-27T07:37:46.142898+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:44.202474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.316769+0000","last_change":"2024-05-27T07:36:04.975392+0000","last_active":"2024-05-27T07:37:46.316769+0000","last_peered":"2024-05-27T07:37:46.316769+0000","last_clean":"2024-05-27T07:37:46.316769+0000","last_became_active":"2024-05-27T07:36:04.975029+0000","last_became_peered":"2024-05-27T07:36:04.975029+0000","last_unstale":"2024-05-27T07:37:46.316769+0000","last_undegraded":"2024-05-27T07:37:46.316769+0000","last_fullsized":"2024-05-27T07:37:46.316769+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:32.022670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.140730+0000","last_change":"2024-05-27T07:36:15.026401+0000","last_active":"2024-05-27T07:37:46.140730+0000","last_peered":"2024-05-27T07:37:46.140730+0000","last_clean":"2024-05-27T07:37:46.140730+0000","last_became_active":"2024-05-27T07:36:15.025963+0000","last_became_peered":"2024-05-27T07:36:15.025963+0000","last_unstale":"2024-05-27T07:37:46.140730+0000","last_undegraded":"2024-05-27T07:37:46.140730+0000","last_fullsized":"2024-05-27T07:37:46.140730+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:33:47.616099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"78'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144868+0000","last_change":"2024-05-27T07:36:02.228776+0000","last_active":"2024-05-27T07:37:46.144868+0000","last_peered":"2024-05-27T07:37:46.144868+0000","last_clean":"2024-05-27T07:37:46.144868+0000","last_became_active":"2024-05-27T07:36:02.228356+0000","last_became_peered":"2024-05-27T07:36:02.228356+0000","last_unstale":"2024-05-27T07:37:46.144868+0000","last_undegraded":"2024-05-27T07:37:46.144868+0000","last_fullsized":"2024-05-27T07:37:46.144868+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:07.289340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.319674+0000","last_change":"2024-05-27T07:36:04.986917+0000","last_active":"2024-05-27T07:37:46.319674+0000","last_peered":"2024-05-27T07:37:46.319674+0000","last_clean":"2024-05-27T07:37:46.319674+0000","last_became_active":"2024-05-27T07:36:04.986432+0000","last_became_peered":"2024-05-27T07:36:04.986432+0000","last_unstale":"2024-05-27T07:37:46.319674+0000","last_undegraded":"2024-05-27T07:37:46.319674+0000","last_fullsized":"2024-05-27T07:37:46.319674+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:39:15.011058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144572+0000","last_change":"2024-05-27T07:36:15.032312+0000","last_active":"2024-05-27T07:37:46.144572+0000","last_peered":"2024-05-27T07:37:46.144572+0000","last_clean":"2024-05-27T07:37:46.144572+0000","last_became_active":"2024-05-27T07:36:15.031475+0000","last_became_peered":"2024-05-27T07:36:15.031475+0000","last_unstale":"2024-05-27T07:37:46.144572+0000","last_undegraded":"2024-05-27T07:37:46.144572+0000","last_fullsized":"2024-05-27T07:37:46.144572+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:40.099421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.314912+0000","last_change":"2024-05-27T07:36:02.243456+0000","last_active":"2024-05-27T07:37:46.314912+0000","last_peered":"2024-05-27T07:37:46.314912+0000","last_clean":"2024-05-27T07:37:46.314912+0000","last_became_active":"2024-05-27T07:36:02.243164+0000","last_became_peered":"2024-05-27T07:36:02.243164+0000","last_unstale":"2024-05-27T07:37:46.314912+0000","last_undegraded":"2024-05-27T07:37:46.314912+0000","last_fullsized":"2024-05-27T07:37:46.314912+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:33.765834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147453+0000","last_change":"2024-05-27T07:36:04.980192+0000","last_active":"2024-05-27T07:37:46.147453+0000","last_peered":"2024-05-27T07:37:46.147453+0000","last_clean":"2024-05-27T07:37:46.147453+0000","last_became_active":"2024-05-27T07:36:04.979797+0000","last_became_peered":"2024-05-27T07:36:04.979797+0000","last_unstale":"2024-05-27T07:37:46.147453+0000","last_undegraded":"2024-05-27T07:37:46.147453+0000","last_fullsized":"2024-05-27T07:37:46.147453+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:47:37.006746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142748+0000","last_change":"2024-05-27T07:36:15.033153+0000","last_active":"2024-05-27T07:37:46.142748+0000","last_peered":"2024-05-27T07:37:46.142748+0000","last_clean":"2024-05-27T07:37:46.142748+0000","last_became_active":"2024-05-27T07:36:15.032783+0000","last_became_peered":"2024-05-27T07:36:15.032783+0000","last_unstale":"2024-05-27T07:37:46.142748+0000","last_undegraded":"2024-05-27T07:37:46.142748+0000","last_fullsized":"2024-05-27T07:37:46.142748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:31:26.237222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.318840+0000","last_change":"2024-05-27T07:36:01.952754+0000","last_active":"2024-05-27T07:37:46.318840+0000","last_peered":"2024-05-27T07:37:46.318840+0000","last_clean":"2024-05-27T07:37:46.318840+0000","last_became_active":"2024-05-27T07:36:01.952397+0000","last_became_peered":"2024-05-27T07:36:01.952397+0000","last_unstale":"2024-05-27T07:37:46.318840+0000","last_undegraded":"2024-05-27T07:37:46.318840+0000","last_fullsized":"2024-05-27T07:37:46.318840+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:04.350169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145139+0000","last_change":"2024-05-27T07:36:04.979193+0000","last_active":"2024-05-27T07:37:46.145139+0000","last_peered":"2024-05-27T07:37:46.145139+0000","last_clean":"2024-05-27T07:37:46.145139+0000","last_became_active":"2024-05-27T07:36:04.978679+0000","last_became_peered":"2024-05-27T07:36:04.978679+0000","last_unstale":"2024-05-27T07:37:46.145139+0000","last_undegraded":"2024-05-27T07:37:46.145139+0000","last_fullsized":"2024-05-27T07:37:46.145139+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:59.310284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.154240+0000","last_change":"2024-05-27T07:37:46.153707+0000","last_active":"2024-05-27T07:37:46.154240+0000","last_peered":"2024-05-27T07:37:46.154240+0000","last_clean":"2024-05-27T07:37:46.154240+0000","last_became_active":"2024-05-27T07:37:46.148703+0000","last_became_peered":"2024-05-27T07:37:46.148703+0000","last_unstale":"2024-05-27T07:37:46.154240+0000","last_undegraded":"2024-05-27T07:37:46.154240+0000","last_fullsized":"2024-05-27T07:37:46.154240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:50:20.099467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145292+0000","last_change":"2024-05-27T07:36:45.140940+0000","last_active":"2024-05-27T07:37:46.145292+0000","last_peered":"2024-05-27T07:37:46.145292+0000","last_clean":"2024-05-27T07:37:46.145292+0000","last_became_active":"2024-05-27T07:36:45.140511+0000","last_became_peered":"2024-05-27T07:36:45.140511+0000","last_unstale":"2024-05-27T07:37:46.145292+0000","last_undegraded":"2024-05-27T07:37:46.145292+0000","last_fullsized":"2024-05-27T07:37:46.145292+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:43:00.507442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.316147+0000","last_change":"2024-05-27T07:36:04.974709+0000","last_active":"2024-05-27T07:37:46.316147+0000","last_peered":"2024-05-27T07:37:46.316147+0000","last_clean":"2024-05-27T07:37:46.316147+0000","last_became_active":"2024-05-27T07:36:04.974147+0000","last_became_peered":"2024-05-27T07:36:04.974147+0000","last_unstale":"2024-05-27T07:37:46.316147+0000","last_undegraded":"2024-05-27T07:37:46.316147+0000","last_fullsized":"2024-05-27T07:37:46.316147+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:05:11.251124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144424+0000","last_change":"2024-05-27T07:36:15.023187+0000","last_active":"2024-05-27T07:37:46.144424+0000","last_peered":"2024-05-27T07:37:46.144424+0000","last_clean":"2024-05-27T07:37:46.144424+0000","last_became_active":"2024-05-27T07:36:15.022687+0000","last_became_peered":"2024-05-27T07:36:15.022687+0000","last_unstale":"2024-05-27T07:37:46.144424+0000","last_undegraded":"2024-05-27T07:37:46.144424+0000","last_fullsized":"2024-05-27T07:37:46.144424+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:51.713222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"78'2","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147714+0000","last_change":"2024-05-27T07:36:01.954751+0000","last_active":"2024-05-27T07:37:46.147714+0000","last_peered":"2024-05-27T07:37:46.147714+0000","last_clean":"2024-05-27T07:37:46.147714+0000","last_became_active":"2024-05-27T07:36:01.954173+0000","last_became_peered":"2024-05-27T07:36:01.954173+0000","last_unstale":"2024-05-27T07:37:46.147714+0000","last_undegraded":"2024-05-27T07:37:46.147714+0000","last_fullsized":"2024-05-27T07:37:46.147714+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:45:10.504504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147029+0000","last_change":"2024-05-27T07:36:04.980294+0000","last_active":"2024-05-27T07:37:46.147029+0000","last_peered":"2024-05-27T07:37:46.147029+0000","last_clean":"2024-05-27T07:37:46.147029+0000","last_became_active":"2024-05-27T07:36:04.979853+0000","last_became_peered":"2024-05-27T07:36:04.979853+0000","last_unstale":"2024-05-27T07:37:46.147029+0000","last_undegraded":"2024-05-27T07:37:46.147029+0000","last_fullsized":"2024-05-27T07:37:46.147029+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:43.131216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315163+0000","last_change":"2024-05-27T07:36:15.025854+0000","last_active":"2024-05-27T07:37:46.315163+0000","last_peered":"2024-05-27T07:37:46.315163+0000","last_clean":"2024-05-27T07:37:46.315163+0000","last_became_active":"2024-05-27T07:36:15.025412+0000","last_became_peered":"2024-05-27T07:36:15.025412+0000","last_unstale":"2024-05-27T07:37:46.315163+0000","last_undegraded":"2024-05-27T07:37:46.315163+0000","last_fullsized":"2024-05-27T07:37:46.315163+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:49.904811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141154+0000","last_change":"2024-05-27T07:36:02.243764+0000","last_active":"2024-05-27T07:37:46.141154+0000","last_peered":"2024-05-27T07:37:46.141154+0000","last_clean":"2024-05-27T07:37:46.141154+0000","last_became_active":"2024-05-27T07:36:02.242935+0000","last_became_peered":"2024-05-27T07:36:02.242935+0000","last_unstale":"2024-05-27T07:37:46.141154+0000","last_undegraded":"2024-05-27T07:37:46.141154+0000","last_fullsized":"2024-05-27T07:37:46.141154+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:27.535658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147851+0000","last_change":"2024-05-27T07:36:04.975291+0000","last_active":"2024-05-27T07:37:46.147851+0000","last_peered":"2024-05-27T07:37:46.147851+0000","last_clean":"2024-05-27T07:37:46.147851+0000","last_became_active":"2024-05-27T07:36:04.974868+0000","last_became_peered":"2024-05-27T07:36:04.974868+0000","last_unstale":"2024-05-27T07:37:46.147851+0000","last_undegraded":"2024-05-27T07:37:46.147851+0000","last_fullsized":"2024-05-27T07:37:46.147851+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:13.890494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.319226+0000","last_change":"2024-05-27T07:36:15.031812+0000","last_active":"2024-05-27T07:37:46.319226+0000","last_peered":"2024-05-27T07:37:46.319226+0000","last_clean":"2024-05-27T07:37:46.319226+0000","last_became_active":"2024-05-27T07:36:15.031399+0000","last_became_peered":"2024-05-27T07:36:15.031399+0000","last_unstale":"2024-05-27T07:37:46.319226+0000","last_undegraded":"2024-05-27T07:37:46.319226+0000","last_fullsized":"2024-05-27T07:37:46.319226+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.595579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315962+0000","last_change":"2024-05-27T07:36:01.954581+0000","last_active":"2024-05-27T07:37:46.315962+0000","last_peered":"2024-05-27T07:37:46.315962+0000","last_clean":"2024-05-27T07:37:46.315962+0000","last_became_active":"2024-05-27T07:36:01.953564+0000","last_became_peered":"2024-05-27T07:36:01.953564+0000","last_unstale":"2024-05-27T07:37:46.315962+0000","last_undegraded":"2024-05-27T07:37:46.315962+0000","last_fullsized":"2024-05-27T07:37:46.315962+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:56.866738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147193+0000","last_change":"2024-05-27T07:36:45.141160+0000","last_active":"2024-05-27T07:37:46.147193+0000","last_peered":"2024-05-27T07:37:46.147193+0000","last_clean":"2024-05-27T07:37:46.147193+0000","last_became_active":"2024-05-27T07:36:45.140840+0000","last_became_peered":"2024-05-27T07:36:45.140840+0000","last_unstale":"2024-05-27T07:37:46.147193+0000","last_undegraded":"2024-05-27T07:37:46.147193+0000","last_fullsized":"2024-05-27T07:37:46.147193+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:31.676808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315220+0000","last_change":"2024-05-27T07:36:15.026483+0000","last_active":"2024-05-27T07:37:46.315220+0000","last_peered":"2024-05-27T07:37:46.315220+0000","last_clean":"2024-05-27T07:37:46.315220+0000","last_became_active":"2024-05-27T07:36:15.025955+0000","last_became_peered":"2024-05-27T07:36:15.025955+0000","last_unstale":"2024-05-27T07:37:46.315220+0000","last_undegraded":"2024-05-27T07:37:46.315220+0000","last_fullsized":"2024-05-27T07:37:46.315220+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:25.468146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145095+0000","last_change":"2024-05-27T07:36:01.952505+0000","last_active":"2024-05-27T07:37:46.145095+0000","last_peered":"2024-05-27T07:37:46.145095+0000","last_clean":"2024-05-27T07:37:46.145095+0000","last_became_active":"2024-05-27T07:36:01.952088+0000","last_became_peered":"2024-05-27T07:36:01.952088+0000","last_unstale":"2024-05-27T07:37:46.145095+0000","last_undegraded":"2024-05-27T07:37:46.145095+0000","last_fullsized":"2024-05-27T07:37:46.145095+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:09.235719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.148086+0000","last_change":"2024-05-27T07:36:04.974331+0000","last_active":"2024-05-27T07:37:46.148086+0000","last_peered":"2024-05-27T07:37:46.148086+0000","last_clean":"2024-05-27T07:37:46.148086+0000","last_became_active":"2024-05-27T07:36:04.973825+0000","last_became_peered":"2024-05-27T07:36:04.973825+0000","last_unstale":"2024-05-27T07:37:46.148086+0000","last_undegraded":"2024-05-27T07:37:46.148086+0000","last_fullsized":"2024-05-27T07:37:46.148086+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:39:42.917309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145188+0000","last_change":"2024-05-27T07:36:45.141224+0000","last_active":"2024-05-27T07:37:46.145188+0000","last_peered":"2024-05-27T07:37:46.145188+0000","last_clean":"2024-05-27T07:37:46.145188+0000","last_became_active":"2024-05-27T07:36:45.140776+0000","last_became_peered":"2024-05-27T07:36:45.140776+0000","last_unstale":"2024-05-27T07:37:46.145188+0000","last_undegraded":"2024-05-27T07:37:46.145188+0000","last_fullsized":"2024-05-27T07:37:46.145188+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:36:59.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141166+0000","last_change":"2024-05-27T07:36:04.988247+0000","last_active":"2024-05-27T07:37:46.141166+0000","last_peered":"2024-05-27T07:37:46.141166+0000","last_clean":"2024-05-27T07:37:46.141166+0000","last_became_active":"2024-05-27T07:36:04.987825+0000","last_became_peered":"2024-05-27T07:36:04.987825+0000","last_unstale":"2024-05-27T07:37:46.141166+0000","last_undegraded":"2024-05-27T07:37:46.141166+0000","last_fullsized":"2024-05-27T07:37:46.141166+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:32.499260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141212+0000","last_change":"2024-05-27T07:36:04.979395+0000","last_active":"2024-05-27T07:37:46.141212+0000","last_peered":"2024-05-27T07:37:46.141212+0000","last_clean":"2024-05-27T07:37:46.141212+0000","last_became_active":"2024-05-27T07:36:04.978983+0000","last_became_peered":"2024-05-27T07:36:04.978983+0000","last_unstale":"2024-05-27T07:37:46.141212+0000","last_undegraded":"2024-05-27T07:37:46.141212+0000","last_fullsized":"2024-05-27T07:37:46.141212+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:53:54.071958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.319142+0000","last_change":"2024-05-27T07:36:04.989442+0000","last_active":"2024-05-27T07:37:46.319142+0000","last_peered":"2024-05-27T07:37:46.319142+0000","last_clean":"2024-05-27T07:37:46.319142+0000","last_became_active":"2024-05-27T07:36:04.989079+0000","last_became_peered":"2024-05-27T07:36:04.989079+0000","last_unstale":"2024-05-27T07:37:46.319142+0000","last_undegraded":"2024-05-27T07:37:46.319142+0000","last_fullsized":"2024-05-27T07:37:46.319142+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:39.411255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144128+0000","last_change":"2024-05-27T07:36:04.981289+0000","last_active":"2024-05-27T07:37:46.144128+0000","last_peered":"2024-05-27T07:37:46.144128+0000","last_clean":"2024-05-27T07:37:46.144128+0000","last_became_active":"2024-05-27T07:36:04.980924+0000","last_became_peered":"2024-05-27T07:36:04.980924+0000","last_unstale":"2024-05-27T07:37:46.144128+0000","last_undegraded":"2024-05-27T07:37:46.144128+0000","last_fullsized":"2024-05-27T07:37:46.144128+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:15:04.741303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145578+0000","last_change":"2024-05-27T07:36:04.985712+0000","last_active":"2024-05-27T07:37:46.145578+0000","last_peered":"2024-05-27T07:37:46.145578+0000","last_clean":"2024-05-27T07:37:46.145578+0000","last_became_active":"2024-05-27T07:36:04.983786+0000","last_became_peered":"2024-05-27T07:36:04.983786+0000","last_unstale":"2024-05-27T07:37:46.145578+0000","last_undegraded":"2024-05-27T07:37:46.145578+0000","last_fullsized":"2024-05-27T07:37:46.145578+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:34.549557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141418+0000","last_change":"2024-05-27T07:36:04.991176+0000","last_active":"2024-05-27T07:37:46.141418+0000","last_peered":"2024-05-27T07:37:46.141418+0000","last_clean":"2024-05-27T07:37:46.141418+0000","last_became_active":"2024-05-27T07:36:04.990774+0000","last_became_peered":"2024-05-27T07:36:04.990774+0000","last_unstale":"2024-05-27T07:37:46.141418+0000","last_undegraded":"2024-05-27T07:37:46.141418+0000","last_fullsized":"2024-05-27T07:37:46.141418+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:37.307581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.316933+0000","last_change":"2024-05-27T07:36:04.982337+0000","last_active":"2024-05-27T07:37:46.316933+0000","last_peered":"2024-05-27T07:37:46.316933+0000","last_clean":"2024-05-27T07:37:46.316933+0000","last_became_active":"2024-05-27T07:36:04.981988+0000","last_became_peered":"2024-05-27T07:36:04.981988+0000","last_unstale":"2024-05-27T07:37:46.316933+0000","last_undegraded":"2024-05-27T07:37:46.316933+0000","last_fullsized":"2024-05-27T07:37:46.316933+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:17:39.763459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315337+0000","last_change":"2024-05-27T07:36:04.982321+0000","last_active":"2024-05-27T07:37:46.315337+0000","last_peered":"2024-05-27T07:37:46.315337+0000","last_clean":"2024-05-27T07:37:46.315337+0000","last_became_active":"2024-05-27T07:36:04.981690+0000","last_became_peered":"2024-05-27T07:36:04.981690+0000","last_unstale":"2024-05-27T07:37:46.315337+0000","last_undegraded":"2024-05-27T07:37:46.315337+0000","last_fullsized":"2024-05-27T07:37:46.315337+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:10.181606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.318945+0000","last_change":"2024-05-27T07:36:04.979411+0000","last_active":"2024-05-27T07:37:46.318945+0000","last_peered":"2024-05-27T07:37:46.318945+0000","last_clean":"2024-05-27T07:37:46.318945+0000","last_became_active":"2024-05-27T07:36:04.979041+0000","last_became_peered":"2024-05-27T07:36:04.979041+0000","last_unstale":"2024-05-27T07:37:46.318945+0000","last_undegraded":"2024-05-27T07:37:46.318945+0000","last_fullsized":"2024-05-27T07:37:46.318945+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:57:29.165997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315486+0000","last_change":"2024-05-27T07:36:04.988564+0000","last_active":"2024-05-27T07:37:46.315486+0000","last_peered":"2024-05-27T07:37:46.315486+0000","last_clean":"2024-05-27T07:37:46.315486+0000","last_became_active":"2024-05-27T07:36:04.986313+0000","last_became_peered":"2024-05-27T07:36:04.986313+0000","last_unstale":"2024-05-27T07:37:46.315486+0000","last_undegraded":"2024-05-27T07:37:46.315486+0000","last_fullsized":"2024-05-27T07:37:46.315486+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:38.809386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.146131+0000","last_change":"2024-05-27T07:36:04.974704+0000","last_active":"2024-05-27T07:37:46.146131+0000","last_peered":"2024-05-27T07:37:46.146131+0000","last_clean":"2024-05-27T07:37:46.146131+0000","last_became_active":"2024-05-27T07:36:04.974311+0000","last_became_peered":"2024-05-27T07:36:04.974311+0000","last_unstale":"2024-05-27T07:37:46.146131+0000","last_undegraded":"2024-05-27T07:37:46.146131+0000","last_fullsized":"2024-05-27T07:37:46.146131+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:43.021149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145443+0000","last_change":"2024-05-27T07:36:04.983951+0000","last_active":"2024-05-27T07:37:46.145443+0000","last_peered":"2024-05-27T07:37:46.145443+0000","last_clean":"2024-05-27T07:37:46.145443+0000","last_became_active":"2024-05-27T07:36:04.983489+0000","last_became_peered":"2024-05-27T07:36:04.983489+0000","last_unstale":"2024-05-27T07:37:46.145443+0000","last_undegraded":"2024-05-27T07:37:46.145443+0000","last_fullsized":"2024-05-27T07:37:46.145443+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:04.229523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280976,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65443,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":211,"num_read_kb":172,"num_write":87,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":196329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710767,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":978944,"data_stored":514628,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906996,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":1169813,"data_compressed":68160,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070521,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":515188,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266750,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":515188,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462979,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":1109652,"data_compressed":68160,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626506,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":510510,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":575155,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738472,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":510600,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":575189,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182489,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":510614,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986260,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":514628,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822734,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":1105156,"data_compressed":68160,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34107,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34107,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34107,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:38:43.408 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:38:43.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:43 smithi163 ceph-mon[36451]: pgmap v378: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:43.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:43 smithi186 ceph-mon[36647]: pgmap v378: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:38:43.647 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:44.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:44 smithi163 ceph-mon[36451]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:38:44.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:44 smithi186 ceph-mon[36647]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:38:44.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:44 smithi114 ceph-mon[32038]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:38:45.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:45 smithi114 ceph-mon[32038]: pgmap v379: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:45.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:45 smithi114 ceph-mon[32038]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-05-27T07:38:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.289+0000 7fccdda01700 1 -- 172.21.15.114:0/2753025714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd8105c90 msgr2=0x7fccd810e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.289+0000 7fccdda01700 1 --2- 172.21.15.114:0/2753025714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd8105c90 0x7fccd810e1e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fccc8009990 tx=0x7fccc8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:38:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.289+0000 7fccdda01700 1 -- 172.21.15.114:0/2753025714 shutdown_connections 2024-05-27T07:38:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.289+0000 7fccdda01700 1 --2- 172.21.15.114:0/2753025714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fccd810e720 0x7fccd8110c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.289+0000 7fccdda01700 1 --2- 172.21.15.114:0/2753025714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccd8105340 0x7fccd8105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.289+0000 7fccdda01700 1 --2- 172.21.15.114:0/2753025714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd8105c90 0x7fccd810e1e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.289+0000 7fccdda01700 1 -- 172.21.15.114:0/2753025714 >> 172.21.15.114:0/2753025714 conn(0x7fccd8078ae0 msgr2=0x7fccd8078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:45.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.289+0000 7fccdda01700 1 -- 172.21.15.114:0/2753025714 shutdown_connections 2024-05-27T07:38:45.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.289+0000 7fccdda01700 1 -- 172.21.15.114:0/2753025714 wait complete. 2024-05-27T07:38:45.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.291+0000 7fccdda01700 1 Processor -- start 2024-05-27T07:38:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.291+0000 7fccdda01700 1 -- start start 2024-05-27T07:38:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccdda01700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccd8105340 0x7fccd81a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccdda01700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fccd8105c90 0x7fccd81a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccdda01700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd810e720 0x7fccd819aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccdda01700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fccd8114280 con 0x7fccd810e720 2024-05-27T07:38:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccdda01700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fccd8114100 con 0x7fccd8105c90 2024-05-27T07:38:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccdda01700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fccd8114400 con 0x7fccd8105340 2024-05-27T07:38:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccd77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd810e720 0x7fccd819aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccd77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd810e720 0x7fccd819aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57466/0 (socket says 172.21.15.114:57466) 2024-05-27T07:38:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccd77fe700 1 -- 172.21.15.114:0/2896835496 learned_addr learned my addr 172.21.15.114:0/2896835496 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccd6ffd700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccd8105340 0x7fccd81a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccd67fc700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fccd8105c90 0x7fccd81a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccd77fe700 1 -- 172.21.15.114:0/2896835496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccd8105340 msgr2=0x7fccd81a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccd77fe700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccd8105340 0x7fccd81a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccd77fe700 1 -- 172.21.15.114:0/2896835496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fccd8105c90 msgr2=0x7fccd81a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccd77fe700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fccd8105c90 0x7fccd81a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccd77fe700 1 -- 172.21.15.114:0/2896835496 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd819b0e0 con 0x7fccd810e720 2024-05-27T07:38:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.292+0000 7fccd6ffd700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccd8105340 0x7fccd81a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:38:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.293+0000 7fccd77fe700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd810e720 0x7fccd819aaa0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fccd000ada0 tx=0x7fccd000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.293+0000 7fccdc9ff700 1 -- 172.21.15.114:0/2896835496 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd0017020 con 0x7fccd810e720 2024-05-27T07:38:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.293+0000 7fccdc9ff700 1 -- 172.21.15.114:0/2896835496 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccd000cd80 con 0x7fccd810e720 2024-05-27T07:38:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.293+0000 7fccdda01700 1 -- 172.21.15.114:0/2896835496 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccd819b3d0 con 0x7fccd810e720 2024-05-27T07:38:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.293+0000 7fccdc9ff700 1 -- 172.21.15.114:0/2896835496 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd0005260 con 0x7fccd810e720 2024-05-27T07:38:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.293+0000 7fccdda01700 1 -- 172.21.15.114:0/2896835496 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccd819b890 con 0x7fccd810e720 2024-05-27T07:38:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.294+0000 7fccdda01700 1 -- 172.21.15.114:0/2896835496 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccd806be40 con 0x7fccd810e720 2024-05-27T07:38:45.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.296+0000 7fccdc9ff700 1 -- 172.21.15.114:0/2896835496 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fccd000a090 con 0x7fccd810e720 2024-05-27T07:38:45.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.297+0000 7fccdc9ff700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fccc00b2880 0x7fccc00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:45.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.297+0000 7fccd6ffd700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fccc00b2880 0x7fccc00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:45.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.297+0000 7fccdc9ff700 1 -- 172.21.15.114:0/2896835496 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fccd00e2240 con 0x7fccd810e720 2024-05-27T07:38:45.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.297+0000 7fccd6ffd700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fccc00b2880 0x7fccc00b4d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fccd819bbe0 tx=0x7fcccc00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:45.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.300+0000 7fccdc9ff700 1 -- 172.21.15.114:0/2896835496 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fccd0021020 con 0x7fccd810e720 2024-05-27T07:38:45.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:45 smithi163 ceph-mon[36451]: pgmap v379: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:45.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:45 smithi163 ceph-mon[36451]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-05-27T07:38:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:45 smithi186 ceph-mon[36647]: pgmap v379: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:45.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:45 smithi186 ceph-mon[36647]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-05-27T07:38:45.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.514+0000 7fccdda01700 1 -- 172.21.15.114:0/2896835496 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fccd819bfd0 con 0x7fccc00b2880 2024-05-27T07:38:45.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.523+0000 7fccdc9ff700 1 -- 172.21.15.114:0/2896835496 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340701 (secure 0 0 0) 0x7fccd819bfd0 con 0x7fccc00b2880 2024-05-27T07:38:45.527 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:38:45.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.527+0000 7fccdda01700 1 -- 172.21.15.114:0/2896835496 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fccc00b2880 msgr2=0x7fccc00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:45.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.527+0000 7fccdda01700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fccc00b2880 0x7fccc00b4d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fccd819bbe0 tx=0x7fcccc00a380 comp rx=0 tx=0).stop 2024-05-27T07:38:45.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.527+0000 7fccdda01700 1 -- 172.21.15.114:0/2896835496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd810e720 msgr2=0x7fccd819aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:45.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.527+0000 7fccdda01700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd810e720 0x7fccd819aaa0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fccd000ada0 tx=0x7fccd000c330 comp rx=0 tx=0).stop 2024-05-27T07:38:45.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.527+0000 7fccdda01700 1 -- 172.21.15.114:0/2896835496 shutdown_connections 2024-05-27T07:38:45.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.527+0000 7fccdda01700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd810e720 0x7fccd819aaa0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:45.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.527+0000 7fccdda01700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fccc00b2880 0x7fccc00b4d90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:45.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.527+0000 7fccdda01700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccd8105340 0x7fccd81a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:45.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.527+0000 7fccdda01700 1 --2- 172.21.15.114:0/2896835496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fccd8105c90 0x7fccd81a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:45.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.527+0000 7fccdda01700 1 -- 172.21.15.114:0/2896835496 >> 172.21.15.114:0/2896835496 conn(0x7fccd8078ae0 msgr2=0x7fccd80fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:45.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.527+0000 7fccdda01700 1 -- 172.21.15.114:0/2896835496 shutdown_connections 2024-05-27T07:38:45.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:45.527+0000 7fccdda01700 1 -- 172.21.15.114:0/2896835496 wait complete. 2024-05-27T07:38:45.532 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-05-27T07:38:46.303 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-05-27T07:38:45.413595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656045,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":292,"num_read_kb":241,"num_write":212,"num_write_kb":2246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2951979,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365180,"kb_used_data":11828,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576644,"statfs":{"total":1151940427776,"available":1151566483456,"internally_reserved":0,"allocated":12111872,"data_stored":8126570,"data_compressed":496341,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003978"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.402099+0000","last_change":"2024-05-27T07:36:04.980329+0000","last_active":"2024-05-27T07:38:43.402099+0000","last_peered":"2024-05-27T07:38:43.402099+0000","last_clean":"2024-05-27T07:38:43.402099+0000","last_became_active":"2024-05-27T07:36:04.980006+0000","last_became_peered":"2024-05-27T07:36:04.980006+0000","last_unstale":"2024-05-27T07:38:43.402099+0000","last_undegraded":"2024-05-27T07:38:43.402099+0000","last_fullsized":"2024-05-27T07:38:43.402099+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:51:56.728465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.393544+0000","last_change":"2024-05-27T07:36:04.983429+0000","last_active":"2024-05-27T07:38:43.393544+0000","last_peered":"2024-05-27T07:38:43.393544+0000","last_clean":"2024-05-27T07:38:43.393544+0000","last_became_active":"2024-05-27T07:36:04.983099+0000","last_became_peered":"2024-05-27T07:36:04.983099+0000","last_unstale":"2024-05-27T07:38:43.393544+0000","last_undegraded":"2024-05-27T07:38:43.393544+0000","last_fullsized":"2024-05-27T07:38:43.393544+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:18:51.059717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.402949+0000","last_change":"2024-05-27T07:36:04.993319+0000","last_active":"2024-05-27T07:38:43.402949+0000","last_peered":"2024-05-27T07:38:43.402949+0000","last_clean":"2024-05-27T07:38:43.402949+0000","last_became_active":"2024-05-27T07:36:04.992983+0000","last_became_peered":"2024-05-27T07:36:04.992983+0000","last_unstale":"2024-05-27T07:38:43.402949+0000","last_undegraded":"2024-05-27T07:38:43.402949+0000","last_fullsized":"2024-05-27T07:38:43.402949+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:50.800194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.143342+0000","last_change":"2024-05-27T07:36:45.140980+0000","last_active":"2024-05-27T07:37:46.143342+0000","last_peered":"2024-05-27T07:37:46.143342+0000","last_clean":"2024-05-27T07:37:46.143342+0000","last_became_active":"2024-05-27T07:36:45.140551+0000","last_became_peered":"2024-05-27T07:36:45.140551+0000","last_unstale":"2024-05-27T07:37:46.143342+0000","last_undegraded":"2024-05-27T07:37:46.143342+0000","last_fullsized":"2024-05-27T07:37:46.143342+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:59:21.646306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147616+0000","last_change":"2024-05-27T07:36:04.977404+0000","last_active":"2024-05-27T07:37:46.147616+0000","last_peered":"2024-05-27T07:37:46.147616+0000","last_clean":"2024-05-27T07:37:46.147616+0000","last_became_active":"2024-05-27T07:36:04.976991+0000","last_became_peered":"2024-05-27T07:36:04.976991+0000","last_unstale":"2024-05-27T07:37:46.147616+0000","last_undegraded":"2024-05-27T07:37:46.147616+0000","last_fullsized":"2024-05-27T07:37:46.147616+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:56.532586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141779+0000","last_change":"2024-05-27T07:36:04.982149+0000","last_active":"2024-05-27T07:37:46.141779+0000","last_peered":"2024-05-27T07:37:46.141779+0000","last_clean":"2024-05-27T07:37:46.141779+0000","last_became_active":"2024-05-27T07:36:04.981751+0000","last_became_peered":"2024-05-27T07:36:04.981751+0000","last_unstale":"2024-05-27T07:37:46.141779+0000","last_undegraded":"2024-05-27T07:37:46.141779+0000","last_fullsized":"2024-05-27T07:37:46.141779+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:42.773458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147276+0000","last_change":"2024-05-27T07:36:04.987491+0000","last_active":"2024-05-27T07:37:46.147276+0000","last_peered":"2024-05-27T07:37:46.147276+0000","last_clean":"2024-05-27T07:37:46.147276+0000","last_became_active":"2024-05-27T07:36:04.987059+0000","last_became_peered":"2024-05-27T07:36:04.987059+0000","last_unstale":"2024-05-27T07:37:46.147276+0000","last_undegraded":"2024-05-27T07:37:46.147276+0000","last_fullsized":"2024-05-27T07:37:46.147276+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:57.611253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141048+0000","last_change":"2024-05-27T07:36:45.126961+0000","last_active":"2024-05-27T07:37:46.141048+0000","last_peered":"2024-05-27T07:37:46.141048+0000","last_clean":"2024-05-27T07:37:46.141048+0000","last_became_active":"2024-05-27T07:36:45.126653+0000","last_became_peered":"2024-05-27T07:36:45.126653+0000","last_unstale":"2024-05-27T07:37:46.141048+0000","last_undegraded":"2024-05-27T07:37:46.141048+0000","last_fullsized":"2024-05-27T07:37:46.141048+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:35.586128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.403306+0000","last_change":"2024-05-27T07:36:04.990469+0000","last_active":"2024-05-27T07:38:43.403306+0000","last_peered":"2024-05-27T07:38:43.403306+0000","last_clean":"2024-05-27T07:38:43.403306+0000","last_became_active":"2024-05-27T07:36:04.989771+0000","last_became_peered":"2024-05-27T07:36:04.989771+0000","last_unstale":"2024-05-27T07:38:43.403306+0000","last_undegraded":"2024-05-27T07:38:43.403306+0000","last_fullsized":"2024-05-27T07:38:43.403306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:53.593566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142268+0000","last_change":"2024-05-27T07:36:04.984746+0000","last_active":"2024-05-27T07:37:46.142268+0000","last_peered":"2024-05-27T07:37:46.142268+0000","last_clean":"2024-05-27T07:37:46.142268+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:37:46.142268+0000","last_undegraded":"2024-05-27T07:37:46.142268+0000","last_fullsized":"2024-05-27T07:37:46.142268+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:18.071981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147629+0000","last_change":"2024-05-27T07:36:04.977167+0000","last_active":"2024-05-27T07:37:46.147629+0000","last_peered":"2024-05-27T07:37:46.147629+0000","last_clean":"2024-05-27T07:37:46.147629+0000","last_became_active":"2024-05-27T07:36:04.976788+0000","last_became_peered":"2024-05-27T07:36:04.976788+0000","last_unstale":"2024-05-27T07:37:46.147629+0000","last_undegraded":"2024-05-27T07:37:46.147629+0000","last_fullsized":"2024-05-27T07:37:46.147629+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:59:44.574191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.394357+0000","last_change":"2024-05-27T07:36:04.986594+0000","last_active":"2024-05-27T07:38:43.394357+0000","last_peered":"2024-05-27T07:38:43.394357+0000","last_clean":"2024-05-27T07:38:43.394357+0000","last_became_active":"2024-05-27T07:36:04.985161+0000","last_became_peered":"2024-05-27T07:36:04.985161+0000","last_unstale":"2024-05-27T07:38:43.394357+0000","last_undegraded":"2024-05-27T07:38:43.394357+0000","last_fullsized":"2024-05-27T07:38:43.394357+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.239461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.146306+0000","last_change":"2024-05-27T07:36:04.984781+0000","last_active":"2024-05-27T07:37:46.146306+0000","last_peered":"2024-05-27T07:37:46.146306+0000","last_clean":"2024-05-27T07:37:46.146306+0000","last_became_active":"2024-05-27T07:36:04.984462+0000","last_became_peered":"2024-05-27T07:36:04.984462+0000","last_unstale":"2024-05-27T07:37:46.146306+0000","last_undegraded":"2024-05-27T07:37:46.146306+0000","last_fullsized":"2024-05-27T07:37:46.146306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:40.051692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.395975+0000","last_change":"2024-05-27T07:36:04.978360+0000","last_active":"2024-05-27T07:38:43.395975+0000","last_peered":"2024-05-27T07:38:43.395975+0000","last_clean":"2024-05-27T07:38:43.395975+0000","last_became_active":"2024-05-27T07:36:04.978105+0000","last_became_peered":"2024-05-27T07:36:04.978105+0000","last_unstale":"2024-05-27T07:38:43.395975+0000","last_undegraded":"2024-05-27T07:38:43.395975+0000","last_fullsized":"2024-05-27T07:38:43.395975+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:04:47.257529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.393832+0000","last_change":"2024-05-27T07:36:04.986510+0000","last_active":"2024-05-27T07:38:43.393832+0000","last_peered":"2024-05-27T07:38:43.393832+0000","last_clean":"2024-05-27T07:38:43.393832+0000","last_became_active":"2024-05-27T07:36:04.986150+0000","last_became_peered":"2024-05-27T07:36:04.986150+0000","last_unstale":"2024-05-27T07:38:43.393832+0000","last_undegraded":"2024-05-27T07:38:43.393832+0000","last_fullsized":"2024-05-27T07:38:43.393832+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:07.349470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.402614+0000","last_change":"2024-05-27T07:36:04.979880+0000","last_active":"2024-05-27T07:38:43.402614+0000","last_peered":"2024-05-27T07:38:43.402614+0000","last_clean":"2024-05-27T07:38:43.402614+0000","last_became_active":"2024-05-27T07:36:04.979516+0000","last_became_peered":"2024-05-27T07:36:04.979516+0000","last_unstale":"2024-05-27T07:38:43.402614+0000","last_undegraded":"2024-05-27T07:38:43.402614+0000","last_fullsized":"2024-05-27T07:38:43.402614+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:41:03.964277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.402345+0000","last_change":"2024-05-27T07:36:04.980231+0000","last_active":"2024-05-27T07:38:43.402345+0000","last_peered":"2024-05-27T07:38:43.402345+0000","last_clean":"2024-05-27T07:38:43.402345+0000","last_became_active":"2024-05-27T07:36:04.979888+0000","last_became_peered":"2024-05-27T07:36:04.979888+0000","last_unstale":"2024-05-27T07:38:43.402345+0000","last_undegraded":"2024-05-27T07:38:43.402345+0000","last_fullsized":"2024-05-27T07:38:43.402345+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.838119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142784+0000","last_change":"2024-05-27T07:36:04.990890+0000","last_active":"2024-05-27T07:37:46.142784+0000","last_peered":"2024-05-27T07:37:46.142784+0000","last_clean":"2024-05-27T07:37:46.142784+0000","last_became_active":"2024-05-27T07:36:04.990581+0000","last_became_peered":"2024-05-27T07:36:04.990581+0000","last_unstale":"2024-05-27T07:37:46.142784+0000","last_undegraded":"2024-05-27T07:37:46.142784+0000","last_fullsized":"2024-05-27T07:37:46.142784+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:33.054395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.397423+0000","last_change":"2024-05-27T07:36:04.981620+0000","last_active":"2024-05-27T07:38:43.397423+0000","last_peered":"2024-05-27T07:38:43.397423+0000","last_clean":"2024-05-27T07:38:43.397423+0000","last_became_active":"2024-05-27T07:36:04.981290+0000","last_became_peered":"2024-05-27T07:36:04.981290+0000","last_unstale":"2024-05-27T07:38:43.397423+0000","last_undegraded":"2024-05-27T07:38:43.397423+0000","last_fullsized":"2024-05-27T07:38:43.397423+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:50.287380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.140727+0000","last_change":"2024-05-27T07:36:15.030740+0000","last_active":"2024-05-27T07:37:46.140727+0000","last_peered":"2024-05-27T07:37:46.140727+0000","last_clean":"2024-05-27T07:37:46.140727+0000","last_became_active":"2024-05-27T07:36:15.029738+0000","last_became_peered":"2024-05-27T07:36:15.029738+0000","last_unstale":"2024-05-27T07:37:46.140727+0000","last_undegraded":"2024-05-27T07:37:46.140727+0000","last_fullsized":"2024-05-27T07:37:46.140727+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:18.101941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"78'4","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.143016+0000","last_change":"2024-05-27T07:36:01.955530+0000","last_active":"2024-05-27T07:37:46.143016+0000","last_peered":"2024-05-27T07:37:46.143016+0000","last_clean":"2024-05-27T07:37:46.143016+0000","last_became_active":"2024-05-27T07:36:01.955225+0000","last_became_peered":"2024-05-27T07:36:01.955225+0000","last_unstale":"2024-05-27T07:37:46.143016+0000","last_undegraded":"2024-05-27T07:37:46.143016+0000","last_fullsized":"2024-05-27T07:37:46.143016+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:02.067672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.317760+0000","last_change":"2024-05-27T07:36:04.984397+0000","last_active":"2024-05-27T07:37:46.317760+0000","last_peered":"2024-05-27T07:37:46.317760+0000","last_clean":"2024-05-27T07:37:46.317760+0000","last_became_active":"2024-05-27T07:36:04.984033+0000","last_became_peered":"2024-05-27T07:36:04.984033+0000","last_unstale":"2024-05-27T07:37:46.317760+0000","last_undegraded":"2024-05-27T07:37:46.317760+0000","last_fullsized":"2024-05-27T07:37:46.317760+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:15.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.322662+0000","last_change":"2024-05-27T07:37:46.322144+0000","last_active":"2024-05-27T07:37:46.322662+0000","last_peered":"2024-05-27T07:37:46.322662+0000","last_clean":"2024-05-27T07:37:46.322662+0000","last_became_active":"2024-05-27T07:37:46.321610+0000","last_became_peered":"2024-05-27T07:37:46.321610+0000","last_unstale":"2024-05-27T07:37:46.322662+0000","last_undegraded":"2024-05-27T07:37:46.322662+0000","last_fullsized":"2024-05-27T07:37:46.322662+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:47.119281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"78'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.317953+0000","last_change":"2024-05-27T07:36:02.243079+0000","last_active":"2024-05-27T07:37:46.317953+0000","last_peered":"2024-05-27T07:37:46.317953+0000","last_clean":"2024-05-27T07:37:46.317953+0000","last_became_active":"2024-05-27T07:36:02.242633+0000","last_became_peered":"2024-05-27T07:36:02.242633+0000","last_unstale":"2024-05-27T07:37:46.317953+0000","last_undegraded":"2024-05-27T07:37:46.317953+0000","last_fullsized":"2024-05-27T07:37:46.317953+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:45:33.980842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.403095+0000","last_change":"2024-05-27T07:36:04.989775+0000","last_active":"2024-05-27T07:38:43.403095+0000","last_peered":"2024-05-27T07:38:43.403095+0000","last_clean":"2024-05-27T07:38:43.403095+0000","last_became_active":"2024-05-27T07:36:04.989289+0000","last_became_peered":"2024-05-27T07:36:04.989289+0000","last_unstale":"2024-05-27T07:38:43.403095+0000","last_undegraded":"2024-05-27T07:38:43.403095+0000","last_fullsized":"2024-05-27T07:38:43.403095+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:24:38.355231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144310+0000","last_change":"2024-05-27T07:36:15.031737+0000","last_active":"2024-05-27T07:37:46.144310+0000","last_peered":"2024-05-27T07:37:46.144310+0000","last_clean":"2024-05-27T07:37:46.144310+0000","last_became_active":"2024-05-27T07:36:15.031314+0000","last_became_peered":"2024-05-27T07:36:15.031314+0000","last_unstale":"2024-05-27T07:37:46.144310+0000","last_undegraded":"2024-05-27T07:37:46.144310+0000","last_fullsized":"2024-05-27T07:37:46.144310+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:32.742298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"78'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.148103+0000","last_change":"2024-05-27T07:36:01.957459+0000","last_active":"2024-05-27T07:37:46.148103+0000","last_peered":"2024-05-27T07:37:46.148103+0000","last_clean":"2024-05-27T07:37:46.148103+0000","last_became_active":"2024-05-27T07:36:01.954985+0000","last_became_peered":"2024-05-27T07:36:01.954985+0000","last_unstale":"2024-05-27T07:37:46.148103+0000","last_undegraded":"2024-05-27T07:37:46.148103+0000","last_fullsized":"2024-05-27T07:37:46.148103+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:27.085486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.400909+0000","last_change":"2024-05-27T07:36:04.986447+0000","last_active":"2024-05-27T07:38:43.400909+0000","last_peered":"2024-05-27T07:38:43.400909+0000","last_clean":"2024-05-27T07:38:43.400909+0000","last_became_active":"2024-05-27T07:36:04.982099+0000","last_became_peered":"2024-05-27T07:36:04.982099+0000","last_unstale":"2024-05-27T07:38:43.400909+0000","last_undegraded":"2024-05-27T07:38:43.400909+0000","last_fullsized":"2024-05-27T07:38:43.400909+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:32.637479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144032+0000","last_change":"2024-05-27T07:36:15.024152+0000","last_active":"2024-05-27T07:37:46.144032+0000","last_peered":"2024-05-27T07:37:46.144032+0000","last_clean":"2024-05-27T07:37:46.144032+0000","last_became_active":"2024-05-27T07:36:15.023711+0000","last_became_peered":"2024-05-27T07:36:15.023711+0000","last_unstale":"2024-05-27T07:37:46.144032+0000","last_undegraded":"2024-05-27T07:37:46.144032+0000","last_fullsized":"2024-05-27T07:37:46.144032+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:33.809364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147668+0000","last_change":"2024-05-27T07:36:01.950531+0000","last_active":"2024-05-27T07:37:46.147668+0000","last_peered":"2024-05-27T07:37:46.147668+0000","last_clean":"2024-05-27T07:37:46.147668+0000","last_became_active":"2024-05-27T07:36:01.950015+0000","last_became_peered":"2024-05-27T07:36:01.950015+0000","last_unstale":"2024-05-27T07:37:46.147668+0000","last_undegraded":"2024-05-27T07:37:46.147668+0000","last_fullsized":"2024-05-27T07:37:46.147668+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:22.400415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.403821+0000","last_change":"2024-05-27T07:36:04.982792+0000","last_active":"2024-05-27T07:38:43.403821+0000","last_peered":"2024-05-27T07:38:43.403821+0000","last_clean":"2024-05-27T07:38:43.403821+0000","last_became_active":"2024-05-27T07:36:04.982400+0000","last_became_peered":"2024-05-27T07:36:04.982400+0000","last_unstale":"2024-05-27T07:38:43.403821+0000","last_undegraded":"2024-05-27T07:38:43.403821+0000","last_fullsized":"2024-05-27T07:38:43.403821+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:27.823654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141113+0000","last_change":"2024-05-27T07:36:15.027932+0000","last_active":"2024-05-27T07:37:46.141113+0000","last_peered":"2024-05-27T07:37:46.141113+0000","last_clean":"2024-05-27T07:37:46.141113+0000","last_became_active":"2024-05-27T07:36:15.027526+0000","last_became_peered":"2024-05-27T07:36:15.027526+0000","last_unstale":"2024-05-27T07:37:46.141113+0000","last_undegraded":"2024-05-27T07:37:46.141113+0000","last_fullsized":"2024-05-27T07:37:46.141113+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:16.165934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"78'1","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315759+0000","last_change":"2024-05-27T07:36:45.127637+0000","last_active":"2024-05-27T07:37:46.315759+0000","last_peered":"2024-05-27T07:37:46.315759+0000","last_clean":"2024-05-27T07:37:46.315759+0000","last_became_active":"2024-05-27T07:36:45.127249+0000","last_became_peered":"2024-05-27T07:36:45.127249+0000","last_unstale":"2024-05-27T07:37:46.315759+0000","last_undegraded":"2024-05-27T07:37:46.315759+0000","last_fullsized":"2024-05-27T07:37:46.315759+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:52.418069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147641+0000","last_change":"2024-05-27T07:36:04.985135+0000","last_active":"2024-05-27T07:37:46.147641+0000","last_peered":"2024-05-27T07:37:46.147641+0000","last_clean":"2024-05-27T07:37:46.147641+0000","last_became_active":"2024-05-27T07:36:04.983621+0000","last_became_peered":"2024-05-27T07:36:04.983621+0000","last_unstale":"2024-05-27T07:37:46.147641+0000","last_undegraded":"2024-05-27T07:37:46.147641+0000","last_fullsized":"2024-05-27T07:37:46.147641+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:11.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.140896+0000","last_change":"2024-05-27T07:36:15.024558+0000","last_active":"2024-05-27T07:37:46.140896+0000","last_peered":"2024-05-27T07:37:46.140896+0000","last_clean":"2024-05-27T07:37:46.140896+0000","last_became_active":"2024-05-27T07:36:15.024148+0000","last_became_peered":"2024-05-27T07:36:15.024148+0000","last_unstale":"2024-05-27T07:37:46.140896+0000","last_undegraded":"2024-05-27T07:37:46.140896+0000","last_fullsized":"2024-05-27T07:37:46.140896+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:03.648451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"78'1","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141941+0000","last_change":"2024-05-27T07:36:45.129413+0000","last_active":"2024-05-27T07:37:46.141941+0000","last_peered":"2024-05-27T07:37:46.141941+0000","last_clean":"2024-05-27T07:37:46.141941+0000","last_became_active":"2024-05-27T07:36:45.128779+0000","last_became_peered":"2024-05-27T07:36:45.128779+0000","last_unstale":"2024-05-27T07:37:46.141941+0000","last_undegraded":"2024-05-27T07:37:46.141941+0000","last_fullsized":"2024-05-27T07:37:46.141941+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:05.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.397756+0000","last_change":"2024-05-27T07:36:04.989878+0000","last_active":"2024-05-27T07:38:43.397756+0000","last_peered":"2024-05-27T07:38:43.397756+0000","last_clean":"2024-05-27T07:38:43.397756+0000","last_became_active":"2024-05-27T07:36:04.989532+0000","last_became_peered":"2024-05-27T07:36:04.989532+0000","last_unstale":"2024-05-27T07:38:43.397756+0000","last_undegraded":"2024-05-27T07:38:43.397756+0000","last_fullsized":"2024-05-27T07:38:43.397756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:54:53.909592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.322124+0000","last_change":"2024-05-27T07:37:46.321582+0000","last_active":"2024-05-27T07:37:46.322124+0000","last_peered":"2024-05-27T07:37:46.322124+0000","last_clean":"2024-05-27T07:37:46.322124+0000","last_became_active":"2024-05-27T07:37:46.321193+0000","last_became_peered":"2024-05-27T07:37:46.321193+0000","last_unstale":"2024-05-27T07:37:46.322124+0000","last_undegraded":"2024-05-27T07:37:46.322124+0000","last_fullsized":"2024-05-27T07:37:46.322124+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:25:25.767300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144205+0000","last_change":"2024-05-27T07:36:01.949684+0000","last_active":"2024-05-27T07:37:46.144205+0000","last_peered":"2024-05-27T07:37:46.144205+0000","last_clean":"2024-05-27T07:37:46.144205+0000","last_became_active":"2024-05-27T07:36:01.949308+0000","last_became_peered":"2024-05-27T07:36:01.949308+0000","last_unstale":"2024-05-27T07:37:46.144205+0000","last_undegraded":"2024-05-27T07:37:46.144205+0000","last_fullsized":"2024-05-27T07:37:46.144205+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:50:54.859172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.317641+0000","last_change":"2024-05-27T07:36:04.971266+0000","last_active":"2024-05-27T07:37:46.317641+0000","last_peered":"2024-05-27T07:37:46.317641+0000","last_clean":"2024-05-27T07:37:46.317641+0000","last_became_active":"2024-05-27T07:36:04.970831+0000","last_became_peered":"2024-05-27T07:36:04.970831+0000","last_unstale":"2024-05-27T07:37:46.317641+0000","last_undegraded":"2024-05-27T07:37:46.317641+0000","last_fullsized":"2024-05-27T07:37:46.317641+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:37.237637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.317455+0000","last_change":"2024-05-27T07:36:15.024415+0000","last_active":"2024-05-27T07:37:46.317455+0000","last_peered":"2024-05-27T07:37:46.317455+0000","last_clean":"2024-05-27T07:37:46.317455+0000","last_became_active":"2024-05-27T07:36:15.023935+0000","last_became_peered":"2024-05-27T07:36:15.023935+0000","last_unstale":"2024-05-27T07:37:46.317455+0000","last_undegraded":"2024-05-27T07:37:46.317455+0000","last_fullsized":"2024-05-27T07:37:46.317455+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:37:30.386933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315347+0000","last_change":"2024-05-27T07:36:01.952256+0000","last_active":"2024-05-27T07:37:46.315347+0000","last_peered":"2024-05-27T07:37:46.315347+0000","last_clean":"2024-05-27T07:37:46.315347+0000","last_became_active":"2024-05-27T07:36:01.951876+0000","last_became_peered":"2024-05-27T07:36:01.951876+0000","last_unstale":"2024-05-27T07:37:46.315347+0000","last_undegraded":"2024-05-27T07:37:46.315347+0000","last_fullsized":"2024-05-27T07:37:46.315347+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:05.201780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141666+0000","last_change":"2024-05-27T07:36:04.980300+0000","last_active":"2024-05-27T07:37:46.141666+0000","last_peered":"2024-05-27T07:37:46.141666+0000","last_clean":"2024-05-27T07:37:46.141666+0000","last_became_active":"2024-05-27T07:36:04.979954+0000","last_became_peered":"2024-05-27T07:36:04.979954+0000","last_unstale":"2024-05-27T07:37:46.141666+0000","last_undegraded":"2024-05-27T07:37:46.141666+0000","last_fullsized":"2024-05-27T07:37:46.141666+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:15.712773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.318738+0000","last_change":"2024-05-27T07:36:15.026685+0000","last_active":"2024-05-27T07:37:46.318738+0000","last_peered":"2024-05-27T07:37:46.318738+0000","last_clean":"2024-05-27T07:37:46.318738+0000","last_became_active":"2024-05-27T07:36:15.026287+0000","last_became_peered":"2024-05-27T07:36:15.026287+0000","last_unstale":"2024-05-27T07:37:46.318738+0000","last_undegraded":"2024-05-27T07:37:46.318738+0000","last_fullsized":"2024-05-27T07:37:46.318738+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:29:26.394435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142109+0000","last_change":"2024-05-27T07:36:01.956566+0000","last_active":"2024-05-27T07:37:46.142109+0000","last_peered":"2024-05-27T07:37:46.142109+0000","last_clean":"2024-05-27T07:37:46.142109+0000","last_became_active":"2024-05-27T07:36:01.955963+0000","last_became_peered":"2024-05-27T07:36:01.955963+0000","last_unstale":"2024-05-27T07:37:46.142109+0000","last_undegraded":"2024-05-27T07:37:46.142109+0000","last_fullsized":"2024-05-27T07:37:46.142109+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:53.081784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.399003+0000","last_change":"2024-05-27T07:36:04.983878+0000","last_active":"2024-05-27T07:38:43.399003+0000","last_peered":"2024-05-27T07:38:43.399003+0000","last_clean":"2024-05-27T07:38:43.399003+0000","last_became_active":"2024-05-27T07:36:04.983339+0000","last_became_peered":"2024-05-27T07:36:04.983339+0000","last_unstale":"2024-05-27T07:38:43.399003+0000","last_undegraded":"2024-05-27T07:38:43.399003+0000","last_fullsized":"2024-05-27T07:38:43.399003+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:57:01.726122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141512+0000","last_change":"2024-05-27T07:36:15.027923+0000","last_active":"2024-05-27T07:37:46.141512+0000","last_peered":"2024-05-27T07:37:46.141512+0000","last_clean":"2024-05-27T07:37:46.141512+0000","last_became_active":"2024-05-27T07:36:15.026012+0000","last_became_peered":"2024-05-27T07:36:15.026012+0000","last_unstale":"2024-05-27T07:37:46.141512+0000","last_undegraded":"2024-05-27T07:37:46.141512+0000","last_fullsized":"2024-05-27T07:37:46.141512+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:27:45.238053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147660+0000","last_change":"2024-05-27T07:36:01.956038+0000","last_active":"2024-05-27T07:37:46.147660+0000","last_peered":"2024-05-27T07:37:46.147660+0000","last_clean":"2024-05-27T07:37:46.147660+0000","last_became_active":"2024-05-27T07:36:01.955639+0000","last_became_peered":"2024-05-27T07:36:01.955639+0000","last_unstale":"2024-05-27T07:37:46.147660+0000","last_undegraded":"2024-05-27T07:37:46.147660+0000","last_fullsized":"2024-05-27T07:37:46.147660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:32.920194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.400666+0000","last_change":"2024-05-27T07:36:04.986676+0000","last_active":"2024-05-27T07:38:43.400666+0000","last_peered":"2024-05-27T07:38:43.400666+0000","last_clean":"2024-05-27T07:38:43.400666+0000","last_became_active":"2024-05-27T07:36:04.986316+0000","last_became_peered":"2024-05-27T07:36:04.986316+0000","last_unstale":"2024-05-27T07:38:43.400666+0000","last_undegraded":"2024-05-27T07:38:43.400666+0000","last_fullsized":"2024-05-27T07:38:43.400666+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:03:50.947781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144481+0000","last_change":"2024-05-27T07:36:15.027643+0000","last_active":"2024-05-27T07:37:46.144481+0000","last_peered":"2024-05-27T07:37:46.144481+0000","last_clean":"2024-05-27T07:37:46.144481+0000","last_became_active":"2024-05-27T07:36:15.027175+0000","last_became_peered":"2024-05-27T07:36:15.027175+0000","last_unstale":"2024-05-27T07:37:46.144481+0000","last_undegraded":"2024-05-27T07:37:46.144481+0000","last_fullsized":"2024-05-27T07:37:46.144481+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:46:32.764271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"78'3","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315206+0000","last_change":"2024-05-27T07:36:02.245746+0000","last_active":"2024-05-27T07:37:46.315206+0000","last_peered":"2024-05-27T07:37:46.315206+0000","last_clean":"2024-05-27T07:37:46.315206+0000","last_became_active":"2024-05-27T07:36:02.245269+0000","last_became_peered":"2024-05-27T07:36:02.245269+0000","last_unstale":"2024-05-27T07:37:46.315206+0000","last_undegraded":"2024-05-27T07:37:46.315206+0000","last_fullsized":"2024-05-27T07:37:46.315206+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:28.202540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.402639+0000","last_change":"2024-05-27T07:36:45.129374+0000","last_active":"2024-05-27T07:38:43.402639+0000","last_peered":"2024-05-27T07:38:43.402639+0000","last_clean":"2024-05-27T07:38:43.402639+0000","last_became_active":"2024-05-27T07:36:45.128956+0000","last_became_peered":"2024-05-27T07:36:45.128956+0000","last_unstale":"2024-05-27T07:38:43.402639+0000","last_undegraded":"2024-05-27T07:38:43.402639+0000","last_fullsized":"2024-05-27T07:38:43.402639+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:17.037598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142054+0000","last_change":"2024-05-27T07:36:15.027097+0000","last_active":"2024-05-27T07:37:46.142054+0000","last_peered":"2024-05-27T07:37:46.142054+0000","last_clean":"2024-05-27T07:37:46.142054+0000","last_became_active":"2024-05-27T07:36:15.026704+0000","last_became_peered":"2024-05-27T07:36:15.026704+0000","last_unstale":"2024-05-27T07:37:46.142054+0000","last_undegraded":"2024-05-27T07:37:46.142054+0000","last_fullsized":"2024-05-27T07:37:46.142054+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:35:03.747199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'31","reported_seq":281,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:40.613959+0000","last_change":"2024-05-27T07:36:01.956926+0000","last_active":"2024-05-27T07:38:40.613959+0000","last_peered":"2024-05-27T07:38:40.613959+0000","last_clean":"2024-05-27T07:38:40.613959+0000","last_became_active":"2024-05-27T07:36:01.956515+0000","last_became_peered":"2024-05-27T07:36:01.956515+0000","last_unstale":"2024-05-27T07:38:40.613959+0000","last_undegraded":"2024-05-27T07:38:40.613959+0000","last_fullsized":"2024-05-27T07:38:40.613959+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:46.566489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64645,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":205,"num_read_kb":166,"num_write":31,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147928+0000","last_change":"2024-05-27T07:36:04.976498+0000","last_active":"2024-05-27T07:37:46.147928+0000","last_peered":"2024-05-27T07:37:46.147928+0000","last_clean":"2024-05-27T07:37:46.147928+0000","last_became_active":"2024-05-27T07:36:04.976020+0000","last_became_peered":"2024-05-27T07:36:04.976020+0000","last_unstale":"2024-05-27T07:37:46.147928+0000","last_undegraded":"2024-05-27T07:37:46.147928+0000","last_fullsized":"2024-05-27T07:37:46.147928+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:25.909259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.316205+0000","last_change":"2024-05-27T07:36:15.022700+0000","last_active":"2024-05-27T07:37:46.316205+0000","last_peered":"2024-05-27T07:37:46.316205+0000","last_clean":"2024-05-27T07:37:46.316205+0000","last_became_active":"2024-05-27T07:36:15.022109+0000","last_became_peered":"2024-05-27T07:36:15.022109+0000","last_unstale":"2024-05-27T07:37:46.316205+0000","last_undegraded":"2024-05-27T07:37:46.316205+0000","last_fullsized":"2024-05-27T07:37:46.316205+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:02:43.498049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"78'2","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.140996+0000","last_change":"2024-05-27T07:36:01.953442+0000","last_active":"2024-05-27T07:37:46.140996+0000","last_peered":"2024-05-27T07:37:46.140996+0000","last_clean":"2024-05-27T07:37:46.140996+0000","last_became_active":"2024-05-27T07:36:01.953051+0000","last_became_peered":"2024-05-27T07:36:01.953051+0000","last_unstale":"2024-05-27T07:37:46.140996+0000","last_undegraded":"2024-05-27T07:37:46.140996+0000","last_fullsized":"2024-05-27T07:37:46.140996+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:47:53.070596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144460+0000","last_change":"2024-05-27T07:36:04.976365+0000","last_active":"2024-05-27T07:37:46.144460+0000","last_peered":"2024-05-27T07:37:46.144460+0000","last_clean":"2024-05-27T07:37:46.144460+0000","last_became_active":"2024-05-27T07:36:04.976050+0000","last_became_peered":"2024-05-27T07:36:04.976050+0000","last_unstale":"2024-05-27T07:37:46.144460+0000","last_undegraded":"2024-05-27T07:37:46.144460+0000","last_fullsized":"2024-05-27T07:37:46.144460+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:19.765746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145211+0000","last_change":"2024-05-27T07:36:15.022561+0000","last_active":"2024-05-27T07:37:46.145211+0000","last_peered":"2024-05-27T07:37:46.145211+0000","last_clean":"2024-05-27T07:37:46.145211+0000","last_became_active":"2024-05-27T07:36:15.021974+0000","last_became_peered":"2024-05-27T07:36:15.021974+0000","last_unstale":"2024-05-27T07:37:46.145211+0000","last_undegraded":"2024-05-27T07:37:46.145211+0000","last_fullsized":"2024-05-27T07:37:46.145211+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:31.520969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315818+0000","last_change":"2024-05-27T07:36:01.952625+0000","last_active":"2024-05-27T07:37:46.315818+0000","last_peered":"2024-05-27T07:37:46.315818+0000","last_clean":"2024-05-27T07:37:46.315818+0000","last_became_active":"2024-05-27T07:36:01.952280+0000","last_became_peered":"2024-05-27T07:36:01.952280+0000","last_unstale":"2024-05-27T07:37:46.315818+0000","last_undegraded":"2024-05-27T07:37:46.315818+0000","last_fullsized":"2024-05-27T07:37:46.315818+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:17.548061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147276+0000","last_change":"2024-05-27T07:36:04.984795+0000","last_active":"2024-05-27T07:37:46.147276+0000","last_peered":"2024-05-27T07:37:46.147276+0000","last_clean":"2024-05-27T07:37:46.147276+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:37:46.147276+0000","last_undegraded":"2024-05-27T07:37:46.147276+0000","last_fullsized":"2024-05-27T07:37:46.147276+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:07.207460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142640+0000","last_change":"2024-05-27T07:36:15.033565+0000","last_active":"2024-05-27T07:37:46.142640+0000","last_peered":"2024-05-27T07:37:46.142640+0000","last_clean":"2024-05-27T07:37:46.142640+0000","last_became_active":"2024-05-27T07:36:15.033145+0000","last_became_peered":"2024-05-27T07:36:15.033145+0000","last_unstale":"2024-05-27T07:37:46.142640+0000","last_undegraded":"2024-05-27T07:37:46.142640+0000","last_fullsized":"2024-05-27T07:37:46.142640+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:38:02.163816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144084+0000","last_change":"2024-05-27T07:36:01.952759+0000","last_active":"2024-05-27T07:37:46.144084+0000","last_peered":"2024-05-27T07:37:46.144084+0000","last_clean":"2024-05-27T07:37:46.144084+0000","last_became_active":"2024-05-27T07:36:01.952339+0000","last_became_peered":"2024-05-27T07:36:01.952339+0000","last_unstale":"2024-05-27T07:37:46.144084+0000","last_undegraded":"2024-05-27T07:37:46.144084+0000","last_fullsized":"2024-05-27T07:37:46.144084+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:14.518123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147455+0000","last_change":"2024-05-27T07:36:04.986312+0000","last_active":"2024-05-27T07:37:46.147455+0000","last_peered":"2024-05-27T07:37:46.147455+0000","last_clean":"2024-05-27T07:37:46.147455+0000","last_became_active":"2024-05-27T07:36:04.985944+0000","last_became_peered":"2024-05-27T07:36:04.985944+0000","last_unstale":"2024-05-27T07:37:46.147455+0000","last_undegraded":"2024-05-27T07:37:46.147455+0000","last_fullsized":"2024-05-27T07:37:46.147455+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:04.955449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144991+0000","last_change":"2024-05-27T07:36:15.020930+0000","last_active":"2024-05-27T07:37:46.144991+0000","last_peered":"2024-05-27T07:37:46.144991+0000","last_clean":"2024-05-27T07:37:46.144991+0000","last_became_active":"2024-05-27T07:36:15.020556+0000","last_became_peered":"2024-05-27T07:36:15.020556+0000","last_unstale":"2024-05-27T07:37:46.144991+0000","last_undegraded":"2024-05-27T07:37:46.144991+0000","last_fullsized":"2024-05-27T07:37:46.144991+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:02.673065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"78'8","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147426+0000","last_change":"2024-05-27T07:36:02.243402+0000","last_active":"2024-05-27T07:37:46.147426+0000","last_peered":"2024-05-27T07:37:46.147426+0000","last_clean":"2024-05-27T07:37:46.147426+0000","last_became_active":"2024-05-27T07:36:02.242978+0000","last_became_peered":"2024-05-27T07:36:02.242978+0000","last_unstale":"2024-05-27T07:37:46.147426+0000","last_undegraded":"2024-05-27T07:37:46.147426+0000","last_fullsized":"2024-05-27T07:37:46.147426+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:25:04.289506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.392605+0000","last_change":"2024-05-27T07:36:04.989885+0000","last_active":"2024-05-27T07:38:43.392605+0000","last_peered":"2024-05-27T07:38:43.392605+0000","last_clean":"2024-05-27T07:38:43.392605+0000","last_became_active":"2024-05-27T07:36:04.989394+0000","last_became_peered":"2024-05-27T07:36:04.989394+0000","last_unstale":"2024-05-27T07:38:43.392605+0000","last_undegraded":"2024-05-27T07:38:43.392605+0000","last_fullsized":"2024-05-27T07:38:43.392605+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:54.138286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315531+0000","last_change":"2024-05-27T07:36:15.021772+0000","last_active":"2024-05-27T07:37:46.315531+0000","last_peered":"2024-05-27T07:37:46.315531+0000","last_clean":"2024-05-27T07:37:46.315531+0000","last_became_active":"2024-05-27T07:36:15.021352+0000","last_became_peered":"2024-05-27T07:36:15.021352+0000","last_unstale":"2024-05-27T07:37:46.315531+0000","last_undegraded":"2024-05-27T07:37:46.315531+0000","last_fullsized":"2024-05-27T07:37:46.315531+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:23.066662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.146222+0000","last_change":"2024-05-27T07:36:45.131900+0000","last_active":"2024-05-27T07:37:46.146222+0000","last_peered":"2024-05-27T07:37:46.146222+0000","last_clean":"2024-05-27T07:37:46.146222+0000","last_became_active":"2024-05-27T07:36:45.127340+0000","last_became_peered":"2024-05-27T07:36:45.127340+0000","last_unstale":"2024-05-27T07:37:46.146222+0000","last_undegraded":"2024-05-27T07:37:46.146222+0000","last_fullsized":"2024-05-27T07:37:46.146222+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:07:57.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.396708+0000","last_change":"2024-05-27T07:36:04.987671+0000","last_active":"2024-05-27T07:38:43.396708+0000","last_peered":"2024-05-27T07:38:43.396708+0000","last_clean":"2024-05-27T07:38:43.396708+0000","last_became_active":"2024-05-27T07:36:04.987317+0000","last_became_peered":"2024-05-27T07:36:04.987317+0000","last_unstale":"2024-05-27T07:38:43.396708+0000","last_undegraded":"2024-05-27T07:38:43.396708+0000","last_fullsized":"2024-05-27T07:38:43.396708+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:35.985968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147631+0000","last_change":"2024-05-27T07:36:15.019967+0000","last_active":"2024-05-27T07:37:46.147631+0000","last_peered":"2024-05-27T07:37:46.147631+0000","last_clean":"2024-05-27T07:37:46.147631+0000","last_became_active":"2024-05-27T07:36:15.019509+0000","last_became_peered":"2024-05-27T07:36:15.019509+0000","last_unstale":"2024-05-27T07:37:46.147631+0000","last_undegraded":"2024-05-27T07:37:46.147631+0000","last_fullsized":"2024-05-27T07:37:46.147631+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:01:56.137335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.143286+0000","last_change":"2024-05-27T07:36:02.243561+0000","last_active":"2024-05-27T07:37:46.143286+0000","last_peered":"2024-05-27T07:37:46.143286+0000","last_clean":"2024-05-27T07:37:46.143286+0000","last_became_active":"2024-05-27T07:36:02.242911+0000","last_became_peered":"2024-05-27T07:36:02.242911+0000","last_unstale":"2024-05-27T07:37:46.143286+0000","last_undegraded":"2024-05-27T07:37:46.143286+0000","last_fullsized":"2024-05-27T07:37:46.143286+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:01:43.915660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141914+0000","last_change":"2024-05-27T07:36:04.971108+0000","last_active":"2024-05-27T07:37:46.141914+0000","last_peered":"2024-05-27T07:37:46.141914+0000","last_clean":"2024-05-27T07:37:46.141914+0000","last_became_active":"2024-05-27T07:36:04.970682+0000","last_became_peered":"2024-05-27T07:36:04.970682+0000","last_unstale":"2024-05-27T07:37:46.141914+0000","last_undegraded":"2024-05-27T07:37:46.141914+0000","last_fullsized":"2024-05-27T07:37:46.141914+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:15:34.940449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144342+0000","last_change":"2024-05-27T07:36:15.021292+0000","last_active":"2024-05-27T07:37:46.144342+0000","last_peered":"2024-05-27T07:37:46.144342+0000","last_clean":"2024-05-27T07:37:46.144342+0000","last_became_active":"2024-05-27T07:36:15.019988+0000","last_became_peered":"2024-05-27T07:36:15.019988+0000","last_unstale":"2024-05-27T07:37:46.144342+0000","last_undegraded":"2024-05-27T07:37:46.144342+0000","last_fullsized":"2024-05-27T07:37:46.144342+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:34:14.325368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.146911+0000","last_change":"2024-05-27T07:36:01.952231+0000","last_active":"2024-05-27T07:37:46.146911+0000","last_peered":"2024-05-27T07:37:46.146911+0000","last_clean":"2024-05-27T07:37:46.146911+0000","last_became_active":"2024-05-27T07:36:01.951818+0000","last_became_peered":"2024-05-27T07:36:01.951818+0000","last_unstale":"2024-05-27T07:37:46.146911+0000","last_undegraded":"2024-05-27T07:37:46.146911+0000","last_fullsized":"2024-05-27T07:37:46.146911+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:52:16.299204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141910+0000","last_change":"2024-05-27T07:36:04.982761+0000","last_active":"2024-05-27T07:37:46.141910+0000","last_peered":"2024-05-27T07:37:46.141910+0000","last_clean":"2024-05-27T07:37:46.141910+0000","last_became_active":"2024-05-27T07:36:04.982303+0000","last_became_peered":"2024-05-27T07:36:04.982303+0000","last_unstale":"2024-05-27T07:37:46.141910+0000","last_undegraded":"2024-05-27T07:37:46.141910+0000","last_fullsized":"2024-05-27T07:37:46.141910+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:13:19.099227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.324049+0000","last_change":"2024-05-27T07:37:46.323619+0000","last_active":"2024-05-27T07:37:46.324049+0000","last_peered":"2024-05-27T07:37:46.324049+0000","last_clean":"2024-05-27T07:37:46.324049+0000","last_became_active":"2024-05-27T07:37:46.322114+0000","last_became_peered":"2024-05-27T07:37:46.322114+0000","last_unstale":"2024-05-27T07:37:46.324049+0000","last_undegraded":"2024-05-27T07:37:46.324049+0000","last_fullsized":"2024-05-27T07:37:46.324049+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:59:08.966863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.399563+0000","last_change":"2024-05-27T07:36:45.142023+0000","last_active":"2024-05-27T07:38:43.399563+0000","last_peered":"2024-05-27T07:38:43.399563+0000","last_clean":"2024-05-27T07:38:43.399563+0000","last_became_active":"2024-05-27T07:36:45.141278+0000","last_became_peered":"2024-05-27T07:36:45.141278+0000","last_unstale":"2024-05-27T07:38:43.399563+0000","last_undegraded":"2024-05-27T07:38:43.399563+0000","last_fullsized":"2024-05-27T07:38:43.399563+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:15.800380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144108+0000","last_change":"2024-05-27T07:36:02.243041+0000","last_active":"2024-05-27T07:37:46.144108+0000","last_peered":"2024-05-27T07:37:46.144108+0000","last_clean":"2024-05-27T07:37:46.144108+0000","last_became_active":"2024-05-27T07:36:02.242689+0000","last_became_peered":"2024-05-27T07:36:02.242689+0000","last_unstale":"2024-05-27T07:37:46.144108+0000","last_undegraded":"2024-05-27T07:37:46.144108+0000","last_fullsized":"2024-05-27T07:37:46.144108+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:44:19.652954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147487+0000","last_change":"2024-05-27T07:36:15.025595+0000","last_active":"2024-05-27T07:37:46.147487+0000","last_peered":"2024-05-27T07:37:46.147487+0000","last_clean":"2024-05-27T07:37:46.147487+0000","last_became_active":"2024-05-27T07:36:15.025177+0000","last_became_peered":"2024-05-27T07:36:15.025177+0000","last_unstale":"2024-05-27T07:37:46.147487+0000","last_undegraded":"2024-05-27T07:37:46.147487+0000","last_fullsized":"2024-05-27T07:37:46.147487+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:08.920778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315492+0000","last_change":"2024-05-27T07:36:04.981422+0000","last_active":"2024-05-27T07:37:46.315492+0000","last_peered":"2024-05-27T07:37:46.315492+0000","last_clean":"2024-05-27T07:37:46.315492+0000","last_became_active":"2024-05-27T07:36:04.981100+0000","last_became_peered":"2024-05-27T07:36:04.981100+0000","last_unstale":"2024-05-27T07:37:46.315492+0000","last_undegraded":"2024-05-27T07:37:46.315492+0000","last_fullsized":"2024-05-27T07:37:46.315492+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:01.076309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144544+0000","last_change":"2024-05-27T07:36:01.953727+0000","last_active":"2024-05-27T07:37:46.144544+0000","last_peered":"2024-05-27T07:37:46.144544+0000","last_clean":"2024-05-27T07:37:46.144544+0000","last_became_active":"2024-05-27T07:36:01.953323+0000","last_became_peered":"2024-05-27T07:36:01.953323+0000","last_unstale":"2024-05-27T07:37:46.144544+0000","last_undegraded":"2024-05-27T07:37:46.144544+0000","last_fullsized":"2024-05-27T07:37:46.144544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:17.186854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":119,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147156+0000","last_change":"2024-05-27T07:34:38.289913+0000","last_active":"2024-05-27T07:37:46.147156+0000","last_peered":"2024-05-27T07:37:46.147156+0000","last_clean":"2024-05-27T07:37:46.147156+0000","last_became_active":"2024-05-27T07:34:38.285523+0000","last_became_peered":"2024-05-27T07:34:38.285523+0000","last_unstale":"2024-05-27T07:37:46.147156+0000","last_undegraded":"2024-05-27T07:37:46.147156+0000","last_fullsized":"2024-05-27T07:37:46.147156+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_clean_scrub_stamp":"2024-05-27T07:31:09.455079+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:59.537055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.316050+0000","last_change":"2024-05-27T07:36:15.027453+0000","last_active":"2024-05-27T07:37:46.316050+0000","last_peered":"2024-05-27T07:37:46.316050+0000","last_clean":"2024-05-27T07:37:46.316050+0000","last_became_active":"2024-05-27T07:36:15.026872+0000","last_became_peered":"2024-05-27T07:36:15.026872+0000","last_unstale":"2024-05-27T07:37:46.316050+0000","last_undegraded":"2024-05-27T07:37:46.316050+0000","last_fullsized":"2024-05-27T07:37:46.316050+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:32.351924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144005+0000","last_change":"2024-05-27T07:36:04.982041+0000","last_active":"2024-05-27T07:37:46.144005+0000","last_peered":"2024-05-27T07:37:46.144005+0000","last_clean":"2024-05-27T07:37:46.144005+0000","last_became_active":"2024-05-27T07:36:04.981719+0000","last_became_peered":"2024-05-27T07:36:04.981719+0000","last_unstale":"2024-05-27T07:37:46.144005+0000","last_undegraded":"2024-05-27T07:37:46.144005+0000","last_fullsized":"2024-05-27T07:37:46.144005+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:40.061892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147061+0000","last_change":"2024-05-27T07:36:01.952238+0000","last_active":"2024-05-27T07:37:46.147061+0000","last_peered":"2024-05-27T07:37:46.147061+0000","last_clean":"2024-05-27T07:37:46.147061+0000","last_became_active":"2024-05-27T07:36:01.951853+0000","last_became_peered":"2024-05-27T07:36:01.951853+0000","last_unstale":"2024-05-27T07:37:46.147061+0000","last_undegraded":"2024-05-27T07:37:46.147061+0000","last_fullsized":"2024-05-27T07:37:46.147061+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:46:41.317531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145516+0000","last_change":"2024-05-27T07:36:15.021848+0000","last_active":"2024-05-27T07:37:46.145516+0000","last_peered":"2024-05-27T07:37:46.145516+0000","last_clean":"2024-05-27T07:37:46.145516+0000","last_became_active":"2024-05-27T07:36:15.021371+0000","last_became_peered":"2024-05-27T07:36:15.021371+0000","last_unstale":"2024-05-27T07:37:46.145516+0000","last_undegraded":"2024-05-27T07:37:46.145516+0000","last_fullsized":"2024-05-27T07:37:46.145516+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:32:21.741229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147962+0000","last_change":"2024-05-27T07:36:02.244283+0000","last_active":"2024-05-27T07:37:46.147962+0000","last_peered":"2024-05-27T07:37:46.147962+0000","last_clean":"2024-05-27T07:37:46.147962+0000","last_became_active":"2024-05-27T07:36:02.243785+0000","last_became_peered":"2024-05-27T07:36:02.243785+0000","last_unstale":"2024-05-27T07:37:46.147962+0000","last_undegraded":"2024-05-27T07:37:46.147962+0000","last_fullsized":"2024-05-27T07:37:46.147962+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:38.596510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144024+0000","last_change":"2024-05-27T07:36:04.979903+0000","last_active":"2024-05-27T07:37:46.144024+0000","last_peered":"2024-05-27T07:37:46.144024+0000","last_clean":"2024-05-27T07:37:46.144024+0000","last_became_active":"2024-05-27T07:36:04.979373+0000","last_became_peered":"2024-05-27T07:36:04.979373+0000","last_unstale":"2024-05-27T07:37:46.144024+0000","last_undegraded":"2024-05-27T07:37:46.144024+0000","last_fullsized":"2024-05-27T07:37:46.144024+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:44:33.763230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.317538+0000","last_change":"2024-05-27T07:36:15.023740+0000","last_active":"2024-05-27T07:37:46.317538+0000","last_peered":"2024-05-27T07:37:46.317538+0000","last_clean":"2024-05-27T07:37:46.317538+0000","last_became_active":"2024-05-27T07:36:15.023364+0000","last_became_peered":"2024-05-27T07:36:15.023364+0000","last_unstale":"2024-05-27T07:37:46.317538+0000","last_undegraded":"2024-05-27T07:37:46.317538+0000","last_fullsized":"2024-05-27T07:37:46.317538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:21.399065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'9","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142898+0000","last_change":"2024-05-27T07:36:01.951451+0000","last_active":"2024-05-27T07:37:46.142898+0000","last_peered":"2024-05-27T07:37:46.142898+0000","last_clean":"2024-05-27T07:37:46.142898+0000","last_became_active":"2024-05-27T07:36:01.951037+0000","last_became_peered":"2024-05-27T07:36:01.951037+0000","last_unstale":"2024-05-27T07:37:46.142898+0000","last_undegraded":"2024-05-27T07:37:46.142898+0000","last_fullsized":"2024-05-27T07:37:46.142898+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:44.202474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.316769+0000","last_change":"2024-05-27T07:36:04.975392+0000","last_active":"2024-05-27T07:37:46.316769+0000","last_peered":"2024-05-27T07:37:46.316769+0000","last_clean":"2024-05-27T07:37:46.316769+0000","last_became_active":"2024-05-27T07:36:04.975029+0000","last_became_peered":"2024-05-27T07:36:04.975029+0000","last_unstale":"2024-05-27T07:37:46.316769+0000","last_undegraded":"2024-05-27T07:37:46.316769+0000","last_fullsized":"2024-05-27T07:37:46.316769+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:32.022670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.140730+0000","last_change":"2024-05-27T07:36:15.026401+0000","last_active":"2024-05-27T07:37:46.140730+0000","last_peered":"2024-05-27T07:37:46.140730+0000","last_clean":"2024-05-27T07:37:46.140730+0000","last_became_active":"2024-05-27T07:36:15.025963+0000","last_became_peered":"2024-05-27T07:36:15.025963+0000","last_unstale":"2024-05-27T07:37:46.140730+0000","last_undegraded":"2024-05-27T07:37:46.140730+0000","last_fullsized":"2024-05-27T07:37:46.140730+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:33:47.616099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"78'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144868+0000","last_change":"2024-05-27T07:36:02.228776+0000","last_active":"2024-05-27T07:37:46.144868+0000","last_peered":"2024-05-27T07:37:46.144868+0000","last_clean":"2024-05-27T07:37:46.144868+0000","last_became_active":"2024-05-27T07:36:02.228356+0000","last_became_peered":"2024-05-27T07:36:02.228356+0000","last_unstale":"2024-05-27T07:37:46.144868+0000","last_undegraded":"2024-05-27T07:37:46.144868+0000","last_fullsized":"2024-05-27T07:37:46.144868+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:07.289340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.395975+0000","last_change":"2024-05-27T07:36:04.986917+0000","last_active":"2024-05-27T07:38:43.395975+0000","last_peered":"2024-05-27T07:38:43.395975+0000","last_clean":"2024-05-27T07:38:43.395975+0000","last_became_active":"2024-05-27T07:36:04.986432+0000","last_became_peered":"2024-05-27T07:36:04.986432+0000","last_unstale":"2024-05-27T07:38:43.395975+0000","last_undegraded":"2024-05-27T07:38:43.395975+0000","last_fullsized":"2024-05-27T07:38:43.395975+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:39:15.011058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144572+0000","last_change":"2024-05-27T07:36:15.032312+0000","last_active":"2024-05-27T07:37:46.144572+0000","last_peered":"2024-05-27T07:37:46.144572+0000","last_clean":"2024-05-27T07:37:46.144572+0000","last_became_active":"2024-05-27T07:36:15.031475+0000","last_became_peered":"2024-05-27T07:36:15.031475+0000","last_unstale":"2024-05-27T07:37:46.144572+0000","last_undegraded":"2024-05-27T07:37:46.144572+0000","last_fullsized":"2024-05-27T07:37:46.144572+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:40.099421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.314912+0000","last_change":"2024-05-27T07:36:02.243456+0000","last_active":"2024-05-27T07:37:46.314912+0000","last_peered":"2024-05-27T07:37:46.314912+0000","last_clean":"2024-05-27T07:37:46.314912+0000","last_became_active":"2024-05-27T07:36:02.243164+0000","last_became_peered":"2024-05-27T07:36:02.243164+0000","last_unstale":"2024-05-27T07:37:46.314912+0000","last_undegraded":"2024-05-27T07:37:46.314912+0000","last_fullsized":"2024-05-27T07:37:46.314912+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:33.765834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147453+0000","last_change":"2024-05-27T07:36:04.980192+0000","last_active":"2024-05-27T07:37:46.147453+0000","last_peered":"2024-05-27T07:37:46.147453+0000","last_clean":"2024-05-27T07:37:46.147453+0000","last_became_active":"2024-05-27T07:36:04.979797+0000","last_became_peered":"2024-05-27T07:36:04.979797+0000","last_unstale":"2024-05-27T07:37:46.147453+0000","last_undegraded":"2024-05-27T07:37:46.147453+0000","last_fullsized":"2024-05-27T07:37:46.147453+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:47:37.006746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.142748+0000","last_change":"2024-05-27T07:36:15.033153+0000","last_active":"2024-05-27T07:37:46.142748+0000","last_peered":"2024-05-27T07:37:46.142748+0000","last_clean":"2024-05-27T07:37:46.142748+0000","last_became_active":"2024-05-27T07:36:15.032783+0000","last_became_peered":"2024-05-27T07:36:15.032783+0000","last_unstale":"2024-05-27T07:37:46.142748+0000","last_undegraded":"2024-05-27T07:37:46.142748+0000","last_fullsized":"2024-05-27T07:37:46.142748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:31:26.237222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.318840+0000","last_change":"2024-05-27T07:36:01.952754+0000","last_active":"2024-05-27T07:37:46.318840+0000","last_peered":"2024-05-27T07:37:46.318840+0000","last_clean":"2024-05-27T07:37:46.318840+0000","last_became_active":"2024-05-27T07:36:01.952397+0000","last_became_peered":"2024-05-27T07:36:01.952397+0000","last_unstale":"2024-05-27T07:37:46.318840+0000","last_undegraded":"2024-05-27T07:37:46.318840+0000","last_fullsized":"2024-05-27T07:37:46.318840+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:04.350169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.394007+0000","last_change":"2024-05-27T07:36:04.979193+0000","last_active":"2024-05-27T07:38:43.394007+0000","last_peered":"2024-05-27T07:38:43.394007+0000","last_clean":"2024-05-27T07:38:43.394007+0000","last_became_active":"2024-05-27T07:36:04.978679+0000","last_became_peered":"2024-05-27T07:36:04.978679+0000","last_unstale":"2024-05-27T07:38:43.394007+0000","last_undegraded":"2024-05-27T07:38:43.394007+0000","last_fullsized":"2024-05-27T07:38:43.394007+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:59.310284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.154240+0000","last_change":"2024-05-27T07:37:46.153707+0000","last_active":"2024-05-27T07:37:46.154240+0000","last_peered":"2024-05-27T07:37:46.154240+0000","last_clean":"2024-05-27T07:37:46.154240+0000","last_became_active":"2024-05-27T07:37:46.148703+0000","last_became_peered":"2024-05-27T07:37:46.148703+0000","last_unstale":"2024-05-27T07:37:46.154240+0000","last_undegraded":"2024-05-27T07:37:46.154240+0000","last_fullsized":"2024-05-27T07:37:46.154240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:50:20.099467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145292+0000","last_change":"2024-05-27T07:36:45.140940+0000","last_active":"2024-05-27T07:37:46.145292+0000","last_peered":"2024-05-27T07:37:46.145292+0000","last_clean":"2024-05-27T07:37:46.145292+0000","last_became_active":"2024-05-27T07:36:45.140511+0000","last_became_peered":"2024-05-27T07:36:45.140511+0000","last_unstale":"2024-05-27T07:37:46.145292+0000","last_undegraded":"2024-05-27T07:37:46.145292+0000","last_fullsized":"2024-05-27T07:37:46.145292+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:43:00.507442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.316147+0000","last_change":"2024-05-27T07:36:04.974709+0000","last_active":"2024-05-27T07:37:46.316147+0000","last_peered":"2024-05-27T07:37:46.316147+0000","last_clean":"2024-05-27T07:37:46.316147+0000","last_became_active":"2024-05-27T07:36:04.974147+0000","last_became_peered":"2024-05-27T07:36:04.974147+0000","last_unstale":"2024-05-27T07:37:46.316147+0000","last_undegraded":"2024-05-27T07:37:46.316147+0000","last_fullsized":"2024-05-27T07:37:46.316147+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:05:11.251124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144424+0000","last_change":"2024-05-27T07:36:15.023187+0000","last_active":"2024-05-27T07:37:46.144424+0000","last_peered":"2024-05-27T07:37:46.144424+0000","last_clean":"2024-05-27T07:37:46.144424+0000","last_became_active":"2024-05-27T07:36:15.022687+0000","last_became_peered":"2024-05-27T07:36:15.022687+0000","last_unstale":"2024-05-27T07:37:46.144424+0000","last_undegraded":"2024-05-27T07:37:46.144424+0000","last_fullsized":"2024-05-27T07:37:46.144424+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:51.713222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"78'2","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147714+0000","last_change":"2024-05-27T07:36:01.954751+0000","last_active":"2024-05-27T07:37:46.147714+0000","last_peered":"2024-05-27T07:37:46.147714+0000","last_clean":"2024-05-27T07:37:46.147714+0000","last_became_active":"2024-05-27T07:36:01.954173+0000","last_became_peered":"2024-05-27T07:36:01.954173+0000","last_unstale":"2024-05-27T07:37:46.147714+0000","last_undegraded":"2024-05-27T07:37:46.147714+0000","last_fullsized":"2024-05-27T07:37:46.147714+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:45:10.504504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147029+0000","last_change":"2024-05-27T07:36:04.980294+0000","last_active":"2024-05-27T07:37:46.147029+0000","last_peered":"2024-05-27T07:37:46.147029+0000","last_clean":"2024-05-27T07:37:46.147029+0000","last_became_active":"2024-05-27T07:36:04.979853+0000","last_became_peered":"2024-05-27T07:36:04.979853+0000","last_unstale":"2024-05-27T07:37:46.147029+0000","last_undegraded":"2024-05-27T07:37:46.147029+0000","last_fullsized":"2024-05-27T07:37:46.147029+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:43.131216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315163+0000","last_change":"2024-05-27T07:36:15.025854+0000","last_active":"2024-05-27T07:37:46.315163+0000","last_peered":"2024-05-27T07:37:46.315163+0000","last_clean":"2024-05-27T07:37:46.315163+0000","last_became_active":"2024-05-27T07:36:15.025412+0000","last_became_peered":"2024-05-27T07:36:15.025412+0000","last_unstale":"2024-05-27T07:37:46.315163+0000","last_undegraded":"2024-05-27T07:37:46.315163+0000","last_fullsized":"2024-05-27T07:37:46.315163+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:49.904811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.141154+0000","last_change":"2024-05-27T07:36:02.243764+0000","last_active":"2024-05-27T07:37:46.141154+0000","last_peered":"2024-05-27T07:37:46.141154+0000","last_clean":"2024-05-27T07:37:46.141154+0000","last_became_active":"2024-05-27T07:36:02.242935+0000","last_became_peered":"2024-05-27T07:36:02.242935+0000","last_unstale":"2024-05-27T07:37:46.141154+0000","last_undegraded":"2024-05-27T07:37:46.141154+0000","last_fullsized":"2024-05-27T07:37:46.141154+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:27.535658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147851+0000","last_change":"2024-05-27T07:36:04.975291+0000","last_active":"2024-05-27T07:37:46.147851+0000","last_peered":"2024-05-27T07:37:46.147851+0000","last_clean":"2024-05-27T07:37:46.147851+0000","last_became_active":"2024-05-27T07:36:04.974868+0000","last_became_peered":"2024-05-27T07:36:04.974868+0000","last_unstale":"2024-05-27T07:37:46.147851+0000","last_undegraded":"2024-05-27T07:37:46.147851+0000","last_fullsized":"2024-05-27T07:37:46.147851+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:13.890494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.319226+0000","last_change":"2024-05-27T07:36:15.031812+0000","last_active":"2024-05-27T07:37:46.319226+0000","last_peered":"2024-05-27T07:37:46.319226+0000","last_clean":"2024-05-27T07:37:46.319226+0000","last_became_active":"2024-05-27T07:36:15.031399+0000","last_became_peered":"2024-05-27T07:36:15.031399+0000","last_unstale":"2024-05-27T07:37:46.319226+0000","last_undegraded":"2024-05-27T07:37:46.319226+0000","last_fullsized":"2024-05-27T07:37:46.319226+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.595579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315962+0000","last_change":"2024-05-27T07:36:01.954581+0000","last_active":"2024-05-27T07:37:46.315962+0000","last_peered":"2024-05-27T07:37:46.315962+0000","last_clean":"2024-05-27T07:37:46.315962+0000","last_became_active":"2024-05-27T07:36:01.953564+0000","last_became_peered":"2024-05-27T07:36:01.953564+0000","last_unstale":"2024-05-27T07:37:46.315962+0000","last_undegraded":"2024-05-27T07:37:46.315962+0000","last_fullsized":"2024-05-27T07:37:46.315962+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:56.866738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.147193+0000","last_change":"2024-05-27T07:36:45.141160+0000","last_active":"2024-05-27T07:37:46.147193+0000","last_peered":"2024-05-27T07:37:46.147193+0000","last_clean":"2024-05-27T07:37:46.147193+0000","last_became_active":"2024-05-27T07:36:45.140840+0000","last_became_peered":"2024-05-27T07:36:45.140840+0000","last_unstale":"2024-05-27T07:37:46.147193+0000","last_undegraded":"2024-05-27T07:37:46.147193+0000","last_fullsized":"2024-05-27T07:37:46.147193+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:31.676808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.315220+0000","last_change":"2024-05-27T07:36:15.026483+0000","last_active":"2024-05-27T07:37:46.315220+0000","last_peered":"2024-05-27T07:37:46.315220+0000","last_clean":"2024-05-27T07:37:46.315220+0000","last_became_active":"2024-05-27T07:36:15.025955+0000","last_became_peered":"2024-05-27T07:36:15.025955+0000","last_unstale":"2024-05-27T07:37:46.315220+0000","last_undegraded":"2024-05-27T07:37:46.315220+0000","last_fullsized":"2024-05-27T07:37:46.315220+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:25.468146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.145095+0000","last_change":"2024-05-27T07:36:01.952505+0000","last_active":"2024-05-27T07:37:46.145095+0000","last_peered":"2024-05-27T07:37:46.145095+0000","last_clean":"2024-05-27T07:37:46.145095+0000","last_became_active":"2024-05-27T07:36:01.952088+0000","last_became_peered":"2024-05-27T07:36:01.952088+0000","last_unstale":"2024-05-27T07:37:46.145095+0000","last_undegraded":"2024-05-27T07:37:46.145095+0000","last_fullsized":"2024-05-27T07:37:46.145095+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:09.235719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.148086+0000","last_change":"2024-05-27T07:36:04.974331+0000","last_active":"2024-05-27T07:37:46.148086+0000","last_peered":"2024-05-27T07:37:46.148086+0000","last_clean":"2024-05-27T07:37:46.148086+0000","last_became_active":"2024-05-27T07:36:04.973825+0000","last_became_peered":"2024-05-27T07:36:04.973825+0000","last_unstale":"2024-05-27T07:37:46.148086+0000","last_undegraded":"2024-05-27T07:37:46.148086+0000","last_fullsized":"2024-05-27T07:37:46.148086+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:39:42.917309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.390376+0000","last_change":"2024-05-27T07:36:45.141224+0000","last_active":"2024-05-27T07:38:43.390376+0000","last_peered":"2024-05-27T07:38:43.390376+0000","last_clean":"2024-05-27T07:38:43.390376+0000","last_became_active":"2024-05-27T07:36:45.140776+0000","last_became_peered":"2024-05-27T07:36:45.140776+0000","last_unstale":"2024-05-27T07:38:43.390376+0000","last_undegraded":"2024-05-27T07:38:43.390376+0000","last_fullsized":"2024-05-27T07:38:43.390376+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:36:59.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.400919+0000","last_change":"2024-05-27T07:36:04.988247+0000","last_active":"2024-05-27T07:38:43.400919+0000","last_peered":"2024-05-27T07:38:43.400919+0000","last_clean":"2024-05-27T07:38:43.400919+0000","last_became_active":"2024-05-27T07:36:04.987825+0000","last_became_peered":"2024-05-27T07:36:04.987825+0000","last_unstale":"2024-05-27T07:38:43.400919+0000","last_undegraded":"2024-05-27T07:38:43.400919+0000","last_fullsized":"2024-05-27T07:38:43.400919+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:32.499260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.375932+0000","last_change":"2024-05-27T07:36:04.979395+0000","last_active":"2024-05-27T07:38:43.375932+0000","last_peered":"2024-05-27T07:38:43.375932+0000","last_clean":"2024-05-27T07:38:43.375932+0000","last_became_active":"2024-05-27T07:36:04.978983+0000","last_became_peered":"2024-05-27T07:36:04.978983+0000","last_unstale":"2024-05-27T07:38:43.375932+0000","last_undegraded":"2024-05-27T07:38:43.375932+0000","last_fullsized":"2024-05-27T07:38:43.375932+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:53:54.071958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.396718+0000","last_change":"2024-05-27T07:36:04.989442+0000","last_active":"2024-05-27T07:38:43.396718+0000","last_peered":"2024-05-27T07:38:43.396718+0000","last_clean":"2024-05-27T07:38:43.396718+0000","last_became_active":"2024-05-27T07:36:04.989079+0000","last_became_peered":"2024-05-27T07:36:04.989079+0000","last_unstale":"2024-05-27T07:38:43.396718+0000","last_undegraded":"2024-05-27T07:38:43.396718+0000","last_fullsized":"2024-05-27T07:38:43.396718+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:39.411255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.144128+0000","last_change":"2024-05-27T07:36:04.981289+0000","last_active":"2024-05-27T07:37:46.144128+0000","last_peered":"2024-05-27T07:37:46.144128+0000","last_clean":"2024-05-27T07:37:46.144128+0000","last_became_active":"2024-05-27T07:36:04.980924+0000","last_became_peered":"2024-05-27T07:36:04.980924+0000","last_unstale":"2024-05-27T07:37:46.144128+0000","last_undegraded":"2024-05-27T07:37:46.144128+0000","last_fullsized":"2024-05-27T07:37:46.144128+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:15:04.741303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.396696+0000","last_change":"2024-05-27T07:36:04.985712+0000","last_active":"2024-05-27T07:38:43.396696+0000","last_peered":"2024-05-27T07:38:43.396696+0000","last_clean":"2024-05-27T07:38:43.396696+0000","last_became_active":"2024-05-27T07:36:04.983786+0000","last_became_peered":"2024-05-27T07:36:04.983786+0000","last_unstale":"2024-05-27T07:38:43.396696+0000","last_undegraded":"2024-05-27T07:38:43.396696+0000","last_fullsized":"2024-05-27T07:38:43.396696+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:34.549557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.388723+0000","last_change":"2024-05-27T07:36:04.991176+0000","last_active":"2024-05-27T07:38:43.388723+0000","last_peered":"2024-05-27T07:38:43.388723+0000","last_clean":"2024-05-27T07:38:43.388723+0000","last_became_active":"2024-05-27T07:36:04.990774+0000","last_became_peered":"2024-05-27T07:36:04.990774+0000","last_unstale":"2024-05-27T07:38:43.388723+0000","last_undegraded":"2024-05-27T07:38:43.388723+0000","last_fullsized":"2024-05-27T07:38:43.388723+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:37.307581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.316933+0000","last_change":"2024-05-27T07:36:04.982337+0000","last_active":"2024-05-27T07:37:46.316933+0000","last_peered":"2024-05-27T07:37:46.316933+0000","last_clean":"2024-05-27T07:37:46.316933+0000","last_became_active":"2024-05-27T07:36:04.981988+0000","last_became_peered":"2024-05-27T07:36:04.981988+0000","last_unstale":"2024-05-27T07:37:46.316933+0000","last_undegraded":"2024-05-27T07:37:46.316933+0000","last_fullsized":"2024-05-27T07:37:46.316933+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:17:39.763459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.399725+0000","last_change":"2024-05-27T07:36:04.982321+0000","last_active":"2024-05-27T07:38:43.399725+0000","last_peered":"2024-05-27T07:38:43.399725+0000","last_clean":"2024-05-27T07:38:43.399725+0000","last_became_active":"2024-05-27T07:36:04.981690+0000","last_became_peered":"2024-05-27T07:36:04.981690+0000","last_unstale":"2024-05-27T07:38:43.399725+0000","last_undegraded":"2024-05-27T07:38:43.399725+0000","last_fullsized":"2024-05-27T07:38:43.399725+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:10.181606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.318945+0000","last_change":"2024-05-27T07:36:04.979411+0000","last_active":"2024-05-27T07:37:46.318945+0000","last_peered":"2024-05-27T07:37:46.318945+0000","last_clean":"2024-05-27T07:37:46.318945+0000","last_became_active":"2024-05-27T07:36:04.979041+0000","last_became_peered":"2024-05-27T07:36:04.979041+0000","last_unstale":"2024-05-27T07:37:46.318945+0000","last_undegraded":"2024-05-27T07:37:46.318945+0000","last_fullsized":"2024-05-27T07:37:46.318945+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:57:29.165997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.402939+0000","last_change":"2024-05-27T07:36:04.988564+0000","last_active":"2024-05-27T07:38:43.402939+0000","last_peered":"2024-05-27T07:38:43.402939+0000","last_clean":"2024-05-27T07:38:43.402939+0000","last_became_active":"2024-05-27T07:36:04.986313+0000","last_became_peered":"2024-05-27T07:36:04.986313+0000","last_unstale":"2024-05-27T07:38:43.402939+0000","last_undegraded":"2024-05-27T07:38:43.402939+0000","last_fullsized":"2024-05-27T07:38:43.402939+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:38.809386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:37:46.146131+0000","last_change":"2024-05-27T07:36:04.974704+0000","last_active":"2024-05-27T07:37:46.146131+0000","last_peered":"2024-05-27T07:37:46.146131+0000","last_clean":"2024-05-27T07:37:46.146131+0000","last_became_active":"2024-05-27T07:36:04.974311+0000","last_became_peered":"2024-05-27T07:36:04.974311+0000","last_unstale":"2024-05-27T07:37:46.146131+0000","last_undegraded":"2024-05-27T07:37:46.146131+0000","last_fullsized":"2024-05-27T07:37:46.146131+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:43.021149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:38:43.383739+0000","last_change":"2024-05-27T07:36:04.983951+0000","last_active":"2024-05-27T07:38:43.383739+0000","last_peered":"2024-05-27T07:38:43.383739+0000","last_clean":"2024-05-27T07:38:43.383739+0000","last_became_active":"2024-05-27T07:36:04.983489+0000","last_became_peered":"2024-05-27T07:36:04.983489+0000","last_unstale":"2024-05-27T07:38:43.383739+0000","last_undegraded":"2024-05-27T07:38:43.383739+0000","last_fullsized":"2024-05-27T07:38:43.383739+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:04.229523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":259,"ondisk_log_size":259,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280976,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65443,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":211,"num_read_kb":172,"num_write":87,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":196827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710768,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":514628,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906996,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":1169813,"data_compressed":68160,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070522,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":515188,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266751,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":515188,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462980,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":1109652,"data_compressed":68160,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626506,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":510510,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":575404,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":510600,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":575189,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182490,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":510614,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986261,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":514628,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822735,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1105156,"data_compressed":68160,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34107,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34107,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34107,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:38:46.308 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-27T07:38:46.308 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-27T07:38:46.309 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-27T07:38:46.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-05-27T07:38:46.552 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:47.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:47 smithi114 ceph-mon[32038]: pgmap v380: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:47.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:47 smithi114 ceph-mon[32038]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:38:47.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:47 smithi163 ceph-mon[36451]: pgmap v380: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:47.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:47 smithi163 ceph-mon[36451]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:38:47.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:47 smithi186 ceph-mon[36647]: pgmap v380: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:38:47.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:47 smithi186 ceph-mon[36647]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:38:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.337+0000 7fc4c1e74700 1 -- 172.21.15.114:0/573159801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4bc109860 msgr2=0x7fc4bc1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.337+0000 7fc4c1e74700 1 --2- 172.21.15.114:0/573159801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4bc109860 0x7fc4bc1111c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400b0d0 tx=0x7fc4b400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:38:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.337+0000 7fc4c1e74700 1 -- 172.21.15.114:0/573159801 shutdown_connections 2024-05-27T07:38:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.337+0000 7fc4c1e74700 1 --2- 172.21.15.114:0/573159801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4bc109860 0x7fc4bc1111c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.337+0000 7fc4c1e74700 1 --2- 172.21.15.114:0/573159801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc1045e0 0x7fc4bc1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.337+0000 7fc4c1e74700 1 --2- 172.21.15.114:0/573159801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4bc108f10 0x7fc4bc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.337+0000 7fc4c1e74700 1 -- 172.21.15.114:0/573159801 >> 172.21.15.114:0/573159801 conn(0x7fc4bc0fffd0 msgr2=0x7fc4bc1023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.337+0000 7fc4c1e74700 1 -- 172.21.15.114:0/573159801 shutdown_connections 2024-05-27T07:38:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.337+0000 7fc4c1e74700 1 -- 172.21.15.114:0/573159801 wait complete. 2024-05-27T07:38:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.338+0000 7fc4c1e74700 1 Processor -- start 2024-05-27T07:38:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.338+0000 7fc4c1e74700 1 -- start start 2024-05-27T07:38:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.338+0000 7fc4c1e74700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4bc1045e0 0x7fc4bc19ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4c1e74700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc108f10 0x7fc4bc19b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4c1e74700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4bc109860 0x7fc4bc19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4c1e74700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc1144e0 con 0x7fc4bc1045e0 2024-05-27T07:38:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4c1e74700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc114360 con 0x7fc4bc108f10 2024-05-27T07:38:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4c1e74700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc114660 con 0x7fc4bc109860 2024-05-27T07:38:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4bb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4bc1045e0 0x7fc4bc19ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4bb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4bc1045e0 0x7fc4bc19ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57488/0 (socket says 172.21.15.114:57488) 2024-05-27T07:38:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4bb7fe700 1 -- 172.21.15.114:0/1224525669 learned_addr learned my addr 172.21.15.114:0/1224525669 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4b3fff700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc108f10 0x7fc4bc19b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4bbfff700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4bc109860 0x7fc4bc19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4bb7fe700 1 -- 172.21.15.114:0/1224525669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4bc109860 msgr2=0x7fc4bc19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4bb7fe700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4bc109860 0x7fc4bc19f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4bb7fe700 1 -- 172.21.15.114:0/1224525669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc108f10 msgr2=0x7fc4bc19b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4bb7fe700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc108f10 0x7fc4bc19b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4bb7fe700 1 -- 172.21.15.114:0/1224525669 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4bc19fca0 con 0x7fc4bc1045e0 2024-05-27T07:38:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4bbfff700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4bc109860 0x7fc4bc19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4bb7fe700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4bc1045e0 0x7fc4bc19ae70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac00c920 tx=0x7fc4ac00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4b97fa700 1 -- 172.21.15.114:0/1224525669 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac007b00 con 0x7fc4bc1045e0 2024-05-27T07:38:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4b97fa700 1 -- 172.21.15.114:0/1224525669 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4ac004ce0 con 0x7fc4bc1045e0 2024-05-27T07:38:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4b97fa700 1 -- 172.21.15.114:0/1224525669 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac005650 con 0x7fc4bc1045e0 2024-05-27T07:38:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4c1e74700 1 -- 172.21.15.114:0/1224525669 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4bc19ff90 con 0x7fc4bc1045e0 2024-05-27T07:38:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.339+0000 7fc4c1e74700 1 -- 172.21.15.114:0/1224525669 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4bc1a78b0 con 0x7fc4bc1045e0 2024-05-27T07:38:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.341+0000 7fc4c1e74700 1 -- 172.21.15.114:0/1224525669 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4bc0512b0 con 0x7fc4bc1045e0 2024-05-27T07:38:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.342+0000 7fc4b97fa700 1 -- 172.21.15.114:0/1224525669 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fc4ac004750 con 0x7fc4bc1045e0 2024-05-27T07:38:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.343+0000 7fc4b97fa700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc4a80b2880 0x7fc4a80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.343+0000 7fc4b3fff700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc4a80b2880 0x7fc4a80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.343+0000 7fc4b97fa700 1 -- 172.21.15.114:0/1224525669 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fc4ac0e25a0 con 0x7fc4bc1045e0 2024-05-27T07:38:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.343+0000 7fc4b3fff700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc4a80b2880 0x7fc4a80b4d90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4005fd0 tx=0x7fc4a4005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.347+0000 7fc4b97fa700 1 -- 172.21.15.114:0/1224525669 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc4ac014030 con 0x7fc4bc1045e0 2024-05-27T07:38:48.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.595+0000 7fc4c1e74700 1 -- 172.21.15.114:0/1224525669 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc4bc02e5e0 con 0x7fc4bc1045e0 2024-05-27T07:38:48.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.596+0000 7fc4b97fa700 1 -- 172.21.15.114:0/1224525669 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fc4ac0a9ef0 con 0x7fc4bc1045e0 2024-05-27T07:38:48.594 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:38:48.594 INFO:teuthology.orchestra.run.smithi114.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-27T07:38:48.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.598+0000 7fc4c1e74700 1 -- 172.21.15.114:0/1224525669 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc4a80b2880 msgr2=0x7fc4a80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:48.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.598+0000 7fc4c1e74700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc4a80b2880 0x7fc4a80b4d90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4005fd0 tx=0x7fc4a4005ee0 comp rx=0 tx=0).stop 2024-05-27T07:38:48.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.598+0000 7fc4c1e74700 1 -- 172.21.15.114:0/1224525669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4bc1045e0 msgr2=0x7fc4bc19ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:48.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.598+0000 7fc4c1e74700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4bc1045e0 0x7fc4bc19ae70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac00c920 tx=0x7fc4ac00cce0 comp rx=0 tx=0).stop 2024-05-27T07:38:48.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.598+0000 7fc4c1e74700 1 -- 172.21.15.114:0/1224525669 shutdown_connections 2024-05-27T07:38:48.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.598+0000 7fc4c1e74700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc4a80b2880 0x7fc4a80b4d90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:48.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.598+0000 7fc4c1e74700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4bc109860 0x7fc4bc19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:48.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.598+0000 7fc4c1e74700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4bc1045e0 0x7fc4bc19ae70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:48.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.598+0000 7fc4c1e74700 1 --2- 172.21.15.114:0/1224525669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc108f10 0x7fc4bc19b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:48.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.598+0000 7fc4c1e74700 1 -- 172.21.15.114:0/1224525669 >> 172.21.15.114:0/1224525669 conn(0x7fc4bc0fffd0 msgr2=0x7fc4bc106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:48.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.598+0000 7fc4c1e74700 1 -- 172.21.15.114:0/1224525669 shutdown_connections 2024-05-27T07:38:48.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:48.598+0000 7fc4c1e74700 1 -- 172.21.15.114:0/1224525669 wait complete. 2024-05-27T07:38:49.236 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:49 smithi114 ceph-mon[32038]: pgmap v381: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:49.236 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:49 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1224525669' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T07:38:49.262 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-27T07:38:49.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:38:49.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:49 smithi163 ceph-mon[36451]: pgmap v381: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:49.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:49 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1224525669' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T07:38:49.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:49 smithi186 ceph-mon[36647]: pgmap v381: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:49.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:49 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1224525669' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T07:38:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:51.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.271+0000 7fb5ce008700 1 -- 172.21.15.114:0/1033764273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5c8108f30 msgr2=0x7fb5c8109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.271+0000 7fb5ce008700 1 --2- 172.21.15.114:0/1033764273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5c8108f30 0x7fb5c8109340 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00099c0 tx=0x7fb5b0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.271+0000 7fb5ce008700 1 -- 172.21.15.114:0/1033764273 shutdown_connections 2024-05-27T07:38:51.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.271+0000 7fb5ce008700 1 --2- 172.21.15.114:0/1033764273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5c8109880 0x7fb5c81111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.271+0000 7fb5ce008700 1 --2- 172.21.15.114:0/1033764273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5c8104600 0x7fb5c8104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.271+0000 7fb5ce008700 1 --2- 172.21.15.114:0/1033764273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5c8108f30 0x7fb5c8109340 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.271+0000 7fb5ce008700 1 -- 172.21.15.114:0/1033764273 >> 172.21.15.114:0/1033764273 conn(0x7fb5c807af60 msgr2=0x7fb5c807b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:51.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.271+0000 7fb5ce008700 1 -- 172.21.15.114:0/1033764273 shutdown_connections 2024-05-27T07:38:51.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.272+0000 7fb5ce008700 1 -- 172.21.15.114:0/1033764273 wait complete. 2024-05-27T07:38:51.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.272+0000 7fb5ce008700 1 Processor -- start 2024-05-27T07:38:51.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.272+0000 7fb5ce008700 1 -- start start 2024-05-27T07:38:51.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.272+0000 7fb5ce008700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5c8104600 0x7fb5c819d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.272+0000 7fb5ce008700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5c8108f30 0x7fb5c819d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.272+0000 7fb5ce008700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5c8109880 0x7fb5c81a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.272+0000 7fb5ce008700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c8114490 con 0x7fb5c8104600 2024-05-27T07:38:51.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.272+0000 7fb5ce008700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c8114310 con 0x7fb5c8109880 2024-05-27T07:38:51.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.272+0000 7fb5ce008700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c8114610 con 0x7fb5c8108f30 2024-05-27T07:38:51.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5c77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5c8104600 0x7fb5c819d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5c77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5c8104600 0x7fb5c819d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57514/0 (socket says 172.21.15.114:57514) 2024-05-27T07:38:51.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5c77fe700 1 -- 172.21.15.114:0/514965418 learned_addr learned my addr 172.21.15.114:0/514965418 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:51.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5bffff700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5c8108f30 0x7fb5c819d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5c7fff700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5c8109880 0x7fb5c81a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5c77fe700 1 -- 172.21.15.114:0/514965418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5c8108f30 msgr2=0x7fb5c819d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5c77fe700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5c8108f30 0x7fb5c819d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5c77fe700 1 -- 172.21.15.114:0/514965418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5c8109880 msgr2=0x7fb5c81a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5c77fe700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5c8109880 0x7fb5c81a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5c77fe700 1 -- 172.21.15.114:0/514965418 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c81a1f90 con 0x7fb5c8104600 2024-05-27T07:38:51.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5c77fe700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5c8104600 0x7fb5c819d0c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800d950 tx=0x7fb5b800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:51.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5c57fa700 1 -- 172.21.15.114:0/514965418 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8016070 con 0x7fb5c8104600 2024-05-27T07:38:51.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5ce008700 1 -- 172.21.15.114:0/514965418 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5c81a2280 con 0x7fb5c8104600 2024-05-27T07:38:51.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5c57fa700 1 -- 172.21.15.114:0/514965418 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5b8009df0 con 0x7fb5c8104600 2024-05-27T07:38:51.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5c57fa700 1 -- 172.21.15.114:0/514965418 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b800f3c0 con 0x7fb5c8104600 2024-05-27T07:38:51.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.273+0000 7fb5ce008700 1 -- 172.21.15.114:0/514965418 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5c81a9ba0 con 0x7fb5c8104600 2024-05-27T07:38:51.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.274+0000 7fb5ce008700 1 -- 172.21.15.114:0/514965418 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5c80512b0 con 0x7fb5c8104600 2024-05-27T07:38:51.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.276+0000 7fb5c57fa700 1 -- 172.21.15.114:0/514965418 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fb5b8019070 con 0x7fb5c8104600 2024-05-27T07:38:51.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.276+0000 7fb5c57fa700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb5b40b2880 0x7fb5b40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.277+0000 7fb5bffff700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb5b40b2880 0x7fb5b40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.277+0000 7fb5c57fa700 1 -- 172.21.15.114:0/514965418 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fb5b80e2200 con 0x7fb5c8104600 2024-05-27T07:38:51.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.277+0000 7fb5bffff700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb5b40b2880 0x7fb5b40b4d90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0006010 tx=0x7fb5b0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:51.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.281+0000 7fb5c57fa700 1 -- 172.21.15.114:0/514965418 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb5b80a9b50 con 0x7fb5c8104600 2024-05-27T07:38:51.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:51 smithi114 ceph-mon[32038]: pgmap v382: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:51.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:51 smithi163 ceph-mon[36451]: pgmap v382: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:51.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:51 smithi186 ceph-mon[36647]: pgmap v382: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:51.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.534+0000 7fb5ce008700 1 -- 172.21.15.114:0/514965418 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5c81a9d70 con 0x7fb5c8104600 2024-05-27T07:38:51.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.535+0000 7fb5c57fa700 1 -- 172.21.15.114:0/514965418 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11827 (secure 0 0 0) 0x7fb5b80a9450 con 0x7fb5c8104600 2024-05-27T07:38:51.532 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:38:51.532 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:37:06.645274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:38:51.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.537+0000 7fb5ce008700 1 -- 172.21.15.114:0/514965418 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb5b40b2880 msgr2=0x7fb5b40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.537+0000 7fb5ce008700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb5b40b2880 0x7fb5b40b4d90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0006010 tx=0x7fb5b0000bc0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.537+0000 7fb5ce008700 1 -- 172.21.15.114:0/514965418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5c8104600 msgr2=0x7fb5c819d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.537+0000 7fb5ce008700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5c8104600 0x7fb5c819d0c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800d950 tx=0x7fb5b800dc60 comp rx=0 tx=0).stop 2024-05-27T07:38:51.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.537+0000 7fb5ce008700 1 -- 172.21.15.114:0/514965418 shutdown_connections 2024-05-27T07:38:51.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.537+0000 7fb5ce008700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5c8109880 0x7fb5c81a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.537+0000 7fb5ce008700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5c8104600 0x7fb5c819d0c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.537+0000 7fb5ce008700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb5b40b2880 0x7fb5b40b4d90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.537+0000 7fb5ce008700 1 --2- 172.21.15.114:0/514965418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5c8108f30 0x7fb5c819d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.537+0000 7fb5ce008700 1 -- 172.21.15.114:0/514965418 >> 172.21.15.114:0/514965418 conn(0x7fb5c807af60 msgr2=0x7fb5c8110eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:51.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.537+0000 7fb5ce008700 1 -- 172.21.15.114:0/514965418 shutdown_connections 2024-05-27T07:38:51.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:51.537+0000 7fb5ce008700 1 -- 172.21.15.114:0/514965418 wait complete. 2024-05-27T07:38:51.536 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:38:52.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:38:52.438 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:52.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:52 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/514965418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:38:52.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:52 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/514965418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:38:52.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:52 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/514965418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:38:53.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:53 smithi163 ceph-mon[36451]: pgmap v383: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:53.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:53 smithi186 ceph-mon[36647]: pgmap v383: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:53.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:53 smithi114 ceph-mon[32038]: pgmap v383: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:54.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.233+0000 7fad8d7d6700 1 -- 172.21.15.114:0/3028314446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad881034b0 msgr2=0x7fad881038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:54.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.233+0000 7fad8d7d6700 1 --2- 172.21.15.114:0/3028314446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad881034b0 0x7fad881038c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fad780099c0 tx=0x7fad78009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:54.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.233+0000 7fad8d7d6700 1 -- 172.21.15.114:0/3028314446 shutdown_connections 2024-05-27T07:38:54.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.233+0000 7fad8d7d6700 1 --2- 172.21.15.114:0/3028314446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad88108730 0x7fad88110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:54.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.233+0000 7fad8d7d6700 1 --2- 172.21.15.114:0/3028314446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad881034b0 0x7fad881038c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:54.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.233+0000 7fad8d7d6700 1 --2- 172.21.15.114:0/3028314446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad88107de0 0x7fad881081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:54.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.233+0000 7fad8d7d6700 1 -- 172.21.15.114:0/3028314446 >> 172.21.15.114:0/3028314446 conn(0x7fad880fedb0 msgr2=0x7fad881011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:54.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.233+0000 7fad8d7d6700 1 -- 172.21.15.114:0/3028314446 shutdown_connections 2024-05-27T07:38:54.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.233+0000 7fad8d7d6700 1 -- 172.21.15.114:0/3028314446 wait complete. 2024-05-27T07:38:54.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.233+0000 7fad8d7d6700 1 Processor -- start 2024-05-27T07:38:54.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.233+0000 7fad8d7d6700 1 -- start start 2024-05-27T07:38:54.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad8d7d6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad881034b0 0x7fad8819f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:54.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad8d7d6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad88107de0 0x7fad8819fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:54.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad8d7d6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad88108730 0x7fad88199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:54.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad8d7d6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fad88113360 con 0x7fad881034b0 2024-05-27T07:38:54.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad8d7d6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fad881131e0 con 0x7fad88108730 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad8d7d6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fad881134e0 con 0x7fad88107de0 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad86ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad881034b0 0x7fad8819f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad86ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad881034b0 0x7fad8819f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57534/0 (socket says 172.21.15.114:57534) 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad86ffd700 1 -- 172.21.15.114:0/2505024444 learned_addr learned my addr 172.21.15.114:0/2505024444 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad877fe700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad88108730 0x7fad88199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad7ffff700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad88107de0 0x7fad8819fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad86ffd700 1 -- 172.21.15.114:0/2505024444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad88107de0 msgr2=0x7fad8819fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad86ffd700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad88107de0 0x7fad8819fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad86ffd700 1 -- 172.21.15.114:0/2505024444 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad88108730 msgr2=0x7fad88199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad86ffd700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad88108730 0x7fad88199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad86ffd700 1 -- 172.21.15.114:0/2505024444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8819a270 con 0x7fad881034b0 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad7ffff700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad88107de0 0x7fad8819fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.234+0000 7fad86ffd700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad881034b0 0x7fad8819f9b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fad7800b580 tx=0x7fad780047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.235+0000 7fad84ff9700 1 -- 172.21.15.114:0/2505024444 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78027070 con 0x7fad881034b0 2024-05-27T07:38:54.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.235+0000 7fad84ff9700 1 -- 172.21.15.114:0/2505024444 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad78003f20 con 0x7fad881034b0 2024-05-27T07:38:54.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.235+0000 7fad8d7d6700 1 -- 172.21.15.114:0/2505024444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad8819a500 con 0x7fad881034b0 2024-05-27T07:38:54.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.235+0000 7fad84ff9700 1 -- 172.21.15.114:0/2505024444 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78022d70 con 0x7fad881034b0 2024-05-27T07:38:54.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.235+0000 7fad8d7d6700 1 -- 172.21.15.114:0/2505024444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad881a6760 con 0x7fad881034b0 2024-05-27T07:38:54.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.236+0000 7fad8d7d6700 1 -- 172.21.15.114:0/2505024444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad8819a9e0 con 0x7fad881034b0 2024-05-27T07:38:54.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.237+0000 7fad84ff9700 1 -- 172.21.15.114:0/2505024444 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fad78022500 con 0x7fad881034b0 2024-05-27T07:38:54.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.238+0000 7fad84ff9700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fad740b2880 0x7fad740b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:54.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.238+0000 7fad7ffff700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fad740b2880 0x7fad740b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:54.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.238+0000 7fad84ff9700 1 -- 172.21.15.114:0/2505024444 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fad780e3700 con 0x7fad881034b0 2024-05-27T07:38:54.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.238+0000 7fad7ffff700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fad740b2880 0x7fad740b4d90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fad70005fd0 tx=0x7fad70005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:54.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.241+0000 7fad84ff9700 1 -- 172.21.15.114:0/2505024444 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fad78013070 con 0x7fad881034b0 2024-05-27T07:38:54.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.489+0000 7fad8d7d6700 1 -- 172.21.15.114:0/2505024444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad8819afe0 con 0x7fad881034b0 2024-05-27T07:38:54.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.490+0000 7fad84ff9700 1 -- 172.21.15.114:0/2505024444 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11827 (secure 0 0 0) 0x7fad780ab0e0 con 0x7fad881034b0 2024-05-27T07:38:54.488 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:38:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:37:06.645274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:38:54.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.492+0000 7fad8d7d6700 1 -- 172.21.15.114:0/2505024444 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fad740b2880 msgr2=0x7fad740b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:54.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.492+0000 7fad8d7d6700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fad740b2880 0x7fad740b4d90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fad70005fd0 tx=0x7fad70005f00 comp rx=0 tx=0).stop 2024-05-27T07:38:54.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.493+0000 7fad8d7d6700 1 -- 172.21.15.114:0/2505024444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad881034b0 msgr2=0x7fad8819f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:54.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.493+0000 7fad8d7d6700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad881034b0 0x7fad8819f9b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fad7800b580 tx=0x7fad780047b0 comp rx=0 tx=0).stop 2024-05-27T07:38:54.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.493+0000 7fad8d7d6700 1 -- 172.21.15.114:0/2505024444 shutdown_connections 2024-05-27T07:38:54.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.493+0000 7fad8d7d6700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad88108730 0x7fad88199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:54.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.493+0000 7fad8d7d6700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad881034b0 0x7fad8819f9b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:54.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.493+0000 7fad8d7d6700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fad740b2880 0x7fad740b4d90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:54.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.493+0000 7fad8d7d6700 1 --2- 172.21.15.114:0/2505024444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad88107de0 0x7fad8819fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:54.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.493+0000 7fad8d7d6700 1 -- 172.21.15.114:0/2505024444 >> 172.21.15.114:0/2505024444 conn(0x7fad880fedb0 msgr2=0x7fad88109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:54.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.493+0000 7fad8d7d6700 1 -- 172.21.15.114:0/2505024444 shutdown_connections 2024-05-27T07:38:54.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:54.493+0000 7fad8d7d6700 1 -- 172.21.15.114:0/2505024444 wait complete. 2024-05-27T07:38:54.491 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:38:55.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:38:55.589 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:55 smithi186 ceph-mon[36647]: pgmap v384: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:55.589 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:55 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2505024444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:38:55.589 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:55 smithi163 ceph-mon[36451]: pgmap v384: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:55.589 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:55 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2505024444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:38:55.589 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:55 smithi114 ceph-mon[32038]: pgmap v384: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:55.589 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:55 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2505024444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:38:55.832 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:57.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:57 smithi114 ceph-mon[32038]: pgmap v385: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:57.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:57 smithi163 ceph-mon[36451]: pgmap v385: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:57.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:57 smithi186 ceph-mon[36647]: pgmap v385: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:57.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.545+0000 7fc57fec1700 1 -- 172.21.15.114:0/1468492887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5781089e0 msgr2=0x7fc578108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:57.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.545+0000 7fc57fec1700 1 --2- 172.21.15.114:0/1468492887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5781089e0 0x7fc578108df0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc57000c210 tx=0x7fc57000c520 comp rx=0 tx=0).stop 2024-05-27T07:38:57.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.545+0000 7fc57fec1700 1 -- 172.21.15.114:0/1468492887 shutdown_connections 2024-05-27T07:38:57.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.545+0000 7fc57fec1700 1 --2- 172.21.15.114:0/1468492887 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc578103300 0x7fc57810f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:57.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.545+0000 7fc57fec1700 1 --2- 172.21.15.114:0/1468492887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5781089e0 0x7fc578108df0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:57.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.545+0000 7fc57fec1700 1 --2- 172.21.15.114:0/1468492887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5781029b0 0x7fc578102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:57.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.545+0000 7fc57fec1700 1 -- 172.21.15.114:0/1468492887 >> 172.21.15.114:0/1468492887 conn(0x7fc5780fe6e0 msgr2=0x7fc578100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:57.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.545+0000 7fc57fec1700 1 -- 172.21.15.114:0/1468492887 shutdown_connections 2024-05-27T07:38:57.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.545+0000 7fc57fec1700 1 -- 172.21.15.114:0/1468492887 wait complete. 2024-05-27T07:38:57.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.545+0000 7fc57fec1700 1 Processor -- start 2024-05-27T07:38:57.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57fec1700 1 -- start start 2024-05-27T07:38:57.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57fec1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5781029b0 0x7fc57819ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:57.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57fec1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc578103300 0x7fc57819b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:57.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57fec1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5781089e0 0x7fc57819f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:57.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57fec1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc5781122d0 con 0x7fc5781029b0 2024-05-27T07:38:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57fec1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc578112150 con 0x7fc578103300 2024-05-27T07:38:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57fec1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc578112450 con 0x7fc5781089e0 2024-05-27T07:38:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57dc5d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5781029b0 0x7fc57819ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57dc5d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5781029b0 0x7fc57819ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55526/0 (socket says 172.21.15.114:55526) 2024-05-27T07:38:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57dc5d700 1 -- 172.21.15.114:0/1002222676 learned_addr learned my addr 172.21.15.114:0/1002222676 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:38:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57e45e700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5781089e0 0x7fc57819f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57d45c700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc578103300 0x7fc57819b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57dc5d700 1 -- 172.21.15.114:0/1002222676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5781089e0 msgr2=0x7fc57819f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57dc5d700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5781089e0 0x7fc57819f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57dc5d700 1 -- 172.21.15.114:0/1002222676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc578103300 msgr2=0x7fc57819b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57dc5d700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc578103300 0x7fc57819b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57dc5d700 1 -- 172.21.15.114:0/1002222676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc57819fd00 con 0x7fc5781029b0 2024-05-27T07:38:57.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57d45c700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc578103300 0x7fc57819b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:57.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57e45e700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5781089e0 0x7fc57819f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:57.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.546+0000 7fc57dc5d700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5781029b0 0x7fc57819ae30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc5700082f0 tx=0x7fc5700083d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:57.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.547+0000 7fc576ffd700 1 -- 172.21.15.114:0/1002222676 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570022070 con 0x7fc5781029b0 2024-05-27T07:38:57.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.547+0000 7fc576ffd700 1 -- 172.21.15.114:0/1002222676 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc570008ae0 con 0x7fc5781029b0 2024-05-27T07:38:57.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.547+0000 7fc576ffd700 1 -- 172.21.15.114:0/1002222676 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57001e850 con 0x7fc5781029b0 2024-05-27T07:38:57.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.547+0000 7fc57fec1700 1 -- 172.21.15.114:0/1002222676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc57819ff90 con 0x7fc5781029b0 2024-05-27T07:38:57.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.547+0000 7fc57fec1700 1 -- 172.21.15.114:0/1002222676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc578198980 con 0x7fc5781029b0 2024-05-27T07:38:57.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.548+0000 7fc57fec1700 1 -- 172.21.15.114:0/1002222676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5781040f0 con 0x7fc5781029b0 2024-05-27T07:38:57.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.549+0000 7fc576ffd700 1 -- 172.21.15.114:0/1002222676 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fc570025070 con 0x7fc5781029b0 2024-05-27T07:38:57.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.550+0000 7fc576ffd700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc5600b2880 0x7fc5600b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:57.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.550+0000 7fc57d45c700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc5600b2880 0x7fc5600b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:57.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.550+0000 7fc576ffd700 1 -- 172.21.15.114:0/1002222676 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fc570010020 con 0x7fc5781029b0 2024-05-27T07:38:57.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.550+0000 7fc57d45c700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc5600b2880 0x7fc5600b4d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc56c005950 tx=0x7fc56c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:57.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.553+0000 7fc576ffd700 1 -- 172.21.15.114:0/1002222676 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc57002c020 con 0x7fc5781029b0 2024-05-27T07:38:57.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.765+0000 7fc57fec1700 1 -- 172.21.15.114:0/1002222676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc5780512b0 con 0x7fc5781029b0 2024-05-27T07:38:57.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.766+0000 7fc576ffd700 1 -- 172.21.15.114:0/1002222676 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24923 (secure 0 0 0) 0x7fc5700abba0 con 0x7fc5781029b0 2024-05-27T07:38:57.764 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:38:57.764 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":89,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T07:37:46.129371+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:38:57.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.769+0000 7fc57fec1700 1 -- 172.21.15.114:0/1002222676 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc5600b2880 msgr2=0x7fc5600b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:57.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.769+0000 7fc57fec1700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc5600b2880 0x7fc5600b4d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc56c005950 tx=0x7fc56c0058e0 comp rx=0 tx=0).stop 2024-05-27T07:38:57.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.769+0000 7fc57fec1700 1 -- 172.21.15.114:0/1002222676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5781029b0 msgr2=0x7fc57819ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:57.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.769+0000 7fc57fec1700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5781029b0 0x7fc57819ae30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc5700082f0 tx=0x7fc5700083d0 comp rx=0 tx=0).stop 2024-05-27T07:38:57.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.769+0000 7fc57fec1700 1 -- 172.21.15.114:0/1002222676 shutdown_connections 2024-05-27T07:38:57.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.769+0000 7fc57fec1700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fc5600b2880 0x7fc5600b4d90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:57.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.769+0000 7fc57fec1700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5781089e0 0x7fc57819f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:57.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.769+0000 7fc57fec1700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5781029b0 0x7fc57819ae30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:57.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.769+0000 7fc57fec1700 1 --2- 172.21.15.114:0/1002222676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc578103300 0x7fc57819b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:57.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.769+0000 7fc57fec1700 1 -- 172.21.15.114:0/1002222676 >> 172.21.15.114:0/1002222676 conn(0x7fc5780fe6e0 msgr2=0x7fc578100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:57.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.769+0000 7fc57fec1700 1 -- 172.21.15.114:0/1002222676 shutdown_connections 2024-05-27T07:38:57.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:38:57.769+0000 7fc57fec1700 1 -- 172.21.15.114:0/1002222676 wait complete. 2024-05-27T07:38:58.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:58 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1002222676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:38:58.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:58 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1002222676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:38:58.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:58 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1002222676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:38:58.544 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:38:58.790 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:59.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:38:59 smithi114 ceph-mon[32038]: pgmap v386: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:59.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:38:59 smithi163 ceph-mon[36451]: pgmap v386: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:38:59.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:38:59 smithi186 ceph-mon[36647]: pgmap v386: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:39:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.584+0000 7ff4b890a700 1 -- 172.21.15.114:0/2150526105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b41034b0 msgr2=0x7ff4b41038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.585+0000 7ff4b890a700 1 --2- 172.21.15.114:0/2150526105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b41034b0 0x7ff4b41038c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80099c0 tx=0x7ff4a8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.585+0000 7ff4b890a700 1 -- 172.21.15.114:0/2150526105 shutdown_connections 2024-05-27T07:39:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.585+0000 7ff4b890a700 1 --2- 172.21.15.114:0/2150526105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4b4108730 0x7ff4b4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.585+0000 7ff4b890a700 1 --2- 172.21.15.114:0/2150526105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b41034b0 0x7ff4b41038c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.585+0000 7ff4b890a700 1 --2- 172.21.15.114:0/2150526105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b4107de0 0x7ff4b41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.585+0000 7ff4b890a700 1 -- 172.21.15.114:0/2150526105 >> 172.21.15.114:0/2150526105 conn(0x7ff4b40fedb0 msgr2=0x7ff4b41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:00.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.585+0000 7ff4b890a700 1 -- 172.21.15.114:0/2150526105 shutdown_connections 2024-05-27T07:39:00.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.585+0000 7ff4b890a700 1 -- 172.21.15.114:0/2150526105 wait complete. 2024-05-27T07:39:00.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.586+0000 7ff4b890a700 1 Processor -- start 2024-05-27T07:39:00.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.586+0000 7ff4b890a700 1 -- start start 2024-05-27T07:39:00.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.586+0000 7ff4b890a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b41034b0 0x7ff4b419bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:00.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.586+0000 7ff4b890a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b4107de0 0x7ff4b419c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:00.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.586+0000 7ff4b890a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4b4108730 0x7ff4b419c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:00.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.586+0000 7ff4b890a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b4113360 con 0x7ff4b41034b0 2024-05-27T07:39:00.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.586+0000 7ff4b890a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b41131e0 con 0x7ff4b4108730 2024-05-27T07:39:00.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.586+0000 7ff4b890a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b41134e0 con 0x7ff4b4107de0 2024-05-27T07:39:00.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4b259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b41034b0 0x7ff4b419bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:00.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4b1d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b4107de0 0x7ff4b419c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:00.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4b259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b41034b0 0x7ff4b419bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55544/0 (socket says 172.21.15.114:55544) 2024-05-27T07:39:00.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4b259c700 1 -- 172.21.15.114:0/1885636854 learned_addr learned my addr 172.21.15.114:0/1885636854 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:00.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4b2d9d700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4b4108730 0x7ff4b419c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4b259c700 1 -- 172.21.15.114:0/1885636854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b4107de0 msgr2=0x7ff4b419c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4b259c700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b4107de0 0x7ff4b419c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4b259c700 1 -- 172.21.15.114:0/1885636854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4b4108730 msgr2=0x7ff4b419c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4b259c700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4b4108730 0x7ff4b419c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4b259c700 1 -- 172.21.15.114:0/1885636854 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b41a6f90 con 0x7ff4b41034b0 2024-05-27T07:39:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4b1d9b700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b4107de0 0x7ff4b419c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4b259c700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b41034b0 0x7ff4b419bc20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800b580 tx=0x7ff4a80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4a37fe700 1 -- 172.21.15.114:0/1885636854 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8027070 con 0x7ff4b41034b0 2024-05-27T07:39:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4a37fe700 1 -- 172.21.15.114:0/1885636854 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4a8003f20 con 0x7ff4b41034b0 2024-05-27T07:39:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.587+0000 7ff4b890a700 1 -- 172.21.15.114:0/1885636854 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b41a7220 con 0x7ff4b41034b0 2024-05-27T07:39:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.588+0000 7ff4a37fe700 1 -- 172.21.15.114:0/1885636854 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8022d10 con 0x7ff4b41034b0 2024-05-27T07:39:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.588+0000 7ff4b890a700 1 -- 172.21.15.114:0/1885636854 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b41a76d0 con 0x7ff4b41034b0 2024-05-27T07:39:00.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.589+0000 7ff4b890a700 1 -- 172.21.15.114:0/1885636854 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4b40512b0 con 0x7ff4b41034b0 2024-05-27T07:39:00.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.590+0000 7ff4a37fe700 1 -- 172.21.15.114:0/1885636854 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7ff4a8022470 con 0x7ff4b41034b0 2024-05-27T07:39:00.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.591+0000 7ff4a37fe700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff49c0b2900 0x7ff49c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:00.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.591+0000 7ff4b1d9b700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff49c0b2900 0x7ff49c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:00.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.591+0000 7ff4a37fe700 1 -- 172.21.15.114:0/1885636854 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7ff4a80e3550 con 0x7ff4b41034b0 2024-05-27T07:39:00.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.591+0000 7ff4b1d9b700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff49c0b2900 0x7ff49c0b4e10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff4b41065b0 tx=0x7ff4a400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:00.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.595+0000 7ff4a37fe700 1 -- 172.21.15.114:0/1885636854 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff4a8013070 con 0x7ff4b41034b0 2024-05-27T07:39:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.841+0000 7ff4b890a700 1 -- 172.21.15.114:0/1885636854 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4b402e5e0 con 0x7ff4b41034b0 2024-05-27T07:39:00.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.841+0000 7ff4a37fe700 1 -- 172.21.15.114:0/1885636854 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11827 (secure 0 0 0) 0x7ff4a80aaf30 con 0x7ff4b41034b0 2024-05-27T07:39:00.839 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:00.839 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:37:06.645274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:39:00.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.844+0000 7ff4b890a700 1 -- 172.21.15.114:0/1885636854 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff49c0b2900 msgr2=0x7ff49c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:00.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.844+0000 7ff4b890a700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff49c0b2900 0x7ff49c0b4e10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff4b41065b0 tx=0x7ff4a400b410 comp rx=0 tx=0).stop 2024-05-27T07:39:00.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.844+0000 7ff4b890a700 1 -- 172.21.15.114:0/1885636854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b41034b0 msgr2=0x7ff4b419bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:00.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.844+0000 7ff4b890a700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b41034b0 0x7ff4b419bc20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800b580 tx=0x7ff4a80047b0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.844+0000 7ff4b890a700 1 -- 172.21.15.114:0/1885636854 shutdown_connections 2024-05-27T07:39:00.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.844+0000 7ff4b890a700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4b4108730 0x7ff4b419c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.844+0000 7ff4b890a700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b41034b0 0x7ff4b419bc20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.844+0000 7ff4b890a700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff49c0b2900 0x7ff49c0b4e10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.844+0000 7ff4b890a700 1 --2- 172.21.15.114:0/1885636854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b4107de0 0x7ff4b419c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.844+0000 7ff4b890a700 1 -- 172.21.15.114:0/1885636854 >> 172.21.15.114:0/1885636854 conn(0x7ff4b40fedb0 msgr2=0x7ff4b4101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:00.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.844+0000 7ff4b890a700 1 -- 172.21.15.114:0/1885636854 shutdown_connections 2024-05-27T07:39:00.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:00.844+0000 7ff4b890a700 1 -- 172.21.15.114:0/1885636854 wait complete. 2024-05-27T07:39:00.842 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:39:01.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:01 smithi114 ceph-mon[32038]: pgmap v387: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:01.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:01 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1885636854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:01.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:01 smithi163 ceph-mon[36451]: pgmap v387: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:01.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:01 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1885636854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:01.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:01 smithi186 ceph-mon[36647]: pgmap v387: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:01.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:01 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1885636854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:01.479 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-27T07:36:06.848027+0000', 'modified': '2024-05-27T07:37:06.645274+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24455': {'gid': 24455, 'name': 'b', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.163:6839/3774280856', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6838', 'nonce': 3774280856}, {'type': 'v1', 'addr': '172.21.15.163:6839', 'nonce': 3774280856}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.186:6839/4115872608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 4115872608}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 4115872608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24463, 'qdb_cluster': [24463]} 2024-05-27T07:39:01.479 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-27T07:39:01.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-05-27T07:39:01.719 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:03.439 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:03 smithi114 ceph-mon[32038]: pgmap v388: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.463+0000 7f3d5b05a700 1 -- 172.21.15.114:0/326806845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d541045c0 msgr2=0x7f3d541049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.463+0000 7f3d5b05a700 1 --2- 172.21.15.114:0/326806845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d541045c0 0x7f3d541049d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3d480099c0 tx=0x7f3d48009cd0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.463+0000 7f3d5b05a700 1 -- 172.21.15.114:0/326806845 shutdown_connections 2024-05-27T07:39:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.463+0000 7f3d5b05a700 1 --2- 172.21.15.114:0/326806845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d54109840 0x7f3d541111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.463+0000 7f3d5b05a700 1 --2- 172.21.15.114:0/326806845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d541045c0 0x7f3d541049d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.463+0000 7f3d5b05a700 1 --2- 172.21.15.114:0/326806845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d54108ef0 0x7f3d54109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.463+0000 7f3d5b05a700 1 -- 172.21.15.114:0/326806845 >> 172.21.15.114:0/326806845 conn(0x7f3d540ffe90 msgr2=0x7f3d541022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.463+0000 7f3d5b05a700 1 -- 172.21.15.114:0/326806845 shutdown_connections 2024-05-27T07:39:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.463+0000 7f3d5b05a700 1 -- 172.21.15.114:0/326806845 wait complete. 2024-05-27T07:39:03.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.464+0000 7f3d5b05a700 1 Processor -- start 2024-05-27T07:39:03.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.464+0000 7f3d5b05a700 1 -- start start 2024-05-27T07:39:03.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.464+0000 7f3d5b05a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d541045c0 0x7f3d5419cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.464+0000 7f3d5b05a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d54108ef0 0x7f3d5419d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.464+0000 7f3d5b05a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d54109840 0x7f3d541a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.464+0000 7f3d5b05a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3d54114490 con 0x7f3d54108ef0 2024-05-27T07:39:03.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.464+0000 7f3d5b05a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3d54114310 con 0x7f3d54109840 2024-05-27T07:39:03.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.464+0000 7f3d5b05a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3d54114610 con 0x7f3d541045c0 2024-05-27T07:39:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.464+0000 7f3d53fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d54108ef0 0x7f3d5419d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.464+0000 7f3d53fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d54108ef0 0x7f3d5419d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55550/0 (socket says 172.21.15.114:55550) 2024-05-27T07:39:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.464+0000 7f3d53fff700 1 -- 172.21.15.114:0/1264649035 learned_addr learned my addr 172.21.15.114:0/1264649035 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.464+0000 7f3d595f7700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d54109840 0x7f3d541a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.464+0000 7f3d58df6700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d541045c0 0x7f3d5419cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.465+0000 7f3d53fff700 1 -- 172.21.15.114:0/1264649035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d541045c0 msgr2=0x7f3d5419cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.465+0000 7f3d53fff700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d541045c0 0x7f3d5419cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.465+0000 7f3d53fff700 1 -- 172.21.15.114:0/1264649035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d54109840 msgr2=0x7f3d541a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.465+0000 7f3d53fff700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d54109840 0x7f3d541a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.465+0000 7f3d53fff700 1 -- 172.21.15.114:0/1264649035 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d541a1eb0 con 0x7f3d54108ef0 2024-05-27T07:39:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.465+0000 7f3d58df6700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d541045c0 0x7f3d5419cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.465+0000 7f3d53fff700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d54108ef0 0x7f3d5419d580 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000d950 tx=0x7f3d4000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:03.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.465+0000 7f3d51ffb700 1 -- 172.21.15.114:0/1264649035 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d40017070 con 0x7f3d54108ef0 2024-05-27T07:39:03.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.465+0000 7f3d5b05a700 1 -- 172.21.15.114:0/1264649035 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d541a21a0 con 0x7f3d54108ef0 2024-05-27T07:39:03.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.465+0000 7f3d51ffb700 1 -- 172.21.15.114:0/1264649035 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d40005020 con 0x7f3d54108ef0 2024-05-27T07:39:03.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.465+0000 7f3d51ffb700 1 -- 172.21.15.114:0/1264649035 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d40009ab0 con 0x7f3d54108ef0 2024-05-27T07:39:03.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.465+0000 7f3d5b05a700 1 -- 172.21.15.114:0/1264649035 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d541a9ac0 con 0x7f3d54108ef0 2024-05-27T07:39:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.469+0000 7f3d51ffb700 1 -- 172.21.15.114:0/1264649035 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f3d40009c50 con 0x7f3d54108ef0 2024-05-27T07:39:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.469+0000 7f3d51ffb700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3d440b2880 0x7f3d440b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.469+0000 7f3d58df6700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3d440b2880 0x7f3d440b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.469+0000 7f3d51ffb700 1 -- 172.21.15.114:0/1264649035 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f3d400e1e70 con 0x7f3d54108ef0 2024-05-27T07:39:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d69a6fb80 1 Processor -- start 2024-05-27T07:39:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d69a6fb80 1 -- start start 2024-05-27T07:39:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d58df6700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3d440b2880 0x7f3d440b4d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3d48003760 tx=0x7f3d4801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d69a6fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24df8df0 0x560d24f7dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d69a6fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f7e020 0x560d24f77b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d69a6fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24f780d0 0x560d24f784e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d69a6fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560d24ea5d20 con 0x560d24f780d0 2024-05-27T07:39:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d69a6fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560d24e958c0 con 0x560d24df8df0 2024-05-27T07:39:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d69a6fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560d24e99900 con 0x560d24f7e020 2024-05-27T07:39:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d595f7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24f780d0 0x560d24f784e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d58df6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24df8df0 0x560d24f7dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d595f7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24f780d0 0x560d24f784e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55556/0 (socket says 172.21.15.114:55556) 2024-05-27T07:39:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d595f7700 1 -- 172.21.15.114:0/1222145684 learned_addr learned my addr 172.21.15.114:0/1222145684 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d53fff700 1 --2- 172.21.15.114:0/1222145684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f7e020 0x560d24f77b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d595f7700 1 -- 172.21.15.114:0/1222145684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f7e020 msgr2=0x560d24f77b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d595f7700 1 --2- 172.21.15.114:0/1222145684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f7e020 0x560d24f77b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d595f7700 1 -- 172.21.15.114:0/1222145684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24df8df0 msgr2=0x560d24f7dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d595f7700 1 --2- 172.21.15.114:0/1222145684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24df8df0 0x560d24f7dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d595f7700 1 -- 172.21.15.114:0/1222145684 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d24e3ef60 con 0x560d24f780d0 2024-05-27T07:39:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d53fff700 1 --2- 172.21.15.114:0/1222145684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f7e020 0x560d24f77b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.470+0000 7f3d58df6700 1 --2- 172.21.15.114:0/1222145684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24df8df0 0x560d24f7dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.471+0000 7f3d595f7700 1 --2- 172.21.15.114:0/1222145684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24f780d0 0x560d24f784e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c011680 tx=0x7f3d4c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1923103a3570fbd9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.471+0000 7f3d3effd700 1 -- 172.21.15.114:0/1222145684 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c019030 con 0x560d24f780d0 2024-05-27T07:39:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.471+0000 7f3d3effd700 1 -- 172.21.15.114:0/1222145684 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d4c0092e0 con 0x560d24f780d0 2024-05-27T07:39:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.471+0000 7f3d3effd700 1 -- 172.21.15.114:0/1222145684 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c007500 con 0x560d24f780d0 2024-05-27T07:39:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.472+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/1222145684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24f780d0 msgr2=0x560d24f784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.472+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/1222145684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24f780d0 0x560d24f784e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c011680 tx=0x7f3d4c009920 comp rx=0 tx=0).stop 2024-05-27T07:39:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.472+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/1222145684 shutdown_connections 2024-05-27T07:39:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.472+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/1222145684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24f780d0 0x560d24f784e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.472+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/1222145684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24df8df0 0x560d24f7dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.472+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/1222145684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f7e020 0x560d24f77b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.472+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/1222145684 >> 172.21.15.114:0/1222145684 conn(0x560d24dfb720 msgr2=0x560d24f72d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.472+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/1222145684 shutdown_connections 2024-05-27T07:39:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.472+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/1222145684 wait complete. 2024-05-27T07:39:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.472+0000 7f3d69a6fb80 1 -- start start 2024-05-27T07:39:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.472+0000 7f3d69a6fb80 1 Processor -- start 2024-05-27T07:39:03.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:03 smithi163 ceph-mon[36451]: pgmap v388: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d69a6fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f817e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d69a6fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f780d0 0x560d24f81d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d69a6fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f7e020 0x560d24f82260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d69a6fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560d24d9bf50 con 0x560d24df8df0 2024-05-27T07:39:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d69a6fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560d24dfdd50 con 0x560d24f780d0 2024-05-27T07:39:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d69a6fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560d24f84b10 con 0x560d24f7e020 2024-05-27T07:39:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d69a6fb80 10 client.0 _ll_register_callbacks cb 0x560d24e2c560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d58df6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f817e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d58df6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f817e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55560/0 (socket says 172.21.15.114:55560) 2024-05-27T07:39:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d58df6700 1 -- 172.21.15.114:0/35284714 learned_addr learned my addr 172.21.15.114:0/35284714 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d595f7700 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f7e020 0x560d24f82260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d53fff700 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f780d0 0x560d24f81d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d58df6700 1 -- 172.21.15.114:0/35284714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f7e020 msgr2=0x560d24f82260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d58df6700 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f7e020 0x560d24f82260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d58df6700 1 -- 172.21.15.114:0/35284714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f780d0 msgr2=0x560d24f81d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d58df6700 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f780d0 0x560d24f81d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d58df6700 1 -- 172.21.15.114:0/35284714 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d24de5880 con 0x560d24df8df0 2024-05-27T07:39:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.473+0000 7f3d53fff700 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f780d0 0x560d24f81d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d58df6700 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f817e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3d480099c0 tx=0x7f3d48009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c3c07bce156bb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d33fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:39:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d33fff700 1 -- 172.21.15.114:0/35284714 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48034060 con 0x560d24df8df0 2024-05-27T07:39:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d33fff700 1 -- 172.21.15.114:0/35284714 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f3d4800f460 con 0x560d24df8df0 2024-05-27T07:39:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/35284714 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d4c0092e0 con 0x560d24df8df0 2024-05-27T07:39:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d33fff700 1 -- 172.21.15.114:0/35284714 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3d4803b050 con 0x560d24df8df0 2024-05-27T07:39:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d69a6fb80 10 client.15030 fetch_fsmap learned FSMap version 21 2024-05-27T07:39:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/35284714 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560d24e4f710 con 0x560d24df8df0 2024-05-27T07:39:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d33fff700 1 -- 172.21.15.114:0/35284714 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d4c0092e0 con 0x560d24df8df0 2024-05-27T07:39:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d33fff700 1 -- 172.21.15.114:0/35284714 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f3d48022070 con 0x560d24df8df0 2024-05-27T07:39:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d69a6fb80 10 client.15030 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:39:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d69a6fb80 10 client.15030 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:39:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d69a6fb80 20 client.15030 populate_metadata read hostname 'smithi114' 2024-05-27T07:39:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560d24f91a10 0x560d24f92e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d69a6fb80 4 client.15030 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.474+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/35284714 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d24ea7c70 con 0x560d24f91a10 2024-05-27T07:39:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.475+0000 7f3d53fff700 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560d24f91a10 0x560d24f92e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.475+0000 7f3d33fff700 10 client.15030 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.475+0000 7f3d53fff700 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560d24f91a10 0x560d24f92e80 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5abce7fa6551a195 server_cookie=70f476ca3f75b225 in_seq=0 out_seq=0 2024-05-27T07:39:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d33fff700 1 -- 172.21.15.114:0/35284714 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560d24ea7c70 con 0x560d24f91a10 2024-05-27T07:39:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d33fff700 10 client.15030 handle_command_reply: tid=0 2024-05-27T07:39:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d69a6fb80 1 client.15030 shutdown 2024-05-27T07:39:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/35284714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 msgr2=0x560d24f817e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f817e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3d480099c0 tx=0x7f3d48009cd0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/35284714 shutdown_connections 2024-05-27T07:39:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f7e020 0x560d24f82260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f817e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560d24f91a10 0x560d24f92e80 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/35284714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f780d0 0x560d24f81d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/35284714 >> 172.21.15.114:0/35284714 conn(0x560d24dfb720 msgr2=0x560d24f76b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d33fff700 0 client.15030 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/35284714 shutdown_connections 2024-05-27T07:39:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.476+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/35284714 wait complete. 2024-05-27T07:39:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.477+0000 7f3d69a6fb80 20 client.15030 trim_cache size 0 max 16384 2024-05-27T07:39:03.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:03 smithi186 ceph-mon[36647]: pgmap v388: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:03.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.498+0000 7f3d69a6fb80 1 Processor -- start 2024-05-27T07:39:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.498+0000 7f3d69a6fb80 1 -- start start 2024-05-27T07:39:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.498+0000 7f3d69a6fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f77b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.498+0000 7f3d69a6fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f780d0 0x560d24f80740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.498+0000 7f3d69a6fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f7e020 0x560d24f8dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.498+0000 7f3d69a6fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560d24f911c0 con 0x560d24df8df0 2024-05-27T07:39:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.498+0000 7f3d69a6fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560d24f91890 con 0x560d24f7e020 2024-05-27T07:39:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.498+0000 7f3d69a6fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560d24f91f60 con 0x560d24f780d0 2024-05-27T07:39:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d58df6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f77b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d58df6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f77b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55570/0 (socket says 172.21.15.114:55570) 2024-05-27T07:39:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d58df6700 1 -- 172.21.15.114:0/2436269484 learned_addr learned my addr 172.21.15.114:0/2436269484 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d53fff700 1 --2- 172.21.15.114:0/2436269484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f780d0 0x560d24f80740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d595f7700 1 --2- 172.21.15.114:0/2436269484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f7e020 0x560d24f8dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d58df6700 1 -- 172.21.15.114:0/2436269484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f780d0 msgr2=0x560d24f80740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d58df6700 1 --2- 172.21.15.114:0/2436269484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f780d0 0x560d24f80740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d58df6700 1 -- 172.21.15.114:0/2436269484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f7e020 msgr2=0x560d24f8dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d58df6700 1 --2- 172.21.15.114:0/2436269484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f7e020 0x560d24f8dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d58df6700 1 -- 172.21.15.114:0/2436269484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d24f6ea50 con 0x560d24df8df0 2024-05-27T07:39:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d595f7700 1 --2- 172.21.15.114:0/2436269484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f7e020 0x560d24f8dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d58df6700 1 --2- 172.21.15.114:0/2436269484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f77b90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3d4802f680 tx=0x7f3d48003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a72be1a1cb257f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d3e7fc700 1 -- 172.21.15.114:0/2436269484 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48034040 con 0x560d24df8df0 2024-05-27T07:39:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d3e7fc700 1 -- 172.21.15.114:0/2436269484 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d4802fda0 con 0x560d24df8df0 2024-05-27T07:39:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.499+0000 7f3d3e7fc700 1 -- 172.21.15.114:0/2436269484 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4800fcc0 con 0x560d24df8df0 2024-05-27T07:39:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.500+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/2436269484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 msgr2=0x560d24f77b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.500+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/2436269484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f77b90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3d4802f680 tx=0x7f3d48003f20 comp rx=0 tx=0).stop 2024-05-27T07:39:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.500+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/2436269484 shutdown_connections 2024-05-27T07:39:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.500+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/2436269484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f7e020 0x560d24f8dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.500+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/2436269484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f77b90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.500+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/2436269484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f780d0 0x560d24f80740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.500+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/2436269484 >> 172.21.15.114:0/2436269484 conn(0x560d24dfb720 msgr2=0x560d24f7c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.500+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/2436269484 shutdown_connections 2024-05-27T07:39:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.500+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/2436269484 wait complete. 2024-05-27T07:39:03.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.501+0000 7f3d69a6fb80 1 -- start start 2024-05-27T07:39:03.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.501+0000 7f3d69a6fb80 1 Processor -- start 2024-05-27T07:39:03.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.501+0000 7f3d69a6fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f76b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.501+0000 7f3d69a6fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f780d0 0x560d24f770c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.501+0000 7f3d69a6fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f7e020 0x560d24f77600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.501+0000 7f3d69a6fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560d24f92d00 con 0x560d24df8df0 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.501+0000 7f3d69a6fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560d24f92630 con 0x560d24f7e020 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.501+0000 7f3d69a6fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560d24f933d0 con 0x560d24f780d0 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.501+0000 7f3d69a6fb80 10 client.0 _ll_register_callbacks cb 0x560d24e2c560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.501+0000 7f3d58df6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f76b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.501+0000 7f3d58df6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f76b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55576/0 (socket says 172.21.15.114:55576) 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.501+0000 7f3d58df6700 1 -- 172.21.15.114:0/166789727 learned_addr learned my addr 172.21.15.114:0/166789727 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.501+0000 7f3d595f7700 1 --2- 172.21.15.114:0/166789727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f7e020 0x560d24f77600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d53fff700 1 --2- 172.21.15.114:0/166789727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f780d0 0x560d24f770c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d58df6700 1 -- 172.21.15.114:0/166789727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f780d0 msgr2=0x560d24f770c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d58df6700 1 --2- 172.21.15.114:0/166789727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f780d0 0x560d24f770c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d58df6700 1 -- 172.21.15.114:0/166789727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f7e020 msgr2=0x560d24f77600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d58df6700 1 --2- 172.21.15.114:0/166789727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f7e020 0x560d24f77600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d58df6700 1 -- 172.21.15.114:0/166789727 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d24f9bad0 con 0x560d24df8df0 2024-05-27T07:39:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d58df6700 1 --2- 172.21.15.114:0/166789727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f76b80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3d48005310 tx=0x7f3d4800ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a4449170960440 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d33fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:39:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d33fff700 1 -- 172.21.15.114:0/166789727 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48034040 con 0x560d24df8df0 2024-05-27T07:39:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d33fff700 1 -- 172.21.15.114:0/166789727 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f3d4802fa70 con 0x560d24df8df0 2024-05-27T07:39:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/166789727 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d4802fda0 con 0x560d24df8df0 2024-05-27T07:39:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d33fff700 1 -- 172.21.15.114:0/166789727 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3d48051050 con 0x560d24df8df0 2024-05-27T07:39:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d69a6fb80 10 client.15036 fetch_fsmap learned FSMap version 21 2024-05-27T07:39:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/166789727 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560d24f9cd50 con 0x560d24df8df0 2024-05-27T07:39:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.502+0000 7f3d33fff700 1 -- 172.21.15.114:0/166789727 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d4802fda0 con 0x560d24df8df0 2024-05-27T07:39:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.503+0000 7f3d33fff700 1 -- 172.21.15.114:0/166789727 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f3d48022030 con 0x560d24df8df0 2024-05-27T07:39:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.503+0000 7f3d69a6fb80 10 client.15036 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:39:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.503+0000 7f3d69a6fb80 10 client.15036 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:39:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.503+0000 7f3d69a6fb80 20 client.15036 populate_metadata read hostname 'smithi114' 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.503+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/166789727 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560d24ffecc0 0x560d25000130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.503+0000 7f3d69a6fb80 4 client.15036 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.503+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/166789727 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "status"}) -- 0x560d24ea1450 con 0x560d24ffecc0 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.503+0000 7f3d53fff700 1 --2- 172.21.15.114:0/166789727 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560d24ffecc0 0x560d25000130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.503+0000 7f3d53fff700 1 --2- 172.21.15.114:0/166789727 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560d24ffecc0 0x560d25000130 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=876f675a2819447b server_cookie=e0eedee2411dcc30 in_seq=0 out_seq=0 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.503+0000 7f3d33fff700 10 client.15036 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d33fff700 1 -- 172.21.15.114:0/166789727 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x560d24ea1450 con 0x560d24ffecc0 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d33fff700 10 client.15036 handle_command_reply: tid=0 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d69a6fb80 1 client.15036 shutdown 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/166789727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 msgr2=0x560d24f76b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/166789727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f76b80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3d48005310 tx=0x7f3d4800ba70 comp rx=0 tx=0).stop 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/166789727 shutdown_connections 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/166789727 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560d24ffecc0 0x560d25000130 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/166789727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560d24f7e020 0x560d24f77600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/166789727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560d24df8df0 0x560d24f76b80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d33fff700 0 client.15036 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d69a6fb80 1 --2- 172.21.15.114:0/166789727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d24f780d0 0x560d24f770c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/166789727 >> 172.21.15.114:0/166789727 conn(0x560d24dfb720 msgr2=0x560d24f7c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/166789727 shutdown_connections 2024-05-27T07:39:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d69a6fb80 1 -- 172.21.15.114:0/166789727 wait complete. 2024-05-27T07:39:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.504+0000 7f3d69a6fb80 20 client.15036 trim_cache size 0 max 16384 2024-05-27T07:39:03.507 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:39:03.507 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "755b3728-1bfa-11ef-bc9b-c7b262605968", 2024-05-27T07:39:03.508 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-05-27T07:39:03.508 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24463, 2024-05-27T07:39:03.508 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-05-27T07:39:03.508 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-05-27T07:39:03.508 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-05-27T07:39:03.508 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 176.529656456, 2024-05-27T07:39:03.508 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 21, 2024-05-27T07:39:03.508 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 89, 2024-05-27T07:39:03.508 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 87, 2024-05-27T07:39:03.508 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 201.46687888400001 2024-05-27T07:39:03.508 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:39:03.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.505+0000 7f3d5b05a700 1 -- 172.21.15.114:0/1264649035 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3d440b2880 msgr2=0x7f3d440b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:03.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.505+0000 7f3d5b05a700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3d440b2880 0x7f3d440b4d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3d48003760 tx=0x7f3d4801a040 comp rx=0 tx=0).stop 2024-05-27T07:39:03.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.505+0000 7f3d5b05a700 1 -- 172.21.15.114:0/1264649035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d54108ef0 msgr2=0x7f3d5419d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:03.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.505+0000 7f3d5b05a700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d54108ef0 0x7f3d5419d580 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000d950 tx=0x7f3d4000dc60 comp rx=0 tx=0).stop 2024-05-27T07:39:03.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.505+0000 7f3d5b05a700 1 -- 172.21.15.114:0/1264649035 shutdown_connections 2024-05-27T07:39:03.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.505+0000 7f3d5b05a700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d54109840 0x7f3d541a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.505+0000 7f3d5b05a700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3d440b2880 0x7f3d440b4d90 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.505+0000 7f3d5b05a700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d541045c0 0x7f3d5419cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.505+0000 7f3d5b05a700 1 --2- 172.21.15.114:0/1264649035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d54108ef0 0x7f3d5419d580 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:03.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.505+0000 7f3d5b05a700 1 -- 172.21.15.114:0/1264649035 >> 172.21.15.114:0/1264649035 conn(0x7f3d540ffe90 msgr2=0x7f3d54110470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:03.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.505+0000 7f3d5b05a700 1 -- 172.21.15.114:0/1264649035 shutdown_connections 2024-05-27T07:39:03.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:03.505+0000 7f3d5b05a700 1 -- 172.21.15.114:0/1264649035 wait complete. 2024-05-27T07:39:04.205 INFO:tasks.cephadm:Setup complete, yielding 2024-05-27T07:39:04.205 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-27T07:39:04.215 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-05-27T07:39:04.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch status' 2024-05-27T07:39:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:05.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:05 smithi163 ceph-mon[36451]: pgmap v389: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:05.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:05 smithi186 ceph-mon[36647]: pgmap v389: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:05.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:05 smithi114 ceph-mon[32038]: pgmap v389: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:06.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.184+0000 7faf6b252700 1 -- 172.21.15.114:0/1672263158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf641049d0 msgr2=0x7faf6410c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:06.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.184+0000 7faf6b252700 1 --2- 172.21.15.114:0/1672263158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf641049d0 0x7faf6410c6e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7faf5c00b0a0 tx=0x7faf5c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:39:06.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.184+0000 7faf6b252700 1 -- 172.21.15.114:0/1672263158 shutdown_connections 2024-05-27T07:39:06.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.184+0000 7faf6b252700 1 --2- 172.21.15.114:0/1672263158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf641049d0 0x7faf6410c6e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:06.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.184+0000 7faf6b252700 1 --2- 172.21.15.114:0/1672263158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf641059a0 0x7faf64105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:06.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.184+0000 7faf6b252700 1 --2- 172.21.15.114:0/1672263158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf64103f50 0x7faf64104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:06.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.184+0000 7faf6b252700 1 -- 172.21.15.114:0/1672263158 >> 172.21.15.114:0/1672263158 conn(0x7faf640ff9c0 msgr2=0x7faf64101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:06.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.184+0000 7faf6b252700 1 -- 172.21.15.114:0/1672263158 shutdown_connections 2024-05-27T07:39:06.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.184+0000 7faf6b252700 1 -- 172.21.15.114:0/1672263158 wait complete. 2024-05-27T07:39:06.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf6b252700 1 Processor -- start 2024-05-27T07:39:06.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf6b252700 1 -- start start 2024-05-27T07:39:06.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf6b252700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf64103f50 0x7faf6419a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:06.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf6b252700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf641049d0 0x7faf6419aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf6b252700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf641059a0 0x7faf641a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf6b252700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410fee0 con 0x7faf64103f50 2024-05-27T07:39:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf6b252700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410fd60 con 0x7faf641049d0 2024-05-27T07:39:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf6b252700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faf64110060 con 0x7faf641059a0 2024-05-27T07:39:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf68fee700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf64103f50 0x7faf6419a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf68fee700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf64103f50 0x7faf6419a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54986/0 (socket says 172.21.15.114:54986) 2024-05-27T07:39:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf68fee700 1 -- 172.21.15.114:0/1539744165 learned_addr learned my addr 172.21.15.114:0/1539744165 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf63fff700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf641049d0 0x7faf6419aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf697ef700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf641059a0 0x7faf641a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf68fee700 1 -- 172.21.15.114:0/1539744165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf641059a0 msgr2=0x7faf641a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf68fee700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf641059a0 0x7faf641a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf68fee700 1 -- 172.21.15.114:0/1539744165 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf641049d0 msgr2=0x7faf6419aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf68fee700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf641049d0 0x7faf6419aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:06.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf68fee700 1 -- 172.21.15.114:0/1539744165 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf641a24c0 con 0x7faf64103f50 2024-05-27T07:39:06.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.185+0000 7faf697ef700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf641059a0 0x7faf641a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:06.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.186+0000 7faf68fee700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf64103f50 0x7faf6419a960 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faf5800c970 tx=0x7faf5800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:06.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.186+0000 7faf61ffb700 1 -- 172.21.15.114:0/1539744165 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5800ce60 con 0x7faf64103f50 2024-05-27T07:39:06.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.186+0000 7faf6b252700 1 -- 172.21.15.114:0/1539744165 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf641a27b0 con 0x7faf64103f50 2024-05-27T07:39:06.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.186+0000 7faf61ffb700 1 -- 172.21.15.114:0/1539744165 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf58007900 con 0x7faf64103f50 2024-05-27T07:39:06.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.186+0000 7faf61ffb700 1 -- 172.21.15.114:0/1539744165 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58003c70 con 0x7faf64103f50 2024-05-27T07:39:06.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.186+0000 7faf6b252700 1 -- 172.21.15.114:0/1539744165 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf641a2d40 con 0x7faf64103f50 2024-05-27T07:39:06.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.187+0000 7faf6b252700 1 -- 172.21.15.114:0/1539744165 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf640512b0 con 0x7faf64103f50 2024-05-27T07:39:06.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.189+0000 7faf61ffb700 1 -- 172.21.15.114:0/1539744165 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7faf58004220 con 0x7faf64103f50 2024-05-27T07:39:06.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.190+0000 7faf61ffb700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7faf540b2880 0x7faf540b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:06.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.190+0000 7faf63fff700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7faf540b2880 0x7faf540b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:06.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.190+0000 7faf61ffb700 1 -- 172.21.15.114:0/1539744165 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7faf580e20f0 con 0x7faf64103f50 2024-05-27T07:39:06.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.191+0000 7faf63fff700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7faf540b2880 0x7faf540b4d90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faf50005950 tx=0x7faf5000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:06.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.195+0000 7faf61ffb700 1 -- 172.21.15.114:0/1539744165 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7faf58014030 con 0x7faf64103f50 2024-05-27T07:39:06.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.419+0000 7faf6b252700 1 -- 172.21.15.114:0/1539744165 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7faf6402e990 con 0x7faf540b2880 2024-05-27T07:39:06.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.420+0000 7faf61ffb700 1 -- 172.21.15.114:0/1539744165 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7faf6402e990 con 0x7faf540b2880 2024-05-27T07:39:06.417 INFO:teuthology.orchestra.run.smithi114.stdout:Backend: cephadm 2024-05-27T07:39:06.417 INFO:teuthology.orchestra.run.smithi114.stdout:Available: Yes 2024-05-27T07:39:06.417 INFO:teuthology.orchestra.run.smithi114.stdout:Paused: No 2024-05-27T07:39:06.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.422+0000 7faf6b252700 1 -- 172.21.15.114:0/1539744165 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7faf540b2880 msgr2=0x7faf540b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:06.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.422+0000 7faf6b252700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7faf540b2880 0x7faf540b4d90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faf50005950 tx=0x7faf5000a300 comp rx=0 tx=0).stop 2024-05-27T07:39:06.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.422+0000 7faf6b252700 1 -- 172.21.15.114:0/1539744165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf64103f50 msgr2=0x7faf6419a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:06.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.422+0000 7faf6b252700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf64103f50 0x7faf6419a960 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faf5800c970 tx=0x7faf5800cc80 comp rx=0 tx=0).stop 2024-05-27T07:39:06.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.422+0000 7faf6b252700 1 -- 172.21.15.114:0/1539744165 shutdown_connections 2024-05-27T07:39:06.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.422+0000 7faf6b252700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7faf540b2880 0x7faf540b4d90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:06.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.422+0000 7faf6b252700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf641059a0 0x7faf641a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:06.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.422+0000 7faf6b252700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf64103f50 0x7faf6419a960 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:06.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.422+0000 7faf6b252700 1 --2- 172.21.15.114:0/1539744165 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf641049d0 0x7faf6419aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:06.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.423+0000 7faf6b252700 1 -- 172.21.15.114:0/1539744165 >> 172.21.15.114:0/1539744165 conn(0x7faf640ff9c0 msgr2=0x7faf641086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:06.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.423+0000 7faf6b252700 1 -- 172.21.15.114:0/1539744165 shutdown_connections 2024-05-27T07:39:06.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:06.423+0000 7faf6b252700 1 -- 172.21.15.114:0/1539744165 wait complete. 2024-05-27T07:39:07.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-27T07:39:07.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:07 smithi163 ceph-mon[36451]: pgmap v390: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:07.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:07 smithi186 ceph-mon[36647]: pgmap v390: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:07.498 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:07.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:07 smithi114 ceph-mon[32038]: pgmap v390: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:08.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:08 smithi114 ceph-mon[32038]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:08.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:08 smithi163 ceph-mon[36451]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:08.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:08 smithi186 ceph-mon[36647]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:09.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.291+0000 7fdedbbbc700 1 -- 172.21.15.114:0/2889228883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded4104360 msgr2=0x7fded4104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.291+0000 7fdedbbbc700 1 --2- 172.21.15.114:0/2889228883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded4104360 0x7fded4104770 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdec4009990 tx=0x7fdec4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.291+0000 7fdedbbbc700 1 -- 172.21.15.114:0/2889228883 shutdown_connections 2024-05-27T07:39:09.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.291+0000 7fdedbbbc700 1 --2- 172.21.15.114:0/2889228883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fded4104d00 0x7fded410b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.291+0000 7fdedbbbc700 1 --2- 172.21.15.114:0/2889228883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded4103230 0x7fded4103640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.291+0000 7fdedbbbc700 1 --2- 172.21.15.114:0/2889228883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded4104360 0x7fded4104770 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.291+0000 7fdedbbbc700 1 -- 172.21.15.114:0/2889228883 >> 172.21.15.114:0/2889228883 conn(0x7fded40fe870 msgr2=0x7fded4100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:09.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.291+0000 7fdedbbbc700 1 -- 172.21.15.114:0/2889228883 shutdown_connections 2024-05-27T07:39:09.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.291+0000 7fdedbbbc700 1 -- 172.21.15.114:0/2889228883 wait complete. 2024-05-27T07:39:09.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.292+0000 7fdedbbbc700 1 Processor -- start 2024-05-27T07:39:09.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.292+0000 7fdedbbbc700 1 -- start start 2024-05-27T07:39:09.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.292+0000 7fdedbbbc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded4103230 0x7fded406c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.292+0000 7fdedbbbc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded4104360 0x7fded419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.292+0000 7fdedbbbc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fded4104d00 0x7fded41a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.292+0000 7fdedbbbc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fded410ecb0 con 0x7fded4104360 2024-05-27T07:39:09.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.292+0000 7fdedbbbc700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fded410eb30 con 0x7fded4103230 2024-05-27T07:39:09.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.292+0000 7fdedbbbc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fded410ee30 con 0x7fded4104d00 2024-05-27T07:39:09.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fded9157700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded4104360 0x7fded419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fded9157700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded4104360 0x7fded419f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55004/0 (socket says 172.21.15.114:55004) 2024-05-27T07:39:09.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fded9157700 1 -- 172.21.15.114:0/1075414770 learned_addr learned my addr 172.21.15.114:0/1075414770 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:09.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fded9958700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded4103230 0x7fded406c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fdeda159700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fded4104d00 0x7fded41a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fded9157700 1 -- 172.21.15.114:0/1075414770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fded4104d00 msgr2=0x7fded41a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fded9157700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fded4104d00 0x7fded41a1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fded9157700 1 -- 172.21.15.114:0/1075414770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded4103230 msgr2=0x7fded406c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fded9157700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded4103230 0x7fded406c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fded9157700 1 -- 172.21.15.114:0/1075414770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded41a2460 con 0x7fded4104360 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fdeda159700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fded4104d00 0x7fded41a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fded9157700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded4104360 0x7fded419f6c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdec4006010 tx=0x7fdec40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fdecaffd700 1 -- 172.21.15.114:0/1075414770 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4027070 con 0x7fded4104360 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fdedbbbc700 1 -- 172.21.15.114:0/1075414770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded41a26f0 con 0x7fded4104360 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fdecaffd700 1 -- 172.21.15.114:0/1075414770 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdec40040c0 con 0x7fded4104360 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fdecaffd700 1 -- 172.21.15.114:0/1075414770 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4022d90 con 0x7fded4104360 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.293+0000 7fdedbbbc700 1 -- 172.21.15.114:0/1075414770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded41a2b40 con 0x7fded4104360 2024-05-27T07:39:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.294+0000 7fdedbbbc700 1 -- 172.21.15.114:0/1075414770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fded40512b0 con 0x7fded4104360 2024-05-27T07:39:09.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.296+0000 7fdecaffd700 1 -- 172.21.15.114:0/1075414770 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fdec402a070 con 0x7fded4104360 2024-05-27T07:39:09.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.297+0000 7fdecaffd700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fdec00b2880 0x7fdec00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.297+0000 7fded9958700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fdec00b2880 0x7fdec00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.297+0000 7fdecaffd700 1 -- 172.21.15.114:0/1075414770 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fdec40e36d0 con 0x7fded4104360 2024-05-27T07:39:09.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.297+0000 7fded9958700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fdec00b2880 0x7fdec00b4d90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fded4104260 tx=0x7fded000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:09.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.300+0000 7fdecaffd700 1 -- 172.21.15.114:0/1075414770 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdec405fdf0 con 0x7fded4104360 2024-05-27T07:39:09.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:09 smithi163 ceph-mon[36451]: pgmap v391: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:09.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:09 smithi186 ceph-mon[36647]: pgmap v391: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:09.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.511+0000 7fdedbbbc700 1 -- 172.21.15.114:0/1075414770 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fded4063bf0 con 0x7fdec00b2880 2024-05-27T07:39:09.508 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:09 smithi114 ceph-mon[32038]: pgmap v391: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:09.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.525+0000 7fdecaffd700 1 -- 172.21.15.114:0/1075414770 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fded4063bf0 con 0x7fdec00b2880 2024-05-27T07:39:09.522 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-27T07:39:09.522 INFO:teuthology.orchestra.run.smithi114.stdout:mds.a smithi114 running (3m) 32s ago 3m 81.0M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 cb013cfd6fb6 2024-05-27T07:39:09.522 INFO:teuthology.orchestra.run.smithi114.stdout:mds.b smithi163 running (3m) 61s ago 3m 86.0M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 9e612e37bd4f 2024-05-27T07:39:09.522 INFO:teuthology.orchestra.run.smithi114.stdout:mds.c smithi186 running (3m) 61s ago 3m 84.3M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 a03d8aed320f 2024-05-27T07:39:09.522 INFO:teuthology.orchestra.run.smithi114.stdout:mds.d smithi114 running (3m) 32s ago 3m 84.4M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 b20fb979dcf7 2024-05-27T07:39:09.522 INFO:teuthology.orchestra.run.smithi114.stdout:mds.e smithi163 running (3m) 61s ago 3m 81.2M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 4e04f6774a3a 2024-05-27T07:39:09.522 INFO:teuthology.orchestra.run.smithi114.stdout:mds.f smithi186 running (3m) 61s ago 3m 82.2M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 65fabfafe5ea 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:mds.g smithi114 running (3m) 32s ago 3m 82.4M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 5542c738fdbb 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:mds.h smithi163 running (3m) 61s ago 3m 81.9M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 ea2b072a7eca 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:mds.i smithi186 running (3m) 61s ago 3m 93.9M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 a502f02b0747 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:mds.j smithi114 running (3m) 32s ago 3m 81.2M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 737e349e4d43 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:mds.k smithi163 running (3m) 61s ago 3m 81.1M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 aa7f78042097 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:mds.l smithi186 running (3m) 61s ago 3m 81.0M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 f84b3ce82512 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.x smithi114 *:9283,8765 running (11m) 32s ago 11m 599M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 4289f83cbe60 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.y smithi163 *:8443,8765 running (9m) 61s ago 9m 535M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 a3f682a56a5e 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.z smithi186 *:8443,8765 running (9m) 61s ago 9m 534M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 32c8c30f71f1 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a smithi114 running (11m) 32s ago 11m 119M 2048M 19.0.0-3872-g38f9d900 e36ea4fc8a58 b13d09b4408f 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b smithi163 running (9m) 61s ago 9m 106M 2048M 19.0.0-3872-g38f9d900 e36ea4fc8a58 61a1565b03ae 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c smithi186 running (9m) 61s ago 9m 108M 2048M 19.0.0-3872-g38f9d900 e36ea4fc8a58 497b6738db89 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (8m) 32s ago 8m 126M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 64f415f677c1 2024-05-27T07:39:09.523 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi163 running (8m) 61s ago 8m 128M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 89eeef1de66c 2024-05-27T07:39:09.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi186 running (8m) 61s ago 8m 127M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 9c78ea804f68 2024-05-27T07:39:09.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi114 running (7m) 32s ago 7m 122M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 b429a7d9ff37 2024-05-27T07:39:09.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi163 running (7m) 61s ago 7m 109M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 e332dc16ccba 2024-05-27T07:39:09.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi186 running (6m) 61s ago 6m 112M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 dd2e496b59ee 2024-05-27T07:39:09.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 smithi114 running (6m) 32s ago 6m 110M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 a6b2ac4595dd 2024-05-27T07:39:09.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 smithi163 running (5m) 61s ago 5m 123M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 ab3fd5519707 2024-05-27T07:39:09.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.8 smithi186 running (5m) 61s ago 5m 111M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 fb489a33d5bb 2024-05-27T07:39:09.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.9 smithi114 running (5m) 32s ago 5m 121M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 227a5431062e 2024-05-27T07:39:09.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.10 smithi163 running (4m) 61s ago 4m 113M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 b2bfae177813 2024-05-27T07:39:09.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.11 smithi186 running (4m) 61s ago 4m 122M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 21c0463cf20f 2024-05-27T07:39:09.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.527+0000 7fdedbbbc700 1 -- 172.21.15.114:0/1075414770 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fdec00b2880 msgr2=0x7fdec00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.527+0000 7fdedbbbc700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fdec00b2880 0x7fdec00b4d90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fded4104260 tx=0x7fded000a380 comp rx=0 tx=0).stop 2024-05-27T07:39:09.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.527+0000 7fdedbbbc700 1 -- 172.21.15.114:0/1075414770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded4104360 msgr2=0x7fded419f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.527+0000 7fdedbbbc700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded4104360 0x7fded419f6c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdec4006010 tx=0x7fdec40047b0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.528+0000 7fdedbbbc700 1 -- 172.21.15.114:0/1075414770 shutdown_connections 2024-05-27T07:39:09.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.528+0000 7fdedbbbc700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fdec00b2880 0x7fdec00b4d90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.528+0000 7fdedbbbc700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fded4104d00 0x7fded41a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.528+0000 7fdedbbbc700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded4103230 0x7fded406c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.528+0000 7fdedbbbc700 1 --2- 172.21.15.114:0/1075414770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded4104360 0x7fded419f6c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.528+0000 7fdedbbbc700 1 -- 172.21.15.114:0/1075414770 >> 172.21.15.114:0/1075414770 conn(0x7fded40fe870 msgr2=0x7fded4107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:09.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.528+0000 7fdedbbbc700 1 -- 172.21.15.114:0/1075414770 shutdown_connections 2024-05-27T07:39:09.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:09.528+0000 7fdedbbbc700 1 -- 172.21.15.114:0/1075414770 wait complete. 2024-05-27T07:39:10.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-27T07:39:10.561 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:11.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:11 smithi114 ceph-mon[32038]: pgmap v392: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:11.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:11 smithi114 ceph-mon[32038]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:11.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:11 smithi163 ceph-mon[36451]: pgmap v392: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:11.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:11 smithi163 ceph-mon[36451]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:11.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:11 smithi186 ceph-mon[36647]: pgmap v392: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:11.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:11 smithi186 ceph-mon[36647]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:12.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.238+0000 7fb14b495700 1 -- 172.21.15.114:0/3570113662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb144104d10 msgr2=0x7fb14410b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:12.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.238+0000 7fb14b495700 1 --2- 172.21.15.114:0/3570113662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb144104d10 0x7fb14410b5c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00b0a0 tx=0x7fb13c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:39:12.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.238+0000 7fb14b495700 1 -- 172.21.15.114:0/3570113662 shutdown_connections 2024-05-27T07:39:12.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.238+0000 7fb14b495700 1 --2- 172.21.15.114:0/3570113662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb144104d10 0x7fb14410b5c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:12.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.238+0000 7fb14b495700 1 --2- 172.21.15.114:0/3570113662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb144103240 0x7fb144103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:12.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.238+0000 7fb14b495700 1 --2- 172.21.15.114:0/3570113662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb144104370 0x7fb144104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:12.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.238+0000 7fb14b495700 1 -- 172.21.15.114:0/3570113662 >> 172.21.15.114:0/3570113662 conn(0x7fb1440fe880 msgr2=0x7fb144100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:12.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.238+0000 7fb14b495700 1 -- 172.21.15.114:0/3570113662 shutdown_connections 2024-05-27T07:39:12.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.238+0000 7fb14b495700 1 -- 172.21.15.114:0/3570113662 wait complete. 2024-05-27T07:39:12.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb14b495700 1 Processor -- start 2024-05-27T07:39:12.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb14b495700 1 -- start start 2024-05-27T07:39:12.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb14b495700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb144103240 0x7fb144076440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:12.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb14b495700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb144104370 0x7fb14419f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:12.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb14b495700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb144104d10 0x7fb1441a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:12.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb14b495700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb14410eca0 con 0x7fb144104370 2024-05-27T07:39:12.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb14b495700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb14410eb20 con 0x7fb144104d10 2024-05-27T07:39:12.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb14b495700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb14410ee20 con 0x7fb144103240 2024-05-27T07:39:12.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb148a30700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb144104370 0x7fb14419f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:12.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb148a30700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb144104370 0x7fb14419f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55018/0 (socket says 172.21.15.114:55018) 2024-05-27T07:39:12.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb148a30700 1 -- 172.21.15.114:0/4208368512 learned_addr learned my addr 172.21.15.114:0/4208368512 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:12.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb149231700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb144103240 0x7fb144076440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:12.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb149a32700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb144104d10 0x7fb1441a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:12.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb148a30700 1 -- 172.21.15.114:0/4208368512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb144103240 msgr2=0x7fb144076440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:12.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb148a30700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb144103240 0x7fb144076440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:12.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb148a30700 1 -- 172.21.15.114:0/4208368512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb144104d10 msgr2=0x7fb1441a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:12.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb148a30700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb144104d10 0x7fb1441a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:12.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.239+0000 7fb148a30700 1 -- 172.21.15.114:0/4208368512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1441a23e0 con 0x7fb144104370 2024-05-27T07:39:12.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.240+0000 7fb148a30700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb144104370 0x7fb14419f640 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb13400d950 tx=0x7fb13400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:12.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.240+0000 7fb13a7fc700 1 -- 172.21.15.114:0/4208368512 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134016070 con 0x7fb144104370 2024-05-27T07:39:12.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.240+0000 7fb13a7fc700 1 -- 172.21.15.114:0/4208368512 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb134009df0 con 0x7fb144104370 2024-05-27T07:39:12.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.240+0000 7fb14b495700 1 -- 172.21.15.114:0/4208368512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1441a26d0 con 0x7fb144104370 2024-05-27T07:39:12.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.240+0000 7fb13a7fc700 1 -- 172.21.15.114:0/4208368512 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13400f3c0 con 0x7fb144104370 2024-05-27T07:39:12.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.240+0000 7fb14b495700 1 -- 172.21.15.114:0/4208368512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1441a2c10 con 0x7fb144104370 2024-05-27T07:39:12.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.241+0000 7fb14b495700 1 -- 172.21.15.114:0/4208368512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1440512b0 con 0x7fb144104370 2024-05-27T07:39:12.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.243+0000 7fb13a7fc700 1 -- 172.21.15.114:0/4208368512 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fb134019070 con 0x7fb144104370 2024-05-27T07:39:12.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.243+0000 7fb13a7fc700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb1300b2880 0x7fb1300b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:12.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.243+0000 7fb149231700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb1300b2880 0x7fb1300b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:12.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.243+0000 7fb13a7fc700 1 -- 172.21.15.114:0/4208368512 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fb1340e21d0 con 0x7fb144104370 2024-05-27T07:39:12.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.244+0000 7fb149231700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb1300b2880 0x7fb1300b4d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb144104270 tx=0x7fb14000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:12.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.247+0000 7fb13a7fc700 1 -- 172.21.15.114:0/4208368512 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb1340a9b20 con 0x7fb144104370 2024-05-27T07:39:12.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.459+0000 7fb14b495700 1 -- 172.21.15.114:0/4208368512 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fb144063cb0 con 0x7fb1300b2880 2024-05-27T07:39:12.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.465+0000 7fb13a7fc700 1 -- 172.21.15.114:0/4208368512 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fb144063cb0 con 0x7fb1300b2880 2024-05-27T07:39:12.461 INFO:teuthology.orchestra.run.smithi114.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-27T07:39:12.461 INFO:teuthology.orchestra.run.smithi114.stdout:mds.all 12/12 64s ago 3m smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-05-27T07:39:12.461 INFO:teuthology.orchestra.run.smithi114.stdout:mgr 3/3 64s ago 9m smithi114=x;smithi163=y;smithi186=z;count:3 2024-05-27T07:39:12.462 INFO:teuthology.orchestra.run.smithi114.stdout:mon 3/3 64s ago 9m smithi114:172.21.15.114=a;smithi163:172.21.15.163=b;smithi186:172.21.15.186=c;count:3 2024-05-27T07:39:12.462 INFO:teuthology.orchestra.run.smithi114.stdout:osd 12 64s ago - 2024-05-27T07:39:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.467+0000 7fb14b495700 1 -- 172.21.15.114:0/4208368512 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb1300b2880 msgr2=0x7fb1300b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.467+0000 7fb14b495700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb1300b2880 0x7fb1300b4d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb144104270 tx=0x7fb14000b410 comp rx=0 tx=0).stop 2024-05-27T07:39:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.467+0000 7fb14b495700 1 -- 172.21.15.114:0/4208368512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb144104370 msgr2=0x7fb14419f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.467+0000 7fb14b495700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb144104370 0x7fb14419f640 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb13400d950 tx=0x7fb13400dc60 comp rx=0 tx=0).stop 2024-05-27T07:39:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.467+0000 7fb14b495700 1 -- 172.21.15.114:0/4208368512 shutdown_connections 2024-05-27T07:39:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.467+0000 7fb14b495700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb144104d10 0x7fb1441a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.467+0000 7fb14b495700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7fb1300b2880 0x7fb1300b4d90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:12.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.467+0000 7fb14b495700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb144103240 0x7fb144076440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:12.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.467+0000 7fb14b495700 1 --2- 172.21.15.114:0/4208368512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb144104370 0x7fb14419f640 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:12.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.467+0000 7fb14b495700 1 -- 172.21.15.114:0/4208368512 >> 172.21.15.114:0/4208368512 conn(0x7fb1440fe880 msgr2=0x7fb1441075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:12.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.468+0000 7fb14b495700 1 -- 172.21.15.114:0/4208368512 shutdown_connections 2024-05-27T07:39:12.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:12.468+0000 7fb14b495700 1 -- 172.21.15.114:0/4208368512 wait complete. 2024-05-27T07:39:13.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-27T07:39:13.327 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:13.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:13 smithi114 ceph-mon[32038]: pgmap v393: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:13.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:13 smithi163 ceph-mon[36451]: pgmap v393: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:13.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:13 smithi186 ceph-mon[36647]: pgmap v393: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:14.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:14 smithi163 ceph-mon[36451]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:14.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:14 smithi186 ceph-mon[36647]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:14.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:14 smithi114 ceph-mon[32038]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:15.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.045+0000 7f04f3734700 1 -- 172.21.15.114:0/332471483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec102340 msgr2=0x7f04ec102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:15.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.045+0000 7f04f3734700 1 --2- 172.21.15.114:0/332471483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec102340 0x7f04ec102750 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009990 tx=0x7f04dc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:15.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.045+0000 7f04f3734700 1 -- 172.21.15.114:0/332471483 shutdown_connections 2024-05-27T07:39:15.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.045+0000 7f04f3734700 1 --2- 172.21.15.114:0/332471483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04ec102c90 0x7f04ec10c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:15.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.045+0000 7f04f3734700 1 --2- 172.21.15.114:0/332471483 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04ec1058f0 0x7f04ec105d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:15.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.045+0000 7f04f3734700 1 --2- 172.21.15.114:0/332471483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec102340 0x7f04ec102750 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:15.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.045+0000 7f04f3734700 1 -- 172.21.15.114:0/332471483 >> 172.21.15.114:0/332471483 conn(0x7f04ec0fe010 msgr2=0x7f04ec1004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:15.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.045+0000 7f04f3734700 1 -- 172.21.15.114:0/332471483 shutdown_connections 2024-05-27T07:39:15.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.045+0000 7f04f3734700 1 -- 172.21.15.114:0/332471483 wait complete. 2024-05-27T07:39:15.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.045+0000 7f04f3734700 1 Processor -- start 2024-05-27T07:39:15.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.045+0000 7f04f3734700 1 -- start start 2024-05-27T07:39:15.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f3734700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec102340 0x7f04ec19a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:15.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f3734700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04ec102c90 0x7f04ec19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:15.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f3734700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04ec1058f0 0x7f04ec1a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:15.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f3734700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec10fdc0 con 0x7f04ec102340 2024-05-27T07:39:15.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f3734700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec10fc40 con 0x7f04ec102c90 2024-05-27T07:39:15.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f3734700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec10ff40 con 0x7f04ec1058f0 2024-05-27T07:39:15.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f14d0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec102340 0x7f04ec19a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:15.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f14d0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec102340 0x7f04ec19a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59012/0 (socket says 172.21.15.114:59012) 2024-05-27T07:39:15.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f14d0700 1 -- 172.21.15.114:0/3123302374 learned_addr learned my addr 172.21.15.114:0/3123302374 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:15.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f1cd1700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04ec1058f0 0x7f04ec1a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:15.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f0ccf700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04ec102c90 0x7f04ec19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:15.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f14d0700 1 -- 172.21.15.114:0/3123302374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04ec1058f0 msgr2=0x7f04ec1a1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:15.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f14d0700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04ec1058f0 0x7f04ec1a1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:15.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f14d0700 1 -- 172.21.15.114:0/3123302374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04ec102c90 msgr2=0x7f04ec19ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:15.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f14d0700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04ec102c90 0x7f04ec19ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:15.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f14d0700 1 -- 172.21.15.114:0/3123302374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec1a2440 con 0x7f04ec102340 2024-05-27T07:39:15.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f1cd1700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04ec1058f0 0x7f04ec1a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:15.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f14d0700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec102340 0x7f04ec19a8e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f04e800d950 tx=0x7f04e800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:15.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04e27fc700 1 -- 172.21.15.114:0/3123302374 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e8017070 con 0x7f04ec102340 2024-05-27T07:39:15.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04e27fc700 1 -- 172.21.15.114:0/3123302374 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04e8009730 con 0x7f04ec102340 2024-05-27T07:39:15.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.046+0000 7f04f3734700 1 -- 172.21.15.114:0/3123302374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ec1a2730 con 0x7f04ec102340 2024-05-27T07:39:15.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.047+0000 7f04e27fc700 1 -- 172.21.15.114:0/3123302374 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e8003a50 con 0x7f04ec102340 2024-05-27T07:39:15.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.047+0000 7f04f3734700 1 -- 172.21.15.114:0/3123302374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ec1a2c70 con 0x7f04ec102340 2024-05-27T07:39:15.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.047+0000 7f04f3734700 1 -- 172.21.15.114:0/3123302374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ec0512b0 con 0x7f04ec102340 2024-05-27T07:39:15.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.049+0000 7f04e27fc700 1 -- 172.21.15.114:0/3123302374 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f04e8003bf0 con 0x7f04ec102340 2024-05-27T07:39:15.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.050+0000 7f04e27fc700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f04d80b2880 0x7f04d80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:15.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.050+0000 7f04f0ccf700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f04d80b2880 0x7f04d80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:15.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.050+0000 7f04e27fc700 1 -- 172.21.15.114:0/3123302374 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f04e80e28d0 con 0x7f04ec102340 2024-05-27T07:39:15.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.050+0000 7f04f0ccf700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f04d80b2880 0x7f04d80b4d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00b5c0 tx=0x7f04dc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:15.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.053+0000 7f04e27fc700 1 -- 172.21.15.114:0/3123302374 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f04e8020030 con 0x7f04ec102340 2024-05-27T07:39:15.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.268+0000 7f04f3734700 1 -- 172.21.15.114:0/3123302374 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f04ec063bf0 con 0x7f04d80b2880 2024-05-27T07:39:15.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.269+0000 7f04e27fc700 1 -- 172.21.15.114:0/3123302374 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f04ec063bf0 con 0x7f04d80b2880 2024-05-27T07:39:15.266 INFO:teuthology.orchestra.run.smithi114.stdout:HOST ADDR LABELS STATUS 2024-05-27T07:39:15.266 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114 172.21.15.114 2024-05-27T07:39:15.266 INFO:teuthology.orchestra.run.smithi114.stdout:smithi163 172.21.15.163 2024-05-27T07:39:15.266 INFO:teuthology.orchestra.run.smithi114.stdout:smithi186 172.21.15.186 2024-05-27T07:39:15.266 INFO:teuthology.orchestra.run.smithi114.stdout:3 hosts in cluster 2024-05-27T07:39:15.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.271+0000 7f04f3734700 1 -- 172.21.15.114:0/3123302374 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f04d80b2880 msgr2=0x7f04d80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:15.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.271+0000 7f04f3734700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f04d80b2880 0x7f04d80b4d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00b5c0 tx=0x7f04dc01a040 comp rx=0 tx=0).stop 2024-05-27T07:39:15.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.271+0000 7f04f3734700 1 -- 172.21.15.114:0/3123302374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec102340 msgr2=0x7f04ec19a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:15.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.271+0000 7f04f3734700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec102340 0x7f04ec19a8e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f04e800d950 tx=0x7f04e800dc60 comp rx=0 tx=0).stop 2024-05-27T07:39:15.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.271+0000 7f04f3734700 1 -- 172.21.15.114:0/3123302374 shutdown_connections 2024-05-27T07:39:15.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.271+0000 7f04f3734700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f04d80b2880 0x7f04d80b4d90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:15.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.271+0000 7f04f3734700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04ec1058f0 0x7f04ec1a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:15.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.271+0000 7f04f3734700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec102340 0x7f04ec19a8e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:15.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.271+0000 7f04f3734700 1 --2- 172.21.15.114:0/3123302374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04ec102c90 0x7f04ec19ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:15.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.271+0000 7f04f3734700 1 -- 172.21.15.114:0/3123302374 >> 172.21.15.114:0/3123302374 conn(0x7f04ec0fe010 msgr2=0x7f04ec108890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:15.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.271+0000 7f04f3734700 1 -- 172.21.15.114:0/3123302374 shutdown_connections 2024-05-27T07:39:15.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:15.271+0000 7f04f3734700 1 -- 172.21.15.114:0/3123302374 wait complete. 2024-05-27T07:39:15.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:15 smithi114 ceph-mon[32038]: pgmap v394: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:15.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:15 smithi163 ceph-mon[36451]: pgmap v394: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:15.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:15 smithi186 ceph-mon[36647]: pgmap v394: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:15.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-27T07:39:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:17.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:17 smithi163 ceph-mon[36451]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:17.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:17 smithi163 ceph-mon[36451]: pgmap v395: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:17.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:17 smithi186 ceph-mon[36647]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:17.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:17 smithi186 ceph-mon[36647]: pgmap v395: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:17.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:17 smithi114 ceph-mon[32038]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:17.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:17 smithi114 ceph-mon[32038]: pgmap v395: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.012+0000 7f3bd2047700 1 -- 172.21.15.114:0/435980021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bcc1041a0 msgr2=0x7f3bcc104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.012+0000 7f3bd2047700 1 --2- 172.21.15.114:0/435980021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bcc1041a0 0x7f3bcc104670 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009990 tx=0x7f3bc0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.012+0000 7f3bd2047700 1 -- 172.21.15.114:0/435980021 shutdown_connections 2024-05-27T07:39:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.012+0000 7f3bd2047700 1 --2- 172.21.15.114:0/435980021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bcc104d00 0x7f3bcc10b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.012+0000 7f3bd2047700 1 --2- 172.21.15.114:0/435980021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bcc1037c0 0x7f3bcc103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.012+0000 7f3bd2047700 1 --2- 172.21.15.114:0/435980021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bcc1041a0 0x7f3bcc104670 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.012+0000 7f3bd2047700 1 -- 172.21.15.114:0/435980021 >> 172.21.15.114:0/435980021 conn(0x7f3bcc0fe010 msgr2=0x7f3bcc1004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.012+0000 7f3bd2047700 1 -- 172.21.15.114:0/435980021 shutdown_connections 2024-05-27T07:39:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.012+0000 7f3bd2047700 1 -- 172.21.15.114:0/435980021 wait complete. 2024-05-27T07:39:18.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.013+0000 7f3bd2047700 1 Processor -- start 2024-05-27T07:39:18.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.013+0000 7f3bd2047700 1 -- start start 2024-05-27T07:39:18.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.013+0000 7f3bd2047700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bcc1037c0 0x7f3bcc102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.013+0000 7f3bd2047700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bcc1041a0 0x7f3bcc19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.013+0000 7f3bd2047700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bcc104d00 0x7f3bcc1a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.013+0000 7f3bd2047700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc10ed00 con 0x7f3bcc104d00 2024-05-27T07:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.013+0000 7f3bd2047700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc10eb80 con 0x7f3bcc1037c0 2024-05-27T07:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.013+0000 7f3bd2047700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc10ee80 con 0x7f3bcc1041a0 2024-05-27T07:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.013+0000 7f3bcbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bcc104d00 0x7f3bcc1a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.013+0000 7f3bcbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bcc104d00 0x7f3bcc1a1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59040/0 (socket says 172.21.15.114:59040) 2024-05-27T07:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.013+0000 7f3bcbfff700 1 -- 172.21.15.114:0/2769255437 learned_addr learned my addr 172.21.15.114:0/2769255437 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.013+0000 7f3bcb7fe700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bcc1037c0 0x7f3bcc102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.013+0000 7f3bcaffd700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bcc1041a0 0x7f3bcc19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.014+0000 7f3bcbfff700 1 -- 172.21.15.114:0/2769255437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bcc1041a0 msgr2=0x7f3bcc19f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.014+0000 7f3bcbfff700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bcc1041a0 0x7f3bcc19f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.014+0000 7f3bcbfff700 1 -- 172.21.15.114:0/2769255437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bcc1037c0 msgr2=0x7f3bcc102ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.014+0000 7f3bcbfff700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bcc1037c0 0x7f3bcc102ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.014+0000 7f3bcbfff700 1 -- 172.21.15.114:0/2769255437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bcc1a2450 con 0x7f3bcc104d00 2024-05-27T07:39:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.014+0000 7f3bcbfff700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bcc104d00 0x7f3bcc1a1c20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4007ee0 tx=0x7f3bc400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.014+0000 7f3bc8ff9700 1 -- 172.21.15.114:0/2769255437 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4004bb0 con 0x7f3bcc104d00 2024-05-27T07:39:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.014+0000 7f3bc8ff9700 1 -- 172.21.15.114:0/2769255437 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bc4004d50 con 0x7f3bcc104d00 2024-05-27T07:39:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.014+0000 7f3bd2047700 1 -- 172.21.15.114:0/2769255437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bcc1a2740 con 0x7f3bcc104d00 2024-05-27T07:39:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.014+0000 7f3bc8ff9700 1 -- 172.21.15.114:0/2769255437 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4005920 con 0x7f3bcc104d00 2024-05-27T07:39:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.014+0000 7f3bd2047700 1 -- 172.21.15.114:0/2769255437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bcc1a2c30 con 0x7f3bcc104d00 2024-05-27T07:39:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.015+0000 7f3bd2047700 1 -- 172.21.15.114:0/2769255437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bcc0512b0 con 0x7f3bcc104d00 2024-05-27T07:39:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.017+0000 7f3bc8ff9700 1 -- 172.21.15.114:0/2769255437 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f3bc401a020 con 0x7f3bcc104d00 2024-05-27T07:39:18.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.018+0000 7f3bc8ff9700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3bb40b2880 0x7f3bb40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.018+0000 7f3bcb7fe700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3bb40b2880 0x7f3bb40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.018+0000 7f3bc8ff9700 1 -- 172.21.15.114:0/2769255437 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f3bc40e22d0 con 0x7f3bcc104d00 2024-05-27T07:39:18.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.018+0000 7f3bcb7fe700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3bb40b2880 0x7f3bb40b4d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc005950 tx=0x7f3bbc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:18.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.021+0000 7f3bc8ff9700 1 -- 172.21.15.114:0/2769255437 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3bc4010070 con 0x7f3bcc104d00 2024-05-27T07:39:18.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.232+0000 7f3bd2047700 1 -- 172.21.15.114:0/2769255437 --> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f3bcc063bf0 con 0x7f3bb40b2880 2024-05-27T07:39:18.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.234+0000 7f3bc8ff9700 1 -- 172.21.15.114:0/2769255437 <== mgr.14152 v2:172.21.15.114:6800/2291864634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f3bcc063bf0 con 0x7f3bb40b2880 2024-05-27T07:39:18.232 INFO:teuthology.orchestra.run.smithi114.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-27T07:39:18.232 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400WB400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-05-27T07:39:18.232 INFO:teuthology.orchestra.run.smithi114.stdout:smithi163 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GU400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-05-27T07:39:18.232 INFO:teuthology.orchestra.run.smithi114.stdout:smithi186 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CB400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-05-27T07:39:18.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.236+0000 7f3bd2047700 1 -- 172.21.15.114:0/2769255437 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3bb40b2880 msgr2=0x7f3bb40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.236+0000 7f3bd2047700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3bb40b2880 0x7f3bb40b4d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc005950 tx=0x7f3bbc00b410 comp rx=0 tx=0).stop 2024-05-27T07:39:18.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.236+0000 7f3bd2047700 1 -- 172.21.15.114:0/2769255437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bcc104d00 msgr2=0x7f3bcc1a1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.236+0000 7f3bd2047700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bcc104d00 0x7f3bcc1a1c20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4007ee0 tx=0x7f3bc400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.237+0000 7f3bd2047700 1 -- 172.21.15.114:0/2769255437 shutdown_connections 2024-05-27T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.237+0000 7f3bd2047700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bcc104d00 0x7f3bcc1a1c20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.237+0000 7f3bd2047700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bcc1037c0 0x7f3bcc102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.237+0000 7f3bd2047700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f3bb40b2880 0x7f3bb40b4d90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.237+0000 7f3bd2047700 1 --2- 172.21.15.114:0/2769255437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bcc1041a0 0x7f3bcc19f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.237+0000 7f3bd2047700 1 -- 172.21.15.114:0/2769255437 >> 172.21.15.114:0/2769255437 conn(0x7f3bcc0fe010 msgr2=0x7f3bcc107770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.237+0000 7f3bd2047700 1 -- 172.21.15.114:0/2769255437 shutdown_connections 2024-05-27T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:18.237+0000 7f3bd2047700 1 -- 172.21.15.114:0/2769255437 wait complete. 2024-05-27T07:39:18.894 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-27T07:39:18.903 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-05-27T07:39:18.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-27T07:39:19.147 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:19.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:19 smithi163 ceph-mon[36451]: pgmap v396: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:19.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:19 smithi186 ceph-mon[36647]: pgmap v396: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:19 smithi114 ceph-mon[32038]: pgmap v396: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:20.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:20 smithi114 ceph-mon[32038]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:20.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:20 smithi163 ceph-mon[36451]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:20.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:20 smithi186 ceph-mon[36647]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:20.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.836+0000 7f8c36a59700 1 -- 172.21.15.114:0/3380261624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c30105480 msgr2=0x7f8c30105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:20.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.836+0000 7f8c36a59700 1 --2- 172.21.15.114:0/3380261624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c30105480 0x7f8c30105890 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009990 tx=0x7f8c18009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.836+0000 7f8c36a59700 1 -- 172.21.15.114:0/3380261624 shutdown_connections 2024-05-27T07:39:20.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.836+0000 7f8c36a59700 1 --2- 172.21.15.114:0/3380261624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c30105e20 0x7f8c3010c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.836+0000 7f8c36a59700 1 --2- 172.21.15.114:0/3380261624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c30104350 0x7f8c30104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.836+0000 7f8c36a59700 1 --2- 172.21.15.114:0/3380261624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c30105480 0x7f8c30105890 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.836+0000 7f8c36a59700 1 -- 172.21.15.114:0/3380261624 >> 172.21.15.114:0/3380261624 conn(0x7f8c300ff9b0 msgr2=0x7f8c30101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:20.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.836+0000 7f8c36a59700 1 -- 172.21.15.114:0/3380261624 shutdown_connections 2024-05-27T07:39:20.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.836+0000 7f8c36a59700 1 -- 172.21.15.114:0/3380261624 wait complete. 2024-05-27T07:39:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c36a59700 1 Processor -- start 2024-05-27T07:39:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c36a59700 1 -- start start 2024-05-27T07:39:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c36a59700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c30104350 0x7f8c3019a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c36a59700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c30105480 0x7f8c3019aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c36a59700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c30105e20 0x7f8c301a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c36a59700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3010fde0 con 0x7f8c30105e20 2024-05-27T07:39:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c36a59700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3010fc60 con 0x7f8c30105480 2024-05-27T07:39:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c36a59700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3010ff60 con 0x7f8c30104350 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c34ff6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c30105e20 0x7f8c301a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c34ff6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c30105e20 0x7f8c301a1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59054/0 (socket says 172.21.15.114:59054) 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c34ff6700 1 -- 172.21.15.114:0/4187535222 learned_addr learned my addr 172.21.15.114:0/4187535222 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c2ffff700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c30104350 0x7f8c3019a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c27fff700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c30105480 0x7f8c3019aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c34ff6700 1 -- 172.21.15.114:0/4187535222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c30104350 msgr2=0x7f8c3019a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c34ff6700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c30104350 0x7f8c3019a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c34ff6700 1 -- 172.21.15.114:0/4187535222 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c30105480 msgr2=0x7f8c3019aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c34ff6700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c30105480 0x7f8c3019aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.837+0000 7f8c34ff6700 1 -- 172.21.15.114:0/4187535222 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c301a24f0 con 0x7f8c30105e20 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.838+0000 7f8c34ff6700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c30105e20 0x7f8c301a1e70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8c28007ee0 tx=0x7f8c2800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.838+0000 7f8c2dffb700 1 -- 172.21.15.114:0/4187535222 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c28004bb0 con 0x7f8c30105e20 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.838+0000 7f8c36a59700 1 -- 172.21.15.114:0/4187535222 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c301a27e0 con 0x7f8c30105e20 2024-05-27T07:39:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.838+0000 7f8c2dffb700 1 -- 172.21.15.114:0/4187535222 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c28004d50 con 0x7f8c30105e20 2024-05-27T07:39:20.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.838+0000 7f8c2dffb700 1 -- 172.21.15.114:0/4187535222 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c28005920 con 0x7f8c30105e20 2024-05-27T07:39:20.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.838+0000 7f8c36a59700 1 -- 172.21.15.114:0/4187535222 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c301a2d20 con 0x7f8c30105e20 2024-05-27T07:39:20.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.839+0000 7f8c36a59700 1 -- 172.21.15.114:0/4187535222 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c300512b0 con 0x7f8c30105e20 2024-05-27T07:39:20.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.841+0000 7f8c2dffb700 1 -- 172.21.15.114:0/4187535222 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f8c2801a020 con 0x7f8c30105e20 2024-05-27T07:39:20.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.841+0000 7f8c2dffb700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8c1c0b2880 0x7f8c1c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.841+0000 7f8c2ffff700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8c1c0b2880 0x7f8c1c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.841+0000 7f8c2dffb700 1 -- 172.21.15.114:0/4187535222 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f8c280e22b0 con 0x7f8c30105e20 2024-05-27T07:39:20.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.842+0000 7f8c2ffff700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8c1c0b2880 0x7f8c1c0b4d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8c30105380 tx=0x7f8c2000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:20.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:20.845+0000 7f8c2dffb700 1 -- 172.21.15.114:0/4187535222 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8c28010070 con 0x7f8c30105e20 2024-05-27T07:39:21.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.101+0000 7f8c36a59700 1 -- 172.21.15.114:0/4187535222 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f8c3002e5e0 con 0x7f8c30105e20 2024-05-27T07:39:21.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.101+0000 7f8c2dffb700 1 -- 172.21.15.114:0/4187535222 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) ==== 76+0+3022 (secure 0 0 0) 0x7f8c280a9c00 con 0x7f8c30105e20 2024-05-27T07:39:21.099 INFO:teuthology.orchestra.run.smithi114.stdout:e21 2024-05-27T07:39:21.099 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T07:39:21.099 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:39:21.099 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-05-27T07:39:21.099 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:21.099 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-05-27T07:39:21.099 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 21 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-05-27T07:36:06.848027+0000 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-05-27T07:37:06.645274+0000 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 300 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 0 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:39:21.100 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 1 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:in 0 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=24463} 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3,4] 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data disabled 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 0 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:qdb_cluster leader: 24463 members: 24463 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.i{0:24463} state up:active seq 8 addr [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b{0:24455} state up:standby-replay seq 1 addr [v2:172.21.15.163:6838/3774280856,v1:172.21.15.163:6839/3774280856] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:39:21.101 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:21.102 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:21.102 INFO:teuthology.orchestra.run.smithi114.stdout:Standby daemons: 2024-05-27T07:39:21.102 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:21.102 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.j{-1:14511} state up:standby seq 1 addr [v2:172.21.15.114:6834/807487035,v1:172.21.15.114:6835/807487035] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:39:21.102 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d{-1:14523} state up:standby seq 1 addr [v2:172.21.15.114:6836/1221779710,v1:172.21.15.114:6837/1221779710] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:39:21.102 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.g{-1:14574} state up:standby seq 1 addr [v2:172.21.15.114:6838/2637229588,v1:172.21.15.114:6839/2637229588] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:39:21.102 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a{-1:14586} state up:standby seq 1 addr [v2:172.21.15.114:6840/3861955816,v1:172.21.15.114:6841/3861955816] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:39:21.102 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.k{-1:24383} state up:standby seq 1 addr [v2:172.21.15.163:6832/1125403618,v1:172.21.15.163:6833/1125403618] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:39:21.102 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.e{-1:24407} state up:standby seq 1 addr [v2:172.21.15.163:6834/2110370477,v1:172.21.15.163:6835/2110370477] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:39:21.102 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.l{-1:24412} state up:standby seq 1 addr [v2:172.21.15.186:6832/1801956554,v1:172.21.15.186:6833/1801956554] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:39:21.102 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.h{-1:24422} state up:standby seq 1 addr [v2:172.21.15.163:6836/869688838,v1:172.21.15.163:6837/869688838] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:39:21.102 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.f{-1:24445} state up:standby seq 1 addr [v2:172.21.15.186:6834/199200469,v1:172.21.15.186:6835/199200469] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:39:21.102 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c{-1:24451} state up:standby seq 1 addr [v2:172.21.15.186:6836/2251439141,v1:172.21.15.186:6837/2251439141] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:39:21.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.103+0000 7f8c36a59700 1 -- 172.21.15.114:0/4187535222 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8c1c0b2880 msgr2=0x7f8c1c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:21.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.103+0000 7f8c36a59700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8c1c0b2880 0x7f8c1c0b4d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8c30105380 tx=0x7f8c2000a380 comp rx=0 tx=0).stop 2024-05-27T07:39:21.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.103+0000 7f8c36a59700 1 -- 172.21.15.114:0/4187535222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c30105e20 msgr2=0x7f8c301a1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:21.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.103+0000 7f8c36a59700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c30105e20 0x7f8c301a1e70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8c28007ee0 tx=0x7f8c2800d3b0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.103+0000 7f8c36a59700 1 -- 172.21.15.114:0/4187535222 shutdown_connections 2024-05-27T07:39:21.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.103+0000 7f8c36a59700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c30105e20 0x7f8c301a1e70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.103+0000 7f8c36a59700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f8c1c0b2880 0x7f8c1c0b4d90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.103+0000 7f8c36a59700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c30104350 0x7f8c3019a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.103+0000 7f8c36a59700 1 --2- 172.21.15.114:0/4187535222 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c30105480 0x7f8c3019aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.103+0000 7f8c36a59700 1 -- 172.21.15.114:0/4187535222 >> 172.21.15.114:0/4187535222 conn(0x7f8c300ff9b0 msgr2=0x7f8c301086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:21.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.104+0000 7f8c36a59700 1 -- 172.21.15.114:0/4187535222 shutdown_connections 2024-05-27T07:39:21.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:21.104+0000 7f8c36a59700 1 -- 172.21.15.114:0/4187535222 wait complete. 2024-05-27T07:39:21.104 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:39:21.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:21 smithi163 ceph-mon[36451]: pgmap v397: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:21.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:21 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/4187535222' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T07:39:21.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:21 smithi186 ceph-mon[36647]: pgmap v397: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:21.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:21 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/4187535222' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T07:39:21.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:21 smithi114 ceph-mon[32038]: pgmap v397: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:21.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:21 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/4187535222' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T07:39:21.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-27T07:39:22.218 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:22.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:22 smithi114 ceph-mon[32038]: pgmap v398: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:22.722 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:22 smithi163 ceph-mon[36451]: pgmap v398: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:22.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:22 smithi186 ceph-mon[36647]: pgmap v398: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:23.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.845+0000 7f68e2504700 1 -- 172.21.15.114:0/3973072554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68dc102c30 msgr2=0x7f68dc105050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:23.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.845+0000 7f68e2504700 1 --2- 172.21.15.114:0/3973072554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68dc102c30 0x7f68dc105050 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009990 tx=0x7f68cc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:23.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.845+0000 7f68e2504700 1 -- 172.21.15.114:0/3973072554 shutdown_connections 2024-05-27T07:39:23.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.845+0000 7f68e2504700 1 --2- 172.21.15.114:0/3973072554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68dc107ef0 0x7f68dc108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:23.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.845+0000 7f68e2504700 1 --2- 172.21.15.114:0/3973072554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68dc102c30 0x7f68dc105050 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:23.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.845+0000 7f68e2504700 1 --2- 172.21.15.114:0/3973072554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68dc105590 0x7f68dc1079b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:23.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.845+0000 7f68e2504700 1 -- 172.21.15.114:0/3973072554 >> 172.21.15.114:0/3973072554 conn(0x7f68dc0fe880 msgr2=0x7f68dc100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:23.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.845+0000 7f68e2504700 1 -- 172.21.15.114:0/3973072554 shutdown_connections 2024-05-27T07:39:23.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.845+0000 7f68e2504700 1 -- 172.21.15.114:0/3973072554 wait complete. 2024-05-27T07:39:23.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.846+0000 7f68e2504700 1 Processor -- start 2024-05-27T07:39:23.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.846+0000 7f68e2504700 1 -- start start 2024-05-27T07:39:23.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.846+0000 7f68e2504700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68dc102c30 0x7f68dc1a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:23.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.846+0000 7f68e2504700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68dc105590 0x7f68dc1a1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:23.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.846+0000 7f68e2504700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68dc107ef0 0x7f68dc1a83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:23.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.846+0000 7f68e2504700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc116370 con 0x7f68dc105590 2024-05-27T07:39:23.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.846+0000 7f68e2504700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc1161f0 con 0x7f68dc102c30 2024-05-27T07:39:23.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.846+0000 7f68e2504700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc1164f0 con 0x7f68dc107ef0 2024-05-27T07:39:23.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.846+0000 7f68d3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68dc105590 0x7f68dc1a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:23.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.846+0000 7f68d3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68dc105590 0x7f68dc1a1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59080/0 (socket says 172.21.15.114:59080) 2024-05-27T07:39:23.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.846+0000 7f68d3fff700 1 -- 172.21.15.114:0/4204246872 learned_addr learned my addr 172.21.15.114:0/4204246872 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:23.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.846+0000 7f68dbfff700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68dc102c30 0x7f68dc1a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:23.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.846+0000 7f68e0aa1700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68dc107ef0 0x7f68dc1a83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:23.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.847+0000 7f68d3fff700 1 -- 172.21.15.114:0/4204246872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68dc107ef0 msgr2=0x7f68dc1a83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:23.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.847+0000 7f68d3fff700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68dc107ef0 0x7f68dc1a83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:23.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.847+0000 7f68d3fff700 1 -- 172.21.15.114:0/4204246872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68dc102c30 msgr2=0x7f68dc1a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:23.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.847+0000 7f68d3fff700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68dc102c30 0x7f68dc1a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:23.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.847+0000 7f68d3fff700 1 -- 172.21.15.114:0/4204246872 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68dc1a8a20 con 0x7f68dc105590 2024-05-27T07:39:23.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.847+0000 7f68d3fff700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68dc105590 0x7f68dc1a1400 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f68c400c920 tx=0x7f68c400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:23.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.847+0000 7f68d9ffb700 1 -- 172.21.15.114:0/4204246872 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c4007b00 con 0x7f68dc105590 2024-05-27T07:39:23.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.847+0000 7f68e2504700 1 -- 172.21.15.114:0/4204246872 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68dc1a8d10 con 0x7f68dc105590 2024-05-27T07:39:23.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.847+0000 7f68d9ffb700 1 -- 172.21.15.114:0/4204246872 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68c4004ce0 con 0x7f68dc105590 2024-05-27T07:39:23.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.847+0000 7f68d9ffb700 1 -- 172.21.15.114:0/4204246872 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c4005550 con 0x7f68dc105590 2024-05-27T07:39:23.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.847+0000 7f68e2504700 1 -- 172.21.15.114:0/4204246872 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68dc1a91a0 con 0x7f68dc105590 2024-05-27T07:39:23.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.848+0000 7f68e2504700 1 -- 172.21.15.114:0/4204246872 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68dc0512b0 con 0x7f68dc105590 2024-05-27T07:39:23.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.850+0000 7f68d9ffb700 1 -- 172.21.15.114:0/4204246872 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f68c4007ca0 con 0x7f68dc105590 2024-05-27T07:39:23.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.851+0000 7f68d9ffb700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f68c80b2880 0x7f68c80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:23.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.851+0000 7f68dbfff700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f68c80b2880 0x7f68c80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:23.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.851+0000 7f68d9ffb700 1 -- 172.21.15.114:0/4204246872 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f68c40e25b0 con 0x7f68dc105590 2024-05-27T07:39:23.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.851+0000 7f68dbfff700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f68c80b2880 0x7f68c80b4d90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00b5c0 tx=0x7f68cc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:23.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:23.854+0000 7f68d9ffb700 1 -- 172.21.15.114:0/4204246872 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f68c4014030 con 0x7f68dc105590 2024-05-27T07:39:24.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.060+0000 7f68e2504700 1 -- 172.21.15.114:0/4204246872 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f68dc02e660 con 0x7f68dc105590 2024-05-27T07:39:24.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.061+0000 7f68d9ffb700 1 -- 172.21.15.114:0/4204246872 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v89) ==== 56+0+5738 (secure 0 0 0) 0x7f68c40a9ea0 con 0x7f68dc105590 2024-05-27T07:39:24.058 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 89 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-05-27T07:27:47.693918+0000 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-05-27T07:37:46.129371+0000 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:crush_version 27 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:full_ratio 0.95 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:backfillfull_ratio 0.9 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:nearfull_ratio 0.85 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:require_min_compat_client luminous 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:min_compat_client luminous 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:require_osd_release squid 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:stretch_mode_enabled false 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-27T07:39:24.059 INFO:teuthology.orchestra.run.smithi114.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:max_osd 12 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6802/2663646464,v1:172.21.15.114:6803/2663646464] [v2:172.21.15.114:6804/2663646464,v1:172.21.15.114:6805/2663646464] exists,up 24622a6e-31e4-46ce-89d3-163303255c27 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6800/3356982241,v1:172.21.15.163:6801/3356982241] [v2:172.21.15.163:6802/3356982241,v1:172.21.15.163:6803/3356982241] exists,up e291f805-5a90-4c10-b6d6-3481a96f3f10 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6800/249211108,v1:172.21.15.186:6801/249211108] [v2:172.21.15.186:6802/249211108,v1:172.21.15.186:6803/249211108] exists,up 5bde05b7-5b12-46fe-a8f3-6992ed96f603 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 up in weight 1 up_from 25 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6810/1973916170,v1:172.21.15.114:6811/1973916170] [v2:172.21.15.114:6812/1973916170,v1:172.21.15.114:6813/1973916170] exists,up d810071c-e2dd-442a-b5c5-080b81828549 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 up in weight 1 up_from 31 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6808/4093855927,v1:172.21.15.163:6809/4093855927] [v2:172.21.15.163:6810/4093855927,v1:172.21.15.163:6811/4093855927] exists,up f013c785-92e2-493f-884a-3c71bb47c24c 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 up in weight 1 up_from 36 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6808/3483853241,v1:172.21.15.186:6809/3483853241] [v2:172.21.15.186:6810/3483853241,v1:172.21.15.186:6811/3483853241] exists,up 1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 up in weight 1 up_from 42 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6818/2237458644,v1:172.21.15.114:6819/2237458644] [v2:172.21.15.114:6820/2237458644,v1:172.21.15.114:6821/2237458644] exists,up 21fe0537-2b7a-4588-82fc-a0df295b078c 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 up in weight 1 up_from 47 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6816/1909415962,v1:172.21.15.163:6817/1909415962] [v2:172.21.15.163:6818/1909415962,v1:172.21.15.163:6819/1909415962] exists,up a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.8 up in weight 1 up_from 53 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6816/3389603353,v1:172.21.15.186:6817/3389603353] [v2:172.21.15.186:6818/3389603353,v1:172.21.15.186:6819/3389603353] exists,up 5acc04eb-36af-4a65-9840-dba989b888a5 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.9 up in weight 1 up_from 59 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6826/3900437915,v1:172.21.15.114:6827/3900437915] [v2:172.21.15.114:6828/3900437915,v1:172.21.15.114:6829/3900437915] exists,up 56bba69d-42d5-4bc9-8e80-1761b4fb12cd 2024-05-27T07:39:24.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.10 up in weight 1 up_from 64 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6824/1908338938,v1:172.21.15.163:6825/1908338938] [v2:172.21.15.163:6826/1908338938,v1:172.21.15.163:6827/1908338938] exists,up 825ee9ad-c4c8-4199-98df-76444163633d 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:osd.11 up in weight 1 up_from 70 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6824/2406495501,v1:172.21.15.186:6825/2406495501] [v2:172.21.15.186:6826/2406495501,v1:172.21.15.186:6827/2406495501] exists,up 920e5b32-7159-48c0-9957-d7cb7db2ac38 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 2.a [2,11] 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 2.f [8,11] 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 2.1a [8,1] 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 2.1b [2,4] 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.0 [4,8] 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.c [5,6] 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.14 [4,7] 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.20 [5,9] 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.38 [3,9] 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.3c [4,9] 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 4.7 [5,8] 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 4.c [9,10] 2024-05-27T07:39:24.061 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 4.18 [9,8] 2024-05-27T07:39:24.062 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 4.1f [9,8] 2024-05-27T07:39:24.062 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/3782422776 expires 2024-05-28T07:28:43.271481+0000 2024-05-27T07:39:24.062 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/3323915605 expires 2024-05-28T07:28:43.271481+0000 2024-05-27T07:39:24.062 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/752776783 expires 2024-05-28T07:28:43.271481+0000 2024-05-27T07:39:24.062 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2604585922 expires 2024-05-28T07:28:16.331472+0000 2024-05-27T07:39:24.062 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2303561217 expires 2024-05-28T07:28:16.331472+0000 2024-05-27T07:39:24.062 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/1568684596 expires 2024-05-28T07:28:16.331472+0000 2024-05-27T07:39:24.062 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/1305604861 expires 2024-05-28T07:28:43.271481+0000 2024-05-27T07:39:24.062 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/1568684596 expires 2024-05-28T07:28:16.331472+0000 2024-05-27T07:39:24.062 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/752776783 expires 2024-05-28T07:28:43.271481+0000 2024-05-27T07:39:24.062 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2442312517 expires 2024-05-28T07:28:16.331472+0000 2024-05-27T07:39:24.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.063+0000 7f68e2504700 1 -- 172.21.15.114:0/4204246872 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f68c80b2880 msgr2=0x7f68c80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:24.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.063+0000 7f68e2504700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f68c80b2880 0x7f68c80b4d90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00b5c0 tx=0x7f68cc01a040 comp rx=0 tx=0).stop 2024-05-27T07:39:24.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.063+0000 7f68e2504700 1 -- 172.21.15.114:0/4204246872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68dc105590 msgr2=0x7f68dc1a1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:24.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.063+0000 7f68e2504700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68dc105590 0x7f68dc1a1400 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f68c400c920 tx=0x7f68c400cc30 comp rx=0 tx=0).stop 2024-05-27T07:39:24.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.063+0000 7f68e2504700 1 -- 172.21.15.114:0/4204246872 shutdown_connections 2024-05-27T07:39:24.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.063+0000 7f68e2504700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f68c80b2880 0x7f68c80b4d90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:24.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.063+0000 7f68e2504700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68dc107ef0 0x7f68dc1a83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:24.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.063+0000 7f68e2504700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68dc102c30 0x7f68dc1a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:24.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.063+0000 7f68e2504700 1 --2- 172.21.15.114:0/4204246872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68dc105590 0x7f68dc1a1400 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:24.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.063+0000 7f68e2504700 1 -- 172.21.15.114:0/4204246872 >> 172.21.15.114:0/4204246872 conn(0x7f68dc0fe880 msgr2=0x7f68dc103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:24.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.063+0000 7f68e2504700 1 -- 172.21.15.114:0/4204246872 shutdown_connections 2024-05-27T07:39:24.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:24.063+0000 7f68e2504700 1 -- 172.21.15.114:0/4204246872 wait complete. 2024-05-27T07:39:24.701 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:24 smithi114 ceph-mon[32038]: pgmap v399: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:24.702 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:24 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/4204246872' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T07:39:24.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:24 smithi186 ceph-mon[36647]: pgmap v399: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:24.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:24 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/4204246872' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T07:39:24.728 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-27T07:39:24.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:39:24.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:24 smithi163 ceph-mon[36451]: pgmap v399: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:24.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:24 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/4204246872' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T07:39:24.979 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:26.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.639+0000 7f640e843700 1 -- 172.21.15.114:0/2484738477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6408109840 msgr2=0x7f64081111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:26.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.639+0000 7f640e843700 1 --2- 172.21.15.114:0/2484738477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6408109840 0x7f64081111a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f640000b0d0 tx=0x7f640000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:39:26.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.639+0000 7f640e843700 1 -- 172.21.15.114:0/2484738477 shutdown_connections 2024-05-27T07:39:26.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.639+0000 7f640e843700 1 --2- 172.21.15.114:0/2484738477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6408109840 0x7f64081111a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:26.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.639+0000 7f640e843700 1 --2- 172.21.15.114:0/2484738477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64081045c0 0x7f64081049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:26.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.639+0000 7f640e843700 1 --2- 172.21.15.114:0/2484738477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6408108ef0 0x7f6408109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:26.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.639+0000 7f640e843700 1 -- 172.21.15.114:0/2484738477 >> 172.21.15.114:0/2484738477 conn(0x7f64080ffe90 msgr2=0x7f64081022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:26.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.639+0000 7f640e843700 1 -- 172.21.15.114:0/2484738477 shutdown_connections 2024-05-27T07:39:26.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.639+0000 7f640e843700 1 -- 172.21.15.114:0/2484738477 wait complete. 2024-05-27T07:39:26.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f640e843700 1 Processor -- start 2024-05-27T07:39:26.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f640e843700 1 -- start start 2024-05-27T07:39:26.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f640e843700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64081045c0 0x7f6408076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:26.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f640e843700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6408108ef0 0x7f6408077100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:26.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f640e843700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6408109840 0x7f6408077640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:26.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f640e843700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6408114490 con 0x7f64081045c0 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f640e843700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6408114310 con 0x7f6408109840 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f640e843700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6408114610 con 0x7f6408108ef0 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f6407fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64081045c0 0x7f6408076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f6407fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64081045c0 0x7f6408076bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48568/0 (socket says 172.21.15.114:48568) 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f6407fff700 1 -- 172.21.15.114:0/1944560728 learned_addr learned my addr 172.21.15.114:0/1944560728 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f640cde0700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6408109840 0x7f6408077640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f63fffff700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6408108ef0 0x7f6408077100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f6407fff700 1 -- 172.21.15.114:0/1944560728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6408108ef0 msgr2=0x7f6408077100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f6407fff700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6408108ef0 0x7f6408077100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f6407fff700 1 -- 172.21.15.114:0/1944560728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6408109840 msgr2=0x7f6408077640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f6407fff700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6408109840 0x7f6408077640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f6407fff700 1 -- 172.21.15.114:0/1944560728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64081a9180 con 0x7f64081045c0 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.640+0000 7f640cde0700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6408109840 0x7f6408077640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.641+0000 7f6407fff700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64081045c0 0x7f6408076bc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f63f800d980 tx=0x7f63f800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:26.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.641+0000 7f6405ffb700 1 -- 172.21.15.114:0/1944560728 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8017070 con 0x7f64081045c0 2024-05-27T07:39:26.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.641+0000 7f640e843700 1 -- 172.21.15.114:0/1944560728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64081a9470 con 0x7f64081045c0 2024-05-27T07:39:26.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.641+0000 7f6405ffb700 1 -- 172.21.15.114:0/1944560728 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63f8004830 con 0x7f64081045c0 2024-05-27T07:39:26.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.641+0000 7f6405ffb700 1 -- 172.21.15.114:0/1944560728 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8005440 con 0x7f64081045c0 2024-05-27T07:39:26.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.641+0000 7f640e843700 1 -- 172.21.15.114:0/1944560728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64081a99b0 con 0x7f64081045c0 2024-05-27T07:39:26.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.642+0000 7f640e843700 1 -- 172.21.15.114:0/1944560728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64080512b0 con 0x7f64081045c0 2024-05-27T07:39:26.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.643+0000 7f6405ffb700 1 -- 172.21.15.114:0/1944560728 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f63f80055e0 con 0x7f64081045c0 2024-05-27T07:39:26.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.644+0000 7f6405ffb700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f63f40b2900 0x7f63f40b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:26.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.644+0000 7f63fffff700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f63f40b2900 0x7f63f40b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:26.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.644+0000 7f6405ffb700 1 -- 172.21.15.114:0/1944560728 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f63f80e2500 con 0x7f64081045c0 2024-05-27T07:39:26.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.645+0000 7f63fffff700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f63f40b2900 0x7f63f40b4e10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f63f0005950 tx=0x7f63f00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:26.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.649+0000 7f6405ffb700 1 -- 172.21.15.114:0/1944560728 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f63f8020020 con 0x7f64081045c0 2024-05-27T07:39:26.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:26 smithi114 ceph-mon[32038]: pgmap v400: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:26.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.901+0000 7f640e843700 1 -- 172.21.15.114:0/1944560728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f640802e5e0 con 0x7f64081045c0 2024-05-27T07:39:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.902+0000 7f6405ffb700 1 -- 172.21.15.114:0/1944560728 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11827 (secure 0 0 0) 0x7f63f80a9e50 con 0x7f64081045c0 2024-05-27T07:39:26.900 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:26.900 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:37:06.645274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:39:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.904+0000 7f640e843700 1 -- 172.21.15.114:0/1944560728 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f63f40b2900 msgr2=0x7f63f40b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.904+0000 7f640e843700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f63f40b2900 0x7f63f40b4e10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f63f0005950 tx=0x7f63f00058e0 comp rx=0 tx=0).stop 2024-05-27T07:39:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.904+0000 7f640e843700 1 -- 172.21.15.114:0/1944560728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64081045c0 msgr2=0x7f6408076bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.904+0000 7f640e843700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64081045c0 0x7f6408076bc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f63f800d980 tx=0x7f63f800dd40 comp rx=0 tx=0).stop 2024-05-27T07:39:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.904+0000 7f640e843700 1 -- 172.21.15.114:0/1944560728 shutdown_connections 2024-05-27T07:39:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.904+0000 7f640e843700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6408109840 0x7f6408077640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.904+0000 7f640e843700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64081045c0 0x7f6408076bc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.904+0000 7f640e843700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f63f40b2900 0x7f63f40b4e10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.904+0000 7f640e843700 1 --2- 172.21.15.114:0/1944560728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6408108ef0 0x7f6408077100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.904+0000 7f640e843700 1 -- 172.21.15.114:0/1944560728 >> 172.21.15.114:0/1944560728 conn(0x7f64080ffe90 msgr2=0x7f6408108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.904+0000 7f640e843700 1 -- 172.21.15.114:0/1944560728 shutdown_connections 2024-05-27T07:39:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:26.904+0000 7f640e843700 1 -- 172.21.15.114:0/1944560728 wait complete. 2024-05-27T07:39:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:39:26.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:26 smithi163 ceph-mon[36451]: pgmap v400: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:26.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:26 smithi186 ceph-mon[36647]: pgmap v400: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:27.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:39:27.778 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:27.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:27 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1944560728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:27.873 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:27 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1944560728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:27.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:27 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1944560728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:28.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:28 smithi114 ceph-mon[32038]: pgmap v401: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:28.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:28 smithi163 ceph-mon[36451]: pgmap v401: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:28.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:28 smithi186 ceph-mon[36647]: pgmap v401: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:29.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.414+0000 7f970b3f1700 1 -- 172.21.15.114:0/4244278065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9704108ef0 msgr2=0x7f9704109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.414+0000 7f970b3f1700 1 --2- 172.21.15.114:0/4244278065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9704108ef0 0x7f9704109300 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f96f80099c0 tx=0x7f96f8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:39:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.414+0000 7f970b3f1700 1 -- 172.21.15.114:0/4244278065 shutdown_connections 2024-05-27T07:39:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.414+0000 7f970b3f1700 1 --2- 172.21.15.114:0/4244278065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9704109840 0x7f97041111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.414+0000 7f970b3f1700 1 --2- 172.21.15.114:0/4244278065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f97041045c0 0x7f97041049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.414+0000 7f970b3f1700 1 --2- 172.21.15.114:0/4244278065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9704108ef0 0x7f9704109300 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.414+0000 7f970b3f1700 1 -- 172.21.15.114:0/4244278065 >> 172.21.15.114:0/4244278065 conn(0x7f97040ffe90 msgr2=0x7f97041022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.414+0000 7f970b3f1700 1 -- 172.21.15.114:0/4244278065 shutdown_connections 2024-05-27T07:39:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.414+0000 7f970b3f1700 1 -- 172.21.15.114:0/4244278065 wait complete. 2024-05-27T07:39:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.414+0000 7f970b3f1700 1 Processor -- start 2024-05-27T07:39:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.414+0000 7f970b3f1700 1 -- start start 2024-05-27T07:39:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970b3f1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97041045c0 0x7f970419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970b3f1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9704108ef0 0x7f970419b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970b3f1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9704109840 0x7f970419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970b3f1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9704114450 con 0x7f9704108ef0 2024-05-27T07:39:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970b3f1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f97041142d0 con 0x7f9704109840 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970b3f1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97041145d0 con 0x7f97041045c0 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970898c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9704108ef0 0x7f970419b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970898c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9704108ef0 0x7f970419b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48584/0 (socket says 172.21.15.114:48584) 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970898c700 1 -- 172.21.15.114:0/508909146 learned_addr learned my addr 172.21.15.114:0/508909146 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970918d700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97041045c0 0x7f970419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970998e700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9704109840 0x7f970419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970898c700 1 -- 172.21.15.114:0/508909146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97041045c0 msgr2=0x7f970419ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970898c700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97041045c0 0x7f970419ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970898c700 1 -- 172.21.15.114:0/508909146 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9704109840 msgr2=0x7f970419f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970898c700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9704109840 0x7f970419f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970898c700 1 -- 172.21.15.114:0/508909146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f970419fc20 con 0x7f9704108ef0 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.415+0000 7f970898c700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9704108ef0 0x7f970419b3a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f96f8006010 tx=0x7f96f80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.416+0000 7f97027fc700 1 -- 172.21.15.114:0/508909146 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f8027070 con 0x7f9704108ef0 2024-05-27T07:39:29.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.416+0000 7f97027fc700 1 -- 172.21.15.114:0/508909146 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96f8003f20 con 0x7f9704108ef0 2024-05-27T07:39:29.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.416+0000 7f970b3f1700 1 -- 172.21.15.114:0/508909146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f970419feb0 con 0x7f9704108ef0 2024-05-27T07:39:29.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.416+0000 7f97027fc700 1 -- 172.21.15.114:0/508909146 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f8022c60 con 0x7f9704108ef0 2024-05-27T07:39:29.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.416+0000 7f970b3f1700 1 -- 172.21.15.114:0/508909146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97041a0390 con 0x7f9704108ef0 2024-05-27T07:39:29.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.417+0000 7f970b3f1700 1 -- 172.21.15.114:0/508909146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97040512b0 con 0x7f9704108ef0 2024-05-27T07:39:29.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.418+0000 7f97027fc700 1 -- 172.21.15.114:0/508909146 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f96f802a070 con 0x7f9704108ef0 2024-05-27T07:39:29.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.419+0000 7f97027fc700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f96ec0b2880 0x7f96ec0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:29.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.419+0000 7f970918d700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f96ec0b2880 0x7f96ec0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:29.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.419+0000 7f97027fc700 1 -- 172.21.15.114:0/508909146 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f96f8022e00 con 0x7f9704108ef0 2024-05-27T07:39:29.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.419+0000 7f970918d700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f96ec0b2880 0x7f96ec0b4d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00f440 tx=0x7f96fc0079e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:29.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.422+0000 7f97027fc700 1 -- 172.21.15.114:0/508909146 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f96f8031080 con 0x7f9704108ef0 2024-05-27T07:39:29.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.679+0000 7f970b3f1700 1 -- 172.21.15.114:0/508909146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f970402e5e0 con 0x7f9704108ef0 2024-05-27T07:39:29.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.680+0000 7f97027fc700 1 -- 172.21.15.114:0/508909146 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11827 (secure 0 0 0) 0x7f96f8013070 con 0x7f9704108ef0 2024-05-27T07:39:29.678 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:29.678 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:37:06.645274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:39:29.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.682+0000 7f970b3f1700 1 -- 172.21.15.114:0/508909146 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f96ec0b2880 msgr2=0x7f96ec0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:29.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.682+0000 7f970b3f1700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f96ec0b2880 0x7f96ec0b4d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00f440 tx=0x7f96fc0079e0 comp rx=0 tx=0).stop 2024-05-27T07:39:29.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.682+0000 7f970b3f1700 1 -- 172.21.15.114:0/508909146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9704108ef0 msgr2=0x7f970419b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:29.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.682+0000 7f970b3f1700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9704108ef0 0x7f970419b3a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f96f8006010 tx=0x7f96f80047b0 comp rx=0 tx=0).stop 2024-05-27T07:39:29.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.682+0000 7f970b3f1700 1 -- 172.21.15.114:0/508909146 shutdown_connections 2024-05-27T07:39:29.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.682+0000 7f970b3f1700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9704109840 0x7f970419f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:29.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.682+0000 7f970b3f1700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f96ec0b2880 0x7f96ec0b4d90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:29.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.682+0000 7f970b3f1700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97041045c0 0x7f970419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:29.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.682+0000 7f970b3f1700 1 --2- 172.21.15.114:0/508909146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9704108ef0 0x7f970419b3a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:29.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.682+0000 7f970b3f1700 1 -- 172.21.15.114:0/508909146 >> 172.21.15.114:0/508909146 conn(0x7f97040ffe90 msgr2=0x7f97041086d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:29.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.683+0000 7f970b3f1700 1 -- 172.21.15.114:0/508909146 shutdown_connections 2024-05-27T07:39:29.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:29.683+0000 7f970b3f1700 1 -- 172.21.15.114:0/508909146 wait complete. 2024-05-27T07:39:29.681 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:39:30.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:39:30.625 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:30.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:30 smithi114 ceph-mon[32038]: pgmap v402: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:30.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:30 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/508909146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:30.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:30 smithi163 ceph-mon[36451]: pgmap v402: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:30.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:30 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/508909146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:30.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:30 smithi186 ceph-mon[36647]: pgmap v402: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:30.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:30 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/508909146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:32.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.299+0000 7f8516f8a700 1 -- 172.21.15.114:0/1888519605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8510108730 msgr2=0x7f8510110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:32.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.299+0000 7f8516f8a700 1 --2- 172.21.15.114:0/1888519605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8510108730 0x7f8510110090 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f850800b0d0 tx=0x7f850800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:39:32.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.299+0000 7f8516f8a700 1 -- 172.21.15.114:0/1888519605 shutdown_connections 2024-05-27T07:39:32.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.299+0000 7f8516f8a700 1 --2- 172.21.15.114:0/1888519605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8510108730 0x7f8510110090 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:32.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.299+0000 7f8516f8a700 1 --2- 172.21.15.114:0/1888519605 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f85101034b0 0x7f85101038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:32.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.299+0000 7f8516f8a700 1 --2- 172.21.15.114:0/1888519605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8510107de0 0x7f85101081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:32.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.299+0000 7f8516f8a700 1 -- 172.21.15.114:0/1888519605 >> 172.21.15.114:0/1888519605 conn(0x7f85100fede0 msgr2=0x7f8510101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:32.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.299+0000 7f8516f8a700 1 -- 172.21.15.114:0/1888519605 shutdown_connections 2024-05-27T07:39:32.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.299+0000 7f8516f8a700 1 -- 172.21.15.114:0/1888519605 wait complete. 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.300+0000 7f8516f8a700 1 Processor -- start 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.300+0000 7f8516f8a700 1 -- start start 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.300+0000 7f8516f8a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85101034b0 0x7f85100ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.300+0000 7f8516f8a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8510107de0 0x7f85100ff800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.300+0000 7f8516f8a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8510108730 0x7f85100ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.300+0000 7f8516f8a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f85101133d0 con 0x7f85101034b0 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.300+0000 7f8516f8a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8510113250 con 0x7f8510108730 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.300+0000 7f8516f8a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8510113550 con 0x7f8510107de0 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.300+0000 7f8514d26700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85101034b0 0x7f85100ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.300+0000 7f8514d26700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85101034b0 0x7f85100ff2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48616/0 (socket says 172.21.15.114:48616) 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.300+0000 7f8514d26700 1 -- 172.21.15.114:0/207127463 learned_addr learned my addr 172.21.15.114:0/207127463 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.300+0000 7f8515527700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8510108730 0x7f85100ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.300+0000 7f850ffff700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8510107de0 0x7f85100ff800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:32.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.301+0000 7f8514d26700 1 -- 172.21.15.114:0/207127463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8510107de0 msgr2=0x7f85100ff800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:32.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.301+0000 7f8514d26700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8510107de0 0x7f85100ff800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:32.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.301+0000 7f8514d26700 1 -- 172.21.15.114:0/207127463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8510108730 msgr2=0x7f85100ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:32.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.301+0000 7f8514d26700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8510108730 0x7f85100ffd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:32.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.301+0000 7f8514d26700 1 -- 172.21.15.114:0/207127463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85101a92d0 con 0x7f85101034b0 2024-05-27T07:39:32.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.301+0000 7f850ffff700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8510107de0 0x7f85100ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:32.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.301+0000 7f8514d26700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85101034b0 0x7f85100ff2c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f850400c920 tx=0x7f850400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:32.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.301+0000 7f850dffb700 1 -- 172.21.15.114:0/207127463 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504007b00 con 0x7f85101034b0 2024-05-27T07:39:32.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.301+0000 7f850dffb700 1 -- 172.21.15.114:0/207127463 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8504004ce0 con 0x7f85101034b0 2024-05-27T07:39:32.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.301+0000 7f8516f8a700 1 -- 172.21.15.114:0/207127463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85101a95c0 con 0x7f85101034b0 2024-05-27T07:39:32.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.301+0000 7f850dffb700 1 -- 172.21.15.114:0/207127463 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504005550 con 0x7f85101034b0 2024-05-27T07:39:32.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.301+0000 7f8516f8a700 1 -- 172.21.15.114:0/207127463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85101a9b50 con 0x7f85101034b0 2024-05-27T07:39:32.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.302+0000 7f8516f8a700 1 -- 172.21.15.114:0/207127463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85100512b0 con 0x7f85101034b0 2024-05-27T07:39:32.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.304+0000 7f850dffb700 1 -- 172.21.15.114:0/207127463 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f8504007ca0 con 0x7f85101034b0 2024-05-27T07:39:32.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.304+0000 7f850dffb700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f85000b2900 0x7f85000b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:32.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.305+0000 7f850ffff700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f85000b2900 0x7f85000b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:32.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.305+0000 7f850dffb700 1 -- 172.21.15.114:0/207127463 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f85040e24d0 con 0x7f85101034b0 2024-05-27T07:39:32.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.305+0000 7f850ffff700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f85000b2900 0x7f85000b4e10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f85101099f0 tx=0x7f84fc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:32.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.308+0000 7f850dffb700 1 -- 172.21.15.114:0/207127463 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8504014030 con 0x7f85101034b0 2024-05-27T07:39:32.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.511+0000 7f8516f8a700 1 -- 172.21.15.114:0/207127463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85101007f0 con 0x7f85101034b0 2024-05-27T07:39:32.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.513+0000 7f850dffb700 1 -- 172.21.15.114:0/207127463 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24923 (secure 0 0 0) 0x7f85040a9ea0 con 0x7f85101034b0 2024-05-27T07:39:32.511 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:32.511 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":89,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T07:37:46.129371+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:39:32.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.516+0000 7f8516f8a700 1 -- 172.21.15.114:0/207127463 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f85000b2900 msgr2=0x7f85000b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:32.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.516+0000 7f8516f8a700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f85000b2900 0x7f85000b4e10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f85101099f0 tx=0x7f84fc008040 comp rx=0 tx=0).stop 2024-05-27T07:39:32.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.516+0000 7f8516f8a700 1 -- 172.21.15.114:0/207127463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85101034b0 msgr2=0x7f85100ff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:32.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.516+0000 7f8516f8a700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85101034b0 0x7f85100ff2c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f850400c920 tx=0x7f850400cc30 comp rx=0 tx=0).stop 2024-05-27T07:39:32.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.516+0000 7f8516f8a700 1 -- 172.21.15.114:0/207127463 shutdown_connections 2024-05-27T07:39:32.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.516+0000 7f8516f8a700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8510108730 0x7f85100ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:32.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.516+0000 7f8516f8a700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85101034b0 0x7f85100ff2c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:32.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.516+0000 7f8516f8a700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f85000b2900 0x7f85000b4e10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:32.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.516+0000 7f8516f8a700 1 --2- 172.21.15.114:0/207127463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8510107de0 0x7f85100ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:32.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.516+0000 7f8516f8a700 1 -- 172.21.15.114:0/207127463 >> 172.21.15.114:0/207127463 conn(0x7f85100fede0 msgr2=0x7f8510075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:32.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.516+0000 7f8516f8a700 1 -- 172.21.15.114:0/207127463 shutdown_connections 2024-05-27T07:39:32.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:32.516+0000 7f8516f8a700 1 -- 172.21.15.114:0/207127463 wait complete. 2024-05-27T07:39:32.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:32 smithi114 ceph-mon[32038]: pgmap v403: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:32.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:32 smithi163 ceph-mon[36451]: pgmap v403: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:32.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:32 smithi186 ceph-mon[36647]: pgmap v403: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-27T07:39:33.177 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-27T07:36:06.848027+0000', 'modified': '2024-05-27T07:37:06.645274+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24455': {'gid': 24455, 'name': 'b', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.163:6839/3774280856', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6838', 'nonce': 3774280856}, {'type': 'v1', 'addr': '172.21.15.163:6839', 'nonce': 3774280856}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.186:6839/4115872608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 4115872608}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 4115872608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24463, 'qdb_cluster': [24463]} 2024-05-27T07:39:33.177 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-27T07:39:33.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-05-27T07:39:33.417 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:33.785 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:33 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/207127463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:39:33.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:33 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/207127463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:39:33.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:33 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/207127463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:39:34.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:34 smithi114 ceph-mon[32038]: pgmap v404: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:34.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:34 smithi163 ceph-mon[36451]: pgmap v404: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:34.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:34 smithi186 ceph-mon[36647]: pgmap v404: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:35.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.093+0000 7f237b512700 1 -- 172.21.15.114:0/1704172167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2374103950 msgr2=0x7f2374103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.093+0000 7f237b512700 1 --2- 172.21.15.114:0/1704172167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2374103950 0x7f2374103d60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2364009990 tx=0x7f2364009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.093+0000 7f237b512700 1 -- 172.21.15.114:0/1704172167 shutdown_connections 2024-05-27T07:39:35.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.093+0000 7f237b512700 1 --2- 172.21.15.114:0/1704172167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23741042a0 0x7f2374110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.093+0000 7f237b512700 1 --2- 172.21.15.114:0/1704172167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23741081f0 0x7f2374108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.093+0000 7f237b512700 1 --2- 172.21.15.114:0/1704172167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2374103950 0x7f2374103d60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.093+0000 7f237b512700 1 -- 172.21.15.114:0/1704172167 >> 172.21.15.114:0/1704172167 conn(0x7f23740fe4d0 msgr2=0x7f2374100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:35.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.094+0000 7f237b512700 1 -- 172.21.15.114:0/1704172167 shutdown_connections 2024-05-27T07:39:35.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.094+0000 7f237b512700 1 -- 172.21.15.114:0/1704172167 wait complete. 2024-05-27T07:39:35.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.094+0000 7f237b512700 1 Processor -- start 2024-05-27T07:39:35.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.094+0000 7f237b512700 1 -- start start 2024-05-27T07:39:35.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.094+0000 7f237b512700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2374103950 0x7f237419ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f237b512700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23741042a0 0x7f237419b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f237b512700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23741081f0 0x7f237419f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f237b512700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2374113370 con 0x7f2374103950 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f237b512700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f23741131f0 con 0x7f23741042a0 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f237b512700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f23741134f0 con 0x7f23741081f0 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f23792ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2374103950 0x7f237419ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f23792ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2374103950 0x7f237419ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43862/0 (socket says 172.21.15.114:43862) 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f23792ae700 1 -- 172.21.15.114:0/1740819675 learned_addr learned my addr 172.21.15.114:0/1740819675 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f2378aad700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23741042a0 0x7f237419b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f2379aaf700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23741081f0 0x7f237419f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f23792ae700 1 -- 172.21.15.114:0/1740819675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23741081f0 msgr2=0x7f237419f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f23792ae700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23741081f0 0x7f237419f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f23792ae700 1 -- 172.21.15.114:0/1740819675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23741042a0 msgr2=0x7f237419b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f23792ae700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23741042a0 0x7f237419b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f23792ae700 1 -- 172.21.15.114:0/1740819675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f237419fc60 con 0x7f2374103950 2024-05-27T07:39:35.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f23792ae700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2374103950 0x7f237419ad90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f237000d950 tx=0x7f237000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.095+0000 7f236a7fc700 1 -- 172.21.15.114:0/1740819675 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2370017070 con 0x7f2374103950 2024-05-27T07:39:35.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.096+0000 7f236a7fc700 1 -- 172.21.15.114:0/1740819675 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2370009730 con 0x7f2374103950 2024-05-27T07:39:35.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.096+0000 7f236a7fc700 1 -- 172.21.15.114:0/1740819675 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2370003a50 con 0x7f2374103950 2024-05-27T07:39:35.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.096+0000 7f237b512700 1 -- 172.21.15.114:0/1740819675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f237419ff50 con 0x7f2374103950 2024-05-27T07:39:35.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.096+0000 7f237b512700 1 -- 172.21.15.114:0/1740819675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23741a7870 con 0x7f2374103950 2024-05-27T07:39:35.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.098+0000 7f236a7fc700 1 -- 172.21.15.114:0/1740819675 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f2370003bf0 con 0x7f2374103950 2024-05-27T07:39:35.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.099+0000 7f236a7fc700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f23600b2900 0x7f23600b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.099+0000 7f2378aad700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f23600b2900 0x7f23600b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.099+0000 7f2389f27b80 1 Processor -- start 2024-05-27T07:39:35.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.099+0000 7f236a7fc700 1 -- 172.21.15.114:0/1740819675 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f23700e20d0 con 0x7f2374103950 2024-05-27T07:39:35.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.099+0000 7f2389f27b80 1 -- start start 2024-05-27T07:39:35.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2389f27b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625798fadf0 0x562579a7e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2389f27b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a7ec00 0x562579a789f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2389f27b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a79340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2389f27b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5625799a7c40 con 0x562579a78f30 2024-05-27T07:39:35.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2389f27b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5625799977e0 con 0x562579a7ec00 2024-05-27T07:39:35.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2389f27b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56257999b820 con 0x5625798fadf0 2024-05-27T07:39:35.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2378aad700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f23600b2900 0x7f23600b4e10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f236400b5c0 tx=0x7f2364005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2379aaf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a79340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2379aaf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a79340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43868/0 (socket says 172.21.15.114:43868) 2024-05-27T07:39:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2379aaf700 1 -- 172.21.15.114:0/3798730029 learned_addr learned my addr 172.21.15.114:0/3798730029 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f23792ae700 1 --2- 172.21.15.114:0/3798730029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625798fadf0 0x562579a7e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2379aaf700 1 -- 172.21.15.114:0/3798730029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625798fadf0 msgr2=0x562579a7e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2378aad700 1 --2- 172.21.15.114:0/3798730029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a7ec00 0x562579a789f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2379aaf700 1 --2- 172.21.15.114:0/3798730029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625798fadf0 0x562579a7e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2379aaf700 1 -- 172.21.15.114:0/3798730029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a7ec00 msgr2=0x562579a789f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2379aaf700 1 --2- 172.21.15.114:0/3798730029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a7ec00 0x562579a789f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2379aaf700 1 -- 172.21.15.114:0/3798730029 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625798e7880 con 0x562579a78f30 2024-05-27T07:39:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f23792ae700 1 --2- 172.21.15.114:0/3798730029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625798fadf0 0x562579a7e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2378aad700 1 --2- 172.21.15.114:0/3798730029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a7ec00 0x562579a789f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.100+0000 7f2379aaf700 1 --2- 172.21.15.114:0/3798730029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a79340 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f236c009dc0 tx=0x7f236c0077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=836e6291fa5b8690 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.101+0000 7f235f7fe700 1 -- 172.21.15.114:0/3798730029 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236c014610 con 0x562579a78f30 2024-05-27T07:39:35.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.101+0000 7f235f7fe700 1 -- 172.21.15.114:0/3798730029 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f236c019070 con 0x562579a78f30 2024-05-27T07:39:35.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.101+0000 7f235f7fe700 1 -- 172.21.15.114:0/3798730029 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236c013900 con 0x562579a78f30 2024-05-27T07:39:35.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.102+0000 7f2389f27b80 1 -- 172.21.15.114:0/3798730029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 msgr2=0x562579a79340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.102+0000 7f2389f27b80 1 --2- 172.21.15.114:0/3798730029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a79340 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f236c009dc0 tx=0x7f236c0077e0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.102+0000 7f2389f27b80 1 -- 172.21.15.114:0/3798730029 shutdown_connections 2024-05-27T07:39:35.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.102+0000 7f2389f27b80 1 --2- 172.21.15.114:0/3798730029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a79340 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.102+0000 7f2389f27b80 1 --2- 172.21.15.114:0/3798730029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625798fadf0 0x562579a7e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.102+0000 7f2389f27b80 1 --2- 172.21.15.114:0/3798730029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a7ec00 0x562579a789f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.102+0000 7f2389f27b80 1 -- 172.21.15.114:0/3798730029 >> 172.21.15.114:0/3798730029 conn(0x562579955aa0 msgr2=0x562579a735f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:35.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.102+0000 7f2389f27b80 1 -- 172.21.15.114:0/3798730029 shutdown_connections 2024-05-27T07:39:35.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.102+0000 7f2389f27b80 1 -- 172.21.15.114:0/3798730029 wait complete. 2024-05-27T07:39:35.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.102+0000 7f2389f27b80 1 -- start start 2024-05-27T07:39:35.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.102+0000 7f2389f27b80 1 Processor -- start 2024-05-27T07:39:35.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2389f27b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5625798fadf0 0x562579a82220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2389f27b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a82760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2389f27b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562579a7ec00 0x562579a82ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2389f27b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56257989df50 con 0x562579a78f30 2024-05-27T07:39:35.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2389f27b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562579939770 con 0x5625798fadf0 2024-05-27T07:39:35.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2389f27b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562579a89b00 con 0x562579a7ec00 2024-05-27T07:39:35.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2389f27b80 10 client.0 _ll_register_callbacks cb 0x5625799328d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:35.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2378aad700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a82760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2378aad700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a82760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43878/0 (socket says 172.21.15.114:43878) 2024-05-27T07:39:35.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2378aad700 1 -- 172.21.15.114:0/2063433251 learned_addr learned my addr 172.21.15.114:0/2063433251 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:35.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2378aad700 1 -- 172.21.15.114:0/2063433251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562579a7ec00 msgr2=0x562579a82ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:39:35.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2378aad700 1 --2- 172.21.15.114:0/2063433251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562579a7ec00 0x562579a82ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2378aad700 1 -- 172.21.15.114:0/2063433251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5625798fadf0 msgr2=0x562579a82220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2378aad700 1 --2- 172.21.15.114:0/2063433251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5625798fadf0 0x562579a82220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2378aad700 1 -- 172.21.15.114:0/2063433251 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562579936640 con 0x562579a78f30 2024-05-27T07:39:35.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f2378aad700 1 --2- 172.21.15.114:0/2063433251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a82760 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2364009990 tx=0x7f2364009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf25f902af695521 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.103+0000 7f235cff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:39:35.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f235cff9700 1 -- 172.21.15.114:0/2063433251 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236400bda0 con 0x562579a78f30 2024-05-27T07:39:35.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f235cff9700 1 -- 172.21.15.114:0/2063433251 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f236400f460 con 0x562579a78f30 2024-05-27T07:39:35.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f2389f27b80 1 -- 172.21.15.114:0/2063433251 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f236c019070 con 0x562579a78f30 2024-05-27T07:39:35.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f235cff9700 1 -- 172.21.15.114:0/2063433251 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f236400f660 con 0x562579a78f30 2024-05-27T07:39:35.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f2389f27b80 10 client.15108 fetch_fsmap learned FSMap version 21 2024-05-27T07:39:35.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f2389f27b80 1 -- 172.21.15.114:0/2063433251 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562579a94430 con 0x562579a78f30 2024-05-27T07:39:35.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f235cff9700 1 -- 172.21.15.114:0/2063433251 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f236c019070 con 0x562579a78f30 2024-05-27T07:39:35.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f235cff9700 1 -- 172.21.15.114:0/2063433251 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2364022070 con 0x562579a78f30 2024-05-27T07:39:35.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f2389f27b80 10 client.15108 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:39:35.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f2389f27b80 10 client.15108 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:39:35.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f2389f27b80 20 client.15108 populate_metadata read hostname 'smithi114' 2024-05-27T07:39:35.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f2389f27b80 1 --2- 172.21.15.114:0/2063433251 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562579a96b60 0x562579a97fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f2389f27b80 4 client.15108 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.104+0000 7f2389f27b80 1 -- 172.21.15.114:0/2063433251 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562579a449a0 con 0x562579a96b60 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.105+0000 7f23792ae700 1 --2- 172.21.15.114:0/2063433251 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562579a96b60 0x562579a97fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.105+0000 7f235cff9700 10 client.15108 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.105+0000 7f23792ae700 1 --2- 172.21.15.114:0/2063433251 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562579a96b60 0x562579a97fd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45561d6a1bf30c5c server_cookie=39b0e8a9b432fd9e in_seq=0 out_seq=0 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f235cff9700 1 -- 172.21.15.114:0/2063433251 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562579a449a0 con 0x562579a96b60 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f235cff9700 10 client.15108 handle_command_reply: tid=0 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f2389f27b80 1 client.15108 shutdown 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f2389f27b80 1 -- 172.21.15.114:0/2063433251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 msgr2=0x562579a82760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f2389f27b80 1 --2- 172.21.15.114:0/2063433251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a82760 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2364009990 tx=0x7f2364009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f2389f27b80 1 -- 172.21.15.114:0/2063433251 shutdown_connections 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f2389f27b80 1 --2- 172.21.15.114:0/2063433251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562579a7ec00 0x562579a82ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f2389f27b80 1 --2- 172.21.15.114:0/2063433251 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562579a96b60 0x562579a97fd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f2389f27b80 1 --2- 172.21.15.114:0/2063433251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5625798fadf0 0x562579a82220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f2389f27b80 1 --2- 172.21.15.114:0/2063433251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a82760 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f2389f27b80 1 -- 172.21.15.114:0/2063433251 >> 172.21.15.114:0/2063433251 conn(0x562579955aa0 msgr2=0x562579a779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:35.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f235cff9700 0 client.15108 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:35.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f2389f27b80 1 -- 172.21.15.114:0/2063433251 shutdown_connections 2024-05-27T07:39:35.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.106+0000 7f2389f27b80 1 -- 172.21.15.114:0/2063433251 wait complete. 2024-05-27T07:39:35.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.107+0000 7f2389f27b80 20 client.15108 trim_cache size 0 max 16384 2024-05-27T07:39:35.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2389f27b80 1 Processor -- start 2024-05-27T07:39:35.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2389f27b80 1 -- start start 2024-05-27T07:39:35.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2389f27b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625798fadf0 0x562579a789f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2389f27b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a82ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2389f27b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a7ec00 0x562579a7c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2389f27b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562579a96310 con 0x562579a78f30 2024-05-27T07:39:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2389f27b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562579a969e0 con 0x562579a7ec00 2024-05-27T07:39:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2389f27b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562579a970b0 con 0x5625798fadf0 2024-05-27T07:39:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2378aad700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a82ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2378aad700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a82ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43890/0 (socket says 172.21.15.114:43890) 2024-05-27T07:39:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2378aad700 1 -- 172.21.15.114:0/4114939320 learned_addr learned my addr 172.21.15.114:0/4114939320 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2379aaf700 1 --2- 172.21.15.114:0/4114939320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a7ec00 0x562579a7c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2378aad700 1 -- 172.21.15.114:0/4114939320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625798fadf0 msgr2=0x562579a789f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:39:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2378aad700 1 --2- 172.21.15.114:0/4114939320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625798fadf0 0x562579a789f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2378aad700 1 -- 172.21.15.114:0/4114939320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a7ec00 msgr2=0x562579a7c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2378aad700 1 --2- 172.21.15.114:0/4114939320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a7ec00 0x562579a7c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.128+0000 7f2378aad700 1 -- 172.21.15.114:0/4114939320 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562579a73b20 con 0x562579a78f30 2024-05-27T07:39:35.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.129+0000 7f2378aad700 1 --2- 172.21.15.114:0/4114939320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a82ae0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f236402f9a0 tx=0x7f236400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbf0e6b919def09 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.129+0000 7f235effd700 1 -- 172.21.15.114:0/4114939320 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236400bba0 con 0x562579a78f30 2024-05-27T07:39:35.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.129+0000 7f235effd700 1 -- 172.21.15.114:0/4114939320 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2364027040 con 0x562579a78f30 2024-05-27T07:39:35.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.129+0000 7f235effd700 1 -- 172.21.15.114:0/4114939320 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2364018400 con 0x562579a78f30 2024-05-27T07:39:35.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.129+0000 7f2389f27b80 1 -- 172.21.15.114:0/4114939320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 msgr2=0x562579a82ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.129+0000 7f2389f27b80 1 --2- 172.21.15.114:0/4114939320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a82ae0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f236402f9a0 tx=0x7f236400f520 comp rx=0 tx=0).stop 2024-05-27T07:39:35.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.130+0000 7f2389f27b80 1 -- 172.21.15.114:0/4114939320 shutdown_connections 2024-05-27T07:39:35.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.130+0000 7f2389f27b80 1 --2- 172.21.15.114:0/4114939320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a7ec00 0x562579a7c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.130+0000 7f2389f27b80 1 --2- 172.21.15.114:0/4114939320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625798fadf0 0x562579a789f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.130+0000 7f2389f27b80 1 --2- 172.21.15.114:0/4114939320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562579a78f30 0x562579a82ae0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.130+0000 7f2389f27b80 1 -- 172.21.15.114:0/4114939320 >> 172.21.15.114:0/4114939320 conn(0x562579955aa0 msgr2=0x562579a7bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:35.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.130+0000 7f2389f27b80 1 -- 172.21.15.114:0/4114939320 shutdown_connections 2024-05-27T07:39:35.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.130+0000 7f2389f27b80 1 -- 172.21.15.114:0/4114939320 wait complete. 2024-05-27T07:39:35.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.130+0000 7f2389f27b80 1 -- start start 2024-05-27T07:39:35.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.130+0000 7f2389f27b80 1 Processor -- start 2024-05-27T07:39:35.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f2389f27b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625798fadf0 0x562579aa1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f2389f27b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a78f30 0x562579aa1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f2389f27b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562579a7ec00 0x562579a9c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f2389f27b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562579a97e50 con 0x5625798fadf0 2024-05-27T07:39:35.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f2389f27b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562579a97780 con 0x562579a78f30 2024-05-27T07:39:35.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f2389f27b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562579a98520 con 0x562579a7ec00 2024-05-27T07:39:35.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f2389f27b80 10 client.0 _ll_register_callbacks cb 0x5625799328d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:35.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f23792ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625798fadf0 0x562579aa1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f23792ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625798fadf0 0x562579aa1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43896/0 (socket says 172.21.15.114:43896) 2024-05-27T07:39:35.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f23792ae700 1 -- 172.21.15.114:0/789267042 learned_addr learned my addr 172.21.15.114:0/789267042 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:35.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f23792ae700 1 -- 172.21.15.114:0/789267042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562579a7ec00 msgr2=0x562579a9c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f2378aad700 1 --2- 172.21.15.114:0/789267042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a78f30 0x562579aa1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f23792ae700 1 --2- 172.21.15.114:0/789267042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562579a7ec00 0x562579a9c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f23792ae700 1 -- 172.21.15.114:0/789267042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a78f30 msgr2=0x562579aa1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f23792ae700 1 --2- 172.21.15.114:0/789267042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a78f30 0x562579aa1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f23792ae700 1 -- 172.21.15.114:0/789267042 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562579a82d00 con 0x5625798fadf0 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f2378aad700 1 --2- 172.21.15.114:0/789267042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a78f30 0x562579aa1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f23792ae700 1 --2- 172.21.15.114:0/789267042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625798fadf0 0x562579aa1450 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f237000b560 tx=0x7f23700a5290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b27ea62467c92ab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f235cff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f235cff9700 1 -- 172.21.15.114:0/789267042 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23700c3060 con 0x5625798fadf0 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f235cff9700 1 -- 172.21.15.114:0/789267042 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f2370009b70 con 0x5625798fadf0 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.131+0000 7f2389f27b80 1 -- 172.21.15.114:0/789267042 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2364027040 con 0x5625798fadf0 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.132+0000 7f235cff9700 1 -- 172.21.15.114:0/789267042 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f23700b2180 con 0x5625798fadf0 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.132+0000 7f2389f27b80 10 client.15114 fetch_fsmap learned FSMap version 21 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.132+0000 7f2389f27b80 1 -- 172.21.15.114:0/789267042 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562579b027f0 con 0x5625798fadf0 2024-05-27T07:39:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.132+0000 7f235cff9700 1 -- 172.21.15.114:0/789267042 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2364027040 con 0x5625798fadf0 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.132+0000 7f235cff9700 1 -- 172.21.15.114:0/789267042 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f23700b1cf0 con 0x5625798fadf0 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.132+0000 7f2389f27b80 10 client.15114 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.132+0000 7f2389f27b80 10 client.15114 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.132+0000 7f2389f27b80 20 client.15114 populate_metadata read hostname 'smithi114' 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.132+0000 7f2389f27b80 1 --2- 172.21.15.114:0/789267042 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562579b04ca0 0x562579b06110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.132+0000 7f2389f27b80 4 client.15114 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.132+0000 7f2389f27b80 1 -- 172.21.15.114:0/789267042 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "status"}) -- 0x5625799a9b90 con 0x562579b04ca0 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.132+0000 7f2378aad700 1 --2- 172.21.15.114:0/789267042 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562579b04ca0 0x562579b06110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f2378aad700 1 --2- 172.21.15.114:0/789267042 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562579b04ca0 0x562579b06110 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd2621cbb053f82e server_cookie=a11c9442b223952c in_seq=0 out_seq=0 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f235cff9700 10 client.15114 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f235cff9700 1 -- 172.21.15.114:0/789267042 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5625799a9b90 con 0x562579b04ca0 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f235cff9700 10 client.15114 handle_command_reply: tid=0 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f2389f27b80 1 client.15114 shutdown 2024-05-27T07:39:35.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f2389f27b80 1 -- 172.21.15.114:0/789267042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625798fadf0 msgr2=0x562579aa1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f2389f27b80 1 --2- 172.21.15.114:0/789267042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625798fadf0 0x562579aa1450 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f237000b560 tx=0x7f23700a5290 comp rx=0 tx=0).stop 2024-05-27T07:39:35.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f2389f27b80 1 -- 172.21.15.114:0/789267042 shutdown_connections 2024-05-27T07:39:35.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f2389f27b80 1 --2- 172.21.15.114:0/789267042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562579a7ec00 0x562579a9c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f2389f27b80 1 --2- 172.21.15.114:0/789267042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625798fadf0 0x562579aa1450 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f2389f27b80 1 --2- 172.21.15.114:0/789267042 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562579b04ca0 0x562579b06110 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f2389f27b80 1 --2- 172.21.15.114:0/789267042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562579a78f30 0x562579aa1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f2389f27b80 1 -- 172.21.15.114:0/789267042 >> 172.21.15.114:0/789267042 conn(0x562579955aa0 msgr2=0x562579a7d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:35.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f235cff9700 0 client.15114 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:35.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f2389f27b80 1 -- 172.21.15.114:0/789267042 shutdown_connections 2024-05-27T07:39:35.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f2389f27b80 1 -- 172.21.15.114:0/789267042 wait complete. 2024-05-27T07:39:35.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.133+0000 7f2389f27b80 20 client.15114 trim_cache size 0 max 16384 2024-05-27T07:39:35.137 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:39:35.137 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "755b3728-1bfa-11ef-bc9b-c7b262605968", 2024-05-27T07:39:35.137 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-05-27T07:39:35.137 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24463, 2024-05-27T07:39:35.137 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-05-27T07:39:35.137 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-05-27T07:39:35.137 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-05-27T07:39:35.137 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 208.15901858000001, 2024-05-27T07:39:35.137 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 21, 2024-05-27T07:39:35.137 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 89, 2024-05-27T07:39:35.137 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 87, 2024-05-27T07:39:35.138 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 233.09624525999999 2024-05-27T07:39:35.138 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:39:35.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.134+0000 7f237b512700 1 -- 172.21.15.114:0/1740819675 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f23600b2900 msgr2=0x7f23600b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.134+0000 7f237b512700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f23600b2900 0x7f23600b4e10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f236400b5c0 tx=0x7f2364005960 comp rx=0 tx=0).stop 2024-05-27T07:39:35.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.134+0000 7f237b512700 1 -- 172.21.15.114:0/1740819675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2374103950 msgr2=0x7f237419ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.134+0000 7f237b512700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2374103950 0x7f237419ad90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f237000d950 tx=0x7f237000dc60 comp rx=0 tx=0).stop 2024-05-27T07:39:35.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.134+0000 7f237b512700 1 -- 172.21.15.114:0/1740819675 shutdown_connections 2024-05-27T07:39:35.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.134+0000 7f237b512700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f23600b2900 0x7f23600b4e10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.134+0000 7f237b512700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23741081f0 0x7f237419f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.134+0000 7f237b512700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2374103950 0x7f237419ad90 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.134+0000 7f237b512700 1 --2- 172.21.15.114:0/1740819675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23741042a0 0x7f237419b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.134+0000 7f237b512700 1 -- 172.21.15.114:0/1740819675 >> 172.21.15.114:0/1740819675 conn(0x7f23740fe4d0 msgr2=0x7f2374102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:35.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.134+0000 7f237b512700 1 -- 172.21.15.114:0/1740819675 shutdown_connections 2024-05-27T07:39:35.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.134+0000 7f237b512700 1 -- 172.21.15.114:0/1740819675 wait complete. 2024-05-27T07:39:35.834 INFO:teuthology.run_tasks:Running task sequential... 2024-05-27T07:39:35.844 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-27T07:39:35.844 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-27T07:39:35.845 INFO:teuthology.task.print:Enabling mgr modules 2024-05-27T07:39:35.845 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-05-27T07:39:35.846 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:39:35.846 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-05-27T07:39:35.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-05-27T07:39:35.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.961+0000 7f601c30d640 1 -- 172.21.15.114:0/2089791417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6014102b50 msgr2=0x7f6014103850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.961+0000 7f601c30d640 1 --2- 172.21.15.114:0/2089791417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6014102b50 0x7f6014103850 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6004009ca0 tx=0x7f600402a8c0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.961+0000 7f601c30d640 1 -- 172.21.15.114:0/2089791417 shutdown_connections 2024-05-27T07:39:35.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.961+0000 7f601c30d640 1 --2- 172.21.15.114:0/2089791417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6014108a90 0x7f601410aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.961+0000 7f601c30d640 1 --2- 172.21.15.114:0/2089791417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6014103d90 0x7f60141041a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.961+0000 7f601c30d640 1 --2- 172.21.15.114:0/2089791417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6014102b50 0x7f6014103850 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.961+0000 7f601c30d640 1 -- 172.21.15.114:0/2089791417 >> 172.21.15.114:0/2089791417 conn(0x7f60140fdd40 msgr2=0x7f60141001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:35.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.961+0000 7f601c30d640 1 -- 172.21.15.114:0/2089791417 shutdown_connections 2024-05-27T07:39:35.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.961+0000 7f601c30d640 1 -- 172.21.15.114:0/2089791417 wait complete. 2024-05-27T07:39:35.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.962+0000 7f601c30d640 1 Processor -- start 2024-05-27T07:39:35.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.962+0000 7f601c30d640 1 -- start start 2024-05-27T07:39:35.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.962+0000 7f601c30d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6014102b50 0x7f6014199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.962+0000 7f601c30d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6014103d90 0x7f6014199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.962+0000 7f601c30d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6014108a90 0x7f60141a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.962+0000 7f601c30d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f601410e6b0 con 0x7f6014102b50 2024-05-27T07:39:35.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.962+0000 7f601c30d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f601410e530 con 0x7f6014103d90 2024-05-27T07:39:35.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.962+0000 7f601c30d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f601410e830 con 0x7f6014108a90 2024-05-27T07:39:35.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.962+0000 7f601a0a9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6014102b50 0x7f6014199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.962+0000 7f601a0a9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6014102b50 0x7f6014199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43908/0 (socket says 172.21.15.114:43908) 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.962+0000 7f601a0a9640 1 -- 172.21.15.114:0/3597207250 learned_addr learned my addr 172.21.15.114:0/3597207250 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.962+0000 7f60198a8640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6014103d90 0x7f6014199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.963+0000 7f601a8aa640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6014108a90 0x7f60141a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.963+0000 7f601a0a9640 1 -- 172.21.15.114:0/3597207250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6014108a90 msgr2=0x7f60141a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.963+0000 7f601a0a9640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6014108a90 0x7f60141a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.963+0000 7f601a0a9640 1 -- 172.21.15.114:0/3597207250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6014103d90 msgr2=0x7f6014199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.963+0000 7f601a0a9640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6014103d90 0x7f6014199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.963+0000 7f601a0a9640 1 -- 172.21.15.114:0/3597207250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60141a0db0 con 0x7f6014102b50 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.963+0000 7f60198a8640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6014103d90 0x7f6014199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.963+0000 7f601a0a9640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6014102b50 0x7f6014199320 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6004033040 tx=0x7f6004003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.963+0000 7f60137fe640 1 -- 172.21.15.114:0/3597207250 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600404a070 con 0x7f6014102b50 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.963+0000 7f601c30d640 1 -- 172.21.15.114:0/3597207250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6014074ef0 con 0x7f6014102b50 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.963+0000 7f60137fe640 1 -- 172.21.15.114:0/3597207250 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60040317c0 con 0x7f6014102b50 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.963+0000 7f60137fe640 1 -- 172.21.15.114:0/3597207250 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600402fc30 con 0x7f6014102b50 2024-05-27T07:39:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.963+0000 7f601c30d640 1 -- 172.21.15.114:0/3597207250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6014075510 con 0x7f6014102b50 2024-05-27T07:39:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.964+0000 7f601c30d640 1 -- 172.21.15.114:0/3597207250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6014107310 con 0x7f6014102b50 2024-05-27T07:39:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.965+0000 7f60137fe640 1 -- 172.21.15.114:0/3597207250 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f600404d070 con 0x7f6014102b50 2024-05-27T07:39:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.967+0000 7f60137fe640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5ffc0b27e0 0x7f5ffc0b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.967+0000 7f60198a8640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5ffc0b27e0 0x7f5ffc0b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.967+0000 7f60137fe640 1 -- 172.21.15.114:0/3597207250 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f6004105110 con 0x7f6014102b50 2024-05-27T07:39:35.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.967+0000 7f60198a8640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5ffc0b27e0 0x7f5ffc0b4cf0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f601419a3c0 tx=0x7f6008008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:35.970+0000 7f60137fe640 1 -- 172.21.15.114:0/3597207250 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6004082df0 con 0x7f6014102b50 2024-05-27T07:39:36.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.200+0000 7f601c30d640 1 -- 172.21.15.114:0/3597207250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f60141a0fe0 con 0x7f6014102b50 2024-05-27T07:39:36.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.517+0000 7f60137fe640 1 -- 172.21.15.114:0/3597207250 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f60040cca00 con 0x7f6014102b50 2024-05-27T07:39:36.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.520+0000 7f601c30d640 1 -- 172.21.15.114:0/3597207250 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5ffc0b27e0 msgr2=0x7f5ffc0b4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.520+0000 7f601c30d640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5ffc0b27e0 0x7f5ffc0b4cf0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f601419a3c0 tx=0x7f6008008040 comp rx=0 tx=0).stop 2024-05-27T07:39:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.520+0000 7f601c30d640 1 -- 172.21.15.114:0/3597207250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6014102b50 msgr2=0x7f6014199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.520+0000 7f601c30d640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6014102b50 0x7f6014199320 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6004033040 tx=0x7f6004003ba0 comp rx=0 tx=0).stop 2024-05-27T07:39:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.520+0000 7f601c30d640 1 -- 172.21.15.114:0/3597207250 shutdown_connections 2024-05-27T07:39:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.520+0000 7f601c30d640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f5ffc0b27e0 0x7f5ffc0b4cf0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.520+0000 7f601c30d640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6014108a90 0x7f60141a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.520+0000 7f601c30d640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6014103d90 0x7f6014199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.520+0000 7f601c30d640 1 --2- 172.21.15.114:0/3597207250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6014102b50 0x7f6014199320 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.520+0000 7f601c30d640 1 -- 172.21.15.114:0/3597207250 >> 172.21.15.114:0/3597207250 conn(0x7f60140fdd40 msgr2=0x7f6014104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.520+0000 7f601c30d640 1 -- 172.21.15.114:0/3597207250 shutdown_connections 2024-05-27T07:39:36.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:36.521+0000 7f601c30d640 1 -- 172.21.15.114:0/3597207250 wait complete. 2024-05-27T07:39:36.540 INFO:teuthology.run_tasks:Running task kclient... 2024-05-27T07:39:36.554 INFO:tasks.kclient:Mounting kernel clients... 2024-05-27T07:39:36.554 INFO:tasks.kclient:config is {'client.0': {}} 2024-05-27T07:39:36.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:39:36.554 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-27T07:39:36.571 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-05-27T07:39:36.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:39:36.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:39:36.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:39:36.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-05-27T07:39:36.740 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-05-27T07:39:36.740 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-27T07:39:36.741 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T07:39:36.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-05-27T07:39:36.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:36 smithi114 ceph-mon[32038]: pgmap v405: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:36.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:36 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3597207250' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T07:39:36.804 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: ignoring --setuser ceph since I am not root 2024-05-27T07:39:36.804 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: ignoring --setgroup ceph since I am not root 2024-05-27T07:39:36.804 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:36.743+0000 7f55d556f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:39:36.912 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:36 smithi186 ceph-mon[36647]: pgmap v405: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:36.912 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:36 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3597207250' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T07:39:36.912 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:36 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: ignoring --setuser ceph since I am not root 2024-05-27T07:39:36.912 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:36 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: ignoring --setgroup ceph since I am not root 2024-05-27T07:39:36.912 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:36 smithi186 ceph-mgr[38500]: -- 172.21.15.186:0/808031779 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eed497c5a0 con 0x55eed9706900 2024-05-27T07:39:36.912 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:36 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:36.726+0000 7f8f5db26200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:39:36.913 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:36 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:36.914+0000 7f8f5db26200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:39:36.916 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:36 smithi163 ceph-mon[36451]: pgmap v405: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:39:36.916 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:36 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3597207250' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T07:39:36.916 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:36 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: ignoring --setuser ceph since I am not root 2024-05-27T07:39:36.916 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:36 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: ignoring --setgroup ceph since I am not root 2024-05-27T07:39:36.916 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:36 smithi163 ceph-mgr[37822]: -- 172.21.15.163:0/3377540840 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b0999865a0 con 0x55b09e710900 2024-05-27T07:39:36.916 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:36 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:36.731+0000 7fde49c79200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:39:36.916 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:36 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:36.918+0000 7fde49c79200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:39:36.991 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:37.223 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:37 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:37.022+0000 7fde49c79200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:39:37.225 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:37.017+0000 7f8f5db26200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:39:37.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:36.944+0000 7f55d556f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:39:37.252 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:37 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:37.069+0000 7f55d556f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:39:37.531 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:37 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:37.394+0000 7f55d556f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:39:37.532 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:37 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:37.485+0000 7f55d556f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:39:37.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:37 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3597207250' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T07:39:37.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:37 smithi163 ceph-mon[36451]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-05-27T07:39:37.723 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:37 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:37.331+0000 7fde49c79200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:39:37.723 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:37 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:37.418+0000 7fde49c79200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:39:37.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:37 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3597207250' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T07:39:37.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:37 smithi186 ceph-mon[36647]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-05-27T07:39:37.726 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:37.326+0000 7f8f5db26200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:39:37.726 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:37.413+0000 7f8f5db26200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:39:37.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:37 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3597207250' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T07:39:37.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:37 smithi114 ceph-mon[32038]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-05-27T07:39:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.646+0000 7f960485c700 1 -- 172.21.15.114:0/3807581324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96001034b0 msgr2=0x7f96001038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.646+0000 7f960485c700 1 --2- 172.21.15.114:0/3807581324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96001034b0 0x7f96001038c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f95f00099c0 tx=0x7f95f0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:39:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.647+0000 7f960485c700 1 -- 172.21.15.114:0/3807581324 shutdown_connections 2024-05-27T07:39:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.647+0000 7f960485c700 1 --2- 172.21.15.114:0/3807581324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9600108730 0x7f9600110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.647+0000 7f960485c700 1 --2- 172.21.15.114:0/3807581324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96001034b0 0x7f96001038c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.647+0000 7f960485c700 1 --2- 172.21.15.114:0/3807581324 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9600107de0 0x7f96001081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.647+0000 7f960485c700 1 -- 172.21.15.114:0/3807581324 >> 172.21.15.114:0/3807581324 conn(0x7f96000fedb0 msgr2=0x7f96001011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.647+0000 7f960485c700 1 -- 172.21.15.114:0/3807581324 shutdown_connections 2024-05-27T07:39:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.647+0000 7f960485c700 1 -- 172.21.15.114:0/3807581324 wait complete. 2024-05-27T07:39:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f960485c700 1 Processor -- start 2024-05-27T07:39:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f960485c700 1 -- start start 2024-05-27T07:39:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f960485c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96001034b0 0x7f9600199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f960485c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9600107de0 0x7f960019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f960485c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9600108730 0x7f960019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f960485c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f96001133d0 con 0x7f9600107de0 2024-05-27T07:39:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f960485c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9600113250 con 0x7f96001034b0 2024-05-27T07:39:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f960485c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9600113550 con 0x7f9600108730 2024-05-27T07:39:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f95fdd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9600107de0 0x7f960019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f95fdd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9600107de0 0x7f960019a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43950/0 (socket says 172.21.15.114:43950) 2024-05-27T07:39:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f95fdd9b700 1 -- 172.21.15.114:0/3898915371 learned_addr learned my addr 172.21.15.114:0/3898915371 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f95fe59c700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96001034b0 0x7f9600199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f95fed9d700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9600108730 0x7f960019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f95fdd9b700 1 -- 172.21.15.114:0/3898915371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9600108730 msgr2=0x7f960019e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f95fdd9b700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9600108730 0x7f960019e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f95fdd9b700 1 -- 172.21.15.114:0/3898915371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96001034b0 msgr2=0x7f9600199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f95fdd9b700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96001034b0 0x7f9600199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f95fdd9b700 1 -- 172.21.15.114:0/3898915371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960019ec10 con 0x7f9600107de0 2024-05-27T07:39:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.648+0000 7f95fe59c700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96001034b0 0x7f9600199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.649+0000 7f95fdd9b700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9600107de0 0x7f960019a2e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f95f40098f0 tx=0x7f95f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:38.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.650+0000 7f95ef7fe700 1 -- 172.21.15.114:0/3898915371 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f400d020 con 0x7f9600107de0 2024-05-27T07:39:38.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.650+0000 7f95ef7fe700 1 -- 172.21.15.114:0/3898915371 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95f400d1c0 con 0x7f9600107de0 2024-05-27T07:39:38.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.650+0000 7f960485c700 1 -- 172.21.15.114:0/3898915371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960019ef00 con 0x7f9600107de0 2024-05-27T07:39:38.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.650+0000 7f95ef7fe700 1 -- 172.21.15.114:0/3898915371 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4007860 con 0x7f9600107de0 2024-05-27T07:39:38.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.650+0000 7f960485c700 1 -- 172.21.15.114:0/3898915371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f960006b600 con 0x7f9600107de0 2024-05-27T07:39:38.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.651+0000 7f960485c700 1 -- 172.21.15.114:0/3898915371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96000512b0 con 0x7f9600107de0 2024-05-27T07:39:38.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.652+0000 7f95ef7fe700 1 -- 172.21.15.114:0/3898915371 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 149670+0+0 (secure 0 0 0) 0x7f95f4007a00 con 0x7f9600107de0 2024-05-27T07:39:38.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.653+0000 7f95ef7fe700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f95e80b2950 0x7f95e80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:38.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.653+0000 7f95fe59c700 1 -- 172.21.15.114:0/3898915371 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f95e80b2950 msgr2=0x7f95e80b4e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2291864634 2024-05-27T07:39:38.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.653+0000 7f95ef7fe700 1 -- 172.21.15.114:0/3898915371 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f95f40e3820 con 0x7f9600107de0 2024-05-27T07:39:38.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.653+0000 7f95fe59c700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f95e80b2950 0x7f95e80b4e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:39:38.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.658+0000 7f95ef7fe700 1 -- 172.21.15.114:0/3898915371 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f95f40ab170 con 0x7f9600107de0 2024-05-27T07:39:38.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.854+0000 7f95fe59c700 1 -- 172.21.15.114:0/3898915371 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f95e80b2950 msgr2=0x7f95e80b4e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2291864634 2024-05-27T07:39:38.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.854+0000 7f95fe59c700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f95e80b2950 0x7f95e80b4e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:39:38.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.913+0000 7f960485c700 1 -- 172.21.15.114:0/3898915371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f960006c080 con 0x7f9600107de0 2024-05-27T07:39:38.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.913+0000 7f95ef7fe700 1 -- 172.21.15.114:0/3898915371 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+88 (secure 0 0 0) 0x7f95f40aaa70 con 0x7f9600107de0 2024-05-27T07:39:38.911 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-27T07:39:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.916+0000 7f960485c700 1 -- 172.21.15.114:0/3898915371 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f95e80b2950 msgr2=0x7f95e80b4e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:39:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.916+0000 7f960485c700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f95e80b2950 0x7f95e80b4e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.916+0000 7f960485c700 1 -- 172.21.15.114:0/3898915371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9600107de0 msgr2=0x7f960019a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.916+0000 7f960485c700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9600107de0 0x7f960019a2e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f95f40098f0 tx=0x7f95f4009c00 comp rx=0 tx=0).stop 2024-05-27T07:39:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.916+0000 7f960485c700 1 -- 172.21.15.114:0/3898915371 shutdown_connections 2024-05-27T07:39:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.916+0000 7f960485c700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f95e80b2950 0x7f95e80b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.916+0000 7f960485c700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9600108730 0x7f960019e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.916+0000 7f960485c700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96001034b0 0x7f9600199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.916+0000 7f960485c700 1 --2- 172.21.15.114:0/3898915371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9600107de0 0x7f960019a2e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.916+0000 7f960485c700 1 -- 172.21.15.114:0/3898915371 >> 172.21.15.114:0/3898915371 conn(0x7f96000fedb0 msgr2=0x7f96001075c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.916+0000 7f960485c700 1 -- 172.21.15.114:0/3898915371 shutdown_connections 2024-05-27T07:39:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:38.916+0000 7f960485c700 1 -- 172.21.15.114:0/3898915371 wait complete. 2024-05-27T07:39:38.941 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:38 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:38.599+0000 7f8f5db26200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:39:38.941 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:38 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:38.771+0000 7f8f5db26200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:39:38.950 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:38 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:38.607+0000 7fde49c79200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:39:38.951 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:38 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:38.781+0000 7fde49c79200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:39:38.951 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:38 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:38.953+0000 7fde49c79200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:39:39.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:38 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3898915371' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:39:39.054 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:38 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:38.724+0000 7f55d556f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:39:39.054 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:38 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:38.906+0000 7f55d556f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:39:39.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:38 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3898915371' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:39:39.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:38 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3898915371' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:39:39.226 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:38 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:38.943+0000 7f8f5db26200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:39:39.389 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:39.085+0000 7f55d556f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:39:39.531 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:39 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:39.236+0000 7f8f5db26200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:39:39.531 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:39 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:39.328+0000 7f8f5db26200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:39:39.531 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:39 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:39.533+0000 7f8f5db26200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:39:39.547 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:39.248+0000 7fde49c79200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:39:39.547 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:39.343+0000 7fde49c79200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:39:39.547 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:39.550+0000 7fde49c79200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:39:39.624 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-27T07:39:39.624 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-27T07:39:39.624 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-05-27T07:39:39.624 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-27T07:39:39.624 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T07:39:39.624 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-27T07:39:39.625 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-27T07:39:39.625 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-27T07:39:39.625 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-27T07:39:39.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:39:39.625 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-05-27T07:39:39.641 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-27T07:39:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-27T07:39:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-05-27T07:39:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:39:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 41994sec preferred_lft 41994sec 2024-05-27T07:39:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link 2024-05-27T07:39:39.643 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:39:39.643 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-27T07:39:39.643 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:39:39.643 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-27T07:39:39.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add name ceph-brx type bridge 2024-05-27T07:39:39.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr flush dev ceph-brx 2024-05-27T07:39:39.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set ceph-brx up 2024-05-27T07:39:39.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-27T07:39:39.643 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-27T07:39:39.697 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:39.391+0000 7f55d556f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:39:39.697 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:39.486+0000 7f55d556f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:39:39.697 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:39.699+0000 7f55d556f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:39:39.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:39:39.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:39:39.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:39:39.786 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-27T07:39:39.853 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-05-27T07:39:39.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:39:39.855 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-05-27T07:39:39.913 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-05-27T07:39:39.913 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-27T07:39:39.913 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-27T07:39:39.913 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-27T07:39:39.913 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-27T07:39:39.914 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:39:39.914 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-27T07:39:39.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-27T07:39:39.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-27T07:39:39.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-27T07:39:39.914 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-27T07:39:39.973 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:39.748+0000 7fde49c79200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:39:39.975 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:39 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:39.731+0000 7f8f5db26200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:39:40.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:39:40.012 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:39.913+0000 7f55d556f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:39:40.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:39:40.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:39:40.044 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-27T07:39:40.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:39:40.059 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2024-05-27T07:39:40.115 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:39:40.115 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-27T07:39:40.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:39:40.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-27T07:39:40.116 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-27T07:39:40.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:39:40.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:39:40.234 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-27T07:39:40.234 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:39:40.235 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-27T07:39:40.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-27T07:39:40.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-27T07:39:40.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-27T07:39:40.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-27T07:39:40.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-27T07:39:40.235 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-27T07:39:40.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:39:40.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:39:40.350 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:39:40.350 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-27T07:39:40.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-05-27T07:39:40.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.0 master ceph-brx 2024-05-27T07:39:40.351 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-27T07:39:40.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:39:40.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:39:40.433 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:39:40.433 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-27T07:39:40.449 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-27T07:39:40.449 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:39:40.449 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-27T07:39:40.505 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-05-27T07:39:40.505 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-05-27T07:39:40.505 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/ce3cee47-a2d9-4102-a357-fb849184a794 2024-05-27T07:39:40.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:39:40.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/ce3cee47-a2d9-4102-a357-fb849184a794 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-27T07:39:40.599 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-27T07:39:40.599 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-05-27T07:39:40.599 INFO:teuthology.orchestra.run.smithi114.stdout:invalid new device string format 2024-05-27T07:39:40.723 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:40 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:40.354+0000 7fde49c79200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:39:40.723 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:40 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:40.579+0000 7fde49c79200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:39:40.726 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:40 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:40.334+0000 7f8f5db26200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:39:40.726 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:40 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:40.560+0000 7f8f5db26200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:39:40.745 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-27T07:39:40.746 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-05-27T07:39:40.746 INFO:teuthology.orchestra.run.smithi114.stdout:invalid new device string format 2024-05-27T07:39:40.746 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:6789,172.21.15.163:6789,172.21.15.186:6789" 2024-05-27T07:39:40.746 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.114:6789,172.21.15.163:6789,172.21.15.186:6789:/volumes/qa/sv_1/ce3cee47-a2d9-4102-a357-fb849184a794 2024-05-27T07:39:40.746 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync,key=0,fsid=755b3728-1bfa-11ef-bc9b-c7b262605968" will pass to kernel 2024-05-27T07:39:40.747 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-27T07:39:40.747 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-05-27T07:39:40.747 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-27T07:39:40.747 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-05-27T07:39:40.747 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-27T07:39:40.749 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-27T07:39:40.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:39:40.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-27T07:39:40.776 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-27T07:39:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T07:39:40.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:39:40.875 INFO:teuthology.orchestra.run.smithi114.stdout:client24856 2024-05-27T07:39:40.875 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:39:40.876 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:39:40.893 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-27T07:39:40.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T07:39:40.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:39:40.993 INFO:teuthology.orchestra.run.smithi114.stdout:client24856 2024-05-27T07:39:40.994 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-27T07:39:41.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T07:39:41.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:39:41.052 INFO:teuthology.orchestra.run.smithi114.stdout:client24856 2024-05-27T07:39:41.052 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968.client24856/status') 2024-05-27T07:39:41.053 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:40 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:40.558+0000 7f55d556f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:39:41.053 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:40 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:40.804+0000 7f55d556f200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:39:41.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:39:41.112 INFO:teuthology.orchestra.run.smithi114.stdout:instance: client.24856 (3)172.21.15.114:0/968719272 2024-05-27T07:39:41.112 INFO:teuthology.orchestra.run.smithi114.stdout:blocklisted: false 2024-05-27T07:39:41.112 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records in 2024-05-27T07:39:41.112 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records out 2024-05-27T07:39:41.112 INFO:teuthology.orchestra.run.smithi114.stderr:71 bytes copied, 2.3538e-05 s, 3.0 MB/s 2024-05-27T07:39:41.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:39:41.120 INFO:teuthology.run_tasks:Running task check-counter... 2024-05-27T07:39:41.133 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-05-27T07:39:41.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T07:39:41.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.266+0000 7ff73b6f2640 1 -- 172.21.15.114:0/3280969633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7341023d0 msgr2=0x7ff73410be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.266+0000 7ff73b6f2640 1 --2- 172.21.15.114:0/3280969633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7341023d0 0x7ff73410be30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009ca0 tx=0x7ff72c02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.266+0000 7ff73b6f2640 1 -- 172.21.15.114:0/3280969633 shutdown_connections 2024-05-27T07:39:41.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.266+0000 7ff73b6f2640 1 --2- 172.21.15.114:0/3280969633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7341023d0 0x7ff73410be30 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.266+0000 7ff73b6f2640 1 --2- 172.21.15.114:0/3280969633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7341018d0 0x7ff734101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.266+0000 7ff73b6f2640 1 --2- 172.21.15.114:0/3280969633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff734105220 0x7ff734105630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.266+0000 7ff73b6f2640 1 -- 172.21.15.114:0/3280969633 >> 172.21.15.114:0/3280969633 conn(0x7ff7340fd4d0 msgr2=0x7ff7340ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:41.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.266+0000 7ff73b6f2640 1 -- 172.21.15.114:0/3280969633 shutdown_connections 2024-05-27T07:39:41.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.266+0000 7ff73b6f2640 1 -- 172.21.15.114:0/3280969633 wait complete. 2024-05-27T07:39:41.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.267+0000 7ff73b6f2640 1 Processor -- start 2024-05-27T07:39:41.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.267+0000 7ff73b6f2640 1 -- start start 2024-05-27T07:39:41.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73b6f2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7341018d0 0x7ff73419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73b6f2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7341023d0 0x7ff73419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73b6f2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff734105220 0x7ff7341a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73b6f2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff73410f740 con 0x7ff7341018d0 2024-05-27T07:39:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73b6f2640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff73410f5c0 con 0x7ff7341023d0 2024-05-27T07:39:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73b6f2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff73410f8c0 con 0x7ff734105220 2024-05-27T07:39:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73948e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7341018d0 0x7ff73419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73948e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7341018d0 0x7ff73419a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43974/0 (socket says 172.21.15.114:43974) 2024-05-27T07:39:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73948e640 1 -- 172.21.15.114:0/2879977957 learned_addr learned my addr 172.21.15.114:0/2879977957 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff739c8f640 1 --2- 172.21.15.114:0/2879977957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff734105220 0x7ff7341a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff738c8d640 1 --2- 172.21.15.114:0/2879977957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7341023d0 0x7ff73419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73948e640 1 -- 172.21.15.114:0/2879977957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff734105220 msgr2=0x7ff7341a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73948e640 1 --2- 172.21.15.114:0/2879977957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff734105220 0x7ff7341a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73948e640 1 -- 172.21.15.114:0/2879977957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7341023d0 msgr2=0x7ff73419a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73948e640 1 --2- 172.21.15.114:0/2879977957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7341023d0 0x7ff73419a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73948e640 1 -- 172.21.15.114:0/2879977957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7341a1dd0 con 0x7ff7341018d0 2024-05-27T07:39:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73948e640 1 --2- 172.21.15.114:0/2879977957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7341018d0 0x7ff73419a290 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff730005d00 tx=0x7ff73000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff72a7fc640 1 -- 172.21.15.114:0/2879977957 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff730017070 con 0x7ff7341018d0 2024-05-27T07:39:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff72a7fc640 1 -- 172.21.15.114:0/2879977957 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff730009c30 con 0x7ff7341018d0 2024-05-27T07:39:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff73b6f2640 1 -- 172.21.15.114:0/2879977957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7341a20c0 con 0x7ff7341018d0 2024-05-27T07:39:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.268+0000 7ff72a7fc640 1 -- 172.21.15.114:0/2879977957 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73000f620 con 0x7ff7341018d0 2024-05-27T07:39:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.269+0000 7ff73b6f2640 1 -- 172.21.15.114:0/2879977957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7341a25c0 con 0x7ff7341018d0 2024-05-27T07:39:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.270+0000 7ff73b6f2640 1 -- 172.21.15.114:0/2879977957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff73410abc0 con 0x7ff7341018d0 2024-05-27T07:39:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.271+0000 7ff72a7fc640 1 -- 172.21.15.114:0/2879977957 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 149670+0+0 (secure 0 0 0) 0x7ff73001a070 con 0x7ff7341018d0 2024-05-27T07:39:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.272+0000 7ff72a7fc640 1 --2- 172.21.15.114:0/2879977957 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff7200b2840 0x7ff7200b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.272+0000 7ff738c8d640 1 -- 172.21.15.114:0/2879977957 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff7200b2840 msgr2=0x7ff7200b4d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2291864634 2024-05-27T07:39:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.272+0000 7ff72a7fc640 1 -- 172.21.15.114:0/2879977957 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7ff7300e1ff0 con 0x7ff7341018d0 2024-05-27T07:39:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.272+0000 7ff738c8d640 1 --2- 172.21.15.114:0/2879977957 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff7200b2840 0x7ff7200b4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:39:41.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.276+0000 7ff72a7fc640 1 -- 172.21.15.114:0/2879977957 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff730009de0 con 0x7ff7341018d0 2024-05-27T07:39:41.381 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:41 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:41.278+0000 7f55d556f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:39:41.382 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:41 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:41.384+0000 7f55d556f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:39:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.464+0000 7ff73b6f2640 1 -- 172.21.15.114:0/2879977957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7340638c0 con 0x7ff7341018d0 2024-05-27T07:39:41.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.466+0000 7ff72a7fc640 1 -- 172.21.15.114:0/2879977957 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24923 (secure 0 0 0) 0x7ff7300a92f0 con 0x7ff7341018d0 2024-05-27T07:39:41.464 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:41.465 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":89,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T07:37:46.129371+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:39:41.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.468+0000 7ff73b6f2640 1 -- 172.21.15.114:0/2879977957 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff7200b2840 msgr2=0x7ff7200b4d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:39:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.468+0000 7ff73b6f2640 1 --2- 172.21.15.114:0/2879977957 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff7200b2840 0x7ff7200b4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.468+0000 7ff73b6f2640 1 -- 172.21.15.114:0/2879977957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7341018d0 msgr2=0x7ff73419a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.468+0000 7ff73b6f2640 1 --2- 172.21.15.114:0/2879977957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7341018d0 0x7ff73419a290 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff730005d00 tx=0x7ff73000dde0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.468+0000 7ff73b6f2640 1 -- 172.21.15.114:0/2879977957 shutdown_connections 2024-05-27T07:39:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.468+0000 7ff73b6f2640 1 --2- 172.21.15.114:0/2879977957 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff7200b2840 0x7ff7200b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.468+0000 7ff73b6f2640 1 --2- 172.21.15.114:0/2879977957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff734105220 0x7ff7341a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.468+0000 7ff73b6f2640 1 --2- 172.21.15.114:0/2879977957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7341023d0 0x7ff73419a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.468+0000 7ff73b6f2640 1 --2- 172.21.15.114:0/2879977957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7341018d0 0x7ff73419a290 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.468+0000 7ff73b6f2640 1 -- 172.21.15.114:0/2879977957 >> 172.21.15.114:0/2879977957 conn(0x7ff7340fd4d0 msgr2=0x7ff734107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.468+0000 7ff73b6f2640 1 -- 172.21.15.114:0/2879977957 shutdown_connections 2024-05-27T07:39:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.468+0000 7ff73b6f2640 1 -- 172.21.15.114:0/2879977957 wait complete. 2024-05-27T07:39:41.468 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:40 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:40.990+0000 7f8f5db26200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:39:41.468 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:41.090+0000 7f8f5db26200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:39:41.472 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:41 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:41.013+0000 7fde49c79200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:39:41.473 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:41 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:41.112+0000 7fde49c79200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:39:41.479 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T07:31:09.365695+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T07:35:59.962762+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T07:36:03.708207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-27T07:36:13.010229+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-27T07:39:41.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T07:39:41.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.601+0000 7f92640e0640 1 -- 172.21.15.114:0/1753272891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c103d50 msgr2=0x7f925c104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.601+0000 7f92640e0640 1 --2- 172.21.15.114:0/1753272891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c103d50 0x7f925c104160 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9250009900 tx=0x7f925002a410 comp rx=0 tx=0).stop 2024-05-27T07:39:41.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.601+0000 7f92640e0640 1 -- 172.21.15.114:0/1753272891 shutdown_connections 2024-05-27T07:39:41.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.601+0000 7f92640e0640 1 --2- 172.21.15.114:0/1753272891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f925c108a50 0x7f925c10ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.601+0000 7f92640e0640 1 --2- 172.21.15.114:0/1753272891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c103d50 0x7f925c104160 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.601+0000 7f92640e0640 1 --2- 172.21.15.114:0/1753272891 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f925c102b10 0x7f925c103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.601+0000 7f92640e0640 1 -- 172.21.15.114:0/1753272891 >> 172.21.15.114:0/1753272891 conn(0x7f925c0fdd00 msgr2=0x7f925c1001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:41.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.601+0000 7f92640e0640 1 -- 172.21.15.114:0/1753272891 shutdown_connections 2024-05-27T07:39:41.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.601+0000 7f92640e0640 1 -- 172.21.15.114:0/1753272891 wait complete. 2024-05-27T07:39:41.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f92640e0640 1 Processor -- start 2024-05-27T07:39:41.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f92640e0640 1 -- start start 2024-05-27T07:39:41.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f92640e0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f925c102b10 0x7f925c19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f92640e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f925c103d50 0x7f925c19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f92640e0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c108a50 0x7f925c1a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f92640e0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f925c10f740 con 0x7f925c108a50 2024-05-27T07:39:41.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f92640e0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f925c10f5c0 con 0x7f925c102b10 2024-05-27T07:39:41.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f92640e0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f925c10f8c0 con 0x7f925c103d50 2024-05-27T07:39:41.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f926267d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c108a50 0x7f925c1a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f926267d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c108a50 0x7f925c1a1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43988/0 (socket says 172.21.15.114:43988) 2024-05-27T07:39:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f926267d640 1 -- 172.21.15.114:0/308849286 learned_addr learned my addr 172.21.15.114:0/308849286 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f926167b640 1 --2- 172.21.15.114:0/308849286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f925c103d50 0x7f925c19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f9261e7c640 1 --2- 172.21.15.114:0/308849286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f925c102b10 0x7f925c19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f926267d640 1 -- 172.21.15.114:0/308849286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f925c103d50 msgr2=0x7f925c19a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f926267d640 1 --2- 172.21.15.114:0/308849286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f925c103d50 0x7f925c19a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f926267d640 1 -- 172.21.15.114:0/308849286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f925c102b10 msgr2=0x7f925c19a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f926267d640 1 --2- 172.21.15.114:0/308849286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f925c102b10 0x7f925c19a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.602+0000 7f926267d640 1 -- 172.21.15.114:0/308849286 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925c1a1ef0 con 0x7f925c108a50 2024-05-27T07:39:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.603+0000 7f926267d640 1 --2- 172.21.15.114:0/308849286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c108a50 0x7f925c1a1880 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f925400eb10 tx=0x7f925400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.603+0000 7f924effd640 1 -- 172.21.15.114:0/308849286 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9254004b20 con 0x7f925c108a50 2024-05-27T07:39:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.603+0000 7f924effd640 1 -- 172.21.15.114:0/308849286 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9254004cc0 con 0x7f925c108a50 2024-05-27T07:39:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.603+0000 7f92640e0640 1 -- 172.21.15.114:0/308849286 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f925c1a21e0 con 0x7f925c108a50 2024-05-27T07:39:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.603+0000 7f924effd640 1 -- 172.21.15.114:0/308849286 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9254010670 con 0x7f925c108a50 2024-05-27T07:39:41.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.603+0000 7f92640e0640 1 -- 172.21.15.114:0/308849286 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f925c1a2730 con 0x7f925c108a50 2024-05-27T07:39:41.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.604+0000 7f92640e0640 1 -- 172.21.15.114:0/308849286 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f925c107860 con 0x7f925c108a50 2024-05-27T07:39:41.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.606+0000 7f924effd640 1 -- 172.21.15.114:0/308849286 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 149670+0+0 (secure 0 0 0) 0x7f9254005000 con 0x7f925c108a50 2024-05-27T07:39:41.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.607+0000 7f924effd640 1 --2- 172.21.15.114:0/308849286 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f92480b2840 0x7f92480b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.607+0000 7f9261e7c640 1 -- 172.21.15.114:0/308849286 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f92480b2840 msgr2=0x7f92480b4d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2291864634 2024-05-27T07:39:41.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.607+0000 7f924effd640 1 -- 172.21.15.114:0/308849286 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f92540e2da0 con 0x7f925c108a50 2024-05-27T07:39:41.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.607+0000 7f9261e7c640 1 --2- 172.21.15.114:0/308849286 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f92480b2840 0x7f92480b4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:39:41.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.611+0000 7f924effd640 1 -- 172.21.15.114:0/308849286 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9254054df0 con 0x7f925c108a50 2024-05-27T07:39:41.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:41 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2879977957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:39:41.726 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:41.468+0000 7f8f5db26200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:39:41.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:41 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2879977957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:39:41.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.798+0000 7f92640e0640 1 -- 172.21.15.114:0/308849286 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f925c02ddb0 con 0x7f925c108a50 2024-05-27T07:39:41.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.799+0000 7f924effd640 1 -- 172.21.15.114:0/308849286 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f92540aa6f0 con 0x7f925c108a50 2024-05-27T07:39:41.796 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-05-27T07:39:41.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.801+0000 7f92640e0640 1 -- 172.21.15.114:0/308849286 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f92480b2840 msgr2=0x7f92480b4d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:39:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.801+0000 7f92640e0640 1 --2- 172.21.15.114:0/308849286 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f92480b2840 0x7f92480b4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.801+0000 7f92640e0640 1 -- 172.21.15.114:0/308849286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c108a50 msgr2=0x7f925c1a1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.801+0000 7f92640e0640 1 --2- 172.21.15.114:0/308849286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c108a50 0x7f925c1a1880 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f925400eb10 tx=0x7f925400c2d0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.801+0000 7f92640e0640 1 -- 172.21.15.114:0/308849286 shutdown_connections 2024-05-27T07:39:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.801+0000 7f92640e0640 1 --2- 172.21.15.114:0/308849286 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f92480b2840 0x7f92480b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.801+0000 7f92640e0640 1 --2- 172.21.15.114:0/308849286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c108a50 0x7f925c1a1880 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.801+0000 7f92640e0640 1 --2- 172.21.15.114:0/308849286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f925c103d50 0x7f925c19a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.801+0000 7f92640e0640 1 --2- 172.21.15.114:0/308849286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f925c102b10 0x7f925c19a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.801+0000 7f92640e0640 1 -- 172.21.15.114:0/308849286 >> 172.21.15.114:0/308849286 conn(0x7f925c0fdd00 msgr2=0x7f925c104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:41.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.801+0000 7f92640e0640 1 -- 172.21.15.114:0/308849286 shutdown_connections 2024-05-27T07:39:41.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.801+0000 7f92640e0640 1 -- 172.21.15.114:0/308849286 wait complete. 2024-05-27T07:39:41.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T07:39:41.845 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:41 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2879977957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:39:41.845 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:41 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:41.490+0000 7fde49c79200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:39:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.929+0000 7ff1b21a8640 1 -- 172.21.15.114:0/3887235527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1ac108a40 msgr2=0x7ff1ac10ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.929+0000 7ff1b21a8640 1 --2- 172.21.15.114:0/3887235527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1ac108a40 0x7ff1ac10ae60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009ca0 tx=0x7ff1a402a8c0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.929+0000 7ff1b21a8640 1 -- 172.21.15.114:0/3887235527 shutdown_connections 2024-05-27T07:39:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.929+0000 7ff1b21a8640 1 --2- 172.21.15.114:0/3887235527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1ac108a40 0x7ff1ac10ae60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.929+0000 7ff1b21a8640 1 --2- 172.21.15.114:0/3887235527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1ac103d40 0x7ff1ac104150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.929+0000 7ff1b21a8640 1 --2- 172.21.15.114:0/3887235527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac102b00 0x7ff1ac103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.929+0000 7ff1b21a8640 1 -- 172.21.15.114:0/3887235527 >> 172.21.15.114:0/3887235527 conn(0x7ff1ac0fdcf0 msgr2=0x7ff1ac1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.929+0000 7ff1b21a8640 1 -- 172.21.15.114:0/3887235527 shutdown_connections 2024-05-27T07:39:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.929+0000 7ff1b21a8640 1 -- 172.21.15.114:0/3887235527 wait complete. 2024-05-27T07:39:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.930+0000 7ff1b21a8640 1 Processor -- start 2024-05-27T07:39:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.930+0000 7ff1b21a8640 1 -- start start 2024-05-27T07:39:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.930+0000 7ff1b21a8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac102b00 0x7ff1ac10d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.930+0000 7ff1b21a8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1ac103d40 0x7ff1ac10ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.930+0000 7ff1b21a8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1ac108a40 0x7ff1ac10bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.930+0000 7ff1b21a8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac10f730 con 0x7ff1ac103d40 2024-05-27T07:39:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.930+0000 7ff1b21a8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac10f5b0 con 0x7ff1ac108a40 2024-05-27T07:39:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.930+0000 7ff1b21a8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac10f8b0 con 0x7ff1ac102b00 2024-05-27T07:39:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.930+0000 7ff1a3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1ac103d40 0x7ff1ac10ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.930+0000 7ff1a3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1ac103d40 0x7ff1ac10ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44006/0 (socket says 172.21.15.114:44006) 2024-05-27T07:39:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.930+0000 7ff1a3fff640 1 -- 172.21.15.114:0/1106289603 learned_addr learned my addr 172.21.15.114:0/1106289603 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.930+0000 7ff1ab7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac102b00 0x7ff1ac10d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.930+0000 7ff1abfff640 1 --2- 172.21.15.114:0/1106289603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1ac108a40 0x7ff1ac10bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.931+0000 7ff1a3fff640 1 -- 172.21.15.114:0/1106289603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac102b00 msgr2=0x7ff1ac10d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.931+0000 7ff1a3fff640 1 --2- 172.21.15.114:0/1106289603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac102b00 0x7ff1ac10d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.931+0000 7ff1a3fff640 1 -- 172.21.15.114:0/1106289603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1ac108a40 msgr2=0x7ff1ac10bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.931+0000 7ff1a3fff640 1 --2- 172.21.15.114:0/1106289603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1ac108a40 0x7ff1ac10bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.931+0000 7ff1a3fff640 1 -- 172.21.15.114:0/1106289603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ac10c6e0 con 0x7ff1ac103d40 2024-05-27T07:39:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.931+0000 7ff1a3fff640 1 --2- 172.21.15.114:0/1106289603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1ac103d40 0x7ff1ac10ba70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff19400d950 tx=0x7ff19400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.931+0000 7ff1a97fa640 1 -- 172.21.15.114:0/1106289603 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff194017030 con 0x7ff1ac103d40 2024-05-27T07:39:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.931+0000 7ff1b21a8640 1 -- 172.21.15.114:0/1106289603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ac1a4670 con 0x7ff1ac103d40 2024-05-27T07:39:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.931+0000 7ff1a97fa640 1 -- 172.21.15.114:0/1106289603 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff194004020 con 0x7ff1ac103d40 2024-05-27T07:39:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.931+0000 7ff1a97fa640 1 -- 172.21.15.114:0/1106289603 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1940047a0 con 0x7ff1ac103d40 2024-05-27T07:39:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.931+0000 7ff1b21a8640 1 -- 172.21.15.114:0/1106289603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ac1a4b60 con 0x7ff1ac103d40 2024-05-27T07:39:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.932+0000 7ff1b21a8640 1 -- 172.21.15.114:0/1106289603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1ac107850 con 0x7ff1ac103d40 2024-05-27T07:39:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.934+0000 7ff1a97fa640 1 -- 172.21.15.114:0/1106289603 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 149670+0+0 (secure 0 0 0) 0x7ff194009a30 con 0x7ff1ac103d40 2024-05-27T07:39:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.935+0000 7ff1a97fa640 1 --2- 172.21.15.114:0/1106289603 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff1980b2840 0x7ff1980b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.936+0000 7ff1ab7fe640 1 -- 172.21.15.114:0/1106289603 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff1980b2840 msgr2=0x7ff1980b4d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2291864634 2024-05-27T07:39:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.936+0000 7ff1a97fa640 1 -- 172.21.15.114:0/1106289603 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7ff1940e1f20 con 0x7ff1ac103d40 2024-05-27T07:39:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.936+0000 7ff1ab7fe640 1 --2- 172.21.15.114:0/1106289603 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff1980b2840 0x7ff1980b4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:39:41.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:41.940+0000 7ff1a97fa640 1 -- 172.21.15.114:0/1106289603 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff1940a9870 con 0x7ff1ac103d40 2024-05-27T07:39:42.053 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:41 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:41.776+0000 7f55d556f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:39:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.127+0000 7ff1b21a8640 1 -- 172.21.15.114:0/1106289603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff1ac02ddb0 con 0x7ff1ac103d40 2024-05-27T07:39:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.128+0000 7ff1a97fa640 1 -- 172.21.15.114:0/1106289603 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7ff1940a9170 con 0x7ff1ac103d40 2024-05-27T07:39:42.126 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-05-27T07:39:42.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.130+0000 7ff1b21a8640 1 -- 172.21.15.114:0/1106289603 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff1980b2840 msgr2=0x7ff1980b4d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:39:42.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.130+0000 7ff1b21a8640 1 --2- 172.21.15.114:0/1106289603 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff1980b2840 0x7ff1980b4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.130+0000 7ff1b21a8640 1 -- 172.21.15.114:0/1106289603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1ac103d40 msgr2=0x7ff1ac10ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:42.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.130+0000 7ff1b21a8640 1 --2- 172.21.15.114:0/1106289603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1ac103d40 0x7ff1ac10ba70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff19400d950 tx=0x7ff19400de10 comp rx=0 tx=0).stop 2024-05-27T07:39:42.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.130+0000 7ff1b21a8640 1 -- 172.21.15.114:0/1106289603 shutdown_connections 2024-05-27T07:39:42.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.130+0000 7ff1b21a8640 1 --2- 172.21.15.114:0/1106289603 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7ff1980b2840 0x7ff1980b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.130+0000 7ff1b21a8640 1 --2- 172.21.15.114:0/1106289603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1ac108a40 0x7ff1ac10bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.130+0000 7ff1b21a8640 1 --2- 172.21.15.114:0/1106289603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1ac103d40 0x7ff1ac10ba70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.130+0000 7ff1b21a8640 1 --2- 172.21.15.114:0/1106289603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac102b00 0x7ff1ac10d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.130+0000 7ff1b21a8640 1 -- 172.21.15.114:0/1106289603 >> 172.21.15.114:0/1106289603 conn(0x7ff1ac0fdcf0 msgr2=0x7ff1ac104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:42.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.130+0000 7ff1b21a8640 1 -- 172.21.15.114:0/1106289603 shutdown_connections 2024-05-27T07:39:42.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.130+0000 7ff1b21a8640 1 -- 172.21.15.114:0/1106289603 wait complete. 2024-05-27T07:39:42.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T07:39:42.169 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:41.827+0000 7f8f5db26200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:39:42.169 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:41.923+0000 7f8f5db26200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:39:42.188 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:41 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:41.848+0000 7fde49c79200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:39:42.188 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:41 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:41.944+0000 7fde49c79200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:39:42.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.258+0000 7f98b09ce640 1 -- 172.21.15.114:0/548456163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98ac1030e0 msgr2=0x7f98ac1034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:42.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.258+0000 7f98b09ce640 1 --2- 172.21.15.114:0/548456163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98ac1030e0 0x7f98ac1034f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f989c009900 tx=0x7f989c02a410 comp rx=0 tx=0).stop 2024-05-27T07:39:42.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.258+0000 7f98b09ce640 1 -- 172.21.15.114:0/548456163 shutdown_connections 2024-05-27T07:39:42.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.258+0000 7f98b09ce640 1 --2- 172.21.15.114:0/548456163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98ac1047a0 0x7f98ac10aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.258+0000 7f98b09ce640 1 --2- 172.21.15.114:0/548456163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98ac103c40 0x7f98ac104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.258+0000 7f98b09ce640 1 --2- 172.21.15.114:0/548456163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98ac1030e0 0x7f98ac1034f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.258+0000 7f98b09ce640 1 -- 172.21.15.114:0/548456163 >> 172.21.15.114:0/548456163 conn(0x7f98ac0fd460 msgr2=0x7f98ac0ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:42.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.258+0000 7f98b09ce640 1 -- 172.21.15.114:0/548456163 shutdown_connections 2024-05-27T07:39:42.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.258+0000 7f98b09ce640 1 -- 172.21.15.114:0/548456163 wait complete. 2024-05-27T07:39:42.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.259+0000 7f98b09ce640 1 Processor -- start 2024-05-27T07:39:42.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.259+0000 7f98b09ce640 1 -- start start 2024-05-27T07:39:42.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.259+0000 7f98b09ce640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98ac1030e0 0x7f98ac19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:42.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.259+0000 7f98b09ce640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98ac103c40 0x7f98ac19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:42.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.259+0000 7f98b09ce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98ac19f620 0x7f98ac1a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:42.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.259+0000 7f98b09ce640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac10e820 con 0x7f98ac19f620 2024-05-27T07:39:42.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.259+0000 7f98b09ce640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac10e6a0 con 0x7f98ac103c40 2024-05-27T07:39:42.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.259+0000 7f98b09ce640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac10e9a0 con 0x7f98ac1030e0 2024-05-27T07:39:42.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.259+0000 7f98aad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98ac19f620 0x7f98ac1a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:42.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.259+0000 7f98aa59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98ac1030e0 0x7f98ac19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:42.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98aad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98ac19f620 0x7f98ac1a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44016/0 (socket says 172.21.15.114:44016) 2024-05-27T07:39:42.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98aa59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98ac1030e0 0x7f98ac19a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:38304/0 (socket says 172.21.15.114:38304) 2024-05-27T07:39:42.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98a3fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98ac103c40 0x7f98ac19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:42.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98aad9d640 1 -- 172.21.15.114:0/2887178082 learned_addr learned my addr 172.21.15.114:0/2887178082 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:42.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98aad9d640 1 -- 172.21.15.114:0/2887178082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98ac1030e0 msgr2=0x7f98ac19a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:42.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98aad9d640 1 --2- 172.21.15.114:0/2887178082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98ac1030e0 0x7f98ac19a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98aad9d640 1 -- 172.21.15.114:0/2887178082 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98ac103c40 msgr2=0x7f98ac19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:42.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98aad9d640 1 --2- 172.21.15.114:0/2887178082 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98ac103c40 0x7f98ac19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98aad9d640 1 -- 172.21.15.114:0/2887178082 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98ac1a20b0 con 0x7f98ac19f620 2024-05-27T07:39:42.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98aad9d640 1 --2- 172.21.15.114:0/2887178082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98ac19f620 0x7f98ac1a1a40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f98a400eb10 tx=0x7f98a400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:42.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98a37fe640 1 -- 172.21.15.114:0/2887178082 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a4004b20 con 0x7f98ac19f620 2024-05-27T07:39:42.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98a37fe640 1 -- 172.21.15.114:0/2887178082 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a4004cc0 con 0x7f98ac19f620 2024-05-27T07:39:42.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98b09ce640 1 -- 172.21.15.114:0/2887178082 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98ac1a23a0 con 0x7f98ac19f620 2024-05-27T07:39:42.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98a37fe640 1 -- 172.21.15.114:0/2887178082 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a4010670 con 0x7f98ac19f620 2024-05-27T07:39:42.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.260+0000 7f98b09ce640 1 -- 172.21.15.114:0/2887178082 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98ac1a28a0 con 0x7f98ac19f620 2024-05-27T07:39:42.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.261+0000 7f98b09ce640 1 -- 172.21.15.114:0/2887178082 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98ac069d90 con 0x7f98ac19f620 2024-05-27T07:39:42.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.263+0000 7f98a37fe640 1 -- 172.21.15.114:0/2887178082 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 149670+0+0 (secure 0 0 0) 0x7f98a4005000 con 0x7f98ac19f620 2024-05-27T07:39:42.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.264+0000 7f98a37fe640 1 --2- 172.21.15.114:0/2887178082 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f98980b2840 0x7f98980b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:42.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.264+0000 7f98aa59c640 1 -- 172.21.15.114:0/2887178082 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f98980b2840 msgr2=0x7f98980b4d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2291864634 2024-05-27T07:39:42.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.264+0000 7f98a37fe640 1 -- 172.21.15.114:0/2887178082 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f98a40e2d90 con 0x7f98ac19f620 2024-05-27T07:39:42.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.264+0000 7f98aa59c640 1 --2- 172.21.15.114:0/2887178082 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f98980b2840 0x7f98980b4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:39:42.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.271+0000 7f98a37fe640 1 -- 172.21.15.114:0/2887178082 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f98a4054df0 con 0x7f98ac19f620 2024-05-27T07:39:42.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.458+0000 7f98b09ce640 1 -- 172.21.15.114:0/2887178082 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f98ac02de70 con 0x7f98ac19f620 2024-05-27T07:39:42.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.459+0000 7f98a37fe640 1 -- 172.21.15.114:0/2887178082 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f98a40aa6e0 con 0x7f98ac19f620 2024-05-27T07:39:42.457 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-05-27T07:39:42.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.461+0000 7f98b09ce640 1 -- 172.21.15.114:0/2887178082 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f98980b2840 msgr2=0x7f98980b4d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:39:42.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.461+0000 7f98b09ce640 1 --2- 172.21.15.114:0/2887178082 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f98980b2840 0x7f98980b4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.461+0000 7f98b09ce640 1 -- 172.21.15.114:0/2887178082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98ac19f620 msgr2=0x7f98ac1a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:42.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.461+0000 7f98b09ce640 1 --2- 172.21.15.114:0/2887178082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98ac19f620 0x7f98ac1a1a40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f98a400eb10 tx=0x7f98a400c2d0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.461+0000 7f98b09ce640 1 -- 172.21.15.114:0/2887178082 shutdown_connections 2024-05-27T07:39:42.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.461+0000 7f98b09ce640 1 --2- 172.21.15.114:0/2887178082 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f98980b2840 0x7f98980b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.461+0000 7f98b09ce640 1 --2- 172.21.15.114:0/2887178082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98ac19f620 0x7f98ac1a1a40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.461+0000 7f98b09ce640 1 --2- 172.21.15.114:0/2887178082 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98ac103c40 0x7f98ac19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.461+0000 7f98b09ce640 1 --2- 172.21.15.114:0/2887178082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98ac1030e0 0x7f98ac19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.461+0000 7f98b09ce640 1 -- 172.21.15.114:0/2887178082 >> 172.21.15.114:0/2887178082 conn(0x7f98ac0fd460 msgr2=0x7f98ac101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:42.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.461+0000 7f98b09ce640 1 -- 172.21.15.114:0/2887178082 shutdown_connections 2024-05-27T07:39:42.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.461+0000 7f98b09ce640 1 -- 172.21.15.114:0/2887178082 wait complete. 2024-05-27T07:39:42.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:42 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:42.163+0000 7f55d556f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:39:42.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:42 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:42.273+0000 7f55d556f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:39:42.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-27T07:39:42.473 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:42.191+0000 7fde49c79200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:39:42.473 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:42.327+0000 7fde49c79200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:39:42.473 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:42.424+0000 7fde49c79200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:39:42.475 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:42 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:42.171+0000 7f8f5db26200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:39:42.476 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:42 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:42.306+0000 7f8f5db26200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:39:42.476 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:42 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:42.403+0000 7f8f5db26200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:39:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.588+0000 7f25602f9640 1 -- 172.21.15.114:0/993582328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2558103a60 msgr2=0x7f2558103f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.588+0000 7f25602f9640 1 --2- 172.21.15.114:0/993582328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2558103a60 0x7f2558103f10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2548009900 tx=0x7f254802a410 comp rx=0 tx=0).stop 2024-05-27T07:39:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.588+0000 7f25602f9640 1 -- 172.21.15.114:0/993582328 shutdown_connections 2024-05-27T07:39:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.588+0000 7f25602f9640 1 --2- 172.21.15.114:0/993582328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25581045a0 0x7f255810ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.588+0000 7f25602f9640 1 --2- 172.21.15.114:0/993582328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2558103a60 0x7f2558103f10 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.588+0000 7f25602f9640 1 --2- 172.21.15.114:0/993582328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2558103080 0x7f2558103490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.588+0000 7f25602f9640 1 -- 172.21.15.114:0/993582328 >> 172.21.15.114:0/993582328 conn(0x7f25580fd4a0 msgr2=0x7f25580ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:42.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.588+0000 7f25602f9640 1 -- 172.21.15.114:0/993582328 shutdown_connections 2024-05-27T07:39:42.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.588+0000 7f25602f9640 1 -- 172.21.15.114:0/993582328 wait complete. 2024-05-27T07:39:42.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.589+0000 7f25602f9640 1 Processor -- start 2024-05-27T07:39:42.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.589+0000 7f25602f9640 1 -- start start 2024-05-27T07:39:42.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.589+0000 7f25602f9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2558103080 0x7f255819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:42.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.589+0000 7f25602f9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2558103a60 0x7f255819a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:42.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.589+0000 7f25602f9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25581045a0 0x7f25581a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.589+0000 7f25602f9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f255810e670 con 0x7f25581045a0 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.589+0000 7f25602f9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f255810e4f0 con 0x7f2558103080 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.589+0000 7f25602f9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f255810e7f0 con 0x7f2558103a60 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.589+0000 7f255e896640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25581045a0 0x7f25581a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.589+0000 7f255e095640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2558103080 0x7f255819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.589+0000 7f255e896640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25581045a0 0x7f25581a1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44024/0 (socket says 172.21.15.114:44024) 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.589+0000 7f255e896640 1 -- 172.21.15.114:0/3359782277 learned_addr learned my addr 172.21.15.114:0/3359782277 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.589+0000 7f255d894640 1 --2- 172.21.15.114:0/3359782277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2558103a60 0x7f255819a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.590+0000 7f255e896640 1 -- 172.21.15.114:0/3359782277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2558103a60 msgr2=0x7f255819a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.590+0000 7f255e896640 1 --2- 172.21.15.114:0/3359782277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2558103a60 0x7f255819a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.590+0000 7f255e896640 1 -- 172.21.15.114:0/3359782277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2558103080 msgr2=0x7f255819a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.590+0000 7f255e896640 1 --2- 172.21.15.114:0/3359782277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2558103080 0x7f255819a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.590+0000 7f255e896640 1 -- 172.21.15.114:0/3359782277 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25581a1ed0 con 0x7f25581045a0 2024-05-27T07:39:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.590+0000 7f255e896640 1 --2- 172.21.15.114:0/3359782277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25581045a0 0x7f25581a1860 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f255000eaa0 tx=0x7f255000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:42.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.590+0000 7f254f7fe640 1 -- 172.21.15.114:0/3359782277 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255000cdb0 con 0x7f25581045a0 2024-05-27T07:39:42.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.590+0000 7f254f7fe640 1 -- 172.21.15.114:0/3359782277 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2550005020 con 0x7f25581045a0 2024-05-27T07:39:42.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.590+0000 7f254f7fe640 1 -- 172.21.15.114:0/3359782277 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25500116d0 con 0x7f25581045a0 2024-05-27T07:39:42.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.590+0000 7f25602f9640 1 -- 172.21.15.114:0/3359782277 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25581a21c0 con 0x7f25581045a0 2024-05-27T07:39:42.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.590+0000 7f25602f9640 1 -- 172.21.15.114:0/3359782277 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25581a26c0 con 0x7f25581045a0 2024-05-27T07:39:42.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.591+0000 7f25602f9640 1 -- 172.21.15.114:0/3359782277 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25581a2980 con 0x7f25581045a0 2024-05-27T07:39:42.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.593+0000 7f254f7fe640 1 -- 172.21.15.114:0/3359782277 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 149670+0+0 (secure 0 0 0) 0x7f25500051d0 con 0x7f25581045a0 2024-05-27T07:39:42.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.594+0000 7f254f7fe640 1 --2- 172.21.15.114:0/3359782277 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f25440b2840 0x7f25440b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:42.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.594+0000 7f255e095640 1 -- 172.21.15.114:0/3359782277 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f25440b2840 msgr2=0x7f25440b4d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2291864634 2024-05-27T07:39:42.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.594+0000 7f254f7fe640 1 -- 172.21.15.114:0/3359782277 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f25500e2b50 con 0x7f25581045a0 2024-05-27T07:39:42.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.594+0000 7f255e095640 1 --2- 172.21.15.114:0/3359782277 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f25440b2840 0x7f25440b4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:39:42.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.601+0000 7f254f7fe640 1 -- 172.21.15.114:0/3359782277 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2550016030 con 0x7f25581045a0 2024-05-27T07:39:42.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.788+0000 7f25602f9640 1 -- 172.21.15.114:0/3359782277 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f25581a2b50 con 0x7f25581045a0 2024-05-27T07:39:42.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.788+0000 7f254f7fe640 1 -- 172.21.15.114:0/3359782277 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f25500a9e50 con 0x7f25581045a0 2024-05-27T07:39:42.786 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-05-27T07:39:42.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.790+0000 7f25602f9640 1 -- 172.21.15.114:0/3359782277 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f25440b2840 msgr2=0x7f25440b4d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:39:42.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.790+0000 7f25602f9640 1 --2- 172.21.15.114:0/3359782277 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f25440b2840 0x7f25440b4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.790+0000 7f25602f9640 1 -- 172.21.15.114:0/3359782277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25581045a0 msgr2=0x7f25581a1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:42.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.790+0000 7f25602f9640 1 --2- 172.21.15.114:0/3359782277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25581045a0 0x7f25581a1860 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f255000eaa0 tx=0x7f255000ef60 comp rx=0 tx=0).stop 2024-05-27T07:39:42.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.790+0000 7f25602f9640 1 -- 172.21.15.114:0/3359782277 shutdown_connections 2024-05-27T07:39:42.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.790+0000 7f25602f9640 1 --2- 172.21.15.114:0/3359782277 >> [v2:172.21.15.114:6800/2291864634,v1:172.21.15.114:6801/2291864634] conn(0x7f25440b2840 0x7f25440b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.790+0000 7f25602f9640 1 --2- 172.21.15.114:0/3359782277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25581045a0 0x7f25581a1860 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.790+0000 7f25602f9640 1 --2- 172.21.15.114:0/3359782277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2558103a60 0x7f255819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.790+0000 7f25602f9640 1 --2- 172.21.15.114:0/3359782277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2558103080 0x7f255819a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:42.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.790+0000 7f25602f9640 1 -- 172.21.15.114:0/3359782277 >> 172.21.15.114:0/3359782277 conn(0x7f25580fd4a0 msgr2=0x7f2558109e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:42.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.790+0000 7f25602f9640 1 -- 172.21.15.114:0/3359782277 shutdown_connections 2024-05-27T07:39:42.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:42.790+0000 7f25602f9640 1 -- 172.21.15.114:0/3359782277 wait complete. 2024-05-27T07:39:42.790 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:42 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/308849286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:39:42.790 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:42 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1106289603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:39:42.790 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:42 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2887178082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:39:42.791 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:42 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:42.539+0000 7f55d556f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:39:42.791 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:42 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:42.688+0000 7f55d556f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:39:42.799 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-05-27T07:39:42.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:39:42.828 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:42 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:42.735+0000 7f8f5db26200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:39:42.828 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:42 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/308849286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:39:42.828 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:42 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1106289603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:39:42.828 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:42 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2887178082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:39:42.849 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:42 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/308849286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:39:42.849 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:42 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1106289603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:39:42.849 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:42 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2887178082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:39:42.849 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:42.757+0000 7fde49c79200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:39:42.850 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:42.852+0000 7fde49c79200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:39:43.053 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:42 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:42.792+0000 7f55d556f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:39:43.063 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:43.225 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:42 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:42.830+0000 7f8f5db26200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:39:43.425 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:43 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:43.154+0000 7f55d556f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:39:43.425 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:43 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:43.254+0000 7f55d556f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:39:43.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:43 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3359782277' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:39:43.723 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:43 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:43.459+0000 7fde49c79200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:39:43.723 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 07:39:43 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y[37818]: 2024-05-27T07:39:43.676+0000 7fde49c79200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:39:43.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:43 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3359782277' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:39:43.726 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:43 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:43.437+0000 7f8f5db26200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:39:43.726 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 07:39:43 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z[38496]: 2024-05-27T07:39:43.656+0000 7f8f5db26200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:39:43.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:43 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3359782277' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:39:44.180 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:43 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:43.891+0000 7f55d556f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:39:44.180 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:44.123+0000 7f55d556f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:39:44.505 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:44.412+0000 7f5533338700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:39:44.505 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:44.412+0000 7f5533338700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:39:44.505 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:44.412+0000 7f5533338700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:39:44.505 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:44.412+0000 7f5533338700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:39:44.505 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:44.412+0000 7f5533338700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:39:44.506 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:44.412+0000 7f553637e700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:39:44.506 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:44.412+0000 7f553637e700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:39:44.506 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:44.412+0000 7f553637e700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:39:44.506 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:44.412+0000 7f553637e700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:39:44.506 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:39:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:39:44.412+0000 7f553637e700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:39:44.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: Standby manager daemon z restarted 2024-05-27T07:39:44.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: Standby manager daemon z started 2024-05-27T07:39:44.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.186:0/3379142781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:39:44.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.186:0/3379142781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:39:44.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.186:0/3379142781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:39:44.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.186:0/3379142781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:39:44.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: Standby manager daemon y restarted 2024-05-27T07:39:44.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: Standby manager daemon y started 2024-05-27T07:39:44.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.163:0/169715530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:39:44.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.163:0/169715530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:39:44.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.163:0/169715530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:39:44.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.? 172.21.15.163:0/169715530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:39:44.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: Active manager daemon x restarted 2024-05-27T07:39:44.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: Activating manager daemon x 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: mgrmap e20: x(active, starting, since 0.0159069s), standbys: y, z 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:39:44.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:39:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:39:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:39:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:39:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:39:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:39:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:39:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:39:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:39:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:39:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:39:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:39:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:39:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:39:44.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:39:44.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:39:44.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:39:44.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:39:44.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:39:44.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:39:44.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:39:44.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:39:44.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: Manager daemon x is now available 2024-05-27T07:39:44.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:44.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:39:44.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:39:44.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: Standby manager daemon z restarted 2024-05-27T07:39:44.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: Standby manager daemon z started 2024-05-27T07:39:44.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.186:0/3379142781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:39:44.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.186:0/3379142781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:39:44.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.186:0/3379142781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:39:44.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.186:0/3379142781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:39:44.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: Standby manager daemon y restarted 2024-05-27T07:39:44.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: Standby manager daemon y started 2024-05-27T07:39:44.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.163:0/169715530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.163:0/169715530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.163:0/169715530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.? 172.21.15.163:0/169715530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: Active manager daemon x restarted 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: Activating manager daemon x 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: mgrmap e20: x(active, starting, since 0.0159069s), standbys: y, z 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:39:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:39:44.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:39:44.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:39:44.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:39:44.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:39:44.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:39:44.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:39:44.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:39:44.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:39:44.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:39:44.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:39:44.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:39:44.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:39:44.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:39:44.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:39:44.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:39:44.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:39:44.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:39:44.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:39:44.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:39:44.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:39:44.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:39:44.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:39:44.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:39:44.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:39:44.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:39:44.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:39:44.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: Manager daemon x is now available 2024-05-27T07:39:44.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:44.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:39:44.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:39:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.746+0000 7f65e9980700 1 -- 172.21.15.114:0/577207541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4108730 msgr2=0x7f65e4110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.746+0000 7f65e9980700 1 --2- 172.21.15.114:0/577207541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4108730 0x7f65e4110090 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f65dc00b0d0 tx=0x7f65dc00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:39:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.746+0000 7f65e9980700 1 -- 172.21.15.114:0/577207541 shutdown_connections 2024-05-27T07:39:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.746+0000 7f65e9980700 1 --2- 172.21.15.114:0/577207541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4108730 0x7f65e4110090 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.746+0000 7f65e9980700 1 --2- 172.21.15.114:0/577207541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65e41034b0 0x7f65e41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.746+0000 7f65e9980700 1 --2- 172.21.15.114:0/577207541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65e4107de0 0x7f65e41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.746+0000 7f65e9980700 1 -- 172.21.15.114:0/577207541 >> 172.21.15.114:0/577207541 conn(0x7f65e40fedb0 msgr2=0x7f65e41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.746+0000 7f65e9980700 1 -- 172.21.15.114:0/577207541 shutdown_connections 2024-05-27T07:39:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.746+0000 7f65e9980700 1 -- 172.21.15.114:0/577207541 wait complete. 2024-05-27T07:39:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.747+0000 7f65e9980700 1 Processor -- start 2024-05-27T07:39:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.747+0000 7f65e9980700 1 -- start start 2024-05-27T07:39:44.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e9980700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65e41034b0 0x7f65e4199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:44.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e9980700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65e4107de0 0x7f65e419a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:44.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e9980700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4108730 0x7f65e419e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:44.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e9980700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65e4113360 con 0x7f65e4108730 2024-05-27T07:39:44.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e9980700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f65e41131e0 con 0x7f65e41034b0 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e9980700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f65e41134e0 con 0x7f65e4107de0 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65e41034b0 0x7f65e4199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e27fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65e4107de0 0x7f65e419a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65e41034b0 0x7f65e4199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:55730/0 (socket says 172.21.15.114:55730) 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e2ffd700 1 -- 172.21.15.114:0/3401243957 learned_addr learned my addr 172.21.15.114:0/3401243957 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e37fe700 1 --2- 172.21.15.114:0/3401243957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4108730 0x7f65e419e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e2ffd700 1 -- 172.21.15.114:0/3401243957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65e4107de0 msgr2=0x7f65e419a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e2ffd700 1 --2- 172.21.15.114:0/3401243957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65e4107de0 0x7f65e419a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e2ffd700 1 -- 172.21.15.114:0/3401243957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4108730 msgr2=0x7f65e419e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e2ffd700 1 --2- 172.21.15.114:0/3401243957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4108730 0x7f65e419e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e2ffd700 1 -- 172.21.15.114:0/3401243957 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e419ec00 con 0x7f65e41034b0 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.748+0000 7f65e27fc700 1 --2- 172.21.15.114:0/3401243957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65e4107de0 0x7f65e419a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.749+0000 7f65e2ffd700 1 --2- 172.21.15.114:0/3401243957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65e41034b0 0x7f65e4199d30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f65d400b890 tx=0x7f65d400bba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.749+0000 7f65e897e700 1 -- 172.21.15.114:0/3401243957 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d400e8b0 con 0x7f65e41034b0 2024-05-27T07:39:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.749+0000 7f65e897e700 1 -- 172.21.15.114:0/3401243957 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65d40173f0 con 0x7f65e41034b0 2024-05-27T07:39:44.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.749+0000 7f65e9980700 1 -- 172.21.15.114:0/3401243957 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e419eef0 con 0x7f65e41034b0 2024-05-27T07:39:44.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.749+0000 7f65e9980700 1 -- 172.21.15.114:0/3401243957 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e406b600 con 0x7f65e41034b0 2024-05-27T07:39:44.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.749+0000 7f65e897e700 1 -- 172.21.15.114:0/3401243957 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d400a610 con 0x7f65e41034b0 2024-05-27T07:39:44.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.750+0000 7f65e9980700 1 -- 172.21.15.114:0/3401243957 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65e40512b0 con 0x7f65e41034b0 2024-05-27T07:39:44.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.752+0000 7f65e897e700 1 -- 172.21.15.114:0/3401243957 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 149337+0+0 (secure 0 0 0) 0x7f65d40175a0 con 0x7f65e41034b0 2024-05-27T07:39:44.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.753+0000 7f65e897e700 1 -- 172.21.15.114:0/3401243957 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f65d40e1d60 con 0x7f65e41034b0 2024-05-27T07:39:44.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:44.756+0000 7f65e897e700 1 -- 172.21.15.114:0/3401243957 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f65d40a99b0 con 0x7f65e41034b0 2024-05-27T07:39:44.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: Standby manager daemon z restarted 2024-05-27T07:39:44.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: Standby manager daemon z started 2024-05-27T07:39:44.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.186:0/3379142781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:39:44.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.186:0/3379142781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:39:44.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.186:0/3379142781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:39:44.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.186:0/3379142781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:39:44.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: Standby manager daemon y restarted 2024-05-27T07:39:44.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: Standby manager daemon y started 2024-05-27T07:39:44.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.163:0/169715530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:39:44.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.163:0/169715530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:39:44.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.163:0/169715530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:39:44.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.? 172.21.15.163:0/169715530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:39:44.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: Active manager daemon x restarted 2024-05-27T07:39:44.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: Activating manager daemon x 2024-05-27T07:39:44.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: mgrmap e20: x(active, starting, since 0.0159069s), standbys: y, z 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:39:44.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:39:44.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:39:44.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:39:44.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:39:44.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:39:44.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:39:44.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:39:44.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:39:44.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:39:44.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: Manager daemon x is now available 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:39:44.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:39:45.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:45.020+0000 7f65e9980700 1 -- 172.21.15.114:0/3401243957 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65e406c080 con 0x7f65e41034b0 2024-05-27T07:39:45.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:45.021+0000 7f65e897e700 1 -- 172.21.15.114:0/3401243957 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11827 (secure 0 0 0) 0x7f65d40a92b0 con 0x7f65e41034b0 2024-05-27T07:39:45.018 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:45.019 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:37:06.645274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:39:45.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:45.023+0000 7f65e9980700 1 -- 172.21.15.114:0/3401243957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65e41034b0 msgr2=0x7f65e4199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:45.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:45.023+0000 7f65e9980700 1 --2- 172.21.15.114:0/3401243957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65e41034b0 0x7f65e4199d30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f65d400b890 tx=0x7f65d400bba0 comp rx=0 tx=0).stop 2024-05-27T07:39:45.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:45.023+0000 7f65e9980700 1 -- 172.21.15.114:0/3401243957 shutdown_connections 2024-05-27T07:39:45.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:45.023+0000 7f65e9980700 1 --2- 172.21.15.114:0/3401243957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4108730 0x7f65e419e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:45.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:45.023+0000 7f65e9980700 1 --2- 172.21.15.114:0/3401243957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65e41034b0 0x7f65e4199d30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:45.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:45.023+0000 7f65e9980700 1 --2- 172.21.15.114:0/3401243957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65e4107de0 0x7f65e419a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:45.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:45.023+0000 7f65e9980700 1 -- 172.21.15.114:0/3401243957 >> 172.21.15.114:0/3401243957 conn(0x7f65e40fedb0 msgr2=0x7f65e4107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:45.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:45.023+0000 7f65e9980700 1 -- 172.21.15.114:0/3401243957 shutdown_connections 2024-05-27T07:39:45.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:45.023+0000 7f65e9980700 1 -- 172.21.15.114:0/3401243957 wait complete. 2024-05-27T07:39:45.021 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:39:45.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:45 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3401243957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:46.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:45 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:45 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:45 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:45 smithi114 ceph-mon[32038]: mgrmap e21: x(active, since 1.0265s), standbys: y, z 2024-05-27T07:39:46.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:45 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:45 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:45 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.261 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:45 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3401243957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:46.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:45 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:45 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:45 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:45 smithi186 ceph-mon[36647]: mgrmap e21: x(active, since 1.0265s), standbys: y, z 2024-05-27T07:39:46.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:45 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:45 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:45 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:45 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3401243957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:46.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:45 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:45 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:45 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:45 smithi163 ceph-mon[36451]: mgrmap e21: x(active, since 1.0265s), standbys: y, z 2024-05-27T07:39:46.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:45 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:45 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:45 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:46.264 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-05-27T07:39:46.264 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-05-27T07:39:46.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T07:39:46.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.384+0000 7f4671ce9640 1 -- 172.21.15.114:0/2188308433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c103090 msgr2=0x7f466c1034a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.384+0000 7f4671ce9640 1 --2- 172.21.15.114:0/2188308433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c103090 0x7f466c1034a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f465c009900 tx=0x7f465c02a410 comp rx=0 tx=0).stop 2024-05-27T07:39:46.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.384+0000 7f4671ce9640 1 -- 172.21.15.114:0/2188308433 shutdown_connections 2024-05-27T07:39:46.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.384+0000 7f4671ce9640 1 --2- 172.21.15.114:0/2188308433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f466c104590 0x7f466c10ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.384+0000 7f4671ce9640 1 --2- 172.21.15.114:0/2188308433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f466c103a70 0x7f466c103f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.384+0000 7f4671ce9640 1 --2- 172.21.15.114:0/2188308433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c103090 0x7f466c1034a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.384+0000 7f4671ce9640 1 -- 172.21.15.114:0/2188308433 >> 172.21.15.114:0/2188308433 conn(0x7f466c0fd4d0 msgr2=0x7f466c0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.384+0000 7f4671ce9640 1 -- 172.21.15.114:0/2188308433 shutdown_connections 2024-05-27T07:39:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.384+0000 7f4671ce9640 1 -- 172.21.15.114:0/2188308433 wait complete. 2024-05-27T07:39:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.384+0000 7f4671ce9640 1 Processor -- start 2024-05-27T07:39:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.384+0000 7f4671ce9640 1 -- start start 2024-05-27T07:39:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f4671ce9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c103090 0x7f466c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f4671ce9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f466c103a70 0x7f466c19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f4671ce9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f466c104590 0x7f466c1a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f4671ce9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f466c10f760 con 0x7f466c103090 2024-05-27T07:39:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f4671ce9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f466c10f5e0 con 0x7f466c104590 2024-05-27T07:39:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f4671ce9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f466c10f8e0 con 0x7f466c103a70 2024-05-27T07:39:46.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f466b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c103090 0x7f466c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f466b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c103090 0x7f466c19a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43532/0 (socket says 172.21.15.114:43532) 2024-05-27T07:39:46.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f466b7fe640 1 -- 172.21.15.114:0/2984204542 learned_addr learned my addr 172.21.15.114:0/2984204542 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:46.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f466bfff640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f466c104590 0x7f466c1a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f4662ffd640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f466c103a70 0x7f466c19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f466b7fe640 1 -- 172.21.15.114:0/2984204542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f466c103a70 msgr2=0x7f466c19a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f466b7fe640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f466c103a70 0x7f466c19a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f466b7fe640 1 -- 172.21.15.114:0/2984204542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f466c104590 msgr2=0x7f466c1a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f466b7fe640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f466c104590 0x7f466c1a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f466b7fe640 1 -- 172.21.15.114:0/2984204542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f466c1a1e20 con 0x7f466c103090 2024-05-27T07:39:46.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.385+0000 7f466b7fe640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c103090 0x7f466c19a2e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f465c006010 tx=0x7f465c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:46.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.386+0000 7f46697fa640 1 -- 172.21.15.114:0/2984204542 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465c03f030 con 0x7f466c103090 2024-05-27T07:39:46.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.386+0000 7f4671ce9640 1 -- 172.21.15.114:0/2984204542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f466c1a20b0 con 0x7f466c103090 2024-05-27T07:39:46.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.386+0000 7f46697fa640 1 -- 172.21.15.114:0/2984204542 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f465c031910 con 0x7f466c103090 2024-05-27T07:39:46.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.386+0000 7f46697fa640 1 -- 172.21.15.114:0/2984204542 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465c035070 con 0x7f466c103090 2024-05-27T07:39:46.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.386+0000 7f4671ce9640 1 -- 172.21.15.114:0/2984204542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f466c1a25f0 con 0x7f466c103090 2024-05-27T07:39:46.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.387+0000 7f4671ce9640 1 -- 172.21.15.114:0/2984204542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f466c107a30 con 0x7f466c103090 2024-05-27T07:39:46.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.388+0000 7f46697fa640 1 -- 172.21.15.114:0/2984204542 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 21) ==== 149464+0+0 (secure 0 0 0) 0x7f465c042030 con 0x7f466c103090 2024-05-27T07:39:46.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.389+0000 7f46697fa640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46580b25e0 0x7f46580b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.389+0000 7f4662ffd640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46580b25e0 0x7f46580b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.389+0000 7f46697fa640 1 -- 172.21.15.114:0/2984204542 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f465c102a40 con 0x7f466c103090 2024-05-27T07:39:46.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.389+0000 7f4662ffd640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46580b25e0 0x7f46580b4af0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f466c19b380 tx=0x7f4654005f90 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:46.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.393+0000 7f46697fa640 1 -- 172.21.15.114:0/2984204542 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f465c0c9f30 con 0x7f466c103090 2024-05-27T07:39:46.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.544+0000 7f46697fa640 1 -- 172.21.15.114:0/2984204542 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 22) ==== 149773+0+0 (secure 0 0 0) 0x7f465c003d10 con 0x7f466c103090 2024-05-27T07:39:46.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.586+0000 7f4671ce9640 1 -- 172.21.15.114:0/2984204542 --> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f466c063100 con 0x7f46580b25e0 2024-05-27T07:39:46.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.596+0000 7f46697fa640 1 -- 172.21.15.114:0/2984204542 <== mgr.15138 v2:172.21.15.114:6800/3311646739 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339584 (secure 0 0 0) 0x7f466c063100 con 0x7f46580b25e0 2024-05-27T07:39:46.594 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:46.604 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-05-27T07:39:46.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.599+0000 7f4671ce9640 1 -- 172.21.15.114:0/2984204542 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46580b25e0 msgr2=0x7f46580b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.599+0000 7f4671ce9640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46580b25e0 0x7f46580b4af0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f466c19b380 tx=0x7f4654005f90 comp rx=0 tx=0).stop 2024-05-27T07:39:46.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.599+0000 7f4671ce9640 1 -- 172.21.15.114:0/2984204542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c103090 msgr2=0x7f466c19a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.599+0000 7f4671ce9640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c103090 0x7f466c19a2e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f465c006010 tx=0x7f465c004270 comp rx=0 tx=0).stop 2024-05-27T07:39:46.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.600+0000 7f4671ce9640 1 -- 172.21.15.114:0/2984204542 shutdown_connections 2024-05-27T07:39:46.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.600+0000 7f4671ce9640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46580b25e0 0x7f46580b4af0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.600+0000 7f4671ce9640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f466c104590 0x7f466c1a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.600+0000 7f4671ce9640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f466c103a70 0x7f466c19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.600+0000 7f4671ce9640 1 --2- 172.21.15.114:0/2984204542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c103090 0x7f466c19a2e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.600+0000 7f4671ce9640 1 -- 172.21.15.114:0/2984204542 >> 172.21.15.114:0/2984204542 conn(0x7f466c0fd4d0 msgr2=0x7f466c104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:46.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.600+0000 7f4671ce9640 1 -- 172.21.15.114:0/2984204542 shutdown_connections 2024-05-27T07:39:46.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.600+0000 7f4671ce9640 1 -- 172.21.15.114:0/2984204542 wait complete. 2024-05-27T07:39:46.609 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-05-27T07:39:46.148375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1187573,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":516,"num_read_kb":393,"num_write":334,"num_write_kb":3567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2951979,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":749,"ondisk_log_size":749,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369520,"kb_used_data":12012,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572304,"statfs":{"total":1151940427776,"available":1151562039296,"internally_reserved":0,"allocated":12300288,"data_stored":8880701,"data_compressed":551551,"data_compressed_allocated":749568,"data_compressed_original":3555328,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.993045"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418062+0000","last_change":"2024-05-27T07:36:04.980329+0000","last_active":"2024-05-27T07:39:44.418062+0000","last_peered":"2024-05-27T07:39:44.418062+0000","last_clean":"2024-05-27T07:39:44.418062+0000","last_became_active":"2024-05-27T07:36:04.980006+0000","last_became_peered":"2024-05-27T07:36:04.980006+0000","last_unstale":"2024-05-27T07:39:44.418062+0000","last_undegraded":"2024-05-27T07:39:44.418062+0000","last_fullsized":"2024-05-27T07:39:44.418062+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:51:56.728465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417978+0000","last_change":"2024-05-27T07:36:04.983429+0000","last_active":"2024-05-27T07:39:44.417978+0000","last_peered":"2024-05-27T07:39:44.417978+0000","last_clean":"2024-05-27T07:39:44.417978+0000","last_became_active":"2024-05-27T07:36:04.983099+0000","last_became_peered":"2024-05-27T07:36:04.983099+0000","last_unstale":"2024-05-27T07:39:44.417978+0000","last_undegraded":"2024-05-27T07:39:44.417978+0000","last_fullsized":"2024-05-27T07:39:44.417978+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:18:51.059717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417722+0000","last_change":"2024-05-27T07:36:04.993319+0000","last_active":"2024-05-27T07:39:44.417722+0000","last_peered":"2024-05-27T07:39:44.417722+0000","last_clean":"2024-05-27T07:39:44.417722+0000","last_became_active":"2024-05-27T07:36:04.992983+0000","last_became_peered":"2024-05-27T07:36:04.992983+0000","last_unstale":"2024-05-27T07:39:44.417722+0000","last_undegraded":"2024-05-27T07:39:44.417722+0000","last_fullsized":"2024-05-27T07:39:44.417722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:50.800194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244278+0000","last_change":"2024-05-27T07:36:45.140980+0000","last_active":"2024-05-27T07:39:44.244278+0000","last_peered":"2024-05-27T07:39:44.244278+0000","last_clean":"2024-05-27T07:39:44.244278+0000","last_became_active":"2024-05-27T07:36:45.140551+0000","last_became_peered":"2024-05-27T07:36:45.140551+0000","last_unstale":"2024-05-27T07:39:44.244278+0000","last_undegraded":"2024-05-27T07:39:44.244278+0000","last_fullsized":"2024-05-27T07:39:44.244278+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:59:21.646306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246654+0000","last_change":"2024-05-27T07:36:04.977404+0000","last_active":"2024-05-27T07:39:44.246654+0000","last_peered":"2024-05-27T07:39:44.246654+0000","last_clean":"2024-05-27T07:39:44.246654+0000","last_became_active":"2024-05-27T07:36:04.976991+0000","last_became_peered":"2024-05-27T07:36:04.976991+0000","last_unstale":"2024-05-27T07:39:44.246654+0000","last_undegraded":"2024-05-27T07:39:44.246654+0000","last_fullsized":"2024-05-27T07:39:44.246654+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:56.532586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133579+0000","last_change":"2024-05-27T07:36:04.982149+0000","last_active":"2024-05-27T07:39:44.133579+0000","last_peered":"2024-05-27T07:39:44.133579+0000","last_clean":"2024-05-27T07:39:44.133579+0000","last_became_active":"2024-05-27T07:36:04.981751+0000","last_became_peered":"2024-05-27T07:36:04.981751+0000","last_unstale":"2024-05-27T07:39:44.133579+0000","last_undegraded":"2024-05-27T07:39:44.133579+0000","last_fullsized":"2024-05-27T07:39:44.133579+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:42.773458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417510+0000","last_change":"2024-05-27T07:36:04.987491+0000","last_active":"2024-05-27T07:39:44.417510+0000","last_peered":"2024-05-27T07:39:44.417510+0000","last_clean":"2024-05-27T07:39:44.417510+0000","last_became_active":"2024-05-27T07:36:04.987059+0000","last_became_peered":"2024-05-27T07:36:04.987059+0000","last_unstale":"2024-05-27T07:39:44.417510+0000","last_undegraded":"2024-05-27T07:39:44.417510+0000","last_fullsized":"2024-05-27T07:39:44.417510+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:57.611253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.243516+0000","last_change":"2024-05-27T07:36:45.126961+0000","last_active":"2024-05-27T07:39:44.243516+0000","last_peered":"2024-05-27T07:39:44.243516+0000","last_clean":"2024-05-27T07:39:44.243516+0000","last_became_active":"2024-05-27T07:36:45.126653+0000","last_became_peered":"2024-05-27T07:36:45.126653+0000","last_unstale":"2024-05-27T07:39:44.243516+0000","last_undegraded":"2024-05-27T07:39:44.243516+0000","last_fullsized":"2024-05-27T07:39:44.243516+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:35.586128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418769+0000","last_change":"2024-05-27T07:36:04.990469+0000","last_active":"2024-05-27T07:39:44.418769+0000","last_peered":"2024-05-27T07:39:44.418769+0000","last_clean":"2024-05-27T07:39:44.418769+0000","last_became_active":"2024-05-27T07:36:04.989771+0000","last_became_peered":"2024-05-27T07:36:04.989771+0000","last_unstale":"2024-05-27T07:39:44.418769+0000","last_undegraded":"2024-05-27T07:39:44.418769+0000","last_fullsized":"2024-05-27T07:39:44.418769+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:53.593566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134126+0000","last_change":"2024-05-27T07:36:04.984746+0000","last_active":"2024-05-27T07:39:44.134126+0000","last_peered":"2024-05-27T07:39:44.134126+0000","last_clean":"2024-05-27T07:39:44.134126+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:39:44.134126+0000","last_undegraded":"2024-05-27T07:39:44.134126+0000","last_fullsized":"2024-05-27T07:39:44.134126+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:18.071981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419074+0000","last_change":"2024-05-27T07:36:04.977167+0000","last_active":"2024-05-27T07:39:44.419074+0000","last_peered":"2024-05-27T07:39:44.419074+0000","last_clean":"2024-05-27T07:39:44.419074+0000","last_became_active":"2024-05-27T07:36:04.976788+0000","last_became_peered":"2024-05-27T07:36:04.976788+0000","last_unstale":"2024-05-27T07:39:44.419074+0000","last_undegraded":"2024-05-27T07:39:44.419074+0000","last_fullsized":"2024-05-27T07:39:44.419074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:59:44.574191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418351+0000","last_change":"2024-05-27T07:36:04.986594+0000","last_active":"2024-05-27T07:39:44.418351+0000","last_peered":"2024-05-27T07:39:44.418351+0000","last_clean":"2024-05-27T07:39:44.418351+0000","last_became_active":"2024-05-27T07:36:04.985161+0000","last_became_peered":"2024-05-27T07:36:04.985161+0000","last_unstale":"2024-05-27T07:39:44.418351+0000","last_undegraded":"2024-05-27T07:39:44.418351+0000","last_fullsized":"2024-05-27T07:39:44.418351+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.239461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133787+0000","last_change":"2024-05-27T07:36:04.984781+0000","last_active":"2024-05-27T07:39:44.133787+0000","last_peered":"2024-05-27T07:39:44.133787+0000","last_clean":"2024-05-27T07:39:44.133787+0000","last_became_active":"2024-05-27T07:36:04.984462+0000","last_became_peered":"2024-05-27T07:36:04.984462+0000","last_unstale":"2024-05-27T07:39:44.133787+0000","last_undegraded":"2024-05-27T07:39:44.133787+0000","last_fullsized":"2024-05-27T07:39:44.133787+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:40.051692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417506+0000","last_change":"2024-05-27T07:36:04.978360+0000","last_active":"2024-05-27T07:39:44.417506+0000","last_peered":"2024-05-27T07:39:44.417506+0000","last_clean":"2024-05-27T07:39:44.417506+0000","last_became_active":"2024-05-27T07:36:04.978105+0000","last_became_peered":"2024-05-27T07:36:04.978105+0000","last_unstale":"2024-05-27T07:39:44.417506+0000","last_undegraded":"2024-05-27T07:39:44.417506+0000","last_fullsized":"2024-05-27T07:39:44.417506+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:04:47.257529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417507+0000","last_change":"2024-05-27T07:36:04.986510+0000","last_active":"2024-05-27T07:39:44.417507+0000","last_peered":"2024-05-27T07:39:44.417507+0000","last_clean":"2024-05-27T07:39:44.417507+0000","last_became_active":"2024-05-27T07:36:04.986150+0000","last_became_peered":"2024-05-27T07:36:04.986150+0000","last_unstale":"2024-05-27T07:39:44.417507+0000","last_undegraded":"2024-05-27T07:39:44.417507+0000","last_fullsized":"2024-05-27T07:39:44.417507+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:07.349470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418178+0000","last_change":"2024-05-27T07:36:04.979880+0000","last_active":"2024-05-27T07:39:44.418178+0000","last_peered":"2024-05-27T07:39:44.418178+0000","last_clean":"2024-05-27T07:39:44.418178+0000","last_became_active":"2024-05-27T07:36:04.979516+0000","last_became_peered":"2024-05-27T07:36:04.979516+0000","last_unstale":"2024-05-27T07:39:44.418178+0000","last_undegraded":"2024-05-27T07:39:44.418178+0000","last_fullsized":"2024-05-27T07:39:44.418178+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:41:03.964277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417955+0000","last_change":"2024-05-27T07:36:04.980231+0000","last_active":"2024-05-27T07:39:44.417955+0000","last_peered":"2024-05-27T07:39:44.417955+0000","last_clean":"2024-05-27T07:39:44.417955+0000","last_became_active":"2024-05-27T07:36:04.979888+0000","last_became_peered":"2024-05-27T07:36:04.979888+0000","last_unstale":"2024-05-27T07:39:44.417955+0000","last_undegraded":"2024-05-27T07:39:44.417955+0000","last_fullsized":"2024-05-27T07:39:44.417955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.838119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244612+0000","last_change":"2024-05-27T07:36:04.990890+0000","last_active":"2024-05-27T07:39:44.244612+0000","last_peered":"2024-05-27T07:39:44.244612+0000","last_clean":"2024-05-27T07:39:44.244612+0000","last_became_active":"2024-05-27T07:36:04.990581+0000","last_became_peered":"2024-05-27T07:36:04.990581+0000","last_unstale":"2024-05-27T07:39:44.244612+0000","last_undegraded":"2024-05-27T07:39:44.244612+0000","last_fullsized":"2024-05-27T07:39:44.244612+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:33.054395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363804+0000","last_change":"2024-05-27T07:36:04.981620+0000","last_active":"2024-05-27T07:39:44.363804+0000","last_peered":"2024-05-27T07:39:44.363804+0000","last_clean":"2024-05-27T07:39:44.363804+0000","last_became_active":"2024-05-27T07:36:04.981290+0000","last_became_peered":"2024-05-27T07:36:04.981290+0000","last_unstale":"2024-05-27T07:39:44.363804+0000","last_undegraded":"2024-05-27T07:39:44.363804+0000","last_fullsized":"2024-05-27T07:39:44.363804+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:50.287380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363074+0000","last_change":"2024-05-27T07:36:15.030740+0000","last_active":"2024-05-27T07:39:44.363074+0000","last_peered":"2024-05-27T07:39:44.363074+0000","last_clean":"2024-05-27T07:39:44.363074+0000","last_became_active":"2024-05-27T07:36:15.029738+0000","last_became_peered":"2024-05-27T07:36:15.029738+0000","last_unstale":"2024-05-27T07:39:44.363074+0000","last_undegraded":"2024-05-27T07:39:44.363074+0000","last_fullsized":"2024-05-27T07:39:44.363074+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:18.101941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"78'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244544+0000","last_change":"2024-05-27T07:36:01.955530+0000","last_active":"2024-05-27T07:39:44.244544+0000","last_peered":"2024-05-27T07:39:44.244544+0000","last_clean":"2024-05-27T07:39:44.244544+0000","last_became_active":"2024-05-27T07:36:01.955225+0000","last_became_peered":"2024-05-27T07:36:01.955225+0000","last_unstale":"2024-05-27T07:39:44.244544+0000","last_undegraded":"2024-05-27T07:39:44.244544+0000","last_fullsized":"2024-05-27T07:39:44.244544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:02.067672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.248070+0000","last_change":"2024-05-27T07:36:04.984397+0000","last_active":"2024-05-27T07:39:44.248070+0000","last_peered":"2024-05-27T07:39:44.248070+0000","last_clean":"2024-05-27T07:39:44.248070+0000","last_became_active":"2024-05-27T07:36:04.984033+0000","last_became_peered":"2024-05-27T07:36:04.984033+0000","last_unstale":"2024-05-27T07:39:44.248070+0000","last_undegraded":"2024-05-27T07:39:44.248070+0000","last_fullsized":"2024-05-27T07:39:44.248070+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:15.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417201+0000","last_change":"2024-05-27T07:37:46.322144+0000","last_active":"2024-05-27T07:39:44.417201+0000","last_peered":"2024-05-27T07:39:44.417201+0000","last_clean":"2024-05-27T07:39:44.417201+0000","last_became_active":"2024-05-27T07:37:46.321610+0000","last_became_peered":"2024-05-27T07:37:46.321610+0000","last_unstale":"2024-05-27T07:39:44.417201+0000","last_undegraded":"2024-05-27T07:39:44.417201+0000","last_fullsized":"2024-05-27T07:39:44.417201+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:47.119281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"78'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.248298+0000","last_change":"2024-05-27T07:36:02.243079+0000","last_active":"2024-05-27T07:39:44.248298+0000","last_peered":"2024-05-27T07:39:44.248298+0000","last_clean":"2024-05-27T07:39:44.248298+0000","last_became_active":"2024-05-27T07:36:02.242633+0000","last_became_peered":"2024-05-27T07:36:02.242633+0000","last_unstale":"2024-05-27T07:39:44.248298+0000","last_undegraded":"2024-05-27T07:39:44.248298+0000","last_fullsized":"2024-05-27T07:39:44.248298+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:45:33.980842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417940+0000","last_change":"2024-05-27T07:36:04.989775+0000","last_active":"2024-05-27T07:39:44.417940+0000","last_peered":"2024-05-27T07:39:44.417940+0000","last_clean":"2024-05-27T07:39:44.417940+0000","last_became_active":"2024-05-27T07:36:04.989289+0000","last_became_peered":"2024-05-27T07:36:04.989289+0000","last_unstale":"2024-05-27T07:39:44.417940+0000","last_undegraded":"2024-05-27T07:39:44.417940+0000","last_fullsized":"2024-05-27T07:39:44.417940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:24:38.355231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417613+0000","last_change":"2024-05-27T07:36:15.031737+0000","last_active":"2024-05-27T07:39:44.417613+0000","last_peered":"2024-05-27T07:39:44.417613+0000","last_clean":"2024-05-27T07:39:44.417613+0000","last_became_active":"2024-05-27T07:36:15.031314+0000","last_became_peered":"2024-05-27T07:36:15.031314+0000","last_unstale":"2024-05-27T07:39:44.417613+0000","last_undegraded":"2024-05-27T07:39:44.417613+0000","last_fullsized":"2024-05-27T07:39:44.417613+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:32.742298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"78'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247148+0000","last_change":"2024-05-27T07:36:01.957459+0000","last_active":"2024-05-27T07:39:44.247148+0000","last_peered":"2024-05-27T07:39:44.247148+0000","last_clean":"2024-05-27T07:39:44.247148+0000","last_became_active":"2024-05-27T07:36:01.954985+0000","last_became_peered":"2024-05-27T07:36:01.954985+0000","last_unstale":"2024-05-27T07:39:44.247148+0000","last_undegraded":"2024-05-27T07:39:44.247148+0000","last_fullsized":"2024-05-27T07:39:44.247148+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:27.085486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418627+0000","last_change":"2024-05-27T07:36:04.986447+0000","last_active":"2024-05-27T07:39:44.418627+0000","last_peered":"2024-05-27T07:39:44.418627+0000","last_clean":"2024-05-27T07:39:44.418627+0000","last_became_active":"2024-05-27T07:36:04.982099+0000","last_became_peered":"2024-05-27T07:36:04.982099+0000","last_unstale":"2024-05-27T07:39:44.418627+0000","last_undegraded":"2024-05-27T07:39:44.418627+0000","last_fullsized":"2024-05-27T07:39:44.418627+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:32.637479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417267+0000","last_change":"2024-05-27T07:36:15.024152+0000","last_active":"2024-05-27T07:39:44.417267+0000","last_peered":"2024-05-27T07:39:44.417267+0000","last_clean":"2024-05-27T07:39:44.417267+0000","last_became_active":"2024-05-27T07:36:15.023711+0000","last_became_peered":"2024-05-27T07:36:15.023711+0000","last_unstale":"2024-05-27T07:39:44.417267+0000","last_undegraded":"2024-05-27T07:39:44.417267+0000","last_fullsized":"2024-05-27T07:39:44.417267+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:33.809364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246823+0000","last_change":"2024-05-27T07:36:01.950531+0000","last_active":"2024-05-27T07:39:44.246823+0000","last_peered":"2024-05-27T07:39:44.246823+0000","last_clean":"2024-05-27T07:39:44.246823+0000","last_became_active":"2024-05-27T07:36:01.950015+0000","last_became_peered":"2024-05-27T07:36:01.950015+0000","last_unstale":"2024-05-27T07:39:44.246823+0000","last_undegraded":"2024-05-27T07:39:44.246823+0000","last_fullsized":"2024-05-27T07:39:44.246823+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:22.400415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135595+0000","last_change":"2024-05-27T07:36:04.982792+0000","last_active":"2024-05-27T07:39:44.135595+0000","last_peered":"2024-05-27T07:39:44.135595+0000","last_clean":"2024-05-27T07:39:44.135595+0000","last_became_active":"2024-05-27T07:36:04.982400+0000","last_became_peered":"2024-05-27T07:36:04.982400+0000","last_unstale":"2024-05-27T07:39:44.135595+0000","last_undegraded":"2024-05-27T07:39:44.135595+0000","last_fullsized":"2024-05-27T07:39:44.135595+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:27.823654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363642+0000","last_change":"2024-05-27T07:36:15.027932+0000","last_active":"2024-05-27T07:39:44.363642+0000","last_peered":"2024-05-27T07:39:44.363642+0000","last_clean":"2024-05-27T07:39:44.363642+0000","last_became_active":"2024-05-27T07:36:15.027526+0000","last_became_peered":"2024-05-27T07:36:15.027526+0000","last_unstale":"2024-05-27T07:39:44.363642+0000","last_undegraded":"2024-05-27T07:39:44.363642+0000","last_fullsized":"2024-05-27T07:39:44.363642+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:16.165934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"78'1","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135427+0000","last_change":"2024-05-27T07:36:45.127637+0000","last_active":"2024-05-27T07:39:44.135427+0000","last_peered":"2024-05-27T07:39:44.135427+0000","last_clean":"2024-05-27T07:39:44.135427+0000","last_became_active":"2024-05-27T07:36:45.127249+0000","last_became_peered":"2024-05-27T07:36:45.127249+0000","last_unstale":"2024-05-27T07:39:44.135427+0000","last_undegraded":"2024-05-27T07:39:44.135427+0000","last_fullsized":"2024-05-27T07:39:44.135427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:52.418069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246700+0000","last_change":"2024-05-27T07:36:04.985135+0000","last_active":"2024-05-27T07:39:44.246700+0000","last_peered":"2024-05-27T07:39:44.246700+0000","last_clean":"2024-05-27T07:39:44.246700+0000","last_became_active":"2024-05-27T07:36:04.983621+0000","last_became_peered":"2024-05-27T07:36:04.983621+0000","last_unstale":"2024-05-27T07:39:44.246700+0000","last_undegraded":"2024-05-27T07:39:44.246700+0000","last_fullsized":"2024-05-27T07:39:44.246700+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:11.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363376+0000","last_change":"2024-05-27T07:36:15.024558+0000","last_active":"2024-05-27T07:39:44.363376+0000","last_peered":"2024-05-27T07:39:44.363376+0000","last_clean":"2024-05-27T07:39:44.363376+0000","last_became_active":"2024-05-27T07:36:15.024148+0000","last_became_peered":"2024-05-27T07:36:15.024148+0000","last_unstale":"2024-05-27T07:39:44.363376+0000","last_undegraded":"2024-05-27T07:39:44.363376+0000","last_fullsized":"2024-05-27T07:39:44.363376+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:03.648451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"78'1","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133382+0000","last_change":"2024-05-27T07:36:45.129413+0000","last_active":"2024-05-27T07:39:44.133382+0000","last_peered":"2024-05-27T07:39:44.133382+0000","last_clean":"2024-05-27T07:39:44.133382+0000","last_became_active":"2024-05-27T07:36:45.128779+0000","last_became_peered":"2024-05-27T07:36:45.128779+0000","last_unstale":"2024-05-27T07:39:44.133382+0000","last_undegraded":"2024-05-27T07:39:44.133382+0000","last_fullsized":"2024-05-27T07:39:44.133382+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:05.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135849+0000","last_change":"2024-05-27T07:36:04.989878+0000","last_active":"2024-05-27T07:39:44.135849+0000","last_peered":"2024-05-27T07:39:44.135849+0000","last_clean":"2024-05-27T07:39:44.135849+0000","last_became_active":"2024-05-27T07:36:04.989532+0000","last_became_peered":"2024-05-27T07:36:04.989532+0000","last_unstale":"2024-05-27T07:39:44.135849+0000","last_undegraded":"2024-05-27T07:39:44.135849+0000","last_fullsized":"2024-05-27T07:39:44.135849+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:54:53.909592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417859+0000","last_change":"2024-05-27T07:37:46.321582+0000","last_active":"2024-05-27T07:39:44.417859+0000","last_peered":"2024-05-27T07:39:44.417859+0000","last_clean":"2024-05-27T07:39:44.417859+0000","last_became_active":"2024-05-27T07:37:46.321193+0000","last_became_peered":"2024-05-27T07:37:46.321193+0000","last_unstale":"2024-05-27T07:39:44.417859+0000","last_undegraded":"2024-05-27T07:39:44.417859+0000","last_fullsized":"2024-05-27T07:39:44.417859+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:25:25.767300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133519+0000","last_change":"2024-05-27T07:36:01.949684+0000","last_active":"2024-05-27T07:39:44.133519+0000","last_peered":"2024-05-27T07:39:44.133519+0000","last_clean":"2024-05-27T07:39:44.133519+0000","last_became_active":"2024-05-27T07:36:01.949308+0000","last_became_peered":"2024-05-27T07:36:01.949308+0000","last_unstale":"2024-05-27T07:39:44.133519+0000","last_undegraded":"2024-05-27T07:39:44.133519+0000","last_fullsized":"2024-05-27T07:39:44.133519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:50:54.859172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247002+0000","last_change":"2024-05-27T07:36:04.971266+0000","last_active":"2024-05-27T07:39:44.247002+0000","last_peered":"2024-05-27T07:39:44.247002+0000","last_clean":"2024-05-27T07:39:44.247002+0000","last_became_active":"2024-05-27T07:36:04.970831+0000","last_became_peered":"2024-05-27T07:36:04.970831+0000","last_unstale":"2024-05-27T07:39:44.247002+0000","last_undegraded":"2024-05-27T07:39:44.247002+0000","last_fullsized":"2024-05-27T07:39:44.247002+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:37.237637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417864+0000","last_change":"2024-05-27T07:36:15.024415+0000","last_active":"2024-05-27T07:39:44.417864+0000","last_peered":"2024-05-27T07:39:44.417864+0000","last_clean":"2024-05-27T07:39:44.417864+0000","last_became_active":"2024-05-27T07:36:15.023935+0000","last_became_peered":"2024-05-27T07:36:15.023935+0000","last_unstale":"2024-05-27T07:39:44.417864+0000","last_undegraded":"2024-05-27T07:39:44.417864+0000","last_fullsized":"2024-05-27T07:39:44.417864+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:37:30.386933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134940+0000","last_change":"2024-05-27T07:36:01.952256+0000","last_active":"2024-05-27T07:39:44.134940+0000","last_peered":"2024-05-27T07:39:44.134940+0000","last_clean":"2024-05-27T07:39:44.134940+0000","last_became_active":"2024-05-27T07:36:01.951876+0000","last_became_peered":"2024-05-27T07:36:01.951876+0000","last_unstale":"2024-05-27T07:39:44.134940+0000","last_undegraded":"2024-05-27T07:39:44.134940+0000","last_fullsized":"2024-05-27T07:39:44.134940+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:05.201780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.243724+0000","last_change":"2024-05-27T07:36:04.980300+0000","last_active":"2024-05-27T07:39:44.243724+0000","last_peered":"2024-05-27T07:39:44.243724+0000","last_clean":"2024-05-27T07:39:44.243724+0000","last_became_active":"2024-05-27T07:36:04.979954+0000","last_became_peered":"2024-05-27T07:36:04.979954+0000","last_unstale":"2024-05-27T07:39:44.243724+0000","last_undegraded":"2024-05-27T07:39:44.243724+0000","last_fullsized":"2024-05-27T07:39:44.243724+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:15.712773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247102+0000","last_change":"2024-05-27T07:36:15.026685+0000","last_active":"2024-05-27T07:39:44.247102+0000","last_peered":"2024-05-27T07:39:44.247102+0000","last_clean":"2024-05-27T07:39:44.247102+0000","last_became_active":"2024-05-27T07:36:15.026287+0000","last_became_peered":"2024-05-27T07:36:15.026287+0000","last_unstale":"2024-05-27T07:39:44.247102+0000","last_undegraded":"2024-05-27T07:39:44.247102+0000","last_fullsized":"2024-05-27T07:39:44.247102+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:29:26.394435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134169+0000","last_change":"2024-05-27T07:36:01.956566+0000","last_active":"2024-05-27T07:39:44.134169+0000","last_peered":"2024-05-27T07:39:44.134169+0000","last_clean":"2024-05-27T07:39:44.134169+0000","last_became_active":"2024-05-27T07:36:01.955963+0000","last_became_peered":"2024-05-27T07:36:01.955963+0000","last_unstale":"2024-05-27T07:39:44.134169+0000","last_undegraded":"2024-05-27T07:39:44.134169+0000","last_fullsized":"2024-05-27T07:39:44.134169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:53.081784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418762+0000","last_change":"2024-05-27T07:36:04.983878+0000","last_active":"2024-05-27T07:39:44.418762+0000","last_peered":"2024-05-27T07:39:44.418762+0000","last_clean":"2024-05-27T07:39:44.418762+0000","last_became_active":"2024-05-27T07:36:04.983339+0000","last_became_peered":"2024-05-27T07:36:04.983339+0000","last_unstale":"2024-05-27T07:39:44.418762+0000","last_undegraded":"2024-05-27T07:39:44.418762+0000","last_fullsized":"2024-05-27T07:39:44.418762+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:57:01.726122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.243682+0000","last_change":"2024-05-27T07:36:15.027923+0000","last_active":"2024-05-27T07:39:44.243682+0000","last_peered":"2024-05-27T07:39:44.243682+0000","last_clean":"2024-05-27T07:39:44.243682+0000","last_became_active":"2024-05-27T07:36:15.026012+0000","last_became_peered":"2024-05-27T07:36:15.026012+0000","last_unstale":"2024-05-27T07:39:44.243682+0000","last_undegraded":"2024-05-27T07:39:44.243682+0000","last_fullsized":"2024-05-27T07:39:44.243682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:27:45.238053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419164+0000","last_change":"2024-05-27T07:36:01.956038+0000","last_active":"2024-05-27T07:39:44.419164+0000","last_peered":"2024-05-27T07:39:44.419164+0000","last_clean":"2024-05-27T07:39:44.419164+0000","last_became_active":"2024-05-27T07:36:01.955639+0000","last_became_peered":"2024-05-27T07:36:01.955639+0000","last_unstale":"2024-05-27T07:39:44.419164+0000","last_undegraded":"2024-05-27T07:39:44.419164+0000","last_fullsized":"2024-05-27T07:39:44.419164+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:32.920194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417936+0000","last_change":"2024-05-27T07:36:04.986676+0000","last_active":"2024-05-27T07:39:44.417936+0000","last_peered":"2024-05-27T07:39:44.417936+0000","last_clean":"2024-05-27T07:39:44.417936+0000","last_became_active":"2024-05-27T07:36:04.986316+0000","last_became_peered":"2024-05-27T07:36:04.986316+0000","last_unstale":"2024-05-27T07:39:44.417936+0000","last_undegraded":"2024-05-27T07:39:44.417936+0000","last_fullsized":"2024-05-27T07:39:44.417936+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:03:50.947781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417814+0000","last_change":"2024-05-27T07:36:15.027643+0000","last_active":"2024-05-27T07:39:44.417814+0000","last_peered":"2024-05-27T07:39:44.417814+0000","last_clean":"2024-05-27T07:39:44.417814+0000","last_became_active":"2024-05-27T07:36:15.027175+0000","last_became_peered":"2024-05-27T07:36:15.027175+0000","last_unstale":"2024-05-27T07:39:44.417814+0000","last_undegraded":"2024-05-27T07:39:44.417814+0000","last_fullsized":"2024-05-27T07:39:44.417814+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:46:32.764271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"78'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135540+0000","last_change":"2024-05-27T07:36:02.245746+0000","last_active":"2024-05-27T07:39:44.135540+0000","last_peered":"2024-05-27T07:39:44.135540+0000","last_clean":"2024-05-27T07:39:44.135540+0000","last_became_active":"2024-05-27T07:36:02.245269+0000","last_became_peered":"2024-05-27T07:36:02.245269+0000","last_unstale":"2024-05-27T07:39:44.135540+0000","last_undegraded":"2024-05-27T07:39:44.135540+0000","last_fullsized":"2024-05-27T07:39:44.135540+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:28.202540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418801+0000","last_change":"2024-05-27T07:36:45.129374+0000","last_active":"2024-05-27T07:39:44.418801+0000","last_peered":"2024-05-27T07:39:44.418801+0000","last_clean":"2024-05-27T07:39:44.418801+0000","last_became_active":"2024-05-27T07:36:45.128956+0000","last_became_peered":"2024-05-27T07:36:45.128956+0000","last_unstale":"2024-05-27T07:39:44.418801+0000","last_undegraded":"2024-05-27T07:39:44.418801+0000","last_fullsized":"2024-05-27T07:39:44.418801+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:17.037598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244073+0000","last_change":"2024-05-27T07:36:15.027097+0000","last_active":"2024-05-27T07:39:44.244073+0000","last_peered":"2024-05-27T07:39:44.244073+0000","last_clean":"2024-05-27T07:39:44.244073+0000","last_became_active":"2024-05-27T07:36:15.026704+0000","last_became_peered":"2024-05-27T07:36:15.026704+0000","last_unstale":"2024-05-27T07:39:44.244073+0000","last_undegraded":"2024-05-27T07:39:44.244073+0000","last_fullsized":"2024-05-27T07:39:44.244073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:35:03.747199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'39","reported_seq":423,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.714753+0000","last_change":"2024-05-27T07:36:01.956926+0000","last_active":"2024-05-27T07:39:44.714753+0000","last_peered":"2024-05-27T07:39:44.714753+0000","last_clean":"2024-05-27T07:39:44.714753+0000","last_became_active":"2024-05-27T07:36:01.956515+0000","last_became_peered":"2024-05-27T07:36:01.956515+0000","last_unstale":"2024-05-27T07:39:44.714753+0000","last_undegraded":"2024-05-27T07:39:44.714753+0000","last_fullsized":"2024-05-27T07:39:44.714753+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:46.566489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71357,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":337,"num_read_kb":239,"num_write":39,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246955+0000","last_change":"2024-05-27T07:36:04.976498+0000","last_active":"2024-05-27T07:39:44.246955+0000","last_peered":"2024-05-27T07:39:44.246955+0000","last_clean":"2024-05-27T07:39:44.246955+0000","last_became_active":"2024-05-27T07:36:04.976020+0000","last_became_peered":"2024-05-27T07:36:04.976020+0000","last_unstale":"2024-05-27T07:39:44.246955+0000","last_undegraded":"2024-05-27T07:39:44.246955+0000","last_fullsized":"2024-05-27T07:39:44.246955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:25.909259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135913+0000","last_change":"2024-05-27T07:36:15.022700+0000","last_active":"2024-05-27T07:39:44.135913+0000","last_peered":"2024-05-27T07:39:44.135913+0000","last_clean":"2024-05-27T07:39:44.135913+0000","last_became_active":"2024-05-27T07:36:15.022109+0000","last_became_peered":"2024-05-27T07:36:15.022109+0000","last_unstale":"2024-05-27T07:39:44.135913+0000","last_undegraded":"2024-05-27T07:39:44.135913+0000","last_fullsized":"2024-05-27T07:39:44.135913+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:02:43.498049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"78'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363210+0000","last_change":"2024-05-27T07:36:01.953442+0000","last_active":"2024-05-27T07:39:44.363210+0000","last_peered":"2024-05-27T07:39:44.363210+0000","last_clean":"2024-05-27T07:39:44.363210+0000","last_became_active":"2024-05-27T07:36:01.953051+0000","last_became_peered":"2024-05-27T07:36:01.953051+0000","last_unstale":"2024-05-27T07:39:44.363210+0000","last_undegraded":"2024-05-27T07:39:44.363210+0000","last_fullsized":"2024-05-27T07:39:44.363210+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:47:53.070596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133518+0000","last_change":"2024-05-27T07:36:04.976365+0000","last_active":"2024-05-27T07:39:44.133518+0000","last_peered":"2024-05-27T07:39:44.133518+0000","last_clean":"2024-05-27T07:39:44.133518+0000","last_became_active":"2024-05-27T07:36:04.976050+0000","last_became_peered":"2024-05-27T07:36:04.976050+0000","last_unstale":"2024-05-27T07:39:44.133518+0000","last_undegraded":"2024-05-27T07:39:44.133518+0000","last_fullsized":"2024-05-27T07:39:44.133518+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:19.765746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417617+0000","last_change":"2024-05-27T07:36:15.022561+0000","last_active":"2024-05-27T07:39:44.417617+0000","last_peered":"2024-05-27T07:39:44.417617+0000","last_clean":"2024-05-27T07:39:44.417617+0000","last_became_active":"2024-05-27T07:36:15.021974+0000","last_became_peered":"2024-05-27T07:36:15.021974+0000","last_unstale":"2024-05-27T07:39:44.417617+0000","last_undegraded":"2024-05-27T07:39:44.417617+0000","last_fullsized":"2024-05-27T07:39:44.417617+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:31.520969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135318+0000","last_change":"2024-05-27T07:36:01.952625+0000","last_active":"2024-05-27T07:39:44.135318+0000","last_peered":"2024-05-27T07:39:44.135318+0000","last_clean":"2024-05-27T07:39:44.135318+0000","last_became_active":"2024-05-27T07:36:01.952280+0000","last_became_peered":"2024-05-27T07:36:01.952280+0000","last_unstale":"2024-05-27T07:39:44.135318+0000","last_undegraded":"2024-05-27T07:39:44.135318+0000","last_fullsized":"2024-05-27T07:39:44.135318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:17.548061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418259+0000","last_change":"2024-05-27T07:36:04.984795+0000","last_active":"2024-05-27T07:39:44.418259+0000","last_peered":"2024-05-27T07:39:44.418259+0000","last_clean":"2024-05-27T07:39:44.418259+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:39:44.418259+0000","last_undegraded":"2024-05-27T07:39:44.418259+0000","last_fullsized":"2024-05-27T07:39:44.418259+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:07.207460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244434+0000","last_change":"2024-05-27T07:36:15.033565+0000","last_active":"2024-05-27T07:39:44.244434+0000","last_peered":"2024-05-27T07:39:44.244434+0000","last_clean":"2024-05-27T07:39:44.244434+0000","last_became_active":"2024-05-27T07:36:15.033145+0000","last_became_peered":"2024-05-27T07:36:15.033145+0000","last_unstale":"2024-05-27T07:39:44.244434+0000","last_undegraded":"2024-05-27T07:39:44.244434+0000","last_fullsized":"2024-05-27T07:39:44.244434+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:38:02.163816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417353+0000","last_change":"2024-05-27T07:36:01.952759+0000","last_active":"2024-05-27T07:39:44.417353+0000","last_peered":"2024-05-27T07:39:44.417353+0000","last_clean":"2024-05-27T07:39:44.417353+0000","last_became_active":"2024-05-27T07:36:01.952339+0000","last_became_peered":"2024-05-27T07:36:01.952339+0000","last_unstale":"2024-05-27T07:39:44.417353+0000","last_undegraded":"2024-05-27T07:39:44.417353+0000","last_fullsized":"2024-05-27T07:39:44.417353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:14.518123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417694+0000","last_change":"2024-05-27T07:36:04.986312+0000","last_active":"2024-05-27T07:39:44.417694+0000","last_peered":"2024-05-27T07:39:44.417694+0000","last_clean":"2024-05-27T07:39:44.417694+0000","last_became_active":"2024-05-27T07:36:04.985944+0000","last_became_peered":"2024-05-27T07:36:04.985944+0000","last_unstale":"2024-05-27T07:39:44.417694+0000","last_undegraded":"2024-05-27T07:39:44.417694+0000","last_fullsized":"2024-05-27T07:39:44.417694+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:04.955449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133923+0000","last_change":"2024-05-27T07:36:15.020930+0000","last_active":"2024-05-27T07:39:44.133923+0000","last_peered":"2024-05-27T07:39:44.133923+0000","last_clean":"2024-05-27T07:39:44.133923+0000","last_became_active":"2024-05-27T07:36:15.020556+0000","last_became_peered":"2024-05-27T07:36:15.020556+0000","last_unstale":"2024-05-27T07:39:44.133923+0000","last_undegraded":"2024-05-27T07:39:44.133923+0000","last_fullsized":"2024-05-27T07:39:44.133923+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:02.673065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"78'8","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246644+0000","last_change":"2024-05-27T07:36:02.243402+0000","last_active":"2024-05-27T07:39:44.246644+0000","last_peered":"2024-05-27T07:39:44.246644+0000","last_clean":"2024-05-27T07:39:44.246644+0000","last_became_active":"2024-05-27T07:36:02.242978+0000","last_became_peered":"2024-05-27T07:36:02.242978+0000","last_unstale":"2024-05-27T07:39:44.246644+0000","last_undegraded":"2024-05-27T07:39:44.246644+0000","last_fullsized":"2024-05-27T07:39:44.246644+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:25:04.289506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135103+0000","last_change":"2024-05-27T07:36:04.989885+0000","last_active":"2024-05-27T07:39:44.135103+0000","last_peered":"2024-05-27T07:39:44.135103+0000","last_clean":"2024-05-27T07:39:44.135103+0000","last_became_active":"2024-05-27T07:36:04.989394+0000","last_became_peered":"2024-05-27T07:36:04.989394+0000","last_unstale":"2024-05-27T07:39:44.135103+0000","last_undegraded":"2024-05-27T07:39:44.135103+0000","last_fullsized":"2024-05-27T07:39:44.135103+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:54.138286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135101+0000","last_change":"2024-05-27T07:36:15.021772+0000","last_active":"2024-05-27T07:39:44.135101+0000","last_peered":"2024-05-27T07:39:44.135101+0000","last_clean":"2024-05-27T07:39:44.135101+0000","last_became_active":"2024-05-27T07:36:15.021352+0000","last_became_peered":"2024-05-27T07:36:15.021352+0000","last_unstale":"2024-05-27T07:39:44.135101+0000","last_undegraded":"2024-05-27T07:39:44.135101+0000","last_fullsized":"2024-05-27T07:39:44.135101+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:23.066662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417811+0000","last_change":"2024-05-27T07:36:45.131900+0000","last_active":"2024-05-27T07:39:44.417811+0000","last_peered":"2024-05-27T07:39:44.417811+0000","last_clean":"2024-05-27T07:39:44.417811+0000","last_became_active":"2024-05-27T07:36:45.127340+0000","last_became_peered":"2024-05-27T07:36:45.127340+0000","last_unstale":"2024-05-27T07:39:44.417811+0000","last_undegraded":"2024-05-27T07:39:44.417811+0000","last_fullsized":"2024-05-27T07:39:44.417811+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:07:57.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417805+0000","last_change":"2024-05-27T07:36:04.987671+0000","last_active":"2024-05-27T07:39:44.417805+0000","last_peered":"2024-05-27T07:39:44.417805+0000","last_clean":"2024-05-27T07:39:44.417805+0000","last_became_active":"2024-05-27T07:36:04.987317+0000","last_became_peered":"2024-05-27T07:36:04.987317+0000","last_unstale":"2024-05-27T07:39:44.417805+0000","last_undegraded":"2024-05-27T07:39:44.417805+0000","last_fullsized":"2024-05-27T07:39:44.417805+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:35.985968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246611+0000","last_change":"2024-05-27T07:36:15.019967+0000","last_active":"2024-05-27T07:39:44.246611+0000","last_peered":"2024-05-27T07:39:44.246611+0000","last_clean":"2024-05-27T07:39:44.246611+0000","last_became_active":"2024-05-27T07:36:15.019509+0000","last_became_peered":"2024-05-27T07:36:15.019509+0000","last_unstale":"2024-05-27T07:39:44.246611+0000","last_undegraded":"2024-05-27T07:39:44.246611+0000","last_fullsized":"2024-05-27T07:39:44.246611+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:01:56.137335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244238+0000","last_change":"2024-05-27T07:36:02.243561+0000","last_active":"2024-05-27T07:39:44.244238+0000","last_peered":"2024-05-27T07:39:44.244238+0000","last_clean":"2024-05-27T07:39:44.244238+0000","last_became_active":"2024-05-27T07:36:02.242911+0000","last_became_peered":"2024-05-27T07:36:02.242911+0000","last_unstale":"2024-05-27T07:39:44.244238+0000","last_undegraded":"2024-05-27T07:39:44.244238+0000","last_fullsized":"2024-05-27T07:39:44.244238+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:01:43.915660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133619+0000","last_change":"2024-05-27T07:36:04.971108+0000","last_active":"2024-05-27T07:39:44.133619+0000","last_peered":"2024-05-27T07:39:44.133619+0000","last_clean":"2024-05-27T07:39:44.133619+0000","last_became_active":"2024-05-27T07:36:04.970682+0000","last_became_peered":"2024-05-27T07:36:04.970682+0000","last_unstale":"2024-05-27T07:39:44.133619+0000","last_undegraded":"2024-05-27T07:39:44.133619+0000","last_fullsized":"2024-05-27T07:39:44.133619+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:15:34.940449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133949+0000","last_change":"2024-05-27T07:36:15.021292+0000","last_active":"2024-05-27T07:39:44.133949+0000","last_peered":"2024-05-27T07:39:44.133949+0000","last_clean":"2024-05-27T07:39:44.133949+0000","last_became_active":"2024-05-27T07:36:15.019988+0000","last_became_peered":"2024-05-27T07:36:15.019988+0000","last_unstale":"2024-05-27T07:39:44.133949+0000","last_undegraded":"2024-05-27T07:39:44.133949+0000","last_fullsized":"2024-05-27T07:39:44.133949+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:34:14.325368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417857+0000","last_change":"2024-05-27T07:36:01.952231+0000","last_active":"2024-05-27T07:39:44.417857+0000","last_peered":"2024-05-27T07:39:44.417857+0000","last_clean":"2024-05-27T07:39:44.417857+0000","last_became_active":"2024-05-27T07:36:01.951818+0000","last_became_peered":"2024-05-27T07:36:01.951818+0000","last_unstale":"2024-05-27T07:39:44.417857+0000","last_undegraded":"2024-05-27T07:39:44.417857+0000","last_fullsized":"2024-05-27T07:39:44.417857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:52:16.299204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133412+0000","last_change":"2024-05-27T07:36:04.982761+0000","last_active":"2024-05-27T07:39:44.133412+0000","last_peered":"2024-05-27T07:39:44.133412+0000","last_clean":"2024-05-27T07:39:44.133412+0000","last_became_active":"2024-05-27T07:36:04.982303+0000","last_became_peered":"2024-05-27T07:36:04.982303+0000","last_unstale":"2024-05-27T07:39:44.133412+0000","last_undegraded":"2024-05-27T07:39:44.133412+0000","last_fullsized":"2024-05-27T07:39:44.133412+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:13:19.099227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133814+0000","last_change":"2024-05-27T07:37:46.323619+0000","last_active":"2024-05-27T07:39:44.133814+0000","last_peered":"2024-05-27T07:39:44.133814+0000","last_clean":"2024-05-27T07:39:44.133814+0000","last_became_active":"2024-05-27T07:37:46.322114+0000","last_became_peered":"2024-05-27T07:37:46.322114+0000","last_unstale":"2024-05-27T07:39:44.133814+0000","last_undegraded":"2024-05-27T07:39:44.133814+0000","last_fullsized":"2024-05-27T07:39:44.133814+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:59:08.966863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135176+0000","last_change":"2024-05-27T07:36:45.142023+0000","last_active":"2024-05-27T07:39:44.135176+0000","last_peered":"2024-05-27T07:39:44.135176+0000","last_clean":"2024-05-27T07:39:44.135176+0000","last_became_active":"2024-05-27T07:36:45.141278+0000","last_became_peered":"2024-05-27T07:36:45.141278+0000","last_unstale":"2024-05-27T07:39:44.135176+0000","last_undegraded":"2024-05-27T07:39:44.135176+0000","last_fullsized":"2024-05-27T07:39:44.135176+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:15.800380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133391+0000","last_change":"2024-05-27T07:36:02.243041+0000","last_active":"2024-05-27T07:39:44.133391+0000","last_peered":"2024-05-27T07:39:44.133391+0000","last_clean":"2024-05-27T07:39:44.133391+0000","last_became_active":"2024-05-27T07:36:02.242689+0000","last_became_peered":"2024-05-27T07:36:02.242689+0000","last_unstale":"2024-05-27T07:39:44.133391+0000","last_undegraded":"2024-05-27T07:39:44.133391+0000","last_fullsized":"2024-05-27T07:39:44.133391+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:44:19.652954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418822+0000","last_change":"2024-05-27T07:36:15.025595+0000","last_active":"2024-05-27T07:39:44.418822+0000","last_peered":"2024-05-27T07:39:44.418822+0000","last_clean":"2024-05-27T07:39:44.418822+0000","last_became_active":"2024-05-27T07:36:15.025177+0000","last_became_peered":"2024-05-27T07:36:15.025177+0000","last_unstale":"2024-05-27T07:39:44.418822+0000","last_undegraded":"2024-05-27T07:39:44.418822+0000","last_fullsized":"2024-05-27T07:39:44.418822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:08.920778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247262+0000","last_change":"2024-05-27T07:36:04.981422+0000","last_active":"2024-05-27T07:39:44.247262+0000","last_peered":"2024-05-27T07:39:44.247262+0000","last_clean":"2024-05-27T07:39:44.247262+0000","last_became_active":"2024-05-27T07:36:04.981100+0000","last_became_peered":"2024-05-27T07:36:04.981100+0000","last_unstale":"2024-05-27T07:39:44.247262+0000","last_undegraded":"2024-05-27T07:39:44.247262+0000","last_fullsized":"2024-05-27T07:39:44.247262+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:01.076309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417708+0000","last_change":"2024-05-27T07:36:01.953727+0000","last_active":"2024-05-27T07:39:44.417708+0000","last_peered":"2024-05-27T07:39:44.417708+0000","last_clean":"2024-05-27T07:39:44.417708+0000","last_became_active":"2024-05-27T07:36:01.953323+0000","last_became_peered":"2024-05-27T07:36:01.953323+0000","last_unstale":"2024-05-27T07:39:44.417708+0000","last_undegraded":"2024-05-27T07:39:44.417708+0000","last_fullsized":"2024-05-27T07:39:44.417708+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:17.186854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417734+0000","last_change":"2024-05-27T07:34:38.289913+0000","last_active":"2024-05-27T07:39:44.417734+0000","last_peered":"2024-05-27T07:39:44.417734+0000","last_clean":"2024-05-27T07:39:44.417734+0000","last_became_active":"2024-05-27T07:34:38.285523+0000","last_became_peered":"2024-05-27T07:34:38.285523+0000","last_unstale":"2024-05-27T07:39:44.417734+0000","last_undegraded":"2024-05-27T07:39:44.417734+0000","last_fullsized":"2024-05-27T07:39:44.417734+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_clean_scrub_stamp":"2024-05-27T07:31:09.455079+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:59.537055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135577+0000","last_change":"2024-05-27T07:36:15.027453+0000","last_active":"2024-05-27T07:39:44.135577+0000","last_peered":"2024-05-27T07:39:44.135577+0000","last_clean":"2024-05-27T07:39:44.135577+0000","last_became_active":"2024-05-27T07:36:15.026872+0000","last_became_peered":"2024-05-27T07:36:15.026872+0000","last_unstale":"2024-05-27T07:39:44.135577+0000","last_undegraded":"2024-05-27T07:39:44.135577+0000","last_fullsized":"2024-05-27T07:39:44.135577+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:32.351924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133387+0000","last_change":"2024-05-27T07:36:04.982041+0000","last_active":"2024-05-27T07:39:44.133387+0000","last_peered":"2024-05-27T07:39:44.133387+0000","last_clean":"2024-05-27T07:39:44.133387+0000","last_became_active":"2024-05-27T07:36:04.981719+0000","last_became_peered":"2024-05-27T07:36:04.981719+0000","last_unstale":"2024-05-27T07:39:44.133387+0000","last_undegraded":"2024-05-27T07:39:44.133387+0000","last_fullsized":"2024-05-27T07:39:44.133387+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:40.061892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417569+0000","last_change":"2024-05-27T07:36:01.952238+0000","last_active":"2024-05-27T07:39:44.417569+0000","last_peered":"2024-05-27T07:39:44.417569+0000","last_clean":"2024-05-27T07:39:44.417569+0000","last_became_active":"2024-05-27T07:36:01.951853+0000","last_became_peered":"2024-05-27T07:36:01.951853+0000","last_unstale":"2024-05-27T07:39:44.417569+0000","last_undegraded":"2024-05-27T07:39:44.417569+0000","last_fullsized":"2024-05-27T07:39:44.417569+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:46:41.317531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417616+0000","last_change":"2024-05-27T07:36:15.021848+0000","last_active":"2024-05-27T07:39:44.417616+0000","last_peered":"2024-05-27T07:39:44.417616+0000","last_clean":"2024-05-27T07:39:44.417616+0000","last_became_active":"2024-05-27T07:36:15.021371+0000","last_became_peered":"2024-05-27T07:36:15.021371+0000","last_unstale":"2024-05-27T07:39:44.417616+0000","last_undegraded":"2024-05-27T07:39:44.417616+0000","last_fullsized":"2024-05-27T07:39:44.417616+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:32:21.741229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246986+0000","last_change":"2024-05-27T07:36:02.244283+0000","last_active":"2024-05-27T07:39:44.246986+0000","last_peered":"2024-05-27T07:39:44.246986+0000","last_clean":"2024-05-27T07:39:44.246986+0000","last_became_active":"2024-05-27T07:36:02.243785+0000","last_became_peered":"2024-05-27T07:36:02.243785+0000","last_unstale":"2024-05-27T07:39:44.246986+0000","last_undegraded":"2024-05-27T07:39:44.246986+0000","last_fullsized":"2024-05-27T07:39:44.246986+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:38.596510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133682+0000","last_change":"2024-05-27T07:36:04.979903+0000","last_active":"2024-05-27T07:39:44.133682+0000","last_peered":"2024-05-27T07:39:44.133682+0000","last_clean":"2024-05-27T07:39:44.133682+0000","last_became_active":"2024-05-27T07:36:04.979373+0000","last_became_peered":"2024-05-27T07:36:04.979373+0000","last_unstale":"2024-05-27T07:39:44.133682+0000","last_undegraded":"2024-05-27T07:39:44.133682+0000","last_fullsized":"2024-05-27T07:39:44.133682+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:44:33.763230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246825+0000","last_change":"2024-05-27T07:36:15.023740+0000","last_active":"2024-05-27T07:39:44.246825+0000","last_peered":"2024-05-27T07:39:44.246825+0000","last_clean":"2024-05-27T07:39:44.246825+0000","last_became_active":"2024-05-27T07:36:15.023364+0000","last_became_peered":"2024-05-27T07:36:15.023364+0000","last_unstale":"2024-05-27T07:39:44.246825+0000","last_undegraded":"2024-05-27T07:39:44.246825+0000","last_fullsized":"2024-05-27T07:39:44.246825+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:21.399065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'46","reported_seq":116,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.372049+0000","last_change":"2024-05-27T07:36:01.951451+0000","last_active":"2024-05-27T07:39:44.372049+0000","last_peered":"2024-05-27T07:39:44.372049+0000","last_clean":"2024-05-27T07:39:44.372049+0000","last_became_active":"2024-05-27T07:36:01.951037+0000","last_became_peered":"2024-05-27T07:36:01.951037+0000","last_unstale":"2024-05-27T07:39:44.372049+0000","last_undegraded":"2024-05-27T07:39:44.372049+0000","last_fullsized":"2024-05-27T07:39:44.372049+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:44.202474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.248504+0000","last_change":"2024-05-27T07:36:04.975392+0000","last_active":"2024-05-27T07:39:44.248504+0000","last_peered":"2024-05-27T07:39:44.248504+0000","last_clean":"2024-05-27T07:39:44.248504+0000","last_became_active":"2024-05-27T07:36:04.975029+0000","last_became_peered":"2024-05-27T07:36:04.975029+0000","last_unstale":"2024-05-27T07:39:44.248504+0000","last_undegraded":"2024-05-27T07:39:44.248504+0000","last_fullsized":"2024-05-27T07:39:44.248504+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:32.022670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363000+0000","last_change":"2024-05-27T07:36:15.026401+0000","last_active":"2024-05-27T07:39:44.363000+0000","last_peered":"2024-05-27T07:39:44.363000+0000","last_clean":"2024-05-27T07:39:44.363000+0000","last_became_active":"2024-05-27T07:36:15.025963+0000","last_became_peered":"2024-05-27T07:36:15.025963+0000","last_unstale":"2024-05-27T07:39:44.363000+0000","last_undegraded":"2024-05-27T07:39:44.363000+0000","last_fullsized":"2024-05-27T07:39:44.363000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:33:47.616099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"78'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418346+0000","last_change":"2024-05-27T07:36:02.228776+0000","last_active":"2024-05-27T07:39:44.418346+0000","last_peered":"2024-05-27T07:39:44.418346+0000","last_clean":"2024-05-27T07:39:44.418346+0000","last_became_active":"2024-05-27T07:36:02.228356+0000","last_became_peered":"2024-05-27T07:36:02.228356+0000","last_unstale":"2024-05-27T07:39:44.418346+0000","last_undegraded":"2024-05-27T07:39:44.418346+0000","last_fullsized":"2024-05-27T07:39:44.418346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:07.289340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418083+0000","last_change":"2024-05-27T07:36:04.986917+0000","last_active":"2024-05-27T07:39:44.418083+0000","last_peered":"2024-05-27T07:39:44.418083+0000","last_clean":"2024-05-27T07:39:44.418083+0000","last_became_active":"2024-05-27T07:36:04.986432+0000","last_became_peered":"2024-05-27T07:36:04.986432+0000","last_unstale":"2024-05-27T07:39:44.418083+0000","last_undegraded":"2024-05-27T07:39:44.418083+0000","last_fullsized":"2024-05-27T07:39:44.418083+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:39:15.011058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418062+0000","last_change":"2024-05-27T07:36:15.032312+0000","last_active":"2024-05-27T07:39:44.418062+0000","last_peered":"2024-05-27T07:39:44.418062+0000","last_clean":"2024-05-27T07:39:44.418062+0000","last_became_active":"2024-05-27T07:36:15.031475+0000","last_became_peered":"2024-05-27T07:36:15.031475+0000","last_unstale":"2024-05-27T07:39:44.418062+0000","last_undegraded":"2024-05-27T07:39:44.418062+0000","last_fullsized":"2024-05-27T07:39:44.418062+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:40.099421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418204+0000","last_change":"2024-05-27T07:36:02.243456+0000","last_active":"2024-05-27T07:39:44.418204+0000","last_peered":"2024-05-27T07:39:44.418204+0000","last_clean":"2024-05-27T07:39:44.418204+0000","last_became_active":"2024-05-27T07:36:02.243164+0000","last_became_peered":"2024-05-27T07:36:02.243164+0000","last_unstale":"2024-05-27T07:39:44.418204+0000","last_undegraded":"2024-05-27T07:39:44.418204+0000","last_fullsized":"2024-05-27T07:39:44.418204+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:33.765834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246597+0000","last_change":"2024-05-27T07:36:04.980192+0000","last_active":"2024-05-27T07:39:44.246597+0000","last_peered":"2024-05-27T07:39:44.246597+0000","last_clean":"2024-05-27T07:39:44.246597+0000","last_became_active":"2024-05-27T07:36:04.979797+0000","last_became_peered":"2024-05-27T07:36:04.979797+0000","last_unstale":"2024-05-27T07:39:44.246597+0000","last_undegraded":"2024-05-27T07:39:44.246597+0000","last_fullsized":"2024-05-27T07:39:44.246597+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:47:37.006746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244501+0000","last_change":"2024-05-27T07:36:15.033153+0000","last_active":"2024-05-27T07:39:44.244501+0000","last_peered":"2024-05-27T07:39:44.244501+0000","last_clean":"2024-05-27T07:39:44.244501+0000","last_became_active":"2024-05-27T07:36:15.032783+0000","last_became_peered":"2024-05-27T07:36:15.032783+0000","last_unstale":"2024-05-27T07:39:44.244501+0000","last_undegraded":"2024-05-27T07:39:44.244501+0000","last_fullsized":"2024-05-27T07:39:44.244501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:31:26.237222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247148+0000","last_change":"2024-05-27T07:36:01.952754+0000","last_active":"2024-05-27T07:39:44.247148+0000","last_peered":"2024-05-27T07:39:44.247148+0000","last_clean":"2024-05-27T07:39:44.247148+0000","last_became_active":"2024-05-27T07:36:01.952397+0000","last_became_peered":"2024-05-27T07:36:01.952397+0000","last_unstale":"2024-05-27T07:39:44.247148+0000","last_undegraded":"2024-05-27T07:39:44.247148+0000","last_fullsized":"2024-05-27T07:39:44.247148+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:04.350169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417457+0000","last_change":"2024-05-27T07:36:04.979193+0000","last_active":"2024-05-27T07:39:44.417457+0000","last_peered":"2024-05-27T07:39:44.417457+0000","last_clean":"2024-05-27T07:39:44.417457+0000","last_became_active":"2024-05-27T07:36:04.978679+0000","last_became_peered":"2024-05-27T07:36:04.978679+0000","last_unstale":"2024-05-27T07:39:44.417457+0000","last_undegraded":"2024-05-27T07:39:44.417457+0000","last_fullsized":"2024-05-27T07:39:44.417457+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:59.310284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418066+0000","last_change":"2024-05-27T07:37:46.153707+0000","last_active":"2024-05-27T07:39:44.418066+0000","last_peered":"2024-05-27T07:39:44.418066+0000","last_clean":"2024-05-27T07:39:44.418066+0000","last_became_active":"2024-05-27T07:37:46.148703+0000","last_became_peered":"2024-05-27T07:37:46.148703+0000","last_unstale":"2024-05-27T07:39:44.418066+0000","last_undegraded":"2024-05-27T07:39:44.418066+0000","last_fullsized":"2024-05-27T07:39:44.418066+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:50:20.099467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417476+0000","last_change":"2024-05-27T07:36:45.140940+0000","last_active":"2024-05-27T07:39:44.417476+0000","last_peered":"2024-05-27T07:39:44.417476+0000","last_clean":"2024-05-27T07:39:44.417476+0000","last_became_active":"2024-05-27T07:36:45.140511+0000","last_became_peered":"2024-05-27T07:36:45.140511+0000","last_unstale":"2024-05-27T07:39:44.417476+0000","last_undegraded":"2024-05-27T07:39:44.417476+0000","last_fullsized":"2024-05-27T07:39:44.417476+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:43:00.507442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247777+0000","last_change":"2024-05-27T07:36:04.974709+0000","last_active":"2024-05-27T07:39:44.247777+0000","last_peered":"2024-05-27T07:39:44.247777+0000","last_clean":"2024-05-27T07:39:44.247777+0000","last_became_active":"2024-05-27T07:36:04.974147+0000","last_became_peered":"2024-05-27T07:36:04.974147+0000","last_unstale":"2024-05-27T07:39:44.247777+0000","last_undegraded":"2024-05-27T07:39:44.247777+0000","last_fullsized":"2024-05-27T07:39:44.247777+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:05:11.251124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417872+0000","last_change":"2024-05-27T07:36:15.023187+0000","last_active":"2024-05-27T07:39:44.417872+0000","last_peered":"2024-05-27T07:39:44.417872+0000","last_clean":"2024-05-27T07:39:44.417872+0000","last_became_active":"2024-05-27T07:36:15.022687+0000","last_became_peered":"2024-05-27T07:36:15.022687+0000","last_unstale":"2024-05-27T07:39:44.417872+0000","last_undegraded":"2024-05-27T07:39:44.417872+0000","last_fullsized":"2024-05-27T07:39:44.417872+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:51.713222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"78'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246746+0000","last_change":"2024-05-27T07:36:01.954751+0000","last_active":"2024-05-27T07:39:44.246746+0000","last_peered":"2024-05-27T07:39:44.246746+0000","last_clean":"2024-05-27T07:39:44.246746+0000","last_became_active":"2024-05-27T07:36:01.954173+0000","last_became_peered":"2024-05-27T07:36:01.954173+0000","last_unstale":"2024-05-27T07:39:44.246746+0000","last_undegraded":"2024-05-27T07:39:44.246746+0000","last_fullsized":"2024-05-27T07:39:44.246746+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:45:10.504504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417752+0000","last_change":"2024-05-27T07:36:04.980294+0000","last_active":"2024-05-27T07:39:44.417752+0000","last_peered":"2024-05-27T07:39:44.417752+0000","last_clean":"2024-05-27T07:39:44.417752+0000","last_became_active":"2024-05-27T07:36:04.979853+0000","last_became_peered":"2024-05-27T07:36:04.979853+0000","last_unstale":"2024-05-27T07:39:44.417752+0000","last_undegraded":"2024-05-27T07:39:44.417752+0000","last_fullsized":"2024-05-27T07:39:44.417752+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:43.131216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418970+0000","last_change":"2024-05-27T07:36:15.025854+0000","last_active":"2024-05-27T07:39:44.418970+0000","last_peered":"2024-05-27T07:39:44.418970+0000","last_clean":"2024-05-27T07:39:44.418970+0000","last_became_active":"2024-05-27T07:36:15.025412+0000","last_became_peered":"2024-05-27T07:36:15.025412+0000","last_unstale":"2024-05-27T07:39:44.418970+0000","last_undegraded":"2024-05-27T07:39:44.418970+0000","last_fullsized":"2024-05-27T07:39:44.418970+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:49.904811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363520+0000","last_change":"2024-05-27T07:36:02.243764+0000","last_active":"2024-05-27T07:39:44.363520+0000","last_peered":"2024-05-27T07:39:44.363520+0000","last_clean":"2024-05-27T07:39:44.363520+0000","last_became_active":"2024-05-27T07:36:02.242935+0000","last_became_peered":"2024-05-27T07:36:02.242935+0000","last_unstale":"2024-05-27T07:39:44.363520+0000","last_undegraded":"2024-05-27T07:39:44.363520+0000","last_fullsized":"2024-05-27T07:39:44.363520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:27.535658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419316+0000","last_change":"2024-05-27T07:36:04.975291+0000","last_active":"2024-05-27T07:39:44.419316+0000","last_peered":"2024-05-27T07:39:44.419316+0000","last_clean":"2024-05-27T07:39:44.419316+0000","last_became_active":"2024-05-27T07:36:04.974868+0000","last_became_peered":"2024-05-27T07:36:04.974868+0000","last_unstale":"2024-05-27T07:39:44.419316+0000","last_undegraded":"2024-05-27T07:39:44.419316+0000","last_fullsized":"2024-05-27T07:39:44.419316+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:13.890494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417509+0000","last_change":"2024-05-27T07:36:15.031812+0000","last_active":"2024-05-27T07:39:44.417509+0000","last_peered":"2024-05-27T07:39:44.417509+0000","last_clean":"2024-05-27T07:39:44.417509+0000","last_became_active":"2024-05-27T07:36:15.031399+0000","last_became_peered":"2024-05-27T07:36:15.031399+0000","last_unstale":"2024-05-27T07:39:44.417509+0000","last_undegraded":"2024-05-27T07:39:44.417509+0000","last_fullsized":"2024-05-27T07:39:44.417509+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.595579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.366476+0000","last_change":"2024-05-27T07:36:01.954581+0000","last_active":"2024-05-27T07:39:44.366476+0000","last_peered":"2024-05-27T07:39:44.366476+0000","last_clean":"2024-05-27T07:39:44.366476+0000","last_became_active":"2024-05-27T07:36:01.953564+0000","last_became_peered":"2024-05-27T07:36:01.953564+0000","last_unstale":"2024-05-27T07:39:44.366476+0000","last_undegraded":"2024-05-27T07:39:44.366476+0000","last_fullsized":"2024-05-27T07:39:44.366476+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:56.866738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418767+0000","last_change":"2024-05-27T07:36:45.141160+0000","last_active":"2024-05-27T07:39:44.418767+0000","last_peered":"2024-05-27T07:39:44.418767+0000","last_clean":"2024-05-27T07:39:44.418767+0000","last_became_active":"2024-05-27T07:36:45.140840+0000","last_became_peered":"2024-05-27T07:36:45.140840+0000","last_unstale":"2024-05-27T07:39:44.418767+0000","last_undegraded":"2024-05-27T07:39:44.418767+0000","last_fullsized":"2024-05-27T07:39:44.418767+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:31.676808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134842+0000","last_change":"2024-05-27T07:36:15.026483+0000","last_active":"2024-05-27T07:39:44.134842+0000","last_peered":"2024-05-27T07:39:44.134842+0000","last_clean":"2024-05-27T07:39:44.134842+0000","last_became_active":"2024-05-27T07:36:15.025955+0000","last_became_peered":"2024-05-27T07:36:15.025955+0000","last_unstale":"2024-05-27T07:39:44.134842+0000","last_undegraded":"2024-05-27T07:39:44.134842+0000","last_fullsized":"2024-05-27T07:39:44.134842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:25.468146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133958+0000","last_change":"2024-05-27T07:36:01.952505+0000","last_active":"2024-05-27T07:39:44.133958+0000","last_peered":"2024-05-27T07:39:44.133958+0000","last_clean":"2024-05-27T07:39:44.133958+0000","last_became_active":"2024-05-27T07:36:01.952088+0000","last_became_peered":"2024-05-27T07:36:01.952088+0000","last_unstale":"2024-05-27T07:39:44.133958+0000","last_undegraded":"2024-05-27T07:39:44.133958+0000","last_fullsized":"2024-05-27T07:39:44.133958+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:09.235719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419335+0000","last_change":"2024-05-27T07:36:04.974331+0000","last_active":"2024-05-27T07:39:44.419335+0000","last_peered":"2024-05-27T07:39:44.419335+0000","last_clean":"2024-05-27T07:39:44.419335+0000","last_became_active":"2024-05-27T07:36:04.973825+0000","last_became_peered":"2024-05-27T07:36:04.973825+0000","last_unstale":"2024-05-27T07:39:44.419335+0000","last_undegraded":"2024-05-27T07:39:44.419335+0000","last_fullsized":"2024-05-27T07:39:44.419335+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:39:42.917309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417542+0000","last_change":"2024-05-27T07:36:45.141224+0000","last_active":"2024-05-27T07:39:44.417542+0000","last_peered":"2024-05-27T07:39:44.417542+0000","last_clean":"2024-05-27T07:39:44.417542+0000","last_became_active":"2024-05-27T07:36:45.140776+0000","last_became_peered":"2024-05-27T07:36:45.140776+0000","last_unstale":"2024-05-27T07:39:44.417542+0000","last_undegraded":"2024-05-27T07:39:44.417542+0000","last_fullsized":"2024-05-27T07:39:44.417542+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:36:59.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363303+0000","last_change":"2024-05-27T07:36:04.988247+0000","last_active":"2024-05-27T07:39:44.363303+0000","last_peered":"2024-05-27T07:39:44.363303+0000","last_clean":"2024-05-27T07:39:44.363303+0000","last_became_active":"2024-05-27T07:36:04.987825+0000","last_became_peered":"2024-05-27T07:36:04.987825+0000","last_unstale":"2024-05-27T07:39:44.363303+0000","last_undegraded":"2024-05-27T07:39:44.363303+0000","last_fullsized":"2024-05-27T07:39:44.363303+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:32.499260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363087+0000","last_change":"2024-05-27T07:36:04.979395+0000","last_active":"2024-05-27T07:39:44.363087+0000","last_peered":"2024-05-27T07:39:44.363087+0000","last_clean":"2024-05-27T07:39:44.363087+0000","last_became_active":"2024-05-27T07:36:04.978983+0000","last_became_peered":"2024-05-27T07:36:04.978983+0000","last_unstale":"2024-05-27T07:39:44.363087+0000","last_undegraded":"2024-05-27T07:39:44.363087+0000","last_fullsized":"2024-05-27T07:39:44.363087+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:53:54.071958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417704+0000","last_change":"2024-05-27T07:36:04.989442+0000","last_active":"2024-05-27T07:39:44.417704+0000","last_peered":"2024-05-27T07:39:44.417704+0000","last_clean":"2024-05-27T07:39:44.417704+0000","last_became_active":"2024-05-27T07:36:04.989079+0000","last_became_peered":"2024-05-27T07:36:04.989079+0000","last_unstale":"2024-05-27T07:39:44.417704+0000","last_undegraded":"2024-05-27T07:39:44.417704+0000","last_fullsized":"2024-05-27T07:39:44.417704+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:39.411255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133776+0000","last_change":"2024-05-27T07:36:04.981289+0000","last_active":"2024-05-27T07:39:44.133776+0000","last_peered":"2024-05-27T07:39:44.133776+0000","last_clean":"2024-05-27T07:39:44.133776+0000","last_became_active":"2024-05-27T07:36:04.980924+0000","last_became_peered":"2024-05-27T07:36:04.980924+0000","last_unstale":"2024-05-27T07:39:44.133776+0000","last_undegraded":"2024-05-27T07:39:44.133776+0000","last_fullsized":"2024-05-27T07:39:44.133776+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:15:04.741303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417916+0000","last_change":"2024-05-27T07:36:04.985712+0000","last_active":"2024-05-27T07:39:44.417916+0000","last_peered":"2024-05-27T07:39:44.417916+0000","last_clean":"2024-05-27T07:39:44.417916+0000","last_became_active":"2024-05-27T07:36:04.983786+0000","last_became_peered":"2024-05-27T07:36:04.983786+0000","last_unstale":"2024-05-27T07:39:44.417916+0000","last_undegraded":"2024-05-27T07:39:44.417916+0000","last_fullsized":"2024-05-27T07:39:44.417916+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:34.549557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363470+0000","last_change":"2024-05-27T07:36:04.991176+0000","last_active":"2024-05-27T07:39:44.363470+0000","last_peered":"2024-05-27T07:39:44.363470+0000","last_clean":"2024-05-27T07:39:44.363470+0000","last_became_active":"2024-05-27T07:36:04.990774+0000","last_became_peered":"2024-05-27T07:36:04.990774+0000","last_unstale":"2024-05-27T07:39:44.363470+0000","last_undegraded":"2024-05-27T07:39:44.363470+0000","last_fullsized":"2024-05-27T07:39:44.363470+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:37.307581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.248721+0000","last_change":"2024-05-27T07:36:04.982337+0000","last_active":"2024-05-27T07:39:44.248721+0000","last_peered":"2024-05-27T07:39:44.248721+0000","last_clean":"2024-05-27T07:39:44.248721+0000","last_became_active":"2024-05-27T07:36:04.981988+0000","last_became_peered":"2024-05-27T07:36:04.981988+0000","last_unstale":"2024-05-27T07:39:44.248721+0000","last_undegraded":"2024-05-27T07:39:44.248721+0000","last_fullsized":"2024-05-27T07:39:44.248721+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:17:39.763459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134985+0000","last_change":"2024-05-27T07:36:04.982321+0000","last_active":"2024-05-27T07:39:44.134985+0000","last_peered":"2024-05-27T07:39:44.134985+0000","last_clean":"2024-05-27T07:39:44.134985+0000","last_became_active":"2024-05-27T07:36:04.981690+0000","last_became_peered":"2024-05-27T07:36:04.981690+0000","last_unstale":"2024-05-27T07:39:44.134985+0000","last_undegraded":"2024-05-27T07:39:44.134985+0000","last_fullsized":"2024-05-27T07:39:44.134985+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:10.181606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247215+0000","last_change":"2024-05-27T07:36:04.979411+0000","last_active":"2024-05-27T07:39:44.247215+0000","last_peered":"2024-05-27T07:39:44.247215+0000","last_clean":"2024-05-27T07:39:44.247215+0000","last_became_active":"2024-05-27T07:36:04.979041+0000","last_became_peered":"2024-05-27T07:36:04.979041+0000","last_unstale":"2024-05-27T07:39:44.247215+0000","last_undegraded":"2024-05-27T07:39:44.247215+0000","last_fullsized":"2024-05-27T07:39:44.247215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:57:29.165997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419164+0000","last_change":"2024-05-27T07:36:04.988564+0000","last_active":"2024-05-27T07:39:44.419164+0000","last_peered":"2024-05-27T07:39:44.419164+0000","last_clean":"2024-05-27T07:39:44.419164+0000","last_became_active":"2024-05-27T07:36:04.986313+0000","last_became_peered":"2024-05-27T07:36:04.986313+0000","last_unstale":"2024-05-27T07:39:44.419164+0000","last_undegraded":"2024-05-27T07:39:44.419164+0000","last_fullsized":"2024-05-27T07:39:44.419164+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:38.809386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133635+0000","last_change":"2024-05-27T07:36:04.974704+0000","last_active":"2024-05-27T07:39:44.133635+0000","last_peered":"2024-05-27T07:39:44.133635+0000","last_clean":"2024-05-27T07:39:44.133635+0000","last_became_active":"2024-05-27T07:36:04.974311+0000","last_became_peered":"2024-05-27T07:36:04.974311+0000","last_unstale":"2024-05-27T07:39:44.133635+0000","last_undegraded":"2024-05-27T07:39:44.133635+0000","last_fullsized":"2024-05-27T07:39:44.133635+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:43.021149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417756+0000","last_change":"2024-05-27T07:36:04.983951+0000","last_active":"2024-05-27T07:39:44.417756+0000","last_peered":"2024-05-27T07:39:44.417756+0000","last_clean":"2024-05-27T07:39:44.417756+0000","last_became_active":"2024-05-27T07:36:04.983489+0000","last_became_peered":"2024-05-27T07:36:04.983489+0000","last_unstale":"2024-05-27T07:39:44.417756+0000","last_undegraded":"2024-05-27T07:39:44.417756+0000","last_fullsized":"2024-05-27T07:39:44.417756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:04.229523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107193,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":596971,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":435,"num_read_kb":324,"num_write":209,"num_write_kb":1415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":1790913,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710779,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":524552,"data_compressed":35398,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877907008,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":1185667,"data_compressed":71129,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070534,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":515188,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266762,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":515188,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462991,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":1119576,"data_compressed":71129,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626518,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":520434,"data_compressed":35398,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":581085,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":520524,"data_compressed":35398,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1069056,"data_stored":591825,"data_compressed":35398,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182501,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":1045354,"data_compressed":60700,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986272,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":514628,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822746,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1069056,"data_stored":1246680,"data_compressed":77285,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":71357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":71391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":524920,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":131782,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":203151,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:39:46.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T07:39:46.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.778+0000 7fce6735e640 1 -- 172.21.15.114:0/1227135409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce60103bf0 msgr2=0x7fce601048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.778+0000 7fce6735e640 1 --2- 172.21.15.114:0/1227135409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce60103bf0 0x7fce601048f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009900 tx=0x7fce5c02a410 comp rx=0 tx=0).stop 2024-05-27T07:39:46.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.778+0000 7fce6735e640 1 -- 172.21.15.114:0/1227135409 shutdown_connections 2024-05-27T07:39:46.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.778+0000 7fce6735e640 1 --2- 172.21.15.114:0/1227135409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce60109b30 0x7fce6010bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.778+0000 7fce6735e640 1 --2- 172.21.15.114:0/1227135409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce60104e30 0x7fce60105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.778+0000 7fce6735e640 1 --2- 172.21.15.114:0/1227135409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce60103bf0 0x7fce601048f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.778+0000 7fce6735e640 1 -- 172.21.15.114:0/1227135409 >> 172.21.15.114:0/1227135409 conn(0x7fce600fede0 msgr2=0x7fce60101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:46.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.779+0000 7fce6735e640 1 -- 172.21.15.114:0/1227135409 shutdown_connections 2024-05-27T07:39:46.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.779+0000 7fce6735e640 1 -- 172.21.15.114:0/1227135409 wait complete. 2024-05-27T07:39:46.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.779+0000 7fce6735e640 1 Processor -- start 2024-05-27T07:39:46.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.779+0000 7fce6735e640 1 -- start start 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.779+0000 7fce6735e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce60103bf0 0x7fce6019a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.779+0000 7fce6735e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce60104e30 0x7fce6019a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.779+0000 7fce6735e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce60109b30 0x7fce601a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.779+0000 7fce6735e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fce6010f720 con 0x7fce60109b30 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.779+0000 7fce6735e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fce6010f5a0 con 0x7fce60103bf0 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.779+0000 7fce6735e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fce6010f8a0 con 0x7fce60104e30 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.780+0000 7fce658fb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce60109b30 0x7fce601a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.780+0000 7fce658fb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce60109b30 0x7fce601a1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43550/0 (socket says 172.21.15.114:43550) 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.780+0000 7fce658fb640 1 -- 172.21.15.114:0/3303529191 learned_addr learned my addr 172.21.15.114:0/3303529191 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.780+0000 7fce648f9640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce60104e30 0x7fce6019a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.780+0000 7fce650fa640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce60103bf0 0x7fce6019a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.780+0000 7fce658fb640 1 -- 172.21.15.114:0/3303529191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce60104e30 msgr2=0x7fce6019a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.780+0000 7fce658fb640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce60104e30 0x7fce6019a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.780+0000 7fce658fb640 1 -- 172.21.15.114:0/3303529191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce60103bf0 msgr2=0x7fce6019a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.780+0000 7fce658fb640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce60103bf0 0x7fce6019a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.780+0000 7fce658fb640 1 -- 172.21.15.114:0/3303529191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce601a1e90 con 0x7fce60109b30 2024-05-27T07:39:46.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.780+0000 7fce648f9640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce60104e30 0x7fce6019a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:46.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.780+0000 7fce658fb640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce60109b30 0x7fce601a1820 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fce5800eb10 tx=0x7fce5800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:46.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.784+0000 7fce567fc640 1 -- 172.21.15.114:0/3303529191 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58004b20 con 0x7fce60109b30 2024-05-27T07:39:46.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.784+0000 7fce6735e640 1 -- 172.21.15.114:0/3303529191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce601a2180 con 0x7fce60109b30 2024-05-27T07:39:46.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.784+0000 7fce567fc640 1 -- 172.21.15.114:0/3303529191 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce58004cc0 con 0x7fce60109b30 2024-05-27T07:39:46.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.784+0000 7fce567fc640 1 -- 172.21.15.114:0/3303529191 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58010670 con 0x7fce60109b30 2024-05-27T07:39:46.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.784+0000 7fce6735e640 1 -- 172.21.15.114:0/3303529191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce601a2680 con 0x7fce60109b30 2024-05-27T07:39:46.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.785+0000 7fce6735e640 1 -- 172.21.15.114:0/3303529191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce60108940 con 0x7fce60109b30 2024-05-27T07:39:46.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.789+0000 7fce567fc640 1 -- 172.21.15.114:0/3303529191 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 22) ==== 149773+0+0 (secure 0 0 0) 0x7fce58005000 con 0x7fce60109b30 2024-05-27T07:39:46.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.789+0000 7fce567fc640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fce4c0b2960 0x7fce4c0b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.789+0000 7fce567fc640 1 -- 172.21.15.114:0/3303529191 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fce580e3190 con 0x7fce60109b30 2024-05-27T07:39:46.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.789+0000 7fce650fa640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fce4c0b2960 0x7fce4c0b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.790+0000 7fce650fa640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fce4c0b2960 0x7fce4c0b4e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fce5c02aa00 tx=0x7fce5c0314a0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:46.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:46.795+0000 7fce567fc640 1 -- 172.21.15.114:0/3303529191 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fce58054df0 con 0x7fce60109b30 2024-05-27T07:39:46.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:46 smithi114 ceph-mon[32038]: [27/May/2024:07:39:44] ENGINE Bus STARTING 2024-05-27T07:39:46.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:46 smithi114 ceph-mon[32038]: [27/May/2024:07:39:44] ENGINE Serving on https://172.21.15.114:7150 2024-05-27T07:39:46.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:46 smithi114 ceph-mon[32038]: [27/May/2024:07:39:44] ENGINE Serving on http://172.21.15.114:8765 2024-05-27T07:39:46.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:46 smithi114 ceph-mon[32038]: [27/May/2024:07:39:44] ENGINE Bus STARTED 2024-05-27T07:39:46.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:46 smithi114 ceph-mon[32038]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:39:46.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:46 smithi163 ceph-mon[36451]: [27/May/2024:07:39:44] ENGINE Bus STARTING 2024-05-27T07:39:46.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:46 smithi163 ceph-mon[36451]: [27/May/2024:07:39:44] ENGINE Serving on https://172.21.15.114:7150 2024-05-27T07:39:46.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:46 smithi163 ceph-mon[36451]: [27/May/2024:07:39:44] ENGINE Serving on http://172.21.15.114:8765 2024-05-27T07:39:46.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:46 smithi163 ceph-mon[36451]: [27/May/2024:07:39:44] ENGINE Bus STARTED 2024-05-27T07:39:46.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:46 smithi163 ceph-mon[36451]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:39:46.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:46 smithi186 ceph-mon[36647]: [27/May/2024:07:39:44] ENGINE Bus STARTING 2024-05-27T07:39:46.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:46 smithi186 ceph-mon[36647]: [27/May/2024:07:39:44] ENGINE Serving on https://172.21.15.114:7150 2024-05-27T07:39:46.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:46 smithi186 ceph-mon[36647]: [27/May/2024:07:39:44] ENGINE Serving on http://172.21.15.114:8765 2024-05-27T07:39:46.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:46 smithi186 ceph-mon[36647]: [27/May/2024:07:39:44] ENGINE Bus STARTED 2024-05-27T07:39:46.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:46 smithi186 ceph-mon[36647]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:39:47.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.021+0000 7fce6735e640 1 -- 172.21.15.114:0/3303529191 --> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fce60063100 con 0x7fce4c0b2960 2024-05-27T07:39:47.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.031+0000 7fce567fc640 1 -- 172.21.15.114:0/3303529191 <== mgr.15138 v2:172.21.15.114:6800/3311646739 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339584 (secure 0 0 0) 0x7fce60063100 con 0x7fce4c0b2960 2024-05-27T07:39:47.033 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:47.038 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-05-27T07:39:47.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.035+0000 7fce6735e640 1 -- 172.21.15.114:0/3303529191 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fce4c0b2960 msgr2=0x7fce4c0b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:47.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.035+0000 7fce6735e640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fce4c0b2960 0x7fce4c0b4e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fce5c02aa00 tx=0x7fce5c0314a0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.035+0000 7fce6735e640 1 -- 172.21.15.114:0/3303529191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce60109b30 msgr2=0x7fce601a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:47.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.035+0000 7fce6735e640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce60109b30 0x7fce601a1820 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fce5800eb10 tx=0x7fce5800c2d0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.035+0000 7fce6735e640 1 -- 172.21.15.114:0/3303529191 shutdown_connections 2024-05-27T07:39:47.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.035+0000 7fce6735e640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fce4c0b2960 0x7fce4c0b4e70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.035+0000 7fce6735e640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce60109b30 0x7fce601a1820 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.035+0000 7fce6735e640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce60104e30 0x7fce6019a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.035+0000 7fce6735e640 1 --2- 172.21.15.114:0/3303529191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce60103bf0 0x7fce6019a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.035+0000 7fce6735e640 1 -- 172.21.15.114:0/3303529191 >> 172.21.15.114:0/3303529191 conn(0x7fce600fede0 msgr2=0x7fce60105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:47.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.035+0000 7fce6735e640 1 -- 172.21.15.114:0/3303529191 shutdown_connections 2024-05-27T07:39:47.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.035+0000 7fce6735e640 1 -- 172.21.15.114:0/3303529191 wait complete. 2024-05-27T07:39:47.048 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-05-27T07:39:46.148375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1187573,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":516,"num_read_kb":393,"num_write":334,"num_write_kb":3567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2951979,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":749,"ondisk_log_size":749,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369520,"kb_used_data":12012,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572304,"statfs":{"total":1151940427776,"available":1151562039296,"internally_reserved":0,"allocated":12300288,"data_stored":8880701,"data_compressed":551551,"data_compressed_allocated":749568,"data_compressed_original":3555328,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.993045"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418062+0000","last_change":"2024-05-27T07:36:04.980329+0000","last_active":"2024-05-27T07:39:44.418062+0000","last_peered":"2024-05-27T07:39:44.418062+0000","last_clean":"2024-05-27T07:39:44.418062+0000","last_became_active":"2024-05-27T07:36:04.980006+0000","last_became_peered":"2024-05-27T07:36:04.980006+0000","last_unstale":"2024-05-27T07:39:44.418062+0000","last_undegraded":"2024-05-27T07:39:44.418062+0000","last_fullsized":"2024-05-27T07:39:44.418062+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:51:56.728465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417978+0000","last_change":"2024-05-27T07:36:04.983429+0000","last_active":"2024-05-27T07:39:44.417978+0000","last_peered":"2024-05-27T07:39:44.417978+0000","last_clean":"2024-05-27T07:39:44.417978+0000","last_became_active":"2024-05-27T07:36:04.983099+0000","last_became_peered":"2024-05-27T07:36:04.983099+0000","last_unstale":"2024-05-27T07:39:44.417978+0000","last_undegraded":"2024-05-27T07:39:44.417978+0000","last_fullsized":"2024-05-27T07:39:44.417978+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:18:51.059717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417722+0000","last_change":"2024-05-27T07:36:04.993319+0000","last_active":"2024-05-27T07:39:44.417722+0000","last_peered":"2024-05-27T07:39:44.417722+0000","last_clean":"2024-05-27T07:39:44.417722+0000","last_became_active":"2024-05-27T07:36:04.992983+0000","last_became_peered":"2024-05-27T07:36:04.992983+0000","last_unstale":"2024-05-27T07:39:44.417722+0000","last_undegraded":"2024-05-27T07:39:44.417722+0000","last_fullsized":"2024-05-27T07:39:44.417722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:50.800194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244278+0000","last_change":"2024-05-27T07:36:45.140980+0000","last_active":"2024-05-27T07:39:44.244278+0000","last_peered":"2024-05-27T07:39:44.244278+0000","last_clean":"2024-05-27T07:39:44.244278+0000","last_became_active":"2024-05-27T07:36:45.140551+0000","last_became_peered":"2024-05-27T07:36:45.140551+0000","last_unstale":"2024-05-27T07:39:44.244278+0000","last_undegraded":"2024-05-27T07:39:44.244278+0000","last_fullsized":"2024-05-27T07:39:44.244278+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:59:21.646306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246654+0000","last_change":"2024-05-27T07:36:04.977404+0000","last_active":"2024-05-27T07:39:44.246654+0000","last_peered":"2024-05-27T07:39:44.246654+0000","last_clean":"2024-05-27T07:39:44.246654+0000","last_became_active":"2024-05-27T07:36:04.976991+0000","last_became_peered":"2024-05-27T07:36:04.976991+0000","last_unstale":"2024-05-27T07:39:44.246654+0000","last_undegraded":"2024-05-27T07:39:44.246654+0000","last_fullsized":"2024-05-27T07:39:44.246654+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:56.532586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133579+0000","last_change":"2024-05-27T07:36:04.982149+0000","last_active":"2024-05-27T07:39:44.133579+0000","last_peered":"2024-05-27T07:39:44.133579+0000","last_clean":"2024-05-27T07:39:44.133579+0000","last_became_active":"2024-05-27T07:36:04.981751+0000","last_became_peered":"2024-05-27T07:36:04.981751+0000","last_unstale":"2024-05-27T07:39:44.133579+0000","last_undegraded":"2024-05-27T07:39:44.133579+0000","last_fullsized":"2024-05-27T07:39:44.133579+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:42.773458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417510+0000","last_change":"2024-05-27T07:36:04.987491+0000","last_active":"2024-05-27T07:39:44.417510+0000","last_peered":"2024-05-27T07:39:44.417510+0000","last_clean":"2024-05-27T07:39:44.417510+0000","last_became_active":"2024-05-27T07:36:04.987059+0000","last_became_peered":"2024-05-27T07:36:04.987059+0000","last_unstale":"2024-05-27T07:39:44.417510+0000","last_undegraded":"2024-05-27T07:39:44.417510+0000","last_fullsized":"2024-05-27T07:39:44.417510+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:57.611253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.243516+0000","last_change":"2024-05-27T07:36:45.126961+0000","last_active":"2024-05-27T07:39:44.243516+0000","last_peered":"2024-05-27T07:39:44.243516+0000","last_clean":"2024-05-27T07:39:44.243516+0000","last_became_active":"2024-05-27T07:36:45.126653+0000","last_became_peered":"2024-05-27T07:36:45.126653+0000","last_unstale":"2024-05-27T07:39:44.243516+0000","last_undegraded":"2024-05-27T07:39:44.243516+0000","last_fullsized":"2024-05-27T07:39:44.243516+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:35.586128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418769+0000","last_change":"2024-05-27T07:36:04.990469+0000","last_active":"2024-05-27T07:39:44.418769+0000","last_peered":"2024-05-27T07:39:44.418769+0000","last_clean":"2024-05-27T07:39:44.418769+0000","last_became_active":"2024-05-27T07:36:04.989771+0000","last_became_peered":"2024-05-27T07:36:04.989771+0000","last_unstale":"2024-05-27T07:39:44.418769+0000","last_undegraded":"2024-05-27T07:39:44.418769+0000","last_fullsized":"2024-05-27T07:39:44.418769+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:53.593566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134126+0000","last_change":"2024-05-27T07:36:04.984746+0000","last_active":"2024-05-27T07:39:44.134126+0000","last_peered":"2024-05-27T07:39:44.134126+0000","last_clean":"2024-05-27T07:39:44.134126+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:39:44.134126+0000","last_undegraded":"2024-05-27T07:39:44.134126+0000","last_fullsized":"2024-05-27T07:39:44.134126+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:18.071981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419074+0000","last_change":"2024-05-27T07:36:04.977167+0000","last_active":"2024-05-27T07:39:44.419074+0000","last_peered":"2024-05-27T07:39:44.419074+0000","last_clean":"2024-05-27T07:39:44.419074+0000","last_became_active":"2024-05-27T07:36:04.976788+0000","last_became_peered":"2024-05-27T07:36:04.976788+0000","last_unstale":"2024-05-27T07:39:44.419074+0000","last_undegraded":"2024-05-27T07:39:44.419074+0000","last_fullsized":"2024-05-27T07:39:44.419074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:59:44.574191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418351+0000","last_change":"2024-05-27T07:36:04.986594+0000","last_active":"2024-05-27T07:39:44.418351+0000","last_peered":"2024-05-27T07:39:44.418351+0000","last_clean":"2024-05-27T07:39:44.418351+0000","last_became_active":"2024-05-27T07:36:04.985161+0000","last_became_peered":"2024-05-27T07:36:04.985161+0000","last_unstale":"2024-05-27T07:39:44.418351+0000","last_undegraded":"2024-05-27T07:39:44.418351+0000","last_fullsized":"2024-05-27T07:39:44.418351+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.239461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133787+0000","last_change":"2024-05-27T07:36:04.984781+0000","last_active":"2024-05-27T07:39:44.133787+0000","last_peered":"2024-05-27T07:39:44.133787+0000","last_clean":"2024-05-27T07:39:44.133787+0000","last_became_active":"2024-05-27T07:36:04.984462+0000","last_became_peered":"2024-05-27T07:36:04.984462+0000","last_unstale":"2024-05-27T07:39:44.133787+0000","last_undegraded":"2024-05-27T07:39:44.133787+0000","last_fullsized":"2024-05-27T07:39:44.133787+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:40.051692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417506+0000","last_change":"2024-05-27T07:36:04.978360+0000","last_active":"2024-05-27T07:39:44.417506+0000","last_peered":"2024-05-27T07:39:44.417506+0000","last_clean":"2024-05-27T07:39:44.417506+0000","last_became_active":"2024-05-27T07:36:04.978105+0000","last_became_peered":"2024-05-27T07:36:04.978105+0000","last_unstale":"2024-05-27T07:39:44.417506+0000","last_undegraded":"2024-05-27T07:39:44.417506+0000","last_fullsized":"2024-05-27T07:39:44.417506+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:04:47.257529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417507+0000","last_change":"2024-05-27T07:36:04.986510+0000","last_active":"2024-05-27T07:39:44.417507+0000","last_peered":"2024-05-27T07:39:44.417507+0000","last_clean":"2024-05-27T07:39:44.417507+0000","last_became_active":"2024-05-27T07:36:04.986150+0000","last_became_peered":"2024-05-27T07:36:04.986150+0000","last_unstale":"2024-05-27T07:39:44.417507+0000","last_undegraded":"2024-05-27T07:39:44.417507+0000","last_fullsized":"2024-05-27T07:39:44.417507+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:07.349470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418178+0000","last_change":"2024-05-27T07:36:04.979880+0000","last_active":"2024-05-27T07:39:44.418178+0000","last_peered":"2024-05-27T07:39:44.418178+0000","last_clean":"2024-05-27T07:39:44.418178+0000","last_became_active":"2024-05-27T07:36:04.979516+0000","last_became_peered":"2024-05-27T07:36:04.979516+0000","last_unstale":"2024-05-27T07:39:44.418178+0000","last_undegraded":"2024-05-27T07:39:44.418178+0000","last_fullsized":"2024-05-27T07:39:44.418178+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:41:03.964277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417955+0000","last_change":"2024-05-27T07:36:04.980231+0000","last_active":"2024-05-27T07:39:44.417955+0000","last_peered":"2024-05-27T07:39:44.417955+0000","last_clean":"2024-05-27T07:39:44.417955+0000","last_became_active":"2024-05-27T07:36:04.979888+0000","last_became_peered":"2024-05-27T07:36:04.979888+0000","last_unstale":"2024-05-27T07:39:44.417955+0000","last_undegraded":"2024-05-27T07:39:44.417955+0000","last_fullsized":"2024-05-27T07:39:44.417955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.838119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244612+0000","last_change":"2024-05-27T07:36:04.990890+0000","last_active":"2024-05-27T07:39:44.244612+0000","last_peered":"2024-05-27T07:39:44.244612+0000","last_clean":"2024-05-27T07:39:44.244612+0000","last_became_active":"2024-05-27T07:36:04.990581+0000","last_became_peered":"2024-05-27T07:36:04.990581+0000","last_unstale":"2024-05-27T07:39:44.244612+0000","last_undegraded":"2024-05-27T07:39:44.244612+0000","last_fullsized":"2024-05-27T07:39:44.244612+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:33.054395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363804+0000","last_change":"2024-05-27T07:36:04.981620+0000","last_active":"2024-05-27T07:39:44.363804+0000","last_peered":"2024-05-27T07:39:44.363804+0000","last_clean":"2024-05-27T07:39:44.363804+0000","last_became_active":"2024-05-27T07:36:04.981290+0000","last_became_peered":"2024-05-27T07:36:04.981290+0000","last_unstale":"2024-05-27T07:39:44.363804+0000","last_undegraded":"2024-05-27T07:39:44.363804+0000","last_fullsized":"2024-05-27T07:39:44.363804+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:50.287380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363074+0000","last_change":"2024-05-27T07:36:15.030740+0000","last_active":"2024-05-27T07:39:44.363074+0000","last_peered":"2024-05-27T07:39:44.363074+0000","last_clean":"2024-05-27T07:39:44.363074+0000","last_became_active":"2024-05-27T07:36:15.029738+0000","last_became_peered":"2024-05-27T07:36:15.029738+0000","last_unstale":"2024-05-27T07:39:44.363074+0000","last_undegraded":"2024-05-27T07:39:44.363074+0000","last_fullsized":"2024-05-27T07:39:44.363074+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:18.101941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"78'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244544+0000","last_change":"2024-05-27T07:36:01.955530+0000","last_active":"2024-05-27T07:39:44.244544+0000","last_peered":"2024-05-27T07:39:44.244544+0000","last_clean":"2024-05-27T07:39:44.244544+0000","last_became_active":"2024-05-27T07:36:01.955225+0000","last_became_peered":"2024-05-27T07:36:01.955225+0000","last_unstale":"2024-05-27T07:39:44.244544+0000","last_undegraded":"2024-05-27T07:39:44.244544+0000","last_fullsized":"2024-05-27T07:39:44.244544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:02.067672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.248070+0000","last_change":"2024-05-27T07:36:04.984397+0000","last_active":"2024-05-27T07:39:44.248070+0000","last_peered":"2024-05-27T07:39:44.248070+0000","last_clean":"2024-05-27T07:39:44.248070+0000","last_became_active":"2024-05-27T07:36:04.984033+0000","last_became_peered":"2024-05-27T07:36:04.984033+0000","last_unstale":"2024-05-27T07:39:44.248070+0000","last_undegraded":"2024-05-27T07:39:44.248070+0000","last_fullsized":"2024-05-27T07:39:44.248070+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:15.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417201+0000","last_change":"2024-05-27T07:37:46.322144+0000","last_active":"2024-05-27T07:39:44.417201+0000","last_peered":"2024-05-27T07:39:44.417201+0000","last_clean":"2024-05-27T07:39:44.417201+0000","last_became_active":"2024-05-27T07:37:46.321610+0000","last_became_peered":"2024-05-27T07:37:46.321610+0000","last_unstale":"2024-05-27T07:39:44.417201+0000","last_undegraded":"2024-05-27T07:39:44.417201+0000","last_fullsized":"2024-05-27T07:39:44.417201+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:47.119281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"78'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.248298+0000","last_change":"2024-05-27T07:36:02.243079+0000","last_active":"2024-05-27T07:39:44.248298+0000","last_peered":"2024-05-27T07:39:44.248298+0000","last_clean":"2024-05-27T07:39:44.248298+0000","last_became_active":"2024-05-27T07:36:02.242633+0000","last_became_peered":"2024-05-27T07:36:02.242633+0000","last_unstale":"2024-05-27T07:39:44.248298+0000","last_undegraded":"2024-05-27T07:39:44.248298+0000","last_fullsized":"2024-05-27T07:39:44.248298+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:45:33.980842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417940+0000","last_change":"2024-05-27T07:36:04.989775+0000","last_active":"2024-05-27T07:39:44.417940+0000","last_peered":"2024-05-27T07:39:44.417940+0000","last_clean":"2024-05-27T07:39:44.417940+0000","last_became_active":"2024-05-27T07:36:04.989289+0000","last_became_peered":"2024-05-27T07:36:04.989289+0000","last_unstale":"2024-05-27T07:39:44.417940+0000","last_undegraded":"2024-05-27T07:39:44.417940+0000","last_fullsized":"2024-05-27T07:39:44.417940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:24:38.355231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417613+0000","last_change":"2024-05-27T07:36:15.031737+0000","last_active":"2024-05-27T07:39:44.417613+0000","last_peered":"2024-05-27T07:39:44.417613+0000","last_clean":"2024-05-27T07:39:44.417613+0000","last_became_active":"2024-05-27T07:36:15.031314+0000","last_became_peered":"2024-05-27T07:36:15.031314+0000","last_unstale":"2024-05-27T07:39:44.417613+0000","last_undegraded":"2024-05-27T07:39:44.417613+0000","last_fullsized":"2024-05-27T07:39:44.417613+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:32.742298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"78'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247148+0000","last_change":"2024-05-27T07:36:01.957459+0000","last_active":"2024-05-27T07:39:44.247148+0000","last_peered":"2024-05-27T07:39:44.247148+0000","last_clean":"2024-05-27T07:39:44.247148+0000","last_became_active":"2024-05-27T07:36:01.954985+0000","last_became_peered":"2024-05-27T07:36:01.954985+0000","last_unstale":"2024-05-27T07:39:44.247148+0000","last_undegraded":"2024-05-27T07:39:44.247148+0000","last_fullsized":"2024-05-27T07:39:44.247148+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:27.085486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418627+0000","last_change":"2024-05-27T07:36:04.986447+0000","last_active":"2024-05-27T07:39:44.418627+0000","last_peered":"2024-05-27T07:39:44.418627+0000","last_clean":"2024-05-27T07:39:44.418627+0000","last_became_active":"2024-05-27T07:36:04.982099+0000","last_became_peered":"2024-05-27T07:36:04.982099+0000","last_unstale":"2024-05-27T07:39:44.418627+0000","last_undegraded":"2024-05-27T07:39:44.418627+0000","last_fullsized":"2024-05-27T07:39:44.418627+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:32.637479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417267+0000","last_change":"2024-05-27T07:36:15.024152+0000","last_active":"2024-05-27T07:39:44.417267+0000","last_peered":"2024-05-27T07:39:44.417267+0000","last_clean":"2024-05-27T07:39:44.417267+0000","last_became_active":"2024-05-27T07:36:15.023711+0000","last_became_peered":"2024-05-27T07:36:15.023711+0000","last_unstale":"2024-05-27T07:39:44.417267+0000","last_undegraded":"2024-05-27T07:39:44.417267+0000","last_fullsized":"2024-05-27T07:39:44.417267+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:33.809364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246823+0000","last_change":"2024-05-27T07:36:01.950531+0000","last_active":"2024-05-27T07:39:44.246823+0000","last_peered":"2024-05-27T07:39:44.246823+0000","last_clean":"2024-05-27T07:39:44.246823+0000","last_became_active":"2024-05-27T07:36:01.950015+0000","last_became_peered":"2024-05-27T07:36:01.950015+0000","last_unstale":"2024-05-27T07:39:44.246823+0000","last_undegraded":"2024-05-27T07:39:44.246823+0000","last_fullsized":"2024-05-27T07:39:44.246823+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:22.400415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135595+0000","last_change":"2024-05-27T07:36:04.982792+0000","last_active":"2024-05-27T07:39:44.135595+0000","last_peered":"2024-05-27T07:39:44.135595+0000","last_clean":"2024-05-27T07:39:44.135595+0000","last_became_active":"2024-05-27T07:36:04.982400+0000","last_became_peered":"2024-05-27T07:36:04.982400+0000","last_unstale":"2024-05-27T07:39:44.135595+0000","last_undegraded":"2024-05-27T07:39:44.135595+0000","last_fullsized":"2024-05-27T07:39:44.135595+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:27.823654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363642+0000","last_change":"2024-05-27T07:36:15.027932+0000","last_active":"2024-05-27T07:39:44.363642+0000","last_peered":"2024-05-27T07:39:44.363642+0000","last_clean":"2024-05-27T07:39:44.363642+0000","last_became_active":"2024-05-27T07:36:15.027526+0000","last_became_peered":"2024-05-27T07:36:15.027526+0000","last_unstale":"2024-05-27T07:39:44.363642+0000","last_undegraded":"2024-05-27T07:39:44.363642+0000","last_fullsized":"2024-05-27T07:39:44.363642+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:16.165934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"78'1","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135427+0000","last_change":"2024-05-27T07:36:45.127637+0000","last_active":"2024-05-27T07:39:44.135427+0000","last_peered":"2024-05-27T07:39:44.135427+0000","last_clean":"2024-05-27T07:39:44.135427+0000","last_became_active":"2024-05-27T07:36:45.127249+0000","last_became_peered":"2024-05-27T07:36:45.127249+0000","last_unstale":"2024-05-27T07:39:44.135427+0000","last_undegraded":"2024-05-27T07:39:44.135427+0000","last_fullsized":"2024-05-27T07:39:44.135427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:52.418069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246700+0000","last_change":"2024-05-27T07:36:04.985135+0000","last_active":"2024-05-27T07:39:44.246700+0000","last_peered":"2024-05-27T07:39:44.246700+0000","last_clean":"2024-05-27T07:39:44.246700+0000","last_became_active":"2024-05-27T07:36:04.983621+0000","last_became_peered":"2024-05-27T07:36:04.983621+0000","last_unstale":"2024-05-27T07:39:44.246700+0000","last_undegraded":"2024-05-27T07:39:44.246700+0000","last_fullsized":"2024-05-27T07:39:44.246700+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:11.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363376+0000","last_change":"2024-05-27T07:36:15.024558+0000","last_active":"2024-05-27T07:39:44.363376+0000","last_peered":"2024-05-27T07:39:44.363376+0000","last_clean":"2024-05-27T07:39:44.363376+0000","last_became_active":"2024-05-27T07:36:15.024148+0000","last_became_peered":"2024-05-27T07:36:15.024148+0000","last_unstale":"2024-05-27T07:39:44.363376+0000","last_undegraded":"2024-05-27T07:39:44.363376+0000","last_fullsized":"2024-05-27T07:39:44.363376+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:03.648451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"78'1","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133382+0000","last_change":"2024-05-27T07:36:45.129413+0000","last_active":"2024-05-27T07:39:44.133382+0000","last_peered":"2024-05-27T07:39:44.133382+0000","last_clean":"2024-05-27T07:39:44.133382+0000","last_became_active":"2024-05-27T07:36:45.128779+0000","last_became_peered":"2024-05-27T07:36:45.128779+0000","last_unstale":"2024-05-27T07:39:44.133382+0000","last_undegraded":"2024-05-27T07:39:44.133382+0000","last_fullsized":"2024-05-27T07:39:44.133382+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:05.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135849+0000","last_change":"2024-05-27T07:36:04.989878+0000","last_active":"2024-05-27T07:39:44.135849+0000","last_peered":"2024-05-27T07:39:44.135849+0000","last_clean":"2024-05-27T07:39:44.135849+0000","last_became_active":"2024-05-27T07:36:04.989532+0000","last_became_peered":"2024-05-27T07:36:04.989532+0000","last_unstale":"2024-05-27T07:39:44.135849+0000","last_undegraded":"2024-05-27T07:39:44.135849+0000","last_fullsized":"2024-05-27T07:39:44.135849+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:54:53.909592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417859+0000","last_change":"2024-05-27T07:37:46.321582+0000","last_active":"2024-05-27T07:39:44.417859+0000","last_peered":"2024-05-27T07:39:44.417859+0000","last_clean":"2024-05-27T07:39:44.417859+0000","last_became_active":"2024-05-27T07:37:46.321193+0000","last_became_peered":"2024-05-27T07:37:46.321193+0000","last_unstale":"2024-05-27T07:39:44.417859+0000","last_undegraded":"2024-05-27T07:39:44.417859+0000","last_fullsized":"2024-05-27T07:39:44.417859+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:25:25.767300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133519+0000","last_change":"2024-05-27T07:36:01.949684+0000","last_active":"2024-05-27T07:39:44.133519+0000","last_peered":"2024-05-27T07:39:44.133519+0000","last_clean":"2024-05-27T07:39:44.133519+0000","last_became_active":"2024-05-27T07:36:01.949308+0000","last_became_peered":"2024-05-27T07:36:01.949308+0000","last_unstale":"2024-05-27T07:39:44.133519+0000","last_undegraded":"2024-05-27T07:39:44.133519+0000","last_fullsized":"2024-05-27T07:39:44.133519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:50:54.859172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247002+0000","last_change":"2024-05-27T07:36:04.971266+0000","last_active":"2024-05-27T07:39:44.247002+0000","last_peered":"2024-05-27T07:39:44.247002+0000","last_clean":"2024-05-27T07:39:44.247002+0000","last_became_active":"2024-05-27T07:36:04.970831+0000","last_became_peered":"2024-05-27T07:36:04.970831+0000","last_unstale":"2024-05-27T07:39:44.247002+0000","last_undegraded":"2024-05-27T07:39:44.247002+0000","last_fullsized":"2024-05-27T07:39:44.247002+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:37.237637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417864+0000","last_change":"2024-05-27T07:36:15.024415+0000","last_active":"2024-05-27T07:39:44.417864+0000","last_peered":"2024-05-27T07:39:44.417864+0000","last_clean":"2024-05-27T07:39:44.417864+0000","last_became_active":"2024-05-27T07:36:15.023935+0000","last_became_peered":"2024-05-27T07:36:15.023935+0000","last_unstale":"2024-05-27T07:39:44.417864+0000","last_undegraded":"2024-05-27T07:39:44.417864+0000","last_fullsized":"2024-05-27T07:39:44.417864+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:37:30.386933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134940+0000","last_change":"2024-05-27T07:36:01.952256+0000","last_active":"2024-05-27T07:39:44.134940+0000","last_peered":"2024-05-27T07:39:44.134940+0000","last_clean":"2024-05-27T07:39:44.134940+0000","last_became_active":"2024-05-27T07:36:01.951876+0000","last_became_peered":"2024-05-27T07:36:01.951876+0000","last_unstale":"2024-05-27T07:39:44.134940+0000","last_undegraded":"2024-05-27T07:39:44.134940+0000","last_fullsized":"2024-05-27T07:39:44.134940+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:05.201780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.243724+0000","last_change":"2024-05-27T07:36:04.980300+0000","last_active":"2024-05-27T07:39:44.243724+0000","last_peered":"2024-05-27T07:39:44.243724+0000","last_clean":"2024-05-27T07:39:44.243724+0000","last_became_active":"2024-05-27T07:36:04.979954+0000","last_became_peered":"2024-05-27T07:36:04.979954+0000","last_unstale":"2024-05-27T07:39:44.243724+0000","last_undegraded":"2024-05-27T07:39:44.243724+0000","last_fullsized":"2024-05-27T07:39:44.243724+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:15.712773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247102+0000","last_change":"2024-05-27T07:36:15.026685+0000","last_active":"2024-05-27T07:39:44.247102+0000","last_peered":"2024-05-27T07:39:44.247102+0000","last_clean":"2024-05-27T07:39:44.247102+0000","last_became_active":"2024-05-27T07:36:15.026287+0000","last_became_peered":"2024-05-27T07:36:15.026287+0000","last_unstale":"2024-05-27T07:39:44.247102+0000","last_undegraded":"2024-05-27T07:39:44.247102+0000","last_fullsized":"2024-05-27T07:39:44.247102+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:29:26.394435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134169+0000","last_change":"2024-05-27T07:36:01.956566+0000","last_active":"2024-05-27T07:39:44.134169+0000","last_peered":"2024-05-27T07:39:44.134169+0000","last_clean":"2024-05-27T07:39:44.134169+0000","last_became_active":"2024-05-27T07:36:01.955963+0000","last_became_peered":"2024-05-27T07:36:01.955963+0000","last_unstale":"2024-05-27T07:39:44.134169+0000","last_undegraded":"2024-05-27T07:39:44.134169+0000","last_fullsized":"2024-05-27T07:39:44.134169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:53.081784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418762+0000","last_change":"2024-05-27T07:36:04.983878+0000","last_active":"2024-05-27T07:39:44.418762+0000","last_peered":"2024-05-27T07:39:44.418762+0000","last_clean":"2024-05-27T07:39:44.418762+0000","last_became_active":"2024-05-27T07:36:04.983339+0000","last_became_peered":"2024-05-27T07:36:04.983339+0000","last_unstale":"2024-05-27T07:39:44.418762+0000","last_undegraded":"2024-05-27T07:39:44.418762+0000","last_fullsized":"2024-05-27T07:39:44.418762+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:57:01.726122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.243682+0000","last_change":"2024-05-27T07:36:15.027923+0000","last_active":"2024-05-27T07:39:44.243682+0000","last_peered":"2024-05-27T07:39:44.243682+0000","last_clean":"2024-05-27T07:39:44.243682+0000","last_became_active":"2024-05-27T07:36:15.026012+0000","last_became_peered":"2024-05-27T07:36:15.026012+0000","last_unstale":"2024-05-27T07:39:44.243682+0000","last_undegraded":"2024-05-27T07:39:44.243682+0000","last_fullsized":"2024-05-27T07:39:44.243682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:27:45.238053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419164+0000","last_change":"2024-05-27T07:36:01.956038+0000","last_active":"2024-05-27T07:39:44.419164+0000","last_peered":"2024-05-27T07:39:44.419164+0000","last_clean":"2024-05-27T07:39:44.419164+0000","last_became_active":"2024-05-27T07:36:01.955639+0000","last_became_peered":"2024-05-27T07:36:01.955639+0000","last_unstale":"2024-05-27T07:39:44.419164+0000","last_undegraded":"2024-05-27T07:39:44.419164+0000","last_fullsized":"2024-05-27T07:39:44.419164+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:32.920194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417936+0000","last_change":"2024-05-27T07:36:04.986676+0000","last_active":"2024-05-27T07:39:44.417936+0000","last_peered":"2024-05-27T07:39:44.417936+0000","last_clean":"2024-05-27T07:39:44.417936+0000","last_became_active":"2024-05-27T07:36:04.986316+0000","last_became_peered":"2024-05-27T07:36:04.986316+0000","last_unstale":"2024-05-27T07:39:44.417936+0000","last_undegraded":"2024-05-27T07:39:44.417936+0000","last_fullsized":"2024-05-27T07:39:44.417936+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:03:50.947781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417814+0000","last_change":"2024-05-27T07:36:15.027643+0000","last_active":"2024-05-27T07:39:44.417814+0000","last_peered":"2024-05-27T07:39:44.417814+0000","last_clean":"2024-05-27T07:39:44.417814+0000","last_became_active":"2024-05-27T07:36:15.027175+0000","last_became_peered":"2024-05-27T07:36:15.027175+0000","last_unstale":"2024-05-27T07:39:44.417814+0000","last_undegraded":"2024-05-27T07:39:44.417814+0000","last_fullsized":"2024-05-27T07:39:44.417814+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:46:32.764271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"78'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135540+0000","last_change":"2024-05-27T07:36:02.245746+0000","last_active":"2024-05-27T07:39:44.135540+0000","last_peered":"2024-05-27T07:39:44.135540+0000","last_clean":"2024-05-27T07:39:44.135540+0000","last_became_active":"2024-05-27T07:36:02.245269+0000","last_became_peered":"2024-05-27T07:36:02.245269+0000","last_unstale":"2024-05-27T07:39:44.135540+0000","last_undegraded":"2024-05-27T07:39:44.135540+0000","last_fullsized":"2024-05-27T07:39:44.135540+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:28.202540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418801+0000","last_change":"2024-05-27T07:36:45.129374+0000","last_active":"2024-05-27T07:39:44.418801+0000","last_peered":"2024-05-27T07:39:44.418801+0000","last_clean":"2024-05-27T07:39:44.418801+0000","last_became_active":"2024-05-27T07:36:45.128956+0000","last_became_peered":"2024-05-27T07:36:45.128956+0000","last_unstale":"2024-05-27T07:39:44.418801+0000","last_undegraded":"2024-05-27T07:39:44.418801+0000","last_fullsized":"2024-05-27T07:39:44.418801+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:17.037598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244073+0000","last_change":"2024-05-27T07:36:15.027097+0000","last_active":"2024-05-27T07:39:44.244073+0000","last_peered":"2024-05-27T07:39:44.244073+0000","last_clean":"2024-05-27T07:39:44.244073+0000","last_became_active":"2024-05-27T07:36:15.026704+0000","last_became_peered":"2024-05-27T07:36:15.026704+0000","last_unstale":"2024-05-27T07:39:44.244073+0000","last_undegraded":"2024-05-27T07:39:44.244073+0000","last_fullsized":"2024-05-27T07:39:44.244073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:35:03.747199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'39","reported_seq":423,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.714753+0000","last_change":"2024-05-27T07:36:01.956926+0000","last_active":"2024-05-27T07:39:44.714753+0000","last_peered":"2024-05-27T07:39:44.714753+0000","last_clean":"2024-05-27T07:39:44.714753+0000","last_became_active":"2024-05-27T07:36:01.956515+0000","last_became_peered":"2024-05-27T07:36:01.956515+0000","last_unstale":"2024-05-27T07:39:44.714753+0000","last_undegraded":"2024-05-27T07:39:44.714753+0000","last_fullsized":"2024-05-27T07:39:44.714753+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:46.566489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71357,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":337,"num_read_kb":239,"num_write":39,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246955+0000","last_change":"2024-05-27T07:36:04.976498+0000","last_active":"2024-05-27T07:39:44.246955+0000","last_peered":"2024-05-27T07:39:44.246955+0000","last_clean":"2024-05-27T07:39:44.246955+0000","last_became_active":"2024-05-27T07:36:04.976020+0000","last_became_peered":"2024-05-27T07:36:04.976020+0000","last_unstale":"2024-05-27T07:39:44.246955+0000","last_undegraded":"2024-05-27T07:39:44.246955+0000","last_fullsized":"2024-05-27T07:39:44.246955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:25.909259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135913+0000","last_change":"2024-05-27T07:36:15.022700+0000","last_active":"2024-05-27T07:39:44.135913+0000","last_peered":"2024-05-27T07:39:44.135913+0000","last_clean":"2024-05-27T07:39:44.135913+0000","last_became_active":"2024-05-27T07:36:15.022109+0000","last_became_peered":"2024-05-27T07:36:15.022109+0000","last_unstale":"2024-05-27T07:39:44.135913+0000","last_undegraded":"2024-05-27T07:39:44.135913+0000","last_fullsized":"2024-05-27T07:39:44.135913+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:02:43.498049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"78'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363210+0000","last_change":"2024-05-27T07:36:01.953442+0000","last_active":"2024-05-27T07:39:44.363210+0000","last_peered":"2024-05-27T07:39:44.363210+0000","last_clean":"2024-05-27T07:39:44.363210+0000","last_became_active":"2024-05-27T07:36:01.953051+0000","last_became_peered":"2024-05-27T07:36:01.953051+0000","last_unstale":"2024-05-27T07:39:44.363210+0000","last_undegraded":"2024-05-27T07:39:44.363210+0000","last_fullsized":"2024-05-27T07:39:44.363210+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:47:53.070596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133518+0000","last_change":"2024-05-27T07:36:04.976365+0000","last_active":"2024-05-27T07:39:44.133518+0000","last_peered":"2024-05-27T07:39:44.133518+0000","last_clean":"2024-05-27T07:39:44.133518+0000","last_became_active":"2024-05-27T07:36:04.976050+0000","last_became_peered":"2024-05-27T07:36:04.976050+0000","last_unstale":"2024-05-27T07:39:44.133518+0000","last_undegraded":"2024-05-27T07:39:44.133518+0000","last_fullsized":"2024-05-27T07:39:44.133518+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:19.765746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417617+0000","last_change":"2024-05-27T07:36:15.022561+0000","last_active":"2024-05-27T07:39:44.417617+0000","last_peered":"2024-05-27T07:39:44.417617+0000","last_clean":"2024-05-27T07:39:44.417617+0000","last_became_active":"2024-05-27T07:36:15.021974+0000","last_became_peered":"2024-05-27T07:36:15.021974+0000","last_unstale":"2024-05-27T07:39:44.417617+0000","last_undegraded":"2024-05-27T07:39:44.417617+0000","last_fullsized":"2024-05-27T07:39:44.417617+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:31.520969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135318+0000","last_change":"2024-05-27T07:36:01.952625+0000","last_active":"2024-05-27T07:39:44.135318+0000","last_peered":"2024-05-27T07:39:44.135318+0000","last_clean":"2024-05-27T07:39:44.135318+0000","last_became_active":"2024-05-27T07:36:01.952280+0000","last_became_peered":"2024-05-27T07:36:01.952280+0000","last_unstale":"2024-05-27T07:39:44.135318+0000","last_undegraded":"2024-05-27T07:39:44.135318+0000","last_fullsized":"2024-05-27T07:39:44.135318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:17.548061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418259+0000","last_change":"2024-05-27T07:36:04.984795+0000","last_active":"2024-05-27T07:39:44.418259+0000","last_peered":"2024-05-27T07:39:44.418259+0000","last_clean":"2024-05-27T07:39:44.418259+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:39:44.418259+0000","last_undegraded":"2024-05-27T07:39:44.418259+0000","last_fullsized":"2024-05-27T07:39:44.418259+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:07.207460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244434+0000","last_change":"2024-05-27T07:36:15.033565+0000","last_active":"2024-05-27T07:39:44.244434+0000","last_peered":"2024-05-27T07:39:44.244434+0000","last_clean":"2024-05-27T07:39:44.244434+0000","last_became_active":"2024-05-27T07:36:15.033145+0000","last_became_peered":"2024-05-27T07:36:15.033145+0000","last_unstale":"2024-05-27T07:39:44.244434+0000","last_undegraded":"2024-05-27T07:39:44.244434+0000","last_fullsized":"2024-05-27T07:39:44.244434+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:38:02.163816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417353+0000","last_change":"2024-05-27T07:36:01.952759+0000","last_active":"2024-05-27T07:39:44.417353+0000","last_peered":"2024-05-27T07:39:44.417353+0000","last_clean":"2024-05-27T07:39:44.417353+0000","last_became_active":"2024-05-27T07:36:01.952339+0000","last_became_peered":"2024-05-27T07:36:01.952339+0000","last_unstale":"2024-05-27T07:39:44.417353+0000","last_undegraded":"2024-05-27T07:39:44.417353+0000","last_fullsized":"2024-05-27T07:39:44.417353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:14.518123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417694+0000","last_change":"2024-05-27T07:36:04.986312+0000","last_active":"2024-05-27T07:39:44.417694+0000","last_peered":"2024-05-27T07:39:44.417694+0000","last_clean":"2024-05-27T07:39:44.417694+0000","last_became_active":"2024-05-27T07:36:04.985944+0000","last_became_peered":"2024-05-27T07:36:04.985944+0000","last_unstale":"2024-05-27T07:39:44.417694+0000","last_undegraded":"2024-05-27T07:39:44.417694+0000","last_fullsized":"2024-05-27T07:39:44.417694+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:04.955449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133923+0000","last_change":"2024-05-27T07:36:15.020930+0000","last_active":"2024-05-27T07:39:44.133923+0000","last_peered":"2024-05-27T07:39:44.133923+0000","last_clean":"2024-05-27T07:39:44.133923+0000","last_became_active":"2024-05-27T07:36:15.020556+0000","last_became_peered":"2024-05-27T07:36:15.020556+0000","last_unstale":"2024-05-27T07:39:44.133923+0000","last_undegraded":"2024-05-27T07:39:44.133923+0000","last_fullsized":"2024-05-27T07:39:44.133923+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:02.673065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"78'8","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246644+0000","last_change":"2024-05-27T07:36:02.243402+0000","last_active":"2024-05-27T07:39:44.246644+0000","last_peered":"2024-05-27T07:39:44.246644+0000","last_clean":"2024-05-27T07:39:44.246644+0000","last_became_active":"2024-05-27T07:36:02.242978+0000","last_became_peered":"2024-05-27T07:36:02.242978+0000","last_unstale":"2024-05-27T07:39:44.246644+0000","last_undegraded":"2024-05-27T07:39:44.246644+0000","last_fullsized":"2024-05-27T07:39:44.246644+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:25:04.289506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135103+0000","last_change":"2024-05-27T07:36:04.989885+0000","last_active":"2024-05-27T07:39:44.135103+0000","last_peered":"2024-05-27T07:39:44.135103+0000","last_clean":"2024-05-27T07:39:44.135103+0000","last_became_active":"2024-05-27T07:36:04.989394+0000","last_became_peered":"2024-05-27T07:36:04.989394+0000","last_unstale":"2024-05-27T07:39:44.135103+0000","last_undegraded":"2024-05-27T07:39:44.135103+0000","last_fullsized":"2024-05-27T07:39:44.135103+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:54.138286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135101+0000","last_change":"2024-05-27T07:36:15.021772+0000","last_active":"2024-05-27T07:39:44.135101+0000","last_peered":"2024-05-27T07:39:44.135101+0000","last_clean":"2024-05-27T07:39:44.135101+0000","last_became_active":"2024-05-27T07:36:15.021352+0000","last_became_peered":"2024-05-27T07:36:15.021352+0000","last_unstale":"2024-05-27T07:39:44.135101+0000","last_undegraded":"2024-05-27T07:39:44.135101+0000","last_fullsized":"2024-05-27T07:39:44.135101+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:23.066662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417811+0000","last_change":"2024-05-27T07:36:45.131900+0000","last_active":"2024-05-27T07:39:44.417811+0000","last_peered":"2024-05-27T07:39:44.417811+0000","last_clean":"2024-05-27T07:39:44.417811+0000","last_became_active":"2024-05-27T07:36:45.127340+0000","last_became_peered":"2024-05-27T07:36:45.127340+0000","last_unstale":"2024-05-27T07:39:44.417811+0000","last_undegraded":"2024-05-27T07:39:44.417811+0000","last_fullsized":"2024-05-27T07:39:44.417811+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:07:57.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417805+0000","last_change":"2024-05-27T07:36:04.987671+0000","last_active":"2024-05-27T07:39:44.417805+0000","last_peered":"2024-05-27T07:39:44.417805+0000","last_clean":"2024-05-27T07:39:44.417805+0000","last_became_active":"2024-05-27T07:36:04.987317+0000","last_became_peered":"2024-05-27T07:36:04.987317+0000","last_unstale":"2024-05-27T07:39:44.417805+0000","last_undegraded":"2024-05-27T07:39:44.417805+0000","last_fullsized":"2024-05-27T07:39:44.417805+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:35.985968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246611+0000","last_change":"2024-05-27T07:36:15.019967+0000","last_active":"2024-05-27T07:39:44.246611+0000","last_peered":"2024-05-27T07:39:44.246611+0000","last_clean":"2024-05-27T07:39:44.246611+0000","last_became_active":"2024-05-27T07:36:15.019509+0000","last_became_peered":"2024-05-27T07:36:15.019509+0000","last_unstale":"2024-05-27T07:39:44.246611+0000","last_undegraded":"2024-05-27T07:39:44.246611+0000","last_fullsized":"2024-05-27T07:39:44.246611+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:01:56.137335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244238+0000","last_change":"2024-05-27T07:36:02.243561+0000","last_active":"2024-05-27T07:39:44.244238+0000","last_peered":"2024-05-27T07:39:44.244238+0000","last_clean":"2024-05-27T07:39:44.244238+0000","last_became_active":"2024-05-27T07:36:02.242911+0000","last_became_peered":"2024-05-27T07:36:02.242911+0000","last_unstale":"2024-05-27T07:39:44.244238+0000","last_undegraded":"2024-05-27T07:39:44.244238+0000","last_fullsized":"2024-05-27T07:39:44.244238+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:01:43.915660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133619+0000","last_change":"2024-05-27T07:36:04.971108+0000","last_active":"2024-05-27T07:39:44.133619+0000","last_peered":"2024-05-27T07:39:44.133619+0000","last_clean":"2024-05-27T07:39:44.133619+0000","last_became_active":"2024-05-27T07:36:04.970682+0000","last_became_peered":"2024-05-27T07:36:04.970682+0000","last_unstale":"2024-05-27T07:39:44.133619+0000","last_undegraded":"2024-05-27T07:39:44.133619+0000","last_fullsized":"2024-05-27T07:39:44.133619+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:15:34.940449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133949+0000","last_change":"2024-05-27T07:36:15.021292+0000","last_active":"2024-05-27T07:39:44.133949+0000","last_peered":"2024-05-27T07:39:44.133949+0000","last_clean":"2024-05-27T07:39:44.133949+0000","last_became_active":"2024-05-27T07:36:15.019988+0000","last_became_peered":"2024-05-27T07:36:15.019988+0000","last_unstale":"2024-05-27T07:39:44.133949+0000","last_undegraded":"2024-05-27T07:39:44.133949+0000","last_fullsized":"2024-05-27T07:39:44.133949+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:34:14.325368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417857+0000","last_change":"2024-05-27T07:36:01.952231+0000","last_active":"2024-05-27T07:39:44.417857+0000","last_peered":"2024-05-27T07:39:44.417857+0000","last_clean":"2024-05-27T07:39:44.417857+0000","last_became_active":"2024-05-27T07:36:01.951818+0000","last_became_peered":"2024-05-27T07:36:01.951818+0000","last_unstale":"2024-05-27T07:39:44.417857+0000","last_undegraded":"2024-05-27T07:39:44.417857+0000","last_fullsized":"2024-05-27T07:39:44.417857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:52:16.299204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133412+0000","last_change":"2024-05-27T07:36:04.982761+0000","last_active":"2024-05-27T07:39:44.133412+0000","last_peered":"2024-05-27T07:39:44.133412+0000","last_clean":"2024-05-27T07:39:44.133412+0000","last_became_active":"2024-05-27T07:36:04.982303+0000","last_became_peered":"2024-05-27T07:36:04.982303+0000","last_unstale":"2024-05-27T07:39:44.133412+0000","last_undegraded":"2024-05-27T07:39:44.133412+0000","last_fullsized":"2024-05-27T07:39:44.133412+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:13:19.099227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133814+0000","last_change":"2024-05-27T07:37:46.323619+0000","last_active":"2024-05-27T07:39:44.133814+0000","last_peered":"2024-05-27T07:39:44.133814+0000","last_clean":"2024-05-27T07:39:44.133814+0000","last_became_active":"2024-05-27T07:37:46.322114+0000","last_became_peered":"2024-05-27T07:37:46.322114+0000","last_unstale":"2024-05-27T07:39:44.133814+0000","last_undegraded":"2024-05-27T07:39:44.133814+0000","last_fullsized":"2024-05-27T07:39:44.133814+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:59:08.966863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135176+0000","last_change":"2024-05-27T07:36:45.142023+0000","last_active":"2024-05-27T07:39:44.135176+0000","last_peered":"2024-05-27T07:39:44.135176+0000","last_clean":"2024-05-27T07:39:44.135176+0000","last_became_active":"2024-05-27T07:36:45.141278+0000","last_became_peered":"2024-05-27T07:36:45.141278+0000","last_unstale":"2024-05-27T07:39:44.135176+0000","last_undegraded":"2024-05-27T07:39:44.135176+0000","last_fullsized":"2024-05-27T07:39:44.135176+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:15.800380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133391+0000","last_change":"2024-05-27T07:36:02.243041+0000","last_active":"2024-05-27T07:39:44.133391+0000","last_peered":"2024-05-27T07:39:44.133391+0000","last_clean":"2024-05-27T07:39:44.133391+0000","last_became_active":"2024-05-27T07:36:02.242689+0000","last_became_peered":"2024-05-27T07:36:02.242689+0000","last_unstale":"2024-05-27T07:39:44.133391+0000","last_undegraded":"2024-05-27T07:39:44.133391+0000","last_fullsized":"2024-05-27T07:39:44.133391+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:44:19.652954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418822+0000","last_change":"2024-05-27T07:36:15.025595+0000","last_active":"2024-05-27T07:39:44.418822+0000","last_peered":"2024-05-27T07:39:44.418822+0000","last_clean":"2024-05-27T07:39:44.418822+0000","last_became_active":"2024-05-27T07:36:15.025177+0000","last_became_peered":"2024-05-27T07:36:15.025177+0000","last_unstale":"2024-05-27T07:39:44.418822+0000","last_undegraded":"2024-05-27T07:39:44.418822+0000","last_fullsized":"2024-05-27T07:39:44.418822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:08.920778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247262+0000","last_change":"2024-05-27T07:36:04.981422+0000","last_active":"2024-05-27T07:39:44.247262+0000","last_peered":"2024-05-27T07:39:44.247262+0000","last_clean":"2024-05-27T07:39:44.247262+0000","last_became_active":"2024-05-27T07:36:04.981100+0000","last_became_peered":"2024-05-27T07:36:04.981100+0000","last_unstale":"2024-05-27T07:39:44.247262+0000","last_undegraded":"2024-05-27T07:39:44.247262+0000","last_fullsized":"2024-05-27T07:39:44.247262+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:01.076309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417708+0000","last_change":"2024-05-27T07:36:01.953727+0000","last_active":"2024-05-27T07:39:44.417708+0000","last_peered":"2024-05-27T07:39:44.417708+0000","last_clean":"2024-05-27T07:39:44.417708+0000","last_became_active":"2024-05-27T07:36:01.953323+0000","last_became_peered":"2024-05-27T07:36:01.953323+0000","last_unstale":"2024-05-27T07:39:44.417708+0000","last_undegraded":"2024-05-27T07:39:44.417708+0000","last_fullsized":"2024-05-27T07:39:44.417708+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:17.186854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417734+0000","last_change":"2024-05-27T07:34:38.289913+0000","last_active":"2024-05-27T07:39:44.417734+0000","last_peered":"2024-05-27T07:39:44.417734+0000","last_clean":"2024-05-27T07:39:44.417734+0000","last_became_active":"2024-05-27T07:34:38.285523+0000","last_became_peered":"2024-05-27T07:34:38.285523+0000","last_unstale":"2024-05-27T07:39:44.417734+0000","last_undegraded":"2024-05-27T07:39:44.417734+0000","last_fullsized":"2024-05-27T07:39:44.417734+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_clean_scrub_stamp":"2024-05-27T07:31:09.455079+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:59.537055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135577+0000","last_change":"2024-05-27T07:36:15.027453+0000","last_active":"2024-05-27T07:39:44.135577+0000","last_peered":"2024-05-27T07:39:44.135577+0000","last_clean":"2024-05-27T07:39:44.135577+0000","last_became_active":"2024-05-27T07:36:15.026872+0000","last_became_peered":"2024-05-27T07:36:15.026872+0000","last_unstale":"2024-05-27T07:39:44.135577+0000","last_undegraded":"2024-05-27T07:39:44.135577+0000","last_fullsized":"2024-05-27T07:39:44.135577+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:32.351924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133387+0000","last_change":"2024-05-27T07:36:04.982041+0000","last_active":"2024-05-27T07:39:44.133387+0000","last_peered":"2024-05-27T07:39:44.133387+0000","last_clean":"2024-05-27T07:39:44.133387+0000","last_became_active":"2024-05-27T07:36:04.981719+0000","last_became_peered":"2024-05-27T07:36:04.981719+0000","last_unstale":"2024-05-27T07:39:44.133387+0000","last_undegraded":"2024-05-27T07:39:44.133387+0000","last_fullsized":"2024-05-27T07:39:44.133387+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:40.061892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417569+0000","last_change":"2024-05-27T07:36:01.952238+0000","last_active":"2024-05-27T07:39:44.417569+0000","last_peered":"2024-05-27T07:39:44.417569+0000","last_clean":"2024-05-27T07:39:44.417569+0000","last_became_active":"2024-05-27T07:36:01.951853+0000","last_became_peered":"2024-05-27T07:36:01.951853+0000","last_unstale":"2024-05-27T07:39:44.417569+0000","last_undegraded":"2024-05-27T07:39:44.417569+0000","last_fullsized":"2024-05-27T07:39:44.417569+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:46:41.317531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417616+0000","last_change":"2024-05-27T07:36:15.021848+0000","last_active":"2024-05-27T07:39:44.417616+0000","last_peered":"2024-05-27T07:39:44.417616+0000","last_clean":"2024-05-27T07:39:44.417616+0000","last_became_active":"2024-05-27T07:36:15.021371+0000","last_became_peered":"2024-05-27T07:36:15.021371+0000","last_unstale":"2024-05-27T07:39:44.417616+0000","last_undegraded":"2024-05-27T07:39:44.417616+0000","last_fullsized":"2024-05-27T07:39:44.417616+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:32:21.741229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246986+0000","last_change":"2024-05-27T07:36:02.244283+0000","last_active":"2024-05-27T07:39:44.246986+0000","last_peered":"2024-05-27T07:39:44.246986+0000","last_clean":"2024-05-27T07:39:44.246986+0000","last_became_active":"2024-05-27T07:36:02.243785+0000","last_became_peered":"2024-05-27T07:36:02.243785+0000","last_unstale":"2024-05-27T07:39:44.246986+0000","last_undegraded":"2024-05-27T07:39:44.246986+0000","last_fullsized":"2024-05-27T07:39:44.246986+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:38.596510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133682+0000","last_change":"2024-05-27T07:36:04.979903+0000","last_active":"2024-05-27T07:39:44.133682+0000","last_peered":"2024-05-27T07:39:44.133682+0000","last_clean":"2024-05-27T07:39:44.133682+0000","last_became_active":"2024-05-27T07:36:04.979373+0000","last_became_peered":"2024-05-27T07:36:04.979373+0000","last_unstale":"2024-05-27T07:39:44.133682+0000","last_undegraded":"2024-05-27T07:39:44.133682+0000","last_fullsized":"2024-05-27T07:39:44.133682+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:44:33.763230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246825+0000","last_change":"2024-05-27T07:36:15.023740+0000","last_active":"2024-05-27T07:39:44.246825+0000","last_peered":"2024-05-27T07:39:44.246825+0000","last_clean":"2024-05-27T07:39:44.246825+0000","last_became_active":"2024-05-27T07:36:15.023364+0000","last_became_peered":"2024-05-27T07:36:15.023364+0000","last_unstale":"2024-05-27T07:39:44.246825+0000","last_undegraded":"2024-05-27T07:39:44.246825+0000","last_fullsized":"2024-05-27T07:39:44.246825+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:21.399065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'46","reported_seq":116,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.372049+0000","last_change":"2024-05-27T07:36:01.951451+0000","last_active":"2024-05-27T07:39:44.372049+0000","last_peered":"2024-05-27T07:39:44.372049+0000","last_clean":"2024-05-27T07:39:44.372049+0000","last_became_active":"2024-05-27T07:36:01.951037+0000","last_became_peered":"2024-05-27T07:36:01.951037+0000","last_unstale":"2024-05-27T07:39:44.372049+0000","last_undegraded":"2024-05-27T07:39:44.372049+0000","last_fullsized":"2024-05-27T07:39:44.372049+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:44.202474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.248504+0000","last_change":"2024-05-27T07:36:04.975392+0000","last_active":"2024-05-27T07:39:44.248504+0000","last_peered":"2024-05-27T07:39:44.248504+0000","last_clean":"2024-05-27T07:39:44.248504+0000","last_became_active":"2024-05-27T07:36:04.975029+0000","last_became_peered":"2024-05-27T07:36:04.975029+0000","last_unstale":"2024-05-27T07:39:44.248504+0000","last_undegraded":"2024-05-27T07:39:44.248504+0000","last_fullsized":"2024-05-27T07:39:44.248504+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:32.022670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363000+0000","last_change":"2024-05-27T07:36:15.026401+0000","last_active":"2024-05-27T07:39:44.363000+0000","last_peered":"2024-05-27T07:39:44.363000+0000","last_clean":"2024-05-27T07:39:44.363000+0000","last_became_active":"2024-05-27T07:36:15.025963+0000","last_became_peered":"2024-05-27T07:36:15.025963+0000","last_unstale":"2024-05-27T07:39:44.363000+0000","last_undegraded":"2024-05-27T07:39:44.363000+0000","last_fullsized":"2024-05-27T07:39:44.363000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:33:47.616099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"78'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418346+0000","last_change":"2024-05-27T07:36:02.228776+0000","last_active":"2024-05-27T07:39:44.418346+0000","last_peered":"2024-05-27T07:39:44.418346+0000","last_clean":"2024-05-27T07:39:44.418346+0000","last_became_active":"2024-05-27T07:36:02.228356+0000","last_became_peered":"2024-05-27T07:36:02.228356+0000","last_unstale":"2024-05-27T07:39:44.418346+0000","last_undegraded":"2024-05-27T07:39:44.418346+0000","last_fullsized":"2024-05-27T07:39:44.418346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:07.289340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418083+0000","last_change":"2024-05-27T07:36:04.986917+0000","last_active":"2024-05-27T07:39:44.418083+0000","last_peered":"2024-05-27T07:39:44.418083+0000","last_clean":"2024-05-27T07:39:44.418083+0000","last_became_active":"2024-05-27T07:36:04.986432+0000","last_became_peered":"2024-05-27T07:36:04.986432+0000","last_unstale":"2024-05-27T07:39:44.418083+0000","last_undegraded":"2024-05-27T07:39:44.418083+0000","last_fullsized":"2024-05-27T07:39:44.418083+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:39:15.011058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418062+0000","last_change":"2024-05-27T07:36:15.032312+0000","last_active":"2024-05-27T07:39:44.418062+0000","last_peered":"2024-05-27T07:39:44.418062+0000","last_clean":"2024-05-27T07:39:44.418062+0000","last_became_active":"2024-05-27T07:36:15.031475+0000","last_became_peered":"2024-05-27T07:36:15.031475+0000","last_unstale":"2024-05-27T07:39:44.418062+0000","last_undegraded":"2024-05-27T07:39:44.418062+0000","last_fullsized":"2024-05-27T07:39:44.418062+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:40.099421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418204+0000","last_change":"2024-05-27T07:36:02.243456+0000","last_active":"2024-05-27T07:39:44.418204+0000","last_peered":"2024-05-27T07:39:44.418204+0000","last_clean":"2024-05-27T07:39:44.418204+0000","last_became_active":"2024-05-27T07:36:02.243164+0000","last_became_peered":"2024-05-27T07:36:02.243164+0000","last_unstale":"2024-05-27T07:39:44.418204+0000","last_undegraded":"2024-05-27T07:39:44.418204+0000","last_fullsized":"2024-05-27T07:39:44.418204+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:33.765834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246597+0000","last_change":"2024-05-27T07:36:04.980192+0000","last_active":"2024-05-27T07:39:44.246597+0000","last_peered":"2024-05-27T07:39:44.246597+0000","last_clean":"2024-05-27T07:39:44.246597+0000","last_became_active":"2024-05-27T07:36:04.979797+0000","last_became_peered":"2024-05-27T07:36:04.979797+0000","last_unstale":"2024-05-27T07:39:44.246597+0000","last_undegraded":"2024-05-27T07:39:44.246597+0000","last_fullsized":"2024-05-27T07:39:44.246597+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:47:37.006746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244501+0000","last_change":"2024-05-27T07:36:15.033153+0000","last_active":"2024-05-27T07:39:44.244501+0000","last_peered":"2024-05-27T07:39:44.244501+0000","last_clean":"2024-05-27T07:39:44.244501+0000","last_became_active":"2024-05-27T07:36:15.032783+0000","last_became_peered":"2024-05-27T07:36:15.032783+0000","last_unstale":"2024-05-27T07:39:44.244501+0000","last_undegraded":"2024-05-27T07:39:44.244501+0000","last_fullsized":"2024-05-27T07:39:44.244501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:31:26.237222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247148+0000","last_change":"2024-05-27T07:36:01.952754+0000","last_active":"2024-05-27T07:39:44.247148+0000","last_peered":"2024-05-27T07:39:44.247148+0000","last_clean":"2024-05-27T07:39:44.247148+0000","last_became_active":"2024-05-27T07:36:01.952397+0000","last_became_peered":"2024-05-27T07:36:01.952397+0000","last_unstale":"2024-05-27T07:39:44.247148+0000","last_undegraded":"2024-05-27T07:39:44.247148+0000","last_fullsized":"2024-05-27T07:39:44.247148+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:04.350169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417457+0000","last_change":"2024-05-27T07:36:04.979193+0000","last_active":"2024-05-27T07:39:44.417457+0000","last_peered":"2024-05-27T07:39:44.417457+0000","last_clean":"2024-05-27T07:39:44.417457+0000","last_became_active":"2024-05-27T07:36:04.978679+0000","last_became_peered":"2024-05-27T07:36:04.978679+0000","last_unstale":"2024-05-27T07:39:44.417457+0000","last_undegraded":"2024-05-27T07:39:44.417457+0000","last_fullsized":"2024-05-27T07:39:44.417457+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:59.310284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418066+0000","last_change":"2024-05-27T07:37:46.153707+0000","last_active":"2024-05-27T07:39:44.418066+0000","last_peered":"2024-05-27T07:39:44.418066+0000","last_clean":"2024-05-27T07:39:44.418066+0000","last_became_active":"2024-05-27T07:37:46.148703+0000","last_became_peered":"2024-05-27T07:37:46.148703+0000","last_unstale":"2024-05-27T07:39:44.418066+0000","last_undegraded":"2024-05-27T07:39:44.418066+0000","last_fullsized":"2024-05-27T07:39:44.418066+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:50:20.099467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417476+0000","last_change":"2024-05-27T07:36:45.140940+0000","last_active":"2024-05-27T07:39:44.417476+0000","last_peered":"2024-05-27T07:39:44.417476+0000","last_clean":"2024-05-27T07:39:44.417476+0000","last_became_active":"2024-05-27T07:36:45.140511+0000","last_became_peered":"2024-05-27T07:36:45.140511+0000","last_unstale":"2024-05-27T07:39:44.417476+0000","last_undegraded":"2024-05-27T07:39:44.417476+0000","last_fullsized":"2024-05-27T07:39:44.417476+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:43:00.507442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247777+0000","last_change":"2024-05-27T07:36:04.974709+0000","last_active":"2024-05-27T07:39:44.247777+0000","last_peered":"2024-05-27T07:39:44.247777+0000","last_clean":"2024-05-27T07:39:44.247777+0000","last_became_active":"2024-05-27T07:36:04.974147+0000","last_became_peered":"2024-05-27T07:36:04.974147+0000","last_unstale":"2024-05-27T07:39:44.247777+0000","last_undegraded":"2024-05-27T07:39:44.247777+0000","last_fullsized":"2024-05-27T07:39:44.247777+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:05:11.251124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417872+0000","last_change":"2024-05-27T07:36:15.023187+0000","last_active":"2024-05-27T07:39:44.417872+0000","last_peered":"2024-05-27T07:39:44.417872+0000","last_clean":"2024-05-27T07:39:44.417872+0000","last_became_active":"2024-05-27T07:36:15.022687+0000","last_became_peered":"2024-05-27T07:36:15.022687+0000","last_unstale":"2024-05-27T07:39:44.417872+0000","last_undegraded":"2024-05-27T07:39:44.417872+0000","last_fullsized":"2024-05-27T07:39:44.417872+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:51.713222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"78'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246746+0000","last_change":"2024-05-27T07:36:01.954751+0000","last_active":"2024-05-27T07:39:44.246746+0000","last_peered":"2024-05-27T07:39:44.246746+0000","last_clean":"2024-05-27T07:39:44.246746+0000","last_became_active":"2024-05-27T07:36:01.954173+0000","last_became_peered":"2024-05-27T07:36:01.954173+0000","last_unstale":"2024-05-27T07:39:44.246746+0000","last_undegraded":"2024-05-27T07:39:44.246746+0000","last_fullsized":"2024-05-27T07:39:44.246746+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:45:10.504504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417752+0000","last_change":"2024-05-27T07:36:04.980294+0000","last_active":"2024-05-27T07:39:44.417752+0000","last_peered":"2024-05-27T07:39:44.417752+0000","last_clean":"2024-05-27T07:39:44.417752+0000","last_became_active":"2024-05-27T07:36:04.979853+0000","last_became_peered":"2024-05-27T07:36:04.979853+0000","last_unstale":"2024-05-27T07:39:44.417752+0000","last_undegraded":"2024-05-27T07:39:44.417752+0000","last_fullsized":"2024-05-27T07:39:44.417752+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:43.131216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418970+0000","last_change":"2024-05-27T07:36:15.025854+0000","last_active":"2024-05-27T07:39:44.418970+0000","last_peered":"2024-05-27T07:39:44.418970+0000","last_clean":"2024-05-27T07:39:44.418970+0000","last_became_active":"2024-05-27T07:36:15.025412+0000","last_became_peered":"2024-05-27T07:36:15.025412+0000","last_unstale":"2024-05-27T07:39:44.418970+0000","last_undegraded":"2024-05-27T07:39:44.418970+0000","last_fullsized":"2024-05-27T07:39:44.418970+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:49.904811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363520+0000","last_change":"2024-05-27T07:36:02.243764+0000","last_active":"2024-05-27T07:39:44.363520+0000","last_peered":"2024-05-27T07:39:44.363520+0000","last_clean":"2024-05-27T07:39:44.363520+0000","last_became_active":"2024-05-27T07:36:02.242935+0000","last_became_peered":"2024-05-27T07:36:02.242935+0000","last_unstale":"2024-05-27T07:39:44.363520+0000","last_undegraded":"2024-05-27T07:39:44.363520+0000","last_fullsized":"2024-05-27T07:39:44.363520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:27.535658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419316+0000","last_change":"2024-05-27T07:36:04.975291+0000","last_active":"2024-05-27T07:39:44.419316+0000","last_peered":"2024-05-27T07:39:44.419316+0000","last_clean":"2024-05-27T07:39:44.419316+0000","last_became_active":"2024-05-27T07:36:04.974868+0000","last_became_peered":"2024-05-27T07:36:04.974868+0000","last_unstale":"2024-05-27T07:39:44.419316+0000","last_undegraded":"2024-05-27T07:39:44.419316+0000","last_fullsized":"2024-05-27T07:39:44.419316+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:13.890494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417509+0000","last_change":"2024-05-27T07:36:15.031812+0000","last_active":"2024-05-27T07:39:44.417509+0000","last_peered":"2024-05-27T07:39:44.417509+0000","last_clean":"2024-05-27T07:39:44.417509+0000","last_became_active":"2024-05-27T07:36:15.031399+0000","last_became_peered":"2024-05-27T07:36:15.031399+0000","last_unstale":"2024-05-27T07:39:44.417509+0000","last_undegraded":"2024-05-27T07:39:44.417509+0000","last_fullsized":"2024-05-27T07:39:44.417509+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.595579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.366476+0000","last_change":"2024-05-27T07:36:01.954581+0000","last_active":"2024-05-27T07:39:44.366476+0000","last_peered":"2024-05-27T07:39:44.366476+0000","last_clean":"2024-05-27T07:39:44.366476+0000","last_became_active":"2024-05-27T07:36:01.953564+0000","last_became_peered":"2024-05-27T07:36:01.953564+0000","last_unstale":"2024-05-27T07:39:44.366476+0000","last_undegraded":"2024-05-27T07:39:44.366476+0000","last_fullsized":"2024-05-27T07:39:44.366476+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:56.866738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418767+0000","last_change":"2024-05-27T07:36:45.141160+0000","last_active":"2024-05-27T07:39:44.418767+0000","last_peered":"2024-05-27T07:39:44.418767+0000","last_clean":"2024-05-27T07:39:44.418767+0000","last_became_active":"2024-05-27T07:36:45.140840+0000","last_became_peered":"2024-05-27T07:36:45.140840+0000","last_unstale":"2024-05-27T07:39:44.418767+0000","last_undegraded":"2024-05-27T07:39:44.418767+0000","last_fullsized":"2024-05-27T07:39:44.418767+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:31.676808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134842+0000","last_change":"2024-05-27T07:36:15.026483+0000","last_active":"2024-05-27T07:39:44.134842+0000","last_peered":"2024-05-27T07:39:44.134842+0000","last_clean":"2024-05-27T07:39:44.134842+0000","last_became_active":"2024-05-27T07:36:15.025955+0000","last_became_peered":"2024-05-27T07:36:15.025955+0000","last_unstale":"2024-05-27T07:39:44.134842+0000","last_undegraded":"2024-05-27T07:39:44.134842+0000","last_fullsized":"2024-05-27T07:39:44.134842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:25.468146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133958+0000","last_change":"2024-05-27T07:36:01.952505+0000","last_active":"2024-05-27T07:39:44.133958+0000","last_peered":"2024-05-27T07:39:44.133958+0000","last_clean":"2024-05-27T07:39:44.133958+0000","last_became_active":"2024-05-27T07:36:01.952088+0000","last_became_peered":"2024-05-27T07:36:01.952088+0000","last_unstale":"2024-05-27T07:39:44.133958+0000","last_undegraded":"2024-05-27T07:39:44.133958+0000","last_fullsized":"2024-05-27T07:39:44.133958+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:09.235719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419335+0000","last_change":"2024-05-27T07:36:04.974331+0000","last_active":"2024-05-27T07:39:44.419335+0000","last_peered":"2024-05-27T07:39:44.419335+0000","last_clean":"2024-05-27T07:39:44.419335+0000","last_became_active":"2024-05-27T07:36:04.973825+0000","last_became_peered":"2024-05-27T07:36:04.973825+0000","last_unstale":"2024-05-27T07:39:44.419335+0000","last_undegraded":"2024-05-27T07:39:44.419335+0000","last_fullsized":"2024-05-27T07:39:44.419335+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:39:42.917309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417542+0000","last_change":"2024-05-27T07:36:45.141224+0000","last_active":"2024-05-27T07:39:44.417542+0000","last_peered":"2024-05-27T07:39:44.417542+0000","last_clean":"2024-05-27T07:39:44.417542+0000","last_became_active":"2024-05-27T07:36:45.140776+0000","last_became_peered":"2024-05-27T07:36:45.140776+0000","last_unstale":"2024-05-27T07:39:44.417542+0000","last_undegraded":"2024-05-27T07:39:44.417542+0000","last_fullsized":"2024-05-27T07:39:44.417542+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:36:59.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363303+0000","last_change":"2024-05-27T07:36:04.988247+0000","last_active":"2024-05-27T07:39:44.363303+0000","last_peered":"2024-05-27T07:39:44.363303+0000","last_clean":"2024-05-27T07:39:44.363303+0000","last_became_active":"2024-05-27T07:36:04.987825+0000","last_became_peered":"2024-05-27T07:36:04.987825+0000","last_unstale":"2024-05-27T07:39:44.363303+0000","last_undegraded":"2024-05-27T07:39:44.363303+0000","last_fullsized":"2024-05-27T07:39:44.363303+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:32.499260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363087+0000","last_change":"2024-05-27T07:36:04.979395+0000","last_active":"2024-05-27T07:39:44.363087+0000","last_peered":"2024-05-27T07:39:44.363087+0000","last_clean":"2024-05-27T07:39:44.363087+0000","last_became_active":"2024-05-27T07:36:04.978983+0000","last_became_peered":"2024-05-27T07:36:04.978983+0000","last_unstale":"2024-05-27T07:39:44.363087+0000","last_undegraded":"2024-05-27T07:39:44.363087+0000","last_fullsized":"2024-05-27T07:39:44.363087+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:53:54.071958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417704+0000","last_change":"2024-05-27T07:36:04.989442+0000","last_active":"2024-05-27T07:39:44.417704+0000","last_peered":"2024-05-27T07:39:44.417704+0000","last_clean":"2024-05-27T07:39:44.417704+0000","last_became_active":"2024-05-27T07:36:04.989079+0000","last_became_peered":"2024-05-27T07:36:04.989079+0000","last_unstale":"2024-05-27T07:39:44.417704+0000","last_undegraded":"2024-05-27T07:39:44.417704+0000","last_fullsized":"2024-05-27T07:39:44.417704+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:39.411255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133776+0000","last_change":"2024-05-27T07:36:04.981289+0000","last_active":"2024-05-27T07:39:44.133776+0000","last_peered":"2024-05-27T07:39:44.133776+0000","last_clean":"2024-05-27T07:39:44.133776+0000","last_became_active":"2024-05-27T07:36:04.980924+0000","last_became_peered":"2024-05-27T07:36:04.980924+0000","last_unstale":"2024-05-27T07:39:44.133776+0000","last_undegraded":"2024-05-27T07:39:44.133776+0000","last_fullsized":"2024-05-27T07:39:44.133776+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:15:04.741303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417916+0000","last_change":"2024-05-27T07:36:04.985712+0000","last_active":"2024-05-27T07:39:44.417916+0000","last_peered":"2024-05-27T07:39:44.417916+0000","last_clean":"2024-05-27T07:39:44.417916+0000","last_became_active":"2024-05-27T07:36:04.983786+0000","last_became_peered":"2024-05-27T07:36:04.983786+0000","last_unstale":"2024-05-27T07:39:44.417916+0000","last_undegraded":"2024-05-27T07:39:44.417916+0000","last_fullsized":"2024-05-27T07:39:44.417916+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:34.549557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363470+0000","last_change":"2024-05-27T07:36:04.991176+0000","last_active":"2024-05-27T07:39:44.363470+0000","last_peered":"2024-05-27T07:39:44.363470+0000","last_clean":"2024-05-27T07:39:44.363470+0000","last_became_active":"2024-05-27T07:36:04.990774+0000","last_became_peered":"2024-05-27T07:36:04.990774+0000","last_unstale":"2024-05-27T07:39:44.363470+0000","last_undegraded":"2024-05-27T07:39:44.363470+0000","last_fullsized":"2024-05-27T07:39:44.363470+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:37.307581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.248721+0000","last_change":"2024-05-27T07:36:04.982337+0000","last_active":"2024-05-27T07:39:44.248721+0000","last_peered":"2024-05-27T07:39:44.248721+0000","last_clean":"2024-05-27T07:39:44.248721+0000","last_became_active":"2024-05-27T07:36:04.981988+0000","last_became_peered":"2024-05-27T07:36:04.981988+0000","last_unstale":"2024-05-27T07:39:44.248721+0000","last_undegraded":"2024-05-27T07:39:44.248721+0000","last_fullsized":"2024-05-27T07:39:44.248721+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:17:39.763459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134985+0000","last_change":"2024-05-27T07:36:04.982321+0000","last_active":"2024-05-27T07:39:44.134985+0000","last_peered":"2024-05-27T07:39:44.134985+0000","last_clean":"2024-05-27T07:39:44.134985+0000","last_became_active":"2024-05-27T07:36:04.981690+0000","last_became_peered":"2024-05-27T07:36:04.981690+0000","last_unstale":"2024-05-27T07:39:44.134985+0000","last_undegraded":"2024-05-27T07:39:44.134985+0000","last_fullsized":"2024-05-27T07:39:44.134985+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:10.181606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247215+0000","last_change":"2024-05-27T07:36:04.979411+0000","last_active":"2024-05-27T07:39:44.247215+0000","last_peered":"2024-05-27T07:39:44.247215+0000","last_clean":"2024-05-27T07:39:44.247215+0000","last_became_active":"2024-05-27T07:36:04.979041+0000","last_became_peered":"2024-05-27T07:36:04.979041+0000","last_unstale":"2024-05-27T07:39:44.247215+0000","last_undegraded":"2024-05-27T07:39:44.247215+0000","last_fullsized":"2024-05-27T07:39:44.247215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:57:29.165997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419164+0000","last_change":"2024-05-27T07:36:04.988564+0000","last_active":"2024-05-27T07:39:44.419164+0000","last_peered":"2024-05-27T07:39:44.419164+0000","last_clean":"2024-05-27T07:39:44.419164+0000","last_became_active":"2024-05-27T07:36:04.986313+0000","last_became_peered":"2024-05-27T07:36:04.986313+0000","last_unstale":"2024-05-27T07:39:44.419164+0000","last_undegraded":"2024-05-27T07:39:44.419164+0000","last_fullsized":"2024-05-27T07:39:44.419164+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:38.809386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133635+0000","last_change":"2024-05-27T07:36:04.974704+0000","last_active":"2024-05-27T07:39:44.133635+0000","last_peered":"2024-05-27T07:39:44.133635+0000","last_clean":"2024-05-27T07:39:44.133635+0000","last_became_active":"2024-05-27T07:36:04.974311+0000","last_became_peered":"2024-05-27T07:36:04.974311+0000","last_unstale":"2024-05-27T07:39:44.133635+0000","last_undegraded":"2024-05-27T07:39:44.133635+0000","last_fullsized":"2024-05-27T07:39:44.133635+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:43.021149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417756+0000","last_change":"2024-05-27T07:36:04.983951+0000","last_active":"2024-05-27T07:39:44.417756+0000","last_peered":"2024-05-27T07:39:44.417756+0000","last_clean":"2024-05-27T07:39:44.417756+0000","last_became_active":"2024-05-27T07:36:04.983489+0000","last_became_peered":"2024-05-27T07:36:04.983489+0000","last_unstale":"2024-05-27T07:39:44.417756+0000","last_undegraded":"2024-05-27T07:39:44.417756+0000","last_fullsized":"2024-05-27T07:39:44.417756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:04.229523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107193,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":596971,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":435,"num_read_kb":324,"num_write":209,"num_write_kb":1415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":1790913,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710779,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":524552,"data_compressed":35398,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877907008,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":1185667,"data_compressed":71129,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070534,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":515188,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266762,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":515188,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462991,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":1119576,"data_compressed":71129,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626518,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":520434,"data_compressed":35398,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":581085,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":520524,"data_compressed":35398,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1069056,"data_stored":591825,"data_compressed":35398,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182501,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":1045354,"data_compressed":60700,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986272,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":514628,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822746,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1069056,"data_stored":1246680,"data_compressed":77285,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":71357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":71391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":524920,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":131782,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":203151,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:39:47.054 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-05-27T07:39:47.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T07:39:47.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.205+0000 7f5ab3084640 1 -- 172.21.15.114:0/2770242712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aac108a20 msgr2=0x7f5aac10ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:47.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.205+0000 7f5ab3084640 1 --2- 172.21.15.114:0/2770242712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aac108a20 0x7f5aac10ae40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5a94009900 tx=0x7f5a9402a410 comp rx=0 tx=0).stop 2024-05-27T07:39:47.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.205+0000 7f5ab3084640 1 -- 172.21.15.114:0/2770242712 shutdown_connections 2024-05-27T07:39:47.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.205+0000 7f5ab3084640 1 --2- 172.21.15.114:0/2770242712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aac108a20 0x7f5aac10ae40 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.205+0000 7f5ab3084640 1 --2- 172.21.15.114:0/2770242712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac103d20 0x7f5aac104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.205+0000 7f5ab3084640 1 --2- 172.21.15.114:0/2770242712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5aac102ae0 0x7f5aac1037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.205+0000 7f5ab3084640 1 -- 172.21.15.114:0/2770242712 >> 172.21.15.114:0/2770242712 conn(0x7f5aac0fdcf0 msgr2=0x7f5aac100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:47.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.205+0000 7f5ab3084640 1 -- 172.21.15.114:0/2770242712 shutdown_connections 2024-05-27T07:39:47.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.205+0000 7f5ab3084640 1 -- 172.21.15.114:0/2770242712 wait complete. 2024-05-27T07:39:47.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.206+0000 7f5ab3084640 1 Processor -- start 2024-05-27T07:39:47.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.206+0000 7f5ab3084640 1 -- start start 2024-05-27T07:39:47.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.206+0000 7f5ab3084640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aac102ae0 0x7f5aac19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:47.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.206+0000 7f5ab3084640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5aac103d20 0x7f5aac19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:47.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.206+0000 7f5ab3084640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac108a20 0x7f5aac1a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.206+0000 7f5ab3084640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac10f710 con 0x7f5aac102ae0 2024-05-27T07:39:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.206+0000 7f5ab3084640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac10f590 con 0x7f5aac108a20 2024-05-27T07:39:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.206+0000 7f5ab3084640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac10f890 con 0x7f5aac103d20 2024-05-27T07:39:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.206+0000 7f5ab0e20640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aac102ae0 0x7f5aac19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.206+0000 7f5ab0e20640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aac102ae0 0x7f5aac19a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43572/0 (socket says 172.21.15.114:43572) 2024-05-27T07:39:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.206+0000 7f5ab0e20640 1 -- 172.21.15.114:0/962055500 learned_addr learned my addr 172.21.15.114:0/962055500 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.206+0000 7f5ab1621640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac108a20 0x7f5aac1a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.206+0000 7f5aabfff640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5aac103d20 0x7f5aac19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.207+0000 7f5ab0e20640 1 -- 172.21.15.114:0/962055500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5aac103d20 msgr2=0x7f5aac19a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.207+0000 7f5ab0e20640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5aac103d20 0x7f5aac19a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.207+0000 7f5ab0e20640 1 -- 172.21.15.114:0/962055500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac108a20 msgr2=0x7f5aac1a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.207+0000 7f5ab0e20640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac108a20 0x7f5aac1a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.207+0000 7f5ab0e20640 1 -- 172.21.15.114:0/962055500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aac1a1e20 con 0x7f5aac102ae0 2024-05-27T07:39:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.207+0000 7f5ab0e20640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aac102ae0 0x7f5aac19a2e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400e9f0 tx=0x7f5aa400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.207+0000 7f5aa9ffb640 1 -- 172.21.15.114:0/962055500 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa400ccc0 con 0x7f5aac102ae0 2024-05-27T07:39:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.207+0000 7f5aa9ffb640 1 -- 172.21.15.114:0/962055500 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5aa40035f0 con 0x7f5aac102ae0 2024-05-27T07:39:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.207+0000 7f5ab3084640 1 -- 172.21.15.114:0/962055500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aac1a2110 con 0x7f5aac102ae0 2024-05-27T07:39:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.207+0000 7f5aa9ffb640 1 -- 172.21.15.114:0/962055500 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa400ce60 con 0x7f5aac102ae0 2024-05-27T07:39:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.207+0000 7f5ab3084640 1 -- 172.21.15.114:0/962055500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aac1a2660 con 0x7f5aac102ae0 2024-05-27T07:39:47.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.208+0000 7f5ab3084640 1 -- 172.21.15.114:0/962055500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aac107830 con 0x7f5aac102ae0 2024-05-27T07:39:47.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.210+0000 7f5aa9ffb640 1 -- 172.21.15.114:0/962055500 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 22) ==== 149773+0+0 (secure 0 0 0) 0x7f5aa4003aa0 con 0x7f5aac102ae0 2024-05-27T07:39:47.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.210+0000 7f5aa9ffb640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5a980b29e0 0x7f5a980b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:47.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.211+0000 7f5aa9ffb640 1 -- 172.21.15.114:0/962055500 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f5aa40e2d70 con 0x7f5aac102ae0 2024-05-27T07:39:47.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.211+0000 7f5aabfff640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5a980b29e0 0x7f5a980b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:47.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.211+0000 7f5aabfff640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5a980b29e0 0x7f5a980b4ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5aac19b380 tx=0x7f5a9c009760 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:47.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.214+0000 7f5aa9ffb640 1 -- 172.21.15.114:0/962055500 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5aa40aa3a0 con 0x7f5aac102ae0 2024-05-27T07:39:47.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.403+0000 7f5ab3084640 1 -- 172.21.15.114:0/962055500 --> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5aac063100 con 0x7f5a980b29e0 2024-05-27T07:39:47.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.413+0000 7f5aa9ffb640 1 -- 172.21.15.114:0/962055500 <== mgr.15138 v2:172.21.15.114:6800/3311646739 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339584 (secure 0 0 0) 0x7f5aac063100 con 0x7f5a980b29e0 2024-05-27T07:39:47.414 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:47.416 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-05-27T07:39:47.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.417+0000 7f5ab3084640 1 -- 172.21.15.114:0/962055500 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5a980b29e0 msgr2=0x7f5a980b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:47.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.417+0000 7f5ab3084640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5a980b29e0 0x7f5a980b4ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5aac19b380 tx=0x7f5a9c009760 comp rx=0 tx=0).stop 2024-05-27T07:39:47.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.417+0000 7f5ab3084640 1 -- 172.21.15.114:0/962055500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aac102ae0 msgr2=0x7f5aac19a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:47.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.417+0000 7f5ab3084640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aac102ae0 0x7f5aac19a2e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400e9f0 tx=0x7f5aa400eeb0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.417+0000 7f5ab3084640 1 -- 172.21.15.114:0/962055500 shutdown_connections 2024-05-27T07:39:47.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.417+0000 7f5ab3084640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5a980b29e0 0x7f5a980b4ef0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.417+0000 7f5ab3084640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac108a20 0x7f5aac1a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.417+0000 7f5ab3084640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5aac103d20 0x7f5aac19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.417+0000 7f5ab3084640 1 --2- 172.21.15.114:0/962055500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aac102ae0 0x7f5aac19a2e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:47.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.417+0000 7f5ab3084640 1 -- 172.21.15.114:0/962055500 >> 172.21.15.114:0/962055500 conn(0x7f5aac0fdcf0 msgr2=0x7f5aac104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:47.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.417+0000 7f5ab3084640 1 -- 172.21.15.114:0/962055500 shutdown_connections 2024-05-27T07:39:47.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:47.417+0000 7f5ab3084640 1 -- 172.21.15.114:0/962055500 wait complete. 2024-05-27T07:39:47.426 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-05-27T07:39:46.148375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1187573,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":516,"num_read_kb":393,"num_write":334,"num_write_kb":3567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2951979,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":749,"ondisk_log_size":749,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369520,"kb_used_data":12012,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572304,"statfs":{"total":1151940427776,"available":1151562039296,"internally_reserved":0,"allocated":12300288,"data_stored":8880701,"data_compressed":551551,"data_compressed_allocated":749568,"data_compressed_original":3555328,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.993045"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418062+0000","last_change":"2024-05-27T07:36:04.980329+0000","last_active":"2024-05-27T07:39:44.418062+0000","last_peered":"2024-05-27T07:39:44.418062+0000","last_clean":"2024-05-27T07:39:44.418062+0000","last_became_active":"2024-05-27T07:36:04.980006+0000","last_became_peered":"2024-05-27T07:36:04.980006+0000","last_unstale":"2024-05-27T07:39:44.418062+0000","last_undegraded":"2024-05-27T07:39:44.418062+0000","last_fullsized":"2024-05-27T07:39:44.418062+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:51:56.728465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417978+0000","last_change":"2024-05-27T07:36:04.983429+0000","last_active":"2024-05-27T07:39:44.417978+0000","last_peered":"2024-05-27T07:39:44.417978+0000","last_clean":"2024-05-27T07:39:44.417978+0000","last_became_active":"2024-05-27T07:36:04.983099+0000","last_became_peered":"2024-05-27T07:36:04.983099+0000","last_unstale":"2024-05-27T07:39:44.417978+0000","last_undegraded":"2024-05-27T07:39:44.417978+0000","last_fullsized":"2024-05-27T07:39:44.417978+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:18:51.059717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417722+0000","last_change":"2024-05-27T07:36:04.993319+0000","last_active":"2024-05-27T07:39:44.417722+0000","last_peered":"2024-05-27T07:39:44.417722+0000","last_clean":"2024-05-27T07:39:44.417722+0000","last_became_active":"2024-05-27T07:36:04.992983+0000","last_became_peered":"2024-05-27T07:36:04.992983+0000","last_unstale":"2024-05-27T07:39:44.417722+0000","last_undegraded":"2024-05-27T07:39:44.417722+0000","last_fullsized":"2024-05-27T07:39:44.417722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:50.800194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244278+0000","last_change":"2024-05-27T07:36:45.140980+0000","last_active":"2024-05-27T07:39:44.244278+0000","last_peered":"2024-05-27T07:39:44.244278+0000","last_clean":"2024-05-27T07:39:44.244278+0000","last_became_active":"2024-05-27T07:36:45.140551+0000","last_became_peered":"2024-05-27T07:36:45.140551+0000","last_unstale":"2024-05-27T07:39:44.244278+0000","last_undegraded":"2024-05-27T07:39:44.244278+0000","last_fullsized":"2024-05-27T07:39:44.244278+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:59:21.646306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246654+0000","last_change":"2024-05-27T07:36:04.977404+0000","last_active":"2024-05-27T07:39:44.246654+0000","last_peered":"2024-05-27T07:39:44.246654+0000","last_clean":"2024-05-27T07:39:44.246654+0000","last_became_active":"2024-05-27T07:36:04.976991+0000","last_became_peered":"2024-05-27T07:36:04.976991+0000","last_unstale":"2024-05-27T07:39:44.246654+0000","last_undegraded":"2024-05-27T07:39:44.246654+0000","last_fullsized":"2024-05-27T07:39:44.246654+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:56.532586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133579+0000","last_change":"2024-05-27T07:36:04.982149+0000","last_active":"2024-05-27T07:39:44.133579+0000","last_peered":"2024-05-27T07:39:44.133579+0000","last_clean":"2024-05-27T07:39:44.133579+0000","last_became_active":"2024-05-27T07:36:04.981751+0000","last_became_peered":"2024-05-27T07:36:04.981751+0000","last_unstale":"2024-05-27T07:39:44.133579+0000","last_undegraded":"2024-05-27T07:39:44.133579+0000","last_fullsized":"2024-05-27T07:39:44.133579+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:42.773458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417510+0000","last_change":"2024-05-27T07:36:04.987491+0000","last_active":"2024-05-27T07:39:44.417510+0000","last_peered":"2024-05-27T07:39:44.417510+0000","last_clean":"2024-05-27T07:39:44.417510+0000","last_became_active":"2024-05-27T07:36:04.987059+0000","last_became_peered":"2024-05-27T07:36:04.987059+0000","last_unstale":"2024-05-27T07:39:44.417510+0000","last_undegraded":"2024-05-27T07:39:44.417510+0000","last_fullsized":"2024-05-27T07:39:44.417510+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:57.611253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.243516+0000","last_change":"2024-05-27T07:36:45.126961+0000","last_active":"2024-05-27T07:39:44.243516+0000","last_peered":"2024-05-27T07:39:44.243516+0000","last_clean":"2024-05-27T07:39:44.243516+0000","last_became_active":"2024-05-27T07:36:45.126653+0000","last_became_peered":"2024-05-27T07:36:45.126653+0000","last_unstale":"2024-05-27T07:39:44.243516+0000","last_undegraded":"2024-05-27T07:39:44.243516+0000","last_fullsized":"2024-05-27T07:39:44.243516+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:35.586128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418769+0000","last_change":"2024-05-27T07:36:04.990469+0000","last_active":"2024-05-27T07:39:44.418769+0000","last_peered":"2024-05-27T07:39:44.418769+0000","last_clean":"2024-05-27T07:39:44.418769+0000","last_became_active":"2024-05-27T07:36:04.989771+0000","last_became_peered":"2024-05-27T07:36:04.989771+0000","last_unstale":"2024-05-27T07:39:44.418769+0000","last_undegraded":"2024-05-27T07:39:44.418769+0000","last_fullsized":"2024-05-27T07:39:44.418769+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:53.593566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134126+0000","last_change":"2024-05-27T07:36:04.984746+0000","last_active":"2024-05-27T07:39:44.134126+0000","last_peered":"2024-05-27T07:39:44.134126+0000","last_clean":"2024-05-27T07:39:44.134126+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:39:44.134126+0000","last_undegraded":"2024-05-27T07:39:44.134126+0000","last_fullsized":"2024-05-27T07:39:44.134126+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:18.071981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419074+0000","last_change":"2024-05-27T07:36:04.977167+0000","last_active":"2024-05-27T07:39:44.419074+0000","last_peered":"2024-05-27T07:39:44.419074+0000","last_clean":"2024-05-27T07:39:44.419074+0000","last_became_active":"2024-05-27T07:36:04.976788+0000","last_became_peered":"2024-05-27T07:36:04.976788+0000","last_unstale":"2024-05-27T07:39:44.419074+0000","last_undegraded":"2024-05-27T07:39:44.419074+0000","last_fullsized":"2024-05-27T07:39:44.419074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:59:44.574191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418351+0000","last_change":"2024-05-27T07:36:04.986594+0000","last_active":"2024-05-27T07:39:44.418351+0000","last_peered":"2024-05-27T07:39:44.418351+0000","last_clean":"2024-05-27T07:39:44.418351+0000","last_became_active":"2024-05-27T07:36:04.985161+0000","last_became_peered":"2024-05-27T07:36:04.985161+0000","last_unstale":"2024-05-27T07:39:44.418351+0000","last_undegraded":"2024-05-27T07:39:44.418351+0000","last_fullsized":"2024-05-27T07:39:44.418351+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.239461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133787+0000","last_change":"2024-05-27T07:36:04.984781+0000","last_active":"2024-05-27T07:39:44.133787+0000","last_peered":"2024-05-27T07:39:44.133787+0000","last_clean":"2024-05-27T07:39:44.133787+0000","last_became_active":"2024-05-27T07:36:04.984462+0000","last_became_peered":"2024-05-27T07:36:04.984462+0000","last_unstale":"2024-05-27T07:39:44.133787+0000","last_undegraded":"2024-05-27T07:39:44.133787+0000","last_fullsized":"2024-05-27T07:39:44.133787+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:40.051692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417506+0000","last_change":"2024-05-27T07:36:04.978360+0000","last_active":"2024-05-27T07:39:44.417506+0000","last_peered":"2024-05-27T07:39:44.417506+0000","last_clean":"2024-05-27T07:39:44.417506+0000","last_became_active":"2024-05-27T07:36:04.978105+0000","last_became_peered":"2024-05-27T07:36:04.978105+0000","last_unstale":"2024-05-27T07:39:44.417506+0000","last_undegraded":"2024-05-27T07:39:44.417506+0000","last_fullsized":"2024-05-27T07:39:44.417506+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:04:47.257529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417507+0000","last_change":"2024-05-27T07:36:04.986510+0000","last_active":"2024-05-27T07:39:44.417507+0000","last_peered":"2024-05-27T07:39:44.417507+0000","last_clean":"2024-05-27T07:39:44.417507+0000","last_became_active":"2024-05-27T07:36:04.986150+0000","last_became_peered":"2024-05-27T07:36:04.986150+0000","last_unstale":"2024-05-27T07:39:44.417507+0000","last_undegraded":"2024-05-27T07:39:44.417507+0000","last_fullsized":"2024-05-27T07:39:44.417507+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:07.349470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418178+0000","last_change":"2024-05-27T07:36:04.979880+0000","last_active":"2024-05-27T07:39:44.418178+0000","last_peered":"2024-05-27T07:39:44.418178+0000","last_clean":"2024-05-27T07:39:44.418178+0000","last_became_active":"2024-05-27T07:36:04.979516+0000","last_became_peered":"2024-05-27T07:36:04.979516+0000","last_unstale":"2024-05-27T07:39:44.418178+0000","last_undegraded":"2024-05-27T07:39:44.418178+0000","last_fullsized":"2024-05-27T07:39:44.418178+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:41:03.964277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417955+0000","last_change":"2024-05-27T07:36:04.980231+0000","last_active":"2024-05-27T07:39:44.417955+0000","last_peered":"2024-05-27T07:39:44.417955+0000","last_clean":"2024-05-27T07:39:44.417955+0000","last_became_active":"2024-05-27T07:36:04.979888+0000","last_became_peered":"2024-05-27T07:36:04.979888+0000","last_unstale":"2024-05-27T07:39:44.417955+0000","last_undegraded":"2024-05-27T07:39:44.417955+0000","last_fullsized":"2024-05-27T07:39:44.417955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.838119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244612+0000","last_change":"2024-05-27T07:36:04.990890+0000","last_active":"2024-05-27T07:39:44.244612+0000","last_peered":"2024-05-27T07:39:44.244612+0000","last_clean":"2024-05-27T07:39:44.244612+0000","last_became_active":"2024-05-27T07:36:04.990581+0000","last_became_peered":"2024-05-27T07:36:04.990581+0000","last_unstale":"2024-05-27T07:39:44.244612+0000","last_undegraded":"2024-05-27T07:39:44.244612+0000","last_fullsized":"2024-05-27T07:39:44.244612+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:33.054395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363804+0000","last_change":"2024-05-27T07:36:04.981620+0000","last_active":"2024-05-27T07:39:44.363804+0000","last_peered":"2024-05-27T07:39:44.363804+0000","last_clean":"2024-05-27T07:39:44.363804+0000","last_became_active":"2024-05-27T07:36:04.981290+0000","last_became_peered":"2024-05-27T07:36:04.981290+0000","last_unstale":"2024-05-27T07:39:44.363804+0000","last_undegraded":"2024-05-27T07:39:44.363804+0000","last_fullsized":"2024-05-27T07:39:44.363804+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:50.287380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363074+0000","last_change":"2024-05-27T07:36:15.030740+0000","last_active":"2024-05-27T07:39:44.363074+0000","last_peered":"2024-05-27T07:39:44.363074+0000","last_clean":"2024-05-27T07:39:44.363074+0000","last_became_active":"2024-05-27T07:36:15.029738+0000","last_became_peered":"2024-05-27T07:36:15.029738+0000","last_unstale":"2024-05-27T07:39:44.363074+0000","last_undegraded":"2024-05-27T07:39:44.363074+0000","last_fullsized":"2024-05-27T07:39:44.363074+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:18.101941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"78'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244544+0000","last_change":"2024-05-27T07:36:01.955530+0000","last_active":"2024-05-27T07:39:44.244544+0000","last_peered":"2024-05-27T07:39:44.244544+0000","last_clean":"2024-05-27T07:39:44.244544+0000","last_became_active":"2024-05-27T07:36:01.955225+0000","last_became_peered":"2024-05-27T07:36:01.955225+0000","last_unstale":"2024-05-27T07:39:44.244544+0000","last_undegraded":"2024-05-27T07:39:44.244544+0000","last_fullsized":"2024-05-27T07:39:44.244544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:02.067672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.248070+0000","last_change":"2024-05-27T07:36:04.984397+0000","last_active":"2024-05-27T07:39:44.248070+0000","last_peered":"2024-05-27T07:39:44.248070+0000","last_clean":"2024-05-27T07:39:44.248070+0000","last_became_active":"2024-05-27T07:36:04.984033+0000","last_became_peered":"2024-05-27T07:36:04.984033+0000","last_unstale":"2024-05-27T07:39:44.248070+0000","last_undegraded":"2024-05-27T07:39:44.248070+0000","last_fullsized":"2024-05-27T07:39:44.248070+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:15.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417201+0000","last_change":"2024-05-27T07:37:46.322144+0000","last_active":"2024-05-27T07:39:44.417201+0000","last_peered":"2024-05-27T07:39:44.417201+0000","last_clean":"2024-05-27T07:39:44.417201+0000","last_became_active":"2024-05-27T07:37:46.321610+0000","last_became_peered":"2024-05-27T07:37:46.321610+0000","last_unstale":"2024-05-27T07:39:44.417201+0000","last_undegraded":"2024-05-27T07:39:44.417201+0000","last_fullsized":"2024-05-27T07:39:44.417201+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:47.119281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"78'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.248298+0000","last_change":"2024-05-27T07:36:02.243079+0000","last_active":"2024-05-27T07:39:44.248298+0000","last_peered":"2024-05-27T07:39:44.248298+0000","last_clean":"2024-05-27T07:39:44.248298+0000","last_became_active":"2024-05-27T07:36:02.242633+0000","last_became_peered":"2024-05-27T07:36:02.242633+0000","last_unstale":"2024-05-27T07:39:44.248298+0000","last_undegraded":"2024-05-27T07:39:44.248298+0000","last_fullsized":"2024-05-27T07:39:44.248298+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:45:33.980842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417940+0000","last_change":"2024-05-27T07:36:04.989775+0000","last_active":"2024-05-27T07:39:44.417940+0000","last_peered":"2024-05-27T07:39:44.417940+0000","last_clean":"2024-05-27T07:39:44.417940+0000","last_became_active":"2024-05-27T07:36:04.989289+0000","last_became_peered":"2024-05-27T07:36:04.989289+0000","last_unstale":"2024-05-27T07:39:44.417940+0000","last_undegraded":"2024-05-27T07:39:44.417940+0000","last_fullsized":"2024-05-27T07:39:44.417940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:24:38.355231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417613+0000","last_change":"2024-05-27T07:36:15.031737+0000","last_active":"2024-05-27T07:39:44.417613+0000","last_peered":"2024-05-27T07:39:44.417613+0000","last_clean":"2024-05-27T07:39:44.417613+0000","last_became_active":"2024-05-27T07:36:15.031314+0000","last_became_peered":"2024-05-27T07:36:15.031314+0000","last_unstale":"2024-05-27T07:39:44.417613+0000","last_undegraded":"2024-05-27T07:39:44.417613+0000","last_fullsized":"2024-05-27T07:39:44.417613+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:32.742298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"78'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247148+0000","last_change":"2024-05-27T07:36:01.957459+0000","last_active":"2024-05-27T07:39:44.247148+0000","last_peered":"2024-05-27T07:39:44.247148+0000","last_clean":"2024-05-27T07:39:44.247148+0000","last_became_active":"2024-05-27T07:36:01.954985+0000","last_became_peered":"2024-05-27T07:36:01.954985+0000","last_unstale":"2024-05-27T07:39:44.247148+0000","last_undegraded":"2024-05-27T07:39:44.247148+0000","last_fullsized":"2024-05-27T07:39:44.247148+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:27.085486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418627+0000","last_change":"2024-05-27T07:36:04.986447+0000","last_active":"2024-05-27T07:39:44.418627+0000","last_peered":"2024-05-27T07:39:44.418627+0000","last_clean":"2024-05-27T07:39:44.418627+0000","last_became_active":"2024-05-27T07:36:04.982099+0000","last_became_peered":"2024-05-27T07:36:04.982099+0000","last_unstale":"2024-05-27T07:39:44.418627+0000","last_undegraded":"2024-05-27T07:39:44.418627+0000","last_fullsized":"2024-05-27T07:39:44.418627+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:32.637479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417267+0000","last_change":"2024-05-27T07:36:15.024152+0000","last_active":"2024-05-27T07:39:44.417267+0000","last_peered":"2024-05-27T07:39:44.417267+0000","last_clean":"2024-05-27T07:39:44.417267+0000","last_became_active":"2024-05-27T07:36:15.023711+0000","last_became_peered":"2024-05-27T07:36:15.023711+0000","last_unstale":"2024-05-27T07:39:44.417267+0000","last_undegraded":"2024-05-27T07:39:44.417267+0000","last_fullsized":"2024-05-27T07:39:44.417267+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:33.809364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246823+0000","last_change":"2024-05-27T07:36:01.950531+0000","last_active":"2024-05-27T07:39:44.246823+0000","last_peered":"2024-05-27T07:39:44.246823+0000","last_clean":"2024-05-27T07:39:44.246823+0000","last_became_active":"2024-05-27T07:36:01.950015+0000","last_became_peered":"2024-05-27T07:36:01.950015+0000","last_unstale":"2024-05-27T07:39:44.246823+0000","last_undegraded":"2024-05-27T07:39:44.246823+0000","last_fullsized":"2024-05-27T07:39:44.246823+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:22.400415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135595+0000","last_change":"2024-05-27T07:36:04.982792+0000","last_active":"2024-05-27T07:39:44.135595+0000","last_peered":"2024-05-27T07:39:44.135595+0000","last_clean":"2024-05-27T07:39:44.135595+0000","last_became_active":"2024-05-27T07:36:04.982400+0000","last_became_peered":"2024-05-27T07:36:04.982400+0000","last_unstale":"2024-05-27T07:39:44.135595+0000","last_undegraded":"2024-05-27T07:39:44.135595+0000","last_fullsized":"2024-05-27T07:39:44.135595+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:27.823654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363642+0000","last_change":"2024-05-27T07:36:15.027932+0000","last_active":"2024-05-27T07:39:44.363642+0000","last_peered":"2024-05-27T07:39:44.363642+0000","last_clean":"2024-05-27T07:39:44.363642+0000","last_became_active":"2024-05-27T07:36:15.027526+0000","last_became_peered":"2024-05-27T07:36:15.027526+0000","last_unstale":"2024-05-27T07:39:44.363642+0000","last_undegraded":"2024-05-27T07:39:44.363642+0000","last_fullsized":"2024-05-27T07:39:44.363642+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:16.165934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"78'1","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135427+0000","last_change":"2024-05-27T07:36:45.127637+0000","last_active":"2024-05-27T07:39:44.135427+0000","last_peered":"2024-05-27T07:39:44.135427+0000","last_clean":"2024-05-27T07:39:44.135427+0000","last_became_active":"2024-05-27T07:36:45.127249+0000","last_became_peered":"2024-05-27T07:36:45.127249+0000","last_unstale":"2024-05-27T07:39:44.135427+0000","last_undegraded":"2024-05-27T07:39:44.135427+0000","last_fullsized":"2024-05-27T07:39:44.135427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:52.418069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246700+0000","last_change":"2024-05-27T07:36:04.985135+0000","last_active":"2024-05-27T07:39:44.246700+0000","last_peered":"2024-05-27T07:39:44.246700+0000","last_clean":"2024-05-27T07:39:44.246700+0000","last_became_active":"2024-05-27T07:36:04.983621+0000","last_became_peered":"2024-05-27T07:36:04.983621+0000","last_unstale":"2024-05-27T07:39:44.246700+0000","last_undegraded":"2024-05-27T07:39:44.246700+0000","last_fullsized":"2024-05-27T07:39:44.246700+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:11.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363376+0000","last_change":"2024-05-27T07:36:15.024558+0000","last_active":"2024-05-27T07:39:44.363376+0000","last_peered":"2024-05-27T07:39:44.363376+0000","last_clean":"2024-05-27T07:39:44.363376+0000","last_became_active":"2024-05-27T07:36:15.024148+0000","last_became_peered":"2024-05-27T07:36:15.024148+0000","last_unstale":"2024-05-27T07:39:44.363376+0000","last_undegraded":"2024-05-27T07:39:44.363376+0000","last_fullsized":"2024-05-27T07:39:44.363376+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:03.648451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"78'1","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133382+0000","last_change":"2024-05-27T07:36:45.129413+0000","last_active":"2024-05-27T07:39:44.133382+0000","last_peered":"2024-05-27T07:39:44.133382+0000","last_clean":"2024-05-27T07:39:44.133382+0000","last_became_active":"2024-05-27T07:36:45.128779+0000","last_became_peered":"2024-05-27T07:36:45.128779+0000","last_unstale":"2024-05-27T07:39:44.133382+0000","last_undegraded":"2024-05-27T07:39:44.133382+0000","last_fullsized":"2024-05-27T07:39:44.133382+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:05.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135849+0000","last_change":"2024-05-27T07:36:04.989878+0000","last_active":"2024-05-27T07:39:44.135849+0000","last_peered":"2024-05-27T07:39:44.135849+0000","last_clean":"2024-05-27T07:39:44.135849+0000","last_became_active":"2024-05-27T07:36:04.989532+0000","last_became_peered":"2024-05-27T07:36:04.989532+0000","last_unstale":"2024-05-27T07:39:44.135849+0000","last_undegraded":"2024-05-27T07:39:44.135849+0000","last_fullsized":"2024-05-27T07:39:44.135849+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:54:53.909592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417859+0000","last_change":"2024-05-27T07:37:46.321582+0000","last_active":"2024-05-27T07:39:44.417859+0000","last_peered":"2024-05-27T07:39:44.417859+0000","last_clean":"2024-05-27T07:39:44.417859+0000","last_became_active":"2024-05-27T07:37:46.321193+0000","last_became_peered":"2024-05-27T07:37:46.321193+0000","last_unstale":"2024-05-27T07:39:44.417859+0000","last_undegraded":"2024-05-27T07:39:44.417859+0000","last_fullsized":"2024-05-27T07:39:44.417859+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:25:25.767300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133519+0000","last_change":"2024-05-27T07:36:01.949684+0000","last_active":"2024-05-27T07:39:44.133519+0000","last_peered":"2024-05-27T07:39:44.133519+0000","last_clean":"2024-05-27T07:39:44.133519+0000","last_became_active":"2024-05-27T07:36:01.949308+0000","last_became_peered":"2024-05-27T07:36:01.949308+0000","last_unstale":"2024-05-27T07:39:44.133519+0000","last_undegraded":"2024-05-27T07:39:44.133519+0000","last_fullsized":"2024-05-27T07:39:44.133519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:50:54.859172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247002+0000","last_change":"2024-05-27T07:36:04.971266+0000","last_active":"2024-05-27T07:39:44.247002+0000","last_peered":"2024-05-27T07:39:44.247002+0000","last_clean":"2024-05-27T07:39:44.247002+0000","last_became_active":"2024-05-27T07:36:04.970831+0000","last_became_peered":"2024-05-27T07:36:04.970831+0000","last_unstale":"2024-05-27T07:39:44.247002+0000","last_undegraded":"2024-05-27T07:39:44.247002+0000","last_fullsized":"2024-05-27T07:39:44.247002+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:37.237637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417864+0000","last_change":"2024-05-27T07:36:15.024415+0000","last_active":"2024-05-27T07:39:44.417864+0000","last_peered":"2024-05-27T07:39:44.417864+0000","last_clean":"2024-05-27T07:39:44.417864+0000","last_became_active":"2024-05-27T07:36:15.023935+0000","last_became_peered":"2024-05-27T07:36:15.023935+0000","last_unstale":"2024-05-27T07:39:44.417864+0000","last_undegraded":"2024-05-27T07:39:44.417864+0000","last_fullsized":"2024-05-27T07:39:44.417864+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:37:30.386933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134940+0000","last_change":"2024-05-27T07:36:01.952256+0000","last_active":"2024-05-27T07:39:44.134940+0000","last_peered":"2024-05-27T07:39:44.134940+0000","last_clean":"2024-05-27T07:39:44.134940+0000","last_became_active":"2024-05-27T07:36:01.951876+0000","last_became_peered":"2024-05-27T07:36:01.951876+0000","last_unstale":"2024-05-27T07:39:44.134940+0000","last_undegraded":"2024-05-27T07:39:44.134940+0000","last_fullsized":"2024-05-27T07:39:44.134940+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:05.201780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.243724+0000","last_change":"2024-05-27T07:36:04.980300+0000","last_active":"2024-05-27T07:39:44.243724+0000","last_peered":"2024-05-27T07:39:44.243724+0000","last_clean":"2024-05-27T07:39:44.243724+0000","last_became_active":"2024-05-27T07:36:04.979954+0000","last_became_peered":"2024-05-27T07:36:04.979954+0000","last_unstale":"2024-05-27T07:39:44.243724+0000","last_undegraded":"2024-05-27T07:39:44.243724+0000","last_fullsized":"2024-05-27T07:39:44.243724+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:15.712773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247102+0000","last_change":"2024-05-27T07:36:15.026685+0000","last_active":"2024-05-27T07:39:44.247102+0000","last_peered":"2024-05-27T07:39:44.247102+0000","last_clean":"2024-05-27T07:39:44.247102+0000","last_became_active":"2024-05-27T07:36:15.026287+0000","last_became_peered":"2024-05-27T07:36:15.026287+0000","last_unstale":"2024-05-27T07:39:44.247102+0000","last_undegraded":"2024-05-27T07:39:44.247102+0000","last_fullsized":"2024-05-27T07:39:44.247102+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:29:26.394435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134169+0000","last_change":"2024-05-27T07:36:01.956566+0000","last_active":"2024-05-27T07:39:44.134169+0000","last_peered":"2024-05-27T07:39:44.134169+0000","last_clean":"2024-05-27T07:39:44.134169+0000","last_became_active":"2024-05-27T07:36:01.955963+0000","last_became_peered":"2024-05-27T07:36:01.955963+0000","last_unstale":"2024-05-27T07:39:44.134169+0000","last_undegraded":"2024-05-27T07:39:44.134169+0000","last_fullsized":"2024-05-27T07:39:44.134169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:53.081784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418762+0000","last_change":"2024-05-27T07:36:04.983878+0000","last_active":"2024-05-27T07:39:44.418762+0000","last_peered":"2024-05-27T07:39:44.418762+0000","last_clean":"2024-05-27T07:39:44.418762+0000","last_became_active":"2024-05-27T07:36:04.983339+0000","last_became_peered":"2024-05-27T07:36:04.983339+0000","last_unstale":"2024-05-27T07:39:44.418762+0000","last_undegraded":"2024-05-27T07:39:44.418762+0000","last_fullsized":"2024-05-27T07:39:44.418762+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:57:01.726122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.243682+0000","last_change":"2024-05-27T07:36:15.027923+0000","last_active":"2024-05-27T07:39:44.243682+0000","last_peered":"2024-05-27T07:39:44.243682+0000","last_clean":"2024-05-27T07:39:44.243682+0000","last_became_active":"2024-05-27T07:36:15.026012+0000","last_became_peered":"2024-05-27T07:36:15.026012+0000","last_unstale":"2024-05-27T07:39:44.243682+0000","last_undegraded":"2024-05-27T07:39:44.243682+0000","last_fullsized":"2024-05-27T07:39:44.243682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:27:45.238053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419164+0000","last_change":"2024-05-27T07:36:01.956038+0000","last_active":"2024-05-27T07:39:44.419164+0000","last_peered":"2024-05-27T07:39:44.419164+0000","last_clean":"2024-05-27T07:39:44.419164+0000","last_became_active":"2024-05-27T07:36:01.955639+0000","last_became_peered":"2024-05-27T07:36:01.955639+0000","last_unstale":"2024-05-27T07:39:44.419164+0000","last_undegraded":"2024-05-27T07:39:44.419164+0000","last_fullsized":"2024-05-27T07:39:44.419164+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:32.920194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417936+0000","last_change":"2024-05-27T07:36:04.986676+0000","last_active":"2024-05-27T07:39:44.417936+0000","last_peered":"2024-05-27T07:39:44.417936+0000","last_clean":"2024-05-27T07:39:44.417936+0000","last_became_active":"2024-05-27T07:36:04.986316+0000","last_became_peered":"2024-05-27T07:36:04.986316+0000","last_unstale":"2024-05-27T07:39:44.417936+0000","last_undegraded":"2024-05-27T07:39:44.417936+0000","last_fullsized":"2024-05-27T07:39:44.417936+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:03:50.947781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417814+0000","last_change":"2024-05-27T07:36:15.027643+0000","last_active":"2024-05-27T07:39:44.417814+0000","last_peered":"2024-05-27T07:39:44.417814+0000","last_clean":"2024-05-27T07:39:44.417814+0000","last_became_active":"2024-05-27T07:36:15.027175+0000","last_became_peered":"2024-05-27T07:36:15.027175+0000","last_unstale":"2024-05-27T07:39:44.417814+0000","last_undegraded":"2024-05-27T07:39:44.417814+0000","last_fullsized":"2024-05-27T07:39:44.417814+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:46:32.764271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"78'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135540+0000","last_change":"2024-05-27T07:36:02.245746+0000","last_active":"2024-05-27T07:39:44.135540+0000","last_peered":"2024-05-27T07:39:44.135540+0000","last_clean":"2024-05-27T07:39:44.135540+0000","last_became_active":"2024-05-27T07:36:02.245269+0000","last_became_peered":"2024-05-27T07:36:02.245269+0000","last_unstale":"2024-05-27T07:39:44.135540+0000","last_undegraded":"2024-05-27T07:39:44.135540+0000","last_fullsized":"2024-05-27T07:39:44.135540+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:28.202540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418801+0000","last_change":"2024-05-27T07:36:45.129374+0000","last_active":"2024-05-27T07:39:44.418801+0000","last_peered":"2024-05-27T07:39:44.418801+0000","last_clean":"2024-05-27T07:39:44.418801+0000","last_became_active":"2024-05-27T07:36:45.128956+0000","last_became_peered":"2024-05-27T07:36:45.128956+0000","last_unstale":"2024-05-27T07:39:44.418801+0000","last_undegraded":"2024-05-27T07:39:44.418801+0000","last_fullsized":"2024-05-27T07:39:44.418801+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:30:17.037598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244073+0000","last_change":"2024-05-27T07:36:15.027097+0000","last_active":"2024-05-27T07:39:44.244073+0000","last_peered":"2024-05-27T07:39:44.244073+0000","last_clean":"2024-05-27T07:39:44.244073+0000","last_became_active":"2024-05-27T07:36:15.026704+0000","last_became_peered":"2024-05-27T07:36:15.026704+0000","last_unstale":"2024-05-27T07:39:44.244073+0000","last_undegraded":"2024-05-27T07:39:44.244073+0000","last_fullsized":"2024-05-27T07:39:44.244073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:35:03.747199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'39","reported_seq":423,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.714753+0000","last_change":"2024-05-27T07:36:01.956926+0000","last_active":"2024-05-27T07:39:44.714753+0000","last_peered":"2024-05-27T07:39:44.714753+0000","last_clean":"2024-05-27T07:39:44.714753+0000","last_became_active":"2024-05-27T07:36:01.956515+0000","last_became_peered":"2024-05-27T07:36:01.956515+0000","last_unstale":"2024-05-27T07:39:44.714753+0000","last_undegraded":"2024-05-27T07:39:44.714753+0000","last_fullsized":"2024-05-27T07:39:44.714753+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:46.566489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71357,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":337,"num_read_kb":239,"num_write":39,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246955+0000","last_change":"2024-05-27T07:36:04.976498+0000","last_active":"2024-05-27T07:39:44.246955+0000","last_peered":"2024-05-27T07:39:44.246955+0000","last_clean":"2024-05-27T07:39:44.246955+0000","last_became_active":"2024-05-27T07:36:04.976020+0000","last_became_peered":"2024-05-27T07:36:04.976020+0000","last_unstale":"2024-05-27T07:39:44.246955+0000","last_undegraded":"2024-05-27T07:39:44.246955+0000","last_fullsized":"2024-05-27T07:39:44.246955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:25.909259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135913+0000","last_change":"2024-05-27T07:36:15.022700+0000","last_active":"2024-05-27T07:39:44.135913+0000","last_peered":"2024-05-27T07:39:44.135913+0000","last_clean":"2024-05-27T07:39:44.135913+0000","last_became_active":"2024-05-27T07:36:15.022109+0000","last_became_peered":"2024-05-27T07:36:15.022109+0000","last_unstale":"2024-05-27T07:39:44.135913+0000","last_undegraded":"2024-05-27T07:39:44.135913+0000","last_fullsized":"2024-05-27T07:39:44.135913+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:02:43.498049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"78'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363210+0000","last_change":"2024-05-27T07:36:01.953442+0000","last_active":"2024-05-27T07:39:44.363210+0000","last_peered":"2024-05-27T07:39:44.363210+0000","last_clean":"2024-05-27T07:39:44.363210+0000","last_became_active":"2024-05-27T07:36:01.953051+0000","last_became_peered":"2024-05-27T07:36:01.953051+0000","last_unstale":"2024-05-27T07:39:44.363210+0000","last_undegraded":"2024-05-27T07:39:44.363210+0000","last_fullsized":"2024-05-27T07:39:44.363210+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:47:53.070596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133518+0000","last_change":"2024-05-27T07:36:04.976365+0000","last_active":"2024-05-27T07:39:44.133518+0000","last_peered":"2024-05-27T07:39:44.133518+0000","last_clean":"2024-05-27T07:39:44.133518+0000","last_became_active":"2024-05-27T07:36:04.976050+0000","last_became_peered":"2024-05-27T07:36:04.976050+0000","last_unstale":"2024-05-27T07:39:44.133518+0000","last_undegraded":"2024-05-27T07:39:44.133518+0000","last_fullsized":"2024-05-27T07:39:44.133518+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:19.765746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417617+0000","last_change":"2024-05-27T07:36:15.022561+0000","last_active":"2024-05-27T07:39:44.417617+0000","last_peered":"2024-05-27T07:39:44.417617+0000","last_clean":"2024-05-27T07:39:44.417617+0000","last_became_active":"2024-05-27T07:36:15.021974+0000","last_became_peered":"2024-05-27T07:36:15.021974+0000","last_unstale":"2024-05-27T07:39:44.417617+0000","last_undegraded":"2024-05-27T07:39:44.417617+0000","last_fullsized":"2024-05-27T07:39:44.417617+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:31.520969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135318+0000","last_change":"2024-05-27T07:36:01.952625+0000","last_active":"2024-05-27T07:39:44.135318+0000","last_peered":"2024-05-27T07:39:44.135318+0000","last_clean":"2024-05-27T07:39:44.135318+0000","last_became_active":"2024-05-27T07:36:01.952280+0000","last_became_peered":"2024-05-27T07:36:01.952280+0000","last_unstale":"2024-05-27T07:39:44.135318+0000","last_undegraded":"2024-05-27T07:39:44.135318+0000","last_fullsized":"2024-05-27T07:39:44.135318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:17.548061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418259+0000","last_change":"2024-05-27T07:36:04.984795+0000","last_active":"2024-05-27T07:39:44.418259+0000","last_peered":"2024-05-27T07:39:44.418259+0000","last_clean":"2024-05-27T07:39:44.418259+0000","last_became_active":"2024-05-27T07:36:04.984384+0000","last_became_peered":"2024-05-27T07:36:04.984384+0000","last_unstale":"2024-05-27T07:39:44.418259+0000","last_undegraded":"2024-05-27T07:39:44.418259+0000","last_fullsized":"2024-05-27T07:39:44.418259+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:07.207460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244434+0000","last_change":"2024-05-27T07:36:15.033565+0000","last_active":"2024-05-27T07:39:44.244434+0000","last_peered":"2024-05-27T07:39:44.244434+0000","last_clean":"2024-05-27T07:39:44.244434+0000","last_became_active":"2024-05-27T07:36:15.033145+0000","last_became_peered":"2024-05-27T07:36:15.033145+0000","last_unstale":"2024-05-27T07:39:44.244434+0000","last_undegraded":"2024-05-27T07:39:44.244434+0000","last_fullsized":"2024-05-27T07:39:44.244434+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:38:02.163816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417353+0000","last_change":"2024-05-27T07:36:01.952759+0000","last_active":"2024-05-27T07:39:44.417353+0000","last_peered":"2024-05-27T07:39:44.417353+0000","last_clean":"2024-05-27T07:39:44.417353+0000","last_became_active":"2024-05-27T07:36:01.952339+0000","last_became_peered":"2024-05-27T07:36:01.952339+0000","last_unstale":"2024-05-27T07:39:44.417353+0000","last_undegraded":"2024-05-27T07:39:44.417353+0000","last_fullsized":"2024-05-27T07:39:44.417353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:14.518123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417694+0000","last_change":"2024-05-27T07:36:04.986312+0000","last_active":"2024-05-27T07:39:44.417694+0000","last_peered":"2024-05-27T07:39:44.417694+0000","last_clean":"2024-05-27T07:39:44.417694+0000","last_became_active":"2024-05-27T07:36:04.985944+0000","last_became_peered":"2024-05-27T07:36:04.985944+0000","last_unstale":"2024-05-27T07:39:44.417694+0000","last_undegraded":"2024-05-27T07:39:44.417694+0000","last_fullsized":"2024-05-27T07:39:44.417694+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:04.955449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133923+0000","last_change":"2024-05-27T07:36:15.020930+0000","last_active":"2024-05-27T07:39:44.133923+0000","last_peered":"2024-05-27T07:39:44.133923+0000","last_clean":"2024-05-27T07:39:44.133923+0000","last_became_active":"2024-05-27T07:36:15.020556+0000","last_became_peered":"2024-05-27T07:36:15.020556+0000","last_unstale":"2024-05-27T07:39:44.133923+0000","last_undegraded":"2024-05-27T07:39:44.133923+0000","last_fullsized":"2024-05-27T07:39:44.133923+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:02.673065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"78'8","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246644+0000","last_change":"2024-05-27T07:36:02.243402+0000","last_active":"2024-05-27T07:39:44.246644+0000","last_peered":"2024-05-27T07:39:44.246644+0000","last_clean":"2024-05-27T07:39:44.246644+0000","last_became_active":"2024-05-27T07:36:02.242978+0000","last_became_peered":"2024-05-27T07:36:02.242978+0000","last_unstale":"2024-05-27T07:39:44.246644+0000","last_undegraded":"2024-05-27T07:39:44.246644+0000","last_fullsized":"2024-05-27T07:39:44.246644+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:25:04.289506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135103+0000","last_change":"2024-05-27T07:36:04.989885+0000","last_active":"2024-05-27T07:39:44.135103+0000","last_peered":"2024-05-27T07:39:44.135103+0000","last_clean":"2024-05-27T07:39:44.135103+0000","last_became_active":"2024-05-27T07:36:04.989394+0000","last_became_peered":"2024-05-27T07:36:04.989394+0000","last_unstale":"2024-05-27T07:39:44.135103+0000","last_undegraded":"2024-05-27T07:39:44.135103+0000","last_fullsized":"2024-05-27T07:39:44.135103+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:54.138286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135101+0000","last_change":"2024-05-27T07:36:15.021772+0000","last_active":"2024-05-27T07:39:44.135101+0000","last_peered":"2024-05-27T07:39:44.135101+0000","last_clean":"2024-05-27T07:39:44.135101+0000","last_became_active":"2024-05-27T07:36:15.021352+0000","last_became_peered":"2024-05-27T07:36:15.021352+0000","last_unstale":"2024-05-27T07:39:44.135101+0000","last_undegraded":"2024-05-27T07:39:44.135101+0000","last_fullsized":"2024-05-27T07:39:44.135101+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:23.066662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417811+0000","last_change":"2024-05-27T07:36:45.131900+0000","last_active":"2024-05-27T07:39:44.417811+0000","last_peered":"2024-05-27T07:39:44.417811+0000","last_clean":"2024-05-27T07:39:44.417811+0000","last_became_active":"2024-05-27T07:36:45.127340+0000","last_became_peered":"2024-05-27T07:36:45.127340+0000","last_unstale":"2024-05-27T07:39:44.417811+0000","last_undegraded":"2024-05-27T07:39:44.417811+0000","last_fullsized":"2024-05-27T07:39:44.417811+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:07:57.264601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417805+0000","last_change":"2024-05-27T07:36:04.987671+0000","last_active":"2024-05-27T07:39:44.417805+0000","last_peered":"2024-05-27T07:39:44.417805+0000","last_clean":"2024-05-27T07:39:44.417805+0000","last_became_active":"2024-05-27T07:36:04.987317+0000","last_became_peered":"2024-05-27T07:36:04.987317+0000","last_unstale":"2024-05-27T07:39:44.417805+0000","last_undegraded":"2024-05-27T07:39:44.417805+0000","last_fullsized":"2024-05-27T07:39:44.417805+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:35.985968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246611+0000","last_change":"2024-05-27T07:36:15.019967+0000","last_active":"2024-05-27T07:39:44.246611+0000","last_peered":"2024-05-27T07:39:44.246611+0000","last_clean":"2024-05-27T07:39:44.246611+0000","last_became_active":"2024-05-27T07:36:15.019509+0000","last_became_peered":"2024-05-27T07:36:15.019509+0000","last_unstale":"2024-05-27T07:39:44.246611+0000","last_undegraded":"2024-05-27T07:39:44.246611+0000","last_fullsized":"2024-05-27T07:39:44.246611+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:01:56.137335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244238+0000","last_change":"2024-05-27T07:36:02.243561+0000","last_active":"2024-05-27T07:39:44.244238+0000","last_peered":"2024-05-27T07:39:44.244238+0000","last_clean":"2024-05-27T07:39:44.244238+0000","last_became_active":"2024-05-27T07:36:02.242911+0000","last_became_peered":"2024-05-27T07:36:02.242911+0000","last_unstale":"2024-05-27T07:39:44.244238+0000","last_undegraded":"2024-05-27T07:39:44.244238+0000","last_fullsized":"2024-05-27T07:39:44.244238+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:01:43.915660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133619+0000","last_change":"2024-05-27T07:36:04.971108+0000","last_active":"2024-05-27T07:39:44.133619+0000","last_peered":"2024-05-27T07:39:44.133619+0000","last_clean":"2024-05-27T07:39:44.133619+0000","last_became_active":"2024-05-27T07:36:04.970682+0000","last_became_peered":"2024-05-27T07:36:04.970682+0000","last_unstale":"2024-05-27T07:39:44.133619+0000","last_undegraded":"2024-05-27T07:39:44.133619+0000","last_fullsized":"2024-05-27T07:39:44.133619+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:15:34.940449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133949+0000","last_change":"2024-05-27T07:36:15.021292+0000","last_active":"2024-05-27T07:39:44.133949+0000","last_peered":"2024-05-27T07:39:44.133949+0000","last_clean":"2024-05-27T07:39:44.133949+0000","last_became_active":"2024-05-27T07:36:15.019988+0000","last_became_peered":"2024-05-27T07:36:15.019988+0000","last_unstale":"2024-05-27T07:39:44.133949+0000","last_undegraded":"2024-05-27T07:39:44.133949+0000","last_fullsized":"2024-05-27T07:39:44.133949+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:34:14.325368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417857+0000","last_change":"2024-05-27T07:36:01.952231+0000","last_active":"2024-05-27T07:39:44.417857+0000","last_peered":"2024-05-27T07:39:44.417857+0000","last_clean":"2024-05-27T07:39:44.417857+0000","last_became_active":"2024-05-27T07:36:01.951818+0000","last_became_peered":"2024-05-27T07:36:01.951818+0000","last_unstale":"2024-05-27T07:39:44.417857+0000","last_undegraded":"2024-05-27T07:39:44.417857+0000","last_fullsized":"2024-05-27T07:39:44.417857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:52:16.299204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133412+0000","last_change":"2024-05-27T07:36:04.982761+0000","last_active":"2024-05-27T07:39:44.133412+0000","last_peered":"2024-05-27T07:39:44.133412+0000","last_clean":"2024-05-27T07:39:44.133412+0000","last_became_active":"2024-05-27T07:36:04.982303+0000","last_became_peered":"2024-05-27T07:36:04.982303+0000","last_unstale":"2024-05-27T07:39:44.133412+0000","last_undegraded":"2024-05-27T07:39:44.133412+0000","last_fullsized":"2024-05-27T07:39:44.133412+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:13:19.099227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133814+0000","last_change":"2024-05-27T07:37:46.323619+0000","last_active":"2024-05-27T07:39:44.133814+0000","last_peered":"2024-05-27T07:39:44.133814+0000","last_clean":"2024-05-27T07:39:44.133814+0000","last_became_active":"2024-05-27T07:37:46.322114+0000","last_became_peered":"2024-05-27T07:37:46.322114+0000","last_unstale":"2024-05-27T07:39:44.133814+0000","last_undegraded":"2024-05-27T07:39:44.133814+0000","last_fullsized":"2024-05-27T07:39:44.133814+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:59:08.966863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135176+0000","last_change":"2024-05-27T07:36:45.142023+0000","last_active":"2024-05-27T07:39:44.135176+0000","last_peered":"2024-05-27T07:39:44.135176+0000","last_clean":"2024-05-27T07:39:44.135176+0000","last_became_active":"2024-05-27T07:36:45.141278+0000","last_became_peered":"2024-05-27T07:36:45.141278+0000","last_unstale":"2024-05-27T07:39:44.135176+0000","last_undegraded":"2024-05-27T07:39:44.135176+0000","last_fullsized":"2024-05-27T07:39:44.135176+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:15.800380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133391+0000","last_change":"2024-05-27T07:36:02.243041+0000","last_active":"2024-05-27T07:39:44.133391+0000","last_peered":"2024-05-27T07:39:44.133391+0000","last_clean":"2024-05-27T07:39:44.133391+0000","last_became_active":"2024-05-27T07:36:02.242689+0000","last_became_peered":"2024-05-27T07:36:02.242689+0000","last_unstale":"2024-05-27T07:39:44.133391+0000","last_undegraded":"2024-05-27T07:39:44.133391+0000","last_fullsized":"2024-05-27T07:39:44.133391+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:44:19.652954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418822+0000","last_change":"2024-05-27T07:36:15.025595+0000","last_active":"2024-05-27T07:39:44.418822+0000","last_peered":"2024-05-27T07:39:44.418822+0000","last_clean":"2024-05-27T07:39:44.418822+0000","last_became_active":"2024-05-27T07:36:15.025177+0000","last_became_peered":"2024-05-27T07:36:15.025177+0000","last_unstale":"2024-05-27T07:39:44.418822+0000","last_undegraded":"2024-05-27T07:39:44.418822+0000","last_fullsized":"2024-05-27T07:39:44.418822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:08.920778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247262+0000","last_change":"2024-05-27T07:36:04.981422+0000","last_active":"2024-05-27T07:39:44.247262+0000","last_peered":"2024-05-27T07:39:44.247262+0000","last_clean":"2024-05-27T07:39:44.247262+0000","last_became_active":"2024-05-27T07:36:04.981100+0000","last_became_peered":"2024-05-27T07:36:04.981100+0000","last_unstale":"2024-05-27T07:39:44.247262+0000","last_undegraded":"2024-05-27T07:39:44.247262+0000","last_fullsized":"2024-05-27T07:39:44.247262+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:01.076309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417708+0000","last_change":"2024-05-27T07:36:01.953727+0000","last_active":"2024-05-27T07:39:44.417708+0000","last_peered":"2024-05-27T07:39:44.417708+0000","last_clean":"2024-05-27T07:39:44.417708+0000","last_became_active":"2024-05-27T07:36:01.953323+0000","last_became_peered":"2024-05-27T07:36:01.953323+0000","last_unstale":"2024-05-27T07:39:44.417708+0000","last_undegraded":"2024-05-27T07:39:44.417708+0000","last_fullsized":"2024-05-27T07:39:44.417708+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:17.186854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417734+0000","last_change":"2024-05-27T07:34:38.289913+0000","last_active":"2024-05-27T07:39:44.417734+0000","last_peered":"2024-05-27T07:39:44.417734+0000","last_clean":"2024-05-27T07:39:44.417734+0000","last_became_active":"2024-05-27T07:34:38.285523+0000","last_became_peered":"2024-05-27T07:34:38.285523+0000","last_unstale":"2024-05-27T07:39:44.417734+0000","last_undegraded":"2024-05-27T07:39:44.417734+0000","last_fullsized":"2024-05-27T07:39:44.417734+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:31:09.455079+0000","last_clean_scrub_stamp":"2024-05-27T07:31:09.455079+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:59.537055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.135577+0000","last_change":"2024-05-27T07:36:15.027453+0000","last_active":"2024-05-27T07:39:44.135577+0000","last_peered":"2024-05-27T07:39:44.135577+0000","last_clean":"2024-05-27T07:39:44.135577+0000","last_became_active":"2024-05-27T07:36:15.026872+0000","last_became_peered":"2024-05-27T07:36:15.026872+0000","last_unstale":"2024-05-27T07:39:44.135577+0000","last_undegraded":"2024-05-27T07:39:44.135577+0000","last_fullsized":"2024-05-27T07:39:44.135577+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:34:32.351924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133387+0000","last_change":"2024-05-27T07:36:04.982041+0000","last_active":"2024-05-27T07:39:44.133387+0000","last_peered":"2024-05-27T07:39:44.133387+0000","last_clean":"2024-05-27T07:39:44.133387+0000","last_became_active":"2024-05-27T07:36:04.981719+0000","last_became_peered":"2024-05-27T07:36:04.981719+0000","last_unstale":"2024-05-27T07:39:44.133387+0000","last_undegraded":"2024-05-27T07:39:44.133387+0000","last_fullsized":"2024-05-27T07:39:44.133387+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:40.061892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417569+0000","last_change":"2024-05-27T07:36:01.952238+0000","last_active":"2024-05-27T07:39:44.417569+0000","last_peered":"2024-05-27T07:39:44.417569+0000","last_clean":"2024-05-27T07:39:44.417569+0000","last_became_active":"2024-05-27T07:36:01.951853+0000","last_became_peered":"2024-05-27T07:36:01.951853+0000","last_unstale":"2024-05-27T07:39:44.417569+0000","last_undegraded":"2024-05-27T07:39:44.417569+0000","last_fullsized":"2024-05-27T07:39:44.417569+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:46:41.317531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417616+0000","last_change":"2024-05-27T07:36:15.021848+0000","last_active":"2024-05-27T07:39:44.417616+0000","last_peered":"2024-05-27T07:39:44.417616+0000","last_clean":"2024-05-27T07:39:44.417616+0000","last_became_active":"2024-05-27T07:36:15.021371+0000","last_became_peered":"2024-05-27T07:36:15.021371+0000","last_unstale":"2024-05-27T07:39:44.417616+0000","last_undegraded":"2024-05-27T07:39:44.417616+0000","last_fullsized":"2024-05-27T07:39:44.417616+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:32:21.741229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246986+0000","last_change":"2024-05-27T07:36:02.244283+0000","last_active":"2024-05-27T07:39:44.246986+0000","last_peered":"2024-05-27T07:39:44.246986+0000","last_clean":"2024-05-27T07:39:44.246986+0000","last_became_active":"2024-05-27T07:36:02.243785+0000","last_became_peered":"2024-05-27T07:36:02.243785+0000","last_unstale":"2024-05-27T07:39:44.246986+0000","last_undegraded":"2024-05-27T07:39:44.246986+0000","last_fullsized":"2024-05-27T07:39:44.246986+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:38.596510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133682+0000","last_change":"2024-05-27T07:36:04.979903+0000","last_active":"2024-05-27T07:39:44.133682+0000","last_peered":"2024-05-27T07:39:44.133682+0000","last_clean":"2024-05-27T07:39:44.133682+0000","last_became_active":"2024-05-27T07:36:04.979373+0000","last_became_peered":"2024-05-27T07:36:04.979373+0000","last_unstale":"2024-05-27T07:39:44.133682+0000","last_undegraded":"2024-05-27T07:39:44.133682+0000","last_fullsized":"2024-05-27T07:39:44.133682+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:44:33.763230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246825+0000","last_change":"2024-05-27T07:36:15.023740+0000","last_active":"2024-05-27T07:39:44.246825+0000","last_peered":"2024-05-27T07:39:44.246825+0000","last_clean":"2024-05-27T07:39:44.246825+0000","last_became_active":"2024-05-27T07:36:15.023364+0000","last_became_peered":"2024-05-27T07:36:15.023364+0000","last_unstale":"2024-05-27T07:39:44.246825+0000","last_undegraded":"2024-05-27T07:39:44.246825+0000","last_fullsized":"2024-05-27T07:39:44.246825+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:21.399065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'46","reported_seq":116,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.372049+0000","last_change":"2024-05-27T07:36:01.951451+0000","last_active":"2024-05-27T07:39:44.372049+0000","last_peered":"2024-05-27T07:39:44.372049+0000","last_clean":"2024-05-27T07:39:44.372049+0000","last_became_active":"2024-05-27T07:36:01.951037+0000","last_became_peered":"2024-05-27T07:36:01.951037+0000","last_unstale":"2024-05-27T07:39:44.372049+0000","last_undegraded":"2024-05-27T07:39:44.372049+0000","last_fullsized":"2024-05-27T07:39:44.372049+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:44.202474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.248504+0000","last_change":"2024-05-27T07:36:04.975392+0000","last_active":"2024-05-27T07:39:44.248504+0000","last_peered":"2024-05-27T07:39:44.248504+0000","last_clean":"2024-05-27T07:39:44.248504+0000","last_became_active":"2024-05-27T07:36:04.975029+0000","last_became_peered":"2024-05-27T07:36:04.975029+0000","last_unstale":"2024-05-27T07:39:44.248504+0000","last_undegraded":"2024-05-27T07:39:44.248504+0000","last_fullsized":"2024-05-27T07:39:44.248504+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:32.022670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363000+0000","last_change":"2024-05-27T07:36:15.026401+0000","last_active":"2024-05-27T07:39:44.363000+0000","last_peered":"2024-05-27T07:39:44.363000+0000","last_clean":"2024-05-27T07:39:44.363000+0000","last_became_active":"2024-05-27T07:36:15.025963+0000","last_became_peered":"2024-05-27T07:36:15.025963+0000","last_unstale":"2024-05-27T07:39:44.363000+0000","last_undegraded":"2024-05-27T07:39:44.363000+0000","last_fullsized":"2024-05-27T07:39:44.363000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:33:47.616099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"78'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418346+0000","last_change":"2024-05-27T07:36:02.228776+0000","last_active":"2024-05-27T07:39:44.418346+0000","last_peered":"2024-05-27T07:39:44.418346+0000","last_clean":"2024-05-27T07:39:44.418346+0000","last_became_active":"2024-05-27T07:36:02.228356+0000","last_became_peered":"2024-05-27T07:36:02.228356+0000","last_unstale":"2024-05-27T07:39:44.418346+0000","last_undegraded":"2024-05-27T07:39:44.418346+0000","last_fullsized":"2024-05-27T07:39:44.418346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:07.289340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418083+0000","last_change":"2024-05-27T07:36:04.986917+0000","last_active":"2024-05-27T07:39:44.418083+0000","last_peered":"2024-05-27T07:39:44.418083+0000","last_clean":"2024-05-27T07:39:44.418083+0000","last_became_active":"2024-05-27T07:36:04.986432+0000","last_became_peered":"2024-05-27T07:36:04.986432+0000","last_unstale":"2024-05-27T07:39:44.418083+0000","last_undegraded":"2024-05-27T07:39:44.418083+0000","last_fullsized":"2024-05-27T07:39:44.418083+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:39:15.011058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418062+0000","last_change":"2024-05-27T07:36:15.032312+0000","last_active":"2024-05-27T07:39:44.418062+0000","last_peered":"2024-05-27T07:39:44.418062+0000","last_clean":"2024-05-27T07:39:44.418062+0000","last_became_active":"2024-05-27T07:36:15.031475+0000","last_became_peered":"2024-05-27T07:36:15.031475+0000","last_unstale":"2024-05-27T07:39:44.418062+0000","last_undegraded":"2024-05-27T07:39:44.418062+0000","last_fullsized":"2024-05-27T07:39:44.418062+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:00:40.099421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418204+0000","last_change":"2024-05-27T07:36:02.243456+0000","last_active":"2024-05-27T07:39:44.418204+0000","last_peered":"2024-05-27T07:39:44.418204+0000","last_clean":"2024-05-27T07:39:44.418204+0000","last_became_active":"2024-05-27T07:36:02.243164+0000","last_became_peered":"2024-05-27T07:36:02.243164+0000","last_unstale":"2024-05-27T07:39:44.418204+0000","last_undegraded":"2024-05-27T07:39:44.418204+0000","last_fullsized":"2024-05-27T07:39:44.418204+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:33.765834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246597+0000","last_change":"2024-05-27T07:36:04.980192+0000","last_active":"2024-05-27T07:39:44.246597+0000","last_peered":"2024-05-27T07:39:44.246597+0000","last_clean":"2024-05-27T07:39:44.246597+0000","last_became_active":"2024-05-27T07:36:04.979797+0000","last_became_peered":"2024-05-27T07:36:04.979797+0000","last_unstale":"2024-05-27T07:39:44.246597+0000","last_undegraded":"2024-05-27T07:39:44.246597+0000","last_fullsized":"2024-05-27T07:39:44.246597+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:47:37.006746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.244501+0000","last_change":"2024-05-27T07:36:15.033153+0000","last_active":"2024-05-27T07:39:44.244501+0000","last_peered":"2024-05-27T07:39:44.244501+0000","last_clean":"2024-05-27T07:39:44.244501+0000","last_became_active":"2024-05-27T07:36:15.032783+0000","last_became_peered":"2024-05-27T07:36:15.032783+0000","last_unstale":"2024-05-27T07:39:44.244501+0000","last_undegraded":"2024-05-27T07:39:44.244501+0000","last_fullsized":"2024-05-27T07:39:44.244501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:31:26.237222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247148+0000","last_change":"2024-05-27T07:36:01.952754+0000","last_active":"2024-05-27T07:39:44.247148+0000","last_peered":"2024-05-27T07:39:44.247148+0000","last_clean":"2024-05-27T07:39:44.247148+0000","last_became_active":"2024-05-27T07:36:01.952397+0000","last_became_peered":"2024-05-27T07:36:01.952397+0000","last_unstale":"2024-05-27T07:39:44.247148+0000","last_undegraded":"2024-05-27T07:39:44.247148+0000","last_fullsized":"2024-05-27T07:39:44.247148+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:04.350169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417457+0000","last_change":"2024-05-27T07:36:04.979193+0000","last_active":"2024-05-27T07:39:44.417457+0000","last_peered":"2024-05-27T07:39:44.417457+0000","last_clean":"2024-05-27T07:39:44.417457+0000","last_became_active":"2024-05-27T07:36:04.978679+0000","last_became_peered":"2024-05-27T07:36:04.978679+0000","last_unstale":"2024-05-27T07:39:44.417457+0000","last_undegraded":"2024-05-27T07:39:44.417457+0000","last_fullsized":"2024-05-27T07:39:44.417457+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:59.310284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418066+0000","last_change":"2024-05-27T07:37:46.153707+0000","last_active":"2024-05-27T07:39:44.418066+0000","last_peered":"2024-05-27T07:39:44.418066+0000","last_clean":"2024-05-27T07:39:44.418066+0000","last_became_active":"2024-05-27T07:37:46.148703+0000","last_became_peered":"2024-05-27T07:37:46.148703+0000","last_unstale":"2024-05-27T07:39:44.418066+0000","last_undegraded":"2024-05-27T07:39:44.418066+0000","last_fullsized":"2024-05-27T07:39:44.418066+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:50:20.099467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417476+0000","last_change":"2024-05-27T07:36:45.140940+0000","last_active":"2024-05-27T07:39:44.417476+0000","last_peered":"2024-05-27T07:39:44.417476+0000","last_clean":"2024-05-27T07:39:44.417476+0000","last_became_active":"2024-05-27T07:36:45.140511+0000","last_became_peered":"2024-05-27T07:36:45.140511+0000","last_unstale":"2024-05-27T07:39:44.417476+0000","last_undegraded":"2024-05-27T07:39:44.417476+0000","last_fullsized":"2024-05-27T07:39:44.417476+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:43:00.507442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247777+0000","last_change":"2024-05-27T07:36:04.974709+0000","last_active":"2024-05-27T07:39:44.247777+0000","last_peered":"2024-05-27T07:39:44.247777+0000","last_clean":"2024-05-27T07:39:44.247777+0000","last_became_active":"2024-05-27T07:36:04.974147+0000","last_became_peered":"2024-05-27T07:36:04.974147+0000","last_unstale":"2024-05-27T07:39:44.247777+0000","last_undegraded":"2024-05-27T07:39:44.247777+0000","last_fullsized":"2024-05-27T07:39:44.247777+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:05:11.251124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417872+0000","last_change":"2024-05-27T07:36:15.023187+0000","last_active":"2024-05-27T07:39:44.417872+0000","last_peered":"2024-05-27T07:39:44.417872+0000","last_clean":"2024-05-27T07:39:44.417872+0000","last_became_active":"2024-05-27T07:36:15.022687+0000","last_became_peered":"2024-05-27T07:36:15.022687+0000","last_unstale":"2024-05-27T07:39:44.417872+0000","last_undegraded":"2024-05-27T07:39:44.417872+0000","last_fullsized":"2024-05-27T07:39:44.417872+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:51.713222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"78'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.246746+0000","last_change":"2024-05-27T07:36:01.954751+0000","last_active":"2024-05-27T07:39:44.246746+0000","last_peered":"2024-05-27T07:39:44.246746+0000","last_clean":"2024-05-27T07:39:44.246746+0000","last_became_active":"2024-05-27T07:36:01.954173+0000","last_became_peered":"2024-05-27T07:36:01.954173+0000","last_unstale":"2024-05-27T07:39:44.246746+0000","last_undegraded":"2024-05-27T07:39:44.246746+0000","last_fullsized":"2024-05-27T07:39:44.246746+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:45:10.504504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417752+0000","last_change":"2024-05-27T07:36:04.980294+0000","last_active":"2024-05-27T07:39:44.417752+0000","last_peered":"2024-05-27T07:39:44.417752+0000","last_clean":"2024-05-27T07:39:44.417752+0000","last_became_active":"2024-05-27T07:36:04.979853+0000","last_became_peered":"2024-05-27T07:36:04.979853+0000","last_unstale":"2024-05-27T07:39:44.417752+0000","last_undegraded":"2024-05-27T07:39:44.417752+0000","last_fullsized":"2024-05-27T07:39:44.417752+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:42:43.131216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418970+0000","last_change":"2024-05-27T07:36:15.025854+0000","last_active":"2024-05-27T07:39:44.418970+0000","last_peered":"2024-05-27T07:39:44.418970+0000","last_clean":"2024-05-27T07:39:44.418970+0000","last_became_active":"2024-05-27T07:36:15.025412+0000","last_became_peered":"2024-05-27T07:36:15.025412+0000","last_unstale":"2024-05-27T07:39:44.418970+0000","last_undegraded":"2024-05-27T07:39:44.418970+0000","last_fullsized":"2024-05-27T07:39:44.418970+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:49.904811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363520+0000","last_change":"2024-05-27T07:36:02.243764+0000","last_active":"2024-05-27T07:39:44.363520+0000","last_peered":"2024-05-27T07:39:44.363520+0000","last_clean":"2024-05-27T07:39:44.363520+0000","last_became_active":"2024-05-27T07:36:02.242935+0000","last_became_peered":"2024-05-27T07:36:02.242935+0000","last_unstale":"2024-05-27T07:39:44.363520+0000","last_undegraded":"2024-05-27T07:39:44.363520+0000","last_fullsized":"2024-05-27T07:39:44.363520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:27.535658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419316+0000","last_change":"2024-05-27T07:36:04.975291+0000","last_active":"2024-05-27T07:39:44.419316+0000","last_peered":"2024-05-27T07:39:44.419316+0000","last_clean":"2024-05-27T07:39:44.419316+0000","last_became_active":"2024-05-27T07:36:04.974868+0000","last_became_peered":"2024-05-27T07:36:04.974868+0000","last_unstale":"2024-05-27T07:39:44.419316+0000","last_undegraded":"2024-05-27T07:39:44.419316+0000","last_fullsized":"2024-05-27T07:39:44.419316+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:13.890494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417509+0000","last_change":"2024-05-27T07:36:15.031812+0000","last_active":"2024-05-27T07:39:44.417509+0000","last_peered":"2024-05-27T07:39:44.417509+0000","last_clean":"2024-05-27T07:39:44.417509+0000","last_became_active":"2024-05-27T07:36:15.031399+0000","last_became_peered":"2024-05-27T07:36:15.031399+0000","last_unstale":"2024-05-27T07:39:44.417509+0000","last_undegraded":"2024-05-27T07:39:44.417509+0000","last_fullsized":"2024-05-27T07:39:44.417509+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.595579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.366476+0000","last_change":"2024-05-27T07:36:01.954581+0000","last_active":"2024-05-27T07:39:44.366476+0000","last_peered":"2024-05-27T07:39:44.366476+0000","last_clean":"2024-05-27T07:39:44.366476+0000","last_became_active":"2024-05-27T07:36:01.953564+0000","last_became_peered":"2024-05-27T07:36:01.953564+0000","last_unstale":"2024-05-27T07:39:44.366476+0000","last_undegraded":"2024-05-27T07:39:44.366476+0000","last_fullsized":"2024-05-27T07:39:44.366476+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:56.866738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.418767+0000","last_change":"2024-05-27T07:36:45.141160+0000","last_active":"2024-05-27T07:39:44.418767+0000","last_peered":"2024-05-27T07:39:44.418767+0000","last_clean":"2024-05-27T07:39:44.418767+0000","last_became_active":"2024-05-27T07:36:45.140840+0000","last_became_peered":"2024-05-27T07:36:45.140840+0000","last_unstale":"2024-05-27T07:39:44.418767+0000","last_undegraded":"2024-05-27T07:39:44.418767+0000","last_fullsized":"2024-05-27T07:39:44.418767+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:31.676808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134842+0000","last_change":"2024-05-27T07:36:15.026483+0000","last_active":"2024-05-27T07:39:44.134842+0000","last_peered":"2024-05-27T07:39:44.134842+0000","last_clean":"2024-05-27T07:39:44.134842+0000","last_became_active":"2024-05-27T07:36:15.025955+0000","last_became_peered":"2024-05-27T07:36:15.025955+0000","last_unstale":"2024-05-27T07:39:44.134842+0000","last_undegraded":"2024-05-27T07:39:44.134842+0000","last_fullsized":"2024-05-27T07:39:44.134842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:14.000363+0000","last_clean_scrub_stamp":"2024-05-27T07:36:14.000363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:25.468146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133958+0000","last_change":"2024-05-27T07:36:01.952505+0000","last_active":"2024-05-27T07:39:44.133958+0000","last_peered":"2024-05-27T07:39:44.133958+0000","last_clean":"2024-05-27T07:39:44.133958+0000","last_became_active":"2024-05-27T07:36:01.952088+0000","last_became_peered":"2024-05-27T07:36:01.952088+0000","last_unstale":"2024-05-27T07:39:44.133958+0000","last_undegraded":"2024-05-27T07:39:44.133958+0000","last_fullsized":"2024-05-27T07:39:44.133958+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:00.930168+0000","last_clean_scrub_stamp":"2024-05-27T07:36:00.930168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:09.235719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419335+0000","last_change":"2024-05-27T07:36:04.974331+0000","last_active":"2024-05-27T07:39:44.419335+0000","last_peered":"2024-05-27T07:39:44.419335+0000","last_clean":"2024-05-27T07:39:44.419335+0000","last_became_active":"2024-05-27T07:36:04.973825+0000","last_became_peered":"2024-05-27T07:36:04.973825+0000","last_unstale":"2024-05-27T07:39:44.419335+0000","last_undegraded":"2024-05-27T07:39:44.419335+0000","last_fullsized":"2024-05-27T07:39:44.419335+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:39:42.917309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417542+0000","last_change":"2024-05-27T07:36:45.141224+0000","last_active":"2024-05-27T07:39:44.417542+0000","last_peered":"2024-05-27T07:39:44.417542+0000","last_clean":"2024-05-27T07:39:44.417542+0000","last_became_active":"2024-05-27T07:36:45.140776+0000","last_became_peered":"2024-05-27T07:36:45.140776+0000","last_unstale":"2024-05-27T07:39:44.417542+0000","last_undegraded":"2024-05-27T07:39:44.417542+0000","last_fullsized":"2024-05-27T07:39:44.417542+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:36:59.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363303+0000","last_change":"2024-05-27T07:36:04.988247+0000","last_active":"2024-05-27T07:39:44.363303+0000","last_peered":"2024-05-27T07:39:44.363303+0000","last_clean":"2024-05-27T07:39:44.363303+0000","last_became_active":"2024-05-27T07:36:04.987825+0000","last_became_peered":"2024-05-27T07:36:04.987825+0000","last_unstale":"2024-05-27T07:39:44.363303+0000","last_undegraded":"2024-05-27T07:39:44.363303+0000","last_fullsized":"2024-05-27T07:39:44.363303+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:32.499260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363087+0000","last_change":"2024-05-27T07:36:04.979395+0000","last_active":"2024-05-27T07:39:44.363087+0000","last_peered":"2024-05-27T07:39:44.363087+0000","last_clean":"2024-05-27T07:39:44.363087+0000","last_became_active":"2024-05-27T07:36:04.978983+0000","last_became_peered":"2024-05-27T07:36:04.978983+0000","last_unstale":"2024-05-27T07:39:44.363087+0000","last_undegraded":"2024-05-27T07:39:44.363087+0000","last_fullsized":"2024-05-27T07:39:44.363087+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:53:54.071958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417704+0000","last_change":"2024-05-27T07:36:04.989442+0000","last_active":"2024-05-27T07:39:44.417704+0000","last_peered":"2024-05-27T07:39:44.417704+0000","last_clean":"2024-05-27T07:39:44.417704+0000","last_became_active":"2024-05-27T07:36:04.989079+0000","last_became_peered":"2024-05-27T07:36:04.989079+0000","last_unstale":"2024-05-27T07:39:44.417704+0000","last_undegraded":"2024-05-27T07:39:44.417704+0000","last_fullsized":"2024-05-27T07:39:44.417704+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:39.411255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133776+0000","last_change":"2024-05-27T07:36:04.981289+0000","last_active":"2024-05-27T07:39:44.133776+0000","last_peered":"2024-05-27T07:39:44.133776+0000","last_clean":"2024-05-27T07:39:44.133776+0000","last_became_active":"2024-05-27T07:36:04.980924+0000","last_became_peered":"2024-05-27T07:36:04.980924+0000","last_unstale":"2024-05-27T07:39:44.133776+0000","last_undegraded":"2024-05-27T07:39:44.133776+0000","last_fullsized":"2024-05-27T07:39:44.133776+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:15:04.741303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417916+0000","last_change":"2024-05-27T07:36:04.985712+0000","last_active":"2024-05-27T07:39:44.417916+0000","last_peered":"2024-05-27T07:39:44.417916+0000","last_clean":"2024-05-27T07:39:44.417916+0000","last_became_active":"2024-05-27T07:36:04.983786+0000","last_became_peered":"2024-05-27T07:36:04.983786+0000","last_unstale":"2024-05-27T07:39:44.417916+0000","last_undegraded":"2024-05-27T07:39:44.417916+0000","last_fullsized":"2024-05-27T07:39:44.417916+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:34.549557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.363470+0000","last_change":"2024-05-27T07:36:04.991176+0000","last_active":"2024-05-27T07:39:44.363470+0000","last_peered":"2024-05-27T07:39:44.363470+0000","last_clean":"2024-05-27T07:39:44.363470+0000","last_became_active":"2024-05-27T07:36:04.990774+0000","last_became_peered":"2024-05-27T07:36:04.990774+0000","last_unstale":"2024-05-27T07:39:44.363470+0000","last_undegraded":"2024-05-27T07:39:44.363470+0000","last_fullsized":"2024-05-27T07:39:44.363470+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:37.307581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.248721+0000","last_change":"2024-05-27T07:36:04.982337+0000","last_active":"2024-05-27T07:39:44.248721+0000","last_peered":"2024-05-27T07:39:44.248721+0000","last_clean":"2024-05-27T07:39:44.248721+0000","last_became_active":"2024-05-27T07:36:04.981988+0000","last_became_peered":"2024-05-27T07:36:04.981988+0000","last_unstale":"2024-05-27T07:39:44.248721+0000","last_undegraded":"2024-05-27T07:39:44.248721+0000","last_fullsized":"2024-05-27T07:39:44.248721+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:17:39.763459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.134985+0000","last_change":"2024-05-27T07:36:04.982321+0000","last_active":"2024-05-27T07:39:44.134985+0000","last_peered":"2024-05-27T07:39:44.134985+0000","last_clean":"2024-05-27T07:39:44.134985+0000","last_became_active":"2024-05-27T07:36:04.981690+0000","last_became_peered":"2024-05-27T07:36:04.981690+0000","last_unstale":"2024-05-27T07:39:44.134985+0000","last_undegraded":"2024-05-27T07:39:44.134985+0000","last_fullsized":"2024-05-27T07:39:44.134985+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:08:10.181606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.247215+0000","last_change":"2024-05-27T07:36:04.979411+0000","last_active":"2024-05-27T07:39:44.247215+0000","last_peered":"2024-05-27T07:39:44.247215+0000","last_clean":"2024-05-27T07:39:44.247215+0000","last_became_active":"2024-05-27T07:36:04.979041+0000","last_became_peered":"2024-05-27T07:36:04.979041+0000","last_unstale":"2024-05-27T07:39:44.247215+0000","last_undegraded":"2024-05-27T07:39:44.247215+0000","last_fullsized":"2024-05-27T07:39:44.247215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:57:29.165997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.419164+0000","last_change":"2024-05-27T07:36:04.988564+0000","last_active":"2024-05-27T07:39:44.419164+0000","last_peered":"2024-05-27T07:39:44.419164+0000","last_clean":"2024-05-27T07:39:44.419164+0000","last_became_active":"2024-05-27T07:36:04.986313+0000","last_became_peered":"2024-05-27T07:36:04.986313+0000","last_unstale":"2024-05-27T07:39:44.419164+0000","last_undegraded":"2024-05-27T07:39:44.419164+0000","last_fullsized":"2024-05-27T07:39:44.419164+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:38.809386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.133635+0000","last_change":"2024-05-27T07:36:04.974704+0000","last_active":"2024-05-27T07:39:44.133635+0000","last_peered":"2024-05-27T07:39:44.133635+0000","last_clean":"2024-05-27T07:39:44.133635+0000","last_became_active":"2024-05-27T07:36:04.974311+0000","last_became_peered":"2024-05-27T07:36:04.974311+0000","last_unstale":"2024-05-27T07:39:44.133635+0000","last_undegraded":"2024-05-27T07:39:44.133635+0000","last_fullsized":"2024-05-27T07:39:44.133635+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:48:43.021149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:39:44.417756+0000","last_change":"2024-05-27T07:36:04.983951+0000","last_active":"2024-05-27T07:39:44.417756+0000","last_peered":"2024-05-27T07:39:44.417756+0000","last_clean":"2024-05-27T07:39:44.417756+0000","last_became_active":"2024-05-27T07:36:04.983489+0000","last_became_peered":"2024-05-27T07:36:04.983489+0000","last_unstale":"2024-05-27T07:39:44.417756+0000","last_undegraded":"2024-05-27T07:39:44.417756+0000","last_fullsized":"2024-05-27T07:39:44.417756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:36:03.949404+0000","last_clean_scrub_stamp":"2024-05-27T07:36:03.949404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:04.229523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107193,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":596971,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":435,"num_read_kb":324,"num_write":209,"num_write_kb":1415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":1790913,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710779,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":524552,"data_compressed":35398,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877907008,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":1185667,"data_compressed":71129,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070534,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":515188,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266762,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":515188,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462991,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":1119576,"data_compressed":71129,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626518,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":520434,"data_compressed":35398,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":581085,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":520524,"data_compressed":35398,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1069056,"data_stored":591825,"data_compressed":35398,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182501,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":1045354,"data_compressed":60700,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986272,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":514628,"data_compressed":32429,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822746,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1069056,"data_stored":1246680,"data_compressed":77285,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35731,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":71357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":71391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":524920,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":131782,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":203151,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:39:47.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:39:47.717 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:47.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:47 smithi114 ceph-mon[32038]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:39:47.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:47 smithi114 ceph-mon[32038]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-05-27T07:39:47.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:47 smithi114 ceph-mon[32038]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:39:47.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:47 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:47 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:47 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:47 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:47 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:47 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:47 smithi163 ceph-mon[36451]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:39:47.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:47 smithi163 ceph-mon[36451]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-05-27T07:39:47.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:47 smithi163 ceph-mon[36451]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:39:47.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:47 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:47 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:47 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:47 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:47 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:47 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:47 smithi186 ceph-mon[36647]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:39:47.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:47 smithi186 ceph-mon[36647]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-05-27T07:39:47.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:47 smithi186 ceph-mon[36647]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:39:47.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:47 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:47 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:47 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:47 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:47 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:47.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:47 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='client.15234 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:39:48.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:39:48.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:48.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:48.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='client.15234 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:39:48.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:39:48.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:48.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:48.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='client.15234 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:39:48.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:39:48.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:48.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:39:48.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:48.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.582+0000 7f75ca084700 1 -- 172.21.15.114:0/2890805119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c41034b0 msgr2=0x7f75c41038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.582+0000 7f75ca084700 1 --2- 172.21.15.114:0/2890805119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c41034b0 0x7f75c41038c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f75b40099c0 tx=0x7f75b4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:39:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.582+0000 7f75ca084700 1 -- 172.21.15.114:0/2890805119 shutdown_connections 2024-05-27T07:39:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.582+0000 7f75ca084700 1 --2- 172.21.15.114:0/2890805119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c4108730 0x7f75c4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.582+0000 7f75ca084700 1 --2- 172.21.15.114:0/2890805119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c41034b0 0x7f75c41038c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.582+0000 7f75ca084700 1 --2- 172.21.15.114:0/2890805119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c4107de0 0x7f75c41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.582+0000 7f75ca084700 1 -- 172.21.15.114:0/2890805119 >> 172.21.15.114:0/2890805119 conn(0x7f75c40fedb0 msgr2=0x7f75c41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.582+0000 7f75ca084700 1 -- 172.21.15.114:0/2890805119 shutdown_connections 2024-05-27T07:39:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.582+0000 7f75ca084700 1 -- 172.21.15.114:0/2890805119 wait complete. 2024-05-27T07:39:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.583+0000 7f75ca084700 1 Processor -- start 2024-05-27T07:39:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.583+0000 7f75ca084700 1 -- start start 2024-05-27T07:39:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.583+0000 7f75ca084700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c41034b0 0x7f75c419bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.583+0000 7f75ca084700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c4107de0 0x7f75c419c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.583+0000 7f75ca084700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c4108730 0x7f75c41a6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.583+0000 7f75ca084700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f75c4113360 con 0x7f75c41034b0 2024-05-27T07:39:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.583+0000 7f75ca084700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41131e0 con 0x7f75c4108730 2024-05-27T07:39:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.583+0000 7f75ca084700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41134e0 con 0x7f75c4107de0 2024-05-27T07:39:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.583+0000 7f75c37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c41034b0 0x7f75c419bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.583+0000 7f75c37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c41034b0 0x7f75c419bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43598/0 (socket says 172.21.15.114:43598) 2024-05-27T07:39:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.583+0000 7f75c37fe700 1 -- 172.21.15.114:0/3324917544 learned_addr learned my addr 172.21.15.114:0/3324917544 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.584+0000 7f75c2ffd700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c4107de0 0x7f75c419c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.584+0000 7f75c3fff700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c4108730 0x7f75c41a6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.584+0000 7f75c37fe700 1 -- 172.21.15.114:0/3324917544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c4107de0 msgr2=0x7f75c419c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.584+0000 7f75c37fe700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c4107de0 0x7f75c419c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.584+0000 7f75c37fe700 1 -- 172.21.15.114:0/3324917544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c4108730 msgr2=0x7f75c41a6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.584+0000 7f75c37fe700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c4108730 0x7f75c41a6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.584+0000 7f75c37fe700 1 -- 172.21.15.114:0/3324917544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c419c970 con 0x7f75c41034b0 2024-05-27T07:39:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.584+0000 7f75c37fe700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c41034b0 0x7f75c419bb80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f75b40098e0 tx=0x7f75b40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.584+0000 7f75c0ff9700 1 -- 172.21.15.114:0/3324917544 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b4027070 con 0x7f75c41034b0 2024-05-27T07:39:49.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.584+0000 7f75ca084700 1 -- 172.21.15.114:0/3324917544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c4199970 con 0x7f75c41034b0 2024-05-27T07:39:49.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.584+0000 7f75c0ff9700 1 -- 172.21.15.114:0/3324917544 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75b4003f20 con 0x7f75c41034b0 2024-05-27T07:39:49.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.585+0000 7f75c0ff9700 1 -- 172.21.15.114:0/3324917544 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b4022ca0 con 0x7f75c41034b0 2024-05-27T07:39:49.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.585+0000 7f75ca084700 1 -- 172.21.15.114:0/3324917544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c4199e50 con 0x7f75c41034b0 2024-05-27T07:39:49.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.585+0000 7f75ca084700 1 -- 172.21.15.114:0/3324917544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75c40512b0 con 0x7f75c41034b0 2024-05-27T07:39:49.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.587+0000 7f75c0ff9700 1 -- 172.21.15.114:0/3324917544 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 23) ==== 149829+0+0 (secure 0 0 0) 0x7f75b402a070 con 0x7f75c41034b0 2024-05-27T07:39:49.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.588+0000 7f75c0ff9700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75ac0b2a80 0x7f75ac0b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:49.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.588+0000 7f75c2ffd700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75ac0b2a80 0x7f75ac0b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:49.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.589+0000 7f75c0ff9700 1 -- 172.21.15.114:0/3324917544 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f75b40e41c0 con 0x7f75c41034b0 2024-05-27T07:39:49.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.589+0000 7f75c2ffd700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75ac0b2a80 0x7f75ac0b4f90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f75c4106540 tx=0x7f75b800b410 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:49.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.593+0000 7f75c0ff9700 1 -- 172.21.15.114:0/3324917544 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f75b4013070 con 0x7f75c41034b0 2024-05-27T07:39:49.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:49 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-27T07:39:49.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:49 smithi114 ceph-mon[32038]: Unable to set osd_memory_target on smithi163 to 184781004: error parsing value: Value '184781004' is below minimum 939524096 2024-05-27T07:39:49.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:49 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-27T07:39:49.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:49 smithi114 ceph-mon[32038]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T07:39:49.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:49 smithi114 ceph-mon[32038]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:39:49.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:49 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi114 to 176.1M 2024-05-27T07:39:49.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:49 smithi114 ceph-mon[32038]: Unable to set osd_memory_target on smithi114 to 184753049: error parsing value: Value '184753049' is below minimum 939524096 2024-05-27T07:39:49.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:49 smithi114 ceph-mon[32038]: Updating smithi114:/etc/ceph/ceph.conf 2024-05-27T07:39:49.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:49 smithi114 ceph-mon[32038]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-27T07:39:49.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:49 smithi114 ceph-mon[32038]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-27T07:39:49.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:49 smithi114 ceph-mon[32038]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-05-27T07:39:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.856+0000 7f75ca084700 1 -- 172.21.15.114:0/3324917544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75c402e5e0 con 0x7f75c41034b0 2024-05-27T07:39:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.857+0000 7f75c0ff9700 1 -- 172.21.15.114:0/3324917544 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11827 (secure 0 0 0) 0x7f75b40ab710 con 0x7f75c41034b0 2024-05-27T07:39:49.855 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:49.855 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:37:06.645274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:39:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.859+0000 7f75ca084700 1 -- 172.21.15.114:0/3324917544 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75ac0b2a80 msgr2=0x7f75ac0b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.859+0000 7f75ca084700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75ac0b2a80 0x7f75ac0b4f90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f75c4106540 tx=0x7f75b800b410 comp rx=0 tx=0).stop 2024-05-27T07:39:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.859+0000 7f75ca084700 1 -- 172.21.15.114:0/3324917544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c41034b0 msgr2=0x7f75c419bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.859+0000 7f75ca084700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c41034b0 0x7f75c419bb80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f75b40098e0 tx=0x7f75b40057c0 comp rx=0 tx=0).stop 2024-05-27T07:39:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.859+0000 7f75ca084700 1 -- 172.21.15.114:0/3324917544 shutdown_connections 2024-05-27T07:39:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.859+0000 7f75ca084700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75ac0b2a80 0x7f75ac0b4f90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.859+0000 7f75ca084700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c4108730 0x7f75c41a6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.859+0000 7f75ca084700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c41034b0 0x7f75c419bb80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:49.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.859+0000 7f75ca084700 1 --2- 172.21.15.114:0/3324917544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c4107de0 0x7f75c419c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:49.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.859+0000 7f75ca084700 1 -- 172.21.15.114:0/3324917544 >> 172.21.15.114:0/3324917544 conn(0x7f75c40fedb0 msgr2=0x7f75c4101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:49.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.859+0000 7f75ca084700 1 -- 172.21.15.114:0/3324917544 shutdown_connections 2024-05-27T07:39:49.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:49.859+0000 7f75ca084700 1 -- 172.21.15.114:0/3324917544 wait complete. 2024-05-27T07:39:49.858 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:39:49.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:49 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-27T07:39:49.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:49 smithi163 ceph-mon[36451]: Unable to set osd_memory_target on smithi163 to 184781004: error parsing value: Value '184781004' is below minimum 939524096 2024-05-27T07:39:49.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:49 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-27T07:39:49.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:49 smithi163 ceph-mon[36451]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T07:39:49.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:49 smithi163 ceph-mon[36451]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:39:49.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:49 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi114 to 176.1M 2024-05-27T07:39:49.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:49 smithi163 ceph-mon[36451]: Unable to set osd_memory_target on smithi114 to 184753049: error parsing value: Value '184753049' is below minimum 939524096 2024-05-27T07:39:49.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:49 smithi163 ceph-mon[36451]: Updating smithi114:/etc/ceph/ceph.conf 2024-05-27T07:39:49.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:49 smithi163 ceph-mon[36451]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-27T07:39:49.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:49 smithi163 ceph-mon[36451]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-27T07:39:49.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:49 smithi163 ceph-mon[36451]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-05-27T07:39:49.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:49 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-27T07:39:49.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:49 smithi186 ceph-mon[36647]: Unable to set osd_memory_target on smithi163 to 184781004: error parsing value: Value '184781004' is below minimum 939524096 2024-05-27T07:39:49.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:49 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-27T07:39:49.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:49 smithi186 ceph-mon[36647]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T07:39:49.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:49 smithi186 ceph-mon[36647]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:39:49.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:49 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi114 to 176.1M 2024-05-27T07:39:49.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:49 smithi186 ceph-mon[36647]: Unable to set osd_memory_target on smithi114 to 184753049: error parsing value: Value '184753049' is below minimum 939524096 2024-05-27T07:39:49.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:49 smithi186 ceph-mon[36647]: Updating smithi114:/etc/ceph/ceph.conf 2024-05-27T07:39:49.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:49 smithi186 ceph-mon[36647]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-27T07:39:49.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:49 smithi186 ceph-mon[36647]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-27T07:39:49.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:49 smithi186 ceph-mon[36647]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-05-27T07:39:50.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:39:50.765 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:50.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: Updating smithi114:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:39:50.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:39:50.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:39:50.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:39:50.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:39:50.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:39:50.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: Updating smithi114:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:39:50.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:39:50.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:39:50.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:50.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:50.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:50.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:50 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3324917544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:50.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: Updating smithi114:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:39:50.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:39:50.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:39:50.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:39:50.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:39:50.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:39:50.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: Updating smithi114:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:50 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3324917544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:50.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: Updating smithi114:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:39:50.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:39:50.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:39:50.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:39:50.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:39:50.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:39:50.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: Updating smithi114:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:39:50.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: Updating smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:39:50.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: Updating smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:39:50.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:50.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:50.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:50.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:50.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:50 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3324917544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:51.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:51 smithi114 ceph-mon[32038]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-05-27T07:39:51.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:51 smithi163 ceph-mon[36451]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-05-27T07:39:51.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:51 smithi186 ceph-mon[36647]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-05-27T07:39:52.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.497+0000 7fafaefb0700 1 -- 172.21.15.114:0/1779291680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa8107e00 msgr2=0x7fafa8108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.497+0000 7fafaefb0700 1 --2- 172.21.15.114:0/1779291680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa8107e00 0x7fafa8108210 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fafa40099c0 tx=0x7fafa4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.497+0000 7fafaefb0700 1 -- 172.21.15.114:0/1779291680 shutdown_connections 2024-05-27T07:39:52.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.497+0000 7fafaefb0700 1 --2- 172.21.15.114:0/1779291680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafa8108750 0x7fafa81100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.497+0000 7fafaefb0700 1 --2- 172.21.15.114:0/1779291680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafa81034d0 0x7fafa81038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.497+0000 7fafaefb0700 1 --2- 172.21.15.114:0/1779291680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa8107e00 0x7fafa8108210 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.497+0000 7fafaefb0700 1 -- 172.21.15.114:0/1779291680 >> 172.21.15.114:0/1779291680 conn(0x7fafa80fef20 msgr2=0x7fafa8101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.497+0000 7fafaefb0700 1 -- 172.21.15.114:0/1779291680 shutdown_connections 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.497+0000 7fafaefb0700 1 -- 172.21.15.114:0/1779291680 wait complete. 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.498+0000 7fafaefb0700 1 Processor -- start 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.498+0000 7fafaefb0700 1 -- start start 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.498+0000 7fafaefb0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafa81034d0 0x7fafa80ff400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.498+0000 7fafaefb0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa8107e00 0x7fafa80ff940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.498+0000 7fafaefb0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafa8108750 0x7fafa80ffe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7fafaefb0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fafa8113360 con 0x7fafa8107e00 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7fafaefb0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fafa81131e0 con 0x7fafa81034d0 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7fafaefb0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fafa81134e0 con 0x7fafa8108750 2024-05-27T07:39:52.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7faf9ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa8107e00 0x7fafa80ff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7faf9ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa8107e00 0x7fafa80ff940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43618/0 (socket says 172.21.15.114:43618) 2024-05-27T07:39:52.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7faf9ffff700 1 -- 172.21.15.114:0/2936287357 learned_addr learned my addr 172.21.15.114:0/2936287357 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:52.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7fafacd4c700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafa81034d0 0x7fafa80ff400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7fafad54d700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafa8108750 0x7fafa80ffe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7faf9ffff700 1 -- 172.21.15.114:0/2936287357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafa8108750 msgr2=0x7fafa80ffe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7faf9ffff700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafa8108750 0x7fafa80ffe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7faf9ffff700 1 -- 172.21.15.114:0/2936287357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafa81034d0 msgr2=0x7fafa80ff400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7faf9ffff700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafa81034d0 0x7fafa80ff400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7faf9ffff700 1 -- 172.21.15.114:0/2936287357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa81a91e0 con 0x7fafa8107e00 2024-05-27T07:39:52.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7fafacd4c700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafa81034d0 0x7fafa80ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:52.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7faf9ffff700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa8107e00 0x7fafa80ff940 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fafa4006010 tx=0x7fafa40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:52.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7faf9dffb700 1 -- 172.21.15.114:0/2936287357 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa4027070 con 0x7fafa8107e00 2024-05-27T07:39:52.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7fafaefb0700 1 -- 172.21.15.114:0/2936287357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa81a9470 con 0x7fafa8107e00 2024-05-27T07:39:52.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7faf9dffb700 1 -- 172.21.15.114:0/2936287357 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafa4003fa0 con 0x7fafa8107e00 2024-05-27T07:39:52.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.499+0000 7faf9dffb700 1 -- 172.21.15.114:0/2936287357 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa4022dd0 con 0x7fafa8107e00 2024-05-27T07:39:52.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.500+0000 7fafaefb0700 1 -- 172.21.15.114:0/2936287357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa81a9920 con 0x7fafa8107e00 2024-05-27T07:39:52.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.501+0000 7fafaefb0700 1 -- 172.21.15.114:0/2936287357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafa80512b0 con 0x7fafa8107e00 2024-05-27T07:39:52.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.503+0000 7faf9dffb700 1 -- 172.21.15.114:0/2936287357 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 23) ==== 149829+0+0 (secure 0 0 0) 0x7fafa402a070 con 0x7fafa8107e00 2024-05-27T07:39:52.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.503+0000 7faf9dffb700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7faf900b2a80 0x7faf900b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.504+0000 7fafacd4c700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7faf900b2a80 0x7faf900b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.504+0000 7faf9dffb700 1 -- 172.21.15.114:0/2936287357 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fafa40e3d10 con 0x7fafa8107e00 2024-05-27T07:39:52.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.504+0000 7fafacd4c700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7faf900b2a80 0x7faf900b4f90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faf98005950 tx=0x7faf98005f50 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:52.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.507+0000 7faf9dffb700 1 -- 172.21.15.114:0/2936287357 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fafa4031080 con 0x7fafa8107e00 2024-05-27T07:39:52.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.713+0000 7fafaefb0700 1 -- 172.21.15.114:0/2936287357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fafa81a9c40 con 0x7fafa8107e00 2024-05-27T07:39:52.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.715+0000 7faf9dffb700 1 -- 172.21.15.114:0/2936287357 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25307 (secure 0 0 0) 0x7fafa4013070 con 0x7fafa8107e00 2024-05-27T07:39:52.712 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:39:52.712 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":90,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T07:39:44.126298+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/1628485461":"2024-05-28T07:39:44.126253+0000","172.21.15.114:0/1761274168":"2024-05-28T07:39:44.126253+0000","172.21.15.114:0/2574023732":"2024-05-28T07:39:44.126253+0000","172.21.15.114:6800/2291864634":"2024-05-28T07:39:44.126253+0000","172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2381281099":"2024-05-28T07:39:44.126253+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/2291864634":"2024-05-28T07:39:44.126253+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:39:52.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.717+0000 7fafaefb0700 1 -- 172.21.15.114:0/2936287357 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7faf900b2a80 msgr2=0x7faf900b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.717+0000 7fafaefb0700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7faf900b2a80 0x7faf900b4f90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faf98005950 tx=0x7faf98005f50 comp rx=0 tx=0).stop 2024-05-27T07:39:52.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.717+0000 7fafaefb0700 1 -- 172.21.15.114:0/2936287357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa8107e00 msgr2=0x7fafa80ff940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.717+0000 7fafaefb0700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa8107e00 0x7fafa80ff940 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fafa4006010 tx=0x7fafa40047b0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.717+0000 7fafaefb0700 1 -- 172.21.15.114:0/2936287357 shutdown_connections 2024-05-27T07:39:52.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.717+0000 7fafaefb0700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafa8108750 0x7fafa80ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.717+0000 7fafaefb0700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7faf900b2a80 0x7faf900b4f90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.717+0000 7fafaefb0700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafa81034d0 0x7fafa80ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.717+0000 7fafaefb0700 1 --2- 172.21.15.114:0/2936287357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa8107e00 0x7fafa80ff940 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.717+0000 7fafaefb0700 1 -- 172.21.15.114:0/2936287357 >> 172.21.15.114:0/2936287357 conn(0x7fafa80fef20 msgr2=0x7fafa81096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:52.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.717+0000 7fafaefb0700 1 -- 172.21.15.114:0/2936287357 shutdown_connections 2024-05-27T07:39:52.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:52.718+0000 7fafaefb0700 1 -- 172.21.15.114:0/2936287357 wait complete. 2024-05-27T07:39:53.415 DEBUG:tasks.fwd_scrub:Yielding 2024-05-27T07:39:53.415 INFO:teuthology.run_tasks:Running task exec... 2024-05-27T07:39:53.424 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:39:53.424 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-05-27T07:39:53.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-05-27T07:39:53.424 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-05-27T07:39:53.424 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:39:53.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:39:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.544+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/3468377960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e410b860 msgr2=0x7fd9e410dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.544+0000 7fd9ebb4d640 1 --2- 172.21.15.114:0/3468377960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e410b860 0x7fd9e410dc80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0009900 tx=0x7fd9e002a510 comp rx=0 tx=0).stop 2024-05-27T07:39:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.544+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/3468377960 shutdown_connections 2024-05-27T07:39:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.544+0000 7fd9ebb4d640 1 --2- 172.21.15.114:0/3468377960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9e410e210 0x7fd9e4110700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.544+0000 7fd9ebb4d640 1 --2- 172.21.15.114:0/3468377960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e410b860 0x7fd9e410dc80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.544+0000 7fd9ebb4d640 1 --2- 172.21.15.114:0/3468377960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9e4104e00 0x7fd9e4105210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.544+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/3468377960 >> 172.21.15.114:0/3468377960 conn(0x7fd9e4079290 msgr2=0x7fd9e40796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.544+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/3468377960 shutdown_connections 2024-05-27T07:39:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.544+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/3468377960 wait complete. 2024-05-27T07:39:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.545+0000 7fd9ebb4d640 1 Processor -- start 2024-05-27T07:39:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.545+0000 7fd9ebb4d640 1 -- start start 2024-05-27T07:39:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.545+0000 7fd9ebb4d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4104e00 0x7fd9e41a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.545+0000 7fd9ebb4d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9e410b860 0x7fd9e41a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.545+0000 7fd9ebb4d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9e410e210 0x7fd9e419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.545+0000 7fd9ebb4d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e4113ed0 con 0x7fd9e4104e00 2024-05-27T07:39:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.545+0000 7fd9ebb4d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e4113d50 con 0x7fd9e410e210 2024-05-27T07:39:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.545+0000 7fd9ebb4d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e4114050 con 0x7fd9e410b860 2024-05-27T07:39:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.545+0000 7fd9e98e9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4104e00 0x7fd9e41a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.545+0000 7fd9e98e9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4104e00 0x7fd9e41a05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43632/0 (socket says 172.21.15.114:43632) 2024-05-27T07:39:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.545+0000 7fd9e98e9640 1 -- 172.21.15.114:0/1344050402 learned_addr learned my addr 172.21.15.114:0/1344050402 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.546+0000 7fd9e98e9640 1 -- 172.21.15.114:0/1344050402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9e410b860 msgr2=0x7fd9e41a0b10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:39:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.546+0000 7fd9e98e9640 1 --2- 172.21.15.114:0/1344050402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9e410b860 0x7fd9e41a0b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.546+0000 7fd9e98e9640 1 -- 172.21.15.114:0/1344050402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9e410e210 msgr2=0x7fd9e419a750 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:39:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.546+0000 7fd9e98e9640 1 --2- 172.21.15.114:0/1344050402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9e410e210 0x7fd9e419a750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.546+0000 7fd9e98e9640 1 -- 172.21.15.114:0/1344050402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e419ae80 con 0x7fd9e4104e00 2024-05-27T07:39:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.546+0000 7fd9e98e9640 1 --2- 172.21.15.114:0/1344050402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4104e00 0x7fd9e41a05d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800b4a0 tx=0x7fd9d800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.546+0000 7fd9d6ffd640 1 -- 172.21.15.114:0/1344050402 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8014070 con 0x7fd9e4104e00 2024-05-27T07:39:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.546+0000 7fd9d6ffd640 1 -- 172.21.15.114:0/1344050402 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9d8010ac0 con 0x7fd9e4104e00 2024-05-27T07:39:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.546+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/1344050402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9e419b110 con 0x7fd9e4104e00 2024-05-27T07:39:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.546+0000 7fd9d6ffd640 1 -- 172.21.15.114:0/1344050402 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8017c30 con 0x7fd9e4104e00 2024-05-27T07:39:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.546+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/1344050402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9e41a7370 con 0x7fd9e4104e00 2024-05-27T07:39:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.547+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/1344050402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9e4075770 con 0x7fd9e4104e00 2024-05-27T07:39:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.549+0000 7fd9d6ffd640 1 -- 172.21.15.114:0/1344050402 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 23) ==== 149829+0+0 (secure 0 0 0) 0x7fd9d8017070 con 0x7fd9e4104e00 2024-05-27T07:39:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.550+0000 7fd9d6ffd640 1 --2- 172.21.15.114:0/1344050402 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd9d00b29e0 0x7fd9d00b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.550+0000 7fd9e90e8640 1 --2- 172.21.15.114:0/1344050402 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd9d00b29e0 0x7fd9d00b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.550+0000 7fd9d6ffd640 1 -- 172.21.15.114:0/1344050402 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fd9d80e2210 con 0x7fd9e4104e00 2024-05-27T07:39:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.550+0000 7fd9e90e8640 1 --2- 172.21.15.114:0/1344050402 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd9d00b29e0 0x7fd9d00b4ef0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd9e002aa30 tx=0x7fd9e0000bc0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.559+0000 7fd9d6ffd640 1 -- 172.21.15.114:0/1344050402 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd9d80a97e0 con 0x7fd9e4104e00 2024-05-27T07:39:53.711 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:53.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.801+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/1344050402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7fd9e41a75e0 con 0x7fd9e4104e00 2024-05-27T07:39:53.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:53 smithi114 ceph-mon[32038]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-27T07:39:53.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:53 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2936287357' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:39:53.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.808+0000 7fd9d6ffd640 1 -- 172.21.15.114:0/1344050402 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v30)=0 v30) ==== 129+0+0 (secure 0 0 0) 0x7fd9d80a90e0 con 0x7fd9e4104e00 2024-05-27T07:39:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.810+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/1344050402 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd9d00b29e0 msgr2=0x7fd9d00b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.810+0000 7fd9ebb4d640 1 --2- 172.21.15.114:0/1344050402 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd9d00b29e0 0x7fd9d00b4ef0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd9e002aa30 tx=0x7fd9e0000bc0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.810+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/1344050402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4104e00 msgr2=0x7fd9e41a05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:53.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.810+0000 7fd9ebb4d640 1 --2- 172.21.15.114:0/1344050402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4104e00 0x7fd9e41a05d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800b4a0 tx=0x7fd9d800b960 comp rx=0 tx=0).stop 2024-05-27T07:39:53.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.810+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/1344050402 shutdown_connections 2024-05-27T07:39:53.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.810+0000 7fd9ebb4d640 1 --2- 172.21.15.114:0/1344050402 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd9d00b29e0 0x7fd9d00b4ef0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.810+0000 7fd9ebb4d640 1 --2- 172.21.15.114:0/1344050402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9e410e210 0x7fd9e419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.810+0000 7fd9ebb4d640 1 --2- 172.21.15.114:0/1344050402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9e410b860 0x7fd9e41a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.810+0000 7fd9ebb4d640 1 --2- 172.21.15.114:0/1344050402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4104e00 0x7fd9e41a05d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.810+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/1344050402 >> 172.21.15.114:0/1344050402 conn(0x7fd9e4079290 msgr2=0x7fd9e4101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:53.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.810+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/1344050402 shutdown_connections 2024-05-27T07:39:53.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.810+0000 7fd9ebb4d640 1 -- 172.21.15.114:0/1344050402 wait complete. 2024-05-27T07:39:53.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-05-27T07:39:53.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.939+0000 7ff3131ee640 1 -- 172.21.15.114:0/772038557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff30c1090b0 msgr2=0x7ff30c103350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:53.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.939+0000 7ff3131ee640 1 --2- 172.21.15.114:0/772038557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff30c1090b0 0x7ff30c103350 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4009900 tx=0x7ff2f402a410 comp rx=0 tx=0).stop 2024-05-27T07:39:53.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.940+0000 7ff3131ee640 1 -- 172.21.15.114:0/772038557 shutdown_connections 2024-05-27T07:39:53.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.940+0000 7ff3131ee640 1 --2- 172.21.15.114:0/772038557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff30c1039c0 0x7ff30c10f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.940+0000 7ff3131ee640 1 --2- 172.21.15.114:0/772038557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff30c1090b0 0x7ff30c103350 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.940+0000 7ff3131ee640 1 --2- 172.21.15.114:0/772038557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff30c108c10 0x7ff30c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.940+0000 7ff3131ee640 1 -- 172.21.15.114:0/772038557 >> 172.21.15.114:0/772038557 conn(0x7ff30c0779d0 msgr2=0x7ff30c077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:53.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.940+0000 7ff3131ee640 1 -- 172.21.15.114:0/772038557 shutdown_connections 2024-05-27T07:39:53.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.940+0000 7ff3131ee640 1 -- 172.21.15.114:0/772038557 wait complete. 2024-05-27T07:39:53.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff3131ee640 1 Processor -- start 2024-05-27T07:39:53.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff3131ee640 1 -- start start 2024-05-27T07:39:53.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff3131ee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff30c1039c0 0x7ff30c19b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:53.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff3131ee640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff30c108c10 0x7ff30c19bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:53.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff3131ee640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff30c1090b0 0x7ff30c19c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:53.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff3131ee640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c112e50 con 0x7ff30c1090b0 2024-05-27T07:39:53.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff3131ee640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c112cd0 con 0x7ff30c1039c0 2024-05-27T07:39:53.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff3131ee640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c112fd0 con 0x7ff30c108c10 2024-05-27T07:39:53.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff31178b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff30c1090b0 0x7ff30c19c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:53.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff31178b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff30c1090b0 0x7ff30c19c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43656/0 (socket says 172.21.15.114:43656) 2024-05-27T07:39:53.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff31178b640 1 -- 172.21.15.114:0/2852133020 learned_addr learned my addr 172.21.15.114:0/2852133020 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:53.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff310f8a640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff30c1039c0 0x7ff30c19b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:53.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff30bfff640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff30c108c10 0x7ff30c19bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff31178b640 1 -- 172.21.15.114:0/2852133020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff30c108c10 msgr2=0x7ff30c19bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff31178b640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff30c108c10 0x7ff30c19bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff31178b640 1 -- 172.21.15.114:0/2852133020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff30c1039c0 msgr2=0x7ff30c19b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff31178b640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff30c1039c0 0x7ff30c19b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff31178b640 1 -- 172.21.15.114:0/2852133020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff30c1a6a80 con 0x7ff30c1090b0 2024-05-27T07:39:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.941+0000 7ff30bfff640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff30c108c10 0x7ff30c19bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.942+0000 7ff31178b640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff30c1090b0 0x7ff30c19c1f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff30400eaa0 tx=0x7ff30400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.942+0000 7ff309ffb640 1 -- 172.21.15.114:0/2852133020 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30400cdb0 con 0x7ff30c1090b0 2024-05-27T07:39:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.942+0000 7ff309ffb640 1 -- 172.21.15.114:0/2852133020 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff304005020 con 0x7ff30c1090b0 2024-05-27T07:39:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.942+0000 7ff309ffb640 1 -- 172.21.15.114:0/2852133020 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3040116d0 con 0x7ff30c1090b0 2024-05-27T07:39:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.942+0000 7ff3131ee640 1 -- 172.21.15.114:0/2852133020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff30c1a6d70 con 0x7ff30c1090b0 2024-05-27T07:39:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.942+0000 7ff3131ee640 1 -- 172.21.15.114:0/2852133020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff30c1a71d0 con 0x7ff30c1090b0 2024-05-27T07:39:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.943+0000 7ff3131ee640 1 -- 172.21.15.114:0/2852133020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff30c050860 con 0x7ff30c1090b0 2024-05-27T07:39:53.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.944+0000 7ff309ffb640 1 -- 172.21.15.114:0/2852133020 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 23) ==== 149829+0+0 (secure 0 0 0) 0x7ff3040051d0 con 0x7ff30c1090b0 2024-05-27T07:39:53.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.945+0000 7ff309ffb640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2f80b29e0 0x7ff2f80b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:53.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.945+0000 7ff310f8a640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2f80b29e0 0x7ff2f80b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:53.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.945+0000 7ff309ffb640 1 -- 172.21.15.114:0/2852133020 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7ff3040e30e0 con 0x7ff30c1090b0 2024-05-27T07:39:53.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.946+0000 7ff310f8a640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2f80b29e0 0x7ff2f80b4ef0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc005d00 tx=0x7ff2fc00b410 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:53.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:53.949+0000 7ff309ffb640 1 -- 172.21.15.114:0/2852133020 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff304016030 con 0x7ff30c1090b0 2024-05-27T07:39:53.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:53 smithi163 ceph-mon[36451]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-27T07:39:53.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:53 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2936287357' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:39:53.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:53 smithi186 ceph-mon[36647]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-27T07:39:53.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:53 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2936287357' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:39:54.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.146+0000 7ff3131ee640 1 -- 172.21.15.114:0/2852133020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7ff30c1022f0 con 0x7ff30c1090b0 2024-05-27T07:39:54.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.151+0000 7ff309ffb640 1 -- 172.21.15.114:0/2852133020 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) ==== 138+0+0 (secure 0 0 0) 0x7ff3040aa5a0 con 0x7ff30c1090b0 2024-05-27T07:39:54.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.153+0000 7ff3131ee640 1 -- 172.21.15.114:0/2852133020 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2f80b29e0 msgr2=0x7ff2f80b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:54.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.153+0000 7ff3131ee640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2f80b29e0 0x7ff2f80b4ef0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc005d00 tx=0x7ff2fc00b410 comp rx=0 tx=0).stop 2024-05-27T07:39:54.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.153+0000 7ff3131ee640 1 -- 172.21.15.114:0/2852133020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff30c1090b0 msgr2=0x7ff30c19c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:54.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.153+0000 7ff3131ee640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff30c1090b0 0x7ff30c19c1f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff30400eaa0 tx=0x7ff30400ef60 comp rx=0 tx=0).stop 2024-05-27T07:39:54.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.153+0000 7ff3131ee640 1 -- 172.21.15.114:0/2852133020 shutdown_connections 2024-05-27T07:39:54.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.153+0000 7ff3131ee640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2f80b29e0 0x7ff2f80b4ef0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.153+0000 7ff3131ee640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff30c1090b0 0x7ff30c19c1f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.154+0000 7ff3131ee640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff30c108c10 0x7ff30c19bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.154+0000 7ff3131ee640 1 --2- 172.21.15.114:0/2852133020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff30c1039c0 0x7ff30c19b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.154+0000 7ff3131ee640 1 -- 172.21.15.114:0/2852133020 >> 172.21.15.114:0/2852133020 conn(0x7ff30c0779d0 msgr2=0x7ff30c100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:54.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.154+0000 7ff3131ee640 1 -- 172.21.15.114:0/2852133020 shutdown_connections 2024-05-27T07:39:54.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.154+0000 7ff3131ee640 1 -- 172.21.15.114:0/2852133020 wait complete. 2024-05-27T07:39:54.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-05-27T07:39:54.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.284+0000 7fe1267c7640 1 -- 172.21.15.114:0/2467947452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe120108b80 msgr2=0x7fe120102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:54.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.284+0000 7fe1267c7640 1 --2- 172.21.15.114:0/2467947452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe120108b80 0x7fe120102720 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe110009900 tx=0x7fe11002a410 comp rx=0 tx=0).stop 2024-05-27T07:39:54.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.284+0000 7fe1267c7640 1 -- 172.21.15.114:0/2467947452 shutdown_connections 2024-05-27T07:39:54.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.284+0000 7fe1267c7640 1 --2- 172.21.15.114:0/2467947452 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1201032d0 0x7fe12010f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.284+0000 7fe1267c7640 1 --2- 172.21.15.114:0/2467947452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe120109020 0x7fe120102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.284+0000 7fe1267c7640 1 --2- 172.21.15.114:0/2467947452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe120108b80 0x7fe120102720 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.284+0000 7fe1267c7640 1 -- 172.21.15.114:0/2467947452 >> 172.21.15.114:0/2467947452 conn(0x7fe1200fe380 msgr2=0x7fe1201007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:54.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.284+0000 7fe1267c7640 1 -- 172.21.15.114:0/2467947452 shutdown_connections 2024-05-27T07:39:54.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.284+0000 7fe1267c7640 1 -- 172.21.15.114:0/2467947452 wait complete. 2024-05-27T07:39:54.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe1267c7640 1 Processor -- start 2024-05-27T07:39:54.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe1267c7640 1 -- start start 2024-05-27T07:39:54.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe1267c7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1201032d0 0x7fe1201997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:54.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe1267c7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe120108b80 0x7fe120199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:54.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe1267c7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe120109020 0x7fe12019e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:54.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe1267c7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe120112d70 con 0x7fe1201032d0 2024-05-27T07:39:54.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe1267c7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe120112bf0 con 0x7fe120108b80 2024-05-27T07:39:54.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe1267c7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe120112ef0 con 0x7fe120109020 2024-05-27T07:39:54.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe11ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1201032d0 0x7fe1201997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:54.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe11ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1201032d0 0x7fe1201997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43676/0 (socket says 172.21.15.114:43676) 2024-05-27T07:39:54.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe11ffff640 1 -- 172.21.15.114:0/2352008632 learned_addr learned my addr 172.21.15.114:0/2352008632 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:54.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe124d64640 1 --2- 172.21.15.114:0/2352008632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe120109020 0x7fe12019e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:54.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe11ffff640 1 -- 172.21.15.114:0/2352008632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe120109020 msgr2=0x7fe12019e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:54.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe11ffff640 1 --2- 172.21.15.114:0/2352008632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe120109020 0x7fe12019e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe11ffff640 1 -- 172.21.15.114:0/2352008632 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe120108b80 msgr2=0x7fe120199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:54.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe11ffff640 1 --2- 172.21.15.114:0/2352008632 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe120108b80 0x7fe120199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.285+0000 7fe11ffff640 1 -- 172.21.15.114:0/2352008632 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe12019e650 con 0x7fe1201032d0 2024-05-27T07:39:54.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.286+0000 7fe11ffff640 1 --2- 172.21.15.114:0/2352008632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1201032d0 0x7fe1201997f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe110006010 tx=0x7fe110004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:54.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.286+0000 7fe11d7fa640 1 -- 172.21.15.114:0/2352008632 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11003f030 con 0x7fe1201032d0 2024-05-27T07:39:54.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.286+0000 7fe11d7fa640 1 -- 172.21.15.114:0/2352008632 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe110003aa0 con 0x7fe1201032d0 2024-05-27T07:39:54.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.286+0000 7fe1267c7640 1 -- 172.21.15.114:0/2352008632 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe12019e8e0 con 0x7fe1201032d0 2024-05-27T07:39:54.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.286+0000 7fe11d7fa640 1 -- 172.21.15.114:0/2352008632 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe110035070 con 0x7fe1201032d0 2024-05-27T07:39:54.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.286+0000 7fe1267c7640 1 -- 172.21.15.114:0/2352008632 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe12019ee20 con 0x7fe1201032d0 2024-05-27T07:39:54.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.287+0000 7fe1267c7640 1 -- 172.21.15.114:0/2352008632 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe120050860 con 0x7fe1201032d0 2024-05-27T07:39:54.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.289+0000 7fe11d7fa640 1 -- 172.21.15.114:0/2352008632 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 23) ==== 149829+0+0 (secure 0 0 0) 0x7fe110042030 con 0x7fe1201032d0 2024-05-27T07:39:54.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.289+0000 7fe11d7fa640 1 --2- 172.21.15.114:0/2352008632 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe1080b2a60 0x7fe1080b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:54.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.289+0000 7fe11f7fe640 1 --2- 172.21.15.114:0/2352008632 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe1080b2a60 0x7fe1080b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:54.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.289+0000 7fe11d7fa640 1 -- 172.21.15.114:0/2352008632 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fe110103410 con 0x7fe1201032d0 2024-05-27T07:39:54.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.290+0000 7fe11f7fe640 1 --2- 172.21.15.114:0/2352008632 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe1080b2a60 0x7fe1080b4f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe120100090 tx=0x7fe114005b80 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:54.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.293+0000 7fe11d7fa640 1 -- 172.21.15.114:0/2352008632 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe1100ca8d0 con 0x7fe1201032d0 2024-05-27T07:39:54.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.481+0000 7fe1267c7640 1 -- 172.21.15.114:0/2352008632 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fe1201a6380 con 0x7fe1201032d0 2024-05-27T07:39:54.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.485+0000 7fe11d7fa640 1 -- 172.21.15.114:0/2352008632 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) ==== 133+0+0 (secure 0 0 0) 0x7fe1100ca1d0 con 0x7fe1201032d0 2024-05-27T07:39:54.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.488+0000 7fe1267c7640 1 -- 172.21.15.114:0/2352008632 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe1080b2a60 msgr2=0x7fe1080b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:54.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.488+0000 7fe1267c7640 1 --2- 172.21.15.114:0/2352008632 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe1080b2a60 0x7fe1080b4f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe120100090 tx=0x7fe114005b80 comp rx=0 tx=0).stop 2024-05-27T07:39:54.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.488+0000 7fe1267c7640 1 -- 172.21.15.114:0/2352008632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1201032d0 msgr2=0x7fe1201997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:54.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.488+0000 7fe1267c7640 1 --2- 172.21.15.114:0/2352008632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1201032d0 0x7fe1201997f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe110006010 tx=0x7fe110004060 comp rx=0 tx=0).stop 2024-05-27T07:39:54.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.488+0000 7fe1267c7640 1 -- 172.21.15.114:0/2352008632 shutdown_connections 2024-05-27T07:39:54.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.488+0000 7fe1267c7640 1 --2- 172.21.15.114:0/2352008632 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe1080b2a60 0x7fe1080b4f70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.488+0000 7fe1267c7640 1 --2- 172.21.15.114:0/2352008632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe120109020 0x7fe12019e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.488+0000 7fe1267c7640 1 --2- 172.21.15.114:0/2352008632 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe120108b80 0x7fe120199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.488+0000 7fe1267c7640 1 --2- 172.21.15.114:0/2352008632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1201032d0 0x7fe1201997f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.488+0000 7fe1267c7640 1 -- 172.21.15.114:0/2352008632 >> 172.21.15.114:0/2352008632 conn(0x7fe1200fe380 msgr2=0x7fe1200fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:54.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.488+0000 7fe1267c7640 1 -- 172.21.15.114:0/2352008632 shutdown_connections 2024-05-27T07:39:54.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.488+0000 7fe1267c7640 1 -- 172.21.15.114:0/2352008632 wait complete. 2024-05-27T07:39:54.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-05-27T07:39:54.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.621+0000 7f03d1261640 1 -- 172.21.15.114:0/3396166395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc1021a0 msgr2=0x7f03cc1025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:54.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.621+0000 7f03d1261640 1 --2- 172.21.15.114:0/3396166395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc1021a0 0x7f03cc1025b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f03b4009900 tx=0x7f03b402a410 comp rx=0 tx=0).stop 2024-05-27T07:39:54.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 -- 172.21.15.114:0/3396166395 shutdown_connections 2024-05-27T07:39:54.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 --2- 172.21.15.114:0/3396166395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03cc10e6a0 0x7f03cc110ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 --2- 172.21.15.114:0/3396166395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc1021a0 0x7f03cc1025b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 --2- 172.21.15.114:0/3396166395 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03cc109cf0 0x7f03cc101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 -- 172.21.15.114:0/3396166395 >> 172.21.15.114:0/3396166395 conn(0x7f03cc0fda10 msgr2=0x7f03cc0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 -- 172.21.15.114:0/3396166395 shutdown_connections 2024-05-27T07:39:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 -- 172.21.15.114:0/3396166395 wait complete. 2024-05-27T07:39:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 Processor -- start 2024-05-27T07:39:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 -- start start 2024-05-27T07:39:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03cc1021a0 0x7f03cc1a0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc109cf0 0x7f03cc1a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03cc10e6a0 0x7f03cc19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc113e60 con 0x7f03cc109cf0 2024-05-27T07:39:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc113ce0 con 0x7f03cc1021a0 2024-05-27T07:39:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.622+0000 7f03d1261640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc113fe0 con 0x7f03cc10e6a0 2024-05-27T07:39:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc109cf0 0x7f03cc1a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc109cf0 0x7f03cc1a0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36840/0 (socket says 172.21.15.114:36840) 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03c3fff640 1 -- 172.21.15.114:0/1263337904 learned_addr learned my addr 172.21.15.114:0/1263337904 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03caffd640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03cc1021a0 0x7f03cc1a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03cb7fe640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03cc10e6a0 0x7f03cc19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03c3fff640 1 -- 172.21.15.114:0/1263337904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03cc10e6a0 msgr2=0x7f03cc19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03c3fff640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03cc10e6a0 0x7f03cc19a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03c3fff640 1 -- 172.21.15.114:0/1263337904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03cc1021a0 msgr2=0x7f03cc1a0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03c3fff640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03cc1021a0 0x7f03cc1a0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03c3fff640 1 -- 172.21.15.114:0/1263337904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03cc19adf0 con 0x7f03cc109cf0 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03c3fff640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc109cf0 0x7f03cc1a0a80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f03b4032fd0 tx=0x7f03b4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03c8ff9640 1 -- 172.21.15.114:0/1263337904 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b403f030 con 0x7f03cc109cf0 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03d1261640 1 -- 172.21.15.114:0/1263337904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03cc19b020 con 0x7f03cc109cf0 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03c8ff9640 1 -- 172.21.15.114:0/1263337904 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03b4004490 con 0x7f03cc109cf0 2024-05-27T07:39:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03c8ff9640 1 -- 172.21.15.114:0/1263337904 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b4035070 con 0x7f03cc109cf0 2024-05-27T07:39:54.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.623+0000 7f03d1261640 1 -- 172.21.15.114:0/1263337904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03cc19b510 con 0x7f03cc109cf0 2024-05-27T07:39:54.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.624+0000 7f03d1261640 1 -- 172.21.15.114:0/1263337904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03cc104f80 con 0x7f03cc109cf0 2024-05-27T07:39:54.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.626+0000 7f03c8ff9640 1 -- 172.21.15.114:0/1263337904 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 23) ==== 149829+0+0 (secure 0 0 0) 0x7f03b4042030 con 0x7f03cc109cf0 2024-05-27T07:39:54.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.626+0000 7f03c8ff9640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03b80b29e0 0x7f03b80b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:54.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.627+0000 7f03caffd640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03b80b29e0 0x7f03b80b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:54.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.627+0000 7f03c8ff9640 1 -- 172.21.15.114:0/1263337904 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f03b4103460 con 0x7f03cc109cf0 2024-05-27T07:39:54.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.627+0000 7f03caffd640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03b80b29e0 0x7f03b80b4ef0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f03bc005fd0 tx=0x7f03bc00b410 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:54.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.630+0000 7f03c8ff9640 1 -- 172.21.15.114:0/1263337904 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f03b40ca920 con 0x7f03cc109cf0 2024-05-27T07:39:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.840+0000 7f03d1261640 1 -- 172.21.15.114:0/1263337904 --> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f03cc0fe530 con 0x7f03b80b29e0 2024-05-27T07:39:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.909+0000 7f03c8ff9640 1 -- 172.21.15.114:0/1263337904 <== mgr.15138 v2:172.21.15.114:6800/3311646739 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f03cc0fe530 con 0x7f03b80b29e0 2024-05-27T07:39:54.906 INFO:teuthology.orchestra.run.smithi114.stdout:Schedule set for path / 2024-05-27T07:39:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.912+0000 7f03d1261640 1 -- 172.21.15.114:0/1263337904 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03b80b29e0 msgr2=0x7f03b80b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.912+0000 7f03d1261640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03b80b29e0 0x7f03b80b4ef0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f03bc005fd0 tx=0x7f03bc00b410 comp rx=0 tx=0).stop 2024-05-27T07:39:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.912+0000 7f03d1261640 1 -- 172.21.15.114:0/1263337904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc109cf0 msgr2=0x7f03cc1a0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.912+0000 7f03d1261640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc109cf0 0x7f03cc1a0a80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f03b4032fd0 tx=0x7f03b4004270 comp rx=0 tx=0).stop 2024-05-27T07:39:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.912+0000 7f03d1261640 1 -- 172.21.15.114:0/1263337904 shutdown_connections 2024-05-27T07:39:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.912+0000 7f03d1261640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03b80b29e0 0x7f03b80b4ef0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.912+0000 7f03d1261640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03cc10e6a0 0x7f03cc19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.912+0000 7f03d1261640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc109cf0 0x7f03cc1a0a80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.912+0000 7f03d1261640 1 --2- 172.21.15.114:0/1263337904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03cc1021a0 0x7f03cc1a0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.912+0000 7f03d1261640 1 -- 172.21.15.114:0/1263337904 >> 172.21.15.114:0/1263337904 conn(0x7f03cc0fda10 msgr2=0x7f03cc0fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.912+0000 7f03d1261640 1 -- 172.21.15.114:0/1263337904 shutdown_connections 2024-05-27T07:39:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:54.912+0000 7f03d1261640 1 -- 172.21.15.114:0/1263337904 wait complete. 2024-05-27T07:39:54.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-05-27T07:39:55.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.040+0000 7f8a9d043640 1 -- 172.21.15.114:0/3923132533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98109040 msgr2=0x7f8a98102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.040+0000 7f8a9d043640 1 --2- 172.21.15.114:0/3923132533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98109040 0x7f8a98102c80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8a80009900 tx=0x7f8a8002a410 comp rx=0 tx=0).stop 2024-05-27T07:39:55.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.040+0000 7f8a9d043640 1 -- 172.21.15.114:0/3923132533 shutdown_connections 2024-05-27T07:39:55.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.040+0000 7f8a9d043640 1 --2- 172.21.15.114:0/3923132533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a981032f0 0x7f8a9810f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.040+0000 7f8a9d043640 1 --2- 172.21.15.114:0/3923132533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98109040 0x7f8a98102c80 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.040+0000 7f8a9d043640 1 --2- 172.21.15.114:0/3923132533 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a98108ba0 0x7f8a98102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.040+0000 7f8a9d043640 1 -- 172.21.15.114:0/3923132533 >> 172.21.15.114:0/3923132533 conn(0x7f8a980782a0 msgr2=0x7f8a98078700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.040+0000 7f8a9d043640 1 -- 172.21.15.114:0/3923132533 shutdown_connections 2024-05-27T07:39:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.040+0000 7f8a9d043640 1 -- 172.21.15.114:0/3923132533 wait complete. 2024-05-27T07:39:55.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.041+0000 7f8a9d043640 1 Processor -- start 2024-05-27T07:39:55.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.041+0000 7f8a9d043640 1 -- start start 2024-05-27T07:39:55.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.041+0000 7f8a9d043640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a981032f0 0x7f8a9819b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.041+0000 7f8a9d043640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98108ba0 0x7f8a9819bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.041+0000 7f8a9d043640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a98109040 0x7f8a9819c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.041+0000 7f8a9d043640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8a98112df0 con 0x7f8a98108ba0 2024-05-27T07:39:55.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.041+0000 7f8a9d043640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8a98112c70 con 0x7f8a981032f0 2024-05-27T07:39:55.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.041+0000 7f8a9d043640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8a98112f70 con 0x7f8a98109040 2024-05-27T07:39:55.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a8e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98108ba0 0x7f8a9819bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a8e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98108ba0 0x7f8a9819bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36862/0 (socket says 172.21.15.114:36862) 2024-05-27T07:39:55.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a8e59c640 1 -- 172.21.15.114:0/3046939595 learned_addr learned my addr 172.21.15.114:0/3046939595 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:55.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a96d9d640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a981032f0 0x7f8a9819b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a8e59c640 1 -- 172.21.15.114:0/3046939595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a98109040 msgr2=0x7f8a9819c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a8e59c640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a98109040 0x7f8a9819c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a8e59c640 1 -- 172.21.15.114:0/3046939595 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a981032f0 msgr2=0x7f8a9819b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a8e59c640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a981032f0 0x7f8a9819b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a8e59c640 1 -- 172.21.15.114:0/3046939595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a981a6a80 con 0x7f8a98108ba0 2024-05-27T07:39:55.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a96d9d640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a981032f0 0x7f8a9819b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:55.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a8e59c640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98108ba0 0x7f8a9819bd10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8a8002a930 tx=0x7f8a80004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a94d99640 1 -- 172.21.15.114:0/3046939595 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8003f030 con 0x7f8a98108ba0 2024-05-27T07:39:55.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a94d99640 1 -- 172.21.15.114:0/3046939595 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a80003cb0 con 0x7f8a98108ba0 2024-05-27T07:39:55.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a9d043640 1 -- 172.21.15.114:0/3046939595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a981a6d10 con 0x7f8a98108ba0 2024-05-27T07:39:55.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a94d99640 1 -- 172.21.15.114:0/3046939595 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a80035070 con 0x7f8a98108ba0 2024-05-27T07:39:55.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.042+0000 7f8a9d043640 1 -- 172.21.15.114:0/3046939595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a981a71b0 con 0x7f8a98108ba0 2024-05-27T07:39:55.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.044+0000 7f8a9d043640 1 -- 172.21.15.114:0/3046939595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a98050860 con 0x7f8a98108ba0 2024-05-27T07:39:55.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.045+0000 7f8a94d99640 1 -- 172.21.15.114:0/3046939595 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 23) ==== 149829+0+0 (secure 0 0 0) 0x7f8a80042030 con 0x7f8a98108ba0 2024-05-27T07:39:55.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.046+0000 7f8a94d99640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8a840b29e0 0x7f8a840b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.046+0000 7f8a96d9d640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8a840b29e0 0x7f8a840b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.046+0000 7f8a94d99640 1 -- 172.21.15.114:0/3046939595 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f8a80103670 con 0x7f8a98108ba0 2024-05-27T07:39:55.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.046+0000 7f8a96d9d640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8a840b29e0 0x7f8a840b4ef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8a88006fd0 tx=0x7f8a88008040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.050+0000 7f8a94d99640 1 -- 172.21.15.114:0/3046939595 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8a800cab30 con 0x7f8a98108ba0 2024-05-27T07:39:55.081 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1344050402' entity='client.admin' 2024-05-27T07:39:55.081 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:55.081 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:55.081 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:55.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:55.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-27T07:39:55.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2852133020' entity='client.admin' 2024-05-27T07:39:55.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:55.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:55.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:55.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:55.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2352008632' entity='client.admin' 2024-05-27T07:39:55.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:55.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:55.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:55.083 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:54 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:55.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1344050402' entity='client.admin' 2024-05-27T07:39:55.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:55.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:55.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:55.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:55.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-27T07:39:55.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2852133020' entity='client.admin' 2024-05-27T07:39:55.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:55.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:55.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:55.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:55.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2352008632' entity='client.admin' 2024-05-27T07:39:55.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:55.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:55.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:55.224 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:54 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:55.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1344050402' entity='client.admin' 2024-05-27T07:39:55.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:55.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:55.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:55.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:55.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-27T07:39:55.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2852133020' entity='client.admin' 2024-05-27T07:39:55.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:55.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:55.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:55.227 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:55.227 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2352008632' entity='client.admin' 2024-05-27T07:39:55.227 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:55.227 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:55.227 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:55.227 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:54 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:39:55.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.267+0000 7f8a9d043640 1 -- 172.21.15.114:0/3046939595 --> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f8a981019d0 con 0x7f8a840b29e0 2024-05-27T07:39:55.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.345+0000 7f8a94d99640 1 -- 172.21.15.114:0/3046939595 <== mgr.15138 v2:172.21.15.114:6800/3311646739 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f8a981019d0 con 0x7f8a840b29e0 2024-05-27T07:39:55.342 INFO:teuthology.orchestra.run.smithi114.stdout:Retention added to path / 2024-05-27T07:39:55.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.348+0000 7f8a9d043640 1 -- 172.21.15.114:0/3046939595 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8a840b29e0 msgr2=0x7f8a840b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.348+0000 7f8a9d043640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8a840b29e0 0x7f8a840b4ef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8a88006fd0 tx=0x7f8a88008040 comp rx=0 tx=0).stop 2024-05-27T07:39:55.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.348+0000 7f8a9d043640 1 -- 172.21.15.114:0/3046939595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98108ba0 msgr2=0x7f8a9819bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.348+0000 7f8a9d043640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98108ba0 0x7f8a9819bd10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8a8002a930 tx=0x7f8a80004270 comp rx=0 tx=0).stop 2024-05-27T07:39:55.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.348+0000 7f8a9d043640 1 -- 172.21.15.114:0/3046939595 shutdown_connections 2024-05-27T07:39:55.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.348+0000 7f8a9d043640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8a840b29e0 0x7f8a840b4ef0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.348+0000 7f8a9d043640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a98109040 0x7f8a9819c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.348+0000 7f8a9d043640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98108ba0 0x7f8a9819bd10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.348+0000 7f8a9d043640 1 --2- 172.21.15.114:0/3046939595 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a981032f0 0x7f8a9819b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.348+0000 7f8a9d043640 1 -- 172.21.15.114:0/3046939595 >> 172.21.15.114:0/3046939595 conn(0x7f8a980782a0 msgr2=0x7f8a981012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.348+0000 7f8a9d043640 1 -- 172.21.15.114:0/3046939595 shutdown_connections 2024-05-27T07:39:55.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.348+0000 7f8a9d043640 1 -- 172.21.15.114:0/3046939595 wait complete. 2024-05-27T07:39:55.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-27T07:39:55.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.476+0000 7f0866934640 1 -- 172.21.15.114:0/479093209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0860109030 msgr2=0x7f0860102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.476+0000 7f0866934640 1 --2- 172.21.15.114:0/479093209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0860109030 0x7f0860102c70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0848009900 tx=0x7f084802a410 comp rx=0 tx=0).stop 2024-05-27T07:39:55.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.476+0000 7f0866934640 1 -- 172.21.15.114:0/479093209 shutdown_connections 2024-05-27T07:39:55.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.476+0000 7f0866934640 1 --2- 172.21.15.114:0/479093209 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08601032e0 0x7f086010f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.476+0000 7f0866934640 1 --2- 172.21.15.114:0/479093209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0860109030 0x7f0860102c70 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.476+0000 7f0866934640 1 --2- 172.21.15.114:0/479093209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860108b90 0x7f0860102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.476+0000 7f0866934640 1 -- 172.21.15.114:0/479093209 >> 172.21.15.114:0/479093209 conn(0x7f08600782d0 msgr2=0x7f0860078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.476+0000 7f0866934640 1 -- 172.21.15.114:0/479093209 shutdown_connections 2024-05-27T07:39:55.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.476+0000 7f0866934640 1 -- 172.21.15.114:0/479093209 wait complete. 2024-05-27T07:39:55.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.477+0000 7f0866934640 1 Processor -- start 2024-05-27T07:39:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.477+0000 7f0866934640 1 -- start start 2024-05-27T07:39:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.477+0000 7f0866934640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08601032e0 0x7f086019f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.477+0000 7f0866934640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860108b90 0x7f086019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.477+0000 7f0866934640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0860109030 0x7f08601996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.477+0000 7f0866934640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0860112d80 con 0x7f08601032e0 2024-05-27T07:39:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.477+0000 7f0866934640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0860112c00 con 0x7f0860109030 2024-05-27T07:39:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.477+0000 7f0866934640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0860112f00 con 0x7f0860108b90 2024-05-27T07:39:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f085ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08601032e0 0x7f086019f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f085ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08601032e0 0x7f086019f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36888/0 (socket says 172.21.15.114:36888) 2024-05-27T07:39:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f085ffff640 1 -- 172.21.15.114:0/3752619212 learned_addr learned my addr 172.21.15.114:0/3752619212 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f0857fff640 1 --2- 172.21.15.114:0/3752619212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860108b90 0x7f086019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f085ffff640 1 -- 172.21.15.114:0/3752619212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860108b90 msgr2=0x7f086019faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f085ffff640 1 --2- 172.21.15.114:0/3752619212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860108b90 0x7f086019faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f085ffff640 1 -- 172.21.15.114:0/3752619212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0860109030 msgr2=0x7f08601996e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:39:55.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f085ffff640 1 --2- 172.21.15.114:0/3752619212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0860109030 0x7f08601996e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f085ffff640 1 -- 172.21.15.114:0/3752619212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0860199e10 con 0x7f08601032e0 2024-05-27T07:39:55.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7efd7e6c5700 1 -- 172.21.15.114:0/947314447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd781034b0 msgr2=0x7efd781038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7efd7e6c5700 1 --2- 172.21.15.114:0/947314447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd781034b0 0x7efd781038c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7efd680099c0 tx=0x7efd68009cd0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7efd7e6c5700 1 -- 172.21.15.114:0/947314447 shutdown_connections 2024-05-27T07:39:55.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7efd7e6c5700 1 --2- 172.21.15.114:0/947314447 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd78108730 0x7efd78110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7efd7e6c5700 1 --2- 172.21.15.114:0/947314447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd781034b0 0x7efd781038c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7efd7e6c5700 1 --2- 172.21.15.114:0/947314447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd78107de0 0x7efd781081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7efd7e6c5700 1 -- 172.21.15.114:0/947314447 >> 172.21.15.114:0/947314447 conn(0x7efd780fedb0 msgr2=0x7efd781011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7efd7e6c5700 1 -- 172.21.15.114:0/947314447 shutdown_connections 2024-05-27T07:39:55.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7efd7e6c5700 1 -- 172.21.15.114:0/947314447 wait complete. 2024-05-27T07:39:55.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f085ffff640 1 --2- 172.21.15.114:0/3752619212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08601032e0 0x7f086019f560 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f085000d740 tx=0x7f085000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f085dffb640 1 -- 172.21.15.114:0/3752619212 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850017070 con 0x7f08601032e0 2024-05-27T07:39:55.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f085dffb640 1 -- 172.21.15.114:0/3752619212 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f085000de50 con 0x7f08601032e0 2024-05-27T07:39:55.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f0866934640 1 -- 172.21.15.114:0/3752619212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f086019a0a0 con 0x7f08601032e0 2024-05-27T07:39:55.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f085dffb640 1 -- 172.21.15.114:0/3752619212 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850004ce0 con 0x7f08601032e0 2024-05-27T07:39:55.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.478+0000 7f0866934640 1 -- 172.21.15.114:0/3752619212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08601a6300 con 0x7f08601032e0 2024-05-27T07:39:55.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.479+0000 7efd7e6c5700 1 Processor -- start 2024-05-27T07:39:55.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.479+0000 7efd7e6c5700 1 -- start start 2024-05-27T07:39:55.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.479+0000 7efd7e6c5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd781034b0 0x7efd78199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.479+0000 7efd7e6c5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd78107de0 0x7efd78199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.479+0000 7efd7e6c5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd78108730 0x7efd7819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.479+0000 7efd7e6c5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd78113400 con 0x7efd78108730 2024-05-27T07:39:55.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.479+0000 7efd7e6c5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd78113280 con 0x7efd781034b0 2024-05-27T07:39:55.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.479+0000 7efd7e6c5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efd78113580 con 0x7efd78107de0 2024-05-27T07:39:55.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.479+0000 7f0866934640 1 -- 172.21.15.114:0/3752619212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0860103e70 con 0x7f08601032e0 2024-05-27T07:39:55.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.479+0000 7efd777fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd78107de0 0x7efd78199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd77fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd781034b0 0x7efd78199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd777fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd78107de0 0x7efd78199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:37050/0 (socket says 172.21.15.114:37050) 2024-05-27T07:39:55.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd777fe700 1 -- 172.21.15.114:0/2935546202 learned_addr learned my addr 172.21.15.114:0/2935546202 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:55.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd777fe700 1 -- 172.21.15.114:0/2935546202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd781034b0 msgr2=0x7efd78199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd777fe700 1 --2- 172.21.15.114:0/2935546202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd781034b0 0x7efd78199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd777fe700 1 -- 172.21.15.114:0/2935546202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd78108730 msgr2=0x7efd7819a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd777fe700 1 --2- 172.21.15.114:0/2935546202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd78108730 0x7efd7819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd777fe700 1 -- 172.21.15.114:0/2935546202 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd781a91f0 con 0x7efd78107de0 2024-05-27T07:39:55.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd777fe700 1 --2- 172.21.15.114:0/2935546202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd78107de0 0x7efd78199fb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00c8f0 tx=0x7efd6c00cc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd757fa700 1 -- 172.21.15.114:0/2935546202 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c007940 con 0x7efd78107de0 2024-05-27T07:39:55.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd7e6c5700 1 -- 172.21.15.114:0/2935546202 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd781a94e0 con 0x7efd78107de0 2024-05-27T07:39:55.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd7e6c5700 1 -- 172.21.15.114:0/2935546202 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd781a9a70 con 0x7efd78107de0 2024-05-27T07:39:55.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd757fa700 1 -- 172.21.15.114:0/2935546202 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd6c00f450 con 0x7efd78107de0 2024-05-27T07:39:55.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.480+0000 7efd757fa700 1 -- 172.21.15.114:0/2935546202 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c018660 con 0x7efd78107de0 2024-05-27T07:39:55.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.482+0000 7f085dffb640 1 -- 172.21.15.114:0/3752619212 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 23) ==== 149829+0+0 (secure 0 0 0) 0x7f0850004e80 con 0x7f08601032e0 2024-05-27T07:39:55.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.483+0000 7f085dffb640 1 --2- 172.21.15.114:0/3752619212 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f084c0b2a60 0x7f084c0b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.483+0000 7f0857fff640 1 --2- 172.21.15.114:0/3752619212 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f084c0b2a60 0x7f084c0b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.483+0000 7f085dffb640 1 -- 172.21.15.114:0/3752619212 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f08500e2830 con 0x7f08601032e0 2024-05-27T07:39:55.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.483+0000 7f0857fff640 1 --2- 172.21.15.114:0/3752619212 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f084c0b2a60 0x7f084c0b4f70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0848009230 tx=0x7f0848031400 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.484+0000 7efd8d0dab80 1 Processor -- start 2024-05-27T07:39:55.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.484+0000 7efd8d0dab80 1 -- start start 2024-05-27T07:39:55.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.484+0000 7efd757fa700 1 -- 172.21.15.114:0/2935546202 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 23) ==== 149829+0+0 (secure 0 0 0) 0x7efd6c018800 con 0x7efd78107de0 2024-05-27T07:39:55.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.484+0000 7efd8d0dab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa588a9ec0 0x55fa58a2d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.484+0000 7efd8d0dab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a2dcd0 0x55fa58a27ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd8d0dab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a28000 0x55fa58a28410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd8d0dab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fa58956d10 con 0x55fa58a2dcd0 2024-05-27T07:39:55.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd8d0dab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fa589468b0 con 0x55fa588a9ec0 2024-05-27T07:39:55.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd8d0dab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fa5894a8f0 con 0x55fa58a28000 2024-05-27T07:39:55.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd77fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a2dcd0 0x55fa58a27ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd77fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a2dcd0 0x55fa58a27ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36918/0 (socket says 172.21.15.114:36918) 2024-05-27T07:39:55.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd77fff700 1 -- 172.21.15.114:0/3625443126 learned_addr learned my addr 172.21.15.114:0/3625443126 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:55.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd7cc62700 1 --2- 172.21.15.114:0/3625443126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa588a9ec0 0x55fa58a2d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd77fff700 1 -- 172.21.15.114:0/3625443126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a28000 msgr2=0x55fa58a28410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd77fff700 1 --2- 172.21.15.114:0/3625443126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a28000 0x55fa58a28410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd77fff700 1 -- 172.21.15.114:0/3625443126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa588a9ec0 msgr2=0x55fa58a2d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd77fff700 1 --2- 172.21.15.114:0/3625443126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa588a9ec0 0x55fa58a2d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd77fff700 1 -- 172.21.15.114:0/3625443126 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa58896950 con 0x55fa58a2dcd0 2024-05-27T07:39:55.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd77fff700 1 --2- 172.21.15.114:0/3625443126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a2dcd0 0x55fa58a27ac0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7efd680099c0 tx=0x7efd680096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401788bc91c74454 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd757fa700 1 --2- 172.21.15.114:0/2935546202 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efd600b2a80 0x7efd600b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.485+0000 7efd7cc62700 1 --2- 172.21.15.114:0/2935546202 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efd600b2a80 0x7efd600b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.486+0000 7efd757fa700 1 -- 172.21.15.114:0/2935546202 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7efd6c0e2980 con 0x7efd78107de0 2024-05-27T07:39:55.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.486+0000 7efd7cc62700 1 --2- 172.21.15.114:0/2935546202 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efd600b2a80 0x7efd600b4f90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x55fa58a28c30 tx=0x7efd70009450 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.488+0000 7efd5e7fc700 1 -- 172.21.15.114:0/3625443126 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68003d50 con 0x55fa58a2dcd0 2024-05-27T07:39:55.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.488+0000 7f085dffb640 1 -- 172.21.15.114:0/3752619212 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f08500a9d70 con 0x7f08601032e0 2024-05-27T07:39:55.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.488+0000 7efd5e7fc700 1 -- 172.21.15.114:0/3625443126 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd68005580 con 0x55fa58a2dcd0 2024-05-27T07:39:55.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.488+0000 7efd5e7fc700 1 -- 172.21.15.114:0/3625443126 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6800f460 con 0x55fa58a2dcd0 2024-05-27T07:39:55.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.489+0000 7efd8d0dab80 1 -- 172.21.15.114:0/3625443126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a2dcd0 msgr2=0x55fa58a27ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.489+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/3625443126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a2dcd0 0x55fa58a27ac0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7efd680099c0 tx=0x7efd680096c0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.489+0000 7efd8d0dab80 1 -- 172.21.15.114:0/3625443126 shutdown_connections 2024-05-27T07:39:55.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.489+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/3625443126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a28000 0x55fa58a28410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.489+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/3625443126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa588a9ec0 0x55fa58a2d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.489+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/3625443126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a2dcd0 0x55fa58a27ac0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.489+0000 7efd8d0dab80 1 -- 172.21.15.114:0/3625443126 >> 172.21.15.114:0/3625443126 conn(0x55fa58904b70 msgr2=0x55fa58a226c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.489+0000 7efd8d0dab80 1 -- 172.21.15.114:0/3625443126 shutdown_connections 2024-05-27T07:39:55.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.489+0000 7efd8d0dab80 1 -- 172.21.15.114:0/3625443126 wait complete. 2024-05-27T07:39:55.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.489+0000 7efd8d0dab80 1 -- start start 2024-05-27T07:39:55.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.489+0000 7efd8d0dab80 1 Processor -- start 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd8d0dab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa588a9ec0 0x55fa58a31320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd8d0dab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a28000 0x55fa58a31860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd8d0dab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a2dcd0 0x55fa58a31da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd8d0dab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fa5884d020 con 0x55fa58a28000 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd8d0dab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fa588e8840 con 0x55fa588a9ec0 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd8d0dab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fa58a34700 con 0x55fa58a2dcd0 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd8d0dab80 10 client.0 _ll_register_callbacks cb 0x55fa588e19a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd777fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a28000 0x55fa58a31860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd777fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a28000 0x55fa58a31860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36924/0 (socket says 172.21.15.114:36924) 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd777fe700 1 -- 172.21.15.114:0/593040210 learned_addr learned my addr 172.21.15.114:0/593040210 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd77fff700 1 --2- 172.21.15.114:0/593040210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa588a9ec0 0x55fa58a31320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd7cc62700 1 --2- 172.21.15.114:0/593040210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a2dcd0 0x55fa58a31da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd777fe700 1 -- 172.21.15.114:0/593040210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a2dcd0 msgr2=0x55fa58a31da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd777fe700 1 --2- 172.21.15.114:0/593040210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a2dcd0 0x55fa58a31da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd777fe700 1 -- 172.21.15.114:0/593040210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa588a9ec0 msgr2=0x55fa58a31320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd777fe700 1 --2- 172.21.15.114:0/593040210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa588a9ec0 0x55fa58a31320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.490+0000 7efd777fe700 1 -- 172.21.15.114:0/593040210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa588e5710 con 0x55fa58a28000 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.491+0000 7efd777fe700 1 --2- 172.21.15.114:0/593040210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a28000 0x55fa58a31860 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0e0320 tx=0x7efd6c0e0210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21394935eb3e586 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.491+0000 7efd577fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.491+0000 7efd577fe700 1 -- 172.21.15.114:0/593040210 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c0a3e60 con 0x55fa58a28000 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.491+0000 7efd8d0dab80 1 -- 172.21.15.114:0/593040210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd68005580 con 0x55fa58a28000 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.491+0000 7efd577fe700 1 -- 172.21.15.114:0/593040210 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7efd6c0b2110 con 0x55fa58a28000 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.491+0000 7efd577fe700 1 -- 172.21.15.114:0/593040210 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7efd6c0a9e40 con 0x55fa58a28000 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.491+0000 7efd8d0dab80 10 client.15297 fetch_fsmap learned FSMap version 21 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.491+0000 7efd8d0dab80 1 -- 172.21.15.114:0/593040210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fa58a3f200 con 0x55fa58a28000 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.491+0000 7efd577fe700 1 -- 172.21.15.114:0/593040210 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd68005580 con 0x55fa58a28000 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.492+0000 7efd577fe700 1 -- 172.21.15.114:0/593040210 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7efd6c0beb90 con 0x55fa58a28000 2024-05-27T07:39:55.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.492+0000 7efd8d0dab80 10 client.15297 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:39:55.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.492+0000 7efd8d0dab80 10 client.15297 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:39:55.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.492+0000 7efd8d0dab80 20 client.15297 populate_metadata read hostname 'smithi114' 2024-05-27T07:39:55.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.492+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/593040210 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fa58a41840 0x55fa58a43c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.492+0000 7efd8d0dab80 4 client.15297 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:55.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.492+0000 7efd8d0dab80 1 -- 172.21.15.114:0/593040210 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa589f3a70 con 0x55fa58a41840 2024-05-27T07:39:55.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.492+0000 7efd77fff700 1 --2- 172.21.15.114:0/593040210 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fa58a41840 0x55fa58a43c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.493+0000 7efd577fe700 10 client.15297 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:55.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.493+0000 7efd77fff700 1 --2- 172.21.15.114:0/593040210 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fa58a41840 0x55fa58a43c60 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3554a262b990539f server_cookie=58e68462a8624f68 in_seq=0 out_seq=0 2024-05-27T07:39:55.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd577fe700 1 -- 172.21.15.114:0/593040210 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fa589f3a70 con 0x55fa58a41840 2024-05-27T07:39:55.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd577fe700 10 client.15297 handle_command_reply: tid=0 2024-05-27T07:39:55.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd8d0dab80 1 client.15297 shutdown 2024-05-27T07:39:55.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd8d0dab80 1 -- 172.21.15.114:0/593040210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a28000 msgr2=0x55fa58a31860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/593040210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a28000 0x55fa58a31860 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0e0320 tx=0x7efd6c0e0210 comp rx=0 tx=0).stop 2024-05-27T07:39:55.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd8d0dab80 1 -- 172.21.15.114:0/593040210 shutdown_connections 2024-05-27T07:39:55.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/593040210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a2dcd0 0x55fa58a31da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/593040210 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fa58a41840 0x55fa58a43c60 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/593040210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa588a9ec0 0x55fa58a31320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/593040210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa58a28000 0x55fa58a31860 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd8d0dab80 1 -- 172.21.15.114:0/593040210 >> 172.21.15.114:0/593040210 conn(0x55fa58904b70 msgr2=0x55fa58a26ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd577fe700 0 client.15297 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:55.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd8d0dab80 1 -- 172.21.15.114:0/593040210 shutdown_connections 2024-05-27T07:39:55.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd8d0dab80 1 -- 172.21.15.114:0/593040210 wait complete. 2024-05-27T07:39:55.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.494+0000 7efd8d0dab80 20 client.15297 trim_cache size 0 max 16384 2024-05-27T07:39:55.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.514+0000 7efd8d0dab80 1 Processor -- start 2024-05-27T07:39:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.514+0000 7efd8d0dab80 1 -- start start 2024-05-27T07:39:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd8d0dab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa58904b70 0x55fa58a27ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd8d0dab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 0x55fa58a31be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd8d0dab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a28000 0x55fa58a3d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd8d0dab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd6400cc60 con 0x55fa588a9ec0 2024-05-27T07:39:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd8d0dab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd6400f430 con 0x55fa58904b70 2024-05-27T07:39:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd8d0dab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efd6400d220 con 0x55fa58a28000 2024-05-27T07:39:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd777fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 0x55fa58a31be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd777fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 0x55fa58a31be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36934/0 (socket says 172.21.15.114:36934) 2024-05-27T07:39:55.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd777fe700 1 -- 172.21.15.114:0/2568508226 learned_addr learned my addr 172.21.15.114:0/2568508226 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:55.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd777fe700 1 -- 172.21.15.114:0/2568508226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a28000 msgr2=0x55fa58a3d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd777fe700 1 --2- 172.21.15.114:0/2568508226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a28000 0x55fa58a3d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd77fff700 1 --2- 172.21.15.114:0/2568508226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa58904b70 0x55fa58a27ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd777fe700 1 -- 172.21.15.114:0/2568508226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa58904b70 msgr2=0x55fa58a27ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd777fe700 1 --2- 172.21.15.114:0/2568508226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa58904b70 0x55fa58a27ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd777fe700 1 -- 172.21.15.114:0/2568508226 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa58a23f50 con 0x55fa588a9ec0 2024-05-27T07:39:55.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.515+0000 7efd777fe700 1 --2- 172.21.15.114:0/2568508226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 0x55fa58a31be0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0a9440 tx=0x7efd6c0a8eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808afc50474c2e85 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.516+0000 7efd5dffb700 1 -- 172.21.15.114:0/2568508226 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c0a3e60 con 0x55fa588a9ec0 2024-05-27T07:39:55.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.516+0000 7efd5dffb700 1 -- 172.21.15.114:0/2568508226 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd6c0912c0 con 0x55fa588a9ec0 2024-05-27T07:39:55.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.516+0000 7efd5dffb700 1 -- 172.21.15.114:0/2568508226 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c0adaf0 con 0x55fa588a9ec0 2024-05-27T07:39:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.516+0000 7efd8d0dab80 1 -- 172.21.15.114:0/2568508226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 msgr2=0x55fa58a31be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.516+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/2568508226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 0x55fa58a31be0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0a9440 tx=0x7efd6c0a8eb0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.517+0000 7efd8d0dab80 1 -- 172.21.15.114:0/2568508226 shutdown_connections 2024-05-27T07:39:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.517+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/2568508226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a28000 0x55fa58a3d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.517+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/2568508226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa58904b70 0x55fa58a27ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.517+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/2568508226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 0x55fa58a31be0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.517+0000 7efd8d0dab80 1 -- 172.21.15.114:0/2568508226 >> 172.21.15.114:0/2568508226 conn(0x55fa58a1ce30 msgr2=0x55fa58a2ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.517+0000 7efd8d0dab80 1 -- 172.21.15.114:0/2568508226 shutdown_connections 2024-05-27T07:39:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.517+0000 7efd8d0dab80 1 -- 172.21.15.114:0/2568508226 wait complete. 2024-05-27T07:39:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.517+0000 7efd8d0dab80 1 -- start start 2024-05-27T07:39:55.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.517+0000 7efd8d0dab80 1 Processor -- start 2024-05-27T07:39:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd8d0dab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 0x55fa58a3a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd8d0dab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa58904b70 0x55fa58a369f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd8d0dab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a28000 0x55fa58a36f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd8d0dab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd6400d8f0 con 0x55fa588a9ec0 2024-05-27T07:39:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd8d0dab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd6400ed60 con 0x55fa58904b70 2024-05-27T07:39:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd8d0dab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efd6400dfc0 con 0x55fa58a28000 2024-05-27T07:39:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd8d0dab80 10 client.0 _ll_register_callbacks cb 0x55fa588e19a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd77fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 0x55fa58a3a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd77fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 0x55fa58a3a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36936/0 (socket says 172.21.15.114:36936) 2024-05-27T07:39:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd77fff700 1 -- 172.21.15.114:0/1336767007 learned_addr learned my addr 172.21.15.114:0/1336767007 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd7cc62700 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a28000 0x55fa58a36f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd777fe700 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa58904b70 0x55fa58a369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd77fff700 1 -- 172.21.15.114:0/1336767007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a28000 msgr2=0x55fa58a36f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd77fff700 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a28000 0x55fa58a36f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd77fff700 1 -- 172.21.15.114:0/1336767007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa58904b70 msgr2=0x55fa58a369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd77fff700 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa58904b70 0x55fa58a369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd77fff700 1 -- 172.21.15.114:0/1336767007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa58a376f0 con 0x55fa588a9ec0 2024-05-27T07:39:55.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd777fe700 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa58904b70 0x55fa58a369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:55.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd77fff700 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 0x55fa58a3a160 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efd68009750 tx=0x7efd68009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466d64c2d62ff68f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd577fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:39:55.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd577fe700 1 -- 172.21.15.114:0/1336767007 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6800bd50 con 0x55fa588a9ec0 2024-05-27T07:39:55.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd8d0dab80 1 -- 172.21.15.114:0/1336767007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd6c0912c0 con 0x55fa588a9ec0 2024-05-27T07:39:55.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.518+0000 7efd577fe700 1 -- 172.21.15.114:0/1336767007 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7efd680050d0 con 0x55fa588a9ec0 2024-05-27T07:39:55.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.519+0000 7efd577fe700 1 -- 172.21.15.114:0/1336767007 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7efd68043050 con 0x55fa588a9ec0 2024-05-27T07:39:55.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.519+0000 7efd8d0dab80 10 client.15303 fetch_fsmap learned FSMap version 21 2024-05-27T07:39:55.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.519+0000 7efd8d0dab80 1 -- 172.21.15.114:0/1336767007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fa58aad1b0 con 0x55fa588a9ec0 2024-05-27T07:39:55.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.519+0000 7efd577fe700 1 -- 172.21.15.114:0/1336767007 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd6c0912c0 con 0x55fa588a9ec0 2024-05-27T07:39:55.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.519+0000 7efd577fe700 1 -- 172.21.15.114:0/1336767007 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7efd6800f990 con 0x55fa588a9ec0 2024-05-27T07:39:55.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.519+0000 7efd8d0dab80 10 client.15303 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:39:55.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.519+0000 7efd8d0dab80 10 client.15303 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:39:55.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.519+0000 7efd8d0dab80 20 client.15303 populate_metadata read hostname 'smithi114' 2024-05-27T07:39:55.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.519+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fa58a2dcd0 0x55fa58aaf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.519+0000 7efd8d0dab80 4 client.15303 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:39:55.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.519+0000 7efd8d0dab80 1 -- 172.21.15.114:0/1336767007 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fa58958c60 con 0x55fa58a2dcd0 2024-05-27T07:39:55.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.519+0000 7efd777fe700 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fa58a2dcd0 0x55fa58aaf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.520+0000 7efd577fe700 10 client.15303 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:55.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.520+0000 7efd777fe700 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fa58a2dcd0 0x55fa58aaf3f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df6b166ae170b197 server_cookie=c3b98a80858523e6 in_seq=0 out_seq=0 2024-05-27T07:39:55.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd577fe700 1 -- 172.21.15.114:0/1336767007 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fa58958c60 con 0x55fa58a2dcd0 2024-05-27T07:39:55.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd577fe700 10 client.15303 handle_command_reply: tid=0 2024-05-27T07:39:55.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd8d0dab80 1 client.15303 shutdown 2024-05-27T07:39:55.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd8d0dab80 1 -- 172.21.15.114:0/1336767007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 msgr2=0x55fa58a3a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 0x55fa58a3a160 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efd68009750 tx=0x7efd68009cd0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd8d0dab80 1 -- 172.21.15.114:0/1336767007 shutdown_connections 2024-05-27T07:39:55.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fa58a28000 0x55fa58a36f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fa588a9ec0 0x55fa58a3a160 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fa58a2dcd0 0x55fa58aaf3f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd8d0dab80 1 --2- 172.21.15.114:0/1336767007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fa58904b70 0x55fa58a369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd577fe700 0 client.15303 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:55.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd8d0dab80 1 -- 172.21.15.114:0/1336767007 >> 172.21.15.114:0/1336767007 conn(0x55fa58a1ce30 msgr2=0x55fa58a2e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd8d0dab80 1 -- 172.21.15.114:0/1336767007 shutdown_connections 2024-05-27T07:39:55.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.521+0000 7efd8d0dab80 1 -- 172.21.15.114:0/1336767007 wait complete. 2024-05-27T07:39:55.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.522+0000 7efd8d0dab80 20 client.15303 trim_cache size 0 max 16384 2024-05-27T07:39:55.522 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:39:55.522 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:39:55.522 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "66995498-c2e5-473f-93cd-3052727f3ace", 2024-05-27T07:39:55.522 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:39:55.523 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:39:55.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.522+0000 7efd7e6c5700 1 -- 172.21.15.114:0/2935546202 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efd600b2a80 msgr2=0x7efd600b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.522+0000 7efd7e6c5700 1 --2- 172.21.15.114:0/2935546202 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efd600b2a80 0x7efd600b4f90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x55fa58a28c30 tx=0x7efd70009450 comp rx=0 tx=0).stop 2024-05-27T07:39:55.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.522+0000 7efd7e6c5700 1 -- 172.21.15.114:0/2935546202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd78107de0 msgr2=0x7efd78199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.522+0000 7efd7e6c5700 1 --2- 172.21.15.114:0/2935546202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd78107de0 0x7efd78199fb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00c8f0 tx=0x7efd6c00cc00 comp rx=0 tx=0).stop 2024-05-27T07:39:55.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.522+0000 7efd7e6c5700 1 -- 172.21.15.114:0/2935546202 shutdown_connections 2024-05-27T07:39:55.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.522+0000 7efd7e6c5700 1 --2- 172.21.15.114:0/2935546202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd78108730 0x7efd7819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.522+0000 7efd7e6c5700 1 --2- 172.21.15.114:0/2935546202 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efd600b2a80 0x7efd600b4f90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.522+0000 7efd7e6c5700 1 --2- 172.21.15.114:0/2935546202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd781034b0 0x7efd78199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.522+0000 7efd7e6c5700 1 --2- 172.21.15.114:0/2935546202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd78107de0 0x7efd78199fb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.522+0000 7efd7e6c5700 1 -- 172.21.15.114:0/2935546202 >> 172.21.15.114:0/2935546202 conn(0x7efd780fedb0 msgr2=0x7efd78101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.522+0000 7efd7e6c5700 1 -- 172.21.15.114:0/2935546202 shutdown_connections 2024-05-27T07:39:55.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.522+0000 7efd7e6c5700 1 -- 172.21.15.114:0/2935546202 wait complete. 2024-05-27T07:39:55.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.701+0000 7f0866934640 1 -- 172.21.15.114:0/3752619212 --> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f08601019c0 con 0x7f084c0b2a60 2024-05-27T07:39:55.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.709+0000 7f085dffb640 1 -- 172.21.15.114:0/3752619212 <== mgr.15138 v2:172.21.15.114:6800/3311646739 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f08601019c0 con 0x7f084c0b2a60 2024-05-27T07:39:55.706 INFO:teuthology.orchestra.run.smithi114.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-27T00:00:00", "created": "2024-05-27T07:39:54", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-05-27T07:39:55.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.711+0000 7f0866934640 1 -- 172.21.15.114:0/3752619212 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f084c0b2a60 msgr2=0x7f084c0b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.712+0000 7f0866934640 1 --2- 172.21.15.114:0/3752619212 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f084c0b2a60 0x7f084c0b4f70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0848009230 tx=0x7f0848031400 comp rx=0 tx=0).stop 2024-05-27T07:39:55.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.712+0000 7f0866934640 1 -- 172.21.15.114:0/3752619212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08601032e0 msgr2=0x7f086019f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.712+0000 7f0866934640 1 --2- 172.21.15.114:0/3752619212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08601032e0 0x7f086019f560 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f085000d740 tx=0x7f085000dc00 comp rx=0 tx=0).stop 2024-05-27T07:39:55.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.712+0000 7f0866934640 1 -- 172.21.15.114:0/3752619212 shutdown_connections 2024-05-27T07:39:55.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.712+0000 7f0866934640 1 --2- 172.21.15.114:0/3752619212 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f084c0b2a60 0x7f084c0b4f70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.712+0000 7f0866934640 1 --2- 172.21.15.114:0/3752619212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0860109030 0x7f08601996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.712+0000 7f0866934640 1 --2- 172.21.15.114:0/3752619212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860108b90 0x7f086019faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.712+0000 7f0866934640 1 --2- 172.21.15.114:0/3752619212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08601032e0 0x7f086019f560 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.712+0000 7f0866934640 1 -- 172.21.15.114:0/3752619212 >> 172.21.15.114:0/3752619212 conn(0x7f08600782d0 msgr2=0x7f08601012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.712+0000 7f0866934640 1 -- 172.21.15.114:0/3752619212 shutdown_connections 2024-05-27T07:39:55.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.712+0000 7f0866934640 1 -- 172.21.15.114:0/3752619212 wait complete. 2024-05-27T07:39:55.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-27T07:39:55.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.838+0000 7fd838d8d640 1 -- 172.21.15.114:0/1027868383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834103a90 msgr2=0x7fd83410f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.838+0000 7fd838d8d640 1 --2- 172.21.15.114:0/1027868383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834103a90 0x7fd83410f6d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009ca0 tx=0x7fd82c02a950 comp rx=0 tx=0).stop 2024-05-27T07:39:55.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.838+0000 7fd838d8d640 1 -- 172.21.15.114:0/1027868383 shutdown_connections 2024-05-27T07:39:55.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.838+0000 7fd838d8d640 1 --2- 172.21.15.114:0/1027868383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834103a90 0x7fd83410f6d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.838+0000 7fd838d8d640 1 --2- 172.21.15.114:0/1027868383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd834108ff0 0x7fd834103400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.838+0000 7fd838d8d640 1 --2- 172.21.15.114:0/1027868383 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd834108be0 0x7fd834102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.838+0000 7fd838d8d640 1 -- 172.21.15.114:0/1027868383 >> 172.21.15.114:0/1027868383 conn(0x7fd8340fd9c0 msgr2=0x7fd8340ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.839+0000 7fd838d8d640 1 -- 172.21.15.114:0/1027868383 shutdown_connections 2024-05-27T07:39:55.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.839+0000 7fd838d8d640 1 -- 172.21.15.114:0/1027868383 wait complete. 2024-05-27T07:39:55.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.839+0000 7fd838d8d640 1 Processor -- start 2024-05-27T07:39:55.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.839+0000 7fd838d8d640 1 -- start start 2024-05-27T07:39:55.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd838d8d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd834108be0 0x7fd834199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd838d8d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834108ff0 0x7fd8341998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd838d8d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd83419dba0 0x7fd83419dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd838d8d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd834112dc0 con 0x7fd834108ff0 2024-05-27T07:39:55.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd838d8d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd834112c40 con 0x7fd834108be0 2024-05-27T07:39:55.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd838d8d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd834112f40 con 0x7fd83419dba0 2024-05-27T07:39:55.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd829d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834108ff0 0x7fd8341998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd829d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834108ff0 0x7fd8341998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36948/0 (socket says 172.21.15.114:36948) 2024-05-27T07:39:55.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd829d9b640 1 -- 172.21.15.114:0/4054306602 learned_addr learned my addr 172.21.15.114:0/4054306602 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:55.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd832d9d640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd83419dba0 0x7fd83419dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd829d9b640 1 -- 172.21.15.114:0/4054306602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd83419dba0 msgr2=0x7fd83419dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd83259c640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd834108be0 0x7fd834199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd829d9b640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd83419dba0 0x7fd83419dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd829d9b640 1 -- 172.21.15.114:0/4054306602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd834108be0 msgr2=0x7fd834199380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd829d9b640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd834108be0 0x7fd834199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd829d9b640 1 -- 172.21.15.114:0/4054306602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd83419e5f0 con 0x7fd834108ff0 2024-05-27T07:39:55.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd832d9d640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd83419dba0 0x7fd83419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:55.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd829d9b640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834108ff0 0x7fd8341998e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00d740 tx=0x7fd81c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.840+0000 7fd82bfff640 1 -- 172.21.15.114:0/4054306602 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c017070 con 0x7fd834108ff0 2024-05-27T07:39:55.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.841+0000 7fd838d8d640 1 -- 172.21.15.114:0/4054306602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd83419e8e0 con 0x7fd834108ff0 2024-05-27T07:39:55.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.841+0000 7fd82bfff640 1 -- 172.21.15.114:0/4054306602 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd81c00de50 con 0x7fd834108ff0 2024-05-27T07:39:55.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.841+0000 7fd82bfff640 1 -- 172.21.15.114:0/4054306602 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c004ce0 con 0x7fd834108ff0 2024-05-27T07:39:55.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.841+0000 7fd838d8d640 1 -- 172.21.15.114:0/4054306602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8341a60e0 con 0x7fd834108ff0 2024-05-27T07:39:55.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.842+0000 7fd838d8d640 1 -- 172.21.15.114:0/4054306602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd834050860 con 0x7fd834108ff0 2024-05-27T07:39:55.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.843+0000 7fd82bfff640 1 -- 172.21.15.114:0/4054306602 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 23) ==== 149829+0+0 (secure 0 0 0) 0x7fd81c004e80 con 0x7fd834108ff0 2024-05-27T07:39:55.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.844+0000 7fd82bfff640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd8200b29e0 0x7fd8200b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.844+0000 7fd83259c640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd8200b29e0 0x7fd8200b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.844+0000 7fd82bfff640 1 -- 172.21.15.114:0/4054306602 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fd81c0e24c0 con 0x7fd834108ff0 2024-05-27T07:39:55.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.845+0000 7fd83259c640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd8200b29e0 0x7fd8200b4ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd8240058c0 tx=0x7fd824005c60 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:55.849+0000 7fd82bfff640 1 -- 172.21.15.114:0/4054306602 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd81c0a9a20 con 0x7fd834108ff0 2024-05-27T07:39:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.059+0000 7fd838d8d640 1 -- 172.21.15.114:0/4054306602 --> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fd8341001f0 con 0x7fd8200b29e0 2024-05-27T07:39:56.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.066+0000 7fd82bfff640 1 -- 172.21.15.114:0/4054306602 <== mgr.15138 v2:172.21.15.114:6800/3311646739 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fd8341001f0 con 0x7fd8200b29e0 2024-05-27T07:39:56.062 INFO:teuthology.orchestra.run.smithi114.stdout:/ 1m 6m3h 2024-05-27T07:39:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.068+0000 7fd838d8d640 1 -- 172.21.15.114:0/4054306602 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd8200b29e0 msgr2=0x7fd8200b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.068+0000 7fd838d8d640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd8200b29e0 0x7fd8200b4ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd8240058c0 tx=0x7fd824005c60 comp rx=0 tx=0).stop 2024-05-27T07:39:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.068+0000 7fd838d8d640 1 -- 172.21.15.114:0/4054306602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834108ff0 msgr2=0x7fd8341998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.068+0000 7fd838d8d640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834108ff0 0x7fd8341998e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00d740 tx=0x7fd81c00dc00 comp rx=0 tx=0).stop 2024-05-27T07:39:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.068+0000 7fd838d8d640 1 -- 172.21.15.114:0/4054306602 shutdown_connections 2024-05-27T07:39:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.068+0000 7fd838d8d640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd8200b29e0 0x7fd8200b4ef0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.068+0000 7fd838d8d640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd83419dba0 0x7fd83419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.068+0000 7fd838d8d640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834108ff0 0x7fd8341998e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:56.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.068+0000 7fd838d8d640 1 --2- 172.21.15.114:0/4054306602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd834108be0 0x7fd834199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:56.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.068+0000 7fd838d8d640 1 -- 172.21.15.114:0/4054306602 >> 172.21.15.114:0/4054306602 conn(0x7fd8340fd9c0 msgr2=0x7fd8340ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:56.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.068+0000 7fd838d8d640 1 -- 172.21.15.114:0/4054306602 shutdown_connections 2024-05-27T07:39:56.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:56.068+0000 7fd838d8d640 1 -- 172.21.15.114:0/4054306602 wait complete. 2024-05-27T07:39:56.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-05-27T07:39:56.102 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-05-27T07:39:56.113 INFO:teuthology.run_tasks:Running task background_exec... 2024-05-27T07:39:56.141 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-05-27T07:39:56.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-05-27T07:39:56.145 INFO:teuthology.run_tasks:Running task workunit... 2024-05-27T07:39:56.157 INFO:tasks.workunit:Pulling workunits from ref 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:39:56.158 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-27T07:39:56.158 INFO:tasks.workunit:timeout=3h 2024-05-27T07:39:56.158 INFO:tasks.workunit:cleanup=True 2024-05-27T07:39:56.158 DEBUG:teuthology.orchestra.run.smithi114:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-27T07:39:56.173 INFO:teuthology.orchestra.run.smithi114.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-27T07:39:56.174 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-27T07:39:56.174 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-05-27T07:39:56.174 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:39:56.174 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-27T07:39:56.174 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-05-27 07:37:18.703823950 +0000 2024-05-27T07:39:56.174 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-05-27 07:37:18.703823950 +0000 2024-05-27T07:39:56.174 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-05-27 07:39:40.776467689 +0000 2024-05-27T07:39:56.174 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-05-27 07:37:18.703823950 +0000 2024-05-27T07:39:56.174 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-27T07:39:56.174 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-27T07:39:56.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:39:56.245 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:39:56.301 INFO:tasks.workunit.client.0.smithi114.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-27T07:39:56.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:56 smithi163 ceph-mon[36451]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:56.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:56 smithi163 ceph-mon[36451]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:56.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:56 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:39:56.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:56 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:56.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:56 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:39:56.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:56 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:56.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:56 smithi186 ceph-mon[36647]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:56.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:56 smithi186 ceph-mon[36647]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:56.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:56 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:39:56.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:56 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:56.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:56 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:39:56.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:56 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:56.525 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:56.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:56 smithi114 ceph-mon[32038]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:56.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:56 smithi114 ceph-mon[32038]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:56.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:56 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:39:56.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:56 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:56.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:56 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:39:56.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:56 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:57.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:57 smithi114 ceph-mon[32038]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:57.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:57 smithi114 ceph-mon[32038]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:57.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:57 smithi114 ceph-mon[32038]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 59 KiB/s wr, 24 op/s 2024-05-27T07:39:57.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:57 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:39:57.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:57 smithi163 ceph-mon[36451]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:57.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:57 smithi163 ceph-mon[36451]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:57.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:57 smithi163 ceph-mon[36451]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 59 KiB/s wr, 24 op/s 2024-05-27T07:39:57.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:57 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:39:57.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:57 smithi186 ceph-mon[36647]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:57.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:57 smithi186 ceph-mon[36647]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:39:57.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:57 smithi186 ceph-mon[36647]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 59 KiB/s wr, 24 op/s 2024-05-27T07:39:57.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:57 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:39:58.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.188+0000 7f0f379cc700 1 -- 172.21.15.114:0/771877452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f30109850 msgr2=0x7f0f301111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:58.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.188+0000 7f0f379cc700 1 --2- 172.21.15.114:0/771877452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f30109850 0x7f0f301111b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800b0d0 tx=0x7f0f2800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.188+0000 7f0f379cc700 1 -- 172.21.15.114:0/771877452 shutdown_connections 2024-05-27T07:39:58.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.188+0000 7f0f379cc700 1 --2- 172.21.15.114:0/771877452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f30109850 0x7f0f301111b0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.188+0000 7f0f379cc700 1 --2- 172.21.15.114:0/771877452 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f301045d0 0x7f0f301049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.188+0000 7f0f379cc700 1 --2- 172.21.15.114:0/771877452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f30108f00 0x7f0f30109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.188+0000 7f0f379cc700 1 -- 172.21.15.114:0/771877452 >> 172.21.15.114:0/771877452 conn(0x7f0f300ffec0 msgr2=0x7f0f30102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:58.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.188+0000 7f0f379cc700 1 -- 172.21.15.114:0/771877452 shutdown_connections 2024-05-27T07:39:58.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.188+0000 7f0f379cc700 1 -- 172.21.15.114:0/771877452 wait complete. 2024-05-27T07:39:58.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.189+0000 7f0f379cc700 1 Processor -- start 2024-05-27T07:39:58.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.189+0000 7f0f379cc700 1 -- start start 2024-05-27T07:39:58.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f379cc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f301045d0 0x7f0f3019ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f379cc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f30108f00 0x7f0f3019b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f379cc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f30109850 0x7f0f3019f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f379cc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301144a0 con 0x7f0f301045d0 2024-05-27T07:39:58.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f379cc700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0f30114320 con 0x7f0f30109850 2024-05-27T07:39:58.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f379cc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0f30114620 con 0x7f0f30108f00 2024-05-27T07:39:58.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f35768700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f301045d0 0x7f0f3019ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f35768700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f301045d0 0x7f0f3019ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36970/0 (socket says 172.21.15.114:36970) 2024-05-27T07:39:58.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f35768700 1 -- 172.21.15.114:0/2175601512 learned_addr learned my addr 172.21.15.114:0/2175601512 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:58.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f34f67700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f30108f00 0x7f0f3019b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f35f69700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f30109850 0x7f0f3019f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f34f67700 1 -- 172.21.15.114:0/2175601512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f30109850 msgr2=0x7f0f3019f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f34f67700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f30109850 0x7f0f3019f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f34f67700 1 -- 172.21.15.114:0/2175601512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f301045d0 msgr2=0x7f0f3019ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f34f67700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f301045d0 0x7f0f3019ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f34f67700 1 -- 172.21.15.114:0/2175601512 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f3019fc60 con 0x7f0f30108f00 2024-05-27T07:39:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f35768700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f301045d0 0x7f0f3019ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.190+0000 7f0f34f67700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f30108f00 0x7f0f3019b330 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000e9c0 tx=0x7f0f2000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.191+0000 7f0f267fc700 1 -- 172.21.15.114:0/2175601512 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f20004220 con 0x7f0f30108f00 2024-05-27T07:39:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.191+0000 7f0f379cc700 1 -- 172.21.15.114:0/2175601512 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f3019ff50 con 0x7f0f30108f00 2024-05-27T07:39:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.191+0000 7f0f267fc700 1 -- 172.21.15.114:0/2175601512 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f200043c0 con 0x7f0f30108f00 2024-05-27T07:39:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.191+0000 7f0f267fc700 1 -- 172.21.15.114:0/2175601512 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f20005020 con 0x7f0f30108f00 2024-05-27T07:39:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.191+0000 7f0f379cc700 1 -- 172.21.15.114:0/2175601512 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f301a7870 con 0x7f0f30108f00 2024-05-27T07:39:58.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.193+0000 7f0f267fc700 1 -- 172.21.15.114:0/2175601512 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 23) ==== 149829+0+0 (secure 0 0 0) 0x7f0f20004750 con 0x7f0f30108f00 2024-05-27T07:39:58.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.194+0000 7f0f267fc700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0f1c0b2a80 0x7f0f1c0b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.194+0000 7f0f35768700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0f1c0b2a80 0x7f0f1c0b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.194+0000 7f0f267fc700 1 -- 172.21.15.114:0/2175601512 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f0f200e2a10 con 0x7f0f30108f00 2024-05-27T07:39:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.194+0000 7f0f35768700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0f1c0b2a80 0x7f0f1c0b4f90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c011550 tx=0x7f0f2c009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.195+0000 7f0f463e1b80 1 Processor -- start 2024-05-27T07:39:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.195+0000 7f0f463e1b80 1 -- start start 2024-05-27T07:39:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.195+0000 7f0f463e1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8a41e40 0x55f0f8bc6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.195+0000 7f0f463e1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc7070 0x55f0f8bc0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.195+0000 7f0f463e1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8bc1120 0x55f0f8bc1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.195+0000 7f0f463e1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f0f8aeed70 con 0x55f0f8bc7070 2024-05-27T07:39:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.195+0000 7f0f463e1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f0f8ade910 con 0x55f0f8bc1120 2024-05-27T07:39:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.195+0000 7f0f463e1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f0f8ae2950 con 0x55f0f8a41e40 2024-05-27T07:39:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.195+0000 7f0f34f67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc7070 0x55f0f8bc0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.195+0000 7f0f34f67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc7070 0x55f0f8bc0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36984/0 (socket says 172.21.15.114:36984) 2024-05-27T07:39:58.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.195+0000 7f0f34f67700 1 -- 172.21.15.114:0/2472728411 learned_addr learned my addr 172.21.15.114:0/2472728411 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:58.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.195+0000 7f0f35f69700 1 --2- 172.21.15.114:0/2472728411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8bc1120 0x55f0f8bc1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.195+0000 7f0f35768700 1 --2- 172.21.15.114:0/2472728411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8a41e40 0x55f0f8bc6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.196+0000 7f0f34f67700 1 -- 172.21.15.114:0/2472728411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8a41e40 msgr2=0x55f0f8bc6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:58.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.196+0000 7f0f34f67700 1 --2- 172.21.15.114:0/2472728411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8a41e40 0x55f0f8bc6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.196+0000 7f0f34f67700 1 -- 172.21.15.114:0/2472728411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8bc1120 msgr2=0x55f0f8bc1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:58.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.196+0000 7f0f34f67700 1 --2- 172.21.15.114:0/2472728411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8bc1120 0x55f0f8bc1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.196+0000 7f0f34f67700 1 -- 172.21.15.114:0/2472728411 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0f8a87fb0 con 0x55f0f8bc7070 2024-05-27T07:39:58.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.196+0000 7f0f34f67700 1 --2- 172.21.15.114:0/2472728411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc7070 0x55f0f8bc0be0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000f010 tx=0x7f0f2000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b0729651a4883d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:58.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.196+0000 7f0f1b7fe700 1 -- 172.21.15.114:0/2472728411 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2001c070 con 0x55f0f8bc7070 2024-05-27T07:39:58.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.196+0000 7f0f1b7fe700 1 -- 172.21.15.114:0/2472728411 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f200a97d0 con 0x55f0f8bc7070 2024-05-27T07:39:58.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.196+0000 7f0f1b7fe700 1 -- 172.21.15.114:0/2472728411 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f200a9a30 con 0x55f0f8bc7070 2024-05-27T07:39:58.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.197+0000 7f0f463e1b80 1 -- 172.21.15.114:0/2472728411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc7070 msgr2=0x55f0f8bc0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:58.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.197+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/2472728411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc7070 0x55f0f8bc0be0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000f010 tx=0x7f0f2000c7e0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.197+0000 7f0f463e1b80 1 -- 172.21.15.114:0/2472728411 shutdown_connections 2024-05-27T07:39:58.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.197+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/2472728411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8bc1120 0x55f0f8bc1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.197+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/2472728411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8a41e40 0x55f0f8bc6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.197+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/2472728411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc7070 0x55f0f8bc0be0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.197+0000 7f0f463e1b80 1 -- 172.21.15.114:0/2472728411 >> 172.21.15.114:0/2472728411 conn(0x55f0f8a44770 msgr2=0x55f0f8bbbd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.197+0000 7f0f463e1b80 1 -- 172.21.15.114:0/2472728411 shutdown_connections 2024-05-27T07:39:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.197+0000 7f0f463e1b80 1 -- 172.21.15.114:0/2472728411 wait complete. 2024-05-27T07:39:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.197+0000 7f0f463e1b80 1 -- start start 2024-05-27T07:39:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f463e1b80 1 Processor -- start 2024-05-27T07:39:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f463e1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8a41e40 0x55f0f8bca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f463e1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 0x55f0f8bcae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f463e1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc7070 0x55f0f8bcb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f463e1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f0f89e4fa0 con 0x55f0f8bc1120 2024-05-27T07:39:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f463e1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f0f8a46da0 con 0x55f0f8a41e40 2024-05-27T07:39:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f463e1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f0f8bcdb60 con 0x55f0f8bc7070 2024-05-27T07:39:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f463e1b80 10 client.0 _ll_register_callbacks cb 0x55f0f8a755b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f34f67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 0x55f0f8bcae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f34f67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 0x55f0f8bcae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36992/0 (socket says 172.21.15.114:36992) 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f34f67700 1 -- 172.21.15.114:0/870225432 learned_addr learned my addr 172.21.15.114:0/870225432 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f35768700 1 --2- 172.21.15.114:0/870225432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8a41e40 0x55f0f8bca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f35f69700 1 --2- 172.21.15.114:0/870225432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc7070 0x55f0f8bcb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f34f67700 1 -- 172.21.15.114:0/870225432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc7070 msgr2=0x55f0f8bcb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f34f67700 1 --2- 172.21.15.114:0/870225432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc7070 0x55f0f8bcb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f34f67700 1 -- 172.21.15.114:0/870225432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8a41e40 msgr2=0x55f0f8bca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f34f67700 1 --2- 172.21.15.114:0/870225432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8a41e40 0x55f0f8bca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.198+0000 7f0f34f67700 1 -- 172.21.15.114:0/870225432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0f8a2e8d0 con 0x55f0f8bc1120 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f34f67700 1 --2- 172.21.15.114:0/870225432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 0x55f0f8bcae30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0f200a5800 tx=0x7f0f200aa000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969090947fc82e69 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f18ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f18ff9700 1 -- 172.21.15.114:0/870225432 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2001c070 con 0x55f0f8bc1120 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f18ff9700 1 -- 172.21.15.114:0/870225432 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f0f200a9a30 con 0x55f0f8bc1120 2024-05-27T07:39:58.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f463e1b80 1 -- 172.21.15.114:0/870225432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f200a97d0 con 0x55f0f8bc1120 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f18ff9700 1 -- 172.21.15.114:0/870225432 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0f200e01b0 con 0x55f0f8bc1120 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f463e1b80 10 client.15321 fetch_fsmap learned FSMap version 21 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f463e1b80 1 -- 172.21.15.114:0/870225432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f0f8a98760 con 0x55f0f8bc1120 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f18ff9700 1 -- 172.21.15.114:0/870225432 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f200a97d0 con 0x55f0f8bc1120 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f18ff9700 1 -- 172.21.15.114:0/870225432 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0f200a8a30 con 0x55f0f8bc1120 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f463e1b80 10 client.15321 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f463e1b80 10 client.15321 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f463e1b80 20 client.15321 populate_metadata read hostname 'smithi114' 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/870225432 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f0f8bdedf0 0x55f0f8be1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f463e1b80 4 client.15321 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.199+0000 7f0f463e1b80 1 -- 172.21.15.114:0/870225432 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0f8af0cc0 con 0x55f0f8bdedf0 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.200+0000 7f0f35768700 1 --2- 172.21.15.114:0/870225432 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f0f8bdedf0 0x55f0f8be1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.200+0000 7f0f18ff9700 10 client.15321 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.200+0000 7f0f35768700 1 --2- 172.21.15.114:0/870225432 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f0f8bdedf0 0x55f0f8be1210 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f02837677ffdc2e5 server_cookie=3aaaa09e458448cb in_seq=0 out_seq=0 2024-05-27T07:39:58.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f18ff9700 1 -- 172.21.15.114:0/870225432 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f0f8af0cc0 con 0x55f0f8bdedf0 2024-05-27T07:39:58.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f18ff9700 10 client.15321 handle_command_reply: tid=0 2024-05-27T07:39:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f463e1b80 1 client.15321 shutdown 2024-05-27T07:39:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f463e1b80 1 -- 172.21.15.114:0/870225432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 msgr2=0x55f0f8bcae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/870225432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 0x55f0f8bcae30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0f200a5800 tx=0x7f0f200aa000 comp rx=0 tx=0).stop 2024-05-27T07:39:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f463e1b80 1 -- 172.21.15.114:0/870225432 shutdown_connections 2024-05-27T07:39:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/870225432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc7070 0x55f0f8bcb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/870225432 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f0f8bdedf0 0x55f0f8be1210 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/870225432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8a41e40 0x55f0f8bca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/870225432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 0x55f0f8bcae30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f18ff9700 0 client.15321 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f463e1b80 1 -- 172.21.15.114:0/870225432 >> 172.21.15.114:0/870225432 conn(0x55f0f8a44770 msgr2=0x55f0f8bbfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f463e1b80 1 -- 172.21.15.114:0/870225432 shutdown_connections 2024-05-27T07:39:58.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.201+0000 7f0f463e1b80 1 -- 172.21.15.114:0/870225432 wait complete. 2024-05-27T07:39:58.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.202+0000 7f0f463e1b80 20 client.15321 trim_cache size 0 max 16384 2024-05-27T07:39:58.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f463e1b80 1 Processor -- start 2024-05-27T07:39:58.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f463e1b80 1 -- start start 2024-05-27T07:39:58.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f463e1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8a41e40 0x55f0f8bbfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f463e1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc1120 0x55f0f8bca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f463e1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8bc7070 0x55f0f8bb5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f463e1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301ac3c0 con 0x55f0f8a41e40 2024-05-27T07:39:58.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f463e1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301aec40 con 0x55f0f8bc7070 2024-05-27T07:39:58.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f463e1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301aca30 con 0x55f0f8bc1120 2024-05-27T07:39:58.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f35768700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8a41e40 0x55f0f8bbfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f35768700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8a41e40 0x55f0f8bbfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37000/0 (socket says 172.21.15.114:37000) 2024-05-27T07:39:58.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f35768700 1 -- 172.21.15.114:0/4023762017 learned_addr learned my addr 172.21.15.114:0/4023762017 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:58.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f34f67700 1 --2- 172.21.15.114:0/4023762017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc1120 0x55f0f8bca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f35f69700 1 --2- 172.21.15.114:0/4023762017 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8bc7070 0x55f0f8bb5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.222+0000 7f0f35768700 1 -- 172.21.15.114:0/4023762017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc1120 msgr2=0x55f0f8bca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:58.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.223+0000 7f0f35768700 1 --2- 172.21.15.114:0/4023762017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc1120 0x55f0f8bca0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.223+0000 7f0f35768700 1 -- 172.21.15.114:0/4023762017 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8bc7070 msgr2=0x55f0f8bb5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:58.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.223+0000 7f0f35768700 1 --2- 172.21.15.114:0/4023762017 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8bc7070 0x55f0f8bb5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.223+0000 7f0f35768700 1 -- 172.21.15.114:0/4023762017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0f8bbd110 con 0x55f0f8a41e40 2024-05-27T07:39:58.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.223+0000 7f0f35768700 1 --2- 172.21.15.114:0/4023762017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8a41e40 0x55f0f8bbfe20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c013010 tx=0x7f0f2c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87639312970d5144 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:58.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.223+0000 7f0f1affd700 1 -- 172.21.15.114:0/4023762017 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c004830 con 0x55f0f8a41e40 2024-05-27T07:39:58.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.223+0000 7f0f1affd700 1 -- 172.21.15.114:0/4023762017 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f2c005020 con 0x55f0f8a41e40 2024-05-27T07:39:58.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.223+0000 7f0f1affd700 1 -- 172.21.15.114:0/4023762017 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c01d6f0 con 0x55f0f8a41e40 2024-05-27T07:39:58.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.224+0000 7f0f463e1b80 1 -- 172.21.15.114:0/4023762017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8a41e40 msgr2=0x55f0f8bbfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:58.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.224+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/4023762017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8a41e40 0x55f0f8bbfe20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c013010 tx=0x7f0f2c014000 comp rx=0 tx=0).stop 2024-05-27T07:39:58.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.224+0000 7f0f463e1b80 1 -- 172.21.15.114:0/4023762017 shutdown_connections 2024-05-27T07:39:58.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.224+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/4023762017 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8bc7070 0x55f0f8bb5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.224+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/4023762017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8a41e40 0x55f0f8bbfe20 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.224+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/4023762017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc1120 0x55f0f8bca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.224+0000 7f0f463e1b80 1 -- 172.21.15.114:0/4023762017 >> 172.21.15.114:0/4023762017 conn(0x55f0f8a44770 msgr2=0x55f0f8bb54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:58.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.224+0000 7f0f463e1b80 1 -- 172.21.15.114:0/4023762017 shutdown_connections 2024-05-27T07:39:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.224+0000 7f0f463e1b80 1 -- 172.21.15.114:0/4023762017 wait complete. 2024-05-27T07:39:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.224+0000 7f0f463e1b80 1 -- start start 2024-05-27T07:39:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.224+0000 7f0f463e1b80 1 Processor -- start 2024-05-27T07:39:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f463e1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8a41e40 0x55f0f8be4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f463e1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 0x55f0f8be1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f463e1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc7070 0x55f0f8be2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f463e1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301ad100 con 0x55f0f8bc1120 2024-05-27T07:39:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f463e1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301ae570 con 0x55f0f8a41e40 2024-05-27T07:39:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f463e1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301ad7d0 con 0x55f0f8bc7070 2024-05-27T07:39:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f463e1b80 10 client.0 _ll_register_callbacks cb 0x55f0f8a755b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f34f67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 0x55f0f8be1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f35768700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8a41e40 0x55f0f8be4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f34f67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 0x55f0f8be1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37002/0 (socket says 172.21.15.114:37002) 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f34f67700 1 -- 172.21.15.114:0/2152174861 learned_addr learned my addr 172.21.15.114:0/2152174861 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f35f69700 1 --2- 172.21.15.114:0/2152174861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc7070 0x55f0f8be2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f35768700 1 -- 172.21.15.114:0/2152174861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc7070 msgr2=0x55f0f8be2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f35768700 1 --2- 172.21.15.114:0/2152174861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc7070 0x55f0f8be2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f35768700 1 -- 172.21.15.114:0/2152174861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 msgr2=0x55f0f8be1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f35768700 1 --2- 172.21.15.114:0/2152174861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 0x55f0f8be1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f35768700 1 -- 172.21.15.114:0/2152174861 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0f8be27c0 con 0x55f0f8a41e40 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f34f67700 1 --2- 172.21.15.114:0/2152174861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 0x55f0f8be1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f35768700 1 --2- 172.21.15.114:0/2152174861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8a41e40 0x55f0f8be4860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c000c00 tx=0x7f0f2c005330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a78edf590b99d3ca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.225+0000 7f0f18ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f18ff9700 1 -- 172.21.15.114:0/2152174861 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c041060 con 0x55f0f8a41e40 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f463e1b80 1 -- 172.21.15.114:0/2152174861 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f2c005020 con 0x55f0f8a41e40 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f18ff9700 1 -- 172.21.15.114:0/2152174861 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f0f2c004750 con 0x55f0f8a41e40 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f18ff9700 1 -- 172.21.15.114:0/2152174861 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0f2c048050 con 0x55f0f8a41e40 2024-05-27T07:39:58.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f463e1b80 10 client.24977 fetch_fsmap learned FSMap version 21 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f463e1b80 1 -- 172.21.15.114:0/2152174861 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f0f8c494a0 con 0x55f0f8a41e40 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f18ff9700 1 -- 172.21.15.114:0/2152174861 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f2c005020 con 0x55f0f8a41e40 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f18ff9700 1 -- 172.21.15.114:0/2152174861 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0f2c01dab0 con 0x55f0f8a41e40 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f463e1b80 10 client.24977 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f463e1b80 10 client.24977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f463e1b80 20 client.24977 populate_metadata read hostname 'smithi114' 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/2152174861 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f0f8c49ad0 0x55f0f8c4bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f463e1b80 4 client.24977 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f463e1b80 1 -- 172.21.15.114:0/2152174861 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f0f8aea4a0 con 0x55f0f8c49ad0 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.226+0000 7f0f34f67700 1 --2- 172.21.15.114:0/2152174861 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f0f8c49ad0 0x55f0f8c4bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f18ff9700 10 client.24977 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f34f67700 1 --2- 172.21.15.114:0/2152174861 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f0f8c49ad0 0x55f0f8c4bef0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28f0eef81feea8fb server_cookie=ece5f8cea8cf5c53 in_seq=0 out_seq=0 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f18ff9700 1 -- 172.21.15.114:0/2152174861 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f0f8aea4a0 con 0x55f0f8c49ad0 2024-05-27T07:39:58.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f18ff9700 10 client.24977 handle_command_reply: tid=0 2024-05-27T07:39:58.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f463e1b80 1 client.24977 shutdown 2024-05-27T07:39:58.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f463e1b80 1 -- 172.21.15.114:0/2152174861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8a41e40 msgr2=0x55f0f8be4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:58.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/2152174861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8a41e40 0x55f0f8be4860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c000c00 tx=0x7f0f2c005330 comp rx=0 tx=0).stop 2024-05-27T07:39:58.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f463e1b80 1 -- 172.21.15.114:0/2152174861 shutdown_connections 2024-05-27T07:39:58.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/2152174861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0f8bc7070 0x55f0f8be2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/2152174861 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f0f8c49ad0 0x55f0f8c4bef0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/2152174861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f0f8a41e40 0x55f0f8be4860 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f463e1b80 1 --2- 172.21.15.114:0/2152174861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0f8bc1120 0x55f0f8be1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f18ff9700 0 client.24977 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:39:58.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f463e1b80 1 -- 172.21.15.114:0/2152174861 >> 172.21.15.114:0/2152174861 conn(0x55f0f8a44770 msgr2=0x55f0f8bc4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:58.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f463e1b80 1 -- 172.21.15.114:0/2152174861 shutdown_connections 2024-05-27T07:39:58.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f463e1b80 1 -- 172.21.15.114:0/2152174861 wait complete. 2024-05-27T07:39:58.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.227+0000 7f0f463e1b80 20 client.24977 trim_cache size 0 max 16384 2024-05-27T07:39:58.230 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:39:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:39:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:39:58.230 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:39:58.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.228+0000 7f0f379cc700 1 -- 172.21.15.114:0/2175601512 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0f1c0b2a80 msgr2=0x7f0f1c0b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:58.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.228+0000 7f0f379cc700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0f1c0b2a80 0x7f0f1c0b4f90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c011550 tx=0x7f0f2c009260 comp rx=0 tx=0).stop 2024-05-27T07:39:58.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.228+0000 7f0f379cc700 1 -- 172.21.15.114:0/2175601512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f30108f00 msgr2=0x7f0f3019b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:58.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.228+0000 7f0f379cc700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f30108f00 0x7f0f3019b330 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000e9c0 tx=0x7f0f2000ecd0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.228+0000 7f0f379cc700 1 -- 172.21.15.114:0/2175601512 shutdown_connections 2024-05-27T07:39:58.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.228+0000 7f0f379cc700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0f1c0b2a80 0x7f0f1c0b4f90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.228+0000 7f0f379cc700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f30109850 0x7f0f3019f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.228+0000 7f0f379cc700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f301045d0 0x7f0f3019ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.228+0000 7f0f379cc700 1 --2- 172.21.15.114:0/2175601512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f30108f00 0x7f0f3019b330 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:58.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.228+0000 7f0f379cc700 1 -- 172.21.15.114:0/2175601512 >> 172.21.15.114:0/2175601512 conn(0x7f0f300ffec0 msgr2=0x7f0f30108670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:58.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.228+0000 7f0f379cc700 1 -- 172.21.15.114:0/2175601512 shutdown_connections 2024-05-27T07:39:58.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:39:58.228+0000 7f0f379cc700 1 -- 172.21.15.114:0/2175601512 wait complete. 2024-05-27T07:39:58.996 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:39:58.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:39:59.240 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:59.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:39:59 smithi114 ceph-mon[32038]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-05-27T07:39:59.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:39:59 smithi163 ceph-mon[36451]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-05-27T07:39:59.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:39:59 smithi186 ceph-mon[36647]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-05-27T07:40:00.354 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:00 smithi114 ceph-mon[32038]: overall HEALTH_OK 2024-05-27T07:40:00.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:00 smithi163 ceph-mon[36451]: overall HEALTH_OK 2024-05-27T07:40:00.475 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:00 smithi186 ceph-mon[36647]: overall HEALTH_OK 2024-05-27T07:40:00.715 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:40:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:40:00.356+0000 7f5510633700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:40:00.715 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:40:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:40:00.356+0000 7f5510633700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:40:00.715 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:40:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:40:00.356+0000 7f5510633700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:40:00.715 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:40:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:40:00.356+0000 7f5510633700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:40:00.715 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 07:40:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T07:40:00.356+0000 7f5510633700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:40:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.979+0000 7ff9fd347700 1 -- 172.21.15.114:0/3101912371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f8108770 msgr2=0x7ff9f81100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.979+0000 7ff9fd347700 1 --2- 172.21.15.114:0/3101912371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f8108770 0x7ff9f81100d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000b0d0 tx=0x7ff9f000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:40:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.979+0000 7ff9fd347700 1 -- 172.21.15.114:0/3101912371 shutdown_connections 2024-05-27T07:40:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.979+0000 7ff9fd347700 1 --2- 172.21.15.114:0/3101912371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f8108770 0x7ff9f81100d0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.979+0000 7ff9fd347700 1 --2- 172.21.15.114:0/3101912371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f8103540 0x7ff9f8103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.979+0000 7ff9fd347700 1 --2- 172.21.15.114:0/3101912371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8107e20 0x7ff9f8108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.979+0000 7ff9fd347700 1 -- 172.21.15.114:0/3101912371 >> 172.21.15.114:0/3101912371 conn(0x7ff9f8078df0 msgr2=0x7ff9f8079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.979+0000 7ff9fd347700 1 -- 172.21.15.114:0/3101912371 shutdown_connections 2024-05-27T07:40:00.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.979+0000 7ff9fd347700 1 -- 172.21.15.114:0/3101912371 wait complete. 2024-05-27T07:40:00.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.980+0000 7ff9fd347700 1 Processor -- start 2024-05-27T07:40:00.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.980+0000 7ff9fd347700 1 -- start start 2024-05-27T07:40:00.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9fd347700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8103540 0x7ff9f819f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:00.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9fd347700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f8107e20 0x7ff9f819fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:00.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9fd347700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f8108770 0x7ff9f8199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9fd347700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f81133a0 con 0x7ff9f8107e20 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9fd347700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f8113220 con 0x7ff9f8108770 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9fd347700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f8113520 con 0x7ff9f8103540 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9effff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f8107e20 0x7ff9f819fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9effff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f8107e20 0x7ff9f819fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37046/0 (socket says 172.21.15.114:37046) 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9effff700 1 -- 172.21.15.114:0/2573879198 learned_addr learned my addr 172.21.15.114:0/2573879198 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9f6ffd700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8103540 0x7ff9f819f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9f77fe700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f8108770 0x7ff9f8199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9effff700 1 -- 172.21.15.114:0/2573879198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8103540 msgr2=0x7ff9f819f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9effff700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8103540 0x7ff9f819f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9effff700 1 -- 172.21.15.114:0/2573879198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f8108770 msgr2=0x7ff9f8199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9effff700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f8108770 0x7ff9f8199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9effff700 1 -- 172.21.15.114:0/2573879198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f819a270 con 0x7ff9f8107e20 2024-05-27T07:40:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9f77fe700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f8108770 0x7ff9f8199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.981+0000 7ff9effff700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f8107e20 0x7ff9f819fef0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00078e0 tx=0x7ff9e0007bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.982+0000 7ff9f4ff9700 1 -- 172.21.15.114:0/2573879198 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0004020 con 0x7ff9f8107e20 2024-05-27T07:40:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.982+0000 7ff9f4ff9700 1 -- 172.21.15.114:0/2573879198 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9e0004830 con 0x7ff9f8107e20 2024-05-27T07:40:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.982+0000 7ff9fd347700 1 -- 172.21.15.114:0/2573879198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9f819a560 con 0x7ff9f8107e20 2024-05-27T07:40:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.982+0000 7ff9fd347700 1 -- 172.21.15.114:0/2573879198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9f80759e0 con 0x7ff9f8107e20 2024-05-27T07:40:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.982+0000 7ff9f4ff9700 1 -- 172.21.15.114:0/2573879198 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0005560 con 0x7ff9f8107e20 2024-05-27T07:40:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.983+0000 7ff9fd347700 1 -- 172.21.15.114:0/2573879198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9d8005330 con 0x7ff9f8107e20 2024-05-27T07:40:00.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.984+0000 7ff9f4ff9700 1 -- 172.21.15.114:0/2573879198 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 23) ==== 149829+0+0 (secure 0 0 0) 0x7ff9e001f020 con 0x7ff9f8107e20 2024-05-27T07:40:00.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.985+0000 7ff9f4ff9700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff9e40b2b00 0x7ff9e40b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:00.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.985+0000 7ff9f6ffd700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff9e40b2b00 0x7ff9e40b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:00.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.985+0000 7ff9f4ff9700 1 -- 172.21.15.114:0/2573879198 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7ff9e00e43d0 con 0x7ff9f8107e20 2024-05-27T07:40:00.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.986+0000 7ff9f6ffd700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff9e40b2b00 0x7ff9e40b5010 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff9e8005950 tx=0x7ff9e800a300 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:00.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:00.989+0000 7ff9f4ff9700 1 -- 172.21.15.114:0/2573879198 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff9e0026030 con 0x7ff9f8107e20 2024-05-27T07:40:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.241+0000 7ff9fd347700 1 -- 172.21.15.114:0/2573879198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9d8005620 con 0x7ff9f8107e20 2024-05-27T07:40:01.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:01 smithi114 ceph-mon[32038]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-05-27T07:40:01.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.242+0000 7ff9f4ff9700 1 -- 172.21.15.114:0/2573879198 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11827 (secure 0 0 0) 0x7ff9e00ab890 con 0x7ff9f8107e20 2024-05-27T07:40:01.239 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:40:01.239 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:37:06.645274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:40:01.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.244+0000 7ff9fd347700 1 -- 172.21.15.114:0/2573879198 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff9e40b2b00 msgr2=0x7ff9e40b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:01.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.244+0000 7ff9fd347700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff9e40b2b00 0x7ff9e40b5010 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff9e8005950 tx=0x7ff9e800a300 comp rx=0 tx=0).stop 2024-05-27T07:40:01.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.244+0000 7ff9fd347700 1 -- 172.21.15.114:0/2573879198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f8107e20 msgr2=0x7ff9f819fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:01.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.244+0000 7ff9fd347700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f8107e20 0x7ff9f819fef0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00078e0 tx=0x7ff9e0007bf0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.244+0000 7ff9fd347700 1 -- 172.21.15.114:0/2573879198 shutdown_connections 2024-05-27T07:40:01.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.244+0000 7ff9fd347700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff9e40b2b00 0x7ff9e40b5010 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.244+0000 7ff9fd347700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f8108770 0x7ff9f8199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.244+0000 7ff9fd347700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8103540 0x7ff9f819f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.244+0000 7ff9fd347700 1 --2- 172.21.15.114:0/2573879198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f8107e20 0x7ff9f819fef0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.244+0000 7ff9fd347700 1 -- 172.21.15.114:0/2573879198 >> 172.21.15.114:0/2573879198 conn(0x7ff9f8078df0 msgr2=0x7ff9f8100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:01.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.245+0000 7ff9fd347700 1 -- 172.21.15.114:0/2573879198 shutdown_connections 2024-05-27T07:40:01.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:01.245+0000 7ff9fd347700 1 -- 172.21.15.114:0/2573879198 wait complete. 2024-05-27T07:40:01.242 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:40:01.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:01 smithi163 ceph-mon[36451]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-05-27T07:40:01.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:01 smithi186 ceph-mon[36647]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-05-27T07:40:01.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:40:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:02.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:02 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2573879198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:40:02.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:02 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2573879198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:40:02.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:02 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2573879198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:40:03.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:03 smithi114 ceph-mon[32038]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 34 op/s 2024-05-27T07:40:03.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:03 smithi114 ceph-mon[32038]: mgrmap e24: x(active, since 18s), standbys: y, z 2024-05-27T07:40:03.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:03 smithi163 ceph-mon[36451]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 34 op/s 2024-05-27T07:40:03.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:03 smithi163 ceph-mon[36451]: mgrmap e24: x(active, since 18s), standbys: y, z 2024-05-27T07:40:03.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:03 smithi186 ceph-mon[36647]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 34 op/s 2024-05-27T07:40:03.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:03 smithi186 ceph-mon[36647]: mgrmap e24: x(active, since 18s), standbys: y, z 2024-05-27T07:40:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.890+0000 7fbc0ec2b700 1 -- 172.21.15.114:0/294382079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc08105e00 msgr2=0x7fbc0810e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.890+0000 7fbc0ec2b700 1 --2- 172.21.15.114:0/294382079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc08105e00 0x7fbc0810e350 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4009990 tx=0x7fbbf4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.890+0000 7fbc0ec2b700 1 -- 172.21.15.114:0/294382079 shutdown_connections 2024-05-27T07:40:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.890+0000 7fbc0ec2b700 1 --2- 172.21.15.114:0/294382079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc0810e8e0 0x7fbc08110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.890+0000 7fbc0ec2b700 1 --2- 172.21.15.114:0/294382079 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbc081054b0 0x7fbc081058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.890+0000 7fbc0ec2b700 1 --2- 172.21.15.114:0/294382079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc08105e00 0x7fbc0810e350 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.890+0000 7fbc0ec2b700 1 -- 172.21.15.114:0/294382079 >> 172.21.15.114:0/294382079 conn(0x7fbc080fedb0 msgr2=0x7fbc081011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.890+0000 7fbc0ec2b700 1 -- 172.21.15.114:0/294382079 shutdown_connections 2024-05-27T07:40:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.890+0000 7fbc0ec2b700 1 -- 172.21.15.114:0/294382079 wait complete. 2024-05-27T07:40:03.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.890+0000 7fbc0ec2b700 1 Processor -- start 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.890+0000 7fbc0ec2b700 1 -- start start 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0ec2b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081054b0 0x7fbc0819ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0ec2b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbc08105e00 0x7fbc0819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0ec2b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc0810e8e0 0x7fbc0819f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0ec2b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbc08114440 con 0x7fbc081054b0 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0ec2b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081142c0 con 0x7fbc08105e00 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0ec2b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081145c0 con 0x7fbc0810e8e0 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0c9c7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081054b0 0x7fbc0819ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0c9c7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081054b0 0x7fbc0819ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37070/0 (socket says 172.21.15.114:37070) 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0c9c7700 1 -- 172.21.15.114:0/381341167 learned_addr learned my addr 172.21.15.114:0/381341167 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0d1c8700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc0810e8e0 0x7fbc0819f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc07fff700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbc08105e00 0x7fbc0819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0c9c7700 1 -- 172.21.15.114:0/381341167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc0810e8e0 msgr2=0x7fbc0819f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc0810e8e0 0x7fbc0819f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0c9c7700 1 -- 172.21.15.114:0/381341167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbc08105e00 msgr2=0x7fbc0819b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:03.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbc08105e00 0x7fbc0819b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0c9c7700 1 -- 172.21.15.114:0/381341167 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0819fc60 con 0x7fbc081054b0 2024-05-27T07:40:03.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0d1c8700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc0810e8e0 0x7fbc0819f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:03.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.891+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081054b0 0x7fbc0819ad90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00c920 tx=0x7fbbfc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:03.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.892+0000 7fbc05ffb700 1 -- 172.21.15.114:0/381341167 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc007b00 con 0x7fbc081054b0 2024-05-27T07:40:03.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.892+0000 7fbc05ffb700 1 -- 172.21.15.114:0/381341167 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbfc004ce0 con 0x7fbc081054b0 2024-05-27T07:40:03.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.892+0000 7fbc0ec2b700 1 -- 172.21.15.114:0/381341167 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0819ff50 con 0x7fbc081054b0 2024-05-27T07:40:03.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.892+0000 7fbc05ffb700 1 -- 172.21.15.114:0/381341167 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc005550 con 0x7fbc081054b0 2024-05-27T07:40:03.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.892+0000 7fbc0ec2b700 1 -- 172.21.15.114:0/381341167 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc081a7870 con 0x7fbc081054b0 2024-05-27T07:40:03.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.894+0000 7fbc05ffb700 1 -- 172.21.15.114:0/381341167 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7fbbfc007ca0 con 0x7fbc081054b0 2024-05-27T07:40:03.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.895+0000 7fbc05ffb700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbbf80b2b10 0x7fbbf80b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.895+0000 7fbc07fff700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbbf80b2b10 0x7fbbf80b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.895+0000 7fbc05ffb700 1 -- 172.21.15.114:0/381341167 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fbbfc0e2b80 con 0x7fbc081054b0 2024-05-27T07:40:03.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.895+0000 7fbc07fff700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbbf80b2b10 0x7fbbf80b5020 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbbf400b5c0 tx=0x7fbbf4005960 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:03.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.895+0000 7fbc1d640b80 1 Processor -- start 2024-05-27T07:40:03.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc1d640b80 1 -- start start 2024-05-27T07:40:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc1d640b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 0x55ca78c5b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc1d640b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78c5bc50 0x55ca78c55a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc1d640b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 0x55ca78c56390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc1d640b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ca78b84c90 con 0x55ca78ad7e40 2024-05-27T07:40:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc1d640b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ca78b74830 con 0x55ca78c5bc50 2024-05-27T07:40:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc1d640b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ca78b78870 con 0x55ca78c55f80 2024-05-27T07:40:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc0c9c7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 0x55ca78c5b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc0c9c7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 0x55ca78c5b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37072/0 (socket says 172.21.15.114:37072) 2024-05-27T07:40:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc0c9c7700 1 -- 172.21.15.114:0/1698522487 learned_addr learned my addr 172.21.15.114:0/1698522487 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc0d1c8700 1 --2- 172.21.15.114:0/1698522487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 0x55ca78c56390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc07fff700 1 --2- 172.21.15.114:0/1698522487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78c5bc50 0x55ca78c55a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc0c9c7700 1 -- 172.21.15.114:0/1698522487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 msgr2=0x55ca78c56390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/1698522487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 0x55ca78c56390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc0c9c7700 1 -- 172.21.15.114:0/1698522487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78c5bc50 msgr2=0x55ca78c55a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/1698522487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78c5bc50 0x55ca78c55a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc0c9c7700 1 -- 172.21.15.114:0/1698522487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca78ac48d0 con 0x55ca78ad7e40 2024-05-27T07:40:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.896+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/1698522487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 0x55ca78c5b710 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc000c00 tx=0x7fbbfc0a9370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce5fb338a075d46 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.897+0000 7fbbf2ffd700 1 -- 172.21.15.114:0/1698522487 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc01d070 con 0x55ca78ad7e40 2024-05-27T07:40:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.897+0000 7fbbf2ffd700 1 -- 172.21.15.114:0/1698522487 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbfc0adc10 con 0x55ca78ad7e40 2024-05-27T07:40:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.897+0000 7fbbf2ffd700 1 -- 172.21.15.114:0/1698522487 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc0a9ac0 con 0x55ca78ad7e40 2024-05-27T07:40:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.897+0000 7fbc1d640b80 1 -- 172.21.15.114:0/1698522487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 msgr2=0x55ca78c5b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.897+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/1698522487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 0x55ca78c5b710 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc000c00 tx=0x7fbbfc0a9370 comp rx=0 tx=0).stop 2024-05-27T07:40:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.898+0000 7fbc1d640b80 1 -- 172.21.15.114:0/1698522487 shutdown_connections 2024-05-27T07:40:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.898+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/1698522487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 0x55ca78c56390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.898+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/1698522487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 0x55ca78c5b710 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.898+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/1698522487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78c5bc50 0x55ca78c55a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.898+0000 7fbc1d640b80 1 -- 172.21.15.114:0/1698522487 >> 172.21.15.114:0/1698522487 conn(0x55ca78b32af0 msgr2=0x55ca78c50640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.898+0000 7fbc1d640b80 1 -- 172.21.15.114:0/1698522487 shutdown_connections 2024-05-27T07:40:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.898+0000 7fbc1d640b80 1 -- 172.21.15.114:0/1698522487 wait complete. 2024-05-27T07:40:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.898+0000 7fbc1d640b80 1 -- start start 2024-05-27T07:40:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.898+0000 7fbc1d640b80 1 Processor -- start 2024-05-27T07:40:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc1d640b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78ad7e40 0x55ca78c5f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc1d640b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 0x55ca78c5f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc1d640b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 0x55ca78c5fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc1d640b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ca78a7afa0 con 0x55ca78c5bc50 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc1d640b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ca78b167c0 con 0x55ca78ad7e40 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc1d640b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ca78c62740 con 0x55ca78c55f80 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc1d640b80 10 client.0 _ll_register_callbacks cb 0x55ca78b0f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc0d1c8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 0x55ca78c5fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc0d1c8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 0x55ca78c5fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37086/0 (socket says 172.21.15.114:37086) 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc0d1c8700 1 -- 172.21.15.114:0/1526998977 learned_addr learned my addr 172.21.15.114:0/1526998977 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/1526998977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78ad7e40 0x55ca78c5f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc0d1c8700 1 -- 172.21.15.114:0/1526998977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 msgr2=0x55ca78c5f900 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc0d1c8700 1 --2- 172.21.15.114:0/1526998977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 0x55ca78c5f900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc0d1c8700 1 -- 172.21.15.114:0/1526998977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78ad7e40 msgr2=0x55ca78c5f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc0d1c8700 1 --2- 172.21.15.114:0/1526998977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78ad7e40 0x55ca78c5f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc0d1c8700 1 -- 172.21.15.114:0/1526998977 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca78b13690 con 0x55ca78c5bc50 2024-05-27T07:40:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/1526998977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78ad7e40 0x55ca78c5f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc0d1c8700 1 --2- 172.21.15.114:0/1526998977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 0x55ca78c5fe40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbc00007940 tx=0x7fbc00007d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9562dd78c5eb54c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbbe7fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbbe7fff700 1 -- 172.21.15.114:0/1526998977 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00019020 con 0x55ca78c5bc50 2024-05-27T07:40:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbbe7fff700 1 -- 172.21.15.114:0/1526998977 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fbc00010860 con 0x55ca78c5bc50 2024-05-27T07:40:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.899+0000 7fbc1d640b80 1 -- 172.21.15.114:0/1526998977 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbfc0adc10 con 0x55ca78c5bc50 2024-05-27T07:40:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.900+0000 7fbbe7fff700 1 -- 172.21.15.114:0/1526998977 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbc00024050 con 0x55ca78c5bc50 2024-05-27T07:40:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.900+0000 7fbc1d640b80 10 client.15351 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.900+0000 7fbc1d640b80 1 -- 172.21.15.114:0/1526998977 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ca78c6d280 con 0x55ca78c5bc50 2024-05-27T07:40:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.900+0000 7fbbe7fff700 1 -- 172.21.15.114:0/1526998977 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbfc0adc10 con 0x55ca78c5bc50 2024-05-27T07:40:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.900+0000 7fbbe7fff700 1 -- 172.21.15.114:0/1526998977 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fbc0001d3a0 con 0x55ca78c5bc50 2024-05-27T07:40:03.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.900+0000 7fbc1d640b80 10 client.15351 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:03.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.900+0000 7fbc1d640b80 10 client.15351 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:03.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.900+0000 7fbc1d640b80 20 client.15351 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:03.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.900+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/1526998977 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca78c6f8c0 0x55ca78c71d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.900+0000 7fbc1d640b80 4 client.15351 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:03.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.900+0000 7fbc1d640b80 1 -- 172.21.15.114:0/1526998977 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca78c219f0 con 0x55ca78c6f8c0 2024-05-27T07:40:03.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.900+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/1526998977 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca78c6f8c0 0x55ca78c71d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.901+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/1526998977 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca78c6f8c0 0x55ca78c71d90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed2a2cfaa338f551 server_cookie=3e5578a14170466c in_seq=0 out_seq=0 2024-05-27T07:40:03.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.901+0000 7fbbe7fff700 10 client.15351 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:03.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbbe7fff700 1 -- 172.21.15.114:0/1526998977 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ca78c219f0 con 0x55ca78c6f8c0 2024-05-27T07:40:03.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbbe7fff700 10 client.15351 handle_command_reply: tid=0 2024-05-27T07:40:03.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbc1d640b80 1 client.15351 shutdown 2024-05-27T07:40:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbc1d640b80 1 -- 172.21.15.114:0/1526998977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 msgr2=0x55ca78c5fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/1526998977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 0x55ca78c5fe40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbc00007940 tx=0x7fbc00007d00 comp rx=0 tx=0).stop 2024-05-27T07:40:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbc1d640b80 1 -- 172.21.15.114:0/1526998977 shutdown_connections 2024-05-27T07:40:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/1526998977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 0x55ca78c5fe40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/1526998977 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca78c6f8c0 0x55ca78c71d90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbbe7fff700 0 client.15351 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/1526998977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78ad7e40 0x55ca78c5f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/1526998977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 0x55ca78c5f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbc1d640b80 1 -- 172.21.15.114:0/1526998977 >> 172.21.15.114:0/1526998977 conn(0x55ca78b32af0 msgr2=0x55ca78c54a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbc1d640b80 1 -- 172.21.15.114:0/1526998977 shutdown_connections 2024-05-27T07:40:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbc1d640b80 1 -- 172.21.15.114:0/1526998977 wait complete. 2024-05-27T07:40:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.902+0000 7fbc1d640b80 20 client.15351 trim_cache size 0 max 16384 2024-05-27T07:40:03.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.917+0000 7fbc1d640b80 1 Processor -- start 2024-05-27T07:40:03.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.917+0000 7fbc1d640b80 1 -- start start 2024-05-27T07:40:03.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.917+0000 7fbc1d640b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78ad7e40 0x55ca78c55a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.917+0000 7fbc1d640b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 0x55ca78c6b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.917+0000 7fbc1d640b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 0x55ca78c59010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.917+0000 7fbc1d640b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081ae5d0 con 0x55ca78c5bc50 2024-05-27T07:40:03.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.917+0000 7fbc1d640b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081b0e50 con 0x55ca78ad7e40 2024-05-27T07:40:03.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.917+0000 7fbc1d640b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081aec40 con 0x55ca78c55f80 2024-05-27T07:40:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbc0d1c8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 0x55ca78c59010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbc0d1c8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 0x55ca78c59010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37096/0 (socket says 172.21.15.114:37096) 2024-05-27T07:40:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbc0d1c8700 1 -- 172.21.15.114:0/3068961628 learned_addr learned my addr 172.21.15.114:0/3068961628 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/3068961628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78ad7e40 0x55ca78c55a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbc07fff700 1 --2- 172.21.15.114:0/3068961628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 0x55ca78c6b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbc0d1c8700 1 -- 172.21.15.114:0/3068961628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 msgr2=0x55ca78c6b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbc0d1c8700 1 --2- 172.21.15.114:0/3068961628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 0x55ca78c6b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbc0d1c8700 1 -- 172.21.15.114:0/3068961628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78ad7e40 msgr2=0x55ca78c55a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbc0d1c8700 1 --2- 172.21.15.114:0/3068961628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78ad7e40 0x55ca78c55a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbc0d1c8700 1 -- 172.21.15.114:0/3068961628 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca78c569f0 con 0x55ca78c5bc50 2024-05-27T07:40:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/3068961628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78ad7e40 0x55ca78c55a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbc0d1c8700 1 --2- 172.21.15.114:0/3068961628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 0x55ca78c59010 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fbc00000f80 tx=0x7fbc000104c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5579aeacf9906348 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbbf27fc700 1 -- 172.21.15.114:0/3068961628 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00019070 con 0x55ca78c5bc50 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbbf27fc700 1 -- 172.21.15.114:0/3068961628 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc0002d070 con 0x55ca78c5bc50 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.918+0000 7fbbf27fc700 1 -- 172.21.15.114:0/3068961628 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00005ec0 con 0x55ca78c5bc50 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.919+0000 7fbc1d640b80 1 -- 172.21.15.114:0/3068961628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 msgr2=0x55ca78c59010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.919+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/3068961628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 0x55ca78c59010 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fbc00000f80 tx=0x7fbc000104c0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.919+0000 7fbc1d640b80 1 -- 172.21.15.114:0/3068961628 shutdown_connections 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.919+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/3068961628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78c5bc50 0x55ca78c59010 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.919+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/3068961628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78ad7e40 0x55ca78c55a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.919+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/3068961628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c55f80 0x55ca78c6b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.919+0000 7fbc1d640b80 1 -- 172.21.15.114:0/3068961628 >> 172.21.15.114:0/3068961628 conn(0x55ca78b32af0 msgr2=0x55ca78c501c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.919+0000 7fbc1d640b80 1 -- 172.21.15.114:0/3068961628 shutdown_connections 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.919+0000 7fbc1d640b80 1 -- 172.21.15.114:0/3068961628 wait complete. 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.919+0000 7fbc1d640b80 1 -- start start 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc1d640b80 1 Processor -- start 2024-05-27T07:40:03.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc1d640b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 0x55ca78c77bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc1d640b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78c55f80 0x55ca78c67380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc1d640b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c5bc50 0x55ca78c678c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc1d640b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081af310 con 0x55ca78ad7e40 2024-05-27T07:40:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc1d640b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081b0780 con 0x55ca78c55f80 2024-05-27T07:40:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc1d640b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081af9e0 con 0x55ca78c5bc50 2024-05-27T07:40:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc1d640b80 10 client.0 _ll_register_callbacks cb 0x55ca78b0f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc0c9c7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 0x55ca78c77bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc0c9c7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 0x55ca78c77bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37108/0 (socket says 172.21.15.114:37108) 2024-05-27T07:40:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc0c9c7700 1 -- 172.21.15.114:0/992819787 learned_addr learned my addr 172.21.15.114:0/992819787 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc07fff700 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78c55f80 0x55ca78c67380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc0c9c7700 1 -- 172.21.15.114:0/992819787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c5bc50 msgr2=0x55ca78c678c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc0d1c8700 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c5bc50 0x55ca78c678c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c5bc50 0x55ca78c678c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc0c9c7700 1 -- 172.21.15.114:0/992819787 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78c55f80 msgr2=0x55ca78c67380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78c55f80 0x55ca78c67380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc0c9c7700 1 -- 172.21.15.114:0/992819787 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca78c67ff0 con 0x55ca78ad7e40 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.920+0000 7fbc0d1c8700 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c5bc50 0x55ca78c678c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbc0c9c7700 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 0x55ca78c77bd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0a9560 tx=0x7fbbfc0a9080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5331529849c2825 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbbe7fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbbe7fff700 1 -- 172.21.15.114:0/992819787 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc01d030 con 0x55ca78ad7e40 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbbe7fff700 1 -- 172.21.15.114:0/992819787 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fbbfc0a5e50 con 0x55ca78ad7e40 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbc1d640b80 1 -- 172.21.15.114:0/992819787 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc0002d070 con 0x55ca78ad7e40 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbbe7fff700 1 -- 172.21.15.114:0/992819787 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbbfc0056f0 con 0x55ca78ad7e40 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbc1d640b80 10 client.15357 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbc1d640b80 1 -- 172.21.15.114:0/992819787 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ca78cde3c0 con 0x55ca78ad7e40 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbbe7fff700 1 -- 172.21.15.114:0/992819787 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc0002d070 con 0x55ca78ad7e40 2024-05-27T07:40:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbbe7fff700 1 -- 172.21.15.114:0/992819787 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fbbfc0a9ac0 con 0x55ca78ad7e40 2024-05-27T07:40:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbc1d640b80 10 client.15357 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbc1d640b80 10 client.15357 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbc1d640b80 20 client.15357 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca78c6f8c0 0x55ca78ce0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbc1d640b80 4 client.15357 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:40:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.921+0000 7fbc1d640b80 1 -- 172.21.15.114:0/992819787 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ca78b86be0 con 0x55ca78c6f8c0 2024-05-27T07:40:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbc07fff700 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca78c6f8c0 0x55ca78ce0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbbe7fff700 10 client.15357 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbc07fff700 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca78c6f8c0 0x55ca78ce0660 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19c3d35127be2624 server_cookie=5ec37c94ed43a9df in_seq=0 out_seq=0 2024-05-27T07:40:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbbe7fff700 1 -- 172.21.15.114:0/992819787 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ca78b86be0 con 0x55ca78c6f8c0 2024-05-27T07:40:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbbe7fff700 10 client.15357 handle_command_reply: tid=0 2024-05-27T07:40:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbc1d640b80 1 client.15357 shutdown 2024-05-27T07:40:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbc1d640b80 1 -- 172.21.15.114:0/992819787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 msgr2=0x55ca78c77bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 0x55ca78c77bd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0a9560 tx=0x7fbbfc0a9080 comp rx=0 tx=0).stop 2024-05-27T07:40:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbc1d640b80 1 -- 172.21.15.114:0/992819787 shutdown_connections 2024-05-27T07:40:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca78c5bc50 0x55ca78c678c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca78ad7e40 0x55ca78c77bd0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca78c6f8c0 0x55ca78ce0660 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbbe7fff700 0 client.15357 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbc1d640b80 1 --2- 172.21.15.114:0/992819787 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca78c55f80 0x55ca78c67380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.922+0000 7fbc1d640b80 1 -- 172.21.15.114:0/992819787 >> 172.21.15.114:0/992819787 conn(0x55ca78b32af0 msgr2=0x55ca78c501c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.923+0000 7fbc1d640b80 1 -- 172.21.15.114:0/992819787 shutdown_connections 2024-05-27T07:40:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.923+0000 7fbc1d640b80 1 -- 172.21.15.114:0/992819787 wait complete. 2024-05-27T07:40:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.923+0000 7fbc1d640b80 20 client.15357 trim_cache size 0 max 16384 2024-05-27T07:40:03.928 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:40:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.923+0000 7fbc0ec2b700 1 -- 172.21.15.114:0/381341167 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbbf80b2b10 msgr2=0x7fbbf80b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.923+0000 7fbc0ec2b700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbbf80b2b10 0x7fbbf80b5020 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbbf400b5c0 tx=0x7fbbf4005960 comp rx=0 tx=0).stop 2024-05-27T07:40:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.923+0000 7fbc0ec2b700 1 -- 172.21.15.114:0/381341167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081054b0 msgr2=0x7fbc0819ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.923+0000 7fbc0ec2b700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081054b0 0x7fbc0819ad90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00c920 tx=0x7fbbfc00cc30 comp rx=0 tx=0).stop 2024-05-27T07:40:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.923+0000 7fbc0ec2b700 1 -- 172.21.15.114:0/381341167 shutdown_connections 2024-05-27T07:40:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.923+0000 7fbc0ec2b700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc0810e8e0 0x7fbc0819f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.923+0000 7fbc0ec2b700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081054b0 0x7fbc0819ad90 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.923+0000 7fbc0ec2b700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbbf80b2b10 0x7fbbf80b5020 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.923+0000 7fbc0ec2b700 1 --2- 172.21.15.114:0/381341167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbc08105e00 0x7fbc0819b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.923+0000 7fbc0ec2b700 1 -- 172.21.15.114:0/381341167 >> 172.21.15.114:0/381341167 conn(0x7fbc080fedb0 msgr2=0x7fbc080ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.924+0000 7fbc0ec2b700 1 -- 172.21.15.114:0/381341167 shutdown_connections 2024-05-27T07:40:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:03.924+0000 7fbc0ec2b700 1 -- 172.21.15.114:0/381341167 wait complete. 2024-05-27T07:40:05.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:05 smithi163 ceph-mon[36451]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 33 op/s 2024-05-27T07:40:05.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:05 smithi186 ceph-mon[36647]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 33 op/s 2024-05-27T07:40:05.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:05 smithi114 ceph-mon[32038]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 33 op/s 2024-05-27T07:40:05.743 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:40:05.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:40:05.995 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:07.429 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:07 smithi114 ceph-mon[32038]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 100 KiB/s wr, 49 op/s 2024-05-27T07:40:07.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:07 smithi163 ceph-mon[36451]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 100 KiB/s wr, 49 op/s 2024-05-27T07:40:07.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:07 smithi186 ceph-mon[36647]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 100 KiB/s wr, 49 op/s 2024-05-27T07:40:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.854+0000 7f8a49e82700 1 -- 172.21.15.114:0/2983528866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a441045c0 msgr2=0x7f8a441049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.854+0000 7f8a49e82700 1 --2- 172.21.15.114:0/2983528866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a441045c0 0x7f8a441049d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8a340099c0 tx=0x7f8a34009cd0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.854+0000 7f8a49e82700 1 -- 172.21.15.114:0/2983528866 shutdown_connections 2024-05-27T07:40:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.854+0000 7f8a49e82700 1 --2- 172.21.15.114:0/2983528866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a44109840 0x7f8a441111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.854+0000 7f8a49e82700 1 --2- 172.21.15.114:0/2983528866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a441045c0 0x7f8a441049d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.854+0000 7f8a49e82700 1 --2- 172.21.15.114:0/2983528866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a44108ef0 0x7f8a44109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.854+0000 7f8a49e82700 1 -- 172.21.15.114:0/2983528866 >> 172.21.15.114:0/2983528866 conn(0x7f8a440ffe90 msgr2=0x7f8a441022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.854+0000 7f8a49e82700 1 -- 172.21.15.114:0/2983528866 shutdown_connections 2024-05-27T07:40:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.854+0000 7f8a49e82700 1 -- 172.21.15.114:0/2983528866 wait complete. 2024-05-27T07:40:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a49e82700 1 Processor -- start 2024-05-27T07:40:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a49e82700 1 -- start start 2024-05-27T07:40:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a49e82700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a441045c0 0x7f8a4419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a49e82700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a44108ef0 0x7f8a4419b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a49e82700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a44109840 0x7f8a4419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a49e82700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8a44114470 con 0x7f8a44109840 2024-05-27T07:40:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a49e82700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441142f0 con 0x7f8a441045c0 2024-05-27T07:40:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a49e82700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441145f0 con 0x7f8a44108ef0 2024-05-27T07:40:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a43fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a44109840 0x7f8a4419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a43fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a44109840 0x7f8a4419f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41232/0 (socket says 172.21.15.114:41232) 2024-05-27T07:40:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a43fff700 1 -- 172.21.15.114:0/1681210965 learned_addr learned my addr 172.21.15.114:0/1681210965 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a437fe700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a441045c0 0x7f8a4419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a3bfff700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a44108ef0 0x7f8a4419b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a43fff700 1 -- 172.21.15.114:0/1681210965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a44108ef0 msgr2=0x7f8a4419b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a43fff700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a44108ef0 0x7f8a4419b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a43fff700 1 -- 172.21.15.114:0/1681210965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a441045c0 msgr2=0x7f8a4419ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a43fff700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a441045c0 0x7f8a4419ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.855+0000 7f8a43fff700 1 -- 172.21.15.114:0/1681210965 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4419fcd0 con 0x7f8a44109840 2024-05-27T07:40:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.856+0000 7f8a43fff700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a44109840 0x7f8a4419f630 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c007ee0 tx=0x7f8a3c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.856+0000 7f8a417fa700 1 -- 172.21.15.114:0/1681210965 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c00dbd0 con 0x7f8a44109840 2024-05-27T07:40:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.856+0000 7f8a417fa700 1 -- 172.21.15.114:0/1681210965 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a3c00dd70 con 0x7f8a44109840 2024-05-27T07:40:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.856+0000 7f8a49e82700 1 -- 172.21.15.114:0/1681210965 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4419ffc0 con 0x7f8a44109840 2024-05-27T07:40:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.856+0000 7f8a417fa700 1 -- 172.21.15.114:0/1681210965 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c005990 con 0x7f8a44109840 2024-05-27T07:40:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.856+0000 7f8a49e82700 1 -- 172.21.15.114:0/1681210965 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a44076af0 con 0x7f8a44109840 2024-05-27T07:40:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.859+0000 7f8a417fa700 1 -- 172.21.15.114:0/1681210965 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7f8a3c01a020 con 0x7f8a44109840 2024-05-27T07:40:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.859+0000 7f8a417fa700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8a300b2b10 0x7f8a300b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.859+0000 7f8a437fe700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8a300b2b10 0x7f8a300b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a417fa700 1 -- 172.21.15.114:0/1681210965 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f8a3c0e23c0 con 0x7f8a44109840 2024-05-27T07:40:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a58897b80 1 Processor -- start 2024-05-27T07:40:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a58897b80 1 -- start start 2024-05-27T07:40:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a437fe700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8a300b2b10 0x7f8a300b5020 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009990 tx=0x7f8a340036e0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a58897b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc1a7ec0 0x556ddc32cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a58897b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc32d0f0 0x556ddc326c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a58897b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 0x556ddc3275b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a58897b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556ddc254df0 con 0x556ddc3271a0 2024-05-27T07:40:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a58897b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556ddc244990 con 0x556ddc1a7ec0 2024-05-27T07:40:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a58897b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556ddc2489d0 con 0x556ddc32d0f0 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a43fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 0x556ddc3275b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a43fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 0x556ddc3275b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41236/0 (socket says 172.21.15.114:41236) 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a43fff700 1 -- 172.21.15.114:0/293862024 learned_addr learned my addr 172.21.15.114:0/293862024 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a437fe700 1 --2- 172.21.15.114:0/293862024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc1a7ec0 0x556ddc32cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.860+0000 7f8a3bfff700 1 --2- 172.21.15.114:0/293862024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc32d0f0 0x556ddc326c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.861+0000 7f8a43fff700 1 -- 172.21.15.114:0/293862024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc32d0f0 msgr2=0x556ddc326c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.861+0000 7f8a43fff700 1 --2- 172.21.15.114:0/293862024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc32d0f0 0x556ddc326c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.861+0000 7f8a43fff700 1 -- 172.21.15.114:0/293862024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc1a7ec0 msgr2=0x556ddc32cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.861+0000 7f8a43fff700 1 --2- 172.21.15.114:0/293862024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc1a7ec0 0x556ddc32cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.861+0000 7f8a43fff700 1 -- 172.21.15.114:0/293862024 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ddc1ee030 con 0x556ddc3271a0 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.861+0000 7f8a43fff700 1 --2- 172.21.15.114:0/293862024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 0x556ddc3275b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0a8da0 tx=0x7f8a3c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857efa5310cf2978 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.861+0000 7f8a39ffb700 1 -- 172.21.15.114:0/293862024 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c0a97a0 con 0x556ddc3271a0 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.861+0000 7f8a39ffb700 1 -- 172.21.15.114:0/293862024 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a3c0a90a0 con 0x556ddc3271a0 2024-05-27T07:40:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.861+0000 7f8a39ffb700 1 -- 172.21.15.114:0/293862024 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c0a9300 con 0x556ddc3271a0 2024-05-27T07:40:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.862+0000 7f8a58897b80 1 -- 172.21.15.114:0/293862024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 msgr2=0x556ddc3275b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.862+0000 7f8a58897b80 1 --2- 172.21.15.114:0/293862024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 0x556ddc3275b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0a8da0 tx=0x7f8a3c007f90 comp rx=0 tx=0).stop 2024-05-27T07:40:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.862+0000 7f8a58897b80 1 -- 172.21.15.114:0/293862024 shutdown_connections 2024-05-27T07:40:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.862+0000 7f8a58897b80 1 --2- 172.21.15.114:0/293862024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 0x556ddc3275b0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.862+0000 7f8a58897b80 1 --2- 172.21.15.114:0/293862024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc1a7ec0 0x556ddc32cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.862+0000 7f8a58897b80 1 --2- 172.21.15.114:0/293862024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc32d0f0 0x556ddc326c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.862+0000 7f8a58897b80 1 -- 172.21.15.114:0/293862024 >> 172.21.15.114:0/293862024 conn(0x556ddc1aa7f0 msgr2=0x556ddc321e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.862+0000 7f8a58897b80 1 -- 172.21.15.114:0/293862024 shutdown_connections 2024-05-27T07:40:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.862+0000 7f8a58897b80 1 -- 172.21.15.114:0/293862024 wait complete. 2024-05-27T07:40:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.862+0000 7f8a58897b80 1 -- start start 2024-05-27T07:40:07.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a58897b80 1 Processor -- start 2024-05-27T07:40:07.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a58897b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 0x556ddc330900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a58897b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc3271a0 0x556ddc330e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a58897b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc32d0f0 0x556ddc331380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a58897b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556ddc14b020 con 0x556ddc1a7ec0 2024-05-27T07:40:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a58897b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556ddc1ace20 con 0x556ddc32d0f0 2024-05-27T07:40:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a58897b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556ddc333be0 con 0x556ddc3271a0 2024-05-27T07:40:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a58897b80 10 client.0 _ll_register_callbacks cb 0x556ddc1db630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 0x556ddc330900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 0x556ddc330900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41238/0 (socket says 172.21.15.114:41238) 2024-05-27T07:40:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a437fe700 1 -- 172.21.15.114:0/3322968906 learned_addr learned my addr 172.21.15.114:0/3322968906 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a43fff700 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc32d0f0 0x556ddc331380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a3bfff700 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc3271a0 0x556ddc330e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a437fe700 1 -- 172.21.15.114:0/3322968906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc3271a0 msgr2=0x556ddc330e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a437fe700 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc3271a0 0x556ddc330e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a437fe700 1 -- 172.21.15.114:0/3322968906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc32d0f0 msgr2=0x556ddc331380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a437fe700 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc32d0f0 0x556ddc331380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a437fe700 1 -- 172.21.15.114:0/3322968906 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ddc194950 con 0x556ddc1a7ec0 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.863+0000 7f8a3bfff700 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc3271a0 0x556ddc330e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a437fe700 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 0x556ddc330900 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8a340099c0 tx=0x7f8a3401a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9b3384ee3c524b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a22ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a22ffd700 1 -- 172.21.15.114:0/3322968906 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3400bd80 con 0x556ddc1a7ec0 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a22ffd700 1 -- 172.21.15.114:0/3322968906 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f8a3400f460 con 0x556ddc1a7ec0 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a58897b80 1 -- 172.21.15.114:0/3322968906 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a3c0a90a0 con 0x556ddc1a7ec0 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a22ffd700 1 -- 172.21.15.114:0/3322968906 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8a3403d050 con 0x556ddc1a7ec0 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a58897b80 10 client.15369 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a58897b80 1 -- 172.21.15.114:0/3322968906 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556ddc1fe7e0 con 0x556ddc1a7ec0 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a22ffd700 1 -- 172.21.15.114:0/3322968906 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a3c0a90a0 con 0x556ddc1a7ec0 2024-05-27T07:40:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a22ffd700 1 -- 172.21.15.114:0/3322968906 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8a34022070 con 0x556ddc1a7ec0 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a58897b80 10 client.15369 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a58897b80 10 client.15369 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a58897b80 20 client.15369 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a58897b80 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ddc340ba0 0x556ddc343050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a58897b80 4 client.15369 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.864+0000 7f8a58897b80 1 -- 172.21.15.114:0/3322968906 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ddc256d40 con 0x556ddc340ba0 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.865+0000 7f8a3bfff700 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ddc340ba0 0x556ddc343050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.865+0000 7f8a22ffd700 10 client.15369 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.865+0000 7f8a3bfff700 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ddc340ba0 0x556ddc343050 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c729eec4076093b server_cookie=a2f5a8df506fa1ce in_seq=0 out_seq=0 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a22ffd700 1 -- 172.21.15.114:0/3322968906 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556ddc256d40 con 0x556ddc340ba0 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a22ffd700 10 client.15369 handle_command_reply: tid=0 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a58897b80 1 client.15369 shutdown 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a58897b80 1 -- 172.21.15.114:0/3322968906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 msgr2=0x556ddc330900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a58897b80 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 0x556ddc330900 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8a340099c0 tx=0x7f8a3401a050 comp rx=0 tx=0).stop 2024-05-27T07:40:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a58897b80 1 -- 172.21.15.114:0/3322968906 shutdown_connections 2024-05-27T07:40:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a58897b80 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ddc340ba0 0x556ddc343050 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a58897b80 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc32d0f0 0x556ddc331380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a22ffd700 0 client.15369 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a58897b80 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 0x556ddc330900 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a58897b80 1 --2- 172.21.15.114:0/3322968906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc3271a0 0x556ddc330e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a58897b80 1 -- 172.21.15.114:0/3322968906 >> 172.21.15.114:0/3322968906 conn(0x556ddc1aa7f0 msgr2=0x556ddc325c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a58897b80 1 -- 172.21.15.114:0/3322968906 shutdown_connections 2024-05-27T07:40:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.866+0000 7f8a58897b80 1 -- 172.21.15.114:0/3322968906 wait complete. 2024-05-27T07:40:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.867+0000 7f8a58897b80 20 client.15369 trim_cache size 0 max 16384 2024-05-27T07:40:07.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a58897b80 1 Processor -- start 2024-05-27T07:40:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a58897b80 1 -- start start 2024-05-27T07:40:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a58897b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 0x556ddc326c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a58897b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc3271a0 0x556ddc3312f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a58897b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc32d0f0 0x556ddc33cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a58897b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441b0850 con 0x556ddc1a7ec0 2024-05-27T07:40:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a58897b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441b30d0 con 0x556ddc3271a0 2024-05-27T07:40:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a58897b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441b0ec0 con 0x556ddc32d0f0 2024-05-27T07:40:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 0x556ddc326c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 0x556ddc326c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41244/0 (socket says 172.21.15.114:41244) 2024-05-27T07:40:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a437fe700 1 -- 172.21.15.114:0/701770956 learned_addr learned my addr 172.21.15.114:0/701770956 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a437fe700 1 -- 172.21.15.114:0/701770956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc32d0f0 msgr2=0x556ddc33cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a3bfff700 1 --2- 172.21.15.114:0/701770956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc3271a0 0x556ddc3312f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a437fe700 1 --2- 172.21.15.114:0/701770956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc32d0f0 0x556ddc33cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a437fe700 1 -- 172.21.15.114:0/701770956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc3271a0 msgr2=0x556ddc3312f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:07.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a437fe700 1 --2- 172.21.15.114:0/701770956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc3271a0 0x556ddc3312f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a437fe700 1 -- 172.21.15.114:0/701770956 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ddc32ee70 con 0x556ddc1a7ec0 2024-05-27T07:40:07.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.887+0000 7f8a3bfff700 1 --2- 172.21.15.114:0/701770956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc3271a0 0x556ddc3312f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:07.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.888+0000 7f8a437fe700 1 --2- 172.21.15.114:0/701770956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 0x556ddc326c60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8a34003a10 tx=0x7f8a34009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc723a482cddb3f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:07.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.888+0000 7f8a397fa700 1 -- 172.21.15.114:0/701770956 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3400bd80 con 0x556ddc1a7ec0 2024-05-27T07:40:07.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.888+0000 7f8a397fa700 1 -- 172.21.15.114:0/701770956 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a34034070 con 0x556ddc1a7ec0 2024-05-27T07:40:07.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.888+0000 7f8a397fa700 1 -- 172.21.15.114:0/701770956 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a34039450 con 0x556ddc1a7ec0 2024-05-27T07:40:07.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.889+0000 7f8a58897b80 1 -- 172.21.15.114:0/701770956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 msgr2=0x556ddc326c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:07.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.889+0000 7f8a58897b80 1 --2- 172.21.15.114:0/701770956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 0x556ddc326c60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8a34003a10 tx=0x7f8a34009cd0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.889+0000 7f8a58897b80 1 -- 172.21.15.114:0/701770956 shutdown_connections 2024-05-27T07:40:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.889+0000 7f8a58897b80 1 --2- 172.21.15.114:0/701770956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc32d0f0 0x556ddc33cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.889+0000 7f8a58897b80 1 --2- 172.21.15.114:0/701770956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc1a7ec0 0x556ddc326c60 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.889+0000 7f8a58897b80 1 --2- 172.21.15.114:0/701770956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc3271a0 0x556ddc3312f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.889+0000 7f8a58897b80 1 -- 172.21.15.114:0/701770956 >> 172.21.15.114:0/701770956 conn(0x556ddc1aa7f0 msgr2=0x556ddc32f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.889+0000 7f8a58897b80 1 -- 172.21.15.114:0/701770956 shutdown_connections 2024-05-27T07:40:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.889+0000 7f8a58897b80 1 -- 172.21.15.114:0/701770956 wait complete. 2024-05-27T07:40:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.889+0000 7f8a58897b80 1 -- start start 2024-05-27T07:40:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.889+0000 7f8a58897b80 1 Processor -- start 2024-05-27T07:40:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a58897b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc1a7ec0 0x556ddc335e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a58897b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 0x556ddc336380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a58897b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc32d0f0 0x556ddc3368c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a58897b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441b1590 con 0x556ddc3271a0 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a58897b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441b2a00 con 0x556ddc32d0f0 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a58897b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441b1c60 con 0x556ddc1a7ec0 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a58897b80 10 client.0 _ll_register_callbacks cb 0x556ddc1db630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a3bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 0x556ddc336380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a3bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 0x556ddc336380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41254/0 (socket says 172.21.15.114:41254) 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a3bfff700 1 -- 172.21.15.114:0/3826525000 learned_addr learned my addr 172.21.15.114:0/3826525000 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a43fff700 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc32d0f0 0x556ddc3368c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a437fe700 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc1a7ec0 0x556ddc335e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a3bfff700 1 -- 172.21.15.114:0/3826525000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc1a7ec0 msgr2=0x556ddc335e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a3bfff700 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc1a7ec0 0x556ddc335e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a3bfff700 1 -- 172.21.15.114:0/3826525000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc32d0f0 msgr2=0x556ddc3368c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a3bfff700 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc32d0f0 0x556ddc3368c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a3bfff700 1 -- 172.21.15.114:0/3826525000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ddc3ab770 con 0x556ddc3271a0 2024-05-27T07:40:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a43fff700 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc32d0f0 0x556ddc3368c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:07.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a3bfff700 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 0x556ddc336380 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8a441074e0 tx=0x7f8a2c00dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341bdbf2d19ba5fe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:07.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a22ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:07.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a22ffd700 1 -- 172.21.15.114:0/3826525000 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c021070 con 0x556ddc3271a0 2024-05-27T07:40:07.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a22ffd700 1 -- 172.21.15.114:0/3826525000 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f8a2c003df0 con 0x556ddc3271a0 2024-05-27T07:40:07.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.890+0000 7f8a58897b80 1 -- 172.21.15.114:0/3826525000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a34034070 con 0x556ddc3271a0 2024-05-27T07:40:07.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.891+0000 7f8a22ffd700 1 -- 172.21.15.114:0/3826525000 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8a2c02d050 con 0x556ddc3271a0 2024-05-27T07:40:07.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.891+0000 7f8a58897b80 10 client.15375 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:07.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.891+0000 7f8a58897b80 1 -- 172.21.15.114:0/3826525000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556ddc3ac9c0 con 0x556ddc3271a0 2024-05-27T07:40:07.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.891+0000 7f8a22ffd700 1 -- 172.21.15.114:0/3826525000 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a34034070 con 0x556ddc3271a0 2024-05-27T07:40:07.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.891+0000 7f8a22ffd700 1 -- 172.21.15.114:0/3826525000 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8a2c011770 con 0x556ddc3271a0 2024-05-27T07:40:07.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.891+0000 7f8a58897b80 10 client.15375 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:07.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.891+0000 7f8a58897b80 10 client.15375 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.891+0000 7f8a58897b80 20 client.15375 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.891+0000 7f8a58897b80 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ddc3ad080 0x556ddc3af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.891+0000 7f8a58897b80 4 client.15375 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.891+0000 7f8a58897b80 1 -- 172.21.15.114:0/3826525000 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556ddc250520 con 0x556ddc3ad080 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.891+0000 7f8a437fe700 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ddc3ad080 0x556ddc3af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.892+0000 7f8a22ffd700 10 client.15375 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.892+0000 7f8a437fe700 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ddc3ad080 0x556ddc3af4a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b3328824a853d3f server_cookie=51bc25bae7b47d7 in_seq=0 out_seq=0 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.893+0000 7f8a22ffd700 1 -- 172.21.15.114:0/3826525000 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556ddc250520 con 0x556ddc3ad080 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.893+0000 7f8a22ffd700 10 client.15375 handle_command_reply: tid=0 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.893+0000 7f8a58897b80 1 client.15375 shutdown 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.893+0000 7f8a58897b80 1 -- 172.21.15.114:0/3826525000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 msgr2=0x556ddc336380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.893+0000 7f8a58897b80 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 0x556ddc336380 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8a441074e0 tx=0x7f8a2c00dda0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a58897b80 1 -- 172.21.15.114:0/3826525000 shutdown_connections 2024-05-27T07:40:07.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a58897b80 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ddc3ad080 0x556ddc3af4a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a58897b80 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ddc32d0f0 0x556ddc3368c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a58897b80 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ddc1a7ec0 0x556ddc335e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a58897b80 1 --2- 172.21.15.114:0/3826525000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ddc3271a0 0x556ddc336380 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a58897b80 1 -- 172.21.15.114:0/3826525000 >> 172.21.15.114:0/3826525000 conn(0x556ddc1aa7f0 msgr2=0x556ddc31b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:07.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a22ffd700 0 client.15375 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:07.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a58897b80 1 -- 172.21.15.114:0/3826525000 shutdown_connections 2024-05-27T07:40:07.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a58897b80 1 -- 172.21.15.114:0/3826525000 wait complete. 2024-05-27T07:40:07.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a58897b80 20 client.15375 trim_cache size 0 max 16384 2024-05-27T07:40:07.898 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:40:07.898 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:40:07.899 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "2f26bcac-8fbb-4376-b819-dfdfb08fe7bc", 2024-05-27T07:40:07.899 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:40:07.899 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:40:07.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a49e82700 1 -- 172.21.15.114:0/1681210965 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8a300b2b10 msgr2=0x7f8a300b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:07.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a49e82700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8a300b2b10 0x7f8a300b5020 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009990 tx=0x7f8a340036e0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a49e82700 1 -- 172.21.15.114:0/1681210965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a44109840 msgr2=0x7f8a4419f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:07.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.894+0000 7f8a49e82700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a44109840 0x7f8a4419f630 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c007ee0 tx=0x7f8a3c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.895+0000 7f8a49e82700 1 -- 172.21.15.114:0/1681210965 shutdown_connections 2024-05-27T07:40:07.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.895+0000 7f8a49e82700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a44109840 0x7f8a4419f630 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.895+0000 7f8a49e82700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8a300b2b10 0x7f8a300b5020 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.895+0000 7f8a49e82700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a441045c0 0x7f8a4419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.895+0000 7f8a49e82700 1 --2- 172.21.15.114:0/1681210965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a44108ef0 0x7f8a4419b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:07.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.895+0000 7f8a49e82700 1 -- 172.21.15.114:0/1681210965 >> 172.21.15.114:0/1681210965 conn(0x7f8a440ffe90 msgr2=0x7f8a44102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:07.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.895+0000 7f8a49e82700 1 -- 172.21.15.114:0/1681210965 shutdown_connections 2024-05-27T07:40:07.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:07.895+0000 7f8a49e82700 1 -- 172.21.15.114:0/1681210965 wait complete. 2024-05-27T07:40:08.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:08 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:40:08.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:08 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:08.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:08 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:40:08.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:08 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:08.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:08 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:40:08.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:08 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:08.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:08 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:40:08.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:08 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:08.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:08 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:40:08.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:08 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:08.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:08 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:40:08.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:08 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:08.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:40:08.938 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:09.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.287+0000 7fcd365c5640 1 -- 172.21.15.114:0/3779239353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3010b7e0 msgr2=0x7fcd3010dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.287+0000 7fcd365c5640 1 --2- 172.21.15.114:0/3779239353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3010b7e0 0x7fcd3010dc00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009900 tx=0x7fcd2402a410 comp rx=0 tx=0).stop 2024-05-27T07:40:09.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.287+0000 7fcd365c5640 1 -- 172.21.15.114:0/3779239353 shutdown_connections 2024-05-27T07:40:09.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.287+0000 7fcd365c5640 1 --2- 172.21.15.114:0/3779239353 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd3010e1d0 0x7fcd301106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.287+0000 7fcd365c5640 1 --2- 172.21.15.114:0/3779239353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3010b7e0 0x7fcd3010dc00 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.287+0000 7fcd365c5640 1 --2- 172.21.15.114:0/3779239353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd30104d80 0x7fcd30105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.287+0000 7fcd365c5640 1 -- 172.21.15.114:0/3779239353 >> 172.21.15.114:0/3779239353 conn(0x7fcd300fe380 msgr2=0x7fcd301007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.287+0000 7fcd365c5640 1 -- 172.21.15.114:0/3779239353 shutdown_connections 2024-05-27T07:40:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.287+0000 7fcd365c5640 1 -- 172.21.15.114:0/3779239353 wait complete. 2024-05-27T07:40:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.288+0000 7fcd365c5640 1 Processor -- start 2024-05-27T07:40:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.288+0000 7fcd365c5640 1 -- start start 2024-05-27T07:40:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.288+0000 7fcd365c5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd30104d80 0x7fcd301a0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd365c5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3010b7e0 0x7fcd301a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd365c5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd3010e1d0 0x7fcd3019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd365c5640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcd30113e30 con 0x7fcd3010b7e0 2024-05-27T07:40:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd365c5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcd30113cb0 con 0x7fcd3010e1d0 2024-05-27T07:40:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd365c5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcd30113fb0 con 0x7fcd30104d80 2024-05-27T07:40:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd2f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3010b7e0 0x7fcd301a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd2f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3010b7e0 0x7fcd301a0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41272/0 (socket says 172.21.15.114:41272) 2024-05-27T07:40:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd2f7fe640 1 -- 172.21.15.114:0/3000548779 learned_addr learned my addr 172.21.15.114:0/3000548779 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd2ffff640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd30104d80 0x7fcd301a0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd34b62640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd3010e1d0 0x7fcd3019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd2f7fe640 1 -- 172.21.15.114:0/3000548779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd30104d80 msgr2=0x7fcd301a0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd2f7fe640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd30104d80 0x7fcd301a0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd2f7fe640 1 -- 172.21.15.114:0/3000548779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd3010e1d0 msgr2=0x7fcd3019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd2f7fe640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd3010e1d0 0x7fcd3019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd2f7fe640 1 -- 172.21.15.114:0/3000548779 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd3019ae00 con 0x7fcd3010b7e0 2024-05-27T07:40:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd2f7fe640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3010b7e0 0x7fcd301a0a90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcd24005940 tx=0x7fcd240042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd2d7fa640 1 -- 172.21.15.114:0/3000548779 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2403f030 con 0x7fcd3010b7e0 2024-05-27T07:40:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd2d7fa640 1 -- 172.21.15.114:0/3000548779 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd24003cb0 con 0x7fcd3010b7e0 2024-05-27T07:40:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd365c5640 1 -- 172.21.15.114:0/3000548779 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd3019b030 con 0x7fcd3010b7e0 2024-05-27T07:40:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd2d7fa640 1 -- 172.21.15.114:0/3000548779 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd24035070 con 0x7fcd3010b7e0 2024-05-27T07:40:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.289+0000 7fcd365c5640 1 -- 172.21.15.114:0/3000548779 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd3019b520 con 0x7fcd3010b7e0 2024-05-27T07:40:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd2d7fa640 1 -- 172.21.15.114:0/3000548779 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7fcd24042030 con 0x7fcd3010b7e0 2024-05-27T07:40:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd3d477740 1 Processor -- start 2024-05-27T07:40:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd3d477740 1 -- start start 2024-05-27T07:40:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd3d477740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad7671c1b0 0x55ad76abd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd3d477740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 0x55ad76abd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd3d477740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766fb4d0 0x55ad76abde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd3d477740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ad76a2fc30 con 0x55ad766f32f0 2024-05-27T07:40:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd3d477740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ad76a1f7d0 con 0x55ad7671c1b0 2024-05-27T07:40:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd3d477740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ad76a23810 con 0x55ad766fb4d0 2024-05-27T07:40:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd2ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 0x55ad76abd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd2ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 0x55ad76abd910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41286/0 (socket says 172.21.15.114:41286) 2024-05-27T07:40:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd2ffff640 1 -- 172.21.15.114:0/2537218824 learned_addr learned my addr 172.21.15.114:0/2537218824 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd34b62640 1 --2- 172.21.15.114:0/2537218824 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad7671c1b0 0x55ad76abd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.292+0000 7fcd2f7fe640 1 --2- 172.21.15.114:0/2537218824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766fb4d0 0x55ad76abde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.293+0000 7fcd2d7fa640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcd180b2a70 0x7fcd180b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.293+0000 7fcd2ffff640 1 -- 172.21.15.114:0/2537218824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766fb4d0 msgr2=0x55ad76abde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.293+0000 7fcd2ffff640 1 --2- 172.21.15.114:0/2537218824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766fb4d0 0x55ad76abde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.293+0000 7fcd2ffff640 1 -- 172.21.15.114:0/2537218824 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad7671c1b0 msgr2=0x55ad76abd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.293+0000 7fcd2ffff640 1 --2- 172.21.15.114:0/2537218824 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad7671c1b0 0x55ad76abd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.293+0000 7fcd2ffff640 1 -- 172.21.15.114:0/2537218824 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad76734210 con 0x55ad766f32f0 2024-05-27T07:40:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.293+0000 7fcd34b62640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcd180b2a70 0x7fcd180b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.293+0000 7fcd2d7fa640 1 -- 172.21.15.114:0/3000548779 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fcd241033e0 con 0x7fcd3010b7e0 2024-05-27T07:40:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.293+0000 7fcd2ffff640 1 --2- 172.21.15.114:0/2537218824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 0x55ad76abd910 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000ac10 tx=0x7fcd2000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcba9f10cd5b20d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.293+0000 7fcd167fc640 1 -- 172.21.15.114:0/2537218824 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2001d070 con 0x55ad766f32f0 2024-05-27T07:40:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.293+0000 7fcd167fc640 1 -- 172.21.15.114:0/2537218824 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd20018af0 con 0x55ad766f32f0 2024-05-27T07:40:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.293+0000 7fcd167fc640 1 -- 172.21.15.114:0/2537218824 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd20010a20 con 0x55ad766f32f0 2024-05-27T07:40:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.294+0000 7fcd3d477740 1 -- 172.21.15.114:0/2537218824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 msgr2=0x55ad76abd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.294+0000 7fcd3d477740 1 --2- 172.21.15.114:0/2537218824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 0x55ad76abd910 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000ac10 tx=0x7fcd2000f790 comp rx=0 tx=0).stop 2024-05-27T07:40:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.294+0000 7fcd34b62640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcd180b2a70 0x7fcd180b4f80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x55ad76aba540 tx=0x7fcd28004a40 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.294+0000 7fcd3d477740 1 -- 172.21.15.114:0/2537218824 shutdown_connections 2024-05-27T07:40:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.294+0000 7fcd3d477740 1 --2- 172.21.15.114:0/2537218824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766fb4d0 0x55ad76abde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.294+0000 7fcd3d477740 1 --2- 172.21.15.114:0/2537218824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 0x55ad76abd910 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.294+0000 7fcd3d477740 1 --2- 172.21.15.114:0/2537218824 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad7671c1b0 0x55ad76abd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.294+0000 7fcd3d477740 1 -- 172.21.15.114:0/2537218824 >> 172.21.15.114:0/2537218824 conn(0x55ad76710300 msgr2=0x55ad76ab9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.294+0000 7fcd3d477740 1 -- 172.21.15.114:0/2537218824 shutdown_connections 2024-05-27T07:40:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.294+0000 7fcd3d477740 1 -- 172.21.15.114:0/2537218824 wait complete. 2024-05-27T07:40:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.294+0000 7fcd3d477740 1 -- start start 2024-05-27T07:40:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd3d477740 1 Processor -- start 2024-05-27T07:40:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd3d477740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 0x55ad76ac9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd3d477740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766fb4d0 0x55ad76ac9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd3d477740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad7671c1b0 0x55ad76ac78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd3d477740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ad76705790 con 0x55ad766f32f0 2024-05-27T07:40:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd3d477740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ad769db890 con 0x55ad766fb4d0 2024-05-27T07:40:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd3d477740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ad76aca500 con 0x55ad7671c1b0 2024-05-27T07:40:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd3d477740 10 client.0 _ll_register_callbacks cb 0x55ad76733a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd2ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 0x55ad76ac9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd2ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 0x55ad76ac9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41302/0 (socket says 172.21.15.114:41302) 2024-05-27T07:40:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd2ffff640 1 -- 172.21.15.114:0/975368550 learned_addr learned my addr 172.21.15.114:0/975368550 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd34b62640 1 --2- 172.21.15.114:0/975368550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad7671c1b0 0x55ad76ac78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd2f7fe640 1 --2- 172.21.15.114:0/975368550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766fb4d0 0x55ad76ac9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd2ffff640 1 -- 172.21.15.114:0/975368550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad7671c1b0 msgr2=0x55ad76ac78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd2ffff640 1 --2- 172.21.15.114:0/975368550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad7671c1b0 0x55ad76ac78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd2ffff640 1 -- 172.21.15.114:0/975368550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766fb4d0 msgr2=0x55ad76ac9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd2ffff640 1 --2- 172.21.15.114:0/975368550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766fb4d0 0x55ad76ac9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd2ffff640 1 -- 172.21.15.114:0/975368550 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad76800620 con 0x55ad766f32f0 2024-05-27T07:40:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.295+0000 7fcd2ffff640 1 --2- 172.21.15.114:0/975368550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 0x55ad76ac9410 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fcd2001b040 tx=0x7fcd2000a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d947f7cca29ef7a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd0f7fe640 1 -- 172.21.15.114:0/975368550 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd20027020 con 0x55ad766f32f0 2024-05-27T07:40:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd0f7fe640 1 -- 172.21.15.114:0/975368550 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fcd20005040 con 0x55ad766f32f0 2024-05-27T07:40:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd3d477740 1 -- 172.21.15.114:0/975368550 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd20018af0 con 0x55ad766f32f0 2024-05-27T07:40:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd0f7fe640 1 -- 172.21.15.114:0/975368550 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcd20032050 con 0x55ad766f32f0 2024-05-27T07:40:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd3d477740 10 client.15387 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd3d477740 1 -- 172.21.15.114:0/975368550 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ad76ad6810 con 0x55ad766f32f0 2024-05-27T07:40:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd0f7fe640 1 -- 172.21.15.114:0/975368550 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd20018af0 con 0x55ad766f32f0 2024-05-27T07:40:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd0f7fe640 1 -- 172.21.15.114:0/975368550 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fcd20010cc0 con 0x55ad766f32f0 2024-05-27T07:40:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd3d477740 10 client.15387 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd3d477740 10 client.15387 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd3d477740 20 client.15387 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd3d477740 1 --2- 172.21.15.114:0/975368550 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ad766dad80 0x55ad76ad9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd3d477740 4 client.15387 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.296+0000 7fcd3d477740 1 -- 172.21.15.114:0/975368550 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad76a31b80 con 0x55ad766dad80 2024-05-27T07:40:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.297+0000 7fcd2f7fe640 1 --2- 172.21.15.114:0/975368550 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ad766dad80 0x55ad76ad9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.297+0000 7fcd0f7fe640 10 client.15387 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.297+0000 7fcd2f7fe640 1 --2- 172.21.15.114:0/975368550 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ad766dad80 0x55ad76ad9c20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aab11b13780f8a6a server_cookie=6ebdb65397252879 in_seq=0 out_seq=0 2024-05-27T07:40:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd0f7fe640 1 -- 172.21.15.114:0/975368550 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ad76a31b80 con 0x55ad766dad80 2024-05-27T07:40:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd0f7fe640 10 client.15387 handle_command_reply: tid=0 2024-05-27T07:40:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd3d477740 1 client.15387 shutdown 2024-05-27T07:40:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd3d477740 1 -- 172.21.15.114:0/975368550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 msgr2=0x55ad76ac9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd3d477740 1 --2- 172.21.15.114:0/975368550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 0x55ad76ac9410 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fcd2001b040 tx=0x7fcd2000a750 comp rx=0 tx=0).stop 2024-05-27T07:40:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd3d477740 1 -- 172.21.15.114:0/975368550 shutdown_connections 2024-05-27T07:40:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd3d477740 1 --2- 172.21.15.114:0/975368550 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ad766dad80 0x55ad76ad9c20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd3d477740 1 --2- 172.21.15.114:0/975368550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad7671c1b0 0x55ad76ac78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd3d477740 1 --2- 172.21.15.114:0/975368550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766fb4d0 0x55ad76ac9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd3d477740 1 --2- 172.21.15.114:0/975368550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad766f32f0 0x55ad76ac9410 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd0f7fe640 0 client.15387 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd3d477740 1 -- 172.21.15.114:0/975368550 >> 172.21.15.114:0/975368550 conn(0x55ad76710300 msgr2=0x55ad76abf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd3d477740 1 -- 172.21.15.114:0/975368550 shutdown_connections 2024-05-27T07:40:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.298+0000 7fcd3d477740 1 -- 172.21.15.114:0/975368550 wait complete. 2024-05-27T07:40:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.299+0000 7fcd3d477740 20 client.15387 trim_cache size 0 max 16384 2024-05-27T07:40:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.313+0000 7fcd3d477740 1 Processor -- start 2024-05-27T07:40:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.313+0000 7fcd3d477740 1 -- start start 2024-05-27T07:40:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd3d477740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 0x55ad76ac15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd3d477740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766f32f0 0x55ad76adcc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd3d477740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766fb4d0 0x55ad76add190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd3d477740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcd301af300 con 0x55ad76710300 2024-05-27T07:40:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd3d477740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcd301b1b80 con 0x55ad766f32f0 2024-05-27T07:40:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd3d477740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcd301af970 con 0x55ad766fb4d0 2024-05-27T07:40:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd2ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 0x55ad76ac15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd2ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 0x55ad76ac15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41318/0 (socket says 172.21.15.114:41318) 2024-05-27T07:40:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd2ffff640 1 -- 172.21.15.114:0/4135192705 learned_addr learned my addr 172.21.15.114:0/4135192705 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd34b62640 1 --2- 172.21.15.114:0/4135192705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766fb4d0 0x55ad76add190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd2f7fe640 1 --2- 172.21.15.114:0/4135192705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766f32f0 0x55ad76adcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd2ffff640 1 -- 172.21.15.114:0/4135192705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766fb4d0 msgr2=0x55ad76add190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd2ffff640 1 --2- 172.21.15.114:0/4135192705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766fb4d0 0x55ad76add190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd2ffff640 1 -- 172.21.15.114:0/4135192705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766f32f0 msgr2=0x55ad76adcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd2ffff640 1 --2- 172.21.15.114:0/4135192705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766f32f0 0x55ad76adcc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd2ffff640 1 -- 172.21.15.114:0/4135192705 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad76ac1ae0 con 0x55ad76710300 2024-05-27T07:40:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd2f7fe640 1 --2- 172.21.15.114:0/4135192705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766f32f0 0x55ad76adcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd2ffff640 1 --2- 172.21.15.114:0/4135192705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 0x55ad76ac15a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000f7d0 tx=0x7fcd2000aae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b796a92b2a37c55 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd15ffb640 1 -- 172.21.15.114:0/4135192705 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2003b070 con 0x55ad76710300 2024-05-27T07:40:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd15ffb640 1 -- 172.21.15.114:0/4135192705 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd20018850 con 0x55ad76710300 2024-05-27T07:40:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.314+0000 7fcd15ffb640 1 -- 172.21.15.114:0/4135192705 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd20027040 con 0x55ad76710300 2024-05-27T07:40:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.315+0000 7fcd3d477740 1 -- 172.21.15.114:0/4135192705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 msgr2=0x55ad76ac15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.315+0000 7fcd3d477740 1 --2- 172.21.15.114:0/4135192705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 0x55ad76ac15a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000f7d0 tx=0x7fcd2000aae0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.315+0000 7fcd3d477740 1 -- 172.21.15.114:0/4135192705 shutdown_connections 2024-05-27T07:40:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.315+0000 7fcd3d477740 1 --2- 172.21.15.114:0/4135192705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766fb4d0 0x55ad76add190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.315+0000 7fcd3d477740 1 --2- 172.21.15.114:0/4135192705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766f32f0 0x55ad76adcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.315+0000 7fcd3d477740 1 --2- 172.21.15.114:0/4135192705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 0x55ad76ac15a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.315+0000 7fcd3d477740 1 -- 172.21.15.114:0/4135192705 >> 172.21.15.114:0/4135192705 conn(0x55ad76ac3260 msgr2=0x55ad76ac0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.315+0000 7fcd3d477740 1 -- 172.21.15.114:0/4135192705 shutdown_connections 2024-05-27T07:40:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.315+0000 7fcd3d477740 1 -- 172.21.15.114:0/4135192705 wait complete. 2024-05-27T07:40:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.315+0000 7fcd3d477740 1 -- start start 2024-05-27T07:40:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd3d477740 1 Processor -- start 2024-05-27T07:40:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd3d477740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766f32f0 0x55ad76b3a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd3d477740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766fb4d0 0x55ad76b3af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd3d477740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 0x55ad76adc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd3d477740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcd301b0040 con 0x55ad76710300 2024-05-27T07:40:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd3d477740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcd301b14b0 con 0x55ad766fb4d0 2024-05-27T07:40:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd3d477740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcd301b0710 con 0x55ad766f32f0 2024-05-27T07:40:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd3d477740 10 client.0 _ll_register_callbacks cb 0x7fcd2001b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd34b62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 0x55ad76adc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd34b62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 0x55ad76adc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41334/0 (socket says 172.21.15.114:41334) 2024-05-27T07:40:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd34b62640 1 -- 172.21.15.114:0/4055345587 learned_addr learned my addr 172.21.15.114:0/4055345587 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd2ffff640 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766f32f0 0x55ad76b3a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd2f7fe640 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766fb4d0 0x55ad76b3af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd34b62640 1 -- 172.21.15.114:0/4055345587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766f32f0 msgr2=0x55ad76b3a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd34b62640 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766f32f0 0x55ad76b3a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd34b62640 1 -- 172.21.15.114:0/4055345587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766fb4d0 msgr2=0x55ad76b3af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd34b62640 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766fb4d0 0x55ad76b3af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd34b62640 1 -- 172.21.15.114:0/4055345587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad76adcdc0 con 0x55ad76710300 2024-05-27T07:40:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.316+0000 7fcd2f7fe640 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766fb4d0 0x55ad76b3af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd34b62640 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 0x55ad76adc7d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcd28010a20 tx=0x7fcd28004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61bb95d83de884ee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd0f7fe640 1 -- 172.21.15.114:0/4055345587 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28003970 con 0x55ad76710300 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd0f7fe640 1 -- 172.21.15.114:0/4055345587 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fcd28003b10 con 0x55ad76710300 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd3d477740 1 -- 172.21.15.114:0/4055345587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd20018850 con 0x55ad76710300 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd0f7fe640 1 -- 172.21.15.114:0/4055345587 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcd28034050 con 0x55ad76710300 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd3d477740 10 client.15393 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd3d477740 1 -- 172.21.15.114:0/4055345587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ad76b42050 con 0x55ad76710300 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd0f7fe640 1 -- 172.21.15.114:0/4055345587 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd20018850 con 0x55ad76710300 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd0f7fe640 1 -- 172.21.15.114:0/4055345587 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fcd2800f040 con 0x55ad76710300 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd3d477740 10 client.15393 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd3d477740 10 client.15393 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd3d477740 20 client.15393 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd3d477740 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ad7671c1b0 0x55ad76b44890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd3d477740 4 client.15393 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:40:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.317+0000 7fcd3d477740 1 -- 172.21.15.114:0/4055345587 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ad76a2b360 con 0x55ad7671c1b0 2024-05-27T07:40:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.318+0000 7fcd2ffff640 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ad7671c1b0 0x55ad76b44890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.318+0000 7fcd2ffff640 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ad7671c1b0 0x55ad76b44890 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ea47718c47e0260 server_cookie=4078886a50a6f55e in_seq=0 out_seq=0 2024-05-27T07:40:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.318+0000 7fcd0f7fe640 10 client.15393 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.331+0000 7fcd0f7fe640 1 -- 172.21.15.114:0/4055345587 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55ad76a2b360 con 0x55ad7671c1b0 2024-05-27T07:40:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.331+0000 7fcd0f7fe640 10 client.15393 handle_command_reply: tid=0 2024-05-27T07:40:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.331+0000 7fcd3d477740 1 client.15393 shutdown 2024-05-27T07:40:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.331+0000 7fcd3d477740 1 -- 172.21.15.114:0/4055345587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 msgr2=0x55ad76adc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.331+0000 7fcd3d477740 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 0x55ad76adc7d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcd28010a20 tx=0x7fcd28004ab0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.331+0000 7fcd3d477740 1 -- 172.21.15.114:0/4055345587 shutdown_connections 2024-05-27T07:40:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.331+0000 7fcd3d477740 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ad7671c1b0 0x55ad76b44890 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd3d477740 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ad76710300 0x55ad76adc7d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd3d477740 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ad766fb4d0 0x55ad76b3af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd0f7fe640 0 client.15393 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd3d477740 1 --2- 172.21.15.114:0/4055345587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad766f32f0 0x55ad76b3a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd3d477740 1 -- 172.21.15.114:0/4055345587 >> 172.21.15.114:0/4055345587 conn(0x55ad76ac3260 msgr2=0x55ad76ac8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd3d477740 1 -- 172.21.15.114:0/4055345587 shutdown_connections 2024-05-27T07:40:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd3d477740 1 -- 172.21.15.114:0/4055345587 wait complete. 2024-05-27T07:40:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd3d477740 20 client.15393 trim_cache size 0 max 16384 2024-05-27T07:40:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:40:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:40:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:40:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd365c5640 1 -- 172.21.15.114:0/3000548779 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcd180b2a70 msgr2=0x7fcd180b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd365c5640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcd180b2a70 0x7fcd180b4f80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x55ad76aba540 tx=0x7fcd28004a40 comp rx=0 tx=0).stop 2024-05-27T07:40:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd365c5640 1 -- 172.21.15.114:0/3000548779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3010b7e0 msgr2=0x7fcd301a0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd365c5640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3010b7e0 0x7fcd301a0a90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcd24005940 tx=0x7fcd240042f0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd365c5640 1 -- 172.21.15.114:0/3000548779 shutdown_connections 2024-05-27T07:40:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd365c5640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcd180b2a70 0x7fcd180b4f80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd365c5640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd3010e1d0 0x7fcd3019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd365c5640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3010b7e0 0x7fcd301a0a90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd365c5640 1 --2- 172.21.15.114:0/3000548779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd30104d80 0x7fcd301a0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd365c5640 1 -- 172.21.15.114:0/3000548779 >> 172.21.15.114:0/3000548779 conn(0x7fcd300fe380 msgr2=0x7fcd3010e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.332+0000 7fcd365c5640 1 -- 172.21.15.114:0/3000548779 shutdown_connections 2024-05-27T07:40:09.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:09.333+0000 7fcd365c5640 1 -- 172.21.15.114:0/3000548779 wait complete. 2024-05-27T07:40:09.334 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:09 smithi114 ceph-mon[32038]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 45 KiB/s wr, 34 op/s 2024-05-27T07:40:09.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:09 smithi163 ceph-mon[36451]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 45 KiB/s wr, 34 op/s 2024-05-27T07:40:09.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:09 smithi186 ceph-mon[36647]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 45 KiB/s wr, 34 op/s 2024-05-27T07:40:10.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.570+0000 7fb3b1664700 1 -- 172.21.15.114:0/411150653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3ac103540 msgr2=0x7fb3ac103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:10.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.570+0000 7fb3b1664700 1 --2- 172.21.15.114:0/411150653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3ac103540 0x7fb3ac103950 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb39c0099c0 tx=0x7fb39c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.570+0000 7fb3b1664700 1 -- 172.21.15.114:0/411150653 shutdown_connections 2024-05-27T07:40:10.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.570+0000 7fb3b1664700 1 --2- 172.21.15.114:0/411150653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3ac108770 0x7fb3ac1100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.570+0000 7fb3b1664700 1 --2- 172.21.15.114:0/411150653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3ac103540 0x7fb3ac103950 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.570+0000 7fb3b1664700 1 --2- 172.21.15.114:0/411150653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac107e20 0x7fb3ac108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.570+0000 7fb3b1664700 1 -- 172.21.15.114:0/411150653 >> 172.21.15.114:0/411150653 conn(0x7fb3ac078df0 msgr2=0x7fb3ac079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:10.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.570+0000 7fb3b1664700 1 -- 172.21.15.114:0/411150653 shutdown_connections 2024-05-27T07:40:10.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.570+0000 7fb3b1664700 1 -- 172.21.15.114:0/411150653 wait complete. 2024-05-27T07:40:10.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.571+0000 7fb3b1664700 1 Processor -- start 2024-05-27T07:40:10.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.571+0000 7fb3b1664700 1 -- start start 2024-05-27T07:40:10.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.571+0000 7fb3b1664700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3ac103540 0x7fb3ac19f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.571+0000 7fb3b1664700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3ac107e20 0x7fb3ac19fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.571+0000 7fb3b1664700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac108770 0x7fb3ac199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.571+0000 7fb3b1664700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1133a0 con 0x7fb3ac107e20 2024-05-27T07:40:10.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.571+0000 7fb3b1664700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac113220 con 0x7fb3ac103540 2024-05-27T07:40:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.571+0000 7fb3b1664700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac113520 con 0x7fb3ac108770 2024-05-27T07:40:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.571+0000 7fb3a3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3ac107e20 0x7fb3ac19fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.571+0000 7fb3a3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3ac107e20 0x7fb3ac19fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41350/0 (socket says 172.21.15.114:41350) 2024-05-27T07:40:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3a3fff700 1 -- 172.21.15.114:0/2307922338 learned_addr learned my addr 172.21.15.114:0/2307922338 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3ab7fe700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac108770 0x7fb3ac199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3ac103540 0x7fb3ac19f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3a3fff700 1 -- 172.21.15.114:0/2307922338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac108770 msgr2=0x7fb3ac199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3a3fff700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac108770 0x7fb3ac199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3a3fff700 1 -- 172.21.15.114:0/2307922338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3ac103540 msgr2=0x7fb3ac19f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3a3fff700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3ac103540 0x7fb3ac19f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3a3fff700 1 -- 172.21.15.114:0/2307922338 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ac19a1e0 con 0x7fb3ac107e20 2024-05-27T07:40:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3ac103540 0x7fb3ac19f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3a3fff700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3ac107e20 0x7fb3ac19fe20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb39400d980 tx=0x7fb39400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:10.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3a8ff9700 1 -- 172.21.15.114:0/2307922338 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb394017070 con 0x7fb3ac107e20 2024-05-27T07:40:10.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3a8ff9700 1 -- 172.21.15.114:0/2307922338 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb394009de0 con 0x7fb3ac107e20 2024-05-27T07:40:10.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3b1664700 1 -- 172.21.15.114:0/2307922338 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3ac19a4d0 con 0x7fb3ac107e20 2024-05-27T07:40:10.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3a8ff9700 1 -- 172.21.15.114:0/2307922338 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39400f3c0 con 0x7fb3ac107e20 2024-05-27T07:40:10.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.572+0000 7fb3b1664700 1 -- 172.21.15.114:0/2307922338 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3ac1a6770 con 0x7fb3ac107e20 2024-05-27T07:40:10.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.576+0000 7fb3a8ff9700 1 -- 172.21.15.114:0/2307922338 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7fb39400f560 con 0x7fb3ac107e20 2024-05-27T07:40:10.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3c0079b80 1 Processor -- start 2024-05-27T07:40:10.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3c0079b80 1 -- start start 2024-05-27T07:40:10.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3a8ff9700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb3980b2b90 0x7fb3980b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3ab7fe700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb3980b2b90 0x7fb3980b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3a8ff9700 1 -- 172.21.15.114:0/2307922338 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fb3940e2790 con 0x7fb3ac107e20 2024-05-27T07:40:10.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3c0079b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 0x55a5277abcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3c0079b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a5277ac210 0x55a5277b6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3c0079b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 0x55a5277b8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3c0079b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a5276d9c90 con 0x55a52762ce40 2024-05-27T07:40:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3c0079b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a5276c9830 con 0x55a5277ac210 2024-05-27T07:40:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3c0079b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a5276cd870 con 0x55a5277ac620 2024-05-27T07:40:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3aaffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 0x55a5277abcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3aaffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 0x55a5277abcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41358/0 (socket says 172.21.15.114:41358) 2024-05-27T07:40:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3aaffd700 1 -- 172.21.15.114:0/1607307780 learned_addr learned my addr 172.21.15.114:0/1607307780 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3ab7fe700 1 --2- 172.21.15.114:0/1607307780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 0x55a5277b8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.577+0000 7fb3a3fff700 1 --2- 172.21.15.114:0/1607307780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a5277ac210 0x55a5277b6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.578+0000 7fb3ab7fe700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb3980b2b90 0x7fb3980b50a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4003ee0 tx=0x7fb3a4009100 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.578+0000 7fb3aaffd700 1 -- 172.21.15.114:0/1607307780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 msgr2=0x55a5277b8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.578+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/1607307780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 0x55a5277b8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.578+0000 7fb3aaffd700 1 -- 172.21.15.114:0/1607307780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a5277ac210 msgr2=0x55a5277b6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.578+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/1607307780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a5277ac210 0x55a5277b6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.578+0000 7fb3aaffd700 1 -- 172.21.15.114:0/1607307780 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5276198d0 con 0x55a52762ce40 2024-05-27T07:40:10.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.578+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/1607307780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 0x55a5277abcd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb39c0099c0 tx=0x7fb39c009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b0f6edb7832246c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:10.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.578+0000 7fb3a17fa700 1 -- 172.21.15.114:0/1607307780 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c003a60 con 0x55a52762ce40 2024-05-27T07:40:10.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.578+0000 7fb3a17fa700 1 -- 172.21.15.114:0/1607307780 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb39c005580 con 0x55a52762ce40 2024-05-27T07:40:10.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.578+0000 7fb3a17fa700 1 -- 172.21.15.114:0/1607307780 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c00bc70 con 0x55a52762ce40 2024-05-27T07:40:10.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.579+0000 7fb3c0079b80 1 -- 172.21.15.114:0/1607307780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 msgr2=0x55a5277abcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.579+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/1607307780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 0x55a5277abcd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb39c0099c0 tx=0x7fb39c009510 comp rx=0 tx=0).stop 2024-05-27T07:40:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.579+0000 7fb3c0079b80 1 -- 172.21.15.114:0/1607307780 shutdown_connections 2024-05-27T07:40:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.579+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/1607307780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 0x55a5277b8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.579+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/1607307780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 0x55a5277abcd0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.579+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/1607307780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a5277ac210 0x55a5277b6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.579+0000 7fb3c0079b80 1 -- 172.21.15.114:0/1607307780 >> 172.21.15.114:0/1607307780 conn(0x55a527687af0 msgr2=0x55a5277a8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.579+0000 7fb3c0079b80 1 -- 172.21.15.114:0/1607307780 shutdown_connections 2024-05-27T07:40:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.579+0000 7fb3c0079b80 1 -- 172.21.15.114:0/1607307780 wait complete. 2024-05-27T07:40:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3c0079b80 1 -- start start 2024-05-27T07:40:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3c0079b80 1 Processor -- start 2024-05-27T07:40:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3c0079b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52762ce40 0x55a5277b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3c0079b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac210 0x55a5277b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3c0079b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac620 0x55a5277b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3c0079b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a5275cffa0 con 0x55a5277ac620 2024-05-27T07:40:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3c0079b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a52766b7c0 con 0x55a52762ce40 2024-05-27T07:40:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3c0079b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a5277b9810 con 0x55a5277ac210 2024-05-27T07:40:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3c0079b80 10 client.0 _ll_register_callbacks cb 0x55a527664920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3ab7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac620 0x55a5277b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3ab7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac620 0x55a5277b6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41374/0 (socket says 172.21.15.114:41374) 2024-05-27T07:40:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3ab7fe700 1 -- 172.21.15.114:0/1517557162 learned_addr learned my addr 172.21.15.114:0/1517557162 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3a3fff700 1 --2- 172.21.15.114:0/1517557162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac210 0x55a5277b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/1517557162 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52762ce40 0x55a5277b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3ab7fe700 1 -- 172.21.15.114:0/1517557162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac210 msgr2=0x55a5277b69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3ab7fe700 1 --2- 172.21.15.114:0/1517557162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac210 0x55a5277b69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3ab7fe700 1 -- 172.21.15.114:0/1517557162 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52762ce40 msgr2=0x55a5277b64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.580+0000 7fb3ab7fe700 1 --2- 172.21.15.114:0/1517557162 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52762ce40 0x55a5277b64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3ab7fe700 1 -- 172.21.15.114:0/1517557162 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a527668690 con 0x55a5277ac620 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3ab7fe700 1 --2- 172.21.15.114:0/1517557162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac620 0x55a5277b6f20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4008ac0 tx=0x7fb3a4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d46799e06acd18 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3927fc700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3927fc700 1 -- 172.21.15.114:0/1517557162 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4026070 con 0x55a5277ac620 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3927fc700 1 -- 172.21.15.114:0/1517557162 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fb3a4015cc0 con 0x55a5277ac620 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3c0079b80 1 -- 172.21.15.114:0/1517557162 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb39c005580 con 0x55a5277ac620 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3927fc700 1 -- 172.21.15.114:0/1517557162 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb3a4032050 con 0x55a5277ac620 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3c0079b80 10 client.15405 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3c0079b80 1 -- 172.21.15.114:0/1517557162 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a5277c4390 con 0x55a5277ac620 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3927fc700 1 -- 172.21.15.114:0/1517557162 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb39c005580 con 0x55a5277ac620 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3927fc700 1 -- 172.21.15.114:0/1517557162 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb3a4020040 con 0x55a5277ac620 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3c0079b80 10 client.15405 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3c0079b80 10 client.15405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.581+0000 7fb3c0079b80 20 client.15405 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.582+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/1517557162 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a5277c6970 0x55a5277c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.582+0000 7fb3c0079b80 4 client.15405 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.582+0000 7fb3c0079b80 1 -- 172.21.15.114:0/1517557162 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5277769f0 con 0x55a5277c6970 2024-05-27T07:40:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.582+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/1517557162 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a5277c6970 0x55a5277c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.582+0000 7fb3927fc700 10 client.15405 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.582+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/1517557162 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a5277c6970 0x55a5277c8db0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7824fdf428a749a9 server_cookie=57bf1d3ca9360b3d in_seq=0 out_seq=0 2024-05-27T07:40:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.583+0000 7fb3927fc700 1 -- 172.21.15.114:0/1517557162 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a5277769f0 con 0x55a5277c6970 2024-05-27T07:40:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.583+0000 7fb3927fc700 10 client.15405 handle_command_reply: tid=0 2024-05-27T07:40:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.583+0000 7fb3c0079b80 1 client.15405 shutdown 2024-05-27T07:40:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.583+0000 7fb3c0079b80 1 -- 172.21.15.114:0/1517557162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac620 msgr2=0x55a5277b6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.583+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/1517557162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac620 0x55a5277b6f20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4008ac0 tx=0x7fb3a4004b40 comp rx=0 tx=0).stop 2024-05-27T07:40:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.583+0000 7fb3c0079b80 1 -- 172.21.15.114:0/1517557162 shutdown_connections 2024-05-27T07:40:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.583+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/1517557162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac620 0x55a5277b6f20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.583+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/1517557162 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a5277c6970 0x55a5277c8db0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.583+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/1517557162 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52762ce40 0x55a5277b64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.583+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/1517557162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac210 0x55a5277b69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.583+0000 7fb3c0079b80 1 -- 172.21.15.114:0/1517557162 >> 172.21.15.114:0/1517557162 conn(0x55a527687af0 msgr2=0x55a5277ad820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.583+0000 7fb3927fc700 0 client.15405 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.583+0000 7fb3c0079b80 1 -- 172.21.15.114:0/1517557162 shutdown_connections 2024-05-27T07:40:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.584+0000 7fb3c0079b80 1 -- 172.21.15.114:0/1517557162 wait complete. 2024-05-27T07:40:10.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.584+0000 7fb3c0079b80 20 client.15405 trim_cache size 0 max 16384 2024-05-27T07:40:10.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.603+0000 7fb3c0079b80 1 Processor -- start 2024-05-27T07:40:10.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.603+0000 7fb3c0079b80 1 -- start start 2024-05-27T07:40:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3c0079b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 0x55a5277ae210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3c0079b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a5277ac210 0x55a5277c28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3c0079b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 0x55a5277b1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3c0079b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1ac2c0 con 0x55a52762ce40 2024-05-27T07:40:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3c0079b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1aeb40 con 0x55a5277ac210 2024-05-27T07:40:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3c0079b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1ac930 con 0x55a5277ac620 2024-05-27T07:40:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3aaffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 0x55a5277ae210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3aaffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 0x55a5277ae210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41390/0 (socket says 172.21.15.114:41390) 2024-05-27T07:40:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3aaffd700 1 -- 172.21.15.114:0/2213479465 learned_addr learned my addr 172.21.15.114:0/2213479465 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3ab7fe700 1 --2- 172.21.15.114:0/2213479465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 0x55a5277b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3a3fff700 1 --2- 172.21.15.114:0/2213479465 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a5277ac210 0x55a5277c28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3aaffd700 1 -- 172.21.15.114:0/2213479465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 msgr2=0x55a5277b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/2213479465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 0x55a5277b1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3aaffd700 1 -- 172.21.15.114:0/2213479465 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a5277ac210 msgr2=0x55a5277c28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/2213479465 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a5277ac210 0x55a5277c28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.604+0000 7fb3aaffd700 1 -- 172.21.15.114:0/2213479465 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5277aca30 con 0x55a52762ce40 2024-05-27T07:40:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.605+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/2213479465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 0x55a5277ae210 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb39c000c00 tx=0x7fb39c009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef43459596116546 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.605+0000 7fb3a0ff9700 1 -- 172.21.15.114:0/2213479465 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c005410 con 0x55a52762ce40 2024-05-27T07:40:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.605+0000 7fb3a0ff9700 1 -- 172.21.15.114:0/2213479465 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb39c003d10 con 0x55a52762ce40 2024-05-27T07:40:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.605+0000 7fb3a0ff9700 1 -- 172.21.15.114:0/2213479465 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c00f460 con 0x55a52762ce40 2024-05-27T07:40:10.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 -- 172.21.15.114:0/2213479465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 msgr2=0x55a5277ae210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:10.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/2213479465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 0x55a5277ae210 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb39c000c00 tx=0x7fb39c009fd0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 -- 172.21.15.114:0/2213479465 shutdown_connections 2024-05-27T07:40:10.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/2213479465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 0x55a5277b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/2213479465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a52762ce40 0x55a5277ae210 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/2213479465 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a5277ac210 0x55a5277c28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 -- 172.21.15.114:0/2213479465 >> 172.21.15.114:0/2213479465 conn(0x55a527687af0 msgr2=0x55a5277b4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:10.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 -- 172.21.15.114:0/2213479465 shutdown_connections 2024-05-27T07:40:10.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 -- 172.21.15.114:0/2213479465 wait complete. 2024-05-27T07:40:10.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 -- start start 2024-05-27T07:40:10.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 Processor -- start 2024-05-27T07:40:10.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52762ce40 0x55a5277ba280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac210 0x55a5277ba7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 0x55a527829440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1ad000 con 0x55a5277ac210 2024-05-27T07:40:10.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1ae470 con 0x55a52762ce40 2024-05-27T07:40:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1ad6d0 con 0x55a5277ac620 2024-05-27T07:40:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.606+0000 7fb3c0079b80 10 client.0 _ll_register_callbacks cb 0x55a527664920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3a3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac210 0x55a5277ba7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3a3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac210 0x55a5277ba7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41398/0 (socket says 172.21.15.114:41398) 2024-05-27T07:40:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3a3fff700 1 -- 172.21.15.114:0/4272064269 learned_addr learned my addr 172.21.15.114:0/4272064269 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3ab7fe700 1 --2- 172.21.15.114:0/4272064269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 0x55a527829440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/4272064269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52762ce40 0x55a5277ba280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3a3fff700 1 -- 172.21.15.114:0/4272064269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 msgr2=0x55a527829440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3a3fff700 1 --2- 172.21.15.114:0/4272064269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 0x55a527829440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3a3fff700 1 -- 172.21.15.114:0/4272064269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52762ce40 msgr2=0x55a5277ba280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3a3fff700 1 --2- 172.21.15.114:0/4272064269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52762ce40 0x55a5277ba280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3a3fff700 1 -- 172.21.15.114:0/4272064269 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a527829a00 con 0x55a5277ac210 2024-05-27T07:40:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3a3fff700 1 --2- 172.21.15.114:0/4272064269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac210 0x55a5277ba7c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb394009520 tx=0x7fb3940a9070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b2003551923681 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3927fc700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3927fc700 1 -- 172.21.15.114:0/4272064269 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb394020040 con 0x55a5277ac210 2024-05-27T07:40:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3927fc700 1 -- 172.21.15.114:0/4272064269 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fb3940e2570 con 0x55a5277ac210 2024-05-27T07:40:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3c0079b80 1 -- 172.21.15.114:0/4272064269 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb39c003d10 con 0x55a5277ac210 2024-05-27T07:40:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3927fc700 1 -- 172.21.15.114:0/4272064269 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb3940ad820 con 0x55a5277ac210 2024-05-27T07:40:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3c0079b80 10 client.15411 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.607+0000 7fb3c0079b80 1 -- 172.21.15.114:0/4272064269 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a527830d40 con 0x55a5277ac210 2024-05-27T07:40:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.608+0000 7fb3927fc700 1 -- 172.21.15.114:0/4272064269 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb39c003d10 con 0x55a5277ac210 2024-05-27T07:40:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.608+0000 7fb3927fc700 1 -- 172.21.15.114:0/4272064269 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb3940e2040 con 0x55a5277ac210 2024-05-27T07:40:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.608+0000 7fb3c0079b80 10 client.15411 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.608+0000 7fb3c0079b80 10 client.15411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.608+0000 7fb3c0079b80 20 client.15411 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.608+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/4272064269 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a527831520 0x55a527833940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.608+0000 7fb3c0079b80 4 client.15411 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:40:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.608+0000 7fb3c0079b80 1 -- 172.21.15.114:0/4272064269 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a5276dbbe0 con 0x55a527831520 2024-05-27T07:40:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.608+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/4272064269 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a527831520 0x55a527833940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.608+0000 7fb3927fc700 10 client.15411 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.608+0000 7fb3aaffd700 1 --2- 172.21.15.114:0/4272064269 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a527831520 0x55a527833940 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d50ed170ab78870d server_cookie=52fc5e0da115741f in_seq=0 out_seq=0 2024-05-27T07:40:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3927fc700 1 -- 172.21.15.114:0/4272064269 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a5276dbbe0 con 0x55a527831520 2024-05-27T07:40:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3927fc700 10 client.15411 handle_command_reply: tid=0 2024-05-27T07:40:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3c0079b80 1 client.15411 shutdown 2024-05-27T07:40:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3c0079b80 1 -- 172.21.15.114:0/4272064269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac210 msgr2=0x55a5277ba7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/4272064269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac210 0x55a5277ba7c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb394009520 tx=0x7fb3940a9070 comp rx=0 tx=0).stop 2024-05-27T07:40:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3c0079b80 1 -- 172.21.15.114:0/4272064269 shutdown_connections 2024-05-27T07:40:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/4272064269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5277ac620 0x55a527829440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/4272064269 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a527831520 0x55a527833940 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/4272064269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52762ce40 0x55a5277ba280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3c0079b80 1 --2- 172.21.15.114:0/4272064269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5277ac210 0x55a5277ba7c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3c0079b80 1 -- 172.21.15.114:0/4272064269 >> 172.21.15.114:0/4272064269 conn(0x55a527687af0 msgr2=0x55a5277b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3927fc700 0 client.15411 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3c0079b80 1 -- 172.21.15.114:0/4272064269 shutdown_connections 2024-05-27T07:40:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3c0079b80 1 -- 172.21.15.114:0/4272064269 wait complete. 2024-05-27T07:40:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.609+0000 7fb3c0079b80 20 client.15411 trim_cache size 0 max 16384 2024-05-27T07:40:10.612 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:40:10.612 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:40:10.612 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:40:10.613 INFO:teuthology.orchestra.run.smithi114.stdout: "2f26bcac-8fbb-4376-b819-dfdfb08fe7bc": { 2024-05-27T07:40:10.613 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:40:10.613 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "2f26bcac-8fbb-4376-b819-dfdfb08fe7bc", 2024-05-27T07:40:10.613 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:40:10.613 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:40:10.613 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:40:10.613 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:40:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.610+0000 7fb3b1664700 1 -- 172.21.15.114:0/2307922338 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb3980b2b90 msgr2=0x7fb3980b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.610+0000 7fb3b1664700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb3980b2b90 0x7fb3980b50a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4003ee0 tx=0x7fb3a4009100 comp rx=0 tx=0).stop 2024-05-27T07:40:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.610+0000 7fb3b1664700 1 -- 172.21.15.114:0/2307922338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3ac107e20 msgr2=0x7fb3ac19fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.610+0000 7fb3b1664700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3ac107e20 0x7fb3ac19fe20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb39400d980 tx=0x7fb39400dc90 comp rx=0 tx=0).stop 2024-05-27T07:40:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.610+0000 7fb3b1664700 1 -- 172.21.15.114:0/2307922338 shutdown_connections 2024-05-27T07:40:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.610+0000 7fb3b1664700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac108770 0x7fb3ac199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.610+0000 7fb3b1664700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb3980b2b90 0x7fb3980b50a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.610+0000 7fb3b1664700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb3ac103540 0x7fb3ac19f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.610+0000 7fb3b1664700 1 --2- 172.21.15.114:0/2307922338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3ac107e20 0x7fb3ac19fe20 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.610+0000 7fb3b1664700 1 -- 172.21.15.114:0/2307922338 >> 172.21.15.114:0/2307922338 conn(0x7fb3ac078df0 msgr2=0x7fb3ac100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.610+0000 7fb3b1664700 1 -- 172.21.15.114:0/2307922338 shutdown_connections 2024-05-27T07:40:10.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:10.610+0000 7fb3b1664700 1 -- 172.21.15.114:0/2307922338 wait complete. 2024-05-27T07:40:11.362 INFO:tasks.cephfs.filesystem:scrub status for tag:2f26bcac-8fbb-4376-b819-dfdfb08fe7bc - {'path': '/', 'tag': '2f26bcac-8fbb-4376-b819-dfdfb08fe7bc', 'options': 'recursive,force'} 2024-05-27T07:40:11.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:11 smithi163 ceph-mon[36451]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 46 KiB/s wr, 38 op/s 2024-05-27T07:40:11.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:11 smithi186 ceph-mon[36647]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 46 KiB/s wr, 38 op/s 2024-05-27T07:40:11.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:11 smithi114 ceph-mon[32038]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 46 KiB/s wr, 38 op/s 2024-05-27T07:40:12.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:12 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:40:12.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:12 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:40:12.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:12 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:40:13.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:13 smithi163 ceph-mon[36451]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-05-27T07:40:13.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:13 smithi186 ceph-mon[36647]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-05-27T07:40:13.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:13 smithi114 ceph-mon[32038]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-05-27T07:40:15.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:15 smithi163 ceph-mon[36451]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-05-27T07:40:15.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:15 smithi186 ceph-mon[36647]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-05-27T07:40:15.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:15 smithi114 ceph-mon[32038]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-05-27T07:40:17.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:17 smithi163 ceph-mon[36451]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-05-27T07:40:17.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:17 smithi186 ceph-mon[36647]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-05-27T07:40:17.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:17 smithi114 ceph-mon[32038]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-05-27T07:40:19.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:19 smithi163 ceph-mon[36451]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-05-27T07:40:19.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:19 smithi186 ceph-mon[36647]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-05-27T07:40:19.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:19 smithi114 ceph-mon[32038]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-05-27T07:40:21.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:21 smithi163 ceph-mon[36451]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-05-27T07:40:21.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:21 smithi186 ceph-mon[36647]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-05-27T07:40:21.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:21 smithi114 ceph-mon[32038]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-05-27T07:40:22.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.444+0000 7f2b61069640 1 -- 172.21.15.114:0/3384384113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b5c1021c0 msgr2=0x7f2b5c1025d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:22.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.444+0000 7f2b61069640 1 --2- 172.21.15.114:0/3384384113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b5c1021c0 0x7f2b5c1025d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2b44009900 tx=0x7f2b4402a410 comp rx=0 tx=0).stop 2024-05-27T07:40:22.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.444+0000 7f2b61069640 1 -- 172.21.15.114:0/3384384113 shutdown_connections 2024-05-27T07:40:22.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.444+0000 7f2b61069640 1 --2- 172.21.15.114:0/3384384113 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b5c10e6c0 0x7f2b5c110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.444+0000 7f2b61069640 1 --2- 172.21.15.114:0/3384384113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b5c1021c0 0x7f2b5c1025d0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.444+0000 7f2b61069640 1 --2- 172.21.15.114:0/3384384113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5c109d10 0x7f2b5c101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.444+0000 7f2b61069640 1 -- 172.21.15.114:0/3384384113 >> 172.21.15.114:0/3384384113 conn(0x7f2b5c0779d0 msgr2=0x7f2b5c077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:22.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.444+0000 7f2b61069640 1 -- 172.21.15.114:0/3384384113 shutdown_connections 2024-05-27T07:40:22.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.444+0000 7f2b61069640 1 -- 172.21.15.114:0/3384384113 wait complete. 2024-05-27T07:40:22.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.445+0000 7f2b61069640 1 Processor -- start 2024-05-27T07:40:22.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.445+0000 7f2b61069640 1 -- start start 2024-05-27T07:40:22.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.445+0000 7f2b61069640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b5c1021c0 0x7f2b5c1a0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.445+0000 7f2b61069640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b5c109d10 0x7f2b5c1a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.445+0000 7f2b61069640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5c10e6c0 0x7f2b5c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.445+0000 7f2b61069640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c113e80 con 0x7f2b5c1021c0 2024-05-27T07:40:22.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.445+0000 7f2b61069640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c113d00 con 0x7f2b5c109d10 2024-05-27T07:40:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.445+0000 7f2b61069640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c114000 con 0x7f2b5c10e6c0 2024-05-27T07:40:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b5ad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b5c1021c0 0x7f2b5c1a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b5ad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b5c1021c0 0x7f2b5c1a0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42388/0 (socket says 172.21.15.114:42388) 2024-05-27T07:40:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b5ad9d640 1 -- 172.21.15.114:0/1785499283 learned_addr learned my addr 172.21.15.114:0/1785499283 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b5a59c640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b5c109d10 0x7f2b5c1a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5c10e6c0 0x7f2b5c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b5ad9d640 1 -- 172.21.15.114:0/1785499283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5c10e6c0 msgr2=0x7f2b5c19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b5ad9d640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5c10e6c0 0x7f2b5c19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b5ad9d640 1 -- 172.21.15.114:0/1785499283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b5c109d10 msgr2=0x7f2b5c1a0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b5ad9d640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b5c109d10 0x7f2b5c1a0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b5ad9d640 1 -- 172.21.15.114:0/1785499283 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b5c19ae10 con 0x7f2b5c1021c0 2024-05-27T07:40:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b5ad9d640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b5c1021c0 0x7f2b5c1a0560 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00ead0 tx=0x7f2b4c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b53fff640 1 -- 172.21.15.114:0/1785499283 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c00ce40 con 0x7f2b5c1021c0 2024-05-27T07:40:22.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b61069640 1 -- 172.21.15.114:0/1785499283 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b5c19b0a0 con 0x7f2b5c1021c0 2024-05-27T07:40:22.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b53fff640 1 -- 172.21.15.114:0/1785499283 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b4c011920 con 0x7f2b5c1021c0 2024-05-27T07:40:22.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b53fff640 1 -- 172.21.15.114:0/1785499283 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c01aae0 con 0x7f2b5c1021c0 2024-05-27T07:40:22.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.446+0000 7f2b61069640 1 -- 172.21.15.114:0/1785499283 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b5c1a7300 con 0x7f2b5c1021c0 2024-05-27T07:40:22.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.449+0000 7f2b67ff5740 1 Processor -- start 2024-05-27T07:40:22.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.449+0000 7f2b67ff5740 1 -- start start 2024-05-27T07:40:22.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.449+0000 7f2b53fff640 1 -- 172.21.15.114:0/1785499283 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7f2b4c01ac80 con 0x7f2b5c1021c0 2024-05-27T07:40:22.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b67ff5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 0x56249fd083d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b67ff5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 0x56249fd08910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b67ff5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 0x56249fd08e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b67ff5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56249fc7ac30 con 0x56249f9671b0 2024-05-27T07:40:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b67ff5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56249fc6a7d0 con 0x56249f9464d0 2024-05-27T07:40:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b67ff5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56249fc6e810 con 0x56249f93e2f0 2024-05-27T07:40:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b5b59e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 0x56249fd083d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b5b59e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 0x56249fd083d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42400/0 (socket says 172.21.15.114:42400) 2024-05-27T07:40:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b5b59e640 1 -- 172.21.15.114:0/195600638 learned_addr learned my addr 172.21.15.114:0/195600638 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b5a59c640 1 --2- 172.21.15.114:0/195600638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 0x56249fd08e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b5ad9d640 1 --2- 172.21.15.114:0/195600638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 0x56249fd08910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b5b59e640 1 -- 172.21.15.114:0/195600638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 msgr2=0x56249fd08910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/195600638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 0x56249fd08910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b5b59e640 1 -- 172.21.15.114:0/195600638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 msgr2=0x56249fd08e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/195600638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 0x56249fd08e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b5b59e640 1 -- 172.21.15.114:0/195600638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56249f97f210 con 0x56249f9671b0 2024-05-27T07:40:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b5a59c640 1 --2- 172.21.15.114:0/195600638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 0x56249fd08e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b53fff640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b480b2a70 0x7f2b480b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/195600638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 0x56249fd083d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2b5400aac0 tx=0x7f2b5400af80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d43ceab8896473 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b5ad9d640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b480b2a70 0x7f2b480b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.450+0000 7f2b517fa640 1 -- 172.21.15.114:0/195600638 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5401d070 con 0x56249f9671b0 2024-05-27T07:40:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.451+0000 7f2b53fff640 1 -- 172.21.15.114:0/1785499283 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f2b4c0e3130 con 0x7f2b5c1021c0 2024-05-27T07:40:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.451+0000 7f2b517fa640 1 -- 172.21.15.114:0/195600638 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b540188f0 con 0x56249f9671b0 2024-05-27T07:40:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.451+0000 7f2b517fa640 1 -- 172.21.15.114:0/195600638 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b540108b0 con 0x56249f9671b0 2024-05-27T07:40:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.451+0000 7f2b67ff5740 1 -- 172.21.15.114:0/195600638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 msgr2=0x56249fd083d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.451+0000 7f2b5ad9d640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b480b2a70 0x7f2b480b4f80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c0e03b0 tx=0x7f2b4c025040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.451+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/195600638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 0x56249fd083d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2b5400aac0 tx=0x7f2b5400af80 comp rx=0 tx=0).stop 2024-05-27T07:40:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 -- 172.21.15.114:0/195600638 shutdown_connections 2024-05-27T07:40:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/195600638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 0x56249fd08e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/195600638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 0x56249fd08910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/195600638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 0x56249fd083d0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 -- 172.21.15.114:0/195600638 >> 172.21.15.114:0/195600638 conn(0x56249f95b300 msgr2=0x56249fd04b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 -- 172.21.15.114:0/195600638 shutdown_connections 2024-05-27T07:40:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 -- 172.21.15.114:0/195600638 wait complete. 2024-05-27T07:40:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 -- start start 2024-05-27T07:40:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 Processor -- start 2024-05-27T07:40:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 0x56249fd14430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 0x56249fd14990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 0x56249fd128f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56249f950790 con 0x56249f9671b0 2024-05-27T07:40:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56249fc26890 con 0x56249f9464d0 2024-05-27T07:40:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.452+0000 7f2b67ff5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56249fd15500 con 0x56249f93e2f0 2024-05-27T07:40:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b67ff5740 10 client.0 _ll_register_callbacks cb 0x56249f97ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b5b59e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 0x56249fd128f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b5b59e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 0x56249fd128f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42404/0 (socket says 172.21.15.114:42404) 2024-05-27T07:40:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b5b59e640 1 -- 172.21.15.114:0/2833642146 learned_addr learned my addr 172.21.15.114:0/2833642146 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/2833642146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 0x56249fd14430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b5a59c640 1 --2- 172.21.15.114:0/2833642146 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 0x56249fd14990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b5b59e640 1 -- 172.21.15.114:0/2833642146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 msgr2=0x56249fd14430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/2833642146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 0x56249fd14430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b5b59e640 1 -- 172.21.15.114:0/2833642146 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 msgr2=0x56249fd14990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/2833642146 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 0x56249fd14990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b5b59e640 1 -- 172.21.15.114:0/2833642146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56249fa4b620 con 0x56249f9671b0 2024-05-27T07:40:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/2833642146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 0x56249fd128f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2b5400ffd0 tx=0x7f2b54010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da2427938eb75b62 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b427fc640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b427fc640 1 -- 172.21.15.114:0/2833642146 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5401d070 con 0x56249f9671b0 2024-05-27T07:40:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2833642146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b540188f0 con 0x56249f9671b0 2024-05-27T07:40:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b427fc640 1 -- 172.21.15.114:0/2833642146 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f2b54005460 con 0x56249f9671b0 2024-05-27T07:40:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b427fc640 1 -- 172.21.15.114:0/2833642146 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2b54010650 con 0x56249f9671b0 2024-05-27T07:40:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b67ff5740 10 client.15423 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.453+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2833642146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56249fd21860 con 0x56249f9671b0 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.454+0000 7f2b427fc640 1 -- 172.21.15.114:0/2833642146 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b540188f0 con 0x56249f9671b0 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.454+0000 7f2b427fc640 1 -- 172.21.15.114:0/2833642146 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2b54004ac0 con 0x56249f9671b0 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.454+0000 7f2b67ff5740 10 client.15423 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.454+0000 7f2b67ff5740 10 client.15423 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.454+0000 7f2b67ff5740 20 client.15423 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.454+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/2833642146 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56249f925d80 0x56249fd24cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.454+0000 7f2b67ff5740 4 client.15423 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.454+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2833642146 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56249fc7cb80 con 0x56249f925d80 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.454+0000 7f2b5a59c640 1 --2- 172.21.15.114:0/2833642146 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56249f925d80 0x56249fd24cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.455+0000 7f2b427fc640 10 client.15423 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.455+0000 7f2b5a59c640 1 --2- 172.21.15.114:0/2833642146 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56249f925d80 0x56249fd24cd0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5d77d95b5908504 server_cookie=f2e700866c0ec5ac in_seq=0 out_seq=0 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.455+0000 7f2b427fc640 1 -- 172.21.15.114:0/2833642146 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56249fc7cb80 con 0x56249f925d80 2024-05-27T07:40:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.455+0000 7f2b427fc640 10 client.15423 handle_command_reply: tid=0 2024-05-27T07:40:22.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.455+0000 7f2b67ff5740 1 client.15423 shutdown 2024-05-27T07:40:22.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.456+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2833642146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 msgr2=0x56249fd128f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:22.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.456+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/2833642146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 0x56249fd128f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2b5400ffd0 tx=0x7f2b54010430 comp rx=0 tx=0).stop 2024-05-27T07:40:22.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.456+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2833642146 shutdown_connections 2024-05-27T07:40:22.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.456+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/2833642146 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56249f925d80 0x56249fd24cd0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.456+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/2833642146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9671b0 0x56249fd128f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.456+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/2833642146 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 0x56249fd14990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.456+0000 7f2b427fc640 0 client.15423 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:22.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.456+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/2833642146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 0x56249fd14430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.456+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2833642146 >> 172.21.15.114:0/2833642146 conn(0x56249f95b300 msgr2=0x56249fd08200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:22.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.456+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2833642146 shutdown_connections 2024-05-27T07:40:22.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.456+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2833642146 wait complete. 2024-05-27T07:40:22.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.456+0000 7f2b67ff5740 20 client.15423 trim_cache size 0 max 16384 2024-05-27T07:40:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.471+0000 7f2b67ff5740 1 Processor -- start 2024-05-27T07:40:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.471+0000 7f2b67ff5740 1 -- start start 2024-05-27T07:40:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.471+0000 7f2b67ff5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f93e2f0 0x56249fd12fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.471+0000 7f2b67ff5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 0x56249fd08160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.471+0000 7f2b67ff5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f9671b0 0x56249fd086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.471+0000 7f2b67ff5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1af3a0 con 0x56249f93e2f0 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.471+0000 7f2b67ff5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1b1c20 con 0x56249f9464d0 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.471+0000 7f2b67ff5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1afa10 con 0x56249f9671b0 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b5b59e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f93e2f0 0x56249fd12fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b5b59e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f93e2f0 0x56249fd12fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42406/0 (socket says 172.21.15.114:42406) 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b5b59e640 1 -- 172.21.15.114:0/959795105 learned_addr learned my addr 172.21.15.114:0/959795105 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/959795105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f9671b0 0x56249fd086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b5a59c640 1 --2- 172.21.15.114:0/959795105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 0x56249fd08160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b5b59e640 1 -- 172.21.15.114:0/959795105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f9671b0 msgr2=0x56249fd086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/959795105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f9671b0 0x56249fd086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b5b59e640 1 -- 172.21.15.114:0/959795105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 msgr2=0x56249fd08160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/959795105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 0x56249fd08160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b5b59e640 1 -- 172.21.15.114:0/959795105 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56249fd25040 con 0x56249f93e2f0 2024-05-27T07:40:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/959795105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f93e2f0 0x56249fd12fc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2b5401b040 tx=0x7f2b5400a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c14d6353fa35c767 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b50ff9640 1 -- 172.21.15.114:0/959795105 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5401d040 con 0x56249f93e2f0 2024-05-27T07:40:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b50ff9640 1 -- 172.21.15.114:0/959795105 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b5401e040 con 0x56249f93e2f0 2024-05-27T07:40:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.472+0000 7f2b50ff9640 1 -- 172.21.15.114:0/959795105 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b540049e0 con 0x56249f93e2f0 2024-05-27T07:40:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.473+0000 7f2b67ff5740 1 -- 172.21.15.114:0/959795105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f93e2f0 msgr2=0x56249fd12fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.473+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/959795105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f93e2f0 0x56249fd12fc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2b5401b040 tx=0x7f2b5400a2d0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.473+0000 7f2b67ff5740 1 -- 172.21.15.114:0/959795105 shutdown_connections 2024-05-27T07:40:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.473+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/959795105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f9671b0 0x56249fd086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.473+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/959795105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9464d0 0x56249fd08160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.473+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/959795105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f93e2f0 0x56249fd12fc0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.473+0000 7f2b67ff5740 1 -- 172.21.15.114:0/959795105 >> 172.21.15.114:0/959795105 conn(0x56249f95b300 msgr2=0x56249fd128b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.473+0000 7f2b67ff5740 1 -- 172.21.15.114:0/959795105 shutdown_connections 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.473+0000 7f2b67ff5740 1 -- 172.21.15.114:0/959795105 wait complete. 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b67ff5740 1 -- start start 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b67ff5740 1 Processor -- start 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b67ff5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 0x56249fd195e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b67ff5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9464d0 0x56249fd19b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b67ff5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9671b0 0x56249fd1a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b67ff5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1b00e0 con 0x56249f9464d0 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b67ff5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1b1550 con 0x56249f9671b0 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b67ff5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1b07b0 con 0x56249f93e2f0 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b67ff5740 10 client.0 _ll_register_callbacks cb 0x7f2b5401b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b5a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9464d0 0x56249fd19b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b5a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9464d0 0x56249fd19b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42410/0 (socket says 172.21.15.114:42410) 2024-05-27T07:40:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b5a59c640 1 -- 172.21.15.114:0/2096515792 learned_addr learned my addr 172.21.15.114:0/2096515792 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9671b0 0x56249fd1a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b5a59c640 1 -- 172.21.15.114:0/2096515792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 msgr2=0x56249fd195e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 0x56249fd195e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b5a59c640 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 0x56249fd195e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b5a59c640 1 -- 172.21.15.114:0/2096515792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9671b0 msgr2=0x56249fd1a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b5a59c640 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9671b0 0x56249fd1a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b5a59c640 1 -- 172.21.15.114:0/2096515792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56249fd8b350 con 0x56249f9464d0 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.474+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9671b0 0x56249fd1a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b5a59c640 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9464d0 0x56249fd19b20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2b44009900 tx=0x7f2b44004130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f137dbee4fb46802 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b427fc640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b427fc640 1 -- 172.21.15.114:0/2096515792 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4404a060 con 0x56249f9464d0 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b427fc640 1 -- 172.21.15.114:0/2096515792 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f2b44005230 con 0x56249f9464d0 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2096515792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b5401e040 con 0x56249f9464d0 2024-05-27T07:40:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b427fc640 1 -- 172.21.15.114:0/2096515792 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2b44051050 con 0x56249f9464d0 2024-05-27T07:40:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b67ff5740 10 client.15429 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2096515792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56249fd8c630 con 0x56249f9464d0 2024-05-27T07:40:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b427fc640 1 -- 172.21.15.114:0/2096515792 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b5401e040 con 0x56249f9464d0 2024-05-27T07:40:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b427fc640 1 -- 172.21.15.114:0/2096515792 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2b440035f0 con 0x56249f9464d0 2024-05-27T07:40:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b67ff5740 10 client.15429 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b67ff5740 10 client.15429 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b67ff5740 20 client.15429 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56249f8fb1c0 0x56249fd8ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b67ff5740 4 client.15429 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:40:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.475+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2096515792 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56249fc76360 con 0x56249f8fb1c0 2024-05-27T07:40:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.476+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56249f8fb1c0 0x56249fd8ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.476+0000 7f2b427fc640 10 client.15429 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.476+0000 7f2b5b59e640 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56249f8fb1c0 0x56249fd8ee20 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfab982cf67c79ed server_cookie=75cb555819747482 in_seq=0 out_seq=0 2024-05-27T07:40:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b427fc640 1 -- 172.21.15.114:0/2096515792 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56249fc76360 con 0x56249f8fb1c0 2024-05-27T07:40:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b427fc640 10 client.15429 handle_command_reply: tid=0 2024-05-27T07:40:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b67ff5740 1 client.15429 shutdown 2024-05-27T07:40:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2096515792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9464d0 msgr2=0x56249fd19b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9464d0 0x56249fd19b20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2b44009900 tx=0x7f2b44004130 comp rx=0 tx=0).stop 2024-05-27T07:40:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2096515792 shutdown_connections 2024-05-27T07:40:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56249f8fb1c0 0x56249fd8ee20 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56249f9671b0 0x56249fd1a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56249f9464d0 0x56249fd19b20 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b427fc640 0 client.15429 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b67ff5740 1 --2- 172.21.15.114:0/2096515792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56249f93e2f0 0x56249fd195e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2096515792 >> 172.21.15.114:0/2096515792 conn(0x56249f95b300 msgr2=0x56249fd08160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2096515792 shutdown_connections 2024-05-27T07:40:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b67ff5740 1 -- 172.21.15.114:0/2096515792 wait complete. 2024-05-27T07:40:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.480+0000 7f2b67ff5740 20 client.15429 trim_cache size 0 max 16384 2024-05-27T07:40:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:40:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:40:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:40:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.481+0000 7f2b61069640 1 -- 172.21.15.114:0/1785499283 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b480b2a70 msgr2=0x7f2b480b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.481+0000 7f2b61069640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b480b2a70 0x7f2b480b4f80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c0e03b0 tx=0x7f2b4c025040 comp rx=0 tx=0).stop 2024-05-27T07:40:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.481+0000 7f2b61069640 1 -- 172.21.15.114:0/1785499283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b5c1021c0 msgr2=0x7f2b5c1a0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.481+0000 7f2b61069640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b5c1021c0 0x7f2b5c1a0560 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00ead0 tx=0x7f2b4c00ef90 comp rx=0 tx=0).stop 2024-05-27T07:40:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.481+0000 7f2b61069640 1 -- 172.21.15.114:0/1785499283 shutdown_connections 2024-05-27T07:40:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.481+0000 7f2b61069640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b480b2a70 0x7f2b480b4f80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.481+0000 7f2b61069640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5c10e6c0 0x7f2b5c19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.481+0000 7f2b61069640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b5c109d10 0x7f2b5c1a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.481+0000 7f2b61069640 1 --2- 172.21.15.114:0/1785499283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b5c1021c0 0x7f2b5c1a0560 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.481+0000 7f2b61069640 1 -- 172.21.15.114:0/1785499283 >> 172.21.15.114:0/1785499283 conn(0x7f2b5c0779d0 msgr2=0x7f2b5c105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.481+0000 7f2b61069640 1 -- 172.21.15.114:0/1785499283 shutdown_connections 2024-05-27T07:40:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:22.481+0000 7f2b61069640 1 -- 172.21.15.114:0/1785499283 wait complete. 2024-05-27T07:40:23.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:23 smithi163 ceph-mon[36451]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 597 B/s wr, 3 op/s 2024-05-27T07:40:23.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:23 smithi186 ceph-mon[36647]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 597 B/s wr, 3 op/s 2024-05-27T07:40:23.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:23 smithi114 ceph-mon[32038]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 597 B/s wr, 3 op/s 2024-05-27T07:40:25.473 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:25 smithi163 ceph-mon[36451]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:40:25.476 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:25 smithi186 ceph-mon[36647]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:40:25.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:25 smithi114 ceph-mon[32038]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:40:27.552 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:27 smithi163 ceph-mon[36451]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:40:27.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:27 smithi114 ceph-mon[32038]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:40:27.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:27 smithi186 ceph-mon[36647]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:40:29.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:29 smithi114 ceph-mon[32038]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:40:29.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:29 smithi163 ceph-mon[36451]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:40:29.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:29 smithi186 ceph-mon[36647]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:40:31.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:31 smithi114 ceph-mon[32038]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-27T07:40:31.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:31 smithi163 ceph-mon[36451]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-27T07:40:31.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:31 smithi186 ceph-mon[36647]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-27T07:40:33.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:33 smithi114 ceph-mon[32038]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-27T07:40:33.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:33 smithi163 ceph-mon[36451]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-27T07:40:33.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:33 smithi186 ceph-mon[36647]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-27T07:40:35.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:35 smithi114 ceph-mon[32038]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-05-27T07:40:35.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.591+0000 7f8774b9d640 1 -- 172.21.15.114:0/591873341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8770109ad0 msgr2=0x7f8770103670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:35.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.591+0000 7f8774b9d640 1 --2- 172.21.15.114:0/591873341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8770109ad0 0x7f8770103670 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8760009900 tx=0x7f876002a440 comp rx=0 tx=0).stop 2024-05-27T07:40:35.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.591+0000 7f8774b9d640 1 -- 172.21.15.114:0/591873341 shutdown_connections 2024-05-27T07:40:35.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.591+0000 7f8774b9d640 1 --2- 172.21.15.114:0/591873341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8770104220 0x7f87701105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.591+0000 7f8774b9d640 1 --2- 172.21.15.114:0/591873341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8770109f70 0x7f8770103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.591+0000 7f8774b9d640 1 --2- 172.21.15.114:0/591873341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8770109ad0 0x7f8770103670 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.591+0000 7f8774b9d640 1 -- 172.21.15.114:0/591873341 >> 172.21.15.114:0/591873341 conn(0x7f87700ff330 msgr2=0x7f8770101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:35.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.591+0000 7f8774b9d640 1 -- 172.21.15.114:0/591873341 shutdown_connections 2024-05-27T07:40:35.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.591+0000 7f8774b9d640 1 -- 172.21.15.114:0/591873341 wait complete. 2024-05-27T07:40:35.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.592+0000 7f8774b9d640 1 Processor -- start 2024-05-27T07:40:35.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.592+0000 7f8774b9d640 1 -- start start 2024-05-27T07:40:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.592+0000 7f8774b9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8770104220 0x7f8770074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.592+0000 7f8774b9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8770109ad0 0x7f8770075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.592+0000 7f8774b9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8770109f70 0x7f8770075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.592+0000 7f8774b9d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8770113dc0 con 0x7f8770104220 2024-05-27T07:40:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.592+0000 7f8774b9d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8770113c40 con 0x7f8770109ad0 2024-05-27T07:40:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.592+0000 7f8774b9d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8770113f40 con 0x7f8770109f70 2024-05-27T07:40:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8770104220 0x7f8770074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8770104220 0x7f8770074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54580/0 (socket says 172.21.15.114:54580) 2024-05-27T07:40:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876f7fe640 1 -- 172.21.15.114:0/3703352457 learned_addr learned my addr 172.21.15.114:0/3703352457 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876ffff640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8770109f70 0x7f8770075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876f7fe640 1 -- 172.21.15.114:0/3703352457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8770109f70 msgr2=0x7f8770075a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876f7fe640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8770109f70 0x7f8770075a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876f7fe640 1 -- 172.21.15.114:0/3703352457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8770109ad0 msgr2=0x7f8770075500 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:40:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876f7fe640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8770109ad0 0x7f8770075500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876f7fe640 1 -- 172.21.15.114:0/3703352457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87701a8c40 con 0x7f8770104220 2024-05-27T07:40:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876ffff640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8770109f70 0x7f8770075a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876f7fe640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8770104220 0x7f8770074fc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8760031a90 tx=0x7f87600042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876cff9640 1 -- 172.21.15.114:0/3703352457 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876003f070 con 0x7f8770104220 2024-05-27T07:40:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876cff9640 1 -- 172.21.15.114:0/3703352457 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8760003aa0 con 0x7f8770104220 2024-05-27T07:40:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f8774b9d640 1 -- 172.21.15.114:0/3703352457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87701a8ed0 con 0x7f8770104220 2024-05-27T07:40:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f876cff9640 1 -- 172.21.15.114:0/3703352457 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760035070 con 0x7f8770104220 2024-05-27T07:40:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.593+0000 7f8774b9d640 1 -- 172.21.15.114:0/3703352457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87701a92d0 con 0x7f8770104220 2024-05-27T07:40:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.596+0000 7f877cd87740 1 Processor -- start 2024-05-27T07:40:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.596+0000 7f877cd87740 1 -- start start 2024-05-27T07:40:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.596+0000 7f876cff9640 1 -- 172.21.15.114:0/3703352457 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7f8760042070 con 0x7f8770104220 2024-05-27T07:40:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.596+0000 7f877cd87740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd771b0 0x563e4e118590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.596+0000 7f877cd87740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd4e2f0 0x563e4e118ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.596+0000 7f877cd87740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e119010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f877cd87740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563e4e08ac30 con 0x563e4dd564d0 2024-05-27T07:40:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f877cd87740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563e4e07a7d0 con 0x563e4dd771b0 2024-05-27T07:40:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f877cd87740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563e4e07e810 con 0x563e4dd4e2f0 2024-05-27T07:40:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f876effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e119010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f876effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e119010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54590/0 (socket says 172.21.15.114:54590) 2024-05-27T07:40:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f876effd640 1 -- 172.21.15.114:0/2645344024 learned_addr learned my addr 172.21.15.114:0/2645344024 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f876ffff640 1 --2- 172.21.15.114:0/2645344024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd771b0 0x563e4e118590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f876effd640 1 -- 172.21.15.114:0/2645344024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd4e2f0 msgr2=0x563e4e118ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f876effd640 1 --2- 172.21.15.114:0/2645344024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd4e2f0 0x563e4e118ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f876effd640 1 -- 172.21.15.114:0/2645344024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd771b0 msgr2=0x563e4e118590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f876effd640 1 --2- 172.21.15.114:0/2645344024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd771b0 0x563e4e118590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f876effd640 1 -- 172.21.15.114:0/2645344024 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e4dd8f210 con 0x563e4dd564d0 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f876cff9640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87580b2a70 0x7f87580b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f876effd640 1 --2- 172.21.15.114:0/2645344024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e119010 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f87640096f0 tx=0x7f8764009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85c4e01c69d0100 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f876ffff640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87580b2a70 0x7f87580b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f8755ffb640 1 -- 172.21.15.114:0/2645344024 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8764015020 con 0x563e4dd564d0 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f876cff9640 1 -- 172.21.15.114:0/3703352457 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f8760103610 con 0x7f8770104220 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f8755ffb640 1 -- 172.21.15.114:0/2645344024 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f876400bd80 con 0x563e4dd564d0 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.597+0000 7f8755ffb640 1 -- 172.21.15.114:0/2645344024 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8764010380 con 0x563e4dd564d0 2024-05-27T07:40:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.598+0000 7f877cd87740 1 -- 172.21.15.114:0/2645344024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 msgr2=0x563e4e119010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.598+0000 7f876ffff640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87580b2a70 0x7f87580b4f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x563e4e115700 tx=0x7f876800e040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.598+0000 7f877cd87740 1 --2- 172.21.15.114:0/2645344024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e119010 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f87640096f0 tx=0x7f8764009bb0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.598+0000 7f877cd87740 1 -- 172.21.15.114:0/2645344024 shutdown_connections 2024-05-27T07:40:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.598+0000 7f877cd87740 1 --2- 172.21.15.114:0/2645344024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e119010 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.598+0000 7f877cd87740 1 --2- 172.21.15.114:0/2645344024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd4e2f0 0x563e4e118ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.598+0000 7f877cd87740 1 --2- 172.21.15.114:0/2645344024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd771b0 0x563e4e118590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.598+0000 7f877cd87740 1 -- 172.21.15.114:0/2645344024 >> 172.21.15.114:0/2645344024 conn(0x563e4dd6b300 msgr2=0x563e4e114d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.598+0000 7f877cd87740 1 -- 172.21.15.114:0/2645344024 shutdown_connections 2024-05-27T07:40:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.598+0000 7f877cd87740 1 -- 172.21.15.114:0/2645344024 wait complete. 2024-05-27T07:40:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.599+0000 7f877cd87740 1 -- start start 2024-05-27T07:40:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.599+0000 7f877cd87740 1 Processor -- start 2024-05-27T07:40:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.599+0000 7f877cd87740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd4e2f0 0x563e4e1245d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.599+0000 7f877cd87740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e124b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.599+0000 7f877cd87740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd771b0 0x563e4e122ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.599+0000 7f877cd87740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563e4dd60790 con 0x563e4dd564d0 2024-05-27T07:40:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.599+0000 7f877cd87740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563e4e036890 con 0x563e4dd4e2f0 2024-05-27T07:40:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.599+0000 7f877cd87740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563e4e1256c0 con 0x563e4dd771b0 2024-05-27T07:40:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.599+0000 7f877cd87740 10 client.0 _ll_register_callbacks cb 0x563e4dd8ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f876effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e124b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f876effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e124b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54596/0 (socket says 172.21.15.114:54596) 2024-05-27T07:40:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f876effd640 1 -- 172.21.15.114:0/3888730964 learned_addr learned my addr 172.21.15.114:0/3888730964 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f876ffff640 1 --2- 172.21.15.114:0/3888730964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd771b0 0x563e4e122ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f876f7fe640 1 --2- 172.21.15.114:0/3888730964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd4e2f0 0x563e4e1245d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f876effd640 1 -- 172.21.15.114:0/3888730964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd771b0 msgr2=0x563e4e122ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f876effd640 1 --2- 172.21.15.114:0/3888730964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd771b0 0x563e4e122ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f876effd640 1 -- 172.21.15.114:0/3888730964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd4e2f0 msgr2=0x563e4e1245d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f876effd640 1 --2- 172.21.15.114:0/3888730964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd4e2f0 0x563e4e1245d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f876effd640 1 -- 172.21.15.114:0/3888730964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e4de5b620 con 0x563e4dd564d0 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f876effd640 1 --2- 172.21.15.114:0/3888730964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e124b30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8764009480 tx=0x7f8764009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a18b2a9ce921691 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f874effd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f874effd640 1 -- 172.21.15.114:0/3888730964 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8764020070 con 0x563e4dd564d0 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f874effd640 1 -- 172.21.15.114:0/3888730964 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f876400bad0 con 0x563e4dd564d0 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f877cd87740 1 -- 172.21.15.114:0/3888730964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f876400bd80 con 0x563e4dd564d0 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f874effd640 1 -- 172.21.15.114:0/3888730964 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8764011920 con 0x563e4dd564d0 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f877cd87740 10 client.15441 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.600+0000 7f877cd87740 1 -- 172.21.15.114:0/3888730964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563e4e131a70 con 0x563e4dd564d0 2024-05-27T07:40:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.601+0000 7f874effd640 1 -- 172.21.15.114:0/3888730964 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f876400bd80 con 0x563e4dd564d0 2024-05-27T07:40:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.601+0000 7f874effd640 1 -- 172.21.15.114:0/3888730964 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8764024400 con 0x563e4dd564d0 2024-05-27T07:40:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.601+0000 7f877cd87740 10 client.15441 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.601+0000 7f877cd87740 10 client.15441 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.601+0000 7f877cd87740 20 client.15441 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.601+0000 7f877cd87740 1 --2- 172.21.15.114:0/3888730964 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563e4dd35d80 0x563e4e134e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.601+0000 7f877cd87740 4 client.15441 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.601+0000 7f877cd87740 1 -- 172.21.15.114:0/3888730964 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e4e08cb80 con 0x563e4dd35d80 2024-05-27T07:40:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.601+0000 7f876f7fe640 1 --2- 172.21.15.114:0/3888730964 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563e4dd35d80 0x563e4e134e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.602+0000 7f874effd640 10 client.15441 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.602+0000 7f876f7fe640 1 --2- 172.21.15.114:0/3888730964 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563e4dd35d80 0x563e4e134e30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cffac80c72df9ef0 server_cookie=3672f7fc9223e7f in_seq=0 out_seq=0 2024-05-27T07:40:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.603+0000 7f874effd640 1 -- 172.21.15.114:0/3888730964 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563e4e08cb80 con 0x563e4dd35d80 2024-05-27T07:40:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.603+0000 7f874effd640 10 client.15441 handle_command_reply: tid=0 2024-05-27T07:40:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.603+0000 7f877cd87740 1 client.15441 shutdown 2024-05-27T07:40:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.603+0000 7f877cd87740 1 -- 172.21.15.114:0/3888730964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 msgr2=0x563e4e124b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.603+0000 7f877cd87740 1 --2- 172.21.15.114:0/3888730964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e124b30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8764009480 tx=0x7f8764009f40 comp rx=0 tx=0).stop 2024-05-27T07:40:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.603+0000 7f877cd87740 1 -- 172.21.15.114:0/3888730964 shutdown_connections 2024-05-27T07:40:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.603+0000 7f877cd87740 1 --2- 172.21.15.114:0/3888730964 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563e4dd35d80 0x563e4e134e30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.603+0000 7f877cd87740 1 --2- 172.21.15.114:0/3888730964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd771b0 0x563e4e122ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.603+0000 7f877cd87740 1 --2- 172.21.15.114:0/3888730964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e124b30 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.604+0000 7f877cd87740 1 --2- 172.21.15.114:0/3888730964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd4e2f0 0x563e4e1245d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.604+0000 7f874effd640 0 client.15441 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.604+0000 7f877cd87740 1 -- 172.21.15.114:0/3888730964 >> 172.21.15.114:0/3888730964 conn(0x563e4dd6b300 msgr2=0x563e4e11a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.604+0000 7f877cd87740 1 -- 172.21.15.114:0/3888730964 shutdown_connections 2024-05-27T07:40:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.604+0000 7f877cd87740 1 -- 172.21.15.114:0/3888730964 wait complete. 2024-05-27T07:40:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.604+0000 7f877cd87740 20 client.15441 trim_cache size 0 max 16384 2024-05-27T07:40:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f877cd87740 1 Processor -- start 2024-05-27T07:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f877cd87740 1 -- start start 2024-05-27T07:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f877cd87740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd4e2f0 0x563e4e123030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f877cd87740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e11c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f877cd87740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd771b0 0x563e4e11c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f877cd87740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f87701b13a0 con 0x563e4dd564d0 2024-05-27T07:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f877cd87740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f87701b3c00 con 0x563e4dd771b0 2024-05-27T07:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f877cd87740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87701b19f0 con 0x563e4dd4e2f0 2024-05-27T07:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f876effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e11c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f876effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e11c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54598/0 (socket says 172.21.15.114:54598) 2024-05-27T07:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f876effd640 1 -- 172.21.15.114:0/868759462 learned_addr learned my addr 172.21.15.114:0/868759462 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f876effd640 1 -- 172.21.15.114:0/868759462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd4e2f0 msgr2=0x563e4e123030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f876effd640 1 --2- 172.21.15.114:0/868759462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd4e2f0 0x563e4e123030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f876effd640 1 -- 172.21.15.114:0/868759462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd771b0 msgr2=0x563e4e11c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f876effd640 1 --2- 172.21.15.114:0/868759462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd771b0 0x563e4e11c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f876effd640 1 -- 172.21.15.114:0/868759462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e4e11ccd0 con 0x563e4dd564d0 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f876effd640 1 --2- 172.21.15.114:0/868759462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e11c250 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8764001eb0 tx=0x7f8764024610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b523f179c349ed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.619+0000 7f87557fa640 1 -- 172.21.15.114:0/868759462 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8764015040 con 0x563e4dd564d0 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.620+0000 7f87557fa640 1 -- 172.21.15.114:0/868759462 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8764032070 con 0x563e4dd564d0 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.620+0000 7f87557fa640 1 -- 172.21.15.114:0/868759462 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876400bb30 con 0x563e4dd564d0 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.620+0000 7f877cd87740 1 -- 172.21.15.114:0/868759462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 msgr2=0x563e4e11c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.620+0000 7f877cd87740 1 --2- 172.21.15.114:0/868759462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e11c250 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8764001eb0 tx=0x7f8764024610 comp rx=0 tx=0).stop 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.620+0000 7f877cd87740 1 -- 172.21.15.114:0/868759462 shutdown_connections 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.620+0000 7f877cd87740 1 --2- 172.21.15.114:0/868759462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd771b0 0x563e4e11c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.620+0000 7f877cd87740 1 --2- 172.21.15.114:0/868759462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd564d0 0x563e4e11c250 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.620+0000 7f877cd87740 1 --2- 172.21.15.114:0/868759462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd4e2f0 0x563e4e123030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.620+0000 7f877cd87740 1 -- 172.21.15.114:0/868759462 >> 172.21.15.114:0/868759462 conn(0x563e4dd6b300 msgr2=0x563e4e1213b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.620+0000 7f877cd87740 1 -- 172.21.15.114:0/868759462 shutdown_connections 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.620+0000 7f877cd87740 1 -- 172.21.15.114:0/868759462 wait complete. 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.621+0000 7f877cd87740 1 -- start start 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.621+0000 7f877cd87740 1 Processor -- start 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.621+0000 7f877cd87740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd4e2f0 0x563e4e127cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.621+0000 7f877cd87740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd564d0 0x563e4e128230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.621+0000 7f877cd87740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd771b0 0x563e4e19b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.621+0000 7f877cd87740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f87701b20c0 con 0x563e4dd4e2f0 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.621+0000 7f877cd87740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f87701b3530 con 0x563e4dd564d0 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.621+0000 7f877cd87740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87701b2790 con 0x563e4dd771b0 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.621+0000 7f877cd87740 10 client.0 _ll_register_callbacks cb 0x7f8768009730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.621+0000 7f876f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd4e2f0 0x563e4e127cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.621+0000 7f876f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd4e2f0 0x563e4e127cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54608/0 (socket says 172.21.15.114:54608) 2024-05-27T07:40:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.621+0000 7f876f7fe640 1 -- 172.21.15.114:0/749083066 learned_addr learned my addr 172.21.15.114:0/749083066 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.621+0000 7f876ffff640 1 --2- 172.21.15.114:0/749083066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd771b0 0x563e4e19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f876f7fe640 1 -- 172.21.15.114:0/749083066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd771b0 msgr2=0x563e4e19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f876f7fe640 1 --2- 172.21.15.114:0/749083066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd771b0 0x563e4e19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f876f7fe640 1 -- 172.21.15.114:0/749083066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd564d0 msgr2=0x563e4e128230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f876f7fe640 1 --2- 172.21.15.114:0/749083066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd564d0 0x563e4e128230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f876f7fe640 1 -- 172.21.15.114:0/749083066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e4e19b8d0 con 0x563e4dd4e2f0 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f876f7fe640 1 --2- 172.21.15.114:0/749083066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd4e2f0 0x563e4e127cf0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8760100fb0 tx=0x7f8760003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123e33ad1a6ccb5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f874effd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f874effd640 1 -- 172.21.15.114:0/749083066 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760103040 con 0x563e4dd4e2f0 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f874effd640 1 -- 172.21.15.114:0/749083066 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f87601031e0 con 0x563e4dd4e2f0 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f877cd87740 1 -- 172.21.15.114:0/749083066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8764032070 con 0x563e4dd4e2f0 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f874effd640 1 -- 172.21.15.114:0/749083066 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8760049040 con 0x563e4dd4e2f0 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f877cd87740 10 client.15447 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f877cd87740 1 -- 172.21.15.114:0/749083066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563e4e19ca10 con 0x563e4dd4e2f0 2024-05-27T07:40:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f874effd640 1 -- 172.21.15.114:0/749083066 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8764032070 con 0x563e4dd4e2f0 2024-05-27T07:40:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f874effd640 1 -- 172.21.15.114:0/749083066 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f87600d2d90 con 0x563e4dd4e2f0 2024-05-27T07:40:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f877cd87740 10 client.15447 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f877cd87740 10 client.15447 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f877cd87740 20 client.15447 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f877cd87740 1 --2- 172.21.15.114:0/749083066 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563e4dd0b1c0 0x563e4e19f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f877cd87740 4 client.15447 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:40:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.622+0000 7f877cd87740 1 -- 172.21.15.114:0/749083066 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563e4e086360 con 0x563e4dd0b1c0 2024-05-27T07:40:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.623+0000 7f876effd640 1 --2- 172.21.15.114:0/749083066 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563e4dd0b1c0 0x563e4e19f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.623+0000 7f874effd640 10 client.15447 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.623+0000 7f876effd640 1 --2- 172.21.15.114:0/749083066 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563e4dd0b1c0 0x563e4e19f180 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=369a811e58bca9a0 server_cookie=2d73893d42a0948d in_seq=0 out_seq=0 2024-05-27T07:40:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.631+0000 7f874effd640 1 -- 172.21.15.114:0/749083066 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x563e4e086360 con 0x563e4dd0b1c0 2024-05-27T07:40:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.631+0000 7f874effd640 10 client.15447 handle_command_reply: tid=0 2024-05-27T07:40:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.631+0000 7f877cd87740 1 client.15447 shutdown 2024-05-27T07:40:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f877cd87740 1 -- 172.21.15.114:0/749083066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd4e2f0 msgr2=0x563e4e127cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f877cd87740 1 --2- 172.21.15.114:0/749083066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd4e2f0 0x563e4e127cf0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8760100fb0 tx=0x7f8760003e80 comp rx=0 tx=0).stop 2024-05-27T07:40:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f877cd87740 1 -- 172.21.15.114:0/749083066 shutdown_connections 2024-05-27T07:40:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f877cd87740 1 --2- 172.21.15.114:0/749083066 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563e4dd0b1c0 0x563e4e19f180 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f877cd87740 1 --2- 172.21.15.114:0/749083066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e4dd771b0 0x563e4e19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f874effd640 0 client.15447 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f877cd87740 1 --2- 172.21.15.114:0/749083066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4dd564d0 0x563e4e128230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f877cd87740 1 --2- 172.21.15.114:0/749083066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e4dd4e2f0 0x563e4e127cf0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f877cd87740 1 -- 172.21.15.114:0/749083066 >> 172.21.15.114:0/749083066 conn(0x563e4dd6b300 msgr2=0x563e4e121310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f877cd87740 1 -- 172.21.15.114:0/749083066 shutdown_connections 2024-05-27T07:40:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f877cd87740 1 -- 172.21.15.114:0/749083066 wait complete. 2024-05-27T07:40:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f877cd87740 20 client.15447 trim_cache size 0 max 16384 2024-05-27T07:40:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:40:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:40:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:40:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f8774b9d640 1 -- 172.21.15.114:0/3703352457 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87580b2a70 msgr2=0x7f87580b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f8774b9d640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87580b2a70 0x7f87580b4f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x563e4e115700 tx=0x7f876800e040 comp rx=0 tx=0).stop 2024-05-27T07:40:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f8774b9d640 1 -- 172.21.15.114:0/3703352457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8770104220 msgr2=0x7f8770074fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f8774b9d640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8770104220 0x7f8770074fc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8760031a90 tx=0x7f87600042e0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f8774b9d640 1 -- 172.21.15.114:0/3703352457 shutdown_connections 2024-05-27T07:40:35.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f8774b9d640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87580b2a70 0x7f87580b4f80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f8774b9d640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8770109f70 0x7f8770075a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f8774b9d640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8770109ad0 0x7f8770075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f8774b9d640 1 --2- 172.21.15.114:0/3703352457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8770104220 0x7f8770074fc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:35.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.632+0000 7f8774b9d640 1 -- 172.21.15.114:0/3703352457 >> 172.21.15.114:0/3703352457 conn(0x7f87700ff330 msgr2=0x7f87700ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:35.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.633+0000 7f8774b9d640 1 -- 172.21.15.114:0/3703352457 shutdown_connections 2024-05-27T07:40:35.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:35.633+0000 7f8774b9d640 1 -- 172.21.15.114:0/3703352457 wait complete. 2024-05-27T07:40:35.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:35 smithi163 ceph-mon[36451]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-05-27T07:40:35.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:35 smithi186 ceph-mon[36647]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-05-27T07:40:37.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:37 smithi114 ceph-mon[32038]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-27T07:40:37.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:37 smithi163 ceph-mon[36451]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-27T07:40:37.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:37 smithi186 ceph-mon[36647]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-27T07:40:39.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:39 smithi114 ceph-mon[32038]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-05-27T07:40:39.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:39 smithi163 ceph-mon[36451]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-05-27T07:40:39.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:39 smithi186 ceph-mon[36647]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-05-27T07:40:41.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:40:41.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:41 smithi114 ceph-mon[32038]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-05-27T07:40:41.615 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:41.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:41 smithi163 ceph-mon[36451]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-05-27T07:40:41.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:41 smithi186 ceph-mon[36647]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-05-27T07:40:43.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.353+0000 7f20db73c700 1 -- 172.21.15.114:0/3155667991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d4105dc0 msgr2=0x7f20d410e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:43.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.353+0000 7f20db73c700 1 --2- 172.21.15.114:0/3155667991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d4105dc0 0x7f20d410e310 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f20c4009990 tx=0x7f20c4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.353+0000 7f20db73c700 1 -- 172.21.15.114:0/3155667991 shutdown_connections 2024-05-27T07:40:43.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.353+0000 7f20db73c700 1 --2- 172.21.15.114:0/3155667991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20d410e8e0 0x7f20d4110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.353+0000 7f20db73c700 1 --2- 172.21.15.114:0/3155667991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20d4105470 0x7f20d4105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.353+0000 7f20db73c700 1 --2- 172.21.15.114:0/3155667991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d4105dc0 0x7f20d410e310 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.353+0000 7f20db73c700 1 -- 172.21.15.114:0/3155667991 >> 172.21.15.114:0/3155667991 conn(0x7f20d40fef20 msgr2=0x7f20d4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:43.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.353+0000 7f20db73c700 1 -- 172.21.15.114:0/3155667991 shutdown_connections 2024-05-27T07:40:43.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.353+0000 7f20db73c700 1 -- 172.21.15.114:0/3155667991 wait complete. 2024-05-27T07:40:43.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20db73c700 1 Processor -- start 2024-05-27T07:40:43.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20db73c700 1 -- start start 2024-05-27T07:40:43.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20db73c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20d4105470 0x7f20d419adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20db73c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d4105dc0 0x7f20d419b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20db73c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20d410e8e0 0x7f20d419f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20db73c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f20d4114440 con 0x7f20d4105dc0 2024-05-27T07:40:43.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20db73c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f20d41142c0 con 0x7f20d410e8e0 2024-05-27T07:40:43.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20db73c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f20d41145c0 con 0x7f20d4105470 2024-05-27T07:40:43.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20d8cd7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d4105dc0 0x7f20d419b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20d8cd7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d4105dc0 0x7f20d419b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54632/0 (socket says 172.21.15.114:54632) 2024-05-27T07:40:43.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20d8cd7700 1 -- 172.21.15.114:0/3369181886 learned_addr learned my addr 172.21.15.114:0/3369181886 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:43.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20d94d8700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20d4105470 0x7f20d419adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20d9cd9700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20d410e8e0 0x7f20d419f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20d8cd7700 1 -- 172.21.15.114:0/3369181886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20d4105470 msgr2=0x7f20d419adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:43.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20d4105470 0x7f20d419adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20d8cd7700 1 -- 172.21.15.114:0/3369181886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20d410e8e0 msgr2=0x7f20d419f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:43.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20d410e8e0 0x7f20d419f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20d8cd7700 1 -- 172.21.15.114:0/3369181886 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d419fbd0 con 0x7f20d4105dc0 2024-05-27T07:40:43.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.354+0000 7f20d9cd9700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20d410e8e0 0x7f20d419f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:43.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.355+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d4105dc0 0x7f20d419b350 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f20c4000c00 tx=0x7f20c4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:43.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.355+0000 7f20ca7fc700 1 -- 172.21.15.114:0/3369181886 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c4027070 con 0x7f20d4105dc0 2024-05-27T07:40:43.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.355+0000 7f20db73c700 1 -- 172.21.15.114:0/3369181886 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d419fe60 con 0x7f20d4105dc0 2024-05-27T07:40:43.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.355+0000 7f20ca7fc700 1 -- 172.21.15.114:0/3369181886 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20c4022470 con 0x7f20d4105dc0 2024-05-27T07:40:43.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.355+0000 7f20db73c700 1 -- 172.21.15.114:0/3369181886 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d41a0390 con 0x7f20d4105dc0 2024-05-27T07:40:43.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.355+0000 7f20ca7fc700 1 -- 172.21.15.114:0/3369181886 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c4017650 con 0x7f20d4105dc0 2024-05-27T07:40:43.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.359+0000 7f20ea151b80 1 Processor -- start 2024-05-27T07:40:43.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.359+0000 7f20ea151b80 1 -- start start 2024-05-27T07:40:43.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.359+0000 7f20ea151b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a7649e40 0x55b9a77ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.359+0000 7f20ea151b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77cf070 0x55b9a77c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.359+0000 7f20ea151b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 0x55b9a77c9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.359+0000 7f20ea151b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b9a76f6d70 con 0x55b9a77c9120 2024-05-27T07:40:43.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.359+0000 7f20ea151b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b9a76e6910 con 0x55b9a77cf070 2024-05-27T07:40:43.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.359+0000 7f20ea151b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b9a76ea950 con 0x55b9a7649e40 2024-05-27T07:40:43.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.359+0000 7f20ca7fc700 1 -- 172.21.15.114:0/3369181886 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7f20c402a070 con 0x7f20d4105dc0 2024-05-27T07:40:43.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.359+0000 7f20d8cd7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 0x55b9a77c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.359+0000 7f20d8cd7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 0x55b9a77c9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54648/0 (socket says 172.21.15.114:54648) 2024-05-27T07:40:43.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.359+0000 7f20d94d8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77cf070 0x55b9a77c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.359+0000 7f20d8cd7700 1 -- 172.21.15.114:0/60752691 learned_addr learned my addr 172.21.15.114:0/60752691 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:43.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.360+0000 7f20d9cd9700 1 --2- 172.21.15.114:0/60752691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a7649e40 0x55b9a77ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.360+0000 7f20d8cd7700 1 -- 172.21.15.114:0/60752691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a7649e40 msgr2=0x55b9a77ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:43.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.360+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/60752691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a7649e40 0x55b9a77ceb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.360+0000 7f20d8cd7700 1 -- 172.21.15.114:0/60752691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77cf070 msgr2=0x55b9a77c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:43.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.360+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/60752691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77cf070 0x55b9a77c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.360+0000 7f20d8cd7700 1 -- 172.21.15.114:0/60752691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9a768ffb0 con 0x55b9a77c9120 2024-05-27T07:40:43.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.360+0000 7f20d9cd9700 1 --2- 172.21.15.114:0/60752691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a7649e40 0x55b9a77ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:43.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.360+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/60752691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 0x55b9a77c9530 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f20c40e1130 tx=0x7f20c40ec3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78fc6f4625eb023a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:43.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.360+0000 7f20bf7fe700 1 -- 172.21.15.114:0/60752691 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40ece20 con 0x55b9a77c9120 2024-05-27T07:40:43.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.360+0000 7f20ca7fc700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f20c00b2b10 0x7f20c00b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.360+0000 7f20bf7fe700 1 -- 172.21.15.114:0/60752691 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20c40f1070 con 0x55b9a77c9120 2024-05-27T07:40:43.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.360+0000 7f20bf7fe700 1 -- 172.21.15.114:0/60752691 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40f57d0 con 0x55b9a77c9120 2024-05-27T07:40:43.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.361+0000 7f20d9cd9700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f20c00b2b10 0x7f20c00b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.361+0000 7f20ca7fc700 1 -- 172.21.15.114:0/3369181886 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f20c40e3e00 con 0x7f20d4105dc0 2024-05-27T07:40:43.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.361+0000 7f20ea151b80 1 -- 172.21.15.114:0/60752691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 msgr2=0x55b9a77c9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:43.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.361+0000 7f20d9cd9700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f20c00b2b10 0x7f20c00b5020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f20cc007500 tx=0x7f20cc009040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:43.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.361+0000 7f20ea151b80 1 --2- 172.21.15.114:0/60752691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 0x55b9a77c9530 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f20c40e1130 tx=0x7f20c40ec3c0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.362+0000 7f20ea151b80 1 -- 172.21.15.114:0/60752691 shutdown_connections 2024-05-27T07:40:43.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.362+0000 7f20ea151b80 1 --2- 172.21.15.114:0/60752691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 0x55b9a77c9530 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.362+0000 7f20ea151b80 1 --2- 172.21.15.114:0/60752691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a7649e40 0x55b9a77ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.362+0000 7f20ea151b80 1 --2- 172.21.15.114:0/60752691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77cf070 0x55b9a77c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.362+0000 7f20ea151b80 1 -- 172.21.15.114:0/60752691 >> 172.21.15.114:0/60752691 conn(0x55b9a764c770 msgr2=0x55b9a77c3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:43.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.362+0000 7f20ea151b80 1 -- 172.21.15.114:0/60752691 shutdown_connections 2024-05-27T07:40:43.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.362+0000 7f20ea151b80 1 -- 172.21.15.114:0/60752691 wait complete. 2024-05-27T07:40:43.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.362+0000 7f20ea151b80 1 -- start start 2024-05-27T07:40:43.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.362+0000 7f20ea151b80 1 Processor -- start 2024-05-27T07:40:43.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20ea151b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 0x55b9a77d2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20ea151b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77c9120 0x55b9a77d2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20ea151b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a77cf070 0x55b9a77d32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20ea151b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b9a75ecfa0 con 0x55b9a7649e40 2024-05-27T07:40:43.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20ea151b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b9a764eda0 con 0x55b9a77c9120 2024-05-27T07:40:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20ea151b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b9a77d5920 con 0x55b9a77cf070 2024-05-27T07:40:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20ea151b80 10 client.0 _ll_register_callbacks cb 0x55b9a767d5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20d94d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 0x55b9a77d2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20d94d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 0x55b9a77d2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54656/0 (socket says 172.21.15.114:54656) 2024-05-27T07:40:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20d94d8700 1 -- 172.21.15.114:0/2316696309 learned_addr learned my addr 172.21.15.114:0/2316696309 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/2316696309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77c9120 0x55b9a77d2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20d9cd9700 1 --2- 172.21.15.114:0/2316696309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a77cf070 0x55b9a77d32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20d94d8700 1 -- 172.21.15.114:0/2316696309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a77cf070 msgr2=0x55b9a77d32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20d94d8700 1 --2- 172.21.15.114:0/2316696309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a77cf070 0x55b9a77d32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20d94d8700 1 -- 172.21.15.114:0/2316696309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77c9120 msgr2=0x55b9a77d2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20d94d8700 1 --2- 172.21.15.114:0/2316696309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77c9120 0x55b9a77d2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20d94d8700 1 -- 172.21.15.114:0/2316696309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9a76368d0 con 0x55b9a7649e40 2024-05-27T07:40:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20d94d8700 1 --2- 172.21.15.114:0/2316696309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 0x55b9a77d2870 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f20d0010a20 tx=0x7f20d0010d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e374b099761d5452 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:43.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.363+0000 7f20bcff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:43.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20bcff9700 1 -- 172.21.15.114:0/2316696309 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0010e60 con 0x55b9a7649e40 2024-05-27T07:40:43.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20bcff9700 1 -- 172.21.15.114:0/2316696309 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f20d0004260 con 0x55b9a7649e40 2024-05-27T07:40:43.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20ea151b80 1 -- 172.21.15.114:0/2316696309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20c40f1070 con 0x55b9a7649e40 2024-05-27T07:40:43.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20bcff9700 1 -- 172.21.15.114:0/2316696309 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f20d0004460 con 0x55b9a7649e40 2024-05-27T07:40:43.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20ea151b80 10 client.15459 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:43.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20ea151b80 1 -- 172.21.15.114:0/2316696309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b9a76a0760 con 0x55b9a7649e40 2024-05-27T07:40:43.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20bcff9700 1 -- 172.21.15.114:0/2316696309 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20c40f1070 con 0x55b9a7649e40 2024-05-27T07:40:43.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20bcff9700 1 -- 172.21.15.114:0/2316696309 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f20d0003b70 con 0x55b9a7649e40 2024-05-27T07:40:43.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20ea151b80 10 client.15459 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:43.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20ea151b80 10 client.15459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:43.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20ea151b80 20 client.15459 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:43.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20ea151b80 1 --2- 172.21.15.114:0/2316696309 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9a77e28d0 0x55b9a77e4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20ea151b80 4 client.15459 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:43.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.364+0000 7f20ea151b80 1 -- 172.21.15.114:0/2316696309 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9a76f8cc0 con 0x55b9a77e28d0 2024-05-27T07:40:43.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.365+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/2316696309 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9a77e28d0 0x55b9a77e4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.365+0000 7f20bcff9700 10 client.15459 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:43.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.365+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/2316696309 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9a77e28d0 0x55b9a77e4cf0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecf13443eb7ae7f server_cookie=64cedd07fee8abf6 in_seq=0 out_seq=0 2024-05-27T07:40:43.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20bcff9700 1 -- 172.21.15.114:0/2316696309 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b9a76f8cc0 con 0x55b9a77e28d0 2024-05-27T07:40:43.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20bcff9700 10 client.15459 handle_command_reply: tid=0 2024-05-27T07:40:43.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20ea151b80 1 client.15459 shutdown 2024-05-27T07:40:43.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20ea151b80 1 -- 172.21.15.114:0/2316696309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 msgr2=0x55b9a77d2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:43.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20ea151b80 1 --2- 172.21.15.114:0/2316696309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 0x55b9a77d2870 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f20d0010a20 tx=0x7f20d0010d30 comp rx=0 tx=0).stop 2024-05-27T07:40:43.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20ea151b80 1 -- 172.21.15.114:0/2316696309 shutdown_connections 2024-05-27T07:40:43.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20ea151b80 1 --2- 172.21.15.114:0/2316696309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a77cf070 0x55b9a77d32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20ea151b80 1 --2- 172.21.15.114:0/2316696309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 0x55b9a77d2870 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20ea151b80 1 --2- 172.21.15.114:0/2316696309 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9a77e28d0 0x55b9a77e4cf0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20ea151b80 1 --2- 172.21.15.114:0/2316696309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77c9120 0x55b9a77d2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20ea151b80 1 -- 172.21.15.114:0/2316696309 >> 172.21.15.114:0/2316696309 conn(0x55b9a764c770 msgr2=0x55b9a77c7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:43.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20bcff9700 0 client.15459 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:43.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20ea151b80 1 -- 172.21.15.114:0/2316696309 shutdown_connections 2024-05-27T07:40:43.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20ea151b80 1 -- 172.21.15.114:0/2316696309 wait complete. 2024-05-27T07:40:43.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.366+0000 7f20ea151b80 20 client.15459 trim_cache size 0 max 16384 2024-05-27T07:40:43.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.387+0000 7f20ea151b80 1 Processor -- start 2024-05-27T07:40:43.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.387+0000 7f20ea151b80 1 -- start start 2024-05-27T07:40:43.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.387+0000 7f20ea151b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 0x55b9a77c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.387+0000 7f20ea151b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77c9120 0x55b9a77d3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.387+0000 7f20ea151b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a77cf070 0x55b9a77de980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.387+0000 7f20ea151b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f20d41b0640 con 0x55b9a7649e40 2024-05-27T07:40:43.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.387+0000 7f20ea151b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f20d41b2ec0 con 0x55b9a77c9120 2024-05-27T07:40:43.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.387+0000 7f20ea151b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f20d41b0cb0 con 0x55b9a77cf070 2024-05-27T07:40:43.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.387+0000 7f20d94d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 0x55b9a77c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.387+0000 7f20d94d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 0x55b9a77c8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54668/0 (socket says 172.21.15.114:54668) 2024-05-27T07:40:43.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.387+0000 7f20d94d8700 1 -- 172.21.15.114:0/1291255530 learned_addr learned my addr 172.21.15.114:0/1291255530 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:43.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.387+0000 7f20d9cd9700 1 --2- 172.21.15.114:0/1291255530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a77cf070 0x55b9a77de980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.387+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/1291255530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77c9120 0x55b9a77d3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.388+0000 7f20d94d8700 1 -- 172.21.15.114:0/1291255530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a77cf070 msgr2=0x55b9a77de980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:43.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.388+0000 7f20d94d8700 1 --2- 172.21.15.114:0/1291255530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a77cf070 0x55b9a77de980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.388+0000 7f20d94d8700 1 -- 172.21.15.114:0/1291255530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77c9120 msgr2=0x55b9a77d3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:43.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.388+0000 7f20d94d8700 1 --2- 172.21.15.114:0/1291255530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77c9120 0x55b9a77d3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.388+0000 7f20d94d8700 1 -- 172.21.15.114:0/1291255530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9a77d0e00 con 0x55b9a7649e40 2024-05-27T07:40:43.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.388+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/1291255530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77c9120 0x55b9a77d3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:43.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.388+0000 7f20d94d8700 1 --2- 172.21.15.114:0/1291255530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 0x55b9a77c8be0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f20d001a4a0 tx=0x7f20d0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1917130c12c54f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:43.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.388+0000 7f20beffd700 1 -- 172.21.15.114:0/1291255530 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0005070 con 0x55b9a7649e40 2024-05-27T07:40:43.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.388+0000 7f20beffd700 1 -- 172.21.15.114:0/1291255530 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20d00056c0 con 0x55b9a7649e40 2024-05-27T07:40:43.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.388+0000 7f20beffd700 1 -- 172.21.15.114:0/1291255530 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d000f960 con 0x55b9a7649e40 2024-05-27T07:40:43.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.389+0000 7f20ea151b80 1 -- 172.21.15.114:0/1291255530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 msgr2=0x55b9a77c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:43.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.389+0000 7f20ea151b80 1 --2- 172.21.15.114:0/1291255530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 0x55b9a77c8be0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f20d001a4a0 tx=0x7f20d0004360 comp rx=0 tx=0).stop 2024-05-27T07:40:43.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.389+0000 7f20ea151b80 1 -- 172.21.15.114:0/1291255530 shutdown_connections 2024-05-27T07:40:43.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.389+0000 7f20ea151b80 1 --2- 172.21.15.114:0/1291255530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a77cf070 0x55b9a77de980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.389+0000 7f20ea151b80 1 --2- 172.21.15.114:0/1291255530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a7649e40 0x55b9a77c8be0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.389+0000 7f20ea151b80 1 --2- 172.21.15.114:0/1291255530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77c9120 0x55b9a77d3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.389+0000 7f20ea151b80 1 -- 172.21.15.114:0/1291255530 >> 172.21.15.114:0/1291255530 conn(0x55b9a764c770 msgr2=0x55b9a77cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:43.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.389+0000 7f20ea151b80 1 -- 172.21.15.114:0/1291255530 shutdown_connections 2024-05-27T07:40:43.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.389+0000 7f20ea151b80 1 -- 172.21.15.114:0/1291255530 wait complete. 2024-05-27T07:40:43.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.389+0000 7f20ea151b80 1 -- start start 2024-05-27T07:40:43.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20ea151b80 1 Processor -- start 2024-05-27T07:40:43.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20ea151b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a7649e40 0x55b9a77db210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20ea151b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 0x55b9a77e9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20ea151b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77cf070 0x55b9a77ea190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20ea151b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f20d41b1380 con 0x55b9a77c9120 2024-05-27T07:40:43.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20ea151b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f20d41b27f0 con 0x55b9a77cf070 2024-05-27T07:40:43.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20ea151b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f20d41b1a50 con 0x55b9a7649e40 2024-05-27T07:40:43.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20ea151b80 10 client.0 _ll_register_callbacks cb 0x55b9a767d5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:43.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20d8cd7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 0x55b9a77e9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20d94d8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a7649e40 0x55b9a77db210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20d8cd7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 0x55b9a77e9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54676/0 (socket says 172.21.15.114:54676) 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20d8cd7700 1 -- 172.21.15.114:0/957244962 learned_addr learned my addr 172.21.15.114:0/957244962 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20d94d8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a7649e40 0x55b9a77db210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:41222/0 (socket says 172.21.15.114:41222) 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.390+0000 7f20d9cd9700 1 --2- 172.21.15.114:0/957244962 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77cf070 0x55b9a77ea190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20d8cd7700 1 -- 172.21.15.114:0/957244962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a7649e40 msgr2=0x55b9a77db210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/957244962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a7649e40 0x55b9a77db210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20d8cd7700 1 -- 172.21.15.114:0/957244962 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77cf070 msgr2=0x55b9a77ea190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/957244962 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77cf070 0x55b9a77ea190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20d8cd7700 1 -- 172.21.15.114:0/957244962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9a77ea7d0 con 0x55b9a77c9120 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20d94d8700 1 --2- 172.21.15.114:0/957244962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a7649e40 0x55b9a77db210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20d8cd7700 1 --2- 172.21.15.114:0/957244962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 0x55b9a77e9c50 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f20c4005f50 tx=0x7f20c40f6f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4276a7076dc627cb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20bcff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20bcff9700 1 -- 172.21.15.114:0/957244962 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40a6a30 con 0x55b9a77c9120 2024-05-27T07:40:43.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20ea151b80 1 -- 172.21.15.114:0/957244962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b9a76f24a0 con 0x55b9a77c9120 2024-05-27T07:40:43.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20bcff9700 1 -- 172.21.15.114:0/957244962 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f20c40e7600 con 0x55b9a77c9120 2024-05-27T07:40:43.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20bcff9700 1 -- 172.21.15.114:0/957244962 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f20c40a6300 con 0x55b9a77c9120 2024-05-27T07:40:43.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20ea151b80 10 client.15465 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:43.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20ea151b80 1 -- 172.21.15.114:0/957244962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b9a784d140 con 0x55b9a77c9120 2024-05-27T07:40:43.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.391+0000 7f20bcff9700 1 -- 172.21.15.114:0/957244962 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b9a76f24a0 con 0x55b9a77c9120 2024-05-27T07:40:43.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.392+0000 7f20bcff9700 1 -- 172.21.15.114:0/957244962 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f20c40a6e00 con 0x55b9a77c9120 2024-05-27T07:40:43.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.392+0000 7f20ea151b80 10 client.15465 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:43.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.392+0000 7f20ea151b80 10 client.15465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:43.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.392+0000 7f20ea151b80 20 client.15465 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:43.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.392+0000 7f20ea151b80 1 --2- 172.21.15.114:0/957244962 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9a784d6a0 0x55b9a784fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:43.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.392+0000 7f20ea151b80 4 client.15465 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:40:43.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.392+0000 7f20ea151b80 1 -- 172.21.15.114:0/957244962 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b9a7569250 con 0x55b9a784d6a0 2024-05-27T07:40:43.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.392+0000 7f20d94d8700 1 --2- 172.21.15.114:0/957244962 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9a784d6a0 0x55b9a784fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:43.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20bcff9700 10 client.15465 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:43.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20d94d8700 1 --2- 172.21.15.114:0/957244962 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9a784d6a0 0x55b9a784fac0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=204cd81da8ce3575 server_cookie=2e89b98e6dc441ba in_seq=0 out_seq=0 2024-05-27T07:40:43.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20bcff9700 1 -- 172.21.15.114:0/957244962 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b9a7569250 con 0x55b9a784d6a0 2024-05-27T07:40:43.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20bcff9700 10 client.15465 handle_command_reply: tid=0 2024-05-27T07:40:43.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20ea151b80 1 client.15465 shutdown 2024-05-27T07:40:43.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20ea151b80 1 -- 172.21.15.114:0/957244962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 msgr2=0x55b9a77e9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:43.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20ea151b80 1 --2- 172.21.15.114:0/957244962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 0x55b9a77e9c50 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f20c4005f50 tx=0x7f20c40f6f60 comp rx=0 tx=0).stop 2024-05-27T07:40:43.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20ea151b80 1 -- 172.21.15.114:0/957244962 shutdown_connections 2024-05-27T07:40:43.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20ea151b80 1 --2- 172.21.15.114:0/957244962 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9a784d6a0 0x55b9a784fac0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20ea151b80 1 --2- 172.21.15.114:0/957244962 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9a77cf070 0x55b9a77ea190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20ea151b80 1 --2- 172.21.15.114:0/957244962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9a7649e40 0x55b9a77db210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20ea151b80 1 --2- 172.21.15.114:0/957244962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9a77c9120 0x55b9a77e9c50 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20ea151b80 1 -- 172.21.15.114:0/957244962 >> 172.21.15.114:0/957244962 conn(0x55b9a764c770 msgr2=0x55b9a77bd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:43.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20bcff9700 0 client.15465 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:43.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20ea151b80 1 -- 172.21.15.114:0/957244962 shutdown_connections 2024-05-27T07:40:43.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.393+0000 7f20ea151b80 1 -- 172.21.15.114:0/957244962 wait complete. 2024-05-27T07:40:43.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.394+0000 7f20ea151b80 20 client.15465 trim_cache size 0 max 16384 2024-05-27T07:40:43.395 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:40:43.395 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:40:43.395 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:40:43.395 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:40:43.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.394+0000 7f20db73c700 1 -- 172.21.15.114:0/3369181886 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f20c00b2b10 msgr2=0x7f20c00b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:43.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.394+0000 7f20db73c700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f20c00b2b10 0x7f20c00b5020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f20cc007500 tx=0x7f20cc009040 comp rx=0 tx=0).stop 2024-05-27T07:40:43.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.394+0000 7f20db73c700 1 -- 172.21.15.114:0/3369181886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d4105dc0 msgr2=0x7f20d419b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:43.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.394+0000 7f20db73c700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d4105dc0 0x7f20d419b350 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f20c4000c00 tx=0x7f20c4009580 comp rx=0 tx=0).stop 2024-05-27T07:40:43.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.394+0000 7f20db73c700 1 -- 172.21.15.114:0/3369181886 shutdown_connections 2024-05-27T07:40:43.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.394+0000 7f20db73c700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f20c00b2b10 0x7f20c00b5020 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.394+0000 7f20db73c700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20d410e8e0 0x7f20d419f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.394+0000 7f20db73c700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20d4105470 0x7f20d419adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.394+0000 7f20db73c700 1 --2- 172.21.15.114:0/3369181886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d4105dc0 0x7f20d419b350 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:43.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.394+0000 7f20db73c700 1 -- 172.21.15.114:0/3369181886 >> 172.21.15.114:0/3369181886 conn(0x7f20d40fef20 msgr2=0x7f20d40ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:43.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.394+0000 7f20db73c700 1 -- 172.21.15.114:0/3369181886 shutdown_connections 2024-05-27T07:40:43.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:43.395+0000 7f20db73c700 1 -- 172.21.15.114:0/3369181886 wait complete. 2024-05-27T07:40:43.521 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:43 smithi114 ceph-mon[32038]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-27T07:40:43.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:43 smithi163 ceph-mon[36451]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-27T07:40:43.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:43 smithi186 ceph-mon[36647]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-27T07:40:44.199 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:40:44.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:40:44.446 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:45.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:45 smithi114 ceph-mon[32038]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-27T07:40:45.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:45 smithi163 ceph-mon[36451]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-27T07:40:45.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:45 smithi186 ceph-mon[36647]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-27T07:40:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.299+0000 7fd683b38700 1 -- 172.21.15.114:0/1548643082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67c108730 msgr2=0x7fd67c110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:46.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.299+0000 7fd683b38700 1 --2- 172.21.15.114:0/1548643082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67c108730 0x7fd67c110090 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd6740099c0 tx=0x7fd674009cd0 comp rx=0 tx=0).stop 2024-05-27T07:40:46.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.299+0000 7fd683b38700 1 -- 172.21.15.114:0/1548643082 shutdown_connections 2024-05-27T07:40:46.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.299+0000 7fd683b38700 1 --2- 172.21.15.114:0/1548643082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67c108730 0x7fd67c110090 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:46.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.299+0000 7fd683b38700 1 --2- 172.21.15.114:0/1548643082 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd67c1034b0 0x7fd67c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:46.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.299+0000 7fd683b38700 1 --2- 172.21.15.114:0/1548643082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd67c107de0 0x7fd67c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:46.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.299+0000 7fd683b38700 1 -- 172.21.15.114:0/1548643082 >> 172.21.15.114:0/1548643082 conn(0x7fd67c0fede0 msgr2=0x7fd67c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:46.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.299+0000 7fd683b38700 1 -- 172.21.15.114:0/1548643082 shutdown_connections 2024-05-27T07:40:46.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.299+0000 7fd683b38700 1 -- 172.21.15.114:0/1548643082 wait complete. 2024-05-27T07:40:46.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.300+0000 7fd683b38700 1 Processor -- start 2024-05-27T07:40:46.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.300+0000 7fd683b38700 1 -- start start 2024-05-27T07:40:46.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.300+0000 7fd683b38700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd67c1034b0 0x7fd67c19be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:46.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.300+0000 7fd683b38700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd67c107de0 0x7fd67c19c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:46.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.300+0000 7fd683b38700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67c108730 0x7fd67c19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:46.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.300+0000 7fd683b38700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c113470 con 0x7fd67c108730 2024-05-27T07:40:46.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.300+0000 7fd683b38700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1132f0 con 0x7fd67c107de0 2024-05-27T07:40:46.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.300+0000 7fd683b38700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1135f0 con 0x7fd67c1034b0 2024-05-27T07:40:46.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.300+0000 7fd6818d4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd67c1034b0 0x7fd67c19be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:46.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.300+0000 7fd6820d5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67c108730 0x7fd67c19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:46.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.300+0000 7fd6810d3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd67c107de0 0x7fd67c19c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:46.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.300+0000 7fd6818d4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd67c1034b0 0x7fd67c19be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:50494/0 (socket says 172.21.15.114:50494) 2024-05-27T07:40:46.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.300+0000 7fd6818d4700 1 -- 172.21.15.114:0/3496233771 learned_addr learned my addr 172.21.15.114:0/3496233771 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:46.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.301+0000 7fd6820d5700 1 -- 172.21.15.114:0/3496233771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd67c1034b0 msgr2=0x7fd67c19be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:46.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.301+0000 7fd6820d5700 1 --2- 172.21.15.114:0/3496233771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd67c1034b0 0x7fd67c19be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:46.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.301+0000 7fd6820d5700 1 -- 172.21.15.114:0/3496233771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd67c107de0 msgr2=0x7fd67c19c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:46.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.301+0000 7fd6820d5700 1 --2- 172.21.15.114:0/3496233771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd67c107de0 0x7fd67c19c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:46.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.301+0000 7fd6820d5700 1 -- 172.21.15.114:0/3496233771 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67c1a7180 con 0x7fd67c108730 2024-05-27T07:40:46.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.301+0000 7fd6820d5700 1 --2- 172.21.15.114:0/3496233771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67c108730 0x7fd67c19c8d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd674006010 tx=0x7fd674005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:46.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.301+0000 7fd67affd700 1 -- 172.21.15.114:0/3496233771 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd674027070 con 0x7fd67c108730 2024-05-27T07:40:46.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.301+0000 7fd683b38700 1 -- 172.21.15.114:0/3496233771 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd67c1a7410 con 0x7fd67c108730 2024-05-27T07:40:46.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.301+0000 7fd67affd700 1 -- 172.21.15.114:0/3496233771 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd674003f90 con 0x7fd67c108730 2024-05-27T07:40:46.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.301+0000 7fd67affd700 1 -- 172.21.15.114:0/3496233771 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd674022be0 con 0x7fd67c108730 2024-05-27T07:40:46.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.301+0000 7fd683b38700 1 -- 172.21.15.114:0/3496233771 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd67c1a78c0 con 0x7fd67c108730 2024-05-27T07:40:46.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.302+0000 7fd683b38700 1 -- 172.21.15.114:0/3496233771 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd67c10b7c0 con 0x7fd67c108730 2024-05-27T07:40:46.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.304+0000 7fd67affd700 1 -- 172.21.15.114:0/3496233771 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7fd67402a070 con 0x7fd67c108730 2024-05-27T07:40:46.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.304+0000 7fd67affd700 1 --2- 172.21.15.114:0/3496233771 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd6640b2b90 0x7fd6640b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:46.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.304+0000 7fd67affd700 1 -- 172.21.15.114:0/3496233771 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fd6740e3b70 con 0x7fd67c108730 2024-05-27T07:40:46.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.304+0000 7fd6818d4700 1 --2- 172.21.15.114:0/3496233771 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd6640b2b90 0x7fd6640b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:46.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.305+0000 7fd6818d4700 1 --2- 172.21.15.114:0/3496233771 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd6640b2b90 0x7fd6640b50a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009ad0 tx=0x7fd66c009210 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:46.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.309+0000 7fd67affd700 1 -- 172.21.15.114:0/3496233771 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd674013070 con 0x7fd67c108730 2024-05-27T07:40:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.565+0000 7fd683b38700 1 -- 172.21.15.114:0/3496233771 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd67c0512b0 con 0x7fd67c108730 2024-05-27T07:40:46.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.566+0000 7fd67affd700 1 -- 172.21.15.114:0/3496233771 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11827 (secure 0 0 0) 0x7fd6740aafe0 con 0x7fd67c108730 2024-05-27T07:40:46.563 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:40:46.563 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:37:06.645274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:40:46.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.568+0000 7fd683b38700 1 -- 172.21.15.114:0/3496233771 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd6640b2b90 msgr2=0x7fd6640b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:46.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.568+0000 7fd683b38700 1 --2- 172.21.15.114:0/3496233771 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd6640b2b90 0x7fd6640b50a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009ad0 tx=0x7fd66c009210 comp rx=0 tx=0).stop 2024-05-27T07:40:46.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.568+0000 7fd683b38700 1 -- 172.21.15.114:0/3496233771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67c108730 msgr2=0x7fd67c19c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:46.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.568+0000 7fd683b38700 1 --2- 172.21.15.114:0/3496233771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67c108730 0x7fd67c19c8d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd674006010 tx=0x7fd674005590 comp rx=0 tx=0).stop 2024-05-27T07:40:46.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.568+0000 7fd683b38700 1 -- 172.21.15.114:0/3496233771 shutdown_connections 2024-05-27T07:40:46.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.568+0000 7fd683b38700 1 --2- 172.21.15.114:0/3496233771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67c108730 0x7fd67c19c8d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:46.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.568+0000 7fd683b38700 1 --2- 172.21.15.114:0/3496233771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd67c1034b0 0x7fd67c19be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:46.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.568+0000 7fd683b38700 1 --2- 172.21.15.114:0/3496233771 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd6640b2b90 0x7fd6640b50a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:46.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.568+0000 7fd683b38700 1 --2- 172.21.15.114:0/3496233771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd67c107de0 0x7fd67c19c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:46.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.568+0000 7fd683b38700 1 -- 172.21.15.114:0/3496233771 >> 172.21.15.114:0/3496233771 conn(0x7fd67c0fede0 msgr2=0x7fd67c109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:46.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.568+0000 7fd683b38700 1 -- 172.21.15.114:0/3496233771 shutdown_connections 2024-05-27T07:40:46.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:46.568+0000 7fd683b38700 1 -- 172.21.15.114:0/3496233771 wait complete. 2024-05-27T07:40:46.567 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:40:47.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:40:47.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:47 smithi114 ceph-mon[32038]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-05-27T07:40:47.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:47 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3496233771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:40:47.665 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:47.893 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:47 smithi186 ceph-mon[36647]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-05-27T07:40:47.893 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:47 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3496233771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:40:47.893 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:47 smithi163 ceph-mon[36451]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-05-27T07:40:47.893 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:47 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3496233771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:40:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.734+0000 7f4b73bd5640 1 -- 172.21.15.114:0/4210980576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c109060 msgr2=0x7f4b6c102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.734+0000 7f4b73bd5640 1 --2- 172.21.15.114:0/4210980576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c109060 0x7f4b6c102ca0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009900 tx=0x7f4b5c02a410 comp rx=0 tx=0).stop 2024-05-27T07:40:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.734+0000 7f4b73bd5640 1 -- 172.21.15.114:0/4210980576 shutdown_connections 2024-05-27T07:40:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.734+0000 7f4b73bd5640 1 --2- 172.21.15.114:0/4210980576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c103310 0x7f4b6c10f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.734+0000 7f4b73bd5640 1 --2- 172.21.15.114:0/4210980576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c109060 0x7f4b6c102ca0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.734+0000 7f4b73bd5640 1 --2- 172.21.15.114:0/4210980576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b6c108bc0 0x7f4b6c102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.734+0000 7f4b73bd5640 1 -- 172.21.15.114:0/4210980576 >> 172.21.15.114:0/4210980576 conn(0x7f4b6c0782c0 msgr2=0x7f4b6c078700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.734+0000 7f4b73bd5640 1 -- 172.21.15.114:0/4210980576 shutdown_connections 2024-05-27T07:40:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.734+0000 7f4b73bd5640 1 -- 172.21.15.114:0/4210980576 wait complete. 2024-05-27T07:40:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.734+0000 7f4b73bd5640 1 Processor -- start 2024-05-27T07:40:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b73bd5640 1 -- start start 2024-05-27T07:40:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b73bd5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b6c103310 0x7f4b6c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b73bd5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c108bc0 0x7f4b6c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b73bd5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c109060 0x7f4b6c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b73bd5640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c111cb0 con 0x7f4b6c108bc0 2024-05-27T07:40:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b73bd5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c111b30 con 0x7f4b6c103310 2024-05-27T07:40:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b73bd5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c111e30 con 0x7f4b6c109060 2024-05-27T07:40:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b71170640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c108bc0 0x7f4b6c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b71170640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c108bc0 0x7f4b6c198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40882/0 (socket says 172.21.15.114:40882) 2024-05-27T07:40:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b71170640 1 -- 172.21.15.114:0/2449755172 learned_addr learned my addr 172.21.15.114:0/2449755172 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b71971640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b6c103310 0x7f4b6c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b72172640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c109060 0x7f4b6c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b71170640 1 -- 172.21.15.114:0/2449755172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c109060 msgr2=0x7f4b6c19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b71170640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c109060 0x7f4b6c19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b71170640 1 -- 172.21.15.114:0/2449755172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b6c103310 msgr2=0x7f4b6c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b71170640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b6c103310 0x7f4b6c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b71170640 1 -- 172.21.15.114:0/2449755172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c19d530 con 0x7f4b6c108bc0 2024-05-27T07:40:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b72172640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c109060 0x7f4b6c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b71971640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b6c103310 0x7f4b6c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.735+0000 7f4b71170640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c108bc0 0x7f4b6c198c30 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c006010 tx=0x7f4b5c004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.736+0000 7f4b62ffd640 1 -- 172.21.15.114:0/2449755172 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c03f030 con 0x7f4b6c108bc0 2024-05-27T07:40:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.736+0000 7f4b73bd5640 1 -- 172.21.15.114:0/2449755172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6c19d7c0 con 0x7f4b6c108bc0 2024-05-27T07:40:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.736+0000 7f4b62ffd640 1 -- 172.21.15.114:0/2449755172 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b5c003aa0 con 0x7f4b6c108bc0 2024-05-27T07:40:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.736+0000 7f4b62ffd640 1 -- 172.21.15.114:0/2449755172 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c035070 con 0x7f4b6c108bc0 2024-05-27T07:40:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.736+0000 7f4b73bd5640 1 -- 172.21.15.114:0/2449755172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b6c19dcb0 con 0x7f4b6c108bc0 2024-05-27T07:40:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.738+0000 7f4b7ab65740 1 Processor -- start 2024-05-27T07:40:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b7ab65740 1 -- start start 2024-05-27T07:40:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b62ffd640 1 -- 172.21.15.114:0/2449755172 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7f4b5c042030 con 0x7f4b6c108bc0 2024-05-27T07:40:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b7ab65740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39721b0 0x5574d3d133d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b7ab65740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 0x5574d3d13910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b7ab65740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39514d0 0x5574d3d13e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b7ab65740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5574d3c85c30 con 0x5574d39492f0 2024-05-27T07:40:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b7ab65740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5574d3c757d0 con 0x5574d39721b0 2024-05-27T07:40:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b7ab65740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5574d3c79810 con 0x5574d39514d0 2024-05-27T07:40:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b71971640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 0x5574d3d13910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b71971640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 0x5574d3d13910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40890/0 (socket says 172.21.15.114:40890) 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b71971640 1 -- 172.21.15.114:0/1281094924 learned_addr learned my addr 172.21.15.114:0/1281094924 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b72172640 1 --2- 172.21.15.114:0/1281094924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39721b0 0x5574d3d133d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b71170640 1 --2- 172.21.15.114:0/1281094924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39514d0 0x5574d3d13e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b71971640 1 -- 172.21.15.114:0/1281094924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39514d0 msgr2=0x5574d3d13e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b71971640 1 --2- 172.21.15.114:0/1281094924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39514d0 0x5574d3d13e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b71971640 1 -- 172.21.15.114:0/1281094924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39721b0 msgr2=0x5574d3d133d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b71971640 1 --2- 172.21.15.114:0/1281094924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39721b0 0x5574d3d133d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.739+0000 7f4b71971640 1 -- 172.21.15.114:0/1281094924 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574d398a210 con 0x5574d39492f0 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.740+0000 7f4b71971640 1 --2- 172.21.15.114:0/1281094924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 0x5574d3d13910 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800dd50 tx=0x7f4b68005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76b31cf37798bae3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.740+0000 7f4b62ffd640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b580b2a70 0x7f4b580b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.740+0000 7f4b57fff640 1 -- 172.21.15.114:0/1281094924 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68009de0 con 0x5574d39492f0 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.740+0000 7f4b72172640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b580b2a70 0x7f4b580b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.740+0000 7f4b62ffd640 1 -- 172.21.15.114:0/2449755172 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f4b5c103160 con 0x7f4b6c108bc0 2024-05-27T07:40:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.740+0000 7f4b57fff640 1 -- 172.21.15.114:0/1281094924 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b680035f0 con 0x5574d39492f0 2024-05-27T07:40:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.740+0000 7f4b57fff640 1 -- 172.21.15.114:0/1281094924 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b680186c0 con 0x5574d39492f0 2024-05-27T07:40:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.741+0000 7f4b7ab65740 1 -- 172.21.15.114:0/1281094924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 msgr2=0x5574d3d13910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.741+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/1281094924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 0x5574d3d13910 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800dd50 tx=0x7f4b68005f90 comp rx=0 tx=0).stop 2024-05-27T07:40:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.741+0000 7f4b72172640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b580b2a70 0x7f4b580b4f80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x5574d3d10540 tx=0x7f4b64009700 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.741+0000 7f4b7ab65740 1 -- 172.21.15.114:0/1281094924 shutdown_connections 2024-05-27T07:40:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.741+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/1281094924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39514d0 0x5574d3d13e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.741+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/1281094924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 0x5574d3d13910 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.741+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/1281094924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39721b0 0x5574d3d133d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.741+0000 7f4b7ab65740 1 -- 172.21.15.114:0/1281094924 >> 172.21.15.114:0/1281094924 conn(0x5574d3966300 msgr2=0x5574d3d0fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.741+0000 7f4b7ab65740 1 -- 172.21.15.114:0/1281094924 shutdown_connections 2024-05-27T07:40:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.741+0000 7f4b7ab65740 1 -- 172.21.15.114:0/1281094924 wait complete. 2024-05-27T07:40:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.741+0000 7f4b7ab65740 1 -- start start 2024-05-27T07:40:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.741+0000 7f4b7ab65740 1 Processor -- start 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b7ab65740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 0x5574d3d1f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b7ab65740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 0x5574d3d1f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b7ab65740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39721b0 0x5574d3d1d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b7ab65740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5574d395b790 con 0x5574d39492f0 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b7ab65740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5574d3c31890 con 0x5574d39514d0 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b7ab65740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5574d3d20500 con 0x5574d39721b0 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b7ab65740 10 client.0 _ll_register_callbacks cb 0x5574d3989a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b71971640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 0x5574d3d1f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b71971640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 0x5574d3d1f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40906/0 (socket says 172.21.15.114:40906) 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b71971640 1 -- 172.21.15.114:0/644741094 learned_addr learned my addr 172.21.15.114:0/644741094 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b71170640 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 0x5574d3d1f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b72172640 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39721b0 0x5574d3d1d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b71971640 1 -- 172.21.15.114:0/644741094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39721b0 msgr2=0x5574d3d1d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b71971640 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39721b0 0x5574d3d1d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b71971640 1 -- 172.21.15.114:0/644741094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 msgr2=0x5574d3d1f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b71971640 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 0x5574d3d1f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b71971640 1 -- 172.21.15.114:0/644741094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574d3a56620 con 0x5574d39492f0 2024-05-27T07:40:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.742+0000 7f4b71170640 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 0x5574d3d1f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b71971640 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 0x5574d3d1f410 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4b68003f70 tx=0x7f4b68003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5e1f3a9132b675 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b557fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b557fa640 1 -- 172.21.15.114:0/644741094 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68004210 con 0x5574d39492f0 2024-05-27T07:40:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b557fa640 1 -- 172.21.15.114:0/644741094 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f4b680043b0 con 0x5574d39492f0 2024-05-27T07:40:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b7ab65740 1 -- 172.21.15.114:0/644741094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b680035f0 con 0x5574d39492f0 2024-05-27T07:40:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b557fa640 1 -- 172.21.15.114:0/644741094 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4b68033050 con 0x5574d39492f0 2024-05-27T07:40:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b7ab65740 10 client.15483 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b7ab65740 1 -- 172.21.15.114:0/644741094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5574d3d2c810 con 0x5574d39492f0 2024-05-27T07:40:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b557fa640 1 -- 172.21.15.114:0/644741094 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b680035f0 con 0x5574d39492f0 2024-05-27T07:40:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b557fa640 1 -- 172.21.15.114:0/644741094 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f4b680049d0 con 0x5574d39492f0 2024-05-27T07:40:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b7ab65740 10 client.15483 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b7ab65740 10 client.15483 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b7ab65740 20 client.15483 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5574d3930d80 0x5574d3d2fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b7ab65740 4 client.15483 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.743+0000 7f4b7ab65740 1 -- 172.21.15.114:0/644741094 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574d3c87b80 con 0x5574d3930d80 2024-05-27T07:40:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.744+0000 7f4b71170640 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5574d3930d80 0x5574d3d2fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.744+0000 7f4b71170640 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5574d3930d80 0x5574d3d2fc80 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22aba4165a955738 server_cookie=94294789f38caf69 in_seq=0 out_seq=0 2024-05-27T07:40:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.744+0000 7f4b557fa640 10 client.15483 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b557fa640 1 -- 172.21.15.114:0/644741094 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5574d3c87b80 con 0x5574d3930d80 2024-05-27T07:40:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b557fa640 10 client.15483 handle_command_reply: tid=0 2024-05-27T07:40:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b7ab65740 1 client.15483 shutdown 2024-05-27T07:40:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b7ab65740 1 -- 172.21.15.114:0/644741094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 msgr2=0x5574d3d1f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 0x5574d3d1f410 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4b68003f70 tx=0x7f4b68003c80 comp rx=0 tx=0).stop 2024-05-27T07:40:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b7ab65740 1 -- 172.21.15.114:0/644741094 shutdown_connections 2024-05-27T07:40:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5574d3930d80 0x5574d3d2fc80 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39721b0 0x5574d3d1d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 0x5574d3d1f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/644741094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d39492f0 0x5574d3d1f410 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b557fa640 0 client.15483 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b7ab65740 1 -- 172.21.15.114:0/644741094 >> 172.21.15.114:0/644741094 conn(0x5574d3966300 msgr2=0x5574d3d150f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b7ab65740 1 -- 172.21.15.114:0/644741094 shutdown_connections 2024-05-27T07:40:48.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b7ab65740 1 -- 172.21.15.114:0/644741094 wait complete. 2024-05-27T07:40:48.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.745+0000 7f4b7ab65740 20 client.15483 trim_cache size 0 max 16384 2024-05-27T07:40:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.760+0000 7f4b7ab65740 1 Processor -- start 2024-05-27T07:40:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.760+0000 7f4b7ab65740 1 -- start start 2024-05-27T07:40:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.760+0000 7f4b7ab65740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 0x5574d3d1c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.760+0000 7f4b7ab65740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39492f0 0x5574d3d1d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.760+0000 7f4b7ab65740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 0x5574d3d32cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.760+0000 7f4b7ab65740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1ad1f0 con 0x5574d3966300 2024-05-27T07:40:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.760+0000 7f4b7ab65740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1afa70 con 0x5574d39514d0 2024-05-27T07:40:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.760+0000 7f4b7ab65740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1ad860 con 0x5574d39492f0 2024-05-27T07:40:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.760+0000 7f4b71971640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 0x5574d3d1c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.760+0000 7f4b71971640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 0x5574d3d1c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40922/0 (socket says 172.21.15.114:40922) 2024-05-27T07:40:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.760+0000 7f4b71971640 1 -- 172.21.15.114:0/1568024639 learned_addr learned my addr 172.21.15.114:0/1568024639 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.761+0000 7f4b71971640 1 -- 172.21.15.114:0/1568024639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39492f0 msgr2=0x5574d3d1d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.761+0000 7f4b71971640 1 --2- 172.21.15.114:0/1568024639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39492f0 0x5574d3d1d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.761+0000 7f4b71971640 1 -- 172.21.15.114:0/1568024639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 msgr2=0x5574d3d32cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.761+0000 7f4b71971640 1 --2- 172.21.15.114:0/1568024639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 0x5574d3d32cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.761+0000 7f4b71971640 1 -- 172.21.15.114:0/1568024639 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574d3d17ac0 con 0x5574d3966300 2024-05-27T07:40:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.761+0000 7f4b71971640 1 --2- 172.21.15.114:0/1568024639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 0x5574d3d1c820 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800d8d0 tx=0x7f4b68018460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c88f1b8134ca7f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.761+0000 7f4b577fe640 1 -- 172.21.15.114:0/1568024639 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68009de0 con 0x5574d3966300 2024-05-27T07:40:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.761+0000 7f4b577fe640 1 -- 172.21.15.114:0/1568024639 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b680035f0 con 0x5574d3966300 2024-05-27T07:40:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.761+0000 7f4b577fe640 1 -- 172.21.15.114:0/1568024639 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b680052a0 con 0x5574d3966300 2024-05-27T07:40:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.762+0000 7f4b7ab65740 1 -- 172.21.15.114:0/1568024639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 msgr2=0x5574d3d1c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.762+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/1568024639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 0x5574d3d1c820 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800d8d0 tx=0x7f4b68018460 comp rx=0 tx=0).stop 2024-05-27T07:40:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.762+0000 7f4b7ab65740 1 -- 172.21.15.114:0/1568024639 shutdown_connections 2024-05-27T07:40:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.762+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/1568024639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 0x5574d3d32cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.762+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/1568024639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39492f0 0x5574d3d1d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.762+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/1568024639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 0x5574d3d1c820 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.762+0000 7f4b7ab65740 1 -- 172.21.15.114:0/1568024639 >> 172.21.15.114:0/1568024639 conn(0x5574d3d19260 msgr2=0x5574d3d1d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.762+0000 7f4b7ab65740 1 -- 172.21.15.114:0/1568024639 shutdown_connections 2024-05-27T07:40:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.762+0000 7f4b7ab65740 1 -- 172.21.15.114:0/1568024639 wait complete. 2024-05-27T07:40:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.762+0000 7f4b7ab65740 1 -- start start 2024-05-27T07:40:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.762+0000 7f4b7ab65740 1 Processor -- start 2024-05-27T07:40:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b7ab65740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39492f0 0x5574d3d32c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b7ab65740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 0x5574d3d33150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b7ab65740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 0x5574d3d8f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b7ab65740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1adf30 con 0x5574d3966300 2024-05-27T07:40:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b7ab65740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1af3a0 con 0x5574d39514d0 2024-05-27T07:40:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b7ab65740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1ae600 con 0x5574d39492f0 2024-05-27T07:40:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b7ab65740 10 client.0 _ll_register_callbacks cb 0x7f4b640095a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b72172640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 0x5574d3d8f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b72172640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 0x5574d3d8f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40930/0 (socket says 172.21.15.114:40930) 2024-05-27T07:40:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b72172640 1 -- 172.21.15.114:0/104602777 learned_addr learned my addr 172.21.15.114:0/104602777 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b71971640 1 --2- 172.21.15.114:0/104602777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39492f0 0x5574d3d32c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b72172640 1 -- 172.21.15.114:0/104602777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39492f0 msgr2=0x5574d3d32c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b72172640 1 --2- 172.21.15.114:0/104602777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39492f0 0x5574d3d32c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b72172640 1 -- 172.21.15.114:0/104602777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 msgr2=0x5574d3d33150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b72172640 1 --2- 172.21.15.114:0/104602777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 0x5574d3d33150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b72172640 1 -- 172.21.15.114:0/104602777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574d3d8fd20 con 0x5574d3966300 2024-05-27T07:40:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b72172640 1 --2- 172.21.15.114:0/104602777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 0x5574d3d8f760 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4b64010900 tx=0x7f4b64004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d93cd386ef8b7d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b557fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b557fa640 1 -- 172.21.15.114:0/104602777 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6400fcd0 con 0x5574d3966300 2024-05-27T07:40:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b557fa640 1 -- 172.21.15.114:0/104602777 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f4b64029470 con 0x5574d3966300 2024-05-27T07:40:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.763+0000 7f4b7ab65740 1 -- 172.21.15.114:0/104602777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b680035f0 con 0x5574d3966300 2024-05-27T07:40:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.764+0000 7f4b557fa640 1 -- 172.21.15.114:0/104602777 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4b64032050 con 0x5574d3966300 2024-05-27T07:40:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.764+0000 7f4b7ab65740 10 client.15489 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.764+0000 7f4b7ab65740 1 -- 172.21.15.114:0/104602777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5574d3d982c0 con 0x5574d3966300 2024-05-27T07:40:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.764+0000 7f4b557fa640 1 -- 172.21.15.114:0/104602777 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b680035f0 con 0x5574d3966300 2024-05-27T07:40:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.764+0000 7f4b557fa640 1 -- 172.21.15.114:0/104602777 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f4b64020520 con 0x5574d3966300 2024-05-27T07:40:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.764+0000 7f4b7ab65740 10 client.15489 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.764+0000 7f4b7ab65740 10 client.15489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.764+0000 7f4b7ab65740 20 client.15489 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.764+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/104602777 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5574d39721b0 0x5574d3d9aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.764+0000 7f4b7ab65740 4 client.15489 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:40:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.764+0000 7f4b7ab65740 1 -- 172.21.15.114:0/104602777 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5574d3c81360 con 0x5574d39721b0 2024-05-27T07:40:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.764+0000 7f4b71971640 1 --2- 172.21.15.114:0/104602777 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5574d39721b0 0x5574d3d9aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.765+0000 7f4b557fa640 10 client.15489 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.765+0000 7f4b71971640 1 --2- 172.21.15.114:0/104602777 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5574d39721b0 0x5574d3d9aab0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=932d7919d01fb207 server_cookie=792ac9afeb952168 in_seq=0 out_seq=0 2024-05-27T07:40:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b557fa640 1 -- 172.21.15.114:0/104602777 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5574d3c81360 con 0x5574d39721b0 2024-05-27T07:40:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b557fa640 10 client.15489 handle_command_reply: tid=0 2024-05-27T07:40:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b7ab65740 1 client.15489 shutdown 2024-05-27T07:40:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b7ab65740 1 -- 172.21.15.114:0/104602777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 msgr2=0x5574d3d8f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/104602777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 0x5574d3d8f760 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4b64010900 tx=0x7f4b64004ab0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b7ab65740 1 -- 172.21.15.114:0/104602777 shutdown_connections 2024-05-27T07:40:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/104602777 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5574d39721b0 0x5574d3d9aab0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/104602777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574d3966300 0x5574d3d8f760 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b557fa640 0 client.15489 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/104602777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5574d39514d0 0x5574d3d33150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b7ab65740 1 --2- 172.21.15.114:0/104602777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574d39492f0 0x5574d3d32c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b7ab65740 1 -- 172.21.15.114:0/104602777 >> 172.21.15.114:0/104602777 conn(0x5574d3d19260 msgr2=0x5574d3d1ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b7ab65740 1 -- 172.21.15.114:0/104602777 shutdown_connections 2024-05-27T07:40:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b7ab65740 1 -- 172.21.15.114:0/104602777 wait complete. 2024-05-27T07:40:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b7ab65740 20 client.15489 trim_cache size 0 max 16384 2024-05-27T07:40:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:40:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:40:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:40:48.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b73bd5640 1 -- 172.21.15.114:0/2449755172 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b580b2a70 msgr2=0x7f4b580b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b73bd5640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b580b2a70 0x7f4b580b4f80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x5574d3d10540 tx=0x7f4b64009700 comp rx=0 tx=0).stop 2024-05-27T07:40:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b73bd5640 1 -- 172.21.15.114:0/2449755172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c108bc0 msgr2=0x7f4b6c198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.769+0000 7f4b73bd5640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c108bc0 0x7f4b6c198c30 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c006010 tx=0x7f4b5c004060 comp rx=0 tx=0).stop 2024-05-27T07:40:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.770+0000 7f4b73bd5640 1 -- 172.21.15.114:0/2449755172 shutdown_connections 2024-05-27T07:40:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.770+0000 7f4b73bd5640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b580b2a70 0x7f4b580b4f80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.770+0000 7f4b73bd5640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c109060 0x7f4b6c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.770+0000 7f4b73bd5640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c108bc0 0x7f4b6c198c30 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.770+0000 7f4b73bd5640 1 --2- 172.21.15.114:0/2449755172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b6c103310 0x7f4b6c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.770+0000 7f4b73bd5640 1 -- 172.21.15.114:0/2449755172 >> 172.21.15.114:0/2449755172 conn(0x7f4b6c0782c0 msgr2=0x7f4b6c1012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.770+0000 7f4b73bd5640 1 -- 172.21.15.114:0/2449755172 shutdown_connections 2024-05-27T07:40:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:40:48.770+0000 7f4b73bd5640 1 -- 172.21.15.114:0/2449755172 wait complete. 2024-05-27T07:40:49.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.475+0000 7f0d78082700 1 -- 172.21.15.114:0/3665390217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d70104600 msgr2=0x7f0d70104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:49.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.475+0000 7f0d78082700 1 --2- 172.21.15.114:0/3665390217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d70104600 0x7f0d70104a10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00b0d0 tx=0x7f0d6c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.475+0000 7f0d78082700 1 -- 172.21.15.114:0/3665390217 shutdown_connections 2024-05-27T07:40:49.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.475+0000 7f0d78082700 1 --2- 172.21.15.114:0/3665390217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d70109880 0x7f0d701111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.475+0000 7f0d78082700 1 --2- 172.21.15.114:0/3665390217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d70104600 0x7f0d70104a10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.475+0000 7f0d78082700 1 --2- 172.21.15.114:0/3665390217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d70108f30 0x7f0d70109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.475+0000 7f0d78082700 1 -- 172.21.15.114:0/3665390217 >> 172.21.15.114:0/3665390217 conn(0x7f0d7007af60 msgr2=0x7f0d7007b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:49.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:49 smithi114 ceph-mon[32038]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-27T07:40:49.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.475+0000 7f0d78082700 1 -- 172.21.15.114:0/3665390217 shutdown_connections 2024-05-27T07:40:49.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.476+0000 7f0d78082700 1 -- 172.21.15.114:0/3665390217 wait complete. 2024-05-27T07:40:49.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.476+0000 7f0d78082700 1 Processor -- start 2024-05-27T07:40:49.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.476+0000 7f0d78082700 1 -- start start 2024-05-27T07:40:49.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d78082700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d70104600 0x7f0d7019aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d78082700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d70108f30 0x7f0d7019b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d78082700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d70109880 0x7f0d7019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d78082700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d701144b0 con 0x7f0d70109880 2024-05-27T07:40:49.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d78082700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0d70114330 con 0x7f0d70104600 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d78082700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d70114630 con 0x7f0d70108f30 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d7661f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d70109880 0x7f0d7019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d7661f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d70109880 0x7f0d7019f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40950/0 (socket says 172.21.15.114:40950) 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d7661f700 1 -- 172.21.15.114:0/1283272894 learned_addr learned my addr 172.21.15.114:0/1283272894 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d75e1e700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d70104600 0x7f0d7019aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d7561d700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d70108f30 0x7f0d7019b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d7661f700 1 -- 172.21.15.114:0/1283272894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d70108f30 msgr2=0x7f0d7019b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d7661f700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d70108f30 0x7f0d7019b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d7661f700 1 -- 172.21.15.114:0/1283272894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d70104600 msgr2=0x7f0d7019aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d7661f700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d70104600 0x7f0d7019aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d7661f700 1 -- 172.21.15.114:0/1283272894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7019fcc0 con 0x7f0d70109880 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d7661f700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d70109880 0x7f0d7019f6d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800d980 tx=0x7f0d6800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.477+0000 7f0d66ffd700 1 -- 172.21.15.114:0/1283272894 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68017070 con 0x7f0d70109880 2024-05-27T07:40:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.478+0000 7f0d66ffd700 1 -- 172.21.15.114:0/1283272894 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d68005020 con 0x7f0d70109880 2024-05-27T07:40:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.478+0000 7f0d66ffd700 1 -- 172.21.15.114:0/1283272894 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6800fa10 con 0x7f0d70109880 2024-05-27T07:40:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.478+0000 7f0d78082700 1 -- 172.21.15.114:0/1283272894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d7019ffb0 con 0x7f0d70109880 2024-05-27T07:40:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.478+0000 7f0d78082700 1 -- 172.21.15.114:0/1283272894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d701a04a0 con 0x7f0d70109880 2024-05-27T07:40:49.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.480+0000 7f0d66ffd700 1 -- 172.21.15.114:0/1283272894 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7f0d6800fbb0 con 0x7f0d70109880 2024-05-27T07:40:49.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.481+0000 7f0d86a97b80 1 Processor -- start 2024-05-27T07:40:49.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.481+0000 7f0d66ffd700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0d5c0b2b10 0x7f0d5c0b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.481+0000 7f0d7561d700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0d5c0b2b10 0x7f0d5c0b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.481+0000 7f0d66ffd700 1 -- 172.21.15.114:0/1283272894 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f0d680e2a00 con 0x7f0d70109880 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.481+0000 7f0d86a97b80 1 -- start start 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d86a97b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6e1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d86a97b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6e1c50 0x56295e6dba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d86a97b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6dbf80 0x56295e6dc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d86a97b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56295e60ac90 con 0x56295e6e1c50 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d86a97b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56295e5fa830 con 0x56295e55de40 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d86a97b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56295e5fe870 con 0x56295e6dbf80 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d7561d700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0d5c0b2b10 0x7f0d5c0b5020 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d70102020 tx=0x7f0d6000a380 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d75e1e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6e1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d75e1e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6e1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39114/0 (socket says 172.21.15.114:39114) 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d7561d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6e1c50 0x56295e6dba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d75e1e700 1 -- 172.21.15.114:0/3294020863 learned_addr learned my addr 172.21.15.114:0/3294020863 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d7661f700 1 --2- 172.21.15.114:0/3294020863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6dbf80 0x56295e6dc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d75e1e700 1 -- 172.21.15.114:0/3294020863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6dbf80 msgr2=0x56295e6dc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d75e1e700 1 --2- 172.21.15.114:0/3294020863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6dbf80 0x56295e6dc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d75e1e700 1 -- 172.21.15.114:0/3294020863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6e1c50 msgr2=0x56295e6dba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d75e1e700 1 --2- 172.21.15.114:0/3294020863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6e1c50 0x56295e6dba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.482+0000 7f0d75e1e700 1 -- 172.21.15.114:0/3294020863 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56295e54a8d0 con 0x56295e55de40 2024-05-27T07:40:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.483+0000 7f0d75e1e700 1 --2- 172.21.15.114:0/3294020863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6e1710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c015000 tx=0x7f0d6c005b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e2100b638e80939 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.483+0000 7f0d5bfff700 1 -- 172.21.15.114:0/3294020863 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c003a00 con 0x56295e55de40 2024-05-27T07:40:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.483+0000 7f0d5bfff700 1 -- 172.21.15.114:0/3294020863 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d6c003ba0 con 0x56295e55de40 2024-05-27T07:40:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.483+0000 7f0d5bfff700 1 -- 172.21.15.114:0/3294020863 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c0052d0 con 0x56295e55de40 2024-05-27T07:40:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.484+0000 7f0d86a97b80 1 -- 172.21.15.114:0/3294020863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 msgr2=0x56295e6e1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.484+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/3294020863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6e1710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c015000 tx=0x7f0d6c005b00 comp rx=0 tx=0).stop 2024-05-27T07:40:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.484+0000 7f0d86a97b80 1 -- 172.21.15.114:0/3294020863 shutdown_connections 2024-05-27T07:40:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.484+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/3294020863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6dbf80 0x56295e6dc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.484+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/3294020863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6e1710 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.484+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/3294020863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6e1c50 0x56295e6dba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.484+0000 7f0d86a97b80 1 -- 172.21.15.114:0/3294020863 >> 172.21.15.114:0/3294020863 conn(0x56295e5b8af0 msgr2=0x56295e6d6640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:49.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.484+0000 7f0d86a97b80 1 -- 172.21.15.114:0/3294020863 shutdown_connections 2024-05-27T07:40:49.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.484+0000 7f0d86a97b80 1 -- 172.21.15.114:0/3294020863 wait complete. 2024-05-27T07:40:49.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.484+0000 7f0d86a97b80 1 -- start start 2024-05-27T07:40:49.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d86a97b80 1 Processor -- start 2024-05-27T07:40:49.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d86a97b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e55de40 0x56295e6e5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d86a97b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6dbf80 0x56295e6e58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d86a97b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e6e1c50 0x56295e6e5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d86a97b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56295e500fa0 con 0x56295e55de40 2024-05-27T07:40:49.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d86a97b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56295e59c7c0 con 0x56295e6e1c50 2024-05-27T07:40:49.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d86a97b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56295e6e86e0 con 0x56295e6dbf80 2024-05-27T07:40:49.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d86a97b80 10 client.0 _ll_register_callbacks cb 0x56295e595920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d75e1e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e55de40 0x56295e6e5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d75e1e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e55de40 0x56295e6e5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40954/0 (socket says 172.21.15.114:40954) 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d75e1e700 1 -- 172.21.15.114:0/938477287 learned_addr learned my addr 172.21.15.114:0/938477287 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d7661f700 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e6e1c50 0x56295e6e5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d7561d700 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6dbf80 0x56295e6e58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d75e1e700 1 -- 172.21.15.114:0/938477287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6dbf80 msgr2=0x56295e6e58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.485+0000 7f0d75e1e700 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6dbf80 0x56295e6e58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.486+0000 7f0d75e1e700 1 -- 172.21.15.114:0/938477287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e6e1c50 msgr2=0x56295e6e5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.486+0000 7f0d75e1e700 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e6e1c50 0x56295e6e5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.486+0000 7f0d75e1e700 1 -- 172.21.15.114:0/938477287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56295e599690 con 0x56295e55de40 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.486+0000 7f0d7561d700 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6dbf80 0x56295e6e58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.486+0000 7f0d75e1e700 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e55de40 0x56295e6e5390 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00d010 tx=0x7f0d6c007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac8dbb1267bc359 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.486+0000 7f0d597fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.486+0000 7f0d597fa700 1 -- 172.21.15.114:0/938477287 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c009310 con 0x56295e55de40 2024-05-27T07:40:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.486+0000 7f0d86a97b80 1 -- 172.21.15.114:0/938477287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d6c003ba0 con 0x56295e55de40 2024-05-27T07:40:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.486+0000 7f0d597fa700 1 -- 172.21.15.114:0/938477287 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f0d6c0050c0 con 0x56295e55de40 2024-05-27T07:40:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.486+0000 7f0d597fa700 1 -- 172.21.15.114:0/938477287 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0d6c036050 con 0x56295e55de40 2024-05-27T07:40:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.486+0000 7f0d86a97b80 10 client.15501 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.486+0000 7f0d86a97b80 1 -- 172.21.15.114:0/938477287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56295e6f31c0 con 0x56295e55de40 2024-05-27T07:40:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.486+0000 7f0d597fa700 1 -- 172.21.15.114:0/938477287 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d6c003ba0 con 0x56295e55de40 2024-05-27T07:40:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.487+0000 7f0d597fa700 1 -- 172.21.15.114:0/938477287 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0d6c011070 con 0x56295e55de40 2024-05-27T07:40:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.487+0000 7f0d86a97b80 10 client.15501 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.487+0000 7f0d86a97b80 10 client.15501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.487+0000 7f0d86a97b80 20 client.15501 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.487+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56295e6f5690 0x56295e6f7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.487+0000 7f0d86a97b80 4 client.15501 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.487+0000 7f0d86a97b80 1 -- 172.21.15.114:0/938477287 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56295e6a79f0 con 0x56295e6f5690 2024-05-27T07:40:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.487+0000 7f0d7561d700 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56295e6f5690 0x56295e6f7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.487+0000 7f0d597fa700 10 client.15501 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.487+0000 7f0d7561d700 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56295e6f5690 0x56295e6f7ab0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfa63986157f9838 server_cookie=6ef05c5d10c003ec in_seq=0 out_seq=0 2024-05-27T07:40:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.488+0000 7f0d597fa700 1 -- 172.21.15.114:0/938477287 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56295e6a79f0 con 0x56295e6f5690 2024-05-27T07:40:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.488+0000 7f0d597fa700 10 client.15501 handle_command_reply: tid=0 2024-05-27T07:40:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.488+0000 7f0d86a97b80 1 client.15501 shutdown 2024-05-27T07:40:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.488+0000 7f0d86a97b80 1 -- 172.21.15.114:0/938477287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e55de40 msgr2=0x56295e6e5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.488+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e55de40 0x56295e6e5390 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00d010 tx=0x7f0d6c007ca0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.488+0000 7f0d86a97b80 1 -- 172.21.15.114:0/938477287 shutdown_connections 2024-05-27T07:40:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.488+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56295e6f5690 0x56295e6f7ab0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.489+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e6e1c50 0x56295e6e5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.489+0000 7f0d597fa700 0 client.15501 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.489+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e55de40 0x56295e6e5390 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.489+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/938477287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6dbf80 0x56295e6e58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.489+0000 7f0d86a97b80 1 -- 172.21.15.114:0/938477287 >> 172.21.15.114:0/938477287 conn(0x56295e5b8af0 msgr2=0x56295e6daa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:49.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.489+0000 7f0d86a97b80 1 -- 172.21.15.114:0/938477287 shutdown_connections 2024-05-27T07:40:49.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.489+0000 7f0d86a97b80 1 -- 172.21.15.114:0/938477287 wait complete. 2024-05-27T07:40:49.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.489+0000 7f0d86a97b80 20 client.15501 trim_cache size 0 max 16384 2024-05-27T07:40:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d86a97b80 1 Processor -- start 2024-05-27T07:40:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d86a97b80 1 -- start start 2024-05-27T07:40:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d86a97b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6dba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d86a97b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 0x56295e6e0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d86a97b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6e1c50 0x56295e6f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d86a97b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d701ae4a0 con 0x56295e6dbf80 2024-05-27T07:40:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d86a97b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0d701b0d20 con 0x56295e55de40 2024-05-27T07:40:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d86a97b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d701aeb10 con 0x56295e6e1c50 2024-05-27T07:40:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d7561d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 0x56295e6e0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d7561d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 0x56295e6e0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40968/0 (socket says 172.21.15.114:40968) 2024-05-27T07:40:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d7561d700 1 -- 172.21.15.114:0/2754628248 learned_addr learned my addr 172.21.15.114:0/2754628248 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d7661f700 1 --2- 172.21.15.114:0/2754628248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6e1c50 0x56295e6f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d7561d700 1 -- 172.21.15.114:0/2754628248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6e1c50 msgr2=0x56295e6f19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d7561d700 1 --2- 172.21.15.114:0/2754628248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6e1c50 0x56295e6f19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d75e1e700 1 --2- 172.21.15.114:0/2754628248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6dba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d7561d700 1 -- 172.21.15.114:0/2754628248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 msgr2=0x56295e6dba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d7561d700 1 --2- 172.21.15.114:0/2754628248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6dba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d7561d700 1 -- 172.21.15.114:0/2754628248 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56295e6e6450 con 0x56295e6dbf80 2024-05-27T07:40:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.504+0000 7f0d7661f700 1 --2- 172.21.15.114:0/2754628248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6e1c50 0x56295e6f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.505+0000 7f0d7561d700 1 --2- 172.21.15.114:0/2754628248 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 0x56295e6e0e70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0d6000aa30 tx=0x7f0d60012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8196cde0a0f3b4dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.505+0000 7f0d5b7fe700 1 -- 172.21.15.114:0/2754628248 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6002c070 con 0x56295e6dbf80 2024-05-27T07:40:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.505+0000 7f0d5b7fe700 1 -- 172.21.15.114:0/2754628248 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d60014070 con 0x56295e6dbf80 2024-05-27T07:40:49.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.505+0000 7f0d5b7fe700 1 -- 172.21.15.114:0/2754628248 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d60016070 con 0x56295e6dbf80 2024-05-27T07:40:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.506+0000 7f0d86a97b80 1 -- 172.21.15.114:0/2754628248 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 msgr2=0x56295e6e0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.506+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/2754628248 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 0x56295e6e0e70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0d6000aa30 tx=0x7f0d60012000 comp rx=0 tx=0).stop 2024-05-27T07:40:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.506+0000 7f0d86a97b80 1 -- 172.21.15.114:0/2754628248 shutdown_connections 2024-05-27T07:40:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.506+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/2754628248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6e1c50 0x56295e6f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.506+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/2754628248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6dba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.506+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/2754628248 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 0x56295e6e0e70 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.506+0000 7f0d86a97b80 1 -- 172.21.15.114:0/2754628248 >> 172.21.15.114:0/2754628248 conn(0x56295e5b8af0 msgr2=0x56295e6df680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.506+0000 7f0d86a97b80 1 -- 172.21.15.114:0/2754628248 shutdown_connections 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.506+0000 7f0d86a97b80 1 -- 172.21.15.114:0/2754628248 wait complete. 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.506+0000 7f0d86a97b80 1 -- start start 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.506+0000 7f0d86a97b80 1 Processor -- start 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d86a97b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6f8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d86a97b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 0x56295e6f87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d86a97b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6e1c50 0x56295e75c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d86a97b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d701af1e0 con 0x56295e6dbf80 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d86a97b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0d701b0650 con 0x56295e55de40 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d86a97b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d701af8b0 con 0x56295e6e1c50 2024-05-27T07:40:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d86a97b80 10 client.0 _ll_register_callbacks cb 0x56295e595920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d7561d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 0x56295e6f87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d7561d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 0x56295e6f87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40974/0 (socket says 172.21.15.114:40974) 2024-05-27T07:40:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d7561d700 1 -- 172.21.15.114:0/3940973100 learned_addr learned my addr 172.21.15.114:0/3940973100 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d7661f700 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6e1c50 0x56295e75c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d75e1e700 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6f8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d7561d700 1 -- 172.21.15.114:0/3940973100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6e1c50 msgr2=0x56295e75c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d7561d700 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6e1c50 0x56295e75c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d7561d700 1 -- 172.21.15.114:0/3940973100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 msgr2=0x56295e6f8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d7561d700 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6f8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d7561d700 1 -- 172.21.15.114:0/3940973100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56295e75c8b0 con 0x56295e6dbf80 2024-05-27T07:40:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d75e1e700 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6f8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d7561d700 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 0x56295e6f87d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0d60003fa0 tx=0x7f0d600045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48dcefb89e8a062b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d597fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d597fa700 1 -- 172.21.15.114:0/3940973100 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d60035060 con 0x56295e6dbf80 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d597fa700 1 -- 172.21.15.114:0/3940973100 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f0d60027c30 con 0x56295e6dbf80 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.507+0000 7f0d86a97b80 1 -- 172.21.15.114:0/3940973100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d60014070 con 0x56295e6dbf80 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.508+0000 7f0d597fa700 1 -- 172.21.15.114:0/3940973100 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0d6002c040 con 0x56295e6dbf80 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.508+0000 7f0d86a97b80 10 client.15507 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.508+0000 7f0d86a97b80 1 -- 172.21.15.114:0/3940973100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56295e763b20 con 0x56295e6dbf80 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.508+0000 7f0d597fa700 1 -- 172.21.15.114:0/3940973100 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d60014070 con 0x56295e6dbf80 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.508+0000 7f0d597fa700 1 -- 172.21.15.114:0/3940973100 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0d600047e0 con 0x56295e6dbf80 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.508+0000 7f0d86a97b80 10 client.15507 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.508+0000 7f0d86a97b80 10 client.15507 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.508+0000 7f0d86a97b80 20 client.15507 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.508+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56295e7641a0 0x56295e7665c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.508+0000 7f0d86a97b80 4 client.15507 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.508+0000 7f0d86a97b80 1 -- 172.21.15.114:0/3940973100 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56295e60cbe0 con 0x56295e7641a0 2024-05-27T07:40:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.508+0000 7f0d75e1e700 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56295e7641a0 0x56295e7665c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d75e1e700 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56295e7641a0 0x56295e7665c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=646fdb9545b46756 server_cookie=b3e66da96abc3538 in_seq=0 out_seq=0 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d597fa700 10 client.15507 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d597fa700 1 -- 172.21.15.114:0/3940973100 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56295e60cbe0 con 0x56295e7641a0 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d597fa700 10 client.15507 handle_command_reply: tid=0 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d86a97b80 1 client.15507 shutdown 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d86a97b80 1 -- 172.21.15.114:0/3940973100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 msgr2=0x56295e6f87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 0x56295e6f87d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0d60003fa0 tx=0x7f0d600045c0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d86a97b80 1 -- 172.21.15.114:0/3940973100 shutdown_connections 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56295e6e1c50 0x56295e75c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56295e7641a0 0x56295e7665c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56295e55de40 0x56295e6f8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d86a97b80 1 --2- 172.21.15.114:0/3940973100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56295e6dbf80 0x56295e6f87d0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d86a97b80 1 -- 172.21.15.114:0/3940973100 >> 172.21.15.114:0/3940973100 conn(0x56295e5b8af0 msgr2=0x56295e6e3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.509+0000 7f0d597fa700 0 client.15507 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:49.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.510+0000 7f0d86a97b80 1 -- 172.21.15.114:0/3940973100 shutdown_connections 2024-05-27T07:40:49.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.510+0000 7f0d86a97b80 1 -- 172.21.15.114:0/3940973100 wait complete. 2024-05-27T07:40:49.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.510+0000 7f0d86a97b80 20 client.15507 trim_cache size 0 max 16384 2024-05-27T07:40:49.513 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:40:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.510+0000 7f0d78082700 1 -- 172.21.15.114:0/1283272894 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0d5c0b2b10 msgr2=0x7f0d5c0b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.510+0000 7f0d78082700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0d5c0b2b10 0x7f0d5c0b5020 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d70102020 tx=0x7f0d6000a380 comp rx=0 tx=0).stop 2024-05-27T07:40:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.510+0000 7f0d78082700 1 -- 172.21.15.114:0/1283272894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d70109880 msgr2=0x7f0d7019f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.510+0000 7f0d78082700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d70109880 0x7f0d7019f6d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800d980 tx=0x7f0d6800dc90 comp rx=0 tx=0).stop 2024-05-27T07:40:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.510+0000 7f0d78082700 1 -- 172.21.15.114:0/1283272894 shutdown_connections 2024-05-27T07:40:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.510+0000 7f0d78082700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d70109880 0x7f0d7019f6d0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.510+0000 7f0d78082700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0d5c0b2b10 0x7f0d5c0b5020 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.510+0000 7f0d78082700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d70104600 0x7f0d7019aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.510+0000 7f0d78082700 1 --2- 172.21.15.114:0/1283272894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d70108f30 0x7f0d7019b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.510+0000 7f0d78082700 1 -- 172.21.15.114:0/1283272894 >> 172.21.15.114:0/1283272894 conn(0x7f0d7007af60 msgr2=0x7f0d70102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.511+0000 7f0d78082700 1 -- 172.21.15.114:0/1283272894 shutdown_connections 2024-05-27T07:40:49.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:49.511+0000 7f0d78082700 1 -- 172.21.15.114:0/1283272894 wait complete. 2024-05-27T07:40:49.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:49 smithi163 ceph-mon[36451]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-27T07:40:49.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:49 smithi186 ceph-mon[36647]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-27T07:40:51.235 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:40:51.236 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:40:51.488 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:51.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:51 smithi114 ceph-mon[32038]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-05-27T07:40:51.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:51 smithi163 ceph-mon[36451]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-05-27T07:40:51.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:51 smithi186 ceph-mon[36647]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-05-27T07:40:53.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.305+0000 7f6092a43700 1 -- 172.21.15.114:0/608799639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f608c108790 msgr2=0x7f608c110130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:53.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.305+0000 7f6092a43700 1 --2- 172.21.15.114:0/608799639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f608c108790 0x7f608c110130 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f608400b0d0 tx=0x7f608400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.306+0000 7f6092a43700 1 -- 172.21.15.114:0/608799639 shutdown_connections 2024-05-27T07:40:53.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.306+0000 7f6092a43700 1 --2- 172.21.15.114:0/608799639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f608c108790 0x7f608c110130 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.306+0000 7f6092a43700 1 --2- 172.21.15.114:0/608799639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f608c103b20 0x7f608c103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.306+0000 7f6092a43700 1 --2- 172.21.15.114:0/608799639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f608c107e40 0x7f608c108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.306+0000 7f6092a43700 1 -- 172.21.15.114:0/608799639 >> 172.21.15.114:0/608799639 conn(0x7f608c0fe510 msgr2=0x7f608c100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:53.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.306+0000 7f6092a43700 1 -- 172.21.15.114:0/608799639 shutdown_connections 2024-05-27T07:40:53.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.306+0000 7f6092a43700 1 -- 172.21.15.114:0/608799639 wait complete. 2024-05-27T07:40:53.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.306+0000 7f6092a43700 1 Processor -- start 2024-05-27T07:40:53.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.306+0000 7f6092a43700 1 -- start start 2024-05-27T07:40:53.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f6092a43700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f608c103b20 0x7f608c19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f6092a43700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f608c107e40 0x7f608c19b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f6092a43700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f608c108790 0x7f608c19f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f6092a43700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f608c113400 con 0x7f608c103b20 2024-05-27T07:40:53.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f6092a43700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f608c113280 con 0x7f608c107e40 2024-05-27T07:40:53.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f6092a43700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f608c113580 con 0x7f608c108790 2024-05-27T07:40:53.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f608bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f608c103b20 0x7f608c19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f608bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f608c103b20 0x7f608c19ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40980/0 (socket says 172.21.15.114:40980) 2024-05-27T07:40:53.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f608bfff700 1 -- 172.21.15.114:0/3962500297 learned_addr learned my addr 172.21.15.114:0/3962500297 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:53.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f6083fff700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f608c107e40 0x7f608c19b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f6090fe0700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f608c108790 0x7f608c19f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f608bfff700 1 -- 172.21.15.114:0/3962500297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f608c108790 msgr2=0x7f608c19f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:53.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f608bfff700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f608c108790 0x7f608c19f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f608bfff700 1 -- 172.21.15.114:0/3962500297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f608c107e40 msgr2=0x7f608c19b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:53.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f608bfff700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f608c107e40 0x7f608c19b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f608bfff700 1 -- 172.21.15.114:0/3962500297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608c19fcf0 con 0x7f608c103b20 2024-05-27T07:40:53.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f6090fe0700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f608c108790 0x7f608c19f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:53.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f608bfff700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f608c103b20 0x7f608c19ae20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f607c00d950 tx=0x7f607c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:53.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f6089ffb700 1 -- 172.21.15.114:0/3962500297 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c016070 con 0x7f608c103b20 2024-05-27T07:40:53.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f6092a43700 1 -- 172.21.15.114:0/3962500297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f608c19ffe0 con 0x7f608c103b20 2024-05-27T07:40:53.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f6089ffb700 1 -- 172.21.15.114:0/3962500297 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607c010460 con 0x7f608c103b20 2024-05-27T07:40:53.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.307+0000 7f6089ffb700 1 -- 172.21.15.114:0/3962500297 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c00f620 con 0x7f608c103b20 2024-05-27T07:40:53.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.308+0000 7f6092a43700 1 -- 172.21.15.114:0/3962500297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f608c1a7900 con 0x7f608c103b20 2024-05-27T07:40:53.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.310+0000 7f6089ffb700 1 -- 172.21.15.114:0/3962500297 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7f607c019070 con 0x7f608c103b20 2024-05-27T07:40:53.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.311+0000 7f6089ffb700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60780b2b10 0x7f60780b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.311+0000 7f6083fff700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60780b2b10 0x7f60780b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.311+0000 7f6089ffb700 1 -- 172.21.15.114:0/3962500297 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f607c0e2940 con 0x7f608c103b20 2024-05-27T07:40:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.311+0000 7f60a1458b80 1 Processor -- start 2024-05-27T07:40:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.311+0000 7f60a1458b80 1 -- start start 2024-05-27T07:40:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.311+0000 7f6083fff700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60780b2b10 0x7f60780b5020 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6074005fd0 tx=0x7f6074005db0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.311+0000 7f60a1458b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd181feec0 0x55dd18382790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.311+0000 7f60a1458b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd18382cd0 0x55dd1837cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.311+0000 7f60a1458b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd1837d000 0x55dd1837d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.311+0000 7f60a1458b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55dd182abd10 con 0x55dd18382cd0 2024-05-27T07:40:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.311+0000 7f60a1458b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dd1829b8b0 con 0x55dd181feec0 2024-05-27T07:40:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.311+0000 7f60a1458b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55dd1829f8f0 con 0x55dd1837d000 2024-05-27T07:40:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f6083fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd18382cd0 0x55dd1837cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f6083fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd18382cd0 0x55dd1837cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40992/0 (socket says 172.21.15.114:40992) 2024-05-27T07:40:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f6083fff700 1 -- 172.21.15.114:0/3941230269 learned_addr learned my addr 172.21.15.114:0/3941230269 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f6090fe0700 1 --2- 172.21.15.114:0/3941230269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd1837d000 0x55dd1837d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f608bfff700 1 --2- 172.21.15.114:0/3941230269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd181feec0 0x55dd18382790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f6083fff700 1 -- 172.21.15.114:0/3941230269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd1837d000 msgr2=0x55dd1837d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f6083fff700 1 --2- 172.21.15.114:0/3941230269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd1837d000 0x55dd1837d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f6083fff700 1 -- 172.21.15.114:0/3941230269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd181feec0 msgr2=0x55dd18382790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f6083fff700 1 --2- 172.21.15.114:0/3941230269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd181feec0 0x55dd18382790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f6083fff700 1 -- 172.21.15.114:0/3941230269 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd181eb950 con 0x55dd18382cd0 2024-05-27T07:40:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f6083fff700 1 --2- 172.21.15.114:0/3941230269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd18382cd0 0x55dd1837cac0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f608c06c9f0 tx=0x7f6074011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39112208c35f2f70 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f60827fc700 1 -- 172.21.15.114:0/3941230269 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074021070 con 0x55dd18382cd0 2024-05-27T07:40:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f60827fc700 1 -- 172.21.15.114:0/3941230269 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607400ee70 con 0x55dd18382cd0 2024-05-27T07:40:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.312+0000 7f60827fc700 1 -- 172.21.15.114:0/3941230269 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607400f620 con 0x55dd18382cd0 2024-05-27T07:40:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.313+0000 7f60a1458b80 1 -- 172.21.15.114:0/3941230269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd18382cd0 msgr2=0x55dd1837cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.313+0000 7f60a1458b80 1 --2- 172.21.15.114:0/3941230269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd18382cd0 0x55dd1837cac0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f608c06c9f0 tx=0x7f6074011000 comp rx=0 tx=0).stop 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.313+0000 7f60a1458b80 1 -- 172.21.15.114:0/3941230269 shutdown_connections 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.313+0000 7f60a1458b80 1 --2- 172.21.15.114:0/3941230269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd1837d000 0x55dd1837d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.313+0000 7f60a1458b80 1 --2- 172.21.15.114:0/3941230269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd181feec0 0x55dd18382790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.313+0000 7f60a1458b80 1 --2- 172.21.15.114:0/3941230269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd18382cd0 0x55dd1837cac0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.313+0000 7f60a1458b80 1 -- 172.21.15.114:0/3941230269 >> 172.21.15.114:0/3941230269 conn(0x55dd18259b70 msgr2=0x55dd183776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.313+0000 7f60a1458b80 1 -- 172.21.15.114:0/3941230269 shutdown_connections 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.313+0000 7f60a1458b80 1 -- 172.21.15.114:0/3941230269 wait complete. 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f60a1458b80 1 -- start start 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f60a1458b80 1 Processor -- start 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f60a1458b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd181feec0 0x55dd183863e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f60a1458b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 0x55dd18386920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f60a1458b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd18382cd0 0x55dd18386e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f60a1458b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55dd181a2020 con 0x55dd1837d000 2024-05-27T07:40:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f60a1458b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dd1823d840 con 0x55dd181feec0 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f60a1458b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55dd183897c0 con 0x55dd18382cd0 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f60a1458b80 10 client.0 _ll_register_callbacks cb 0x55dd182369a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f6083fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 0x55dd18386920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f6083fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 0x55dd18386920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40996/0 (socket says 172.21.15.114:40996) 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f6083fff700 1 -- 172.21.15.114:0/4119910112 learned_addr learned my addr 172.21.15.114:0/4119910112 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f608bfff700 1 --2- 172.21.15.114:0/4119910112 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd181feec0 0x55dd183863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.314+0000 7f6090fe0700 1 --2- 172.21.15.114:0/4119910112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd18382cd0 0x55dd18386e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f6083fff700 1 -- 172.21.15.114:0/4119910112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd18382cd0 msgr2=0x55dd18386e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f6083fff700 1 --2- 172.21.15.114:0/4119910112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd18382cd0 0x55dd18386e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f6083fff700 1 -- 172.21.15.114:0/4119910112 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd181feec0 msgr2=0x55dd183863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f6083fff700 1 --2- 172.21.15.114:0/4119910112 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd181feec0 0x55dd183863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f6083fff700 1 -- 172.21.15.114:0/4119910112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd1823a710 con 0x55dd1837d000 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f6083fff700 1 --2- 172.21.15.114:0/4119910112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 0x55dd18386920 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6074005950 tx=0x7f607401d680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee1bfe312745a99 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f606b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:53.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f606b7fe700 1 -- 172.21.15.114:0/4119910112 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074034060 con 0x55dd1837d000 2024-05-27T07:40:53.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f606b7fe700 1 -- 172.21.15.114:0/4119910112 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f6074005050 con 0x55dd1837d000 2024-05-27T07:40:53.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f60a1458b80 1 -- 172.21.15.114:0/4119910112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f607400ee70 con 0x55dd1837d000 2024-05-27T07:40:53.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f606b7fe700 1 -- 172.21.15.114:0/4119910112 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f607403b050 con 0x55dd1837d000 2024-05-27T07:40:53.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f60a1458b80 10 client.15519 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:53.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f60a1458b80 1 -- 172.21.15.114:0/4119910112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dd18394240 con 0x55dd1837d000 2024-05-27T07:40:53.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.315+0000 7f606b7fe700 1 -- 172.21.15.114:0/4119910112 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607400ee70 con 0x55dd1837d000 2024-05-27T07:40:53.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.316+0000 7f606b7fe700 1 -- 172.21.15.114:0/4119910112 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6074004bc0 con 0x55dd1837d000 2024-05-27T07:40:53.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.316+0000 7f60a1458b80 10 client.15519 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:53.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.316+0000 7f60a1458b80 10 client.15519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:53.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.316+0000 7f60a1458b80 20 client.15519 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:53.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.316+0000 7f60a1458b80 1 --2- 172.21.15.114:0/4119910112 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dd183967c0 0x55dd18398be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.316+0000 7f60a1458b80 4 client.15519 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.316+0000 7f60a1458b80 1 -- 172.21.15.114:0/4119910112 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd18348a70 con 0x55dd183967c0 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.316+0000 7f608bfff700 1 --2- 172.21.15.114:0/4119910112 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dd183967c0 0x55dd18398be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.317+0000 7f606b7fe700 10 client.15519 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.317+0000 7f608bfff700 1 --2- 172.21.15.114:0/4119910112 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dd183967c0 0x55dd18398be0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13912992f03c9337 server_cookie=cef902adc05cf371 in_seq=0 out_seq=0 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f606b7fe700 1 -- 172.21.15.114:0/4119910112 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dd18348a70 con 0x55dd183967c0 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f606b7fe700 10 client.15519 handle_command_reply: tid=0 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f60a1458b80 1 client.15519 shutdown 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f60a1458b80 1 -- 172.21.15.114:0/4119910112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 msgr2=0x55dd18386920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f60a1458b80 1 --2- 172.21.15.114:0/4119910112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 0x55dd18386920 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6074005950 tx=0x7f607401d680 comp rx=0 tx=0).stop 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f60a1458b80 1 -- 172.21.15.114:0/4119910112 shutdown_connections 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f60a1458b80 1 --2- 172.21.15.114:0/4119910112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd18382cd0 0x55dd18386e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f60a1458b80 1 --2- 172.21.15.114:0/4119910112 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dd183967c0 0x55dd18398be0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f606b7fe700 0 client.15519 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f60a1458b80 1 --2- 172.21.15.114:0/4119910112 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd181feec0 0x55dd183863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f60a1458b80 1 --2- 172.21.15.114:0/4119910112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 0x55dd18386920 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f60a1458b80 1 -- 172.21.15.114:0/4119910112 >> 172.21.15.114:0/4119910112 conn(0x55dd18259b70 msgr2=0x55dd1837bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:53.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f60a1458b80 1 -- 172.21.15.114:0/4119910112 shutdown_connections 2024-05-27T07:40:53.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f60a1458b80 1 -- 172.21.15.114:0/4119910112 wait complete. 2024-05-27T07:40:53.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.318+0000 7f60a1458b80 20 client.15519 trim_cache size 0 max 16384 2024-05-27T07:40:53.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.338+0000 7f60a1458b80 1 Processor -- start 2024-05-27T07:40:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.338+0000 7f60a1458b80 1 -- start start 2024-05-27T07:40:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.338+0000 7f60a1458b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd181feec0 0x55dd1837cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.338+0000 7f60a1458b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd1837d000 0x55dd18380ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.338+0000 7f60a1458b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd18382cd0 0x55dd18392a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.338+0000 7f60a1458b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f608c1ae6b0 con 0x55dd181feec0 2024-05-27T07:40:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.338+0000 7f60a1458b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f608c1b0f30 con 0x55dd1837d000 2024-05-27T07:40:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.338+0000 7f60a1458b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f608c1aed20 con 0x55dd18382cd0 2024-05-27T07:40:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.338+0000 7f608bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd181feec0 0x55dd1837cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.338+0000 7f608bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd181feec0 0x55dd1837cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41012/0 (socket says 172.21.15.114:41012) 2024-05-27T07:40:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.338+0000 7f608bfff700 1 -- 172.21.15.114:0/3323215603 learned_addr learned my addr 172.21.15.114:0/3323215603 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.339+0000 7f6083fff700 1 --2- 172.21.15.114:0/3323215603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd1837d000 0x55dd18380ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.339+0000 7f6090fe0700 1 --2- 172.21.15.114:0/3323215603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd18382cd0 0x55dd18392a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.339+0000 7f608bfff700 1 -- 172.21.15.114:0/3323215603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd18382cd0 msgr2=0x55dd18392a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.339+0000 7f608bfff700 1 --2- 172.21.15.114:0/3323215603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd18382cd0 0x55dd18392a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.339+0000 7f608bfff700 1 -- 172.21.15.114:0/3323215603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd1837d000 msgr2=0x55dd18380ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.339+0000 7f608bfff700 1 --2- 172.21.15.114:0/3323215603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd1837d000 0x55dd18380ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.339+0000 7f608bfff700 1 -- 172.21.15.114:0/3323215603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd18379100 con 0x55dd181feec0 2024-05-27T07:40:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.339+0000 7f608bfff700 1 --2- 172.21.15.114:0/3323215603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd181feec0 0x55dd1837cac0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f607c0e02e0 tx=0x7f607c005910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=868de38f80788b8b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.339+0000 7f6081ffb700 1 -- 172.21.15.114:0/3323215603 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c020040 con 0x55dd181feec0 2024-05-27T07:40:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.339+0000 7f6081ffb700 1 -- 172.21.15.114:0/3323215603 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607c0a58b0 con 0x55dd181feec0 2024-05-27T07:40:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.339+0000 7f6081ffb700 1 -- 172.21.15.114:0/3323215603 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c0dfe40 con 0x55dd181feec0 2024-05-27T07:40:53.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.340+0000 7f60a1458b80 1 -- 172.21.15.114:0/3323215603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd181feec0 msgr2=0x55dd1837cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:53.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.340+0000 7f60a1458b80 1 --2- 172.21.15.114:0/3323215603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd181feec0 0x55dd1837cac0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f607c0e02e0 tx=0x7f607c005910 comp rx=0 tx=0).stop 2024-05-27T07:40:53.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.340+0000 7f60a1458b80 1 -- 172.21.15.114:0/3323215603 shutdown_connections 2024-05-27T07:40:53.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.340+0000 7f60a1458b80 1 --2- 172.21.15.114:0/3323215603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd18382cd0 0x55dd18392a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.340+0000 7f60a1458b80 1 --2- 172.21.15.114:0/3323215603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd181feec0 0x55dd1837cac0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.340+0000 7f60a1458b80 1 --2- 172.21.15.114:0/3323215603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd1837d000 0x55dd18380ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.340+0000 7f60a1458b80 1 -- 172.21.15.114:0/3323215603 >> 172.21.15.114:0/3323215603 conn(0x55dd18259b70 msgr2=0x55dd18381b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:53.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.340+0000 7f60a1458b80 1 -- 172.21.15.114:0/3323215603 shutdown_connections 2024-05-27T07:40:53.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.340+0000 7f60a1458b80 1 -- 172.21.15.114:0/3323215603 wait complete. 2024-05-27T07:40:53.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.340+0000 7f60a1458b80 1 -- start start 2024-05-27T07:40:53.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f60a1458b80 1 Processor -- start 2024-05-27T07:40:53.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f60a1458b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd181feec0 0x55dd1839c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f60a1458b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 0x55dd1839a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f60a1458b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd18382cd0 0x55dd1839ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f60a1458b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f608c1af3f0 con 0x55dd1837d000 2024-05-27T07:40:53.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f60a1458b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f608c1b0860 con 0x55dd18382cd0 2024-05-27T07:40:53.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f60a1458b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f608c1afac0 con 0x55dd181feec0 2024-05-27T07:40:53.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f60a1458b80 10 client.0 _ll_register_callbacks cb 0x55dd182369a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:53.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f6083fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 0x55dd1839a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f6083fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 0x55dd1839a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41016/0 (socket says 172.21.15.114:41016) 2024-05-27T07:40:53.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f6083fff700 1 -- 172.21.15.114:0/250650185 learned_addr learned my addr 172.21.15.114:0/250650185 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:53.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f6090fe0700 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd18382cd0 0x55dd1839ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f608bfff700 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd181feec0 0x55dd1839c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f6083fff700 1 -- 172.21.15.114:0/250650185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd181feec0 msgr2=0x55dd1839c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:53.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f6083fff700 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd181feec0 0x55dd1839c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f6083fff700 1 -- 172.21.15.114:0/250650185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd18382cd0 msgr2=0x55dd1839ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:53.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f6083fff700 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd18382cd0 0x55dd1839ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f6083fff700 1 -- 172.21.15.114:0/250650185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd1839b4b0 con 0x55dd1837d000 2024-05-27T07:40:53.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.341+0000 7f6090fe0700 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd18382cd0 0x55dd1839ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:53.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f6083fff700 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 0x55dd1839a7c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f60740265c0 tx=0x7f607401d7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b32024233bec6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:53.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f606b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:53.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f606b7fe700 1 -- 172.21.15.114:0/250650185 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074034070 con 0x55dd1837d000 2024-05-27T07:40:53.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f606b7fe700 1 -- 172.21.15.114:0/250650185 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f6074026ac0 con 0x55dd1837d000 2024-05-27T07:40:53.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f60a1458b80 1 -- 172.21.15.114:0/250650185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f607c0a58b0 con 0x55dd1837d000 2024-05-27T07:40:53.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f606b7fe700 1 -- 172.21.15.114:0/250650185 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6074014040 con 0x55dd1837d000 2024-05-27T07:40:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f60a1458b80 10 client.15525 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f60a1458b80 1 -- 172.21.15.114:0/250650185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dd18401fd0 con 0x55dd1837d000 2024-05-27T07:40:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f606b7fe700 1 -- 172.21.15.114:0/250650185 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607c0a58b0 con 0x55dd1837d000 2024-05-27T07:40:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f606b7fe700 1 -- 172.21.15.114:0/250650185 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6074004fe0 con 0x55dd1837d000 2024-05-27T07:40:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f60a1458b80 10 client.15525 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f60a1458b80 10 client.15525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f60a1458b80 20 client.15525 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f60a1458b80 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dd183967c0 0x55dd184046b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f60a1458b80 4 client.15525 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:40:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.342+0000 7f60a1458b80 1 -- 172.21.15.114:0/250650185 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dd182adc60 con 0x55dd183967c0 2024-05-27T07:40:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.343+0000 7f608bfff700 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dd183967c0 0x55dd184046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.343+0000 7f606b7fe700 10 client.15525 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.343+0000 7f608bfff700 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dd183967c0 0x55dd184046b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56ed2ad57d7b01fa server_cookie=38e078d34eb7e547 in_seq=0 out_seq=0 2024-05-27T07:40:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.344+0000 7f606b7fe700 1 -- 172.21.15.114:0/250650185 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dd182adc60 con 0x55dd183967c0 2024-05-27T07:40:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.344+0000 7f606b7fe700 10 client.15525 handle_command_reply: tid=0 2024-05-27T07:40:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.344+0000 7f60a1458b80 1 client.15525 shutdown 2024-05-27T07:40:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.344+0000 7f60a1458b80 1 -- 172.21.15.114:0/250650185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 msgr2=0x55dd1839a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.344+0000 7f60a1458b80 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 0x55dd1839a7c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f60740265c0 tx=0x7f607401d7a0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.345+0000 7f60a1458b80 1 -- 172.21.15.114:0/250650185 shutdown_connections 2024-05-27T07:40:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.345+0000 7f60a1458b80 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dd183967c0 0x55dd184046b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.345+0000 7f60a1458b80 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd18382cd0 0x55dd1839ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.345+0000 7f606b7fe700 0 client.15525 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.345+0000 7f60a1458b80 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd181feec0 0x55dd1839c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.345+0000 7f60a1458b80 1 --2- 172.21.15.114:0/250650185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dd1837d000 0x55dd1839a7c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.345+0000 7f60a1458b80 1 -- 172.21.15.114:0/250650185 >> 172.21.15.114:0/250650185 conn(0x55dd18259b70 msgr2=0x55dd183863a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.345+0000 7f60a1458b80 1 -- 172.21.15.114:0/250650185 shutdown_connections 2024-05-27T07:40:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.345+0000 7f60a1458b80 1 -- 172.21.15.114:0/250650185 wait complete. 2024-05-27T07:40:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.345+0000 7f60a1458b80 20 client.15525 trim_cache size 0 max 16384 2024-05-27T07:40:53.353 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:40:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:40:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "2684f505-3ca5-40fe-9991-5fd169e125a1", 2024-05-27T07:40:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:40:53.354 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:40:53.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.346+0000 7f6092a43700 1 -- 172.21.15.114:0/3962500297 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60780b2b10 msgr2=0x7f60780b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:53.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.346+0000 7f6092a43700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60780b2b10 0x7f60780b5020 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6074005fd0 tx=0x7f6074005db0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.346+0000 7f6092a43700 1 -- 172.21.15.114:0/3962500297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f608c103b20 msgr2=0x7f608c19ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:53.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.346+0000 7f6092a43700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f608c103b20 0x7f608c19ae20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f607c00d950 tx=0x7f607c00dd10 comp rx=0 tx=0).stop 2024-05-27T07:40:53.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.346+0000 7f6092a43700 1 -- 172.21.15.114:0/3962500297 shutdown_connections 2024-05-27T07:40:53.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.346+0000 7f6092a43700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f608c108790 0x7f608c19f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.346+0000 7f6092a43700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f608c103b20 0x7f608c19ae20 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.346+0000 7f6092a43700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60780b2b10 0x7f60780b5020 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.346+0000 7f6092a43700 1 --2- 172.21.15.114:0/3962500297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f608c107e40 0x7f608c19b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:53.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.346+0000 7f6092a43700 1 -- 172.21.15.114:0/3962500297 >> 172.21.15.114:0/3962500297 conn(0x7f608c0fe510 msgr2=0x7f608c102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:53.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.346+0000 7f6092a43700 1 -- 172.21.15.114:0/3962500297 shutdown_connections 2024-05-27T07:40:53.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:53.346+0000 7f6092a43700 1 -- 172.21.15.114:0/3962500297 wait complete. 2024-05-27T07:40:53.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:53 smithi114 ceph-mon[32038]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:40:53.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:53 smithi163 ceph-mon[36451]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:40:53.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:53 smithi186 ceph-mon[36647]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:40:54.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:40:54.370 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:54.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:54 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:40:54.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:54 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:54.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:54 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:40:54.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:54 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:54.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:54 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:40:54.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:54 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:54.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:54 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:40:54.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:54 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:54.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:54 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:40:54.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:54 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:54.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:54 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:40:54.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:54 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:55.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:55 smithi114 ceph-mon[32038]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:40:55.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:55 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:40:55.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:55 smithi163 ceph-mon[36451]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:40:55.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:55 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:40:55.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:55 smithi186 ceph-mon[36647]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:40:55.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:55 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:40:56.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.165+0000 7f2e302bc700 1 -- 172.21.15.114:0/3720087780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e28107de0 msgr2=0x7f2e281081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.165+0000 7f2e302bc700 1 --2- 172.21.15.114:0/3720087780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e28107de0 0x7f2e281081f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c0099c0 tx=0x7f2e1c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.166+0000 7f2e302bc700 1 -- 172.21.15.114:0/3720087780 shutdown_connections 2024-05-27T07:40:56.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.166+0000 7f2e302bc700 1 --2- 172.21.15.114:0/3720087780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e28108730 0x7f2e28110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.166+0000 7f2e302bc700 1 --2- 172.21.15.114:0/3720087780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e281034b0 0x7f2e281038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.166+0000 7f2e302bc700 1 --2- 172.21.15.114:0/3720087780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e28107de0 0x7f2e281081f0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.166+0000 7f2e302bc700 1 -- 172.21.15.114:0/3720087780 >> 172.21.15.114:0/3720087780 conn(0x7f2e280fedb0 msgr2=0x7f2e281011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:56.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.166+0000 7f2e302bc700 1 -- 172.21.15.114:0/3720087780 shutdown_connections 2024-05-27T07:40:56.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.166+0000 7f2e302bc700 1 -- 172.21.15.114:0/3720087780 wait complete. 2024-05-27T07:40:56.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.166+0000 7f2e302bc700 1 Processor -- start 2024-05-27T07:40:56.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.166+0000 7f2e302bc700 1 -- start start 2024-05-27T07:40:56.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e302bc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e281034b0 0x7f2e2819f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e302bc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e28107de0 0x7f2e2819fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e302bc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e28108730 0x7f2e28199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e302bc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2e28113360 con 0x7f2e28107de0 2024-05-27T07:40:56.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e302bc700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281131e0 con 0x7f2e28108730 2024-05-27T07:40:56.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e302bc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281134e0 con 0x7f2e281034b0 2024-05-27T07:40:56.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e2d857700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e28107de0 0x7f2e2819fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e2d857700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e28107de0 0x7f2e2819fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36876/0 (socket says 172.21.15.114:36876) 2024-05-27T07:40:56.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e2d857700 1 -- 172.21.15.114:0/717023151 learned_addr learned my addr 172.21.15.114:0/717023151 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:56.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e2e058700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e281034b0 0x7f2e2819f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e2e859700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e28108730 0x7f2e28199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e2d857700 1 -- 172.21.15.114:0/717023151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e281034b0 msgr2=0x7f2e2819f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e2d857700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e281034b0 0x7f2e2819f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e2d857700 1 -- 172.21.15.114:0/717023151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e28108730 msgr2=0x7f2e28199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e2d857700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e28108730 0x7f2e28199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.167+0000 7f2e2d857700 1 -- 172.21.15.114:0/717023151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2819a270 con 0x7f2e28107de0 2024-05-27T07:40:56.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.168+0000 7f2e2d857700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e28107de0 0x7f2e2819fef0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c006010 tx=0x7f2e1c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:56.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.168+0000 7f2e277fe700 1 -- 172.21.15.114:0/717023151 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c027070 con 0x7f2e28107de0 2024-05-27T07:40:56.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.168+0000 7f2e277fe700 1 -- 172.21.15.114:0/717023151 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e1c003f20 con 0x7f2e28107de0 2024-05-27T07:40:56.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.168+0000 7f2e302bc700 1 -- 172.21.15.114:0/717023151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e2819a500 con 0x7f2e28107de0 2024-05-27T07:40:56.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.168+0000 7f2e277fe700 1 -- 172.21.15.114:0/717023151 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c022c60 con 0x7f2e28107de0 2024-05-27T07:40:56.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.168+0000 7f2e302bc700 1 -- 172.21.15.114:0/717023151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e281a6760 con 0x7f2e28107de0 2024-05-27T07:40:56.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.171+0000 7f2e277fe700 1 -- 172.21.15.114:0/717023151 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7f2e1c02a070 con 0x7f2e28107de0 2024-05-27T07:40:56.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.171+0000 7f2e277fe700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2e100b2b10 0x7f2e100b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.171+0000 7f2e2e058700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2e100b2b10 0x7f2e100b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.172+0000 7f2e277fe700 1 -- 172.21.15.114:0/717023151 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f2e1c022e00 con 0x7f2e28107de0 2024-05-27T07:40:56.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.172+0000 7f2e2e058700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2e100b2b10 0x7f2e100b5020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2e180060b0 tx=0x7f2e1800c040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:56.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.172+0000 7f2e3ecd1b80 1 Processor -- start 2024-05-27T07:40:56.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.172+0000 7f2e3ecd1b80 1 -- start start 2024-05-27T07:40:56.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.172+0000 7f2e3ecd1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb73c4e40 0x55ecb75460b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.172+0000 7f2e3ecd1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb75465f0 0x55ecb754a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.172+0000 7f2e3ecd1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 0x55ecb754b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.172+0000 7f2e3ecd1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ecb7471d70 con 0x55ecb754ade0 2024-05-27T07:40:56.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.172+0000 7f2e3ecd1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ecb7461910 con 0x55ecb75465f0 2024-05-27T07:40:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e3ecd1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ecb7465950 con 0x55ecb73c4e40 2024-05-27T07:40:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e2e859700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 0x55ecb754b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e2e859700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 0x55ecb754b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36882/0 (socket says 172.21.15.114:36882) 2024-05-27T07:40:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e2e859700 1 -- 172.21.15.114:0/991083041 learned_addr learned my addr 172.21.15.114:0/991083041 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e2d857700 1 --2- 172.21.15.114:0/991083041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb75465f0 0x55ecb754a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e2e058700 1 --2- 172.21.15.114:0/991083041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb73c4e40 0x55ecb75460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e2d857700 1 -- 172.21.15.114:0/991083041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb73c4e40 msgr2=0x55ecb75460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e2d857700 1 --2- 172.21.15.114:0/991083041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb73c4e40 0x55ecb75460b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e2d857700 1 -- 172.21.15.114:0/991083041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 msgr2=0x55ecb754b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e2d857700 1 --2- 172.21.15.114:0/991083041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 0x55ecb754b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e2d857700 1 -- 172.21.15.114:0/991083041 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecb740afb0 con 0x55ecb75465f0 2024-05-27T07:40:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e2e859700 1 --2- 172.21.15.114:0/991083041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 0x55ecb754b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:40:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e2d857700 1 --2- 172.21.15.114:0/991083041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb75465f0 0x55ecb754a8a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c0e0cc0 tx=0x7f2e1c0aa070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f04a3be57243f39 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:56.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.173+0000 7f2e24d47700 1 -- 172.21.15.114:0/991083041 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c0cd0d0 con 0x55ecb75465f0 2024-05-27T07:40:56.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.174+0000 7f2e24d47700 1 -- 172.21.15.114:0/991083041 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e1c0b10f0 con 0x55ecb75465f0 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.174+0000 7f2e24d47700 1 -- 172.21.15.114:0/991083041 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c0aacd0 con 0x55ecb75465f0 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.174+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/991083041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb75465f0 msgr2=0x55ecb754a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.174+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/991083041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb75465f0 0x55ecb754a8a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c0e0cc0 tx=0x7f2e1c0aa070 comp rx=0 tx=0).stop 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.174+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/991083041 shutdown_connections 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.174+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/991083041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 0x55ecb754b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.174+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/991083041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb73c4e40 0x55ecb75460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.174+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/991083041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb75465f0 0x55ecb754a8a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.174+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/991083041 >> 172.21.15.114:0/991083041 conn(0x55ecb73c7770 msgr2=0x55ecb753ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.175+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/991083041 shutdown_connections 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.175+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/991083041 wait complete. 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.175+0000 7f2e3ecd1b80 1 -- start start 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.175+0000 7f2e3ecd1b80 1 Processor -- start 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.175+0000 7f2e3ecd1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 0x55ecb754fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.175+0000 7f2e3ecd1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb75465f0 0x55ecb7550090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.175+0000 7f2e3ecd1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb754ade0 0x55ecb75505d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.175+0000 7f2e3ecd1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ecb7367fa0 con 0x55ecb73c4e40 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.175+0000 7f2e3ecd1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ecb73c9da0 con 0x55ecb754ade0 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e3ecd1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ecb7552b50 con 0x55ecb75465f0 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e2e058700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 0x55ecb754fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e3ecd1b80 10 client.0 _ll_register_callbacks cb 0x55ecb73f85b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e2e058700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 0x55ecb754fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36892/0 (socket says 172.21.15.114:36892) 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e2e058700 1 -- 172.21.15.114:0/1586586091 learned_addr learned my addr 172.21.15.114:0/1586586091 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e2d857700 1 --2- 172.21.15.114:0/1586586091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb75465f0 0x55ecb7550090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e2e859700 1 --2- 172.21.15.114:0/1586586091 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb754ade0 0x55ecb75505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e2e058700 1 -- 172.21.15.114:0/1586586091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb75465f0 msgr2=0x55ecb7550090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e2e058700 1 --2- 172.21.15.114:0/1586586091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb75465f0 0x55ecb7550090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e2e058700 1 -- 172.21.15.114:0/1586586091 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb754ade0 msgr2=0x55ecb75505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:56.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e2e058700 1 --2- 172.21.15.114:0/1586586091 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb754ade0 0x55ecb75505d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e2e058700 1 -- 172.21.15.114:0/1586586091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecb73b18d0 con 0x55ecb73c4e40 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e2e058700 1 --2- 172.21.15.114:0/1586586091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 0x55ecb754fb50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f2e18016cd0 tx=0x7f2e18004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d888f08f2ab26695 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e0dffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e0dffb700 1 -- 172.21.15.114:0/1586586091 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18004bb0 con 0x55ecb73c4e40 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e0dffb700 1 -- 172.21.15.114:0/1586586091 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f2e1801f040 con 0x55ecb73c4e40 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/1586586091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e1c0b10f0 con 0x55ecb73c4e40 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e0dffb700 1 -- 172.21.15.114:0/1586586091 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2e1802f050 con 0x55ecb73c4e40 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e3ecd1b80 10 client.15537 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.176+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/1586586091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ecb741b760 con 0x55ecb73c4e40 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.177+0000 7f2e0dffb700 1 -- 172.21.15.114:0/1586586091 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e1c0b10f0 con 0x55ecb73c4e40 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.177+0000 7f2e0dffb700 1 -- 172.21.15.114:0/1586586091 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2e18022930 con 0x55ecb73c4e40 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.177+0000 7f2e3ecd1b80 10 client.15537 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.177+0000 7f2e3ecd1b80 10 client.15537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.177+0000 7f2e3ecd1b80 20 client.15537 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:56.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.177+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/1586586091 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecb755fa30 0x55ecb7561eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.177+0000 7f2e3ecd1b80 4 client.15537 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:56.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.177+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/1586586091 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecb7473cc0 con 0x55ecb755fa30 2024-05-27T07:40:56.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.177+0000 7f2e2d857700 1 --2- 172.21.15.114:0/1586586091 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecb755fa30 0x55ecb7561eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.178+0000 7f2e0dffb700 10 client.15537 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:56.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.178+0000 7f2e2d857700 1 --2- 172.21.15.114:0/1586586091 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecb755fa30 0x55ecb7561eb0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2070ab658847206a server_cookie=9571b12c6089614 in_seq=0 out_seq=0 2024-05-27T07:40:56.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.178+0000 7f2e0dffb700 1 -- 172.21.15.114:0/1586586091 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ecb7473cc0 con 0x55ecb755fa30 2024-05-27T07:40:56.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.178+0000 7f2e0dffb700 10 client.15537 handle_command_reply: tid=0 2024-05-27T07:40:56.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.178+0000 7f2e3ecd1b80 1 client.15537 shutdown 2024-05-27T07:40:56.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.179+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/1586586091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 msgr2=0x55ecb754fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.179+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/1586586091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 0x55ecb754fb50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f2e18016cd0 tx=0x7f2e18004ad0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.179+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/1586586091 shutdown_connections 2024-05-27T07:40:56.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.179+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/1586586091 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecb755fa30 0x55ecb7561eb0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.179+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/1586586091 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb754ade0 0x55ecb75505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.179+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/1586586091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 0x55ecb754fb50 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.179+0000 7f2e0dffb700 0 client.15537 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:56.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.179+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/1586586091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb75465f0 0x55ecb7550090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.179+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/1586586091 >> 172.21.15.114:0/1586586091 conn(0x55ecb73c7770 msgr2=0x55ecb7542bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:56.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.179+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/1586586091 shutdown_connections 2024-05-27T07:40:56.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.179+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/1586586091 wait complete. 2024-05-27T07:40:56.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.179+0000 7f2e3ecd1b80 20 client.15537 trim_cache size 0 max 16384 2024-05-27T07:40:56.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e3ecd1b80 1 Processor -- start 2024-05-27T07:40:56.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e3ecd1b80 1 -- start start 2024-05-27T07:40:56.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e3ecd1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 0x55ecb754eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e3ecd1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb75465f0 0x55ecb7550410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e3ecd1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb754ade0 0x55ecb7545610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e3ecd1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281ac2e0 con 0x55ecb73c4e40 2024-05-27T07:40:56.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e3ecd1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281aeb60 con 0x55ecb75465f0 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e3ecd1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281ac950 con 0x55ecb754ade0 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e2e058700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 0x55ecb754eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e2e058700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 0x55ecb754eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36894/0 (socket says 172.21.15.114:36894) 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e2e058700 1 -- 172.21.15.114:0/2461302901 learned_addr learned my addr 172.21.15.114:0/2461302901 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e2d857700 1 --2- 172.21.15.114:0/2461302901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb75465f0 0x55ecb7550410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e2e859700 1 --2- 172.21.15.114:0/2461302901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb754ade0 0x55ecb7545610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e2e058700 1 -- 172.21.15.114:0/2461302901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb754ade0 msgr2=0x55ecb7545610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.201+0000 7f2e2e058700 1 --2- 172.21.15.114:0/2461302901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb754ade0 0x55ecb7545610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.202+0000 7f2e2e058700 1 -- 172.21.15.114:0/2461302901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb75465f0 msgr2=0x55ecb7550410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.202+0000 7f2e2e058700 1 --2- 172.21.15.114:0/2461302901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb75465f0 0x55ecb7550410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.202+0000 7f2e2e058700 1 -- 172.21.15.114:0/2461302901 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecb7550950 con 0x55ecb73c4e40 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.202+0000 7f2e2e058700 1 --2- 172.21.15.114:0/2461302901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 0x55ecb754eb10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2e18022dd0 tx=0x7f2e18007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18bef62adf2b8477 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.202+0000 7f2e0ffff700 1 -- 172.21.15.114:0/2461302901 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18004b40 con 0x55ecb73c4e40 2024-05-27T07:40:56.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.202+0000 7f2e0ffff700 1 -- 172.21.15.114:0/2461302901 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e18004ce0 con 0x55ecb73c4e40 2024-05-27T07:40:56.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.202+0000 7f2e0ffff700 1 -- 172.21.15.114:0/2461302901 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18033430 con 0x55ecb73c4e40 2024-05-27T07:40:56.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.203+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/2461302901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 msgr2=0x55ecb754eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.203+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/2461302901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 0x55ecb754eb10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2e18022dd0 tx=0x7f2e18007f90 comp rx=0 tx=0).stop 2024-05-27T07:40:56.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.203+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/2461302901 shutdown_connections 2024-05-27T07:40:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.203+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/2461302901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb754ade0 0x55ecb7545610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.203+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/2461302901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb73c4e40 0x55ecb754eb10 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.203+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/2461302901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb75465f0 0x55ecb7550410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.203+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/2461302901 >> 172.21.15.114:0/2461302901 conn(0x55ecb73c7770 msgr2=0x55ecb75384f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.203+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/2461302901 shutdown_connections 2024-05-27T07:40:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.203+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/2461302901 wait complete. 2024-05-27T07:40:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.203+0000 7f2e3ecd1b80 1 -- start start 2024-05-27T07:40:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.203+0000 7f2e3ecd1b80 1 Processor -- start 2024-05-27T07:40:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e3ecd1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb73c4e40 0x55ecb75671a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e3ecd1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb75465f0 0x55ecb75676e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e3ecd1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 0x55ecb7567c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e3ecd1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281ad020 con 0x55ecb754ade0 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e3ecd1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281ae490 con 0x55ecb73c4e40 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e3ecd1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281ad6f0 con 0x55ecb75465f0 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e3ecd1b80 10 client.0 _ll_register_callbacks cb 0x55ecb754ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e2e859700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 0x55ecb7567c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e2e859700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 0x55ecb7567c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36902/0 (socket says 172.21.15.114:36902) 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e2e859700 1 -- 172.21.15.114:0/2663819711 learned_addr learned my addr 172.21.15.114:0/2663819711 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e2e058700 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb73c4e40 0x55ecb75671a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e2d857700 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb75465f0 0x55ecb75676e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e2e859700 1 -- 172.21.15.114:0/2663819711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb75465f0 msgr2=0x55ecb75676e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e2e859700 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb75465f0 0x55ecb75676e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e2e859700 1 -- 172.21.15.114:0/2663819711 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb73c4e40 msgr2=0x55ecb75671a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e2e859700 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb73c4e40 0x55ecb75671a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e2e859700 1 -- 172.21.15.114:0/2663819711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecb75561e0 con 0x55ecb754ade0 2024-05-27T07:40:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e2d857700 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb75465f0 0x55ecb75676e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e2e859700 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 0x55ecb7567c20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000bcd0 tx=0x7f2e20003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8b48da248eb6ec server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e0dffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:40:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.204+0000 7f2e0dffb700 1 -- 172.21.15.114:0/2663819711 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e20011640 con 0x55ecb754ade0 2024-05-27T07:40:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/2663819711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e18004ce0 con 0x55ecb754ade0 2024-05-27T07:40:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e0dffb700 1 -- 172.21.15.114:0/2663819711 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f2e200117e0 con 0x55ecb754ade0 2024-05-27T07:40:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e0dffb700 1 -- 172.21.15.114:0/2663819711 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2e2002a050 con 0x55ecb754ade0 2024-05-27T07:40:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e3ecd1b80 10 client.15543 fetch_fsmap learned FSMap version 21 2024-05-27T07:40:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/2663819711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ecb75ccb70 con 0x55ecb754ade0 2024-05-27T07:40:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e0dffb700 1 -- 172.21.15.114:0/2663819711 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e18004ce0 con 0x55ecb754ade0 2024-05-27T07:40:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e0dffb700 1 -- 172.21.15.114:0/2663819711 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2e2001cb30 con 0x55ecb754ade0 2024-05-27T07:40:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e3ecd1b80 10 client.15543 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:40:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e3ecd1b80 10 client.15543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:40:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e3ecd1b80 20 client.15543 populate_metadata read hostname 'smithi114' 2024-05-27T07:40:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecb75cd280 0x55ecb75cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e3ecd1b80 4 client.15543 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:40:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/2663819711 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ecb746d4a0 con 0x55ecb75cd280 2024-05-27T07:40:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.205+0000 7f2e2e058700 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecb75cd280 0x55ecb75cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e2e058700 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecb75cd280 0x55ecb75cf6a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f4176134858e052 server_cookie=b2aa72895de43cb4 in_seq=0 out_seq=0 2024-05-27T07:40:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e0dffb700 10 client.15543 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e0dffb700 1 -- 172.21.15.114:0/2663819711 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ecb746d4a0 con 0x55ecb75cd280 2024-05-27T07:40:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e0dffb700 10 client.15543 handle_command_reply: tid=0 2024-05-27T07:40:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e3ecd1b80 1 client.15543 shutdown 2024-05-27T07:40:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/2663819711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 msgr2=0x55ecb7567c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 0x55ecb7567c20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000bcd0 tx=0x7f2e20003680 comp rx=0 tx=0).stop 2024-05-27T07:40:56.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/2663819711 shutdown_connections 2024-05-27T07:40:56.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecb754ade0 0x55ecb7567c20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecb75cd280 0x55ecb75cf6a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecb73c4e40 0x55ecb75671a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e3ecd1b80 1 --2- 172.21.15.114:0/2663819711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecb75465f0 0x55ecb75676e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e0dffb700 0 client.15543 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:40:56.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/2663819711 >> 172.21.15.114:0/2663819711 conn(0x55ecb73c7770 msgr2=0x55ecb7538c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:56.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/2663819711 shutdown_connections 2024-05-27T07:40:56.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.206+0000 7f2e3ecd1b80 1 -- 172.21.15.114:0/2663819711 wait complete. 2024-05-27T07:40:56.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.207+0000 7f2e3ecd1b80 20 client.15543 trim_cache size 0 max 16384 2024-05-27T07:40:56.211 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:40:56.211 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:40:56.211 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:40:56.211 INFO:teuthology.orchestra.run.smithi114.stdout: "2684f505-3ca5-40fe-9991-5fd169e125a1": { 2024-05-27T07:40:56.211 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:40:56.211 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "2684f505-3ca5-40fe-9991-5fd169e125a1", 2024-05-27T07:40:56.211 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:40:56.211 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:40:56.211 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:40:56.211 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:40:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.207+0000 7f2e302bc700 1 -- 172.21.15.114:0/717023151 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2e100b2b10 msgr2=0x7f2e100b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.207+0000 7f2e302bc700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2e100b2b10 0x7f2e100b5020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2e180060b0 tx=0x7f2e1800c040 comp rx=0 tx=0).stop 2024-05-27T07:40:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.207+0000 7f2e302bc700 1 -- 172.21.15.114:0/717023151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e28107de0 msgr2=0x7f2e2819fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.207+0000 7f2e302bc700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e28107de0 0x7f2e2819fef0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c006010 tx=0x7f2e1c0047b0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.207+0000 7f2e302bc700 1 -- 172.21.15.114:0/717023151 shutdown_connections 2024-05-27T07:40:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.207+0000 7f2e302bc700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2e100b2b10 0x7f2e100b5020 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.207+0000 7f2e302bc700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e28108730 0x7f2e28199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.207+0000 7f2e302bc700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e281034b0 0x7f2e2819f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.207+0000 7f2e302bc700 1 --2- 172.21.15.114:0/717023151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e28107de0 0x7f2e2819fef0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.207+0000 7f2e302bc700 1 -- 172.21.15.114:0/717023151 >> 172.21.15.114:0/717023151 conn(0x7f2e280fedb0 msgr2=0x7f2e28109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.207+0000 7f2e302bc700 1 -- 172.21.15.114:0/717023151 shutdown_connections 2024-05-27T07:40:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:40:56.207+0000 7f2e302bc700 1 -- 172.21.15.114:0/717023151 wait complete. 2024-05-27T07:40:56.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:56 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:40:56.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:56 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:40:56.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:56 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:40:56.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:56 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:40:56.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:56 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:40:56.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:56 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:40:56.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:56 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:40:56.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:56 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:40:56.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:56 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:40:57.075 INFO:tasks.cephfs.filesystem:scrub status for tag:2684f505-3ca5-40fe-9991-5fd169e125a1 - {'path': '/', 'tag': '2684f505-3ca5-40fe-9991-5fd169e125a1', 'options': 'recursive,force'} 2024-05-27T07:40:57.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:57 smithi114 ceph-mon[32038]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:40:57.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:57 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:40:57.624 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:57 smithi163 ceph-mon[36451]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:40:57.625 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:57 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:40:57.627 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:57 smithi186 ceph-mon[36647]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:40:57.627 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:57 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:40:59.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:40:59 smithi114 ceph-mon[32038]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:40:59.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:40:59 smithi163 ceph-mon[36451]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:40:59.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:40:59 smithi186 ceph-mon[36647]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:41:01.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:01 smithi114 ceph-mon[32038]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:41:01.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:01 smithi163 ceph-mon[36451]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:41:01.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:01 smithi186 ceph-mon[36647]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:41:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.869+0000 7fa7e30d1640 1 -- 172.21.15.114:0/2280856697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7dc104e20 msgr2=0x7fa7dc105230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.869+0000 7fa7e30d1640 1 --2- 172.21.15.114:0/2280856697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7dc104e20 0x7fa7dc105230 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009900 tx=0x7fa7d002a480 comp rx=0 tx=0).stop 2024-05-27T07:41:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.869+0000 7fa7e30d1640 1 -- 172.21.15.114:0/2280856697 shutdown_connections 2024-05-27T07:41:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.869+0000 7fa7e30d1640 1 --2- 172.21.15.114:0/2280856697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7dc10e230 0x7fa7dc110700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.869+0000 7fa7e30d1640 1 --2- 172.21.15.114:0/2280856697 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7dc10b880 0x7fa7dc10dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.869+0000 7fa7e30d1640 1 --2- 172.21.15.114:0/2280856697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7dc104e20 0x7fa7dc105230 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.869+0000 7fa7e30d1640 1 -- 172.21.15.114:0/2280856697 >> 172.21.15.114:0/2280856697 conn(0x7fa7dc078270 msgr2=0x7fa7dc0786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.869+0000 7fa7e30d1640 1 -- 172.21.15.114:0/2280856697 shutdown_connections 2024-05-27T07:41:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.869+0000 7fa7e30d1640 1 -- 172.21.15.114:0/2280856697 wait complete. 2024-05-27T07:41:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.869+0000 7fa7e30d1640 1 Processor -- start 2024-05-27T07:41:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7e30d1640 1 -- start start 2024-05-27T07:41:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7e30d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7dc104e20 0x7fa7dc1a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7e30d1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7dc10b880 0x7fa7dc1a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7e30d1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7dc10e230 0x7fa7dc19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7e30d1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc113e70 con 0x7fa7dc10b880 2024-05-27T07:41:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7e30d1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc113cf0 con 0x7fa7dc10e230 2024-05-27T07:41:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7e30d1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc113ff0 con 0x7fa7dc104e20 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7dbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7dc10b880 0x7fa7dc1a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7dbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7dc10b880 0x7fa7dc1a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36918/0 (socket says 172.21.15.114:36918) 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7dbfff640 1 -- 172.21.15.114:0/323342985 learned_addr learned my addr 172.21.15.114:0/323342985 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7dc104e20 0x7fa7dc1a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7e166e640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7dc10e230 0x7fa7dc19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7dbfff640 1 -- 172.21.15.114:0/323342985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7dc104e20 msgr2=0x7fa7dc1a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7dbfff640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7dc104e20 0x7fa7dc1a0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7dbfff640 1 -- 172.21.15.114:0/323342985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7dc10e230 msgr2=0x7fa7dc19a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7dbfff640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7dc10e230 0x7fa7dc19a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7dbfff640 1 -- 172.21.15.114:0/323342985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7dc19adc0 con 0x7fa7dc10b880 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7dc104e20 0x7fa7dc1a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.870+0000 7fa7dbfff640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7dc10b880 0x7fa7dc1a0a50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800d950 tx=0x7fa7c800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.871+0000 7fa7d9ffb640 1 -- 172.21.15.114:0/323342985 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8017070 con 0x7fa7dc10b880 2024-05-27T07:41:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.871+0000 7fa7d9ffb640 1 -- 172.21.15.114:0/323342985 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7c8009c30 con 0x7fa7dc10b880 2024-05-27T07:41:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.871+0000 7fa7e30d1640 1 -- 172.21.15.114:0/323342985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7dc19b050 con 0x7fa7dc10b880 2024-05-27T07:41:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.871+0000 7fa7d9ffb640 1 -- 172.21.15.114:0/323342985 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c800f620 con 0x7fa7dc10b880 2024-05-27T07:41:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.871+0000 7fa7e30d1640 1 -- 172.21.15.114:0/323342985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7dc1a72b0 con 0x7fa7dc10b880 2024-05-27T07:41:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.873+0000 7fa7e9f81740 1 Processor -- start 2024-05-27T07:41:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e9f81740 1 -- start start 2024-05-27T07:41:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7d9ffb640 1 -- 172.21.15.114:0/323342985 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7fa7c8010460 con 0x7fa7dc10b880 2024-05-27T07:41:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e9f81740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afc61b0 0x557f6b3673d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e9f81740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b367910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e9f81740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afa54d0 0x557f6b367e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e9f81740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557f6b2d9c30 con 0x557f6af9d2f0 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e9f81740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557f6b2c97d0 con 0x557f6afc61b0 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e9f81740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557f6b2cd810 con 0x557f6afa54d0 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e0e6d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b367910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e0e6d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b367910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36922/0 (socket says 172.21.15.114:36922) 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e0e6d640 1 -- 172.21.15.114:0/2416954255 learned_addr learned my addr 172.21.15.114:0/2416954255 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7dbfff640 1 --2- 172.21.15.114:0/2416954255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afa54d0 0x557f6b367e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e166e640 1 --2- 172.21.15.114:0/2416954255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afc61b0 0x557f6b3673d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e0e6d640 1 -- 172.21.15.114:0/2416954255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afa54d0 msgr2=0x557f6b367e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/2416954255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afa54d0 0x557f6b367e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e0e6d640 1 -- 172.21.15.114:0/2416954255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afc61b0 msgr2=0x557f6b3673d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/2416954255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afc61b0 0x557f6b3673d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e0e6d640 1 -- 172.21.15.114:0/2416954255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f6afde210 con 0x557f6af9d2f0 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.874+0000 7fa7e166e640 1 --2- 172.21.15.114:0/2416954255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afc61b0 0x557f6b3673d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.875+0000 7fa7d9ffb640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7cc0b2a70 0x7fa7cc0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.875+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/2416954255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b367910 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009900 tx=0x7fa7d0032e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=714907fb1e337c00 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.875+0000 7fa7e166e640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7cc0b2a70 0x7fa7cc0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.875+0000 7fa7c6ffd640 1 -- 172.21.15.114:0/2416954255 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d003f070 con 0x557f6af9d2f0 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.875+0000 7fa7c6ffd640 1 -- 172.21.15.114:0/2416954255 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7d00316a0 con 0x557f6af9d2f0 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.875+0000 7fa7d9ffb640 1 -- 172.21.15.114:0/323342985 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fa7c80e2980 con 0x7fa7dc10b880 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.875+0000 7fa7c6ffd640 1 -- 172.21.15.114:0/2416954255 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0035070 con 0x557f6af9d2f0 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.876+0000 7fa7e9f81740 1 -- 172.21.15.114:0/2416954255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 msgr2=0x557f6b367910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.876+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/2416954255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b367910 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009900 tx=0x7fa7d0032e70 comp rx=0 tx=0).stop 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.876+0000 7fa7e166e640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7cc0b2a70 0x7fa7cc0b4f80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x557f6b364540 tx=0x7fa7d400a3d0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.876+0000 7fa7e9f81740 1 -- 172.21.15.114:0/2416954255 shutdown_connections 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.876+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/2416954255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afa54d0 0x557f6b367e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.876+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/2416954255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b367910 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.876+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/2416954255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afc61b0 0x557f6b3673d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.876+0000 7fa7e9f81740 1 -- 172.21.15.114:0/2416954255 >> 172.21.15.114:0/2416954255 conn(0x557f6afba300 msgr2=0x557f6b363b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.876+0000 7fa7e9f81740 1 -- 172.21.15.114:0/2416954255 shutdown_connections 2024-05-27T07:41:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.876+0000 7fa7e9f81740 1 -- 172.21.15.114:0/2416954255 wait complete. 2024-05-27T07:41:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.876+0000 7fa7e9f81740 1 -- start start 2024-05-27T07:41:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.876+0000 7fa7e9f81740 1 Processor -- start 2024-05-27T07:41:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e9f81740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b371ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e9f81740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afa54d0 0x557f6b372030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e9f81740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afc61b0 0x557f6b377810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e9f81740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557f6afaf790 con 0x557f6af9d2f0 2024-05-27T07:41:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e9f81740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557f6b285890 con 0x557f6afa54d0 2024-05-27T07:41:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e9f81740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557f6b3744a0 con 0x557f6afc61b0 2024-05-27T07:41:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e9f81740 10 client.0 _ll_register_callbacks cb 0x557f6afdda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e0e6d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b371ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e0e6d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b371ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36928/0 (socket says 172.21.15.114:36928) 2024-05-27T07:41:01.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e0e6d640 1 -- 172.21.15.114:0/1414261706 learned_addr learned my addr 172.21.15.114:0/1414261706 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:01.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7dbfff640 1 --2- 172.21.15.114:0/1414261706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afa54d0 0x557f6b372030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e166e640 1 --2- 172.21.15.114:0/1414261706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afc61b0 0x557f6b377810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e0e6d640 1 -- 172.21.15.114:0/1414261706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afc61b0 msgr2=0x557f6b377810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:01.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/1414261706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afc61b0 0x557f6b377810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e0e6d640 1 -- 172.21.15.114:0/1414261706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afa54d0 msgr2=0x557f6b372030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:01.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/1414261706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afa54d0 0x557f6b372030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e0e6d640 1 -- 172.21.15.114:0/1414261706 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f6b0aa620 con 0x557f6af9d2f0 2024-05-27T07:41:01.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/1414261706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b371ad0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0042000 tx=0x7fa7d0005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749b27d61a3dc5b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:01.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.877+0000 7fa7bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:41:01.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7bbfff640 1 -- 172.21.15.114:0/1414261706 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d004b060 con 0x557f6af9d2f0 2024-05-27T07:41:01.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7bbfff640 1 -- 172.21.15.114:0/1414261706 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fa7d0005560 con 0x557f6af9d2f0 2024-05-27T07:41:01.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7e9f81740 1 -- 172.21.15.114:0/1414261706 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7d00316a0 con 0x557f6af9d2f0 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7bbfff640 1 -- 172.21.15.114:0/1414261706 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa7d0052050 con 0x557f6af9d2f0 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7e9f81740 10 client.15555 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7e9f81740 1 -- 172.21.15.114:0/1414261706 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557f6b380750 con 0x557f6af9d2f0 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7bbfff640 1 -- 172.21.15.114:0/1414261706 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7d00316a0 con 0x557f6af9d2f0 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7bbfff640 1 -- 172.21.15.114:0/1414261706 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fa7d00035f0 con 0x557f6af9d2f0 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7e9f81740 10 client.15555 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7e9f81740 10 client.15555 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7e9f81740 20 client.15555 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/1414261706 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f6af84d80 0x557f6b383ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7e9f81740 4 client.15555 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.878+0000 7fa7e9f81740 1 -- 172.21.15.114:0/1414261706 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f6b2dbb80 con 0x557f6af84d80 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.879+0000 7fa7dbfff640 1 --2- 172.21.15.114:0/1414261706 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f6af84d80 0x557f6b383ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.879+0000 7fa7dbfff640 1 --2- 172.21.15.114:0/1414261706 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f6af84d80 0x557f6b383ab0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33831306f6a0cd29 server_cookie=3198e2cf12423302 in_seq=0 out_seq=0 2024-05-27T07:41:01.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.879+0000 7fa7bbfff640 10 client.15555 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:01.883 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7bbfff640 1 -- 172.21.15.114:0/1414261706 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557f6b2dbb80 con 0x557f6af84d80 2024-05-27T07:41:01.883 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7bbfff640 10 client.15555 handle_command_reply: tid=0 2024-05-27T07:41:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7e9f81740 1 client.15555 shutdown 2024-05-27T07:41:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7e9f81740 1 -- 172.21.15.114:0/1414261706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 msgr2=0x557f6b371ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/1414261706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b371ad0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0042000 tx=0x7fa7d0005340 comp rx=0 tx=0).stop 2024-05-27T07:41:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7e9f81740 1 -- 172.21.15.114:0/1414261706 shutdown_connections 2024-05-27T07:41:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/1414261706 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f6af84d80 0x557f6b383ab0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/1414261706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afc61b0 0x557f6b377810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/1414261706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afa54d0 0x557f6b372030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7bbfff640 0 client.15555 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/1414261706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b371ad0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7e9f81740 1 -- 172.21.15.114:0/1414261706 >> 172.21.15.114:0/1414261706 conn(0x557f6afba300 msgr2=0x557f6b3690f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7e9f81740 1 -- 172.21.15.114:0/1414261706 shutdown_connections 2024-05-27T07:41:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7e9f81740 1 -- 172.21.15.114:0/1414261706 wait complete. 2024-05-27T07:41:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.880+0000 7fa7e9f81740 20 client.15555 trim_cache size 0 max 16384 2024-05-27T07:41:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.895+0000 7fa7e9f81740 1 Processor -- start 2024-05-27T07:41:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.895+0000 7fa7e9f81740 1 -- start start 2024-05-27T07:41:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e9f81740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b37e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e9f81740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afa54d0 0x557f6b368170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e9f81740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afc61b0 0x557f6b36b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e9f81740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc1af270 con 0x557f6af9d2f0 2024-05-27T07:41:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e9f81740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc1b1af0 con 0x557f6afa54d0 2024-05-27T07:41:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e9f81740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc1af8e0 con 0x557f6afc61b0 2024-05-27T07:41:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e0e6d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b37e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e0e6d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b37e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36932/0 (socket says 172.21.15.114:36932) 2024-05-27T07:41:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e0e6d640 1 -- 172.21.15.114:0/541596988 learned_addr learned my addr 172.21.15.114:0/541596988 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e166e640 1 --2- 172.21.15.114:0/541596988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afc61b0 0x557f6b36b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7dbfff640 1 --2- 172.21.15.114:0/541596988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afa54d0 0x557f6b368170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e0e6d640 1 -- 172.21.15.114:0/541596988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afc61b0 msgr2=0x557f6b36b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/541596988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afc61b0 0x557f6b36b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e0e6d640 1 -- 172.21.15.114:0/541596988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afa54d0 msgr2=0x557f6b368170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/541596988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afa54d0 0x557f6b368170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e0e6d640 1 -- 172.21.15.114:0/541596988 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f6b364aa0 con 0x557f6af9d2f0 2024-05-27T07:41:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e166e640 1 --2- 172.21.15.114:0/541596988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afc61b0 0x557f6b36b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7dbfff640 1 --2- 172.21.15.114:0/541596988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afa54d0 0x557f6b368170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/541596988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b37e560 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0032eb0 tx=0x7fa7d0005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc0ad6e667261b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.896+0000 7fa7c67fc640 1 -- 172.21.15.114:0/541596988 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d005b070 con 0x557f6af9d2f0 2024-05-27T07:41:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.897+0000 7fa7c67fc640 1 -- 172.21.15.114:0/541596988 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7d00314a0 con 0x557f6af9d2f0 2024-05-27T07:41:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.897+0000 7fa7c67fc640 1 -- 172.21.15.114:0/541596988 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d004b070 con 0x557f6af9d2f0 2024-05-27T07:41:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.897+0000 7fa7e9f81740 1 -- 172.21.15.114:0/541596988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 msgr2=0x557f6b37e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.897+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/541596988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b37e560 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0032eb0 tx=0x7fa7d0005ea0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.897+0000 7fa7e9f81740 1 -- 172.21.15.114:0/541596988 shutdown_connections 2024-05-27T07:41:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.897+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/541596988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afc61b0 0x557f6b36b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.897+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/541596988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6afa54d0 0x557f6b368170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.897+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/541596988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6af9d2f0 0x557f6b37e560 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.897+0000 7fa7e9f81740 1 -- 172.21.15.114:0/541596988 >> 172.21.15.114:0/541596988 conn(0x557f6afba300 msgr2=0x557f6b372d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.897+0000 7fa7e9f81740 1 -- 172.21.15.114:0/541596988 shutdown_connections 2024-05-27T07:41:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.897+0000 7fa7e9f81740 1 -- 172.21.15.114:0/541596988 wait complete. 2024-05-27T07:41:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.898+0000 7fa7e9f81740 1 -- start start 2024-05-27T07:41:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.898+0000 7fa7e9f81740 1 Processor -- start 2024-05-27T07:41:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.898+0000 7fa7e9f81740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6af9d2f0 0x557f6b37ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.898+0000 7fa7e9f81740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afa54d0 0x557f6b37b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.898+0000 7fa7e9f81740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6afc61b0 0x557f6b37b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.898+0000 7fa7e9f81740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc1affb0 con 0x557f6afc61b0 2024-05-27T07:41:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.898+0000 7fa7e9f81740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc1b1420 con 0x557f6af9d2f0 2024-05-27T07:41:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.898+0000 7fa7e9f81740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc1b0680 con 0x557f6afa54d0 2024-05-27T07:41:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.898+0000 7fa7e9f81740 10 client.0 _ll_register_callbacks cb 0x7fa7d0032d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.898+0000 7fa7e166e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6afc61b0 0x557f6b37b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.898+0000 7fa7e166e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6afc61b0 0x557f6b37b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36944/0 (socket says 172.21.15.114:36944) 2024-05-27T07:41:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.898+0000 7fa7e166e640 1 -- 172.21.15.114:0/2804486945 learned_addr learned my addr 172.21.15.114:0/2804486945 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7dbfff640 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afa54d0 0x557f6b37b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6af9d2f0 0x557f6b37ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7e166e640 1 -- 172.21.15.114:0/2804486945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afa54d0 msgr2=0x557f6b37b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7e166e640 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afa54d0 0x557f6b37b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7e166e640 1 -- 172.21.15.114:0/2804486945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6af9d2f0 msgr2=0x557f6b37ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7e166e640 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6af9d2f0 0x557f6b37ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7e166e640 1 -- 172.21.15.114:0/2804486945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f6b3ea120 con 0x557f6afc61b0 2024-05-27T07:41:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6af9d2f0 0x557f6b37ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7e166e640 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6afc61b0 0x557f6b37b7e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400a4d0 tx=0x7fa7d4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de62f3866b3bacfb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:41:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7bbfff640 1 -- 172.21.15.114:0/2804486945 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d401fb20 con 0x557f6afc61b0 2024-05-27T07:41:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7e9f81740 1 -- 172.21.15.114:0/2804486945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7d00314a0 con 0x557f6afc61b0 2024-05-27T07:41:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7bbfff640 1 -- 172.21.15.114:0/2804486945 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fa7d401fcc0 con 0x557f6afc61b0 2024-05-27T07:41:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7bbfff640 1 -- 172.21.15.114:0/2804486945 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa7d4020440 con 0x557f6afc61b0 2024-05-27T07:41:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7e9f81740 10 client.15561 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.899+0000 7fa7e9f81740 1 -- 172.21.15.114:0/2804486945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557f6b3eb3d0 con 0x557f6afc61b0 2024-05-27T07:41:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.900+0000 7fa7bbfff640 1 -- 172.21.15.114:0/2804486945 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7d00314a0 con 0x557f6afc61b0 2024-05-27T07:41:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.900+0000 7fa7bbfff640 1 -- 172.21.15.114:0/2804486945 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fa7d4029660 con 0x557f6afc61b0 2024-05-27T07:41:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.900+0000 7fa7e9f81740 10 client.15561 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.900+0000 7fa7e9f81740 10 client.15561 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.900+0000 7fa7e9f81740 20 client.15561 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.900+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f6af5a1c0 0x557f6b3edb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.900+0000 7fa7e9f81740 4 client.15561 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:41:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.900+0000 7fa7e9f81740 1 -- 172.21.15.114:0/2804486945 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557f6b2d5360 con 0x557f6af5a1c0 2024-05-27T07:41:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.900+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f6af5a1c0 0x557f6b3edb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.901+0000 7fa7bbfff640 10 client.15561 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:01.901+0000 7fa7e0e6d640 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f6af5a1c0 0x557f6b3edb90 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5da375883666399b server_cookie=be0259baa09b7126 in_seq=0 out_seq=0 2024-05-27T07:41:02.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7bbfff640 1 -- 172.21.15.114:0/2804486945 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x557f6b2d5360 con 0x557f6af5a1c0 2024-05-27T07:41:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7bbfff640 10 client.15561 handle_command_reply: tid=0 2024-05-27T07:41:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7e9f81740 1 client.15561 shutdown 2024-05-27T07:41:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7e9f81740 1 -- 172.21.15.114:0/2804486945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6afc61b0 msgr2=0x557f6b37b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6afc61b0 0x557f6b37b7e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400a4d0 tx=0x7fa7d4004ab0 comp rx=0 tx=0).stop 2024-05-27T07:41:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7e9f81740 1 -- 172.21.15.114:0/2804486945 shutdown_connections 2024-05-27T07:41:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f6af5a1c0 0x557f6b3edb90 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f6afc61b0 0x557f6b37b7e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f6afa54d0 0x557f6b37b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7bbfff640 0 client.15561 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7e9f81740 1 --2- 172.21.15.114:0/2804486945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f6af9d2f0 0x557f6b37ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7e9f81740 1 -- 172.21.15.114:0/2804486945 >> 172.21.15.114:0/2804486945 conn(0x557f6afba300 msgr2=0x557f6b371890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7e9f81740 1 -- 172.21.15.114:0/2804486945 shutdown_connections 2024-05-27T07:41:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7e9f81740 1 -- 172.21.15.114:0/2804486945 wait complete. 2024-05-27T07:41:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.050+0000 7fa7e9f81740 20 client.15561 trim_cache size 0 max 16384 2024-05-27T07:41:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:41:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:41:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:41:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.051+0000 7fa7e30d1640 1 -- 172.21.15.114:0/323342985 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7cc0b2a70 msgr2=0x7fa7cc0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.051+0000 7fa7e30d1640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7cc0b2a70 0x7fa7cc0b4f80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x557f6b364540 tx=0x7fa7d400a3d0 comp rx=0 tx=0).stop 2024-05-27T07:41:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.051+0000 7fa7e30d1640 1 -- 172.21.15.114:0/323342985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7dc10b880 msgr2=0x7fa7dc1a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.051+0000 7fa7e30d1640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7dc10b880 0x7fa7dc1a0a50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800d950 tx=0x7fa7c800de10 comp rx=0 tx=0).stop 2024-05-27T07:41:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.051+0000 7fa7e30d1640 1 -- 172.21.15.114:0/323342985 shutdown_connections 2024-05-27T07:41:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.051+0000 7fa7e30d1640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7cc0b2a70 0x7fa7cc0b4f80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.051+0000 7fa7e30d1640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7dc10e230 0x7fa7dc19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.051+0000 7fa7e30d1640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7dc10b880 0x7fa7dc1a0a50 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.051+0000 7fa7e30d1640 1 --2- 172.21.15.114:0/323342985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7dc104e20 0x7fa7dc1a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.051+0000 7fa7e30d1640 1 -- 172.21.15.114:0/323342985 >> 172.21.15.114:0/323342985 conn(0x7fa7dc078270 msgr2=0x7fa7dc101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.051+0000 7fa7e30d1640 1 -- 172.21.15.114:0/323342985 shutdown_connections 2024-05-27T07:41:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:02.051+0000 7fa7e30d1640 1 -- 172.21.15.114:0/323342985 wait complete. 2024-05-27T07:41:03.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:03 smithi114 ceph-mon[32038]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-05-27T07:41:03.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:03 smithi163 ceph-mon[36451]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-05-27T07:41:03.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:03 smithi186 ceph-mon[36647]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-05-27T07:41:05.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:05 smithi114 ceph-mon[32038]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-05-27T07:41:05.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:05 smithi163 ceph-mon[36451]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-05-27T07:41:05.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:05 smithi186 ceph-mon[36647]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-05-27T07:41:07.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:07 smithi163 ceph-mon[36451]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-05-27T07:41:07.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:07 smithi186 ceph-mon[36647]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-05-27T07:41:07.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:07 smithi114 ceph-mon[32038]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-05-27T07:41:09.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:09 smithi163 ceph-mon[36451]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-05-27T07:41:09.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:09 smithi186 ceph-mon[36647]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-05-27T07:41:09.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:09 smithi114 ceph-mon[32038]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-05-27T07:41:11.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:11 smithi163 ceph-mon[36451]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-05-27T07:41:11.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:11 smithi186 ceph-mon[36647]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-05-27T07:41:11.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:11 smithi114 ceph-mon[32038]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-05-27T07:41:13.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:13 smithi163 ceph-mon[36451]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-05-27T07:41:13.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:13 smithi186 ceph-mon[36647]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-05-27T07:41:13.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:13 smithi114 ceph-mon[32038]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-05-27T07:41:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.151+0000 7fa5bac05640 1 -- 172.21.15.114:0/3194664070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b4108b90 msgr2=0x7fa5b4102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.151+0000 7fa5bac05640 1 --2- 172.21.15.114:0/3194664070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b4108b90 0x7fa5b4102730 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009900 tx=0x7fa5a802a410 comp rx=0 tx=0).stop 2024-05-27T07:41:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.151+0000 7fa5bac05640 1 -- 172.21.15.114:0/3194664070 shutdown_connections 2024-05-27T07:41:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.151+0000 7fa5bac05640 1 --2- 172.21.15.114:0/3194664070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa5b41032e0 0x7fa5b410f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.151+0000 7fa5bac05640 1 --2- 172.21.15.114:0/3194664070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5b4109030 0x7fa5b4102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.151+0000 7fa5bac05640 1 --2- 172.21.15.114:0/3194664070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b4108b90 0x7fa5b4102730 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.151+0000 7fa5bac05640 1 -- 172.21.15.114:0/3194664070 >> 172.21.15.114:0/3194664070 conn(0x7fa5b40782d0 msgr2=0x7fa5b4078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.151+0000 7fa5bac05640 1 -- 172.21.15.114:0/3194664070 shutdown_connections 2024-05-27T07:41:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.151+0000 7fa5bac05640 1 -- 172.21.15.114:0/3194664070 wait complete. 2024-05-27T07:41:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.152+0000 7fa5bac05640 1 Processor -- start 2024-05-27T07:41:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.152+0000 7fa5bac05640 1 -- start start 2024-05-27T07:41:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.152+0000 7fa5bac05640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa5b41032e0 0x7fa5b41a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.152+0000 7fa5bac05640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5b4108b90 0x7fa5b41a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.152+0000 7fa5bac05640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b4109030 0x7fa5b419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.152+0000 7fa5bac05640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b4112d80 con 0x7fa5b4109030 2024-05-27T07:41:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.152+0000 7fa5bac05640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b4112c00 con 0x7fa5b41032e0 2024-05-27T07:41:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.152+0000 7fa5bac05640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b4112f00 con 0x7fa5b4108b90 2024-05-27T07:41:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.152+0000 7fa5b91a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b4109030 0x7fa5b419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.152+0000 7fa5b91a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b4109030 0x7fa5b419a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59782/0 (socket says 172.21.15.114:59782) 2024-05-27T07:41:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.152+0000 7fa5b91a2640 1 -- 172.21.15.114:0/3523183035 learned_addr learned my addr 172.21.15.114:0/3523183035 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.152+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5b4108b90 0x7fa5b41a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.152+0000 7fa5b89a1640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa5b41032e0 0x7fa5b41a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.153+0000 7fa5b91a2640 1 -- 172.21.15.114:0/3523183035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5b4108b90 msgr2=0x7fa5b41a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.153+0000 7fa5b91a2640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5b4108b90 0x7fa5b41a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.153+0000 7fa5b91a2640 1 -- 172.21.15.114:0/3523183035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa5b41032e0 msgr2=0x7fa5b41a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.153+0000 7fa5b91a2640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa5b41032e0 0x7fa5b41a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.153+0000 7fa5b91a2640 1 -- 172.21.15.114:0/3523183035 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b419ae40 con 0x7fa5b4109030 2024-05-27T07:41:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.153+0000 7fa5b91a2640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b4109030 0x7fa5b419a710 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00eaa0 tx=0x7fa5ac00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.153+0000 7fa5b1ffb640 1 -- 172.21.15.114:0/3523183035 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac00cdb0 con 0x7fa5b4109030 2024-05-27T07:41:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.153+0000 7fa5bac05640 1 -- 172.21.15.114:0/3523183035 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b419b0d0 con 0x7fa5b4109030 2024-05-27T07:41:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.153+0000 7fa5b1ffb640 1 -- 172.21.15.114:0/3523183035 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5ac005020 con 0x7fa5b4109030 2024-05-27T07:41:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.153+0000 7fa5b1ffb640 1 -- 172.21.15.114:0/3523183035 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac0116d0 con 0x7fa5b4109030 2024-05-27T07:41:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.154+0000 7fa5bac05640 1 -- 172.21.15.114:0/3523183035 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b40ff2a0 con 0x7fa5b4109030 2024-05-27T07:41:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.157+0000 7fa5c1b91740 1 Processor -- start 2024-05-27T07:41:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.157+0000 7fa5b1ffb640 1 -- 172.21.15.114:0/3523183035 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 149886+0+0 (secure 0 0 0) 0x7fa5ac0051d0 con 0x7fa5b4109030 2024-05-27T07:41:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.157+0000 7fa5c1b91740 1 -- start start 2024-05-27T07:41:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.158+0000 7fa5c1b91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630fe1b0 0x55826349f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.158+0000 7fa5c1b91740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630d52f0 0x55826349f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.158+0000 7fa5c1b91740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630dd4d0 0x55826349fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.158+0000 7fa5c1b91740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558263411c30 con 0x5582630d52f0 2024-05-27T07:41:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.158+0000 7fa5c1b91740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5582634017d0 con 0x5582630dd4d0 2024-05-27T07:41:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.158+0000 7fa5c1b91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558263405810 con 0x5582630fe1b0 2024-05-27T07:41:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.158+0000 7fa5b89a1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630d52f0 0x55826349f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.158+0000 7fa5b89a1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630d52f0 0x55826349f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59796/0 (socket says 172.21.15.114:59796) 2024-05-27T07:41:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.158+0000 7fa5b89a1640 1 -- 172.21.15.114:0/3370462660 learned_addr learned my addr 172.21.15.114:0/3370462660 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.158+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/3370462660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630dd4d0 0x55826349fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.158+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/3370462660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630fe1b0 0x55826349f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.158+0000 7fa5b89a1640 1 -- 172.21.15.114:0/3370462660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630fe1b0 msgr2=0x55826349f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.159+0000 7fa5b89a1640 1 --2- 172.21.15.114:0/3370462660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630fe1b0 0x55826349f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.159+0000 7fa5b89a1640 1 -- 172.21.15.114:0/3370462660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630dd4d0 msgr2=0x55826349fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.159+0000 7fa5b89a1640 1 --2- 172.21.15.114:0/3370462660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630dd4d0 0x55826349fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.159+0000 7fa5b89a1640 1 -- 172.21.15.114:0/3370462660 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558263116210 con 0x5582630d52f0 2024-05-27T07:41:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.159+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/3370462660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630fe1b0 0x55826349f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.159+0000 7fa5b1ffb640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa5a40b2a70 0x7fa5a40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.159+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa5a40b2a70 0x7fa5a40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.159+0000 7fa5b1ffb640 1 -- 172.21.15.114:0/3523183035 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fa5ac0e2db0 con 0x7fa5b4109030 2024-05-27T07:41:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.159+0000 7fa5b89a1640 1 --2- 172.21.15.114:0/3370462660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630d52f0 0x55826349f800 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009900 tx=0x7fa5a8032bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4137d86f6de6a694 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.159+0000 7fa59effd640 1 -- 172.21.15.114:0/3370462660 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8040070 con 0x5582630d52f0 2024-05-27T07:41:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.159+0000 7fa59effd640 1 -- 172.21.15.114:0/3370462660 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5a8004030 con 0x5582630d52f0 2024-05-27T07:41:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.159+0000 7fa59effd640 1 -- 172.21.15.114:0/3370462660 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8035040 con 0x5582630d52f0 2024-05-27T07:41:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.160+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa5a40b2a70 0x7fa5a40b4f80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x5582634a10c0 tx=0x7fa5a00096a0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.160+0000 7fa5c1b91740 1 -- 172.21.15.114:0/3370462660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630d52f0 msgr2=0x55826349f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.160+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/3370462660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630d52f0 0x55826349f800 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009900 tx=0x7fa5a8032bd0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.161+0000 7fa5c1b91740 1 -- 172.21.15.114:0/3370462660 shutdown_connections 2024-05-27T07:41:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.161+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/3370462660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630dd4d0 0x55826349fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.161+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/3370462660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630d52f0 0x55826349f800 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.161+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/3370462660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630fe1b0 0x55826349f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.161+0000 7fa5c1b91740 1 -- 172.21.15.114:0/3370462660 >> 172.21.15.114:0/3370462660 conn(0x5582630f2300 msgr2=0x55826349bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.161+0000 7fa5c1b91740 1 -- 172.21.15.114:0/3370462660 shutdown_connections 2024-05-27T07:41:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.161+0000 7fa5c1b91740 1 -- 172.21.15.114:0/3370462660 wait complete. 2024-05-27T07:41:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.161+0000 7fa5c1b91740 1 -- start start 2024-05-27T07:41:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.161+0000 7fa5c1b91740 1 Processor -- start 2024-05-27T07:41:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5c1b91740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630d52f0 0x5582634b3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5c1b91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630dd4d0 0x5582634b4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5c1b91740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630fe1b0 0x5582634b2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5c1b91740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5582630e7790 con 0x5582630fe1b0 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5c1b91740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5582633bd890 con 0x5582630d52f0 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5c1b91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5582634b4cb0 con 0x5582630dd4d0 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5c1b91740 10 client.0 _ll_register_callbacks cb 0x558263115a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5b3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630dd4d0 0x5582634b4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5b91a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630fe1b0 0x5582634b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5b91a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630fe1b0 0x5582634b2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59802/0 (socket says 172.21.15.114:59802) 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5b91a2640 1 -- 172.21.15.114:0/1860561530 learned_addr learned my addr 172.21.15.114:0/1860561530 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5b3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630dd4d0 0x5582634b4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40236/0 (socket says 172.21.15.114:40236) 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5b89a1640 1 --2- 172.21.15.114:0/1860561530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630d52f0 0x5582634b3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5b3fff640 1 -- 172.21.15.114:0/1860561530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630d52f0 msgr2=0x5582634b3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/1860561530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630d52f0 0x5582634b3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5b3fff640 1 -- 172.21.15.114:0/1860561530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630fe1b0 msgr2=0x5582634b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.162+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/1860561530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630fe1b0 0x5582634b2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.163+0000 7fa5b3fff640 1 -- 172.21.15.114:0/1860561530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582631e2620 con 0x5582630dd4d0 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.163+0000 7fa5b91a2640 1 --2- 172.21.15.114:0/1860561530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630fe1b0 0x5582634b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.163+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/1860561530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630dd4d0 0x5582634b4210 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009710 tx=0x7fa5a0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5720b77b6d5a9c2b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.163+0000 7fa593fff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.163+0000 7fa593fff640 1 -- 172.21.15.114:0/1860561530 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0005020 con 0x5582630dd4d0 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.163+0000 7fa5c1b91740 1 -- 172.21.15.114:0/1860561530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5a8004030 con 0x5582630dd4d0 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.163+0000 7fa593fff640 1 -- 172.21.15.114:0/1860561530 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fa5a00051c0 con 0x5582630dd4d0 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.163+0000 7fa593fff640 1 -- 172.21.15.114:0/1860561530 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa5a0033050 con 0x5582630dd4d0 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.163+0000 7fa5c1b91740 10 client.25192 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.163+0000 7fa5c1b91740 1 -- 172.21.15.114:0/1860561530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5582634c10c0 con 0x5582630dd4d0 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.164+0000 7fa593fff640 1 -- 172.21.15.114:0/1860561530 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5a8004030 con 0x5582630dd4d0 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.164+0000 7fa593fff640 1 -- 172.21.15.114:0/1860561530 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fa5a0010700 con 0x5582630dd4d0 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.164+0000 7fa5c1b91740 10 client.25192 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.164+0000 7fa5c1b91740 10 client.25192 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.164+0000 7fa5c1b91740 20 client.25192 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:15.166 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.164+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/1860561530 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5582630bcd80 0x5582634c4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.166 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.164+0000 7fa5c1b91740 4 client.25192 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:15.166 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.164+0000 7fa5c1b91740 1 -- 172.21.15.114:0/1860561530 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558263413b80 con 0x5582630bcd80 2024-05-27T07:41:15.166 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.164+0000 7fa5b89a1640 1 --2- 172.21.15.114:0/1860561530 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5582630bcd80 0x5582634c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.165+0000 7fa5b89a1640 1 --2- 172.21.15.114:0/1860561530 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5582630bcd80 0x5582634c4590 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bedf07b24d6b6213 server_cookie=ba2328f54c224396 in_seq=0 out_seq=0 2024-05-27T07:41:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.165+0000 7fa593fff640 10 client.25192 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa593fff640 1 -- 172.21.15.114:0/1860561530 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558263413b80 con 0x5582630bcd80 2024-05-27T07:41:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa593fff640 10 client.25192 handle_command_reply: tid=0 2024-05-27T07:41:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa5c1b91740 1 client.25192 shutdown 2024-05-27T07:41:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa5c1b91740 1 -- 172.21.15.114:0/1860561530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630dd4d0 msgr2=0x5582634b4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/1860561530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630dd4d0 0x5582634b4210 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009710 tx=0x7fa5a0004730 comp rx=0 tx=0).stop 2024-05-27T07:41:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa5c1b91740 1 -- 172.21.15.114:0/1860561530 shutdown_connections 2024-05-27T07:41:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/1860561530 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5582630bcd80 0x5582634c4590 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/1860561530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630fe1b0 0x5582634b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa593fff640 0 client.25192 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/1860561530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630dd4d0 0x5582634b4210 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/1860561530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630d52f0 0x5582634b3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa5c1b91740 1 -- 172.21.15.114:0/1860561530 >> 172.21.15.114:0/1860561530 conn(0x5582630f2300 msgr2=0x55826349f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa5c1b91740 1 -- 172.21.15.114:0/1860561530 shutdown_connections 2024-05-27T07:41:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.166+0000 7fa5c1b91740 1 -- 172.21.15.114:0/1860561530 wait complete. 2024-05-27T07:41:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.167+0000 7fa5c1b91740 20 client.25192 trim_cache size 0 max 16384 2024-05-27T07:41:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.195+0000 7fa5c1b91740 1 Processor -- start 2024-05-27T07:41:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.195+0000 7fa5c1b91740 1 -- start start 2024-05-27T07:41:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5c1b91740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630f2300 0x55826349f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5c1b91740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630bcd80 0x5582634aba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5c1b91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630d52f0 0x5582634abf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5c1b91740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b41ad190 con 0x5582630bcd80 2024-05-27T07:41:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5c1b91740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b41afa10 con 0x5582630f2300 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5c1b91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b41ad800 con 0x5582630d52f0 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5b3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630bcd80 0x5582634aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5b3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630bcd80 0x5582634aba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59818/0 (socket says 172.21.15.114:59818) 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5b3fff640 1 -- 172.21.15.114:0/107581898 learned_addr learned my addr 172.21.15.114:0/107581898 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5b89a1640 1 --2- 172.21.15.114:0/107581898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630f2300 0x55826349f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5b91a2640 1 --2- 172.21.15.114:0/107581898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630d52f0 0x5582634abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5b3fff640 1 -- 172.21.15.114:0/107581898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630d52f0 msgr2=0x5582634abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/107581898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630d52f0 0x5582634abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5b3fff640 1 -- 172.21.15.114:0/107581898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630f2300 msgr2=0x55826349f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/107581898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630f2300 0x55826349f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5b3fff640 1 -- 172.21.15.114:0/107581898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582634a0350 con 0x5582630bcd80 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5b91a2640 1 --2- 172.21.15.114:0/107581898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630d52f0 0x5582634abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.196+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/107581898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630bcd80 0x5582634aba10 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0000c00 tx=0x7fa5a0009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0b3629f7f7e3ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.197+0000 7fa59e7fc640 1 -- 172.21.15.114:0/107581898 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0010d40 con 0x5582630bcd80 2024-05-27T07:41:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.197+0000 7fa59e7fc640 1 -- 172.21.15.114:0/107581898 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5a0026070 con 0x5582630bcd80 2024-05-27T07:41:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.197+0000 7fa59e7fc640 1 -- 172.21.15.114:0/107581898 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0004a00 con 0x5582630bcd80 2024-05-27T07:41:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.197+0000 7fa5c1b91740 1 -- 172.21.15.114:0/107581898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630bcd80 msgr2=0x5582634aba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.197+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/107581898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630bcd80 0x5582634aba10 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0000c00 tx=0x7fa5a0009f00 comp rx=0 tx=0).stop 2024-05-27T07:41:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.198+0000 7fa5c1b91740 1 -- 172.21.15.114:0/107581898 shutdown_connections 2024-05-27T07:41:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.198+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/107581898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630d52f0 0x5582634abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.198+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/107581898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630bcd80 0x5582634aba10 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.198+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/107581898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630f2300 0x55826349f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.198+0000 7fa5c1b91740 1 -- 172.21.15.114:0/107581898 >> 172.21.15.114:0/107581898 conn(0x5582634c4900 msgr2=0x55826349f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.198+0000 7fa5c1b91740 1 -- 172.21.15.114:0/107581898 shutdown_connections 2024-05-27T07:41:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.198+0000 7fa5c1b91740 1 -- 172.21.15.114:0/107581898 wait complete. 2024-05-27T07:41:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.198+0000 7fa5c1b91740 1 -- start start 2024-05-27T07:41:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.198+0000 7fa5c1b91740 1 Processor -- start 2024-05-27T07:41:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.199+0000 7fa5c1b91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630bcd80 0x5582634b9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.199+0000 7fa5c1b91740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630d52f0 0x5582634ba4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.199+0000 7fa5c1b91740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630f2300 0x5582634b5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.199+0000 7fa5c1b91740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b41aded0 con 0x5582630f2300 2024-05-27T07:41:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.199+0000 7fa5c1b91740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b41af340 con 0x5582630d52f0 2024-05-27T07:41:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.199+0000 7fa5c1b91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b41ae5a0 con 0x5582630bcd80 2024-05-27T07:41:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.199+0000 7fa5b91a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630f2300 0x5582634b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.199+0000 7fa5c1b91740 10 client.0 _ll_register_callbacks cb 0x7fa5a0009c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.199+0000 7fa5b91a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630f2300 0x5582634b5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59834/0 (socket says 172.21.15.114:59834) 2024-05-27T07:41:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.199+0000 7fa5b91a2640 1 -- 172.21.15.114:0/2554286971 learned_addr learned my addr 172.21.15.114:0/2554286971 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.199+0000 7fa5b89a1640 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630bcd80 0x5582634b9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.199+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630d52f0 0x5582634ba4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa5b89a1640 1 -- 172.21.15.114:0/2554286971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630d52f0 msgr2=0x5582634ba4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa5b89a1640 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630d52f0 0x5582634ba4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa5b89a1640 1 -- 172.21.15.114:0/2554286971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630f2300 msgr2=0x5582634b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa5b89a1640 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630f2300 0x5582634b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa5b89a1640 1 -- 172.21.15.114:0/2554286971 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582634b5ba0 con 0x5582630bcd80 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa5b91a2640 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630f2300 0x5582634b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa5b89a1640 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630bcd80 0x5582634b9fa0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8006010 tx=0x7fa5a8005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d6b08e3a67e5245 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa593fff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa593fff640 1 -- 172.21.15.114:0/2554286971 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8035040 con 0x5582630bcd80 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa593fff640 1 -- 172.21.15.114:0/2554286971 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fa5a80316b0 con 0x5582630bcd80 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa5c1b91740 1 -- 172.21.15.114:0/2554286971 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5a0026070 con 0x5582630bcd80 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa593fff640 1 -- 172.21.15.114:0/2554286971 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa5a8062050 con 0x5582630bcd80 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa5c1b91740 10 client.25198 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.200+0000 7fa5c1b91740 1 -- 172.21.15.114:0/2554286971 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55826352c5b0 con 0x5582630bcd80 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.201+0000 7fa593fff640 1 -- 172.21.15.114:0/2554286971 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5a0026070 con 0x5582630bcd80 2024-05-27T07:41:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.201+0000 7fa593fff640 1 -- 172.21.15.114:0/2554286971 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fa5a80035f0 con 0x5582630bcd80 2024-05-27T07:41:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.201+0000 7fa5c1b91740 10 client.25198 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.201+0000 7fa5c1b91740 10 client.25198 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.201+0000 7fa5c1b91740 20 client.25198 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.201+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5582630dd4d0 0x55826352e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.201+0000 7fa5c1b91740 4 client.25198 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:41:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.201+0000 7fa5c1b91740 1 -- 172.21.15.114:0/2554286971 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55826340d360 con 0x5582630dd4d0 2024-05-27T07:41:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.201+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5582630dd4d0 0x55826352e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.202+0000 7fa593fff640 10 client.25198 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.202+0000 7fa5b3fff640 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5582630dd4d0 0x55826352e950 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ac91dfaa87a78af server_cookie=8e5555d44f98ea78 in_seq=0 out_seq=0 2024-05-27T07:41:15.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.207+0000 7fa593fff640 1 -- 172.21.15.114:0/2554286971 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55826340d360 con 0x5582630dd4d0 2024-05-27T07:41:15.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.207+0000 7fa593fff640 10 client.25198 handle_command_reply: tid=0 2024-05-27T07:41:15.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.208+0000 7fa5c1b91740 1 client.25198 shutdown 2024-05-27T07:41:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.208+0000 7fa5c1b91740 1 -- 172.21.15.114:0/2554286971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630bcd80 msgr2=0x5582634b9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.208+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630bcd80 0x5582634b9fa0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8006010 tx=0x7fa5a8005a20 comp rx=0 tx=0).stop 2024-05-27T07:41:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.208+0000 7fa5c1b91740 1 -- 172.21.15.114:0/2554286971 shutdown_connections 2024-05-27T07:41:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.208+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5582630dd4d0 0x55826352e950 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.208+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582630f2300 0x5582634b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.208+0000 7fa593fff640 0 client.25198 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.208+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582630d52f0 0x5582634ba4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.208+0000 7fa5c1b91740 1 --2- 172.21.15.114:0/2554286971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582630bcd80 0x5582634b9fa0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.208+0000 7fa5c1b91740 1 -- 172.21.15.114:0/2554286971 >> 172.21.15.114:0/2554286971 conn(0x5582634c4900 msgr2=0x5582634a10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.208+0000 7fa5c1b91740 1 -- 172.21.15.114:0/2554286971 shutdown_connections 2024-05-27T07:41:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.208+0000 7fa5c1b91740 1 -- 172.21.15.114:0/2554286971 wait complete. 2024-05-27T07:41:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.208+0000 7fa5c1b91740 20 client.25198 trim_cache size 0 max 16384 2024-05-27T07:41:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:41:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:41:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:41:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.209+0000 7fa5bac05640 1 -- 172.21.15.114:0/3523183035 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa5a40b2a70 msgr2=0x7fa5a40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.209+0000 7fa5bac05640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa5a40b2a70 0x7fa5a40b4f80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x5582634a10c0 tx=0x7fa5a00096a0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.209+0000 7fa5bac05640 1 -- 172.21.15.114:0/3523183035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b4109030 msgr2=0x7fa5b419a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.209+0000 7fa5bac05640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b4109030 0x7fa5b419a710 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00eaa0 tx=0x7fa5ac00ef60 comp rx=0 tx=0).stop 2024-05-27T07:41:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.209+0000 7fa5bac05640 1 -- 172.21.15.114:0/3523183035 shutdown_connections 2024-05-27T07:41:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.209+0000 7fa5bac05640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa5a40b2a70 0x7fa5a40b4f80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.209+0000 7fa5bac05640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b4109030 0x7fa5b419a710 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.209+0000 7fa5bac05640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5b4108b90 0x7fa5b41a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.209+0000 7fa5bac05640 1 --2- 172.21.15.114:0/3523183035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa5b41032e0 0x7fa5b41a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.209+0000 7fa5bac05640 1 -- 172.21.15.114:0/3523183035 >> 172.21.15.114:0/3523183035 conn(0x7fa5b40782d0 msgr2=0x7fa5b406a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.209+0000 7fa5bac05640 1 -- 172.21.15.114:0/3523183035 shutdown_connections 2024-05-27T07:41:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:15.209+0000 7fa5bac05640 1 -- 172.21.15.114:0/3523183035 wait complete. 2024-05-27T07:41:15.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:15 smithi114 ceph-mon[32038]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-05-27T07:41:15.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:15 smithi163 ceph-mon[36451]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-05-27T07:41:15.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:15 smithi186 ceph-mon[36647]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-05-27T07:41:17.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:17 smithi163 ceph-mon[36451]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-05-27T07:41:17.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:17 smithi163 ceph-mon[36451]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-05-27T07:41:17.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:17 smithi186 ceph-mon[36647]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-05-27T07:41:17.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:17 smithi186 ceph-mon[36647]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-05-27T07:41:17.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:17 smithi114 ceph-mon[32038]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-05-27T07:41:17.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:17 smithi114 ceph-mon[32038]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-05-27T07:41:19.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:19 smithi163 ceph-mon[36451]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:19.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:19 smithi186 ceph-mon[36647]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:19.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:19 smithi114 ceph-mon[32038]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:21.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:21 smithi163 ceph-mon[36451]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-27T07:41:21.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:21 smithi186 ceph-mon[36647]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-27T07:41:21.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:21 smithi114 ceph-mon[32038]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-27T07:41:23.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:23 smithi163 ceph-mon[36451]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-05-27T07:41:23.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:23 smithi186 ceph-mon[36647]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-05-27T07:41:23.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:23 smithi114 ceph-mon[32038]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-05-27T07:41:25.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:25 smithi163 ceph-mon[36451]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-05-27T07:41:25.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:25 smithi186 ceph-mon[36647]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-05-27T07:41:25.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:25 smithi114 ceph-mon[32038]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-05-27T07:41:27.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:41:27.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:27 smithi114 ceph-mon[32038]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-27T07:41:27.447 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:27.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:27 smithi163 ceph-mon[36451]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-27T07:41:27.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:27 smithi186 ceph-mon[36647]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-27T07:41:28.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.351+0000 7f48c80bf640 1 -- 172.21.15.114:0/1502010538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48c010bea0 msgr2=0x7f48c0105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:28.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.351+0000 7f48c80bf640 1 --2- 172.21.15.114:0/1502010538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48c010bea0 0x7f48c0105a40 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009900 tx=0x7f48bc02a410 comp rx=0 tx=0).stop 2024-05-27T07:41:28.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.351+0000 7f48c80bf640 1 -- 172.21.15.114:0/1502010538 shutdown_connections 2024-05-27T07:41:28.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.351+0000 7f48c80bf640 1 --2- 172.21.15.114:0/1502010538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48c01065f0 0x7f48c01129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.351+0000 7f48c80bf640 1 --2- 172.21.15.114:0/1502010538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c010c340 0x7f48c0105f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.351+0000 7f48c80bf640 1 --2- 172.21.15.114:0/1502010538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48c010bea0 0x7f48c0105a40 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.351+0000 7f48c80bf640 1 -- 172.21.15.114:0/1502010538 >> 172.21.15.114:0/1502010538 conn(0x7f48c0101520 msgr2=0x7f48c0103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.351+0000 7f48c80bf640 1 -- 172.21.15.114:0/1502010538 shutdown_connections 2024-05-27T07:41:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.352+0000 7f48c80bf640 1 -- 172.21.15.114:0/1502010538 wait complete. 2024-05-27T07:41:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.352+0000 7f48c80bf640 1 Processor -- start 2024-05-27T07:41:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.353+0000 7f48c80bf640 1 -- start start 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.353+0000 7f48c80bf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48c01065f0 0x7f48c01a2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.353+0000 7f48c80bf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c010bea0 0x7f48c01a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.353+0000 7f48c80bf640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48c010c340 0x7f48c019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.353+0000 7f48c80bf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f48c01160a0 con 0x7f48c01065f0 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.353+0000 7f48c80bf640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f48c0115f20 con 0x7f48c010c340 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.353+0000 7f48c80bf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48c0116220 con 0x7f48c010bea0 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.353+0000 7f48c5e5b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48c01065f0 0x7f48c01a2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.353+0000 7f48c5e5b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48c01065f0 0x7f48c01a2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39084/0 (socket says 172.21.15.114:39084) 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.353+0000 7f48c5e5b640 1 -- 172.21.15.114:0/3758568567 learned_addr learned my addr 172.21.15.114:0/3758568567 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.353+0000 7f48c565a640 1 --2- 172.21.15.114:0/3758568567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c010bea0 0x7f48c01a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.354+0000 7f48c5e5b640 1 -- 172.21.15.114:0/3758568567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c010bea0 msgr2=0x7f48c01a2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.354+0000 7f48c5e5b640 1 --2- 172.21.15.114:0/3758568567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c010bea0 0x7f48c01a2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.354+0000 7f48c5e5b640 1 -- 172.21.15.114:0/3758568567 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48c010c340 msgr2=0x7f48c019c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.354+0000 7f48c5e5b640 1 --2- 172.21.15.114:0/3758568567 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48c010c340 0x7f48c019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.354+0000 7f48c5e5b640 1 -- 172.21.15.114:0/3758568567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c019cf50 con 0x7f48c01065f0 2024-05-27T07:41:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.354+0000 7f48c5e5b640 1 --2- 172.21.15.114:0/3758568567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48c01065f0 0x7f48c01a2750 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f48bc006010 tx=0x7f48bc004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.354+0000 7f48b6ffd640 1 -- 172.21.15.114:0/3758568567 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc03f030 con 0x7f48c01065f0 2024-05-27T07:41:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.354+0000 7f48c80bf640 1 -- 172.21.15.114:0/3758568567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48c019d1e0 con 0x7f48c01065f0 2024-05-27T07:41:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.354+0000 7f48b6ffd640 1 -- 172.21.15.114:0/3758568567 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48bc003aa0 con 0x7f48c01065f0 2024-05-27T07:41:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.354+0000 7f48b6ffd640 1 -- 172.21.15.114:0/3758568567 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc035070 con 0x7f48c01065f0 2024-05-27T07:41:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.354+0000 7f48c80bf640 1 -- 172.21.15.114:0/3758568567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48c019d720 con 0x7f48c01065f0 2024-05-27T07:41:28.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.357+0000 7f48b6ffd640 1 -- 172.21.15.114:0/3758568567 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f48bc042030 con 0x7f48c01065f0 2024-05-27T07:41:28.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.357+0000 7f48cef70740 1 Processor -- start 2024-05-27T07:41:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.357+0000 7f48cef70740 1 -- start start 2024-05-27T07:41:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48cef70740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262411b0 0x55e3265e23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48cef70740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262182f0 0x55e3265e2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48cef70740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262204d0 0x55e3265e2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48cef70740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e326554c30 con 0x55e3262411b0 2024-05-27T07:41:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48cef70740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e3265447d0 con 0x55e3262204d0 2024-05-27T07:41:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48cef70740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e326548810 con 0x55e3262182f0 2024-05-27T07:41:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48c665c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262411b0 0x55e3265e23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48c665c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262411b0 0x55e3265e23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39086/0 (socket says 172.21.15.114:39086) 2024-05-27T07:41:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48c665c640 1 -- 172.21.15.114:0/2506925019 learned_addr learned my addr 172.21.15.114:0/2506925019 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48b6ffd640 1 --2- 172.21.15.114:0/3758568567 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f48ac0b29d0 0x7f48ac0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48c665c640 1 --2- 172.21.15.114:0/3758568567 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f48ac0b29d0 0x7f48ac0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48b6ffd640 1 -- 172.21.15.114:0/3758568567 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f48bc102f00 con 0x7f48c01065f0 2024-05-27T07:41:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48c565a640 1 --2- 172.21.15.114:0/2506925019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262204d0 0x55e3265e2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48c5e5b640 1 --2- 172.21.15.114:0/2506925019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262182f0 0x55e3265e2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48c665c640 1 -- 172.21.15.114:0/2506925019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262182f0 msgr2=0x55e3265e2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48c665c640 1 --2- 172.21.15.114:0/2506925019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262182f0 0x55e3265e2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48c665c640 1 -- 172.21.15.114:0/2506925019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262204d0 msgr2=0x55e3265e2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.358+0000 7f48c665c640 1 --2- 172.21.15.114:0/2506925019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262204d0 0x55e3265e2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.359+0000 7f48c665c640 1 -- 172.21.15.114:0/2506925019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e326259210 con 0x55e3262411b0 2024-05-27T07:41:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.359+0000 7f48c565a640 1 --2- 172.21.15.114:0/2506925019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262204d0 0x55e3265e2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.359+0000 7f48c665c640 1 --2- 172.21.15.114:0/2506925019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262411b0 0x55e3265e23d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f48b80138d0 tx=0x7f48b8013d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700b8884ca33409f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.359+0000 7f48abfff640 1 -- 172.21.15.114:0/2506925019 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b8003970 con 0x55e3262411b0 2024-05-27T07:41:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.359+0000 7f48c665c640 1 --2- 172.21.15.114:0/3758568567 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f48ac0b29d0 0x7f48ac0b4ee0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f48b800a000 tx=0x7f48b8019000 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.359+0000 7f48abfff640 1 -- 172.21.15.114:0/2506925019 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b800c7d0 con 0x55e3262411b0 2024-05-27T07:41:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.359+0000 7f48abfff640 1 -- 172.21.15.114:0/2506925019 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b8004290 con 0x55e3262411b0 2024-05-27T07:41:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.360+0000 7f48cef70740 1 -- 172.21.15.114:0/2506925019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262411b0 msgr2=0x55e3265e23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.360+0000 7f48cef70740 1 --2- 172.21.15.114:0/2506925019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262411b0 0x55e3265e23d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f48b80138d0 tx=0x7f48b8013d90 comp rx=0 tx=0).stop 2024-05-27T07:41:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.360+0000 7f48cef70740 1 -- 172.21.15.114:0/2506925019 shutdown_connections 2024-05-27T07:41:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.360+0000 7f48cef70740 1 --2- 172.21.15.114:0/2506925019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262204d0 0x55e3265e2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.360+0000 7f48cef70740 1 --2- 172.21.15.114:0/2506925019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262182f0 0x55e3265e2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.360+0000 7f48cef70740 1 --2- 172.21.15.114:0/2506925019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262411b0 0x55e3265e23d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.360+0000 7f48cef70740 1 -- 172.21.15.114:0/2506925019 >> 172.21.15.114:0/2506925019 conn(0x55e326235300 msgr2=0x55e3265deb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.360+0000 7f48cef70740 1 -- 172.21.15.114:0/2506925019 shutdown_connections 2024-05-27T07:41:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.360+0000 7f48cef70740 1 -- 172.21.15.114:0/2506925019 wait complete. 2024-05-27T07:41:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.361+0000 7f48cef70740 1 -- start start 2024-05-27T07:41:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.361+0000 7f48cef70740 1 Processor -- start 2024-05-27T07:41:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.361+0000 7f48cef70740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262182f0 0x55e3265ee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.361+0000 7f48cef70740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 0x55e3265eea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.361+0000 7f48cef70740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262411b0 0x55e3265ec950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.361+0000 7f48cef70740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e32622a790 con 0x55e3262204d0 2024-05-27T07:41:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.361+0000 7f48cef70740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e326500890 con 0x55e3262411b0 2024-05-27T07:41:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.361+0000 7f48cef70740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e3265ef440 con 0x55e3262182f0 2024-05-27T07:41:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.361+0000 7f48c565a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 0x55e3265eea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.361+0000 7f48c565a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 0x55e3265eea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39088/0 (socket says 172.21.15.114:39088) 2024-05-27T07:41:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.361+0000 7f48c565a640 1 -- 172.21.15.114:0/2553517304 learned_addr learned my addr 172.21.15.114:0/2553517304 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.361+0000 7f48c665c640 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262411b0 0x55e3265ec950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48c5e5b640 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262182f0 0x55e3265ee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.361+0000 7f48cef70740 10 client.0 _ll_register_callbacks cb 0x55e326258a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48c565a640 1 -- 172.21.15.114:0/2553517304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262182f0 msgr2=0x55e3265ee4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48c565a640 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262182f0 0x55e3265ee4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48c565a640 1 -- 172.21.15.114:0/2553517304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262411b0 msgr2=0x55e3265ec950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48c565a640 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262411b0 0x55e3265ec950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48c565a640 1 -- 172.21.15.114:0/2553517304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e326325620 con 0x55e3262204d0 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48c5e5b640 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262182f0 0x55e3265ee4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48c665c640 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262411b0 0x55e3265ec950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48cef70740 1 -- 172.21.15.114:0/2553517304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48b800c7d0 con 0x55e3262204d0 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48c565a640 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 0x55e3265eea10 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009f10 tx=0x7f48b00127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ba1edead996581 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48a97fa640 10 client.15591 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48a97fa640 1 -- 172.21.15.114:0/2553517304 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0004220 con 0x55e3262204d0 2024-05-27T07:41:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48a97fa640 1 -- 172.21.15.114:0/2553517304 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f48b00043c0 con 0x55e3262204d0 2024-05-27T07:41:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48a97fa640 1 -- 172.21.15.114:0/2553517304 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f48b00106d0 con 0x55e3262204d0 2024-05-27T07:41:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48cef70740 10 client.15591 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.362+0000 7f48cef70740 1 -- 172.21.15.114:0/2553517304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55e3265fb670 con 0x55e3262204d0 2024-05-27T07:41:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.363+0000 7f48a97fa640 1 -- 172.21.15.114:0/2553517304 <== mon.0 v2:172.21.15.114:3300/0 4 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f48b0019030 con 0x55e3262204d0 2024-05-27T07:41:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.363+0000 7f48cef70740 10 client.15591 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.363+0000 7f48cef70740 10 client.15591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.363+0000 7f48cef70740 20 client.15591 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.363+0000 7f48cef70740 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e3261ffd80 0x55e3265feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.363+0000 7f48cef70740 4 client.15591 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.363+0000 7f48cef70740 1 -- 172.21.15.114:0/2553517304 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e326556b80 con 0x55e3261ffd80 2024-05-27T07:41:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.363+0000 7f48c5e5b640 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e3261ffd80 0x55e3265feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.364+0000 7f48c5e5b640 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e3261ffd80 0x55e3265feb10 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=432858318ecc1e4b server_cookie=c4f722a9662fda60 in_seq=0 out_seq=0 2024-05-27T07:41:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.364+0000 7f48a97fa640 10 client.15591 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.364+0000 7f48a97fa640 1 -- 172.21.15.114:0/2553517304 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e326556b80 con 0x55e3261ffd80 2024-05-27T07:41:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.364+0000 7f48a97fa640 10 client.15591 handle_command_reply: tid=0 2024-05-27T07:41:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.364+0000 7f48cef70740 1 client.15591 shutdown 2024-05-27T07:41:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.364+0000 7f48cef70740 1 -- 172.21.15.114:0/2553517304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 msgr2=0x55e3265eea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.364+0000 7f48cef70740 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 0x55e3265eea10 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009f10 tx=0x7f48b00127c0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.365+0000 7f48cef70740 1 -- 172.21.15.114:0/2553517304 shutdown_connections 2024-05-27T07:41:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.365+0000 7f48cef70740 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e3261ffd80 0x55e3265feb10 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.365+0000 7f48cef70740 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262411b0 0x55e3265ec950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.365+0000 7f48cef70740 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 0x55e3265eea10 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.365+0000 7f48cef70740 1 --2- 172.21.15.114:0/2553517304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262182f0 0x55e3265ee4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.365+0000 7f48cef70740 1 -- 172.21.15.114:0/2553517304 >> 172.21.15.114:0/2553517304 conn(0x55e326235300 msgr2=0x55e3265e2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.365+0000 7f48a97fa640 0 client.15591 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.365+0000 7f48cef70740 1 -- 172.21.15.114:0/2553517304 shutdown_connections 2024-05-27T07:41:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.365+0000 7f48cef70740 1 -- 172.21.15.114:0/2553517304 wait complete. 2024-05-27T07:41:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.365+0000 7f48cef70740 20 client.15591 trim_cache size 0 max 16384 2024-05-27T07:41:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48cef70740 1 Processor -- start 2024-05-27T07:41:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48cef70740 1 -- start start 2024-05-27T07:41:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48cef70740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262182f0 0x55e3265e67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48cef70740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 0x55e3265de540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48cef70740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262411b0 0x55e3265dea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48cef70740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f48c01b1430 con 0x55e3262204d0 2024-05-27T07:41:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48cef70740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f48c01b3cb0 con 0x55e3262182f0 2024-05-27T07:41:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48cef70740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48c01b1aa0 con 0x55e3262411b0 2024-05-27T07:41:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48c565a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 0x55e3265de540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48c565a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 0x55e3265de540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39100/0 (socket says 172.21.15.114:39100) 2024-05-27T07:41:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48c565a640 1 -- 172.21.15.114:0/1880789106 learned_addr learned my addr 172.21.15.114:0/1880789106 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48c5e5b640 1 --2- 172.21.15.114:0/1880789106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262182f0 0x55e3265e67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48c565a640 1 -- 172.21.15.114:0/1880789106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262411b0 msgr2=0x55e3265dea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48c565a640 1 --2- 172.21.15.114:0/1880789106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262411b0 0x55e3265dea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48c565a640 1 -- 172.21.15.114:0/1880789106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262182f0 msgr2=0x55e3265e67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48c565a640 1 --2- 172.21.15.114:0/1880789106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262182f0 0x55e3265e67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48c565a640 1 -- 172.21.15.114:0/1880789106 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3265da340 con 0x55e3262204d0 2024-05-27T07:41:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.380+0000 7f48c5e5b640 1 --2- 172.21.15.114:0/1880789106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262182f0 0x55e3265e67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.381+0000 7f48c565a640 1 --2- 172.21.15.114:0/1880789106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 0x55e3265de540 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f48b0012800 tx=0x7f48b0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8918908f33495d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.381+0000 7f48ab7fe640 1 -- 172.21.15.114:0/1880789106 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0005030 con 0x55e3262204d0 2024-05-27T07:41:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.381+0000 7f48ab7fe640 1 -- 172.21.15.114:0/1880789106 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b800c7d0 con 0x55e3262204d0 2024-05-27T07:41:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.381+0000 7f48ab7fe640 1 -- 172.21.15.114:0/1880789106 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0024690 con 0x55e3262204d0 2024-05-27T07:41:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.382+0000 7f48cef70740 1 -- 172.21.15.114:0/1880789106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 msgr2=0x55e3265de540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.382+0000 7f48cef70740 1 --2- 172.21.15.114:0/1880789106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 0x55e3265de540 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f48b0012800 tx=0x7f48b0015000 comp rx=0 tx=0).stop 2024-05-27T07:41:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.382+0000 7f48cef70740 1 -- 172.21.15.114:0/1880789106 shutdown_connections 2024-05-27T07:41:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.382+0000 7f48cef70740 1 --2- 172.21.15.114:0/1880789106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262411b0 0x55e3265dea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.382+0000 7f48cef70740 1 --2- 172.21.15.114:0/1880789106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262204d0 0x55e3265de540 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.382+0000 7f48cef70740 1 --2- 172.21.15.114:0/1880789106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262182f0 0x55e3265e67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.382+0000 7f48cef70740 1 -- 172.21.15.114:0/1880789106 >> 172.21.15.114:0/1880789106 conn(0x55e326235300 msgr2=0x55e3265e60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.382+0000 7f48cef70740 1 -- 172.21.15.114:0/1880789106 shutdown_connections 2024-05-27T07:41:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.382+0000 7f48cef70740 1 -- 172.21.15.114:0/1880789106 wait complete. 2024-05-27T07:41:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.382+0000 7f48cef70740 1 -- start start 2024-05-27T07:41:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.382+0000 7f48cef70740 1 Processor -- start 2024-05-27T07:41:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48cef70740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262182f0 0x55e3265f4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48cef70740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262204d0 0x55e3265f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48cef70740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262411b0 0x55e3265f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48cef70740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f48c01b2170 con 0x55e3262182f0 2024-05-27T07:41:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48cef70740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f48c01b35e0 con 0x55e3262204d0 2024-05-27T07:41:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48c5e5b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262182f0 0x55e3265f4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48cef70740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48c01b2840 con 0x55e3262411b0 2024-05-27T07:41:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48c5e5b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262182f0 0x55e3265f4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39106/0 (socket says 172.21.15.114:39106) 2024-05-27T07:41:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48c5e5b640 1 -- 172.21.15.114:0/280893537 learned_addr learned my addr 172.21.15.114:0/280893537 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48cef70740 10 client.0 _ll_register_callbacks cb 0x7f48b0005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48c565a640 1 --2- 172.21.15.114:0/280893537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262204d0 0x55e3265f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48c5e5b640 1 -- 172.21.15.114:0/280893537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262411b0 msgr2=0x55e3265f1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48c5e5b640 1 --2- 172.21.15.114:0/280893537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262411b0 0x55e3265f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48c5e5b640 1 -- 172.21.15.114:0/280893537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262204d0 msgr2=0x55e3265f4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48c5e5b640 1 --2- 172.21.15.114:0/280893537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262204d0 0x55e3265f4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48c5e5b640 1 -- 172.21.15.114:0/280893537 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3265f21e0 con 0x55e3262182f0 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.383+0000 7f48c565a640 1 --2- 172.21.15.114:0/280893537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262204d0 0x55e3265f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.384+0000 7f48c5e5b640 1 --2- 172.21.15.114:0/280893537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262182f0 0x55e3265f4680 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f48bc031440 tx=0x7f48bc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948a2f30a619d96d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.384+0000 7f48a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.384+0000 7f48a97fa640 1 -- 172.21.15.114:0/280893537 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc049040 con 0x55e3262182f0 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.384+0000 7f48a97fa640 1 -- 172.21.15.114:0/280893537 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f48bc0cd040 con 0x55e3262182f0 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.384+0000 7f48cef70740 1 -- 172.21.15.114:0/280893537 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48b800c7d0 con 0x55e3262182f0 2024-05-27T07:41:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.384+0000 7f48a97fa640 1 -- 172.21.15.114:0/280893537 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f48bc0d2c90 con 0x55e3262182f0 2024-05-27T07:41:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.384+0000 7f48cef70740 10 client.15597 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.384+0000 7f48cef70740 1 -- 172.21.15.114:0/280893537 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e326666ae0 con 0x55e3262182f0 2024-05-27T07:41:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.384+0000 7f48a97fa640 1 -- 172.21.15.114:0/280893537 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48bc0c6420 con 0x55e3262182f0 2024-05-27T07:41:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.384+0000 7f48a97fa640 1 -- 172.21.15.114:0/280893537 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f48bc0e1040 con 0x55e3262182f0 2024-05-27T07:41:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.384+0000 7f48cef70740 10 client.15597 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.384+0000 7f48cef70740 10 client.15597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.384+0000 7f48cef70740 20 client.15597 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.385+0000 7f48cef70740 1 --2- 172.21.15.114:0/280893537 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e3261d51c0 0x55e3266691b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.385+0000 7f48cef70740 4 client.15597 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:41:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.385+0000 7f48cef70740 1 -- 172.21.15.114:0/280893537 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f48b800c7d0 con 0x55e3261d51c0 2024-05-27T07:41:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.385+0000 7f48c565a640 1 --2- 172.21.15.114:0/280893537 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e3261d51c0 0x55e3266691b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.385+0000 7f48a97fa640 10 client.15597 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.385+0000 7f48c565a640 1 --2- 172.21.15.114:0/280893537 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e3261d51c0 0x55e3266691b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c018a90625b643e server_cookie=9fff4aca6313adb2 in_seq=0 out_seq=0 2024-05-27T07:41:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.389+0000 7f48a97fa640 1 -- 172.21.15.114:0/280893537 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7f48b800c7d0 con 0x55e3261d51c0 2024-05-27T07:41:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.389+0000 7f48a97fa640 10 client.15597 handle_command_reply: tid=0 2024-05-27T07:41:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.389+0000 7f48cef70740 1 client.15597 shutdown 2024-05-27T07:41:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.389+0000 7f48cef70740 1 -- 172.21.15.114:0/280893537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262182f0 msgr2=0x55e3265f4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.389+0000 7f48cef70740 1 --2- 172.21.15.114:0/280893537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262182f0 0x55e3265f4680 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f48bc031440 tx=0x7f48bc000bc0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.390+0000 7f48cef70740 1 -- 172.21.15.114:0/280893537 shutdown_connections 2024-05-27T07:41:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.390+0000 7f48cef70740 1 --2- 172.21.15.114:0/280893537 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e3261d51c0 0x55e3266691b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.390+0000 7f48cef70740 1 --2- 172.21.15.114:0/280893537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3262411b0 0x55e3265f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.390+0000 7f48a97fa640 0 client.15597 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.390+0000 7f48cef70740 1 --2- 172.21.15.114:0/280893537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e3262204d0 0x55e3265f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.390+0000 7f48cef70740 1 --2- 172.21.15.114:0/280893537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e3262182f0 0x55e3265f4680 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.390+0000 7f48cef70740 1 -- 172.21.15.114:0/280893537 >> 172.21.15.114:0/280893537 conn(0x55e326235300 msgr2=0x55e3265de540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.390+0000 7f48cef70740 1 -- 172.21.15.114:0/280893537 shutdown_connections 2024-05-27T07:41:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.390+0000 7f48cef70740 1 -- 172.21.15.114:0/280893537 wait complete. 2024-05-27T07:41:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.390+0000 7f48cef70740 20 client.15597 trim_cache size 0 max 16384 2024-05-27T07:41:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:41:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:41:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:41:28.390 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.390+0000 7f48c80bf640 1 -- 172.21.15.114:0/3758568567 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f48ac0b29d0 msgr2=0x7f48ac0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:28.390 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.391+0000 7f48c80bf640 1 --2- 172.21.15.114:0/3758568567 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f48ac0b29d0 0x7f48ac0b4ee0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f48b800a000 tx=0x7f48b8019000 comp rx=0 tx=0).stop 2024-05-27T07:41:28.390 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.391+0000 7f48c80bf640 1 -- 172.21.15.114:0/3758568567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48c01065f0 msgr2=0x7f48c01a2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:28.390 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.391+0000 7f48c80bf640 1 --2- 172.21.15.114:0/3758568567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48c01065f0 0x7f48c01a2750 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f48bc006010 tx=0x7f48bc004060 comp rx=0 tx=0).stop 2024-05-27T07:41:28.390 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.391+0000 7f48c80bf640 1 -- 172.21.15.114:0/3758568567 shutdown_connections 2024-05-27T07:41:28.390 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.391+0000 7f48c80bf640 1 --2- 172.21.15.114:0/3758568567 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f48ac0b29d0 0x7f48ac0b4ee0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.390 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.391+0000 7f48c80bf640 1 --2- 172.21.15.114:0/3758568567 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48c010c340 0x7f48c019c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.390 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.391+0000 7f48c80bf640 1 --2- 172.21.15.114:0/3758568567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c010bea0 0x7f48c01a2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.390 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.391+0000 7f48c80bf640 1 --2- 172.21.15.114:0/3758568567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48c01065f0 0x7f48c01a2750 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:28.391 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.391+0000 7f48c80bf640 1 -- 172.21.15.114:0/3758568567 >> 172.21.15.114:0/3758568567 conn(0x7f48c0101520 msgr2=0x7f48c01022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:28.391 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.391+0000 7f48c80bf640 1 -- 172.21.15.114:0/3758568567 shutdown_connections 2024-05-27T07:41:28.391 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:28.391+0000 7f48c80bf640 1 -- 172.21.15.114:0/3758568567 wait complete. 2024-05-27T07:41:29.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.341+0000 7fad8ff62700 1 -- 172.21.15.114:0/314546992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad881029b0 msgr2=0x7fad88102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:29.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.341+0000 7fad8ff62700 1 --2- 172.21.15.114:0/314546992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad881029b0 0x7fad88102dc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fad7800aaa0 tx=0x7fad7800adb0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.341+0000 7fad8ff62700 1 -- 172.21.15.114:0/314546992 shutdown_connections 2024-05-27T07:41:29.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.341+0000 7fad8ff62700 1 --2- 172.21.15.114:0/314546992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad88103300 0x7fad8810f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.341+0000 7fad8ff62700 1 --2- 172.21.15.114:0/314546992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad881089e0 0x7fad88108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.341+0000 7fad8ff62700 1 --2- 172.21.15.114:0/314546992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad881029b0 0x7fad88102dc0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.341+0000 7fad8ff62700 1 -- 172.21.15.114:0/314546992 >> 172.21.15.114:0/314546992 conn(0x7fad880fe6e0 msgr2=0x7fad88100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:29.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.341+0000 7fad8ff62700 1 -- 172.21.15.114:0/314546992 shutdown_connections 2024-05-27T07:41:29.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.341+0000 7fad8ff62700 1 -- 172.21.15.114:0/314546992 wait complete. 2024-05-27T07:41:29.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.342+0000 7fad8ff62700 1 Processor -- start 2024-05-27T07:41:29.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.342+0000 7fad8ff62700 1 -- start start 2024-05-27T07:41:29.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.343+0000 7fad8ff62700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad881029b0 0x7fad8819aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.343+0000 7fad8ff62700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad88103300 0x7fad8819b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.343+0000 7fad8ff62700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad881089e0 0x7fad8819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.343+0000 7fad8ff62700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fad881122d0 con 0x7fad88103300 2024-05-27T07:41:29.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.343+0000 7fad8ff62700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fad88112150 con 0x7fad881029b0 2024-05-27T07:41:29.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.343+0000 7fad8ff62700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fad88112450 con 0x7fad881089e0 2024-05-27T07:41:29.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.343+0000 7fad8d4fd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad88103300 0x7fad8819b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.343+0000 7fad8d4fd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad88103300 0x7fad8819b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39124/0 (socket says 172.21.15.114:39124) 2024-05-27T07:41:29.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.343+0000 7fad8d4fd700 1 -- 172.21.15.114:0/1459398360 learned_addr learned my addr 172.21.15.114:0/1459398360 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:29.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.343+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad881029b0 0x7fad8819aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.344+0000 7fad8e4ff700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad881089e0 0x7fad8819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.344+0000 7fad8d4fd700 1 -- 172.21.15.114:0/1459398360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad881089e0 msgr2=0x7fad8819f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:29.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.344+0000 7fad8d4fd700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad881089e0 0x7fad8819f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.344+0000 7fad8d4fd700 1 -- 172.21.15.114:0/1459398360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad881029b0 msgr2=0x7fad8819aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:29.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.344+0000 7fad8d4fd700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad881029b0 0x7fad8819aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.344+0000 7fad8d4fd700 1 -- 172.21.15.114:0/1459398360 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8819fd90 con 0x7fad88103300 2024-05-27T07:41:29.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.344+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad881029b0 0x7fad8819aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:29.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.344+0000 7fad8e4ff700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad881089e0 0x7fad8819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:29.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.344+0000 7fad8d4fd700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad88103300 0x7fad8819b460 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fad78000c00 tx=0x7fad78003930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.345+0000 7fad7effd700 1 -- 172.21.15.114:0/1459398360 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78003e60 con 0x7fad88103300 2024-05-27T07:41:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.345+0000 7fad7effd700 1 -- 172.21.15.114:0/1459398360 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad78024070 con 0x7fad88103300 2024-05-27T07:41:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.345+0000 7fad7effd700 1 -- 172.21.15.114:0/1459398360 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78029470 con 0x7fad88103300 2024-05-27T07:41:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.345+0000 7fad8ff62700 1 -- 172.21.15.114:0/1459398360 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad881a0020 con 0x7fad88103300 2024-05-27T07:41:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.345+0000 7fad8ff62700 1 -- 172.21.15.114:0/1459398360 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad881a04b0 con 0x7fad88103300 2024-05-27T07:41:29.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.350+0000 7fad9e977b80 1 Processor -- start 2024-05-27T07:41:29.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.350+0000 7fad9e977b80 1 -- start start 2024-05-27T07:41:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.351+0000 7fad9e977b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d649e40 0x55a71d7ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.351+0000 7fad9e977b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d7cf070 0x55a71d7c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.351+0000 7fad9e977b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7c9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.351+0000 7fad9e977b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a71d6f6d70 con 0x55a71d7c9120 2024-05-27T07:41:29.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.351+0000 7fad9e977b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a71d6e6910 con 0x55a71d7cf070 2024-05-27T07:41:29.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.351+0000 7fad9e977b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a71d6ea950 con 0x55a71d649e40 2024-05-27T07:41:29.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.351+0000 7fad8dcfe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d7cf070 0x55a71d7c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.351+0000 7fad8dcfe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d7cf070 0x55a71d7c8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:49970/0 (socket says 172.21.15.114:49970) 2024-05-27T07:41:29.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.351+0000 7fad8dcfe700 1 -- 172.21.15.114:0/1086796638 learned_addr learned my addr 172.21.15.114:0/1086796638 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:29.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.352+0000 7fad8dcfe700 1 -- 172.21.15.114:0/1086796638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d649e40 msgr2=0x55a71d7ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:29.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.352+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/1086796638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d649e40 0x55a71d7ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.352+0000 7fad8dcfe700 1 -- 172.21.15.114:0/1086796638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 msgr2=0x55a71d7c9530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:41:29.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.352+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/1086796638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7c9530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.352+0000 7fad8dcfe700 1 -- 172.21.15.114:0/1086796638 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a71d68ffb0 con 0x55a71d7cf070 2024-05-27T07:41:29.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.352+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/1086796638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d7cf070 0x55a71d7c8be0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fad84009d60 tx=0x7fad840106b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=796d430d82e70f33 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:29.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.352+0000 7fad73fff700 1 -- 172.21.15.114:0/1086796638 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8401c480 con 0x55a71d7cf070 2024-05-27T07:41:29.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.352+0000 7fad73fff700 1 -- 172.21.15.114:0/1086796638 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad8401cad0 con 0x55a71d7cf070 2024-05-27T07:41:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.353+0000 7fad7effd700 1 -- 172.21.15.114:0/1459398360 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fad78020780 con 0x7fad88103300 2024-05-27T07:41:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.353+0000 7fad73fff700 1 -- 172.21.15.114:0/1086796638 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad840118e0 con 0x55a71d7cf070 2024-05-27T07:41:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.354+0000 7fad9e977b80 1 -- 172.21.15.114:0/1086796638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d7cf070 msgr2=0x55a71d7c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.354+0000 7fad9e977b80 1 --2- 172.21.15.114:0/1086796638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d7cf070 0x55a71d7c8be0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fad84009d60 tx=0x7fad840106b0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.354+0000 7fad9e977b80 1 -- 172.21.15.114:0/1086796638 shutdown_connections 2024-05-27T07:41:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.354+0000 7fad9e977b80 1 --2- 172.21.15.114:0/1086796638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.354+0000 7fad9e977b80 1 --2- 172.21.15.114:0/1086796638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d649e40 0x55a71d7ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.354+0000 7fad9e977b80 1 --2- 172.21.15.114:0/1086796638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d7cf070 0x55a71d7c8be0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.354+0000 7fad9e977b80 1 -- 172.21.15.114:0/1086796638 >> 172.21.15.114:0/1086796638 conn(0x55a71d64c770 msgr2=0x55a71d7c3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.355+0000 7fad9e977b80 1 -- 172.21.15.114:0/1086796638 shutdown_connections 2024-05-27T07:41:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.355+0000 7fad9e977b80 1 -- 172.21.15.114:0/1086796638 wait complete. 2024-05-27T07:41:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.355+0000 7fad7effd700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fad740b2a60 0x7fad740b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.355+0000 7fad9e977b80 1 -- start start 2024-05-27T07:41:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.356+0000 7fad8e4ff700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fad740b2a60 0x7fad740b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.356+0000 7fad9e977b80 1 Processor -- start 2024-05-27T07:41:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.356+0000 7fad9e977b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d649e40 0x55a71d7d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.356+0000 7fad9e977b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7d2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.356+0000 7fad9e977b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d7cf070 0x55a71d7d32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.356+0000 7fad9e977b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a71d5ecfa0 con 0x55a71d7c9120 2024-05-27T07:41:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.356+0000 7fad9e977b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a71d64eda0 con 0x55a71d649e40 2024-05-27T07:41:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.356+0000 7fad9e977b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a71d7d5c00 con 0x55a71d7cf070 2024-05-27T07:41:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.356+0000 7fad9e977b80 10 client.0 _ll_register_callbacks cb 0x55a71d67d5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.357+0000 7fad8dcfe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d649e40 0x55a71d7d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.357+0000 7fad8dcfe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d649e40 0x55a71d7d2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:49982/0 (socket says 172.21.15.114:49982) 2024-05-27T07:41:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.357+0000 7fad8dcfe700 1 -- 172.21.15.114:0/559238394 learned_addr learned my addr 172.21.15.114:0/559238394 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.357+0000 7fad7effd700 1 -- 172.21.15.114:0/1459398360 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fad780e7cc0 con 0x7fad88103300 2024-05-27T07:41:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.357+0000 7fad8e4ff700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fad740b2a60 0x7fad740b4f70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fad80009fd0 tx=0x7fad80009920 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.357+0000 7fad8d4fd700 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7d2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.357+0000 7fad8e4ff700 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d7cf070 0x55a71d7d32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.358+0000 7fad8dcfe700 1 -- 172.21.15.114:0/559238394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d7cf070 msgr2=0x55a71d7d32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.358+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d7cf070 0x55a71d7d32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.358+0000 7fad8dcfe700 1 -- 172.21.15.114:0/559238394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 msgr2=0x55a71d7d2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.358+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7d2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.358+0000 7fad8dcfe700 1 -- 172.21.15.114:0/559238394 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a71d6368d0 con 0x55a71d649e40 2024-05-27T07:41:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.358+0000 7fad8d4fd700 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7d2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.358+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d649e40 0x55a71d7d2830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fad840096a0 tx=0x7fad8401ce90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=956e6bd1b31a0587 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.358+0000 7fad717fa700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:41:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad717fa700 1 -- 172.21.15.114:0/559238394 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad84004440 con 0x55a71d649e40 2024-05-27T07:41:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad717fa700 1 -- 172.21.15.114:0/559238394 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fad840268d0 con 0x55a71d649e40 2024-05-27T07:41:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad9e977b80 1 -- 172.21.15.114:0/559238394 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad8401cad0 con 0x55a71d649e40 2024-05-27T07:41:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad717fa700 1 -- 172.21.15.114:0/559238394 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fad84036050 con 0x55a71d649e40 2024-05-27T07:41:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad9e977b80 10 client.25172 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad9e977b80 1 -- 172.21.15.114:0/559238394 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a71d6a0760 con 0x55a71d649e40 2024-05-27T07:41:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad717fa700 1 -- 172.21.15.114:0/559238394 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad84005020 con 0x55a71d649e40 2024-05-27T07:41:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad717fa700 1 -- 172.21.15.114:0/559238394 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fad840048f0 con 0x55a71d649e40 2024-05-27T07:41:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad9e977b80 10 client.25172 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad9e977b80 10 client.25172 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad9e977b80 20 client.25172 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad9e977b80 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a71d7e2750 0x55a71d7e4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad9e977b80 4 client.25172 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.359+0000 7fad9e977b80 1 -- 172.21.15.114:0/559238394 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fad8401cad0 con 0x55a71d7e2750 2024-05-27T07:41:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.360+0000 7fad8d4fd700 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a71d7e2750 0x55a71d7e4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.361+0000 7fad717fa700 10 client.25172 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.361+0000 7fad8d4fd700 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a71d7e2750 0x55a71d7e4b70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5462e4f6ffc1f18e server_cookie=5cc903732ee07fa9 in_seq=0 out_seq=0 2024-05-27T07:41:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.362+0000 7fad717fa700 1 -- 172.21.15.114:0/559238394 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fad8401cad0 con 0x55a71d7e2750 2024-05-27T07:41:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.362+0000 7fad717fa700 10 client.25172 handle_command_reply: tid=0 2024-05-27T07:41:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.362+0000 7fad9e977b80 1 client.25172 shutdown 2024-05-27T07:41:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.362+0000 7fad9e977b80 1 -- 172.21.15.114:0/559238394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d649e40 msgr2=0x55a71d7d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.362+0000 7fad9e977b80 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d649e40 0x55a71d7d2830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fad840096a0 tx=0x7fad8401ce90 comp rx=0 tx=0).stop 2024-05-27T07:41:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.362+0000 7fad9e977b80 1 -- 172.21.15.114:0/559238394 shutdown_connections 2024-05-27T07:41:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.362+0000 7fad9e977b80 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d7cf070 0x55a71d7d32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.362+0000 7fad9e977b80 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a71d7e2750 0x55a71d7e4b70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.362+0000 7fad9e977b80 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d649e40 0x55a71d7d2830 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.362+0000 7fad9e977b80 1 --2- 172.21.15.114:0/559238394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7d2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.362+0000 7fad9e977b80 1 -- 172.21.15.114:0/559238394 >> 172.21.15.114:0/559238394 conn(0x55a71d64c770 msgr2=0x55a71d7c7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.362+0000 7fad717fa700 0 client.25172 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.363+0000 7fad9e977b80 1 -- 172.21.15.114:0/559238394 shutdown_connections 2024-05-27T07:41:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.363+0000 7fad9e977b80 1 -- 172.21.15.114:0/559238394 wait complete. 2024-05-27T07:41:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.363+0000 7fad9e977b80 20 client.25172 trim_cache size 0 max 16384 2024-05-27T07:41:29.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.403+0000 7fad9e977b80 1 Processor -- start 2024-05-27T07:41:29.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.403+0000 7fad9e977b80 1 -- start start 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.403+0000 7fad9e977b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d649e40 0x55a71d7c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.403+0000 7fad9e977b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7deea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad8d4fd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7deea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad8d4fd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7deea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39132/0 (socket says 172.21.15.114:39132) 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad8d4fd700 1 -- 172.21.15.114:0/345022798 learned_addr learned my addr 172.21.15.114:0/345022798 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/345022798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d649e40 0x55a71d7c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad9e977b80 1 --2- 172.21.15.114:0/345022798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d7cf070 0x55a71d7d2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad9e977b80 1 -- 172.21.15.114:0/345022798 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fad881ac2f0 con 0x55a71d7c9120 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad9e977b80 1 -- 172.21.15.114:0/345022798 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fad881aeb70 con 0x55a71d7cf070 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad9e977b80 1 -- 172.21.15.114:0/345022798 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fad881ac960 con 0x55a71d649e40 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad8dcfe700 1 -- 172.21.15.114:0/345022798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d7cf070 msgr2=0x55a71d7d2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/345022798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d7cf070 0x55a71d7d2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad8dcfe700 1 -- 172.21.15.114:0/345022798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 msgr2=0x55a71d7deea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/345022798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7deea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad8dcfe700 1 -- 172.21.15.114:0/345022798 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a71d7d0da0 con 0x55a71d649e40 2024-05-27T07:41:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad8d4fd700 1 --2- 172.21.15.114:0/345022798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7deea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.404+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/345022798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d649e40 0x55a71d7c8be0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fad84004e60 tx=0x7fad840106b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ad7e7f41e1d427e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.405+0000 7fad737fe700 1 -- 172.21.15.114:0/345022798 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8403f070 con 0x55a71d649e40 2024-05-27T07:41:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.405+0000 7fad737fe700 1 -- 172.21.15.114:0/345022798 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad8401ca90 con 0x55a71d649e40 2024-05-27T07:41:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.405+0000 7fad737fe700 1 -- 172.21.15.114:0/345022798 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad84005120 con 0x55a71d649e40 2024-05-27T07:41:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.406+0000 7fad9e977b80 1 -- 172.21.15.114:0/345022798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d649e40 msgr2=0x55a71d7c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.406+0000 7fad9e977b80 1 --2- 172.21.15.114:0/345022798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d649e40 0x55a71d7c8be0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fad84004e60 tx=0x7fad840106b0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.406+0000 7fad9e977b80 1 -- 172.21.15.114:0/345022798 shutdown_connections 2024-05-27T07:41:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.406+0000 7fad9e977b80 1 --2- 172.21.15.114:0/345022798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d7cf070 0x55a71d7d2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.406+0000 7fad9e977b80 1 --2- 172.21.15.114:0/345022798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d649e40 0x55a71d7c8be0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.406+0000 7fad9e977b80 1 --2- 172.21.15.114:0/345022798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7c9120 0x55a71d7deea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.406+0000 7fad9e977b80 1 -- 172.21.15.114:0/345022798 >> 172.21.15.114:0/345022798 conn(0x55a71d64c770 msgr2=0x55a71d7cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.406+0000 7fad9e977b80 1 -- 172.21.15.114:0/345022798 shutdown_connections 2024-05-27T07:41:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.406+0000 7fad9e977b80 1 -- 172.21.15.114:0/345022798 wait complete. 2024-05-27T07:41:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.407+0000 7fad9e977b80 1 -- start start 2024-05-27T07:41:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.407+0000 7fad9e977b80 1 Processor -- start 2024-05-27T07:41:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.407+0000 7fad9e977b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d649e40 0x55a71d7e4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.407+0000 7fad9e977b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d7c9120 0x55a71d7e5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.407+0000 7fad9e977b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7cf070 0x55a71d843f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.407+0000 7fad9e977b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fad881ad030 con 0x55a71d7cf070 2024-05-27T07:41:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.407+0000 7fad9e977b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fad881ae4a0 con 0x55a71d649e40 2024-05-27T07:41:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.407+0000 7fad9e977b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fad881ad700 con 0x55a71d7c9120 2024-05-27T07:41:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.407+0000 7fad9e977b80 10 client.0 _ll_register_callbacks cb 0x55a71d67d5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.408+0000 7fad8e4ff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7cf070 0x55a71d843f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.408+0000 7fad8e4ff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7cf070 0x55a71d843f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39140/0 (socket says 172.21.15.114:39140) 2024-05-27T07:41:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.408+0000 7fad8e4ff700 1 -- 172.21.15.114:0/1389366015 learned_addr learned my addr 172.21.15.114:0/1389366015 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.408+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/1389366015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d649e40 0x55a71d7e4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.408+0000 7fad8d4fd700 1 --2- 172.21.15.114:0/1389366015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d7c9120 0x55a71d7e5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.408+0000 7fad8dcfe700 1 -- 172.21.15.114:0/1389366015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d7c9120 msgr2=0x55a71d7e5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.408+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/1389366015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d7c9120 0x55a71d7e5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.408+0000 7fad8dcfe700 1 -- 172.21.15.114:0/1389366015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7cf070 msgr2=0x55a71d843f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:29.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.408+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/1389366015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7cf070 0x55a71d843f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.408+0000 7fad8dcfe700 1 -- 172.21.15.114:0/1389366015 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a71d844530 con 0x55a71d649e40 2024-05-27T07:41:29.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.408+0000 7fad8dcfe700 1 --2- 172.21.15.114:0/1389366015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d649e40 0x55a71d7e4fe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55a71d7ce510 tx=0x7fad8401ce90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3469a68836c2b605 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:29.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.408+0000 7fad717fa700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:41:29.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.409+0000 7fad717fa700 1 -- 172.21.15.114:0/1389366015 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8403f030 con 0x55a71d649e40 2024-05-27T07:41:29.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.409+0000 7fad9e977b80 1 -- 172.21.15.114:0/1389366015 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad8401ca90 con 0x55a71d649e40 2024-05-27T07:41:29.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.409+0000 7fad717fa700 1 -- 172.21.15.114:0/1389366015 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fad84026ae0 con 0x55a71d649e40 2024-05-27T07:41:29.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.409+0000 7fad717fa700 1 -- 172.21.15.114:0/1389366015 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fad8404b050 con 0x55a71d649e40 2024-05-27T07:41:29.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.409+0000 7fad9e977b80 10 client.25175 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:29.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.409+0000 7fad9e977b80 1 -- 172.21.15.114:0/1389366015 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a71d84ca70 con 0x55a71d649e40 2024-05-27T07:41:29.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.409+0000 7fad717fa700 1 -- 172.21.15.114:0/1389366015 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad8401ca90 con 0x55a71d649e40 2024-05-27T07:41:29.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.410+0000 7fad717fa700 1 -- 172.21.15.114:0/1389366015 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fad84005530 con 0x55a71d649e40 2024-05-27T07:41:29.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.410+0000 7fad9e977b80 10 client.25175 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:29.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.410+0000 7fad9e977b80 10 client.25175 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:29.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.410+0000 7fad9e977b80 20 client.25175 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:29.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.410+0000 7fad9e977b80 1 --2- 172.21.15.114:0/1389366015 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a71d84d120 0x55a71d84f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:29.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.410+0000 7fad9e977b80 4 client.25175 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:41:29.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.410+0000 7fad9e977b80 1 -- 172.21.15.114:0/1389366015 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a71d6f8cc0 con 0x55a71d84d120 2024-05-27T07:41:29.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.410+0000 7fad8d4fd700 1 --2- 172.21.15.114:0/1389366015 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a71d84d120 0x55a71d84f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:29.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.411+0000 7fad717fa700 10 client.25175 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:29.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.411+0000 7fad8d4fd700 1 --2- 172.21.15.114:0/1389366015 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a71d84d120 0x55a71d84f540 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b59968e47f080704 server_cookie=c3d147da3246ac5d in_seq=0 out_seq=0 2024-05-27T07:41:29.410 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:29 smithi114 ceph-mon[32038]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:29.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.412+0000 7fad717fa700 1 -- 172.21.15.114:0/1389366015 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a71d6f8cc0 con 0x55a71d84d120 2024-05-27T07:41:29.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.412+0000 7fad717fa700 10 client.25175 handle_command_reply: tid=0 2024-05-27T07:41:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.412+0000 7fad9e977b80 1 client.25175 shutdown 2024-05-27T07:41:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.412+0000 7fad9e977b80 1 -- 172.21.15.114:0/1389366015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d649e40 msgr2=0x55a71d7e4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.412+0000 7fad9e977b80 1 --2- 172.21.15.114:0/1389366015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d649e40 0x55a71d7e4fe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55a71d7ce510 tx=0x7fad8401ce90 comp rx=0 tx=0).stop 2024-05-27T07:41:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.412+0000 7fad9e977b80 1 -- 172.21.15.114:0/1389366015 shutdown_connections 2024-05-27T07:41:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.412+0000 7fad9e977b80 1 --2- 172.21.15.114:0/1389366015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a71d7cf070 0x55a71d843f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.412+0000 7fad9e977b80 1 --2- 172.21.15.114:0/1389366015 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a71d84d120 0x55a71d84f540 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.412+0000 7fad717fa700 0 client.25175 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.412+0000 7fad9e977b80 1 --2- 172.21.15.114:0/1389366015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a71d649e40 0x55a71d7e4fe0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.412+0000 7fad9e977b80 1 --2- 172.21.15.114:0/1389366015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d7c9120 0x55a71d7e5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.412+0000 7fad9e977b80 1 -- 172.21.15.114:0/1389366015 >> 172.21.15.114:0/1389366015 conn(0x55a71d64c770 msgr2=0x55a71d7cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.413+0000 7fad9e977b80 1 -- 172.21.15.114:0/1389366015 shutdown_connections 2024-05-27T07:41:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.413+0000 7fad9e977b80 1 -- 172.21.15.114:0/1389366015 wait complete. 2024-05-27T07:41:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.413+0000 7fad9e977b80 20 client.25175 trim_cache size 0 max 16384 2024-05-27T07:41:29.413 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:41:29.413 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:41:29.413 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:41:29.413 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:41:29.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.414+0000 7fad8ff62700 1 -- 172.21.15.114:0/1459398360 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fad740b2a60 msgr2=0x7fad740b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.414+0000 7fad8ff62700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fad740b2a60 0x7fad740b4f70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fad80009fd0 tx=0x7fad80009920 comp rx=0 tx=0).stop 2024-05-27T07:41:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.414+0000 7fad8ff62700 1 -- 172.21.15.114:0/1459398360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad88103300 msgr2=0x7fad8819b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.414+0000 7fad8ff62700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad88103300 0x7fad8819b460 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fad78000c00 tx=0x7fad78003930 comp rx=0 tx=0).stop 2024-05-27T07:41:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.414+0000 7fad8ff62700 1 -- 172.21.15.114:0/1459398360 shutdown_connections 2024-05-27T07:41:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.414+0000 7fad8ff62700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad881089e0 0x7fad8819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.414+0000 7fad8ff62700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fad740b2a60 0x7fad740b4f70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.414+0000 7fad8ff62700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad881029b0 0x7fad8819aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.414+0000 7fad8ff62700 1 --2- 172.21.15.114:0/1459398360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad88103300 0x7fad8819b460 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.414+0000 7fad8ff62700 1 -- 172.21.15.114:0/1459398360 >> 172.21.15.114:0/1459398360 conn(0x7fad880fe6e0 msgr2=0x7fad88106c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.415+0000 7fad8ff62700 1 -- 172.21.15.114:0/1459398360 shutdown_connections 2024-05-27T07:41:29.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:29.415+0000 7fad8ff62700 1 -- 172.21.15.114:0/1459398360 wait complete. 2024-05-27T07:41:29.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:29 smithi163 ceph-mon[36451]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:29.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:29 smithi186 ceph-mon[36647]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:30.117 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:41:30.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:41:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:31.677 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:31 smithi114 ceph-mon[32038]: pgmap v56: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-27T07:41:31.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:31 smithi163 ceph-mon[36451]: pgmap v56: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-27T07:41:31.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:31 smithi186 ceph-mon[36647]: pgmap v56: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-27T07:41:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.400+0000 7f6d12973700 1 -- 172.21.15.114:0/4247905224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d0c1034b0 msgr2=0x7f6d0c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.400+0000 7f6d12973700 1 --2- 172.21.15.114:0/4247905224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d0c1034b0 0x7f6d0c1038c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0099c0 tx=0x7f6cfc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:41:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.400+0000 7f6d12973700 1 -- 172.21.15.114:0/4247905224 shutdown_connections 2024-05-27T07:41:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.400+0000 7f6d12973700 1 --2- 172.21.15.114:0/4247905224 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d0c108730 0x7f6d0c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.400+0000 7f6d12973700 1 --2- 172.21.15.114:0/4247905224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d0c1034b0 0x7f6d0c1038c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.400+0000 7f6d12973700 1 --2- 172.21.15.114:0/4247905224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d0c107de0 0x7f6d0c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.400+0000 7f6d12973700 1 -- 172.21.15.114:0/4247905224 >> 172.21.15.114:0/4247905224 conn(0x7f6d0c0fedb0 msgr2=0x7f6d0c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.400+0000 7f6d12973700 1 -- 172.21.15.114:0/4247905224 shutdown_connections 2024-05-27T07:41:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.400+0000 7f6d12973700 1 -- 172.21.15.114:0/4247905224 wait complete. 2024-05-27T07:41:32.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.401+0000 7f6d12973700 1 Processor -- start 2024-05-27T07:41:32.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.401+0000 7f6d12973700 1 -- start start 2024-05-27T07:41:32.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.402+0000 7f6d12973700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d0c1034b0 0x7f6d0c19d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:32.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.402+0000 7f6d12973700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d0c107de0 0x7f6d0c19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.402+0000 7f6d12973700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d0c108730 0x7f6d0c1a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.402+0000 7f6d12973700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c116650 con 0x7f6d0c108730 2024-05-27T07:41:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.402+0000 7f6d12973700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c1164d0 con 0x7f6d0c107de0 2024-05-27T07:41:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.402+0000 7f6d12973700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c1167d0 con 0x7f6d0c1034b0 2024-05-27T07:41:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.402+0000 7f6d10f10700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d0c108730 0x7f6d0c1a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.402+0000 7f6d10f10700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d0c108730 0x7f6d0c1a1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39160/0 (socket says 172.21.15.114:39160) 2024-05-27T07:41:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.402+0000 7f6d10f10700 1 -- 172.21.15.114:0/2965460516 learned_addr learned my addr 172.21.15.114:0/2965460516 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.402+0000 7f6d0bfff700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d0c1034b0 0x7f6d0c19d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.402+0000 7f6d03fff700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d0c107de0 0x7f6d0c19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.403+0000 7f6d10f10700 1 -- 172.21.15.114:0/2965460516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d0c1034b0 msgr2=0x7f6d0c19d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.403+0000 7f6d10f10700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d0c1034b0 0x7f6d0c19d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.403+0000 7f6d10f10700 1 -- 172.21.15.114:0/2965460516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d0c107de0 msgr2=0x7f6d0c19d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.403+0000 7f6d10f10700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d0c107de0 0x7f6d0c19d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:32.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.403+0000 7f6d10f10700 1 -- 172.21.15.114:0/2965460516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0c1a1f30 con 0x7f6d0c108730 2024-05-27T07:41:32.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.403+0000 7f6d10f10700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d0c108730 0x7f6d0c1a1890 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6d04007ee0 tx=0x7f6d0400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:32.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.403+0000 7f6d09ffb700 1 -- 172.21.15.114:0/2965460516 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0400ddd0 con 0x7f6d0c108730 2024-05-27T07:41:32.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.403+0000 7f6d09ffb700 1 -- 172.21.15.114:0/2965460516 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d04004020 con 0x7f6d0c108730 2024-05-27T07:41:32.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.403+0000 7f6d12973700 1 -- 172.21.15.114:0/2965460516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0c1a2220 con 0x7f6d0c108730 2024-05-27T07:41:32.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.404+0000 7f6d12973700 1 -- 172.21.15.114:0/2965460516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d0c1a2650 con 0x7f6d0c108730 2024-05-27T07:41:32.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.404+0000 7f6d09ffb700 1 -- 172.21.15.114:0/2965460516 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04015600 con 0x7f6d0c108730 2024-05-27T07:41:32.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.406+0000 7f6d12973700 1 -- 172.21.15.114:0/2965460516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d0c0512b0 con 0x7f6d0c108730 2024-05-27T07:41:32.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.408+0000 7f6d09ffb700 1 -- 172.21.15.114:0/2965460516 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6d040041d0 con 0x7f6d0c108730 2024-05-27T07:41:32.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.409+0000 7f6d09ffb700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6cf80b29f0 0x7f6cf80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:32.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.409+0000 7f6d09ffb700 1 -- 172.21.15.114:0/2965460516 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f6d040e2370 con 0x7f6d0c108730 2024-05-27T07:41:32.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.410+0000 7f6d0bfff700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6cf80b29f0 0x7f6cf80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:32.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.410+0000 7f6d0bfff700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6cf80b29f0 0x7f6cf80b4f00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc000c00 tx=0x7f6cfc01a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:32.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.419+0000 7f6d09ffb700 1 -- 172.21.15.114:0/2965460516 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6d04010070 con 0x7f6d0c108730 2024-05-27T07:41:32.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.924+0000 7f6d12973700 1 -- 172.21.15.114:0/2965460516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d0c19e3c0 con 0x7f6d0c108730 2024-05-27T07:41:32.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.925+0000 7f6d09ffb700 1 -- 172.21.15.114:0/2965460516 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11827 (secure 0 0 0) 0x7f6d040a99a0 con 0x7f6d0c108730 2024-05-27T07:41:32.923 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:41:32.923 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:37:06.645274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:41:32.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.928+0000 7f6d12973700 1 -- 172.21.15.114:0/2965460516 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6cf80b29f0 msgr2=0x7f6cf80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:32.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.928+0000 7f6d12973700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6cf80b29f0 0x7f6cf80b4f00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc000c00 tx=0x7f6cfc01a040 comp rx=0 tx=0).stop 2024-05-27T07:41:32.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.928+0000 7f6d12973700 1 -- 172.21.15.114:0/2965460516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d0c108730 msgr2=0x7f6d0c1a1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:32.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.928+0000 7f6d12973700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d0c108730 0x7f6d0c1a1890 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6d04007ee0 tx=0x7f6d0400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:41:32.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.928+0000 7f6d12973700 1 -- 172.21.15.114:0/2965460516 shutdown_connections 2024-05-27T07:41:32.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.928+0000 7f6d12973700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d0c108730 0x7f6d0c1a1890 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:32.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.928+0000 7f6d12973700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d0c1034b0 0x7f6d0c19d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:32.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.928+0000 7f6d12973700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6cf80b29f0 0x7f6cf80b4f00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:32.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.928+0000 7f6d12973700 1 --2- 172.21.15.114:0/2965460516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d0c107de0 0x7f6d0c19d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:32.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.928+0000 7f6d12973700 1 -- 172.21.15.114:0/2965460516 >> 172.21.15.114:0/2965460516 conn(0x7f6d0c0fedb0 msgr2=0x7f6d0c101d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:32.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.928+0000 7f6d12973700 1 -- 172.21.15.114:0/2965460516 shutdown_connections 2024-05-27T07:41:32.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:32.928+0000 7f6d12973700 1 -- 172.21.15.114:0/2965460516 wait complete. 2024-05-27T07:41:32.928 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:41:33.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:33 smithi114 ceph-mon[32038]: pgmap v57: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:33.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:33 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2965460516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:41:33.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:41:33.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:33 smithi163 ceph-mon[36451]: pgmap v57: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:33.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:33 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2965460516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:41:33.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:33 smithi186 ceph-mon[36647]: pgmap v57: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:33.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:33 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2965460516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:41:33.982 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:35.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:35 smithi114 ceph-mon[32038]: pgmap v58: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:35.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:35 smithi163 ceph-mon[36451]: pgmap v58: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:35.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:35 smithi186 ceph-mon[36647]: pgmap v58: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:35.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.903+0000 7f83dcda2700 1 -- 172.21.15.114:0/1628671572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83d8102780 msgr2=0x7f83d8102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:35.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.903+0000 7f83dcda2700 1 --2- 172.21.15.114:0/1628671572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83d8102780 0x7f83d8102b90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f83cc009990 tx=0x7f83cc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.903+0000 7f83dcda2700 1 -- 172.21.15.114:0/1628671572 shutdown_connections 2024-05-27T07:41:35.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.903+0000 7f83dcda2700 1 --2- 172.21.15.114:0/1628671572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d81030d0 0x7f83d8111160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.903+0000 7f83dcda2700 1 --2- 172.21.15.114:0/1628671572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83d81092f0 0x7f83d8109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.903+0000 7f83dcda2700 1 --2- 172.21.15.114:0/1628671572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83d8102780 0x7f83d8102b90 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.903+0000 7f83dcda2700 1 -- 172.21.15.114:0/1628671572 >> 172.21.15.114:0/1628671572 conn(0x7f83d80fe530 msgr2=0x7f83d8100980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:35.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.903+0000 7f83dcda2700 1 -- 172.21.15.114:0/1628671572 shutdown_connections 2024-05-27T07:41:35.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.903+0000 7f83dcda2700 1 -- 172.21.15.114:0/1628671572 wait complete. 2024-05-27T07:41:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.904+0000 7f83dcda2700 1 Processor -- start 2024-05-27T07:41:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.904+0000 7f83dcda2700 1 -- start start 2024-05-27T07:41:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.905+0000 7f83dcda2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d8102780 0x7f83d819af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.905+0000 7f83dcda2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83d81030d0 0x7f83d819b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.905+0000 7f83dcda2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83d81092f0 0x7f83d819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.905+0000 7f83d5d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83d81030d0 0x7f83d819b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.905+0000 7f83d5d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83d81030d0 0x7f83d819b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39822/0 (socket says 172.21.15.114:39822) 2024-05-27T07:41:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.905+0000 7f83d5d9b700 1 -- 172.21.15.114:0/2957770740 learned_addr learned my addr 172.21.15.114:0/2957770740 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.905+0000 7f83dcda2700 1 -- 172.21.15.114:0/2957770740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83d8114590 con 0x7f83d81030d0 2024-05-27T07:41:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.905+0000 7f83dcda2700 1 -- 172.21.15.114:0/2957770740 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f83d8114410 con 0x7f83d81092f0 2024-05-27T07:41:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.905+0000 7f83dcda2700 1 -- 172.21.15.114:0/2957770740 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83d8114710 con 0x7f83d8102780 2024-05-27T07:41:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.905+0000 7f83d659c700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d8102780 0x7f83d819af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.906+0000 7f83d6d9d700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83d81092f0 0x7f83d819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.906+0000 7f83d5d9b700 1 -- 172.21.15.114:0/2957770740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d8102780 msgr2=0x7f83d819af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.906+0000 7f83d5d9b700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d8102780 0x7f83d819af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.906+0000 7f83d5d9b700 1 -- 172.21.15.114:0/2957770740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83d81092f0 msgr2=0x7f83d819f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.906+0000 7f83d5d9b700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83d81092f0 0x7f83d819f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.906+0000 7f83d5d9b700 1 -- 172.21.15.114:0/2957770740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d819fdd0 con 0x7f83d81030d0 2024-05-27T07:41:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.906+0000 7f83d5d9b700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83d81030d0 0x7f83d819b4a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.906+0000 7f83c77fe700 1 -- 172.21.15.114:0/2957770740 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc027070 con 0x7f83d81030d0 2024-05-27T07:41:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.906+0000 7f83c77fe700 1 -- 172.21.15.114:0/2957770740 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83cc022470 con 0x7f83d81030d0 2024-05-27T07:41:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.906+0000 7f83c77fe700 1 -- 172.21.15.114:0/2957770740 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc017650 con 0x7f83d81030d0 2024-05-27T07:41:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.906+0000 7f83dcda2700 1 -- 172.21.15.114:0/2957770740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d81a0060 con 0x7f83d81030d0 2024-05-27T07:41:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.907+0000 7f83dcda2700 1 -- 172.21.15.114:0/2957770740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d81a04f0 con 0x7f83d81030d0 2024-05-27T07:41:35.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.910+0000 7f83c77fe700 1 -- 172.21.15.114:0/2957770740 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f83cc02a070 con 0x7f83d81030d0 2024-05-27T07:41:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.912+0000 7f83c77fe700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83c00b2a70 0x7f83c00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.912+0000 7f83c77fe700 1 -- 172.21.15.114:0/2957770740 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f83cc0e39b0 con 0x7f83d81030d0 2024-05-27T07:41:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.912+0000 7f83d659c700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83c00b2a70 0x7f83c00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.913+0000 7f83d659c700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83c00b2a70 0x7f83c00b4f80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f83c8009680 tx=0x7f83c8009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.913+0000 7f83eb7b7b80 1 Processor -- start 2024-05-27T07:41:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.913+0000 7f83eb7b7b80 1 -- start start 2024-05-27T07:41:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.914+0000 7f83eb7b7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764841ce40 0x5576485a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.914+0000 7f83eb7b7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x55764859bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.914+0000 7f83eb7b7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 0x55764859c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.914+0000 7f83d5d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x55764859bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.914+0000 7f83d5d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x55764859bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39830/0 (socket says 172.21.15.114:39830) 2024-05-27T07:41:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.914+0000 7f83d5d9b700 1 -- 172.21.15.114:0/833325110 learned_addr learned my addr 172.21.15.114:0/833325110 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.914+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/833325110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5576484c9d70 con 0x5576485a2070 2024-05-27T07:41:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.914+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/833325110 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5576484b9910 con 0x55764859c120 2024-05-27T07:41:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.914+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/833325110 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5576484bd950 con 0x55764841ce40 2024-05-27T07:41:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.914+0000 7f83d6d9d700 1 --2- 172.21.15.114:0/833325110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 0x55764859c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.915+0000 7f83d659c700 1 --2- 172.21.15.114:0/833325110 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764841ce40 0x5576485a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.915+0000 7f83d5d9b700 1 -- 172.21.15.114:0/833325110 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764841ce40 msgr2=0x5576485a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.915+0000 7f83d5d9b700 1 --2- 172.21.15.114:0/833325110 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764841ce40 0x5576485a1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.915+0000 7f83d5d9b700 1 -- 172.21.15.114:0/833325110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 msgr2=0x55764859c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.915+0000 7f83d5d9b700 1 --2- 172.21.15.114:0/833325110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 0x55764859c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.915+0000 7f83d5d9b700 1 -- 172.21.15.114:0/833325110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557648462fb0 con 0x5576485a2070 2024-05-27T07:41:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.915+0000 7f83d6d9d700 1 --2- 172.21.15.114:0/833325110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 0x55764859c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.915+0000 7f83d659c700 1 --2- 172.21.15.114:0/833325110 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764841ce40 0x5576485a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.915+0000 7f83d5d9b700 1 --2- 172.21.15.114:0/833325110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x55764859bbe0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f83cc0a69f0 tx=0x7f83cc0aa210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941188b14606bb3c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.915+0000 7f83c4ff9700 1 -- 172.21.15.114:0/833325110 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc0aa970 con 0x5576485a2070 2024-05-27T07:41:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.915+0000 7f83c4ff9700 1 -- 172.21.15.114:0/833325110 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83cc0aed20 con 0x5576485a2070 2024-05-27T07:41:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.915+0000 7f83c4ff9700 1 -- 172.21.15.114:0/833325110 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc022940 con 0x5576485a2070 2024-05-27T07:41:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.917+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/833325110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 msgr2=0x55764859bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.917+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/833325110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x55764859bbe0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f83cc0a69f0 tx=0x7f83cc0aa210 comp rx=0 tx=0).stop 2024-05-27T07:41:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.917+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/833325110 shutdown_connections 2024-05-27T07:41:35.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.917+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/833325110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 0x55764859c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.917+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/833325110 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764841ce40 0x5576485a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.917+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/833325110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x55764859bbe0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.917+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/833325110 >> 172.21.15.114:0/833325110 conn(0x55764841f770 msgr2=0x557648596d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:35.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.917+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/833325110 shutdown_connections 2024-05-27T07:41:35.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.917+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/833325110 wait complete. 2024-05-27T07:41:35.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.918+0000 7f83eb7b7b80 1 -- start start 2024-05-27T07:41:35.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.918+0000 7f83eb7b7b80 1 Processor -- start 2024-05-27T07:41:35.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.919+0000 7f83eb7b7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55764841ce40 0x5576485a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.919+0000 7f83eb7b7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 0x5576485a5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.919+0000 7f83eb7b7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576485a2070 0x5576485a6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.919+0000 7f83eb7b7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5576483bffa0 con 0x55764841ce40 2024-05-27T07:41:35.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.919+0000 7f83eb7b7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557648421da0 con 0x55764859c120 2024-05-27T07:41:35.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.919+0000 7f83eb7b7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5576485a8920 con 0x5576485a2070 2024-05-27T07:41:35.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.919+0000 7f83d659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55764841ce40 0x5576485a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.919+0000 7f83d659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55764841ce40 0x5576485a5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39836/0 (socket says 172.21.15.114:39836) 2024-05-27T07:41:35.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.919+0000 7f83d659c700 1 -- 172.21.15.114:0/1457514945 learned_addr learned my addr 172.21.15.114:0/1457514945 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:35.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.919+0000 7f83eb7b7b80 10 client.0 _ll_register_callbacks cb 0x5576484505b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.919+0000 7f83d5d9b700 1 --2- 172.21.15.114:0/1457514945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 0x5576485a5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.919+0000 7f83d6d9d700 1 --2- 172.21.15.114:0/1457514945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576485a2070 0x5576485a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.920+0000 7f83d659c700 1 -- 172.21.15.114:0/1457514945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576485a2070 msgr2=0x5576485a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.920+0000 7f83d659c700 1 --2- 172.21.15.114:0/1457514945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576485a2070 0x5576485a6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.920+0000 7f83d659c700 1 -- 172.21.15.114:0/1457514945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 msgr2=0x5576485a5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.920+0000 7f83d659c700 1 --2- 172.21.15.114:0/1457514945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 0x5576485a5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.920+0000 7f83d659c700 1 -- 172.21.15.114:0/1457514945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576484098d0 con 0x55764841ce40 2024-05-27T07:41:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.920+0000 7f83d659c700 1 --2- 172.21.15.114:0/1457514945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55764841ce40 0x5576485a5a10 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f83c8012a20 tx=0x7f83c80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb7739ac331d504 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.920+0000 7f83bdffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:41:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.920+0000 7f83bdffb700 1 -- 172.21.15.114:0/1457514945 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8004a30 con 0x55764841ce40 2024-05-27T07:41:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.920+0000 7f83bdffb700 1 -- 172.21.15.114:0/1457514945 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f83c8004bd0 con 0x55764841ce40 2024-05-27T07:41:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.920+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/1457514945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83cc0aed20 con 0x55764841ce40 2024-05-27T07:41:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.921+0000 7f83bdffb700 1 -- 172.21.15.114:0/1457514945 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f83c8004dd0 con 0x55764841ce40 2024-05-27T07:41:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.921+0000 7f83eb7b7b80 10 client.15630 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.921+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/1457514945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557648473760 con 0x55764841ce40 2024-05-27T07:41:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.921+0000 7f83bdffb700 1 -- 172.21.15.114:0/1457514945 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83cc0aed20 con 0x55764841ce40 2024-05-27T07:41:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.921+0000 7f83bdffb700 1 -- 172.21.15.114:0/1457514945 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f83c80053b0 con 0x55764841ce40 2024-05-27T07:41:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.921+0000 7f83eb7b7b80 10 client.15630 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.921+0000 7f83eb7b7b80 10 client.15630 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.921+0000 7f83eb7b7b80 20 client.15630 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.921+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/1457514945 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576485b5850 0x5576485b7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.921+0000 7f83eb7b7b80 4 client.15630 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.921+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/1457514945 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576484cbcc0 con 0x5576485b5850 2024-05-27T07:41:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.922+0000 7f83d5d9b700 1 --2- 172.21.15.114:0/1457514945 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576485b5850 0x5576485b7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.922+0000 7f83bdffb700 10 client.15630 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.923+0000 7f83d5d9b700 1 --2- 172.21.15.114:0/1457514945 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576485b5850 0x5576485b7cb0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13fb1cf8c66a2120 server_cookie=ad0eb38c0c23492c in_seq=0 out_seq=0 2024-05-27T07:41:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.923+0000 7f83bdffb700 1 -- 172.21.15.114:0/1457514945 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5576484cbcc0 con 0x5576485b5850 2024-05-27T07:41:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.923+0000 7f83bdffb700 10 client.15630 handle_command_reply: tid=0 2024-05-27T07:41:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.923+0000 7f83eb7b7b80 1 client.15630 shutdown 2024-05-27T07:41:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.924+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/1457514945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55764841ce40 msgr2=0x5576485a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.924+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/1457514945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55764841ce40 0x5576485a5a10 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f83c8012a20 tx=0x7f83c80047c0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.924+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/1457514945 shutdown_connections 2024-05-27T07:41:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.924+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/1457514945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576485a2070 0x5576485a6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.924+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/1457514945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55764841ce40 0x5576485a5a10 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.924+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/1457514945 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576485b5850 0x5576485b7cb0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.924+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/1457514945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 0x5576485a5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.924+0000 7f83bdffb700 0 client.15630 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.924+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/1457514945 >> 172.21.15.114:0/1457514945 conn(0x55764841f770 msgr2=0x55764859abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.924+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/1457514945 shutdown_connections 2024-05-27T07:41:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.924+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/1457514945 wait complete. 2024-05-27T07:41:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.924+0000 7f83eb7b7b80 20 client.15630 trim_cache size 0 max 16384 2024-05-27T07:41:35.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.954+0000 7f83eb7b7b80 1 Processor -- start 2024-05-27T07:41:35.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.954+0000 7f83eb7b7b80 1 -- start start 2024-05-27T07:41:35.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.954+0000 7f83eb7b7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764841ce40 0x55764859bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.954+0000 7f83eb7b7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764859c120 0x5576485a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.954+0000 7f83eb7b7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x5576485b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.954+0000 7f83eb7b7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83d81ae540 con 0x5576485a2070 2024-05-27T07:41:35.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.955+0000 7f83eb7b7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f83d81b0dc0 con 0x55764841ce40 2024-05-27T07:41:35.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.955+0000 7f83eb7b7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83d81aebb0 con 0x55764859c120 2024-05-27T07:41:35.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.955+0000 7f83d6d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x5576485b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.955+0000 7f83d6d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x5576485b18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39852/0 (socket says 172.21.15.114:39852) 2024-05-27T07:41:35.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.955+0000 7f83d6d9d700 1 -- 172.21.15.114:0/3957328168 learned_addr learned my addr 172.21.15.114:0/3957328168 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:35.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.955+0000 7f83d659c700 1 --2- 172.21.15.114:0/3957328168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764841ce40 0x55764859bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.955+0000 7f83d6d9d700 1 -- 172.21.15.114:0/3957328168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764859c120 msgr2=0x5576485a60e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:41:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.955+0000 7f83d6d9d700 1 --2- 172.21.15.114:0/3957328168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764859c120 0x5576485a60e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.955+0000 7f83d6d9d700 1 -- 172.21.15.114:0/3957328168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764841ce40 msgr2=0x55764859bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.955+0000 7f83d6d9d700 1 --2- 172.21.15.114:0/3957328168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764841ce40 0x55764859bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.955+0000 7f83d6d9d700 1 -- 172.21.15.114:0/3957328168 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576485a3f80 con 0x5576485a2070 2024-05-27T07:41:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.955+0000 7f83d6d9d700 1 --2- 172.21.15.114:0/3957328168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x5576485b18b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009530 tx=0x7f83d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e8e7ed060160cd3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.956+0000 7f83bffff700 1 -- 172.21.15.114:0/3957328168 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d0014560 con 0x5576485a2070 2024-05-27T07:41:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.956+0000 7f83bffff700 1 -- 172.21.15.114:0/3957328168 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83d001a070 con 0x5576485a2070 2024-05-27T07:41:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.956+0000 7f83bffff700 1 -- 172.21.15.114:0/3957328168 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d0015ad0 con 0x5576485a2070 2024-05-27T07:41:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.957+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/3957328168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 msgr2=0x5576485b18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.957+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/3957328168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x5576485b18b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009530 tx=0x7f83d000b040 comp rx=0 tx=0).stop 2024-05-27T07:41:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.957+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/3957328168 shutdown_connections 2024-05-27T07:41:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.957+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/3957328168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x5576485b18b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.957+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/3957328168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764841ce40 0x55764859bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.957+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/3957328168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764859c120 0x5576485a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.957+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/3957328168 >> 172.21.15.114:0/3957328168 conn(0x55764841f770 msgr2=0x5576485a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.957+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/3957328168 shutdown_connections 2024-05-27T07:41:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.958+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/3957328168 wait complete. 2024-05-27T07:41:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.958+0000 7f83eb7b7b80 1 -- start start 2024-05-27T07:41:35.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.958+0000 7f83eb7b7b80 1 Processor -- start 2024-05-27T07:41:35.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83eb7b7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764841ce40 0x5576485adce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83eb7b7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 0x5576485ae220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83eb7b7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x5576485bca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83eb7b7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83d81af280 con 0x5576485a2070 2024-05-27T07:41:35.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83eb7b7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f83d81b06f0 con 0x55764859c120 2024-05-27T07:41:35.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83eb7b7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83d81af950 con 0x55764841ce40 2024-05-27T07:41:35.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83eb7b7b80 10 client.0 _ll_register_callbacks cb 0x5576484505b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83d6d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x5576485bca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83d6d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x5576485bca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39862/0 (socket says 172.21.15.114:39862) 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83d6d9d700 1 -- 172.21.15.114:0/694572287 learned_addr learned my addr 172.21.15.114:0/694572287 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83d659c700 1 --2- 172.21.15.114:0/694572287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764841ce40 0x5576485adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83d6d9d700 1 -- 172.21.15.114:0/694572287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764841ce40 msgr2=0x5576485adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83d6d9d700 1 --2- 172.21.15.114:0/694572287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764841ce40 0x5576485adce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83d6d9d700 1 -- 172.21.15.114:0/694572287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 msgr2=0x5576485ae220 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83d6d9d700 1 --2- 172.21.15.114:0/694572287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 0x5576485ae220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.959+0000 7f83d6d9d700 1 -- 172.21.15.114:0/694572287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576485bd200 con 0x5576485a2070 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.960+0000 7f83d6d9d700 1 --2- 172.21.15.114:0/694572287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x5576485bca60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f83d000b010 tx=0x7f83d0016a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b1deed567f6603 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.960+0000 7f83bdffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.960+0000 7f83bdffb700 1 -- 172.21.15.114:0/694572287 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d0023070 con 0x5576485a2070 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.960+0000 7f83bdffb700 1 -- 172.21.15.114:0/694572287 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f83d0014760 con 0x5576485a2070 2024-05-27T07:41:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.960+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/694572287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83d001a070 con 0x5576485a2070 2024-05-27T07:41:35.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.960+0000 7f83bdffb700 1 -- 172.21.15.114:0/694572287 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f83d0031050 con 0x5576485a2070 2024-05-27T07:41:35.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.960+0000 7f83eb7b7b80 10 client.15636 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:35.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.960+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/694572287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55764861fd20 con 0x5576485a2070 2024-05-27T07:41:35.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.961+0000 7f83bdffb700 1 -- 172.21.15.114:0/694572287 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83d001a070 con 0x5576485a2070 2024-05-27T07:41:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.961+0000 7f83bdffb700 1 -- 172.21.15.114:0/694572287 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f83d001f640 con 0x5576485a2070 2024-05-27T07:41:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.961+0000 7f83eb7b7b80 10 client.15636 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.961+0000 7f83eb7b7b80 10 client.15636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.961+0000 7f83eb7b7b80 20 client.15636 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.961+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/694572287 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576486203a0 0x5576486227c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.961+0000 7f83eb7b7b80 4 client.15636 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:41:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.961+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/694572287 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5576484c54a0 con 0x5576486203a0 2024-05-27T07:41:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.962+0000 7f83d659c700 1 --2- 172.21.15.114:0/694572287 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576486203a0 0x5576486227c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:35.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.962+0000 7f83bdffb700 10 client.15636 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.962+0000 7f83d659c700 1 --2- 172.21.15.114:0/694572287 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576486203a0 0x5576486227c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4bb5a243d4bd4b1 server_cookie=f0ce506baece0ab7 in_seq=0 out_seq=0 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.962+0000 7f83bdffb700 1 -- 172.21.15.114:0/694572287 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5576484c54a0 con 0x5576486203a0 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.962+0000 7f83bdffb700 10 client.15636 handle_command_reply: tid=0 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.962+0000 7f83eb7b7b80 1 client.15636 shutdown 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.962+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/694572287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 msgr2=0x5576485bca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.962+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/694572287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x5576485bca60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f83d000b010 tx=0x7f83d0016a10 comp rx=0 tx=0).stop 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.963+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/694572287 shutdown_connections 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.963+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/694572287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576485a2070 0x5576485bca60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.963+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/694572287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55764841ce40 0x5576485adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.963+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/694572287 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576486203a0 0x5576486227c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.963+0000 7f83eb7b7b80 1 --2- 172.21.15.114:0/694572287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55764859c120 0x5576485ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.963+0000 7f83bdffb700 0 client.15636 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.963+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/694572287 >> 172.21.15.114:0/694572287 conn(0x55764841f770 msgr2=0x5576485904f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:35.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.963+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/694572287 shutdown_connections 2024-05-27T07:41:35.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.963+0000 7f83eb7b7b80 1 -- 172.21.15.114:0/694572287 wait complete. 2024-05-27T07:41:35.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.963+0000 7f83eb7b7b80 20 client.15636 trim_cache size 0 max 16384 2024-05-27T07:41:35.965 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:41:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.964+0000 7f83dcda2700 1 -- 172.21.15.114:0/2957770740 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83c00b2a70 msgr2=0x7f83c00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.964+0000 7f83dcda2700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83c00b2a70 0x7f83c00b4f80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f83c8009680 tx=0x7f83c8009260 comp rx=0 tx=0).stop 2024-05-27T07:41:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.964+0000 7f83dcda2700 1 -- 172.21.15.114:0/2957770740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83d81030d0 msgr2=0x7f83d819b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.964+0000 7f83dcda2700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83d81030d0 0x7f83d819b4a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc00bfd0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.964+0000 7f83dcda2700 1 -- 172.21.15.114:0/2957770740 shutdown_connections 2024-05-27T07:41:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.964+0000 7f83dcda2700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83c00b2a70 0x7f83c00b4f80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.964+0000 7f83dcda2700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83d81092f0 0x7f83d819f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.964+0000 7f83dcda2700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d8102780 0x7f83d819af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.964+0000 7f83dcda2700 1 --2- 172.21.15.114:0/2957770740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83d81030d0 0x7f83d819b4a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.964+0000 7f83dcda2700 1 -- 172.21.15.114:0/2957770740 >> 172.21.15.114:0/2957770740 conn(0x7f83d80fe530 msgr2=0x7f83d8107160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.964+0000 7f83dcda2700 1 -- 172.21.15.114:0/2957770740 shutdown_connections 2024-05-27T07:41:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:35.964+0000 7f83dcda2700 1 -- 172.21.15.114:0/2957770740 wait complete. 2024-05-27T07:41:37.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:37 smithi163 ceph-mon[36451]: pgmap v59: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:37.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:37 smithi186 ceph-mon[36647]: pgmap v59: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:37.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:37 smithi114 ceph-mon[32038]: pgmap v59: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:37.865 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:41:37.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:41:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:39.676 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:39 smithi114 ceph-mon[32038]: pgmap v60: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:39.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:39 smithi163 ceph-mon[36451]: pgmap v60: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:39.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:39 smithi186 ceph-mon[36647]: pgmap v60: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:41.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.385+0000 7f033c84a700 1 -- 172.21.15.114:0/4231175341 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0330012650 con 0x7f0338108740 2024-05-27T07:41:41.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.386+0000 7f033fab0700 1 -- 172.21.15.114:0/4231175341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0338108740 msgr2=0x7f03381100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.386+0000 7f033fab0700 1 --2- 172.21.15.114:0/4231175341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0338108740 0x7f03381100a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f033000b0d0 tx=0x7f033000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.384 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:41 smithi114 ceph-mon[32038]: pgmap v61: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:41.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.386+0000 7f033fab0700 1 -- 172.21.15.114:0/4231175341 shutdown_connections 2024-05-27T07:41:41.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.386+0000 7f033fab0700 1 --2- 172.21.15.114:0/4231175341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0338108740 0x7f03381100a0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.386+0000 7f033fab0700 1 --2- 172.21.15.114:0/4231175341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0338103bb0 0x7f0338103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.386+0000 7f033fab0700 1 --2- 172.21.15.114:0/4231175341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0338107df0 0x7f0338108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.386+0000 7f033fab0700 1 -- 172.21.15.114:0/4231175341 >> 172.21.15.114:0/4231175341 conn(0x7f03380fe6e0 msgr2=0x7f0338100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:41.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.386+0000 7f033fab0700 1 -- 172.21.15.114:0/4231175341 shutdown_connections 2024-05-27T07:41:41.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.386+0000 7f033fab0700 1 -- 172.21.15.114:0/4231175341 wait complete. 2024-05-27T07:41:41.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.387+0000 7f033fab0700 1 Processor -- start 2024-05-27T07:41:41.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.387+0000 7f033fab0700 1 -- start start 2024-05-27T07:41:41.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.388+0000 7f033fab0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0338103bb0 0x7f033819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.388+0000 7f033fab0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0338107df0 0x7f033819fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.388+0000 7f033fab0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0338108740 0x7f0338199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.388+0000 7f033fab0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0338113370 con 0x7f0338103bb0 2024-05-27T07:41:41.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.388+0000 7f033fab0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f03381131f0 con 0x7f0338107df0 2024-05-27T07:41:41.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.388+0000 7f033fab0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f03381134f0 con 0x7f0338108740 2024-05-27T07:41:41.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.388+0000 7f033d84c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0338103bb0 0x7f033819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.388+0000 7f033d84c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0338103bb0 0x7f033819f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39886/0 (socket says 172.21.15.114:39886) 2024-05-27T07:41:41.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.388+0000 7f033d84c700 1 -- 172.21.15.114:0/1144970900 learned_addr learned my addr 172.21.15.114:0/1144970900 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:41.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.389+0000 7f033e04d700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0338108740 0x7f0338199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.389+0000 7f033d04b700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0338107df0 0x7f033819fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.389+0000 7f033d84c700 1 -- 172.21.15.114:0/1144970900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0338108740 msgr2=0x7f0338199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.389+0000 7f033d84c700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0338108740 0x7f0338199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.389+0000 7f033d84c700 1 -- 172.21.15.114:0/1144970900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0338107df0 msgr2=0x7f033819fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.389+0000 7f033d84c700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0338107df0 0x7f033819fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.389+0000 7f033d84c700 1 -- 172.21.15.114:0/1144970900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f033819a120 con 0x7f0338103bb0 2024-05-27T07:41:41.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.389+0000 7f033d04b700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0338107df0 0x7f033819fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:41.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.389+0000 7f033d84c700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0338103bb0 0x7f033819f850 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f033400bf20 tx=0x7f033400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:41.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.390+0000 7f032effd700 1 -- 172.21.15.114:0/1144970900 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0334010a90 con 0x7f0338103bb0 2024-05-27T07:41:41.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.390+0000 7f032effd700 1 -- 172.21.15.114:0/1144970900 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0334010c30 con 0x7f0338103bb0 2024-05-27T07:41:41.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.390+0000 7f033fab0700 1 -- 172.21.15.114:0/1144970900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f033819a410 con 0x7f0338103bb0 2024-05-27T07:41:41.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.390+0000 7f032effd700 1 -- 172.21.15.114:0/1144970900 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03340096f0 con 0x7f0338103bb0 2024-05-27T07:41:41.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.390+0000 7f033fab0700 1 -- 172.21.15.114:0/1144970900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0338102810 con 0x7f0338103bb0 2024-05-27T07:41:41.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.394+0000 7f032effd700 1 -- 172.21.15.114:0/1144970900 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f0334009890 con 0x7f0338103bb0 2024-05-27T07:41:41.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.394+0000 7f034e4c5b80 1 Processor -- start 2024-05-27T07:41:41.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.395+0000 7f034e4c5b80 1 -- start start 2024-05-27T07:41:41.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.395+0000 7f034e4c5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19db21ec0 0x55b19dcaf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.395+0000 7f034e4c5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 0x55b19dca9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.395+0000 7f034e4c5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19dca99b0 0x55b19dca9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.395+0000 7f033e04d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19db21ec0 0x55b19dcaf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.395+0000 7f033e04d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19db21ec0 0x55b19dcaf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40124/0 (socket says 172.21.15.114:40124) 2024-05-27T07:41:41.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.395+0000 7f033d84c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 0x55b19dca9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.395+0000 7f033d84c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 0x55b19dca9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:60174/0 (socket says 172.21.15.114:60174) 2024-05-27T07:41:41.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.396+0000 7f033e04d700 1 -- 172.21.15.114:0/1272581213 learned_addr learned my addr 172.21.15.114:0/1272581213 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:41.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.396+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1272581213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b19dbcedf0 con 0x55b19dca99b0 2024-05-27T07:41:41.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.396+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1272581213 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b19dbbe990 con 0x55b19dcaf8e0 2024-05-27T07:41:41.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.396+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1272581213 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b19dbc29d0 con 0x55b19db21ec0 2024-05-27T07:41:41.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.396+0000 7f032effd700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03240b2a70 0x7f03240b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.396+0000 7f033d04b700 1 --2- 172.21.15.114:0/1272581213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19dca99b0 0x55b19dca9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.396+0000 7f033e04d700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03240b2a70 0x7f03240b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.396+0000 7f032effd700 1 -- 172.21.15.114:0/1144970900 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f03340e2d80 con 0x7f0338103bb0 2024-05-27T07:41:41.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.396+0000 7f033d04b700 1 -- 172.21.15.114:0/1272581213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19db21ec0 msgr2=0x55b19dcaf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.396+0000 7f033d04b700 1 --2- 172.21.15.114:0/1272581213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19db21ec0 0x55b19dcaf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.396+0000 7f033d04b700 1 -- 172.21.15.114:0/1272581213 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 msgr2=0x55b19dca9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.396+0000 7f033d04b700 1 --2- 172.21.15.114:0/1272581213 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 0x55b19dca9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.396+0000 7f033d04b700 1 -- 172.21.15.114:0/1272581213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b19db68030 con 0x55b19dca99b0 2024-05-27T07:41:41.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.397+0000 7f033d04b700 1 --2- 172.21.15.114:0/1272581213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19dca99b0 0x55b19dca9dc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0338106750 tx=0x7f0328009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5befe2e454a0c8f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:41.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.397+0000 7f033e04d700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03240b2a70 0x7f03240b4f80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x55b19dcaa6e0 tx=0x7f033000b3e0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:41.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.397+0000 7f0323fff700 1 -- 172.21.15.114:0/1272581213 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0328016070 con 0x55b19dca99b0 2024-05-27T07:41:41.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.397+0000 7f0323fff700 1 -- 172.21.15.114:0/1272581213 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0328005020 con 0x55b19dca99b0 2024-05-27T07:41:41.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.397+0000 7f0323fff700 1 -- 172.21.15.114:0/1272581213 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032800eb70 con 0x55b19dca99b0 2024-05-27T07:41:41.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.398+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1272581213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19dca99b0 msgr2=0x55b19dca9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.398+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/1272581213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19dca99b0 0x55b19dca9dc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0338106750 tx=0x7f0328009e40 comp rx=0 tx=0).stop 2024-05-27T07:41:41.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.399+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1272581213 shutdown_connections 2024-05-27T07:41:41.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.399+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/1272581213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19dca99b0 0x55b19dca9dc0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.399+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/1272581213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19db21ec0 0x55b19dcaf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.399+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/1272581213 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 0x55b19dca9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.399+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1272581213 >> 172.21.15.114:0/1272581213 conn(0x55b19db247f0 msgr2=0x55b19dc9be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:41.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.399+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1272581213 shutdown_connections 2024-05-27T07:41:41.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.399+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1272581213 wait complete. 2024-05-27T07:41:41.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.399+0000 7f034e4c5b80 1 -- start start 2024-05-27T07:41:41.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.400+0000 7f034e4c5b80 1 Processor -- start 2024-05-27T07:41:41.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.400+0000 7f034e4c5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19db21ec0 0x55b19dcb3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.400+0000 7f034e4c5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcaf8e0 0x55b19dcb37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.400+0000 7f034e4c5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19dcb3cf0 0x55b19dcbf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.400+0000 7f034e4c5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b19dac5020 con 0x55b19dcb3cf0 2024-05-27T07:41:41.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.400+0000 7f034e4c5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b19db26e20 con 0x55b19db21ec0 2024-05-27T07:41:41.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.400+0000 7f034e4c5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b19dcb6410 con 0x55b19dcaf8e0 2024-05-27T07:41:41.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.401+0000 7f033d84c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19db21ec0 0x55b19dcb3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.401+0000 7f033d84c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19db21ec0 0x55b19dcb3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:60176/0 (socket says 172.21.15.114:60176) 2024-05-27T07:41:41.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.401+0000 7f033d84c700 1 -- 172.21.15.114:0/738624159 learned_addr learned my addr 172.21.15.114:0/738624159 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:41.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.401+0000 7f033d04b700 1 --2- 172.21.15.114:0/738624159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcaf8e0 0x55b19dcb37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.401+0000 7f034e4c5b80 10 client.0 _ll_register_callbacks cb 0x55b19db55630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:41.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.401+0000 7f033d84c700 1 -- 172.21.15.114:0/738624159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcaf8e0 msgr2=0x55b19dcb37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.401+0000 7f033d84c700 1 --2- 172.21.15.114:0/738624159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcaf8e0 0x55b19dcb37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.401+0000 7f033d84c700 1 -- 172.21.15.114:0/738624159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19dcb3cf0 msgr2=0x55b19dcbf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.401+0000 7f033d84c700 1 --2- 172.21.15.114:0/738624159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19dcb3cf0 0x55b19dcbf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.401+0000 7f033d84c700 1 -- 172.21.15.114:0/738624159 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b19db0e950 con 0x55b19db21ec0 2024-05-27T07:41:41.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.401+0000 7f033d84c700 1 --2- 172.21.15.114:0/738624159 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19db21ec0 0x55b19dcb3270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0334000c00 tx=0x7f0334015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af64a41eb315cd6a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:41.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.401+0000 7f03217fa700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:41:41.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.402+0000 7f03217fa700 1 -- 172.21.15.114:0/738624159 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0334010400 con 0x55b19db21ec0 2024-05-27T07:41:41.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.402+0000 7f034e4c5b80 1 -- 172.21.15.114:0/738624159 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0328005020 con 0x55b19db21ec0 2024-05-27T07:41:41.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.402+0000 7f03217fa700 1 -- 172.21.15.114:0/738624159 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f0334009d90 con 0x55b19db21ec0 2024-05-27T07:41:41.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.402+0000 7f03217fa700 1 -- 172.21.15.114:0/738624159 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f03340a98d0 con 0x55b19db21ec0 2024-05-27T07:41:41.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.402+0000 7f034e4c5b80 10 client.25202 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:41.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.402+0000 7f034e4c5b80 1 -- 172.21.15.114:0/738624159 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b19db787e0 con 0x55b19db21ec0 2024-05-27T07:41:41.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.402+0000 7f03217fa700 1 -- 172.21.15.114:0/738624159 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0328005020 con 0x55b19db21ec0 2024-05-27T07:41:41.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.402+0000 7f03217fa700 1 -- 172.21.15.114:0/738624159 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f03340c2030 con 0x55b19db21ec0 2024-05-27T07:41:41.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.403+0000 7f034e4c5b80 10 client.25202 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:41.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.403+0000 7f034e4c5b80 10 client.25202 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:41.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.403+0000 7f034e4c5b80 20 client.25202 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:41.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.403+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/738624159 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b19dcc3dd0 0x55b19dcc6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.403+0000 7f034e4c5b80 4 client.25202 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:41.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.403+0000 7f034e4c5b80 1 -- 172.21.15.114:0/738624159 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b19dbd0d40 con 0x55b19dcc3dd0 2024-05-27T07:41:41.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.403+0000 7f033d04b700 1 --2- 172.21.15.114:0/738624159 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b19dcc3dd0 0x55b19dcc6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.404+0000 7f033d04b700 1 --2- 172.21.15.114:0/738624159 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b19dcc3dd0 0x55b19dcc6290 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a81ba85ebf3cfd server_cookie=93a9643edd459a38 in_seq=0 out_seq=0 2024-05-27T07:41:41.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.404+0000 7f03217fa700 10 client.25202 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:41.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.404+0000 7f03217fa700 1 -- 172.21.15.114:0/738624159 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b19dbd0d40 con 0x55b19dcc3dd0 2024-05-27T07:41:41.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.404+0000 7f03217fa700 10 client.25202 handle_command_reply: tid=0 2024-05-27T07:41:41.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.404+0000 7f034e4c5b80 1 client.25202 shutdown 2024-05-27T07:41:41.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.405+0000 7f034e4c5b80 1 -- 172.21.15.114:0/738624159 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19db21ec0 msgr2=0x55b19dcb3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.405+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/738624159 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19db21ec0 0x55b19dcb3270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0334000c00 tx=0x7f0334015000 comp rx=0 tx=0).stop 2024-05-27T07:41:41.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.405+0000 7f034e4c5b80 1 -- 172.21.15.114:0/738624159 shutdown_connections 2024-05-27T07:41:41.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.405+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/738624159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19dcb3cf0 0x55b19dcbf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.405+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/738624159 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b19dcc3dd0 0x55b19dcc6290 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.405+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/738624159 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19db21ec0 0x55b19dcb3270 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.405+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/738624159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcaf8e0 0x55b19dcb37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.405+0000 7f034e4c5b80 1 -- 172.21.15.114:0/738624159 >> 172.21.15.114:0/738624159 conn(0x55b19db247f0 msgr2=0x55b19dc9fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:41.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.405+0000 7f03217fa700 0 client.25202 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:41.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.405+0000 7f034e4c5b80 1 -- 172.21.15.114:0/738624159 shutdown_connections 2024-05-27T07:41:41.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.405+0000 7f034e4c5b80 1 -- 172.21.15.114:0/738624159 wait complete. 2024-05-27T07:41:41.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.405+0000 7f034e4c5b80 20 client.25202 trim_cache size 0 max 16384 2024-05-27T07:41:41.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.426+0000 7f034e4c5b80 1 Processor -- start 2024-05-27T07:41:41.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.426+0000 7f034e4c5b80 1 -- start start 2024-05-27T07:41:41.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.427+0000 7f034e4c5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 0x55b19dcbf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.427+0000 7f034e4c5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 0x55b19dc95c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.427+0000 7f034e4c5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcb3cf0 0x55b19dccba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.427+0000 7f034e4c5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f03381ac4a0 con 0x55b19db21ec0 2024-05-27T07:41:41.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.427+0000 7f034e4c5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f03381aed20 con 0x55b19dcaf8e0 2024-05-27T07:41:41.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.427+0000 7f034e4c5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f03381acb10 con 0x55b19dcb3cf0 2024-05-27T07:41:41.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.427+0000 7f033d84c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 0x55b19dcbf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.427+0000 7f033d84c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 0x55b19dcbf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39906/0 (socket says 172.21.15.114:39906) 2024-05-27T07:41:41.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.427+0000 7f033d84c700 1 -- 172.21.15.114:0/4051496624 learned_addr learned my addr 172.21.15.114:0/4051496624 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:41.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.427+0000 7f033d04b700 1 --2- 172.21.15.114:0/4051496624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 0x55b19dc95c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.428+0000 7f033e04d700 1 --2- 172.21.15.114:0/4051496624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcb3cf0 0x55b19dccba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.428+0000 7f033d84c700 1 -- 172.21.15.114:0/4051496624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcb3cf0 msgr2=0x55b19dccba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.428+0000 7f033d84c700 1 --2- 172.21.15.114:0/4051496624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcb3cf0 0x55b19dccba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.428+0000 7f033d84c700 1 -- 172.21.15.114:0/4051496624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 msgr2=0x55b19dc95c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.428+0000 7f033d84c700 1 --2- 172.21.15.114:0/4051496624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 0x55b19dc95c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.428+0000 7f033d84c700 1 -- 172.21.15.114:0/4051496624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b19dcc3bb0 con 0x55b19db21ec0 2024-05-27T07:41:41.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.428+0000 7f033d84c700 1 --2- 172.21.15.114:0/4051496624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 0x55b19dcbf470 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f03340a9a70 tx=0x7f03340e30e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08e06bcb6832a47 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:41.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.429+0000 7f03237fe700 1 -- 172.21.15.114:0/4051496624 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03340a5a80 con 0x55b19db21ec0 2024-05-27T07:41:41.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.429+0000 7f03237fe700 1 -- 172.21.15.114:0/4051496624 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03340aa4c0 con 0x55b19db21ec0 2024-05-27T07:41:41.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.429+0000 7f03237fe700 1 -- 172.21.15.114:0/4051496624 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03340a8a90 con 0x55b19db21ec0 2024-05-27T07:41:41.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.430+0000 7f034e4c5b80 1 -- 172.21.15.114:0/4051496624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 msgr2=0x55b19dcbf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.430+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/4051496624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 0x55b19dcbf470 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f03340a9a70 tx=0x7f03340e30e0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.430+0000 7f034e4c5b80 1 -- 172.21.15.114:0/4051496624 shutdown_connections 2024-05-27T07:41:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.430+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/4051496624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcb3cf0 0x55b19dccba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.430+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/4051496624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 0x55b19dcbf470 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.430+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/4051496624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 0x55b19dc95c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.430+0000 7f034e4c5b80 1 -- 172.21.15.114:0/4051496624 >> 172.21.15.114:0/4051496624 conn(0x55b19db247f0 msgr2=0x55b19dc95570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.430+0000 7f034e4c5b80 1 -- 172.21.15.114:0/4051496624 shutdown_connections 2024-05-27T07:41:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.430+0000 7f034e4c5b80 1 -- 172.21.15.114:0/4051496624 wait complete. 2024-05-27T07:41:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.431+0000 7f034e4c5b80 1 -- start start 2024-05-27T07:41:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.431+0000 7f034e4c5b80 1 Processor -- start 2024-05-27T07:41:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.432+0000 7f034e4c5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 0x55b19dcbb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.432+0000 7f033d84c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 0x55b19dcbb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.432+0000 7f033d84c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 0x55b19dcbb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39914/0 (socket says 172.21.15.114:39914) 2024-05-27T07:41:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.432+0000 7f033d84c700 1 -- 172.21.15.114:0/1369989010 learned_addr learned my addr 172.21.15.114:0/1369989010 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.432+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 0x55b19dcbbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.432+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcb3cf0 0x55b19dcb8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.432+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1369989010 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f03381ad1e0 con 0x55b19db21ec0 2024-05-27T07:41:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.433+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1369989010 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f03381ae650 con 0x55b19dcaf8e0 2024-05-27T07:41:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.433+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1369989010 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f03381ad8b0 con 0x55b19dcb3cf0 2024-05-27T07:41:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.433+0000 7f033d04b700 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 0x55b19dcbbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.433+0000 7f033e04d700 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcb3cf0 0x55b19dcb8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.433+0000 7f033e04d700 1 -- 172.21.15.114:0/1369989010 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 msgr2=0x55b19dcbbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.433+0000 7f033e04d700 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 0x55b19dcbbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.433+0000 7f033e04d700 1 -- 172.21.15.114:0/1369989010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 msgr2=0x55b19dcbb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.433+0000 7f033e04d700 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 0x55b19dcbb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.433+0000 7f033e04d700 1 -- 172.21.15.114:0/1369989010 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b19dcbc130 con 0x55b19dcb3cf0 2024-05-27T07:41:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.433+0000 7f034e4c5b80 10 client.0 _ll_register_callbacks cb 0x55b19db55630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.433+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1369989010 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03340aa4c0 con 0x55b19dcb3cf0 2024-05-27T07:41:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.434+0000 7f033d84c700 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 0x55b19dcbb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.434+0000 7f033e04d700 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcb3cf0 0x55b19dcb8630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0330009e60 tx=0x7f03300099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8061a232123ac7af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.434+0000 7f03217fa700 10 client.25252 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:41:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.434+0000 7f03217fa700 1 -- 172.21.15.114:0/1369989010 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03300077c0 con 0x55b19dcb3cf0 2024-05-27T07:41:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.434+0000 7f03217fa700 1 -- 172.21.15.114:0/1369989010 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f0330007960 con 0x55b19dcb3cf0 2024-05-27T07:41:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.434+0000 7f03217fa700 1 -- 172.21.15.114:0/1369989010 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0330007bc0 con 0x55b19dcb3cf0 2024-05-27T07:41:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.434+0000 7f034e4c5b80 10 client.25252 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.434+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1369989010 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55b19dd2ec10 con 0x55b19dcb3cf0 2024-05-27T07:41:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.435+0000 7f03217fa700 1 -- 172.21.15.114:0/1369989010 <== mon.1 v2:172.21.15.186:3300/0 4 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0330011070 con 0x55b19dcb3cf0 2024-05-27T07:41:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.435+0000 7f034e4c5b80 10 client.25252 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.435+0000 7f034e4c5b80 10 client.25252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.435+0000 7f034e4c5b80 20 client.25252 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.435+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b19dd2f3b0 0x55b19dd31850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.435+0000 7f034e4c5b80 4 client.25252 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:41:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.435+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1369989010 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b19dbca520 con 0x55b19dd2f3b0 2024-05-27T07:41:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.435+0000 7f033d84c700 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b19dd2f3b0 0x55b19dd31850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.436+0000 7f033d84c700 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b19dd2f3b0 0x55b19dd31850 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8354fe2c74daee2 server_cookie=c50f191e303af74b in_seq=0 out_seq=0 2024-05-27T07:41:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.436+0000 7f03217fa700 10 client.25252 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.437+0000 7f03217fa700 1 -- 172.21.15.114:0/1369989010 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b19dbca520 con 0x55b19dd2f3b0 2024-05-27T07:41:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.437+0000 7f03217fa700 10 client.25252 handle_command_reply: tid=0 2024-05-27T07:41:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.437+0000 7f034e4c5b80 1 client.25252 shutdown 2024-05-27T07:41:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.437+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1369989010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcb3cf0 msgr2=0x55b19dcb8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.437+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcb3cf0 0x55b19dcb8630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0330009e60 tx=0x7f03300099e0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.438+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1369989010 shutdown_connections 2024-05-27T07:41:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.438+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b19dcb3cf0 0x55b19dcb8630 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.438+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b19db21ec0 0x55b19dcbb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.438+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b19dd2f3b0 0x55b19dd31850 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.438+0000 7f034e4c5b80 1 --2- 172.21.15.114:0/1369989010 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b19dcaf8e0 0x55b19dcbbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.438+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1369989010 >> 172.21.15.114:0/1369989010 conn(0x55b19db247f0 msgr2=0x55b19dca03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.438+0000 7f03217fa700 0 client.25252 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.438+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1369989010 shutdown_connections 2024-05-27T07:41:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.438+0000 7f034e4c5b80 1 -- 172.21.15.114:0/1369989010 wait complete. 2024-05-27T07:41:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.438+0000 7f034e4c5b80 20 client.25252 trim_cache size 0 max 16384 2024-05-27T07:41:41.437 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:41:41.438 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:41:41.438 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "79e39209-38ec-4d20-8972-4b02aa737ebb", 2024-05-27T07:41:41.438 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:41:41.438 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:41:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.440+0000 7f033fab0700 1 -- 172.21.15.114:0/1144970900 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03240b2a70 msgr2=0x7f03240b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.440+0000 7f033fab0700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03240b2a70 0x7f03240b4f80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x55b19dcaa6e0 tx=0x7f033000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.440+0000 7f033fab0700 1 -- 172.21.15.114:0/1144970900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0338103bb0 msgr2=0x7f033819f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.440+0000 7f033fab0700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0338103bb0 0x7f033819f850 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f033400bf20 tx=0x7f033400dc30 comp rx=0 tx=0).stop 2024-05-27T07:41:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.440+0000 7f033fab0700 1 -- 172.21.15.114:0/1144970900 shutdown_connections 2024-05-27T07:41:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.440+0000 7f033fab0700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0338108740 0x7f0338199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.440+0000 7f033fab0700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0338103bb0 0x7f033819f850 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.440+0000 7f033fab0700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03240b2a70 0x7f03240b4f80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.440+0000 7f033fab0700 1 --2- 172.21.15.114:0/1144970900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0338107df0 0x7f033819fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.440+0000 7f033fab0700 1 -- 172.21.15.114:0/1144970900 >> 172.21.15.114:0/1144970900 conn(0x7f03380fe6e0 msgr2=0x7f0338107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.441+0000 7f033fab0700 1 -- 172.21.15.114:0/1144970900 shutdown_connections 2024-05-27T07:41:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:41.441+0000 7f033fab0700 1 -- 172.21.15.114:0/1144970900 wait complete. 2024-05-27T07:41:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.531+0000 7f16da2f0640 1 -- 172.21.15.114:0/3853612578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d4104d90 msgr2=0x7f16d41051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.531+0000 7f16da2f0640 1 --2- 172.21.15.114:0/3853612578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d4104d90 0x7f16d41051a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f16c4009860 tx=0x7f16c402a340 comp rx=0 tx=0).stop 2024-05-27T07:41:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.532+0000 7f16da2f0640 1 -- 172.21.15.114:0/3853612578 shutdown_connections 2024-05-27T07:41:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.532+0000 7f16da2f0640 1 --2- 172.21.15.114:0/3853612578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16d410e1e0 0x7f16d41106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.532+0000 7f16da2f0640 1 --2- 172.21.15.114:0/3853612578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d410b7f0 0x7f16d410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.532+0000 7f16da2f0640 1 --2- 172.21.15.114:0/3853612578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d4104d90 0x7f16d41051a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.532+0000 7f16da2f0640 1 -- 172.21.15.114:0/3853612578 >> 172.21.15.114:0/3853612578 conn(0x7f16d40782d0 msgr2=0x7f16d4078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.532+0000 7f16da2f0640 1 -- 172.21.15.114:0/3853612578 shutdown_connections 2024-05-27T07:41:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.532+0000 7f16da2f0640 1 -- 172.21.15.114:0/3853612578 wait complete. 2024-05-27T07:41:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.533+0000 7f16da2f0640 1 Processor -- start 2024-05-27T07:41:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.533+0000 7f16da2f0640 1 -- start start 2024-05-27T07:41:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16da2f0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d4104d90 0x7f16d41a0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16da2f0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16d410b7f0 0x7f16d41a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16da2f0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d410e1e0 0x7f16d419a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16da2f0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f16d4113e40 con 0x7f16d4104d90 2024-05-27T07:41:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16da2f0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f16d4113cc0 con 0x7f16d410b7f0 2024-05-27T07:41:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16da2f0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f16d4113fc0 con 0x7f16d410e1e0 2024-05-27T07:41:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16d3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d4104d90 0x7f16d41a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16d888d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d410e1e0 0x7f16d419a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16d888d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d410e1e0 0x7f16d419a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40158/0 (socket says 172.21.15.114:40158) 2024-05-27T07:41:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16d888d640 1 -- 172.21.15.114:0/2273045719 learned_addr learned my addr 172.21.15.114:0/2273045719 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16d3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d4104d90 0x7f16d41a0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39932/0 (socket says 172.21.15.114:39932) 2024-05-27T07:41:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16d37fe640 1 --2- 172.21.15.114:0/2273045719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16d410b7f0 0x7f16d41a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16d888d640 1 -- 172.21.15.114:0/2273045719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16d410b7f0 msgr2=0x7f16d41a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16d888d640 1 --2- 172.21.15.114:0/2273045719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16d410b7f0 0x7f16d41a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16d888d640 1 -- 172.21.15.114:0/2273045719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d4104d90 msgr2=0x7f16d41a0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16d888d640 1 --2- 172.21.15.114:0/2273045719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d4104d90 0x7f16d41a0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.534+0000 7f16d888d640 1 -- 172.21.15.114:0/2273045719 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d419adf0 con 0x7f16d410e1e0 2024-05-27T07:41:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.535+0000 7f16d888d640 1 --2- 172.21.15.114:0/2273045719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d410e1e0 0x7f16d419a6c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00eb10 tx=0x7f16cc00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.535+0000 7f16d17fa640 1 -- 172.21.15.114:0/2273045719 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16cc004b20 con 0x7f16d410e1e0 2024-05-27T07:41:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.535+0000 7f16d17fa640 1 -- 172.21.15.114:0/2273045719 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16cc004cc0 con 0x7f16d410e1e0 2024-05-27T07:41:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.535+0000 7f16d17fa640 1 -- 172.21.15.114:0/2273045719 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16cc010670 con 0x7f16d410e1e0 2024-05-27T07:41:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.535+0000 7f16da2f0640 1 -- 172.21.15.114:0/2273045719 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d419b080 con 0x7f16d410e1e0 2024-05-27T07:41:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.535+0000 7f16da2f0640 1 -- 172.21.15.114:0/2273045719 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d41a72e0 con 0x7f16d410e1e0 2024-05-27T07:41:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.539+0000 7f16d17fa640 1 -- 172.21.15.114:0/2273045719 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f16cc005020 con 0x7f16d410e1e0 2024-05-27T07:41:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.539+0000 7f16e11a0740 1 Processor -- start 2024-05-27T07:41:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.539+0000 7f16e11a0740 1 -- start start 2024-05-27T07:41:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.539+0000 7f16e11a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa44c1b0 0x55c1fa7fc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.539+0000 7f16e11a0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa4232f0 0x55c1fa7fc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.539+0000 7f16e11a0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 0x55c1fa7fb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.539+0000 7f16e11a0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1fa75fc30 con 0x55c1fa42b4d0 2024-05-27T07:41:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.539+0000 7f16e11a0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c1fa74f7d0 con 0x55c1fa4232f0 2024-05-27T07:41:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.539+0000 7f16e11a0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c1fa753810 con 0x55c1fa44c1b0 2024-05-27T07:41:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.539+0000 7f16d37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 0x55c1fa7fb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.539+0000 7f16d37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 0x55c1fa7fb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39948/0 (socket says 172.21.15.114:39948) 2024-05-27T07:41:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.539+0000 7f16d37fe640 1 -- 172.21.15.114:0/3409024978 learned_addr learned my addr 172.21.15.114:0/3409024978 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.540+0000 7f16d3fff640 1 --2- 172.21.15.114:0/3409024978 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa4232f0 0x55c1fa7fc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.540+0000 7f16d37fe640 1 --2- 172.21.15.114:0/3409024978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa44c1b0 0x55c1fa7fc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.540+0000 7f16d17fa640 1 --2- 172.21.15.114:0/2273045719 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f16bc0b29d0 0x7f16bc0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.540+0000 7f16d888d640 1 --2- 172.21.15.114:0/2273045719 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f16bc0b29d0 0x7f16bc0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.540+0000 7f16d17fa640 1 -- 172.21.15.114:0/2273045719 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f16cc0e2e50 con 0x7f16d410e1e0 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.540+0000 7f16d37fe640 1 -- 172.21.15.114:0/3409024978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa44c1b0 msgr2=0x55c1fa7fc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.540+0000 7f16d37fe640 1 --2- 172.21.15.114:0/3409024978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa44c1b0 0x55c1fa7fc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.540+0000 7f16d37fe640 1 -- 172.21.15.114:0/3409024978 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa4232f0 msgr2=0x55c1fa7fc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.540+0000 7f16d37fe640 1 --2- 172.21.15.114:0/3409024978 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa4232f0 0x55c1fa7fc980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.540+0000 7f16d37fe640 1 -- 172.21.15.114:0/3409024978 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1fa464210 con 0x55c1fa42b4d0 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.540+0000 7f16d37fe640 1 --2- 172.21.15.114:0/3409024978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 0x55c1fa7fb0e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009820 tx=0x7f16c80097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91dbb63897c3f6fd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.541+0000 7f16ba7fc640 1 -- 172.21.15.114:0/3409024978 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8005020 con 0x55c1fa42b4d0 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.541+0000 7f16ba7fc640 1 -- 172.21.15.114:0/3409024978 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16c80047a0 con 0x55c1fa42b4d0 2024-05-27T07:41:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.541+0000 7f16ba7fc640 1 -- 172.21.15.114:0/3409024978 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c801d630 con 0x55c1fa42b4d0 2024-05-27T07:41:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.542+0000 7f16e11a0740 1 -- 172.21.15.114:0/3409024978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 msgr2=0x55c1fa7fb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.542+0000 7f16e11a0740 1 --2- 172.21.15.114:0/3409024978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 0x55c1fa7fb0e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009820 tx=0x7f16c80097b0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.542+0000 7f16d888d640 1 --2- 172.21.15.114:0/2273045719 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f16bc0b29d0 0x7f16bc0b4ee0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0e02a0 tx=0x7f16cc01bac0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.542+0000 7f16e11a0740 1 -- 172.21.15.114:0/3409024978 shutdown_connections 2024-05-27T07:41:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.542+0000 7f16e11a0740 1 --2- 172.21.15.114:0/3409024978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 0x55c1fa7fb0e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.542+0000 7f16e11a0740 1 --2- 172.21.15.114:0/3409024978 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa4232f0 0x55c1fa7fc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.542+0000 7f16e11a0740 1 --2- 172.21.15.114:0/3409024978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa44c1b0 0x55c1fa7fc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.542+0000 7f16e11a0740 1 -- 172.21.15.114:0/3409024978 >> 172.21.15.114:0/3409024978 conn(0x55c1fa440300 msgr2=0x55c1fa7e9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.542+0000 7f16e11a0740 1 -- 172.21.15.114:0/3409024978 shutdown_connections 2024-05-27T07:41:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.542+0000 7f16e11a0740 1 -- 172.21.15.114:0/3409024978 wait complete. 2024-05-27T07:41:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.543+0000 7f16e11a0740 1 -- start start 2024-05-27T07:41:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.543+0000 7f16e11a0740 1 Processor -- start 2024-05-27T07:41:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.543+0000 7f16e11a0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa4232f0 0x55c1fa8062e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.543+0000 7f16e11a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa42b4d0 0x55c1fa806820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.543+0000 7f16e11a0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa44c1b0 0x55c1fa8006d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16e11a0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1fa435790 con 0x55c1fa44c1b0 2024-05-27T07:41:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16e11a0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c1fa70b890 con 0x55c1fa4232f0 2024-05-27T07:41:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16e11a0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c1fa7fbb10 con 0x55c1fa42b4d0 2024-05-27T07:41:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16e11a0740 10 client.0 _ll_register_callbacks cb 0x55c1fa463a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16d3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa42b4d0 0x55c1fa806820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16d3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa42b4d0 0x55c1fa806820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40168/0 (socket says 172.21.15.114:40168) 2024-05-27T07:41:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16d3fff640 1 -- 172.21.15.114:0/1010323596 learned_addr learned my addr 172.21.15.114:0/1010323596 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16d37fe640 1 --2- 172.21.15.114:0/1010323596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa44c1b0 0x55c1fa8006d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16d3fff640 1 -- 172.21.15.114:0/1010323596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa4232f0 msgr2=0x55c1fa8062e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16d3fff640 1 --2- 172.21.15.114:0/1010323596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa4232f0 0x55c1fa8062e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16d3fff640 1 -- 172.21.15.114:0/1010323596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa44c1b0 msgr2=0x55c1fa8006d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16d3fff640 1 --2- 172.21.15.114:0/1010323596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa44c1b0 0x55c1fa8006d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16d3fff640 1 -- 172.21.15.114:0/1010323596 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1fa530620 con 0x55c1fa42b4d0 2024-05-27T07:41:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.544+0000 7f16d37fe640 1 --2- 172.21.15.114:0/1010323596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa44c1b0 0x55c1fa8006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.545+0000 7f16d3fff640 1 --2- 172.21.15.114:0/1010323596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa42b4d0 0x55c1fa806820 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f16c4009860 tx=0x7f16c4030d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=975986a40fc93e32 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.545+0000 7f16b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:41:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.545+0000 7f16b37fe640 1 -- 172.21.15.114:0/1010323596 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4042070 con 0x55c1fa42b4d0 2024-05-27T07:41:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.545+0000 7f16e11a0740 1 -- 172.21.15.114:0/1010323596 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16c80047a0 con 0x55c1fa42b4d0 2024-05-27T07:41:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.545+0000 7f16b37fe640 1 -- 172.21.15.114:0/1010323596 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f16c4004240 con 0x55c1fa42b4d0 2024-05-27T07:41:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.545+0000 7f16b37fe640 1 -- 172.21.15.114:0/1010323596 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f16c404e050 con 0x55c1fa42b4d0 2024-05-27T07:41:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.545+0000 7f16e11a0740 10 client.25264 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.545+0000 7f16e11a0740 1 -- 172.21.15.114:0/1010323596 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c1fa80f0d0 con 0x55c1fa42b4d0 2024-05-27T07:41:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.546+0000 7f16b37fe640 1 -- 172.21.15.114:0/1010323596 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16c80047a0 con 0x55c1fa42b4d0 2024-05-27T07:41:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.546+0000 7f16b37fe640 1 -- 172.21.15.114:0/1010323596 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f16c4003a30 con 0x55c1fa42b4d0 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.546+0000 7f16e11a0740 10 client.25264 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.546+0000 7f16e11a0740 10 client.25264 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.546+0000 7f16e11a0740 20 client.25264 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.546+0000 7f16e11a0740 1 --2- 172.21.15.114:0/1010323596 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c1fa40ad80 0x55c1fa812ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.546+0000 7f16e11a0740 4 client.25264 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.546+0000 7f16e11a0740 1 -- 172.21.15.114:0/1010323596 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1fa761b80 con 0x55c1fa40ad80 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.547+0000 7f16d37fe640 1 --2- 172.21.15.114:0/1010323596 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c1fa40ad80 0x55c1fa812ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.547+0000 7f16b37fe640 10 client.25264 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.547+0000 7f16d37fe640 1 --2- 172.21.15.114:0/1010323596 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c1fa40ad80 0x55c1fa812ac0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22d5657d449ffad server_cookie=61525869c50d3711 in_seq=0 out_seq=0 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.548+0000 7f16b37fe640 1 -- 172.21.15.114:0/1010323596 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c1fa761b80 con 0x55c1fa40ad80 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.548+0000 7f16b37fe640 10 client.25264 handle_command_reply: tid=0 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.548+0000 7f16e11a0740 1 client.25264 shutdown 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.548+0000 7f16e11a0740 1 -- 172.21.15.114:0/1010323596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa42b4d0 msgr2=0x55c1fa806820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.548+0000 7f16e11a0740 1 --2- 172.21.15.114:0/1010323596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa42b4d0 0x55c1fa806820 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f16c4009860 tx=0x7f16c4030d80 comp rx=0 tx=0).stop 2024-05-27T07:41:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.548+0000 7f16e11a0740 1 -- 172.21.15.114:0/1010323596 shutdown_connections 2024-05-27T07:41:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.548+0000 7f16e11a0740 1 --2- 172.21.15.114:0/1010323596 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c1fa40ad80 0x55c1fa812ac0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.548+0000 7f16e11a0740 1 --2- 172.21.15.114:0/1010323596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa44c1b0 0x55c1fa8006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.548+0000 7f16b37fe640 0 client.25264 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.548+0000 7f16e11a0740 1 --2- 172.21.15.114:0/1010323596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa42b4d0 0x55c1fa806820 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.548+0000 7f16e11a0740 1 --2- 172.21.15.114:0/1010323596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa4232f0 0x55c1fa8062e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.549+0000 7f16e11a0740 1 -- 172.21.15.114:0/1010323596 >> 172.21.15.114:0/1010323596 conn(0x55c1fa440300 msgr2=0x55c1fa7e9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.549+0000 7f16e11a0740 1 -- 172.21.15.114:0/1010323596 shutdown_connections 2024-05-27T07:41:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.549+0000 7f16e11a0740 1 -- 172.21.15.114:0/1010323596 wait complete. 2024-05-27T07:41:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.549+0000 7f16e11a0740 20 client.25264 trim_cache size 0 max 16384 2024-05-27T07:41:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.578+0000 7f16e11a0740 1 Processor -- start 2024-05-27T07:41:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.578+0000 7f16e11a0740 1 -- start start 2024-05-27T07:41:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.579+0000 7f16e11a0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa40ad80 0x55c1fa7fe080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.579+0000 7f16e11a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa4232f0 0x55c1fa7f9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.579+0000 7f16e11a0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 0x55c1fa8062e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.579+0000 7f16e11a0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f16d41ad0c0 con 0x55c1fa42b4d0 2024-05-27T07:41:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.579+0000 7f16e11a0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f16d41af940 con 0x55c1fa40ad80 2024-05-27T07:41:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.579+0000 7f16e11a0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f16d41ad730 con 0x55c1fa4232f0 2024-05-27T07:41:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.579+0000 7f16d37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 0x55c1fa8062e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.579+0000 7f16d3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa4232f0 0x55c1fa7f9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.579+0000 7f16d37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 0x55c1fa8062e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39972/0 (socket says 172.21.15.114:39972) 2024-05-27T07:41:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.579+0000 7f16d3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa4232f0 0x55c1fa7f9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40172/0 (socket says 172.21.15.114:40172) 2024-05-27T07:41:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.579+0000 7f16d37fe640 1 -- 172.21.15.114:0/3647179024 learned_addr learned my addr 172.21.15.114:0/3647179024 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:41.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.580+0000 7f16d3fff640 1 -- 172.21.15.114:0/3647179024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa40ad80 msgr2=0x55c1fa7fe080 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:41:41.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.580+0000 7f16d3fff640 1 --2- 172.21.15.114:0/3647179024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa40ad80 0x55c1fa7fe080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.580+0000 7f16d3fff640 1 -- 172.21.15.114:0/3647179024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 msgr2=0x55c1fa8062e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.580+0000 7f16d3fff640 1 --2- 172.21.15.114:0/3647179024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 0x55c1fa8062e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.580+0000 7f16d3fff640 1 -- 172.21.15.114:0/3647179024 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1fa7ecdf0 con 0x55c1fa4232f0 2024-05-27T07:41:41.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.580+0000 7f16d37fe640 1 --2- 172.21.15.114:0/3647179024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 0x55c1fa8062e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:41.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.580+0000 7f16d3fff640 1 --2- 172.21.15.114:0/3647179024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa4232f0 0x55c1fa7f9050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f16c4005940 tx=0x7f16c402aa40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5ac862d8c8b7d51 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:41.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.580+0000 7f16b9ffb640 1 -- 172.21.15.114:0/3647179024 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4042040 con 0x55c1fa4232f0 2024-05-27T07:41:41.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.580+0000 7f16b9ffb640 1 -- 172.21.15.114:0/3647179024 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16c4005d30 con 0x55c1fa4232f0 2024-05-27T07:41:41.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.580+0000 7f16b9ffb640 1 -- 172.21.15.114:0/3647179024 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4037040 con 0x55c1fa4232f0 2024-05-27T07:41:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.581+0000 7f16e11a0740 1 -- 172.21.15.114:0/3647179024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa4232f0 msgr2=0x55c1fa7f9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.581+0000 7f16e11a0740 1 --2- 172.21.15.114:0/3647179024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa4232f0 0x55c1fa7f9050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f16c4005940 tx=0x7f16c402aa40 comp rx=0 tx=0).stop 2024-05-27T07:41:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.582+0000 7f16e11a0740 1 -- 172.21.15.114:0/3647179024 shutdown_connections 2024-05-27T07:41:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.582+0000 7f16e11a0740 1 --2- 172.21.15.114:0/3647179024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa42b4d0 0x55c1fa8062e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.582+0000 7f16e11a0740 1 --2- 172.21.15.114:0/3647179024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa4232f0 0x55c1fa7f9050 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.582+0000 7f16e11a0740 1 --2- 172.21.15.114:0/3647179024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa40ad80 0x55c1fa7fe080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.582+0000 7f16e11a0740 1 -- 172.21.15.114:0/3647179024 >> 172.21.15.114:0/3647179024 conn(0x55c1fa440300 msgr2=0x55c1fa7fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.582+0000 7f16e11a0740 1 -- 172.21.15.114:0/3647179024 shutdown_connections 2024-05-27T07:41:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.582+0000 7f16e11a0740 1 -- 172.21.15.114:0/3647179024 wait complete. 2024-05-27T07:41:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.582+0000 7f16e11a0740 1 -- start start 2024-05-27T07:41:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.582+0000 7f16e11a0740 1 Processor -- start 2024-05-27T07:41:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.583+0000 7f16e11a0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa40ad80 0x55c1fa80c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.583+0000 7f16e11a0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa4232f0 0x55c1fa80ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.583+0000 7f16e11a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa42b4d0 0x55c1fa808ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.583+0000 7f16d3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa4232f0 0x55c1fa80ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.583+0000 7f16d3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa4232f0 0x55c1fa80ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39988/0 (socket says 172.21.15.114:39988) 2024-05-27T07:41:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.583+0000 7f16d3fff640 1 -- 172.21.15.114:0/950909347 learned_addr learned my addr 172.21.15.114:0/950909347 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.583+0000 7f16e11a0740 1 -- 172.21.15.114:0/950909347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f16d41ade00 con 0x55c1fa4232f0 2024-05-27T07:41:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.583+0000 7f16d37fe640 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa40ad80 0x55c1fa80c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.583+0000 7f16e11a0740 1 -- 172.21.15.114:0/950909347 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f16d41af270 con 0x55c1fa40ad80 2024-05-27T07:41:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.583+0000 7f16e11a0740 1 -- 172.21.15.114:0/950909347 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f16d41ae4d0 con 0x55c1fa42b4d0 2024-05-27T07:41:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.584+0000 7f16d37fe640 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa42b4d0 0x55c1fa808ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.584+0000 7f16d37fe640 1 -- 172.21.15.114:0/950909347 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa42b4d0 msgr2=0x55c1fa808ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.584+0000 7f16d37fe640 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa42b4d0 0x55c1fa808ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.584+0000 7f16d37fe640 1 -- 172.21.15.114:0/950909347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa4232f0 msgr2=0x55c1fa80ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.584+0000 7f16d37fe640 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa4232f0 0x55c1fa80ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.584+0000 7f16d37fe640 1 -- 172.21.15.114:0/950909347 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1fa809760 con 0x55c1fa40ad80 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.584+0000 7f16d3fff640 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa4232f0 0x55c1fa80ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.584+0000 7f16e11a0740 10 client.0 _ll_register_callbacks cb 0x7f16cc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.584+0000 7f16e11a0740 1 -- 172.21.15.114:0/950909347 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16c4005d30 con 0x55c1fa40ad80 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.584+0000 7f16d37fe640 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa40ad80 0x55c1fa80c780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f16c80057c0 tx=0x7f16c8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bba59e77fe12a2a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.584+0000 7f16b37fe640 10 client.25220 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.584+0000 7f16b37fe640 1 -- 172.21.15.114:0/950909347 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8005350 con 0x55c1fa40ad80 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.584+0000 7f16b37fe640 1 -- 172.21.15.114:0/950909347 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f16c80054f0 con 0x55c1fa40ad80 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.585+0000 7f16b37fe640 1 -- 172.21.15.114:0/950909347 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f16c802a2c0 con 0x55c1fa40ad80 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.585+0000 7f16e11a0740 10 client.25220 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.585+0000 7f16e11a0740 1 -- 172.21.15.114:0/950909347 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55c1fa879aa0 con 0x55c1fa40ad80 2024-05-27T07:41:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.585+0000 7f16b37fe640 1 -- 172.21.15.114:0/950909347 <== mon.2 v2:172.21.15.163:3300/0 4 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f16c802a460 con 0x55c1fa40ad80 2024-05-27T07:41:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.585+0000 7f16e11a0740 10 client.25220 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.585+0000 7f16e11a0740 10 client.25220 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.585+0000 7f16e11a0740 20 client.25220 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.585+0000 7f16e11a0740 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c1fa44c1b0 0x55c1fa87c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.585+0000 7f16e11a0740 4 client.25220 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:41:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.585+0000 7f16e11a0740 1 -- 172.21.15.114:0/950909347 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c1fa75b360 con 0x55c1fa44c1b0 2024-05-27T07:41:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.585+0000 7f16d3fff640 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c1fa44c1b0 0x55c1fa87c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.586+0000 7f16b37fe640 10 client.25220 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.586+0000 7f16d3fff640 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c1fa44c1b0 0x55c1fa87c000 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3080717dbf619c5 server_cookie=913699136a1f9aa2 in_seq=0 out_seq=0 2024-05-27T07:41:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.590+0000 7f16b37fe640 1 -- 172.21.15.114:0/950909347 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55c1fa75b360 con 0x55c1fa44c1b0 2024-05-27T07:41:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.590+0000 7f16b37fe640 10 client.25220 handle_command_reply: tid=0 2024-05-27T07:41:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.590+0000 7f16e11a0740 1 client.25220 shutdown 2024-05-27T07:41:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.590+0000 7f16e11a0740 1 -- 172.21.15.114:0/950909347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa40ad80 msgr2=0x55c1fa80c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.590+0000 7f16e11a0740 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa40ad80 0x55c1fa80c780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f16c80057c0 tx=0x7f16c8005fd0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.590+0000 7f16e11a0740 1 -- 172.21.15.114:0/950909347 shutdown_connections 2024-05-27T07:41:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.590+0000 7f16e11a0740 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c1fa44c1b0 0x55c1fa87c000 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.590+0000 7f16e11a0740 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1fa42b4d0 0x55c1fa808ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.590+0000 7f16e11a0740 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1fa4232f0 0x55c1fa80ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.590+0000 7f16e11a0740 1 --2- 172.21.15.114:0/950909347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1fa40ad80 0x55c1fa80c780 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.590+0000 7f16e11a0740 1 -- 172.21.15.114:0/950909347 >> 172.21.15.114:0/950909347 conn(0x55c1fa440300 msgr2=0x55c1fa7f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.590+0000 7f16b37fe640 0 client.25220 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.591+0000 7f16e11a0740 1 -- 172.21.15.114:0/950909347 shutdown_connections 2024-05-27T07:41:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.591+0000 7f16e11a0740 1 -- 172.21.15.114:0/950909347 wait complete. 2024-05-27T07:41:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.591+0000 7f16e11a0740 20 client.25220 trim_cache size 0 max 16384 2024-05-27T07:41:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:41:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:41:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:41:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.591+0000 7f16da2f0640 1 -- 172.21.15.114:0/2273045719 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f16bc0b29d0 msgr2=0x7f16bc0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.591+0000 7f16da2f0640 1 --2- 172.21.15.114:0/2273045719 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f16bc0b29d0 0x7f16bc0b4ee0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0e02a0 tx=0x7f16cc01bac0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.591+0000 7f16da2f0640 1 -- 172.21.15.114:0/2273045719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d410e1e0 msgr2=0x7f16d419a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.591+0000 7f16da2f0640 1 --2- 172.21.15.114:0/2273045719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d410e1e0 0x7f16d419a6c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00eb10 tx=0x7f16cc00c2d0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.592+0000 7f16da2f0640 1 -- 172.21.15.114:0/2273045719 shutdown_connections 2024-05-27T07:41:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.592+0000 7f16da2f0640 1 --2- 172.21.15.114:0/2273045719 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f16bc0b29d0 0x7f16bc0b4ee0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.592+0000 7f16da2f0640 1 --2- 172.21.15.114:0/2273045719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d410e1e0 0x7f16d419a6c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.592+0000 7f16da2f0640 1 --2- 172.21.15.114:0/2273045719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16d410b7f0 0x7f16d41a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.592+0000 7f16da2f0640 1 --2- 172.21.15.114:0/2273045719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d4104d90 0x7f16d41a0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.592+0000 7f16da2f0640 1 -- 172.21.15.114:0/2273045719 >> 172.21.15.114:0/2273045719 conn(0x7f16d40782d0 msgr2=0x7f16d41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.592+0000 7f16da2f0640 1 -- 172.21.15.114:0/2273045719 shutdown_connections 2024-05-27T07:41:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:41.592+0000 7f16da2f0640 1 -- 172.21.15.114:0/2273045719 wait complete. 2024-05-27T07:41:41.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:41 smithi163 ceph-mon[36451]: pgmap v61: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:41.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:41 smithi186 ceph-mon[36647]: pgmap v61: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-27T07:41:42.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:41:42.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:42 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:41:42.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:42 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:42.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:42 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:41:42.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:42 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:42.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:42 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:41:42.568 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:42.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:42 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:41:42.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:42 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:42.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:42 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:41:42.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:42 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:42.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:42 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:41:42.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:42 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:41:42.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:42 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:42.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:42 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:41:42.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:42 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:42.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:42 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:41:43.723 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:43 smithi114 ceph-mon[32038]: pgmap v62: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:41:43.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:43 smithi163 ceph-mon[36451]: pgmap v62: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:41:43.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:43 smithi186 ceph-mon[36647]: pgmap v62: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:41:44.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.666+0000 7f11f145f700 1 -- 172.21.15.114:0/640435463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ec1045e0 msgr2=0x7f11ec1049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:44.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.666+0000 7f11f145f700 1 --2- 172.21.15.114:0/640435463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ec1045e0 0x7f11ec1049f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f11e00099c0 tx=0x7f11e0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.666+0000 7f11f145f700 1 -- 172.21.15.114:0/640435463 shutdown_connections 2024-05-27T07:41:44.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.666+0000 7f11f145f700 1 --2- 172.21.15.114:0/640435463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11ec109860 0x7f11ec1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.666+0000 7f11f145f700 1 --2- 172.21.15.114:0/640435463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ec1045e0 0x7f11ec1049f0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.666+0000 7f11f145f700 1 --2- 172.21.15.114:0/640435463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11ec108f10 0x7f11ec109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.666+0000 7f11f145f700 1 -- 172.21.15.114:0/640435463 >> 172.21.15.114:0/640435463 conn(0x7f11ec100030 msgr2=0x7f11ec102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:44.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.666+0000 7f11f145f700 1 -- 172.21.15.114:0/640435463 shutdown_connections 2024-05-27T07:41:44.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.667+0000 7f11f145f700 1 -- 172.21.15.114:0/640435463 wait complete. 2024-05-27T07:41:44.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.667+0000 7f11f145f700 1 Processor -- start 2024-05-27T07:41:44.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.667+0000 7f11f145f700 1 -- start start 2024-05-27T07:41:44.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11f145f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11ec1045e0 0x7f11ec075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11f145f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ec108f10 0x7f11ec075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11f145f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11ec109860 0x7f11ec076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11f145f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec114510 con 0x7f11ec108f10 2024-05-27T07:41:44.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11f145f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec114390 con 0x7f11ec109860 2024-05-27T07:41:44.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11f145f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec114690 con 0x7f11ec1045e0 2024-05-27T07:41:44.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11ea7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ec108f10 0x7f11ec075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11ea7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ec108f10 0x7f11ec075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47618/0 (socket says 172.21.15.114:47618) 2024-05-27T07:41:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11ea7fc700 1 -- 172.21.15.114:0/3837050289 learned_addr learned my addr 172.21.15.114:0/3837050289 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11eb7fe700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11ec109860 0x7f11ec076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11eaffd700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11ec1045e0 0x7f11ec075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11ea7fc700 1 -- 172.21.15.114:0/3837050289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11ec1045e0 msgr2=0x7f11ec075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11ec1045e0 0x7f11ec075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11ea7fc700 1 -- 172.21.15.114:0/3837050289 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11ec109860 msgr2=0x7f11ec076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11ec109860 0x7f11ec076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11ea7fc700 1 -- 172.21.15.114:0/3837050289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec1a9200 con 0x7f11ec108f10 2024-05-27T07:41:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11eaffd700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11ec1045e0 0x7f11ec075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.668+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ec108f10 0x7f11ec075ff0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00c8f0 tx=0x7f11dc00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.669+0000 7f11d3fff700 1 -- 172.21.15.114:0/3837050289 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc007940 con 0x7f11ec108f10 2024-05-27T07:41:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.669+0000 7f11f145f700 1 -- 172.21.15.114:0/3837050289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11ec1a94f0 con 0x7f11ec108f10 2024-05-27T07:41:44.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.669+0000 7f11d3fff700 1 -- 172.21.15.114:0/3837050289 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11dc00f450 con 0x7f11ec108f10 2024-05-27T07:41:44.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.669+0000 7f11d3fff700 1 -- 172.21.15.114:0/3837050289 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc018660 con 0x7f11ec108f10 2024-05-27T07:41:44.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.669+0000 7f11f145f700 1 -- 172.21.15.114:0/3837050289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11ec1a9a80 con 0x7f11ec108f10 2024-05-27T07:41:44.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.672+0000 7f11d3fff700 1 -- 172.21.15.114:0/3837050289 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f11dc018800 con 0x7f11ec108f10 2024-05-27T07:41:44.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.672+0000 7f11d3fff700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f11d40b29f0 0x7f11d40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11eaffd700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f11d40b29f0 0x7f11d40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11d3fff700 1 -- 172.21.15.114:0/3837050289 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f11dc0e28b0 con 0x7f11ec108f10 2024-05-27T07:41:44.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11ffe74b80 1 Processor -- start 2024-05-27T07:41:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11ffe74b80 1 -- start start 2024-05-27T07:41:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11ffe74b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9501e40 0x560df9686b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11ffe74b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 0x560df9680be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11ffe74b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9681120 0x560df9681530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11eaffd700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f11d40b29f0 0x7f11d40b4f00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009fd0 tx=0x7f11e001a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11ffe74b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560df95aed70 con 0x560df9687070 2024-05-27T07:41:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11ffe74b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560df959e910 con 0x560df9501e40 2024-05-27T07:41:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11ffe74b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560df95a2950 con 0x560df9681120 2024-05-27T07:41:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11ea7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 0x560df9680be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11ea7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 0x560df9680be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47622/0 (socket says 172.21.15.114:47622) 2024-05-27T07:41:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11ea7fc700 1 -- 172.21.15.114:0/3039612812 learned_addr learned my addr 172.21.15.114:0/3039612812 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.673+0000 7f11eb7fe700 1 --2- 172.21.15.114:0/3039612812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9681120 0x560df9681530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.674+0000 7f11eaffd700 1 --2- 172.21.15.114:0/3039612812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9501e40 0x560df9686b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.674+0000 7f11ea7fc700 1 -- 172.21.15.114:0/3039612812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9681120 msgr2=0x560df9681530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.674+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/3039612812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9681120 0x560df9681530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.674+0000 7f11ea7fc700 1 -- 172.21.15.114:0/3039612812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9501e40 msgr2=0x560df9686b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.674+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/3039612812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9501e40 0x560df9686b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.674+0000 7f11ea7fc700 1 -- 172.21.15.114:0/3039612812 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560df9547fb0 con 0x560df9687070 2024-05-27T07:41:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.674+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/3039612812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 0x560df9680be0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f11dc0a5860 tx=0x7f11dc0a9260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa787d850f132a80 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.674+0000 7f11d17fa700 1 -- 172.21.15.114:0/3039612812 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc0b2640 con 0x560df9687070 2024-05-27T07:41:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.674+0000 7f11d17fa700 1 -- 172.21.15.114:0/3039612812 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11dc00f600 con 0x560df9687070 2024-05-27T07:41:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.674+0000 7f11d17fa700 1 -- 172.21.15.114:0/3039612812 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc0a9a40 con 0x560df9687070 2024-05-27T07:41:44.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.675+0000 7f11ffe74b80 1 -- 172.21.15.114:0/3039612812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 msgr2=0x560df9680be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:44.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.675+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/3039612812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 0x560df9680be0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f11dc0a5860 tx=0x7f11dc0a9260 comp rx=0 tx=0).stop 2024-05-27T07:41:44.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.675+0000 7f11ffe74b80 1 -- 172.21.15.114:0/3039612812 shutdown_connections 2024-05-27T07:41:44.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.675+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/3039612812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9681120 0x560df9681530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.675+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/3039612812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9501e40 0x560df9686b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.675+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/3039612812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 0x560df9680be0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.675+0000 7f11ffe74b80 1 -- 172.21.15.114:0/3039612812 >> 172.21.15.114:0/3039612812 conn(0x560df9504770 msgr2=0x560df967bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.675+0000 7f11ffe74b80 1 -- 172.21.15.114:0/3039612812 shutdown_connections 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.675+0000 7f11ffe74b80 1 -- 172.21.15.114:0/3039612812 wait complete. 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11ffe74b80 1 -- start start 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11ffe74b80 1 Processor -- start 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11ffe74b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df968a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11ffe74b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9681120 0x560df968aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11ffe74b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 0x560df968b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11ffe74b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560df94a4fa0 con 0x560df9687070 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11ffe74b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560df9506da0 con 0x560df9681120 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11ffe74b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560df968db60 con 0x560df9501e40 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11ffe74b80 10 client.0 _ll_register_callbacks cb 0x560df95355b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11eb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 0x560df968b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11eb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 0x560df968b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47634/0 (socket says 172.21.15.114:47634) 2024-05-27T07:41:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11eb7fe700 1 -- 172.21.15.114:0/149735036 learned_addr learned my addr 172.21.15.114:0/149735036 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9681120 0x560df968aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11eaffd700 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df968a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11eb7fe700 1 -- 172.21.15.114:0/149735036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 msgr2=0x560df968a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11eb7fe700 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df968a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11eb7fe700 1 -- 172.21.15.114:0/149735036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9681120 msgr2=0x560df968aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11eb7fe700 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9681120 0x560df968aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.676+0000 7f11eb7fe700 1 -- 172.21.15.114:0/149735036 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560df94ee8d0 con 0x560df9687070 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.677+0000 7f11eaffd700 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df968a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.677+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9681120 0x560df968aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.677+0000 7f11eb7fe700 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 0x560df968b420 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f11e400ab40 tx=0x7f11e400af00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec430a6199b4a92a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.677+0000 7f11ceff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.677+0000 7f11ceff5700 1 -- 172.21.15.114:0/149735036 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e401a070 con 0x560df9687070 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.677+0000 7f11ffe74b80 1 -- 172.21.15.114:0/149735036 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11dc00f600 con 0x560df9687070 2024-05-27T07:41:44.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.677+0000 7f11ceff5700 1 -- 172.21.15.114:0/149735036 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f11e4011c20 con 0x560df9687070 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.677+0000 7f11ceff5700 1 -- 172.21.15.114:0/149735036 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f11e4026050 con 0x560df9687070 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.677+0000 7f11ffe74b80 10 client.15681 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.677+0000 7f11ffe74b80 1 -- 172.21.15.114:0/149735036 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560df9558760 con 0x560df9687070 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.677+0000 7f11ceff5700 1 -- 172.21.15.114:0/149735036 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11dc00f600 con 0x560df9687070 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.678+0000 7f11ceff5700 1 -- 172.21.15.114:0/149735036 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f11e401e3d0 con 0x560df9687070 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.678+0000 7f11ffe74b80 10 client.15681 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.678+0000 7f11ffe74b80 10 client.15681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.678+0000 7f11ffe74b80 20 client.15681 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.678+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560df969afc0 0x560df969d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.678+0000 7f11ffe74b80 4 client.15681 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.678+0000 7f11ffe74b80 1 -- 172.21.15.114:0/149735036 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560df95b0cc0 con 0x560df969afc0 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.678+0000 7f11eaffd700 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560df969afc0 0x560df969d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.679+0000 7f11ceff5700 10 client.15681 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.679+0000 7f11eaffd700 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560df969afc0 0x560df969d450 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5bec97daa4586a2 server_cookie=434d8b7522c87a74 in_seq=0 out_seq=0 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.679+0000 7f11ceff5700 1 -- 172.21.15.114:0/149735036 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560df95b0cc0 con 0x560df969afc0 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.679+0000 7f11ceff5700 10 client.15681 handle_command_reply: tid=0 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.679+0000 7f11ffe74b80 1 client.15681 shutdown 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.680+0000 7f11ffe74b80 1 -- 172.21.15.114:0/149735036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 msgr2=0x560df968b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.680+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 0x560df968b420 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f11e400ab40 tx=0x7f11e400af00 comp rx=0 tx=0).stop 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.680+0000 7f11ffe74b80 1 -- 172.21.15.114:0/149735036 shutdown_connections 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.680+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9687070 0x560df968b420 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.680+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df968a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.680+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560df969afc0 0x560df969d450 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.680+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/149735036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9681120 0x560df968aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.680+0000 7f11ffe74b80 1 -- 172.21.15.114:0/149735036 >> 172.21.15.114:0/149735036 conn(0x560df9504770 msgr2=0x560df967fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.680+0000 7f11ceff5700 0 client.15681 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.680+0000 7f11ffe74b80 1 -- 172.21.15.114:0/149735036 shutdown_connections 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.680+0000 7f11ffe74b80 1 -- 172.21.15.114:0/149735036 wait complete. 2024-05-27T07:41:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.680+0000 7f11ffe74b80 20 client.15681 trim_cache size 0 max 16384 2024-05-27T07:41:44.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.715+0000 7f11ffe74b80 1 Processor -- start 2024-05-27T07:41:44.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.715+0000 7f11ffe74b80 1 -- start start 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.715+0000 7f11ffe74b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df968b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.715+0000 7f11ffe74b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 0x560df9689290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.715+0000 7f11ffe74b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9687070 0x560df96897d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.715+0000 7f11ffe74b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b0840 con 0x560df9681120 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.715+0000 7f11ffe74b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b30c0 con 0x560df9687070 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.715+0000 7f11ffe74b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b0eb0 con 0x560df9501e40 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.715+0000 7f11ea7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 0x560df9689290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.715+0000 7f11ea7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 0x560df9689290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47636/0 (socket says 172.21.15.114:47636) 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.715+0000 7f11ea7fc700 1 -- 172.21.15.114:0/2460714178 learned_addr learned my addr 172.21.15.114:0/2460714178 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.715+0000 7f11eaffd700 1 --2- 172.21.15.114:0/2460714178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df968b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.715+0000 7f11eb7fe700 1 --2- 172.21.15.114:0/2460714178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9687070 0x560df96897d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.716+0000 7f11ea7fc700 1 -- 172.21.15.114:0/2460714178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 msgr2=0x560df968b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.716+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/2460714178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df968b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.716+0000 7f11ea7fc700 1 -- 172.21.15.114:0/2460714178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9687070 msgr2=0x560df96897d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:44.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.716+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/2460714178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9687070 0x560df96897d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.716+0000 7f11ea7fc700 1 -- 172.21.15.114:0/2460714178 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560df9688f10 con 0x560df9681120 2024-05-27T07:41:44.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.716+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/2460714178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 0x560df9689290 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f11dc0a94e0 tx=0x7f11dc00ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d720ca6677106249 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:44.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.716+0000 7f11d0ff9700 1 -- 172.21.15.114:0/2460714178 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc0b2640 con 0x560df9681120 2024-05-27T07:41:44.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.716+0000 7f11d0ff9700 1 -- 172.21.15.114:0/2460714178 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11dc00f600 con 0x560df9681120 2024-05-27T07:41:44.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.716+0000 7f11d0ff9700 1 -- 172.21.15.114:0/2460714178 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc00fa60 con 0x560df9681120 2024-05-27T07:41:44.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.718+0000 7f11ffe74b80 1 -- 172.21.15.114:0/2460714178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 msgr2=0x560df9689290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:44.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.718+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/2460714178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 0x560df9689290 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f11dc0a94e0 tx=0x7f11dc00ff90 comp rx=0 tx=0).stop 2024-05-27T07:41:44.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.718+0000 7f11ffe74b80 1 -- 172.21.15.114:0/2460714178 shutdown_connections 2024-05-27T07:41:44.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.718+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/2460714178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9687070 0x560df96897d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.718+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/2460714178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df968b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.718+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/2460714178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 0x560df9689290 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.718+0000 7f11ffe74b80 1 -- 172.21.15.114:0/2460714178 >> 172.21.15.114:0/2460714178 conn(0x560df9504770 msgr2=0x560df969cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:44.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.718+0000 7f11ffe74b80 1 -- 172.21.15.114:0/2460714178 shutdown_connections 2024-05-27T07:41:44.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.718+0000 7f11ffe74b80 1 -- 172.21.15.114:0/2460714178 wait complete. 2024-05-27T07:41:44.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.718+0000 7f11ffe74b80 1 -- start start 2024-05-27T07:41:44.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.718+0000 7f11ffe74b80 1 Processor -- start 2024-05-27T07:41:44.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.719+0000 7f11ffe74b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df9692df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.719+0000 7f11ffe74b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 0x560df9693330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.719+0000 7f11ffe74b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9687070 0x560df96a1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.719+0000 7f11ffe74b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b1580 con 0x560df9681120 2024-05-27T07:41:44.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.719+0000 7f11ffe74b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b29f0 con 0x560df9687070 2024-05-27T07:41:44.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.719+0000 7f11ffe74b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b1c50 con 0x560df9501e40 2024-05-27T07:41:44.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.719+0000 7f11ea7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 0x560df9693330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.719+0000 7f11ffe74b80 10 client.0 _ll_register_callbacks cb 0x560df95355b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:44.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.719+0000 7f11ea7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 0x560df9693330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47650/0 (socket says 172.21.15.114:47650) 2024-05-27T07:41:44.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.719+0000 7f11ea7fc700 1 -- 172.21.15.114:0/2310080633 learned_addr learned my addr 172.21.15.114:0/2310080633 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:44.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.719+0000 7f11eaffd700 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df9692df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.719+0000 7f11eb7fe700 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9687070 0x560df96a1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11ea7fc700 1 -- 172.21.15.114:0/2310080633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 msgr2=0x560df9692df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df9692df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11ea7fc700 1 -- 172.21.15.114:0/2310080633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9687070 msgr2=0x560df96a1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9687070 0x560df96a1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11ea7fc700 1 -- 172.21.15.114:0/2310080633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560df96a2180 con 0x560df9681120 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11eb7fe700 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9687070 0x560df96a1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11eaffd700 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df9692df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11ea7fc700 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 0x560df9693330 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f11dc0a9490 tx=0x7f11dc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a678b5bfe3e83fcd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11ceff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11ceff5700 1 -- 172.21.15.114:0/2310080633 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc0c6070 con 0x560df9681120 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11ceff5700 1 -- 172.21.15.114:0/2310080633 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f11dc017360 con 0x560df9681120 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11ffe74b80 1 -- 172.21.15.114:0/2310080633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11dc00f600 con 0x560df9681120 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11ceff5700 1 -- 172.21.15.114:0/2310080633 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f11dc0a56a0 con 0x560df9681120 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11ffe74b80 10 client.15687 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.720+0000 7f11ffe74b80 1 -- 172.21.15.114:0/2310080633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560df9705180 con 0x560df9681120 2024-05-27T07:41:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.721+0000 7f11ceff5700 1 -- 172.21.15.114:0/2310080633 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11dc00f600 con 0x560df9681120 2024-05-27T07:41:44.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.721+0000 7f11ceff5700 1 -- 172.21.15.114:0/2310080633 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f11dc0b21b0 con 0x560df9681120 2024-05-27T07:41:44.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.721+0000 7f11ffe74b80 10 client.15687 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:44.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.721+0000 7f11ffe74b80 10 client.15687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:44.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.721+0000 7f11ffe74b80 20 client.15687 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:44.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.721+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560df9705350 0x560df9707770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:44.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.721+0000 7f11ffe74b80 4 client.15687 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:41:44.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.721+0000 7f11ffe74b80 1 -- 172.21.15.114:0/2310080633 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560df95aa4a0 con 0x560df9705350 2024-05-27T07:41:44.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.721+0000 7f11eaffd700 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560df9705350 0x560df9707770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:44.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ceff5700 10 client.15687 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:44.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11eaffd700 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560df9705350 0x560df9707770 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3af7439b9340c45 server_cookie=8db539adb2e8d18d in_seq=0 out_seq=0 2024-05-27T07:41:44.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ceff5700 1 -- 172.21.15.114:0/2310080633 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560df95aa4a0 con 0x560df9705350 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ceff5700 10 client.15687 handle_command_reply: tid=0 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ffe74b80 1 client.15687 shutdown 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ffe74b80 1 -- 172.21.15.114:0/2310080633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 msgr2=0x560df9693330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 0x560df9693330 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f11dc0a9490 tx=0x7f11dc012000 comp rx=0 tx=0).stop 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ffe74b80 1 -- 172.21.15.114:0/2310080633 shutdown_connections 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560df9705350 0x560df9707770 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560df9687070 0x560df96a1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560df9501e40 0x560df9692df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ffe74b80 1 --2- 172.21.15.114:0/2310080633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560df9681120 0x560df9693330 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ffe74b80 1 -- 172.21.15.114:0/2310080633 >> 172.21.15.114:0/2310080633 conn(0x560df9504770 msgr2=0x560df9689120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ceff5700 0 client.15687 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ffe74b80 1 -- 172.21.15.114:0/2310080633 shutdown_connections 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ffe74b80 1 -- 172.21.15.114:0/2310080633 wait complete. 2024-05-27T07:41:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.722+0000 7f11ffe74b80 20 client.15687 trim_cache size 0 max 16384 2024-05-27T07:41:44.725 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:41:44.725 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:41:44.725 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:41:44.725 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:41:44.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.723+0000 7f11f145f700 1 -- 172.21.15.114:0/3837050289 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f11d40b29f0 msgr2=0x7f11d40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:44.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.723+0000 7f11f145f700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f11d40b29f0 0x7f11d40b4f00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009fd0 tx=0x7f11e001a040 comp rx=0 tx=0).stop 2024-05-27T07:41:44.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.723+0000 7f11f145f700 1 -- 172.21.15.114:0/3837050289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ec108f10 msgr2=0x7f11ec075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:44.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.723+0000 7f11f145f700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ec108f10 0x7f11ec075ff0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00c8f0 tx=0x7f11dc00cc00 comp rx=0 tx=0).stop 2024-05-27T07:41:44.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.723+0000 7f11f145f700 1 -- 172.21.15.114:0/3837050289 shutdown_connections 2024-05-27T07:41:44.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.723+0000 7f11f145f700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f11d40b29f0 0x7f11d40b4f00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.723+0000 7f11f145f700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11ec109860 0x7f11ec076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.723+0000 7f11f145f700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11ec1045e0 0x7f11ec075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.723+0000 7f11f145f700 1 --2- 172.21.15.114:0/3837050289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ec108f10 0x7f11ec075ff0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:44.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.723+0000 7f11f145f700 1 -- 172.21.15.114:0/3837050289 >> 172.21.15.114:0/3837050289 conn(0x7f11ec100030 msgr2=0x7f11ec108220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:44.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.723+0000 7f11f145f700 1 -- 172.21.15.114:0/3837050289 shutdown_connections 2024-05-27T07:41:44.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:44.723+0000 7f11f145f700 1 -- 172.21.15.114:0/3837050289 wait complete. 2024-05-27T07:41:45.228 INFO:tasks.workunit.client.0.smithi114.stderr:Updating files: 92% (11409/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-05-27T07:41:45.510 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:41:45.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:41:45.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:45 smithi163 ceph-mon[36451]: pgmap v63: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:41:45.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:45 smithi186 ceph-mon[36647]: pgmap v63: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:41:45.766 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:45.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:45 smithi114 ceph-mon[32038]: pgmap v63: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-27T07:41:45.815 INFO:tasks.workunit.client.0.smithi114.stderr:Note: switching to '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'. 2024-05-27T07:41:45.815 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-05-27T07:41:45.815 INFO:tasks.workunit.client.0.smithi114.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-27T07:41:45.815 INFO:tasks.workunit.client.0.smithi114.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-27T07:41:45.815 INFO:tasks.workunit.client.0.smithi114.stderr:state without impacting any branches by switching back to a branch. 2024-05-27T07:41:45.815 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-05-27T07:41:45.815 INFO:tasks.workunit.client.0.smithi114.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-27T07:41:45.816 INFO:tasks.workunit.client.0.smithi114.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-27T07:41:45.816 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-05-27T07:41:45.816 INFO:tasks.workunit.client.0.smithi114.stderr: git switch -c 2024-05-27T07:41:45.816 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-05-27T07:41:45.816 INFO:tasks.workunit.client.0.smithi114.stderr:Or undo this operation with: 2024-05-27T07:41:45.816 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-05-27T07:41:45.816 INFO:tasks.workunit.client.0.smithi114.stderr: git switch - 2024-05-27T07:41:45.816 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-05-27T07:41:45.816 INFO:tasks.workunit.client.0.smithi114.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-27T07:41:45.816 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-05-27T07:41:45.816 INFO:tasks.workunit.client.0.smithi114.stderr:HEAD is now at 38f9d900c83 Merge PR #55758 into wip-vshankar-testing-20240521.063558-debug 2024-05-27T07:41:45.822 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-27T07:41:45.887 INFO:tasks.workunit.client.0.smithi114.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-27T07:41:45.889 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T07:41:45.890 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-27T07:41:47.664 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:47 smithi114 ceph-mon[32038]: pgmap v64: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:41:47.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:47 smithi163 ceph-mon[36451]: pgmap v64: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:41:47.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:47 smithi186 ceph-mon[36647]: pgmap v64: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:41:47.853 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-27T07:41:47.978 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-27T07:41:48.018 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T07:41:48.020 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T07:41:48.020 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-27T07:41:48.060 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T07:41:48.063 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T07:41:48.063 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-27T07:41:48.079 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-05-27T07:41:48.081 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-05-27T07:41:48.081 DEBUG:teuthology.orchestra.run.smithi114:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-05-27T07:41:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.088+0000 7f436c871700 1 -- 172.21.15.114:0/3438122488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43681029b0 msgr2=0x7f4368102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.088+0000 7f436c871700 1 --2- 172.21.15.114:0/3438122488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43681029b0 0x7f4368102dc0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f435c00aaa0 tx=0x7f435c00adb0 comp rx=0 tx=0).stop 2024-05-27T07:41:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.088+0000 7f436c871700 1 -- 172.21.15.114:0/3438122488 shutdown_connections 2024-05-27T07:41:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.088+0000 7f436c871700 1 --2- 172.21.15.114:0/3438122488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4368103300 0x7f436810f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.088+0000 7f436c871700 1 --2- 172.21.15.114:0/3438122488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43681089e0 0x7f4368108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.088+0000 7f436c871700 1 --2- 172.21.15.114:0/3438122488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43681029b0 0x7f4368102dc0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.088+0000 7f436c871700 1 -- 172.21.15.114:0/3438122488 >> 172.21.15.114:0/3438122488 conn(0x7f43680fe510 msgr2=0x7f4368100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.088+0000 7f436c871700 1 -- 172.21.15.114:0/3438122488 shutdown_connections 2024-05-27T07:41:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.088+0000 7f436c871700 1 -- 172.21.15.114:0/3438122488 wait complete. 2024-05-27T07:41:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.089+0000 7f436c871700 1 Processor -- start 2024-05-27T07:41:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.089+0000 7f436c871700 1 -- start start 2024-05-27T07:41:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.089+0000 7f436c871700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43681029b0 0x7f436819e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.089+0000 7f436c871700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4368103300 0x7f436819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.089+0000 7f436c871700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43681089e0 0x7f4368198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.089+0000 7f436c871700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f43681122d0 con 0x7f43681029b0 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.089+0000 7f436c871700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4368112150 con 0x7f4368103300 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.089+0000 7f436c871700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4368112450 con 0x7f43681089e0 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.089+0000 7f436659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43681029b0 0x7f436819e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.089+0000 7f436659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43681029b0 0x7f436819e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47664/0 (socket says 172.21.15.114:47664) 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.089+0000 7f436659c700 1 -- 172.21.15.114:0/529111377 learned_addr learned my addr 172.21.15.114:0/529111377 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.089+0000 7f4365d9b700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4368103300 0x7f436819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.089+0000 7f4366d9d700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43681089e0 0x7f4368198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.090+0000 7f436659c700 1 -- 172.21.15.114:0/529111377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43681089e0 msgr2=0x7f4368198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.090+0000 7f436659c700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43681089e0 0x7f4368198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.090+0000 7f436659c700 1 -- 172.21.15.114:0/529111377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4368103300 msgr2=0x7f436819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.090+0000 7f436659c700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4368103300 0x7f436819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.090+0000 7f436659c700 1 -- 172.21.15.114:0/529111377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4368199160 con 0x7f43681029b0 2024-05-27T07:41:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.090+0000 7f436659c700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43681029b0 0x7f436819e890 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f435800c8f0 tx=0x7f435800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.090+0000 7f43577fe700 1 -- 172.21.15.114:0/529111377 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43580078d0 con 0x7f43681029b0 2024-05-27T07:41:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.090+0000 7f436c871700 1 -- 172.21.15.114:0/529111377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4368199450 con 0x7f43681029b0 2024-05-27T07:41:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.090+0000 7f43577fe700 1 -- 172.21.15.114:0/529111377 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f435800f450 con 0x7f43681029b0 2024-05-27T07:41:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.090+0000 7f43577fe700 1 -- 172.21.15.114:0/529111377 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4358018660 con 0x7f43681029b0 2024-05-27T07:41:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.090+0000 7f436c871700 1 -- 172.21.15.114:0/529111377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43681a5720 con 0x7f43681029b0 2024-05-27T07:41:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.091+0000 7f436c871700 1 -- 172.21.15.114:0/529111377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43680512b0 con 0x7f43681029b0 2024-05-27T07:41:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.092+0000 7f43577fe700 1 -- 172.21.15.114:0/529111377 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f435800f900 con 0x7f43681029b0 2024-05-27T07:41:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.093+0000 7f43577fe700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f43500b29f0 0x7f43500b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.093+0000 7f4365d9b700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f43500b29f0 0x7f43500b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.093+0000 7f43577fe700 1 -- 172.21.15.114:0/529111377 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f43580e2790 con 0x7f43681029b0 2024-05-27T07:41:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.094+0000 7f4365d9b700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f43500b29f0 0x7f43500b4f00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f435c00c6d0 tx=0x7f435c005960 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:48.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.098+0000 7f43577fe700 1 -- 172.21.15.114:0/529111377 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4358014030 con 0x7f43681029b0 2024-05-27T07:41:48.120 INFO:tasks.workunit.client.0.smithi114.stderr:+ dbench 1 2024-05-27T07:41:48.130 INFO:tasks.workunit.client.0.smithi114.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-05-27T07:41:48.130 INFO:tasks.workunit.client.0.smithi114.stdout: 2024-05-27T07:41:48.130 INFO:tasks.workunit.client.0.smithi114.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-05-27T07:41:48.135 INFO:tasks.workunit.client.0.smithi114.stdout:0 of 1 processes prepared for launch 0 sec 2024-05-27T07:41:48.235 INFO:tasks.workunit.client.0.smithi114.stdout:1 of 1 processes prepared for launch 0 sec 2024-05-27T07:41:48.236 INFO:tasks.workunit.client.0.smithi114.stdout:releasing clients 2024-05-27T07:41:48.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.367+0000 7f436c871700 1 -- 172.21.15.114:0/529111377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4368199fe0 con 0x7f43681029b0 2024-05-27T07:41:48.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.368+0000 7f43577fe700 1 -- 172.21.15.114:0/529111377 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11827 (secure 0 0 0) 0x7f43580a9dc0 con 0x7f43681029b0 2024-05-27T07:41:48.366 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:41:48.367 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:37:06.645274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:41:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.371+0000 7f436c871700 1 -- 172.21.15.114:0/529111377 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f43500b29f0 msgr2=0x7f43500b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.371+0000 7f436c871700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f43500b29f0 0x7f43500b4f00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f435c00c6d0 tx=0x7f435c005960 comp rx=0 tx=0).stop 2024-05-27T07:41:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.371+0000 7f436c871700 1 -- 172.21.15.114:0/529111377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43681029b0 msgr2=0x7f436819e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.371+0000 7f436c871700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43681029b0 0x7f436819e890 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f435800c8f0 tx=0x7f435800cc00 comp rx=0 tx=0).stop 2024-05-27T07:41:48.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.371+0000 7f436c871700 1 -- 172.21.15.114:0/529111377 shutdown_connections 2024-05-27T07:41:48.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.371+0000 7f436c871700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43681089e0 0x7f4368198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:48.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.371+0000 7f436c871700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43681029b0 0x7f436819e890 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:48.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.371+0000 7f436c871700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f43500b29f0 0x7f43500b4f00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:48.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.371+0000 7f436c871700 1 --2- 172.21.15.114:0/529111377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4368103300 0x7f436819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:48.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.371+0000 7f436c871700 1 -- 172.21.15.114:0/529111377 >> 172.21.15.114:0/529111377 conn(0x7f43680fe510 msgr2=0x7f4368100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:48.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.371+0000 7f436c871700 1 -- 172.21.15.114:0/529111377 shutdown_connections 2024-05-27T07:41:48.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:48.371+0000 7f436c871700 1 -- 172.21.15.114:0/529111377 wait complete. 2024-05-27T07:41:48.370 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:41:48.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:48 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/529111377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:41:48.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:48 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/529111377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:41:48.749 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:48 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/529111377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:41:49.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:41:49.236 INFO:tasks.workunit.client.0.smithi114.stdout: 1 3663 38.72 MB/sec warmup 1 sec latency 8.806 ms 2024-05-27T07:41:49.313 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:49.680 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:49 smithi114 ceph-mon[32038]: pgmap v65: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:41:49.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:49 smithi163 ceph-mon[36451]: pgmap v65: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:41:49.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:49 smithi186 ceph-mon[36647]: pgmap v65: 129 pgs: 129 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-27T07:41:50.235 INFO:tasks.workunit.client.0.smithi114.stdout: 1 6486 26.30 MB/sec warmup 2 sec latency 17.292 ms 2024-05-27T07:41:51.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.087+0000 7fdadc5cf700 1 -- 172.21.15.114:0/3516530992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdad4108730 msgr2=0x7fdad4110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:51.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.087+0000 7fdadc5cf700 1 --2- 172.21.15.114:0/3516530992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdad4108730 0x7fdad4110090 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00b0d0 tx=0x7fdacc00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.088+0000 7fdadc5cf700 1 -- 172.21.15.114:0/3516530992 shutdown_connections 2024-05-27T07:41:51.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.088+0000 7fdadc5cf700 1 --2- 172.21.15.114:0/3516530992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdad4108730 0x7fdad4110090 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.088+0000 7fdadc5cf700 1 --2- 172.21.15.114:0/3516530992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad41034b0 0x7fdad41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.088+0000 7fdadc5cf700 1 --2- 172.21.15.114:0/3516530992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdad4107de0 0x7fdad41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.088+0000 7fdadc5cf700 1 -- 172.21.15.114:0/3516530992 >> 172.21.15.114:0/3516530992 conn(0x7fdad40fedb0 msgr2=0x7fdad41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:51.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.088+0000 7fdadc5cf700 1 -- 172.21.15.114:0/3516530992 shutdown_connections 2024-05-27T07:41:51.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.088+0000 7fdadc5cf700 1 -- 172.21.15.114:0/3516530992 wait complete. 2024-05-27T07:41:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.089+0000 7fdadc5cf700 1 Processor -- start 2024-05-27T07:41:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.089+0000 7fdadc5cf700 1 -- start start 2024-05-27T07:41:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.089+0000 7fdadc5cf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdad41034b0 0x7fdad419bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.089+0000 7fdadc5cf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad4107de0 0x7fdad419c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.089+0000 7fdadc5cf700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdad4108730 0x7fdad41a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.089+0000 7fdadc5cf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdad4113360 con 0x7fdad41034b0 2024-05-27T07:41:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.089+0000 7fdadc5cf700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdad41131e0 con 0x7fdad4108730 2024-05-27T07:41:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.089+0000 7fdadc5cf700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdad41134e0 con 0x7fdad4107de0 2024-05-27T07:41:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.089+0000 7fdada36b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdad41034b0 0x7fdad419bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdada36b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdad41034b0 0x7fdad419bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47696/0 (socket says 172.21.15.114:47696) 2024-05-27T07:41:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdad9b6a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad4107de0 0x7fdad419c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdada36b700 1 -- 172.21.15.114:0/3760342650 learned_addr learned my addr 172.21.15.114:0/3760342650 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdadab6c700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdad4108730 0x7fdad41a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdada36b700 1 -- 172.21.15.114:0/3760342650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad4107de0 msgr2=0x7fdad419c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:51.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdada36b700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad4107de0 0x7fdad419c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdada36b700 1 -- 172.21.15.114:0/3760342650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdad4108730 msgr2=0x7fdad41a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:51.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdada36b700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdad4108730 0x7fdad41a6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdada36b700 1 -- 172.21.15.114:0/3760342650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad419c9d0 con 0x7fdad41034b0 2024-05-27T07:41:51.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdad9b6a700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad4107de0 0x7fdad419c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:51.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdadab6c700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdad4108730 0x7fdad41a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:51.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdada36b700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdad41034b0 0x7fdad419bbe0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdad000e9a0 tx=0x7fdad000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:51.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdacb7fe700 1 -- 172.21.15.114:0/3760342650 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad000ca10 con 0x7fdad41034b0 2024-05-27T07:41:51.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdacb7fe700 1 -- 172.21.15.114:0/3760342650 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdad000cbb0 con 0x7fdad41034b0 2024-05-27T07:41:51.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdadc5cf700 1 -- 172.21.15.114:0/3760342650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad41a72c0 con 0x7fdad41034b0 2024-05-27T07:41:51.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdacb7fe700 1 -- 172.21.15.114:0/3760342650 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0018680 con 0x7fdad41034b0 2024-05-27T07:41:51.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.090+0000 7fdadc5cf700 1 -- 172.21.15.114:0/3760342650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad41a77d0 con 0x7fdad41034b0 2024-05-27T07:41:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.093+0000 7fdacb7fe700 1 -- 172.21.15.114:0/3760342650 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fdad0018820 con 0x7fdad41034b0 2024-05-27T07:41:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.094+0000 7fdacb7fe700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdac00b29f0 0x7fdac00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.094+0000 7fdad9b6a700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdac00b29f0 0x7fdac00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.094+0000 7fdacb7fe700 1 -- 172.21.15.114:0/3760342650 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fdad00e2bd0 con 0x7fdad41034b0 2024-05-27T07:41:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.094+0000 7fdad9b6a700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdac00b29f0 0x7fdac00b4f00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdad4102230 tx=0x7fdac4009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.094+0000 7fdaeafe4b80 1 Processor -- start 2024-05-27T07:41:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdaeafe4b80 1 -- start start 2024-05-27T07:41:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdaeafe4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ac5ae40 0x560a6addfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdaeafe4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ade0070 0x560a6add9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdaeafe4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 0x560a6adda530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdaeafe4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560a6ad07d70 con 0x560a6adda120 2024-05-27T07:41:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdaeafe4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560a6acf7910 con 0x560a6ade0070 2024-05-27T07:41:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdaeafe4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560a6acfb950 con 0x560a6ac5ae40 2024-05-27T07:41:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdadab6c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 0x560a6adda530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdadab6c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 0x560a6adda530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47702/0 (socket says 172.21.15.114:47702) 2024-05-27T07:41:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdadab6c700 1 -- 172.21.15.114:0/3871755756 learned_addr learned my addr 172.21.15.114:0/3871755756 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdada36b700 1 --2- 172.21.15.114:0/3871755756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ac5ae40 0x560a6addfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdad9b6a700 1 --2- 172.21.15.114:0/3871755756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ade0070 0x560a6add9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdadab6c700 1 -- 172.21.15.114:0/3871755756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ac5ae40 msgr2=0x560a6addfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdadab6c700 1 --2- 172.21.15.114:0/3871755756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ac5ae40 0x560a6addfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdadab6c700 1 -- 172.21.15.114:0/3871755756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ade0070 msgr2=0x560a6add9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdadab6c700 1 --2- 172.21.15.114:0/3871755756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ade0070 0x560a6add9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdadab6c700 1 -- 172.21.15.114:0/3871755756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a6aca0fb0 con 0x560a6adda120 2024-05-27T07:41:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.095+0000 7fdad9b6a700 1 --2- 172.21.15.114:0/3871755756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ade0070 0x560a6add9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.096+0000 7fdadab6c700 1 --2- 172.21.15.114:0/3871755756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 0x560a6adda530 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdacc005af0 tx=0x7fdacc005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5eff0cfdf99da88 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.096+0000 7fdac8ff9700 1 -- 172.21.15.114:0/3871755756 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc00be50 con 0x560a6adda120 2024-05-27T07:41:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.096+0000 7fdac8ff9700 1 -- 172.21.15.114:0/3871755756 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdacc0094e0 con 0x560a6adda120 2024-05-27T07:41:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.096+0000 7fdac8ff9700 1 -- 172.21.15.114:0/3871755756 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc0075c0 con 0x560a6adda120 2024-05-27T07:41:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.097+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/3871755756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 msgr2=0x560a6adda530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.097+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/3871755756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 0x560a6adda530 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdacc005af0 tx=0x7fdacc005b20 comp rx=0 tx=0).stop 2024-05-27T07:41:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.097+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/3871755756 shutdown_connections 2024-05-27T07:41:51.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.097+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/3871755756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 0x560a6adda530 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.097+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/3871755756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ac5ae40 0x560a6addfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.097+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/3871755756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ade0070 0x560a6add9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.097+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/3871755756 >> 172.21.15.114:0/3871755756 conn(0x560a6ac5d770 msgr2=0x560a6add4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:51.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.097+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/3871755756 shutdown_connections 2024-05-27T07:41:51.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.097+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/3871755756 wait complete. 2024-05-27T07:41:51.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.097+0000 7fdaeafe4b80 1 -- start start 2024-05-27T07:41:51.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.097+0000 7fdaeafe4b80 1 Processor -- start 2024-05-27T07:41:51.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdaeafe4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ac5ae40 0x560a6ade3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdaeafe4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6adda120 0x560a6ade3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdaeafe4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ade0070 0x560a6ade4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdaeafe4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560a6abfdfa0 con 0x560a6ade0070 2024-05-27T07:41:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdaeafe4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560a6ac5fda0 con 0x560a6adda120 2024-05-27T07:41:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdaeafe4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560a6ade6b00 con 0x560a6ac5ae40 2024-05-27T07:41:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdaeafe4b80 10 client.0 _ll_register_callbacks cb 0x560a6ac8e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdadab6c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ade0070 0x560a6ade4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdadab6c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ade0070 0x560a6ade4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47716/0 (socket says 172.21.15.114:47716) 2024-05-27T07:41:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdad9b6a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6adda120 0x560a6ade3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdadab6c700 1 -- 172.21.15.114:0/425942900 learned_addr learned my addr 172.21.15.114:0/425942900 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdada36b700 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ac5ae40 0x560a6ade3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdadab6c700 1 -- 172.21.15.114:0/425942900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ac5ae40 msgr2=0x560a6ade3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdadab6c700 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ac5ae40 0x560a6ade3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdadab6c700 1 -- 172.21.15.114:0/425942900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6adda120 msgr2=0x560a6ade3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdadab6c700 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6adda120 0x560a6ade3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdadab6c700 1 -- 172.21.15.114:0/425942900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a6ac478d0 con 0x560a6ade0070 2024-05-27T07:41:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdada36b700 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ac5ae40 0x560a6ade3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdad9b6a700 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6adda120 0x560a6ade3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdadab6c700 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ade0070 0x560a6ade4310 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0059b0 tx=0x7fdacc005370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82793d9941936a9a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdabdffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:41:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.098+0000 7fdabdffb700 1 -- 172.21.15.114:0/425942900 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc02d070 con 0x560a6ade0070 2024-05-27T07:41:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdabdffb700 1 -- 172.21.15.114:0/425942900 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fdacc003a00 con 0x560a6ade0070 2024-05-27T07:41:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/425942900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560a6ad09cc0 con 0x560a6ade0070 2024-05-27T07:41:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdabdffb700 1 -- 172.21.15.114:0/425942900 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdacc039050 con 0x560a6ade0070 2024-05-27T07:41:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdaeafe4b80 10 client.15705 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/425942900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560a6acb1760 con 0x560a6ade0070 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdabdffb700 1 -- 172.21.15.114:0/425942900 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560a6ad09cc0 con 0x560a6ade0070 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdabdffb700 1 -- 172.21.15.114:0/425942900 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fdacc011070 con 0x560a6ade0070 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdaeafe4b80 10 client.15705 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdaeafe4b80 10 client.15705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdaeafe4b80 20 client.15705 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a6adf3900 0x560a6adf5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdaeafe4b80 4 client.15705 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/425942900 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a6ad034a0 con 0x560a6adf3900 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.099+0000 7fdada36b700 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a6adf3900 0x560a6adf5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.100+0000 7fdabdffb700 10 client.15705 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.100+0000 7fdada36b700 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a6adf3900 0x560a6adf5d20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ae40ec8997e9558 server_cookie=69fb16bf9f3a46cf in_seq=0 out_seq=0 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdabdffb700 1 -- 172.21.15.114:0/425942900 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560a6ad034a0 con 0x560a6adf3900 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdabdffb700 10 client.15705 handle_command_reply: tid=0 2024-05-27T07:41:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdaeafe4b80 1 client.15705 shutdown 2024-05-27T07:41:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/425942900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ade0070 msgr2=0x560a6ade4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ade0070 0x560a6ade4310 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0059b0 tx=0x7fdacc005370 comp rx=0 tx=0).stop 2024-05-27T07:41:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/425942900 shutdown_connections 2024-05-27T07:41:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ade0070 0x560a6ade4310 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ac5ae40 0x560a6ade3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a6adf3900 0x560a6adf5d20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/425942900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6adda120 0x560a6ade3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdabdffb700 0 client.15705 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/425942900 >> 172.21.15.114:0/425942900 conn(0x560a6ac5d770 msgr2=0x560a6add8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/425942900 shutdown_connections 2024-05-27T07:41:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/425942900 wait complete. 2024-05-27T07:41:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.101+0000 7fdaeafe4b80 20 client.15705 trim_cache size 0 max 16384 2024-05-27T07:41:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdaeafe4b80 1 Processor -- start 2024-05-27T07:41:51.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdaeafe4b80 1 -- start start 2024-05-27T07:41:51.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdaeafe4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ac5ae40 0x560a6add9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdaeafe4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 0x560a6adddec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdaeafe4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ade0070 0x560a6adefda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdaeafe4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdad41ae580 con 0x560a6adda120 2024-05-27T07:41:51.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdaeafe4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdad41b0e00 con 0x560a6ac5ae40 2024-05-27T07:41:51.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdaeafe4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdad41aebf0 con 0x560a6ade0070 2024-05-27T07:41:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdad9b6a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 0x560a6adddec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdad9b6a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 0x560a6adddec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47726/0 (socket says 172.21.15.114:47726) 2024-05-27T07:41:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdad9b6a700 1 -- 172.21.15.114:0/1315789668 learned_addr learned my addr 172.21.15.114:0/1315789668 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdadab6c700 1 --2- 172.21.15.114:0/1315789668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ade0070 0x560a6adefda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdada36b700 1 --2- 172.21.15.114:0/1315789668 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ac5ae40 0x560a6add9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdad9b6a700 1 -- 172.21.15.114:0/1315789668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ade0070 msgr2=0x560a6adefda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:51.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdad9b6a700 1 --2- 172.21.15.114:0/1315789668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ade0070 0x560a6adefda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdad9b6a700 1 -- 172.21.15.114:0/1315789668 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ac5ae40 msgr2=0x560a6add9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:51.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdad9b6a700 1 --2- 172.21.15.114:0/1315789668 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ac5ae40 0x560a6add9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.117+0000 7fdad9b6a700 1 -- 172.21.15.114:0/1315789668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a6add5ab0 con 0x560a6adda120 2024-05-27T07:41:51.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.118+0000 7fdad9b6a700 1 --2- 172.21.15.114:0/1315789668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 0x560a6adddec0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fdac400ed50 tx=0x7fdac4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d38915024db5f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:51.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.118+0000 7fdabffff700 1 -- 172.21.15.114:0/1315789668 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac401fcf0 con 0x560a6adda120 2024-05-27T07:41:51.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.118+0000 7fdabffff700 1 -- 172.21.15.114:0/1315789668 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdac402a070 con 0x560a6adda120 2024-05-27T07:41:51.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.118+0000 7fdabffff700 1 -- 172.21.15.114:0/1315789668 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac401d5e0 con 0x560a6adda120 2024-05-27T07:41:51.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/1315789668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 msgr2=0x560a6adddec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:51.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/1315789668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 0x560a6adddec0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fdac400ed50 tx=0x7fdac4014000 comp rx=0 tx=0).stop 2024-05-27T07:41:51.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/1315789668 shutdown_connections 2024-05-27T07:41:51.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/1315789668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6ade0070 0x560a6adefda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/1315789668 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ac5ae40 0x560a6add9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/1315789668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6adda120 0x560a6adddec0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/1315789668 >> 172.21.15.114:0/1315789668 conn(0x560a6ac5d770 msgr2=0x560a6adde980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:51.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/1315789668 shutdown_connections 2024-05-27T07:41:51.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/1315789668 wait complete. 2024-05-27T07:41:51.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 -- start start 2024-05-27T07:41:51.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 Processor -- start 2024-05-27T07:41:51.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ac5ae40 0x560a6adf7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6adda120 0x560a6adf8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ade0070 0x560a6adf8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdad41af2c0 con 0x560a6ac5ae40 2024-05-27T07:41:51.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdad41b0730 con 0x560a6ade0070 2024-05-27T07:41:51.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.119+0000 7fdaeafe4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdad41af990 con 0x560a6adda120 2024-05-27T07:41:51.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdaeafe4b80 10 client.0 _ll_register_callbacks cb 0x560a6ac8e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:51.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdada36b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ac5ae40 0x560a6adf7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdada36b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ac5ae40 0x560a6adf7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47742/0 (socket says 172.21.15.114:47742) 2024-05-27T07:41:51.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdada36b700 1 -- 172.21.15.114:0/3738336417 learned_addr learned my addr 172.21.15.114:0/3738336417 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:51.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdad9b6a700 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6adda120 0x560a6adf8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdadab6c700 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ade0070 0x560a6adf8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdad9b6a700 1 -- 172.21.15.114:0/3738336417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ade0070 msgr2=0x560a6adf8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:51.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdad9b6a700 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ade0070 0x560a6adf8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdad9b6a700 1 -- 172.21.15.114:0/3738336417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ac5ae40 msgr2=0x560a6adf7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:51.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdad9b6a700 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ac5ae40 0x560a6adf7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdad9b6a700 1 -- 172.21.15.114:0/3738336417 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a6ae5c220 con 0x560a6adda120 2024-05-27T07:41:51.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdada36b700 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ac5ae40 0x560a6adf7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:51.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdad9b6a700 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6adda120 0x560a6adf8100 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x560a6ad9b260 tx=0x7fdac400eec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9eeb53675bb680f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:51.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdabdffb700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:41:51.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdabdffb700 1 -- 172.21.15.114:0/3738336417 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac4004250 con 0x560a6adda120 2024-05-27T07:41:51.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/3738336417 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdac402a070 con 0x560a6adda120 2024-05-27T07:41:51.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdabdffb700 1 -- 172.21.15.114:0/3738336417 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fdac40043f0 con 0x560a6adda120 2024-05-27T07:41:51.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdabdffb700 1 -- 172.21.15.114:0/3738336417 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdac4040050 con 0x560a6adda120 2024-05-27T07:41:51.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdaeafe4b80 10 client.25306 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:51.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.120+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/3738336417 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560a6ae5d460 con 0x560a6adda120 2024-05-27T07:41:51.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.121+0000 7fdabdffb700 1 -- 172.21.15.114:0/3738336417 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdac402a070 con 0x560a6adda120 2024-05-27T07:41:51.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.121+0000 7fdabdffb700 1 -- 172.21.15.114:0/3738336417 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fdac40267a0 con 0x560a6adda120 2024-05-27T07:41:51.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.121+0000 7fdaeafe4b80 10 client.25306 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:51.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.121+0000 7fdaeafe4b80 10 client.25306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:51.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.121+0000 7fdaeafe4b80 20 client.25306 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:51.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.121+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a6adf3900 0x560a6ae5fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:51.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.121+0000 7fdaeafe4b80 4 client.25306 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:41:51.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.121+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/3738336417 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560a6ab7a250 con 0x560a6adf3900 2024-05-27T07:41:51.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.121+0000 7fdada36b700 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a6adf3900 0x560a6ae5fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:51.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.121+0000 7fdabdffb700 10 client.25306 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:51.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.122+0000 7fdada36b700 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a6adf3900 0x560a6ae5fab0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1aee09379786af77 server_cookie=9de30796d084e425 in_seq=0 out_seq=0 2024-05-27T07:41:51.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.123+0000 7fdabdffb700 1 -- 172.21.15.114:0/3738336417 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560a6ab7a250 con 0x560a6adf3900 2024-05-27T07:41:51.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.123+0000 7fdabdffb700 10 client.25306 handle_command_reply: tid=0 2024-05-27T07:41:51.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.123+0000 7fdaeafe4b80 1 client.25306 shutdown 2024-05-27T07:41:51.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/3738336417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6adda120 msgr2=0x560a6adf8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:51.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6adda120 0x560a6adf8100 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x560a6ad9b260 tx=0x7fdac400eec0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/3738336417 shutdown_connections 2024-05-27T07:41:51.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a6adf3900 0x560a6ae5fab0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a6ade0070 0x560a6adf8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a6ac5ae40 0x560a6adf7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdaeafe4b80 1 --2- 172.21.15.114:0/3738336417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a6adda120 0x560a6adf8100 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdabdffb700 0 client.25306 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:51.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/3738336417 >> 172.21.15.114:0/3738336417 conn(0x560a6ac5d770 msgr2=0x560a6addd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:51.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/3738336417 shutdown_connections 2024-05-27T07:41:51.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdaeafe4b80 1 -- 172.21.15.114:0/3738336417 wait complete. 2024-05-27T07:41:51.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdaeafe4b80 20 client.25306 trim_cache size 0 max 16384 2024-05-27T07:41:51.135 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:41:51.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdadc5cf700 1 -- 172.21.15.114:0/3760342650 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdac00b29f0 msgr2=0x7fdac00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:51.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdadc5cf700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdac00b29f0 0x7fdac00b4f00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdad4102230 tx=0x7fdac4009260 comp rx=0 tx=0).stop 2024-05-27T07:41:51.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdadc5cf700 1 -- 172.21.15.114:0/3760342650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdad41034b0 msgr2=0x7fdad419bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:51.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdadc5cf700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdad41034b0 0x7fdad419bbe0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdad000e9a0 tx=0x7fdad000ecb0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdadc5cf700 1 -- 172.21.15.114:0/3760342650 shutdown_connections 2024-05-27T07:41:51.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdadc5cf700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdac00b29f0 0x7fdac00b4f00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdadc5cf700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdad4108730 0x7fdad41a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdadc5cf700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdad41034b0 0x7fdad419bbe0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdadc5cf700 1 --2- 172.21.15.114:0/3760342650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad4107de0 0x7fdad419c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:51.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.124+0000 7fdadc5cf700 1 -- 172.21.15.114:0/3760342650 >> 172.21.15.114:0/3760342650 conn(0x7fdad40fedb0 msgr2=0x7fdad4107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:51.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.125+0000 7fdadc5cf700 1 -- 172.21.15.114:0/3760342650 shutdown_connections 2024-05-27T07:41:51.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:51.125+0000 7fdadc5cf700 1 -- 172.21.15.114:0/3760342650 wait complete. 2024-05-27T07:41:51.236 INFO:tasks.workunit.client.0.smithi114.stdout: 1 9543 22.11 MB/sec warmup 3 sec latency 14.468 ms 2024-05-27T07:41:51.705 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:51 smithi114 ceph-mon[32038]: pgmap v66: 129 pgs: 129 active+clean; 4.7 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 318 KiB/s wr, 38 op/s 2024-05-27T07:41:51.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:51 smithi163 ceph-mon[36451]: pgmap v66: 129 pgs: 129 active+clean; 4.7 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 318 KiB/s wr, 38 op/s 2024-05-27T07:41:51.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:51 smithi186 ceph-mon[36647]: pgmap v66: 129 pgs: 129 active+clean; 4.7 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 318 KiB/s wr, 38 op/s 2024-05-27T07:41:52.236 INFO:tasks.workunit.client.0.smithi114.stdout: 1 11907 20.11 MB/sec warmup 4 sec latency 36.483 ms 2024-05-27T07:41:52.779 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:41:52.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:41:53.029 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:53.236 INFO:tasks.workunit.client.0.smithi114.stdout: 1 15001 19.59 MB/sec warmup 5 sec latency 21.847 ms 2024-05-27T07:41:53.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:53 smithi114 ceph-mon[32038]: pgmap v67: 129 pgs: 129 active+clean; 4.7 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 318 KiB/s wr, 37 op/s 2024-05-27T07:41:53.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:53 smithi163 ceph-mon[36451]: pgmap v67: 129 pgs: 129 active+clean; 4.7 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 318 KiB/s wr, 37 op/s 2024-05-27T07:41:53.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:53 smithi186 ceph-mon[36647]: pgmap v67: 129 pgs: 129 active+clean; 4.7 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 318 KiB/s wr, 37 op/s 2024-05-27T07:41:54.236 INFO:tasks.workunit.client.0.smithi114.stdout: 1 18310 19.43 MB/sec warmup 6 sec latency 7.230 ms 2024-05-27T07:41:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.699+0000 7f67515c0640 1 -- 172.21.15.114:0/1262707692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f674c108bb0 msgr2=0x7f674c102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.699+0000 7f67515c0640 1 --2- 172.21.15.114:0/1262707692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f674c108bb0 0x7f674c102750 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f673c009900 tx=0x7f673c02a410 comp rx=0 tx=0).stop 2024-05-27T07:41:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.700+0000 7f67515c0640 1 -- 172.21.15.114:0/1262707692 shutdown_connections 2024-05-27T07:41:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.700+0000 7f67515c0640 1 --2- 172.21.15.114:0/1262707692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c103300 0x7f674c10f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.700+0000 7f67515c0640 1 --2- 172.21.15.114:0/1262707692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f674c109050 0x7f674c102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.700+0000 7f67515c0640 1 --2- 172.21.15.114:0/1262707692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f674c108bb0 0x7f674c102750 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.700+0000 7f67515c0640 1 -- 172.21.15.114:0/1262707692 >> 172.21.15.114:0/1262707692 conn(0x7f674c0fe380 msgr2=0x7f674c1007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.700+0000 7f67515c0640 1 -- 172.21.15.114:0/1262707692 shutdown_connections 2024-05-27T07:41:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.700+0000 7f67515c0640 1 -- 172.21.15.114:0/1262707692 wait complete. 2024-05-27T07:41:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.700+0000 7f67515c0640 1 Processor -- start 2024-05-27T07:41:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.700+0000 7f67515c0640 1 -- start start 2024-05-27T07:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f67515c0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c103300 0x7f674c19b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f67515c0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f674c108bb0 0x7f674c19bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f67515c0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f674c109050 0x7f674c19c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f67515c0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f674c112df0 con 0x7f674c108bb0 2024-05-27T07:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f67515c0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f674c112c70 con 0x7f674c109050 2024-05-27T07:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f67515c0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f674c112f70 con 0x7f674c103300 2024-05-27T07:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f6743fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f674c108bb0 0x7f674c19bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f674affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c103300 0x7f674c19b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f6743fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f674c108bb0 0x7f674c19bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51892/0 (socket says 172.21.15.114:51892) 2024-05-27T07:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f674affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c103300 0x7f674c19b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34472/0 (socket says 172.21.15.114:34472) 2024-05-27T07:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f6743fff640 1 -- 172.21.15.114:0/3724655104 learned_addr learned my addr 172.21.15.114:0/3724655104 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f674b7fe640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f674c109050 0x7f674c19c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f6743fff640 1 -- 172.21.15.114:0/3724655104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c103300 msgr2=0x7f674c19b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f6743fff640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c103300 0x7f674c19b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f6743fff640 1 -- 172.21.15.114:0/3724655104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f674c109050 msgr2=0x7f674c19c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f6743fff640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f674c109050 0x7f674c19c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f6743fff640 1 -- 172.21.15.114:0/3724655104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c1a6aa0 con 0x7f674c108bb0 2024-05-27T07:41:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f674affd640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c103300 0x7f674c19b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f6743fff640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f674c108bb0 0x7f674c19bcd0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f673400e9f0 tx=0x7f673400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.701+0000 7f6748ff9640 1 -- 172.21.15.114:0/3724655104 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673400ccc0 con 0x7f674c108bb0 2024-05-27T07:41:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.702+0000 7f6748ff9640 1 -- 172.21.15.114:0/3724655104 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67340035f0 con 0x7f674c108bb0 2024-05-27T07:41:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.702+0000 7f67515c0640 1 -- 172.21.15.114:0/3724655104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674c1a6d90 con 0x7f674c108bb0 2024-05-27T07:41:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.702+0000 7f6748ff9640 1 -- 172.21.15.114:0/3724655104 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673400ce60 con 0x7f674c108bb0 2024-05-27T07:41:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.702+0000 7f67515c0640 1 -- 172.21.15.114:0/3724655104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674c1a7250 con 0x7f674c108bb0 2024-05-27T07:41:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.704+0000 7f6758470740 1 Processor -- start 2024-05-27T07:41:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.704+0000 7f6748ff9640 1 -- 172.21.15.114:0/3724655104 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6734003aa0 con 0x7f674c108bb0 2024-05-27T07:41:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.704+0000 7f6758470740 1 -- start start 2024-05-27T07:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f6758470740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83541b0 0x562ae86f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f6758470740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae832b2f0 0x562ae86f5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f6758470740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae83334d0 0x562ae86f5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f6758470740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562ae8667c30 con 0x562ae832b2f0 2024-05-27T07:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f6758470740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562ae86577d0 con 0x562ae83334d0 2024-05-27T07:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f6758470740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562ae865b810 con 0x562ae83541b0 2024-05-27T07:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f674b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83541b0 0x562ae86f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f674affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae832b2f0 0x562ae86f5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f674b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83541b0 0x562ae86f53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34482/0 (socket says 172.21.15.114:34482) 2024-05-27T07:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f674b7fe640 1 -- 172.21.15.114:0/2960412680 learned_addr learned my addr 172.21.15.114:0/2960412680 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f674affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae832b2f0 0x562ae86f5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51908/0 (socket says 172.21.15.114:51908) 2024-05-27T07:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f6743fff640 1 --2- 172.21.15.114:0/2960412680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae83334d0 0x562ae86f5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f674b7fe640 1 -- 172.21.15.114:0/2960412680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae83334d0 msgr2=0x562ae86f5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f674b7fe640 1 --2- 172.21.15.114:0/2960412680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae83334d0 0x562ae86f5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f674b7fe640 1 -- 172.21.15.114:0/2960412680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae832b2f0 msgr2=0x562ae86f5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f674b7fe640 1 --2- 172.21.15.114:0/2960412680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae832b2f0 0x562ae86f5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f6748ff9640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67380b29d0 0x7f67380b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f674b7fe640 1 -- 172.21.15.114:0/2960412680 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ae836c210 con 0x562ae83541b0 2024-05-27T07:41:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f6743fff640 1 --2- 172.21.15.114:0/2960412680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae83334d0 0x562ae86f5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f6748ff9640 1 -- 172.21.15.114:0/3724655104 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f67340e2be0 con 0x7f674c108bb0 2024-05-27T07:41:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f674b7fe640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67380b29d0 0x7f67380b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.705+0000 7f674b7fe640 1 --2- 172.21.15.114:0/2960412680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83541b0 0x562ae86f53d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6744009f90 tx=0x7f67440118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c19b97429564598 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.706+0000 7f67417fa640 1 -- 172.21.15.114:0/2960412680 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744004050 con 0x562ae83541b0 2024-05-27T07:41:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.706+0000 7f67417fa640 1 -- 172.21.15.114:0/2960412680 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6744004800 con 0x562ae83541b0 2024-05-27T07:41:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.706+0000 7f67417fa640 1 -- 172.21.15.114:0/2960412680 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744019660 con 0x562ae83541b0 2024-05-27T07:41:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.707+0000 7f6758470740 1 -- 172.21.15.114:0/2960412680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83541b0 msgr2=0x562ae86f53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.707+0000 7f674b7fe640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67380b29d0 0x7f67380b4ee0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f674401d000 tx=0x7f674401e000 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.707+0000 7f6758470740 1 --2- 172.21.15.114:0/2960412680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83541b0 0x562ae86f53d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6744009f90 tx=0x7f67440118d0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.707+0000 7f6758470740 1 -- 172.21.15.114:0/2960412680 shutdown_connections 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.707+0000 7f6758470740 1 --2- 172.21.15.114:0/2960412680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae83334d0 0x562ae86f5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.707+0000 7f6758470740 1 --2- 172.21.15.114:0/2960412680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae832b2f0 0x562ae86f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.707+0000 7f6758470740 1 --2- 172.21.15.114:0/2960412680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83541b0 0x562ae86f53d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.707+0000 7f6758470740 1 -- 172.21.15.114:0/2960412680 >> 172.21.15.114:0/2960412680 conn(0x562ae8348300 msgr2=0x562ae86f1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.707+0000 7f6758470740 1 -- 172.21.15.114:0/2960412680 shutdown_connections 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.707+0000 7f6758470740 1 -- 172.21.15.114:0/2960412680 wait complete. 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.707+0000 7f6758470740 1 -- start start 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.707+0000 7f6758470740 1 Processor -- start 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6758470740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae832b2f0 0x562ae87013e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6758470740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 0x562ae8701920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6758470740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83541b0 0x562ae86ff8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6758470740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562ae833d790 con 0x562ae83334d0 2024-05-27T07:41:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6758470740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562ae8613890 con 0x562ae832b2f0 2024-05-27T07:41:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6758470740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562ae87024a0 con 0x562ae83541b0 2024-05-27T07:41:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6743fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 0x562ae8701920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6758470740 10 client.0 _ll_register_callbacks cb 0x562ae836ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6743fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 0x562ae8701920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51922/0 (socket says 172.21.15.114:51922) 2024-05-27T07:41:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6743fff640 1 -- 172.21.15.114:0/1363321058 learned_addr learned my addr 172.21.15.114:0/1363321058 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f674affd640 1 --2- 172.21.15.114:0/1363321058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae832b2f0 0x562ae87013e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f674b7fe640 1 --2- 172.21.15.114:0/1363321058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83541b0 0x562ae86ff8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6743fff640 1 -- 172.21.15.114:0/1363321058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83541b0 msgr2=0x562ae86ff8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6743fff640 1 --2- 172.21.15.114:0/1363321058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83541b0 0x562ae86ff8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6743fff640 1 -- 172.21.15.114:0/1363321058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae832b2f0 msgr2=0x562ae87013e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6743fff640 1 --2- 172.21.15.114:0/1363321058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae832b2f0 0x562ae87013e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6743fff640 1 -- 172.21.15.114:0/1363321058 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ae8438620 con 0x562ae83334d0 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6743fff640 1 --2- 172.21.15.114:0/1363321058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 0x562ae8701920 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f673400e9c0 tx=0x7f67340aaf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef71559635959c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f67327fc640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f67327fc640 1 -- 172.21.15.114:0/1363321058 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340a9620 con 0x562ae83334d0 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f67327fc640 1 -- 172.21.15.114:0/1363321058 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f67340a5da0 con 0x562ae83334d0 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.708+0000 7f6758470740 1 -- 172.21.15.114:0/1363321058 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6744004800 con 0x562ae83334d0 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.709+0000 7f67327fc640 1 -- 172.21.15.114:0/1363321058 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f67340aa210 con 0x562ae83334d0 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.709+0000 7f6758470740 10 client.15723 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.709+0000 7f6758470740 1 -- 172.21.15.114:0/1363321058 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562ae870e6f0 con 0x562ae83334d0 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.709+0000 7f67327fc640 1 -- 172.21.15.114:0/1363321058 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6744004800 con 0x562ae83334d0 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.709+0000 7f67327fc640 1 -- 172.21.15.114:0/1363321058 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f67340c4080 con 0x562ae83334d0 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.709+0000 7f6758470740 10 client.15723 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.709+0000 7f6758470740 10 client.15723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.709+0000 7f6758470740 20 client.15723 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.709+0000 7f6758470740 1 --2- 172.21.15.114:0/1363321058 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562ae8312d80 0x562ae8711c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.709+0000 7f6758470740 4 client.15723 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.709+0000 7f6758470740 1 -- 172.21.15.114:0/1363321058 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ae8669b80 con 0x562ae8312d80 2024-05-27T07:41:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.709+0000 7f674affd640 1 --2- 172.21.15.114:0/1363321058 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562ae8312d80 0x562ae8711c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.710+0000 7f67327fc640 10 client.15723 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.710+0000 7f674affd640 1 --2- 172.21.15.114:0/1363321058 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562ae8312d80 0x562ae8711c10 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcb952f16307a56 server_cookie=f54b956d955407e6 in_seq=0 out_seq=0 2024-05-27T07:41:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f67327fc640 1 -- 172.21.15.114:0/1363321058 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562ae8669b80 con 0x562ae8312d80 2024-05-27T07:41:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f67327fc640 10 client.15723 handle_command_reply: tid=0 2024-05-27T07:41:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f6758470740 1 client.15723 shutdown 2024-05-27T07:41:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f6758470740 1 -- 172.21.15.114:0/1363321058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 msgr2=0x562ae8701920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f6758470740 1 --2- 172.21.15.114:0/1363321058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 0x562ae8701920 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f673400e9c0 tx=0x7f67340aaf00 comp rx=0 tx=0).stop 2024-05-27T07:41:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f6758470740 1 -- 172.21.15.114:0/1363321058 shutdown_connections 2024-05-27T07:41:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f6758470740 1 --2- 172.21.15.114:0/1363321058 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562ae8312d80 0x562ae8711c10 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f6758470740 1 --2- 172.21.15.114:0/1363321058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83541b0 0x562ae86ff8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f6758470740 1 --2- 172.21.15.114:0/1363321058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 0x562ae8701920 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f6758470740 1 --2- 172.21.15.114:0/1363321058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae832b2f0 0x562ae87013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f6758470740 1 -- 172.21.15.114:0/1363321058 >> 172.21.15.114:0/1363321058 conn(0x562ae8348300 msgr2=0x562ae86f5200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f67327fc640 0 client.15723 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f6758470740 1 -- 172.21.15.114:0/1363321058 shutdown_connections 2024-05-27T07:41:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f6758470740 1 -- 172.21.15.114:0/1363321058 wait complete. 2024-05-27T07:41:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.711+0000 7f6758470740 20 client.15723 trim_cache size 0 max 16384 2024-05-27T07:41:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.731+0000 7f6758470740 1 Processor -- start 2024-05-27T07:41:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.731+0000 7f6758470740 1 -- start start 2024-05-27T07:41:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f6758470740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae832b2f0 0x562ae86f9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f6758470740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83334d0 0x562ae86f9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f6758470740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83541b0 0x562ae86ff160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f6758470740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f674c1af390 con 0x562ae83541b0 2024-05-27T07:41:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f6758470740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f674c1b1c10 con 0x562ae832b2f0 2024-05-27T07:41:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f6758470740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f674c1afa00 con 0x562ae83334d0 2024-05-27T07:41:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f674b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83541b0 0x562ae86ff160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f6743fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83334d0 0x562ae86f9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f674b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83541b0 0x562ae86ff160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51926/0 (socket says 172.21.15.114:51926) 2024-05-27T07:41:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f674b7fe640 1 -- 172.21.15.114:0/3833230753 learned_addr learned my addr 172.21.15.114:0/3833230753 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f674affd640 1 --2- 172.21.15.114:0/3833230753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae832b2f0 0x562ae86f9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f674b7fe640 1 -- 172.21.15.114:0/3833230753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83334d0 msgr2=0x562ae86f9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f674b7fe640 1 --2- 172.21.15.114:0/3833230753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83334d0 0x562ae86f9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f674b7fe640 1 -- 172.21.15.114:0/3833230753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae832b2f0 msgr2=0x562ae86f9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f674b7fe640 1 --2- 172.21.15.114:0/3833230753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae832b2f0 0x562ae86f9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f674b7fe640 1 -- 172.21.15.114:0/3833230753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ae8312fb0 con 0x562ae83541b0 2024-05-27T07:41:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f674affd640 1 --2- 172.21.15.114:0/3833230753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae832b2f0 0x562ae86f9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.732+0000 7f674b7fe640 1 --2- 172.21.15.114:0/3833230753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83541b0 0x562ae86ff160 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x562ae86f2540 tx=0x7f6744026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368a45020548b3d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.733+0000 7f6740ff9640 1 -- 172.21.15.114:0/3833230753 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744034070 con 0x562ae83541b0 2024-05-27T07:41:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.733+0000 7f6740ff9640 1 -- 172.21.15.114:0/3833230753 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6744019e10 con 0x562ae83541b0 2024-05-27T07:41:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.733+0000 7f6740ff9640 1 -- 172.21.15.114:0/3833230753 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744028040 con 0x562ae83541b0 2024-05-27T07:41:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.734+0000 7f6758470740 1 -- 172.21.15.114:0/3833230753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83541b0 msgr2=0x562ae86ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.734+0000 7f6758470740 1 --2- 172.21.15.114:0/3833230753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83541b0 0x562ae86ff160 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x562ae86f2540 tx=0x7f6744026000 comp rx=0 tx=0).stop 2024-05-27T07:41:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.734+0000 7f6758470740 1 -- 172.21.15.114:0/3833230753 shutdown_connections 2024-05-27T07:41:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.734+0000 7f6758470740 1 --2- 172.21.15.114:0/3833230753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83541b0 0x562ae86ff160 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.734+0000 7f6758470740 1 --2- 172.21.15.114:0/3833230753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae83334d0 0x562ae86f9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.734+0000 7f6758470740 1 --2- 172.21.15.114:0/3833230753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae832b2f0 0x562ae86f9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.734+0000 7f6758470740 1 -- 172.21.15.114:0/3833230753 >> 172.21.15.114:0/3833230753 conn(0x562ae8348300 msgr2=0x562ae86fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.734+0000 7f6758470740 1 -- 172.21.15.114:0/3833230753 shutdown_connections 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.734+0000 7f6758470740 1 -- 172.21.15.114:0/3833230753 wait complete. 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.734+0000 7f6758470740 1 -- start start 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.734+0000 7f6758470740 1 Processor -- start 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.734+0000 7f6758470740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae832b2f0 0x562ae87067d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6758470740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 0x562ae8706d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6758470740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae83541b0 0x562ae87782a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6758470740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f674c1b00d0 con 0x562ae83334d0 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6758470740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f674c1b1540 con 0x562ae83541b0 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6758470740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f674c1b07a0 con 0x562ae832b2f0 2024-05-27T07:41:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6758470740 10 client.0 _ll_register_callbacks cb 0x7f67340046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6743fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 0x562ae8706d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6743fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 0x562ae8706d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51930/0 (socket says 172.21.15.114:51930) 2024-05-27T07:41:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6743fff640 1 -- 172.21.15.114:0/1839998973 learned_addr learned my addr 172.21.15.114:0/1839998973 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f674affd640 1 --2- 172.21.15.114:0/1839998973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae832b2f0 0x562ae87067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f674b7fe640 1 --2- 172.21.15.114:0/1839998973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae83541b0 0x562ae87782a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6743fff640 1 -- 172.21.15.114:0/1839998973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae832b2f0 msgr2=0x562ae87067d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6743fff640 1 --2- 172.21.15.114:0/1839998973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae832b2f0 0x562ae87067d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6743fff640 1 -- 172.21.15.114:0/1839998973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae83541b0 msgr2=0x562ae87782a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6743fff640 1 --2- 172.21.15.114:0/1839998973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae83541b0 0x562ae87782a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6743fff640 1 -- 172.21.15.114:0/1839998973 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ae87074f0 con 0x562ae83334d0 2024-05-27T07:41:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6743fff640 1 --2- 172.21.15.114:0/1839998973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 0x562ae8706d10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f67340a9cd0 tx=0x7f6734003e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0cf365664ca179 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f67327fc640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:41:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f67327fc640 1 -- 172.21.15.114:0/1839998973 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340a9280 con 0x562ae83334d0 2024-05-27T07:41:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f67327fc640 1 -- 172.21.15.114:0/1839998973 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f67340a6100 con 0x562ae83334d0 2024-05-27T07:41:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6758470740 1 -- 172.21.15.114:0/1839998973 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6744019e10 con 0x562ae83334d0 2024-05-27T07:41:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f67327fc640 1 -- 172.21.15.114:0/1839998973 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f67340a59d0 con 0x562ae83334d0 2024-05-27T07:41:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6758470740 10 client.15729 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.735+0000 7f6758470740 1 -- 172.21.15.114:0/1839998973 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562ae8779780 con 0x562ae83334d0 2024-05-27T07:41:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.736+0000 7f67327fc640 1 -- 172.21.15.114:0/1839998973 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6744019e10 con 0x562ae83334d0 2024-05-27T07:41:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.736+0000 7f67327fc640 1 -- 172.21.15.114:0/1839998973 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f67340a87e0 con 0x562ae83334d0 2024-05-27T07:41:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.736+0000 7f6758470740 10 client.15729 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.736+0000 7f6758470740 10 client.15729 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.736+0000 7f6758470740 20 client.15729 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.736+0000 7f6758470740 1 --2- 172.21.15.114:0/1839998973 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562ae82e81c0 0x562ae877bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.736+0000 7f6758470740 4 client.15729 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:41:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.736+0000 7f6758470740 1 -- 172.21.15.114:0/1839998973 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562ae8663360 con 0x562ae82e81c0 2024-05-27T07:41:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.736+0000 7f674affd640 1 --2- 172.21.15.114:0/1839998973 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562ae82e81c0 0x562ae877bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.736+0000 7f67327fc640 10 client.15729 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.737+0000 7f674affd640 1 --2- 172.21.15.114:0/1839998973 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562ae82e81c0 0x562ae877bd10 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf8fcc571524fb8b server_cookie=24089fded44dba13 in_seq=0 out_seq=0 2024-05-27T07:41:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f67327fc640 1 -- 172.21.15.114:0/1839998973 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562ae8663360 con 0x562ae82e81c0 2024-05-27T07:41:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f67327fc640 10 client.15729 handle_command_reply: tid=0 2024-05-27T07:41:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f6758470740 1 client.15729 shutdown 2024-05-27T07:41:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f6758470740 1 -- 172.21.15.114:0/1839998973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 msgr2=0x562ae8706d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f6758470740 1 --2- 172.21.15.114:0/1839998973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 0x562ae8706d10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f67340a9cd0 tx=0x7f6734003e70 comp rx=0 tx=0).stop 2024-05-27T07:41:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f6758470740 1 -- 172.21.15.114:0/1839998973 shutdown_connections 2024-05-27T07:41:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f6758470740 1 --2- 172.21.15.114:0/1839998973 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562ae82e81c0 0x562ae877bd10 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f6758470740 1 --2- 172.21.15.114:0/1839998973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562ae83541b0 0x562ae87782a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f67327fc640 0 client.15729 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f6758470740 1 --2- 172.21.15.114:0/1839998973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ae83334d0 0x562ae8706d10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f6758470740 1 --2- 172.21.15.114:0/1839998973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562ae832b2f0 0x562ae87067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f6758470740 1 -- 172.21.15.114:0/1839998973 >> 172.21.15.114:0/1839998973 conn(0x562ae8348300 msgr2=0x562ae86f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f6758470740 1 -- 172.21.15.114:0/1839998973 shutdown_connections 2024-05-27T07:41:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f6758470740 1 -- 172.21.15.114:0/1839998973 wait complete. 2024-05-27T07:41:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.931+0000 7f6758470740 20 client.15729 trim_cache size 0 max 16384 2024-05-27T07:41:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:41:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:41:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:41:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.932+0000 7f67515c0640 1 -- 172.21.15.114:0/3724655104 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67380b29d0 msgr2=0x7f67380b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.932+0000 7f67515c0640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67380b29d0 0x7f67380b4ee0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f674401d000 tx=0x7f674401e000 comp rx=0 tx=0).stop 2024-05-27T07:41:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.932+0000 7f67515c0640 1 -- 172.21.15.114:0/3724655104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f674c108bb0 msgr2=0x7f674c19bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.932+0000 7f67515c0640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f674c108bb0 0x7f674c19bcd0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f673400e9f0 tx=0x7f673400eeb0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.932+0000 7f67515c0640 1 -- 172.21.15.114:0/3724655104 shutdown_connections 2024-05-27T07:41:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.932+0000 7f67515c0640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67380b29d0 0x7f67380b4ee0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.932+0000 7f67515c0640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f674c109050 0x7f674c19c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.932+0000 7f67515c0640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f674c108bb0 0x7f674c19bcd0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.932+0000 7f67515c0640 1 --2- 172.21.15.114:0/3724655104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c103300 0x7f674c19b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.932+0000 7f67515c0640 1 -- 172.21.15.114:0/3724655104 >> 172.21.15.114:0/3724655104 conn(0x7f674c0fe380 msgr2=0x7f674c0fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.932+0000 7f67515c0640 1 -- 172.21.15.114:0/3724655104 shutdown_connections 2024-05-27T07:41:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:41:54.932+0000 7f67515c0640 1 -- 172.21.15.114:0/3724655104 wait complete. 2024-05-27T07:41:55.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.022+0000 7fb86ffcb700 1 -- 172.21.15.114:0/109752231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb868108750 msgr2=0x7fb8681100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:55.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.022+0000 7fb86ffcb700 1 --2- 172.21.15.114:0/109752231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb868108750 0x7fb8681100b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb8600099c0 tx=0x7fb860009cd0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.022+0000 7fb86ffcb700 1 -- 172.21.15.114:0/109752231 shutdown_connections 2024-05-27T07:41:55.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.022+0000 7fb86ffcb700 1 --2- 172.21.15.114:0/109752231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb868108750 0x7fb8681100b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.022+0000 7fb86ffcb700 1 --2- 172.21.15.114:0/109752231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8681034d0 0x7fb8681038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.022+0000 7fb86ffcb700 1 --2- 172.21.15.114:0/109752231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb868107e00 0x7fb868108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.022+0000 7fb86ffcb700 1 -- 172.21.15.114:0/109752231 >> 172.21.15.114:0/109752231 conn(0x7fb8680fef20 msgr2=0x7fb868101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:55.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.022+0000 7fb86ffcb700 1 -- 172.21.15.114:0/109752231 shutdown_connections 2024-05-27T07:41:55.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.022+0000 7fb86ffcb700 1 -- 172.21.15.114:0/109752231 wait complete. 2024-05-27T07:41:55.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.023+0000 7fb86ffcb700 1 Processor -- start 2024-05-27T07:41:55.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.023+0000 7fb86ffcb700 1 -- start start 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.023+0000 7fb86ffcb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8681034d0 0x7fb8680ff400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86ffcb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb868107e00 0x7fb8680ff940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86ffcb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb868108750 0x7fb8680ffe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86ffcb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb868113380 con 0x7fb868108750 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86ffcb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb868113200 con 0x7fb868107e00 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86ffcb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb868113500 con 0x7fb8681034d0 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86dd67700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8681034d0 0x7fb8680ff400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86dd67700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8681034d0 0x7fb8680ff400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34536/0 (socket says 172.21.15.114:34536) 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86dd67700 1 -- 172.21.15.114:0/2813902569 learned_addr learned my addr 172.21.15.114:0/2813902569 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86e568700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb868108750 0x7fb8680ffe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86d566700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb868107e00 0x7fb8680ff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86dd67700 1 -- 172.21.15.114:0/2813902569 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb868107e00 msgr2=0x7fb8680ff940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86dd67700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb868107e00 0x7fb8680ff940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86dd67700 1 -- 172.21.15.114:0/2813902569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb868108750 msgr2=0x7fb8680ffe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:55.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86dd67700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb868108750 0x7fb8680ffe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86dd67700 1 -- 172.21.15.114:0/2813902569 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8681a9140 con 0x7fb8681034d0 2024-05-27T07:41:55.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb86dd67700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8681034d0 0x7fb8680ff400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb86400ef10 tx=0x7fb86400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:55.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.024+0000 7fb85effd700 1 -- 172.21.15.114:0/2813902569 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86400ca70 con 0x7fb8681034d0 2024-05-27T07:41:55.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.025+0000 7fb85effd700 1 -- 172.21.15.114:0/2813902569 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb864019070 con 0x7fb8681034d0 2024-05-27T07:41:55.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.025+0000 7fb86ffcb700 1 -- 172.21.15.114:0/2813902569 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8681a93d0 con 0x7fb8681034d0 2024-05-27T07:41:55.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.025+0000 7fb86ffcb700 1 -- 172.21.15.114:0/2813902569 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8681a9910 con 0x7fb8681034d0 2024-05-27T07:41:55.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.025+0000 7fb85effd700 1 -- 172.21.15.114:0/2813902569 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb864014480 con 0x7fb8681034d0 2024-05-27T07:41:55.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.027+0000 7fb85effd700 1 -- 172.21.15.114:0/2813902569 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fb864012050 con 0x7fb8681034d0 2024-05-27T07:41:55.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.028+0000 7fb85effd700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8540b29f0 0x7fb8540b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.028+0000 7fb86d566700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8540b29f0 0x7fb8540b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.028+0000 7fb85effd700 1 -- 172.21.15.114:0/2813902569 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fb8640e2a80 con 0x7fb8681034d0 2024-05-27T07:41:55.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.028+0000 7fb87e9e0b80 1 Processor -- start 2024-05-27T07:41:55.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb86d566700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8540b29f0 0x7fb8540b4f00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb868109a10 tx=0x7fb858009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:55.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb87e9e0b80 1 -- start start 2024-05-27T07:41:55.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb87e9e0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05700ec0 0x559e05884790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb87e9e0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05884cd0 0x559e0587eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb87e9e0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e0587f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb87e9e0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559e057add10 con 0x559e05700ec0 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb87e9e0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559e0579d8b0 con 0x559e0587f000 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb87e9e0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559e057a18f0 con 0x559e05884cd0 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb86dd67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05700ec0 0x559e05884790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb86dd67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05700ec0 0x559e05884790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51950/0 (socket says 172.21.15.114:51950) 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb86dd67700 1 -- 172.21.15.114:0/1009239868 learned_addr learned my addr 172.21.15.114:0/1009239868 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb86d566700 1 --2- 172.21.15.114:0/1009239868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05884cd0 0x559e0587eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb86e568700 1 --2- 172.21.15.114:0/1009239868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e0587f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb86dd67700 1 -- 172.21.15.114:0/1009239868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05884cd0 msgr2=0x559e0587eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb86dd67700 1 --2- 172.21.15.114:0/1009239868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05884cd0 0x559e0587eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb86dd67700 1 -- 172.21.15.114:0/1009239868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 msgr2=0x559e0587f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb86dd67700 1 --2- 172.21.15.114:0/1009239868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e0587f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb86dd67700 1 -- 172.21.15.114:0/1009239868 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e056ed950 con 0x559e05700ec0 2024-05-27T07:41:55.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.029+0000 7fb86d566700 1 --2- 172.21.15.114:0/1009239868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05884cd0 0x559e0587eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:55.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.030+0000 7fb86dd67700 1 --2- 172.21.15.114:0/1009239868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05700ec0 0x559e05884790 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb8640a5a30 tx=0x7fb8640a9120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38f0a7f19bacb15 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:55.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.030+0000 7fb853fff700 1 -- 172.21.15.114:0/1009239868 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8640b2150 con 0x559e05700ec0 2024-05-27T07:41:55.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.030+0000 7fb853fff700 1 -- 172.21.15.114:0/1009239868 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8640a94c0 con 0x559e05700ec0 2024-05-27T07:41:55.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.030+0000 7fb853fff700 1 -- 172.21.15.114:0/1009239868 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8640a9e50 con 0x559e05700ec0 2024-05-27T07:41:55.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.031+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/1009239868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05700ec0 msgr2=0x559e05884790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:55.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.031+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/1009239868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05700ec0 0x559e05884790 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb8640a5a30 tx=0x7fb8640a9120 comp rx=0 tx=0).stop 2024-05-27T07:41:55.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.031+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/1009239868 shutdown_connections 2024-05-27T07:41:55.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.031+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/1009239868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e0587f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.031+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/1009239868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05700ec0 0x559e05884790 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.031+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/1009239868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05884cd0 0x559e0587eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.031+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/1009239868 >> 172.21.15.114:0/1009239868 conn(0x559e0575bb70 msgr2=0x559e058796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:55.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.031+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/1009239868 shutdown_connections 2024-05-27T07:41:55.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.031+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/1009239868 wait complete. 2024-05-27T07:41:55.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.031+0000 7fb87e9e0b80 1 -- start start 2024-05-27T07:41:55.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.031+0000 7fb87e9e0b80 1 Processor -- start 2024-05-27T07:41:55.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.032+0000 7fb87e9e0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e05888480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.032+0000 7fb87e9e0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e058889c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.032+0000 7fb87e9e0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e05888f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.032+0000 7fb87e9e0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559e056a4020 con 0x559e05884cd0 2024-05-27T07:41:55.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.032+0000 7fb87e9e0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559e0573f840 con 0x559e0587f000 2024-05-27T07:41:55.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.032+0000 7fb87e9e0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559e0588b7c0 con 0x559e05700ec0 2024-05-27T07:41:55.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.032+0000 7fb87e9e0b80 10 client.0 _ll_register_callbacks cb 0x559e057389a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:55.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.032+0000 7fb86e568700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e05888f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.032+0000 7fb86dd67700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e05888480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.032+0000 7fb86dd67700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e05888480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34556/0 (socket says 172.21.15.114:34556) 2024-05-27T07:41:55.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.032+0000 7fb86dd67700 1 -- 172.21.15.114:0/2311035120 learned_addr learned my addr 172.21.15.114:0/2311035120 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:55.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.032+0000 7fb86e568700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e05888f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51952/0 (socket says 172.21.15.114:51952) 2024-05-27T07:41:55.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.032+0000 7fb86d566700 1 --2- 172.21.15.114:0/2311035120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e058889c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb86dd67700 1 -- 172.21.15.114:0/2311035120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 msgr2=0x559e058889c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb86dd67700 1 --2- 172.21.15.114:0/2311035120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e058889c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb86dd67700 1 -- 172.21.15.114:0/2311035120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 msgr2=0x559e05888f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb86dd67700 1 --2- 172.21.15.114:0/2311035120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e05888f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb86dd67700 1 -- 172.21.15.114:0/2311035120 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e0573c710 con 0x559e05700ec0 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb86d566700 1 --2- 172.21.15.114:0/2311035120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e058889c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb86dd67700 1 --2- 172.21.15.114:0/2311035120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e05888480 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb8640dfed0 tx=0x7fb8640b25b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9116ac3d0ab36100 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb8517fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb8517fa700 1 -- 172.21.15.114:0/2311035120 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8640c3070 con 0x559e05700ec0 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2311035120 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8640a94c0 con 0x559e05700ec0 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb8517fa700 1 -- 172.21.15.114:0/2311035120 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fb8640a5fa0 con 0x559e05700ec0 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb8517fa700 1 -- 172.21.15.114:0/2311035120 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb8640b2150 con 0x559e05700ec0 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb87e9e0b80 10 client.25336 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2311035120 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559e058962e0 con 0x559e05700ec0 2024-05-27T07:41:55.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.033+0000 7fb8517fa700 1 -- 172.21.15.114:0/2311035120 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8640a94c0 con 0x559e05700ec0 2024-05-27T07:41:55.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.034+0000 7fb8517fa700 1 -- 172.21.15.114:0/2311035120 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb8640aace0 con 0x559e05700ec0 2024-05-27T07:41:55.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.034+0000 7fb87e9e0b80 10 client.25336 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:55.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.034+0000 7fb87e9e0b80 10 client.25336 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:55.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.034+0000 7fb87e9e0b80 20 client.25336 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:55.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.034+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/2311035120 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559e058987b0 0x559e0589abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.034+0000 7fb87e9e0b80 4 client.25336 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:55.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.034+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2311035120 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e0584aa70 con 0x559e058987b0 2024-05-27T07:41:55.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.034+0000 7fb86d566700 1 --2- 172.21.15.114:0/2311035120 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559e058987b0 0x559e0589abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.034+0000 7fb8517fa700 10 client.25336 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:55.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.034+0000 7fb86d566700 1 --2- 172.21.15.114:0/2311035120 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559e058987b0 0x559e0589abd0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=688861f2018cd96c server_cookie=ab85b57f008bc16 in_seq=0 out_seq=0 2024-05-27T07:41:55.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.036+0000 7fb8517fa700 1 -- 172.21.15.114:0/2311035120 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559e0584aa70 con 0x559e058987b0 2024-05-27T07:41:55.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.036+0000 7fb8517fa700 10 client.25336 handle_command_reply: tid=0 2024-05-27T07:41:55.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.037+0000 7fb87e9e0b80 1 client.25336 shutdown 2024-05-27T07:41:55.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.037+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2311035120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 msgr2=0x559e05888480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.037+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/2311035120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e05888480 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb8640dfed0 tx=0x7fb8640b25b0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.037+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2311035120 shutdown_connections 2024-05-27T07:41:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.037+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/2311035120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e05888f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.037+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/2311035120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e05888480 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.037+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/2311035120 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559e058987b0 0x559e0589abd0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.037+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/2311035120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e058889c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.037+0000 7fb8517fa700 0 client.25336 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.037+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2311035120 >> 172.21.15.114:0/2311035120 conn(0x559e0575bb70 msgr2=0x559e0587dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.037+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2311035120 shutdown_connections 2024-05-27T07:41:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.037+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2311035120 wait complete. 2024-05-27T07:41:55.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.037+0000 7fb87e9e0b80 20 client.25336 trim_cache size 0 max 16384 2024-05-27T07:41:55.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb87e9e0b80 1 Processor -- start 2024-05-27T07:41:55.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb87e9e0b80 1 -- start start 2024-05-27T07:41:55.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb87e9e0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e0587eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb87e9e0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e058860f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb87e9e0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e05882ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb87e9e0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb8681b0860 con 0x559e05884cd0 2024-05-27T07:41:55.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb87e9e0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb8681b30e0 con 0x559e0587f000 2024-05-27T07:41:55.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb87e9e0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8681b0ed0 con 0x559e05700ec0 2024-05-27T07:41:55.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb86e568700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e05882ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb86d566700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e058860f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb86dd67700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e0587eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb86e568700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e05882ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51968/0 (socket says 172.21.15.114:51968) 2024-05-27T07:41:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb86d566700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e058860f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37342/0 (socket says 172.21.15.114:37342) 2024-05-27T07:41:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.058+0000 7fb86e568700 1 -- 172.21.15.114:0/252452704 learned_addr learned my addr 172.21.15.114:0/252452704 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.059+0000 7fb86e568700 1 -- 172.21.15.114:0/252452704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 msgr2=0x559e0587eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.059+0000 7fb86e568700 1 --2- 172.21.15.114:0/252452704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e0587eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.059+0000 7fb86e568700 1 -- 172.21.15.114:0/252452704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 msgr2=0x559e058860f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.059+0000 7fb86e568700 1 --2- 172.21.15.114:0/252452704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e058860f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.059+0000 7fb86e568700 1 -- 172.21.15.114:0/252452704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e0587af50 con 0x559e05884cd0 2024-05-27T07:41:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.059+0000 7fb86d566700 1 --2- 172.21.15.114:0/252452704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e058860f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.059+0000 7fb86e568700 1 --2- 172.21.15.114:0/252452704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e05882ac0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb860003760 tx=0x7fb860005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec529db216976fff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.059+0000 7fb8537fe700 1 -- 172.21.15.114:0/252452704 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb860005330 con 0x559e05884cd0 2024-05-27T07:41:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.059+0000 7fb8537fe700 1 -- 172.21.15.114:0/252452704 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb860003c70 con 0x559e05884cd0 2024-05-27T07:41:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.059+0000 7fb8537fe700 1 -- 172.21.15.114:0/252452704 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86000bc50 con 0x559e05884cd0 2024-05-27T07:41:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.060+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/252452704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 msgr2=0x559e05882ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.060+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/252452704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e05882ac0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb860003760 tx=0x7fb860005040 comp rx=0 tx=0).stop 2024-05-27T07:41:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.060+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/252452704 shutdown_connections 2024-05-27T07:41:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.060+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/252452704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e05882ac0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.060+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/252452704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e0587eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.060+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/252452704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e058860f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.060+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/252452704 >> 172.21.15.114:0/252452704 conn(0x559e0575bb70 msgr2=0x559e05883b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.060+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/252452704 shutdown_connections 2024-05-27T07:41:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.060+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/252452704 wait complete. 2024-05-27T07:41:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.061+0000 7fb87e9e0b80 1 -- start start 2024-05-27T07:41:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.061+0000 7fb87e9e0b80 1 Processor -- start 2024-05-27T07:41:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.061+0000 7fb87e9e0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e0589ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.061+0000 7fb87e9e0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e0589cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.061+0000 7fb87e9e0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e0589d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.061+0000 7fb87e9e0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb8681b15a0 con 0x559e05884cd0 2024-05-27T07:41:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.061+0000 7fb87e9e0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb8681b2a10 con 0x559e0587f000 2024-05-27T07:41:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.061+0000 7fb87e9e0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8681b1c70 con 0x559e05700ec0 2024-05-27T07:41:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.061+0000 7fb87e9e0b80 10 client.0 _ll_register_callbacks cb 0x559e057389a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.061+0000 7fb86dd67700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e0589ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.061+0000 7fb86e568700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e0589d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.061+0000 7fb86dd67700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e0589ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34560/0 (socket says 172.21.15.114:34560) 2024-05-27T07:41:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.061+0000 7fb86dd67700 1 -- 172.21.15.114:0/2638335616 learned_addr learned my addr 172.21.15.114:0/2638335616 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb86d566700 1 --2- 172.21.15.114:0/2638335616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e0589cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb86dd67700 1 -- 172.21.15.114:0/2638335616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 msgr2=0x559e0589cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb86dd67700 1 --2- 172.21.15.114:0/2638335616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e0589cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb86dd67700 1 -- 172.21.15.114:0/2638335616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 msgr2=0x559e0589d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb86dd67700 1 --2- 172.21.15.114:0/2638335616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e0589d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb86dd67700 1 -- 172.21.15.114:0/2638335616 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e05874490 con 0x559e05700ec0 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb86e568700 1 --2- 172.21.15.114:0/2638335616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e0589d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb86dd67700 1 --2- 172.21.15.114:0/2638335616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e0589ca90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb8640aa010 tx=0x7fb8640a5620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1574eb570866410 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb8517fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb8517fa700 1 -- 172.21.15.114:0/2638335616 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8640c3040 con 0x559e05700ec0 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2638335616 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb860003c70 con 0x559e05700ec0 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb8517fa700 1 -- 172.21.15.114:0/2638335616 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fb8640a5fa0 con 0x559e05700ec0 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb8517fa700 1 -- 172.21.15.114:0/2638335616 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb8640a5870 con 0x559e05700ec0 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb87e9e0b80 10 client.25342 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2638335616 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559e05904180 con 0x559e05700ec0 2024-05-27T07:41:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.062+0000 7fb8517fa700 1 -- 172.21.15.114:0/2638335616 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb860003c70 con 0x559e05700ec0 2024-05-27T07:41:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.063+0000 7fb8517fa700 1 -- 172.21.15.114:0/2638335616 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb8640aace0 con 0x559e05700ec0 2024-05-27T07:41:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.063+0000 7fb87e9e0b80 10 client.25342 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.063+0000 7fb87e9e0b80 10 client.25342 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.063+0000 7fb87e9e0b80 20 client.25342 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.063+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/2638335616 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559e05904690 0x559e05906ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.063+0000 7fb87e9e0b80 4 client.25342 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:41:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.063+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2638335616 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559e057afc60 con 0x559e05904690 2024-05-27T07:41:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.063+0000 7fb86d566700 1 --2- 172.21.15.114:0/2638335616 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559e05904690 0x559e05906ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.063+0000 7fb8517fa700 10 client.25342 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.063+0000 7fb86d566700 1 --2- 172.21.15.114:0/2638335616 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559e05904690 0x559e05906ab0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6ff70f63caa764 server_cookie=38020a91bc0080f4 in_seq=0 out_seq=0 2024-05-27T07:41:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb8517fa700 1 -- 172.21.15.114:0/2638335616 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559e057afc60 con 0x559e05904690 2024-05-27T07:41:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb8517fa700 10 client.25342 handle_command_reply: tid=0 2024-05-27T07:41:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb87e9e0b80 1 client.25342 shutdown 2024-05-27T07:41:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2638335616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 msgr2=0x559e0589ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/2638335616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e0589ca90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb8640aa010 tx=0x7fb8640a5620 comp rx=0 tx=0).stop 2024-05-27T07:41:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2638335616 shutdown_connections 2024-05-27T07:41:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/2638335616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e05884cd0 0x559e0589d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/2638335616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559e05700ec0 0x559e0589ca90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/2638335616 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559e05904690 0x559e05906ab0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb87e9e0b80 1 --2- 172.21.15.114:0/2638335616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559e0587f000 0x559e0589cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2638335616 >> 172.21.15.114:0/2638335616 conn(0x559e0575bb70 msgr2=0x559e05883b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb8517fa700 0 client.25342 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2638335616 shutdown_connections 2024-05-27T07:41:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb87e9e0b80 1 -- 172.21.15.114:0/2638335616 wait complete. 2024-05-27T07:41:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.065+0000 7fb87e9e0b80 20 client.25342 trim_cache size 0 max 16384 2024-05-27T07:41:55.067 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:41:55.068 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:41:55.068 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "425f224b-8d2a-4b8d-a0ec-248d9f1b7a92", 2024-05-27T07:41:55.068 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:41:55.068 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:41:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.066+0000 7fb86ffcb700 1 -- 172.21.15.114:0/2813902569 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8540b29f0 msgr2=0x7fb8540b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.066+0000 7fb86ffcb700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8540b29f0 0x7fb8540b4f00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb868109a10 tx=0x7fb858009260 comp rx=0 tx=0).stop 2024-05-27T07:41:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.066+0000 7fb86ffcb700 1 -- 172.21.15.114:0/2813902569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8681034d0 msgr2=0x7fb8680ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.066+0000 7fb86ffcb700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8681034d0 0x7fb8680ff400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb86400ef10 tx=0x7fb86400c3a0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.066+0000 7fb86ffcb700 1 -- 172.21.15.114:0/2813902569 shutdown_connections 2024-05-27T07:41:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.066+0000 7fb86ffcb700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb868108750 0x7fb8680ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.066+0000 7fb86ffcb700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8681034d0 0x7fb8680ff400 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.066+0000 7fb86ffcb700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8540b29f0 0x7fb8540b4f00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.066+0000 7fb86ffcb700 1 --2- 172.21.15.114:0/2813902569 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb868107e00 0x7fb8680ff940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.066+0000 7fb86ffcb700 1 -- 172.21.15.114:0/2813902569 >> 172.21.15.114:0/2813902569 conn(0x7fb8680fef20 msgr2=0x7fb86806c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.066+0000 7fb86ffcb700 1 -- 172.21.15.114:0/2813902569 shutdown_connections 2024-05-27T07:41:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:55.066+0000 7fb86ffcb700 1 -- 172.21.15.114:0/2813902569 wait complete. 2024-05-27T07:41:55.236 INFO:tasks.workunit.client.0.smithi114.stdout: 1 21404 19.18 MB/sec warmup 7 sec latency 99.599 ms 2024-05-27T07:41:55.714 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:55 smithi114 ceph-mon[32038]: pgmap v68: 129 pgs: 129 active+clean; 4.7 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 318 KiB/s wr, 37 op/s 2024-05-27T07:41:55.714 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:55 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:41:55.714 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:55 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:55.714 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:55 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:41:55.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:55 smithi163 ceph-mon[36451]: pgmap v68: 129 pgs: 129 active+clean; 4.7 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 318 KiB/s wr, 37 op/s 2024-05-27T07:41:55.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:55 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:41:55.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:55 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:55.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:55 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:41:55.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:55 smithi186 ceph-mon[36647]: pgmap v68: 129 pgs: 129 active+clean; 4.7 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 318 KiB/s wr, 37 op/s 2024-05-27T07:41:55.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:55 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:41:55.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:55 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:55.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:55 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:41:55.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:41:55.989 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:56.236 INFO:tasks.workunit.client.0.smithi114.stdout: 1 24085 18.34 MB/sec warmup 8 sec latency 34.736 ms 2024-05-27T07:41:56.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:56 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:56.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:56 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:41:56.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:56 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:56.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:56 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:41:56.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:56 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:56.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:56 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:41:57.236 INFO:tasks.workunit.client.0.smithi114.stdout: 1 26683 17.80 MB/sec warmup 9 sec latency 37.940 ms 2024-05-27T07:41:57.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:57 smithi114 ceph-mon[32038]: pgmap v69: 129 pgs: 129 active+clean; 7.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.0 MiB/s wr, 256 op/s 2024-05-27T07:41:57.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:57 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:41:57.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:57 smithi163 ceph-mon[36451]: pgmap v69: 129 pgs: 129 active+clean; 7.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.0 MiB/s wr, 256 op/s 2024-05-27T07:41:57.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:57 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:41:57.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:57 smithi186 ceph-mon[36647]: pgmap v69: 129 pgs: 129 active+clean; 7.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.0 MiB/s wr, 256 op/s 2024-05-27T07:41:57.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:57 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:41:57.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.813+0000 7f2b994ef700 1 -- 172.21.15.114:0/2191788900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b94108730 msgr2=0x7f2b94110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:57.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.813+0000 7f2b994ef700 1 --2- 172.21.15.114:0/2191788900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b94108730 0x7f2b94110090 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00b0d0 tx=0x7f2b8c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.813+0000 7f2b994ef700 1 -- 172.21.15.114:0/2191788900 shutdown_connections 2024-05-27T07:41:57.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.813+0000 7f2b994ef700 1 --2- 172.21.15.114:0/2191788900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b94108730 0x7f2b94110090 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.813+0000 7f2b994ef700 1 --2- 172.21.15.114:0/2191788900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b941034b0 0x7f2b941038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.813+0000 7f2b994ef700 1 --2- 172.21.15.114:0/2191788900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b94107de0 0x7f2b941081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.813+0000 7f2b994ef700 1 -- 172.21.15.114:0/2191788900 >> 172.21.15.114:0/2191788900 conn(0x7f2b940fedb0 msgr2=0x7f2b941011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:57.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.814+0000 7f2b994ef700 1 -- 172.21.15.114:0/2191788900 shutdown_connections 2024-05-27T07:41:57.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.814+0000 7f2b994ef700 1 -- 172.21.15.114:0/2191788900 wait complete. 2024-05-27T07:41:57.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.814+0000 7f2b994ef700 1 Processor -- start 2024-05-27T07:41:57.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b994ef700 1 -- start start 2024-05-27T07:41:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b994ef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b941034b0 0x7f2b9419ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b994ef700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b94107de0 0x7f2b9419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b994ef700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b94108730 0x7f2b9419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b994ef700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b94113360 con 0x7f2b941034b0 2024-05-27T07:41:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b994ef700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941131e0 con 0x7f2b94107de0 2024-05-27T07:41:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b994ef700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941134e0 con 0x7f2b94108730 2024-05-27T07:41:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b92ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b941034b0 0x7f2b9419ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b92ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b941034b0 0x7f2b9419ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51986/0 (socket says 172.21.15.114:51986) 2024-05-27T07:41:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b92ffd700 1 -- 172.21.15.114:0/1685594067 learned_addr learned my addr 172.21.15.114:0/1685594067 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b937fe700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b94108730 0x7f2b9419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b8a7fc700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b94107de0 0x7f2b9419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b92ffd700 1 -- 172.21.15.114:0/1685594067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b94108730 msgr2=0x7f2b9419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b92ffd700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b94108730 0x7f2b9419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b92ffd700 1 -- 172.21.15.114:0/1685594067 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b94107de0 msgr2=0x7f2b9419b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b92ffd700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b94107de0 0x7f2b9419b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.815+0000 7f2b92ffd700 1 -- 172.21.15.114:0/1685594067 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9419fc50 con 0x7f2b941034b0 2024-05-27T07:41:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.816+0000 7f2b92ffd700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b941034b0 0x7f2b9419ad80 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400e9a0 tx=0x7f2b8400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.816+0000 7f2b90ff9700 1 -- 172.21.15.114:0/1685594067 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b84024070 con 0x7f2b941034b0 2024-05-27T07:41:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.816+0000 7f2b994ef700 1 -- 172.21.15.114:0/1685594067 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b9419ff40 con 0x7f2b941034b0 2024-05-27T07:41:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.816+0000 7f2b90ff9700 1 -- 172.21.15.114:0/1685594067 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b8400cb90 con 0x7f2b941034b0 2024-05-27T07:41:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.816+0000 7f2b90ff9700 1 -- 172.21.15.114:0/1685594067 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8401fbb0 con 0x7f2b941034b0 2024-05-27T07:41:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.816+0000 7f2b994ef700 1 -- 172.21.15.114:0/1685594067 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b940ff1c0 con 0x7f2b941034b0 2024-05-27T07:41:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.819+0000 7f2b90ff9700 1 -- 172.21.15.114:0/1685594067 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f2b84027070 con 0x7f2b941034b0 2024-05-27T07:41:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.819+0000 7f2b90ff9700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b800b29f0 0x7f2b800b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.819+0000 7f2b8a7fc700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b800b29f0 0x7f2b800b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.819+0000 7f2b90ff9700 1 -- 172.21.15.114:0/1685594067 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f2b840eacb0 con 0x7f2b941034b0 2024-05-27T07:41:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.820+0000 7f2ba7f04b80 1 Processor -- start 2024-05-27T07:41:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.820+0000 7f2b8a7fc700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b800b29f0 0x7f2b800b4f00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2b94109820 tx=0x7f2b7c009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.820+0000 7f2ba7f04b80 1 -- start start 2024-05-27T07:41:57.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.820+0000 7f2ba7f04b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 0x55873a69a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.820+0000 7f2ba7f04b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a69ac50 0x55873a694a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.820+0000 7f2ba7f04b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a694f80 0x55873a695390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.820+0000 7f2ba7f04b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55873a5c3c90 con 0x55873a516e40 2024-05-27T07:41:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.820+0000 7f2ba7f04b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55873a5b3830 con 0x55873a694f80 2024-05-27T07:41:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.820+0000 7f2ba7f04b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55873a5b7870 con 0x55873a69ac50 2024-05-27T07:41:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b92ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 0x55873a69a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b8a7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a69ac50 0x55873a694a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b92ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 0x55873a69a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52000/0 (socket says 172.21.15.114:52000) 2024-05-27T07:41:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b92ffd700 1 -- 172.21.15.114:0/381690937 learned_addr learned my addr 172.21.15.114:0/381690937 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b937fe700 1 --2- 172.21.15.114:0/381690937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a694f80 0x55873a695390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b8a7fc700 1 -- 172.21.15.114:0/381690937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a694f80 msgr2=0x55873a695390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:57.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b8a7fc700 1 --2- 172.21.15.114:0/381690937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a694f80 0x55873a695390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b8a7fc700 1 -- 172.21.15.114:0/381690937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 msgr2=0x55873a69a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:57.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b8a7fc700 1 --2- 172.21.15.114:0/381690937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 0x55873a69a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b8a7fc700 1 -- 172.21.15.114:0/381690937 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55873a5038d0 con 0x55873a69ac50 2024-05-27T07:41:57.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b92ffd700 1 --2- 172.21.15.114:0/381690937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 0x55873a69a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:57.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b8a7fc700 1 --2- 172.21.15.114:0/381690937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a69ac50 0x55873a694a40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009fd0 tx=0x7f2b7c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e863f0825100355 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:57.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b897fa700 1 -- 172.21.15.114:0/381690937 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c004020 con 0x55873a69ac50 2024-05-27T07:41:57.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b897fa700 1 -- 172.21.15.114:0/381690937 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b7c00e790 con 0x55873a69ac50 2024-05-27T07:41:57.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.821+0000 7f2b897fa700 1 -- 172.21.15.114:0/381690937 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c003ae0 con 0x55873a69ac50 2024-05-27T07:41:57.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.822+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/381690937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a69ac50 msgr2=0x55873a694a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:57.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.822+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/381690937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a69ac50 0x55873a694a40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009fd0 tx=0x7f2b7c014000 comp rx=0 tx=0).stop 2024-05-27T07:41:57.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.822+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/381690937 shutdown_connections 2024-05-27T07:41:57.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.822+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/381690937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a694f80 0x55873a695390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.822+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/381690937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 0x55873a69a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.822+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/381690937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a69ac50 0x55873a694a40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.822+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/381690937 >> 172.21.15.114:0/381690937 conn(0x55873a571af0 msgr2=0x55873a68f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:57.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.822+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/381690937 shutdown_connections 2024-05-27T07:41:57.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.822+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/381690937 wait complete. 2024-05-27T07:41:57.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2ba7f04b80 1 -- start start 2024-05-27T07:41:57.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2ba7f04b80 1 Processor -- start 2024-05-27T07:41:57.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2ba7f04b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a516e40 0x55873a69e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2ba7f04b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a694f80 0x55873a69e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2ba7f04b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a69ac50 0x55873a69ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2ba7f04b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55873a4b9fa0 con 0x55873a694f80 2024-05-27T07:41:57.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2ba7f04b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55873a5557c0 con 0x55873a516e40 2024-05-27T07:41:57.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2ba7f04b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55873a6a1740 con 0x55873a69ac50 2024-05-27T07:41:57.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2ba7f04b80 10 client.0 _ll_register_callbacks cb 0x55873a54e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:57.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2b92ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a516e40 0x55873a69e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2b8a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a694f80 0x55873a69e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2b92ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a516e40 0x55873a69e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37392/0 (socket says 172.21.15.114:37392) 2024-05-27T07:41:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2b92ffd700 1 -- 172.21.15.114:0/1878524016 learned_addr learned my addr 172.21.15.114:0/1878524016 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.823+0000 7f2b937fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a69ac50 0x55873a69ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2b92ffd700 1 -- 172.21.15.114:0/1878524016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a69ac50 msgr2=0x55873a69ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:57.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2b92ffd700 1 --2- 172.21.15.114:0/1878524016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a69ac50 0x55873a69ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2b92ffd700 1 -- 172.21.15.114:0/1878524016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a694f80 msgr2=0x55873a69e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2b92ffd700 1 --2- 172.21.15.114:0/1878524016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a694f80 0x55873a69e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2b92ffd700 1 -- 172.21.15.114:0/1878524016 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55873a552690 con 0x55873a516e40 2024-05-27T07:41:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2b8a7fc700 1 --2- 172.21.15.114:0/1878524016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a694f80 0x55873a69e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2b92ffd700 1 --2- 172.21.15.114:0/1878524016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a516e40 0x55873a69e3d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2b840e8210 tx=0x7f2b840b1880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5df22ab093da5121 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2b7a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:41:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2b7a7fc700 1 -- 172.21.15.114:0/1878524016 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b840b1cf0 con 0x55873a516e40 2024-05-27T07:41:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2b7a7fc700 1 -- 172.21.15.114:0/1878524016 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f2b840cc380 con 0x55873a516e40 2024-05-27T07:41:57.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/1878524016 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b7c00e790 con 0x55873a516e40 2024-05-27T07:41:57.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2b7a7fc700 1 -- 172.21.15.114:0/1878524016 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2b840b60a0 con 0x55873a516e40 2024-05-27T07:41:57.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2ba7f04b80 10 client.25298 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:57.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/1878524016 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55873a6ac160 con 0x55873a516e40 2024-05-27T07:41:57.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.824+0000 7f2b7a7fc700 1 -- 172.21.15.114:0/1878524016 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b7c00e790 con 0x55873a516e40 2024-05-27T07:41:57.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.825+0000 7f2b7a7fc700 1 -- 172.21.15.114:0/1878524016 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2b840ad660 con 0x55873a516e40 2024-05-27T07:41:57.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.825+0000 7f2ba7f04b80 10 client.25298 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:57.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.825+0000 7f2ba7f04b80 10 client.25298 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:57.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.825+0000 7f2ba7f04b80 20 client.25298 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.825+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/1878524016 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55873a6ae7a0 0x55873a6b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.825+0000 7f2ba7f04b80 4 client.25298 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.825+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/1878524016 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55873a6609f0 con 0x55873a6ae7a0 2024-05-27T07:41:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.825+0000 7f2b8a7fc700 1 --2- 172.21.15.114:0/1878524016 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55873a6ae7a0 0x55873a6b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.825+0000 7f2b7a7fc700 10 client.25298 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.825+0000 7f2b8a7fc700 1 --2- 172.21.15.114:0/1878524016 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55873a6ae7a0 0x55873a6b0c00 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea0c4e0f73844131 server_cookie=d6924a5162549ca5 in_seq=0 out_seq=0 2024-05-27T07:41:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.826+0000 7f2b7a7fc700 1 -- 172.21.15.114:0/1878524016 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55873a6609f0 con 0x55873a6ae7a0 2024-05-27T07:41:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.826+0000 7f2b7a7fc700 10 client.25298 handle_command_reply: tid=0 2024-05-27T07:41:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.826+0000 7f2ba7f04b80 1 client.25298 shutdown 2024-05-27T07:41:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.826+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/1878524016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a516e40 msgr2=0x55873a69e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.826+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/1878524016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a516e40 0x55873a69e3d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2b840e8210 tx=0x7f2b840b1880 comp rx=0 tx=0).stop 2024-05-27T07:41:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.827+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/1878524016 shutdown_connections 2024-05-27T07:41:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.827+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/1878524016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a69ac50 0x55873a69ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.827+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/1878524016 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55873a6ae7a0 0x55873a6b0c00 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.827+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/1878524016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a516e40 0x55873a69e3d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.827+0000 7f2b7a7fc700 0 client.25298 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.827+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/1878524016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a694f80 0x55873a69e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.827+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/1878524016 >> 172.21.15.114:0/1878524016 conn(0x55873a571af0 msgr2=0x55873a693a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.827+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/1878524016 shutdown_connections 2024-05-27T07:41:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.827+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/1878524016 wait complete. 2024-05-27T07:41:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.827+0000 7f2ba7f04b80 20 client.25298 trim_cache size 0 max 16384 2024-05-27T07:41:57.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.847+0000 7f2ba7f04b80 1 Processor -- start 2024-05-27T07:41:57.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.847+0000 7f2ba7f04b80 1 -- start start 2024-05-27T07:41:57.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.847+0000 7f2ba7f04b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 0x55873a694a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2ba7f04b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a694f80 0x55873a6aa7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2ba7f04b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a69ac50 0x55873a69d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2ba7f04b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941ac390 con 0x55873a516e40 2024-05-27T07:41:57.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2ba7f04b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941aec10 con 0x55873a69ac50 2024-05-27T07:41:57.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2ba7f04b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941aca00 con 0x55873a694f80 2024-05-27T07:41:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b92ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 0x55873a694a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b92ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 0x55873a694a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52016/0 (socket says 172.21.15.114:52016) 2024-05-27T07:41:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b92ffd700 1 -- 172.21.15.114:0/2338741122 learned_addr learned my addr 172.21.15.114:0/2338741122 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b937fe700 1 --2- 172.21.15.114:0/2338741122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a69ac50 0x55873a69d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b8a7fc700 1 --2- 172.21.15.114:0/2338741122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a694f80 0x55873a6aa7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b92ffd700 1 -- 172.21.15.114:0/2338741122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a694f80 msgr2=0x55873a6aa7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b92ffd700 1 --2- 172.21.15.114:0/2338741122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a694f80 0x55873a6aa7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b92ffd700 1 -- 172.21.15.114:0/2338741122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a69ac50 msgr2=0x55873a69d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b92ffd700 1 --2- 172.21.15.114:0/2338741122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a69ac50 0x55873a69d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b92ffd700 1 -- 172.21.15.114:0/2338741122 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55873a6959f0 con 0x55873a516e40 2024-05-27T07:41:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b937fe700 1 --2- 172.21.15.114:0/2338741122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a69ac50 0x55873a69d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b92ffd700 1 --2- 172.21.15.114:0/2338741122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 0x55873a694a40 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2b840aef60 tx=0x7f2b840adce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dac77dc35329675 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b88ff9700 1 -- 172.21.15.114:0/2338741122 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b840bab50 con 0x55873a516e40 2024-05-27T07:41:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b88ff9700 1 -- 172.21.15.114:0/2338741122 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b840ba8d0 con 0x55873a516e40 2024-05-27T07:41:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.848+0000 7f2b88ff9700 1 -- 172.21.15.114:0/2338741122 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b840b0f50 con 0x55873a516e40 2024-05-27T07:41:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.849+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/2338741122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 msgr2=0x55873a694a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.849+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/2338741122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 0x55873a694a40 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2b840aef60 tx=0x7f2b840adce0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.849+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/2338741122 shutdown_connections 2024-05-27T07:41:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.849+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/2338741122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a69ac50 0x55873a69d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.849+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/2338741122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a516e40 0x55873a694a40 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.849+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/2338741122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a694f80 0x55873a6aa7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.849+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/2338741122 >> 172.21.15.114:0/2338741122 conn(0x55873a571af0 msgr2=0x55873a69cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.849+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/2338741122 shutdown_connections 2024-05-27T07:41:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.849+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/2338741122 wait complete. 2024-05-27T07:41:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2ba7f04b80 1 -- start start 2024-05-27T07:41:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2ba7f04b80 1 Processor -- start 2024-05-27T07:41:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2ba7f04b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a516e40 0x55873a6b5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2ba7f04b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a694f80 0x55873a6b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2ba7f04b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a69ac50 0x55873a6b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2ba7f04b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941ad0d0 con 0x55873a69ac50 2024-05-27T07:41:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2ba7f04b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941ae540 con 0x55873a516e40 2024-05-27T07:41:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2ba7f04b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941ad7a0 con 0x55873a694f80 2024-05-27T07:41:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2b937fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a69ac50 0x55873a6b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2ba7f04b80 10 client.0 _ll_register_callbacks cb 0x55873a6958b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2b937fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a69ac50 0x55873a6b69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52018/0 (socket says 172.21.15.114:52018) 2024-05-27T07:41:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2b8a7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a694f80 0x55873a6b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2b937fe700 1 -- 172.21.15.114:0/995423730 learned_addr learned my addr 172.21.15.114:0/995423730 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:41:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.850+0000 7f2b92ffd700 1 --2- 172.21.15.114:0/995423730 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a516e40 0x55873a6b5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2b937fe700 1 -- 172.21.15.114:0/995423730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a694f80 msgr2=0x55873a6b64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2b937fe700 1 --2- 172.21.15.114:0/995423730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a694f80 0x55873a6b64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2b937fe700 1 -- 172.21.15.114:0/995423730 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a516e40 msgr2=0x55873a6b5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2b937fe700 1 --2- 172.21.15.114:0/995423730 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a516e40 0x55873a6b5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2b937fe700 1 -- 172.21.15.114:0/995423730 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55873a718290 con 0x55873a69ac50 2024-05-27T07:41:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2b937fe700 1 --2- 172.21.15.114:0/995423730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a69ac50 0x55873a6b69e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00bda0 tx=0x7f2b8c009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5771af5873ab96 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2b7a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:41:57.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2b7a7fc700 1 -- 172.21.15.114:0/995423730 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c02b060 con 0x55873a69ac50 2024-05-27T07:41:57.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2b7a7fc700 1 -- 172.21.15.114:0/995423730 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f2b8c0075c0 con 0x55873a69ac50 2024-05-27T07:41:57.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/995423730 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b840ba8d0 con 0x55873a69ac50 2024-05-27T07:41:57.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2b7a7fc700 1 -- 172.21.15.114:0/995423730 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2b8c003e10 con 0x55873a69ac50 2024-05-27T07:41:57.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2ba7f04b80 10 client.15765 fetch_fsmap learned FSMap version 21 2024-05-27T07:41:57.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/995423730 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55873a7194f0 con 0x55873a69ac50 2024-05-27T07:41:57.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.851+0000 7f2b7a7fc700 1 -- 172.21.15.114:0/995423730 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b840ba8d0 con 0x55873a69ac50 2024-05-27T07:41:57.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.852+0000 7f2b7a7fc700 1 -- 172.21.15.114:0/995423730 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2b8c011070 con 0x55873a69ac50 2024-05-27T07:41:57.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.852+0000 7f2ba7f04b80 10 client.15765 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:41:57.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.852+0000 7f2ba7f04b80 10 client.15765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:41:57.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.852+0000 7f2ba7f04b80 20 client.15765 populate_metadata read hostname 'smithi114' 2024-05-27T07:41:57.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.852+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/995423730 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55873a719c00 0x55873a71c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:57.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.852+0000 7f2ba7f04b80 4 client.15765 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:41:57.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.852+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/995423730 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55873a5c5be0 con 0x55873a719c00 2024-05-27T07:41:57.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.852+0000 7f2b92ffd700 1 --2- 172.21.15.114:0/995423730 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55873a719c00 0x55873a71c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:57.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.852+0000 7f2b92ffd700 1 --2- 172.21.15.114:0/995423730 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55873a719c00 0x55873a71c020 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d144a0e69c48bd3 server_cookie=61ea3ff82c74125e in_seq=0 out_seq=0 2024-05-27T07:41:57.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.852+0000 7f2b7a7fc700 10 client.15765 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:57.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.852+0000 7f2b7a7fc700 1 -- 172.21.15.114:0/995423730 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55873a5c5be0 con 0x55873a719c00 2024-05-27T07:41:57.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.852+0000 7f2b7a7fc700 10 client.15765 handle_command_reply: tid=0 2024-05-27T07:41:57.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.852+0000 7f2ba7f04b80 1 client.15765 shutdown 2024-05-27T07:41:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.853+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/995423730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a69ac50 msgr2=0x55873a6b69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.853+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/995423730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a69ac50 0x55873a6b69e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00bda0 tx=0x7f2b8c009e20 comp rx=0 tx=0).stop 2024-05-27T07:41:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.853+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/995423730 shutdown_connections 2024-05-27T07:41:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.853+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/995423730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55873a69ac50 0x55873a6b69e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.853+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/995423730 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55873a719c00 0x55873a71c020 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.853+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/995423730 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55873a516e40 0x55873a6b5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.853+0000 7f2ba7f04b80 1 --2- 172.21.15.114:0/995423730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55873a694f80 0x55873a6b64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.853+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/995423730 >> 172.21.15.114:0/995423730 conn(0x55873a571af0 msgr2=0x55873a6987a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.853+0000 7f2b7a7fc700 0 client.15765 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:41:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.853+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/995423730 shutdown_connections 2024-05-27T07:41:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.853+0000 7f2ba7f04b80 1 -- 172.21.15.114:0/995423730 wait complete. 2024-05-27T07:41:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.853+0000 7f2ba7f04b80 20 client.15765 trim_cache size 0 max 16384 2024-05-27T07:41:57.858 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:41:57.858 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:41:57.858 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:41:57.859 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:41:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.853+0000 7f2b994ef700 1 -- 172.21.15.114:0/1685594067 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b800b29f0 msgr2=0x7f2b800b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:57.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.854+0000 7f2b994ef700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b800b29f0 0x7f2b800b4f00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2b94109820 tx=0x7f2b7c009260 comp rx=0 tx=0).stop 2024-05-27T07:41:57.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.854+0000 7f2b994ef700 1 -- 172.21.15.114:0/1685594067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b941034b0 msgr2=0x7f2b9419ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:57.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.854+0000 7f2b994ef700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b941034b0 0x7f2b9419ad80 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400e9a0 tx=0x7f2b8400ecb0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.854+0000 7f2b994ef700 1 -- 172.21.15.114:0/1685594067 shutdown_connections 2024-05-27T07:41:57.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.854+0000 7f2b994ef700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b94108730 0x7f2b9419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.854+0000 7f2b994ef700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b941034b0 0x7f2b9419ad80 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.854+0000 7f2b994ef700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b800b29f0 0x7f2b800b4f00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.854+0000 7f2b994ef700 1 --2- 172.21.15.114:0/1685594067 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b94107de0 0x7f2b9419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:57.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.854+0000 7f2b994ef700 1 -- 172.21.15.114:0/1685594067 >> 172.21.15.114:0/1685594067 conn(0x7f2b940fedb0 msgr2=0x7f2b9406b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:57.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.854+0000 7f2b994ef700 1 -- 172.21.15.114:0/1685594067 shutdown_connections 2024-05-27T07:41:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:41:57.854+0000 7f2b994ef700 1 -- 172.21.15.114:0/1685594067 wait complete. 2024-05-27T07:41:58.236 INFO:tasks.workunit.client.0.smithi114.stdout: 1 29396 17.48 MB/sec warmup 10 sec latency 38.927 ms 2024-05-27T07:41:58.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:58 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:41:58.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:58 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:41:58.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:58 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:41:58.679 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:41:58.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:41:58.723 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:58 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:41:58.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:58 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:41:58.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:58 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:41:58.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:58 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:41:58.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:58 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:41:58.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:58 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:41:58.926 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:59.236 INFO:tasks.workunit.client.0.smithi114.stdout: 1 32631 17.56 MB/sec warmup 11 sec latency 23.690 ms 2024-05-27T07:41:59.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:41:59 smithi114 ceph-mon[32038]: pgmap v70: 129 pgs: 129 active+clean; 7.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 1.0 MiB/s wr, 254 op/s 2024-05-27T07:41:59.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:41:59 smithi163 ceph-mon[36451]: pgmap v70: 129 pgs: 129 active+clean; 7.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 1.0 MiB/s wr, 254 op/s 2024-05-27T07:41:59.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:41:59 smithi186 ceph-mon[36647]: pgmap v70: 129 pgs: 129 active+clean; 7.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 1.0 MiB/s wr, 254 op/s 2024-05-27T07:42:00.236 INFO:tasks.workunit.client.0.smithi114.stdout: 1 35895 17.68 MB/sec warmup 12 sec latency 25.027 ms 2024-05-27T07:42:00.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.712+0000 7f4f1a96f700 1 -- 172.21.15.114:0/1372338574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f141028d0 msgr2=0x7f4f14102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:00.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.712+0000 7f4f1a96f700 1 --2- 172.21.15.114:0/1372338574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f141028d0 0x7f4f14102ce0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4efc009990 tx=0x7f4efc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.712+0000 7f4f1a96f700 1 -- 172.21.15.114:0/1372338574 shutdown_connections 2024-05-27T07:42:00.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.712+0000 7f4f1a96f700 1 --2- 172.21.15.114:0/1372338574 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f14103220 0x7f4f14111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.712+0000 7f4f1a96f700 1 --2- 172.21.15.114:0/1372338574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f141092c0 0x7f4f141096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.712+0000 7f4f1a96f700 1 --2- 172.21.15.114:0/1372338574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f141028d0 0x7f4f14102ce0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.712+0000 7f4f1a96f700 1 -- 172.21.15.114:0/1372338574 >> 172.21.15.114:0/1372338574 conn(0x7f4f140fe6e0 msgr2=0x7f4f14100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:00.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.712+0000 7f4f1a96f700 1 -- 172.21.15.114:0/1372338574 shutdown_connections 2024-05-27T07:42:00.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.712+0000 7f4f1a96f700 1 -- 172.21.15.114:0/1372338574 wait complete. 2024-05-27T07:42:00.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.713+0000 7f4f1a96f700 1 Processor -- start 2024-05-27T07:42:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.713+0000 7f4f1a96f700 1 -- start start 2024-05-27T07:42:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.713+0000 7f4f1a96f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f141028d0 0x7f4f14103ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.713+0000 7f4f1a96f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f14103220 0x7f4f14103ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.713+0000 7f4f1a96f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f141092c0 0x7f4f14104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.713+0000 7f4f1a96f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4f14114580 con 0x7f4f141028d0 2024-05-27T07:42:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.713+0000 7f4f1a96f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4f14114400 con 0x7f4f14103220 2024-05-27T07:42:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.713+0000 7f4f1a96f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4f14114700 con 0x7f4f141092c0 2024-05-27T07:42:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.713+0000 7f4f13fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f141028d0 0x7f4f14103ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.713+0000 7f4f0bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f14103220 0x7f4f14103ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.713+0000 7f4f13fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f141028d0 0x7f4f14103ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52038/0 (socket says 172.21.15.114:52038) 2024-05-27T07:42:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.713+0000 7f4f13fff700 1 -- 172.21.15.114:0/3842853138 learned_addr learned my addr 172.21.15.114:0/3842853138 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.713+0000 7f4f18f0c700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f141092c0 0x7f4f14104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.714+0000 7f4f18f0c700 1 -- 172.21.15.114:0/3842853138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f14103220 msgr2=0x7f4f14103ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.714+0000 7f4f18f0c700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f14103220 0x7f4f14103ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.714+0000 7f4f18f0c700 1 -- 172.21.15.114:0/3842853138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f141028d0 msgr2=0x7f4f14103ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.714+0000 7f4f18f0c700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f141028d0 0x7f4f14103ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.714+0000 7f4f18f0c700 1 -- 172.21.15.114:0/3842853138 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f141a9290 con 0x7f4f141092c0 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.714+0000 7f4f13fff700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f141028d0 0x7f4f14103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.714+0000 7f4f0bfff700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f14103220 0x7f4f14103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.714+0000 7f4f18f0c700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f141092c0 0x7f4f14104530 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00efc0 tx=0x7f4f0c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.714+0000 7f4f11ffb700 1 -- 172.21.15.114:0/3842853138 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c0092e0 con 0x7f4f141092c0 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.714+0000 7f4f1a96f700 1 -- 172.21.15.114:0/3842853138 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f141a9520 con 0x7f4f141092c0 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.714+0000 7f4f11ffb700 1 -- 172.21.15.114:0/3842853138 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f0c004bb0 con 0x7f4f141092c0 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.714+0000 7f4f11ffb700 1 -- 172.21.15.114:0/3842853138 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c004020 con 0x7f4f141092c0 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.714+0000 7f4f1a96f700 1 -- 172.21.15.114:0/3842853138 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f141a99e0 con 0x7f4f141092c0 2024-05-27T07:42:00.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.715+0000 7f4f1a96f700 1 -- 172.21.15.114:0/3842853138 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f140512b0 con 0x7f4f141092c0 2024-05-27T07:42:00.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.717+0000 7f4f11ffb700 1 -- 172.21.15.114:0/3842853138 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f4f0c01b030 con 0x7f4f141092c0 2024-05-27T07:42:00.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.718+0000 7f4f11ffb700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4f000b29f0 0x7f4f000b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:00.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.718+0000 7f4f13fff700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4f000b29f0 0x7f4f000b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:00.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.718+0000 7f4f11ffb700 1 -- 172.21.15.114:0/3842853138 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f4f0c0e2640 con 0x7f4f141092c0 2024-05-27T07:42:00.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.718+0000 7f4f13fff700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4f000b29f0 0x7f4f000b4f00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4f04005950 tx=0x7f4f04009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:00.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.721+0000 7f4f11ffb700 1 -- 172.21.15.114:0/3842853138 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4f0c022020 con 0x7f4f141092c0 2024-05-27T07:42:00.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.995+0000 7f4f1a96f700 1 -- 172.21.15.114:0/3842853138 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f1402e5e0 con 0x7f4f141092c0 2024-05-27T07:42:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.996+0000 7f4f11ffb700 1 -- 172.21.15.114:0/3842853138 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11827 (secure 0 0 0) 0x7f4f0c010040 con 0x7f4f141092c0 2024-05-27T07:42:00.995 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:42:00.995 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:37:06.645274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3774280856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3774280856},{"type":"v1","addr":"172.21.15.163:6839","nonce":3774280856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:42:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.999+0000 7f4f1a96f700 1 -- 172.21.15.114:0/3842853138 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4f000b29f0 msgr2=0x7f4f000b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.999+0000 7f4f1a96f700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4f000b29f0 0x7f4f000b4f00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4f04005950 tx=0x7f4f04009260 comp rx=0 tx=0).stop 2024-05-27T07:42:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.999+0000 7f4f1a96f700 1 -- 172.21.15.114:0/3842853138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f141092c0 msgr2=0x7f4f14104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.999+0000 7f4f1a96f700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f141092c0 0x7f4f14104530 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00efc0 tx=0x7f4f0c00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.999+0000 7f4f1a96f700 1 -- 172.21.15.114:0/3842853138 shutdown_connections 2024-05-27T07:42:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.999+0000 7f4f1a96f700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f141092c0 0x7f4f14104530 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.999+0000 7f4f1a96f700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f141028d0 0x7f4f14103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.999+0000 7f4f1a96f700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4f000b29f0 0x7f4f000b4f00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.999+0000 7f4f1a96f700 1 --2- 172.21.15.114:0/3842853138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f14103220 0x7f4f14103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.999+0000 7f4f1a96f700 1 -- 172.21.15.114:0/3842853138 >> 172.21.15.114:0/3842853138 conn(0x7f4f140fe6e0 msgr2=0x7f4f1410a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.999+0000 7f4f1a96f700 1 -- 172.21.15.114:0/3842853138 shutdown_connections 2024-05-27T07:42:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:00.999+0000 7f4f1a96f700 1 -- 172.21.15.114:0/3842853138 wait complete. 2024-05-27T07:42:00.999 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-05-27T07:42:01.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 36542 16.46 MB/sec warmup 13 sec latency 53.999 ms 2024-05-27T07:42:01.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-mon[36451]: pgmap v71: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 1.7 MiB/s wr, 360 op/s 2024-05-27T07:42:01.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3842853138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:42:01.724 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7a61d63700 time 2024-05-27T07:42:01.450008+0000 2024-05-27T07:42:01.724 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:42:01.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:01.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7a745a8ba6] 2024-05-27T07:42:01.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7a745a8dd1] 2024-05-27T07:42:01.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6ed6a923d] 2024-05-27T07:42:01.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c6ed694971] 2024-05-27T07:42:01.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c6ed692fa0] 2024-05-27T07:42:01.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6ed693651] 2024-05-27T07:42:01.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6ed628347] 2024-05-27T07:42:01.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6ed32487e] 2024-05-27T07:42:01.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c6ed329e94] 2024-05-27T07:42:01.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c6ed334969] 2024-05-27T07:42:01.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c6ed6cd48c] 2024-05-27T07:42:01.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c6ed6d69cb] 2024-05-27T07:42:01.726 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 13: (Thread::entry_wrapper()+0x43) [0x7f7a7457772d] 2024-05-27T07:42:01.726 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7a74577749] 2024-05-27T07:42:01.726 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7a7311b1ca] 2024-05-27T07:42:01.726 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 16: clone() 2024-05-27T07:42:01.726 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.450+0000 7f7a61d63700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7a61d63700 time 2024-05-27T07:42:01.450008+0000 2024-05-27T07:42:01.727 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:42:01.727 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.727 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:01.727 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7a745a8ba6] 2024-05-27T07:42:01.727 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7a745a8dd1] 2024-05-27T07:42:01.727 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6ed6a923d] 2024-05-27T07:42:01.727 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c6ed694971] 2024-05-27T07:42:01.727 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c6ed692fa0] 2024-05-27T07:42:01.727 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6ed693651] 2024-05-27T07:42:01.727 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6ed628347] 2024-05-27T07:42:01.728 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6ed32487e] 2024-05-27T07:42:01.728 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c6ed329e94] 2024-05-27T07:42:01.728 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c6ed334969] 2024-05-27T07:42:01.728 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c6ed6cd48c] 2024-05-27T07:42:01.728 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c6ed6d69cb] 2024-05-27T07:42:01.728 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 13: (Thread::entry_wrapper()+0x43) [0x7f7a7457772d] 2024-05-27T07:42:01.728 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7a74577749] 2024-05-27T07:42:01.728 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7a7311b1ca] 2024-05-27T07:42:01.728 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 16: clone() 2024-05-27T07:42:01.728 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.728 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: *** Caught signal (Aborted) ** 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: in thread 7f7a61d63700 thread_name:md_log_replay 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7a73125d20] 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2: gsignal() 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 3: abort() 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7a745a8ced] 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7a745a8dd1] 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6ed6a923d] 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c6ed694971] 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c6ed692fa0] 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6ed693651] 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6ed628347] 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6ed32487e] 2024-05-27T07:42:01.729 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c6ed329e94] 2024-05-27T07:42:01.730 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c6ed334969] 2024-05-27T07:42:01.730 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c6ed6cd48c] 2024-05-27T07:42:01.730 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c6ed6d69cb] 2024-05-27T07:42:01.730 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 16: (Thread::entry_wrapper()+0x43) [0x7f7a7457772d] 2024-05-27T07:42:01.730 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7a74577749] 2024-05-27T07:42:01.730 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7a7311b1ca] 2024-05-27T07:42:01.730 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 19: clone() 2024-05-27T07:42:01.730 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.453+0000 7f7a61d63700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:42:01.730 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: in thread 7f7a61d63700 thread_name:md_log_replay 2024-05-27T07:42:01.730 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.730 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:01.730 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7a73125d20] 2024-05-27T07:42:01.730 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2: gsignal() 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 3: abort() 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7a745a8ced] 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7a745a8dd1] 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6ed6a923d] 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c6ed694971] 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c6ed692fa0] 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6ed693651] 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6ed628347] 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6ed32487e] 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c6ed329e94] 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c6ed334969] 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c6ed6cd48c] 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c6ed6d69cb] 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 16: (Thread::entry_wrapper()+0x43) [0x7f7a7457772d] 2024-05-27T07:42:01.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7a74577749] 2024-05-27T07:42:01.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7a7311b1ca] 2024-05-27T07:42:01.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 19: clone() 2024-05-27T07:42:01.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:42:01.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: -1> 2024-05-27T07:42:01.450+0000 7f7a61d63700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7a61d63700 time 2024-05-27T07:42:01.450008+0000 2024-05-27T07:42:01.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:42:01.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:01.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7a745a8ba6] 2024-05-27T07:42:01.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7a745a8dd1] 2024-05-27T07:42:01.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6ed6a923d] 2024-05-27T07:42:01.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c6ed694971] 2024-05-27T07:42:01.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c6ed692fa0] 2024-05-27T07:42:01.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6ed693651] 2024-05-27T07:42:01.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6ed628347] 2024-05-27T07:42:01.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6ed32487e] 2024-05-27T07:42:01.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c6ed329e94] 2024-05-27T07:42:01.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c6ed334969] 2024-05-27T07:42:01.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c6ed6cd48c] 2024-05-27T07:42:01.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c6ed6d69cb] 2024-05-27T07:42:01.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 13: (Thread::entry_wrapper()+0x43) [0x7f7a7457772d] 2024-05-27T07:42:01.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7a74577749] 2024-05-27T07:42:01.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7a7311b1ca] 2024-05-27T07:42:01.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 16: clone() 2024-05-27T07:42:01.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.734 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 0> 2024-05-27T07:42:01.453+0000 7f7a61d63700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:42:01.734 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: in thread 7f7a61d63700 thread_name:md_log_replay 2024-05-27T07:42:01.734 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.734 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:01.734 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7a73125d20] 2024-05-27T07:42:01.734 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2: gsignal() 2024-05-27T07:42:01.734 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 3: abort() 2024-05-27T07:42:01.734 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7a745a8ced] 2024-05-27T07:42:01.734 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7a745a8dd1] 2024-05-27T07:42:01.735 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6ed6a923d] 2024-05-27T07:42:01.735 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c6ed694971] 2024-05-27T07:42:01.735 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c6ed692fa0] 2024-05-27T07:42:01.735 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6ed693651] 2024-05-27T07:42:01.735 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6ed628347] 2024-05-27T07:42:01.735 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6ed32487e] 2024-05-27T07:42:01.735 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c6ed329e94] 2024-05-27T07:42:01.735 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c6ed334969] 2024-05-27T07:42:01.735 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c6ed6cd48c] 2024-05-27T07:42:01.735 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c6ed6d69cb] 2024-05-27T07:42:01.735 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 16: (Thread::entry_wrapper()+0x43) [0x7f7a7457772d] 2024-05-27T07:42:01.736 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7a74577749] 2024-05-27T07:42:01.736 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7a7311b1ca] 2024-05-27T07:42:01.736 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 19: clone() 2024-05-27T07:42:01.736 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:42:01.736 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.736 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: -1> 2024-05-27T07:42:01.450+0000 7f7a61d63700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7a61d63700 time 2024-05-27T07:42:01.450008+0000 2024-05-27T07:42:01.736 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:42:01.736 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.737 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:01.737 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7a745a8ba6] 2024-05-27T07:42:01.737 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7a745a8dd1] 2024-05-27T07:42:01.737 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6ed6a923d] 2024-05-27T07:42:01.737 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c6ed694971] 2024-05-27T07:42:01.737 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c6ed692fa0] 2024-05-27T07:42:01.737 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6ed693651] 2024-05-27T07:42:01.737 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6ed628347] 2024-05-27T07:42:01.737 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6ed32487e] 2024-05-27T07:42:01.737 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c6ed329e94] 2024-05-27T07:42:01.737 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c6ed334969] 2024-05-27T07:42:01.737 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c6ed6cd48c] 2024-05-27T07:42:01.738 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c6ed6d69cb] 2024-05-27T07:42:01.738 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 13: (Thread::entry_wrapper()+0x43) [0x7f7a7457772d] 2024-05-27T07:42:01.738 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7a74577749] 2024-05-27T07:42:01.738 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7a7311b1ca] 2024-05-27T07:42:01.738 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 16: clone() 2024-05-27T07:42:01.738 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.738 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 0> 2024-05-27T07:42:01.453+0000 7f7a61d63700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:42:01.738 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: in thread 7f7a61d63700 thread_name:md_log_replay 2024-05-27T07:42:01.738 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.738 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:01.738 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7a73125d20] 2024-05-27T07:42:01.738 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2: gsignal() 2024-05-27T07:42:01.739 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 3: abort() 2024-05-27T07:42:01.739 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7a745a8ced] 2024-05-27T07:42:01.739 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7a745a8dd1] 2024-05-27T07:42:01.739 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6ed6a923d] 2024-05-27T07:42:01.739 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c6ed694971] 2024-05-27T07:42:01.739 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c6ed692fa0] 2024-05-27T07:42:01.739 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6ed693651] 2024-05-27T07:42:01.739 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6ed628347] 2024-05-27T07:42:01.739 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6ed32487e] 2024-05-27T07:42:01.739 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c6ed329e94] 2024-05-27T07:42:01.739 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c6ed334969] 2024-05-27T07:42:01.739 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c6ed6cd48c] 2024-05-27T07:42:01.740 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c6ed6d69cb] 2024-05-27T07:42:01.740 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 16: (Thread::entry_wrapper()+0x43) [0x7f7a7457772d] 2024-05-27T07:42:01.740 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7a74577749] 2024-05-27T07:42:01.740 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7a7311b1ca] 2024-05-27T07:42:01.740 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 19: clone() 2024-05-27T07:42:01.740 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:42:01.740 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[66323]: 2024-05-27T07:42:01.740 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:01 smithi163 podman[70111]: 2024-05-27 07:42:01.516230801 +0000 UTC m=+0.019616894 container died 9e612e37bd4f6b4935012bee7ba5a2727fb911092a3e708a73ca42252ef53211 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:42:01.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:01 smithi114 ceph-mon[32038]: pgmap v71: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 1.7 MiB/s wr, 360 op/s 2024-05-27T07:42:01.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:01 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3842853138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:42:01.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:42:01.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:01 smithi186 ceph-mon[36647]: pgmap v71: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 1.7 MiB/s wr, 360 op/s 2024-05-27T07:42:01.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:01 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3842853138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:42:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:02.237 INFO:tasks.workunit.client.0.smithi114.stdout: 1 39523 16.52 MB/sec warmup 14 sec latency 91.085 ms 2024-05-27T07:42:02.524 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:02 smithi163 podman[70111]: 2024-05-27 07:42:02.27456795 +0000 UTC m=+0.777954035 container remove 9e612e37bd4f6b4935012bee7ba5a2727fb911092a3e708a73ca42252ef53211 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2) 2024-05-27T07:42:02.524 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:02 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:42:02.974 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:02 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-05-27T07:42:02.974 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:02 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Consumed 1.992s CPU time. 2024-05-27T07:42:03.237 INFO:tasks.workunit.client.0.smithi114.stdout: 1 42839 16.67 MB/sec warmup 15 sec latency 13.124 ms 2024-05-27T07:42:03.662 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:03 smithi114 ceph-mon[32038]: pgmap v72: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.4 MiB/s wr, 325 op/s 2024-05-27T07:42:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.912+0000 7f79acc11700 1 -- 172.21.15.114:0/2684658732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a81034d0 msgr2=0x7f79a81038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.912+0000 7f79acc11700 1 --2- 172.21.15.114:0/2684658732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a81034d0 0x7f79a81038e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f79a000b0d0 tx=0x7f79a000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.912+0000 7f79acc11700 1 -- 172.21.15.114:0/2684658732 shutdown_connections 2024-05-27T07:42:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.912+0000 7f79acc11700 1 --2- 172.21.15.114:0/2684658732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a8108750 0x7f79a81100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.912+0000 7f79acc11700 1 --2- 172.21.15.114:0/2684658732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a81034d0 0x7f79a81038e0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.912+0000 7f79acc11700 1 --2- 172.21.15.114:0/2684658732 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79a8107e00 0x7f79a8108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.912+0000 7f79acc11700 1 -- 172.21.15.114:0/2684658732 >> 172.21.15.114:0/2684658732 conn(0x7f79a80fef20 msgr2=0x7f79a8101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:03.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.912+0000 7f79acc11700 1 -- 172.21.15.114:0/2684658732 shutdown_connections 2024-05-27T07:42:03.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.912+0000 7f79acc11700 1 -- 172.21.15.114:0/2684658732 wait complete. 2024-05-27T07:42:03.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79acc11700 1 Processor -- start 2024-05-27T07:42:03.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79acc11700 1 -- start start 2024-05-27T07:42:03.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79acc11700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79a81034d0 0x7f79a819bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79acc11700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a8107e00 0x7f79a819c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79acc11700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a8108750 0x7f79a819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79acc11700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79a8113380 con 0x7f79a8108750 2024-05-27T07:42:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79acc11700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79a8113200 con 0x7f79a81034d0 2024-05-27T07:42:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79acc11700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79a8113500 con 0x7f79a8107e00 2024-05-27T07:42:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79a6d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a8108750 0x7f79a819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79a659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79a81034d0 0x7f79a819bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79a5d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a8107e00 0x7f79a819c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79a6d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a8108750 0x7f79a819c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52048/0 (socket says 172.21.15.114:52048) 2024-05-27T07:42:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79a659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79a81034d0 0x7f79a819bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37436/0 (socket says 172.21.15.114:37436) 2024-05-27T07:42:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.913+0000 7f79a6d9d700 1 -- 172.21.15.114:0/390106675 learned_addr learned my addr 172.21.15.114:0/390106675 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.914+0000 7f79a6d9d700 1 -- 172.21.15.114:0/390106675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a8107e00 msgr2=0x7f79a819c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.914+0000 7f79a6d9d700 1 --2- 172.21.15.114:0/390106675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a8107e00 0x7f79a819c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.914+0000 7f79a6d9d700 1 -- 172.21.15.114:0/390106675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79a81034d0 msgr2=0x7f79a819bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.914+0000 7f79a6d9d700 1 --2- 172.21.15.114:0/390106675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79a81034d0 0x7f79a819bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.914+0000 7f79a6d9d700 1 -- 172.21.15.114:0/390106675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a81a6ff0 con 0x7f79a8108750 2024-05-27T07:42:03.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.914+0000 7f79a5d9b700 1 --2- 172.21.15.114:0/390106675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a8107e00 0x7f79a819c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:03.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.914+0000 7f79a6d9d700 1 --2- 172.21.15.114:0/390106675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a8108750 0x7f79a819c740 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f799800e940 tx=0x7f799800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:03.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.914+0000 7f799f7fe700 1 -- 172.21.15.114:0/390106675 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7998004a30 con 0x7f79a8108750 2024-05-27T07:42:03.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.914+0000 7f799f7fe700 1 -- 172.21.15.114:0/390106675 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79980054d0 con 0x7f79a8108750 2024-05-27T07:42:03.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.914+0000 7f79acc11700 1 -- 172.21.15.114:0/390106675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a81a72e0 con 0x7f79a8108750 2024-05-27T07:42:03.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.914+0000 7f799f7fe700 1 -- 172.21.15.114:0/390106675 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799800ccf0 con 0x7f79a8108750 2024-05-27T07:42:03.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.914+0000 7f79acc11700 1 -- 172.21.15.114:0/390106675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a81a77d0 con 0x7f79a8108750 2024-05-27T07:42:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.917+0000 7f799f7fe700 1 -- 172.21.15.114:0/390106675 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f7998005020 con 0x7f79a8108750 2024-05-27T07:42:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.918+0000 7f799f7fe700 1 --2- 172.21.15.114:0/390106675 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f79940b29f0 0x7f79940b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.918+0000 7f79a659c700 1 --2- 172.21.15.114:0/390106675 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f79940b29f0 0x7f79940b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.918+0000 7f799f7fe700 1 -- 172.21.15.114:0/390106675 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f79980e2af0 con 0x7f79a8108750 2024-05-27T07:42:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.919+0000 7f79a659c700 1 --2- 172.21.15.114:0/390106675 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f79940b29f0 0x7f79940b4f00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f79a0009fd0 tx=0x7f79a000b040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.919+0000 7f79bb626b80 1 Processor -- start 2024-05-27T07:42:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79bb626b80 1 -- start start 2024-05-27T07:42:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79bb626b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 0x562b787f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79bb626b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787f4070 0x562b787edbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79bb626b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787ee120 0x562b787ee530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79bb626b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562b7871bd70 con 0x562b7866ee40 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79bb626b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562b7870b910 con 0x562b787ee120 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79bb626b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562b7870f950 con 0x562b787f4070 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79a659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 0x562b787f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79a659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 0x562b787f3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52064/0 (socket says 172.21.15.114:52064) 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79a659c700 1 -- 172.21.15.114:0/4055184357 learned_addr learned my addr 172.21.15.114:0/4055184357 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79a5d9b700 1 --2- 172.21.15.114:0/4055184357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787f4070 0x562b787edbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79a6d9d700 1 --2- 172.21.15.114:0/4055184357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787ee120 0x562b787ee530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79a659c700 1 -- 172.21.15.114:0/4055184357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787f4070 msgr2=0x562b787edbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79a659c700 1 --2- 172.21.15.114:0/4055184357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787f4070 0x562b787edbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79a659c700 1 -- 172.21.15.114:0/4055184357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787ee120 msgr2=0x562b787ee530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79a659c700 1 --2- 172.21.15.114:0/4055184357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787ee120 0x562b787ee530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.920+0000 7f79a659c700 1 -- 172.21.15.114:0/4055184357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b786b4fb0 con 0x562b7866ee40 2024-05-27T07:42:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.921+0000 7f79a659c700 1 --2- 172.21.15.114:0/4055184357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 0x562b787f3b30 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f79a000baf0 tx=0x7f79a001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beeecf02e4f930e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.921+0000 7f799cff9700 1 -- 172.21.15.114:0/4055184357 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a002c070 con 0x562b7866ee40 2024-05-27T07:42:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.921+0000 7f799cff9700 1 -- 172.21.15.114:0/4055184357 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79a0007c70 con 0x562b7866ee40 2024-05-27T07:42:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.921+0000 7f799cff9700 1 -- 172.21.15.114:0/4055184357 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0022070 con 0x562b7866ee40 2024-05-27T07:42:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.922+0000 7f79bb626b80 1 -- 172.21.15.114:0/4055184357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 msgr2=0x562b787f3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.922+0000 7f79bb626b80 1 --2- 172.21.15.114:0/4055184357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 0x562b787f3b30 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f79a000baf0 tx=0x7f79a001e000 comp rx=0 tx=0).stop 2024-05-27T07:42:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.922+0000 7f79bb626b80 1 -- 172.21.15.114:0/4055184357 shutdown_connections 2024-05-27T07:42:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.922+0000 7f79bb626b80 1 --2- 172.21.15.114:0/4055184357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787ee120 0x562b787ee530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.922+0000 7f79bb626b80 1 --2- 172.21.15.114:0/4055184357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 0x562b787f3b30 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.922+0000 7f79bb626b80 1 --2- 172.21.15.114:0/4055184357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787f4070 0x562b787edbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.922+0000 7f79bb626b80 1 -- 172.21.15.114:0/4055184357 >> 172.21.15.114:0/4055184357 conn(0x562b78671770 msgr2=0x562b787e8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.922+0000 7f79bb626b80 1 -- 172.21.15.114:0/4055184357 shutdown_connections 2024-05-27T07:42:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.922+0000 7f79bb626b80 1 -- 172.21.15.114:0/4055184357 wait complete. 2024-05-27T07:42:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.922+0000 7f79bb626b80 1 -- start start 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.922+0000 7f79bb626b80 1 Processor -- start 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79bb626b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 0x562b787f7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79bb626b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787ee120 0x562b787f7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79bb626b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787f4070 0x562b787f8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79bb626b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562b78611fa0 con 0x562b7866ee40 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79bb626b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562b78673da0 con 0x562b787f4070 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79bb626b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562b787fab00 con 0x562b787ee120 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79bb626b80 10 client.0 _ll_register_callbacks cb 0x562b786a25b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79a659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 0x562b787f7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79a5d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787ee120 0x562b787f7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79a659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 0x562b787f7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52080/0 (socket says 172.21.15.114:52080) 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79a659c700 1 -- 172.21.15.114:0/2930086144 learned_addr learned my addr 172.21.15.114:0/2930086144 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79a5d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787ee120 0x562b787f7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34648/0 (socket says 172.21.15.114:34648) 2024-05-27T07:42:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79a6d9d700 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787f4070 0x562b787f8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79a659c700 1 -- 172.21.15.114:0/2930086144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787ee120 msgr2=0x562b787f7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79a659c700 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787ee120 0x562b787f7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79a659c700 1 -- 172.21.15.114:0/2930086144 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787f4070 msgr2=0x562b787f8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79a659c700 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787f4070 0x562b787f8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79a659c700 1 -- 172.21.15.114:0/2930086144 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b7865b8d0 con 0x562b7866ee40 2024-05-27T07:42:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79a6d9d700 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787f4070 0x562b787f8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.923+0000 7f79a5d9b700 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787ee120 0x562b787f7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.924+0000 7f79a659c700 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 0x562b787f7810 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f79a000b770 tx=0x7f79a0028ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ed62a49e18d6edb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.924+0000 7f798dffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.924+0000 7f798dffb700 1 -- 172.21.15.114:0/2930086144 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a002c070 con 0x562b7866ee40 2024-05-27T07:42:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.924+0000 7f798dffb700 1 -- 172.21.15.114:0/2930086144 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f79a00079d0 con 0x562b7866ee40 2024-05-27T07:42:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.924+0000 7f79bb626b80 1 -- 172.21.15.114:0/2930086144 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79a0007c70 con 0x562b7866ee40 2024-05-27T07:42:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.924+0000 7f798dffb700 1 -- 172.21.15.114:0/2930086144 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f79a0039440 con 0x562b7866ee40 2024-05-27T07:42:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.924+0000 7f79bb626b80 10 client.15783 fetch_fsmap learned FSMap version 21 2024-05-27T07:42:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.924+0000 7f79bb626b80 1 -- 172.21.15.114:0/2930086144 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562b786c5760 con 0x562b7866ee40 2024-05-27T07:42:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.924+0000 7f798dffb700 1 -- 172.21.15.114:0/2930086144 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79a0007c70 con 0x562b7866ee40 2024-05-27T07:42:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.925+0000 7f798dffb700 1 -- 172.21.15.114:0/2930086144 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f79a0011070 con 0x562b7866ee40 2024-05-27T07:42:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.925+0000 7f79bb626b80 10 client.15783 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:42:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.925+0000 7f79bb626b80 10 client.15783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.925+0000 7f79bb626b80 20 client.15783 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.925+0000 7f79bb626b80 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b788078a0 0x562b78809cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.925+0000 7f79bb626b80 4 client.15783 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.925+0000 7f79bb626b80 1 -- 172.21.15.114:0/2930086144 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b7871dcc0 con 0x562b788078a0 2024-05-27T07:42:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.925+0000 7f79a5d9b700 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b788078a0 0x562b78809cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.925+0000 7f798dffb700 10 client.15783 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.925+0000 7f79a5d9b700 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b788078a0 0x562b78809cc0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c0e6f94bf4640f9 server_cookie=d89a3a6c887dd788 in_seq=0 out_seq=0 2024-05-27T07:42:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f798dffb700 1 -- 172.21.15.114:0/2930086144 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562b7871dcc0 con 0x562b788078a0 2024-05-27T07:42:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f798dffb700 10 client.15783 handle_command_reply: tid=0 2024-05-27T07:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f79bb626b80 1 client.15783 shutdown 2024-05-27T07:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f79bb626b80 1 -- 172.21.15.114:0/2930086144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 msgr2=0x562b787f7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f79bb626b80 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 0x562b787f7810 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f79a000b770 tx=0x7f79a0028ca0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f79bb626b80 1 -- 172.21.15.114:0/2930086144 shutdown_connections 2024-05-27T07:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f79bb626b80 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b788078a0 0x562b78809cc0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f79bb626b80 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787f4070 0x562b787f8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f79bb626b80 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b7866ee40 0x562b787f7810 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f798dffb700 0 client.15783 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f79bb626b80 1 --2- 172.21.15.114:0/2930086144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787ee120 0x562b787f7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f79bb626b80 1 -- 172.21.15.114:0/2930086144 >> 172.21.15.114:0/2930086144 conn(0x562b78671770 msgr2=0x562b787ecbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f79bb626b80 1 -- 172.21.15.114:0/2930086144 shutdown_connections 2024-05-27T07:42:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.928+0000 7f79bb626b80 1 -- 172.21.15.114:0/2930086144 wait complete. 2024-05-27T07:42:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.929+0000 7f79bb626b80 20 client.15783 trim_cache size 0 max 16384 2024-05-27T07:42:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.946+0000 7f79bb626b80 1 Processor -- start 2024-05-27T07:42:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.946+0000 7f79bb626b80 1 -- start start 2024-05-27T07:42:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.946+0000 7f79bb626b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b7866ee40 0x562b787edbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.946+0000 7f79bb626b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b787ee120 0x562b78803da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.946+0000 7f79bb626b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787f4070 0x562b787f77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.946+0000 7f79bb626b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79a81ac3f0 con 0x562b787ee120 2024-05-27T07:42:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.946+0000 7f79bb626b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79a81aec50 con 0x562b787f4070 2024-05-27T07:42:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.946+0000 7f79bb626b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79a81aca40 con 0x562b7866ee40 2024-05-27T07:42:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.946+0000 7f79a5d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b787ee120 0x562b78803da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.947+0000 7f79a6d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787f4070 0x562b787f77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.947+0000 7f79a6d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787f4070 0x562b787f77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37458/0 (socket says 172.21.15.114:37458) 2024-05-27T07:42:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.947+0000 7f79a6d9d700 1 -- 172.21.15.114:0/1207406999 learned_addr learned my addr 172.21.15.114:0/1207406999 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.947+0000 7f79a659c700 1 --2- 172.21.15.114:0/1207406999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b7866ee40 0x562b787edbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.947+0000 7f79a6d9d700 1 -- 172.21.15.114:0/1207406999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b7866ee40 msgr2=0x562b787edbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.947+0000 7f79a6d9d700 1 --2- 172.21.15.114:0/1207406999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b7866ee40 0x562b787edbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.947+0000 7f79a6d9d700 1 -- 172.21.15.114:0/1207406999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b787ee120 msgr2=0x562b78803da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.947+0000 7f79a6d9d700 1 --2- 172.21.15.114:0/1207406999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b787ee120 0x562b78803da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.947+0000 7f79a6d9d700 1 -- 172.21.15.114:0/1207406999 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b78804780 con 0x562b787f4070 2024-05-27T07:42:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.947+0000 7f79a6d9d700 1 --2- 172.21.15.114:0/1207406999 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787f4070 0x562b787f77d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f79980a9720 tx=0x7f79980097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8c8920cecba9fac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.947+0000 7f798ffff700 1 -- 172.21.15.114:0/1207406999 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7998005310 con 0x562b787f4070 2024-05-27T07:42:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.947+0000 7f798ffff700 1 -- 172.21.15.114:0/1207406999 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79980b21c0 con 0x562b787f4070 2024-05-27T07:42:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.947+0000 7f798ffff700 1 -- 172.21.15.114:0/1207406999 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79980b2880 con 0x562b787f4070 2024-05-27T07:42:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.948+0000 7f79bb626b80 1 -- 172.21.15.114:0/1207406999 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787f4070 msgr2=0x562b787f77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.948+0000 7f79bb626b80 1 --2- 172.21.15.114:0/1207406999 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787f4070 0x562b787f77d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f79980a9720 tx=0x7f79980097b0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.948+0000 7f79bb626b80 1 -- 172.21.15.114:0/1207406999 shutdown_connections 2024-05-27T07:42:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.948+0000 7f79bb626b80 1 --2- 172.21.15.114:0/1207406999 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b787f4070 0x562b787f77d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.948+0000 7f79bb626b80 1 --2- 172.21.15.114:0/1207406999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b7866ee40 0x562b787edbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.948+0000 7f79bb626b80 1 --2- 172.21.15.114:0/1207406999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b787ee120 0x562b78803da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.948+0000 7f79bb626b80 1 -- 172.21.15.114:0/1207406999 >> 172.21.15.114:0/1207406999 conn(0x562b78671770 msgr2=0x562b787f2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.948+0000 7f79bb626b80 1 -- 172.21.15.114:0/1207406999 shutdown_connections 2024-05-27T07:42:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.948+0000 7f79bb626b80 1 -- 172.21.15.114:0/1207406999 wait complete. 2024-05-27T07:42:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79bb626b80 1 -- start start 2024-05-27T07:42:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79bb626b80 1 Processor -- start 2024-05-27T07:42:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79bb626b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b7866ee40 0x562b7880d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79bb626b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787ee120 0x562b7880a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79bb626b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b787f4070 0x562b7880a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79bb626b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79a81ad110 con 0x562b787f4070 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79bb626b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79a81ae580 con 0x562b7866ee40 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79bb626b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79a81ad7e0 con 0x562b787ee120 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79bb626b80 10 client.0 _ll_register_callbacks cb 0x562b786a25b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79a6d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b787f4070 0x562b7880a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79a6d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b787f4070 0x562b7880a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52102/0 (socket says 172.21.15.114:52102) 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79a6d9d700 1 -- 172.21.15.114:0/2927822686 learned_addr learned my addr 172.21.15.114:0/2927822686 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79a5d9b700 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787ee120 0x562b7880a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.949+0000 7f79a659c700 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b7866ee40 0x562b7880d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f79a6d9d700 1 -- 172.21.15.114:0/2927822686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787ee120 msgr2=0x562b7880a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f79a6d9d700 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787ee120 0x562b7880a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f79a6d9d700 1 -- 172.21.15.114:0/2927822686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b7866ee40 msgr2=0x562b7880d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f79a6d9d700 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b7866ee40 0x562b7880d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f79a6d9d700 1 -- 172.21.15.114:0/2927822686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b7880aea0 con 0x562b787f4070 2024-05-27T07:42:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f79a5d9b700 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787ee120 0x562b7880a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f79a6d9d700 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b787f4070 0x562b7880a960 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f79980a5aa0 tx=0x7f7998016680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e1057e6cdda8fd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f798dffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f798dffb700 1 -- 172.21.15.114:0/2927822686 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799806d030 con 0x562b787f4070 2024-05-27T07:42:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f798dffb700 1 -- 172.21.15.114:0/2927822686 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f79980a6010 con 0x562b787f4070 2024-05-27T07:42:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f79bb626b80 1 -- 172.21.15.114:0/2927822686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562b787174a0 con 0x562b787f4070 2024-05-27T07:42:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f798dffb700 1 -- 172.21.15.114:0/2927822686 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7998005310 con 0x562b787f4070 2024-05-27T07:42:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f79bb626b80 10 client.15789 fetch_fsmap learned FSMap version 21 2024-05-27T07:42:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f79bb626b80 1 -- 172.21.15.114:0/2927822686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562b78871cd0 con 0x562b787f4070 2024-05-27T07:42:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.950+0000 7f798dffb700 1 -- 172.21.15.114:0/2927822686 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562b787174a0 con 0x562b787f4070 2024-05-27T07:42:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.951+0000 7f798dffb700 1 -- 172.21.15.114:0/2927822686 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f799806f2c0 con 0x562b787f4070 2024-05-27T07:42:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.951+0000 7f79bb626b80 10 client.15789 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:42:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.951+0000 7f79bb626b80 10 client.15789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.951+0000 7f79bb626b80 20 client.15789 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.951+0000 7f79bb626b80 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b78872370 0x562b78874790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.951+0000 7f79bb626b80 4 client.15789 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:42:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.951+0000 7f79bb626b80 1 -- 172.21.15.114:0/2927822686 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562b7858e250 con 0x562b78872370 2024-05-27T07:42:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.951+0000 7f79a659c700 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b78872370 0x562b78874790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.951+0000 7f798dffb700 10 client.15789 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.951+0000 7f79a659c700 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b78872370 0x562b78874790 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5782b1fcacdd4a7f server_cookie=f4ccc9028e122f76 in_seq=0 out_seq=0 2024-05-27T07:42:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f798dffb700 1 -- 172.21.15.114:0/2927822686 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562b7858e250 con 0x562b78872370 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f798dffb700 10 client.15789 handle_command_reply: tid=0 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f79bb626b80 1 client.15789 shutdown 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f79bb626b80 1 -- 172.21.15.114:0/2927822686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b787f4070 msgr2=0x562b7880a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f79bb626b80 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b787f4070 0x562b7880a960 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f79980a5aa0 tx=0x7f7998016680 comp rx=0 tx=0).stop 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f79bb626b80 1 -- 172.21.15.114:0/2927822686 shutdown_connections 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f79bb626b80 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b787f4070 0x562b7880a960 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f79bb626b80 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b78872370 0x562b78874790 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f79bb626b80 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b7866ee40 0x562b7880d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f79bb626b80 1 --2- 172.21.15.114:0/2927822686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b787ee120 0x562b7880a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f79bb626b80 1 -- 172.21.15.114:0/2927822686 >> 172.21.15.114:0/2927822686 conn(0x562b78671770 msgr2=0x562b787f2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f798dffb700 0 client.15789 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f79bb626b80 1 -- 172.21.15.114:0/2927822686 shutdown_connections 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f79bb626b80 1 -- 172.21.15.114:0/2927822686 wait complete. 2024-05-27T07:42:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.952+0000 7f79bb626b80 20 client.15789 trim_cache size 0 max 16384 2024-05-27T07:42:03.956 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:42:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.953+0000 7f79acc11700 1 -- 172.21.15.114:0/390106675 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f79940b29f0 msgr2=0x7f79940b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.953+0000 7f79acc11700 1 --2- 172.21.15.114:0/390106675 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f79940b29f0 0x7f79940b4f00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f79a0009fd0 tx=0x7f79a000b040 comp rx=0 tx=0).stop 2024-05-27T07:42:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.953+0000 7f79acc11700 1 -- 172.21.15.114:0/390106675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a8108750 msgr2=0x7f79a819c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.953+0000 7f79acc11700 1 --2- 172.21.15.114:0/390106675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a8108750 0x7f79a819c740 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f799800e940 tx=0x7f799800ec50 comp rx=0 tx=0).stop 2024-05-27T07:42:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.953+0000 7f79acc11700 1 -- 172.21.15.114:0/390106675 shutdown_connections 2024-05-27T07:42:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.953+0000 7f79acc11700 1 --2- 172.21.15.114:0/390106675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a8108750 0x7f79a819c740 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.953+0000 7f79acc11700 1 --2- 172.21.15.114:0/390106675 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f79940b29f0 0x7f79940b4f00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.953+0000 7f79acc11700 1 --2- 172.21.15.114:0/390106675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79a81034d0 0x7f79a819bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.953+0000 7f79acc11700 1 --2- 172.21.15.114:0/390106675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a8107e00 0x7f79a819c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.953+0000 7f79acc11700 1 -- 172.21.15.114:0/390106675 >> 172.21.15.114:0/390106675 conn(0x7f79a80fef20 msgr2=0x7f79a81096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.953+0000 7f79acc11700 1 -- 172.21.15.114:0/390106675 shutdown_connections 2024-05-27T07:42:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:03.953+0000 7f79acc11700 1 -- 172.21.15.114:0/390106675 wait complete. 2024-05-27T07:42:03.973 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:03 smithi163 ceph-mon[36451]: pgmap v72: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.4 MiB/s wr, 325 op/s 2024-05-27T07:42:03.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:03 smithi186 ceph-mon[36647]: pgmap v72: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.4 MiB/s wr, 325 op/s 2024-05-27T07:42:04.237 INFO:tasks.workunit.client.0.smithi114.stdout: 1 45357 16.34 MB/sec warmup 16 sec latency 32.964 ms 2024-05-27T07:42:05.237 INFO:tasks.workunit.client.0.smithi114.stdout: 1 48210 16.19 MB/sec warmup 17 sec latency 35.917 ms 2024-05-27T07:42:05.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:05 smithi114 ceph-mon[32038]: pgmap v73: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.4 MiB/s wr, 325 op/s 2024-05-27T07:42:05.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:05 smithi163 ceph-mon[36451]: pgmap v73: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.4 MiB/s wr, 325 op/s 2024-05-27T07:42:05.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:05 smithi186 ceph-mon[36647]: pgmap v73: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.4 MiB/s wr, 325 op/s 2024-05-27T07:42:06.120 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:42:06.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:42:06.237 INFO:tasks.workunit.client.0.smithi114.stdout: 1 50745 16.14 MB/sec warmup 18 sec latency 13.401 ms 2024-05-27T07:42:06.388 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:07.237 INFO:tasks.workunit.client.0.smithi114.stdout: 1 53856 16.20 MB/sec warmup 19 sec latency 10.268 ms 2024-05-27T07:42:07.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:07 smithi114 ceph-mon[32038]: pgmap v74: 129 pgs: 129 active+clean; 21 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 2.2 MiB/s wr, 434 op/s 2024-05-27T07:42:07.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:07 smithi163 ceph-mon[36451]: pgmap v74: 129 pgs: 129 active+clean; 21 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 2.2 MiB/s wr, 434 op/s 2024-05-27T07:42:07.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:07 smithi186 ceph-mon[36647]: pgmap v74: 129 pgs: 129 active+clean; 21 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 2.2 MiB/s wr, 434 op/s 2024-05-27T07:42:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.038+0000 7fe9c8a69640 1 -- 172.21.15.114:0/4009587500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c4107e60 msgr2=0x7fe9c410f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.038+0000 7fe9c8a69640 1 --2- 172.21.15.114:0/4009587500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c4107e60 0x7fe9c410f690 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009ca0 tx=0x7fe9bc02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.038+0000 7fe9c8a69640 1 -- 172.21.15.114:0/4009587500 shutdown_connections 2024-05-27T07:42:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.038+0000 7fe9c8a69640 1 --2- 172.21.15.114:0/4009587500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c4107e60 0x7fe9c410f690 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.038+0000 7fe9c8a69640 1 --2- 172.21.15.114:0/4009587500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9c41073e0 0x7fe9c41077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.038+0000 7fe9c8a69640 1 --2- 172.21.15.114:0/4009587500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9c4103520 0x7fe9c4103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.038+0000 7fe9c8a69640 1 -- 172.21.15.114:0/4009587500 >> 172.21.15.114:0/4009587500 conn(0x7fe9c40fd9c0 msgr2=0x7fe9c40ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.038+0000 7fe9c8a69640 1 -- 172.21.15.114:0/4009587500 shutdown_connections 2024-05-27T07:42:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.038+0000 7fe9c8a69640 1 -- 172.21.15.114:0/4009587500 wait complete. 2024-05-27T07:42:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.039+0000 7fe9c8a69640 1 Processor -- start 2024-05-27T07:42:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.039+0000 7fe9c8a69640 1 -- start start 2024-05-27T07:42:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.039+0000 7fe9c8a69640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c4103520 0x7fe9c419b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.039+0000 7fe9c8a69640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9c41073e0 0x7fe9c419bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.039+0000 7fe9c8a69640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9c4107e60 0x7fe9c419c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.039+0000 7fe9c8a69640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c4112d80 con 0x7fe9c4103520 2024-05-27T07:42:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.039+0000 7fe9c8a69640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c4112c00 con 0x7fe9c41073e0 2024-05-27T07:42:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.039+0000 7fe9c8a69640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c4112f00 con 0x7fe9c4107e60 2024-05-27T07:42:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.039+0000 7fe9c259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c4103520 0x7fe9c419b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.039+0000 7fe9c259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c4103520 0x7fe9c419b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33146/0 (socket says 172.21.15.114:33146) 2024-05-27T07:42:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.039+0000 7fe9c259c640 1 -- 172.21.15.114:0/3735362972 learned_addr learned my addr 172.21.15.114:0/3735362972 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.039+0000 7fe9c2d9d640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9c4107e60 0x7fe9c419c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.039+0000 7fe9b9d9b640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9c41073e0 0x7fe9c419bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.040+0000 7fe9c259c640 1 -- 172.21.15.114:0/3735362972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9c4107e60 msgr2=0x7fe9c419c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.040+0000 7fe9c259c640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9c4107e60 0x7fe9c419c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.040+0000 7fe9c259c640 1 -- 172.21.15.114:0/3735362972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9c41073e0 msgr2=0x7fe9c419bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.040+0000 7fe9c259c640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9c41073e0 0x7fe9c419bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.040+0000 7fe9c259c640 1 -- 172.21.15.114:0/3735362972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c41a6960 con 0x7fe9c4103520 2024-05-27T07:42:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.040+0000 7fe9c259c640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c4103520 0x7fe9c419b620 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400e9f0 tx=0x7fe9b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.040+0000 7fe9bbfff640 1 -- 172.21.15.114:0/3735362972 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b400ccc0 con 0x7fe9c4103520 2024-05-27T07:42:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.040+0000 7fe9c8a69640 1 -- 172.21.15.114:0/3735362972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c41a6c50 con 0x7fe9c4103520 2024-05-27T07:42:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.040+0000 7fe9bbfff640 1 -- 172.21.15.114:0/3735362972 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9b40054d0 con 0x7fe9c4103520 2024-05-27T07:42:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.040+0000 7fe9bbfff640 1 -- 172.21.15.114:0/3735362972 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4010430 con 0x7fe9c4103520 2024-05-27T07:42:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.040+0000 7fe9c8a69640 1 -- 172.21.15.114:0/3735362972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c41a71a0 con 0x7fe9c4103520 2024-05-27T07:42:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.043+0000 7fe9bbfff640 1 -- 172.21.15.114:0/3735362972 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fe9b40046c0 con 0x7fe9c4103520 2024-05-27T07:42:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.043+0000 7fe9cf9f5740 1 Processor -- start 2024-05-27T07:42:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.043+0000 7fe9cf9f5740 1 -- start start 2024-05-27T07:42:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9cf9f5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f2e1b0 0x5610a12cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9cf9f5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f052f0 0x5610a12cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9cf9f5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f0d4d0 0x5610a12cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9cf9f5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5610a1241c30 con 0x5610a0f0d4d0 2024-05-27T07:42:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9cf9f5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5610a12317d0 con 0x5610a0f052f0 2024-05-27T07:42:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9cf9f5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5610a1235810 con 0x5610a0f2e1b0 2024-05-27T07:42:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9c2d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f2e1b0 0x5610a12cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9c2d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f2e1b0 0x5610a12cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54360/0 (socket says 172.21.15.114:54360) 2024-05-27T07:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9b9d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f0d4d0 0x5610a12cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9c2d9d640 1 -- 172.21.15.114:0/3603482334 learned_addr learned my addr 172.21.15.114:0/3603482334 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9bbfff640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe9b00b2950 0x7fe9b00b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9c259c640 1 --2- 172.21.15.114:0/3603482334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f052f0 0x5610a12cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9c2d9d640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe9b00b2950 0x7fe9b00b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9c2d9d640 1 -- 172.21.15.114:0/3603482334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f052f0 msgr2=0x5610a12cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9c2d9d640 1 --2- 172.21.15.114:0/3603482334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f052f0 0x5610a12cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9c2d9d640 1 -- 172.21.15.114:0/3603482334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f0d4d0 msgr2=0x5610a12cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9c2d9d640 1 --2- 172.21.15.114:0/3603482334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f0d4d0 0x5610a12cfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9c2d9d640 1 -- 172.21.15.114:0/3603482334 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610a0f46210 con 0x5610a0f2e1b0 2024-05-27T07:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9b9d9b640 1 --2- 172.21.15.114:0/3603482334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f0d4d0 0x5610a12cfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.044+0000 7fe9bbfff640 1 -- 172.21.15.114:0/3735362972 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fe9b40e2d20 con 0x7fe9c4103520 2024-05-27T07:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.045+0000 7fe9c2d9d640 1 --2- 172.21.15.114:0/3603482334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f2e1b0 0x5610a12cf3d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009ca0 tx=0x7fe9bc031d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63ac7e4e94805cad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.045+0000 7fe9b8d99640 1 -- 172.21.15.114:0/3603482334 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc004d40 con 0x5610a0f2e1b0 2024-05-27T07:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.045+0000 7fe9c2d9d640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe9b00b2950 0x7fe9b00b4e60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc04e000 tx=0x7fe9bc04c000 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.045+0000 7fe9b8d99640 1 -- 172.21.15.114:0/3603482334 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9bc036070 con 0x5610a0f2e1b0 2024-05-27T07:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.045+0000 7fe9b8d99640 1 -- 172.21.15.114:0/3603482334 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc004600 con 0x5610a0f2e1b0 2024-05-27T07:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.046+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/3603482334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f2e1b0 msgr2=0x5610a12cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.046+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/3603482334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f2e1b0 0x5610a12cf3d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009ca0 tx=0x7fe9bc031d90 comp rx=0 tx=0).stop 2024-05-27T07:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.046+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/3603482334 shutdown_connections 2024-05-27T07:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.046+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/3603482334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f0d4d0 0x5610a12cfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.046+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/3603482334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f052f0 0x5610a12cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.046+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/3603482334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f2e1b0 0x5610a12cf3d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.046+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/3603482334 >> 172.21.15.114:0/3603482334 conn(0x5610a0f22300 msgr2=0x5610a12cbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.046+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/3603482334 shutdown_connections 2024-05-27T07:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.046+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/3603482334 wait complete. 2024-05-27T07:42:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.046+0000 7fe9cf9f5740 1 -- start start 2024-05-27T07:42:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.047+0000 7fe9cf9f5740 1 Processor -- start 2024-05-27T07:42:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.047+0000 7fe9cf9f5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f052f0 0x5610a12db410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.047+0000 7fe9cf9f5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f0d4d0 0x5610a12db970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.047+0000 7fe9cf9f5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f2e1b0 0x5610a12d9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.047+0000 7fe9cf9f5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5610a0f17790 con 0x5610a0f0d4d0 2024-05-27T07:42:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.047+0000 7fe9cf9f5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5610a11ed890 con 0x5610a0f2e1b0 2024-05-27T07:42:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.047+0000 7fe9cf9f5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5610a12dc380 con 0x5610a0f052f0 2024-05-27T07:42:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.047+0000 7fe9b9d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f0d4d0 0x5610a12db970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.047+0000 7fe9c259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f052f0 0x5610a12db410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.047+0000 7fe9b9d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f0d4d0 0x5610a12db970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33152/0 (socket says 172.21.15.114:33152) 2024-05-27T07:42:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.047+0000 7fe9b9d9b640 1 -- 172.21.15.114:0/1763219996 learned_addr learned my addr 172.21.15.114:0/1763219996 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.047+0000 7fe9c2d9d640 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f2e1b0 0x5610a12d9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.047+0000 7fe9cf9f5740 10 client.0 _ll_register_callbacks cb 0x5610a0f45a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9b9d9b640 1 -- 172.21.15.114:0/1763219996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f052f0 msgr2=0x5610a12db410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9b9d9b640 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f052f0 0x5610a12db410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9b9d9b640 1 -- 172.21.15.114:0/1763219996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f2e1b0 msgr2=0x5610a12d9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9b9d9b640 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f2e1b0 0x5610a12d9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9b9d9b640 1 -- 172.21.15.114:0/1763219996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610a1012620 con 0x5610a0f0d4d0 2024-05-27T07:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9c259c640 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f052f0 0x5610a12db410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9c2d9d640 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f2e1b0 0x5610a12d9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9b9d9b640 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f0d4d0 0x5610a12db970 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac012700 tx=0x7fe9ac012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23eac7ba8612d38d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9a9ffb640 1 -- 172.21.15.114:0/1763219996 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac00f920 con 0x5610a0f0d4d0 2024-05-27T07:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/1763219996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9bc036070 con 0x5610a0f0d4d0 2024-05-27T07:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9a9ffb640 1 -- 172.21.15.114:0/1763219996 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fe9ac00fac0 con 0x5610a0f0d4d0 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9a9ffb640 1 -- 172.21.15.114:0/1763219996 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe9ac00fcc0 con 0x5610a0f0d4d0 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9cf9f5740 10 client.15801 fetch_fsmap learned FSMap version 21 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/1763219996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5610a12e8750 con 0x5610a0f0d4d0 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.048+0000 7fe9a9ffb640 1 -- 172.21.15.114:0/1763219996 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9bc036070 con 0x5610a0f0d4d0 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.049+0000 7fe9a9ffb640 1 -- 172.21.15.114:0/1763219996 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fe9ac01c420 con 0x5610a0f0d4d0 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.049+0000 7fe9cf9f5740 10 client.15801 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.049+0000 7fe9cf9f5740 10 client.15801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.049+0000 7fe9cf9f5740 20 client.15801 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.049+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5610a0eecd80 0x5610a12ebcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.049+0000 7fe9cf9f5740 4 client.15801 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.049+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/1763219996 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610a1243b80 con 0x5610a0eecd80 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.049+0000 7fe9c259c640 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5610a0eecd80 0x5610a12ebcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.049+0000 7fe9a9ffb640 10 client.15801 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.049+0000 7fe9c259c640 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5610a0eecd80 0x5610a12ebcd0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2156a388f8ef567 server_cookie=658a5e8b28636de in_seq=0 out_seq=0 2024-05-27T07:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.051+0000 7fe9a9ffb640 1 -- 172.21.15.114:0/1763219996 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5610a1243b80 con 0x5610a0eecd80 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.051+0000 7fe9a9ffb640 10 client.15801 handle_command_reply: tid=0 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.051+0000 7fe9cf9f5740 1 client.15801 shutdown 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.051+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/1763219996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f0d4d0 msgr2=0x5610a12db970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.051+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f0d4d0 0x5610a12db970 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac012700 tx=0x7fe9ac012bc0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.052+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/1763219996 shutdown_connections 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.052+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5610a0eecd80 0x5610a12ebcd0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.052+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f2e1b0 0x5610a12d9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.052+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f0d4d0 0x5610a12db970 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.052+0000 7fe9a9ffb640 0 client.15801 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.052+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/1763219996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f052f0 0x5610a12db410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.052+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/1763219996 >> 172.21.15.114:0/1763219996 conn(0x5610a0f22300 msgr2=0x5610a12cf200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.052+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/1763219996 shutdown_connections 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.052+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/1763219996 wait complete. 2024-05-27T07:42:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.052+0000 7fe9cf9f5740 20 client.15801 trim_cache size 0 max 16384 2024-05-27T07:42:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.067+0000 7fe9cf9f5740 1 Processor -- start 2024-05-27T07:42:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.067+0000 7fe9cf9f5740 1 -- start start 2024-05-27T07:42:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.067+0000 7fe9cf9f5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f052f0 0x5610a12d3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.067+0000 7fe9cf9f5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f0d4d0 0x5610a12f1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.067+0000 7fe9cf9f5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f2e1b0 0x5610a12f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.067+0000 7fe9cf9f5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c41af310 con 0x5610a0f2e1b0 2024-05-27T07:42:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.067+0000 7fe9cf9f5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c41b1b90 con 0x5610a0f052f0 2024-05-27T07:42:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.067+0000 7fe9cf9f5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c41af980 con 0x5610a0f0d4d0 2024-05-27T07:42:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.067+0000 7fe9c2d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f2e1b0 0x5610a12f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.067+0000 7fe9c2d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f2e1b0 0x5610a12f1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33160/0 (socket says 172.21.15.114:33160) 2024-05-27T07:42:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.067+0000 7fe9c2d9d640 1 -- 172.21.15.114:0/1336275957 learned_addr learned my addr 172.21.15.114:0/1336275957 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.067+0000 7fe9c259c640 1 --2- 172.21.15.114:0/1336275957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f052f0 0x5610a12d3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.067+0000 7fe9b9d9b640 1 --2- 172.21.15.114:0/1336275957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f0d4d0 0x5610a12f1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.068+0000 7fe9c2d9d640 1 -- 172.21.15.114:0/1336275957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f0d4d0 msgr2=0x5610a12f1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.068+0000 7fe9c2d9d640 1 --2- 172.21.15.114:0/1336275957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f0d4d0 0x5610a12f1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.068+0000 7fe9c2d9d640 1 -- 172.21.15.114:0/1336275957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f052f0 msgr2=0x5610a12d3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.068+0000 7fe9c2d9d640 1 --2- 172.21.15.114:0/1336275957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f052f0 0x5610a12d3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.068+0000 7fe9c2d9d640 1 -- 172.21.15.114:0/1336275957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610a12ced60 con 0x5610a0f2e1b0 2024-05-27T07:42:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.068+0000 7fe9c2d9d640 1 --2- 172.21.15.114:0/1336275957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f2e1b0 0x5610a12f1a60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0042d0 tx=0x7fe9bc0041c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ee31503219eb02 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.068+0000 7fe9abfff640 1 -- 172.21.15.114:0/1336275957 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc063070 con 0x5610a0f2e1b0 2024-05-27T07:42:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.068+0000 7fe9abfff640 1 -- 172.21.15.114:0/1336275957 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9bc036070 con 0x5610a0f2e1b0 2024-05-27T07:42:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.068+0000 7fe9abfff640 1 -- 172.21.15.114:0/1336275957 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc004860 con 0x5610a0f2e1b0 2024-05-27T07:42:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.069+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/1336275957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f2e1b0 msgr2=0x5610a12f1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.069+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/1336275957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f2e1b0 0x5610a12f1a60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0042d0 tx=0x7fe9bc0041c0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.069+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/1336275957 shutdown_connections 2024-05-27T07:42:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.069+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/1336275957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f2e1b0 0x5610a12f1a60 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.069+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/1336275957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f0d4d0 0x5610a12f1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.069+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/1336275957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f052f0 0x5610a12d3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.069+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/1336275957 >> 172.21.15.114:0/1336275957 conn(0x5610a0f22300 msgr2=0x5610a12d3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.069+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/1336275957 shutdown_connections 2024-05-27T07:42:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.069+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/1336275957 wait complete. 2024-05-27T07:42:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.069+0000 7fe9cf9f5740 1 -- start start 2024-05-27T07:42:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.069+0000 7fe9cf9f5740 1 Processor -- start 2024-05-27T07:42:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9cf9f5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f052f0 0x5610a12e16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9cf9f5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f0d4d0 0x5610a12e1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9cf9f5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f2e1b0 0x5610a12debf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9cf9f5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c41b0050 con 0x5610a0f052f0 2024-05-27T07:42:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9cf9f5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c41b14c0 con 0x5610a0f2e1b0 2024-05-27T07:42:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9cf9f5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c41b0720 con 0x5610a0f0d4d0 2024-05-27T07:42:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9cf9f5740 10 client.0 _ll_register_callbacks cb 0x7fe9ac009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9c259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f052f0 0x5610a12e16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9c259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f052f0 0x5610a12e16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33166/0 (socket says 172.21.15.114:33166) 2024-05-27T07:42:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9c259c640 1 -- 172.21.15.114:0/3118118039 learned_addr learned my addr 172.21.15.114:0/3118118039 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9b9d9b640 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f0d4d0 0x5610a12e1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9c2d9d640 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f2e1b0 0x5610a12debf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9c259c640 1 -- 172.21.15.114:0/3118118039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f0d4d0 msgr2=0x5610a12e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9c259c640 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f0d4d0 0x5610a12e1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9c259c640 1 -- 172.21.15.114:0/3118118039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f2e1b0 msgr2=0x5610a12debf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9c259c640 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f2e1b0 0x5610a12debf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9c259c640 1 -- 172.21.15.114:0/3118118039 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610a12df1e0 con 0x5610a0f052f0 2024-05-27T07:42:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9c2d9d640 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f2e1b0 0x5610a12debf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9c259c640 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f052f0 0x5610a12e16e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40a5cd0 tx=0x7fe9b40734f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f53b84a2582294b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9a9ffb640 1 -- 172.21.15.114:0/3118118039 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4091040 con 0x5610a0f052f0 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9a9ffb640 1 -- 172.21.15.114:0/3118118039 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fe9b40a6240 con 0x5610a0f052f0 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/3118118039 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9bc036070 con 0x5610a0f052f0 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9a9ffb640 1 -- 172.21.15.114:0/3118118039 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe9b40b23f0 con 0x5610a0f052f0 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9cf9f5740 10 client.15807 fetch_fsmap learned FSMap version 21 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.070+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/3118118039 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5610a1353e80 con 0x5610a0f052f0 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.071+0000 7fe9a9ffb640 1 -- 172.21.15.114:0/3118118039 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9bc036070 con 0x5610a0f052f0 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.071+0000 7fe9a9ffb640 1 -- 172.21.15.114:0/3118118039 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fe9b40731b0 con 0x5610a0f052f0 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.071+0000 7fe9cf9f5740 10 client.15807 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.071+0000 7fe9cf9f5740 10 client.15807 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.071+0000 7fe9cf9f5740 20 client.15807 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.071+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5610a0ec21c0 0x5610a1356640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.071+0000 7fe9cf9f5740 4 client.15807 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:42:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.071+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/3118118039 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5610a123d360 con 0x5610a0ec21c0 2024-05-27T07:42:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.071+0000 7fe9b9d9b640 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5610a0ec21c0 0x5610a1356640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.071+0000 7fe9b9d9b640 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5610a0ec21c0 0x5610a1356640 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b66674b527596541 server_cookie=6ef7c63b5e1179c5 in_seq=0 out_seq=0 2024-05-27T07:42:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.071+0000 7fe9a9ffb640 10 client.15807 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:08.237 INFO:tasks.workunit.client.0.smithi114.stdout: 1 56860 16.27 MB/sec warmup 20 sec latency 127.572 ms 2024-05-27T07:42:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.327+0000 7fe9a9ffb640 1 -- 172.21.15.114:0/3118118039 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5610a123d360 con 0x5610a0ec21c0 2024-05-27T07:42:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.327+0000 7fe9a9ffb640 10 client.15807 handle_command_reply: tid=0 2024-05-27T07:42:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.327+0000 7fe9cf9f5740 1 client.15807 shutdown 2024-05-27T07:42:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/3118118039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f052f0 msgr2=0x5610a12e16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f052f0 0x5610a12e16e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40a5cd0 tx=0x7fe9b40734f0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/3118118039 shutdown_connections 2024-05-27T07:42:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5610a0ec21c0 0x5610a1356640 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5610a0f2e1b0 0x5610a12debf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5610a0f0d4d0 0x5610a12e1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9cf9f5740 1 --2- 172.21.15.114:0/3118118039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610a0f052f0 0x5610a12e16e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/3118118039 >> 172.21.15.114:0/3118118039 conn(0x5610a0f22300 msgr2=0x5610a12dafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9a9ffb640 0 client.15807 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/3118118039 shutdown_connections 2024-05-27T07:42:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9cf9f5740 1 -- 172.21.15.114:0/3118118039 wait complete. 2024-05-27T07:42:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9cf9f5740 20 client.15807 trim_cache size 0 max 16384 2024-05-27T07:42:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:42:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:42:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:42:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9c8a69640 1 -- 172.21.15.114:0/3735362972 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe9b00b2950 msgr2=0x7fe9b00b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9c8a69640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe9b00b2950 0x7fe9b00b4e60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc04e000 tx=0x7fe9bc04c000 comp rx=0 tx=0).stop 2024-05-27T07:42:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9c8a69640 1 -- 172.21.15.114:0/3735362972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c4103520 msgr2=0x7fe9c419b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.328+0000 7fe9c8a69640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c4103520 0x7fe9c419b620 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400e9f0 tx=0x7fe9b400eeb0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.329+0000 7fe9c8a69640 1 -- 172.21.15.114:0/3735362972 shutdown_connections 2024-05-27T07:42:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.329+0000 7fe9c8a69640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe9b00b2950 0x7fe9b00b4e60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.329+0000 7fe9c8a69640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9c4107e60 0x7fe9c419c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.329+0000 7fe9c8a69640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9c41073e0 0x7fe9c419bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.329+0000 7fe9c8a69640 1 --2- 172.21.15.114:0/3735362972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c4103520 0x7fe9c419b620 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.329+0000 7fe9c8a69640 1 -- 172.21.15.114:0/3735362972 >> 172.21.15.114:0/3735362972 conn(0x7fe9c40fd9c0 msgr2=0x7fe9c41054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.329+0000 7fe9c8a69640 1 -- 172.21.15.114:0/3735362972 shutdown_connections 2024-05-27T07:42:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:08.329+0000 7fe9c8a69640 1 -- 172.21.15.114:0/3735362972 wait complete. 2024-05-27T07:42:08.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.513+0000 7fb5b314d700 1 -- 172.21.15.114:0/2687400694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5ac108730 msgr2=0x7fb5ac110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.513+0000 7fb5b314d700 1 --2- 172.21.15.114:0/2687400694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5ac108730 0x7fb5ac110090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400b0a0 tx=0x7fb5a400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.513+0000 7fb5b314d700 1 -- 172.21.15.114:0/2687400694 shutdown_connections 2024-05-27T07:42:08.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.513+0000 7fb5b314d700 1 --2- 172.21.15.114:0/2687400694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5ac108730 0x7fb5ac110090 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.513+0000 7fb5b314d700 1 --2- 172.21.15.114:0/2687400694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac1034b0 0x7fb5ac1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.513+0000 7fb5b314d700 1 --2- 172.21.15.114:0/2687400694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5ac107de0 0x7fb5ac1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.513+0000 7fb5b314d700 1 -- 172.21.15.114:0/2687400694 >> 172.21.15.114:0/2687400694 conn(0x7fb5ac0fedb0 msgr2=0x7fb5ac1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.514+0000 7fb5b314d700 1 -- 172.21.15.114:0/2687400694 shutdown_connections 2024-05-27T07:42:08.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.514+0000 7fb5b314d700 1 -- 172.21.15.114:0/2687400694 wait complete. 2024-05-27T07:42:08.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.514+0000 7fb5b314d700 1 Processor -- start 2024-05-27T07:42:08.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.514+0000 7fb5b314d700 1 -- start start 2024-05-27T07:42:08.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5b314d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5ac1034b0 0x7fb5ac19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5b314d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac107de0 0x7fb5ac19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5b314d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5ac108730 0x7fb5ac19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5b314d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac113380 con 0x7fb5ac108730 2024-05-27T07:42:08.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5b314d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac113200 con 0x7fb5ac1034b0 2024-05-27T07:42:08.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5b314d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac113500 con 0x7fb5ac107de0 2024-05-27T07:42:08.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5b16ea700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5ac108730 0x7fb5ac19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5abfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac107de0 0x7fb5ac19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5b16ea700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5ac108730 0x7fb5ac19f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33184/0 (socket says 172.21.15.114:33184) 2024-05-27T07:42:08.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5b16ea700 1 -- 172.21.15.114:0/4177644587 learned_addr learned my addr 172.21.15.114:0/4177644587 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:08.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5b0ee9700 1 --2- 172.21.15.114:0/4177644587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5ac1034b0 0x7fb5ac19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5abfff700 1 -- 172.21.15.114:0/4177644587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5ac1034b0 msgr2=0x7fb5ac19ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5abfff700 1 --2- 172.21.15.114:0/4177644587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5ac1034b0 0x7fb5ac19ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5abfff700 1 -- 172.21.15.114:0/4177644587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5ac108730 msgr2=0x7fb5ac19f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5abfff700 1 --2- 172.21.15.114:0/4177644587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5ac108730 0x7fb5ac19f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5abfff700 1 -- 172.21.15.114:0/4177644587 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac19fcd0 con 0x7fb5ac107de0 2024-05-27T07:42:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.515+0000 7fb5abfff700 1 --2- 172.21.15.114:0/4177644587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac107de0 0x7fb5ac19b3a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb59800e9c0 tx=0x7fb59800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.516+0000 7fb5a9ffb700 1 -- 172.21.15.114:0/4177644587 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59800c9c0 con 0x7fb5ac107de0 2024-05-27T07:42:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.516+0000 7fb5a9ffb700 1 -- 172.21.15.114:0/4177644587 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb59800cb60 con 0x7fb5ac107de0 2024-05-27T07:42:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.516+0000 7fb5b314d700 1 -- 172.21.15.114:0/4177644587 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ac19ffc0 con 0x7fb5ac107de0 2024-05-27T07:42:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.516+0000 7fb5a9ffb700 1 -- 172.21.15.114:0/4177644587 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598018680 con 0x7fb5ac107de0 2024-05-27T07:42:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.516+0000 7fb5b314d700 1 -- 172.21.15.114:0/4177644587 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ac1a78e0 con 0x7fb5ac107de0 2024-05-27T07:42:08.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.519+0000 7fb5a9ffb700 1 -- 172.21.15.114:0/4177644587 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fb598010430 con 0x7fb5ac107de0 2024-05-27T07:42:08.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.519+0000 7fb5a9ffb700 1 --2- 172.21.15.114:0/4177644587 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb59c0b2a70 0x7fb59c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.520+0000 7fb5b0ee9700 1 --2- 172.21.15.114:0/4177644587 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb59c0b2a70 0x7fb59c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.520+0000 7fb5a9ffb700 1 -- 172.21.15.114:0/4177644587 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fb5980e2a40 con 0x7fb5ac107de0 2024-05-27T07:42:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.520+0000 7fb5b0ee9700 1 --2- 172.21.15.114:0/4177644587 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb59c0b2a70 0x7fb59c0b4f80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009c60 tx=0x7fb5a0009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.520+0000 7fb5c1b62b80 1 Processor -- start 2024-05-27T07:42:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.520+0000 7fb5c1b62b80 1 -- start start 2024-05-27T07:42:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.520+0000 7fb5c1b62b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03e6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.520+0000 7fb5c1b62b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e6cd0 0x5606c03e0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.520+0000 7fb5c1b62b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e1000 0x5606c03e1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.520+0000 7fb5c1b62b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5606c030fd10 con 0x5606c03e6cd0 2024-05-27T07:42:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.520+0000 7fb5c1b62b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5606c02ff8b0 con 0x5606c03e1000 2024-05-27T07:42:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.520+0000 7fb5c1b62b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5606c03038f0 con 0x5606c0262ec0 2024-05-27T07:42:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb5b0ee9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03e6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb5abfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e6cd0 0x5606c03e0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb5b0ee9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03e6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54416/0 (socket says 172.21.15.114:54416) 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb5b0ee9700 1 -- 172.21.15.114:0/2318452832 learned_addr learned my addr 172.21.15.114:0/2318452832 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb5b16ea700 1 --2- 172.21.15.114:0/2318452832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e1000 0x5606c03e1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb5abfff700 1 -- 172.21.15.114:0/2318452832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 msgr2=0x5606c03e6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb5abfff700 1 --2- 172.21.15.114:0/2318452832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03e6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb5abfff700 1 -- 172.21.15.114:0/2318452832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e1000 msgr2=0x5606c03e1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb5abfff700 1 --2- 172.21.15.114:0/2318452832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e1000 0x5606c03e1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb5abfff700 1 -- 172.21.15.114:0/2318452832 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606c024f950 con 0x5606c03e6cd0 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb5b0ee9700 1 --2- 172.21.15.114:0/2318452832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03e6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb5b16ea700 1 --2- 172.21.15.114:0/2318452832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e1000 0x5606c03e1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb5abfff700 1 --2- 172.21.15.114:0/2318452832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e6cd0 0x5606c03e0ac0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb5980dfde0 tx=0x7fb5980a5200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d343cfc044ecf857 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb596ffd700 1 -- 172.21.15.114:0/2318452832 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5980aa070 con 0x5606c03e6cd0 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb596ffd700 1 -- 172.21.15.114:0/2318452832 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5980a9970 con 0x5606c03e6cd0 2024-05-27T07:42:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.521+0000 7fb596ffd700 1 -- 172.21.15.114:0/2318452832 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5980a9e30 con 0x5606c03e6cd0 2024-05-27T07:42:08.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.522+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/2318452832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e6cd0 msgr2=0x5606c03e0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.522+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/2318452832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e6cd0 0x5606c03e0ac0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb5980dfde0 tx=0x7fb5980a5200 comp rx=0 tx=0).stop 2024-05-27T07:42:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.522+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/2318452832 shutdown_connections 2024-05-27T07:42:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.522+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/2318452832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e1000 0x5606c03e1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.522+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/2318452832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03e6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.522+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/2318452832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e6cd0 0x5606c03e0ac0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.522+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/2318452832 >> 172.21.15.114:0/2318452832 conn(0x5606c02bdb70 msgr2=0x5606c03db6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.522+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/2318452832 shutdown_connections 2024-05-27T07:42:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.522+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/2318452832 wait complete. 2024-05-27T07:42:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.523+0000 7fb5c1b62b80 1 -- start start 2024-05-27T07:42:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.523+0000 7fb5c1b62b80 1 Processor -- start 2024-05-27T07:42:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.523+0000 7fb5c1b62b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03ea4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.523+0000 7fb5c1b62b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e1000 0x5606c03ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.523+0000 7fb5c1b62b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e6cd0 0x5606c03eaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5c1b62b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5606c0206020 con 0x5606c03e6cd0 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5c1b62b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5606c02a1840 con 0x5606c03e1000 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5c1b62b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5606c03ed760 con 0x5606c0262ec0 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5c1b62b80 10 client.0 _ll_register_callbacks cb 0x5606c029a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5abfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e1000 0x5606c03ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5abfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e1000 0x5606c03ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39798/0 (socket says 172.21.15.114:39798) 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5b0ee9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03ea4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5abfff700 1 -- 172.21.15.114:0/3352853387 learned_addr learned my addr 172.21.15.114:0/3352853387 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5b16ea700 1 --2- 172.21.15.114:0/3352853387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e6cd0 0x5606c03eaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5abfff700 1 -- 172.21.15.114:0/3352853387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 msgr2=0x5606c03ea4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5abfff700 1 --2- 172.21.15.114:0/3352853387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03ea4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5abfff700 1 -- 172.21.15.114:0/3352853387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e6cd0 msgr2=0x5606c03eaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5abfff700 1 --2- 172.21.15.114:0/3352853387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e6cd0 0x5606c03eaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5abfff700 1 -- 172.21.15.114:0/3352853387 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606c029e710 con 0x5606c03e1000 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5abfff700 1 --2- 172.21.15.114:0/3352853387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e1000 0x5606c03ea9e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb59800f010 tx=0x7fb5980cc270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2cba6ff6556cf8a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb58bfff700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb58bfff700 1 -- 172.21.15.114:0/3352853387 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5980c2070 con 0x5606c03e1000 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3352853387 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5980a9970 con 0x5606c03e1000 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.524+0000 7fb58bfff700 1 -- 172.21.15.114:0/3352853387 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fb5980b2550 con 0x5606c03e1000 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.525+0000 7fb58bfff700 1 -- 172.21.15.114:0/3352853387 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb5980add20 con 0x5606c03e1000 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.525+0000 7fb5c1b62b80 10 client.25355 fetch_fsmap learned FSMap version 21 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.525+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3352853387 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5606c03fc6c0 con 0x5606c03e1000 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.525+0000 7fb58bfff700 1 -- 172.21.15.114:0/3352853387 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5980a9970 con 0x5606c03e1000 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.525+0000 7fb58bfff700 1 -- 172.21.15.114:0/3352853387 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb5980c6020 con 0x5606c03e1000 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.525+0000 7fb5c1b62b80 10 client.25355 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.525+0000 7fb5c1b62b80 10 client.25355 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.525+0000 7fb5c1b62b80 20 client.25355 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:08.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.525+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/3352853387 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5606c03feb90 0x5606c0400fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.525+0000 7fb5c1b62b80 4 client.25355 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:08.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.525+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3352853387 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606c03aca70 con 0x5606c03feb90 2024-05-27T07:42:08.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.525+0000 7fb5b0ee9700 1 --2- 172.21.15.114:0/3352853387 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5606c03feb90 0x5606c0400fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.526+0000 7fb58bfff700 10 client.25355 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:08.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.526+0000 7fb5b0ee9700 1 --2- 172.21.15.114:0/3352853387 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5606c03feb90 0x5606c0400fb0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7463d776bc1fed08 server_cookie=1363c35327ca4488 in_seq=0 out_seq=0 2024-05-27T07:42:08.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.527+0000 7fb58bfff700 1 -- 172.21.15.114:0/3352853387 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5606c03aca70 con 0x5606c03feb90 2024-05-27T07:42:08.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.527+0000 7fb58bfff700 10 client.25355 handle_command_reply: tid=0 2024-05-27T07:42:08.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.527+0000 7fb5c1b62b80 1 client.25355 shutdown 2024-05-27T07:42:08.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.528+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3352853387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e1000 msgr2=0x5606c03ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.528+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/3352853387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e1000 0x5606c03ea9e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb59800f010 tx=0x7fb5980cc270 comp rx=0 tx=0).stop 2024-05-27T07:42:08.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.528+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3352853387 shutdown_connections 2024-05-27T07:42:08.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.528+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/3352853387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e6cd0 0x5606c03eaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.528+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/3352853387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03ea4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.528+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/3352853387 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5606c03feb90 0x5606c0400fb0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.528+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/3352853387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e1000 0x5606c03ea9e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.528+0000 7fb58bfff700 0 client.25355 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:08.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.528+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3352853387 >> 172.21.15.114:0/3352853387 conn(0x5606c02bdb70 msgr2=0x5606c03dfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.528+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3352853387 shutdown_connections 2024-05-27T07:42:08.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.528+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3352853387 wait complete. 2024-05-27T07:42:08.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.528+0000 7fb5c1b62b80 20 client.25355 trim_cache size 0 max 16384 2024-05-27T07:42:08.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.549+0000 7fb5c1b62b80 1 Processor -- start 2024-05-27T07:42:08.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.549+0000 7fb5c1b62b80 1 -- start start 2024-05-27T07:42:08.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5c1b62b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03e0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5c1b62b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e1000 0x5606c03eae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5c1b62b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e6cd0 0x5606c03e9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5c1b62b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1ac380 con 0x5606c03e1000 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5c1b62b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1aec00 con 0x5606c03e6cd0 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5c1b62b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1ac9f0 con 0x5606c0262ec0 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5abfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e1000 0x5606c03eae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5abfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e1000 0x5606c03eae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33222/0 (socket says 172.21.15.114:33222) 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5abfff700 1 -- 172.21.15.114:0/956027277 learned_addr learned my addr 172.21.15.114:0/956027277 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5b16ea700 1 --2- 172.21.15.114:0/956027277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e6cd0 0x5606c03e9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5b0ee9700 1 --2- 172.21.15.114:0/956027277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03e0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5b16ea700 1 -- 172.21.15.114:0/956027277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 msgr2=0x5606c03e0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5b16ea700 1 --2- 172.21.15.114:0/956027277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03e0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5b16ea700 1 -- 172.21.15.114:0/956027277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e1000 msgr2=0x5606c03eae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5b16ea700 1 --2- 172.21.15.114:0/956027277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e1000 0x5606c03eae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5b16ea700 1 -- 172.21.15.114:0/956027277 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606c03e8a40 con 0x5606c03e6cd0 2024-05-27T07:42:08.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5abfff700 1 --2- 172.21.15.114:0/956027277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e1000 0x5606c03eae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:08.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.550+0000 7fb5b16ea700 1 --2- 172.21.15.114:0/956027277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e6cd0 0x5606c03e9da0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009fd0 tx=0x7fb5a40095d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72bcf9dfde826ae server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.551+0000 7fb5967fc700 1 -- 172.21.15.114:0/956027277 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4003a00 con 0x5606c03e6cd0 2024-05-27T07:42:08.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.551+0000 7fb5967fc700 1 -- 172.21.15.114:0/956027277 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5a4003ba0 con 0x5606c03e6cd0 2024-05-27T07:42:08.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.551+0000 7fb5967fc700 1 -- 172.21.15.114:0/956027277 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a40276e0 con 0x5606c03e6cd0 2024-05-27T07:42:08.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.552+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/956027277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e6cd0 msgr2=0x5606c03e9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.552+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/956027277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e6cd0 0x5606c03e9da0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009fd0 tx=0x7fb5a40095d0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.552+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/956027277 shutdown_connections 2024-05-27T07:42:08.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.552+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/956027277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e6cd0 0x5606c03e9da0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.552+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/956027277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c0262ec0 0x5606c03e0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.552+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/956027277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c03e1000 0x5606c03eae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.552+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/956027277 >> 172.21.15.114:0/956027277 conn(0x5606c02bdb70 msgr2=0x5606c03f59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.552+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/956027277 shutdown_connections 2024-05-27T07:42:08.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.552+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/956027277 wait complete. 2024-05-27T07:42:08.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.552+0000 7fb5c1b62b80 1 -- start start 2024-05-27T07:42:08.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.552+0000 7fb5c1b62b80 1 Processor -- start 2024-05-27T07:42:08.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5c1b62b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c0262ec0 0x5606c04040e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5c1b62b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c03e1000 0x5606c0404620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5c1b62b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e6cd0 0x5606c0402630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5c1b62b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1ad0c0 con 0x5606c0262ec0 2024-05-27T07:42:08.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5c1b62b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1ae530 con 0x5606c03e6cd0 2024-05-27T07:42:08.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5c1b62b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1ad790 con 0x5606c03e1000 2024-05-27T07:42:08.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5c1b62b80 10 client.0 _ll_register_callbacks cb 0x5606c029a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:08.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5b0ee9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c0262ec0 0x5606c04040e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5b0ee9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c0262ec0 0x5606c04040e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33226/0 (socket says 172.21.15.114:33226) 2024-05-27T07:42:08.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5b0ee9700 1 -- 172.21.15.114:0/3863811307 learned_addr learned my addr 172.21.15.114:0/3863811307 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5b16ea700 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e6cd0 0x5606c0402630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5abfff700 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c03e1000 0x5606c0404620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5b0ee9700 1 -- 172.21.15.114:0/3863811307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c03e1000 msgr2=0x5606c0404620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5b0ee9700 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c03e1000 0x5606c0404620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5b0ee9700 1 -- 172.21.15.114:0/3863811307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e6cd0 msgr2=0x5606c0402630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5b0ee9700 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e6cd0 0x5606c0402630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5b0ee9700 1 -- 172.21.15.114:0/3863811307 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606c0402c70 con 0x5606c0262ec0 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5b16ea700 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e6cd0 0x5606c0402630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb5b0ee9700 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c0262ec0 0x5606c04040e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009af0 tx=0x7fb5a00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15cbd88a1cbb8de6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.553+0000 7fb58bfff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb58bfff700 1 -- 172.21.15.114:0/3863811307 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a002e070 con 0x5606c0262ec0 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3863811307 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5a4003ba0 con 0x5606c0262ec0 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb58bfff700 1 -- 172.21.15.114:0/3863811307 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fb5a002a790 con 0x5606c0262ec0 2024-05-27T07:42:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb58bfff700 1 -- 172.21.15.114:0/3863811307 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb5a0003e60 con 0x5606c0262ec0 2024-05-27T07:42:08.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb5c1b62b80 10 client.15825 fetch_fsmap learned FSMap version 21 2024-05-27T07:42:08.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3863811307 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5606c0469f70 con 0x5606c0262ec0 2024-05-27T07:42:08.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb58bfff700 1 -- 172.21.15.114:0/3863811307 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5a4003ba0 con 0x5606c0262ec0 2024-05-27T07:42:08.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb58bfff700 1 -- 172.21.15.114:0/3863811307 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb5a001e420 con 0x5606c0262ec0 2024-05-27T07:42:08.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb5c1b62b80 10 client.15825 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:42:08.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb5c1b62b80 10 client.15825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:08.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb5c1b62b80 20 client.15825 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:08.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5606c046a660 0x5606c046ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb5c1b62b80 4 client.15825 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:42:08.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.554+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3863811307 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5606c0311c60 con 0x5606c046a660 2024-05-27T07:42:08.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.555+0000 7fb5abfff700 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5606c046a660 0x5606c046ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.555+0000 7fb58bfff700 10 client.15825 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:08.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.555+0000 7fb5abfff700 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5606c046a660 0x5606c046ca80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba3bdcc27748e8f server_cookie=b623d110fa58e9ef in_seq=0 out_seq=0 2024-05-27T07:42:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.557+0000 7fb58bfff700 1 -- 172.21.15.114:0/3863811307 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5606c0311c60 con 0x5606c046a660 2024-05-27T07:42:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.557+0000 7fb58bfff700 10 client.15825 handle_command_reply: tid=0 2024-05-27T07:42:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.557+0000 7fb5c1b62b80 1 client.15825 shutdown 2024-05-27T07:42:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.558+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3863811307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c0262ec0 msgr2=0x5606c04040e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.558+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c0262ec0 0x5606c04040e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009af0 tx=0x7fb5a00047c0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.558+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3863811307 shutdown_connections 2024-05-27T07:42:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.558+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5606c046a660 0x5606c046ca80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.558+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606c03e6cd0 0x5606c0402630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.558+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606c0262ec0 0x5606c04040e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.558+0000 7fb5c1b62b80 1 --2- 172.21.15.114:0/3863811307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606c03e1000 0x5606c0404620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.558+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3863811307 >> 172.21.15.114:0/3863811307 conn(0x5606c02bdb70 msgr2=0x5606c03eabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.558+0000 7fb58bfff700 0 client.15825 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.558+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3863811307 shutdown_connections 2024-05-27T07:42:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.558+0000 7fb5c1b62b80 1 -- 172.21.15.114:0/3863811307 wait complete. 2024-05-27T07:42:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.558+0000 7fb5c1b62b80 20 client.15825 trim_cache size 0 max 16384 2024-05-27T07:42:08.559 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:42:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:42:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "b43ee4b2-5ea0-4cb3-a548-4882b6e95049", 2024-05-27T07:42:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:42:08.559 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:42:08.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.559+0000 7fb5b314d700 1 -- 172.21.15.114:0/4177644587 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb59c0b2a70 msgr2=0x7fb59c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.559+0000 7fb5b314d700 1 --2- 172.21.15.114:0/4177644587 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb59c0b2a70 0x7fb59c0b4f80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009c60 tx=0x7fb5a0009260 comp rx=0 tx=0).stop 2024-05-27T07:42:08.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.559+0000 7fb5b314d700 1 -- 172.21.15.114:0/4177644587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac107de0 msgr2=0x7fb5ac19b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.559+0000 7fb5b314d700 1 --2- 172.21.15.114:0/4177644587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac107de0 0x7fb5ac19b3a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb59800e9c0 tx=0x7fb59800ecd0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.559+0000 7fb5b314d700 1 -- 172.21.15.114:0/4177644587 shutdown_connections 2024-05-27T07:42:08.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.559+0000 7fb5b314d700 1 --2- 172.21.15.114:0/4177644587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5ac108730 0x7fb5ac19f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.559+0000 7fb5b314d700 1 --2- 172.21.15.114:0/4177644587 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb59c0b2a70 0x7fb59c0b4f80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.559+0000 7fb5b314d700 1 --2- 172.21.15.114:0/4177644587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5ac1034b0 0x7fb5ac19ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.559+0000 7fb5b314d700 1 --2- 172.21.15.114:0/4177644587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac107de0 0x7fb5ac19b3a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.559+0000 7fb5b314d700 1 -- 172.21.15.114:0/4177644587 >> 172.21.15.114:0/4177644587 conn(0x7fb5ac0fedb0 msgr2=0x7fb5ac107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.559+0000 7fb5b314d700 1 -- 172.21.15.114:0/4177644587 shutdown_connections 2024-05-27T07:42:08.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:08.559+0000 7fb5b314d700 1 -- 172.21.15.114:0/4177644587 wait complete. 2024-05-27T07:42:09.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 59278 15.94 MB/sec warmup 21 sec latency 165.100 ms 2024-05-27T07:42:09.705 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:09 smithi114 ceph-mon[32038]: pgmap v75: 129 pgs: 129 active+clean; 21 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.5 MiB/s wr, 215 op/s 2024-05-27T07:42:09.705 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:09 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:42:09.706 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:09 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:09.706 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:09 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:42:09.706 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:09 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:09.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:42:09.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:09 smithi163 ceph-mon[36451]: pgmap v75: 129 pgs: 129 active+clean; 21 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.5 MiB/s wr, 215 op/s 2024-05-27T07:42:09.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:09 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:42:09.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:09 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:09.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:09 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:42:09.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:09 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:09.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:09 smithi186 ceph-mon[36647]: pgmap v75: 129 pgs: 129 active+clean; 21 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.5 MiB/s wr, 215 op/s 2024-05-27T07:42:09.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:09 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:42:09.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:09 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:09.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:09 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:42:09.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:09 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:10.036 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:10.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 61938 15.97 MB/sec warmup 22 sec latency 14.609 ms 2024-05-27T07:42:11.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 64675 15.90 MB/sec warmup 23 sec latency 44.557 ms 2024-05-27T07:42:11.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:11 smithi114 ceph-mon[32038]: pgmap v76: 129 pgs: 129 active+clean; 17 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 2.2 MiB/s wr, 466 op/s 2024-05-27T07:42:11.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.835+0000 7f35d2ea9700 1 -- 172.21.15.114:0/456941424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35cc1034d0 msgr2=0x7f35cc1038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:11.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.835+0000 7f35d2ea9700 1 --2- 172.21.15.114:0/456941424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35cc1034d0 0x7f35cc1038e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f35c00099c0 tx=0x7f35c0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.835+0000 7f35d2ea9700 1 -- 172.21.15.114:0/456941424 shutdown_connections 2024-05-27T07:42:11.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.835+0000 7f35d2ea9700 1 --2- 172.21.15.114:0/456941424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc108750 0x7f35cc1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.835+0000 7f35d2ea9700 1 --2- 172.21.15.114:0/456941424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35cc1034d0 0x7f35cc1038e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.835+0000 7f35d2ea9700 1 --2- 172.21.15.114:0/456941424 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc107e00 0x7f35cc108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.835+0000 7f35d2ea9700 1 -- 172.21.15.114:0/456941424 >> 172.21.15.114:0/456941424 conn(0x7f35cc0fef20 msgr2=0x7f35cc101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:11.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.835+0000 7f35d2ea9700 1 -- 172.21.15.114:0/456941424 shutdown_connections 2024-05-27T07:42:11.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.835+0000 7f35d2ea9700 1 -- 172.21.15.114:0/456941424 wait complete. 2024-05-27T07:42:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.836+0000 7f35d2ea9700 1 Processor -- start 2024-05-27T07:42:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.836+0000 7f35d2ea9700 1 -- start start 2024-05-27T07:42:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.836+0000 7f35d2ea9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc1034d0 0x7f35cc19bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.836+0000 7f35d2ea9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc107e00 0x7f35cc19c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d2ea9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35cc108750 0x7f35cc19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d2ea9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1133d0 con 0x7f35cc108750 2024-05-27T07:42:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d0c45700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc1034d0 0x7f35cc19bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d0c45700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc1034d0 0x7f35cc19bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54472/0 (socket says 172.21.15.114:54472) 2024-05-27T07:42:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d0c45700 1 -- 172.21.15.114:0/2373676789 learned_addr learned my addr 172.21.15.114:0/2373676789 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d2ea9700 1 -- 172.21.15.114:0/2373676789 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc113250 con 0x7f35cc107e00 2024-05-27T07:42:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d2ea9700 1 -- 172.21.15.114:0/2373676789 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc113550 con 0x7f35cc1034d0 2024-05-27T07:42:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d1446700 1 --2- 172.21.15.114:0/2373676789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35cc108750 0x7f35cc19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d0c45700 1 -- 172.21.15.114:0/2373676789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc107e00 msgr2=0x7f35cc19c230 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:42:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d0c45700 1 --2- 172.21.15.114:0/2373676789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc107e00 0x7f35cc19c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d0c45700 1 -- 172.21.15.114:0/2373676789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35cc108750 msgr2=0x7f35cc19c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d0c45700 1 --2- 172.21.15.114:0/2373676789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35cc108750 0x7f35cc19c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d0c45700 1 -- 172.21.15.114:0/2373676789 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35cc1a7040 con 0x7f35cc1034d0 2024-05-27T07:42:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.837+0000 7f35d0c45700 1 --2- 172.21.15.114:0/2373676789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc1034d0 0x7f35cc19bcf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f35c0005910 tx=0x7f35c0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.838+0000 7f35c9ffb700 1 -- 172.21.15.114:0/2373676789 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c0026070 con 0x7f35cc1034d0 2024-05-27T07:42:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.838+0000 7f35c9ffb700 1 -- 172.21.15.114:0/2373676789 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35c0003c70 con 0x7f35cc1034d0 2024-05-27T07:42:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.838+0000 7f35d2ea9700 1 -- 172.21.15.114:0/2373676789 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35cc1a72d0 con 0x7f35cc1034d0 2024-05-27T07:42:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.838+0000 7f35c9ffb700 1 -- 172.21.15.114:0/2373676789 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c000fb50 con 0x7f35cc1034d0 2024-05-27T07:42:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.838+0000 7f35d2ea9700 1 -- 172.21.15.114:0/2373676789 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35cc1a7720 con 0x7f35cc1034d0 2024-05-27T07:42:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.841+0000 7f35c9ffb700 1 -- 172.21.15.114:0/2373676789 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f35c0029070 con 0x7f35cc1034d0 2024-05-27T07:42:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35e18beb80 1 Processor -- start 2024-05-27T07:42:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35c9ffb700 1 --2- 172.21.15.114:0/2373676789 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35bc0b2a70 0x7f35bc0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35e18beb80 1 -- start start 2024-05-27T07:42:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35cbfff700 1 --2- 172.21.15.114:0/2373676789 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35bc0b2a70 0x7f35bc0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35c9ffb700 1 -- 172.21.15.114:0/2373676789 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f35c00e3200 con 0x7f35cc1034d0 2024-05-27T07:42:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35e18beb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 0x564c67951b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35e18beb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c6794be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35e18beb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c6794c390 0x564c6794c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35e18beb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564c67876c90 con 0x564c6794c390 2024-05-27T07:42:11.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35e18beb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564c67866830 con 0x564c67952060 2024-05-27T07:42:11.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35e18beb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564c6786a870 con 0x564c677c9e40 2024-05-27T07:42:11.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35d0c45700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 0x564c67951b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35d1446700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c6794c390 0x564c6794c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35d0c45700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 0x564c67951b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54484/0 (socket says 172.21.15.114:54484) 2024-05-27T07:42:11.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35d0c45700 1 -- 172.21.15.114:0/3159838794 learned_addr learned my addr 172.21.15.114:0/3159838794 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:11.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35cbfff700 1 --2- 172.21.15.114:0/3159838794 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c6794be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35cbfff700 1 --2- 172.21.15.114:0/2373676789 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35bc0b2a70 0x7f35bc0b4f80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f35cc1068e0 tx=0x7f35b8009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:11.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35d0c45700 1 -- 172.21.15.114:0/3159838794 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 msgr2=0x564c6794be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:11.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35d0c45700 1 --2- 172.21.15.114:0/3159838794 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c6794be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35d0c45700 1 -- 172.21.15.114:0/3159838794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c6794c390 msgr2=0x564c6794c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:11.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35d0c45700 1 --2- 172.21.15.114:0/3159838794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c6794c390 0x564c6794c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.842+0000 7f35d0c45700 1 -- 172.21.15.114:0/3159838794 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c677b68d0 con 0x564c677c9e40 2024-05-27T07:42:11.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.843+0000 7f35d0c45700 1 --2- 172.21.15.114:0/3159838794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 0x564c67951b20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f35c00aa150 tx=0x7f35c00036f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c0b88310206cf95 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:11.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.843+0000 7f35b6ffd700 1 -- 172.21.15.114:0/3159838794 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c00aa8c0 con 0x564c677c9e40 2024-05-27T07:42:11.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.843+0000 7f35b6ffd700 1 -- 172.21.15.114:0/3159838794 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35c00aa1c0 con 0x564c677c9e40 2024-05-27T07:42:11.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.843+0000 7f35b6ffd700 1 -- 172.21.15.114:0/3159838794 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c00b2d60 con 0x564c677c9e40 2024-05-27T07:42:11.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.844+0000 7f35e18beb80 1 -- 172.21.15.114:0/3159838794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 msgr2=0x564c67951b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:11.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.844+0000 7f35e18beb80 1 --2- 172.21.15.114:0/3159838794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 0x564c67951b20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f35c00aa150 tx=0x7f35c00036f0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.844+0000 7f35e18beb80 1 -- 172.21.15.114:0/3159838794 shutdown_connections 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.844+0000 7f35e18beb80 1 --2- 172.21.15.114:0/3159838794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c6794c390 0x564c6794c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.844+0000 7f35e18beb80 1 --2- 172.21.15.114:0/3159838794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 0x564c67951b20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.844+0000 7f35e18beb80 1 --2- 172.21.15.114:0/3159838794 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c6794be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.844+0000 7f35e18beb80 1 -- 172.21.15.114:0/3159838794 >> 172.21.15.114:0/3159838794 conn(0x564c67824af0 msgr2=0x564c67942640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.844+0000 7f35e18beb80 1 -- 172.21.15.114:0/3159838794 shutdown_connections 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.844+0000 7f35e18beb80 1 -- 172.21.15.114:0/3159838794 wait complete. 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.844+0000 7f35e18beb80 1 -- start start 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.845+0000 7f35e18beb80 1 Processor -- start 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.845+0000 7f35e18beb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 0x564c67955870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.845+0000 7f35d0c45700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 0x564c67955870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.845+0000 7f35d0c45700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 0x564c67955870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33252/0 (socket says 172.21.15.114:33252) 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.845+0000 7f35e18beb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c6794c390 0x564c67955db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.845+0000 7f35e18beb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c679562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.845+0000 7f35e18beb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564c6776cfa0 con 0x564c677c9e40 2024-05-27T07:42:11.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.845+0000 7f35e18beb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564c678087c0 con 0x564c67952060 2024-05-27T07:42:11.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.845+0000 7f35e18beb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564c67958910 con 0x564c6794c390 2024-05-27T07:42:11.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.845+0000 7f35d0c45700 1 -- 172.21.15.114:0/1579469869 learned_addr learned my addr 172.21.15.114:0/1579469869 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:11.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.846+0000 7f35e18beb80 10 client.0 _ll_register_callbacks cb 0x564c67801920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:11.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.846+0000 7f35d1446700 1 --2- 172.21.15.114:0/1579469869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c679562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.846+0000 7f35d0c45700 1 -- 172.21.15.114:0/1579469869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c6794c390 msgr2=0x564c67955db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:11.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.846+0000 7f35d0c45700 1 --2- 172.21.15.114:0/1579469869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c6794c390 0x564c67955db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.846+0000 7f35d0c45700 1 -- 172.21.15.114:0/1579469869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 msgr2=0x564c679562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:11.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.846+0000 7f35d0c45700 1 --2- 172.21.15.114:0/1579469869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c679562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.846+0000 7f35d0c45700 1 -- 172.21.15.114:0/1579469869 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c67805690 con 0x564c677c9e40 2024-05-27T07:42:11.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.846+0000 7f35d1446700 1 --2- 172.21.15.114:0/1579469869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c679562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:42:11.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.846+0000 7f35d0c45700 1 --2- 172.21.15.114:0/1579469869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 0x564c67955870 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f35c00e0ba0 tx=0x7f35c0003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=838c4cec51ed52b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:11.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.846+0000 7f35abfff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:11.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35abfff700 1 -- 172.21.15.114:0/1579469869 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c00b2f90 con 0x564c677c9e40 2024-05-27T07:42:11.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35abfff700 1 -- 172.21.15.114:0/1579469869 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f35c00aa630 con 0x564c677c9e40 2024-05-27T07:42:11.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35e18beb80 1 -- 172.21.15.114:0/1579469869 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35c00aa1c0 con 0x564c677c9e40 2024-05-27T07:42:11.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35abfff700 1 -- 172.21.15.114:0/1579469869 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f35c00aa8c0 con 0x564c677c9e40 2024-05-27T07:42:11.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35e18beb80 10 client.15837 fetch_fsmap learned FSMap version 21 2024-05-27T07:42:11.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35e18beb80 1 -- 172.21.15.114:0/1579469869 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564c67963330 con 0x564c677c9e40 2024-05-27T07:42:11.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35abfff700 1 -- 172.21.15.114:0/1579469869 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35c00aa1c0 con 0x564c677c9e40 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35abfff700 1 -- 172.21.15.114:0/1579469869 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f35c00a9540 con 0x564c677c9e40 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35e18beb80 10 client.15837 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35e18beb80 10 client.15837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35e18beb80 20 client.15837 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35e18beb80 1 --2- 172.21.15.114:0/1579469869 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c67965800 0x564c67967c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35e18beb80 4 client.15837 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.847+0000 7f35e18beb80 1 -- 172.21.15.114:0/1579469869 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c679139f0 con 0x564c67965800 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.848+0000 7f35cbfff700 1 --2- 172.21.15.114:0/1579469869 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c67965800 0x564c67967c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.848+0000 7f35abfff700 10 client.15837 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.848+0000 7f35cbfff700 1 --2- 172.21.15.114:0/1579469869 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c67965800 0x564c67967c20 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=390390684dedf57d server_cookie=147990c7203a052d in_seq=0 out_seq=0 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.849+0000 7f35abfff700 1 -- 172.21.15.114:0/1579469869 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564c679139f0 con 0x564c67965800 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.849+0000 7f35abfff700 10 client.15837 handle_command_reply: tid=0 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.849+0000 7f35e18beb80 1 client.15837 shutdown 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.849+0000 7f35e18beb80 1 -- 172.21.15.114:0/1579469869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 msgr2=0x564c67955870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:11.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.849+0000 7f35e18beb80 1 --2- 172.21.15.114:0/1579469869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 0x564c67955870 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f35c00e0ba0 tx=0x7f35c0003f40 comp rx=0 tx=0).stop 2024-05-27T07:42:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.849+0000 7f35e18beb80 1 -- 172.21.15.114:0/1579469869 shutdown_connections 2024-05-27T07:42:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.849+0000 7f35e18beb80 1 --2- 172.21.15.114:0/1579469869 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c67965800 0x564c67967c20 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.849+0000 7f35e18beb80 1 --2- 172.21.15.114:0/1579469869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c679562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.849+0000 7f35e18beb80 1 --2- 172.21.15.114:0/1579469869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 0x564c67955870 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.849+0000 7f35e18beb80 1 --2- 172.21.15.114:0/1579469869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c6794c390 0x564c67955db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.849+0000 7f35e18beb80 1 -- 172.21.15.114:0/1579469869 >> 172.21.15.114:0/1579469869 conn(0x564c67824af0 msgr2=0x564c6794ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.849+0000 7f35abfff700 0 client.15837 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.850+0000 7f35e18beb80 1 -- 172.21.15.114:0/1579469869 shutdown_connections 2024-05-27T07:42:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.850+0000 7f35e18beb80 1 -- 172.21.15.114:0/1579469869 wait complete. 2024-05-27T07:42:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.850+0000 7f35e18beb80 20 client.15837 trim_cache size 0 max 16384 2024-05-27T07:42:11.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.873+0000 7f35e18beb80 1 Processor -- start 2024-05-27T07:42:11.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.873+0000 7f35e18beb80 1 -- start start 2024-05-27T07:42:11.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.873+0000 7f35e18beb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 0x564c6794be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.873+0000 7f35e18beb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c6794c390 0x564c67956130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.873+0000 7f35e18beb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c67954870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.873+0000 7f35e18beb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1ac310 con 0x564c6794c390 2024-05-27T07:42:11.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.873+0000 7f35e18beb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1aeb90 con 0x564c67952060 2024-05-27T07:42:11.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.873+0000 7f35e18beb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1ac980 con 0x564c677c9e40 2024-05-27T07:42:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35cbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c6794c390 0x564c67956130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35d0c45700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 0x564c6794be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35cbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c6794c390 0x564c67956130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33266/0 (socket says 172.21.15.114:33266) 2024-05-27T07:42:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35cbfff700 1 -- 172.21.15.114:0/1566503408 learned_addr learned my addr 172.21.15.114:0/1566503408 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:11.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35d0c45700 1 --2- 172.21.15.114:0/1566503408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 0x564c6794be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54510/0 (socket says 172.21.15.114:54510) 2024-05-27T07:42:11.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35d1446700 1 --2- 172.21.15.114:0/1566503408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c67954870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35d0c45700 1 -- 172.21.15.114:0/1566503408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 msgr2=0x564c67954870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:11.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35d0c45700 1 --2- 172.21.15.114:0/1566503408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c67954870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35d0c45700 1 -- 172.21.15.114:0/1566503408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c6794c390 msgr2=0x564c67956130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:11.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35d0c45700 1 --2- 172.21.15.114:0/1566503408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c6794c390 0x564c67956130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35d0c45700 1 -- 172.21.15.114:0/1566503408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c67943ed0 con 0x564c677c9e40 2024-05-27T07:42:11.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35d1446700 1 --2- 172.21.15.114:0/1566503408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c67954870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:42:11.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35d0c45700 1 --2- 172.21.15.114:0/1566503408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 0x564c6794be50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f35c0003fb0 tx=0x7f35c0009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b49e1be31e01a36 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:11.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35b67fc700 1 -- 172.21.15.114:0/1566503408 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c00b2f90 con 0x564c677c9e40 2024-05-27T07:42:11.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35b67fc700 1 -- 172.21.15.114:0/1566503408 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35c00aa1c0 con 0x564c677c9e40 2024-05-27T07:42:11.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.874+0000 7f35b67fc700 1 -- 172.21.15.114:0/1566503408 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c00a9420 con 0x564c677c9e40 2024-05-27T07:42:11.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.875+0000 7f35e18beb80 1 -- 172.21.15.114:0/1566503408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 msgr2=0x564c6794be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:11.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.875+0000 7f35e18beb80 1 --2- 172.21.15.114:0/1566503408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 0x564c6794be50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f35c0003fb0 tx=0x7f35c0009310 comp rx=0 tx=0).stop 2024-05-27T07:42:11.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.875+0000 7f35e18beb80 1 -- 172.21.15.114:0/1566503408 shutdown_connections 2024-05-27T07:42:11.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.875+0000 7f35e18beb80 1 --2- 172.21.15.114:0/1566503408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c67952060 0x564c67954870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.875+0000 7f35e18beb80 1 --2- 172.21.15.114:0/1566503408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c677c9e40 0x564c6794be50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.875+0000 7f35e18beb80 1 --2- 172.21.15.114:0/1566503408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c6794c390 0x564c67956130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.875+0000 7f35e18beb80 1 -- 172.21.15.114:0/1566503408 >> 172.21.15.114:0/1566503408 conn(0x564c67824af0 msgr2=0x564c67954160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:11.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.875+0000 7f35e18beb80 1 -- 172.21.15.114:0/1566503408 shutdown_connections 2024-05-27T07:42:11.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.875+0000 7f35e18beb80 1 -- 172.21.15.114:0/1566503408 wait complete. 2024-05-27T07:42:11.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.876+0000 7f35e18beb80 1 -- start start 2024-05-27T07:42:11.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.876+0000 7f35e18beb80 1 Processor -- start 2024-05-27T07:42:11.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.876+0000 7f35e18beb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 0x564c6795db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.876+0000 7f35e18beb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c6794c390 0x564c6795e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.876+0000 7f35e18beb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c67952060 0x564c6796c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.876+0000 7f35e18beb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1ad050 con 0x564c677c9e40 2024-05-27T07:42:11.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.876+0000 7f35e18beb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1ae4c0 con 0x564c6794c390 2024-05-27T07:42:11.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.876+0000 7f35e18beb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1ad720 con 0x564c67952060 2024-05-27T07:42:11.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.876+0000 7f35e18beb80 10 client.0 _ll_register_callbacks cb 0x564c67801920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:11.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.876+0000 7f35d0c45700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 0x564c6795db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35d0c45700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 0x564c6795db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33268/0 (socket says 172.21.15.114:33268) 2024-05-27T07:42:11.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35d0c45700 1 -- 172.21.15.114:0/2216676297 learned_addr learned my addr 172.21.15.114:0/2216676297 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:11.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35d1446700 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c67952060 0x564c6796c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35cbfff700 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c6794c390 0x564c6795e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35d0c45700 1 -- 172.21.15.114:0/2216676297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c67952060 msgr2=0x564c6796c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:11.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35d0c45700 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c67952060 0x564c6796c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35d0c45700 1 -- 172.21.15.114:0/2216676297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c6794c390 msgr2=0x564c6795e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:11.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35d0c45700 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c6794c390 0x564c6795e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35d0c45700 1 -- 172.21.15.114:0/2216676297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c6796c940 con 0x564c677c9e40 2024-05-27T07:42:11.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35cbfff700 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c6794c390 0x564c6795e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:11.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35d0c45700 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 0x564c6795db60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f35c00a99d0 tx=0x7f35c00a5b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f068a2ad89819763 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:11.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35abfff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:11.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35abfff700 1 -- 172.21.15.114:0/2216676297 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c00b2b00 con 0x564c677c9e40 2024-05-27T07:42:11.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35abfff700 1 -- 172.21.15.114:0/2216676297 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f35c00b2ca0 con 0x564c677c9e40 2024-05-27T07:42:11.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35e18beb80 1 -- 172.21.15.114:0/2216676297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35c00aa1c0 con 0x564c677c9e40 2024-05-27T07:42:11.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35abfff700 1 -- 172.21.15.114:0/2216676297 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f35c00a9420 con 0x564c677c9e40 2024-05-27T07:42:11.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35e18beb80 10 client.15843 fetch_fsmap learned FSMap version 21 2024-05-27T07:42:11.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.877+0000 7f35e18beb80 1 -- 172.21.15.114:0/2216676297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564c6796c940 con 0x564c677c9e40 2024-05-27T07:42:11.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.878+0000 7f35abfff700 1 -- 172.21.15.114:0/2216676297 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35c00a95c0 con 0x564c677c9e40 2024-05-27T07:42:11.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.878+0000 7f35abfff700 1 -- 172.21.15.114:0/2216676297 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f35c00c4a20 con 0x564c677c9e40 2024-05-27T07:42:11.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.878+0000 7f35e18beb80 10 client.15843 fetch_fsmap finished waiting for FSMap version 21 2024-05-27T07:42:11.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.878+0000 7f35e18beb80 10 client.15843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:11.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.878+0000 7f35e18beb80 20 client.15843 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:11.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.878+0000 7f35e18beb80 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c679cfc40 0x564c679d2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:11.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.878+0000 7f35e18beb80 4 client.15843 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:42:11.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.878+0000 7f35e18beb80 1 -- 172.21.15.114:0/2216676297 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f35c00aa1c0 con 0x564c679cfc40 2024-05-27T07:42:11.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.878+0000 7f35cbfff700 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c679cfc40 0x564c679d2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:11.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.878+0000 7f35abfff700 10 client.15843 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:11.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.878+0000 7f35cbfff700 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c679cfc40 0x564c679d2060 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb77ea092d70a97 server_cookie=6666d12136be07d2 in_seq=0 out_seq=0 2024-05-27T07:42:11.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35abfff700 1 -- 172.21.15.114:0/2216676297 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f35c00aa1c0 con 0x564c679cfc40 2024-05-27T07:42:11.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35abfff700 10 client.15843 handle_command_reply: tid=0 2024-05-27T07:42:11.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35e18beb80 1 client.15843 shutdown 2024-05-27T07:42:11.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35e18beb80 1 -- 172.21.15.114:0/2216676297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 msgr2=0x564c6795db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:11.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35e18beb80 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 0x564c6795db60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f35c00a99d0 tx=0x7f35c00a5b80 comp rx=0 tx=0).stop 2024-05-27T07:42:11.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35e18beb80 1 -- 172.21.15.114:0/2216676297 shutdown_connections 2024-05-27T07:42:11.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35e18beb80 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c67952060 0x564c6796c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35e18beb80 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c677c9e40 0x564c6795db60 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35e18beb80 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c679cfc40 0x564c679d2060 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35e18beb80 1 --2- 172.21.15.114:0/2216676297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c6794c390 0x564c6795e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35abfff700 0 client.15843 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:11.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35e18beb80 1 -- 172.21.15.114:0/2216676297 >> 172.21.15.114:0/2216676297 conn(0x564c67824af0 msgr2=0x564c6794f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:11.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35e18beb80 1 -- 172.21.15.114:0/2216676297 shutdown_connections 2024-05-27T07:42:11.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35e18beb80 1 -- 172.21.15.114:0/2216676297 wait complete. 2024-05-27T07:42:11.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.879+0000 7f35e18beb80 20 client.15843 trim_cache size 0 max 16384 2024-05-27T07:42:11.885 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:42:11.885 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:42:11.886 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:42:11.886 INFO:teuthology.orchestra.run.smithi114.stdout: "b43ee4b2-5ea0-4cb3-a548-4882b6e95049": { 2024-05-27T07:42:11.886 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:42:11.886 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "b43ee4b2-5ea0-4cb3-a548-4882b6e95049", 2024-05-27T07:42:11.886 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:42:11.886 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:42:11.886 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:42:11.886 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:42:11.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.880+0000 7f35d2ea9700 1 -- 172.21.15.114:0/2373676789 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35bc0b2a70 msgr2=0x7f35bc0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:11.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.880+0000 7f35d2ea9700 1 --2- 172.21.15.114:0/2373676789 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35bc0b2a70 0x7f35bc0b4f80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f35cc1068e0 tx=0x7f35b8009260 comp rx=0 tx=0).stop 2024-05-27T07:42:11.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.880+0000 7f35d2ea9700 1 -- 172.21.15.114:0/2373676789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc1034d0 msgr2=0x7f35cc19bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:11.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.880+0000 7f35d2ea9700 1 --2- 172.21.15.114:0/2373676789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc1034d0 0x7f35cc19bcf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f35c0005910 tx=0x7f35c0004270 comp rx=0 tx=0).stop 2024-05-27T07:42:11.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.880+0000 7f35d2ea9700 1 -- 172.21.15.114:0/2373676789 shutdown_connections 2024-05-27T07:42:11.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.880+0000 7f35d2ea9700 1 --2- 172.21.15.114:0/2373676789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35cc108750 0x7f35cc19c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.880+0000 7f35d2ea9700 1 --2- 172.21.15.114:0/2373676789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc1034d0 0x7f35cc19bcf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.880+0000 7f35d2ea9700 1 --2- 172.21.15.114:0/2373676789 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35bc0b2a70 0x7f35bc0b4f80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.880+0000 7f35d2ea9700 1 --2- 172.21.15.114:0/2373676789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc107e00 0x7f35cc19c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:11.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.880+0000 7f35d2ea9700 1 -- 172.21.15.114:0/2373676789 >> 172.21.15.114:0/2373676789 conn(0x7f35cc0fef20 msgr2=0x7f35cc1096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:11.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.880+0000 7f35d2ea9700 1 -- 172.21.15.114:0/2373676789 shutdown_connections 2024-05-27T07:42:11.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:11.880+0000 7f35d2ea9700 1 -- 172.21.15.114:0/2373676789 wait complete. 2024-05-27T07:42:11.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:11 smithi163 ceph-mon[36451]: pgmap v76: 129 pgs: 129 active+clean; 17 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 2.2 MiB/s wr, 466 op/s 2024-05-27T07:42:11.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:11 smithi186 ceph-mon[36647]: pgmap v76: 129 pgs: 129 active+clean; 17 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 2.2 MiB/s wr, 466 op/s 2024-05-27T07:42:12.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 67984 16.01 MB/sec warmup 24 sec latency 7.848 ms 2024-05-27T07:42:12.664 INFO:tasks.cephfs.filesystem:scrub status for tag:b43ee4b2-5ea0-4cb3-a548-4882b6e95049 - {'path': '/', 'tag': 'b43ee4b2-5ea0-4cb3-a548-4882b6e95049', 'options': 'recursive,force'} 2024-05-27T07:42:12.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:12 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:42:12.948 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:12 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:42:12.948 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:12 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:42:12.948 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:12 smithi163 systemd[1]: Stopped Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:42:12.948 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:12 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Consumed 1.992s CPU time. 2024-05-27T07:42:12.949 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:12 smithi163 systemd[1]: Starting Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:42:12.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:12 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:42:13.224 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:12 smithi163 podman[70200]: 2024-05-27 07:42:12.853610802 +0000 UTC m=+0.017938953 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:42:13.224 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:12 smithi163 podman[70200]: 2024-05-27 07:42:12.983716295 +0000 UTC m=+0.148044445 container create ee3fa36442456b1013fa905ab00a55decc75fc632b228dd12e09b4db4193f479 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS) 2024-05-27T07:42:13.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 71289 16.12 MB/sec warmup 25 sec latency 33.186 ms 2024-05-27T07:42:13.515 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 podman[70200]: 2024-05-27 07:42:13.312032015 +0000 UTC m=+0.476360167 container init ee3fa36442456b1013fa905ab00a55decc75fc632b228dd12e09b4db4193f479 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:42:13.515 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 podman[70200]: 2024-05-27 07:42:13.314633795 +0000 UTC m=+0.478961959 container start ee3fa36442456b1013fa905ab00a55decc75fc632b228dd12e09b4db4193f479 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-27T07:42:13.515 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-mds[70218]: -- 172.21.15.163:0/883745156 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ba35a9ba40 con 0x55ba35ae3180 2024-05-27T07:42:13.515 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: starting mds.b at 2024-05-27T07:42:13.516 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 bash[70200]: ee3fa36442456b1013fa905ab00a55decc75fc632b228dd12e09b4db4193f479 2024-05-27T07:42:13.516 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 systemd[1]: Started Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:42:13.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:13 smithi114 ceph-mon[32038]: pgmap v77: 129 pgs: 129 active+clean; 17 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 1.5 MiB/s wr, 360 op/s 2024-05-27T07:42:13.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:13 smithi114 ceph-mon[32038]: daemon mds.b restarted 2024-05-27T07:42:13.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-mon[36451]: pgmap v77: 129 pgs: 129 active+clean; 17 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 1.5 MiB/s wr, 360 op/s 2024-05-27T07:42:13.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-mon[36451]: daemon mds.b restarted 2024-05-27T07:42:13.974 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9e81d15700 time 2024-05-27T07:42:13.672839+0000 2024-05-27T07:42:13.974 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:42:13.974 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9e9455aba6] 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9e9455add1] 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ba3485023d] 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ba3483b971] 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ba34839fa0] 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ba3483a651] 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ba347cf347] 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ba344cb87e] 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55ba344d0f2f] 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ba344db969] 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ba3487448c] 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ba3487d9cb] 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 13: (Thread::entry_wrapper()+0x43) [0x7f9e9452972d] 2024-05-27T07:42:13.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9e94529749] 2024-05-27T07:42:13.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9e930cd1ca] 2024-05-27T07:42:13.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 16: clone() 2024-05-27T07:42:13.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.674+0000 7f9e81d15700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9e81d15700 time 2024-05-27T07:42:13.672839+0000 2024-05-27T07:42:13.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:42:13.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:13.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9e9455aba6] 2024-05-27T07:42:13.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9e9455add1] 2024-05-27T07:42:13.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ba3485023d] 2024-05-27T07:42:13.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ba3483b971] 2024-05-27T07:42:13.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ba34839fa0] 2024-05-27T07:42:13.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ba3483a651] 2024-05-27T07:42:13.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ba347cf347] 2024-05-27T07:42:13.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ba344cb87e] 2024-05-27T07:42:13.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55ba344d0f2f] 2024-05-27T07:42:13.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ba344db969] 2024-05-27T07:42:13.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ba3487448c] 2024-05-27T07:42:13.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ba3487d9cb] 2024-05-27T07:42:13.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 13: (Thread::entry_wrapper()+0x43) [0x7f9e9452972d] 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9e94529749] 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9e930cd1ca] 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 16: clone() 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: *** Caught signal (Aborted) ** 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: in thread 7f9e81d15700 thread_name:md_log_replay 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9e930d7d20] 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2: gsignal() 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 3: abort() 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9e9455aced] 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9e9455add1] 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ba3485023d] 2024-05-27T07:42:13.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ba3483b971] 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ba34839fa0] 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ba3483a651] 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ba347cf347] 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ba344cb87e] 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55ba344d0f2f] 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ba344db969] 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ba3487448c] 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ba3487d9cb] 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 16: (Thread::entry_wrapper()+0x43) [0x7f9e9452972d] 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9e94529749] 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9e930cd1ca] 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 19: clone() 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.676+0000 7f9e81d15700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: in thread 7f9e81d15700 thread_name:md_log_replay 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.979 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9e930d7d20] 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2: gsignal() 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 3: abort() 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9e9455aced] 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9e9455add1] 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ba3485023d] 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ba3483b971] 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ba34839fa0] 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ba3483a651] 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ba347cf347] 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ba344cb87e] 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55ba344d0f2f] 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ba344db969] 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ba3487448c] 2024-05-27T07:42:13.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ba3487d9cb] 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 16: (Thread::entry_wrapper()+0x43) [0x7f9e9452972d] 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9e94529749] 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9e930cd1ca] 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 19: clone() 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: -1> 2024-05-27T07:42:13.674+0000 7f9e81d15700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9e81d15700 time 2024-05-27T07:42:13.672839+0000 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9e9455aba6] 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9e9455add1] 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ba3485023d] 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ba3483b971] 2024-05-27T07:42:13.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ba34839fa0] 2024-05-27T07:42:13.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ba3483a651] 2024-05-27T07:42:13.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ba347cf347] 2024-05-27T07:42:13.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ba344cb87e] 2024-05-27T07:42:13.982 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:13 smithi186 ceph-mon[36647]: pgmap v77: 129 pgs: 129 active+clean; 17 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 1.5 MiB/s wr, 360 op/s 2024-05-27T07:42:13.982 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:13 smithi186 ceph-mon[36647]: daemon mds.b restarted 2024-05-27T07:42:13.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55ba344d0f2f] 2024-05-27T07:42:13.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ba344db969] 2024-05-27T07:42:13.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ba3487448c] 2024-05-27T07:42:13.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ba3487d9cb] 2024-05-27T07:42:13.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 13: (Thread::entry_wrapper()+0x43) [0x7f9e9452972d] 2024-05-27T07:42:13.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9e94529749] 2024-05-27T07:42:13.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9e930cd1ca] 2024-05-27T07:42:13.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 16: clone() 2024-05-27T07:42:13.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 0> 2024-05-27T07:42:13.676+0000 7f9e81d15700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:42:13.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: in thread 7f9e81d15700 thread_name:md_log_replay 2024-05-27T07:42:13.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9e930d7d20] 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2: gsignal() 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 3: abort() 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9e9455aced] 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9e9455add1] 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ba3485023d] 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ba3483b971] 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ba34839fa0] 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ba3483a651] 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ba347cf347] 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ba344cb87e] 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55ba344d0f2f] 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ba344db969] 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ba3487448c] 2024-05-27T07:42:13.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ba3487d9cb] 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 16: (Thread::entry_wrapper()+0x43) [0x7f9e9452972d] 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9e94529749] 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9e930cd1ca] 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 19: clone() 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: -1> 2024-05-27T07:42:13.674+0000 7f9e81d15700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9e81d15700 time 2024-05-27T07:42:13.672839+0000 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9e9455aba6] 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9e9455add1] 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ba3485023d] 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ba3483b971] 2024-05-27T07:42:13.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ba34839fa0] 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ba3483a651] 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ba347cf347] 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ba344cb87e] 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55ba344d0f2f] 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ba344db969] 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ba3487448c] 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ba3487d9cb] 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 13: (Thread::entry_wrapper()+0x43) [0x7f9e9452972d] 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9e94529749] 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9e930cd1ca] 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 16: clone() 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 0> 2024-05-27T07:42:13.676+0000 7f9e81d15700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:42:13.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: in thread 7f9e81d15700 thread_name:md_log_replay 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9e930d7d20] 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2: gsignal() 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 3: abort() 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9e9455aced] 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9e9455add1] 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ba3485023d] 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ba3483b971] 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ba34839fa0] 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ba3483a651] 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ba347cf347] 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ba344cb87e] 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55ba344d0f2f] 2024-05-27T07:42:13.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ba344db969] 2024-05-27T07:42:13.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ba3487448c] 2024-05-27T07:42:13.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ba3487d9cb] 2024-05-27T07:42:13.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 16: (Thread::entry_wrapper()+0x43) [0x7f9e9452972d] 2024-05-27T07:42:13.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9e94529749] 2024-05-27T07:42:13.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9e930cd1ca] 2024-05-27T07:42:13.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 19: clone() 2024-05-27T07:42:13.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:42:13.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70214]: 2024-05-27T07:42:13.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:13 smithi163 podman[70249]: 2024-05-27 07:42:13.731631753 +0000 UTC m=+0.018450327 container died ee3fa36442456b1013fa905ab00a55decc75fc632b228dd12e09b4db4193f479 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-27T07:42:14.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 74548 16.19 MB/sec warmup 26 sec latency 14.131 ms 2024-05-27T07:42:14.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:14 smithi163 ceph-mon[36451]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T07:42:14.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:14 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6838/1630381172,v1:172.21.15.163:6839/1630381172] up:boot 2024-05-27T07:42:14.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:14 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:42:14.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:14 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:42:14.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:14 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:42:14.724 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:14 smithi163 podman[70249]: 2024-05-27 07:42:14.367360768 +0000 UTC m=+0.654179380 container remove ee3fa36442456b1013fa905ab00a55decc75fc632b228dd12e09b4db4193f479 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-27T07:42:14.724 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:14 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:42:14.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:14 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-05-27T07:42:14.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:14 smithi114 ceph-mon[32038]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T07:42:14.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:14 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6838/1630381172,v1:172.21.15.163:6839/1630381172] up:boot 2024-05-27T07:42:14.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:14 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:42:14.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:14 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:42:14.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:14 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:42:14.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:14 smithi186 ceph-mon[36647]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T07:42:14.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:14 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6838/1630381172,v1:172.21.15.163:6839/1630381172] up:boot 2024-05-27T07:42:14.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:14 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:42:14.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:14 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:42:14.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:14 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:42:15.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 77476 16.11 MB/sec warmup 27 sec latency 15.365 ms 2024-05-27T07:42:15.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:15 smithi114 ceph-mon[32038]: pgmap v79: 129 pgs: 129 active+clean; 17 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.9 MiB/s wr, 432 op/s 2024-05-27T07:42:15.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:15 smithi163 ceph-mon[36451]: pgmap v79: 129 pgs: 129 active+clean; 17 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.9 MiB/s wr, 432 op/s 2024-05-27T07:42:15.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:15 smithi186 ceph-mon[36647]: pgmap v79: 129 pgs: 129 active+clean; 17 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.9 MiB/s wr, 432 op/s 2024-05-27T07:42:16.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 80455 16.02 MB/sec warmup 28 sec latency 25.527 ms 2024-05-27T07:42:17.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 83088 16.06 MB/sec warmup 29 sec latency 15.216 ms 2024-05-27T07:42:17.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:17 smithi114 ceph-mon[32038]: pgmap v80: 129 pgs: 129 active+clean; 25 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.9 MiB/s wr, 423 op/s 2024-05-27T07:42:17.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:17 smithi163 ceph-mon[36451]: pgmap v80: 129 pgs: 129 active+clean; 25 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.9 MiB/s wr, 423 op/s 2024-05-27T07:42:17.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:17 smithi186 ceph-mon[36647]: pgmap v80: 129 pgs: 129 active+clean; 25 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.9 MiB/s wr, 423 op/s 2024-05-27T07:42:18.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 86063 16.02 MB/sec warmup 30 sec latency 15.329 ms 2024-05-27T07:42:19.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 89200 16.10 MB/sec warmup 31 sec latency 25.707 ms 2024-05-27T07:42:19.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:19 smithi114 ceph-mon[32038]: pgmap v81: 129 pgs: 129 active+clean; 25 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.9 MiB/s wr, 423 op/s 2024-05-27T07:42:19.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:19 smithi163 ceph-mon[36451]: pgmap v81: 129 pgs: 129 active+clean; 25 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.9 MiB/s wr, 423 op/s 2024-05-27T07:42:19.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:19 smithi186 ceph-mon[36647]: pgmap v81: 129 pgs: 129 active+clean; 25 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.9 MiB/s wr, 423 op/s 2024-05-27T07:42:20.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 89436 15.60 MB/sec warmup 32 sec latency 383.560 ms 2024-05-27T07:42:21.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 91492 15.46 MB/sec warmup 33 sec latency 458.603 ms 2024-05-27T07:42:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.494+0000 7f8df628b640 1 -- 172.21.15.114:0/3281724979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df0108b70 msgr2=0x7f8df0101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.494+0000 7f8df628b640 1 --2- 172.21.15.114:0/3281724979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df0108b70 0x7f8df0101c80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009860 tx=0x7f8de002a3f0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.494+0000 7f8df628b640 1 -- 172.21.15.114:0/3281724979 shutdown_connections 2024-05-27T07:42:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.494+0000 7f8df628b640 1 --2- 172.21.15.114:0/3281724979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df0107410 0x7f8df0107820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.494+0000 7f8df628b640 1 --2- 172.21.15.114:0/3281724979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8df0108f80 0x7f8df0102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.494+0000 7f8df628b640 1 --2- 172.21.15.114:0/3281724979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df0108b70 0x7f8df0101c80 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.494+0000 7f8df628b640 1 -- 172.21.15.114:0/3281724979 >> 172.21.15.114:0/3281724979 conn(0x7f8df00779d0 msgr2=0x7f8df0077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.495+0000 7f8df628b640 1 -- 172.21.15.114:0/3281724979 shutdown_connections 2024-05-27T07:42:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.495+0000 7f8df628b640 1 -- 172.21.15.114:0/3281724979 wait complete. 2024-05-27T07:42:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.496+0000 7f8df628b640 1 Processor -- start 2024-05-27T07:42:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.496+0000 7f8df628b640 1 -- start start 2024-05-27T07:42:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.497+0000 7f8df628b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df0107410 0x7f8df01a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.497+0000 7f8df628b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df0108b70 0x7f8df01a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.497+0000 7f8df628b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8df0108f80 0x7f8df01a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.497+0000 7f8df628b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8df011a4a0 con 0x7f8df0108b70 2024-05-27T07:42:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.497+0000 7f8df628b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8df011a320 con 0x7f8df0108f80 2024-05-27T07:42:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.497+0000 7f8df628b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8df011a620 con 0x7f8df0107410 2024-05-27T07:42:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.498+0000 7f8deffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df0107410 0x7f8df01a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.498+0000 7f8deffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df0107410 0x7f8df01a0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35336/0 (socket says 172.21.15.114:35336) 2024-05-27T07:42:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.498+0000 7f8deffff640 1 -- 172.21.15.114:0/4231335381 learned_addr learned my addr 172.21.15.114:0/4231335381 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.498+0000 7f8df4828640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8df0108f80 0x7f8df01a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.499+0000 7f8def7fe640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df0108b70 0x7f8df01a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.500+0000 7f8deffff640 1 -- 172.21.15.114:0/4231335381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8df0108f80 msgr2=0x7f8df01a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.500+0000 7f8deffff640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8df0108f80 0x7f8df01a5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.500+0000 7f8deffff640 1 -- 172.21.15.114:0/4231335381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df0108b70 msgr2=0x7f8df01a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.500+0000 7f8deffff640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df0108b70 0x7f8df01a1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.500+0000 7f8deffff640 1 -- 172.21.15.114:0/4231335381 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df01a5d40 con 0x7f8df0107410 2024-05-27T07:42:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.501+0000 7f8deffff640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df0107410 0x7f8df01a0ee0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8de0030f70 tx=0x7f8de0030fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:21.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.502+0000 7f8ded7fa640 1 -- 172.21.15.114:0/4231335381 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0040020 con 0x7f8df0107410 2024-05-27T07:42:21.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.502+0000 7f8ded7fa640 1 -- 172.21.15.114:0/4231335381 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8de0005520 con 0x7f8df0107410 2024-05-27T07:42:21.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.502+0000 7f8ded7fa640 1 -- 172.21.15.114:0/4231335381 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de00046c0 con 0x7f8df0107410 2024-05-27T07:42:21.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.502+0000 7f8df628b640 1 -- 172.21.15.114:0/4231335381 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df01a5fd0 con 0x7f8df0107410 2024-05-27T07:42:21.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.502+0000 7f8df628b640 1 -- 172.21.15.114:0/4231335381 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df01a64c0 con 0x7f8df0107410 2024-05-27T07:42:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.506+0000 7f8dfd211740 1 Processor -- start 2024-05-27T07:42:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.506+0000 7f8dfd211740 1 -- start start 2024-05-27T07:42:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.507+0000 7f8dfd211740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbccc1b0 0x55edbc06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.507+0000 7f8dfd211740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbca32f0 0x55edbc06d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.507+0000 7f8dfd211740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbcab4d0 0x55edbc06de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.507+0000 7f8dfd211740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55edbbfdfc30 con 0x55edbbcab4d0 2024-05-27T07:42:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.507+0000 7f8dfd211740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55edbbfcf7d0 con 0x55edbbca32f0 2024-05-27T07:42:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.507+0000 7f8dfd211740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55edbbfd3810 con 0x55edbbccc1b0 2024-05-27T07:42:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.507+0000 7f8def7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbcab4d0 0x55edbc06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.507+0000 7f8def7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbcab4d0 0x55edbc06de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41660/0 (socket says 172.21.15.114:41660) 2024-05-27T07:42:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.507+0000 7f8def7fe640 1 -- 172.21.15.114:0/1666649989 learned_addr learned my addr 172.21.15.114:0/1666649989 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.507+0000 7f8df4828640 1 --2- 172.21.15.114:0/1666649989 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbccc1b0 0x55edbc06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.508+0000 7f8ded7fa640 1 -- 172.21.15.114:0/4231335381 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f8de0043020 con 0x7f8df0107410 2024-05-27T07:42:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.508+0000 7f8def7fe640 1 -- 172.21.15.114:0/1666649989 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbccc1b0 msgr2=0x55edbc06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.508+0000 7f8def7fe640 1 --2- 172.21.15.114:0/1666649989 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbccc1b0 0x55edbc06d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.508+0000 7f8def7fe640 1 -- 172.21.15.114:0/1666649989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbca32f0 msgr2=0x55edbc06d910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:42:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.508+0000 7f8def7fe640 1 --2- 172.21.15.114:0/1666649989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbca32f0 0x55edbc06d910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.508+0000 7f8def7fe640 1 -- 172.21.15.114:0/1666649989 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edbbce4210 con 0x55edbbcab4d0 2024-05-27T07:42:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.508+0000 7f8def7fe640 1 --2- 172.21.15.114:0/1666649989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbcab4d0 0x55edbc06de50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc009c20 tx=0x7f8ddc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=533ab264184218e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.509+0000 7f8ded7fa640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8dd40b2b60 0x7f8dd40b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.509+0000 7f8ded7fa640 1 -- 172.21.15.114:0/4231335381 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9466+0+0 (secure 0 0 0) 0x7f8de01036d0 con 0x7f8df0107410 2024-05-27T07:42:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.509+0000 7f8de67fc640 1 -- 172.21.15.114:0/1666649989 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc01e070 con 0x55edbbcab4d0 2024-05-27T07:42:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.509+0000 7f8de67fc640 1 -- 172.21.15.114:0/1666649989 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ddc009dd0 con 0x55edbbcab4d0 2024-05-27T07:42:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.509+0000 7f8de67fc640 1 -- 172.21.15.114:0/1666649989 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc018730 con 0x55edbbcab4d0 2024-05-27T07:42:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.509+0000 7f8df4828640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8dd40b2b60 0x7f8dd40b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.511+0000 7f8dfd211740 1 -- 172.21.15.114:0/1666649989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbcab4d0 msgr2=0x55edbc06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.511+0000 7f8df4828640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8dd40b2b60 0x7f8dd40b5070 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55edbc06a540 tx=0x7f8de8010040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.511+0000 7f8dfd211740 1 --2- 172.21.15.114:0/1666649989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbcab4d0 0x55edbc06de50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc009c20 tx=0x7f8ddc011810 comp rx=0 tx=0).stop 2024-05-27T07:42:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.511+0000 7f8dfd211740 1 -- 172.21.15.114:0/1666649989 shutdown_connections 2024-05-27T07:42:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.511+0000 7f8dfd211740 1 --2- 172.21.15.114:0/1666649989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbcab4d0 0x55edbc06de50 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.511+0000 7f8dfd211740 1 --2- 172.21.15.114:0/1666649989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbca32f0 0x55edbc06d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.511+0000 7f8dfd211740 1 --2- 172.21.15.114:0/1666649989 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbccc1b0 0x55edbc06d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.511+0000 7f8dfd211740 1 -- 172.21.15.114:0/1666649989 >> 172.21.15.114:0/1666649989 conn(0x55edbbcc0300 msgr2=0x55edbc069b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.511+0000 7f8dfd211740 1 -- 172.21.15.114:0/1666649989 shutdown_connections 2024-05-27T07:42:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.511+0000 7f8dfd211740 1 -- 172.21.15.114:0/1666649989 wait complete. 2024-05-27T07:42:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.512+0000 7f8dfd211740 1 -- start start 2024-05-27T07:42:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.512+0000 7f8dfd211740 1 Processor -- start 2024-05-27T07:42:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8dfd211740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbca32f0 0x55edbc079420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8dfd211740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbcab4d0 0x55edbc079960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8dfd211740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 0x55edbc077930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8dfd211740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55edbbcb5790 con 0x55edbbccc1b0 2024-05-27T07:42:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8dfd211740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55edbbf8b890 con 0x55edbbcab4d0 2024-05-27T07:42:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8dfd211740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55edbc07a500 con 0x55edbbca32f0 2024-05-27T07:42:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8dfd211740 10 client.0 _ll_register_callbacks cb 0x55edbbce3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8deffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbca32f0 0x55edbc079420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8deffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbca32f0 0x55edbc079420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35346/0 (socket says 172.21.15.114:35346) 2024-05-27T07:42:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8deffff640 1 -- 172.21.15.114:0/2890983460 learned_addr learned my addr 172.21.15.114:0/2890983460 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8def7fe640 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbcab4d0 0x55edbc079960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8df4828640 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 0x55edbc077930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8deffff640 1 -- 172.21.15.114:0/2890983460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbcab4d0 msgr2=0x55edbc079960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8deffff640 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbcab4d0 0x55edbc079960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8deffff640 1 -- 172.21.15.114:0/2890983460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 msgr2=0x55edbc077930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8deffff640 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 0x55edbc077930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8deffff640 1 -- 172.21.15.114:0/2890983460 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edbbdb0620 con 0x55edbbca32f0 2024-05-27T07:42:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.513+0000 7f8df4828640 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 0x55edbc077930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.514+0000 7f8deffff640 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbca32f0 0x55edbc079420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8de0004650 tx=0x7f8de0100fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2ceaa1110c7343a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.514+0000 7f8dd37fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:42:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.515+0000 7f8dd37fe640 1 -- 172.21.15.114:0/2890983460 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0004470 con 0x55edbbca32f0 2024-05-27T07:42:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.515+0000 7f8dd37fe640 1 -- 172.21.15.114:0/2890983460 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9466+0+0 (secure 0 0 0) 0x7f8de00c6b00 con 0x55edbbca32f0 2024-05-27T07:42:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.515+0000 7f8dfd211740 1 -- 172.21.15.114:0/2890983460 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ddc009dd0 con 0x55edbbca32f0 2024-05-27T07:42:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.515+0000 7f8dd37fe640 1 -- 172.21.15.114:0/2890983460 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8de0031400 con 0x55edbbca32f0 2024-05-27T07:42:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.515+0000 7f8dfd211740 10 client.25453 fetch_fsmap learned FSMap version 23 2024-05-27T07:42:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.515+0000 7f8dfd211740 1 -- 172.21.15.114:0/2890983460 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55edbc086850 con 0x55edbbca32f0 2024-05-27T07:42:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.516+0000 7f8dd37fe640 1 -- 172.21.15.114:0/2890983460 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ddc009dd0 con 0x55edbbca32f0 2024-05-27T07:42:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.516+0000 7f8dd37fe640 1 -- 172.21.15.114:0/2890983460 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f8de00d2f70 con 0x55edbbca32f0 2024-05-27T07:42:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.516+0000 7f8dfd211740 10 client.25453 fetch_fsmap finished waiting for FSMap version 23 2024-05-27T07:42:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.516+0000 7f8dfd211740 10 client.25453 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.516+0000 7f8dfd211740 20 client.25453 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.516+0000 7f8dfd211740 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edbbc8ad80 0x55edbc089d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.516+0000 7f8dfd211740 4 client.25453 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.516+0000 7f8dfd211740 1 -- 172.21.15.114:0/2890983460 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edbbfe1b80 con 0x55edbbc8ad80 2024-05-27T07:42:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.517+0000 7f8def7fe640 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edbbc8ad80 0x55edbc089d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.517+0000 7f8def7fe640 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edbbc8ad80 0x55edbc089d10 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa6387b3af38d05a server_cookie=3f6baf37efcb128a in_seq=0 out_seq=0 2024-05-27T07:42:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.518+0000 7f8dd37fe640 10 client.25453 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.519+0000 7f8dd37fe640 1 -- 172.21.15.114:0/2890983460 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55edbbfe1b80 con 0x55edbbc8ad80 2024-05-27T07:42:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.519+0000 7f8dd37fe640 10 client.25453 handle_command_reply: tid=0 2024-05-27T07:42:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.519+0000 7f8dfd211740 1 client.25453 shutdown 2024-05-27T07:42:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.519+0000 7f8dfd211740 1 -- 172.21.15.114:0/2890983460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbca32f0 msgr2=0x55edbc079420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.519+0000 7f8dfd211740 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbca32f0 0x55edbc079420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8de0004650 tx=0x7f8de0100fd0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.520+0000 7f8dfd211740 1 -- 172.21.15.114:0/2890983460 shutdown_connections 2024-05-27T07:42:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.520+0000 7f8dfd211740 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edbbc8ad80 0x55edbc089d10 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.520+0000 7f8dfd211740 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 0x55edbc077930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.520+0000 7f8dfd211740 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbcab4d0 0x55edbc079960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.520+0000 7f8dfd211740 1 --2- 172.21.15.114:0/2890983460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbca32f0 0x55edbc079420 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8de0004650 tx=0x7f8de0100fd0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.520+0000 7f8dfd211740 1 -- 172.21.15.114:0/2890983460 >> 172.21.15.114:0/2890983460 conn(0x55edbbcc0300 msgr2=0x55edbc06f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:21.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.520+0000 7f8dd37fe640 0 client.25453 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:21.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.520+0000 7f8dfd211740 1 -- 172.21.15.114:0/2890983460 shutdown_connections 2024-05-27T07:42:21.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.520+0000 7f8dfd211740 1 -- 172.21.15.114:0/2890983460 wait complete. 2024-05-27T07:42:21.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.520+0000 7f8dfd211740 20 client.25453 trim_cache size 0 max 16384 2024-05-27T07:42:21.551 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.550+0000 7f8dfd211740 1 Processor -- start 2024-05-27T07:42:21.551 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.551+0000 7f8dfd211740 1 -- start start 2024-05-27T07:42:21.551 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.551+0000 7f8dfd211740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbcab4d0 0x55edbc06f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.551 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.551+0000 7f8dfd211740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 0x55edbc08f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.551+0000 7f8dfd211740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbc601c0 0x55edbc08f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.552+0000 7f8deffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbcab4d0 0x55edbc06f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.552+0000 7f8deffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbcab4d0 0x55edbc06f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35362/0 (socket says 172.21.15.114:35362) 2024-05-27T07:42:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.552+0000 7f8deffff640 1 -- 172.21.15.114:0/901367880 learned_addr learned my addr 172.21.15.114:0/901367880 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.552+0000 7f8dfd211740 1 -- 172.21.15.114:0/901367880 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8df01b5910 con 0x55edbbccc1b0 2024-05-27T07:42:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.552+0000 7f8dfd211740 1 -- 172.21.15.114:0/901367880 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8df01b8190 con 0x55edbbc601c0 2024-05-27T07:42:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.552+0000 7f8dfd211740 1 -- 172.21.15.114:0/901367880 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8df01b5f80 con 0x55edbbcab4d0 2024-05-27T07:42:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.552+0000 7f8df4828640 1 --2- 172.21.15.114:0/901367880 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbc601c0 0x55edbc08f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.553+0000 7f8deffff640 1 -- 172.21.15.114:0/901367880 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbc601c0 msgr2=0x55edbc08f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.553+0000 7f8deffff640 1 --2- 172.21.15.114:0/901367880 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbc601c0 0x55edbc08f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.553+0000 7f8deffff640 1 -- 172.21.15.114:0/901367880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 msgr2=0x55edbc08f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.553+0000 7f8deffff640 1 --2- 172.21.15.114:0/901367880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 0x55edbc08f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.553+0000 7f8deffff640 1 -- 172.21.15.114:0/901367880 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edbc06a770 con 0x55edbbcab4d0 2024-05-27T07:42:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.553+0000 7f8deffff640 1 --2- 172.21.15.114:0/901367880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbcab4d0 0x55edbc06f790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8de00c6ab0 tx=0x7f8de0103600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9af4f708253b2c73 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.553+0000 7f8de5ffb640 1 -- 172.21.15.114:0/901367880 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de00c9ef0 con 0x55edbbcab4d0 2024-05-27T07:42:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.553+0000 7f8de5ffb640 1 -- 172.21.15.114:0/901367880 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8de00c8780 con 0x55edbbcab4d0 2024-05-27T07:42:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.553+0000 7f8de5ffb640 1 -- 172.21.15.114:0/901367880 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de00caca0 con 0x55edbbcab4d0 2024-05-27T07:42:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.555+0000 7f8dfd211740 1 -- 172.21.15.114:0/901367880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbcab4d0 msgr2=0x55edbc06f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.555+0000 7f8dfd211740 1 --2- 172.21.15.114:0/901367880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbcab4d0 0x55edbc06f790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8de00c6ab0 tx=0x7f8de0103600 comp rx=0 tx=0).stop 2024-05-27T07:42:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.555+0000 7f8dfd211740 1 -- 172.21.15.114:0/901367880 shutdown_connections 2024-05-27T07:42:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.555+0000 7f8dfd211740 1 --2- 172.21.15.114:0/901367880 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbc601c0 0x55edbc08f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.555+0000 7f8dfd211740 1 --2- 172.21.15.114:0/901367880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 0x55edbc08f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.555+0000 7f8dfd211740 1 --2- 172.21.15.114:0/901367880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbcab4d0 0x55edbc06f790 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.555+0000 7f8dfd211740 1 -- 172.21.15.114:0/901367880 >> 172.21.15.114:0/901367880 conn(0x55edbbcc0300 msgr2=0x55edbc06f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.556+0000 7f8dfd211740 1 -- 172.21.15.114:0/901367880 shutdown_connections 2024-05-27T07:42:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.556+0000 7f8dfd211740 1 -- 172.21.15.114:0/901367880 wait complete. 2024-05-27T07:42:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.557+0000 7f8dfd211740 1 -- start start 2024-05-27T07:42:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.557+0000 7f8dfd211740 1 Processor -- start 2024-05-27T07:42:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.558+0000 7f8dfd211740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbc601c0 0x55edbc07f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.558+0000 7f8dfd211740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbcab4d0 0x55edbc07fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.558+0000 7f8dfd211740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 0x55edbc07cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.559+0000 7f8dfd211740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8df01b6650 con 0x55edbbccc1b0 2024-05-27T07:42:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.559+0000 7f8dfd211740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8df01b7ac0 con 0x55edbbc601c0 2024-05-27T07:42:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.559+0000 7f8dfd211740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8df01b6d20 con 0x55edbbcab4d0 2024-05-27T07:42:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.559+0000 7f8dfd211740 10 client.0 _ll_register_callbacks cb 0x7f8de0009540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.560+0000 7f8deffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbc601c0 0x55edbc07f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.561+0000 7f8df4828640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 0x55edbc07cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.561+0000 7f8df4828640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 0x55edbc07cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41700/0 (socket says 172.21.15.114:41700) 2024-05-27T07:42:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.561+0000 7f8df4828640 1 -- 172.21.15.114:0/3876593539 learned_addr learned my addr 172.21.15.114:0/3876593539 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.561+0000 7f8def7fe640 1 --2- 172.21.15.114:0/3876593539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbcab4d0 0x55edbc07fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.562+0000 7f8deffff640 1 -- 172.21.15.114:0/3876593539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbcab4d0 msgr2=0x55edbc07fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.562+0000 7f8deffff640 1 --2- 172.21.15.114:0/3876593539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbcab4d0 0x55edbc07fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.562+0000 7f8deffff640 1 -- 172.21.15.114:0/3876593539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 msgr2=0x55edbc07cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.562+0000 7f8deffff640 1 --2- 172.21.15.114:0/3876593539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 0x55edbc07cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.562+0000 7f8deffff640 1 -- 172.21.15.114:0/3876593539 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edbc07d2c0 con 0x55edbbc601c0 2024-05-27T07:42:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.562+0000 7f8deffff640 1 --2- 172.21.15.114:0/3876593539 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbc601c0 0x55edbc07f790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8de0004650 tx=0x7f8de0100fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaa663f479417cd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.563+0000 7f8dd37fe640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:42:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.563+0000 7f8dd37fe640 1 -- 172.21.15.114:0/3876593539 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de00c9ef0 con 0x55edbbc601c0 2024-05-27T07:42:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.563+0000 7f8dd37fe640 1 -- 172.21.15.114:0/3876593539 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9466+0+0 (secure 0 0 0) 0x7f8de00c9a10 con 0x55edbbc601c0 2024-05-27T07:42:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.563+0000 7f8dfd211740 1 -- 172.21.15.114:0/3876593539 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8de00c8780 con 0x55edbbc601c0 2024-05-27T07:42:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.564+0000 7f8dd37fe640 1 -- 172.21.15.114:0/3876593539 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8de0004470 con 0x55edbbc601c0 2024-05-27T07:42:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.566+0000 7f8dfd211740 10 client.25391 fetch_fsmap learned FSMap version 23 2024-05-27T07:42:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.566+0000 7f8dfd211740 1 -- 172.21.15.114:0/3876593539 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55edbc07d2c0 con 0x55edbbc601c0 2024-05-27T07:42:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.566+0000 7f8dd37fe640 1 -- 172.21.15.114:0/3876593539 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8de00ca0b0 con 0x55edbbc601c0 2024-05-27T07:42:21.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.567+0000 7f8dd37fe640 1 -- 172.21.15.114:0/3876593539 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f8de00c9270 con 0x55edbbc601c0 2024-05-27T07:42:21.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.567+0000 7f8dfd211740 10 client.25391 fetch_fsmap finished waiting for FSMap version 23 2024-05-27T07:42:21.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.568+0000 7f8dfd211740 10 client.25391 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:21.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.568+0000 7f8dfd211740 20 client.25391 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.568+0000 7f8dfd211740 1 --2- 172.21.15.114:0/3876593539 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edbc0f1f20 0x55edbc0f4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.568+0000 7f8dfd211740 4 client.25391 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:42:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.568+0000 7f8dfd211740 1 -- 172.21.15.114:0/3876593539 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f8de00c8780 con 0x55edbc0f1f20 2024-05-27T07:42:21.568 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.569+0000 7f8def7fe640 1 --2- 172.21.15.114:0/3876593539 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edbc0f1f20 0x55edbc0f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:21.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.571+0000 7f8def7fe640 1 --2- 172.21.15.114:0/3876593539 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edbc0f1f20 0x55edbc0f4340 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b5f8d9d6f1737d1 server_cookie=edac984d85d774d5 in_seq=0 out_seq=0 2024-05-27T07:42:21.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.572+0000 7f8dd37fe640 10 client.25391 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.665+0000 7f8dd37fe640 1 -- 172.21.15.114:0/3876593539 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7f8de00c8780 con 0x55edbc0f1f20 2024-05-27T07:42:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.665+0000 7f8dd37fe640 10 client.25391 handle_command_reply: tid=0 2024-05-27T07:42:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.665+0000 7f8dfd211740 1 client.25391 shutdown 2024-05-27T07:42:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.665+0000 7f8dfd211740 1 -- 172.21.15.114:0/3876593539 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbc601c0 msgr2=0x55edbc07f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.665+0000 7f8dfd211740 1 --2- 172.21.15.114:0/3876593539 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbc601c0 0x55edbc07f790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8de0004650 tx=0x7f8de0100fd0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.665+0000 7f8dfd211740 1 -- 172.21.15.114:0/3876593539 shutdown_connections 2024-05-27T07:42:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.665+0000 7f8dfd211740 1 --2- 172.21.15.114:0/3876593539 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edbc0f1f20 0x55edbc0f4340 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.665+0000 7f8dfd211740 1 --2- 172.21.15.114:0/3876593539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edbbccc1b0 0x55edbc07cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.665+0000 7f8dd37fe640 0 client.25391 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.665+0000 7f8dfd211740 1 --2- 172.21.15.114:0/3876593539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edbbcab4d0 0x55edbc07fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.665+0000 7f8dfd211740 1 --2- 172.21.15.114:0/3876593539 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edbbc601c0 0x55edbc07f790 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.665+0000 7f8dfd211740 1 -- 172.21.15.114:0/3876593539 >> 172.21.15.114:0/3876593539 conn(0x55edbbcc0300 msgr2=0x55edbc078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.665+0000 7f8dfd211740 1 -- 172.21.15.114:0/3876593539 shutdown_connections 2024-05-27T07:42:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.666+0000 7f8dfd211740 1 -- 172.21.15.114:0/3876593539 wait complete. 2024-05-27T07:42:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.666+0000 7f8dfd211740 20 client.25391 trim_cache size 0 max 16384 2024-05-27T07:42:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:42:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:42:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:42:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.667+0000 7f8df628b640 1 -- 172.21.15.114:0/4231335381 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8dd40b2b60 msgr2=0x7f8dd40b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.667+0000 7f8df628b640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8dd40b2b60 0x7f8dd40b5070 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55edbc06a540 tx=0x7f8de8010040 comp rx=0 tx=0).stop 2024-05-27T07:42:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.667+0000 7f8df628b640 1 -- 172.21.15.114:0/4231335381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df0107410 msgr2=0x7f8df01a0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.667+0000 7f8df628b640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df0107410 0x7f8df01a0ee0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8de0030f70 tx=0x7f8de0030fa0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.667+0000 7f8df628b640 1 -- 172.21.15.114:0/4231335381 shutdown_connections 2024-05-27T07:42:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.667+0000 7f8df628b640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8dd40b2b60 0x7f8dd40b5070 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.667+0000 7f8df628b640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8df0108f80 0x7f8df01a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.667+0000 7f8df628b640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df0108b70 0x7f8df01a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.667+0000 7f8df628b640 1 --2- 172.21.15.114:0/4231335381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df0107410 0x7f8df01a0ee0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.667+0000 7f8df628b640 1 -- 172.21.15.114:0/4231335381 >> 172.21.15.114:0/4231335381 conn(0x7f8df00779d0 msgr2=0x7f8df01053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.667+0000 7f8df628b640 1 -- 172.21.15.114:0/4231335381 shutdown_connections 2024-05-27T07:42:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:21.667+0000 7f8df628b640 1 -- 172.21.15.114:0/4231335381 wait complete. 2024-05-27T07:42:21.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:21 smithi114 ceph-mon[32038]: pgmap v82: 129 pgs: 129 active+clean; 119 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 11 MiB/s wr, 340 op/s 2024-05-27T07:42:21.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:21 smithi163 ceph-mon[36451]: pgmap v82: 129 pgs: 129 active+clean; 119 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 11 MiB/s wr, 340 op/s 2024-05-27T07:42:21.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:21 smithi186 ceph-mon[36647]: pgmap v82: 129 pgs: 129 active+clean; 119 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 11 MiB/s wr, 340 op/s 2024-05-27T07:42:22.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 93009 15.25 MB/sec warmup 34 sec latency 37.262 ms 2024-05-27T07:42:23.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 96206 15.35 MB/sec warmup 35 sec latency 13.297 ms 2024-05-27T07:42:23.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:23 smithi114 ceph-mon[32038]: pgmap v83: 129 pgs: 129 active+clean; 119 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 11 MiB/s wr, 340 op/s 2024-05-27T07:42:23.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:23 smithi163 ceph-mon[36451]: pgmap v83: 129 pgs: 129 active+clean; 119 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 11 MiB/s wr, 340 op/s 2024-05-27T07:42:23.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:23 smithi186 ceph-mon[36647]: pgmap v83: 129 pgs: 129 active+clean; 119 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 11 MiB/s wr, 340 op/s 2024-05-27T07:42:24.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 99178 15.33 MB/sec warmup 36 sec latency 57.768 ms 2024-05-27T07:42:25.224 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:24 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:42:25.224 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:24 smithi163 systemd[1]: Stopped Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:42:25.224 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:24 smithi163 systemd[1]: Starting Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:42:25.224 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:24 smithi163 podman[70340]: 2024-05-27 07:42:24.860088763 +0000 UTC m=+0.018963544 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:42:25.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 101137 15.18 MB/sec warmup 37 sec latency 84.186 ms 2024-05-27T07:42:25.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:25 smithi114 ceph-mon[32038]: pgmap v84: 129 pgs: 129 active+clean; 119 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 10 MiB/s wr, 319 op/s 2024-05-27T07:42:25.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:25 smithi163 ceph-mon[36451]: pgmap v84: 129 pgs: 129 active+clean; 119 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 10 MiB/s wr, 319 op/s 2024-05-27T07:42:25.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:25 smithi186 ceph-mon[36647]: pgmap v84: 129 pgs: 129 active+clean; 119 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 10 MiB/s wr, 319 op/s 2024-05-27T07:42:26.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 103690 15.16 MB/sec warmup 38 sec latency 36.553 ms 2024-05-27T07:42:26.248 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:25 smithi163 podman[70340]: 2024-05-27 07:42:25.985777351 +0000 UTC m=+1.144652130 container create 5b5c72894a08bcbddc7c182f94a942198b1164b2a6e5980ce8f4f5e8cce56766 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-27T07:42:26.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:26 smithi163 ceph-mon[36451]: daemon mds.b restarted 2024-05-27T07:42:26.724 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:26 smithi163 podman[70340]: 2024-05-27 07:42:26.35538487 +0000 UTC m=+1.514259650 container init 5b5c72894a08bcbddc7c182f94a942198b1164b2a6e5980ce8f4f5e8cce56766 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-27T07:42:26.724 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:26 smithi163 podman[70340]: 2024-05-27 07:42:26.358120591 +0000 UTC m=+1.516995378 container start 5b5c72894a08bcbddc7c182f94a942198b1164b2a6e5980ce8f4f5e8cce56766 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-27T07:42:26.724 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:26 smithi163 ceph-mds[70359]: -- 172.21.15.163:0/2421655391 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b6c3659a40 con 0x55b6c36a1a80 2024-05-27T07:42:26.724 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: starting mds.b at 2024-05-27T07:42:26.724 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:26 smithi163 bash[70340]: 5b5c72894a08bcbddc7c182f94a942198b1164b2a6e5980ce8f4f5e8cce56766 2024-05-27T07:42:26.724 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:42:26 smithi163 systemd[1]: Started Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:42:26.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:26 smithi114 ceph-mon[32038]: daemon mds.b restarted 2024-05-27T07:42:26.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:26 smithi186 ceph-mon[36647]: daemon mds.b restarted 2024-05-27T07:42:27.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 106818 15.25 MB/sec warmup 39 sec latency 18.616 ms 2024-05-27T07:42:27.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:27 smithi114 ceph-mon[32038]: pgmap v85: 129 pgs: 129 active+clean; 28 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 9.6 MiB/s wr, 565 op/s 2024-05-27T07:42:27.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:27 smithi114 ceph-mon[32038]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T07:42:27.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:27 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6838/1365473025,v1:172.21.15.163:6839/1365473025] up:boot 2024-05-27T07:42:27.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:27 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:42:27.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:27 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:42:27.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:27 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:42:27.877 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:27 smithi186 ceph-mon[36647]: pgmap v85: 129 pgs: 129 active+clean; 28 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 9.6 MiB/s wr, 565 op/s 2024-05-27T07:42:27.877 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:27 smithi186 ceph-mon[36647]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T07:42:27.877 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:27 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6838/1365473025,v1:172.21.15.163:6839/1365473025] up:boot 2024-05-27T07:42:27.877 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:27 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:42:27.878 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:27 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:42:27.878 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:27 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:42:27.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:27 smithi163 ceph-mon[36451]: pgmap v85: 129 pgs: 129 active+clean; 28 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 9.6 MiB/s wr, 565 op/s 2024-05-27T07:42:27.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:27 smithi163 ceph-mon[36451]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T07:42:27.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:27 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6838/1365473025,v1:172.21.15.163:6839/1365473025] up:boot 2024-05-27T07:42:27.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:27 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:42:27.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:27 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:42:27.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:27 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:42:28.238 INFO:tasks.workunit.client.0.smithi114.stdout: 1 109828 15.23 MB/sec warmup 40 sec latency 19.200 ms 2024-05-27T07:42:29.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 112397 15.14 MB/sec warmup 41 sec latency 14.495 ms 2024-05-27T07:42:29.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:29 smithi114 ceph-mon[32038]: pgmap v87: 129 pgs: 129 active+clean; 28 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 11 MiB/s wr, 557 op/s 2024-05-27T07:42:29.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:29 smithi163 ceph-mon[36451]: pgmap v87: 129 pgs: 129 active+clean; 28 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 11 MiB/s wr, 557 op/s 2024-05-27T07:42:29.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:29 smithi186 ceph-mon[36647]: pgmap v87: 129 pgs: 129 active+clean; 28 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 11 MiB/s wr, 557 op/s 2024-05-27T07:42:30.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 114669 15.12 MB/sec warmup 42 sec latency 25.882 ms 2024-05-27T07:42:31.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 117574 15.16 MB/sec warmup 43 sec latency 43.566 ms 2024-05-27T07:42:31.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:31 smithi114 ceph-mon[32038]: pgmap v88: 129 pgs: 129 active+clean; 35 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 443 op/s 2024-05-27T07:42:31.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:31 smithi163 ceph-mon[36451]: pgmap v88: 129 pgs: 129 active+clean; 35 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 443 op/s 2024-05-27T07:42:31.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:31 smithi186 ceph-mon[36647]: pgmap v88: 129 pgs: 129 active+clean; 35 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 443 op/s 2024-05-27T07:42:32.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 120723 15.22 MB/sec warmup 44 sec latency 19.401 ms 2024-05-27T07:42:33.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 123057 15.07 MB/sec warmup 45 sec latency 25.976 ms 2024-05-27T07:42:33.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:33 smithi114 ceph-mon[32038]: pgmap v89: 129 pgs: 129 active+clean; 35 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 443 op/s 2024-05-27T07:42:33.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:33 smithi163 ceph-mon[36451]: pgmap v89: 129 pgs: 129 active+clean; 35 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 443 op/s 2024-05-27T07:42:33.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:33 smithi186 ceph-mon[36647]: pgmap v89: 129 pgs: 129 active+clean; 35 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 443 op/s 2024-05-27T07:42:34.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 125185 15.03 MB/sec warmup 46 sec latency 14.441 ms 2024-05-27T07:42:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.770+0000 7fae5c669640 1 -- 172.21.15.114:0/3560146376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae541032e0 msgr2=0x7fae5410f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.770+0000 7fae5c669640 1 --2- 172.21.15.114:0/3560146376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae541032e0 0x7fae5410f6b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fae4c009ca0 tx=0x7fae4c02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.770+0000 7fae5c669640 1 -- 172.21.15.114:0/3560146376 shutdown_connections 2024-05-27T07:42:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.770+0000 7fae5c669640 1 --2- 172.21.15.114:0/3560146376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae541032e0 0x7fae5410f6b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.770+0000 7fae5c669640 1 --2- 172.21.15.114:0/3560146376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae54109030 0x7fae54102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.770+0000 7fae5c669640 1 --2- 172.21.15.114:0/3560146376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae54108b90 0x7fae54102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.770+0000 7fae5c669640 1 -- 172.21.15.114:0/3560146376 >> 172.21.15.114:0/3560146376 conn(0x7fae540782d0 msgr2=0x7fae54078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.770+0000 7fae5c669640 1 -- 172.21.15.114:0/3560146376 shutdown_connections 2024-05-27T07:42:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.770+0000 7fae5c669640 1 -- 172.21.15.114:0/3560146376 wait complete. 2024-05-27T07:42:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.771+0000 7fae5c669640 1 Processor -- start 2024-05-27T07:42:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.771+0000 7fae5c669640 1 -- start start 2024-05-27T07:42:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.771+0000 7fae5c669640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae541032e0 0x7fae541a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.771+0000 7fae5c669640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae54108b90 0x7fae541a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.771+0000 7fae5c669640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae54109030 0x7fae5419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.771+0000 7fae5c669640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fae54112da0 con 0x7fae541032e0 2024-05-27T07:42:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.771+0000 7fae5c669640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fae54112c20 con 0x7fae54109030 2024-05-27T07:42:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.771+0000 7fae5c669640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae54112f20 con 0x7fae54108b90 2024-05-27T07:42:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae5a405640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae541032e0 0x7fae541a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae5a405640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae541032e0 0x7fae541a05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54420/0 (socket says 172.21.15.114:54420) 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae5a405640 1 -- 172.21.15.114:0/2073026052 learned_addr learned my addr 172.21.15.114:0/2073026052 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae59c04640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae54108b90 0x7fae541a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae5ac06640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae54109030 0x7fae5419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae5a405640 1 -- 172.21.15.114:0/2073026052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae54108b90 msgr2=0x7fae541a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae5a405640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae54108b90 0x7fae541a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae5a405640 1 -- 172.21.15.114:0/2073026052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae54109030 msgr2=0x7fae5419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae5a405640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae54109030 0x7fae5419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae5a405640 1 -- 172.21.15.114:0/2073026052 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5419ae80 con 0x7fae541032e0 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae5a405640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae541032e0 0x7fae541a05d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fae4800e9f0 tx=0x7fae4800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae477fe640 1 -- 172.21.15.114:0/2073026052 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4800ccc0 con 0x7fae541032e0 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae5c669640 1 -- 172.21.15.114:0/2073026052 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae5419b110 con 0x7fae541032e0 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae477fe640 1 -- 172.21.15.114:0/2073026052 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae480054d0 con 0x7fae541032e0 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae477fe640 1 -- 172.21.15.114:0/2073026052 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae48010430 con 0x7fae541032e0 2024-05-27T07:42:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.772+0000 7fae5c669640 1 -- 172.21.15.114:0/2073026052 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae541a7370 con 0x7fae541032e0 2024-05-27T07:42:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.775+0000 7fae477fe640 1 -- 172.21.15.114:0/2073026052 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fae480046c0 con 0x7fae541032e0 2024-05-27T07:42:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.775+0000 7fae635f3740 1 Processor -- start 2024-05-27T07:42:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae635f3740 1 -- start start 2024-05-27T07:42:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae635f3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 0x55b7ed5753d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae635f3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed575910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae635f3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed575e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae635f3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b7ed4e7c30 con 0x55b7ed1b34d0 2024-05-27T07:42:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae635f3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b7ed4d77d0 con 0x55b7ed1ab2f0 2024-05-27T07:42:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae635f3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b7ed4db810 con 0x55b7ed1d41b0 2024-05-27T07:42:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae59c04640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed575e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae5ac06640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 0x55b7ed5753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae5a405640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed575910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae5ac06640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 0x55b7ed5753d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:60700/0 (socket says 172.21.15.114:60700) 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae5ac06640 1 -- 172.21.15.114:0/1677974252 learned_addr learned my addr 172.21.15.114:0/1677974252 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae477fe640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fae400b2950 0x7fae400b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae5ac06640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fae400b2950 0x7fae400b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae477fe640 1 -- 172.21.15.114:0/2073026052 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7fae480e2db0 con 0x7fae541032e0 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae5a405640 1 -- 172.21.15.114:0/1677974252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 msgr2=0x55b7ed5753d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae5a405640 1 --2- 172.21.15.114:0/1677974252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 0x55b7ed5753d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae5a405640 1 -- 172.21.15.114:0/1677974252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 msgr2=0x55b7ed575e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae5a405640 1 --2- 172.21.15.114:0/1677974252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed575e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.776+0000 7fae5a405640 1 -- 172.21.15.114:0/1677974252 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7ed1ec210 con 0x55b7ed1ab2f0 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.777+0000 7fae5a405640 1 --2- 172.21.15.114:0/1677974252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed575910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fae480e0150 tx=0x7fae4800ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88f3f8176e7c1551 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.777+0000 7fae5ac06640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fae400b2950 0x7fae400b4e60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fae4c009a60 tx=0x7fae4c02a8c0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.777+0000 7fae44bfe640 1 -- 172.21.15.114:0/1677974252 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae480a97f0 con 0x55b7ed1ab2f0 2024-05-27T07:42:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.777+0000 7fae44bfe640 1 -- 172.21.15.114:0/1677974252 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae480e6510 con 0x55b7ed1ab2f0 2024-05-27T07:42:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.777+0000 7fae44bfe640 1 -- 172.21.15.114:0/1677974252 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae480b2910 con 0x55b7ed1ab2f0 2024-05-27T07:42:34.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.778+0000 7fae635f3740 1 -- 172.21.15.114:0/1677974252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 msgr2=0x55b7ed575910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:34.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.778+0000 7fae635f3740 1 --2- 172.21.15.114:0/1677974252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed575910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fae480e0150 tx=0x7fae4800ce60 comp rx=0 tx=0).stop 2024-05-27T07:42:34.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.778+0000 7fae635f3740 1 -- 172.21.15.114:0/1677974252 shutdown_connections 2024-05-27T07:42:34.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.778+0000 7fae635f3740 1 --2- 172.21.15.114:0/1677974252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed575e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.778+0000 7fae635f3740 1 --2- 172.21.15.114:0/1677974252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed575910 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.778+0000 7fae635f3740 1 --2- 172.21.15.114:0/1677974252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 0x55b7ed5753d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.778+0000 7fae635f3740 1 -- 172.21.15.114:0/1677974252 >> 172.21.15.114:0/1677974252 conn(0x55b7ed1c8300 msgr2=0x55b7ed571b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.778+0000 7fae635f3740 1 -- 172.21.15.114:0/1677974252 shutdown_connections 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.778+0000 7fae635f3740 1 -- 172.21.15.114:0/1677974252 wait complete. 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.778+0000 7fae635f3740 1 -- start start 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae635f3740 1 Processor -- start 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae635f3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed5814c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae635f3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed581a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae635f3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 0x55b7ed57f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae635f3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b7ed1bd790 con 0x55b7ed1b34d0 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae635f3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b7ed493890 con 0x55b7ed1ab2f0 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae635f3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b7ed5824a0 con 0x55b7ed1d41b0 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae635f3740 10 client.0 _ll_register_callbacks cb 0x55b7ed1eba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae59c04640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed581a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae5a405640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed5814c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae59c04640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed581a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54432/0 (socket says 172.21.15.114:54432) 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae59c04640 1 -- 172.21.15.114:0/1062060993 learned_addr learned my addr 172.21.15.114:0/1062060993 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae5ac06640 1 --2- 172.21.15.114:0/1062060993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 0x55b7ed57f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae59c04640 1 -- 172.21.15.114:0/1062060993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 msgr2=0x55b7ed57f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae59c04640 1 --2- 172.21.15.114:0/1062060993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 0x55b7ed57f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae59c04640 1 -- 172.21.15.114:0/1062060993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 msgr2=0x55b7ed5814c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae59c04640 1 --2- 172.21.15.114:0/1062060993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed5814c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae59c04640 1 -- 172.21.15.114:0/1062060993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7ed2b8620 con 0x55b7ed1b34d0 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.779+0000 7fae5a405640 1 --2- 172.21.15.114:0/1062060993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed5814c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae59c04640 1 --2- 172.21.15.114:0/1062060993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed581a00 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fae50012700 tx=0x7fae50012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0ed531d1640107 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae3dffb640 1 -- 172.21.15.114:0/1062060993 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae50004050 con 0x55b7ed1b34d0 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae3dffb640 1 -- 172.21.15.114:0/1062060993 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7fae500041f0 con 0x55b7ed1b34d0 2024-05-27T07:42:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae635f3740 1 -- 172.21.15.114:0/1062060993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b7ed4e9b80 con 0x55b7ed1b34d0 2024-05-27T07:42:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae3dffb640 1 -- 172.21.15.114:0/1062060993 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fae50025050 con 0x55b7ed1b34d0 2024-05-27T07:42:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae635f3740 10 client.15882 fetch_fsmap learned FSMap version 25 2024-05-27T07:42:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae635f3740 1 -- 172.21.15.114:0/1062060993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b7ed58e860 con 0x55b7ed1b34d0 2024-05-27T07:42:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae3dffb640 1 -- 172.21.15.114:0/1062060993 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b7ed4e9b80 con 0x55b7ed1b34d0 2024-05-27T07:42:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae3dffb640 1 -- 172.21.15.114:0/1062060993 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fae50003a30 con 0x55b7ed1b34d0 2024-05-27T07:42:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae635f3740 10 client.15882 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:42:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae635f3740 10 client.15882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae635f3740 20 client.15882 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae635f3740 1 --2- 172.21.15.114:0/1062060993 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b7ed192d80 0x55b7ed591d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae635f3740 4 client.15882 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.780+0000 7fae635f3740 1 -- 172.21.15.114:0/1062060993 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7ed4e3360 con 0x55b7ed192d80 2024-05-27T07:42:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.781+0000 7fae5a405640 1 --2- 172.21.15.114:0/1062060993 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b7ed192d80 0x55b7ed591d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.781+0000 7fae3dffb640 10 client.15882 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.781+0000 7fae5a405640 1 --2- 172.21.15.114:0/1062060993 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b7ed192d80 0x55b7ed591d50 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a27f86de3ae34a0b server_cookie=50bb0e2f0e59172 in_seq=0 out_seq=0 2024-05-27T07:42:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae3dffb640 1 -- 172.21.15.114:0/1062060993 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b7ed4e3360 con 0x55b7ed192d80 2024-05-27T07:42:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae3dffb640 10 client.15882 handle_command_reply: tid=0 2024-05-27T07:42:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae635f3740 1 client.15882 shutdown 2024-05-27T07:42:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae635f3740 1 -- 172.21.15.114:0/1062060993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 msgr2=0x55b7ed581a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae635f3740 1 --2- 172.21.15.114:0/1062060993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed581a00 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fae50012700 tx=0x7fae50012bc0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae635f3740 1 -- 172.21.15.114:0/1062060993 shutdown_connections 2024-05-27T07:42:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae635f3740 1 --2- 172.21.15.114:0/1062060993 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b7ed192d80 0x55b7ed591d50 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae635f3740 1 --2- 172.21.15.114:0/1062060993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 0x55b7ed57f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae635f3740 1 --2- 172.21.15.114:0/1062060993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed581a00 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae3dffb640 0 client.15882 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae635f3740 1 --2- 172.21.15.114:0/1062060993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed5814c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae635f3740 1 -- 172.21.15.114:0/1062060993 >> 172.21.15.114:0/1062060993 conn(0x55b7ed1c8300 msgr2=0x55b7ed575200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:34.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae635f3740 1 -- 172.21.15.114:0/1062060993 shutdown_connections 2024-05-27T07:42:34.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae635f3740 1 -- 172.21.15.114:0/1062060993 wait complete. 2024-05-27T07:42:34.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.782+0000 7fae635f3740 20 client.15882 trim_cache size 0 max 16384 2024-05-27T07:42:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.797+0000 7fae635f3740 1 Processor -- start 2024-05-27T07:42:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.797+0000 7fae635f3740 1 -- start start 2024-05-27T07:42:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae635f3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed5796a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae635f3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1b34d0 0x55b7ed597550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae635f3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1d41b0 0x55b7ed597a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae635f3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fae541af3c0 con 0x55b7ed1d41b0 2024-05-27T07:42:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae635f3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fae541b1c40 con 0x55b7ed1b34d0 2024-05-27T07:42:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae635f3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae541afa30 con 0x55b7ed1ab2f0 2024-05-27T07:42:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae5ac06640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1d41b0 0x55b7ed597a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae5a405640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed5796a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae5ac06640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1d41b0 0x55b7ed597a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54448/0 (socket says 172.21.15.114:54448) 2024-05-27T07:42:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae5ac06640 1 -- 172.21.15.114:0/2256869183 learned_addr learned my addr 172.21.15.114:0/2256869183 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae59c04640 1 --2- 172.21.15.114:0/2256869183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1b34d0 0x55b7ed597550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae5ac06640 1 -- 172.21.15.114:0/2256869183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1ab2f0 msgr2=0x55b7ed5796a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae5ac06640 1 --2- 172.21.15.114:0/2256869183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed5796a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae5ac06640 1 -- 172.21.15.114:0/2256869183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1b34d0 msgr2=0x55b7ed597550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:34.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae5ac06640 1 --2- 172.21.15.114:0/2256869183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1b34d0 0x55b7ed597550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae5ac06640 1 -- 172.21.15.114:0/2256869183 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7ed5762f0 con 0x55b7ed1d41b0 2024-05-27T07:42:34.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae5a405640 1 --2- 172.21.15.114:0/2256869183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed5796a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:34.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae59c04640 1 --2- 172.21.15.114:0/2256869183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1b34d0 0x55b7ed597550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:34.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae5ac06640 1 --2- 172.21.15.114:0/2256869183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1d41b0 0x55b7ed597a90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fae4c005cc0 tx=0x7fae4c005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47bc40501154d74e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:34.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae3ffff640 1 -- 172.21.15.114:0/2256869183 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4c045070 con 0x55b7ed1d41b0 2024-05-27T07:42:34.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae3ffff640 1 -- 172.21.15.114:0/2256869183 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae4c036040 con 0x55b7ed1d41b0 2024-05-27T07:42:34.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.798+0000 7fae3ffff640 1 -- 172.21.15.114:0/2256869183 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4c02fb00 con 0x55b7ed1d41b0 2024-05-27T07:42:34.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.799+0000 7fae635f3740 1 -- 172.21.15.114:0/2256869183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1d41b0 msgr2=0x55b7ed597a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:34.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.799+0000 7fae635f3740 1 --2- 172.21.15.114:0/2256869183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1d41b0 0x55b7ed597a90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fae4c005cc0 tx=0x7fae4c005d80 comp rx=0 tx=0).stop 2024-05-27T07:42:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.799+0000 7fae635f3740 1 -- 172.21.15.114:0/2256869183 shutdown_connections 2024-05-27T07:42:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.799+0000 7fae635f3740 1 --2- 172.21.15.114:0/2256869183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1d41b0 0x55b7ed597a90 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.799+0000 7fae635f3740 1 --2- 172.21.15.114:0/2256869183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1b34d0 0x55b7ed597550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.799+0000 7fae635f3740 1 --2- 172.21.15.114:0/2256869183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed5796a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.799+0000 7fae635f3740 1 -- 172.21.15.114:0/2256869183 >> 172.21.15.114:0/2256869183 conn(0x55b7ed1c8300 msgr2=0x55b7ed578f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.799+0000 7fae635f3740 1 -- 172.21.15.114:0/2256869183 shutdown_connections 2024-05-27T07:42:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.799+0000 7fae635f3740 1 -- 172.21.15.114:0/2256869183 wait complete. 2024-05-27T07:42:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae635f3740 1 -- start start 2024-05-27T07:42:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae635f3740 1 Processor -- start 2024-05-27T07:42:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae635f3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed587a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae635f3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed587f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae635f3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 0x55b7ed584e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae635f3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fae541b0100 con 0x55b7ed1b34d0 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae635f3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fae541b1570 con 0x55b7ed1ab2f0 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae635f3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae541b07d0 con 0x55b7ed1d41b0 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae635f3740 10 client.0 _ll_register_callbacks cb 0x7fae50009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae59c04640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed587f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae5a405640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed587a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae59c04640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed587f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54460/0 (socket says 172.21.15.114:54460) 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae59c04640 1 -- 172.21.15.114:0/3519993012 learned_addr learned my addr 172.21.15.114:0/3519993012 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae5ac06640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 0x55b7ed584e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae59c04640 1 -- 172.21.15.114:0/3519993012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 msgr2=0x55b7ed584e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae59c04640 1 --2- 172.21.15.114:0/3519993012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 0x55b7ed584e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae59c04640 1 -- 172.21.15.114:0/3519993012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 msgr2=0x55b7ed587a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae59c04640 1 --2- 172.21.15.114:0/3519993012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed587a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae59c04640 1 -- 172.21.15.114:0/3519993012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7ed5855d0 con 0x55b7ed1b34d0 2024-05-27T07:42:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.800+0000 7fae5a405640 1 --2- 172.21.15.114:0/3519993012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed587a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae59c04640 1 --2- 172.21.15.114:0/3519993012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed587f50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x55b7ed56bf40 tx=0x7fae50010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142b4c8f7cb1e652 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae3dffb640 1 -- 172.21.15.114:0/3519993012 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5000fb30 con 0x55b7ed1b34d0 2024-05-27T07:42:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae635f3740 1 -- 172.21.15.114:0/3519993012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae4c036040 con 0x55b7ed1b34d0 2024-05-27T07:42:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae3dffb640 1 -- 172.21.15.114:0/3519993012 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7fae5000fcd0 con 0x55b7ed1b34d0 2024-05-27T07:42:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae3dffb640 1 -- 172.21.15.114:0/3519993012 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fae50003fa0 con 0x55b7ed1b34d0 2024-05-27T07:42:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae635f3740 10 client.15888 fetch_fsmap learned FSMap version 25 2024-05-27T07:42:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae635f3740 1 -- 172.21.15.114:0/3519993012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b7ed5f9f00 con 0x55b7ed1b34d0 2024-05-27T07:42:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae3dffb640 1 -- 172.21.15.114:0/3519993012 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae4c036040 con 0x55b7ed1b34d0 2024-05-27T07:42:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae3dffb640 1 -- 172.21.15.114:0/3519993012 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fae50010830 con 0x55b7ed1b34d0 2024-05-27T07:42:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae635f3740 10 client.15888 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:42:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae635f3740 10 client.15888 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae635f3740 20 client.15888 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae635f3740 1 --2- 172.21.15.114:0/3519993012 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b7ed1681c0 0x55b7ed5fc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae635f3740 4 client.15888 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:42:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.801+0000 7fae635f3740 1 -- 172.21.15.114:0/3519993012 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b7ed45d5f0 con 0x55b7ed1681c0 2024-05-27T07:42:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.802+0000 7fae5a405640 1 --2- 172.21.15.114:0/3519993012 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b7ed1681c0 0x55b7ed5fc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.802+0000 7fae3dffb640 10 client.15888 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:34.802+0000 7fae5a405640 1 --2- 172.21.15.114:0/3519993012 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b7ed1681c0 0x55b7ed5fc6c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb8bc76fdd7957a7 server_cookie=c7a9b54e8ed8f17c in_seq=0 out_seq=0 2024-05-27T07:42:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae3dffb640 1 -- 172.21.15.114:0/3519993012 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b7ed45d5f0 con 0x55b7ed1681c0 2024-05-27T07:42:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae3dffb640 10 client.15888 handle_command_reply: tid=0 2024-05-27T07:42:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae635f3740 1 client.15888 shutdown 2024-05-27T07:42:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae635f3740 1 -- 172.21.15.114:0/3519993012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 msgr2=0x55b7ed587f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae635f3740 1 --2- 172.21.15.114:0/3519993012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed587f50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x55b7ed56bf40 tx=0x7fae50010660 comp rx=0 tx=0).stop 2024-05-27T07:42:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae635f3740 1 -- 172.21.15.114:0/3519993012 shutdown_connections 2024-05-27T07:42:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae635f3740 1 --2- 172.21.15.114:0/3519993012 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b7ed1681c0 0x55b7ed5fc6c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae635f3740 1 --2- 172.21.15.114:0/3519993012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7ed1d41b0 0x55b7ed584e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae635f3740 1 --2- 172.21.15.114:0/3519993012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b7ed1b34d0 0x55b7ed587f50 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae635f3740 1 --2- 172.21.15.114:0/3519993012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7ed1ab2f0 0x55b7ed587a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae635f3740 1 -- 172.21.15.114:0/3519993012 >> 172.21.15.114:0/3519993012 conn(0x55b7ed1c8300 msgr2=0x55b7ed57f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae3dffb640 0 client.15888 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae635f3740 1 -- 172.21.15.114:0/3519993012 shutdown_connections 2024-05-27T07:42:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae635f3740 1 -- 172.21.15.114:0/3519993012 wait complete. 2024-05-27T07:42:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.098+0000 7fae635f3740 20 client.15888 trim_cache size 0 max 16384 2024-05-27T07:42:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:42:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:42:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:42:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.099+0000 7fae5c669640 1 -- 172.21.15.114:0/2073026052 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fae400b2950 msgr2=0x7fae400b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.099+0000 7fae5c669640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fae400b2950 0x7fae400b4e60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fae4c009a60 tx=0x7fae4c02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:42:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.099+0000 7fae5c669640 1 -- 172.21.15.114:0/2073026052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae541032e0 msgr2=0x7fae541a05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.099+0000 7fae5c669640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae541032e0 0x7fae541a05d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fae4800e9f0 tx=0x7fae4800eeb0 comp rx=0 tx=0).stop 2024-05-27T07:42:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.099+0000 7fae5c669640 1 -- 172.21.15.114:0/2073026052 shutdown_connections 2024-05-27T07:42:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.099+0000 7fae5c669640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fae400b2950 0x7fae400b4e60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.099+0000 7fae5c669640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae54109030 0x7fae5419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.099+0000 7fae5c669640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae54108b90 0x7fae541a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.099+0000 7fae5c669640 1 --2- 172.21.15.114:0/2073026052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae541032e0 0x7fae541a05d0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.099+0000 7fae5c669640 1 -- 172.21.15.114:0/2073026052 >> 172.21.15.114:0/2073026052 conn(0x7fae540782d0 msgr2=0x7fae5406aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.099+0000 7fae5c669640 1 -- 172.21.15.114:0/2073026052 shutdown_connections 2024-05-27T07:42:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:35.099+0000 7fae5c669640 1 -- 172.21.15.114:0/2073026052 wait complete. 2024-05-27T07:42:35.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 127838 15.06 MB/sec warmup 47 sec latency 229.704 ms 2024-05-27T07:42:35.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:35 smithi114 ceph-mon[32038]: pgmap v90: 129 pgs: 129 active+clean; 35 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 443 op/s 2024-05-27T07:42:35.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:35 smithi163 ceph-mon[36451]: pgmap v90: 129 pgs: 129 active+clean; 35 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 443 op/s 2024-05-27T07:42:35.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:35 smithi186 ceph-mon[36647]: pgmap v90: 129 pgs: 129 active+clean; 35 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 443 op/s 2024-05-27T07:42:36.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 130227 14.94 MB/sec warmup 48 sec latency 28.804 ms 2024-05-27T07:42:37.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 132279 14.89 MB/sec warmup 49 sec latency 14.496 ms 2024-05-27T07:42:37.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:37 smithi114 ceph-mon[32038]: pgmap v91: 129 pgs: 129 active+clean; 30 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 395 op/s 2024-05-27T07:42:37.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:37 smithi163 ceph-mon[36451]: pgmap v91: 129 pgs: 129 active+clean; 30 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 395 op/s 2024-05-27T07:42:37.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:37 smithi186 ceph-mon[36647]: pgmap v91: 129 pgs: 129 active+clean; 30 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 395 op/s 2024-05-27T07:42:38.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 135383 14.95 MB/sec warmup 50 sec latency 9.122 ms 2024-05-27T07:42:39.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 138516 15.01 MB/sec warmup 51 sec latency 16.851 ms 2024-05-27T07:42:39.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:39 smithi114 ceph-mon[32038]: pgmap v92: 129 pgs: 129 active+clean; 30 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 339 op/s 2024-05-27T07:42:39.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:39 smithi163 ceph-mon[36451]: pgmap v92: 129 pgs: 129 active+clean; 30 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 339 op/s 2024-05-27T07:42:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:39 smithi186 ceph-mon[36647]: pgmap v92: 129 pgs: 129 active+clean; 30 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 339 op/s 2024-05-27T07:42:40.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 141146 14.96 MB/sec warmup 52 sec latency 14.482 ms 2024-05-27T07:42:41.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 143551 14.93 MB/sec warmup 53 sec latency 15.147 ms 2024-05-27T07:42:41.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:41 smithi114 ceph-mon[32038]: pgmap v93: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.1 MiB/s wr, 426 op/s 2024-05-27T07:42:41.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:41 smithi163 ceph-mon[36451]: pgmap v93: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.1 MiB/s wr, 426 op/s 2024-05-27T07:42:41.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:41 smithi186 ceph-mon[36647]: pgmap v93: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.1 MiB/s wr, 426 op/s 2024-05-27T07:42:42.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 146250 14.92 MB/sec warmup 54 sec latency 14.641 ms 2024-05-27T07:42:42.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:42:42.917 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:43.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 149408 14.98 MB/sec warmup 55 sec latency 22.729 ms 2024-05-27T07:42:43.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:43 smithi163 ceph-mon[36451]: pgmap v94: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 339 op/s 2024-05-27T07:42:43.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:43 smithi186 ceph-mon[36647]: pgmap v94: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 339 op/s 2024-05-27T07:42:44.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:43 smithi114 ceph-mon[32038]: pgmap v94: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 339 op/s 2024-05-27T07:42:44.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 152560 15.03 MB/sec warmup 56 sec latency 18.435 ms 2024-05-27T07:42:44.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.745+0000 7eff97bf7700 1 -- 172.21.15.114:0/1999544909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff901045c0 msgr2=0x7eff901049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:44.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.745+0000 7eff97bf7700 1 --2- 172.21.15.114:0/1999544909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff901045c0 0x7eff901049d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0099c0 tx=0x7eff8c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.745+0000 7eff97bf7700 1 -- 172.21.15.114:0/1999544909 shutdown_connections 2024-05-27T07:42:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.745+0000 7eff97bf7700 1 --2- 172.21.15.114:0/1999544909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff90109840 0x7eff901111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.745+0000 7eff97bf7700 1 --2- 172.21.15.114:0/1999544909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff901045c0 0x7eff901049d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.745+0000 7eff97bf7700 1 --2- 172.21.15.114:0/1999544909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff90108ef0 0x7eff90109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.745+0000 7eff97bf7700 1 -- 172.21.15.114:0/1999544909 >> 172.21.15.114:0/1999544909 conn(0x7eff900ffe90 msgr2=0x7eff901022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.745+0000 7eff97bf7700 1 -- 172.21.15.114:0/1999544909 shutdown_connections 2024-05-27T07:42:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.745+0000 7eff97bf7700 1 -- 172.21.15.114:0/1999544909 wait complete. 2024-05-27T07:42:44.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.746+0000 7eff97bf7700 1 Processor -- start 2024-05-27T07:42:44.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.746+0000 7eff97bf7700 1 -- start start 2024-05-27T07:42:44.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.746+0000 7eff97bf7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff901045c0 0x7eff9019ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.746+0000 7eff97bf7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff90108ef0 0x7eff9019b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.746+0000 7eff97bf7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff90109840 0x7eff9019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.746+0000 7eff97bf7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7eff90114470 con 0x7eff901045c0 2024-05-27T07:42:44.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.746+0000 7eff97bf7700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7eff901142f0 con 0x7eff90108ef0 2024-05-27T07:42:44.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff97bf7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7eff901145f0 con 0x7eff90109840 2024-05-27T07:42:44.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff95993700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff901045c0 0x7eff9019ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff95993700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff901045c0 0x7eff9019ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33320/0 (socket says 172.21.15.114:33320) 2024-05-27T07:42:44.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff95993700 1 -- 172.21.15.114:0/4073185602 learned_addr learned my addr 172.21.15.114:0/4073185602 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:44.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff95192700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff90108ef0 0x7eff9019b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff96194700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff90109840 0x7eff9019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff95993700 1 -- 172.21.15.114:0/4073185602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff90109840 msgr2=0x7eff9019f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:44.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff95993700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff90109840 0x7eff9019f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff95993700 1 -- 172.21.15.114:0/4073185602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff90108ef0 msgr2=0x7eff9019b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:44.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff95993700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff90108ef0 0x7eff9019b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff95993700 1 -- 172.21.15.114:0/4073185602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9019fd30 con 0x7eff901045c0 2024-05-27T07:42:44.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff96194700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff90109840 0x7eff9019f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:44.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff95993700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff901045c0 0x7eff9019ae60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7eff8c006010 tx=0x7eff8c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:44.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff86ffd700 1 -- 172.21.15.114:0/4073185602 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c026070 con 0x7eff901045c0 2024-05-27T07:42:44.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff97bf7700 1 -- 172.21.15.114:0/4073185602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff9019ffc0 con 0x7eff901045c0 2024-05-27T07:42:44.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff86ffd700 1 -- 172.21.15.114:0/4073185602 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff8c00f460 con 0x7eff901045c0 2024-05-27T07:42:44.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff86ffd700 1 -- 172.21.15.114:0/4073185602 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c009e60 con 0x7eff901045c0 2024-05-27T07:42:44.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.747+0000 7eff97bf7700 1 -- 172.21.15.114:0/4073185602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff901a7940 con 0x7eff901045c0 2024-05-27T07:42:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.751+0000 7eff86ffd700 1 -- 172.21.15.114:0/4073185602 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7eff8c029070 con 0x7eff901045c0 2024-05-27T07:42:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.751+0000 7eff86ffd700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7eff7c0b29f0 0x7eff7c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7effa660cb80 1 Processor -- start 2024-05-27T07:42:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7eff95192700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7eff7c0b29f0 0x7eff7c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7eff86ffd700 1 -- 172.21.15.114:0/4073185602 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7eff8c0e33f0 con 0x7eff901045c0 2024-05-27T07:42:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7effa660cb80 1 -- start start 2024-05-27T07:42:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7eff95192700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7eff7c0b29f0 0x7eff7c0b4f00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7eff901074e0 tx=0x7eff8000b410 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7effa660cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1028ae40 0x564d10412b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7effa660cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d1040ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7effa660cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1040d390 0x564d1040d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7effa660cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564d10337c90 con 0x564d10413060 2024-05-27T07:42:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7effa660cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564d10327830 con 0x564d1028ae40 2024-05-27T07:42:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7effa660cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d1032b870 con 0x564d1040d390 2024-05-27T07:42:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7eff95192700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d1040ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7eff95192700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d1040ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33334/0 (socket says 172.21.15.114:33334) 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7eff95192700 1 -- 172.21.15.114:0/1702301296 learned_addr learned my addr 172.21.15.114:0/1702301296 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7eff95993700 1 --2- 172.21.15.114:0/1702301296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1028ae40 0x564d10412b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.752+0000 7eff96194700 1 --2- 172.21.15.114:0/1702301296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1040d390 0x564d1040d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.753+0000 7eff95993700 1 -- 172.21.15.114:0/1702301296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1040d390 msgr2=0x564d1040d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.753+0000 7eff95993700 1 --2- 172.21.15.114:0/1702301296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1040d390 0x564d1040d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.753+0000 7eff95993700 1 -- 172.21.15.114:0/1702301296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 msgr2=0x564d1040ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.753+0000 7eff95993700 1 --2- 172.21.15.114:0/1702301296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d1040ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.753+0000 7eff95993700 1 -- 172.21.15.114:0/1702301296 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d102778d0 con 0x564d1028ae40 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.753+0000 7eff95192700 1 --2- 172.21.15.114:0/1702301296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d1040ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.753+0000 7eff95993700 1 --2- 172.21.15.114:0/1702301296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1028ae40 0x564d10412b20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0e0790 tx=0x7eff8c0a9b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=850d7108d6bc59fa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.753+0000 7eff7bfff700 1 -- 172.21.15.114:0/1702301296 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0aa3b0 con 0x564d1028ae40 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.753+0000 7eff7bfff700 1 -- 172.21.15.114:0/1702301296 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff8c0ae760 con 0x564d1028ae40 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.753+0000 7eff7bfff700 1 -- 172.21.15.114:0/1702301296 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0aa850 con 0x564d1028ae40 2024-05-27T07:42:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.754+0000 7effa660cb80 1 -- 172.21.15.114:0/1702301296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1028ae40 msgr2=0x564d10412b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.754+0000 7effa660cb80 1 --2- 172.21.15.114:0/1702301296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1028ae40 0x564d10412b20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0e0790 tx=0x7eff8c0a9b20 comp rx=0 tx=0).stop 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.754+0000 7effa660cb80 1 -- 172.21.15.114:0/1702301296 shutdown_connections 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.754+0000 7effa660cb80 1 --2- 172.21.15.114:0/1702301296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1040d390 0x564d1040d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.754+0000 7effa660cb80 1 --2- 172.21.15.114:0/1702301296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1028ae40 0x564d10412b20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.754+0000 7effa660cb80 1 --2- 172.21.15.114:0/1702301296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d1040ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.754+0000 7effa660cb80 1 -- 172.21.15.114:0/1702301296 >> 172.21.15.114:0/1702301296 conn(0x564d102e5af0 msgr2=0x564d10403640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.754+0000 7effa660cb80 1 -- 172.21.15.114:0/1702301296 shutdown_connections 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.754+0000 7effa660cb80 1 -- 172.21.15.114:0/1702301296 wait complete. 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.755+0000 7effa660cb80 1 -- start start 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.755+0000 7effa660cb80 1 Processor -- start 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.755+0000 7effa660cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d1028ae40 0x564d104167a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.755+0000 7effa660cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1040d390 0x564d10416ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.755+0000 7effa660cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d10413060 0x564d10417220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.755+0000 7effa660cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564d1022dfa0 con 0x564d1028ae40 2024-05-27T07:42:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.755+0000 7effa660cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564d102c97c0 con 0x564d1040d390 2024-05-27T07:42:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.755+0000 7effa660cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d10419910 con 0x564d10413060 2024-05-27T07:42:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.755+0000 7effa660cb80 10 client.0 _ll_register_callbacks cb 0x564d102c2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.755+0000 7eff95993700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d1028ae40 0x564d104167a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.755+0000 7eff95993700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d1028ae40 0x564d104167a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33342/0 (socket says 172.21.15.114:33342) 2024-05-27T07:42:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.755+0000 7eff95993700 1 -- 172.21.15.114:0/4086710342 learned_addr learned my addr 172.21.15.114:0/4086710342 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.755+0000 7eff96194700 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d10413060 0x564d10417220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff95192700 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1040d390 0x564d10416ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff95993700 1 -- 172.21.15.114:0/4086710342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d10413060 msgr2=0x564d10417220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff95993700 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d10413060 0x564d10417220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff95993700 1 -- 172.21.15.114:0/4086710342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1040d390 msgr2=0x564d10416ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff95993700 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1040d390 0x564d10416ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff95993700 1 -- 172.21.15.114:0/4086710342 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d102c6690 con 0x564d1028ae40 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff95192700 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1040d390 0x564d10416ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff96194700 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d10413060 0x564d10417220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff95993700 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d1028ae40 0x564d104167a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009d70 tx=0x7eff8c0e3350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=382e9a25965a7c44 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff797fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff797fa700 1 -- 172.21.15.114:0/4086710342 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0aa3b0 con 0x564d1028ae40 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff797fa700 1 -- 172.21.15.114:0/4086710342 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7eff8c0aa820 con 0x564d1028ae40 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7effa660cb80 1 -- 172.21.15.114:0/4086710342 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff8c0ae760 con 0x564d1028ae40 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff797fa700 1 -- 172.21.15.114:0/4086710342 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7eff8c0b3180 con 0x564d1028ae40 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7effa660cb80 10 client.15900 fetch_fsmap learned FSMap version 25 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7effa660cb80 1 -- 172.21.15.114:0/4086710342 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564d10424330 con 0x564d1028ae40 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.756+0000 7eff797fa700 1 -- 172.21.15.114:0/4086710342 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff8c0ae760 con 0x564d1028ae40 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.757+0000 7eff797fa700 1 -- 172.21.15.114:0/4086710342 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7eff8c0a4ad0 con 0x564d1028ae40 2024-05-27T07:42:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.757+0000 7effa660cb80 10 client.15900 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:42:44.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.757+0000 7effa660cb80 10 client.15900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:44.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.757+0000 7effa660cb80 20 client.15900 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:44.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.757+0000 7effa660cb80 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564d10426800 0x564d10428c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.757+0000 7effa660cb80 4 client.15900 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:44.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.757+0000 7effa660cb80 1 -- 172.21.15.114:0/4086710342 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d103d49f0 con 0x564d10426800 2024-05-27T07:42:44.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.757+0000 7eff95192700 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564d10426800 0x564d10428c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.757+0000 7eff797fa700 10 client.15900 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:44.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.757+0000 7eff95192700 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564d10426800 0x564d10428c20 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=346319135a7af32c server_cookie=4a3d1804b8f4918c in_seq=0 out_seq=0 2024-05-27T07:42:44.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7eff797fa700 1 -- 172.21.15.114:0/4086710342 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564d103d49f0 con 0x564d10426800 2024-05-27T07:42:44.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7eff797fa700 10 client.15900 handle_command_reply: tid=0 2024-05-27T07:42:44.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7effa660cb80 1 client.15900 shutdown 2024-05-27T07:42:44.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7effa660cb80 1 -- 172.21.15.114:0/4086710342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d1028ae40 msgr2=0x564d104167a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:44.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7effa660cb80 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d1028ae40 0x564d104167a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009d70 tx=0x7eff8c0e3350 comp rx=0 tx=0).stop 2024-05-27T07:42:44.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7effa660cb80 1 -- 172.21.15.114:0/4086710342 shutdown_connections 2024-05-27T07:42:44.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7effa660cb80 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d10413060 0x564d10417220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7effa660cb80 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d1028ae40 0x564d104167a0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7effa660cb80 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564d10426800 0x564d10428c20 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7effa660cb80 1 --2- 172.21.15.114:0/4086710342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1040d390 0x564d10416ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7eff797fa700 0 client.15900 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:44.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7effa660cb80 1 -- 172.21.15.114:0/4086710342 >> 172.21.15.114:0/4086710342 conn(0x564d102e5af0 msgr2=0x564d1040be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:44.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7effa660cb80 1 -- 172.21.15.114:0/4086710342 shutdown_connections 2024-05-27T07:42:44.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7effa660cb80 1 -- 172.21.15.114:0/4086710342 wait complete. 2024-05-27T07:42:44.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.759+0000 7effa660cb80 20 client.15900 trim_cache size 0 max 16384 2024-05-27T07:42:44.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.781+0000 7effa660cb80 1 Processor -- start 2024-05-27T07:42:44.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.781+0000 7effa660cb80 1 -- start start 2024-05-27T07:42:44.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7effa660cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1028ae40 0x564d10410e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7effa660cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1040d390 0x564d10417060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7effa660cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d10422970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7effa660cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7eff901ac1b0 con 0x564d10413060 2024-05-27T07:42:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7effa660cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7eff901aea30 con 0x564d1028ae40 2024-05-27T07:42:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7effa660cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7eff901ac820 con 0x564d1040d390 2024-05-27T07:42:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7eff95192700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1040d390 0x564d10417060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7eff95993700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1028ae40 0x564d10410e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7eff95192700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1040d390 0x564d10417060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40790/0 (socket says 172.21.15.114:40790) 2024-05-27T07:42:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7eff95192700 1 -- 172.21.15.114:0/2470984157 learned_addr learned my addr 172.21.15.114:0/2470984157 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7eff96194700 1 --2- 172.21.15.114:0/2470984157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d10422970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7eff95993700 1 -- 172.21.15.114:0/2470984157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1040d390 msgr2=0x564d10417060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7eff95993700 1 --2- 172.21.15.114:0/2470984157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1040d390 0x564d10417060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7eff95993700 1 -- 172.21.15.114:0/2470984157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 msgr2=0x564d10422970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7eff95993700 1 --2- 172.21.15.114:0/2470984157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d10422970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7eff95993700 1 -- 172.21.15.114:0/2470984157 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d1040de00 con 0x564d1028ae40 2024-05-27T07:42:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.782+0000 7eff95192700 1 --2- 172.21.15.114:0/2470984157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1040d390 0x564d10417060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.783+0000 7eff95993700 1 --2- 172.21.15.114:0/2470984157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1028ae40 0x564d10410e50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009350 tx=0x7eff8c00fc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb1a487608b127d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.783+0000 7eff7b7fe700 1 -- 172.21.15.114:0/2470984157 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0ae760 con 0x564d1028ae40 2024-05-27T07:42:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.783+0000 7eff7b7fe700 1 -- 172.21.15.114:0/2470984157 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff8c0aa3b0 con 0x564d1028ae40 2024-05-27T07:42:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.783+0000 7eff7b7fe700 1 -- 172.21.15.114:0/2470984157 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0a4ad0 con 0x564d1028ae40 2024-05-27T07:42:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.784+0000 7effa660cb80 1 -- 172.21.15.114:0/2470984157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1028ae40 msgr2=0x564d10410e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.784+0000 7effa660cb80 1 --2- 172.21.15.114:0/2470984157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1028ae40 0x564d10410e50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009350 tx=0x7eff8c00fc60 comp rx=0 tx=0).stop 2024-05-27T07:42:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.784+0000 7effa660cb80 1 -- 172.21.15.114:0/2470984157 shutdown_connections 2024-05-27T07:42:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.784+0000 7effa660cb80 1 --2- 172.21.15.114:0/2470984157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d10422970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.784+0000 7effa660cb80 1 --2- 172.21.15.114:0/2470984157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1028ae40 0x564d10410e50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.784+0000 7effa660cb80 1 --2- 172.21.15.114:0/2470984157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1040d390 0x564d10417060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.784+0000 7effa660cb80 1 -- 172.21.15.114:0/2470984157 >> 172.21.15.114:0/2470984157 conn(0x564d102e5af0 msgr2=0x564d10411ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.784+0000 7effa660cb80 1 -- 172.21.15.114:0/2470984157 shutdown_connections 2024-05-27T07:42:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.784+0000 7effa660cb80 1 -- 172.21.15.114:0/2470984157 wait complete. 2024-05-27T07:42:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.784+0000 7effa660cb80 1 -- start start 2024-05-27T07:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.784+0000 7effa660cb80 1 Processor -- start 2024-05-27T07:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7effa660cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1028ae40 0x564d1042df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7effa660cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1040d390 0x564d1042e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7effa660cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d1042e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7effa660cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7eff901acef0 con 0x564d10413060 2024-05-27T07:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7effa660cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7eff901ae360 con 0x564d1040d390 2024-05-27T07:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7effa660cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7eff901ad5c0 con 0x564d1028ae40 2024-05-27T07:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7effa660cb80 10 client.0 _ll_register_callbacks cb 0x564d102c2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7eff96194700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d1042e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7eff96194700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d1042e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33360/0 (socket says 172.21.15.114:33360) 2024-05-27T07:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7eff96194700 1 -- 172.21.15.114:0/1668914828 learned_addr learned my addr 172.21.15.114:0/1668914828 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7eff95993700 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1028ae40 0x564d1042df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7eff95192700 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1040d390 0x564d1042e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7eff96194700 1 -- 172.21.15.114:0/1668914828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1028ae40 msgr2=0x564d1042df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:44.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7eff96194700 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1028ae40 0x564d1042df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7eff96194700 1 -- 172.21.15.114:0/1668914828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1040d390 msgr2=0x564d1042e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:44.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7eff96194700 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1040d390 0x564d1042e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7eff96194700 1 -- 172.21.15.114:0/1668914828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d10490250 con 0x564d10413060 2024-05-27T07:42:44.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.785+0000 7eff95192700 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1040d390 0x564d1042e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:44.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.786+0000 7eff96194700 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d1042e9b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7eff88018700 tx=0x7eff88018730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=676120f995f4e6e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:44.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.786+0000 7eff797fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:44.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.786+0000 7eff797fa700 1 -- 172.21.15.114:0/1668914828 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff88003c00 con 0x564d10413060 2024-05-27T07:42:44.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.786+0000 7effa660cb80 1 -- 172.21.15.114:0/1668914828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff8c0aa3b0 con 0x564d10413060 2024-05-27T07:42:44.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.786+0000 7eff797fa700 1 -- 172.21.15.114:0/1668914828 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7eff88003da0 con 0x564d10413060 2024-05-27T07:42:44.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.786+0000 7eff797fa700 1 -- 172.21.15.114:0/1668914828 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7eff88029050 con 0x564d10413060 2024-05-27T07:42:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.786+0000 7effa660cb80 10 client.15906 fetch_fsmap learned FSMap version 25 2024-05-27T07:42:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.786+0000 7effa660cb80 1 -- 172.21.15.114:0/1668914828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564d10491570 con 0x564d10413060 2024-05-27T07:42:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.787+0000 7eff797fa700 1 -- 172.21.15.114:0/1668914828 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff8c0aa3b0 con 0x564d10413060 2024-05-27T07:42:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.787+0000 7eff797fa700 1 -- 172.21.15.114:0/1668914828 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7eff880043a0 con 0x564d10413060 2024-05-27T07:42:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.787+0000 7effa660cb80 10 client.15906 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:42:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.787+0000 7effa660cb80 10 client.15906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.787+0000 7effa660cb80 20 client.15906 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.787+0000 7effa660cb80 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564d10491c60 0x564d10494080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.787+0000 7effa660cb80 4 client.15906 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:42:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.787+0000 7effa660cb80 1 -- 172.21.15.114:0/1668914828 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564d10339be0 con 0x564d10491c60 2024-05-27T07:42:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.787+0000 7eff95993700 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564d10491c60 0x564d10494080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.787+0000 7eff797fa700 10 client.15906 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.787+0000 7eff95993700 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564d10491c60 0x564d10494080 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a21fdc232aea0a7 server_cookie=eab951f8ae88a4e in_seq=0 out_seq=0 2024-05-27T07:42:44.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7eff797fa700 1 -- 172.21.15.114:0/1668914828 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564d10339be0 con 0x564d10491c60 2024-05-27T07:42:44.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7eff797fa700 10 client.15906 handle_command_reply: tid=0 2024-05-27T07:42:44.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7effa660cb80 1 client.15906 shutdown 2024-05-27T07:42:44.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7effa660cb80 1 -- 172.21.15.114:0/1668914828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 msgr2=0x564d1042e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:44.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7effa660cb80 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d1042e9b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7eff88018700 tx=0x7eff88018730 comp rx=0 tx=0).stop 2024-05-27T07:42:44.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7effa660cb80 1 -- 172.21.15.114:0/1668914828 shutdown_connections 2024-05-27T07:42:44.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7effa660cb80 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d10413060 0x564d1042e9b0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7effa660cb80 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d1028ae40 0x564d1042df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7effa660cb80 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564d10491c60 0x564d10494080 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7effa660cb80 1 --2- 172.21.15.114:0/1668914828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564d1040d390 0x564d1042e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7eff797fa700 0 client.15906 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:44.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7effa660cb80 1 -- 172.21.15.114:0/1668914828 >> 172.21.15.114:0/1668914828 conn(0x564d102e5af0 msgr2=0x564d10414c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:44.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7effa660cb80 1 -- 172.21.15.114:0/1668914828 shutdown_connections 2024-05-27T07:42:44.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7effa660cb80 1 -- 172.21.15.114:0/1668914828 wait complete. 2024-05-27T07:42:44.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.790+0000 7effa660cb80 20 client.15906 trim_cache size 0 max 16384 2024-05-27T07:42:44.795 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:42:44.795 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:42:44.795 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:42:44.795 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:42:44.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.791+0000 7eff97bf7700 1 -- 172.21.15.114:0/4073185602 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7eff7c0b29f0 msgr2=0x7eff7c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:44.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.791+0000 7eff97bf7700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7eff7c0b29f0 0x7eff7c0b4f00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7eff901074e0 tx=0x7eff8000b410 comp rx=0 tx=0).stop 2024-05-27T07:42:44.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.791+0000 7eff97bf7700 1 -- 172.21.15.114:0/4073185602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff901045c0 msgr2=0x7eff9019ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:44.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.791+0000 7eff97bf7700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff901045c0 0x7eff9019ae60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7eff8c006010 tx=0x7eff8c0045e0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.791+0000 7eff97bf7700 1 -- 172.21.15.114:0/4073185602 shutdown_connections 2024-05-27T07:42:44.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.791+0000 7eff97bf7700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff90109840 0x7eff9019f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.791+0000 7eff97bf7700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff901045c0 0x7eff9019ae60 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.791+0000 7eff97bf7700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7eff7c0b29f0 0x7eff7c0b4f00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.791+0000 7eff97bf7700 1 --2- 172.21.15.114:0/4073185602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff90108ef0 0x7eff9019b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.791+0000 7eff97bf7700 1 -- 172.21.15.114:0/4073185602 >> 172.21.15.114:0/4073185602 conn(0x7eff900ffe90 msgr2=0x7eff90102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:44.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.791+0000 7eff97bf7700 1 -- 172.21.15.114:0/4073185602 shutdown_connections 2024-05-27T07:42:44.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:44.791+0000 7eff97bf7700 1 -- 172.21.15.114:0/4073185602 wait complete. 2024-05-27T07:42:45.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 154888 14.91 MB/sec warmup 57 sec latency 14.619 ms 2024-05-27T07:42:45.652 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:42:45.652 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:42:45.904 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:45.946 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:45 smithi114 ceph-mon[32038]: pgmap v95: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 339 op/s 2024-05-27T07:42:45.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:45 smithi163 ceph-mon[36451]: pgmap v95: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 339 op/s 2024-05-27T07:42:45.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:45 smithi186 ceph-mon[36647]: pgmap v95: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 339 op/s 2024-05-27T07:42:46.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 157200 14.90 MB/sec warmup 58 sec latency 15.267 ms 2024-05-27T07:42:47.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 160229 14.94 MB/sec warmup 59 sec latency 20.604 ms 2024-05-27T07:42:47.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.717+0000 7facd1497700 1 -- 172.21.15.114:0/1346913224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faccc107e00 msgr2=0x7faccc108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:47.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.717+0000 7facd1497700 1 --2- 172.21.15.114:0/1346913224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faccc107e00 0x7faccc108210 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7facc00099c0 tx=0x7facc0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:42:47.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.717+0000 7facd1497700 1 -- 172.21.15.114:0/1346913224 shutdown_connections 2024-05-27T07:42:47.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.717+0000 7facd1497700 1 --2- 172.21.15.114:0/1346913224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faccc108750 0x7faccc1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:47.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.717+0000 7facd1497700 1 --2- 172.21.15.114:0/1346913224 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faccc1034d0 0x7faccc1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:47.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.717+0000 7facd1497700 1 --2- 172.21.15.114:0/1346913224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faccc107e00 0x7faccc108210 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:47.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.717+0000 7facd1497700 1 -- 172.21.15.114:0/1346913224 >> 172.21.15.114:0/1346913224 conn(0x7faccc0fef20 msgr2=0x7faccc101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:47.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.717+0000 7facd1497700 1 -- 172.21.15.114:0/1346913224 shutdown_connections 2024-05-27T07:42:47.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.717+0000 7facd1497700 1 -- 172.21.15.114:0/1346913224 wait complete. 2024-05-27T07:42:47.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.718+0000 7facd1497700 1 Processor -- start 2024-05-27T07:42:47.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.718+0000 7facd1497700 1 -- start start 2024-05-27T07:42:47.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facd1497700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faccc1034d0 0x7faccc19bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:47.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facd1497700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faccc107e00 0x7faccc19c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:47.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facd1497700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faccc108750 0x7faccc1a6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:47.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facd1497700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faccc113360 con 0x7faccc107e00 2024-05-27T07:42:47.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facd1497700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faccc1131e0 con 0x7faccc108750 2024-05-27T07:42:47.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facd1497700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faccc1134e0 con 0x7faccc1034d0 2024-05-27T07:42:47.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facca7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faccc107e00 0x7faccc19c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7faccaffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faccc1034d0 0x7faccc19bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facca7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faccc107e00 0x7faccc19c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33380/0 (socket says 172.21.15.114:33380) 2024-05-27T07:42:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facca7fc700 1 -- 172.21.15.114:0/487156725 learned_addr learned my addr 172.21.15.114:0/487156725 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7faccb7fe700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faccc108750 0x7faccc1a6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facca7fc700 1 -- 172.21.15.114:0/487156725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faccc1034d0 msgr2=0x7faccc19bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facca7fc700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faccc1034d0 0x7faccc19bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facca7fc700 1 -- 172.21.15.114:0/487156725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faccc108750 msgr2=0x7faccc1a6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facca7fc700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faccc108750 0x7faccc1a6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facca7fc700 1 -- 172.21.15.114:0/487156725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faccc19cad0 con 0x7faccc107e00 2024-05-27T07:42:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7faccaffd700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faccc1034d0 0x7faccc19bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.719+0000 7facca7fc700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faccc107e00 0x7faccc19c220 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7facc0005f50 tx=0x7facc0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.720+0000 7facb3fff700 1 -- 172.21.15.114:0/487156725 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc0026070 con 0x7faccc107e00 2024-05-27T07:42:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.720+0000 7facd1497700 1 -- 172.21.15.114:0/487156725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faccc1a7360 con 0x7faccc107e00 2024-05-27T07:42:47.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.720+0000 7facb3fff700 1 -- 172.21.15.114:0/487156725 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facc0003c70 con 0x7faccc107e00 2024-05-27T07:42:47.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.720+0000 7facb3fff700 1 -- 172.21.15.114:0/487156725 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc000f9c0 con 0x7faccc107e00 2024-05-27T07:42:47.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.720+0000 7facd1497700 1 -- 172.21.15.114:0/487156725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faccc1a77b0 con 0x7faccc107e00 2024-05-27T07:42:47.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.721+0000 7facd1497700 1 -- 172.21.15.114:0/487156725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faccc0512b0 con 0x7faccc107e00 2024-05-27T07:42:47.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.722+0000 7facb3fff700 1 -- 172.21.15.114:0/487156725 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7facc0029070 con 0x7faccc107e00 2024-05-27T07:42:47.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.724+0000 7facb3fff700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7facb40b29f0 0x7facb40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:47.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.724+0000 7faccaffd700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7facb40b29f0 0x7facb40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:47.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.724+0000 7facb3fff700 1 -- 172.21.15.114:0/487156725 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7facc00e3600 con 0x7faccc107e00 2024-05-27T07:42:47.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.724+0000 7faccaffd700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7facb40b29f0 0x7facb40b4f00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7facbc011490 tx=0x7facbc009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:47.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:47.727+0000 7facb3fff700 1 -- 172.21.15.114:0/487156725 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7facc0030080 con 0x7faccc107e00 2024-05-27T07:42:47.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:47 smithi163 ceph-mon[36451]: pgmap v96: 129 pgs: 129 active+clean; 43 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.1 MiB/s wr, 426 op/s 2024-05-27T07:42:47.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:47 smithi186 ceph-mon[36647]: pgmap v96: 129 pgs: 129 active+clean; 43 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.1 MiB/s wr, 426 op/s 2024-05-27T07:42:48.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.014+0000 7facd1497700 1 -- 172.21.15.114:0/487156725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faccc02e5e0 con 0x7faccc107e00 2024-05-27T07:42:48.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:47 smithi114 ceph-mon[32038]: pgmap v96: 129 pgs: 129 active+clean; 43 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.1 MiB/s wr, 426 op/s 2024-05-27T07:42:48.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.015+0000 7facb3fff700 1 -- 172.21.15.114:0/487156725 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11827 (secure 0 0 0) 0x7facc0013070 con 0x7faccc107e00 2024-05-27T07:42:48.014 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:42:48.014 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:42:26.549486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25397":{"gid":25397,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/1365473025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":1365473025},{"type":"v1","addr":"172.21.15.163:6839","nonce":1365473025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:42:48.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.017+0000 7facd1497700 1 -- 172.21.15.114:0/487156725 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7facb40b29f0 msgr2=0x7facb40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:48.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.017+0000 7facd1497700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7facb40b29f0 0x7facb40b4f00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7facbc011490 tx=0x7facbc009260 comp rx=0 tx=0).stop 2024-05-27T07:42:48.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.017+0000 7facd1497700 1 -- 172.21.15.114:0/487156725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faccc107e00 msgr2=0x7faccc19c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:48.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.017+0000 7facd1497700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faccc107e00 0x7faccc19c220 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7facc0005f50 tx=0x7facc0004270 comp rx=0 tx=0).stop 2024-05-27T07:42:48.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.018+0000 7facd1497700 1 -- 172.21.15.114:0/487156725 shutdown_connections 2024-05-27T07:42:48.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.018+0000 7facd1497700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7facb40b29f0 0x7facb40b4f00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.018+0000 7facd1497700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faccc108750 0x7faccc1a6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.018+0000 7facd1497700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faccc1034d0 0x7faccc19bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.018+0000 7facd1497700 1 --2- 172.21.15.114:0/487156725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faccc107e00 0x7faccc19c220 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.018+0000 7facd1497700 1 -- 172.21.15.114:0/487156725 >> 172.21.15.114:0/487156725 conn(0x7faccc0fef20 msgr2=0x7faccc107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:48.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.018+0000 7facd1497700 1 -- 172.21.15.114:0/487156725 shutdown_connections 2024-05-27T07:42:48.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:48.018+0000 7facd1497700 1 -- 172.21.15.114:0/487156725 wait complete. 2024-05-27T07:42:48.018 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 25 2024-05-27T07:42:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.204+0000 7ff310205640 1 -- 172.21.15.114:0/1629253991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3081032e0 msgr2=0x7ff30810f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.204+0000 7ff310205640 1 --2- 172.21.15.114:0/1629253991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3081032e0 0x7ff30810f6b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff300009ca0 tx=0x7ff30002a8c0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.204+0000 7ff310205640 1 -- 172.21.15.114:0/1629253991 shutdown_connections 2024-05-27T07:42:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.204+0000 7ff310205640 1 --2- 172.21.15.114:0/1629253991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3081032e0 0x7ff30810f6b0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.204+0000 7ff310205640 1 --2- 172.21.15.114:0/1629253991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff308109030 0x7ff308102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.204+0000 7ff310205640 1 --2- 172.21.15.114:0/1629253991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff308108b90 0x7ff308102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.204+0000 7ff310205640 1 -- 172.21.15.114:0/1629253991 >> 172.21.15.114:0/1629253991 conn(0x7ff3080782d0 msgr2=0x7ff308078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.204+0000 7ff310205640 1 -- 172.21.15.114:0/1629253991 shutdown_connections 2024-05-27T07:42:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.204+0000 7ff310205640 1 -- 172.21.15.114:0/1629253991 wait complete. 2024-05-27T07:42:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff310205640 1 Processor -- start 2024-05-27T07:42:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff310205640 1 -- start start 2024-05-27T07:42:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff310205640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3081032e0 0x7ff30819f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff310205640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff308108b90 0x7ff30819fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff310205640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff308109030 0x7ff308199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff310205640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff308112da0 con 0x7ff308109030 2024-05-27T07:42:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff310205640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff308112c20 con 0x7ff3081032e0 2024-05-27T07:42:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff310205640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff308112f20 con 0x7ff308108b90 2024-05-27T07:42:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff30e7a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff308109030 0x7ff308199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff30e7a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff308109030 0x7ff308199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33402/0 (socket says 172.21.15.114:33402) 2024-05-27T07:42:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff30e7a2640 1 -- 172.21.15.114:0/1063949785 learned_addr learned my addr 172.21.15.114:0/1063949785 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff308108b90 0x7ff30819fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff30dfa1640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3081032e0 0x7ff30819f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff30e7a2640 1 -- 172.21.15.114:0/1063949785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff308108b90 msgr2=0x7ff30819fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff30e7a2640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff308108b90 0x7ff30819fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff30e7a2640 1 -- 172.21.15.114:0/1063949785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3081032e0 msgr2=0x7ff30819f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff30e7a2640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3081032e0 0x7ff30819f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff30e7a2640 1 -- 172.21.15.114:0/1063949785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff308199d90 con 0x7ff308109030 2024-05-27T07:42:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.205+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff308108b90 0x7ff30819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.206+0000 7ff30e7a2640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff308109030 0x7ff308199660 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff3000098d0 tx=0x7ff3000317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.206+0000 7ff2feffd640 1 -- 172.21.15.114:0/1063949785 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff300003b70 con 0x7ff308109030 2024-05-27T07:42:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.206+0000 7ff2feffd640 1 -- 172.21.15.114:0/1063949785 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff300003d10 con 0x7ff308109030 2024-05-27T07:42:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.206+0000 7ff310205640 1 -- 172.21.15.114:0/1063949785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff308199fc0 con 0x7ff308109030 2024-05-27T07:42:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.206+0000 7ff2feffd640 1 -- 172.21.15.114:0/1063949785 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30002fa20 con 0x7ff308109030 2024-05-27T07:42:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.206+0000 7ff310205640 1 -- 172.21.15.114:0/1063949785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3081a6280 con 0x7ff308109030 2024-05-27T07:42:48.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff2feffd640 1 -- 172.21.15.114:0/1063949785 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ff30002fc40 con 0x7ff308109030 2024-05-27T07:42:48.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff317193740 1 Processor -- start 2024-05-27T07:42:48.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff317193740 1 -- start start 2024-05-27T07:42:48.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff317193740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499df71b0 0x55c49a19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff317193740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 0x55c49a19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff317193740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499dd64d0 0x55c49a19b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff317193740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c49a10ac30 con 0x55c499dce2f0 2024-05-27T07:42:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff317193740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c49a0fa7d0 con 0x55c499df71b0 2024-05-27T07:42:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff317193740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c49a0fe810 con 0x55c499dd64d0 2024-05-27T07:42:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff30dfa1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 0x55c49a19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff30d7a0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499dd64d0 0x55c49a19b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff30dfa1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 0x55c49a19ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33412/0 (socket says 172.21.15.114:33412) 2024-05-27T07:42:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff30dfa1640 1 -- 172.21.15.114:0/3954535261 learned_addr learned my addr 172.21.15.114:0/3954535261 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/3954535261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499df71b0 0x55c49a19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.209+0000 7ff2feffd640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2f40b2950 0x7ff2f40b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.210+0000 7ff30d7a0640 1 -- 172.21.15.114:0/3954535261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499df71b0 msgr2=0x55c49a19a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.210+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/3954535261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499df71b0 0x55c49a19a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.210+0000 7ff30d7a0640 1 -- 172.21.15.114:0/3954535261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 msgr2=0x55c49a19ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.210+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/3954535261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 0x55c49a19ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.210+0000 7ff30d7a0640 1 -- 172.21.15.114:0/3954535261 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c499e0f210 con 0x55c499dd64d0 2024-05-27T07:42:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.210+0000 7ff30dfa1640 1 --2- 172.21.15.114:0/3954535261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 0x55c49a19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.210+0000 7ff30e7a2640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2f40b2950 0x7ff2f40b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.210+0000 7ff2feffd640 1 -- 172.21.15.114:0/1063949785 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7ff300103d60 con 0x7ff308109030 2024-05-27T07:42:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.210+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/3954535261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499dd64d0 0x55c49a19b060 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009710 tx=0x7ff2f80096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19d1127472e14f4f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.210+0000 7ff2f3fff640 1 -- 172.21.15.114:0/3954535261 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8004030 con 0x55c499dd64d0 2024-05-27T07:42:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.210+0000 7ff30e7a2640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2f40b2950 0x7ff2f40b4e60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff3001012c0 tx=0x7ff30002a8c0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.210+0000 7ff2f3fff640 1 -- 172.21.15.114:0/3954535261 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2f800f910 con 0x55c499dd64d0 2024-05-27T07:42:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.210+0000 7ff2f3fff640 1 -- 172.21.15.114:0/3954535261 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8003a10 con 0x55c499dd64d0 2024-05-27T07:42:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.211+0000 7ff317193740 1 -- 172.21.15.114:0/3954535261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499dd64d0 msgr2=0x55c49a19b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:48.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.211+0000 7ff317193740 1 --2- 172.21.15.114:0/3954535261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499dd64d0 0x55c49a19b060 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009710 tx=0x7ff2f80096a0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.211+0000 7ff317193740 1 -- 172.21.15.114:0/3954535261 shutdown_connections 2024-05-27T07:42:48.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.211+0000 7ff317193740 1 --2- 172.21.15.114:0/3954535261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499dd64d0 0x55c49a19b060 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.211+0000 7ff317193740 1 --2- 172.21.15.114:0/3954535261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 0x55c49a19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.211+0000 7ff317193740 1 --2- 172.21.15.114:0/3954535261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499df71b0 0x55c49a19a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.211+0000 7ff317193740 1 -- 172.21.15.114:0/3954535261 >> 172.21.15.114:0/3954535261 conn(0x55c499deb300 msgr2=0x55c49a196d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:48.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.211+0000 7ff317193740 1 -- 172.21.15.114:0/3954535261 shutdown_connections 2024-05-27T07:42:48.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.211+0000 7ff317193740 1 -- 172.21.15.114:0/3954535261 wait complete. 2024-05-27T07:42:48.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff317193740 1 -- start start 2024-05-27T07:42:48.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff317193740 1 Processor -- start 2024-05-27T07:42:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff317193740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 0x55c49a1a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff317193740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499dd64d0 0x55c49a1a6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff317193740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499df71b0 0x55c49a1a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff317193740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c499de0790 con 0x55c499dce2f0 2024-05-27T07:42:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff317193740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c49a0b6890 con 0x55c499df71b0 2024-05-27T07:42:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff317193740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c49a1a7710 con 0x55c499dd64d0 2024-05-27T07:42:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff317193740 10 client.0 _ll_register_callbacks cb 0x55c499e0ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff30d7a0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 0x55c49a1a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff30d7a0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 0x55c49a1a65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33428/0 (socket says 172.21.15.114:33428) 2024-05-27T07:42:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff30d7a0640 1 -- 172.21.15.114:0/2093114171 learned_addr learned my addr 172.21.15.114:0/2093114171 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff30dfa1640 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499dd64d0 0x55c49a1a6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.212+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499df71b0 0x55c49a1a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff30dfa1640 1 -- 172.21.15.114:0/2093114171 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499df71b0 msgr2=0x55c49a1a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff30dfa1640 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499df71b0 0x55c49a1a4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff30dfa1640 1 -- 172.21.15.114:0/2093114171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 msgr2=0x55c49a1a65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff30dfa1640 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 0x55c49a1a65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff30dfa1640 1 -- 172.21.15.114:0/2093114171 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c499edb620 con 0x55c499dd64d0 2024-05-27T07:42:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 0x55c49a1a65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff30dfa1640 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499dd64d0 0x55c49a1a6b30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x55c49a197d20 tx=0x7ff304011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b4249080c89fafd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff2f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:42:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff2f17fa640 1 -- 172.21.15.114:0/2093114171 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304004020 con 0x55c499dd64d0 2024-05-27T07:42:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff317193740 1 -- 172.21.15.114:0/2093114171 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2f800f910 con 0x55c499dd64d0 2024-05-27T07:42:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff2f17fa640 1 -- 172.21.15.114:0/2093114171 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7ff3040041c0 con 0x55c499dd64d0 2024-05-27T07:42:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff2f17fa640 1 -- 172.21.15.114:0/2093114171 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff304025050 con 0x55c499dd64d0 2024-05-27T07:42:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff317193740 10 client.25519 fetch_fsmap learned FSMap version 25 2024-05-27T07:42:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff317193740 1 -- 172.21.15.114:0/2093114171 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c49a1b39f0 con 0x55c499dd64d0 2024-05-27T07:42:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.213+0000 7ff2f17fa640 1 -- 172.21.15.114:0/2093114171 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2f800f910 con 0x55c499dd64d0 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.214+0000 7ff2f17fa640 1 -- 172.21.15.114:0/2093114171 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7ff304003790 con 0x55c499dd64d0 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.214+0000 7ff317193740 10 client.25519 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.214+0000 7ff317193740 10 client.25519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.214+0000 7ff317193740 20 client.25519 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.214+0000 7ff317193740 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c499db5d80 0x55c49a1b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.214+0000 7ff317193740 4 client.25519 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.214+0000 7ff317193740 1 -- 172.21.15.114:0/2093114171 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c49a10cb80 con 0x55c499db5d80 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.214+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c499db5d80 0x55c49a1b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.214+0000 7ff2f17fa640 10 client.25519 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.214+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c499db5d80 0x55c49a1b6f10 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c0d053b8572bd30 server_cookie=9c98a511e09a5960 in_seq=0 out_seq=0 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff2f17fa640 1 -- 172.21.15.114:0/2093114171 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c49a10cb80 con 0x55c499db5d80 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff2f17fa640 10 client.25519 handle_command_reply: tid=0 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff317193740 1 client.25519 shutdown 2024-05-27T07:42:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff317193740 1 -- 172.21.15.114:0/2093114171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499dd64d0 msgr2=0x55c49a1a6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff317193740 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499dd64d0 0x55c49a1a6b30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x55c49a197d20 tx=0x7ff304011810 comp rx=0 tx=0).stop 2024-05-27T07:42:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff317193740 1 -- 172.21.15.114:0/2093114171 shutdown_connections 2024-05-27T07:42:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff317193740 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c499db5d80 0x55c49a1b6f10 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff317193740 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499df71b0 0x55c49a1a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff317193740 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499dd64d0 0x55c49a1a6b30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff317193740 1 --2- 172.21.15.114:0/2093114171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dce2f0 0x55c49a1a65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff2f17fa640 0 client.25519 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff317193740 1 -- 172.21.15.114:0/2093114171 >> 172.21.15.114:0/2093114171 conn(0x55c499deb300 msgr2=0x55c49a19c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff317193740 1 -- 172.21.15.114:0/2093114171 shutdown_connections 2024-05-27T07:42:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff317193740 1 -- 172.21.15.114:0/2093114171 wait complete. 2024-05-27T07:42:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.215+0000 7ff317193740 20 client.25519 trim_cache size 0 max 16384 2024-05-27T07:42:48.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.231+0000 7ff317193740 1 Processor -- start 2024-05-27T07:42:48.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.231+0000 7ff317193740 1 -- start start 2024-05-27T07:42:48.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.231+0000 7ff317193740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499dce2f0 0x55c49a19e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.231+0000 7ff317193740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 0x55c49a19e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff317193740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499df71b0 0x55c49a1bc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff317193740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff3081ad150 con 0x55c499dd64d0 2024-05-27T07:42:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff317193740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff3081af9d0 con 0x55c499dce2f0 2024-05-27T07:42:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff317193740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff3081ad7c0 con 0x55c499df71b0 2024-05-27T07:42:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff30dfa1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 0x55c49a19e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff30dfa1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 0x55c49a19e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33438/0 (socket says 172.21.15.114:33438) 2024-05-27T07:42:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff30dfa1640 1 -- 172.21.15.114:0/2797374278 learned_addr learned my addr 172.21.15.114:0/2797374278 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/2797374278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499dce2f0 0x55c49a19e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/2797374278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499df71b0 0x55c49a1bc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff30dfa1640 1 -- 172.21.15.114:0/2797374278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499df71b0 msgr2=0x55c49a1bc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff30dfa1640 1 --2- 172.21.15.114:0/2797374278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499df71b0 0x55c49a1bc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff30dfa1640 1 -- 172.21.15.114:0/2797374278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499dce2f0 msgr2=0x55c49a19e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff30dfa1640 1 --2- 172.21.15.114:0/2797374278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499dce2f0 0x55c49a19e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff30dfa1640 1 -- 172.21.15.114:0/2797374278 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c49a1a3030 con 0x55c499dd64d0 2024-05-27T07:42:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff30dfa1640 1 --2- 172.21.15.114:0/2797374278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 0x55c49a19e770 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff304003bb0 tx=0x7ff3040058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35381af6180466ff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff2f37fe640 1 -- 172.21.15.114:0/2797374278 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30402d030 con 0x55c499dd64d0 2024-05-27T07:42:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff2f37fe640 1 -- 172.21.15.114:0/2797374278 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff304004000 con 0x55c499dd64d0 2024-05-27T07:42:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.232+0000 7ff2f37fe640 1 -- 172.21.15.114:0/2797374278 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3040053b0 con 0x55c499dd64d0 2024-05-27T07:42:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.233+0000 7ff317193740 1 -- 172.21.15.114:0/2797374278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 msgr2=0x55c49a19e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.233+0000 7ff317193740 1 --2- 172.21.15.114:0/2797374278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 0x55c49a19e770 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff304003bb0 tx=0x7ff3040058d0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.233+0000 7ff317193740 1 -- 172.21.15.114:0/2797374278 shutdown_connections 2024-05-27T07:42:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.233+0000 7ff317193740 1 --2- 172.21.15.114:0/2797374278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499df71b0 0x55c49a1bc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.233+0000 7ff317193740 1 --2- 172.21.15.114:0/2797374278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 0x55c49a19e770 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.233+0000 7ff317193740 1 --2- 172.21.15.114:0/2797374278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499dce2f0 0x55c49a19e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.233+0000 7ff317193740 1 -- 172.21.15.114:0/2797374278 >> 172.21.15.114:0/2797374278 conn(0x55c499deb300 msgr2=0x55c49a1a33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.233+0000 7ff317193740 1 -- 172.21.15.114:0/2797374278 shutdown_connections 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.233+0000 7ff317193740 1 -- 172.21.15.114:0/2797374278 wait complete. 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.234+0000 7ff317193740 1 -- start start 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.234+0000 7ff317193740 1 Processor -- start 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.234+0000 7ff317193740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499dce2f0 0x55c49a1acec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.234+0000 7ff317193740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 0x55c49a1aa230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.234+0000 7ff317193740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499df71b0 0x55c49a1aa770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.234+0000 7ff317193740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff3081ade90 con 0x55c499dd64d0 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.234+0000 7ff317193740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff3081af300 con 0x55c499dce2f0 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.234+0000 7ff317193740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff3081ae560 con 0x55c499df71b0 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.234+0000 7ff317193740 10 client.0 _ll_register_callbacks cb 0x7ff30400bb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff30dfa1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 0x55c49a1aa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff30dfa1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 0x55c49a1aa230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33446/0 (socket says 172.21.15.114:33446) 2024-05-27T07:42:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff30d7a0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499df71b0 0x55c49a1aa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff30dfa1640 1 -- 172.21.15.114:0/251422673 learned_addr learned my addr 172.21.15.114:0/251422673 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/251422673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499dce2f0 0x55c49a1acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff30dfa1640 1 -- 172.21.15.114:0/251422673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499df71b0 msgr2=0x55c49a1aa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff30dfa1640 1 --2- 172.21.15.114:0/251422673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499df71b0 0x55c49a1aa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff30dfa1640 1 -- 172.21.15.114:0/251422673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499dce2f0 msgr2=0x55c49a1acec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff30dfa1640 1 --2- 172.21.15.114:0/251422673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499dce2f0 0x55c49a1acec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff30dfa1640 1 -- 172.21.15.114:0/251422673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c49a1aacb0 con 0x55c499dd64d0 2024-05-27T07:42:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/251422673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499df71b0 0x55c49a1aa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff30dfa1640 1 --2- 172.21.15.114:0/251422673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 0x55c49a1aa230 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff304003cd0 tx=0x7ff3040043e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c8675824d4cab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff2f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff2f17fa640 1 -- 172.21.15.114:0/251422673 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30402d030 con 0x55c499dd64d0 2024-05-27T07:42:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff2f17fa640 1 -- 172.21.15.114:0/251422673 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7ff304003800 con 0x55c499dd64d0 2024-05-27T07:42:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff317193740 1 -- 172.21.15.114:0/251422673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff304004000 con 0x55c499dd64d0 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff2f17fa640 1 -- 172.21.15.114:0/251422673 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff304041050 con 0x55c499dd64d0 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff317193740 10 client.15930 fetch_fsmap learned FSMap version 25 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.235+0000 7ff317193740 1 -- 172.21.15.114:0/251422673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c49a21f2f0 con 0x55c499dd64d0 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.236+0000 7ff2f17fa640 1 -- 172.21.15.114:0/251422673 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff304004000 con 0x55c499dd64d0 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.236+0000 7ff2f17fa640 1 -- 172.21.15.114:0/251422673 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7ff304010c90 con 0x55c499dd64d0 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.236+0000 7ff317193740 10 client.15930 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.236+0000 7ff317193740 10 client.15930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.236+0000 7ff317193740 20 client.15930 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.236+0000 7ff317193740 1 --2- 172.21.15.114:0/251422673 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c499d8b1c0 0x55c49a221980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.236+0000 7ff317193740 4 client.15930 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.236+0000 7ff317193740 1 -- 172.21.15.114:0/251422673 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c49a106360 con 0x55c499d8b1c0 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.236+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/251422673 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c499d8b1c0 0x55c49a221980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.237+0000 7ff2f17fa640 10 client.15930 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.237+0000 7ff30d7a0640 1 --2- 172.21.15.114:0/251422673 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c499d8b1c0 0x55c49a221980 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8404635be81edde8 server_cookie=d50012b69b9c373a in_seq=0 out_seq=0 2024-05-27T07:42:48.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 163349 14.99 MB/sec warmup 60 sec latency 20.679 ms 2024-05-27T07:42:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.489+0000 7ff2f17fa640 1 -- 172.21.15.114:0/251422673 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55c49a106360 con 0x55c499d8b1c0 2024-05-27T07:42:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.489+0000 7ff2f17fa640 10 client.15930 handle_command_reply: tid=0 2024-05-27T07:42:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.489+0000 7ff317193740 1 client.15930 shutdown 2024-05-27T07:42:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.490+0000 7ff317193740 1 -- 172.21.15.114:0/251422673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 msgr2=0x55c49a1aa230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.490+0000 7ff317193740 1 --2- 172.21.15.114:0/251422673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 0x55c49a1aa230 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff304003cd0 tx=0x7ff3040043e0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.490+0000 7ff317193740 1 -- 172.21.15.114:0/251422673 shutdown_connections 2024-05-27T07:42:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.490+0000 7ff317193740 1 --2- 172.21.15.114:0/251422673 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c499d8b1c0 0x55c49a221980 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.490+0000 7ff317193740 1 --2- 172.21.15.114:0/251422673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c499df71b0 0x55c49a1aa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.490+0000 7ff2f17fa640 0 client.15930 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.490+0000 7ff317193740 1 --2- 172.21.15.114:0/251422673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c499dd64d0 0x55c49a1aa230 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.490+0000 7ff317193740 1 --2- 172.21.15.114:0/251422673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c499dce2f0 0x55c49a1acec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.490+0000 7ff317193740 1 -- 172.21.15.114:0/251422673 >> 172.21.15.114:0/251422673 conn(0x55c499deb300 msgr2=0x55c49a19e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.490+0000 7ff317193740 1 -- 172.21.15.114:0/251422673 shutdown_connections 2024-05-27T07:42:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.490+0000 7ff317193740 1 -- 172.21.15.114:0/251422673 wait complete. 2024-05-27T07:42:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.490+0000 7ff317193740 20 client.15930 trim_cache size 0 max 16384 2024-05-27T07:42:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:42:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:42:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.491+0000 7ff310205640 1 -- 172.21.15.114:0/1063949785 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2f40b2950 msgr2=0x7ff2f40b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.491+0000 7ff310205640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2f40b2950 0x7ff2f40b4e60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff3001012c0 tx=0x7ff30002a8c0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.491+0000 7ff310205640 1 -- 172.21.15.114:0/1063949785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff308109030 msgr2=0x7ff308199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.491+0000 7ff310205640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff308109030 0x7ff308199660 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff3000098d0 tx=0x7ff3000317b0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.491+0000 7ff310205640 1 -- 172.21.15.114:0/1063949785 shutdown_connections 2024-05-27T07:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.491+0000 7ff310205640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2f40b2950 0x7ff2f40b4e60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.491+0000 7ff310205640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff308109030 0x7ff308199660 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.491+0000 7ff310205640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff308108b90 0x7ff30819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.491+0000 7ff310205640 1 --2- 172.21.15.114:0/1063949785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3081032e0 0x7ff30819f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.491+0000 7ff310205640 1 -- 172.21.15.114:0/1063949785 >> 172.21.15.114:0/1063949785 conn(0x7ff3080782d0 msgr2=0x7ff3080fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.491+0000 7ff310205640 1 -- 172.21.15.114:0/1063949785 shutdown_connections 2024-05-27T07:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:42:48.491+0000 7ff310205640 1 -- 172.21.15.114:0/1063949785 wait complete. 2024-05-27T07:42:48.721 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:48 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/487156725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:42:48.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:42:48.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:48 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/487156725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:42:48.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:48 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/487156725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:42:49.013 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:49.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 164871 14.85 MB/sec warmup 61 sec latency 160.089 ms 2024-05-27T07:42:49.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:49 smithi114 ceph-mon[32038]: pgmap v97: 129 pgs: 129 active+clean; 43 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2024-05-27T07:42:49.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:49 smithi163 ceph-mon[36451]: pgmap v97: 129 pgs: 129 active+clean; 43 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2024-05-27T07:42:49.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:49 smithi186 ceph-mon[36647]: pgmap v97: 129 pgs: 129 active+clean; 43 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2024-05-27T07:42:50.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 167456 14.84 MB/sec warmup 62 sec latency 27.991 ms 2024-05-27T07:42:51.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.083+0000 7f9cf72e0700 1 -- 172.21.15.114:0/1263347761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9cf01034b0 msgr2=0x7f9cf01038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:51.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.083+0000 7f9cf72e0700 1 --2- 172.21.15.114:0/1263347761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9cf01034b0 0x7f9cf01038c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0099c0 tx=0x7f9cec009cd0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.083+0000 7f9cf72e0700 1 -- 172.21.15.114:0/1263347761 shutdown_connections 2024-05-27T07:42:51.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.083+0000 7f9cf72e0700 1 --2- 172.21.15.114:0/1263347761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cf0108730 0x7f9cf0110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.083+0000 7f9cf72e0700 1 --2- 172.21.15.114:0/1263347761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9cf01034b0 0x7f9cf01038c0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.083+0000 7f9cf72e0700 1 --2- 172.21.15.114:0/1263347761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cf0107de0 0x7f9cf01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.083+0000 7f9cf72e0700 1 -- 172.21.15.114:0/1263347761 >> 172.21.15.114:0/1263347761 conn(0x7f9cf00fedb0 msgr2=0x7f9cf01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:51.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.083+0000 7f9cf72e0700 1 -- 172.21.15.114:0/1263347761 shutdown_connections 2024-05-27T07:42:51.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.083+0000 7f9cf72e0700 1 -- 172.21.15.114:0/1263347761 wait complete. 2024-05-27T07:42:51.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.084+0000 7f9cf72e0700 1 Processor -- start 2024-05-27T07:42:51.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.084+0000 7f9cf72e0700 1 -- start start 2024-05-27T07:42:51.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.084+0000 7f9cf72e0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9cf01034b0 0x7f9cf019f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.084+0000 7f9cf72e0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cf0107de0 0x7f9cf019fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.084+0000 7f9cf72e0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cf0108730 0x7f9cf0199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.084+0000 7f9cf72e0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf0113360 con 0x7f9cf01034b0 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.084+0000 7f9cf72e0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf01131e0 con 0x7f9cf0107de0 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.084+0000 7f9cf72e0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf01134e0 con 0x7f9cf0108730 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.084+0000 7f9cf507c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9cf01034b0 0x7f9cf019f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.084+0000 7f9cf507c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9cf01034b0 0x7f9cf019f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33458/0 (socket says 172.21.15.114:33458) 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.084+0000 7f9cf507c700 1 -- 172.21.15.114:0/3983546159 learned_addr learned my addr 172.21.15.114:0/3983546159 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.084+0000 7f9cf487b700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cf0107de0 0x7f9cf019fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.084+0000 7f9cf587d700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cf0108730 0x7f9cf0199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.085+0000 7f9cf507c700 1 -- 172.21.15.114:0/3983546159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cf0108730 msgr2=0x7f9cf0199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.085+0000 7f9cf507c700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cf0108730 0x7f9cf0199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.085+0000 7f9cf507c700 1 -- 172.21.15.114:0/3983546159 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cf0107de0 msgr2=0x7f9cf019fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.085+0000 7f9cf507c700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cf0107de0 0x7f9cf019fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.085+0000 7f9cf507c700 1 -- 172.21.15.114:0/3983546159 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf019a1c0 con 0x7f9cf01034b0 2024-05-27T07:42:51.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.085+0000 7f9cf507c700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9cf01034b0 0x7f9cf019f8a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9cec006010 tx=0x7f9cec0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:51.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.085+0000 7f9ce67fc700 1 -- 172.21.15.114:0/3983546159 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cec028070 con 0x7f9cf01034b0 2024-05-27T07:42:51.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.085+0000 7f9ce67fc700 1 -- 172.21.15.114:0/3983546159 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cec019600 con 0x7f9cf01034b0 2024-05-27T07:42:51.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.085+0000 7f9cf72e0700 1 -- 172.21.15.114:0/3983546159 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cf019a450 con 0x7f9cf01034b0 2024-05-27T07:42:51.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.085+0000 7f9ce67fc700 1 -- 172.21.15.114:0/3983546159 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cec009e60 con 0x7f9cf01034b0 2024-05-27T07:42:51.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.085+0000 7f9cf72e0700 1 -- 172.21.15.114:0/3983546159 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cf006b600 con 0x7f9cf01034b0 2024-05-27T07:42:51.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.088+0000 7f9ce67fc700 1 -- 172.21.15.114:0/3983546159 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f9cec02b070 con 0x7f9cf01034b0 2024-05-27T07:42:51.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.089+0000 7f9ce67fc700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9cdc0b29f0 0x7f9cdc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.089+0000 7f9cf487b700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9cdc0b29f0 0x7f9cdc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.089+0000 7f9ce67fc700 1 -- 172.21.15.114:0/3983546159 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f9cec0e53e0 con 0x7f9cf01034b0 2024-05-27T07:42:51.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.089+0000 7f9d05cf5b80 1 Processor -- start 2024-05-27T07:42:51.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.089+0000 7f9d05cf5b80 1 -- start start 2024-05-27T07:42:51.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.089+0000 7f9cf487b700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9cdc0b29f0 0x7f9cdc0b4f00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9cf01063d0 tx=0x7f9ce0009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:51.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9d05cf5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991a9ee40 0x558991c22710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9d05cf5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c1ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9d05cf5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991c1cf80 0x558991c1d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9d05cf5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558991b4bc90 con 0x558991c22c50 2024-05-27T07:42:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9d05cf5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558991b3b830 con 0x558991c1cf80 2024-05-27T07:42:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9d05cf5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558991b3f870 con 0x558991a9ee40 2024-05-27T07:42:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9cf487b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c1ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9cf487b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c1ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33460/0 (socket says 172.21.15.114:33460) 2024-05-27T07:42:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9cf487b700 1 -- 172.21.15.114:0/317832828 learned_addr learned my addr 172.21.15.114:0/317832828 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9cf507c700 1 --2- 172.21.15.114:0/317832828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991a9ee40 0x558991c22710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9cf587d700 1 --2- 172.21.15.114:0/317832828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991c1cf80 0x558991c1d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9cf507c700 1 -- 172.21.15.114:0/317832828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991c1cf80 msgr2=0x558991c1d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9cf507c700 1 --2- 172.21.15.114:0/317832828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991c1cf80 0x558991c1d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9cf507c700 1 -- 172.21.15.114:0/317832828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 msgr2=0x558991c1ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:51.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9cf507c700 1 --2- 172.21.15.114:0/317832828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c1ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9cf507c700 1 -- 172.21.15.114:0/317832828 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558991a8b8d0 con 0x558991a9ee40 2024-05-27T07:42:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9cf487b700 1 --2- 172.21.15.114:0/317832828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c1ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.090+0000 7f9cf507c700 1 --2- 172.21.15.114:0/317832828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991a9ee40 0x558991c22710 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0e2780 tx=0x7f9cec0abb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebbab119ac9b01ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.091+0000 7f9cdb7fe700 1 -- 172.21.15.114:0/317832828 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cec0ac3a0 con 0x558991a9ee40 2024-05-27T07:42:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.091+0000 7f9cdb7fe700 1 -- 172.21.15.114:0/317832828 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cec0b0750 con 0x558991a9ee40 2024-05-27T07:42:51.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.091+0000 7f9cdb7fe700 1 -- 172.21.15.114:0/317832828 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cec0ac600 con 0x558991a9ee40 2024-05-27T07:42:51.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.092+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/317832828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991a9ee40 msgr2=0x558991c22710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:51.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.092+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/317832828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991a9ee40 0x558991c22710 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0e2780 tx=0x7f9cec0abb10 comp rx=0 tx=0).stop 2024-05-27T07:42:51.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.092+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/317832828 shutdown_connections 2024-05-27T07:42:51.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.092+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/317832828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991c1cf80 0x558991c1d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.092+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/317832828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991a9ee40 0x558991c22710 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.092+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/317832828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c1ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.092+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/317832828 >> 172.21.15.114:0/317832828 conn(0x558991af9af0 msgr2=0x558991c17640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:51.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.092+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/317832828 shutdown_connections 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.092+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/317832828 wait complete. 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.092+0000 7f9d05cf5b80 1 -- start start 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.092+0000 7f9d05cf5b80 1 Processor -- start 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9d05cf5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c263e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9d05cf5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c1cf80 0x558991c26920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9d05cf5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c26e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9d05cf5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558991a41fa0 con 0x558991c22c50 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9d05cf5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558991add7c0 con 0x558991a9ee40 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9d05cf5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558991c29500 con 0x558991c1cf80 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9d05cf5b80 10 client.0 _ll_register_callbacks cb 0x558991ad6920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9cf507c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c263e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9cf487b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c1cf80 0x558991c26920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9cf587d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c26e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9cf507c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c263e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51600/0 (socket says 172.21.15.114:51600) 2024-05-27T07:42:51.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9cf507c700 1 -- 172.21.15.114:0/1150737108 learned_addr learned my addr 172.21.15.114:0/1150737108 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9cf507c700 1 -- 172.21.15.114:0/1150737108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c1cf80 msgr2=0x558991c26920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9cf507c700 1 --2- 172.21.15.114:0/1150737108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c1cf80 0x558991c26920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9cf507c700 1 -- 172.21.15.114:0/1150737108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 msgr2=0x558991c26e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9cf507c700 1 --2- 172.21.15.114:0/1150737108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c26e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.093+0000 7f9cf507c700 1 -- 172.21.15.114:0/1150737108 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558991ada690 con 0x558991a9ee40 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9cf507c700 1 --2- 172.21.15.114:0/1150737108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c263e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0e2d80 tx=0x7f9cec0e27a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23063ca0d0d44f1e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9cd8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9cd8ff9700 1 -- 172.21.15.114:0/1150737108 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cec0a8630 con 0x558991a9ee40 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/1150737108 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cec0b0750 con 0x558991a9ee40 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9cd8ff9700 1 -- 172.21.15.114:0/1150737108 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f9cec0ac600 con 0x558991a9ee40 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9cd8ff9700 1 -- 172.21.15.114:0/1150737108 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9cec0ac800 con 0x558991a9ee40 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9d05cf5b80 10 client.25469 fetch_fsmap learned FSMap version 25 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/1150737108 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558991c33f40 con 0x558991a9ee40 2024-05-27T07:42:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9cd8ff9700 1 -- 172.21.15.114:0/1150737108 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cec0b0750 con 0x558991a9ee40 2024-05-27T07:42:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9cd8ff9700 1 -- 172.21.15.114:0/1150737108 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f9cec0a7da0 con 0x558991a9ee40 2024-05-27T07:42:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9d05cf5b80 10 client.25469 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:42:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9d05cf5b80 10 client.25469 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9d05cf5b80 20 client.25469 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/1150737108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558991c36240 0x558991c38660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9d05cf5b80 4 client.25469 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.094+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/1150737108 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558991be89f0 con 0x558991c36240 2024-05-27T07:42:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.095+0000 7f9cf487b700 1 --2- 172.21.15.114:0/1150737108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558991c36240 0x558991c38660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.095+0000 7f9cd8ff9700 10 client.25469 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:51.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.095+0000 7f9cf487b700 1 --2- 172.21.15.114:0/1150737108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558991c36240 0x558991c38660 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=452a6e5c34be4823 server_cookie=2ea00eb7e1cd1c50 in_seq=0 out_seq=0 2024-05-27T07:42:51.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.096+0000 7f9cd8ff9700 1 -- 172.21.15.114:0/1150737108 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558991be89f0 con 0x558991c36240 2024-05-27T07:42:51.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.096+0000 7f9cd8ff9700 10 client.25469 handle_command_reply: tid=0 2024-05-27T07:42:51.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.096+0000 7f9d05cf5b80 1 client.25469 shutdown 2024-05-27T07:42:51.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.096+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/1150737108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 msgr2=0x558991c263e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:51.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.096+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/1150737108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c263e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0e2d80 tx=0x7f9cec0e27a0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.096+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/1150737108 shutdown_connections 2024-05-27T07:42:51.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.096+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/1150737108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c26e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.096+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/1150737108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558991c36240 0x558991c38660 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.096+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/1150737108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c263e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.096+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/1150737108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c1cf80 0x558991c26920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.096+0000 7f9cd8ff9700 0 client.25469 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:51.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.096+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/1150737108 >> 172.21.15.114:0/1150737108 conn(0x558991af9af0 msgr2=0x558991c1ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:51.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.097+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/1150737108 shutdown_connections 2024-05-27T07:42:51.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.097+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/1150737108 wait complete. 2024-05-27T07:42:51.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.097+0000 7f9d05cf5b80 20 client.25469 trim_cache size 0 max 16384 2024-05-27T07:42:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.112+0000 7f9d05cf5b80 1 Processor -- start 2024-05-27T07:42:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.112+0000 7f9d05cf5b80 1 -- start start 2024-05-27T07:42:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.112+0000 7f9d05cf5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c1ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.112+0000 7f9d05cf5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c1cf80 0x558991c28950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.112+0000 7f9d05cf5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c22c50 0x558991c253e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.112+0000 7f9d05cf5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf01ac2b0 con 0x558991c1cf80 2024-05-27T07:42:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.112+0000 7f9d05cf5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf01aeb30 con 0x558991a9ee40 2024-05-27T07:42:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.112+0000 7f9d05cf5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf01ac920 con 0x558991c22c50 2024-05-27T07:42:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.112+0000 7f9cf487b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c1cf80 0x558991c28950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.112+0000 7f9cf487b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c1cf80 0x558991c28950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33476/0 (socket says 172.21.15.114:33476) 2024-05-27T07:42:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.112+0000 7f9cf487b700 1 -- 172.21.15.114:0/1542942991 learned_addr learned my addr 172.21.15.114:0/1542942991 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.112+0000 7f9cf587d700 1 --2- 172.21.15.114:0/1542942991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c22c50 0x558991c253e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.112+0000 7f9cf507c700 1 --2- 172.21.15.114:0/1542942991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c1ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.113+0000 7f9cf487b700 1 -- 172.21.15.114:0/1542942991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c22c50 msgr2=0x558991c253e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.113+0000 7f9cf487b700 1 --2- 172.21.15.114:0/1542942991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c22c50 0x558991c253e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.113+0000 7f9cf487b700 1 -- 172.21.15.114:0/1542942991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 msgr2=0x558991c1ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.113+0000 7f9cf487b700 1 --2- 172.21.15.114:0/1542942991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c1ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.113+0000 7f9cf487b700 1 -- 172.21.15.114:0/1542942991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558991c18eb0 con 0x558991c1cf80 2024-05-27T07:42:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.113+0000 7f9cf507c700 1 --2- 172.21.15.114:0/1542942991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c1ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.113+0000 7f9cf487b700 1 --2- 172.21.15.114:0/1542942991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c1cf80 0x558991c28950 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x558991c38c00 tx=0x7f9ce0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87b41c3264b7b8f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.113+0000 7f9cdaffd700 1 -- 172.21.15.114:0/1542942991 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0003ce0 con 0x558991c1cf80 2024-05-27T07:42:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.113+0000 7f9cdaffd700 1 -- 172.21.15.114:0/1542942991 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ce0004560 con 0x558991c1cf80 2024-05-27T07:42:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.113+0000 7f9cdaffd700 1 -- 172.21.15.114:0/1542942991 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce001d650 con 0x558991c1cf80 2024-05-27T07:42:51.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.114+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/1542942991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c1cf80 msgr2=0x558991c28950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:51.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.114+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/1542942991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c1cf80 0x558991c28950 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x558991c38c00 tx=0x7f9ce0014000 comp rx=0 tx=0).stop 2024-05-27T07:42:51.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.114+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/1542942991 shutdown_connections 2024-05-27T07:42:51.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.114+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/1542942991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c22c50 0x558991c253e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.114+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/1542942991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c1ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.114+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/1542942991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c1cf80 0x558991c28950 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.114+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/1542942991 >> 172.21.15.114:0/1542942991 conn(0x558991af9af0 msgr2=0x558991c24cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.114+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/1542942991 shutdown_connections 2024-05-27T07:42:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.114+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/1542942991 wait complete. 2024-05-27T07:42:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.114+0000 7f9d05cf5b80 1 -- start start 2024-05-27T07:42:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.114+0000 7f9d05cf5b80 1 Processor -- start 2024-05-27T07:42:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9d05cf5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c2e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9d05cf5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c1cf80 0x558991c2ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9d05cf5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c3cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9d05cf5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf01acff0 con 0x558991c22c50 2024-05-27T07:42:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9d05cf5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf01ae460 con 0x558991a9ee40 2024-05-27T07:42:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9d05cf5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf01ad6c0 con 0x558991c1cf80 2024-05-27T07:42:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9d05cf5b80 10 client.0 _ll_register_callbacks cb 0x558991ad6920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9cf587d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c3cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9cf587d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c3cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33478/0 (socket says 172.21.15.114:33478) 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9cf587d700 1 -- 172.21.15.114:0/3288795422 learned_addr learned my addr 172.21.15.114:0/3288795422 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9cf487b700 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c1cf80 0x558991c2ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9cf507c700 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c2e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9cf587d700 1 -- 172.21.15.114:0/3288795422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c1cf80 msgr2=0x558991c2ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9cf587d700 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c1cf80 0x558991c2ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9cf587d700 1 -- 172.21.15.114:0/3288795422 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 msgr2=0x558991c2e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9cf587d700 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c2e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9cf587d700 1 -- 172.21.15.114:0/3288795422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558991c3d270 con 0x558991c22c50 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.115+0000 7f9cf487b700 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c1cf80 0x558991c2ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.116+0000 7f9cf587d700 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c3cd30 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8013b70 tx=0x7f9ce8013e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca3876a9593628f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.116+0000 7f9cd8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.116+0000 7f9cd8ff9700 1 -- 172.21.15.114:0/3288795422 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8004db0 con 0x558991c22c50 2024-05-27T07:42:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.116+0000 7f9cd8ff9700 1 -- 172.21.15.114:0/3288795422 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f9ce80044b0 con 0x558991c22c50 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.116+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/3288795422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ce0004560 con 0x558991c22c50 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.116+0000 7f9cd8ff9700 1 -- 172.21.15.114:0/3288795422 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9ce802a050 con 0x558991c22c50 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.116+0000 7f9d05cf5b80 10 client.15948 fetch_fsmap learned FSMap version 25 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.116+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/3288795422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558991ca02f0 con 0x558991c22c50 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.116+0000 7f9cd8ff9700 1 -- 172.21.15.114:0/3288795422 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ce0004560 con 0x558991c22c50 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.116+0000 7f9cd8ff9700 1 -- 172.21.15.114:0/3288795422 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f9ce801f410 con 0x558991c22c50 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.116+0000 7f9d05cf5b80 10 client.15948 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.116+0000 7f9d05cf5b80 10 client.15948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.116+0000 7f9d05cf5b80 20 client.15948 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.117+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558991ca0ab0 0x558991ca2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.117+0000 7f9d05cf5b80 4 client.15948 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.117+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/3288795422 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558991b4dbe0 con 0x558991ca0ab0 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.117+0000 7f9cf507c700 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558991ca0ab0 0x558991ca2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.117+0000 7f9cd8ff9700 10 client.15948 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:51.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.117+0000 7f9cf507c700 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558991ca0ab0 0x558991ca2ed0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75dac043819fdf25 server_cookie=190ab9a63a1a38f8 in_seq=0 out_seq=0 2024-05-27T07:42:51.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.118+0000 7f9cd8ff9700 1 -- 172.21.15.114:0/3288795422 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558991b4dbe0 con 0x558991ca0ab0 2024-05-27T07:42:51.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.118+0000 7f9cd8ff9700 10 client.15948 handle_command_reply: tid=0 2024-05-27T07:42:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.118+0000 7f9d05cf5b80 1 client.15948 shutdown 2024-05-27T07:42:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/3288795422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 msgr2=0x558991c3cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c3cd30 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8013b70 tx=0x7f9ce8013e80 comp rx=0 tx=0).stop 2024-05-27T07:42:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/3288795422 shutdown_connections 2024-05-27T07:42:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558991c22c50 0x558991c3cd30 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558991ca0ab0 0x558991ca2ed0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558991a9ee40 0x558991c2e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9d05cf5b80 1 --2- 172.21.15.114:0/3288795422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558991c1cf80 0x558991c2ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/3288795422 >> 172.21.15.114:0/3288795422 conn(0x558991af9af0 msgr2=0x558991c20010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9cd8ff9700 0 client.15948 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/3288795422 shutdown_connections 2024-05-27T07:42:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9d05cf5b80 1 -- 172.21.15.114:0/3288795422 wait complete. 2024-05-27T07:42:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9d05cf5b80 20 client.15948 trim_cache size 0 max 16384 2024-05-27T07:42:51.122 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:42:51.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9cf72e0700 1 -- 172.21.15.114:0/3983546159 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9cdc0b29f0 msgr2=0x7f9cdc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:51.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9cf72e0700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9cdc0b29f0 0x7f9cdc0b4f00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9cf01063d0 tx=0x7f9ce0009260 comp rx=0 tx=0).stop 2024-05-27T07:42:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9cf72e0700 1 -- 172.21.15.114:0/3983546159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9cf01034b0 msgr2=0x7f9cf019f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.119+0000 7f9cf72e0700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9cf01034b0 0x7f9cf019f8a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9cec006010 tx=0x7f9cec0045e0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.120+0000 7f9cf72e0700 1 -- 172.21.15.114:0/3983546159 shutdown_connections 2024-05-27T07:42:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.120+0000 7f9cf72e0700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cf0108730 0x7f9cf0199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.120+0000 7f9cf72e0700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9cf01034b0 0x7f9cf019f8a0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.120+0000 7f9cf72e0700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9cdc0b29f0 0x7f9cdc0b4f00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.120+0000 7f9cf72e0700 1 --2- 172.21.15.114:0/3983546159 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cf0107de0 0x7f9cf019fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.120+0000 7f9cf72e0700 1 -- 172.21.15.114:0/3983546159 >> 172.21.15.114:0/3983546159 conn(0x7f9cf00fedb0 msgr2=0x7f9cf0101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.120+0000 7f9cf72e0700 1 -- 172.21.15.114:0/3983546159 shutdown_connections 2024-05-27T07:42:51.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:51.120+0000 7f9cf72e0700 1 -- 172.21.15.114:0/3983546159 wait complete. 2024-05-27T07:42:51.239 INFO:tasks.workunit.client.0.smithi114.stdout: 1 170569 14.90 MB/sec warmup 63 sec latency 21.030 ms 2024-05-27T07:42:51.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:51 smithi114 ceph-mon[32038]: pgmap v98: 129 pgs: 129 active+clean; 39 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.2 MiB/s wr, 424 op/s 2024-05-27T07:42:51.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:51 smithi163 ceph-mon[36451]: pgmap v98: 129 pgs: 129 active+clean; 39 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.2 MiB/s wr, 424 op/s 2024-05-27T07:42:51.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:51 smithi186 ceph-mon[36647]: pgmap v98: 129 pgs: 129 active+clean; 39 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.2 MiB/s wr, 424 op/s 2024-05-27T07:42:52.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 173272 14.87 MB/sec warmup 64 sec latency 25.699 ms 2024-05-27T07:42:53.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 176200 14.85 MB/sec warmup 65 sec latency 14.508 ms 2024-05-27T07:42:53.262 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:42:53.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:42:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:53.887 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:53 smithi114 ceph-mon[32038]: pgmap v99: 129 pgs: 129 active+clean; 39 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 327 op/s 2024-05-27T07:42:53.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:53 smithi163 ceph-mon[36451]: pgmap v99: 129 pgs: 129 active+clean; 39 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 327 op/s 2024-05-27T07:42:53.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:53 smithi186 ceph-mon[36647]: pgmap v99: 129 pgs: 129 active+clean; 39 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 327 op/s 2024-05-27T07:42:54.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178413 14.83 MB/sec warmup 66 sec latency 30.956 ms 2024-05-27T07:42:55.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 181359 14.87 MB/sec warmup 67 sec latency 22.318 ms 2024-05-27T07:42:55.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.359+0000 7f666e8d1700 1 -- 172.21.15.114:0/2143331544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666810e8f0 msgr2=0x7f6668110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:55.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.359+0000 7f666e8d1700 1 --2- 172.21.15.114:0/2143331544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666810e8f0 0x7f6668110db0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f666000b0a0 tx=0x7f666000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.360+0000 7f666e8d1700 1 -- 172.21.15.114:0/2143331544 shutdown_connections 2024-05-27T07:42:55.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.360+0000 7f666e8d1700 1 --2- 172.21.15.114:0/2143331544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666810e8f0 0x7f6668110db0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.360+0000 7f666e8d1700 1 --2- 172.21.15.114:0/2143331544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6668105510 0x7f6668105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.360+0000 7f666e8d1700 1 --2- 172.21.15.114:0/2143331544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6668105e60 0x7f666810e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.360+0000 7f666e8d1700 1 -- 172.21.15.114:0/2143331544 >> 172.21.15.114:0/2143331544 conn(0x7f66680fedb0 msgr2=0x7f66681011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:55.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.360+0000 7f666e8d1700 1 -- 172.21.15.114:0/2143331544 shutdown_connections 2024-05-27T07:42:55.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.360+0000 7f666e8d1700 1 -- 172.21.15.114:0/2143331544 wait complete. 2024-05-27T07:42:55.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.360+0000 7f666e8d1700 1 Processor -- start 2024-05-27T07:42:55.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.360+0000 7f666e8d1700 1 -- start start 2024-05-27T07:42:55.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f666e8d1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6668105510 0x7f666819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f666e8d1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6668105e60 0x7f666819b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f666e8d1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666810e8f0 0x7f666819f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f666e8d1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6668114430 con 0x7f666810e8f0 2024-05-27T07:42:55.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f666e8d1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f66681142b0 con 0x7f6668105e60 2024-05-27T07:42:55.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f666e8d1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f66681145b0 con 0x7f6668105510 2024-05-27T07:42:55.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f666ce6e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666810e8f0 0x7f666819f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f6667fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6668105510 0x7f666819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f6667fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6668105510 0x7f666819ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:52292/0 (socket says 172.21.15.114:52292) 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f6667fff700 1 -- 172.21.15.114:0/3357031260 learned_addr learned my addr 172.21.15.114:0/3357031260 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f66677fe700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6668105e60 0x7f666819b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f6667fff700 1 -- 172.21.15.114:0/3357031260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6668105e60 msgr2=0x7f666819b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f6667fff700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6668105e60 0x7f666819b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f6667fff700 1 -- 172.21.15.114:0/3357031260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666810e8f0 msgr2=0x7f666819f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f6667fff700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666810e8f0 0x7f666819f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f6667fff700 1 -- 172.21.15.114:0/3357031260 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f666819fc10 con 0x7f6668105510 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.361+0000 7f666ce6e700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666810e8f0 0x7f666819f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.362+0000 7f6667fff700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6668105510 0x7f666819ad40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f665800e9a0 tx=0x7f665800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.362+0000 7f66657fa700 1 -- 172.21.15.114:0/3357031260 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665800ca10 con 0x7f6668105510 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.362+0000 7f666e8d1700 1 -- 172.21.15.114:0/3357031260 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f666819ff00 con 0x7f6668105510 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.362+0000 7f66657fa700 1 -- 172.21.15.114:0/3357031260 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f665800cbb0 con 0x7f6668105510 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.362+0000 7f66657fa700 1 -- 172.21.15.114:0/3357031260 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6658018680 con 0x7f6668105510 2024-05-27T07:42:55.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.362+0000 7f666e8d1700 1 -- 172.21.15.114:0/3357031260 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66681a7820 con 0x7f6668105510 2024-05-27T07:42:55.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.364+0000 7f66657fa700 1 -- 172.21.15.114:0/3357031260 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6658018820 con 0x7f6668105510 2024-05-27T07:42:55.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.365+0000 7f66657fa700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f66500b29f0 0x7f66500b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.365+0000 7f66677fe700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f66500b29f0 0x7f66500b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.365+0000 7f66657fa700 1 -- 172.21.15.114:0/3357031260 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f66580e2be0 con 0x7f6668105510 2024-05-27T07:42:55.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f667d2e6b80 1 Processor -- start 2024-05-27T07:42:55.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f667d2e6b80 1 -- start start 2024-05-27T07:42:55.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f66677fe700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f66500b29f0 0x7f66500b4f00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f66681021e0 tx=0x7f665c009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:55.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f667d2e6b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6d3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f667d2e6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6d3da0 0x55a66d6d41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f667d2e6b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6dc8d0 0x55a66d6decf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f667d2e6b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a66d5f7df0 con 0x55a66d54aec0 2024-05-27T07:42:55.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f667d2e6b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a66d5e7990 con 0x55a66d6dc8d0 2024-05-27T07:42:55.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f667d2e6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a66d5eb9d0 con 0x55a66d6d3da0 2024-05-27T07:42:55.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f6667fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6d3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f6667fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6d3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54188/0 (socket says 172.21.15.114:54188) 2024-05-27T07:42:55.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f6667fff700 1 -- 172.21.15.114:0/2493352869 learned_addr learned my addr 172.21.15.114:0/2493352869 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:55.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f66677fe700 1 --2- 172.21.15.114:0/2493352869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6d3da0 0x55a66d6d41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f666ce6e700 1 --2- 172.21.15.114:0/2493352869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6dc8d0 0x55a66d6decf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f66677fe700 1 -- 172.21.15.114:0/2493352869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6dc8d0 msgr2=0x55a66d6decf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:55.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f66677fe700 1 --2- 172.21.15.114:0/2493352869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6dc8d0 0x55a66d6decf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f66677fe700 1 -- 172.21.15.114:0/2493352869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 msgr2=0x55a66d6d3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:55.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f66677fe700 1 --2- 172.21.15.114:0/2493352869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6d3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f66677fe700 1 -- 172.21.15.114:0/2493352869 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a66d591030 con 0x55a66d6d3da0 2024-05-27T07:42:55.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.366+0000 7f6667fff700 1 --2- 172.21.15.114:0/2493352869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6d3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:55.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.367+0000 7f66677fe700 1 --2- 172.21.15.114:0/2493352869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6d3da0 0x55a66d6d41b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f665c009fd0 tx=0x7f665c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f78ac66d91aecb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:55.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.367+0000 7f664e7fc700 1 -- 172.21.15.114:0/2493352869 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c004020 con 0x55a66d6d3da0 2024-05-27T07:42:55.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.367+0000 7f664e7fc700 1 -- 172.21.15.114:0/2493352869 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f665c00e810 con 0x55a66d6d3da0 2024-05-27T07:42:55.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.367+0000 7f664e7fc700 1 -- 172.21.15.114:0/2493352869 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c003bc0 con 0x55a66d6d3da0 2024-05-27T07:42:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.368+0000 7f667d2e6b80 1 -- 172.21.15.114:0/2493352869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6d3da0 msgr2=0x55a66d6d41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.368+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/2493352869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6d3da0 0x55a66d6d41b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f665c009fd0 tx=0x7f665c014000 comp rx=0 tx=0).stop 2024-05-27T07:42:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.368+0000 7f667d2e6b80 1 -- 172.21.15.114:0/2493352869 shutdown_connections 2024-05-27T07:42:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.368+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/2493352869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6dc8d0 0x55a66d6decf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.368+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/2493352869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6d3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.368+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/2493352869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6d3da0 0x55a66d6d41b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.368+0000 7f667d2e6b80 1 -- 172.21.15.114:0/2493352869 >> 172.21.15.114:0/2493352869 conn(0x55a66d54d7f0 msgr2=0x55a66d6c50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.368+0000 7f667d2e6b80 1 -- 172.21.15.114:0/2493352869 shutdown_connections 2024-05-27T07:42:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.368+0000 7f667d2e6b80 1 -- 172.21.15.114:0/2493352869 wait complete. 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.368+0000 7f667d2e6b80 1 -- start start 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.368+0000 7f667d2e6b80 1 Processor -- start 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f667d2e6b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6d9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f667d2e6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6d3da0 0x55a66d6d9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f667d2e6b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6dc8d0 0x55a66d6da2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f667d2e6b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a66d4ee020 con 0x55a66d54aec0 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f667d2e6b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a66d54fe20 con 0x55a66d6dc8d0 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f667d2e6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a66d6df3e0 con 0x55a66d6d3da0 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f667d2e6b80 10 client.0 _ll_register_callbacks cb 0x55a66d57e630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f6667fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6d9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f66677fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6d3da0 0x55a66d6d9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f6667fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6d9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54196/0 (socket says 172.21.15.114:54196) 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f6667fff700 1 -- 172.21.15.114:0/2943345927 learned_addr learned my addr 172.21.15.114:0/2943345927 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f666ce6e700 1 --2- 172.21.15.114:0/2943345927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6dc8d0 0x55a66d6da2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f6667fff700 1 -- 172.21.15.114:0/2943345927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6d3da0 msgr2=0x55a66d6d9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f6667fff700 1 --2- 172.21.15.114:0/2943345927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6d3da0 0x55a66d6d9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f6667fff700 1 -- 172.21.15.114:0/2943345927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6dc8d0 msgr2=0x55a66d6da2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f6667fff700 1 --2- 172.21.15.114:0/2943345927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6dc8d0 0x55a66d6da2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f6667fff700 1 -- 172.21.15.114:0/2943345927 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a66d537950 con 0x55a66d54aec0 2024-05-27T07:42:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.369+0000 7f66677fe700 1 --2- 172.21.15.114:0/2943345927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6d3da0 0x55a66d6d9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.370+0000 7f6667fff700 1 --2- 172.21.15.114:0/2943345927 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6d9830 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f66580098c0 tx=0x7f66580b29f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90a555e26ae81c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.370+0000 7f66477fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:42:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.370+0000 7f66477fe700 1 -- 172.21.15.114:0/2943345927 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66580a8f30 con 0x55a66d54aec0 2024-05-27T07:42:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.370+0000 7f66477fe700 1 -- 172.21.15.114:0/2943345927 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f66580a90d0 con 0x55a66d54aec0 2024-05-27T07:42:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.370+0000 7f667d2e6b80 1 -- 172.21.15.114:0/2943345927 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f665c00e810 con 0x55a66d54aec0 2024-05-27T07:42:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.370+0000 7f66477fe700 1 -- 172.21.15.114:0/2943345927 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f66580c4080 con 0x55a66d54aec0 2024-05-27T07:42:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.370+0000 7f667d2e6b80 10 client.15960 fetch_fsmap learned FSMap version 25 2024-05-27T07:42:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.370+0000 7f667d2e6b80 1 -- 172.21.15.114:0/2943345927 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a66d5a17e0 con 0x55a66d54aec0 2024-05-27T07:42:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.371+0000 7f66477fe700 1 -- 172.21.15.114:0/2943345927 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f665c00e810 con 0x55a66d54aec0 2024-05-27T07:42:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.371+0000 7f66477fe700 1 -- 172.21.15.114:0/2943345927 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f66580a89a0 con 0x55a66d54aec0 2024-05-27T07:42:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.371+0000 7f667d2e6b80 10 client.15960 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:42:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.371+0000 7f667d2e6b80 10 client.15960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.371+0000 7f667d2e6b80 20 client.15960 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.371+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/2943345927 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a66d6ec350 0x55a66d6ee770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.371+0000 7f667d2e6b80 4 client.15960 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.371+0000 7f667d2e6b80 1 -- 172.21.15.114:0/2943345927 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a66d5f9d40 con 0x55a66d6ec350 2024-05-27T07:42:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.371+0000 7f66677fe700 1 --2- 172.21.15.114:0/2943345927 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a66d6ec350 0x55a66d6ee770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.372+0000 7f66677fe700 1 --2- 172.21.15.114:0/2943345927 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a66d6ec350 0x55a66d6ee770 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c67e901a247286f server_cookie=98208ceda188ee63 in_seq=0 out_seq=0 2024-05-27T07:42:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.372+0000 7f66477fe700 10 client.15960 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:55.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.378+0000 7f66477fe700 1 -- 172.21.15.114:0/2943345927 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a66d5f9d40 con 0x55a66d6ec350 2024-05-27T07:42:55.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.378+0000 7f66477fe700 10 client.15960 handle_command_reply: tid=0 2024-05-27T07:42:55.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.378+0000 7f667d2e6b80 1 client.15960 shutdown 2024-05-27T07:42:55.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.379+0000 7f667d2e6b80 1 -- 172.21.15.114:0/2943345927 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 msgr2=0x55a66d6d9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:55.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.379+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/2943345927 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6d9830 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f66580098c0 tx=0x7f66580b29f0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.379+0000 7f667d2e6b80 1 -- 172.21.15.114:0/2943345927 shutdown_connections 2024-05-27T07:42:55.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.379+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/2943345927 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a66d6ec350 0x55a66d6ee770 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.379+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/2943345927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6dc8d0 0x55a66d6da2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.379+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/2943345927 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6d9830 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.379+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/2943345927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6d3da0 0x55a66d6d9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.379+0000 7f667d2e6b80 1 -- 172.21.15.114:0/2943345927 >> 172.21.15.114:0/2943345927 conn(0x55a66d54d7f0 msgr2=0x55a66d6c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.379+0000 7f66477fe700 0 client.15960 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.379+0000 7f667d2e6b80 1 -- 172.21.15.114:0/2943345927 shutdown_connections 2024-05-27T07:42:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.379+0000 7f667d2e6b80 1 -- 172.21.15.114:0/2943345927 wait complete. 2024-05-27T07:42:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.379+0000 7f667d2e6b80 20 client.15960 trim_cache size 0 max 16384 2024-05-27T07:42:55.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.401+0000 7f667d2e6b80 1 Processor -- start 2024-05-27T07:42:55.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.401+0000 7f667d2e6b80 1 -- start start 2024-05-27T07:42:55.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.401+0000 7f667d2e6b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6befa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.401+0000 7f667d2e6b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6d3da0 0x55a66d6da0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.401+0000 7f667d2e6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6dc8d0 0x55a66d6e8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.401+0000 7f667d2e6b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f66681ac460 con 0x55a66d54aec0 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.401+0000 7f667d2e6b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f66681aece0 con 0x55a66d6d3da0 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.401+0000 7f667d2e6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f66681acad0 con 0x55a66d6dc8d0 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f6667fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6befa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f6667fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6befa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54200/0 (socket says 172.21.15.114:54200) 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f6667fff700 1 -- 172.21.15.114:0/505641739 learned_addr learned my addr 172.21.15.114:0/505641739 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f666ce6e700 1 --2- 172.21.15.114:0/505641739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6dc8d0 0x55a66d6e8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f66677fe700 1 --2- 172.21.15.114:0/505641739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6d3da0 0x55a66d6da0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f6667fff700 1 -- 172.21.15.114:0/505641739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6dc8d0 msgr2=0x55a66d6e8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f6667fff700 1 --2- 172.21.15.114:0/505641739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6dc8d0 0x55a66d6e8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f6667fff700 1 -- 172.21.15.114:0/505641739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6d3da0 msgr2=0x55a66d6da0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f6667fff700 1 --2- 172.21.15.114:0/505641739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6d3da0 0x55a66d6da0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f6667fff700 1 -- 172.21.15.114:0/505641739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a66d6c0b20 con 0x55a66d54aec0 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f66677fe700 1 --2- 172.21.15.114:0/505641739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6d3da0 0x55a66d6da0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:55.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f666ce6e700 1 --2- 172.21.15.114:0/505641739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6dc8d0 0x55a66d6e8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:55.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f6667fff700 1 --2- 172.21.15.114:0/505641739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6befa0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f665800c2d0 tx=0x7f66580a9bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94ac9444c9c48efe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:55.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f664dffb700 1 -- 172.21.15.114:0/505641739 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66580cc200 con 0x55a66d54aec0 2024-05-27T07:42:55.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f664dffb700 1 -- 172.21.15.114:0/505641739 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66580a89a0 con 0x55a66d54aec0 2024-05-27T07:42:55.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.402+0000 7f664dffb700 1 -- 172.21.15.114:0/505641739 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66580af340 con 0x55a66d54aec0 2024-05-27T07:42:55.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.403+0000 7f667d2e6b80 1 -- 172.21.15.114:0/505641739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 msgr2=0x55a66d6befa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:55.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.403+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/505641739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6befa0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f665800c2d0 tx=0x7f66580a9bd0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.403+0000 7f667d2e6b80 1 -- 172.21.15.114:0/505641739 shutdown_connections 2024-05-27T07:42:55.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.403+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/505641739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6dc8d0 0x55a66d6e8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.403+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/505641739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6befa0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.403+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/505641739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6d3da0 0x55a66d6da0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.403+0000 7f667d2e6b80 1 -- 172.21.15.114:0/505641739 >> 172.21.15.114:0/505641739 conn(0x55a66d54d7f0 msgr2=0x55a66d6d88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:55.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.403+0000 7f667d2e6b80 1 -- 172.21.15.114:0/505641739 shutdown_connections 2024-05-27T07:42:55.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.403+0000 7f667d2e6b80 1 -- 172.21.15.114:0/505641739 wait complete. 2024-05-27T07:42:55.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.404+0000 7f667d2e6b80 1 -- start start 2024-05-27T07:42:55.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.404+0000 7f667d2e6b80 1 Processor -- start 2024-05-27T07:42:55.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.404+0000 7f667d2e6b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6e0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.404+0000 7f667d2e6b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6d3da0 0x55a66d6e0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.404+0000 7f667d2e6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6dc8d0 0x55a66d6e0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.404+0000 7f667d2e6b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f66681ad1a0 con 0x55a66d54aec0 2024-05-27T07:42:55.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.404+0000 7f667d2e6b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f66681ae610 con 0x55a66d6d3da0 2024-05-27T07:42:55.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.404+0000 7f667d2e6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f66681ad870 con 0x55a66d6dc8d0 2024-05-27T07:42:55.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.404+0000 7f667d2e6b80 10 client.0 _ll_register_callbacks cb 0x55a66d57e630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:55.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.404+0000 7f6667fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6e0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.404+0000 7f666ce6e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6dc8d0 0x55a66d6e0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.404+0000 7f6667fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6e0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54212/0 (socket says 172.21.15.114:54212) 2024-05-27T07:42:55.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.404+0000 7f6667fff700 1 -- 172.21.15.114:0/733968320 learned_addr learned my addr 172.21.15.114:0/733968320 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:55.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f66677fe700 1 --2- 172.21.15.114:0/733968320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6d3da0 0x55a66d6e0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f666ce6e700 1 -- 172.21.15.114:0/733968320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6d3da0 msgr2=0x55a66d6e0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:55.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f666ce6e700 1 --2- 172.21.15.114:0/733968320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6d3da0 0x55a66d6e0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f666ce6e700 1 -- 172.21.15.114:0/733968320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 msgr2=0x55a66d6e0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:55.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f666ce6e700 1 --2- 172.21.15.114:0/733968320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6e0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f666ce6e700 1 -- 172.21.15.114:0/733968320 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a66d755d60 con 0x55a66d6dc8d0 2024-05-27T07:42:55.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f666ce6e700 1 --2- 172.21.15.114:0/733968320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6dc8d0 0x55a66d6e0da0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6660009fd0 tx=0x7f6660003dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87f08a016228ffeb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:55.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f66477fe700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:42:55.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f66477fe700 1 -- 172.21.15.114:0/733968320 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66600079d0 con 0x55a66d6dc8d0 2024-05-27T07:42:55.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f667d2e6b80 1 -- 172.21.15.114:0/733968320 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66580a89a0 con 0x55a66d6dc8d0 2024-05-27T07:42:55.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f66477fe700 1 -- 172.21.15.114:0/733968320 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f6660007b70 con 0x55a66d6dc8d0 2024-05-27T07:42:55.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f66477fe700 1 -- 172.21.15.114:0/733968320 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6660032050 con 0x55a66d6dc8d0 2024-05-27T07:42:55.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f667d2e6b80 10 client.25561 fetch_fsmap learned FSMap version 25 2024-05-27T07:42:55.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.405+0000 7f667d2e6b80 1 -- 172.21.15.114:0/733968320 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a66d756f40 con 0x55a66d6dc8d0 2024-05-27T07:42:55.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.406+0000 7f66477fe700 1 -- 172.21.15.114:0/733968320 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66580a89a0 con 0x55a66d6dc8d0 2024-05-27T07:42:55.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.406+0000 7f66477fe700 1 -- 172.21.15.114:0/733968320 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f666002d030 con 0x55a66d6dc8d0 2024-05-27T07:42:55.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.406+0000 7f667d2e6b80 10 client.25561 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:42:55.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.406+0000 7f667d2e6b80 10 client.25561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:55.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.406+0000 7f667d2e6b80 20 client.25561 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:55.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.406+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/733968320 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a66d7575f0 0x55a66d759a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:55.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.406+0000 7f667d2e6b80 4 client.25561 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:42:55.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.406+0000 7f667d2e6b80 1 -- 172.21.15.114:0/733968320 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a66d5f3520 con 0x55a66d7575f0 2024-05-27T07:42:55.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.406+0000 7f6667fff700 1 --2- 172.21.15.114:0/733968320 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a66d7575f0 0x55a66d759a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:55.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.406+0000 7f66477fe700 10 client.25561 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:55.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.407+0000 7f6667fff700 1 --2- 172.21.15.114:0/733968320 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a66d7575f0 0x55a66d759a10 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=655a4c4601238e71 server_cookie=55def05fe773fa6e in_seq=0 out_seq=0 2024-05-27T07:42:55.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f66477fe700 1 -- 172.21.15.114:0/733968320 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a66d5f3520 con 0x55a66d7575f0 2024-05-27T07:42:55.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f66477fe700 10 client.25561 handle_command_reply: tid=0 2024-05-27T07:42:55.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f667d2e6b80 1 client.25561 shutdown 2024-05-27T07:42:55.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f667d2e6b80 1 -- 172.21.15.114:0/733968320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6dc8d0 msgr2=0x55a66d6e0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:55.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/733968320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6dc8d0 0x55a66d6e0da0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6660009fd0 tx=0x7f6660003dd0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f667d2e6b80 1 -- 172.21.15.114:0/733968320 shutdown_connections 2024-05-27T07:42:55.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/733968320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a66d6dc8d0 0x55a66d6e0da0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/733968320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a66d54aec0 0x55a66d6e0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/733968320 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a66d7575f0 0x55a66d759a10 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f667d2e6b80 1 --2- 172.21.15.114:0/733968320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a66d6d3da0 0x55a66d6e0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f66477fe700 0 client.25561 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:55.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f667d2e6b80 1 -- 172.21.15.114:0/733968320 >> 172.21.15.114:0/733968320 conn(0x55a66d54d7f0 msgr2=0x55a66d6bed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:55.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f667d2e6b80 1 -- 172.21.15.114:0/733968320 shutdown_connections 2024-05-27T07:42:55.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f667d2e6b80 1 -- 172.21.15.114:0/733968320 wait complete. 2024-05-27T07:42:55.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.409+0000 7f667d2e6b80 20 client.25561 trim_cache size 0 max 16384 2024-05-27T07:42:55.411 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:42:55.411 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:42:55.411 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "ff6b8f9c-5f9b-49f2-ac89-2b3e0bf6c76c", 2024-05-27T07:42:55.411 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:42:55.411 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:42:55.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.410+0000 7f666e8d1700 1 -- 172.21.15.114:0/3357031260 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f66500b29f0 msgr2=0x7f66500b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:55.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.410+0000 7f666e8d1700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f66500b29f0 0x7f66500b4f00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f66681021e0 tx=0x7f665c009260 comp rx=0 tx=0).stop 2024-05-27T07:42:55.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.410+0000 7f666e8d1700 1 -- 172.21.15.114:0/3357031260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6668105510 msgr2=0x7f666819ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:55.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.410+0000 7f666e8d1700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6668105510 0x7f666819ad40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f665800e9a0 tx=0x7f665800ecb0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.410+0000 7f666e8d1700 1 -- 172.21.15.114:0/3357031260 shutdown_connections 2024-05-27T07:42:55.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.410+0000 7f666e8d1700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666810e8f0 0x7f666819f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.410+0000 7f666e8d1700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6668105510 0x7f666819ad40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.410+0000 7f666e8d1700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f66500b29f0 0x7f66500b4f00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.410+0000 7f666e8d1700 1 --2- 172.21.15.114:0/3357031260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6668105e60 0x7f666819b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:55.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.410+0000 7f666e8d1700 1 -- 172.21.15.114:0/3357031260 >> 172.21.15.114:0/3357031260 conn(0x7f66680fedb0 msgr2=0x7f66680ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:55.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.410+0000 7f666e8d1700 1 -- 172.21.15.114:0/3357031260 shutdown_connections 2024-05-27T07:42:55.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:55.410+0000 7f666e8d1700 1 -- 172.21.15.114:0/3357031260 wait complete. 2024-05-27T07:42:55.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:55 smithi163 ceph-mon[36451]: pgmap v100: 129 pgs: 129 active+clean; 39 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 327 op/s 2024-05-27T07:42:55.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:55 smithi186 ceph-mon[36647]: pgmap v100: 129 pgs: 129 active+clean; 39 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 327 op/s 2024-05-27T07:42:56.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:55 smithi114 ceph-mon[32038]: pgmap v100: 129 pgs: 129 active+clean; 39 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 327 op/s 2024-05-27T07:42:56.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 183687 14.81 MB/sec warmup 68 sec latency 50.681 ms 2024-05-27T07:42:56.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:42:56.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:56 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:42:56.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:56 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:56.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:56 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:42:56.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:56 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:56.912 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:56.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:56 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:42:56.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:56 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:56.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:56 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:42:56.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:56 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:56.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:56 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:42:56.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:56 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:56.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:56 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:42:56.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:56 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:57.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 185904 14.79 MB/sec warmup 69 sec latency 23.561 ms 2024-05-27T07:42:57.875 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:57 smithi163 ceph-mon[36451]: pgmap v101: 129 pgs: 129 active+clean; 46 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.2 MiB/s wr, 417 op/s 2024-05-27T07:42:57.875 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:57 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:42:57.877 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:57 smithi186 ceph-mon[36647]: pgmap v101: 129 pgs: 129 active+clean; 46 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.2 MiB/s wr, 417 op/s 2024-05-27T07:42:57.877 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:57 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:42:58.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:57 smithi114 ceph-mon[32038]: pgmap v101: 129 pgs: 129 active+clean; 46 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.2 MiB/s wr, 417 op/s 2024-05-27T07:42:58.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:57 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:42:58.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 188468 14.78 MB/sec warmup 70 sec latency 30.133 ms 2024-05-27T07:42:58.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:58 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:42:58.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:58 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:42:59.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:58 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:42:59.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 191380 14.77 MB/sec warmup 71 sec latency 57.174 ms 2024-05-27T07:42:59.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.678+0000 7fbe279a5700 1 -- 172.21.15.114:0/4103355688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe20108740 msgr2=0x7fbe201100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:59.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.678+0000 7fbe279a5700 1 --2- 172.21.15.114:0/4103355688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe20108740 0x7fbe201100a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fbe1800b0d0 tx=0x7fbe1800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.678+0000 7fbe279a5700 1 -- 172.21.15.114:0/4103355688 shutdown_connections 2024-05-27T07:42:59.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.678+0000 7fbe279a5700 1 --2- 172.21.15.114:0/4103355688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe20108740 0x7fbe201100a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.678+0000 7fbe279a5700 1 --2- 172.21.15.114:0/4103355688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe20103c50 0x7fbe20104060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.678+0000 7fbe279a5700 1 --2- 172.21.15.114:0/4103355688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe20107df0 0x7fbe20108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.678+0000 7fbe279a5700 1 -- 172.21.15.114:0/4103355688 >> 172.21.15.114:0/4103355688 conn(0x7fbe200fe6e0 msgr2=0x7fbe20100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:59.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.678+0000 7fbe279a5700 1 -- 172.21.15.114:0/4103355688 shutdown_connections 2024-05-27T07:42:59.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.678+0000 7fbe279a5700 1 -- 172.21.15.114:0/4103355688 wait complete. 2024-05-27T07:42:59.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.679+0000 7fbe279a5700 1 Processor -- start 2024-05-27T07:42:59.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.679+0000 7fbe279a5700 1 -- start start 2024-05-27T07:42:59.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.679+0000 7fbe279a5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe20103c50 0x7fbe2019bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.679+0000 7fbe279a5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe20107df0 0x7fbe2019c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.679+0000 7fbe279a5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe20108740 0x7fbe201a6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.679+0000 7fbe279a5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbe20113370 con 0x7fbe20108740 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.679+0000 7fbe279a5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe201131f0 con 0x7fbe20107df0 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.679+0000 7fbe279a5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbe201134f0 con 0x7fbe20103c50 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.679+0000 7fbe25f42700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe20108740 0x7fbe201a6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.679+0000 7fbe25f42700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe20108740 0x7fbe201a6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54244/0 (socket says 172.21.15.114:54244) 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.679+0000 7fbe25f42700 1 -- 172.21.15.114:0/859860400 learned_addr learned my addr 172.21.15.114:0/859860400 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.679+0000 7fbe24f40700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe20107df0 0x7fbe2019c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.679+0000 7fbe25741700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe20103c50 0x7fbe2019bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.680+0000 7fbe24f40700 1 -- 172.21.15.114:0/859860400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe20103c50 msgr2=0x7fbe2019bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.680+0000 7fbe24f40700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe20103c50 0x7fbe2019bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.680+0000 7fbe24f40700 1 -- 172.21.15.114:0/859860400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe20108740 msgr2=0x7fbe201a6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.680+0000 7fbe24f40700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe20108740 0x7fbe201a6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.680+0000 7fbe24f40700 1 -- 172.21.15.114:0/859860400 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2019ca20 con 0x7fbe20107df0 2024-05-27T07:42:59.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.680+0000 7fbe25f42700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe20108740 0x7fbe201a6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:59.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.680+0000 7fbe24f40700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe20107df0 0x7fbe2019c170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbe1000e9c0 tx=0x7fbe1000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:59.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.680+0000 7fbe167fc700 1 -- 172.21.15.114:0/859860400 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1000c9c0 con 0x7fbe20107df0 2024-05-27T07:42:59.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.680+0000 7fbe167fc700 1 -- 172.21.15.114:0/859860400 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe1000cb60 con 0x7fbe20107df0 2024-05-27T07:42:59.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.680+0000 7fbe279a5700 1 -- 172.21.15.114:0/859860400 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe20199a80 con 0x7fbe20107df0 2024-05-27T07:42:59.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.680+0000 7fbe167fc700 1 -- 172.21.15.114:0/859860400 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe10018680 con 0x7fbe20107df0 2024-05-27T07:42:59.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.680+0000 7fbe279a5700 1 -- 172.21.15.114:0/859860400 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe20199f70 con 0x7fbe20107df0 2024-05-27T07:42:59.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.684+0000 7fbe167fc700 1 -- 172.21.15.114:0/859860400 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fbe10010430 con 0x7fbe20107df0 2024-05-27T07:42:59.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe167fc700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbe0c0b29f0 0x7fbe0c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe363bab80 1 Processor -- start 2024-05-27T07:42:59.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe25741700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbe0c0b29f0 0x7fbe0c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe167fc700 1 -- 172.21.15.114:0/859860400 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7fbe100e2ad0 con 0x7fbe20107df0 2024-05-27T07:42:59.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe363bab80 1 -- start start 2024-05-27T07:42:59.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe363bab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026ec710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe363bab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026ecc50 0x563f026e6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe363bab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026e6f80 0x563f026e7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe363bab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563f02615c90 con 0x563f02568e40 2024-05-27T07:42:59.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe363bab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563f02605830 con 0x563f026ecc50 2024-05-27T07:42:59.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe363bab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563f02609870 con 0x563f026e6f80 2024-05-27T07:42:59.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe25741700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbe0c0b29f0 0x7fbe0c0b4f00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c010d80 tx=0x7fbe1c012040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:59.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe25741700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026ec710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe25741700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026ec710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54258/0 (socket says 172.21.15.114:54258) 2024-05-27T07:42:59.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe25741700 1 -- 172.21.15.114:0/2807362210 learned_addr learned my addr 172.21.15.114:0/2807362210 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:59.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe24f40700 1 --2- 172.21.15.114:0/2807362210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026ecc50 0x563f026e6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.685+0000 7fbe25f42700 1 --2- 172.21.15.114:0/2807362210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026e6f80 0x563f026e7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.686+0000 7fbe25741700 1 -- 172.21.15.114:0/2807362210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026e6f80 msgr2=0x563f026e7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:59.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.686+0000 7fbe25741700 1 --2- 172.21.15.114:0/2807362210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026e6f80 0x563f026e7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.686+0000 7fbe25741700 1 -- 172.21.15.114:0/2807362210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026ecc50 msgr2=0x563f026e6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:59.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.686+0000 7fbe25741700 1 --2- 172.21.15.114:0/2807362210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026ecc50 0x563f026e6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.686+0000 7fbe25741700 1 -- 172.21.15.114:0/2807362210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f025558d0 con 0x563f02568e40 2024-05-27T07:42:59.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.686+0000 7fbe24f40700 1 --2- 172.21.15.114:0/2807362210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026ecc50 0x563f026e6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:59.685 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:59 smithi114 ceph-mon[32038]: pgmap v102: 129 pgs: 129 active+clean; 46 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 330 op/s 2024-05-27T07:42:59.685 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:59 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:42:59.685 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:59 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:42:59.685 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:42:59 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:42:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.686+0000 7fbe25741700 1 --2- 172.21.15.114:0/2807362210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026ec710 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c01c520 tx=0x7fbe1c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5287db4b60a498 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.687+0000 7fbe0b7fe700 1 -- 172.21.15.114:0/2807362210 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c022040 con 0x563f02568e40 2024-05-27T07:42:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.687+0000 7fbe0b7fe700 1 -- 172.21.15.114:0/2807362210 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe1c004ce0 con 0x563f02568e40 2024-05-27T07:42:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.687+0000 7fbe0b7fe700 1 -- 172.21.15.114:0/2807362210 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c005650 con 0x563f02568e40 2024-05-27T07:42:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.688+0000 7fbe363bab80 1 -- 172.21.15.114:0/2807362210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 msgr2=0x563f026ec710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.688+0000 7fbe363bab80 1 --2- 172.21.15.114:0/2807362210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026ec710 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c01c520 tx=0x7fbe1c018000 comp rx=0 tx=0).stop 2024-05-27T07:42:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.688+0000 7fbe363bab80 1 -- 172.21.15.114:0/2807362210 shutdown_connections 2024-05-27T07:42:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.688+0000 7fbe363bab80 1 --2- 172.21.15.114:0/2807362210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026e6f80 0x563f026e7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.688+0000 7fbe363bab80 1 --2- 172.21.15.114:0/2807362210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026ec710 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.688+0000 7fbe363bab80 1 --2- 172.21.15.114:0/2807362210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026ecc50 0x563f026e6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.688+0000 7fbe363bab80 1 -- 172.21.15.114:0/2807362210 >> 172.21.15.114:0/2807362210 conn(0x563f025c3af0 msgr2=0x563f026e1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.688+0000 7fbe363bab80 1 -- 172.21.15.114:0/2807362210 shutdown_connections 2024-05-27T07:42:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.688+0000 7fbe363bab80 1 -- 172.21.15.114:0/2807362210 wait complete. 2024-05-27T07:42:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.689+0000 7fbe363bab80 1 -- start start 2024-05-27T07:42:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.689+0000 7fbe363bab80 1 Processor -- start 2024-05-27T07:42:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.689+0000 7fbe363bab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026f0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.689+0000 7fbe363bab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026e6f80 0x563f026f0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.689+0000 7fbe363bab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026ecc50 0x563f026f0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.689+0000 7fbe363bab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563f0250bfa0 con 0x563f02568e40 2024-05-27T07:42:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.689+0000 7fbe363bab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563f025a77c0 con 0x563f026ecc50 2024-05-27T07:42:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.689+0000 7fbe363bab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563f026f3740 con 0x563f026e6f80 2024-05-27T07:42:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.689+0000 7fbe363bab80 10 client.0 _ll_register_callbacks cb 0x563f025a0920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.689+0000 7fbe25741700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026f0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.689+0000 7fbe25741700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026f0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54264/0 (socket says 172.21.15.114:54264) 2024-05-27T07:42:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe25741700 1 -- 172.21.15.114:0/3811534734 learned_addr learned my addr 172.21.15.114:0/3811534734 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe24f40700 1 --2- 172.21.15.114:0/3811534734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026e6f80 0x563f026f0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe25f42700 1 --2- 172.21.15.114:0/3811534734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026ecc50 0x563f026f0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe24f40700 1 -- 172.21.15.114:0/3811534734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026ecc50 msgr2=0x563f026f0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe24f40700 1 --2- 172.21.15.114:0/3811534734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026ecc50 0x563f026f0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe24f40700 1 -- 172.21.15.114:0/3811534734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 msgr2=0x563f026f0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe24f40700 1 --2- 172.21.15.114:0/3811534734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026f0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe24f40700 1 -- 172.21.15.114:0/3811534734 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f025a4690 con 0x563f026e6f80 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe24f40700 1 --2- 172.21.15.114:0/3811534734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026e6f80 0x563f026f0880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbe100dfe70 tx=0x7fbe100a9f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea151ef0f8f406e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe08ff9700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe08ff9700 1 -- 172.21.15.114:0/3811534734 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe100a9510 con 0x563f026e6f80 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe08ff9700 1 -- 172.21.15.114:0/3811534734 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7fbe100cc2c0 con 0x563f026e6f80 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe363bab80 1 -- 172.21.15.114:0/3811534734 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe1c004ce0 con 0x563f026e6f80 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe08ff9700 1 -- 172.21.15.114:0/3811534734 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbe100aa100 con 0x563f026e6f80 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe363bab80 10 client.25573 fetch_fsmap learned FSMap version 25 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.690+0000 7fbe363bab80 1 -- 172.21.15.114:0/3811534734 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563f026fe200 con 0x563f026e6f80 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.691+0000 7fbe08ff9700 1 -- 172.21.15.114:0/3811534734 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe100bee50 con 0x563f026e6f80 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.691+0000 7fbe08ff9700 1 -- 172.21.15.114:0/3811534734 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fbe100a5370 con 0x563f026e6f80 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.691+0000 7fbe363bab80 10 client.25573 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:42:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.691+0000 7fbe363bab80 10 client.25573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.691+0000 7fbe363bab80 20 client.25573 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.691+0000 7fbe363bab80 1 --2- 172.21.15.114:0/3811534734 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563f02700790 0x563f02702bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.691+0000 7fbe363bab80 4 client.25573 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.691+0000 7fbe363bab80 1 -- 172.21.15.114:0/3811534734 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbe1c004ce0 con 0x563f02700790 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.691+0000 7fbe25741700 1 --2- 172.21.15.114:0/3811534734 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563f02700790 0x563f02702bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.692+0000 7fbe08ff9700 10 client.25573 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.692+0000 7fbe25741700 1 --2- 172.21.15.114:0/3811534734 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563f02700790 0x563f02702bb0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c02c51f83ad7e95 server_cookie=b18de9e6f1b99517 in_seq=0 out_seq=0 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.692+0000 7fbe08ff9700 1 -- 172.21.15.114:0/3811534734 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fbe1c004ce0 con 0x563f02700790 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.692+0000 7fbe08ff9700 10 client.25573 handle_command_reply: tid=0 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.692+0000 7fbe363bab80 1 client.25573 shutdown 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.693+0000 7fbe363bab80 1 -- 172.21.15.114:0/3811534734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026e6f80 msgr2=0x563f026f0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.693+0000 7fbe363bab80 1 --2- 172.21.15.114:0/3811534734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026e6f80 0x563f026f0880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbe100dfe70 tx=0x7fbe100a9f40 comp rx=0 tx=0).stop 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.693+0000 7fbe363bab80 1 -- 172.21.15.114:0/3811534734 shutdown_connections 2024-05-27T07:42:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.693+0000 7fbe363bab80 1 --2- 172.21.15.114:0/3811534734 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563f02700790 0x563f02702bb0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.693+0000 7fbe363bab80 1 --2- 172.21.15.114:0/3811534734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026ecc50 0x563f026f0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.693+0000 7fbe363bab80 1 --2- 172.21.15.114:0/3811534734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026f0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.693+0000 7fbe363bab80 1 --2- 172.21.15.114:0/3811534734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026e6f80 0x563f026f0880 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.693+0000 7fbe08ff9700 0 client.25573 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:59.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.693+0000 7fbe363bab80 1 -- 172.21.15.114:0/3811534734 >> 172.21.15.114:0/3811534734 conn(0x563f025c3af0 msgr2=0x563f026e5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:59.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.693+0000 7fbe363bab80 1 -- 172.21.15.114:0/3811534734 shutdown_connections 2024-05-27T07:42:59.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.693+0000 7fbe363bab80 1 -- 172.21.15.114:0/3811534734 wait complete. 2024-05-27T07:42:59.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.693+0000 7fbe363bab80 20 client.25573 trim_cache size 0 max 16384 2024-05-27T07:42:59.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.713+0000 7fbe363bab80 1 Processor -- start 2024-05-27T07:42:59.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.713+0000 7fbe363bab80 1 -- start start 2024-05-27T07:42:59.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe363bab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026ea680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe363bab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026e6f80 0x563f026ebe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe363bab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026ecc50 0x563f026f0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe363bab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbe201ac480 con 0x563f02568e40 2024-05-27T07:42:59.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe363bab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe201aed00 con 0x563f026e6f80 2024-05-27T07:42:59.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe363bab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbe201acaf0 con 0x563f026ecc50 2024-05-27T07:42:59.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe25741700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026ea680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe25f42700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026ecc50 0x563f026f0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe25741700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026ea680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54272/0 (socket says 172.21.15.114:54272) 2024-05-27T07:42:59.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe25741700 1 -- 172.21.15.114:0/3022691547 learned_addr learned my addr 172.21.15.114:0/3022691547 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:59.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe24f40700 1 --2- 172.21.15.114:0/3022691547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026e6f80 0x563f026ebe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe25f42700 1 -- 172.21.15.114:0/3022691547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026e6f80 msgr2=0x563f026ebe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:59.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe25f42700 1 --2- 172.21.15.114:0/3022691547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026e6f80 0x563f026ebe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe25f42700 1 -- 172.21.15.114:0/3022691547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 msgr2=0x563f026ea680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:59.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe25f42700 1 --2- 172.21.15.114:0/3022691547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026ea680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe25f42700 1 -- 172.21.15.114:0/3022691547 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f026ee8d0 con 0x563f026ecc50 2024-05-27T07:42:59.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe25741700 1 --2- 172.21.15.114:0/3022691547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026ea680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:59.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe24f40700 1 --2- 172.21.15.114:0/3022691547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026e6f80 0x563f026ebe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:59.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.714+0000 7fbe25f42700 1 --2- 172.21.15.114:0/3022691547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026ecc50 0x563f026f0c00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009fd0 tx=0x7fbe18005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17d611bfeef6bb9a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:59.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.715+0000 7fbe0affd700 1 -- 172.21.15.114:0/3022691547 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1802b070 con 0x563f026ecc50 2024-05-27T07:42:59.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.715+0000 7fbe0affd700 1 -- 172.21.15.114:0/3022691547 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe18003c00 con 0x563f026ecc50 2024-05-27T07:42:59.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.715+0000 7fbe0affd700 1 -- 172.21.15.114:0/3022691547 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe18005310 con 0x563f026ecc50 2024-05-27T07:42:59.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.715+0000 7fbe363bab80 1 -- 172.21.15.114:0/3022691547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026ecc50 msgr2=0x563f026f0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:59.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.715+0000 7fbe363bab80 1 --2- 172.21.15.114:0/3022691547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026ecc50 0x563f026f0c00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009fd0 tx=0x7fbe18005040 comp rx=0 tx=0).stop 2024-05-27T07:42:59.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.715+0000 7fbe363bab80 1 -- 172.21.15.114:0/3022691547 shutdown_connections 2024-05-27T07:42:59.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.715+0000 7fbe363bab80 1 --2- 172.21.15.114:0/3022691547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026ecc50 0x563f026f0c00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.715+0000 7fbe363bab80 1 --2- 172.21.15.114:0/3022691547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f02568e40 0x563f026ea680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.715+0000 7fbe363bab80 1 --2- 172.21.15.114:0/3022691547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f026e6f80 0x563f026ebe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.715+0000 7fbe363bab80 1 -- 172.21.15.114:0/3022691547 >> 172.21.15.114:0/3022691547 conn(0x563f025c3af0 msgr2=0x563f026f19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:59.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.716+0000 7fbe363bab80 1 -- 172.21.15.114:0/3022691547 shutdown_connections 2024-05-27T07:42:59.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.716+0000 7fbe363bab80 1 -- 172.21.15.114:0/3022691547 wait complete. 2024-05-27T07:42:59.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.716+0000 7fbe363bab80 1 -- start start 2024-05-27T07:42:59.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.716+0000 7fbe363bab80 1 Processor -- start 2024-05-27T07:42:59.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe363bab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f02568e40 0x563f026f91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe363bab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f026e6f80 0x563f026f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe363bab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026ecc50 0x563f026f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe363bab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbe201ad1c0 con 0x563f026e6f80 2024-05-27T07:42:59.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe363bab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe201ae630 con 0x563f02568e40 2024-05-27T07:42:59.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe363bab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbe201ad890 con 0x563f026ecc50 2024-05-27T07:42:59.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe25741700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f02568e40 0x563f026f91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe24f40700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f026e6f80 0x563f026f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe25741700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f02568e40 0x563f026f91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:41822/0 (socket says 172.21.15.114:41822) 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe25741700 1 -- 172.21.15.114:0/1575193199 learned_addr learned my addr 172.21.15.114:0/1575193199 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe363bab80 10 client.0 _ll_register_callbacks cb 0x563f025a0920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe25f42700 1 --2- 172.21.15.114:0/1575193199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026ecc50 0x563f026f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe25741700 1 -- 172.21.15.114:0/1575193199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026ecc50 msgr2=0x563f026f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe25741700 1 --2- 172.21.15.114:0/1575193199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026ecc50 0x563f026f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe25741700 1 -- 172.21.15.114:0/1575193199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f026e6f80 msgr2=0x563f026f59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe25741700 1 --2- 172.21.15.114:0/1575193199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f026e6f80 0x563f026f59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe25741700 1 -- 172.21.15.114:0/1575193199 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f026f6740 con 0x563f02568e40 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe25741700 1 --2- 172.21.15.114:0/1575193199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f02568e40 0x563f026f91c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c01cb10 tx=0x7fbe1c01cb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c45b3d7ef2db1d9d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.717+0000 7fbe08ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe08ff9700 1 -- 172.21.15.114:0/1575193199 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c03c070 con 0x563f02568e40 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe363bab80 1 -- 172.21.15.114:0/1575193199 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe18003c00 con 0x563f02568e40 2024-05-27T07:42:59.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe08ff9700 1 -- 172.21.15.114:0/1575193199 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7fbe1c005090 con 0x563f02568e40 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe08ff9700 1 -- 172.21.15.114:0/1575193199 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbe1c048050 con 0x563f02568e40 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe363bab80 10 client.25511 fetch_fsmap learned FSMap version 25 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe363bab80 1 -- 172.21.15.114:0/1575193199 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563f0276c1e0 con 0x563f02568e40 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe08ff9700 1 -- 172.21.15.114:0/1575193199 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe18003c00 con 0x563f02568e40 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe08ff9700 1 -- 172.21.15.114:0/1575193199 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fbe1c025030 con 0x563f02568e40 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe363bab80 10 client.25511 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe363bab80 10 client.25511 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe363bab80 20 client.25511 populate_metadata read hostname 'smithi114' 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe363bab80 1 --2- 172.21.15.114:0/1575193199 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563f0276c610 0x563f0276ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe363bab80 4 client.25511 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe363bab80 1 -- 172.21.15.114:0/1575193199 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563f026b29f0 con 0x563f0276c610 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.718+0000 7fbe24f40700 1 --2- 172.21.15.114:0/1575193199 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563f0276c610 0x563f0276ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.719+0000 7fbe08ff9700 10 client.25511 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:59.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.719+0000 7fbe24f40700 1 --2- 172.21.15.114:0/1575193199 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563f0276c610 0x563f0276ea30 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b634c753ad44625f server_cookie=b79b6447173e1b0c in_seq=0 out_seq=0 2024-05-27T07:42:59.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.721+0000 7fbe08ff9700 1 -- 172.21.15.114:0/1575193199 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563f026b29f0 con 0x563f0276c610 2024-05-27T07:42:59.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.721+0000 7fbe08ff9700 10 client.25511 handle_command_reply: tid=0 2024-05-27T07:42:59.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.721+0000 7fbe363bab80 1 client.25511 shutdown 2024-05-27T07:42:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.721+0000 7fbe363bab80 1 -- 172.21.15.114:0/1575193199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f02568e40 msgr2=0x563f026f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.721+0000 7fbe363bab80 1 --2- 172.21.15.114:0/1575193199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f02568e40 0x563f026f91c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c01cb10 tx=0x7fbe1c01cb40 comp rx=0 tx=0).stop 2024-05-27T07:42:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe363bab80 1 -- 172.21.15.114:0/1575193199 shutdown_connections 2024-05-27T07:42:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe363bab80 1 --2- 172.21.15.114:0/1575193199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f026ecc50 0x563f026f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe363bab80 1 --2- 172.21.15.114:0/1575193199 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563f0276c610 0x563f0276ea30 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe363bab80 1 --2- 172.21.15.114:0/1575193199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563f02568e40 0x563f026f91c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe363bab80 1 --2- 172.21.15.114:0/1575193199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f026e6f80 0x563f026f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe08ff9700 0 client.25511 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:42:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe363bab80 1 -- 172.21.15.114:0/1575193199 >> 172.21.15.114:0/1575193199 conn(0x563f025c3af0 msgr2=0x563f026ebb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe363bab80 1 -- 172.21.15.114:0/1575193199 shutdown_connections 2024-05-27T07:42:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe363bab80 1 -- 172.21.15.114:0/1575193199 wait complete. 2024-05-27T07:42:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe363bab80 20 client.25511 trim_cache size 0 max 16384 2024-05-27T07:42:59.723 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:42:59.723 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:42:59.723 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:42:59.723 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:42:59.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe279a5700 1 -- 172.21.15.114:0/859860400 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbe0c0b29f0 msgr2=0x7fbe0c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:59.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe279a5700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbe0c0b29f0 0x7fbe0c0b4f00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c010d80 tx=0x7fbe1c012040 comp rx=0 tx=0).stop 2024-05-27T07:42:59.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe279a5700 1 -- 172.21.15.114:0/859860400 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe20107df0 msgr2=0x7fbe2019c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:59.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe279a5700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe20107df0 0x7fbe2019c170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbe1000e9c0 tx=0x7fbe1000ecd0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe279a5700 1 -- 172.21.15.114:0/859860400 shutdown_connections 2024-05-27T07:42:59.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe279a5700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe20108740 0x7fbe201a6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe279a5700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe20103c50 0x7fbe2019bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe279a5700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbe0c0b29f0 0x7fbe0c0b4f00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe279a5700 1 --2- 172.21.15.114:0/859860400 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe20107df0 0x7fbe2019c170 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:59.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.722+0000 7fbe279a5700 1 -- 172.21.15.114:0/859860400 >> 172.21.15.114:0/859860400 conn(0x7fbe200fe6e0 msgr2=0x7fbe201074d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:59.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.723+0000 7fbe279a5700 1 -- 172.21.15.114:0/859860400 shutdown_connections 2024-05-27T07:42:59.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:42:59.723+0000 7fbe279a5700 1 -- 172.21.15.114:0/859860400 wait complete. 2024-05-27T07:42:59.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:59 smithi163 ceph-mon[36451]: pgmap v102: 129 pgs: 129 active+clean; 46 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 330 op/s 2024-05-27T07:42:59.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:59 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:42:59.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:59 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:42:59.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:42:59 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:42:59.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:59 smithi186 ceph-mon[36647]: pgmap v102: 129 pgs: 129 active+clean; 46 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 330 op/s 2024-05-27T07:42:59.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:59 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:42:59.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:59 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:42:59.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:42:59 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:43:00.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 193462 14.71 MB/sec warmup 72 sec latency 33.936 ms 2024-05-27T07:43:00.426 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:43:00.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:43:00.727 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:01.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 194803 14.62 MB/sec warmup 73 sec latency 73.120 ms 2024-05-27T07:43:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.621+0000 7f10229c7640 1 -- 172.21.15.114:0/1809995826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c106b00 msgr2=0x7f101c106f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.621+0000 7f10229c7640 1 --2- 172.21.15.114:0/1809995826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c106b00 0x7f101c106f30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1004009900 tx=0x7f100402a410 comp rx=0 tx=0).stop 2024-05-27T07:43:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.621+0000 7f10229c7640 1 -- 172.21.15.114:0/1809995826 shutdown_connections 2024-05-27T07:43:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.621+0000 7f10229c7640 1 --2- 172.21.15.114:0/1809995826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f101c1075c0 0x7f101c10ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.621+0000 7f10229c7640 1 --2- 172.21.15.114:0/1809995826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c106b00 0x7f101c106f30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.621+0000 7f10229c7640 1 --2- 172.21.15.114:0/1809995826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f101c102570 0x7f101c102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.621+0000 7f10229c7640 1 -- 172.21.15.114:0/1809995826 >> 172.21.15.114:0/1809995826 conn(0x7f101c0779d0 msgr2=0x7f101c077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.621+0000 7f10229c7640 1 -- 172.21.15.114:0/1809995826 shutdown_connections 2024-05-27T07:43:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.621+0000 7f10229c7640 1 -- 172.21.15.114:0/1809995826 wait complete. 2024-05-27T07:43:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.622+0000 7f10229c7640 1 Processor -- start 2024-05-27T07:43:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.622+0000 7f10229c7640 1 -- start start 2024-05-27T07:43:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.622+0000 7f10229c7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f101c102570 0x7f101c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.622+0000 7f10229c7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c106b00 0x7f101c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.622+0000 7f10229c7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f101c1075c0 0x7f101c19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.622+0000 7f10229c7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f101c111c50 con 0x7f101c1075c0 2024-05-27T07:43:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.622+0000 7f10229c7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f101c111ad0 con 0x7f101c102570 2024-05-27T07:43:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.622+0000 7f10229c7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f101c111dd0 con 0x7f101c106b00 2024-05-27T07:43:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.622+0000 7f1020f64640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f101c1075c0 0x7f101c19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.622+0000 7f101bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f101c102570 0x7f101c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f1013fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c106b00 0x7f101c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f1020f64640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f101c1075c0 0x7f101c19ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54294/0 (socket says 172.21.15.114:54294) 2024-05-27T07:43:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f1020f64640 1 -- 172.21.15.114:0/4171090708 learned_addr learned my addr 172.21.15.114:0/4171090708 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f101bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f101c102570 0x7f101c198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:41836/0 (socket says 172.21.15.114:41836) 2024-05-27T07:43:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f1020f64640 1 -- 172.21.15.114:0/4171090708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c106b00 msgr2=0x7f101c198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f1020f64640 1 --2- 172.21.15.114:0/4171090708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c106b00 0x7f101c198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f1020f64640 1 -- 172.21.15.114:0/4171090708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f101c102570 msgr2=0x7f101c198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f1020f64640 1 --2- 172.21.15.114:0/4171090708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f101c102570 0x7f101c198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f1020f64640 1 -- 172.21.15.114:0/4171090708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101c19d4d0 con 0x7f101c1075c0 2024-05-27T07:43:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f101bfff640 1 --2- 172.21.15.114:0/4171090708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f101c102570 0x7f101c198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f1020f64640 1 --2- 172.21.15.114:0/4171090708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f101c1075c0 0x7f101c19ce90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f101400afd0 tx=0x7f101400d870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f1019ffb640 1 -- 172.21.15.114:0/4171090708 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1014004d20 con 0x7f101c1075c0 2024-05-27T07:43:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f10229c7640 1 -- 172.21.15.114:0/4171090708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f101c19d7c0 con 0x7f101c1075c0 2024-05-27T07:43:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f1019ffb640 1 -- 172.21.15.114:0/4171090708 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f101400cb50 con 0x7f101c1075c0 2024-05-27T07:43:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f1019ffb640 1 -- 172.21.15.114:0/4171090708 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10140057d0 con 0x7f101c1075c0 2024-05-27T07:43:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.623+0000 7f10229c7640 1 -- 172.21.15.114:0/4171090708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f101c1a5140 con 0x7f101c1075c0 2024-05-27T07:43:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1019ffb640 1 -- 172.21.15.114:0/4171090708 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f101400cd00 con 0x7f101c1075c0 2024-05-27T07:43:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1029878740 1 Processor -- start 2024-05-27T07:43:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1029878740 1 -- start start 2024-05-27T07:43:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1029878740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 0x55ca25f8f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1029878740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25f8f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1029878740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 0x55ca25f8fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1029878740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ca25f01c30 con 0x55ca25bee1b0 2024-05-27T07:43:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1029878740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ca25ef17d0 con 0x55ca25bc52f0 2024-05-27T07:43:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1029878740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ca25ef5810 con 0x55ca25bcd4d0 2024-05-27T07:43:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1013fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 0x55ca25f8f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1013fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 0x55ca25f8f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54304/0 (socket says 172.21.15.114:54304) 2024-05-27T07:43:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f101bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25f8f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1013fff640 1 -- 172.21.15.114:0/2574706221 learned_addr learned my addr 172.21.15.114:0/2574706221 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1019ffb640 1 --2- 172.21.15.114:0/4171090708 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f10080b2950 0x7f10080b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.626+0000 7f1013fff640 1 --2- 172.21.15.114:0/2574706221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 0x55ca25f8fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.627+0000 7f1020f64640 1 --2- 172.21.15.114:0/4171090708 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f10080b2950 0x7f10080b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.627+0000 7f101bfff640 1 -- 172.21.15.114:0/2574706221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 msgr2=0x55ca25f8fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.627+0000 7f101bfff640 1 --2- 172.21.15.114:0/2574706221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 0x55ca25f8fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.627+0000 7f101bfff640 1 -- 172.21.15.114:0/2574706221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 msgr2=0x55ca25f8f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.627+0000 7f101bfff640 1 --2- 172.21.15.114:0/2574706221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 0x55ca25f8f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.627+0000 7f101bfff640 1 -- 172.21.15.114:0/2574706221 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca25c06210 con 0x55ca25bc52f0 2024-05-27T07:43:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.627+0000 7f1013fff640 1 --2- 172.21.15.114:0/2574706221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 0x55ca25f8f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.627+0000 7f1019ffb640 1 -- 172.21.15.114:0/4171090708 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f10140e3e90 con 0x7f101c1075c0 2024-05-27T07:43:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.627+0000 7f101bfff640 1 --2- 172.21.15.114:0/2574706221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25f8f720 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f100c009fd0 tx=0x7f100c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1c7c14f1bad17fa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.627+0000 7f1020f64640 1 --2- 172.21.15.114:0/4171090708 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f10080b2950 0x7f10080b4e60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f10140e12e0 tx=0x7f101409f780 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.627+0000 7f10127fc640 1 -- 172.21.15.114:0/2574706221 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c004030 con 0x55ca25bc52f0 2024-05-27T07:43:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.627+0000 7f10127fc640 1 -- 172.21.15.114:0/2574706221 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f100c0054d0 con 0x55ca25bc52f0 2024-05-27T07:43:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.627+0000 7f10127fc640 1 -- 172.21.15.114:0/2574706221 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c004bc0 con 0x55ca25bc52f0 2024-05-27T07:43:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.628+0000 7f1029878740 1 -- 172.21.15.114:0/2574706221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 msgr2=0x55ca25f8f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.628+0000 7f1029878740 1 --2- 172.21.15.114:0/2574706221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25f8f720 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f100c009fd0 tx=0x7f100c0118d0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.628+0000 7f1029878740 1 -- 172.21.15.114:0/2574706221 shutdown_connections 2024-05-27T07:43:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.628+0000 7f1029878740 1 --2- 172.21.15.114:0/2574706221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 0x55ca25f8fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.628+0000 7f1029878740 1 --2- 172.21.15.114:0/2574706221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25f8f720 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.628+0000 7f1029878740 1 --2- 172.21.15.114:0/2574706221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 0x55ca25f8f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.628+0000 7f1029878740 1 -- 172.21.15.114:0/2574706221 >> 172.21.15.114:0/2574706221 conn(0x55ca25be2300 msgr2=0x55ca25f8bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.628+0000 7f1029878740 1 -- 172.21.15.114:0/2574706221 shutdown_connections 2024-05-27T07:43:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.628+0000 7f1029878740 1 -- 172.21.15.114:0/2574706221 wait complete. 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.628+0000 7f1029878740 1 -- start start 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1029878740 1 Processor -- start 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1029878740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25f9bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1029878740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 0x55ca25f9c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1029878740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 0x55ca25fa1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1029878740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ca25bd7790 con 0x55ca25bee1b0 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1029878740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ca25ead890 con 0x55ca25bc52f0 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1029878740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ca25f9e770 con 0x55ca25bcd4d0 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1029878740 10 client.0 _ll_register_callbacks cb 0x55ca25c05a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1013fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 0x55ca25fa1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1013fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 0x55ca25fa1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54306/0 (socket says 172.21.15.114:54306) 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1013fff640 1 -- 172.21.15.114:0/354443862 learned_addr learned my addr 172.21.15.114:0/354443862 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f101bfff640 1 --2- 172.21.15.114:0/354443862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 0x55ca25f9c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1013fff640 1 --2- 172.21.15.114:0/354443862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25f9bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1013fff640 1 -- 172.21.15.114:0/354443862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 msgr2=0x55ca25f9c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1013fff640 1 --2- 172.21.15.114:0/354443862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 0x55ca25f9c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1013fff640 1 -- 172.21.15.114:0/354443862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 msgr2=0x55ca25f9bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.629+0000 7f1013fff640 1 --2- 172.21.15.114:0/354443862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25f9bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.630+0000 7f1013fff640 1 -- 172.21.15.114:0/354443862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca25cd2620 con 0x55ca25bee1b0 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.630+0000 7f1013fff640 1 --2- 172.21.15.114:0/354443862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 0x55ca25fa1a70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1004032fd0 tx=0x7f1004031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e859d858a3b1938d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.630+0000 7f0ffb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.630+0000 7f0ffb7fe640 1 -- 172.21.15.114:0/354443862 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1004035070 con 0x55ca25bee1b0 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.630+0000 7f1029878740 1 -- 172.21.15.114:0/354443862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f100c0054d0 con 0x55ca25bee1b0 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.630+0000 7f0ffb7fe640 1 -- 172.21.15.114:0/354443862 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f1004004480 con 0x55ca25bee1b0 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.630+0000 7f0ffb7fe640 1 -- 172.21.15.114:0/354443862 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f10040318f0 con 0x55ca25bee1b0 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.630+0000 7f1029878740 10 client.15996 fetch_fsmap learned FSMap version 25 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.630+0000 7f1029878740 1 -- 172.21.15.114:0/354443862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ca25faa9d0 con 0x55ca25bee1b0 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.630+0000 7f0ffb7fe640 1 -- 172.21.15.114:0/354443862 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f100c0054d0 con 0x55ca25bee1b0 2024-05-27T07:43:01.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.630+0000 7f0ffb7fe640 1 -- 172.21.15.114:0/354443862 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f1004004ba0 con 0x55ca25bee1b0 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.631+0000 7f1029878740 10 client.15996 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.631+0000 7f1029878740 10 client.15996 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.631+0000 7f1029878740 20 client.15996 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.631+0000 7f1029878740 1 --2- 172.21.15.114:0/354443862 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca25bacd80 0x55ca25fadea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.631+0000 7f1029878740 4 client.15996 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.631+0000 7f1029878740 1 -- 172.21.15.114:0/354443862 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca25f03b80 con 0x55ca25bacd80 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.631+0000 7f101bfff640 1 --2- 172.21.15.114:0/354443862 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca25bacd80 0x55ca25fadea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.631+0000 7f0ffb7fe640 10 client.15996 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.631+0000 7f101bfff640 1 --2- 172.21.15.114:0/354443862 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca25bacd80 0x55ca25fadea0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=399ca3ec4f2a2642 server_cookie=e6736adead107e56 in_seq=0 out_seq=0 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f0ffb7fe640 1 -- 172.21.15.114:0/354443862 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ca25f03b80 con 0x55ca25bacd80 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f0ffb7fe640 10 client.15996 handle_command_reply: tid=0 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f1029878740 1 client.15996 shutdown 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f1029878740 1 -- 172.21.15.114:0/354443862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 msgr2=0x55ca25fa1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f1029878740 1 --2- 172.21.15.114:0/354443862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 0x55ca25fa1a70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1004032fd0 tx=0x7f1004031400 comp rx=0 tx=0).stop 2024-05-27T07:43:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f1029878740 1 -- 172.21.15.114:0/354443862 shutdown_connections 2024-05-27T07:43:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f1029878740 1 --2- 172.21.15.114:0/354443862 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca25bacd80 0x55ca25fadea0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f1029878740 1 --2- 172.21.15.114:0/354443862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bee1b0 0x55ca25fa1a70 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f0ffb7fe640 0 client.15996 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f1029878740 1 --2- 172.21.15.114:0/354443862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 0x55ca25f9c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f1029878740 1 --2- 172.21.15.114:0/354443862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25f9bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f1029878740 1 -- 172.21.15.114:0/354443862 >> 172.21.15.114:0/354443862 conn(0x55ca25be2300 msgr2=0x55ca25f91080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f1029878740 1 -- 172.21.15.114:0/354443862 shutdown_connections 2024-05-27T07:43:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f1029878740 1 -- 172.21.15.114:0/354443862 wait complete. 2024-05-27T07:43:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.632+0000 7f1029878740 20 client.15996 trim_cache size 0 max 16384 2024-05-27T07:43:01.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.647+0000 7f1029878740 1 Processor -- start 2024-05-27T07:43:01.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f1029878740 1 -- start start 2024-05-27T07:43:01.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f1029878740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bacd80 0x55ca25f8fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f1029878740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25fa19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f1029878740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bcd4d0 0x55ca25fa1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f1029878740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f101c1ad140 con 0x55ca25bcd4d0 2024-05-27T07:43:01.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f1029878740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f101c1af9c0 con 0x55ca25bc52f0 2024-05-27T07:43:01.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f1029878740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f101c1ad7b0 con 0x55ca25bacd80 2024-05-27T07:43:01.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f1013fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bacd80 0x55ca25f8fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f101bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25fa19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f101bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25fa19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:41880/0 (socket says 172.21.15.114:41880) 2024-05-27T07:43:01.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f1013fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bcd4d0 0x55ca25fa1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f101bfff640 1 -- 172.21.15.114:0/2964676355 learned_addr learned my addr 172.21.15.114:0/2964676355 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:01.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f101bfff640 1 -- 172.21.15.114:0/2964676355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bacd80 msgr2=0x55ca25f8fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:01.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f101bfff640 1 --2- 172.21.15.114:0/2964676355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bacd80 0x55ca25f8fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f101bfff640 1 -- 172.21.15.114:0/2964676355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bcd4d0 msgr2=0x55ca25fa1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:01.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f101bfff640 1 --2- 172.21.15.114:0/2964676355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bcd4d0 0x55ca25fa1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f101bfff640 1 -- 172.21.15.114:0/2964676355 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca25f85e60 con 0x55ca25bc52f0 2024-05-27T07:43:01.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.648+0000 7f101bfff640 1 --2- 172.21.15.114:0/2964676355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25fa19a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f100c004630 tx=0x7f100c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4426aa35fc48221d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:01.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.649+0000 7f1011ffb640 1 -- 172.21.15.114:0/2964676355 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c00f880 con 0x55ca25bc52f0 2024-05-27T07:43:01.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.649+0000 7f1011ffb640 1 -- 172.21.15.114:0/2964676355 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f100c02b330 con 0x55ca25bc52f0 2024-05-27T07:43:01.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.649+0000 7f1011ffb640 1 -- 172.21.15.114:0/2964676355 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c0196b0 con 0x55ca25bc52f0 2024-05-27T07:43:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.649+0000 7f1029878740 1 -- 172.21.15.114:0/2964676355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 msgr2=0x55ca25fa19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.649+0000 7f1029878740 1 --2- 172.21.15.114:0/2964676355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25fa19a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f100c004630 tx=0x7f100c0118d0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.649+0000 7f1029878740 1 -- 172.21.15.114:0/2964676355 shutdown_connections 2024-05-27T07:43:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.649+0000 7f1029878740 1 --2- 172.21.15.114:0/2964676355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bcd4d0 0x55ca25fa1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.649+0000 7f1029878740 1 --2- 172.21.15.114:0/2964676355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bc52f0 0x55ca25fa19a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.649+0000 7f1029878740 1 --2- 172.21.15.114:0/2964676355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bacd80 0x55ca25f8fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.649+0000 7f1029878740 1 -- 172.21.15.114:0/2964676355 >> 172.21.15.114:0/2964676355 conn(0x55ca25be2300 msgr2=0x55ca25f9a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.649+0000 7f1029878740 1 -- 172.21.15.114:0/2964676355 shutdown_connections 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.649+0000 7f1029878740 1 -- 172.21.15.114:0/2964676355 wait complete. 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.650+0000 7f1029878740 1 -- start start 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.650+0000 7f1029878740 1 Processor -- start 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.650+0000 7f1029878740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bacd80 0x55ca25fa8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.650+0000 7f1029878740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bc52f0 0x55ca25fa34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.650+0000 7f1029878740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 0x55ca25fa3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.650+0000 7f1029878740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f101c1ade80 con 0x55ca25bc52f0 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.650+0000 7f1029878740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f101c1af2f0 con 0x55ca25bacd80 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.650+0000 7f1029878740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f101c1ae550 con 0x55ca25bcd4d0 2024-05-27T07:43:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.650+0000 7f1029878740 10 client.0 _ll_register_callbacks cb 0x7f10040317b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.650+0000 7f101bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bc52f0 0x55ca25fa34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.650+0000 7f101bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bc52f0 0x55ca25fa34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54328/0 (socket says 172.21.15.114:54328) 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.650+0000 7f101bfff640 1 -- 172.21.15.114:0/4107193609 learned_addr learned my addr 172.21.15.114:0/4107193609 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.650+0000 7f1013fff640 1 --2- 172.21.15.114:0/4107193609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 0x55ca25fa3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f1013fff640 1 --2- 172.21.15.114:0/4107193609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bacd80 0x55ca25fa8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f101bfff640 1 -- 172.21.15.114:0/4107193609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 msgr2=0x55ca25fa3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f101bfff640 1 --2- 172.21.15.114:0/4107193609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 0x55ca25fa3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f101bfff640 1 -- 172.21.15.114:0/4107193609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bacd80 msgr2=0x55ca25fa8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f101bfff640 1 --2- 172.21.15.114:0/4107193609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bacd80 0x55ca25fa8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f101bfff640 1 -- 172.21.15.114:0/4107193609 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca25fa3f40 con 0x55ca25bc52f0 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f101bfff640 1 --2- 172.21.15.114:0/4107193609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bc52f0 0x55ca25fa34c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f100c02bcb0 tx=0x7f100c011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb4c8688ed17184 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f0ffb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f0ffb7fe640 1 -- 172.21.15.114:0/4107193609 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c03a020 con 0x55ca25bc52f0 2024-05-27T07:43:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f0ffb7fe640 1 -- 172.21.15.114:0/4107193609 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f100c02b6f0 con 0x55ca25bc52f0 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f1029878740 1 -- 172.21.15.114:0/4107193609 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f100c02b330 con 0x55ca25bc52f0 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f0ffb7fe640 1 -- 172.21.15.114:0/4107193609 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f100c045050 con 0x55ca25bc52f0 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f1029878740 10 client.16002 fetch_fsmap learned FSMap version 25 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f1029878740 1 -- 172.21.15.114:0/4107193609 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ca26016200 con 0x55ca25bc52f0 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.651+0000 7f0ffb7fe640 1 -- 172.21.15.114:0/4107193609 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f100c02b330 con 0x55ca25bc52f0 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.652+0000 7f0ffb7fe640 1 -- 172.21.15.114:0/4107193609 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f100c036460 con 0x55ca25bc52f0 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.652+0000 7f1029878740 10 client.16002 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.652+0000 7f1029878740 10 client.16002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.652+0000 7f1029878740 20 client.16002 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.652+0000 7f1029878740 1 --2- 172.21.15.114:0/4107193609 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca25bee1b0 0x55ca26018780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.652+0000 7f1029878740 4 client.16002 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.652+0000 7f1029878740 1 -- 172.21.15.114:0/4107193609 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ca25efd360 con 0x55ca25bee1b0 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.652+0000 7f1013fff640 1 --2- 172.21.15.114:0/4107193609 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca25bee1b0 0x55ca26018780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.652+0000 7f0ffb7fe640 10 client.16002 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.652+0000 7f1013fff640 1 --2- 172.21.15.114:0/4107193609 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca25bee1b0 0x55ca26018780 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4860929956effc10 server_cookie=7ac40ba14ea0c75b in_seq=0 out_seq=0 2024-05-27T07:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.901+0000 7f0ffb7fe640 1 -- 172.21.15.114:0/4107193609 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55ca25efd360 con 0x55ca25bee1b0 2024-05-27T07:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.901+0000 7f0ffb7fe640 10 client.16002 handle_command_reply: tid=0 2024-05-27T07:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.901+0000 7f1029878740 1 client.16002 shutdown 2024-05-27T07:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.902+0000 7f1029878740 1 -- 172.21.15.114:0/4107193609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bc52f0 msgr2=0x55ca25fa34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.902+0000 7f1029878740 1 --2- 172.21.15.114:0/4107193609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bc52f0 0x55ca25fa34c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f100c02bcb0 tx=0x7f100c011dd0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.902+0000 7f1029878740 1 -- 172.21.15.114:0/4107193609 shutdown_connections 2024-05-27T07:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.902+0000 7f1029878740 1 --2- 172.21.15.114:0/4107193609 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca25bee1b0 0x55ca26018780 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.902+0000 7f1029878740 1 --2- 172.21.15.114:0/4107193609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca25bcd4d0 0x55ca25fa3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.902+0000 7f1029878740 1 --2- 172.21.15.114:0/4107193609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca25bc52f0 0x55ca25fa34c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.902+0000 7f1029878740 1 --2- 172.21.15.114:0/4107193609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca25bacd80 0x55ca25fa8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.902+0000 7f1029878740 1 -- 172.21.15.114:0/4107193609 >> 172.21.15.114:0/4107193609 conn(0x55ca25be2300 msgr2=0x55ca25f90ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.902+0000 7f0ffb7fe640 0 client.16002 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.902+0000 7f1029878740 1 -- 172.21.15.114:0/4107193609 shutdown_connections 2024-05-27T07:43:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.902+0000 7f1029878740 1 -- 172.21.15.114:0/4107193609 wait complete. 2024-05-27T07:43:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.902+0000 7f1029878740 20 client.16002 trim_cache size 0 max 16384 2024-05-27T07:43:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:43:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:43:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:43:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.903+0000 7f10229c7640 1 -- 172.21.15.114:0/4171090708 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f10080b2950 msgr2=0x7f10080b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.903+0000 7f10229c7640 1 --2- 172.21.15.114:0/4171090708 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f10080b2950 0x7f10080b4e60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f10140e12e0 tx=0x7f101409f780 comp rx=0 tx=0).stop 2024-05-27T07:43:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.903+0000 7f10229c7640 1 -- 172.21.15.114:0/4171090708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f101c1075c0 msgr2=0x7f101c19ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.903+0000 7f10229c7640 1 --2- 172.21.15.114:0/4171090708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f101c1075c0 0x7f101c19ce90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f101400afd0 tx=0x7f101400d870 comp rx=0 tx=0).stop 2024-05-27T07:43:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.903+0000 7f10229c7640 1 -- 172.21.15.114:0/4171090708 shutdown_connections 2024-05-27T07:43:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.903+0000 7f10229c7640 1 --2- 172.21.15.114:0/4171090708 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f10080b2950 0x7f10080b4e60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.903+0000 7f10229c7640 1 --2- 172.21.15.114:0/4171090708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f101c1075c0 0x7f101c19ce90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.903+0000 7f10229c7640 1 --2- 172.21.15.114:0/4171090708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c106b00 0x7f101c198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.903+0000 7f10229c7640 1 --2- 172.21.15.114:0/4171090708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f101c102570 0x7f101c198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.903+0000 7f10229c7640 1 -- 172.21.15.114:0/4171090708 >> 172.21.15.114:0/4171090708 conn(0x7f101c0779d0 msgr2=0x7f101c100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.903+0000 7f10229c7640 1 -- 172.21.15.114:0/4171090708 shutdown_connections 2024-05-27T07:43:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:01.903+0000 7f10229c7640 1 -- 172.21.15.114:0/4171090708 wait complete. 2024-05-27T07:43:01.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:01 smithi163 ceph-mon[36451]: pgmap v103: 129 pgs: 129 active+clean; 51 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 458 op/s 2024-05-27T07:43:01.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:01 smithi186 ceph-mon[36647]: pgmap v103: 129 pgs: 129 active+clean; 51 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 458 op/s 2024-05-27T07:43:02.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:01 smithi114 ceph-mon[32038]: pgmap v103: 129 pgs: 129 active+clean; 51 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 458 op/s 2024-05-27T07:43:02.241 INFO:tasks.workunit.client.0.smithi114.stdout: 1 195640 14.50 MB/sec warmup 74 sec latency 141.438 ms 2024-05-27T07:43:03.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.147+0000 7f2f2a150700 1 -- 172.21.15.114:0/3286577487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f241081f0 msgr2=0x7f2f24108600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:03.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.147+0000 7f2f2a150700 1 --2- 172.21.15.114:0/3286577487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f241081f0 0x7f2f24108600 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009990 tx=0x7f2f14009ca0 comp rx=0 tx=0).stop 2024-05-27T07:43:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.147+0000 7f2f2a150700 1 -- 172.21.15.114:0/3286577487 shutdown_connections 2024-05-27T07:43:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.147+0000 7f2f2a150700 1 --2- 172.21.15.114:0/3286577487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2f241042e0 0x7f2f24110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.147+0000 7f2f2a150700 1 --2- 172.21.15.114:0/3286577487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f241081f0 0x7f2f24108600 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.147+0000 7f2f2a150700 1 --2- 172.21.15.114:0/3286577487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24103990 0x7f2f24103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.147+0000 7f2f2a150700 1 -- 172.21.15.114:0/3286577487 >> 172.21.15.114:0/3286577487 conn(0x7f2f240fe510 msgr2=0x7f2f24100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.147+0000 7f2f2a150700 1 -- 172.21.15.114:0/3286577487 shutdown_connections 2024-05-27T07:43:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.147+0000 7f2f2a150700 1 -- 172.21.15.114:0/3286577487 wait complete. 2024-05-27T07:43:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.148+0000 7f2f2a150700 1 Processor -- start 2024-05-27T07:43:03.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.148+0000 7f2f2a150700 1 -- start start 2024-05-27T07:43:03.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.148+0000 7f2f2a150700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24103990 0x7f2f2419ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:03.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.148+0000 7f2f2a150700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2f241042e0 0x7f2f2419b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:03.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.148+0000 7f2f2a150700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f241081f0 0x7f2f2419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:03.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.148+0000 7f2f2a150700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f24114470 con 0x7f2f241081f0 2024-05-27T07:43:03.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.148+0000 7f2f2a150700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2f241142f0 con 0x7f2f241042e0 2024-05-27T07:43:03.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.148+0000 7f2f2a150700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2f241145f0 con 0x7f2f24103990 2024-05-27T07:43:03.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.148+0000 7f2f23fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f241081f0 0x7f2f2419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:03.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.148+0000 7f2f22ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2f241042e0 0x7f2f2419b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.148+0000 7f2f23fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f241081f0 0x7f2f2419f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54350/0 (socket says 172.21.15.114:54350) 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.148+0000 7f2f23fff700 1 -- 172.21.15.114:0/1442935250 learned_addr learned my addr 172.21.15.114:0/1442935250 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.148+0000 7f2f237fe700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24103990 0x7f2f2419ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.149+0000 7f2f23fff700 1 -- 172.21.15.114:0/1442935250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24103990 msgr2=0x7f2f2419ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.149+0000 7f2f23fff700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24103990 0x7f2f2419ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.149+0000 7f2f23fff700 1 -- 172.21.15.114:0/1442935250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2f241042e0 msgr2=0x7f2f2419b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.149+0000 7f2f23fff700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2f241042e0 0x7f2f2419b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.149+0000 7f2f23fff700 1 -- 172.21.15.114:0/1442935250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2419fc70 con 0x7f2f241081f0 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.149+0000 7f2f237fe700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24103990 0x7f2f2419ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.149+0000 7f2f22ffd700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2f241042e0 0x7f2f2419b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.149+0000 7f2f23fff700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f241081f0 0x7f2f2419f5d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c007ee0 tx=0x7f2f1c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.149+0000 7f2f20ff9700 1 -- 172.21.15.114:0/1442935250 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c004bb0 con 0x7f2f241081f0 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.149+0000 7f2f2a150700 1 -- 172.21.15.114:0/1442935250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2419ff60 con 0x7f2f241081f0 2024-05-27T07:43:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.149+0000 7f2f20ff9700 1 -- 172.21.15.114:0/1442935250 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f1c004d50 con 0x7f2f241081f0 2024-05-27T07:43:03.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.149+0000 7f2f20ff9700 1 -- 172.21.15.114:0/1442935250 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c005920 con 0x7f2f241081f0 2024-05-27T07:43:03.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.149+0000 7f2f2a150700 1 -- 172.21.15.114:0/1442935250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f241109a0 con 0x7f2f241081f0 2024-05-27T07:43:03.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.150+0000 7f2f2a150700 1 -- 172.21.15.114:0/1442935250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f241049a0 con 0x7f2f241081f0 2024-05-27T07:43:03.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.152+0000 7f2f20ff9700 1 -- 172.21.15.114:0/1442935250 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f2f1c01a020 con 0x7f2f241081f0 2024-05-27T07:43:03.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.153+0000 7f2f20ff9700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2f0c0b29f0 0x7f2f0c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:03.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.153+0000 7f2f237fe700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2f0c0b29f0 0x7f2f0c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:03.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.153+0000 7f2f20ff9700 1 -- 172.21.15.114:0/1442935250 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f2f1c0e26c0 con 0x7f2f241081f0 2024-05-27T07:43:03.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.153+0000 7f2f237fe700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2f0c0b29f0 0x7f2f0c0b4f00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f140059d0 tx=0x7f2f14005960 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:03.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.157+0000 7f2f20ff9700 1 -- 172.21.15.114:0/1442935250 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2f1c010070 con 0x7f2f241081f0 2024-05-27T07:43:03.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 198646 14.54 MB/sec warmup 75 sec latency 29.718 ms 2024-05-27T07:43:03.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.435+0000 7f2f2a150700 1 -- 172.21.15.114:0/1442935250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f2410b790 con 0x7f2f241081f0 2024-05-27T07:43:03.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.435+0000 7f2f20ff9700 1 -- 172.21.15.114:0/1442935250 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11827 (secure 0 0 0) 0x7f2f1c0a9cf0 con 0x7f2f241081f0 2024-05-27T07:43:03.434 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:43:03.434 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:42:26.549486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25397":{"gid":25397,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/1365473025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":1365473025},{"type":"v1","addr":"172.21.15.163:6839","nonce":1365473025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:43:03.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.438+0000 7f2f2a150700 1 -- 172.21.15.114:0/1442935250 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2f0c0b29f0 msgr2=0x7f2f0c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:03.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.438+0000 7f2f2a150700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2f0c0b29f0 0x7f2f0c0b4f00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f140059d0 tx=0x7f2f14005960 comp rx=0 tx=0).stop 2024-05-27T07:43:03.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.438+0000 7f2f2a150700 1 -- 172.21.15.114:0/1442935250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f241081f0 msgr2=0x7f2f2419f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:03.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.438+0000 7f2f2a150700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f241081f0 0x7f2f2419f5d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c007ee0 tx=0x7f2f1c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:43:03.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.438+0000 7f2f2a150700 1 -- 172.21.15.114:0/1442935250 shutdown_connections 2024-05-27T07:43:03.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.438+0000 7f2f2a150700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f241081f0 0x7f2f2419f5d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:03.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.438+0000 7f2f2a150700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24103990 0x7f2f2419ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:03.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.438+0000 7f2f2a150700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2f0c0b29f0 0x7f2f0c0b4f00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:03.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.438+0000 7f2f2a150700 1 --2- 172.21.15.114:0/1442935250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2f241042e0 0x7f2f2419b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:03.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.438+0000 7f2f2a150700 1 -- 172.21.15.114:0/1442935250 >> 172.21.15.114:0/1442935250 conn(0x7f2f240fe510 msgr2=0x7f2f24109680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:03.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.438+0000 7f2f2a150700 1 -- 172.21.15.114:0/1442935250 shutdown_connections 2024-05-27T07:43:03.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:03.438+0000 7f2f2a150700 1 -- 172.21.15.114:0/1442935250 wait complete. 2024-05-27T07:43:03.437 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 25 2024-05-27T07:43:03.589 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7c9d855700 time 2024-05-27T07:43:03.277268+0000 2024-05-27T07:43:03.590 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:43:03.590 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:03.590 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7cb009aba6] 2024-05-27T07:43:03.590 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cb009add1] 2024-05-27T07:43:03.590 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b6c249e23d] 2024-05-27T07:43:03.590 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b6c2489971] 2024-05-27T07:43:03.590 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b6c2487fa0] 2024-05-27T07:43:03.590 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b6c2488651] 2024-05-27T07:43:03.590 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b6c241d347] 2024-05-27T07:43:03.590 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b6c211987e] 2024-05-27T07:43:03.590 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b6c211ee94] 2024-05-27T07:43:03.590 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b6c2129969] 2024-05-27T07:43:03.591 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b6c24c248c] 2024-05-27T07:43:03.591 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b6c24cb9cb] 2024-05-27T07:43:03.591 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 13: (Thread::entry_wrapper()+0x43) [0x7f7cb006972d] 2024-05-27T07:43:03.591 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7cb0069749] 2024-05-27T07:43:03.591 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7caec0d1ca] 2024-05-27T07:43:03.591 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 16: clone() 2024-05-27T07:43:03.591 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: *** Caught signal (Aborted) ** 2024-05-27T07:43:03.591 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: in thread 7f7c9d855700 thread_name:md_log_replay 2024-05-27T07:43:03.592 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.278+0000 7f7c9d855700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7c9d855700 time 2024-05-27T07:43:03.277268+0000 2024-05-27T07:43:03.592 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:43:03.592 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.592 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:03.592 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7cb009aba6] 2024-05-27T07:43:03.592 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cb009add1] 2024-05-27T07:43:03.593 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b6c249e23d] 2024-05-27T07:43:03.593 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b6c2489971] 2024-05-27T07:43:03.593 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b6c2487fa0] 2024-05-27T07:43:03.593 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b6c2488651] 2024-05-27T07:43:03.593 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b6c241d347] 2024-05-27T07:43:03.593 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b6c211987e] 2024-05-27T07:43:03.593 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b6c211ee94] 2024-05-27T07:43:03.593 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b6c2129969] 2024-05-27T07:43:03.593 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b6c24c248c] 2024-05-27T07:43:03.593 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b6c24cb9cb] 2024-05-27T07:43:03.593 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 13: (Thread::entry_wrapper()+0x43) [0x7f7cb006972d] 2024-05-27T07:43:03.593 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7cb0069749] 2024-05-27T07:43:03.593 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7caec0d1ca] 2024-05-27T07:43:03.594 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 16: clone() 2024-05-27T07:43:03.594 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.594 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:03.594 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7caec17d20] 2024-05-27T07:43:03.594 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2: gsignal() 2024-05-27T07:43:03.594 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 3: abort() 2024-05-27T07:43:03.594 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7cb009aced] 2024-05-27T07:43:03.594 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cb009add1] 2024-05-27T07:43:03.594 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b6c249e23d] 2024-05-27T07:43:03.594 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b6c2489971] 2024-05-27T07:43:03.594 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b6c2487fa0] 2024-05-27T07:43:03.594 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b6c2488651] 2024-05-27T07:43:03.594 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b6c241d347] 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b6c211987e] 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b6c211ee94] 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b6c2129969] 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b6c24c248c] 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b6c24cb9cb] 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 16: (Thread::entry_wrapper()+0x43) [0x7f7cb006972d] 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7cb0069749] 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7caec0d1ca] 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 19: clone() 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.280+0000 7f7c9d855700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: in thread 7f7c9d855700 thread_name:md_log_replay 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7caec17d20] 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2: gsignal() 2024-05-27T07:43:03.595 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 3: abort() 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7cb009aced] 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cb009add1] 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b6c249e23d] 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b6c2489971] 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b6c2487fa0] 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b6c2488651] 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b6c241d347] 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b6c211987e] 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b6c211ee94] 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b6c2129969] 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b6c24c248c] 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b6c24cb9cb] 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 16: (Thread::entry_wrapper()+0x43) [0x7f7cb006972d] 2024-05-27T07:43:03.596 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7cb0069749] 2024-05-27T07:43:03.597 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7caec0d1ca] 2024-05-27T07:43:03.597 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 19: clone() 2024-05-27T07:43:03.597 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:43:03.597 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.597 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: -1> 2024-05-27T07:43:03.278+0000 7f7c9d855700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7c9d855700 time 2024-05-27T07:43:03.277268+0000 2024-05-27T07:43:03.597 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:43:03.597 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.597 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:03.597 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7cb009aba6] 2024-05-27T07:43:03.597 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cb009add1] 2024-05-27T07:43:03.597 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b6c249e23d] 2024-05-27T07:43:03.597 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b6c2489971] 2024-05-27T07:43:03.597 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b6c2487fa0] 2024-05-27T07:43:03.598 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b6c2488651] 2024-05-27T07:43:03.598 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b6c241d347] 2024-05-27T07:43:03.598 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b6c211987e] 2024-05-27T07:43:03.598 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b6c211ee94] 2024-05-27T07:43:03.598 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b6c2129969] 2024-05-27T07:43:03.598 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b6c24c248c] 2024-05-27T07:43:03.598 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b6c24cb9cb] 2024-05-27T07:43:03.598 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 13: (Thread::entry_wrapper()+0x43) [0x7f7cb006972d] 2024-05-27T07:43:03.598 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7cb0069749] 2024-05-27T07:43:03.598 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7caec0d1ca] 2024-05-27T07:43:03.598 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 16: clone() 2024-05-27T07:43:03.598 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.598 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 0> 2024-05-27T07:43:03.280+0000 7f7c9d855700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: in thread 7f7c9d855700 thread_name:md_log_replay 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7caec17d20] 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2: gsignal() 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 3: abort() 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7cb009aced] 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cb009add1] 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b6c249e23d] 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b6c2489971] 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b6c2487fa0] 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b6c2488651] 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b6c241d347] 2024-05-27T07:43:03.599 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b6c211987e] 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b6c211ee94] 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b6c2129969] 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b6c24c248c] 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b6c24cb9cb] 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 16: (Thread::entry_wrapper()+0x43) [0x7f7cb006972d] 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7cb0069749] 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7caec0d1ca] 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 19: clone() 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: -1> 2024-05-27T07:43:03.278+0000 7f7c9d855700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7c9d855700 time 2024-05-27T07:43:03.277268+0000 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.600 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7cb009aba6] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cb009add1] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b6c249e23d] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b6c2489971] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b6c2487fa0] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b6c2488651] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b6c241d347] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b6c211987e] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b6c211ee94] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b6c2129969] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b6c24c248c] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b6c24cb9cb] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 13: (Thread::entry_wrapper()+0x43) [0x7f7cb006972d] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7cb0069749] 2024-05-27T07:43:03.601 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7caec0d1ca] 2024-05-27T07:43:03.602 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 16: clone() 2024-05-27T07:43:03.602 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.602 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 0> 2024-05-27T07:43:03.280+0000 7f7c9d855700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:43:03.602 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: in thread 7f7c9d855700 thread_name:md_log_replay 2024-05-27T07:43:03.602 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.602 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:03.602 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7caec17d20] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2: gsignal() 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 3: abort() 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7cb009aced] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cb009add1] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b6c249e23d] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b6c2489971] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b6c2487fa0] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b6c2488651] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b6c241d347] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b6c211987e] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b6c211ee94] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b6c2129969] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b6c24c248c] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b6c24cb9cb] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 16: (Thread::entry_wrapper()+0x43) [0x7f7cb006972d] 2024-05-27T07:43:03.603 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7cb0069749] 2024-05-27T07:43:03.604 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7caec0d1ca] 2024-05-27T07:43:03.604 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 19: clone() 2024-05-27T07:43:03.604 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:43:03.604 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70355]: 2024-05-27T07:43:03.604 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:03 smithi163 podman[70566]: 2024-05-27 07:43:03.344645371 +0000 UTC m=+0.019852394 container died 5b5c72894a08bcbddc7c182f94a942198b1164b2a6e5980ce8f4f5e8cce56766 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:43:03.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:03 smithi114 ceph-mon[32038]: pgmap v104: 129 pgs: 129 active+clean; 51 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 217 op/s 2024-05-27T07:43:03.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:03 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1442935250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:03.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-mon[36451]: pgmap v104: 129 pgs: 129 active+clean; 51 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 217 op/s 2024-05-27T07:43:03.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:03 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1442935250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:03.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:03 smithi186 ceph-mon[36647]: pgmap v104: 129 pgs: 129 active+clean; 51 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 217 op/s 2024-05-27T07:43:03.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:03 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1442935250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:04.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 201107 14.47 MB/sec warmup 76 sec latency 15.038 ms 2024-05-27T07:43:04.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:43:04.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:04 smithi163 podman[70566]: 2024-05-27 07:43:04.043249381 +0000 UTC m=+0.718456401 container remove 5b5c72894a08bcbddc7c182f94a942198b1164b2a6e5980ce8f4f5e8cce56766 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-27T07:43:04.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:04 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:43:04.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:04 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-05-27T07:43:04.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:04 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Consumed 3.420s CPU time. 2024-05-27T07:43:04.656 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:05.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 203621 14.49 MB/sec warmup 77 sec latency 14.408 ms 2024-05-27T07:43:05.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:05 smithi114 ceph-mon[32038]: pgmap v105: 129 pgs: 129 active+clean; 51 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 217 op/s 2024-05-27T07:43:05.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:05 smithi163 ceph-mon[36451]: pgmap v105: 129 pgs: 129 active+clean; 51 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 217 op/s 2024-05-27T07:43:05.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:05 smithi186 ceph-mon[36647]: pgmap v105: 129 pgs: 129 active+clean; 51 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 217 op/s 2024-05-27T07:43:06.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 206121 14.49 MB/sec warmup 78 sec latency 59.473 ms 2024-05-27T07:43:06.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.484+0000 7f51ebec1700 1 -- 172.21.15.114:0/3152028706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e410a920 msgr2=0x7f51e410ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:06.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.484+0000 7f51ebec1700 1 --2- 172.21.15.114:0/3152028706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e410a920 0x7f51e410ad30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f51e0009990 tx=0x7f51e0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.485+0000 7f51ebec1700 1 -- 172.21.15.114:0/3152028706 shutdown_connections 2024-05-27T07:43:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.485+0000 7f51ebec1700 1 --2- 172.21.15.114:0/3152028706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51e411d920 0x7f51e411fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.485+0000 7f51ebec1700 1 --2- 172.21.15.114:0/3152028706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e410a920 0x7f51e410ad30 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.485+0000 7f51ebec1700 1 --2- 172.21.15.114:0/3152028706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51e410b270 0x7f51e411d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.485+0000 7f51ebec1700 1 -- 172.21.15.114:0/3152028706 >> 172.21.15.114:0/3152028706 conn(0x7f51e406f4c0 msgr2=0x7f51e406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.486+0000 7f51ebec1700 1 -- 172.21.15.114:0/3152028706 shutdown_connections 2024-05-27T07:43:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.486+0000 7f51ebec1700 1 -- 172.21.15.114:0/3152028706 wait complete. 2024-05-27T07:43:06.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.487+0000 7f51ebec1700 1 Processor -- start 2024-05-27T07:43:06.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.487+0000 7f51ebec1700 1 -- start start 2024-05-27T07:43:06.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.488+0000 7f51ebec1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51e410a920 0x7f51e41af780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.488+0000 7f51ebec1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51e410b270 0x7f51e41afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.488+0000 7f51ebec1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e411d920 0x7f51e41a99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.488+0000 7f51ebec1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f51e4123370 con 0x7f51e411d920 2024-05-27T07:43:06.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.488+0000 7f51ebec1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f51e41231f0 con 0x7f51e410a920 2024-05-27T07:43:06.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.488+0000 7f51ebec1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51e41234f0 con 0x7f51e410b270 2024-05-27T07:43:06.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.489+0000 7f51ea45e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e411d920 0x7f51e41a99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.489+0000 7f51ea45e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e411d920 0x7f51e41a99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40696/0 (socket says 172.21.15.114:40696) 2024-05-27T07:43:06.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.489+0000 7f51ea45e700 1 -- 172.21.15.114:0/3598938662 learned_addr learned my addr 172.21.15.114:0/3598938662 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:06.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.489+0000 7f51e9c5d700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51e410a920 0x7f51e41af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.490+0000 7f51e945c700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51e410b270 0x7f51e41afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.490+0000 7f51ea45e700 1 -- 172.21.15.114:0/3598938662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51e410b270 msgr2=0x7f51e41afcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:06.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.490+0000 7f51ea45e700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51e410b270 0x7f51e41afcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.490+0000 7f51ea45e700 1 -- 172.21.15.114:0/3598938662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51e410a920 msgr2=0x7f51e41af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:06.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.490+0000 7f51ea45e700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51e410a920 0x7f51e41af780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.490+0000 7f51ea45e700 1 -- 172.21.15.114:0/3598938662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e41aa050 con 0x7f51e411d920 2024-05-27T07:43:06.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.490+0000 7f51ea45e700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e411d920 0x7f51e41a99a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0094f0 tx=0x7f51dc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:06.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.491+0000 7f51daffd700 1 -- 172.21.15.114:0/3598938662 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc004db0 con 0x7f51e411d920 2024-05-27T07:43:06.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.491+0000 7f51daffd700 1 -- 172.21.15.114:0/3598938662 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51dc00db60 con 0x7f51e411d920 2024-05-27T07:43:06.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.491+0000 7f51daffd700 1 -- 172.21.15.114:0/3598938662 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc015400 con 0x7f51e411d920 2024-05-27T07:43:06.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.491+0000 7f51ebec1700 1 -- 172.21.15.114:0/3598938662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e41aa340 con 0x7f51e411d920 2024-05-27T07:43:06.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.491+0000 7f51ebec1700 1 -- 172.21.15.114:0/3598938662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e41b6610 con 0x7f51e411d920 2024-05-27T07:43:06.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.497+0000 7f51daffd700 1 -- 172.21.15.114:0/3598938662 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f51dc0155c0 con 0x7f51e411d920 2024-05-27T07:43:06.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.498+0000 7f51fa8d6b80 1 Processor -- start 2024-05-27T07:43:06.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.498+0000 7f51fa8d6b80 1 -- start start 2024-05-27T07:43:06.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.498+0000 7f51fa8d6b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868484e40 0x560868608710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.499+0000 7f51fa8d6b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x560868602a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.499+0000 7f51fa8d6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868602f80 0x560868603390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.499+0000 7f51fa8d6b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560868531c90 con 0x560868608c50 2024-05-27T07:43:06.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.499+0000 7f51fa8d6b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560868521830 con 0x560868484e40 2024-05-27T07:43:06.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.499+0000 7f51fa8d6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560868525870 con 0x560868602f80 2024-05-27T07:43:06.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.499+0000 7f51e945c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868484e40 0x560868608710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.499+0000 7f51e945c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868602f80 0x560868603390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.499+0000 7f51daffd700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f51d00b2b40 0x7f51d00b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.499+0000 7f51e9c5d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x560868602a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.499+0000 7f51e9c5d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x560868602a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40700/0 (socket says 172.21.15.114:40700) 2024-05-27T07:43:06.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.499+0000 7f51e9c5d700 1 -- 172.21.15.114:0/1654357705 learned_addr learned my addr 172.21.15.114:0/1654357705 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:06.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.500+0000 7f51daffd700 1 -- 172.21.15.114:0/3598938662 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f51dc0e2420 con 0x7f51e411d920 2024-05-27T07:43:06.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.505+0000 7f51ea45e700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f51d00b2b40 0x7f51d00b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.506+0000 7f51ea45e700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f51d00b2b40 0x7f51d00b5050 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0df820 tx=0x7f51dc00b040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:06.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.507+0000 7f51e9c5d700 1 -- 172.21.15.114:0/1654357705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868602f80 msgr2=0x560868603390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:06.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.507+0000 7f51e9c5d700 1 --2- 172.21.15.114:0/1654357705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868602f80 0x560868603390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.507+0000 7f51e9c5d700 1 -- 172.21.15.114:0/1654357705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868484e40 msgr2=0x560868608710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:06.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.507+0000 7f51e9c5d700 1 --2- 172.21.15.114:0/1654357705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868484e40 0x560868608710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.507+0000 7f51e9c5d700 1 -- 172.21.15.114:0/1654357705 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608684718d0 con 0x560868608c50 2024-05-27T07:43:06.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.517+0000 7f51e9c5d700 1 --2- 172.21.15.114:0/1654357705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x560868602a40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f51e0003810 tx=0x7f51e0003840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713c1b4b7d5942b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:06.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.519+0000 7f51cffff700 1 -- 172.21.15.114:0/1654357705 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0003b00 con 0x560868608c50 2024-05-27T07:43:06.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.519+0000 7f51cffff700 1 -- 172.21.15.114:0/1654357705 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51e0005580 con 0x560868608c50 2024-05-27T07:43:06.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.519+0000 7f51cffff700 1 -- 172.21.15.114:0/1654357705 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e000bd10 con 0x560868608c50 2024-05-27T07:43:06.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.522+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/1654357705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 msgr2=0x560868602a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:06.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.522+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/1654357705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x560868602a40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f51e0003810 tx=0x7f51e0003840 comp rx=0 tx=0).stop 2024-05-27T07:43:06.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.523+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/1654357705 shutdown_connections 2024-05-27T07:43:06.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.523+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/1654357705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868602f80 0x560868603390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.523+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/1654357705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868484e40 0x560868608710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.523+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/1654357705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x560868602a40 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.523+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/1654357705 >> 172.21.15.114:0/1654357705 conn(0x5608684dfaf0 msgr2=0x5608685fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:06.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.524+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/1654357705 shutdown_connections 2024-05-27T07:43:06.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.524+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/1654357705 wait complete. 2024-05-27T07:43:06.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.525+0000 7f51fa8d6b80 1 -- start start 2024-05-27T07:43:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.525+0000 7f51fa8d6b80 1 Processor -- start 2024-05-27T07:43:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.526+0000 7f51fa8d6b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868484e40 0x56086860c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.526+0000 7f51fa8d6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868602f80 0x56086860c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.526+0000 7f51fa8d6b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868608c50 0x56086860ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.526+0000 7f51fa8d6b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560868427fa0 con 0x560868484e40 2024-05-27T07:43:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.526+0000 7f51fa8d6b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5608684c37c0 con 0x560868608c50 2024-05-27T07:43:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.526+0000 7f51fa8d6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56086860f6e0 con 0x560868602f80 2024-05-27T07:43:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.526+0000 7f51fa8d6b80 10 client.0 _ll_register_callbacks cb 0x5608684bc920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.526+0000 7f51e9c5d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868602f80 0x56086860c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.526+0000 7f51e9c5d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868602f80 0x56086860c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36394/0 (socket says 172.21.15.114:36394) 2024-05-27T07:43:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.526+0000 7f51e9c5d700 1 -- 172.21.15.114:0/431966530 learned_addr learned my addr 172.21.15.114:0/431966530 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.527+0000 7f51e945c700 1 --2- 172.21.15.114:0/431966530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868608c50 0x56086860ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.527+0000 7f51e945c700 1 --2- 172.21.15.114:0/431966530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868484e40 0x56086860c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.527+0000 7f51e945c700 1 -- 172.21.15.114:0/431966530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868602f80 msgr2=0x56086860c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:06.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.527+0000 7f51e945c700 1 --2- 172.21.15.114:0/431966530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868602f80 0x56086860c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.527+0000 7f51e945c700 1 -- 172.21.15.114:0/431966530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868608c50 msgr2=0x56086860ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:06.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.527+0000 7f51e945c700 1 --2- 172.21.15.114:0/431966530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868608c50 0x56086860ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.527+0000 7f51e945c700 1 -- 172.21.15.114:0/431966530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608684c0690 con 0x560868484e40 2024-05-27T07:43:06.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.528+0000 7f51e945c700 1 --2- 172.21.15.114:0/431966530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868484e40 0x56086860c390 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f51e411d770 tx=0x7f51d4006c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae6c93c7fb3e413 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:06.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.528+0000 7f51cd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:06.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.528+0000 7f51cd7fa700 1 -- 172.21.15.114:0/431966530 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d400f040 con 0x560868484e40 2024-05-27T07:43:06.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.528+0000 7f51cd7fa700 1 -- 172.21.15.114:0/431966530 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f51d4017b40 con 0x560868484e40 2024-05-27T07:43:06.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.528+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/431966530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51e0005580 con 0x560868484e40 2024-05-27T07:43:06.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.529+0000 7f51cd7fa700 1 -- 172.21.15.114:0/431966530 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f51d4019e20 con 0x560868484e40 2024-05-27T07:43:06.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.529+0000 7f51fa8d6b80 10 client.16020 fetch_fsmap learned FSMap version 25 2024-05-27T07:43:06.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.529+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/431966530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56086861a1f0 con 0x560868484e40 2024-05-27T07:43:06.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.529+0000 7f51cd7fa700 1 -- 172.21.15.114:0/431966530 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51e0005580 con 0x560868484e40 2024-05-27T07:43:06.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.529+0000 7f51cd7fa700 1 -- 172.21.15.114:0/431966530 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f51d402a020 con 0x560868484e40 2024-05-27T07:43:06.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.529+0000 7f51fa8d6b80 10 client.16020 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:43:06.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.529+0000 7f51fa8d6b80 10 client.16020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:06.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.529+0000 7f51fa8d6b80 20 client.16020 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:06.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.529+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/431966530 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56086861cc30 0x56086861f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.529+0000 7f51fa8d6b80 4 client.16020 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:06.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.529+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/431966530 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608685ce9f0 con 0x56086861cc30 2024-05-27T07:43:06.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.532+0000 7f51e9c5d700 1 --2- 172.21.15.114:0/431966530 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56086861cc30 0x56086861f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.540+0000 7f51e9c5d700 1 --2- 172.21.15.114:0/431966530 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56086861cc30 0x56086861f0e0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=243dd61b9e709eec server_cookie=36ff40ce83f322e7 in_seq=0 out_seq=0 2024-05-27T07:43:06.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.540+0000 7f51cd7fa700 10 client.16020 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:06.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.542+0000 7f51cd7fa700 1 -- 172.21.15.114:0/431966530 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5608685ce9f0 con 0x56086861cc30 2024-05-27T07:43:06.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.543+0000 7f51cd7fa700 10 client.16020 handle_command_reply: tid=0 2024-05-27T07:43:06.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.543+0000 7f51fa8d6b80 1 client.16020 shutdown 2024-05-27T07:43:06.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.544+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/431966530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868484e40 msgr2=0x56086860c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:06.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.544+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/431966530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868484e40 0x56086860c390 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f51e411d770 tx=0x7f51d4006c00 comp rx=0 tx=0).stop 2024-05-27T07:43:06.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.545+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/431966530 shutdown_connections 2024-05-27T07:43:06.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.545+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/431966530 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56086861cc30 0x56086861f0e0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.545+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/431966530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868608c50 0x56086860ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.545+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/431966530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868484e40 0x56086860c390 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.545+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/431966530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868602f80 0x56086860c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.545+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/431966530 >> 172.21.15.114:0/431966530 conn(0x5608684dfaf0 msgr2=0x560868601a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.545+0000 7f51cd7fa700 0 client.16020 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.545+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/431966530 shutdown_connections 2024-05-27T07:43:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.545+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/431966530 wait complete. 2024-05-27T07:43:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.546+0000 7f51fa8d6b80 20 client.16020 trim_cache size 0 max 16384 2024-05-27T07:43:06.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.574+0000 7f51fa8d6b80 1 Processor -- start 2024-05-27T07:43:06.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.574+0000 7f51fa8d6b80 1 -- start start 2024-05-27T07:43:06.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.574+0000 7f51fa8d6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868484e40 0x560868602a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.574+0000 7f51fa8d6b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868602f80 0x560868618780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51fa8d6b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x56086860e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51fa8d6b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f51e41bd350 con 0x560868608c50 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51fa8d6b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f51e41bfbd0 con 0x560868602f80 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51fa8d6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51e41bd9c0 con 0x560868484e40 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51e9c5d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868602f80 0x560868618780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51e9c5d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868602f80 0x560868618780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:53428/0 (socket says 172.21.15.114:53428) 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51e9c5d700 1 -- 172.21.15.114:0/1169280278 learned_addr learned my addr 172.21.15.114:0/1169280278 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51e945c700 1 --2- 172.21.15.114:0/1169280278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x56086860e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51e945c700 1 --2- 172.21.15.114:0/1169280278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868484e40 0x560868602a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51e9c5d700 1 -- 172.21.15.114:0/1169280278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868484e40 msgr2=0x560868602a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51e9c5d700 1 --2- 172.21.15.114:0/1169280278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868484e40 0x560868602a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51e9c5d700 1 -- 172.21.15.114:0/1169280278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 msgr2=0x56086860e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51e9c5d700 1 --2- 172.21.15.114:0/1169280278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x56086860e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.575+0000 7f51e9c5d700 1 -- 172.21.15.114:0/1169280278 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560868618cc0 con 0x560868602f80 2024-05-27T07:43:06.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.576+0000 7f51e9c5d700 1 --2- 172.21.15.114:0/1169280278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868602f80 0x560868618780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f51e0005b40 tx=0x7f51e000b7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a857d4d49b4efccb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:06.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.576+0000 7f51cf7fe700 1 -- 172.21.15.114:0/1169280278 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0027040 con 0x560868602f80 2024-05-27T07:43:06.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.576+0000 7f51cf7fe700 1 -- 172.21.15.114:0/1169280278 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51e000bd90 con 0x560868602f80 2024-05-27T07:43:06.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.576+0000 7f51cf7fe700 1 -- 172.21.15.114:0/1169280278 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e00054f0 con 0x560868602f80 2024-05-27T07:43:06.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.577+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/1169280278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868602f80 msgr2=0x560868618780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:06.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.577+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/1169280278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868602f80 0x560868618780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f51e0005b40 tx=0x7f51e000b7d0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.578+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/1169280278 shutdown_connections 2024-05-27T07:43:06.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.578+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/1169280278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x56086860e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.578+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/1169280278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868484e40 0x560868602a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.578+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/1169280278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868602f80 0x560868618780 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.578+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/1169280278 >> 172.21.15.114:0/1169280278 conn(0x5608684dfaf0 msgr2=0x56086860ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:06.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.578+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/1169280278 shutdown_connections 2024-05-27T07:43:06.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.578+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/1169280278 wait complete. 2024-05-27T07:43:06.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.578+0000 7f51fa8d6b80 1 -- start start 2024-05-27T07:43:06.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.578+0000 7f51fa8d6b80 1 Processor -- start 2024-05-27T07:43:06.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.579+0000 7f51fa8d6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868484e40 0x560868614b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.579+0000 7f51fa8d6b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868602f80 0x5608686150d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.579+0000 7f51fa8d6b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x5608686238d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.579+0000 7f51fa8d6b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f51e41be090 con 0x560868608c50 2024-05-27T07:43:06.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.579+0000 7f51fa8d6b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f51e41bf500 con 0x560868602f80 2024-05-27T07:43:06.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.579+0000 7f51fa8d6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51e41be760 con 0x560868484e40 2024-05-27T07:43:06.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.579+0000 7f51fa8d6b80 10 client.0 _ll_register_callbacks cb 0x5608684bc920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:06.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.579+0000 7f51e9c5d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868602f80 0x5608686150d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.580+0000 7f51e945c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868484e40 0x560868614b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.580+0000 7f51e945c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868484e40 0x560868614b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36418/0 (socket says 172.21.15.114:36418) 2024-05-27T07:43:06.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.580+0000 7f51e945c700 1 -- 172.21.15.114:0/3470362604 learned_addr learned my addr 172.21.15.114:0/3470362604 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:06.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.580+0000 7f51e945c700 1 --2- 172.21.15.114:0/3470362604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x5608686238d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.580+0000 7f51e945c700 1 -- 172.21.15.114:0/3470362604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868484e40 msgr2=0x560868614b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:06.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.580+0000 7f51e945c700 1 --2- 172.21.15.114:0/3470362604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868484e40 0x560868614b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.580+0000 7f51e945c700 1 -- 172.21.15.114:0/3470362604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868602f80 msgr2=0x5608686150d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.580+0000 7f51e945c700 1 --2- 172.21.15.114:0/3470362604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868602f80 0x5608686150d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.580+0000 7f51e945c700 1 -- 172.21.15.114:0/3470362604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560868624070 con 0x560868608c50 2024-05-27T07:43:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.581+0000 7f51e945c700 1 --2- 172.21.15.114:0/3470362604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x5608686238d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f51d401afd0 tx=0x7f51d401b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e91225c3a89b5a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.581+0000 7f51cd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.581+0000 7f51cd7fa700 1 -- 172.21.15.114:0/3470362604 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4015470 con 0x560868608c50 2024-05-27T07:43:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.581+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/3470362604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51e000bd90 con 0x560868608c50 2024-05-27T07:43:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.581+0000 7f51cd7fa700 1 -- 172.21.15.114:0/3470362604 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f51d4017b50 con 0x560868608c50 2024-05-27T07:43:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.582+0000 7f51cd7fa700 1 -- 172.21.15.114:0/3470362604 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f51d4017d50 con 0x560868608c50 2024-05-27T07:43:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.582+0000 7f51fa8d6b80 10 client.16026 fetch_fsmap learned FSMap version 25 2024-05-27T07:43:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.582+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/3470362604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5608686252f0 con 0x560868608c50 2024-05-27T07:43:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.582+0000 7f51cd7fa700 1 -- 172.21.15.114:0/3470362604 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51e000bd90 con 0x560868608c50 2024-05-27T07:43:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.583+0000 7f51cd7fa700 1 -- 172.21.15.114:0/3470362604 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f51d402a040 con 0x560868608c50 2024-05-27T07:43:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.583+0000 7f51fa8d6b80 10 client.16026 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:43:06.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.583+0000 7f51fa8d6b80 10 client.16026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:06.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.583+0000 7f51fa8d6b80 20 client.16026 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:06.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.583+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/3470362604 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560868686b20 0x560868688f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:06.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.583+0000 7f51fa8d6b80 4 client.16026 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:43:06.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.583+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/3470362604 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560868533be0 con 0x560868686b20 2024-05-27T07:43:06.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.583+0000 7f51e9c5d700 1 --2- 172.21.15.114:0/3470362604 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560868686b20 0x560868688f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:06.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.588+0000 7f51e9c5d700 1 --2- 172.21.15.114:0/3470362604 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560868686b20 0x560868688f40 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb6bd206f9a7ac2c server_cookie=2bcc35b12881c863 in_seq=0 out_seq=0 2024-05-27T07:43:06.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.588+0000 7f51cd7fa700 10 client.16026 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:06.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.589+0000 7f51cd7fa700 1 -- 172.21.15.114:0/3470362604 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560868533be0 con 0x560868686b20 2024-05-27T07:43:06.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.589+0000 7f51cd7fa700 10 client.16026 handle_command_reply: tid=0 2024-05-27T07:43:06.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.589+0000 7f51fa8d6b80 1 client.16026 shutdown 2024-05-27T07:43:06.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.589+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/3470362604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 msgr2=0x5608686238d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:06.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.589+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/3470362604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x5608686238d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f51d401afd0 tx=0x7f51d401b040 comp rx=0 tx=0).stop 2024-05-27T07:43:06.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.590+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/3470362604 shutdown_connections 2024-05-27T07:43:06.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.590+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/3470362604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560868608c50 0x5608686238d0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.590+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/3470362604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560868484e40 0x560868614b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.590+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/3470362604 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560868686b20 0x560868688f40 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.590+0000 7f51fa8d6b80 1 --2- 172.21.15.114:0/3470362604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560868602f80 0x5608686150d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.590+0000 7f51cd7fa700 0 client.16026 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:06.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.590+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/3470362604 >> 172.21.15.114:0/3470362604 conn(0x5608684dfaf0 msgr2=0x560868606010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:06.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.590+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/3470362604 shutdown_connections 2024-05-27T07:43:06.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.591+0000 7f51fa8d6b80 1 -- 172.21.15.114:0/3470362604 wait complete. 2024-05-27T07:43:06.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.591+0000 7f51fa8d6b80 20 client.16026 trim_cache size 0 max 16384 2024-05-27T07:43:06.591 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:43:06.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.592+0000 7f51ebec1700 1 -- 172.21.15.114:0/3598938662 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f51d00b2b40 msgr2=0x7f51d00b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:06.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.592+0000 7f51ebec1700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f51d00b2b40 0x7f51d00b5050 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0df820 tx=0x7f51dc00b040 comp rx=0 tx=0).stop 2024-05-27T07:43:06.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.592+0000 7f51ebec1700 1 -- 172.21.15.114:0/3598938662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e411d920 msgr2=0x7f51e41a99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:06.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.592+0000 7f51ebec1700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e411d920 0x7f51e41a99a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0094f0 tx=0x7f51dc007f90 comp rx=0 tx=0).stop 2024-05-27T07:43:06.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.592+0000 7f51ebec1700 1 -- 172.21.15.114:0/3598938662 shutdown_connections 2024-05-27T07:43:06.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.592+0000 7f51ebec1700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e411d920 0x7f51e41a99a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.592+0000 7f51ebec1700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f51d00b2b40 0x7f51d00b5050 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.592+0000 7f51ebec1700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51e410a920 0x7f51e41af780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.592+0000 7f51ebec1700 1 --2- 172.21.15.114:0/3598938662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51e410b270 0x7f51e41afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:06.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.592+0000 7f51ebec1700 1 -- 172.21.15.114:0/3598938662 >> 172.21.15.114:0/3598938662 conn(0x7f51e406f4c0 msgr2=0x7f51e411b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:06.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.592+0000 7f51ebec1700 1 -- 172.21.15.114:0/3598938662 shutdown_connections 2024-05-27T07:43:06.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:06.593+0000 7f51ebec1700 1 -- 172.21.15.114:0/3598938662 wait complete. 2024-05-27T07:43:07.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 208505 14.44 MB/sec warmup 79 sec latency 50.594 ms 2024-05-27T07:43:07.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:07 smithi163 ceph-mon[36451]: pgmap v106: 129 pgs: 129 active+clean; 43 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 511 op/s 2024-05-27T07:43:07.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:07 smithi186 ceph-mon[36647]: pgmap v106: 129 pgs: 129 active+clean; 43 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 511 op/s 2024-05-27T07:43:08.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:07 smithi114 ceph-mon[32038]: pgmap v106: 129 pgs: 129 active+clean; 43 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 511 op/s 2024-05-27T07:43:08.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 210784 14.43 MB/sec warmup 80 sec latency 45.067 ms 2024-05-27T07:43:08.688 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:43:08.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:43:08.940 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:09.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 213624 14.43 MB/sec warmup 81 sec latency 14.065 ms 2024-05-27T07:43:09.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:09 smithi114 ceph-mon[32038]: pgmap v107: 129 pgs: 129 active+clean; 43 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 421 op/s 2024-05-27T07:43:09.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:09 smithi163 ceph-mon[36451]: pgmap v107: 129 pgs: 129 active+clean; 43 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 421 op/s 2024-05-27T07:43:09.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:09 smithi186 ceph-mon[36647]: pgmap v107: 129 pgs: 129 active+clean; 43 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 421 op/s 2024-05-27T07:43:10.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 216802 14.48 MB/sec warmup 82 sec latency 26.497 ms 2024-05-27T07:43:10.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.752+0000 7fb0cb2d6700 1 -- 172.21.15.114:0/656698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c41092f0 msgr2=0x7fb0c4109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:10.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.752+0000 7fb0cb2d6700 1 --2- 172.21.15.114:0/656698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c41092f0 0x7fb0c4109700 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009990 tx=0x7fb0c0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.752+0000 7fb0cb2d6700 1 -- 172.21.15.114:0/656698 shutdown_connections 2024-05-27T07:43:10.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.752+0000 7fb0cb2d6700 1 --2- 172.21.15.114:0/656698 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0c41030d0 0x7fb0c4111160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.752+0000 7fb0cb2d6700 1 --2- 172.21.15.114:0/656698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c41092f0 0x7fb0c4109700 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.752+0000 7fb0cb2d6700 1 --2- 172.21.15.114:0/656698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0c4102780 0x7fb0c4102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.752+0000 7fb0cb2d6700 1 -- 172.21.15.114:0/656698 >> 172.21.15.114:0/656698 conn(0x7fb0c40fe510 msgr2=0x7fb0c4100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:10.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.752+0000 7fb0cb2d6700 1 -- 172.21.15.114:0/656698 shutdown_connections 2024-05-27T07:43:10.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.752+0000 7fb0cb2d6700 1 -- 172.21.15.114:0/656698 wait complete. 2024-05-27T07:43:10.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0cb2d6700 1 Processor -- start 2024-05-27T07:43:10.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0cb2d6700 1 -- start start 2024-05-27T07:43:10.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0cb2d6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0c4102780 0x7fb0c4103960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0cb2d6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c41030d0 0x7fb0c4103ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0cb2d6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0c41092f0 0x7fb0c41043e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0cb2d6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c4114580 con 0x7fb0c41030d0 2024-05-27T07:43:10.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0cb2d6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c4114400 con 0x7fb0c4102780 2024-05-27T07:43:10.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0cb2d6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c4114700 con 0x7fb0c41092f0 2024-05-27T07:43:10.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0c8871700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c41030d0 0x7fb0c4103ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0c9072700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0c4102780 0x7fb0c4103960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0c8871700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c41030d0 0x7fb0c4103ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40746/0 (socket says 172.21.15.114:40746) 2024-05-27T07:43:10.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0c9072700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0c4102780 0x7fb0c4103960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:53456/0 (socket says 172.21.15.114:53456) 2024-05-27T07:43:10.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0c8871700 1 -- 172.21.15.114:0/4279213768 learned_addr learned my addr 172.21.15.114:0/4279213768 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:10.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.753+0000 7fb0c9873700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0c41092f0 0x7fb0c41043e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.754+0000 7fb0c9072700 1 -- 172.21.15.114:0/4279213768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0c41092f0 msgr2=0x7fb0c41043e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:10.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.754+0000 7fb0c9072700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0c41092f0 0x7fb0c41043e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.754+0000 7fb0c9072700 1 -- 172.21.15.114:0/4279213768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c41030d0 msgr2=0x7fb0c4103ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:10.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.754+0000 7fb0c9072700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c41030d0 0x7fb0c4103ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.754+0000 7fb0c9072700 1 -- 172.21.15.114:0/4279213768 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c41a9290 con 0x7fb0c4102780 2024-05-27T07:43:10.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.754+0000 7fb0c8871700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c41030d0 0x7fb0c4103ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:10.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.754+0000 7fb0c9072700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0c4102780 0x7fb0c4103960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0000c00 tx=0x7fb0c00047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:10.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.754+0000 7fb0ba7fc700 1 -- 172.21.15.114:0/4279213768 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0027070 con 0x7fb0c4102780 2024-05-27T07:43:10.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.754+0000 7fb0cb2d6700 1 -- 172.21.15.114:0/4279213768 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c41a9520 con 0x7fb0c4102780 2024-05-27T07:43:10.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.754+0000 7fb0cb2d6700 1 -- 172.21.15.114:0/4279213768 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c41a99d0 con 0x7fb0c4102780 2024-05-27T07:43:10.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.754+0000 7fb0ba7fc700 1 -- 172.21.15.114:0/4279213768 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0c0022470 con 0x7fb0c4102780 2024-05-27T07:43:10.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.754+0000 7fb0ba7fc700 1 -- 172.21.15.114:0/4279213768 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0017650 con 0x7fb0c4102780 2024-05-27T07:43:10.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.757+0000 7fb0ba7fc700 1 -- 172.21.15.114:0/4279213768 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fb0c0009d10 con 0x7fb0c4102780 2024-05-27T07:43:10.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.758+0000 7fb0ba7fc700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0b00b2a70 0x7fb0b00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.758+0000 7fb0ba7fc700 1 -- 172.21.15.114:0/4279213768 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7fb0c00e34b0 con 0x7fb0c4102780 2024-05-27T07:43:10.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.758+0000 7fb0c8871700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0b00b2a70 0x7fb0b00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.758+0000 7fb0d9cebb80 1 Processor -- start 2024-05-27T07:43:10.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.758+0000 7fb0d9cebb80 1 -- start start 2024-05-27T07:43:10.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0c8871700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0b00b2a70 0x7fb0b00b4f80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4108a70 tx=0x7fb0b4009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:10.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0d9cebb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565036236ec0 0x5650363c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0d9cebb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363c48e0 0x5650363be470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0d9cebb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 0x5650363bedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0d9cebb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5650362e3df0 con 0x5650363be9b0 2024-05-27T07:43:10.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0d9cebb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5650362d3990 con 0x565036236ec0 2024-05-27T07:43:10.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0d9cebb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5650362d79d0 con 0x5650363c48e0 2024-05-27T07:43:10.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0c9873700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 0x5650363bedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0c8871700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363c48e0 0x5650363be470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0c9072700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565036236ec0 0x5650363c43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0c9873700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 0x5650363bedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40750/0 (socket says 172.21.15.114:40750) 2024-05-27T07:43:10.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0c8871700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363c48e0 0x5650363be470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36446/0 (socket says 172.21.15.114:36446) 2024-05-27T07:43:10.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0c9873700 1 -- 172.21.15.114:0/4040038851 learned_addr learned my addr 172.21.15.114:0/4040038851 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:10.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0c9873700 1 -- 172.21.15.114:0/4040038851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363c48e0 msgr2=0x5650363be470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:10.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0c9873700 1 --2- 172.21.15.114:0/4040038851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363c48e0 0x5650363be470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0c9873700 1 -- 172.21.15.114:0/4040038851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565036236ec0 msgr2=0x5650363c43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:10.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0c9873700 1 --2- 172.21.15.114:0/4040038851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565036236ec0 0x5650363c43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.759+0000 7fb0c9873700 1 -- 172.21.15.114:0/4040038851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56503627d030 con 0x5650363be9b0 2024-05-27T07:43:10.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.760+0000 7fb0c9873700 1 --2- 172.21.15.114:0/4040038851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 0x5650363bedc0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc011680 tx=0x7fb0bc009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f199b4877e1eb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:10.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.760+0000 7fb0af7fe700 1 -- 172.21.15.114:0/4040038851 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc019030 con 0x5650363be9b0 2024-05-27T07:43:10.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.760+0000 7fb0af7fe700 1 -- 172.21.15.114:0/4040038851 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0bc0092e0 con 0x5650363be9b0 2024-05-27T07:43:10.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.760+0000 7fb0af7fe700 1 -- 172.21.15.114:0/4040038851 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc007500 con 0x5650363be9b0 2024-05-27T07:43:10.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.761+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/4040038851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 msgr2=0x5650363bedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:10.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.761+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/4040038851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 0x5650363bedc0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc011680 tx=0x7fb0bc009920 comp rx=0 tx=0).stop 2024-05-27T07:43:10.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.761+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/4040038851 shutdown_connections 2024-05-27T07:43:10.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.761+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/4040038851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 0x5650363bedc0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.761+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/4040038851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565036236ec0 0x5650363c43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.761+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/4040038851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363c48e0 0x5650363be470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.761+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/4040038851 >> 172.21.15.114:0/4040038851 conn(0x5650362397f0 msgr2=0x5650363b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:10.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.761+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/4040038851 shutdown_connections 2024-05-27T07:43:10.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.761+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/4040038851 wait complete. 2024-05-27T07:43:10.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0d9cebb80 1 -- start start 2024-05-27T07:43:10.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0d9cebb80 1 Processor -- start 2024-05-27T07:43:10.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0d9cebb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565036236ec0 0x5650363c8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0d9cebb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650363be9b0 0x5650363c8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0d9cebb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363c48e0 0x5650363c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0d9cebb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5650361da020 con 0x5650363c48e0 2024-05-27T07:43:10.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0d9cebb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56503623be20 con 0x5650363be9b0 2024-05-27T07:43:10.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0d9cebb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5650363cb410 con 0x565036236ec0 2024-05-27T07:43:10.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0d9cebb80 10 client.0 _ll_register_callbacks cb 0x56503626a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:10.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0c9072700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565036236ec0 0x5650363c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0c8871700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650363be9b0 0x5650363c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0c9072700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565036236ec0 0x5650363c8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36452/0 (socket says 172.21.15.114:36452) 2024-05-27T07:43:10.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0c9072700 1 -- 172.21.15.114:0/3533607600 learned_addr learned my addr 172.21.15.114:0/3533607600 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:10.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.762+0000 7fb0c9873700 1 --2- 172.21.15.114:0/3533607600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363c48e0 0x5650363c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.763+0000 7fb0c8871700 1 -- 172.21.15.114:0/3533607600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565036236ec0 msgr2=0x5650363c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:10.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.763+0000 7fb0c8871700 1 --2- 172.21.15.114:0/3533607600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565036236ec0 0x5650363c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.763+0000 7fb0c8871700 1 -- 172.21.15.114:0/3533607600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363c48e0 msgr2=0x5650363c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:10.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.763+0000 7fb0c8871700 1 --2- 172.21.15.114:0/3533607600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363c48e0 0x5650363c8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.763+0000 7fb0c8871700 1 -- 172.21.15.114:0/3533607600 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565036223950 con 0x5650363be9b0 2024-05-27T07:43:10.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.763+0000 7fb0c8871700 1 --2- 172.21.15.114:0/3533607600 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650363be9b0 0x5650363c8560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4005950 tx=0x7fb0b40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85ea1fd403f90430 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:10.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.763+0000 7fb0acff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:43:10.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.763+0000 7fb0acff9700 1 -- 172.21.15.114:0/3533607600 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b401f7e0 con 0x5650363be9b0 2024-05-27T07:43:10.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.763+0000 7fb0acff9700 1 -- 172.21.15.114:0/3533607600 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7fb0b401f980 con 0x5650363be9b0 2024-05-27T07:43:10.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.763+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3533607600 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0bc0092e0 con 0x5650363be9b0 2024-05-27T07:43:10.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.764+0000 7fb0acff9700 1 -- 172.21.15.114:0/3533607600 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb0b4031050 con 0x5650363be9b0 2024-05-27T07:43:10.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.764+0000 7fb0d9cebb80 10 client.25562 fetch_fsmap learned FSMap version 25 2024-05-27T07:43:10.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.764+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3533607600 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56503628d7e0 con 0x5650363be9b0 2024-05-27T07:43:10.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.764+0000 7fb0acff9700 1 -- 172.21.15.114:0/3533607600 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0bc0092e0 con 0x5650363be9b0 2024-05-27T07:43:10.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.764+0000 7fb0acff9700 1 -- 172.21.15.114:0/3533607600 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fb0b4026020 con 0x5650363be9b0 2024-05-27T07:43:10.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.764+0000 7fb0d9cebb80 10 client.25562 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:43:10.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.764+0000 7fb0d9cebb80 10 client.25562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:10.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.764+0000 7fb0d9cebb80 20 client.25562 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:10.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.764+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3533607600 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5650363d8340 0x5650363da7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.764+0000 7fb0d9cebb80 4 client.25562 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:10.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.764+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3533607600 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650362e5d40 con 0x5650363d8340 2024-05-27T07:43:10.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.764+0000 7fb0c9072700 1 --2- 172.21.15.114:0/3533607600 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5650363d8340 0x5650363da7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.765+0000 7fb0acff9700 10 client.25562 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:10.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.765+0000 7fb0c9072700 1 --2- 172.21.15.114:0/3533607600 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5650363d8340 0x5650363da7d0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e87598e75f406a00 server_cookie=17831d4063decfea in_seq=0 out_seq=0 2024-05-27T07:43:10.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0acff9700 1 -- 172.21.15.114:0/3533607600 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5650362e5d40 con 0x5650363d8340 2024-05-27T07:43:10.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0acff9700 10 client.25562 handle_command_reply: tid=0 2024-05-27T07:43:10.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0d9cebb80 1 client.25562 shutdown 2024-05-27T07:43:10.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3533607600 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650363be9b0 msgr2=0x5650363c8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:10.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3533607600 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650363be9b0 0x5650363c8560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4005950 tx=0x7fb0b40047c0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3533607600 shutdown_connections 2024-05-27T07:43:10.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3533607600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363c48e0 0x5650363c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3533607600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565036236ec0 0x5650363c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3533607600 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5650363d8340 0x5650363da7d0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3533607600 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650363be9b0 0x5650363c8560 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3533607600 >> 172.21.15.114:0/3533607600 conn(0x5650362397f0 msgr2=0x5650363b4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:10.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0acff9700 0 client.25562 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:10.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3533607600 shutdown_connections 2024-05-27T07:43:10.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.766+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3533607600 wait complete. 2024-05-27T07:43:10.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.767+0000 7fb0d9cebb80 20 client.25562 trim_cache size 0 max 16384 2024-05-27T07:43:10.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.790+0000 7fb0d9cebb80 1 Processor -- start 2024-05-27T07:43:10.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.790+0000 7fb0d9cebb80 1 -- start start 2024-05-27T07:43:10.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.790+0000 7fb0d9cebb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565036236ec0 0x5650363b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.790+0000 7fb0d9cebb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 0x5650363c24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.790+0000 7fb0d9cebb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363c48e0 0x5650363d44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.790+0000 7fb0d9cebb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a800cd40 con 0x5650363be9b0 2024-05-27T07:43:10.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.790+0000 7fb0d9cebb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a800f5c0 con 0x565036236ec0 2024-05-27T07:43:10.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.790+0000 7fb0d9cebb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a800d3b0 con 0x5650363c48e0 2024-05-27T07:43:10.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.790+0000 7fb0c8871700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 0x5650363c24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.790+0000 7fb0c8871700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 0x5650363c24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40764/0 (socket says 172.21.15.114:40764) 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.790+0000 7fb0c8871700 1 -- 172.21.15.114:0/3859370747 learned_addr learned my addr 172.21.15.114:0/3859370747 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.790+0000 7fb0c9072700 1 --2- 172.21.15.114:0/3859370747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565036236ec0 0x5650363b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.790+0000 7fb0c9873700 1 --2- 172.21.15.114:0/3859370747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363c48e0 0x5650363d44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.791+0000 7fb0c8871700 1 -- 172.21.15.114:0/3859370747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363c48e0 msgr2=0x5650363d44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.791+0000 7fb0c8871700 1 --2- 172.21.15.114:0/3859370747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363c48e0 0x5650363d44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.791+0000 7fb0c8871700 1 -- 172.21.15.114:0/3859370747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565036236ec0 msgr2=0x5650363b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.791+0000 7fb0c8871700 1 --2- 172.21.15.114:0/3859370747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565036236ec0 0x5650363b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.791+0000 7fb0c8871700 1 -- 172.21.15.114:0/3859370747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650363c65c0 con 0x5650363be9b0 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.791+0000 7fb0c9072700 1 --2- 172.21.15.114:0/3859370747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565036236ec0 0x5650363b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.791+0000 7fb0c8871700 1 --2- 172.21.15.114:0/3859370747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 0x5650363c24d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb0b400eda0 tx=0x7fb0b4003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55189804aee7de55 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.791+0000 7fb0aeffd700 1 -- 172.21.15.114:0/3859370747 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4004a30 con 0x5650363be9b0 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.791+0000 7fb0aeffd700 1 -- 172.21.15.114:0/3859370747 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0b40054e0 con 0x5650363be9b0 2024-05-27T07:43:10.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.791+0000 7fb0aeffd700 1 -- 172.21.15.114:0/3859370747 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b401faa0 con 0x5650363be9b0 2024-05-27T07:43:10.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.792+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3859370747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 msgr2=0x5650363c24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:10.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.792+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3859370747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 0x5650363c24d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb0b400eda0 tx=0x7fb0b4003d60 comp rx=0 tx=0).stop 2024-05-27T07:43:10.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.792+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3859370747 shutdown_connections 2024-05-27T07:43:10.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.792+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3859370747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363c48e0 0x5650363d44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.792+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3859370747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565036236ec0 0x5650363b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.792+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3859370747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650363be9b0 0x5650363c24d0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.792+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3859370747 >> 172.21.15.114:0/3859370747 conn(0x5650362397f0 msgr2=0x5650363c1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:10.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.792+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3859370747 shutdown_connections 2024-05-27T07:43:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.792+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3859370747 wait complete. 2024-05-27T07:43:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.793+0000 7fb0d9cebb80 1 -- start start 2024-05-27T07:43:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.793+0000 7fb0d9cebb80 1 Processor -- start 2024-05-27T07:43:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.793+0000 7fb0d9cebb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565036236ec0 0x5650363d06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.793+0000 7fb0d9cebb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363be9b0 0x5650363d0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.793+0000 7fb0d9cebb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650363c48e0 0x5650363cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.793+0000 7fb0d9cebb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a800da80 con 0x565036236ec0 2024-05-27T07:43:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.793+0000 7fb0d9cebb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a800eef0 con 0x5650363c48e0 2024-05-27T07:43:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.793+0000 7fb0d9cebb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a800e150 con 0x5650363be9b0 2024-05-27T07:43:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.793+0000 7fb0c9072700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565036236ec0 0x5650363d06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.793+0000 7fb0d9cebb80 10 client.0 _ll_register_callbacks cb 0x56503626a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:10.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0c9072700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565036236ec0 0x5650363d06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40776/0 (socket says 172.21.15.114:40776) 2024-05-27T07:43:10.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0c9072700 1 -- 172.21.15.114:0/3200191088 learned_addr learned my addr 172.21.15.114:0/3200191088 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:10.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0c9873700 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650363c48e0 0x5650363cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0c8871700 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363be9b0 0x5650363d0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0c9072700 1 -- 172.21.15.114:0/3200191088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363be9b0 msgr2=0x5650363d0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:10.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0c9072700 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363be9b0 0x5650363d0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0c9072700 1 -- 172.21.15.114:0/3200191088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650363c48e0 msgr2=0x5650363cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:10.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0c9072700 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650363c48e0 0x5650363cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0c9072700 1 -- 172.21.15.114:0/3200191088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650363cdc20 con 0x565036236ec0 2024-05-27T07:43:10.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0c9873700 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650363c48e0 0x5650363cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:43:10.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0c9072700 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565036236ec0 0x5650363d06c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0022e50 tx=0x7fb0c00e07e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ba30da30d30878 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:10.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0acff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:10.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0acff9700 1 -- 172.21.15.114:0/3200191088 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c00aab10 con 0x565036236ec0 2024-05-27T07:43:10.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0acff9700 1 -- 172.21.15.114:0/3200191088 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7fb0c00aa670 con 0x565036236ec0 2024-05-27T07:43:10.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3200191088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0b40054e0 con 0x565036236ec0 2024-05-27T07:43:10.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0acff9700 1 -- 172.21.15.114:0/3200191088 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb0c00aa410 con 0x565036236ec0 2024-05-27T07:43:10.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.794+0000 7fb0d9cebb80 10 client.16044 fetch_fsmap learned FSMap version 25 2024-05-27T07:43:10.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.795+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3200191088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x565036443750 con 0x565036236ec0 2024-05-27T07:43:10.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.795+0000 7fb0acff9700 1 -- 172.21.15.114:0/3200191088 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0b40054e0 con 0x565036236ec0 2024-05-27T07:43:10.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.795+0000 7fb0acff9700 1 -- 172.21.15.114:0/3200191088 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fb0c00cb120 con 0x565036236ec0 2024-05-27T07:43:10.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.795+0000 7fb0d9cebb80 10 client.16044 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:43:10.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.795+0000 7fb0d9cebb80 10 client.16044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:10.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.795+0000 7fb0d9cebb80 20 client.16044 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:10.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.795+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x565036443e60 0x565036446280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:10.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.795+0000 7fb0d9cebb80 4 client.16044 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:43:10.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.795+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3200191088 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5650362df520 con 0x565036443e60 2024-05-27T07:43:10.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.795+0000 7fb0c8871700 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x565036443e60 0x565036446280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:10.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.796+0000 7fb0acff9700 10 client.16044 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:10.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.796+0000 7fb0c8871700 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x565036443e60 0x565036446280 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dd752b86f940bbf server_cookie=a1068b38e524da31 in_seq=0 out_seq=0 2024-05-27T07:43:10.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.798+0000 7fb0acff9700 1 -- 172.21.15.114:0/3200191088 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5650362df520 con 0x565036443e60 2024-05-27T07:43:10.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.798+0000 7fb0acff9700 10 client.16044 handle_command_reply: tid=0 2024-05-27T07:43:10.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.798+0000 7fb0d9cebb80 1 client.16044 shutdown 2024-05-27T07:43:10.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.798+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3200191088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565036236ec0 msgr2=0x5650363d06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:10.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.798+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565036236ec0 0x5650363d06c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0022e50 tx=0x7fb0c00e07e0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.798+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3200191088 shutdown_connections 2024-05-27T07:43:10.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.798+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x565036443e60 0x565036446280 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.798+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650363c48e0 0x5650363cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.798+0000 7fb0acff9700 0 client.16044 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.798+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565036236ec0 0x5650363d06c0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.798+0000 7fb0d9cebb80 1 --2- 172.21.15.114:0/3200191088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650363be9b0 0x5650363d0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.798+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3200191088 >> 172.21.15.114:0/3200191088 conn(0x5650362397f0 msgr2=0x5650363aa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3200191088 shutdown_connections 2024-05-27T07:43:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0d9cebb80 1 -- 172.21.15.114:0/3200191088 wait complete. 2024-05-27T07:43:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0d9cebb80 20 client.16044 trim_cache size 0 max 16384 2024-05-27T07:43:10.801 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:43:10.801 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:43:10.802 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "eb248c19-32d5-4d3e-808d-16903660f4a2", 2024-05-27T07:43:10.802 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:43:10.802 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:43:10.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0cb2d6700 1 -- 172.21.15.114:0/4279213768 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0b00b2a70 msgr2=0x7fb0b00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:10.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0cb2d6700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0b00b2a70 0x7fb0b00b4f80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4108a70 tx=0x7fb0b4009260 comp rx=0 tx=0).stop 2024-05-27T07:43:10.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0cb2d6700 1 -- 172.21.15.114:0/4279213768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0c4102780 msgr2=0x7fb0c4103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:10.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0cb2d6700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0c4102780 0x7fb0c4103960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0000c00 tx=0x7fb0c00047b0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0cb2d6700 1 -- 172.21.15.114:0/4279213768 shutdown_connections 2024-05-27T07:43:10.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0cb2d6700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0c41092f0 0x7fb0c41043e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0cb2d6700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0b00b2a70 0x7fb0b00b4f80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0cb2d6700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0c4102780 0x7fb0c4103960 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0cb2d6700 1 --2- 172.21.15.114:0/4279213768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c41030d0 0x7fb0c4103ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:10.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0cb2d6700 1 -- 172.21.15.114:0/4279213768 >> 172.21.15.114:0/4279213768 conn(0x7fb0c40fe510 msgr2=0x7fb0c4107170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:10.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0cb2d6700 1 -- 172.21.15.114:0/4279213768 shutdown_connections 2024-05-27T07:43:10.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:10.799+0000 7fb0cb2d6700 1 -- 172.21.15.114:0/4279213768 wait complete. 2024-05-27T07:43:11.240 INFO:tasks.workunit.client.0.smithi114.stdout: 1 219018 14.40 MB/sec warmup 83 sec latency 63.500 ms 2024-05-27T07:43:11.617 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:43:11.865 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:11.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:11 smithi163 ceph-mon[36451]: pgmap v108: 129 pgs: 129 active+clean; 49 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.8 MiB/s wr, 550 op/s 2024-05-27T07:43:11.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:11 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:43:11.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:11 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:11.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:11 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:43:11.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:11 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:11.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:11 smithi186 ceph-mon[36647]: pgmap v108: 129 pgs: 129 active+clean; 49 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.8 MiB/s wr, 550 op/s 2024-05-27T07:43:11.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:11 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:43:11.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:11 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:11.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:11 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:43:11.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:11 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:11.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:11 smithi114 ceph-mon[32038]: pgmap v108: 129 pgs: 129 active+clean; 49 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.8 MiB/s wr, 550 op/s 2024-05-27T07:43:11.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:11 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:43:11.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:11 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:11.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:11 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:43:11.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:11 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:12.241 INFO:tasks.workunit.client.0.smithi114.stdout: 1 220902 14.37 MB/sec warmup 84 sec latency 24.072 ms 2024-05-27T07:43:12.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:12 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:43:12.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:12 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:43:13.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:12 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:43:13.241 INFO:tasks.workunit.client.0.smithi114.stdout: 1 223988 14.42 MB/sec warmup 85 sec latency 8.720 ms 2024-05-27T07:43:13.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:13 smithi114 ceph-mon[32038]: pgmap v109: 129 pgs: 129 active+clean; 49 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 422 op/s 2024-05-27T07:43:13.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:13 smithi163 ceph-mon[36451]: pgmap v109: 129 pgs: 129 active+clean; 49 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 422 op/s 2024-05-27T07:43:13.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:13 smithi186 ceph-mon[36647]: pgmap v109: 129 pgs: 129 active+clean; 49 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 422 op/s 2024-05-27T07:43:14.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.100+0000 7f7f08113700 1 -- 172.21.15.114:0/992199584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f00103300 msgr2=0x7f7f0010f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:14.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.100+0000 7f7f08113700 1 --2- 172.21.15.114:0/992199584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f00103300 0x7f7f0010f880 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7ef800b0a0 tx=0x7f7ef800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.100+0000 7f7f08113700 1 -- 172.21.15.114:0/992199584 shutdown_connections 2024-05-27T07:43:14.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.100+0000 7f7f08113700 1 --2- 172.21.15.114:0/992199584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f00103300 0x7f7f0010f880 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.100+0000 7f7f08113700 1 --2- 172.21.15.114:0/992199584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f001089e0 0x7f7f00108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.100+0000 7f7f08113700 1 --2- 172.21.15.114:0/992199584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f001029b0 0x7f7f00102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.100+0000 7f7f08113700 1 -- 172.21.15.114:0/992199584 >> 172.21.15.114:0/992199584 conn(0x7f7f000fe4d0 msgr2=0x7f7f00100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:14.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.100+0000 7f7f08113700 1 -- 172.21.15.114:0/992199584 shutdown_connections 2024-05-27T07:43:14.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.100+0000 7f7f08113700 1 -- 172.21.15.114:0/992199584 wait complete. 2024-05-27T07:43:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.101+0000 7f7f08113700 1 Processor -- start 2024-05-27T07:43:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.101+0000 7f7f08113700 1 -- start start 2024-05-27T07:43:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f08113700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f001029b0 0x7f7f0019aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f08113700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f00103300 0x7f7f0019b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f08113700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f001089e0 0x7f7f0019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f08113700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7f00112330 con 0x7f7f001029b0 2024-05-27T07:43:14.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f08113700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001121b0 con 0x7f7f001089e0 2024-05-27T07:43:14.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f08113700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001124b0 con 0x7f7f00103300 2024-05-27T07:43:14.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f05eaf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f001029b0 0x7f7f0019aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f05eaf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f001029b0 0x7f7f0019aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40788/0 (socket says 172.21.15.114:40788) 2024-05-27T07:43:14.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f05eaf700 1 -- 172.21.15.114:0/34398626 learned_addr learned my addr 172.21.15.114:0/34398626 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:14.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f056ae700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f00103300 0x7f7f0019b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f066b0700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f001089e0 0x7f7f0019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f05eaf700 1 -- 172.21.15.114:0/34398626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f00103300 msgr2=0x7f7f0019b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:14.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f00103300 0x7f7f0019b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f05eaf700 1 -- 172.21.15.114:0/34398626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f001089e0 msgr2=0x7f7f0019f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:14.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f001089e0 0x7f7f0019f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f05eaf700 1 -- 172.21.15.114:0/34398626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0019fd70 con 0x7f7f001029b0 2024-05-27T07:43:14.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f066b0700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f001089e0 0x7f7f0019f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:43:14.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f001029b0 0x7f7f0019aea0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000d980 tx=0x7f7ef000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:14.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7ef6ffd700 1 -- 172.21.15.114:0/34398626 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0017070 con 0x7f7f001029b0 2024-05-27T07:43:14.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.102+0000 7f7f08113700 1 -- 172.21.15.114:0/34398626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f001a0060 con 0x7f7f001029b0 2024-05-27T07:43:14.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.103+0000 7f7ef6ffd700 1 -- 172.21.15.114:0/34398626 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ef0009de0 con 0x7f7f001029b0 2024-05-27T07:43:14.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.103+0000 7f7f08113700 1 -- 172.21.15.114:0/34398626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f001a7980 con 0x7f7f001029b0 2024-05-27T07:43:14.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.103+0000 7f7ef6ffd700 1 -- 172.21.15.114:0/34398626 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef000f3c0 con 0x7f7f001029b0 2024-05-27T07:43:14.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.105+0000 7f7ef6ffd700 1 -- 172.21.15.114:0/34398626 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f7ef000f560 con 0x7f7f001029b0 2024-05-27T07:43:14.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.106+0000 7f7ef6ffd700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7eec0b2a70 0x7f7eec0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.106+0000 7f7f056ae700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7eec0b2a70 0x7f7eec0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.106+0000 7f7ef6ffd700 1 -- 172.21.15.114:0/34398626 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f7ef00e2800 con 0x7f7f001029b0 2024-05-27T07:43:14.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.107+0000 7f7f056ae700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7eec0b2a70 0x7f7eec0b4f80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7f00107f40 tx=0x7f7efc009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:14.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.108+0000 7f7f16b28b80 1 Processor -- start 2024-05-27T07:43:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.108+0000 7f7f16b28b80 1 -- start start 2024-05-27T07:43:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.108+0000 7f7f16b28b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 0x563b57937b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.108+0000 7f7f16b28b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 0x563b57931be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.108+0000 7f7f16b28b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b57932120 0x563b57932530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.108+0000 7f7f16b28b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563b5785fd70 con 0x563b577b2e40 2024-05-27T07:43:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.108+0000 7f7f16b28b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563b5784f910 con 0x563b57932120 2024-05-27T07:43:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.108+0000 7f7f16b28b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563b57853950 con 0x563b57938070 2024-05-27T07:43:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7f05eaf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 0x563b57937b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7f05eaf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 0x563b57937b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40790/0 (socket says 172.21.15.114:40790) 2024-05-27T07:43:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7f05eaf700 1 -- 172.21.15.114:0/1677160239 learned_addr learned my addr 172.21.15.114:0/1677160239 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7f056ae700 1 --2- 172.21.15.114:0/1677160239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 0x563b57931be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7f066b0700 1 --2- 172.21.15.114:0/1677160239 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b57932120 0x563b57932530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7f05eaf700 1 -- 172.21.15.114:0/1677160239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 msgr2=0x563b57931be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:14.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/1677160239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 0x563b57931be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7f05eaf700 1 -- 172.21.15.114:0/1677160239 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b57932120 msgr2=0x563b57932530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:14.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/1677160239 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b57932120 0x563b57932530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7f05eaf700 1 -- 172.21.15.114:0/1677160239 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b577f8fb0 con 0x563b577b2e40 2024-05-27T07:43:14.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/1677160239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 0x563b57937b30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00dfc00 tx=0x7f7ef00aa0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ecd60d29feb6e3f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:14.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7eebfff700 1 -- 172.21.15.114:0/1677160239 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0020070 con 0x563b577b2e40 2024-05-27T07:43:14.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7eebfff700 1 -- 172.21.15.114:0/1677160239 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ef00aa5d0 con 0x563b577b2e40 2024-05-27T07:43:14.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.109+0000 7f7eebfff700 1 -- 172.21.15.114:0/1677160239 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef00e23f0 con 0x563b577b2e40 2024-05-27T07:43:14.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.110+0000 7f7f16b28b80 1 -- 172.21.15.114:0/1677160239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 msgr2=0x563b57937b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:14.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.110+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/1677160239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 0x563b57937b30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00dfc00 tx=0x7f7ef00aa0e0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.110+0000 7f7f16b28b80 1 -- 172.21.15.114:0/1677160239 shutdown_connections 2024-05-27T07:43:14.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.110+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/1677160239 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b57932120 0x563b57932530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.110+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/1677160239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 0x563b57937b30 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.110+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/1677160239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 0x563b57931be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.110+0000 7f7f16b28b80 1 -- 172.21.15.114:0/1677160239 >> 172.21.15.114:0/1677160239 conn(0x563b577b5770 msgr2=0x563b5792cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:14.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.111+0000 7f7f16b28b80 1 -- 172.21.15.114:0/1677160239 shutdown_connections 2024-05-27T07:43:14.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.111+0000 7f7f16b28b80 1 -- 172.21.15.114:0/1677160239 wait complete. 2024-05-27T07:43:14.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.111+0000 7f7f16b28b80 1 -- start start 2024-05-27T07:43:14.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.111+0000 7f7f16b28b80 1 Processor -- start 2024-05-27T07:43:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f16b28b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b577b2e40 0x563b5793b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f16b28b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57932120 0x563b5793bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f16b28b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 0x563b5793c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f16b28b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563b57755fa0 con 0x563b57932120 2024-05-27T07:43:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f16b28b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563b577b7da0 con 0x563b577b2e40 2024-05-27T07:43:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f16b28b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563b5793eb00 con 0x563b57938070 2024-05-27T07:43:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f16b28b80 10 client.0 _ll_register_callbacks cb 0x563b577e65b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f056ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57932120 0x563b5793bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f056ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57932120 0x563b5793bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40804/0 (socket says 172.21.15.114:40804) 2024-05-27T07:43:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f056ae700 1 -- 172.21.15.114:0/2542232858 learned_addr learned my addr 172.21.15.114:0/2542232858 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f066b0700 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 0x563b5793c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b577b2e40 0x563b5793b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f056ae700 1 -- 172.21.15.114:0/2542232858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 msgr2=0x563b5793c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:14.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f056ae700 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 0x563b5793c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f056ae700 1 -- 172.21.15.114:0/2542232858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b577b2e40 msgr2=0x563b5793b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:14.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f056ae700 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b577b2e40 0x563b5793b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f056ae700 1 -- 172.21.15.114:0/2542232858 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b5779f8d0 con 0x563b57932120 2024-05-27T07:43:14.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b577b2e40 0x563b5793b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:43:14.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f056ae700 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57932120 0x563b5793bea0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f7efc011fb0 tx=0x7f7efc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7bf962a2768d1f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:14.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7ee97fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:14.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7ee97fa700 1 -- 172.21.15.114:0/2542232858 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc003e20 con 0x563b57932120 2024-05-27T07:43:14.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7ee97fa700 1 -- 172.21.15.114:0/2542232858 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f7efc01d620 con 0x563b57932120 2024-05-27T07:43:14.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.112+0000 7f7f16b28b80 1 -- 172.21.15.114:0/2542232858 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ef00aa5d0 con 0x563b57932120 2024-05-27T07:43:14.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.113+0000 7f7ee97fa700 1 -- 172.21.15.114:0/2542232858 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7efc01d820 con 0x563b57932120 2024-05-27T07:43:14.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.113+0000 7f7f16b28b80 10 client.16056 fetch_fsmap learned FSMap version 25 2024-05-27T07:43:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.113+0000 7f7f16b28b80 1 -- 172.21.15.114:0/2542232858 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563b57809760 con 0x563b57932120 2024-05-27T07:43:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.113+0000 7f7ee97fa700 1 -- 172.21.15.114:0/2542232858 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ef00aa5d0 con 0x563b57932120 2024-05-27T07:43:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.113+0000 7f7ee97fa700 1 -- 172.21.15.114:0/2542232858 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f7efc0278d0 con 0x563b57932120 2024-05-27T07:43:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.113+0000 7f7f16b28b80 10 client.16056 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:43:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.113+0000 7f7f16b28b80 10 client.16056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.113+0000 7f7f16b28b80 20 client.16056 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.113+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563b5794bbb0 0x563b5794dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.113+0000 7f7f16b28b80 4 client.16056 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.113+0000 7f7f16b28b80 1 -- 172.21.15.114:0/2542232858 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b57861cc0 con 0x563b5794bbb0 2024-05-27T07:43:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.113+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563b5794bbb0 0x563b5794dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.114+0000 7f7ee97fa700 10 client.16056 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.114+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563b5794bbb0 0x563b5794dfd0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d52238671ec46de2 server_cookie=56042580fc65391e in_seq=0 out_seq=0 2024-05-27T07:43:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7ee97fa700 1 -- 172.21.15.114:0/2542232858 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563b57861cc0 con 0x563b5794bbb0 2024-05-27T07:43:14.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7ee97fa700 10 client.16056 handle_command_reply: tid=0 2024-05-27T07:43:14.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7f16b28b80 1 client.16056 shutdown 2024-05-27T07:43:14.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7f16b28b80 1 -- 172.21.15.114:0/2542232858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57932120 msgr2=0x563b5793bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:14.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57932120 0x563b5793bea0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f7efc011fb0 tx=0x7f7efc0047c0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7f16b28b80 1 -- 172.21.15.114:0/2542232858 shutdown_connections 2024-05-27T07:43:14.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 0x563b5793c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563b5794bbb0 0x563b5794dfd0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b577b2e40 0x563b5793b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/2542232858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57932120 0x563b5793bea0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7ee97fa700 0 client.16056 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:14.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7f16b28b80 1 -- 172.21.15.114:0/2542232858 >> 172.21.15.114:0/2542232858 conn(0x563b577b5770 msgr2=0x563b57930bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:14.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7f16b28b80 1 -- 172.21.15.114:0/2542232858 shutdown_connections 2024-05-27T07:43:14.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7f16b28b80 1 -- 172.21.15.114:0/2542232858 wait complete. 2024-05-27T07:43:14.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.115+0000 7f7f16b28b80 20 client.16056 trim_cache size 0 max 16384 2024-05-27T07:43:14.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.137+0000 7f7f16b28b80 1 Processor -- start 2024-05-27T07:43:14.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.137+0000 7f7f16b28b80 1 -- start start 2024-05-27T07:43:14.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.137+0000 7f7f16b28b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b577b2e40 0x563b57935ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.137+0000 7f7f16b28b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57932120 0x563b57947ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.137+0000 7f7f16b28b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57938070 0x563b579536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.137+0000 7f7f16b28b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001ae6d0 con 0x563b57938070 2024-05-27T07:43:14.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.137+0000 7f7f16b28b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001b0f50 con 0x563b577b2e40 2024-05-27T07:43:14.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.137+0000 7f7f16b28b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001aed40 con 0x563b57932120 2024-05-27T07:43:14.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.137+0000 7f7f066b0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57938070 0x563b579536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.137+0000 7f7f066b0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57938070 0x563b579536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40818/0 (socket says 172.21.15.114:40818) 2024-05-27T07:43:14.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.137+0000 7f7f066b0700 1 -- 172.21.15.114:0/446326417 learned_addr learned my addr 172.21.15.114:0/446326417 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:14.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.137+0000 7f7f056ae700 1 --2- 172.21.15.114:0/446326417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57932120 0x563b57947ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.137+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/446326417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b577b2e40 0x563b57935ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.138+0000 7f7f066b0700 1 -- 172.21.15.114:0/446326417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57932120 msgr2=0x563b57947ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:14.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.138+0000 7f7f066b0700 1 --2- 172.21.15.114:0/446326417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57932120 0x563b57947ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.138+0000 7f7f066b0700 1 -- 172.21.15.114:0/446326417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b577b2e40 msgr2=0x563b57935ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:14.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.138+0000 7f7f066b0700 1 --2- 172.21.15.114:0/446326417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b577b2e40 0x563b57935ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.138+0000 7f7f066b0700 1 -- 172.21.15.114:0/446326417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b57924950 con 0x563b57938070 2024-05-27T07:43:14.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.138+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/446326417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b577b2e40 0x563b57935ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:14.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.138+0000 7f7f066b0700 1 --2- 172.21.15.114:0/446326417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57938070 0x563b579536f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009520 tx=0x7f7ef8009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=485f4429ab6c4154 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:14.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.138+0000 7f7eeb7fe700 1 -- 172.21.15.114:0/446326417 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8015040 con 0x563b57938070 2024-05-27T07:43:14.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.138+0000 7f7eeb7fe700 1 -- 172.21.15.114:0/446326417 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ef800bab0 con 0x563b57938070 2024-05-27T07:43:14.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.138+0000 7f7eeb7fe700 1 -- 172.21.15.114:0/446326417 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8003c40 con 0x563b57938070 2024-05-27T07:43:14.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.139+0000 7f7f16b28b80 1 -- 172.21.15.114:0/446326417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57938070 msgr2=0x563b579536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:14.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.139+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/446326417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57938070 0x563b579536f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009520 tx=0x7f7ef8009550 comp rx=0 tx=0).stop 2024-05-27T07:43:14.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.139+0000 7f7f16b28b80 1 -- 172.21.15.114:0/446326417 shutdown_connections 2024-05-27T07:43:14.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.139+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/446326417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b57938070 0x563b579536f0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.139+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/446326417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b577b2e40 0x563b57935ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.139+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/446326417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57932120 0x563b57947ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.139+0000 7f7f16b28b80 1 -- 172.21.15.114:0/446326417 >> 172.21.15.114:0/446326417 conn(0x563b577b5770 msgr2=0x563b57935490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.139+0000 7f7f16b28b80 1 -- 172.21.15.114:0/446326417 shutdown_connections 2024-05-27T07:43:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.139+0000 7f7f16b28b80 1 -- 172.21.15.114:0/446326417 wait complete. 2024-05-27T07:43:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.139+0000 7f7f16b28b80 1 -- start start 2024-05-27T07:43:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.139+0000 7f7f16b28b80 1 Processor -- start 2024-05-27T07:43:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f16b28b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 0x563b57944350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f16b28b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b57932120 0x563b57940d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f16b28b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 0x563b579412a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f16b28b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001af410 con 0x563b577b2e40 2024-05-27T07:43:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f16b28b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001b0880 con 0x563b57932120 2024-05-27T07:43:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f16b28b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001afae0 con 0x563b57938070 2024-05-27T07:43:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f05eaf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 0x563b57944350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f16b28b80 10 client.0 _ll_register_callbacks cb 0x563b577e65b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f05eaf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 0x563b57944350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40828/0 (socket says 172.21.15.114:40828) 2024-05-27T07:43:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f05eaf700 1 -- 172.21.15.114:0/1835290621 learned_addr learned my addr 172.21.15.114:0/1835290621 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f066b0700 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 0x563b579412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f056ae700 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b57932120 0x563b57940d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f066b0700 1 -- 172.21.15.114:0/1835290621 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b57932120 msgr2=0x563b57940d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f066b0700 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b57932120 0x563b57940d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f066b0700 1 -- 172.21.15.114:0/1835290621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 msgr2=0x563b57944350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f066b0700 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 0x563b57944350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f066b0700 1 -- 172.21.15.114:0/1835290621 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b57941810 con 0x563b57938070 2024-05-27T07:43:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.140+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 0x563b57944350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7f066b0700 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 0x563b579412a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009610 tx=0x7f7ef80075c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30f62c8b9ff4f858 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7ee97fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:43:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7ee97fa700 1 -- 172.21.15.114:0/1835290621 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8007670 con 0x563b57938070 2024-05-27T07:43:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7f16b28b80 1 -- 172.21.15.114:0/1835290621 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ef800bab0 con 0x563b57938070 2024-05-27T07:43:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7ee97fa700 1 -- 172.21.15.114:0/1835290621 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f7ef8003c10 con 0x563b57938070 2024-05-27T07:43:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7ee97fa700 1 -- 172.21.15.114:0/1835290621 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7ef8003e10 con 0x563b57938070 2024-05-27T07:43:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7f16b28b80 10 client.25651 fetch_fsmap learned FSMap version 25 2024-05-27T07:43:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7f16b28b80 1 -- 172.21.15.114:0/1835290621 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563b579ae580 con 0x563b57938070 2024-05-27T07:43:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7ee97fa700 1 -- 172.21.15.114:0/1835290621 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ef800bab0 con 0x563b57938070 2024-05-27T07:43:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7ee97fa700 1 -- 172.21.15.114:0/1835290621 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f7ef801a5e0 con 0x563b57938070 2024-05-27T07:43:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7f16b28b80 10 client.25651 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:43:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7f16b28b80 10 client.25651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7f16b28b80 20 client.25651 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563b579aec40 0x563b579b1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7f16b28b80 4 client.25651 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:43:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.141+0000 7f7f16b28b80 1 -- 172.21.15.114:0/1835290621 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563b5785b4a0 con 0x563b579aec40 2024-05-27T07:43:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.142+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563b579aec40 0x563b579b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.142+0000 7f7ee97fa700 10 client.25651 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.142+0000 7f7f05eaf700 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563b579aec40 0x563b579b1060 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34d659aff5dbb65f server_cookie=245b5a41c1984406 in_seq=0 out_seq=0 2024-05-27T07:43:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.143+0000 7f7ee97fa700 1 -- 172.21.15.114:0/1835290621 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563b5785b4a0 con 0x563b579aec40 2024-05-27T07:43:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.143+0000 7f7ee97fa700 10 client.25651 handle_command_reply: tid=0 2024-05-27T07:43:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.143+0000 7f7f16b28b80 1 client.25651 shutdown 2024-05-27T07:43:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.143+0000 7f7f16b28b80 1 -- 172.21.15.114:0/1835290621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 msgr2=0x563b579412a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.143+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 0x563b579412a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009610 tx=0x7f7ef80075c0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.143+0000 7f7f16b28b80 1 -- 172.21.15.114:0/1835290621 shutdown_connections 2024-05-27T07:43:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.143+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b57938070 0x563b579412a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.143+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563b577b2e40 0x563b57944350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.143+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563b579aec40 0x563b579b1060 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f16b28b80 1 --2- 172.21.15.114:0/1835290621 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b57932120 0x563b57940d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7ee97fa700 0 client.25651 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f16b28b80 1 -- 172.21.15.114:0/1835290621 >> 172.21.15.114:0/1835290621 conn(0x563b577b5770 msgr2=0x563b57937050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f16b28b80 1 -- 172.21.15.114:0/1835290621 shutdown_connections 2024-05-27T07:43:14.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f16b28b80 1 -- 172.21.15.114:0/1835290621 wait complete. 2024-05-27T07:43:14.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f16b28b80 20 client.25651 trim_cache size 0 max 16384 2024-05-27T07:43:14.150 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:43:14.150 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:43:14.150 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:43:14.150 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:43:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f08113700 1 -- 172.21.15.114:0/34398626 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7eec0b2a70 msgr2=0x7f7eec0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f08113700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7eec0b2a70 0x7f7eec0b4f80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7f00107f40 tx=0x7f7efc009260 comp rx=0 tx=0).stop 2024-05-27T07:43:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f08113700 1 -- 172.21.15.114:0/34398626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f001029b0 msgr2=0x7f7f0019aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f08113700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f001029b0 0x7f7f0019aea0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000d980 tx=0x7f7ef000dc90 comp rx=0 tx=0).stop 2024-05-27T07:43:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f08113700 1 -- 172.21.15.114:0/34398626 shutdown_connections 2024-05-27T07:43:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f08113700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7eec0b2a70 0x7f7eec0b4f80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f08113700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f001089e0 0x7f7f0019f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f08113700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f001029b0 0x7f7f0019aea0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f08113700 1 --2- 172.21.15.114:0/34398626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f00103300 0x7f7f0019b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:14.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.144+0000 7f7f08113700 1 -- 172.21.15.114:0/34398626 >> 172.21.15.114:0/34398626 conn(0x7f7f000fe4d0 msgr2=0x7f7f00105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:14.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.145+0000 7f7f08113700 1 -- 172.21.15.114:0/34398626 shutdown_connections 2024-05-27T07:43:14.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:14.145+0000 7f7f08113700 1 -- 172.21.15.114:0/34398626 wait complete. 2024-05-27T07:43:14.241 INFO:tasks.workunit.client.0.smithi114.stdout: 1 227085 14.46 MB/sec warmup 86 sec latency 18.623 ms 2024-05-27T07:43:14.733 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:14 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:43:14.734 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:14 smithi163 systemd[1]: Stopped Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:43:14.734 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:14 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Consumed 3.420s CPU time. 2024-05-27T07:43:14.734 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:14 smithi163 systemd[1]: Starting Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:43:14.734 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:14 smithi163 podman[70658]: 2024-05-27 07:43:14.606734337 +0000 UTC m=+0.017463972 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:43:14.994 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:14 smithi163 podman[70658]: 2024-05-27 07:43:14.735494796 +0000 UTC m=+0.146224416 container create d4a80d6b9be1cb250812733b1452b9f51f00bd7b1d5f6a7a3e1d804a0475a27a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:43:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.009+0000 7f66682aa640 1 -- 172.21.15.114:0/2711279064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66601032e0 msgr2=0x7f666010f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.009+0000 7f66682aa640 1 --2- 172.21.15.114:0/2711279064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66601032e0 0x7f666010f6b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6658009ca0 tx=0x7f665802a8c0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.009+0000 7f66682aa640 1 -- 172.21.15.114:0/2711279064 shutdown_connections 2024-05-27T07:43:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.009+0000 7f66682aa640 1 --2- 172.21.15.114:0/2711279064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66601032e0 0x7f666010f6b0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.009+0000 7f66682aa640 1 --2- 172.21.15.114:0/2711279064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6660109030 0x7f6660102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.009+0000 7f66682aa640 1 --2- 172.21.15.114:0/2711279064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6660108b90 0x7f6660102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.009+0000 7f66682aa640 1 -- 172.21.15.114:0/2711279064 >> 172.21.15.114:0/2711279064 conn(0x7f66600782d0 msgr2=0x7f6660078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.009+0000 7f66682aa640 1 -- 172.21.15.114:0/2711279064 shutdown_connections 2024-05-27T07:43:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.009+0000 7f66682aa640 1 -- 172.21.15.114:0/2711279064 wait complete. 2024-05-27T07:43:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.010+0000 7f66682aa640 1 Processor -- start 2024-05-27T07:43:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.010+0000 7f66682aa640 1 -- start start 2024-05-27T07:43:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.010+0000 7f66682aa640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66601032e0 0x7f666010d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.010+0000 7f66682aa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6660108b90 0x7f666010dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.010+0000 7f66682aa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6660109030 0x7f666010e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.010+0000 7f66682aa640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6660112df0 con 0x7f6660109030 2024-05-27T07:43:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f66682aa640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6660112c70 con 0x7f66601032e0 2024-05-27T07:43:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f66682aa640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6660112f70 con 0x7f6660108b90 2024-05-27T07:43:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f6665845640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6660108b90 0x7f666010dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f6666046640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66601032e0 0x7f666010d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f6666046640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66601032e0 0x7f666010d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:40330/0 (socket says 172.21.15.114:40330) 2024-05-27T07:43:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f6665845640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6660108b90 0x7f666010dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:52882/0 (socket says 172.21.15.114:52882) 2024-05-27T07:43:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f6666046640 1 -- 172.21.15.114:0/388127994 learned_addr learned my addr 172.21.15.114:0/388127994 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f6666847640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6660109030 0x7f666010e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f6666847640 1 -- 172.21.15.114:0/388127994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6660108b90 msgr2=0x7f666010dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f6666847640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6660108b90 0x7f666010dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f6666847640 1 -- 172.21.15.114:0/388127994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66601032e0 msgr2=0x7f666010d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f6666847640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66601032e0 0x7f666010d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f6666847640 1 -- 172.21.15.114:0/388127994 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66601a8be0 con 0x7f6660109030 2024-05-27T07:43:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f6666046640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66601032e0 0x7f666010d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f6666847640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6660109030 0x7f666010e190 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f6658005e30 tx=0x7f66580317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f66577fe640 1 -- 172.21.15.114:0/388127994 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6658003b70 con 0x7f6660109030 2024-05-27T07:43:15.015 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:43:15.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:43:15.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f66682aa640 1 -- 172.21.15.114:0/388127994 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66601a8e70 con 0x7f6660109030 2024-05-27T07:43:15.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f66577fe640 1 -- 172.21.15.114:0/388127994 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6658003d10 con 0x7f6660109030 2024-05-27T07:43:15.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f66577fe640 1 -- 172.21.15.114:0/388127994 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665802f970 con 0x7f6660109030 2024-05-27T07:43:15.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.011+0000 7f66682aa640 1 -- 172.21.15.114:0/388127994 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66601a9350 con 0x7f6660109030 2024-05-27T07:43:15.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.014+0000 7f666f234740 1 Processor -- start 2024-05-27T07:43:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.015+0000 7f66577fe640 1 -- 172.21.15.114:0/388127994 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f665802fb10 con 0x7f6660109030 2024-05-27T07:43:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.015+0000 7f666f234740 1 -- start start 2024-05-27T07:43:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.015+0000 7f666f234740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594226741b0 0x559422a153d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.015+0000 7f666f234740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55942264b2f0 0x559422a15910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.015+0000 7f666f234740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594226534d0 0x559422a15e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.015+0000 7f666f234740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559422987c30 con 0x5594226741b0 2024-05-27T07:43:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.015+0000 7f666f234740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5594229777d0 con 0x55942264b2f0 2024-05-27T07:43:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.015+0000 7f666f234740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55942297b810 con 0x5594226534d0 2024-05-27T07:43:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.015+0000 7f6665845640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594226741b0 0x559422a153d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.015+0000 7f6665845640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594226741b0 0x559422a153d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56984/0 (socket says 172.21.15.114:56984) 2024-05-27T07:43:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.015+0000 7f6665845640 1 -- 172.21.15.114:0/2037974690 learned_addr learned my addr 172.21.15.114:0/2037974690 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f6665845640 1 --2- 172.21.15.114:0/2037974690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594226534d0 0x559422a15e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f6666046640 1 --2- 172.21.15.114:0/2037974690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55942264b2f0 0x559422a15910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f66577fe640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f664c0b2950 0x7f664c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f6665845640 1 -- 172.21.15.114:0/2037974690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594226534d0 msgr2=0x559422a15e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f6665845640 1 --2- 172.21.15.114:0/2037974690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594226534d0 0x559422a15e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f6665845640 1 -- 172.21.15.114:0/2037974690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55942264b2f0 msgr2=0x559422a15910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f6665845640 1 --2- 172.21.15.114:0/2037974690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55942264b2f0 0x559422a15910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f6665845640 1 -- 172.21.15.114:0/2037974690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55942268c210 con 0x5594226741b0 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f6666847640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f664c0b2950 0x7f664c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f6665845640 1 --2- 172.21.15.114:0/2037974690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594226741b0 0x559422a153d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6650013700 tx=0x7f6650009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995d822e2ea7d5cd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f66577fe640 1 -- 172.21.15.114:0/388127994 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f6658103ab0 con 0x7f6660109030 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f6654bfe640 1 -- 172.21.15.114:0/2037974690 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6650013d40 con 0x5594226741b0 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f6654bfe640 1 -- 172.21.15.114:0/2037974690 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66500054d0 con 0x5594226741b0 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.016+0000 7f6654bfe640 1 -- 172.21.15.114:0/2037974690 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66500117e0 con 0x5594226741b0 2024-05-27T07:43:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.017+0000 7f666f234740 1 -- 172.21.15.114:0/2037974690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594226741b0 msgr2=0x559422a153d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.017+0000 7f666f234740 1 --2- 172.21.15.114:0/2037974690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594226741b0 0x559422a153d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6650013700 tx=0x7f6650009570 comp rx=0 tx=0).stop 2024-05-27T07:43:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.017+0000 7f6666847640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f664c0b2950 0x7f664c0b4e60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6658004af0 tx=0x7f66580095c0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.017+0000 7f666f234740 1 -- 172.21.15.114:0/2037974690 shutdown_connections 2024-05-27T07:43:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.017+0000 7f666f234740 1 --2- 172.21.15.114:0/2037974690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594226534d0 0x559422a15e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.017+0000 7f666f234740 1 --2- 172.21.15.114:0/2037974690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55942264b2f0 0x559422a15910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.017+0000 7f666f234740 1 --2- 172.21.15.114:0/2037974690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594226741b0 0x559422a153d0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.017+0000 7f666f234740 1 -- 172.21.15.114:0/2037974690 >> 172.21.15.114:0/2037974690 conn(0x559422668300 msgr2=0x559422a11b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.017+0000 7f666f234740 1 -- 172.21.15.114:0/2037974690 shutdown_connections 2024-05-27T07:43:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.017+0000 7f666f234740 1 -- 172.21.15.114:0/2037974690 wait complete. 2024-05-27T07:43:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.018+0000 7f666f234740 1 -- start start 2024-05-27T07:43:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.018+0000 7f666f234740 1 Processor -- start 2024-05-27T07:43:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.018+0000 7f666f234740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55942264b2f0 0x559422a21570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.018+0000 7f666f234740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594226534d0 0x559422a21ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.018+0000 7f666f234740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594226741b0 0x559422a1f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.018+0000 7f666f234740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55942265d790 con 0x55942264b2f0 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.018+0000 7f666f234740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559422933890 con 0x5594226741b0 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.018+0000 7f666f234740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559422a224a0 con 0x5594226534d0 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.018+0000 7f666f234740 10 client.0 _ll_register_callbacks cb 0x55942268ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.018+0000 7f6666046640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594226534d0 0x559422a21ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.018+0000 7f6665845640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55942264b2f0 0x559422a21570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.018+0000 7f6666046640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594226534d0 0x559422a21ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:52914/0 (socket says 172.21.15.114:52914) 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.018+0000 7f6666046640 1 -- 172.21.15.114:0/3761451445 learned_addr learned my addr 172.21.15.114:0/3761451445 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f6665845640 1 --2- 172.21.15.114:0/3761451445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594226741b0 0x559422a1f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f6666046640 1 -- 172.21.15.114:0/3761451445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594226741b0 msgr2=0x559422a1f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f6666046640 1 --2- 172.21.15.114:0/3761451445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594226741b0 0x559422a1f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f6666046640 1 -- 172.21.15.114:0/3761451445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55942264b2f0 msgr2=0x559422a21570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f6666046640 1 --2- 172.21.15.114:0/3761451445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55942264b2f0 0x559422a21570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f6666046640 1 -- 172.21.15.114:0/3761451445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559422758620 con 0x5594226534d0 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f6666046640 1 --2- 172.21.15.114:0/3761451445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594226534d0 0x559422a21ab0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f665c012d20 tx=0x7f665c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e31d6aa1b4153248 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f6649ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f6649ffb640 1 -- 172.21.15.114:0/3761451445 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c0120d0 con 0x5594226534d0 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f666f234740 1 -- 172.21.15.114:0/3761451445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66500054d0 con 0x5594226534d0 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f6649ffb640 1 -- 172.21.15.114:0/3761451445 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f665c0047a0 con 0x5594226534d0 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f6649ffb640 1 -- 172.21.15.114:0/3761451445 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f665c012270 con 0x5594226534d0 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f666f234740 10 client.25663 fetch_fsmap learned FSMap version 25 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f666f234740 1 -- 172.21.15.114:0/3761451445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559422a2e8f0 con 0x5594226534d0 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.019+0000 7f6649ffb640 1 -- 172.21.15.114:0/3761451445 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66500054d0 con 0x5594226534d0 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.020+0000 7f6649ffb640 1 -- 172.21.15.114:0/3761451445 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f665c010700 con 0x5594226534d0 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.020+0000 7f666f234740 10 client.25663 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.020+0000 7f666f234740 10 client.25663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.020+0000 7f666f234740 20 client.25663 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:15.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.020+0000 7f666f234740 1 --2- 172.21.15.114:0/3761451445 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559422632d80 0x559422a31d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.020+0000 7f666f234740 4 client.25663 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:15.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.020+0000 7f666f234740 1 -- 172.21.15.114:0/3761451445 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559422989b80 con 0x559422632d80 2024-05-27T07:43:15.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.020+0000 7f6665845640 1 --2- 172.21.15.114:0/3761451445 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559422632d80 0x559422a31d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.020+0000 7f6649ffb640 10 client.25663 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:15.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.020+0000 7f6665845640 1 --2- 172.21.15.114:0/3761451445 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559422632d80 0x559422a31d70 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f32305933402806 server_cookie=43165fc1d778fdd8 in_seq=0 out_seq=0 2024-05-27T07:43:15.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.022+0000 7f6649ffb640 1 -- 172.21.15.114:0/3761451445 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559422989b80 con 0x559422632d80 2024-05-27T07:43:15.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.022+0000 7f6649ffb640 10 client.25663 handle_command_reply: tid=0 2024-05-27T07:43:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.022+0000 7f666f234740 1 client.25663 shutdown 2024-05-27T07:43:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.023+0000 7f666f234740 1 -- 172.21.15.114:0/3761451445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594226534d0 msgr2=0x559422a21ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.023+0000 7f666f234740 1 --2- 172.21.15.114:0/3761451445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594226534d0 0x559422a21ab0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f665c012d20 tx=0x7f665c00f790 comp rx=0 tx=0).stop 2024-05-27T07:43:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.023+0000 7f666f234740 1 -- 172.21.15.114:0/3761451445 shutdown_connections 2024-05-27T07:43:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.023+0000 7f666f234740 1 --2- 172.21.15.114:0/3761451445 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559422632d80 0x559422a31d70 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.023+0000 7f666f234740 1 --2- 172.21.15.114:0/3761451445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594226741b0 0x559422a1f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.023+0000 7f6649ffb640 0 client.25663 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.023+0000 7f666f234740 1 --2- 172.21.15.114:0/3761451445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594226534d0 0x559422a21ab0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.023+0000 7f666f234740 1 --2- 172.21.15.114:0/3761451445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55942264b2f0 0x559422a21570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.023+0000 7f666f234740 1 -- 172.21.15.114:0/3761451445 >> 172.21.15.114:0/3761451445 conn(0x559422668300 msgr2=0x559422a15200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.023+0000 7f666f234740 1 -- 172.21.15.114:0/3761451445 shutdown_connections 2024-05-27T07:43:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.023+0000 7f666f234740 1 -- 172.21.15.114:0/3761451445 wait complete. 2024-05-27T07:43:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.023+0000 7f666f234740 20 client.25663 trim_cache size 0 max 16384 2024-05-27T07:43:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.040+0000 7f666f234740 1 Processor -- start 2024-05-27T07:43:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.040+0000 7f666f234740 1 -- start start 2024-05-27T07:43:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.040+0000 7f666f234740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559422632d80 0x559422a1f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.040+0000 7f666f234740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55942264b2f0 0x559422a1f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.040+0000 7f666f234740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594226534d0 0x559422a19020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.040+0000 7f666f234740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f66601b1440 con 0x55942264b2f0 2024-05-27T07:43:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.040+0000 7f666f234740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f66601b3cc0 con 0x5594226534d0 2024-05-27T07:43:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.040+0000 7f666f234740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f66601b1ab0 con 0x559422632d80 2024-05-27T07:43:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.040+0000 7f6666046640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55942264b2f0 0x559422a1f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.040+0000 7f6666046640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55942264b2f0 0x559422a1f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57002/0 (socket says 172.21.15.114:57002) 2024-05-27T07:43:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.040+0000 7f6666046640 1 -- 172.21.15.114:0/3096078278 learned_addr learned my addr 172.21.15.114:0/3096078278 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.040+0000 7f6665845640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559422632d80 0x559422a1f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.040+0000 7f6665845640 1 --2- 172.21.15.114:0/3096078278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594226534d0 0x559422a19020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.041+0000 7f6666046640 1 -- 172.21.15.114:0/3096078278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559422632d80 msgr2=0x559422a1f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.041+0000 7f6666046640 1 --2- 172.21.15.114:0/3096078278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559422632d80 0x559422a1f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.041+0000 7f6666046640 1 -- 172.21.15.114:0/3096078278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594226534d0 msgr2=0x559422a19020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.041+0000 7f6666046640 1 --2- 172.21.15.114:0/3096078278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594226534d0 0x559422a19020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.041+0000 7f6666046640 1 -- 172.21.15.114:0/3096078278 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559422a1fc90 con 0x55942264b2f0 2024-05-27T07:43:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.041+0000 7f6666046640 1 --2- 172.21.15.114:0/3096078278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55942264b2f0 0x559422a1f750 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f665c000c00 tx=0x7f665c012990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1722159b2113393 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.041+0000 7f664bfff640 1 -- 172.21.15.114:0/3096078278 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c011ea0 con 0x55942264b2f0 2024-05-27T07:43:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.041+0000 7f664bfff640 1 -- 172.21.15.114:0/3096078278 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f665c0108e0 con 0x55942264b2f0 2024-05-27T07:43:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.041+0000 7f664bfff640 1 -- 172.21.15.114:0/3096078278 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c004900 con 0x55942264b2f0 2024-05-27T07:43:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.042+0000 7f666f234740 1 -- 172.21.15.114:0/3096078278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55942264b2f0 msgr2=0x559422a1f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.042+0000 7f666f234740 1 --2- 172.21.15.114:0/3096078278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55942264b2f0 0x559422a1f750 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f665c000c00 tx=0x7f665c012990 comp rx=0 tx=0).stop 2024-05-27T07:43:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.042+0000 7f666f234740 1 -- 172.21.15.114:0/3096078278 shutdown_connections 2024-05-27T07:43:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.042+0000 7f666f234740 1 --2- 172.21.15.114:0/3096078278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594226534d0 0x559422a19020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.042+0000 7f666f234740 1 --2- 172.21.15.114:0/3096078278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55942264b2f0 0x559422a1f750 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.042+0000 7f666f234740 1 --2- 172.21.15.114:0/3096078278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559422632d80 0x559422a1f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.042+0000 7f666f234740 1 -- 172.21.15.114:0/3096078278 >> 172.21.15.114:0/3096078278 conn(0x559422668300 msgr2=0x559422a1e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.042+0000 7f666f234740 1 -- 172.21.15.114:0/3096078278 shutdown_connections 2024-05-27T07:43:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.042+0000 7f666f234740 1 -- 172.21.15.114:0/3096078278 wait complete. 2024-05-27T07:43:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.042+0000 7f666f234740 1 -- start start 2024-05-27T07:43:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.042+0000 7f666f234740 1 Processor -- start 2024-05-27T07:43:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f666f234740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559422632d80 0x559422a26e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f666f234740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55942264b2f0 0x559422a27340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f666f234740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594226534d0 0x559422a92490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f666f234740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f66601b2180 con 0x5594226534d0 2024-05-27T07:43:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f666f234740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f66601b35f0 con 0x55942264b2f0 2024-05-27T07:43:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f666f234740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f66601b2850 con 0x559422632d80 2024-05-27T07:43:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f666f234740 10 client.0 _ll_register_callbacks cb 0x7f665c009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f6665845640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594226534d0 0x559422a92490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f6665845640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559422632d80 0x559422a26e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f6665845640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559422632d80 0x559422a26e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:52930/0 (socket says 172.21.15.114:52930) 2024-05-27T07:43:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f6665845640 1 -- 172.21.15.114:0/1545647399 learned_addr learned my addr 172.21.15.114:0/1545647399 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f6666046640 1 --2- 172.21.15.114:0/1545647399 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55942264b2f0 0x559422a27340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f6666046640 1 -- 172.21.15.114:0/1545647399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559422632d80 msgr2=0x559422a26e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f6666046640 1 --2- 172.21.15.114:0/1545647399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559422632d80 0x559422a26e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f6666046640 1 -- 172.21.15.114:0/1545647399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594226534d0 msgr2=0x559422a92490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f6666046640 1 --2- 172.21.15.114:0/1545647399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594226534d0 0x559422a92490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f6666046640 1 -- 172.21.15.114:0/1545647399 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559422a92a20 con 0x55942264b2f0 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.043+0000 7f6665845640 1 --2- 172.21.15.114:0/1545647399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559422632d80 0x559422a26e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f6666046640 1 --2- 172.21.15.114:0/1545647399 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55942264b2f0 0x559422a27340 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f665c00a010 tx=0x7f665c010660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc8316b432860e4c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f6649ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f6649ffb640 1 -- 172.21.15.114:0/1545647399 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c036030 con 0x55942264b2f0 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f6649ffb640 1 -- 172.21.15.114:0/1545647399 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9466+0+0 (secure 0 0 0) 0x7f665c018040 con 0x55942264b2f0 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f666f234740 1 -- 172.21.15.114:0/1545647399 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f665c0108e0 con 0x55942264b2f0 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f6649ffb640 1 -- 172.21.15.114:0/1545647399 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f665c005250 con 0x55942264b2f0 2024-05-27T07:43:15.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f666f234740 10 client.25610 fetch_fsmap learned FSMap version 25 2024-05-27T07:43:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f666f234740 1 -- 172.21.15.114:0/1545647399 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559422a99d70 con 0x55942264b2f0 2024-05-27T07:43:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f6649ffb640 1 -- 172.21.15.114:0/1545647399 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f665c0108e0 con 0x55942264b2f0 2024-05-27T07:43:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f6649ffb640 1 -- 172.21.15.114:0/1545647399 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f665c01c540 con 0x55942264b2f0 2024-05-27T07:43:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f666f234740 10 client.25610 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T07:43:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f666f234740 10 client.25610 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f666f234740 20 client.25610 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f666f234740 1 --2- 172.21.15.114:0/1545647399 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5594226741b0 0x559422a9c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f666f234740 4 client.25610 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:43:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.044+0000 7f666f234740 1 -- 172.21.15.114:0/1545647399 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559422983360 con 0x5594226741b0 2024-05-27T07:43:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.045+0000 7f6665845640 1 --2- 172.21.15.114:0/1545647399 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5594226741b0 0x559422a9c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.045+0000 7f6649ffb640 10 client.25610 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.045+0000 7f6665845640 1 --2- 172.21.15.114:0/1545647399 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5594226741b0 0x559422a9c3d0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19634b552640e2e9 server_cookie=1dc0131e634e827c in_seq=0 out_seq=0 2024-05-27T07:43:15.241 INFO:tasks.workunit.client.0.smithi114.stdout: 1 228882 14.37 MB/sec warmup 87 sec latency 149.875 ms 2024-05-27T07:43:15.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.364+0000 7f6649ffb640 1 -- 172.21.15.114:0/1545647399 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x559422983360 con 0x5594226741b0 2024-05-27T07:43:15.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.364+0000 7f6649ffb640 10 client.25610 handle_command_reply: tid=0 2024-05-27T07:43:15.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.364+0000 7f666f234740 1 client.25610 shutdown 2024-05-27T07:43:15.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.365+0000 7f666f234740 1 -- 172.21.15.114:0/1545647399 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55942264b2f0 msgr2=0x559422a27340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.365+0000 7f666f234740 1 --2- 172.21.15.114:0/1545647399 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55942264b2f0 0x559422a27340 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f665c00a010 tx=0x7f665c010660 comp rx=0 tx=0).stop 2024-05-27T07:43:15.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.365+0000 7f666f234740 1 -- 172.21.15.114:0/1545647399 shutdown_connections 2024-05-27T07:43:15.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.365+0000 7f666f234740 1 --2- 172.21.15.114:0/1545647399 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5594226741b0 0x559422a9c3d0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.365+0000 7f666f234740 1 --2- 172.21.15.114:0/1545647399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594226534d0 0x559422a92490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.365+0000 7f666f234740 1 --2- 172.21.15.114:0/1545647399 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55942264b2f0 0x559422a27340 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.365+0000 7f666f234740 1 --2- 172.21.15.114:0/1545647399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559422632d80 0x559422a26e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.365+0000 7f666f234740 1 -- 172.21.15.114:0/1545647399 >> 172.21.15.114:0/1545647399 conn(0x559422668300 msgr2=0x559422a1e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:15.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.365+0000 7f6649ffb640 0 client.25610 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:15.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.365+0000 7f666f234740 1 -- 172.21.15.114:0/1545647399 shutdown_connections 2024-05-27T07:43:15.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.365+0000 7f666f234740 1 -- 172.21.15.114:0/1545647399 wait complete. 2024-05-27T07:43:15.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.365+0000 7f666f234740 20 client.25610 trim_cache size 0 max 16384 2024-05-27T07:43:15.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:43:15.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:43:15.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:43:15.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.366+0000 7f66682aa640 1 -- 172.21.15.114:0/388127994 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f664c0b2950 msgr2=0x7f664c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.366+0000 7f66682aa640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f664c0b2950 0x7f664c0b4e60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6658004af0 tx=0x7f66580095c0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.366+0000 7f66682aa640 1 -- 172.21.15.114:0/388127994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6660109030 msgr2=0x7f666010e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.366+0000 7f66682aa640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6660109030 0x7f666010e190 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f6658005e30 tx=0x7f66580317c0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.366+0000 7f66682aa640 1 -- 172.21.15.114:0/388127994 shutdown_connections 2024-05-27T07:43:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.366+0000 7f66682aa640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f664c0b2950 0x7f664c0b4e60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.366+0000 7f66682aa640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6660109030 0x7f666010e190 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.366+0000 7f66682aa640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6660108b90 0x7f666010dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.366+0000 7f66682aa640 1 --2- 172.21.15.114:0/388127994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66601032e0 0x7f666010d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.366+0000 7f66682aa640 1 -- 172.21.15.114:0/388127994 >> 172.21.15.114:0/388127994 conn(0x7f66600782d0 msgr2=0x7f66601008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.366+0000 7f66682aa640 1 -- 172.21.15.114:0/388127994 shutdown_connections 2024-05-27T07:43:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:15.366+0000 7f66682aa640 1 -- 172.21.15.114:0/388127994 wait complete. 2024-05-27T07:43:15.387 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:15.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:14 smithi163 podman[70658]: 2024-05-27 07:43:14.996584533 +0000 UTC m=+0.407314163 container init d4a80d6b9be1cb250812733b1452b9f51f00bd7b1d5f6a7a3e1d804a0475a27a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-27T07:43:15.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:14 smithi163 podman[70658]: 2024-05-27 07:43:14.99930029 +0000 UTC m=+0.410029919 container start d4a80d6b9be1cb250812733b1452b9f51f00bd7b1d5f6a7a3e1d804a0475a27a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:43:15.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:15 smithi163 ceph-mds[70677]: -- 172.21.15.163:0/80234775 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c4f314da40 con 0x55c4f3195180 2024-05-27T07:43:15.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: starting mds.b at 2024-05-27T07:43:15.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:15 smithi163 bash[70658]: d4a80d6b9be1cb250812733b1452b9f51f00bd7b1d5f6a7a3e1d804a0475a27a 2024-05-27T07:43:15.475 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:15 smithi163 systemd[1]: Started Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:43:15.890 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:15 smithi114 ceph-mon[32038]: pgmap v110: 129 pgs: 129 active+clean; 49 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 422 op/s 2024-05-27T07:43:15.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:15 smithi114 ceph-mon[32038]: daemon mds.b restarted 2024-05-27T07:43:15.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:15 smithi163 ceph-mon[36451]: pgmap v110: 129 pgs: 129 active+clean; 49 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 422 op/s 2024-05-27T07:43:15.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:15 smithi163 ceph-mon[36451]: daemon mds.b restarted 2024-05-27T07:43:15.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:15 smithi186 ceph-mon[36647]: pgmap v110: 129 pgs: 129 active+clean; 49 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 422 op/s 2024-05-27T07:43:15.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:15 smithi186 ceph-mon[36647]: daemon mds.b restarted 2024-05-27T07:43:16.241 INFO:tasks.workunit.client.0.smithi114.stdout: 1 231034 14.36 MB/sec warmup 88 sec latency 169.009 ms 2024-05-27T07:43:16.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:16 smithi114 ceph-mon[32038]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T07:43:16.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:16 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6838/2656639979,v1:172.21.15.163:6839/2656639979] up:boot 2024-05-27T07:43:16.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:16 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:43:16.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:16 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:43:16.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:16 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:43:16.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:16 smithi163 ceph-mon[36451]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T07:43:16.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:16 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6838/2656639979,v1:172.21.15.163:6839/2656639979] up:boot 2024-05-27T07:43:16.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:16 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:43:16.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:16 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:43:16.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:16 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:43:16.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:16 smithi186 ceph-mon[36647]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T07:43:16.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:16 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6838/2656639979,v1:172.21.15.163:6839/2656639979] up:boot 2024-05-27T07:43:16.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:16 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:43:16.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:16 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:43:16.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:16 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:43:17.241 INFO:tasks.workunit.client.0.smithi114.stdout: 1 233546 14.34 MB/sec warmup 89 sec latency 26.441 ms 2024-05-27T07:43:17.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.255+0000 7f322302e700 1 -- 172.21.15.114:0/4245594228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c107e00 msgr2=0x7f321c108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.255+0000 7f322302e700 1 --2- 172.21.15.114:0/4245594228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c107e00 0x7f321c108210 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f32080099c0 tx=0x7f3208009cd0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.255+0000 7f322302e700 1 -- 172.21.15.114:0/4245594228 shutdown_connections 2024-05-27T07:43:17.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.255+0000 7f322302e700 1 --2- 172.21.15.114:0/4245594228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f321c108750 0x7f321c1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.255+0000 7f322302e700 1 --2- 172.21.15.114:0/4245594228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f321c1034d0 0x7f321c1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.255+0000 7f322302e700 1 --2- 172.21.15.114:0/4245594228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c107e00 0x7f321c108210 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.255+0000 7f322302e700 1 -- 172.21.15.114:0/4245594228 >> 172.21.15.114:0/4245594228 conn(0x7f321c0fef20 msgr2=0x7f321c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:17.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.255+0000 7f322302e700 1 -- 172.21.15.114:0/4245594228 shutdown_connections 2024-05-27T07:43:17.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.255+0000 7f322302e700 1 -- 172.21.15.114:0/4245594228 wait complete. 2024-05-27T07:43:17.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.256+0000 7f322302e700 1 Processor -- start 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.256+0000 7f322302e700 1 -- start start 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.256+0000 7f322302e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c1034d0 0x7f321c19bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.256+0000 7f322302e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f321c107e00 0x7f321c19c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.256+0000 7f322302e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f321c108750 0x7f321c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.256+0000 7f322302e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f321c113360 con 0x7f321c1034d0 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.256+0000 7f322302e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f321c1131e0 con 0x7f321c107e00 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.256+0000 7f322302e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f321c1134e0 con 0x7f321c108750 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.256+0000 7f3220dca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c1034d0 0x7f321c19bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.256+0000 7f3220dca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c1034d0 0x7f321c19bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57016/0 (socket says 172.21.15.114:57016) 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.256+0000 7f3220dca700 1 -- 172.21.15.114:0/3199856931 learned_addr learned my addr 172.21.15.114:0/3199856931 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.256+0000 7f321bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f321c107e00 0x7f321c19c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.256+0000 7f32215cb700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f321c108750 0x7f321c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.257+0000 7f3220dca700 1 -- 172.21.15.114:0/3199856931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f321c108750 msgr2=0x7f321c19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.257+0000 7f3220dca700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f321c108750 0x7f321c19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.257+0000 7f3220dca700 1 -- 172.21.15.114:0/3199856931 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f321c107e00 msgr2=0x7f321c19c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.257+0000 7f3220dca700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f321c107e00 0x7f321c19c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.257+0000 7f3220dca700 1 -- 172.21.15.114:0/3199856931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321c1a7070 con 0x7f321c1034d0 2024-05-27T07:43:17.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.257+0000 7f321bfff700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f321c107e00 0x7f321c19c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:17.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.257+0000 7f3220dca700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c1034d0 0x7f321c19bd20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f321000e9a0 tx=0x7f321000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:17.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.257+0000 7f3219ffb700 1 -- 172.21.15.114:0/3199856931 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321000ca10 con 0x7f321c1034d0 2024-05-27T07:43:17.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.257+0000 7f3219ffb700 1 -- 172.21.15.114:0/3199856931 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f321000cbb0 con 0x7f321c1034d0 2024-05-27T07:43:17.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.257+0000 7f322302e700 1 -- 172.21.15.114:0/3199856931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f321c1a7360 con 0x7f321c1034d0 2024-05-27T07:43:17.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.257+0000 7f3219ffb700 1 -- 172.21.15.114:0/3199856931 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3210018680 con 0x7f321c1034d0 2024-05-27T07:43:17.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.257+0000 7f322302e700 1 -- 172.21.15.114:0/3199856931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f321c1a7870 con 0x7f321c1034d0 2024-05-27T07:43:17.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.258+0000 7f322302e700 1 -- 172.21.15.114:0/3199856931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f321c0512b0 con 0x7f321c1034d0 2024-05-27T07:43:17.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.260+0000 7f3219ffb700 1 -- 172.21.15.114:0/3199856931 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f3210010430 con 0x7f321c1034d0 2024-05-27T07:43:17.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.260+0000 7f3219ffb700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f320c0b29f0 0x7f320c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:17.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.261+0000 7f321bfff700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f320c0b29f0 0x7f320c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:17.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.261+0000 7f3219ffb700 1 -- 172.21.15.114:0/3199856931 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9466+0+0 (secure 0 0 0) 0x7f32100e2e80 con 0x7f321c1034d0 2024-05-27T07:43:17.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.261+0000 7f321bfff700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f320c0b29f0 0x7f320c0b4f00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3208000c00 tx=0x7f320801a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:17.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.264+0000 7f3219ffb700 1 -- 172.21.15.114:0/3199856931 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f32100aa4b0 con 0x7f321c1034d0 2024-05-27T07:43:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.550+0000 7f322302e700 1 -- 172.21.15.114:0/3199856931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f321c02e5e0 con 0x7f321c1034d0 2024-05-27T07:43:17.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.551+0000 7f3219ffb700 1 -- 172.21.15.114:0/3199856931 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11827 (secure 0 0 0) 0x7f32100a9db0 con 0x7f321c1034d0 2024-05-27T07:43:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:43:17.551 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:43:15.618543+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25604":{"gid":25604,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/2656639979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":2656639979},{"type":"v1","addr":"172.21.15.163:6839","nonce":2656639979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:43:17.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.554+0000 7f322302e700 1 -- 172.21.15.114:0/3199856931 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f320c0b29f0 msgr2=0x7f320c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.554+0000 7f322302e700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f320c0b29f0 0x7f320c0b4f00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3208000c00 tx=0x7f320801a040 comp rx=0 tx=0).stop 2024-05-27T07:43:17.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.554+0000 7f322302e700 1 -- 172.21.15.114:0/3199856931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c1034d0 msgr2=0x7f321c19bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.554+0000 7f322302e700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c1034d0 0x7f321c19bd20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f321000e9a0 tx=0x7f321000ecb0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.554+0000 7f322302e700 1 -- 172.21.15.114:0/3199856931 shutdown_connections 2024-05-27T07:43:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.554+0000 7f322302e700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f321c108750 0x7f321c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.554+0000 7f322302e700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f321c1034d0 0x7f321c19bd20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.554+0000 7f322302e700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f320c0b29f0 0x7f320c0b4f00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.554+0000 7f322302e700 1 --2- 172.21.15.114:0/3199856931 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f321c107e00 0x7f321c19c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.554+0000 7f322302e700 1 -- 172.21.15.114:0/3199856931 >> 172.21.15.114:0/3199856931 conn(0x7f321c0fef20 msgr2=0x7f321c1096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.554+0000 7f322302e700 1 -- 172.21.15.114:0/3199856931 shutdown_connections 2024-05-27T07:43:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:17.554+0000 7f322302e700 1 -- 172.21.15.114:0/3199856931 wait complete. 2024-05-27T07:43:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 27 2024-05-27T07:43:17.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:17 smithi114 ceph-mon[32038]: pgmap v112: 129 pgs: 129 active+clean; 55 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 382 op/s 2024-05-27T07:43:17.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:17 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3199856931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:17.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-mon[36451]: pgmap v112: 129 pgs: 129 active+clean; 55 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 382 op/s 2024-05-27T07:43:17.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3199856931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:17.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f07c9ac8700 time 2024-05-27T07:43:17.743252+0000 2024-05-27T07:43:17.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:43:17.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:17.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f07dc30dba6] 2024-05-27T07:43:17.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f07dc30ddd1] 2024-05-27T07:43:17.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c4f086923d] 2024-05-27T07:43:17.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c4f0854971] 2024-05-27T07:43:17.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c4f0852fa0] 2024-05-27T07:43:17.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c4f0853651] 2024-05-27T07:43:17.975 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c4f07e8347] 2024-05-27T07:43:17.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c4f04e487e] 2024-05-27T07:43:17.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c4f04e9f2f] 2024-05-27T07:43:17.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c4f04f4969] 2024-05-27T07:43:17.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c4f088d48c] 2024-05-27T07:43:17.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c4f08969cb] 2024-05-27T07:43:17.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 13: (Thread::entry_wrapper()+0x43) [0x7f07dc2dc72d] 2024-05-27T07:43:17.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 14: (Thread::_entry_func(void*)+0xd) [0x7f07dc2dc749] 2024-05-27T07:43:17.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f07dae801ca] 2024-05-27T07:43:17.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 16: clone() 2024-05-27T07:43:17.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.744+0000 7f07c9ac8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f07c9ac8700 time 2024-05-27T07:43:17.743252+0000 2024-05-27T07:43:17.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:43:17.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.976 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f07dc30dba6] 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f07dc30ddd1] 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c4f086923d] 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c4f0854971] 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c4f0852fa0] 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c4f0853651] 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c4f07e8347] 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c4f04e487e] 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c4f04e9f2f] 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c4f04f4969] 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c4f088d48c] 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c4f08969cb] 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 13: (Thread::entry_wrapper()+0x43) [0x7f07dc2dc72d] 2024-05-27T07:43:17.977 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 14: (Thread::_entry_func(void*)+0xd) [0x7f07dc2dc749] 2024-05-27T07:43:17.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f07dae801ca] 2024-05-27T07:43:17.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 16: clone() 2024-05-27T07:43:17.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: *** Caught signal (Aborted) ** 2024-05-27T07:43:17.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: in thread 7f07c9ac8700 thread_name:md_log_replay 2024-05-27T07:43:17.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:17.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f07dae8ad20] 2024-05-27T07:43:17.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2: gsignal() 2024-05-27T07:43:17.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 3: abort() 2024-05-27T07:43:17.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f07dc30dced] 2024-05-27T07:43:17.978 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f07dc30ddd1] 2024-05-27T07:43:17.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:17 smithi186 ceph-mon[36647]: pgmap v112: 129 pgs: 129 active+clean; 55 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 382 op/s 2024-05-27T07:43:17.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:17 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3199856931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:17.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c4f086923d] 2024-05-27T07:43:17.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c4f0854971] 2024-05-27T07:43:17.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c4f0852fa0] 2024-05-27T07:43:17.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c4f0853651] 2024-05-27T07:43:17.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c4f07e8347] 2024-05-27T07:43:17.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c4f04e487e] 2024-05-27T07:43:17.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c4f04e9f2f] 2024-05-27T07:43:17.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c4f04f4969] 2024-05-27T07:43:17.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c4f088d48c] 2024-05-27T07:43:17.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c4f08969cb] 2024-05-27T07:43:17.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 16: (Thread::entry_wrapper()+0x43) [0x7f07dc2dc72d] 2024-05-27T07:43:17.980 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 17: (Thread::_entry_func(void*)+0xd) [0x7f07dc2dc749] 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f07dae801ca] 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 19: clone() 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.746+0000 7f07c9ac8700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: in thread 7f07c9ac8700 thread_name:md_log_replay 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f07dae8ad20] 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2: gsignal() 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 3: abort() 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f07dc30dced] 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f07dc30ddd1] 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c4f086923d] 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c4f0854971] 2024-05-27T07:43:17.981 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c4f0852fa0] 2024-05-27T07:43:17.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c4f0853651] 2024-05-27T07:43:17.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c4f07e8347] 2024-05-27T07:43:17.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c4f04e487e] 2024-05-27T07:43:17.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c4f04e9f2f] 2024-05-27T07:43:17.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c4f04f4969] 2024-05-27T07:43:17.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c4f088d48c] 2024-05-27T07:43:17.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c4f08969cb] 2024-05-27T07:43:17.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 16: (Thread::entry_wrapper()+0x43) [0x7f07dc2dc72d] 2024-05-27T07:43:17.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 17: (Thread::_entry_func(void*)+0xd) [0x7f07dc2dc749] 2024-05-27T07:43:17.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f07dae801ca] 2024-05-27T07:43:17.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 19: clone() 2024-05-27T07:43:17.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:43:17.982 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: -1> 2024-05-27T07:43:17.744+0000 7f07c9ac8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f07c9ac8700 time 2024-05-27T07:43:17.743252+0000 2024-05-27T07:43:17.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:43:17.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:17.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f07dc30dba6] 2024-05-27T07:43:17.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f07dc30ddd1] 2024-05-27T07:43:17.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c4f086923d] 2024-05-27T07:43:17.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c4f0854971] 2024-05-27T07:43:17.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c4f0852fa0] 2024-05-27T07:43:17.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c4f0853651] 2024-05-27T07:43:17.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c4f07e8347] 2024-05-27T07:43:17.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c4f04e487e] 2024-05-27T07:43:17.983 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c4f04e9f2f] 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c4f04f4969] 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c4f088d48c] 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c4f08969cb] 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 13: (Thread::entry_wrapper()+0x43) [0x7f07dc2dc72d] 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 14: (Thread::_entry_func(void*)+0xd) [0x7f07dc2dc749] 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f07dae801ca] 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 16: clone() 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 0> 2024-05-27T07:43:17.746+0000 7f07c9ac8700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: in thread 7f07c9ac8700 thread_name:md_log_replay 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f07dae8ad20] 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2: gsignal() 2024-05-27T07:43:17.984 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 3: abort() 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f07dc30dced] 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f07dc30ddd1] 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c4f086923d] 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c4f0854971] 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c4f0852fa0] 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c4f0853651] 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c4f07e8347] 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c4f04e487e] 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c4f04e9f2f] 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c4f04f4969] 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c4f088d48c] 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c4f08969cb] 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 16: (Thread::entry_wrapper()+0x43) [0x7f07dc2dc72d] 2024-05-27T07:43:17.985 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 17: (Thread::_entry_func(void*)+0xd) [0x7f07dc2dc749] 2024-05-27T07:43:17.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f07dae801ca] 2024-05-27T07:43:17.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 19: clone() 2024-05-27T07:43:17.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:43:17.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: -1> 2024-05-27T07:43:17.744+0000 7f07c9ac8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f07c9ac8700 time 2024-05-27T07:43:17.743252+0000 2024-05-27T07:43:17.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:43:17.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:17.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f07dc30dba6] 2024-05-27T07:43:17.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f07dc30ddd1] 2024-05-27T07:43:17.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c4f086923d] 2024-05-27T07:43:17.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c4f0854971] 2024-05-27T07:43:17.986 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c4f0852fa0] 2024-05-27T07:43:17.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c4f0853651] 2024-05-27T07:43:17.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c4f07e8347] 2024-05-27T07:43:17.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c4f04e487e] 2024-05-27T07:43:17.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c4f04e9f2f] 2024-05-27T07:43:17.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c4f04f4969] 2024-05-27T07:43:17.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c4f088d48c] 2024-05-27T07:43:17.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c4f08969cb] 2024-05-27T07:43:17.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 13: (Thread::entry_wrapper()+0x43) [0x7f07dc2dc72d] 2024-05-27T07:43:17.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 14: (Thread::_entry_func(void*)+0xd) [0x7f07dc2dc749] 2024-05-27T07:43:17.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f07dae801ca] 2024-05-27T07:43:17.987 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 16: clone() 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 0> 2024-05-27T07:43:17.746+0000 7f07c9ac8700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: in thread 7f07c9ac8700 thread_name:md_log_replay 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f07dae8ad20] 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2: gsignal() 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 3: abort() 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f07dc30dced] 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f07dc30ddd1] 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c4f086923d] 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c4f0854971] 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c4f0852fa0] 2024-05-27T07:43:17.988 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c4f0853651] 2024-05-27T07:43:17.989 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c4f07e8347] 2024-05-27T07:43:17.989 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c4f04e487e] 2024-05-27T07:43:17.989 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c4f04e9f2f] 2024-05-27T07:43:17.989 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c4f04f4969] 2024-05-27T07:43:17.989 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c4f088d48c] 2024-05-27T07:43:17.989 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c4f08969cb] 2024-05-27T07:43:17.989 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 16: (Thread::entry_wrapper()+0x43) [0x7f07dc2dc72d] 2024-05-27T07:43:17.989 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 17: (Thread::_entry_func(void*)+0xd) [0x7f07dc2dc749] 2024-05-27T07:43:17.989 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f07dae801ca] 2024-05-27T07:43:17.989 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 19: clone() 2024-05-27T07:43:17.989 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:43:17.989 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70673]: 2024-05-27T07:43:17.989 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:17 smithi163 podman[70710]: 2024-05-27 07:43:17.80310798 +0000 UTC m=+0.018071375 container died d4a80d6b9be1cb250812733b1452b9f51f00bd7b1d5f6a7a3e1d804a0475a27a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:43:18.241 INFO:tasks.workunit.client.0.smithi114.stdout: 1 235769 14.31 MB/sec warmup 90 sec latency 67.954 ms 2024-05-27T07:43:18.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:43:18.789 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:18.974 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:18 smithi163 podman[70710]: 2024-05-27 07:43:18.522093902 +0000 UTC m=+0.737057291 container remove d4a80d6b9be1cb250812733b1452b9f51f00bd7b1d5f6a7a3e1d804a0475a27a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-27T07:43:18.974 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:18 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:43:18.974 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:18 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-05-27T07:43:19.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 238429 14.31 MB/sec warmup 91 sec latency 23.781 ms 2024-05-27T07:43:19.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:19 smithi114 ceph-mon[32038]: pgmap v113: 129 pgs: 129 active+clean; 55 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 382 op/s 2024-05-27T07:43:19.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:19 smithi163 ceph-mon[36451]: pgmap v113: 129 pgs: 129 active+clean; 55 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 382 op/s 2024-05-27T07:43:19.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:19 smithi186 ceph-mon[36647]: pgmap v113: 129 pgs: 129 active+clean; 55 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 382 op/s 2024-05-27T07:43:20.241 INFO:tasks.workunit.client.0.smithi114.stdout: 1 241505 14.36 MB/sec warmup 92 sec latency 25.452 ms 2024-05-27T07:43:20.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.810+0000 7f0d0a8e4700 1 -- 172.21.15.114:0/601797678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0410a920 msgr2=0x7f0d0410ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.810+0000 7f0d0a8e4700 1 --2- 172.21.15.114:0/601797678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0410a920 0x7f0d0410ad30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00b0a0 tx=0x7f0cfc00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.810+0000 7f0d0a8e4700 1 -- 172.21.15.114:0/601797678 shutdown_connections 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.810+0000 7f0d0a8e4700 1 --2- 172.21.15.114:0/601797678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d0410b270 0x7f0d040759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.810+0000 7f0d0a8e4700 1 --2- 172.21.15.114:0/601797678 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d04110950 0x7f0d04110d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.810+0000 7f0d0a8e4700 1 --2- 172.21.15.114:0/601797678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0410a920 0x7f0d0410ad30 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.810+0000 7f0d0a8e4700 1 -- 172.21.15.114:0/601797678 >> 172.21.15.114:0/601797678 conn(0x7f0d0406f4c0 msgr2=0x7f0d0406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:20.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.814+0000 7f0d0a8e4700 1 -- 172.21.15.114:0/601797678 shutdown_connections 2024-05-27T07:43:20.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.814+0000 7f0d0a8e4700 1 -- 172.21.15.114:0/601797678 wait complete. 2024-05-27T07:43:20.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.815+0000 7f0d0a8e4700 1 Processor -- start 2024-05-27T07:43:20.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.815+0000 7f0d0a8e4700 1 -- start start 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.816+0000 7f0d0a8e4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d0410a920 0x7f0d04078d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.816+0000 7f0d0a8e4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0410b270 0x7f0d04079290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.816+0000 7f0d0a8e4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d04110950 0x7f0d040797d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.816+0000 7f0d0a8e4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d040832c0 con 0x7f0d0410b270 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.816+0000 7f0d0a8e4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0d04083140 con 0x7f0d0410a920 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.816+0000 7f0d0a8e4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d04083440 con 0x7f0d04110950 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.816+0000 7f0d03fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d0410a920 0x7f0d04078d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.816+0000 7f0d03fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d0410a920 0x7f0d04078d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:40394/0 (socket says 172.21.15.114:40394) 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.817+0000 7f0d03fff700 1 -- 172.21.15.114:0/3968079336 learned_addr learned my addr 172.21.15.114:0/3968079336 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.817+0000 7f0d08e81700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d04110950 0x7f0d040797d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.818+0000 7f0d037fe700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0410b270 0x7f0d04079290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.818+0000 7f0d08e81700 1 -- 172.21.15.114:0/3968079336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d0410a920 msgr2=0x7f0d04078d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.818+0000 7f0d08e81700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d0410a920 0x7f0d04078d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.818+0000 7f0d08e81700 1 -- 172.21.15.114:0/3968079336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0410b270 msgr2=0x7f0d04079290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.818+0000 7f0d08e81700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0410b270 0x7f0d04079290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.818+0000 7f0d08e81700 1 -- 172.21.15.114:0/3968079336 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d041be6a0 con 0x7f0d04110950 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.818+0000 7f0d037fe700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0410b270 0x7f0d04079290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.819+0000 7f0d08e81700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d04110950 0x7f0d040797d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00e970 tx=0x7f0cec00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.819+0000 7f0d017fa700 1 -- 172.21.15.114:0/3968079336 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec00cbd0 con 0x7f0d04110950 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.819+0000 7f0d017fa700 1 -- 172.21.15.114:0/3968079336 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cec00ee60 con 0x7f0d04110950 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.819+0000 7f0d017fa700 1 -- 172.21.15.114:0/3968079336 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec005020 con 0x7f0d04110950 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.819+0000 7f0d0a8e4700 1 -- 172.21.15.114:0/3968079336 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d041be990 con 0x7f0d04110950 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.820+0000 7f0d0a8e4700 1 -- 172.21.15.114:0/3968079336 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d041bee50 con 0x7f0d04110950 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.826+0000 7f0d192f9b80 1 Processor -- start 2024-05-27T07:43:20.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.827+0000 7f0d192f9b80 1 -- start start 2024-05-27T07:43:20.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.828+0000 7f0d192f9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564689ea8e40 0x56468a02c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.828+0000 7f0d192f9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a02cc50 0x56468a026a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.828+0000 7f0d192f9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a026f80 0x56468a027390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.828+0000 7f0d037fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564689ea8e40 0x56468a02c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.828+0000 7f0d017fa700 1 -- 172.21.15.114:0/3968079336 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f0cec0051c0 con 0x7f0d04110950 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.828+0000 7f0d037fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564689ea8e40 0x56468a02c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57040/0 (socket says 172.21.15.114:57040) 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.828+0000 7f0d037fe700 1 -- 172.21.15.114:0/2580659127 learned_addr learned my addr 172.21.15.114:0/2580659127 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.828+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2580659127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564689f55c90 con 0x564689ea8e40 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.828+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2580659127 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564689f45830 con 0x56468a026f80 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.828+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2580659127 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564689f49870 con 0x56468a02cc50 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.828+0000 7f0d037fe700 1 --2- 172.21.15.114:0/2580659127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a026f80 0x56468a027390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.830+0000 7f0d03fff700 1 --2- 172.21.15.114:0/2580659127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a02cc50 0x56468a026a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.830+0000 7f0d017fa700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0cf00b2b20 0x7f0cf00b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.830+0000 7f0d017fa700 1 -- 172.21.15.114:0/3968079336 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9466+0+0 (secure 0 0 0) 0x7f0cec0e28a0 con 0x7f0d04110950 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.831+0000 7f0d037fe700 1 -- 172.21.15.114:0/2580659127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a02cc50 msgr2=0x56468a026a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.831+0000 7f0d037fe700 1 --2- 172.21.15.114:0/2580659127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a02cc50 0x56468a026a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.831+0000 7f0d037fe700 1 -- 172.21.15.114:0/2580659127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a026f80 msgr2=0x56468a027390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.831+0000 7f0d037fe700 1 --2- 172.21.15.114:0/2580659127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a026f80 0x56468a027390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.831+0000 7f0d037fe700 1 -- 172.21.15.114:0/2580659127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564689e958d0 con 0x564689ea8e40 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.831+0000 7f0d037fe700 1 --2- 172.21.15.114:0/2580659127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564689ea8e40 0x56468a02c710 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009900 tx=0x7f0cfc009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56eccb6ecc8dfad9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.832+0000 7f0d08e81700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0cf00b2b20 0x7f0cf00b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.832+0000 7f0cfa7fc700 1 -- 172.21.15.114:0/2580659127 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc003c00 con 0x564689ea8e40 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.832+0000 7f0cfa7fc700 1 -- 172.21.15.114:0/2580659127 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cfc007a70 con 0x564689ea8e40 2024-05-27T07:43:20.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.832+0000 7f0cfa7fc700 1 -- 172.21.15.114:0/2580659127 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc005630 con 0x564689ea8e40 2024-05-27T07:43:20.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.833+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2580659127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564689ea8e40 msgr2=0x56468a02c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:20.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.833+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/2580659127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564689ea8e40 0x56468a02c710 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009900 tx=0x7f0cfc009890 comp rx=0 tx=0).stop 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.833+0000 7f0d08e81700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0cf00b2b20 0x7f0cf00b5030 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0cec0a9470 tx=0x7f0cec004940 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.833+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2580659127 shutdown_connections 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.833+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/2580659127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a026f80 0x56468a027390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.833+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/2580659127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564689ea8e40 0x56468a02c710 secure :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009900 tx=0x7f0cfc009890 comp rx=0 tx=0).stop 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.833+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/2580659127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a02cc50 0x56468a026a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.833+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2580659127 >> 172.21.15.114:0/2580659127 conn(0x564689f03af0 msgr2=0x56468a021640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.834+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2580659127 shutdown_connections 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.834+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2580659127 wait complete. 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.835+0000 7f0d192f9b80 1 -- start start 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.835+0000 7f0d192f9b80 1 Processor -- start 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.836+0000 7f0d192f9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a026f80 0x56468a030490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.837+0000 7f0d192f9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a02cc50 0x56468a0309d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.837+0000 7f0d192f9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a030f10 0x56468a03cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.837+0000 7f0d192f9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564689e4bfa0 con 0x56468a030f10 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.837+0000 7f0d192f9b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564689ee77c0 con 0x56468a026f80 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.837+0000 7f0d192f9b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56468a033730 con 0x56468a02cc50 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.837+0000 7f0d192f9b80 10 client.0 _ll_register_callbacks cb 0x564689ee0920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.838+0000 7f0d037fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a026f80 0x56468a030490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.838+0000 7f0d037fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a026f80 0x56468a030490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:40420/0 (socket says 172.21.15.114:40420) 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.838+0000 7f0d037fe700 1 -- 172.21.15.114:0/1283522935 learned_addr learned my addr 172.21.15.114:0/1283522935 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.838+0000 7f0d037fe700 1 --2- 172.21.15.114:0/1283522935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a030f10 0x56468a03cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.838+0000 7f0d03fff700 1 --2- 172.21.15.114:0/1283522935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a02cc50 0x56468a0309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.840+0000 7f0d037fe700 1 -- 172.21.15.114:0/1283522935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a02cc50 msgr2=0x56468a0309d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.840+0000 7f0d037fe700 1 --2- 172.21.15.114:0/1283522935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a02cc50 0x56468a0309d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.840+0000 7f0d037fe700 1 -- 172.21.15.114:0/1283522935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a030f10 msgr2=0x56468a03cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.840+0000 7f0d037fe700 1 --2- 172.21.15.114:0/1283522935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a030f10 0x56468a03cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.840+0000 7f0d037fe700 1 -- 172.21.15.114:0/1283522935 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564689ee4690 con 0x56468a026f80 2024-05-27T07:43:20.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.842+0000 7f0d037fe700 1 --2- 172.21.15.114:0/1283522935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a026f80 0x56468a030490 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00b800 tx=0x7f0cfc022040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=336fe14b9ea17a62 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:20.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.843+0000 7f0ce37fe700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.845+0000 7f0ce37fe700 1 -- 172.21.15.114:0/1283522935 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc026040 con 0x56468a026f80 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.845+0000 7f0ce37fe700 1 -- 172.21.15.114:0/1283522935 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9466+0+0 (secure 0 0 0) 0x7f0cfc0055d0 con 0x56468a026f80 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.845+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1283522935 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cfc007a70 con 0x56468a026f80 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.845+0000 7f0ce37fe700 1 -- 172.21.15.114:0/1283522935 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0cfc037050 con 0x56468a026f80 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.846+0000 7f0d192f9b80 10 client.25625 fetch_fsmap learned FSMap version 27 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.846+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1283522935 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56468a03e7f0 con 0x56468a026f80 2024-05-27T07:43:20.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.847+0000 7f0ce37fe700 1 -- 172.21.15.114:0/1283522935 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cfc003a00 con 0x56468a026f80 2024-05-27T07:43:20.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.847+0000 7f0ce37fe700 1 -- 172.21.15.114:0/1283522935 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f0cfc02c040 con 0x56468a026f80 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.847+0000 7f0d192f9b80 10 client.25625 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.847+0000 7f0d192f9b80 10 client.25625 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.847+0000 7f0d192f9b80 20 client.25625 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.847+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/1283522935 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56468a040cc0 0x56468a0430e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.847+0000 7f0d192f9b80 4 client.25625 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.847+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1283522935 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0cfc007a70 con 0x56468a040cc0 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.849+0000 7f0d03fff700 1 --2- 172.21.15.114:0/1283522935 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56468a040cc0 0x56468a0430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.850+0000 7f0d03fff700 1 --2- 172.21.15.114:0/1283522935 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56468a040cc0 0x56468a0430e0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad629fc4164639b2 server_cookie=ffac9fb9bda63229 in_seq=0 out_seq=0 2024-05-27T07:43:20.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.850+0000 7f0ce37fe700 10 client.25625 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:20.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.853+0000 7f0ce37fe700 1 -- 172.21.15.114:0/1283522935 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f0cfc007a70 con 0x56468a040cc0 2024-05-27T07:43:20.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.853+0000 7f0ce37fe700 10 client.25625 handle_command_reply: tid=0 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.854+0000 7f0d192f9b80 1 client.25625 shutdown 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.854+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1283522935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a026f80 msgr2=0x56468a030490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.854+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/1283522935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a026f80 0x56468a030490 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00b800 tx=0x7f0cfc022040 comp rx=0 tx=0).stop 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.855+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1283522935 shutdown_connections 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.855+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/1283522935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a030f10 0x56468a03cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.855+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/1283522935 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56468a040cc0 0x56468a0430e0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.855+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/1283522935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a026f80 0x56468a030490 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00b800 tx=0x7f0cfc022040 comp rx=0 tx=0).stop 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.855+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/1283522935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a02cc50 0x56468a0309d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.855+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1283522935 >> 172.21.15.114:0/1283522935 conn(0x564689f03af0 msgr2=0x56468a025a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.855+0000 7f0ce37fe700 0 client.25625 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.855+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1283522935 shutdown_connections 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.855+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1283522935 wait complete. 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.856+0000 7f0d192f9b80 20 client.25625 trim_cache size 0 max 16384 2024-05-27T07:43:20.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.892+0000 7f0d192f9b80 1 Processor -- start 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.892+0000 7f0d192f9b80 1 -- start start 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.892+0000 7f0d192f9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a02cc50 0x56468a026a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.892+0000 7f0d192f9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a030f10 0x56468a03ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.892+0000 7f0d037fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a02cc50 0x56468a026a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.892+0000 7f0d037fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a02cc50 0x56468a026a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57062/0 (socket says 172.21.15.114:57062) 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.892+0000 7f0d037fe700 1 -- 172.21.15.114:0/1098029566 learned_addr learned my addr 172.21.15.114:0/1098029566 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.892+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/1098029566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a02fd70 0x56468a01b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.892+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1098029566 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d041d6bf0 con 0x56468a02cc50 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.892+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1098029566 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0d041d9470 con 0x56468a02fd70 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.892+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1098029566 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d041d7260 con 0x56468a030f10 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.893+0000 7f0d03fff700 1 --2- 172.21.15.114:0/1098029566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a030f10 0x56468a03ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.893+0000 7f0d037fe700 1 --2- 172.21.15.114:0/1098029566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a02fd70 0x56468a01b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.893+0000 7f0d037fe700 1 -- 172.21.15.114:0/1098029566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a030f10 msgr2=0x56468a03ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.893+0000 7f0d037fe700 1 --2- 172.21.15.114:0/1098029566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a030f10 0x56468a03ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.893+0000 7f0d037fe700 1 -- 172.21.15.114:0/1098029566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a02fd70 msgr2=0x56468a01b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.893+0000 7f0d037fe700 1 --2- 172.21.15.114:0/1098029566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a02fd70 0x56468a01b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.893+0000 7f0d037fe700 1 -- 172.21.15.114:0/1098029566 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56468a02ca60 con 0x56468a02cc50 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.893+0000 7f0d03fff700 1 --2- 172.21.15.114:0/1098029566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a030f10 0x56468a03ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.893+0000 7f0d037fe700 1 --2- 172.21.15.114:0/1098029566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a02cc50 0x56468a026a40 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0051a0 tx=0x7f0cfc00cfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6516331394b76f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.893+0000 7f0cf9ffb700 1 -- 172.21.15.114:0/1098029566 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc01fa50 con 0x56468a02cc50 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.894+0000 7f0cf9ffb700 1 -- 172.21.15.114:0/1098029566 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cfc01f430 con 0x56468a02cc50 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.894+0000 7f0cf9ffb700 1 -- 172.21.15.114:0/1098029566 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc0075c0 con 0x56468a02cc50 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.894+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1098029566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a02cc50 msgr2=0x56468a026a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.894+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/1098029566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a02cc50 0x56468a026a40 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0051a0 tx=0x7f0cfc00cfd0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.894+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1098029566 shutdown_connections 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.894+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/1098029566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a02fd70 0x56468a01b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.895+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/1098029566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a02cc50 0x56468a026a40 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.895+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/1098029566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a030f10 0x56468a03ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.895+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1098029566 >> 172.21.15.114:0/1098029566 conn(0x564689f03af0 msgr2=0x56468a0211c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.895+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1098029566 shutdown_connections 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.895+0000 7f0d192f9b80 1 -- 172.21.15.114:0/1098029566 wait complete. 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.895+0000 7f0d192f9b80 1 -- start start 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.895+0000 7f0d192f9b80 1 Processor -- start 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.895+0000 7f0d192f9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a02cc50 0x56468a03c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d192f9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a02fd70 0x56468a03c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d192f9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a030f10 0x56468a0a34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d192f9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d041d7930 con 0x56468a02fd70 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d192f9b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0d041d8da0 con 0x56468a02cc50 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d192f9b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d041d8000 con 0x56468a030f10 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d192f9b80 10 client.0 _ll_register_callbacks cb 0x564689ee0920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d037fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a02cc50 0x56468a03c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d037fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a02cc50 0x56468a03c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:40450/0 (socket says 172.21.15.114:40450) 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d037fe700 1 -- 172.21.15.114:0/2569796191 learned_addr learned my addr 172.21.15.114:0/2569796191 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d037fe700 1 --2- 172.21.15.114:0/2569796191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a030f10 0x56468a0a34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d03fff700 1 --2- 172.21.15.114:0/2569796191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a02fd70 0x56468a03c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d037fe700 1 -- 172.21.15.114:0/2569796191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a030f10 msgr2=0x56468a0a34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d037fe700 1 --2- 172.21.15.114:0/2569796191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a030f10 0x56468a0a34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d037fe700 1 -- 172.21.15.114:0/2569796191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a02fd70 msgr2=0x56468a03c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d037fe700 1 --2- 172.21.15.114:0/2569796191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a02fd70 0x56468a03c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d037fe700 1 -- 172.21.15.114:0/2569796191 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56468a0a3a80 con 0x56468a02cc50 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d037fe700 1 --2- 172.21.15.114:0/2569796191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a02cc50 0x56468a03c010 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc01f660 tx=0x7f0cfc021040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2cd5c6da5ab7456 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0ce37fe700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0ce37fe700 1 -- 172.21.15.114:0/2569796191 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc01fa60 con 0x56468a02cc50 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0ce37fe700 1 -- 172.21.15.114:0/2569796191 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9466+0+0 (secure 0 0 0) 0x7f0cfc005390 con 0x56468a02cc50 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.896+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2569796191 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cfc01f430 con 0x56468a02cc50 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.897+0000 7f0ce37fe700 1 -- 172.21.15.114:0/2569796191 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0cfc005590 con 0x56468a02cc50 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.897+0000 7f0d192f9b80 10 client.25631 fetch_fsmap learned FSMap version 27 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.897+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2569796191 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56468a0aada0 con 0x56468a02cc50 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.897+0000 7f0ce37fe700 1 -- 172.21.15.114:0/2569796191 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cfc01f430 con 0x56468a02cc50 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.897+0000 7f0ce37fe700 1 -- 172.21.15.114:0/2569796191 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f0cfc02c070 con 0x56468a02cc50 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.897+0000 7f0d192f9b80 10 client.25631 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.897+0000 7f0d192f9b80 10 client.25631 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.897+0000 7f0d192f9b80 20 client.25631 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.897+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/2569796191 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56468a0ab380 0x56468a0ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.897+0000 7f0d192f9b80 4 client.25631 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.897+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2569796191 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564689ff29f0 con 0x56468a0ab380 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.897+0000 7f0d03fff700 1 --2- 172.21.15.114:0/2569796191 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56468a0ab380 0x56468a0ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.898+0000 7f0ce37fe700 10 client.25631 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.898+0000 7f0d03fff700 1 --2- 172.21.15.114:0/2569796191 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56468a0ab380 0x56468a0ad7a0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=531513c34d361cd5 server_cookie=f7bced2ac9fd5faf in_seq=0 out_seq=0 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0ce37fe700 1 -- 172.21.15.114:0/2569796191 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564689ff29f0 con 0x56468a0ab380 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0ce37fe700 10 client.25631 handle_command_reply: tid=0 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0d192f9b80 1 client.25631 shutdown 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2569796191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a02cc50 msgr2=0x56468a03c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/2569796191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a02cc50 0x56468a03c010 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc01f660 tx=0x7f0cfc021040 comp rx=0 tx=0).stop 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2569796191 shutdown_connections 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/2569796191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56468a030f10 0x56468a0a34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/2569796191 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56468a0ab380 0x56468a0ad7a0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/2569796191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56468a02cc50 0x56468a03c010 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0d192f9b80 1 --2- 172.21.15.114:0/2569796191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56468a02fd70 0x56468a03c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0ce37fe700 0 client.25631 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2569796191 >> 172.21.15.114:0/2569796191 conn(0x564689f03af0 msgr2=0x56468a0211c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2569796191 shutdown_connections 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0d192f9b80 1 -- 172.21.15.114:0/2569796191 wait complete. 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.899+0000 7f0d192f9b80 20 client.25631 trim_cache size 0 max 16384 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.900+0000 7f0d0a8e4700 1 -- 172.21.15.114:0/3968079336 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0cf00b2b20 msgr2=0x7f0cf00b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.900+0000 7f0d0a8e4700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0cf00b2b20 0x7f0cf00b5030 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0cec0a9470 tx=0x7f0cec004940 comp rx=0 tx=0).stop 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.900+0000 7f0d0a8e4700 1 -- 172.21.15.114:0/3968079336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d04110950 msgr2=0x7f0d040797d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.900+0000 7f0d0a8e4700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d04110950 0x7f0d040797d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00e970 tx=0x7f0cec00ec80 comp rx=0 tx=0).stop 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.900+0000 7f0d0a8e4700 1 -- 172.21.15.114:0/3968079336 shutdown_connections 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.900+0000 7f0d0a8e4700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d04110950 0x7f0d040797d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.900+0000 7f0d0a8e4700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0cf00b2b20 0x7f0cf00b5030 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.900+0000 7f0d0a8e4700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d0410a920 0x7f0d04078d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.900+0000 7f0d0a8e4700 1 --2- 172.21.15.114:0/3968079336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0410b270 0x7f0d04079290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.900+0000 7f0d0a8e4700 1 -- 172.21.15.114:0/3968079336 >> 172.21.15.114:0/3968079336 conn(0x7f0d0406f4c0 msgr2=0x7f0d04085a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.900+0000 7f0d0a8e4700 1 -- 172.21.15.114:0/3968079336 shutdown_connections 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:20.900+0000 7f0d0a8e4700 1 -- 172.21.15.114:0/3968079336 wait complete. 2024-05-27T07:43:21.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 243647 14.28 MB/sec warmup 93 sec latency 97.015 ms 2024-05-27T07:43:21.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:21 smithi163 ceph-mon[36451]: pgmap v114: 129 pgs: 129 active+clean; 53 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 434 op/s 2024-05-27T07:43:21.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:21 smithi186 ceph-mon[36647]: pgmap v114: 129 pgs: 129 active+clean; 53 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 434 op/s 2024-05-27T07:43:22.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:21 smithi114 ceph-mon[32038]: pgmap v114: 129 pgs: 129 active+clean; 53 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 434 op/s 2024-05-27T07:43:22.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 245765 14.27 MB/sec warmup 94 sec latency 17.461 ms 2024-05-27T07:43:22.681 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:43:22.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:43:22.955 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:23.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 248788 14.31 MB/sec warmup 95 sec latency 26.516 ms 2024-05-27T07:43:23.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:23 smithi114 ceph-mon[32038]: pgmap v115: 129 pgs: 129 active+clean; 45 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 522 op/s 2024-05-27T07:43:23.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:23 smithi163 ceph-mon[36451]: pgmap v115: 129 pgs: 129 active+clean; 45 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 522 op/s 2024-05-27T07:43:23.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:23 smithi186 ceph-mon[36647]: pgmap v115: 129 pgs: 129 active+clean; 45 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 522 op/s 2024-05-27T07:43:24.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 251864 14.34 MB/sec warmup 96 sec latency 24.434 ms 2024-05-27T07:43:25.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 254269 14.29 MB/sec warmup 97 sec latency 45.800 ms 2024-05-27T07:43:25.869 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:25 smithi114 ceph-mon[32038]: pgmap v116: 129 pgs: 129 active+clean; 45 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 522 op/s 2024-05-27T07:43:25.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:25 smithi163 ceph-mon[36451]: pgmap v116: 129 pgs: 129 active+clean; 45 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 522 op/s 2024-05-27T07:43:25.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:25 smithi186 ceph-mon[36647]: pgmap v116: 129 pgs: 129 active+clean; 45 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 522 op/s 2024-05-27T07:43:26.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.131+0000 7f7a1e83d700 1 -- 172.21.15.114:0/2599770372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18109860 msgr2=0x7f7a181111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:26.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.131+0000 7f7a1e83d700 1 --2- 172.21.15.114:0/2599770372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18109860 0x7f7a181111c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000b0d0 tx=0x7f7a1000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.131+0000 7f7a1e83d700 1 -- 172.21.15.114:0/2599770372 shutdown_connections 2024-05-27T07:43:26.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.131+0000 7f7a1e83d700 1 --2- 172.21.15.114:0/2599770372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18109860 0x7f7a181111c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.131+0000 7f7a1e83d700 1 --2- 172.21.15.114:0/2599770372 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a181045e0 0x7f7a181049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.131+0000 7f7a1e83d700 1 --2- 172.21.15.114:0/2599770372 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a18108f10 0x7f7a18109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.131+0000 7f7a1e83d700 1 -- 172.21.15.114:0/2599770372 >> 172.21.15.114:0/2599770372 conn(0x7f7a18100030 msgr2=0x7f7a18102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:26.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.131+0000 7f7a1e83d700 1 -- 172.21.15.114:0/2599770372 shutdown_connections 2024-05-27T07:43:26.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.131+0000 7f7a1e83d700 1 -- 172.21.15.114:0/2599770372 wait complete. 2024-05-27T07:43:26.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.132+0000 7f7a1e83d700 1 Processor -- start 2024-05-27T07:43:26.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.132+0000 7f7a1e83d700 1 -- start start 2024-05-27T07:43:26.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.132+0000 7f7a1e83d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a181045e0 0x7f7a1819ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.132+0000 7f7a1e83d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18108f10 0x7f7a1819b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.132+0000 7f7a1e83d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a18109860 0x7f7a1819f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a1e83d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7a181144b0 con 0x7f7a18108f10 2024-05-27T07:43:26.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a1e83d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7a18114330 con 0x7f7a181045e0 2024-05-27T07:43:26.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a1e83d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a18114630 con 0x7f7a18109860 2024-05-27T07:43:26.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a0ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18108f10 0x7f7a1819b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a0ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18108f10 0x7f7a1819b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54564/0 (socket says 172.21.15.114:54564) 2024-05-27T07:43:26.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a0ffff700 1 -- 172.21.15.114:0/292074700 learned_addr learned my addr 172.21.15.114:0/292074700 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:26.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a1cdda700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a18109860 0x7f7a1819f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a17fff700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a181045e0 0x7f7a1819ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a0ffff700 1 -- 172.21.15.114:0/292074700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a18109860 msgr2=0x7f7a1819f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:26.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a0ffff700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a18109860 0x7f7a1819f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a0ffff700 1 -- 172.21.15.114:0/292074700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a181045e0 msgr2=0x7f7a1819ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:26.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a0ffff700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a181045e0 0x7f7a1819ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a0ffff700 1 -- 172.21.15.114:0/292074700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1819fcd0 con 0x7f7a18108f10 2024-05-27T07:43:26.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a1cdda700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a18109860 0x7f7a1819f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:43:26.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a0ffff700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18108f10 0x7f7a1819b3a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000e9f0 tx=0x7f7a0000edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:26.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a15ffb700 1 -- 172.21.15.114:0/292074700 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0000cbd0 con 0x7f7a18108f10 2024-05-27T07:43:26.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a1e83d700 1 -- 172.21.15.114:0/292074700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a1819ffc0 con 0x7f7a18108f10 2024-05-27T07:43:26.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a15ffb700 1 -- 172.21.15.114:0/292074700 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a0000cd70 con 0x7f7a18108f10 2024-05-27T07:43:26.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a15ffb700 1 -- 172.21.15.114:0/292074700 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a00010430 con 0x7f7a18108f10 2024-05-27T07:43:26.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.133+0000 7f7a1e83d700 1 -- 172.21.15.114:0/292074700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a181a78e0 con 0x7f7a18108f10 2024-05-27T07:43:26.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.137+0000 7f7a15ffb700 1 -- 172.21.15.114:0/292074700 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f7a000105d0 con 0x7f7a18108f10 2024-05-27T07:43:26.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a15ffb700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7a040b29f0 0x7f7a040b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a17fff700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7a040b29f0 0x7f7a040b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a2d252b80 1 Processor -- start 2024-05-27T07:43:26.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a15ffb700 1 -- 172.21.15.114:0/292074700 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9466+0+0 (secure 0 0 0) 0x7f7a000e2d00 con 0x7f7a18108f10 2024-05-27T07:43:26.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a2d252b80 1 -- start start 2024-05-27T07:43:26.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a2d252b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c27077790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a2d252b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27077cd0 0x564c27071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a2d252b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27072000 0x564c27072410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a2d252b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564c26fa0d10 con 0x564c27077cd0 2024-05-27T07:43:26.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a2d252b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564c26f908b0 con 0x564c27072000 2024-05-27T07:43:26.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a2d252b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564c26f948f0 con 0x564c26ef3ec0 2024-05-27T07:43:26.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a17fff700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7a040b29f0 0x7f7a040b4f00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7a08005fd0 tx=0x7f7a08005f60 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:26.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a0ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27077cd0 0x564c27071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a1cdda700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27072000 0x564c27072410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a1cdda700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27072000 0x564c27072410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:55134/0 (socket says 172.21.15.114:55134) 2024-05-27T07:43:26.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a0ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27077cd0 0x564c27071ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54580/0 (socket says 172.21.15.114:54580) 2024-05-27T07:43:26.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a1cdda700 1 -- 172.21.15.114:0/3416022158 learned_addr learned my addr 172.21.15.114:0/3416022158 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:26.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.138+0000 7f7a17fff700 1 --2- 172.21.15.114:0/3416022158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c27077790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.139+0000 7f7a1cdda700 1 -- 172.21.15.114:0/3416022158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 msgr2=0x564c27077790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:26.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.139+0000 7f7a1cdda700 1 --2- 172.21.15.114:0/3416022158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c27077790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.139+0000 7f7a1cdda700 1 -- 172.21.15.114:0/3416022158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27077cd0 msgr2=0x564c27071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:26.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.139+0000 7f7a1cdda700 1 --2- 172.21.15.114:0/3416022158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27077cd0 0x564c27071ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.139+0000 7f7a1cdda700 1 -- 172.21.15.114:0/3416022158 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c26ee0950 con 0x564c27072000 2024-05-27T07:43:26.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.139+0000 7f7a17fff700 1 --2- 172.21.15.114:0/3416022158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c27077790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:26.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.139+0000 7f7a1cdda700 1 --2- 172.21.15.114:0/3416022158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27072000 0x564c27072410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7a10009dc0 tx=0x7f7a10009df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c719e6804e737db7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:26.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.139+0000 7f7a0e7fc700 1 -- 172.21.15.114:0/3416022158 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a10022070 con 0x564c27072000 2024-05-27T07:43:26.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.139+0000 7f7a0e7fc700 1 -- 172.21.15.114:0/3416022158 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a100075c0 con 0x564c27072000 2024-05-27T07:43:26.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.139+0000 7f7a0e7fc700 1 -- 172.21.15.114:0/3416022158 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1001c6a0 con 0x564c27072000 2024-05-27T07:43:26.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.140+0000 7f7a2d252b80 1 -- 172.21.15.114:0/3416022158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27072000 msgr2=0x564c27072410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:26.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.140+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/3416022158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27072000 0x564c27072410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7a10009dc0 tx=0x7f7a10009df0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.140+0000 7f7a2d252b80 1 -- 172.21.15.114:0/3416022158 shutdown_connections 2024-05-27T07:43:26.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.140+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/3416022158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27072000 0x564c27072410 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.140+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/3416022158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c27077790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.140+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/3416022158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27077cd0 0x564c27071ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.140+0000 7f7a2d252b80 1 -- 172.21.15.114:0/3416022158 >> 172.21.15.114:0/3416022158 conn(0x564c26f4eb70 msgr2=0x564c2706c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:26.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.140+0000 7f7a2d252b80 1 -- 172.21.15.114:0/3416022158 shutdown_connections 2024-05-27T07:43:26.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.140+0000 7f7a2d252b80 1 -- 172.21.15.114:0/3416022158 wait complete. 2024-05-27T07:43:26.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a2d252b80 1 -- start start 2024-05-27T07:43:26.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a2d252b80 1 Processor -- start 2024-05-27T07:43:26.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a2d252b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c2707b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a2d252b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 0x564c2707b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a2d252b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 0x564c2707bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a2d252b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564c26e97020 con 0x564c27072000 2024-05-27T07:43:26.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a2d252b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564c26f32840 con 0x564c27077cd0 2024-05-27T07:43:26.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a2d252b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564c27082b70 con 0x564c26ef3ec0 2024-05-27T07:43:26.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a2d252b80 10 client.0 _ll_register_callbacks cb 0x564c26f2b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:26.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a0ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 0x564c2707b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a0ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 0x564c2707b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54594/0 (socket says 172.21.15.114:54594) 2024-05-27T07:43:26.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a0ffff700 1 -- 172.21.15.114:0/2167464188 learned_addr learned my addr 172.21.15.114:0/2167464188 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:26.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a17fff700 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c2707b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.141+0000 7f7a1cdda700 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 0x564c2707bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f7a0ffff700 1 -- 172.21.15.114:0/2167464188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 msgr2=0x564c2707b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:26.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f7a0ffff700 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c2707b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f7a0ffff700 1 -- 172.21.15.114:0/2167464188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 msgr2=0x564c2707bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:26.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f7a0ffff700 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 0x564c2707bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f7a0ffff700 1 -- 172.21.15.114:0/2167464188 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c26f2f710 con 0x564c27072000 2024-05-27T07:43:26.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f7a17fff700 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c2707b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:26.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f7a0ffff700 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 0x564c2707b890 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7a000e06a0 tx=0x7f7a000a5430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312ef8124085e299 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:26.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f79f77fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:26.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f79f77fe700 1 -- 172.21.15.114:0/2167464188 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a000a9740 con 0x564c27072000 2024-05-27T07:43:26.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f79f77fe700 1 -- 172.21.15.114:0/2167464188 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9466+0+0 (secure 0 0 0) 0x7f7a000a6220 con 0x564c27072000 2024-05-27T07:43:26.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f7a2d252b80 1 -- 172.21.15.114:0/2167464188 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a100075c0 con 0x564c27072000 2024-05-27T07:43:26.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f79f77fe700 1 -- 172.21.15.114:0/2167464188 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7a000aa330 con 0x564c27072000 2024-05-27T07:43:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f7a2d252b80 10 client.16122 fetch_fsmap learned FSMap version 27 2024-05-27T07:43:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f7a2d252b80 1 -- 172.21.15.114:0/2167464188 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564c2708d5c0 con 0x564c27072000 2024-05-27T07:43:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.142+0000 7f79f77fe700 1 -- 172.21.15.114:0/2167464188 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a100075c0 con 0x564c27072000 2024-05-27T07:43:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.143+0000 7f79f77fe700 1 -- 172.21.15.114:0/2167464188 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f7a000a55a0 con 0x564c27072000 2024-05-27T07:43:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.143+0000 7f7a2d252b80 10 client.16122 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:43:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.143+0000 7f7a2d252b80 10 client.16122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.143+0000 7f7a2d252b80 20 client.16122 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.143+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c2708feb0 0x564c27092360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.143+0000 7f7a2d252b80 4 client.16122 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.143+0000 7f7a2d252b80 1 -- 172.21.15.114:0/2167464188 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c2703da70 con 0x564c2708feb0 2024-05-27T07:43:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.143+0000 7f7a17fff700 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c2708feb0 0x564c27092360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.143+0000 7f79f77fe700 10 client.16122 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.143+0000 7f7a17fff700 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c2708feb0 0x564c27092360 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4209c34f15c52c2c server_cookie=a7c3359ce2df8fa8 in_seq=0 out_seq=0 2024-05-27T07:43:26.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.144+0000 7f79f77fe700 1 -- 172.21.15.114:0/2167464188 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564c2703da70 con 0x564c2708feb0 2024-05-27T07:43:26.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.144+0000 7f79f77fe700 10 client.16122 handle_command_reply: tid=0 2024-05-27T07:43:26.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.144+0000 7f7a2d252b80 1 client.16122 shutdown 2024-05-27T07:43:26.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.145+0000 7f7a2d252b80 1 -- 172.21.15.114:0/2167464188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 msgr2=0x564c2707b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:26.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.145+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 0x564c2707b890 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7a000e06a0 tx=0x7f7a000a5430 comp rx=0 tx=0).stop 2024-05-27T07:43:26.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.145+0000 7f7a2d252b80 1 -- 172.21.15.114:0/2167464188 shutdown_connections 2024-05-27T07:43:26.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.145+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c2708feb0 0x564c27092360 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.145+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 0x564c2707bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.145+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c2707b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.145+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/2167464188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 0x564c2707b890 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.145+0000 7f7a2d252b80 1 -- 172.21.15.114:0/2167464188 >> 172.21.15.114:0/2167464188 conn(0x564c26f4eb70 msgr2=0x564c27070ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:26.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.145+0000 7f79f77fe700 0 client.16122 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:26.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.145+0000 7f7a2d252b80 1 -- 172.21.15.114:0/2167464188 shutdown_connections 2024-05-27T07:43:26.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.145+0000 7f7a2d252b80 1 -- 172.21.15.114:0/2167464188 wait complete. 2024-05-27T07:43:26.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.145+0000 7f7a2d252b80 20 client.16122 trim_cache size 0 max 16384 2024-05-27T07:43:26.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a2d252b80 1 Processor -- start 2024-05-27T07:43:26.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a2d252b80 1 -- start start 2024-05-27T07:43:26.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a2d252b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c26ef3ec0 0x564c27071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a2d252b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c27072000 0x564c2708bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a2d252b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 0x564c2707b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a2d252b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7a181ac280 con 0x564c26ef3ec0 2024-05-27T07:43:26.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a2d252b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7a181aeb00 con 0x564c27077cd0 2024-05-27T07:43:26.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a2d252b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a181ac8f0 con 0x564c27072000 2024-05-27T07:43:26.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a17fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c26ef3ec0 0x564c27071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a17fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c26ef3ec0 0x564c27071ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54606/0 (socket says 172.21.15.114:54606) 2024-05-27T07:43:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a17fff700 1 -- 172.21.15.114:0/1687613430 learned_addr learned my addr 172.21.15.114:0/1687613430 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a0ffff700 1 --2- 172.21.15.114:0/1687613430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c27072000 0x564c2708bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a1cdda700 1 --2- 172.21.15.114:0/1687613430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 0x564c2707b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.168+0000 7f7a17fff700 1 -- 172.21.15.114:0/1687613430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c27072000 msgr2=0x564c2708bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.169+0000 7f7a17fff700 1 --2- 172.21.15.114:0/1687613430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c27072000 0x564c2708bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.169+0000 7f7a17fff700 1 -- 172.21.15.114:0/1687613430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 msgr2=0x564c2707b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.169+0000 7f7a17fff700 1 --2- 172.21.15.114:0/1687613430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 0x564c2707b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.169+0000 7f7a17fff700 1 -- 172.21.15.114:0/1687613430 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c2706de50 con 0x564c26ef3ec0 2024-05-27T07:43:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.169+0000 7f7a17fff700 1 --2- 172.21.15.114:0/1687613430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c26ef3ec0 0x564c27071ac0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7a0800b960 tx=0x7f7a08015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d75d3258f491dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.169+0000 7f7a0dffb700 1 -- 172.21.15.114:0/1687613430 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a08021bf0 con 0x564c26ef3ec0 2024-05-27T07:43:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.169+0000 7f7a0dffb700 1 -- 172.21.15.114:0/1687613430 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a08021d90 con 0x564c26ef3ec0 2024-05-27T07:43:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.169+0000 7f7a0dffb700 1 -- 172.21.15.114:0/1687613430 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a080117b0 con 0x564c26ef3ec0 2024-05-27T07:43:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.170+0000 7f7a2d252b80 1 -- 172.21.15.114:0/1687613430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c26ef3ec0 msgr2=0x564c27071ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.170+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/1687613430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c26ef3ec0 0x564c27071ac0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7a0800b960 tx=0x7f7a08015000 comp rx=0 tx=0).stop 2024-05-27T07:43:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.170+0000 7f7a2d252b80 1 -- 172.21.15.114:0/1687613430 shutdown_connections 2024-05-27T07:43:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.170+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/1687613430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 0x564c2707b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.170+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/1687613430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c26ef3ec0 0x564c27071ac0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.170+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/1687613430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c27072000 0x564c2708bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.170+0000 7f7a2d252b80 1 -- 172.21.15.114:0/1687613430 >> 172.21.15.114:0/1687613430 conn(0x564c26f4eb70 msgr2=0x564c27075700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.170+0000 7f7a2d252b80 1 -- 172.21.15.114:0/1687613430 shutdown_connections 2024-05-27T07:43:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.170+0000 7f7a2d252b80 1 -- 172.21.15.114:0/1687613430 wait complete. 2024-05-27T07:43:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.170+0000 7f7a2d252b80 1 -- start start 2024-05-27T07:43:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.170+0000 7f7a2d252b80 1 Processor -- start 2024-05-27T07:43:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a2d252b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c27088610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a2d252b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 0x564c270974c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a2d252b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 0x564c27097ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a2d252b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7a181acfc0 con 0x564c27072000 2024-05-27T07:43:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a2d252b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7a181ae430 con 0x564c27077cd0 2024-05-27T07:43:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a2d252b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a181ad690 con 0x564c26ef3ec0 2024-05-27T07:43:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a2d252b80 10 client.0 _ll_register_callbacks cb 0x564c26f2b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a0ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 0x564c270974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a0ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 0x564c270974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54622/0 (socket says 172.21.15.114:54622) 2024-05-27T07:43:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a0ffff700 1 -- 172.21.15.114:0/270303956 learned_addr learned my addr 172.21.15.114:0/270303956 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a17fff700 1 --2- 172.21.15.114:0/270303956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c27088610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a1cdda700 1 --2- 172.21.15.114:0/270303956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 0x564c27097ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a0ffff700 1 -- 172.21.15.114:0/270303956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 msgr2=0x564c27088610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a0ffff700 1 --2- 172.21.15.114:0/270303956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c27088610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a0ffff700 1 -- 172.21.15.114:0/270303956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 msgr2=0x564c27097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a0ffff700 1 --2- 172.21.15.114:0/270303956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 0x564c27097ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a0ffff700 1 -- 172.21.15.114:0/270303956 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c27098290 con 0x564c27072000 2024-05-27T07:43:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f7a0ffff700 1 --2- 172.21.15.114:0/270303956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 0x564c270974c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7a000a9ab0 tx=0x7f7a0000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c2019ac4e7323d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f79f77fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.171+0000 7f79f77fe700 1 -- 172.21.15.114:0/270303956 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a000a9740 con 0x564c27072000 2024-05-27T07:43:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f79f77fe700 1 -- 172.21.15.114:0/270303956 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9466+0+0 (secure 0 0 0) 0x7f7a000a6220 con 0x564c27072000 2024-05-27T07:43:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f7a2d252b80 1 -- 172.21.15.114:0/270303956 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a08021d90 con 0x564c27072000 2024-05-27T07:43:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f79f77fe700 1 -- 172.21.15.114:0/270303956 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7a000c42d0 con 0x564c27072000 2024-05-27T07:43:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f7a2d252b80 10 client.16128 fetch_fsmap learned FSMap version 27 2024-05-27T07:43:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f7a2d252b80 1 -- 172.21.15.114:0/270303956 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564c270faa90 con 0x564c27072000 2024-05-27T07:43:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f79f77fe700 1 -- 172.21.15.114:0/270303956 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a08021d90 con 0x564c27072000 2024-05-27T07:43:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f79f77fe700 1 -- 172.21.15.114:0/270303956 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f7a00085060 con 0x564c27072000 2024-05-27T07:43:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f7a2d252b80 10 client.16128 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:43:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f7a2d252b80 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f7a2d252b80 20 client.16128 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/270303956 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c270fb010 0x564c270fd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f7a2d252b80 4 client.16128 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:43:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f7a2d252b80 1 -- 172.21.15.114:0/270303956 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564c26fa2c60 con 0x564c270fb010 2024-05-27T07:43:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.172+0000 7f7a17fff700 1 --2- 172.21.15.114:0/270303956 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c270fb010 0x564c270fd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.173+0000 7f79f77fe700 10 client.16128 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.173+0000 7f7a17fff700 1 --2- 172.21.15.114:0/270303956 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c270fb010 0x564c270fd430 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1329304a1d9e28b2 server_cookie=915ac54f7737181f in_seq=0 out_seq=0 2024-05-27T07:43:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f79f77fe700 1 -- 172.21.15.114:0/270303956 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564c26fa2c60 con 0x564c270fb010 2024-05-27T07:43:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f79f77fe700 10 client.16128 handle_command_reply: tid=0 2024-05-27T07:43:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f7a2d252b80 1 client.16128 shutdown 2024-05-27T07:43:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f7a2d252b80 1 -- 172.21.15.114:0/270303956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 msgr2=0x564c270974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/270303956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 0x564c270974c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7a000a9ab0 tx=0x7f7a0000ef90 comp rx=0 tx=0).stop 2024-05-27T07:43:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f7a2d252b80 1 -- 172.21.15.114:0/270303956 shutdown_connections 2024-05-27T07:43:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/270303956 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564c270fb010 0x564c270fd430 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/270303956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564c27077cd0 0x564c27097ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/270303956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564c26ef3ec0 0x564c27088610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f79f77fe700 0 client.16128 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:26.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f7a2d252b80 1 --2- 172.21.15.114:0/270303956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c27072000 0x564c270974c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f7a2d252b80 1 -- 172.21.15.114:0/270303956 >> 172.21.15.114:0/270303956 conn(0x564c26f4eb70 msgr2=0x564c27075700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:26.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f7a2d252b80 1 -- 172.21.15.114:0/270303956 shutdown_connections 2024-05-27T07:43:26.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f7a2d252b80 1 -- 172.21.15.114:0/270303956 wait complete. 2024-05-27T07:43:26.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.175+0000 7f7a2d252b80 20 client.16128 trim_cache size 0 max 16384 2024-05-27T07:43:26.180 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:43:26.180 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:43:26.180 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "280f262b-916a-41b4-89ee-147be5a56c6a", 2024-05-27T07:43:26.180 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:43:26.181 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:43:26.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.176+0000 7f7a1e83d700 1 -- 172.21.15.114:0/292074700 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7a040b29f0 msgr2=0x7f7a040b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:26.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.176+0000 7f7a1e83d700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7a040b29f0 0x7f7a040b4f00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7a08005fd0 tx=0x7f7a08005f60 comp rx=0 tx=0).stop 2024-05-27T07:43:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.176+0000 7f7a1e83d700 1 -- 172.21.15.114:0/292074700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18108f10 msgr2=0x7f7a1819b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.176+0000 7f7a1e83d700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18108f10 0x7f7a1819b3a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000e9f0 tx=0x7f7a0000edb0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.176+0000 7f7a1e83d700 1 -- 172.21.15.114:0/292074700 shutdown_connections 2024-05-27T07:43:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.176+0000 7f7a1e83d700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a18109860 0x7f7a1819f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.176+0000 7f7a1e83d700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7a040b29f0 0x7f7a040b4f00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.176+0000 7f7a1e83d700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a181045e0 0x7f7a1819ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.176+0000 7f7a1e83d700 1 --2- 172.21.15.114:0/292074700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18108f10 0x7f7a1819b3a0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.176+0000 7f7a1e83d700 1 -- 172.21.15.114:0/292074700 >> 172.21.15.114:0/292074700 conn(0x7f7a18100030 msgr2=0x7f7a18108650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.176+0000 7f7a1e83d700 1 -- 172.21.15.114:0/292074700 shutdown_connections 2024-05-27T07:43:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:26.176+0000 7f7a1e83d700 1 -- 172.21.15.114:0/292074700 wait complete. 2024-05-27T07:43:26.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 256155 14.27 MB/sec warmup 98 sec latency 42.111 ms 2024-05-27T07:43:26.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:26 smithi163 ceph-mon[36451]: pgmap v117: 129 pgs: 129 active+clean; 51 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 MiB/s wr, 424 op/s 2024-05-27T07:43:26.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:26 smithi186 ceph-mon[36647]: pgmap v117: 129 pgs: 129 active+clean; 51 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 MiB/s wr, 424 op/s 2024-05-27T07:43:27.008 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:26 smithi114 ceph-mon[32038]: pgmap v117: 129 pgs: 129 active+clean; 51 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 MiB/s wr, 424 op/s 2024-05-27T07:43:27.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:43:27.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 259211 14.31 MB/sec warmup 99 sec latency 27.154 ms 2024-05-27T07:43:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:27.679 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:27 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:43:27.679 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:27 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:27.679 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:27 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:43:27.679 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:27 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:27.679 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:27 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:43:27.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:27 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:43:27.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:27 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:27.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:27 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:43:27.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:27 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:27.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:27 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:43:27.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:27 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:43:27.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:27 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:27.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:27 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:43:27.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:27 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:27.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:27 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:43:28.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 261576 14.25 MB/sec warmup 100 sec latency 68.462 ms 2024-05-27T07:43:28.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.478+0000 7f9ca5c96640 1 -- 172.21.15.114:0/1480261164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0109030 msgr2=0x7f9ca0102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.478+0000 7f9ca5c96640 1 --2- 172.21.15.114:0/1480261164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0109030 0x7f9ca0102c70 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009900 tx=0x7f9c9402a410 comp rx=0 tx=0).stop 2024-05-27T07:43:28.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.478+0000 7f9ca5c96640 1 -- 172.21.15.114:0/1480261164 shutdown_connections 2024-05-27T07:43:28.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.478+0000 7f9ca5c96640 1 --2- 172.21.15.114:0/1480261164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ca01032e0 0x7f9ca010f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.478+0000 7f9ca5c96640 1 --2- 172.21.15.114:0/1480261164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0109030 0x7f9ca0102c70 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.478+0000 7f9ca5c96640 1 --2- 172.21.15.114:0/1480261164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ca0108b90 0x7f9ca0102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.478+0000 7f9ca5c96640 1 -- 172.21.15.114:0/1480261164 >> 172.21.15.114:0/1480261164 conn(0x7f9ca00782d0 msgr2=0x7f9ca0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:28.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.479+0000 7f9ca5c96640 1 -- 172.21.15.114:0/1480261164 shutdown_connections 2024-05-27T07:43:28.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.479+0000 7f9ca5c96640 1 -- 172.21.15.114:0/1480261164 wait complete. 2024-05-27T07:43:28.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.479+0000 7f9ca5c96640 1 Processor -- start 2024-05-27T07:43:28.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.479+0000 7f9ca5c96640 1 -- start start 2024-05-27T07:43:28.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9ca5c96640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ca01032e0 0x7f9ca01a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9ca5c96640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ca0108b90 0x7f9ca01a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9ca5c96640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0109030 0x7f9ca019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9ca5c96640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0112d80 con 0x7f9ca0109030 2024-05-27T07:43:28.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9ca5c96640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0112c00 con 0x7f9ca01032e0 2024-05-27T07:43:28.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9ca5c96640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0112f00 con 0x7f9ca0108b90 2024-05-27T07:43:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9c9ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0109030 0x7f9ca019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9c9ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0109030 0x7f9ca019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54644/0 (socket says 172.21.15.114:54644) 2024-05-27T07:43:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9c9ffff640 1 -- 172.21.15.114:0/3152028535 learned_addr learned my addr 172.21.15.114:0/3152028535 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9c9effd640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ca0108b90 0x7f9ca01a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9c9f7fe640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ca01032e0 0x7f9ca01a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9c9ffff640 1 -- 172.21.15.114:0/3152028535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ca0108b90 msgr2=0x7f9ca01a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9c9ffff640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ca0108b90 0x7f9ca01a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9c9ffff640 1 -- 172.21.15.114:0/3152028535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ca01032e0 msgr2=0x7f9ca01a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9c9ffff640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ca01032e0 0x7f9ca01a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9c9ffff640 1 -- 172.21.15.114:0/3152028535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca019ae40 con 0x7f9ca0109030 2024-05-27T07:43:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9c9ffff640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0109030 0x7f9ca019a710 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800eb10 tx=0x7f9c9800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.480+0000 7f9c9cff9640 1 -- 172.21.15.114:0/3152028535 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98004b20 con 0x7f9ca0109030 2024-05-27T07:43:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.481+0000 7f9c9cff9640 1 -- 172.21.15.114:0/3152028535 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c98004cc0 con 0x7f9ca0109030 2024-05-27T07:43:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.481+0000 7f9c9cff9640 1 -- 172.21.15.114:0/3152028535 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98010670 con 0x7f9ca0109030 2024-05-27T07:43:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.481+0000 7f9ca5c96640 1 -- 172.21.15.114:0/3152028535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca019b0d0 con 0x7f9ca0109030 2024-05-27T07:43:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.481+0000 7f9ca5c96640 1 -- 172.21.15.114:0/3152028535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca01a7330 con 0x7f9ca0109030 2024-05-27T07:43:28.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.483+0000 7f9cacc28740 1 Processor -- start 2024-05-27T07:43:28.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.483+0000 7f9c9cff9640 1 -- 172.21.15.114:0/3152028535 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f9c98005020 con 0x7f9ca0109030 2024-05-27T07:43:28.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.483+0000 7f9cacc28740 1 -- start start 2024-05-27T07:43:28.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.483+0000 7f9cacc28740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c28c1b0 0x561b2c62d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.483+0000 7f9cacc28740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c2632f0 0x561b2c62d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.483+0000 7f9cacc28740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c26b4d0 0x561b2c62de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.483+0000 7f9cacc28740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561b2c59fc30 con 0x561b2c26b4d0 2024-05-27T07:43:28.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.483+0000 7f9cacc28740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561b2c58f7d0 con 0x561b2c28c1b0 2024-05-27T07:43:28.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.483+0000 7f9cacc28740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561b2c593810 con 0x561b2c2632f0 2024-05-27T07:43:28.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c26b4d0 0x561b2c62de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c2632f0 0x561b2c62d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c2632f0 0x561b2c62d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36526/0 (socket says 172.21.15.114:36526) 2024-05-27T07:43:28.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c26b4d0 0x561b2c62de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54658/0 (socket says 172.21.15.114:54658) 2024-05-27T07:43:28.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9f7fe640 1 -- 172.21.15.114:0/3557652591 learned_addr learned my addr 172.21.15.114:0/3557652591 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:28.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9effd640 1 --2- 172.21.15.114:0/3557652591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c28c1b0 0x561b2c62d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9cff9640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9c880b2950 0x7f9c880b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9f7fe640 1 -- 172.21.15.114:0/3557652591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c28c1b0 msgr2=0x561b2c62d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:28.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9f7fe640 1 --2- 172.21.15.114:0/3557652591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c28c1b0 0x561b2c62d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9f7fe640 1 -- 172.21.15.114:0/3557652591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c26b4d0 msgr2=0x561b2c62de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:28.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9f7fe640 1 --2- 172.21.15.114:0/3557652591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c26b4d0 0x561b2c62de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9f7fe640 1 -- 172.21.15.114:0/3557652591 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b2c2a4210 con 0x561b2c2632f0 2024-05-27T07:43:28.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9effd640 1 --2- 172.21.15.114:0/3557652591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c28c1b0 0x561b2c62d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:43:28.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9ffff640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9c880b2950 0x7f9c880b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9f7fe640 1 --2- 172.21.15.114:0/3557652591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c2632f0 0x561b2c62d910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009bb0 tx=0x7f9c900117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8e8a6d560a234aa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c9cff9640 1 -- 172.21.15.114:0/3152028535 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9466+0+0 (secure 0 0 0) 0x7f9c980e3050 con 0x7f9ca0109030 2024-05-27T07:43:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c85ffb640 1 -- 172.21.15.114:0/3557652591 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90012450 con 0x561b2c2632f0 2024-05-27T07:43:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c85ffb640 1 -- 172.21.15.114:0/3557652591 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c90012aa0 con 0x561b2c2632f0 2024-05-27T07:43:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.484+0000 7f9c85ffb640 1 -- 172.21.15.114:0/3557652591 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9000f9e0 con 0x561b2c2632f0 2024-05-27T07:43:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.485+0000 7f9cacc28740 1 -- 172.21.15.114:0/3557652591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c2632f0 msgr2=0x561b2c62d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.485+0000 7f9c9ffff640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9c880b2950 0x7f9c880b4e60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9c980049d0 tx=0x7f9c980d9590 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.485+0000 7f9cacc28740 1 --2- 172.21.15.114:0/3557652591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c2632f0 0x561b2c62d910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009bb0 tx=0x7f9c900117b0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.485+0000 7f9cacc28740 1 -- 172.21.15.114:0/3557652591 shutdown_connections 2024-05-27T07:43:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.485+0000 7f9cacc28740 1 --2- 172.21.15.114:0/3557652591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c26b4d0 0x561b2c62de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.485+0000 7f9cacc28740 1 --2- 172.21.15.114:0/3557652591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c2632f0 0x561b2c62d910 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.485+0000 7f9cacc28740 1 --2- 172.21.15.114:0/3557652591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c28c1b0 0x561b2c62d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.485+0000 7f9cacc28740 1 -- 172.21.15.114:0/3557652591 >> 172.21.15.114:0/3557652591 conn(0x561b2c280300 msgr2=0x561b2c629b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.485+0000 7f9cacc28740 1 -- 172.21.15.114:0/3557652591 shutdown_connections 2024-05-27T07:43:28.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.485+0000 7f9cacc28740 1 -- 172.21.15.114:0/3557652591 wait complete. 2024-05-27T07:43:28.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9cacc28740 1 -- start start 2024-05-27T07:43:28.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9cacc28740 1 Processor -- start 2024-05-27T07:43:28.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9cacc28740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 0x561b2c637a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9cacc28740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 0x561b2c637fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9cacc28740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c28c1b0 0x561b2c63d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9cacc28740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561b2c275790 con 0x561b2c28c1b0 2024-05-27T07:43:28.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9cacc28740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561b2c54b890 con 0x561b2c2632f0 2024-05-27T07:43:28.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9cacc28740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561b2c63a500 con 0x561b2c26b4d0 2024-05-27T07:43:28.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9cacc28740 10 client.0 _ll_register_callbacks cb 0x561b2c2a3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:28.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9c9effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c28c1b0 0x561b2c63d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9c9f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 0x561b2c637fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9c9effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 0x561b2c637a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9c9effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 0x561b2c637a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:55192/0 (socket says 172.21.15.114:55192) 2024-05-27T07:43:28.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9c9effd640 1 -- 172.21.15.114:0/2032528719 learned_addr learned my addr 172.21.15.114:0/2032528719 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:28.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.486+0000 7f9c9f7fe640 1 --2- 172.21.15.114:0/2032528719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 0x561b2c637fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36528/0 (socket says 172.21.15.114:36528) 2024-05-27T07:43:28.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9c9effd640 1 -- 172.21.15.114:0/2032528719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 msgr2=0x561b2c637fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:28.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9c9effd640 1 --2- 172.21.15.114:0/2032528719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 0x561b2c637fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9c9effd640 1 -- 172.21.15.114:0/2032528719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c28c1b0 msgr2=0x561b2c63d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9c9effd640 1 --2- 172.21.15.114:0/2032528719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c28c1b0 0x561b2c63d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9c9effd640 1 -- 172.21.15.114:0/2032528719 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b2c370620 con 0x561b2c2632f0 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9c9effd640 1 --2- 172.21.15.114:0/2032528719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 0x561b2c637a80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9c940096c0 tx=0x7f9c94000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4f238dda1207ee9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9c7effd640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9c7effd640 1 -- 172.21.15.114:0/2032528719 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c94041020 con 0x561b2c2632f0 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9cacc28740 1 -- 172.21.15.114:0/2032528719 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c90012aa0 con 0x561b2c2632f0 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9c7effd640 1 -- 172.21.15.114:0/2032528719 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9466+0+0 (secure 0 0 0) 0x7f9c94031b10 con 0x561b2c2632f0 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9c7effd640 1 -- 172.21.15.114:0/2032528719 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9c94053050 con 0x561b2c2632f0 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9cacc28740 10 client.25661 fetch_fsmap learned FSMap version 27 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9cacc28740 1 -- 172.21.15.114:0/2032528719 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561b2c646800 con 0x561b2c2632f0 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.487+0000 7f9c7effd640 1 -- 172.21.15.114:0/2032528719 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c90012aa0 con 0x561b2c2632f0 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.488+0000 7f9c7effd640 1 -- 172.21.15.114:0/2032528719 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f9c94005240 con 0x561b2c2632f0 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.488+0000 7f9cacc28740 10 client.25661 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:43:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.488+0000 7f9cacc28740 10 client.25661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.488+0000 7f9cacc28740 20 client.25661 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.488+0000 7f9cacc28740 1 --2- 172.21.15.114:0/2032528719 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561b2c24ad80 0x561b2c649c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.488+0000 7f9cacc28740 4 client.25661 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.488+0000 7f9cacc28740 1 -- 172.21.15.114:0/2032528719 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b2c5a1b80 con 0x561b2c24ad80 2024-05-27T07:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.488+0000 7f9c9f7fe640 1 --2- 172.21.15.114:0/2032528719 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561b2c24ad80 0x561b2c649c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.488+0000 7f9c7effd640 10 client.25661 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.488+0000 7f9c9f7fe640 1 --2- 172.21.15.114:0/2032528719 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561b2c24ad80 0x561b2c649c80 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abec8102b8644fc3 server_cookie=8eaeaf6422413cce in_seq=0 out_seq=0 2024-05-27T07:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9c7effd640 1 -- 172.21.15.114:0/2032528719 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561b2c5a1b80 con 0x561b2c24ad80 2024-05-27T07:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9c7effd640 10 client.25661 handle_command_reply: tid=0 2024-05-27T07:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9cacc28740 1 client.25661 shutdown 2024-05-27T07:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9cacc28740 1 -- 172.21.15.114:0/2032528719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 msgr2=0x561b2c637a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9cacc28740 1 --2- 172.21.15.114:0/2032528719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 0x561b2c637a80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9c940096c0 tx=0x7f9c94000bc0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9cacc28740 1 -- 172.21.15.114:0/2032528719 shutdown_connections 2024-05-27T07:43:28.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9cacc28740 1 --2- 172.21.15.114:0/2032528719 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561b2c24ad80 0x561b2c649c80 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9cacc28740 1 --2- 172.21.15.114:0/2032528719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c28c1b0 0x561b2c63d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9c7effd640 0 client.25661 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:28.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9cacc28740 1 --2- 172.21.15.114:0/2032528719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 0x561b2c637fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9cacc28740 1 --2- 172.21.15.114:0/2032528719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 0x561b2c637a80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9cacc28740 1 -- 172.21.15.114:0/2032528719 >> 172.21.15.114:0/2032528719 conn(0x561b2c280300 msgr2=0x561b2c62d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:28.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9cacc28740 1 -- 172.21.15.114:0/2032528719 shutdown_connections 2024-05-27T07:43:28.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9cacc28740 1 -- 172.21.15.114:0/2032528719 wait complete. 2024-05-27T07:43:28.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.490+0000 7f9cacc28740 20 client.25661 trim_cache size 0 max 16384 2024-05-27T07:43:28.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.514+0000 7f9cacc28740 1 Processor -- start 2024-05-27T07:43:28.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.514+0000 7f9cacc28740 1 -- start start 2024-05-27T07:43:28.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.515+0000 7f9cacc28740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c24ad80 0x561b2c62d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.515+0000 7f9cacc28740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 0x561b2c64f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.515+0000 7f9cacc28740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 0x561b2c64f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.515+0000 7f9cacc28740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01af390 con 0x561b2c24ad80 2024-05-27T07:43:28.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.515+0000 7f9cacc28740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01b1c10 con 0x561b2c2632f0 2024-05-27T07:43:28.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.515+0000 7f9cacc28740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01afa00 con 0x561b2c26b4d0 2024-05-27T07:43:28.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.515+0000 7f9c9f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 0x561b2c64f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.515+0000 7f9c9f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 0x561b2c64f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:55204/0 (socket says 172.21.15.114:55204) 2024-05-27T07:43:28.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.515+0000 7f9c9f7fe640 1 -- 172.21.15.114:0/2746620012 learned_addr learned my addr 172.21.15.114:0/2746620012 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:28.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.516+0000 7f9c9f7fe640 1 -- 172.21.15.114:0/2746620012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 msgr2=0x561b2c64f980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:43:28.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.516+0000 7f9c9f7fe640 1 --2- 172.21.15.114:0/2746620012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 0x561b2c64f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.516+0000 7f9c9f7fe640 1 -- 172.21.15.114:0/2746620012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c24ad80 msgr2=0x561b2c62d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:28.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.516+0000 7f9c9f7fe640 1 --2- 172.21.15.114:0/2746620012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c24ad80 0x561b2c62d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.516 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.516+0000 7f9c9f7fe640 1 -- 172.21.15.114:0/2746620012 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b2c635dc0 con 0x561b2c2632f0 2024-05-27T07:43:28.516 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.517+0000 7f9c9f7fe640 1 --2- 172.21.15.114:0/2746620012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 0x561b2c64f440 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9c90012470 tx=0x7f9c90009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9870473a84e9ee51 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:28.516 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.517+0000 7f9c857fa640 1 -- 172.21.15.114:0/2746620012 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9002d070 con 0x561b2c2632f0 2024-05-27T07:43:28.516 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.517+0000 7f9c857fa640 1 -- 172.21.15.114:0/2746620012 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c900054f0 con 0x561b2c2632f0 2024-05-27T07:43:28.516 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.517+0000 7f9c857fa640 1 -- 172.21.15.114:0/2746620012 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9000fdb0 con 0x561b2c2632f0 2024-05-27T07:43:28.517 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.518+0000 7f9cacc28740 1 -- 172.21.15.114:0/2746620012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 msgr2=0x561b2c64f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.517 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.518+0000 7f9cacc28740 1 --2- 172.21.15.114:0/2746620012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 0x561b2c64f440 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9c90012470 tx=0x7f9c90009b00 comp rx=0 tx=0).stop 2024-05-27T07:43:28.518 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.519+0000 7f9cacc28740 1 -- 172.21.15.114:0/2746620012 shutdown_connections 2024-05-27T07:43:28.518 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.519+0000 7f9cacc28740 1 --2- 172.21.15.114:0/2746620012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 0x561b2c64f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.518 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.519+0000 7f9cacc28740 1 --2- 172.21.15.114:0/2746620012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c2632f0 0x561b2c64f440 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.518 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.519+0000 7f9cacc28740 1 --2- 172.21.15.114:0/2746620012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c24ad80 0x561b2c62d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.518 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.519+0000 7f9cacc28740 1 -- 172.21.15.114:0/2746620012 >> 172.21.15.114:0/2746620012 conn(0x561b2c280300 msgr2=0x561b2c62d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:28.521 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.521+0000 7f9cacc28740 1 -- 172.21.15.114:0/2746620012 shutdown_connections 2024-05-27T07:43:28.521 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.521+0000 7f9cacc28740 1 -- 172.21.15.114:0/2746620012 wait complete. 2024-05-27T07:43:28.521 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.521+0000 7f9cacc28740 1 -- start start 2024-05-27T07:43:28.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.522+0000 7f9cacc28740 1 Processor -- start 2024-05-27T07:43:28.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.522+0000 7f9cacc28740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c24ad80 0x561b2c643b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.522+0000 7f9cacc28740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c2632f0 0x561b2c640ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.522+0000 7f9cacc28740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 0x561b2c641420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.522+0000 7f9cacc28740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01b00d0 con 0x561b2c2632f0 2024-05-27T07:43:28.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.522+0000 7f9cacc28740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01b1540 con 0x561b2c24ad80 2024-05-27T07:43:28.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.522+0000 7f9cacc28740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01b07a0 con 0x561b2c26b4d0 2024-05-27T07:43:28.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.522+0000 7f9cacc28740 10 client.0 _ll_register_callbacks cb 0x7f9c94009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:28.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.523+0000 7f9c9effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 0x561b2c641420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.523+0000 7f9c9effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c24ad80 0x561b2c643b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.523+0000 7f9c9effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c24ad80 0x561b2c643b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:55218/0 (socket says 172.21.15.114:55218) 2024-05-27T07:43:28.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.523+0000 7f9c9effd640 1 -- 172.21.15.114:0/197922277 learned_addr learned my addr 172.21.15.114:0/197922277 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:28.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.524+0000 7f9c9effd640 1 -- 172.21.15.114:0/197922277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 msgr2=0x561b2c641420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:28.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.524+0000 7f9c9effd640 1 --2- 172.21.15.114:0/197922277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 0x561b2c641420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.524+0000 7f9c9effd640 1 -- 172.21.15.114:0/197922277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c2632f0 msgr2=0x561b2c640ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:28.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.524+0000 7f9c9effd640 1 --2- 172.21.15.114:0/197922277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c2632f0 0x561b2c640ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.524+0000 7f9c9effd640 1 -- 172.21.15.114:0/197922277 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b2c641960 con 0x561b2c24ad80 2024-05-27T07:43:28.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.525+0000 7f9c9effd640 1 --2- 172.21.15.114:0/197922277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c24ad80 0x561b2c643b70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9c9402a930 tx=0x7f9c94000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dba81ef68676d77 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:28.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.525+0000 7f9c7effd640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:43:28.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.525+0000 7f9c7effd640 1 -- 172.21.15.114:0/197922277 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c94041070 con 0x561b2c24ad80 2024-05-27T07:43:28.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.525+0000 7f9cacc28740 1 -- 172.21.15.114:0/197922277 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c900054f0 con 0x561b2c24ad80 2024-05-27T07:43:28.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.525+0000 7f9c7effd640 1 -- 172.21.15.114:0/197922277 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9466+0+0 (secure 0 0 0) 0x7f9c940046c0 con 0x561b2c24ad80 2024-05-27T07:43:28.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.525+0000 7f9c7effd640 1 -- 172.21.15.114:0/197922277 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9c94063050 con 0x561b2c24ad80 2024-05-27T07:43:28.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.527+0000 7f9cacc28740 10 client.25667 fetch_fsmap learned FSMap version 27 2024-05-27T07:43:28.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.527+0000 7f9cacc28740 1 -- 172.21.15.114:0/197922277 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561b2c6afee0 con 0x561b2c24ad80 2024-05-27T07:43:28.527 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.528+0000 7f9c7effd640 1 -- 172.21.15.114:0/197922277 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c900054f0 con 0x561b2c24ad80 2024-05-27T07:43:28.528 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.528+0000 7f9c7effd640 1 -- 172.21.15.114:0/197922277 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f9c94003680 con 0x561b2c24ad80 2024-05-27T07:43:28.528 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.528+0000 7f9cacc28740 10 client.25667 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:43:28.528 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.528+0000 7f9cacc28740 10 client.25667 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:28.528 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.528+0000 7f9cacc28740 20 client.25667 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:28.528 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.528+0000 7f9cacc28740 1 --2- 172.21.15.114:0/197922277 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561b2c28c1b0 0x561b2c6b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.528 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.528+0000 7f9cacc28740 4 client.25667 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:43:28.528 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.528+0000 7f9cacc28740 1 -- 172.21.15.114:0/197922277 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561b2c59b360 con 0x561b2c28c1b0 2024-05-27T07:43:28.528 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.529+0000 7f9c9f7fe640 1 --2- 172.21.15.114:0/197922277 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561b2c28c1b0 0x561b2c6b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.529 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.529+0000 7f9c7effd640 10 client.25667 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:28.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.529+0000 7f9c9f7fe640 1 --2- 172.21.15.114:0/197922277 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561b2c28c1b0 0x561b2c6b2670 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a26227392a9bb4 server_cookie=6eb87ab1e81e4745 in_seq=0 out_seq=0 2024-05-27T07:43:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9c7effd640 1 -- 172.21.15.114:0/197922277 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x561b2c59b360 con 0x561b2c28c1b0 2024-05-27T07:43:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9c7effd640 10 client.25667 handle_command_reply: tid=0 2024-05-27T07:43:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9cacc28740 1 client.25667 shutdown 2024-05-27T07:43:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9cacc28740 1 -- 172.21.15.114:0/197922277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c24ad80 msgr2=0x561b2c643b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9cacc28740 1 --2- 172.21.15.114:0/197922277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c24ad80 0x561b2c643b70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9c9402a930 tx=0x7f9c94000bc0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9cacc28740 1 -- 172.21.15.114:0/197922277 shutdown_connections 2024-05-27T07:43:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9cacc28740 1 --2- 172.21.15.114:0/197922277 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561b2c28c1b0 0x561b2c6b2670 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9cacc28740 1 --2- 172.21.15.114:0/197922277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b2c26b4d0 0x561b2c641420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9cacc28740 1 --2- 172.21.15.114:0/197922277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b2c2632f0 0x561b2c640ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9cacc28740 1 --2- 172.21.15.114:0/197922277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561b2c24ad80 0x561b2c643b70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9cacc28740 1 -- 172.21.15.114:0/197922277 >> 172.21.15.114:0/197922277 conn(0x561b2c280300 msgr2=0x561b2c638d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9c7effd640 0 client.25667 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9cacc28740 1 -- 172.21.15.114:0/197922277 shutdown_connections 2024-05-27T07:43:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.799+0000 7f9cacc28740 1 -- 172.21.15.114:0/197922277 wait complete. 2024-05-27T07:43:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.800+0000 7f9cacc28740 20 client.25667 trim_cache size 0 max 16384 2024-05-27T07:43:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:43:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:43:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:43:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.800+0000 7f9ca5c96640 1 -- 172.21.15.114:0/3152028535 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9c880b2950 msgr2=0x7f9c880b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.800+0000 7f9ca5c96640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9c880b2950 0x7f9c880b4e60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9c980049d0 tx=0x7f9c980d9590 comp rx=0 tx=0).stop 2024-05-27T07:43:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.800+0000 7f9ca5c96640 1 -- 172.21.15.114:0/3152028535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0109030 msgr2=0x7f9ca019a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.800+0000 7f9ca5c96640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0109030 0x7f9ca019a710 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800eb10 tx=0x7f9c9800c2d0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.800+0000 7f9ca5c96640 1 -- 172.21.15.114:0/3152028535 shutdown_connections 2024-05-27T07:43:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.800+0000 7f9ca5c96640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9c880b2950 0x7f9c880b4e60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.800+0000 7f9ca5c96640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0109030 0x7f9ca019a710 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.800+0000 7f9ca5c96640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ca0108b90 0x7f9ca01a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.800+0000 7f9ca5c96640 1 --2- 172.21.15.114:0/3152028535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ca01032e0 0x7f9ca01a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.800+0000 7f9ca5c96640 1 -- 172.21.15.114:0/3152028535 >> 172.21.15.114:0/3152028535 conn(0x7f9ca00782d0 msgr2=0x7f9ca0106240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.801+0000 7f9ca5c96640 1 -- 172.21.15.114:0/3152028535 shutdown_connections 2024-05-27T07:43:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:28.802+0000 7f9ca5c96640 1 -- 172.21.15.114:0/3152028535 wait complete. 2024-05-27T07:43:28.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:28 smithi163 ceph-mon[36451]: pgmap v118: 129 pgs: 129 active+clean; 51 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 374 op/s 2024-05-27T07:43:28.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:28 smithi186 ceph-mon[36647]: pgmap v118: 129 pgs: 129 active+clean; 51 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 374 op/s 2024-05-27T07:43:29.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:28 smithi114 ceph-mon[32038]: pgmap v118: 129 pgs: 129 active+clean; 51 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 374 op/s 2024-05-27T07:43:29.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 263234 14.22 MB/sec warmup 101 sec latency 126.681 ms 2024-05-27T07:43:29.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.392+0000 7f0547fa9700 1 -- 172.21.15.114:0/3561506292 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05401034b0 msgr2=0x7f05401038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:29.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.392+0000 7f0547fa9700 1 --2- 172.21.15.114:0/3561506292 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05401034b0 0x7f05401038c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f053c009990 tx=0x7f053c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.392+0000 7f0547fa9700 1 -- 172.21.15.114:0/3561506292 shutdown_connections 2024-05-27T07:43:29.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.392+0000 7f0547fa9700 1 --2- 172.21.15.114:0/3561506292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0540108730 0x7f0540110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.392+0000 7f0547fa9700 1 --2- 172.21.15.114:0/3561506292 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05401034b0 0x7f05401038c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.392+0000 7f0547fa9700 1 --2- 172.21.15.114:0/3561506292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0540107de0 0x7f05401081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.392+0000 7f0547fa9700 1 -- 172.21.15.114:0/3561506292 >> 172.21.15.114:0/3561506292 conn(0x7f05400fedb0 msgr2=0x7f05401011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:29.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.392+0000 7f0547fa9700 1 -- 172.21.15.114:0/3561506292 shutdown_connections 2024-05-27T07:43:29.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.392+0000 7f0547fa9700 1 -- 172.21.15.114:0/3561506292 wait complete. 2024-05-27T07:43:29.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.393+0000 7f0547fa9700 1 Processor -- start 2024-05-27T07:43:29.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.393+0000 7f0547fa9700 1 -- start start 2024-05-27T07:43:29.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.393+0000 7f0547fa9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05401034b0 0x7f054019aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.393+0000 7f0547fa9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0540107de0 0x7f054019b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.393+0000 7f0547fa9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0540108730 0x7f054019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.393+0000 7f0547fa9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0540113360 con 0x7f05401034b0 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.393+0000 7f0547fa9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f05401131e0 con 0x7f0540107de0 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.393+0000 7f0547fa9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f05401134e0 con 0x7f0540108730 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0545d45700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05401034b0 0x7f054019aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0545544700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0540107de0 0x7f054019b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0545d45700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05401034b0 0x7f054019aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54704/0 (socket says 172.21.15.114:54704) 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0545d45700 1 -- 172.21.15.114:0/475572102 learned_addr learned my addr 172.21.15.114:0/475572102 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0546546700 1 --2- 172.21.15.114:0/475572102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0540108730 0x7f054019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0545544700 1 -- 172.21.15.114:0/475572102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0540108730 msgr2=0x7f054019f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0545544700 1 --2- 172.21.15.114:0/475572102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0540108730 0x7f054019f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0545544700 1 -- 172.21.15.114:0/475572102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05401034b0 msgr2=0x7f054019aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0545544700 1 --2- 172.21.15.114:0/475572102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05401034b0 0x7f054019aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0545544700 1 -- 172.21.15.114:0/475572102 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054019fdb0 con 0x7f0540107de0 2024-05-27T07:43:29.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0545544700 1 --2- 172.21.15.114:0/475572102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0540107de0 0x7f054019b480 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f053000e9c0 tx=0x7f053000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:29.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0536ffd700 1 -- 172.21.15.114:0/475572102 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053000c9c0 con 0x7f0540107de0 2024-05-27T07:43:29.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0536ffd700 1 -- 172.21.15.114:0/475572102 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f053000cb60 con 0x7f0540107de0 2024-05-27T07:43:29.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0547fa9700 1 -- 172.21.15.114:0/475572102 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05401a00a0 con 0x7f0540107de0 2024-05-27T07:43:29.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0536ffd700 1 -- 172.21.15.114:0/475572102 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0530018680 con 0x7f0540107de0 2024-05-27T07:43:29.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.394+0000 7f0547fa9700 1 -- 172.21.15.114:0/475572102 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05401a79c0 con 0x7f0540107de0 2024-05-27T07:43:29.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.397+0000 7f0536ffd700 1 -- 172.21.15.114:0/475572102 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f0530018820 con 0x7f0540107de0 2024-05-27T07:43:29.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.398+0000 7f0536ffd700 1 --2- 172.21.15.114:0/475572102 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f052c0b2a70 0x7f052c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.398+0000 7f0545d45700 1 --2- 172.21.15.114:0/475572102 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f052c0b2a70 0x7f052c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.398+0000 7f0536ffd700 1 -- 172.21.15.114:0/475572102 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9466+0+0 (secure 0 0 0) 0x7f05300e2be0 con 0x7f0540107de0 2024-05-27T07:43:29.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.398+0000 7f0545d45700 1 --2- 172.21.15.114:0/475572102 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f052c0b2a70 0x7f052c0b4f80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f053c009750 tx=0x7f053c00b6f0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:29.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.398+0000 7f05569beb80 1 Processor -- start 2024-05-27T07:43:29.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f05569beb80 1 -- start start 2024-05-27T07:43:29.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f05569beb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6815d7e40 0x55f68175cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f05569beb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f68175d070 0x55f681756be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f05569beb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f681757120 0x55f681757530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f05569beb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f681684d70 con 0x55f6815d7e40 2024-05-27T07:43:29.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f05569beb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f681674910 con 0x55f68175d070 2024-05-27T07:43:29.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f05569beb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f681678950 con 0x55f681757120 2024-05-27T07:43:29.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f0546546700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f681757120 0x55f681757530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f0546546700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f681757120 0x55f681757530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36562/0 (socket says 172.21.15.114:36562) 2024-05-27T07:43:29.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f0546546700 1 -- 172.21.15.114:0/66102634 learned_addr learned my addr 172.21.15.114:0/66102634 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:29.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f0545544700 1 --2- 172.21.15.114:0/66102634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f68175d070 0x55f681756be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f0545d45700 1 --2- 172.21.15.114:0/66102634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6815d7e40 0x55f68175cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f0545544700 1 -- 172.21.15.114:0/66102634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f681757120 msgr2=0x55f681757530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:29.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f0545544700 1 --2- 172.21.15.114:0/66102634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f681757120 0x55f681757530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f0545544700 1 -- 172.21.15.114:0/66102634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6815d7e40 msgr2=0x55f68175cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:29.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.399+0000 7f0545544700 1 --2- 172.21.15.114:0/66102634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6815d7e40 0x55f68175cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.400+0000 7f0545544700 1 -- 172.21.15.114:0/66102634 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f68161dfb0 con 0x55f68175d070 2024-05-27T07:43:29.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.400+0000 7f0545544700 1 --2- 172.21.15.114:0/66102634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f68175d070 0x55f681756be0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f05300dff20 tx=0x7f05300a9a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc990f7df72f56c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:29.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.400+0000 7f052bfff700 1 -- 172.21.15.114:0/66102634 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05300a6160 con 0x55f68175d070 2024-05-27T07:43:29.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.400+0000 7f052bfff700 1 -- 172.21.15.114:0/66102634 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05300a6300 con 0x55f68175d070 2024-05-27T07:43:29.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.400+0000 7f052bfff700 1 -- 172.21.15.114:0/66102634 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0530010630 con 0x55f68175d070 2024-05-27T07:43:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.401+0000 7f05569beb80 1 -- 172.21.15.114:0/66102634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f68175d070 msgr2=0x55f681756be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.401+0000 7f05569beb80 1 --2- 172.21.15.114:0/66102634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f68175d070 0x55f681756be0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f05300dff20 tx=0x7f05300a9a30 comp rx=0 tx=0).stop 2024-05-27T07:43:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.401+0000 7f05569beb80 1 -- 172.21.15.114:0/66102634 shutdown_connections 2024-05-27T07:43:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.401+0000 7f05569beb80 1 --2- 172.21.15.114:0/66102634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f681757120 0x55f681757530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.401+0000 7f05569beb80 1 --2- 172.21.15.114:0/66102634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6815d7e40 0x55f68175cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.401+0000 7f05569beb80 1 --2- 172.21.15.114:0/66102634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f68175d070 0x55f681756be0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.401+0000 7f05569beb80 1 -- 172.21.15.114:0/66102634 >> 172.21.15.114:0/66102634 conn(0x55f6815da770 msgr2=0x55f681751d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:29.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.401+0000 7f05569beb80 1 -- 172.21.15.114:0/66102634 shutdown_connections 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.401+0000 7f05569beb80 1 -- 172.21.15.114:0/66102634 wait complete. 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.402+0000 7f05569beb80 1 -- start start 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.402+0000 7f05569beb80 1 Processor -- start 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.402+0000 7f05569beb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f6815d7e40 0x55f681760960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.402+0000 7f05569beb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f681757120 0x55f681760ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.402+0000 7f05569beb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f6817613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.402+0000 7f05569beb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f68157afa0 con 0x55f681757120 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.402+0000 7f05569beb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f6815dcda0 con 0x55f6815d7e40 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.402+0000 7f05569beb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f681767f70 con 0x55f68175d070 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.402+0000 7f05569beb80 10 client.0 _ll_register_callbacks cb 0x55f68160b5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.402+0000 7f0546546700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f6817613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.402+0000 7f0545544700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f681757120 0x55f681760ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f0546546700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f6817613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36576/0 (socket says 172.21.15.114:36576) 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f0546546700 1 -- 172.21.15.114:0/1181759645 learned_addr learned my addr 172.21.15.114:0/1181759645 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f0545d45700 1 --2- 172.21.15.114:0/1181759645 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f6815d7e40 0x55f681760960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f0546546700 1 -- 172.21.15.114:0/1181759645 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f6815d7e40 msgr2=0x55f681760960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f0546546700 1 --2- 172.21.15.114:0/1181759645 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f6815d7e40 0x55f681760960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f0546546700 1 -- 172.21.15.114:0/1181759645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f681757120 msgr2=0x55f681760ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f0546546700 1 --2- 172.21.15.114:0/1181759645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f681757120 0x55f681760ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f0546546700 1 -- 172.21.15.114:0/1181759645 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6815c48d0 con 0x55f68175d070 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f0546546700 1 --2- 172.21.15.114:0/1181759645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f6817613e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f05380047a0 tx=0x7f0538014600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a9ff51c2cabe3b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f05297fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f05297fa700 1 -- 172.21.15.114:0/1181759645 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05380048f0 con 0x55f68175d070 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f05569beb80 1 -- 172.21.15.114:0/1181759645 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05300a6300 con 0x55f68175d070 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f05297fa700 1 -- 172.21.15.114:0/1181759645 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9466+0+0 (secure 0 0 0) 0x7f0538014a60 con 0x55f68175d070 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f05297fa700 1 -- 172.21.15.114:0/1181759645 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0538014c60 con 0x55f68175d070 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f05569beb80 10 client.25744 fetch_fsmap learned FSMap version 27 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.403+0000 7f05569beb80 1 -- 172.21.15.114:0/1181759645 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f68162e760 con 0x55f68175d070 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.404+0000 7f05297fa700 1 -- 172.21.15.114:0/1181759645 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05300a6300 con 0x55f68175d070 2024-05-27T07:43:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.404+0000 7f05297fa700 1 -- 172.21.15.114:0/1181759645 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f053801e420 con 0x55f68175d070 2024-05-27T07:43:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.404+0000 7f05569beb80 10 client.25744 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:43:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.404+0000 7f05569beb80 10 client.25744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.404+0000 7f05569beb80 20 client.25744 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.404+0000 7f05569beb80 1 --2- 172.21.15.114:0/1181759645 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f681774fc0 0x55f6817773e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.404+0000 7f05569beb80 4 client.25744 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.404+0000 7f05569beb80 1 -- 172.21.15.114:0/1181759645 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f681686cc0 con 0x55f681774fc0 2024-05-27T07:43:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.404+0000 7f0545d45700 1 --2- 172.21.15.114:0/1181759645 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f681774fc0 0x55f6817773e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.405+0000 7f05297fa700 10 client.25744 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.405+0000 7f0545d45700 1 --2- 172.21.15.114:0/1181759645 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f681774fc0 0x55f6817773e0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60ec18c5d4ede540 server_cookie=c782064534ce7c14 in_seq=0 out_seq=0 2024-05-27T07:43:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05297fa700 1 -- 172.21.15.114:0/1181759645 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f681686cc0 con 0x55f681774fc0 2024-05-27T07:43:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05297fa700 10 client.25744 handle_command_reply: tid=0 2024-05-27T07:43:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05569beb80 1 client.25744 shutdown 2024-05-27T07:43:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05569beb80 1 -- 172.21.15.114:0/1181759645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 msgr2=0x55f6817613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:29.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05569beb80 1 --2- 172.21.15.114:0/1181759645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f6817613e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f05380047a0 tx=0x7f0538014600 comp rx=0 tx=0).stop 2024-05-27T07:43:29.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05569beb80 1 -- 172.21.15.114:0/1181759645 shutdown_connections 2024-05-27T07:43:29.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05569beb80 1 --2- 172.21.15.114:0/1181759645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f6817613e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05569beb80 1 --2- 172.21.15.114:0/1181759645 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f681774fc0 0x55f6817773e0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05569beb80 1 --2- 172.21.15.114:0/1181759645 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f6815d7e40 0x55f681760960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05569beb80 1 --2- 172.21.15.114:0/1181759645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f681757120 0x55f681760ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05569beb80 1 -- 172.21.15.114:0/1181759645 >> 172.21.15.114:0/1181759645 conn(0x55f6815da770 msgr2=0x55f681755bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:29.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05297fa700 0 client.25744 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:29.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05569beb80 1 -- 172.21.15.114:0/1181759645 shutdown_connections 2024-05-27T07:43:29.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.406+0000 7f05569beb80 1 -- 172.21.15.114:0/1181759645 wait complete. 2024-05-27T07:43:29.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.407+0000 7f05569beb80 20 client.25744 trim_cache size 0 max 16384 2024-05-27T07:43:29.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.427+0000 7f05569beb80 1 Processor -- start 2024-05-27T07:43:29.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.427+0000 7f05569beb80 1 -- start start 2024-05-27T07:43:29.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f05569beb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f6815d7e40 0x55f68175aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f05569beb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f681757120 0x55f68177cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f05569beb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f68177d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f05569beb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f05401ac490 con 0x55f681757120 2024-05-27T07:43:29.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f05569beb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f05401aed10 con 0x55f6815d7e40 2024-05-27T07:43:29.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f05569beb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f05401acb00 con 0x55f68175d070 2024-05-27T07:43:29.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f0545544700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f681757120 0x55f68177cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f0545d45700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f6815d7e40 0x55f68175aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f0545544700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f681757120 0x55f68177cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54724/0 (socket says 172.21.15.114:54724) 2024-05-27T07:43:29.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f0545544700 1 -- 172.21.15.114:0/3257443189 learned_addr learned my addr 172.21.15.114:0/3257443189 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:29.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f0546546700 1 --2- 172.21.15.114:0/3257443189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f68177d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f0545d45700 1 -- 172.21.15.114:0/3257443189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 msgr2=0x55f68177d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:29.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f0545d45700 1 --2- 172.21.15.114:0/3257443189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f68177d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f0545d45700 1 -- 172.21.15.114:0/3257443189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f681757120 msgr2=0x55f68177cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:29.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f0545d45700 1 --2- 172.21.15.114:0/3257443189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f681757120 0x55f68177cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f0545d45700 1 -- 172.21.15.114:0/3257443189 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f68175eef0 con 0x55f6815d7e40 2024-05-27T07:43:29.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f0545544700 1 --2- 172.21.15.114:0/3257443189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f681757120 0x55f68177cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:29.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.428+0000 7f0545d45700 1 --2- 172.21.15.114:0/3257443189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f6815d7e40 0x55f68175aec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x55f681777b40 tx=0x7f053c01e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca1e44ed6a08d4f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:29.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.429+0000 7f052b7fe700 1 -- 172.21.15.114:0/3257443189 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c003900 con 0x55f6815d7e40 2024-05-27T07:43:29.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.429+0000 7f052b7fe700 1 -- 172.21.15.114:0/3257443189 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f053c034430 con 0x55f6815d7e40 2024-05-27T07:43:29.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.429+0000 7f052b7fe700 1 -- 172.21.15.114:0/3257443189 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c00f6c0 con 0x55f6815d7e40 2024-05-27T07:43:29.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.429+0000 7f05569beb80 1 -- 172.21.15.114:0/3257443189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f6815d7e40 msgr2=0x55f68175aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:29.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.429+0000 7f05569beb80 1 --2- 172.21.15.114:0/3257443189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f6815d7e40 0x55f68175aec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x55f681777b40 tx=0x7f053c01e000 comp rx=0 tx=0).stop 2024-05-27T07:43:29.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.429+0000 7f05569beb80 1 -- 172.21.15.114:0/3257443189 shutdown_connections 2024-05-27T07:43:29.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.430+0000 7f05569beb80 1 --2- 172.21.15.114:0/3257443189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f68177d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.430+0000 7f05569beb80 1 --2- 172.21.15.114:0/3257443189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f6815d7e40 0x55f68175aec0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.430+0000 7f05569beb80 1 --2- 172.21.15.114:0/3257443189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f681757120 0x55f68177cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.430+0000 7f05569beb80 1 -- 172.21.15.114:0/3257443189 >> 172.21.15.114:0/3257443189 conn(0x55f6815da770 msgr2=0x55f68175b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:29.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.430+0000 7f05569beb80 1 -- 172.21.15.114:0/3257443189 shutdown_connections 2024-05-27T07:43:29.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.430+0000 7f05569beb80 1 -- 172.21.15.114:0/3257443189 wait complete. 2024-05-27T07:43:29.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.430+0000 7f05569beb80 1 -- start start 2024-05-27T07:43:29.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.430+0000 7f05569beb80 1 Processor -- start 2024-05-27T07:43:29.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f05569beb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6815d7e40 0x55f68176b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f05569beb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f681757120 0x55f68176b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f05569beb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f6817dea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f05569beb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f05401ad1d0 con 0x55f6815d7e40 2024-05-27T07:43:29.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f05569beb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f05401ae640 con 0x55f681757120 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f05569beb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f05401ad8a0 con 0x55f68175d070 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f05569beb80 10 client.0 _ll_register_callbacks cb 0x55f68160b5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f0545d45700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6815d7e40 0x55f68176b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f0545d45700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6815d7e40 0x55f68176b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54728/0 (socket says 172.21.15.114:54728) 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f0545d45700 1 -- 172.21.15.114:0/1502267183 learned_addr learned my addr 172.21.15.114:0/1502267183 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f0545544700 1 --2- 172.21.15.114:0/1502267183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f681757120 0x55f68176b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f0546546700 1 --2- 172.21.15.114:0/1502267183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f6817dea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f0545d45700 1 -- 172.21.15.114:0/1502267183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 msgr2=0x55f6817dea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f0545d45700 1 --2- 172.21.15.114:0/1502267183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f6817dea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f0545d45700 1 -- 172.21.15.114:0/1502267183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f681757120 msgr2=0x55f68176b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f0545d45700 1 --2- 172.21.15.114:0/1502267183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f681757120 0x55f68176b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f0545d45700 1 -- 172.21.15.114:0/1502267183 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f68176c090 con 0x55f6815d7e40 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f0545d45700 1 --2- 172.21.15.114:0/1502267183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6815d7e40 0x55f68176b2c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f053c009ce0 tx=0x7f053c00bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4cd06850db8181 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f05297fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f05297fa700 1 -- 172.21.15.114:0/1502267183 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c042060 con 0x55f6815d7e40 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.431+0000 7f05297fa700 1 -- 172.21.15.114:0/1502267183 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9466+0+0 (secure 0 0 0) 0x7f053c003880 con 0x55f6815d7e40 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.432+0000 7f05569beb80 1 -- 172.21.15.114:0/1502267183 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f053c034430 con 0x55f6815d7e40 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.432+0000 7f05297fa700 1 -- 172.21.15.114:0/1502267183 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f053c003a80 con 0x55f6815d7e40 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.432+0000 7f05569beb80 10 client.16158 fetch_fsmap learned FSMap version 27 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.432+0000 7f05569beb80 1 -- 172.21.15.114:0/1502267183 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f6817dffe0 con 0x55f6815d7e40 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.432+0000 7f05297fa700 1 -- 172.21.15.114:0/1502267183 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f053c034430 con 0x55f6815d7e40 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.432+0000 7f05297fa700 1 -- 172.21.15.114:0/1502267183 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f053c022070 con 0x55f6815d7e40 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.432+0000 7f05569beb80 10 client.16158 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.432+0000 7f05569beb80 10 client.16158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.432+0000 7f05569beb80 20 client.16158 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.432+0000 7f05569beb80 1 --2- 172.21.15.114:0/1502267183 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f6817e0660 0x55f6817e2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.432+0000 7f05569beb80 4 client.16158 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:43:29.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.432+0000 7f05569beb80 1 -- 172.21.15.114:0/1502267183 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f6816804a0 con 0x55f6817e0660 2024-05-27T07:43:29.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.432+0000 7f0545544700 1 --2- 172.21.15.114:0/1502267183 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f6817e0660 0x55f6817e2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:29.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.433+0000 7f05297fa700 10 client.16158 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:29.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.433+0000 7f0545544700 1 --2- 172.21.15.114:0/1502267183 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f6817e0660 0x55f6817e2a80 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=422e5b33da1301de server_cookie=f16dac00e8bcca1 in_seq=0 out_seq=0 2024-05-27T07:43:29.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05297fa700 1 -- 172.21.15.114:0/1502267183 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f6816804a0 con 0x55f6817e0660 2024-05-27T07:43:29.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05297fa700 10 client.16158 handle_command_reply: tid=0 2024-05-27T07:43:29.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05569beb80 1 client.16158 shutdown 2024-05-27T07:43:29.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05569beb80 1 -- 172.21.15.114:0/1502267183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6815d7e40 msgr2=0x55f68176b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:29.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05569beb80 1 --2- 172.21.15.114:0/1502267183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6815d7e40 0x55f68176b2c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f053c009ce0 tx=0x7f053c00bfb0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05569beb80 1 -- 172.21.15.114:0/1502267183 shutdown_connections 2024-05-27T07:43:29.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05569beb80 1 --2- 172.21.15.114:0/1502267183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f68175d070 0x55f6817dea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05569beb80 1 --2- 172.21.15.114:0/1502267183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6815d7e40 0x55f68176b2c0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05569beb80 1 --2- 172.21.15.114:0/1502267183 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f6817e0660 0x55f6817e2a80 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05569beb80 1 --2- 172.21.15.114:0/1502267183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f681757120 0x55f68176b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05297fa700 0 client.16158 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05569beb80 1 -- 172.21.15.114:0/1502267183 >> 172.21.15.114:0/1502267183 conn(0x55f6815da770 msgr2=0x55f68175b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05569beb80 1 -- 172.21.15.114:0/1502267183 shutdown_connections 2024-05-27T07:43:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05569beb80 1 -- 172.21.15.114:0/1502267183 wait complete. 2024-05-27T07:43:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.434+0000 7f05569beb80 20 client.16158 trim_cache size 0 max 16384 2024-05-27T07:43:29.439 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:43:29.439 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:43:29.439 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:43:29.439 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:43:29.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.435+0000 7f0547fa9700 1 -- 172.21.15.114:0/475572102 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f052c0b2a70 msgr2=0x7f052c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:29.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.435+0000 7f0547fa9700 1 --2- 172.21.15.114:0/475572102 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f052c0b2a70 0x7f052c0b4f80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f053c009750 tx=0x7f053c00b6f0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.435+0000 7f0547fa9700 1 -- 172.21.15.114:0/475572102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0540107de0 msgr2=0x7f054019b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:29.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.435+0000 7f0547fa9700 1 --2- 172.21.15.114:0/475572102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0540107de0 0x7f054019b480 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f053000e9c0 tx=0x7f053000ecd0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.435+0000 7f0547fa9700 1 -- 172.21.15.114:0/475572102 shutdown_connections 2024-05-27T07:43:29.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.435+0000 7f0547fa9700 1 --2- 172.21.15.114:0/475572102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0540108730 0x7f054019f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.435+0000 7f0547fa9700 1 --2- 172.21.15.114:0/475572102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05401034b0 0x7f054019aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.435+0000 7f0547fa9700 1 --2- 172.21.15.114:0/475572102 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f052c0b2a70 0x7f052c0b4f80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.435+0000 7f0547fa9700 1 --2- 172.21.15.114:0/475572102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0540107de0 0x7f054019b480 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:29.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.435+0000 7f0547fa9700 1 -- 172.21.15.114:0/475572102 >> 172.21.15.114:0/475572102 conn(0x7f05400fedb0 msgr2=0x7f054006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:29.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.435+0000 7f0547fa9700 1 -- 172.21.15.114:0/475572102 shutdown_connections 2024-05-27T07:43:29.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:29.435+0000 7f0547fa9700 1 -- 172.21.15.114:0/475572102 wait complete. 2024-05-27T07:43:29.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:28 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:43:29.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:28 smithi163 systemd[1]: Stopped Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:43:29.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:28 smithi163 systemd[1]: Starting Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:43:29.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:29 smithi163 podman[70799]: 2024-05-27 07:43:29.109721599 +0000 UTC m=+0.018576759 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:43:30.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 265780 14.22 MB/sec warmup 102 sec latency 128.929 ms 2024-05-27T07:43:30.291 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:43:30.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:43:30.544 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:30.974 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:30 smithi163 podman[70799]: 2024-05-27 07:43:30.61136426 +0000 UTC m=+1.520219409 container create 240d1e072553d1f6192bb7e87dcc9a4776f1701132c9c64ba264d1f49a44bd0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-05-27T07:43:31.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 267248 14.15 MB/sec warmup 103 sec latency 126.834 ms 2024-05-27T07:43:31.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:31 smithi114 ceph-mon[32038]: pgmap v119: 129 pgs: 129 active+clean; 41 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 558 op/s 2024-05-27T07:43:31.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:31 smithi163 ceph-mon[36451]: pgmap v119: 129 pgs: 129 active+clean; 41 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 558 op/s 2024-05-27T07:43:31.724 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:31 smithi163 podman[70799]: 2024-05-27 07:43:31.372702795 +0000 UTC m=+2.281557943 container init 240d1e072553d1f6192bb7e87dcc9a4776f1701132c9c64ba264d1f49a44bd0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-27T07:43:31.724 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:31 smithi163 podman[70799]: 2024-05-27 07:43:31.375793554 +0000 UTC m=+2.284648703 container start 240d1e072553d1f6192bb7e87dcc9a4776f1701132c9c64ba264d1f49a44bd0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:43:31.724 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:31 smithi163 ceph-mds[70818]: -- 172.21.15.163:0/3206536179 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560b231c1a40 con 0x560b23209180 2024-05-27T07:43:31.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:31 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: starting mds.b at 2024-05-27T07:43:31.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:31 smithi163 bash[70799]: 240d1e072553d1f6192bb7e87dcc9a4776f1701132c9c64ba264d1f49a44bd0c 2024-05-27T07:43:31.725 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:43:31 smithi163 systemd[1]: Started Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:43:31.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:31 smithi186 ceph-mon[36647]: pgmap v119: 129 pgs: 129 active+clean; 41 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 558 op/s 2024-05-27T07:43:32.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 270149 14.18 MB/sec warmup 104 sec latency 32.779 ms 2024-05-27T07:43:32.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.340+0000 7f149f7fe700 1 -- 172.21.15.114:0/3202297252 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1498003b40 con 0x7f14a8107de0 2024-05-27T07:43:32.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.341+0000 7f14af20e700 1 -- 172.21.15.114:0/3202297252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14a8107de0 msgr2=0x7f14a81081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:32.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.341+0000 7f14af20e700 1 --2- 172.21.15.114:0/3202297252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14a8107de0 0x7f14a81081f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f14980099c0 tx=0x7f1498009cd0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.341+0000 7f14af20e700 1 -- 172.21.15.114:0/3202297252 shutdown_connections 2024-05-27T07:43:32.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.341+0000 7f14af20e700 1 --2- 172.21.15.114:0/3202297252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14a8108730 0x7f14a8110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.341+0000 7f14af20e700 1 --2- 172.21.15.114:0/3202297252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14a81034b0 0x7f14a81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.341+0000 7f14af20e700 1 --2- 172.21.15.114:0/3202297252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14a8107de0 0x7f14a81081f0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.341+0000 7f14af20e700 1 -- 172.21.15.114:0/3202297252 >> 172.21.15.114:0/3202297252 conn(0x7f14a80fedb0 msgr2=0x7f14a81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:32.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.341+0000 7f14af20e700 1 -- 172.21.15.114:0/3202297252 shutdown_connections 2024-05-27T07:43:32.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.341+0000 7f14af20e700 1 -- 172.21.15.114:0/3202297252 wait complete. 2024-05-27T07:43:32.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.342+0000 7f14af20e700 1 Processor -- start 2024-05-27T07:43:32.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.342+0000 7f14af20e700 1 -- start start 2024-05-27T07:43:32.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.342+0000 7f14af20e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14a81034b0 0x7f14a819f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:32.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.342+0000 7f14af20e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14a8107de0 0x7f14a819fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:32.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.342+0000 7f14af20e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14a8108730 0x7f14a8199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:32.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.342+0000 7f14af20e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f14a8113340 con 0x7f14a8107de0 2024-05-27T07:43:32.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.342+0000 7f14af20e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f14a81131c0 con 0x7f14a81034b0 2024-05-27T07:43:32.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.342+0000 7f14af20e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f14a81134c0 con 0x7f14a8108730 2024-05-27T07:43:32.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.342+0000 7f149ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14a8107de0 0x7f14a819fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.342+0000 7f149ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14a8107de0 0x7f14a819fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54748/0 (socket says 172.21.15.114:54748) 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.342+0000 7f149ffff700 1 -- 172.21.15.114:0/3280602707 learned_addr learned my addr 172.21.15.114:0/3280602707 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.343+0000 7f14ad7ab700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14a8108730 0x7f14a8199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.342+0000 7f14acfaa700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14a81034b0 0x7f14a819f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.343+0000 7f14acfaa700 1 -- 172.21.15.114:0/3280602707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14a8108730 msgr2=0x7f14a8199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.343+0000 7f14acfaa700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14a8108730 0x7f14a8199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.343+0000 7f14acfaa700 1 -- 172.21.15.114:0/3280602707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14a8107de0 msgr2=0x7f14a819fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.343+0000 7f14acfaa700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14a8107de0 0x7f14a819fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.343+0000 7f14acfaa700 1 -- 172.21.15.114:0/3280602707 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a819a1c0 con 0x7f14a81034b0 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.343+0000 7f149ffff700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14a8107de0 0x7f14a819fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.343+0000 7f14acfaa700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14a81034b0 0x7f14a819f8a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f14a400ef10 tx=0x7f14a400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.343+0000 7f149dffb700 1 -- 172.21.15.114:0/3280602707 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a400c9c0 con 0x7f14a81034b0 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.343+0000 7f14af20e700 1 -- 172.21.15.114:0/3280602707 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14a819a450 con 0x7f14a81034b0 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.343+0000 7f149dffb700 1 -- 172.21.15.114:0/3280602707 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14a4019070 con 0x7f14a81034b0 2024-05-27T07:43:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.343+0000 7f149dffb700 1 -- 172.21.15.114:0/3280602707 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a4014480 con 0x7f14a81034b0 2024-05-27T07:43:32.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.344+0000 7f14af20e700 1 -- 172.21.15.114:0/3280602707 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14a806b600 con 0x7f14a81034b0 2024-05-27T07:43:32.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.345+0000 7f14af20e700 1 -- 172.21.15.114:0/3280602707 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14a8104960 con 0x7f14a81034b0 2024-05-27T07:43:32.345 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:32 smithi114 ceph-mon[32038]: daemon mds.b restarted 2024-05-27T07:43:32.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.346+0000 7f149dffb700 1 -- 172.21.15.114:0/3280602707 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f14a4012050 con 0x7f14a81034b0 2024-05-27T07:43:32.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.347+0000 7f149dffb700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f14900b29f0 0x7f14900b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:32.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.347+0000 7f149ffff700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f14900b29f0 0x7f14900b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:32.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.347+0000 7f149dffb700 1 -- 172.21.15.114:0/3280602707 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f14a40e2d20 con 0x7f14a81034b0 2024-05-27T07:43:32.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.348+0000 7f149ffff700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f14900b29f0 0x7f14900b4f00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f14a8107d40 tx=0x7f149801c040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:32.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.351+0000 7f149dffb700 1 -- 172.21.15.114:0/3280602707 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f14a4010040 con 0x7f14a81034b0 2024-05-27T07:43:32.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.666+0000 7f14af20e700 1 -- 172.21.15.114:0/3280602707 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14a819afd0 con 0x7f14a81034b0 2024-05-27T07:43:32.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.666+0000 7f149dffb700 1 -- 172.21.15.114:0/3280602707 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11827 (secure 0 0 0) 0x7f14a40aa350 con 0x7f14a81034b0 2024-05-27T07:43:32.666 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:43:32.666 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:43:32.272613+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25691":{"gid":25691,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3475666132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3475666132},{"type":"v1","addr":"172.21.15.163:6839","nonce":3475666132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:43:32.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.670+0000 7f14af20e700 1 -- 172.21.15.114:0/3280602707 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f14900b29f0 msgr2=0x7f14900b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:32.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.670+0000 7f14af20e700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f14900b29f0 0x7f14900b4f00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f14a8107d40 tx=0x7f149801c040 comp rx=0 tx=0).stop 2024-05-27T07:43:32.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.670+0000 7f14af20e700 1 -- 172.21.15.114:0/3280602707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14a81034b0 msgr2=0x7f14a819f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:32.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.670+0000 7f14af20e700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14a81034b0 0x7f14a819f8a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f14a400ef10 tx=0x7f14a400c3a0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.670+0000 7f14af20e700 1 -- 172.21.15.114:0/3280602707 shutdown_connections 2024-05-27T07:43:32.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.670+0000 7f14af20e700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14a8108730 0x7f14a8199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.670+0000 7f14af20e700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f14900b29f0 0x7f14900b4f00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.670+0000 7f14af20e700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14a81034b0 0x7f14a819f8a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.670+0000 7f14af20e700 1 --2- 172.21.15.114:0/3280602707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14a8107de0 0x7f14a819fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.670+0000 7f14af20e700 1 -- 172.21.15.114:0/3280602707 >> 172.21.15.114:0/3280602707 conn(0x7f14a80fedb0 msgr2=0x7f14a81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:32.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.670+0000 7f14af20e700 1 -- 172.21.15.114:0/3280602707 shutdown_connections 2024-05-27T07:43:32.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:32.670+0000 7f14af20e700 1 -- 172.21.15.114:0/3280602707 wait complete. 2024-05-27T07:43:32.670 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 29 2024-05-27T07:43:32.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:32 smithi163 ceph-mon[36451]: daemon mds.b restarted 2024-05-27T07:43:32.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:32 smithi186 ceph-mon[36647]: daemon mds.b restarted 2024-05-27T07:43:33.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 273269 14.21 MB/sec warmup 105 sec latency 24.566 ms 2024-05-27T07:43:33.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:43:33.560 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:33 smithi114 ceph-mon[32038]: pgmap v120: 129 pgs: 129 active+clean; 40 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 478 op/s 2024-05-27T07:43:33.561 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:33 smithi114 ceph-mon[32038]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T07:43:33.561 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:33 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6838/3475666132,v1:172.21.15.163:6839/3475666132] up:boot 2024-05-27T07:43:33.561 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:33 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:43:33.561 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:43:33.561 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:33 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:43:33.561 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:33 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3280602707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:33.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:33 smithi163 ceph-mon[36451]: pgmap v120: 129 pgs: 129 active+clean; 40 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 478 op/s 2024-05-27T07:43:33.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:33 smithi163 ceph-mon[36451]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T07:43:33.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:33 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6838/3475666132,v1:172.21.15.163:6839/3475666132] up:boot 2024-05-27T07:43:33.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:33 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:43:33.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:43:33.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:33 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:43:33.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:33 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3280602707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:33.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:33 smithi186 ceph-mon[36647]: pgmap v120: 129 pgs: 129 active+clean; 40 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 478 op/s 2024-05-27T07:43:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:33 smithi186 ceph-mon[36647]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T07:43:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:33 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6838/3475666132,v1:172.21.15.163:6839/3475666132] up:boot 2024-05-27T07:43:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:33 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-27T07:43:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:43:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:33 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-05-27T07:43:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:33 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3280602707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:33.735 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:34.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 276027 14.19 MB/sec warmup 106 sec latency 14.803 ms 2024-05-27T07:43:35.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 278251 14.18 MB/sec warmup 107 sec latency 23.281 ms 2024-05-27T07:43:35.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:35 smithi114 ceph-mon[32038]: pgmap v122: 129 pgs: 129 active+clean; 40 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 486 op/s 2024-05-27T07:43:35.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:35 smithi163 ceph-mon[36451]: pgmap v122: 129 pgs: 129 active+clean; 40 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 486 op/s 2024-05-27T07:43:35.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:35 smithi186 ceph-mon[36647]: pgmap v122: 129 pgs: 129 active+clean; 40 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 486 op/s 2024-05-27T07:43:35.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.810+0000 7ff8ff85d700 1 -- 172.21.15.114:0/3574657083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8f8108f10 msgr2=0x7ff8f8109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:35.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.810+0000 7ff8ff85d700 1 --2- 172.21.15.114:0/3574657083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8f8108f10 0x7ff8f8109320 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80098f0 tx=0x7ff8e8009c00 comp rx=0 tx=0).stop 2024-05-27T07:43:35.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.810+0000 7ff8ff85d700 1 -- 172.21.15.114:0/3574657083 shutdown_connections 2024-05-27T07:43:35.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.810+0000 7ff8ff85d700 1 --2- 172.21.15.114:0/3574657083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f8109860 0x7ff8f81111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.810+0000 7ff8ff85d700 1 --2- 172.21.15.114:0/3574657083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8f81045e0 0x7ff8f81049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.810+0000 7ff8ff85d700 1 --2- 172.21.15.114:0/3574657083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8f8108f10 0x7ff8f8109320 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.810+0000 7ff8ff85d700 1 -- 172.21.15.114:0/3574657083 >> 172.21.15.114:0/3574657083 conn(0x7ff8f8100030 msgr2=0x7ff8f8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:35.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.811+0000 7ff8ff85d700 1 -- 172.21.15.114:0/3574657083 shutdown_connections 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.811+0000 7ff8ff85d700 1 -- 172.21.15.114:0/3574657083 wait complete. 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.811+0000 7ff8ff85d700 1 Processor -- start 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.811+0000 7ff8ff85d700 1 -- start start 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8ff85d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f81045e0 0x7ff8f819aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8ff85d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8f8108f10 0x7ff8f819b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8ff85d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8f8109860 0x7ff8f819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8ff85d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f81144e0 con 0x7ff8f8108f10 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8ff85d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f8114360 con 0x7ff8f8109860 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8ff85d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f8114660 con 0x7ff8f81045e0 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8fcdf8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8f8108f10 0x7ff8f819b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8fd5f9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f81045e0 0x7ff8f819aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8fcdf8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8f8108f10 0x7ff8f819b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48076/0 (socket says 172.21.15.114:48076) 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8fcdf8700 1 -- 172.21.15.114:0/1748577256 learned_addr learned my addr 172.21.15.114:0/1748577256 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:35.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8fddfa700 1 --2- 172.21.15.114:0/1748577256 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8f8109860 0x7ff8f819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8fcdf8700 1 -- 172.21.15.114:0/1748577256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f81045e0 msgr2=0x7ff8f819aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:35.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8fcdf8700 1 --2- 172.21.15.114:0/1748577256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f81045e0 0x7ff8f819aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8fcdf8700 1 -- 172.21.15.114:0/1748577256 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8f8109860 msgr2=0x7ff8f819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:35.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8fcdf8700 1 --2- 172.21.15.114:0/1748577256 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8f8109860 0x7ff8f819f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8fcdf8700 1 -- 172.21.15.114:0/1748577256 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f819fdc0 con 0x7ff8f8108f10 2024-05-27T07:43:35.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8fcdf8700 1 --2- 172.21.15.114:0/1748577256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8f8108f10 0x7ff8f819b490 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800c010 tx=0x7ff8e800f9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:35.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.812+0000 7ff8ee7fc700 1 -- 172.21.15.114:0/1748577256 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8027070 con 0x7ff8f8108f10 2024-05-27T07:43:35.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.813+0000 7ff8ff85d700 1 -- 172.21.15.114:0/1748577256 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f81a0050 con 0x7ff8f8108f10 2024-05-27T07:43:35.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.813+0000 7ff8ee7fc700 1 -- 172.21.15.114:0/1748577256 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e80054f0 con 0x7ff8f8108f10 2024-05-27T07:43:35.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.813+0000 7ff8ee7fc700 1 -- 172.21.15.114:0/1748577256 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8004930 con 0x7ff8f8108f10 2024-05-27T07:43:35.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.813+0000 7ff8ff85d700 1 -- 172.21.15.114:0/1748577256 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f81a0530 con 0x7ff8f8108f10 2024-05-27T07:43:35.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.815+0000 7ff8ee7fc700 1 -- 172.21.15.114:0/1748577256 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ff8e802a070 con 0x7ff8f8108f10 2024-05-27T07:43:35.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.816+0000 7ff8ee7fc700 1 --2- 172.21.15.114:0/1748577256 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff8e40b6e00 0x7ff8e40b9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.816+0000 7ff8fd5f9700 1 --2- 172.21.15.114:0/1748577256 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff8e40b6e00 0x7ff8e40b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.816+0000 7ff8ee7fc700 1 -- 172.21.15.114:0/1748577256 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff8e80e3390 con 0x7ff8f8108f10 2024-05-27T07:43:35.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff90e272b80 1 Processor -- start 2024-05-27T07:43:35.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff8fd5f9700 1 --2- 172.21.15.114:0/1748577256 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff8e40b6e00 0x7ff8e40b9310 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4009750 tx=0x7ff8f4009310 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:35.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff90e272b80 1 -- start start 2024-05-27T07:43:35.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff90e272b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d3abe40 0x56343d52f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff90e272b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d52fc50 0x56343d529a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff90e272b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d52a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff90e272b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56343d458c90 con 0x56343d529f80 2024-05-27T07:43:35.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff90e272b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56343d448830 con 0x56343d52fc50 2024-05-27T07:43:35.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff90e272b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56343d44c870 con 0x56343d3abe40 2024-05-27T07:43:35.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff8fd5f9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d3abe40 0x56343d52f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff8fddfa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d52a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff8fddfa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d52a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48082/0 (socket says 172.21.15.114:48082) 2024-05-27T07:43:35.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff8fcdf8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d52fc50 0x56343d529a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff8fddfa700 1 -- 172.21.15.114:0/4173613791 learned_addr learned my addr 172.21.15.114:0/4173613791 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:35.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.817+0000 7ff8fd5f9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d3abe40 0x56343d52f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:43288/0 (socket says 172.21.15.114:43288) 2024-05-27T07:43:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.818+0000 7ff8fddfa700 1 -- 172.21.15.114:0/4173613791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d3abe40 msgr2=0x56343d52f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.818+0000 7ff8fddfa700 1 --2- 172.21.15.114:0/4173613791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d3abe40 0x56343d52f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.818+0000 7ff8fddfa700 1 -- 172.21.15.114:0/4173613791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d52fc50 msgr2=0x56343d529a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.818+0000 7ff8fddfa700 1 --2- 172.21.15.114:0/4173613791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d52fc50 0x56343d529a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.818+0000 7ff8fddfa700 1 -- 172.21.15.114:0/4173613791 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56343d3988d0 con 0x56343d529f80 2024-05-27T07:43:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.818+0000 7ff8fcdf8700 1 --2- 172.21.15.114:0/4173613791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d52fc50 0x56343d529a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.818+0000 7ff8fddfa700 1 --2- 172.21.15.114:0/4173613791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d52a390 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0011680 tx=0x7ff8f0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80732c84113bf5b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.818+0000 7ff8e37fe700 1 -- 172.21.15.114:0/4173613791 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0019030 con 0x56343d529f80 2024-05-27T07:43:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.818+0000 7ff8e37fe700 1 -- 172.21.15.114:0/4173613791 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8f00092e0 con 0x56343d529f80 2024-05-27T07:43:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.818+0000 7ff8e37fe700 1 -- 172.21.15.114:0/4173613791 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0007500 con 0x56343d529f80 2024-05-27T07:43:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.819+0000 7ff90e272b80 1 -- 172.21.15.114:0/4173613791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 msgr2=0x56343d52a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.819+0000 7ff90e272b80 1 --2- 172.21.15.114:0/4173613791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d52a390 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0011680 tx=0x7ff8f0009920 comp rx=0 tx=0).stop 2024-05-27T07:43:35.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.819+0000 7ff90e272b80 1 -- 172.21.15.114:0/4173613791 shutdown_connections 2024-05-27T07:43:35.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.819+0000 7ff90e272b80 1 --2- 172.21.15.114:0/4173613791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d52a390 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.819+0000 7ff90e272b80 1 --2- 172.21.15.114:0/4173613791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d3abe40 0x56343d52f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.819+0000 7ff90e272b80 1 --2- 172.21.15.114:0/4173613791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d52fc50 0x56343d529a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.819+0000 7ff90e272b80 1 -- 172.21.15.114:0/4173613791 >> 172.21.15.114:0/4173613791 conn(0x56343d406af0 msgr2=0x56343d524640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:35.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.819+0000 7ff90e272b80 1 -- 172.21.15.114:0/4173613791 shutdown_connections 2024-05-27T07:43:35.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.819+0000 7ff90e272b80 1 -- 172.21.15.114:0/4173613791 wait complete. 2024-05-27T07:43:35.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.820+0000 7ff90e272b80 1 -- start start 2024-05-27T07:43:35.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.820+0000 7ff90e272b80 1 Processor -- start 2024-05-27T07:43:35.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.820+0000 7ff90e272b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d3abe40 0x56343d533360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.820+0000 7ff90e272b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d5338a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff90e272b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d52fc50 0x56343d533de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff90e272b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56343d34efa0 con 0x56343d529f80 2024-05-27T07:43:35.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff90e272b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56343d3ea7c0 con 0x56343d3abe40 2024-05-27T07:43:35.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff90e272b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56343d536740 con 0x56343d52fc50 2024-05-27T07:43:35.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff8fcdf8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d5338a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff90e272b80 10 client.0 _ll_register_callbacks cb 0x56343d3e3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:35.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff8fcdf8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d5338a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48086/0 (socket says 172.21.15.114:48086) 2024-05-27T07:43:35.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff8fcdf8700 1 -- 172.21.15.114:0/135707571 learned_addr learned my addr 172.21.15.114:0/135707571 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:35.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff8fd5f9700 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d3abe40 0x56343d533360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff8fddfa700 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d52fc50 0x56343d533de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff8fcdf8700 1 -- 172.21.15.114:0/135707571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d52fc50 msgr2=0x56343d533de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:35.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff8fcdf8700 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d52fc50 0x56343d533de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff8fcdf8700 1 -- 172.21.15.114:0/135707571 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d3abe40 msgr2=0x56343d533360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff8fcdf8700 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d3abe40 0x56343d533360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff8fcdf8700 1 -- 172.21.15.114:0/135707571 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56343d3e7690 con 0x56343d529f80 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff8fddfa700 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d52fc50 0x56343d533de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff8fcdf8700 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d5338a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80a75c0 tx=0x7ff8e80a6840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2455b894de3c00fe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.821+0000 7ff8e0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff8e0ff9700 1 -- 172.21.15.114:0/135707571 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e80a9e60 con 0x56343d529f80 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff8e0ff9700 1 -- 172.21.15.114:0/135707571 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff8e80aa5b0 con 0x56343d529f80 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff90e272b80 1 -- 172.21.15.114:0/135707571 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8f00092e0 con 0x56343d529f80 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff8e0ff9700 1 -- 172.21.15.114:0/135707571 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff8e80aaa50 con 0x56343d529f80 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff90e272b80 10 client.16179 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff90e272b80 1 -- 172.21.15.114:0/135707571 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56343d5411c0 con 0x56343d529f80 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff8e0ff9700 1 -- 172.21.15.114:0/135707571 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8f00092e0 con 0x56343d529f80 2024-05-27T07:43:35.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff8e0ff9700 1 -- 172.21.15.114:0/135707571 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7ff8e80a95b0 con 0x56343d529f80 2024-05-27T07:43:35.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff90e272b80 10 client.16179 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:35.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff90e272b80 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:35.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff90e272b80 20 client.16179 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:35.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff90e272b80 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56343d543690 0x56343d545ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff90e272b80 4 client.16179 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:35.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff90e272b80 1 -- 172.21.15.114:0/135707571 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56343d4f59f0 con 0x56343d543690 2024-05-27T07:43:35.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.822+0000 7ff8fd5f9700 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56343d543690 0x56343d545ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.823+0000 7ff8e0ff9700 10 client.16179 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:35.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.823+0000 7ff8fd5f9700 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56343d543690 0x56343d545ab0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=247b9bbbcc6c9f6b server_cookie=5b0e49b2a1ffc3 in_seq=0 out_seq=0 2024-05-27T07:43:35.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.824+0000 7ff8e0ff9700 1 -- 172.21.15.114:0/135707571 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56343d4f59f0 con 0x56343d543690 2024-05-27T07:43:35.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.824+0000 7ff8e0ff9700 10 client.16179 handle_command_reply: tid=0 2024-05-27T07:43:35.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.824+0000 7ff90e272b80 1 client.16179 shutdown 2024-05-27T07:43:35.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.824+0000 7ff90e272b80 1 -- 172.21.15.114:0/135707571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 msgr2=0x56343d5338a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.824+0000 7ff90e272b80 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d5338a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80a75c0 tx=0x7ff8e80a6840 comp rx=0 tx=0).stop 2024-05-27T07:43:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.824+0000 7ff90e272b80 1 -- 172.21.15.114:0/135707571 shutdown_connections 2024-05-27T07:43:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.824+0000 7ff90e272b80 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d52fc50 0x56343d533de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.824+0000 7ff90e272b80 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56343d543690 0x56343d545ab0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.824+0000 7ff90e272b80 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d3abe40 0x56343d533360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.825+0000 7ff90e272b80 1 --2- 172.21.15.114:0/135707571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d5338a0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.825+0000 7ff8e0ff9700 0 client.16179 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.825+0000 7ff90e272b80 1 -- 172.21.15.114:0/135707571 >> 172.21.15.114:0/135707571 conn(0x56343d406af0 msgr2=0x56343d528a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.825+0000 7ff90e272b80 1 -- 172.21.15.114:0/135707571 shutdown_connections 2024-05-27T07:43:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.825+0000 7ff90e272b80 1 -- 172.21.15.114:0/135707571 wait complete. 2024-05-27T07:43:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.825+0000 7ff90e272b80 20 client.16179 trim_cache size 0 max 16384 2024-05-27T07:43:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff90e272b80 1 Processor -- start 2024-05-27T07:43:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff90e272b80 1 -- start start 2024-05-27T07:43:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff90e272b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d3abe40 0x56343d529a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff90e272b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d53f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff90e272b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d52fc50 0x56343d52d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff90e272b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f81ac350 con 0x56343d529f80 2024-05-27T07:43:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff90e272b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f81aebd0 con 0x56343d3abe40 2024-05-27T07:43:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff90e272b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f81ac9c0 con 0x56343d52fc50 2024-05-27T07:43:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff8fcdf8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d53f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff8fcdf8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d53f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48090/0 (socket says 172.21.15.114:48090) 2024-05-27T07:43:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff8fcdf8700 1 -- 172.21.15.114:0/3003362628 learned_addr learned my addr 172.21.15.114:0/3003362628 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff8fddfa700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d52fc50 0x56343d52d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff8fd5f9700 1 --2- 172.21.15.114:0/3003362628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d3abe40 0x56343d529a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff8fcdf8700 1 -- 172.21.15.114:0/3003362628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d52fc50 msgr2=0x56343d52d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff8fcdf8700 1 --2- 172.21.15.114:0/3003362628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d52fc50 0x56343d52d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff8fcdf8700 1 -- 172.21.15.114:0/3003362628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d3abe40 msgr2=0x56343d529a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff8fcdf8700 1 --2- 172.21.15.114:0/3003362628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d3abe40 0x56343d529a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.842+0000 7ff8fcdf8700 1 -- 172.21.15.114:0/3003362628 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56343d534420 con 0x56343d529f80 2024-05-27T07:43:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.843+0000 7ff8fcdf8700 1 --2- 172.21.15.114:0/3003362628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d53f7e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80a63d0 tx=0x7ff8e80a6340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579d3649463aae37 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.843+0000 7ff8e2ffd700 1 -- 172.21.15.114:0/3003362628 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e80aa350 con 0x56343d529f80 2024-05-27T07:43:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.843+0000 7ff8e2ffd700 1 -- 172.21.15.114:0/3003362628 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e80a5cc0 con 0x56343d529f80 2024-05-27T07:43:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.843+0000 7ff8e2ffd700 1 -- 172.21.15.114:0/3003362628 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e80a95b0 con 0x56343d529f80 2024-05-27T07:43:35.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.844+0000 7ff90e272b80 1 -- 172.21.15.114:0/3003362628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 msgr2=0x56343d53f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:35.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.844+0000 7ff90e272b80 1 --2- 172.21.15.114:0/3003362628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d53f7e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80a63d0 tx=0x7ff8e80a6340 comp rx=0 tx=0).stop 2024-05-27T07:43:35.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.844+0000 7ff90e272b80 1 -- 172.21.15.114:0/3003362628 shutdown_connections 2024-05-27T07:43:35.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.844+0000 7ff90e272b80 1 --2- 172.21.15.114:0/3003362628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d52fc50 0x56343d52d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.844+0000 7ff90e272b80 1 --2- 172.21.15.114:0/3003362628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d3abe40 0x56343d529a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.844+0000 7ff90e272b80 1 --2- 172.21.15.114:0/3003362628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d529f80 0x56343d53f7e0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.844+0000 7ff90e272b80 1 -- 172.21.15.114:0/3003362628 >> 172.21.15.114:0/3003362628 conn(0x56343d406af0 msgr2=0x56343d52d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:35.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.844+0000 7ff90e272b80 1 -- 172.21.15.114:0/3003362628 shutdown_connections 2024-05-27T07:43:35.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.844+0000 7ff90e272b80 1 -- 172.21.15.114:0/3003362628 wait complete. 2024-05-27T07:43:35.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.844+0000 7ff90e272b80 1 -- start start 2024-05-27T07:43:35.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.844+0000 7ff90e272b80 1 Processor -- start 2024-05-27T07:43:35.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.845+0000 7ff90e272b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d3abe40 0x56343d5391c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.845+0000 7ff90e272b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d529f80 0x56343d539700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.845+0000 7ff90e272b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d52fc50 0x56343d53b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.845+0000 7ff8fd5f9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d3abe40 0x56343d5391c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.845+0000 7ff8fddfa700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d52fc50 0x56343d53b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.845+0000 7ff8fd5f9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d3abe40 0x56343d5391c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48096/0 (socket says 172.21.15.114:48096) 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.845+0000 7ff8fcdf8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d529f80 0x56343d539700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.845+0000 7ff8fddfa700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d52fc50 0x56343d53b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39822/0 (socket says 172.21.15.114:39822) 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.845+0000 7ff8fcdf8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d529f80 0x56343d539700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:43302/0 (socket says 172.21.15.114:43302) 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.845+0000 7ff8fd5f9700 1 -- 172.21.15.114:0/4051878976 learned_addr learned my addr 172.21.15.114:0/4051878976 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.845+0000 7ff90e272b80 1 -- 172.21.15.114:0/4051878976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f81ad090 con 0x56343d3abe40 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.845+0000 7ff90e272b80 1 -- 172.21.15.114:0/4051878976 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f81ae500 con 0x56343d52fc50 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.845+0000 7ff90e272b80 1 -- 172.21.15.114:0/4051878976 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f81ad760 con 0x56343d529f80 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff90e272b80 10 client.0 _ll_register_callbacks cb 0x56343d3e3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff8fddfa700 1 -- 172.21.15.114:0/4051878976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d529f80 msgr2=0x56343d539700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff8fddfa700 1 --2- 172.21.15.114:0/4051878976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d529f80 0x56343d539700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff8fddfa700 1 -- 172.21.15.114:0/4051878976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d3abe40 msgr2=0x56343d5391c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff8fddfa700 1 --2- 172.21.15.114:0/4051878976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d3abe40 0x56343d5391c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff8fddfa700 1 -- 172.21.15.114:0/4051878976 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56343d53bce0 con 0x56343d52fc50 2024-05-27T07:43:35.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff8fd5f9700 1 --2- 172.21.15.114:0/4051878976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d3abe40 0x56343d5391c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:35.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff8fcdf8700 1 --2- 172.21.15.114:0/4051878976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d529f80 0x56343d539700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:35.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff8fddfa700 1 --2- 172.21.15.114:0/4051878976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d52fc50 0x56343d53b720 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0000f80 tx=0x7ff8f0004110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b467b419b72623c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:35.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff8e0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:43:35.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff8e0ff9700 1 -- 172.21.15.114:0/4051878976 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0005550 con 0x56343d52fc50 2024-05-27T07:43:35.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff90e272b80 1 -- 172.21.15.114:0/4051878976 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8e80a5cc0 con 0x56343d52fc50 2024-05-27T07:43:35.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff8e0ff9700 1 -- 172.21.15.114:0/4051878976 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff8f00056f0 con 0x56343d52fc50 2024-05-27T07:43:35.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.846+0000 7ff8e0ff9700 1 -- 172.21.15.114:0/4051878976 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff8f002f050 con 0x56343d52fc50 2024-05-27T07:43:35.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.847+0000 7ff90e272b80 10 client.25715 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:35.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.847+0000 7ff90e272b80 1 -- 172.21.15.114:0/4051878976 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56343d5adb70 con 0x56343d52fc50 2024-05-27T07:43:35.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.847+0000 7ff8e0ff9700 1 -- 172.21.15.114:0/4051878976 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e80a5cc0 con 0x56343d52fc50 2024-05-27T07:43:35.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.847+0000 7ff8e0ff9700 1 -- 172.21.15.114:0/4051878976 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7ff8f0027020 con 0x56343d52fc50 2024-05-27T07:43:35.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.847+0000 7ff90e272b80 10 client.25715 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:35.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.847+0000 7ff90e272b80 10 client.25715 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:35.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.847+0000 7ff90e272b80 20 client.25715 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:35.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.847+0000 7ff90e272b80 1 --2- 172.21.15.114:0/4051878976 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56343d5ae0d0 0x56343d5b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:35.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.847+0000 7ff90e272b80 4 client.25715 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:43:35.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.847+0000 7ff90e272b80 1 -- 172.21.15.114:0/4051878976 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56343d45abe0 con 0x56343d5ae0d0 2024-05-27T07:43:35.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.847+0000 7ff8fd5f9700 1 --2- 172.21.15.114:0/4051878976 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56343d5ae0d0 0x56343d5b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:35.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff8e0ff9700 10 client.25715 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:35.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff8fd5f9700 1 --2- 172.21.15.114:0/4051878976 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56343d5ae0d0 0x56343d5b04f0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3da73a98e553fdc4 server_cookie=31f3a2199c609e68 in_seq=0 out_seq=0 2024-05-27T07:43:35.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff8e0ff9700 1 -- 172.21.15.114:0/4051878976 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56343d45abe0 con 0x56343d5ae0d0 2024-05-27T07:43:35.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff8e0ff9700 10 client.25715 handle_command_reply: tid=0 2024-05-27T07:43:35.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff90e272b80 1 client.25715 shutdown 2024-05-27T07:43:35.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff90e272b80 1 -- 172.21.15.114:0/4051878976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d52fc50 msgr2=0x56343d53b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:35.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff90e272b80 1 --2- 172.21.15.114:0/4051878976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d52fc50 0x56343d53b720 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0000f80 tx=0x7ff8f0004110 comp rx=0 tx=0).stop 2024-05-27T07:43:35.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff90e272b80 1 -- 172.21.15.114:0/4051878976 shutdown_connections 2024-05-27T07:43:35.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff90e272b80 1 --2- 172.21.15.114:0/4051878976 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56343d5ae0d0 0x56343d5b04f0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff90e272b80 1 --2- 172.21.15.114:0/4051878976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56343d52fc50 0x56343d53b720 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff90e272b80 1 --2- 172.21.15.114:0/4051878976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56343d3abe40 0x56343d5391c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff90e272b80 1 --2- 172.21.15.114:0/4051878976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56343d529f80 0x56343d539700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff8e0ff9700 0 client.25715 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:35.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff90e272b80 1 -- 172.21.15.114:0/4051878976 >> 172.21.15.114:0/4051878976 conn(0x56343d406af0 msgr2=0x56343d51ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:35.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff90e272b80 1 -- 172.21.15.114:0/4051878976 shutdown_connections 2024-05-27T07:43:35.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff90e272b80 1 -- 172.21.15.114:0/4051878976 wait complete. 2024-05-27T07:43:35.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.848+0000 7ff90e272b80 20 client.25715 trim_cache size 0 max 16384 2024-05-27T07:43:35.852 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:43:35.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.849+0000 7ff8ff85d700 1 -- 172.21.15.114:0/1748577256 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff8e40b6e00 msgr2=0x7ff8e40b9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:35.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.849+0000 7ff8ff85d700 1 --2- 172.21.15.114:0/1748577256 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff8e40b6e00 0x7ff8e40b9310 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4009750 tx=0x7ff8f4009310 comp rx=0 tx=0).stop 2024-05-27T07:43:35.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.849+0000 7ff8ff85d700 1 -- 172.21.15.114:0/1748577256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8f8108f10 msgr2=0x7ff8f819b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:35.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.849+0000 7ff8ff85d700 1 --2- 172.21.15.114:0/1748577256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8f8108f10 0x7ff8f819b490 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800c010 tx=0x7ff8e800f9b0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.849+0000 7ff8ff85d700 1 -- 172.21.15.114:0/1748577256 shutdown_connections 2024-05-27T07:43:35.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.849+0000 7ff8ff85d700 1 --2- 172.21.15.114:0/1748577256 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff8e40b6e00 0x7ff8e40b9310 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.849+0000 7ff8ff85d700 1 --2- 172.21.15.114:0/1748577256 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8f8109860 0x7ff8f819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.849+0000 7ff8ff85d700 1 --2- 172.21.15.114:0/1748577256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f81045e0 0x7ff8f819aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.849+0000 7ff8ff85d700 1 --2- 172.21.15.114:0/1748577256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8f8108f10 0x7ff8f819b490 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:35.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.849+0000 7ff8ff85d700 1 -- 172.21.15.114:0/1748577256 >> 172.21.15.114:0/1748577256 conn(0x7ff8f8100030 msgr2=0x7ff8f810a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:35.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.849+0000 7ff8ff85d700 1 -- 172.21.15.114:0/1748577256 shutdown_connections 2024-05-27T07:43:35.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:35.849+0000 7ff8ff85d700 1 -- 172.21.15.114:0/1748577256 wait complete. 2024-05-27T07:43:36.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 280823 14.18 MB/sec warmup 108 sec latency 37.700 ms 2024-05-27T07:43:37.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 283775 14.21 MB/sec warmup 109 sec latency 45.837 ms 2024-05-27T07:43:37.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:37 smithi114 ceph-mon[32038]: pgmap v123: 129 pgs: 129 active+clean; 46 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 482 op/s 2024-05-27T07:43:37.617 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:43:37.617 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:43:37.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:37 smithi163 ceph-mon[36451]: pgmap v123: 129 pgs: 129 active+clean; 46 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 482 op/s 2024-05-27T07:43:37.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:37 smithi186 ceph-mon[36647]: pgmap v123: 129 pgs: 129 active+clean; 46 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 482 op/s 2024-05-27T07:43:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:38.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 285987 14.15 MB/sec warmup 110 sec latency 87.037 ms 2024-05-27T07:43:39.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 288142 14.15 MB/sec warmup 111 sec latency 17.615 ms 2024-05-27T07:43:39.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:39 smithi114 ceph-mon[32038]: pgmap v124: 129 pgs: 129 active+clean; 46 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 482 op/s 2024-05-27T07:43:39.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:39 smithi163 ceph-mon[36451]: pgmap v124: 129 pgs: 129 active+clean; 46 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 482 op/s 2024-05-27T07:43:39.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.728+0000 7f29c3870700 1 -- 172.21.15.114:0/1542536630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29bc1042e0 msgr2=0x7f29bc110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:39.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.728+0000 7f29c3870700 1 --2- 172.21.15.114:0/1542536630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29bc1042e0 0x7f29bc110060 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f29b400b0a0 tx=0x7f29b400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.728+0000 7f29c3870700 1 -- 172.21.15.114:0/1542536630 shutdown_connections 2024-05-27T07:43:39.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.728+0000 7f29c3870700 1 --2- 172.21.15.114:0/1542536630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29bc1042e0 0x7f29bc110060 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.728+0000 7f29c3870700 1 --2- 172.21.15.114:0/1542536630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29bc1081f0 0x7f29bc108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.728+0000 7f29c3870700 1 --2- 172.21.15.114:0/1542536630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29bc103990 0x7f29bc103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.728+0000 7f29c3870700 1 -- 172.21.15.114:0/1542536630 >> 172.21.15.114:0/1542536630 conn(0x7f29bc0fe4d0 msgr2=0x7f29bc100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:39.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.728+0000 7f29c3870700 1 -- 172.21.15.114:0/1542536630 shutdown_connections 2024-05-27T07:43:39.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.728+0000 7f29c3870700 1 -- 172.21.15.114:0/1542536630 wait complete. 2024-05-27T07:43:39.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:39 smithi186 ceph-mon[36647]: pgmap v124: 129 pgs: 129 active+clean; 46 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 482 op/s 2024-05-27T07:43:39.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.729+0000 7f29c3870700 1 Processor -- start 2024-05-27T07:43:39.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.729+0000 7f29c3870700 1 -- start start 2024-05-27T07:43:39.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.729+0000 7f29c3870700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29bc103990 0x7f29bc19bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.729+0000 7f29c3870700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29bc1042e0 0x7f29bc19c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.729+0000 7f29c3870700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29bc1081f0 0x7f29bc19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.729+0000 7f29c3870700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc113380 con 0x7f29bc1042e0 2024-05-27T07:43:39.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.729+0000 7f29c3870700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc113200 con 0x7f29bc103990 2024-05-27T07:43:39.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.729+0000 7f29c3870700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc113500 con 0x7f29bc1081f0 2024-05-27T07:43:39.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.729+0000 7f29c160c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29bc103990 0x7f29bc19bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.729+0000 7f29c0e0b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29bc1042e0 0x7f29bc19c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.729+0000 7f29c160c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29bc103990 0x7f29bc19bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39842/0 (socket says 172.21.15.114:39842) 2024-05-27T07:43:39.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.729+0000 7f29c160c700 1 -- 172.21.15.114:0/2093175776 learned_addr learned my addr 172.21.15.114:0/2093175776 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:39.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.729+0000 7f29c1e0d700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29bc1081f0 0x7f29bc19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.730+0000 7f29c0e0b700 1 -- 172.21.15.114:0/2093175776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29bc1081f0 msgr2=0x7f29bc19c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:39.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.730+0000 7f29c0e0b700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29bc1081f0 0x7f29bc19c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.730+0000 7f29c0e0b700 1 -- 172.21.15.114:0/2093175776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29bc103990 msgr2=0x7f29bc19bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:39.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.730+0000 7f29c0e0b700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29bc103990 0x7f29bc19bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.730+0000 7f29c0e0b700 1 -- 172.21.15.114:0/2093175776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29bc1a6f10 con 0x7f29bc1042e0 2024-05-27T07:43:39.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.730+0000 7f29c160c700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29bc103990 0x7f29bc19bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:39.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.730+0000 7f29c1e0d700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29bc1081f0 0x7f29bc19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:39.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.730+0000 7f29c0e0b700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29bc1042e0 0x7f29bc19c0e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f29ac00e9f0 tx=0x7f29ac00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:39.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.730+0000 7f29b27fc700 1 -- 172.21.15.114:0/2093175776 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac00c9c0 con 0x7f29bc1042e0 2024-05-27T07:43:39.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.730+0000 7f29c3870700 1 -- 172.21.15.114:0/2093175776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29bc1a7200 con 0x7f29bc1042e0 2024-05-27T07:43:39.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.730+0000 7f29b27fc700 1 -- 172.21.15.114:0/2093175776 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29ac00cb60 con 0x7f29bc1042e0 2024-05-27T07:43:39.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.730+0000 7f29b27fc700 1 -- 172.21.15.114:0/2093175776 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac018680 con 0x7f29bc1042e0 2024-05-27T07:43:39.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.730+0000 7f29c3870700 1 -- 172.21.15.114:0/2093175776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29bc1a7740 con 0x7f29bc1042e0 2024-05-27T07:43:39.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.733+0000 7f29b27fc700 1 -- 172.21.15.114:0/2093175776 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f29ac018820 con 0x7f29bc1042e0 2024-05-27T07:43:39.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.734+0000 7f29b27fc700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f29a80b29f0 0x7f29a80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.734+0000 7f29d2285b80 1 Processor -- start 2024-05-27T07:43:39.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.734+0000 7f29c160c700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f29a80b29f0 0x7f29a80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.734+0000 7f29b27fc700 1 -- 172.21.15.114:0/2093175776 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f29ac0e2ba0 con 0x7f29bc1042e0 2024-05-27T07:43:39.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.734+0000 7f29d2285b80 1 -- start start 2024-05-27T07:43:39.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29d2285b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc767790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29d2285b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc767cd0 0x55b4cc761ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29c160c700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f29a80b29f0 0x7f29a80b4f00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f29b8011490 tx=0x7f29b8009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:39.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29d2285b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc762000 0x55b4cc762410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29d2285b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b4cc690d10 con 0x55b4cc767cd0 2024-05-27T07:43:39.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29d2285b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b4cc6808b0 con 0x55b4cc762000 2024-05-27T07:43:39.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29d2285b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b4cc6848f0 con 0x55b4cc5e3ec0 2024-05-27T07:43:39.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29c0e0b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc767cd0 0x55b4cc761ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29c160c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc767790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29c160c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc767790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:43330/0 (socket says 172.21.15.114:43330) 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29c160c700 1 -- 172.21.15.114:0/3927296882 learned_addr learned my addr 172.21.15.114:0/3927296882 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29c0e0b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc767cd0 0x55b4cc761ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48126/0 (socket says 172.21.15.114:48126) 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29c1e0d700 1 --2- 172.21.15.114:0/3927296882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc762000 0x55b4cc762410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29c160c700 1 -- 172.21.15.114:0/3927296882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc762000 msgr2=0x55b4cc762410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29c160c700 1 --2- 172.21.15.114:0/3927296882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc762000 0x55b4cc762410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29c160c700 1 -- 172.21.15.114:0/3927296882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc767cd0 msgr2=0x55b4cc761ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29c160c700 1 --2- 172.21.15.114:0/3927296882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc767cd0 0x55b4cc761ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29c160c700 1 -- 172.21.15.114:0/3927296882 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4cc5d0950 con 0x55b4cc5e3ec0 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.735+0000 7f29c160c700 1 --2- 172.21.15.114:0/3927296882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc767790 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f29b800e730 tx=0x7f29b8014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5176579f1b0d0b1e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.736+0000 7f29a77fe700 1 -- 172.21.15.114:0/3927296882 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b801e420 con 0x55b4cc5e3ec0 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.736+0000 7f29a77fe700 1 -- 172.21.15.114:0/3927296882 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29b8011e50 con 0x55b4cc5e3ec0 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.736+0000 7f29a77fe700 1 -- 172.21.15.114:0/3927296882 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b80268c0 con 0x55b4cc5e3ec0 2024-05-27T07:43:39.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.737+0000 7f29d2285b80 1 -- 172.21.15.114:0/3927296882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc5e3ec0 msgr2=0x55b4cc767790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:39.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.737+0000 7f29d2285b80 1 --2- 172.21.15.114:0/3927296882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc767790 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f29b800e730 tx=0x7f29b8014000 comp rx=0 tx=0).stop 2024-05-27T07:43:39.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.737+0000 7f29d2285b80 1 -- 172.21.15.114:0/3927296882 shutdown_connections 2024-05-27T07:43:39.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.737+0000 7f29d2285b80 1 --2- 172.21.15.114:0/3927296882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc762000 0x55b4cc762410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.737+0000 7f29d2285b80 1 --2- 172.21.15.114:0/3927296882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc767790 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.737+0000 7f29d2285b80 1 --2- 172.21.15.114:0/3927296882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc767cd0 0x55b4cc761ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.737+0000 7f29d2285b80 1 -- 172.21.15.114:0/3927296882 >> 172.21.15.114:0/3927296882 conn(0x55b4cc63eb70 msgr2=0x55b4cc75c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:39.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.737+0000 7f29d2285b80 1 -- 172.21.15.114:0/3927296882 shutdown_connections 2024-05-27T07:43:39.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.737+0000 7f29d2285b80 1 -- 172.21.15.114:0/3927296882 wait complete. 2024-05-27T07:43:39.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.737+0000 7f29d2285b80 1 -- start start 2024-05-27T07:43:39.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.737+0000 7f29d2285b80 1 Processor -- start 2024-05-27T07:43:39.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29d2285b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc76b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29d2285b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc762000 0x55b4cc76b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29d2285b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 0x55b4cc76be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29d2285b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b4cc587020 con 0x55b4cc762000 2024-05-27T07:43:39.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29d2285b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b4cc622840 con 0x55b4cc5e3ec0 2024-05-27T07:43:39.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29d2285b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b4cc76e7c0 con 0x55b4cc767cd0 2024-05-27T07:43:39.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29d2285b80 10 client.0 _ll_register_callbacks cb 0x55b4cc61b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:39.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29c0e0b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc762000 0x55b4cc76b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29c160c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc76b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29c160c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc76b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39856/0 (socket says 172.21.15.114:39856) 2024-05-27T07:43:39.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29c160c700 1 -- 172.21.15.114:0/3366182438 learned_addr learned my addr 172.21.15.114:0/3366182438 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:39.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29c1e0d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 0x55b4cc76be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29c0e0b700 1 -- 172.21.15.114:0/3366182438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 msgr2=0x55b4cc76be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:39.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29c0e0b700 1 --2- 172.21.15.114:0/3366182438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 0x55b4cc76be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29c0e0b700 1 -- 172.21.15.114:0/3366182438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc5e3ec0 msgr2=0x55b4cc76b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29c0e0b700 1 --2- 172.21.15.114:0/3366182438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc76b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29c0e0b700 1 -- 172.21.15.114:0/3366182438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4cc61f710 con 0x55b4cc762000 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29c160c700 1 --2- 172.21.15.114:0/3366182438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc76b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29c0e0b700 1 --2- 172.21.15.114:0/3366182438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc762000 0x55b4cc76b920 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0dff40 tx=0x7f29ac0a52d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2bf420e0a190696 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.738+0000 7f29a4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29a4ff9700 1 -- 172.21.15.114:0/3366182438 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac0a95e0 con 0x55b4cc762000 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29a4ff9700 1 -- 172.21.15.114:0/3366182438 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f29ac0a60c0 con 0x55b4cc762000 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29d2285b80 1 -- 172.21.15.114:0/3366182438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29b8011e50 con 0x55b4cc762000 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29a4ff9700 1 -- 172.21.15.114:0/3366182438 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f29ac0aa1d0 con 0x55b4cc762000 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29d2285b80 10 client.16197 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29d2285b80 1 -- 172.21.15.114:0/3366182438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b4cc779240 con 0x55b4cc762000 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29a4ff9700 1 -- 172.21.15.114:0/3366182438 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29b8011e50 con 0x55b4cc762000 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29a4ff9700 1 -- 172.21.15.114:0/3366182438 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f29ac0cc0c0 con 0x55b4cc762000 2024-05-27T07:43:39.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29d2285b80 10 client.16197 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:39.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29d2285b80 10 client.16197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:39.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29d2285b80 20 client.16197 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:39.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29d2285b80 1 --2- 172.21.15.114:0/3366182438 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b4cc77b880 0x55b4cc77dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29d2285b80 4 client.16197 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:39.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29d2285b80 1 -- 172.21.15.114:0/3366182438 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4cc72da70 con 0x55b4cc77b880 2024-05-27T07:43:39.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.739+0000 7f29c160c700 1 --2- 172.21.15.114:0/3366182438 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b4cc77b880 0x55b4cc77dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.740+0000 7f29a4ff9700 10 client.16197 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:39.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.740+0000 7f29c160c700 1 --2- 172.21.15.114:0/3366182438 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b4cc77b880 0x55b4cc77dca0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6afcfb5261f6815 server_cookie=ff1e1dc8a7f64eed in_seq=0 out_seq=0 2024-05-27T07:43:39.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.741+0000 7f29a4ff9700 1 -- 172.21.15.114:0/3366182438 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b4cc72da70 con 0x55b4cc77b880 2024-05-27T07:43:39.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.741+0000 7f29a4ff9700 10 client.16197 handle_command_reply: tid=0 2024-05-27T07:43:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.741+0000 7f29d2285b80 1 client.16197 shutdown 2024-05-27T07:43:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.741+0000 7f29d2285b80 1 -- 172.21.15.114:0/3366182438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc762000 msgr2=0x55b4cc76b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.741+0000 7f29d2285b80 1 --2- 172.21.15.114:0/3366182438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc762000 0x55b4cc76b920 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0dff40 tx=0x7f29ac0a52d0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.741+0000 7f29d2285b80 1 -- 172.21.15.114:0/3366182438 shutdown_connections 2024-05-27T07:43:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.741+0000 7f29d2285b80 1 --2- 172.21.15.114:0/3366182438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 0x55b4cc76be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.741+0000 7f29d2285b80 1 --2- 172.21.15.114:0/3366182438 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b4cc77b880 0x55b4cc77dca0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.741+0000 7f29d2285b80 1 --2- 172.21.15.114:0/3366182438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc76b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.741+0000 7f29d2285b80 1 --2- 172.21.15.114:0/3366182438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc762000 0x55b4cc76b920 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.741+0000 7f29d2285b80 1 -- 172.21.15.114:0/3366182438 >> 172.21.15.114:0/3366182438 conn(0x55b4cc63eb70 msgr2=0x55b4cc760ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.741+0000 7f29a4ff9700 0 client.16197 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.742+0000 7f29d2285b80 1 -- 172.21.15.114:0/3366182438 shutdown_connections 2024-05-27T07:43:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.742+0000 7f29d2285b80 1 -- 172.21.15.114:0/3366182438 wait complete. 2024-05-27T07:43:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.742+0000 7f29d2285b80 20 client.16197 trim_cache size 0 max 16384 2024-05-27T07:43:39.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29d2285b80 1 Processor -- start 2024-05-27T07:43:39.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29d2285b80 1 -- start start 2024-05-27T07:43:39.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29d2285b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc761ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29d2285b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc762000 0x55b4cc76a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29d2285b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 0x55b4cc777860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29d2285b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1ae780 con 0x55b4cc5e3ec0 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29d2285b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1b1000 con 0x55b4cc762000 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29d2285b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1aedf0 con 0x55b4cc767cd0 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29c160c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc761ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29c160c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc761ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48148/0 (socket says 172.21.15.114:48148) 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29c160c700 1 -- 172.21.15.114:0/835907878 learned_addr learned my addr 172.21.15.114:0/835907878 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29c1e0d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 0x55b4cc777860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29c0e0b700 1 --2- 172.21.15.114:0/835907878 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc762000 0x55b4cc76a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29c160c700 1 -- 172.21.15.114:0/835907878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 msgr2=0x55b4cc777860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29c160c700 1 --2- 172.21.15.114:0/835907878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 0x55b4cc777860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29c160c700 1 -- 172.21.15.114:0/835907878 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc762000 msgr2=0x55b4cc76a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.764+0000 7f29c160c700 1 --2- 172.21.15.114:0/835907878 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc762000 0x55b4cc76a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.765+0000 7f29c160c700 1 -- 172.21.15.114:0/835907878 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4cc765e10 con 0x55b4cc5e3ec0 2024-05-27T07:43:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.765+0000 7f29c1e0d700 1 --2- 172.21.15.114:0/835907878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 0x55b4cc777860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:39.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.765+0000 7f29c160c700 1 --2- 172.21.15.114:0/835907878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc761ac0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f29b800efd0 tx=0x7f29b800eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a720eaa9a79b4a1e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:39.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.765+0000 7f29a6ffd700 1 -- 172.21.15.114:0/835907878 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b801e7f0 con 0x55b4cc5e3ec0 2024-05-27T07:43:39.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.765+0000 7f29a6ffd700 1 -- 172.21.15.114:0/835907878 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29b8005040 con 0x55b4cc5e3ec0 2024-05-27T07:43:39.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.765+0000 7f29a6ffd700 1 -- 172.21.15.114:0/835907878 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b8004930 con 0x55b4cc5e3ec0 2024-05-27T07:43:39.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.766+0000 7f29d2285b80 1 -- 172.21.15.114:0/835907878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc5e3ec0 msgr2=0x55b4cc761ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:39.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.766+0000 7f29d2285b80 1 --2- 172.21.15.114:0/835907878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc761ac0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f29b800efd0 tx=0x7f29b800eed0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.766+0000 7f29d2285b80 1 -- 172.21.15.114:0/835907878 shutdown_connections 2024-05-27T07:43:39.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.766+0000 7f29d2285b80 1 --2- 172.21.15.114:0/835907878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 0x55b4cc777860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.766+0000 7f29d2285b80 1 --2- 172.21.15.114:0/835907878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc761ac0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.766+0000 7f29d2285b80 1 --2- 172.21.15.114:0/835907878 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc762000 0x55b4cc76a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.766+0000 7f29d2285b80 1 -- 172.21.15.114:0/835907878 >> 172.21.15.114:0/835907878 conn(0x55b4cc63eb70 msgr2=0x55b4cc765700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:39.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.766+0000 7f29d2285b80 1 -- 172.21.15.114:0/835907878 shutdown_connections 2024-05-27T07:43:39.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.766+0000 7f29d2285b80 1 -- 172.21.15.114:0/835907878 wait complete. 2024-05-27T07:43:39.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.766+0000 7f29d2285b80 1 -- start start 2024-05-27T07:43:39.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.766+0000 7f29d2285b80 1 Processor -- start 2024-05-27T07:43:39.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29d2285b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc7e1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29d2285b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc762000 0x55b4cc7e02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29d2285b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 0x55b4cc7e0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29d2285b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1af4c0 con 0x55b4cc762000 2024-05-27T07:43:39.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29d2285b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1b0930 con 0x55b4cc5e3ec0 2024-05-27T07:43:39.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29d2285b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1afb90 con 0x55b4cc767cd0 2024-05-27T07:43:39.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29d2285b80 10 client.0 _ll_register_callbacks cb 0x55b4cc61b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29c0e0b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc762000 0x55b4cc7e02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29c0e0b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc762000 0x55b4cc7e02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48152/0 (socket says 172.21.15.114:48152) 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29c0e0b700 1 -- 172.21.15.114:0/2276613706 learned_addr learned my addr 172.21.15.114:0/2276613706 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29c1e0d700 1 --2- 172.21.15.114:0/2276613706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 0x55b4cc7e0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29c160c700 1 --2- 172.21.15.114:0/2276613706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc7e1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29c0e0b700 1 -- 172.21.15.114:0/2276613706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 msgr2=0x55b4cc7e0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29c0e0b700 1 --2- 172.21.15.114:0/2276613706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 0x55b4cc7e0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29c0e0b700 1 -- 172.21.15.114:0/2276613706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc5e3ec0 msgr2=0x55b4cc7e1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29c0e0b700 1 --2- 172.21.15.114:0/2276613706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc7e1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.767+0000 7f29c0e0b700 1 -- 172.21.15.114:0/2276613706 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4cc7e0d60 con 0x55b4cc762000 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29c0e0b700 1 --2- 172.21.15.114:0/2276613706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc762000 0x55b4cc7e02e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0cc480 tx=0x7f29ac0aa010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b1b22b2b8d3b12 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29a4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29a4ff9700 1 -- 172.21.15.114:0/2276613706 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac06d060 con 0x55b4cc762000 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29a4ff9700 1 -- 172.21.15.114:0/2276613706 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f29ac0a60c0 con 0x55b4cc762000 2024-05-27T07:43:39.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29d2285b80 1 -- 172.21.15.114:0/2276613706 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29b8005040 con 0x55b4cc762000 2024-05-27T07:43:39.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29a4ff9700 1 -- 172.21.15.114:0/2276613706 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f29ac0b2930 con 0x55b4cc762000 2024-05-27T07:43:39.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29d2285b80 10 client.16203 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:39.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29d2285b80 1 -- 172.21.15.114:0/2276613706 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b4cc7736d0 con 0x55b4cc762000 2024-05-27T07:43:39.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29a4ff9700 1 -- 172.21.15.114:0/2276613706 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29b8005040 con 0x55b4cc762000 2024-05-27T07:43:39.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29a4ff9700 1 -- 172.21.15.114:0/2276613706 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f29ac0a5640 con 0x55b4cc762000 2024-05-27T07:43:39.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29d2285b80 10 client.16203 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:39.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29d2285b80 10 client.16203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:39.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29d2285b80 20 client.16203 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:39.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.768+0000 7f29d2285b80 1 --2- 172.21.15.114:0/2276613706 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b4cc773d90 0x55b4cc7741a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:39.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.769+0000 7f29d2285b80 4 client.16203 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:43:39.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.769+0000 7f29d2285b80 1 -- 172.21.15.114:0/2276613706 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b4cc692c60 con 0x55b4cc773d90 2024-05-27T07:43:39.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.769+0000 7f29c160c700 1 --2- 172.21.15.114:0/2276613706 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b4cc773d90 0x55b4cc7741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:39.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.769+0000 7f29a4ff9700 10 client.16203 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:39.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.769+0000 7f29c160c700 1 --2- 172.21.15.114:0/2276613706 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b4cc773d90 0x55b4cc7741a0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38383d1507842409 server_cookie=fbf02deebaf268a6 in_seq=0 out_seq=0 2024-05-27T07:43:39.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.770+0000 7f29a4ff9700 1 -- 172.21.15.114:0/2276613706 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b4cc692c60 con 0x55b4cc773d90 2024-05-27T07:43:39.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.770+0000 7f29a4ff9700 10 client.16203 handle_command_reply: tid=0 2024-05-27T07:43:39.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.770+0000 7f29d2285b80 1 client.16203 shutdown 2024-05-27T07:43:39.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.770+0000 7f29d2285b80 1 -- 172.21.15.114:0/2276613706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc762000 msgr2=0x55b4cc7e02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:39.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.770+0000 7f29d2285b80 1 --2- 172.21.15.114:0/2276613706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc762000 0x55b4cc7e02e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0cc480 tx=0x7f29ac0aa010 comp rx=0 tx=0).stop 2024-05-27T07:43:39.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.770+0000 7f29d2285b80 1 -- 172.21.15.114:0/2276613706 shutdown_connections 2024-05-27T07:43:39.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.770+0000 7f29d2285b80 1 --2- 172.21.15.114:0/2276613706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4cc767cd0 0x55b4cc7e0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.770+0000 7f29d2285b80 1 --2- 172.21.15.114:0/2276613706 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b4cc773d90 0x55b4cc7741a0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.770+0000 7f29d2285b80 1 --2- 172.21.15.114:0/2276613706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4cc5e3ec0 0x55b4cc7e1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.770+0000 7f29d2285b80 1 --2- 172.21.15.114:0/2276613706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b4cc762000 0x55b4cc7e02e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.770+0000 7f29d2285b80 1 -- 172.21.15.114:0/2276613706 >> 172.21.15.114:0/2276613706 conn(0x55b4cc63eb70 msgr2=0x55b4cc765700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:39.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.770+0000 7f29a4ff9700 0 client.16203 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:39.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.771+0000 7f29d2285b80 1 -- 172.21.15.114:0/2276613706 shutdown_connections 2024-05-27T07:43:39.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.771+0000 7f29d2285b80 1 -- 172.21.15.114:0/2276613706 wait complete. 2024-05-27T07:43:39.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.771+0000 7f29d2285b80 20 client.16203 trim_cache size 0 max 16384 2024-05-27T07:43:39.774 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:43:39.774 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:43:39.774 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "adca9979-8a72-43c7-a9ba-c32e9d4ed60a", 2024-05-27T07:43:39.774 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:43:39.774 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:43:39.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.771+0000 7f29c3870700 1 -- 172.21.15.114:0/2093175776 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f29a80b29f0 msgr2=0x7f29a80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:39.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.771+0000 7f29c3870700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f29a80b29f0 0x7f29a80b4f00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f29b8011490 tx=0x7f29b8009260 comp rx=0 tx=0).stop 2024-05-27T07:43:39.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.771+0000 7f29c3870700 1 -- 172.21.15.114:0/2093175776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29bc1042e0 msgr2=0x7f29bc19c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:39.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.771+0000 7f29c3870700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29bc1042e0 0x7f29bc19c0e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f29ac00e9f0 tx=0x7f29ac00ed00 comp rx=0 tx=0).stop 2024-05-27T07:43:39.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.771+0000 7f29c3870700 1 -- 172.21.15.114:0/2093175776 shutdown_connections 2024-05-27T07:43:39.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.771+0000 7f29c3870700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29bc1081f0 0x7f29bc19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.771+0000 7f29c3870700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f29a80b29f0 0x7f29a80b4f00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.771+0000 7f29c3870700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29bc103990 0x7f29bc19bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.771+0000 7f29c3870700 1 --2- 172.21.15.114:0/2093175776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29bc1042e0 0x7f29bc19c0e0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:39.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.771+0000 7f29c3870700 1 -- 172.21.15.114:0/2093175776 >> 172.21.15.114:0/2093175776 conn(0x7f29bc0fe4d0 msgr2=0x7f29bc1075c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:39.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.772+0000 7f29c3870700 1 -- 172.21.15.114:0/2093175776 shutdown_connections 2024-05-27T07:43:39.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:39.772+0000 7f29c3870700 1 -- 172.21.15.114:0/2093175776 wait complete. 2024-05-27T07:43:40.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 291203 14.18 MB/sec warmup 112 sec latency 40.506 ms 2024-05-27T07:43:40.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:40 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:43:40.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:40 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:40.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:40 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:43:40.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:40 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:40.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:43:40.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:40 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:43:40.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:40 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:40.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:40 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:43:40.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:40 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:40.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:40 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:43:40.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:40 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:40.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:40 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:43:40.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:40 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:41.242 INFO:tasks.workunit.client.0.smithi114.stdout: 1 294076 14.18 MB/sec warmup 113 sec latency 22.352 ms 2024-05-27T07:43:41.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:41 smithi114 ceph-mon[32038]: pgmap v125: 129 pgs: 129 active+clean; 51 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 362 op/s 2024-05-27T07:43:41.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:41 smithi163 ceph-mon[36451]: pgmap v125: 129 pgs: 129 active+clean; 51 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 362 op/s 2024-05-27T07:43:41.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:41 smithi186 ceph-mon[36647]: pgmap v125: 129 pgs: 129 active+clean; 51 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 362 op/s 2024-05-27T07:43:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.908+0000 7fcf18243640 1 -- 172.21.15.114:0/935064662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf101021a0 msgr2=0x7fcf101025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.908+0000 7fcf18243640 1 --2- 172.21.15.114:0/935064662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf101021a0 0x7fcf101025b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009900 tx=0x7fcf0002a410 comp rx=0 tx=0).stop 2024-05-27T07:43:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.908+0000 7fcf18243640 1 -- 172.21.15.114:0/935064662 shutdown_connections 2024-05-27T07:43:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.908+0000 7fcf18243640 1 --2- 172.21.15.114:0/935064662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf1010e6a0 0x7fcf10110ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.908+0000 7fcf18243640 1 --2- 172.21.15.114:0/935064662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf101021a0 0x7fcf101025b0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.908+0000 7fcf18243640 1 --2- 172.21.15.114:0/935064662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcf10109cf0 0x7fcf10101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.908+0000 7fcf18243640 1 -- 172.21.15.114:0/935064662 >> 172.21.15.114:0/935064662 conn(0x7fcf100fda10 msgr2=0x7fcf100ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.908+0000 7fcf18243640 1 -- 172.21.15.114:0/935064662 shutdown_connections 2024-05-27T07:43:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.908+0000 7fcf18243640 1 -- 172.21.15.114:0/935064662 wait complete. 2024-05-27T07:43:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.909+0000 7fcf18243640 1 Processor -- start 2024-05-27T07:43:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.909+0000 7fcf18243640 1 -- start start 2024-05-27T07:43:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.909+0000 7fcf18243640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf101021a0 0x7fcf101a0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.909+0000 7fcf18243640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcf10109cf0 0x7fcf101a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.909+0000 7fcf18243640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf1010e6a0 0x7fcf1019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.909+0000 7fcf18243640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcf10113e60 con 0x7fcf101021a0 2024-05-27T07:43:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.909+0000 7fcf18243640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcf10113ce0 con 0x7fcf10109cf0 2024-05-27T07:43:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.909+0000 7fcf18243640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcf10113fe0 con 0x7fcf1010e6a0 2024-05-27T07:43:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf15fdf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf101021a0 0x7fcf101a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf15fdf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf101021a0 0x7fcf101a0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48180/0 (socket says 172.21.15.114:48180) 2024-05-27T07:43:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf15fdf640 1 -- 172.21.15.114:0/3586443532 learned_addr learned my addr 172.21.15.114:0/3586443532 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf157de640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcf10109cf0 0x7fcf101a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf167e0640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf1010e6a0 0x7fcf1019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf15fdf640 1 -- 172.21.15.114:0/3586443532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf1010e6a0 msgr2=0x7fcf1019a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf15fdf640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf1010e6a0 0x7fcf1019a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf15fdf640 1 -- 172.21.15.114:0/3586443532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcf10109cf0 msgr2=0x7fcf101a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf15fdf640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcf10109cf0 0x7fcf101a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf15fdf640 1 -- 172.21.15.114:0/3586443532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1019adf0 con 0x7fcf101021a0 2024-05-27T07:43:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf15fdf640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf101021a0 0x7fcf101a0540 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c00e9f0 tx=0x7fcf0c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf06ffd640 1 -- 172.21.15.114:0/3586443532 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c00ccc0 con 0x7fcf101021a0 2024-05-27T07:43:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf18243640 1 -- 172.21.15.114:0/3586443532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1019b080 con 0x7fcf101021a0 2024-05-27T07:43:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf06ffd640 1 -- 172.21.15.114:0/3586443532 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf0c0035f0 con 0x7fcf101021a0 2024-05-27T07:43:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf06ffd640 1 -- 172.21.15.114:0/3586443532 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c00ce60 con 0x7fcf101021a0 2024-05-27T07:43:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.910+0000 7fcf18243640 1 -- 172.21.15.114:0/3586443532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf101a72e0 con 0x7fcf101021a0 2024-05-27T07:43:41.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.914+0000 7fcf1f1da740 1 Processor -- start 2024-05-27T07:43:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.914+0000 7fcf1f1da740 1 -- start start 2024-05-27T07:43:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.914+0000 7fcf1f1da740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238ef1b0 0x557523c902a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.914+0000 7fcf1f1da740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238c62f0 0x557523c907e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.914+0000 7fcf1f1da740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c90d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.914+0000 7fcf1f1da740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557523c02c30 con 0x5575238ef1b0 2024-05-27T07:43:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.914+0000 7fcf1f1da740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557523bf27d0 con 0x5575238ce4d0 2024-05-27T07:43:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.914+0000 7fcf1f1da740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557523bf6810 con 0x5575238c62f0 2024-05-27T07:43:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.914+0000 7fcf167e0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238ef1b0 0x557523c902a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.914+0000 7fcf167e0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238ef1b0 0x557523c902a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48186/0 (socket says 172.21.15.114:48186) 2024-05-27T07:43:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.914+0000 7fcf167e0640 1 -- 172.21.15.114:0/2976276020 learned_addr learned my addr 172.21.15.114:0/2976276020 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.914+0000 7fcf06ffd640 1 -- 172.21.15.114:0/3586443532 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fcf0c010690 con 0x7fcf101021a0 2024-05-27T07:43:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcf157de640 1 --2- 172.21.15.114:0/2976276020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c90d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcf15fdf640 1 --2- 172.21.15.114:0/2976276020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238c62f0 0x557523c907e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcf167e0640 1 -- 172.21.15.114:0/2976276020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238c62f0 msgr2=0x557523c907e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcf167e0640 1 --2- 172.21.15.114:0/2976276020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238c62f0 0x557523c907e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcf167e0640 1 -- 172.21.15.114:0/2976276020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 msgr2=0x557523c90d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcf167e0640 1 --2- 172.21.15.114:0/2976276020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c90d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcf167e0640 1 -- 172.21.15.114:0/2976276020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557523907210 con 0x5575238ef1b0 2024-05-27T07:43:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcf157de640 1 --2- 172.21.15.114:0/2976276020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c90d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:43:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcf15fdf640 1 --2- 172.21.15.114:0/2976276020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238c62f0 0x557523c907e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:43:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcf167e0640 1 --2- 172.21.15.114:0/2976276020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238ef1b0 0x557523c902a0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcf08009fc0 tx=0x7fcf080118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50028a20ed6c2935 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcefbfff640 1 -- 172.21.15.114:0/2976276020 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08004d20 con 0x5575238ef1b0 2024-05-27T07:43:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcefbfff640 1 -- 172.21.15.114:0/2976276020 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf08004050 con 0x5575238ef1b0 2024-05-27T07:43:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcefbfff640 1 -- 172.21.15.114:0/2976276020 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08005260 con 0x5575238ef1b0 2024-05-27T07:43:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcf06ffd640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcefc0b2950 0x7fcefc0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.915+0000 7fcf15fdf640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcefc0b2950 0x7fcefc0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.916+0000 7fcf06ffd640 1 -- 172.21.15.114:0/3586443532 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fcf0c0e2cc0 con 0x7fcf101021a0 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.916+0000 7fcf15fdf640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcefc0b2950 0x7fcefc0b4e60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0e0110 tx=0x7fcf0c003e70 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.916+0000 7fcf1f1da740 1 -- 172.21.15.114:0/2976276020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238ef1b0 msgr2=0x557523c902a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.916+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/2976276020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238ef1b0 0x557523c902a0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcf08009fc0 tx=0x7fcf080118d0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.916+0000 7fcf1f1da740 1 -- 172.21.15.114:0/2976276020 shutdown_connections 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.916+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/2976276020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c90d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.916+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/2976276020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238c62f0 0x557523c907e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.916+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/2976276020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238ef1b0 0x557523c902a0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.916+0000 7fcf1f1da740 1 -- 172.21.15.114:0/2976276020 >> 172.21.15.114:0/2976276020 conn(0x5575238e3300 msgr2=0x557523c8cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.916+0000 7fcf1f1da740 1 -- 172.21.15.114:0/2976276020 shutdown_connections 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.916+0000 7fcf1f1da740 1 -- 172.21.15.114:0/2976276020 wait complete. 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf1f1da740 1 -- start start 2024-05-27T07:43:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf1f1da740 1 Processor -- start 2024-05-27T07:43:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf1f1da740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238c62f0 0x557523c9eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf1f1da740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c9f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf1f1da740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238ef1b0 0x557523ca4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf1f1da740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5575238d8790 con 0x5575238c62f0 2024-05-27T07:43:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf1f1da740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557523bae890 con 0x5575238ce4d0 2024-05-27T07:43:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf1f1da740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557523ca19c0 con 0x5575238ef1b0 2024-05-27T07:43:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf1f1da740 10 client.0 _ll_register_callbacks cb 0x557523906a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf167e0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238c62f0 0x557523c9eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf157de640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c9f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf157de640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c9f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39912/0 (socket says 172.21.15.114:39912) 2024-05-27T07:43:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf157de640 1 -- 172.21.15.114:0/1025204273 learned_addr learned my addr 172.21.15.114:0/1025204273 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.917+0000 7fcf167e0640 1 --2- 172.21.15.114:0/1025204273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238ef1b0 0x557523ca4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcf167e0640 1 -- 172.21.15.114:0/1025204273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238ef1b0 msgr2=0x557523ca4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcf167e0640 1 --2- 172.21.15.114:0/1025204273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238ef1b0 0x557523ca4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcf167e0640 1 -- 172.21.15.114:0/1025204273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 msgr2=0x557523c9f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcf167e0640 1 --2- 172.21.15.114:0/1025204273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c9f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcf167e0640 1 -- 172.21.15.114:0/1025204273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575239d3620 con 0x5575238c62f0 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcf157de640 1 --2- 172.21.15.114:0/1025204273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c9f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcf167e0640 1 --2- 172.21.15.114:0/1025204273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238c62f0 0x557523c9eeb0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fcf08011dd0 tx=0x7fcf08009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df4906d2e9add11 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcef97fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcef97fa640 1 -- 172.21.15.114:0/1025204273 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0802c060 con 0x5575238c62f0 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcef97fa640 1 -- 172.21.15.114:0/1025204273 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fcf08003c30 con 0x5575238c62f0 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcf1f1da740 1 -- 172.21.15.114:0/1025204273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf08004050 con 0x5575238c62f0 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcef97fa640 1 -- 172.21.15.114:0/1025204273 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf08033050 con 0x5575238c62f0 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcf1f1da740 10 client.16215 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcf1f1da740 1 -- 172.21.15.114:0/1025204273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557523cadc10 con 0x5575238c62f0 2024-05-27T07:43:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcef97fa640 1 -- 172.21.15.114:0/1025204273 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf08004050 con 0x5575238c62f0 2024-05-27T07:43:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.918+0000 7fcef97fa640 1 -- 172.21.15.114:0/1025204273 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fcf08025040 con 0x5575238c62f0 2024-05-27T07:43:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.919+0000 7fcf1f1da740 10 client.16215 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.919+0000 7fcf1f1da740 10 client.16215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.919+0000 7fcf1f1da740 20 client.16215 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.919+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/1025204273 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575238add80 0x557523cb1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.919+0000 7fcf1f1da740 4 client.16215 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.919+0000 7fcf1f1da740 1 -- 172.21.15.114:0/1025204273 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557523c04b80 con 0x5575238add80 2024-05-27T07:43:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.919+0000 7fcf157de640 1 --2- 172.21.15.114:0/1025204273 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575238add80 0x557523cb1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.919+0000 7fcf157de640 1 --2- 172.21.15.114:0/1025204273 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575238add80 0x557523cb1080 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8268038d6be47f10 server_cookie=f59c62498583d3d1 in_seq=0 out_seq=0 2024-05-27T07:43:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.919+0000 7fcef97fa640 10 client.16215 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.920+0000 7fcef97fa640 1 -- 172.21.15.114:0/1025204273 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557523c04b80 con 0x5575238add80 2024-05-27T07:43:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.920+0000 7fcef97fa640 10 client.16215 handle_command_reply: tid=0 2024-05-27T07:43:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.920+0000 7fcf1f1da740 1 client.16215 shutdown 2024-05-27T07:43:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.920+0000 7fcf1f1da740 1 -- 172.21.15.114:0/1025204273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238c62f0 msgr2=0x557523c9eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.920+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/1025204273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238c62f0 0x557523c9eeb0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fcf08011dd0 tx=0x7fcf08009be0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.920+0000 7fcf1f1da740 1 -- 172.21.15.114:0/1025204273 shutdown_connections 2024-05-27T07:43:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.920+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/1025204273 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575238add80 0x557523cb1080 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.920+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/1025204273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238ef1b0 0x557523ca4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.920+0000 7fcef97fa640 0 client.16215 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.920+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/1025204273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c9f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.920+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/1025204273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238c62f0 0x557523c9eeb0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.920+0000 7fcf1f1da740 1 -- 172.21.15.114:0/1025204273 >> 172.21.15.114:0/1025204273 conn(0x5575238e3300 msgr2=0x557523c92ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.921+0000 7fcf1f1da740 1 -- 172.21.15.114:0/1025204273 shutdown_connections 2024-05-27T07:43:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.921+0000 7fcf1f1da740 1 -- 172.21.15.114:0/1025204273 wait complete. 2024-05-27T07:43:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.921+0000 7fcf1f1da740 20 client.16215 trim_cache size 0 max 16384 2024-05-27T07:43:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf1f1da740 1 Processor -- start 2024-05-27T07:43:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf1f1da740 1 -- start start 2024-05-27T07:43:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf1f1da740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238add80 0x557523c92b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf1f1da740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238c62f0 0x557523c98610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf1f1da740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c98b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf1f1da740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcf101ad180 con 0x5575238c62f0 2024-05-27T07:43:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf1f1da740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcf101afa00 con 0x5575238ce4d0 2024-05-27T07:43:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf1f1da740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcf101ad7f0 con 0x5575238add80 2024-05-27T07:43:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf157de640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238c62f0 0x557523c98610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf157de640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238c62f0 0x557523c98610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48198/0 (socket says 172.21.15.114:48198) 2024-05-27T07:43:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf157de640 1 -- 172.21.15.114:0/939342199 learned_addr learned my addr 172.21.15.114:0/939342199 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf167e0640 1 --2- 172.21.15.114:0/939342199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238add80 0x557523c92b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf157de640 1 -- 172.21.15.114:0/939342199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238add80 msgr2=0x557523c92b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf157de640 1 --2- 172.21.15.114:0/939342199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238add80 0x557523c92b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf167e0640 1 --2- 172.21.15.114:0/939342199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c98b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf157de640 1 -- 172.21.15.114:0/939342199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 msgr2=0x557523c98b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf157de640 1 --2- 172.21.15.114:0/939342199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c98b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf157de640 1 -- 172.21.15.114:0/939342199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557523c9f7b0 con 0x5575238c62f0 2024-05-27T07:43:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.937+0000 7fcf157de640 1 --2- 172.21.15.114:0/939342199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238c62f0 0x557523c98610 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcf1006c180 tx=0x7fcf000092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43c926998721ba8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.938+0000 7fcefb7fe640 1 -- 172.21.15.114:0/939342199 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0004a070 con 0x5575238c62f0 2024-05-27T07:43:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.938+0000 7fcefb7fe640 1 -- 172.21.15.114:0/939342199 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf000054f0 con 0x5575238c62f0 2024-05-27T07:43:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.938+0000 7fcefb7fe640 1 -- 172.21.15.114:0/939342199 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf00035040 con 0x5575238c62f0 2024-05-27T07:43:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.939+0000 7fcf1f1da740 1 -- 172.21.15.114:0/939342199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238c62f0 msgr2=0x557523c98610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.939+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/939342199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238c62f0 0x557523c98610 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcf1006c180 tx=0x7fcf000092d0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.939+0000 7fcf1f1da740 1 -- 172.21.15.114:0/939342199 shutdown_connections 2024-05-27T07:43:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.939+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/939342199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238ce4d0 0x557523c98b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.939+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/939342199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238c62f0 0x557523c98610 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.939+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/939342199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238add80 0x557523c92b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.939+0000 7fcf1f1da740 1 -- 172.21.15.114:0/939342199 >> 172.21.15.114:0/939342199 conn(0x5575238e3300 msgr2=0x557523ca0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.939+0000 7fcf1f1da740 1 -- 172.21.15.114:0/939342199 shutdown_connections 2024-05-27T07:43:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.939+0000 7fcf1f1da740 1 -- 172.21.15.114:0/939342199 wait complete. 2024-05-27T07:43:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.939+0000 7fcf1f1da740 1 -- start start 2024-05-27T07:43:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.939+0000 7fcf1f1da740 1 Processor -- start 2024-05-27T07:43:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf1f1da740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238add80 0x557523ca5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf1f1da740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238c62f0 0x557523ca6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf1f1da740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238ce4d0 0x557523ca66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf1f1da740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcf101adec0 con 0x5575238ce4d0 2024-05-27T07:43:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf1f1da740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcf101af330 con 0x5575238c62f0 2024-05-27T07:43:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf1f1da740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcf101ae590 con 0x5575238add80 2024-05-27T07:43:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf1f1da740 10 client.0 _ll_register_callbacks cb 0x7fcf08009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf167e0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238ce4d0 0x557523ca66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf167e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238add80 0x557523ca5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf157de640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238c62f0 0x557523ca6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf167e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238add80 0x557523ca5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:43450/0 (socket says 172.21.15.114:43450) 2024-05-27T07:43:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf157de640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238c62f0 0x557523ca6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39930/0 (socket says 172.21.15.114:39930) 2024-05-27T07:43:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf167e0640 1 -- 172.21.15.114:0/2186631710 learned_addr learned my addr 172.21.15.114:0/2186631710 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf157de640 1 -- 172.21.15.114:0/2186631710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238add80 msgr2=0x557523ca5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf157de640 1 --2- 172.21.15.114:0/2186631710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238add80 0x557523ca5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf157de640 1 -- 172.21.15.114:0/2186631710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238ce4d0 msgr2=0x557523ca66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf157de640 1 --2- 172.21.15.114:0/2186631710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238ce4d0 0x557523ca66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf157de640 1 -- 172.21.15.114:0/2186631710 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557523d17540 con 0x5575238c62f0 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.940+0000 7fcf167e0640 1 --2- 172.21.15.114:0/2186631710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238add80 0x557523ca5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.941+0000 7fcf157de640 1 --2- 172.21.15.114:0/2186631710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238c62f0 0x557523ca6170 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcf00048040 tx=0x7fcf00009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c1941092b961eba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.941+0000 7fcef97fa640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.941+0000 7fcef97fa640 1 -- 172.21.15.114:0/2186631710 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0004a040 con 0x5575238c62f0 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.941+0000 7fcf1f1da740 1 -- 172.21.15.114:0/2186631710 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf000054f0 con 0x5575238c62f0 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.941+0000 7fcef97fa640 1 -- 172.21.15.114:0/2186631710 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fcf00005240 con 0x5575238c62f0 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.941+0000 7fcef97fa640 1 -- 172.21.15.114:0/2186631710 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf00035040 con 0x5575238c62f0 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.941+0000 7fcf1f1da740 10 client.25748 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.941+0000 7fcf1f1da740 1 -- 172.21.15.114:0/2186631710 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557523d186f0 con 0x5575238c62f0 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.941+0000 7fcef97fa640 1 -- 172.21.15.114:0/2186631710 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf000054f0 con 0x5575238c62f0 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.941+0000 7fcef97fa640 1 -- 172.21.15.114:0/2186631710 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fcf000035f0 con 0x5575238c62f0 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.941+0000 7fcf1f1da740 10 client.25748 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.941+0000 7fcf1f1da740 10 client.25748 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.941+0000 7fcf1f1da740 20 client.25748 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.942+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/2186631710 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575238ef1b0 0x557523d1ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.942+0000 7fcf1f1da740 4 client.25748 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:43:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.942+0000 7fcf1f1da740 1 -- 172.21.15.114:0/2186631710 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557523bfe360 con 0x5575238ef1b0 2024-05-27T07:43:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.942+0000 7fcf167e0640 1 --2- 172.21.15.114:0/2186631710 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575238ef1b0 0x557523d1ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.942+0000 7fcef97fa640 10 client.25748 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:41.942+0000 7fcf167e0640 1 --2- 172.21.15.114:0/2186631710 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575238ef1b0 0x557523d1ae30 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36fc67c40e10065b server_cookie=a3ab23ede2dda039 in_seq=0 out_seq=0 2024-05-27T07:43:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcef97fa640 1 -- 172.21.15.114:0/2186631710 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x557523bfe360 con 0x5575238ef1b0 2024-05-27T07:43:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcef97fa640 10 client.25748 handle_command_reply: tid=0 2024-05-27T07:43:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcf1f1da740 1 client.25748 shutdown 2024-05-27T07:43:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcf1f1da740 1 -- 172.21.15.114:0/2186631710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238c62f0 msgr2=0x557523ca6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/2186631710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238c62f0 0x557523ca6170 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcf00048040 tx=0x7fcf00009640 comp rx=0 tx=0).stop 2024-05-27T07:43:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcf1f1da740 1 -- 172.21.15.114:0/2186631710 shutdown_connections 2024-05-27T07:43:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/2186631710 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575238ef1b0 0x557523d1ae30 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/2186631710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575238ce4d0 0x557523ca66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/2186631710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575238c62f0 0x557523ca6170 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcf1f1da740 1 --2- 172.21.15.114:0/2186631710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575238add80 0x557523ca5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcef97fa640 0 client.25748 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcf1f1da740 1 -- 172.21.15.114:0/2186631710 >> 172.21.15.114:0/2186631710 conn(0x5575238e3300 msgr2=0x557523c90160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcf1f1da740 1 -- 172.21.15.114:0/2186631710 shutdown_connections 2024-05-27T07:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcf1f1da740 1 -- 172.21.15.114:0/2186631710 wait complete. 2024-05-27T07:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.244+0000 7fcf1f1da740 20 client.25748 trim_cache size 0 max 16384 2024-05-27T07:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:43:42.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 295090 14.11 MB/sec warmup 114 sec latency 113.073 ms 2024-05-27T07:43:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.245+0000 7fcf18243640 1 -- 172.21.15.114:0/3586443532 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcefc0b2950 msgr2=0x7fcefc0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.245+0000 7fcf18243640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcefc0b2950 0x7fcefc0b4e60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0e0110 tx=0x7fcf0c003e70 comp rx=0 tx=0).stop 2024-05-27T07:43:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.245+0000 7fcf18243640 1 -- 172.21.15.114:0/3586443532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf101021a0 msgr2=0x7fcf101a0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.245+0000 7fcf18243640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf101021a0 0x7fcf101a0540 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c00e9f0 tx=0x7fcf0c00eeb0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.245+0000 7fcf18243640 1 -- 172.21.15.114:0/3586443532 shutdown_connections 2024-05-27T07:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.245+0000 7fcf18243640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcefc0b2950 0x7fcefc0b4e60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.245+0000 7fcf18243640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf1010e6a0 0x7fcf1019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.245+0000 7fcf18243640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcf10109cf0 0x7fcf101a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.245+0000 7fcf18243640 1 --2- 172.21.15.114:0/3586443532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf101021a0 0x7fcf101a0540 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.245+0000 7fcf18243640 1 -- 172.21.15.114:0/3586443532 >> 172.21.15.114:0/3586443532 conn(0x7fcf100fda10 msgr2=0x7fcf100fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.245+0000 7fcf18243640 1 -- 172.21.15.114:0/3586443532 shutdown_connections 2024-05-27T07:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:42.245+0000 7fcf18243640 1 -- 172.21.15.114:0/3586443532 wait complete. 2024-05-27T07:43:42.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:42 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:43:42.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.710+0000 7f4005ae8700 1 -- 172.21.15.114:0/1905322969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4000108f10 msgr2=0x7f4000109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:42.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.710+0000 7f4005ae8700 1 --2- 172.21.15.114:0/1905322969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4000108f10 0x7f4000109320 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40099c0 tx=0x7f3ff4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.710+0000 7f4005ae8700 1 -- 172.21.15.114:0/1905322969 shutdown_connections 2024-05-27T07:43:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.710+0000 7f4005ae8700 1 --2- 172.21.15.114:0/1905322969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4000109860 0x7f40001111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.710+0000 7f4005ae8700 1 --2- 172.21.15.114:0/1905322969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40001045e0 0x7f40001049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.710+0000 7f4005ae8700 1 --2- 172.21.15.114:0/1905322969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4000108f10 0x7f4000109320 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.710+0000 7f4005ae8700 1 -- 172.21.15.114:0/1905322969 >> 172.21.15.114:0/1905322969 conn(0x7f4000100030 msgr2=0x7f4000102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.710+0000 7f4005ae8700 1 -- 172.21.15.114:0/1905322969 shutdown_connections 2024-05-27T07:43:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.710+0000 7f4005ae8700 1 -- 172.21.15.114:0/1905322969 wait complete. 2024-05-27T07:43:42.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.711+0000 7f4005ae8700 1 Processor -- start 2024-05-27T07:43:42.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.711+0000 7f4005ae8700 1 -- start start 2024-05-27T07:43:42.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f4005ae8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40001045e0 0x7f400019aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f4005ae8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4000108f10 0x7f400019b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f4005ae8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4000109860 0x7f400019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f4005ae8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4000114470 con 0x7f4000109860 2024-05-27T07:43:42.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f4005ae8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f40001142f0 con 0x7f4000108f10 2024-05-27T07:43:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f4005ae8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f40001145f0 con 0x7f40001045e0 2024-05-27T07:43:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f3fff7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40001045e0 0x7f400019aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f3fff7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40001045e0 0x7f400019aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:43460/0 (socket says 172.21.15.114:43460) 2024-05-27T07:43:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f3fff7fe700 1 -- 172.21.15.114:0/3409355898 learned_addr learned my addr 172.21.15.114:0/3409355898 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f3ffeffd700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4000108f10 0x7f400019b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f3ffffff700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4000109860 0x7f400019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f3fff7fe700 1 -- 172.21.15.114:0/3409355898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4000108f10 msgr2=0x7f400019b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f3fff7fe700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4000108f10 0x7f400019b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f3fff7fe700 1 -- 172.21.15.114:0/3409355898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4000109860 msgr2=0x7f400019f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f3fff7fe700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4000109860 0x7f400019f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f3fff7fe700 1 -- 172.21.15.114:0/3409355898 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400019fdb0 con 0x7f40001045e0 2024-05-27T07:43:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f3ffffff700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4000109860 0x7f400019f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.712+0000 7f3fff7fe700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40001045e0 0x7f400019aee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000e9a0 tx=0x7f3ff000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.713+0000 7f3ffcff9700 1 -- 172.21.15.114:0/3409355898 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff000ca10 con 0x7f40001045e0 2024-05-27T07:43:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.713+0000 7f4005ae8700 1 -- 172.21.15.114:0/3409355898 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40001a00a0 con 0x7f40001045e0 2024-05-27T07:43:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.713+0000 7f3ffcff9700 1 -- 172.21.15.114:0/3409355898 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff000cbb0 con 0x7f40001045e0 2024-05-27T07:43:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.713+0000 7f3ffcff9700 1 -- 172.21.15.114:0/3409355898 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0018680 con 0x7f40001045e0 2024-05-27T07:43:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.713+0000 7f4005ae8700 1 -- 172.21.15.114:0/3409355898 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40000759e0 con 0x7f40001045e0 2024-05-27T07:43:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.716+0000 7f3ffcff9700 1 -- 172.21.15.114:0/3409355898 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f3ff0018820 con 0x7f40001045e0 2024-05-27T07:43:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.716+0000 7f3ffcff9700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3fe80b29f0 0x7f3fe80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f40144fdb80 1 Processor -- start 2024-05-27T07:43:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f3ffeffd700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3fe80b29f0 0x7f3fe80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f3ffcff9700 1 -- 172.21.15.114:0/3409355898 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3ff00e2c00 con 0x7f40001045e0 2024-05-27T07:43:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f40144fdb80 1 -- start start 2024-05-27T07:43:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f40144fdb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 0x55ac12adfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f40144fdb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12ad9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f40144fdb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ada530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f40144fdb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ac12a07d70 con 0x55ac12ae0070 2024-05-27T07:43:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f40144fdb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ac129f7910 con 0x55ac1295ae40 2024-05-27T07:43:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f40144fdb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ac129fb950 con 0x55ac12ada120 2024-05-27T07:43:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f3ffeffd700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3fe80b29f0 0x7f3fe80b4f00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4000c00 tx=0x7f3ff401a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f3ffeffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12ad9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f3ffeffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12ad9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48240/0 (socket says 172.21.15.114:48240) 2024-05-27T07:43:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f3ffeffd700 1 -- 172.21.15.114:0/1846331983 learned_addr learned my addr 172.21.15.114:0/1846331983 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f3ffffff700 1 --2- 172.21.15.114:0/1846331983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ada530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.717+0000 7f3fff7fe700 1 --2- 172.21.15.114:0/1846331983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 0x55ac12adfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.718+0000 7f3ffeffd700 1 -- 172.21.15.114:0/1846331983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 msgr2=0x55ac12ada530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.718+0000 7f3ffeffd700 1 --2- 172.21.15.114:0/1846331983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ada530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.718+0000 7f3ffeffd700 1 -- 172.21.15.114:0/1846331983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 msgr2=0x55ac12adfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.718+0000 7f3ffeffd700 1 --2- 172.21.15.114:0/1846331983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 0x55ac12adfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.718+0000 7f3ffeffd700 1 -- 172.21.15.114:0/1846331983 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac129a0fb0 con 0x55ac12ae0070 2024-05-27T07:43:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.718+0000 7f3fff7fe700 1 --2- 172.21.15.114:0/1846331983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 0x55ac12adfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:43:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.718+0000 7f3ffeffd700 1 --2- 172.21.15.114:0/1846331983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12ad9be0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40099c0 tx=0x7f3ff4009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c198ce2da7b038b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.718+0000 7f3fe5ffb700 1 -- 172.21.15.114:0/1846331983 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4003e20 con 0x55ac12ae0070 2024-05-27T07:43:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.718+0000 7f3fe5ffb700 1 -- 172.21.15.114:0/1846331983 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff402b070 con 0x55ac12ae0070 2024-05-27T07:43:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.718+0000 7f3fe5ffb700 1 -- 172.21.15.114:0/1846331983 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff402faa0 con 0x55ac12ae0070 2024-05-27T07:43:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.719+0000 7f40144fdb80 1 -- 172.21.15.114:0/1846331983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 msgr2=0x55ac12ad9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.719+0000 7f40144fdb80 1 --2- 172.21.15.114:0/1846331983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12ad9be0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40099c0 tx=0x7f3ff4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.719+0000 7f40144fdb80 1 -- 172.21.15.114:0/1846331983 shutdown_connections 2024-05-27T07:43:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.719+0000 7f40144fdb80 1 --2- 172.21.15.114:0/1846331983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ada530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.719+0000 7f40144fdb80 1 --2- 172.21.15.114:0/1846331983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 0x55ac12adfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.719+0000 7f40144fdb80 1 --2- 172.21.15.114:0/1846331983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12ad9be0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.719+0000 7f40144fdb80 1 -- 172.21.15.114:0/1846331983 >> 172.21.15.114:0/1846331983 conn(0x55ac1295d770 msgr2=0x55ac12ad4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.719+0000 7f40144fdb80 1 -- 172.21.15.114:0/1846331983 shutdown_connections 2024-05-27T07:43:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.719+0000 7f40144fdb80 1 -- 172.21.15.114:0/1846331983 wait complete. 2024-05-27T07:43:42.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.719+0000 7f40144fdb80 1 -- start start 2024-05-27T07:43:42.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.720+0000 7f40144fdb80 1 Processor -- start 2024-05-27T07:43:42.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.720+0000 7f40144fdb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 0x55ac12ae38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.720+0000 7f40144fdb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ae3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.720+0000 7f40144fdb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12ae4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.720+0000 7f40144fdb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ac128fdfa0 con 0x55ac12ae0070 2024-05-27T07:43:42.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.720+0000 7f40144fdb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ac1295fda0 con 0x55ac1295ae40 2024-05-27T07:43:42.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.720+0000 7f40144fdb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ac12ae6b00 con 0x55ac12ada120 2024-05-27T07:43:42.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.720+0000 7f40144fdb80 10 client.0 _ll_register_callbacks cb 0x55ac1298e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:42.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.720+0000 7f3ffeffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ae3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.720+0000 7f3ffeffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ae3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:43474/0 (socket says 172.21.15.114:43474) 2024-05-27T07:43:42.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.720+0000 7f3ffeffd700 1 -- 172.21.15.114:0/5729713 learned_addr learned my addr 172.21.15.114:0/5729713 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:42.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.720+0000 7f3fff7fe700 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 0x55ac12ae38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.721+0000 7f3ffffff700 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12ae4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.721+0000 7f3ffeffd700 1 -- 172.21.15.114:0/5729713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 msgr2=0x55ac12ae38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:42.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.721+0000 7f3ffeffd700 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 0x55ac12ae38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.721+0000 7f3ffeffd700 1 -- 172.21.15.114:0/5729713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 msgr2=0x55ac12ae4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:42.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.721+0000 7f3ffeffd700 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12ae4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.721+0000 7f3ffeffd700 1 -- 172.21.15.114:0/5729713 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac129478d0 con 0x55ac12ada120 2024-05-27T07:43:42.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.721+0000 7f3ffffff700 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12ae4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:43:42.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.721+0000 7f3ffeffd700 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ae3de0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4005080 tx=0x7f3ff402fa20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a275f19ed0bea726 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:42.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.721+0000 7f3fdeffd700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:43:42.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.721+0000 7f3fdeffd700 1 -- 172.21.15.114:0/5729713 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff402f410 con 0x55ac12ada120 2024-05-27T07:43:42.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.721+0000 7f40144fdb80 1 -- 172.21.15.114:0/5729713 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ff402b070 con 0x55ac12ada120 2024-05-27T07:43:42.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.721+0000 7f3fdeffd700 1 -- 172.21.15.114:0/5729713 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3ff400b9a0 con 0x55ac12ada120 2024-05-27T07:43:42.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.721+0000 7f3fdeffd700 1 -- 172.21.15.114:0/5729713 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3ff400bba0 con 0x55ac12ada120 2024-05-27T07:43:42.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.722+0000 7f40144fdb80 10 client.25825 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:42.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.722+0000 7f40144fdb80 1 -- 172.21.15.114:0/5729713 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ac129b1760 con 0x55ac12ada120 2024-05-27T07:43:42.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.722+0000 7f3fdeffd700 1 -- 172.21.15.114:0/5729713 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff402b070 con 0x55ac12ada120 2024-05-27T07:43:42.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.722+0000 7f3fdeffd700 1 -- 172.21.15.114:0/5729713 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f3ff4021070 con 0x55ac12ada120 2024-05-27T07:43:42.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.722+0000 7f40144fdb80 10 client.25825 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:42.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.722+0000 7f40144fdb80 10 client.25825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:42.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.722+0000 7f40144fdb80 20 client.25825 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:42.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.722+0000 7f40144fdb80 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac12af3920 0x55ac12af5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.722+0000 7f40144fdb80 4 client.25825 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:42.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.722+0000 7f40144fdb80 1 -- 172.21.15.114:0/5729713 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac12a09cc0 con 0x55ac12af3920 2024-05-27T07:43:42.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.722+0000 7f3fff7fe700 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac12af3920 0x55ac12af5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.723+0000 7f3fdeffd700 10 client.25825 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:42.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.723+0000 7f3fff7fe700 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac12af3920 0x55ac12af5d40 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99736bb86b7ef5c7 server_cookie=75857058c111c585 in_seq=0 out_seq=0 2024-05-27T07:43:42.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.725+0000 7f3fdeffd700 1 -- 172.21.15.114:0/5729713 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ac12a09cc0 con 0x55ac12af3920 2024-05-27T07:43:42.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.725+0000 7f3fdeffd700 10 client.25825 handle_command_reply: tid=0 2024-05-27T07:43:42.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.725+0000 7f40144fdb80 1 client.25825 shutdown 2024-05-27T07:43:42.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.725+0000 7f40144fdb80 1 -- 172.21.15.114:0/5729713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 msgr2=0x55ac12ae3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:42.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.726+0000 7f40144fdb80 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ae3de0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4005080 tx=0x7f3ff402fa20 comp rx=0 tx=0).stop 2024-05-27T07:43:42.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.726+0000 7f40144fdb80 1 -- 172.21.15.114:0/5729713 shutdown_connections 2024-05-27T07:43:42.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.726+0000 7f40144fdb80 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12ae4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.726+0000 7f40144fdb80 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac12af3920 0x55ac12af5d40 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.726+0000 7f40144fdb80 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 0x55ac12ae38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.726+0000 7f3fdeffd700 0 client.25825 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:42.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.726+0000 7f40144fdb80 1 --2- 172.21.15.114:0/5729713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ae3de0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.726+0000 7f40144fdb80 1 -- 172.21.15.114:0/5729713 >> 172.21.15.114:0/5729713 conn(0x55ac1295d770 msgr2=0x55ac12ad8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:42.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.726+0000 7f40144fdb80 1 -- 172.21.15.114:0/5729713 shutdown_connections 2024-05-27T07:43:42.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.726+0000 7f40144fdb80 1 -- 172.21.15.114:0/5729713 wait complete. 2024-05-27T07:43:42.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.726+0000 7f40144fdb80 20 client.25825 trim_cache size 0 max 16384 2024-05-27T07:43:42.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:42 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:43:42.731 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:42 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:43:42.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.748+0000 7f40144fdb80 1 Processor -- start 2024-05-27T07:43:42.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.748+0000 7f40144fdb80 1 -- start start 2024-05-27T07:43:42.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.748+0000 7f40144fdb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 0x55ac12ad9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.748+0000 7f40144fdb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ae5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.748+0000 7f40144fdb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12aefda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.748+0000 7f40144fdb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f40001b0770 con 0x55ac12ae0070 2024-05-27T07:43:42.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.748+0000 7f40144fdb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f40001b2ff0 con 0x55ac1295ae40 2024-05-27T07:43:42.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.748+0000 7f40144fdb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f40001b0de0 con 0x55ac12ada120 2024-05-27T07:43:42.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.748+0000 7f3ffeffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ae5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.748+0000 7f3ffffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12aefda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.748+0000 7f3ffeffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ae5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:43486/0 (socket says 172.21.15.114:43486) 2024-05-27T07:43:42.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.748+0000 7f3ffeffd700 1 -- 172.21.15.114:0/2274132704 learned_addr learned my addr 172.21.15.114:0/2274132704 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:42.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.749+0000 7f3fff7fe700 1 --2- 172.21.15.114:0/2274132704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 0x55ac12ad9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.749+0000 7f3ffffff700 1 -- 172.21.15.114:0/2274132704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 msgr2=0x55ac12ae5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:42.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.749+0000 7f3ffffff700 1 --2- 172.21.15.114:0/2274132704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ae5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.749+0000 7f3ffffff700 1 -- 172.21.15.114:0/2274132704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 msgr2=0x55ac12ad9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:42.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.749+0000 7f3ffffff700 1 --2- 172.21.15.114:0/2274132704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 0x55ac12ad9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.749+0000 7f3ffffff700 1 -- 172.21.15.114:0/2274132704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac12ae4960 con 0x55ac12ae0070 2024-05-27T07:43:42.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.749+0000 7f3ffeffd700 1 --2- 172.21.15.114:0/2274132704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ae5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:42.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.749+0000 7f3ffffff700 1 --2- 172.21.15.114:0/2274132704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12aefda0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8008770 tx=0x7f3ff80078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22ad8976d3becd5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:42.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.749+0000 7f3fe57fa700 1 -- 172.21.15.114:0/2274132704 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff801b030 con 0x55ac12ae0070 2024-05-27T07:43:42.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.749+0000 7f3fe57fa700 1 -- 172.21.15.114:0/2274132704 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff8017640 con 0x55ac12ae0070 2024-05-27T07:43:42.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.749+0000 7f3fe57fa700 1 -- 172.21.15.114:0/2274132704 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8004260 con 0x55ac12ae0070 2024-05-27T07:43:42.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.750+0000 7f40144fdb80 1 -- 172.21.15.114:0/2274132704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 msgr2=0x55ac12aefda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:42.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.750+0000 7f40144fdb80 1 --2- 172.21.15.114:0/2274132704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12aefda0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8008770 tx=0x7f3ff80078d0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.750+0000 7f40144fdb80 1 -- 172.21.15.114:0/2274132704 shutdown_connections 2024-05-27T07:43:42.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.750+0000 7f40144fdb80 1 --2- 172.21.15.114:0/2274132704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac12ae0070 0x55ac12aefda0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.750+0000 7f40144fdb80 1 --2- 172.21.15.114:0/2274132704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac1295ae40 0x55ac12ad9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.750+0000 7f40144fdb80 1 --2- 172.21.15.114:0/2274132704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12ae5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.750+0000 7f40144fdb80 1 -- 172.21.15.114:0/2274132704 >> 172.21.15.114:0/2274132704 conn(0x55ac1295d770 msgr2=0x55ac12ace4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:42.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.750+0000 7f40144fdb80 1 -- 172.21.15.114:0/2274132704 shutdown_connections 2024-05-27T07:43:42.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.750+0000 7f40144fdb80 1 -- 172.21.15.114:0/2274132704 wait complete. 2024-05-27T07:43:42.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.751+0000 7f40144fdb80 1 -- start start 2024-05-27T07:43:42.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.751+0000 7f40144fdb80 1 Processor -- start 2024-05-27T07:43:42.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.751+0000 7f40144fdb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac1295ae40 0x55ac12af6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.751+0000 7f40144fdb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12af6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.751+0000 7f40144fdb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac12ae0070 0x55ac12af7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.751+0000 7f40144fdb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f40001b14b0 con 0x55ac1295ae40 2024-05-27T07:43:42.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.751+0000 7f40144fdb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f40001b2920 con 0x55ac12ae0070 2024-05-27T07:43:42.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.751+0000 7f40144fdb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f40001b1b80 con 0x55ac12ada120 2024-05-27T07:43:42.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.751+0000 7f40144fdb80 10 client.0 _ll_register_callbacks cb 0x55ac1298e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:42.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.751+0000 7f3ffeffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12af6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.751+0000 7f3fff7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac1295ae40 0x55ac12af6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.751+0000 7f3ffeffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12af6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:43494/0 (socket says 172.21.15.114:43494) 2024-05-27T07:43:42.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.751+0000 7f3ffeffd700 1 -- 172.21.15.114:0/495950348 learned_addr learned my addr 172.21.15.114:0/495950348 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:42.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f3ffffff700 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac12ae0070 0x55ac12af7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f3fff7fe700 1 -- 172.21.15.114:0/495950348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 msgr2=0x55ac12af6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:42.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f3fff7fe700 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12af6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f3fff7fe700 1 -- 172.21.15.114:0/495950348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac12ae0070 msgr2=0x55ac12af7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:42.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f3fff7fe700 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac12ae0070 0x55ac12af7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f3fff7fe700 1 -- 172.21.15.114:0/495950348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac12b5d190 con 0x55ac1295ae40 2024-05-27T07:43:42.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f3ffffff700 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac12ae0070 0x55ac12af7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:42.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f3ffeffd700 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12af6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:42.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f3fff7fe700 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac1295ae40 0x55ac12af6aa0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3ff00dffa0 tx=0x7f3ff00e2040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd80dd9d78ab1bb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:42.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f3fdeffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:42.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f3fdeffd700 1 -- 172.21.15.114:0/495950348 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff00aae60 con 0x55ac1295ae40 2024-05-27T07:43:42.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f3fdeffd700 1 -- 172.21.15.114:0/495950348 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3ff00a9d90 con 0x55ac1295ae40 2024-05-27T07:43:42.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f40144fdb80 1 -- 172.21.15.114:0/495950348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ff8017640 con 0x55ac1295ae40 2024-05-27T07:43:42.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f3fdeffd700 1 -- 172.21.15.114:0/495950348 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3ff00a9b30 con 0x55ac1295ae40 2024-05-27T07:43:42.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f40144fdb80 10 client.16239 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:42.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.752+0000 7f40144fdb80 1 -- 172.21.15.114:0/495950348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ac12b66bc0 con 0x55ac1295ae40 2024-05-27T07:43:42.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.753+0000 7f3fdeffd700 1 -- 172.21.15.114:0/495950348 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff8017640 con 0x55ac1295ae40 2024-05-27T07:43:42.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.753+0000 7f3fdeffd700 1 -- 172.21.15.114:0/495950348 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f3ff00b2500 con 0x55ac1295ae40 2024-05-27T07:43:42.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.753+0000 7f40144fdb80 10 client.16239 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:42.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.753+0000 7f40144fdb80 10 client.16239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:42.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.753+0000 7f40144fdb80 20 client.16239 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:42.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.753+0000 7f40144fdb80 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac12af3920 0x55ac12b69280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:42.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.753+0000 7f40144fdb80 4 client.16239 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:43:42.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.753+0000 7f40144fdb80 1 -- 172.21.15.114:0/495950348 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ac12ad9f70 con 0x55ac12af3920 2024-05-27T07:43:42.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.753+0000 7f3ffeffd700 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac12af3920 0x55ac12b69280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:42.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.753+0000 7f3fdeffd700 10 client.16239 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:42.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.754+0000 7f3ffeffd700 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac12af3920 0x55ac12b69280 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed679460a0e311b server_cookie=7fceef4a56461661 in_seq=0 out_seq=0 2024-05-27T07:43:42.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f3fdeffd700 1 -- 172.21.15.114:0/495950348 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ac12ad9f70 con 0x55ac12af3920 2024-05-27T07:43:42.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f3fdeffd700 10 client.16239 handle_command_reply: tid=0 2024-05-27T07:43:42.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f40144fdb80 1 client.16239 shutdown 2024-05-27T07:43:42.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f40144fdb80 1 -- 172.21.15.114:0/495950348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac1295ae40 msgr2=0x55ac12af6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:42.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f40144fdb80 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac1295ae40 0x55ac12af6aa0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3ff00dffa0 tx=0x7f3ff00e2040 comp rx=0 tx=0).stop 2024-05-27T07:43:42.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f40144fdb80 1 -- 172.21.15.114:0/495950348 shutdown_connections 2024-05-27T07:43:42.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f40144fdb80 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac12af3920 0x55ac12b69280 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f40144fdb80 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac12ae0070 0x55ac12af7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f40144fdb80 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac1295ae40 0x55ac12af6aa0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f40144fdb80 1 --2- 172.21.15.114:0/495950348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac12ada120 0x55ac12af6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f3fdeffd700 0 client.16239 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:42.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f40144fdb80 1 -- 172.21.15.114:0/495950348 >> 172.21.15.114:0/495950348 conn(0x55ac1295d770 msgr2=0x55ac12addc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:42.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f40144fdb80 1 -- 172.21.15.114:0/495950348 shutdown_connections 2024-05-27T07:43:42.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f40144fdb80 1 -- 172.21.15.114:0/495950348 wait complete. 2024-05-27T07:43:42.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.755+0000 7f40144fdb80 20 client.16239 trim_cache size 0 max 16384 2024-05-27T07:43:42.759 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:43:42.759 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:43:42.759 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:43:42.759 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:43:42.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.756+0000 7f4005ae8700 1 -- 172.21.15.114:0/3409355898 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3fe80b29f0 msgr2=0x7f3fe80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:42.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.756+0000 7f4005ae8700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3fe80b29f0 0x7f3fe80b4f00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4000c00 tx=0x7f3ff401a040 comp rx=0 tx=0).stop 2024-05-27T07:43:42.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.756+0000 7f4005ae8700 1 -- 172.21.15.114:0/3409355898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40001045e0 msgr2=0x7f400019aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:42.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.756+0000 7f4005ae8700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40001045e0 0x7f400019aee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000e9a0 tx=0x7f3ff000ecb0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.756+0000 7f4005ae8700 1 -- 172.21.15.114:0/3409355898 shutdown_connections 2024-05-27T07:43:42.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.756+0000 7f4005ae8700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4000109860 0x7f400019f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.756+0000 7f4005ae8700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40001045e0 0x7f400019aee0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.756+0000 7f4005ae8700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3fe80b29f0 0x7f3fe80b4f00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.756+0000 7f4005ae8700 1 --2- 172.21.15.114:0/3409355898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4000108f10 0x7f400019b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:42.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.756+0000 7f4005ae8700 1 -- 172.21.15.114:0/3409355898 >> 172.21.15.114:0/3409355898 conn(0x7f4000100030 msgr2=0x7f400010a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:42.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.756+0000 7f4005ae8700 1 -- 172.21.15.114:0/3409355898 shutdown_connections 2024-05-27T07:43:42.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:42.756+0000 7f4005ae8700 1 -- 172.21.15.114:0/3409355898 wait complete. 2024-05-27T07:43:43.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 298071 14.14 MB/sec warmup 115 sec latency 22.648 ms 2024-05-27T07:43:43.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:43 smithi114 ceph-mon[32038]: pgmap v126: 129 pgs: 129 active+clean; 52 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 303 op/s 2024-05-27T07:43:43.552 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:43:43.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:43:43.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:43 smithi163 ceph-mon[36451]: pgmap v126: 129 pgs: 129 active+clean; 52 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 303 op/s 2024-05-27T07:43:43.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:43 smithi186 ceph-mon[36647]: pgmap v126: 129 pgs: 129 active+clean; 52 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 303 op/s 2024-05-27T07:43:43.814 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:44.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 300398 14.09 MB/sec warmup 116 sec latency 25.125 ms 2024-05-27T07:43:45.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 302560 14.09 MB/sec warmup 117 sec latency 18.217 ms 2024-05-27T07:43:45.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:45 smithi114 ceph-mon[32038]: pgmap v127: 129 pgs: 129 active+clean; 52 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 253 op/s 2024-05-27T07:43:45.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:45 smithi163 ceph-mon[36451]: pgmap v127: 129 pgs: 129 active+clean; 52 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 253 op/s 2024-05-27T07:43:45.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:45 smithi186 ceph-mon[36647]: pgmap v127: 129 pgs: 129 active+clean; 52 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 253 op/s 2024-05-27T07:43:46.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305514 14.11 MB/sec warmup 118 sec latency 24.886 ms 2024-05-27T07:43:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.098+0000 7fedb1add700 1 -- 172.21.15.114:0/1878675577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedac1034b0 msgr2=0x7fedac1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.098+0000 7fedb1add700 1 --2- 172.21.15.114:0/1878675577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedac1034b0 0x7fedac1038c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fed9c0099c0 tx=0x7fed9c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:43:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.098+0000 7fedb1add700 1 -- 172.21.15.114:0/1878675577 shutdown_connections 2024-05-27T07:43:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.098+0000 7fedb1add700 1 --2- 172.21.15.114:0/1878675577 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedac108730 0x7fedac110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.098+0000 7fedb1add700 1 --2- 172.21.15.114:0/1878675577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedac1034b0 0x7fedac1038c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.098+0000 7fedb1add700 1 --2- 172.21.15.114:0/1878675577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedac107de0 0x7fedac1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.098+0000 7fedb1add700 1 -- 172.21.15.114:0/1878675577 >> 172.21.15.114:0/1878675577 conn(0x7fedac0fedb0 msgr2=0x7fedac1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.098+0000 7fedb1add700 1 -- 172.21.15.114:0/1878675577 shutdown_connections 2024-05-27T07:43:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.098+0000 7fedb1add700 1 -- 172.21.15.114:0/1878675577 wait complete. 2024-05-27T07:43:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.099+0000 7fedb1add700 1 Processor -- start 2024-05-27T07:43:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.099+0000 7fedb1add700 1 -- start start 2024-05-27T07:43:46.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.099+0000 7fedb1add700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedac1034b0 0x7fedac19f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.099+0000 7fedb1add700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedac107de0 0x7fedac19fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.099+0000 7fedb1add700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedac108730 0x7fedac199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.099+0000 7fedb1add700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fedac113360 con 0x7fedac108730 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.099+0000 7fedabfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedac108730 0x7fedac199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.099+0000 7fedb1add700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fedac1131e0 con 0x7fedac107de0 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.099+0000 7fedaaffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedac107de0 0x7fedac19fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.099+0000 7fedb1add700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fedac1134e0 con 0x7fedac1034b0 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.099+0000 7fedabfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedac108730 0x7fedac199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55878/0 (socket says 172.21.15.114:55878) 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.099+0000 7fedabfff700 1 -- 172.21.15.114:0/2134604117 learned_addr learned my addr 172.21.15.114:0/2134604117 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.099+0000 7fedab7fe700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedac1034b0 0x7fedac19f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.100+0000 7fedaaffd700 1 -- 172.21.15.114:0/2134604117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedac1034b0 msgr2=0x7fedac19f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.100+0000 7fedaaffd700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedac1034b0 0x7fedac19f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.100+0000 7fedaaffd700 1 -- 172.21.15.114:0/2134604117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedac108730 msgr2=0x7fedac199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.100+0000 7fedaaffd700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedac108730 0x7fedac199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.100+0000 7fedaaffd700 1 -- 172.21.15.114:0/2134604117 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedac19a270 con 0x7fedac107de0 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.100+0000 7fedabfff700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedac108730 0x7fedac199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.100+0000 7fedaaffd700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedac107de0 0x7fedac19fef0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feda000e9c0 tx=0x7feda000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.100+0000 7feda8ff9700 1 -- 172.21.15.114:0/2134604117 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda000c9c0 con 0x7fedac107de0 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.100+0000 7fedb1add700 1 -- 172.21.15.114:0/2134604117 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedac19a560 con 0x7fedac107de0 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.100+0000 7feda8ff9700 1 -- 172.21.15.114:0/2134604117 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feda000cb60 con 0x7fedac107de0 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.100+0000 7feda8ff9700 1 -- 172.21.15.114:0/2134604117 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda0018680 con 0x7fedac107de0 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.100+0000 7fedb1add700 1 -- 172.21.15.114:0/2134604117 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedac1a6760 con 0x7fedac107de0 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.101+0000 7fedb1add700 1 -- 172.21.15.114:0/2134604117 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedac104960 con 0x7fedac107de0 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.103+0000 7feda8ff9700 1 -- 172.21.15.114:0/2134604117 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7feda0010430 con 0x7fedac107de0 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.104+0000 7feda8ff9700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fed940b29f0 0x7fed940b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.104+0000 7fedab7fe700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fed940b29f0 0x7fed940b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.104+0000 7feda8ff9700 1 -- 172.21.15.114:0/2134604117 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7feda00e2f00 con 0x7fedac107de0 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.104+0000 7fedab7fe700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fed940b29f0 0x7fed940b4f00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fed9c000c00 tx=0x7fed9c01a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:46.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.107+0000 7feda8ff9700 1 -- 172.21.15.114:0/2134604117 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7feda00aa4d0 con 0x7fedac107de0 2024-05-27T07:43:46.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.394+0000 7fedb1add700 1 -- 172.21.15.114:0/2134604117 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedac19afe0 con 0x7fedac107de0 2024-05-27T07:43:46.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.400+0000 7feda8ff9700 1 -- 172.21.15.114:0/2134604117 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11827 (secure 0 0 0) 0x7feda00a9dd0 con 0x7fedac107de0 2024-05-27T07:43:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:43:46.399 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:43:32.272613+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25691":{"gid":25691,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3475666132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3475666132},{"type":"v1","addr":"172.21.15.163:6839","nonce":3475666132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:43:46.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.405+0000 7fedb1add700 1 -- 172.21.15.114:0/2134604117 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fed940b29f0 msgr2=0x7fed940b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:46.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.405+0000 7fedb1add700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fed940b29f0 0x7fed940b4f00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fed9c000c00 tx=0x7fed9c01a040 comp rx=0 tx=0).stop 2024-05-27T07:43:46.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.405+0000 7fedb1add700 1 -- 172.21.15.114:0/2134604117 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedac107de0 msgr2=0x7fedac19fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:46.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.405+0000 7fedb1add700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedac107de0 0x7fedac19fef0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feda000e9c0 tx=0x7feda000ecd0 comp rx=0 tx=0).stop 2024-05-27T07:43:46.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.406+0000 7fedb1add700 1 -- 172.21.15.114:0/2134604117 shutdown_connections 2024-05-27T07:43:46.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.406+0000 7fedb1add700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedac108730 0x7fedac199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:46.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.406+0000 7fedb1add700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedac1034b0 0x7fedac19f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:46.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.406+0000 7fedb1add700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fed940b29f0 0x7fed940b4f00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:46.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.406+0000 7fedb1add700 1 --2- 172.21.15.114:0/2134604117 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedac107de0 0x7fedac19fef0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:46.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.406+0000 7fedb1add700 1 -- 172.21.15.114:0/2134604117 >> 172.21.15.114:0/2134604117 conn(0x7fedac0fedb0 msgr2=0x7fedac1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:46.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.409+0000 7fedb1add700 1 -- 172.21.15.114:0/2134604117 shutdown_connections 2024-05-27T07:43:46.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:46.409+0000 7fedb1add700 1 -- 172.21.15.114:0/2134604117 wait complete. 2024-05-27T07:43:46.409 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 29 2024-05-27T07:43:47.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:43:47.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 307636 14.06 MB/sec warmup 119 sec latency 113.747 ms 2024-05-27T07:43:47.359 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:47.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:47 smithi114 ceph-mon[32038]: pgmap v128: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 537 op/s 2024-05-27T07:43:47.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:47 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2134604117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:47.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:47 smithi163 ceph-mon[36451]: pgmap v128: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 537 op/s 2024-05-27T07:43:47.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:47 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2134604117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:47.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:47 smithi186 ceph-mon[36647]: pgmap v128: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 537 op/s 2024-05-27T07:43:47.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:47 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2134604117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:49.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.120+0000 7fbb4f557700 1 -- 172.21.15.114:0/4145773087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb48107de0 msgr2=0x7fbb481081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:49.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.120+0000 7fbb4f557700 1 --2- 172.21.15.114:0/4145773087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb48107de0 0x7fbb481081f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fbb380099c0 tx=0x7fbb38009cd0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.120+0000 7fbb4f557700 1 -- 172.21.15.114:0/4145773087 shutdown_connections 2024-05-27T07:43:49.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.120+0000 7fbb4f557700 1 --2- 172.21.15.114:0/4145773087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb48108730 0x7fbb48110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.120+0000 7fbb4f557700 1 --2- 172.21.15.114:0/4145773087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb481034b0 0x7fbb481038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.120+0000 7fbb4f557700 1 --2- 172.21.15.114:0/4145773087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb48107de0 0x7fbb481081f0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.120+0000 7fbb4f557700 1 -- 172.21.15.114:0/4145773087 >> 172.21.15.114:0/4145773087 conn(0x7fbb480fedb0 msgr2=0x7fbb481011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.120+0000 7fbb4f557700 1 -- 172.21.15.114:0/4145773087 shutdown_connections 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.120+0000 7fbb4f557700 1 -- 172.21.15.114:0/4145773087 wait complete. 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.121+0000 7fbb4f557700 1 Processor -- start 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.121+0000 7fbb4f557700 1 -- start start 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.121+0000 7fbb4f557700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb481034b0 0x7fbb4819d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.121+0000 7fbb4f557700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb48107de0 0x7fbb4819d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.121+0000 7fbb4f557700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb48108730 0x7fbb481a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.121+0000 7fbb4f557700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbb48113340 con 0x7fbb481034b0 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.121+0000 7fbb4f557700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbb481131c0 con 0x7fbb48108730 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.121+0000 7fbb4f557700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbb481134c0 con 0x7fbb48107de0 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.121+0000 7fbb4d2f3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb481034b0 0x7fbb4819d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.121+0000 7fbb4d2f3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb481034b0 0x7fbb4819d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55896/0 (socket says 172.21.15.114:55896) 2024-05-27T07:43:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.121+0000 7fbb4d2f3700 1 -- 172.21.15.114:0/274257786 learned_addr learned my addr 172.21.15.114:0/274257786 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.121+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb48107de0 0x7fbb4819d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.121+0000 7fbb4daf4700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb48108730 0x7fbb481a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.122+0000 7fbb4d2f3700 1 -- 172.21.15.114:0/274257786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb48107de0 msgr2=0x7fbb4819d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:49.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.122+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb48107de0 0x7fbb4819d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.122+0000 7fbb4d2f3700 1 -- 172.21.15.114:0/274257786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb48108730 msgr2=0x7fbb481a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:49.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.122+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb48108730 0x7fbb481a1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.122+0000 7fbb4d2f3700 1 -- 172.21.15.114:0/274257786 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb481a1f00 con 0x7fbb481034b0 2024-05-27T07:43:49.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.122+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb481034b0 0x7fbb4819d030 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400e9a0 tx=0x7fbb4400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:49.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.122+0000 7fbb3e7fc700 1 -- 172.21.15.114:0/274257786 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4400c9c0 con 0x7fbb481034b0 2024-05-27T07:43:49.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.122+0000 7fbb3e7fc700 1 -- 172.21.15.114:0/274257786 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb4400cb60 con 0x7fbb481034b0 2024-05-27T07:43:49.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.122+0000 7fbb4f557700 1 -- 172.21.15.114:0/274257786 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb481a21f0 con 0x7fbb481034b0 2024-05-27T07:43:49.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.122+0000 7fbb3e7fc700 1 -- 172.21.15.114:0/274257786 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb44018680 con 0x7fbb481034b0 2024-05-27T07:43:49.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.122+0000 7fbb4f557700 1 -- 172.21.15.114:0/274257786 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb481a9b10 con 0x7fbb481034b0 2024-05-27T07:43:49.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.125+0000 7fbb3e7fc700 1 -- 172.21.15.114:0/274257786 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fbb44018820 con 0x7fbb481034b0 2024-05-27T07:43:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb3e7fc700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbb340b2a70 0x7fbb340b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb5df6cb80 1 Processor -- start 2024-05-27T07:43:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb5df6cb80 1 -- start start 2024-05-27T07:43:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbb340b2a70 0x7fbb340b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb3e7fc700 1 -- 172.21.15.114:0/274257786 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fbb440e2c60 con 0x7fbb481034b0 2024-05-27T07:43:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb5df6cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699d41e40 0x563699ec5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb5df6cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ec5c50 0x563699ebfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb5df6cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ebff80 0x563699ec0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb5df6cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563699deec90 con 0x563699ec5c50 2024-05-27T07:43:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb5df6cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563699dde830 con 0x563699ebff80 2024-05-27T07:43:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb5df6cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563699de2870 con 0x563699d41e40 2024-05-27T07:43:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb4d2f3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699d41e40 0x563699ec5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb4d2f3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699d41e40 0x563699ec5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34872/0 (socket says 172.21.15.114:34872) 2024-05-27T07:43:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb4d2f3700 1 -- 172.21.15.114:0/1064075576 learned_addr learned my addr 172.21.15.114:0/1064075576 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.126+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/1064075576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ec5c50 0x563699ebfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.127+0000 7fbb4daf4700 1 --2- 172.21.15.114:0/1064075576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ebff80 0x563699ec0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.127+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbb340b2a70 0x7fbb340b4f80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbb38000c00 tx=0x7fbb38005040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.127+0000 7fbb4d2f3700 1 -- 172.21.15.114:0/1064075576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ebff80 msgr2=0x563699ec0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.127+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/1064075576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ebff80 0x563699ec0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.127+0000 7fbb4d2f3700 1 -- 172.21.15.114:0/1064075576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ec5c50 msgr2=0x563699ebfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.127+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/1064075576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ec5c50 0x563699ebfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.127+0000 7fbb4d2f3700 1 -- 172.21.15.114:0/1064075576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563699d2e8d0 con 0x563699d41e40 2024-05-27T07:43:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.127+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/1064075576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699d41e40 0x563699ec5710 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbb440a9890 tx=0x7fbb440b23b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb83ff591fd61e71 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.127+0000 7fbb337fe700 1 -- 172.21.15.114:0/1064075576 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb440aa290 con 0x563699d41e40 2024-05-27T07:43:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.127+0000 7fbb337fe700 1 -- 172.21.15.114:0/1064075576 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb440a9b90 con 0x563699d41e40 2024-05-27T07:43:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.127+0000 7fbb337fe700 1 -- 172.21.15.114:0/1064075576 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb440b2560 con 0x563699d41e40 2024-05-27T07:43:49.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.128+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1064075576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699d41e40 msgr2=0x563699ec5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:49.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.128+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1064075576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699d41e40 0x563699ec5710 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbb440a9890 tx=0x7fbb440b23b0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.128+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1064075576 shutdown_connections 2024-05-27T07:43:49.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.128+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1064075576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ebff80 0x563699ec0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.128+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1064075576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699d41e40 0x563699ec5710 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.128+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1064075576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ec5c50 0x563699ebfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.128+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1064075576 >> 172.21.15.114:0/1064075576 conn(0x563699d9caf0 msgr2=0x563699eba640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:49.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.128+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1064075576 shutdown_connections 2024-05-27T07:43:49.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.128+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1064075576 wait complete. 2024-05-27T07:43:49.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.128+0000 7fbb5df6cb80 1 -- start start 2024-05-27T07:43:49.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.129+0000 7fbb5df6cb80 1 Processor -- start 2024-05-27T07:43:49.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.129+0000 7fbb5df6cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699d41e40 0x563699ec9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.129+0000 7fbb5df6cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ebff80 0x563699ec98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.129+0000 7fbb5df6cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699ec5c50 0x563699ec9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.129+0000 7fbb5df6cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563699ce4fa0 con 0x563699ebff80 2024-05-27T07:43:49.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.129+0000 7fbb5df6cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563699d807c0 con 0x563699d41e40 2024-05-27T07:43:49.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.129+0000 7fbb5df6cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563699ecc6e0 con 0x563699ec5c50 2024-05-27T07:43:49.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.129+0000 7fbb5df6cb80 10 client.0 _ll_register_callbacks cb 0x563699d79920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:49.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.129+0000 7fbb4caf2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ebff80 0x563699ec98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.129+0000 7fbb4caf2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ebff80 0x563699ec98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55906/0 (socket says 172.21.15.114:55906) 2024-05-27T07:43:49.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.129+0000 7fbb4caf2700 1 -- 172.21.15.114:0/1857296450 learned_addr learned my addr 172.21.15.114:0/1857296450 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:49.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.129+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699d41e40 0x563699ec9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.129+0000 7fbb4daf4700 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699ec5c50 0x563699ec9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb4caf2700 1 -- 172.21.15.114:0/1857296450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699ec5c50 msgr2=0x563699ec9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:49.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699ec5c50 0x563699ec9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb4caf2700 1 -- 172.21.15.114:0/1857296450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699d41e40 msgr2=0x563699ec9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:49.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699d41e40 0x563699ec9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb4caf2700 1 -- 172.21.15.114:0/1857296450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563699d7d690 con 0x563699ebff80 2024-05-27T07:43:49.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699d41e40 0x563699ec9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:49.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ebff80 0x563699ec98d0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbb48076aa0 tx=0x7fbb38009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a707476f01c553c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:49.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb30ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:49.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb30ff9700 1 -- 172.21.15.114:0/1857296450 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb38034060 con 0x563699ebff80 2024-05-27T07:43:49.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb30ff9700 1 -- 172.21.15.114:0/1857296450 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fbb38021070 con 0x563699ebff80 2024-05-27T07:43:49.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1857296450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb440a9b90 con 0x563699ebff80 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb30ff9700 1 -- 172.21.15.114:0/1857296450 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb3803b050 con 0x563699ebff80 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb5df6cb80 10 client.16257 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1857296450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563699ed71c0 con 0x563699ebff80 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.130+0000 7fbb30ff9700 1 -- 172.21.15.114:0/1857296450 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb440a9b90 con 0x563699ebff80 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.131+0000 7fbb30ff9700 1 -- 172.21.15.114:0/1857296450 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fbb3800fbe0 con 0x563699ebff80 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.131+0000 7fbb5df6cb80 10 client.16257 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.131+0000 7fbb5df6cb80 10 client.16257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.131+0000 7fbb5df6cb80 20 client.16257 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.131+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563699ed9800 0x563699edbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.131+0000 7fbb5df6cb80 4 client.16257 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.131+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1857296450 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563699e8b9f0 con 0x563699ed9800 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.131+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563699ed9800 0x563699edbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.131+0000 7fbb30ff9700 10 client.16257 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.131+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563699ed9800 0x563699edbc20 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec5e7766aa6bbbb8 server_cookie=afce2e2ad33c1af4 in_seq=0 out_seq=0 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb30ff9700 1 -- 172.21.15.114:0/1857296450 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563699e8b9f0 con 0x563699ed9800 2024-05-27T07:43:49.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb30ff9700 10 client.16257 handle_command_reply: tid=0 2024-05-27T07:43:49.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb5df6cb80 1 client.16257 shutdown 2024-05-27T07:43:49.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1857296450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ebff80 msgr2=0x563699ec98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:49.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ebff80 0x563699ec98d0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbb48076aa0 tx=0x7fbb38009310 comp rx=0 tx=0).stop 2024-05-27T07:43:49.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1857296450 shutdown_connections 2024-05-27T07:43:49.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699ec5c50 0x563699ec9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563699ed9800 0x563699edbc20 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699d41e40 0x563699ec9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1857296450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ebff80 0x563699ec98d0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1857296450 >> 172.21.15.114:0/1857296450 conn(0x563699d9caf0 msgr2=0x563699ebea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:49.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb30ff9700 0 client.16257 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:49.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1857296450 shutdown_connections 2024-05-27T07:43:49.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1857296450 wait complete. 2024-05-27T07:43:49.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.133+0000 7fbb5df6cb80 20 client.16257 trim_cache size 0 max 16384 2024-05-27T07:43:49.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.148+0000 7fbb5df6cb80 1 Processor -- start 2024-05-27T07:43:49.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.148+0000 7fbb5df6cb80 1 -- start start 2024-05-27T07:43:49.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb5df6cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699d41e40 0x563699ebfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb5df6cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699ebff80 0x563699ec4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb5df6cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ec5c50 0x563699ecb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb5df6cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbb481b0830 con 0x563699d41e40 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb5df6cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbb481b30b0 con 0x563699ec5c50 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb5df6cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbb481b0ea0 con 0x563699ebff80 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb4d2f3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699d41e40 0x563699ebfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb4d2f3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699d41e40 0x563699ebfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55916/0 (socket says 172.21.15.114:55916) 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb4d2f3700 1 -- 172.21.15.114:0/2678181521 learned_addr learned my addr 172.21.15.114:0/2678181521 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/2678181521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699ebff80 0x563699ec4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb4daf4700 1 --2- 172.21.15.114:0/2678181521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ec5c50 0x563699ecb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb4caf2700 1 -- 172.21.15.114:0/2678181521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ec5c50 msgr2=0x563699ecb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/2678181521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ec5c50 0x563699ecb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb4caf2700 1 -- 172.21.15.114:0/2678181521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699d41e40 msgr2=0x563699ebfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/2678181521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699d41e40 0x563699ebfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb4caf2700 1 -- 172.21.15.114:0/2678181521 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563699ec9160 con 0x563699ebff80 2024-05-27T07:43:49.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb4daf4700 1 --2- 172.21.15.114:0/2678181521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ec5c50 0x563699ecb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:49.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.149+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/2678181521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699d41e40 0x563699ebfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:49.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.150+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/2678181521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699ebff80 0x563699ec4b50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800b580 tx=0x7fbb380057b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ab29dee7e750c41 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:49.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.150+0000 7fbb32ffd700 1 -- 172.21.15.114:0/2678181521 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb38044070 con 0x563699ebff80 2024-05-27T07:43:49.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.150+0000 7fbb32ffd700 1 -- 172.21.15.114:0/2678181521 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb38026620 con 0x563699ebff80 2024-05-27T07:43:49.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.150+0000 7fbb32ffd700 1 -- 172.21.15.114:0/2678181521 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb38034040 con 0x563699ebff80 2024-05-27T07:43:49.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.151+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/2678181521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699ebff80 msgr2=0x563699ec4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:49.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.151+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/2678181521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699ebff80 0x563699ec4b50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800b580 tx=0x7fbb380057b0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.151+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/2678181521 shutdown_connections 2024-05-27T07:43:49.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.151+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/2678181521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ec5c50 0x563699ecb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.151+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/2678181521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699d41e40 0x563699ebfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.151+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/2678181521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699ebff80 0x563699ec4b50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.151+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/2678181521 >> 172.21.15.114:0/2678181521 conn(0x563699d9caf0 msgr2=0x563699ec4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.151+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/2678181521 shutdown_connections 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.151+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/2678181521 wait complete. 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.151+0000 7fbb5df6cb80 1 -- start start 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.151+0000 7fbb5df6cb80 1 Processor -- start 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb5df6cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699d41e40 0x563699ed19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb5df6cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ebff80 0x563699ed1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb5df6cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ec5c50 0x563699ee0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb5df6cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbb481b1570 con 0x563699ec5c50 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb5df6cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbb481b29e0 con 0x563699ebff80 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb5df6cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbb481b1c40 con 0x563699d41e40 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb4d2f3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699d41e40 0x563699ed19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb4caf2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ebff80 0x563699ed1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb5df6cb80 10 client.0 _ll_register_callbacks cb 0x563699d79920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb4caf2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ebff80 0x563699ed1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:41796/0 (socket says 172.21.15.114:41796) 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb4caf2700 1 -- 172.21.15.114:0/1729096249 learned_addr learned my addr 172.21.15.114:0/1729096249 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb4daf4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ec5c50 0x563699ee0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb4d2f3700 1 -- 172.21.15.114:0/1729096249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ebff80 msgr2=0x563699ed1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/1729096249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ebff80 0x563699ed1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb4d2f3700 1 -- 172.21.15.114:0/1729096249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ec5c50 msgr2=0x563699ee0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/1729096249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ec5c50 0x563699ee0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb4d2f3700 1 -- 172.21.15.114:0/1729096249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563699ed25c0 con 0x563699d41e40 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.152+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/1729096249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ebff80 0x563699ed1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.153+0000 7fbb4d2f3700 1 --2- 172.21.15.114:0/1729096249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699d41e40 0x563699ed19a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbb440b24c0 tx=0x7fbb44014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78baa01b65e21e88 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.153+0000 7fbb30ff9700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.153+0000 7fbb30ff9700 1 -- 172.21.15.114:0/1729096249 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb440c2070 con 0x563699d41e40 2024-05-27T07:43:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.153+0000 7fbb30ff9700 1 -- 172.21.15.114:0/1729096249 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fbb440b2560 con 0x563699d41e40 2024-05-27T07:43:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.153+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1729096249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb38026620 con 0x563699d41e40 2024-05-27T07:43:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.153+0000 7fbb30ff9700 1 -- 172.21.15.114:0/1729096249 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb440aa290 con 0x563699d41e40 2024-05-27T07:43:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.153+0000 7fbb5df6cb80 10 client.25855 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.153+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1729096249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563699f43800 con 0x563699d41e40 2024-05-27T07:43:49.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.154+0000 7fbb30ff9700 1 -- 172.21.15.114:0/1729096249 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb38026620 con 0x563699d41e40 2024-05-27T07:43:49.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.154+0000 7fbb30ff9700 1 -- 172.21.15.114:0/1729096249 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fbb440a5500 con 0x563699d41e40 2024-05-27T07:43:49.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.154+0000 7fbb5df6cb80 10 client.25855 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:49.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.154+0000 7fbb5df6cb80 10 client.25855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:49.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.154+0000 7fbb5df6cb80 20 client.25855 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:49.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.154+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1729096249 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563699f43a90 0x563699f45eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.154+0000 7fbb5df6cb80 4 client.25855 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.154+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1729096249 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563699df0be0 con 0x563699f43a90 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.154+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/1729096249 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563699f43a90 0x563699f45eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.154+0000 7fbb30ff9700 10 client.25855 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.155+0000 7fbb4caf2700 1 --2- 172.21.15.114:0/1729096249 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563699f43a90 0x563699f45eb0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c015f90744cbe68b server_cookie=b53ca2a2eac33336 in_seq=0 out_seq=0 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.156+0000 7fbb30ff9700 1 -- 172.21.15.114:0/1729096249 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563699df0be0 con 0x563699f43a90 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.156+0000 7fbb30ff9700 10 client.25855 handle_command_reply: tid=0 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.156+0000 7fbb5df6cb80 1 client.25855 shutdown 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.157+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1729096249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699d41e40 msgr2=0x563699ed19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.157+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1729096249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699d41e40 0x563699ed19a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbb440b24c0 tx=0x7fbb44014000 comp rx=0 tx=0).stop 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.157+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1729096249 shutdown_connections 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.157+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1729096249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563699ec5c50 0x563699ee0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.157+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1729096249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563699d41e40 0x563699ed19a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.157+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1729096249 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563699f43a90 0x563699f45eb0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.157+0000 7fbb5df6cb80 1 --2- 172.21.15.114:0/1729096249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563699ebff80 0x563699ed1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.157+0000 7fbb30ff9700 0 client.25855 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:49.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.157+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1729096249 >> 172.21.15.114:0/1729096249 conn(0x563699d9caf0 msgr2=0x563699ec7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:49.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.157+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1729096249 shutdown_connections 2024-05-27T07:43:49.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.157+0000 7fbb5df6cb80 1 -- 172.21.15.114:0/1729096249 wait complete. 2024-05-27T07:43:49.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.157+0000 7fbb5df6cb80 20 client.25855 trim_cache size 0 max 16384 2024-05-27T07:43:49.157 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:43:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.158+0000 7fbb4f557700 1 -- 172.21.15.114:0/274257786 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbb340b2a70 msgr2=0x7fbb340b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.158+0000 7fbb4f557700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbb340b2a70 0x7fbb340b4f80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbb38000c00 tx=0x7fbb38005040 comp rx=0 tx=0).stop 2024-05-27T07:43:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.158+0000 7fbb4f557700 1 -- 172.21.15.114:0/274257786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb481034b0 msgr2=0x7fbb4819d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.158+0000 7fbb4f557700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb481034b0 0x7fbb4819d030 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400e9a0 tx=0x7fbb4400ecb0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.158+0000 7fbb4f557700 1 -- 172.21.15.114:0/274257786 shutdown_connections 2024-05-27T07:43:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.158+0000 7fbb4f557700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbb340b2a70 0x7fbb340b4f80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.158+0000 7fbb4f557700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb48108730 0x7fbb481a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.158+0000 7fbb4f557700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb481034b0 0x7fbb4819d030 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.158+0000 7fbb4f557700 1 --2- 172.21.15.114:0/274257786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb48107de0 0x7fbb4819d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.158+0000 7fbb4f557700 1 -- 172.21.15.114:0/274257786 >> 172.21.15.114:0/274257786 conn(0x7fbb480fedb0 msgr2=0x7fbb4810fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.158+0000 7fbb4f557700 1 -- 172.21.15.114:0/274257786 shutdown_connections 2024-05-27T07:43:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:49.158+0000 7fbb4f557700 1 -- 172.21.15.114:0/274257786 wait complete. 2024-05-27T07:43:49.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 312775 16.99 MB/sec execute 1 sec latency 14.601 ms 2024-05-27T07:43:49.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:49 smithi114 ceph-mon[32038]: pgmap v129: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 412 op/s 2024-05-27T07:43:49.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:49 smithi163 ceph-mon[36451]: pgmap v129: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 412 op/s 2024-05-27T07:43:49.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:49 smithi186 ceph-mon[36647]: pgmap v129: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 412 op/s 2024-05-27T07:43:50.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 315797 17.33 MB/sec execute 2 sec latency 26.726 ms 2024-05-27T07:43:50.960 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:43:50.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:43:51.219 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:51.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 318128 14.41 MB/sec execute 3 sec latency 14.501 ms 2024-05-27T07:43:51.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:51 smithi114 ceph-mon[32038]: pgmap v130: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 505 op/s 2024-05-27T07:43:51.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:51 smithi163 ceph-mon[36451]: pgmap v130: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 505 op/s 2024-05-27T07:43:51.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:51 smithi186 ceph-mon[36647]: pgmap v130: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 505 op/s 2024-05-27T07:43:52.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 319979 13.89 MB/sec execute 4 sec latency 125.097 ms 2024-05-27T07:43:53.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 323003 14.80 MB/sec execute 5 sec latency 34.710 ms 2024-05-27T07:43:53.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:53 smithi114 ceph-mon[32038]: pgmap v131: 129 pgs: 129 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 450 op/s 2024-05-27T07:43:53.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:53 smithi163 ceph-mon[36451]: pgmap v131: 129 pgs: 129 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 450 op/s 2024-05-27T07:43:53.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:53 smithi186 ceph-mon[36647]: pgmap v131: 129 pgs: 129 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 450 op/s 2024-05-27T07:43:54.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 325694 14.37 MB/sec execute 6 sec latency 29.571 ms 2024-05-27T07:43:54.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.686+0000 7fcb6cf97700 1 -- 172.21.15.114:0/2891212450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb68108740 msgr2=0x7fcb681100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:54.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.686+0000 7fcb6cf97700 1 --2- 172.21.15.114:0/2891212450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb68108740 0x7fcb681100a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000b0d0 tx=0x7fcb6000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.686+0000 7fcb6cf97700 1 -- 172.21.15.114:0/2891212450 shutdown_connections 2024-05-27T07:43:54.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.686+0000 7fcb6cf97700 1 --2- 172.21.15.114:0/2891212450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb68108740 0x7fcb681100a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.686+0000 7fcb6cf97700 1 --2- 172.21.15.114:0/2891212450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb68103c50 0x7fcb68104060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.686+0000 7fcb6cf97700 1 --2- 172.21.15.114:0/2891212450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb68107df0 0x7fcb68108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.686+0000 7fcb6cf97700 1 -- 172.21.15.114:0/2891212450 >> 172.21.15.114:0/2891212450 conn(0x7fcb680fe6e0 msgr2=0x7fcb68100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:54.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.686+0000 7fcb6cf97700 1 -- 172.21.15.114:0/2891212450 shutdown_connections 2024-05-27T07:43:54.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.686+0000 7fcb6cf97700 1 -- 172.21.15.114:0/2891212450 wait complete. 2024-05-27T07:43:54.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.687+0000 7fcb6cf97700 1 Processor -- start 2024-05-27T07:43:54.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.687+0000 7fcb6cf97700 1 -- start start 2024-05-27T07:43:54.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.687+0000 7fcb6cf97700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb68103c50 0x7fcb6819f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.687+0000 7fcb6cf97700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb68107df0 0x7fcb6819fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.687+0000 7fcb6cf97700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb68108740 0x7fcb68199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.687+0000 7fcb6cf97700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcb68113370 con 0x7fcb68107df0 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.687+0000 7fcb6cf97700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcb681131f0 con 0x7fcb68108740 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.687+0000 7fcb6cf97700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb681134f0 con 0x7fcb68103c50 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.687+0000 7fcb65d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb68107df0 0x7fcb6819fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.687+0000 7fcb6659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb68103c50 0x7fcb6819f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.687+0000 7fcb65d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb68107df0 0x7fcb6819fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57482/0 (socket says 172.21.15.114:57482) 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.687+0000 7fcb6659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb68103c50 0x7fcb6819f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:32804/0 (socket says 172.21.15.114:32804) 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.687+0000 7fcb65d9b700 1 -- 172.21.15.114:0/3328010017 learned_addr learned my addr 172.21.15.114:0/3328010017 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.688+0000 7fcb66d9d700 1 --2- 172.21.15.114:0/3328010017 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb68108740 0x7fcb68199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.688+0000 7fcb65d9b700 1 -- 172.21.15.114:0/3328010017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb68103c50 msgr2=0x7fcb6819f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.688+0000 7fcb65d9b700 1 --2- 172.21.15.114:0/3328010017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb68103c50 0x7fcb6819f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.688+0000 7fcb65d9b700 1 -- 172.21.15.114:0/3328010017 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb68108740 msgr2=0x7fcb68199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.688+0000 7fcb65d9b700 1 --2- 172.21.15.114:0/3328010017 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb68108740 0x7fcb68199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.688+0000 7fcb65d9b700 1 -- 172.21.15.114:0/3328010017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6819a1d0 con 0x7fcb68107df0 2024-05-27T07:43:54.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.688+0000 7fcb65d9b700 1 --2- 172.21.15.114:0/3328010017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb68107df0 0x7fcb6819fe10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00d950 tx=0x7fcb5c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:54.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.688+0000 7fcb577fe700 1 -- 172.21.15.114:0/3328010017 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c016070 con 0x7fcb68107df0 2024-05-27T07:43:54.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.688+0000 7fcb6cf97700 1 -- 172.21.15.114:0/3328010017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6819a4c0 con 0x7fcb68107df0 2024-05-27T07:43:54.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.688+0000 7fcb577fe700 1 -- 172.21.15.114:0/3328010017 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb5c004830 con 0x7fcb68107df0 2024-05-27T07:43:54.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.688+0000 7fcb577fe700 1 -- 172.21.15.114:0/3328010017 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c005520 con 0x7fcb68107df0 2024-05-27T07:43:54.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.688+0000 7fcb6cf97700 1 -- 172.21.15.114:0/3328010017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb681a6760 con 0x7fcb68107df0 2024-05-27T07:43:54.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.691+0000 7fcb577fe700 1 -- 172.21.15.114:0/3328010017 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fcb5c00f690 con 0x7fcb68107df0 2024-05-27T07:43:54.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.691+0000 7fcb577fe700 1 --2- 172.21.15.114:0/3328010017 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcb500b29f0 0x7fcb500b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.692+0000 7fcb6659c700 1 --2- 172.21.15.114:0/3328010017 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcb500b29f0 0x7fcb500b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.692+0000 7fcb577fe700 1 -- 172.21.15.114:0/3328010017 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fcb5c0e2680 con 0x7fcb68107df0 2024-05-27T07:43:54.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.692+0000 7fcb6659c700 1 --2- 172.21.15.114:0/3328010017 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcb500b29f0 0x7fcb500b4f00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcb6819ac30 tx=0x7fcb58009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:54.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.692+0000 7fcb7b9acb80 1 Processor -- start 2024-05-27T07:43:54.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.692+0000 7fcb7b9acb80 1 -- start start 2024-05-27T07:43:54.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb7b9acb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbc3bec0 0x5579cbdc0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb7b9acb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdc10f0 0x5579cbdbac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb7b9acb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbdbb1a0 0x5579cbdbb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb7b9acb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5579cbce8df0 con 0x5579cbc3bec0 2024-05-27T07:43:54.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb7b9acb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5579cbcd8990 con 0x5579cbdbb1a0 2024-05-27T07:43:54.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb7b9acb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5579cbcdc9d0 con 0x5579cbdc10f0 2024-05-27T07:43:54.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb6659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbc3bec0 0x5579cbdc0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb6659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbc3bec0 0x5579cbdc0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57490/0 (socket says 172.21.15.114:57490) 2024-05-27T07:43:54.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb65d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdc10f0 0x5579cbdbac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb6659c700 1 -- 172.21.15.114:0/1257347340 learned_addr learned my addr 172.21.15.114:0/1257347340 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:54.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb66d9d700 1 --2- 172.21.15.114:0/1257347340 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbdbb1a0 0x5579cbdbb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb65d9b700 1 -- 172.21.15.114:0/1257347340 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbdbb1a0 msgr2=0x5579cbdbb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:54.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb65d9b700 1 --2- 172.21.15.114:0/1257347340 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbdbb1a0 0x5579cbdbb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb65d9b700 1 -- 172.21.15.114:0/1257347340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbc3bec0 msgr2=0x5579cbdc0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:54.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb65d9b700 1 --2- 172.21.15.114:0/1257347340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbc3bec0 0x5579cbdc0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb65d9b700 1 -- 172.21.15.114:0/1257347340 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579cbc82030 con 0x5579cbdc10f0 2024-05-27T07:43:54.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb6659c700 1 --2- 172.21.15.114:0/1257347340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbc3bec0 0x5579cbdc0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:54.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.693+0000 7fcb65d9b700 1 --2- 172.21.15.114:0/1257347340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdc10f0 0x5579cbdbac60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0dfa20 tx=0x7fcb5c0a5c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6deb669bbb8ef471 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:54.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.694+0000 7fcb54ff9700 1 -- 172.21.15.114:0/1257347340 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0e1040 con 0x5579cbdc10f0 2024-05-27T07:43:54.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.694+0000 7fcb54ff9700 1 -- 172.21.15.114:0/1257347340 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb5c0920c0 con 0x5579cbdc10f0 2024-05-27T07:43:54.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.694+0000 7fcb54ff9700 1 -- 172.21.15.114:0/1257347340 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0a96b0 con 0x5579cbdc10f0 2024-05-27T07:43:54.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.695+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/1257347340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdc10f0 msgr2=0x5579cbdbac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:54.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.695+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/1257347340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdc10f0 0x5579cbdbac60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0dfa20 tx=0x7fcb5c0a5c70 comp rx=0 tx=0).stop 2024-05-27T07:43:54.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.695+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/1257347340 shutdown_connections 2024-05-27T07:43:54.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.695+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/1257347340 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbdbb1a0 0x5579cbdbb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.695+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/1257347340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbc3bec0 0x5579cbdc0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.695+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/1257347340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdc10f0 0x5579cbdbac60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.695+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/1257347340 >> 172.21.15.114:0/1257347340 conn(0x5579cbc3e7f0 msgr2=0x5579cbdb5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.695+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/1257347340 shutdown_connections 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.695+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/1257347340 wait complete. 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.695+0000 7fcb7b9acb80 1 -- start start 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.695+0000 7fcb7b9acb80 1 Processor -- start 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb7b9acb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbc3bec0 0x5579cbdc4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb7b9acb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdbb1a0 0x5579cbdc4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb7b9acb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdc10f0 0x5579cbdc54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb7b9acb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5579cbbdf020 con 0x5579cbdbb1a0 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb7b9acb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5579cbc40e20 con 0x5579cbc3bec0 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb7b9acb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5579cbdc7be0 con 0x5579cbdc10f0 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb7b9acb80 10 client.0 _ll_register_callbacks cb 0x5579cbc6f630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:54.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb65d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdbb1a0 0x5579cbdc4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb65d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdbb1a0 0x5579cbdc4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57492/0 (socket says 172.21.15.114:57492) 2024-05-27T07:43:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb65d9b700 1 -- 172.21.15.114:0/150256844 learned_addr learned my addr 172.21.15.114:0/150256844 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb66d9d700 1 --2- 172.21.15.114:0/150256844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdc10f0 0x5579cbdc54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb6659c700 1 --2- 172.21.15.114:0/150256844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbc3bec0 0x5579cbdc4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb65d9b700 1 -- 172.21.15.114:0/150256844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdc10f0 msgr2=0x5579cbdc54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb65d9b700 1 --2- 172.21.15.114:0/150256844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdc10f0 0x5579cbdc54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb65d9b700 1 -- 172.21.15.114:0/150256844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbc3bec0 msgr2=0x5579cbdc4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb65d9b700 1 --2- 172.21.15.114:0/150256844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbc3bec0 0x5579cbdc4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb65d9b700 1 -- 172.21.15.114:0/150256844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579cbc28950 con 0x5579cbdbb1a0 2024-05-27T07:43:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb65d9b700 1 --2- 172.21.15.114:0/150256844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdbb1a0 0x5579cbdc4fb0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0a77b0 tx=0x7fcb5c0a64d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1018c52357acd6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb4dffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb4dffb700 1 -- 172.21.15.114:0/150256844 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0c2020 con 0x5579cbdbb1a0 2024-05-27T07:43:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb4dffb700 1 -- 172.21.15.114:0/150256844 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fcb5c0e2480 con 0x5579cbdbb1a0 2024-05-27T07:43:54.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.696+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/150256844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb5c0920c0 con 0x5579cbdbb1a0 2024-05-27T07:43:54.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.697+0000 7fcb4dffb700 1 -- 172.21.15.114:0/150256844 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcb5c0a9450 con 0x5579cbdbb1a0 2024-05-27T07:43:54.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.697+0000 7fcb7b9acb80 10 client.16275 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:54.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.697+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/150256844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5579cbc927e0 con 0x5579cbdbb1a0 2024-05-27T07:43:54.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.697+0000 7fcb4dffb700 1 -- 172.21.15.114:0/150256844 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb5c0920c0 con 0x5579cbdbb1a0 2024-05-27T07:43:54.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.697+0000 7fcb4dffb700 1 -- 172.21.15.114:0/150256844 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fcb5c0b0ce0 con 0x5579cbdbb1a0 2024-05-27T07:43:54.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.697+0000 7fcb7b9acb80 10 client.16275 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:54.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.697+0000 7fcb7b9acb80 10 client.16275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:54.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.697+0000 7fcb7b9acb80 20 client.16275 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:54.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.697+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/150256844 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5579cbdd4b50 0x5579cbdd6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.697+0000 7fcb7b9acb80 4 client.16275 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:54.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.697+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/150256844 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579cbcead40 con 0x5579cbdd4b50 2024-05-27T07:43:54.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.697+0000 7fcb6659c700 1 --2- 172.21.15.114:0/150256844 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5579cbdd4b50 0x5579cbdd6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.698+0000 7fcb4dffb700 10 client.16275 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:54.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.698+0000 7fcb6659c700 1 --2- 172.21.15.114:0/150256844 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5579cbdd4b50 0x5579cbdd6f70 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c33286ad4788cdb8 server_cookie=8c009ef649883042 in_seq=0 out_seq=0 2024-05-27T07:43:54.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.698+0000 7fcb4dffb700 1 -- 172.21.15.114:0/150256844 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5579cbcead40 con 0x5579cbdd4b50 2024-05-27T07:43:54.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.698+0000 7fcb4dffb700 10 client.16275 handle_command_reply: tid=0 2024-05-27T07:43:54.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.698+0000 7fcb7b9acb80 1 client.16275 shutdown 2024-05-27T07:43:54.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.698+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/150256844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdbb1a0 msgr2=0x5579cbdc4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:54.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.698+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/150256844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdbb1a0 0x5579cbdc4fb0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0a77b0 tx=0x7fcb5c0a64d0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.698+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/150256844 shutdown_connections 2024-05-27T07:43:54.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.698+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/150256844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdc10f0 0x5579cbdc54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.699+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/150256844 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5579cbdd4b50 0x5579cbdd6f70 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.699+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/150256844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbc3bec0 0x5579cbdc4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.699+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/150256844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdbb1a0 0x5579cbdc4fb0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.833 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:43:54.833 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:43:54.833 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "a8aa2006-1a5a-480b-9ece-70839119390e", 2024-05-27T07:43:54.833 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:43:54.833 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:43:54.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.699+0000 7fcb4dffb700 0 client.16275 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:54.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.699+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/150256844 >> 172.21.15.114:0/150256844 conn(0x5579cbc3e7f0 msgr2=0x5579cbdb9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:54.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.699+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/150256844 shutdown_connections 2024-05-27T07:43:54.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.699+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/150256844 wait complete. 2024-05-27T07:43:54.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.699+0000 7fcb7b9acb80 20 client.16275 trim_cache size 0 max 16384 2024-05-27T07:43:54.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.720+0000 7fcb7b9acb80 1 Processor -- start 2024-05-27T07:43:54.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.720+0000 7fcb7b9acb80 1 -- start start 2024-05-27T07:43:54.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.720+0000 7fcb7b9acb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbc3bec0 0x5579cbdc39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.720+0000 7fcb7b9acb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdbb1a0 0x5579cbdbec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.720+0000 7fcb7b9acb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 0x5579cbdd9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.720+0000 7fcb7b9acb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcb681ac4d0 con 0x5579cbdc10f0 2024-05-27T07:43:54.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.720+0000 7fcb7b9acb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcb681aed50 con 0x5579cbc3bec0 2024-05-27T07:43:54.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.720+0000 7fcb7b9acb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb681acb40 con 0x5579cbdbb1a0 2024-05-27T07:43:54.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.720+0000 7fcb66d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 0x5579cbdd9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.720+0000 7fcb66d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 0x5579cbdd9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57494/0 (socket says 172.21.15.114:57494) 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.720+0000 7fcb66d9d700 1 -- 172.21.15.114:0/477129948 learned_addr learned my addr 172.21.15.114:0/477129948 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.720+0000 7fcb65d9b700 1 --2- 172.21.15.114:0/477129948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdbb1a0 0x5579cbdbec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.720+0000 7fcb6659c700 1 --2- 172.21.15.114:0/477129948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbc3bec0 0x5579cbdc39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.721+0000 7fcb66d9d700 1 -- 172.21.15.114:0/477129948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdbb1a0 msgr2=0x5579cbdbec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.721+0000 7fcb66d9d700 1 --2- 172.21.15.114:0/477129948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdbb1a0 0x5579cbdbec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.721+0000 7fcb66d9d700 1 -- 172.21.15.114:0/477129948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbc3bec0 msgr2=0x5579cbdc39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.721+0000 7fcb66d9d700 1 --2- 172.21.15.114:0/477129948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbc3bec0 0x5579cbdc39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.721+0000 7fcb66d9d700 1 -- 172.21.15.114:0/477129948 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579cbdad9d0 con 0x5579cbdc10f0 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.721+0000 7fcb66d9d700 1 --2- 172.21.15.114:0/477129948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 0x5579cbdd9f90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fcb60009580 tx=0x7fcb6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46093b04f816cdb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.721+0000 7fcb4ffff700 1 -- 172.21.15.114:0/477129948 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6002b070 con 0x5579cbdc10f0 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.721+0000 7fcb4ffff700 1 -- 172.21.15.114:0/477129948 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb6000bca0 con 0x5579cbdc10f0 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.721+0000 7fcb4ffff700 1 -- 172.21.15.114:0/477129948 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6001b530 con 0x5579cbdc10f0 2024-05-27T07:43:54.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.722+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/477129948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 msgr2=0x5579cbdd9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:54.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.722+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/477129948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 0x5579cbdd9f90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fcb60009580 tx=0x7fcb6000b040 comp rx=0 tx=0).stop 2024-05-27T07:43:54.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.722+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/477129948 shutdown_connections 2024-05-27T07:43:54.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.722+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/477129948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 0x5579cbdd9f90 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.722+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/477129948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbc3bec0 0x5579cbdc39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.722+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/477129948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbdbb1a0 0x5579cbdbec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.722+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/477129948 >> 172.21.15.114:0/477129948 conn(0x5579cbc3e7f0 msgr2=0x5579cbdbe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:54.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.722+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/477129948 shutdown_connections 2024-05-27T07:43:54.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.722+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/477129948 wait complete. 2024-05-27T07:43:54.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.722+0000 7fcb7b9acb80 1 -- start start 2024-05-27T07:43:54.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.722+0000 7fcb7b9acb80 1 Processor -- start 2024-05-27T07:43:54.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb7b9acb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbc3bec0 0x5579cbdd8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb7b9acb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbdbb1a0 0x5579cbdd9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb7b9acb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 0x5579cbdd9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb7b9acb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcb681ad210 con 0x5579cbdc10f0 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb7b9acb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcb681ae680 con 0x5579cbdbb1a0 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb7b9acb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb681ad8e0 con 0x5579cbc3bec0 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb7b9acb80 10 client.0 _ll_register_callbacks cb 0x5579cbc6f630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb66d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 0x5579cbdd9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb66d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 0x5579cbdd9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57502/0 (socket says 172.21.15.114:57502) 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb66d9d700 1 -- 172.21.15.114:0/2860092023 learned_addr learned my addr 172.21.15.114:0/2860092023 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb6659c700 1 --2- 172.21.15.114:0/2860092023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbc3bec0 0x5579cbdd8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb65d9b700 1 --2- 172.21.15.114:0/2860092023 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbdbb1a0 0x5579cbdd9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb66d9d700 1 -- 172.21.15.114:0/2860092023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbc3bec0 msgr2=0x5579cbdd8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb66d9d700 1 --2- 172.21.15.114:0/2860092023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbc3bec0 0x5579cbdd8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb66d9d700 1 -- 172.21.15.114:0/2860092023 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbdbb1a0 msgr2=0x5579cbdd9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb66d9d700 1 --2- 172.21.15.114:0/2860092023 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbdbb1a0 0x5579cbdd9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb66d9d700 1 -- 172.21.15.114:0/2860092023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579cbe3f3d0 con 0x5579cbdc10f0 2024-05-27T07:43:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb66d9d700 1 --2- 172.21.15.114:0/2860092023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 0x5579cbdd9870 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fcb600061c0 tx=0x7fcb600099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78c5829ffb25751 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.723+0000 7fcb4dffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb4dffb700 1 -- 172.21.15.114:0/2860092023 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60034020 con 0x5579cbdc10f0 2024-05-27T07:43:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb4dffb700 1 -- 172.21.15.114:0/2860092023 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fcb6000baa0 con 0x5579cbdc10f0 2024-05-27T07:43:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/2860092023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb6000bca0 con 0x5579cbdc10f0 2024-05-27T07:43:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb4dffb700 1 -- 172.21.15.114:0/2860092023 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcb6002b070 con 0x5579cbdc10f0 2024-05-27T07:43:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb7b9acb80 10 client.16281 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/2860092023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5579cbe40670 con 0x5579cbdc10f0 2024-05-27T07:43:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb4dffb700 1 -- 172.21.15.114:0/2860092023 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb6000bca0 con 0x5579cbdc10f0 2024-05-27T07:43:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb4dffb700 1 -- 172.21.15.114:0/2860092023 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fcb60011070 con 0x5579cbdc10f0 2024-05-27T07:43:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb7b9acb80 10 client.16281 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb7b9acb80 10 client.16281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb7b9acb80 20 client.16281 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/2860092023 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5579cbe40c60 0x5579cbe43080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb7b9acb80 4 client.16281 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/2860092023 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5579cbce4520 con 0x5579cbe40c60 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.724+0000 7fcb6659c700 1 --2- 172.21.15.114:0/2860092023 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5579cbe40c60 0x5579cbe43080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.725+0000 7fcb4dffb700 10 client.16281 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.725+0000 7fcb6659c700 1 --2- 172.21.15.114:0/2860092023 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5579cbe40c60 0x5579cbe43080 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1f7f3dc9379bb1a server_cookie=c84d4a9573856daa in_seq=0 out_seq=0 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.727+0000 7fcb4dffb700 1 -- 172.21.15.114:0/2860092023 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5579cbce4520 con 0x5579cbe40c60 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb4dffb700 10 client.16281 handle_command_reply: tid=0 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb7b9acb80 1 client.16281 shutdown 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/2860092023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 msgr2=0x5579cbdd9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/2860092023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 0x5579cbdd9870 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fcb600061c0 tx=0x7fcb600099e0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/2860092023 shutdown_connections 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/2860092023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5579cbdc10f0 0x5579cbdd9870 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/2860092023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579cbc3bec0 0x5579cbdd8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/2860092023 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5579cbe40c60 0x5579cbe43080 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb7b9acb80 1 --2- 172.21.15.114:0/2860092023 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579cbdbb1a0 0x5579cbdd9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb4dffb700 0 client.16281 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:54.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/2860092023 >> 172.21.15.114:0/2860092023 conn(0x5579cbc3e7f0 msgr2=0x5579cbdc4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:54.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/2860092023 shutdown_connections 2024-05-27T07:43:54.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb7b9acb80 1 -- 172.21.15.114:0/2860092023 wait complete. 2024-05-27T07:43:54.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.728+0000 7fcb7b9acb80 20 client.16281 trim_cache size 0 max 16384 2024-05-27T07:43:54.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.729+0000 7fcb6cf97700 1 -- 172.21.15.114:0/3328010017 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcb500b29f0 msgr2=0x7fcb500b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:54.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.729+0000 7fcb6cf97700 1 --2- 172.21.15.114:0/3328010017 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcb500b29f0 0x7fcb500b4f00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcb6819ac30 tx=0x7fcb58009260 comp rx=0 tx=0).stop 2024-05-27T07:43:54.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.729+0000 7fcb6cf97700 1 -- 172.21.15.114:0/3328010017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb68107df0 msgr2=0x7fcb6819fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:54.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.729+0000 7fcb6cf97700 1 --2- 172.21.15.114:0/3328010017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb68107df0 0x7fcb6819fe10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00d950 tx=0x7fcb5c00dc60 comp rx=0 tx=0).stop 2024-05-27T07:43:54.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.729+0000 7fcb6cf97700 1 -- 172.21.15.114:0/3328010017 shutdown_connections 2024-05-27T07:43:54.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.729+0000 7fcb6cf97700 1 --2- 172.21.15.114:0/3328010017 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcb500b29f0 0x7fcb500b4f00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.729+0000 7fcb6cf97700 1 --2- 172.21.15.114:0/3328010017 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb68108740 0x7fcb68199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.729+0000 7fcb6cf97700 1 --2- 172.21.15.114:0/3328010017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb68103c50 0x7fcb6819f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.729+0000 7fcb6cf97700 1 --2- 172.21.15.114:0/3328010017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb68107df0 0x7fcb6819fe10 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:54.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.729+0000 7fcb6cf97700 1 -- 172.21.15.114:0/3328010017 >> 172.21.15.114:0/3328010017 conn(0x7fcb680fe6e0 msgr2=0x7fcb68107560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:54.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.729+0000 7fcb6cf97700 1 -- 172.21.15.114:0/3328010017 shutdown_connections 2024-05-27T07:43:54.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:54.729+0000 7fcb6cf97700 1 -- 172.21.15.114:0/3328010017 wait complete. 2024-05-27T07:43:55.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 327785 14.07 MB/sec execute 7 sec latency 27.288 ms 2024-05-27T07:43:55.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.351+0000 7fea1bb8c640 1 -- 172.21.15.114:0/3993247239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea14109030 msgr2=0x7fea14102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:55.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.351+0000 7fea1bb8c640 1 --2- 172.21.15.114:0/3993247239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea14109030 0x7fea14102c70 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fea04009900 tx=0x7fea0402a410 comp rx=0 tx=0).stop 2024-05-27T07:43:55.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.351+0000 7fea1bb8c640 1 -- 172.21.15.114:0/3993247239 shutdown_connections 2024-05-27T07:43:55.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.351+0000 7fea1bb8c640 1 --2- 172.21.15.114:0/3993247239 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea141032e0 0x7fea1410f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.351+0000 7fea1bb8c640 1 --2- 172.21.15.114:0/3993247239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea14109030 0x7fea14102c70 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.351+0000 7fea1bb8c640 1 --2- 172.21.15.114:0/3993247239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea14108b90 0x7fea14102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.351+0000 7fea1bb8c640 1 -- 172.21.15.114:0/3993247239 >> 172.21.15.114:0/3993247239 conn(0x7fea140782d0 msgr2=0x7fea14078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:55.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.351+0000 7fea1bb8c640 1 -- 172.21.15.114:0/3993247239 shutdown_connections 2024-05-27T07:43:55.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.351+0000 7fea1bb8c640 1 -- 172.21.15.114:0/3993247239 wait complete. 2024-05-27T07:43:55.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.352+0000 7fea1bb8c640 1 Processor -- start 2024-05-27T07:43:55.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.352+0000 7fea1bb8c640 1 -- start start 2024-05-27T07:43:55.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea1bb8c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea141032e0 0x7fea141a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea1bb8c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea14108b90 0x7fea141a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea1bb8c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea14109030 0x7fea1419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea1bb8c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fea14112dd0 con 0x7fea14108b90 2024-05-27T07:43:55.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea1bb8c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fea14112c50 con 0x7fea14109030 2024-05-27T07:43:55.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea1bb8c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fea14112f50 con 0x7fea141032e0 2024-05-27T07:43:55.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea19127640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea14108b90 0x7fea141a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea19127640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea14108b90 0x7fea141a0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57518/0 (socket says 172.21.15.114:57518) 2024-05-27T07:43:55.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea19127640 1 -- 172.21.15.114:0/3353849743 learned_addr learned my addr 172.21.15.114:0/3353849743 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:55.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea19928640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea141032e0 0x7fea141a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea1a129640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea14109030 0x7fea1419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea19127640 1 -- 172.21.15.114:0/3353849743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea141032e0 msgr2=0x7fea141a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:55.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea19127640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea141032e0 0x7fea141a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea19127640 1 -- 172.21.15.114:0/3353849743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea14109030 msgr2=0x7fea1419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:55.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea19127640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea14109030 0x7fea1419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea19127640 1 -- 172.21.15.114:0/3353849743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1419ae90 con 0x7fea14108b90 2024-05-27T07:43:55.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea19928640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea141032e0 0x7fea141a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:55.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.353+0000 7fea1a129640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea14109030 0x7fea1419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:55.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.354+0000 7fea19127640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea14108b90 0x7fea141a0b20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fea04005ab0 tx=0x7fea04003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:55.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.354+0000 7fea0affd640 1 -- 172.21.15.114:0/3353849743 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0403f070 con 0x7fea14108b90 2024-05-27T07:43:55.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.354+0000 7fea1bb8c640 1 -- 172.21.15.114:0/3353849743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea1419b0c0 con 0x7fea14108b90 2024-05-27T07:43:55.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.354+0000 7fea0affd640 1 -- 172.21.15.114:0/3353849743 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea04031b50 con 0x7fea14108b90 2024-05-27T07:43:55.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.354+0000 7fea0affd640 1 -- 172.21.15.114:0/3353849743 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea04035030 con 0x7fea14108b90 2024-05-27T07:43:55.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.354+0000 7fea1bb8c640 1 -- 172.21.15.114:0/3353849743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea1419b5b0 con 0x7fea14108b90 2024-05-27T07:43:55.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.357+0000 7fea0affd640 1 -- 172.21.15.114:0/3353849743 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fea04042070 con 0x7fea14108b90 2024-05-27T07:43:55.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.357+0000 7fea22a3d740 1 Processor -- start 2024-05-27T07:43:55.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.357+0000 7fea22a3d740 1 -- start start 2024-05-27T07:43:55.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.357+0000 7fea22a3d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 0x562dead593d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.357+0000 7fea22a3d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea98f2f0 0x562dead59910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.357+0000 7fea22a3d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 0x562dead59e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.357+0000 7fea22a3d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562deaccbc30 con 0x562dea9b81b0 2024-05-27T07:43:55.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.357+0000 7fea22a3d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562deacbb7d0 con 0x562dea9974d0 2024-05-27T07:43:55.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea22a3d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562deacbf810 con 0x562dea98f2f0 2024-05-27T07:43:55.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea1a129640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 0x562dead593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea1a129640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 0x562dead593d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57522/0 (socket says 172.21.15.114:57522) 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea1a129640 1 -- 172.21.15.114:0/56651255 learned_addr learned my addr 172.21.15.114:0/56651255 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea19928640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea98f2f0 0x562dead59910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea19127640 1 --2- 172.21.15.114:0/56651255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 0x562dead59e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea0affd640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fea000b2950 0x7fea000b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea1a129640 1 -- 172.21.15.114:0/56651255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea98f2f0 msgr2=0x562dead59910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea1a129640 1 --2- 172.21.15.114:0/56651255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea98f2f0 0x562dead59910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea1a129640 1 -- 172.21.15.114:0/56651255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 msgr2=0x562dead59e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea1a129640 1 --2- 172.21.15.114:0/56651255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 0x562dead59e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea1a129640 1 -- 172.21.15.114:0/56651255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dea9d0210 con 0x562dea9b81b0 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea19928640 1 --2- 172.21.15.114:0/56651255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea98f2f0 0x562dead59910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea0affd640 1 -- 172.21.15.114:0/3353849743 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fea04102f00 con 0x7fea14108b90 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea1a129640 1 --2- 172.21.15.114:0/56651255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 0x562dead593d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fea0c0117e0 tx=0x7fea0c011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e2f3f5f673cc66 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fea1a129640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fea000b2950 0x7fea000b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fe9fffff640 1 -- 172.21.15.114:0/56651255 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c011290 con 0x562dea9b81b0 2024-05-27T07:43:55.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fe9fffff640 1 -- 172.21.15.114:0/56651255 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea0c003bd0 con 0x562dea9b81b0 2024-05-27T07:43:55.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.358+0000 7fe9fffff640 1 -- 172.21.15.114:0/56651255 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c01a3f0 con 0x562dea9b81b0 2024-05-27T07:43:55.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.359+0000 7fea22a3d740 1 -- 172.21.15.114:0/56651255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 msgr2=0x562dead593d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:55.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.359+0000 7fea22a3d740 1 --2- 172.21.15.114:0/56651255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 0x562dead593d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fea0c0117e0 tx=0x7fea0c011ca0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.359+0000 7fea1a129640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fea000b2950 0x7fea000b4e60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fea0c01b000 tx=0x7fea0c01a700 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:55.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.359+0000 7fea22a3d740 1 -- 172.21.15.114:0/56651255 shutdown_connections 2024-05-27T07:43:55.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.359+0000 7fea22a3d740 1 --2- 172.21.15.114:0/56651255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 0x562dead59e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.359+0000 7fea22a3d740 1 --2- 172.21.15.114:0/56651255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea98f2f0 0x562dead59910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.359+0000 7fea22a3d740 1 --2- 172.21.15.114:0/56651255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 0x562dead593d0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.359+0000 7fea22a3d740 1 -- 172.21.15.114:0/56651255 >> 172.21.15.114:0/56651255 conn(0x562dea9ac300 msgr2=0x562dead55b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:55.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.360+0000 7fea22a3d740 1 -- 172.21.15.114:0/56651255 shutdown_connections 2024-05-27T07:43:55.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.360+0000 7fea22a3d740 1 -- 172.21.15.114:0/56651255 wait complete. 2024-05-27T07:43:55.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.360+0000 7fea22a3d740 1 -- start start 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.360+0000 7fea22a3d740 1 Processor -- start 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.360+0000 7fea22a3d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 0x562dead63a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.360+0000 7fea22a3d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 0x562dead63fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.360+0000 7fea22a3d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea9b81b0 0x562dead69870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.360+0000 7fea22a3d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562dea9a1790 con 0x562dea98f2f0 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.360+0000 7fea22a3d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562deac77890 con 0x562dea9974d0 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.360+0000 7fea22a3d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562dead66500 con 0x562dea9b81b0 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.360+0000 7fea22a3d740 10 client.0 _ll_register_callbacks cb 0x562dea9cfa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea19928640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 0x562dead63a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea19928640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 0x562dead63a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57524/0 (socket says 172.21.15.114:57524) 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea19928640 1 -- 172.21.15.114:0/2070515264 learned_addr learned my addr 172.21.15.114:0/2070515264 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea1a129640 1 --2- 172.21.15.114:0/2070515264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea9b81b0 0x562dead69870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea19127640 1 --2- 172.21.15.114:0/2070515264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 0x562dead63fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea19928640 1 -- 172.21.15.114:0/2070515264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea9b81b0 msgr2=0x562dead69870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea19928640 1 --2- 172.21.15.114:0/2070515264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea9b81b0 0x562dead69870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea19928640 1 -- 172.21.15.114:0/2070515264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 msgr2=0x562dead63fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea19928640 1 --2- 172.21.15.114:0/2070515264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 0x562dead63fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea19928640 1 -- 172.21.15.114:0/2070515264 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562deaa9c620 con 0x562dea98f2f0 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea19928640 1 --2- 172.21.15.114:0/2070515264 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 0x562dead63a80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fea10012d20 tx=0x7fea1000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5fbf18c67a7cd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fe9fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fe9fd7fa640 1 -- 172.21.15.114:0/2070515264 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1000fe60 con 0x562dea98f2f0 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea22a3d740 1 -- 172.21.15.114:0/2070515264 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea0c003bd0 con 0x562dea98f2f0 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fe9fd7fa640 1 -- 172.21.15.114:0/2070515264 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fea100120d0 con 0x562dea98f2f0 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fe9fd7fa640 1 -- 172.21.15.114:0/2070515264 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fea10025050 con 0x562dea98f2f0 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea22a3d740 10 client.16293 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.361+0000 7fea22a3d740 1 -- 172.21.15.114:0/2070515264 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562dead72860 con 0x562dea98f2f0 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.362+0000 7fe9fd7fa640 1 -- 172.21.15.114:0/2070515264 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea0c003bd0 con 0x562dea98f2f0 2024-05-27T07:43:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.362+0000 7fe9fd7fa640 1 -- 172.21.15.114:0/2070515264 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fea100104a0 con 0x562dea98f2f0 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.362+0000 7fea22a3d740 10 client.16293 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.362+0000 7fea22a3d740 10 client.16293 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.362+0000 7fea22a3d740 20 client.16293 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.362+0000 7fea22a3d740 1 --2- 172.21.15.114:0/2070515264 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562dea976d80 0x562dead75da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.362+0000 7fea22a3d740 4 client.16293 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.362+0000 7fea22a3d740 1 -- 172.21.15.114:0/2070515264 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562deaccdb80 con 0x562dea976d80 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.362+0000 7fea19127640 1 --2- 172.21.15.114:0/2070515264 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562dea976d80 0x562dead75da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.362+0000 7fea19127640 1 --2- 172.21.15.114:0/2070515264 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562dea976d80 0x562dead75da0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e8b34a8b0312bd7 server_cookie=d36eefe0e2d4e368 in_seq=0 out_seq=0 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.362+0000 7fe9fd7fa640 10 client.16293 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fe9fd7fa640 1 -- 172.21.15.114:0/2070515264 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562deaccdb80 con 0x562dea976d80 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fe9fd7fa640 10 client.16293 handle_command_reply: tid=0 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fea22a3d740 1 client.16293 shutdown 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fea22a3d740 1 -- 172.21.15.114:0/2070515264 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 msgr2=0x562dead63a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fea22a3d740 1 --2- 172.21.15.114:0/2070515264 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 0x562dead63a80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fea10012d20 tx=0x7fea1000f790 comp rx=0 tx=0).stop 2024-05-27T07:43:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fea22a3d740 1 -- 172.21.15.114:0/2070515264 shutdown_connections 2024-05-27T07:43:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fea22a3d740 1 --2- 172.21.15.114:0/2070515264 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562dea976d80 0x562dead75da0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fea22a3d740 1 --2- 172.21.15.114:0/2070515264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea9b81b0 0x562dead69870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fea22a3d740 1 --2- 172.21.15.114:0/2070515264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 0x562dead63fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fe9fd7fa640 0 client.16293 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fea22a3d740 1 --2- 172.21.15.114:0/2070515264 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 0x562dead63a80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fea22a3d740 1 -- 172.21.15.114:0/2070515264 >> 172.21.15.114:0/2070515264 conn(0x562dea9ac300 msgr2=0x562dead59200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fea22a3d740 1 -- 172.21.15.114:0/2070515264 shutdown_connections 2024-05-27T07:43:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fea22a3d740 1 -- 172.21.15.114:0/2070515264 wait complete. 2024-05-27T07:43:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.364+0000 7fea22a3d740 20 client.16293 trim_cache size 0 max 16384 2024-05-27T07:43:55.380 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea22a3d740 1 Processor -- start 2024-05-27T07:43:55.380 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea22a3d740 1 -- start start 2024-05-27T07:43:55.380 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea22a3d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 0x562dead59870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.380 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea22a3d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea9974d0 0x562dead70560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.380 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea22a3d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9b81b0 0x562dead69770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.380 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea22a3d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fea141af3c0 con 0x562dea98f2f0 2024-05-27T07:43:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea22a3d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fea141b1c40 con 0x562dea9b81b0 2024-05-27T07:43:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea22a3d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fea141afa30 con 0x562dea9974d0 2024-05-27T07:43:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea19928640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 0x562dead59870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea19928640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 0x562dead59870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57530/0 (socket says 172.21.15.114:57530) 2024-05-27T07:43:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea19928640 1 -- 172.21.15.114:0/1831364907 learned_addr learned my addr 172.21.15.114:0/1831364907 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea19127640 1 --2- 172.21.15.114:0/1831364907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea9974d0 0x562dead70560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea1a129640 1 --2- 172.21.15.114:0/1831364907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9b81b0 0x562dead69770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea19928640 1 -- 172.21.15.114:0/1831364907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea9974d0 msgr2=0x562dead70560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea19928640 1 --2- 172.21.15.114:0/1831364907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea9974d0 0x562dead70560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea19928640 1 -- 172.21.15.114:0/1831364907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9b81b0 msgr2=0x562dead69770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea19928640 1 --2- 172.21.15.114:0/1831364907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9b81b0 0x562dead69770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea19928640 1 -- 172.21.15.114:0/1831364907 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dead56aa0 con 0x562dea98f2f0 2024-05-27T07:43:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.380+0000 7fea1a129640 1 --2- 172.21.15.114:0/1831364907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9b81b0 0x562dead69770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:43:55.382 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.381+0000 7fea19928640 1 --2- 172.21.15.114:0/1831364907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 0x562dead59870 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fea10012870 tx=0x7fea10012990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f1b5cd11e93d1a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:55.382 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.381+0000 7fe9ff7fe640 1 -- 172.21.15.114:0/1831364907 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea100120d0 con 0x562dea98f2f0 2024-05-27T07:43:55.382 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.381+0000 7fe9ff7fe640 1 -- 172.21.15.114:0/1831364907 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea10012270 con 0x562dea98f2f0 2024-05-27T07:43:55.382 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.381+0000 7fe9ff7fe640 1 -- 172.21.15.114:0/1831364907 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea10010430 con 0x562dea98f2f0 2024-05-27T07:43:55.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.382+0000 7fea22a3d740 1 -- 172.21.15.114:0/1831364907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 msgr2=0x562dead59870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.382+0000 7fea22a3d740 1 --2- 172.21.15.114:0/1831364907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 0x562dead59870 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fea10012870 tx=0x7fea10012990 comp rx=0 tx=0).stop 2024-05-27T07:43:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.382+0000 7fea22a3d740 1 -- 172.21.15.114:0/1831364907 shutdown_connections 2024-05-27T07:43:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.382+0000 7fea22a3d740 1 --2- 172.21.15.114:0/1831364907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9b81b0 0x562dead69770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.382+0000 7fea22a3d740 1 --2- 172.21.15.114:0/1831364907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea9974d0 0x562dead70560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.382+0000 7fea22a3d740 1 --2- 172.21.15.114:0/1831364907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea98f2f0 0x562dead59870 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.382+0000 7fea22a3d740 1 -- 172.21.15.114:0/1831364907 >> 172.21.15.114:0/1831364907 conn(0x562dea9ac300 msgr2=0x562dead59160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.382+0000 7fea22a3d740 1 -- 172.21.15.114:0/1831364907 shutdown_connections 2024-05-27T07:43:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.382+0000 7fea22a3d740 1 -- 172.21.15.114:0/1831364907 wait complete. 2024-05-27T07:43:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.382+0000 7fea22a3d740 1 -- start start 2024-05-27T07:43:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.382+0000 7fea22a3d740 1 Processor -- start 2024-05-27T07:43:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea22a3d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea98f2f0 0x562dead6ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea22a3d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 0x562dead6b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea22a3d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 0x562dead6b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea22a3d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fea141b0100 con 0x562dea9b81b0 2024-05-27T07:43:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea22a3d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fea141b1570 con 0x562dea9974d0 2024-05-27T07:43:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea22a3d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fea141b07d0 con 0x562dea98f2f0 2024-05-27T07:43:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea22a3d740 10 client.0 _ll_register_callbacks cb 0x7fea10009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea1a129640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 0x562dead6b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea1a129640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 0x562dead6b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57538/0 (socket says 172.21.15.114:57538) 2024-05-27T07:43:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea1a129640 1 -- 172.21.15.114:0/2312367725 learned_addr learned my addr 172.21.15.114:0/2312367725 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea19928640 1 --2- 172.21.15.114:0/2312367725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea98f2f0 0x562dead6ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea19127640 1 --2- 172.21.15.114:0/2312367725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 0x562dead6b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea1a129640 1 -- 172.21.15.114:0/2312367725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea98f2f0 msgr2=0x562dead6ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea1a129640 1 --2- 172.21.15.114:0/2312367725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea98f2f0 0x562dead6ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea1a129640 1 -- 172.21.15.114:0/2312367725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 msgr2=0x562dead6b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea1a129640 1 --2- 172.21.15.114:0/2312367725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 0x562dead6b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea1a129640 1 -- 172.21.15.114:0/2312367725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562deaddc660 con 0x562dea9b81b0 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea1a129640 1 --2- 172.21.15.114:0/2312367725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 0x562dead6b6f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00f790 tx=0x7fea0c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d668e49fab09c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fe9fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fe9fd7fa640 1 -- 172.21.15.114:0/2312367725 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c038020 con 0x562dea9b81b0 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fe9fd7fa640 1 -- 172.21.15.114:0/2312367725 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fea0c003a90 con 0x562dea9b81b0 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.383+0000 7fea22a3d740 1 -- 172.21.15.114:0/2312367725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea10012270 con 0x562dea9b81b0 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.384+0000 7fe9fd7fa640 1 -- 172.21.15.114:0/2312367725 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fea0c043050 con 0x562dea9b81b0 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.384+0000 7fea22a3d740 10 client.16299 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.384+0000 7fea22a3d740 1 -- 172.21.15.114:0/2312367725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562deaddd820 con 0x562dea9b81b0 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.384+0000 7fe9fd7fa640 1 -- 172.21.15.114:0/2312367725 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea10012270 con 0x562dea9b81b0 2024-05-27T07:43:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.384+0000 7fe9fd7fa640 1 -- 172.21.15.114:0/2312367725 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fea0c033070 con 0x562dea9b81b0 2024-05-27T07:43:55.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.384+0000 7fea22a3d740 10 client.16299 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:55.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.384+0000 7fea22a3d740 10 client.16299 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:55.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.384+0000 7fea22a3d740 20 client.16299 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:55.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.384+0000 7fea22a3d740 1 --2- 172.21.15.114:0/2312367725 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562dea94c1c0 0x562deaddfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:55.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.384+0000 7fea22a3d740 4 client.16299 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:43:55.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.384+0000 7fea22a3d740 1 -- 172.21.15.114:0/2312367725 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562deacc7360 con 0x562dea94c1c0 2024-05-27T07:43:55.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.384+0000 7fea19928640 1 --2- 172.21.15.114:0/2312367725 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562dea94c1c0 0x562deaddfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:55.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.385+0000 7fe9fd7fa640 10 client.16299 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:55.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.385+0000 7fea19928640 1 --2- 172.21.15.114:0/2312367725 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562dea94c1c0 0x562deaddfe90 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=889eb10fb27c74a9 server_cookie=8c1c5f93851e7666 in_seq=0 out_seq=0 2024-05-27T07:43:55.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:55 smithi114 ceph-mon[32038]: pgmap v132: 129 pgs: 129 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 406 op/s 2024-05-27T07:43:55.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:55 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:43:55.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:55 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:55.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:55 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:43:55.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:55 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:55.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:43:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:43:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:43:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:43:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.666+0000 7fe9fd7fa640 1 -- 172.21.15.114:0/2312367725 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562deacc7360 con 0x562dea94c1c0 2024-05-27T07:43:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.666+0000 7fe9fd7fa640 10 client.16299 handle_command_reply: tid=0 2024-05-27T07:43:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.666+0000 7fea22a3d740 1 client.16299 shutdown 2024-05-27T07:43:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.666+0000 7fea22a3d740 1 -- 172.21.15.114:0/2312367725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 msgr2=0x562dead6b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.666+0000 7fea22a3d740 1 --2- 172.21.15.114:0/2312367725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 0x562dead6b6f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00f790 tx=0x7fea0c018000 comp rx=0 tx=0).stop 2024-05-27T07:43:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.667+0000 7fea22a3d740 1 -- 172.21.15.114:0/2312367725 shutdown_connections 2024-05-27T07:43:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.667+0000 7fea22a3d740 1 --2- 172.21.15.114:0/2312367725 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562dea94c1c0 0x562deaddfe90 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.667+0000 7fea22a3d740 1 --2- 172.21.15.114:0/2312367725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dea9b81b0 0x562dead6b6f0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.667+0000 7fea22a3d740 1 --2- 172.21.15.114:0/2312367725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dea9974d0 0x562dead6b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.667+0000 7fea22a3d740 1 --2- 172.21.15.114:0/2312367725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562dea98f2f0 0x562dead6ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.667+0000 7fea22a3d740 1 -- 172.21.15.114:0/2312367725 >> 172.21.15.114:0/2312367725 conn(0x562dea9ac300 msgr2=0x562dead65870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.667+0000 7fe9fd7fa640 0 client.16299 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.667+0000 7fea22a3d740 1 -- 172.21.15.114:0/2312367725 shutdown_connections 2024-05-27T07:43:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.667+0000 7fea22a3d740 1 -- 172.21.15.114:0/2312367725 wait complete. 2024-05-27T07:43:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.667+0000 7fea22a3d740 20 client.16299 trim_cache size 0 max 16384 2024-05-27T07:43:55.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.669+0000 7fea1bb8c640 1 -- 172.21.15.114:0/3353849743 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fea000b2950 msgr2=0x7fea000b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:55.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.669+0000 7fea1bb8c640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fea000b2950 0x7fea000b4e60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fea0c01b000 tx=0x7fea0c01a700 comp rx=0 tx=0).stop 2024-05-27T07:43:55.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.669+0000 7fea1bb8c640 1 -- 172.21.15.114:0/3353849743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea14108b90 msgr2=0x7fea141a0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:55.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.669+0000 7fea1bb8c640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea14108b90 0x7fea141a0b20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fea04005ab0 tx=0x7fea04003b90 comp rx=0 tx=0).stop 2024-05-27T07:43:55.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.670+0000 7fea1bb8c640 1 -- 172.21.15.114:0/3353849743 shutdown_connections 2024-05-27T07:43:55.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.670+0000 7fea1bb8c640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fea000b2950 0x7fea000b4e60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.670+0000 7fea1bb8c640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea14109030 0x7fea1419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.670+0000 7fea1bb8c640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea14108b90 0x7fea141a0b20 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.670+0000 7fea1bb8c640 1 --2- 172.21.15.114:0/3353849743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea141032e0 0x7fea141a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.670+0000 7fea1bb8c640 1 -- 172.21.15.114:0/3353849743 >> 172.21.15.114:0/3353849743 conn(0x7fea140782d0 msgr2=0x7fea14106240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.671+0000 7fea1bb8c640 1 -- 172.21.15.114:0/3353849743 shutdown_connections 2024-05-27T07:43:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:43:55.671+0000 7fea1bb8c640 1 -- 172.21.15.114:0/3353849743 wait complete. 2024-05-27T07:43:55.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:55 smithi163 ceph-mon[36451]: pgmap v132: 129 pgs: 129 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 406 op/s 2024-05-27T07:43:55.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:55 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:43:55.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:55 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:55.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:55 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:43:55.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:55 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:55.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:55 smithi186 ceph-mon[36647]: pgmap v132: 129 pgs: 129 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 406 op/s 2024-05-27T07:43:55.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:55 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:43:55.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:55 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:55.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:55 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:43:55.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:55 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:43:55.913 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:56.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 330286 14.07 MB/sec execute 8 sec latency 162.789 ms 2024-05-27T07:43:57.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 332476 13.40 MB/sec execute 9 sec latency 180.143 ms 2024-05-27T07:43:57.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:57 smithi114 ceph-mon[32038]: pgmap v133: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.9 MiB/s wr, 545 op/s 2024-05-27T07:43:57.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:57 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:43:57.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:57 smithi163 ceph-mon[36451]: pgmap v133: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.9 MiB/s wr, 545 op/s 2024-05-27T07:43:57.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:57 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:43:57.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:57 smithi186 ceph-mon[36647]: pgmap v133: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.9 MiB/s wr, 545 op/s 2024-05-27T07:43:57.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:57 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:43:57.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.815+0000 7fa7cc699700 1 -- 172.21.15.114:0/3194436326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c4105490 msgr2=0x7fa7c41058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:57.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.815+0000 7fa7cc699700 1 --2- 172.21.15.114:0/3194436326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c4105490 0x7fa7c41058a0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009990 tx=0x7fa7c0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.816+0000 7fa7cc699700 1 -- 172.21.15.114:0/3194436326 shutdown_connections 2024-05-27T07:43:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.816+0000 7fa7cc699700 1 --2- 172.21.15.114:0/3194436326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7c410e900 0x7fa7c4110e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.816+0000 7fa7cc699700 1 --2- 172.21.15.114:0/3194436326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c4105490 0x7fa7c41058a0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.816+0000 7fa7cc699700 1 --2- 172.21.15.114:0/3194436326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7c4105de0 0x7fa7c410e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.816+0000 7fa7cc699700 1 -- 172.21.15.114:0/3194436326 >> 172.21.15.114:0/3194436326 conn(0x7fa7c4078e40 msgr2=0x7fa7c4079280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.816+0000 7fa7cc699700 1 -- 172.21.15.114:0/3194436326 shutdown_connections 2024-05-27T07:43:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.816+0000 7fa7cc699700 1 -- 172.21.15.114:0/3194436326 wait complete. 2024-05-27T07:43:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.816+0000 7fa7cc699700 1 Processor -- start 2024-05-27T07:43:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.816+0000 7fa7cc699700 1 -- start start 2024-05-27T07:43:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7cc699700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7c4105490 0x7fa7c419aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7cc699700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7c4105de0 0x7fa7c419b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7cc699700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c410e900 0x7fa7c419f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7cc699700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c41144b0 con 0x7fa7c410e900 2024-05-27T07:43:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7cc699700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c4114330 con 0x7fa7c4105490 2024-05-27T07:43:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7cc699700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c4114630 con 0x7fa7c4105de0 2024-05-27T07:43:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7cac36700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c410e900 0x7fa7c419f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7cac36700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c410e900 0x7fa7c419f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57554/0 (socket says 172.21.15.114:57554) 2024-05-27T07:43:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7c9c34700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7c4105de0 0x7fa7c419b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7ca435700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7c4105490 0x7fa7c419aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7cac36700 1 -- 172.21.15.114:0/631157519 learned_addr learned my addr 172.21.15.114:0/631157519 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7c9c34700 1 -- 172.21.15.114:0/631157519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7c4105490 msgr2=0x7fa7c419aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7c9c34700 1 --2- 172.21.15.114:0/631157519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7c4105490 0x7fa7c419aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7c9c34700 1 -- 172.21.15.114:0/631157519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c410e900 msgr2=0x7fa7c419f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7c9c34700 1 --2- 172.21.15.114:0/631157519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c410e900 0x7fa7c419f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7c9c34700 1 -- 172.21.15.114:0/631157519 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c419fd70 con 0x7fa7c4105de0 2024-05-27T07:43:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7cac36700 1 --2- 172.21.15.114:0/631157519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c410e900 0x7fa7c419f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.817+0000 7fa7c9c34700 1 --2- 172.21.15.114:0/631157519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7c4105de0 0x7fa7c419b440 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400e9c0 tx=0x7fa7b400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.818+0000 7fa7bb7fe700 1 -- 172.21.15.114:0/631157519 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b400c9c0 con 0x7fa7c4105de0 2024-05-27T07:43:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.818+0000 7fa7bb7fe700 1 -- 172.21.15.114:0/631157519 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7b400cb60 con 0x7fa7c4105de0 2024-05-27T07:43:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.818+0000 7fa7cc699700 1 -- 172.21.15.114:0/631157519 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7c41a0060 con 0x7fa7c4105de0 2024-05-27T07:43:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.818+0000 7fa7bb7fe700 1 -- 172.21.15.114:0/631157519 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4018680 con 0x7fa7c4105de0 2024-05-27T07:43:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.818+0000 7fa7cc699700 1 -- 172.21.15.114:0/631157519 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7c41a7980 con 0x7fa7c4105de0 2024-05-27T07:43:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.820+0000 7fa7bb7fe700 1 -- 172.21.15.114:0/631157519 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa7b4018820 con 0x7fa7c4105de0 2024-05-27T07:43:57.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.821+0000 7fa7bb7fe700 1 --2- 172.21.15.114:0/631157519 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7b00b29f0 0x7fa7b00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.821+0000 7fa7ca435700 1 --2- 172.21.15.114:0/631157519 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7b00b29f0 0x7fa7b00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.821+0000 7fa7db0aeb80 1 Processor -- start 2024-05-27T07:43:57.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.821+0000 7fa7bb7fe700 1 -- 172.21.15.114:0/631157519 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa7b40e2be0 con 0x7fa7c4105de0 2024-05-27T07:43:57.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7db0aeb80 1 -- start start 2024-05-27T07:43:57.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7ca435700 1 --2- 172.21.15.114:0/631157519 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7b00b29f0 0x7fa7b00b4f00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0006010 tx=0x7fa7c0005960 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7db0aeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d858e40 0x56079d9dc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7db0aeb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9dcc50 0x56079d9d6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7db0aeb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d9d6f80 0x56079d9d7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7db0aeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56079d905c90 con 0x56079d858e40 2024-05-27T07:43:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7db0aeb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56079d8f5830 con 0x56079d9d6f80 2024-05-27T07:43:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7db0aeb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56079d8f9870 con 0x56079d9dcc50 2024-05-27T07:43:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7ca435700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d858e40 0x56079d9dc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7ca435700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d858e40 0x56079d9dc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57556/0 (socket says 172.21.15.114:57556) 2024-05-27T07:43:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7cac36700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d9d6f80 0x56079d9d7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7ca435700 1 -- 172.21.15.114:0/952663610 learned_addr learned my addr 172.21.15.114:0/952663610 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:57.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7c9c34700 1 --2- 172.21.15.114:0/952663610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9dcc50 0x56079d9d6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7cac36700 1 -- 172.21.15.114:0/952663610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9dcc50 msgr2=0x56079d9d6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7cac36700 1 --2- 172.21.15.114:0/952663610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9dcc50 0x56079d9d6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7cac36700 1 -- 172.21.15.114:0/952663610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d858e40 msgr2=0x56079d9dc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7cac36700 1 --2- 172.21.15.114:0/952663610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d858e40 0x56079d9dc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.822+0000 7fa7cac36700 1 -- 172.21.15.114:0/952663610 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56079d8458d0 con 0x56079d9d6f80 2024-05-27T07:43:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.823+0000 7fa7cac36700 1 --2- 172.21.15.114:0/952663610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d9d6f80 0x56079d9d7390 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009430 tx=0x7fa7bc009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b85e0651686c525d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.823+0000 7fa7b8ff9700 1 -- 172.21.15.114:0/952663610 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc019070 con 0x56079d9d6f80 2024-05-27T07:43:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.823+0000 7fa7b8ff9700 1 -- 172.21.15.114:0/952663610 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7bc004db0 con 0x56079d9d6f80 2024-05-27T07:43:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.823+0000 7fa7b8ff9700 1 -- 172.21.15.114:0/952663610 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc0053e0 con 0x56079d9d6f80 2024-05-27T07:43:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.824+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/952663610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d9d6f80 msgr2=0x56079d9d7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.824+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/952663610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d9d6f80 0x56079d9d7390 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009430 tx=0x7fa7bc009920 comp rx=0 tx=0).stop 2024-05-27T07:43:57.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.824+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/952663610 shutdown_connections 2024-05-27T07:43:57.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.824+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/952663610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d9d6f80 0x56079d9d7390 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.824+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/952663610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d858e40 0x56079d9dc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.824+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/952663610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9dcc50 0x56079d9d6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.824+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/952663610 >> 172.21.15.114:0/952663610 conn(0x56079d8b3af0 msgr2=0x56079d9d1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:57.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.824+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/952663610 shutdown_connections 2024-05-27T07:43:57.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.824+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/952663610 wait complete. 2024-05-27T07:43:57.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.824+0000 7fa7db0aeb80 1 -- start start 2024-05-27T07:43:57.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.824+0000 7fa7db0aeb80 1 Processor -- start 2024-05-27T07:43:57.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7db0aeb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d858e40 0x56079d9e0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7db0aeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9d6f80 0x56079d9e0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7db0aeb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d9dcc50 0x56079d9e0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7db0aeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56079d7fbfa0 con 0x56079d9d6f80 2024-05-27T07:43:57.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7db0aeb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56079d8977c0 con 0x56079d9dcc50 2024-05-27T07:43:57.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7db0aeb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56079d9e3740 con 0x56079d858e40 2024-05-27T07:43:57.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7db0aeb80 10 client.0 _ll_register_callbacks cb 0x56079d890920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:57.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7c9c34700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9d6f80 0x56079d9e0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7c9c34700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9d6f80 0x56079d9e0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57566/0 (socket says 172.21.15.114:57566) 2024-05-27T07:43:57.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7c9c34700 1 -- 172.21.15.114:0/856972609 learned_addr learned my addr 172.21.15.114:0/856972609 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:57.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7cac36700 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d9dcc50 0x56079d9e0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7ca435700 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d858e40 0x56079d9e0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7cac36700 1 -- 172.21.15.114:0/856972609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d858e40 msgr2=0x56079d9e0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:57.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7cac36700 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d858e40 0x56079d9e0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7cac36700 1 -- 172.21.15.114:0/856972609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9d6f80 msgr2=0x56079d9e0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:57.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7cac36700 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9d6f80 0x56079d9e0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7cac36700 1 -- 172.21.15.114:0/856972609 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56079d894690 con 0x56079d9dcc50 2024-05-27T07:43:57.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7c9c34700 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9d6f80 0x56079d9e0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:57.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7cac36700 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d9dcc50 0x56079d9e0da0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000f80 tx=0x7fa7bc00ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=833932b3337c92e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:57.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.825+0000 7fa7adffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:43:57.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7adffb700 1 -- 172.21.15.114:0/856972609 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc02b060 con 0x56079d9dcc50 2024-05-27T07:43:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7adffb700 1 -- 172.21.15.114:0/856972609 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa7bc026410 con 0x56079d9dcc50 2024-05-27T07:43:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/856972609 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7bc004db0 con 0x56079d9dcc50 2024-05-27T07:43:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7adffb700 1 -- 172.21.15.114:0/856972609 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa7bc032050 con 0x56079d9dcc50 2024-05-27T07:43:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7db0aeb80 10 client.25832 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/856972609 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56079d9ee190 con 0x56079d9dcc50 2024-05-27T07:43:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7adffb700 1 -- 172.21.15.114:0/856972609 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7bc004db0 con 0x56079d9dcc50 2024-05-27T07:43:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7adffb700 1 -- 172.21.15.114:0/856972609 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fa7bc01d490 con 0x56079d9dcc50 2024-05-27T07:43:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7db0aeb80 10 client.25832 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7db0aeb80 10 client.25832 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7db0aeb80 20 client.25832 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56079d9f0710 0x56079d9f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7db0aeb80 4 client.25832 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:57.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.826+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/856972609 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56079d9a29f0 con 0x56079d9f0710 2024-05-27T07:43:57.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.827+0000 7fa7ca435700 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56079d9f0710 0x56079d9f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.827+0000 7fa7adffb700 10 client.25832 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.827+0000 7fa7ca435700 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56079d9f0710 0x56079d9f2b30 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb0765d0f940c229 server_cookie=a149fa2d7a6a0aa3 in_seq=0 out_seq=0 2024-05-27T07:43:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7adffb700 1 -- 172.21.15.114:0/856972609 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56079d9a29f0 con 0x56079d9f0710 2024-05-27T07:43:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7adffb700 10 client.25832 handle_command_reply: tid=0 2024-05-27T07:43:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7db0aeb80 1 client.25832 shutdown 2024-05-27T07:43:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/856972609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d9dcc50 msgr2=0x56079d9e0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:57.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d9dcc50 0x56079d9e0da0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000f80 tx=0x7fa7bc00ec00 comp rx=0 tx=0).stop 2024-05-27T07:43:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/856972609 shutdown_connections 2024-05-27T07:43:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56079d9f0710 0x56079d9f2b30 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d9dcc50 0x56079d9e0da0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d858e40 0x56079d9e0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7adffb700 0 client.25832 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/856972609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9d6f80 0x56079d9e0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/856972609 >> 172.21.15.114:0/856972609 conn(0x56079d8b3af0 msgr2=0x56079d9d5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/856972609 shutdown_connections 2024-05-27T07:43:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/856972609 wait complete. 2024-05-27T07:43:57.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.828+0000 7fa7db0aeb80 20 client.25832 trim_cache size 0 max 16384 2024-05-27T07:43:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.849+0000 7fa7db0aeb80 1 Processor -- start 2024-05-27T07:43:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.849+0000 7fa7db0aeb80 1 -- start start 2024-05-27T07:43:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.849+0000 7fa7db0aeb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d858e40 0x56079d9ec7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.849+0000 7fa7db0aeb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9d6f80 0x56079d9e1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.849+0000 7fa7db0aeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 0x56079d9e2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.849+0000 7fa7db0aeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c41ae670 con 0x56079d9dcc50 2024-05-27T07:43:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.849+0000 7fa7db0aeb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c41b0ef0 con 0x56079d858e40 2024-05-27T07:43:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.849+0000 7fa7db0aeb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c41aece0 con 0x56079d9d6f80 2024-05-27T07:43:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.849+0000 7fa7cac36700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 0x56079d9e2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.849+0000 7fa7cac36700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 0x56079d9e2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57580/0 (socket says 172.21.15.114:57580) 2024-05-27T07:43:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.849+0000 7fa7cac36700 1 -- 172.21.15.114:0/1770713212 learned_addr learned my addr 172.21.15.114:0/1770713212 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.849+0000 7fa7ca435700 1 --2- 172.21.15.114:0/1770713212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d858e40 0x56079d9ec7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.849+0000 7fa7c9c34700 1 --2- 172.21.15.114:0/1770713212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9d6f80 0x56079d9e1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.850+0000 7fa7cac36700 1 -- 172.21.15.114:0/1770713212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9d6f80 msgr2=0x56079d9e1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.850+0000 7fa7cac36700 1 --2- 172.21.15.114:0/1770713212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9d6f80 0x56079d9e1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.850+0000 7fa7cac36700 1 -- 172.21.15.114:0/1770713212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d858e40 msgr2=0x56079d9ec7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.850+0000 7fa7cac36700 1 --2- 172.21.15.114:0/1770713212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d858e40 0x56079d9ec7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.850+0000 7fa7cac36700 1 -- 172.21.15.114:0/1770713212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56079d9de8b0 con 0x56079d9dcc50 2024-05-27T07:43:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.850+0000 7fa7ca435700 1 --2- 172.21.15.114:0/1770713212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d858e40 0x56079d9ec7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.850+0000 7fa7c9c34700 1 --2- 172.21.15.114:0/1770713212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9d6f80 0x56079d9e1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.850+0000 7fa7cac36700 1 --2- 172.21.15.114:0/1770713212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 0x56079d9e2400 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc002230 tx=0x7fa7bc007be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa106db790e7592d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.850+0000 7fa7affff700 1 -- 172.21.15.114:0/1770713212 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc02b070 con 0x56079d9dcc50 2024-05-27T07:43:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.850+0000 7fa7affff700 1 -- 172.21.15.114:0/1770713212 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7bc01d400 con 0x56079d9dcc50 2024-05-27T07:43:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.850+0000 7fa7affff700 1 -- 172.21.15.114:0/1770713212 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc019040 con 0x56079d9dcc50 2024-05-27T07:43:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.851+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/1770713212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 msgr2=0x56079d9e2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.851+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/1770713212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 0x56079d9e2400 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc002230 tx=0x7fa7bc007be0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.851+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/1770713212 shutdown_connections 2024-05-27T07:43:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.851+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/1770713212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 0x56079d9e2400 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.851+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/1770713212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d858e40 0x56079d9ec7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.851+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/1770713212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9d6f80 0x56079d9e1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.851+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/1770713212 >> 172.21.15.114:0/1770713212 conn(0x56079d8b3af0 msgr2=0x56079d9e17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.851+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/1770713212 shutdown_connections 2024-05-27T07:43:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.851+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/1770713212 wait complete. 2024-05-27T07:43:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.851+0000 7fa7db0aeb80 1 -- start start 2024-05-27T07:43:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7db0aeb80 1 Processor -- start 2024-05-27T07:43:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7db0aeb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d858e40 0x56079d9e8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7db0aeb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9d6f80 0x56079d9e6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7db0aeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 0x56079d9e68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7db0aeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c41af3b0 con 0x56079d9dcc50 2024-05-27T07:43:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7db0aeb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c41b0820 con 0x56079d858e40 2024-05-27T07:43:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7db0aeb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c41afa80 con 0x56079d9d6f80 2024-05-27T07:43:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7db0aeb80 10 client.0 _ll_register_callbacks cb 0x56079d890920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7cac36700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 0x56079d9e68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7cac36700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 0x56079d9e68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57588/0 (socket says 172.21.15.114:57588) 2024-05-27T07:43:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7cac36700 1 -- 172.21.15.114:0/1672440441 learned_addr learned my addr 172.21.15.114:0/1672440441 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:43:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7ca435700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d858e40 0x56079d9e8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7c9c34700 1 --2- 172.21.15.114:0/1672440441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9d6f80 0x56079d9e6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7cac36700 1 -- 172.21.15.114:0/1672440441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9d6f80 msgr2=0x56079d9e6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7cac36700 1 --2- 172.21.15.114:0/1672440441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9d6f80 0x56079d9e6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7cac36700 1 -- 172.21.15.114:0/1672440441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d858e40 msgr2=0x56079d9e8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7cac36700 1 --2- 172.21.15.114:0/1672440441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d858e40 0x56079d9e8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7cac36700 1 -- 172.21.15.114:0/1672440441 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56079d9e6f40 con 0x56079d9dcc50 2024-05-27T07:43:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.852+0000 7fa7ca435700 1 --2- 172.21.15.114:0/1672440441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d858e40 0x56079d9e8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:43:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7cac36700 1 --2- 172.21.15.114:0/1672440441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 0x56079d9e68a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc01d860 tx=0x7fa7bc00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e918ef892419ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7adffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:43:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7adffb700 1 -- 172.21.15.114:0/1672440441 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc03d070 con 0x56079d9dcc50 2024-05-27T07:43:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7adffb700 1 -- 172.21.15.114:0/1672440441 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa7bc026d90 con 0x56079d9dcc50 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/1672440441 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7bc01d400 con 0x56079d9dcc50 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7adffb700 1 -- 172.21.15.114:0/1672440441 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa7bc046050 con 0x56079d9dcc50 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7db0aeb80 10 client.16317 fetch_fsmap learned FSMap version 29 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/1672440441 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56079da5f240 con 0x56079d9dcc50 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7adffb700 1 -- 172.21.15.114:0/1672440441 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7bc01d400 con 0x56079d9dcc50 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7adffb700 1 -- 172.21.15.114:0/1672440441 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fa7bc040020 con 0x56079d9dcc50 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7db0aeb80 10 client.16317 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7db0aeb80 10 client.16317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7db0aeb80 20 client.16317 populate_metadata read hostname 'smithi114' 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/1672440441 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56079da5f890 0x56079da61cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7db0aeb80 4 client.16317 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.853+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/1672440441 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56079d907be0 con 0x56079da5f890 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7ca435700 1 --2- 172.21.15.114:0/1672440441 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56079da5f890 0x56079da61cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7ca435700 1 --2- 172.21.15.114:0/1672440441 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56079da5f890 0x56079da61cb0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b51c3356486ee178 server_cookie=9eb3ecba72284bbe in_seq=0 out_seq=0 2024-05-27T07:43:57.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7adffb700 10 client.16317 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7adffb700 1 -- 172.21.15.114:0/1672440441 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56079d907be0 con 0x56079da5f890 2024-05-27T07:43:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7adffb700 10 client.16317 handle_command_reply: tid=0 2024-05-27T07:43:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7db0aeb80 1 client.16317 shutdown 2024-05-27T07:43:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/1672440441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 msgr2=0x56079d9e68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/1672440441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 0x56079d9e68a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc01d860 tx=0x7fa7bc00ef00 comp rx=0 tx=0).stop 2024-05-27T07:43:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/1672440441 shutdown_connections 2024-05-27T07:43:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/1672440441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56079d9dcc50 0x56079d9e68a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/1672440441 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56079da5f890 0x56079da61cb0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/1672440441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56079d858e40 0x56079d9e8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7db0aeb80 1 --2- 172.21.15.114:0/1672440441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56079d9d6f80 0x56079d9e6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.854+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/1672440441 >> 172.21.15.114:0/1672440441 conn(0x56079d8b3af0 msgr2=0x56079d9dba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:57.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7adffb700 0 client.16317 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:43:57.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/1672440441 shutdown_connections 2024-05-27T07:43:57.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7db0aeb80 1 -- 172.21.15.114:0/1672440441 wait complete. 2024-05-27T07:43:57.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7db0aeb80 20 client.16317 trim_cache size 0 max 16384 2024-05-27T07:43:57.862 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:43:57.862 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:43:57.862 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:43:57.862 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:43:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7cc699700 1 -- 172.21.15.114:0/631157519 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7b00b29f0 msgr2=0x7fa7b00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7cc699700 1 --2- 172.21.15.114:0/631157519 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7b00b29f0 0x7fa7b00b4f00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0006010 tx=0x7fa7c0005960 comp rx=0 tx=0).stop 2024-05-27T07:43:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7cc699700 1 -- 172.21.15.114:0/631157519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7c4105de0 msgr2=0x7fa7c419b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7cc699700 1 --2- 172.21.15.114:0/631157519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7c4105de0 0x7fa7c419b440 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400e9c0 tx=0x7fa7b400ecd0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7cc699700 1 -- 172.21.15.114:0/631157519 shutdown_connections 2024-05-27T07:43:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7cc699700 1 --2- 172.21.15.114:0/631157519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c410e900 0x7fa7c419f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7cc699700 1 --2- 172.21.15.114:0/631157519 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7b00b29f0 0x7fa7b00b4f00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7cc699700 1 --2- 172.21.15.114:0/631157519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7c4105490 0x7fa7c419aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7cc699700 1 --2- 172.21.15.114:0/631157519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7c4105de0 0x7fa7c419b440 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:57.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.855+0000 7fa7cc699700 1 -- 172.21.15.114:0/631157519 >> 172.21.15.114:0/631157519 conn(0x7fa7c4078e40 msgr2=0x7fa7c4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:57.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.856+0000 7fa7cc699700 1 -- 172.21.15.114:0/631157519 shutdown_connections 2024-05-27T07:43:57.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:43:57.856+0000 7fa7cc699700 1 -- 172.21.15.114:0/631157519 wait complete. 2024-05-27T07:43:58.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 334660 13.38 MB/sec execute 10 sec latency 29.789 ms 2024-05-27T07:43:58.811 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:43:58.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:43:59.117 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:59.243 INFO:tasks.workunit.client.0.smithi114.stdout: 1 337565 13.71 MB/sec execute 11 sec latency 26.682 ms 2024-05-27T07:43:59.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:59 smithi114 ceph-mon[32038]: pgmap v134: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 259 op/s 2024-05-27T07:43:59.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:43:59 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:43:59.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:59 smithi163 ceph-mon[36451]: pgmap v134: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 259 op/s 2024-05-27T07:43:59.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:43:59 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:43:59.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:59 smithi186 ceph-mon[36647]: pgmap v134: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 259 op/s 2024-05-27T07:43:59.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:43:59 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:44:00.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 340598 14.04 MB/sec execute 12 sec latency 23.003 ms 2024-05-27T07:44:00.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:00 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:44:00.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:00 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:44:00.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:00 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:44:00.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:44:00.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:44:00.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:44:00.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:00 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:44:00.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:00 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:44:00.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:00 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:44:01.244 INFO:tasks.workunit.client.0.smithi114.stdout: 1 341547 13.18 MB/sec execute 13 sec latency 58.198 ms 2024-05-27T07:44:01.358 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0d1347f700 time 2024-05-27T07:44:00.983068+0000 2024-05-27T07:44:01.358 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:44:01.358 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:01.358 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0d25cc4ba6] 2024-05-27T07:44:01.358 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0d25cc4dd1] 2024-05-27T07:44:01.358 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560b223aa23d] 2024-05-27T07:44:01.358 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 4: (SnapRealm::build_snap_set() const+0x323) [0x560b22395971] 2024-05-27T07:44:01.358 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 5: (SnapRealm::check_cache() const+0x13e) [0x560b22393fa0] 2024-05-27T07:44:01.358 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560b22394651] 2024-05-27T07:44:01.359 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560b22329347] 2024-05-27T07:44:01.359 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560b2202587e] 2024-05-27T07:44:01.359 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560b2202ae94] 2024-05-27T07:44:01.359 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560b22035969] 2024-05-27T07:44:01.359 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 11: (MDLog::_replay_thread()+0x1f38) [0x560b223ce48c] 2024-05-27T07:44:01.359 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 12: (MDLog::ReplayThread::entry()+0x11) [0x560b223d79cb] 2024-05-27T07:44:01.359 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 13: (Thread::entry_wrapper()+0x43) [0x7f0d25c9372d] 2024-05-27T07:44:01.359 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0d25c93749] 2024-05-27T07:44:01.359 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0d248371ca] 2024-05-27T07:44:01.359 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 16: clone() 2024-05-27T07:44:01.359 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:00.984+0000 7f0d1347f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0d1347f700 time 2024-05-27T07:44:00.983068+0000 2024-05-27T07:44:01.360 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:44:01.360 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:01.360 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:01.360 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0d25cc4ba6] 2024-05-27T07:44:01.360 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0d25cc4dd1] 2024-05-27T07:44:01.360 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560b223aa23d] 2024-05-27T07:44:01.360 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 4: (SnapRealm::build_snap_set() const+0x323) [0x560b22395971] 2024-05-27T07:44:01.360 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 5: (SnapRealm::check_cache() const+0x13e) [0x560b22393fa0] 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560b22394651] 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560b22329347] 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560b2202587e] 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560b2202ae94] 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560b22035969] 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 11: (MDLog::_replay_thread()+0x1f38) [0x560b223ce48c] 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 12: (MDLog::ReplayThread::entry()+0x11) [0x560b223d79cb] 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 13: (Thread::entry_wrapper()+0x43) [0x7f0d25c9372d] 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0d25c93749] 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0d248371ca] 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 16: clone() 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: *** Caught signal (Aborted) ** 2024-05-27T07:44:01.361 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: in thread 7f0d1347f700 thread_name:md_log_replay 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0d24841d20] 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2: gsignal() 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 3: abort() 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0d25cc4ced] 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0d25cc4dd1] 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560b223aa23d] 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 7: (SnapRealm::build_snap_set() const+0x323) [0x560b22395971] 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 8: (SnapRealm::check_cache() const+0x13e) [0x560b22393fa0] 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560b22394651] 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560b22329347] 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560b2202587e] 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560b2202ae94] 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560b22035969] 2024-05-27T07:44:01.362 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 14: (MDLog::_replay_thread()+0x1f38) [0x560b223ce48c] 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 15: (MDLog::ReplayThread::entry()+0x11) [0x560b223d79cb] 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 16: (Thread::entry_wrapper()+0x43) [0x7f0d25c9372d] 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0d25c93749] 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0d248371ca] 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 19: clone() 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:00.987+0000 7f0d1347f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: in thread 7f0d1347f700 thread_name:md_log_replay 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0d24841d20] 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2: gsignal() 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 3: abort() 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0d25cc4ced] 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0d25cc4dd1] 2024-05-27T07:44:01.363 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560b223aa23d] 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 7: (SnapRealm::build_snap_set() const+0x323) [0x560b22395971] 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 8: (SnapRealm::check_cache() const+0x13e) [0x560b22393fa0] 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560b22394651] 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560b22329347] 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560b2202587e] 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560b2202ae94] 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560b22035969] 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 14: (MDLog::_replay_thread()+0x1f38) [0x560b223ce48c] 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 15: (MDLog::ReplayThread::entry()+0x11) [0x560b223d79cb] 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 16: (Thread::entry_wrapper()+0x43) [0x7f0d25c9372d] 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0d25c93749] 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0d248371ca] 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 19: clone() 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:44:01.364 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:01.365 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: -1> 2024-05-27T07:44:00.984+0000 7f0d1347f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0d1347f700 time 2024-05-27T07:44:00.983068+0000 2024-05-27T07:44:01.365 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:44:01.365 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:01.365 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:01.365 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0d25cc4ba6] 2024-05-27T07:44:01.365 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0d25cc4dd1] 2024-05-27T07:44:01.365 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560b223aa23d] 2024-05-27T07:44:01.365 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 4: (SnapRealm::build_snap_set() const+0x323) [0x560b22395971] 2024-05-27T07:44:01.365 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 5: (SnapRealm::check_cache() const+0x13e) [0x560b22393fa0] 2024-05-27T07:44:01.365 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560b22394651] 2024-05-27T07:44:01.365 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560b22329347] 2024-05-27T07:44:01.365 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560b2202587e] 2024-05-27T07:44:01.366 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560b2202ae94] 2024-05-27T07:44:01.366 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560b22035969] 2024-05-27T07:44:01.366 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 11: (MDLog::_replay_thread()+0x1f38) [0x560b223ce48c] 2024-05-27T07:44:01.366 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 12: (MDLog::ReplayThread::entry()+0x11) [0x560b223d79cb] 2024-05-27T07:44:01.366 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 13: (Thread::entry_wrapper()+0x43) [0x7f0d25c9372d] 2024-05-27T07:44:01.366 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0d25c93749] 2024-05-27T07:44:01.366 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0d248371ca] 2024-05-27T07:44:01.366 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 16: clone() 2024-05-27T07:44:01.366 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 0> 2024-05-27T07:44:00.987+0000 7f0d1347f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: in thread 7f0d1347f700 thread_name:md_log_replay 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0d24841d20] 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2: gsignal() 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 3: abort() 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0d25cc4ced] 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0d25cc4dd1] 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560b223aa23d] 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 7: (SnapRealm::build_snap_set() const+0x323) [0x560b22395971] 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 8: (SnapRealm::check_cache() const+0x13e) [0x560b22393fa0] 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560b22394651] 2024-05-27T07:44:01.367 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560b22329347] 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560b2202587e] 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560b2202ae94] 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560b22035969] 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 14: (MDLog::_replay_thread()+0x1f38) [0x560b223ce48c] 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 15: (MDLog::ReplayThread::entry()+0x11) [0x560b223d79cb] 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 16: (Thread::entry_wrapper()+0x43) [0x7f0d25c9372d] 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0d25c93749] 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0d248371ca] 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 19: clone() 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: -1> 2024-05-27T07:44:00.984+0000 7f0d1347f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0d1347f700 time 2024-05-27T07:44:00.983068+0000 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:01.368 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0d25cc4ba6] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0d25cc4dd1] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560b223aa23d] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 4: (SnapRealm::build_snap_set() const+0x323) [0x560b22395971] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 5: (SnapRealm::check_cache() const+0x13e) [0x560b22393fa0] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560b22394651] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560b22329347] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560b2202587e] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560b2202ae94] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560b22035969] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 11: (MDLog::_replay_thread()+0x1f38) [0x560b223ce48c] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 12: (MDLog::ReplayThread::entry()+0x11) [0x560b223d79cb] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 13: (Thread::entry_wrapper()+0x43) [0x7f0d25c9372d] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0d25c93749] 2024-05-27T07:44:01.369 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0d248371ca] 2024-05-27T07:44:01.370 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 16: clone() 2024-05-27T07:44:01.370 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:01.370 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 0> 2024-05-27T07:44:00.987+0000 7f0d1347f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:44:01.370 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: in thread 7f0d1347f700 thread_name:md_log_replay 2024-05-27T07:44:01.370 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:01.370 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:01.370 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0d24841d20] 2024-05-27T07:44:01.370 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2: gsignal() 2024-05-27T07:44:01.370 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 3: abort() 2024-05-27T07:44:01.370 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0d25cc4ced] 2024-05-27T07:44:01.370 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0d25cc4dd1] 2024-05-27T07:44:01.370 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560b223aa23d] 2024-05-27T07:44:01.370 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 7: (SnapRealm::build_snap_set() const+0x323) [0x560b22395971] 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 8: (SnapRealm::check_cache() const+0x13e) [0x560b22393fa0] 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560b22394651] 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560b22329347] 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560b2202587e] 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560b2202ae94] 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560b22035969] 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 14: (MDLog::_replay_thread()+0x1f38) [0x560b223ce48c] 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 15: (MDLog::ReplayThread::entry()+0x11) [0x560b223d79cb] 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 16: (Thread::entry_wrapper()+0x43) [0x7f0d25c9372d] 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0d25c93749] 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0d248371ca] 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 19: clone() 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b[70814]: 2024-05-27T07:44:01.371 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 podman[71006]: 2024-05-27 07:44:01.057093594 +0000 UTC m=+0.021731255 container died 240d1e072553d1f6192bb7e87dcc9a4776f1701132c9c64ba264d1f49a44bd0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-27T07:44:01.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.555+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/1321918696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b14110c40 msgr2=0x7f4b14113060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:01.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.555+0000 7f4b1bd5c700 1 --2- 172.21.15.114:0/1321918696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b14110c40 0x7f4b14113060 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00b0a0 tx=0x7f4b0c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:44:01.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.555+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/1321918696 shutdown_connections 2024-05-27T07:44:01.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.555+0000 7f4b1bd5c700 1 --2- 172.21.15.114:0/1321918696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b14110c40 0x7f4b14113060 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:01.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.555+0000 7f4b1bd5c700 1 --2- 172.21.15.114:0/1321918696 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b14103090 0x7f4b141034a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:01.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.555+0000 7f4b1bd5c700 1 --2- 172.21.15.114:0/1321918696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b14103a70 0x7f4b14110700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:01.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.555+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/1321918696 >> 172.21.15.114:0/1321918696 conn(0x7f4b140fef20 msgr2=0x7f4b14101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:01.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.555+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/1321918696 shutdown_connections 2024-05-27T07:44:01.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.555+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/1321918696 wait complete. 2024-05-27T07:44:01.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.556+0000 7f4b1bd5c700 1 Processor -- start 2024-05-27T07:44:01.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.556+0000 7f4b1bd5c700 1 -- start start 2024-05-27T07:44:01.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.556+0000 7f4b1bd5c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b14103090 0x7f4b1419cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:01.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.556+0000 7f4b1bd5c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b14103a70 0x7f4b1419d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:01.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.556+0000 7f4b1bd5c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b14110c40 0x7f4b141a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:01.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.556+0000 7f4b1bd5c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4b141166a0 con 0x7f4b14110c40 2024-05-27T07:44:01.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.556+0000 7f4b1bd5c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4b14116520 con 0x7f4b14103a70 2024-05-27T07:44:01.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.556+0000 7f4b1bd5c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b14116820 con 0x7f4b14103090 2024-05-27T07:44:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.557+0000 7f4b192f7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b14103a70 0x7f4b1419d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.557+0000 7f4b192f7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b14103a70 0x7f4b1419d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:55940/0 (socket says 172.21.15.114:55940) 2024-05-27T07:44:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.557+0000 7f4b192f7700 1 -- 172.21.15.114:0/3057468627 learned_addr learned my addr 172.21.15.114:0/3057468627 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.557+0000 7f4b1a2f9700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b14110c40 0x7f4b141a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.557+0000 7f4b19af8700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b14103090 0x7f4b1419cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.557+0000 7f4b1a2f9700 1 -- 172.21.15.114:0/3057468627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b14103090 msgr2=0x7f4b1419cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.557+0000 7f4b1a2f9700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b14103090 0x7f4b1419cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.557+0000 7f4b1a2f9700 1 -- 172.21.15.114:0/3057468627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b14103a70 msgr2=0x7f4b1419d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.557+0000 7f4b1a2f9700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b14103a70 0x7f4b1419d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.557+0000 7f4b1a2f9700 1 -- 172.21.15.114:0/3057468627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b141a1e60 con 0x7f4b14110c40 2024-05-27T07:44:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.557+0000 7f4b19af8700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b14103090 0x7f4b1419cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.557+0000 7f4b1a2f9700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b14110c40 0x7f4b141a17c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00b070 tx=0x7f4b0c004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.558+0000 7f4b0affd700 1 -- 172.21.15.114:0/3057468627 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c021070 con 0x7f4b14110c40 2024-05-27T07:44:01.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.558+0000 7f4b0affd700 1 -- 172.21.15.114:0/3057468627 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b0c007e30 con 0x7f4b14110c40 2024-05-27T07:44:01.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.558+0000 7f4b0affd700 1 -- 172.21.15.114:0/3057468627 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c0045d0 con 0x7f4b14110c40 2024-05-27T07:44:01.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.558+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/3057468627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b141a20f0 con 0x7f4b14110c40 2024-05-27T07:44:01.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.558+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/3057468627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b141a25e0 con 0x7f4b14110c40 2024-05-27T07:44:01.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.559+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/3057468627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b140512b0 con 0x7f4b14110c40 2024-05-27T07:44:01.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.561+0000 7f4b0affd700 1 -- 172.21.15.114:0/3057468627 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f4b0c003c00 con 0x7f4b14110c40 2024-05-27T07:44:01.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.561+0000 7f4b0affd700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b000b29f0 0x7f4b000b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.561+0000 7f4b19af8700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b000b29f0 0x7f4b000b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.561+0000 7f4b0affd700 1 -- 172.21.15.114:0/3057468627 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f4b0c00f020 con 0x7f4b14110c40 2024-05-27T07:44:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.562+0000 7f4b19af8700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b000b29f0 0x7f4b000b4f00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4b10011490 tx=0x7f4b10009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:01.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.565+0000 7f4b0affd700 1 -- 172.21.15.114:0/3057468627 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4b0c02a020 con 0x7f4b14110c40 2024-05-27T07:44:01.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:01 smithi163 ceph-mon[36451]: pgmap v135: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 413 op/s 2024-05-27T07:44:01.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:01 smithi114 ceph-mon[32038]: pgmap v135: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 413 op/s 2024-05-27T07:44:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.901+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/3057468627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b1402e5e0 con 0x7f4b14110c40 2024-05-27T07:44:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.902+0000 7f4b0affd700 1 -- 172.21.15.114:0/3057468627 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11827 (secure 0 0 0) 0x7f4b0c02e3e0 con 0x7f4b14110c40 2024-05-27T07:44:01.901 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:44:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2251439141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2251439141},{"type":"v1","addr":"172.21.15.186:6837","nonce":2251439141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:43:32.272613+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25691":{"gid":25691,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6839/3475666132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":3475666132},{"type":"v1","addr":"172.21.15.163:6839","nonce":3475666132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:44:01.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.904+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/3057468627 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b000b29f0 msgr2=0x7f4b000b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:01.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.904+0000 7f4b1bd5c700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b000b29f0 0x7f4b000b4f00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4b10011490 tx=0x7f4b10009260 comp rx=0 tx=0).stop 2024-05-27T07:44:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.904+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/3057468627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b14110c40 msgr2=0x7f4b141a17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.904+0000 7f4b1bd5c700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b14110c40 0x7f4b141a17c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00b070 tx=0x7f4b0c004d00 comp rx=0 tx=0).stop 2024-05-27T07:44:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.904+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/3057468627 shutdown_connections 2024-05-27T07:44:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.904+0000 7f4b1bd5c700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b14110c40 0x7f4b141a17c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.904+0000 7f4b1bd5c700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b14103090 0x7f4b1419cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.904+0000 7f4b1bd5c700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b000b29f0 0x7f4b000b4f00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.904+0000 7f4b1bd5c700 1 --2- 172.21.15.114:0/3057468627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b14103a70 0x7f4b1419d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.904+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/3057468627 >> 172.21.15.114:0/3057468627 conn(0x7f4b140fef20 msgr2=0x7f4b141125e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.904+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/3057468627 shutdown_connections 2024-05-27T07:44:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:01.904+0000 7f4b1bd5c700 1 -- 172.21.15.114:0/3057468627 wait complete. 2024-05-27T07:44:01.905 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 29 2024-05-27T07:44:01.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:01 smithi186 ceph-mon[36647]: pgmap v135: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 413 op/s 2024-05-27T07:44:02.001 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 podman[71006]: 2024-05-27 07:44:01.73764149 +0000 UTC m=+0.702279150 container remove 240d1e072553d1f6192bb7e87dcc9a4776f1701132c9c64ba264d1f49a44bd0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:44:02.001 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:01 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:44:02.244 INFO:tasks.workunit.client.0.smithi114.stdout: 1 343396 12.94 MB/sec execute 14 sec latency 62.225 ms 2024-05-27T07:44:02.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:02 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-05-27T07:44:02.474 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:02 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Consumed 2.969s CPU time. 2024-05-27T07:44:02.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:44:02.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:02 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3057468627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:44:02.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:02 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3057468627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:44:03.004 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:44:03.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:02 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3057468627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:44:03.244 INFO:tasks.workunit.client.0.smithi114.stdout: 1 345222 12.74 MB/sec execute 15 sec latency 30.965 ms 2024-05-27T07:44:03.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:03 smithi114 ceph-mon[32038]: pgmap v136: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 453 op/s 2024-05-27T07:44:03.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:03 smithi163 ceph-mon[36451]: pgmap v136: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 453 op/s 2024-05-27T07:44:03.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:03 smithi186 ceph-mon[36647]: pgmap v136: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 453 op/s 2024-05-27T07:44:04.244 INFO:tasks.workunit.client.0.smithi114.stdout: 1 348230 13.01 MB/sec execute 16 sec latency 12.371 ms 2024-05-27T07:44:04.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.846+0000 7ff83cb44700 1 -- 172.21.15.114:0/2630977836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8381031c0 msgr2=0x7ff838111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:04.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.846+0000 7ff83cb44700 1 --2- 172.21.15.114:0/2630977836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8381031c0 0x7ff838111160 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff83000b0a0 tx=0x7ff83000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.846+0000 7ff83cb44700 1 -- 172.21.15.114:0/2630977836 shutdown_connections 2024-05-27T07:44:04.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.846+0000 7ff83cb44700 1 --2- 172.21.15.114:0/2630977836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8381031c0 0x7ff838111160 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.846+0000 7ff83cb44700 1 --2- 172.21.15.114:0/2630977836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8381092f0 0x7ff838109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.846+0000 7ff83cb44700 1 --2- 172.21.15.114:0/2630977836 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff838102870 0x7ff838102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.846+0000 7ff83cb44700 1 -- 172.21.15.114:0/2630977836 >> 172.21.15.114:0/2630977836 conn(0x7ff8380fe6e0 msgr2=0x7ff838100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:04.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.846+0000 7ff83cb44700 1 -- 172.21.15.114:0/2630977836 shutdown_connections 2024-05-27T07:44:04.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.846+0000 7ff83cb44700 1 -- 172.21.15.114:0/2630977836 wait complete. 2024-05-27T07:44:04.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.847+0000 7ff83cb44700 1 Processor -- start 2024-05-27T07:44:04.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.847+0000 7ff83cb44700 1 -- start start 2024-05-27T07:44:04.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.847+0000 7ff83cb44700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff838102870 0x7ff83819ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.847+0000 7ff83cb44700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8381031c0 0x7ff83819b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.847+0000 7ff83cb44700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8381092f0 0x7ff83819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.847+0000 7ff83cb44700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff838114590 con 0x7ff838102870 2024-05-27T07:44:04.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.847+0000 7ff83cb44700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff838114410 con 0x7ff8381031c0 2024-05-27T07:44:04.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.847+0000 7ff83cb44700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff838114710 con 0x7ff8381092f0 2024-05-27T07:44:04.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.847+0000 7ff83659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff838102870 0x7ff83819ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.847+0000 7ff83659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff838102870 0x7ff83819ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60704/0 (socket says 172.21.15.114:60704) 2024-05-27T07:44:04.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.847+0000 7ff83659c700 1 -- 172.21.15.114:0/2463405874 learned_addr learned my addr 172.21.15.114:0/2463405874 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:04.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.847+0000 7ff836d9d700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8381092f0 0x7ff83819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.847+0000 7ff835d9b700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8381031c0 0x7ff83819b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.848+0000 7ff83659c700 1 -- 172.21.15.114:0/2463405874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8381092f0 msgr2=0x7ff83819f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:04.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.848+0000 7ff83659c700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8381092f0 0x7ff83819f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.848+0000 7ff83659c700 1 -- 172.21.15.114:0/2463405874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8381031c0 msgr2=0x7ff83819b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:04.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.848+0000 7ff83659c700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8381031c0 0x7ff83819b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.848+0000 7ff83659c700 1 -- 172.21.15.114:0/2463405874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83819fd50 con 0x7ff838102870 2024-05-27T07:44:04.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.848+0000 7ff836d9d700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8381092f0 0x7ff83819f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:04.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.848+0000 7ff835d9b700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8381031c0 0x7ff83819b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:44:04.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.848+0000 7ff83659c700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff838102870 0x7ff83819ae80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff82800e940 tx=0x7ff82800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:04.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.848+0000 7ff8277fe700 1 -- 172.21.15.114:0/2463405874 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82800cbd0 con 0x7ff838102870 2024-05-27T07:44:04.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.848+0000 7ff83cb44700 1 -- 172.21.15.114:0/2463405874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8381a0040 con 0x7ff838102870 2024-05-27T07:44:04.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.848+0000 7ff8277fe700 1 -- 172.21.15.114:0/2463405874 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82800ee90 con 0x7ff838102870 2024-05-27T07:44:04.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.848+0000 7ff8277fe700 1 -- 172.21.15.114:0/2463405874 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828003dc0 con 0x7ff838102870 2024-05-27T07:44:04.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.848+0000 7ff83cb44700 1 -- 172.21.15.114:0/2463405874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8381a7960 con 0x7ff838102870 2024-05-27T07:44:04.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.851+0000 7ff8277fe700 1 -- 172.21.15.114:0/2463405874 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ff828018430 con 0x7ff838102870 2024-05-27T07:44:04.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.852+0000 7ff84b559b80 1 Processor -- start 2024-05-27T07:44:04.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.852+0000 7ff8277fe700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff8200b29f0 0x7ff8200b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.852+0000 7ff836d9d700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff8200b29f0 0x7ff8200b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.852+0000 7ff8277fe700 1 -- 172.21.15.114:0/2463405874 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff8280e2c60 con 0x7ff838102870 2024-05-27T07:44:04.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff84b559b80 1 -- start start 2024-05-27T07:44:04.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff84b559b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b401b0e40 0x557b40335b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff84b559b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 0x557b4032fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff84b559b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 0x557b40330530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff84b559b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557b4025dd70 con 0x557b40336070 2024-05-27T07:44:04.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff84b559b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557b4024d910 con 0x557b40330120 2024-05-27T07:44:04.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff84b559b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557b40251950 con 0x557b401b0e40 2024-05-27T07:44:04.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff835d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 0x557b4032fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff836d9d700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff8200b29f0 0x7ff8200b4f00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff830009900 tx=0x7ff83000b720 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:04.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff835d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 0x557b4032fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60714/0 (socket says 172.21.15.114:60714) 2024-05-27T07:44:04.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff835d9b700 1 -- 172.21.15.114:0/1626801712 learned_addr learned my addr 172.21.15.114:0/1626801712 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff83659c700 1 --2- 172.21.15.114:0/1626801712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b401b0e40 0x557b40335b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff836d9d700 1 --2- 172.21.15.114:0/1626801712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 0x557b40330530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff835d9b700 1 -- 172.21.15.114:0/1626801712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b401b0e40 msgr2=0x557b40335b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff835d9b700 1 --2- 172.21.15.114:0/1626801712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b401b0e40 0x557b40335b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff835d9b700 1 -- 172.21.15.114:0/1626801712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 msgr2=0x557b40330530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff835d9b700 1 --2- 172.21.15.114:0/1626801712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 0x557b40330530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff835d9b700 1 -- 172.21.15.114:0/1626801712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b401f6fb0 con 0x557b40336070 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.853+0000 7ff83659c700 1 --2- 172.21.15.114:0/1626801712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b401b0e40 0x557b40335b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.854+0000 7ff835d9b700 1 --2- 172.21.15.114:0/1626801712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 0x557b4032fbe0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00bd30 tx=0x7ff82c00f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46eeaa38a21c3f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.854+0000 7ff824ff9700 1 -- 172.21.15.114:0/1626801712 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c00fda0 con 0x557b40336070 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.854+0000 7ff824ff9700 1 -- 172.21.15.114:0/1626801712 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c0118f0 con 0x557b40336070 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.854+0000 7ff824ff9700 1 -- 172.21.15.114:0/1626801712 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c01b710 con 0x557b40336070 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.855+0000 7ff84b559b80 1 -- 172.21.15.114:0/1626801712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 msgr2=0x557b4032fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.855+0000 7ff84b559b80 1 --2- 172.21.15.114:0/1626801712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 0x557b4032fbe0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00bd30 tx=0x7ff82c00f640 comp rx=0 tx=0).stop 2024-05-27T07:44:04.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.855+0000 7ff84b559b80 1 -- 172.21.15.114:0/1626801712 shutdown_connections 2024-05-27T07:44:04.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.855+0000 7ff84b559b80 1 --2- 172.21.15.114:0/1626801712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 0x557b40330530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.855+0000 7ff84b559b80 1 --2- 172.21.15.114:0/1626801712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b401b0e40 0x557b40335b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.855+0000 7ff84b559b80 1 --2- 172.21.15.114:0/1626801712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 0x557b4032fbe0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.855+0000 7ff84b559b80 1 -- 172.21.15.114:0/1626801712 >> 172.21.15.114:0/1626801712 conn(0x557b401b3770 msgr2=0x557b4032ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:04.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.855+0000 7ff84b559b80 1 -- 172.21.15.114:0/1626801712 shutdown_connections 2024-05-27T07:44:04.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.855+0000 7ff84b559b80 1 -- 172.21.15.114:0/1626801712 wait complete. 2024-05-27T07:44:04.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.855+0000 7ff84b559b80 1 -- start start 2024-05-27T07:44:04.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff84b559b80 1 Processor -- start 2024-05-27T07:44:04.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff84b559b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b401b0e40 0x557b40339960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff84b559b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 0x557b40339ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff84b559b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 0x557b4033a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff84b559b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557b40153fa0 con 0x557b40336070 2024-05-27T07:44:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff84b559b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557b401b5da0 con 0x557b40330120 2024-05-27T07:44:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff84b559b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557b4033cb60 con 0x557b401b0e40 2024-05-27T07:44:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff84b559b80 10 client.0 _ll_register_callbacks cb 0x557b401e45b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff836d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 0x557b4033a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff83659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b401b0e40 0x557b40339960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff836d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 0x557b4033a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60724/0 (socket says 172.21.15.114:60724) 2024-05-27T07:44:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff83659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b401b0e40 0x557b40339960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34512/0 (socket says 172.21.15.114:34512) 2024-05-27T07:44:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff836d9d700 1 -- 172.21.15.114:0/4268309155 learned_addr learned my addr 172.21.15.114:0/4268309155 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff835d9b700 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 0x557b40339ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff83659c700 1 -- 172.21.15.114:0/4268309155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 msgr2=0x557b40339ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff83659c700 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 0x557b40339ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff83659c700 1 -- 172.21.15.114:0/4268309155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 msgr2=0x557b4033a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff83659c700 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 0x557b4033a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff83659c700 1 -- 172.21.15.114:0/4268309155 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b4019d8d0 con 0x557b401b0e40 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff836d9d700 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 0x557b4033a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.856+0000 7ff835d9b700 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 0x557b40339ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff83659c700 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b401b0e40 0x557b40339960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff8280e0000 tx=0x7ff8280a60a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96762f1478437f4e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff81dffb700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff81dffb700 1 -- 172.21.15.114:0/4268309155 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8280a96a0 con 0x557b401b0e40 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff81dffb700 1 -- 172.21.15.114:0/4268309155 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff8280aa000 con 0x557b401b0e40 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff84b559b80 1 -- 172.21.15.114:0/4268309155 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff82c0118f0 con 0x557b401b0e40 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff81dffb700 1 -- 172.21.15.114:0/4268309155 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff8280aa290 con 0x557b401b0e40 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff84b559b80 10 client.25927 fetch_fsmap learned FSMap version 29 2024-05-27T07:44:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff84b559b80 1 -- 172.21.15.114:0/4268309155 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557b40207760 con 0x557b401b0e40 2024-05-27T07:44:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff81dffb700 1 -- 172.21.15.114:0/4268309155 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c0118f0 con 0x557b401b0e40 2024-05-27T07:44:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff81dffb700 1 -- 172.21.15.114:0/4268309155 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7ff82806f250 con 0x557b401b0e40 2024-05-27T07:44:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff84b559b80 10 client.25927 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:44:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff84b559b80 10 client.25927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff84b559b80 20 client.25927 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff84b559b80 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b40349c10 0x557b4034c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff84b559b80 4 client.25927 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:44:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.857+0000 7ff84b559b80 1 -- 172.21.15.114:0/4268309155 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b4025fcc0 con 0x557b40349c10 2024-05-27T07:44:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.858+0000 7ff835d9b700 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b40349c10 0x557b4034c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.858+0000 7ff81dffb700 10 client.25927 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.858+0000 7ff835d9b700 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b40349c10 0x557b4034c080 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a23befb611002036 server_cookie=1d500227ea3b7748 in_seq=0 out_seq=0 2024-05-27T07:44:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff81dffb700 1 -- 172.21.15.114:0/4268309155 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557b4025fcc0 con 0x557b40349c10 2024-05-27T07:44:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff81dffb700 10 client.25927 handle_command_reply: tid=0 2024-05-27T07:44:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff84b559b80 1 client.25927 shutdown 2024-05-27T07:44:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff84b559b80 1 -- 172.21.15.114:0/4268309155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b401b0e40 msgr2=0x557b40339960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff84b559b80 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b401b0e40 0x557b40339960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff8280e0000 tx=0x7ff8280a60a0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff84b559b80 1 -- 172.21.15.114:0/4268309155 shutdown_connections 2024-05-27T07:44:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff84b559b80 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40336070 0x557b4033a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff84b559b80 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b401b0e40 0x557b40339960 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff84b559b80 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b40349c10 0x557b4034c080 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff81dffb700 0 client.25927 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff84b559b80 1 --2- 172.21.15.114:0/4268309155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 0x557b40339ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff84b559b80 1 -- 172.21.15.114:0/4268309155 >> 172.21.15.114:0/4268309155 conn(0x557b401b3770 msgr2=0x557b4032ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff84b559b80 1 -- 172.21.15.114:0/4268309155 shutdown_connections 2024-05-27T07:44:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.859+0000 7ff84b559b80 1 -- 172.21.15.114:0/4268309155 wait complete. 2024-05-27T07:44:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.860+0000 7ff84b559b80 20 client.25927 trim_cache size 0 max 16384 2024-05-27T07:44:04.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.876+0000 7ff84b559b80 1 Processor -- start 2024-05-27T07:44:04.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.876+0000 7ff84b559b80 1 -- start start 2024-05-27T07:44:04.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.876+0000 7ff84b559b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b401b0e40 0x557b4032fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.876+0000 7ff84b559b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40330120 0x557b40338ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.876+0000 7ff84b559b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b40336070 0x557b40351890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.876+0000 7ff84b559b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff8381ae620 con 0x557b40330120 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.876+0000 7ff84b559b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff8381b0ea0 con 0x557b401b0e40 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.876+0000 7ff84b559b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff8381aec90 con 0x557b40336070 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.876+0000 7ff835d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40330120 0x557b40338ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.877+0000 7ff835d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40330120 0x557b40338ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60734/0 (socket says 172.21.15.114:60734) 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.877+0000 7ff835d9b700 1 -- 172.21.15.114:0/4204139889 learned_addr learned my addr 172.21.15.114:0/4204139889 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.877+0000 7ff836d9d700 1 --2- 172.21.15.114:0/4204139889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b40336070 0x557b40351890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.877+0000 7ff83659c700 1 --2- 172.21.15.114:0/4204139889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b401b0e40 0x557b4032fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.877+0000 7ff835d9b700 1 -- 172.21.15.114:0/4204139889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b40336070 msgr2=0x557b40351890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.877+0000 7ff835d9b700 1 --2- 172.21.15.114:0/4204139889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b40336070 0x557b40351890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.877+0000 7ff835d9b700 1 -- 172.21.15.114:0/4204139889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b401b0e40 msgr2=0x557b4032fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.877+0000 7ff835d9b700 1 --2- 172.21.15.114:0/4204139889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b401b0e40 0x557b4032fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.877+0000 7ff835d9b700 1 -- 172.21.15.114:0/4204139889 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b40334f30 con 0x557b40330120 2024-05-27T07:44:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.877+0000 7ff835d9b700 1 --2- 172.21.15.114:0/4204139889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40330120 0x557b40338ca0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00f990 tx=0x7ff82c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d979d6dc38afbc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.877+0000 7ff81ffff700 1 -- 172.21.15.114:0/4204139889 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c004240 con 0x557b40330120 2024-05-27T07:44:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.877+0000 7ff81ffff700 1 -- 172.21.15.114:0/4204139889 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c0043e0 con 0x557b40330120 2024-05-27T07:44:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.877+0000 7ff81ffff700 1 -- 172.21.15.114:0/4204139889 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c004c10 con 0x557b40330120 2024-05-27T07:44:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.878+0000 7ff84b559b80 1 -- 172.21.15.114:0/4204139889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40330120 msgr2=0x557b40338ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.878+0000 7ff84b559b80 1 --2- 172.21.15.114:0/4204139889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40330120 0x557b40338ca0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00f990 tx=0x7ff82c00be80 comp rx=0 tx=0).stop 2024-05-27T07:44:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.878+0000 7ff84b559b80 1 -- 172.21.15.114:0/4204139889 shutdown_connections 2024-05-27T07:44:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.878+0000 7ff84b559b80 1 --2- 172.21.15.114:0/4204139889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b40336070 0x557b40351890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.878+0000 7ff84b559b80 1 --2- 172.21.15.114:0/4204139889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b401b0e40 0x557b4032fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.878+0000 7ff84b559b80 1 --2- 172.21.15.114:0/4204139889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b40330120 0x557b40338ca0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.878+0000 7ff84b559b80 1 -- 172.21.15.114:0/4204139889 >> 172.21.15.114:0/4204139889 conn(0x557b401b3770 msgr2=0x557b4032aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.879+0000 7ff84b559b80 1 -- 172.21.15.114:0/4204139889 shutdown_connections 2024-05-27T07:44:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.879+0000 7ff84b559b80 1 -- 172.21.15.114:0/4204139889 wait complete. 2024-05-27T07:44:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.879+0000 7ff84b559b80 1 -- start start 2024-05-27T07:44:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.879+0000 7ff84b559b80 1 Processor -- start 2024-05-27T07:44:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.879+0000 7ff84b559b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b401b0e40 0x557b40341df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.879+0000 7ff84b559b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 0x557b40342330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.879+0000 7ff84b559b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b40336070 0x557b40350880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.879+0000 7ff84b559b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff8381af360 con 0x557b401b0e40 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.879+0000 7ff84b559b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff8381b07d0 con 0x557b40330120 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.879+0000 7ff84b559b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff8381afa30 con 0x557b40336070 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.879+0000 7ff84b559b80 10 client.0 _ll_register_callbacks cb 0x557b401e45b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff83659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b401b0e40 0x557b40341df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff836d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b40336070 0x557b40350880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff83659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b401b0e40 0x557b40341df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60744/0 (socket says 172.21.15.114:60744) 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff83659c700 1 -- 172.21.15.114:0/1798095453 learned_addr learned my addr 172.21.15.114:0/1798095453 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff835d9b700 1 --2- 172.21.15.114:0/1798095453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 0x557b40342330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff836d9d700 1 -- 172.21.15.114:0/1798095453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 msgr2=0x557b40342330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff836d9d700 1 --2- 172.21.15.114:0/1798095453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 0x557b40342330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff836d9d700 1 -- 172.21.15.114:0/1798095453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b401b0e40 msgr2=0x557b40341df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff836d9d700 1 --2- 172.21.15.114:0/1798095453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b401b0e40 0x557b40341df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff836d9d700 1 -- 172.21.15.114:0/1798095453 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b40352290 con 0x557b40336070 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff836d9d700 1 --2- 172.21.15.114:0/1798095453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b40336070 0x557b40350880 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff830009fd0 tx=0x7ff83001d070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2f1e7a8237b4704 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff81dffb700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff81dffb700 1 -- 172.21.15.114:0/1798095453 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff830037060 con 0x557b40336070 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff84b559b80 1 -- 172.21.15.114:0/1798095453 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff82c0043e0 con 0x557b40336070 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.880+0000 7ff81dffb700 1 -- 172.21.15.114:0/1798095453 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff830005830 con 0x557b40336070 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.881+0000 7ff81dffb700 1 -- 172.21.15.114:0/1798095453 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff83003e050 con 0x557b40336070 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.881+0000 7ff84b559b80 10 client.25933 fetch_fsmap learned FSMap version 29 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.881+0000 7ff84b559b80 1 -- 172.21.15.114:0/1798095453 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557b403b3b70 con 0x557b40336070 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.881+0000 7ff81dffb700 1 -- 172.21.15.114:0/1798095453 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c0043e0 con 0x557b40336070 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.881+0000 7ff81dffb700 1 -- 172.21.15.114:0/1798095453 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7ff830011070 con 0x557b40336070 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.881+0000 7ff84b559b80 10 client.25933 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:44:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.881+0000 7ff84b559b80 10 client.25933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.881+0000 7ff84b559b80 20 client.25933 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.881+0000 7ff84b559b80 1 --2- 172.21.15.114:0/1798095453 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b40349c10 0x557b403b6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.881+0000 7ff84b559b80 4 client.25933 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:44:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.881+0000 7ff84b559b80 1 -- 172.21.15.114:0/1798095453 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b402594a0 con 0x557b40349c10 2024-05-27T07:44:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.881+0000 7ff83659c700 1 --2- 172.21.15.114:0/1798095453 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b40349c10 0x557b403b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff81dffb700 10 client.25933 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff83659c700 1 --2- 172.21.15.114:0/1798095453 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b40349c10 0x557b403b6210 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d90fed6fe527199 server_cookie=6b5e9be7f3227f71 in_seq=0 out_seq=0 2024-05-27T07:44:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff81dffb700 1 -- 172.21.15.114:0/1798095453 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b402594a0 con 0x557b40349c10 2024-05-27T07:44:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff81dffb700 10 client.25933 handle_command_reply: tid=0 2024-05-27T07:44:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff84b559b80 1 client.25933 shutdown 2024-05-27T07:44:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff84b559b80 1 -- 172.21.15.114:0/1798095453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b40336070 msgr2=0x557b40350880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff84b559b80 1 --2- 172.21.15.114:0/1798095453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b40336070 0x557b40350880 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff830009fd0 tx=0x7ff83001d070 comp rx=0 tx=0).stop 2024-05-27T07:44:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff84b559b80 1 -- 172.21.15.114:0/1798095453 shutdown_connections 2024-05-27T07:44:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff84b559b80 1 --2- 172.21.15.114:0/1798095453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b40336070 0x557b40350880 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff84b559b80 1 --2- 172.21.15.114:0/1798095453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b401b0e40 0x557b40341df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff84b559b80 1 --2- 172.21.15.114:0/1798095453 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b40349c10 0x557b403b6210 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff84b559b80 1 --2- 172.21.15.114:0/1798095453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b40330120 0x557b40342330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff81dffb700 0 client.25933 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff84b559b80 1 -- 172.21.15.114:0/1798095453 >> 172.21.15.114:0/1798095453 conn(0x557b401b3770 msgr2=0x557b40333490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff84b559b80 1 -- 172.21.15.114:0/1798095453 shutdown_connections 2024-05-27T07:44:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.882+0000 7ff84b559b80 1 -- 172.21.15.114:0/1798095453 wait complete. 2024-05-27T07:44:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.883+0000 7ff84b559b80 20 client.25933 trim_cache size 0 max 16384 2024-05-27T07:44:04.886 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:44:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.883+0000 7ff83cb44700 1 -- 172.21.15.114:0/2463405874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff8200b29f0 msgr2=0x7ff8200b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.883+0000 7ff83cb44700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff8200b29f0 0x7ff8200b4f00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff830009900 tx=0x7ff83000b720 comp rx=0 tx=0).stop 2024-05-27T07:44:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.883+0000 7ff83cb44700 1 -- 172.21.15.114:0/2463405874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff838102870 msgr2=0x7ff83819ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.883+0000 7ff83cb44700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff838102870 0x7ff83819ae80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff82800e940 tx=0x7ff82800ec50 comp rx=0 tx=0).stop 2024-05-27T07:44:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.883+0000 7ff83cb44700 1 -- 172.21.15.114:0/2463405874 shutdown_connections 2024-05-27T07:44:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.883+0000 7ff83cb44700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8381092f0 0x7ff83819f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.883+0000 7ff83cb44700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff838102870 0x7ff83819ae80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.883+0000 7ff83cb44700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff8200b29f0 0x7ff8200b4f00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.883+0000 7ff83cb44700 1 --2- 172.21.15.114:0/2463405874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8381031c0 0x7ff83819b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.883+0000 7ff83cb44700 1 -- 172.21.15.114:0/2463405874 >> 172.21.15.114:0/2463405874 conn(0x7ff8380fe6e0 msgr2=0x7ff838108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.883+0000 7ff83cb44700 1 -- 172.21.15.114:0/2463405874 shutdown_connections 2024-05-27T07:44:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:04.883+0000 7ff83cb44700 1 -- 172.21.15.114:0/2463405874 wait complete. 2024-05-27T07:44:05.244 INFO:tasks.workunit.client.0.smithi114.stdout: 1 351243 13.28 MB/sec execute 17 sec latency 28.955 ms 2024-05-27T07:44:05.751 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:05 smithi114 ceph-mon[32038]: pgmap v137: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 425 op/s 2024-05-27T07:44:05.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:05 smithi163 ceph-mon[36451]: pgmap v137: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 425 op/s 2024-05-27T07:44:05.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:05 smithi186 ceph-mon[36647]: pgmap v137: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 425 op/s 2024-05-27T07:44:06.244 INFO:tasks.workunit.client.0.smithi114.stdout: 1 353637 13.06 MB/sec execute 18 sec latency 26.361 ms 2024-05-27T07:44:06.782 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:44:06.782 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:44:07.080 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:44:07.244 INFO:tasks.workunit.client.0.smithi114.stdout: 1 355423 12.98 MB/sec execute 19 sec latency 58.442 ms 2024-05-27T07:44:07.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:07 smithi114 ceph-mon[32038]: pgmap v138: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.9 MiB/s wr, 571 op/s 2024-05-27T07:44:07.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:07 smithi163 ceph-mon[36451]: pgmap v138: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.9 MiB/s wr, 571 op/s 2024-05-27T07:44:07.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:07 smithi186 ceph-mon[36647]: pgmap v138: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.9 MiB/s wr, 571 op/s 2024-05-27T07:44:08.244 INFO:tasks.workunit.client.0.smithi114.stdout: 1 358334 13.20 MB/sec execute 20 sec latency 31.740 ms 2024-05-27T07:44:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.778+0000 7fd687667640 1 -- 172.21.15.114:0/4042023563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd680108b90 msgr2=0x7fd680102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.778+0000 7fd687667640 1 --2- 172.21.15.114:0/4042023563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd680108b90 0x7fd680102730 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009900 tx=0x7fd67c02a410 comp rx=0 tx=0).stop 2024-05-27T07:44:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.778+0000 7fd687667640 1 -- 172.21.15.114:0/4042023563 shutdown_connections 2024-05-27T07:44:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.778+0000 7fd687667640 1 --2- 172.21.15.114:0/4042023563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6801032e0 0x7fd68010f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.778+0000 7fd687667640 1 --2- 172.21.15.114:0/4042023563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd680109030 0x7fd680102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.778+0000 7fd687667640 1 --2- 172.21.15.114:0/4042023563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd680108b90 0x7fd680102730 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.778+0000 7fd687667640 1 -- 172.21.15.114:0/4042023563 >> 172.21.15.114:0/4042023563 conn(0x7fd6800782d0 msgr2=0x7fd680078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.778+0000 7fd687667640 1 -- 172.21.15.114:0/4042023563 shutdown_connections 2024-05-27T07:44:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.778+0000 7fd687667640 1 -- 172.21.15.114:0/4042023563 wait complete. 2024-05-27T07:44:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.779+0000 7fd687667640 1 Processor -- start 2024-05-27T07:44:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.779+0000 7fd687667640 1 -- start start 2024-05-27T07:44:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.779+0000 7fd687667640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6801032e0 0x7fd6801a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.779+0000 7fd687667640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd680108b90 0x7fd6801a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.779+0000 7fd687667640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd680109030 0x7fd68019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.779+0000 7fd687667640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd680112d80 con 0x7fd6801032e0 2024-05-27T07:44:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.779+0000 7fd687667640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd680112c00 con 0x7fd680109030 2024-05-27T07:44:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.779+0000 7fd687667640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd680112f00 con 0x7fd680108b90 2024-05-27T07:44:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.780+0000 7fd685403640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6801032e0 0x7fd6801a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.780+0000 7fd685403640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6801032e0 0x7fd6801a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60760/0 (socket says 172.21.15.114:60760) 2024-05-27T07:44:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.780+0000 7fd685403640 1 -- 172.21.15.114:0/2950030973 learned_addr learned my addr 172.21.15.114:0/2950030973 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:08.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.780+0000 7fd684c02640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd680108b90 0x7fd6801a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.780+0000 7fd685c04640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd680109030 0x7fd68019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.780+0000 7fd685403640 1 -- 172.21.15.114:0/2950030973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd680108b90 msgr2=0x7fd6801a0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:08.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.780+0000 7fd685403640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd680108b90 0x7fd6801a0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.780+0000 7fd685403640 1 -- 172.21.15.114:0/2950030973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd680109030 msgr2=0x7fd68019a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.780+0000 7fd685403640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd680109030 0x7fd68019a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.780+0000 7fd685403640 1 -- 172.21.15.114:0/2950030973 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68019ae20 con 0x7fd6801032e0 2024-05-27T07:44:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.780+0000 7fd685c04640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd680109030 0x7fd68019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.780+0000 7fd685403640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6801032e0 0x7fd6801a0570 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd67c032fd0 tx=0x7fd67c031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.780+0000 7fd6767fc640 1 -- 172.21.15.114:0/2950030973 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c03f030 con 0x7fd6801032e0 2024-05-27T07:44:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.781+0000 7fd687667640 1 -- 172.21.15.114:0/2950030973 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd68019b050 con 0x7fd6801032e0 2024-05-27T07:44:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.781+0000 7fd687667640 1 -- 172.21.15.114:0/2950030973 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6801a7310 con 0x7fd6801032e0 2024-05-27T07:44:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.781+0000 7fd6767fc640 1 -- 172.21.15.114:0/2950030973 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd67c004490 con 0x7fd6801032e0 2024-05-27T07:44:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.781+0000 7fd6767fc640 1 -- 172.21.15.114:0/2950030973 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c035070 con 0x7fd6801032e0 2024-05-27T07:44:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd68e5f7740 1 Processor -- start 2024-05-27T07:44:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd68e5f7740 1 -- start start 2024-05-27T07:44:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd6767fc640 1 -- 172.21.15.114:0/2950030973 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fd67c042030 con 0x7fd6801032e0 2024-05-27T07:44:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd68e5f7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1c21b0 0x55d78c5633d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd68e5f7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1992f0 0x55d78c563910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd68e5f7740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 0x55d78c563e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd68e5f7740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d78c4d5c30 con 0x55d78c1a14d0 2024-05-27T07:44:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd68e5f7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d78c4c57d0 con 0x55d78c1c21b0 2024-05-27T07:44:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd68e5f7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d78c4c9810 con 0x55d78c1992f0 2024-05-27T07:44:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd684c02640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 0x55d78c563e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd685403640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1992f0 0x55d78c563910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd684c02640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 0x55d78c563e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60762/0 (socket says 172.21.15.114:60762) 2024-05-27T07:44:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd684c02640 1 -- 172.21.15.114:0/1647012788 learned_addr learned my addr 172.21.15.114:0/1647012788 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.784+0000 7fd685c04640 1 --2- 172.21.15.114:0/1647012788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1c21b0 0x55d78c5633d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.785+0000 7fd684c02640 1 -- 172.21.15.114:0/1647012788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1992f0 msgr2=0x55d78c563910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.785+0000 7fd684c02640 1 --2- 172.21.15.114:0/1647012788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1992f0 0x55d78c563910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.785+0000 7fd684c02640 1 -- 172.21.15.114:0/1647012788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1c21b0 msgr2=0x55d78c5633d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.785+0000 7fd684c02640 1 --2- 172.21.15.114:0/1647012788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1c21b0 0x55d78c5633d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.785+0000 7fd684c02640 1 -- 172.21.15.114:0/1647012788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d78c1da210 con 0x55d78c1a14d0 2024-05-27T07:44:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.785+0000 7fd685403640 1 --2- 172.21.15.114:0/1647012788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1992f0 0x55d78c563910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.785+0000 7fd684c02640 1 --2- 172.21.15.114:0/1647012788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 0x55d78c563e50 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd670009c20 tx=0x7fd670011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6234ad975845b83 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.785+0000 7fd6767fc640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd66c0b29d0 0x7fd66c0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.785+0000 7fd66b7fe640 1 -- 172.21.15.114:0/1647012788 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670009dd0 con 0x55d78c1a14d0 2024-05-27T07:44:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.785+0000 7fd685c04640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd66c0b29d0 0x7fd66c0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.785+0000 7fd66b7fe640 1 -- 172.21.15.114:0/1647012788 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd670005020 con 0x55d78c1a14d0 2024-05-27T07:44:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.785+0000 7fd6767fc640 1 -- 172.21.15.114:0/2950030973 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd67c102c30 con 0x7fd6801032e0 2024-05-27T07:44:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.785+0000 7fd66b7fe640 1 -- 172.21.15.114:0/1647012788 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6700049e0 con 0x55d78c1a14d0 2024-05-27T07:44:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.786+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1647012788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 msgr2=0x55d78c563e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.786+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1647012788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 0x55d78c563e50 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd670009c20 tx=0x7fd670011810 comp rx=0 tx=0).stop 2024-05-27T07:44:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.786+0000 7fd685c04640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd66c0b29d0 0x7fd66c0b4ee0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55d78c560540 tx=0x7fd678009500 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.786+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1647012788 shutdown_connections 2024-05-27T07:44:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.786+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1647012788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 0x55d78c563e50 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.786+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1647012788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1992f0 0x55d78c563910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.786+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1647012788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1c21b0 0x55d78c5633d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.786+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1647012788 >> 172.21.15.114:0/1647012788 conn(0x55d78c1b6300 msgr2=0x55d78c55fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.787+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1647012788 shutdown_connections 2024-05-27T07:44:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.787+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1647012788 wait complete. 2024-05-27T07:44:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.787+0000 7fd68e5f7740 1 -- start start 2024-05-27T07:44:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.787+0000 7fd68e5f7740 1 Processor -- start 2024-05-27T07:44:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd68e5f7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1992f0 0x55d78c56da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd68e5f7740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 0x55d78c56dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd68e5f7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 0x55d78c573870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd68e5f7740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d78c1ab790 con 0x55d78c1a14d0 2024-05-27T07:44:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd68e5f7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d78c481890 con 0x55d78c1992f0 2024-05-27T07:44:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd68e5f7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d78c570500 con 0x55d78c1c21b0 2024-05-27T07:44:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd68e5f7740 10 client.0 _ll_register_callbacks cb 0x55d78c1d9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd684c02640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 0x55d78c56dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd684c02640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 0x55d78c56dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60776/0 (socket says 172.21.15.114:60776) 2024-05-27T07:44:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd684c02640 1 -- 172.21.15.114:0/1771393370 learned_addr learned my addr 172.21.15.114:0/1771393370 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd685403640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1992f0 0x55d78c56da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd685c04640 1 --2- 172.21.15.114:0/1771393370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 0x55d78c573870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd685403640 1 -- 172.21.15.114:0/1771393370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 msgr2=0x55d78c573870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd685403640 1 --2- 172.21.15.114:0/1771393370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 0x55d78c573870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd685403640 1 -- 172.21.15.114:0/1771393370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 msgr2=0x55d78c56dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd685403640 1 --2- 172.21.15.114:0/1771393370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 0x55d78c56dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd685403640 1 -- 172.21.15.114:0/1771393370 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d78c2a6620 con 0x55d78c1992f0 2024-05-27T07:44:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd684c02640 1 --2- 172.21.15.114:0/1771393370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 0x55d78c56dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd685403640 1 --2- 172.21.15.114:0/1771393370 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1992f0 0x55d78c56da80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd67c0fffd0 tx=0x7fd67c02a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96121219a07ead59 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.788+0000 7fd668ff9640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:44:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd668ff9640 1 -- 172.21.15.114:0/1771393370 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c049070 con 0x55d78c1992f0 2024-05-27T07:44:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd668ff9640 1 -- 172.21.15.114:0/1771393370 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd67c0c9e70 con 0x55d78c1992f0 2024-05-27T07:44:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1771393370 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd670005020 con 0x55d78c1992f0 2024-05-27T07:44:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd668ff9640 1 -- 172.21.15.114:0/1771393370 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd67c0c99e0 con 0x55d78c1992f0 2024-05-27T07:44:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd68e5f7740 10 client.25874 fetch_fsmap learned FSMap version 29 2024-05-27T07:44:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1771393370 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d78c57c800 con 0x55d78c1992f0 2024-05-27T07:44:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd668ff9640 1 -- 172.21.15.114:0/1771393370 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd670005020 con 0x55d78c1992f0 2024-05-27T07:44:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd668ff9640 1 -- 172.21.15.114:0/1771393370 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fd67c0d2530 con 0x55d78c1992f0 2024-05-27T07:44:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd68e5f7740 10 client.25874 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:44:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd68e5f7740 10 client.25874 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd68e5f7740 20 client.25874 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1771393370 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d78c180d80 0x55d78c57fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd68e5f7740 4 client.25874 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:44:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1771393370 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d78c4d7b80 con 0x55d78c180d80 2024-05-27T07:44:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.789+0000 7fd684c02640 1 --2- 172.21.15.114:0/1771393370 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d78c180d80 0x55d78c57fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.790+0000 7fd668ff9640 10 client.25874 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.790+0000 7fd684c02640 1 --2- 172.21.15.114:0/1771393370 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d78c180d80 0x55d78c57fce0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f6e26dc73bb79b9 server_cookie=49b63ef490e6a262 in_seq=0 out_seq=0 2024-05-27T07:44:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.791+0000 7fd668ff9640 1 -- 172.21.15.114:0/1771393370 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d78c4d7b80 con 0x55d78c180d80 2024-05-27T07:44:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.791+0000 7fd668ff9640 10 client.25874 handle_command_reply: tid=0 2024-05-27T07:44:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.791+0000 7fd68e5f7740 1 client.25874 shutdown 2024-05-27T07:44:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.791+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1771393370 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1992f0 msgr2=0x55d78c56da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.791+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1771393370 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1992f0 0x55d78c56da80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd67c0fffd0 tx=0x7fd67c02a410 comp rx=0 tx=0).stop 2024-05-27T07:44:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.791+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1771393370 shutdown_connections 2024-05-27T07:44:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.791+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1771393370 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d78c180d80 0x55d78c57fce0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.791+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1771393370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 0x55d78c573870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.791+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1771393370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1a14d0 0x55d78c56dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.791+0000 7fd668ff9640 0 client.25874 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.791+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1771393370 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1992f0 0x55d78c56da80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.791+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1771393370 >> 172.21.15.114:0/1771393370 conn(0x55d78c1b6300 msgr2=0x55d78c5650f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.792+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1771393370 shutdown_connections 2024-05-27T07:44:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.792+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1771393370 wait complete. 2024-05-27T07:44:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.792+0000 7fd68e5f7740 20 client.25874 trim_cache size 0 max 16384 2024-05-27T07:44:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.807+0000 7fd68e5f7740 1 Processor -- start 2024-05-27T07:44:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd68e5f7740 1 -- start start 2024-05-27T07:44:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd68e5f7740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 0x55d78c573770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd68e5f7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1a14d0 0x55d78c565720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd68e5f7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 0x55d78c567130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd68e5f7740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd6801af390 con 0x55d78c1992f0 2024-05-27T07:44:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd68e5f7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd6801b1c10 con 0x55d78c1a14d0 2024-05-27T07:44:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd68e5f7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6801afa00 con 0x55d78c1c21b0 2024-05-27T07:44:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd685403640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 0x55d78c573770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd685403640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 0x55d78c573770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60782/0 (socket says 172.21.15.114:60782) 2024-05-27T07:44:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd685c04640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 0x55d78c567130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd685403640 1 -- 172.21.15.114:0/3209459828 learned_addr learned my addr 172.21.15.114:0/3209459828 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd684c02640 1 --2- 172.21.15.114:0/3209459828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1a14d0 0x55d78c565720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd685403640 1 -- 172.21.15.114:0/3209459828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 msgr2=0x55d78c567130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd685403640 1 --2- 172.21.15.114:0/3209459828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 0x55d78c567130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd685403640 1 -- 172.21.15.114:0/3209459828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1a14d0 msgr2=0x55d78c565720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd685403640 1 --2- 172.21.15.114:0/3209459828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1a14d0 0x55d78c565720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd685403640 1 -- 172.21.15.114:0/3209459828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d78c565c60 con 0x55d78c1992f0 2024-05-27T07:44:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.808+0000 7fd685403640 1 --2- 172.21.15.114:0/3209459828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 0x55d78c573770 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd67c032af0 tx=0x7fd67c009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b14d5f3fd841f26 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.809+0000 7fd66affd640 1 -- 172.21.15.114:0/3209459828 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c049040 con 0x55d78c1992f0 2024-05-27T07:44:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.809+0000 7fd66affd640 1 -- 172.21.15.114:0/3209459828 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd67c105070 con 0x55d78c1992f0 2024-05-27T07:44:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.809+0000 7fd66affd640 1 -- 172.21.15.114:0/3209459828 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c0c9380 con 0x55d78c1992f0 2024-05-27T07:44:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.809+0000 7fd68e5f7740 1 -- 172.21.15.114:0/3209459828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 msgr2=0x55d78c573770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.809+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/3209459828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 0x55d78c573770 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd67c032af0 tx=0x7fd67c009680 comp rx=0 tx=0).stop 2024-05-27T07:44:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.810+0000 7fd68e5f7740 1 -- 172.21.15.114:0/3209459828 shutdown_connections 2024-05-27T07:44:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.810+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/3209459828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 0x55d78c567130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.810+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/3209459828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1a14d0 0x55d78c565720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.810+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/3209459828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 0x55d78c573770 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.810+0000 7fd68e5f7740 1 -- 172.21.15.114:0/3209459828 >> 172.21.15.114:0/3209459828 conn(0x55d78c1b6300 msgr2=0x55d78c565010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.810+0000 7fd68e5f7740 1 -- 172.21.15.114:0/3209459828 shutdown_connections 2024-05-27T07:44:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.810+0000 7fd68e5f7740 1 -- 172.21.15.114:0/3209459828 wait complete. 2024-05-27T07:44:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.810+0000 7fd68e5f7740 1 -- start start 2024-05-27T07:44:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.810+0000 7fd68e5f7740 1 Processor -- start 2024-05-27T07:44:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd68e5f7740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 0x55d78c5e48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd68e5f7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1a14d0 0x55d78c5e4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd68e5f7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 0x55d78c574d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd68e5f7740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd6801b00d0 con 0x55d78c1992f0 2024-05-27T07:44:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd68e5f7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd6801b1540 con 0x55d78c1a14d0 2024-05-27T07:44:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd68e5f7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6801b07a0 con 0x55d78c1c21b0 2024-05-27T07:44:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd685403640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 0x55d78c5e48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd68e5f7740 10 client.0 _ll_register_callbacks cb 0x7fd67c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd685403640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 0x55d78c5e48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60792/0 (socket says 172.21.15.114:60792) 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd685403640 1 -- 172.21.15.114:0/1925745834 learned_addr learned my addr 172.21.15.114:0/1925745834 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd685c04640 1 --2- 172.21.15.114:0/1925745834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 0x55d78c574d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd684c02640 1 --2- 172.21.15.114:0/1925745834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1a14d0 0x55d78c5e4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd685403640 1 -- 172.21.15.114:0/1925745834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 msgr2=0x55d78c574d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd685403640 1 --2- 172.21.15.114:0/1925745834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 0x55d78c574d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd685403640 1 -- 172.21.15.114:0/1925745834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1a14d0 msgr2=0x55d78c5e4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd685403640 1 --2- 172.21.15.114:0/1925745834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1a14d0 0x55d78c5e4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd685403640 1 -- 172.21.15.114:0/1925745834 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d78c5753b0 con 0x55d78c1992f0 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd685403640 1 --2- 172.21.15.114:0/1925745834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 0x55d78c5e48c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd67c0c5c90 tx=0x7fd67c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3dc92f5bd2eaaf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.811+0000 7fd668ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd668ff9640 1 -- 172.21.15.114:0/1925745834 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c049070 con 0x55d78c1992f0 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd668ff9640 1 -- 172.21.15.114:0/1925745834 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd67c0c9080 con 0x55d78c1992f0 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1925745834 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd67c105070 con 0x55d78c1992f0 2024-05-27T07:44:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd668ff9640 1 -- 172.21.15.114:0/1925745834 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd67c0ae050 con 0x55d78c1992f0 2024-05-27T07:44:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd68e5f7740 10 client.16359 fetch_fsmap learned FSMap version 29 2024-05-27T07:44:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1925745834 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d78c5753b0 con 0x55d78c1992f0 2024-05-27T07:44:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd668ff9640 1 -- 172.21.15.114:0/1925745834 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd67c105070 con 0x55d78c1992f0 2024-05-27T07:44:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd668ff9640 1 -- 172.21.15.114:0/1925745834 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fd67c0ec110 con 0x55d78c1992f0 2024-05-27T07:44:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd68e5f7740 10 client.16359 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:44:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd68e5f7740 10 client.16359 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd68e5f7740 20 client.16359 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1925745834 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d78c1561c0 0x55d78c5ee130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd68e5f7740 4 client.16359 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:44:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1925745834 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d78c4d1360 con 0x55d78c1561c0 2024-05-27T07:44:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.812+0000 7fd684c02640 1 --2- 172.21.15.114:0/1925745834 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d78c1561c0 0x55d78c5ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.813+0000 7fd668ff9640 10 client.16359 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:08.813+0000 7fd684c02640 1 --2- 172.21.15.114:0/1925745834 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d78c1561c0 0x55d78c5ee130 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2b333ab3b402482 server_cookie=17b0f242e51eaa52 in_seq=0 out_seq=0 2024-05-27T07:44:08.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.988+0000 7ffadcf4f700 1 -- 172.21.15.114:0/2409714927 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffad810a920 msgr2=0x7ffad810ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:08.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.988+0000 7ffadcf4f700 1 --2- 172.21.15.114:0/2409714927 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffad810a920 0x7ffad810ad30 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ffac0009990 tx=0x7ffac0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.988+0000 7ffadcf4f700 1 -- 172.21.15.114:0/2409714927 shutdown_connections 2024-05-27T07:44:08.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.988+0000 7ffadcf4f700 1 --2- 172.21.15.114:0/2409714927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffad811d920 0x7ffad811fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.988+0000 7ffadcf4f700 1 --2- 172.21.15.114:0/2409714927 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffad810a920 0x7ffad810ad30 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.988+0000 7ffadcf4f700 1 --2- 172.21.15.114:0/2409714927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffad810b270 0x7ffad811d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.988+0000 7ffadcf4f700 1 -- 172.21.15.114:0/2409714927 >> 172.21.15.114:0/2409714927 conn(0x7ffad806f4c0 msgr2=0x7ffad806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:08.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.992+0000 7ffadcf4f700 1 -- 172.21.15.114:0/2409714927 shutdown_connections 2024-05-27T07:44:08.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.992+0000 7ffadcf4f700 1 -- 172.21.15.114:0/2409714927 wait complete. 2024-05-27T07:44:08.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.993+0000 7ffadcf4f700 1 Processor -- start 2024-05-27T07:44:08.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.995+0000 7ffadcf4f700 1 -- start start 2024-05-27T07:44:08.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.996+0000 7ffadcf4f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffad810a920 0x7ffad81a9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.996+0000 7ffadcf4f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffad810b270 0x7ffad81aa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.996+0000 7ffadcf4f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffad811d920 0x7ffad81ae630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:08.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.996+0000 7ffadcf4f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffad8123430 con 0x7ffad811d920 2024-05-27T07:44:08.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.996+0000 7ffadcf4f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffad81232b0 con 0x7ffad810a920 2024-05-27T07:44:08.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.996+0000 7ffadcf4f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffad81235b0 con 0x7ffad810b270 2024-05-27T07:44:08.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.996+0000 7ffad6d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffad811d920 0x7ffad81ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.996+0000 7ffad6d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffad811d920 0x7ffad81ae630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60796/0 (socket says 172.21.15.114:60796) 2024-05-27T07:44:08.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.996+0000 7ffad6d9d700 1 -- 172.21.15.114:0/3032362018 learned_addr learned my addr 172.21.15.114:0/3032362018 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:08.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.996+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffad810b270 0x7ffad81aa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.997+0000 7ffad659c700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffad810a920 0x7ffad81a9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:08.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.997+0000 7ffad6d9d700 1 -- 172.21.15.114:0/3032362018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffad810b270 msgr2=0x7ffad81aa3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:08.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.997+0000 7ffad6d9d700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffad810b270 0x7ffad81aa3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.997+0000 7ffad6d9d700 1 -- 172.21.15.114:0/3032362018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffad810a920 msgr2=0x7ffad81a9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:08.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.997+0000 7ffad6d9d700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffad810a920 0x7ffad81a9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:08.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.997+0000 7ffad6d9d700 1 -- 172.21.15.114:0/3032362018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad81aecd0 con 0x7ffad811d920 2024-05-27T07:44:08.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.998+0000 7ffad6d9d700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffad811d920 0x7ffad81ae630 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ffad000b7b0 tx=0x7ffad000bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:08.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.998+0000 7ffacf7fe700 1 -- 172.21.15.114:0/3032362018 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad0014070 con 0x7ffad811d920 2024-05-27T07:44:08.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.998+0000 7ffacf7fe700 1 -- 172.21.15.114:0/3032362018 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffad0010cb0 con 0x7ffad811d920 2024-05-27T07:44:08.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.998+0000 7ffacf7fe700 1 -- 172.21.15.114:0/3032362018 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad0017d30 con 0x7ffad811d920 2024-05-27T07:44:08.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.999+0000 7ffadcf4f700 1 -- 172.21.15.114:0/3032362018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad81aefc0 con 0x7ffad811d920 2024-05-27T07:44:08.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:08.999+0000 7ffadcf4f700 1 -- 172.21.15.114:0/3032362018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad81b68e0 con 0x7ffad811d920 2024-05-27T07:44:09.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.007+0000 7ffaeb964b80 1 Processor -- start 2024-05-27T07:44:09.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.008+0000 7ffaeb964b80 1 -- start start 2024-05-27T07:44:09.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.008+0000 7ffacf7fe700 1 -- 172.21.15.114:0/3032362018 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ffad0017070 con 0x7ffad811d920 2024-05-27T07:44:09.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.010+0000 7ffacf7fe700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffac40b2aa0 0x7ffac40b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.010+0000 7ffacf7fe700 1 -- 172.21.15.114:0/3032362018 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ffad00e23a0 con 0x7ffad811d920 2024-05-27T07:44:09.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.011+0000 7ffaeb964b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd45790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.011+0000 7ffaeb964b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd3fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.011+0000 7ffaeb964b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd40410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.011+0000 7ffaeb964b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55abdfc6ed10 con 0x55abdfd40000 2024-05-27T07:44:09.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.011+0000 7ffaeb964b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55abdfc5e8b0 con 0x55abdfbc1ec0 2024-05-27T07:44:09.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.011+0000 7ffaeb964b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55abdfc628f0 con 0x55abdfd45cd0 2024-05-27T07:44:09.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.011+0000 7ffad5d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd3fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:09.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.011+0000 7ffad5d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd3fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34604/0 (socket says 172.21.15.114:34604) 2024-05-27T07:44:09.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.012+0000 7ffad5d9b700 1 -- 172.21.15.114:0/1919894326 learned_addr learned my addr 172.21.15.114:0/1919894326 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:09.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.013+0000 7ffad5d9b700 1 -- 172.21.15.114:0/1919894326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 msgr2=0x55abdfd45790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:44:09.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.013+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/1919894326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd45790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.013+0000 7ffad5d9b700 1 -- 172.21.15.114:0/1919894326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 msgr2=0x55abdfd40410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:09.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.013+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/1919894326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd40410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.013+0000 7ffad5d9b700 1 -- 172.21.15.114:0/1919894326 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abdfbae950 con 0x55abdfd45cd0 2024-05-27T07:44:09.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.013+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/1919894326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd3fac0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffac800ca80 tx=0x7ffac800cd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3f3922d6838f38d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:09.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.013+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffac40b2aa0 0x7ffac40b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:09.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.014+0000 7ffaccff9700 1 -- 172.21.15.114:0/1919894326 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac800f040 con 0x55abdfd45cd0 2024-05-27T07:44:09.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.014+0000 7ffaccff9700 1 -- 172.21.15.114:0/1919894326 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffac8018070 con 0x55abdfd45cd0 2024-05-27T07:44:09.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.014+0000 7ffaccff9700 1 -- 172.21.15.114:0/1919894326 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac8012390 con 0x55abdfd45cd0 2024-05-27T07:44:09.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.015+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffac40b2aa0 0x7ffac40b4fb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ffac8013fd0 tx=0x7ffac8013f00 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:09.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.016+0000 7ffaeb964b80 1 -- 172.21.15.114:0/1919894326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 msgr2=0x55abdfd3fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:09.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.016+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/1919894326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd3fac0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffac800ca80 tx=0x7ffac800cd90 comp rx=0 tx=0).stop 2024-05-27T07:44:09.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.019+0000 7ffaeb964b80 1 -- 172.21.15.114:0/1919894326 shutdown_connections 2024-05-27T07:44:09.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.019+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/1919894326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd40410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.019+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/1919894326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd45790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.019+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/1919894326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd3fac0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.019+0000 7ffaeb964b80 1 -- 172.21.15.114:0/1919894326 >> 172.21.15.114:0/1919894326 conn(0x55abdfc1cb70 msgr2=0x55abdfd3a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:09.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.019+0000 7ffaeb964b80 1 -- 172.21.15.114:0/1919894326 shutdown_connections 2024-05-27T07:44:09.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.019+0000 7ffaeb964b80 1 -- 172.21.15.114:0/1919894326 wait complete. 2024-05-27T07:44:09.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.020+0000 7ffaeb964b80 1 -- start start 2024-05-27T07:44:09.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.020+0000 7ffaeb964b80 1 Processor -- start 2024-05-27T07:44:09.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.021+0000 7ffaeb964b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd49530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.021+0000 7ffaeb964b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd49a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.021+0000 7ffaeb964b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd49fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.021+0000 7ffad659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd49530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:09.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.021+0000 7ffad659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd49530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:47396/0 (socket says 172.21.15.114:47396) 2024-05-27T07:44:09.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.021+0000 7ffad659c700 1 -- 172.21.15.114:0/427717870 learned_addr learned my addr 172.21.15.114:0/427717870 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:09.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.021+0000 7ffaeb964b80 1 -- 172.21.15.114:0/427717870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55abdfb65020 con 0x55abdfd40000 2024-05-27T07:44:09.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.021+0000 7ffaeb964b80 1 -- 172.21.15.114:0/427717870 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55abdfc00840 con 0x55abdfbc1ec0 2024-05-27T07:44:09.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.021+0000 7ffaeb964b80 1 -- 172.21.15.114:0/427717870 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55abdfd4c810 con 0x55abdfd45cd0 2024-05-27T07:44:09.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.022+0000 7ffaeb964b80 10 client.0 _ll_register_callbacks cb 0x55abdfbf99a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:09.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.022+0000 7ffad659c700 1 -- 172.21.15.114:0/427717870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 msgr2=0x55abdfd49fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:09.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.022+0000 7ffad659c700 1 --2- 172.21.15.114:0/427717870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd49fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.022+0000 7ffad659c700 1 -- 172.21.15.114:0/427717870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 msgr2=0x55abdfd49a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:09.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.022+0000 7ffad659c700 1 --2- 172.21.15.114:0/427717870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd49a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.022+0000 7ffad659c700 1 -- 172.21.15.114:0/427717870 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abdfbfd710 con 0x55abdfbc1ec0 2024-05-27T07:44:09.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.023+0000 7ffad659c700 1 --2- 172.21.15.114:0/427717870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd49530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffac0015000 tx=0x7ffac0005330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2ede9417725bcc1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:09.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.023+0000 7ffabdffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:44:09.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.023+0000 7ffabdffb700 1 -- 172.21.15.114:0/427717870 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac000b9d0 con 0x55abdfbc1ec0 2024-05-27T07:44:09.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.023+0000 7ffabdffb700 1 -- 172.21.15.114:0/427717870 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ffac000bb70 con 0x55abdfbc1ec0 2024-05-27T07:44:09.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.025+0000 7ffaeb964b80 1 -- 172.21.15.114:0/427717870 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffac8018070 con 0x55abdfbc1ec0 2024-05-27T07:44:09.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.025+0000 7ffabdffb700 1 -- 172.21.15.114:0/427717870 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffac0003680 con 0x55abdfbc1ec0 2024-05-27T07:44:09.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.025+0000 7ffaeb964b80 10 client.25886 fetch_fsmap learned FSMap version 29 2024-05-27T07:44:09.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.025+0000 7ffaeb964b80 1 -- 172.21.15.114:0/427717870 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55abdfd57160 con 0x55abdfbc1ec0 2024-05-27T07:44:09.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.026+0000 7ffabdffb700 1 -- 172.21.15.114:0/427717870 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffac000bdb0 con 0x55abdfbc1ec0 2024-05-27T07:44:09.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.026+0000 7ffabdffb700 1 -- 172.21.15.114:0/427717870 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7ffac0021420 con 0x55abdfbc1ec0 2024-05-27T07:44:09.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.026+0000 7ffaeb964b80 10 client.25886 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:44:09.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.026+0000 7ffaeb964b80 10 client.25886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:09.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.027+0000 7ffaeb964b80 20 client.25886 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:09.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.027+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/427717870 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55abdfd597a0 0x55abdfd5bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.027+0000 7ffaeb964b80 4 client.25886 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:44:09.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.027+0000 7ffaeb964b80 1 -- 172.21.15.114:0/427717870 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ffac8018070 con 0x55abdfd597a0 2024-05-27T07:44:09.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.028+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/427717870 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55abdfd597a0 0x55abdfd5bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:09.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.031+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/427717870 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55abdfd597a0 0x55abdfd5bbc0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1f12ce039b152a5 server_cookie=87e9e9ddfc4c8b21 in_seq=0 out_seq=0 2024-05-27T07:44:09.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.031+0000 7ffabdffb700 10 client.25886 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:09.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.033+0000 7ffabdffb700 1 -- 172.21.15.114:0/427717870 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7ffac8018070 con 0x55abdfd597a0 2024-05-27T07:44:09.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.033+0000 7ffabdffb700 10 client.25886 handle_command_reply: tid=0 2024-05-27T07:44:09.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.033+0000 7ffaeb964b80 1 client.25886 shutdown 2024-05-27T07:44:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.037+0000 7ffaeb964b80 1 -- 172.21.15.114:0/427717870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 msgr2=0x55abdfd49530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.037+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/427717870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd49530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffac0015000 tx=0x7ffac0005330 comp rx=0 tx=0).stop 2024-05-27T07:44:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.037+0000 7ffaeb964b80 1 -- 172.21.15.114:0/427717870 shutdown_connections 2024-05-27T07:44:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.037+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/427717870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd49fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.037+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/427717870 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55abdfd597a0 0x55abdfd5bbc0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.037+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/427717870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd49530 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.037+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/427717870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd49a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.037+0000 7ffaeb964b80 1 -- 172.21.15.114:0/427717870 >> 172.21.15.114:0/427717870 conn(0x55abdfc1cb70 msgr2=0x55abdfd3eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.037+0000 7ffabdffb700 0 client.25886 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:09.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.039+0000 7ffaeb964b80 1 -- 172.21.15.114:0/427717870 shutdown_connections 2024-05-27T07:44:09.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.039+0000 7ffaeb964b80 1 -- 172.21.15.114:0/427717870 wait complete. 2024-05-27T07:44:09.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.040+0000 7ffaeb964b80 20 client.25886 trim_cache size 0 max 16384 2024-05-27T07:44:09.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.079+0000 7ffaeb964b80 1 Processor -- start 2024-05-27T07:44:09.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.079+0000 7ffaeb964b80 1 -- start start 2024-05-27T07:44:09.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.079+0000 7ffaeb964b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd3fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.079+0000 7ffaeb964b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd48490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.079+0000 7ffaeb964b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd43ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.079+0000 7ffaeb964b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffad81bd600 con 0x55abdfd40000 2024-05-27T07:44:09.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.079+0000 7ffaeb964b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffad81bfe30 con 0x55abdfbc1ec0 2024-05-27T07:44:09.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.079+0000 7ffaeb964b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffad81bdc70 con 0x55abdfd45cd0 2024-05-27T07:44:09.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.079+0000 7ffad5d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd48490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:09.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.080+0000 7ffad5d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd48490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60806/0 (socket says 172.21.15.114:60806) 2024-05-27T07:44:09.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.080+0000 7ffad5d9b700 1 -- 172.21.15.114:0/2988113606 learned_addr learned my addr 172.21.15.114:0/2988113606 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:09.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.080+0000 7ffad659c700 1 --2- 172.21.15.114:0/2988113606 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd3fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:09.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.080+0000 7ffad6d9d700 1 --2- 172.21.15.114:0/2988113606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd43ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:09.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.080+0000 7ffad5d9b700 1 -- 172.21.15.114:0/2988113606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 msgr2=0x55abdfd43ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:09.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.080+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/2988113606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd43ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.080+0000 7ffad5d9b700 1 -- 172.21.15.114:0/2988113606 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 msgr2=0x55abdfd3fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:09.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.080+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/2988113606 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd3fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.080+0000 7ffad5d9b700 1 -- 172.21.15.114:0/2988113606 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abdfd5b7b0 con 0x55abdfd40000 2024-05-27T07:44:09.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.080+0000 7ffad659c700 1 --2- 172.21.15.114:0/2988113606 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd3fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:09.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.080+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/2988113606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd48490 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ffac800ca50 tx=0x7ffac801f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd805d94f702223c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:09.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.080+0000 7ffabffff700 1 -- 172.21.15.114:0/2988113606 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac8012390 con 0x55abdfd40000 2024-05-27T07:44:09.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.080+0000 7ffabffff700 1 -- 172.21.15.114:0/2988113606 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffac8018040 con 0x55abdfd40000 2024-05-27T07:44:09.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.081+0000 7ffabffff700 1 -- 172.21.15.114:0/2988113606 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac801db40 con 0x55abdfd40000 2024-05-27T07:44:09.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.081+0000 7ffaeb964b80 1 -- 172.21.15.114:0/2988113606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 msgr2=0x55abdfd48490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:09.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.081+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/2988113606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd48490 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ffac800ca50 tx=0x7ffac801f040 comp rx=0 tx=0).stop 2024-05-27T07:44:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.082+0000 7ffaeb964b80 1 -- 172.21.15.114:0/2988113606 shutdown_connections 2024-05-27T07:44:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.082+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/2988113606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd43ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.082+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/2988113606 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd3fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.082+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/2988113606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd48490 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.082+0000 7ffaeb964b80 1 -- 172.21.15.114:0/2988113606 >> 172.21.15.114:0/2988113606 conn(0x55abdfc1cb70 msgr2=0x55abdfd463a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.082+0000 7ffaeb964b80 1 -- 172.21.15.114:0/2988113606 shutdown_connections 2024-05-27T07:44:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.082+0000 7ffaeb964b80 1 -- 172.21.15.114:0/2988113606 wait complete. 2024-05-27T07:44:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.082+0000 7ffaeb964b80 1 -- start start 2024-05-27T07:44:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.082+0000 7ffaeb964b80 1 Processor -- start 2024-05-27T07:44:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffaeb964b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd52250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffaeb964b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd4ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffaeb964b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd4f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffaeb964b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffad81be2f0 con 0x55abdfd40000 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffaeb964b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffad81bf760 con 0x55abdfbc1ec0 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffaeb964b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffad81be9c0 con 0x55abdfd45cd0 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffad659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd52250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffad659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd52250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:47414/0 (socket says 172.21.15.114:47414) 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffad659c700 1 -- 172.21.15.114:0/3710326299 learned_addr learned my addr 172.21.15.114:0/3710326299 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffaeb964b80 10 client.0 _ll_register_callbacks cb 0x55abdfbf99a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/3710326299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd4ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffad6d9d700 1 --2- 172.21.15.114:0/3710326299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd4f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffad5d9b700 1 -- 172.21.15.114:0/3710326299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 msgr2=0x55abdfd4f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/3710326299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd4f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffad5d9b700 1 -- 172.21.15.114:0/3710326299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 msgr2=0x55abdfd52250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/3710326299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd52250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffad5d9b700 1 -- 172.21.15.114:0/3710326299 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abdfd4f7d0 con 0x55abdfd40000 2024-05-27T07:44:09.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffad5d9b700 1 --2- 172.21.15.114:0/3710326299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd4ea70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ffac800ca50 tx=0x7ffac8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f3a81ed177d5c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:09.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.083+0000 7ffabdffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:44:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.084+0000 7fd668ff9640 1 -- 172.21.15.114:0/1925745834 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55d78c4d1360 con 0x55d78c1561c0 2024-05-27T07:44:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.084+0000 7fd668ff9640 10 client.16359 handle_command_reply: tid=0 2024-05-27T07:44:09.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.084+0000 7ffabdffb700 1 -- 172.21.15.114:0/3710326299 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac8012630 con 0x55abdfd40000 2024-05-27T07:44:09.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.084+0000 7ffabdffb700 1 -- 172.21.15.114:0/3710326299 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ffac8005a00 con 0x55abdfd40000 2024-05-27T07:44:09.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.084+0000 7fd68e5f7740 1 client.16359 shutdown 2024-05-27T07:44:09.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.084+0000 7ffaeb964b80 1 -- 172.21.15.114:0/3710326299 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffac8018040 con 0x55abdfd40000 2024-05-27T07:44:09.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.084+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1925745834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 msgr2=0x55d78c5e48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:09.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.084+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1925745834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 0x55d78c5e48c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd67c0c5c90 tx=0x7fd67c009260 comp rx=0 tx=0).stop 2024-05-27T07:44:09.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.084+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1925745834 shutdown_connections 2024-05-27T07:44:09.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.084+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1925745834 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d78c1561c0 0x55d78c5ee130 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.084+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1925745834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d78c1c21b0 0x55d78c574d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.084+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1925745834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d78c1a14d0 0x55d78c5e4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.084+0000 7fd68e5f7740 1 --2- 172.21.15.114:0/1925745834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d78c1992f0 0x55d78c5e48c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.084+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1925745834 >> 172.21.15.114:0/1925745834 conn(0x55d78c1b6300 msgr2=0x55d78c56ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:09.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.084+0000 7fd668ff9640 0 client.16359 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:09.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.085+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1925745834 shutdown_connections 2024-05-27T07:44:09.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.085+0000 7fd68e5f7740 1 -- 172.21.15.114:0/1925745834 wait complete. 2024-05-27T07:44:09.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.085+0000 7fd68e5f7740 20 client.16359 trim_cache size 0 max 16384 2024-05-27T07:44:09.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:44:09.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:44:09.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:44:09.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.086+0000 7fd687667640 1 -- 172.21.15.114:0/2950030973 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd66c0b29d0 msgr2=0x7fd66c0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.086+0000 7fd687667640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd66c0b29d0 0x7fd66c0b4ee0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55d78c560540 tx=0x7fd678009500 comp rx=0 tx=0).stop 2024-05-27T07:44:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.086+0000 7fd687667640 1 -- 172.21.15.114:0/2950030973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6801032e0 msgr2=0x7fd6801a0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.087+0000 7fd687667640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6801032e0 0x7fd6801a0570 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd67c032fd0 tx=0x7fd67c031a20 comp rx=0 tx=0).stop 2024-05-27T07:44:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.087+0000 7fd687667640 1 -- 172.21.15.114:0/2950030973 shutdown_connections 2024-05-27T07:44:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.087+0000 7fd687667640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd66c0b29d0 0x7fd66c0b4ee0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.087+0000 7fd687667640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd680109030 0x7fd68019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.087+0000 7fd687667640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd680108b90 0x7fd6801a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.087+0000 7fd687667640 1 --2- 172.21.15.114:0/2950030973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6801032e0 0x7fd6801a0570 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.087+0000 7fd687667640 1 -- 172.21.15.114:0/2950030973 >> 172.21.15.114:0/2950030973 conn(0x7fd6800782d0 msgr2=0x7fd68006a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.087+0000 7fd687667640 1 -- 172.21.15.114:0/2950030973 shutdown_connections 2024-05-27T07:44:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:09.087+0000 7fd687667640 1 -- 172.21.15.114:0/2950030973 wait complete. 2024-05-27T07:44:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.088+0000 7ffabdffb700 1 -- 172.21.15.114:0/3710326299 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffac8005c00 con 0x55abdfd40000 2024-05-27T07:44:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.088+0000 7ffaeb964b80 10 client.16371 fetch_fsmap learned FSMap version 29 2024-05-27T07:44:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.088+0000 7ffaeb964b80 1 -- 172.21.15.114:0/3710326299 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55abdfdc5230 con 0x55abdfd40000 2024-05-27T07:44:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.089+0000 7ffabdffb700 1 -- 172.21.15.114:0/3710326299 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffac8018040 con 0x55abdfd40000 2024-05-27T07:44:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.090+0000 7ffabdffb700 1 -- 172.21.15.114:0/3710326299 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7ffac8012070 con 0x55abdfd40000 2024-05-27T07:44:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.091+0000 7ffaeb964b80 10 client.16371 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:44:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.091+0000 7ffaeb964b80 10 client.16371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.091+0000 7ffaeb964b80 20 client.16371 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.091+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/3710326299 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55abdfdc5620 0x55abdfdc7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.091+0000 7ffaeb964b80 4 client.16371 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:44:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.091+0000 7ffaeb964b80 1 -- 172.21.15.114:0/3710326299 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55abdfd0ba70 con 0x55abdfdc5620 2024-05-27T07:44:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.093+0000 7ffad659c700 1 --2- 172.21.15.114:0/3710326299 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55abdfdc5620 0x55abdfdc7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.094+0000 7ffabdffb700 10 client.16371 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.094+0000 7ffad659c700 1 --2- 172.21.15.114:0/3710326299 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55abdfdc5620 0x55abdfdc7a40 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f8278a9ff7799c4 server_cookie=854fb98054fa7349 in_seq=0 out_seq=0 2024-05-27T07:44:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.095+0000 7ffabdffb700 1 -- 172.21.15.114:0/3710326299 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55abdfd0ba70 con 0x55abdfdc5620 2024-05-27T07:44:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.095+0000 7ffabdffb700 10 client.16371 handle_command_reply: tid=0 2024-05-27T07:44:09.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.095+0000 7ffaeb964b80 1 client.16371 shutdown 2024-05-27T07:44:09.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.096+0000 7ffaeb964b80 1 -- 172.21.15.114:0/3710326299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 msgr2=0x55abdfd4ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:09.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.096+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/3710326299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd4ea70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ffac800ca50 tx=0x7ffac8013000 comp rx=0 tx=0).stop 2024-05-27T07:44:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.096+0000 7ffaeb964b80 1 -- 172.21.15.114:0/3710326299 shutdown_connections 2024-05-27T07:44:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.096+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/3710326299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abdfd45cd0 0x55abdfd4f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.096+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/3710326299 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55abdfdc5620 0x55abdfdc7a40 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.096+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/3710326299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55abdfbc1ec0 0x55abdfd52250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.096+0000 7ffaeb964b80 1 --2- 172.21.15.114:0/3710326299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55abdfd40000 0x55abdfd4ea70 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.096+0000 7ffaeb964b80 1 -- 172.21.15.114:0/3710326299 >> 172.21.15.114:0/3710326299 conn(0x55abdfc1cb70 msgr2=0x55abdfd43820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.096+0000 7ffabdffb700 0 client.16371 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.096+0000 7ffaeb964b80 1 -- 172.21.15.114:0/3710326299 shutdown_connections 2024-05-27T07:44:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.096+0000 7ffaeb964b80 1 -- 172.21.15.114:0/3710326299 wait complete. 2024-05-27T07:44:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.096+0000 7ffaeb964b80 20 client.16371 trim_cache size 0 max 16384 2024-05-27T07:44:09.096 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:44:09.097 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:44:09.097 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "448e6cb1-e84d-40c5-a121-ffda8e564730", 2024-05-27T07:44:09.097 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:44:09.097 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:44:09.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.097+0000 7ffadcf4f700 1 -- 172.21.15.114:0/3032362018 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffac40b2aa0 msgr2=0x7ffac40b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:09.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.097+0000 7ffadcf4f700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffac40b2aa0 0x7ffac40b4fb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ffac8013fd0 tx=0x7ffac8013f00 comp rx=0 tx=0).stop 2024-05-27T07:44:09.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.097+0000 7ffadcf4f700 1 -- 172.21.15.114:0/3032362018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffad811d920 msgr2=0x7ffad81ae630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:09.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.097+0000 7ffadcf4f700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffad811d920 0x7ffad81ae630 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ffad000b7b0 tx=0x7ffad000bac0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.097+0000 7ffadcf4f700 1 -- 172.21.15.114:0/3032362018 shutdown_connections 2024-05-27T07:44:09.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.097+0000 7ffadcf4f700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffad811d920 0x7ffad81ae630 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.097+0000 7ffadcf4f700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffac40b2aa0 0x7ffac40b4fb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.097+0000 7ffadcf4f700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffad810a920 0x7ffad81a9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.097+0000 7ffadcf4f700 1 --2- 172.21.15.114:0/3032362018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffad810b270 0x7ffad81aa3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:09.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.097+0000 7ffadcf4f700 1 -- 172.21.15.114:0/3032362018 >> 172.21.15.114:0/3032362018 conn(0x7ffad806f4c0 msgr2=0x7ffad811b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:09.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.097+0000 7ffadcf4f700 1 -- 172.21.15.114:0/3032362018 shutdown_connections 2024-05-27T07:44:09.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:09.097+0000 7ffadcf4f700 1 -- 172.21.15.114:0/3032362018 wait complete. 2024-05-27T07:44:09.244 INFO:tasks.workunit.client.0.smithi114.stdout: 1 359533 12.75 MB/sec execute 21 sec latency 171.742 ms 2024-05-27T07:44:09.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:09 smithi114 ceph-mon[32038]: pgmap v139: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 432 op/s 2024-05-27T07:44:09.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:09 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:44:09.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:09 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:44:09.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:09 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:44:09.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:44:09.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:09 smithi163 ceph-mon[36451]: pgmap v139: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 432 op/s 2024-05-27T07:44:09.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:09 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:44:09.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:09 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:44:09.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:09 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:44:09.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:09 smithi186 ceph-mon[36647]: pgmap v139: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 432 op/s 2024-05-27T07:44:09.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:09 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:44:09.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:09 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:44:09.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:09 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:44:10.117 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:44:10.244 INFO:tasks.workunit.client.0.smithi114.stdout: 1 362458 12.94 MB/sec execute 22 sec latency 12.159 ms 2024-05-27T07:44:10.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:10 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:44:10.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:10 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:44:10.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:10 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:44:11.244 INFO:tasks.workunit.client.0.smithi114.stdout: 1 365553 13.19 MB/sec execute 23 sec latency 14.841 ms 2024-05-27T07:44:11.889 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:11 smithi114 ceph-mon[32038]: pgmap v140: 129 pgs: 129 active+clean; 49 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 578 op/s 2024-05-27T07:44:11.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.904+0000 7f04d9388700 1 -- 172.21.15.114:0/2806579108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04d4107de0 msgr2=0x7f04d41081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:11.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.904+0000 7f04d9388700 1 --2- 172.21.15.114:0/2806579108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04d4107de0 0x7f04d41081f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f04c80099c0 tx=0x7f04c8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.904+0000 7f04d9388700 1 -- 172.21.15.114:0/2806579108 shutdown_connections 2024-05-27T07:44:11.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.904+0000 7f04d9388700 1 --2- 172.21.15.114:0/2806579108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04d4108730 0x7f04d4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.904+0000 7f04d9388700 1 --2- 172.21.15.114:0/2806579108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d41034b0 0x7f04d41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.904+0000 7f04d9388700 1 --2- 172.21.15.114:0/2806579108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04d4107de0 0x7f04d41081f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.904+0000 7f04d9388700 1 -- 172.21.15.114:0/2806579108 >> 172.21.15.114:0/2806579108 conn(0x7f04d40fede0 msgr2=0x7f04d4101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:11.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.904+0000 7f04d9388700 1 -- 172.21.15.114:0/2806579108 shutdown_connections 2024-05-27T07:44:11.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.904+0000 7f04d9388700 1 -- 172.21.15.114:0/2806579108 wait complete. 2024-05-27T07:44:11.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.904+0000 7f04d9388700 1 Processor -- start 2024-05-27T07:44:11.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d9388700 1 -- start start 2024-05-27T07:44:11.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d9388700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04d41034b0 0x7f04d419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d9388700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04d4107de0 0x7f04d419b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d9388700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d4108730 0x7f04d419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d9388700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04d4113340 con 0x7f04d4107de0 2024-05-27T07:44:11.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d9388700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04d41131c0 con 0x7f04d4108730 2024-05-27T07:44:11.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d9388700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04d41134c0 con 0x7f04d41034b0 2024-05-27T07:44:11.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04d41034b0 0x7f04d419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d27fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04d4107de0 0x7f04d419b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04d41034b0 0x7f04d419ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34648/0 (socket says 172.21.15.114:34648) 2024-05-27T07:44:11.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d27fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04d4107de0 0x7f04d419b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60842/0 (socket says 172.21.15.114:60842) 2024-05-27T07:44:11.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d2ffd700 1 -- 172.21.15.114:0/2408073022 learned_addr learned my addr 172.21.15.114:0/2408073022 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:11.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d37fe700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d4108730 0x7f04d419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d37fe700 1 -- 172.21.15.114:0/2408073022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04d41034b0 msgr2=0x7f04d419ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:11.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d37fe700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04d41034b0 0x7f04d419ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d37fe700 1 -- 172.21.15.114:0/2408073022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04d4107de0 msgr2=0x7f04d419b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:11.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d37fe700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04d4107de0 0x7f04d419b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d37fe700 1 -- 172.21.15.114:0/2408073022 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d419fcd0 con 0x7f04d4108730 2024-05-27T07:44:11.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.905+0000 7f04d27fc700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04d4107de0 0x7f04d419b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:11.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.906+0000 7f04d37fe700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d4108730 0x7f04d419f630 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f04cc00efc0 tx=0x7f04cc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:11.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.906+0000 7f04bbfff700 1 -- 172.21.15.114:0/2408073022 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc0092e0 con 0x7f04d4108730 2024-05-27T07:44:11.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.906+0000 7f04d9388700 1 -- 172.21.15.114:0/2408073022 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d419ff60 con 0x7f04d4108730 2024-05-27T07:44:11.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.906+0000 7f04bbfff700 1 -- 172.21.15.114:0/2408073022 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04cc004bb0 con 0x7f04d4108730 2024-05-27T07:44:11.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.906+0000 7f04bbfff700 1 -- 172.21.15.114:0/2408073022 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc004020 con 0x7f04d4108730 2024-05-27T07:44:11.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.906+0000 7f04d9388700 1 -- 172.21.15.114:0/2408073022 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d41a0470 con 0x7f04d4108730 2024-05-27T07:44:11.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.909+0000 7f04bbfff700 1 -- 172.21.15.114:0/2408073022 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f04cc01b030 con 0x7f04d4108730 2024-05-27T07:44:11.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.910+0000 7f04bbfff700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04bc0b29f0 0x7f04bc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.910+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04bc0b29f0 0x7f04bc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.910+0000 7f04e7d9db80 1 Processor -- start 2024-05-27T07:44:11.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.910+0000 7f04bbfff700 1 -- 172.21.15.114:0/2408073022 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f04cc0e2460 con 0x7f04d4108730 2024-05-27T07:44:11.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.910+0000 7f04e7d9db80 1 -- start start 2024-05-27T07:44:11.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.910+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04bc0b29f0 0x7f04bc0b4f00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009d20 tx=0x7f04c4009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:11.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.910+0000 7f04e7d9db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 0x5632264d7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.910+0000 7f04e7d9db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04e7d9db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d1f80 0x5632264d2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04e7d9db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563226400c90 con 0x5632264d1f80 2024-05-27T07:44:11.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04e7d9db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5632263f0830 con 0x563226353e40 2024-05-27T07:44:11.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04e7d9db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5632263f4870 con 0x5632264d7c50 2024-05-27T07:44:11.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 0x5632264d7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04d27fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 0x5632264d7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:47456/0 (socket says 172.21.15.114:47456) 2024-05-27T07:44:11.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04d37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d1f80 0x5632264d2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04d2ffd700 1 -- 172.21.15.114:0/1140113992 learned_addr learned my addr 172.21.15.114:0/1140113992 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:11.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04d27fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264d1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34652/0 (socket says 172.21.15.114:34652) 2024-05-27T07:44:11.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04d2ffd700 1 -- 172.21.15.114:0/1140113992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 msgr2=0x5632264d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:11.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/1140113992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264d1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04d2ffd700 1 -- 172.21.15.114:0/1140113992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d1f80 msgr2=0x5632264d2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:11.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/1140113992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d1f80 0x5632264d2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04d2ffd700 1 -- 172.21.15.114:0/1140113992 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632263408d0 con 0x563226353e40 2024-05-27T07:44:11.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04d27fc700 1 --2- 172.21.15.114:0/1140113992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264d1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:11.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/1140113992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 0x5632264d7710 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f04c400e540 tx=0x7f04c4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fc50c22e42cc38a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:11.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04b97fa700 1 -- 172.21.15.114:0/1140113992 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c4026430 con 0x563226353e40 2024-05-27T07:44:11.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04b97fa700 1 -- 172.21.15.114:0/1140113992 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04c401c8c0 con 0x563226353e40 2024-05-27T07:44:11.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.911+0000 7f04b97fa700 1 -- 172.21.15.114:0/1140113992 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c401d8f0 con 0x563226353e40 2024-05-27T07:44:11.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.912+0000 7f04e7d9db80 1 -- 172.21.15.114:0/1140113992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 msgr2=0x5632264d7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:11.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.912+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/1140113992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 0x5632264d7710 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f04c400e540 tx=0x7f04c4014000 comp rx=0 tx=0).stop 2024-05-27T07:44:11.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.913+0000 7f04e7d9db80 1 -- 172.21.15.114:0/1140113992 shutdown_connections 2024-05-27T07:44:11.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.913+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/1140113992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d1f80 0x5632264d2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.913+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/1140113992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 0x5632264d7710 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.913+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/1140113992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264d1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.913+0000 7f04e7d9db80 1 -- 172.21.15.114:0/1140113992 >> 172.21.15.114:0/1140113992 conn(0x5632263aeaf0 msgr2=0x5632264cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:11.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.913+0000 7f04e7d9db80 1 -- 172.21.15.114:0/1140113992 shutdown_connections 2024-05-27T07:44:11.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.913+0000 7f04e7d9db80 1 -- 172.21.15.114:0/1140113992 wait complete. 2024-05-27T07:44:11.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.913+0000 7f04e7d9db80 1 -- start start 2024-05-27T07:44:11.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.913+0000 7f04e7d9db80 1 Processor -- start 2024-05-27T07:44:11.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04e7d9db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563226353e40 0x5632264db340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04e7d9db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632264d1f80 0x5632264db880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04e7d9db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d7c50 0x5632264dbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04e7d9db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5632262f6fa0 con 0x5632264d7c50 2024-05-27T07:44:11.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04e7d9db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5632263927c0 con 0x5632264d1f80 2024-05-27T07:44:11.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04e7d9db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5632264de740 con 0x563226353e40 2024-05-27T07:44:11.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04e7d9db80 10 client.0 _ll_register_callbacks cb 0x56322638b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:11.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04d37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d7c50 0x5632264dbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563226353e40 0x5632264db340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563226353e40 0x5632264db340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34654/0 (socket says 172.21.15.114:34654) 2024-05-27T07:44:11.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04d2ffd700 1 -- 172.21.15.114:0/3668272125 learned_addr learned my addr 172.21.15.114:0/3668272125 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:11.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04d27fc700 1 --2- 172.21.15.114:0/3668272125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632264d1f80 0x5632264db880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04d2ffd700 1 -- 172.21.15.114:0/3668272125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632264d1f80 msgr2=0x5632264db880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:11.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/3668272125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632264d1f80 0x5632264db880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04d2ffd700 1 -- 172.21.15.114:0/3668272125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d7c50 msgr2=0x5632264dbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:11.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/3668272125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d7c50 0x5632264dbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04d2ffd700 1 -- 172.21.15.114:0/3668272125 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56322638f690 con 0x563226353e40 2024-05-27T07:44:11.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/3668272125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563226353e40 0x5632264db340 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f04c400eee0 tx=0x7f04c401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80a0bb056e2beded server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:11.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04b6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:44:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.914+0000 7f04b6ff5700 1 -- 172.21.15.114:0/3668272125 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c401ca70 con 0x563226353e40 2024-05-27T07:44:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04e7d9db80 1 -- 172.21.15.114:0/3668272125 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04c401c8c0 con 0x563226353e40 2024-05-27T07:44:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04b6ff5700 1 -- 172.21.15.114:0/3668272125 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f04c401cc10 con 0x563226353e40 2024-05-27T07:44:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04b6ff5700 1 -- 172.21.15.114:0/3668272125 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f04c4005040 con 0x563226353e40 2024-05-27T07:44:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04e7d9db80 10 client.25972 fetch_fsmap learned FSMap version 29 2024-05-27T07:44:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04e7d9db80 1 -- 172.21.15.114:0/3668272125 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5632264e91a0 con 0x563226353e40 2024-05-27T07:44:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04b6ff5700 1 -- 172.21.15.114:0/3668272125 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04c401c8c0 con 0x563226353e40 2024-05-27T07:44:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04b6ff5700 1 -- 172.21.15.114:0/3668272125 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f04c4004c70 con 0x563226353e40 2024-05-27T07:44:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04e7d9db80 10 client.25972 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:44:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04e7d9db80 10 client.25972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04e7d9db80 20 client.25972 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/3668272125 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5632264eb6d0 0x5632264edaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04e7d9db80 4 client.25972 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:44:11.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04e7d9db80 1 -- 172.21.15.114:0/3668272125 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56322649d9f0 con 0x5632264eb6d0 2024-05-27T07:44:11.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.915+0000 7f04d27fc700 1 --2- 172.21.15.114:0/3668272125 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5632264eb6d0 0x5632264edaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.916+0000 7f04b6ff5700 10 client.25972 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:11.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.916+0000 7f04d27fc700 1 --2- 172.21.15.114:0/3668272125 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5632264eb6d0 0x5632264edaf0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76d77b0c9e42b942 server_cookie=e2f431940ff8cbb7 in_seq=0 out_seq=0 2024-05-27T07:44:11.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.916+0000 7f04b6ff5700 1 -- 172.21.15.114:0/3668272125 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56322649d9f0 con 0x5632264eb6d0 2024-05-27T07:44:11.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.916+0000 7f04b6ff5700 10 client.25972 handle_command_reply: tid=0 2024-05-27T07:44:11.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.917+0000 7f04e7d9db80 1 client.25972 shutdown 2024-05-27T07:44:11.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.917+0000 7f04e7d9db80 1 -- 172.21.15.114:0/3668272125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563226353e40 msgr2=0x5632264db340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:11.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.917+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/3668272125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563226353e40 0x5632264db340 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f04c400eee0 tx=0x7f04c401c410 comp rx=0 tx=0).stop 2024-05-27T07:44:11.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.917+0000 7f04e7d9db80 1 -- 172.21.15.114:0/3668272125 shutdown_connections 2024-05-27T07:44:11.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.917+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/3668272125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d7c50 0x5632264dbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.917+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/3668272125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563226353e40 0x5632264db340 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.917+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/3668272125 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5632264eb6d0 0x5632264edaf0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.917+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/3668272125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632264d1f80 0x5632264db880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.917+0000 7f04b6ff5700 0 client.25972 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:11.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.917+0000 7f04e7d9db80 1 -- 172.21.15.114:0/3668272125 >> 172.21.15.114:0/3668272125 conn(0x5632263aeaf0 msgr2=0x5632264d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:11.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.917+0000 7f04e7d9db80 1 -- 172.21.15.114:0/3668272125 shutdown_connections 2024-05-27T07:44:11.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.917+0000 7f04e7d9db80 1 -- 172.21.15.114:0/3668272125 wait complete. 2024-05-27T07:44:11.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.917+0000 7f04e7d9db80 20 client.25972 trim_cache size 0 max 16384 2024-05-27T07:44:11.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.938+0000 7f04e7d9db80 1 Processor -- start 2024-05-27T07:44:11.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.938+0000 7f04e7d9db80 1 -- start start 2024-05-27T07:44:11.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04e7d9db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 0x5632264d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04e7d9db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d1f80 0x5632264e79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04e7d9db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264d5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04e7d9db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04d41ac410 con 0x5632264d1f80 2024-05-27T07:44:11.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04e7d9db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04d41aec70 con 0x563226353e40 2024-05-27T07:44:11.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04e7d9db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04d41aca60 con 0x5632264d7c50 2024-05-27T07:44:11.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 0x5632264d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 0x5632264d1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:47482/0 (socket says 172.21.15.114:47482) 2024-05-27T07:44:11.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04d27fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d1f80 0x5632264e79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04d2ffd700 1 -- 172.21.15.114:0/1760934556 learned_addr learned my addr 172.21.15.114:0/1760934556 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:11.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04d37fe700 1 --2- 172.21.15.114:0/1760934556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264d5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04d2ffd700 1 -- 172.21.15.114:0/1760934556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 msgr2=0x5632264d5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:11.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/1760934556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264d5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04d2ffd700 1 -- 172.21.15.114:0/1760934556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d1f80 msgr2=0x5632264e79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:11.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/1760934556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d1f80 0x5632264e79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04d2ffd700 1 -- 172.21.15.114:0/1760934556 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632264d98b0 con 0x563226353e40 2024-05-27T07:44:11.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.939+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/1760934556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 0x5632264d1a40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f04c4011ab0 tx=0x7f04c401d6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46f9863033cca946 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:11.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.940+0000 7f04b8ff9700 1 -- 172.21.15.114:0/1760934556 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c4009e60 con 0x563226353e40 2024-05-27T07:44:11.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.940+0000 7f04b8ff9700 1 -- 172.21.15.114:0/1760934556 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04c4005690 con 0x563226353e40 2024-05-27T07:44:11.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.940+0000 7f04b8ff9700 1 -- 172.21.15.114:0/1760934556 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c401ca20 con 0x563226353e40 2024-05-27T07:44:11.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.940+0000 7f04e7d9db80 1 -- 172.21.15.114:0/1760934556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 msgr2=0x5632264d1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:11.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.940+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/1760934556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 0x5632264d1a40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f04c4011ab0 tx=0x7f04c401d6e0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 -- 172.21.15.114:0/1760934556 shutdown_connections 2024-05-27T07:44:11.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/1760934556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264d5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/1760934556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563226353e40 0x5632264d1a40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/1760934556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632264d1f80 0x5632264e79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 -- 172.21.15.114:0/1760934556 >> 172.21.15.114:0/1760934556 conn(0x5632263aeaf0 msgr2=0x5632264d5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:11.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 -- 172.21.15.114:0/1760934556 shutdown_connections 2024-05-27T07:44:11.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 -- 172.21.15.114:0/1760934556 wait complete. 2024-05-27T07:44:11.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 -- start start 2024-05-27T07:44:11.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 Processor -- start 2024-05-27T07:44:11.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563226353e40 0x5632264f12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632264d1f80 0x5632264ef0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264ef720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04d41ad130 con 0x563226353e40 2024-05-27T07:44:11.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04d41ae5a0 con 0x5632264d1f80 2024-05-27T07:44:11.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04d41ad800 con 0x5632264d7c50 2024-05-27T07:44:11.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04e7d9db80 10 client.0 _ll_register_callbacks cb 0x56322638b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:11.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563226353e40 0x5632264f12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563226353e40 0x5632264f12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60862/0 (socket says 172.21.15.114:60862) 2024-05-27T07:44:11.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04d2ffd700 1 -- 172.21.15.114:0/465290886 learned_addr learned my addr 172.21.15.114:0/465290886 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:11.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.941+0000 7f04d37fe700 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264ef720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.942+0000 7f04d27fc700 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632264d1f80 0x5632264ef0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.942+0000 7f04d2ffd700 1 -- 172.21.15.114:0/465290886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 msgr2=0x5632264ef720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:11.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.942+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264ef720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.942+0000 7f04d2ffd700 1 -- 172.21.15.114:0/465290886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632264d1f80 msgr2=0x5632264ef0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:11.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.942+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632264d1f80 0x5632264ef0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.942+0000 7f04d2ffd700 1 -- 172.21.15.114:0/465290886 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632264efea0 con 0x563226353e40 2024-05-27T07:44:11.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.942+0000 7f04d37fe700 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264ef720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:11.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.942+0000 7f04d2ffd700 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563226353e40 0x5632264f12d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f04c4005fa0 tx=0x7f04c4009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bedcd6c42dc452a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:11.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.942+0000 7f04b6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:44:11.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.942+0000 7f04b6ff5700 1 -- 172.21.15.114:0/465290886 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c4049070 con 0x563226353e40 2024-05-27T07:44:11.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.942+0000 7f04e7d9db80 1 -- 172.21.15.114:0/465290886 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04c4005690 con 0x563226353e40 2024-05-27T07:44:11.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.942+0000 7f04b6ff5700 1 -- 172.21.15.114:0/465290886 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f04c401de00 con 0x563226353e40 2024-05-27T07:44:11.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.943+0000 7f04b6ff5700 1 -- 172.21.15.114:0/465290886 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f04c40051e0 con 0x563226353e40 2024-05-27T07:44:11.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.943+0000 7f04e7d9db80 10 client.16389 fetch_fsmap learned FSMap version 29 2024-05-27T07:44:11.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.943+0000 7f04e7d9db80 1 -- 172.21.15.114:0/465290886 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563226555f70 con 0x563226353e40 2024-05-27T07:44:11.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.943+0000 7f04b6ff5700 1 -- 172.21.15.114:0/465290886 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04c4005690 con 0x563226353e40 2024-05-27T07:44:11.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.943+0000 7f04b6ff5700 1 -- 172.21.15.114:0/465290886 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f04c40047a0 con 0x563226353e40 2024-05-27T07:44:11.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.943+0000 7f04e7d9db80 10 client.16389 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:44:11.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.943+0000 7f04e7d9db80 10 client.16389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:11.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.943+0000 7f04e7d9db80 20 client.16389 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:11.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.943+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563226556560 0x563226558980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:11.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.943+0000 7f04e7d9db80 4 client.16389 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:44:11.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.943+0000 7f04e7d9db80 1 -- 172.21.15.114:0/465290886 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563226402be0 con 0x563226556560 2024-05-27T07:44:11.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.943+0000 7f04d27fc700 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563226556560 0x563226558980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:11.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.944+0000 7f04b6ff5700 10 client.16389 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:11.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.944+0000 7f04d27fc700 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563226556560 0x563226558980 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3f48589e507a1cb server_cookie=e2887de2d3e30579 in_seq=0 out_seq=0 2024-05-27T07:44:11.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04b6ff5700 1 -- 172.21.15.114:0/465290886 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563226402be0 con 0x563226556560 2024-05-27T07:44:11.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04b6ff5700 10 client.16389 handle_command_reply: tid=0 2024-05-27T07:44:11.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04e7d9db80 1 client.16389 shutdown 2024-05-27T07:44:11.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04e7d9db80 1 -- 172.21.15.114:0/465290886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563226353e40 msgr2=0x5632264f12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:11.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563226353e40 0x5632264f12d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f04c4005fa0 tx=0x7f04c4009360 comp rx=0 tx=0).stop 2024-05-27T07:44:11.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04e7d9db80 1 -- 172.21.15.114:0/465290886 shutdown_connections 2024-05-27T07:44:11.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632264d7c50 0x5632264ef720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563226353e40 0x5632264f12d0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563226556560 0x563226558980 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04e7d9db80 1 --2- 172.21.15.114:0/465290886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632264d1f80 0x5632264ef0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04e7d9db80 1 -- 172.21.15.114:0/465290886 >> 172.21.15.114:0/465290886 conn(0x5632263aeaf0 msgr2=0x5632264d6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:11.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04b6ff5700 0 client.16389 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:11.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04e7d9db80 1 -- 172.21.15.114:0/465290886 shutdown_connections 2024-05-27T07:44:11.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04e7d9db80 1 -- 172.21.15.114:0/465290886 wait complete. 2024-05-27T07:44:11.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.945+0000 7f04e7d9db80 20 client.16389 trim_cache size 0 max 16384 2024-05-27T07:44:11.950 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:44:11.950 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:44:11.950 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:44:11.950 INFO:teuthology.orchestra.run.smithi114.stdout: "448e6cb1-e84d-40c5-a121-ffda8e564730": { 2024-05-27T07:44:11.950 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:44:11.950 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "448e6cb1-e84d-40c5-a121-ffda8e564730", 2024-05-27T07:44:11.950 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:44:11.950 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:44:11.951 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:44:11.951 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:44:11.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.946+0000 7f04d9388700 1 -- 172.21.15.114:0/2408073022 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04bc0b29f0 msgr2=0x7f04bc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:11.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.946+0000 7f04d9388700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04bc0b29f0 0x7f04bc0b4f00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009d20 tx=0x7f04c4009260 comp rx=0 tx=0).stop 2024-05-27T07:44:11.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.946+0000 7f04d9388700 1 -- 172.21.15.114:0/2408073022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d4108730 msgr2=0x7f04d419f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:11.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.946+0000 7f04d9388700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d4108730 0x7f04d419f630 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f04cc00efc0 tx=0x7f04cc00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.946+0000 7f04d9388700 1 -- 172.21.15.114:0/2408073022 shutdown_connections 2024-05-27T07:44:11.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.946+0000 7f04d9388700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04bc0b29f0 0x7f04bc0b4f00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.946+0000 7f04d9388700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d4108730 0x7f04d419f630 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.946+0000 7f04d9388700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04d41034b0 0x7f04d419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.946+0000 7f04d9388700 1 --2- 172.21.15.114:0/2408073022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04d4107de0 0x7f04d419b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:11.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.946+0000 7f04d9388700 1 -- 172.21.15.114:0/2408073022 >> 172.21.15.114:0/2408073022 conn(0x7f04d40fede0 msgr2=0x7f04d4075d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:11.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.946+0000 7f04d9388700 1 -- 172.21.15.114:0/2408073022 shutdown_connections 2024-05-27T07:44:11.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:11.946+0000 7f04d9388700 1 -- 172.21.15.114:0/2408073022 wait complete. 2024-05-27T07:44:11.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:11 smithi163 ceph-mon[36451]: pgmap v140: 129 pgs: 129 active+clean; 49 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 578 op/s 2024-05-27T07:44:11.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:11 smithi186 ceph-mon[36647]: pgmap v140: 129 pgs: 129 active+clean; 49 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 578 op/s 2024-05-27T07:44:12.244 INFO:tasks.workunit.client.0.smithi114.stdout: 1 367540 12.90 MB/sec execute 24 sec latency 74.935 ms 2024-05-27T07:44:12.624 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:12 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:44:12.624 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:12 smithi163 systemd[1]: Stopped Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:44:12.625 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:12 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Consumed 2.969s CPU time. 2024-05-27T07:44:12.625 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:12 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Start request repeated too quickly. 2024-05-27T07:44:12.625 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:12 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-05-27T07:44:12.625 INFO:journalctl@ceph.mds.b.smithi163.stdout:May 27 07:44:12 smithi163 systemd[1]: Failed to start Ceph mds.b for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:44:12.841 INFO:tasks.cephfs.filesystem:scrub status for tag:448e6cb1-e84d-40c5-a121-ffda8e564730 - {'path': '/', 'tag': '448e6cb1-e84d-40c5-a121-ffda8e564730', 'options': 'recursive,force'} 2024-05-27T07:44:12.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:12 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:44:12.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:12 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:44:13.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:12 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:44:13.244 INFO:tasks.workunit.client.0.smithi114.stdout: 1 369550 12.92 MB/sec execute 25 sec latency 45.246 ms 2024-05-27T07:44:13.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:13 smithi163 ceph-mon[36451]: pgmap v141: 129 pgs: 129 active+clean; 41 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 557 op/s 2024-05-27T07:44:13.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:13 smithi186 ceph-mon[36647]: pgmap v141: 129 pgs: 129 active+clean; 41 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 557 op/s 2024-05-27T07:44:14.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:13 smithi114 ceph-mon[32038]: pgmap v141: 129 pgs: 129 active+clean; 41 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 557 op/s 2024-05-27T07:44:14.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 372680 13.14 MB/sec execute 26 sec latency 26.536 ms 2024-05-27T07:44:15.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 375402 13.13 MB/sec execute 27 sec latency 24.578 ms 2024-05-27T07:44:15.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:15 smithi163 ceph-mon[36451]: pgmap v142: 129 pgs: 129 active+clean; 41 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 424 op/s 2024-05-27T07:44:15.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:15 smithi186 ceph-mon[36647]: pgmap v142: 129 pgs: 129 active+clean; 41 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 424 op/s 2024-05-27T07:44:16.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:15 smithi114 ceph-mon[32038]: pgmap v142: 129 pgs: 129 active+clean; 41 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 424 op/s 2024-05-27T07:44:16.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 377660 13.08 MB/sec execute 28 sec latency 19.127 ms 2024-05-27T07:44:17.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 380139 13.11 MB/sec execute 29 sec latency 45.648 ms 2024-05-27T07:44:17.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:17 smithi163 ceph-mon[36451]: pgmap v143: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 548 op/s 2024-05-27T07:44:17.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:17 smithi186 ceph-mon[36647]: pgmap v143: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 548 op/s 2024-05-27T07:44:18.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:17 smithi114 ceph-mon[32038]: pgmap v143: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 548 op/s 2024-05-27T07:44:18.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 382533 13.09 MB/sec execute 30 sec latency 135.883 ms 2024-05-27T07:44:19.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 384808 13.05 MB/sec execute 31 sec latency 14.246 ms 2024-05-27T07:44:19.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:19 smithi163 ceph-mon[36451]: pgmap v144: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 402 op/s 2024-05-27T07:44:19.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:19 smithi163 ceph-mon[36451]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-05-27T07:44:19.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:19 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:44:19.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:19 smithi186 ceph-mon[36647]: pgmap v144: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 402 op/s 2024-05-27T07:44:19.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:19 smithi186 ceph-mon[36647]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-05-27T07:44:19.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:19 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:44:20.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:19 smithi114 ceph-mon[32038]: pgmap v144: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 402 op/s 2024-05-27T07:44:20.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:19 smithi114 ceph-mon[32038]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-05-27T07:44:20.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:19 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:44:20.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 387421 13.09 MB/sec execute 32 sec latency 24.323 ms 2024-05-27T07:44:21.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 390597 13.26 MB/sec execute 33 sec latency 27.746 ms 2024-05-27T07:44:21.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:21 smithi163 ceph-mon[36451]: pgmap v145: 129 pgs: 129 active+clean; 49 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 461 op/s 2024-05-27T07:44:21.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:21 smithi186 ceph-mon[36647]: pgmap v145: 129 pgs: 129 active+clean; 49 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 461 op/s 2024-05-27T07:44:22.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:21 smithi114 ceph-mon[32038]: pgmap v145: 129 pgs: 129 active+clean; 49 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 461 op/s 2024-05-27T07:44:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.199+0000 7f957a14d640 1 -- 172.21.15.114:0/586460883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f957410e1f0 msgr2=0x7f95741106e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.199+0000 7f957a14d640 1 --2- 172.21.15.114:0/586460883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f957410e1f0 0x7f95741106e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f956c009ca0 tx=0x7f956c02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.199+0000 7f957a14d640 1 -- 172.21.15.114:0/586460883 shutdown_connections 2024-05-27T07:44:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.199+0000 7f957a14d640 1 --2- 172.21.15.114:0/586460883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f957410e1f0 0x7f95741106e0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.199+0000 7f957a14d640 1 --2- 172.21.15.114:0/586460883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f957410b800 0x7f957410dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.199+0000 7f957a14d640 1 --2- 172.21.15.114:0/586460883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9574104da0 0x7f95741051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.199+0000 7f957a14d640 1 -- 172.21.15.114:0/586460883 >> 172.21.15.114:0/586460883 conn(0x7f9574078270 msgr2=0x7f95740786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.199+0000 7f957a14d640 1 -- 172.21.15.114:0/586460883 shutdown_connections 2024-05-27T07:44:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.199+0000 7f957a14d640 1 -- 172.21.15.114:0/586460883 wait complete. 2024-05-27T07:44:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.200+0000 7f957a14d640 1 Processor -- start 2024-05-27T07:44:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.200+0000 7f957a14d640 1 -- start start 2024-05-27T07:44:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.200+0000 7f957a14d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9574104da0 0x7f957419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.200+0000 7f957a14d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f957410b800 0x7f957419ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.200+0000 7f957a14d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f957410e1f0 0x7f957419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.200+0000 7f957a14d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9574113e50 con 0x7f957410b800 2024-05-27T07:44:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.200+0000 7f957a14d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9574113cd0 con 0x7f9574104da0 2024-05-27T07:44:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.200+0000 7f957a14d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9574113fd0 con 0x7f957410e1f0 2024-05-27T07:44:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f9573fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f957410e1f0 0x7f957419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f956bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f957410b800 0x7f957419ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f9573fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f957410e1f0 0x7f957419f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58912/0 (socket says 172.21.15.114:58912) 2024-05-27T07:44:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f9573fff640 1 -- 172.21.15.114:0/305673127 learned_addr learned my addr 172.21.15.114:0/305673127 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f956bfff640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f957410b800 0x7f957419ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49194/0 (socket says 172.21.15.114:49194) 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f95737fe640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9574104da0 0x7f957419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f956bfff640 1 -- 172.21.15.114:0/305673127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f957410e1f0 msgr2=0x7f957419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f956bfff640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f957410e1f0 0x7f957419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f956bfff640 1 -- 172.21.15.114:0/305673127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9574104da0 msgr2=0x7f957419a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f956bfff640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9574104da0 0x7f957419a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f956bfff640 1 -- 172.21.15.114:0/305673127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957419f650 con 0x7f957410b800 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f9573fff640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f957410e1f0 0x7f957419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f956bfff640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f957410b800 0x7f957419ad50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f955c00e9f0 tx=0x7f955c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f95717fa640 1 -- 172.21.15.114:0/305673127 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c00ccc0 con 0x7f957410b800 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f957a14d640 1 -- 172.21.15.114:0/305673127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f957419f940 con 0x7f957410b800 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f95717fa640 1 -- 172.21.15.114:0/305673127 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f955c005020 con 0x7f957410b800 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f95717fa640 1 -- 172.21.15.114:0/305673127 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c004a00 con 0x7f957410b800 2024-05-27T07:44:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.201+0000 7f957a14d640 1 -- 172.21.15.114:0/305673127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95741a72c0 con 0x7f957410b800 2024-05-27T07:44:22.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.204+0000 7f95717fa640 1 -- 172.21.15.114:0/305673127 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f955c004ba0 con 0x7f957410b800 2024-05-27T07:44:22.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.204+0000 7f9581001740 1 Processor -- start 2024-05-27T07:44:22.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.204+0000 7f9581001740 1 -- start start 2024-05-27T07:44:22.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9581001740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 0x5589c0b103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9581001740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 0x5589c0b10910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9581001740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c074e4d0 0x5589c0b10e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9581001740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5589c0a82c30 con 0x5589c074e4d0 2024-05-27T07:44:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9581001740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5589c0a727d0 con 0x5589c07462f0 2024-05-27T07:44:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9581001740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5589c0a76810 con 0x5589c076f1b0 2024-05-27T07:44:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9573fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 0x5589c0b103d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9573fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 0x5589c0b103d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58916/0 (socket says 172.21.15.114:58916) 2024-05-27T07:44:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9573fff640 1 -- 172.21.15.114:0/2208938752 learned_addr learned my addr 172.21.15.114:0/2208938752 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f956bfff640 1 --2- 172.21.15.114:0/2208938752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c074e4d0 0x5589c0b10e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f95737fe640 1 --2- 172.21.15.114:0/2208938752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 0x5589c0b10910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9573fff640 1 -- 172.21.15.114:0/2208938752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 msgr2=0x5589c0b10910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9573fff640 1 --2- 172.21.15.114:0/2208938752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 0x5589c0b10910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9573fff640 1 -- 172.21.15.114:0/2208938752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c074e4d0 msgr2=0x5589c0b10e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9573fff640 1 --2- 172.21.15.114:0/2208938752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c074e4d0 0x5589c0b10e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9573fff640 1 -- 172.21.15.114:0/2208938752 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589c0787210 con 0x5589c076f1b0 2024-05-27T07:44:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f95737fe640 1 --2- 172.21.15.114:0/2208938752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 0x5589c0b10910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:44:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f95717fa640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95600b2950 0x7f95600b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9573fff640 1 --2- 172.21.15.114:0/2208938752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 0x5589c0b103d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f956c0094e0 tx=0x7f956c02f390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9224f193a64eb096 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f95737fe640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95600b2950 0x7f95600b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f95717fa640 1 -- 172.21.15.114:0/305673127 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f955c0e28c0 con 0x7f957410b800 2024-05-27T07:44:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.205+0000 7f9569ffb640 1 -- 172.21.15.114:0/2208938752 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c031650 con 0x5589c076f1b0 2024-05-27T07:44:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.206+0000 7f9569ffb640 1 -- 172.21.15.114:0/2208938752 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f956c036040 con 0x5589c076f1b0 2024-05-27T07:44:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.206+0000 7f9569ffb640 1 -- 172.21.15.114:0/2208938752 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c004500 con 0x5589c076f1b0 2024-05-27T07:44:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.206+0000 7f9581001740 1 -- 172.21.15.114:0/2208938752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 msgr2=0x5589c0b103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.206+0000 7f95737fe640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95600b2950 0x7f95600b4e60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x5589c0b11780 tx=0x7f956400b390 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.206+0000 7f9581001740 1 --2- 172.21.15.114:0/2208938752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 0x5589c0b103d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f956c0094e0 tx=0x7f956c02f390 comp rx=0 tx=0).stop 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 -- 172.21.15.114:0/2208938752 shutdown_connections 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 --2- 172.21.15.114:0/2208938752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c074e4d0 0x5589c0b10e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 --2- 172.21.15.114:0/2208938752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 0x5589c0b10910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 --2- 172.21.15.114:0/2208938752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 0x5589c0b103d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 -- 172.21.15.114:0/2208938752 >> 172.21.15.114:0/2208938752 conn(0x5589c0763300 msgr2=0x5589c0b0cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 -- 172.21.15.114:0/2208938752 shutdown_connections 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 -- 172.21.15.114:0/2208938752 wait complete. 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 -- start start 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 Processor -- start 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 0x5589c0b1aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c074e4d0 0x5589c0b1aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 0x5589c0b20690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5589c0758790 con 0x5589c076f1b0 2024-05-27T07:44:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5589c0a2e890 con 0x5589c07462f0 2024-05-27T07:44:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.207+0000 7f9581001740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5589c0b1d320 con 0x5589c074e4d0 2024-05-27T07:44:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f9581001740 10 client.0 _ll_register_callbacks cb 0x5589c0786a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f9573fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 0x5589c0b20690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f9573fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 0x5589c0b20690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49214/0 (socket says 172.21.15.114:49214) 2024-05-27T07:44:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f956bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c074e4d0 0x5589c0b1aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f9573fff640 1 -- 172.21.15.114:0/2415526851 learned_addr learned my addr 172.21.15.114:0/2415526851 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f95737fe640 1 --2- 172.21.15.114:0/2415526851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 0x5589c0b1aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f956bfff640 1 -- 172.21.15.114:0/2415526851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 msgr2=0x5589c0b1aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f956bfff640 1 --2- 172.21.15.114:0/2415526851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 0x5589c0b1aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f956bfff640 1 -- 172.21.15.114:0/2415526851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 msgr2=0x5589c0b20690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f956bfff640 1 --2- 172.21.15.114:0/2415526851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 0x5589c0b20690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f956bfff640 1 -- 172.21.15.114:0/2415526851 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589c0853620 con 0x5589c074e4d0 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f9573fff640 1 --2- 172.21.15.114:0/2415526851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 0x5589c0b20690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f956bfff640 1 --2- 172.21.15.114:0/2415526851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c074e4d0 0x5589c0b1aff0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f955c0dfc60 tx=0x7f955c005460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ef2faacc8846500 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f9552ffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f9552ffd640 1 -- 172.21.15.114:0/2415526851 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c01c070 con 0x5589c074e4d0 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f9581001740 1 -- 172.21.15.114:0/2415526851 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f956c036040 con 0x5589c074e4d0 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f9552ffd640 1 -- 172.21.15.114:0/2415526851 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f955c0a5a80 con 0x5589c074e4d0 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f9552ffd640 1 -- 172.21.15.114:0/2415526851 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f955c0a95c0 con 0x5589c074e4d0 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f9581001740 10 client.25990 fetch_fsmap learned FSMap version 30 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.208+0000 7f9581001740 1 -- 172.21.15.114:0/2415526851 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5589c0b296f0 con 0x5589c074e4d0 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.209+0000 7f9552ffd640 1 -- 172.21.15.114:0/2415526851 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f956c036040 con 0x5589c074e4d0 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.209+0000 7f9552ffd640 1 -- 172.21.15.114:0/2415526851 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 30) ==== 7725+0+0 (secure 0 0 0) 0x7f955c0b21c0 con 0x5589c074e4d0 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.209+0000 7f9581001740 10 client.25990 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.209+0000 7f9581001740 10 client.25990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.209+0000 7f9581001740 20 client.25990 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.209+0000 7f9581001740 1 --2- 172.21.15.114:0/2415526851 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5589c072dd80 0x5589c0b2cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.209+0000 7f9581001740 4 client.25990 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.209+0000 7f9581001740 1 -- 172.21.15.114:0/2415526851 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589c0a84b80 con 0x5589c072dd80 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.209+0000 7f95737fe640 1 --2- 172.21.15.114:0/2415526851 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5589c072dd80 0x5589c0b2cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.209+0000 7f9552ffd640 10 client.25990 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.209+0000 7f95737fe640 1 --2- 172.21.15.114:0/2415526851 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5589c072dd80 0x5589c0b2cc10 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3b882d3de4c196a server_cookie=264e0921813b72c6 in_seq=0 out_seq=0 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.210+0000 7f9552ffd640 1 -- 172.21.15.114:0/2415526851 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5589c0a84b80 con 0x5589c072dd80 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.210+0000 7f9552ffd640 10 client.25990 handle_command_reply: tid=0 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.210+0000 7f9581001740 1 client.25990 shutdown 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.210+0000 7f9581001740 1 -- 172.21.15.114:0/2415526851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c074e4d0 msgr2=0x5589c0b1aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.210+0000 7f9581001740 1 --2- 172.21.15.114:0/2415526851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c074e4d0 0x5589c0b1aff0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f955c0dfc60 tx=0x7f955c005460 comp rx=0 tx=0).stop 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.210+0000 7f9581001740 1 -- 172.21.15.114:0/2415526851 shutdown_connections 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.210+0000 7f9581001740 1 --2- 172.21.15.114:0/2415526851 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5589c072dd80 0x5589c0b2cc10 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.210+0000 7f9581001740 1 --2- 172.21.15.114:0/2415526851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 0x5589c0b20690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.210+0000 7f9552ffd640 0 client.25990 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.210+0000 7f9581001740 1 --2- 172.21.15.114:0/2415526851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c074e4d0 0x5589c0b1aff0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.211+0000 7f9581001740 1 --2- 172.21.15.114:0/2415526851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 0x5589c0b1aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.211+0000 7f9581001740 1 -- 172.21.15.114:0/2415526851 >> 172.21.15.114:0/2415526851 conn(0x5589c0763300 msgr2=0x5589c0b10200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.211+0000 7f9581001740 1 -- 172.21.15.114:0/2415526851 shutdown_connections 2024-05-27T07:44:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.211+0000 7f9581001740 1 -- 172.21.15.114:0/2415526851 wait complete. 2024-05-27T07:44:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.211+0000 7f9581001740 20 client.25990 trim_cache size 0 max 16384 2024-05-27T07:44:22.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 393555 13.30 MB/sec execute 34 sec latency 29.145 ms 2024-05-27T07:44:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.252+0000 7f9581001740 1 Processor -- start 2024-05-27T07:44:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.253+0000 7f9581001740 1 -- start start 2024-05-27T07:44:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.253+0000 7f9581001740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c07462f0 0x5589c0b10870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.253+0000 7f9581001740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c074e4d0 0x5589c0b20590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.253+0000 7f9581001740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 0x5589c0b20ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.253+0000 7f9581001740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95741b2130 con 0x5589c076f1b0 2024-05-27T07:44:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.253+0000 7f9581001740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f95741af1e0 con 0x5589c074e4d0 2024-05-27T07:44:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.253+0000 7f9581001740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95741b1390 con 0x5589c07462f0 2024-05-27T07:44:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.254+0000 7f9573fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 0x5589c0b20ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.254+0000 7f9573fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 0x5589c0b20ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49230/0 (socket says 172.21.15.114:49230) 2024-05-27T07:44:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.254+0000 7f9573fff640 1 -- 172.21.15.114:0/545016445 learned_addr learned my addr 172.21.15.114:0/545016445 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.255+0000 7f95737fe640 1 --2- 172.21.15.114:0/545016445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c07462f0 0x5589c0b10870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.255+0000 7f9573fff640 1 -- 172.21.15.114:0/545016445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c07462f0 msgr2=0x5589c0b10870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.255+0000 7f9573fff640 1 --2- 172.21.15.114:0/545016445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c07462f0 0x5589c0b10870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.255+0000 7f9573fff640 1 -- 172.21.15.114:0/545016445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c074e4d0 msgr2=0x5589c0b20590 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:44:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.255+0000 7f9573fff640 1 --2- 172.21.15.114:0/545016445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c074e4d0 0x5589c0b20590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.255+0000 7f9573fff640 1 -- 172.21.15.114:0/545016445 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589c0b0d750 con 0x5589c076f1b0 2024-05-27T07:44:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.255+0000 7f9573fff640 1 --2- 172.21.15.114:0/545016445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 0x5589c0b20ad0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f956c0097b0 tx=0x7f956c02b060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a16dee2feb8ece server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:22.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.256+0000 7f95697fa640 1 -- 172.21.15.114:0/545016445 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c003970 con 0x5589c076f1b0 2024-05-27T07:44:22.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.256+0000 7f95697fa640 1 -- 172.21.15.114:0/545016445 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f956c036040 con 0x5589c076f1b0 2024-05-27T07:44:22.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.256+0000 7f95697fa640 1 -- 172.21.15.114:0/545016445 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c02f660 con 0x5589c076f1b0 2024-05-27T07:44:22.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.257+0000 7f9581001740 1 -- 172.21.15.114:0/545016445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 msgr2=0x5589c0b20ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:22.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.257+0000 7f9581001740 1 --2- 172.21.15.114:0/545016445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 0x5589c0b20ad0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f956c0097b0 tx=0x7f956c02b060 comp rx=0 tx=0).stop 2024-05-27T07:44:22.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.258+0000 7f9581001740 1 -- 172.21.15.114:0/545016445 shutdown_connections 2024-05-27T07:44:22.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.258+0000 7f9581001740 1 --2- 172.21.15.114:0/545016445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c076f1b0 0x5589c0b20ad0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.258+0000 7f9581001740 1 --2- 172.21.15.114:0/545016445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c074e4d0 0x5589c0b20590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.258+0000 7f9581001740 1 --2- 172.21.15.114:0/545016445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c07462f0 0x5589c0b10870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.258+0000 7f9581001740 1 -- 172.21.15.114:0/545016445 >> 172.21.15.114:0/545016445 conn(0x5589c0763300 msgr2=0x5589c0b10160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.260+0000 7f9581001740 1 -- 172.21.15.114:0/545016445 shutdown_connections 2024-05-27T07:44:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.261+0000 7f9581001740 1 -- 172.21.15.114:0/545016445 wait complete. 2024-05-27T07:44:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.261+0000 7f9581001740 1 -- start start 2024-05-27T07:44:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.262+0000 7f9581001740 1 Processor -- start 2024-05-27T07:44:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.262+0000 7f9581001740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 0x5589c0b21c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.262+0000 7f9581001740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c074e4d0 0x5589c0b221c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.262+0000 7f9581001740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 0x5589c0ba4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.262+0000 7f9581001740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95741b0cc0 con 0x5589c074e4d0 2024-05-27T07:44:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.262+0000 7f9581001740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f95741af850 con 0x5589c07462f0 2024-05-27T07:44:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.262+0000 7f9581001740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95741aff20 con 0x5589c076f1b0 2024-05-27T07:44:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.262+0000 7f9581001740 10 client.0 _ll_register_callbacks cb 0x7f955c0a5050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.262+0000 7f9573fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 0x5589c0ba4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.262+0000 7f9573fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 0x5589c0ba4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58950/0 (socket says 172.21.15.114:58950) 2024-05-27T07:44:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.262+0000 7f9573fff640 1 -- 172.21.15.114:0/1622086551 learned_addr learned my addr 172.21.15.114:0/1622086551 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.263+0000 7f95737fe640 1 --2- 172.21.15.114:0/1622086551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 0x5589c0b21c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.263+0000 7f9573fff640 1 -- 172.21.15.114:0/1622086551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 msgr2=0x5589c0b21c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.263+0000 7f9573fff640 1 --2- 172.21.15.114:0/1622086551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 0x5589c0b21c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.263+0000 7f9573fff640 1 -- 172.21.15.114:0/1622086551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c074e4d0 msgr2=0x5589c0b221c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:44:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.263+0000 7f9573fff640 1 --2- 172.21.15.114:0/1622086551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c074e4d0 0x5589c0b221c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.263+0000 7f9573fff640 1 -- 172.21.15.114:0/1622086551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589c0b22a90 con 0x5589c076f1b0 2024-05-27T07:44:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.264+0000 7f9573fff640 1 --2- 172.21.15.114:0/1622086551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 0x5589c0ba4290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f956c005de0 tx=0x7f956c0099f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff8720c782d9aa0f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:22.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.265+0000 7f9552ffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:44:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.266+0000 7f9552ffd640 1 -- 172.21.15.114:0/1622086551 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c004860 con 0x5589c076f1b0 2024-05-27T07:44:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.266+0000 7f9581001740 1 -- 172.21.15.114:0/1622086551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f956c036040 con 0x5589c076f1b0 2024-05-27T07:44:22.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.266+0000 7f9552ffd640 1 -- 172.21.15.114:0/1622086551 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f956c003b70 con 0x5589c076f1b0 2024-05-27T07:44:22.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.266+0000 7f9552ffd640 1 -- 172.21.15.114:0/1622086551 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f956c08c050 con 0x5589c076f1b0 2024-05-27T07:44:22.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.267+0000 7f9581001740 10 client.25993 fetch_fsmap learned FSMap version 30 2024-05-27T07:44:22.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.267+0000 7f9581001740 1 -- 172.21.15.114:0/1622086551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5589c0b8d4b0 con 0x5589c076f1b0 2024-05-27T07:44:22.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.267+0000 7f9552ffd640 1 -- 172.21.15.114:0/1622086551 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f956c036040 con 0x5589c076f1b0 2024-05-27T07:44:22.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.267+0000 7f9552ffd640 1 -- 172.21.15.114:0/1622086551 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 30) ==== 7725+0+0 (secure 0 0 0) 0x7f956c044070 con 0x5589c076f1b0 2024-05-27T07:44:22.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.267+0000 7f9581001740 10 client.25993 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:44:22.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.267+0000 7f9581001740 10 client.25993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:22.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.267+0000 7f9581001740 20 client.25993 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:22.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.267+0000 7f9581001740 1 --2- 172.21.15.114:0/1622086551 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5589c07031c0 0x5589c0b8fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:22.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.267+0000 7f9581001740 4 client.25993 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:44:22.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.267+0000 7f9581001740 1 -- 172.21.15.114:0/1622086551 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5589c0a7e360 con 0x5589c07031c0 2024-05-27T07:44:22.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.268+0000 7f95737fe640 1 --2- 172.21.15.114:0/1622086551 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5589c07031c0 0x5589c0b8fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.272+0000 7f95737fe640 1 --2- 172.21.15.114:0/1622086551 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5589c07031c0 0x5589c0b8fb80 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60f3a96736205877 server_cookie=7ae3ea5fa6d2a60d in_seq=0 out_seq=0 2024-05-27T07:44:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.272+0000 7f9552ffd640 10 client.25993 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.666+0000 7f9552ffd640 1 -- 172.21.15.114:0/1622086551 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5589c0a7e360 con 0x5589c07031c0 2024-05-27T07:44:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.666+0000 7f9552ffd640 10 client.25993 handle_command_reply: tid=0 2024-05-27T07:44:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.666+0000 7f9581001740 1 client.25993 shutdown 2024-05-27T07:44:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.666+0000 7f9581001740 1 -- 172.21.15.114:0/1622086551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 msgr2=0x5589c0ba4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.666+0000 7f9581001740 1 --2- 172.21.15.114:0/1622086551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 0x5589c0ba4290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f956c005de0 tx=0x7f956c0099f0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.666+0000 7f9581001740 1 -- 172.21.15.114:0/1622086551 shutdown_connections 2024-05-27T07:44:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.666+0000 7f9581001740 1 --2- 172.21.15.114:0/1622086551 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5589c07031c0 0x5589c0b8fb80 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.666+0000 7f9581001740 1 --2- 172.21.15.114:0/1622086551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589c076f1b0 0x5589c0ba4290 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.666+0000 7f9581001740 1 --2- 172.21.15.114:0/1622086551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5589c074e4d0 0x5589c0b221c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.666+0000 7f9581001740 1 --2- 172.21.15.114:0/1622086551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5589c07462f0 0x5589c0b21c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.666+0000 7f9581001740 1 -- 172.21.15.114:0/1622086551 >> 172.21.15.114:0/1622086551 conn(0x5589c0763300 msgr2=0x5589c0b1ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.666+0000 7f9552ffd640 0 client.25993 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.666+0000 7f9581001740 1 -- 172.21.15.114:0/1622086551 shutdown_connections 2024-05-27T07:44:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.667+0000 7f9581001740 1 -- 172.21.15.114:0/1622086551 wait complete. 2024-05-27T07:44:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.667+0000 7f9581001740 20 client.25993 trim_cache size 0 max 16384 2024-05-27T07:44:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:44:22.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:44:22.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:44:22.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.667+0000 7f957a14d640 1 -- 172.21.15.114:0/305673127 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95600b2950 msgr2=0x7f95600b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:22.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.667+0000 7f957a14d640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95600b2950 0x7f95600b4e60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x5589c0b11780 tx=0x7f956400b390 comp rx=0 tx=0).stop 2024-05-27T07:44:22.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.667+0000 7f957a14d640 1 -- 172.21.15.114:0/305673127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f957410b800 msgr2=0x7f957419ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:22.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.667+0000 7f957a14d640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f957410b800 0x7f957419ad50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f955c00e9f0 tx=0x7f955c00eeb0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.667+0000 7f957a14d640 1 -- 172.21.15.114:0/305673127 shutdown_connections 2024-05-27T07:44:22.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.667+0000 7f957a14d640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95600b2950 0x7f95600b4e60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.667+0000 7f957a14d640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f957410e1f0 0x7f957419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.667+0000 7f957a14d640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f957410b800 0x7f957419ad50 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.667+0000 7f957a14d640 1 --2- 172.21.15.114:0/305673127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9574104da0 0x7f957419a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:22.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.668+0000 7f957a14d640 1 -- 172.21.15.114:0/305673127 >> 172.21.15.114:0/305673127 conn(0x7f9574078270 msgr2=0x7f9574101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:22.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.668+0000 7f957a14d640 1 -- 172.21.15.114:0/305673127 shutdown_connections 2024-05-27T07:44:22.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:22.668+0000 7f957a14d640 1 -- 172.21.15.114:0/305673127 wait complete. 2024-05-27T07:44:23.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 394433 13.05 MB/sec execute 35 sec latency 143.831 ms 2024-05-27T07:44:23.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:23 smithi163 ceph-mon[36451]: pgmap v146: 129 pgs: 129 active+clean; 53 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.3 MiB/s wr, 378 op/s 2024-05-27T07:44:23.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:23 smithi186 ceph-mon[36647]: pgmap v146: 129 pgs: 129 active+clean; 53 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.3 MiB/s wr, 378 op/s 2024-05-27T07:44:24.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:23 smithi114 ceph-mon[32038]: pgmap v146: 129 pgs: 129 active+clean; 53 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.3 MiB/s wr, 378 op/s 2024-05-27T07:44:24.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 397550 13.21 MB/sec execute 36 sec latency 15.770 ms 2024-05-27T07:44:25.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 399905 13.08 MB/sec execute 37 sec latency 15.158 ms 2024-05-27T07:44:25.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:25 smithi163 ceph-mon[36451]: pgmap v147: 129 pgs: 129 active+clean; 53 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 245 op/s 2024-05-27T07:44:25.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:25 smithi186 ceph-mon[36647]: pgmap v147: 129 pgs: 129 active+clean; 53 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 245 op/s 2024-05-27T07:44:26.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:25 smithi114 ceph-mon[32038]: pgmap v147: 129 pgs: 129 active+clean; 53 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 245 op/s 2024-05-27T07:44:26.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 402311 13.16 MB/sec execute 38 sec latency 22.769 ms 2024-05-27T07:44:27.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 405160 13.19 MB/sec execute 39 sec latency 15.687 ms 2024-05-27T07:44:27.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:27 smithi163 ceph-mon[36451]: pgmap v148: 129 pgs: 129 active+clean; 44 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 525 op/s 2024-05-27T07:44:27.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:27 smithi186 ceph-mon[36647]: pgmap v148: 129 pgs: 129 active+clean; 44 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 525 op/s 2024-05-27T07:44:28.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:27 smithi114 ceph-mon[32038]: pgmap v148: 129 pgs: 129 active+clean; 44 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 525 op/s 2024-05-27T07:44:28.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 407595 13.19 MB/sec execute 40 sec latency 93.267 ms 2024-05-27T07:44:29.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 410332 13.18 MB/sec execute 41 sec latency 13.983 ms 2024-05-27T07:44:29.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:29 smithi163 ceph-mon[36451]: pgmap v149: 129 pgs: 129 active+clean; 44 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 400 op/s 2024-05-27T07:44:29.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:29 smithi186 ceph-mon[36647]: pgmap v149: 129 pgs: 129 active+clean; 44 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 400 op/s 2024-05-27T07:44:30.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:29 smithi114 ceph-mon[32038]: pgmap v149: 129 pgs: 129 active+clean; 44 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 400 op/s 2024-05-27T07:44:30.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 412484 13.18 MB/sec execute 42 sec latency 23.946 ms 2024-05-27T07:44:31.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 415499 13.29 MB/sec execute 43 sec latency 18.073 ms 2024-05-27T07:44:31.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:31 smithi163 ceph-mon[36451]: pgmap v150: 129 pgs: 129 active+clean; 46 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 464 op/s 2024-05-27T07:44:31.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:31 smithi186 ceph-mon[36647]: pgmap v150: 129 pgs: 129 active+clean; 46 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 464 op/s 2024-05-27T07:44:32.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:31 smithi114 ceph-mon[32038]: pgmap v150: 129 pgs: 129 active+clean; 46 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 464 op/s 2024-05-27T07:44:32.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 418672 13.39 MB/sec execute 44 sec latency 19.523 ms 2024-05-27T07:44:33.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 419816 13.17 MB/sec execute 45 sec latency 62.692 ms 2024-05-27T07:44:33.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-mon[36647]: pgmap v151: 129 pgs: 129 active+clean; 50 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 469 op/s 2024-05-27T07:44:33.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f89a794b700 time 2024-05-27T07:44:33.460366+0000 2024-05-27T07:44:33.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:44:33.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:33.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f89ba190ba6] 2024-05-27T07:44:33.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f89ba190dd1] 2024-05-27T07:44:33.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56505872923d] 2024-05-27T07:44:33.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 4: (SnapRealm::build_snap_set() const+0x323) [0x565058714971] 2024-05-27T07:44:33.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 5: (SnapRealm::check_cache() const+0x13e) [0x565058712fa0] 2024-05-27T07:44:33.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x565058713651] 2024-05-27T07:44:33.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650586a8347] 2024-05-27T07:44:33.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650583a487e] 2024-05-27T07:44:33.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5650583a9f2f] 2024-05-27T07:44:33.729 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5650583b4969] 2024-05-27T07:44:33.729 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 11: (MDLog::_replay_thread()+0x1f38) [0x56505874d48c] 2024-05-27T07:44:33.729 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5650587569cb] 2024-05-27T07:44:33.729 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 13: (Thread::entry_wrapper()+0x43) [0x7f89ba15f72d] 2024-05-27T07:44:33.729 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 14: (Thread::_entry_func(void*)+0xd) [0x7f89ba15f749] 2024-05-27T07:44:33.729 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f89b8d031ca] 2024-05-27T07:44:33.729 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 16: clone() 2024-05-27T07:44:33.729 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.461+0000 7f89a794b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f89a794b700 time 2024-05-27T07:44:33.460366+0000 2024-05-27T07:44:33.729 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:44:33.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:33.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f89ba190ba6] 2024-05-27T07:44:33.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f89ba190dd1] 2024-05-27T07:44:33.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56505872923d] 2024-05-27T07:44:33.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 4: (SnapRealm::build_snap_set() const+0x323) [0x565058714971] 2024-05-27T07:44:33.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 5: (SnapRealm::check_cache() const+0x13e) [0x565058712fa0] 2024-05-27T07:44:33.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x565058713651] 2024-05-27T07:44:33.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650586a8347] 2024-05-27T07:44:33.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650583a487e] 2024-05-27T07:44:33.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5650583a9f2f] 2024-05-27T07:44:33.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5650583b4969] 2024-05-27T07:44:33.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 11: (MDLog::_replay_thread()+0x1f38) [0x56505874d48c] 2024-05-27T07:44:33.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5650587569cb] 2024-05-27T07:44:33.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 13: (Thread::entry_wrapper()+0x43) [0x7f89ba15f72d] 2024-05-27T07:44:33.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 14: (Thread::_entry_func(void*)+0xd) [0x7f89ba15f749] 2024-05-27T07:44:33.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f89b8d031ca] 2024-05-27T07:44:33.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 16: clone() 2024-05-27T07:44:33.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: *** Caught signal (Aborted) ** 2024-05-27T07:44:33.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: in thread 7f89a794b700 thread_name:md_log_replay 2024-05-27T07:44:33.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:33.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f89b8d0dd20] 2024-05-27T07:44:33.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2: gsignal() 2024-05-27T07:44:33.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 3: abort() 2024-05-27T07:44:33.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f89ba190ced] 2024-05-27T07:44:33.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f89ba190dd1] 2024-05-27T07:44:33.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56505872923d] 2024-05-27T07:44:33.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 7: (SnapRealm::build_snap_set() const+0x323) [0x565058714971] 2024-05-27T07:44:33.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 8: (SnapRealm::check_cache() const+0x13e) [0x565058712fa0] 2024-05-27T07:44:33.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x565058713651] 2024-05-27T07:44:33.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650586a8347] 2024-05-27T07:44:33.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650583a487e] 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5650583a9f2f] 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5650583b4969] 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 14: (MDLog::_replay_thread()+0x1f38) [0x56505874d48c] 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5650587569cb] 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 16: (Thread::entry_wrapper()+0x43) [0x7f89ba15f72d] 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 17: (Thread::_entry_func(void*)+0xd) [0x7f89ba15f749] 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f89b8d031ca] 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 19: clone() 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.464+0000 7f89a794b700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: in thread 7f89a794b700 thread_name:md_log_replay 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f89b8d0dd20] 2024-05-27T07:44:33.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2: gsignal() 2024-05-27T07:44:33.734 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 3: abort() 2024-05-27T07:44:33.734 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f89ba190ced] 2024-05-27T07:44:33.734 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f89ba190dd1] 2024-05-27T07:44:33.734 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56505872923d] 2024-05-27T07:44:33.734 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 7: (SnapRealm::build_snap_set() const+0x323) [0x565058714971] 2024-05-27T07:44:33.734 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 8: (SnapRealm::check_cache() const+0x13e) [0x565058712fa0] 2024-05-27T07:44:33.734 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x565058713651] 2024-05-27T07:44:33.734 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650586a8347] 2024-05-27T07:44:33.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650583a487e] 2024-05-27T07:44:33.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5650583a9f2f] 2024-05-27T07:44:33.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5650583b4969] 2024-05-27T07:44:33.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 14: (MDLog::_replay_thread()+0x1f38) [0x56505874d48c] 2024-05-27T07:44:33.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5650587569cb] 2024-05-27T07:44:33.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 16: (Thread::entry_wrapper()+0x43) [0x7f89ba15f72d] 2024-05-27T07:44:33.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 17: (Thread::_entry_func(void*)+0xd) [0x7f89ba15f749] 2024-05-27T07:44:33.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f89b8d031ca] 2024-05-27T07:44:33.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 19: clone() 2024-05-27T07:44:33.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:44:33.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: -1> 2024-05-27T07:44:33.461+0000 7f89a794b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f89a794b700 time 2024-05-27T07:44:33.460366+0000 2024-05-27T07:44:33.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:44:33.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:33.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f89ba190ba6] 2024-05-27T07:44:33.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f89ba190dd1] 2024-05-27T07:44:33.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56505872923d] 2024-05-27T07:44:33.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 4: (SnapRealm::build_snap_set() const+0x323) [0x565058714971] 2024-05-27T07:44:33.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 5: (SnapRealm::check_cache() const+0x13e) [0x565058712fa0] 2024-05-27T07:44:33.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x565058713651] 2024-05-27T07:44:33.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650586a8347] 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650583a487e] 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5650583a9f2f] 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5650583b4969] 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 11: (MDLog::_replay_thread()+0x1f38) [0x56505874d48c] 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5650587569cb] 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 13: (Thread::entry_wrapper()+0x43) [0x7f89ba15f72d] 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 14: (Thread::_entry_func(void*)+0xd) [0x7f89ba15f749] 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f89b8d031ca] 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 16: clone() 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 0> 2024-05-27T07:44:33.464+0000 7f89a794b700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: in thread 7f89a794b700 thread_name:md_log_replay 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f89b8d0dd20] 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2: gsignal() 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 3: abort() 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f89ba190ced] 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f89ba190dd1] 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56505872923d] 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 7: (SnapRealm::build_snap_set() const+0x323) [0x565058714971] 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 8: (SnapRealm::check_cache() const+0x13e) [0x565058712fa0] 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x565058713651] 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650586a8347] 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650583a487e] 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5650583a9f2f] 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5650583b4969] 2024-05-27T07:44:33.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 14: (MDLog::_replay_thread()+0x1f38) [0x56505874d48c] 2024-05-27T07:44:33.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5650587569cb] 2024-05-27T07:44:33.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 16: (Thread::entry_wrapper()+0x43) [0x7f89ba15f72d] 2024-05-27T07:44:33.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 17: (Thread::_entry_func(void*)+0xd) [0x7f89ba15f749] 2024-05-27T07:44:33.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f89b8d031ca] 2024-05-27T07:44:33.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 19: clone() 2024-05-27T07:44:33.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:44:33.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: -1> 2024-05-27T07:44:33.461+0000 7f89a794b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f89a794b700 time 2024-05-27T07:44:33.460366+0000 2024-05-27T07:44:33.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:44:33.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:33.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f89ba190ba6] 2024-05-27T07:44:33.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f89ba190dd1] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56505872923d] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 4: (SnapRealm::build_snap_set() const+0x323) [0x565058714971] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 5: (SnapRealm::check_cache() const+0x13e) [0x565058712fa0] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x565058713651] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650586a8347] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650583a487e] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5650583a9f2f] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5650583b4969] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 11: (MDLog::_replay_thread()+0x1f38) [0x56505874d48c] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5650587569cb] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 13: (Thread::entry_wrapper()+0x43) [0x7f89ba15f72d] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 14: (Thread::_entry_func(void*)+0xd) [0x7f89ba15f749] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f89b8d031ca] 2024-05-27T07:44:33.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 16: clone() 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 0> 2024-05-27T07:44:33.464+0000 7f89a794b700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: in thread 7f89a794b700 thread_name:md_log_replay 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f89b8d0dd20] 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2: gsignal() 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 3: abort() 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f89ba190ced] 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f89ba190dd1] 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56505872923d] 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 7: (SnapRealm::build_snap_set() const+0x323) [0x565058714971] 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 8: (SnapRealm::check_cache() const+0x13e) [0x565058712fa0] 2024-05-27T07:44:33.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x565058713651] 2024-05-27T07:44:33.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650586a8347] 2024-05-27T07:44:33.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650583a487e] 2024-05-27T07:44:33.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5650583a9f2f] 2024-05-27T07:44:33.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5650583b4969] 2024-05-27T07:44:33.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 14: (MDLog::_replay_thread()+0x1f38) [0x56505874d48c] 2024-05-27T07:44:33.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5650587569cb] 2024-05-27T07:44:33.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 16: (Thread::entry_wrapper()+0x43) [0x7f89ba15f72d] 2024-05-27T07:44:33.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 17: (Thread::_entry_func(void*)+0xd) [0x7f89ba15f749] 2024-05-27T07:44:33.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f89b8d031ca] 2024-05-27T07:44:33.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 19: clone() 2024-05-27T07:44:33.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:44:33.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[67691]: 2024-05-27T07:44:33.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:33 smithi186 podman[72039]: 2024-05-27 07:44:33.533596061 +0000 UTC m=+0.021634771 container died a03d8aed320f71d84db154665a48737696953dec88b4e314ecf575f6616dc1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, ceph=True) 2024-05-27T07:44:33.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:33 smithi163 ceph-mon[36451]: pgmap v151: 129 pgs: 129 active+clean; 50 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 469 op/s 2024-05-27T07:44:34.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:33 smithi114 ceph-mon[32038]: pgmap v151: 129 pgs: 129 active+clean; 50 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 469 op/s 2024-05-27T07:44:34.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 422725 13.26 MB/sec execute 46 sec latency 7.721 ms 2024-05-27T07:44:34.789 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:34 smithi186 podman[72039]: 2024-05-27 07:44:34.487721902 +0000 UTC m=+0.975760611 container remove a03d8aed320f71d84db154665a48737696953dec88b4e314ecf575f6616dc1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:44:34.789 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:34 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:44:35.227 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:34 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:44:35.227 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:34 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Consumed 2.886s CPU time. 2024-05-27T07:44:35.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 425783 13.35 MB/sec execute 47 sec latency 19.155 ms 2024-05-27T07:44:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.774+0000 7fc4bab7d640 1 -- 172.21.15.114:0/3097187473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4b4103920 msgr2=0x7fc4b410f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.774+0000 7fc4bab7d640 1 --2- 172.21.15.114:0/3097187473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4b4103920 0x7fc4b410f690 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009ca0 tx=0x7fc4ac02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.774+0000 7fc4bab7d640 1 -- 172.21.15.114:0/3097187473 shutdown_connections 2024-05-27T07:44:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.774+0000 7fc4bab7d640 1 --2- 172.21.15.114:0/3097187473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4b4103920 0x7fc4b410f690 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.774+0000 7fc4bab7d640 1 --2- 172.21.15.114:0/3097187473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4b4109010 0x7fc4b41032b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.774+0000 7fc4bab7d640 1 --2- 172.21.15.114:0/3097187473 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4b4108b70 0x7fc4b4102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.774+0000 7fc4bab7d640 1 -- 172.21.15.114:0/3097187473 >> 172.21.15.114:0/3097187473 conn(0x7fc4b40fda10 msgr2=0x7fc4b40ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.774+0000 7fc4bab7d640 1 -- 172.21.15.114:0/3097187473 shutdown_connections 2024-05-27T07:44:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.774+0000 7fc4bab7d640 1 -- 172.21.15.114:0/3097187473 wait complete. 2024-05-27T07:44:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.775+0000 7fc4bab7d640 1 Processor -- start 2024-05-27T07:44:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.775+0000 7fc4bab7d640 1 -- start start 2024-05-27T07:44:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.775+0000 7fc4bab7d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4b4103920 0x7fc4b419b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.775+0000 7fc4bab7d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4b4108b70 0x7fc4b419bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.775+0000 7fc4bab7d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4b4109010 0x7fc4b419c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.775+0000 7fc4bab7d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b4112d80 con 0x7fc4b4109010 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.775+0000 7fc4bab7d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b4112c00 con 0x7fc4b4103920 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.775+0000 7fc4bab7d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b4112f00 con 0x7fc4b4108b70 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.775+0000 7fc4b911a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4b4109010 0x7fc4b419c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.775+0000 7fc4b8919640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4b4103920 0x7fc4b419b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.775+0000 7fc4b3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4b4108b70 0x7fc4b419bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.775+0000 7fc4b911a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4b4109010 0x7fc4b419c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54874/0 (socket says 172.21.15.114:54874) 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.775+0000 7fc4b911a640 1 -- 172.21.15.114:0/599246990 learned_addr learned my addr 172.21.15.114:0/599246990 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.776+0000 7fc4b911a640 1 -- 172.21.15.114:0/599246990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4b4108b70 msgr2=0x7fc4b419bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.776+0000 7fc4b911a640 1 --2- 172.21.15.114:0/599246990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4b4108b70 0x7fc4b419bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.776+0000 7fc4b911a640 1 -- 172.21.15.114:0/599246990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4b4103920 msgr2=0x7fc4b419b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.776+0000 7fc4b911a640 1 --2- 172.21.15.114:0/599246990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4b4103920 0x7fc4b419b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.776+0000 7fc4b911a640 1 -- 172.21.15.114:0/599246990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b41a6990 con 0x7fc4b4109010 2024-05-27T07:44:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.776+0000 7fc4b8919640 1 --2- 172.21.15.114:0/599246990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4b4103920 0x7fc4b419b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.776+0000 7fc4b911a640 1 --2- 172.21.15.114:0/599246990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4b4109010 0x7fc4b419c100 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac005cc0 tx=0x7fc4ac031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.776+0000 7fc4b1ffb640 1 -- 172.21.15.114:0/599246990 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac003b70 con 0x7fc4b4109010 2024-05-27T07:44:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.776+0000 7fc4b1ffb640 1 -- 172.21.15.114:0/599246990 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4ac0318f0 con 0x7fc4b4109010 2024-05-27T07:44:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.776+0000 7fc4bab7d640 1 -- 172.21.15.114:0/599246990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b41a6c20 con 0x7fc4b4109010 2024-05-27T07:44:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.776+0000 7fc4b1ffb640 1 -- 172.21.15.114:0/599246990 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac02f950 con 0x7fc4b4109010 2024-05-27T07:44:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.776+0000 7fc4bab7d640 1 -- 172.21.15.114:0/599246990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b41a71c0 con 0x7fc4b4109010 2024-05-27T07:44:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4c1a33740 1 Processor -- start 2024-05-27T07:44:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4b1ffb640 1 -- 172.21.15.114:0/599246990 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fc4ac02fb30 con 0x7fc4b4109010 2024-05-27T07:44:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4c1a33740 1 -- start start 2024-05-27T07:44:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4c1a33740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828311b0 0x563982bd23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4c1a33740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639828082f0 0x563982bd2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4c1a33740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982bd2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4c1a33740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563982b44c30 con 0x5639828104d0 2024-05-27T07:44:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4c1a33740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563982b347d0 con 0x5639828082f0 2024-05-27T07:44:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4c1a33740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563982b38810 con 0x5639828311b0 2024-05-27T07:44:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4b3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982bd2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4b8919640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639828082f0 0x563982bd2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4b3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982bd2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54888/0 (socket says 172.21.15.114:54888) 2024-05-27T07:44:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4b3fff640 1 -- 172.21.15.114:0/868011921 learned_addr learned my addr 172.21.15.114:0/868011921 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.779+0000 7fc4b3fff640 1 --2- 172.21.15.114:0/868011921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828311b0 0x563982bd23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.780+0000 7fc4b8919640 1 -- 172.21.15.114:0/868011921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828311b0 msgr2=0x563982bd23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.780+0000 7fc4b1ffb640 1 --2- 172.21.15.114:0/599246990 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc4a40b29d0 0x7fc4a40b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.780+0000 7fc4b8919640 1 --2- 172.21.15.114:0/868011921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828311b0 0x563982bd23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.780+0000 7fc4b8919640 1 -- 172.21.15.114:0/868011921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 msgr2=0x563982bd2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.780+0000 7fc4b8919640 1 --2- 172.21.15.114:0/868011921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982bd2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.780+0000 7fc4b8919640 1 -- 172.21.15.114:0/868011921 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563982849210 con 0x5639828082f0 2024-05-27T07:44:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.780+0000 7fc4b1ffb640 1 -- 172.21.15.114:0/599246990 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc4ac103b60 con 0x7fc4b4109010 2024-05-27T07:44:35.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.786+0000 7fc4b911a640 1 --2- 172.21.15.114:0/599246990 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc4a40b29d0 0x7fc4a40b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.786+0000 7fc4b8919640 1 --2- 172.21.15.114:0/868011921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639828082f0 0x563982bd2910 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009fa0 tx=0x7fc4a80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cf0a56c2238509 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.787+0000 7fc4b911a640 1 --2- 172.21.15.114:0/599246990 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc4a40b29d0 0x7fc4a40b4ee0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac0c6c20 tx=0x7fc4ac0095c0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.788+0000 7fc49effd640 1 -- 172.21.15.114:0/868011921 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8004030 con 0x5639828082f0 2024-05-27T07:44:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.788+0000 7fc49effd640 1 -- 172.21.15.114:0/868011921 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4a80054d0 con 0x5639828082f0 2024-05-27T07:44:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.788+0000 7fc49effd640 1 -- 172.21.15.114:0/868011921 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8004ce0 con 0x5639828082f0 2024-05-27T07:44:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.789+0000 7fc4c1a33740 1 -- 172.21.15.114:0/868011921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639828082f0 msgr2=0x563982bd2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.790+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/868011921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639828082f0 0x563982bd2910 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009fa0 tx=0x7fc4a80118d0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.790+0000 7fc4c1a33740 1 -- 172.21.15.114:0/868011921 shutdown_connections 2024-05-27T07:44:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.790+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/868011921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982bd2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.790+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/868011921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639828082f0 0x563982bd2910 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009fa0 tx=0x7fc4a80118d0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.790+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/868011921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828311b0 0x563982bd23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.790+0000 7fc4c1a33740 1 -- 172.21.15.114:0/868011921 >> 172.21.15.114:0/868011921 conn(0x563982825300 msgr2=0x563982bceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.792+0000 7fc4c1a33740 1 -- 172.21.15.114:0/868011921 shutdown_connections 2024-05-27T07:44:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.793+0000 7fc4c1a33740 1 -- 172.21.15.114:0/868011921 wait complete. 2024-05-27T07:44:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.793+0000 7fc4c1a33740 1 -- start start 2024-05-27T07:44:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.793+0000 7fc4c1a33740 1 Processor -- start 2024-05-27T07:44:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.794+0000 7fc4c1a33740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828104d0 0x563982bde500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.794+0000 7fc4c1a33740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828311b0 0x563982bdea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.794+0000 7fc4c1a33740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 0x563982bdca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.794+0000 7fc4c1a33740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56398281a790 con 0x5639828311b0 2024-05-27T07:44:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.794+0000 7fc4c1a33740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563982af0890 con 0x5639827efd80 2024-05-27T07:44:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.794+0000 7fc4c1a33740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563982bdf500 con 0x5639828104d0 2024-05-27T07:44:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.794+0000 7fc4c1a33740 10 client.0 _ll_register_callbacks cb 0x563982848a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.795+0000 7fc4b3fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 0x563982bdca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.795+0000 7fc4b3fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 0x563982bdca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:49282/0 (socket says 172.21.15.114:49282) 2024-05-27T07:44:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.795+0000 7fc4b3fff640 1 -- 172.21.15.114:0/2244046753 learned_addr learned my addr 172.21.15.114:0/2244046753 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.795+0000 7fc4b3fff640 1 --2- 172.21.15.114:0/2244046753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828104d0 0x563982bde500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.796+0000 7fc4b3fff640 1 -- 172.21.15.114:0/2244046753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828104d0 msgr2=0x563982bde500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.796+0000 7fc4b3fff640 1 --2- 172.21.15.114:0/2244046753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828104d0 0x563982bde500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.796+0000 7fc4b3fff640 1 -- 172.21.15.114:0/2244046753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828311b0 msgr2=0x563982bdea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:44:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.796+0000 7fc4b3fff640 1 --2- 172.21.15.114:0/2244046753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828311b0 0x563982bdea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.796+0000 7fc4b3fff640 1 -- 172.21.15.114:0/2244046753 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563982915620 con 0x5639827efd80 2024-05-27T07:44:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.798+0000 7fc4b3fff640 1 --2- 172.21.15.114:0/2244046753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 0x563982bdca10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000f410 tx=0x7fc4a0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dee1c17df34b6e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.798+0000 7fc493fff640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:44:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.799+0000 7fc493fff640 1 -- 172.21.15.114:0/2244046753 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0005020 con 0x5639827efd80 2024-05-27T07:44:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.799+0000 7fc493fff640 1 -- 172.21.15.114:0/2244046753 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc4a00051c0 con 0x5639827efd80 2024-05-27T07:44:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.799+0000 7fc4c1a33740 1 -- 172.21.15.114:0/2244046753 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4a80054d0 con 0x5639827efd80 2024-05-27T07:44:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.801+0000 7fc493fff640 1 -- 172.21.15.114:0/2244046753 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc4a002a050 con 0x5639827efd80 2024-05-27T07:44:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.801+0000 7fc4c1a33740 10 client.25937 fetch_fsmap learned FSMap version 30 2024-05-27T07:44:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.801+0000 7fc4c1a33740 1 -- 172.21.15.114:0/2244046753 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563982c051e0 con 0x5639827efd80 2024-05-27T07:44:35.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.802+0000 7fc493fff640 1 -- 172.21.15.114:0/2244046753 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4a80054d0 con 0x5639827efd80 2024-05-27T07:44:35.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.802+0000 7fc493fff640 1 -- 172.21.15.114:0/2244046753 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 30) ==== 7725+0+0 (secure 0 0 0) 0x7fc4a0020040 con 0x5639827efd80 2024-05-27T07:44:35.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.802+0000 7fc4c1a33740 10 client.25937 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:44:35.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.802+0000 7fc4c1a33740 10 client.25937 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:35.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.802+0000 7fc4c1a33740 20 client.25937 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:35.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.802+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/2244046753 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5639827c51c0 0x563982c08b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.802+0000 7fc4c1a33740 4 client.25937 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:44:35.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.802+0000 7fc4c1a33740 1 -- 172.21.15.114:0/2244046753 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563982b46b80 con 0x5639827c51c0 2024-05-27T07:44:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.807+0000 7fc4b8919640 1 --2- 172.21.15.114:0/2244046753 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5639827c51c0 0x563982c08b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.812+0000 7fc4b8919640 1 --2- 172.21.15.114:0/2244046753 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5639827c51c0 0x563982c08b40 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82c28747d77702d6 server_cookie=5c9cbf757b49de0f in_seq=0 out_seq=0 2024-05-27T07:44:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.812+0000 7fc493fff640 10 client.25937 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:35.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.813+0000 7fc493fff640 1 -- 172.21.15.114:0/2244046753 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563982b46b80 con 0x5639827c51c0 2024-05-27T07:44:35.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.813+0000 7fc493fff640 10 client.25937 handle_command_reply: tid=0 2024-05-27T07:44:35.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.813+0000 7fc4c1a33740 1 client.25937 shutdown 2024-05-27T07:44:35.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.814+0000 7fc4c1a33740 1 -- 172.21.15.114:0/2244046753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 msgr2=0x563982bdca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:35.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.814+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/2244046753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 0x563982bdca10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000f410 tx=0x7fc4a0004730 comp rx=0 tx=0).stop 2024-05-27T07:44:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.814+0000 7fc4c1a33740 1 -- 172.21.15.114:0/2244046753 shutdown_connections 2024-05-27T07:44:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.814+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/2244046753 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5639827c51c0 0x563982c08b40 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.814+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/2244046753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 0x563982bdca10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.814+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/2244046753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828311b0 0x563982bdea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.814+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/2244046753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828104d0 0x563982bde500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.814+0000 7fc4c1a33740 1 -- 172.21.15.114:0/2244046753 >> 172.21.15.114:0/2244046753 conn(0x563982825300 msgr2=0x563982bd2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.814+0000 7fc493fff640 0 client.25937 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.814+0000 7fc4c1a33740 1 -- 172.21.15.114:0/2244046753 shutdown_connections 2024-05-27T07:44:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.814+0000 7fc4c1a33740 1 -- 172.21.15.114:0/2244046753 wait complete. 2024-05-27T07:44:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.814+0000 7fc4c1a33740 20 client.25937 trim_cache size 0 max 16384 2024-05-27T07:44:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.830+0000 7fc4c1a33740 1 Processor -- start 2024-05-27T07:44:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.830+0000 7fc4c1a33740 1 -- start start 2024-05-27T07:44:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.830+0000 7fc4c1a33740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639827efd80 0x563982bd2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.830+0000 7fc4c1a33740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982c0e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.830+0000 7fc4c1a33740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639828311b0 0x563982c0e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.830+0000 7fc4c1a33740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b41a7810 con 0x5639828104d0 2024-05-27T07:44:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.830+0000 7fc4c1a33740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b4072410 con 0x5639828311b0 2024-05-27T07:44:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.830+0000 7fc4c1a33740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b40745c0 con 0x5639827efd80 2024-05-27T07:44:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc4b8919640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982c0e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc4b8919640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982c0e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54892/0 (socket says 172.21.15.114:54892) 2024-05-27T07:44:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc4b8919640 1 -- 172.21.15.114:0/521427587 learned_addr learned my addr 172.21.15.114:0/521427587 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc4b3fff640 1 --2- 172.21.15.114:0/521427587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639827efd80 0x563982bd2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc4b3fff640 1 --2- 172.21.15.114:0/521427587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639828311b0 0x563982c0e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc4b8919640 1 -- 172.21.15.114:0/521427587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639827efd80 msgr2=0x563982bd2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:35.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc4b8919640 1 --2- 172.21.15.114:0/521427587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639827efd80 0x563982bd2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc4b8919640 1 -- 172.21.15.114:0/521427587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639828311b0 msgr2=0x563982c0e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:35.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc4b8919640 1 --2- 172.21.15.114:0/521427587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639828311b0 0x563982c0e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc4b8919640 1 -- 172.21.15.114:0/521427587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563982bd85d0 con 0x5639828104d0 2024-05-27T07:44:35.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc4b8919640 1 --2- 172.21.15.114:0/521427587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982c0e340 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009fa0 tx=0x7fc4a8004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b931ece214a9f12 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:35.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc49e7fc640 1 -- 172.21.15.114:0/521427587 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a801d040 con 0x5639828104d0 2024-05-27T07:44:35.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc49e7fc640 1 -- 172.21.15.114:0/521427587 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4a801e040 con 0x5639828104d0 2024-05-27T07:44:35.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.831+0000 7fc49e7fc640 1 -- 172.21.15.114:0/521427587 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a80294b0 con 0x5639828104d0 2024-05-27T07:44:35.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.832+0000 7fc4c1a33740 1 -- 172.21.15.114:0/521427587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 msgr2=0x563982c0e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:35.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.832+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/521427587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982c0e340 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009fa0 tx=0x7fc4a8004840 comp rx=0 tx=0).stop 2024-05-27T07:44:35.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.832+0000 7fc4c1a33740 1 -- 172.21.15.114:0/521427587 shutdown_connections 2024-05-27T07:44:35.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.832+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/521427587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639828311b0 0x563982c0e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.832+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/521427587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982c0e340 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.832+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/521427587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639827efd80 0x563982bd2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.832+0000 7fc4c1a33740 1 -- 172.21.15.114:0/521427587 >> 172.21.15.114:0/521427587 conn(0x563982825300 msgr2=0x563982bd2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:35.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.832+0000 7fc4c1a33740 1 -- 172.21.15.114:0/521427587 shutdown_connections 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.832+0000 7fc4c1a33740 1 -- 172.21.15.114:0/521427587 wait complete. 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4c1a33740 1 -- start start 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4c1a33740 1 Processor -- start 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4c1a33740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 0x563982bea5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4c1a33740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982beab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4c1a33740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828311b0 0x563982c6f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4c1a33740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b4073ef0 con 0x5639828104d0 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4c1a33740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b4072a80 con 0x5639827efd80 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4c1a33740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b4073150 con 0x5639828311b0 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4c1a33740 10 client.0 _ll_register_callbacks cb 0x7fc4a0009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4b3fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 0x563982bea5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4b8919640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982beab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4b3fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 0x563982bea5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:49302/0 (socket says 172.21.15.114:49302) 2024-05-27T07:44:35.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4b3fff640 1 -- 172.21.15.114:0/115200758 learned_addr learned my addr 172.21.15.114:0/115200758 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:35.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.833+0000 7fc4b3fff640 1 --2- 172.21.15.114:0/115200758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828311b0 0x563982c6f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc4b3fff640 1 -- 172.21.15.114:0/115200758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828311b0 msgr2=0x563982c6f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:35.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc4b3fff640 1 --2- 172.21.15.114:0/115200758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828311b0 0x563982c6f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc4b3fff640 1 -- 172.21.15.114:0/115200758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 msgr2=0x563982beab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:35.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc4b3fff640 1 --2- 172.21.15.114:0/115200758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982beab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:35.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc4b3fff640 1 -- 172.21.15.114:0/115200758 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563982c6f750 con 0x5639827efd80 2024-05-27T07:44:35.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc4b8919640 1 --2- 172.21.15.114:0/115200758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982beab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:35.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc4b3fff640 1 --2- 172.21.15.114:0/115200758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 0x563982bea5e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009c10 tx=0x7fc4a0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3070256d63928f23 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:35.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc493fff640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:44:35.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc493fff640 1 -- 172.21.15.114:0/115200758 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0027040 con 0x5639827efd80 2024-05-27T07:44:35.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc4c1a33740 1 -- 172.21.15.114:0/115200758 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4a801e040 con 0x5639827efd80 2024-05-27T07:44:35.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc493fff640 1 -- 172.21.15.114:0/115200758 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc4a0005020 con 0x5639827efd80 2024-05-27T07:44:35.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc493fff640 1 -- 172.21.15.114:0/115200758 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc4a0040050 con 0x5639827efd80 2024-05-27T07:44:35.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc4c1a33740 10 client.25943 fetch_fsmap learned FSMap version 30 2024-05-27T07:44:35.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc4c1a33740 1 -- 172.21.15.114:0/115200758 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563982c70980 con 0x5639827efd80 2024-05-27T07:44:35.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.834+0000 7fc493fff640 1 -- 172.21.15.114:0/115200758 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4a801e040 con 0x5639827efd80 2024-05-27T07:44:35.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.835+0000 7fc493fff640 1 -- 172.21.15.114:0/115200758 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 30) ==== 7725+0+0 (secure 0 0 0) 0x7fc4a0004b90 con 0x5639827efd80 2024-05-27T07:44:35.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.835+0000 7fc4c1a33740 10 client.25943 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:44:35.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.835+0000 7fc4c1a33740 10 client.25943 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:35.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.835+0000 7fc4c1a33740 20 client.25943 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:35.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.835+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/115200758 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563982c70fb0 0x563982c733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:35.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.835+0000 7fc4c1a33740 4 client.25943 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:44:35.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.835+0000 7fc4c1a33740 1 -- 172.21.15.114:0/115200758 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563982bdedd0 con 0x563982c70fb0 2024-05-27T07:44:35.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.835+0000 7fc4b8919640 1 --2- 172.21.15.114:0/115200758 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563982c70fb0 0x563982c733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:35.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.835+0000 7fc493fff640 10 client.25943 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:35.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:35.835+0000 7fc4b8919640 1 --2- 172.21.15.114:0/115200758 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563982c70fb0 0x563982c733d0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=648bb7bc0f69ffb9 server_cookie=5e98b0eb7aae568 in_seq=0 out_seq=0 2024-05-27T07:44:35.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:35 smithi163 ceph-mon[36451]: pgmap v152: 129 pgs: 129 active+clean; 50 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 407 op/s 2024-05-27T07:44:35.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:35 smithi186 ceph-mon[36647]: pgmap v152: 129 pgs: 129 active+clean; 50 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 407 op/s 2024-05-27T07:44:36.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:35 smithi114 ceph-mon[32038]: pgmap v152: 129 pgs: 129 active+clean; 50 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 407 op/s 2024-05-27T07:44:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.121+0000 7fc493fff640 1 -- 172.21.15.114:0/115200758 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x563982bdedd0 con 0x563982c70fb0 2024-05-27T07:44:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.121+0000 7fc493fff640 10 client.25943 handle_command_reply: tid=0 2024-05-27T07:44:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.122+0000 7fc4c1a33740 1 client.25943 shutdown 2024-05-27T07:44:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.122+0000 7fc4c1a33740 1 -- 172.21.15.114:0/115200758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 msgr2=0x563982bea5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.122+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/115200758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 0x563982bea5e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009c10 tx=0x7fc4a0004730 comp rx=0 tx=0).stop 2024-05-27T07:44:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.122+0000 7fc4c1a33740 1 -- 172.21.15.114:0/115200758 shutdown_connections 2024-05-27T07:44:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.122+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/115200758 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563982c70fb0 0x563982c733d0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.122+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/115200758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639828311b0 0x563982c6f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.122+0000 7fc493fff640 0 client.25943 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.122+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/115200758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5639828104d0 0x563982beab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.122+0000 7fc4c1a33740 1 --2- 172.21.15.114:0/115200758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5639827efd80 0x563982bea5e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.122+0000 7fc4c1a33740 1 -- 172.21.15.114:0/115200758 >> 172.21.15.114:0/115200758 conn(0x563982825300 msgr2=0x563982bdbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.122+0000 7fc4c1a33740 1 -- 172.21.15.114:0/115200758 shutdown_connections 2024-05-27T07:44:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.122+0000 7fc4c1a33740 1 -- 172.21.15.114:0/115200758 wait complete. 2024-05-27T07:44:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.122+0000 7fc4c1a33740 20 client.25943 trim_cache size 0 max 16384 2024-05-27T07:44:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:44:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:44:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:44:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.123+0000 7fc4bab7d640 1 -- 172.21.15.114:0/599246990 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc4a40b29d0 msgr2=0x7fc4a40b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.123+0000 7fc4bab7d640 1 --2- 172.21.15.114:0/599246990 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc4a40b29d0 0x7fc4a40b4ee0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac0c6c20 tx=0x7fc4ac0095c0 comp rx=0 tx=0).stop 2024-05-27T07:44:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.123+0000 7fc4bab7d640 1 -- 172.21.15.114:0/599246990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4b4109010 msgr2=0x7fc4b419c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.123+0000 7fc4bab7d640 1 --2- 172.21.15.114:0/599246990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4b4109010 0x7fc4b419c100 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac005cc0 tx=0x7fc4ac031700 comp rx=0 tx=0).stop 2024-05-27T07:44:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.123+0000 7fc4bab7d640 1 -- 172.21.15.114:0/599246990 shutdown_connections 2024-05-27T07:44:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.123+0000 7fc4bab7d640 1 --2- 172.21.15.114:0/599246990 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc4a40b29d0 0x7fc4a40b4ee0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.123+0000 7fc4bab7d640 1 --2- 172.21.15.114:0/599246990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4b4109010 0x7fc4b419c100 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.123+0000 7fc4bab7d640 1 --2- 172.21.15.114:0/599246990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4b4108b70 0x7fc4b419bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.123+0000 7fc4bab7d640 1 --2- 172.21.15.114:0/599246990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4b4103920 0x7fc4b419b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.123+0000 7fc4bab7d640 1 -- 172.21.15.114:0/599246990 >> 172.21.15.114:0/599246990 conn(0x7fc4b40fda10 msgr2=0x7fc4b41001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.123+0000 7fc4bab7d640 1 -- 172.21.15.114:0/599246990 shutdown_connections 2024-05-27T07:44:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:36.123+0000 7fc4bab7d640 1 -- 172.21.15.114:0/599246990 wait complete. 2024-05-27T07:44:36.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 427108 13.21 MB/sec execute 48 sec latency 205.798 ms 2024-05-27T07:44:36.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:36 smithi163 ceph-mon[36451]: pgmap v153: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.0 MiB/s wr, 526 op/s 2024-05-27T07:44:36.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:36 smithi186 ceph-mon[36647]: pgmap v153: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.0 MiB/s wr, 526 op/s 2024-05-27T07:44:37.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:36 smithi114 ceph-mon[32038]: pgmap v153: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.0 MiB/s wr, 526 op/s 2024-05-27T07:44:37.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 429920 13.23 MB/sec execute 49 sec latency 15.199 ms 2024-05-27T07:44:38.245 INFO:tasks.workunit.client.0.smithi114.stdout: 1 432416 13.23 MB/sec execute 50 sec latency 100.320 ms 2024-05-27T07:44:39.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 434652 13.19 MB/sec execute 51 sec latency 15.346 ms 2024-05-27T07:44:39.474 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:39 smithi163 ceph-mon[36451]: pgmap v154: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 247 op/s 2024-05-27T07:44:39.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:39 smithi114 ceph-mon[32038]: pgmap v154: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 247 op/s 2024-05-27T07:44:39.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:39 smithi186 ceph-mon[36647]: pgmap v154: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 247 op/s 2024-05-27T07:44:40.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 437197 13.21 MB/sec execute 52 sec latency 15.238 ms 2024-05-27T07:44:41.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 440317 13.31 MB/sec execute 53 sec latency 15.162 ms 2024-05-27T07:44:41.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:41 smithi114 ceph-mon[32038]: pgmap v155: 129 pgs: 129 active+clean; 48 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 397 op/s 2024-05-27T07:44:41.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:41 smithi163 ceph-mon[36451]: pgmap v155: 129 pgs: 129 active+clean; 48 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 397 op/s 2024-05-27T07:44:41.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:41 smithi186 ceph-mon[36647]: pgmap v155: 129 pgs: 129 active+clean; 48 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 397 op/s 2024-05-27T07:44:42.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 443159 13.33 MB/sec execute 54 sec latency 24.731 ms 2024-05-27T07:44:42.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:44:43.108 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:44:43.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 444684 13.21 MB/sec execute 55 sec latency 64.690 ms 2024-05-27T07:44:43.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:43 smithi114 ceph-mon[32038]: pgmap v156: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.0 MiB/s wr, 451 op/s 2024-05-27T07:44:43.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:43 smithi163 ceph-mon[36451]: pgmap v156: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.0 MiB/s wr, 451 op/s 2024-05-27T07:44:43.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:43 smithi186 ceph-mon[36647]: pgmap v156: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.0 MiB/s wr, 451 op/s 2024-05-27T07:44:44.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 447396 13.28 MB/sec execute 56 sec latency 28.868 ms 2024-05-27T07:44:45.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.173+0000 7f1fbc234700 1 -- 172.21.15.114:0/1367460317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb4107de0 msgr2=0x7f1fb41081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:45.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.173+0000 7f1fbc234700 1 --2- 172.21.15.114:0/1367460317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb4107de0 0x7f1fb41081f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40099c0 tx=0x7f1fa4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.173+0000 7f1fbc234700 1 -- 172.21.15.114:0/1367460317 shutdown_connections 2024-05-27T07:44:45.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.173+0000 7f1fbc234700 1 --2- 172.21.15.114:0/1367460317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fb4108730 0x7f1fb4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.173+0000 7f1fbc234700 1 --2- 172.21.15.114:0/1367460317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb41034b0 0x7f1fb41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.173+0000 7f1fbc234700 1 --2- 172.21.15.114:0/1367460317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb4107de0 0x7f1fb41081f0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.173+0000 7f1fbc234700 1 -- 172.21.15.114:0/1367460317 >> 172.21.15.114:0/1367460317 conn(0x7f1fb40fedb0 msgr2=0x7f1fb41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:45.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.173+0000 7f1fbc234700 1 -- 172.21.15.114:0/1367460317 shutdown_connections 2024-05-27T07:44:45.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.173+0000 7f1fbc234700 1 -- 172.21.15.114:0/1367460317 wait complete. 2024-05-27T07:44:45.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.174+0000 7f1fbc234700 1 Processor -- start 2024-05-27T07:44:45.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.174+0000 7f1fbc234700 1 -- start start 2024-05-27T07:44:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.174+0000 7f1fbc234700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb41034b0 0x7f1fb419bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.174+0000 7f1fbc234700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb4107de0 0x7f1fb419c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.174+0000 7f1fbc234700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fb4108730 0x7f1fb419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.174+0000 7f1fbc234700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb4113390 con 0x7f1fb4107de0 2024-05-27T07:44:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.174+0000 7f1fbc234700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb4113210 con 0x7f1fb4108730 2024-05-27T07:44:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.174+0000 7f1fbc234700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb4113510 con 0x7f1fb41034b0 2024-05-27T07:44:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.174+0000 7f1fb9fd0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb41034b0 0x7f1fb419bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.174+0000 7f1fb9fd0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb41034b0 0x7f1fb419bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55404/0 (socket says 172.21.15.114:55404) 2024-05-27T07:44:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.174+0000 7f1fb97cf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb4107de0 0x7f1fb419c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.174+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/1774018957 learned_addr learned my addr 172.21.15.114:0/1774018957 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.174+0000 7f1fba7d1700 1 --2- 172.21.15.114:0/1774018957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fb4108730 0x7f1fb419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.175+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/1774018957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fb4108730 msgr2=0x7f1fb419c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:45.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.175+0000 7f1fb9fd0700 1 --2- 172.21.15.114:0/1774018957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fb4108730 0x7f1fb419c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.175+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/1774018957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb4107de0 msgr2=0x7f1fb419c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:45.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.175+0000 7f1fb9fd0700 1 --2- 172.21.15.114:0/1774018957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb4107de0 0x7f1fb419c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.175+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/1774018957 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb41a7020 con 0x7f1fb41034b0 2024-05-27T07:44:45.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.175+0000 7f1fb9fd0700 1 --2- 172.21.15.114:0/1774018957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb41034b0 0x7f1fb419bcd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000e9a0 tx=0x7f1fb000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:45.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.175+0000 7f1faaffd700 1 -- 172.21.15.114:0/1774018957 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb000ca10 con 0x7f1fb41034b0 2024-05-27T07:44:45.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.175+0000 7f1faaffd700 1 -- 172.21.15.114:0/1774018957 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fb000cbb0 con 0x7f1fb41034b0 2024-05-27T07:44:45.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.175+0000 7f1fbc234700 1 -- 172.21.15.114:0/1774018957 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb41a7310 con 0x7f1fb41034b0 2024-05-27T07:44:45.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.175+0000 7f1faaffd700 1 -- 172.21.15.114:0/1774018957 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0018680 con 0x7f1fb41034b0 2024-05-27T07:44:45.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.175+0000 7f1fbc234700 1 -- 172.21.15.114:0/1774018957 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb41a7820 con 0x7f1fb41034b0 2024-05-27T07:44:45.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.179+0000 7f1faaffd700 1 -- 172.21.15.114:0/1774018957 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f1fb0010430 con 0x7f1fb41034b0 2024-05-27T07:44:45.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.179+0000 7f1fcac49b80 1 Processor -- start 2024-05-27T07:44:45.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.179+0000 7f1fcac49b80 1 -- start start 2024-05-27T07:44:45.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.179+0000 7f1fcac49b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d101f8be40 0x55d102110b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fcac49b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d102111070 0x55d10210abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1faaffd700 1 --2- 172.21.15.114:0/1774018957 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1fa00b29f0 0x7f1fa00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fcac49b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 0x55d10210b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fcac49b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d102038d70 con 0x55d102111070 2024-05-27T07:44:45.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fcac49b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d102028910 con 0x55d10210b120 2024-05-27T07:44:45.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fcac49b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d10202c950 con 0x55d101f8be40 2024-05-27T07:44:45.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fb9fd0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d102111070 0x55d10210abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fba7d1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d101f8be40 0x55d102110b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fb9fd0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d102111070 0x55d10210abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41612/0 (socket says 172.21.15.114:41612) 2024-05-27T07:44:45.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/4058572461 learned_addr learned my addr 172.21.15.114:0/4058572461 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:45.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fba7d1700 1 --2- 172.21.15.114:0/4058572461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d101f8be40 0x55d102110b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55418/0 (socket says 172.21.15.114:55418) 2024-05-27T07:44:45.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fba7d1700 1 --2- 172.21.15.114:0/4058572461 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 0x55d10210b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fb97cf700 1 --2- 172.21.15.114:0/1774018957 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1fa00b29f0 0x7f1fa00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fba7d1700 1 -- 172.21.15.114:0/4058572461 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 msgr2=0x55d10210b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:45.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fba7d1700 1 --2- 172.21.15.114:0/4058572461 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 0x55d10210b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fba7d1700 1 -- 172.21.15.114:0/4058572461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d102111070 msgr2=0x55d10210abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:45.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fba7d1700 1 --2- 172.21.15.114:0/4058572461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d102111070 0x55d10210abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fba7d1700 1 -- 172.21.15.114:0/4058572461 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d101fd1fb0 con 0x55d101f8be40 2024-05-27T07:44:45.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1faaffd700 1 -- 172.21.15.114:0/1774018957 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1fb00e2b70 con 0x7f1fb41034b0 2024-05-27T07:44:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.180+0000 7f1fba7d1700 1 --2- 172.21.15.114:0/4058572461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d101f8be40 0x55d102110b30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1fb419ccb0 tx=0x7f1fac00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78af57931c814d87 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.181+0000 7f1f9ffff700 1 -- 172.21.15.114:0/4058572461 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac003a00 con 0x55d101f8be40 2024-05-27T07:44:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.181+0000 7f1f9ffff700 1 -- 172.21.15.114:0/4058572461 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fac003ba0 con 0x55d101f8be40 2024-05-27T07:44:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.181+0000 7f1f9ffff700 1 -- 172.21.15.114:0/4058572461 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac01a3e0 con 0x55d101f8be40 2024-05-27T07:44:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.182+0000 7f1fcac49b80 1 -- 172.21.15.114:0/4058572461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d101f8be40 msgr2=0x55d102110b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.182+0000 7f1fb97cf700 1 --2- 172.21.15.114:0/1774018957 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1fa00b29f0 0x7f1fa00b4f00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4000c00 tx=0x7f1fa401a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.182+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/4058572461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d101f8be40 0x55d102110b30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1fb419ccb0 tx=0x7f1fac00b040 comp rx=0 tx=0).stop 2024-05-27T07:44:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.182+0000 7f1fcac49b80 1 -- 172.21.15.114:0/4058572461 shutdown_connections 2024-05-27T07:44:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.182+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/4058572461 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 0x55d10210b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.182+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/4058572461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d101f8be40 0x55d102110b30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.182+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/4058572461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d102111070 0x55d10210abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.182+0000 7f1fcac49b80 1 -- 172.21.15.114:0/4058572461 >> 172.21.15.114:0/4058572461 conn(0x55d101f8e770 msgr2=0x55d102105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.182+0000 7f1fcac49b80 1 -- 172.21.15.114:0/4058572461 shutdown_connections 2024-05-27T07:44:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.182+0000 7f1fcac49b80 1 -- 172.21.15.114:0/4058572461 wait complete. 2024-05-27T07:44:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.182+0000 7f1fcac49b80 1 -- start start 2024-05-27T07:44:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.182+0000 7f1fcac49b80 1 Processor -- start 2024-05-27T07:44:45.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fcac49b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d101f8be40 0x55d102114920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fcac49b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d10210b120 0x55d102114e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fcac49b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d102111070 0x55d1021153a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fcac49b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d101f2efa0 con 0x55d102111070 2024-05-27T07:44:45.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fcac49b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d101f90da0 con 0x55d101f8be40 2024-05-27T07:44:45.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fcac49b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d10211bf70 con 0x55d10210b120 2024-05-27T07:44:45.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fcac49b80 10 client.0 _ll_register_callbacks cb 0x55d101fbf5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:45.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fb97cf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d10210b120 0x55d102114e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fb97cf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d10210b120 0x55d102114e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55420/0 (socket says 172.21.15.114:55420) 2024-05-27T07:44:45.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fba7d1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d102111070 0x55d1021153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fb97cf700 1 -- 172.21.15.114:0/3772176743 learned_addr learned my addr 172.21.15.114:0/3772176743 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:45.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fb9fd0700 1 --2- 172.21.15.114:0/3772176743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d101f8be40 0x55d102114920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/3772176743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d10210b120 msgr2=0x55d102114e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:45.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fb9fd0700 1 --2- 172.21.15.114:0/3772176743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d10210b120 0x55d102114e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.183+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/3772176743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d102111070 msgr2=0x55d1021153a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:45.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1fb9fd0700 1 --2- 172.21.15.114:0/3772176743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d102111070 0x55d1021153a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/3772176743 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d101f788d0 con 0x55d101f8be40 2024-05-27T07:44:45.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1fba7d1700 1 --2- 172.21.15.114:0/3772176743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d102111070 0x55d1021153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:45.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1fb9fd0700 1 --2- 172.21.15.114:0/3772176743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d101f8be40 0x55d102114920 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1fb00e0510 tx=0x7f1fb0073340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=489dbbb0b4a4e39a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:45.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1f9d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:44:45.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1f9d7fa700 1 -- 172.21.15.114:0/3772176743 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb00d8420 con 0x55d101f8be40 2024-05-27T07:44:45.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1f9d7fa700 1 -- 172.21.15.114:0/3772176743 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1fb00a6090 con 0x55d101f8be40 2024-05-27T07:44:45.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3772176743 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fac003ba0 con 0x55d101f8be40 2024-05-27T07:44:45.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1f9d7fa700 1 -- 172.21.15.114:0/3772176743 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1fb00b2240 con 0x55d101f8be40 2024-05-27T07:44:45.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1fcac49b80 10 client.25955 fetch_fsmap learned FSMap version 30 2024-05-27T07:44:45.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3772176743 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d101fe2760 con 0x55d101f8be40 2024-05-27T07:44:45.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1f9d7fa700 1 -- 172.21.15.114:0/3772176743 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fac003ba0 con 0x55d101f8be40 2024-05-27T07:44:45.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1f9d7fa700 1 -- 172.21.15.114:0/3772176743 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 30) ==== 7725+0+0 (secure 0 0 0) 0x7f1fb00a9d00 con 0x55d101f8be40 2024-05-27T07:44:45.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1fcac49b80 10 client.25955 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:44:45.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.184+0000 7f1fcac49b80 10 client.25955 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:45.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.185+0000 7f1fcac49b80 20 client.25955 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:45.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.185+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/3772176743 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d102128ed0 0x55d10212b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.185+0000 7f1fcac49b80 4 client.25955 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:44:45.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.185+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3772176743 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d10203acc0 con 0x55d102128ed0 2024-05-27T07:44:45.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.185+0000 7f1fb97cf700 1 --2- 172.21.15.114:0/3772176743 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d102128ed0 0x55d10212b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.185+0000 7f1f9d7fa700 10 client.25955 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:45.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.185+0000 7f1fb97cf700 1 --2- 172.21.15.114:0/3772176743 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d102128ed0 0x55d10212b360 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=430ffd4b913e25b6 server_cookie=f89597db67f70912 in_seq=0 out_seq=0 2024-05-27T07:44:45.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.186+0000 7f1f9d7fa700 1 -- 172.21.15.114:0/3772176743 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d10203acc0 con 0x55d102128ed0 2024-05-27T07:44:45.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.186+0000 7f1f9d7fa700 10 client.25955 handle_command_reply: tid=0 2024-05-27T07:44:45.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.186+0000 7f1fcac49b80 1 client.25955 shutdown 2024-05-27T07:44:45.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.186+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3772176743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d101f8be40 msgr2=0x55d102114920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:45.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.186+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/3772176743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d101f8be40 0x55d102114920 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1fb00e0510 tx=0x7f1fb0073340 comp rx=0 tx=0).stop 2024-05-27T07:44:45.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.186+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3772176743 shutdown_connections 2024-05-27T07:44:45.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.186+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/3772176743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d102111070 0x55d1021153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.186+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/3772176743 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d102128ed0 0x55d10212b360 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.186+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/3772176743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d101f8be40 0x55d102114920 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.186+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/3772176743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d10210b120 0x55d102114e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.186+0000 7f1f9d7fa700 0 client.25955 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:45.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.186+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3772176743 >> 172.21.15.114:0/3772176743 conn(0x55d101f8e770 msgr2=0x55d102109bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:45.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.187+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3772176743 shutdown_connections 2024-05-27T07:44:45.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.187+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3772176743 wait complete. 2024-05-27T07:44:45.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.187+0000 7f1fcac49b80 20 client.25955 trim_cache size 0 max 16384 2024-05-27T07:44:45.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.209+0000 7f1fcac49b80 1 Processor -- start 2024-05-27T07:44:45.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.209+0000 7f1fcac49b80 1 -- start start 2024-05-27T07:44:45.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.209+0000 7f1fcac49b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 0x55d1021151e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.209+0000 7f1fcac49b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 0x55d10210e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.209+0000 7f1fcac49b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d102111070 0x55d10210e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.209+0000 7f1fcac49b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41af3b0 con 0x55d101f8be40 2024-05-27T07:44:45.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.209+0000 7f1fcac49b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41ac460 con 0x55d10210b120 2024-05-27T07:44:45.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.209+0000 7f1fcac49b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41ae610 con 0x55d102111070 2024-05-27T07:44:45.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.209+0000 7f1fb9fd0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 0x55d1021151e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1fb97cf700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 0x55d10210e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1fb9fd0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 0x55d1021151e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41632/0 (socket says 172.21.15.114:41632) 2024-05-27T07:44:45.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/4024960832 learned_addr learned my addr 172.21.15.114:0/4024960832 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:45.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1fba7d1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d102111070 0x55d10210e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/4024960832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d102111070 msgr2=0x55d10210e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:45.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1fb9fd0700 1 --2- 172.21.15.114:0/4024960832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d102111070 0x55d10210e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/4024960832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 msgr2=0x55d10210e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:45.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1fb9fd0700 1 --2- 172.21.15.114:0/4024960832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 0x55d10210e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/4024960832 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d102112e90 con 0x55d101f8be40 2024-05-27T07:44:45.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1fb97cf700 1 --2- 172.21.15.114:0/4024960832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 0x55d10210e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:45.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1fb9fd0700 1 --2- 172.21.15.114:0/4024960832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 0x55d1021151e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1fb00098c0 tx=0x7f1fb0010c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892609019b9cb728 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:45.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1f9f7fe700 1 -- 172.21.15.114:0/4024960832 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb00a95b0 con 0x55d101f8be40 2024-05-27T07:44:45.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1f9f7fe700 1 -- 172.21.15.114:0/4024960832 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fb00aa1a0 con 0x55d101f8be40 2024-05-27T07:44:45.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.210+0000 7f1f9f7fe700 1 -- 172.21.15.114:0/4024960832 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb00b2470 con 0x55d101f8be40 2024-05-27T07:44:45.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.211+0000 7f1fcac49b80 1 -- 172.21.15.114:0/4024960832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 msgr2=0x55d1021151e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:45.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.211+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/4024960832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 0x55d1021151e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1fb00098c0 tx=0x7f1fb0010c10 comp rx=0 tx=0).stop 2024-05-27T07:44:45.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.211+0000 7f1fcac49b80 1 -- 172.21.15.114:0/4024960832 shutdown_connections 2024-05-27T07:44:45.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.211+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/4024960832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d102111070 0x55d10210e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.211+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/4024960832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 0x55d1021151e0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.211+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/4024960832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 0x55d10210e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.211+0000 7f1fcac49b80 1 -- 172.21.15.114:0/4024960832 >> 172.21.15.114:0/4024960832 conn(0x55d101f8e770 msgr2=0x55d10210f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:45.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.211+0000 7f1fcac49b80 1 -- 172.21.15.114:0/4024960832 shutdown_connections 2024-05-27T07:44:45.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.211+0000 7f1fcac49b80 1 -- 172.21.15.114:0/4024960832 wait complete. 2024-05-27T07:44:45.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.212+0000 7f1fcac49b80 1 -- start start 2024-05-27T07:44:45.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.212+0000 7f1fcac49b80 1 Processor -- start 2024-05-27T07:44:45.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.212+0000 7f1fcac49b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 0x55d102121400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.212+0000 7f1fcac49b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 0x55d102121940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.212+0000 7f1fcac49b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d102111070 0x55d10212ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.212+0000 7f1fcac49b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41adf40 con 0x55d101f8be40 2024-05-27T07:44:45.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.212+0000 7f1fcac49b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41acad0 con 0x55d10210b120 2024-05-27T07:44:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.212+0000 7f1fcac49b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41ad1a0 con 0x55d102111070 2024-05-27T07:44:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.212+0000 7f1fcac49b80 10 client.0 _ll_register_callbacks cb 0x55d101fbf5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.212+0000 7f1fb9fd0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 0x55d102121400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1fb9fd0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 0x55d102121400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41640/0 (socket says 172.21.15.114:41640) 2024-05-27T07:44:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/3762642773 learned_addr learned my addr 172.21.15.114:0/3762642773 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1fba7d1700 1 --2- 172.21.15.114:0/3762642773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d102111070 0x55d10212ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1fb97cf700 1 --2- 172.21.15.114:0/3762642773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 0x55d102121940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/3762642773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d102111070 msgr2=0x55d10212ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1fb9fd0700 1 --2- 172.21.15.114:0/3762642773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d102111070 0x55d10212ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/3762642773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 msgr2=0x55d102121940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1fb9fd0700 1 --2- 172.21.15.114:0/3762642773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 0x55d102121940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1fb9fd0700 1 -- 172.21.15.114:0/3762642773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d102130690 con 0x55d101f8be40 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1fb9fd0700 1 --2- 172.21.15.114:0/3762642773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 0x55d102121400 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1fb00a9a30 tx=0x7f1fb00b26a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98af80d9f1e5e18 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1f9d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1f9d7fa700 1 -- 172.21.15.114:0/3762642773 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb00ade50 con 0x55d101f8be40 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1f9d7fa700 1 -- 172.21.15.114:0/3762642773 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1fb00a6090 con 0x55d101f8be40 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.213+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3762642773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fb00aa1a0 con 0x55d101f8be40 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.214+0000 7f1f9d7fa700 1 -- 172.21.15.114:0/3762642773 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1fb00a9aa0 con 0x55d101f8be40 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.214+0000 7f1fcac49b80 10 client.16437 fetch_fsmap learned FSMap version 30 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.214+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3762642773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d1021933d0 con 0x55d101f8be40 2024-05-27T07:44:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.214+0000 7f1f9d7fa700 1 -- 172.21.15.114:0/3762642773 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fb00aa1a0 con 0x55d101f8be40 2024-05-27T07:44:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.214+0000 7f1f9d7fa700 1 -- 172.21.15.114:0/3762642773 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 7725+0+0 (secure 0 0 0) 0x7f1fb00a5410 con 0x55d101f8be40 2024-05-27T07:44:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.214+0000 7f1fcac49b80 10 client.16437 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:44:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.214+0000 7f1fcac49b80 10 client.16437 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.214+0000 7f1fcac49b80 20 client.16437 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.214+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/3762642773 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d1021936f0 0x55d102195b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.214+0000 7f1fcac49b80 4 client.16437 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:44:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.214+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3762642773 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d1020344a0 con 0x55d1021936f0 2024-05-27T07:44:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.214+0000 7f1fb97cf700 1 --2- 172.21.15.114:0/3762642773 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d1021936f0 0x55d102195b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.215+0000 7f1f9d7fa700 10 client.16437 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.215+0000 7f1fb97cf700 1 --2- 172.21.15.114:0/3762642773 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d1021936f0 0x55d102195b10 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3cf2d0fddf5b0ca server_cookie=6684d12d9970710e in_seq=0 out_seq=0 2024-05-27T07:44:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.215+0000 7f1f9d7fa700 1 -- 172.21.15.114:0/3762642773 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d1020344a0 con 0x55d1021936f0 2024-05-27T07:44:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.215+0000 7f1f9d7fa700 10 client.16437 handle_command_reply: tid=0 2024-05-27T07:44:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.215+0000 7f1fcac49b80 1 client.16437 shutdown 2024-05-27T07:44:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.215+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3762642773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 msgr2=0x55d102121400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.215+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/3762642773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 0x55d102121400 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1fb00a9a30 tx=0x7f1fb00b26a0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3762642773 shutdown_connections 2024-05-27T07:44:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/3762642773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d102111070 0x55d10212ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/3762642773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d101f8be40 0x55d102121400 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/3762642773 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d1021936f0 0x55d102195b10 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fcac49b80 1 --2- 172.21.15.114:0/3762642773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d10210b120 0x55d102121940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3762642773 >> 172.21.15.114:0/3762642773 conn(0x55d101f8e770 msgr2=0x55d10210e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1f9d7fa700 0 client.16437 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3762642773 shutdown_connections 2024-05-27T07:44:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fcac49b80 1 -- 172.21.15.114:0/3762642773 wait complete. 2024-05-27T07:44:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fcac49b80 20 client.16437 trim_cache size 0 max 16384 2024-05-27T07:44:45.232 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:44:45.232 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:44:45.232 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:44:45.232 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:44:45.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fbc234700 1 -- 172.21.15.114:0/1774018957 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1fa00b29f0 msgr2=0x7f1fa00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:45.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fbc234700 1 --2- 172.21.15.114:0/1774018957 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1fa00b29f0 0x7f1fa00b4f00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4000c00 tx=0x7f1fa401a040 comp rx=0 tx=0).stop 2024-05-27T07:44:45.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fbc234700 1 -- 172.21.15.114:0/1774018957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb41034b0 msgr2=0x7f1fb419bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:45.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fbc234700 1 --2- 172.21.15.114:0/1774018957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb41034b0 0x7f1fb419bcd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000e9a0 tx=0x7f1fb000ecb0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fbc234700 1 -- 172.21.15.114:0/1774018957 shutdown_connections 2024-05-27T07:44:45.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.216+0000 7f1fbc234700 1 --2- 172.21.15.114:0/1774018957 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1fa00b29f0 0x7f1fa00b4f00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.217+0000 7f1fbc234700 1 --2- 172.21.15.114:0/1774018957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fb4108730 0x7f1fb419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.217+0000 7f1fbc234700 1 --2- 172.21.15.114:0/1774018957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb41034b0 0x7f1fb419bcd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.217+0000 7f1fbc234700 1 --2- 172.21.15.114:0/1774018957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb4107de0 0x7f1fb419c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:45.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.217+0000 7f1fbc234700 1 -- 172.21.15.114:0/1774018957 >> 172.21.15.114:0/1774018957 conn(0x7f1fb40fedb0 msgr2=0x7f1fb4109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:45.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.217+0000 7f1fbc234700 1 -- 172.21.15.114:0/1774018957 shutdown_connections 2024-05-27T07:44:45.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:45.217+0000 7f1fbc234700 1 -- 172.21.15.114:0/1774018957 wait complete. 2024-05-27T07:44:45.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 450310 13.30 MB/sec execute 57 sec latency 14.963 ms 2024-05-27T07:44:45.282 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:44 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:44:45.283 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:44 smithi186 systemd[1]: Stopped Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:44:45.283 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:44 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Consumed 2.886s CPU time. 2024-05-27T07:44:45.283 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:44 smithi186 systemd[1]: Starting Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:44:45.283 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:45 smithi186 podman[72130]: 2024-05-27 07:44:45.112674708 +0000 UTC m=+0.018088992 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:44:45.283 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:45 smithi186 podman[72130]: 2024-05-27 07:44:45.223839225 +0000 UTC m=+0.129253480 container create 4e63ad794a713e120359b2a193bc8168f93fc63bdfa0142615c8fcf860c7aa30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:44:45.442 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:45 smithi114 ceph-mon[32038]: pgmap v157: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 387 op/s 2024-05-27T07:44:45.577 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:45 smithi186 ceph-mon[36647]: pgmap v157: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 387 op/s 2024-05-27T07:44:45.578 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:45 smithi186 podman[72130]: 2024-05-27 07:44:45.486729417 +0000 UTC m=+0.392143692 container init 4e63ad794a713e120359b2a193bc8168f93fc63bdfa0142615c8fcf860c7aa30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-27T07:44:45.578 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:45 smithi186 podman[72130]: 2024-05-27 07:44:45.490162008 +0000 UTC m=+0.395576287 container start 4e63ad794a713e120359b2a193bc8168f93fc63bdfa0142615c8fcf860c7aa30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:44:45.578 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:45 smithi186 ceph-mds[72149]: -- 172.21.15.186:0/477924560 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56246d1d3a40 con 0x56246d21b600 2024-05-27T07:44:45.578 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:45 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: starting mds.c at 2024-05-27T07:44:45.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:45 smithi163 ceph-mon[36451]: pgmap v157: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 387 op/s 2024-05-27T07:44:45.891 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:45 smithi186 bash[72130]: 4e63ad794a713e120359b2a193bc8168f93fc63bdfa0142615c8fcf860c7aa30 2024-05-27T07:44:45.891 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:45 smithi186 systemd[1]: Started Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:44:46.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 452863 13.27 MB/sec execute 58 sec latency 21.395 ms 2024-05-27T07:44:46.335 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:44:46.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:44:46.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:46 smithi114 ceph-mon[32038]: daemon mds.c restarted 2024-05-27T07:44:46.586 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:44:46.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:46 smithi163 ceph-mon[36451]: daemon mds.c restarted 2024-05-27T07:44:46.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:46 smithi186 ceph-mon[36647]: daemon mds.c restarted 2024-05-27T07:44:47.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 454782 13.26 MB/sec execute 59 sec latency 16.007 ms 2024-05-27T07:44:47.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:47 smithi186 ceph-mon[36647]: pgmap v158: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 509 op/s 2024-05-27T07:44:47.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:47 smithi186 ceph-mon[36647]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T07:44:47.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:47 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6836/3710253414,v1:172.21.15.186:6837/3710253414] up:boot 2024-05-27T07:44:47.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:47 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:44:47.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:47 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:44:47.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:47 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:44:47.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:47 smithi114 ceph-mon[32038]: pgmap v158: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 509 op/s 2024-05-27T07:44:47.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:47 smithi114 ceph-mon[32038]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T07:44:47.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:47 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6836/3710253414,v1:172.21.15.186:6837/3710253414] up:boot 2024-05-27T07:44:47.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:47 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:44:47.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:47 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:44:47.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:47 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:44:47.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:47 smithi163 ceph-mon[36451]: pgmap v158: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 509 op/s 2024-05-27T07:44:47.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:47 smithi163 ceph-mon[36451]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T07:44:47.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:47 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6836/3710253414,v1:172.21.15.186:6837/3710253414] up:boot 2024-05-27T07:44:47.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:47 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:44:47.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:47 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:44:47.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:47 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:44:48.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 457691 13.33 MB/sec execute 60 sec latency 75.913 ms 2024-05-27T07:44:48.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.275+0000 7f9595d8b700 1 -- 172.21.15.114:0/3322282314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9590108730 msgr2=0x7f9590110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.275+0000 7f9595d8b700 1 --2- 172.21.15.114:0/3322282314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9590108730 0x7f9590110090 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f958800b0d0 tx=0x7f958800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:44:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.275+0000 7f9595d8b700 1 -- 172.21.15.114:0/3322282314 shutdown_connections 2024-05-27T07:44:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.275+0000 7f9595d8b700 1 --2- 172.21.15.114:0/3322282314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9590108730 0x7f9590110090 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.275+0000 7f9595d8b700 1 --2- 172.21.15.114:0/3322282314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95901034b0 0x7f95901038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.275+0000 7f9595d8b700 1 --2- 172.21.15.114:0/3322282314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9590107de0 0x7f95901081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.275+0000 7f9595d8b700 1 -- 172.21.15.114:0/3322282314 >> 172.21.15.114:0/3322282314 conn(0x7f95900fedb0 msgr2=0x7f95901011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.275+0000 7f9595d8b700 1 -- 172.21.15.114:0/3322282314 shutdown_connections 2024-05-27T07:44:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.275+0000 7f9595d8b700 1 -- 172.21.15.114:0/3322282314 wait complete. 2024-05-27T07:44:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.276+0000 7f9595d8b700 1 Processor -- start 2024-05-27T07:44:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.276+0000 7f9595d8b700 1 -- start start 2024-05-27T07:44:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.276+0000 7f9595d8b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95901034b0 0x7f959019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.276+0000 7f9595d8b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9590107de0 0x7f959019fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.276+0000 7f9595d8b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9590108730 0x7f9590199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.276+0000 7f9595d8b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9590113360 con 0x7f95901034b0 2024-05-27T07:44:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.276+0000 7f9595d8b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f95901131e0 con 0x7f9590107de0 2024-05-27T07:44:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.276+0000 7f9595d8b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95901134e0 con 0x7f9590108730 2024-05-27T07:44:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.276+0000 7f958f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95901034b0 0x7f959019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.276+0000 7f958f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95901034b0 0x7f959019f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41644/0 (socket says 172.21.15.114:41644) 2024-05-27T07:44:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.276+0000 7f958ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9590108730 0x7f9590199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.276+0000 7f958f7fe700 1 -- 172.21.15.114:0/1534067918 learned_addr learned my addr 172.21.15.114:0/1534067918 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.277+0000 7f958effd700 1 --2- 172.21.15.114:0/1534067918 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9590107de0 0x7f959019fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.277+0000 7f958ffff700 1 -- 172.21.15.114:0/1534067918 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9590107de0 msgr2=0x7f959019fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.277+0000 7f958ffff700 1 --2- 172.21.15.114:0/1534067918 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9590107de0 0x7f959019fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.277+0000 7f958ffff700 1 -- 172.21.15.114:0/1534067918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95901034b0 msgr2=0x7f959019f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.277+0000 7f958ffff700 1 --2- 172.21.15.114:0/1534067918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95901034b0 0x7f959019f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.277+0000 7f958ffff700 1 -- 172.21.15.114:0/1534067918 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959019a0a0 con 0x7f9590108730 2024-05-27T07:44:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.277+0000 7f958ffff700 1 --2- 172.21.15.114:0/1534067918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9590108730 0x7f9590199a60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f958800b0d0 tx=0x7f9588007a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.277+0000 7f958cff9700 1 -- 172.21.15.114:0/1534067918 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958801f070 con 0x7f9590108730 2024-05-27T07:44:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.277+0000 7f958cff9700 1 -- 172.21.15.114:0/1534067918 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9588004c50 con 0x7f9590108730 2024-05-27T07:44:48.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.277+0000 7f9595d8b700 1 -- 172.21.15.114:0/1534067918 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f959019a330 con 0x7f9590108730 2024-05-27T07:44:48.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.277+0000 7f958cff9700 1 -- 172.21.15.114:0/1534067918 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958801b690 con 0x7f9590108730 2024-05-27T07:44:48.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.277+0000 7f9595d8b700 1 -- 172.21.15.114:0/1534067918 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f959019a810 con 0x7f9590108730 2024-05-27T07:44:48.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.278+0000 7f9595d8b700 1 -- 172.21.15.114:0/1534067918 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95900512b0 con 0x7f9590108730 2024-05-27T07:44:48.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.280+0000 7f958cff9700 1 -- 172.21.15.114:0/1534067918 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f9588022070 con 0x7f9590108730 2024-05-27T07:44:48.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.281+0000 7f958cff9700 1 --2- 172.21.15.114:0/1534067918 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95780b29f0 0x7f95780b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:48.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.281+0000 7f958f7fe700 1 --2- 172.21.15.114:0/1534067918 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95780b29f0 0x7f95780b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:48.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.281+0000 7f958cff9700 1 -- 172.21.15.114:0/1534067918 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f958800f020 con 0x7f9590108730 2024-05-27T07:44:48.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.281+0000 7f958f7fe700 1 --2- 172.21.15.114:0/1534067918 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95780b29f0 0x7f95780b4f00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f959019aba0 tx=0x7f9580005f90 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:48.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.288+0000 7f958cff9700 1 -- 172.21.15.114:0/1534067918 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9588029020 con 0x7f9590108730 2024-05-27T07:44:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.566+0000 7f9595d8b700 1 -- 172.21.15.114:0/1534067918 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f959002e5e0 con 0x7f9590108730 2024-05-27T07:44:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.567+0000 7f958cff9700 1 -- 172.21.15.114:0/1534067918 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+10999 (secure 0 0 0) 0x7f95880ab490 con 0x7f9590108730 2024-05-27T07:44:48.566 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:44:48.567 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:44:46.296164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26035":{"gid":26035,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3710253414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3710253414},{"type":"v1","addr":"172.21.15.186:6837","nonce":3710253414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:44:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.569+0000 7f9595d8b700 1 -- 172.21.15.114:0/1534067918 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95780b29f0 msgr2=0x7f95780b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.569+0000 7f9595d8b700 1 --2- 172.21.15.114:0/1534067918 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95780b29f0 0x7f95780b4f00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f959019aba0 tx=0x7f9580005f90 comp rx=0 tx=0).stop 2024-05-27T07:44:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.569+0000 7f9595d8b700 1 -- 172.21.15.114:0/1534067918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9590108730 msgr2=0x7f9590199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.569+0000 7f9595d8b700 1 --2- 172.21.15.114:0/1534067918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9590108730 0x7f9590199a60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f958800b0d0 tx=0x7f9588007a00 comp rx=0 tx=0).stop 2024-05-27T07:44:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.570+0000 7f9595d8b700 1 -- 172.21.15.114:0/1534067918 shutdown_connections 2024-05-27T07:44:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.570+0000 7f9595d8b700 1 --2- 172.21.15.114:0/1534067918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9590108730 0x7f9590199a60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.570+0000 7f9595d8b700 1 --2- 172.21.15.114:0/1534067918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95901034b0 0x7f959019f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.570+0000 7f9595d8b700 1 --2- 172.21.15.114:0/1534067918 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95780b29f0 0x7f95780b4f00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.570+0000 7f9595d8b700 1 --2- 172.21.15.114:0/1534067918 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9590107de0 0x7f959019fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.570+0000 7f9595d8b700 1 -- 172.21.15.114:0/1534067918 >> 172.21.15.114:0/1534067918 conn(0x7f95900fedb0 msgr2=0x7f9590107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.570+0000 7f9595d8b700 1 -- 172.21.15.114:0/1534067918 shutdown_connections 2024-05-27T07:44:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:48.570+0000 7f9595d8b700 1 -- 172.21.15.114:0/1534067918 wait complete. 2024-05-27T07:44:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 32 2024-05-27T07:44:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.226+0000 7f2093abd640 1 -- 172.21.15.114:0/2925713407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c109030 msgr2=0x7f208c102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.226+0000 7f2093abd640 1 --2- 172.21.15.114:0/2925713407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c109030 0x7f208c102c70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f207c009900 tx=0x7f207c02a410 comp rx=0 tx=0).stop 2024-05-27T07:44:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.226+0000 7f2093abd640 1 -- 172.21.15.114:0/2925713407 shutdown_connections 2024-05-27T07:44:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.226+0000 7f2093abd640 1 --2- 172.21.15.114:0/2925713407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f208c1032e0 0x7f208c10f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.226+0000 7f2093abd640 1 --2- 172.21.15.114:0/2925713407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c109030 0x7f208c102c70 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.226+0000 7f2093abd640 1 --2- 172.21.15.114:0/2925713407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f208c108b90 0x7f208c102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.226+0000 7f2093abd640 1 -- 172.21.15.114:0/2925713407 >> 172.21.15.114:0/2925713407 conn(0x7f208c0782d0 msgr2=0x7f208c078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.226+0000 7f2093abd640 1 -- 172.21.15.114:0/2925713407 shutdown_connections 2024-05-27T07:44:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.226+0000 7f2093abd640 1 -- 172.21.15.114:0/2925713407 wait complete. 2024-05-27T07:44:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2093abd640 1 Processor -- start 2024-05-27T07:44:49.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2093abd640 1 -- start start 2024-05-27T07:44:49.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2093abd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f208c1032e0 0x7f208c19b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2093abd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c108b90 0x7f208c19bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2093abd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f208c109030 0x7f208c19c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2093abd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f208c112d80 con 0x7f208c108b90 2024-05-27T07:44:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2093abd640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f208c112c00 con 0x7f208c1032e0 2024-05-27T07:44:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2093abd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f208c112f00 con 0x7f208c109030 2024-05-27T07:44:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2091058640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c108b90 0x7f208c19bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2091058640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c108b90 0x7f208c19bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41664/0 (socket says 172.21.15.114:41664) 2024-05-27T07:44:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2091058640 1 -- 172.21.15.114:0/4226438682 learned_addr learned my addr 172.21.15.114:0/4226438682 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f209205a640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f208c109030 0x7f208c19c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2091859640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f208c1032e0 0x7f208c19b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2091058640 1 -- 172.21.15.114:0/4226438682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f208c109030 msgr2=0x7f208c19c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2091058640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f208c109030 0x7f208c19c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2091058640 1 -- 172.21.15.114:0/4226438682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f208c1032e0 msgr2=0x7f208c19b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2091058640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f208c1032e0 0x7f208c19b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f2091058640 1 -- 172.21.15.114:0/4226438682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f208c1a6a70 con 0x7f208c108b90 2024-05-27T07:44:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.227+0000 7f209205a640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f208c109030 0x7f208c19c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.228+0000 7f2091058640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c108b90 0x7f208c19bca0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f207c006010 tx=0x7f207c004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.228+0000 7f2082ffd640 1 -- 172.21.15.114:0/4226438682 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c03f030 con 0x7f208c108b90 2024-05-27T07:44:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.228+0000 7f2093abd640 1 -- 172.21.15.114:0/4226438682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f208c1a6d00 con 0x7f208c108b90 2024-05-27T07:44:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.228+0000 7f2082ffd640 1 -- 172.21.15.114:0/4226438682 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f207c003aa0 con 0x7f208c108b90 2024-05-27T07:44:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.228+0000 7f2082ffd640 1 -- 172.21.15.114:0/4226438682 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c035070 con 0x7f208c108b90 2024-05-27T07:44:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.228+0000 7f2093abd640 1 -- 172.21.15.114:0/4226438682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f208c1a7190 con 0x7f208c108b90 2024-05-27T07:44:49.232 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f494201c700 time 2024-05-27T07:44:48.875776+0000 2024-05-27T07:44:49.232 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:44:49.232 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:49.232 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4954861ba6] 2024-05-27T07:44:49.232 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4954861dd1] 2024-05-27T07:44:49.232 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56246b81723d] 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56246b802971] 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 5: (SnapRealm::check_cache() const+0x13e) [0x56246b800fa0] 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56246b801651] 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56246b796347] 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56246b49287e] 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56246b497f2f] 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56246b4a2969] 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 11: (MDLog::_replay_thread()+0x1f38) [0x56246b83b48c] 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56246b8449cb] 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 13: (Thread::entry_wrapper()+0x43) [0x7f495483072d] 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4954830749] 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f49533d41ca] 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 16: clone() 2024-05-27T07:44:49.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:48.877+0000 7f494201c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f494201c700 time 2024-05-27T07:44:48.875776+0000 2024-05-27T07:44:49.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:44:49.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:49.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:49.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4954861ba6] 2024-05-27T07:44:49.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4954861dd1] 2024-05-27T07:44:49.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56246b81723d] 2024-05-27T07:44:49.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56246b802971] 2024-05-27T07:44:49.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 5: (SnapRealm::check_cache() const+0x13e) [0x56246b800fa0] 2024-05-27T07:44:49.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56246b801651] 2024-05-27T07:44:49.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56246b796347] 2024-05-27T07:44:49.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56246b49287e] 2024-05-27T07:44:49.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56246b497f2f] 2024-05-27T07:44:49.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56246b4a2969] 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 11: (MDLog::_replay_thread()+0x1f38) [0x56246b83b48c] 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56246b8449cb] 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 13: (Thread::entry_wrapper()+0x43) [0x7f495483072d] 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4954830749] 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f49533d41ca] 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 16: clone() 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: *** Caught signal (Aborted) ** 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: in thread 7f494201c700 thread_name:md_log_replay 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f49533ded20] 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2: gsignal() 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 3: abort() 2024-05-27T07:44:49.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4954861ced] 2024-05-27T07:44:49.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4954861dd1] 2024-05-27T07:44:49.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56246b81723d] 2024-05-27T07:44:49.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56246b802971] 2024-05-27T07:44:49.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 8: (SnapRealm::check_cache() const+0x13e) [0x56246b800fa0] 2024-05-27T07:44:49.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56246b801651] 2024-05-27T07:44:49.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56246b796347] 2024-05-27T07:44:49.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56246b49287e] 2024-05-27T07:44:49.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56246b497f2f] 2024-05-27T07:44:49.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56246b4a2969] 2024-05-27T07:44:49.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 14: (MDLog::_replay_thread()+0x1f38) [0x56246b83b48c] 2024-05-27T07:44:49.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56246b8449cb] 2024-05-27T07:44:49.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 16: (Thread::entry_wrapper()+0x43) [0x7f495483072d] 2024-05-27T07:44:49.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4954830749] 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f49533d41ca] 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 19: clone() 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:48.879+0000 7f494201c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: in thread 7f494201c700 thread_name:md_log_replay 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f49533ded20] 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2: gsignal() 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 3: abort() 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4954861ced] 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4954861dd1] 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56246b81723d] 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56246b802971] 2024-05-27T07:44:49.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 8: (SnapRealm::check_cache() const+0x13e) [0x56246b800fa0] 2024-05-27T07:44:49.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56246b801651] 2024-05-27T07:44:49.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56246b796347] 2024-05-27T07:44:49.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56246b49287e] 2024-05-27T07:44:49.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56246b497f2f] 2024-05-27T07:44:49.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56246b4a2969] 2024-05-27T07:44:49.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 14: (MDLog::_replay_thread()+0x1f38) [0x56246b83b48c] 2024-05-27T07:44:49.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56246b8449cb] 2024-05-27T07:44:49.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 16: (Thread::entry_wrapper()+0x43) [0x7f495483072d] 2024-05-27T07:44:49.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4954830749] 2024-05-27T07:44:49.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f49533d41ca] 2024-05-27T07:44:49.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 19: clone() 2024-05-27T07:44:49.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:44:49.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: -1> 2024-05-27T07:44:48.877+0000 7f494201c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f494201c700 time 2024-05-27T07:44:48.875776+0000 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4954861ba6] 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4954861dd1] 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56246b81723d] 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56246b802971] 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 5: (SnapRealm::check_cache() const+0x13e) [0x56246b800fa0] 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56246b801651] 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56246b796347] 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56246b49287e] 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56246b497f2f] 2024-05-27T07:44:49.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56246b4a2969] 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 11: (MDLog::_replay_thread()+0x1f38) [0x56246b83b48c] 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56246b8449cb] 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 13: (Thread::entry_wrapper()+0x43) [0x7f495483072d] 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4954830749] 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f49533d41ca] 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 16: clone() 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 0> 2024-05-27T07:44:48.879+0000 7f494201c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: in thread 7f494201c700 thread_name:md_log_replay 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f49533ded20] 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2: gsignal() 2024-05-27T07:44:49.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 3: abort() 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4954861ced] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4954861dd1] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56246b81723d] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56246b802971] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 8: (SnapRealm::check_cache() const+0x13e) [0x56246b800fa0] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56246b801651] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56246b796347] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56246b49287e] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56246b497f2f] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56246b4a2969] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 14: (MDLog::_replay_thread()+0x1f38) [0x56246b83b48c] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56246b8449cb] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 16: (Thread::entry_wrapper()+0x43) [0x7f495483072d] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4954830749] 2024-05-27T07:44:49.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f49533d41ca] 2024-05-27T07:44:49.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 19: clone() 2024-05-27T07:44:49.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:44:49.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:49.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: -1> 2024-05-27T07:44:48.877+0000 7f494201c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f494201c700 time 2024-05-27T07:44:48.875776+0000 2024-05-27T07:44:49.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:44:49.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:49.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:49.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4954861ba6] 2024-05-27T07:44:49.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4954861dd1] 2024-05-27T07:44:49.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56246b81723d] 2024-05-27T07:44:49.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56246b802971] 2024-05-27T07:44:49.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 5: (SnapRealm::check_cache() const+0x13e) [0x56246b800fa0] 2024-05-27T07:44:49.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56246b801651] 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56246b796347] 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56246b49287e] 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56246b497f2f] 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56246b4a2969] 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 11: (MDLog::_replay_thread()+0x1f38) [0x56246b83b48c] 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56246b8449cb] 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 13: (Thread::entry_wrapper()+0x43) [0x7f495483072d] 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4954830749] 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f49533d41ca] 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 16: clone() 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 0> 2024-05-27T07:44:48.879+0000 7f494201c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: in thread 7f494201c700 thread_name:md_log_replay 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:49.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f49533ded20] 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2: gsignal() 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 3: abort() 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4954861ced] 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4954861dd1] 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56246b81723d] 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56246b802971] 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 8: (SnapRealm::check_cache() const+0x13e) [0x56246b800fa0] 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56246b801651] 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56246b796347] 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56246b49287e] 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56246b497f2f] 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56246b4a2969] 2024-05-27T07:44:49.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 14: (MDLog::_replay_thread()+0x1f38) [0x56246b83b48c] 2024-05-27T07:44:49.245 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56246b8449cb] 2024-05-27T07:44:49.245 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 16: (Thread::entry_wrapper()+0x43) [0x7f495483072d] 2024-05-27T07:44:49.245 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4954830749] 2024-05-27T07:44:49.245 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f49533d41ca] 2024-05-27T07:44:49.245 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 19: clone() 2024-05-27T07:44:49.245 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:44:49.245 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72145]: 2024-05-27T07:44:49.245 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:48 smithi186 podman[72194]: 2024-05-27 07:44:48.973846637 +0000 UTC m=+0.042692998 container died 4e63ad794a713e120359b2a193bc8168f93fc63bdfa0142615c8fcf860c7aa30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-27T07:44:49.250 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.231+0000 7f2082ffd640 1 -- 172.21.15.114:0/4226438682 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f207c042030 con 0x7f208c108b90 2024-05-27T07:44:49.250 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.231+0000 7f209a974740 1 Processor -- start 2024-05-27T07:44:49.250 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.231+0000 7f209a974740 1 -- start start 2024-05-27T07:44:49.250 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.231+0000 7f209a974740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c351b0 0x565145fd6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.231+0000 7f209a974740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fd67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.231+0000 7f209a974740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c144d0 0x565145fd6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.231+0000 7f209a974740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x565145f48c30 con 0x565145c0c2f0 2024-05-27T07:44:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.231+0000 7f209a974740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x565145f387d0 con 0x565145c351b0 2024-05-27T07:44:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.231+0000 7f209a974740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x565145f3c810 con 0x565145c144d0 2024-05-27T07:44:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.231+0000 7f2091859640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fd67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.231+0000 7f2091058640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c144d0 0x565145fd6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f2091859640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fd67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41676/0 (socket says 172.21.15.114:41676) 2024-05-27T07:44:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f2091859640 1 -- 172.21.15.114:0/4016390275 learned_addr learned my addr 172.21.15.114:0/4016390275 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f209205a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c351b0 0x565145fd6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f2082ffd640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f20780b2950 0x7f20780b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f2091058640 1 -- 172.21.15.114:0/4016390275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c351b0 msgr2=0x565145fd6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f2091058640 1 --2- 172.21.15.114:0/4016390275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c351b0 0x565145fd6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f2091058640 1 -- 172.21.15.114:0/4016390275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 msgr2=0x565145fd67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f2091058640 1 --2- 172.21.15.114:0/4016390275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fd67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f2091058640 1 -- 172.21.15.114:0/4016390275 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565145c4d210 con 0x565145c144d0 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f209205a640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f20780b2950 0x7f20780b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f2082ffd640 1 -- 172.21.15.114:0/4226438682 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f207c102e00 con 0x7f208c108b90 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f2091058640 1 --2- 172.21.15.114:0/4016390275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c144d0 0x565145fd6ce0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f207c1001a0 tx=0x7f207c08f380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77e85f4411ed18e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f2077fff640 1 -- 172.21.15.114:0/4016390275 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c0c9730 con 0x565145c144d0 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f209205a640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f20780b2950 0x7f20780b4e60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x565145fd3570 tx=0x7f2084004a40 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f2077fff640 1 -- 172.21.15.114:0/4016390275 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f207c0c98f0 con 0x565145c144d0 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.232+0000 7f2077fff640 1 -- 172.21.15.114:0/4016390275 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c0f91a0 con 0x565145c144d0 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.233+0000 7f209a974740 1 -- 172.21.15.114:0/4016390275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c144d0 msgr2=0x565145fd6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.233+0000 7f209a974740 1 --2- 172.21.15.114:0/4016390275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c144d0 0x565145fd6ce0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f207c1001a0 tx=0x7f207c08f380 comp rx=0 tx=0).stop 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.233+0000 7f209a974740 1 -- 172.21.15.114:0/4016390275 shutdown_connections 2024-05-27T07:44:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.233+0000 7f209a974740 1 --2- 172.21.15.114:0/4016390275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c144d0 0x565145fd6ce0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.233+0000 7f209a974740 1 --2- 172.21.15.114:0/4016390275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fd67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.233+0000 7f209a974740 1 --2- 172.21.15.114:0/4016390275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c351b0 0x565145fd6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.233+0000 7f209a974740 1 -- 172.21.15.114:0/4016390275 >> 172.21.15.114:0/4016390275 conn(0x565145c29300 msgr2=0x565145fd2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.233+0000 7f209a974740 1 -- 172.21.15.114:0/4016390275 shutdown_connections 2024-05-27T07:44:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.233+0000 7f209a974740 1 -- 172.21.15.114:0/4016390275 wait complete. 2024-05-27T07:44:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.234+0000 7f209a974740 1 -- start start 2024-05-27T07:44:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.234+0000 7f209a974740 1 Processor -- start 2024-05-27T07:44:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.234+0000 7f209a974740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fe4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.234+0000 7f209a974740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c144d0 0x565145fe53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.234+0000 7f209a974740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c351b0 0x565145fea990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.234+0000 7f209a974740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x565145c1e790 con 0x565145c0c2f0 2024-05-27T07:44:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.234+0000 7f209a974740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x565145ef4890 con 0x565145c144d0 2024-05-27T07:44:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.234+0000 7f209a974740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x565145fe7730 con 0x565145c351b0 2024-05-27T07:44:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.234+0000 7f209a974740 10 client.0 _ll_register_callbacks cb 0x565145c4ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f2091859640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fe4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f209205a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c351b0 0x565145fea990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f209205a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c351b0 0x565145fea990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55512/0 (socket says 172.21.15.114:55512) 2024-05-27T07:44:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f2091859640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fe4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41686/0 (socket says 172.21.15.114:41686) 2024-05-27T07:44:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f2091859640 1 -- 172.21.15.114:0/3726118727 learned_addr learned my addr 172.21.15.114:0/3726118727 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f2091058640 1 --2- 172.21.15.114:0/3726118727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c144d0 0x565145fe53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f209205a640 1 -- 172.21.15.114:0/3726118727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c144d0 msgr2=0x565145fe53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f209205a640 1 --2- 172.21.15.114:0/3726118727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c144d0 0x565145fe53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f209205a640 1 -- 172.21.15.114:0/3726118727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 msgr2=0x565145fe4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f209205a640 1 --2- 172.21.15.114:0/3726118727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fe4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f209205a640 1 -- 172.21.15.114:0/3726118727 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565145d19620 con 0x565145c351b0 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f209205a640 1 --2- 172.21.15.114:0/3726118727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c351b0 0x565145fea990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2084009fd0 tx=0x7f2084004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f42615180ff44f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f20757fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f20757fa640 1 -- 172.21.15.114:0/3726118727 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208402d060 con 0x565145c351b0 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f209a974740 1 -- 172.21.15.114:0/3726118727 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f207c0c98f0 con 0x565145c351b0 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f20757fa640 1 -- 172.21.15.114:0/3726118727 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f2084028600 con 0x565145c351b0 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.235+0000 7f20757fa640 1 -- 172.21.15.114:0/3726118727 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2084034050 con 0x565145c351b0 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.236+0000 7f209a974740 10 client.26053 fetch_fsmap learned FSMap version 32 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.236+0000 7f209a974740 1 -- 172.21.15.114:0/3726118727 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x565145ff3900 con 0x565145c351b0 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.236+0000 7f20757fa640 1 -- 172.21.15.114:0/3726118727 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f207c0c98f0 con 0x565145c351b0 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.236+0000 7f20757fa640 1 -- 172.21.15.114:0/3726118727 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 32) ==== 7725+0+0 (secure 0 0 0) 0x7f208400f7a0 con 0x565145c351b0 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.236+0000 7f209a974740 10 client.26053 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.236+0000 7f209a974740 10 client.26053 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.236+0000 7f209a974740 20 client.26053 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.236+0000 7f209a974740 1 --2- 172.21.15.114:0/3726118727 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x565145bf3d80 0x565145ff6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.236+0000 7f209a974740 4 client.26053 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.236+0000 7f209a974740 1 -- 172.21.15.114:0/3726118727 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565145f4ab80 con 0x565145bf3d80 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.236+0000 7f2091859640 1 --2- 172.21.15.114:0/3726118727 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x565145bf3d80 0x565145ff6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.237+0000 7f20757fa640 10 client.26053 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.237+0000 7f2091859640 1 --2- 172.21.15.114:0/3726118727 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x565145bf3d80 0x565145ff6e20 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9eead3789a1a7a88 server_cookie=39e84ae61b7a58d4 in_seq=0 out_seq=0 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f20757fa640 1 -- 172.21.15.114:0/3726118727 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x565145f4ab80 con 0x565145bf3d80 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f20757fa640 10 client.26053 handle_command_reply: tid=0 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f209a974740 1 client.26053 shutdown 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f209a974740 1 -- 172.21.15.114:0/3726118727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c351b0 msgr2=0x565145fea990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f209a974740 1 --2- 172.21.15.114:0/3726118727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c351b0 0x565145fea990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2084009fd0 tx=0x7f2084004ab0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f209a974740 1 -- 172.21.15.114:0/3726118727 shutdown_connections 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f209a974740 1 --2- 172.21.15.114:0/3726118727 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x565145bf3d80 0x565145ff6e20 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f209a974740 1 --2- 172.21.15.114:0/3726118727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c351b0 0x565145fea990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f209a974740 1 --2- 172.21.15.114:0/3726118727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c144d0 0x565145fe53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f20757fa640 0 client.26053 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:49.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f209a974740 1 --2- 172.21.15.114:0/3726118727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fe4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f209a974740 1 -- 172.21.15.114:0/3726118727 >> 172.21.15.114:0/3726118727 conn(0x565145c29300 msgr2=0x565145fd80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:49.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f209a974740 1 -- 172.21.15.114:0/3726118727 shutdown_connections 2024-05-27T07:44:49.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f209a974740 1 -- 172.21.15.114:0/3726118727 wait complete. 2024-05-27T07:44:49.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.238+0000 7f209a974740 20 client.26053 trim_cache size 0 max 16384 2024-05-27T07:44:49.257 INFO:tasks.workunit.client.0.smithi114.stdout: 1 459008 13.48 MB/sec execute 61 sec latency 101.050 ms 2024-05-27T07:44:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209a974740 1 Processor -- start 2024-05-27T07:44:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209a974740 1 -- start start 2024-05-27T07:44:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209a974740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145bf3d80 0x565145fd6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209a974740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c0c2f0 0x565145fea820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209a974740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c144d0 0x565145fead60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209a974740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f208c1b0080 con 0x565145c144d0 2024-05-27T07:44:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209a974740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f208c1ad130 con 0x565145c0c2f0 2024-05-27T07:44:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209a974740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f208c1af2e0 con 0x565145bf3d80 2024-05-27T07:44:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209205a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c144d0 0x565145fead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209205a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c144d0 0x565145fead60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41700/0 (socket says 172.21.15.114:41700) 2024-05-27T07:44:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f2091859640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145bf3d80 0x565145fd6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209205a640 1 -- 172.21.15.114:0/4214532869 learned_addr learned my addr 172.21.15.114:0/4214532869 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f2091058640 1 --2- 172.21.15.114:0/4214532869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c0c2f0 0x565145fea820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209205a640 1 -- 172.21.15.114:0/4214532869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145bf3d80 msgr2=0x565145fd6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209205a640 1 --2- 172.21.15.114:0/4214532869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145bf3d80 0x565145fd6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209205a640 1 -- 172.21.15.114:0/4214532869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c0c2f0 msgr2=0x565145fea820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209205a640 1 --2- 172.21.15.114:0/4214532869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c0c2f0 0x565145fea820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.255+0000 7f209205a640 1 -- 172.21.15.114:0/4214532869 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565145feb2a0 con 0x565145c144d0 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.256+0000 7f209205a640 1 --2- 172.21.15.114:0/4214532869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c144d0 0x565145fead60 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f208400fb70 tx=0x7f2084005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46f21e7e67bf946 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.256+0000 7f20777fe640 1 -- 172.21.15.114:0/4214532869 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208400fea0 con 0x565145c144d0 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.256+0000 7f20777fe640 1 -- 172.21.15.114:0/4214532869 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f208403a440 con 0x565145c144d0 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.256+0000 7f20777fe640 1 -- 172.21.15.114:0/4214532869 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2084028660 con 0x565145c144d0 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.257+0000 7f209a974740 1 -- 172.21.15.114:0/4214532869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c144d0 msgr2=0x565145fead60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.257+0000 7f209a974740 1 --2- 172.21.15.114:0/4214532869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c144d0 0x565145fead60 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f208400fb70 tx=0x7f2084005f30 comp rx=0 tx=0).stop 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.257+0000 7f209a974740 1 -- 172.21.15.114:0/4214532869 shutdown_connections 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.257+0000 7f209a974740 1 --2- 172.21.15.114:0/4214532869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c144d0 0x565145fead60 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.257+0000 7f209a974740 1 --2- 172.21.15.114:0/4214532869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145c0c2f0 0x565145fea820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.257+0000 7f209a974740 1 --2- 172.21.15.114:0/4214532869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145bf3d80 0x565145fd6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.257+0000 7f209a974740 1 -- 172.21.15.114:0/4214532869 >> 172.21.15.114:0/4214532869 conn(0x565145c29300 msgr2=0x565145fd6160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.257+0000 7f209a974740 1 -- 172.21.15.114:0/4214532869 shutdown_connections 2024-05-27T07:44:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.257+0000 7f209a974740 1 -- 172.21.15.114:0/4214532869 wait complete. 2024-05-27T07:44:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.257+0000 7f209a974740 1 -- start start 2024-05-27T07:44:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.257+0000 7f209a974740 1 Processor -- start 2024-05-27T07:44:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f209a974740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145bf3d80 0x565145ff1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f209a974740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fec730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f209a974740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c144d0 0x565145fecc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f209a974740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f208c1aec10 con 0x565145c0c2f0 2024-05-27T07:44:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f209a974740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f208c1ad7a0 con 0x565145bf3d80 2024-05-27T07:44:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f209a974740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f208c1ade70 con 0x565145c144d0 2024-05-27T07:44:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f209a974740 10 client.0 _ll_register_callbacks cb 0x7f2084003bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f2091058640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fec730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f2091859640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145bf3d80 0x565145ff1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f2091058640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fec730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41702/0 (socket says 172.21.15.114:41702) 2024-05-27T07:44:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f2091058640 1 -- 172.21.15.114:0/1869476904 learned_addr learned my addr 172.21.15.114:0/1869476904 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f209205a640 1 --2- 172.21.15.114:0/1869476904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c144d0 0x565145fecc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f2091859640 1 -- 172.21.15.114:0/1869476904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c144d0 msgr2=0x565145fecc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f2091859640 1 --2- 172.21.15.114:0/1869476904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c144d0 0x565145fecc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f2091859640 1 -- 172.21.15.114:0/1869476904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 msgr2=0x565145fec730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f2091859640 1 --2- 172.21.15.114:0/1869476904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fec730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.258+0000 7f2091859640 1 -- 172.21.15.114:0/1869476904 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565145fed260 con 0x565145bf3d80 2024-05-27T07:44:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.259+0000 7f2091859640 1 --2- 172.21.15.114:0/1869476904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145bf3d80 0x565145ff1390 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x565145fe5a90 tx=0x7f20880128e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c72470abb633b0d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.259+0000 7f20757fa640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.259+0000 7f20757fa640 1 -- 172.21.15.114:0/1869476904 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2088004220 con 0x565145bf3d80 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.259+0000 7f20757fa640 1 -- 172.21.15.114:0/1869476904 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f20880043c0 con 0x565145bf3d80 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.259+0000 7f209a974740 1 -- 172.21.15.114:0/1869476904 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f208403a440 con 0x565145bf3d80 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.259+0000 7f20757fa640 1 -- 172.21.15.114:0/1869476904 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2088005020 con 0x565145bf3d80 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.259+0000 7f209a974740 10 client.25988 fetch_fsmap learned FSMap version 32 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.259+0000 7f209a974740 1 -- 172.21.15.114:0/1869476904 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56514605f280 con 0x565145bf3d80 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.259+0000 7f20757fa640 1 -- 172.21.15.114:0/1869476904 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f208403a440 con 0x565145bf3d80 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.260+0000 7f20757fa640 1 -- 172.21.15.114:0/1869476904 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 32) ==== 7725+0+0 (secure 0 0 0) 0x7f208801d6c0 con 0x565145bf3d80 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.260+0000 7f209a974740 10 client.25988 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.260+0000 7f209a974740 10 client.25988 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.260+0000 7f209a974740 20 client.25988 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.260+0000 7f209a974740 1 --2- 172.21.15.114:0/1869476904 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x565145c351b0 0x565146061a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.260+0000 7f209a974740 4 client.25988 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:44:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.260+0000 7f209a974740 1 -- 172.21.15.114:0/1869476904 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565145f44360 con 0x565145c351b0 2024-05-27T07:44:49.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.260+0000 7f2091058640 1 --2- 172.21.15.114:0/1869476904 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x565145c351b0 0x565146061a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:49.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.260+0000 7f20757fa640 10 client.25988 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:49.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.260+0000 7f2091058640 1 --2- 172.21.15.114:0/1869476904 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x565145c351b0 0x565146061a40 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e5f3188fddfda98 server_cookie=2f78d8b58fd438f5 in_seq=0 out_seq=0 2024-05-27T07:44:49.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:49 smithi114 ceph-mon[32038]: pgmap v160: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 467 op/s 2024-05-27T07:44:49.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:49 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1534067918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:44:49.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.579+0000 7f20757fa640 1 -- 172.21.15.114:0/1869476904 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x565145f44360 con 0x565145c351b0 2024-05-27T07:44:49.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f20757fa640 10 client.25988 handle_command_reply: tid=0 2024-05-27T07:44:49.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f209a974740 1 client.25988 shutdown 2024-05-27T07:44:49.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f209a974740 1 -- 172.21.15.114:0/1869476904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145bf3d80 msgr2=0x565145ff1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:49.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f209a974740 1 --2- 172.21.15.114:0/1869476904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145bf3d80 0x565145ff1390 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x565145fe5a90 tx=0x7f20880128e0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f209a974740 1 -- 172.21.15.114:0/1869476904 shutdown_connections 2024-05-27T07:44:49.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f209a974740 1 --2- 172.21.15.114:0/1869476904 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x565145c351b0 0x565146061a40 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f20757fa640 0 client.25988 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:49.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f209a974740 1 --2- 172.21.15.114:0/1869476904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565145c144d0 0x565145fecc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f209a974740 1 --2- 172.21.15.114:0/1869476904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565145c0c2f0 0x565145fec730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f209a974740 1 --2- 172.21.15.114:0/1869476904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565145bf3d80 0x565145ff1390 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f209a974740 1 -- 172.21.15.114:0/1869476904 >> 172.21.15.114:0/1869476904 conn(0x565145c29300 msgr2=0x565145fd8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:49.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f209a974740 1 -- 172.21.15.114:0/1869476904 shutdown_connections 2024-05-27T07:44:49.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f209a974740 1 -- 172.21.15.114:0/1869476904 wait complete. 2024-05-27T07:44:49.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.580+0000 7f209a974740 20 client.25988 trim_cache size 0 max 16384 2024-05-27T07:44:49.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:44:49.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:44:49.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:44:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.581+0000 7f2093abd640 1 -- 172.21.15.114:0/4226438682 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f20780b2950 msgr2=0x7f20780b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.581+0000 7f2093abd640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f20780b2950 0x7f20780b4e60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x565145fd3570 tx=0x7f2084004a40 comp rx=0 tx=0).stop 2024-05-27T07:44:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.581+0000 7f2093abd640 1 -- 172.21.15.114:0/4226438682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c108b90 msgr2=0x7f208c19bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.581+0000 7f2093abd640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c108b90 0x7f208c19bca0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f207c006010 tx=0x7f207c004060 comp rx=0 tx=0).stop 2024-05-27T07:44:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.581+0000 7f2093abd640 1 -- 172.21.15.114:0/4226438682 shutdown_connections 2024-05-27T07:44:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.581+0000 7f2093abd640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f20780b2950 0x7f20780b4e60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.581+0000 7f2093abd640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f208c109030 0x7f208c19c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.581+0000 7f2093abd640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c108b90 0x7f208c19bca0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.581+0000 7f2093abd640 1 --2- 172.21.15.114:0/4226438682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f208c1032e0 0x7f208c19b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:49.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.581+0000 7f2093abd640 1 -- 172.21.15.114:0/4226438682 >> 172.21.15.114:0/4226438682 conn(0x7f208c0782d0 msgr2=0x7f208c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:49.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.581+0000 7f2093abd640 1 -- 172.21.15.114:0/4226438682 shutdown_connections 2024-05-27T07:44:49.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:44:49.581+0000 7f2093abd640 1 -- 172.21.15.114:0/4226438682 wait complete. 2024-05-27T07:44:49.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:44:49.716 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:49 smithi186 ceph-mon[36647]: pgmap v160: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 467 op/s 2024-05-27T07:44:49.717 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:49 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1534067918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:44:49.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:49 smithi163 ceph-mon[36451]: pgmap v160: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 467 op/s 2024-05-27T07:44:49.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:49 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1534067918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:44:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:44:49.977 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:49 smithi186 podman[72194]: 2024-05-27 07:44:49.718452968 +0000 UTC m=+0.787299310 container remove 4e63ad794a713e120359b2a193bc8168f93fc63bdfa0142615c8fcf860c7aa30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-27T07:44:49.977 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:49 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:44:50.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 461266 13.46 MB/sec execute 62 sec latency 127.411 ms 2024-05-27T07:44:50.477 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:50 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:44:50.478 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:44:50 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.723s CPU time. 2024-05-27T07:44:51.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 463881 13.44 MB/sec execute 63 sec latency 23.466 ms 2024-05-27T07:44:51.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:51 smithi114 ceph-mon[32038]: pgmap v161: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 511 op/s 2024-05-27T07:44:51.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:51 smithi163 ceph-mon[36451]: pgmap v161: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 511 op/s 2024-05-27T07:44:51.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:51 smithi186 ceph-mon[36647]: pgmap v161: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 511 op/s 2024-05-27T07:44:51.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.831+0000 7ff7f7905700 1 -- 172.21.15.114:0/3130090218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f01092f0 msgr2=0x7ff7f0109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:51.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.831+0000 7ff7f7905700 1 --2- 172.21.15.114:0/3130090218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f01092f0 0x7ff7f0109700 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009990 tx=0x7ff7ec009ca0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.831+0000 7ff7f7905700 1 -- 172.21.15.114:0/3130090218 shutdown_connections 2024-05-27T07:44:51.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.831+0000 7ff7f7905700 1 --2- 172.21.15.114:0/3130090218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7f01030d0 0x7ff7f0111160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.831+0000 7ff7f7905700 1 --2- 172.21.15.114:0/3130090218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f01092f0 0x7ff7f0109700 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.831+0000 7ff7f7905700 1 --2- 172.21.15.114:0/3130090218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7f0102780 0x7ff7f0102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.831+0000 7ff7f7905700 1 -- 172.21.15.114:0/3130090218 >> 172.21.15.114:0/3130090218 conn(0x7ff7f00fe530 msgr2=0x7ff7f0100980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:51.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.831+0000 7ff7f7905700 1 -- 172.21.15.114:0/3130090218 shutdown_connections 2024-05-27T07:44:51.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.831+0000 7ff7f7905700 1 -- 172.21.15.114:0/3130090218 wait complete. 2024-05-27T07:44:51.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f7905700 1 Processor -- start 2024-05-27T07:44:51.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f7905700 1 -- start start 2024-05-27T07:44:51.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f7905700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f0102780 0x7ff7f019af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f7905700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7f01030d0 0x7ff7f019b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f7905700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7f01092f0 0x7ff7f019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f7905700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f01145d0 con 0x7ff7f0102780 2024-05-27T07:44:51.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f7905700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f0114450 con 0x7ff7f01030d0 2024-05-27T07:44:51.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f7905700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f0114750 con 0x7ff7f01092f0 2024-05-27T07:44:51.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f56a1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f0102780 0x7ff7f019af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f56a1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f0102780 0x7ff7f019af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41724/0 (socket says 172.21.15.114:41724) 2024-05-27T07:44:51.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f56a1700 1 -- 172.21.15.114:0/1345547154 learned_addr learned my addr 172.21.15.114:0/1345547154 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:51.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7f01030d0 0x7ff7f019b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f5ea2700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7f01092f0 0x7ff7f019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f56a1700 1 -- 172.21.15.114:0/1345547154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7f01092f0 msgr2=0x7ff7f019f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:51.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f56a1700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7f01092f0 0x7ff7f019f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f56a1700 1 -- 172.21.15.114:0/1345547154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7f01030d0 msgr2=0x7ff7f019b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:51.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f56a1700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7f01030d0 0x7ff7f019b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f56a1700 1 -- 172.21.15.114:0/1345547154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f019fde0 con 0x7ff7f0102780 2024-05-27T07:44:51.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.832+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7f01030d0 0x7ff7f019b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:51.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.833+0000 7ff7f56a1700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f0102780 0x7ff7f019af60 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec000c00 tx=0x7ff7ec0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:51.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.833+0000 7ff7e67fc700 1 -- 172.21.15.114:0/1345547154 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec026070 con 0x7ff7f0102780 2024-05-27T07:44:51.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.833+0000 7ff7e67fc700 1 -- 172.21.15.114:0/1345547154 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7ec021410 con 0x7ff7f0102780 2024-05-27T07:44:51.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.833+0000 7ff7f7905700 1 -- 172.21.15.114:0/1345547154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f01a0070 con 0x7ff7f0102780 2024-05-27T07:44:51.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.833+0000 7ff7e67fc700 1 -- 172.21.15.114:0/1345547154 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec017650 con 0x7ff7f0102780 2024-05-27T07:44:51.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.833+0000 7ff7f7905700 1 -- 172.21.15.114:0/1345547154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f01a79f0 con 0x7ff7f0102780 2024-05-27T07:44:51.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.835+0000 7ff7e67fc700 1 -- 172.21.15.114:0/1345547154 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ff7ec029070 con 0x7ff7f0102780 2024-05-27T07:44:51.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.836+0000 7ff7e67fc700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff7dc0b2a70 0x7ff7dc0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.836+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff7dc0b2a70 0x7ff7dc0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.836+0000 7ff7e67fc700 1 -- 172.21.15.114:0/1345547154 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7ff7ec0e34c0 con 0x7ff7f0102780 2024-05-27T07:44:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.837+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff7dc0b2a70 0x7ff7dc0b4f80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0108930 tx=0x7ff7e0009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.837+0000 7ff80631ab80 1 Processor -- start 2024-05-27T07:44:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.837+0000 7ff80631ab80 1 -- start start 2024-05-27T07:44:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff80631ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a374afe40 0x563a37637ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff80631ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a376380e0 0x563a37631e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff80631ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a376327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff80631ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563a3755cc90 con 0x563a374afe40 2024-05-27T07:44:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff80631ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563a3754c830 con 0x563a376380e0 2024-05-27T07:44:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff80631ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563a37550870 con 0x563a37632390 2024-05-27T07:44:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff7f56a1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a374afe40 0x563a37637ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff7f56a1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a374afe40 0x563a37637ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41726/0 (socket says 172.21.15.114:41726) 2024-05-27T07:44:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff7f56a1700 1 -- 172.21.15.114:0/3858551972 learned_addr learned my addr 172.21.15.114:0/3858551972 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/3858551972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a376380e0 0x563a37631e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff7f5ea2700 1 --2- 172.21.15.114:0/3858551972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a376327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff7f4ea0700 1 -- 172.21.15.114:0/3858551972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 msgr2=0x563a376327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/3858551972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a376327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff7f4ea0700 1 -- 172.21.15.114:0/3858551972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a374afe40 msgr2=0x563a37637ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:51.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/3858551972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a374afe40 0x563a37637ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff7f4ea0700 1 -- 172.21.15.114:0/3858551972 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a3749c8d0 con 0x563a376380e0 2024-05-27T07:44:51.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff7f56a1700 1 --2- 172.21.15.114:0/3858551972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a374afe40 0x563a37637ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:51.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff7f5ea2700 1 --2- 172.21.15.114:0/3858551972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a376327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:51.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.838+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/3858551972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a376380e0 0x563a37631e50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff7e00042f0 tx=0x7ff7e0014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bd38ddf2cc27a6e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:51.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.839+0000 7ff7db7fe700 1 -- 172.21.15.114:0/3858551972 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e0003d10 con 0x563a376380e0 2024-05-27T07:44:51.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.839+0000 7ff7db7fe700 1 -- 172.21.15.114:0/3858551972 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7e00268b0 con 0x563a376380e0 2024-05-27T07:44:51.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.839+0000 7ff7db7fe700 1 -- 172.21.15.114:0/3858551972 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e001e770 con 0x563a376380e0 2024-05-27T07:44:51.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.840+0000 7ff80631ab80 1 -- 172.21.15.114:0/3858551972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a376380e0 msgr2=0x563a37631e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:51.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.840+0000 7ff80631ab80 1 --2- 172.21.15.114:0/3858551972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a376380e0 0x563a37631e50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff7e00042f0 tx=0x7ff7e0014000 comp rx=0 tx=0).stop 2024-05-27T07:44:51.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.840+0000 7ff80631ab80 1 -- 172.21.15.114:0/3858551972 shutdown_connections 2024-05-27T07:44:51.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.840+0000 7ff80631ab80 1 --2- 172.21.15.114:0/3858551972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a376327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.840+0000 7ff80631ab80 1 --2- 172.21.15.114:0/3858551972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a374afe40 0x563a37637ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.840+0000 7ff80631ab80 1 --2- 172.21.15.114:0/3858551972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a376380e0 0x563a37631e50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.840+0000 7ff80631ab80 1 -- 172.21.15.114:0/3858551972 >> 172.21.15.114:0/3858551972 conn(0x563a3750aaf0 msgr2=0x563a37628640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.840+0000 7ff80631ab80 1 -- 172.21.15.114:0/3858551972 shutdown_connections 2024-05-27T07:44:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.840+0000 7ff80631ab80 1 -- 172.21.15.114:0/3858551972 wait complete. 2024-05-27T07:44:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.840+0000 7ff80631ab80 1 -- start start 2024-05-27T07:44:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.840+0000 7ff80631ab80 1 Processor -- start 2024-05-27T07:44:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff80631ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a374afe40 0x563a3763b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff80631ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a3763bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff80631ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a376380e0 0x563a3763c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff80631ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563a37452fa0 con 0x563a376380e0 2024-05-27T07:44:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff80631ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563a374ee7c0 con 0x563a374afe40 2024-05-27T07:44:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff80631ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563a3763eb10 con 0x563a37632390 2024-05-27T07:44:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff7f5ea2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a376380e0 0x563a3763c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff7f4ea0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a3763bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff80631ab80 10 client.0 _ll_register_callbacks cb 0x563a374e7920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff7f56a1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a374afe40 0x563a3763b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff7f4ea0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a3763bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55586/0 (socket says 172.21.15.114:55586) 2024-05-27T07:44:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff7f5ea2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a376380e0 0x563a3763c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41734/0 (socket says 172.21.15.114:41734) 2024-05-27T07:44:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff7f4ea0700 1 -- 172.21.15.114:0/1450111957 learned_addr learned my addr 172.21.15.114:0/1450111957 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff7f56a1700 1 -- 172.21.15.114:0/1450111957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 msgr2=0x563a3763bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff7f56a1700 1 --2- 172.21.15.114:0/1450111957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a3763bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff7f56a1700 1 -- 172.21.15.114:0/1450111957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a376380e0 msgr2=0x563a3763c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff7f56a1700 1 --2- 172.21.15.114:0/1450111957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a376380e0 0x563a3763c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.841+0000 7ff7f56a1700 1 -- 172.21.15.114:0/1450111957 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a374eb690 con 0x563a374afe40 2024-05-27T07:44:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff7f56a1700 1 --2- 172.21.15.114:0/1450111957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a374afe40 0x563a3763b7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0aa410 tx=0x7ff7ec0a9ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ed0d530c5771b05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff7d8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:44:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff7d8ff9700 1 -- 172.21.15.114:0/1450111957 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec0aa480 con 0x563a374afe40 2024-05-27T07:44:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff7d8ff9700 1 -- 172.21.15.114:0/1450111957 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7ff7ec0e3170 con 0x563a374afe40 2024-05-27T07:44:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff80631ab80 1 -- 172.21.15.114:0/1450111957 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7e00268b0 con 0x563a374afe40 2024-05-27T07:44:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff7d8ff9700 1 -- 172.21.15.114:0/1450111957 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff7ec0ae830 con 0x563a374afe40 2024-05-27T07:44:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff80631ab80 10 client.26000 fetch_fsmap learned FSMap version 32 2024-05-27T07:44:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff80631ab80 1 -- 172.21.15.114:0/1450111957 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563a37649590 con 0x563a374afe40 2024-05-27T07:44:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff7d8ff9700 1 -- 172.21.15.114:0/1450111957 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7e00268b0 con 0x563a374afe40 2024-05-27T07:44:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff7d8ff9700 1 -- 172.21.15.114:0/1450111957 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 32) ==== 7725+0+0 (secure 0 0 0) 0x7ff7ec0c4ce0 con 0x563a374afe40 2024-05-27T07:44:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff80631ab80 10 client.26000 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T07:44:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff80631ab80 10 client.26000 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff80631ab80 20 client.26000 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff80631ab80 1 --2- 172.21.15.114:0/1450111957 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a3764bc40 0x563a3764e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff80631ab80 4 client.26000 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:44:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.842+0000 7ff80631ab80 1 -- 172.21.15.114:0/1450111957 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a375f99f0 con 0x563a3764bc40 2024-05-27T07:44:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.843+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/1450111957 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a3764bc40 0x563a3764e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.843+0000 7ff7d8ff9700 10 client.26000 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.843+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/1450111957 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a3764bc40 0x563a3764e0b0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c067de46c4db68b server_cookie=ed5e9215021abeb0 in_seq=0 out_seq=0 2024-05-27T07:44:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.844+0000 7ff7d8ff9700 1 -- 172.21.15.114:0/1450111957 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563a375f99f0 con 0x563a3764bc40 2024-05-27T07:44:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.844+0000 7ff7d8ff9700 10 client.26000 handle_command_reply: tid=0 2024-05-27T07:44:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.844+0000 7ff80631ab80 1 client.26000 shutdown 2024-05-27T07:44:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.845+0000 7ff80631ab80 1 -- 172.21.15.114:0/1450111957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a374afe40 msgr2=0x563a3763b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.845+0000 7ff80631ab80 1 --2- 172.21.15.114:0/1450111957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a374afe40 0x563a3763b7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0aa410 tx=0x7ff7ec0a9ce0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.845+0000 7ff80631ab80 1 -- 172.21.15.114:0/1450111957 shutdown_connections 2024-05-27T07:44:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.845+0000 7ff80631ab80 1 --2- 172.21.15.114:0/1450111957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a376380e0 0x563a3763c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.845+0000 7ff80631ab80 1 --2- 172.21.15.114:0/1450111957 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a3764bc40 0x563a3764e0b0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.845+0000 7ff80631ab80 1 --2- 172.21.15.114:0/1450111957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a374afe40 0x563a3763b7a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.845+0000 7ff80631ab80 1 --2- 172.21.15.114:0/1450111957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a3763bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.845+0000 7ff7d8ff9700 0 client.26000 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.845+0000 7ff80631ab80 1 -- 172.21.15.114:0/1450111957 >> 172.21.15.114:0/1450111957 conn(0x563a3750aaf0 msgr2=0x563a3762ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.845+0000 7ff80631ab80 1 -- 172.21.15.114:0/1450111957 shutdown_connections 2024-05-27T07:44:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.845+0000 7ff80631ab80 1 -- 172.21.15.114:0/1450111957 wait complete. 2024-05-27T07:44:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.845+0000 7ff80631ab80 20 client.26000 trim_cache size 0 max 16384 2024-05-27T07:44:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff80631ab80 1 Processor -- start 2024-05-27T07:44:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff80631ab80 1 -- start start 2024-05-27T07:44:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff80631ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a374afe40 0x563a37636f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff80631ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a3763c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff80631ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a376380e0 0x563a37622e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff80631ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f01af1d0 con 0x563a374afe40 2024-05-27T07:44:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff80631ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f01ac280 con 0x563a376380e0 2024-05-27T07:44:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff80631ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f01ae430 con 0x563a37632390 2024-05-27T07:44:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff7f4ea0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a3763c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff7f5ea2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a376380e0 0x563a37622e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff7f4ea0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a3763c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55588/0 (socket says 172.21.15.114:55588) 2024-05-27T07:44:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff7f4ea0700 1 -- 172.21.15.114:0/1952880103 learned_addr learned my addr 172.21.15.114:0/1952880103 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff7f56a1700 1 --2- 172.21.15.114:0/1952880103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a374afe40 0x563a37636f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff7f5ea2700 1 -- 172.21.15.114:0/1952880103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 msgr2=0x563a3763c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff7f5ea2700 1 --2- 172.21.15.114:0/1952880103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a3763c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff7f5ea2700 1 -- 172.21.15.114:0/1952880103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a374afe40 msgr2=0x563a37636f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff7f5ea2700 1 --2- 172.21.15.114:0/1952880103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a374afe40 0x563a37636f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.861+0000 7ff7f5ea2700 1 -- 172.21.15.114:0/1952880103 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a37639d10 con 0x563a376380e0 2024-05-27T07:44:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.862+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/1952880103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a3763c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.862+0000 7ff7f5ea2700 1 --2- 172.21.15.114:0/1952880103 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a376380e0 0x563a37622e00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8007660 tx=0x7ff7e8009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c53864d5dde9c100 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.862+0000 7ff7daffd700 1 -- 172.21.15.114:0/1952880103 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8004bb0 con 0x563a376380e0 2024-05-27T07:44:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.862+0000 7ff7daffd700 1 -- 172.21.15.114:0/1952880103 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7e8004d50 con 0x563a376380e0 2024-05-27T07:44:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.862+0000 7ff7daffd700 1 -- 172.21.15.114:0/1952880103 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8016e30 con 0x563a376380e0 2024-05-27T07:44:51.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.863+0000 7ff80631ab80 1 -- 172.21.15.114:0/1952880103 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a376380e0 msgr2=0x563a37622e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:51.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.863+0000 7ff80631ab80 1 --2- 172.21.15.114:0/1952880103 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a376380e0 0x563a37622e00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8007660 tx=0x7ff7e8009920 comp rx=0 tx=0).stop 2024-05-27T07:44:51.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.863+0000 7ff80631ab80 1 -- 172.21.15.114:0/1952880103 shutdown_connections 2024-05-27T07:44:51.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.863+0000 7ff80631ab80 1 --2- 172.21.15.114:0/1952880103 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a376380e0 0x563a37622e00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.863+0000 7ff80631ab80 1 --2- 172.21.15.114:0/1952880103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a374afe40 0x563a37636f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.863+0000 7ff80631ab80 1 --2- 172.21.15.114:0/1952880103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a37632390 0x563a3763c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.863+0000 7ff80631ab80 1 -- 172.21.15.114:0/1952880103 >> 172.21.15.114:0/1952880103 conn(0x563a3750aaf0 msgr2=0x563a376226f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:51.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.863+0000 7ff80631ab80 1 -- 172.21.15.114:0/1952880103 shutdown_connections 2024-05-27T07:44:51.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.863+0000 7ff80631ab80 1 -- 172.21.15.114:0/1952880103 wait complete. 2024-05-27T07:44:51.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.863+0000 7ff80631ab80 1 -- start start 2024-05-27T07:44:51.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.863+0000 7ff80631ab80 1 Processor -- start 2024-05-27T07:44:51.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.864+0000 7ff80631ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a374afe40 0x563a37644270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.864+0000 7ff80631ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a37632390 0x563a37652e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.864+0000 7ff80631ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a376380e0 0x563a376533c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.864+0000 7ff80631ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f01add60 con 0x563a376380e0 2024-05-27T07:44:51.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.864+0000 7ff80631ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f01ac8f0 con 0x563a37632390 2024-05-27T07:44:51.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.864+0000 7ff80631ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f01acfc0 con 0x563a374afe40 2024-05-27T07:44:51.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.864+0000 7ff7f5ea2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a376380e0 0x563a376533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.864+0000 7ff80631ab80 10 client.0 _ll_register_callbacks cb 0x563a374e7920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:51.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.864+0000 7ff7f5ea2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a376380e0 0x563a376533c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41748/0 (socket says 172.21.15.114:41748) 2024-05-27T07:44:51.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.864+0000 7ff7f56a1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a374afe40 0x563a37644270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.864+0000 7ff7f5ea2700 1 -- 172.21.15.114:0/2413612988 learned_addr learned my addr 172.21.15.114:0/2413612988 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:51.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.864+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/2413612988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a37632390 0x563a37652e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff7f4ea0700 1 -- 172.21.15.114:0/2413612988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a374afe40 msgr2=0x563a37644270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:51.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/2413612988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a374afe40 0x563a37644270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff7f4ea0700 1 -- 172.21.15.114:0/2413612988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a376380e0 msgr2=0x563a376533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:51.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/2413612988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a376380e0 0x563a376533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff7f4ea0700 1 -- 172.21.15.114:0/2413612988 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a37653900 con 0x563a37632390 2024-05-27T07:44:51.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff7f5ea2700 1 --2- 172.21.15.114:0/2413612988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a376380e0 0x563a376533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:51.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff7f4ea0700 1 --2- 172.21.15.114:0/2413612988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a37632390 0x563a37652e80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0005830 tx=0x7ff7e0026400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=761ef7b4fdb6faba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:51.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff7d8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:44:51.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff7d8ff9700 1 -- 172.21.15.114:0/2413612988 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e0038020 con 0x563a37632390 2024-05-27T07:44:51.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff7d8ff9700 1 -- 172.21.15.114:0/2413612988 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7ff7e0005040 con 0x563a37632390 2024-05-27T07:44:51.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff80631ab80 1 -- 172.21.15.114:0/2413612988 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7e8004d50 con 0x563a37632390 2024-05-27T07:44:51.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff7d8ff9700 1 -- 172.21.15.114:0/2413612988 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff7e0043050 con 0x563a37632390 2024-05-27T07:44:51.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff80631ab80 10 client.26006 fetch_fsmap learned FSMap version 32 2024-05-27T07:44:51.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.865+0000 7ff80631ab80 1 -- 172.21.15.114:0/2413612988 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563a376b63c0 con 0x563a37632390 2024-05-27T07:44:51.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.866+0000 7ff7d8ff9700 1 -- 172.21.15.114:0/2413612988 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7e8004d50 con 0x563a37632390 2024-05-27T07:44:51.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.866+0000 7ff7d8ff9700 1 -- 172.21.15.114:0/2413612988 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 32) ==== 7725+0+0 (secure 0 0 0) 0x7ff7e00046c0 con 0x563a37632390 2024-05-27T07:44:51.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.866+0000 7ff80631ab80 10 client.26006 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T07:44:51.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.866+0000 7ff80631ab80 10 client.26006 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:51.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.866+0000 7ff80631ab80 20 client.26006 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:51.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.866+0000 7ff80631ab80 1 --2- 172.21.15.114:0/2413612988 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a376b6990 0x563a376b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:51.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.866+0000 7ff80631ab80 4 client.26006 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:44:51.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.866+0000 7ff80631ab80 1 -- 172.21.15.114:0/2413612988 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a3755ebe0 con 0x563a376b6990 2024-05-27T07:44:51.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.867+0000 7ff7f56a1700 1 --2- 172.21.15.114:0/2413612988 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a376b6990 0x563a376b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:51.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.867+0000 7ff7d8ff9700 10 client.26006 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:51.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.867+0000 7ff7f56a1700 1 --2- 172.21.15.114:0/2413612988 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a376b6990 0x563a376b8db0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29830fe99107c3e0 server_cookie=13a0b1c9676dd688 in_seq=0 out_seq=0 2024-05-27T07:44:51.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff7d8ff9700 1 -- 172.21.15.114:0/2413612988 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a3755ebe0 con 0x563a376b6990 2024-05-27T07:44:51.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff7d8ff9700 10 client.26006 handle_command_reply: tid=0 2024-05-27T07:44:51.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff80631ab80 1 client.26006 shutdown 2024-05-27T07:44:51.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff80631ab80 1 -- 172.21.15.114:0/2413612988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a37632390 msgr2=0x563a37652e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:51.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff80631ab80 1 --2- 172.21.15.114:0/2413612988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a37632390 0x563a37652e80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0005830 tx=0x7ff7e0026400 comp rx=0 tx=0).stop 2024-05-27T07:44:51.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff80631ab80 1 -- 172.21.15.114:0/2413612988 shutdown_connections 2024-05-27T07:44:51.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff80631ab80 1 --2- 172.21.15.114:0/2413612988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a376380e0 0x563a376533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff80631ab80 1 --2- 172.21.15.114:0/2413612988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a374afe40 0x563a37644270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff80631ab80 1 --2- 172.21.15.114:0/2413612988 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a376b6990 0x563a376b8db0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff80631ab80 1 --2- 172.21.15.114:0/2413612988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a37632390 0x563a37652e80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff7d8ff9700 0 client.26006 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:51.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff80631ab80 1 -- 172.21.15.114:0/2413612988 >> 172.21.15.114:0/2413612988 conn(0x563a3750aaf0 msgr2=0x563a3762ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:51.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff80631ab80 1 -- 172.21.15.114:0/2413612988 shutdown_connections 2024-05-27T07:44:51.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff80631ab80 1 -- 172.21.15.114:0/2413612988 wait complete. 2024-05-27T07:44:51.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.869+0000 7ff80631ab80 20 client.26006 trim_cache size 0 max 16384 2024-05-27T07:44:51.872 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:44:51.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.870+0000 7ff7f7905700 1 -- 172.21.15.114:0/1345547154 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff7dc0b2a70 msgr2=0x7ff7dc0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:51.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.870+0000 7ff7f7905700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff7dc0b2a70 0x7ff7dc0b4f80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0108930 tx=0x7ff7e0009260 comp rx=0 tx=0).stop 2024-05-27T07:44:51.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.870+0000 7ff7f7905700 1 -- 172.21.15.114:0/1345547154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f0102780 msgr2=0x7ff7f019af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:51.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.870+0000 7ff7f7905700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f0102780 0x7ff7f019af60 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec000c00 tx=0x7ff7ec0047b0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.870+0000 7ff7f7905700 1 -- 172.21.15.114:0/1345547154 shutdown_connections 2024-05-27T07:44:51.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.870+0000 7ff7f7905700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7f01092f0 0x7ff7f019f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.870+0000 7ff7f7905700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f0102780 0x7ff7f019af60 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.870+0000 7ff7f7905700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff7dc0b2a70 0x7ff7dc0b4f80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.870+0000 7ff7f7905700 1 --2- 172.21.15.114:0/1345547154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7f01030d0 0x7ff7f019b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:51.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.870+0000 7ff7f7905700 1 -- 172.21.15.114:0/1345547154 >> 172.21.15.114:0/1345547154 conn(0x7ff7f00fe530 msgr2=0x7ff7f01071f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:51.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.870+0000 7ff7f7905700 1 -- 172.21.15.114:0/1345547154 shutdown_connections 2024-05-27T07:44:51.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:51.870+0000 7ff7f7905700 1 -- 172.21.15.114:0/1345547154 wait complete. 2024-05-27T07:44:52.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 466505 13.47 MB/sec execute 64 sec latency 15.119 ms 2024-05-27T07:44:53.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 469263 13.53 MB/sec execute 65 sec latency 53.246 ms 2024-05-27T07:44:53.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:53 smithi114 ceph-mon[32038]: pgmap v162: 129 pgs: 129 active+clean; 43 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 487 op/s 2024-05-27T07:44:53.686 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:44:53.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:44:53.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:53 smithi163 ceph-mon[36451]: pgmap v162: 129 pgs: 129 active+clean; 43 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 487 op/s 2024-05-27T07:44:53.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:53 smithi186 ceph-mon[36647]: pgmap v162: 129 pgs: 129 active+clean; 43 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 487 op/s 2024-05-27T07:44:54.010 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:44:54.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 471603 13.45 MB/sec execute 66 sec latency 30.319 ms 2024-05-27T07:44:55.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 474069 13.49 MB/sec execute 67 sec latency 41.084 ms 2024-05-27T07:44:55.518 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:55 smithi114 ceph-mon[32038]: pgmap v163: 129 pgs: 129 active+clean; 43 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 487 op/s 2024-05-27T07:44:55.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:55 smithi163 ceph-mon[36451]: pgmap v163: 129 pgs: 129 active+clean; 43 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 487 op/s 2024-05-27T07:44:55.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:55 smithi186 ceph-mon[36647]: pgmap v163: 129 pgs: 129 active+clean; 43 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 487 op/s 2024-05-27T07:44:56.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 476855 13.50 MB/sec execute 68 sec latency 22.026 ms 2024-05-27T07:44:56.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.076+0000 7f64db7c2700 1 -- 172.21.15.114:0/1529066407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64d4102780 msgr2=0x7f64d4102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:56.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.076+0000 7f64db7c2700 1 --2- 172.21.15.114:0/1529066407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64d4102780 0x7f64d4102b90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009990 tx=0x7f64c4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.076+0000 7f64db7c2700 1 -- 172.21.15.114:0/1529066407 shutdown_connections 2024-05-27T07:44:56.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.076+0000 7f64db7c2700 1 --2- 172.21.15.114:0/1529066407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d41030d0 0x7f64d4111160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.076+0000 7f64db7c2700 1 --2- 172.21.15.114:0/1529066407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64d41092f0 0x7f64d4109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.076+0000 7f64db7c2700 1 --2- 172.21.15.114:0/1529066407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64d4102780 0x7f64d4102b90 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.076+0000 7f64db7c2700 1 -- 172.21.15.114:0/1529066407 >> 172.21.15.114:0/1529066407 conn(0x7f64d40fe510 msgr2=0x7f64d4100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:56.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.076+0000 7f64db7c2700 1 -- 172.21.15.114:0/1529066407 shutdown_connections 2024-05-27T07:44:56.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.076+0000 7f64db7c2700 1 -- 172.21.15.114:0/1529066407 wait complete. 2024-05-27T07:44:56.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64db7c2700 1 Processor -- start 2024-05-27T07:44:56.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64db7c2700 1 -- start start 2024-05-27T07:44:56.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64db7c2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64d4102780 0x7f64d419aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64db7c2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64d41030d0 0x7f64d419b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64db7c2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d41092f0 0x7f64d419f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64db7c2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f64d4114590 con 0x7f64d4102780 2024-05-27T07:44:56.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64db7c2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f64d4114410 con 0x7f64d41030d0 2024-05-27T07:44:56.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64db7c2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64d4114710 con 0x7f64d41092f0 2024-05-27T07:44:56.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64d955e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64d4102780 0x7f64d419aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64d955e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64d4102780 0x7f64d419aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60612/0 (socket says 172.21.15.114:60612) 2024-05-27T07:44:56.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64d955e700 1 -- 172.21.15.114:0/1208568563 learned_addr learned my addr 172.21.15.114:0/1208568563 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:56.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64d9d5f700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d41092f0 0x7f64d419f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64d41030d0 0x7f64d419b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64d955e700 1 -- 172.21.15.114:0/1208568563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d41092f0 msgr2=0x7f64d419f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:56.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64d955e700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d41092f0 0x7f64d419f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64d955e700 1 -- 172.21.15.114:0/1208568563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64d41030d0 msgr2=0x7f64d419b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:56.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64d955e700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64d41030d0 0x7f64d419b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.077+0000 7f64d955e700 1 -- 172.21.15.114:0/1208568563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d419fd70 con 0x7f64d4102780 2024-05-27T07:44:56.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.078+0000 7f64d955e700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64d4102780 0x7f64d419aea0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f64d000e940 tx=0x7f64d000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:56.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.078+0000 7f64ca7fc700 1 -- 172.21.15.114:0/1208568563 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d0004220 con 0x7f64d4102780 2024-05-27T07:44:56.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.078+0000 7f64db7c2700 1 -- 172.21.15.114:0/1208568563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d41a0060 con 0x7f64d4102780 2024-05-27T07:44:56.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.078+0000 7f64ca7fc700 1 -- 172.21.15.114:0/1208568563 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64d00043c0 con 0x7f64d4102780 2024-05-27T07:44:56.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.078+0000 7f64ca7fc700 1 -- 172.21.15.114:0/1208568563 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d0003c60 con 0x7f64d4102780 2024-05-27T07:44:56.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.078+0000 7f64db7c2700 1 -- 172.21.15.114:0/1208568563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d41a7980 con 0x7f64d4102780 2024-05-27T07:44:56.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.081+0000 7f64ca7fc700 1 -- 172.21.15.114:0/1208568563 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f64d000c7e0 con 0x7f64d4102780 2024-05-27T07:44:56.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.081+0000 7f64ca7fc700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64c00b29f0 0x7f64c00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.081+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64c00b29f0 0x7f64c00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.081+0000 7f64ca7fc700 1 -- 172.21.15.114:0/1208568563 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f64d00e2ba0 con 0x7f64d4102780 2024-05-27T07:44:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64ea1d7b80 1 Processor -- start 2024-05-27T07:44:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64ea1d7b80 1 -- start start 2024-05-27T07:44:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64c00b29f0 0x7f64c00b4f00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f64c4006010 tx=0x7f64c4005960 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64ea1d7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c5f1cec0 0x55d6c60a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64ea1d7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 0x55d6c609bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64ea1d7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c609c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64ea1d7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d6c5fc9df0 con 0x55d6c60a20f0 2024-05-27T07:44:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64ea1d7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d6c5fb9990 con 0x55d6c609c1a0 2024-05-27T07:44:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64ea1d7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d6c5fbd9d0 con 0x55d6c5f1cec0 2024-05-27T07:44:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64d8d5d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 0x55d6c609bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64d8d5d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 0x55d6c609bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60628/0 (socket says 172.21.15.114:60628) 2024-05-27T07:44:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64d8d5d700 1 -- 172.21.15.114:0/3839815902 learned_addr learned my addr 172.21.15.114:0/3839815902 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:56.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64d955e700 1 --2- 172.21.15.114:0/3839815902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c5f1cec0 0x55d6c60a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64d9d5f700 1 --2- 172.21.15.114:0/3839815902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c609c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.082+0000 7f64d8d5d700 1 -- 172.21.15.114:0/3839815902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c5f1cec0 msgr2=0x55d6c60a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:56.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.083+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/3839815902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c5f1cec0 0x55d6c60a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.083+0000 7f64d8d5d700 1 -- 172.21.15.114:0/3839815902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 msgr2=0x55d6c609c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:56.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.083+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/3839815902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c609c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.083+0000 7f64d8d5d700 1 -- 172.21.15.114:0/3839815902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6c5f63030 con 0x55d6c60a20f0 2024-05-27T07:44:56.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.083+0000 7f64d9d5f700 1 --2- 172.21.15.114:0/3839815902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c609c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:44:56.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.083+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/3839815902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 0x55d6c609bc60 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f64c4015000 tx=0x7f64c401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b317566cb6012bb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:56.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.083+0000 7f64bf7fe700 1 -- 172.21.15.114:0/3839815902 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c400be10 con 0x55d6c60a20f0 2024-05-27T07:44:56.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.083+0000 7f64bf7fe700 1 -- 172.21.15.114:0/3839815902 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64c4003b30 con 0x55d6c60a20f0 2024-05-27T07:44:56.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.083+0000 7f64bf7fe700 1 -- 172.21.15.114:0/3839815902 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c400fb60 con 0x55d6c60a20f0 2024-05-27T07:44:56.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.084+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/3839815902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 msgr2=0x55d6c609bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:56.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.084+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/3839815902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 0x55d6c609bc60 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f64c4015000 tx=0x7f64c401e000 comp rx=0 tx=0).stop 2024-05-27T07:44:56.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.084+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/3839815902 shutdown_connections 2024-05-27T07:44:56.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.084+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/3839815902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c609c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.084+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/3839815902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c5f1cec0 0x55d6c60a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.084+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/3839815902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 0x55d6c609bc60 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.084+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/3839815902 >> 172.21.15.114:0/3839815902 conn(0x55d6c5f1f7f0 msgr2=0x55d6c6096e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:56.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.084+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/3839815902 shutdown_connections 2024-05-27T07:44:56.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.084+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/3839815902 wait complete. 2024-05-27T07:44:56.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.084+0000 7f64ea1d7b80 1 -- start start 2024-05-27T07:44:56.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.084+0000 7f64ea1d7b80 1 Processor -- start 2024-05-27T07:44:56.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64ea1d7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 0x55d6c60a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64ea1d7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c60a5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64ea1d7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c60a20f0 0x55d6c60a6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64ea1d7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d6c5ec0020 con 0x55d6c5f1cec0 2024-05-27T07:44:56.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64ea1d7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d6c5f21e20 con 0x55d6c609c1a0 2024-05-27T07:44:56.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64ea1d7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d6c60a8b20 con 0x55d6c60a20f0 2024-05-27T07:44:56.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64ea1d7b80 10 client.0 _ll_register_callbacks cb 0x55d6c5f50630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:56.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64d955e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 0x55d6c60a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64d955e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 0x55d6c60a58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60640/0 (socket says 172.21.15.114:60640) 2024-05-27T07:44:56.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64d955e700 1 -- 172.21.15.114:0/2797606462 learned_addr learned my addr 172.21.15.114:0/2797606462 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:56.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64d9d5f700 1 --2- 172.21.15.114:0/2797606462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c60a20f0 0x55d6c60a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/2797606462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c60a5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64d955e700 1 -- 172.21.15.114:0/2797606462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c60a20f0 msgr2=0x55d6c60a6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:56.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64d955e700 1 --2- 172.21.15.114:0/2797606462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c60a20f0 0x55d6c60a6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64d955e700 1 -- 172.21.15.114:0/2797606462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 msgr2=0x55d6c60a5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:56.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64d955e700 1 --2- 172.21.15.114:0/2797606462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c60a5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.085+0000 7f64d955e700 1 -- 172.21.15.114:0/2797606462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6c5f09950 con 0x55d6c5f1cec0 2024-05-27T07:44:56.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64d955e700 1 --2- 172.21.15.114:0/2797606462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 0x55d6c60a58f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f64d000e910 tx=0x7f64d00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37a02bbd0aeca3a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:56.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64bcff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:44:56.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64bcff9700 1 -- 172.21.15.114:0/2797606462 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d00c2020 con 0x55d6c5f1cec0 2024-05-27T07:44:56.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64bcff9700 1 -- 172.21.15.114:0/2797606462 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f64d00a60c0 con 0x55d6c5f1cec0 2024-05-27T07:44:56.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/2797606462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64c4003b30 con 0x55d6c5f1cec0 2024-05-27T07:44:56.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64bcff9700 1 -- 172.21.15.114:0/2797606462 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f64d00a95e0 con 0x55d6c5f1cec0 2024-05-27T07:44:56.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64ea1d7b80 10 client.16497 fetch_fsmap learned FSMap version 32 2024-05-27T07:44:56.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/2797606462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d6c5f737e0 con 0x55d6c5f1cec0 2024-05-27T07:44:56.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64bcff9700 1 -- 172.21.15.114:0/2797606462 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64c4003b30 con 0x55d6c5f1cec0 2024-05-27T07:44:56.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64bcff9700 1 -- 172.21.15.114:0/2797606462 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 32) ==== 7725+0+0 (secure 0 0 0) 0x7f64d000cad0 con 0x55d6c5f1cec0 2024-05-27T07:44:56.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64ea1d7b80 10 client.16497 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T07:44:56.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64ea1d7b80 10 client.16497 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:56.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64ea1d7b80 20 client.16497 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:56.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/2797606462 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6c60b5a60 0x55d6c60b7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64ea1d7b80 4 client.16497 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:44:56.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/2797606462 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6c5fcbd40 con 0x55d6c60b5a60 2024-05-27T07:44:56.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.086+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/2797606462 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6c60b5a60 0x55d6c60b7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.087+0000 7f64bcff9700 10 client.16497 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:56.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.087+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/2797606462 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6c60b5a60 0x55d6c60b7e80 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=787ffdac53d05d36 server_cookie=80e51bcc7f75fe5f in_seq=0 out_seq=0 2024-05-27T07:44:56.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.088+0000 7f64bcff9700 1 -- 172.21.15.114:0/2797606462 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d6c5fcbd40 con 0x55d6c60b5a60 2024-05-27T07:44:56.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.088+0000 7f64bcff9700 10 client.16497 handle_command_reply: tid=0 2024-05-27T07:44:56.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.088+0000 7f64ea1d7b80 1 client.16497 shutdown 2024-05-27T07:44:56.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.088+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/2797606462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 msgr2=0x55d6c60a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:56.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.088+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/2797606462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 0x55d6c60a58f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f64d000e910 tx=0x7f64d00045e0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.088+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/2797606462 shutdown_connections 2024-05-27T07:44:56.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.088+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/2797606462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c60a20f0 0x55d6c60a6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.088+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/2797606462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 0x55d6c60a58f0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.088+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/2797606462 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6c60b5a60 0x55d6c60b7e80 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.088+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/2797606462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c60a5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.088+0000 7f64bcff9700 0 client.16497 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:56.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.089+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/2797606462 >> 172.21.15.114:0/2797606462 conn(0x55d6c5f1f7f0 msgr2=0x55d6c609ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.089+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/2797606462 shutdown_connections 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.089+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/2797606462 wait complete. 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.089+0000 7f64ea1d7b80 20 client.16497 trim_cache size 0 max 16384 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64ea1d7b80 1 Processor -- start 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64ea1d7b80 1 -- start start 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64ea1d7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c5f1cec0 0x55d6c609bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64ea1d7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c609ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64ea1d7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 0x55d6c60bd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64ea1d7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41b1450 con 0x55d6c60a20f0 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64ea1d7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41ae500 con 0x55d6c609c1a0 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64ea1d7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41b06b0 con 0x55d6c5f1cec0 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64d9d5f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 0x55d6c60bd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64d955e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c5f1cec0 0x55d6c609bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64d9d5f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 0x55d6c60bd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60654/0 (socket says 172.21.15.114:60654) 2024-05-27T07:44:56.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64d9d5f700 1 -- 172.21.15.114:0/1107978022 learned_addr learned my addr 172.21.15.114:0/1107978022 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:56.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/1107978022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c609ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64d9d5f700 1 -- 172.21.15.114:0/1107978022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c5f1cec0 msgr2=0x55d6c609bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:56.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64d9d5f700 1 --2- 172.21.15.114:0/1107978022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c5f1cec0 0x55d6c609bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64d9d5f700 1 -- 172.21.15.114:0/1107978022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 msgr2=0x55d6c609ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:56.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64d9d5f700 1 --2- 172.21.15.114:0/1107978022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c609ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64d9d5f700 1 -- 172.21.15.114:0/1107978022 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6c60a8190 con 0x55d6c60a20f0 2024-05-27T07:44:56.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/1107978022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c609ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:56.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.111+0000 7f64d955e700 1 --2- 172.21.15.114:0/1107978022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c5f1cec0 0x55d6c609bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:56.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.112+0000 7f64d9d5f700 1 --2- 172.21.15.114:0/1107978022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 0x55d6c60bd610 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f64cc007690 tx=0x7f64cc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f438f3cfbdbe31 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:56.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.112+0000 7f64beffd700 1 -- 172.21.15.114:0/1107978022 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc007e60 con 0x55d6c60a20f0 2024-05-27T07:44:56.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.112+0000 7f64beffd700 1 -- 172.21.15.114:0/1107978022 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64cc003c00 con 0x55d6c60a20f0 2024-05-27T07:44:56.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.112+0000 7f64beffd700 1 -- 172.21.15.114:0/1107978022 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc0177d0 con 0x55d6c60a20f0 2024-05-27T07:44:56.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.113+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/1107978022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 msgr2=0x55d6c60bd610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:56.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.113+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/1107978022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 0x55d6c60bd610 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f64cc007690 tx=0x7f64cc00b040 comp rx=0 tx=0).stop 2024-05-27T07:44:56.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.113+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/1107978022 shutdown_connections 2024-05-27T07:44:56.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.113+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/1107978022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c60a20f0 0x55d6c60bd610 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.113+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/1107978022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c5f1cec0 0x55d6c609bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.113+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/1107978022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c609ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.113+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/1107978022 >> 172.21.15.114:0/1107978022 conn(0x55d6c5f1f7f0 msgr2=0x55d6c60a5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:56.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.113+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/1107978022 shutdown_connections 2024-05-27T07:44:56.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.113+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/1107978022 wait complete. 2024-05-27T07:44:56.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.113+0000 7f64ea1d7b80 1 -- start start 2024-05-27T07:44:56.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.113+0000 7f64ea1d7b80 1 Processor -- start 2024-05-27T07:44:56.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64ea1d7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 0x55d6c60ae300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64ea1d7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c60aad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64ea1d7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c60a20f0 0x55d6c60ab2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64ea1d7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41affe0 con 0x55d6c5f1cec0 2024-05-27T07:44:56.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64ea1d7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41aeb70 con 0x55d6c609c1a0 2024-05-27T07:44:56.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64ea1d7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41af240 con 0x55d6c60a20f0 2024-05-27T07:44:56.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64ea1d7b80 10 client.0 _ll_register_callbacks cb 0x55d6c5f50630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:56.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64d955e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 0x55d6c60ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64d955e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 0x55d6c60ae300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60660/0 (socket says 172.21.15.114:60660) 2024-05-27T07:44:56.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64d955e700 1 -- 172.21.15.114:0/3264817257 learned_addr learned my addr 172.21.15.114:0/3264817257 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:56.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64d9d5f700 1 --2- 172.21.15.114:0/3264817257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c60a20f0 0x55d6c60ab2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/3264817257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c60aad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64d955e700 1 -- 172.21.15.114:0/3264817257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c60a20f0 msgr2=0x55d6c60ab2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:56.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64d955e700 1 --2- 172.21.15.114:0/3264817257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c60a20f0 0x55d6c60ab2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64d955e700 1 -- 172.21.15.114:0/3264817257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 msgr2=0x55d6c60aad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64d955e700 1 --2- 172.21.15.114:0/3264817257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c60aad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64d955e700 1 -- 172.21.15.114:0/3264817257 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6c60ab880 con 0x55d6c5f1cec0 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64d955e700 1 --2- 172.21.15.114:0/3264817257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 0x55d6c60ae300 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f64d00a9a10 tx=0x7f64d00dfff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e488b102021aed22 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64bcff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64bcff9700 1 -- 172.21.15.114:0/3264817257 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d00c2040 con 0x55d6c5f1cec0 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64bcff9700 1 -- 172.21.15.114:0/3264817257 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f64d00a60c0 con 0x55d6c5f1cec0 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.114+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/3264817257 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64cc003c00 con 0x55d6c5f1cec0 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.115+0000 7f64bcff9700 1 -- 172.21.15.114:0/3264817257 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f64d00a9ad0 con 0x55d6c5f1cec0 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.115+0000 7f64ea1d7b80 10 client.16503 fetch_fsmap learned FSMap version 32 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.115+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/3264817257 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d6c6121280 con 0x55d6c5f1cec0 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.115+0000 7f64bcff9700 1 -- 172.21.15.114:0/3264817257 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64cc003c00 con 0x55d6c5f1cec0 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.115+0000 7f64bcff9700 1 -- 172.21.15.114:0/3264817257 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 32) ==== 7725+0+0 (secure 0 0 0) 0x7f64d00a54d0 con 0x55d6c5f1cec0 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.115+0000 7f64ea1d7b80 10 client.16503 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.115+0000 7f64ea1d7b80 10 client.16503 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:56.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.115+0000 7f64ea1d7b80 20 client.16503 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.115+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/3264817257 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6c61217c0 0x55d6c6123be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.115+0000 7f64ea1d7b80 4 client.16503 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.115+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/3264817257 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d6c5fc5520 con 0x55d6c61217c0 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.115+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/3264817257 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6c61217c0 0x55d6c6123be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.116+0000 7f64bcff9700 10 client.16503 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.116+0000 7f64d8d5d700 1 --2- 172.21.15.114:0/3264817257 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6c61217c0 0x55d6c6123be0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc336f7f58efdb4e server_cookie=73f91dc580406acb in_seq=0 out_seq=0 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64bcff9700 1 -- 172.21.15.114:0/3264817257 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d6c5fc5520 con 0x55d6c61217c0 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64bcff9700 10 client.16503 handle_command_reply: tid=0 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64ea1d7b80 1 client.16503 shutdown 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/3264817257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 msgr2=0x55d6c60ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/3264817257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 0x55d6c60ae300 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f64d00a9a10 tx=0x7f64d00dfff0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/3264817257 shutdown_connections 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/3264817257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6c60a20f0 0x55d6c60ab2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/3264817257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6c5f1cec0 0x55d6c60ae300 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/3264817257 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6c61217c0 0x55d6c6123be0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64ea1d7b80 1 --2- 172.21.15.114:0/3264817257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6c609c1a0 0x55d6c60aad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/3264817257 >> 172.21.15.114:0/3264817257 conn(0x55d6c5f1f7f0 msgr2=0x55d6c6090570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:56.273 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:44:56.273 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:44:56.273 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "6d852ded-5dc9-4123-8a26-2f22c5b2bf2f", 2024-05-27T07:44:56.273 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:44:56.273 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:44:56.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64bcff9700 0 client.16503 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:56.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/3264817257 shutdown_connections 2024-05-27T07:44:56.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.117+0000 7f64ea1d7b80 1 -- 172.21.15.114:0/3264817257 wait complete. 2024-05-27T07:44:56.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.118+0000 7f64ea1d7b80 20 client.16503 trim_cache size 0 max 16384 2024-05-27T07:44:56.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.118+0000 7f64db7c2700 1 -- 172.21.15.114:0/1208568563 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64c00b29f0 msgr2=0x7f64c00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:56.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.118+0000 7f64db7c2700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64c00b29f0 0x7f64c00b4f00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f64c4006010 tx=0x7f64c4005960 comp rx=0 tx=0).stop 2024-05-27T07:44:56.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.118+0000 7f64db7c2700 1 -- 172.21.15.114:0/1208568563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64d4102780 msgr2=0x7f64d419aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:56.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.118+0000 7f64db7c2700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64d4102780 0x7f64d419aea0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f64d000e940 tx=0x7f64d000ec50 comp rx=0 tx=0).stop 2024-05-27T07:44:56.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.118+0000 7f64db7c2700 1 -- 172.21.15.114:0/1208568563 shutdown_connections 2024-05-27T07:44:56.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.118+0000 7f64db7c2700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d41092f0 0x7f64d419f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.118+0000 7f64db7c2700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64d4102780 0x7f64d419aea0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.118+0000 7f64db7c2700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64c00b29f0 0x7f64c00b4f00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.118+0000 7f64db7c2700 1 --2- 172.21.15.114:0/1208568563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64d41030d0 0x7f64d419b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:56.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.118+0000 7f64db7c2700 1 -- 172.21.15.114:0/1208568563 >> 172.21.15.114:0/1208568563 conn(0x7f64d40fe510 msgr2=0x7f64d4107110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:56.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.118+0000 7f64db7c2700 1 -- 172.21.15.114:0/1208568563 shutdown_connections 2024-05-27T07:44:56.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:56.118+0000 7f64db7c2700 1 -- 172.21.15.114:0/1208568563 wait complete. 2024-05-27T07:44:57.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:44:57.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 480072 13.58 MB/sec execute 69 sec latency 9.064 ms 2024-05-27T07:44:57.383 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:44:57.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:57 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:44:57.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:57 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:44:57.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:57 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:44:57.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:57 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:44:57.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:57 smithi186 ceph-mon[36647]: pgmap v164: 129 pgs: 129 active+clean; 49 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.0 MiB/s wr, 489 op/s 2024-05-27T07:44:57.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:57 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:44:57.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:57 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:44:57.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:57 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:44:57.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:57 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:44:57.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:57 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:44:57.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:57 smithi114 ceph-mon[32038]: pgmap v164: 129 pgs: 129 active+clean; 49 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.0 MiB/s wr, 489 op/s 2024-05-27T07:44:57.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:57 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:44:57.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:57 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:44:57.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:57 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:44:57.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:57 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:44:57.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:57 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:44:57.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:57 smithi163 ceph-mon[36451]: pgmap v164: 129 pgs: 129 active+clean; 49 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.0 MiB/s wr, 489 op/s 2024-05-27T07:44:57.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:57 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:44:58.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 482547 13.55 MB/sec execute 70 sec latency 92.678 ms 2024-05-27T07:44:59.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 484256 13.49 MB/sec execute 71 sec latency 39.711 ms 2024-05-27T07:44:59.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:44:59 smithi114 ceph-mon[32038]: pgmap v165: 129 pgs: 129 active+clean; 49 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 409 op/s 2024-05-27T07:44:59.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:44:59 smithi163 ceph-mon[36451]: pgmap v165: 129 pgs: 129 active+clean; 49 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 409 op/s 2024-05-27T07:44:59.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:44:59 smithi186 ceph-mon[36647]: pgmap v165: 129 pgs: 129 active+clean; 49 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 409 op/s 2024-05-27T07:44:59.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.656+0000 7f0434beb700 1 -- 172.21.15.114:0/881453689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04301034b0 msgr2=0x7f04301038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:59.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.656+0000 7f0434beb700 1 --2- 172.21.15.114:0/881453689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04301034b0 0x7f04301038c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f04200099c0 tx=0x7f0420009cd0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.656+0000 7f0434beb700 1 -- 172.21.15.114:0/881453689 shutdown_connections 2024-05-27T07:44:59.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.656+0000 7f0434beb700 1 --2- 172.21.15.114:0/881453689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0430108730 0x7f0430110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.656+0000 7f0434beb700 1 --2- 172.21.15.114:0/881453689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04301034b0 0x7f04301038c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.656+0000 7f0434beb700 1 --2- 172.21.15.114:0/881453689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0430107de0 0x7f04301081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.656+0000 7f0434beb700 1 -- 172.21.15.114:0/881453689 >> 172.21.15.114:0/881453689 conn(0x7f04300fedb0 msgr2=0x7f04301011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:59.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.656+0000 7f0434beb700 1 -- 172.21.15.114:0/881453689 shutdown_connections 2024-05-27T07:44:59.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.656+0000 7f0434beb700 1 -- 172.21.15.114:0/881453689 wait complete. 2024-05-27T07:44:59.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.657+0000 7f0434beb700 1 Processor -- start 2024-05-27T07:44:59.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.657+0000 7f0434beb700 1 -- start start 2024-05-27T07:44:59.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.657+0000 7f0434beb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04301034b0 0x7f04300ffcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.657+0000 7f0434beb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0430107de0 0x7f0430100200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.657+0000 7f0434beb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0430108730 0x7f0430100740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.657+0000 7f0434beb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0430113360 con 0x7f04301034b0 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.657+0000 7f0434beb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04301131e0 con 0x7f0430107de0 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.657+0000 7f0434beb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04301134e0 con 0x7f0430108730 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.657+0000 7f042e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04301034b0 0x7f04300ffcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.657+0000 7f042e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04301034b0 0x7f04300ffcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60690/0 (socket says 172.21.15.114:60690) 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.657+0000 7f042e59c700 1 -- 172.21.15.114:0/2931716918 learned_addr learned my addr 172.21.15.114:0/2931716918 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.657+0000 7f042ed9d700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0430108730 0x7f0430100740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.657+0000 7f042dd9b700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0430107de0 0x7f0430100200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.658+0000 7f042ed9d700 1 -- 172.21.15.114:0/2931716918 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0430107de0 msgr2=0x7f0430100200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.658+0000 7f042ed9d700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0430107de0 0x7f0430100200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.658+0000 7f042ed9d700 1 -- 172.21.15.114:0/2931716918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04301034b0 msgr2=0x7f04300ffcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.658+0000 7f042ed9d700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04301034b0 0x7f04300ffcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.658+0000 7f042ed9d700 1 -- 172.21.15.114:0/2931716918 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04301a9160 con 0x7f0430108730 2024-05-27T07:44:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.658+0000 7f042e59c700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04301034b0 0x7f04300ffcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.658+0000 7f042ed9d700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0430108730 0x7f0430100740 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f042800efc0 tx=0x7f042800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.658+0000 7f041f7fe700 1 -- 172.21.15.114:0/2931716918 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04280092e0 con 0x7f0430108730 2024-05-27T07:44:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.658+0000 7f0434beb700 1 -- 172.21.15.114:0/2931716918 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04301a93f0 con 0x7f0430108730 2024-05-27T07:44:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.658+0000 7f041f7fe700 1 -- 172.21.15.114:0/2931716918 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0428004bb0 con 0x7f0430108730 2024-05-27T07:44:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.658+0000 7f041f7fe700 1 -- 172.21.15.114:0/2931716918 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428004020 con 0x7f0430108730 2024-05-27T07:44:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.658+0000 7f0434beb700 1 -- 172.21.15.114:0/2931716918 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04301a98e0 con 0x7f0430108730 2024-05-27T07:44:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.662+0000 7f041f7fe700 1 -- 172.21.15.114:0/2931716918 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f042801b030 con 0x7f0430108730 2024-05-27T07:44:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.662+0000 7f041f7fe700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04180b29f0 0x7f04180b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.662+0000 7f042e59c700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04180b29f0 0x7f04180b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.662+0000 7f041f7fe700 1 -- 172.21.15.114:0/2931716918 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f04280e2430 con 0x7f0430108730 2024-05-27T07:44:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f0443600b80 1 Processor -- start 2024-05-27T07:44:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f0443600b80 1 -- start start 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f042e59c700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04180b29f0 0x7f04180b4f00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0420009990 tx=0x7f042001a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f0443600b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78893e40 0x563a78a17710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f0443600b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78a17c50 0x563a78a11a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f0443600b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a12390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f0443600b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563a78940c90 con 0x563a78a11f80 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f0443600b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563a78930830 con 0x563a78893e40 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f0443600b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563a78934870 con 0x563a78a17c50 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f042ed9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a12390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f042dd9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78a17c50 0x563a78a11a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f042ed9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a12390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60698/0 (socket says 172.21.15.114:60698) 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f042ed9d700 1 -- 172.21.15.114:0/2222676639 learned_addr learned my addr 172.21.15.114:0/2222676639 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.663+0000 7f042dd9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78a17c50 0x563a78a11a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:56094/0 (socket says 172.21.15.114:56094) 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.664+0000 7f042e59c700 1 --2- 172.21.15.114:0/2222676639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78893e40 0x563a78a17710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.664+0000 7f042ed9d700 1 -- 172.21.15.114:0/2222676639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78a17c50 msgr2=0x563a78a11a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.664+0000 7f042ed9d700 1 --2- 172.21.15.114:0/2222676639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78a17c50 0x563a78a11a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.664+0000 7f042ed9d700 1 -- 172.21.15.114:0/2222676639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78893e40 msgr2=0x563a78a17710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.664+0000 7f042ed9d700 1 --2- 172.21.15.114:0/2222676639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78893e40 0x563a78a17710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.664+0000 7f042ed9d700 1 -- 172.21.15.114:0/2222676639 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a788808d0 con 0x563a78a11f80 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.664+0000 7f042e59c700 1 --2- 172.21.15.114:0/2222676639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78893e40 0x563a78a17710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.664+0000 7f042dd9b700 1 --2- 172.21.15.114:0/2222676639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78a17c50 0x563a78a11a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.664+0000 7f042ed9d700 1 --2- 172.21.15.114:0/2222676639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a12390 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f04280090a0 tx=0x7f04280a8f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21365958e96ec272 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.664+0000 7f041cff9700 1 -- 172.21.15.114:0/2222676639 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04280ad7a0 con 0x563a78a11f80 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.664+0000 7f041cff9700 1 -- 172.21.15.114:0/2222676639 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04280b21c0 con 0x563a78a11f80 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.664+0000 7f041cff9700 1 -- 172.21.15.114:0/2222676639 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04280a3b10 con 0x563a78a11f80 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.665+0000 7f0443600b80 1 -- 172.21.15.114:0/2222676639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 msgr2=0x563a78a12390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.665+0000 7f0443600b80 1 --2- 172.21.15.114:0/2222676639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a12390 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f04280090a0 tx=0x7f04280a8f00 comp rx=0 tx=0).stop 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.665+0000 7f0443600b80 1 -- 172.21.15.114:0/2222676639 shutdown_connections 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.665+0000 7f0443600b80 1 --2- 172.21.15.114:0/2222676639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a12390 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.665+0000 7f0443600b80 1 --2- 172.21.15.114:0/2222676639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78893e40 0x563a78a17710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.665+0000 7f0443600b80 1 --2- 172.21.15.114:0/2222676639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78a17c50 0x563a78a11a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.665+0000 7f0443600b80 1 -- 172.21.15.114:0/2222676639 >> 172.21.15.114:0/2222676639 conn(0x563a788eeaf0 msgr2=0x563a78a0c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.665+0000 7f0443600b80 1 -- 172.21.15.114:0/2222676639 shutdown_connections 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.665+0000 7f0443600b80 1 -- 172.21.15.114:0/2222676639 wait complete. 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f0443600b80 1 -- start start 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f0443600b80 1 Processor -- start 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f0443600b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78893e40 0x563a78a1b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f0443600b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a11f80 0x563a78a1b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f0443600b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78a17c50 0x563a78a1be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f0443600b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563a78836fa0 con 0x563a78893e40 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f0443600b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563a788d27c0 con 0x563a78a11f80 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f0443600b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563a78a1e740 con 0x563a78a17c50 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f0443600b80 10 client.0 _ll_register_callbacks cb 0x563a788cb920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f042e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78893e40 0x563a78a1b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f042dd9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a11f80 0x563a78a1b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f042e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78893e40 0x563a78a1b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60712/0 (socket says 172.21.15.114:60712) 2024-05-27T07:44:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f042e59c700 1 -- 172.21.15.114:0/1460616578 learned_addr learned my addr 172.21.15.114:0/1460616578 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f042dd9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a11f80 0x563a78a1b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:52906/0 (socket says 172.21.15.114:52906) 2024-05-27T07:44:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.666+0000 7f042ed9d700 1 --2- 172.21.15.114:0/1460616578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78a17c50 0x563a78a1be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f042e59c700 1 -- 172.21.15.114:0/1460616578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78a17c50 msgr2=0x563a78a1be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f042e59c700 1 --2- 172.21.15.114:0/1460616578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78a17c50 0x563a78a1be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f042e59c700 1 -- 172.21.15.114:0/1460616578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a11f80 msgr2=0x563a78a1b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f042e59c700 1 --2- 172.21.15.114:0/1460616578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a11f80 0x563a78a1b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f042e59c700 1 -- 172.21.15.114:0/1460616578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a788cf690 con 0x563a78893e40 2024-05-27T07:44:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f042e59c700 1 --2- 172.21.15.114:0/1460616578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78893e40 0x563a78a1b3d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0420000c00 tx=0x7f0420009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01af4d373166597 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f0415ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:44:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f0415ffb700 1 -- 172.21.15.114:0/1460616578 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042000be60 con 0x563a78893e40 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f0415ffb700 1 -- 172.21.15.114:0/1460616578 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f0420031460 con 0x563a78893e40 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f0443600b80 1 -- 172.21.15.114:0/1460616578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04280b21c0 con 0x563a78893e40 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f0415ffb700 1 -- 172.21.15.114:0/1460616578 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f042003a050 con 0x563a78893e40 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f0443600b80 10 client.16515 fetch_fsmap learned FSMap version 32 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f0443600b80 1 -- 172.21.15.114:0/1460616578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563a78a29240 con 0x563a78893e40 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f0415ffb700 1 -- 172.21.15.114:0/1460616578 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04280b21c0 con 0x563a78893e40 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.667+0000 7f0415ffb700 1 -- 172.21.15.114:0/1460616578 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 32) ==== 7725+0+0 (secure 0 0 0) 0x7f042000f760 con 0x563a78893e40 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.668+0000 7f0443600b80 10 client.16515 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.668+0000 7f0443600b80 10 client.16515 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.668+0000 7f0443600b80 20 client.16515 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.668+0000 7f0443600b80 1 --2- 172.21.15.114:0/1460616578 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a78a2b8d0 0x563a78a2dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.668+0000 7f0443600b80 4 client.16515 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.668+0000 7f0443600b80 1 -- 172.21.15.114:0/1460616578 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a789dd9f0 con 0x563a78a2b8d0 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.668+0000 7f042dd9b700 1 --2- 172.21.15.114:0/1460616578 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a78a2b8d0 0x563a78a2dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.668+0000 7f0415ffb700 10 client.16515 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:59.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.668+0000 7f042dd9b700 1 --2- 172.21.15.114:0/1460616578 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a78a2b8d0 0x563a78a2dd40 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f7b214d9ba2ec7e server_cookie=4fc5df9fd0a2dcad in_seq=0 out_seq=0 2024-05-27T07:44:59.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0415ffb700 1 -- 172.21.15.114:0/1460616578 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563a789dd9f0 con 0x563a78a2b8d0 2024-05-27T07:44:59.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0415ffb700 10 client.16515 handle_command_reply: tid=0 2024-05-27T07:44:59.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0443600b80 1 client.16515 shutdown 2024-05-27T07:44:59.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0443600b80 1 -- 172.21.15.114:0/1460616578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78893e40 msgr2=0x563a78a1b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:59.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0443600b80 1 --2- 172.21.15.114:0/1460616578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78893e40 0x563a78a1b3d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0420000c00 tx=0x7f0420009310 comp rx=0 tx=0).stop 2024-05-27T07:44:59.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0443600b80 1 -- 172.21.15.114:0/1460616578 shutdown_connections 2024-05-27T07:44:59.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0443600b80 1 --2- 172.21.15.114:0/1460616578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78a17c50 0x563a78a1be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0443600b80 1 --2- 172.21.15.114:0/1460616578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78893e40 0x563a78a1b3d0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0443600b80 1 --2- 172.21.15.114:0/1460616578 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a78a2b8d0 0x563a78a2dd40 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0443600b80 1 --2- 172.21.15.114:0/1460616578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a11f80 0x563a78a1b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0443600b80 1 -- 172.21.15.114:0/1460616578 >> 172.21.15.114:0/1460616578 conn(0x563a788eeaf0 msgr2=0x563a78a10a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:59.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0415ffb700 0 client.16515 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0443600b80 1 -- 172.21.15.114:0/1460616578 shutdown_connections 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0443600b80 1 -- 172.21.15.114:0/1460616578 wait complete. 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.669+0000 7f0443600b80 20 client.16515 trim_cache size 0 max 16384 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.690+0000 7f0443600b80 1 Processor -- start 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.690+0000 7f0443600b80 1 -- start start 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.690+0000 7f0443600b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78893e40 0x563a78a15720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.690+0000 7f0443600b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a277e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.690+0000 7f0443600b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a17c50 0x563a78a1d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.690+0000 7f0443600b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04301b3630 con 0x563a78a11f80 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.690+0000 7f0443600b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04301b06e0 con 0x563a78a17c50 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.690+0000 7f0443600b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04301b2890 con 0x563a78893e40 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.690+0000 7f042dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a277e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.690+0000 7f042dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a277e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60718/0 (socket says 172.21.15.114:60718) 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.690+0000 7f042dd9b700 1 -- 172.21.15.114:0/1479652850 learned_addr learned my addr 172.21.15.114:0/1479652850 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:59.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.690+0000 7f042ed9d700 1 --2- 172.21.15.114:0/1479652850 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a17c50 0x563a78a1d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.690+0000 7f042e59c700 1 --2- 172.21.15.114:0/1479652850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78893e40 0x563a78a15720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.691+0000 7f042dd9b700 1 -- 172.21.15.114:0/1479652850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78893e40 msgr2=0x563a78a15720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.691+0000 7f042dd9b700 1 --2- 172.21.15.114:0/1479652850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78893e40 0x563a78a15720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.691+0000 7f042dd9b700 1 -- 172.21.15.114:0/1479652850 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a17c50 msgr2=0x563a78a1d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.691+0000 7f042dd9b700 1 --2- 172.21.15.114:0/1479652850 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a17c50 0x563a78a1d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.691+0000 7f042dd9b700 1 -- 172.21.15.114:0/1479652850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a78a1a390 con 0x563a78a11f80 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.691+0000 7f042dd9b700 1 --2- 172.21.15.114:0/1479652850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a277e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x563a78a0e0a0 tx=0x7f0424009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174a7d30d9989cb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.691+0000 7f0417fff700 1 -- 172.21.15.114:0/1479652850 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04240120d0 con 0x563a78a11f80 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.691+0000 7f0417fff700 1 -- 172.21.15.114:0/1479652850 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0424012270 con 0x563a78a11f80 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.691+0000 7f0417fff700 1 -- 172.21.15.114:0/1479652850 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0424010430 con 0x563a78a11f80 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.692+0000 7f0443600b80 1 -- 172.21.15.114:0/1479652850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 msgr2=0x563a78a277e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.692+0000 7f0443600b80 1 --2- 172.21.15.114:0/1479652850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a277e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x563a78a0e0a0 tx=0x7f0424009260 comp rx=0 tx=0).stop 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.692+0000 7f0443600b80 1 -- 172.21.15.114:0/1479652850 shutdown_connections 2024-05-27T07:44:59.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.692+0000 7f0443600b80 1 --2- 172.21.15.114:0/1479652850 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a17c50 0x563a78a1d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.692+0000 7f0443600b80 1 --2- 172.21.15.114:0/1479652850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78893e40 0x563a78a15720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.692+0000 7f0443600b80 1 --2- 172.21.15.114:0/1479652850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a277e0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.692+0000 7f0443600b80 1 -- 172.21.15.114:0/1479652850 >> 172.21.15.114:0/1479652850 conn(0x563a788eeaf0 msgr2=0x563a78a15010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.692+0000 7f0443600b80 1 -- 172.21.15.114:0/1479652850 shutdown_connections 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.692+0000 7f0443600b80 1 -- 172.21.15.114:0/1479652850 wait complete. 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.692+0000 7f0443600b80 1 -- start start 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.692+0000 7f0443600b80 1 Processor -- start 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f0443600b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78893e40 0x563a78a241b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f0443600b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a32e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f0443600b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a17c50 0x563a78a33440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f0443600b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04301b21c0 con 0x563a78a11f80 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f0443600b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04301b0d50 con 0x563a78a17c50 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f0443600b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04301b1420 con 0x563a78893e40 2024-05-27T07:44:59.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f0443600b80 10 client.0 _ll_register_callbacks cb 0x7f04200053c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f042dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a32e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f042e59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78893e40 0x563a78a241b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f042dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a32e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60728/0 (socket says 172.21.15.114:60728) 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f042e59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78893e40 0x563a78a241b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:56110/0 (socket says 172.21.15.114:56110) 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f042dd9b700 1 -- 172.21.15.114:0/2516446932 learned_addr learned my addr 172.21.15.114:0/2516446932 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f042ed9d700 1 --2- 172.21.15.114:0/2516446932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a17c50 0x563a78a33440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f042dd9b700 1 -- 172.21.15.114:0/2516446932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78893e40 msgr2=0x563a78a241b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f042dd9b700 1 --2- 172.21.15.114:0/2516446932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78893e40 0x563a78a241b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f042dd9b700 1 -- 172.21.15.114:0/2516446932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a17c50 msgr2=0x563a78a33440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f042dd9b700 1 --2- 172.21.15.114:0/2516446932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a17c50 0x563a78a33440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f042dd9b700 1 -- 172.21.15.114:0/2516446932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a78a33c30 con 0x563a78a11f80 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f042ed9d700 1 --2- 172.21.15.114:0/2516446932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a17c50 0x563a78a33440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f042dd9b700 1 --2- 172.21.15.114:0/2516446932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a32e40 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0424000c00 tx=0x7f0424005150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68662dcc76fa95f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:44:59.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f0415ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:44:59.857 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:44:59.857 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:44:59.857 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:44:59.857 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:44:59.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f0415ffb700 1 -- 172.21.15.114:0/2516446932 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0424005600 con 0x563a78a11f80 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f0415ffb700 1 -- 172.21.15.114:0/2516446932 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f0424004750 con 0x563a78a11f80 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.693+0000 7f0443600b80 1 -- 172.21.15.114:0/2516446932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0424012270 con 0x563a78a11f80 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.694+0000 7f0415ffb700 1 -- 172.21.15.114:0/2516446932 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f042403a050 con 0x563a78a11f80 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.694+0000 7f0443600b80 10 client.16521 fetch_fsmap learned FSMap version 32 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.694+0000 7f0443600b80 1 -- 172.21.15.114:0/2516446932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563a78a963c0 con 0x563a78a11f80 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.694+0000 7f0415ffb700 1 -- 172.21.15.114:0/2516446932 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0424012270 con 0x563a78a11f80 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.694+0000 7f0415ffb700 1 -- 172.21.15.114:0/2516446932 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 32) ==== 7725+0+0 (secure 0 0 0) 0x7f042402f020 con 0x563a78a11f80 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.694+0000 7f0443600b80 10 client.16521 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.694+0000 7f0443600b80 10 client.16521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.694+0000 7f0443600b80 20 client.16521 populate_metadata read hostname 'smithi114' 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.694+0000 7f0443600b80 1 --2- 172.21.15.114:0/2516446932 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a78a96a20 0x563a78a98e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.694+0000 7f0443600b80 4 client.16521 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.694+0000 7f0443600b80 1 -- 172.21.15.114:0/2516446932 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563a78942be0 con 0x563a78a96a20 2024-05-27T07:44:59.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.694+0000 7f042e59c700 1 --2- 172.21.15.114:0/2516446932 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a78a96a20 0x563a78a98e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0415ffb700 10 client.16521 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f042e59c700 1 --2- 172.21.15.114:0/2516446932 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a78a96a20 0x563a78a98e40 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c4bfaff8f67ecae server_cookie=2a1048a97de46bf1 in_seq=0 out_seq=0 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0415ffb700 1 -- 172.21.15.114:0/2516446932 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563a78942be0 con 0x563a78a96a20 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0415ffb700 10 client.16521 handle_command_reply: tid=0 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0443600b80 1 client.16521 shutdown 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0443600b80 1 -- 172.21.15.114:0/2516446932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 msgr2=0x563a78a32e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0443600b80 1 --2- 172.21.15.114:0/2516446932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a32e40 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0424000c00 tx=0x7f0424005150 comp rx=0 tx=0).stop 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0443600b80 1 -- 172.21.15.114:0/2516446932 shutdown_connections 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0443600b80 1 --2- 172.21.15.114:0/2516446932 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563a78a96a20 0x563a78a98e40 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0443600b80 1 --2- 172.21.15.114:0/2516446932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563a78a17c50 0x563a78a33440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0443600b80 1 --2- 172.21.15.114:0/2516446932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a78893e40 0x563a78a241b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0415ffb700 0 client.16521 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0443600b80 1 --2- 172.21.15.114:0/2516446932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a78a11f80 0x563a78a32e40 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0443600b80 1 -- 172.21.15.114:0/2516446932 >> 172.21.15.114:0/2516446932 conn(0x563a788eeaf0 msgr2=0x563a78a16ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0443600b80 1 -- 172.21.15.114:0/2516446932 shutdown_connections 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0443600b80 1 -- 172.21.15.114:0/2516446932 wait complete. 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.695+0000 7f0443600b80 20 client.16521 trim_cache size 0 max 16384 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.696+0000 7f0434beb700 1 -- 172.21.15.114:0/2931716918 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04180b29f0 msgr2=0x7f04180b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.696+0000 7f0434beb700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04180b29f0 0x7f04180b4f00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0420009990 tx=0x7f042001a040 comp rx=0 tx=0).stop 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.696+0000 7f0434beb700 1 -- 172.21.15.114:0/2931716918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0430108730 msgr2=0x7f0430100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.696+0000 7f0434beb700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0430108730 0x7f0430100740 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f042800efc0 tx=0x7f042800c3a0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.696+0000 7f0434beb700 1 -- 172.21.15.114:0/2931716918 shutdown_connections 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.696+0000 7f0434beb700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0430108730 0x7f0430100740 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.696+0000 7f0434beb700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04301034b0 0x7f04300ffcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.696+0000 7f0434beb700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04180b29f0 0x7f04180b4f00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.696+0000 7f0434beb700 1 --2- 172.21.15.114:0/2931716918 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0430107de0 0x7f0430100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.696+0000 7f0434beb700 1 -- 172.21.15.114:0/2931716918 >> 172.21.15.114:0/2931716918 conn(0x7f04300fedb0 msgr2=0x7f0430101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:44:59.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.696+0000 7f0434beb700 1 -- 172.21.15.114:0/2931716918 shutdown_connections 2024-05-27T07:44:59.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:44:59.696+0000 7f0434beb700 1 -- 172.21.15.114:0/2931716918 wait complete. 2024-05-27T07:45:00.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 487178 13.54 MB/sec execute 72 sec latency 9.344 ms 2024-05-27T07:45:00.684 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:00 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:45:00.713 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:45:00.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:45:00.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:00 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:45:00.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:00 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:45:00.727 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:00 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:45:00.727 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:00 smithi186 systemd[1]: Stopped Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:45:00.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:00 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.723s CPU time. 2024-05-27T07:45:00.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:00 smithi186 systemd[1]: Starting Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:45:00.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:00 smithi186 podman[72363]: 2024-05-27 07:45:00.381812868 +0000 UTC m=+0.018286162 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:45:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:01.246 INFO:tasks.workunit.client.0.smithi114.stdout: 1 489027 13.44 MB/sec execute 73 sec latency 85.339 ms 2024-05-27T07:45:01.414 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:01 smithi114 ceph-mon[32038]: pgmap v166: 129 pgs: 129 active+clean; 54 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 508 op/s 2024-05-27T07:45:01.414 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:01 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:45:01.414 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:01 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:45:01.414 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:01 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:45:01.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:01 smithi163 ceph-mon[36451]: pgmap v166: 129 pgs: 129 active+clean; 54 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 508 op/s 2024-05-27T07:45:01.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:01 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:45:01.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:01 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:45:01.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:01 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:45:01.727 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:01 smithi186 podman[72363]: 2024-05-27 07:45:01.387141028 +0000 UTC m=+1.023614301 container create 6f41e07e18921c088441505e2e68f1fc482aa4fdcb271770fec6508109065adf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:45:01.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:01 smithi186 ceph-mon[36647]: pgmap v166: 129 pgs: 129 active+clean; 54 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 508 op/s 2024-05-27T07:45:01.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:01 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:45:01.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:01 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:45:01.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:01 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:45:02.247 INFO:tasks.workunit.client.0.smithi114.stdout: 1 489259 13.28 MB/sec execute 74 sec latency 160.581 ms 2024-05-27T07:45:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.687+0000 7f742b784640 1 -- 172.21.15.114:0/824726501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7424108ba0 msgr2=0x7f7424102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.687+0000 7f742b784640 1 --2- 172.21.15.114:0/824726501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7424108ba0 0x7f7424102740 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f7420009900 tx=0x7f742002a410 comp rx=0 tx=0).stop 2024-05-27T07:45:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.687+0000 7f742b784640 1 -- 172.21.15.114:0/824726501 shutdown_connections 2024-05-27T07:45:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.687+0000 7f742b784640 1 --2- 172.21.15.114:0/824726501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74241032f0 0x7f742410f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.687+0000 7f742b784640 1 --2- 172.21.15.114:0/824726501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7424109040 0x7f7424102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.687+0000 7f742b784640 1 --2- 172.21.15.114:0/824726501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7424108ba0 0x7f7424102740 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.687+0000 7f742b784640 1 -- 172.21.15.114:0/824726501 >> 172.21.15.114:0/824726501 conn(0x7f7424078270 msgr2=0x7f74240786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.688+0000 7f742b784640 1 -- 172.21.15.114:0/824726501 shutdown_connections 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.688+0000 7f742b784640 1 -- 172.21.15.114:0/824726501 wait complete. 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.688+0000 7f742b784640 1 Processor -- start 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.688+0000 7f742b784640 1 -- start start 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.688+0000 7f742b784640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74241032f0 0x7f742419b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.688+0000 7f742b784640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7424108ba0 0x7f742419bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.688+0000 7f742b784640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7424109040 0x7f742419c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.688+0000 7f742b784640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7424112d90 con 0x7f74241032f0 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.688+0000 7f742b784640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7424112c10 con 0x7f7424108ba0 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.688+0000 7f742b784640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7424112f10 con 0x7f7424109040 2024-05-27T07:45:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.689+0000 7f7429520640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74241032f0 0x7f742419b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.689+0000 7f7429520640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74241032f0 0x7f742419b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60732/0 (socket says 172.21.15.114:60732) 2024-05-27T07:45:02.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.689+0000 7f7429520640 1 -- 172.21.15.114:0/2146789481 learned_addr learned my addr 172.21.15.114:0/2146789481 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:02.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.689+0000 7f7429d21640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7424109040 0x7f742419c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.689+0000 7f7428d1f640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7424108ba0 0x7f742419bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.689+0000 7f7429d21640 1 -- 172.21.15.114:0/2146789481 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7424108ba0 msgr2=0x7f742419bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.689+0000 7f7429d21640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7424108ba0 0x7f742419bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.689+0000 7f7429d21640 1 -- 172.21.15.114:0/2146789481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74241032f0 msgr2=0x7f742419b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.689+0000 7f7429d21640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74241032f0 0x7f742419b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.689+0000 7f7429d21640 1 -- 172.21.15.114:0/2146789481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74241a6a00 con 0x7f7424109040 2024-05-27T07:45:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.689+0000 7f7429520640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74241032f0 0x7f742419b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.689+0000 7f7429d21640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7424109040 0x7f742419c170 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f741c009fd0 tx=0x7f741c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.689+0000 7f741a7fc640 1 -- 172.21.15.114:0/2146789481 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c00ce20 con 0x7f7424109040 2024-05-27T07:45:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.690+0000 7f742b784640 1 -- 172.21.15.114:0/2146789481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74241a6cf0 con 0x7f7424109040 2024-05-27T07:45:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.690+0000 7f741a7fc640 1 -- 172.21.15.114:0/2146789481 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f741c005020 con 0x7f7424109040 2024-05-27T07:45:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.690+0000 7f741a7fc640 1 -- 172.21.15.114:0/2146789481 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c010690 con 0x7f7424109040 2024-05-27T07:45:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.690+0000 7f742b784640 1 -- 172.21.15.114:0/2146789481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74241a71d0 con 0x7f7424109040 2024-05-27T07:45:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.692+0000 7f741a7fc640 1 -- 172.21.15.114:0/2146789481 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f741c0054d0 con 0x7f7424109040 2024-05-27T07:45:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.692+0000 7f743263c740 1 Processor -- start 2024-05-27T07:45:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.692+0000 7f743263c740 1 -- start start 2024-05-27T07:45:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f743263c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e023d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f743263c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a382f0 0x560150e02910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f743263c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e02e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f743263c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560150d74c30 con 0x560150a382f0 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f743263c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560150d647d0 con 0x560150a404d0 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f743263c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560150d68810 con 0x560150a611b0 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f7428d1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e023d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f7429520640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a382f0 0x560150e02910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f7428d1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e023d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:56136/0 (socket says 172.21.15.114:56136) 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f7428d1f640 1 -- 172.21.15.114:0/3259075593 learned_addr learned my addr 172.21.15.114:0/3259075593 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f7428d1f640 1 --2- 172.21.15.114:0/3259075593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e02e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f7428d1f640 1 -- 172.21.15.114:0/3259075593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 msgr2=0x560150e02e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f7428d1f640 1 --2- 172.21.15.114:0/3259075593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e02e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f7428d1f640 1 -- 172.21.15.114:0/3259075593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a382f0 msgr2=0x560150e02910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f7428d1f640 1 --2- 172.21.15.114:0/3259075593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a382f0 0x560150e02910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f7428d1f640 1 -- 172.21.15.114:0/3259075593 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560150a79210 con 0x560150a611b0 2024-05-27T07:45:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f741a7fc640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f74100b2950 0x7f74100b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f7429520640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f74100b2950 0x7f74100b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f7428d1f640 1 --2- 172.21.15.114:0/3259075593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e023d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7414013450 tx=0x7f741400b5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b532aa62cdd8b0ac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.693+0000 7f741a7fc640 1 -- 172.21.15.114:0/2146789481 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f741c0e3140 con 0x7f7424109040 2024-05-27T07:45:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.694+0000 7f740f7fe640 1 -- 172.21.15.114:0/3259075593 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7414019030 con 0x560150a611b0 2024-05-27T07:45:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.694+0000 7f740f7fe640 1 -- 172.21.15.114:0/3259075593 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74140035f0 con 0x560150a611b0 2024-05-27T07:45:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.694+0000 7f740f7fe640 1 -- 172.21.15.114:0/3259075593 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7414011680 con 0x560150a611b0 2024-05-27T07:45:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.694+0000 7f743263c740 1 -- 172.21.15.114:0/3259075593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 msgr2=0x560150e023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.694+0000 7f7429520640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f74100b2950 0x7f74100b4e60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x560150dffb10 tx=0x7f7420032ed0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.694+0000 7f743263c740 1 --2- 172.21.15.114:0/3259075593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e023d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7414013450 tx=0x7f741400b5a0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.694+0000 7f743263c740 1 -- 172.21.15.114:0/3259075593 shutdown_connections 2024-05-27T07:45:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.694+0000 7f743263c740 1 --2- 172.21.15.114:0/3259075593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e02e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.694+0000 7f743263c740 1 --2- 172.21.15.114:0/3259075593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a382f0 0x560150e02910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.694+0000 7f743263c740 1 --2- 172.21.15.114:0/3259075593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e023d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.694+0000 7f743263c740 1 -- 172.21.15.114:0/3259075593 >> 172.21.15.114:0/3259075593 conn(0x560150a55300 msgr2=0x560150dfeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.695+0000 7f743263c740 1 -- 172.21.15.114:0/3259075593 shutdown_connections 2024-05-27T07:45:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.695+0000 7f743263c740 1 -- 172.21.15.114:0/3259075593 wait complete. 2024-05-27T07:45:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.695+0000 7f743263c740 1 -- start start 2024-05-27T07:45:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.695+0000 7f743263c740 1 Processor -- start 2024-05-27T07:45:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.695+0000 7f743263c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a382f0 0x560150e0e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.695+0000 7f743263c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e0eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f743263c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a611b0 0x560150e0c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f743263c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560150a4a790 con 0x560150a611b0 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f743263c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560150d20890 con 0x560150a404d0 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f743263c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560150e0f500 con 0x560150a382f0 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f7429520640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a382f0 0x560150e0e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f743263c740 10 client.0 _ll_register_callbacks cb 0x560150a78a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f7429d21640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a611b0 0x560150e0c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f7429520640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a382f0 0x560150e0e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:56140/0 (socket says 172.21.15.114:56140) 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f7429520640 1 -- 172.21.15.114:0/515220008 learned_addr learned my addr 172.21.15.114:0/515220008 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f7428d1f640 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e0eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f7429d21640 1 -- 172.21.15.114:0/515220008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a382f0 msgr2=0x560150e0e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f7429d21640 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a382f0 0x560150e0e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f7429d21640 1 -- 172.21.15.114:0/515220008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 msgr2=0x560150e0eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f7429d21640 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e0eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f7429d21640 1 -- 172.21.15.114:0/515220008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560150b45620 con 0x560150a611b0 2024-05-27T07:45:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f7429520640 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a382f0 0x560150e0e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f7428d1f640 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e0eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:45:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f7429d21640 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a611b0 0x560150e0c970 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f741c0e04e0 tx=0x7f741c0aa830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55da5c2f8e4c52e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f740cff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f740cff9640 1 -- 172.21.15.114:0/515220008 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c06d020 con 0x560150a611b0 2024-05-27T07:45:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.696+0000 7f740cff9640 1 -- 172.21.15.114:0/515220008 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f741c0a6300 con 0x560150a611b0 2024-05-27T07:45:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.697+0000 7f743263c740 1 -- 172.21.15.114:0/515220008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74140035f0 con 0x560150a611b0 2024-05-27T07:45:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.697+0000 7f740cff9640 1 -- 172.21.15.114:0/515220008 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f741c0b2ed0 con 0x560150a611b0 2024-05-27T07:45:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.697+0000 7f743263c740 10 client.16533 fetch_fsmap learned FSMap version 32 2024-05-27T07:45:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.697+0000 7f743263c740 1 -- 172.21.15.114:0/515220008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560150e1b910 con 0x560150a611b0 2024-05-27T07:45:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.697+0000 7f740cff9640 1 -- 172.21.15.114:0/515220008 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74140035f0 con 0x560150a611b0 2024-05-27T07:45:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.697+0000 7f740cff9640 1 -- 172.21.15.114:0/515220008 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 32) ==== 7725+0+0 (secure 0 0 0) 0x7f741c0a5b00 con 0x560150a611b0 2024-05-27T07:45:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.697+0000 7f743263c740 10 client.16533 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T07:45:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.697+0000 7f743263c740 10 client.16533 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.697+0000 7f743263c740 20 client.16533 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.697+0000 7f743263c740 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560150a1fd80 0x560150e1ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.697+0000 7f743263c740 4 client.16533 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.697+0000 7f743263c740 1 -- 172.21.15.114:0/515220008 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560150d76b80 con 0x560150a1fd80 2024-05-27T07:45:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.697+0000 7f7429520640 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560150a1fd80 0x560150e1ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.698+0000 7f740cff9640 10 client.16533 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.698+0000 7f7429520640 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560150a1fd80 0x560150e1ee50 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=409726200437c1f server_cookie=b25af9d29c25b27c in_seq=0 out_seq=0 2024-05-27T07:45:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f740cff9640 1 -- 172.21.15.114:0/515220008 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560150d76b80 con 0x560150a1fd80 2024-05-27T07:45:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f740cff9640 10 client.16533 handle_command_reply: tid=0 2024-05-27T07:45:02.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f743263c740 1 client.16533 shutdown 2024-05-27T07:45:02.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f743263c740 1 -- 172.21.15.114:0/515220008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a611b0 msgr2=0x560150e0c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:02.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f743263c740 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a611b0 0x560150e0c970 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f741c0e04e0 tx=0x7f741c0aa830 comp rx=0 tx=0).stop 2024-05-27T07:45:02.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f743263c740 1 -- 172.21.15.114:0/515220008 shutdown_connections 2024-05-27T07:45:02.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f743263c740 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560150a1fd80 0x560150e1ee50 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f743263c740 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a611b0 0x560150e0c970 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f743263c740 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e0eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f740cff9640 0 client.16533 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:02.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f743263c740 1 --2- 172.21.15.114:0/515220008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a382f0 0x560150e0e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f743263c740 1 -- 172.21.15.114:0/515220008 >> 172.21.15.114:0/515220008 conn(0x560150a55300 msgr2=0x560150e02200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:02.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f743263c740 1 -- 172.21.15.114:0/515220008 shutdown_connections 2024-05-27T07:45:02.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f743263c740 1 -- 172.21.15.114:0/515220008 wait complete. 2024-05-27T07:45:02.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.700+0000 7f743263c740 20 client.16533 trim_cache size 0 max 16384 2024-05-27T07:45:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.716+0000 7f743263c740 1 Processor -- start 2024-05-27T07:45:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.716+0000 7f743263c740 1 -- start start 2024-05-27T07:45:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.716+0000 7f743263c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a382f0 0x560150e06730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.716+0000 7f743263c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a404d0 0x560150e02160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.716+0000 7f743263c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e026a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.716+0000 7f743263c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f74241afec0 con 0x560150a404d0 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.716+0000 7f743263c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f74241acf70 con 0x560150a382f0 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.716+0000 7f743263c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f74241af120 con 0x560150a611b0 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.716+0000 7f7428d1f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a404d0 0x560150e02160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.716+0000 7f7428d1f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a404d0 0x560150e02160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60764/0 (socket says 172.21.15.114:60764) 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.716+0000 7f7429d21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e026a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.716+0000 7f7428d1f640 1 -- 172.21.15.114:0/2938060401 learned_addr learned my addr 172.21.15.114:0/2938060401 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.716+0000 7f7429520640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a382f0 0x560150e06730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.717+0000 7f7429d21640 1 -- 172.21.15.114:0/2938060401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a382f0 msgr2=0x560150e06730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.717+0000 7f7429d21640 1 --2- 172.21.15.114:0/2938060401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a382f0 0x560150e06730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.717+0000 7f7429d21640 1 -- 172.21.15.114:0/2938060401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a404d0 msgr2=0x560150e02160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.717+0000 7f7429d21640 1 --2- 172.21.15.114:0/2938060401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a404d0 0x560150e02160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.717+0000 7f7429d21640 1 -- 172.21.15.114:0/2938060401 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560150dfa2d0 con 0x560150a611b0 2024-05-27T07:45:02.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.717+0000 7f7428d1f640 1 --2- 172.21.15.114:0/2938060401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a404d0 0x560150e02160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:02.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.717+0000 7f7429d21640 1 --2- 172.21.15.114:0/2938060401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e026a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f741c0c8040 tx=0x7f741c0ae5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb0f8ac9e6873fa4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:02.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.717+0000 7f740effd640 1 -- 172.21.15.114:0/2938060401 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c06d070 con 0x560150a611b0 2024-05-27T07:45:02.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.717+0000 7f740effd640 1 -- 172.21.15.114:0/2938060401 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f741c0a6300 con 0x560150a611b0 2024-05-27T07:45:02.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.717+0000 7f740effd640 1 -- 172.21.15.114:0/2938060401 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c0b2ca0 con 0x560150a611b0 2024-05-27T07:45:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.718+0000 7f743263c740 1 -- 172.21.15.114:0/2938060401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 msgr2=0x560150e026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.718+0000 7f743263c740 1 --2- 172.21.15.114:0/2938060401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e026a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f741c0c8040 tx=0x7f741c0ae5a0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.718+0000 7f743263c740 1 -- 172.21.15.114:0/2938060401 shutdown_connections 2024-05-27T07:45:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.718+0000 7f743263c740 1 --2- 172.21.15.114:0/2938060401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e026a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.718+0000 7f743263c740 1 --2- 172.21.15.114:0/2938060401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a404d0 0x560150e02160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.718+0000 7f743263c740 1 --2- 172.21.15.114:0/2938060401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a382f0 0x560150e06730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.718+0000 7f743263c740 1 -- 172.21.15.114:0/2938060401 >> 172.21.15.114:0/2938060401 conn(0x560150a55300 msgr2=0x560150e06020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.718+0000 7f743263c740 1 -- 172.21.15.114:0/2938060401 shutdown_connections 2024-05-27T07:45:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.718+0000 7f743263c740 1 -- 172.21.15.114:0/2938060401 wait complete. 2024-05-27T07:45:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.718+0000 7f743263c740 1 -- start start 2024-05-27T07:45:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.718+0000 7f743263c740 1 Processor -- start 2024-05-27T07:45:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f743263c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a382f0 0x560150e13ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f743263c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e14400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f743263c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e7f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f743263c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f74241aea50 con 0x560150a382f0 2024-05-27T07:45:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f743263c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f74241ad5e0 con 0x560150a404d0 2024-05-27T07:45:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f743263c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f74241adcb0 con 0x560150a611b0 2024-05-27T07:45:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f743263c740 10 client.0 _ll_register_callbacks cb 0x7f741c0a5f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f7429520640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a382f0 0x560150e13ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f7429d21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e7f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f7429520640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a382f0 0x560150e13ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60774/0 (socket says 172.21.15.114:60774) 2024-05-27T07:45:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f7429520640 1 -- 172.21.15.114:0/3515079131 learned_addr learned my addr 172.21.15.114:0/3515079131 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f7428d1f640 1 --2- 172.21.15.114:0/3515079131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e14400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f7429d21640 1 -- 172.21.15.114:0/3515079131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 msgr2=0x560150e14400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f7429d21640 1 --2- 172.21.15.114:0/3515079131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e14400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f7429d21640 1 -- 172.21.15.114:0/3515079131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a382f0 msgr2=0x560150e13ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f7429d21640 1 --2- 172.21.15.114:0/3515079131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a382f0 0x560150e13ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f7429d21640 1 -- 172.21.15.114:0/3515079131 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560150e01da0 con 0x560150a611b0 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f7428d1f640 1 --2- 172.21.15.114:0/3515079131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e14400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f7429d21640 1 --2- 172.21.15.114:0/3515079131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e7f570 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f741c0a9d70 tx=0x7f741c0b2c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6ae21f389b8c72c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.719+0000 7f740cff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f740cff9640 1 -- 172.21.15.114:0/3515079131 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c06d030 con 0x560150a611b0 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f740cff9640 1 -- 172.21.15.114:0/3515079131 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f741c0a5bf0 con 0x560150a611b0 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f743263c740 1 -- 172.21.15.114:0/3515079131 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f741c0a6300 con 0x560150a611b0 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f740cff9640 1 -- 172.21.15.114:0/3515079131 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f741c0a97e0 con 0x560150a611b0 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f743263c740 10 client.26131 fetch_fsmap learned FSMap version 32 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f743263c740 1 -- 172.21.15.114:0/3515079131 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560150e01da0 con 0x560150a611b0 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f740cff9640 1 -- 172.21.15.114:0/3515079131 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f741c0a6300 con 0x560150a611b0 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f740cff9640 1 -- 172.21.15.114:0/3515079131 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 32) ==== 7725+0+0 (secure 0 0 0) 0x7f741c069bb0 con 0x560150a611b0 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f743263c740 10 client.26131 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f743263c740 10 client.26131 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f743263c740 20 client.26131 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f743263c740 1 --2- 172.21.15.114:0/3515079131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5601509f51c0 0x560150e88f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f743263c740 4 client.26131 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:45:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.720+0000 7f743263c740 1 -- 172.21.15.114:0/3515079131 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560150d70360 con 0x5601509f51c0 2024-05-27T07:45:02.753 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.754+0000 7f7429520640 1 --2- 172.21.15.114:0/3515079131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5601509f51c0 0x560150e88f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:02.754 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.754+0000 7f7429520640 1 --2- 172.21.15.114:0/3515079131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5601509f51c0 0x560150e88f20 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f209baadf15acab4 server_cookie=9e138f1933e3d790 in_seq=0 out_seq=0 2024-05-27T07:45:02.754 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:02.754+0000 7f740cff9640 10 client.26131 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:03.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f740cff9640 1 -- 172.21.15.114:0/3515079131 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x560150d70360 con 0x5601509f51c0 2024-05-27T07:45:03.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f740cff9640 10 client.26131 handle_command_reply: tid=0 2024-05-27T07:45:03.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f743263c740 1 client.26131 shutdown 2024-05-27T07:45:03.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f743263c740 1 -- 172.21.15.114:0/3515079131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 msgr2=0x560150e7f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:03.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f743263c740 1 --2- 172.21.15.114:0/3515079131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e7f570 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f741c0a9d70 tx=0x7f741c0b2c50 comp rx=0 tx=0).stop 2024-05-27T07:45:03.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f743263c740 1 -- 172.21.15.114:0/3515079131 shutdown_connections 2024-05-27T07:45:03.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f743263c740 1 --2- 172.21.15.114:0/3515079131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5601509f51c0 0x560150e88f20 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f743263c740 1 --2- 172.21.15.114:0/3515079131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560150a611b0 0x560150e7f570 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f740cff9640 0 client.26131 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:03.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f743263c740 1 --2- 172.21.15.114:0/3515079131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560150a404d0 0x560150e14400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f743263c740 1 --2- 172.21.15.114:0/3515079131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560150a382f0 0x560150e13ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f743263c740 1 -- 172.21.15.114:0/3515079131 >> 172.21.15.114:0/3515079131 conn(0x560150a55300 msgr2=0x560150e02160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:03.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f743263c740 1 -- 172.21.15.114:0/3515079131 shutdown_connections 2024-05-27T07:45:03.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f743263c740 1 -- 172.21.15.114:0/3515079131 wait complete. 2024-05-27T07:45:03.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.119+0000 7f743263c740 20 client.26131 trim_cache size 0 max 16384 2024-05-27T07:45:03.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:45:03.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:45:03.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:45:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.120+0000 7f742b784640 1 -- 172.21.15.114:0/2146789481 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f74100b2950 msgr2=0x7f74100b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.120+0000 7f742b784640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f74100b2950 0x7f74100b4e60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x560150dffb10 tx=0x7f7420032ed0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.120+0000 7f742b784640 1 -- 172.21.15.114:0/2146789481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7424109040 msgr2=0x7f742419c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.120+0000 7f742b784640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7424109040 0x7f742419c170 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f741c009fd0 tx=0x7f741c00ef10 comp rx=0 tx=0).stop 2024-05-27T07:45:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.120+0000 7f742b784640 1 -- 172.21.15.114:0/2146789481 shutdown_connections 2024-05-27T07:45:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.120+0000 7f742b784640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f74100b2950 0x7f74100b4e60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.120+0000 7f742b784640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7424109040 0x7f742419c170 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.120+0000 7f742b784640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7424108ba0 0x7f742419bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.120+0000 7f742b784640 1 --2- 172.21.15.114:0/2146789481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74241032f0 0x7f742419b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.120+0000 7f742b784640 1 -- 172.21.15.114:0/2146789481 >> 172.21.15.114:0/2146789481 conn(0x7f7424078270 msgr2=0x7f7424104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.120+0000 7f742b784640 1 -- 172.21.15.114:0/2146789481 shutdown_connections 2024-05-27T07:45:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:03.120+0000 7f742b784640 1 -- 172.21.15.114:0/2146789481 wait complete. 2024-05-27T07:45:03.247 INFO:tasks.workunit.client.0.smithi114.stdout: 1 490784 13.26 MB/sec execute 75 sec latency 285.789 ms 2024-05-27T07:45:03.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.373+0000 7f7c99529700 1 -- 172.21.15.114:0/1020702339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c94108730 msgr2=0x7f7c94110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:03.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.373+0000 7f7c99529700 1 --2- 172.21.15.114:0/1020702339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c94108730 0x7f7c94110090 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c00b0d0 tx=0x7f7c8c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.373+0000 7f7c99529700 1 -- 172.21.15.114:0/1020702339 shutdown_connections 2024-05-27T07:45:03.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.373+0000 7f7c99529700 1 --2- 172.21.15.114:0/1020702339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c94108730 0x7f7c94110090 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.373+0000 7f7c99529700 1 --2- 172.21.15.114:0/1020702339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c941034b0 0x7f7c941038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.373+0000 7f7c99529700 1 --2- 172.21.15.114:0/1020702339 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c94107de0 0x7f7c941081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.373+0000 7f7c99529700 1 -- 172.21.15.114:0/1020702339 >> 172.21.15.114:0/1020702339 conn(0x7f7c940fedb0 msgr2=0x7f7c941011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:03.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.373+0000 7f7c99529700 1 -- 172.21.15.114:0/1020702339 shutdown_connections 2024-05-27T07:45:03.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.373+0000 7f7c99529700 1 -- 172.21.15.114:0/1020702339 wait complete. 2024-05-27T07:45:03.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:03 smithi114 ceph-mon[32038]: pgmap v167: 129 pgs: 129 active+clean; 54 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 3.8 MiB/s wr, 350 op/s 2024-05-27T07:45:03.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.374+0000 7f7c99529700 1 Processor -- start 2024-05-27T07:45:03.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.375+0000 7f7c99529700 1 -- start start 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.375+0000 7f7c99529700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c941034b0 0x7f7c9419ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.375+0000 7f7c99529700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c94107de0 0x7f7c9419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.375+0000 7f7c99529700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c94108730 0x7f7c9419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.375+0000 7f7c99529700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7c94113360 con 0x7f7c94107de0 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.375+0000 7f7c927fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c94107de0 0x7f7c9419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.375+0000 7f7c92ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c941034b0 0x7f7c9419ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.375+0000 7f7c99529700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7c941131e0 con 0x7f7c94108730 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.375+0000 7f7c92ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c941034b0 0x7f7c9419ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:56156/0 (socket says 172.21.15.114:56156) 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.375+0000 7f7c92ffd700 1 -- 172.21.15.114:0/459794266 learned_addr learned my addr 172.21.15.114:0/459794266 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.375+0000 7f7c99529700 1 -- 172.21.15.114:0/459794266 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c941134e0 con 0x7f7c941034b0 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.375+0000 7f7c937fe700 1 --2- 172.21.15.114:0/459794266 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c94108730 0x7f7c9419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.376+0000 7f7c92ffd700 1 -- 172.21.15.114:0/459794266 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c94108730 msgr2=0x7f7c9419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.376+0000 7f7c92ffd700 1 --2- 172.21.15.114:0/459794266 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c94108730 0x7f7c9419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.376+0000 7f7c92ffd700 1 -- 172.21.15.114:0/459794266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c94107de0 msgr2=0x7f7c9419b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:03.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.376+0000 7f7c92ffd700 1 --2- 172.21.15.114:0/459794266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c94107de0 0x7f7c9419b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.376+0000 7f7c92ffd700 1 -- 172.21.15.114:0/459794266 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c9419fc50 con 0x7f7c941034b0 2024-05-27T07:45:03.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.376+0000 7f7c92ffd700 1 --2- 172.21.15.114:0/459794266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c941034b0 0x7f7c9419ad80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7c8400e9a0 tx=0x7f7c8400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:03.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.376+0000 7f7c7bfff700 1 -- 172.21.15.114:0/459794266 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8400ca10 con 0x7f7c941034b0 2024-05-27T07:45:03.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.376+0000 7f7c7bfff700 1 -- 172.21.15.114:0/459794266 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c8400cbb0 con 0x7f7c941034b0 2024-05-27T07:45:03.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.376+0000 7f7c99529700 1 -- 172.21.15.114:0/459794266 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c9419ff40 con 0x7f7c941034b0 2024-05-27T07:45:03.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.376+0000 7f7c7bfff700 1 -- 172.21.15.114:0/459794266 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84018680 con 0x7f7c941034b0 2024-05-27T07:45:03.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.376+0000 7f7c99529700 1 -- 172.21.15.114:0/459794266 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c941a7860 con 0x7f7c941034b0 2024-05-27T07:45:03.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.379+0000 7f7c99529700 1 -- 172.21.15.114:0/459794266 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c940512b0 con 0x7f7c941034b0 2024-05-27T07:45:03.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.379+0000 7f7c7bfff700 1 -- 172.21.15.114:0/459794266 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f7c84010430 con 0x7f7c941034b0 2024-05-27T07:45:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.379+0000 7f7c7bfff700 1 --2- 172.21.15.114:0/459794266 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7c7c0b2a70 0x7f7c7c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.380+0000 7f7c927fc700 1 --2- 172.21.15.114:0/459794266 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7c7c0b2a70 0x7f7c7c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.380+0000 7f7c7bfff700 1 -- 172.21.15.114:0/459794266 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f7c840e2ac0 con 0x7f7c941034b0 2024-05-27T07:45:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.380+0000 7f7c927fc700 1 --2- 172.21.15.114:0/459794266 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7c7c0b2a70 0x7f7c7c0b4f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7c94109820 tx=0x7f7c88009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:03.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.384+0000 7f7c7bfff700 1 -- 172.21.15.114:0/459794266 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7c840aa0f0 con 0x7f7c941034b0 2024-05-27T07:45:03.564 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:03 smithi186 ceph-mon[36647]: pgmap v167: 129 pgs: 129 active+clean; 54 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 3.8 MiB/s wr, 350 op/s 2024-05-27T07:45:03.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.658+0000 7f7c99529700 1 -- 172.21.15.114:0/459794266 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c9402e5e0 con 0x7f7c941034b0 2024-05-27T07:45:03.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.659+0000 7f7c7bfff700 1 -- 172.21.15.114:0/459794266 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+10999 (secure 0 0 0) 0x7f7c840b2170 con 0x7f7c941034b0 2024-05-27T07:45:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:45:03.659 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:44:46.296164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26035":{"gid":26035,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3710253414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3710253414},{"type":"v1","addr":"172.21.15.186:6837","nonce":3710253414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:45:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.662+0000 7f7c99529700 1 -- 172.21.15.114:0/459794266 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7c7c0b2a70 msgr2=0x7f7c7c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.662+0000 7f7c99529700 1 --2- 172.21.15.114:0/459794266 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7c7c0b2a70 0x7f7c7c0b4f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7c94109820 tx=0x7f7c88009260 comp rx=0 tx=0).stop 2024-05-27T07:45:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.662+0000 7f7c99529700 1 -- 172.21.15.114:0/459794266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c941034b0 msgr2=0x7f7c9419ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.662+0000 7f7c99529700 1 --2- 172.21.15.114:0/459794266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c941034b0 0x7f7c9419ad80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7c8400e9a0 tx=0x7f7c8400ecb0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.662+0000 7f7c99529700 1 -- 172.21.15.114:0/459794266 shutdown_connections 2024-05-27T07:45:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.662+0000 7f7c99529700 1 --2- 172.21.15.114:0/459794266 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7c7c0b2a70 0x7f7c7c0b4f80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.662+0000 7f7c99529700 1 --2- 172.21.15.114:0/459794266 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c94108730 0x7f7c9419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.662+0000 7f7c99529700 1 --2- 172.21.15.114:0/459794266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c941034b0 0x7f7c9419ad80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.662+0000 7f7c99529700 1 --2- 172.21.15.114:0/459794266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c94107de0 0x7f7c9419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:03.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.662+0000 7f7c99529700 1 -- 172.21.15.114:0/459794266 >> 172.21.15.114:0/459794266 conn(0x7f7c940fedb0 msgr2=0x7f7c9406b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:03.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.662+0000 7f7c99529700 1 -- 172.21.15.114:0/459794266 shutdown_connections 2024-05-27T07:45:03.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:03.662+0000 7f7c99529700 1 -- 172.21.15.114:0/459794266 wait complete. 2024-05-27T07:45:03.662 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 32 2024-05-27T07:45:03.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:03 smithi163 ceph-mon[36451]: pgmap v167: 129 pgs: 129 active+clean; 54 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 3.8 MiB/s wr, 350 op/s 2024-05-27T07:45:03.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:03 smithi186 podman[72363]: 2024-05-27 07:45:03.566145454 +0000 UTC m=+3.202618752 container init 6f41e07e18921c088441505e2e68f1fc482aa4fdcb271770fec6508109065adf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:45:03.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:03 smithi186 podman[72363]: 2024-05-27 07:45:03.568873417 +0000 UTC m=+3.205346710 container start 6f41e07e18921c088441505e2e68f1fc482aa4fdcb271770fec6508109065adf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-27T07:45:03.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:03 smithi186 ceph-mds[72431]: -- 172.21.15.186:0/2526780147 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5566015fba40 con 0x556601643600 2024-05-27T07:45:03.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:03 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: starting mds.c at 2024-05-27T07:45:04.247 INFO:tasks.workunit.client.0.smithi114.stdout: 1 491492 13.11 MB/sec execute 76 sec latency 195.995 ms 2024-05-27T07:45:04.557 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-mon[36647]: daemon mds.c restarted 2024-05-27T07:45:04.557 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/459794266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:45:04.557 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-mon[36647]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T07:45:04.557 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6836/2533834448,v1:172.21.15.186:6837/2533834448] up:boot 2024-05-27T07:45:04.557 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:45:04.557 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:45:04.557 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fc062328700 time 2024-05-27T07:45:04.262821+0000 2024-05-27T07:45:04.558 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:45:04.558 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:04.558 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc074b6dba6] 2024-05-27T07:45:04.558 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc074b6ddd1] 2024-05-27T07:45:04.558 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5565feaae23d] 2024-05-27T07:45:04.558 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5565fea99971] 2024-05-27T07:45:04.558 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 5: (SnapRealm::check_cache() const+0x13e) [0x5565fea97fa0] 2024-05-27T07:45:04.558 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5565fea98651] 2024-05-27T07:45:04.558 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5565fea2d347] 2024-05-27T07:45:04.558 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5565fe72987e] 2024-05-27T07:45:04.558 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5565fe72ef2f] 2024-05-27T07:45:04.558 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5565fe739969] 2024-05-27T07:45:04.558 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 11: (MDLog::_replay_thread()+0x1f38) [0x5565fead248c] 2024-05-27T07:45:04.559 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5565feadb9cb] 2024-05-27T07:45:04.559 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 13: (Thread::entry_wrapper()+0x43) [0x7fc074b3c72d] 2024-05-27T07:45:04.559 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 14: (Thread::_entry_func(void*)+0xd) [0x7fc074b3c749] 2024-05-27T07:45:04.559 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc0736e01ca] 2024-05-27T07:45:04.559 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 16: clone() 2024-05-27T07:45:04.559 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.264+0000 7fc062328700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fc062328700 time 2024-05-27T07:45:04.262821+0000 2024-05-27T07:45:04.560 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:45:04.560 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.560 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:04.560 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc074b6dba6] 2024-05-27T07:45:04.560 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc074b6ddd1] 2024-05-27T07:45:04.560 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5565feaae23d] 2024-05-27T07:45:04.560 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5565fea99971] 2024-05-27T07:45:04.560 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 5: (SnapRealm::check_cache() const+0x13e) [0x5565fea97fa0] 2024-05-27T07:45:04.560 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5565fea98651] 2024-05-27T07:45:04.560 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5565fea2d347] 2024-05-27T07:45:04.561 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5565fe72987e] 2024-05-27T07:45:04.561 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5565fe72ef2f] 2024-05-27T07:45:04.561 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5565fe739969] 2024-05-27T07:45:04.561 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 11: (MDLog::_replay_thread()+0x1f38) [0x5565fead248c] 2024-05-27T07:45:04.561 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5565feadb9cb] 2024-05-27T07:45:04.561 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 13: (Thread::entry_wrapper()+0x43) [0x7fc074b3c72d] 2024-05-27T07:45:04.561 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 14: (Thread::_entry_func(void*)+0xd) [0x7fc074b3c749] 2024-05-27T07:45:04.561 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc0736e01ca] 2024-05-27T07:45:04.561 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 16: clone() 2024-05-27T07:45:04.561 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.561 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: *** Caught signal (Aborted) ** 2024-05-27T07:45:04.561 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: in thread 7fc062328700 thread_name:md_log_replay 2024-05-27T07:45:04.561 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc0736ead20] 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2: gsignal() 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 3: abort() 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc074b6dced] 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc074b6ddd1] 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5565feaae23d] 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5565fea99971] 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 8: (SnapRealm::check_cache() const+0x13e) [0x5565fea97fa0] 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5565fea98651] 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5565fea2d347] 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5565fe72987e] 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5565fe72ef2f] 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5565fe739969] 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 14: (MDLog::_replay_thread()+0x1f38) [0x5565fead248c] 2024-05-27T07:45:04.562 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5565feadb9cb] 2024-05-27T07:45:04.563 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 16: (Thread::entry_wrapper()+0x43) [0x7fc074b3c72d] 2024-05-27T07:45:04.563 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 17: (Thread::_entry_func(void*)+0xd) [0x7fc074b3c749] 2024-05-27T07:45:04.563 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc0736e01ca] 2024-05-27T07:45:04.563 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 19: clone() 2024-05-27T07:45:04.563 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.266+0000 7fc062328700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:45:04.563 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: in thread 7fc062328700 thread_name:md_log_replay 2024-05-27T07:45:04.563 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.563 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:04.563 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc0736ead20] 2024-05-27T07:45:04.563 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2: gsignal() 2024-05-27T07:45:04.563 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 3: abort() 2024-05-27T07:45:04.563 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc074b6dced] 2024-05-27T07:45:04.563 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc074b6ddd1] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5565feaae23d] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5565fea99971] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 8: (SnapRealm::check_cache() const+0x13e) [0x5565fea97fa0] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5565fea98651] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5565fea2d347] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5565fe72987e] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5565fe72ef2f] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5565fe739969] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 14: (MDLog::_replay_thread()+0x1f38) [0x5565fead248c] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5565feadb9cb] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 16: (Thread::entry_wrapper()+0x43) [0x7fc074b3c72d] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 17: (Thread::_entry_func(void*)+0xd) [0x7fc074b3c749] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc0736e01ca] 2024-05-27T07:45:04.564 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 19: clone() 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: -1> 2024-05-27T07:45:04.264+0000 7fc062328700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fc062328700 time 2024-05-27T07:45:04.262821+0000 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc074b6dba6] 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc074b6ddd1] 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5565feaae23d] 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5565fea99971] 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 5: (SnapRealm::check_cache() const+0x13e) [0x5565fea97fa0] 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5565fea98651] 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5565fea2d347] 2024-05-27T07:45:04.565 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5565fe72987e] 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5565fe72ef2f] 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5565fe739969] 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 11: (MDLog::_replay_thread()+0x1f38) [0x5565fead248c] 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5565feadb9cb] 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 13: (Thread::entry_wrapper()+0x43) [0x7fc074b3c72d] 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 14: (Thread::_entry_func(void*)+0xd) [0x7fc074b3c749] 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc0736e01ca] 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 16: clone() 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 0> 2024-05-27T07:45:04.266+0000 7fc062328700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: in thread 7fc062328700 thread_name:md_log_replay 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:04.566 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc0736ead20] 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2: gsignal() 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 3: abort() 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc074b6dced] 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc074b6ddd1] 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5565feaae23d] 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5565fea99971] 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 8: (SnapRealm::check_cache() const+0x13e) [0x5565fea97fa0] 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5565fea98651] 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5565fea2d347] 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5565fe72987e] 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5565fe72ef2f] 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5565fe739969] 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 14: (MDLog::_replay_thread()+0x1f38) [0x5565fead248c] 2024-05-27T07:45:04.567 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5565feadb9cb] 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 16: (Thread::entry_wrapper()+0x43) [0x7fc074b3c72d] 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 17: (Thread::_entry_func(void*)+0xd) [0x7fc074b3c749] 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc0736e01ca] 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 19: clone() 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: -1> 2024-05-27T07:45:04.264+0000 7fc062328700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fc062328700 time 2024-05-27T07:45:04.262821+0000 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc074b6dba6] 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc074b6ddd1] 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5565feaae23d] 2024-05-27T07:45:04.568 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5565fea99971] 2024-05-27T07:45:04.569 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 5: (SnapRealm::check_cache() const+0x13e) [0x5565fea97fa0] 2024-05-27T07:45:04.569 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5565fea98651] 2024-05-27T07:45:04.569 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5565fea2d347] 2024-05-27T07:45:04.569 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5565fe72987e] 2024-05-27T07:45:04.569 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5565fe72ef2f] 2024-05-27T07:45:04.569 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5565fe739969] 2024-05-27T07:45:04.569 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 11: (MDLog::_replay_thread()+0x1f38) [0x5565fead248c] 2024-05-27T07:45:04.569 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5565feadb9cb] 2024-05-27T07:45:04.569 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 13: (Thread::entry_wrapper()+0x43) [0x7fc074b3c72d] 2024-05-27T07:45:04.569 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 14: (Thread::_entry_func(void*)+0xd) [0x7fc074b3c749] 2024-05-27T07:45:04.569 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc0736e01ca] 2024-05-27T07:45:04.569 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 16: clone() 2024-05-27T07:45:04.570 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.570 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 0> 2024-05-27T07:45:04.266+0000 7fc062328700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:45:04.570 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: in thread 7fc062328700 thread_name:md_log_replay 2024-05-27T07:45:04.570 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.570 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:04.570 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc0736ead20] 2024-05-27T07:45:04.570 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2: gsignal() 2024-05-27T07:45:04.570 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 3: abort() 2024-05-27T07:45:04.570 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc074b6dced] 2024-05-27T07:45:04.570 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc074b6ddd1] 2024-05-27T07:45:04.570 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5565feaae23d] 2024-05-27T07:45:04.570 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5565fea99971] 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 8: (SnapRealm::check_cache() const+0x13e) [0x5565fea97fa0] 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5565fea98651] 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5565fea2d347] 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5565fe72987e] 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5565fe72ef2f] 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5565fe739969] 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 14: (MDLog::_replay_thread()+0x1f38) [0x5565fead248c] 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5565feadb9cb] 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 16: (Thread::entry_wrapper()+0x43) [0x7fc074b3c72d] 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 17: (Thread::_entry_func(void*)+0xd) [0x7fc074b3c749] 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc0736e01ca] 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 19: clone() 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:45:04.571 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72427]: 2024-05-27T07:45:04.572 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 bash[72363]: 6f41e07e18921c088441505e2e68f1fc482aa4fdcb271770fec6508109065adf 2024-05-27T07:45:04.572 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 systemd[1]: Started Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:45:04.680 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:04 smithi114 ceph-mon[32038]: daemon mds.c restarted 2024-05-27T07:45:04.680 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:04 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/459794266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:45:04.680 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:04 smithi114 ceph-mon[32038]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T07:45:04.680 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:04 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6836/2533834448,v1:172.21.15.186:6837/2533834448] up:boot 2024-05-27T07:45:04.680 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:04 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:45:04.680 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:04 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:45:04.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:45:04.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:04 smithi163 ceph-mon[36451]: daemon mds.c restarted 2024-05-27T07:45:04.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:04 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/459794266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:45:04.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:04 smithi163 ceph-mon[36451]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T07:45:04.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:04 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6836/2533834448,v1:172.21.15.186:6837/2533834448] up:boot 2024-05-27T07:45:04.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:04 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:45:04.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:04 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:45:04.970 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:04.977 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:04 smithi186 podman[72462]: 2024-05-27 07:45:04.639323348 +0000 UTC m=+0.349874894 container died 6f41e07e18921c088441505e2e68f1fc482aa4fdcb271770fec6508109065adf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:45:05.247 INFO:tasks.workunit.client.0.smithi114.stdout: 1 494326 13.16 MB/sec execute 77 sec latency 85.289 ms 2024-05-27T07:45:05.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:05 smithi114 ceph-mon[32038]: pgmap v169: 129 pgs: 129 active+clean; 54 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 4.1 MiB/s wr, 304 op/s 2024-05-27T07:45:05.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:05 smithi163 ceph-mon[36451]: pgmap v169: 129 pgs: 129 active+clean; 54 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 4.1 MiB/s wr, 304 op/s 2024-05-27T07:45:05.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:05 smithi186 ceph-mon[36647]: pgmap v169: 129 pgs: 129 active+clean; 54 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 4.1 MiB/s wr, 304 op/s 2024-05-27T07:45:06.247 INFO:tasks.workunit.client.0.smithi114.stdout: 1 496782 13.15 MB/sec execute 78 sec latency 80.856 ms 2024-05-27T07:45:06.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.698+0000 7f51df756700 1 -- 172.21.15.114:0/4242758125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d8108730 msgr2=0x7f51d8110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:06.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.698+0000 7f51df756700 1 --2- 172.21.15.114:0/4242758125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d8108730 0x7f51d8110090 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f51d000b0d0 tx=0x7f51d000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.698+0000 7f51df756700 1 -- 172.21.15.114:0/4242758125 shutdown_connections 2024-05-27T07:45:06.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.698+0000 7f51df756700 1 --2- 172.21.15.114:0/4242758125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d8108730 0x7f51d8110090 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.698+0000 7f51df756700 1 --2- 172.21.15.114:0/4242758125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51d81034b0 0x7f51d81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.698+0000 7f51df756700 1 --2- 172.21.15.114:0/4242758125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51d8107de0 0x7f51d81081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.698+0000 7f51df756700 1 -- 172.21.15.114:0/4242758125 >> 172.21.15.114:0/4242758125 conn(0x7f51d80fede0 msgr2=0x7f51d8101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:06.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.698+0000 7f51df756700 1 -- 172.21.15.114:0/4242758125 shutdown_connections 2024-05-27T07:45:06.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.698+0000 7f51df756700 1 -- 172.21.15.114:0/4242758125 wait complete. 2024-05-27T07:45:06.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.699+0000 7f51df756700 1 Processor -- start 2024-05-27T07:45:06.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.699+0000 7f51df756700 1 -- start start 2024-05-27T07:45:06.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.699+0000 7f51df756700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51d81034b0 0x7f51d819f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.699+0000 7f51df756700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51d8107de0 0x7f51d819fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.699+0000 7f51df756700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d8108730 0x7f51d8199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.699+0000 7f51df756700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81133b0 con 0x7f51d8108730 2024-05-27T07:45:06.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.699+0000 7f51df756700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f51d8113230 con 0x7f51d81034b0 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.699+0000 7f51df756700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51d8113530 con 0x7f51d8107de0 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.699+0000 7f51ddcf3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d8108730 0x7f51d8199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.699+0000 7f51dccf1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51d8107de0 0x7f51d819fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.699+0000 7f51ddcf3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d8108730 0x7f51d8199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57438/0 (socket says 172.21.15.114:57438) 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51ddcf3700 1 -- 172.21.15.114:0/2095618724 learned_addr learned my addr 172.21.15.114:0/2095618724 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51d81034b0 0x7f51d819f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51dccf1700 1 -- 172.21.15.114:0/2095618724 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51d81034b0 msgr2=0x7f51d819f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51dccf1700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51d81034b0 0x7f51d819f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51dccf1700 1 -- 172.21.15.114:0/2095618724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d8108730 msgr2=0x7f51d8199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51dccf1700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d8108730 0x7f51d8199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51dccf1700 1 -- 172.21.15.114:0/2095618724 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d819a1b0 con 0x7f51d8107de0 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51ddcf3700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d8108730 0x7f51d8199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51dccf1700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51d8107de0 0x7f51d819fdf0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f51c800e9c0 tx=0x7f51c800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51ce7fc700 1 -- 172.21.15.114:0/2095618724 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c800c9c0 con 0x7f51d8107de0 2024-05-27T07:45:06.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51df756700 1 -- 172.21.15.114:0/2095618724 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d819a4a0 con 0x7f51d8107de0 2024-05-27T07:45:06.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51ce7fc700 1 -- 172.21.15.114:0/2095618724 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51c800cb60 con 0x7f51d8107de0 2024-05-27T07:45:06.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51ce7fc700 1 -- 172.21.15.114:0/2095618724 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c8018680 con 0x7f51d8107de0 2024-05-27T07:45:06.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.700+0000 7f51df756700 1 -- 172.21.15.114:0/2095618724 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d81a6740 con 0x7f51d8107de0 2024-05-27T07:45:06.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.703+0000 7f51ce7fc700 1 -- 172.21.15.114:0/2095618724 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f51c8010430 con 0x7f51d8107de0 2024-05-27T07:45:06.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.704+0000 7f51ce7fc700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f51c40b29f0 0x7f51c40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.704+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f51c40b29f0 0x7f51c40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.704+0000 7f51ce7fc700 1 -- 172.21.15.114:0/2095618724 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f51c80e2b30 con 0x7f51d8107de0 2024-05-27T07:45:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.704+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f51c40b29f0 0x7f51c40b4f00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f51d819ac10 tx=0x7f51d4009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.704+0000 7f51ee16bb80 1 Processor -- start 2024-05-27T07:45:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.704+0000 7f51ee16bb80 1 -- start start 2024-05-27T07:45:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51ee16bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bbaee40 0x561f9bd349a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51ee16bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd34ee0 0x561f9bd2ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51ee16bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bd2f190 0x561f9bd2f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51ee16bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561f9bc5bc90 con 0x561f9bd2f190 2024-05-27T07:45:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51ee16bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561f9bc4b830 con 0x561f9bd34ee0 2024-05-27T07:45:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51ee16bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561f9bc4f870 con 0x561f9bbaee40 2024-05-27T07:45:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51dd4f2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bbaee40 0x561f9bd349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51dd4f2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bbaee40 0x561f9bd349a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:42528/0 (socket says 172.21.15.114:42528) 2024-05-27T07:45:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51dd4f2700 1 -- 172.21.15.114:0/3013824957 learned_addr learned my addr 172.21.15.114:0/3013824957 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51ddcf3700 1 --2- 172.21.15.114:0/3013824957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bd2f190 0x561f9bd2f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51dccf1700 1 --2- 172.21.15.114:0/3013824957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd34ee0 0x561f9bd2ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51dd4f2700 1 -- 172.21.15.114:0/3013824957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd34ee0 msgr2=0x561f9bd2ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/3013824957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd34ee0 0x561f9bd2ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51dd4f2700 1 -- 172.21.15.114:0/3013824957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bd2f190 msgr2=0x561f9bd2f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/3013824957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bd2f190 0x561f9bd2f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51dd4f2700 1 -- 172.21.15.114:0/3013824957 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f9bb9b8d0 con 0x561f9bbaee40 2024-05-27T07:45:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51ddcf3700 1 --2- 172.21.15.114:0/3013824957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bd2f190 0x561f9bd2f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.705+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/3013824957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bbaee40 0x561f9bd349a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f51d4011d50 tx=0x7f51d4014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29a509bdfdb1b8f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.706+0000 7f51c37fe700 1 -- 172.21.15.114:0/3013824957 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d400eb20 con 0x561f9bbaee40 2024-05-27T07:45:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.706+0000 7f51c37fe700 1 -- 172.21.15.114:0/3013824957 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d40044d0 con 0x561f9bbaee40 2024-05-27T07:45:06.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.706+0000 7f51c37fe700 1 -- 172.21.15.114:0/3013824957 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d401c410 con 0x561f9bbaee40 2024-05-27T07:45:06.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.707+0000 7f51ee16bb80 1 -- 172.21.15.114:0/3013824957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bbaee40 msgr2=0x561f9bd349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:06.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.707+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/3013824957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bbaee40 0x561f9bd349a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f51d4011d50 tx=0x7f51d4014000 comp rx=0 tx=0).stop 2024-05-27T07:45:06.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.707+0000 7f51ee16bb80 1 -- 172.21.15.114:0/3013824957 shutdown_connections 2024-05-27T07:45:06.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.707+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/3013824957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bd2f190 0x561f9bd2f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.707+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/3013824957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bbaee40 0x561f9bd349a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.707+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/3013824957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd34ee0 0x561f9bd2ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.707+0000 7f51ee16bb80 1 -- 172.21.15.114:0/3013824957 >> 172.21.15.114:0/3013824957 conn(0x561f9bc09af0 msgr2=0x561f9bd27640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.707+0000 7f51ee16bb80 1 -- 172.21.15.114:0/3013824957 shutdown_connections 2024-05-27T07:45:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.707+0000 7f51ee16bb80 1 -- 172.21.15.114:0/3013824957 wait complete. 2024-05-27T07:45:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.707+0000 7f51ee16bb80 1 -- start start 2024-05-27T07:45:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.707+0000 7f51ee16bb80 1 Processor -- start 2024-05-27T07:45:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ee16bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bbaee40 0x561f9bd385e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ee16bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd2f190 0x561f9bd38b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ee16bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bd34ee0 0x561f9bd39060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ee16bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561f9bb51fa0 con 0x561f9bd34ee0 2024-05-27T07:45:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ee16bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561f9bbed7c0 con 0x561f9bd2f190 2024-05-27T07:45:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ee16bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561f9bd3b910 con 0x561f9bbaee40 2024-05-27T07:45:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ee16bb80 10 client.0 _ll_register_callbacks cb 0x561f9bbe6920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ddcf3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bd34ee0 0x561f9bd39060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ddcf3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bd34ee0 0x561f9bd39060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57468/0 (socket says 172.21.15.114:57468) 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51dd4f2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bbaee40 0x561f9bd385e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ddcf3700 1 -- 172.21.15.114:0/2447670143 learned_addr learned my addr 172.21.15.114:0/2447670143 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51dccf1700 1 --2- 172.21.15.114:0/2447670143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd2f190 0x561f9bd38b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ddcf3700 1 -- 172.21.15.114:0/2447670143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bbaee40 msgr2=0x561f9bd385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ddcf3700 1 --2- 172.21.15.114:0/2447670143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bbaee40 0x561f9bd385e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ddcf3700 1 -- 172.21.15.114:0/2447670143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd2f190 msgr2=0x561f9bd38b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ddcf3700 1 --2- 172.21.15.114:0/2447670143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd2f190 0x561f9bd38b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ddcf3700 1 -- 172.21.15.114:0/2447670143 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f9bbea690 con 0x561f9bd34ee0 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/2447670143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bbaee40 0x561f9bd385e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51ddcf3700 1 --2- 172.21.15.114:0/2447670143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bd34ee0 0x561f9bd39060 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f51d0004da0 tx=0x7f51d0004dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7889101efc74b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.708+0000 7f51c0ff9700 1 -- 172.21.15.114:0/2447670143 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d002a060 con 0x561f9bd34ee0 2024-05-27T07:45:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51ee16bb80 1 -- 172.21.15.114:0/2447670143 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51d40044d0 con 0x561f9bd34ee0 2024-05-27T07:45:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51c0ff9700 1 -- 172.21.15.114:0/2447670143 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f51d00077d0 con 0x561f9bd34ee0 2024-05-27T07:45:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51c0ff9700 1 -- 172.21.15.114:0/2447670143 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f51d0031050 con 0x561f9bd34ee0 2024-05-27T07:45:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51ee16bb80 10 client.16563 fetch_fsmap learned FSMap version 33 2024-05-27T07:45:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51ee16bb80 1 -- 172.21.15.114:0/2447670143 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561f9bd463c0 con 0x561f9bd34ee0 2024-05-27T07:45:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51c0ff9700 1 -- 172.21.15.114:0/2447670143 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d40044d0 con 0x561f9bd34ee0 2024-05-27T07:45:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51c0ff9700 1 -- 172.21.15.114:0/2447670143 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 33) ==== 7725+0+0 (secure 0 0 0) 0x7f51d0011070 con 0x561f9bd34ee0 2024-05-27T07:45:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51ee16bb80 10 client.16563 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T07:45:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51ee16bb80 10 client.16563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51ee16bb80 20 client.16563 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/2447670143 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561f9bd48a00 0x561f9bd4ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51ee16bb80 4 client.16563 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51ee16bb80 1 -- 172.21.15.114:0/2447670143 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f9bcf89f0 con 0x561f9bd48a00 2024-05-27T07:45:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.709+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/2447670143 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561f9bd48a00 0x561f9bd4ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.710+0000 7f51c0ff9700 10 client.16563 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:06.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.710+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/2447670143 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561f9bd48a00 0x561f9bd4ae40 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d37c98d7e461836 server_cookie=7e84c5828376333 in_seq=0 out_seq=0 2024-05-27T07:45:06.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.711+0000 7f51c0ff9700 1 -- 172.21.15.114:0/2447670143 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561f9bcf89f0 con 0x561f9bd48a00 2024-05-27T07:45:06.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.711+0000 7f51c0ff9700 10 client.16563 handle_command_reply: tid=0 2024-05-27T07:45:06.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.712+0000 7f51ee16bb80 1 client.16563 shutdown 2024-05-27T07:45:06.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.712+0000 7f51ee16bb80 1 -- 172.21.15.114:0/2447670143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bd34ee0 msgr2=0x561f9bd39060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:06.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.712+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/2447670143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bd34ee0 0x561f9bd39060 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f51d0004da0 tx=0x7f51d0004dd0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.712+0000 7f51ee16bb80 1 -- 172.21.15.114:0/2447670143 shutdown_connections 2024-05-27T07:45:06.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.712+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/2447670143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bd34ee0 0x561f9bd39060 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.712+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/2447670143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bbaee40 0x561f9bd385e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.712+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/2447670143 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561f9bd48a00 0x561f9bd4ae40 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.712+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/2447670143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd2f190 0x561f9bd38b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.712+0000 7f51c0ff9700 0 client.16563 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:06.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.712+0000 7f51ee16bb80 1 -- 172.21.15.114:0/2447670143 >> 172.21.15.114:0/2447670143 conn(0x561f9bc09af0 msgr2=0x561f9bd2ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:06.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.712+0000 7f51ee16bb80 1 -- 172.21.15.114:0/2447670143 shutdown_connections 2024-05-27T07:45:06.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.712+0000 7f51ee16bb80 1 -- 172.21.15.114:0/2447670143 wait complete. 2024-05-27T07:45:06.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.712+0000 7f51ee16bb80 20 client.16563 trim_cache size 0 max 16384 2024-05-27T07:45:06.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.727+0000 7f51ee16bb80 1 Processor -- start 2024-05-27T07:45:06.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.727+0000 7f51ee16bb80 1 -- start start 2024-05-27T07:45:06.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51ee16bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 0x561f9bd349a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51ee16bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd2f190 0x561f9bd38ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51ee16bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bd34ee0 0x561f9bd44970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51ee16bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81af270 con 0x561f9bbaee40 2024-05-27T07:45:06.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51ee16bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81ac320 con 0x561f9bd2f190 2024-05-27T07:45:06.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51ee16bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81ae4d0 con 0x561f9bd34ee0 2024-05-27T07:45:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51dd4f2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 0x561f9bd349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51dd4f2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 0x561f9bd349a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57480/0 (socket says 172.21.15.114:57480) 2024-05-27T07:45:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51dd4f2700 1 -- 172.21.15.114:0/2160919956 learned_addr learned my addr 172.21.15.114:0/2160919956 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51dccf1700 1 --2- 172.21.15.114:0/2160919956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd2f190 0x561f9bd38ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51ddcf3700 1 --2- 172.21.15.114:0/2160919956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bd34ee0 0x561f9bd44970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51dd4f2700 1 -- 172.21.15.114:0/2160919956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bd34ee0 msgr2=0x561f9bd44970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/2160919956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bd34ee0 0x561f9bd44970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51dd4f2700 1 -- 172.21.15.114:0/2160919956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd2f190 msgr2=0x561f9bd38ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/2160919956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd2f190 0x561f9bd38ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51dd4f2700 1 -- 172.21.15.114:0/2160919956 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f9bd2f960 con 0x561f9bbaee40 2024-05-27T07:45:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.728+0000 7f51ddcf3700 1 --2- 172.21.15.114:0/2160919956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bd34ee0 0x561f9bd44970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:45:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.729+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/2160919956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 0x561f9bd349a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x561f9bd282c0 tx=0x7f51d4029040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0d92b1b7ff9809 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:06.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.729+0000 7f51c2ffd700 1 -- 172.21.15.114:0/2160919956 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4003840 con 0x561f9bbaee40 2024-05-27T07:45:06.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.729+0000 7f51c2ffd700 1 -- 172.21.15.114:0/2160919956 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d400ed20 con 0x561f9bbaee40 2024-05-27T07:45:06.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.729+0000 7f51c2ffd700 1 -- 172.21.15.114:0/2160919956 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d401c970 con 0x561f9bbaee40 2024-05-27T07:45:06.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.730+0000 7f51ee16bb80 1 -- 172.21.15.114:0/2160919956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 msgr2=0x561f9bd349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:06.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.730+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/2160919956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 0x561f9bd349a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x561f9bd282c0 tx=0x7f51d4029040 comp rx=0 tx=0).stop 2024-05-27T07:45:06.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.730+0000 7f51ee16bb80 1 -- 172.21.15.114:0/2160919956 shutdown_connections 2024-05-27T07:45:06.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.730+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/2160919956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bd34ee0 0x561f9bd44970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.730+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/2160919956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 0x561f9bd349a0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.730+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/2160919956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd2f190 0x561f9bd38ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.730+0000 7f51ee16bb80 1 -- 172.21.15.114:0/2160919956 >> 172.21.15.114:0/2160919956 conn(0x561f9bc09af0 msgr2=0x561f9bd32260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.730+0000 7f51ee16bb80 1 -- 172.21.15.114:0/2160919956 shutdown_connections 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.730+0000 7f51ee16bb80 1 -- 172.21.15.114:0/2160919956 wait complete. 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.730+0000 7f51ee16bb80 1 -- start start 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.730+0000 7f51ee16bb80 1 Processor -- start 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51ee16bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 0x561f9bd2e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51ee16bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bd2f190 0x561f9bd2c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51ee16bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd34ee0 0x561f9bd2ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51ee16bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81ade00 con 0x561f9bbaee40 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51ee16bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81ac990 con 0x561f9bd34ee0 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51ee16bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81ad060 con 0x561f9bd2f190 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51ee16bb80 10 client.0 _ll_register_callbacks cb 0x561f9bbe6920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51dd4f2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 0x561f9bd2e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51dd4f2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 0x561f9bd2e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57482/0 (socket says 172.21.15.114:57482) 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51dd4f2700 1 -- 172.21.15.114:0/75170100 learned_addr learned my addr 172.21.15.114:0/75170100 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51dccf1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bd2f190 0x561f9bd2c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51ddcf3700 1 --2- 172.21.15.114:0/75170100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd34ee0 0x561f9bd2ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51dd4f2700 1 -- 172.21.15.114:0/75170100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bd2f190 msgr2=0x561f9bd2c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/75170100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bd2f190 0x561f9bd2c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51dd4f2700 1 -- 172.21.15.114:0/75170100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd34ee0 msgr2=0x561f9bd2ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/75170100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd34ee0 0x561f9bd2ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51dd4f2700 1 -- 172.21.15.114:0/75170100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f9bd2d5d0 con 0x561f9bbaee40 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51dd4f2700 1 --2- 172.21.15.114:0/75170100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 0x561f9bd2e560 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f51d4014040 tx=0x7f51d400ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca94539675ace2c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51c0ff9700 1 -- 172.21.15.114:0/75170100 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d40050d0 con 0x561f9bbaee40 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51c0ff9700 1 -- 172.21.15.114:0/75170100 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f51d4005270 con 0x561f9bbaee40 2024-05-27T07:45:06.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.731+0000 7f51ee16bb80 1 -- 172.21.15.114:0/75170100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51d400ed20 con 0x561f9bbaee40 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.732+0000 7f51c0ff9700 1 -- 172.21.15.114:0/75170100 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f51d4003840 con 0x561f9bbaee40 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.732+0000 7f51ee16bb80 10 client.16569 fetch_fsmap learned FSMap version 33 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.732+0000 7f51ee16bb80 1 -- 172.21.15.114:0/75170100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561f9bd40a50 con 0x561f9bbaee40 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.732+0000 7f51c0ff9700 1 -- 172.21.15.114:0/75170100 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d400ed20 con 0x561f9bbaee40 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.732+0000 7f51c0ff9700 1 -- 172.21.15.114:0/75170100 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 33) ==== 7725+0+0 (secure 0 0 0) 0x7f51d401cc00 con 0x561f9bbaee40 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.732+0000 7f51ee16bb80 10 client.16569 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.732+0000 7f51ee16bb80 10 client.16569 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.732+0000 7f51ee16bb80 20 client.16569 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.732+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/75170100 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561f9bd40fa0 0x561f9bd413b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.732+0000 7f51ee16bb80 4 client.16569 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.732+0000 7f51ee16bb80 1 -- 172.21.15.114:0/75170100 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561f9bc5dbe0 con 0x561f9bd40fa0 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.732+0000 7f51dccf1700 1 --2- 172.21.15.114:0/75170100 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561f9bd40fa0 0x561f9bd413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.733+0000 7f51c0ff9700 10 client.16569 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:06.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.733+0000 7f51dccf1700 1 --2- 172.21.15.114:0/75170100 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561f9bd40fa0 0x561f9bd413b0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3eb76740fadadf6b server_cookie=d0713f0b3402407a in_seq=0 out_seq=0 2024-05-27T07:45:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.734+0000 7f51c0ff9700 1 -- 172.21.15.114:0/75170100 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561f9bc5dbe0 con 0x561f9bd40fa0 2024-05-27T07:45:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.734+0000 7f51c0ff9700 10 client.16569 handle_command_reply: tid=0 2024-05-27T07:45:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.734+0000 7f51ee16bb80 1 client.16569 shutdown 2024-05-27T07:45:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.734+0000 7f51ee16bb80 1 -- 172.21.15.114:0/75170100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 msgr2=0x561f9bd2e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.734+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/75170100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 0x561f9bd2e560 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f51d4014040 tx=0x7f51d400ef80 comp rx=0 tx=0).stop 2024-05-27T07:45:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.734+0000 7f51ee16bb80 1 -- 172.21.15.114:0/75170100 shutdown_connections 2024-05-27T07:45:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.734+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/75170100 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561f9bd40fa0 0x561f9bd413b0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.734+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/75170100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561f9bd34ee0 0x561f9bd2ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.734+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/75170100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9bbaee40 0x561f9bd2e560 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.734+0000 7f51ee16bb80 1 --2- 172.21.15.114:0/75170100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561f9bd2f190 0x561f9bd2c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.734+0000 7f51ee16bb80 1 -- 172.21.15.114:0/75170100 >> 172.21.15.114:0/75170100 conn(0x561f9bc09af0 msgr2=0x561f9bd21e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.734+0000 7f51c0ff9700 0 client.16569 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.735+0000 7f51ee16bb80 1 -- 172.21.15.114:0/75170100 shutdown_connections 2024-05-27T07:45:06.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.735+0000 7f51ee16bb80 1 -- 172.21.15.114:0/75170100 wait complete. 2024-05-27T07:45:06.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.735+0000 7f51ee16bb80 20 client.16569 trim_cache size 0 max 16384 2024-05-27T07:45:06.738 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:45:06.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.735+0000 7f51df756700 1 -- 172.21.15.114:0/2095618724 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f51c40b29f0 msgr2=0x7f51c40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:06.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.735+0000 7f51df756700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f51c40b29f0 0x7f51c40b4f00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f51d819ac10 tx=0x7f51d4009260 comp rx=0 tx=0).stop 2024-05-27T07:45:06.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.735+0000 7f51df756700 1 -- 172.21.15.114:0/2095618724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51d8107de0 msgr2=0x7f51d819fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:06.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.735+0000 7f51df756700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51d8107de0 0x7f51d819fdf0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f51c800e9c0 tx=0x7f51c800ecd0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.735+0000 7f51df756700 1 -- 172.21.15.114:0/2095618724 shutdown_connections 2024-05-27T07:45:06.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.735+0000 7f51df756700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d8108730 0x7f51d8199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.735+0000 7f51df756700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f51c40b29f0 0x7f51c40b4f00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.735+0000 7f51df756700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51d81034b0 0x7f51d819f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.735+0000 7f51df756700 1 --2- 172.21.15.114:0/2095618724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51d8107de0 0x7f51d819fdf0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:06.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.735+0000 7f51df756700 1 -- 172.21.15.114:0/2095618724 >> 172.21.15.114:0/2095618724 conn(0x7f51d80fede0 msgr2=0x7f51d8107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:06.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.736+0000 7f51df756700 1 -- 172.21.15.114:0/2095618724 shutdown_connections 2024-05-27T07:45:06.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:06.736+0000 7f51df756700 1 -- 172.21.15.114:0/2095618724 wait complete. 2024-05-27T07:45:07.247 INFO:tasks.workunit.client.0.smithi114.stdout: 1 498919 13.13 MB/sec execute 79 sec latency 60.885 ms 2024-05-27T07:45:07.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:07 smithi114 ceph-mon[32038]: pgmap v170: 129 pgs: 129 active+clean; 43 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 3.7 MiB/s wr, 495 op/s 2024-05-27T07:45:07.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:07 smithi186 ceph-mon[36647]: pgmap v170: 129 pgs: 129 active+clean; 43 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 3.7 MiB/s wr, 495 op/s 2024-05-27T07:45:07.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:07 smithi163 ceph-mon[36451]: pgmap v170: 129 pgs: 129 active+clean; 43 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 3.7 MiB/s wr, 495 op/s 2024-05-27T07:45:08.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 501486 13.15 MB/sec execute 80 sec latency 80.397 ms 2024-05-27T07:45:09.588 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:45:09.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:45:09.590 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:09 smithi186 ceph-mon[36647]: pgmap v171: 129 pgs: 129 active+clean; 43 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 3.7 MiB/s wr, 495 op/s 2024-05-27T07:45:09.590 INFO:tasks.workunit.client.0.smithi114.stdout: 1 503420 13.08 MB/sec execute 81 sec latency 43.575 ms 2024-05-27T07:45:09.590 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:09 smithi186 podman[72462]: 2024-05-27 07:45:09.267356699 +0000 UTC m=+4.977908227 container remove 6f41e07e18921c088441505e2e68f1fc482aa4fdcb271770fec6508109065adf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=) 2024-05-27T07:45:09.591 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:09 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:45:09.613 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:09 smithi114 ceph-mon[32038]: pgmap v171: 129 pgs: 129 active+clean; 43 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 3.7 MiB/s wr, 495 op/s 2024-05-27T07:45:09.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:09 smithi163 ceph-mon[36451]: pgmap v171: 129 pgs: 129 active+clean; 43 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 3.7 MiB/s wr, 495 op/s 2024-05-27T07:45:09.843 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:09.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:09 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:45:10.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 505658 13.08 MB/sec execute 82 sec latency 16.572 ms 2024-05-27T07:45:11.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 508635 13.13 MB/sec execute 83 sec latency 35.430 ms 2024-05-27T07:45:11.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:11 smithi114 ceph-mon[32038]: pgmap v172: 129 pgs: 129 active+clean; 44 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 3.2 MiB/s wr, 430 op/s 2024-05-27T07:45:11.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:11 smithi163 ceph-mon[36451]: pgmap v172: 129 pgs: 129 active+clean; 44 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 3.2 MiB/s wr, 430 op/s 2024-05-27T07:45:11.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:11 smithi186 ceph-mon[36647]: pgmap v172: 129 pgs: 129 active+clean; 44 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 3.2 MiB/s wr, 430 op/s 2024-05-27T07:45:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.825+0000 7f14c6208700 1 -- 172.21.15.114:0/1226375324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14c01054f0 msgr2=0x7f14c0105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.825+0000 7f14c6208700 1 --2- 172.21.15.114:0/1226375324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14c01054f0 0x7f14c0105900 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009990 tx=0x7f14b0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.825+0000 7f14c6208700 1 -- 172.21.15.114:0/1226375324 shutdown_connections 2024-05-27T07:45:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.825+0000 7f14c6208700 1 --2- 172.21.15.114:0/1226375324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14c010e8d0 0x7f14c0110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.825+0000 7f14c6208700 1 --2- 172.21.15.114:0/1226375324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14c01054f0 0x7f14c0105900 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.825+0000 7f14c6208700 1 --2- 172.21.15.114:0/1226375324 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14c0105e40 0x7f14c010e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.825+0000 7f14c6208700 1 -- 172.21.15.114:0/1226375324 >> 172.21.15.114:0/1226375324 conn(0x7f14c00fedb0 msgr2=0x7f14c01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.825+0000 7f14c6208700 1 -- 172.21.15.114:0/1226375324 shutdown_connections 2024-05-27T07:45:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.825+0000 7f14c6208700 1 -- 172.21.15.114:0/1226375324 wait complete. 2024-05-27T07:45:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.825+0000 7f14c6208700 1 Processor -- start 2024-05-27T07:45:11.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.826+0000 7f14c6208700 1 -- start start 2024-05-27T07:45:11.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.826+0000 7f14c6208700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14c01054f0 0x7f14c019ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.826+0000 7f14c6208700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14c0105e40 0x7f14c019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.826+0000 7f14c6208700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14c010e8d0 0x7f14c019f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.826+0000 7f14c6208700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f14c0114420 con 0x7f14c01054f0 2024-05-27T07:45:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.826+0000 7f14c6208700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f14c01142a0 con 0x7f14c010e8d0 2024-05-27T07:45:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.826+0000 7f14c6208700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f14c01145a0 con 0x7f14c0105e40 2024-05-27T07:45:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.826+0000 7f14bf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14c01054f0 0x7f14c019ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.826+0000 7f14beffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14c0105e40 0x7f14c019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.826+0000 7f14bf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14c01054f0 0x7f14c019ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57502/0 (socket says 172.21.15.114:57502) 2024-05-27T07:45:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.826+0000 7f14bf7fe700 1 -- 172.21.15.114:0/3293855552 learned_addr learned my addr 172.21.15.114:0/3293855552 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.826+0000 7f14bffff700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14c010e8d0 0x7f14c019f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.827+0000 7f14beffd700 1 -- 172.21.15.114:0/3293855552 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14c010e8d0 msgr2=0x7f14c019f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.827+0000 7f14beffd700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14c010e8d0 0x7f14c019f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.827+0000 7f14beffd700 1 -- 172.21.15.114:0/3293855552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14c01054f0 msgr2=0x7f14c019ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.827+0000 7f14beffd700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14c01054f0 0x7f14c019ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.827+0000 7f14beffd700 1 -- 172.21.15.114:0/3293855552 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c019fb90 con 0x7f14c0105e40 2024-05-27T07:45:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.827+0000 7f14bf7fe700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14c01054f0 0x7f14c019ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.827+0000 7f14beffd700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14c0105e40 0x7f14c019b310 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f14b400c970 tx=0x7f14b400cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.827+0000 7f14bcff9700 1 -- 172.21.15.114:0/3293855552 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4014070 con 0x7f14c0105e40 2024-05-27T07:45:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.827+0000 7f14c6208700 1 -- 172.21.15.114:0/3293855552 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c019fe80 con 0x7f14c0105e40 2024-05-27T07:45:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.827+0000 7f14bcff9700 1 -- 172.21.15.114:0/3293855552 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14b4004440 con 0x7f14c0105e40 2024-05-27T07:45:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.827+0000 7f14bcff9700 1 -- 172.21.15.114:0/3293855552 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4007a30 con 0x7f14c0105e40 2024-05-27T07:45:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.827+0000 7f14c6208700 1 -- 172.21.15.114:0/3293855552 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c01a03c0 con 0x7f14c0105e40 2024-05-27T07:45:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.830+0000 7f14bcff9700 1 -- 172.21.15.114:0/3293855552 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f14b4017070 con 0x7f14c0105e40 2024-05-27T07:45:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.831+0000 7f14bcff9700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f14a80b29f0 0x7f14a80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.831+0000 7f14bf7fe700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f14a80b29f0 0x7f14a80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.831+0000 7f14d4c1db80 1 Processor -- start 2024-05-27T07:45:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.831+0000 7f14d4c1db80 1 -- start start 2024-05-27T07:45:11.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.831+0000 7f14d4c1db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc1131ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.831+0000 7f14d4c1db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc112bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.831+0000 7f14bcff9700 1 -- 172.21.15.114:0/3293855552 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f14b40e22e0 con 0x7f14c0105e40 2024-05-27T07:45:11.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.831+0000 7f14d4c1db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc112c410 0x564bc112c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.831+0000 7f14bf7fe700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f14a80b29f0 0x7f14a80b4f00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f14b000b2e0 tx=0x7f14b0004710 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:11.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14d4c1db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564bc1056d10 con 0x564bc0fa9ec0 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14d4c1db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564bc10468b0 con 0x564bc112c410 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14d4c1db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564bc104a8f0 con 0x564bc1131ff0 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14bf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc1131ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14bf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc1131ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57518/0 (socket says 172.21.15.114:57518) 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14bf7fe700 1 -- 172.21.15.114:0/3398873895 learned_addr learned my addr 172.21.15.114:0/3398873895 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14beffd700 1 --2- 172.21.15.114:0/3398873895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc112bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14bffff700 1 --2- 172.21.15.114:0/3398873895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc112c410 0x564bc112c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14beffd700 1 -- 172.21.15.114:0/3398873895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc112c410 msgr2=0x564bc112c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14beffd700 1 --2- 172.21.15.114:0/3398873895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc112c410 0x564bc112c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14beffd700 1 -- 172.21.15.114:0/3398873895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 msgr2=0x564bc1131ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14beffd700 1 --2- 172.21.15.114:0/3398873895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc1131ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14beffd700 1 -- 172.21.15.114:0/3398873895 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bc0f96950 con 0x564bc1131ff0 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14bffff700 1 --2- 172.21.15.114:0/3398873895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc112c410 0x564bc112c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.832+0000 7f14beffd700 1 --2- 172.21.15.114:0/3398873895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc112bed0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f14b40df620 tx=0x7f14b40058b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5c1e8fd044a3be2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:11.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.833+0000 7f14a5ffb700 1 -- 172.21.15.114:0/3398873895 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b40a91b0 con 0x564bc1131ff0 2024-05-27T07:45:11.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.833+0000 7f14a5ffb700 1 -- 172.21.15.114:0/3398873895 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14b40ad560 con 0x564bc1131ff0 2024-05-27T07:45:11.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.833+0000 7f14a5ffb700 1 -- 172.21.15.114:0/3398873895 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b40ad120 con 0x564bc1131ff0 2024-05-27T07:45:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.833+0000 7f14d4c1db80 1 -- 172.21.15.114:0/3398873895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 msgr2=0x564bc112bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.833+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/3398873895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc112bed0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f14b40df620 tx=0x7f14b40058b0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.834+0000 7f14d4c1db80 1 -- 172.21.15.114:0/3398873895 shutdown_connections 2024-05-27T07:45:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.834+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/3398873895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc112c410 0x564bc112c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.834+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/3398873895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc1131ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.834+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/3398873895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc112bed0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.834+0000 7f14d4c1db80 1 -- 172.21.15.114:0/3398873895 >> 172.21.15.114:0/3398873895 conn(0x564bc1004b70 msgr2=0x564bc11226c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.834+0000 7f14d4c1db80 1 -- 172.21.15.114:0/3398873895 shutdown_connections 2024-05-27T07:45:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.834+0000 7f14d4c1db80 1 -- 172.21.15.114:0/3398873895 wait complete. 2024-05-27T07:45:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.834+0000 7f14d4c1db80 1 -- start start 2024-05-27T07:45:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.834+0000 7f14d4c1db80 1 Processor -- start 2024-05-27T07:45:11.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14d4c1db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc0fa9ec0 0x564bc1135a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14d4c1db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc112c410 0x564bc1135f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14d4c1db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc1136490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14d4c1db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564bc0f4d020 con 0x564bc112c410 2024-05-27T07:45:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14d4c1db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564bc0fe8840 con 0x564bc0fa9ec0 2024-05-27T07:45:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14d4c1db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564bc1138b30 con 0x564bc1131ff0 2024-05-27T07:45:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14d4c1db80 10 client.0 _ll_register_callbacks cb 0x564bc0fe19a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14beffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc112c410 0x564bc1135f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14bffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc1136490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14bffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc1136490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:42586/0 (socket says 172.21.15.114:42586) 2024-05-27T07:45:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14bffff700 1 -- 172.21.15.114:0/2620153381 learned_addr learned my addr 172.21.15.114:0/2620153381 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14beffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc112c410 0x564bc1135f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57522/0 (socket says 172.21.15.114:57522) 2024-05-27T07:45:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14bf7fe700 1 --2- 172.21.15.114:0/2620153381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc0fa9ec0 0x564bc1135a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14bffff700 1 -- 172.21.15.114:0/2620153381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc0fa9ec0 msgr2=0x564bc1135a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:11.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14bffff700 1 --2- 172.21.15.114:0/2620153381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc0fa9ec0 0x564bc1135a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14bffff700 1 -- 172.21.15.114:0/2620153381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc112c410 msgr2=0x564bc1135f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:11.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14bffff700 1 --2- 172.21.15.114:0/2620153381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc112c410 0x564bc1135f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14bffff700 1 -- 172.21.15.114:0/2620153381 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bc0fe5710 con 0x564bc1131ff0 2024-05-27T07:45:11.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14beffd700 1 --2- 172.21.15.114:0/2620153381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc112c410 0x564bc1135f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:11.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f14bffff700 1 --2- 172.21.15.114:0/2620153381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc1136490 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f14b8007d10 tx=0x7f14b80116a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=772ce1b5ea7b2319 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:11.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f149effd700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:45:11.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.835+0000 7f149effd700 1 -- 172.21.15.114:0/2620153381 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b801a070 con 0x564bc1131ff0 2024-05-27T07:45:11.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2620153381 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14b40ad560 con 0x564bc1131ff0 2024-05-27T07:45:11.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f149effd700 1 -- 172.21.15.114:0/2620153381 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f14b801f460 con 0x564bc1131ff0 2024-05-27T07:45:11.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f149effd700 1 -- 172.21.15.114:0/2620153381 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f14b8028050 con 0x564bc1131ff0 2024-05-27T07:45:11.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f14d4c1db80 10 client.26173 fetch_fsmap learned FSMap version 33 2024-05-27T07:45:11.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2620153381 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564bc11437f0 con 0x564bc1131ff0 2024-05-27T07:45:11.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f149effd700 1 -- 172.21.15.114:0/2620153381 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14b40ad560 con 0x564bc1131ff0 2024-05-27T07:45:11.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f149effd700 1 -- 172.21.15.114:0/2620153381 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 33) ==== 7725+0+0 (secure 0 0 0) 0x7f14b8016770 con 0x564bc1131ff0 2024-05-27T07:45:11.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f14d4c1db80 10 client.26173 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T07:45:11.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f14d4c1db80 10 client.26173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:11.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f14d4c1db80 20 client.26173 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:11.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/2620153381 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564bc1145e30 0x564bc1148270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f14d4c1db80 4 client.26173 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:11.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2620153381 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564bc10f3a70 con 0x564bc1145e30 2024-05-27T07:45:11.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.836+0000 7f14bf7fe700 1 --2- 172.21.15.114:0/2620153381 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564bc1145e30 0x564bc1148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.837+0000 7f149effd700 10 client.26173 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.837+0000 7f14bf7fe700 1 --2- 172.21.15.114:0/2620153381 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564bc1145e30 0x564bc1148270 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4847247d4c75bb4 server_cookie=4eb395cea5935084 in_seq=0 out_seq=0 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f149effd700 1 -- 172.21.15.114:0/2620153381 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564bc10f3a70 con 0x564bc1145e30 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f149effd700 10 client.26173 handle_command_reply: tid=0 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f14d4c1db80 1 client.26173 shutdown 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2620153381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 msgr2=0x564bc1136490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/2620153381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc1136490 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f14b8007d10 tx=0x7f14b80116a0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2620153381 shutdown_connections 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/2620153381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc1136490 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/2620153381 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564bc1145e30 0x564bc1148270 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/2620153381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc0fa9ec0 0x564bc1135a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/2620153381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc112c410 0x564bc1135f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2620153381 >> 172.21.15.114:0/2620153381 conn(0x564bc1004b70 msgr2=0x564bc1126ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:11.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f149effd700 0 client.26173 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:11.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2620153381 shutdown_connections 2024-05-27T07:45:11.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2620153381 wait complete. 2024-05-27T07:45:11.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.838+0000 7f14d4c1db80 20 client.26173 trim_cache size 0 max 16384 2024-05-27T07:45:11.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.861+0000 7f14d4c1db80 1 Processor -- start 2024-05-27T07:45:11.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.861+0000 7f14d4c1db80 1 -- start start 2024-05-27T07:45:11.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.861+0000 7f14d4c1db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc1134d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.861+0000 7f14d4c1db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc112c410 0x564bc1133680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.861+0000 7f14d4c1db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc1131ff0 0x564bc111d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.861+0000 7f14d4c1db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f14c01af2a0 con 0x564bc0fa9ec0 2024-05-27T07:45:11.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.861+0000 7f14d4c1db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f14c01ac350 con 0x564bc1131ff0 2024-05-27T07:45:11.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.861+0000 7f14d4c1db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f14c01ae500 con 0x564bc112c410 2024-05-27T07:45:11.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.861+0000 7f14bf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc1134d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.861+0000 7f14bf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc1134d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57524/0 (socket says 172.21.15.114:57524) 2024-05-27T07:45:11.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.861+0000 7f14bf7fe700 1 -- 172.21.15.114:0/155555442 learned_addr learned my addr 172.21.15.114:0/155555442 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:11.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.861+0000 7f14beffd700 1 --2- 172.21.15.114:0/155555442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc112c410 0x564bc1133680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.862+0000 7f14bffff700 1 --2- 172.21.15.114:0/155555442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc1131ff0 0x564bc111d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.862+0000 7f14bf7fe700 1 -- 172.21.15.114:0/155555442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc112c410 msgr2=0x564bc1133680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:11.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.862+0000 7f14bf7fe700 1 --2- 172.21.15.114:0/155555442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc112c410 0x564bc1133680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.862+0000 7f14bf7fe700 1 -- 172.21.15.114:0/155555442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc1131ff0 msgr2=0x564bc111d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:11.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.862+0000 7f14bf7fe700 1 --2- 172.21.15.114:0/155555442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc1131ff0 0x564bc111d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.862+0000 7f14bf7fe700 1 -- 172.21.15.114:0/155555442 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bc1122240 con 0x564bc0fa9ec0 2024-05-27T07:45:11.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.862+0000 7f14bffff700 1 --2- 172.21.15.114:0/155555442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc1131ff0 0x564bc111d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:45:11.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.862+0000 7f14bf7fe700 1 --2- 172.21.15.114:0/155555442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc1134d30 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f14b00098e0 tx=0x7f14b0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca613f444384137 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:11.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.862+0000 7f14a57fa700 1 -- 172.21.15.114:0/155555442 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0004800 con 0x564bc0fa9ec0 2024-05-27T07:45:11.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.862+0000 7f14a57fa700 1 -- 172.21.15.114:0/155555442 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14b0004e50 con 0x564bc0fa9ec0 2024-05-27T07:45:11.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.862+0000 7f14a57fa700 1 -- 172.21.15.114:0/155555442 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0022450 con 0x564bc0fa9ec0 2024-05-27T07:45:11.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.863+0000 7f14d4c1db80 1 -- 172.21.15.114:0/155555442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 msgr2=0x564bc1134d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:11.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.863+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/155555442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc1134d30 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f14b00098e0 tx=0x7f14b0018000 comp rx=0 tx=0).stop 2024-05-27T07:45:11.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.863+0000 7f14d4c1db80 1 -- 172.21.15.114:0/155555442 shutdown_connections 2024-05-27T07:45:11.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.863+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/155555442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc1131ff0 0x564bc111d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.863+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/155555442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc1134d30 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.863+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/155555442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc112c410 0x564bc1133680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.863+0000 7f14d4c1db80 1 -- 172.21.15.114:0/155555442 >> 172.21.15.114:0/155555442 conn(0x564bc1004b70 msgr2=0x564bc1126ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.863+0000 7f14d4c1db80 1 -- 172.21.15.114:0/155555442 shutdown_connections 2024-05-27T07:45:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.863+0000 7f14d4c1db80 1 -- 172.21.15.114:0/155555442 wait complete. 2024-05-27T07:45:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.864+0000 7f14d4c1db80 1 -- start start 2024-05-27T07:45:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.864+0000 7f14d4c1db80 1 Processor -- start 2024-05-27T07:45:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.864+0000 7f14d4c1db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc114d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.864+0000 7f14d4c1db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc112c410 0x564bc114dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.864+0000 7f14d4c1db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc114e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.864+0000 7f14d4c1db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f14c01ade30 con 0x564bc0fa9ec0 2024-05-27T07:45:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.864+0000 7f14d4c1db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f14c01ac9c0 con 0x564bc112c410 2024-05-27T07:45:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.864+0000 7f14d4c1db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f14c01ad090 con 0x564bc1131ff0 2024-05-27T07:45:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.864+0000 7f14d4c1db80 10 client.0 _ll_register_callbacks cb 0x564bc0fe19a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.864+0000 7f14bf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc114d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.864+0000 7f14bf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc114d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57536/0 (socket says 172.21.15.114:57536) 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.864+0000 7f14bf7fe700 1 -- 172.21.15.114:0/2776967575 learned_addr learned my addr 172.21.15.114:0/2776967575 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.864+0000 7f14bffff700 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc114e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f14beffd700 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc112c410 0x564bc114dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f14bf7fe700 1 -- 172.21.15.114:0/2776967575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 msgr2=0x564bc114e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f14bf7fe700 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc114e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f14bf7fe700 1 -- 172.21.15.114:0/2776967575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc112c410 msgr2=0x564bc114dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f14bf7fe700 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc112c410 0x564bc114dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f14bf7fe700 1 -- 172.21.15.114:0/2776967575 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bc11afa00 con 0x564bc0fa9ec0 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f14beffd700 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc112c410 0x564bc114dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f14bf7fe700 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc114d6b0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009ce0 tx=0x7f14b0005660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1742d323ef636f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f149effd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f149effd700 1 -- 172.21.15.114:0/2776967575 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0043020 con 0x564bc0fa9ec0 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2776967575 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14b0004e50 con 0x564bc0fa9ec0 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f149effd700 1 -- 172.21.15.114:0/2776967575 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f14b00047d0 con 0x564bc0fa9ec0 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f149effd700 1 -- 172.21.15.114:0/2776967575 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f14b004e050 con 0x564bc0fa9ec0 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f14d4c1db80 10 client.16587 fetch_fsmap learned FSMap version 33 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.865+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2776967575 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564bc11b0bb0 con 0x564bc0fa9ec0 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.866+0000 7f149effd700 1 -- 172.21.15.114:0/2776967575 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14b0004e50 con 0x564bc0fa9ec0 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.866+0000 7f149effd700 1 -- 172.21.15.114:0/2776967575 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 33) ==== 7725+0+0 (secure 0 0 0) 0x7f14b0022c70 con 0x564bc0fa9ec0 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.866+0000 7f14d4c1db80 10 client.16587 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.866+0000 7f14d4c1db80 10 client.16587 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.866+0000 7f14d4c1db80 20 client.16587 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.866+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564bc11b11b0 0x564bc11b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.866+0000 7f14d4c1db80 4 client.16587 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.866+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2776967575 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564bc1058c60 con 0x564bc11b11b0 2024-05-27T07:45:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.866+0000 7f14beffd700 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564bc11b11b0 0x564bc11b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:11.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.866+0000 7f149effd700 10 client.16587 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:11.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.867+0000 7f14beffd700 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564bc11b11b0 0x564bc11b35d0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47a18089597692ac server_cookie=6265331d3a6a4a8e in_seq=0 out_seq=0 2024-05-27T07:45:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f149effd700 1 -- 172.21.15.114:0/2776967575 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564bc1058c60 con 0x564bc11b11b0 2024-05-27T07:45:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f149effd700 10 client.16587 handle_command_reply: tid=0 2024-05-27T07:45:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f14d4c1db80 1 client.16587 shutdown 2024-05-27T07:45:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2776967575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 msgr2=0x564bc114d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc114d6b0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009ce0 tx=0x7f14b0005660 comp rx=0 tx=0).stop 2024-05-27T07:45:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2776967575 shutdown_connections 2024-05-27T07:45:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564bc1131ff0 0x564bc114e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564bc0fa9ec0 0x564bc114d6b0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564bc11b11b0 0x564bc11b35d0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f14d4c1db80 1 --2- 172.21.15.114:0/2776967575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564bc112c410 0x564bc114dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f149effd700 0 client.16587 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2776967575 >> 172.21.15.114:0/2776967575 conn(0x564bc1004b70 msgr2=0x564bc1134a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:11.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2776967575 shutdown_connections 2024-05-27T07:45:11.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f14d4c1db80 1 -- 172.21.15.114:0/2776967575 wait complete. 2024-05-27T07:45:11.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.869+0000 7f14d4c1db80 20 client.16587 trim_cache size 0 max 16384 2024-05-27T07:45:11.871 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:45:11.871 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:45:11.871 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "d42297f2-c2cf-43b8-9826-b6c3c4651e44", 2024-05-27T07:45:11.871 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:45:11.871 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:45:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.870+0000 7f14c6208700 1 -- 172.21.15.114:0/3293855552 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f14a80b29f0 msgr2=0x7f14a80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.870+0000 7f14c6208700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f14a80b29f0 0x7f14a80b4f00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f14b000b2e0 tx=0x7f14b0004710 comp rx=0 tx=0).stop 2024-05-27T07:45:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.870+0000 7f14c6208700 1 -- 172.21.15.114:0/3293855552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14c0105e40 msgr2=0x7f14c019b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.870+0000 7f14c6208700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14c0105e40 0x7f14c019b310 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f14b400c970 tx=0x7f14b400cc80 comp rx=0 tx=0).stop 2024-05-27T07:45:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.870+0000 7f14c6208700 1 -- 172.21.15.114:0/3293855552 shutdown_connections 2024-05-27T07:45:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.870+0000 7f14c6208700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f14a80b29f0 0x7f14a80b4f00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.870+0000 7f14c6208700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14c010e8d0 0x7f14c019f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.870+0000 7f14c6208700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14c01054f0 0x7f14c019ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.870+0000 7f14c6208700 1 --2- 172.21.15.114:0/3293855552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14c0105e40 0x7f14c019b310 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.870+0000 7f14c6208700 1 -- 172.21.15.114:0/3293855552 >> 172.21.15.114:0/3293855552 conn(0x7f14c00fedb0 msgr2=0x7f14c00ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:11.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.870+0000 7f14c6208700 1 -- 172.21.15.114:0/3293855552 shutdown_connections 2024-05-27T07:45:11.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:11.870+0000 7f14c6208700 1 -- 172.21.15.114:0/3293855552 wait complete. 2024-05-27T07:45:12.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 511793 13.20 MB/sec execute 84 sec latency 14.691 ms 2024-05-27T07:45:12.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:12 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:45:12.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:12 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:12.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:12 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:45:12.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:12 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:12.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:12 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:45:12.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:12 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:45:12.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:12 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:12.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:12 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:45:12.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:12 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:12.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:12 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:45:12.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:12 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:45:12.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:12 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:12.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:12 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:45:12.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:12 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:12.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:12 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:45:12.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:45:13.182 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:13.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 514163 13.14 MB/sec execute 85 sec latency 18.850 ms 2024-05-27T07:45:13.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:13 smithi114 ceph-mon[32038]: pgmap v173: 129 pgs: 129 active+clean; 49 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 3.9 MiB/s wr, 476 op/s 2024-05-27T07:45:13.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:13 smithi163 ceph-mon[36451]: pgmap v173: 129 pgs: 129 active+clean; 49 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 3.9 MiB/s wr, 476 op/s 2024-05-27T07:45:13.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:13 smithi186 ceph-mon[36647]: pgmap v173: 129 pgs: 129 active+clean; 49 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 3.9 MiB/s wr, 476 op/s 2024-05-27T07:45:14.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 515869 13.12 MB/sec execute 86 sec latency 57.675 ms 2024-05-27T07:45:14.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.937+0000 7f95b341a700 1 -- 172.21.15.114:0/256841603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95ac102870 msgr2=0x7f95ac102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:14.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.937+0000 7f95b341a700 1 --2- 172.21.15.114:0/256841603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95ac102870 0x7f95ac102c80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f959c009990 tx=0x7f959c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.937+0000 7f95b341a700 1 -- 172.21.15.114:0/256841603 shutdown_connections 2024-05-27T07:45:14.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.937+0000 7f95b341a700 1 --2- 172.21.15.114:0/256841603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95ac1031c0 0x7f95ac111160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.937+0000 7f95b341a700 1 --2- 172.21.15.114:0/256841603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95ac1092f0 0x7f95ac109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.937+0000 7f95b341a700 1 --2- 172.21.15.114:0/256841603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95ac102870 0x7f95ac102c80 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.937+0000 7f95b341a700 1 -- 172.21.15.114:0/256841603 >> 172.21.15.114:0/256841603 conn(0x7f95ac0fe6e0 msgr2=0x7f95ac100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:14.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.938+0000 7f95b341a700 1 -- 172.21.15.114:0/256841603 shutdown_connections 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.938+0000 7f95b341a700 1 -- 172.21.15.114:0/256841603 wait complete. 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.938+0000 7f95b341a700 1 Processor -- start 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.938+0000 7f95b341a700 1 -- start start 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.938+0000 7f95b341a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95ac102870 0x7f95ac103a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.938+0000 7f95b341a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95ac1031c0 0x7f95ac103f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.938+0000 7f95b341a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95ac1092f0 0x7f95ac1044d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b341a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac114590 con 0x7f95ac1031c0 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b341a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac114410 con 0x7f95ac102870 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b341a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac114710 con 0x7f95ac1092f0 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b09b5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95ac1031c0 0x7f95ac103f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b09b5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95ac1031c0 0x7f95ac103f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48374/0 (socket says 172.21.15.114:48374) 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b09b5700 1 -- 172.21.15.114:0/766327459 learned_addr learned my addr 172.21.15.114:0/766327459 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b11b6700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95ac102870 0x7f95ac103a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b19b7700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95ac1092f0 0x7f95ac1044d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b09b5700 1 -- 172.21.15.114:0/766327459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95ac1092f0 msgr2=0x7f95ac1044d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:14.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b09b5700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95ac1092f0 0x7f95ac1044d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b09b5700 1 -- 172.21.15.114:0/766327459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95ac102870 msgr2=0x7f95ac103a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:14.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b09b5700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95ac102870 0x7f95ac103a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b09b5700 1 -- 172.21.15.114:0/766327459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ac1a9300 con 0x7f95ac1031c0 2024-05-27T07:45:14.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b19b7700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95ac1092f0 0x7f95ac1044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:14.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b11b6700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95ac102870 0x7f95ac103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:14.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95b09b5700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95ac1031c0 0x7f95ac103f90 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f959c000c00 tx=0x7f959c00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:14.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.939+0000 7f95a27fc700 1 -- 172.21.15.114:0/766327459 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959c027070 con 0x7f95ac1031c0 2024-05-27T07:45:14.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.940+0000 7f95a27fc700 1 -- 172.21.15.114:0/766327459 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f959c022470 con 0x7f95ac1031c0 2024-05-27T07:45:14.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.940+0000 7f95a27fc700 1 -- 172.21.15.114:0/766327459 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959c017650 con 0x7f95ac1031c0 2024-05-27T07:45:14.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.940+0000 7f95b341a700 1 -- 172.21.15.114:0/766327459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95ac1a9590 con 0x7f95ac1031c0 2024-05-27T07:45:14.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.940+0000 7f95b341a700 1 -- 172.21.15.114:0/766327459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95ac1a9990 con 0x7f95ac1031c0 2024-05-27T07:45:14.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.942+0000 7f95a27fc700 1 -- 172.21.15.114:0/766327459 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f959c02a070 con 0x7f95ac1031c0 2024-05-27T07:45:14.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.943+0000 7f95a27fc700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95980b2a70 0x7f95980b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.943+0000 7f95b11b6700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95980b2a70 0x7f95980b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.943+0000 7f95c1e2fb80 1 Processor -- start 2024-05-27T07:45:14.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.943+0000 7f95a27fc700 1 -- 172.21.15.114:0/766327459 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f959c0e3860 con 0x7f95ac1031c0 2024-05-27T07:45:14.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95c1e2fb80 1 -- start start 2024-05-27T07:45:14.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95c1e2fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29258ae40 0x55b29270fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95c1e2fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b292710070 0x55b292709be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95b11b6700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95980b2a70 0x7f95980b4f80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f95ac076270 tx=0x7f95a8009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:14.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95c1e2fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b29270a120 0x55b29270a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95c1e2fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b292637d70 con 0x55b29270a120 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95c1e2fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b292627910 con 0x55b292710070 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95c1e2fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b29262b950 con 0x55b29258ae40 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95b11b6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29258ae40 0x55b29270fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95b19b7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b29270a120 0x55b29270a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95b09b5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b292710070 0x55b292709be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95b11b6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29258ae40 0x55b29270fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54194/0 (socket says 172.21.15.114:54194) 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95b11b6700 1 -- 172.21.15.114:0/2073143172 learned_addr learned my addr 172.21.15.114:0/2073143172 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95b11b6700 1 -- 172.21.15.114:0/2073143172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b292710070 msgr2=0x55b292709be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95b11b6700 1 --2- 172.21.15.114:0/2073143172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b292710070 0x55b292709be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95b11b6700 1 -- 172.21.15.114:0/2073143172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b29270a120 msgr2=0x55b29270a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95b11b6700 1 --2- 172.21.15.114:0/2073143172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b29270a120 0x55b29270a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.944+0000 7f95b11b6700 1 -- 172.21.15.114:0/2073143172 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2925d0fb0 con 0x55b29258ae40 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.945+0000 7f95b11b6700 1 --2- 172.21.15.114:0/2073143172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29258ae40 0x55b29270fb30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f95a800e700 tx=0x7f95a8014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5c31e4aa2ea4cca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.945+0000 7f95977fe700 1 -- 172.21.15.114:0/2073143172 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a8004230 con 0x55b29258ae40 2024-05-27T07:45:14.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.945+0000 7f95977fe700 1 -- 172.21.15.114:0/2073143172 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95a80043d0 con 0x55b29258ae40 2024-05-27T07:45:14.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.945+0000 7f95977fe700 1 -- 172.21.15.114:0/2073143172 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a801e770 con 0x55b29258ae40 2024-05-27T07:45:14.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.946+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/2073143172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29258ae40 msgr2=0x55b29270fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:14.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.946+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/2073143172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29258ae40 0x55b29270fb30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f95a800e700 tx=0x7f95a8014000 comp rx=0 tx=0).stop 2024-05-27T07:45:14.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.946+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/2073143172 shutdown_connections 2024-05-27T07:45:14.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.946+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/2073143172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b29270a120 0x55b29270a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.946+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/2073143172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29258ae40 0x55b29270fb30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.946+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/2073143172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b292710070 0x55b292709be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.946+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/2073143172 >> 172.21.15.114:0/2073143172 conn(0x55b29258d770 msgr2=0x55b292704d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:14.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.946+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/2073143172 shutdown_connections 2024-05-27T07:45:14.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.946+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/2073143172 wait complete. 2024-05-27T07:45:14.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.946+0000 7f95c1e2fb80 1 -- start start 2024-05-27T07:45:14.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.946+0000 7f95c1e2fb80 1 Processor -- start 2024-05-27T07:45:14.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95c1e2fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29258ae40 0x55b292713840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95c1e2fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29270a120 0x55b292713d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95c1e2fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 0x55b2927142c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95c1e2fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b29252dfa0 con 0x55b292710070 2024-05-27T07:45:14.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95c1e2fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b29258fda0 con 0x55b29270a120 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95c1e2fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b292716b60 con 0x55b29258ae40 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95c1e2fb80 10 client.0 _ll_register_callbacks cb 0x55b2925be5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95b19b7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 0x55b2927142c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95b11b6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29258ae40 0x55b292713840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95b19b7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 0x55b2927142c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48400/0 (socket says 172.21.15.114:48400) 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95b19b7700 1 -- 172.21.15.114:0/3816015976 learned_addr learned my addr 172.21.15.114:0/3816015976 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95b09b5700 1 --2- 172.21.15.114:0/3816015976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29270a120 0x55b292713d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95b19b7700 1 -- 172.21.15.114:0/3816015976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29258ae40 msgr2=0x55b292713840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95b19b7700 1 --2- 172.21.15.114:0/3816015976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29258ae40 0x55b292713840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95b19b7700 1 -- 172.21.15.114:0/3816015976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29270a120 msgr2=0x55b292713d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95b19b7700 1 --2- 172.21.15.114:0/3816015976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29270a120 0x55b292713d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95b19b7700 1 -- 172.21.15.114:0/3816015976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2925778d0 con 0x55b292710070 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.947+0000 7f95b11b6700 1 --2- 172.21.15.114:0/3816015976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29258ae40 0x55b292713840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f95b19b7700 1 --2- 172.21.15.114:0/3816015976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 0x55b2927142c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f95a4003d80 tx=0x7f95a400e7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c94d99a6e615521 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:14.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f9594ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f9594ff9700 1 -- 172.21.15.114:0/3816015976 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a40092d0 con 0x55b292710070 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f9594ff9700 1 -- 172.21.15.114:0/3816015976 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f95a4007500 con 0x55b292710070 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3816015976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95a80043d0 con 0x55b292710070 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f9594ff9700 1 -- 172.21.15.114:0/3816015976 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f95a4027050 con 0x55b292710070 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f95c1e2fb80 10 client.16599 fetch_fsmap learned FSMap version 33 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3816015976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b2925e1760 con 0x55b292710070 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f9594ff9700 1 -- 172.21.15.114:0/3816015976 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95a80043d0 con 0x55b292710070 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f9594ff9700 1 -- 172.21.15.114:0/3816015976 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 33) ==== 7725+0+0 (secure 0 0 0) 0x7f95a401e520 con 0x55b292710070 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f95c1e2fb80 10 client.16599 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f95c1e2fb80 10 client.16599 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f95c1e2fb80 20 client.16599 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.948+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3816015976 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b292723b00 0x55b292725f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.949+0000 7f95c1e2fb80 4 client.16599 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:14.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.949+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3816015976 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b292639cc0 con 0x55b292723b00 2024-05-27T07:45:14.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.949+0000 7f95b11b6700 1 --2- 172.21.15.114:0/3816015976 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b292723b00 0x55b292725f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.949+0000 7f9594ff9700 10 client.16599 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:14.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.949+0000 7f95b11b6700 1 --2- 172.21.15.114:0/3816015976 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b292723b00 0x55b292725f40 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=beea823bb0712a5e server_cookie=4bff179700d06b33 in_seq=0 out_seq=0 2024-05-27T07:45:14.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f9594ff9700 1 -- 172.21.15.114:0/3816015976 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b292639cc0 con 0x55b292723b00 2024-05-27T07:45:14.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f9594ff9700 10 client.16599 handle_command_reply: tid=0 2024-05-27T07:45:14.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f95c1e2fb80 1 client.16599 shutdown 2024-05-27T07:45:14.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3816015976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 msgr2=0x55b2927142c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:14.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3816015976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 0x55b2927142c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f95a4003d80 tx=0x7f95a400e7d0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3816015976 shutdown_connections 2024-05-27T07:45:14.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3816015976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 0x55b2927142c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3816015976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29258ae40 0x55b292713840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3816015976 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b292723b00 0x55b292725f40 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f9594ff9700 0 client.16599 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:14.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3816015976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29270a120 0x55b292713d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3816015976 >> 172.21.15.114:0/3816015976 conn(0x55b29258d770 msgr2=0x55b292708bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:14.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3816015976 shutdown_connections 2024-05-27T07:45:14.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3816015976 wait complete. 2024-05-27T07:45:14.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.950+0000 7f95c1e2fb80 20 client.16599 trim_cache size 0 max 16384 2024-05-27T07:45:14.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.972+0000 7f95c1e2fb80 1 Processor -- start 2024-05-27T07:45:14.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.972+0000 7f95c1e2fb80 1 -- start start 2024-05-27T07:45:14.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95c1e2fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29258ae40 0x55b29271fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95c1e2fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29270a120 0x55b29272b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95c1e2fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 0x55b29272bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95c1e2fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac1b3640 con 0x55b292710070 2024-05-27T07:45:14.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95c1e2fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac1b06f0 con 0x55b29258ae40 2024-05-27T07:45:14.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95c1e2fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac1b28a0 con 0x55b29270a120 2024-05-27T07:45:14.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95b19b7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 0x55b29272bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95b19b7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 0x55b29272bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48410/0 (socket says 172.21.15.114:48410) 2024-05-27T07:45:14.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95b19b7700 1 -- 172.21.15.114:0/3726347212 learned_addr learned my addr 172.21.15.114:0/3726347212 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:14.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95b09b5700 1 --2- 172.21.15.114:0/3726347212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29270a120 0x55b29272b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95b11b6700 1 --2- 172.21.15.114:0/3726347212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29258ae40 0x55b29271fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95b19b7700 1 -- 172.21.15.114:0/3726347212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29270a120 msgr2=0x55b29272b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:14.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95b19b7700 1 --2- 172.21.15.114:0/3726347212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29270a120 0x55b29272b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95b19b7700 1 -- 172.21.15.114:0/3726347212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29258ae40 msgr2=0x55b29271fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:14.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95b19b7700 1 --2- 172.21.15.114:0/3726347212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29258ae40 0x55b29271fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95b19b7700 1 -- 172.21.15.114:0/3726347212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b29270aa50 con 0x55b292710070 2024-05-27T07:45:14.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95b09b5700 1 --2- 172.21.15.114:0/3726347212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29270a120 0x55b29272b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:14.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.973+0000 7f95b19b7700 1 --2- 172.21.15.114:0/3726347212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 0x55b29272bc40 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f95a401eab0 tx=0x7f95a400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64648183735569bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:14.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.974+0000 7f9596ffd700 1 -- 172.21.15.114:0/3726347212 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a401ec30 con 0x55b292710070 2024-05-27T07:45:14.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.974+0000 7f9596ffd700 1 -- 172.21.15.114:0/3726347212 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95a401e400 con 0x55b292710070 2024-05-27T07:45:14.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.974+0000 7f9596ffd700 1 -- 172.21.15.114:0/3726347212 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a4007500 con 0x55b292710070 2024-05-27T07:45:14.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.974+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3726347212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 msgr2=0x55b29272bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:14.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.974+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3726347212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 0x55b29272bc40 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f95a401eab0 tx=0x7f95a400ef40 comp rx=0 tx=0).stop 2024-05-27T07:45:14.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.975+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3726347212 shutdown_connections 2024-05-27T07:45:14.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.975+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3726347212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b292710070 0x55b29272bc40 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.975+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3726347212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29258ae40 0x55b29271fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.975+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3726347212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b29270a120 0x55b29272b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.975+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3726347212 >> 172.21.15.114:0/3726347212 conn(0x55b29258d770 msgr2=0x55b292709480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.975+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3726347212 shutdown_connections 2024-05-27T07:45:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.975+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3726347212 wait complete. 2024-05-27T07:45:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.975+0000 7f95c1e2fb80 1 -- start start 2024-05-27T07:45:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.975+0000 7f95c1e2fb80 1 Processor -- start 2024-05-27T07:45:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95c1e2fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b29258ae40 0x55b29271bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95c1e2fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29270a120 0x55b29271c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95c1e2fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b292710070 0x55b29272a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95c1e2fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac1b21d0 con 0x55b29258ae40 2024-05-27T07:45:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95c1e2fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac1b0d60 con 0x55b29270a120 2024-05-27T07:45:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95c1e2fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac1b1430 con 0x55b292710070 2024-05-27T07:45:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95c1e2fb80 10 client.0 _ll_register_callbacks cb 0x55b2925be5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95b11b6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b29258ae40 0x55b29271bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95b11b6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b29258ae40 0x55b29271bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48412/0 (socket says 172.21.15.114:48412) 2024-05-27T07:45:14.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95b19b7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b292710070 0x55b29272a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95b11b6700 1 -- 172.21.15.114:0/3675208883 learned_addr learned my addr 172.21.15.114:0/3675208883 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:14.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95b09b5700 1 --2- 172.21.15.114:0/3675208883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29270a120 0x55b29271c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95b19b7700 1 -- 172.21.15.114:0/3675208883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29270a120 msgr2=0x55b29271c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:14.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95b19b7700 1 --2- 172.21.15.114:0/3675208883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29270a120 0x55b29271c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95b19b7700 1 -- 172.21.15.114:0/3675208883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b29258ae40 msgr2=0x55b29271bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:14.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95b19b7700 1 --2- 172.21.15.114:0/3675208883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b29258ae40 0x55b29271bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95b19b7700 1 -- 172.21.15.114:0/3675208883 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b292713610 con 0x55b292710070 2024-05-27T07:45:14.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95b11b6700 1 --2- 172.21.15.114:0/3675208883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b29258ae40 0x55b29271bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:14.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f95b19b7700 1 --2- 172.21.15.114:0/3675208883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b292710070 0x55b29272a6f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f95a402d040 tx=0x7f95a4004dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=894b639c43c5f777 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:14.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.976+0000 7f9594ff9700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:45:14.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f9594ff9700 1 -- 172.21.15.114:0/3675208883 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a403a020 con 0x55b292710070 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f9594ff9700 1 -- 172.21.15.114:0/3675208883 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f95a4030040 con 0x55b292710070 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3675208883 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95a401e400 con 0x55b292710070 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f9594ff9700 1 -- 172.21.15.114:0/3675208883 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f95a4045050 con 0x55b292710070 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f95c1e2fb80 10 client.26197 fetch_fsmap learned FSMap version 33 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3675208883 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b29278d940 con 0x55b292710070 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f9594ff9700 1 -- 172.21.15.114:0/3675208883 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95a401e400 con 0x55b292710070 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f9594ff9700 1 -- 172.21.15.114:0/3675208883 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 33) ==== 7725+0+0 (secure 0 0 0) 0x7f95a40077c0 con 0x55b292710070 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f95c1e2fb80 10 client.26197 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f95c1e2fb80 10 client.26197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f95c1e2fb80 20 client.26197 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3675208883 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b292723b00 0x55b29278fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f95c1e2fb80 4 client.26197 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.977+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3675208883 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b2926334a0 con 0x55b292723b00 2024-05-27T07:45:14.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.978+0000 7f95b11b6700 1 --2- 172.21.15.114:0/3675208883 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b292723b00 0x55b29278fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:14.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.978+0000 7f9594ff9700 10 client.26197 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:14.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.978+0000 7f95b11b6700 1 --2- 172.21.15.114:0/3675208883 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b292723b00 0x55b29278fee0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9ba1dee4d2132ad server_cookie=c6e12bb1367590a7 in_seq=0 out_seq=0 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.980+0000 7f9594ff9700 1 -- 172.21.15.114:0/3675208883 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b2926334a0 con 0x55b292723b00 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.980+0000 7f9594ff9700 10 client.26197 handle_command_reply: tid=0 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.980+0000 7f95c1e2fb80 1 client.26197 shutdown 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3675208883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b292710070 msgr2=0x55b29272a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3675208883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b292710070 0x55b29272a6f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f95a402d040 tx=0x7f95a4004dd0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3675208883 shutdown_connections 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3675208883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b292710070 0x55b29272a6f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3675208883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b29258ae40 0x55b29271bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3675208883 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b292723b00 0x55b29278fee0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95c1e2fb80 1 --2- 172.21.15.114:0/3675208883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b29270a120 0x55b29271c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f9594ff9700 0 client.26197 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3675208883 >> 172.21.15.114:0/3675208883 conn(0x55b29258d770 msgr2=0x55b29270d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3675208883 shutdown_connections 2024-05-27T07:45:14.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95c1e2fb80 1 -- 172.21.15.114:0/3675208883 wait complete. 2024-05-27T07:45:14.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95c1e2fb80 20 client.26197 trim_cache size 0 max 16384 2024-05-27T07:45:14.982 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:45:14.982 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:45:14.982 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:45:14.982 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:45:14.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95b341a700 1 -- 172.21.15.114:0/766327459 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95980b2a70 msgr2=0x7f95980b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:14.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95b341a700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95980b2a70 0x7f95980b4f80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f95ac076270 tx=0x7f95a8009260 comp rx=0 tx=0).stop 2024-05-27T07:45:14.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95b341a700 1 -- 172.21.15.114:0/766327459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95ac1031c0 msgr2=0x7f95ac103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:14.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.981+0000 7f95b341a700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95ac1031c0 0x7f95ac103f90 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f959c000c00 tx=0x7f959c00bfd0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.982+0000 7f95b341a700 1 -- 172.21.15.114:0/766327459 shutdown_connections 2024-05-27T07:45:14.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.982+0000 7f95b341a700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95ac1092f0 0x7f95ac1044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.982+0000 7f95b341a700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95980b2a70 0x7f95980b4f80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.982+0000 7f95b341a700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95ac102870 0x7f95ac103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.982+0000 7f95b341a700 1 --2- 172.21.15.114:0/766327459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95ac1031c0 0x7f95ac103f90 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:14.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.982+0000 7f95b341a700 1 -- 172.21.15.114:0/766327459 >> 172.21.15.114:0/766327459 conn(0x7f95ac0fe6e0 msgr2=0x7f95ac10a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:14.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.982+0000 7f95b341a700 1 -- 172.21.15.114:0/766327459 shutdown_connections 2024-05-27T07:45:14.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:14.982+0000 7f95b341a700 1 -- 172.21.15.114:0/766327459 wait complete. 2024-05-27T07:45:15.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 518932 13.18 MB/sec execute 87 sec latency 39.403 ms 2024-05-27T07:45:15.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:15 smithi163 ceph-mon[36451]: pgmap v174: 129 pgs: 129 active+clean; 49 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 3.9 MiB/s wr, 476 op/s 2024-05-27T07:45:15.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:15 smithi186 ceph-mon[36647]: pgmap v174: 129 pgs: 129 active+clean; 49 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 3.9 MiB/s wr, 476 op/s 2024-05-27T07:45:15.751 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:15 smithi114 ceph-mon[32038]: pgmap v174: 129 pgs: 129 active+clean; 49 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 3.9 MiB/s wr, 476 op/s 2024-05-27T07:45:16.079 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:45:16.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:45:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.225+0000 7f42cb938640 1 -- 172.21.15.114:0/4047973019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c410a140 msgr2=0x7f42c4103d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.225+0000 7f42cb938640 1 --2- 172.21.15.114:0/4047973019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c410a140 0x7f42c4103d80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f42b4009900 tx=0x7f42b402a410 comp rx=0 tx=0).stop 2024-05-27T07:45:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.225+0000 7f42cb938640 1 -- 172.21.15.114:0/4047973019 shutdown_connections 2024-05-27T07:45:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.225+0000 7f42cb938640 1 --2- 172.21.15.114:0/4047973019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42c41043f0 0x7f42c41107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.225+0000 7f42cb938640 1 --2- 172.21.15.114:0/4047973019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c410a140 0x7f42c4103d80 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.225+0000 7f42cb938640 1 --2- 172.21.15.114:0/4047973019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42c4109ca0 0x7f42c4103840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.225+0000 7f42cb938640 1 -- 172.21.15.114:0/4047973019 >> 172.21.15.114:0/4047973019 conn(0x7f42c4079390 msgr2=0x7f42c40797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.225+0000 7f42cb938640 1 -- 172.21.15.114:0/4047973019 shutdown_connections 2024-05-27T07:45:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.225+0000 7f42cb938640 1 -- 172.21.15.114:0/4047973019 wait complete. 2024-05-27T07:45:16.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.226+0000 7f42cb938640 1 Processor -- start 2024-05-27T07:45:16.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.226+0000 7f42cb938640 1 -- start start 2024-05-27T07:45:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42cb938640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c41043f0 0x7f42c4074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42cb938640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42c4109ca0 0x7f42c4075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42cb938640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42c410a140 0x7f42c4075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42cb938640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f42c4113e90 con 0x7f42c41043f0 2024-05-27T07:45:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42cb938640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f42c4113d10 con 0x7f42c410a140 2024-05-27T07:45:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42cb938640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f42c4114010 con 0x7f42c4109ca0 2024-05-27T07:45:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42c96d4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c41043f0 0x7f42c4074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42c8ed3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42c4109ca0 0x7f42c4075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42c96d4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c41043f0 0x7f42c4074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48434/0 (socket says 172.21.15.114:48434) 2024-05-27T07:45:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42c96d4640 1 -- 172.21.15.114:0/1984739884 learned_addr learned my addr 172.21.15.114:0/1984739884 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42c9ed5640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42c410a140 0x7f42c4075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42c96d4640 1 -- 172.21.15.114:0/1984739884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42c4109ca0 msgr2=0x7f42c4075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42c96d4640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42c4109ca0 0x7f42c4075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42c96d4640 1 -- 172.21.15.114:0/1984739884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42c410a140 msgr2=0x7f42c4075a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42c96d4640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42c410a140 0x7f42c4075a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42c96d4640 1 -- 172.21.15.114:0/1984739884 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c41a8c70 con 0x7f42c41043f0 2024-05-27T07:45:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42c8ed3640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42c4109ca0 0x7f42c4075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42c96d4640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c41043f0 0x7f42c4074fc0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f42c000e9f0 tx=0x7f42c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42ba7fc640 1 -- 172.21.15.114:0/1984739884 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c000ccc0 con 0x7f42c41043f0 2024-05-27T07:45:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.227+0000 7f42ba7fc640 1 -- 172.21.15.114:0/1984739884 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42c00054d0 con 0x7f42c41043f0 2024-05-27T07:45:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.228+0000 7f42ba7fc640 1 -- 172.21.15.114:0/1984739884 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c0010430 con 0x7f42c41043f0 2024-05-27T07:45:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.228+0000 7f42cb938640 1 -- 172.21.15.114:0/1984739884 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c41a8f60 con 0x7f42c41043f0 2024-05-27T07:45:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.228+0000 7f42cb938640 1 -- 172.21.15.114:0/1984739884 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c41a93d0 con 0x7f42c41043f0 2024-05-27T07:45:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.231+0000 7f42d27ed740 1 Processor -- start 2024-05-27T07:45:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.231+0000 7f42d27ed740 1 -- start start 2024-05-27T07:45:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.231+0000 7f42d27ed740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71f61b0 0x563fe75973d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.231+0000 7f42d27ed740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe7597910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.231+0000 7f42d27ed740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe7597e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.231+0000 7f42d27ed740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563fe7509c30 con 0x563fe71f61b0 2024-05-27T07:45:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.231+0000 7f42d27ed740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563fe74f97d0 con 0x563fe71cd2f0 2024-05-27T07:45:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.231+0000 7f42d27ed740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563fe74fd810 con 0x563fe71d54d0 2024-05-27T07:45:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.231+0000 7f42c9ed5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71f61b0 0x563fe75973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.231+0000 7f42c8ed3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe7597e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.231+0000 7f42c9ed5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71f61b0 0x563fe75973d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48438/0 (socket says 172.21.15.114:48438) 2024-05-27T07:45:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.231+0000 7f42c9ed5640 1 -- 172.21.15.114:0/3358116170 learned_addr learned my addr 172.21.15.114:0/3358116170 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.232+0000 7f42ba7fc640 1 -- 172.21.15.114:0/1984739884 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f42c00105d0 con 0x7f42c41043f0 2024-05-27T07:45:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.232+0000 7f42c8ed3640 1 -- 172.21.15.114:0/3358116170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 msgr2=0x563fe7597910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.232+0000 7f42c8ed3640 1 --2- 172.21.15.114:0/3358116170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe7597910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.232+0000 7f42c8ed3640 1 -- 172.21.15.114:0/3358116170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71f61b0 msgr2=0x563fe75973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.232+0000 7f42c8ed3640 1 --2- 172.21.15.114:0/3358116170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71f61b0 0x563fe75973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.232+0000 7f42c8ed3640 1 -- 172.21.15.114:0/3358116170 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fe720e210 con 0x563fe71d54d0 2024-05-27T07:45:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.232+0000 7f42c8ed3640 1 --2- 172.21.15.114:0/3358116170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe7597e50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f42b402a930 tx=0x7f42b4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dadf988bafcb3b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.232+0000 7f42af7fe640 1 -- 172.21.15.114:0/3358116170 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4040070 con 0x563fe71d54d0 2024-05-27T07:45:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.232+0000 7f42af7fe640 1 -- 172.21.15.114:0/3358116170 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42b4031720 con 0x563fe71d54d0 2024-05-27T07:45:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.232+0000 7f42af7fe640 1 -- 172.21.15.114:0/3358116170 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4035040 con 0x563fe71d54d0 2024-05-27T07:45:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.233+0000 7f42ba7fc640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f42b00b2950 0x7f42b00b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.233+0000 7f42c9ed5640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f42b00b2950 0x7f42b00b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.233+0000 7f42ba7fc640 1 -- 172.21.15.114:0/1984739884 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f42c00e2cc0 con 0x7f42c41043f0 2024-05-27T07:45:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.233+0000 7f42d27ed740 1 -- 172.21.15.114:0/3358116170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 msgr2=0x563fe7597e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.233+0000 7f42c9ed5640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f42b00b2950 0x7f42b00b4e60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x563fe7594540 tx=0x7f42bc009500 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.233+0000 7f42d27ed740 1 --2- 172.21.15.114:0/3358116170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe7597e50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f42b402a930 tx=0x7f42b4000bc0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.233+0000 7f42d27ed740 1 -- 172.21.15.114:0/3358116170 shutdown_connections 2024-05-27T07:45:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.233+0000 7f42d27ed740 1 --2- 172.21.15.114:0/3358116170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe7597e50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.233+0000 7f42d27ed740 1 --2- 172.21.15.114:0/3358116170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe7597910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.233+0000 7f42d27ed740 1 --2- 172.21.15.114:0/3358116170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71f61b0 0x563fe75973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.233+0000 7f42d27ed740 1 -- 172.21.15.114:0/3358116170 >> 172.21.15.114:0/3358116170 conn(0x563fe71ea300 msgr2=0x563fe7593b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.234+0000 7f42d27ed740 1 -- 172.21.15.114:0/3358116170 shutdown_connections 2024-05-27T07:45:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.234+0000 7f42d27ed740 1 -- 172.21.15.114:0/3358116170 wait complete. 2024-05-27T07:45:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.234+0000 7f42d27ed740 1 -- start start 2024-05-27T07:45:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.234+0000 7f42d27ed740 1 Processor -- start 2024-05-27T07:45:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42d27ed740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe75a3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42d27ed740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe75a3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42d27ed740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71f61b0 0x563fe75a1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42d27ed740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563fe71df790 con 0x563fe71f61b0 2024-05-27T07:45:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42d27ed740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563fe74b5890 con 0x563fe71cd2f0 2024-05-27T07:45:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42d27ed740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563fe75a44a0 con 0x563fe71d54d0 2024-05-27T07:45:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42d27ed740 10 client.0 _ll_register_callbacks cb 0x563fe720da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42c9ed5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71f61b0 0x563fe75a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42c9ed5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71f61b0 0x563fe75a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48446/0 (socket says 172.21.15.114:48446) 2024-05-27T07:45:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42c9ed5640 1 -- 172.21.15.114:0/2817420634 learned_addr learned my addr 172.21.15.114:0/2817420634 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42c8ed3640 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe75a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42c96d4640 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe75a3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42c9ed5640 1 -- 172.21.15.114:0/2817420634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 msgr2=0x563fe75a3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42c9ed5640 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe75a3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42c9ed5640 1 -- 172.21.15.114:0/2817420634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 msgr2=0x563fe75a3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42c9ed5640 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe75a3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42c9ed5640 1 -- 172.21.15.114:0/2817420634 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fe72da620 con 0x563fe71f61b0 2024-05-27T07:45:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42c8ed3640 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe75a3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42c9ed5640 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71f61b0 0x563fe75a1930 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00f700 tx=0x7f42bc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2541936e5366914f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.235+0000 7f42acff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42acff9640 1 -- 172.21.15.114:0/2817420634 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc003970 con 0x563fe71f61b0 2024-05-27T07:45:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42acff9640 1 -- 172.21.15.114:0/2817420634 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f42bc017030 con 0x563fe71f61b0 2024-05-27T07:45:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42d27ed740 1 -- 172.21.15.114:0/2817420634 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42b4031720 con 0x563fe71f61b0 2024-05-27T07:45:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42acff9640 1 -- 172.21.15.114:0/2817420634 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f42bc032050 con 0x563fe71f61b0 2024-05-27T07:45:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42d27ed740 10 client.16617 fetch_fsmap learned FSMap version 33 2024-05-27T07:45:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42d27ed740 1 -- 172.21.15.114:0/2817420634 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563fe75b0850 con 0x563fe71f61b0 2024-05-27T07:45:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42acff9640 1 -- 172.21.15.114:0/2817420634 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42b4031720 con 0x563fe71f61b0 2024-05-27T07:45:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42acff9640 1 -- 172.21.15.114:0/2817420634 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 33) ==== 7725+0+0 (secure 0 0 0) 0x7f42bc004150 con 0x563fe71f61b0 2024-05-27T07:45:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42d27ed740 10 client.16617 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T07:45:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42d27ed740 10 client.16617 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42d27ed740 20 client.16617 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42d27ed740 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563fe71b4d80 0x563fe75b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42d27ed740 4 client.16617 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42d27ed740 1 -- 172.21.15.114:0/2817420634 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fe750bb80 con 0x563fe71b4d80 2024-05-27T07:45:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.236+0000 7f42c96d4640 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563fe71b4d80 0x563fe75b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.237+0000 7f42acff9640 10 client.16617 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.237+0000 7f42c96d4640 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563fe71b4d80 0x563fe75b3d90 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbadbcc927d67c73 server_cookie=339c988a8d53afc6 in_seq=0 out_seq=0 2024-05-27T07:45:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42acff9640 1 -- 172.21.15.114:0/2817420634 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563fe750bb80 con 0x563fe71b4d80 2024-05-27T07:45:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42acff9640 10 client.16617 handle_command_reply: tid=0 2024-05-27T07:45:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42d27ed740 1 client.16617 shutdown 2024-05-27T07:45:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42d27ed740 1 -- 172.21.15.114:0/2817420634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71f61b0 msgr2=0x563fe75a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42d27ed740 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71f61b0 0x563fe75a1930 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00f700 tx=0x7f42bc004ab0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42d27ed740 1 -- 172.21.15.114:0/2817420634 shutdown_connections 2024-05-27T07:45:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42d27ed740 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563fe71b4d80 0x563fe75b3d90 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42acff9640 0 client.16617 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42d27ed740 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71f61b0 0x563fe75a1930 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42d27ed740 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe75a3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42d27ed740 1 --2- 172.21.15.114:0/2817420634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe75a3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42d27ed740 1 -- 172.21.15.114:0/2817420634 >> 172.21.15.114:0/2817420634 conn(0x563fe71ea300 msgr2=0x563fe75990f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42d27ed740 1 -- 172.21.15.114:0/2817420634 shutdown_connections 2024-05-27T07:45:16.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42d27ed740 1 -- 172.21.15.114:0/2817420634 wait complete. 2024-05-27T07:45:16.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.238+0000 7f42d27ed740 20 client.16617 trim_cache size 0 max 16384 2024-05-27T07:45:16.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 521432 13.16 MB/sec execute 88 sec latency 14.192 ms 2024-05-27T07:45:16.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.254+0000 7f42d27ed740 1 Processor -- start 2024-05-27T07:45:16.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.254+0000 7f42d27ed740 1 -- start start 2024-05-27T07:45:16.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42d27ed740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71b4d80 0x563fe759b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42d27ed740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe759b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42d27ed740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71d54d0 0x563fe75b95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42d27ed740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f42c41b4440 con 0x563fe71d54d0 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42d27ed740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f42c41b14f0 con 0x563fe71cd2f0 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42d27ed740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f42c41b36a0 con 0x563fe71b4d80 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42c96d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71b4d80 0x563fe759b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42c9ed5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71d54d0 0x563fe75b95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42c9ed5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71d54d0 0x563fe75b95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48454/0 (socket says 172.21.15.114:48454) 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42c96d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71b4d80 0x563fe759b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54260/0 (socket says 172.21.15.114:54260) 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42c9ed5640 1 -- 172.21.15.114:0/1386606196 learned_addr learned my addr 172.21.15.114:0/1386606196 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42c8ed3640 1 --2- 172.21.15.114:0/1386606196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe759b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42c96d4640 1 -- 172.21.15.114:0/1386606196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 msgr2=0x563fe759b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42c96d4640 1 --2- 172.21.15.114:0/1386606196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe759b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42c96d4640 1 -- 172.21.15.114:0/1386606196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71d54d0 msgr2=0x563fe75b95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42c96d4640 1 --2- 172.21.15.114:0/1386606196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71d54d0 0x563fe75b95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42c96d4640 1 -- 172.21.15.114:0/1386606196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fe7598170 con 0x563fe71b4d80 2024-05-27T07:45:16.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.255+0000 7f42c9ed5640 1 --2- 172.21.15.114:0/1386606196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71d54d0 0x563fe75b95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:16.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.256+0000 7f42c96d4640 1 --2- 172.21.15.114:0/1386606196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71b4d80 0x563fe759b080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f42c00e0060 tx=0x7f42c0004a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a26b8dc1fca3f3df server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:16.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.256+0000 7f42aeffd640 1 -- 172.21.15.114:0/1386606196 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c00e4040 con 0x563fe71b4d80 2024-05-27T07:45:16.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.256+0000 7f42aeffd640 1 -- 172.21.15.114:0/1386606196 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42c008d070 con 0x563fe71b4d80 2024-05-27T07:45:16.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.256+0000 7f42aeffd640 1 -- 172.21.15.114:0/1386606196 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c00b2a50 con 0x563fe71b4d80 2024-05-27T07:45:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.257+0000 7f42d27ed740 1 -- 172.21.15.114:0/1386606196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71b4d80 msgr2=0x563fe759b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.257+0000 7f42d27ed740 1 --2- 172.21.15.114:0/1386606196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71b4d80 0x563fe759b080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f42c00e0060 tx=0x7f42c0004a60 comp rx=0 tx=0).stop 2024-05-27T07:45:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.257+0000 7f42d27ed740 1 -- 172.21.15.114:0/1386606196 shutdown_connections 2024-05-27T07:45:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.257+0000 7f42d27ed740 1 --2- 172.21.15.114:0/1386606196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71d54d0 0x563fe75b95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.257+0000 7f42d27ed740 1 --2- 172.21.15.114:0/1386606196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe759b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.257+0000 7f42d27ed740 1 --2- 172.21.15.114:0/1386606196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71b4d80 0x563fe759b080 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.257+0000 7f42d27ed740 1 -- 172.21.15.114:0/1386606196 >> 172.21.15.114:0/1386606196 conn(0x563fe71ea300 msgr2=0x563fe75a0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.257+0000 7f42d27ed740 1 -- 172.21.15.114:0/1386606196 shutdown_connections 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.257+0000 7f42d27ed740 1 -- 172.21.15.114:0/1386606196 wait complete. 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.257+0000 7f42d27ed740 1 -- start start 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.258+0000 7f42d27ed740 1 Processor -- start 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.258+0000 7f42d27ed740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71b4d80 0x563fe75a9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.258+0000 7f42d27ed740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe75a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.258+0000 7f42d27ed740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe75a5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.258+0000 7f42d27ed740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f42c41b2fd0 con 0x563fe71b4d80 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.258+0000 7f42d27ed740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f42c41b1b60 con 0x563fe71cd2f0 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.258+0000 7f42d27ed740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f42c41b2230 con 0x563fe71d54d0 2024-05-27T07:45:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.258+0000 7f42d27ed740 10 client.0 _ll_register_callbacks cb 0x7f42bc009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.258+0000 7f42c96d4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71b4d80 0x563fe75a9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.258+0000 7f42c96d4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71b4d80 0x563fe75a9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48456/0 (socket says 172.21.15.114:48456) 2024-05-27T07:45:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.258+0000 7f42c9ed5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe75a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.258+0000 7f42c96d4640 1 -- 172.21.15.114:0/1444878330 learned_addr learned my addr 172.21.15.114:0/1444878330 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.259+0000 7f42c8ed3640 1 --2- 172.21.15.114:0/1444878330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe75a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.259+0000 7f42c9ed5640 1 -- 172.21.15.114:0/1444878330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 msgr2=0x563fe75a4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.259+0000 7f42c9ed5640 1 --2- 172.21.15.114:0/1444878330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe75a4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.259+0000 7f42c9ed5640 1 -- 172.21.15.114:0/1444878330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71b4d80 msgr2=0x563fe75a9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.259+0000 7f42c9ed5640 1 --2- 172.21.15.114:0/1444878330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71b4d80 0x563fe75a9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.259+0000 7f42c9ed5640 1 -- 172.21.15.114:0/1444878330 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fe75a5990 con 0x563fe71d54d0 2024-05-27T07:45:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.259+0000 7f42c9ed5640 1 --2- 172.21.15.114:0/1444878330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe75a5450 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009d90 tx=0x7f42bc005c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fe6ce5a54505809 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.259+0000 7f42acff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:45:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.259+0000 7f42acff9640 1 -- 172.21.15.114:0/1444878330 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc01f880 con 0x563fe71d54d0 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.259+0000 7f42acff9640 1 -- 172.21.15.114:0/1444878330 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f42bc01fa20 con 0x563fe71d54d0 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.260+0000 7f42d27ed740 1 -- 172.21.15.114:0/1444878330 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42c008d070 con 0x563fe71d54d0 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.260+0000 7f42acff9640 1 -- 172.21.15.114:0/1444878330 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f42bc043050 con 0x563fe71d54d0 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.260+0000 7f42d27ed740 10 client.26215 fetch_fsmap learned FSMap version 33 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.260+0000 7f42d27ed740 1 -- 172.21.15.114:0/1444878330 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563fe761c100 con 0x563fe71d54d0 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.260+0000 7f42acff9640 1 -- 172.21.15.114:0/1444878330 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42c008d070 con 0x563fe71d54d0 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.260+0000 7f42acff9640 1 -- 172.21.15.114:0/1444878330 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 33) ==== 7725+0+0 (secure 0 0 0) 0x7f42bc03e030 con 0x563fe71d54d0 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.260+0000 7f42d27ed740 10 client.26215 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.260+0000 7f42d27ed740 10 client.26215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.260+0000 7f42d27ed740 20 client.26215 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.260+0000 7f42d27ed740 1 --2- 172.21.15.114:0/1444878330 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563fe71f61b0 0x563fe761e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.260+0000 7f42d27ed740 4 client.26215 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.260+0000 7f42d27ed740 1 -- 172.21.15.114:0/1444878330 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563fe7505360 con 0x563fe71f61b0 2024-05-27T07:45:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.261+0000 7f42c96d4640 1 --2- 172.21.15.114:0/1444878330 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563fe71f61b0 0x563fe761e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.261+0000 7f42acff9640 10 client.26215 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.261+0000 7f42c96d4640 1 --2- 172.21.15.114:0/1444878330 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563fe71f61b0 0x563fe761e820 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b31384c9c09a8545 server_cookie=78f30a38d47de3c9 in_seq=0 out_seq=0 2024-05-27T07:45:16.382 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42acff9640 1 -- 172.21.15.114:0/1444878330 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x563fe7505360 con 0x563fe71f61b0 2024-05-27T07:45:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42acff9640 10 client.26215 handle_command_reply: tid=0 2024-05-27T07:45:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42d27ed740 1 client.26215 shutdown 2024-05-27T07:45:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42d27ed740 1 -- 172.21.15.114:0/1444878330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 msgr2=0x563fe75a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42d27ed740 1 --2- 172.21.15.114:0/1444878330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe75a5450 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009d90 tx=0x7f42bc005c40 comp rx=0 tx=0).stop 2024-05-27T07:45:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42d27ed740 1 -- 172.21.15.114:0/1444878330 shutdown_connections 2024-05-27T07:45:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42d27ed740 1 --2- 172.21.15.114:0/1444878330 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563fe71f61b0 0x563fe761e820 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42d27ed740 1 --2- 172.21.15.114:0/1444878330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563fe71d54d0 0x563fe75a5450 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42d27ed740 1 --2- 172.21.15.114:0/1444878330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563fe71cd2f0 0x563fe75a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42d27ed740 1 --2- 172.21.15.114:0/1444878330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563fe71b4d80 0x563fe75a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42d27ed740 1 -- 172.21.15.114:0/1444878330 >> 172.21.15.114:0/1444878330 conn(0x563fe71ea300 msgr2=0x563fe7597160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42acff9640 0 client.26215 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42d27ed740 1 -- 172.21.15.114:0/1444878330 shutdown_connections 2024-05-27T07:45:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.601+0000 7f42d27ed740 1 -- 172.21.15.114:0/1444878330 wait complete. 2024-05-27T07:45:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.602+0000 7f42d27ed740 20 client.26215 trim_cache size 0 max 16384 2024-05-27T07:45:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:45:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:45:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:45:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.602+0000 7f42cb938640 1 -- 172.21.15.114:0/1984739884 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f42b00b2950 msgr2=0x7f42b00b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.602+0000 7f42cb938640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f42b00b2950 0x7f42b00b4e60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x563fe7594540 tx=0x7f42bc009500 comp rx=0 tx=0).stop 2024-05-27T07:45:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.602+0000 7f42cb938640 1 -- 172.21.15.114:0/1984739884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c41043f0 msgr2=0x7f42c4074fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.602+0000 7f42cb938640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c41043f0 0x7f42c4074fc0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f42c000e9f0 tx=0x7f42c000eeb0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.602+0000 7f42cb938640 1 -- 172.21.15.114:0/1984739884 shutdown_connections 2024-05-27T07:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.602+0000 7f42cb938640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f42b00b2950 0x7f42b00b4e60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.602+0000 7f42cb938640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42c410a140 0x7f42c4075a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.602+0000 7f42cb938640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42c4109ca0 0x7f42c4075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.602+0000 7f42cb938640 1 --2- 172.21.15.114:0/1984739884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c41043f0 0x7f42c4074fc0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.602+0000 7f42cb938640 1 -- 172.21.15.114:0/1984739884 >> 172.21.15.114:0/1984739884 conn(0x7f42c4079390 msgr2=0x7f42c4102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.602+0000 7f42cb938640 1 -- 172.21.15.114:0/1984739884 shutdown_connections 2024-05-27T07:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:16.602+0000 7f42cb938640 1 -- 172.21.15.114:0/1984739884 wait complete. 2024-05-27T07:45:17.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 522796 13.11 MB/sec execute 89 sec latency 136.709 ms 2024-05-27T07:45:17.603 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:17 smithi114 ceph-mon[32038]: pgmap v175: 129 pgs: 129 active+clean; 56 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 4.9 MiB/s wr, 525 op/s 2024-05-27T07:45:17.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:17 smithi163 ceph-mon[36451]: pgmap v175: 129 pgs: 129 active+clean; 56 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 4.9 MiB/s wr, 525 op/s 2024-05-27T07:45:17.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:17 smithi186 ceph-mon[36647]: pgmap v175: 129 pgs: 129 active+clean; 56 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 4.9 MiB/s wr, 525 op/s 2024-05-27T07:45:18.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 525920 13.16 MB/sec execute 90 sec latency 15.619 ms 2024-05-27T07:45:18.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.440+0000 7f84a7dac700 1 -- 172.21.15.114:0/902070066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a0109850 msgr2=0x7f84a01111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:18.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.440+0000 7f84a7dac700 1 --2- 172.21.15.114:0/902070066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a0109850 0x7f84a01111b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f849800b0d0 tx=0x7f849800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:45:18.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.440+0000 7f84a7dac700 1 -- 172.21.15.114:0/902070066 shutdown_connections 2024-05-27T07:45:18.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.440+0000 7f84a7dac700 1 --2- 172.21.15.114:0/902070066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a0109850 0x7f84a01111b0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:18.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.440+0000 7f84a7dac700 1 --2- 172.21.15.114:0/902070066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84a01045d0 0x7f84a01049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:18.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.440+0000 7f84a7dac700 1 --2- 172.21.15.114:0/902070066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a0108f00 0x7f84a0109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:18.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.440+0000 7f84a7dac700 1 -- 172.21.15.114:0/902070066 >> 172.21.15.114:0/902070066 conn(0x7f84a00ffe60 msgr2=0x7f84a01022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:18.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.440+0000 7f84a7dac700 1 -- 172.21.15.114:0/902070066 shutdown_connections 2024-05-27T07:45:18.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.440+0000 7f84a7dac700 1 -- 172.21.15.114:0/902070066 wait complete. 2024-05-27T07:45:18.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.441+0000 7f84a7dac700 1 Processor -- start 2024-05-27T07:45:18.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.441+0000 7f84a7dac700 1 -- start start 2024-05-27T07:45:18.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.441+0000 7f84a7dac700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a01045d0 0x7f84a019adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:18.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.441+0000 7f84a7dac700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a0108f00 0x7f84a019b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:18.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.441+0000 7f84a7dac700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84a0109850 0x7f84a019f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:18.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.441+0000 7f84a7dac700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f84a01144a0 con 0x7f84a0108f00 2024-05-27T07:45:18.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.441+0000 7f84a7dac700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f84a0114320 con 0x7f84a0109850 2024-05-27T07:45:18.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.441+0000 7f84a7dac700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f84a0114620 con 0x7f84a01045d0 2024-05-27T07:45:18.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.441+0000 7f84a5347700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a0108f00 0x7f84a019b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:18.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.441+0000 7f84a5347700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a0108f00 0x7f84a019b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48470/0 (socket says 172.21.15.114:48470) 2024-05-27T07:45:18.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.441+0000 7f84a5347700 1 -- 172.21.15.114:0/2901710496 learned_addr learned my addr 172.21.15.114:0/2901710496 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:18.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.441+0000 7f84a5b48700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a01045d0 0x7f84a019adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:18.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f84a6349700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84a0109850 0x7f84a019f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:18.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f84a5b48700 1 -- 172.21.15.114:0/2901710496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84a0109850 msgr2=0x7f84a019f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:18.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f84a5b48700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84a0109850 0x7f84a019f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:18.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f84a5b48700 1 -- 172.21.15.114:0/2901710496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a0108f00 msgr2=0x7f84a019b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:18.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f84a5b48700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a0108f00 0x7f84a019b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:18.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f84a5b48700 1 -- 172.21.15.114:0/2901710496 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a019fc70 con 0x7f84a01045d0 2024-05-27T07:45:18.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f84a6349700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84a0109850 0x7f84a019f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:45:18.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f84a5347700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a0108f00 0x7f84a019b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:18.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f84a5b48700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a01045d0 0x7f84a019adf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f849c00e9f0 tx=0x7f849c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:18.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f8496ffd700 1 -- 172.21.15.114:0/2901710496 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849c00cbd0 con 0x7f84a01045d0 2024-05-27T07:45:18.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f84a7dac700 1 -- 172.21.15.114:0/2901710496 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84a019ff60 con 0x7f84a01045d0 2024-05-27T07:45:18.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f8496ffd700 1 -- 172.21.15.114:0/2901710496 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f849c00cd70 con 0x7f84a01045d0 2024-05-27T07:45:18.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f8496ffd700 1 -- 172.21.15.114:0/2901710496 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849c018680 con 0x7f84a01045d0 2024-05-27T07:45:18.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.442+0000 7f84a7dac700 1 -- 172.21.15.114:0/2901710496 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84a01a7880 con 0x7f84a01045d0 2024-05-27T07:45:18.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.443+0000 7f84a7dac700 1 -- 172.21.15.114:0/2901710496 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84a00512b0 con 0x7f84a01045d0 2024-05-27T07:45:18.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.445+0000 7f8496ffd700 1 -- 172.21.15.114:0/2901710496 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f849c010430 con 0x7f84a01045d0 2024-05-27T07:45:18.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.446+0000 7f8496ffd700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f848c0b29f0 0x7f848c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:18.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.446+0000 7f8496ffd700 1 -- 172.21.15.114:0/2901710496 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f849c0e30a0 con 0x7f84a01045d0 2024-05-27T07:45:18.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.446+0000 7f84a5347700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f848c0b29f0 0x7f848c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:18.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.446+0000 7f84a5347700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f848c0b29f0 0x7f848c0b4f00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f84a01087e0 tx=0x7f8490009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.453+0000 7f8496ffd700 1 -- 172.21.15.114:0/2901710496 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f849c0aa670 con 0x7f84a01045d0 2024-05-27T07:45:18.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.801+0000 7f84a7dac700 1 -- 172.21.15.114:0/2901710496 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84a019c100 con 0x7f84a01045d0 2024-05-27T07:45:18.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.802+0000 7f8496ffd700 1 -- 172.21.15.114:0/2901710496 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+10999 (secure 0 0 0) 0x7f849c0a9f70 con 0x7f84a01045d0 2024-05-27T07:45:18.801 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:45:18.801 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/199200469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":199200469},{"type":"v1","addr":"172.21.15.186:6835","nonce":199200469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:45:04.221300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26143":{"gid":26143,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/2533834448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2533834448},{"type":"v1","addr":"172.21.15.186:6837","nonce":2533834448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:45:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.804+0000 7f84a7dac700 1 -- 172.21.15.114:0/2901710496 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f848c0b29f0 msgr2=0x7f848c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.804+0000 7f84a7dac700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f848c0b29f0 0x7f848c0b4f00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f84a01087e0 tx=0x7f8490009260 comp rx=0 tx=0).stop 2024-05-27T07:45:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.804+0000 7f84a7dac700 1 -- 172.21.15.114:0/2901710496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a01045d0 msgr2=0x7f84a019adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.804+0000 7f84a7dac700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a01045d0 0x7f84a019adf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f849c00e9f0 tx=0x7f849c00ed00 comp rx=0 tx=0).stop 2024-05-27T07:45:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.804+0000 7f84a7dac700 1 -- 172.21.15.114:0/2901710496 shutdown_connections 2024-05-27T07:45:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.804+0000 7f84a7dac700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f848c0b29f0 0x7f848c0b4f00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.804+0000 7f84a7dac700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84a0109850 0x7f84a019f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.804+0000 7f84a7dac700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a01045d0 0x7f84a019adf0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:18.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.804+0000 7f84a7dac700 1 --2- 172.21.15.114:0/2901710496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a0108f00 0x7f84a019b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:18.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.804+0000 7f84a7dac700 1 -- 172.21.15.114:0/2901710496 >> 172.21.15.114:0/2901710496 conn(0x7f84a00ffe60 msgr2=0x7f84a0107030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:18.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.805+0000 7f84a7dac700 1 -- 172.21.15.114:0/2901710496 shutdown_connections 2024-05-27T07:45:18.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:18.805+0000 7f84a7dac700 1 -- 172.21.15.114:0/2901710496 wait complete. 2024-05-27T07:45:18.804 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 33 2024-05-27T07:45:19.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 527404 13.08 MB/sec execute 91 sec latency 39.794 ms 2024-05-27T07:45:19.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:19 smithi114 ceph-mon[32038]: pgmap v176: 129 pgs: 129 active+clean; 56 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 3.3 MiB/s wr, 243 op/s 2024-05-27T07:45:19.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:19 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2901710496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:45:19.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:19 smithi114 ceph-mon[32038]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:45:19.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:19 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:45:19.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:45:19.673 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:19 smithi186 ceph-mon[36647]: pgmap v176: 129 pgs: 129 active+clean; 56 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 3.3 MiB/s wr, 243 op/s 2024-05-27T07:45:19.674 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:19 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2901710496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:45:19.674 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:19 smithi186 ceph-mon[36647]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:45:19.674 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:19 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:45:19.674 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fad60cd7700 time 2024-05-27T07:45:19.368750+0000 2024-05-27T07:45:19.674 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:45:19.674 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:19.675 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fad7351cba6] 2024-05-27T07:45:19.675 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fad7351cdd1] 2024-05-27T07:45:19.675 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4ac45923d] 2024-05-27T07:45:19.675 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e4ac444971] 2024-05-27T07:45:19.675 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e4ac442fa0] 2024-05-27T07:45:19.675 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4ac443651] 2024-05-27T07:45:19.675 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4ac3d8347] 2024-05-27T07:45:19.675 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4ac0d487e] 2024-05-27T07:45:19.675 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e4ac0d9f2f] 2024-05-27T07:45:19.675 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e4ac0e4969] 2024-05-27T07:45:19.675 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e4ac47d48c] 2024-05-27T07:45:19.675 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e4ac4869cb] 2024-05-27T07:45:19.675 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 13: (Thread::entry_wrapper()+0x43) [0x7fad734eb72d] 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 14: (Thread::_entry_func(void*)+0xd) [0x7fad734eb749] 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fad7208f1ca] 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 16: clone() 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.370+0000 7fad60cd7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fad60cd7700 time 2024-05-27T07:45:19.368750+0000 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fad7351cba6] 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fad7351cdd1] 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4ac45923d] 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e4ac444971] 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e4ac442fa0] 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4ac443651] 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4ac3d8347] 2024-05-27T07:45:19.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4ac0d487e] 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e4ac0d9f2f] 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e4ac0e4969] 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e4ac47d48c] 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e4ac4869cb] 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 13: (Thread::entry_wrapper()+0x43) [0x7fad734eb72d] 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 14: (Thread::_entry_func(void*)+0xd) [0x7fad734eb749] 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fad7208f1ca] 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 16: clone() 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: *** Caught signal (Aborted) ** 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: in thread 7fad60cd7700 thread_name:md_log_replay 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fad72099d20] 2024-05-27T07:45:19.677 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2: gsignal() 2024-05-27T07:45:19.678 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 3: abort() 2024-05-27T07:45:19.678 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fad7351cced] 2024-05-27T07:45:19.678 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fad7351cdd1] 2024-05-27T07:45:19.679 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4ac45923d] 2024-05-27T07:45:19.679 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e4ac444971] 2024-05-27T07:45:19.679 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e4ac442fa0] 2024-05-27T07:45:19.679 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4ac443651] 2024-05-27T07:45:19.679 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4ac3d8347] 2024-05-27T07:45:19.679 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4ac0d487e] 2024-05-27T07:45:19.679 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e4ac0d9f2f] 2024-05-27T07:45:19.679 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e4ac0e4969] 2024-05-27T07:45:19.679 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e4ac47d48c] 2024-05-27T07:45:19.679 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e4ac4869cb] 2024-05-27T07:45:19.679 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 16: (Thread::entry_wrapper()+0x43) [0x7fad734eb72d] 2024-05-27T07:45:19.679 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 17: (Thread::_entry_func(void*)+0xd) [0x7fad734eb749] 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fad7208f1ca] 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 19: clone() 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.372+0000 7fad60cd7700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: in thread 7fad60cd7700 thread_name:md_log_replay 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fad72099d20] 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2: gsignal() 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 3: abort() 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fad7351cced] 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fad7351cdd1] 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4ac45923d] 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e4ac444971] 2024-05-27T07:45:19.680 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e4ac442fa0] 2024-05-27T07:45:19.681 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4ac443651] 2024-05-27T07:45:19.681 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4ac3d8347] 2024-05-27T07:45:19.681 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4ac0d487e] 2024-05-27T07:45:19.681 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e4ac0d9f2f] 2024-05-27T07:45:19.681 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e4ac0e4969] 2024-05-27T07:45:19.681 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e4ac47d48c] 2024-05-27T07:45:19.681 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e4ac4869cb] 2024-05-27T07:45:19.681 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 16: (Thread::entry_wrapper()+0x43) [0x7fad734eb72d] 2024-05-27T07:45:19.681 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 17: (Thread::_entry_func(void*)+0xd) [0x7fad734eb749] 2024-05-27T07:45:19.681 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fad7208f1ca] 2024-05-27T07:45:19.682 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 19: clone() 2024-05-27T07:45:19.682 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:45:19.682 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.682 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: -1> 2024-05-27T07:45:19.370+0000 7fad60cd7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fad60cd7700 time 2024-05-27T07:45:19.368750+0000 2024-05-27T07:45:19.682 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:45:19.682 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.682 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:19.682 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fad7351cba6] 2024-05-27T07:45:19.682 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fad7351cdd1] 2024-05-27T07:45:19.682 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4ac45923d] 2024-05-27T07:45:19.682 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e4ac444971] 2024-05-27T07:45:19.682 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e4ac442fa0] 2024-05-27T07:45:19.682 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4ac443651] 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4ac3d8347] 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4ac0d487e] 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e4ac0d9f2f] 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e4ac0e4969] 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e4ac47d48c] 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e4ac4869cb] 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 13: (Thread::entry_wrapper()+0x43) [0x7fad734eb72d] 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 14: (Thread::_entry_func(void*)+0xd) [0x7fad734eb749] 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fad7208f1ca] 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 16: clone() 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 0> 2024-05-27T07:45:19.372+0000 7fad60cd7700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: in thread 7fad60cd7700 thread_name:md_log_replay 2024-05-27T07:45:19.683 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fad72099d20] 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2: gsignal() 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 3: abort() 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fad7351cced] 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fad7351cdd1] 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4ac45923d] 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e4ac444971] 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e4ac442fa0] 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4ac443651] 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4ac3d8347] 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4ac0d487e] 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e4ac0d9f2f] 2024-05-27T07:45:19.684 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e4ac0e4969] 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e4ac47d48c] 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e4ac4869cb] 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 16: (Thread::entry_wrapper()+0x43) [0x7fad734eb72d] 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 17: (Thread::_entry_func(void*)+0xd) [0x7fad734eb749] 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fad7208f1ca] 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 19: clone() 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: -1> 2024-05-27T07:45:19.370+0000 7fad60cd7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fad60cd7700 time 2024-05-27T07:45:19.368750+0000 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fad7351cba6] 2024-05-27T07:45:19.685 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fad7351cdd1] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4ac45923d] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e4ac444971] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e4ac442fa0] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4ac443651] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4ac3d8347] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4ac0d487e] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e4ac0d9f2f] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e4ac0e4969] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e4ac47d48c] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e4ac4869cb] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 13: (Thread::entry_wrapper()+0x43) [0x7fad734eb72d] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 14: (Thread::_entry_func(void*)+0xd) [0x7fad734eb749] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fad7208f1ca] 2024-05-27T07:45:19.686 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 16: clone() 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 0> 2024-05-27T07:45:19.372+0000 7fad60cd7700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: in thread 7fad60cd7700 thread_name:md_log_replay 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fad72099d20] 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2: gsignal() 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 3: abort() 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fad7351cced] 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fad7351cdd1] 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4ac45923d] 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e4ac444971] 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e4ac442fa0] 2024-05-27T07:45:19.687 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4ac443651] 2024-05-27T07:45:19.688 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4ac3d8347] 2024-05-27T07:45:19.688 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4ac0d487e] 2024-05-27T07:45:19.688 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e4ac0d9f2f] 2024-05-27T07:45:19.688 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e4ac0e4969] 2024-05-27T07:45:19.688 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e4ac47d48c] 2024-05-27T07:45:19.688 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e4ac4869cb] 2024-05-27T07:45:19.688 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 16: (Thread::entry_wrapper()+0x43) [0x7fad734eb72d] 2024-05-27T07:45:19.688 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 17: (Thread::_entry_func(void*)+0xd) [0x7fad734eb749] 2024-05-27T07:45:19.688 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fad7208f1ca] 2024-05-27T07:45:19.688 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 19: clone() 2024-05-27T07:45:19.688 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:45:19.688 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[67389]: 2024-05-27T07:45:19.688 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:19 smithi186 podman[72517]: 2024-05-27 07:45:19.43350142 +0000 UTC m=+0.019989206 container died 65fabfafe5eab5db11d7a13dfbf51d877e354d13d6c287d4c1b5e55f3318c679 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:45:19.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:19 smithi163 ceph-mon[36451]: pgmap v176: 129 pgs: 129 active+clean; 56 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 3.3 MiB/s wr, 243 op/s 2024-05-27T07:45:19.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:19 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2901710496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:45:19.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:19 smithi163 ceph-mon[36451]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:45:19.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:19 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:45:19.795 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:19.977 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:19 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:45:19.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:19 smithi186 systemd[1]: Stopped Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:45:19.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:19 smithi186 systemd[1]: Starting Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:45:20.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 530029 13.10 MB/sec execute 92 sec latency 24.982 ms 2024-05-27T07:45:20.349 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:20 smithi186 podman[72517]: 2024-05-27 07:45:20.073824319 +0000 UTC m=+0.660312105 container remove 65fabfafe5eab5db11d7a13dfbf51d877e354d13d6c287d4c1b5e55f3318c679 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521) 2024-05-27T07:45:20.350 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:20 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:45:20.652 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:20 smithi186 podman[72598]: 2024-05-27 07:45:20.256158414 +0000 UTC m=+0.023721532 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:45:20.652 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:20 smithi186 podman[72598]: 2024-05-27 07:45:20.408665866 +0000 UTC m=+0.176228968 container create e3552dde5a37224eadfd8a9f0f65bcf0ddfed049861957fcfc2497b3c66ea931 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:45:20.977 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:20 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:45:20.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:20 smithi186 podman[72598]: 2024-05-27 07:45:20.653559328 +0000 UTC m=+0.421122431 container init e3552dde5a37224eadfd8a9f0f65bcf0ddfed049861957fcfc2497b3c66ea931 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:45:20.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:20 smithi186 podman[72598]: 2024-05-27 07:45:20.656690164 +0000 UTC m=+0.424253267 container start e3552dde5a37224eadfd8a9f0f65bcf0ddfed049861957fcfc2497b3c66ea931 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-27T07:45:20.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:20 smithi186 ceph-mds[72623]: -- 172.21.15.186:0/1792863470 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c017797a40 con 0x55c0177df600 2024-05-27T07:45:20.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: starting mds.c at 2024-05-27T07:45:20.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:20 smithi186 bash[72598]: e3552dde5a37224eadfd8a9f0f65bcf0ddfed049861957fcfc2497b3c66ea931 2024-05-27T07:45:20.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:45:20 smithi186 systemd[1]: Started Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:45:21.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 533192 13.16 MB/sec execute 93 sec latency 9.602 ms 2024-05-27T07:45:21.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:21 smithi114 ceph-mon[32038]: pgmap v177: 129 pgs: 129 active+clean; 59 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-05-27T07:45:21.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:21 smithi163 ceph-mon[36451]: pgmap v177: 129 pgs: 129 active+clean; 59 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-05-27T07:45:21.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:21 smithi186 ceph-mon[36647]: pgmap v177: 129 pgs: 129 active+clean; 59 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-05-27T07:45:22.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.107+0000 7f71c776d700 1 -- 172.21.15.114:0/696816410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c01029b0 msgr2=0x7f71c0102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:22.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.107+0000 7f71c776d700 1 --2- 172.21.15.114:0/696816410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c01029b0 0x7f71c0102dc0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f71b00098f0 tx=0x7f71b0009c00 comp rx=0 tx=0).stop 2024-05-27T07:45:22.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.107+0000 7f71c776d700 1 -- 172.21.15.114:0/696816410 shutdown_connections 2024-05-27T07:45:22.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.107+0000 7f71c776d700 1 --2- 172.21.15.114:0/696816410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71c0103300 0x7f71c010f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.107+0000 7f71c776d700 1 --2- 172.21.15.114:0/696816410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c01089e0 0x7f71c0108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.107+0000 7f71c776d700 1 --2- 172.21.15.114:0/696816410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c01029b0 0x7f71c0102dc0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.107+0000 7f71c776d700 1 -- 172.21.15.114:0/696816410 >> 172.21.15.114:0/696816410 conn(0x7f71c00fe6e0 msgr2=0x7f71c0100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:22.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.107+0000 7f71c776d700 1 -- 172.21.15.114:0/696816410 shutdown_connections 2024-05-27T07:45:22.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.108+0000 7f71c776d700 1 -- 172.21.15.114:0/696816410 wait complete. 2024-05-27T07:45:22.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.108+0000 7f71c776d700 1 Processor -- start 2024-05-27T07:45:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.108+0000 7f71c776d700 1 -- start start 2024-05-27T07:45:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c776d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71c01029b0 0x7f71c019aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c776d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c0103300 0x7f71c019b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c776d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c01089e0 0x7f71c019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c776d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01122d0 con 0x7f71c0103300 2024-05-27T07:45:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c776d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0112150 con 0x7f71c01089e0 2024-05-27T07:45:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c776d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0112450 con 0x7f71c01029b0 2024-05-27T07:45:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c4d08700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c0103300 0x7f71c019b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c4d08700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c0103300 0x7f71c019b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48490/0 (socket says 172.21.15.114:48490) 2024-05-27T07:45:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c4d08700 1 -- 172.21.15.114:0/500859870 learned_addr learned my addr 172.21.15.114:0/500859870 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c5509700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71c01029b0 0x7f71c019aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c5d0a700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c01089e0 0x7f71c019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c4d08700 1 -- 172.21.15.114:0/500859870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71c01029b0 msgr2=0x7f71c019aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:22.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c4d08700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71c01029b0 0x7f71c019aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c4d08700 1 -- 172.21.15.114:0/500859870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c01089e0 msgr2=0x7f71c019f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:22.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c4d08700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c01089e0 0x7f71c019f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.109+0000 7f71c4d08700 1 -- 172.21.15.114:0/500859870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c019fd70 con 0x7f71c0103300 2024-05-27T07:45:22.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.110+0000 7f71c4d08700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c0103300 0x7f71c019b440 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009230 tx=0x7f71b000f990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:22.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.110+0000 7f71b67fc700 1 -- 172.21.15.114:0/500859870 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b00050c0 con 0x7f71c0103300 2024-05-27T07:45:22.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.110+0000 7f71b67fc700 1 -- 172.21.15.114:0/500859870 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71b000b560 con 0x7f71c0103300 2024-05-27T07:45:22.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.110+0000 7f71c776d700 1 -- 172.21.15.114:0/500859870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71c01a0000 con 0x7f71c0103300 2024-05-27T07:45:22.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.110+0000 7f71b67fc700 1 -- 172.21.15.114:0/500859870 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0003a40 con 0x7f71c0103300 2024-05-27T07:45:22.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.110+0000 7f71c776d700 1 -- 172.21.15.114:0/500859870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71c01a04e0 con 0x7f71c0103300 2024-05-27T07:45:22.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.113+0000 7f71b67fc700 1 -- 172.21.15.114:0/500859870 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f71b0003be0 con 0x7f71c0103300 2024-05-27T07:45:22.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71d6182b80 1 Processor -- start 2024-05-27T07:45:22.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71b67fc700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f71ac0b29f0 0x7f71ac0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71d6182b80 1 -- start start 2024-05-27T07:45:22.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71c5d0a700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f71ac0b29f0 0x7f71ac0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71b67fc700 1 -- 172.21.15.114:0/500859870 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f71b00e38b0 con 0x7f71c0103300 2024-05-27T07:45:22.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71d6182b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750f9cee40 0x55750fb52710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71d6182b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb52c50 0x55750fb4ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71d6182b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb4cf80 0x55750fb4d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71d6182b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55750fa7bc90 con 0x55750f9cee40 2024-05-27T07:45:22.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71d6182b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55750fa6b830 con 0x55750fb4cf80 2024-05-27T07:45:22.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71c5509700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750f9cee40 0x55750fb52710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71d6182b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55750fa6f870 con 0x55750fb52c50 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71c5509700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750f9cee40 0x55750fb52710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48506/0 (socket says 172.21.15.114:48506) 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71c5509700 1 -- 172.21.15.114:0/2583862160 learned_addr learned my addr 172.21.15.114:0/2583862160 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.114+0000 7f71c4d08700 1 --2- 172.21.15.114:0/2583862160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb52c50 0x55750fb4ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.115+0000 7f71c5d0a700 1 --2- 172.21.15.114:0/2583862160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb4cf80 0x55750fb4d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.115+0000 7f71c5509700 1 -- 172.21.15.114:0/2583862160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb52c50 msgr2=0x55750fb4ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.115+0000 7f71c5509700 1 --2- 172.21.15.114:0/2583862160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb52c50 0x55750fb4ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.115+0000 7f71c5509700 1 -- 172.21.15.114:0/2583862160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb4cf80 msgr2=0x55750fb4d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.115+0000 7f71c5509700 1 --2- 172.21.15.114:0/2583862160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb4cf80 0x55750fb4d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.115+0000 7f71c5509700 1 -- 172.21.15.114:0/2583862160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55750f9bb8d0 con 0x55750f9cee40 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.115+0000 7f71c5509700 1 --2- 172.21.15.114:0/2583862160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750f9cee40 0x55750fb52710 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009ca0 tx=0x7f71bc011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ceb708e96d4d55 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.115+0000 7f71c5d0a700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f71ac0b29f0 0x7f71ac0b4f00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f71b8008590 tx=0x7f71b8003d90 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.115+0000 7f71ab7fe700 1 -- 172.21.15.114:0/2583862160 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc011d80 con 0x55750f9cee40 2024-05-27T07:45:22.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.116+0000 7f71ab7fe700 1 -- 172.21.15.114:0/2583862160 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71bc004ce0 con 0x55750f9cee40 2024-05-27T07:45:22.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.116+0000 7f71ab7fe700 1 -- 172.21.15.114:0/2583862160 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc019690 con 0x55750f9cee40 2024-05-27T07:45:22.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.117+0000 7f71d6182b80 1 -- 172.21.15.114:0/2583862160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750f9cee40 msgr2=0x55750fb52710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:22.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.117+0000 7f71d6182b80 1 --2- 172.21.15.114:0/2583862160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750f9cee40 0x55750fb52710 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009ca0 tx=0x7f71bc011440 comp rx=0 tx=0).stop 2024-05-27T07:45:22.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.117+0000 7f71d6182b80 1 -- 172.21.15.114:0/2583862160 shutdown_connections 2024-05-27T07:45:22.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.117+0000 7f71d6182b80 1 --2- 172.21.15.114:0/2583862160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb4cf80 0x55750fb4d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.117+0000 7f71d6182b80 1 --2- 172.21.15.114:0/2583862160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750f9cee40 0x55750fb52710 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.117+0000 7f71d6182b80 1 --2- 172.21.15.114:0/2583862160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb52c50 0x55750fb4ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.117+0000 7f71d6182b80 1 -- 172.21.15.114:0/2583862160 >> 172.21.15.114:0/2583862160 conn(0x55750fa29af0 msgr2=0x55750fb47640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.117+0000 7f71d6182b80 1 -- 172.21.15.114:0/2583862160 shutdown_connections 2024-05-27T07:45:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.117+0000 7f71d6182b80 1 -- 172.21.15.114:0/2583862160 wait complete. 2024-05-27T07:45:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.117+0000 7f71d6182b80 1 -- start start 2024-05-27T07:45:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.118+0000 7f71d6182b80 1 Processor -- start 2024-05-27T07:45:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.118+0000 7f71d6182b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750f9cee40 0x55750fb56340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.118+0000 7f71d6182b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb4cf80 0x55750fb56880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.118+0000 7f71d6182b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 0x55750fb56dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.118+0000 7f71d6182b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55750f971fa0 con 0x55750fb52c50 2024-05-27T07:45:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.118+0000 7f71d6182b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55750fa0d7c0 con 0x55750f9cee40 2024-05-27T07:45:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.118+0000 7f71d6182b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55750fb5dba0 con 0x55750fb4cf80 2024-05-27T07:45:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.118+0000 7f71d6182b80 10 client.0 _ll_register_callbacks cb 0x55750fa06920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.118+0000 7f71c5d0a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 0x55750fb56dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.118+0000 7f71c4d08700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb4cf80 0x55750fb56880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.118+0000 7f71c5d0a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 0x55750fb56dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48512/0 (socket says 172.21.15.114:48512) 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.118+0000 7f71c5d0a700 1 -- 172.21.15.114:0/3441763357 learned_addr learned my addr 172.21.15.114:0/3441763357 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.118+0000 7f71c5509700 1 --2- 172.21.15.114:0/3441763357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750f9cee40 0x55750fb56340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71c5d0a700 1 -- 172.21.15.114:0/3441763357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb4cf80 msgr2=0x55750fb56880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71c5d0a700 1 --2- 172.21.15.114:0/3441763357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb4cf80 0x55750fb56880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71c5d0a700 1 -- 172.21.15.114:0/3441763357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750f9cee40 msgr2=0x55750fb56340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71c5d0a700 1 --2- 172.21.15.114:0/3441763357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750f9cee40 0x55750fb56340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71c5d0a700 1 -- 172.21.15.114:0/3441763357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55750fa0a690 con 0x55750fb52c50 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71c4d08700 1 --2- 172.21.15.114:0/3441763357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb4cf80 0x55750fb56880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71c5d0a700 1 --2- 172.21.15.114:0/3441763357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 0x55750fb56dc0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f71b800a6b0 tx=0x7f71b8004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2765a760e809b0c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71a8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71a8ff9700 1 -- 172.21.15.114:0/3441763357 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b8027020 con 0x55750fb52c50 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71a8ff9700 1 -- 172.21.15.114:0/3441763357 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f71b80074a0 con 0x55750fb52c50 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71d6182b80 1 -- 172.21.15.114:0/3441763357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71bc004ce0 con 0x55750fb52c50 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71a8ff9700 1 -- 172.21.15.114:0/3441763357 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f71b8032050 con 0x55750fb52c50 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71d6182b80 10 client.16647 fetch_fsmap learned FSMap version 35 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.119+0000 7f71d6182b80 1 -- 172.21.15.114:0/3441763357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55750fb68600 con 0x55750fb52c50 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.120+0000 7f71a8ff9700 1 -- 172.21.15.114:0/3441763357 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71bc004ce0 con 0x55750fb52c50 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.120+0000 7f71a8ff9700 1 -- 172.21.15.114:0/3441763357 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 35) ==== 7725+0+0 (secure 0 0 0) 0x7f71b8018750 con 0x55750fb52c50 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.120+0000 7f71d6182b80 10 client.16647 fetch_fsmap finished waiting for FSMap version 35 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.120+0000 7f71d6182b80 10 client.16647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.120+0000 7f71d6182b80 20 client.16647 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.120+0000 7f71d6182b80 1 --2- 172.21.15.114:0/3441763357 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55750fb6ac40 0x55750fb6d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.120+0000 7f71d6182b80 4 client.16647 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.120+0000 7f71d6182b80 1 -- 172.21.15.114:0/3441763357 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55750fb189f0 con 0x55750fb6ac40 2024-05-27T07:45:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.120+0000 7f71c5509700 1 --2- 172.21.15.114:0/3441763357 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55750fb6ac40 0x55750fb6d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.120+0000 7f71a8ff9700 10 client.16647 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:22.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.121+0000 7f71c5509700 1 --2- 172.21.15.114:0/3441763357 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55750fb6ac40 0x55750fb6d0b0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcf1dd08e2095532 server_cookie=ccffce8a28a6380 in_seq=0 out_seq=0 2024-05-27T07:45:22.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.121+0000 7f71a8ff9700 1 -- 172.21.15.114:0/3441763357 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55750fb189f0 con 0x55750fb6ac40 2024-05-27T07:45:22.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.121+0000 7f71a8ff9700 10 client.16647 handle_command_reply: tid=0 2024-05-27T07:45:22.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.121+0000 7f71d6182b80 1 client.16647 shutdown 2024-05-27T07:45:22.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.121+0000 7f71d6182b80 1 -- 172.21.15.114:0/3441763357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 msgr2=0x55750fb56dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:22.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.122+0000 7f71d6182b80 1 --2- 172.21.15.114:0/3441763357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 0x55750fb56dc0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f71b800a6b0 tx=0x7f71b8004b40 comp rx=0 tx=0).stop 2024-05-27T07:45:22.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.122+0000 7f71d6182b80 1 -- 172.21.15.114:0/3441763357 shutdown_connections 2024-05-27T07:45:22.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.122+0000 7f71d6182b80 1 --2- 172.21.15.114:0/3441763357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 0x55750fb56dc0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.122+0000 7f71d6182b80 1 --2- 172.21.15.114:0/3441763357 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55750fb6ac40 0x55750fb6d0b0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.122+0000 7f71d6182b80 1 --2- 172.21.15.114:0/3441763357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750f9cee40 0x55750fb56340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.122+0000 7f71d6182b80 1 --2- 172.21.15.114:0/3441763357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb4cf80 0x55750fb56880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.122+0000 7f71a8ff9700 0 client.16647 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:22.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.122+0000 7f71d6182b80 1 -- 172.21.15.114:0/3441763357 >> 172.21.15.114:0/3441763357 conn(0x55750fa29af0 msgr2=0x55750fb4ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:22.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.122+0000 7f71d6182b80 1 -- 172.21.15.114:0/3441763357 shutdown_connections 2024-05-27T07:45:22.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.122+0000 7f71d6182b80 1 -- 172.21.15.114:0/3441763357 wait complete. 2024-05-27T07:45:22.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.122+0000 7f71d6182b80 20 client.16647 trim_cache size 0 max 16384 2024-05-27T07:45:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.138+0000 7f71d6182b80 1 Processor -- start 2024-05-27T07:45:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71d6182b80 1 -- start start 2024-05-27T07:45:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71d6182b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750f9cee40 0x55750fb4ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71d6182b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb4cf80 0x55750fb56c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71d6182b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 0x55750fb66c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71d6182b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01af230 con 0x55750fb52c50 2024-05-27T07:45:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71d6182b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01ac300 con 0x55750fb4cf80 2024-05-27T07:45:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71d6182b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01ae490 con 0x55750f9cee40 2024-05-27T07:45:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71c5d0a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 0x55750fb66c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71c5d0a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 0x55750fb66c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48516/0 (socket says 172.21.15.114:48516) 2024-05-27T07:45:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71c5d0a700 1 -- 172.21.15.114:0/1565496138 learned_addr learned my addr 172.21.15.114:0/1565496138 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71c5509700 1 --2- 172.21.15.114:0/1565496138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750f9cee40 0x55750fb4ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71c4d08700 1 --2- 172.21.15.114:0/1565496138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb4cf80 0x55750fb56c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71c5d0a700 1 -- 172.21.15.114:0/1565496138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750f9cee40 msgr2=0x55750fb4ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71c5d0a700 1 --2- 172.21.15.114:0/1565496138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750f9cee40 0x55750fb4ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71c5d0a700 1 -- 172.21.15.114:0/1565496138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb4cf80 msgr2=0x55750fb56c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71c5d0a700 1 --2- 172.21.15.114:0/1565496138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb4cf80 0x55750fb56c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71c5d0a700 1 -- 172.21.15.114:0/1565496138 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55750fb423d0 con 0x55750fb52c50 2024-05-27T07:45:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.139+0000 7f71c5d0a700 1 --2- 172.21.15.114:0/1565496138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 0x55750fb66c90 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f71b8008450 tx=0x7f71b800a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e09145d341dbd6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:22.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.140+0000 7f71aaffd700 1 -- 172.21.15.114:0/1565496138 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b803a030 con 0x55750fb52c50 2024-05-27T07:45:22.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.140+0000 7f71aaffd700 1 -- 172.21.15.114:0/1565496138 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71b8018440 con 0x55750fb52c50 2024-05-27T07:45:22.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.140+0000 7f71aaffd700 1 -- 172.21.15.114:0/1565496138 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b8027040 con 0x55750fb52c50 2024-05-27T07:45:22.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.141+0000 7f71d6182b80 1 -- 172.21.15.114:0/1565496138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 msgr2=0x55750fb66c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:22.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.141+0000 7f71d6182b80 1 --2- 172.21.15.114:0/1565496138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 0x55750fb66c90 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f71b8008450 tx=0x7f71b800a5b0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.141+0000 7f71d6182b80 1 -- 172.21.15.114:0/1565496138 shutdown_connections 2024-05-27T07:45:22.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.141+0000 7f71d6182b80 1 --2- 172.21.15.114:0/1565496138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750fb52c50 0x55750fb66c90 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.141+0000 7f71d6182b80 1 --2- 172.21.15.114:0/1565496138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750f9cee40 0x55750fb4ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.141+0000 7f71d6182b80 1 --2- 172.21.15.114:0/1565496138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb4cf80 0x55750fb56c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.141+0000 7f71d6182b80 1 -- 172.21.15.114:0/1565496138 >> 172.21.15.114:0/1565496138 conn(0x55750fa29af0 msgr2=0x55750fb471c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.141+0000 7f71d6182b80 1 -- 172.21.15.114:0/1565496138 shutdown_connections 2024-05-27T07:45:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.141+0000 7f71d6182b80 1 -- 172.21.15.114:0/1565496138 wait complete. 2024-05-27T07:45:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.141+0000 7f71d6182b80 1 -- start start 2024-05-27T07:45:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.141+0000 7f71d6182b80 1 Processor -- start 2024-05-27T07:45:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.142+0000 7f71d6182b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750f9cee40 0x55750fb63630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.142+0000 7f71d6182b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb4cf80 0x55750fb5de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.142+0000 7f71d6182b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb52c50 0x55750fb5e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.142+0000 7f71d6182b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01addc0 con 0x55750f9cee40 2024-05-27T07:45:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.142+0000 7f71d6182b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01ac950 con 0x55750fb52c50 2024-05-27T07:45:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.142+0000 7f71d6182b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01ad020 con 0x55750fb4cf80 2024-05-27T07:45:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.142+0000 7f71d6182b80 10 client.0 _ll_register_callbacks cb 0x55750fa06920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.142+0000 7f71c4d08700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb4cf80 0x55750fb5de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.142+0000 7f71c4d08700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb4cf80 0x55750fb5de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54346/0 (socket says 172.21.15.114:54346) 2024-05-27T07:45:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.142+0000 7f71c4d08700 1 -- 172.21.15.114:0/2132375839 learned_addr learned my addr 172.21.15.114:0/2132375839 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.142+0000 7f71c5d0a700 1 --2- 172.21.15.114:0/2132375839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb52c50 0x55750fb5e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.142+0000 7f71c5509700 1 --2- 172.21.15.114:0/2132375839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750f9cee40 0x55750fb63630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.143+0000 7f71c4d08700 1 -- 172.21.15.114:0/2132375839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb52c50 msgr2=0x55750fb5e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.143+0000 7f71c4d08700 1 --2- 172.21.15.114:0/2132375839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb52c50 0x55750fb5e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.143+0000 7f71c4d08700 1 -- 172.21.15.114:0/2132375839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750f9cee40 msgr2=0x55750fb63630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.143+0000 7f71c4d08700 1 --2- 172.21.15.114:0/2132375839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750f9cee40 0x55750fb63630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.143+0000 7f71c4d08700 1 -- 172.21.15.114:0/2132375839 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55750fb5eba0 con 0x55750fb4cf80 2024-05-27T07:45:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.143+0000 7f71c4d08700 1 --2- 172.21.15.114:0/2132375839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb4cf80 0x55750fb5de40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f71b00a68f0 tx=0x7f71b00a4f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18e8d862da0990d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.143+0000 7f71a8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:45:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.143+0000 7f71a8ff9700 1 -- 172.21.15.114:0/2132375839 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b00c3020 con 0x55750fb4cf80 2024-05-27T07:45:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.143+0000 7f71d6182b80 1 -- 172.21.15.114:0/2132375839 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71b8018440 con 0x55750fb4cf80 2024-05-27T07:45:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.143+0000 7f71a8ff9700 1 -- 172.21.15.114:0/2132375839 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f71b00a6b00 con 0x55750fb4cf80 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.143+0000 7f71a8ff9700 1 -- 172.21.15.114:0/2132375839 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f71b00aa380 con 0x55750fb4cf80 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.143+0000 7f71d6182b80 10 client.26245 fetch_fsmap learned FSMap version 35 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.143+0000 7f71d6182b80 1 -- 172.21.15.114:0/2132375839 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55750fbd47e0 con 0x55750fb4cf80 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.144+0000 7f71a8ff9700 1 -- 172.21.15.114:0/2132375839 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71b8018440 con 0x55750fb4cf80 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.144+0000 7f71a8ff9700 1 -- 172.21.15.114:0/2132375839 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 35) ==== 7725+0+0 (secure 0 0 0) 0x7f71b00cb020 con 0x55750fb4cf80 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.144+0000 7f71d6182b80 10 client.26245 fetch_fsmap finished waiting for FSMap version 35 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.144+0000 7f71d6182b80 10 client.26245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.144+0000 7f71d6182b80 20 client.26245 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.144+0000 7f71d6182b80 1 --2- 172.21.15.114:0/2132375839 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55750fbd4ea0 0x55750fbd72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.144+0000 7f71d6182b80 4 client.26245 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.144+0000 7f71d6182b80 1 -- 172.21.15.114:0/2132375839 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55750fa7dbe0 con 0x55750fbd4ea0 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.144+0000 7f71c5509700 1 --2- 172.21.15.114:0/2132375839 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55750fbd4ea0 0x55750fbd72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.145+0000 7f71a8ff9700 10 client.26245 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.145+0000 7f71c5509700 1 --2- 172.21.15.114:0/2132375839 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55750fbd4ea0 0x55750fbd72c0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16e73de6f385b4a server_cookie=ce774dd23b5258f9 in_seq=0 out_seq=0 2024-05-27T07:45:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71a8ff9700 1 -- 172.21.15.114:0/2132375839 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55750fa7dbe0 con 0x55750fbd4ea0 2024-05-27T07:45:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71a8ff9700 10 client.26245 handle_command_reply: tid=0 2024-05-27T07:45:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71d6182b80 1 client.26245 shutdown 2024-05-27T07:45:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71d6182b80 1 -- 172.21.15.114:0/2132375839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb4cf80 msgr2=0x55750fb5de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71d6182b80 1 --2- 172.21.15.114:0/2132375839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb4cf80 0x55750fb5de40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f71b00a68f0 tx=0x7f71b00a4f90 comp rx=0 tx=0).stop 2024-05-27T07:45:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71d6182b80 1 -- 172.21.15.114:0/2132375839 shutdown_connections 2024-05-27T07:45:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71d6182b80 1 --2- 172.21.15.114:0/2132375839 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55750fbd4ea0 0x55750fbd72c0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71d6182b80 1 --2- 172.21.15.114:0/2132375839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55750fb52c50 0x55750fb5e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71d6182b80 1 --2- 172.21.15.114:0/2132375839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55750f9cee40 0x55750fb63630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71a8ff9700 0 client.26245 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71d6182b80 1 --2- 172.21.15.114:0/2132375839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55750fb4cf80 0x55750fb5de40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71d6182b80 1 -- 172.21.15.114:0/2132375839 >> 172.21.15.114:0/2132375839 conn(0x55750fa29af0 msgr2=0x55750fb56c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:22.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71d6182b80 1 -- 172.21.15.114:0/2132375839 shutdown_connections 2024-05-27T07:45:22.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71d6182b80 1 -- 172.21.15.114:0/2132375839 wait complete. 2024-05-27T07:45:22.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.147+0000 7f71d6182b80 20 client.26245 trim_cache size 0 max 16384 2024-05-27T07:45:22.149 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:45:22.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.148+0000 7f71c776d700 1 -- 172.21.15.114:0/500859870 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f71ac0b29f0 msgr2=0x7f71ac0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:22.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.148+0000 7f71c776d700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f71ac0b29f0 0x7f71ac0b4f00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f71b8008590 tx=0x7f71b8003d90 comp rx=0 tx=0).stop 2024-05-27T07:45:22.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.148+0000 7f71c776d700 1 -- 172.21.15.114:0/500859870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c0103300 msgr2=0x7f71c019b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:22.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.148+0000 7f71c776d700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c0103300 0x7f71c019b440 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009230 tx=0x7f71b000f990 comp rx=0 tx=0).stop 2024-05-27T07:45:22.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.148+0000 7f71c776d700 1 -- 172.21.15.114:0/500859870 shutdown_connections 2024-05-27T07:45:22.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.148+0000 7f71c776d700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f71ac0b29f0 0x7f71ac0b4f00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.148+0000 7f71c776d700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c01089e0 0x7f71c019f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.148+0000 7f71c776d700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71c01029b0 0x7f71c019aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.148+0000 7f71c776d700 1 --2- 172.21.15.114:0/500859870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c0103300 0x7f71c019b440 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:22.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.148+0000 7f71c776d700 1 -- 172.21.15.114:0/500859870 >> 172.21.15.114:0/500859870 conn(0x7f71c00fe6e0 msgr2=0x7f71c0105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:22.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.148+0000 7f71c776d700 1 -- 172.21.15.114:0/500859870 shutdown_connections 2024-05-27T07:45:22.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:22.148+0000 7f71c776d700 1 -- 172.21.15.114:0/500859870 wait complete. 2024-05-27T07:45:22.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 535766 13.14 MB/sec execute 94 sec latency 24.676 ms 2024-05-27T07:45:22.465 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:22 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6834/2936519309,v1:172.21.15.186:6835/2936519309] up:boot 2024-05-27T07:45:22.465 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:22 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:45:22.465 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:22 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:45:22.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:22 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6834/2936519309,v1:172.21.15.186:6835/2936519309] up:boot 2024-05-27T07:45:22.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:22 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:45:22.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:22 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:45:22.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:22 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6834/2936519309,v1:172.21.15.186:6835/2936519309] up:boot 2024-05-27T07:45:22.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:22 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:45:22.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:22 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:45:23.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 537692 13.11 MB/sec execute 95 sec latency 44.875 ms 2024-05-27T07:45:23.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:23 smithi186 ceph-mon[36647]: pgmap v178: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 4.4 MiB/s wr, 502 op/s 2024-05-27T07:45:23.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:23 smithi163 ceph-mon[36451]: pgmap v178: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 4.4 MiB/s wr, 502 op/s 2024-05-27T07:45:23.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:23 smithi114 ceph-mon[32038]: pgmap v178: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 4.4 MiB/s wr, 502 op/s 2024-05-27T07:45:23.901 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:45:23.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:45:24.205 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:24.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 540215 13.14 MB/sec execute 96 sec latency 91.584 ms 2024-05-27T07:45:25.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 542949 13.14 MB/sec execute 97 sec latency 16.326 ms 2024-05-27T07:45:25.713 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:25 smithi114 ceph-mon[32038]: pgmap v179: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 3.4 MiB/s wr, 433 op/s 2024-05-27T07:45:25.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:25 smithi163 ceph-mon[36451]: pgmap v179: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 3.4 MiB/s wr, 433 op/s 2024-05-27T07:45:25.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:25 smithi186 ceph-mon[36647]: pgmap v179: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 3.4 MiB/s wr, 433 op/s 2024-05-27T07:45:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.171+0000 7f046af72700 1 -- 172.21.15.114:0/4154127620 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0464109850 msgr2=0x7f04641111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.171+0000 7f046af72700 1 --2- 172.21.15.114:0/4154127620 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0464109850 0x7f04641111b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f045c00b0a0 tx=0x7f045c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.171+0000 7f046af72700 1 -- 172.21.15.114:0/4154127620 shutdown_connections 2024-05-27T07:45:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.171+0000 7f046af72700 1 --2- 172.21.15.114:0/4154127620 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0464109850 0x7f04641111b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.171+0000 7f046af72700 1 --2- 172.21.15.114:0/4154127620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04641045d0 0x7f04641049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.171+0000 7f046af72700 1 --2- 172.21.15.114:0/4154127620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0464108f00 0x7f0464109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.171+0000 7f046af72700 1 -- 172.21.15.114:0/4154127620 >> 172.21.15.114:0/4154127620 conn(0x7f04640ffec0 msgr2=0x7f0464102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.171+0000 7f046af72700 1 -- 172.21.15.114:0/4154127620 shutdown_connections 2024-05-27T07:45:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.172+0000 7f046af72700 1 -- 172.21.15.114:0/4154127620 wait complete. 2024-05-27T07:45:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.172+0000 7f046af72700 1 Processor -- start 2024-05-27T07:45:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.172+0000 7f046af72700 1 -- start start 2024-05-27T07:45:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f046af72700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04641045d0 0x7f04641a21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f046af72700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0464108f00 0x7f04641a26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f046af72700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0464109850 0x7f04641ad230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f046af72700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04641144d0 con 0x7f04641045d0 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f046af72700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0464114350 con 0x7f0464108f00 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f046af72700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0464114650 con 0x7f0464109850 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f0463fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0464108f00 0x7f04641a26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f0468d0e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04641045d0 0x7f04641a21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f0463fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0464108f00 0x7f04641a26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:41680/0 (socket says 172.21.15.114:41680) 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f0463fff700 1 -- 172.21.15.114:0/3375377510 learned_addr learned my addr 172.21.15.114:0/3375377510 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f046950f700 1 --2- 172.21.15.114:0/3375377510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0464109850 0x7f04641ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f0463fff700 1 -- 172.21.15.114:0/3375377510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0464109850 msgr2=0x7f04641ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f0463fff700 1 --2- 172.21.15.114:0/3375377510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0464109850 0x7f04641ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f0463fff700 1 -- 172.21.15.114:0/3375377510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04641045d0 msgr2=0x7f04641a21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f0463fff700 1 --2- 172.21.15.114:0/3375377510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04641045d0 0x7f04641a21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.173+0000 7f0463fff700 1 -- 172.21.15.114:0/3375377510 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04641ad770 con 0x7f0464108f00 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.174+0000 7f0463fff700 1 --2- 172.21.15.114:0/3375377510 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0464108f00 0x7f04641a26f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f045800e970 tx=0x7f045800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.174+0000 7f0461ffb700 1 -- 172.21.15.114:0/3375377510 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458004a30 con 0x7f0464108f00 2024-05-27T07:45:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.174+0000 7f0461ffb700 1 -- 172.21.15.114:0/3375377510 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f045800cc90 con 0x7f0464108f00 2024-05-27T07:45:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.174+0000 7f0461ffb700 1 -- 172.21.15.114:0/3375377510 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458005610 con 0x7f0464108f00 2024-05-27T07:45:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.174+0000 7f046af72700 1 -- 172.21.15.114:0/3375377510 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04641ad940 con 0x7f0464108f00 2024-05-27T07:45:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.174+0000 7f046af72700 1 -- 172.21.15.114:0/3375377510 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04641ade30 con 0x7f0464108f00 2024-05-27T07:45:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.177+0000 7f0461ffb700 1 -- 172.21.15.114:0/3375377510 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f045800c7e0 con 0x7f0464108f00 2024-05-27T07:45:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.179+0000 7f0461ffb700 1 --2- 172.21.15.114:0/3375377510 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f044c0b2af0 0x7f044c0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.179+0000 7f0468d0e700 1 --2- 172.21.15.114:0/3375377510 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f044c0b2af0 0x7f044c0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.179+0000 7f0461ffb700 1 -- 172.21.15.114:0/3375377510 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f04580e2ad0 con 0x7f0464108f00 2024-05-27T07:45:26.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.179+0000 7f0479987b80 1 Processor -- start 2024-05-27T07:45:26.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.179+0000 7f0479987b80 1 -- start start 2024-05-27T07:45:26.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.179+0000 7f0468d0e700 1 --2- 172.21.15.114:0/3375377510 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f044c0b2af0 0x7f044c0b5000 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0454009680 tx=0x7f0454009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f0479987b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1a9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f0479987b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1a3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f0479987b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f0479987b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e7ae0d1df0 con 0x55e7ae024ec0 2024-05-27T07:45:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f0479987b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e7ae0c1990 con 0x55e7ae1aa0f0 2024-05-27T07:45:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f0479987b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e7ae0c59d0 con 0x55e7ae1a41a0 2024-05-27T07:45:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f046950f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f046950f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1a45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:38364/0 (socket says 172.21.15.114:38364) 2024-05-27T07:45:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f046950f700 1 -- 172.21.15.114:0/1890203251 learned_addr learned my addr 172.21.15.114:0/1890203251 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f0463fff700 1 --2- 172.21.15.114:0/1890203251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1a3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f0468d0e700 1 --2- 172.21.15.114:0/1890203251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1a9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f046950f700 1 -- 172.21.15.114:0/1890203251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 msgr2=0x55e7ae1a3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:26.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f046950f700 1 --2- 172.21.15.114:0/1890203251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1a3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f046950f700 1 -- 172.21.15.114:0/1890203251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 msgr2=0x55e7ae1a9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:26.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f046950f700 1 --2- 172.21.15.114:0/1890203251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1a9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f046950f700 1 -- 172.21.15.114:0/1890203251 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7ae06b030 con 0x55e7ae1a41a0 2024-05-27T07:45:26.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f0468d0e700 1 --2- 172.21.15.114:0/1890203251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1a9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:45:26.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f046950f700 1 --2- 172.21.15.114:0/1890203251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1a45b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f045c015000 tx=0x7f045c009db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=351b85566f308847 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:26.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.180+0000 7f044affd700 1 -- 172.21.15.114:0/1890203251 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045c020070 con 0x55e7ae1a41a0 2024-05-27T07:45:26.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.181+0000 7f044affd700 1 -- 172.21.15.114:0/1890203251 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f045c00bcb0 con 0x55e7ae1a41a0 2024-05-27T07:45:26.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.181+0000 7f044affd700 1 -- 172.21.15.114:0/1890203251 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045c007c70 con 0x55e7ae1a41a0 2024-05-27T07:45:26.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.181+0000 7f0479987b80 1 -- 172.21.15.114:0/1890203251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1a41a0 msgr2=0x55e7ae1a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:26.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.181+0000 7f0479987b80 1 --2- 172.21.15.114:0/1890203251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1a45b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f045c015000 tx=0x7f045c009db0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.182+0000 7f0479987b80 1 -- 172.21.15.114:0/1890203251 shutdown_connections 2024-05-27T07:45:26.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.182+0000 7f0479987b80 1 --2- 172.21.15.114:0/1890203251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1a45b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.182+0000 7f0479987b80 1 --2- 172.21.15.114:0/1890203251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1a9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.182+0000 7f0479987b80 1 --2- 172.21.15.114:0/1890203251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1a3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.182+0000 7f0479987b80 1 -- 172.21.15.114:0/1890203251 >> 172.21.15.114:0/1890203251 conn(0x55e7ae0277f0 msgr2=0x55e7ae19ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:26.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.182+0000 7f0479987b80 1 -- 172.21.15.114:0/1890203251 shutdown_connections 2024-05-27T07:45:26.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.182+0000 7f0479987b80 1 -- 172.21.15.114:0/1890203251 wait complete. 2024-05-27T07:45:26.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.182+0000 7f0479987b80 1 -- start start 2024-05-27T07:45:26.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.182+0000 7f0479987b80 1 Processor -- start 2024-05-27T07:45:26.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f0479987b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1ad920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f0479987b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1ade60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f0479987b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f0479987b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e7adfc8020 con 0x55e7ae024ec0 2024-05-27T07:45:26.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f0479987b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e7ae029e20 con 0x55e7ae1a41a0 2024-05-27T07:45:26.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f0479987b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e7ae1b0b80 con 0x55e7ae1aa0f0 2024-05-27T07:45:26.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f0479987b80 10 client.0 _ll_register_callbacks cb 0x55e7ae058630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:26.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f046950f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f046950f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1ae3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:38378/0 (socket says 172.21.15.114:38378) 2024-05-27T07:45:26.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f046950f700 1 -- 172.21.15.114:0/2473631511 learned_addr learned my addr 172.21.15.114:0/2473631511 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:26.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f0468d0e700 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f0463fff700 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f046950f700 1 -- 172.21.15.114:0/2473631511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1a41a0 msgr2=0x55e7ae1ade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:26.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f046950f700 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1ade60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f046950f700 1 -- 172.21.15.114:0/2473631511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 msgr2=0x55e7ae1ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:26.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f046950f700 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f046950f700 1 -- 172.21.15.114:0/2473631511 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7ae011950 con 0x55e7ae1aa0f0 2024-05-27T07:45:26.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f0463fff700 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:26.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f046950f700 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1ae3a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f045c007a80 tx=0x7f045c00b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2aba8c964fde147c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:26.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.183+0000 7f0443fff700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:45:26.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0443fff700 1 -- 172.21.15.114:0/2473631511 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045c020040 con 0x55e7ae1aa0f0 2024-05-27T07:45:26.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0479987b80 1 -- 172.21.15.114:0/2473631511 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f045c00bcb0 con 0x55e7ae1aa0f0 2024-05-27T07:45:26.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0443fff700 1 -- 172.21.15.114:0/2473631511 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f045c005040 con 0x55e7ae1aa0f0 2024-05-27T07:45:26.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0443fff700 1 -- 172.21.15.114:0/2473631511 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f045c035050 con 0x55e7ae1aa0f0 2024-05-27T07:45:26.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0479987b80 10 client.26257 fetch_fsmap learned FSMap version 35 2024-05-27T07:45:26.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0479987b80 1 -- 172.21.15.114:0/2473631511 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e7ae07b7e0 con 0x55e7ae1aa0f0 2024-05-27T07:45:26.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0443fff700 1 -- 172.21.15.114:0/2473631511 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f045c00bcb0 con 0x55e7ae1aa0f0 2024-05-27T07:45:26.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0443fff700 1 -- 172.21.15.114:0/2473631511 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 35) ==== 7725+0+0 (secure 0 0 0) 0x7f045c02b030 con 0x55e7ae1aa0f0 2024-05-27T07:45:26.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0479987b80 10 client.26257 fetch_fsmap finished waiting for FSMap version 35 2024-05-27T07:45:26.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0479987b80 10 client.26257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:26.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0479987b80 20 client.26257 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:26.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0479987b80 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e7ae1bd700 0x55e7ae1bfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0479987b80 4 client.26257 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:26.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.184+0000 7f0479987b80 1 -- 172.21.15.114:0/2473631511 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7ae0d3d40 con 0x55e7ae1bd700 2024-05-27T07:45:26.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.185+0000 7f0468d0e700 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e7ae1bd700 0x55e7ae1bfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.185+0000 7f0443fff700 10 client.26257 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.185+0000 7f0468d0e700 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e7ae1bd700 0x55e7ae1bfb20 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43c5a545bab5a2b4 server_cookie=c202f00795b25bc in_seq=0 out_seq=0 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.186+0000 7f0443fff700 1 -- 172.21.15.114:0/2473631511 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e7ae0d3d40 con 0x55e7ae1bd700 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.186+0000 7f0443fff700 10 client.26257 handle_command_reply: tid=0 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.186+0000 7f0479987b80 1 client.26257 shutdown 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.187+0000 7f0479987b80 1 -- 172.21.15.114:0/2473631511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1aa0f0 msgr2=0x55e7ae1ae3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.187+0000 7f0479987b80 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1ae3a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f045c007a80 tx=0x7f045c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.187+0000 7f0479987b80 1 -- 172.21.15.114:0/2473631511 shutdown_connections 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.187+0000 7f0479987b80 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1ae3a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.187+0000 7f0479987b80 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.187+0000 7f0479987b80 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e7ae1bd700 0x55e7ae1bfb20 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.187+0000 7f0479987b80 1 --2- 172.21.15.114:0/2473631511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.187+0000 7f0479987b80 1 -- 172.21.15.114:0/2473631511 >> 172.21.15.114:0/2473631511 conn(0x55e7ae0277f0 msgr2=0x55e7ae1a2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.187+0000 7f0443fff700 0 client.26257 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:26.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.187+0000 7f0479987b80 1 -- 172.21.15.114:0/2473631511 shutdown_connections 2024-05-27T07:45:26.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.187+0000 7f0479987b80 1 -- 172.21.15.114:0/2473631511 wait complete. 2024-05-27T07:45:26.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.187+0000 7f0479987b80 20 client.26257 trim_cache size 0 max 16384 2024-05-27T07:45:26.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0479987b80 1 Processor -- start 2024-05-27T07:45:26.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0479987b80 1 -- start start 2024-05-27T07:45:26.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0479987b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1a3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0479987b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1b9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0479987b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0479987b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04641b7a30 con 0x55e7ae024ec0 2024-05-27T07:45:26.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0479987b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04641b4ae0 con 0x55e7ae1aa0f0 2024-05-27T07:45:26.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0479987b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04641b6c90 con 0x55e7ae1a41a0 2024-05-27T07:45:26.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0468d0e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1a3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0468d0e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1a3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49880/0 (socket says 172.21.15.114:49880) 2024-05-27T07:45:26.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0468d0e700 1 -- 172.21.15.114:0/1179576980 learned_addr learned my addr 172.21.15.114:0/1179576980 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:26.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0463fff700 1 --2- 172.21.15.114:0/1179576980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f046950f700 1 --2- 172.21.15.114:0/1179576980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0468d0e700 1 -- 172.21.15.114:0/1179576980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1a41a0 msgr2=0x55e7ae1b9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:26.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0468d0e700 1 --2- 172.21.15.114:0/1179576980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1b9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0468d0e700 1 -- 172.21.15.114:0/1179576980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 msgr2=0x55e7ae1adeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:26.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.210+0000 7f0468d0e700 1 --2- 172.21.15.114:0/1179576980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1adeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.211+0000 7f0468d0e700 1 -- 172.21.15.114:0/1179576980 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7ae1a8f60 con 0x55e7ae024ec0 2024-05-27T07:45:26.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.211+0000 7f046950f700 1 --2- 172.21.15.114:0/1179576980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:26.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.211+0000 7f0463fff700 1 --2- 172.21.15.114:0/1179576980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1b9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:26.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.211+0000 7f0468d0e700 1 --2- 172.21.15.114:0/1179576980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1a3c60 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f0454011cd0 tx=0x7f0454014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ae6ba84f84be50 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:26.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.211+0000 7f044a7fc700 1 -- 172.21.15.114:0/1179576980 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045402e070 con 0x55e7ae024ec0 2024-05-27T07:45:26.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.211+0000 7f044a7fc700 1 -- 172.21.15.114:0/1179576980 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0454003b30 con 0x55e7ae024ec0 2024-05-27T07:45:26.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.211+0000 7f044a7fc700 1 -- 172.21.15.114:0/1179576980 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0454029b10 con 0x55e7ae024ec0 2024-05-27T07:45:26.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.212+0000 7f0479987b80 1 -- 172.21.15.114:0/1179576980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 msgr2=0x55e7ae1a3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:26.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.212+0000 7f0479987b80 1 --2- 172.21.15.114:0/1179576980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1a3c60 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f0454011cd0 tx=0x7f0454014000 comp rx=0 tx=0).stop 2024-05-27T07:45:26.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.212+0000 7f0479987b80 1 -- 172.21.15.114:0/1179576980 shutdown_connections 2024-05-27T07:45:26.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.212+0000 7f0479987b80 1 --2- 172.21.15.114:0/1179576980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.212+0000 7f0479987b80 1 --2- 172.21.15.114:0/1179576980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae024ec0 0x55e7ae1a3c60 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.212+0000 7f0479987b80 1 --2- 172.21.15.114:0/1179576980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1b9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.212+0000 7f0479987b80 1 -- 172.21.15.114:0/1179576980 >> 172.21.15.114:0/1179576980 conn(0x55e7ae0277f0 msgr2=0x55e7ae1a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:26.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.212+0000 7f0479987b80 1 -- 172.21.15.114:0/1179576980 shutdown_connections 2024-05-27T07:45:26.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.212+0000 7f0479987b80 1 -- 172.21.15.114:0/1179576980 wait complete. 2024-05-27T07:45:26.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.212+0000 7f0479987b80 1 -- start start 2024-05-27T07:45:26.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0479987b80 1 Processor -- start 2024-05-27T07:45:26.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0479987b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae024ec0 0x55e7ae1c2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0479987b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1c0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0479987b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1c1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0479987b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04641b65c0 con 0x55e7ae1a41a0 2024-05-27T07:45:26.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0479987b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04641b5150 con 0x55e7ae1aa0f0 2024-05-27T07:45:26.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0479987b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04641b5820 con 0x55e7ae024ec0 2024-05-27T07:45:26.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0479987b80 10 client.0 _ll_register_callbacks cb 0x55e7ae058630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:26.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0468d0e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae024ec0 0x55e7ae1c2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0468d0e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae024ec0 0x55e7ae1c2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:38404/0 (socket says 172.21.15.114:38404) 2024-05-27T07:45:26.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0468d0e700 1 -- 172.21.15.114:0/4045977542 learned_addr learned my addr 172.21.15.114:0/4045977542 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:26.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0463fff700 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1c0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f046950f700 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1c1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0468d0e700 1 -- 172.21.15.114:0/4045977542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 msgr2=0x55e7ae1c1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:26.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0468d0e700 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1c1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0468d0e700 1 -- 172.21.15.114:0/4045977542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae1a41a0 msgr2=0x55e7ae1c0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:26.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.213+0000 7f0468d0e700 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1c0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.214+0000 7f0468d0e700 1 -- 172.21.15.114:0/4045977542 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7ae1c18a0 con 0x55e7ae024ec0 2024-05-27T07:45:26.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.214+0000 7f046950f700 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1c1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:45:26.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.214+0000 7f0468d0e700 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae024ec0 0x55e7ae1c2a90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0454009500 tx=0x7f0454003d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21feb0cf2819af01 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:26.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.214+0000 7f0443fff700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:45:26.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.214+0000 7f0443fff700 1 -- 172.21.15.114:0/4045977542 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0454037060 con 0x55e7ae024ec0 2024-05-27T07:45:26.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.214+0000 7f0479987b80 1 -- 172.21.15.114:0/4045977542 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0454003b30 con 0x55e7ae024ec0 2024-05-27T07:45:26.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.214+0000 7f0443fff700 1 -- 172.21.15.114:0/4045977542 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f0454003740 con 0x55e7ae024ec0 2024-05-27T07:45:26.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.214+0000 7f0443fff700 1 -- 172.21.15.114:0/4045977542 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f045403e050 con 0x55e7ae024ec0 2024-05-27T07:45:26.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.214+0000 7f0479987b80 10 client.26263 fetch_fsmap learned FSMap version 35 2024-05-27T07:45:26.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.214+0000 7f0479987b80 1 -- 172.21.15.114:0/4045977542 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e7ae228b80 con 0x55e7ae024ec0 2024-05-27T07:45:26.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.214+0000 7f0443fff700 1 -- 172.21.15.114:0/4045977542 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0454003b30 con 0x55e7ae024ec0 2024-05-27T07:45:26.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.215+0000 7f0443fff700 1 -- 172.21.15.114:0/4045977542 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 35) ==== 7725+0+0 (secure 0 0 0) 0x7f0454004ac0 con 0x55e7ae024ec0 2024-05-27T07:45:26.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.215+0000 7f0479987b80 10 client.26263 fetch_fsmap finished waiting for FSMap version 35 2024-05-27T07:45:26.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.215+0000 7f0479987b80 10 client.26263 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:26.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.215+0000 7f0479987b80 20 client.26263 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:26.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.215+0000 7f0479987b80 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e7ae229130 0x55e7ae22b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:26.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.215+0000 7f0479987b80 4 client.26263 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:45:26.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.215+0000 7f0479987b80 1 -- 172.21.15.114:0/4045977542 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e7ae1a9410 con 0x55e7ae229130 2024-05-27T07:45:26.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.215+0000 7f0463fff700 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e7ae229130 0x55e7ae22b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:26.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.215+0000 7f0443fff700 10 client.26263 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:26.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.215+0000 7f0463fff700 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e7ae229130 0x55e7ae22b550 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=449b7fd967604cb2 server_cookie=69bd0d79ce8cc653 in_seq=0 out_seq=0 2024-05-27T07:45:26.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.218+0000 7f0443fff700 1 -- 172.21.15.114:0/4045977542 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e7ae1a9410 con 0x55e7ae229130 2024-05-27T07:45:26.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.218+0000 7f0443fff700 10 client.26263 handle_command_reply: tid=0 2024-05-27T07:45:26.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.218+0000 7f0479987b80 1 client.26263 shutdown 2024-05-27T07:45:26.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.218+0000 7f0479987b80 1 -- 172.21.15.114:0/4045977542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae024ec0 msgr2=0x55e7ae1c2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:26.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.218+0000 7f0479987b80 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae024ec0 0x55e7ae1c2a90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0454009500 tx=0x7f0454003d60 comp rx=0 tx=0).stop 2024-05-27T07:45:26.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.218+0000 7f0479987b80 1 -- 172.21.15.114:0/4045977542 shutdown_connections 2024-05-27T07:45:26.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.218+0000 7f0479987b80 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e7ae229130 0x55e7ae22b550 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.218+0000 7f0479987b80 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7ae1aa0f0 0x55e7ae1c1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.218+0000 7f0479987b80 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ae024ec0 0x55e7ae1c2a90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.218+0000 7f0443fff700 0 client.26263 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:26.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.218+0000 7f0479987b80 1 --2- 172.21.15.114:0/4045977542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e7ae1a41a0 0x55e7ae1c0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.218+0000 7f0479987b80 1 -- 172.21.15.114:0/4045977542 >> 172.21.15.114:0/4045977542 conn(0x55e7ae0277f0 msgr2=0x55e7ae1a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:26.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.219+0000 7f0479987b80 1 -- 172.21.15.114:0/4045977542 shutdown_connections 2024-05-27T07:45:26.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.219+0000 7f0479987b80 1 -- 172.21.15.114:0/4045977542 wait complete. 2024-05-27T07:45:26.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.219+0000 7f0479987b80 20 client.26263 trim_cache size 0 max 16384 2024-05-27T07:45:26.223 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:45:26.223 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:45:26.223 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "8f194f2e-8dab-4374-adf0-68c8db37bb20", 2024-05-27T07:45:26.223 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:45:26.223 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:45:26.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.219+0000 7f046af72700 1 -- 172.21.15.114:0/3375377510 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f044c0b2af0 msgr2=0x7f044c0b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:26.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.219+0000 7f046af72700 1 --2- 172.21.15.114:0/3375377510 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f044c0b2af0 0x7f044c0b5000 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0454009680 tx=0x7f0454009260 comp rx=0 tx=0).stop 2024-05-27T07:45:26.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.219+0000 7f046af72700 1 -- 172.21.15.114:0/3375377510 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0464108f00 msgr2=0x7f04641a26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:26.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.219+0000 7f046af72700 1 --2- 172.21.15.114:0/3375377510 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0464108f00 0x7f04641a26f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f045800e970 tx=0x7f045800ec80 comp rx=0 tx=0).stop 2024-05-27T07:45:26.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.219+0000 7f046af72700 1 -- 172.21.15.114:0/3375377510 shutdown_connections 2024-05-27T07:45:26.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.220+0000 7f046af72700 1 --2- 172.21.15.114:0/3375377510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0464109850 0x7f04641ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.220+0000 7f046af72700 1 --2- 172.21.15.114:0/3375377510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04641045d0 0x7f04641a21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.220+0000 7f046af72700 1 --2- 172.21.15.114:0/3375377510 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f044c0b2af0 0x7f044c0b5000 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.220+0000 7f046af72700 1 --2- 172.21.15.114:0/3375377510 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0464108f00 0x7f04641a26f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:26.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.220+0000 7f046af72700 1 -- 172.21.15.114:0/3375377510 >> 172.21.15.114:0/3375377510 conn(0x7f04640ffec0 msgr2=0x7f0464108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:26.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.220+0000 7f046af72700 1 -- 172.21.15.114:0/3375377510 shutdown_connections 2024-05-27T07:45:26.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:26.220+0000 7f046af72700 1 -- 172.21.15.114:0/3375377510 wait complete. 2024-05-27T07:45:26.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 544771 13.09 MB/sec execute 98 sec latency 19.109 ms 2024-05-27T07:45:27.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:45:27.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 547689 13.13 MB/sec execute 99 sec latency 10.466 ms 2024-05-27T07:45:27.261 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:27.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:27 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:45:27.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:27 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:27.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:27 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:45:27.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:27 smithi114 ceph-mon[32038]: pgmap v180: 129 pgs: 129 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 6.5 MiB/s wr, 552 op/s 2024-05-27T07:45:27.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:27 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:27.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:27 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:45:27.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:27 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:45:27.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:27 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:27.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:27 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:45:27.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:27 smithi163 ceph-mon[36451]: pgmap v180: 129 pgs: 129 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 6.5 MiB/s wr, 552 op/s 2024-05-27T07:45:27.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:27 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:27.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:27 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:45:27.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:27 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:45:27.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:27 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:27.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:27 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:45:27.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:27 smithi186 ceph-mon[36647]: pgmap v180: 129 pgs: 129 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 6.5 MiB/s wr, 552 op/s 2024-05-27T07:45:27.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:27 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:27.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:27 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:45:28.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 550730 13.18 MB/sec execute 100 sec latency 22.798 ms 2024-05-27T07:45:29.249 INFO:tasks.workunit.client.0.smithi114.stdout: 1 551864 13.08 MB/sec execute 101 sec latency 96.223 ms 2024-05-27T07:45:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.264+0000 7fefb7c12700 1 -- 172.21.15.114:0/1991660414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb0109240 msgr2=0x7fefb0109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.264+0000 7fefb7c12700 1 --2- 172.21.15.114:0/1991660414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb0109240 0x7fefb0109650 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fefac0098f0 tx=0x7fefac009c00 comp rx=0 tx=0).stop 2024-05-27T07:45:29.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.268+0000 7fefb7c12700 1 -- 172.21.15.114:0/1991660414 shutdown_connections 2024-05-27T07:45:29.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.268+0000 7fefb7c12700 1 --2- 172.21.15.114:0/1991660414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefb01030d0 0x7fefb0107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.268+0000 7fefb7c12700 1 --2- 172.21.15.114:0/1991660414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb0109240 0x7fefb0109650 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.268+0000 7fefb7c12700 1 --2- 172.21.15.114:0/1991660414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefb0102780 0x7fefb0102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.268+0000 7fefb7c12700 1 -- 172.21.15.114:0/1991660414 >> 172.21.15.114:0/1991660414 conn(0x7fefb00fe4d0 msgr2=0x7fefb0100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.270+0000 7fefb7c12700 1 -- 172.21.15.114:0/1991660414 shutdown_connections 2024-05-27T07:45:29.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.271+0000 7fefb7c12700 1 -- 172.21.15.114:0/1991660414 wait complete. 2024-05-27T07:45:29.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.272+0000 7fefb7c12700 1 Processor -- start 2024-05-27T07:45:29.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.274+0000 7fefb7c12700 1 -- start start 2024-05-27T07:45:29.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.274+0000 7fefb7c12700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefb0102780 0x7fefb01a1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.274+0000 7fefb7c12700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefb01030d0 0x7fefb01a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.274+0000 7fefb7c12700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb0109240 0x7fefb01a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.274+0000 7fefb7c12700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fefb011aaf0 con 0x7fefb0102780 2024-05-27T07:45:29.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.274+0000 7fefb7c12700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fefb011a970 con 0x7fefb01030d0 2024-05-27T07:45:29.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.274+0000 7fefb7c12700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fefb011ac70 con 0x7fefb0109240 2024-05-27T07:45:29.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.275+0000 7fefb51ad700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefb01030d0 0x7fefb01a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.275+0000 7fefb51ad700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefb01030d0 0x7fefb01a1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:41738/0 (socket says 172.21.15.114:41738) 2024-05-27T07:45:29.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.275+0000 7fefb51ad700 1 -- 172.21.15.114:0/2721957100 learned_addr learned my addr 172.21.15.114:0/2721957100 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:29.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.275+0000 7fefb61af700 1 --2- 172.21.15.114:0/2721957100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb0109240 0x7fefb01a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.275+0000 7fefb61af700 1 -- 172.21.15.114:0/2721957100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefb01030d0 msgr2=0x7fefb01a1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.275+0000 7fefb61af700 1 --2- 172.21.15.114:0/2721957100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefb01030d0 0x7fefb01a1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.275+0000 7fefb61af700 1 -- 172.21.15.114:0/2721957100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefb0102780 msgr2=0x7fefb01a1500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:45:29.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.275+0000 7fefb61af700 1 --2- 172.21.15.114:0/2721957100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefb0102780 0x7fefb01a1500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.275+0000 7fefb61af700 1 -- 172.21.15.114:0/2721957100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefb01a63d0 con 0x7fefb0109240 2024-05-27T07:45:29.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.276+0000 7fefb61af700 1 --2- 172.21.15.114:0/2721957100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb0109240 0x7fefb01a5d30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fefa800bb00 tx=0x7fefa800be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:29.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.276+0000 7fefa6ffd700 1 -- 172.21.15.114:0/2721957100 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa803b070 con 0x7fefb0109240 2024-05-27T07:45:29.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.276+0000 7fefa6ffd700 1 -- 172.21.15.114:0/2721957100 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefa8036440 con 0x7fefb0109240 2024-05-27T07:45:29.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.276+0000 7fefa6ffd700 1 -- 172.21.15.114:0/2721957100 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8034560 con 0x7fefb0109240 2024-05-27T07:45:29.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.277+0000 7fefb7c12700 1 -- 172.21.15.114:0/2721957100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefb01a66c0 con 0x7fefb0109240 2024-05-27T07:45:29.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.277+0000 7fefb7c12700 1 -- 172.21.15.114:0/2721957100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefb006d1d0 con 0x7fefb0109240 2024-05-27T07:45:29.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.283+0000 7fefc6627b80 1 Processor -- start 2024-05-27T07:45:29.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.286+0000 7fefc6627b80 1 -- start start 2024-05-27T07:45:29.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.286+0000 7fefa6ffd700 1 -- 172.21.15.114:0/2721957100 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fefa803e070 con 0x7fefb0109240 2024-05-27T07:45:29.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.287+0000 7fefc6627b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672cf1ee40 0x55672d0a2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.287+0000 7fefc6627b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d0a2c50 0x55672d09ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.287+0000 7fefc6627b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d09cf80 0x55672d09d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.287+0000 7fefc6627b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55672cfcbc90 con 0x55672cf1ee40 2024-05-27T07:45:29.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.287+0000 7fefc6627b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55672cfbb830 con 0x55672d0a2c50 2024-05-27T07:45:29.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.287+0000 7fefc6627b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55672cfbf870 con 0x55672d09cf80 2024-05-27T07:45:29.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.287+0000 7fefb59ae700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d0a2c50 0x55672d09ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.287+0000 7fefb51ad700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672cf1ee40 0x55672d0a2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.287+0000 7fefb51ad700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672cf1ee40 0x55672d0a2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49928/0 (socket says 172.21.15.114:49928) 2024-05-27T07:45:29.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.287+0000 7fefb51ad700 1 -- 172.21.15.114:0/3015712967 learned_addr learned my addr 172.21.15.114:0/3015712967 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:29.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.289+0000 7fefb51ad700 1 --2- 172.21.15.114:0/3015712967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d09cf80 0x55672d09d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.289+0000 7fefa6ffd700 1 --2- 172.21.15.114:0/2721957100 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fef9c0b2bc0 0x7fef9c0b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.290+0000 7fefa6ffd700 1 -- 172.21.15.114:0/2721957100 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fefa8103b50 con 0x7fefb0109240 2024-05-27T07:45:29.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.290+0000 7fefb51ad700 1 -- 172.21.15.114:0/3015712967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d09cf80 msgr2=0x55672d09d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.290+0000 7fefb51ad700 1 --2- 172.21.15.114:0/3015712967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d09cf80 0x55672d09d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.290+0000 7fefb51ad700 1 -- 172.21.15.114:0/3015712967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d0a2c50 msgr2=0x55672d09ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.290+0000 7fefb51ad700 1 --2- 172.21.15.114:0/3015712967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d0a2c50 0x55672d09ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.290+0000 7fefb51ad700 1 -- 172.21.15.114:0/3015712967 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55672cf0b8d0 con 0x55672cf1ee40 2024-05-27T07:45:29.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.291+0000 7fefb59ae700 1 --2- 172.21.15.114:0/3015712967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d0a2c50 0x55672d09ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:45:29.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.291+0000 7fefb51ad700 1 --2- 172.21.15.114:0/3015712967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672cf1ee40 0x55672d0a2710 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fefa0012960 tx=0x7fefa000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99a8ae6bbc42df1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:29.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.292+0000 7fef9bfff700 1 -- 172.21.15.114:0/3015712967 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa00120d0 con 0x55672cf1ee40 2024-05-27T07:45:29.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.292+0000 7fef9bfff700 1 -- 172.21.15.114:0/3015712967 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefa00044e0 con 0x55672cf1ee40 2024-05-27T07:45:29.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.292+0000 7fef9bfff700 1 -- 172.21.15.114:0/3015712967 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa0005020 con 0x55672cf1ee40 2024-05-27T07:45:29.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.292+0000 7fefb61af700 1 --2- 172.21.15.114:0/2721957100 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fef9c0b2bc0 0x7fef9c0b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.293+0000 7fefc6627b80 1 -- 172.21.15.114:0/3015712967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672cf1ee40 msgr2=0x55672d0a2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.293+0000 7fefc6627b80 1 --2- 172.21.15.114:0/3015712967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672cf1ee40 0x55672d0a2710 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fefa0012960 tx=0x7fefa000b900 comp rx=0 tx=0).stop 2024-05-27T07:45:29.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.294+0000 7fefb61af700 1 --2- 172.21.15.114:0/2721957100 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fef9c0b2bc0 0x7fef9c0b50d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fefa80caac0 tx=0x7fefa8030040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:29.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.295+0000 7fefc6627b80 1 -- 172.21.15.114:0/3015712967 shutdown_connections 2024-05-27T07:45:29.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.295+0000 7fefc6627b80 1 --2- 172.21.15.114:0/3015712967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d09cf80 0x55672d09d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.295+0000 7fefc6627b80 1 --2- 172.21.15.114:0/3015712967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672cf1ee40 0x55672d0a2710 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.295+0000 7fefc6627b80 1 --2- 172.21.15.114:0/3015712967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d0a2c50 0x55672d09ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.295+0000 7fefc6627b80 1 -- 172.21.15.114:0/3015712967 >> 172.21.15.114:0/3015712967 conn(0x55672cf79af0 msgr2=0x55672d097640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:29.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.295+0000 7fefc6627b80 1 -- 172.21.15.114:0/3015712967 shutdown_connections 2024-05-27T07:45:29.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.295+0000 7fefc6627b80 1 -- 172.21.15.114:0/3015712967 wait complete. 2024-05-27T07:45:29.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.296+0000 7fefc6627b80 1 -- start start 2024-05-27T07:45:29.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.296+0000 7fefc6627b80 1 Processor -- start 2024-05-27T07:45:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.297+0000 7fefc6627b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672cf1ee40 0x55672d0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.297+0000 7fefc6627b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d09cf80 0x55672d0a6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.298+0000 7fefc6627b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d0a2c50 0x55672d0a6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.298+0000 7fefc6627b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55672cec1fa0 con 0x55672d09cf80 2024-05-27T07:45:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.298+0000 7fefc6627b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55672cf5d7c0 con 0x55672cf1ee40 2024-05-27T07:45:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.298+0000 7fefc6627b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55672d0a96e0 con 0x55672d0a2c50 2024-05-27T07:45:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.298+0000 7fefc6627b80 10 client.0 _ll_register_callbacks cb 0x55672cf56920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.298+0000 7fefb51ad700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672cf1ee40 0x55672d0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.298+0000 7fefb51ad700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672cf1ee40 0x55672d0a6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:41752/0 (socket says 172.21.15.114:41752) 2024-05-27T07:45:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.298+0000 7fefb51ad700 1 -- 172.21.15.114:0/3364170498 learned_addr learned my addr 172.21.15.114:0/3364170498 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.298+0000 7fefb59ae700 1 --2- 172.21.15.114:0/3364170498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d09cf80 0x55672d0a6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.299+0000 7fefb51ad700 1 --2- 172.21.15.114:0/3364170498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d0a2c50 0x55672d0a6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.300+0000 7fefb51ad700 1 -- 172.21.15.114:0/3364170498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d0a2c50 msgr2=0x55672d0a6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.300+0000 7fefb51ad700 1 --2- 172.21.15.114:0/3364170498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d0a2c50 0x55672d0a6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.300+0000 7fefb51ad700 1 -- 172.21.15.114:0/3364170498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d09cf80 msgr2=0x55672d0a6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.300+0000 7fefb51ad700 1 --2- 172.21.15.114:0/3364170498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d09cf80 0x55672d0a6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.300+0000 7fefb51ad700 1 -- 172.21.15.114:0/3364170498 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55672cf5a690 con 0x55672cf1ee40 2024-05-27T07:45:29.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.301+0000 7fefb51ad700 1 --2- 172.21.15.114:0/3364170498 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672cf1ee40 0x55672d0a6320 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fefa000b540 tx=0x7fefa0004a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4acb21b890fa7c26 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:29.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.301+0000 7fef997fa700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:45:29.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.301+0000 7fef997fa700 1 -- 172.21.15.114:0/3364170498 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa0004fc0 con 0x55672cf1ee40 2024-05-27T07:45:29.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.301+0000 7fef997fa700 1 -- 172.21.15.114:0/3364170498 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fefa0005160 con 0x55672cf1ee40 2024-05-27T07:45:29.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.301+0000 7fefc6627b80 1 -- 172.21.15.114:0/3364170498 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefa00044e0 con 0x55672cf1ee40 2024-05-27T07:45:29.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.302+0000 7fef997fa700 1 -- 172.21.15.114:0/3364170498 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fefa0005360 con 0x55672cf1ee40 2024-05-27T07:45:29.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.302+0000 7fefc6627b80 10 client.26183 fetch_fsmap learned FSMap version 35 2024-05-27T07:45:29.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.303+0000 7fefc6627b80 1 -- 172.21.15.114:0/3364170498 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55672d0b4170 con 0x55672cf1ee40 2024-05-27T07:45:29.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.303+0000 7fef997fa700 1 -- 172.21.15.114:0/3364170498 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefa0005500 con 0x55672cf1ee40 2024-05-27T07:45:29.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.304+0000 7fef997fa700 1 -- 172.21.15.114:0/3364170498 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 35) ==== 7725+0+0 (secure 0 0 0) 0x7fefa0021460 con 0x55672cf1ee40 2024-05-27T07:45:29.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.304+0000 7fefc6627b80 10 client.26183 fetch_fsmap finished waiting for FSMap version 35 2024-05-27T07:45:29.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.304+0000 7fefc6627b80 10 client.26183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:29.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.304+0000 7fefc6627b80 20 client.26183 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:29.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.304+0000 7fefc6627b80 1 --2- 172.21.15.114:0/3364170498 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55672d0b6640 0x55672d0b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.304+0000 7fefc6627b80 4 client.26183 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:29.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.304+0000 7fefc6627b80 1 -- 172.21.15.114:0/3364170498 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fefa00044e0 con 0x55672d0b6640 2024-05-27T07:45:29.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.309+0000 7fefb59ae700 1 --2- 172.21.15.114:0/3364170498 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55672d0b6640 0x55672d0b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.311+0000 7fefb59ae700 1 --2- 172.21.15.114:0/3364170498 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55672d0b6640 0x55672d0b8a60 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4219ff2be9184f79 server_cookie=13b5fe1853a25274 in_seq=0 out_seq=0 2024-05-27T07:45:29.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.311+0000 7fef997fa700 10 client.26183 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:29.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.315+0000 7fef997fa700 1 -- 172.21.15.114:0/3364170498 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fefa00044e0 con 0x55672d0b6640 2024-05-27T07:45:29.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.315+0000 7fef997fa700 10 client.26183 handle_command_reply: tid=0 2024-05-27T07:45:29.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.315+0000 7fefc6627b80 1 client.26183 shutdown 2024-05-27T07:45:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.318+0000 7fefc6627b80 1 -- 172.21.15.114:0/3364170498 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672cf1ee40 msgr2=0x55672d0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.318+0000 7fefc6627b80 1 --2- 172.21.15.114:0/3364170498 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672cf1ee40 0x55672d0a6320 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fefa000b540 tx=0x7fefa0004a70 comp rx=0 tx=0).stop 2024-05-27T07:45:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.318+0000 7fefc6627b80 1 -- 172.21.15.114:0/3364170498 shutdown_connections 2024-05-27T07:45:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.318+0000 7fefc6627b80 1 --2- 172.21.15.114:0/3364170498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d0a2c50 0x55672d0a6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.318+0000 7fefc6627b80 1 --2- 172.21.15.114:0/3364170498 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55672d0b6640 0x55672d0b8a60 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.318+0000 7fefc6627b80 1 --2- 172.21.15.114:0/3364170498 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672cf1ee40 0x55672d0a6320 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.318+0000 7fefc6627b80 1 --2- 172.21.15.114:0/3364170498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d09cf80 0x55672d0a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.318+0000 7fefc6627b80 1 -- 172.21.15.114:0/3364170498 >> 172.21.15.114:0/3364170498 conn(0x55672cf79af0 msgr2=0x55672d09ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:29.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.318+0000 7fef997fa700 0 client.26183 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:29.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.319+0000 7fefc6627b80 1 -- 172.21.15.114:0/3364170498 shutdown_connections 2024-05-27T07:45:29.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.319+0000 7fefc6627b80 1 -- 172.21.15.114:0/3364170498 wait complete. 2024-05-27T07:45:29.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.319+0000 7fefc6627b80 20 client.26183 trim_cache size 0 max 16384 2024-05-27T07:45:29.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.346+0000 7fefc6627b80 1 Processor -- start 2024-05-27T07:45:29.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.346+0000 7fefc6627b80 1 -- start start 2024-05-27T07:45:29.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.346+0000 7fefc6627b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 0x55672d09ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.346+0000 7fefc6627b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d09cf80 0x55672d0b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.346+0000 7fefc6627b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d0a2c50 0x55672d0a8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.346+0000 7fefc6627b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fef9400fd30 con 0x55672d09cf80 2024-05-27T07:45:29.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.346+0000 7fefc6627b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fef9400cde0 con 0x55672d0a2c50 2024-05-27T07:45:29.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.346+0000 7fefc6627b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef9400ef90 con 0x55672cf1ee40 2024-05-27T07:45:29.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fefb59ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d09cf80 0x55672d0b29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fefb51ad700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 0x55672d09ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fefb59ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d09cf80 0x55672d0b29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49954/0 (socket says 172.21.15.114:49954) 2024-05-27T07:45:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fefb51ad700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 0x55672d09ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:38440/0 (socket says 172.21.15.114:38440) 2024-05-27T07:45:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fefb59ae700 1 -- 172.21.15.114:0/2180711255 learned_addr learned my addr 172.21.15.114:0/2180711255 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fefb51ad700 1 --2- 172.21.15.114:0/2180711255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d0a2c50 0x55672d0a8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fefb51ad700 1 -- 172.21.15.114:0/2180711255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d0a2c50 msgr2=0x55672d0a8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fefb51ad700 1 --2- 172.21.15.114:0/2180711255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d0a2c50 0x55672d0a8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fefb51ad700 1 -- 172.21.15.114:0/2180711255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d09cf80 msgr2=0x55672d0b29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fefb51ad700 1 --2- 172.21.15.114:0/2180711255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d09cf80 0x55672d0b29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fefb51ad700 1 -- 172.21.15.114:0/2180711255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55672d0a48b0 con 0x55672cf1ee40 2024-05-27T07:45:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fefb51ad700 1 --2- 172.21.15.114:0/2180711255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 0x55672d09ca40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fefa000b540 tx=0x7fefa0025040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a2a9b6dfddbecf1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fef9b7fe700 1 -- 172.21.15.114:0/2180711255 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa000b9f0 con 0x55672cf1ee40 2024-05-27T07:45:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fef9b7fe700 1 -- 172.21.15.114:0/2180711255 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefa0004fc0 con 0x55672cf1ee40 2024-05-27T07:45:29.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.347+0000 7fef9b7fe700 1 -- 172.21.15.114:0/2180711255 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa00216c0 con 0x55672cf1ee40 2024-05-27T07:45:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.348+0000 7fefc6627b80 1 -- 172.21.15.114:0/2180711255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 msgr2=0x55672d09ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.348+0000 7fefc6627b80 1 --2- 172.21.15.114:0/2180711255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 0x55672d09ca40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fefa000b540 tx=0x7fefa0025040 comp rx=0 tx=0).stop 2024-05-27T07:45:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.349+0000 7fefc6627b80 1 -- 172.21.15.114:0/2180711255 shutdown_connections 2024-05-27T07:45:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.349+0000 7fefc6627b80 1 --2- 172.21.15.114:0/2180711255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d0a2c50 0x55672d0a8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.349+0000 7fefc6627b80 1 --2- 172.21.15.114:0/2180711255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 0x55672d09ca40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.349+0000 7fefc6627b80 1 --2- 172.21.15.114:0/2180711255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d09cf80 0x55672d0b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.349+0000 7fefc6627b80 1 -- 172.21.15.114:0/2180711255 >> 172.21.15.114:0/2180711255 conn(0x55672cf79af0 msgr2=0x55672d0a62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.349+0000 7fefc6627b80 1 -- 172.21.15.114:0/2180711255 shutdown_connections 2024-05-27T07:45:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.349+0000 7fefc6627b80 1 -- 172.21.15.114:0/2180711255 wait complete. 2024-05-27T07:45:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.349+0000 7fefc6627b80 1 -- start start 2024-05-27T07:45:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.349+0000 7fefc6627b80 1 Processor -- start 2024-05-27T07:45:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.349+0000 7fefc6627b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 0x55672d0b8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefc6627b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d09cf80 0x55672d0b9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefc6627b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d0a2c50 0x55672d11eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefc6627b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fef9400e8c0 con 0x55672d0a2c50 2024-05-27T07:45:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefc6627b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fef9400d450 con 0x55672d09cf80 2024-05-27T07:45:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefb51ad700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 0x55672d0b8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefc6627b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef9400db20 con 0x55672cf1ee40 2024-05-27T07:45:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefb51ad700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 0x55672d0b8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:38450/0 (socket says 172.21.15.114:38450) 2024-05-27T07:45:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefb51ad700 1 -- 172.21.15.114:0/2489734591 learned_addr learned my addr 172.21.15.114:0/2489734591 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefc6627b80 10 client.0 _ll_register_callbacks cb 0x55672cf56920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefb59ae700 1 --2- 172.21.15.114:0/2489734591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d09cf80 0x55672d0b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefb51ad700 1 --2- 172.21.15.114:0/2489734591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d0a2c50 0x55672d11eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefb51ad700 1 -- 172.21.15.114:0/2489734591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d09cf80 msgr2=0x55672d0b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefb51ad700 1 --2- 172.21.15.114:0/2489734591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d09cf80 0x55672d0b9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefb51ad700 1 -- 172.21.15.114:0/2489734591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d0a2c50 msgr2=0x55672d11eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefb51ad700 1 --2- 172.21.15.114:0/2489734591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d0a2c50 0x55672d11eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.350+0000 7fefb51ad700 1 -- 172.21.15.114:0/2489734591 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55672d11f430 con 0x55672cf1ee40 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.351+0000 7fefb51ad700 1 --2- 172.21.15.114:0/2489734591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 0x55672d0b8c50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fefa0021e30 tx=0x7fefa0021fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f472470ba36738b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.351+0000 7fef997fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.351+0000 7fef997fa700 1 -- 172.21.15.114:0/2489734591 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa0005710 con 0x55672cf1ee40 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.351+0000 7fef997fa700 1 -- 172.21.15.114:0/2489734591 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fefa0022440 con 0x55672cf1ee40 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.351+0000 7fefc6627b80 1 -- 172.21.15.114:0/2489734591 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefa0004fc0 con 0x55672cf1ee40 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.351+0000 7fef997fa700 1 -- 172.21.15.114:0/2489734591 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fefa0048050 con 0x55672cf1ee40 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.351+0000 7fefc6627b80 10 client.26281 fetch_fsmap learned FSMap version 35 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.351+0000 7fefc6627b80 1 -- 172.21.15.114:0/2489734591 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55672d1205f0 con 0x55672cf1ee40 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.351+0000 7fef997fa700 1 -- 172.21.15.114:0/2489734591 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefa00058b0 con 0x55672cf1ee40 2024-05-27T07:45:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.352+0000 7fef997fa700 1 -- 172.21.15.114:0/2489734591 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 35) ==== 7725+0+0 (secure 0 0 0) 0x7fefa0034030 con 0x55672cf1ee40 2024-05-27T07:45:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.352+0000 7fefc6627b80 10 client.26281 fetch_fsmap finished waiting for FSMap version 35 2024-05-27T07:45:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.352+0000 7fefc6627b80 10 client.26281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.352+0000 7fefc6627b80 20 client.26281 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.352+0000 7fefc6627b80 1 --2- 172.21.15.114:0/2489734591 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55672d120b30 0x55672d122f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.352+0000 7fefc6627b80 4 client.26281 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:45:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.352+0000 7fefc6627b80 1 -- 172.21.15.114:0/2489734591 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fefa0004fc0 con 0x55672d120b30 2024-05-27T07:45:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.352+0000 7fefb59ae700 1 --2- 172.21.15.114:0/2489734591 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55672d120b30 0x55672d122f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fefb59ae700 1 --2- 172.21.15.114:0/2489734591 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55672d120b30 0x55672d122f50 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebb525f154cb5039 server_cookie=ec0f69281f48ea2 in_seq=0 out_seq=0 2024-05-27T07:45:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fef997fa700 10 client.26281 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fef997fa700 1 -- 172.21.15.114:0/2489734591 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fefa0004fc0 con 0x55672d120b30 2024-05-27T07:45:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fef997fa700 10 client.26281 handle_command_reply: tid=0 2024-05-27T07:45:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fefc6627b80 1 client.26281 shutdown 2024-05-27T07:45:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fefc6627b80 1 -- 172.21.15.114:0/2489734591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 msgr2=0x55672d0b8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fefc6627b80 1 --2- 172.21.15.114:0/2489734591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 0x55672d0b8c50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fefa0021e30 tx=0x7fefa0021fa0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fefc6627b80 1 -- 172.21.15.114:0/2489734591 shutdown_connections 2024-05-27T07:45:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fefc6627b80 1 --2- 172.21.15.114:0/2489734591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55672d0a2c50 0x55672d11eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fefc6627b80 1 --2- 172.21.15.114:0/2489734591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672cf1ee40 0x55672d0b8c50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fefc6627b80 1 --2- 172.21.15.114:0/2489734591 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55672d120b30 0x55672d122f50 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fefc6627b80 1 --2- 172.21.15.114:0/2489734591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55672d09cf80 0x55672d0b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fef997fa700 0 client.26281 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fefc6627b80 1 -- 172.21.15.114:0/2489734591 >> 172.21.15.114:0/2489734591 conn(0x55672cf79af0 msgr2=0x55672d09fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fefc6627b80 1 -- 172.21.15.114:0/2489734591 shutdown_connections 2024-05-27T07:45:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.353+0000 7fefc6627b80 1 -- 172.21.15.114:0/2489734591 wait complete. 2024-05-27T07:45:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.354+0000 7fefc6627b80 20 client.26281 trim_cache size 0 max 16384 2024-05-27T07:45:29.358 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:45:29.358 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:45:29.358 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:45:29.358 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:45:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.354+0000 7fefb7c12700 1 -- 172.21.15.114:0/2721957100 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fef9c0b2bc0 msgr2=0x7fef9c0b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.354+0000 7fefb7c12700 1 --2- 172.21.15.114:0/2721957100 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fef9c0b2bc0 0x7fef9c0b50d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fefa80caac0 tx=0x7fefa8030040 comp rx=0 tx=0).stop 2024-05-27T07:45:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.354+0000 7fefb7c12700 1 -- 172.21.15.114:0/2721957100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb0109240 msgr2=0x7fefb01a5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.354+0000 7fefb7c12700 1 --2- 172.21.15.114:0/2721957100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb0109240 0x7fefb01a5d30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fefa800bb00 tx=0x7fefa800be10 comp rx=0 tx=0).stop 2024-05-27T07:45:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.354+0000 7fefb7c12700 1 -- 172.21.15.114:0/2721957100 shutdown_connections 2024-05-27T07:45:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.354+0000 7fefb7c12700 1 --2- 172.21.15.114:0/2721957100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb0109240 0x7fefb01a5d30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.354+0000 7fefb7c12700 1 --2- 172.21.15.114:0/2721957100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefb0102780 0x7fefb01a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.354+0000 7fefb7c12700 1 --2- 172.21.15.114:0/2721957100 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fef9c0b2bc0 0x7fef9c0b50d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.354+0000 7fefb7c12700 1 --2- 172.21.15.114:0/2721957100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefb01030d0 0x7fefb01a1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.354+0000 7fefb7c12700 1 -- 172.21.15.114:0/2721957100 >> 172.21.15.114:0/2721957100 conn(0x7fefb00fe4d0 msgr2=0x7fefb0105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.354+0000 7fefb7c12700 1 -- 172.21.15.114:0/2721957100 shutdown_connections 2024-05-27T07:45:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:29.354+0000 7fefb7c12700 1 -- 172.21.15.114:0/2721957100 wait complete. 2024-05-27T07:45:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.706+0000 7f2711c6e640 1 -- 172.21.15.114:0/32771976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f270c104d90 msgr2=0x7f270c1051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.706+0000 7f2711c6e640 1 --2- 172.21.15.114:0/32771976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f270c104d90 0x7f270c1051a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f26fc009900 tx=0x7f26fc02a410 comp rx=0 tx=0).stop 2024-05-27T07:45:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.706+0000 7f2711c6e640 1 -- 172.21.15.114:0/32771976 shutdown_connections 2024-05-27T07:45:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.706+0000 7f2711c6e640 1 --2- 172.21.15.114:0/32771976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f270c10e1e0 0x7f270c1106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.706+0000 7f2711c6e640 1 --2- 172.21.15.114:0/32771976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c10b7f0 0x7f270c10dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.706+0000 7f2711c6e640 1 --2- 172.21.15.114:0/32771976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f270c104d90 0x7f270c1051a0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.706+0000 7f2711c6e640 1 -- 172.21.15.114:0/32771976 >> 172.21.15.114:0/32771976 conn(0x7f270c0782d0 msgr2=0x7f270c078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.706+0000 7f2711c6e640 1 -- 172.21.15.114:0/32771976 shutdown_connections 2024-05-27T07:45:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.706+0000 7f2711c6e640 1 -- 172.21.15.114:0/32771976 wait complete. 2024-05-27T07:45:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.707+0000 7f2711c6e640 1 Processor -- start 2024-05-27T07:45:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.707+0000 7f2711c6e640 1 -- start start 2024-05-27T07:45:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.707+0000 7f2711c6e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c104d90 0x7f270c1a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.707+0000 7f2711c6e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f270c10b7f0 0x7f270c1a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.707+0000 7f2711c6e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f270c10e1e0 0x7f270c19a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.707+0000 7f2711c6e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f270c113e40 con 0x7f270c10b7f0 2024-05-27T07:45:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.707+0000 7f2711c6e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f270c113cc0 con 0x7f270c104d90 2024-05-27T07:45:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.707+0000 7f2711c6e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f270c113fc0 con 0x7f270c10e1e0 2024-05-27T07:45:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.707+0000 7f270affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f270c10b7f0 0x7f270c1a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f270affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f270c10b7f0 0x7f270c1a0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49978/0 (socket says 172.21.15.114:49978) 2024-05-27T07:45:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f270affd640 1 -- 172.21.15.114:0/3568213107 learned_addr learned my addr 172.21.15.114:0/3568213107 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f270bfff640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f270c10e1e0 0x7f270c19a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f270b7fe640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c104d90 0x7f270c1a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f270affd640 1 -- 172.21.15.114:0/3568213107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f270c10e1e0 msgr2=0x7f270c19a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f270affd640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f270c10e1e0 0x7f270c19a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f270affd640 1 -- 172.21.15.114:0/3568213107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c104d90 msgr2=0x7f270c1a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f270affd640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c104d90 0x7f270c1a0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f270affd640 1 -- 172.21.15.114:0/3568213107 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270c19adb0 con 0x7f270c10b7f0 2024-05-27T07:45:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f270bfff640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f270c10e1e0 0x7f270c19a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f270affd640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f270c10b7f0 0x7f270c1a0a40 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f270000e9f0 tx=0x7f270000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f2708ff9640 1 -- 172.21.15.114:0/3568213107 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270000cd80 con 0x7f270c10b7f0 2024-05-27T07:45:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f2708ff9640 1 -- 172.21.15.114:0/3568213107 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27000108e0 con 0x7f270c10b7f0 2024-05-27T07:45:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f2711c6e640 1 -- 172.21.15.114:0/3568213107 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f270c19b040 con 0x7f270c10b7f0 2024-05-27T07:45:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f2708ff9640 1 -- 172.21.15.114:0/3568213107 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2700018980 con 0x7f270c10b7f0 2024-05-27T07:45:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.708+0000 7f2711c6e640 1 -- 172.21.15.114:0/3568213107 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f270c1a72a0 con 0x7f270c10b7f0 2024-05-27T07:45:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.711+0000 7f2708ff9640 1 -- 172.21.15.114:0/3568213107 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f2700018b20 con 0x7f270c10b7f0 2024-05-27T07:45:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.711+0000 7f2718c02740 1 Processor -- start 2024-05-27T07:45:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.711+0000 7f2718c02740 1 -- start start 2024-05-27T07:45:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.711+0000 7f2718c02740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663b51b0 0x55fb667563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.711+0000 7f2718c02740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66756910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.711+0000 7f2718c02740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663944d0 0x55fb66756e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.711+0000 7f2718c02740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fb666c8c30 con 0x55fb663944d0 2024-05-27T07:45:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.711+0000 7f2718c02740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fb666b87d0 con 0x55fb663b51b0 2024-05-27T07:45:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.711+0000 7f2718c02740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fb666bc810 con 0x55fb6638c2f0 2024-05-27T07:45:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.711+0000 7f270affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663944d0 0x55fb66756e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f270bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663b51b0 0x55fb667563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f270b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66756910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f270affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663944d0 0x55fb66756e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49986/0 (socket says 172.21.15.114:49986) 2024-05-27T07:45:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f270bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663b51b0 0x55fb667563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:41806/0 (socket says 172.21.15.114:41806) 2024-05-27T07:45:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f270affd640 1 -- 172.21.15.114:0/1546050016 learned_addr learned my addr 172.21.15.114:0/1546050016 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f2708ff9640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f26f40b2950 0x7f26f40b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f270bfff640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f26f40b2950 0x7f26f40b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f270b7fe640 1 -- 172.21.15.114:0/1546050016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663b51b0 msgr2=0x55fb667563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f270b7fe640 1 --2- 172.21.15.114:0/1546050016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663b51b0 0x55fb667563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f270b7fe640 1 -- 172.21.15.114:0/1546050016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663944d0 msgr2=0x55fb66756e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f270b7fe640 1 --2- 172.21.15.114:0/1546050016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663944d0 0x55fb66756e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f2708ff9640 1 -- 172.21.15.114:0/3568213107 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f27000e2d60 con 0x7f270c10b7f0 2024-05-27T07:45:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f270b7fe640 1 -- 172.21.15.114:0/1546050016 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb663cd210 con 0x55fb6638c2f0 2024-05-27T07:45:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f270b7fe640 1 --2- 172.21.15.114:0/1546050016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66756910 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f26fc009230 tx=0x7f26fc032b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bcd8dedb90e3c28 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f270bfff640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f26f40b2950 0x7f26f40b4e60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55fb66753540 tx=0x7f2704004ab0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f26f1ffb640 1 -- 172.21.15.114:0/1546050016 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc040070 con 0x55fb6638c2f0 2024-05-27T07:45:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f26f1ffb640 1 -- 172.21.15.114:0/1546050016 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26fc004030 con 0x55fb6638c2f0 2024-05-27T07:45:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.712+0000 7f26f1ffb640 1 -- 172.21.15.114:0/1546050016 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc035040 con 0x55fb6638c2f0 2024-05-27T07:45:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.713+0000 7f2718c02740 1 -- 172.21.15.114:0/1546050016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 msgr2=0x55fb66756910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.713+0000 7f2718c02740 1 --2- 172.21.15.114:0/1546050016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66756910 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f26fc009230 tx=0x7f26fc032b90 comp rx=0 tx=0).stop 2024-05-27T07:45:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.713+0000 7f2718c02740 1 -- 172.21.15.114:0/1546050016 shutdown_connections 2024-05-27T07:45:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.713+0000 7f2718c02740 1 --2- 172.21.15.114:0/1546050016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663944d0 0x55fb66756e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.713+0000 7f2718c02740 1 --2- 172.21.15.114:0/1546050016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66756910 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.713+0000 7f2718c02740 1 --2- 172.21.15.114:0/1546050016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663b51b0 0x55fb667563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.713+0000 7f2718c02740 1 -- 172.21.15.114:0/1546050016 >> 172.21.15.114:0/1546050016 conn(0x55fb663a9300 msgr2=0x55fb66752b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.713+0000 7f2718c02740 1 -- 172.21.15.114:0/1546050016 shutdown_connections 2024-05-27T07:45:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.713+0000 7f2718c02740 1 -- 172.21.15.114:0/1546050016 wait complete. 2024-05-27T07:45:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f2718c02740 1 -- start start 2024-05-27T07:45:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f2718c02740 1 Processor -- start 2024-05-27T07:45:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f2718c02740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66760ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f2718c02740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663944d0 0x55fb66761030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f2718c02740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663b51b0 0x55fb66766810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f2718c02740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fb6639e790 con 0x55fb663b51b0 2024-05-27T07:45:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f2718c02740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fb66674890 con 0x55fb663944d0 2024-05-27T07:45:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f2718c02740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fb667634a0 con 0x55fb6638c2f0 2024-05-27T07:45:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f2718c02740 10 client.0 _ll_register_callbacks cb 0x55fb663cca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f270bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663b51b0 0x55fb66766810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f270b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66760ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f270b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66760ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:38494/0 (socket says 172.21.15.114:38494) 2024-05-27T07:45:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f270b7fe640 1 -- 172.21.15.114:0/2582349752 learned_addr learned my addr 172.21.15.114:0/2582349752 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.714+0000 7f270affd640 1 --2- 172.21.15.114:0/2582349752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663944d0 0x55fb66761030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f270b7fe640 1 -- 172.21.15.114:0/2582349752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663944d0 msgr2=0x55fb66761030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f270b7fe640 1 --2- 172.21.15.114:0/2582349752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663944d0 0x55fb66761030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f270b7fe640 1 -- 172.21.15.114:0/2582349752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663b51b0 msgr2=0x55fb66766810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f270b7fe640 1 --2- 172.21.15.114:0/2582349752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663b51b0 0x55fb66766810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f270b7fe640 1 -- 172.21.15.114:0/2582349752 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb66499620 con 0x55fb6638c2f0 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f270bfff640 1 --2- 172.21.15.114:0/2582349752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663b51b0 0x55fb66766810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f270b7fe640 1 --2- 172.21.15.114:0/2582349752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66760ad0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f26fc02a930 tx=0x7f26fc009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e33cdaa2537bd40d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f26eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f26eaffd640 1 -- 172.21.15.114:0/2582349752 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc040070 con 0x55fb6638c2f0 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f2718c02740 1 -- 172.21.15.114:0/2582349752 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26fc004030 con 0x55fb6638c2f0 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f26eaffd640 1 -- 172.21.15.114:0/2582349752 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f26fc005260 con 0x55fb6638c2f0 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f26eaffd640 1 -- 172.21.15.114:0/2582349752 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f26fc056050 con 0x55fb6638c2f0 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f2718c02740 10 client.26293 fetch_fsmap learned FSMap version 35 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f2718c02740 1 -- 172.21.15.114:0/2582349752 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55fb6676f7e0 con 0x55fb6638c2f0 2024-05-27T07:45:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.715+0000 7f26eaffd640 1 -- 172.21.15.114:0/2582349752 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26fc004030 con 0x55fb6638c2f0 2024-05-27T07:45:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.716+0000 7f26eaffd640 1 -- 172.21.15.114:0/2582349752 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 35) ==== 7725+0+0 (secure 0 0 0) 0x7f26fc004b00 con 0x55fb6638c2f0 2024-05-27T07:45:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.716+0000 7f2718c02740 10 client.26293 fetch_fsmap finished waiting for FSMap version 35 2024-05-27T07:45:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.716+0000 7f2718c02740 10 client.26293 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.716+0000 7f2718c02740 20 client.26293 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.716+0000 7f2718c02740 1 --2- 172.21.15.114:0/2582349752 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fb66373d80 0x55fb66772b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.716+0000 7f2718c02740 4 client.26293 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.716+0000 7f2718c02740 1 -- 172.21.15.114:0/2582349752 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb666cab80 con 0x55fb66373d80 2024-05-27T07:45:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.716+0000 7f270affd640 1 --2- 172.21.15.114:0/2582349752 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fb66373d80 0x55fb66772b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.716+0000 7f26eaffd640 10 client.26293 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.716+0000 7f270affd640 1 --2- 172.21.15.114:0/2582349752 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fb66373d80 0x55fb66772b40 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54cd0fa3f248cefd server_cookie=d576ab4f6dfa9eee in_seq=0 out_seq=0 2024-05-27T07:45:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.717+0000 7f26eaffd640 1 -- 172.21.15.114:0/2582349752 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fb666cab80 con 0x55fb66373d80 2024-05-27T07:45:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.717+0000 7f26eaffd640 10 client.26293 handle_command_reply: tid=0 2024-05-27T07:45:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.717+0000 7f2718c02740 1 client.26293 shutdown 2024-05-27T07:45:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.718+0000 7f2718c02740 1 -- 172.21.15.114:0/2582349752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 msgr2=0x55fb66760ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.718+0000 7f2718c02740 1 --2- 172.21.15.114:0/2582349752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66760ad0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f26fc02a930 tx=0x7f26fc009260 comp rx=0 tx=0).stop 2024-05-27T07:45:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.718+0000 7f2718c02740 1 -- 172.21.15.114:0/2582349752 shutdown_connections 2024-05-27T07:45:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.718+0000 7f2718c02740 1 --2- 172.21.15.114:0/2582349752 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fb66373d80 0x55fb66772b40 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.718+0000 7f2718c02740 1 --2- 172.21.15.114:0/2582349752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663b51b0 0x55fb66766810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.718+0000 7f2718c02740 1 --2- 172.21.15.114:0/2582349752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663944d0 0x55fb66761030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.718+0000 7f26eaffd640 0 client.26293 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.718+0000 7f2718c02740 1 --2- 172.21.15.114:0/2582349752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66760ad0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.718+0000 7f2718c02740 1 -- 172.21.15.114:0/2582349752 >> 172.21.15.114:0/2582349752 conn(0x55fb663a9300 msgr2=0x55fb667580f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.718+0000 7f2718c02740 1 -- 172.21.15.114:0/2582349752 shutdown_connections 2024-05-27T07:45:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.718+0000 7f2718c02740 1 -- 172.21.15.114:0/2582349752 wait complete. 2024-05-27T07:45:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.718+0000 7f2718c02740 20 client.26293 trim_cache size 0 max 16384 2024-05-27T07:45:29.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:29 smithi114 ceph-mon[32038]: pgmap v181: 129 pgs: 129 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 4.9 MiB/s wr, 423 op/s 2024-05-27T07:45:29.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:29 smithi163 ceph-mon[36451]: pgmap v181: 129 pgs: 129 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 4.9 MiB/s wr, 423 op/s 2024-05-27T07:45:29.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:29 smithi186 ceph-mon[36647]: pgmap v181: 129 pgs: 129 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 4.9 MiB/s wr, 423 op/s 2024-05-27T07:45:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.734+0000 7f2718c02740 1 Processor -- start 2024-05-27T07:45:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.734+0000 7f2718c02740 1 -- start start 2024-05-27T07:45:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.734+0000 7f2718c02740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb6638c2f0 0x55fb66758870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.734+0000 7f2718c02740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb663944d0 0x55fb66757170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.734+0000 7f2718c02740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663b51b0 0x55fb66762530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.734+0000 7f2718c02740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f270c1b21c0 con 0x55fb6638c2f0 2024-05-27T07:45:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.734+0000 7f2718c02740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f270c1af270 con 0x55fb663b51b0 2024-05-27T07:45:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.734+0000 7f2718c02740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f270c1b1420 con 0x55fb663944d0 2024-05-27T07:45:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f270b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb6638c2f0 0x55fb66758870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f270b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb6638c2f0 0x55fb66758870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50000/0 (socket says 172.21.15.114:50000) 2024-05-27T07:45:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f270b7fe640 1 -- 172.21.15.114:0/4013675943 learned_addr learned my addr 172.21.15.114:0/4013675943 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f270bfff640 1 --2- 172.21.15.114:0/4013675943 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663b51b0 0x55fb66762530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f270affd640 1 --2- 172.21.15.114:0/4013675943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb663944d0 0x55fb66757170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f270b7fe640 1 -- 172.21.15.114:0/4013675943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb663944d0 msgr2=0x55fb66757170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f270b7fe640 1 --2- 172.21.15.114:0/4013675943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb663944d0 0x55fb66757170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f270b7fe640 1 -- 172.21.15.114:0/4013675943 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663b51b0 msgr2=0x55fb66762530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f270b7fe640 1 --2- 172.21.15.114:0/4013675943 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663b51b0 0x55fb66762530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f270b7fe640 1 -- 172.21.15.114:0/4013675943 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb667576b0 con 0x55fb6638c2f0 2024-05-27T07:45:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f270b7fe640 1 --2- 172.21.15.114:0/4013675943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb6638c2f0 0x55fb66758870 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f26fc031440 tx=0x7f26fc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ec23433a4ed3f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f26f17fa640 1 -- 172.21.15.114:0/4013675943 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc040040 con 0x55fb6638c2f0 2024-05-27T07:45:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f26f17fa640 1 -- 172.21.15.114:0/4013675943 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26fc04d070 con 0x55fb6638c2f0 2024-05-27T07:45:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.735+0000 7f26f17fa640 1 -- 172.21.15.114:0/4013675943 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc035040 con 0x55fb6638c2f0 2024-05-27T07:45:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.736+0000 7f2718c02740 1 -- 172.21.15.114:0/4013675943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb6638c2f0 msgr2=0x55fb66758870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.736+0000 7f2718c02740 1 --2- 172.21.15.114:0/4013675943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb6638c2f0 0x55fb66758870 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f26fc031440 tx=0x7f26fc000bc0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.736+0000 7f2718c02740 1 -- 172.21.15.114:0/4013675943 shutdown_connections 2024-05-27T07:45:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.736+0000 7f2718c02740 1 --2- 172.21.15.114:0/4013675943 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663b51b0 0x55fb66762530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.736+0000 7f2718c02740 1 --2- 172.21.15.114:0/4013675943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb663944d0 0x55fb66757170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.736+0000 7f2718c02740 1 --2- 172.21.15.114:0/4013675943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb6638c2f0 0x55fb66758870 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.736+0000 7f2718c02740 1 -- 172.21.15.114:0/4013675943 >> 172.21.15.114:0/4013675943 conn(0x55fb663a9300 msgr2=0x55fb66761e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.736+0000 7f2718c02740 1 -- 172.21.15.114:0/4013675943 shutdown_connections 2024-05-27T07:45:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.736+0000 7f2718c02740 1 -- 172.21.15.114:0/4013675943 wait complete. 2024-05-27T07:45:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f2718c02740 1 -- start start 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f2718c02740 1 Processor -- start 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f2718c02740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66769970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f2718c02740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663944d0 0x55fb66769eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f2718c02740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663b51b0 0x55fb6676a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f2718c02740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f270c1b0d50 con 0x55fb663b51b0 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f2718c02740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f270c1af8e0 con 0x55fb663944d0 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f2718c02740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f270c1affb0 con 0x55fb6638c2f0 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f2718c02740 10 client.0 _ll_register_callbacks cb 0x7f27000a5c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f270bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663b51b0 0x55fb6676a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f270affd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663944d0 0x55fb66769eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f270b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66769970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f270bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663b51b0 0x55fb6676a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50002/0 (socket says 172.21.15.114:50002) 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f270bfff640 1 -- 172.21.15.114:0/1908520754 learned_addr learned my addr 172.21.15.114:0/1908520754 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f270bfff640 1 -- 172.21.15.114:0/1908520754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 msgr2=0x55fb66769970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.737+0000 7f270bfff640 1 --2- 172.21.15.114:0/1908520754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66769970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f270bfff640 1 -- 172.21.15.114:0/1908520754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663944d0 msgr2=0x55fb66769eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f270bfff640 1 --2- 172.21.15.114:0/1908520754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663944d0 0x55fb66769eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f270bfff640 1 -- 172.21.15.114:0/1908520754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb667d8f80 con 0x55fb663b51b0 2024-05-27T07:45:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f270affd640 1 --2- 172.21.15.114:0/1908520754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663944d0 0x55fb66769eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f270bfff640 1 --2- 172.21.15.114:0/1908520754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663b51b0 0x55fb6676a3f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2704013fc0 tx=0x7f2704004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202af6d5c8bb3e82 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f26eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f26eaffd640 1 -- 172.21.15.114:0/1908520754 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704022480 con 0x55fb663b51b0 2024-05-27T07:45:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f26eaffd640 1 -- 172.21.15.114:0/1908520754 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2704022620 con 0x55fb663b51b0 2024-05-27T07:45:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f2718c02740 1 -- 172.21.15.114:0/1908520754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26fc04d070 con 0x55fb663b51b0 2024-05-27T07:45:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f26eaffd640 1 -- 172.21.15.114:0/1908520754 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2704035050 con 0x55fb663b51b0 2024-05-27T07:45:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f2718c02740 10 client.16701 fetch_fsmap learned FSMap version 35 2024-05-27T07:45:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f2718c02740 1 -- 172.21.15.114:0/1908520754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55fb667da250 con 0x55fb663b51b0 2024-05-27T07:45:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f26eaffd640 1 -- 172.21.15.114:0/1908520754 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26fc04d070 con 0x55fb663b51b0 2024-05-27T07:45:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f26eaffd640 1 -- 172.21.15.114:0/1908520754 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 35) ==== 7725+0+0 (secure 0 0 0) 0x7f270401f040 con 0x55fb663b51b0 2024-05-27T07:45:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f2718c02740 10 client.16701 fetch_fsmap finished waiting for FSMap version 35 2024-05-27T07:45:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f2718c02740 10 client.16701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f2718c02740 20 client.16701 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f2718c02740 1 --2- 172.21.15.114:0/1908520754 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fb663491c0 0x55fb667dca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f2718c02740 4 client.16701 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:45:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.738+0000 7f2718c02740 1 -- 172.21.15.114:0/1908520754 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fb666c4360 con 0x55fb663491c0 2024-05-27T07:45:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.739+0000 7f270b7fe640 1 --2- 172.21.15.114:0/1908520754 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fb663491c0 0x55fb667dca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.739+0000 7f26eaffd640 10 client.16701 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:29.739+0000 7f270b7fe640 1 --2- 172.21.15.114:0/1908520754 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fb663491c0 0x55fb667dca70 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfda4cfbaa8206d4 server_cookie=cdac32e291d4422 in_seq=0 out_seq=0 2024-05-27T07:45:30.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f26eaffd640 1 -- 172.21.15.114:0/1908520754 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55fb666c4360 con 0x55fb663491c0 2024-05-27T07:45:30.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f26eaffd640 10 client.16701 handle_command_reply: tid=0 2024-05-27T07:45:30.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f2718c02740 1 client.16701 shutdown 2024-05-27T07:45:30.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f2718c02740 1 -- 172.21.15.114:0/1908520754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663b51b0 msgr2=0x55fb6676a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:30.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f2718c02740 1 --2- 172.21.15.114:0/1908520754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663b51b0 0x55fb6676a3f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2704013fc0 tx=0x7f2704004a40 comp rx=0 tx=0).stop 2024-05-27T07:45:30.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f2718c02740 1 -- 172.21.15.114:0/1908520754 shutdown_connections 2024-05-27T07:45:30.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f2718c02740 1 --2- 172.21.15.114:0/1908520754 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fb663491c0 0x55fb667dca70 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:30.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f26eaffd640 0 client.16701 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f2718c02740 1 --2- 172.21.15.114:0/1908520754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb663b51b0 0x55fb6676a3f0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f2718c02740 1 --2- 172.21.15.114:0/1908520754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb663944d0 0x55fb66769eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f2718c02740 1 --2- 172.21.15.114:0/1908520754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fb6638c2f0 0x55fb66769970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f2718c02740 1 -- 172.21.15.114:0/1908520754 >> 172.21.15.114:0/1908520754 conn(0x55fb663a9300 msgr2=0x55fb66758550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f2718c02740 1 -- 172.21.15.114:0/1908520754 shutdown_connections 2024-05-27T07:45:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f2718c02740 1 -- 172.21.15.114:0/1908520754 wait complete. 2024-05-27T07:45:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.030+0000 7f2718c02740 20 client.16701 trim_cache size 0 max 16384 2024-05-27T07:45:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:45:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:45:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:45:30.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.031+0000 7f2711c6e640 1 -- 172.21.15.114:0/3568213107 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f26f40b2950 msgr2=0x7f26f40b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:30.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.031+0000 7f2711c6e640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f26f40b2950 0x7f26f40b4e60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55fb66753540 tx=0x7f2704004ab0 comp rx=0 tx=0).stop 2024-05-27T07:45:30.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.031+0000 7f2711c6e640 1 -- 172.21.15.114:0/3568213107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f270c10b7f0 msgr2=0x7f270c1a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:30.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.031+0000 7f2711c6e640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f270c10b7f0 0x7f270c1a0a40 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f270000e9f0 tx=0x7f270000eeb0 comp rx=0 tx=0).stop 2024-05-27T07:45:30.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.031+0000 7f2711c6e640 1 -- 172.21.15.114:0/3568213107 shutdown_connections 2024-05-27T07:45:30.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.031+0000 7f2711c6e640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f26f40b2950 0x7f26f40b4e60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:30.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.031+0000 7f2711c6e640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f270c10e1e0 0x7f270c19a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:30.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.031+0000 7f2711c6e640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f270c10b7f0 0x7f270c1a0a40 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:30.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.031+0000 7f2711c6e640 1 --2- 172.21.15.114:0/3568213107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c104d90 0x7f270c1a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:30.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.031+0000 7f2711c6e640 1 -- 172.21.15.114:0/3568213107 >> 172.21.15.114:0/3568213107 conn(0x7f270c0782d0 msgr2=0x7f270c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:30.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.032+0000 7f2711c6e640 1 -- 172.21.15.114:0/3568213107 shutdown_connections 2024-05-27T07:45:30.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:30.032+0000 7f2711c6e640 1 -- 172.21.15.114:0/3568213107 wait complete. 2024-05-27T07:45:30.034 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:45:30.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:45:30.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 554278 13.11 MB/sec execute 102 sec latency 226.917 ms 2024-05-27T07:45:30.286 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:31.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:30 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:45:31.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:30 smithi186 systemd[1]: Stopped Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:45:31.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:30 smithi186 systemd[1]: Starting Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:45:31.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:31 smithi186 podman[72706]: 2024-05-27 07:45:31.126482555 +0000 UTC m=+0.018759292 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:45:31.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:31 smithi186 podman[72706]: 2024-05-27 07:45:31.238783017 +0000 UTC m=+0.131059741 container create f49e98eb1e98f5920a6af0b617ac56a4156b3966668aa6fc0b9f0a49ecefd806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:45:31.249 INFO:tasks.workunit.client.0.smithi114.stdout: 1 556639 13.07 MB/sec execute 103 sec latency 25.189 ms 2024-05-27T07:45:31.509 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:31 smithi186 ceph-mon[36647]: pgmap v182: 129 pgs: 129 active+clean; 60 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 6.0 MiB/s wr, 597 op/s 2024-05-27T07:45:31.510 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:31 smithi186 podman[72706]: 2024-05-27 07:45:31.475310578 +0000 UTC m=+0.367587316 container init f49e98eb1e98f5920a6af0b617ac56a4156b3966668aa6fc0b9f0a49ecefd806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:45:31.510 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:31 smithi186 podman[72706]: 2024-05-27 07:45:31.478429045 +0000 UTC m=+0.370705778 container start f49e98eb1e98f5920a6af0b617ac56a4156b3966668aa6fc0b9f0a49ecefd806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-05-27T07:45:31.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:31 smithi163 ceph-mon[36451]: pgmap v182: 129 pgs: 129 active+clean; 60 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 6.0 MiB/s wr, 597 op/s 2024-05-27T07:45:31.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:31 smithi114 ceph-mon[32038]: pgmap v182: 129 pgs: 129 active+clean; 60 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 6.0 MiB/s wr, 597 op/s 2024-05-27T07:45:31.977 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:31 smithi186 ceph-mds[72725]: -- 172.21.15.186:0/3084021498 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d9d213da40 con 0x55d9d2185180 2024-05-27T07:45:31.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:31 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: starting mds.f at 2024-05-27T07:45:31.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:31 smithi186 bash[72706]: f49e98eb1e98f5920a6af0b617ac56a4156b3966668aa6fc0b9f0a49ecefd806 2024-05-27T07:45:31.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:45:31 smithi186 systemd[1]: Started Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:45:32.249 INFO:tasks.workunit.client.0.smithi114.stdout: 1 558813 13.07 MB/sec execute 104 sec latency 18.174 ms 2024-05-27T07:45:32.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.281+0000 7f42140e4700 1 -- 172.21.15.114:0/3743277847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f420c1034b0 msgr2=0x7f420c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:32.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.281+0000 7f42140e4700 1 --2- 172.21.15.114:0/3743277847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f420c1034b0 0x7f420c1038c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f42080099c0 tx=0x7f4208009cd0 comp rx=0 tx=0).stop 2024-05-27T07:45:32.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.281+0000 7f42140e4700 1 -- 172.21.15.114:0/3743277847 shutdown_connections 2024-05-27T07:45:32.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.281+0000 7f42140e4700 1 --2- 172.21.15.114:0/3743277847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f420c108730 0x7f420c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:32.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.281+0000 7f42140e4700 1 --2- 172.21.15.114:0/3743277847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f420c1034b0 0x7f420c1038c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:32.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.281+0000 7f42140e4700 1 --2- 172.21.15.114:0/3743277847 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c107de0 0x7f420c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:32.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.281+0000 7f42140e4700 1 -- 172.21.15.114:0/3743277847 >> 172.21.15.114:0/3743277847 conn(0x7f420c0fedb0 msgr2=0x7f420c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:32.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.281+0000 7f42140e4700 1 -- 172.21.15.114:0/3743277847 shutdown_connections 2024-05-27T07:45:32.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.281+0000 7f42140e4700 1 -- 172.21.15.114:0/3743277847 wait complete. 2024-05-27T07:45:32.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.282+0000 7f42140e4700 1 Processor -- start 2024-05-27T07:45:32.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.282+0000 7f42140e4700 1 -- start start 2024-05-27T07:45:32.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.282+0000 7f42140e4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034b0 0x7f420c19bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:32.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.282+0000 7f42140e4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f420c107de0 0x7f420c19c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:32.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.282+0000 7f42140e4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f420c108730 0x7f420c19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:32.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.282+0000 7f42140e4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f420c1133b0 con 0x7f420c107de0 2024-05-27T07:45:32.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.282+0000 7f42140e4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f420c113230 con 0x7f420c1034b0 2024-05-27T07:45:32.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.282+0000 7f42140e4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f420c113530 con 0x7f420c108730 2024-05-27T07:45:32.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.282+0000 7f421167f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f420c107de0 0x7f420c19c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.282+0000 7f4211e80700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034b0 0x7f420c19bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.282+0000 7f4211e80700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034b0 0x7f420c19bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:41870/0 (socket says 172.21.15.114:41870) 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.282+0000 7f4211e80700 1 -- 172.21.15.114:0/1198897480 learned_addr learned my addr 172.21.15.114:0/1198897480 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.282+0000 7f4212681700 1 --2- 172.21.15.114:0/1198897480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f420c108730 0x7f420c19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.283+0000 7f4211e80700 1 -- 172.21.15.114:0/1198897480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f420c108730 msgr2=0x7f420c19c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.283+0000 7f4211e80700 1 --2- 172.21.15.114:0/1198897480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f420c108730 0x7f420c19c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.283+0000 7f4211e80700 1 -- 172.21.15.114:0/1198897480 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f420c107de0 msgr2=0x7f420c19c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.283+0000 7f4211e80700 1 --2- 172.21.15.114:0/1198897480 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f420c107de0 0x7f420c19c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.283+0000 7f4211e80700 1 -- 172.21.15.114:0/1198897480 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420c1a7000 con 0x7f420c1034b0 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.283+0000 7f4211e80700 1 --2- 172.21.15.114:0/1198897480 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034b0 0x7f420c19bcb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4208005910 tx=0x7f4208004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.283+0000 7f4202ffd700 1 -- 172.21.15.114:0/1198897480 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4208026070 con 0x7f420c1034b0 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.283+0000 7f42140e4700 1 -- 172.21.15.114:0/1198897480 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420c1a7290 con 0x7f420c1034b0 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.283+0000 7f4202ffd700 1 -- 172.21.15.114:0/1198897480 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4208003c70 con 0x7f420c1034b0 2024-05-27T07:45:32.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.283+0000 7f4202ffd700 1 -- 172.21.15.114:0/1198897480 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420800fa40 con 0x7f420c1034b0 2024-05-27T07:45:32.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.283+0000 7f42140e4700 1 -- 172.21.15.114:0/1198897480 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f420c1a76f0 con 0x7f420c1034b0 2024-05-27T07:45:32.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.284+0000 7f42140e4700 1 -- 172.21.15.114:0/1198897480 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f420c0512b0 con 0x7f420c1034b0 2024-05-27T07:45:32.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.286+0000 7f4202ffd700 1 -- 172.21.15.114:0/1198897480 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f4208029070 con 0x7f420c1034b0 2024-05-27T07:45:32.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.286+0000 7f4202ffd700 1 --2- 172.21.15.114:0/1198897480 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f41f80b29f0 0x7f41f80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:32.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.287+0000 7f421167f700 1 --2- 172.21.15.114:0/1198897480 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f41f80b29f0 0x7f41f80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:32.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.287+0000 7f4202ffd700 1 -- 172.21.15.114:0/1198897480 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f42080e37e0 con 0x7f420c1034b0 2024-05-27T07:45:32.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.287+0000 7f421167f700 1 --2- 172.21.15.114:0/1198897480 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f41f80b29f0 0x7f41f80b4f00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f420c1024a0 tx=0x7f41fc009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:32.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.290+0000 7f4202ffd700 1 -- 172.21.15.114:0/1198897480 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f420805edf0 con 0x7f420c1034b0 2024-05-27T07:45:32.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:32 smithi114 ceph-mon[32038]: daemon mds.f restarted 2024-05-27T07:45:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.571+0000 7f42140e4700 1 -- 172.21.15.114:0/1198897480 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f420c02e5e0 con 0x7f420c1034b0 2024-05-27T07:45:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.571+0000 7f4202ffd700 1 -- 172.21.15.114:0/1198897480 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11000 (secure 0 0 0) 0x7f4208013070 con 0x7f420c1034b0 2024-05-27T07:45:32.571 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:45:32.571 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":26227,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2936519309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2936519309},{"type":"v1","addr":"172.21.15.186:6835","nonce":2936519309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:45:32.379189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26305":{"gid":26305,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/684229511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":684229511},{"type":"v1","addr":"172.21.15.186:6837","nonce":684229511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:45:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.574+0000 7f42140e4700 1 -- 172.21.15.114:0/1198897480 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f41f80b29f0 msgr2=0x7f41f80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.574+0000 7f42140e4700 1 --2- 172.21.15.114:0/1198897480 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f41f80b29f0 0x7f41f80b4f00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f420c1024a0 tx=0x7f41fc009260 comp rx=0 tx=0).stop 2024-05-27T07:45:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.574+0000 7f42140e4700 1 -- 172.21.15.114:0/1198897480 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034b0 msgr2=0x7f420c19bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.574+0000 7f42140e4700 1 --2- 172.21.15.114:0/1198897480 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034b0 0x7f420c19bcb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4208005910 tx=0x7f4208004270 comp rx=0 tx=0).stop 2024-05-27T07:45:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.574+0000 7f42140e4700 1 -- 172.21.15.114:0/1198897480 shutdown_connections 2024-05-27T07:45:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.574+0000 7f42140e4700 1 --2- 172.21.15.114:0/1198897480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f420c108730 0x7f420c19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.574+0000 7f42140e4700 1 --2- 172.21.15.114:0/1198897480 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f41f80b29f0 0x7f41f80b4f00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.574+0000 7f42140e4700 1 --2- 172.21.15.114:0/1198897480 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034b0 0x7f420c19bcb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.574+0000 7f42140e4700 1 --2- 172.21.15.114:0/1198897480 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f420c107de0 0x7f420c19c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.574+0000 7f42140e4700 1 -- 172.21.15.114:0/1198897480 >> 172.21.15.114:0/1198897480 conn(0x7f420c0fedb0 msgr2=0x7f420c109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.574+0000 7f42140e4700 1 -- 172.21.15.114:0/1198897480 shutdown_connections 2024-05-27T07:45:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:32.574+0000 7f42140e4700 1 -- 172.21.15.114:0/1198897480 wait complete. 2024-05-27T07:45:32.575 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 37 2024-05-27T07:45:32.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:32 smithi163 ceph-mon[36451]: daemon mds.f restarted 2024-05-27T07:45:32.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:32 smithi186 ceph-mon[36647]: daemon mds.f restarted 2024-05-27T07:45:33.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:45:33.248 INFO:tasks.workunit.client.0.smithi114.stdout: 1 561634 13.11 MB/sec execute 105 sec latency 25.059 ms 2024-05-27T07:45:33.477 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:33 smithi186 ceph-mon[36647]: pgmap v183: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 5.5 MiB/s wr, 497 op/s 2024-05-27T07:45:33.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:33 smithi186 ceph-mon[36647]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T07:45:33.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:33 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6836/684229511,v1:172.21.15.186:6837/684229511] up:boot 2024-05-27T07:45:33.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:33 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:45:33.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:45:33.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:33 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:45:33.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:33 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1198897480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:45:33.495 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:33.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:33 smithi114 ceph-mon[32038]: pgmap v183: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 5.5 MiB/s wr, 497 op/s 2024-05-27T07:45:33.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:33 smithi114 ceph-mon[32038]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T07:45:33.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:33 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6836/684229511,v1:172.21.15.186:6837/684229511] up:boot 2024-05-27T07:45:33.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:33 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:45:33.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:45:33.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:33 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:45:33.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:33 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1198897480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:45:33.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:33 smithi163 ceph-mon[36451]: pgmap v183: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 5.5 MiB/s wr, 497 op/s 2024-05-27T07:45:33.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:33 smithi163 ceph-mon[36451]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T07:45:33.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:33 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6836/684229511,v1:172.21.15.186:6837/684229511] up:boot 2024-05-27T07:45:33.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:33 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:45:33.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:45:33.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:33 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:45:33.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:33 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1198897480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:45:34.249 INFO:tasks.workunit.client.0.smithi114.stdout: 1 563850 13.06 MB/sec execute 106 sec latency 49.131 ms 2024-05-27T07:45:35.249 INFO:tasks.workunit.client.0.smithi114.stdout: 1 565835 13.06 MB/sec execute 107 sec latency 55.206 ms 2024-05-27T07:45:35.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.299+0000 7f1f09b0c700 1 -- 172.21.15.114:0/982520717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f04107de0 msgr2=0x7f1f041081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:35.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.299+0000 7f1f09b0c700 1 --2- 172.21.15.114:0/982520717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f04107de0 0x7f1f041081f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80099c0 tx=0x7f1ef8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.299+0000 7f1f09b0c700 1 -- 172.21.15.114:0/982520717 shutdown_connections 2024-05-27T07:45:35.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.299+0000 7f1f09b0c700 1 --2- 172.21.15.114:0/982520717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f04108730 0x7f1f04110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.299+0000 7f1f09b0c700 1 --2- 172.21.15.114:0/982520717 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f041034b0 0x7f1f041038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.299+0000 7f1f09b0c700 1 --2- 172.21.15.114:0/982520717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f04107de0 0x7f1f041081f0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.299+0000 7f1f09b0c700 1 -- 172.21.15.114:0/982520717 >> 172.21.15.114:0/982520717 conn(0x7f1f040fedb0 msgr2=0x7f1f041011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:35.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.299+0000 7f1f09b0c700 1 -- 172.21.15.114:0/982520717 shutdown_connections 2024-05-27T07:45:35.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.299+0000 7f1f09b0c700 1 -- 172.21.15.114:0/982520717 wait complete. 2024-05-27T07:45:35.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.300+0000 7f1f09b0c700 1 Processor -- start 2024-05-27T07:45:35.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f09b0c700 1 -- start start 2024-05-27T07:45:35.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f09b0c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f041034b0 0x7f1f0419d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f09b0c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f04108730 0x7f1f0419d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f09b0c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f041a1920 0x7f1f041a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f09b0c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1f04113360 con 0x7f1f041034b0 2024-05-27T07:45:35.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f09b0c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1f041131e0 con 0x7f1f04108730 2024-05-27T07:45:35.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f09b0c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1f041134e0 con 0x7f1f041a1920 2024-05-27T07:45:35.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f037fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f041034b0 0x7f1f0419d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f037fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f041034b0 0x7f1f0419d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52822/0 (socket says 172.21.15.114:52822) 2024-05-27T07:45:35.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f037fe700 1 -- 172.21.15.114:0/2954995139 learned_addr learned my addr 172.21.15.114:0/2954995139 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:35.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f04108730 0x7f1f0419d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f03fff700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f041a1920 0x7f1f041a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f02ffd700 1 -- 172.21.15.114:0/2954995139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f041a1920 msgr2=0x7f1f041a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:35.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.301+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f041a1920 0x7f1f041a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.302+0000 7f1f02ffd700 1 -- 172.21.15.114:0/2954995139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f041034b0 msgr2=0x7f1f0419d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:35.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.302+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f041034b0 0x7f1f0419d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.302+0000 7f1f02ffd700 1 -- 172.21.15.114:0/2954995139 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f041a2470 con 0x7f1f04108730 2024-05-27T07:45:35.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.302+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f04108730 0x7f1f0419d750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1f041042e0 tx=0x7f1ef8004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:35.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.302+0000 7f1f00ff9700 1 -- 172.21.15.114:0/2954995139 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8009e60 con 0x7f1f04108730 2024-05-27T07:45:35.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.302+0000 7f1f00ff9700 1 -- 172.21.15.114:0/2954995139 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ef8003c70 con 0x7f1f04108730 2024-05-27T07:45:35.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.302+0000 7f1f09b0c700 1 -- 172.21.15.114:0/2954995139 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f041a2700 con 0x7f1f04108730 2024-05-27T07:45:35.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.302+0000 7f1f00ff9700 1 -- 172.21.15.114:0/2954995139 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef800faa0 con 0x7f1f04108730 2024-05-27T07:45:35.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.302+0000 7f1f09b0c700 1 -- 172.21.15.114:0/2954995139 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f041a9df0 con 0x7f1f04108730 2024-05-27T07:45:35.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.306+0000 7f1f00ff9700 1 -- 172.21.15.114:0/2954995139 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f1ef800fca0 con 0x7f1f04108730 2024-05-27T07:45:35.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.306+0000 7f1f00ff9700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1eec0b29f0 0x7f1eec0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.307+0000 7f1f18521b80 1 Processor -- start 2024-05-27T07:45:35.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.307+0000 7f1f18521b80 1 -- start start 2024-05-27T07:45:35.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.307+0000 7f1f037fe700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1eec0b29f0 0x7f1eec0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.307+0000 7f1f18521b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1ddbae40 0x55bd1df3e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.307+0000 7f1f18521b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 0x55bd1df38a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.307+0000 7f1f18521b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 0x55bd1df39390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.307+0000 7f1f18521b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bd1de67c90 con 0x55bd1df3ec50 2024-05-27T07:45:35.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.307+0000 7f1f18521b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bd1de57830 con 0x55bd1ddbae40 2024-05-27T07:45:35.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.307+0000 7f1f18521b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bd1de5b870 con 0x55bd1df38f80 2024-05-27T07:45:35.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.307+0000 7f1f03fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 0x55bd1df39390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.307+0000 7f1f03fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 0x55bd1df39390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58882/0 (socket says 172.21.15.114:58882) 2024-05-27T07:45:35.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.307+0000 7f1f03fff700 1 -- 172.21.15.114:0/120932841 learned_addr learned my addr 172.21.15.114:0/120932841 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:35.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.308+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/120932841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 0x55bd1df38a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.308+0000 7f1f03fff700 1 -- 172.21.15.114:0/120932841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1ddbae40 msgr2=0x55bd1df3e710 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:45:35.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.308+0000 7f1f03fff700 1 --2- 172.21.15.114:0/120932841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1ddbae40 0x55bd1df3e710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.308+0000 7f1f00ff9700 1 -- 172.21.15.114:0/2954995139 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f1ef80e34c0 con 0x7f1f04108730 2024-05-27T07:45:35.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.308+0000 7f1f03fff700 1 -- 172.21.15.114:0/120932841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 msgr2=0x55bd1df38a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:35.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.308+0000 7f1f037fe700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1eec0b29f0 0x7f1eec0b4f00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009d70 tx=0x7f1ef4009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:35.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.308+0000 7f1f03fff700 1 --2- 172.21.15.114:0/120932841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 0x55bd1df38a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.308+0000 7f1f03fff700 1 -- 172.21.15.114:0/120932841 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd1dda78d0 con 0x55bd1df38f80 2024-05-27T07:45:35.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.308+0000 7f1f03fff700 1 --2- 172.21.15.114:0/120932841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 0x55bd1df39390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1efc011680 tx=0x7f1efc009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d88f212bd772972 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:35.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.308+0000 7f1ee9ffb700 1 -- 172.21.15.114:0/120932841 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1efc019030 con 0x55bd1df38f80 2024-05-27T07:45:35.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.309+0000 7f1ee9ffb700 1 -- 172.21.15.114:0/120932841 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1efc0092e0 con 0x55bd1df38f80 2024-05-27T07:45:35.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.309+0000 7f1ee9ffb700 1 -- 172.21.15.114:0/120932841 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1efc007500 con 0x55bd1df38f80 2024-05-27T07:45:35.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.309+0000 7f1f18521b80 1 -- 172.21.15.114:0/120932841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 msgr2=0x55bd1df39390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.309+0000 7f1f18521b80 1 --2- 172.21.15.114:0/120932841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 0x55bd1df39390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1efc011680 tx=0x7f1efc009920 comp rx=0 tx=0).stop 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.309+0000 7f1f18521b80 1 -- 172.21.15.114:0/120932841 shutdown_connections 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.309+0000 7f1f18521b80 1 --2- 172.21.15.114:0/120932841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 0x55bd1df39390 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.309+0000 7f1f18521b80 1 --2- 172.21.15.114:0/120932841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1ddbae40 0x55bd1df3e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.309+0000 7f1f18521b80 1 --2- 172.21.15.114:0/120932841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 0x55bd1df38a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.309+0000 7f1f18521b80 1 -- 172.21.15.114:0/120932841 >> 172.21.15.114:0/120932841 conn(0x55bd1de15af0 msgr2=0x55bd1df33640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.309+0000 7f1f18521b80 1 -- 172.21.15.114:0/120932841 shutdown_connections 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.310+0000 7f1f18521b80 1 -- 172.21.15.114:0/120932841 wait complete. 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.310+0000 7f1f18521b80 1 -- start start 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.310+0000 7f1f18521b80 1 Processor -- start 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.310+0000 7f1f18521b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1ddbae40 0x55bd1df42380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.310+0000 7f1f18521b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df38f80 0x55bd1df428c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.310+0000 7f1f18521b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 0x55bd1df42e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.310+0000 7f1f18521b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bd1dd5dfa0 con 0x55bd1df3ec50 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f18521b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bd1ddf97c0 con 0x55bd1df38f80 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f18521b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bd1df45740 con 0x55bd1ddbae40 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f18521b80 10 client.0 _ll_register_callbacks cb 0x55bd1ddf2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f037fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1ddbae40 0x55bd1df42380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f037fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1ddbae40 0x55bd1df42380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58898/0 (socket says 172.21.15.114:58898) 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f037fe700 1 -- 172.21.15.114:0/3766973642 learned_addr learned my addr 172.21.15.114:0/3766973642 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f03fff700 1 --2- 172.21.15.114:0/3766973642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 0x55bd1df42e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/3766973642 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df38f80 0x55bd1df428c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f037fe700 1 -- 172.21.15.114:0/3766973642 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df38f80 msgr2=0x55bd1df428c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f037fe700 1 --2- 172.21.15.114:0/3766973642 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df38f80 0x55bd1df428c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f037fe700 1 -- 172.21.15.114:0/3766973642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 msgr2=0x55bd1df42e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f037fe700 1 --2- 172.21.15.114:0/3766973642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 0x55bd1df42e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f037fe700 1 -- 172.21.15.114:0/3766973642 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd1ddf6690 con 0x55bd1ddbae40 2024-05-27T07:45:35.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f037fe700 1 --2- 172.21.15.114:0/3766973642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1ddbae40 0x55bd1df42380 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1ef400f970 tx=0x7f1ef40047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae5a2e2c0bb49cc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1ee2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1ee2ffd700 1 -- 172.21.15.114:0/3766973642 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4012e10 con 0x55bd1ddbae40 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1f18521b80 1 -- 172.21.15.114:0/3766973642 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1efc0092e0 con 0x55bd1ddbae40 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.311+0000 7f1ee2ffd700 1 -- 172.21.15.114:0/3766973642 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f1ef4026480 con 0x55bd1ddbae40 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.312+0000 7f1ee2ffd700 1 -- 172.21.15.114:0/3766973642 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1ef402f050 con 0x55bd1ddbae40 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.312+0000 7f1f18521b80 10 client.26323 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.312+0000 7f1f18521b80 1 -- 172.21.15.114:0/3766973642 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bd1df501a0 con 0x55bd1ddbae40 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.312+0000 7f1ee2ffd700 1 -- 172.21.15.114:0/3766973642 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1efc0092e0 con 0x55bd1ddbae40 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.312+0000 7f1ee2ffd700 1 -- 172.21.15.114:0/3766973642 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f1ef401d880 con 0x55bd1ddbae40 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.312+0000 7f1f18521b80 10 client.26323 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.312+0000 7f1f18521b80 10 client.26323 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.312+0000 7f1f18521b80 20 client.26323 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.312+0000 7f1f18521b80 1 --2- 172.21.15.114:0/3766973642 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bd1df52bf0 0x55bd1df550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.312+0000 7f1f18521b80 4 client.26323 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.312+0000 7f1f18521b80 1 -- 172.21.15.114:0/3766973642 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd1df049f0 con 0x55bd1df52bf0 2024-05-27T07:45:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.312+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/3766973642 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bd1df52bf0 0x55bd1df550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.313+0000 7f1ee2ffd700 10 client.26323 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.313+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/3766973642 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bd1df52bf0 0x55bd1df550c0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c33270d63c557892 server_cookie=e55d10f2ec96fca6 in_seq=0 out_seq=0 2024-05-27T07:45:35.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.314+0000 7f1ee2ffd700 1 -- 172.21.15.114:0/3766973642 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bd1df049f0 con 0x55bd1df52bf0 2024-05-27T07:45:35.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.314+0000 7f1ee2ffd700 10 client.26323 handle_command_reply: tid=0 2024-05-27T07:45:35.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.314+0000 7f1f18521b80 1 client.26323 shutdown 2024-05-27T07:45:35.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.315+0000 7f1f18521b80 1 -- 172.21.15.114:0/3766973642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1ddbae40 msgr2=0x55bd1df42380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:35.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.315+0000 7f1f18521b80 1 --2- 172.21.15.114:0/3766973642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1ddbae40 0x55bd1df42380 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1ef400f970 tx=0x7f1ef40047c0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.315+0000 7f1f18521b80 1 -- 172.21.15.114:0/3766973642 shutdown_connections 2024-05-27T07:45:35.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.315+0000 7f1f18521b80 1 --2- 172.21.15.114:0/3766973642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 0x55bd1df42e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.315+0000 7f1f18521b80 1 --2- 172.21.15.114:0/3766973642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1ddbae40 0x55bd1df42380 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.315+0000 7f1f18521b80 1 --2- 172.21.15.114:0/3766973642 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bd1df52bf0 0x55bd1df550c0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.315+0000 7f1f18521b80 1 --2- 172.21.15.114:0/3766973642 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df38f80 0x55bd1df428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.315+0000 7f1f18521b80 1 -- 172.21.15.114:0/3766973642 >> 172.21.15.114:0/3766973642 conn(0x55bd1de15af0 msgr2=0x55bd1df37a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:35.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.315+0000 7f1ee2ffd700 0 client.26323 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:35.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.315+0000 7f1f18521b80 1 -- 172.21.15.114:0/3766973642 shutdown_connections 2024-05-27T07:45:35.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.315+0000 7f1f18521b80 1 -- 172.21.15.114:0/3766973642 wait complete. 2024-05-27T07:45:35.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.315+0000 7f1f18521b80 20 client.26323 trim_cache size 0 max 16384 2024-05-27T07:45:35.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.332+0000 7f1f18521b80 1 Processor -- start 2024-05-27T07:45:35.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.332+0000 7f1f18521b80 1 -- start start 2024-05-27T07:45:35.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f18521b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 0x55bd1df38a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f18521b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df3ec50 0x55bd1df4e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f18521b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df41380 0x55bd1df3d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f18521b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1f041b3af0 con 0x55bd1df41380 2024-05-27T07:45:35.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f18521b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1f041b0ba0 con 0x55bd1df3ec50 2024-05-27T07:45:35.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f18521b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1f041b2d50 con 0x55bd1df38f80 2024-05-27T07:45:35.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f037fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 0x55bd1df38a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f03fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df41380 0x55bd1df3d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f037fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 0x55bd1df38a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58900/0 (socket says 172.21.15.114:58900) 2024-05-27T07:45:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f037fe700 1 -- 172.21.15.114:0/326308437 learned_addr learned my addr 172.21.15.114:0/326308437 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/326308437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df3ec50 0x55bd1df4e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f037fe700 1 -- 172.21.15.114:0/326308437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df3ec50 msgr2=0x55bd1df4e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f037fe700 1 --2- 172.21.15.114:0/326308437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df3ec50 0x55bd1df4e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f037fe700 1 -- 172.21.15.114:0/326308437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df41380 msgr2=0x55bd1df3d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f037fe700 1 --2- 172.21.15.114:0/326308437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df41380 0x55bd1df3d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f037fe700 1 -- 172.21.15.114:0/326308437 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd1df347a0 con 0x55bd1df38f80 2024-05-27T07:45:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/326308437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df3ec50 0x55bd1df4e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:45:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f03fff700 1 --2- 172.21.15.114:0/326308437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df41380 0x55bd1df3d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1f037fe700 1 --2- 172.21.15.114:0/326308437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 0x55bd1df38a40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1ef401dc50 tx=0x7f1ef4009fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d06615957a9123f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.333+0000 7f1ee97fa700 1 -- 172.21.15.114:0/326308437 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef401d420 con 0x55bd1df38f80 2024-05-27T07:45:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.334+0000 7f1ee97fa700 1 -- 172.21.15.114:0/326308437 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ef401c620 con 0x55bd1df38f80 2024-05-27T07:45:35.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.334+0000 7f1ee97fa700 1 -- 172.21.15.114:0/326308437 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4026480 con 0x55bd1df38f80 2024-05-27T07:45:35.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.334+0000 7f1f18521b80 1 -- 172.21.15.114:0/326308437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 msgr2=0x55bd1df38a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:35.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.334+0000 7f1f18521b80 1 --2- 172.21.15.114:0/326308437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 0x55bd1df38a40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1ef401dc50 tx=0x7f1ef4009fd0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.334+0000 7f1f18521b80 1 -- 172.21.15.114:0/326308437 shutdown_connections 2024-05-27T07:45:35.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.334+0000 7f1f18521b80 1 --2- 172.21.15.114:0/326308437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df41380 0x55bd1df3d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.334+0000 7f1f18521b80 1 --2- 172.21.15.114:0/326308437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df38f80 0x55bd1df38a40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.334+0000 7f1f18521b80 1 --2- 172.21.15.114:0/326308437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df3ec50 0x55bd1df4e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.334+0000 7f1f18521b80 1 -- 172.21.15.114:0/326308437 >> 172.21.15.114:0/326308437 conn(0x55bd1de15af0 msgr2=0x55bd1df40c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:35.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.335+0000 7f1f18521b80 1 -- 172.21.15.114:0/326308437 shutdown_connections 2024-05-27T07:45:35.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.335+0000 7f1f18521b80 1 -- 172.21.15.114:0/326308437 wait complete. 2024-05-27T07:45:35.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.335+0000 7f1f18521b80 1 -- start start 2024-05-27T07:45:35.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.335+0000 7f1f18521b80 1 Processor -- start 2024-05-27T07:45:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.335+0000 7f1f18521b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df38f80 0x55bd1df4ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.335+0000 7f1f18521b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 0x55bd1df4b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f18521b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df41380 0x55bd1df59c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f18521b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1f041b2680 con 0x55bd1df3ec50 2024-05-27T07:45:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f18521b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1f041b1210 con 0x55bd1df38f80 2024-05-27T07:45:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f18521b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1f041b18e0 con 0x55bd1df41380 2024-05-27T07:45:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f18521b80 10 client.0 _ll_register_callbacks cb 0x55bd1ddf2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f03fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df41380 0x55bd1df59c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f03fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df41380 0x55bd1df59c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58902/0 (socket says 172.21.15.114:58902) 2024-05-27T07:45:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f03fff700 1 -- 172.21.15.114:0/350323430 learned_addr learned my addr 172.21.15.114:0/350323430 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 0x55bd1df4b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f037fe700 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df38f80 0x55bd1df4ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f02ffd700 1 -- 172.21.15.114:0/350323430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df41380 msgr2=0x55bd1df59c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df41380 0x55bd1df59c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f02ffd700 1 -- 172.21.15.114:0/350323430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df38f80 msgr2=0x55bd1df4ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df38f80 0x55bd1df4ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f02ffd700 1 -- 172.21.15.114:0/350323430 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd1df5a430 con 0x55bd1df3ec50 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f03fff700 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df41380 0x55bd1df59c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1f02ffd700 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 0x55bd1df4b140 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80e0860 tx=0x7f1ef8005860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76225c2bbb47caa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.336+0000 7f1ee2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.337+0000 7f1ee2ffd700 1 -- 172.21.15.114:0/350323430 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef80a9f90 con 0x55bd1df3ec50 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.337+0000 7f1f18521b80 1 -- 172.21.15.114:0/350323430 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ef401c620 con 0x55bd1df3ec50 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.337+0000 7f1ee2ffd700 1 -- 172.21.15.114:0/350323430 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f1ef8069050 con 0x55bd1df3ec50 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.337+0000 7f1ee2ffd700 1 -- 172.21.15.114:0/350323430 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1ef80aab80 con 0x55bd1df3ec50 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.337+0000 7f1f18521b80 10 client.16731 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.337+0000 7f1f18521b80 1 -- 172.21.15.114:0/350323430 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bd1dfbc7b0 con 0x55bd1df3ec50 2024-05-27T07:45:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.337+0000 7f1ee2ffd700 1 -- 172.21.15.114:0/350323430 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ef8068020 con 0x55bd1df3ec50 2024-05-27T07:45:35.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.337+0000 7f1ee2ffd700 1 -- 172.21.15.114:0/350323430 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f1ef80e3160 con 0x55bd1df3ec50 2024-05-27T07:45:35.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.337+0000 7f1f18521b80 10 client.16731 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:35.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.337+0000 7f1f18521b80 10 client.16731 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:35.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.338+0000 7f1f18521b80 20 client.16731 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:35.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.338+0000 7f1f18521b80 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bd1dfbce10 0x55bd1dfbf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:35.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.338+0000 7f1f18521b80 4 client.16731 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:45:35.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.338+0000 7f1f18521b80 1 -- 172.21.15.114:0/350323430 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f1ef401c620 con 0x55bd1dfbce10 2024-05-27T07:45:35.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.338+0000 7f1f037fe700 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bd1dfbce10 0x55bd1dfbf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:35.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.338+0000 7f1ee2ffd700 10 client.16731 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:35.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.338+0000 7f1f037fe700 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bd1dfbce10 0x55bd1dfbf230 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d366ef63c844c83e server_cookie=723cc6251ef7f31e in_seq=0 out_seq=0 2024-05-27T07:45:35.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.340+0000 7f1ee2ffd700 1 -- 172.21.15.114:0/350323430 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f1ef401c620 con 0x55bd1dfbce10 2024-05-27T07:45:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.340+0000 7f1ee2ffd700 10 client.16731 handle_command_reply: tid=0 2024-05-27T07:45:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.340+0000 7f1f18521b80 1 client.16731 shutdown 2024-05-27T07:45:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.340+0000 7f1f18521b80 1 -- 172.21.15.114:0/350323430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 msgr2=0x55bd1df4b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.340+0000 7f1f18521b80 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 0x55bd1df4b140 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80e0860 tx=0x7f1ef8005860 comp rx=0 tx=0).stop 2024-05-27T07:45:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.341+0000 7f1f18521b80 1 -- 172.21.15.114:0/350323430 shutdown_connections 2024-05-27T07:45:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.341+0000 7f1f18521b80 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd1df41380 0x55bd1df59c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.341+0000 7f1f18521b80 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bd1dfbce10 0x55bd1dfbf230 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.341+0000 7f1f18521b80 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd1df38f80 0x55bd1df4ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.341+0000 7f1f18521b80 1 --2- 172.21.15.114:0/350323430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bd1df3ec50 0x55bd1df4b140 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.341+0000 7f1ee2ffd700 0 client.16731 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.341+0000 7f1f18521b80 1 -- 172.21.15.114:0/350323430 >> 172.21.15.114:0/350323430 conn(0x55bd1de15af0 msgr2=0x55bd1df3d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.341+0000 7f1f18521b80 1 -- 172.21.15.114:0/350323430 shutdown_connections 2024-05-27T07:45:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.341+0000 7f1f18521b80 1 -- 172.21.15.114:0/350323430 wait complete. 2024-05-27T07:45:35.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.341+0000 7f1f18521b80 20 client.16731 trim_cache size 0 max 16384 2024-05-27T07:45:35.343 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:45:35.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.342+0000 7f1f09b0c700 1 -- 172.21.15.114:0/2954995139 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1eec0b29f0 msgr2=0x7f1eec0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:35.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.342+0000 7f1f09b0c700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1eec0b29f0 0x7f1eec0b4f00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009d70 tx=0x7f1ef4009260 comp rx=0 tx=0).stop 2024-05-27T07:45:35.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.342+0000 7f1f09b0c700 1 -- 172.21.15.114:0/2954995139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f04108730 msgr2=0x7f1f0419d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:35.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.342+0000 7f1f09b0c700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f04108730 0x7f1f0419d750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1f041042e0 tx=0x7f1ef8004270 comp rx=0 tx=0).stop 2024-05-27T07:45:35.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.342+0000 7f1f09b0c700 1 -- 172.21.15.114:0/2954995139 shutdown_connections 2024-05-27T07:45:35.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.342+0000 7f1f09b0c700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f041a1920 0x7f1f041a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.342+0000 7f1f09b0c700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f041034b0 0x7f1f0419d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.342+0000 7f1f09b0c700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1eec0b29f0 0x7f1eec0b4f00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.342+0000 7f1f09b0c700 1 --2- 172.21.15.114:0/2954995139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f04108730 0x7f1f0419d750 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:35.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.342+0000 7f1f09b0c700 1 -- 172.21.15.114:0/2954995139 >> 172.21.15.114:0/2954995139 conn(0x7f1f040fedb0 msgr2=0x7f1f041097d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:35.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.342+0000 7f1f09b0c700 1 -- 172.21.15.114:0/2954995139 shutdown_connections 2024-05-27T07:45:35.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:35.342+0000 7f1f09b0c700 1 -- 172.21.15.114:0/2954995139 wait complete. 2024-05-27T07:45:35.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:35 smithi114 ceph-mon[32038]: pgmap v185: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 5.9 MiB/s wr, 467 op/s 2024-05-27T07:45:35.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:35 smithi163 ceph-mon[36451]: pgmap v185: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 5.9 MiB/s wr, 467 op/s 2024-05-27T07:45:35.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:35 smithi186 ceph-mon[36647]: pgmap v185: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 5.9 MiB/s wr, 467 op/s 2024-05-27T07:45:36.249 INFO:tasks.workunit.client.0.smithi114.stdout: 1 568798 13.10 MB/sec execute 108 sec latency 13.944 ms 2024-05-27T07:45:37.037 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:45:37.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:45:37.249 INFO:tasks.workunit.client.0.smithi114.stdout: 1 571636 13.11 MB/sec execute 109 sec latency 68.788 ms 2024-05-27T07:45:37.307 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:37.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:37 smithi114 ceph-mon[32038]: pgmap v186: 129 pgs: 129 active+clean; 66 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-05-27T07:45:37.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:37 smithi163 ceph-mon[36451]: pgmap v186: 129 pgs: 129 active+clean; 66 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-05-27T07:45:37.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:37 smithi186 ceph-mon[36647]: pgmap v186: 129 pgs: 129 active+clean; 66 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-05-27T07:45:38.249 INFO:tasks.workunit.client.0.smithi114.stdout: 1 573749 13.09 MB/sec execute 110 sec latency 26.816 ms 2024-05-27T07:45:39.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.047+0000 7f9d264e3700 1 -- 172.21.15.114:0/406476424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d2010a920 msgr2=0x7f9d2010ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:39.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.047+0000 7f9d264e3700 1 --2- 172.21.15.114:0/406476424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d2010a920 0x7f9d2010ad30 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f9d140098f0 tx=0x7f9d14009c00 comp rx=0 tx=0).stop 2024-05-27T07:45:39.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.049+0000 7f9d264e3700 1 -- 172.21.15.114:0/406476424 shutdown_connections 2024-05-27T07:45:39.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.049+0000 7f9d264e3700 1 --2- 172.21.15.114:0/406476424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d2010b270 0x7f9d201198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.049+0000 7f9d264e3700 1 --2- 172.21.15.114:0/406476424 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d20111bc0 0x7f9d20111fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.049+0000 7f9d264e3700 1 --2- 172.21.15.114:0/406476424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d2010a920 0x7f9d2010ad30 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.049+0000 7f9d264e3700 1 -- 172.21.15.114:0/406476424 >> 172.21.15.114:0/406476424 conn(0x7f9d2006f4c0 msgr2=0x7f9d2006f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:39.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.052+0000 7f9d264e3700 1 -- 172.21.15.114:0/406476424 shutdown_connections 2024-05-27T07:45:39.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.052+0000 7f9d264e3700 1 -- 172.21.15.114:0/406476424 wait complete. 2024-05-27T07:45:39.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.053+0000 7f9d264e3700 1 Processor -- start 2024-05-27T07:45:39.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.053+0000 7f9d264e3700 1 -- start start 2024-05-27T07:45:39.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.054+0000 7f9d264e3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d2010a920 0x7f9d201a3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.054+0000 7f9d264e3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d2010b270 0x7f9d201a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.054+0000 7f9d264e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d20111bc0 0x7f9d201a7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.054+0000 7f9d264e3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2011cce0 con 0x7f9d20111bc0 2024-05-27T07:45:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.054+0000 7f9d264e3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2011cb60 con 0x7f9d2010b270 2024-05-27T07:45:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.054+0000 7f9d24a80700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d20111bc0 0x7f9d201a7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.055+0000 7f9d264e3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2011ce60 con 0x7f9d2010a920 2024-05-27T07:45:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.055+0000 7f9d1f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d2010b270 0x7f9d201a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.055+0000 7f9d1f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d2010b270 0x7f9d201a39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:40042/0 (socket says 172.21.15.114:40042) 2024-05-27T07:45:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.055+0000 7f9d1f7fe700 1 -- 172.21.15.114:0/3265800874 learned_addr learned my addr 172.21.15.114:0/3265800874 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:39.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.056+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/3265800874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d2010a920 0x7f9d201a3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.058+0000 7f9d1f7fe700 1 -- 172.21.15.114:0/3265800874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d2010a920 msgr2=0x7f9d201a3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:39.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.058+0000 7f9d1f7fe700 1 --2- 172.21.15.114:0/3265800874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d2010a920 0x7f9d201a3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.058+0000 7f9d1f7fe700 1 -- 172.21.15.114:0/3265800874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d20111bc0 msgr2=0x7f9d201a7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:39.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.058+0000 7f9d1f7fe700 1 --2- 172.21.15.114:0/3265800874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d20111bc0 0x7f9d201a7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.058+0000 7f9d1f7fe700 1 -- 172.21.15.114:0/3265800874 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d201a8300 con 0x7f9d2010b270 2024-05-27T07:45:39.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.059+0000 7f9d1f7fe700 1 --2- 172.21.15.114:0/3265800874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d2010b270 0x7f9d201a39d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9d14009fd0 tx=0x7f9d1400b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:39.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.061+0000 7f9d1d7fa700 1 -- 172.21.15.114:0/3265800874 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d14025070 con 0x7f9d2010b270 2024-05-27T07:45:39.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.061+0000 7f9d1d7fa700 1 -- 172.21.15.114:0/3265800874 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d1400bb10 con 0x7f9d2010b270 2024-05-27T07:45:39.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.061+0000 7f9d1d7fa700 1 -- 172.21.15.114:0/3265800874 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d14004e50 con 0x7f9d2010b270 2024-05-27T07:45:39.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.061+0000 7f9d264e3700 1 -- 172.21.15.114:0/3265800874 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d201a8590 con 0x7f9d2010b270 2024-05-27T07:45:39.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.061+0000 7f9d264e3700 1 -- 172.21.15.114:0/3265800874 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d201a8a20 con 0x7f9d2010b270 2024-05-27T07:45:39.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.070+0000 7f9d34ef8b80 1 Processor -- start 2024-05-27T07:45:39.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.070+0000 7f9d34ef8b80 1 -- start start 2024-05-27T07:45:39.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.071+0000 7f9d34ef8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c6b9ec0 0x55641c83ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.071+0000 7f9d34ef8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c838c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.071+0000 7f9d34ef8b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 0x55641c8395b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.071+0000 7f9d34ef8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55641c766df0 con 0x55641c83f0f0 2024-05-27T07:45:39.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.071+0000 7f9d34ef8b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55641c756990 con 0x55641c8391a0 2024-05-27T07:45:39.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.071+0000 7f9d34ef8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55641c75a9d0 con 0x55641c6b9ec0 2024-05-27T07:45:39.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.072+0000 7f9d1ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c838c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.072+0000 7f9d1ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c838c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52876/0 (socket says 172.21.15.114:52876) 2024-05-27T07:45:39.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.072+0000 7f9d1ffff700 1 -- 172.21.15.114:0/1429799715 learned_addr learned my addr 172.21.15.114:0/1429799715 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:39.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.073+0000 7f9d1ffff700 1 -- 172.21.15.114:0/1429799715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c6b9ec0 msgr2=0x55641c83ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:39.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.073+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/1429799715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c6b9ec0 0x55641c83ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.073+0000 7f9d1d7fa700 1 -- 172.21.15.114:0/3265800874 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f9d14028070 con 0x7f9d2010b270 2024-05-27T07:45:39.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.073+0000 7f9d1ffff700 1 -- 172.21.15.114:0/1429799715 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 msgr2=0x55641c8395b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:45:39.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.073+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/1429799715 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 0x55641c8395b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.073+0000 7f9d1ffff700 1 -- 172.21.15.114:0/1429799715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55641c700030 con 0x55641c83f0f0 2024-05-27T07:45:39.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.074+0000 7f9d1d7fa700 1 --2- 172.21.15.114:0/3265800874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9d080b2da0 0x7f9d080b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.074+0000 7f9d24a80700 1 --2- 172.21.15.114:0/3265800874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9d080b2da0 0x7f9d080b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.074+0000 7f9d1d7fa700 1 -- 172.21.15.114:0/3265800874 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f9d140e3190 con 0x7f9d2010b270 2024-05-27T07:45:39.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.075+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/1429799715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c838c60 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9d18011600 tx=0x7f9d180099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef8520425c074a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:39.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.075+0000 7f9d24a80700 1 --2- 172.21.15.114:0/3265800874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9d080b2da0 0x7f9d080b52b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9d10005950 tx=0x7f9d100058e0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:39.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.076+0000 7f9d067fc700 1 -- 172.21.15.114:0/1429799715 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18003a00 con 0x55641c83f0f0 2024-05-27T07:45:39.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.076+0000 7f9d067fc700 1 -- 172.21.15.114:0/1429799715 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d18003ba0 con 0x55641c83f0f0 2024-05-27T07:45:39.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.076+0000 7f9d067fc700 1 -- 172.21.15.114:0/1429799715 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18004290 con 0x55641c83f0f0 2024-05-27T07:45:39.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.078+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1429799715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 msgr2=0x55641c838c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:39.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.078+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1429799715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c838c60 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9d18011600 tx=0x7f9d180099e0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.079+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1429799715 shutdown_connections 2024-05-27T07:45:39.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.079+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1429799715 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 0x55641c8395b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.079+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1429799715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c6b9ec0 0x55641c83ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.079+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1429799715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c838c60 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.079+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1429799715 >> 172.21.15.114:0/1429799715 conn(0x55641c6bc7f0 msgr2=0x55641c833e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:39.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.080+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1429799715 shutdown_connections 2024-05-27T07:45:39.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.082+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1429799715 wait complete. 2024-05-27T07:45:39.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.083+0000 7f9d34ef8b80 1 -- start start 2024-05-27T07:45:39.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.084+0000 7f9d34ef8b80 1 Processor -- start 2024-05-27T07:45:39.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.084+0000 7f9d34ef8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c6b9ec0 0x55641c842900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.085+0000 7f9d34ef8b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 0x55641c842e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.085+0000 7f9d34ef8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c843380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.085+0000 7f9d24a80700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c843380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.085+0000 7f9d24a80700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c843380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52882/0 (socket says 172.21.15.114:52882) 2024-05-27T07:45:39.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.085+0000 7f9d24a80700 1 -- 172.21.15.114:0/1751769076 learned_addr learned my addr 172.21.15.114:0/1751769076 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:39.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.085+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/1751769076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c6b9ec0 0x55641c842900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.085+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1751769076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55641c65d020 con 0x55641c83f0f0 2024-05-27T07:45:39.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.085+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1751769076 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55641c6bee20 con 0x55641c8391a0 2024-05-27T07:45:39.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.085+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1751769076 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55641c845be0 con 0x55641c6b9ec0 2024-05-27T07:45:39.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.086+0000 7f9d34ef8b80 10 client.0 _ll_register_callbacks cb 0x55641c6ed630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:39.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.087+0000 7f9d1ffff700 1 -- 172.21.15.114:0/1751769076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 msgr2=0x55641c842e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:39.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.087+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/1751769076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 0x55641c842e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.087+0000 7f9d1ffff700 1 -- 172.21.15.114:0/1751769076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 msgr2=0x55641c843380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:39.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.087+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/1751769076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c843380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.087+0000 7f9d1ffff700 1 -- 172.21.15.114:0/1751769076 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55641c6a6950 con 0x55641c6b9ec0 2024-05-27T07:45:39.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.088+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/1751769076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c6b9ec0 0x55641c842900 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9d18012010 tx=0x7f9d18016690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=419f70fc8ee92866 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:39.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.089+0000 7f9cff7fe700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:45:39.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.089+0000 7f9cff7fe700 1 -- 172.21.15.114:0/1751769076 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18003a00 con 0x55641c6b9ec0 2024-05-27T07:45:39.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.089+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1751769076 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d18003ba0 con 0x55641c6b9ec0 2024-05-27T07:45:39.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.089+0000 7f9cff7fe700 1 -- 172.21.15.114:0/1751769076 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f9d18016ac0 con 0x55641c6b9ec0 2024-05-27T07:45:39.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.089+0000 7f9cff7fe700 1 -- 172.21.15.114:0/1751769076 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9d18033050 con 0x55641c6b9ec0 2024-05-27T07:45:39.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.089+0000 7f9d34ef8b80 10 client.26335 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:39.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.089+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1751769076 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55641c7107e0 con 0x55641c6b9ec0 2024-05-27T07:45:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.090+0000 7f9cff7fe700 1 -- 172.21.15.114:0/1751769076 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d18003ba0 con 0x55641c6b9ec0 2024-05-27T07:45:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.091+0000 7f9cff7fe700 1 -- 172.21.15.114:0/1751769076 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f9d1801fc90 con 0x55641c6b9ec0 2024-05-27T07:45:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.091+0000 7f9d34ef8b80 10 client.26335 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.091+0000 7f9d34ef8b80 10 client.26335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.091+0000 7f9d34ef8b80 20 client.26335 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.091+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1751769076 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55641c857260 0x55641c859770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.091+0000 7f9d34ef8b80 4 client.26335 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.091+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1751769076 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55641c768d40 con 0x55641c857260 2024-05-27T07:45:39.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.092+0000 7f9d1f7fe700 1 --2- 172.21.15.114:0/1751769076 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55641c857260 0x55641c859770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.096+0000 7f9d1f7fe700 1 --2- 172.21.15.114:0/1751769076 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55641c857260 0x55641c859770 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d89748a6060fcac server_cookie=5cef04dc2c95a32a in_seq=0 out_seq=0 2024-05-27T07:45:39.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.096+0000 7f9cff7fe700 10 client.26335 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:39.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.099+0000 7f9cff7fe700 1 -- 172.21.15.114:0/1751769076 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55641c768d40 con 0x55641c857260 2024-05-27T07:45:39.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.099+0000 7f9cff7fe700 10 client.26335 handle_command_reply: tid=0 2024-05-27T07:45:39.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.099+0000 7f9d34ef8b80 1 client.26335 shutdown 2024-05-27T07:45:39.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.100+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1751769076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c6b9ec0 msgr2=0x55641c842900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:39.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.100+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1751769076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c6b9ec0 0x55641c842900 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9d18012010 tx=0x7f9d18016690 comp rx=0 tx=0).stop 2024-05-27T07:45:39.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.100+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1751769076 shutdown_connections 2024-05-27T07:45:39.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.100+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1751769076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c843380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.100+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1751769076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c6b9ec0 0x55641c842900 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9d18012010 tx=0x7f9d18016690 comp rx=0 tx=0).stop 2024-05-27T07:45:39.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.100+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1751769076 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55641c857260 0x55641c859770 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.100+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1751769076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 0x55641c842e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.100+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1751769076 >> 172.21.15.114:0/1751769076 conn(0x55641c6bc7f0 msgr2=0x55641c837c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:39.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.101+0000 7f9cff7fe700 0 client.26335 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:39.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.104+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1751769076 shutdown_connections 2024-05-27T07:45:39.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.104+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1751769076 wait complete. 2024-05-27T07:45:39.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.104+0000 7f9d34ef8b80 20 client.26335 trim_cache size 0 max 16384 2024-05-27T07:45:39.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.143+0000 7f9d34ef8b80 1 Processor -- start 2024-05-27T07:45:39.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.143+0000 7f9d34ef8b80 1 -- start start 2024-05-27T07:45:39.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.143+0000 7f9d34ef8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c8391a0 0x55641c838c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.143+0000 7f9d34ef8b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c83f0f0 0x55641c8431f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.143+0000 7f9d34ef8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c840700 0x55641c83ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.143+0000 7f9d34ef8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c00fb90 con 0x55641c840700 2024-05-27T07:45:39.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.143+0000 7f9d34ef8b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c00ccf0 con 0x55641c83f0f0 2024-05-27T07:45:39.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.143+0000 7f9d34ef8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c00edf0 con 0x55641c8391a0 2024-05-27T07:45:39.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.143+0000 7f9d1ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c8391a0 0x55641c838c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.143+0000 7f9d1ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c8391a0 0x55641c838c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58950/0 (socket says 172.21.15.114:58950) 2024-05-27T07:45:39.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.143+0000 7f9d1ffff700 1 -- 172.21.15.114:0/1654887636 learned_addr learned my addr 172.21.15.114:0/1654887636 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:39.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.143+0000 7f9d24a80700 1 --2- 172.21.15.114:0/1654887636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c840700 0x55641c83ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.143+0000 7f9d1f7fe700 1 --2- 172.21.15.114:0/1654887636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c83f0f0 0x55641c8431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.144+0000 7f9d1ffff700 1 -- 172.21.15.114:0/1654887636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c83f0f0 msgr2=0x55641c8431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:39.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.144+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/1654887636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c83f0f0 0x55641c8431f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.144+0000 7f9d1ffff700 1 -- 172.21.15.114:0/1654887636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c840700 msgr2=0x55641c83ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:39.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.144+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/1654887636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c840700 0x55641c83ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.144+0000 7f9d1ffff700 1 -- 172.21.15.114:0/1654887636 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55641c835610 con 0x55641c8391a0 2024-05-27T07:45:39.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.144+0000 7f9d1f7fe700 1 --2- 172.21.15.114:0/1654887636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c83f0f0 0x55641c8431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:39.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.144+0000 7f9d24a80700 1 --2- 172.21.15.114:0/1654887636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c840700 0x55641c83ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.144+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/1654887636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c8391a0 0x55641c838c60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9d1801e4f0 tx=0x7f9d18016830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d51e6428a80d184d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.144+0000 7f9d05ffb700 1 -- 172.21.15.114:0/1654887636 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1803c070 con 0x55641c8391a0 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.144+0000 7f9d05ffb700 1 -- 172.21.15.114:0/1654887636 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d1801e860 con 0x55641c8391a0 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.144+0000 7f9d05ffb700 1 -- 172.21.15.114:0/1654887636 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18016d20 con 0x55641c8391a0 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.145+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1654887636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c8391a0 msgr2=0x55641c838c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.145+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1654887636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c8391a0 0x55641c838c60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9d1801e4f0 tx=0x7f9d18016830 comp rx=0 tx=0).stop 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.146+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1654887636 shutdown_connections 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.146+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1654887636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c840700 0x55641c83ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.146+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1654887636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c8391a0 0x55641c838c60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.146+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/1654887636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c83f0f0 0x55641c8431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.146+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1654887636 >> 172.21.15.114:0/1654887636 conn(0x55641c6bc7f0 msgr2=0x55641c82d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.146+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1654887636 shutdown_connections 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.146+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/1654887636 wait complete. 2024-05-27T07:45:39.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.146+0000 7f9d34ef8b80 1 -- start start 2024-05-27T07:45:39.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.147+0000 7f9d34ef8b80 1 Processor -- start 2024-05-27T07:45:39.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.147+0000 7f9d34ef8b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 0x55641c84aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.147+0000 7f9d34ef8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c84b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.147+0000 7f9d34ef8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c840700 0x55641c85e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.147+0000 7f9d34ef8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c00e720 con 0x55641c83f0f0 2024-05-27T07:45:39.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.147+0000 7f9d1ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 0x55641c84aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.147+0000 7f9d1ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 0x55641c84aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:40080/0 (socket says 172.21.15.114:40080) 2024-05-27T07:45:39.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.147+0000 7f9d1ffff700 1 -- 172.21.15.114:0/2377521267 learned_addr learned my addr 172.21.15.114:0/2377521267 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:39.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.147+0000 7f9d1f7fe700 1 --2- 172.21.15.114:0/2377521267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c84b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.147+0000 7f9d34ef8b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c00d2b0 con 0x55641c8391a0 2024-05-27T07:45:39.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.147+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/2377521267 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c00d980 con 0x55641c840700 2024-05-27T07:45:39.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.147+0000 7f9d24a80700 1 --2- 172.21.15.114:0/2377521267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c840700 0x55641c85e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.147+0000 7f9d34ef8b80 10 client.0 _ll_register_callbacks cb 0x55641c6ed630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.148+0000 7f9d1ffff700 1 -- 172.21.15.114:0/2377521267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c840700 msgr2=0x55641c85e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.148+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/2377521267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c840700 0x55641c85e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.148+0000 7f9d1ffff700 1 -- 172.21.15.114:0/2377521267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 msgr2=0x55641c84b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.148+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/2377521267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c84b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.148+0000 7f9d1ffff700 1 -- 172.21.15.114:0/2377521267 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55641c85e730 con 0x55641c8391a0 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.148+0000 7f9d1ffff700 1 --2- 172.21.15.114:0/2377521267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 0x55641c84aed0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009ac0 tx=0x7f9d18009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e8e2eebb71cc5a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.148+0000 7f9cff7fe700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.148+0000 7f9cff7fe700 1 -- 172.21.15.114:0/2377521267 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1803c070 con 0x55641c8391a0 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.148+0000 7f9cff7fe700 1 -- 172.21.15.114:0/2377521267 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f9d18005d70 con 0x55641c8391a0 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.148+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/2377521267 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d1801e860 con 0x55641c8391a0 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.148+0000 7f9cff7fe700 1 -- 172.21.15.114:0/2377521267 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9d18048050 con 0x55641c8391a0 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.148+0000 7f9d34ef8b80 10 client.26243 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.148+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/2377521267 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55641c8c16e0 con 0x55641c8391a0 2024-05-27T07:45:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.149+0000 7f9cff7fe700 1 -- 172.21.15.114:0/2377521267 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d1801e860 con 0x55641c8391a0 2024-05-27T07:45:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.149+0000 7f9cff7fe700 1 -- 172.21.15.114:0/2377521267 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f9d18003b20 con 0x55641c8391a0 2024-05-27T07:45:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.149+0000 7f9d34ef8b80 10 client.26243 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.149+0000 7f9d34ef8b80 10 client.26243 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.149+0000 7f9d34ef8b80 20 client.26243 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.149+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/2377521267 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55641c8c1ca0 0x55641c8c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.149+0000 7f9d34ef8b80 4 client.26243 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:45:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.149+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/2377521267 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55641c762520 con 0x55641c8c1ca0 2024-05-27T07:45:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.149+0000 7f9d1f7fe700 1 --2- 172.21.15.114:0/2377521267 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55641c8c1ca0 0x55641c8c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.149+0000 7f9cff7fe700 10 client.26243 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.150+0000 7f9d1f7fe700 1 --2- 172.21.15.114:0/2377521267 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55641c8c1ca0 0x55641c8c40c0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=392d3a96ac0ef299 server_cookie=cf26bd7334280137 in_seq=0 out_seq=0 2024-05-27T07:45:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.157+0000 7f9cff7fe700 1 -- 172.21.15.114:0/2377521267 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55641c762520 con 0x55641c8c1ca0 2024-05-27T07:45:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.157+0000 7f9cff7fe700 10 client.26243 handle_command_reply: tid=0 2024-05-27T07:45:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.158+0000 7f9d34ef8b80 1 client.26243 shutdown 2024-05-27T07:45:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.158+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/2377521267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 msgr2=0x55641c84aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.158+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/2377521267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 0x55641c84aed0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009ac0 tx=0x7f9d18009b80 comp rx=0 tx=0).stop 2024-05-27T07:45:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.158+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/2377521267 shutdown_connections 2024-05-27T07:45:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.158+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/2377521267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55641c840700 0x55641c85e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.158+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/2377521267 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55641c8c1ca0 0x55641c8c40c0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.158+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/2377521267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55641c8391a0 0x55641c84aed0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.158+0000 7f9cff7fe700 0 client.26243 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.158+0000 7f9d34ef8b80 1 --2- 172.21.15.114:0/2377521267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55641c83f0f0 0x55641c84b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.158+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/2377521267 >> 172.21.15.114:0/2377521267 conn(0x55641c6bc7f0 msgr2=0x55641c83f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.158+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/2377521267 shutdown_connections 2024-05-27T07:45:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.158+0000 7f9d34ef8b80 1 -- 172.21.15.114:0/2377521267 wait complete. 2024-05-27T07:45:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.158+0000 7f9d34ef8b80 20 client.26243 trim_cache size 0 max 16384 2024-05-27T07:45:39.467 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:45:39.467 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:45:39.467 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "41ccb17e-30ff-4d13-b326-6550dbe09481", 2024-05-27T07:45:39.468 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:45:39.468 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:45:39.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.159+0000 7f9d264e3700 1 -- 172.21.15.114:0/3265800874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9d080b2da0 msgr2=0x7f9d080b52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:39.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.159+0000 7f9d264e3700 1 --2- 172.21.15.114:0/3265800874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9d080b2da0 0x7f9d080b52b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9d10005950 tx=0x7f9d100058e0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.159+0000 7f9d264e3700 1 -- 172.21.15.114:0/3265800874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d2010b270 msgr2=0x7f9d201a39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:39.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.159+0000 7f9d264e3700 1 --2- 172.21.15.114:0/3265800874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d2010b270 0x7f9d201a39d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9d14009fd0 tx=0x7f9d1400b790 comp rx=0 tx=0).stop 2024-05-27T07:45:39.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.159+0000 7f9d264e3700 1 -- 172.21.15.114:0/3265800874 shutdown_connections 2024-05-27T07:45:39.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.159+0000 7f9d264e3700 1 --2- 172.21.15.114:0/3265800874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d20111bc0 0x7f9d201a7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.159+0000 7f9d264e3700 1 --2- 172.21.15.114:0/3265800874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d2010a920 0x7f9d201a3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.159+0000 7f9d264e3700 1 --2- 172.21.15.114:0/3265800874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9d080b2da0 0x7f9d080b52b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.159+0000 7f9d264e3700 1 --2- 172.21.15.114:0/3265800874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d2010b270 0x7f9d201a39d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:39.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.159+0000 7f9d264e3700 1 -- 172.21.15.114:0/3265800874 >> 172.21.15.114:0/3265800874 conn(0x7f9d2006f4c0 msgr2=0x7f9d200728d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:39.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.159+0000 7f9d264e3700 1 -- 172.21.15.114:0/3265800874 shutdown_connections 2024-05-27T07:45:39.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:39.159+0000 7f9d264e3700 1 -- 172.21.15.114:0/3265800874 wait complete. 2024-05-27T07:45:39.469 INFO:tasks.workunit.client.0.smithi114.stdout: 1 575881 13.09 MB/sec execute 111 sec latency 83.937 ms 2024-05-27T07:45:39.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:39 smithi163 ceph-mon[36451]: pgmap v187: 129 pgs: 129 active+clean; 66 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-05-27T07:45:39.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:39 smithi186 ceph-mon[36647]: pgmap v187: 129 pgs: 129 active+clean; 66 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-05-27T07:45:39.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:39 smithi114 ceph-mon[32038]: pgmap v187: 129 pgs: 129 active+clean; 66 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-05-27T07:45:40.249 INFO:tasks.workunit.client.0.smithi114.stdout: 1 578433 13.09 MB/sec execute 112 sec latency 71.739 ms 2024-05-27T07:45:40.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:45:40.544 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:40.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:40 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:45:40.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:40 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:40.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:40 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:45:40.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:40 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:40.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:40 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:45:40.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:40 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:40.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:40 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:45:40.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:40 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:40.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:40 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:45:40.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:40 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:40.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:40 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:45:40.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:40 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:41.249 INFO:tasks.workunit.client.0.smithi114.stdout: 1 580465 13.07 MB/sec execute 113 sec latency 30.441 ms 2024-05-27T07:45:41.696 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:41 smithi114 ceph-mon[32038]: pgmap v188: 129 pgs: 129 active+clean; 72 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 310 op/s 2024-05-27T07:45:41.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:41 smithi163 ceph-mon[36451]: pgmap v188: 129 pgs: 129 active+clean; 72 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 310 op/s 2024-05-27T07:45:41.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:41 smithi186 ceph-mon[36647]: pgmap v188: 129 pgs: 129 active+clean; 72 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 310 op/s 2024-05-27T07:45:42.249 INFO:tasks.workunit.client.0.smithi114.stdout: 1 583193 13.08 MB/sec execute 114 sec latency 20.030 ms 2024-05-27T07:45:42.659 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:42 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:45:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.670+0000 7f398aec9700 1 -- 172.21.15.114:0/1818784225 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3984108730 msgr2=0x7f3984110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.670+0000 7f398aec9700 1 --2- 172.21.15.114:0/1818784225 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3984108730 0x7f3984110090 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f397c00b0d0 tx=0x7f397c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.670+0000 7f398aec9700 1 -- 172.21.15.114:0/1818784225 shutdown_connections 2024-05-27T07:45:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.670+0000 7f398aec9700 1 --2- 172.21.15.114:0/1818784225 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3984108730 0x7f3984110090 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.670+0000 7f398aec9700 1 --2- 172.21.15.114:0/1818784225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39841034b0 0x7f39841038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.670+0000 7f398aec9700 1 --2- 172.21.15.114:0/1818784225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3984107de0 0x7f39841081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.670+0000 7f398aec9700 1 -- 172.21.15.114:0/1818784225 >> 172.21.15.114:0/1818784225 conn(0x7f39840fedb0 msgr2=0x7f39841011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.671+0000 7f398aec9700 1 -- 172.21.15.114:0/1818784225 shutdown_connections 2024-05-27T07:45:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.671+0000 7f398aec9700 1 -- 172.21.15.114:0/1818784225 wait complete. 2024-05-27T07:45:42.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.671+0000 7f398aec9700 1 Processor -- start 2024-05-27T07:45:42.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.671+0000 7f398aec9700 1 -- start start 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f398aec9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39841034b0 0x7f398419bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f398aec9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3984107de0 0x7f398419c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f398aec9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3984108730 0x7f398419c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f398aec9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3984113360 con 0x7f3984107de0 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f398aec9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f39841131e0 con 0x7f39841034b0 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f398aec9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f39841134e0 con 0x7f3984108730 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f3983fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3984107de0 0x7f398419c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f3988c65700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39841034b0 0x7f398419bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f3983fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3984107de0 0x7f398419c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52910/0 (socket says 172.21.15.114:52910) 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f3983fff700 1 -- 172.21.15.114:0/304177303 learned_addr learned my addr 172.21.15.114:0/304177303 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f3989466700 1 --2- 172.21.15.114:0/304177303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3984108730 0x7f398419c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f3983fff700 1 -- 172.21.15.114:0/304177303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3984108730 msgr2=0x7f398419c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f3983fff700 1 --2- 172.21.15.114:0/304177303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3984108730 0x7f398419c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f3983fff700 1 -- 172.21.15.114:0/304177303 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39841034b0 msgr2=0x7f398419bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f3983fff700 1 --2- 172.21.15.114:0/304177303 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39841034b0 0x7f398419bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f3983fff700 1 -- 172.21.15.114:0/304177303 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39841a6f90 con 0x7f3984107de0 2024-05-27T07:45:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.672+0000 7f3983fff700 1 --2- 172.21.15.114:0/304177303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3984107de0 0x7f398419c160 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f397000d980 tx=0x7f397000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.673+0000 7f3981ffb700 1 -- 172.21.15.114:0/304177303 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3970017070 con 0x7f3984107de0 2024-05-27T07:45:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.673+0000 7f3981ffb700 1 -- 172.21.15.114:0/304177303 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f397000fd40 con 0x7f3984107de0 2024-05-27T07:45:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.673+0000 7f398aec9700 1 -- 172.21.15.114:0/304177303 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39841a7280 con 0x7f3984107de0 2024-05-27T07:45:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.673+0000 7f3981ffb700 1 -- 172.21.15.114:0/304177303 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397000e3b0 con 0x7f3984107de0 2024-05-27T07:45:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.673+0000 7f398aec9700 1 -- 172.21.15.114:0/304177303 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39841a77c0 con 0x7f3984107de0 2024-05-27T07:45:42.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.676+0000 7f3981ffb700 1 -- 172.21.15.114:0/304177303 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f3970010610 con 0x7f3984107de0 2024-05-27T07:45:42.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.676+0000 7f3981ffb700 1 --2- 172.21.15.114:0/304177303 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f39740b29f0 0x7f39740b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.676+0000 7f3988c65700 1 --2- 172.21.15.114:0/304177303 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f39740b29f0 0x7f39740b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.676+0000 7f3981ffb700 1 -- 172.21.15.114:0/304177303 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f39700e23d0 con 0x7f3984107de0 2024-05-27T07:45:42.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f3988c65700 1 --2- 172.21.15.114:0/304177303 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f39740b29f0 0x7f39740b4f00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3978009c60 tx=0x7f3978009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:42.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f39998deb80 1 Processor -- start 2024-05-27T07:45:42.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f39998deb80 1 -- start start 2024-05-27T07:45:42.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f39998deb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c118523290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f39998deb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1185237d0 0x55c118527a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f39998deb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c118527fb0 0x55c1185283c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f39998deb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c11844cd70 con 0x55c11839fe40 2024-05-27T07:45:42.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f39998deb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c11843c910 con 0x55c1185237d0 2024-05-27T07:45:42.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f39998deb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c118440950 con 0x55c118527fb0 2024-05-27T07:45:42.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f3988c65700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c118523290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f3988c65700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c118523290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52916/0 (socket says 172.21.15.114:52916) 2024-05-27T07:45:42.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f3988c65700 1 -- 172.21.15.114:0/3258311378 learned_addr learned my addr 172.21.15.114:0/3258311378 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:42.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f3989466700 1 --2- 172.21.15.114:0/3258311378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c118527fb0 0x55c1185283c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.677+0000 7f3983fff700 1 --2- 172.21.15.114:0/3258311378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1185237d0 0x55c118527a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.678+0000 7f3988c65700 1 -- 172.21.15.114:0/3258311378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c118527fb0 msgr2=0x55c1185283c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:42.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.678+0000 7f3988c65700 1 --2- 172.21.15.114:0/3258311378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c118527fb0 0x55c1185283c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.678+0000 7f3988c65700 1 -- 172.21.15.114:0/3258311378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1185237d0 msgr2=0x55c118527a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:42.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.678+0000 7f3988c65700 1 --2- 172.21.15.114:0/3258311378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1185237d0 0x55c118527a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.678+0000 7f3988c65700 1 -- 172.21.15.114:0/3258311378 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1183e5fb0 con 0x55c11839fe40 2024-05-27T07:45:42.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.678+0000 7f3989466700 1 --2- 172.21.15.114:0/3258311378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c118527fb0 0x55c1185283c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:42.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.678+0000 7f3983fff700 1 --2- 172.21.15.114:0/3258311378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1185237d0 0x55c118527a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:42.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.678+0000 7f3988c65700 1 --2- 172.21.15.114:0/3258311378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c118523290 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f397800e4e0 tx=0x7f3978014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29bd4efe7025bf2e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:42.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.678+0000 7f396effd700 1 -- 172.21.15.114:0/3258311378 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397800ee00 con 0x55c11839fe40 2024-05-27T07:45:42.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.678+0000 7f396effd700 1 -- 172.21.15.114:0/3258311378 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f397801c8c0 con 0x55c11839fe40 2024-05-27T07:45:42.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.678+0000 7f396effd700 1 -- 172.21.15.114:0/3258311378 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397801d9c0 con 0x55c11839fe40 2024-05-27T07:45:42.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.679+0000 7f39998deb80 1 -- 172.21.15.114:0/3258311378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 msgr2=0x55c118523290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:42.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.679+0000 7f39998deb80 1 --2- 172.21.15.114:0/3258311378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c118523290 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f397800e4e0 tx=0x7f3978014000 comp rx=0 tx=0).stop 2024-05-27T07:45:42.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.679+0000 7f39998deb80 1 -- 172.21.15.114:0/3258311378 shutdown_connections 2024-05-27T07:45:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.679+0000 7f39998deb80 1 --2- 172.21.15.114:0/3258311378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c118527fb0 0x55c1185283c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.679+0000 7f39998deb80 1 --2- 172.21.15.114:0/3258311378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c118523290 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.679+0000 7f39998deb80 1 --2- 172.21.15.114:0/3258311378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1185237d0 0x55c118527a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.679+0000 7f39998deb80 1 -- 172.21.15.114:0/3258311378 >> 172.21.15.114:0/3258311378 conn(0x55c1183a2770 msgr2=0x55c118519d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.679+0000 7f39998deb80 1 -- 172.21.15.114:0/3258311378 shutdown_connections 2024-05-27T07:45:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.679+0000 7f39998deb80 1 -- 172.21.15.114:0/3258311378 wait complete. 2024-05-27T07:45:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.679+0000 7f39998deb80 1 -- start start 2024-05-27T07:45:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.680+0000 7f39998deb80 1 Processor -- start 2024-05-27T07:45:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.680+0000 7f39998deb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c11852ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.680+0000 7f39998deb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1185237d0 0x55c11852d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.680+0000 7f39998deb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c118527fb0 0x55c11852d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.680+0000 7f39998deb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c118342fa0 con 0x55c11839fe40 2024-05-27T07:45:42.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.680+0000 7f39998deb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c1183a4da0 con 0x55c1185237d0 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.680+0000 7f39998deb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c11852ffc0 con 0x55c118527fb0 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.680+0000 7f39998deb80 10 client.0 _ll_register_callbacks cb 0x55c1183d35b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.680+0000 7f3988c65700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c11852ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.680+0000 7f3988c65700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c11852ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52924/0 (socket says 172.21.15.114:52924) 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.680+0000 7f3988c65700 1 -- 172.21.15.114:0/2162426249 learned_addr learned my addr 172.21.15.114:0/2162426249 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.680+0000 7f3983fff700 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1185237d0 0x55c11852d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.680+0000 7f3989466700 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c118527fb0 0x55c11852d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f3988c65700 1 -- 172.21.15.114:0/2162426249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c118527fb0 msgr2=0x55c11852d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f3988c65700 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c118527fb0 0x55c11852d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f3988c65700 1 -- 172.21.15.114:0/2162426249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1185237d0 msgr2=0x55c11852d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f3988c65700 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1185237d0 0x55c11852d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f3988c65700 1 -- 172.21.15.114:0/2162426249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c11838c8d0 con 0x55c11839fe40 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f3983fff700 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1185237d0 0x55c11852d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:42.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f3988c65700 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c11852ccc0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f397801d5c0 tx=0x7f397800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7ebe0501e02e69 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f3963fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f3963fff700 1 -- 172.21.15.114:0/2162426249 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3978033060 con 0x55c11839fe40 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f3963fff700 1 -- 172.21.15.114:0/2162426249 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f397801c620 con 0x55c11839fe40 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f39998deb80 1 -- 172.21.15.114:0/2162426249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f397801c8c0 con 0x55c11839fe40 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f3963fff700 1 -- 172.21.15.114:0/2162426249 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f397803a050 con 0x55c11839fe40 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f39998deb80 10 client.16761 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f39998deb80 1 -- 172.21.15.114:0/2162426249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c1183f6760 con 0x55c11839fe40 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f3963fff700 1 -- 172.21.15.114:0/2162426249 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f397801c8c0 con 0x55c11839fe40 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.681+0000 7f3963fff700 1 -- 172.21.15.114:0/2162426249 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f3978003680 con 0x55c11839fe40 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.682+0000 7f39998deb80 10 client.16761 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.682+0000 7f39998deb80 10 client.16761 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.682+0000 7f39998deb80 20 client.16761 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.682+0000 7f39998deb80 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c11853cdf0 0x55c11853f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.682+0000 7f39998deb80 4 client.16761 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:42.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.682+0000 7f39998deb80 1 -- 172.21.15.114:0/2162426249 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c11844ecc0 con 0x55c11853cdf0 2024-05-27T07:45:42.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.682+0000 7f3983fff700 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c11853cdf0 0x55c11853f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.682+0000 7f3963fff700 10 client.16761 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:42.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.682+0000 7f3983fff700 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c11853cdf0 0x55c11853f210 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45835a164f129a5 server_cookie=2c7189eff2686043 in_seq=0 out_seq=0 2024-05-27T07:45:42.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.684+0000 7f3963fff700 1 -- 172.21.15.114:0/2162426249 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c11844ecc0 con 0x55c11853cdf0 2024-05-27T07:45:42.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.684+0000 7f3963fff700 10 client.16761 handle_command_reply: tid=0 2024-05-27T07:45:42.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.684+0000 7f39998deb80 1 client.16761 shutdown 2024-05-27T07:45:42.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.684+0000 7f39998deb80 1 -- 172.21.15.114:0/2162426249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 msgr2=0x55c11852ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:42.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.684+0000 7f39998deb80 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c11852ccc0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f397801d5c0 tx=0x7f397800ed40 comp rx=0 tx=0).stop 2024-05-27T07:45:42.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.684+0000 7f39998deb80 1 -- 172.21.15.114:0/2162426249 shutdown_connections 2024-05-27T07:45:42.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.684+0000 7f39998deb80 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c118527fb0 0x55c11852d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.684+0000 7f39998deb80 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c11852ccc0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.684+0000 7f39998deb80 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c11853cdf0 0x55c11853f210 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.684+0000 7f39998deb80 1 --2- 172.21.15.114:0/2162426249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c1185237d0 0x55c11852d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.684+0000 7f39998deb80 1 -- 172.21.15.114:0/2162426249 >> 172.21.15.114:0/2162426249 conn(0x55c1183a2770 msgr2=0x55c11851dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.684+0000 7f3963fff700 0 client.16761 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.684+0000 7f39998deb80 1 -- 172.21.15.114:0/2162426249 shutdown_connections 2024-05-27T07:45:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.685+0000 7f39998deb80 1 -- 172.21.15.114:0/2162426249 wait complete. 2024-05-27T07:45:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.685+0000 7f39998deb80 20 client.16761 trim_cache size 0 max 16384 2024-05-27T07:45:42.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.707+0000 7f39998deb80 1 Processor -- start 2024-05-27T07:45:42.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.707+0000 7f39998deb80 1 -- start start 2024-05-27T07:45:42.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.707+0000 7f39998deb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c11839fe40 0x55c11852d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.707+0000 7f39998deb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1185237d0 0x55c118539020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.707+0000 7f39998deb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c118527fb0 0x55c118513c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.707+0000 7f39998deb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f39841af400 con 0x55c118527fb0 2024-05-27T07:45:42.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.707+0000 7f39998deb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f39841ac4b0 con 0x55c11839fe40 2024-05-27T07:45:42.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.707+0000 7f39998deb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f39841ae660 con 0x55c1185237d0 2024-05-27T07:45:42.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.707+0000 7f3989466700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c118527fb0 0x55c118513c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.707+0000 7f3988c65700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c11839fe40 0x55c11852d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.707+0000 7f3989466700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c118527fb0 0x55c118513c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52940/0 (socket says 172.21.15.114:52940) 2024-05-27T07:45:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.707+0000 7f3989466700 1 -- 172.21.15.114:0/139608382 learned_addr learned my addr 172.21.15.114:0/139608382 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.708+0000 7f3983fff700 1 --2- 172.21.15.114:0/139608382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1185237d0 0x55c118539020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.708+0000 7f3988c65700 1 -- 172.21.15.114:0/139608382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1185237d0 msgr2=0x55c118539020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.708+0000 7f3988c65700 1 --2- 172.21.15.114:0/139608382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1185237d0 0x55c118539020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.708+0000 7f3988c65700 1 -- 172.21.15.114:0/139608382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c118527fb0 msgr2=0x55c118513c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.708+0000 7f3988c65700 1 --2- 172.21.15.114:0/139608382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c118527fb0 0x55c118513c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.708+0000 7f3988c65700 1 -- 172.21.15.114:0/139608382 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c118515aa0 con 0x55c11839fe40 2024-05-27T07:45:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.708+0000 7f3988c65700 1 --2- 172.21.15.114:0/139608382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c11839fe40 0x55c11852d580 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3978003c40 tx=0x7f3978009370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1eefb9f41d743f90 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.708+0000 7f396e7fc700 1 -- 172.21.15.114:0/139608382 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3978033040 con 0x55c11839fe40 2024-05-27T07:45:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.708+0000 7f396e7fc700 1 -- 172.21.15.114:0/139608382 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f397800edb0 con 0x55c11839fe40 2024-05-27T07:45:42.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.708+0000 7f396e7fc700 1 -- 172.21.15.114:0/139608382 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397801d8a0 con 0x55c11839fe40 2024-05-27T07:45:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.709+0000 7f39998deb80 1 -- 172.21.15.114:0/139608382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c11839fe40 msgr2=0x55c11852d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.709+0000 7f39998deb80 1 --2- 172.21.15.114:0/139608382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c11839fe40 0x55c11852d580 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3978003c40 tx=0x7f3978009370 comp rx=0 tx=0).stop 2024-05-27T07:45:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.709+0000 7f39998deb80 1 -- 172.21.15.114:0/139608382 shutdown_connections 2024-05-27T07:45:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.709+0000 7f39998deb80 1 --2- 172.21.15.114:0/139608382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c118527fb0 0x55c118513c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.709+0000 7f39998deb80 1 --2- 172.21.15.114:0/139608382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c11839fe40 0x55c11852d580 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.709+0000 7f39998deb80 1 --2- 172.21.15.114:0/139608382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1185237d0 0x55c118539020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.709+0000 7f39998deb80 1 -- 172.21.15.114:0/139608382 >> 172.21.15.114:0/139608382 conn(0x55c1183a2770 msgr2=0x55c1185134f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:42.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.709+0000 7f39998deb80 1 -- 172.21.15.114:0/139608382 shutdown_connections 2024-05-27T07:45:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.709+0000 7f39998deb80 1 -- 172.21.15.114:0/139608382 wait complete. 2024-05-27T07:45:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f39998deb80 1 -- start start 2024-05-27T07:45:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f39998deb80 1 Processor -- start 2024-05-27T07:45:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f39998deb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c118535210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f39998deb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1185237d0 0x55c118535750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f39998deb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c118527fb0 0x55c118529390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f39998deb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f39841adf90 con 0x55c11839fe40 2024-05-27T07:45:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f39998deb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f39841acb20 con 0x55c118527fb0 2024-05-27T07:45:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f39998deb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f39841ad1f0 con 0x55c1185237d0 2024-05-27T07:45:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f39998deb80 10 client.0 _ll_register_callbacks cb 0x55c1183d35b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f3988c65700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c118535210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f3988c65700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c118535210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52950/0 (socket says 172.21.15.114:52950) 2024-05-27T07:45:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f3988c65700 1 -- 172.21.15.114:0/1421181876 learned_addr learned my addr 172.21.15.114:0/1421181876 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f3983fff700 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1185237d0 0x55c118535750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.710+0000 7f3989466700 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c118527fb0 0x55c118529390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f3988c65700 1 -- 172.21.15.114:0/1421181876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1185237d0 msgr2=0x55c118535750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f3988c65700 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1185237d0 0x55c118535750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f3988c65700 1 -- 172.21.15.114:0/1421181876 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c118527fb0 msgr2=0x55c118529390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f3988c65700 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c118527fb0 0x55c118529390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f3988c65700 1 -- 172.21.15.114:0/1421181876 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1185298d0 con 0x55c11839fe40 2024-05-27T07:45:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f3983fff700 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1185237d0 0x55c118535750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f3988c65700 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c118535210 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3978003c90 tx=0x7f3978005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de145c209d12270 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f3963fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f3963fff700 1 -- 172.21.15.114:0/1421181876 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3978043020 con 0x55c11839fe40 2024-05-27T07:45:42.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f3963fff700 1 -- 172.21.15.114:0/1421181876 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f397801d5f0 con 0x55c11839fe40 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f39998deb80 1 -- 172.21.15.114:0/1421181876 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f397800edb0 con 0x55c11839fe40 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f3963fff700 1 -- 172.21.15.114:0/1421181876 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f397804e050 con 0x55c11839fe40 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f39998deb80 10 client.16767 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f39998deb80 1 -- 172.21.15.114:0/1421181876 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c1185a6e00 con 0x55c11839fe40 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.711+0000 7f3963fff700 1 -- 172.21.15.114:0/1421181876 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f397800edb0 con 0x55c11839fe40 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.712+0000 7f3963fff700 1 -- 172.21.15.114:0/1421181876 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f397801c860 con 0x55c11839fe40 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.712+0000 7f39998deb80 10 client.16767 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.712+0000 7f39998deb80 10 client.16767 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.712+0000 7f39998deb80 20 client.16767 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.712+0000 7f39998deb80 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c1185a7410 0x55c1185a9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.712+0000 7f39998deb80 4 client.16767 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.712+0000 7f39998deb80 1 -- 172.21.15.114:0/1421181876 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c1184484a0 con 0x55c1185a7410 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.712+0000 7f3983fff700 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c1185a7410 0x55c1185a9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.712+0000 7f3983fff700 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c1185a7410 0x55c1185a9830 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c12eb4d73e0da1cb server_cookie=7e11e7d49ad0cc22 in_seq=0 out_seq=0 2024-05-27T07:45:42.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.712+0000 7f3963fff700 10 client.16767 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.714+0000 7f3963fff700 1 -- 172.21.15.114:0/1421181876 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c1184484a0 con 0x55c1185a7410 2024-05-27T07:45:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.714+0000 7f3963fff700 10 client.16767 handle_command_reply: tid=0 2024-05-27T07:45:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.714+0000 7f39998deb80 1 client.16767 shutdown 2024-05-27T07:45:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f39998deb80 1 -- 172.21.15.114:0/1421181876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 msgr2=0x55c118535210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f39998deb80 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c118535210 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3978003c90 tx=0x7f3978005250 comp rx=0 tx=0).stop 2024-05-27T07:45:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f39998deb80 1 -- 172.21.15.114:0/1421181876 shutdown_connections 2024-05-27T07:45:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f39998deb80 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c1185a7410 0x55c1185a9830 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f39998deb80 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c118527fb0 0x55c118529390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f39998deb80 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c11839fe40 0x55c118535210 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f3963fff700 0 client.16767 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f39998deb80 1 --2- 172.21.15.114:0/1421181876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c1185237d0 0x55c118535750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f39998deb80 1 -- 172.21.15.114:0/1421181876 >> 172.21.15.114:0/1421181876 conn(0x55c1183a2770 msgr2=0x55c11852e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f39998deb80 1 -- 172.21.15.114:0/1421181876 shutdown_connections 2024-05-27T07:45:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f39998deb80 1 -- 172.21.15.114:0/1421181876 wait complete. 2024-05-27T07:45:42.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f39998deb80 20 client.16767 trim_cache size 0 max 16384 2024-05-27T07:45:42.718 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:45:42.718 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:45:42.718 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:45:42.718 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:45:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f398aec9700 1 -- 172.21.15.114:0/304177303 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f39740b29f0 msgr2=0x7f39740b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f398aec9700 1 --2- 172.21.15.114:0/304177303 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f39740b29f0 0x7f39740b4f00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3978009c60 tx=0x7f3978009260 comp rx=0 tx=0).stop 2024-05-27T07:45:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f398aec9700 1 -- 172.21.15.114:0/304177303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3984107de0 msgr2=0x7f398419c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.715+0000 7f398aec9700 1 --2- 172.21.15.114:0/304177303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3984107de0 0x7f398419c160 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f397000d980 tx=0x7f397000dc90 comp rx=0 tx=0).stop 2024-05-27T07:45:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.716+0000 7f398aec9700 1 -- 172.21.15.114:0/304177303 shutdown_connections 2024-05-27T07:45:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.716+0000 7f398aec9700 1 --2- 172.21.15.114:0/304177303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3984108730 0x7f398419c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.716+0000 7f398aec9700 1 --2- 172.21.15.114:0/304177303 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f39740b29f0 0x7f39740b4f00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.716+0000 7f398aec9700 1 --2- 172.21.15.114:0/304177303 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39841034b0 0x7f398419bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.716+0000 7f398aec9700 1 --2- 172.21.15.114:0/304177303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3984107de0 0x7f398419c160 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.716+0000 7f398aec9700 1 -- 172.21.15.114:0/304177303 >> 172.21.15.114:0/304177303 conn(0x7f39840fedb0 msgr2=0x7f3984107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.716+0000 7f398aec9700 1 -- 172.21.15.114:0/304177303 shutdown_connections 2024-05-27T07:45:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:42.716+0000 7f398aec9700 1 -- 172.21.15.114:0/304177303 wait complete. 2024-05-27T07:45:42.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:42 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:45:42.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:42 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:45:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.137+0000 7fe694ebd640 1 -- 172.21.15.114:0/3181309633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe69010e1e0 msgr2=0x7fe6901106d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.137+0000 7fe694ebd640 1 --2- 172.21.15.114:0/3181309633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe69010e1e0 0x7fe6901106d0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fe688009ca0 tx=0x7fe68802a8c0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.138+0000 7fe694ebd640 1 -- 172.21.15.114:0/3181309633 shutdown_connections 2024-05-27T07:45:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.138+0000 7fe694ebd640 1 --2- 172.21.15.114:0/3181309633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe69010e1e0 0x7fe6901106d0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.138+0000 7fe694ebd640 1 --2- 172.21.15.114:0/3181309633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe69010b7f0 0x7fe69010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.138+0000 7fe694ebd640 1 --2- 172.21.15.114:0/3181309633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe690104d90 0x7fe6901051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.138+0000 7fe694ebd640 1 -- 172.21.15.114:0/3181309633 >> 172.21.15.114:0/3181309633 conn(0x7fe6900782d0 msgr2=0x7fe690078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.138+0000 7fe694ebd640 1 -- 172.21.15.114:0/3181309633 shutdown_connections 2024-05-27T07:45:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.138+0000 7fe694ebd640 1 -- 172.21.15.114:0/3181309633 wait complete. 2024-05-27T07:45:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.138+0000 7fe694ebd640 1 Processor -- start 2024-05-27T07:45:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe694ebd640 1 -- start start 2024-05-27T07:45:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe694ebd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe690104d90 0x7fe6901a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe694ebd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe69010b7f0 0x7fe6901a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe694ebd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe69010e1e0 0x7fe69019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe694ebd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe690113e40 con 0x7fe690104d90 2024-05-27T07:45:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe694ebd640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe690113cc0 con 0x7fe69010b7f0 2024-05-27T07:45:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe694ebd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe690113fc0 con 0x7fe69010e1e0 2024-05-27T07:45:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe68e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe690104d90 0x7fe6901a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe68e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe690104d90 0x7fe6901a04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52968/0 (socket says 172.21.15.114:52968) 2024-05-27T07:45:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe68e59c640 1 -- 172.21.15.114:0/848391307 learned_addr learned my addr 172.21.15.114:0/848391307 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe68ed9d640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe69010e1e0 0x7fe69019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe687fff640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe69010b7f0 0x7fe6901a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe68e59c640 1 -- 172.21.15.114:0/848391307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe69010e1e0 msgr2=0x7fe69019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe68e59c640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe69010e1e0 0x7fe69019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe68e59c640 1 -- 172.21.15.114:0/848391307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe69010b7f0 msgr2=0x7fe6901a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe68e59c640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe69010b7f0 0x7fe6901a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.139+0000 7fe68e59c640 1 -- 172.21.15.114:0/848391307 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69019ad90 con 0x7fe690104d90 2024-05-27T07:45:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.140+0000 7fe68e59c640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe690104d90 0x7fe6901a04e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fe68000e9f0 tx=0x7fe68000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.140+0000 7fe6877fe640 1 -- 172.21.15.114:0/848391307 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68000cd80 con 0x7fe690104d90 2024-05-27T07:45:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.140+0000 7fe694ebd640 1 -- 172.21.15.114:0/848391307 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe69019b020 con 0x7fe690104d90 2024-05-27T07:45:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.140+0000 7fe6877fe640 1 -- 172.21.15.114:0/848391307 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6800054d0 con 0x7fe690104d90 2024-05-27T07:45:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.140+0000 7fe6877fe640 1 -- 172.21.15.114:0/848391307 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe680010690 con 0x7fe690104d90 2024-05-27T07:45:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.140+0000 7fe694ebd640 1 -- 172.21.15.114:0/848391307 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6901a7280 con 0x7fe690104d90 2024-05-27T07:45:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.142+0000 7fe6877fe640 1 -- 172.21.15.114:0/848391307 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fe680010830 con 0x7fe690104d90 2024-05-27T07:45:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.143+0000 7fe69bd73740 1 Processor -- start 2024-05-27T07:45:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.143+0000 7fe69bd73740 1 -- start start 2024-05-27T07:45:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.143+0000 7fe69bd73740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9784101b0 0x55b9787b13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.143+0000 7fe69bd73740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783e72f0 0x55b9787b1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.143+0000 7fe69bd73740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.143+0000 7fe69bd73740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b978723c30 con 0x55b9783e72f0 2024-05-27T07:45:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.143+0000 7fe69bd73740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b9787137d0 con 0x55b9783ef4d0 2024-05-27T07:45:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.143+0000 7fe69bd73740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b978717810 con 0x55b9784101b0 2024-05-27T07:45:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.143+0000 7fe68e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783e72f0 0x55b9787b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.143+0000 7fe68ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9784101b0 0x55b9787b13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.143+0000 7fe68ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9784101b0 0x55b9787b13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59012/0 (socket says 172.21.15.114:59012) 2024-05-27T07:45:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe68ed9d640 1 -- 172.21.15.114:0/2379871700 learned_addr learned my addr 172.21.15.114:0/2379871700 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe6877fe640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe67c0b2950 0x7fe67c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe687fff640 1 --2- 172.21.15.114:0/2379871700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe68ed9d640 1 -- 172.21.15.114:0/2379871700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 msgr2=0x55b9787b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe68ed9d640 1 --2- 172.21.15.114:0/2379871700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe68ed9d640 1 -- 172.21.15.114:0/2379871700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783e72f0 msgr2=0x55b9787b1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe68ed9d640 1 --2- 172.21.15.114:0/2379871700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783e72f0 0x55b9787b1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe68ed9d640 1 -- 172.21.15.114:0/2379871700 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b978428210 con 0x55b9784101b0 2024-05-27T07:45:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe68ed9d640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe67c0b2950 0x7fe67c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe6877fe640 1 -- 172.21.15.114:0/848391307 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7fe6800e2e10 con 0x7fe690104d90 2024-05-27T07:45:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe68ed9d640 1 --2- 172.21.15.114:0/2379871700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9784101b0 0x55b9787b13d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe688009480 tx=0x7fe688009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47d15b8ce45eee06 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe684ff9640 1 -- 172.21.15.114:0/2379871700 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe688042040 con 0x55b9784101b0 2024-05-27T07:45:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe68ed9d640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe67c0b2950 0x7fe67c0b4e60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe68804d000 tx=0x7fe68804e000 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe684ff9640 1 -- 172.21.15.114:0/2379871700 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe688036070 con 0x55b9784101b0 2024-05-27T07:45:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.144+0000 7fe684ff9640 1 -- 172.21.15.114:0/2379871700 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe688004810 con 0x55b9784101b0 2024-05-27T07:45:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.145+0000 7fe69bd73740 1 -- 172.21.15.114:0/2379871700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9784101b0 msgr2=0x55b9787b13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.145+0000 7fe69bd73740 1 --2- 172.21.15.114:0/2379871700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9784101b0 0x55b9787b13d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe688009480 tx=0x7fe688009800 comp rx=0 tx=0).stop 2024-05-27T07:45:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.145+0000 7fe69bd73740 1 -- 172.21.15.114:0/2379871700 shutdown_connections 2024-05-27T07:45:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.145+0000 7fe69bd73740 1 --2- 172.21.15.114:0/2379871700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.145+0000 7fe69bd73740 1 --2- 172.21.15.114:0/2379871700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783e72f0 0x55b9787b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.145+0000 7fe69bd73740 1 --2- 172.21.15.114:0/2379871700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9784101b0 0x55b9787b13d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.145+0000 7fe69bd73740 1 -- 172.21.15.114:0/2379871700 >> 172.21.15.114:0/2379871700 conn(0x55b978404300 msgr2=0x55b9787adb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.145+0000 7fe69bd73740 1 -- 172.21.15.114:0/2379871700 shutdown_connections 2024-05-27T07:45:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.145+0000 7fe69bd73740 1 -- 172.21.15.114:0/2379871700 wait complete. 2024-05-27T07:45:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe69bd73740 1 -- start start 2024-05-27T07:45:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe69bd73740 1 Processor -- start 2024-05-27T07:45:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe69bd73740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783e72f0 0x55b9787bd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe69bd73740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787bd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe69bd73740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9784101b0 0x55b9787bb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe69bd73740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b9783f9790 con 0x55b9783e72f0 2024-05-27T07:45:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe69bd73740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b9786cf890 con 0x55b9783ef4d0 2024-05-27T07:45:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe69bd73740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b9787be500 con 0x55b9784101b0 2024-05-27T07:45:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe69bd73740 10 client.0 _ll_register_callbacks cb 0x55b978427a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe68e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783e72f0 0x55b9787bd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe68ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9784101b0 0x55b9787bb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe687fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787bd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe68e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783e72f0 0x55b9787bd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52978/0 (socket says 172.21.15.114:52978) 2024-05-27T07:45:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.146+0000 7fe68e59c640 1 -- 172.21.15.114:0/1594001664 learned_addr learned my addr 172.21.15.114:0/1594001664 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe687fff640 1 -- 172.21.15.114:0/1594001664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9784101b0 msgr2=0x55b9787bb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe687fff640 1 --2- 172.21.15.114:0/1594001664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9784101b0 0x55b9787bb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe687fff640 1 -- 172.21.15.114:0/1594001664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783e72f0 msgr2=0x55b9787bd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe687fff640 1 --2- 172.21.15.114:0/1594001664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783e72f0 0x55b9787bd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe687fff640 1 -- 172.21.15.114:0/1594001664 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9784f4620 con 0x55b9783ef4d0 2024-05-27T07:45:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe68e59c640 1 --2- 172.21.15.114:0/1594001664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783e72f0 0x55b9787bd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe687fff640 1 --2- 172.21.15.114:0/1594001664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787bd990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe69019be90 tx=0x7fe678012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61013487283befc5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe675ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:45:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe675ffb640 1 -- 172.21.15.114:0/1594001664 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe678004030 con 0x55b9783ef4d0 2024-05-27T07:45:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe675ffb640 1 -- 172.21.15.114:0/1594001664 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7fe6780041d0 con 0x55b9783ef4d0 2024-05-27T07:45:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe69bd73740 1 -- 172.21.15.114:0/1594001664 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe688036070 con 0x55b9783ef4d0 2024-05-27T07:45:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe675ffb640 1 -- 172.21.15.114:0/1594001664 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe6780043d0 con 0x55b9783ef4d0 2024-05-27T07:45:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe69bd73740 10 client.26273 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe69bd73740 1 -- 172.21.15.114:0/1594001664 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b9787ca8c0 con 0x55b9783ef4d0 2024-05-27T07:45:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.147+0000 7fe675ffb640 1 -- 172.21.15.114:0/1594001664 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe688036070 con 0x55b9783ef4d0 2024-05-27T07:45:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.148+0000 7fe675ffb640 1 -- 172.21.15.114:0/1594001664 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7fe6780039b0 con 0x55b9783ef4d0 2024-05-27T07:45:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.148+0000 7fe69bd73740 10 client.26273 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.148+0000 7fe69bd73740 10 client.26273 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.148+0000 7fe69bd73740 20 client.26273 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.148+0000 7fe69bd73740 1 --2- 172.21.15.114:0/1594001664 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9783ced80 0x55b9787cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.148+0000 7fe69bd73740 4 client.26273 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.148+0000 7fe69bd73740 1 -- 172.21.15.114:0/1594001664 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b978725b80 con 0x55b9783ced80 2024-05-27T07:45:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.148+0000 7fe68e59c640 1 --2- 172.21.15.114:0/1594001664 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9783ced80 0x55b9787cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.148+0000 7fe675ffb640 10 client.26273 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.149+0000 7fe68e59c640 1 --2- 172.21.15.114:0/1594001664 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9783ced80 0x55b9787cde10 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa820524f23b3eca server_cookie=4f412562bef78ddd in_seq=0 out_seq=0 2024-05-27T07:45:43.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe675ffb640 1 -- 172.21.15.114:0/1594001664 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b978725b80 con 0x55b9783ced80 2024-05-27T07:45:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe675ffb640 10 client.26273 handle_command_reply: tid=0 2024-05-27T07:45:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe69bd73740 1 client.26273 shutdown 2024-05-27T07:45:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe69bd73740 1 -- 172.21.15.114:0/1594001664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 msgr2=0x55b9787bd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe69bd73740 1 --2- 172.21.15.114:0/1594001664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787bd990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe69019be90 tx=0x7fe678012b60 comp rx=0 tx=0).stop 2024-05-27T07:45:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe69bd73740 1 -- 172.21.15.114:0/1594001664 shutdown_connections 2024-05-27T07:45:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe69bd73740 1 --2- 172.21.15.114:0/1594001664 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9783ced80 0x55b9787cde10 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe69bd73740 1 --2- 172.21.15.114:0/1594001664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9784101b0 0x55b9787bb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe675ffb640 0 client.26273 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe69bd73740 1 --2- 172.21.15.114:0/1594001664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787bd990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe69bd73740 1 --2- 172.21.15.114:0/1594001664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783e72f0 0x55b9787bd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe69bd73740 1 -- 172.21.15.114:0/1594001664 >> 172.21.15.114:0/1594001664 conn(0x55b978404300 msgr2=0x55b9787b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe69bd73740 1 -- 172.21.15.114:0/1594001664 shutdown_connections 2024-05-27T07:45:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.151+0000 7fe69bd73740 1 -- 172.21.15.114:0/1594001664 wait complete. 2024-05-27T07:45:43.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.152+0000 7fe69bd73740 20 client.26273 trim_cache size 0 max 16384 2024-05-27T07:45:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.175+0000 7fe69bd73740 1 Processor -- start 2024-05-27T07:45:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.175+0000 7fe69bd73740 1 -- start start 2024-05-27T07:45:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.175+0000 7fe69bd73740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783ced80 0x55b9787b5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.175+0000 7fe69bd73740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9783e72f0 0x55b9787bb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.176+0000 7fe69bd73740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787bbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.176+0000 7fe687fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9783e72f0 0x55b9787bb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.176+0000 7fe687fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9783e72f0 0x55b9787bb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59024/0 (socket says 172.21.15.114:59024) 2024-05-27T07:45:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.176+0000 7fe687fff640 1 -- 172.21.15.114:0/799953051 learned_addr learned my addr 172.21.15.114:0/799953051 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.176+0000 7fe68ed9d640 1 --2- 172.21.15.114:0/799953051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787bbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.176+0000 7fe69bd73740 1 -- 172.21.15.114:0/799953051 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6901b2160 con 0x55b9783ced80 2024-05-27T07:45:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.176+0000 7fe69bd73740 1 -- 172.21.15.114:0/799953051 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe6901af210 con 0x55b9783ef4d0 2024-05-27T07:45:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.176+0000 7fe69bd73740 1 -- 172.21.15.114:0/799953051 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe6901b13c0 con 0x55b9783e72f0 2024-05-27T07:45:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.178+0000 7fe68e59c640 1 --2- 172.21.15.114:0/799953051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783ced80 0x55b9787b5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.178+0000 7fe687fff640 1 -- 172.21.15.114:0/799953051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 msgr2=0x55b9787bbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.178+0000 7fe687fff640 1 --2- 172.21.15.114:0/799953051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787bbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.178+0000 7fe687fff640 1 -- 172.21.15.114:0/799953051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783ced80 msgr2=0x55b9787b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.178+0000 7fe687fff640 1 --2- 172.21.15.114:0/799953051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783ced80 0x55b9787b5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.178+0000 7fe687fff640 1 -- 172.21.15.114:0/799953051 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9787b9e40 con 0x55b9783e72f0 2024-05-27T07:45:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.179+0000 7fe687fff640 1 --2- 172.21.15.114:0/799953051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9783e72f0 0x55b9787bb8b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe678001eb0 tx=0x7fe67800f810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea2497bee83e5c85 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.179+0000 7fe68e59c640 1 --2- 172.21.15.114:0/799953051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783ced80 0x55b9787b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:45:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.180+0000 7fe677fff640 1 -- 172.21.15.114:0/799953051 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67802e070 con 0x55b9783e72f0 2024-05-27T07:45:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.180+0000 7fe677fff640 1 -- 172.21.15.114:0/799953051 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe67800fdd0 con 0x55b9783e72f0 2024-05-27T07:45:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.180+0000 7fe677fff640 1 -- 172.21.15.114:0/799953051 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67801cae0 con 0x55b9783e72f0 2024-05-27T07:45:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.181+0000 7fe69bd73740 1 -- 172.21.15.114:0/799953051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9783e72f0 msgr2=0x55b9787bb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.181+0000 7fe69bd73740 1 --2- 172.21.15.114:0/799953051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9783e72f0 0x55b9787bb8b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe678001eb0 tx=0x7fe67800f810 comp rx=0 tx=0).stop 2024-05-27T07:45:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.181+0000 7fe69bd73740 1 -- 172.21.15.114:0/799953051 shutdown_connections 2024-05-27T07:45:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.181+0000 7fe69bd73740 1 --2- 172.21.15.114:0/799953051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787bbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.181+0000 7fe69bd73740 1 --2- 172.21.15.114:0/799953051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9783e72f0 0x55b9787bb8b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.181+0000 7fe69bd73740 1 --2- 172.21.15.114:0/799953051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783ced80 0x55b9787b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.181+0000 7fe69bd73740 1 -- 172.21.15.114:0/799953051 >> 172.21.15.114:0/799953051 conn(0x55b978404300 msgr2=0x55b9787b5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.181+0000 7fe69bd73740 1 -- 172.21.15.114:0/799953051 shutdown_connections 2024-05-27T07:45:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.181+0000 7fe69bd73740 1 -- 172.21.15.114:0/799953051 wait complete. 2024-05-27T07:45:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.182+0000 7fe69bd73740 1 -- start start 2024-05-27T07:45:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.182+0000 7fe69bd73740 1 Processor -- start 2024-05-27T07:45:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.182+0000 7fe69bd73740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783ced80 0x55b9787c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.183+0000 7fe69bd73740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9783e72f0 0x55b9787c2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.183+0000 7fe69bd73740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787c2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.183+0000 7fe68e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783ced80 0x55b9787c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.183+0000 7fe69bd73740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6901b0cf0 con 0x55b9783ced80 2024-05-27T07:45:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.183+0000 7fe69bd73740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe6901af880 con 0x55b9783ef4d0 2024-05-27T07:45:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.183+0000 7fe68e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783ced80 0x55b9787c3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52988/0 (socket says 172.21.15.114:52988) 2024-05-27T07:45:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.183+0000 7fe68e59c640 1 -- 172.21.15.114:0/88706007 learned_addr learned my addr 172.21.15.114:0/88706007 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.183+0000 7fe69bd73740 1 -- 172.21.15.114:0/88706007 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe6901aff50 con 0x55b9783e72f0 2024-05-27T07:45:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.183+0000 7fe69bd73740 10 client.0 _ll_register_callbacks cb 0x7fe678009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.183+0000 7fe687fff640 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9783e72f0 0x55b9787c2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.184+0000 7fe68e59c640 1 -- 172.21.15.114:0/88706007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9783e72f0 msgr2=0x55b9787c2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.184+0000 7fe68e59c640 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9783e72f0 0x55b9787c2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.184+0000 7fe68e59c640 1 -- 172.21.15.114:0/88706007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 msgr2=0x55b9787c2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.184+0000 7fe68ed9d640 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787c2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.184+0000 7fe68e59c640 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787c2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.184+0000 7fe68e59c640 1 -- 172.21.15.114:0/88706007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9787c33a0 con 0x55b9783ced80 2024-05-27T07:45:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.184+0000 7fe68ed9d640 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787c2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:45:43.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.187+0000 7fe68e59c640 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783ced80 0x55b9787c3ff0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fe6800e07b0 tx=0x7fe6800a64c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af43ba4ad1eafba6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:43.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.188+0000 7fe675ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:43.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.189+0000 7fe675ffb640 1 -- 172.21.15.114:0/88706007 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6800a5c00 con 0x55b9783ced80 2024-05-27T07:45:43.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.189+0000 7fe675ffb640 1 -- 172.21.15.114:0/88706007 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7fe6800a5fd0 con 0x55b9783ced80 2024-05-27T07:45:43.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.189+0000 7fe69bd73740 1 -- 172.21.15.114:0/88706007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe67800fdd0 con 0x55b9783ced80 2024-05-27T07:45:43.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.190+0000 7fe675ffb640 1 -- 172.21.15.114:0/88706007 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe6800a9850 con 0x55b9783ced80 2024-05-27T07:45:43.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.190+0000 7fe69bd73740 10 client.16785 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:43.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.191+0000 7fe69bd73740 1 -- 172.21.15.114:0/88706007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b97883e140 con 0x55b9783ced80 2024-05-27T07:45:43.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.191+0000 7fe675ffb640 1 -- 172.21.15.114:0/88706007 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe680005680 con 0x55b9783ced80 2024-05-27T07:45:43.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.191+0000 7fe675ffb640 1 -- 172.21.15.114:0/88706007 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7fe6800b2710 con 0x55b9783ced80 2024-05-27T07:45:43.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.191+0000 7fe69bd73740 10 client.16785 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:43.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.191+0000 7fe69bd73740 10 client.16785 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:43.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.191+0000 7fe69bd73740 20 client.16785 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:43.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.191+0000 7fe69bd73740 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9784101b0 0x55b9788408b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.192+0000 7fe69bd73740 4 client.16785 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:45:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.192+0000 7fe69bd73740 1 -- 172.21.15.114:0/88706007 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fe67800fdd0 con 0x55b9784101b0 2024-05-27T07:45:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.192+0000 7fe687fff640 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9784101b0 0x55b9788408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.193+0000 7fe687fff640 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9784101b0 0x55b9788408b0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb9cc3edd397309d server_cookie=d545b8efd3cdb0a1 in_seq=0 out_seq=0 2024-05-27T07:45:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.193+0000 7fe675ffb640 10 client.16785 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:43.249 INFO:tasks.workunit.client.0.smithi114.stdout: 1 586162 13.12 MB/sec execute 115 sec latency 29.832 ms 2024-05-27T07:45:43.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.503+0000 7fe675ffb640 1 -- 172.21.15.114:0/88706007 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7fe67800fdd0 con 0x55b9784101b0 2024-05-27T07:45:43.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.503+0000 7fe675ffb640 10 client.16785 handle_command_reply: tid=0 2024-05-27T07:45:43.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.503+0000 7fe69bd73740 1 client.16785 shutdown 2024-05-27T07:45:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.503+0000 7fe69bd73740 1 -- 172.21.15.114:0/88706007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783ced80 msgr2=0x55b9787c3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.503+0000 7fe69bd73740 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783ced80 0x55b9787c3ff0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fe6800e07b0 tx=0x7fe6800a64c0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.503+0000 7fe69bd73740 1 -- 172.21.15.114:0/88706007 shutdown_connections 2024-05-27T07:45:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.503+0000 7fe69bd73740 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b9784101b0 0x55b9788408b0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.503+0000 7fe69bd73740 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9783ef4d0 0x55b9787c2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.503+0000 7fe69bd73740 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9783e72f0 0x55b9787c2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.503+0000 7fe69bd73740 1 --2- 172.21.15.114:0/88706007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b9783ced80 0x55b9787c3ff0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.503+0000 7fe675ffb640 0 client.16785 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.503+0000 7fe69bd73740 1 -- 172.21.15.114:0/88706007 >> 172.21.15.114:0/88706007 conn(0x55b978404300 msgr2=0x55b9787b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.504+0000 7fe69bd73740 1 -- 172.21.15.114:0/88706007 shutdown_connections 2024-05-27T07:45:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.504+0000 7fe69bd73740 1 -- 172.21.15.114:0/88706007 wait complete. 2024-05-27T07:45:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.504+0000 7fe69bd73740 20 client.16785 trim_cache size 0 max 16384 2024-05-27T07:45:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:45:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:45:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:45:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.505+0000 7fe694ebd640 1 -- 172.21.15.114:0/848391307 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe67c0b2950 msgr2=0x7fe67c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.505+0000 7fe694ebd640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe67c0b2950 0x7fe67c0b4e60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe68804d000 tx=0x7fe68804e000 comp rx=0 tx=0).stop 2024-05-27T07:45:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.505+0000 7fe694ebd640 1 -- 172.21.15.114:0/848391307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe690104d90 msgr2=0x7fe6901a04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.505+0000 7fe694ebd640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe690104d90 0x7fe6901a04e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fe68000e9f0 tx=0x7fe68000edf0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.505+0000 7fe694ebd640 1 -- 172.21.15.114:0/848391307 shutdown_connections 2024-05-27T07:45:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.505+0000 7fe694ebd640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe67c0b2950 0x7fe67c0b4e60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.505+0000 7fe694ebd640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe69010e1e0 0x7fe69019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.505+0000 7fe694ebd640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe69010b7f0 0x7fe6901a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.505+0000 7fe694ebd640 1 --2- 172.21.15.114:0/848391307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe690104d90 0x7fe6901a04e0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.506+0000 7fe694ebd640 1 -- 172.21.15.114:0/848391307 >> 172.21.15.114:0/848391307 conn(0x7fe6900782d0 msgr2=0x7fe6901012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.506+0000 7fe694ebd640 1 -- 172.21.15.114:0/848391307 shutdown_connections 2024-05-27T07:45:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:43.506+0000 7fe694ebd640 1 -- 172.21.15.114:0/848391307 wait complete. 2024-05-27T07:45:43.568 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:45:43.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:45:43.724 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:43 smithi163 ceph-mon[36451]: pgmap v189: 129 pgs: 129 active+clean; 76 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 268 op/s 2024-05-27T07:45:43.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:43 smithi186 ceph-mon[36647]: pgmap v189: 129 pgs: 129 active+clean; 76 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 268 op/s 2024-05-27T07:45:43.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:43 smithi114 ceph-mon[32038]: pgmap v189: 129 pgs: 129 active+clean; 76 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 268 op/s 2024-05-27T07:45:43.814 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:44.249 INFO:tasks.workunit.client.0.smithi114.stdout: 1 587323 13.04 MB/sec execute 116 sec latency 92.100 ms 2024-05-27T07:45:45.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 590028 13.07 MB/sec execute 117 sec latency 89.264 ms 2024-05-27T07:45:45.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:45 smithi114 ceph-mon[32038]: pgmap v190: 129 pgs: 129 active+clean; 76 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-05-27T07:45:45.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:45 smithi163 ceph-mon[36451]: pgmap v190: 129 pgs: 129 active+clean; 76 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-05-27T07:45:45.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:45 smithi186 ceph-mon[36647]: pgmap v190: 129 pgs: 129 active+clean; 76 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-05-27T07:45:45.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.897+0000 7f599fe0c700 1 -- 172.21.15.114:0/3552659165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5998104600 msgr2=0x7f5998104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:45.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.897+0000 7f599fe0c700 1 --2- 172.21.15.114:0/3552659165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5998104600 0x7f5998104a10 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f59940099c0 tx=0x7f5994009cd0 comp rx=0 tx=0).stop 2024-05-27T07:45:45.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.897+0000 7f599fe0c700 1 -- 172.21.15.114:0/3552659165 shutdown_connections 2024-05-27T07:45:45.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.897+0000 7f599fe0c700 1 --2- 172.21.15.114:0/3552659165 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5998109880 0x7f59981111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:45.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.897+0000 7f599fe0c700 1 --2- 172.21.15.114:0/3552659165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5998104600 0x7f5998104a10 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:45.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.897+0000 7f599fe0c700 1 --2- 172.21.15.114:0/3552659165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5998108f30 0x7f5998109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:45.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.897+0000 7f599fe0c700 1 -- 172.21.15.114:0/3552659165 >> 172.21.15.114:0/3552659165 conn(0x7f599807af60 msgr2=0x7f599807b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:45.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.897+0000 7f599fe0c700 1 -- 172.21.15.114:0/3552659165 shutdown_connections 2024-05-27T07:45:45.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.897+0000 7f599fe0c700 1 -- 172.21.15.114:0/3552659165 wait complete. 2024-05-27T07:45:45.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599fe0c700 1 Processor -- start 2024-05-27T07:45:45.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599fe0c700 1 -- start start 2024-05-27T07:45:45.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599fe0c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5998104600 0x7f599819aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:45.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599fe0c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5998108f30 0x7f599819b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:45.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599fe0c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5998109880 0x7f599819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:45.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599fe0c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f59981144d0 con 0x7f5998109880 2024-05-27T07:45:45.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599fe0c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5998114350 con 0x7f5998108f30 2024-05-27T07:45:45.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599fe0c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5998114650 con 0x7f5998104600 2024-05-27T07:45:45.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599dba8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5998104600 0x7f599819aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:45.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599d3a7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5998108f30 0x7f599819b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599e3a9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5998109880 0x7f599819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599dba8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5998104600 0x7f599819aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:48412/0 (socket says 172.21.15.114:48412) 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599d3a7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5998108f30 0x7f599819b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:40698/0 (socket says 172.21.15.114:40698) 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599dba8700 1 -- 172.21.15.114:0/3666407791 learned_addr learned my addr 172.21.15.114:0/3666407791 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599d3a7700 1 -- 172.21.15.114:0/3666407791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5998104600 msgr2=0x7f599819aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.898+0000 7f599d3a7700 1 --2- 172.21.15.114:0/3666407791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5998104600 0x7f599819aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.899+0000 7f599d3a7700 1 -- 172.21.15.114:0/3666407791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5998109880 msgr2=0x7f599819f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.899+0000 7f599d3a7700 1 --2- 172.21.15.114:0/3666407791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5998109880 0x7f599819f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.899+0000 7f599d3a7700 1 -- 172.21.15.114:0/3666407791 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f599819fd90 con 0x7f5998108f30 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.899+0000 7f599d3a7700 1 --2- 172.21.15.114:0/3666407791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5998108f30 0x7f599819b460 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f598800e9c0 tx=0x7f598800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.899+0000 7f598effd700 1 -- 172.21.15.114:0/3666407791 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5988004220 con 0x7f5998108f30 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.899+0000 7f598effd700 1 -- 172.21.15.114:0/3666407791 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59880043c0 con 0x7f5998108f30 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.899+0000 7f599fe0c700 1 -- 172.21.15.114:0/3666407791 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59981a0080 con 0x7f5998108f30 2024-05-27T07:45:45.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.899+0000 7f598effd700 1 -- 172.21.15.114:0/3666407791 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598800c740 con 0x7f5998108f30 2024-05-27T07:45:45.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.899+0000 7f599fe0c700 1 -- 172.21.15.114:0/3666407791 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59980759e0 con 0x7f5998108f30 2024-05-27T07:45:45.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.900+0000 7f599fe0c700 1 -- 172.21.15.114:0/3666407791 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5998105ab0 con 0x7f5998108f30 2024-05-27T07:45:45.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.902+0000 7f598effd700 1 -- 172.21.15.114:0/3666407791 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f5988004750 con 0x7f5998108f30 2024-05-27T07:45:45.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.902+0000 7f598effd700 1 --2- 172.21.15.114:0/3666407791 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f59840b29f0 0x7f59840b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:45.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.903+0000 7f599dba8700 1 --2- 172.21.15.114:0/3666407791 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f59840b29f0 0x7f59840b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:45.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.903+0000 7f598effd700 1 -- 172.21.15.114:0/3666407791 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f59880e29f0 con 0x7f5998108f30 2024-05-27T07:45:45.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.903+0000 7f599dba8700 1 --2- 172.21.15.114:0/3666407791 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f59840b29f0 0x7f59840b4f00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5994003760 tx=0x7f599401a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:45.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:45.906+0000 7f598effd700 1 -- 172.21.15.114:0/3666407791 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f59880aa050 con 0x7f5998108f30 2024-05-27T07:45:46.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.201+0000 7f599fe0c700 1 -- 172.21.15.114:0/3666407791 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f599819c220 con 0x7f5998108f30 2024-05-27T07:45:46.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.204+0000 7f598effd700 1 -- 172.21.15.114:0/3666407791 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11000 (secure 0 0 0) 0x7f59880a9950 con 0x7f5998108f30 2024-05-27T07:45:46.204 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:45:46.204 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":26227,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2936519309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2936519309},{"type":"v1","addr":"172.21.15.186:6835","nonce":2936519309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:45:32.379189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26305":{"gid":26305,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/684229511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":684229511},{"type":"v1","addr":"172.21.15.186:6837","nonce":684229511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:45:46.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.209+0000 7f599fe0c700 1 -- 172.21.15.114:0/3666407791 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f59840b29f0 msgr2=0x7f59840b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:46.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.209+0000 7f599fe0c700 1 --2- 172.21.15.114:0/3666407791 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f59840b29f0 0x7f59840b4f00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5994003760 tx=0x7f599401a040 comp rx=0 tx=0).stop 2024-05-27T07:45:46.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.209+0000 7f599fe0c700 1 -- 172.21.15.114:0/3666407791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5998108f30 msgr2=0x7f599819b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:46.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.209+0000 7f599fe0c700 1 --2- 172.21.15.114:0/3666407791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5998108f30 0x7f599819b460 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f598800e9c0 tx=0x7f598800ecd0 comp rx=0 tx=0).stop 2024-05-27T07:45:46.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.209+0000 7f599fe0c700 1 -- 172.21.15.114:0/3666407791 shutdown_connections 2024-05-27T07:45:46.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.210+0000 7f599fe0c700 1 --2- 172.21.15.114:0/3666407791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5998109880 0x7f599819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:46.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.210+0000 7f599fe0c700 1 --2- 172.21.15.114:0/3666407791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5998104600 0x7f599819aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:46.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.210+0000 7f599fe0c700 1 --2- 172.21.15.114:0/3666407791 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f59840b29f0 0x7f59840b4f00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:46.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.210+0000 7f599fe0c700 1 --2- 172.21.15.114:0/3666407791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5998108f30 0x7f599819b460 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:46.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.210+0000 7f599fe0c700 1 -- 172.21.15.114:0/3666407791 >> 172.21.15.114:0/3666407791 conn(0x7f599807af60 msgr2=0x7f5998102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:46.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.210+0000 7f599fe0c700 1 -- 172.21.15.114:0/3666407791 shutdown_connections 2024-05-27T07:45:46.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:46.210+0000 7f599fe0c700 1 -- 172.21.15.114:0/3666407791 wait complete. 2024-05-27T07:45:46.211 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 37 2024-05-27T07:45:46.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 592618 13.07 MB/sec execute 118 sec latency 29.805 ms 2024-05-27T07:45:46.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:46 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3666407791' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:45:46.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:46 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3666407791' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:45:46.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:46 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3666407791' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:45:47.187 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:45:47.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 594723 13.06 MB/sec execute 119 sec latency 36.162 ms 2024-05-27T07:45:47.466 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:47.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:47 smithi114 ceph-mon[32038]: pgmap v191: 129 pgs: 129 active+clean; 74 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 467 op/s 2024-05-27T07:45:47.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:47 smithi163 ceph-mon[36451]: pgmap v191: 129 pgs: 129 active+clean; 74 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 467 op/s 2024-05-27T07:45:47.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:47 smithi186 ceph-mon[36647]: pgmap v191: 129 pgs: 129 active+clean; 74 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 467 op/s 2024-05-27T07:45:48.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 597273 13.07 MB/sec execute 120 sec latency 17.095 ms 2024-05-27T07:45:49.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 600263 13.10 MB/sec execute 121 sec latency 33.089 ms 2024-05-27T07:45:49.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.366+0000 7f411bafa700 1 -- 172.21.15.114:0/2888844926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411410a960 msgr2=0x7f411410ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:49.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.366+0000 7f411bafa700 1 --2- 172.21.15.114:0/2888844926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411410a960 0x7f411410ad70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4104009990 tx=0x7f4104009ca0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.367+0000 7f411bafa700 1 -- 172.21.15.114:0/2888844926 shutdown_connections 2024-05-27T07:45:49.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.367+0000 7f411bafa700 1 --2- 172.21.15.114:0/2888844926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411410b2b0 0x7f4114117830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.367+0000 7f411bafa700 1 --2- 172.21.15.114:0/2888844926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4114110990 0x7f4114110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.367+0000 7f411bafa700 1 --2- 172.21.15.114:0/2888844926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411410a960 0x7f411410ad70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.367+0000 7f411bafa700 1 -- 172.21.15.114:0/2888844926 >> 172.21.15.114:0/2888844926 conn(0x7f4114076cb0 msgr2=0x7f41140770c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:49.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.367+0000 7f411bafa700 1 -- 172.21.15.114:0/2888844926 shutdown_connections 2024-05-27T07:45:49.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.367+0000 7f411bafa700 1 -- 172.21.15.114:0/2888844926 wait complete. 2024-05-27T07:45:49.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.368+0000 7f411bafa700 1 Processor -- start 2024-05-27T07:45:49.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.368+0000 7f411bafa700 1 -- start start 2024-05-27T07:45:49.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.368+0000 7f411bafa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411410a960 0x7f41141a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.368+0000 7f411bafa700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411410b2b0 0x7f41141a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.369+0000 7f411bafa700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4114110990 0x7f41141a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.369+0000 7f411bafa700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f411411aab0 con 0x7f411410a960 2024-05-27T07:45:49.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.369+0000 7f411bafa700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f411411a930 con 0x7f4114110990 2024-05-27T07:45:49.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.369+0000 7f411bafa700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f411411ac30 con 0x7f411410b2b0 2024-05-27T07:45:49.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.369+0000 7f4119896700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411410a960 0x7f41141a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.369+0000 7f4119896700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411410a960 0x7f41141a1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60976/0 (socket says 172.21.15.114:60976) 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.369+0000 7f4119896700 1 -- 172.21.15.114:0/3849320784 learned_addr learned my addr 172.21.15.114:0/3849320784 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.369+0000 7f4119095700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411410b2b0 0x7f41141a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.369+0000 7f411a097700 1 --2- 172.21.15.114:0/3849320784 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4114110990 0x7f41141a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.370+0000 7f4119896700 1 -- 172.21.15.114:0/3849320784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411410b2b0 msgr2=0x7f41141a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.370+0000 7f4119896700 1 --2- 172.21.15.114:0/3849320784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411410b2b0 0x7f41141a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.370+0000 7f4119896700 1 -- 172.21.15.114:0/3849320784 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4114110990 msgr2=0x7f41141a5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.370+0000 7f4119896700 1 --2- 172.21.15.114:0/3849320784 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4114110990 0x7f41141a5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.370+0000 7f4119896700 1 -- 172.21.15.114:0/3849320784 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41141a6310 con 0x7f411410a960 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.370+0000 7f4119896700 1 --2- 172.21.15.114:0/3849320784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411410a960 0x7f41141a1440 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f411000d950 tx=0x7f411000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.371+0000 7f410affd700 1 -- 172.21.15.114:0/3849320784 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110016070 con 0x7f411410a960 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.371+0000 7f410affd700 1 -- 172.21.15.114:0/3849320784 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4110004830 con 0x7f411410a960 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.371+0000 7f410affd700 1 -- 172.21.15.114:0/3849320784 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110005440 con 0x7f411410a960 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.371+0000 7f411bafa700 1 -- 172.21.15.114:0/3849320784 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41141a6600 con 0x7f411410a960 2024-05-27T07:45:49.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.371+0000 7f411bafa700 1 -- 172.21.15.114:0/3849320784 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41141adf20 con 0x7f411410a960 2024-05-27T07:45:49.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.375+0000 7f410affd700 1 -- 172.21.15.114:0/3849320784 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f411000b750 con 0x7f411410a960 2024-05-27T07:45:49.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.376+0000 7f412a50fb80 1 Processor -- start 2024-05-27T07:45:49.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.376+0000 7f412a50fb80 1 -- start start 2024-05-27T07:45:49.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.376+0000 7f410affd700 1 --2- 172.21.15.114:0/3849320784 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f41000b2b40 0x7f41000b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f412a50fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x5608906f8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f412a50fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f9070 0x5608906f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f412a50fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f3120 0x5608906f3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f412a50fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560890620d70 con 0x5608906f9070 2024-05-27T07:45:49.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f412a50fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560890610910 con 0x5608906f3120 2024-05-27T07:45:49.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f412a50fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560890614950 con 0x560890573e40 2024-05-27T07:45:49.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f411a097700 1 --2- 172.21.15.114:0/3849320784 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f41000b2b40 0x7f41000b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f410affd700 1 -- 172.21.15.114:0/3849320784 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f41100e2630 con 0x7f411410a960 2024-05-27T07:45:49.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f4119095700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f9070 0x5608906f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f4119095700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f9070 0x5608906f2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60984/0 (socket says 172.21.15.114:60984) 2024-05-27T07:45:49.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f4119095700 1 -- 172.21.15.114:0/1126822879 learned_addr learned my addr 172.21.15.114:0/1126822879 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:49.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f4119896700 1 --2- 172.21.15.114:0/1126822879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x5608906f8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f411a097700 1 --2- 172.21.15.114:0/1126822879 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f3120 0x5608906f3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f4119095700 1 -- 172.21.15.114:0/1126822879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 msgr2=0x5608906f8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:49.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f4119095700 1 --2- 172.21.15.114:0/1126822879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x5608906f8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f4119095700 1 -- 172.21.15.114:0/1126822879 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f3120 msgr2=0x5608906f3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:49.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f4119095700 1 --2- 172.21.15.114:0/1126822879 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f3120 0x5608906f3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f4119095700 1 -- 172.21.15.114:0/1126822879 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608905b9fb0 con 0x5608906f9070 2024-05-27T07:45:49.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f4119896700 1 --2- 172.21.15.114:0/1126822879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x5608906f8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:49.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f411a097700 1 --2- 172.21.15.114:0/3849320784 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f41000b2b40 0x7f41000b5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f410c007b60 tx=0x7f410c0076b0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:49.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.377+0000 7f4119095700 1 --2- 172.21.15.114:0/1126822879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f9070 0x5608906f2be0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f4104009990 tx=0x7f4104005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3110e5d754806c6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:49.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.378+0000 7f40fffff700 1 -- 172.21.15.114:0/1126822879 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410400b720 con 0x5608906f9070 2024-05-27T07:45:49.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.378+0000 7f40fffff700 1 -- 172.21.15.114:0/1126822879 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f410400bd70 con 0x5608906f9070 2024-05-27T07:45:49.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.378+0000 7f40fffff700 1 -- 172.21.15.114:0/1126822879 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104003d70 con 0x5608906f9070 2024-05-27T07:45:49.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.379+0000 7f412a50fb80 1 -- 172.21.15.114:0/1126822879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f9070 msgr2=0x5608906f2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:49.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.379+0000 7f412a50fb80 1 --2- 172.21.15.114:0/1126822879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f9070 0x5608906f2be0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f4104009990 tx=0x7f4104005ea0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.379+0000 7f412a50fb80 1 -- 172.21.15.114:0/1126822879 shutdown_connections 2024-05-27T07:45:49.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.379+0000 7f412a50fb80 1 --2- 172.21.15.114:0/1126822879 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f3120 0x5608906f3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.379+0000 7f412a50fb80 1 --2- 172.21.15.114:0/1126822879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x5608906f8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.379+0000 7f412a50fb80 1 --2- 172.21.15.114:0/1126822879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f9070 0x5608906f2be0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.379+0000 7f412a50fb80 1 -- 172.21.15.114:0/1126822879 >> 172.21.15.114:0/1126822879 conn(0x560890576770 msgr2=0x5608906edd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:49.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.379+0000 7f412a50fb80 1 -- 172.21.15.114:0/1126822879 shutdown_connections 2024-05-27T07:45:49.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.380+0000 7f412a50fb80 1 -- 172.21.15.114:0/1126822879 wait complete. 2024-05-27T07:45:49.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.380+0000 7f412a50fb80 1 -- start start 2024-05-27T07:45:49.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.380+0000 7f412a50fb80 1 Processor -- start 2024-05-27T07:45:49.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f412a50fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x5608906fc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f412a50fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f3120 0x5608906fcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f412a50fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 0x5608906fd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f412a50fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560890516fa0 con 0x5608906f3120 2024-05-27T07:45:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f412a50fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560890578da0 con 0x5608906f9070 2024-05-27T07:45:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f412a50fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5608906ffaa0 con 0x560890573e40 2024-05-27T07:45:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f4119095700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f3120 0x5608906fcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f412a50fb80 10 client.0 _ll_register_callbacks cb 0x5608905a75b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f4119095700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f3120 0x5608906fcdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60994/0 (socket says 172.21.15.114:60994) 2024-05-27T07:45:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f4119095700 1 -- 172.21.15.114:0/4219924405 learned_addr learned my addr 172.21.15.114:0/4219924405 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f411a097700 1 --2- 172.21.15.114:0/4219924405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 0x5608906fd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f4119896700 1 --2- 172.21.15.114:0/4219924405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x5608906fc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f4119095700 1 -- 172.21.15.114:0/4219924405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 msgr2=0x5608906fc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f4119095700 1 --2- 172.21.15.114:0/4219924405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x5608906fc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f4119095700 1 -- 172.21.15.114:0/4219924405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 msgr2=0x5608906fd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:49.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f4119095700 1 --2- 172.21.15.114:0/4219924405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 0x5608906fd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f4119095700 1 -- 172.21.15.114:0/4219924405 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608905608d0 con 0x5608906f3120 2024-05-27T07:45:49.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f4119095700 1 --2- 172.21.15.114:0/4219924405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f3120 0x5608906fcdb0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4104009ce0 tx=0x7f41040038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd564dda6509abb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:49.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f40fd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:49.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.381+0000 7f40fd7fa700 1 -- 172.21.15.114:0/4219924405 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410402f020 con 0x5608906f3120 2024-05-27T07:45:49.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.382+0000 7f412a50fb80 1 -- 172.21.15.114:0/4219924405 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f410400bd70 con 0x5608906f3120 2024-05-27T07:45:49.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.382+0000 7f40fd7fa700 1 -- 172.21.15.114:0/4219924405 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f4104003c00 con 0x5608906f3120 2024-05-27T07:45:49.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.382+0000 7f40fd7fa700 1 -- 172.21.15.114:0/4219924405 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f410403a050 con 0x5608906f3120 2024-05-27T07:45:49.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.382+0000 7f412a50fb80 10 client.16803 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:49.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.382+0000 7f412a50fb80 1 -- 172.21.15.114:0/4219924405 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5608905ca760 con 0x5608906f3120 2024-05-27T07:45:49.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.382+0000 7f40fd7fa700 1 -- 172.21.15.114:0/4219924405 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f410400bd70 con 0x5608906f3120 2024-05-27T07:45:49.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.382+0000 7f40fd7fa700 1 -- 172.21.15.114:0/4219924405 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f410400f840 con 0x5608906f3120 2024-05-27T07:45:49.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.382+0000 7f412a50fb80 10 client.16803 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:49.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.382+0000 7f412a50fb80 10 client.16803 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:49.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.382+0000 7f412a50fb80 20 client.16803 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:49.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.382+0000 7f412a50fb80 1 --2- 172.21.15.114:0/4219924405 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56089070cb00 0x56089070ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.382+0000 7f412a50fb80 4 client.16803 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:49.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.382+0000 7f412a50fb80 1 -- 172.21.15.114:0/4219924405 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560890622cc0 con 0x56089070cb00 2024-05-27T07:45:49.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.383+0000 7f4119896700 1 --2- 172.21.15.114:0/4219924405 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56089070cb00 0x56089070ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.383+0000 7f40fd7fa700 10 client.16803 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:49.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.383+0000 7f4119896700 1 --2- 172.21.15.114:0/4219924405 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56089070cb00 0x56089070ef20 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec6aafd06ed24869 server_cookie=98a7b80ef3ad4977 in_seq=0 out_seq=0 2024-05-27T07:45:49.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.386+0000 7f40fd7fa700 1 -- 172.21.15.114:0/4219924405 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560890622cc0 con 0x56089070cb00 2024-05-27T07:45:49.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.386+0000 7f40fd7fa700 10 client.16803 handle_command_reply: tid=0 2024-05-27T07:45:49.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.386+0000 7f412a50fb80 1 client.16803 shutdown 2024-05-27T07:45:49.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.386+0000 7f412a50fb80 1 -- 172.21.15.114:0/4219924405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f3120 msgr2=0x5608906fcdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:49.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.386+0000 7f412a50fb80 1 --2- 172.21.15.114:0/4219924405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f3120 0x5608906fcdb0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4104009ce0 tx=0x7f41040038b0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.386+0000 7f412a50fb80 1 -- 172.21.15.114:0/4219924405 shutdown_connections 2024-05-27T07:45:49.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.386+0000 7f412a50fb80 1 --2- 172.21.15.114:0/4219924405 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56089070cb00 0x56089070ef20 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.386+0000 7f412a50fb80 1 --2- 172.21.15.114:0/4219924405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 0x5608906fd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.386+0000 7f40fd7fa700 0 client.16803 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:49.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.386+0000 7f412a50fb80 1 --2- 172.21.15.114:0/4219924405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x5608906fc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.386+0000 7f412a50fb80 1 --2- 172.21.15.114:0/4219924405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f3120 0x5608906fcdb0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.386+0000 7f412a50fb80 1 -- 172.21.15.114:0/4219924405 >> 172.21.15.114:0/4219924405 conn(0x560890576770 msgr2=0x5608906f1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:49.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.387+0000 7f412a50fb80 1 -- 172.21.15.114:0/4219924405 shutdown_connections 2024-05-27T07:45:49.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.387+0000 7f412a50fb80 1 -- 172.21.15.114:0/4219924405 wait complete. 2024-05-27T07:45:49.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.387+0000 7f412a50fb80 20 client.16803 trim_cache size 0 max 16384 2024-05-27T07:45:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.403+0000 7f412a50fb80 1 Processor -- start 2024-05-27T07:45:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.403+0000 7f412a50fb80 1 -- start start 2024-05-27T07:45:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f412a50fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560890573e40 0x5608906f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f412a50fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608906f3120 0x5608906fd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f412a50fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 0x560890708b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f412a50fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f41141b79d0 con 0x560890573e40 2024-05-27T07:45:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f412a50fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f41141b4a80 con 0x5608906f9070 2024-05-27T07:45:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f412a50fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41141b6c30 con 0x5608906f3120 2024-05-27T07:45:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f4119896700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560890573e40 0x5608906f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f4119095700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608906f3120 0x5608906fd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f4119896700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560890573e40 0x5608906f2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60996/0 (socket says 172.21.15.114:60996) 2024-05-27T07:45:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f4119896700 1 -- 172.21.15.114:0/3556850535 learned_addr learned my addr 172.21.15.114:0/3556850535 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f411a097700 1 --2- 172.21.15.114:0/3556850535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 0x560890708b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f4119095700 1 -- 172.21.15.114:0/3556850535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 msgr2=0x560890708b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f4119095700 1 --2- 172.21.15.114:0/3556850535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 0x560890708b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f4119095700 1 -- 172.21.15.114:0/3556850535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560890573e40 msgr2=0x5608906f2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f4119095700 1 --2- 172.21.15.114:0/3556850535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560890573e40 0x5608906f2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f4119095700 1 -- 172.21.15.114:0/3556850535 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608906fbbb0 con 0x5608906f3120 2024-05-27T07:45:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f4119896700 1 --2- 172.21.15.114:0/3556850535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560890573e40 0x5608906f2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.404+0000 7f4119095700 1 --2- 172.21.15.114:0/3556850535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608906f3120 0x5608906fd130 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f410400b2e0 tx=0x7f4104000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7074c0bec0632fcf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.405+0000 7f40ff7fe700 1 -- 172.21.15.114:0/3556850535 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410402f040 con 0x5608906f3120 2024-05-27T07:45:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.405+0000 7f40ff7fe700 1 -- 172.21.15.114:0/3556850535 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4104005960 con 0x5608906f3120 2024-05-27T07:45:49.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.405+0000 7f40ff7fe700 1 -- 172.21.15.114:0/3556850535 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104022730 con 0x5608906f3120 2024-05-27T07:45:49.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.406+0000 7f412a50fb80 1 -- 172.21.15.114:0/3556850535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608906f3120 msgr2=0x5608906fd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:49.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.406+0000 7f412a50fb80 1 --2- 172.21.15.114:0/3556850535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608906f3120 0x5608906fd130 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f410400b2e0 tx=0x7f4104000bc0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.406+0000 7f412a50fb80 1 -- 172.21.15.114:0/3556850535 shutdown_connections 2024-05-27T07:45:49.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.406+0000 7f412a50fb80 1 --2- 172.21.15.114:0/3556850535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 0x560890708b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.406+0000 7f412a50fb80 1 --2- 172.21.15.114:0/3556850535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560890573e40 0x5608906f2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.406+0000 7f412a50fb80 1 --2- 172.21.15.114:0/3556850535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608906f3120 0x5608906fd130 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.406+0000 7f412a50fb80 1 -- 172.21.15.114:0/3556850535 >> 172.21.15.114:0/3556850535 conn(0x560890576770 msgr2=0x5608906f6490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.406+0000 7f412a50fb80 1 -- 172.21.15.114:0/3556850535 shutdown_connections 2024-05-27T07:45:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.406+0000 7f412a50fb80 1 -- 172.21.15.114:0/3556850535 wait complete. 2024-05-27T07:45:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.406+0000 7f412a50fb80 1 -- start start 2024-05-27T07:45:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.407+0000 7f412a50fb80 1 Processor -- start 2024-05-27T07:45:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.407+0000 7f412a50fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x560890704d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.407+0000 7f412a50fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f3120 0x560890705300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.407+0000 7f412a50fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 0x560890713820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.407+0000 7f412a50fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f41141b6560 con 0x5608906f3120 2024-05-27T07:45:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.407+0000 7f412a50fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f41141b50f0 con 0x5608906f9070 2024-05-27T07:45:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.407+0000 7f412a50fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41141b57c0 con 0x560890573e40 2024-05-27T07:45:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.407+0000 7f4119896700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x560890704d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.407+0000 7f4119095700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f3120 0x560890705300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.407+0000 7f4119896700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x560890704d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:48464/0 (socket says 172.21.15.114:48464) 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.407+0000 7f4119896700 1 -- 172.21.15.114:0/3630483136 learned_addr learned my addr 172.21.15.114:0/3630483136 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.407+0000 7f412a50fb80 10 client.0 _ll_register_callbacks cb 0x5608905a75b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f411a097700 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 0x560890713820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f4119896700 1 -- 172.21.15.114:0/3630483136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 msgr2=0x560890713820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f4119896700 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 0x560890713820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f4119896700 1 -- 172.21.15.114:0/3630483136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f3120 msgr2=0x560890705300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f4119896700 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f3120 0x560890705300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f4119896700 1 -- 172.21.15.114:0/3630483136 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560890713e30 con 0x560890573e40 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f4119095700 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f3120 0x560890705300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f411a097700 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 0x560890713820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f4119896700 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x560890704d60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f41100dffd0 tx=0x7f411000df90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24ce0a8d12f5dccf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f40fd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f40fd7fa700 1 -- 172.21.15.114:0/3630483136 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41100c7070 con 0x560890573e40 2024-05-27T07:45:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f40fd7fa700 1 -- 172.21.15.114:0/3630483136 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f41100a57f0 con 0x560890573e40 2024-05-27T07:45:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f412a50fb80 1 -- 172.21.15.114:0/3630483136 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4104005960 con 0x560890573e40 2024-05-27T07:45:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f40fd7fa700 1 -- 172.21.15.114:0/3630483136 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f411004f470 con 0x560890573e40 2024-05-27T07:45:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f412a50fb80 10 client.26401 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f412a50fb80 1 -- 172.21.15.114:0/3630483136 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560890776e60 con 0x560890573e40 2024-05-27T07:45:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.408+0000 7f40fd7fa700 1 -- 172.21.15.114:0/3630483136 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41100c3e50 con 0x560890573e40 2024-05-27T07:45:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.409+0000 7f40fd7fa700 1 -- 172.21.15.114:0/3630483136 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f41100e2200 con 0x560890573e40 2024-05-27T07:45:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.409+0000 7f412a50fb80 10 client.26401 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.409+0000 7f412a50fb80 10 client.26401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.409+0000 7f412a50fb80 20 client.26401 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.409+0000 7f412a50fb80 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56089070cb00 0x5608907795b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.409+0000 7f412a50fb80 4 client.26401 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:45:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.409+0000 7f412a50fb80 1 -- 172.21.15.114:0/3630483136 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f4104005960 con 0x56089070cb00 2024-05-27T07:45:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.409+0000 7f4119095700 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56089070cb00 0x5608907795b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.409+0000 7f40fd7fa700 10 client.26401 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.409+0000 7f4119095700 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56089070cb00 0x5608907795b0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e83ee03b4efeea5 server_cookie=4e69cda7ced3c198 in_seq=0 out_seq=0 2024-05-27T07:45:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.410+0000 7f40fd7fa700 1 -- 172.21.15.114:0/3630483136 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f4104005960 con 0x56089070cb00 2024-05-27T07:45:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.410+0000 7f40fd7fa700 10 client.26401 handle_command_reply: tid=0 2024-05-27T07:45:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.410+0000 7f412a50fb80 1 client.26401 shutdown 2024-05-27T07:45:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.410+0000 7f412a50fb80 1 -- 172.21.15.114:0/3630483136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 msgr2=0x560890704d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.410+0000 7f412a50fb80 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x560890704d60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f41100dffd0 tx=0x7f411000df90 comp rx=0 tx=0).stop 2024-05-27T07:45:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.410+0000 7f412a50fb80 1 -- 172.21.15.114:0/3630483136 shutdown_connections 2024-05-27T07:45:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.410+0000 7f412a50fb80 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56089070cb00 0x5608907795b0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.410+0000 7f412a50fb80 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5608906f9070 0x560890713820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.410+0000 7f412a50fb80 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560890573e40 0x560890704d60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.410+0000 7f40fd7fa700 0 client.26401 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.410+0000 7f412a50fb80 1 --2- 172.21.15.114:0/3630483136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5608906f3120 0x560890705300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.411+0000 7f412a50fb80 1 -- 172.21.15.114:0/3630483136 >> 172.21.15.114:0/3630483136 conn(0x560890576770 msgr2=0x5608906e74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.411+0000 7f412a50fb80 1 -- 172.21.15.114:0/3630483136 shutdown_connections 2024-05-27T07:45:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.411+0000 7f412a50fb80 1 -- 172.21.15.114:0/3630483136 wait complete. 2024-05-27T07:45:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.411+0000 7f412a50fb80 20 client.26401 trim_cache size 0 max 16384 2024-05-27T07:45:49.416 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:45:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.411+0000 7f411bafa700 1 -- 172.21.15.114:0/3849320784 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f41000b2b40 msgr2=0x7f41000b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.411+0000 7f411bafa700 1 --2- 172.21.15.114:0/3849320784 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f41000b2b40 0x7f41000b5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f410c007b60 tx=0x7f410c0076b0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.411+0000 7f411bafa700 1 -- 172.21.15.114:0/3849320784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411410a960 msgr2=0x7f41141a1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.411+0000 7f411bafa700 1 --2- 172.21.15.114:0/3849320784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411410a960 0x7f41141a1440 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f411000d950 tx=0x7f411000dc60 comp rx=0 tx=0).stop 2024-05-27T07:45:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.411+0000 7f411bafa700 1 -- 172.21.15.114:0/3849320784 shutdown_connections 2024-05-27T07:45:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.412+0000 7f411bafa700 1 --2- 172.21.15.114:0/3849320784 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f41000b2b40 0x7f41000b5050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.412+0000 7f411bafa700 1 --2- 172.21.15.114:0/3849320784 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4114110990 0x7f41141a5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.412+0000 7f411bafa700 1 --2- 172.21.15.114:0/3849320784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411410a960 0x7f41141a1440 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.412+0000 7f411bafa700 1 --2- 172.21.15.114:0/3849320784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411410b2b0 0x7f41141a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:49.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.412+0000 7f411bafa700 1 -- 172.21.15.114:0/3849320784 >> 172.21.15.114:0/3849320784 conn(0x7f4114076cb0 msgr2=0x7f4114108850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:49.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.412+0000 7f411bafa700 1 -- 172.21.15.114:0/3849320784 shutdown_connections 2024-05-27T07:45:49.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:49.412+0000 7f411bafa700 1 -- 172.21.15.114:0/3849320784 wait complete. 2024-05-27T07:45:49.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:49 smithi114 ceph-mon[32038]: pgmap v192: 129 pgs: 129 active+clean; 74 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 363 op/s 2024-05-27T07:45:49.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:49 smithi163 ceph-mon[36451]: pgmap v192: 129 pgs: 129 active+clean; 74 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 363 op/s 2024-05-27T07:45:49.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:49 smithi186 ceph-mon[36647]: pgmap v192: 129 pgs: 129 active+clean; 74 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 363 op/s 2024-05-27T07:45:50.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 601438 13.03 MB/sec execute 122 sec latency 96.988 ms 2024-05-27T07:45:51.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 604181 13.06 MB/sec execute 123 sec latency 34.243 ms 2024-05-27T07:45:51.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:51 smithi163 ceph-mon[36451]: pgmap v193: 129 pgs: 129 active+clean; 74 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.9 MiB/s wr, 418 op/s 2024-05-27T07:45:51.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:51 smithi186 ceph-mon[36647]: pgmap v193: 129 pgs: 129 active+clean; 74 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.9 MiB/s wr, 418 op/s 2024-05-27T07:45:51.796 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:45:51.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:45:51.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:51 smithi114 ceph-mon[32038]: pgmap v193: 129 pgs: 129 active+clean; 74 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.9 MiB/s wr, 418 op/s 2024-05-27T07:45:52.060 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:52.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 606977 13.06 MB/sec execute 124 sec latency 16.207 ms 2024-05-27T07:45:53.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 608903 13.04 MB/sec execute 125 sec latency 16.969 ms 2024-05-27T07:45:53.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:53 smithi114 ceph-mon[32038]: pgmap v194: 129 pgs: 129 active+clean; 79 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 424 op/s 2024-05-27T07:45:53.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:53 smithi163 ceph-mon[36451]: pgmap v194: 129 pgs: 129 active+clean; 79 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 424 op/s 2024-05-27T07:45:53.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:53 smithi186 ceph-mon[36647]: pgmap v194: 129 pgs: 129 active+clean; 79 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 424 op/s 2024-05-27T07:45:54.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 611474 13.05 MB/sec execute 126 sec latency 10.798 ms 2024-05-27T07:45:54.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.291+0000 7efd0f007700 1 -- 172.21.15.114:0/2904295671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08108730 msgr2=0x7efd08110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:54.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.291+0000 7efd0f007700 1 --2- 172.21.15.114:0/2904295671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08108730 0x7efd08110090 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efd0000b0d0 tx=0x7efd0000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.291+0000 7efd0f007700 1 -- 172.21.15.114:0/2904295671 shutdown_connections 2024-05-27T07:45:54.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.291+0000 7efd0f007700 1 --2- 172.21.15.114:0/2904295671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08108730 0x7efd08110090 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.291+0000 7efd0f007700 1 --2- 172.21.15.114:0/2904295671 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd081034b0 0x7efd081038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.291+0000 7efd0f007700 1 --2- 172.21.15.114:0/2904295671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd08107de0 0x7efd081081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.291+0000 7efd0f007700 1 -- 172.21.15.114:0/2904295671 >> 172.21.15.114:0/2904295671 conn(0x7efd080fedb0 msgr2=0x7efd081011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:54.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.291+0000 7efd0f007700 1 -- 172.21.15.114:0/2904295671 shutdown_connections 2024-05-27T07:45:54.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.291+0000 7efd0f007700 1 -- 172.21.15.114:0/2904295671 wait complete. 2024-05-27T07:45:54.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.292+0000 7efd0f007700 1 Processor -- start 2024-05-27T07:45:54.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.292+0000 7efd0f007700 1 -- start start 2024-05-27T07:45:54.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.292+0000 7efd0f007700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd081034b0 0x7efd0819f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.292+0000 7efd0f007700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08107de0 0x7efd0819fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.292+0000 7efd0f007700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd08108730 0x7efd08199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.292+0000 7efd0f007700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd08113360 con 0x7efd08107de0 2024-05-27T07:45:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.292+0000 7efd0f007700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd081131e0 con 0x7efd081034b0 2024-05-27T07:45:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.292+0000 7efd0f007700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efd081134e0 con 0x7efd08108730 2024-05-27T07:45:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.292+0000 7efd07fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08107de0 0x7efd0819fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.292+0000 7efd07fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08107de0 0x7efd0819fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32796/0 (socket says 172.21.15.114:32796) 2024-05-27T07:45:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.292+0000 7efd0cda3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd081034b0 0x7efd0819f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.292+0000 7efd07fff700 1 -- 172.21.15.114:0/4227921257 learned_addr learned my addr 172.21.15.114:0/4227921257 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.292+0000 7efd0d5a4700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd08108730 0x7efd08199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.293+0000 7efd07fff700 1 -- 172.21.15.114:0/4227921257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd08108730 msgr2=0x7efd08199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.293+0000 7efd07fff700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd08108730 0x7efd08199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.293+0000 7efd07fff700 1 -- 172.21.15.114:0/4227921257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd081034b0 msgr2=0x7efd0819f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.293+0000 7efd07fff700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd081034b0 0x7efd0819f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.293+0000 7efd07fff700 1 -- 172.21.15.114:0/4227921257 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0819a110 con 0x7efd08107de0 2024-05-27T07:45:54.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.293+0000 7efd0cda3700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd081034b0 0x7efd0819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:54.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.293+0000 7efd07fff700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08107de0 0x7efd0819fd80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7efcf400e970 tx=0x7efcf400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:54.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.293+0000 7efd05ffb700 1 -- 172.21.15.114:0/4227921257 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4004a30 con 0x7efd08107de0 2024-05-27T07:45:54.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.293+0000 7efd05ffb700 1 -- 172.21.15.114:0/4227921257 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcf400cc90 con 0x7efd08107de0 2024-05-27T07:45:54.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.293+0000 7efd0f007700 1 -- 172.21.15.114:0/4227921257 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0819a400 con 0x7efd08107de0 2024-05-27T07:45:54.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.293+0000 7efd05ffb700 1 -- 172.21.15.114:0/4227921257 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4005610 con 0x7efd08107de0 2024-05-27T07:45:54.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.293+0000 7efd0f007700 1 -- 172.21.15.114:0/4227921257 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd0806b600 con 0x7efd08107de0 2024-05-27T07:45:54.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.296+0000 7efd05ffb700 1 -- 172.21.15.114:0/4227921257 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7efcf400c7e0 con 0x7efd08107de0 2024-05-27T07:45:54.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.297+0000 7efd05ffb700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efcf80b4c00 0x7efcf80b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.297+0000 7efd05ffb700 1 -- 172.21.15.114:0/4227921257 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7efcf40e2ac0 con 0x7efd08107de0 2024-05-27T07:45:54.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.297+0000 7efd0cda3700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efcf80b4c00 0x7efcf80b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.298+0000 7efd0cda3700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efcf80b4c00 0x7efcf80b7110 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efd0819aba0 tx=0x7efcfc009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:54.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.298+0000 7efd1da1cb80 1 Processor -- start 2024-05-27T07:45:54.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.298+0000 7efd1da1cb80 1 -- start start 2024-05-27T07:45:54.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd1da1cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d3fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd1da1cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d400f0 0x5567f4d39c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd1da1cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d3a1a0 0x5567f4d3a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd1da1cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5567f4c67df0 con 0x5567f4bbaec0 2024-05-27T07:45:54.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd1da1cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5567f4c57990 con 0x5567f4d400f0 2024-05-27T07:45:54.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd1da1cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5567f4c5b9d0 con 0x5567f4d3a1a0 2024-05-27T07:45:54.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd0cda3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d3fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd0cda3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d3fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32812/0 (socket says 172.21.15.114:32812) 2024-05-27T07:45:54.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd0cda3700 1 -- 172.21.15.114:0/3091488059 learned_addr learned my addr 172.21.15.114:0/3091488059 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:54.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd0d5a4700 1 --2- 172.21.15.114:0/3091488059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d3a1a0 0x5567f4d3a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd07fff700 1 --2- 172.21.15.114:0/3091488059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d400f0 0x5567f4d39c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd0cda3700 1 -- 172.21.15.114:0/3091488059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d3a1a0 msgr2=0x5567f4d3a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:54.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd0cda3700 1 --2- 172.21.15.114:0/3091488059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d3a1a0 0x5567f4d3a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd0cda3700 1 -- 172.21.15.114:0/3091488059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d400f0 msgr2=0x5567f4d39c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:54.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd0cda3700 1 --2- 172.21.15.114:0/3091488059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d400f0 0x5567f4d39c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd0cda3700 1 -- 172.21.15.114:0/3091488059 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567f4c01030 con 0x5567f4bbaec0 2024-05-27T07:45:54.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.299+0000 7efd07fff700 1 --2- 172.21.15.114:0/3091488059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d400f0 0x5567f4d39c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:45:54.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.300+0000 7efd0cda3700 1 --2- 172.21.15.114:0/3091488059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d3fbb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0043c0 tx=0x7efcfc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8b9a9c867f9e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:54.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.300+0000 7efcf2ffd700 1 -- 172.21.15.114:0/3091488059 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc003d10 con 0x5567f4bbaec0 2024-05-27T07:45:54.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.300+0000 7efcf2ffd700 1 -- 172.21.15.114:0/3091488059 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcfc01e8d0 con 0x5567f4bbaec0 2024-05-27T07:45:54.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.300+0000 7efcf2ffd700 1 -- 172.21.15.114:0/3091488059 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc026800 con 0x5567f4bbaec0 2024-05-27T07:45:54.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.301+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3091488059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 msgr2=0x5567f4d3fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:54.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.301+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3091488059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d3fbb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0043c0 tx=0x7efcfc014000 comp rx=0 tx=0).stop 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.301+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3091488059 shutdown_connections 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.301+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3091488059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d3a1a0 0x5567f4d3a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.301+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3091488059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d3fbb0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.301+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3091488059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d400f0 0x5567f4d39c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.301+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3091488059 >> 172.21.15.114:0/3091488059 conn(0x5567f4bbd7f0 msgr2=0x5567f4d34e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.301+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3091488059 shutdown_connections 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.301+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3091488059 wait complete. 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.301+0000 7efd1da1cb80 1 -- start start 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.301+0000 7efd1da1cb80 1 Processor -- start 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd1da1cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d439d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd1da1cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d3a1a0 0x5567f4d43f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd1da1cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d44450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd1da1cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5567f4b5e020 con 0x5567f4bbaec0 2024-05-27T07:45:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd1da1cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5567f4bbfe20 con 0x5567f4d3a1a0 2024-05-27T07:45:54.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd1da1cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5567f4d4aff0 con 0x5567f4d400f0 2024-05-27T07:45:54.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd1da1cb80 10 client.0 _ll_register_callbacks cb 0x5567f4bee630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:54.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd0cda3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd0d5a4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d44450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd0d5a4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d44450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:48488/0 (socket says 172.21.15.114:48488) 2024-05-27T07:45:54.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd0d5a4700 1 -- 172.21.15.114:0/3850669551 learned_addr learned my addr 172.21.15.114:0/3850669551 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:54.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd07fff700 1 --2- 172.21.15.114:0/3850669551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d3a1a0 0x5567f4d43f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd0d5a4700 1 -- 172.21.15.114:0/3850669551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d3a1a0 msgr2=0x5567f4d43f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:54.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd0d5a4700 1 --2- 172.21.15.114:0/3850669551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d3a1a0 0x5567f4d43f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd0d5a4700 1 -- 172.21.15.114:0/3850669551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 msgr2=0x5567f4d439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:54.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd0d5a4700 1 --2- 172.21.15.114:0/3850669551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d439d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd0d5a4700 1 -- 172.21.15.114:0/3850669551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567f4ba7950 con 0x5567f4d400f0 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.302+0000 7efd07fff700 1 --2- 172.21.15.114:0/3850669551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d3a1a0 0x5567f4d43f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.303+0000 7efd0d5a4700 1 --2- 172.21.15.114:0/3850669551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d44450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efd00005b60 tx=0x7efd00005b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c91807076ca7a0a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.303+0000 7efce7fff700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.303+0000 7efce7fff700 1 -- 172.21.15.114:0/3850669551 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0000bde0 con 0x5567f4d400f0 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.303+0000 7efce7fff700 1 -- 172.21.15.114:0/3850669551 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7efd00003a00 con 0x5567f4d400f0 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.303+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3850669551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcfc01e8d0 con 0x5567f4d400f0 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.303+0000 7efce7fff700 1 -- 172.21.15.114:0/3850669551 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7efd0002f050 con 0x5567f4d400f0 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.303+0000 7efd1da1cb80 10 client.26413 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.303+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3850669551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5567f4c117e0 con 0x5567f4d400f0 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.303+0000 7efce7fff700 1 -- 172.21.15.114:0/3850669551 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcfc01e8d0 con 0x5567f4d400f0 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.304+0000 7efce7fff700 1 -- 172.21.15.114:0/3850669551 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7efd00011070 con 0x5567f4d400f0 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.304+0000 7efd1da1cb80 10 client.26413 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.304+0000 7efd1da1cb80 10 client.26413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.304+0000 7efd1da1cb80 20 client.26413 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.304+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3850669551 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5567f4d48d50 0x5567f4d49160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.304+0000 7efd1da1cb80 4 client.26413 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.304+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3850669551 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567f4c69d40 con 0x5567f4d48d50 2024-05-27T07:45:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.304+0000 7efd0cda3700 1 --2- 172.21.15.114:0/3850669551 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5567f4d48d50 0x5567f4d49160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.305+0000 7efce7fff700 10 client.26413 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.305+0000 7efd0cda3700 1 --2- 172.21.15.114:0/3850669551 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5567f4d48d50 0x5567f4d49160 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91a4709ebf2c4be6 server_cookie=8da36c7c91c36e51 in_seq=0 out_seq=0 2024-05-27T07:45:54.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efce7fff700 1 -- 172.21.15.114:0/3850669551 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5567f4c69d40 con 0x5567f4d48d50 2024-05-27T07:45:54.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efce7fff700 10 client.26413 handle_command_reply: tid=0 2024-05-27T07:45:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efd1da1cb80 1 client.26413 shutdown 2024-05-27T07:45:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3850669551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 msgr2=0x5567f4d44450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3850669551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d44450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efd00005b60 tx=0x7efd00005b90 comp rx=0 tx=0).stop 2024-05-27T07:45:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3850669551 shutdown_connections 2024-05-27T07:45:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3850669551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d44450 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3850669551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3850669551 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5567f4d48d50 0x5567f4d49160 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efce7fff700 0 client.26413 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3850669551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d3a1a0 0x5567f4d43f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3850669551 >> 172.21.15.114:0/3850669551 conn(0x5567f4bbd7f0 msgr2=0x5567f4d38c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3850669551 shutdown_connections 2024-05-27T07:45:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3850669551 wait complete. 2024-05-27T07:45:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.307+0000 7efd1da1cb80 20 client.26413 trim_cache size 0 max 16384 2024-05-27T07:45:54.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.329+0000 7efd1da1cb80 1 Processor -- start 2024-05-27T07:45:54.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.329+0000 7efd1da1cb80 1 -- start start 2024-05-27T07:45:54.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.329+0000 7efd1da1cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4bbaec0 0x5567f4d42930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.329+0000 7efd1da1cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4d3a1a0 0x5567f4d54290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.329+0000 7efd1da1cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d44290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.329+0000 7efd1da1cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd081af1f0 con 0x5567f4d3a1a0 2024-05-27T07:45:54.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.329+0000 7efd1da1cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd081ac2a0 con 0x5567f4bbaec0 2024-05-27T07:45:54.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.329+0000 7efd1da1cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efd081ae450 con 0x5567f4d400f0 2024-05-27T07:45:54.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.329+0000 7efd07fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4d3a1a0 0x5567f4d54290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efd07fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4d3a1a0 0x5567f4d54290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32836/0 (socket says 172.21.15.114:32836) 2024-05-27T07:45:54.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efd07fff700 1 -- 172.21.15.114:0/3088726569 learned_addr learned my addr 172.21.15.114:0/3088726569 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:54.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efd0d5a4700 1 --2- 172.21.15.114:0/3088726569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d44290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efd0cda3700 1 --2- 172.21.15.114:0/3088726569 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4bbaec0 0x5567f4d42930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efd0d5a4700 1 -- 172.21.15.114:0/3088726569 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4bbaec0 msgr2=0x5567f4d42930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:54.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efd0d5a4700 1 --2- 172.21.15.114:0/3088726569 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4bbaec0 0x5567f4d42930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efd0d5a4700 1 -- 172.21.15.114:0/3088726569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4d3a1a0 msgr2=0x5567f4d54290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:54.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efd0d5a4700 1 --2- 172.21.15.114:0/3088726569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4d3a1a0 0x5567f4d54290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efd0d5a4700 1 -- 172.21.15.114:0/3088726569 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567f4d49680 con 0x5567f4d400f0 2024-05-27T07:45:54.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efd07fff700 1 --2- 172.21.15.114:0/3088726569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4d3a1a0 0x5567f4d54290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:54.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efd0d5a4700 1 --2- 172.21.15.114:0/3088726569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d44290 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efd00009a20 tx=0x7efd00006180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb06c7a66823acd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:54.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efcf27fc700 1 -- 172.21.15.114:0/3088726569 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd00007cd0 con 0x5567f4d400f0 2024-05-27T07:45:54.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efcf27fc700 1 -- 172.21.15.114:0/3088726569 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd00005c60 con 0x5567f4d400f0 2024-05-27T07:45:54.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.330+0000 7efcf27fc700 1 -- 172.21.15.114:0/3088726569 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd00005110 con 0x5567f4d400f0 2024-05-27T07:45:54.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.331+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3088726569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 msgr2=0x5567f4d44290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.331+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3088726569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d44290 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efd00009a20 tx=0x7efd00006180 comp rx=0 tx=0).stop 2024-05-27T07:45:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.331+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3088726569 shutdown_connections 2024-05-27T07:45:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.331+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3088726569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d44290 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.331+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3088726569 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4bbaec0 0x5567f4d42930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.331+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/3088726569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4d3a1a0 0x5567f4d54290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.331+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3088726569 >> 172.21.15.114:0/3088726569 conn(0x5567f4bbd7f0 msgr2=0x5567f4d3d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.331+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3088726569 shutdown_connections 2024-05-27T07:45:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.331+0000 7efd1da1cb80 1 -- 172.21.15.114:0/3088726569 wait complete. 2024-05-27T07:45:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd1da1cb80 1 -- start start 2024-05-27T07:45:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd1da1cb80 1 Processor -- start 2024-05-27T07:45:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd1da1cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d60390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd1da1cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d3a1a0 0x5567f4d608d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd1da1cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d60e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd1da1cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd081add80 con 0x5567f4bbaec0 2024-05-27T07:45:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd1da1cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd081ac910 con 0x5567f4d3a1a0 2024-05-27T07:45:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd1da1cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efd081acfe0 con 0x5567f4d400f0 2024-05-27T07:45:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd1da1cb80 10 client.0 _ll_register_callbacks cb 0x5567f4bee630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd0cda3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d60390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd0cda3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d60390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32838/0 (socket says 172.21.15.114:32838) 2024-05-27T07:45:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd0cda3700 1 -- 172.21.15.114:0/54889633 learned_addr learned my addr 172.21.15.114:0/54889633 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd0d5a4700 1 --2- 172.21.15.114:0/54889633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d60e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.332+0000 7efd07fff700 1 --2- 172.21.15.114:0/54889633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d3a1a0 0x5567f4d608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efd0cda3700 1 -- 172.21.15.114:0/54889633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 msgr2=0x5567f4d60e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efd0cda3700 1 --2- 172.21.15.114:0/54889633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d60e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efd0cda3700 1 -- 172.21.15.114:0/54889633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d3a1a0 msgr2=0x5567f4d608d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efd0cda3700 1 --2- 172.21.15.114:0/54889633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d3a1a0 0x5567f4d608d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efd0cda3700 1 -- 172.21.15.114:0/54889633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567f4dc6280 con 0x5567f4bbaec0 2024-05-27T07:45:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efd0cda3700 1 --2- 172.21.15.114:0/54889633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d60390 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7efcfc003f20 tx=0x7efcfc003b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e5ffa8684fd7b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efce7fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efce7fff700 1 -- 172.21.15.114:0/54889633 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc005310 con 0x5567f4bbaec0 2024-05-27T07:45:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efce7fff700 1 -- 172.21.15.114:0/54889633 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7efcfc0054b0 con 0x5567f4bbaec0 2024-05-27T07:45:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efd1da1cb80 1 -- 172.21.15.114:0/54889633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd00005c60 con 0x5567f4bbaec0 2024-05-27T07:45:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efce7fff700 1 -- 172.21.15.114:0/54889633 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7efcfc0056b0 con 0x5567f4bbaec0 2024-05-27T07:45:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efd1da1cb80 10 client.16827 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efd1da1cb80 1 -- 172.21.15.114:0/54889633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5567f4dc7480 con 0x5567f4bbaec0 2024-05-27T07:45:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.333+0000 7efce7fff700 1 -- 172.21.15.114:0/54889633 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd00005c60 con 0x5567f4bbaec0 2024-05-27T07:45:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.334+0000 7efce7fff700 1 -- 172.21.15.114:0/54889633 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7efcfc030040 con 0x5567f4bbaec0 2024-05-27T07:45:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.334+0000 7efd1da1cb80 10 client.16827 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:54.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.334+0000 7efd1da1cb80 10 client.16827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:54.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.334+0000 7efd1da1cb80 20 client.16827 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:54.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.334+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/54889633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5567f4d48d50 0x5567f4dc9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:54.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.334+0000 7efd1da1cb80 4 client.16827 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:45:54.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.334+0000 7efd1da1cb80 1 -- 172.21.15.114:0/54889633 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5567f4c63520 con 0x5567f4d48d50 2024-05-27T07:45:54.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.334+0000 7efd07fff700 1 --2- 172.21.15.114:0/54889633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5567f4d48d50 0x5567f4dc9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:54.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.334+0000 7efce7fff700 10 client.16827 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:54.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.334+0000 7efd07fff700 1 --2- 172.21.15.114:0/54889633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5567f4d48d50 0x5567f4dc9950 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bdd42b16719dff server_cookie=96f2c8c4deed3e53 in_seq=0 out_seq=0 2024-05-27T07:45:54.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.337+0000 7efce7fff700 1 -- 172.21.15.114:0/54889633 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5567f4c63520 con 0x5567f4d48d50 2024-05-27T07:45:54.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.337+0000 7efce7fff700 10 client.16827 handle_command_reply: tid=0 2024-05-27T07:45:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.337+0000 7efd1da1cb80 1 client.16827 shutdown 2024-05-27T07:45:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.337+0000 7efd1da1cb80 1 -- 172.21.15.114:0/54889633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 msgr2=0x5567f4d60390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.337+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/54889633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d60390 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7efcfc003f20 tx=0x7efcfc003b10 comp rx=0 tx=0).stop 2024-05-27T07:45:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.338+0000 7efd1da1cb80 1 -- 172.21.15.114:0/54889633 shutdown_connections 2024-05-27T07:45:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.338+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/54889633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5567f4d400f0 0x5567f4d60e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.338+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/54889633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567f4bbaec0 0x5567f4d60390 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.338+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/54889633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5567f4d48d50 0x5567f4dc9950 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.338+0000 7efce7fff700 0 client.16827 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.338+0000 7efd1da1cb80 1 --2- 172.21.15.114:0/54889633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5567f4d3a1a0 0x5567f4d608d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.338+0000 7efd1da1cb80 1 -- 172.21.15.114:0/54889633 >> 172.21.15.114:0/54889633 conn(0x5567f4bbd7f0 msgr2=0x5567f4d3dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.338+0000 7efd1da1cb80 1 -- 172.21.15.114:0/54889633 shutdown_connections 2024-05-27T07:45:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.338+0000 7efd1da1cb80 1 -- 172.21.15.114:0/54889633 wait complete. 2024-05-27T07:45:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.338+0000 7efd1da1cb80 20 client.16827 trim_cache size 0 max 16384 2024-05-27T07:45:54.341 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:45:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:45:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "c0042df7-5afa-4903-b32c-a2f7350bfa05", 2024-05-27T07:45:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:45:54.341 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:45:54.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.339+0000 7efd0f007700 1 -- 172.21.15.114:0/4227921257 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efcf80b4c00 msgr2=0x7efcf80b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:54.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.339+0000 7efd0f007700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efcf80b4c00 0x7efcf80b7110 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efd0819aba0 tx=0x7efcfc009260 comp rx=0 tx=0).stop 2024-05-27T07:45:54.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.339+0000 7efd0f007700 1 -- 172.21.15.114:0/4227921257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08107de0 msgr2=0x7efd0819fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:54.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.339+0000 7efd0f007700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08107de0 0x7efd0819fd80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7efcf400e970 tx=0x7efcf400ec80 comp rx=0 tx=0).stop 2024-05-27T07:45:54.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.339+0000 7efd0f007700 1 -- 172.21.15.114:0/4227921257 shutdown_connections 2024-05-27T07:45:54.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.339+0000 7efd0f007700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd08108730 0x7efd08199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.339+0000 7efd0f007700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efcf80b4c00 0x7efcf80b7110 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.339+0000 7efd0f007700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd081034b0 0x7efd0819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:54.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.339+0000 7efd0f007700 1 --2- 172.21.15.114:0/4227921257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08107de0 0x7efd0819fd80 secure :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7efcf400e970 tx=0x7efcf400ec80 comp rx=0 tx=0).stop 2024-05-27T07:45:54.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.339+0000 7efd0f007700 1 -- 172.21.15.114:0/4227921257 >> 172.21.15.114:0/4227921257 conn(0x7efd080fedb0 msgr2=0x7efd08107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:54.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.339+0000 7efd0f007700 1 -- 172.21.15.114:0/4227921257 shutdown_connections 2024-05-27T07:45:54.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:54.339+0000 7efd0f007700 1 -- 172.21.15.114:0/4227921257 wait complete. 2024-05-27T07:45:55.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:45:55.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 613489 13.01 MB/sec execute 127 sec latency 91.810 ms 2024-05-27T07:45:55.517 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:55.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:55 smithi163 ceph-mon[36451]: pgmap v195: 129 pgs: 129 active+clean; 79 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 361 op/s 2024-05-27T07:45:55.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:55 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:45:55.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:55 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:55.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:55 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:45:55.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:55 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:55.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:55 smithi186 ceph-mon[36647]: pgmap v195: 129 pgs: 129 active+clean; 79 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 361 op/s 2024-05-27T07:45:55.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:55 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:45:55.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:55 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:55.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:55 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:45:55.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:55 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:55.765 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:55 smithi114 ceph-mon[32038]: pgmap v195: 129 pgs: 129 active+clean; 79 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 361 op/s 2024-05-27T07:45:55.765 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:55 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:45:55.765 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:55 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:55.765 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:55 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:45:55.765 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:55 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:45:56.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 615415 13.00 MB/sec execute 128 sec latency 39.624 ms 2024-05-27T07:45:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.617+0000 7f9ee1c6e640 1 -- 172.21.15.114:0/394910654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc108b90 msgr2=0x7f9edc102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.617+0000 7f9ee1c6e640 1 --2- 172.21.15.114:0/394910654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc108b90 0x7f9edc102730 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009860 tx=0x7f9ecc02a340 comp rx=0 tx=0).stop 2024-05-27T07:45:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.617+0000 7f9ee1c6e640 1 -- 172.21.15.114:0/394910654 shutdown_connections 2024-05-27T07:45:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.617+0000 7f9ee1c6e640 1 --2- 172.21.15.114:0/394910654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9edc1032e0 0x7f9edc10f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.617+0000 7f9ee1c6e640 1 --2- 172.21.15.114:0/394910654 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9edc109030 0x7f9edc102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.617+0000 7f9ee1c6e640 1 --2- 172.21.15.114:0/394910654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc108b90 0x7f9edc102730 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.617+0000 7f9ee1c6e640 1 -- 172.21.15.114:0/394910654 >> 172.21.15.114:0/394910654 conn(0x7f9edc0782d0 msgr2=0x7f9edc078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.617+0000 7f9ee1c6e640 1 -- 172.21.15.114:0/394910654 shutdown_connections 2024-05-27T07:45:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.617+0000 7f9ee1c6e640 1 -- 172.21.15.114:0/394910654 wait complete. 2024-05-27T07:45:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.617+0000 7f9ee1c6e640 1 Processor -- start 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.617+0000 7f9ee1c6e640 1 -- start start 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9ee1c6e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9edc1032e0 0x7f9edc1997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9ee1c6e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9edc108b90 0x7f9edc199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9ee1c6e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc109030 0x7f9edc19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9ee1c6e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc112d80 con 0x7f9edc109030 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9ee1c6e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc112c00 con 0x7f9edc108b90 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9ee1c6e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc112f00 con 0x7f9edc1032e0 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9edbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc109030 0x7f9edc19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9edb7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9edc1032e0 0x7f9edc1997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9edb7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9edc1032e0 0x7f9edc1997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59406/0 (socket says 172.21.15.114:59406) 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9edbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc109030 0x7f9edc19dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44396/0 (socket says 172.21.15.114:44396) 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9edb7fe640 1 -- 172.21.15.114:0/1198105407 learned_addr learned my addr 172.21.15.114:0/1198105407 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9ed3fff640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9edc108b90 0x7f9edc199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9edbfff640 1 -- 172.21.15.114:0/1198105407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9edc1032e0 msgr2=0x7f9edc1997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9edbfff640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9edc1032e0 0x7f9edc1997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9edbfff640 1 -- 172.21.15.114:0/1198105407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9edc108b90 msgr2=0x7f9edc199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9edbfff640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9edc108b90 0x7f9edc199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9edbfff640 1 -- 172.21.15.114:0/1198105407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9edc19e600 con 0x7f9edc109030 2024-05-27T07:45:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9edb7fe640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9edc1032e0 0x7f9edc1997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.618+0000 7f9edbfff640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc109030 0x7f9edc19dfc0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400ef00 tx=0x7f9ed400c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.619+0000 7f9ed97fa640 1 -- 172.21.15.114:0/1198105407 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4004b20 con 0x7f9edc109030 2024-05-27T07:45:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.619+0000 7f9ed97fa640 1 -- 172.21.15.114:0/1198105407 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ed4003e20 con 0x7f9edc109030 2024-05-27T07:45:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.619+0000 7f9ee1c6e640 1 -- 172.21.15.114:0/1198105407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9edc19e8f0 con 0x7f9edc109030 2024-05-27T07:45:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.619+0000 7f9ed97fa640 1 -- 172.21.15.114:0/1198105407 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed400b570 con 0x7f9edc109030 2024-05-27T07:45:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.619+0000 7f9ee1c6e640 1 -- 172.21.15.114:0/1198105407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9edc19edf0 con 0x7f9edc109030 2024-05-27T07:45:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.621+0000 7f9ee8c00740 1 Processor -- start 2024-05-27T07:45:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.621+0000 7f9ed97fa640 1 -- 172.21.15.114:0/1198105407 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f9ed401d020 con 0x7f9edc109030 2024-05-27T07:45:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.622+0000 7f9ee8c00740 1 -- start start 2024-05-27T07:45:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.622+0000 7f9ee8c00740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13e31b0 0x55e5c1784220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.622+0000 7f9ee8c00740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13ba2f0 0x55e5c1784760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.622+0000 7f9ee8c00740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13c24d0 0x55e5c1784ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.622+0000 7f9ee8c00740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e5c16f6c30 con 0x55e5c13ba2f0 2024-05-27T07:45:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.622+0000 7f9ee8c00740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e5c16e67d0 con 0x55e5c13e31b0 2024-05-27T07:45:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.622+0000 7f9ee8c00740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e5c16ea810 con 0x55e5c13c24d0 2024-05-27T07:45:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.622+0000 7f9ed3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13c24d0 0x55e5c1784ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.622+0000 7f9edb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13ba2f0 0x55e5c1784760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.622+0000 7f9edb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13ba2f0 0x55e5c1784760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44402/0 (socket says 172.21.15.114:44402) 2024-05-27T07:45:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.622+0000 7f9edb7fe640 1 -- 172.21.15.114:0/2403490464 learned_addr learned my addr 172.21.15.114:0/2403490464 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.623+0000 7f9ed3fff640 1 --2- 172.21.15.114:0/2403490464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13e31b0 0x55e5c1784220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.623+0000 7f9ed3fff640 1 -- 172.21.15.114:0/2403490464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13e31b0 msgr2=0x55e5c1784220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.623+0000 7f9ed3fff640 1 --2- 172.21.15.114:0/2403490464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13e31b0 0x55e5c1784220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.623+0000 7f9ed3fff640 1 -- 172.21.15.114:0/2403490464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13ba2f0 msgr2=0x55e5c1784760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.623+0000 7f9ed3fff640 1 --2- 172.21.15.114:0/2403490464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13ba2f0 0x55e5c1784760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.623+0000 7f9ed3fff640 1 -- 172.21.15.114:0/2403490464 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5c13fb210 con 0x55e5c13c24d0 2024-05-27T07:45:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.623+0000 7f9ed97fa640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9ec80b2950 0x7f9ec80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.623+0000 7f9ed3fff640 1 --2- 172.21.15.114:0/2403490464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13c24d0 0x55e5c1784ca0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9edc100820 tx=0x7f9ec4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbf8f69bbc2beb19 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.623+0000 7f9edb7fe640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9ec80b2950 0x7f9ec80b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.623+0000 7f9ed1ffb640 1 -- 172.21.15.114:0/2403490464 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4011050 con 0x55e5c13c24d0 2024-05-27T07:45:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.623+0000 7f9ed97fa640 1 -- 172.21.15.114:0/1198105407 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f9ed40e2890 con 0x7f9edc109030 2024-05-27T07:45:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.623+0000 7f9ed1ffb640 1 -- 172.21.15.114:0/2403490464 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ec40111f0 con 0x55e5c13c24d0 2024-05-27T07:45:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.623+0000 7f9ed1ffb640 1 -- 172.21.15.114:0/2403490464 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec40047a0 con 0x55e5c13c24d0 2024-05-27T07:45:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.624+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2403490464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13c24d0 msgr2=0x55e5c1784ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.624+0000 7f9edb7fe640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9ec80b2950 0x7f9ec80b4e60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc005eb0 tx=0x7f9ecc02c040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.624+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2403490464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13c24d0 0x55e5c1784ca0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9edc100820 tx=0x7f9ec4004730 comp rx=0 tx=0).stop 2024-05-27T07:45:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.624+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2403490464 shutdown_connections 2024-05-27T07:45:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.624+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2403490464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13c24d0 0x55e5c1784ca0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.624+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2403490464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13ba2f0 0x55e5c1784760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.624+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2403490464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13e31b0 0x55e5c1784220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.624+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2403490464 >> 172.21.15.114:0/2403490464 conn(0x55e5c13d7300 msgr2=0x55e5c1780b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:56.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.624+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2403490464 shutdown_connections 2024-05-27T07:45:56.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.624+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2403490464 wait complete. 2024-05-27T07:45:56.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.625+0000 7f9ee8c00740 1 -- start start 2024-05-27T07:45:56.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.625+0000 7f9ee8c00740 1 Processor -- start 2024-05-27T07:45:56.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.625+0000 7f9ee8c00740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13ba2f0 0x55e5c1792690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.625+0000 7f9ee8c00740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13c24d0 0x55e5c1792bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.625+0000 7f9ee8c00740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13e31b0 0x55e5c1790b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.625+0000 7f9ee8c00740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e5c13cc790 con 0x55e5c13ba2f0 2024-05-27T07:45:56.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.625+0000 7f9ee8c00740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e5c16a2890 con 0x55e5c13e31b0 2024-05-27T07:45:56.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.625+0000 7f9ee8c00740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e5c1793710 con 0x55e5c13c24d0 2024-05-27T07:45:56.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.625+0000 7f9ee8c00740 10 client.0 _ll_register_callbacks cb 0x55e5c13faa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:56.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9edb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13ba2f0 0x55e5c1792690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9edb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13ba2f0 0x55e5c1792690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44416/0 (socket says 172.21.15.114:44416) 2024-05-27T07:45:56.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9edb7fe640 1 -- 172.21.15.114:0/2430417557 learned_addr learned my addr 172.21.15.114:0/2430417557 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:56.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9ed3fff640 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13c24d0 0x55e5c1792bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9edbfff640 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13e31b0 0x55e5c1790b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9edb7fe640 1 -- 172.21.15.114:0/2430417557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13c24d0 msgr2=0x55e5c1792bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:56.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9edb7fe640 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13c24d0 0x55e5c1792bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9edb7fe640 1 -- 172.21.15.114:0/2430417557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13e31b0 msgr2=0x55e5c1790b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:56.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9edb7fe640 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13e31b0 0x55e5c1790b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9edb7fe640 1 -- 172.21.15.114:0/2430417557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5c14c7620 con 0x55e5c13ba2f0 2024-05-27T07:45:56.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9ed3fff640 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13c24d0 0x55e5c1792bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:56.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9edb7fe640 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13ba2f0 0x55e5c1792690 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009830 tx=0x7f9ecc02a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60fd3fd7a116bd7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:56.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9ebaffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:56.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9ebaffd640 1 -- 172.21.15.114:0/2430417557 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc037040 con 0x55e5c13ba2f0 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9ebaffd640 1 -- 172.21.15.114:0/2430417557 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f9ecc0327f0 con 0x55e5c13ba2f0 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2430417557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ec40111f0 con 0x55e5c13ba2f0 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9ebaffd640 1 -- 172.21.15.114:0/2430417557 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9ecc058050 con 0x55e5c13ba2f0 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9ee8c00740 10 client.16839 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.626+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2430417557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e5c179fa20 con 0x55e5c13ba2f0 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.627+0000 7f9ebaffd640 1 -- 172.21.15.114:0/2430417557 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ec40111f0 con 0x55e5c13ba2f0 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.627+0000 7f9ebaffd640 1 -- 172.21.15.114:0/2430417557 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f9ecc003c40 con 0x55e5c13ba2f0 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.627+0000 7f9ee8c00740 10 client.16839 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.627+0000 7f9ee8c00740 10 client.16839 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.627+0000 7f9ee8c00740 20 client.16839 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.627+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e5c13a1d80 0x55e5c17a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.627+0000 7f9ee8c00740 4 client.16839 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.627+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2430417557 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5c16f8b80 con 0x55e5c13a1d80 2024-05-27T07:45:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.627+0000 7f9ed3fff640 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e5c13a1d80 0x55e5c17a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.628+0000 7f9ebaffd640 10 client.16839 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.628+0000 7f9ed3fff640 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e5c13a1d80 0x55e5c17a2fb0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6d73e7e3f3887d server_cookie=7c1519792a245125 in_seq=0 out_seq=0 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.628+0000 7f9ebaffd640 1 -- 172.21.15.114:0/2430417557 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e5c16f8b80 con 0x55e5c13a1d80 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.628+0000 7f9ebaffd640 10 client.16839 handle_command_reply: tid=0 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.628+0000 7f9ee8c00740 1 client.16839 shutdown 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.629+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2430417557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13ba2f0 msgr2=0x55e5c1792690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.629+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13ba2f0 0x55e5c1792690 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009830 tx=0x7f9ecc02a340 comp rx=0 tx=0).stop 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.629+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2430417557 shutdown_connections 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.629+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e5c13a1d80 0x55e5c17a2fb0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.629+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13e31b0 0x55e5c1790b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.629+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13c24d0 0x55e5c1792bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.629+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2430417557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13ba2f0 0x55e5c1792690 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.629+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2430417557 >> 172.21.15.114:0/2430417557 conn(0x55e5c13d7300 msgr2=0x55e5c1786080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.629+0000 7f9ebaffd640 0 client.16839 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:56.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.629+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2430417557 shutdown_connections 2024-05-27T07:45:56.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.629+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2430417557 wait complete. 2024-05-27T07:45:56.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.629+0000 7f9ee8c00740 20 client.16839 trim_cache size 0 max 16384 2024-05-27T07:45:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.646+0000 7f9ee8c00740 1 Processor -- start 2024-05-27T07:45:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.646+0000 7f9ee8c00740 1 -- start start 2024-05-27T07:45:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.646+0000 7f9ee8c00740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13ba2f0 0x55e5c177f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.646+0000 7f9ee8c00740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13c24d0 0x55e5c178a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.646+0000 7f9ee8c00740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 0x55e5c178a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.646+0000 7f9ee8c00740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc1affa0 con 0x55e5c13e31b0 2024-05-27T07:45:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.646+0000 7f9ee8c00740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc1ad050 con 0x55e5c13c24d0 2024-05-27T07:45:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.646+0000 7f9ee8c00740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc1af200 con 0x55e5c13ba2f0 2024-05-27T07:45:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9edbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 0x55e5c178a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9edb7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13ba2f0 0x55e5c177f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9edbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 0x55e5c178a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44420/0 (socket says 172.21.15.114:44420) 2024-05-27T07:45:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9edbfff640 1 -- 172.21.15.114:0/2010502706 learned_addr learned my addr 172.21.15.114:0/2010502706 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9ed3fff640 1 --2- 172.21.15.114:0/2010502706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13c24d0 0x55e5c178a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9edbfff640 1 -- 172.21.15.114:0/2010502706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13ba2f0 msgr2=0x55e5c177f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9edbfff640 1 --2- 172.21.15.114:0/2010502706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13ba2f0 0x55e5c177f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9edbfff640 1 -- 172.21.15.114:0/2010502706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13c24d0 msgr2=0x55e5c178a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9edbfff640 1 --2- 172.21.15.114:0/2010502706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13c24d0 0x55e5c178a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9edbfff640 1 -- 172.21.15.114:0/2010502706 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5c1786fa0 con 0x55e5c13e31b0 2024-05-27T07:45:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9edbfff640 1 --2- 172.21.15.114:0/2010502706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 0x55e5c178a950 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9ed40e0230 tx=0x7f9ed4004890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b10488bc802f16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9ed17fa640 1 -- 172.21.15.114:0/2010502706 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed401a040 con 0x55e5c13e31b0 2024-05-27T07:45:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9ed17fa640 1 -- 172.21.15.114:0/2010502706 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ed40a97c0 con 0x55e5c13e31b0 2024-05-27T07:45:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.647+0000 7f9ed17fa640 1 -- 172.21.15.114:0/2010502706 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed40a5a50 con 0x55e5c13e31b0 2024-05-27T07:45:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.648+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2010502706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 msgr2=0x55e5c178a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.648+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2010502706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 0x55e5c178a950 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9ed40e0230 tx=0x7f9ed4004890 comp rx=0 tx=0).stop 2024-05-27T07:45:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.649+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2010502706 shutdown_connections 2024-05-27T07:45:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.649+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2010502706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 0x55e5c178a950 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.649+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2010502706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13c24d0 0x55e5c178a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.649+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2010502706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13ba2f0 0x55e5c177f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.649+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2010502706 >> 172.21.15.114:0/2010502706 conn(0x55e5c13d7300 msgr2=0x55e5c177f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.649+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2010502706 shutdown_connections 2024-05-27T07:45:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.649+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2010502706 wait complete. 2024-05-27T07:45:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.649+0000 7f9ee8c00740 1 -- start start 2024-05-27T07:45:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.649+0000 7f9ee8c00740 1 Processor -- start 2024-05-27T07:45:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9ee8c00740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13ba2f0 0x55e5c1793920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9ee8c00740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13c24d0 0x55e5c1793e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9ee8c00740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 0x55e5c17943a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9ee8c00740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc1aeb30 con 0x55e5c13e31b0 2024-05-27T07:45:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9ee8c00740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc1ad6c0 con 0x55e5c13c24d0 2024-05-27T07:45:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9ee8c00740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc1add90 con 0x55e5c13ba2f0 2024-05-27T07:45:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9ee8c00740 10 client.0 _ll_register_callbacks cb 0x7f9ed4009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9edbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 0x55e5c17943a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9edbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 0x55e5c17943a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44436/0 (socket says 172.21.15.114:44436) 2024-05-27T07:45:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9edbfff640 1 -- 172.21.15.114:0/2045571424 learned_addr learned my addr 172.21.15.114:0/2045571424 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9ed3fff640 1 --2- 172.21.15.114:0/2045571424 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13c24d0 0x55e5c1793e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9edb7fe640 1 --2- 172.21.15.114:0/2045571424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13ba2f0 0x55e5c1793920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9edbfff640 1 -- 172.21.15.114:0/2045571424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13ba2f0 msgr2=0x55e5c1793920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9edbfff640 1 --2- 172.21.15.114:0/2045571424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13ba2f0 0x55e5c1793920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9edbfff640 1 -- 172.21.15.114:0/2045571424 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13c24d0 msgr2=0x55e5c1793e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9edbfff640 1 --2- 172.21.15.114:0/2045571424 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13c24d0 0x55e5c1793e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9edbfff640 1 -- 172.21.15.114:0/2045571424 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5c1809670 con 0x55e5c13e31b0 2024-05-27T07:45:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9edbfff640 1 --2- 172.21.15.114:0/2045571424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 0x55e5c17943a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9ed40a94c0 tx=0x7f9ed40b2620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae3b2695a1da679f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9ebaffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.650+0000 7f9ebaffd640 1 -- 172.21.15.114:0/2045571424 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed401a070 con 0x55e5c13e31b0 2024-05-27T07:45:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2045571424 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ed40a97c0 con 0x55e5c13e31b0 2024-05-27T07:45:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9ebaffd640 1 -- 172.21.15.114:0/2045571424 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f9ed40b2390 con 0x55e5c13e31b0 2024-05-27T07:45:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9ebaffd640 1 -- 172.21.15.114:0/2045571424 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9ed4091080 con 0x55e5c13e31b0 2024-05-27T07:45:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9ee8c00740 10 client.16845 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2045571424 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e5c180a7a0 con 0x55e5c13e31b0 2024-05-27T07:45:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9ebaffd640 1 -- 172.21.15.114:0/2045571424 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ed40a97c0 con 0x55e5c13e31b0 2024-05-27T07:45:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9ebaffd640 1 -- 172.21.15.114:0/2045571424 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f9ed408fce0 con 0x55e5c13e31b0 2024-05-27T07:45:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9ee8c00740 10 client.16845 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9ee8c00740 10 client.16845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9ee8c00740 20 client.16845 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2045571424 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e5c13771c0 0x55e5c180ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9ee8c00740 4 client.16845 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:45:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2045571424 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e5c16f2360 con 0x55e5c13771c0 2024-05-27T07:45:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.651+0000 7f9edb7fe640 1 --2- 172.21.15.114:0/2045571424 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e5c13771c0 0x55e5c180ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.652+0000 7f9ebaffd640 10 client.16845 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:56.652+0000 7f9edb7fe640 1 --2- 172.21.15.114:0/2045571424 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e5c13771c0 0x55e5c180ce60 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8416368a8c56f691 server_cookie=eac5fc732d6d9532 in_seq=0 out_seq=0 2024-05-27T07:45:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.010+0000 7f9ebaffd640 1 -- 172.21.15.114:0/2045571424 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55e5c16f2360 con 0x55e5c13771c0 2024-05-27T07:45:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.010+0000 7f9ebaffd640 10 client.16845 handle_command_reply: tid=0 2024-05-27T07:45:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.010+0000 7f9ee8c00740 1 client.16845 shutdown 2024-05-27T07:45:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.010+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2045571424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 msgr2=0x55e5c17943a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.010+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2045571424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 0x55e5c17943a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9ed40a94c0 tx=0x7f9ed40b2620 comp rx=0 tx=0).stop 2024-05-27T07:45:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.010+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2045571424 shutdown_connections 2024-05-27T07:45:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.010+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2045571424 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e5c13771c0 0x55e5c180ce60 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.010+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2045571424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e5c13e31b0 0x55e5c17943a0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.010+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2045571424 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e5c13c24d0 0x55e5c1793e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.010+0000 7f9ee8c00740 1 --2- 172.21.15.114:0/2045571424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5c13ba2f0 0x55e5c1793920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.010+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2045571424 >> 172.21.15.114:0/2045571424 conn(0x55e5c13d7300 msgr2=0x55e5c1780b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.010+0000 7f9ebaffd640 0 client.16845 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.011+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2045571424 shutdown_connections 2024-05-27T07:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.011+0000 7f9ee8c00740 1 -- 172.21.15.114:0/2045571424 wait complete. 2024-05-27T07:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.011+0000 7f9ee8c00740 20 client.16845 trim_cache size 0 max 16384 2024-05-27T07:45:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:45:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:45:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:45:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.011+0000 7f9ee1c6e640 1 -- 172.21.15.114:0/1198105407 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9ec80b2950 msgr2=0x7f9ec80b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.011+0000 7f9ee1c6e640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9ec80b2950 0x7f9ec80b4e60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc005eb0 tx=0x7f9ecc02c040 comp rx=0 tx=0).stop 2024-05-27T07:45:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.011+0000 7f9ee1c6e640 1 -- 172.21.15.114:0/1198105407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc109030 msgr2=0x7f9edc19dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.011+0000 7f9ee1c6e640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc109030 0x7f9edc19dfc0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400ef00 tx=0x7f9ed400c740 comp rx=0 tx=0).stop 2024-05-27T07:45:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.012+0000 7f9ee1c6e640 1 -- 172.21.15.114:0/1198105407 shutdown_connections 2024-05-27T07:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.012+0000 7f9ee1c6e640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9ec80b2950 0x7f9ec80b4e60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.012+0000 7f9ee1c6e640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc109030 0x7f9edc19dfc0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.012+0000 7f9ee1c6e640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9edc108b90 0x7f9edc199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.012+0000 7f9ee1c6e640 1 --2- 172.21.15.114:0/1198105407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9edc1032e0 0x7f9edc1997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.012+0000 7f9ee1c6e640 1 -- 172.21.15.114:0/1198105407 >> 172.21.15.114:0/1198105407 conn(0x7f9edc0782d0 msgr2=0x7f9edc1008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.012+0000 7f9ee1c6e640 1 -- 172.21.15.114:0/1198105407 shutdown_connections 2024-05-27T07:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:45:57.012+0000 7f9ee1c6e640 1 -- 172.21.15.114:0/1198105407 wait complete. 2024-05-27T07:45:57.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 617934 13.03 MB/sec execute 129 sec latency 268.917 ms 2024-05-27T07:45:57.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.428+0000 7fc8912d7700 1 -- 172.21.15.114:0/3926158025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c108750 msgr2=0x7fc88c1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:57.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.428+0000 7fc8912d7700 1 --2- 172.21.15.114:0/3926158025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c108750 0x7fc88c1100b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc88400b0d0 tx=0x7fc88400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.428+0000 7fc8912d7700 1 -- 172.21.15.114:0/3926158025 shutdown_connections 2024-05-27T07:45:57.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.428+0000 7fc8912d7700 1 --2- 172.21.15.114:0/3926158025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c108750 0x7fc88c1100b0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.428+0000 7fc8912d7700 1 --2- 172.21.15.114:0/3926158025 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc88c1034d0 0x7fc88c1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.428+0000 7fc8912d7700 1 --2- 172.21.15.114:0/3926158025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c107e00 0x7fc88c108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.428+0000 7fc8912d7700 1 -- 172.21.15.114:0/3926158025 >> 172.21.15.114:0/3926158025 conn(0x7fc88c0fef20 msgr2=0x7fc88c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:57.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.428+0000 7fc8912d7700 1 -- 172.21.15.114:0/3926158025 shutdown_connections 2024-05-27T07:45:57.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.428+0000 7fc8912d7700 1 -- 172.21.15.114:0/3926158025 wait complete. 2024-05-27T07:45:57.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.428+0000 7fc8912d7700 1 Processor -- start 2024-05-27T07:45:57.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.429+0000 7fc8912d7700 1 -- start start 2024-05-27T07:45:57.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.429+0000 7fc8912d7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc88c1034d0 0x7fc88c19d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.429+0000 7fc8912d7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c107e00 0x7fc88c19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.429+0000 7fc8912d7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c108750 0x7fc88c1a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.429+0000 7fc8912d7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c112270 con 0x7fc88c107e00 2024-05-27T07:45:57.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.429+0000 7fc8912d7700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1120f0 con 0x7fc88c1034d0 2024-05-27T07:45:57.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.429+0000 7fc8912d7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1123f0 con 0x7fc88c108750 2024-05-27T07:45:57.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.429+0000 7fc88a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c107e00 0x7fc88c19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.429+0000 7fc88a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c107e00 0x7fc88c19d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44442/0 (socket says 172.21.15.114:44442) 2024-05-27T07:45:57.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.429+0000 7fc88a7fc700 1 -- 172.21.15.114:0/493194466 learned_addr learned my addr 172.21.15.114:0/493194466 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:57.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.429+0000 7fc88b7fe700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c108750 0x7fc88c1a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.429+0000 7fc88affd700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc88c1034d0 0x7fc88c19d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.430+0000 7fc88b7fe700 1 -- 172.21.15.114:0/493194466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc88c1034d0 msgr2=0x7fc88c19d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:57.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.430+0000 7fc88b7fe700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc88c1034d0 0x7fc88c19d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.430+0000 7fc88b7fe700 1 -- 172.21.15.114:0/493194466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c107e00 msgr2=0x7fc88c19d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:57.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.430+0000 7fc88b7fe700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c107e00 0x7fc88c19d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.430+0000 7fc88b7fe700 1 -- 172.21.15.114:0/493194466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c1a1ef0 con 0x7fc88c108750 2024-05-27T07:45:57.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.430+0000 7fc88b7fe700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c108750 0x7fc88c1a1850 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc88400b0d0 tx=0x7fc884007a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:57.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.430+0000 7fc883fff700 1 -- 172.21.15.114:0/493194466 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88401f070 con 0x7fc88c108750 2024-05-27T07:45:57.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.430+0000 7fc8912d7700 1 -- 172.21.15.114:0/493194466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc88c1a2180 con 0x7fc88c108750 2024-05-27T07:45:57.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.430+0000 7fc883fff700 1 -- 172.21.15.114:0/493194466 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc884004c50 con 0x7fc88c108750 2024-05-27T07:45:57.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.430+0000 7fc883fff700 1 -- 172.21.15.114:0/493194466 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88401b430 con 0x7fc88c108750 2024-05-27T07:45:57.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.430+0000 7fc8912d7700 1 -- 172.21.15.114:0/493194466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc88c1a9b00 con 0x7fc88c108750 2024-05-27T07:45:57.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.433+0000 7fc883fff700 1 -- 172.21.15.114:0/493194466 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fc884022070 con 0x7fc88c108750 2024-05-27T07:45:57.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.434+0000 7fc883fff700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc8780b29f0 0x7fc8780b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.434+0000 7fc88affd700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc8780b29f0 0x7fc8780b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.434+0000 7fc89fcecb80 1 Processor -- start 2024-05-27T07:45:57.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.434+0000 7fc89fcecb80 1 -- start start 2024-05-27T07:45:57.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.434+0000 7fc883fff700 1 -- 172.21.15.114:0/493194466 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7fc88400f020 con 0x7fc88c108750 2024-05-27T07:45:57.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc88affd700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc8780b29f0 0x7fc8780b4f00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc88c19e0b0 tx=0x7fc874009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:57.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc89fcecb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d19be40 0x55a45d31f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc89fcecb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d31fc50 0x55a45d319a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc89fcecb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 0x55a45d31a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc89fcecb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a45d248c90 con 0x55a45d31fc50 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc89fcecb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a45d238830 con 0x55a45d19be40 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc89fcecb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a45d23c870 con 0x55a45d319f80 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc88a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d31fc50 0x55a45d319a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc88a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d31fc50 0x55a45d319a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44452/0 (socket says 172.21.15.114:44452) 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc88a7fc700 1 -- 172.21.15.114:0/1233820979 learned_addr learned my addr 172.21.15.114:0/1233820979 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc88b7fe700 1 --2- 172.21.15.114:0/1233820979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 0x55a45d31a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc88affd700 1 --2- 172.21.15.114:0/1233820979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d19be40 0x55a45d31f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc88a7fc700 1 -- 172.21.15.114:0/1233820979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 msgr2=0x55a45d31a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc88a7fc700 1 --2- 172.21.15.114:0/1233820979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 0x55a45d31a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc88a7fc700 1 -- 172.21.15.114:0/1233820979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d19be40 msgr2=0x55a45d31f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:57.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc88a7fc700 1 --2- 172.21.15.114:0/1233820979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d19be40 0x55a45d31f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc88a7fc700 1 -- 172.21.15.114:0/1233820979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a45d1888d0 con 0x55a45d31fc50 2024-05-27T07:45:57.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.435+0000 7fc88a7fc700 1 --2- 172.21.15.114:0/1233820979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d31fc50 0x55a45d319a40 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009d30 tx=0x7fc87c011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c5832e253a2a09 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:57.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.436+0000 7fc8817fa700 1 -- 172.21.15.114:0/1233820979 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c004a30 con 0x55a45d31fc50 2024-05-27T07:45:57.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.436+0000 7fc8817fa700 1 -- 172.21.15.114:0/1233820979 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc87c0054d0 con 0x55a45d31fc50 2024-05-27T07:45:57.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.436+0000 7fc8817fa700 1 -- 172.21.15.114:0/1233820979 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c019970 con 0x55a45d31fc50 2024-05-27T07:45:57.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.437+0000 7fc89fcecb80 1 -- 172.21.15.114:0/1233820979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d31fc50 msgr2=0x55a45d319a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:57.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.437+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/1233820979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d31fc50 0x55a45d319a40 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009d30 tx=0x7fc87c011720 comp rx=0 tx=0).stop 2024-05-27T07:45:57.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.437+0000 7fc89fcecb80 1 -- 172.21.15.114:0/1233820979 shutdown_connections 2024-05-27T07:45:57.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.437+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/1233820979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 0x55a45d31a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.437+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/1233820979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d19be40 0x55a45d31f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.437+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/1233820979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d31fc50 0x55a45d319a40 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.437+0000 7fc89fcecb80 1 -- 172.21.15.114:0/1233820979 >> 172.21.15.114:0/1233820979 conn(0x55a45d1f6af0 msgr2=0x55a45d314640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:57.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.437+0000 7fc89fcecb80 1 -- 172.21.15.114:0/1233820979 shutdown_connections 2024-05-27T07:45:57.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.437+0000 7fc89fcecb80 1 -- 172.21.15.114:0/1233820979 wait complete. 2024-05-27T07:45:57.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.437+0000 7fc89fcecb80 1 -- start start 2024-05-27T07:45:57.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.437+0000 7fc89fcecb80 1 Processor -- start 2024-05-27T07:45:57.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.437+0000 7fc89fcecb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d19be40 0x55a45d323360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc89fcecb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d319f80 0x55a45d3238a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc89fcecb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d31fc50 0x55a45d323de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc89fcecb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a45d13efa0 con 0x55a45d319f80 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc89fcecb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a45d1da7c0 con 0x55a45d31fc50 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc89fcecb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a45d326740 con 0x55a45d19be40 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc89fcecb80 10 client.0 _ll_register_callbacks cb 0x55a45d1d3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc88affd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d19be40 0x55a45d323360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc88a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d319f80 0x55a45d3238a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc88affd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d19be40 0x55a45d323360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59480/0 (socket says 172.21.15.114:59480) 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc88affd700 1 -- 172.21.15.114:0/2113696723 learned_addr learned my addr 172.21.15.114:0/2113696723 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc88b7fe700 1 --2- 172.21.15.114:0/2113696723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d31fc50 0x55a45d323de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc88a7fc700 1 -- 172.21.15.114:0/2113696723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d19be40 msgr2=0x55a45d323360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc88a7fc700 1 --2- 172.21.15.114:0/2113696723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d19be40 0x55a45d323360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc88a7fc700 1 -- 172.21.15.114:0/2113696723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d31fc50 msgr2=0x55a45d323de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc88a7fc700 1 --2- 172.21.15.114:0/2113696723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d31fc50 0x55a45d323de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc88a7fc700 1 -- 172.21.15.114:0/2113696723 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a45d1d7690 con 0x55a45d319f80 2024-05-27T07:45:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc88affd700 1 --2- 172.21.15.114:0/2113696723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d19be40 0x55a45d323360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:45:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc88a7fc700 1 --2- 172.21.15.114:0/2113696723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d319f80 0x55a45d3238a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc87c01b040 tx=0x7fc87c004980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c51a99dd9693d26 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc8727fc700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:45:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc8727fc700 1 -- 172.21.15.114:0/2113696723 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c029030 con 0x55a45d319f80 2024-05-27T07:45:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc8727fc700 1 -- 172.21.15.114:0/2113696723 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7fc87c005680 con 0x55a45d319f80 2024-05-27T07:45:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.438+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2113696723 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc87c0054d0 con 0x55a45d319f80 2024-05-27T07:45:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.439+0000 7fc8727fc700 1 -- 172.21.15.114:0/2113696723 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc87c033050 con 0x55a45d319f80 2024-05-27T07:45:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.439+0000 7fc89fcecb80 10 client.16857 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.439+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2113696723 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a45d331170 con 0x55a45d319f80 2024-05-27T07:45:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.439+0000 7fc8727fc700 1 -- 172.21.15.114:0/2113696723 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc87c0054d0 con 0x55a45d319f80 2024-05-27T07:45:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.439+0000 7fc8727fc700 1 -- 172.21.15.114:0/2113696723 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7fc87c025470 con 0x55a45d319f80 2024-05-27T07:45:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.439+0000 7fc89fcecb80 10 client.16857 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.439+0000 7fc89fcecb80 10 client.16857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.439+0000 7fc89fcecb80 20 client.16857 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.439+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/2113696723 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a45d3336a0 0x55a45d335ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.439+0000 7fc89fcecb80 4 client.16857 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:45:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.439+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2113696723 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a45d2e59f0 con 0x55a45d3336a0 2024-05-27T07:45:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.439+0000 7fc88affd700 1 --2- 172.21.15.114:0/2113696723 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a45d3336a0 0x55a45d335ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.440+0000 7fc88affd700 1 --2- 172.21.15.114:0/2113696723 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a45d3336a0 0x55a45d335ac0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86842fbc86a59783 server_cookie=62fcd994006f4e90 in_seq=0 out_seq=0 2024-05-27T07:45:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.440+0000 7fc8727fc700 10 client.16857 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:57.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc8727fc700 1 -- 172.21.15.114:0/2113696723 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a45d2e59f0 con 0x55a45d3336a0 2024-05-27T07:45:57.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc8727fc700 10 client.16857 handle_command_reply: tid=0 2024-05-27T07:45:57.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc89fcecb80 1 client.16857 shutdown 2024-05-27T07:45:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2113696723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d319f80 msgr2=0x55a45d3238a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/2113696723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d319f80 0x55a45d3238a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc87c01b040 tx=0x7fc87c004980 comp rx=0 tx=0).stop 2024-05-27T07:45:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2113696723 shutdown_connections 2024-05-27T07:45:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/2113696723 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a45d3336a0 0x55a45d335ac0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/2113696723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d31fc50 0x55a45d323de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/2113696723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d19be40 0x55a45d323360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/2113696723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d319f80 0x55a45d3238a0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2113696723 >> 172.21.15.114:0/2113696723 conn(0x55a45d1f6af0 msgr2=0x55a45d318a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc8727fc700 0 client.16857 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2113696723 shutdown_connections 2024-05-27T07:45:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2113696723 wait complete. 2024-05-27T07:45:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.441+0000 7fc89fcecb80 20 client.16857 trim_cache size 0 max 16384 2024-05-27T07:45:57.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.464+0000 7fc89fcecb80 1 Processor -- start 2024-05-27T07:45:57.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.464+0000 7fc89fcecb80 1 -- start start 2024-05-27T07:45:57.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.464+0000 7fc89fcecb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d19be40 0x55a45d319a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.464+0000 7fc89fcecb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 0x55a45d322c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.464+0000 7fc89fcecb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d31fc50 0x55a45d32f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.464+0000 7fc89fcecb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1b3750 con 0x55a45d19be40 2024-05-27T07:45:57.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.464+0000 7fc89fcecb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1b0800 con 0x55a45d31fc50 2024-05-27T07:45:57.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.464+0000 7fc89fcecb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1b29b0 con 0x55a45d319f80 2024-05-27T07:45:57.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.464+0000 7fc88affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d19be40 0x55a45d319a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.464+0000 7fc88affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d19be40 0x55a45d319a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44460/0 (socket says 172.21.15.114:44460) 2024-05-27T07:45:57.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.464+0000 7fc88affd700 1 -- 172.21.15.114:0/1821831761 learned_addr learned my addr 172.21.15.114:0/1821831761 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:57.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.464+0000 7fc88b7fe700 1 --2- 172.21.15.114:0/1821831761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d31fc50 0x55a45d32f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.464+0000 7fc88a7fc700 1 --2- 172.21.15.114:0/1821831761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 0x55a45d322c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.465+0000 7fc88affd700 1 -- 172.21.15.114:0/1821831761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 msgr2=0x55a45d322c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:57.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.465+0000 7fc88affd700 1 --2- 172.21.15.114:0/1821831761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 0x55a45d322c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.465+0000 7fc88affd700 1 -- 172.21.15.114:0/1821831761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d31fc50 msgr2=0x55a45d32f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.465+0000 7fc88affd700 1 --2- 172.21.15.114:0/1821831761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d31fc50 0x55a45d32f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.465+0000 7fc88affd700 1 -- 172.21.15.114:0/1821831761 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a45d3218d0 con 0x55a45d19be40 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.465+0000 7fc88affd700 1 --2- 172.21.15.114:0/1821831761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d19be40 0x55a45d319a40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc874011a40 tx=0x7fc874014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee6a392145fb19c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.465+0000 7fc880ff9700 1 -- 172.21.15.114:0/1821831761 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc874003e10 con 0x55a45d19be40 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.465+0000 7fc880ff9700 1 -- 172.21.15.114:0/1821831761 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc874004ce0 con 0x55a45d19be40 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.465+0000 7fc880ff9700 1 -- 172.21.15.114:0/1821831761 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc874032410 con 0x55a45d19be40 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.466+0000 7fc89fcecb80 1 -- 172.21.15.114:0/1821831761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d19be40 msgr2=0x55a45d319a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.466+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/1821831761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d19be40 0x55a45d319a40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc874011a40 tx=0x7fc874014000 comp rx=0 tx=0).stop 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.466+0000 7fc89fcecb80 1 -- 172.21.15.114:0/1821831761 shutdown_connections 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.466+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/1821831761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d31fc50 0x55a45d32f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.466+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/1821831761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d19be40 0x55a45d319a40 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.466+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/1821831761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 0x55a45d322c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.466+0000 7fc89fcecb80 1 -- 172.21.15.114:0/1821831761 >> 172.21.15.114:0/1821831761 conn(0x55a45d1f6af0 msgr2=0x55a45d323320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.466+0000 7fc89fcecb80 1 -- 172.21.15.114:0/1821831761 shutdown_connections 2024-05-27T07:45:57.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.466+0000 7fc89fcecb80 1 -- 172.21.15.114:0/1821831761 wait complete. 2024-05-27T07:45:57.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.466+0000 7fc89fcecb80 1 -- start start 2024-05-27T07:45:57.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc89fcecb80 1 Processor -- start 2024-05-27T07:45:57.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc89fcecb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d19be40 0x55a45d32bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc89fcecb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 0x55a45d32c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc89fcecb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d31fc50 0x55a45d33a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc89fcecb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1b22e0 con 0x55a45d31fc50 2024-05-27T07:45:57.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc89fcecb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1b0e70 con 0x55a45d19be40 2024-05-27T07:45:57.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc89fcecb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1b1540 con 0x55a45d319f80 2024-05-27T07:45:57.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc88affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d19be40 0x55a45d32bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc88a7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 0x55a45d32c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc89fcecb80 10 client.0 _ll_register_callbacks cb 0x55a45d1d3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:45:57.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc88affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d19be40 0x55a45d32bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:58696/0 (socket says 172.21.15.114:58696) 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc88affd700 1 -- 172.21.15.114:0/2624048586 learned_addr learned my addr 172.21.15.114:0/2624048586 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc88b7fe700 1 --2- 172.21.15.114:0/2624048586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d31fc50 0x55a45d33a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc88affd700 1 -- 172.21.15.114:0/2624048586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 msgr2=0x55a45d32c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc88affd700 1 --2- 172.21.15.114:0/2624048586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 0x55a45d32c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc88affd700 1 -- 172.21.15.114:0/2624048586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d31fc50 msgr2=0x55a45d33a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc88affd700 1 --2- 172.21.15.114:0/2624048586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d31fc50 0x55a45d33a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.467+0000 7fc88affd700 1 -- 172.21.15.114:0/2624048586 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a45d33a8a0 con 0x55a45d19be40 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.468+0000 7fc88affd700 1 --2- 172.21.15.114:0/2624048586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d19be40 0x55a45d32bbd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc8740117d0 tx=0x7fc874003f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf31ef94520b0fbc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.468+0000 7fc8727fc700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.468+0000 7fc8727fc700 1 -- 172.21.15.114:0/2624048586 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc874005690 con 0x55a45d19be40 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.468+0000 7fc8727fc700 1 -- 172.21.15.114:0/2624048586 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7fc87402f040 con 0x55a45d19be40 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.468+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2624048586 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc874004ce0 con 0x55a45d19be40 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.468+0000 7fc8727fc700 1 -- 172.21.15.114:0/2624048586 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc874043050 con 0x55a45d19be40 2024-05-27T07:45:57.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.468+0000 7fc89fcecb80 10 client.26351 fetch_fsmap learned FSMap version 37 2024-05-27T07:45:57.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.468+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2624048586 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a45d39d800 con 0x55a45d19be40 2024-05-27T07:45:57.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.468+0000 7fc8727fc700 1 -- 172.21.15.114:0/2624048586 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc874004ce0 con 0x55a45d19be40 2024-05-27T07:45:57.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.469+0000 7fc8727fc700 1 -- 172.21.15.114:0/2624048586 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7fc87401d9b0 con 0x55a45d19be40 2024-05-27T07:45:57.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.469+0000 7fc89fcecb80 10 client.26351 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:45:57.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.469+0000 7fc89fcecb80 10 client.26351 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:45:57.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.469+0000 7fc89fcecb80 20 client.26351 populate_metadata read hostname 'smithi114' 2024-05-27T07:45:57.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.469+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/2624048586 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a45d39de60 0x55a45d3a0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:45:57.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.469+0000 7fc89fcecb80 4 client.26351 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:45:57.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.469+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2624048586 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a45d24abe0 con 0x55a45d39de60 2024-05-27T07:45:57.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.469+0000 7fc88a7fc700 1 --2- 172.21.15.114:0/2624048586 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a45d39de60 0x55a45d3a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:45:57.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.469+0000 7fc8727fc700 10 client.26351 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:57.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.469+0000 7fc88a7fc700 1 --2- 172.21.15.114:0/2624048586 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a45d39de60 0x55a45d3a0280 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ddb477376f47ad1 server_cookie=4fe6bcec2b60b8c6 in_seq=0 out_seq=0 2024-05-27T07:45:57.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc8727fc700 1 -- 172.21.15.114:0/2624048586 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a45d24abe0 con 0x55a45d39de60 2024-05-27T07:45:57.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc8727fc700 10 client.26351 handle_command_reply: tid=0 2024-05-27T07:45:57.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc89fcecb80 1 client.26351 shutdown 2024-05-27T07:45:57.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2624048586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d19be40 msgr2=0x55a45d32bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:57.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/2624048586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d19be40 0x55a45d32bbd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc8740117d0 tx=0x7fc874003f20 comp rx=0 tx=0).stop 2024-05-27T07:45:57.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2624048586 shutdown_connections 2024-05-27T07:45:57.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/2624048586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a45d31fc50 0x55a45d33a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/2624048586 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a45d39de60 0x55a45d3a0280 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/2624048586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a45d19be40 0x55a45d32bbd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc89fcecb80 1 --2- 172.21.15.114:0/2624048586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a45d319f80 0x55a45d32c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc8727fc700 0 client.26351 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:45:57.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2624048586 >> 172.21.15.114:0/2624048586 conn(0x55a45d1f6af0 msgr2=0x55a45d31d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:57.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2624048586 shutdown_connections 2024-05-27T07:45:57.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc89fcecb80 1 -- 172.21.15.114:0/2624048586 wait complete. 2024-05-27T07:45:57.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.471+0000 7fc89fcecb80 20 client.26351 trim_cache size 0 max 16384 2024-05-27T07:45:57.476 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:45:57.476 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:45:57.476 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:45:57.476 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:45:57.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.472+0000 7fc8912d7700 1 -- 172.21.15.114:0/493194466 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc8780b29f0 msgr2=0x7fc8780b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.472+0000 7fc8912d7700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc8780b29f0 0x7fc8780b4f00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc88c19e0b0 tx=0x7fc874009260 comp rx=0 tx=0).stop 2024-05-27T07:45:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.472+0000 7fc8912d7700 1 -- 172.21.15.114:0/493194466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c108750 msgr2=0x7fc88c1a1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:45:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.472+0000 7fc8912d7700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c108750 0x7fc88c1a1850 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc88400b0d0 tx=0x7fc884007a00 comp rx=0 tx=0).stop 2024-05-27T07:45:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.472+0000 7fc8912d7700 1 -- 172.21.15.114:0/493194466 shutdown_connections 2024-05-27T07:45:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.472+0000 7fc8912d7700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c108750 0x7fc88c1a1850 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.472+0000 7fc8912d7700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc8780b29f0 0x7fc8780b4f00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.472+0000 7fc8912d7700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc88c1034d0 0x7fc88c19d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.472+0000 7fc8912d7700 1 --2- 172.21.15.114:0/493194466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c107e00 0x7fc88c19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:45:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.472+0000 7fc8912d7700 1 -- 172.21.15.114:0/493194466 >> 172.21.15.114:0/493194466 conn(0x7fc88c0fef20 msgr2=0x7fc88c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:45:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.472+0000 7fc8912d7700 1 -- 172.21.15.114:0/493194466 shutdown_connections 2024-05-27T07:45:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:45:57.472+0000 7fc8912d7700 1 -- 172.21.15.114:0/493194466 wait complete. 2024-05-27T07:45:57.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:57 smithi114 ceph-mon[32038]: pgmap v196: 129 pgs: 129 active+clean; 89 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.1 MiB/s wr, 486 op/s 2024-05-27T07:45:57.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:57 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:45:57.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:57 smithi163 ceph-mon[36451]: pgmap v196: 129 pgs: 129 active+clean; 89 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.1 MiB/s wr, 486 op/s 2024-05-27T07:45:57.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:57 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:45:57.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:57 smithi186 ceph-mon[36647]: pgmap v196: 129 pgs: 129 active+clean; 89 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.1 MiB/s wr, 486 op/s 2024-05-27T07:45:57.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:57 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:45:58.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 619875 12.99 MB/sec execute 130 sec latency 16.754 ms 2024-05-27T07:45:58.257 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:45:58.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:45:58.545 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:45:59.251 INFO:tasks.workunit.client.0.smithi114.stdout: 1 622249 12.99 MB/sec execute 131 sec latency 55.820 ms 2024-05-27T07:45:59.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:45:59 smithi114 ceph-mon[32038]: pgmap v197: 129 pgs: 129 active+clean; 89 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 243 op/s 2024-05-27T07:45:59.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:45:59 smithi163 ceph-mon[36451]: pgmap v197: 129 pgs: 129 active+clean; 89 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 243 op/s 2024-05-27T07:45:59.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:45:59 smithi186 ceph-mon[36647]: pgmap v197: 129 pgs: 129 active+clean; 89 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 243 op/s 2024-05-27T07:46:00.251 INFO:tasks.workunit.client.0.smithi114.stdout: 1 625054 13.02 MB/sec execute 132 sec latency 93.158 ms 2024-05-27T07:46:00.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.315+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2280948301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e810f930 msgr2=0x7fd8e810fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:00.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.315+0000 7fd8ef9fc700 1 --2- 172.21.15.114:0/2280948301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e810f930 0x7fd8e810fd40 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000ca60 tx=0x7fd8e000cd70 comp rx=0 tx=0).stop 2024-05-27T07:46:00.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.316+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2280948301 shutdown_connections 2024-05-27T07:46:00.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.316+0000 7fd8ef9fc700 1 --2- 172.21.15.114:0/2280948301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8e8110280 0x7fd8e8117be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:00.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.316+0000 7fd8ef9fc700 1 --2- 172.21.15.114:0/2280948301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8e810b000 0x7fd8e810b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:00.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.316+0000 7fd8ef9fc700 1 --2- 172.21.15.114:0/2280948301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e810f930 0x7fd8e810fd40 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:00.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.316+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2280948301 >> 172.21.15.114:0/2280948301 conn(0x7fd8e806f4c0 msgr2=0x7fd8e806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:00.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.318+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2280948301 shutdown_connections 2024-05-27T07:46:00.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.318+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2280948301 wait complete. 2024-05-27T07:46:00.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.319+0000 7fd8ef9fc700 1 Processor -- start 2024-05-27T07:46:00.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.320+0000 7fd8ef9fc700 1 -- start start 2024-05-27T07:46:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.320+0000 7fd8ef9fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8e810b000 0x7fd8e81a9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.320+0000 7fd8ef9fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8e810f930 0x7fd8e81a9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.320+0000 7fd8ef9fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e8110280 0x7fd8e81a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.320+0000 7fd8ef9fc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e811cdc0 con 0x7fd8e8110280 2024-05-27T07:46:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.320+0000 7fd8ef9fc700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e811cc40 con 0x7fd8e810b000 2024-05-27T07:46:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.320+0000 7fd8ef9fc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e811cf40 con 0x7fd8e810f930 2024-05-27T07:46:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.322+0000 7fd8edf99700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e8110280 0x7fd8e81a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.322+0000 7fd8edf99700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e8110280 0x7fd8e81a36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44478/0 (socket says 172.21.15.114:44478) 2024-05-27T07:46:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.322+0000 7fd8edf99700 1 -- 172.21.15.114:0/2810721349 learned_addr learned my addr 172.21.15.114:0/2810721349 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.322+0000 7fd8ed798700 1 --2- 172.21.15.114:0/2810721349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8e810b000 0x7fd8e81a9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.322+0000 7fd8edf99700 1 -- 172.21.15.114:0/2810721349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8e810f930 msgr2=0x7fd8e81a9a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:46:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.322+0000 7fd8edf99700 1 --2- 172.21.15.114:0/2810721349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8e810f930 0x7fd8e81a9a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.322+0000 7fd8edf99700 1 -- 172.21.15.114:0/2810721349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8e810b000 msgr2=0x7fd8e81a9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.322+0000 7fd8edf99700 1 --2- 172.21.15.114:0/2810721349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8e810b000 0x7fd8e81a9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.322+0000 7fd8edf99700 1 -- 172.21.15.114:0/2810721349 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e81a3dd0 con 0x7fd8e8110280 2024-05-27T07:46:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.323+0000 7fd8edf99700 1 --2- 172.21.15.114:0/2810721349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e8110280 0x7fd8e81a36a0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd8d800b4a0 tx=0x7fd8d800b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:00.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.327+0000 7fd8de7fc700 1 -- 172.21.15.114:0/2810721349 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d801d070 con 0x7fd8e8110280 2024-05-27T07:46:00.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.327+0000 7fd8de7fc700 1 -- 172.21.15.114:0/2810721349 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8d8010e20 con 0x7fd8e8110280 2024-05-27T07:46:00.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.328+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2810721349 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e81a40c0 con 0x7fd8e8110280 2024-05-27T07:46:00.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.328+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2810721349 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e811a450 con 0x7fd8e8110280 2024-05-27T07:46:00.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.328+0000 7fd8de7fc700 1 -- 172.21.15.114:0/2810721349 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8016340 con 0x7fd8e8110280 2024-05-27T07:46:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.331+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2810721349 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8cc005330 con 0x7fd8e8110280 2024-05-27T07:46:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.333+0000 7fd8de7fc700 1 -- 172.21.15.114:0/2810721349 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fd8d8020070 con 0x7fd8e8110280 2024-05-27T07:46:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.335+0000 7fd8de7fc700 1 --2- 172.21.15.114:0/2810721349 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd8d40b2c00 0x7fd8d40b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.335+0000 7fd8de7fc700 1 -- 172.21.15.114:0/2810721349 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7fd8d80e5900 con 0x7fd8e8110280 2024-05-27T07:46:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.336+0000 7fd8ed798700 1 --2- 172.21.15.114:0/2810721349 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd8d40b2c00 0x7fd8d40b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.336+0000 7fd8ed798700 1 --2- 172.21.15.114:0/2810721349 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd8d40b2c00 0x7fd8d40b5110 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4006fd0 tx=0x7fd8e4009190 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:00.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.347+0000 7fd8de7fc700 1 -- 172.21.15.114:0/2810721349 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd8d8055df0 con 0x7fd8e8110280 2024-05-27T07:46:00.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.679+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2810721349 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8cc0058f0 con 0x7fd8e8110280 2024-05-27T07:46:00.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.680+0000 7fd8de7fc700 1 -- 172.21.15.114:0/2810721349 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11000 (secure 0 0 0) 0x7fd8d80acfe0 con 0x7fd8e8110280 2024-05-27T07:46:00.679 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:46:00.679 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":26227,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2936519309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2936519309},{"type":"v1","addr":"172.21.15.186:6835","nonce":2936519309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:45:32.379189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26305":{"gid":26305,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/684229511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":684229511},{"type":"v1","addr":"172.21.15.186:6837","nonce":684229511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:46:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.683+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2810721349 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd8d40b2c00 msgr2=0x7fd8d40b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.683+0000 7fd8ef9fc700 1 --2- 172.21.15.114:0/2810721349 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd8d40b2c00 0x7fd8d40b5110 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4006fd0 tx=0x7fd8e4009190 comp rx=0 tx=0).stop 2024-05-27T07:46:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.683+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2810721349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e8110280 msgr2=0x7fd8e81a36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.683+0000 7fd8ef9fc700 1 --2- 172.21.15.114:0/2810721349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e8110280 0x7fd8e81a36a0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd8d800b4a0 tx=0x7fd8d800b7b0 comp rx=0 tx=0).stop 2024-05-27T07:46:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.683+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2810721349 shutdown_connections 2024-05-27T07:46:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.683+0000 7fd8ef9fc700 1 --2- 172.21.15.114:0/2810721349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e8110280 0x7fd8e81a36a0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.683+0000 7fd8ef9fc700 1 --2- 172.21.15.114:0/2810721349 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd8d40b2c00 0x7fd8d40b5110 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.683+0000 7fd8ef9fc700 1 --2- 172.21.15.114:0/2810721349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8e810b000 0x7fd8e81a9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.683+0000 7fd8ef9fc700 1 --2- 172.21.15.114:0/2810721349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8e810f930 0x7fd8e81a9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.683+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2810721349 >> 172.21.15.114:0/2810721349 conn(0x7fd8e806f4c0 msgr2=0x7fd8e810da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:00.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.683+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2810721349 shutdown_connections 2024-05-27T07:46:00.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:00.683+0000 7fd8ef9fc700 1 -- 172.21.15.114:0/2810721349 wait complete. 2024-05-27T07:46:00.683 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 37 2024-05-27T07:46:01.251 INFO:tasks.workunit.client.0.smithi114.stdout: 1 626092 12.95 MB/sec execute 133 sec latency 64.337 ms 2024-05-27T07:46:01.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:46:01.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:01 smithi114 ceph-mon[32038]: pgmap v198: 129 pgs: 129 active+clean; 78 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 424 op/s 2024-05-27T07:46:01.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:01 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2810721349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:46:01.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:01 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:46:01.641 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:46:01.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:01 smithi163 ceph-mon[36451]: pgmap v198: 129 pgs: 129 active+clean; 78 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 424 op/s 2024-05-27T07:46:01.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:01 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2810721349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:46:01.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:01 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:46:01.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:01 smithi186 ceph-mon[36647]: pgmap v198: 129 pgs: 129 active+clean; 78 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 424 op/s 2024-05-27T07:46:01.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:01 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2810721349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:46:01.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:01 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:46:02.229 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f71334f6700 time 2024-05-27T07:46:01.956799+0000 2024-05-27T07:46:02.229 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:46:02.229 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:02.229 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7145d3bba6] 2024-05-27T07:46:02.229 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7145d3bdd1] 2024-05-27T07:46:02.229 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d9d049d23d] 2024-05-27T07:46:02.229 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d9d0488971] 2024-05-27T07:46:02.229 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d9d0486fa0] 2024-05-27T07:46:02.229 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d9d0487651] 2024-05-27T07:46:02.229 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d9d041c347] 2024-05-27T07:46:02.230 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d9d011887e] 2024-05-27T07:46:02.230 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d9d011de94] 2024-05-27T07:46:02.230 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d9d0128969] 2024-05-27T07:46:02.230 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d9d04c148c] 2024-05-27T07:46:02.230 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d9d04ca9cb] 2024-05-27T07:46:02.230 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 13: (Thread::entry_wrapper()+0x43) [0x7f7145d0a72d] 2024-05-27T07:46:02.230 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7145d0a749] 2024-05-27T07:46:02.230 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f71448ae1ca] 2024-05-27T07:46:02.230 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 16: clone() 2024-05-27T07:46:02.230 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:01.958+0000 7f71334f6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f71334f6700 time 2024-05-27T07:46:01.956799+0000 2024-05-27T07:46:02.231 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:46:02.231 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:02.231 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:02.232 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7145d3bba6] 2024-05-27T07:46:02.232 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7145d3bdd1] 2024-05-27T07:46:02.232 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d9d049d23d] 2024-05-27T07:46:02.232 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d9d0488971] 2024-05-27T07:46:02.232 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d9d0486fa0] 2024-05-27T07:46:02.232 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d9d0487651] 2024-05-27T07:46:02.232 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d9d041c347] 2024-05-27T07:46:02.232 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d9d011887e] 2024-05-27T07:46:02.232 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d9d011de94] 2024-05-27T07:46:02.232 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d9d0128969] 2024-05-27T07:46:02.232 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d9d04c148c] 2024-05-27T07:46:02.232 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d9d04ca9cb] 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 13: (Thread::entry_wrapper()+0x43) [0x7f7145d0a72d] 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7145d0a749] 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f71448ae1ca] 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 16: clone() 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: *** Caught signal (Aborted) ** 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: in thread 7f71334f6700 thread_name:md_log_replay 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71448b8d20] 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2: gsignal() 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 3: abort() 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7145d3bced] 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7145d3bdd1] 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d9d049d23d] 2024-05-27T07:46:02.233 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d9d0488971] 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d9d0486fa0] 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d9d0487651] 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d9d041c347] 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d9d011887e] 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d9d011de94] 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d9d0128969] 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d9d04c148c] 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d9d04ca9cb] 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 16: (Thread::entry_wrapper()+0x43) [0x7f7145d0a72d] 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7145d0a749] 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f71448ae1ca] 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 19: clone() 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:01.961+0000 7f71334f6700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:46:02.234 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: in thread 7f71334f6700 thread_name:md_log_replay 2024-05-27T07:46:02.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:02.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:02.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71448b8d20] 2024-05-27T07:46:02.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2: gsignal() 2024-05-27T07:46:02.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 3: abort() 2024-05-27T07:46:02.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7145d3bced] 2024-05-27T07:46:02.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7145d3bdd1] 2024-05-27T07:46:02.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d9d049d23d] 2024-05-27T07:46:02.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d9d0488971] 2024-05-27T07:46:02.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d9d0486fa0] 2024-05-27T07:46:02.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d9d0487651] 2024-05-27T07:46:02.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d9d041c347] 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d9d011887e] 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d9d011de94] 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d9d0128969] 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d9d04c148c] 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d9d04ca9cb] 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 16: (Thread::entry_wrapper()+0x43) [0x7f7145d0a72d] 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7145d0a749] 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f71448ae1ca] 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 19: clone() 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: -1> 2024-05-27T07:46:01.958+0000 7f71334f6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f71334f6700 time 2024-05-27T07:46:01.956799+0000 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:46:02.236 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7145d3bba6] 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7145d3bdd1] 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d9d049d23d] 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d9d0488971] 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d9d0486fa0] 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d9d0487651] 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d9d041c347] 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d9d011887e] 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d9d011de94] 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d9d0128969] 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d9d04c148c] 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d9d04ca9cb] 2024-05-27T07:46:02.237 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 13: (Thread::entry_wrapper()+0x43) [0x7f7145d0a72d] 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7145d0a749] 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f71448ae1ca] 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 16: clone() 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 0> 2024-05-27T07:46:01.961+0000 7f71334f6700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: in thread 7f71334f6700 thread_name:md_log_replay 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71448b8d20] 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2: gsignal() 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 3: abort() 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7145d3bced] 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7145d3bdd1] 2024-05-27T07:46:02.238 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d9d049d23d] 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d9d0488971] 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d9d0486fa0] 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d9d0487651] 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d9d041c347] 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d9d011887e] 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d9d011de94] 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d9d0128969] 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d9d04c148c] 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d9d04ca9cb] 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 16: (Thread::entry_wrapper()+0x43) [0x7f7145d0a72d] 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7145d0a749] 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f71448ae1ca] 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 19: clone() 2024-05-27T07:46:02.239 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: -1> 2024-05-27T07:46:01.958+0000 7f71334f6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f71334f6700 time 2024-05-27T07:46:01.956799+0000 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7145d3bba6] 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7145d3bdd1] 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d9d049d23d] 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d9d0488971] 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d9d0486fa0] 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d9d0487651] 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d9d041c347] 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d9d011887e] 2024-05-27T07:46:02.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d9d011de94] 2024-05-27T07:46:02.241 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d9d0128969] 2024-05-27T07:46:02.241 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d9d04c148c] 2024-05-27T07:46:02.241 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d9d04ca9cb] 2024-05-27T07:46:02.241 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 13: (Thread::entry_wrapper()+0x43) [0x7f7145d0a72d] 2024-05-27T07:46:02.241 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7145d0a749] 2024-05-27T07:46:02.241 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f71448ae1ca] 2024-05-27T07:46:02.241 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 16: clone() 2024-05-27T07:46:02.241 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:02.241 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 0> 2024-05-27T07:46:01.961+0000 7f71334f6700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:46:02.241 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: in thread 7f71334f6700 thread_name:md_log_replay 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71448b8d20] 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2: gsignal() 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 3: abort() 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7145d3bced] 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7145d3bdd1] 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d9d049d23d] 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d9d0488971] 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d9d0486fa0] 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d9d0487651] 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d9d041c347] 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d9d011887e] 2024-05-27T07:46:02.242 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d9d011de94] 2024-05-27T07:46:02.243 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d9d0128969] 2024-05-27T07:46:02.243 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d9d04c148c] 2024-05-27T07:46:02.243 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d9d04ca9cb] 2024-05-27T07:46:02.243 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 16: (Thread::entry_wrapper()+0x43) [0x7f7145d0a72d] 2024-05-27T07:46:02.243 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7145d0a749] 2024-05-27T07:46:02.243 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f71448ae1ca] 2024-05-27T07:46:02.243 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 19: clone() 2024-05-27T07:46:02.243 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:46:02.243 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[72721]: 2024-05-27T07:46:02.243 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:02 smithi186 podman[72882]: 2024-05-27 07:46:02.030022892 +0000 UTC m=+0.021200785 container died f49e98eb1e98f5920a6af0b617ac56a4156b3966668aa6fc0b9f0a49ecefd806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:46:02.250 INFO:tasks.workunit.client.0.smithi114.stdout: 1 627056 12.88 MB/sec execute 134 sec latency 104.704 ms 2024-05-27T07:46:02.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:02 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:46:02.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:02 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:46:02.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:02 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:46:02.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:02 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:46:02.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:02 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:46:02.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:02 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:46:02.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:02 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:46:02.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:02 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:46:02.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:02 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:46:02.959 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:02 smithi186 podman[72882]: 2024-05-27 07:46:02.68204354 +0000 UTC m=+0.673221425 container remove f49e98eb1e98f5920a6af0b617ac56a4156b3966668aa6fc0b9f0a49ecefd806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T07:46:02.960 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:02 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:46:03.228 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:02 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:46:03.228 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:02 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Consumed 3.116s CPU time. 2024-05-27T07:46:03.261 INFO:tasks.workunit.client.0.smithi114.stdout: 1 628987 12.89 MB/sec execute 135 sec latency 36.758 ms 2024-05-27T07:46:03.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:03 smithi114 ceph-mon[32038]: pgmap v199: 129 pgs: 129 active+clean; 77 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.3 MiB/s wr, 467 op/s 2024-05-27T07:46:03.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:03 smithi114 ceph-mon[32038]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T07:46:03.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:03 smithi163 ceph-mon[36451]: pgmap v199: 129 pgs: 129 active+clean; 77 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.3 MiB/s wr, 467 op/s 2024-05-27T07:46:03.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:03 smithi163 ceph-mon[36451]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T07:46:03.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:03 smithi186 ceph-mon[36647]: pgmap v199: 129 pgs: 129 active+clean; 77 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.3 MiB/s wr, 467 op/s 2024-05-27T07:46:03.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:03 smithi186 ceph-mon[36647]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T07:46:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.203+0000 7f4041c0a700 1 -- 172.21.15.114:0/1347625180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f403c103b90 msgr2=0x7f403c1080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.203+0000 7f4041c0a700 1 --2- 172.21.15.114:0/1347625180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f403c103b90 0x7f403c1080c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f403400b0a0 tx=0x7f403400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.203+0000 7f4041c0a700 1 -- 172.21.15.114:0/1347625180 shutdown_connections 2024-05-27T07:46:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.203+0000 7f4041c0a700 1 --2- 172.21.15.114:0/1347625180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f403c108690 0x7f403c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.203+0000 7f4041c0a700 1 --2- 172.21.15.114:0/1347625180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f403c103240 0x7f403c103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.203+0000 7f4041c0a700 1 --2- 172.21.15.114:0/1347625180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f403c103b90 0x7f403c1080c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.203+0000 7f4041c0a700 1 -- 172.21.15.114:0/1347625180 >> 172.21.15.114:0/1347625180 conn(0x7f403c0fedb0 msgr2=0x7f403c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.204+0000 7f4041c0a700 1 -- 172.21.15.114:0/1347625180 shutdown_connections 2024-05-27T07:46:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.204+0000 7f4041c0a700 1 -- 172.21.15.114:0/1347625180 wait complete. 2024-05-27T07:46:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.206+0000 7f4041c0a700 1 Processor -- start 2024-05-27T07:46:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.206+0000 7f4041c0a700 1 -- start start 2024-05-27T07:46:04.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.207+0000 7f4041c0a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f403c103240 0x7f403c1a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.207+0000 7f4041c0a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f403c103b90 0x7f403c1a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.207+0000 7f4041c0a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f403c108690 0x7f403c1a5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.208+0000 7f403affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f403c103b90 0x7f403c1a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.208+0000 7f403b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f403c103240 0x7f403c1a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.208+0000 7f403b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f403c103240 0x7f403c1a1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59520/0 (socket says 172.21.15.114:59520) 2024-05-27T07:46:04.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.208+0000 7f4041c0a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f403c11aab0 con 0x7f403c108690 2024-05-27T07:46:04.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.208+0000 7f403b7fe700 1 -- 172.21.15.114:0/3337217596 learned_addr learned my addr 172.21.15.114:0/3337217596 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:04.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.208+0000 7f4041c0a700 1 -- 172.21.15.114:0/3337217596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f403c11a930 con 0x7f403c103b90 2024-05-27T07:46:04.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.208+0000 7f4041c0a700 1 -- 172.21.15.114:0/3337217596 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f403c11ac30 con 0x7f403c103240 2024-05-27T07:46:04.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.209+0000 7f403bfff700 1 --2- 172.21.15.114:0/3337217596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f403c108690 0x7f403c1a5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.210+0000 7f403affd700 1 -- 172.21.15.114:0/3337217596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f403c103240 msgr2=0x7f403c1a1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:04.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.210+0000 7f403affd700 1 --2- 172.21.15.114:0/3337217596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f403c103240 0x7f403c1a1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.210+0000 7f403affd700 1 -- 172.21.15.114:0/3337217596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f403c108690 msgr2=0x7f403c1a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:04.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.210+0000 7f403affd700 1 --2- 172.21.15.114:0/3337217596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f403c108690 0x7f403c1a5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.210+0000 7f403affd700 1 -- 172.21.15.114:0/3337217596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f403c1a62f0 con 0x7f403c103b90 2024-05-27T07:46:04.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.211+0000 7f403affd700 1 --2- 172.21.15.114:0/3337217596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f403c103b90 0x7f403c1a19c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4034006070 tx=0x7f4034009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:04.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.214+0000 7f4038ff9700 1 -- 172.21.15.114:0/3337217596 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4034021070 con 0x7f403c103b90 2024-05-27T07:46:04.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.214+0000 7f4038ff9700 1 -- 172.21.15.114:0/3337217596 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4034007e30 con 0x7f403c103b90 2024-05-27T07:46:04.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.214+0000 7f4041c0a700 1 -- 172.21.15.114:0/3337217596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f403c1a6580 con 0x7f403c103b90 2024-05-27T07:46:04.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.215+0000 7f4041c0a700 1 -- 172.21.15.114:0/3337217596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f403c1adf00 con 0x7f403c103b90 2024-05-27T07:46:04.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.215+0000 7f4038ff9700 1 -- 172.21.15.114:0/3337217596 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4034004710 con 0x7f403c103b90 2024-05-27T07:46:04.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.219+0000 7f4038ff9700 1 -- 172.21.15.114:0/3337217596 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f403401d450 con 0x7f403c103b90 2024-05-27T07:46:04.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.224+0000 7f405061fb80 1 Processor -- start 2024-05-27T07:46:04.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.224+0000 7f405061fb80 1 -- start start 2024-05-27T07:46:04.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f405061fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6ada2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f405061fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 0x55ea6ada7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f405061fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada7bb0 0x55ea6ada7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f405061fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ea6acbfd70 con 0x55ea6ada33d0 2024-05-27T07:46:04.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f405061fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ea6acaf910 con 0x55ea6ac12e40 2024-05-27T07:46:04.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f405061fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ea6acb3950 con 0x55ea6ada7bb0 2024-05-27T07:46:04.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f403bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6ada2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f403bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6ada2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:58744/0 (socket says 172.21.15.114:58744) 2024-05-27T07:46:04.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f403bfff700 1 -- 172.21.15.114:0/3190986849 learned_addr learned my addr 172.21.15.114:0/3190986849 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:04.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f4038ff9700 1 --2- 172.21.15.114:0/3337217596 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f40240b2b80 0x7f40240b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f403affd700 1 --2- 172.21.15.114:0/3190986849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada7bb0 0x55ea6ada7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f403bfff700 1 -- 172.21.15.114:0/3190986849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada7bb0 msgr2=0x55ea6ada7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:04.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f403bfff700 1 --2- 172.21.15.114:0/3190986849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada7bb0 0x55ea6ada7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f403bfff700 1 -- 172.21.15.114:0/3190986849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 msgr2=0x55ea6ada7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:04.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f403bfff700 1 --2- 172.21.15.114:0/3190986849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 0x55ea6ada7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f403bfff700 1 -- 172.21.15.114:0/3190986849 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea6ac58fb0 con 0x55ea6ac12e40 2024-05-27T07:46:04.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.225+0000 7f403bfff700 1 --2- 172.21.15.114:0/3337217596 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f40240b2b80 0x7f40240b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.226+0000 7f403affd700 1 --2- 172.21.15.114:0/3190986849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada7bb0 0x55ea6ada7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:46:04.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.226+0000 7f4038ff9700 1 -- 172.21.15.114:0/3337217596 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f40340e38c0 con 0x7f403c103b90 2024-05-27T07:46:04.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.228+0000 7f403bfff700 1 --2- 172.21.15.114:0/3190986849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6ada2e90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f402c00acf0 tx=0x7f402c0113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=266d74199e0d4696 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:04.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.228+0000 7f403bfff700 1 --2- 172.21.15.114:0/3337217596 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f40240b2b80 0x7f40240b5090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f402c021000 tx=0x7f402c01e000 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:04.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.229+0000 7f4021ffb700 1 -- 172.21.15.114:0/3190986849 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c012d80 con 0x55ea6ac12e40 2024-05-27T07:46:04.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.229+0000 7f4021ffb700 1 -- 172.21.15.114:0/3190986849 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f402c00e860 con 0x55ea6ac12e40 2024-05-27T07:46:04.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.229+0000 7f4021ffb700 1 -- 172.21.15.114:0/3190986849 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c0196d0 con 0x55ea6ac12e40 2024-05-27T07:46:04.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.231+0000 7f405061fb80 1 -- 172.21.15.114:0/3190986849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 msgr2=0x55ea6ada2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:04.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.231+0000 7f405061fb80 1 --2- 172.21.15.114:0/3190986849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6ada2e90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f402c00acf0 tx=0x7f402c0113e0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.231+0000 7f405061fb80 1 -- 172.21.15.114:0/3190986849 shutdown_connections 2024-05-27T07:46:04.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.231+0000 7f405061fb80 1 --2- 172.21.15.114:0/3190986849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada7bb0 0x55ea6ada7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.231+0000 7f405061fb80 1 --2- 172.21.15.114:0/3190986849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6ada2e90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.231+0000 7f405061fb80 1 --2- 172.21.15.114:0/3190986849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 0x55ea6ada7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.231+0000 7f405061fb80 1 -- 172.21.15.114:0/3190986849 >> 172.21.15.114:0/3190986849 conn(0x55ea6ac15770 msgr2=0x55ea6ad8cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:04.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.232+0000 7f405061fb80 1 -- 172.21.15.114:0/3190986849 shutdown_connections 2024-05-27T07:46:04.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.233+0000 7f405061fb80 1 -- 172.21.15.114:0/3190986849 wait complete. 2024-05-27T07:46:04.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.233+0000 7f405061fb80 1 -- start start 2024-05-27T07:46:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.234+0000 7f405061fb80 1 Processor -- start 2024-05-27T07:46:04.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.234+0000 7f405061fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ac12e40 0x55ea6adac920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.235+0000 7f405061fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 0x55ea6adace60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.235+0000 7f403b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ac12e40 0x55ea6adac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.235+0000 7f403b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ac12e40 0x55ea6adac920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59546/0 (socket says 172.21.15.114:59546) 2024-05-27T07:46:04.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.235+0000 7f403affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 0x55ea6adace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.235+0000 7f403affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 0x55ea6adace60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44510/0 (socket says 172.21.15.114:44510) 2024-05-27T07:46:04.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.235+0000 7f403b7fe700 1 -- 172.21.15.114:0/3354350267 learned_addr learned my addr 172.21.15.114:0/3354350267 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:04.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.235+0000 7f405061fb80 1 --2- 172.21.15.114:0/3354350267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ada7bb0 0x55ea6adad3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.236+0000 7f405061fb80 1 -- 172.21.15.114:0/3354350267 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ea6abb5fa0 con 0x55ea6ada33d0 2024-05-27T07:46:04.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.236+0000 7f403bfff700 1 --2- 172.21.15.114:0/3354350267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ada7bb0 0x55ea6adad3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.236+0000 7f405061fb80 1 -- 172.21.15.114:0/3354350267 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ea6ac17da0 con 0x55ea6ada7bb0 2024-05-27T07:46:04.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.236+0000 7f405061fb80 1 -- 172.21.15.114:0/3354350267 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ea6adafbc0 con 0x55ea6ac12e40 2024-05-27T07:46:04.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.236+0000 7f405061fb80 10 client.0 _ll_register_callbacks cb 0x55ea6ac465b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:04.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.238+0000 7f403affd700 1 -- 172.21.15.114:0/3354350267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ac12e40 msgr2=0x55ea6adac920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:04.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.238+0000 7f403affd700 1 --2- 172.21.15.114:0/3354350267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ac12e40 0x55ea6adac920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.238+0000 7f403affd700 1 -- 172.21.15.114:0/3354350267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ada7bb0 msgr2=0x55ea6adad3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:04.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.238+0000 7f403affd700 1 --2- 172.21.15.114:0/3354350267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ada7bb0 0x55ea6adad3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.238+0000 7f403affd700 1 -- 172.21.15.114:0/3354350267 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea6abff8d0 con 0x55ea6ada33d0 2024-05-27T07:46:04.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.239+0000 7f403affd700 1 --2- 172.21.15.114:0/3354350267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 0x55ea6adace60 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f40340e0bb0 tx=0x7f40340aa1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e04b343951a0b4a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:04.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.239+0000 7f401affd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:04.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.242+0000 7f401affd700 1 -- 172.21.15.114:0/3354350267 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403402a070 con 0x55ea6ada33d0 2024-05-27T07:46:04.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.242+0000 7f401affd700 1 -- 172.21.15.114:0/3354350267 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f40340a6dc0 con 0x55ea6ada33d0 2024-05-27T07:46:04.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.242+0000 7f405061fb80 1 -- 172.21.15.114:0/3354350267 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f402c00e860 con 0x55ea6ada33d0 2024-05-27T07:46:04.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.242+0000 7f401affd700 1 -- 172.21.15.114:0/3354350267 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f40340aa2e0 con 0x55ea6ada33d0 2024-05-27T07:46:04.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.243+0000 7f405061fb80 10 client.16878 fetch_fsmap learned FSMap version 37 2024-05-27T07:46:04.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.243+0000 7f405061fb80 1 -- 172.21.15.114:0/3354350267 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ea6ac69760 con 0x55ea6ada33d0 2024-05-27T07:46:04.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.246+0000 7f401affd700 1 -- 172.21.15.114:0/3354350267 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40340c9050 con 0x55ea6ada33d0 2024-05-27T07:46:04.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.246+0000 7f401affd700 1 -- 172.21.15.114:0/3354350267 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f4034011070 con 0x55ea6ada33d0 2024-05-27T07:46:04.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.246+0000 7f405061fb80 10 client.16878 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:46:04.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.246+0000 7f405061fb80 10 client.16878 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:04.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.247+0000 7f405061fb80 20 client.16878 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:04.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.247+0000 7f405061fb80 1 --2- 172.21.15.114:0/3354350267 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ea6adbcb50 0x55ea6adbefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.247+0000 7f405061fb80 4 client.16878 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:46:04.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.247+0000 7f405061fb80 1 -- 172.21.15.114:0/3354350267 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f402c00e860 con 0x55ea6adbcb50 2024-05-27T07:46:04.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.247+0000 7f403b7fe700 1 --2- 172.21.15.114:0/3354350267 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ea6adbcb50 0x55ea6adbefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.248+0000 7f403b7fe700 1 --2- 172.21.15.114:0/3354350267 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ea6adbcb50 0x55ea6adbefa0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67fcd3846f54281 server_cookie=4c07c335f3246432 in_seq=0 out_seq=0 2024-05-27T07:46:04.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.248+0000 7f401affd700 10 client.16878 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:04.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.251+0000 7f401affd700 1 -- 172.21.15.114:0/3354350267 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f402c00e860 con 0x55ea6adbcb50 2024-05-27T07:46:04.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.251+0000 7f401affd700 10 client.16878 handle_command_reply: tid=0 2024-05-27T07:46:04.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.252+0000 7f405061fb80 1 client.16878 shutdown 2024-05-27T07:46:04.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.253+0000 7f405061fb80 1 -- 172.21.15.114:0/3354350267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 msgr2=0x55ea6adace60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:04.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.253+0000 7f405061fb80 1 --2- 172.21.15.114:0/3354350267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 0x55ea6adace60 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f40340e0bb0 tx=0x7f40340aa1b0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.253+0000 7f405061fb80 1 -- 172.21.15.114:0/3354350267 shutdown_connections 2024-05-27T07:46:04.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.253+0000 7f405061fb80 1 --2- 172.21.15.114:0/3354350267 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ea6adbcb50 0x55ea6adbefa0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.253+0000 7f405061fb80 1 --2- 172.21.15.114:0/3354350267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ada7bb0 0x55ea6adad3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.253+0000 7f405061fb80 1 --2- 172.21.15.114:0/3354350267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ac12e40 0x55ea6adac920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.253+0000 7f405061fb80 1 --2- 172.21.15.114:0/3354350267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 0x55ea6adace60 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.253+0000 7f405061fb80 1 -- 172.21.15.114:0/3354350267 >> 172.21.15.114:0/3354350267 conn(0x55ea6ac15770 msgr2=0x55ea6ad90bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:04.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.253+0000 7f401affd700 0 client.16878 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:04.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.254+0000 7f405061fb80 1 -- 172.21.15.114:0/3354350267 shutdown_connections 2024-05-27T07:46:04.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.257+0000 7f405061fb80 1 -- 172.21.15.114:0/3354350267 wait complete. 2024-05-27T07:46:04.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.258+0000 7f405061fb80 20 client.16878 trim_cache size 0 max 16384 2024-05-27T07:46:04.260 INFO:tasks.workunit.client.0.smithi114.stdout: 1 629283 12.79 MB/sec execute 136 sec latency 170.787 ms 2024-05-27T07:46:04.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.287+0000 7f405061fb80 1 Processor -- start 2024-05-27T07:46:04.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.287+0000 7f405061fb80 1 -- start start 2024-05-27T07:46:04.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.288+0000 7f405061fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6adab880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.288+0000 7f405061fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada33d0 0x55ea6adad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.288+0000 7f405061fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada7bb0 0x55ea6adb8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.288+0000 7f405061fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f402800ff10 con 0x55ea6ada7bb0 2024-05-27T07:46:04.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.288+0000 7f405061fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f402800cfe0 con 0x55ea6ac12e40 2024-05-27T07:46:04.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.288+0000 7f405061fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f402800f170 con 0x55ea6ada33d0 2024-05-27T07:46:04.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.288+0000 7f403b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6adab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.288+0000 7f403b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6adab880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:58758/0 (socket says 172.21.15.114:58758) 2024-05-27T07:46:04.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.288+0000 7f403b7fe700 1 -- 172.21.15.114:0/1589825294 learned_addr learned my addr 172.21.15.114:0/1589825294 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:04.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.288+0000 7f403bfff700 1 --2- 172.21.15.114:0/1589825294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada7bb0 0x55ea6adb8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.288+0000 7f403affd700 1 --2- 172.21.15.114:0/1589825294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada33d0 0x55ea6adad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.289+0000 7f403bfff700 1 -- 172.21.15.114:0/1589825294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada33d0 msgr2=0x55ea6adad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:04.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.289+0000 7f403bfff700 1 --2- 172.21.15.114:0/1589825294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada33d0 0x55ea6adad1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.289+0000 7f403bfff700 1 -- 172.21.15.114:0/1589825294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 msgr2=0x55ea6adab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:04.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.289+0000 7f403bfff700 1 --2- 172.21.15.114:0/1589825294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6adab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.289+0000 7f403bfff700 1 -- 172.21.15.114:0/1589825294 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea6adb9200 con 0x55ea6ada7bb0 2024-05-27T07:46:04.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.289+0000 7f403bfff700 1 --2- 172.21.15.114:0/1589825294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada7bb0 0x55ea6adb8cc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f402c011b50 tx=0x7f402c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9705ac878f19273 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:04.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.292+0000 7f40217fa700 1 -- 172.21.15.114:0/1589825294 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c012b80 con 0x55ea6ada7bb0 2024-05-27T07:46:04.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.292+0000 7f40217fa700 1 -- 172.21.15.114:0/1589825294 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f402c012d20 con 0x55ea6ada7bb0 2024-05-27T07:46:04.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.292+0000 7f40217fa700 1 -- 172.21.15.114:0/1589825294 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c00fe10 con 0x55ea6ada7bb0 2024-05-27T07:46:04.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.294+0000 7f405061fb80 1 -- 172.21.15.114:0/1589825294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada7bb0 msgr2=0x55ea6adb8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:04.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.294+0000 7f405061fb80 1 --2- 172.21.15.114:0/1589825294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada7bb0 0x55ea6adb8cc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f402c011b50 tx=0x7f402c011000 comp rx=0 tx=0).stop 2024-05-27T07:46:04.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.294+0000 7f405061fb80 1 -- 172.21.15.114:0/1589825294 shutdown_connections 2024-05-27T07:46:04.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.294+0000 7f405061fb80 1 --2- 172.21.15.114:0/1589825294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada7bb0 0x55ea6adb8cc0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.294+0000 7f405061fb80 1 --2- 172.21.15.114:0/1589825294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6adab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.294+0000 7f405061fb80 1 --2- 172.21.15.114:0/1589825294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada33d0 0x55ea6adad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.294+0000 7f405061fb80 1 -- 172.21.15.114:0/1589825294 >> 172.21.15.114:0/1589825294 conn(0x55ea6ac15770 msgr2=0x55ea6ad864f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:04.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.296+0000 7f405061fb80 1 -- 172.21.15.114:0/1589825294 shutdown_connections 2024-05-27T07:46:04.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.296+0000 7f405061fb80 1 -- 172.21.15.114:0/1589825294 wait complete. 2024-05-27T07:46:04.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.297+0000 7f405061fb80 1 -- start start 2024-05-27T07:46:04.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.297+0000 7f405061fb80 1 Processor -- start 2024-05-27T07:46:04.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.298+0000 7f405061fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6adb5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.298+0000 7f405061fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 0x55ea6adb5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.298+0000 7f405061fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada7bb0 0x55ea6adb05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.298+0000 7f405061fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f402800eaa0 con 0x55ea6ada33d0 2024-05-27T07:46:04.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.298+0000 7f405061fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f402800d630 con 0x55ea6ac12e40 2024-05-27T07:46:04.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.299+0000 7f405061fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f402800dd00 con 0x55ea6ada7bb0 2024-05-27T07:46:04.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.299+0000 7f403b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6adb5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.299+0000 7f403b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6adb5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:58774/0 (socket says 172.21.15.114:58774) 2024-05-27T07:46:04.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.299+0000 7f403b7fe700 1 -- 172.21.15.114:0/1468479894 learned_addr learned my addr 172.21.15.114:0/1468479894 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:04.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.299+0000 7f403bfff700 1 --2- 172.21.15.114:0/1468479894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada7bb0 0x55ea6adb05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.299+0000 7f403affd700 1 --2- 172.21.15.114:0/1468479894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 0x55ea6adb5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.299+0000 7f405061fb80 10 client.0 _ll_register_callbacks cb 0x55ea6ac465b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:04.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.299+0000 7f403b7fe700 1 -- 172.21.15.114:0/1468479894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada7bb0 msgr2=0x55ea6adb05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:04.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.299+0000 7f403b7fe700 1 --2- 172.21.15.114:0/1468479894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada7bb0 0x55ea6adb05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.299+0000 7f403b7fe700 1 -- 172.21.15.114:0/1468479894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 msgr2=0x55ea6adb5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:04.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.299+0000 7f403b7fe700 1 --2- 172.21.15.114:0/1468479894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 0x55ea6adb5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.299+0000 7f403b7fe700 1 -- 172.21.15.114:0/1468479894 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea6adb0d60 con 0x55ea6ac12e40 2024-05-27T07:46:04.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.300+0000 7f403b7fe700 1 --2- 172.21.15.114:0/1468479894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6adb5130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f403000a980 tx=0x7f4030012810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c3dad60998edaca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:04.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.302+0000 7f401affd700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:46:04.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.302+0000 7f401affd700 1 -- 172.21.15.114:0/1468479894 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030004220 con 0x55ea6ac12e40 2024-05-27T07:46:04.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.302+0000 7f401affd700 1 -- 172.21.15.114:0/1468479894 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f40300043c0 con 0x55ea6ac12e40 2024-05-27T07:46:04.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.302+0000 7f405061fb80 1 -- 172.21.15.114:0/1468479894 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f402c012d20 con 0x55ea6ac12e40 2024-05-27T07:46:04.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.304+0000 7f401affd700 1 -- 172.21.15.114:0/1468479894 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4030004830 con 0x55ea6ac12e40 2024-05-27T07:46:04.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.304+0000 7f405061fb80 10 client.26369 fetch_fsmap learned FSMap version 37 2024-05-27T07:46:04.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.304+0000 7f405061fb80 1 -- 172.21.15.114:0/1468479894 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ea6ae270b0 con 0x55ea6ac12e40 2024-05-27T07:46:04.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.307+0000 7f401affd700 1 -- 172.21.15.114:0/1468479894 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4030021070 con 0x55ea6ac12e40 2024-05-27T07:46:04.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.307+0000 7f401affd700 1 -- 172.21.15.114:0/1468479894 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f4030003680 con 0x55ea6ac12e40 2024-05-27T07:46:04.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.307+0000 7f405061fb80 10 client.26369 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:46:04.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.307+0000 7f405061fb80 10 client.26369 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:04.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.307+0000 7f405061fb80 20 client.26369 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:04.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.307+0000 7f405061fb80 1 --2- 172.21.15.114:0/1468479894 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ea6ae277f0 0x55ea6ae29c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:04.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.307+0000 7f405061fb80 4 client.26369 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:46:04.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.307+0000 7f405061fb80 1 -- 172.21.15.114:0/1468479894 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f402c012d20 con 0x55ea6ae277f0 2024-05-27T07:46:04.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.308+0000 7f403affd700 1 --2- 172.21.15.114:0/1468479894 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ea6ae277f0 0x55ea6ae29c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.309+0000 7f403affd700 1 --2- 172.21.15.114:0/1468479894 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ea6ae277f0 0x55ea6ae29c10 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8232cda021eaa158 server_cookie=703906a39ee3511e in_seq=0 out_seq=0 2024-05-27T07:46:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.310+0000 7f401affd700 10 client.26369 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.310+0000 7f401affd700 1 -- 172.21.15.114:0/1468479894 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f402c012d20 con 0x55ea6ae277f0 2024-05-27T07:46:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.310+0000 7f401affd700 10 client.26369 handle_command_reply: tid=0 2024-05-27T07:46:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.310+0000 7f405061fb80 1 client.26369 shutdown 2024-05-27T07:46:04.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.313+0000 7f405061fb80 1 -- 172.21.15.114:0/1468479894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 msgr2=0x55ea6adb5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:04.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.313+0000 7f405061fb80 1 --2- 172.21.15.114:0/1468479894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6adb5130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f403000a980 tx=0x7f4030012810 comp rx=0 tx=0).stop 2024-05-27T07:46:04.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.314+0000 7f405061fb80 1 -- 172.21.15.114:0/1468479894 shutdown_connections 2024-05-27T07:46:04.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.314+0000 7f405061fb80 1 --2- 172.21.15.114:0/1468479894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea6ada7bb0 0x55ea6adb05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.314+0000 7f405061fb80 1 --2- 172.21.15.114:0/1468479894 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ea6ae277f0 0x55ea6ae29c10 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.314+0000 7f405061fb80 1 --2- 172.21.15.114:0/1468479894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ea6ac12e40 0x55ea6adb5130 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.314+0000 7f405061fb80 1 --2- 172.21.15.114:0/1468479894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea6ada33d0 0x55ea6adb5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.314+0000 7f405061fb80 1 -- 172.21.15.114:0/1468479894 >> 172.21.15.114:0/1468479894 conn(0x55ea6ac15770 msgr2=0x55ea6adadd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.315+0000 7f401affd700 0 client.26369 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.315+0000 7f405061fb80 1 -- 172.21.15.114:0/1468479894 shutdown_connections 2024-05-27T07:46:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.316+0000 7f405061fb80 1 -- 172.21.15.114:0/1468479894 wait complete. 2024-05-27T07:46:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.316+0000 7f405061fb80 20 client.26369 trim_cache size 0 max 16384 2024-05-27T07:46:04.315 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:46:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.317+0000 7f4041c0a700 1 -- 172.21.15.114:0/3337217596 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f40240b2b80 msgr2=0x7f40240b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.317+0000 7f4041c0a700 1 --2- 172.21.15.114:0/3337217596 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f40240b2b80 0x7f40240b5090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f402c021000 tx=0x7f402c01e000 comp rx=0 tx=0).stop 2024-05-27T07:46:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.317+0000 7f4041c0a700 1 -- 172.21.15.114:0/3337217596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f403c103b90 msgr2=0x7f403c1a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.317+0000 7f4041c0a700 1 --2- 172.21.15.114:0/3337217596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f403c103b90 0x7f403c1a19c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4034006070 tx=0x7f4034009700 comp rx=0 tx=0).stop 2024-05-27T07:46:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.317+0000 7f4041c0a700 1 -- 172.21.15.114:0/3337217596 shutdown_connections 2024-05-27T07:46:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.317+0000 7f4041c0a700 1 --2- 172.21.15.114:0/3337217596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f403c108690 0x7f403c1a5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.318+0000 7f4041c0a700 1 --2- 172.21.15.114:0/3337217596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f403c103240 0x7f403c1a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.318+0000 7f4041c0a700 1 --2- 172.21.15.114:0/3337217596 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f40240b2b80 0x7f40240b5090 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.318+0000 7f4041c0a700 1 --2- 172.21.15.114:0/3337217596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f403c103b90 0x7f403c1a19c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.318+0000 7f4041c0a700 1 -- 172.21.15.114:0/3337217596 >> 172.21.15.114:0/3337217596 conn(0x7f403c0fedb0 msgr2=0x7f403c105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.319+0000 7f4041c0a700 1 -- 172.21.15.114:0/3337217596 shutdown_connections 2024-05-27T07:46:04.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:04.319+0000 7f4041c0a700 1 -- 172.21.15.114:0/3337217596 wait complete. 2024-05-27T07:46:05.261 INFO:tasks.workunit.client.0.smithi114.stdout: 1 629500 12.71 MB/sec execute 137 sec latency 157.823 ms 2024-05-27T07:46:05.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:05 smithi163 ceph-mon[36451]: pgmap v201: 129 pgs: 129 active+clean; 77 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 484 op/s 2024-05-27T07:46:05.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:05 smithi186 ceph-mon[36647]: pgmap v201: 129 pgs: 129 active+clean; 77 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 484 op/s 2024-05-27T07:46:05.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:05 smithi114 ceph-mon[32038]: pgmap v201: 129 pgs: 129 active+clean; 77 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 484 op/s 2024-05-27T07:46:06.039 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:46:06.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:46:06.261 INFO:tasks.workunit.client.0.smithi114.stdout: 1 630596 12.65 MB/sec execute 138 sec latency 117.488 ms 2024-05-27T07:46:06.447 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:46:07.261 INFO:tasks.workunit.client.0.smithi114.stdout: 1 632575 12.65 MB/sec execute 139 sec latency 19.552 ms 2024-05-27T07:46:07.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:07 smithi163 ceph-mon[36451]: pgmap v202: 129 pgs: 14 active+clean+snaptrim, 75 active+clean, 40 active+clean+snaptrim_wait; 82 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 471 op/s 2024-05-27T07:46:07.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:07 smithi186 ceph-mon[36647]: pgmap v202: 129 pgs: 14 active+clean+snaptrim, 75 active+clean, 40 active+clean+snaptrim_wait; 82 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 471 op/s 2024-05-27T07:46:07.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:07 smithi114 ceph-mon[32038]: pgmap v202: 129 pgs: 14 active+clean+snaptrim, 75 active+clean, 40 active+clean+snaptrim_wait; 82 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 471 op/s 2024-05-27T07:46:08.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.231+0000 7fba03ce2700 1 -- 172.21.15.114:0/2625403364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc103540 msgr2=0x7fb9fc103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:08.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.231+0000 7fba03ce2700 1 --2- 172.21.15.114:0/2625403364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc103540 0x7fb9fc103950 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb9f80099c0 tx=0x7fb9f8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.231+0000 7fba03ce2700 1 -- 172.21.15.114:0/2625403364 shutdown_connections 2024-05-27T07:46:08.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.231+0000 7fba03ce2700 1 --2- 172.21.15.114:0/2625403364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9fc108770 0x7fb9fc1100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.231+0000 7fba03ce2700 1 --2- 172.21.15.114:0/2625403364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc103540 0x7fb9fc103950 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.231+0000 7fba03ce2700 1 --2- 172.21.15.114:0/2625403364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9fc107e20 0x7fb9fc108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.231+0000 7fba03ce2700 1 -- 172.21.15.114:0/2625403364 >> 172.21.15.114:0/2625403364 conn(0x7fb9fc078df0 msgr2=0x7fb9fc079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.231+0000 7fba03ce2700 1 -- 172.21.15.114:0/2625403364 shutdown_connections 2024-05-27T07:46:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.231+0000 7fba03ce2700 1 -- 172.21.15.114:0/2625403364 wait complete. 2024-05-27T07:46:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.232+0000 7fba03ce2700 1 Processor -- start 2024-05-27T07:46:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.232+0000 7fba03ce2700 1 -- start start 2024-05-27T07:46:08.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.232+0000 7fba03ce2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9fc103540 0x7fb9fc19f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.232+0000 7fba03ce2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc107e20 0x7fb9fc19fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.232+0000 7fba03ce2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9fc108770 0x7fb9fc199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.232+0000 7fba03ce2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc1133a0 con 0x7fb9fc107e20 2024-05-27T07:46:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.232+0000 7fba03ce2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc113220 con 0x7fb9fc108770 2024-05-27T07:46:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.232+0000 7fba03ce2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc113520 con 0x7fb9fc103540 2024-05-27T07:46:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.232+0000 7fba0127d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc107e20 0x7fb9fc19fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.232+0000 7fba01a7e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9fc103540 0x7fb9fc19f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.232+0000 7fba0227f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9fc108770 0x7fb9fc199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.232+0000 7fba0127d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc107e20 0x7fb9fc19fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55086/0 (socket says 172.21.15.114:55086) 2024-05-27T07:46:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.232+0000 7fba0127d700 1 -- 172.21.15.114:0/3130831111 learned_addr learned my addr 172.21.15.114:0/3130831111 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.233+0000 7fba01a7e700 1 -- 172.21.15.114:0/3130831111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9fc108770 msgr2=0x7fb9fc199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.233+0000 7fba01a7e700 1 --2- 172.21.15.114:0/3130831111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9fc108770 0x7fb9fc199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.233+0000 7fba01a7e700 1 -- 172.21.15.114:0/3130831111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc107e20 msgr2=0x7fb9fc19fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.233+0000 7fba01a7e700 1 --2- 172.21.15.114:0/3130831111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc107e20 0x7fb9fc19fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.233+0000 7fba01a7e700 1 -- 172.21.15.114:0/3130831111 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9fc19a270 con 0x7fb9fc103540 2024-05-27T07:46:08.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.233+0000 7fba0127d700 1 --2- 172.21.15.114:0/3130831111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc107e20 0x7fb9fc19fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:08.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.233+0000 7fba01a7e700 1 --2- 172.21.15.114:0/3130831111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9fc103540 0x7fb9fc19f9b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb9f80098e0 tx=0x7fb9f800bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:08.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.233+0000 7fb9f2ffd700 1 -- 172.21.15.114:0/3130831111 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8009e60 con 0x7fb9fc103540 2024-05-27T07:46:08.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.233+0000 7fba03ce2700 1 -- 172.21.15.114:0/3130831111 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9fc19a500 con 0x7fb9fc103540 2024-05-27T07:46:08.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.233+0000 7fb9f2ffd700 1 -- 172.21.15.114:0/3130831111 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9f8003c70 con 0x7fb9fc103540 2024-05-27T07:46:08.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.233+0000 7fb9f2ffd700 1 -- 172.21.15.114:0/3130831111 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8019910 con 0x7fb9fc103540 2024-05-27T07:46:08.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.233+0000 7fba03ce2700 1 -- 172.21.15.114:0/3130831111 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9fc0759e0 con 0x7fb9fc103540 2024-05-27T07:46:08.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.237+0000 7fb9f2ffd700 1 -- 172.21.15.114:0/3130831111 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fb9f80037c0 con 0x7fb9fc103540 2024-05-27T07:46:08.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.237+0000 7fba126f7b80 1 Processor -- start 2024-05-27T07:46:08.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fba126f7b80 1 -- start start 2024-05-27T07:46:08.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fb9f2ffd700 1 --2- 172.21.15.114:0/3130831111 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb9e80b29f0 0x7fb9e80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fba126f7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380e3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fba126f7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380e3cd0 0x55f4380ddac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fba126f7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380de000 0x55f4380de410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fb9f2ffd700 1 -- 172.21.15.114:0/3130831111 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7fb9f80e39d0 con 0x7fb9fc103540 2024-05-27T07:46:08.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fba0227f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380de000 0x55f4380de410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fba0227f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380de000 0x55f4380de410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55096/0 (socket says 172.21.15.114:55096) 2024-05-27T07:46:08.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fba0227f700 1 -- 172.21.15.114:0/671983323 learned_addr learned my addr 172.21.15.114:0/671983323 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:08.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fba01a7e700 1 --2- 172.21.15.114:0/671983323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380e3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fba0127d700 1 --2- 172.21.15.114:0/671983323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380e3cd0 0x55f4380ddac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fba126f7b80 1 -- 172.21.15.114:0/671983323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f43800cd10 con 0x55f4380de000 2024-05-27T07:46:08.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fba126f7b80 1 -- 172.21.15.114:0/671983323 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f437ffc8b0 con 0x55f437f5fec0 2024-05-27T07:46:08.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fba126f7b80 1 -- 172.21.15.114:0/671983323 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f4380008f0 con 0x55f4380e3cd0 2024-05-27T07:46:08.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.238+0000 7fba0227f700 1 --2- 172.21.15.114:0/3130831111 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb9e80b29f0 0x7fb9e80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.239+0000 7fba01a7e700 1 -- 172.21.15.114:0/671983323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380e3cd0 msgr2=0x55f4380ddac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.239+0000 7fba01a7e700 1 --2- 172.21.15.114:0/671983323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380e3cd0 0x55f4380ddac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.239+0000 7fba01a7e700 1 -- 172.21.15.114:0/671983323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380de000 msgr2=0x55f4380de410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.239+0000 7fba01a7e700 1 --2- 172.21.15.114:0/671983323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380de000 0x55f4380de410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.239+0000 7fba01a7e700 1 -- 172.21.15.114:0/671983323 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f437f4c950 con 0x55f437f5fec0 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.239+0000 7fba01a7e700 1 --2- 172.21.15.114:0/671983323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380e3790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb9f80e0c60 tx=0x7fb9f8005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92c13f9e4f6ce165 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.239+0000 7fba0227f700 1 --2- 172.21.15.114:0/3130831111 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb9e80b29f0 0x7fb9e80b4f00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009fd0 tx=0x7fb9f40090a0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.239+0000 7fb9e7fff700 1 -- 172.21.15.114:0/671983323 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8003ab0 con 0x55f437f5fec0 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.239+0000 7fb9e7fff700 1 -- 172.21.15.114:0/671983323 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9f80aa390 con 0x55f437f5fec0 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.239+0000 7fb9e7fff700 1 -- 172.21.15.114:0/671983323 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f80aec30 con 0x55f437f5fec0 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.240+0000 7fba126f7b80 1 -- 172.21.15.114:0/671983323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 msgr2=0x55f4380e3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.240+0000 7fba126f7b80 1 --2- 172.21.15.114:0/671983323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380e3790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb9f80e0c60 tx=0x7fb9f8005e80 comp rx=0 tx=0).stop 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.240+0000 7fba126f7b80 1 -- 172.21.15.114:0/671983323 shutdown_connections 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.240+0000 7fba126f7b80 1 --2- 172.21.15.114:0/671983323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380de000 0x55f4380de410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.240+0000 7fba126f7b80 1 --2- 172.21.15.114:0/671983323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380e3790 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.240+0000 7fba126f7b80 1 --2- 172.21.15.114:0/671983323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380e3cd0 0x55f4380ddac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.241+0000 7fba126f7b80 1 -- 172.21.15.114:0/671983323 >> 172.21.15.114:0/671983323 conn(0x55f437fbab70 msgr2=0x55f4380d86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:08.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.241+0000 7fba126f7b80 1 -- 172.21.15.114:0/671983323 shutdown_connections 2024-05-27T07:46:08.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.241+0000 7fba126f7b80 1 -- 172.21.15.114:0/671983323 wait complete. 2024-05-27T07:46:08.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.241+0000 7fba126f7b80 1 -- start start 2024-05-27T07:46:08.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.241+0000 7fba126f7b80 1 Processor -- start 2024-05-27T07:46:08.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba126f7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f437f5fec0 0x55f4380e73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba126f7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4380de000 0x55f4380e78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba126f7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380e3cd0 0x55f4380e7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba126f7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f437f03020 con 0x55f437f5fec0 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba126f7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f437f9e840 con 0x55f4380de000 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba126f7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f437ffc8b0 con 0x55f4380e3cd0 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba126f7b80 10 client.0 _ll_register_callbacks cb 0x55f437f979a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba01a7e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f437f5fec0 0x55f4380e73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba0127d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4380de000 0x55f4380e78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba0227f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380e3cd0 0x55f4380e7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba0127d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4380de000 0x55f4380e78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:41582/0 (socket says 172.21.15.114:41582) 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba0127d700 1 -- 172.21.15.114:0/1565624806 learned_addr learned my addr 172.21.15.114:0/1565624806 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba0127d700 1 -- 172.21.15.114:0/1565624806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380e3cd0 msgr2=0x55f4380e7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba0127d700 1 --2- 172.21.15.114:0/1565624806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380e3cd0 0x55f4380e7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba0127d700 1 -- 172.21.15.114:0/1565624806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f437f5fec0 msgr2=0x55f4380e73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba0127d700 1 --2- 172.21.15.114:0/1565624806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f437f5fec0 0x55f4380e73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba0127d700 1 -- 172.21.15.114:0/1565624806 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f437f9b710 con 0x55f4380de000 2024-05-27T07:46:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fba0127d700 1 --2- 172.21.15.114:0/1565624806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4380de000 0x55f4380e78e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec012d20 tx=0x7fb9ec00f6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbe600b502db1e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.242+0000 7fb9e57fa700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fb9e57fa700 1 -- 172.21.15.114:0/1565624806 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec00fb80 con 0x55f4380de000 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fb9e57fa700 1 -- 172.21.15.114:0/1565624806 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7fb9ec00fd20 con 0x55f4380de000 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fba126f7b80 1 -- 172.21.15.114:0/1565624806 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9f80aa390 con 0x55f4380de000 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fb9e57fa700 1 -- 172.21.15.114:0/1565624806 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb9ec004050 con 0x55f4380de000 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fba126f7b80 10 client.26381 fetch_fsmap learned FSMap version 37 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fba126f7b80 1 -- 172.21.15.114:0/1565624806 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f4380f4e20 con 0x55f4380de000 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fb9e57fa700 1 -- 172.21.15.114:0/1565624806 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9f80aa390 con 0x55f4380de000 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fb9e57fa700 1 -- 172.21.15.114:0/1565624806 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7fb9ec010430 con 0x55f4380de000 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fba126f7b80 10 client.26381 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fba126f7b80 10 client.26381 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fba126f7b80 20 client.26381 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fba126f7b80 1 --2- 172.21.15.114:0/1565624806 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f4380f7460 0x55f4380f9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fba126f7b80 4 client.26381 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:46:08.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.243+0000 7fba126f7b80 1 -- 172.21.15.114:0/1565624806 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4380a9a70 con 0x55f4380f7460 2024-05-27T07:46:08.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.244+0000 7fba01a7e700 1 --2- 172.21.15.114:0/1565624806 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f4380f7460 0x55f4380f9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.244+0000 7fb9e57fa700 10 client.26381 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:08.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.244+0000 7fba01a7e700 1 --2- 172.21.15.114:0/1565624806 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f4380f7460 0x55f4380f9880 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd17cc5a7676c486 server_cookie=4643f8c317f3000a in_seq=0 out_seq=0 2024-05-27T07:46:08.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fb9e57fa700 1 -- 172.21.15.114:0/1565624806 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f4380a9a70 con 0x55f4380f7460 2024-05-27T07:46:08.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fb9e57fa700 10 client.26381 handle_command_reply: tid=0 2024-05-27T07:46:08.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fba126f7b80 1 client.26381 shutdown 2024-05-27T07:46:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fba126f7b80 1 -- 172.21.15.114:0/1565624806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4380de000 msgr2=0x55f4380e78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fba126f7b80 1 --2- 172.21.15.114:0/1565624806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4380de000 0x55f4380e78e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec012d20 tx=0x7fb9ec00f6a0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fba126f7b80 1 -- 172.21.15.114:0/1565624806 shutdown_connections 2024-05-27T07:46:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fba126f7b80 1 --2- 172.21.15.114:0/1565624806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380e3cd0 0x55f4380e7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fba126f7b80 1 --2- 172.21.15.114:0/1565624806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f437f5fec0 0x55f4380e73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fba126f7b80 1 --2- 172.21.15.114:0/1565624806 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f4380f7460 0x55f4380f9880 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fba126f7b80 1 --2- 172.21.15.114:0/1565624806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4380de000 0x55f4380e78e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fb9e57fa700 0 client.26381 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fba126f7b80 1 -- 172.21.15.114:0/1565624806 >> 172.21.15.114:0/1565624806 conn(0x55f437fbab70 msgr2=0x55f4380dcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fba126f7b80 1 -- 172.21.15.114:0/1565624806 shutdown_connections 2024-05-27T07:46:08.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fba126f7b80 1 -- 172.21.15.114:0/1565624806 wait complete. 2024-05-27T07:46:08.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.245+0000 7fba126f7b80 20 client.26381 trim_cache size 0 max 16384 2024-05-27T07:46:08.261 INFO:tasks.workunit.client.0.smithi114.stdout: 1 634664 12.62 MB/sec execute 140 sec latency 86.255 ms 2024-05-27T07:46:08.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.276+0000 7fba126f7b80 1 Processor -- start 2024-05-27T07:46:08.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.277+0000 7fba126f7b80 1 -- start start 2024-05-27T07:46:08.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.277+0000 7fba126f7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380ddac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.277+0000 7fba126f7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380de000 0x55f4380e9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.277+0000 7fba126f7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380e3cd0 0x55f4380f3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.278+0000 7fba126f7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc1af530 con 0x55f4380e3cd0 2024-05-27T07:46:08.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.278+0000 7fba126f7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc1ac600 con 0x55f437f5fec0 2024-05-27T07:46:08.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.278+0000 7fba126f7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc1ae790 con 0x55f4380de000 2024-05-27T07:46:08.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.278+0000 7fba0227f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380e3cd0 0x55f4380f3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.278+0000 7fba0227f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380e3cd0 0x55f4380f3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55114/0 (socket says 172.21.15.114:55114) 2024-05-27T07:46:08.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.278+0000 7fba0227f700 1 -- 172.21.15.114:0/3093524196 learned_addr learned my addr 172.21.15.114:0/3093524196 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:08.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.278+0000 7fba0127d700 1 --2- 172.21.15.114:0/3093524196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380de000 0x55f4380e9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.278+0000 7fba0227f700 1 -- 172.21.15.114:0/3093524196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380de000 msgr2=0x55f4380e9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:08.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.278+0000 7fba0227f700 1 --2- 172.21.15.114:0/3093524196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380de000 0x55f4380e9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.278+0000 7fba0227f700 1 -- 172.21.15.114:0/3093524196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 msgr2=0x55f4380ddac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:46:08.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.278+0000 7fba0227f700 1 --2- 172.21.15.114:0/3093524196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380ddac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.278+0000 7fba0227f700 1 -- 172.21.15.114:0/3093524196 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4380f39a0 con 0x55f4380e3cd0 2024-05-27T07:46:08.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.279+0000 7fba0227f700 1 --2- 172.21.15.114:0/3093524196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380e3cd0 0x55f4380f3460 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4008ec0 tx=0x7fb9f4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc3dd96cb06280f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:08.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.283+0000 7fb9e77fe700 1 -- 172.21.15.114:0/3093524196 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4004db0 con 0x55f4380e3cd0 2024-05-27T07:46:08.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.283+0000 7fb9e77fe700 1 -- 172.21.15.114:0/3093524196 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9f4003a00 con 0x55f4380e3cd0 2024-05-27T07:46:08.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.283+0000 7fb9e77fe700 1 -- 172.21.15.114:0/3093524196 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f40044e0 con 0x55f4380e3cd0 2024-05-27T07:46:08.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.284+0000 7fba126f7b80 1 -- 172.21.15.114:0/3093524196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380e3cd0 msgr2=0x55f4380f3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:08.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.284+0000 7fba126f7b80 1 --2- 172.21.15.114:0/3093524196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380e3cd0 0x55f4380f3460 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4008ec0 tx=0x7fb9f4004b40 comp rx=0 tx=0).stop 2024-05-27T07:46:08.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.284+0000 7fba126f7b80 1 -- 172.21.15.114:0/3093524196 shutdown_connections 2024-05-27T07:46:08.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.284+0000 7fba126f7b80 1 --2- 172.21.15.114:0/3093524196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380e3cd0 0x55f4380f3460 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.284+0000 7fba126f7b80 1 --2- 172.21.15.114:0/3093524196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380ddac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.284+0000 7fba126f7b80 1 --2- 172.21.15.114:0/3093524196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380de000 0x55f4380e9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.284+0000 7fba126f7b80 1 -- 172.21.15.114:0/3093524196 >> 172.21.15.114:0/3093524196 conn(0x55f437fbab70 msgr2=0x55f4380e7360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:08.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.286+0000 7fba126f7b80 1 -- 172.21.15.114:0/3093524196 shutdown_connections 2024-05-27T07:46:08.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.286+0000 7fba126f7b80 1 -- 172.21.15.114:0/3093524196 wait complete. 2024-05-27T07:46:08.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.287+0000 7fba126f7b80 1 -- start start 2024-05-27T07:46:08.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.287+0000 7fba126f7b80 1 Processor -- start 2024-05-27T07:46:08.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.287+0000 7fba126f7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380ef9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.287+0000 7fba126f7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380de000 0x55f43815cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.287+0000 7fba126f7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380e3cd0 0x55f43815d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba126f7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc1ae0c0 con 0x55f4380e3cd0 2024-05-27T07:46:08.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba126f7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc1acc50 con 0x55f437f5fec0 2024-05-27T07:46:08.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba126f7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc1ad320 con 0x55f4380de000 2024-05-27T07:46:08.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba126f7b80 10 client.0 _ll_register_callbacks cb 0x55f437f979a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:08.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba01a7e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380ef9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba01a7e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380ef9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:41594/0 (socket says 172.21.15.114:41594) 2024-05-27T07:46:08.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba01a7e700 1 -- 172.21.15.114:0/2798341246 learned_addr learned my addr 172.21.15.114:0/2798341246 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:08.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba0127d700 1 --2- 172.21.15.114:0/2798341246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380de000 0x55f43815cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba01a7e700 1 -- 172.21.15.114:0/2798341246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380de000 msgr2=0x55f43815cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:08.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba01a7e700 1 --2- 172.21.15.114:0/2798341246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380de000 0x55f43815cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba01a7e700 1 -- 172.21.15.114:0/2798341246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380e3cd0 msgr2=0x55f43815d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:08.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba01a7e700 1 --2- 172.21.15.114:0/2798341246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380e3cd0 0x55f43815d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba01a7e700 1 -- 172.21.15.114:0/2798341246 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f43815da00 con 0x55f437f5fec0 2024-05-27T07:46:08.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.288+0000 7fba01a7e700 1 --2- 172.21.15.114:0/2798341246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380ef9f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800b580 tx=0x7fb9f80aa700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67c62df868d12606 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:08.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.289+0000 7fb9e57fa700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:46:08.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.289+0000 7fb9e57fa700 1 -- 172.21.15.114:0/2798341246 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f80aa390 con 0x55f437f5fec0 2024-05-27T07:46:08.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.289+0000 7fb9e57fa700 1 -- 172.21.15.114:0/2798341246 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7fb9f80e0ee0 con 0x55f437f5fec0 2024-05-27T07:46:08.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.289+0000 7fba126f7b80 1 -- 172.21.15.114:0/2798341246 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9f4003a00 con 0x55f437f5fec0 2024-05-27T07:46:08.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.289+0000 7fb9e57fa700 1 -- 172.21.15.114:0/2798341246 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb9f80aaf80 con 0x55f437f5fec0 2024-05-27T07:46:08.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.289+0000 7fba126f7b80 10 client.26384 fetch_fsmap learned FSMap version 37 2024-05-27T07:46:08.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.289+0000 7fba126f7b80 1 -- 172.21.15.114:0/2798341246 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f43815cb90 con 0x55f437f5fec0 2024-05-27T07:46:08.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.289+0000 7fb9e57fa700 1 -- 172.21.15.114:0/2798341246 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9f4003a00 con 0x55f437f5fec0 2024-05-27T07:46:08.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.289+0000 7fb9e57fa700 1 -- 172.21.15.114:0/2798341246 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7fb9f8021070 con 0x55f437f5fec0 2024-05-27T07:46:08.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.290+0000 7fba126f7b80 10 client.26384 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:46:08.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.290+0000 7fba126f7b80 10 client.26384 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:08.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.290+0000 7fba126f7b80 20 client.26384 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:08.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.290+0000 7fba126f7b80 1 --2- 172.21.15.114:0/2798341246 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f4380ee9c0 0x55f4380eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:08.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.290+0000 7fba126f7b80 4 client.26384 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:46:08.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.290+0000 7fba126f7b80 1 -- 172.21.15.114:0/2798341246 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f43800ec60 con 0x55f4380ee9c0 2024-05-27T07:46:08.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.290+0000 7fba0127d700 1 --2- 172.21.15.114:0/2798341246 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f4380ee9c0 0x55f4380eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:08.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.290+0000 7fb9e57fa700 10 client.26384 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:08.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.290+0000 7fba0127d700 1 --2- 172.21.15.114:0/2798341246 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f4380ee9c0 0x55f4380eedd0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d659f7c3ffce5704 server_cookie=a4ad099c25f25133 in_seq=0 out_seq=0 2024-05-27T07:46:08.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.295+0000 7fb9e57fa700 1 -- 172.21.15.114:0/2798341246 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f43800ec60 con 0x55f4380ee9c0 2024-05-27T07:46:08.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.295+0000 7fb9e57fa700 10 client.26384 handle_command_reply: tid=0 2024-05-27T07:46:08.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.295+0000 7fba126f7b80 1 client.26384 shutdown 2024-05-27T07:46:08.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.295+0000 7fba126f7b80 1 -- 172.21.15.114:0/2798341246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 msgr2=0x55f4380ef9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.295+0000 7fba126f7b80 1 --2- 172.21.15.114:0/2798341246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380ef9f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800b580 tx=0x7fb9f80aa700 comp rx=0 tx=0).stop 2024-05-27T07:46:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.295+0000 7fba126f7b80 1 -- 172.21.15.114:0/2798341246 shutdown_connections 2024-05-27T07:46:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.295+0000 7fba126f7b80 1 --2- 172.21.15.114:0/2798341246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f4380e3cd0 0x55f43815d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.295+0000 7fba126f7b80 1 --2- 172.21.15.114:0/2798341246 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f4380ee9c0 0x55f4380eedd0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.295+0000 7fba126f7b80 1 --2- 172.21.15.114:0/2798341246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f437f5fec0 0x55f4380ef9f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.295+0000 7fb9e57fa700 0 client.26384 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.295+0000 7fba126f7b80 1 --2- 172.21.15.114:0/2798341246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4380de000 0x55f43815cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba126f7b80 1 -- 172.21.15.114:0/2798341246 >> 172.21.15.114:0/2798341246 conn(0x55f437fbab70 msgr2=0x55f4380e1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba126f7b80 1 -- 172.21.15.114:0/2798341246 shutdown_connections 2024-05-27T07:46:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba126f7b80 1 -- 172.21.15.114:0/2798341246 wait complete. 2024-05-27T07:46:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba126f7b80 20 client.26384 trim_cache size 0 max 16384 2024-05-27T07:46:08.297 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:46:08.297 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:46:08.297 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "52bc62a7-5c93-4cf9-bd5e-b6c01e441e1a", 2024-05-27T07:46:08.297 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:46:08.297 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:46:08.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba03ce2700 1 -- 172.21.15.114:0/3130831111 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb9e80b29f0 msgr2=0x7fb9e80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:08.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba03ce2700 1 --2- 172.21.15.114:0/3130831111 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb9e80b29f0 0x7fb9e80b4f00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009fd0 tx=0x7fb9f40090a0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba03ce2700 1 -- 172.21.15.114:0/3130831111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9fc103540 msgr2=0x7fb9fc19f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:08.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba03ce2700 1 --2- 172.21.15.114:0/3130831111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9fc103540 0x7fb9fc19f9b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb9f80098e0 tx=0x7fb9f800bee0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba03ce2700 1 -- 172.21.15.114:0/3130831111 shutdown_connections 2024-05-27T07:46:08.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba03ce2700 1 --2- 172.21.15.114:0/3130831111 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb9e80b29f0 0x7fb9e80b4f00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba03ce2700 1 --2- 172.21.15.114:0/3130831111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9fc108770 0x7fb9fc199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba03ce2700 1 --2- 172.21.15.114:0/3130831111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9fc103540 0x7fb9fc19f9b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba03ce2700 1 --2- 172.21.15.114:0/3130831111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc107e20 0x7fb9fc19fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:08.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.296+0000 7fba03ce2700 1 -- 172.21.15.114:0/3130831111 >> 172.21.15.114:0/3130831111 conn(0x7fb9fc078df0 msgr2=0x7fb9fc101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:08.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.297+0000 7fba03ce2700 1 -- 172.21.15.114:0/3130831111 shutdown_connections 2024-05-27T07:46:08.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:08.297+0000 7fba03ce2700 1 -- 172.21.15.114:0/3130831111 wait complete. 2024-05-27T07:46:09.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:46:09.261 INFO:tasks.workunit.client.0.smithi114.stdout: 1 636802 12.63 MB/sec execute 141 sec latency 24.020 ms 2024-05-27T07:46:09.286 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:46:09.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:09 smithi186 ceph-mon[36647]: pgmap v203: 129 pgs: 14 active+clean+snaptrim, 75 active+clean, 40 active+clean+snaptrim_wait; 82 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 471 op/s 2024-05-27T07:46:09.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:09 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:46:09.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:09 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:09.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:09 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:46:09.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:09 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:09.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:09 smithi163 ceph-mon[36451]: pgmap v203: 129 pgs: 14 active+clean+snaptrim, 75 active+clean, 40 active+clean+snaptrim_wait; 82 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 471 op/s 2024-05-27T07:46:09.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:09 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:46:09.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:09 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:09.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:09 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:46:09.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:09 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:09.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:09 smithi114 ceph-mon[32038]: pgmap v203: 129 pgs: 14 active+clean+snaptrim, 75 active+clean, 40 active+clean+snaptrim_wait; 82 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 471 op/s 2024-05-27T07:46:09.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:09 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:46:09.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:09 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:09.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:09 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:46:09.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:09 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.120+0000 7f79fe896640 1 -- 172.21.15.114:0/841562465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f8108b80 msgr2=0x7f79f8102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.120+0000 7f79fe896640 1 --2- 172.21.15.114:0/841562465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f8108b80 0x7f79f8102720 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009900 tx=0x7f79e802a410 comp rx=0 tx=0).stop 2024-05-27T07:46:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.121+0000 7f79fe896640 1 -- 172.21.15.114:0/841562465 shutdown_connections 2024-05-27T07:46:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.121+0000 7f79fe896640 1 --2- 172.21.15.114:0/841562465 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79f81032d0 0x7f79f810f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.121+0000 7f79fe896640 1 --2- 172.21.15.114:0/841562465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79f8109020 0x7f79f8102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.121+0000 7f79fe896640 1 --2- 172.21.15.114:0/841562465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f8108b80 0x7f79f8102720 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.121+0000 7f79fe896640 1 -- 172.21.15.114:0/841562465 >> 172.21.15.114:0/841562465 conn(0x7f79f80fe380 msgr2=0x7f79f81007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.121+0000 7f79fe896640 1 -- 172.21.15.114:0/841562465 shutdown_connections 2024-05-27T07:46:10.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.121+0000 7f79fe896640 1 -- 172.21.15.114:0/841562465 wait complete. 2024-05-27T07:46:10.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.121+0000 7f79fe896640 1 Processor -- start 2024-05-27T07:46:10.122 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.121+0000 7f79fe896640 1 -- start start 2024-05-27T07:46:10.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79fe896640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f81032d0 0x7f79f81a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79fe896640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79f8108b80 0x7f79f81a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79fe896640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79f8109020 0x7f79f819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79fe896640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79f8112d70 con 0x7f79f81032d0 2024-05-27T07:46:10.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79fe896640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79f8112bf0 con 0x7f79f8109020 2024-05-27T07:46:10.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79fe896640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79f8112ef0 con 0x7f79f8108b80 2024-05-27T07:46:10.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79f7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f81032d0 0x7f79f81a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79f7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f81032d0 0x7f79f81a05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55136/0 (socket says 172.21.15.114:55136) 2024-05-27T07:46:10.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79f7fff640 1 -- 172.21.15.114:0/2152553276 learned_addr learned my addr 172.21.15.114:0/2152553276 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:10.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79fce33640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79f8109020 0x7f79f819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79f77fe640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79f8108b80 0x7f79f81a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79f7fff640 1 -- 172.21.15.114:0/2152553276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79f8108b80 msgr2=0x7f79f81a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:10.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79f7fff640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79f8108b80 0x7f79f81a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79f7fff640 1 -- 172.21.15.114:0/2152553276 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79f8109020 msgr2=0x7f79f819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.122+0000 7f79f7fff640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79f8109020 0x7f79f819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.123+0000 7f79f7fff640 1 -- 172.21.15.114:0/2152553276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f819ae90 con 0x7f79f81032d0 2024-05-27T07:46:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.123+0000 7f79fce33640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79f8109020 0x7f79f819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.123+0000 7f79f7fff640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f81032d0 0x7f79f81a05e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f79e8032fd0 tx=0x7f79e8031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.123+0000 7f79f57fa640 1 -- 172.21.15.114:0/2152553276 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e803f030 con 0x7f79f81032d0 2024-05-27T07:46:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.123+0000 7f79fe896640 1 -- 172.21.15.114:0/2152553276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f819b0c0 con 0x7f79f81032d0 2024-05-27T07:46:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.123+0000 7f79f57fa640 1 -- 172.21.15.114:0/2152553276 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79e8004490 con 0x7f79f81032d0 2024-05-27T07:46:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.123+0000 7f79fe896640 1 -- 172.21.15.114:0/2152553276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f81a7380 con 0x7f79f81032d0 2024-05-27T07:46:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.123+0000 7f79f57fa640 1 -- 172.21.15.114:0/2152553276 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8035070 con 0x7f79f81032d0 2024-05-27T07:46:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.126+0000 7f7a05827740 1 Processor -- start 2024-05-27T07:46:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.126+0000 7f7a05827740 1 -- start start 2024-05-27T07:46:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.126+0000 7f79f57fa640 1 -- 172.21.15.114:0/2152553276 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f79e8042030 con 0x7f79f81032d0 2024-05-27T07:46:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.126+0000 7f7a05827740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706c03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.126+0000 7f7a05827740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702f62f0 0x5646706c0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.126+0000 7f7a05827740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702fe4d0 0x5646706c0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.126+0000 7f7a05827740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564670632c30 con 0x5646702fe4d0 2024-05-27T07:46:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.126+0000 7f7a05827740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5646706227d0 con 0x56467031f1b0 2024-05-27T07:46:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.126+0000 7f7a05827740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564670626810 con 0x5646702f62f0 2024-05-27T07:46:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79fce33640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706c03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79f77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702fe4d0 0x5646706c0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79f7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702f62f0 0x5646706c0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79fce33640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706c03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:41608/0 (socket says 172.21.15.114:41608) 2024-05-27T07:46:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79fce33640 1 -- 172.21.15.114:0/149270410 learned_addr learned my addr 172.21.15.114:0/149270410 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79fce33640 1 -- 172.21.15.114:0/149270410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702f62f0 msgr2=0x5646706c0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79fce33640 1 --2- 172.21.15.114:0/149270410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702f62f0 0x5646706c0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79fce33640 1 -- 172.21.15.114:0/149270410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702fe4d0 msgr2=0x5646706c0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79fce33640 1 --2- 172.21.15.114:0/149270410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702fe4d0 0x5646706c0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79fce33640 1 -- 172.21.15.114:0/149270410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564670337210 con 0x56467031f1b0 2024-05-27T07:46:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79f77fe640 1 --2- 172.21.15.114:0/149270410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702fe4d0 0x5646706c0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79fce33640 1 --2- 172.21.15.114:0/149270410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706c03d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f79f00117e0 tx=0x7f79f0011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c22236739fda7ef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79f57fa640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f79e00b29d0 0x7f79e00b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79f7fff640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f79e00b29d0 0x7f79e00b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79de7fc640 1 -- 172.21.15.114:0/149270410 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f0011310 con 0x56467031f1b0 2024-05-27T07:46:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79de7fc640 1 -- 172.21.15.114:0/149270410 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79f0003e20 con 0x56467031f1b0 2024-05-27T07:46:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79de7fc640 1 -- 172.21.15.114:0/149270410 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f00044f0 con 0x56467031f1b0 2024-05-27T07:46:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.127+0000 7f79f57fa640 1 -- 172.21.15.114:0/2152553276 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f79e8102d10 con 0x7f79f81032d0 2024-05-27T07:46:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.128+0000 7f7a05827740 1 -- 172.21.15.114:0/149270410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 msgr2=0x5646706c03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.128+0000 7f7a05827740 1 --2- 172.21.15.114:0/149270410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706c03d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f79f00117e0 tx=0x7f79f0011ca0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.128+0000 7f79f7fff640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f79e00b29d0 0x7f79e00b4ee0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f79e81000b0 tx=0x7f79e810b040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.128+0000 7f7a05827740 1 -- 172.21.15.114:0/149270410 shutdown_connections 2024-05-27T07:46:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.128+0000 7f7a05827740 1 --2- 172.21.15.114:0/149270410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702fe4d0 0x5646706c0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.128+0000 7f7a05827740 1 --2- 172.21.15.114:0/149270410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702f62f0 0x5646706c0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.128+0000 7f7a05827740 1 --2- 172.21.15.114:0/149270410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706c03d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.128+0000 7f7a05827740 1 -- 172.21.15.114:0/149270410 >> 172.21.15.114:0/149270410 conn(0x564670313300 msgr2=0x5646706bcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.128+0000 7f7a05827740 1 -- 172.21.15.114:0/149270410 shutdown_connections 2024-05-27T07:46:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.128+0000 7f7a05827740 1 -- 172.21.15.114:0/149270410 wait complete. 2024-05-27T07:46:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.129+0000 7f7a05827740 1 -- start start 2024-05-27T07:46:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.129+0000 7f7a05827740 1 Processor -- start 2024-05-27T07:46:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.129+0000 7f7a05827740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 0x5646706cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.129+0000 7f7a05827740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702fe4d0 0x5646706cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.129+0000 7f7a05827740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706ca920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.129+0000 7f7a05827740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564670308790 con 0x5646702f62f0 2024-05-27T07:46:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.129+0000 7f7a05827740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5646705de890 con 0x56467031f1b0 2024-05-27T07:46:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.129+0000 7f7a05827740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5646706cd500 con 0x5646702fe4d0 2024-05-27T07:46:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.129+0000 7f7a05827740 10 client.0 _ll_register_callbacks cb 0x564670336a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.129+0000 7f79fce33640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 0x5646706cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.129+0000 7f79fce33640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 0x5646706cc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55162/0 (socket says 172.21.15.114:55162) 2024-05-27T07:46:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.129+0000 7f79fce33640 1 -- 172.21.15.114:0/3621047179 learned_addr learned my addr 172.21.15.114:0/3621047179 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.129+0000 7f79f77fe640 1 --2- 172.21.15.114:0/3621047179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702fe4d0 0x5646706cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f79fce33640 1 --2- 172.21.15.114:0/3621047179 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f79fce33640 1 -- 172.21.15.114:0/3621047179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702fe4d0 msgr2=0x5646706cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f79fce33640 1 --2- 172.21.15.114:0/3621047179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702fe4d0 0x5646706cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f79fce33640 1 -- 172.21.15.114:0/3621047179 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 msgr2=0x5646706ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f79fce33640 1 --2- 172.21.15.114:0/3621047179 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f79fce33640 1 -- 172.21.15.114:0/3621047179 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564670403620 con 0x5646702f62f0 2024-05-27T07:46:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f79fce33640 1 --2- 172.21.15.114:0/3621047179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 0x5646706cc410 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f79f0004110 tx=0x7f79f0005ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a908839f37d32389 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f79d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f79d77fe640 1 -- 172.21.15.114:0/3621047179 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f0020020 con 0x5646702f62f0 2024-05-27T07:46:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f79d77fe640 1 -- 172.21.15.114:0/3621047179 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f79f0004a40 con 0x5646702f62f0 2024-05-27T07:46:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f7a05827740 1 -- 172.21.15.114:0/3621047179 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79f0003e20 con 0x5646702f62f0 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f79d77fe640 1 -- 172.21.15.114:0/3621047179 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f79f0033050 con 0x5646702f62f0 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f7a05827740 10 client.16911 fetch_fsmap learned FSMap version 37 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f7a05827740 1 -- 172.21.15.114:0/3621047179 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5646706d9900 con 0x5646702f62f0 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.130+0000 7f79d77fe640 1 -- 172.21.15.114:0/3621047179 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79f0003e20 con 0x5646702f62f0 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.131+0000 7f79d77fe640 1 -- 172.21.15.114:0/3621047179 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f79f0018020 con 0x5646702f62f0 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.131+0000 7f7a05827740 10 client.16911 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.131+0000 7f7a05827740 10 client.16911 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.131+0000 7f7a05827740 20 client.16911 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.131+0000 7f7a05827740 1 --2- 172.21.15.114:0/3621047179 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5646702ddd80 0x5646706dcd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.131+0000 7f7a05827740 4 client.16911 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.131+0000 7f7a05827740 1 -- 172.21.15.114:0/3621047179 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564670634b80 con 0x5646702ddd80 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.131+0000 7f79f77fe640 1 --2- 172.21.15.114:0/3621047179 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5646702ddd80 0x5646706dcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.131+0000 7f79d77fe640 10 client.16911 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.131+0000 7f79f77fe640 1 --2- 172.21.15.114:0/3621047179 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5646702ddd80 0x5646706dcd20 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85da01db3ff9ef25 server_cookie=b6f2bd971931db7f in_seq=0 out_seq=0 2024-05-27T07:46:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f79d77fe640 1 -- 172.21.15.114:0/3621047179 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564670634b80 con 0x5646702ddd80 2024-05-27T07:46:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f79d77fe640 10 client.16911 handle_command_reply: tid=0 2024-05-27T07:46:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f7a05827740 1 client.16911 shutdown 2024-05-27T07:46:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f7a05827740 1 -- 172.21.15.114:0/3621047179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 msgr2=0x5646706cc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f7a05827740 1 --2- 172.21.15.114:0/3621047179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 0x5646706cc410 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f79f0004110 tx=0x7f79f0005ad0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f7a05827740 1 -- 172.21.15.114:0/3621047179 shutdown_connections 2024-05-27T07:46:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f7a05827740 1 --2- 172.21.15.114:0/3621047179 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5646702ddd80 0x5646706dcd20 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f7a05827740 1 --2- 172.21.15.114:0/3621047179 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f7a05827740 1 --2- 172.21.15.114:0/3621047179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702fe4d0 0x5646706cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f79d77fe640 0 client.16911 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f7a05827740 1 --2- 172.21.15.114:0/3621047179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 0x5646706cc410 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f7a05827740 1 -- 172.21.15.114:0/3621047179 >> 172.21.15.114:0/3621047179 conn(0x564670313300 msgr2=0x5646706c0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f7a05827740 1 -- 172.21.15.114:0/3621047179 shutdown_connections 2024-05-27T07:46:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.133+0000 7f7a05827740 1 -- 172.21.15.114:0/3621047179 wait complete. 2024-05-27T07:46:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.134+0000 7f7a05827740 20 client.16911 trim_cache size 0 max 16384 2024-05-27T07:46:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f7a05827740 1 Processor -- start 2024-05-27T07:46:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f7a05827740 1 -- start start 2024-05-27T07:46:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f7a05827740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5646702f62f0 0x5646706cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f7a05827740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702fe4d0 0x5646706e2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f7a05827740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56467031f1b0 0x5646706e2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f7a05827740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79f81b0310 con 0x5646702fe4d0 2024-05-27T07:46:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f7a05827740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79f81ad3e0 con 0x5646702f62f0 2024-05-27T07:46:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f7a05827740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79f81af570 con 0x56467031f1b0 2024-05-27T07:46:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f79f77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702fe4d0 0x5646706e2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f79f77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702fe4d0 0x5646706e2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55178/0 (socket says 172.21.15.114:55178) 2024-05-27T07:46:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f79f77fe640 1 -- 172.21.15.114:0/2849428586 learned_addr learned my addr 172.21.15.114:0/2849428586 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f79fce33640 1 --2- 172.21.15.114:0/2849428586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56467031f1b0 0x5646706e2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f79fce33640 1 --2- 172.21.15.114:0/2849428586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5646702f62f0 0x5646706cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f79f77fe640 1 -- 172.21.15.114:0/2849428586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56467031f1b0 msgr2=0x5646706e2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f79f77fe640 1 --2- 172.21.15.114:0/2849428586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56467031f1b0 0x5646706e2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f79f77fe640 1 -- 172.21.15.114:0/2849428586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5646702f62f0 msgr2=0x5646706cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.149+0000 7f79f77fe640 1 --2- 172.21.15.114:0/2849428586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5646702f62f0 0x5646706cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.150+0000 7f79f77fe640 1 -- 172.21.15.114:0/2849428586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646706c8e70 con 0x5646702fe4d0 2024-05-27T07:46:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.150+0000 7f79f77fe640 1 --2- 172.21.15.114:0/2849428586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702fe4d0 0x5646706e2400 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f79ec015010 tx=0x7f79ec005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766eca39ed83e47e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.150+0000 7f79ddffb640 1 -- 172.21.15.114:0/2849428586 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec005220 con 0x5646702fe4d0 2024-05-27T07:46:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.150+0000 7f79ddffb640 1 -- 172.21.15.114:0/2849428586 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79ec009e10 con 0x5646702fe4d0 2024-05-27T07:46:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.150+0000 7f79ddffb640 1 -- 172.21.15.114:0/2849428586 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec003930 con 0x5646702fe4d0 2024-05-27T07:46:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.151+0000 7f7a05827740 1 -- 172.21.15.114:0/2849428586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702fe4d0 msgr2=0x5646706e2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.151+0000 7f7a05827740 1 --2- 172.21.15.114:0/2849428586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702fe4d0 0x5646706e2400 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f79ec015010 tx=0x7f79ec005f90 comp rx=0 tx=0).stop 2024-05-27T07:46:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.151+0000 7f7a05827740 1 -- 172.21.15.114:0/2849428586 shutdown_connections 2024-05-27T07:46:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.151+0000 7f7a05827740 1 --2- 172.21.15.114:0/2849428586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56467031f1b0 0x5646706e2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.151+0000 7f7a05827740 1 --2- 172.21.15.114:0/2849428586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702fe4d0 0x5646706e2400 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.151+0000 7f7a05827740 1 --2- 172.21.15.114:0/2849428586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5646702f62f0 0x5646706cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.151+0000 7f7a05827740 1 -- 172.21.15.114:0/2849428586 >> 172.21.15.114:0/2849428586 conn(0x564670313300 msgr2=0x5646706cc040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.151+0000 7f7a05827740 1 -- 172.21.15.114:0/2849428586 shutdown_connections 2024-05-27T07:46:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.151+0000 7f7a05827740 1 -- 172.21.15.114:0/2849428586 wait complete. 2024-05-27T07:46:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.151+0000 7f7a05827740 1 -- start start 2024-05-27T07:46:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.151+0000 7f7a05827740 1 Processor -- start 2024-05-27T07:46:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f7a05827740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 0x5646706d2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f7a05827740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702fe4d0 0x5646706d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f7a05827740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f7a05827740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79f81aeea0 con 0x5646702f62f0 2024-05-27T07:46:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f7a05827740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79f81ada30 con 0x56467031f1b0 2024-05-27T07:46:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f7a05827740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79f81ae100 con 0x5646702fe4d0 2024-05-27T07:46:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f7a05827740 10 client.0 _ll_register_callbacks cb 0x7f79ec0137d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79fce33640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 0x5646706d2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79fce33640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 0x5646706d2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55182/0 (socket says 172.21.15.114:55182) 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79fce33640 1 -- 172.21.15.114:0/467707851 learned_addr learned my addr 172.21.15.114:0/467707851 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79f77fe640 1 --2- 172.21.15.114:0/467707851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702fe4d0 0x5646706d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79fce33640 1 --2- 172.21.15.114:0/467707851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79fce33640 1 -- 172.21.15.114:0/467707851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702fe4d0 msgr2=0x5646706d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79fce33640 1 --2- 172.21.15.114:0/467707851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702fe4d0 0x5646706d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79fce33640 1 -- 172.21.15.114:0/467707851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 msgr2=0x5646706cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79fce33640 1 --2- 172.21.15.114:0/467707851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706cff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79fce33640 1 -- 172.21.15.114:0/467707851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646706d06b0 con 0x5646702f62f0 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79fce33640 1 --2- 172.21.15.114:0/467707851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 0x5646706d2ac0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f79f0005d70 tx=0x7f79f00053c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13d70ddac86bb85 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79d77fe640 1 -- 172.21.15.114:0/467707851 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f0020070 con 0x5646702f62f0 2024-05-27T07:46:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f79d77fe640 1 -- 172.21.15.114:0/467707851 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f79f0005640 con 0x5646702f62f0 2024-05-27T07:46:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.152+0000 7f7a05827740 1 -- 172.21.15.114:0/467707851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79ec009e10 con 0x5646702f62f0 2024-05-27T07:46:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.153+0000 7f79d77fe640 1 -- 172.21.15.114:0/467707851 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f79f0043050 con 0x5646702f62f0 2024-05-27T07:46:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.153+0000 7f7a05827740 10 client.16917 fetch_fsmap learned FSMap version 37 2024-05-27T07:46:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.153+0000 7f7a05827740 1 -- 172.21.15.114:0/467707851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564670744df0 con 0x5646702f62f0 2024-05-27T07:46:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.153+0000 7f79d77fe640 1 -- 172.21.15.114:0/467707851 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79ec009e10 con 0x5646702f62f0 2024-05-27T07:46:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.153+0000 7f79d77fe640 1 -- 172.21.15.114:0/467707851 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f79f0018050 con 0x5646702f62f0 2024-05-27T07:46:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.153+0000 7f7a05827740 10 client.16917 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:46:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.153+0000 7f7a05827740 10 client.16917 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.153+0000 7f7a05827740 20 client.16917 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.153+0000 7f7a05827740 1 --2- 172.21.15.114:0/467707851 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5646702b31c0 0x564670747460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.153+0000 7f7a05827740 4 client.16917 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:46:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.153+0000 7f7a05827740 1 -- 172.21.15.114:0/467707851 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56467062e360 con 0x5646702b31c0 2024-05-27T07:46:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.153+0000 7f79f77fe640 1 --2- 172.21.15.114:0/467707851 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5646702b31c0 0x564670747460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.154+0000 7f79d77fe640 10 client.16917 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.154+0000 7f79f77fe640 1 --2- 172.21.15.114:0/467707851 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5646702b31c0 0x564670747460 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=328da3db97cb671a server_cookie=1f3070aecdaa813b in_seq=0 out_seq=0 2024-05-27T07:46:10.261 INFO:tasks.workunit.client.0.smithi114.stdout: 1 639236 12.64 MB/sec execute 142 sec latency 71.675 ms 2024-05-27T07:46:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.445+0000 7f79d77fe640 1 -- 172.21.15.114:0/467707851 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56467062e360 con 0x5646702b31c0 2024-05-27T07:46:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.445+0000 7f79d77fe640 10 client.16917 handle_command_reply: tid=0 2024-05-27T07:46:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.445+0000 7f7a05827740 1 client.16917 shutdown 2024-05-27T07:46:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.445+0000 7f7a05827740 1 -- 172.21.15.114:0/467707851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 msgr2=0x5646706d2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.445+0000 7f7a05827740 1 --2- 172.21.15.114:0/467707851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 0x5646706d2ac0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f79f0005d70 tx=0x7f79f00053c0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.446+0000 7f7a05827740 1 -- 172.21.15.114:0/467707851 shutdown_connections 2024-05-27T07:46:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.446+0000 7f7a05827740 1 --2- 172.21.15.114:0/467707851 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5646702b31c0 0x564670747460 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.446+0000 7f79d77fe640 0 client.16917 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.446+0000 7f7a05827740 1 --2- 172.21.15.114:0/467707851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56467031f1b0 0x5646706cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.446+0000 7f7a05827740 1 --2- 172.21.15.114:0/467707851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5646702fe4d0 0x5646706d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.446+0000 7f7a05827740 1 --2- 172.21.15.114:0/467707851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5646702f62f0 0x5646706d2ac0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.446+0000 7f7a05827740 1 -- 172.21.15.114:0/467707851 >> 172.21.15.114:0/467707851 conn(0x564670313300 msgr2=0x5646706c3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.446+0000 7f7a05827740 1 -- 172.21.15.114:0/467707851 shutdown_connections 2024-05-27T07:46:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.446+0000 7f7a05827740 1 -- 172.21.15.114:0/467707851 wait complete. 2024-05-27T07:46:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.446+0000 7f7a05827740 20 client.16917 trim_cache size 0 max 16384 2024-05-27T07:46:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:46:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:46:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:46:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.447+0000 7f79fe896640 1 -- 172.21.15.114:0/2152553276 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f79e00b29d0 msgr2=0x7f79e00b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.447+0000 7f79fe896640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f79e00b29d0 0x7f79e00b4ee0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f79e81000b0 tx=0x7f79e810b040 comp rx=0 tx=0).stop 2024-05-27T07:46:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.447+0000 7f79fe896640 1 -- 172.21.15.114:0/2152553276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f81032d0 msgr2=0x7f79f81a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.447+0000 7f79fe896640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f81032d0 0x7f79f81a05e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f79e8032fd0 tx=0x7f79e8031a20 comp rx=0 tx=0).stop 2024-05-27T07:46:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.447+0000 7f79fe896640 1 -- 172.21.15.114:0/2152553276 shutdown_connections 2024-05-27T07:46:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.447+0000 7f79fe896640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f79e00b29d0 0x7f79e00b4ee0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.447+0000 7f79fe896640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79f8109020 0x7f79f819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.447+0000 7f79fe896640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79f8108b80 0x7f79f81a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.447+0000 7f79fe896640 1 --2- 172.21.15.114:0/2152553276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f81032d0 0x7f79f81a05e0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.447+0000 7f79fe896640 1 -- 172.21.15.114:0/2152553276 >> 172.21.15.114:0/2152553276 conn(0x7f79f80fe380 msgr2=0x7f79f8100c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.447+0000 7f79fe896640 1 -- 172.21.15.114:0/2152553276 shutdown_connections 2024-05-27T07:46:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:10.447+0000 7f79fe896640 1 -- 172.21.15.114:0/2152553276 wait complete. 2024-05-27T07:46:11.261 INFO:tasks.workunit.client.0.smithi114.stdout: 1 640691 12.60 MB/sec execute 143 sec latency 95.744 ms 2024-05-27T07:46:11.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:11 smithi163 ceph-mon[36451]: pgmap v204: 129 pgs: 9 active+clean+snaptrim, 94 active+clean, 26 active+clean+snaptrim_wait; 86 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 320 op/s 2024-05-27T07:46:11.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:11 smithi186 ceph-mon[36647]: pgmap v204: 129 pgs: 9 active+clean+snaptrim, 94 active+clean, 26 active+clean+snaptrim_wait; 86 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 320 op/s 2024-05-27T07:46:11.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:11 smithi114 ceph-mon[32038]: pgmap v204: 129 pgs: 9 active+clean+snaptrim, 94 active+clean, 26 active+clean+snaptrim_wait; 86 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 320 op/s 2024-05-27T07:46:12.261 INFO:tasks.workunit.client.0.smithi114.stdout: 1 643532 12.62 MB/sec execute 144 sec latency 14.516 ms 2024-05-27T07:46:12.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:12 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:46:12.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:12 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:46:12.766 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:12 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:46:13.261 INFO:tasks.workunit.client.0.smithi114.stdout: 1 646419 12.65 MB/sec execute 145 sec latency 80.894 ms 2024-05-27T07:46:13.446 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:12 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:46:13.446 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:12 smithi186 systemd[1]: Stopped Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:46:13.446 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:12 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Consumed 3.116s CPU time. 2024-05-27T07:46:13.446 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:12 smithi186 systemd[1]: Starting Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:46:13.446 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:13 smithi186 podman[73009]: 2024-05-27 07:46:13.115089849 +0000 UTC m=+0.017297135 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:46:13.629 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:13 smithi114 ceph-mon[32038]: pgmap v205: 129 pgs: 129 active+clean; 85 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 310 op/s 2024-05-27T07:46:13.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:13 smithi163 ceph-mon[36451]: pgmap v205: 129 pgs: 129 active+clean; 85 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 310 op/s 2024-05-27T07:46:13.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:13 smithi186 ceph-mon[36647]: pgmap v205: 129 pgs: 129 active+clean; 85 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 310 op/s 2024-05-27T07:46:13.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.745+0000 7f9a3aee9700 1 -- 172.21.15.114:0/1103816718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a34108750 msgr2=0x7f9a341100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:13.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.745+0000 7f9a3aee9700 1 --2- 172.21.15.114:0/1103816718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a34108750 0x7f9a341100b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0099c0 tx=0x7f9a1c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.745+0000 7f9a3aee9700 1 -- 172.21.15.114:0/1103816718 shutdown_connections 2024-05-27T07:46:13.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.745+0000 7f9a3aee9700 1 --2- 172.21.15.114:0/1103816718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a34108750 0x7f9a341100b0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.745+0000 7f9a3aee9700 1 --2- 172.21.15.114:0/1103816718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a341034d0 0x7f9a341038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.745+0000 7f9a3aee9700 1 --2- 172.21.15.114:0/1103816718 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a34107e00 0x7f9a34108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.745+0000 7f9a3aee9700 1 -- 172.21.15.114:0/1103816718 >> 172.21.15.114:0/1103816718 conn(0x7f9a340fef40 msgr2=0x7f9a34101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:13.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.745+0000 7f9a3aee9700 1 -- 172.21.15.114:0/1103816718 shutdown_connections 2024-05-27T07:46:13.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.745+0000 7f9a3aee9700 1 -- 172.21.15.114:0/1103816718 wait complete. 2024-05-27T07:46:13.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.746+0000 7f9a3aee9700 1 Processor -- start 2024-05-27T07:46:13.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.746+0000 7f9a3aee9700 1 -- start start 2024-05-27T07:46:13.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a3aee9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a341034d0 0x7f9a3419bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a3aee9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a34107e00 0x7f9a3419c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a3aee9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a34108750 0x7f9a3419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a3aee9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a34113380 con 0x7f9a34107e00 2024-05-27T07:46:13.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a3aee9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9a34113200 con 0x7f9a34108750 2024-05-27T07:46:13.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a3aee9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a34113500 con 0x7f9a341034d0 2024-05-27T07:46:13.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a38c85700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a341034d0 0x7f9a3419bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a38c85700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a341034d0 0x7f9a3419bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47750/0 (socket says 172.21.15.114:47750) 2024-05-27T07:46:13.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a38c85700 1 -- 172.21.15.114:0/676505036 learned_addr learned my addr 172.21.15.114:0/676505036 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:13.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a39486700 1 --2- 172.21.15.114:0/676505036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a34108750 0x7f9a3419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a38c85700 1 -- 172.21.15.114:0/676505036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a34108750 msgr2=0x7f9a3419c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:13.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a38c85700 1 --2- 172.21.15.114:0/676505036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a34108750 0x7f9a3419c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a38c85700 1 -- 172.21.15.114:0/676505036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a34107e00 msgr2=0x7f9a3419c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:13.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a38c85700 1 --2- 172.21.15.114:0/676505036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a34107e00 0x7f9a3419c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.747+0000 7f9a38c85700 1 -- 172.21.15.114:0/676505036 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a341a6f50 con 0x7f9a341034d0 2024-05-27T07:46:13.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.748+0000 7f9a38c85700 1 --2- 172.21.15.114:0/676505036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a341034d0 0x7f9a3419bbe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00e9a0 tx=0x7f9a2c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:13.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.748+0000 7f9a31ffb700 1 -- 172.21.15.114:0/676505036 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c00cbd0 con 0x7f9a341034d0 2024-05-27T07:46:13.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.748+0000 7f9a31ffb700 1 -- 172.21.15.114:0/676505036 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a2c00cd70 con 0x7f9a341034d0 2024-05-27T07:46:13.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.748+0000 7f9a31ffb700 1 -- 172.21.15.114:0/676505036 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c018680 con 0x7f9a341034d0 2024-05-27T07:46:13.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.748+0000 7f9a3aee9700 1 -- 172.21.15.114:0/676505036 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a34199a30 con 0x7f9a341034d0 2024-05-27T07:46:13.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.748+0000 7f9a3aee9700 1 -- 172.21.15.114:0/676505036 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a34199e60 con 0x7f9a341034d0 2024-05-27T07:46:13.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.751+0000 7f9a31ffb700 1 -- 172.21.15.114:0/676505036 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f9a2c018820 con 0x7f9a341034d0 2024-05-27T07:46:13.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.752+0000 7f9a31ffb700 1 --2- 172.21.15.114:0/676505036 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a200b2a70 0x7f9a200b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.752+0000 7f9a33fff700 1 --2- 172.21.15.114:0/676505036 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a200b2a70 0x7f9a200b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.752+0000 7f9a31ffb700 1 -- 172.21.15.114:0/676505036 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f9a2c0e2db0 con 0x7f9a341034d0 2024-05-27T07:46:13.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.752+0000 7f9a498feb80 1 Processor -- start 2024-05-27T07:46:13.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.752+0000 7f9a33fff700 1 --2- 172.21.15.114:0/676505036 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a200b2a70 0x7f9a200b4f80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9a34102a00 tx=0x7f9a2400d040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:13.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.752+0000 7f9a498feb80 1 -- start start 2024-05-27T07:46:13.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.752+0000 7f9a498feb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d63de40 0x55aa9d7c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.752+0000 7f9a498feb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d7c3070 0x55aa9d7bcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.752+0000 7f9a498feb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7bd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.752+0000 7f9a498feb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55aa9d6ead70 con 0x55aa9d7bd120 2024-05-27T07:46:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a498feb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55aa9d6da910 con 0x55aa9d7c3070 2024-05-27T07:46:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a498feb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55aa9d6de950 con 0x55aa9d63de40 2024-05-27T07:46:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a38c85700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d63de40 0x55aa9d7c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a38c85700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d63de40 0x55aa9d7c2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47764/0 (socket says 172.21.15.114:47764) 2024-05-27T07:46:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a38c85700 1 -- 172.21.15.114:0/2192305739 learned_addr learned my addr 172.21.15.114:0/2192305739 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a39486700 1 --2- 172.21.15.114:0/2192305739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7bd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a33fff700 1 --2- 172.21.15.114:0/2192305739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d7c3070 0x55aa9d7bcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a38c85700 1 -- 172.21.15.114:0/2192305739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d7c3070 msgr2=0x55aa9d7bcbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a38c85700 1 --2- 172.21.15.114:0/2192305739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d7c3070 0x55aa9d7bcbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a38c85700 1 -- 172.21.15.114:0/2192305739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 msgr2=0x55aa9d7bd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a38c85700 1 --2- 172.21.15.114:0/2192305739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7bd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a38c85700 1 -- 172.21.15.114:0/2192305739 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa9d683fb0 con 0x55aa9d63de40 2024-05-27T07:46:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a33fff700 1 --2- 172.21.15.114:0/2192305739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d7c3070 0x55aa9d7bcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:46:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a38c85700 1 --2- 172.21.15.114:0/2192305739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d63de40 0x55aa9d7c2b30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0a9930 tx=0x7f9a2c0a9810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f701292efe7fffc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a2affd700 1 -- 172.21.15.114:0/2192305739 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c0adfe0 con 0x55aa9d63de40 2024-05-27T07:46:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a2affd700 1 -- 172.21.15.114:0/2192305739 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a2c0b2a90 con 0x55aa9d63de40 2024-05-27T07:46:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.753+0000 7f9a2affd700 1 -- 172.21.15.114:0/2192305739 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c0b2860 con 0x55aa9d63de40 2024-05-27T07:46:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.754+0000 7f9a498feb80 1 -- 172.21.15.114:0/2192305739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d63de40 msgr2=0x55aa9d7c2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.754+0000 7f9a498feb80 1 --2- 172.21.15.114:0/2192305739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d63de40 0x55aa9d7c2b30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0a9930 tx=0x7f9a2c0a9810 comp rx=0 tx=0).stop 2024-05-27T07:46:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.754+0000 7f9a498feb80 1 -- 172.21.15.114:0/2192305739 shutdown_connections 2024-05-27T07:46:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.754+0000 7f9a498feb80 1 --2- 172.21.15.114:0/2192305739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7bd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.754+0000 7f9a498feb80 1 --2- 172.21.15.114:0/2192305739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d63de40 0x55aa9d7c2b30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.754+0000 7f9a498feb80 1 --2- 172.21.15.114:0/2192305739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d7c3070 0x55aa9d7bcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.754+0000 7f9a498feb80 1 -- 172.21.15.114:0/2192305739 >> 172.21.15.114:0/2192305739 conn(0x55aa9d640770 msgr2=0x55aa9d7b7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.755+0000 7f9a498feb80 1 -- 172.21.15.114:0/2192305739 shutdown_connections 2024-05-27T07:46:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.755+0000 7f9a498feb80 1 -- 172.21.15.114:0/2192305739 wait complete. 2024-05-27T07:46:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.755+0000 7f9a498feb80 1 -- start start 2024-05-27T07:46:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.755+0000 7f9a498feb80 1 Processor -- start 2024-05-27T07:46:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a498feb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d63de40 0x55aa9d7c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a498feb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7bd120 0x55aa9d7c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a498feb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d7c3070 0x55aa9d7c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a498feb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55aa9d5e0fa0 con 0x55aa9d63de40 2024-05-27T07:46:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a498feb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55aa9d642da0 con 0x55aa9d7c3070 2024-05-27T07:46:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a498feb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55aa9d7c9b60 con 0x55aa9d7bd120 2024-05-27T07:46:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a498feb80 10 client.0 _ll_register_callbacks cb 0x55aa9d6715b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a33fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7bd120 0x55aa9d7c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a38c85700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d63de40 0x55aa9d7c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a33fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7bd120 0x55aa9d7c6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47772/0 (socket says 172.21.15.114:47772) 2024-05-27T07:46:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a33fff700 1 -- 172.21.15.114:0/3674478098 learned_addr learned my addr 172.21.15.114:0/3674478098 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a39486700 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d7c3070 0x55aa9d7c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a33fff700 1 -- 172.21.15.114:0/3674478098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d7c3070 msgr2=0x55aa9d7c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a33fff700 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d7c3070 0x55aa9d7c7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a33fff700 1 -- 172.21.15.114:0/3674478098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d63de40 msgr2=0x55aa9d7c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a33fff700 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d63de40 0x55aa9d7c69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a33fff700 1 -- 172.21.15.114:0/3674478098 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa9d62a8d0 con 0x55aa9d7bd120 2024-05-27T07:46:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a39486700 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d7c3070 0x55aa9d7c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:46:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.756+0000 7f9a38c85700 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d63de40 0x55aa9d7c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.757+0000 7f9a33fff700 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7bd120 0x55aa9d7c6ef0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9a24016e00 tx=0x7f9a24009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63284412c47ad2cd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.757+0000 7f9a13fff700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:46:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.757+0000 7f9a13fff700 1 -- 172.21.15.114:0/3674478098 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a24004bb0 con 0x55aa9d7bd120 2024-05-27T07:46:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.757+0000 7f9a13fff700 1 -- 172.21.15.114:0/3674478098 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f9a24004d50 con 0x55aa9d7bd120 2024-05-27T07:46:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.757+0000 7f9a498feb80 1 -- 172.21.15.114:0/3674478098 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a2c0b2a90 con 0x55aa9d7bd120 2024-05-27T07:46:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.757+0000 7f9a13fff700 1 -- 172.21.15.114:0/3674478098 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9a2402f050 con 0x55aa9d7bd120 2024-05-27T07:46:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.757+0000 7f9a498feb80 10 client.26515 fetch_fsmap learned FSMap version 37 2024-05-27T07:46:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.757+0000 7f9a498feb80 1 -- 172.21.15.114:0/3674478098 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55aa9d694760 con 0x55aa9d7bd120 2024-05-27T07:46:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.757+0000 7f9a13fff700 1 -- 172.21.15.114:0/3674478098 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a2c0b2a90 con 0x55aa9d7bd120 2024-05-27T07:46:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.758+0000 7f9a13fff700 1 -- 172.21.15.114:0/3674478098 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f9a24022b50 con 0x55aa9d7bd120 2024-05-27T07:46:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.758+0000 7f9a498feb80 10 client.26515 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:46:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.758+0000 7f9a498feb80 10 client.26515 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.758+0000 7f9a498feb80 20 client.26515 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.758+0000 7f9a498feb80 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa9d7df420 0x55aa9d7e1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.758+0000 7f9a498feb80 4 client.26515 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:46:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.758+0000 7f9a498feb80 1 -- 172.21.15.114:0/3674478098 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa9d6eccc0 con 0x55aa9d7df420 2024-05-27T07:46:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.758+0000 7f9a38c85700 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa9d7df420 0x55aa9d7e1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.758+0000 7f9a13fff700 10 client.26515 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.758+0000 7f9a38c85700 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa9d7df420 0x55aa9d7e1840 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfc506d2e175579a server_cookie=30cb04a4fe107c0b in_seq=0 out_seq=0 2024-05-27T07:46:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.759+0000 7f9a13fff700 1 -- 172.21.15.114:0/3674478098 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55aa9d6eccc0 con 0x55aa9d7df420 2024-05-27T07:46:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.759+0000 7f9a13fff700 10 client.26515 handle_command_reply: tid=0 2024-05-27T07:46:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.759+0000 7f9a498feb80 1 client.26515 shutdown 2024-05-27T07:46:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.759+0000 7f9a498feb80 1 -- 172.21.15.114:0/3674478098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7bd120 msgr2=0x55aa9d7c6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.759+0000 7f9a498feb80 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7bd120 0x55aa9d7c6ef0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9a24016e00 tx=0x7f9a24009920 comp rx=0 tx=0).stop 2024-05-27T07:46:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.759+0000 7f9a498feb80 1 -- 172.21.15.114:0/3674478098 shutdown_connections 2024-05-27T07:46:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.759+0000 7f9a498feb80 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa9d7df420 0x55aa9d7e1840 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.759+0000 7f9a498feb80 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d7c3070 0x55aa9d7c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.759+0000 7f9a498feb80 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d63de40 0x55aa9d7c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.759+0000 7f9a13fff700 0 client.26515 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.759+0000 7f9a498feb80 1 --2- 172.21.15.114:0/3674478098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7bd120 0x55aa9d7c6ef0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.759+0000 7f9a498feb80 1 -- 172.21.15.114:0/3674478098 >> 172.21.15.114:0/3674478098 conn(0x55aa9d640770 msgr2=0x55aa9d7bbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.760+0000 7f9a498feb80 1 -- 172.21.15.114:0/3674478098 shutdown_connections 2024-05-27T07:46:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.760+0000 7f9a498feb80 1 -- 172.21.15.114:0/3674478098 wait complete. 2024-05-27T07:46:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.760+0000 7f9a498feb80 20 client.26515 trim_cache size 0 max 16384 2024-05-27T07:46:13.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a498feb80 1 Processor -- start 2024-05-27T07:46:13.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a498feb80 1 -- start start 2024-05-27T07:46:13.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a498feb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d63de40 0x55aa9d7bc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a498feb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7c7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a498feb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7c3070 0x55aa9d7db490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a498feb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a341af640 con 0x55aa9d7bd120 2024-05-27T07:46:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a498feb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9a341ac710 con 0x55aa9d63de40 2024-05-27T07:46:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a498feb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a341ae8a0 con 0x55aa9d7c3070 2024-05-27T07:46:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a33fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7c7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a33fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7c7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55214/0 (socket says 172.21.15.114:55214) 2024-05-27T07:46:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a33fff700 1 -- 172.21.15.114:0/3445102676 learned_addr learned my addr 172.21.15.114:0/3445102676 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a39486700 1 --2- 172.21.15.114:0/3445102676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7c3070 0x55aa9d7db490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a38c85700 1 --2- 172.21.15.114:0/3445102676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d63de40 0x55aa9d7bc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a33fff700 1 -- 172.21.15.114:0/3445102676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7c3070 msgr2=0x55aa9d7db490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a33fff700 1 --2- 172.21.15.114:0/3445102676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7c3070 0x55aa9d7db490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a33fff700 1 -- 172.21.15.114:0/3445102676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d63de40 msgr2=0x55aa9d7bc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a33fff700 1 --2- 172.21.15.114:0/3445102676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d63de40 0x55aa9d7bc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a33fff700 1 -- 172.21.15.114:0/3445102676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa9d7b12f0 con 0x55aa9d7bd120 2024-05-27T07:46:13.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.784+0000 7f9a38c85700 1 --2- 172.21.15.114:0/3445102676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d63de40 0x55aa9d7bc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:46:13.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.785+0000 7f9a33fff700 1 --2- 172.21.15.114:0/3445102676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7c7270 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9a24004b10 tx=0x7f9a240060a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=740087cf4da82fb7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:13.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.785+0000 7f9a2a7fc700 1 -- 172.21.15.114:0/3445102676 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a24037070 con 0x55aa9d7bd120 2024-05-27T07:46:13.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.785+0000 7f9a2a7fc700 1 -- 172.21.15.114:0/3445102676 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a24005a50 con 0x55aa9d7bd120 2024-05-27T07:46:13.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.785+0000 7f9a2a7fc700 1 -- 172.21.15.114:0/3445102676 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a24015ea0 con 0x55aa9d7bd120 2024-05-27T07:46:13.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.786+0000 7f9a498feb80 1 -- 172.21.15.114:0/3445102676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 msgr2=0x55aa9d7c7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.786+0000 7f9a498feb80 1 --2- 172.21.15.114:0/3445102676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7c7270 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9a24004b10 tx=0x7f9a240060a0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.786+0000 7f9a498feb80 1 -- 172.21.15.114:0/3445102676 shutdown_connections 2024-05-27T07:46:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.786+0000 7f9a498feb80 1 --2- 172.21.15.114:0/3445102676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7c3070 0x55aa9d7db490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.786+0000 7f9a498feb80 1 --2- 172.21.15.114:0/3445102676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d63de40 0x55aa9d7bc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.786+0000 7f9a498feb80 1 --2- 172.21.15.114:0/3445102676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7c7270 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.786+0000 7f9a498feb80 1 -- 172.21.15.114:0/3445102676 >> 172.21.15.114:0/3445102676 conn(0x55aa9d640770 msgr2=0x55aa9d7c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.786+0000 7f9a498feb80 1 -- 172.21.15.114:0/3445102676 shutdown_connections 2024-05-27T07:46:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.786+0000 7f9a498feb80 1 -- 172.21.15.114:0/3445102676 wait complete. 2024-05-27T07:46:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.786+0000 7f9a498feb80 1 -- start start 2024-05-27T07:46:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.786+0000 7f9a498feb80 1 Processor -- start 2024-05-27T07:46:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a498feb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d63de40 0x55aa9d7d7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a498feb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7d7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a498feb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7c3070 0x55aa9d7e5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a498feb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a341ae1d0 con 0x55aa9d7bd120 2024-05-27T07:46:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a498feb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9a341acd60 con 0x55aa9d63de40 2024-05-27T07:46:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a498feb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a341ad430 con 0x55aa9d7c3070 2024-05-27T07:46:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a33fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7d7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a498feb80 10 client.0 _ll_register_callbacks cb 0x55aa9d6715b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a33fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7d7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55228/0 (socket says 172.21.15.114:55228) 2024-05-27T07:46:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a33fff700 1 -- 172.21.15.114:0/1895383735 learned_addr learned my addr 172.21.15.114:0/1895383735 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a39486700 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7c3070 0x55aa9d7e5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a38c85700 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d63de40 0x55aa9d7d7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a33fff700 1 -- 172.21.15.114:0/1895383735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7c3070 msgr2=0x55aa9d7e5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a33fff700 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7c3070 0x55aa9d7e5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a33fff700 1 -- 172.21.15.114:0/1895383735 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d63de40 msgr2=0x55aa9d7d7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a33fff700 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d63de40 0x55aa9d7d7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a33fff700 1 -- 172.21.15.114:0/1895383735 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa9d7e6530 con 0x55aa9d7bd120 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a39486700 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7c3070 0x55aa9d7e5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a33fff700 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7d7bc0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9a24016e00 tx=0x7f9a24015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5465440fb71b39cb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.787+0000 7f9a13fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a13fff700 1 -- 172.21.15.114:0/1895383735 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a24037040 con 0x55aa9d7bd120 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a13fff700 1 -- 172.21.15.114:0/1895383735 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f9a24004dc0 con 0x55aa9d7bd120 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a498feb80 1 -- 172.21.15.114:0/1895383735 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a24005a50 con 0x55aa9d7bd120 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a13fff700 1 -- 172.21.15.114:0/1895383735 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9a240051c0 con 0x55aa9d7bd120 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a498feb80 10 client.16932 fetch_fsmap learned FSMap version 37 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a498feb80 1 -- 172.21.15.114:0/1895383735 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55aa9d8494f0 con 0x55aa9d7bd120 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a13fff700 1 -- 172.21.15.114:0/1895383735 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a24005a50 con 0x55aa9d7bd120 2024-05-27T07:46:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a13fff700 1 -- 172.21.15.114:0/1895383735 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 7725+0+0 (secure 0 0 0) 0x7f9a24022740 con 0x55aa9d7bd120 2024-05-27T07:46:13.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a498feb80 10 client.16932 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:46:13.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a498feb80 10 client.16932 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:13.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a498feb80 20 client.16932 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:13.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a498feb80 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa9d7df420 0x55aa9d84ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:13.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a498feb80 4 client.16932 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:46:13.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a498feb80 1 -- 172.21.15.114:0/1895383735 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55aa9d6e64a0 con 0x55aa9d7df420 2024-05-27T07:46:13.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.788+0000 7f9a38c85700 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa9d7df420 0x55aa9d84ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:13.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a13fff700 10 client.16932 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:13.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a38c85700 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa9d7df420 0x55aa9d84ba50 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4efe899e1600f94c server_cookie=3d2fd46e2988b581 in_seq=0 out_seq=0 2024-05-27T07:46:13.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a13fff700 1 -- 172.21.15.114:0/1895383735 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55aa9d6e64a0 con 0x55aa9d7df420 2024-05-27T07:46:13.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a13fff700 10 client.16932 handle_command_reply: tid=0 2024-05-27T07:46:13.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a498feb80 1 client.16932 shutdown 2024-05-27T07:46:13.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a498feb80 1 -- 172.21.15.114:0/1895383735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 msgr2=0x55aa9d7d7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:13.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a498feb80 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7d7bc0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9a24016e00 tx=0x7f9a24015000 comp rx=0 tx=0).stop 2024-05-27T07:46:13.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a498feb80 1 -- 172.21.15.114:0/1895383735 shutdown_connections 2024-05-27T07:46:13.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a498feb80 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa9d7c3070 0x55aa9d7e5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a498feb80 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa9d7df420 0x55aa9d84ba50 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a498feb80 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa9d63de40 0x55aa9d7d7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a498feb80 1 --2- 172.21.15.114:0/1895383735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa9d7bd120 0x55aa9d7d7bc0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a13fff700 0 client.16932 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:13.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a498feb80 1 -- 172.21.15.114:0/1895383735 >> 172.21.15.114:0/1895383735 conn(0x55aa9d640770 msgr2=0x55aa9d7b14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:13.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.789+0000 7f9a498feb80 1 -- 172.21.15.114:0/1895383735 shutdown_connections 2024-05-27T07:46:13.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.790+0000 7f9a498feb80 1 -- 172.21.15.114:0/1895383735 wait complete. 2024-05-27T07:46:13.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.790+0000 7f9a498feb80 20 client.16932 trim_cache size 0 max 16384 2024-05-27T07:46:13.795 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:46:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:46:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:46:13.796 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:46:13.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.790+0000 7f9a3aee9700 1 -- 172.21.15.114:0/676505036 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a200b2a70 msgr2=0x7f9a200b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:13.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.790+0000 7f9a3aee9700 1 --2- 172.21.15.114:0/676505036 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a200b2a70 0x7f9a200b4f80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9a34102a00 tx=0x7f9a2400d040 comp rx=0 tx=0).stop 2024-05-27T07:46:13.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.790+0000 7f9a3aee9700 1 -- 172.21.15.114:0/676505036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a341034d0 msgr2=0x7f9a3419bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:13.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.790+0000 7f9a3aee9700 1 --2- 172.21.15.114:0/676505036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a341034d0 0x7f9a3419bbe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00e9a0 tx=0x7f9a2c00ecb0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.790+0000 7f9a3aee9700 1 -- 172.21.15.114:0/676505036 shutdown_connections 2024-05-27T07:46:13.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.790+0000 7f9a3aee9700 1 --2- 172.21.15.114:0/676505036 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a200b2a70 0x7f9a200b4f80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.790+0000 7f9a3aee9700 1 --2- 172.21.15.114:0/676505036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a34108750 0x7f9a3419c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.790+0000 7f9a3aee9700 1 --2- 172.21.15.114:0/676505036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a341034d0 0x7f9a3419bbe0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.790+0000 7f9a3aee9700 1 --2- 172.21.15.114:0/676505036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a34107e00 0x7f9a3419c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:13.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.790+0000 7f9a3aee9700 1 -- 172.21.15.114:0/676505036 >> 172.21.15.114:0/676505036 conn(0x7f9a340fef40 msgr2=0x7f9a34107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:13.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.791+0000 7f9a3aee9700 1 -- 172.21.15.114:0/676505036 shutdown_connections 2024-05-27T07:46:13.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:13.791+0000 7f9a3aee9700 1 -- 172.21.15.114:0/676505036 wait complete. 2024-05-27T07:46:14.261 INFO:tasks.workunit.client.0.smithi114.stdout: 1 647110 12.57 MB/sec execute 146 sec latency 158.739 ms 2024-05-27T07:46:14.743 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:46:14.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:46:15.037 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:46:15.228 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:14 smithi186 podman[73009]: 2024-05-27 07:46:14.825806902 +0000 UTC m=+1.728014169 container create f66355e80bc2ff85613798be25f68f3013675016b2dbd047f6760c25a3daab9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2) 2024-05-27T07:46:15.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 648441 12.53 MB/sec execute 147 sec latency 213.822 ms 2024-05-27T07:46:15.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:15 smithi114 ceph-mon[32038]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T07:46:15.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:15 smithi114 ceph-mon[32038]: pgmap v207: 129 pgs: 129 active+clean; 85 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 310 op/s 2024-05-27T07:46:15.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:15 smithi163 ceph-mon[36451]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T07:46:15.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:15 smithi163 ceph-mon[36451]: pgmap v207: 129 pgs: 129 active+clean; 85 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 310 op/s 2024-05-27T07:46:15.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:15 smithi186 ceph-mon[36647]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T07:46:15.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:15 smithi186 ceph-mon[36647]: pgmap v207: 129 pgs: 129 active+clean; 85 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 310 op/s 2024-05-27T07:46:16.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 650043 12.52 MB/sec execute 148 sec latency 169.146 ms 2024-05-27T07:46:16.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.941+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/2796043556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2f81034b0 msgr2=0x7fc2f81038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:16.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.941+0000 7fc2ff1f6700 1 --2- 172.21.15.114:0/2796043556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2f81034b0 0x7fc2f81038c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0099c0 tx=0x7fc2ec009cd0 comp rx=0 tx=0).stop 2024-05-27T07:46:16.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.941+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/2796043556 shutdown_connections 2024-05-27T07:46:16.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.941+0000 7fc2ff1f6700 1 --2- 172.21.15.114:0/2796043556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2f8108730 0x7fc2f8110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:16.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.941+0000 7fc2ff1f6700 1 --2- 172.21.15.114:0/2796043556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2f81034b0 0x7fc2f81038c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:16.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.941+0000 7fc2ff1f6700 1 --2- 172.21.15.114:0/2796043556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107de0 0x7fc2f81081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:16.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.941+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/2796043556 >> 172.21.15.114:0/2796043556 conn(0x7fc2f80fedb0 msgr2=0x7fc2f81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:16.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.941+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/2796043556 shutdown_connections 2024-05-27T07:46:16.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.941+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/2796043556 wait complete. 2024-05-27T07:46:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.942+0000 7fc2ff1f6700 1 Processor -- start 2024-05-27T07:46:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.942+0000 7fc2ff1f6700 1 -- start start 2024-05-27T07:46:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2ff1f6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2f81034b0 0x7fc2f819f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2ff1f6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107de0 0x7fc2f819fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2ff1f6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2f8108730 0x7fc2f8199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2ff1f6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f8113360 con 0x7fc2f81034b0 2024-05-27T07:46:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2ff1f6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f81131e0 con 0x7fc2f8108730 2024-05-27T07:46:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2ff1f6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f81134e0 con 0x7fc2f8107de0 2024-05-27T07:46:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2f7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107de0 0x7fc2f819fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2fcf92700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2f81034b0 0x7fc2f819f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2f7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107de0 0x7fc2f819fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36678/0 (socket says 172.21.15.114:36678) 2024-05-27T07:46:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2f7fff700 1 -- 172.21.15.114:0/3058557709 learned_addr learned my addr 172.21.15.114:0/3058557709 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2fd793700 1 --2- 172.21.15.114:0/3058557709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2f8108730 0x7fc2f8199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2f7fff700 1 -- 172.21.15.114:0/3058557709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2f8108730 msgr2=0x7fc2f8199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2f7fff700 1 --2- 172.21.15.114:0/3058557709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2f8108730 0x7fc2f8199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2f7fff700 1 -- 172.21.15.114:0/3058557709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2f81034b0 msgr2=0x7fc2f819f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2f7fff700 1 --2- 172.21.15.114:0/3058557709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2f81034b0 0x7fc2f819f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2f7fff700 1 -- 172.21.15.114:0/3058557709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f819a1c0 con 0x7fc2f8107de0 2024-05-27T07:46:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.943+0000 7fc2f7fff700 1 --2- 172.21.15.114:0/3058557709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107de0 0x7fc2f819fe00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400e970 tx=0x7fc2e400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.944+0000 7fc2f5ffb700 1 -- 172.21.15.114:0/3058557709 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4004a30 con 0x7fc2f8107de0 2024-05-27T07:46:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.944+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/3058557709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f819a4b0 con 0x7fc2f8107de0 2024-05-27T07:46:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.944+0000 7fc2f5ffb700 1 -- 172.21.15.114:0/3058557709 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2e400cc90 con 0x7fc2f8107de0 2024-05-27T07:46:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.944+0000 7fc2f5ffb700 1 -- 172.21.15.114:0/3058557709 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4005610 con 0x7fc2f8107de0 2024-05-27T07:46:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.944+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/3058557709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f81a6750 con 0x7fc2f8107de0 2024-05-27T07:46:16.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.945+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/3058557709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f80512b0 con 0x7fc2f8107de0 2024-05-27T07:46:16.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.947+0000 7fc2f5ffb700 1 -- 172.21.15.114:0/3058557709 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fc2e400c7e0 con 0x7fc2f8107de0 2024-05-27T07:46:16.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.947+0000 7fc2f5ffb700 1 --2- 172.21.15.114:0/3058557709 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2e80b29f0 0x7fc2e80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:16.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.948+0000 7fc2fcf92700 1 --2- 172.21.15.114:0/3058557709 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2e80b29f0 0x7fc2e80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:16.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.948+0000 7fc2f5ffb700 1 -- 172.21.15.114:0/3058557709 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9554+0+0 (secure 0 0 0) 0x7fc2e40e2e10 con 0x7fc2f8107de0 2024-05-27T07:46:16.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.948+0000 7fc2fcf92700 1 --2- 172.21.15.114:0/3058557709 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2e80b29f0 0x7fc2e80b4f00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec000c00 tx=0x7fc2ec01a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:16.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:16.951+0000 7fc2f5ffb700 1 -- 172.21.15.114:0/3058557709 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc2e4054df0 con 0x7fc2f8107de0 2024-05-27T07:46:16.955 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:16 smithi186 podman[73009]: 2024-05-27 07:46:16.713242972 +0000 UTC m=+3.615450251 container init f66355e80bc2ff85613798be25f68f3013675016b2dbd047f6760c25a3daab9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-27T07:46:16.955 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:16 smithi186 podman[73009]: 2024-05-27 07:46:16.717085389 +0000 UTC m=+3.619292657 container start f66355e80bc2ff85613798be25f68f3013675016b2dbd047f6760c25a3daab9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-05-27T07:46:16.955 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:16 smithi186 ceph-mds[73029]: -- 172.21.15.186:0/2177054265 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563c03bbda40 con 0x563c03c05600 2024-05-27T07:46:16.956 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: starting mds.f at 2024-05-27T07:46:16.956 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:16 smithi186 bash[73009]: f66355e80bc2ff85613798be25f68f3013675016b2dbd047f6760c25a3daab9f 2024-05-27T07:46:16.956 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:16 smithi186 systemd[1]: Started Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:46:17.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.231+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/3058557709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2f819afd0 con 0x7fc2f8107de0 2024-05-27T07:46:17.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.232+0000 7fc2f5ffb700 1 -- 172.21.15.114:0/3058557709 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11000 (secure 0 0 0) 0x7fc2e40aa3e0 con 0x7fc2f8107de0 2024-05-27T07:46:17.232 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:46:17.232 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":26227,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2936519309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2936519309},{"type":"v1","addr":"172.21.15.186:6835","nonce":2936519309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:45:32.379189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26305":{"gid":26305,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/684229511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":684229511},{"type":"v1","addr":"172.21.15.186:6837","nonce":684229511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:46:17.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.234+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/3058557709 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2e80b29f0 msgr2=0x7fc2e80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:17.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.234+0000 7fc2ff1f6700 1 --2- 172.21.15.114:0/3058557709 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2e80b29f0 0x7fc2e80b4f00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec000c00 tx=0x7fc2ec01a040 comp rx=0 tx=0).stop 2024-05-27T07:46:17.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.234+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/3058557709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107de0 msgr2=0x7fc2f819fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:17.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.234+0000 7fc2ff1f6700 1 --2- 172.21.15.114:0/3058557709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107de0 0x7fc2f819fe00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400e970 tx=0x7fc2e400ec80 comp rx=0 tx=0).stop 2024-05-27T07:46:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.234+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/3058557709 shutdown_connections 2024-05-27T07:46:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.234+0000 7fc2ff1f6700 1 --2- 172.21.15.114:0/3058557709 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2e80b29f0 0x7fc2e80b4f00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.234+0000 7fc2ff1f6700 1 --2- 172.21.15.114:0/3058557709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2f8108730 0x7fc2f8199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.234+0000 7fc2ff1f6700 1 --2- 172.21.15.114:0/3058557709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2f81034b0 0x7fc2f819f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.234+0000 7fc2ff1f6700 1 --2- 172.21.15.114:0/3058557709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107de0 0x7fc2f819fe00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.234+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/3058557709 >> 172.21.15.114:0/3058557709 conn(0x7fc2f80fedb0 msgr2=0x7fc2f8101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.234+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/3058557709 shutdown_connections 2024-05-27T07:46:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:17.234+0000 7fc2ff1f6700 1 -- 172.21.15.114:0/3058557709 wait complete. 2024-05-27T07:46:17.236 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 37 2024-05-27T07:46:17.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 651410 12.48 MB/sec execute 149 sec latency 97.471 ms 2024-05-27T07:46:17.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:17 smithi114 ceph-mon[32038]: pgmap v208: 129 pgs: 129 active+clean; 75 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 387 op/s 2024-05-27T07:46:17.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:17 smithi114 ceph-mon[32038]: daemon mds.f restarted 2024-05-27T07:46:17.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:17 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3058557709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:46:17.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:17 smithi163 ceph-mon[36451]: pgmap v208: 129 pgs: 129 active+clean; 75 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 387 op/s 2024-05-27T07:46:17.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:17 smithi163 ceph-mon[36451]: daemon mds.f restarted 2024-05-27T07:46:17.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:17 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3058557709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:46:17.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:17 smithi186 ceph-mon[36647]: pgmap v208: 129 pgs: 129 active+clean; 75 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 387 op/s 2024-05-27T07:46:17.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:17 smithi186 ceph-mon[36647]: daemon mds.f restarted 2024-05-27T07:46:17.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:17 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3058557709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:46:17.728 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd406466700 time 2024-05-27T07:46:17.439816+0000 2024-05-27T07:46:17.728 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:46:17.728 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd418cabba6] 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd418cabdd1] 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563c01d0b23d] 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563c01cf6971] 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 5: (SnapRealm::check_cache() const+0x13e) [0x563c01cf4fa0] 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563c01cf5651] 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563c01c8a347] 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563c0198687e] 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563c0198bf2f] 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563c01996969] 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 11: (MDLog::_replay_thread()+0x1f38) [0x563c01d2f48c] 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563c01d389cb] 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 13: (Thread::entry_wrapper()+0x43) [0x7fd418c7a72d] 2024-05-27T07:46:17.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd418c7a749] 2024-05-27T07:46:17.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd41781e1ca] 2024-05-27T07:46:17.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 16: clone() 2024-05-27T07:46:17.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.441+0000 7fd406466700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd406466700 time 2024-05-27T07:46:17.439816+0000 2024-05-27T07:46:17.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:46:17.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:17.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd418cabba6] 2024-05-27T07:46:17.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd418cabdd1] 2024-05-27T07:46:17.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563c01d0b23d] 2024-05-27T07:46:17.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563c01cf6971] 2024-05-27T07:46:17.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 5: (SnapRealm::check_cache() const+0x13e) [0x563c01cf4fa0] 2024-05-27T07:46:17.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563c01cf5651] 2024-05-27T07:46:17.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563c01c8a347] 2024-05-27T07:46:17.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563c0198687e] 2024-05-27T07:46:17.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563c0198bf2f] 2024-05-27T07:46:17.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563c01996969] 2024-05-27T07:46:17.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 11: (MDLog::_replay_thread()+0x1f38) [0x563c01d2f48c] 2024-05-27T07:46:17.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563c01d389cb] 2024-05-27T07:46:17.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 13: (Thread::entry_wrapper()+0x43) [0x7fd418c7a72d] 2024-05-27T07:46:17.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd418c7a749] 2024-05-27T07:46:17.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd41781e1ca] 2024-05-27T07:46:17.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 16: clone() 2024-05-27T07:46:17.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: *** Caught signal (Aborted) ** 2024-05-27T07:46:17.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: in thread 7fd406466700 thread_name:md_log_replay 2024-05-27T07:46:17.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:17.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd417828d20] 2024-05-27T07:46:17.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2: gsignal() 2024-05-27T07:46:17.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 3: abort() 2024-05-27T07:46:17.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd418cabced] 2024-05-27T07:46:17.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd418cabdd1] 2024-05-27T07:46:17.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563c01d0b23d] 2024-05-27T07:46:17.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563c01cf6971] 2024-05-27T07:46:17.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 8: (SnapRealm::check_cache() const+0x13e) [0x563c01cf4fa0] 2024-05-27T07:46:17.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563c01cf5651] 2024-05-27T07:46:17.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563c01c8a347] 2024-05-27T07:46:17.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563c0198687e] 2024-05-27T07:46:17.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563c0198bf2f] 2024-05-27T07:46:17.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563c01996969] 2024-05-27T07:46:17.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 14: (MDLog::_replay_thread()+0x1f38) [0x563c01d2f48c] 2024-05-27T07:46:17.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563c01d389cb] 2024-05-27T07:46:17.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 16: (Thread::entry_wrapper()+0x43) [0x7fd418c7a72d] 2024-05-27T07:46:17.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd418c7a749] 2024-05-27T07:46:17.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd41781e1ca] 2024-05-27T07:46:17.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 19: clone() 2024-05-27T07:46:17.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.443+0000 7fd406466700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:46:17.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: in thread 7fd406466700 thread_name:md_log_replay 2024-05-27T07:46:17.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:17.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd417828d20] 2024-05-27T07:46:17.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2: gsignal() 2024-05-27T07:46:17.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 3: abort() 2024-05-27T07:46:17.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd418cabced] 2024-05-27T07:46:17.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd418cabdd1] 2024-05-27T07:46:17.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563c01d0b23d] 2024-05-27T07:46:17.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563c01cf6971] 2024-05-27T07:46:17.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 8: (SnapRealm::check_cache() const+0x13e) [0x563c01cf4fa0] 2024-05-27T07:46:17.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563c01cf5651] 2024-05-27T07:46:17.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563c01c8a347] 2024-05-27T07:46:17.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563c0198687e] 2024-05-27T07:46:17.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563c0198bf2f] 2024-05-27T07:46:17.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563c01996969] 2024-05-27T07:46:17.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 14: (MDLog::_replay_thread()+0x1f38) [0x563c01d2f48c] 2024-05-27T07:46:17.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563c01d389cb] 2024-05-27T07:46:17.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 16: (Thread::entry_wrapper()+0x43) [0x7fd418c7a72d] 2024-05-27T07:46:17.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd418c7a749] 2024-05-27T07:46:17.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd41781e1ca] 2024-05-27T07:46:17.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 19: clone() 2024-05-27T07:46:17.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:46:17.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: -1> 2024-05-27T07:46:17.441+0000 7fd406466700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd406466700 time 2024-05-27T07:46:17.439816+0000 2024-05-27T07:46:17.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:46:17.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:17.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd418cabba6] 2024-05-27T07:46:17.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd418cabdd1] 2024-05-27T07:46:17.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563c01d0b23d] 2024-05-27T07:46:17.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563c01cf6971] 2024-05-27T07:46:17.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 5: (SnapRealm::check_cache() const+0x13e) [0x563c01cf4fa0] 2024-05-27T07:46:17.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563c01cf5651] 2024-05-27T07:46:17.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563c01c8a347] 2024-05-27T07:46:17.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563c0198687e] 2024-05-27T07:46:17.739 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563c0198bf2f] 2024-05-27T07:46:17.739 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563c01996969] 2024-05-27T07:46:17.739 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 11: (MDLog::_replay_thread()+0x1f38) [0x563c01d2f48c] 2024-05-27T07:46:17.739 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563c01d389cb] 2024-05-27T07:46:17.739 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 13: (Thread::entry_wrapper()+0x43) [0x7fd418c7a72d] 2024-05-27T07:46:17.739 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd418c7a749] 2024-05-27T07:46:17.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd41781e1ca] 2024-05-27T07:46:17.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 16: clone() 2024-05-27T07:46:17.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 0> 2024-05-27T07:46:17.443+0000 7fd406466700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:46:17.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: in thread 7fd406466700 thread_name:md_log_replay 2024-05-27T07:46:17.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:17.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd417828d20] 2024-05-27T07:46:17.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2: gsignal() 2024-05-27T07:46:17.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 3: abort() 2024-05-27T07:46:17.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd418cabced] 2024-05-27T07:46:17.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd418cabdd1] 2024-05-27T07:46:17.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563c01d0b23d] 2024-05-27T07:46:17.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563c01cf6971] 2024-05-27T07:46:17.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 8: (SnapRealm::check_cache() const+0x13e) [0x563c01cf4fa0] 2024-05-27T07:46:17.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563c01cf5651] 2024-05-27T07:46:17.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563c01c8a347] 2024-05-27T07:46:17.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563c0198687e] 2024-05-27T07:46:17.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563c0198bf2f] 2024-05-27T07:46:17.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563c01996969] 2024-05-27T07:46:17.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 14: (MDLog::_replay_thread()+0x1f38) [0x563c01d2f48c] 2024-05-27T07:46:17.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563c01d389cb] 2024-05-27T07:46:17.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 16: (Thread::entry_wrapper()+0x43) [0x7fd418c7a72d] 2024-05-27T07:46:17.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd418c7a749] 2024-05-27T07:46:17.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd41781e1ca] 2024-05-27T07:46:17.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 19: clone() 2024-05-27T07:46:17.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:46:17.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: -1> 2024-05-27T07:46:17.441+0000 7fd406466700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd406466700 time 2024-05-27T07:46:17.439816+0000 2024-05-27T07:46:17.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:46:17.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:17.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd418cabba6] 2024-05-27T07:46:17.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd418cabdd1] 2024-05-27T07:46:17.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563c01d0b23d] 2024-05-27T07:46:17.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563c01cf6971] 2024-05-27T07:46:17.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 5: (SnapRealm::check_cache() const+0x13e) [0x563c01cf4fa0] 2024-05-27T07:46:17.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563c01cf5651] 2024-05-27T07:46:17.744 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563c01c8a347] 2024-05-27T07:46:17.744 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563c0198687e] 2024-05-27T07:46:17.744 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563c0198bf2f] 2024-05-27T07:46:17.744 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563c01996969] 2024-05-27T07:46:17.744 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 11: (MDLog::_replay_thread()+0x1f38) [0x563c01d2f48c] 2024-05-27T07:46:17.744 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563c01d389cb] 2024-05-27T07:46:17.744 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 13: (Thread::entry_wrapper()+0x43) [0x7fd418c7a72d] 2024-05-27T07:46:17.744 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd418c7a749] 2024-05-27T07:46:17.744 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd41781e1ca] 2024-05-27T07:46:17.744 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 16: clone() 2024-05-27T07:46:17.745 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.745 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 0> 2024-05-27T07:46:17.443+0000 7fd406466700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:46:17.745 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: in thread 7fd406466700 thread_name:md_log_replay 2024-05-27T07:46:17.745 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.745 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:17.745 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd417828d20] 2024-05-27T07:46:17.745 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2: gsignal() 2024-05-27T07:46:17.745 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 3: abort() 2024-05-27T07:46:17.745 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd418cabced] 2024-05-27T07:46:17.745 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd418cabdd1] 2024-05-27T07:46:17.745 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563c01d0b23d] 2024-05-27T07:46:17.745 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563c01cf6971] 2024-05-27T07:46:17.745 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 8: (SnapRealm::check_cache() const+0x13e) [0x563c01cf4fa0] 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563c01cf5651] 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563c01c8a347] 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563c0198687e] 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563c0198bf2f] 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563c01996969] 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 14: (MDLog::_replay_thread()+0x1f38) [0x563c01d2f48c] 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563c01d389cb] 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 16: (Thread::entry_wrapper()+0x43) [0x7fd418c7a72d] 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd418c7a749] 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd41781e1ca] 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 19: clone() 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73024]: 2024-05-27T07:46:17.746 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:17 smithi186 podman[73070]: 2024-05-27 07:46:17.505176309 +0000 UTC m=+0.020165338 container died f66355e80bc2ff85613798be25f68f3013675016b2dbd047f6760c25a3daab9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T07:46:17.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:46:18.194 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:46:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 653998 12.49 MB/sec execute 150 sec latency 30.118 ms 2024-05-27T07:46:18.417 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:18 smithi186 ceph-mon[36647]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T07:46:18.417 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:18 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6836/15711889,v1:172.21.15.186:6837/15711889] up:boot 2024-05-27T07:46:18.417 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:18 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:46:18.418 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:18 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:46:18.418 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:18 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:46:18.418 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:18 smithi186 podman[73070]: 2024-05-27 07:46:18.145118195 +0000 UTC m=+0.660107215 container remove f66355e80bc2ff85613798be25f68f3013675016b2dbd047f6760c25a3daab9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-27T07:46:18.418 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:18 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:46:18.601 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:18 smithi114 ceph-mon[32038]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T07:46:18.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:18 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6836/15711889,v1:172.21.15.186:6837/15711889] up:boot 2024-05-27T07:46:18.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:18 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:46:18.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:18 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:46:18.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:18 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:46:18.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:18 smithi163 ceph-mon[36451]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T07:46:18.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:18 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6836/15711889,v1:172.21.15.186:6837/15711889] up:boot 2024-05-27T07:46:18.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:18 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:46:18.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:18 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:46:18.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:18 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:46:18.727 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:18 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:46:19.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 656764 12.50 MB/sec execute 151 sec latency 32.673 ms 2024-05-27T07:46:19.695 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:19 smithi114 ceph-mon[32038]: pgmap v210: 129 pgs: 129 active+clean; 75 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 3.5 MiB/s wr, 401 op/s 2024-05-27T07:46:19.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:19 smithi163 ceph-mon[36451]: pgmap v210: 129 pgs: 129 active+clean; 75 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 3.5 MiB/s wr, 401 op/s 2024-05-27T07:46:19.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:19 smithi186 ceph-mon[36647]: pgmap v210: 129 pgs: 129 active+clean; 75 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 3.5 MiB/s wr, 401 op/s 2024-05-27T07:46:19.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.962+0000 7fbcd05f0700 1 -- 172.21.15.114:0/1962665912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcc8105560 msgr2=0x7fbcc8105970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:19.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.962+0000 7fbcd05f0700 1 --2- 172.21.15.114:0/1962665912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcc8105560 0x7fbcc8105970 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fbcc40098f0 tx=0x7fbcc4009c00 comp rx=0 tx=0).stop 2024-05-27T07:46:19.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.962+0000 7fbcd05f0700 1 -- 172.21.15.114:0/1962665912 shutdown_connections 2024-05-27T07:46:19.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.962+0000 7fbcd05f0700 1 --2- 172.21.15.114:0/1962665912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcc810e940 0x7fbcc8110e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.962+0000 7fbcd05f0700 1 --2- 172.21.15.114:0/1962665912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcc8105560 0x7fbcc8105970 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.962+0000 7fbcd05f0700 1 --2- 172.21.15.114:0/1962665912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcc8105eb0 0x7fbcc810e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.962+0000 7fbcd05f0700 1 -- 172.21.15.114:0/1962665912 >> 172.21.15.114:0/1962665912 conn(0x7fbcc8078df0 msgr2=0x7fbcc8079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:19.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.962+0000 7fbcd05f0700 1 -- 172.21.15.114:0/1962665912 shutdown_connections 2024-05-27T07:46:19.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.962+0000 7fbcd05f0700 1 -- 172.21.15.114:0/1962665912 wait complete. 2024-05-27T07:46:19.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbcd05f0700 1 Processor -- start 2024-05-27T07:46:19.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbcd05f0700 1 -- start start 2024-05-27T07:46:19.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbcd05f0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcc8105560 0x7fbcc819ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:19.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbcd05f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcc8105eb0 0x7fbcc819b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:19.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbcd05f0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcc810e940 0x7fbcc819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:19.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbcd05f0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc8114460 con 0x7fbcc8105eb0 2024-05-27T07:46:19.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbcd05f0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc81142e0 con 0x7fbcc810e940 2024-05-27T07:46:19.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbcd05f0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc81145e0 con 0x7fbcc8105560 2024-05-27T07:46:19.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbccdb8b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcc8105eb0 0x7fbcc819b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:19.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbcce38c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcc8105560 0x7fbcc819ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:19.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbccdb8b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcc8105eb0 0x7fbcc819b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46748/0 (socket says 172.21.15.114:46748) 2024-05-27T07:46:19.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbcce38c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcc8105560 0x7fbcc819ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36694/0 (socket says 172.21.15.114:36694) 2024-05-27T07:46:19.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbccdb8b700 1 -- 172.21.15.114:0/996193825 learned_addr learned my addr 172.21.15.114:0/996193825 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:19.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.963+0000 7fbcceb8d700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcc810e940 0x7fbcc819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.964+0000 7fbccdb8b700 1 -- 172.21.15.114:0/996193825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcc8105560 msgr2=0x7fbcc819ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.964+0000 7fbccdb8b700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcc8105560 0x7fbcc819ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.964+0000 7fbccdb8b700 1 -- 172.21.15.114:0/996193825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcc810e940 msgr2=0x7fbcc819f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.964+0000 7fbccdb8b700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcc810e940 0x7fbcc819f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.964+0000 7fbccdb8b700 1 -- 172.21.15.114:0/996193825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcc819fd20 con 0x7fbcc8105eb0 2024-05-27T07:46:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.964+0000 7fbcce38c700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcc8105560 0x7fbcc819ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.964+0000 7fbccdb8b700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcc8105eb0 0x7fbcc819b3f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fbcb800ec10 tx=0x7fbcb800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.964+0000 7fbcbf7fe700 1 -- 172.21.15.114:0/996193825 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb800cc90 con 0x7fbcc8105eb0 2024-05-27T07:46:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.964+0000 7fbcd05f0700 1 -- 172.21.15.114:0/996193825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcc81a0010 con 0x7fbcc8105eb0 2024-05-27T07:46:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.964+0000 7fbcbf7fe700 1 -- 172.21.15.114:0/996193825 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcb800ce30 con 0x7fbcc8105eb0 2024-05-27T07:46:19.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.964+0000 7fbcbf7fe700 1 -- 172.21.15.114:0/996193825 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8010430 con 0x7fbcc8105eb0 2024-05-27T07:46:19.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.964+0000 7fbcd05f0700 1 -- 172.21.15.114:0/996193825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcc81a7930 con 0x7fbcc8105eb0 2024-05-27T07:46:19.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.968+0000 7fbcbf7fe700 1 -- 172.21.15.114:0/996193825 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fbcb80105d0 con 0x7fbcc8105eb0 2024-05-27T07:46:19.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.969+0000 7fbcbf7fe700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbcb40b2a70 0x7fbcb40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:19.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.969+0000 7fbcce38c700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbcb40b2a70 0x7fbcb40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:19.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.969+0000 7fbcbf7fe700 1 -- 172.21.15.114:0/996193825 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7fbcb80e2c40 con 0x7fbcc8105eb0 2024-05-27T07:46:19.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.969+0000 7fbcdf005b80 1 Processor -- start 2024-05-27T07:46:19.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.969+0000 7fbcdf005b80 1 -- start start 2024-05-27T07:46:19.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbcce38c700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbcb40b2a70 0x7fbcb40b4f80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009fd0 tx=0x7fbcc4019040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:19.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbcdf005b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbde2be40 0x563cbdfb3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:19.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbcdf005b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbdfade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:19.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbcdf005b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbdfae390 0x563cbdfae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:19.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbcdf005b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563cbded8c90 con 0x563cbdfb3f70 2024-05-27T07:46:19.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbcdf005b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563cbdec8830 con 0x563cbdfae390 2024-05-27T07:46:19.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbcdf005b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563cbdecc870 con 0x563cbde2be40 2024-05-27T07:46:19.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbccdb8b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbdfade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:19.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbccdb8b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbdfade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46762/0 (socket says 172.21.15.114:46762) 2024-05-27T07:46:19.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbccdb8b700 1 -- 172.21.15.114:0/2160196912 learned_addr learned my addr 172.21.15.114:0/2160196912 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:19.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbcce38c700 1 --2- 172.21.15.114:0/2160196912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbde2be40 0x563cbdfb3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:19.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbcceb8d700 1 --2- 172.21.15.114:0/2160196912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbdfae390 0x563cbdfae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:19.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbccdb8b700 1 -- 172.21.15.114:0/2160196912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbde2be40 msgr2=0x563cbdfb3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:19.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbccdb8b700 1 --2- 172.21.15.114:0/2160196912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbde2be40 0x563cbdfb3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbccdb8b700 1 -- 172.21.15.114:0/2160196912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbdfae390 msgr2=0x563cbdfae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:19.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbccdb8b700 1 --2- 172.21.15.114:0/2160196912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbdfae390 0x563cbdfae7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbccdb8b700 1 -- 172.21.15.114:0/2160196912 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cbde188d0 con 0x563cbdfb3f70 2024-05-27T07:46:19.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.970+0000 7fbccdb8b700 1 --2- 172.21.15.114:0/2160196912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbdfade50 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8005d70 tx=0x7fbcb80a94c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e45867033ff282 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:19.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.971+0000 7fbcbcff9700 1 -- 172.21.15.114:0/2160196912 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb80a5300 con 0x563cbdfb3f70 2024-05-27T07:46:19.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.971+0000 7fbcbcff9700 1 -- 172.21.15.114:0/2160196912 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcb80b2320 con 0x563cbdfb3f70 2024-05-27T07:46:19.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.971+0000 7fbcbcff9700 1 -- 172.21.15.114:0/2160196912 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb80a9dd0 con 0x563cbdfb3f70 2024-05-27T07:46:19.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.972+0000 7fbcdf005b80 1 -- 172.21.15.114:0/2160196912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 msgr2=0x563cbdfade50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:19.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.972+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/2160196912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbdfade50 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8005d70 tx=0x7fbcb80a94c0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.972+0000 7fbcdf005b80 1 -- 172.21.15.114:0/2160196912 shutdown_connections 2024-05-27T07:46:19.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.972+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/2160196912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbdfae390 0x563cbdfae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.972+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/2160196912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbde2be40 0x563cbdfb3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.972+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/2160196912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbdfade50 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.972+0000 7fbcdf005b80 1 -- 172.21.15.114:0/2160196912 >> 172.21.15.114:0/2160196912 conn(0x563cbde86af0 msgr2=0x563cbdfa4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:19.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.972+0000 7fbcdf005b80 1 -- 172.21.15.114:0/2160196912 shutdown_connections 2024-05-27T07:46:19.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.972+0000 7fbcdf005b80 1 -- 172.21.15.114:0/2160196912 wait complete. 2024-05-27T07:46:19.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.972+0000 7fbcdf005b80 1 -- start start 2024-05-27T07:46:19.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.972+0000 7fbcdf005b80 1 Processor -- start 2024-05-27T07:46:19.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.973+0000 7fbcdf005b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbde2be40 0x563cbdfba2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:19.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.973+0000 7fbcdf005b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfae390 0x563cbdfb78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:19.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.973+0000 7fbcdf005b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbdfb7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:19.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.973+0000 7fbcdf005b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563cbddcefa0 con 0x563cbdfb3f70 2024-05-27T07:46:19.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.973+0000 7fbcdf005b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563cbde6a7c0 con 0x563cbde2be40 2024-05-27T07:46:19.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.973+0000 7fbcdf005b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563cbdfbab10 con 0x563cbdfae390 2024-05-27T07:46:19.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.973+0000 7fbcdf005b80 10 client.0 _ll_register_callbacks cb 0x563cbde63920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:19.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.973+0000 7fbcce38c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbde2be40 0x563cbdfba2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:19.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.973+0000 7fbcce38c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbde2be40 0x563cbdfba2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:45116/0 (socket says 172.21.15.114:45116) 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.973+0000 7fbcce38c700 1 -- 172.21.15.114:0/434461496 learned_addr learned my addr 172.21.15.114:0/434461496 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.973+0000 7fbcceb8d700 1 --2- 172.21.15.114:0/434461496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbdfb7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.973+0000 7fbccdb8b700 1 --2- 172.21.15.114:0/434461496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfae390 0x563cbdfb78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.974+0000 7fbcceb8d700 1 -- 172.21.15.114:0/434461496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfae390 msgr2=0x563cbdfb78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.974+0000 7fbcceb8d700 1 --2- 172.21.15.114:0/434461496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfae390 0x563cbdfb78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.974+0000 7fbcceb8d700 1 -- 172.21.15.114:0/434461496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbde2be40 msgr2=0x563cbdfba2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.974+0000 7fbcceb8d700 1 --2- 172.21.15.114:0/434461496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbde2be40 0x563cbdfba2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.974+0000 7fbcceb8d700 1 -- 172.21.15.114:0/434461496 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cbde67690 con 0x563cbdfb3f70 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.974+0000 7fbcceb8d700 1 --2- 172.21.15.114:0/434461496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbdfb7de0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0011e90 tx=0x7fbcc0014600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4640e992064a88dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.974+0000 7fbcb1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.974+0000 7fbcb1ffb700 1 -- 172.21.15.114:0/434461496 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0004bb0 con 0x563cbdfb3f70 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.974+0000 7fbcb1ffb700 1 -- 172.21.15.114:0/434461496 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7fbcc0004d50 con 0x563cbdfb3f70 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.974+0000 7fbcdf005b80 1 -- 172.21.15.114:0/434461496 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcb80b2320 con 0x563cbdfb3f70 2024-05-27T07:46:19.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.975+0000 7fbcb1ffb700 1 -- 172.21.15.114:0/434461496 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fbcc0026050 con 0x563cbdfb3f70 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.975+0000 7fbcdf005b80 10 client.16956 fetch_fsmap learned FSMap version 39 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.975+0000 7fbcdf005b80 1 -- 172.21.15.114:0/434461496 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563cbdfc9b10 con 0x563cbdfb3f70 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.975+0000 7fbcb1ffb700 1 -- 172.21.15.114:0/434461496 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcb80b2320 con 0x563cbdfb3f70 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.975+0000 7fbcb1ffb700 1 -- 172.21.15.114:0/434461496 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 39) ==== 7725+0+0 (secure 0 0 0) 0x7fbcc0004330 con 0x563cbdfb3f70 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.975+0000 7fbcdf005b80 10 client.16956 fetch_fsmap finished waiting for FSMap version 39 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.975+0000 7fbcdf005b80 10 client.16956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.975+0000 7fbcdf005b80 20 client.16956 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.975+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/434461496 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563cbdfcc1a0 0x563cbdfce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.975+0000 7fbcdf005b80 4 client.16956 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.975+0000 7fbcdf005b80 1 -- 172.21.15.114:0/434461496 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cbdf759f0 con 0x563cbdfcc1a0 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.976+0000 7fbcce38c700 1 --2- 172.21.15.114:0/434461496 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563cbdfcc1a0 0x563cbdfce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.976+0000 7fbcb1ffb700 10 client.16956 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.976+0000 7fbcce38c700 1 --2- 172.21.15.114:0/434461496 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563cbdfcc1a0 0x563cbdfce5f0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a80c8526fbc6bc6 server_cookie=1a3628d84a8ef49f in_seq=0 out_seq=0 2024-05-27T07:46:19.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.977+0000 7fbcb1ffb700 1 -- 172.21.15.114:0/434461496 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563cbdf759f0 con 0x563cbdfcc1a0 2024-05-27T07:46:19.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.977+0000 7fbcb1ffb700 10 client.16956 handle_command_reply: tid=0 2024-05-27T07:46:19.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.977+0000 7fbcdf005b80 1 client.16956 shutdown 2024-05-27T07:46:19.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.978+0000 7fbcdf005b80 1 -- 172.21.15.114:0/434461496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 msgr2=0x563cbdfb7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:19.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.978+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/434461496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbdfb7de0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0011e90 tx=0x7fbcc0014600 comp rx=0 tx=0).stop 2024-05-27T07:46:19.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.978+0000 7fbcdf005b80 1 -- 172.21.15.114:0/434461496 shutdown_connections 2024-05-27T07:46:19.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.978+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/434461496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbdfb7de0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.978+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/434461496 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563cbdfcc1a0 0x563cbdfce5f0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.978+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/434461496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbde2be40 0x563cbdfba2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.978+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/434461496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfae390 0x563cbdfb78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:19.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.978+0000 7fbcdf005b80 1 -- 172.21.15.114:0/434461496 >> 172.21.15.114:0/434461496 conn(0x563cbde86af0 msgr2=0x563cbdfa8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:19.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.978+0000 7fbcb1ffb700 0 client.16956 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:19.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.978+0000 7fbcdf005b80 1 -- 172.21.15.114:0/434461496 shutdown_connections 2024-05-27T07:46:19.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.978+0000 7fbcdf005b80 1 -- 172.21.15.114:0/434461496 wait complete. 2024-05-27T07:46:19.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:19.978+0000 7fbcdf005b80 20 client.16956 trim_cache size 0 max 16384 2024-05-27T07:46:20.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.016+0000 7fbcdf005b80 1 Processor -- start 2024-05-27T07:46:20.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.016+0000 7fbcdf005b80 1 -- start start 2024-05-27T07:46:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.016+0000 7fbcdf005b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbde2be40 0x563cbdfa4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.017+0000 7fbcdf005b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbdfae390 0x563cbdfb7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.017+0000 7fbcdf005b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfb3f70 0x563cbdfb9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.017+0000 7fbcdf005b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc81af3b0 con 0x563cbde2be40 2024-05-27T07:46:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.017+0000 7fbcdf005b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc81ac460 con 0x563cbdfae390 2024-05-27T07:46:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.017+0000 7fbcdf005b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc81ae610 con 0x563cbdfb3f70 2024-05-27T07:46:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.017+0000 7fbccdb8b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbdfae390 0x563cbdfb7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.017+0000 7fbccdb8b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbdfae390 0x563cbdfb7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:45124/0 (socket says 172.21.15.114:45124) 2024-05-27T07:46:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.017+0000 7fbccdb8b700 1 -- 172.21.15.114:0/285151926 learned_addr learned my addr 172.21.15.114:0/285151926 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.018+0000 7fbcceb8d700 1 --2- 172.21.15.114:0/285151926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfb3f70 0x563cbdfb9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.018+0000 7fbccdb8b700 1 -- 172.21.15.114:0/285151926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfb3f70 msgr2=0x563cbdfb9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.018+0000 7fbccdb8b700 1 --2- 172.21.15.114:0/285151926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfb3f70 0x563cbdfb9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.018+0000 7fbccdb8b700 1 -- 172.21.15.114:0/285151926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbde2be40 msgr2=0x563cbdfa4800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:46:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.018+0000 7fbccdb8b700 1 --2- 172.21.15.114:0/285151926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbde2be40 0x563cbdfa4800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.018+0000 7fbccdb8b700 1 -- 172.21.15.114:0/285151926 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cbdf9c970 con 0x563cbdfae390 2024-05-27T07:46:20.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.022+0000 7fbccdb8b700 1 --2- 172.21.15.114:0/285151926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbdfae390 0x563cbdfb7e10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80039d0 tx=0x7fbcb800ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5ac1c3d16372ec4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:20.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.023+0000 7fbcb3fff700 1 -- 172.21.15.114:0/285151926 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8086070 con 0x563cbdfae390 2024-05-27T07:46:20.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.023+0000 7fbcb3fff700 1 -- 172.21.15.114:0/285151926 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcb80a9680 con 0x563cbdfae390 2024-05-27T07:46:20.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.023+0000 7fbcb3fff700 1 -- 172.21.15.114:0/285151926 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb80a9b70 con 0x563cbdfae390 2024-05-27T07:46:20.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.024+0000 7fbcdf005b80 1 -- 172.21.15.114:0/285151926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbdfae390 msgr2=0x563cbdfb7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:20.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.024+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/285151926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbdfae390 0x563cbdfb7e10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80039d0 tx=0x7fbcb800ef90 comp rx=0 tx=0).stop 2024-05-27T07:46:20.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.025+0000 7fbcdf005b80 1 -- 172.21.15.114:0/285151926 shutdown_connections 2024-05-27T07:46:20.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.025+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/285151926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfb3f70 0x563cbdfb9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.025+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/285151926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbde2be40 0x563cbdfa4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.025+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/285151926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbdfae390 0x563cbdfb7e10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.025+0000 7fbcdf005b80 1 -- 172.21.15.114:0/285151926 >> 172.21.15.114:0/285151926 conn(0x563cbde86af0 msgr2=0x563cbdfb29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:20.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.028+0000 7fbcdf005b80 1 -- 172.21.15.114:0/285151926 shutdown_connections 2024-05-27T07:46:20.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.029+0000 7fbcdf005b80 1 -- 172.21.15.114:0/285151926 wait complete. 2024-05-27T07:46:20.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.029+0000 7fbcdf005b80 1 -- start start 2024-05-27T07:46:20.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.029+0000 7fbcdf005b80 1 Processor -- start 2024-05-27T07:46:20.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.030+0000 7fbcdf005b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbde2be40 0x563cbdfc5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:20.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.030+0000 7fbcdf005b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfae390 0x563cbdfc6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:20.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.030+0000 7fbcdf005b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbe034c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:20.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.030+0000 7fbcdf005b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc81adf40 con 0x563cbdfb3f70 2024-05-27T07:46:20.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.030+0000 7fbcdf005b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc81acad0 con 0x563cbde2be40 2024-05-27T07:46:20.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.030+0000 7fbcdf005b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc81ad1a0 con 0x563cbdfae390 2024-05-27T07:46:20.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.030+0000 7fbcdf005b80 10 client.0 _ll_register_callbacks cb 0x563cbde63920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:20.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.031+0000 7fbcceb8d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbe034c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:20.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.031+0000 7fbcceb8d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbe034c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46780/0 (socket says 172.21.15.114:46780) 2024-05-27T07:46:20.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.031+0000 7fbcceb8d700 1 -- 172.21.15.114:0/2880397835 learned_addr learned my addr 172.21.15.114:0/2880397835 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:20.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.031+0000 7fbcce38c700 1 --2- 172.21.15.114:0/2880397835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbde2be40 0x563cbdfc5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:20.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.031+0000 7fbcceb8d700 1 -- 172.21.15.114:0/2880397835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfae390 msgr2=0x563cbdfc6360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:46:20.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.031+0000 7fbcceb8d700 1 --2- 172.21.15.114:0/2880397835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfae390 0x563cbdfc6360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.031+0000 7fbcceb8d700 1 -- 172.21.15.114:0/2880397835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbde2be40 msgr2=0x563cbdfc5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:20.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.031+0000 7fbcceb8d700 1 --2- 172.21.15.114:0/2880397835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbde2be40 0x563cbdfc5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.031+0000 7fbcceb8d700 1 -- 172.21.15.114:0/2880397835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cbdfbade0 con 0x563cbdfb3f70 2024-05-27T07:46:20.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.032+0000 7fbcceb8d700 1 --2- 172.21.15.114:0/2880397835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbe034c80 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000b010 tx=0x7fbcc0009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b34763035baa1f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:20.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.033+0000 7fbcb1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:20.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.035+0000 7fbcb1ffb700 1 -- 172.21.15.114:0/2880397835 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0014a00 con 0x563cbdfb3f70 2024-05-27T07:46:20.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.035+0000 7fbcb1ffb700 1 -- 172.21.15.114:0/2880397835 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7fbcc0014ba0 con 0x563cbdfb3f70 2024-05-27T07:46:20.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.035+0000 7fbcdf005b80 1 -- 172.21.15.114:0/2880397835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcb80a9680 con 0x563cbdfb3f70 2024-05-27T07:46:20.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.036+0000 7fbcb1ffb700 1 -- 172.21.15.114:0/2880397835 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fbcc0014dc0 con 0x563cbdfb3f70 2024-05-27T07:46:20.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.038+0000 7fbcdf005b80 10 client.16959 fetch_fsmap learned FSMap version 39 2024-05-27T07:46:20.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.038+0000 7fbcdf005b80 1 -- 172.21.15.114:0/2880397835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563cbdfbc060 con 0x563cbdfb3f70 2024-05-27T07:46:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.040+0000 7fbcb1ffb700 1 -- 172.21.15.114:0/2880397835 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcb80a9680 con 0x563cbdfb3f70 2024-05-27T07:46:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.040+0000 7fbcb1ffb700 1 -- 172.21.15.114:0/2880397835 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 39) ==== 7725+0+0 (secure 0 0 0) 0x7fbcc002a7b0 con 0x563cbdfb3f70 2024-05-27T07:46:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.040+0000 7fbcdf005b80 10 client.16959 fetch_fsmap finished waiting for FSMap version 39 2024-05-27T07:46:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.040+0000 7fbcdf005b80 10 client.16959 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.041+0000 7fbcdf005b80 20 client.16959 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.041+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/2880397835 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563cbdfbc7f0 0x563cbdfbec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.041+0000 7fbcdf005b80 4 client.16959 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:46:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.041+0000 7fbcdf005b80 1 -- 172.21.15.114:0/2880397835 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563cbdedabe0 con 0x563cbdfbc7f0 2024-05-27T07:46:20.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.049+0000 7fbcce38c700 1 --2- 172.21.15.114:0/2880397835 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563cbdfbc7f0 0x563cbdfbec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:20.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.058+0000 7fbcce38c700 1 --2- 172.21.15.114:0/2880397835 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563cbdfbc7f0 0x563cbdfbec10 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f5e908c754f1689 server_cookie=c247a76153605ccf in_seq=0 out_seq=0 2024-05-27T07:46:20.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.058+0000 7fbcb1ffb700 10 client.16959 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:20.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.062+0000 7fbcb1ffb700 1 -- 172.21.15.114:0/2880397835 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563cbdedabe0 con 0x563cbdfbc7f0 2024-05-27T07:46:20.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.062+0000 7fbcb1ffb700 10 client.16959 handle_command_reply: tid=0 2024-05-27T07:46:20.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.062+0000 7fbcdf005b80 1 client.16959 shutdown 2024-05-27T07:46:20.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.062+0000 7fbcdf005b80 1 -- 172.21.15.114:0/2880397835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 msgr2=0x563cbe034c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:20.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.062+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/2880397835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbe034c80 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000b010 tx=0x7fbcc0009510 comp rx=0 tx=0).stop 2024-05-27T07:46:20.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.063+0000 7fbcdf005b80 1 -- 172.21.15.114:0/2880397835 shutdown_connections 2024-05-27T07:46:20.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.063+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/2880397835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563cbdfb3f70 0x563cbe034c80 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.063+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/2880397835 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563cbdfbc7f0 0x563cbdfbec10 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.063+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/2880397835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563cbde2be40 0x563cbdfc5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.063+0000 7fbcdf005b80 1 --2- 172.21.15.114:0/2880397835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563cbdfae390 0x563cbdfc6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.063+0000 7fbcdf005b80 1 -- 172.21.15.114:0/2880397835 >> 172.21.15.114:0/2880397835 conn(0x563cbde86af0 msgr2=0x563cbdf9e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:20.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.063+0000 7fbcb1ffb700 0 client.16959 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:20.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.064+0000 7fbcdf005b80 1 -- 172.21.15.114:0/2880397835 shutdown_connections 2024-05-27T07:46:20.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.064+0000 7fbcdf005b80 1 -- 172.21.15.114:0/2880397835 wait complete. 2024-05-27T07:46:20.065 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:46:20.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.064+0000 7fbcdf005b80 20 client.16959 trim_cache size 0 max 16384 2024-05-27T07:46:20.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.065+0000 7fbcd05f0700 1 -- 172.21.15.114:0/996193825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbcb40b2a70 msgr2=0x7fbcb40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:20.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.065+0000 7fbcd05f0700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbcb40b2a70 0x7fbcb40b4f80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009fd0 tx=0x7fbcc4019040 comp rx=0 tx=0).stop 2024-05-27T07:46:20.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.066+0000 7fbcd05f0700 1 -- 172.21.15.114:0/996193825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcc8105eb0 msgr2=0x7fbcc819b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:20.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.066+0000 7fbcd05f0700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcc8105eb0 0x7fbcc819b3f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fbcb800ec10 tx=0x7fbcb800ef20 comp rx=0 tx=0).stop 2024-05-27T07:46:20.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.066+0000 7fbcd05f0700 1 -- 172.21.15.114:0/996193825 shutdown_connections 2024-05-27T07:46:20.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.066+0000 7fbcd05f0700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbcb40b2a70 0x7fbcb40b4f80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.066+0000 7fbcd05f0700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcc810e940 0x7fbcc819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.066+0000 7fbcd05f0700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcc8105560 0x7fbcc819ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.066+0000 7fbcd05f0700 1 --2- 172.21.15.114:0/996193825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcc8105eb0 0x7fbcc819b3f0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:20.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.066+0000 7fbcd05f0700 1 -- 172.21.15.114:0/996193825 >> 172.21.15.114:0/996193825 conn(0x7fbcc8078df0 msgr2=0x7fbcc8100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:20.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.066+0000 7fbcd05f0700 1 -- 172.21.15.114:0/996193825 shutdown_connections 2024-05-27T07:46:20.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:20.066+0000 7fbcd05f0700 1 -- 172.21.15.114:0/996193825 wait complete. 2024-05-27T07:46:20.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 658138 12.47 MB/sec execute 152 sec latency 55.075 ms 2024-05-27T07:46:21.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 660849 12.49 MB/sec execute 153 sec latency 38.828 ms 2024-05-27T07:46:21.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:21 smithi163 ceph-mon[36451]: pgmap v211: 129 pgs: 129 active+clean; 79 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 355 op/s 2024-05-27T07:46:21.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:21 smithi186 ceph-mon[36647]: pgmap v211: 129 pgs: 129 active+clean; 79 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 355 op/s 2024-05-27T07:46:21.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:21 smithi114 ceph-mon[32038]: pgmap v211: 129 pgs: 129 active+clean; 79 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 355 op/s 2024-05-27T07:46:22.073 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:46:22.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:46:22.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 663513 12.50 MB/sec execute 154 sec latency 18.183 ms 2024-05-27T07:46:22.344 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:46:23.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 665631 12.49 MB/sec execute 155 sec latency 23.907 ms 2024-05-27T07:46:23.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.565+0000 7f5db90c6640 1 -- 172.21.15.114:0/2001770902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5db4109040 msgr2=0x7f5db4102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.565+0000 7f5db90c6640 1 --2- 172.21.15.114:0/2001770902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5db4109040 0x7f5db4102c80 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009900 tx=0x7f5da802a410 comp rx=0 tx=0).stop 2024-05-27T07:46:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.565+0000 7f5db90c6640 1 -- 172.21.15.114:0/2001770902 shutdown_connections 2024-05-27T07:46:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.565+0000 7f5db90c6640 1 --2- 172.21.15.114:0/2001770902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5db41032f0 0x7f5db410f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.565+0000 7f5db90c6640 1 --2- 172.21.15.114:0/2001770902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5db4109040 0x7f5db4102c80 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.565+0000 7f5db90c6640 1 --2- 172.21.15.114:0/2001770902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db4108ba0 0x7f5db4102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.565+0000 7f5db90c6640 1 -- 172.21.15.114:0/2001770902 >> 172.21.15.114:0/2001770902 conn(0x7f5db4078270 msgr2=0x7f5db40786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.565+0000 7f5db90c6640 1 -- 172.21.15.114:0/2001770902 shutdown_connections 2024-05-27T07:46:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.565+0000 7f5db90c6640 1 -- 172.21.15.114:0/2001770902 wait complete. 2024-05-27T07:46:23.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.566+0000 7f5db90c6640 1 Processor -- start 2024-05-27T07:46:23.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.567+0000 7f5db90c6640 1 -- start start 2024-05-27T07:46:23.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.567+0000 7f5db90c6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5db41032f0 0x7f5db41997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.567+0000 7f5db90c6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db4108ba0 0x7f5db4199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.567+0000 7f5db90c6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5db4109040 0x7f5db419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.567+0000 7f5db90c6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5db4112de0 con 0x7f5db4109040 2024-05-27T07:46:23.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.567+0000 7f5db90c6640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5db4112c60 con 0x7f5db41032f0 2024-05-27T07:46:23.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.567+0000 7f5db90c6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5db4112f60 con 0x7f5db4108ba0 2024-05-27T07:46:23.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.567+0000 7f5db359e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5db4109040 0x7f5db419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.567+0000 7f5db359e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5db4109040 0x7f5db419dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46792/0 (socket says 172.21.15.114:46792) 2024-05-27T07:46:23.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.567+0000 7f5db359e640 1 -- 172.21.15.114:0/117453915 learned_addr learned my addr 172.21.15.114:0/117453915 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.567+0000 7f5db259c640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db4108ba0 0x7f5db4199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.567+0000 7f5db2d9d640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5db41032f0 0x7f5db41997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.568+0000 7f5db259c640 1 -- 172.21.15.114:0/117453915 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5db41032f0 msgr2=0x7f5db41997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.568+0000 7f5db259c640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5db41032f0 0x7f5db41997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.568+0000 7f5db259c640 1 -- 172.21.15.114:0/117453915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5db4109040 msgr2=0x7f5db419dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.568+0000 7f5db259c640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5db4109040 0x7f5db419dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.568+0000 7f5db259c640 1 -- 172.21.15.114:0/117453915 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db419e620 con 0x7f5db4108ba0 2024-05-27T07:46:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.568+0000 7f5db359e640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5db4109040 0x7f5db419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.568+0000 7f5db259c640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db4108ba0 0x7f5db4199d20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009230 tx=0x7f5da8003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.569+0000 7f5d9bfff640 1 -- 172.21.15.114:0/117453915 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da803f070 con 0x7f5db4108ba0 2024-05-27T07:46:23.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.569+0000 7f5d9bfff640 1 -- 172.21.15.114:0/117453915 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da8031b60 con 0x7f5db4108ba0 2024-05-27T07:46:23.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.569+0000 7f5d9bfff640 1 -- 172.21.15.114:0/117453915 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8035030 con 0x7f5db4108ba0 2024-05-27T07:46:23.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.569+0000 7f5db90c6640 1 -- 172.21.15.114:0/117453915 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db419e8b0 con 0x7f5db4108ba0 2024-05-27T07:46:23.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.569+0000 7f5db90c6640 1 -- 172.21.15.114:0/117453915 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db419ed20 con 0x7f5db4108ba0 2024-05-27T07:46:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.571+0000 7f5d9bfff640 1 -- 172.21.15.114:0/117453915 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f5da8042070 con 0x7f5db4108ba0 2024-05-27T07:46:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5dbff77740 1 Processor -- start 2024-05-27T07:46:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5dbff77740 1 -- start start 2024-05-27T07:46:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5dbff77740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab3051b0 0x55a7ab6a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5dbff77740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5dbff77740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6a6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5dbff77740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a7ab618c30 con 0x55a7ab2dc2f0 2024-05-27T07:46:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5dbff77740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a7ab6087d0 con 0x55a7ab2e44d0 2024-05-27T07:46:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5dbff77740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a7ab60c810 con 0x55a7ab3051b0 2024-05-27T07:46:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5d9bfff640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5d9c0b29d0 0x7f5d9c0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5db2d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5db359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab3051b0 0x55a7ab6a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5db259c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6a6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5db2d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6a67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46806/0 (socket says 172.21.15.114:46806) 2024-05-27T07:46:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5db359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab3051b0 0x55a7ab6a62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36748/0 (socket says 172.21.15.114:36748) 2024-05-27T07:46:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5db2d9d640 1 -- 172.21.15.114:0/3202514798 learned_addr learned my addr 172.21.15.114:0/3202514798 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5d9bfff640 1 -- 172.21.15.114:0/117453915 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7f5da8102ee0 con 0x7f5db4108ba0 2024-05-27T07:46:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.572+0000 7f5db359e640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5d9c0b29d0 0x7f5d9c0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.573+0000 7f5db2d9d640 1 -- 172.21.15.114:0/3202514798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab3051b0 msgr2=0x55a7ab6a62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.573+0000 7f5db2d9d640 1 --2- 172.21.15.114:0/3202514798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab3051b0 0x55a7ab6a62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.573+0000 7f5db2d9d640 1 -- 172.21.15.114:0/3202514798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2e44d0 msgr2=0x55a7ab6a6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.573+0000 7f5db2d9d640 1 --2- 172.21.15.114:0/3202514798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6a6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.573+0000 7f5db2d9d640 1 -- 172.21.15.114:0/3202514798 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7ab31d210 con 0x55a7ab2dc2f0 2024-05-27T07:46:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.573+0000 7f5db259c640 1 --2- 172.21.15.114:0/3202514798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6a6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.573+0000 7f5db2d9d640 1 --2- 172.21.15.114:0/3202514798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6a67e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5da4009c50 tx=0x7f5da4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e85d8087052863 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.573+0000 7f5db359e640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5d9c0b29d0 0x7f5d9c0b4ee0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0094f0 tx=0x7f5dac009680 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.573+0000 7f5d997fa640 1 -- 172.21.15.114:0/3202514798 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4004230 con 0x55a7ab2dc2f0 2024-05-27T07:46:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.573+0000 7f5d997fa640 1 -- 172.21.15.114:0/3202514798 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da40043d0 con 0x55a7ab2dc2f0 2024-05-27T07:46:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.573+0000 7f5d997fa640 1 -- 172.21.15.114:0/3202514798 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4004d20 con 0x55a7ab2dc2f0 2024-05-27T07:46:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.574+0000 7f5dbff77740 1 -- 172.21.15.114:0/3202514798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2dc2f0 msgr2=0x55a7ab6a67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.574+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3202514798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6a67e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5da4009c50 tx=0x7f5da4011810 comp rx=0 tx=0).stop 2024-05-27T07:46:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.574+0000 7f5dbff77740 1 -- 172.21.15.114:0/3202514798 shutdown_connections 2024-05-27T07:46:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.574+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3202514798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6a6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.574+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3202514798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6a67e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.574+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3202514798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab3051b0 0x55a7ab6a62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.574+0000 7f5dbff77740 1 -- 172.21.15.114:0/3202514798 >> 172.21.15.114:0/3202514798 conn(0x55a7ab2f9300 msgr2=0x55a7ab6a2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.574+0000 7f5dbff77740 1 -- 172.21.15.114:0/3202514798 shutdown_connections 2024-05-27T07:46:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.574+0000 7f5dbff77740 1 -- 172.21.15.114:0/3202514798 wait complete. 2024-05-27T07:46:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.574+0000 7f5dbff77740 1 -- start start 2024-05-27T07:46:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5dbff77740 1 Processor -- start 2024-05-27T07:46:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5dbff77740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5dbff77740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6b3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5dbff77740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 0x55a7ab6b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5dbff77740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a7ab2ee790 con 0x55a7ab3051b0 2024-05-27T07:46:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5dbff77740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a7ab5c4890 con 0x55a7ab2dc2f0 2024-05-27T07:46:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5dbff77740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a7ab6b57c0 con 0x55a7ab2e44d0 2024-05-27T07:46:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5dbff77740 10 client.0 _ll_register_callbacks cb 0x55a7ab31ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5db359e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 0x55a7ab6b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5db359e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 0x55a7ab6b8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46816/0 (socket says 172.21.15.114:46816) 2024-05-27T07:46:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5db259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5db359e640 1 -- 172.21.15.114:0/3720105636 learned_addr learned my addr 172.21.15.114:0/3720105636 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5db2d9d640 1 --2- 172.21.15.114:0/3720105636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5db359e640 1 -- 172.21.15.114:0/3720105636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2e44d0 msgr2=0x55a7ab6b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5db359e640 1 --2- 172.21.15.114:0/3720105636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5db359e640 1 -- 172.21.15.114:0/3720105636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2dc2f0 msgr2=0x55a7ab6b2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5db359e640 1 --2- 172.21.15.114:0/3720105636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6b2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5db359e640 1 -- 172.21.15.114:0/3720105636 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7ab3e9620 con 0x55a7ab3051b0 2024-05-27T07:46:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5db359e640 1 --2- 172.21.15.114:0/3720105636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 0x55a7ab6b8af0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f5dac011ba0 tx=0x7f5dac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3365ab5de2155b31 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.575+0000 7f5d96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5d96ff5640 1 -- 172.21.15.114:0/3720105636 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac00ee50 con 0x55a7ab3051b0 2024-05-27T07:46:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5d96ff5640 1 -- 172.21.15.114:0/3720105636 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7f5dac004b20 con 0x55a7ab3051b0 2024-05-27T07:46:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5dbff77740 1 -- 172.21.15.114:0/3720105636 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5da40043d0 con 0x55a7ab3051b0 2024-05-27T07:46:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5d96ff5640 1 -- 172.21.15.114:0/3720105636 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f5dac004d20 con 0x55a7ab3051b0 2024-05-27T07:46:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5dbff77740 10 client.16971 fetch_fsmap learned FSMap version 39 2024-05-27T07:46:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5dbff77740 1 -- 172.21.15.114:0/3720105636 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a7ab6c1a20 con 0x55a7ab3051b0 2024-05-27T07:46:23.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5d96ff5640 1 -- 172.21.15.114:0/3720105636 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da40043d0 con 0x55a7ab3051b0 2024-05-27T07:46:23.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5d96ff5640 1 -- 172.21.15.114:0/3720105636 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 39) ==== 7725+0+0 (secure 0 0 0) 0x7f5dac00f040 con 0x55a7ab3051b0 2024-05-27T07:46:23.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5dbff77740 10 client.16971 fetch_fsmap finished waiting for FSMap version 39 2024-05-27T07:46:23.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5dbff77740 10 client.16971 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:23.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5dbff77740 20 client.16971 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:23.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3720105636 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a7ab2c3d80 0x55a7ab6c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5dbff77740 4 client.16971 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:46:23.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.576+0000 7f5dbff77740 1 -- 172.21.15.114:0/3720105636 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7ab61ab80 con 0x55a7ab2c3d80 2024-05-27T07:46:23.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.577+0000 7f5db2d9d640 1 --2- 172.21.15.114:0/3720105636 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a7ab2c3d80 0x55a7ab6c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.577+0000 7f5d96ff5640 10 client.16971 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:23.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.577+0000 7f5db2d9d640 1 --2- 172.21.15.114:0/3720105636 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a7ab2c3d80 0x55a7ab6c4f60 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d927f4a837e5835 server_cookie=febf3cdd7f73588 in_seq=0 out_seq=0 2024-05-27T07:46:23.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5d96ff5640 1 -- 172.21.15.114:0/3720105636 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a7ab61ab80 con 0x55a7ab2c3d80 2024-05-27T07:46:23.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5d96ff5640 10 client.16971 handle_command_reply: tid=0 2024-05-27T07:46:23.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5dbff77740 1 client.16971 shutdown 2024-05-27T07:46:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5dbff77740 1 -- 172.21.15.114:0/3720105636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 msgr2=0x55a7ab6b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3720105636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 0x55a7ab6b8af0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f5dac011ba0 tx=0x7f5dac004a40 comp rx=0 tx=0).stop 2024-05-27T07:46:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5dbff77740 1 -- 172.21.15.114:0/3720105636 shutdown_connections 2024-05-27T07:46:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3720105636 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a7ab2c3d80 0x55a7ab6c4f60 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3720105636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 0x55a7ab6b8af0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3720105636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3720105636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5dbff77740 1 -- 172.21.15.114:0/3720105636 >> 172.21.15.114:0/3720105636 conn(0x55a7ab2f9300 msgr2=0x55a7ab6a8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5d96ff5640 0 client.16971 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5dbff77740 1 -- 172.21.15.114:0/3720105636 shutdown_connections 2024-05-27T07:46:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5dbff77740 1 -- 172.21.15.114:0/3720105636 wait complete. 2024-05-27T07:46:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.579+0000 7f5dbff77740 20 client.16971 trim_cache size 0 max 16384 2024-05-27T07:46:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.596+0000 7f5dbff77740 1 Processor -- start 2024-05-27T07:46:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.596+0000 7f5dbff77740 1 -- start start 2024-05-27T07:46:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5dbff77740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5dbff77740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5dbff77740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab3051b0 0x55a7ab6ac950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5dbff77740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5db41aff90 con 0x55a7ab2e44d0 2024-05-27T07:46:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5dbff77740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5db41ad040 con 0x55a7ab3051b0 2024-05-27T07:46:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5dbff77740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5db41af1f0 con 0x55a7ab2dc2f0 2024-05-27T07:46:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5db259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5db2d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5db259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6ac410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46830/0 (socket says 172.21.15.114:46830) 2024-05-27T07:46:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5db2d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36762/0 (socket says 172.21.15.114:36762) 2024-05-27T07:46:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5db259c640 1 -- 172.21.15.114:0/3483379839 learned_addr learned my addr 172.21.15.114:0/3483379839 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5db359e640 1 --2- 172.21.15.114:0/3483379839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab3051b0 0x55a7ab6ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5db259c640 1 -- 172.21.15.114:0/3483379839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2dc2f0 msgr2=0x55a7ab6b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5db259c640 1 --2- 172.21.15.114:0/3483379839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5db259c640 1 -- 172.21.15.114:0/3483379839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab3051b0 msgr2=0x55a7ab6ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5db259c640 1 --2- 172.21.15.114:0/3483379839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab3051b0 0x55a7ab6ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5db259c640 1 -- 172.21.15.114:0/3483379839 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7ab2c3fb0 con 0x55a7ab2e44d0 2024-05-27T07:46:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5db2d9d640 1 --2- 172.21.15.114:0/3483379839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5db259c640 1 --2- 172.21.15.114:0/3483379839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6ac410 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5da802a9d0 tx=0x7f5da803b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d99521d23574574 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.597+0000 7f5d98ff9640 1 -- 172.21.15.114:0/3483379839 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8049040 con 0x55a7ab2e44d0 2024-05-27T07:46:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.598+0000 7f5d98ff9640 1 -- 172.21.15.114:0/3483379839 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da80ce1c0 con 0x55a7ab2e44d0 2024-05-27T07:46:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.598+0000 7f5d98ff9640 1 -- 172.21.15.114:0/3483379839 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da80491e0 con 0x55a7ab2e44d0 2024-05-27T07:46:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.598+0000 7f5dbff77740 1 -- 172.21.15.114:0/3483379839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2e44d0 msgr2=0x55a7ab6ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.598+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3483379839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6ac410 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5da802a9d0 tx=0x7f5da803b790 comp rx=0 tx=0).stop 2024-05-27T07:46:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.598+0000 7f5dbff77740 1 -- 172.21.15.114:0/3483379839 shutdown_connections 2024-05-27T07:46:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.598+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3483379839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab3051b0 0x55a7ab6ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.598+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3483379839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6ac410 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.598+0000 7f5dbff77740 1 --2- 172.21.15.114:0/3483379839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.598+0000 7f5dbff77740 1 -- 172.21.15.114:0/3483379839 >> 172.21.15.114:0/3483379839 conn(0x55a7ab2f9300 msgr2=0x55a7ab6b12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.598+0000 7f5dbff77740 1 -- 172.21.15.114:0/3483379839 shutdown_connections 2024-05-27T07:46:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.598+0000 7f5dbff77740 1 -- 172.21.15.114:0/3483379839 wait complete. 2024-05-27T07:46:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5dbff77740 1 -- start start 2024-05-27T07:46:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5dbff77740 1 Processor -- start 2024-05-27T07:46:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5dbff77740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6bf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5dbff77740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5dbff77740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 0x55a7ab6ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5dbff77740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5db41aeb20 con 0x55a7ab3051b0 2024-05-27T07:46:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5dbff77740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5db41ad6b0 con 0x55a7ab2dc2f0 2024-05-27T07:46:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5dbff77740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5db41add80 con 0x55a7ab2e44d0 2024-05-27T07:46:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5dbff77740 10 client.0 _ll_register_callbacks cb 0x7f5dac011820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5db359e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 0x55a7ab6ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5db359e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 0x55a7ab6ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46840/0 (socket says 172.21.15.114:46840) 2024-05-27T07:46:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5db259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5db359e640 1 -- 172.21.15.114:0/1937716612 learned_addr learned my addr 172.21.15.114:0/1937716612 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.599+0000 7f5db2d9d640 1 --2- 172.21.15.114:0/1937716612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6bf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5db259c640 1 -- 172.21.15.114:0/1937716612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2dc2f0 msgr2=0x55a7ab6bf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5db259c640 1 --2- 172.21.15.114:0/1937716612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6bf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5db259c640 1 -- 172.21.15.114:0/1937716612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 msgr2=0x55a7ab6ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5db259c640 1 --2- 172.21.15.114:0/1937716612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 0x55a7ab6ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5db259c640 1 -- 172.21.15.114:0/1937716612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7ab6caf50 con 0x55a7ab2e44d0 2024-05-27T07:46:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5db359e640 1 --2- 172.21.15.114:0/1937716612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 0x55a7ab6ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5db259c640 1 --2- 172.21.15.114:0/1937716612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6ca2d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5da8100330 tx=0x7f5da8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49cd22ca4e1abeb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5d96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:46:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5d96ff5640 1 -- 172.21.15.114:0/1937716612 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8049050 con 0x55a7ab2e44d0 2024-05-27T07:46:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5dbff77740 1 -- 172.21.15.114:0/1937716612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5da80ce1c0 con 0x55a7ab2e44d0 2024-05-27T07:46:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5d96ff5640 1 -- 172.21.15.114:0/1937716612 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7f5da80ca120 con 0x55a7ab2e44d0 2024-05-27T07:46:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5d96ff5640 1 -- 172.21.15.114:0/1937716612 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f5da80d2c70 con 0x55a7ab2e44d0 2024-05-27T07:46:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5dbff77740 10 client.26563 fetch_fsmap learned FSMap version 39 2024-05-27T07:46:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5dbff77740 1 -- 172.21.15.114:0/1937716612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a7ab72d640 con 0x55a7ab2e44d0 2024-05-27T07:46:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5d96ff5640 1 -- 172.21.15.114:0/1937716612 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da80ce1c0 con 0x55a7ab2e44d0 2024-05-27T07:46:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5d96ff5640 1 -- 172.21.15.114:0/1937716612 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 39) ==== 7725+0+0 (secure 0 0 0) 0x7f5da80c9120 con 0x55a7ab2e44d0 2024-05-27T07:46:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5dbff77740 10 client.26563 fetch_fsmap finished waiting for FSMap version 39 2024-05-27T07:46:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5dbff77740 10 client.26563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5dbff77740 20 client.26563 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.600+0000 7f5dbff77740 1 --2- 172.21.15.114:0/1937716612 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a7ab2991c0 0x55a7ab72fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.601+0000 7f5dbff77740 4 client.26563 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:46:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.601+0000 7f5dbff77740 1 -- 172.21.15.114:0/1937716612 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a7ab614360 con 0x55a7ab2991c0 2024-05-27T07:46:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.601+0000 7f5db2d9d640 1 --2- 172.21.15.114:0/1937716612 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a7ab2991c0 0x55a7ab72fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.601+0000 7f5d96ff5640 10 client.26563 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.601+0000 7f5db2d9d640 1 --2- 172.21.15.114:0/1937716612 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a7ab2991c0 0x55a7ab72fc00 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65b3cc106766dd30 server_cookie=cb75b7b58d0c5c27 in_seq=0 out_seq=0 2024-05-27T07:46:23.662 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:23 smithi114 ceph-mon[32038]: pgmap v212: 129 pgs: 129 active+clean; 81 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.6 MiB/s wr, 426 op/s 2024-05-27T07:46:23.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:23 smithi163 ceph-mon[36451]: pgmap v212: 129 pgs: 129 active+clean; 81 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.6 MiB/s wr, 426 op/s 2024-05-27T07:46:23.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:23 smithi186 ceph-mon[36647]: pgmap v212: 129 pgs: 129 active+clean; 81 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.6 MiB/s wr, 426 op/s 2024-05-27T07:46:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.898+0000 7f5d96ff5640 1 -- 172.21.15.114:0/1937716612 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a7ab614360 con 0x55a7ab2991c0 2024-05-27T07:46:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.898+0000 7f5d96ff5640 10 client.26563 handle_command_reply: tid=0 2024-05-27T07:46:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.898+0000 7f5dbff77740 1 client.26563 shutdown 2024-05-27T07:46:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.898+0000 7f5dbff77740 1 -- 172.21.15.114:0/1937716612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2e44d0 msgr2=0x55a7ab6ca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.898+0000 7f5dbff77740 1 --2- 172.21.15.114:0/1937716612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6ca2d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5da8100330 tx=0x7f5da8000bc0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.899+0000 7f5dbff77740 1 -- 172.21.15.114:0/1937716612 shutdown_connections 2024-05-27T07:46:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.899+0000 7f5dbff77740 1 --2- 172.21.15.114:0/1937716612 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a7ab2991c0 0x55a7ab72fc00 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.899+0000 7f5d96ff5640 0 client.26563 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.899+0000 7f5dbff77740 1 --2- 172.21.15.114:0/1937716612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7ab3051b0 0x55a7ab6ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.899+0000 7f5dbff77740 1 --2- 172.21.15.114:0/1937716612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7ab2e44d0 0x55a7ab6ca2d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.899+0000 7f5dbff77740 1 --2- 172.21.15.114:0/1937716612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7ab2dc2f0 0x55a7ab6bf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.899+0000 7f5dbff77740 1 -- 172.21.15.114:0/1937716612 >> 172.21.15.114:0/1937716612 conn(0x55a7ab2f9300 msgr2=0x55a7ab6a6160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.899+0000 7f5dbff77740 1 -- 172.21.15.114:0/1937716612 shutdown_connections 2024-05-27T07:46:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.899+0000 7f5dbff77740 1 -- 172.21.15.114:0/1937716612 wait complete. 2024-05-27T07:46:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.901+0000 7f5dbff77740 20 client.26563 trim_cache size 0 max 16384 2024-05-27T07:46:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:46:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:46:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:46:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.904+0000 7f5db90c6640 1 -- 172.21.15.114:0/117453915 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5d9c0b29d0 msgr2=0x7f5d9c0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.904+0000 7f5db90c6640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5d9c0b29d0 0x7f5d9c0b4ee0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0094f0 tx=0x7f5dac009680 comp rx=0 tx=0).stop 2024-05-27T07:46:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.904+0000 7f5db90c6640 1 -- 172.21.15.114:0/117453915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db4108ba0 msgr2=0x7f5db4199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.904+0000 7f5db90c6640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db4108ba0 0x7f5db4199d20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009230 tx=0x7f5da8003c10 comp rx=0 tx=0).stop 2024-05-27T07:46:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.904+0000 7f5db90c6640 1 -- 172.21.15.114:0/117453915 shutdown_connections 2024-05-27T07:46:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.904+0000 7f5db90c6640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5d9c0b29d0 0x7f5d9c0b4ee0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.904+0000 7f5db90c6640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5db4109040 0x7f5db419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.904+0000 7f5db90c6640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db4108ba0 0x7f5db4199d20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.904+0000 7f5db90c6640 1 --2- 172.21.15.114:0/117453915 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5db41032f0 0x7f5db41997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.904+0000 7f5db90c6640 1 -- 172.21.15.114:0/117453915 >> 172.21.15.114:0/117453915 conn(0x7f5db4078270 msgr2=0x7f5db4101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.904+0000 7f5db90c6640 1 -- 172.21.15.114:0/117453915 shutdown_connections 2024-05-27T07:46:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:23.904+0000 7f5db90c6640 1 -- 172.21.15.114:0/117453915 wait complete. 2024-05-27T07:46:24.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 667883 12.50 MB/sec execute 156 sec latency 186.844 ms 2024-05-27T07:46:24.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.285+0000 7f467533e700 1 -- 172.21.15.114:0/2261521980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4670108730 msgr2=0x7f4670110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:24.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.285+0000 7f467533e700 1 --2- 172.21.15.114:0/2261521980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4670108730 0x7f4670110090 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f466800b0d0 tx=0x7f466800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.285+0000 7f467533e700 1 -- 172.21.15.114:0/2261521980 shutdown_connections 2024-05-27T07:46:24.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.285+0000 7f467533e700 1 --2- 172.21.15.114:0/2261521980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4670108730 0x7f4670110090 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.285+0000 7f467533e700 1 --2- 172.21.15.114:0/2261521980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46701034b0 0x7f46701038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.285+0000 7f467533e700 1 --2- 172.21.15.114:0/2261521980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4670107de0 0x7f46701081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.285+0000 7f467533e700 1 -- 172.21.15.114:0/2261521980 >> 172.21.15.114:0/2261521980 conn(0x7f46700fedb0 msgr2=0x7f46701011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:24.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.285+0000 7f467533e700 1 -- 172.21.15.114:0/2261521980 shutdown_connections 2024-05-27T07:46:24.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.285+0000 7f467533e700 1 -- 172.21.15.114:0/2261521980 wait complete. 2024-05-27T07:46:24.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.286+0000 7f467533e700 1 Processor -- start 2024-05-27T07:46:24.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.286+0000 7f467533e700 1 -- start start 2024-05-27T07:46:24.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.286+0000 7f467533e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46701034b0 0x7f467019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.286+0000 7f467533e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4670107de0 0x7f467019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.286+0000 7f467533e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4670108730 0x7f467019f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.286+0000 7f467533e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4670113360 con 0x7f46701034b0 2024-05-27T07:46:24.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.286+0000 7f467533e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f46701131e0 con 0x7f4670107de0 2024-05-27T07:46:24.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.286+0000 7f467533e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f46701134e0 con 0x7f4670108730 2024-05-27T07:46:24.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.286+0000 7f466effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46701034b0 0x7f467019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.286+0000 7f466effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46701034b0 0x7f467019adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46860/0 (socket says 172.21.15.114:46860) 2024-05-27T07:46:24.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.286+0000 7f466e7fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4670107de0 0x7f467019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.286+0000 7f466effd700 1 -- 172.21.15.114:0/3946728874 learned_addr learned my addr 172.21.15.114:0/3946728874 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:24.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.287+0000 7f466e7fc700 1 -- 172.21.15.114:0/3946728874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4670108730 msgr2=0x7f467019f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:24.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.287+0000 7f466e7fc700 1 --2- 172.21.15.114:0/3946728874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4670108730 0x7f467019f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.287+0000 7f466e7fc700 1 -- 172.21.15.114:0/3946728874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46701034b0 msgr2=0x7f467019adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:24.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.287+0000 7f466e7fc700 1 --2- 172.21.15.114:0/3946728874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46701034b0 0x7f467019adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.287+0000 7f466e7fc700 1 -- 172.21.15.114:0/3946728874 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467019fc90 con 0x7f4670107de0 2024-05-27T07:46:24.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.287+0000 7f466effd700 1 --2- 172.21.15.114:0/3946728874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46701034b0 0x7f467019adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:24.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.287+0000 7f466e7fc700 1 --2- 172.21.15.114:0/3946728874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4670107de0 0x7f467019b360 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4670076aa0 tx=0x7f466400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:24.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.287+0000 7f4657fff700 1 -- 172.21.15.114:0/3946728874 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4664017070 con 0x7f4670107de0 2024-05-27T07:46:24.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.287+0000 7f467533e700 1 -- 172.21.15.114:0/3946728874 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f467019ff80 con 0x7f4670107de0 2024-05-27T07:46:24.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.287+0000 7f467533e700 1 -- 172.21.15.114:0/3946728874 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46700ff1c0 con 0x7f4670107de0 2024-05-27T07:46:24.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.287+0000 7f4657fff700 1 -- 172.21.15.114:0/3946728874 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4664009de0 con 0x7f4670107de0 2024-05-27T07:46:24.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.287+0000 7f4657fff700 1 -- 172.21.15.114:0/3946728874 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4664010d90 con 0x7f4670107de0 2024-05-27T07:46:24.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.291+0000 7f4657fff700 1 -- 172.21.15.114:0/3946728874 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f466401a070 con 0x7f4670107de0 2024-05-27T07:46:24.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.292+0000 7f4657fff700 1 --2- 172.21.15.114:0/3946728874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46580b29f0 0x7f46580b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.292+0000 7f4683d53b80 1 Processor -- start 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.292+0000 7f4683d53b80 1 -- start start 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.292+0000 7f466effd700 1 --2- 172.21.15.114:0/3946728874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46580b29f0 0x7f46580b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.292+0000 7f4657fff700 1 -- 172.21.15.114:0/3946728874 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7f46640e22d0 con 0x7f4670107de0 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.292+0000 7f4683d53b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc83dec0 0x55d5bc9c2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.292+0000 7f4683d53b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9bcc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.292+0000 7f4683d53b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9bd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.292+0000 7f4683d53b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d5bc8eadf0 con 0x55d5bc9bd1a0 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.292+0000 7f4683d53b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d5bc8da990 con 0x55d5bc9c30f0 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.292+0000 7f4683d53b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d5bc8de9d0 con 0x55d5bc83dec0 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.293+0000 7f466e7fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9bcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.293+0000 7f466e7fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9bcc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:45226/0 (socket says 172.21.15.114:45226) 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.293+0000 7f466e7fc700 1 -- 172.21.15.114:0/1875860562 learned_addr learned my addr 172.21.15.114:0/1875860562 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.293+0000 7f466effd700 1 --2- 172.21.15.114:0/3946728874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46580b29f0 0x7f46580b4f00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4660011550 tx=0x7f4660009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:24.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.293+0000 7f466effd700 1 --2- 172.21.15.114:0/1875860562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc83dec0 0x55d5bc9c2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.293+0000 7f466f7fe700 1 --2- 172.21.15.114:0/1875860562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9bd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.293+0000 7f466effd700 1 -- 172.21.15.114:0/1875860562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc9c30f0 msgr2=0x55d5bc9bcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:24.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.293+0000 7f466effd700 1 --2- 172.21.15.114:0/1875860562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9bcc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.293+0000 7f466effd700 1 -- 172.21.15.114:0/1875860562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9bd1a0 msgr2=0x55d5bc9bd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:24.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.293+0000 7f466effd700 1 --2- 172.21.15.114:0/1875860562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9bd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.293+0000 7f466effd700 1 -- 172.21.15.114:0/1875860562 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5bc884030 con 0x55d5bc83dec0 2024-05-27T07:46:24.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.294+0000 7f466effd700 1 --2- 172.21.15.114:0/1875860562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc83dec0 0x55d5bc9c2bb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f466000e5e0 tx=0x7f4660014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fb5c8ab703c88b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:24.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.294+0000 7f46557fa700 1 -- 172.21.15.114:0/1875860562 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4660026630 con 0x55d5bc83dec0 2024-05-27T07:46:24.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.294+0000 7f46557fa700 1 -- 172.21.15.114:0/1875860562 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f466000ee40 con 0x55d5bc83dec0 2024-05-27T07:46:24.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.294+0000 7f46557fa700 1 -- 172.21.15.114:0/1875860562 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4660025700 con 0x55d5bc83dec0 2024-05-27T07:46:24.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.295+0000 7f4683d53b80 1 -- 172.21.15.114:0/1875860562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc83dec0 msgr2=0x55d5bc9c2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:24.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.295+0000 7f4683d53b80 1 --2- 172.21.15.114:0/1875860562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc83dec0 0x55d5bc9c2bb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f466000e5e0 tx=0x7f4660014000 comp rx=0 tx=0).stop 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.295+0000 7f4683d53b80 1 -- 172.21.15.114:0/1875860562 shutdown_connections 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.295+0000 7f4683d53b80 1 --2- 172.21.15.114:0/1875860562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9bd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.295+0000 7f4683d53b80 1 --2- 172.21.15.114:0/1875860562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc83dec0 0x55d5bc9c2bb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.295+0000 7f4683d53b80 1 --2- 172.21.15.114:0/1875860562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9bcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.295+0000 7f4683d53b80 1 -- 172.21.15.114:0/1875860562 >> 172.21.15.114:0/1875860562 conn(0x55d5bc8407f0 msgr2=0x55d5bc9b7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.295+0000 7f4683d53b80 1 -- 172.21.15.114:0/1875860562 shutdown_connections 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.295+0000 7f4683d53b80 1 -- 172.21.15.114:0/1875860562 wait complete. 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f4683d53b80 1 -- start start 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f4683d53b80 1 Processor -- start 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f4683d53b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc83dec0 0x55d5bc9c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f4683d53b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f4683d53b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9c7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f4683d53b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d5bc7e1020 con 0x55d5bc9c30f0 2024-05-27T07:46:24.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f4683d53b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d5bc842e20 con 0x55d5bc83dec0 2024-05-27T07:46:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f4683d53b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d5bc9c9be0 con 0x55d5bc9bd1a0 2024-05-27T07:46:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f4683d53b80 10 client.0 _ll_register_callbacks cb 0x55d5bc871630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f466f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9c7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f466effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc83dec0 0x55d5bc9c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f466e7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f466effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc83dec0 0x55d5bc9c69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:45230/0 (socket says 172.21.15.114:45230) 2024-05-27T07:46:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f466e7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9c6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36826/0 (socket says 172.21.15.114:36826) 2024-05-27T07:46:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f466effd700 1 -- 172.21.15.114:0/2214716129 learned_addr learned my addr 172.21.15.114:0/2214716129 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.296+0000 7f466f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9c7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46880/0 (socket says 172.21.15.114:46880) 2024-05-27T07:46:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f466f7fe700 1 -- 172.21.15.114:0/2214716129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9bd1a0 msgr2=0x55d5bc9c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f466f7fe700 1 --2- 172.21.15.114:0/2214716129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f466f7fe700 1 -- 172.21.15.114:0/2214716129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc83dec0 msgr2=0x55d5bc9c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f466f7fe700 1 --2- 172.21.15.114:0/2214716129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc83dec0 0x55d5bc9c69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f466f7fe700 1 -- 172.21.15.114:0/2214716129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5bc82a950 con 0x55d5bc9c30f0 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f466e7fc700 1 --2- 172.21.15.114:0/2214716129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f466effd700 1 --2- 172.21.15.114:0/2214716129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc83dec0 0x55d5bc9c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f466f7fe700 1 --2- 172.21.15.114:0/2214716129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9c7460 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f466800b0d0 tx=0x7f4668009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d4fbf564802be1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f4652ff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f4652ff5700 1 -- 172.21.15.114:0/2214716129 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668003a00 con 0x55d5bc9c30f0 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f4652ff5700 1 -- 172.21.15.114:0/2214716129 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7f4668003ba0 con 0x55d5bc9c30f0 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f4683d53b80 1 -- 172.21.15.114:0/2214716129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d5bc8ecd40 con 0x55d5bc9c30f0 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f4652ff5700 1 -- 172.21.15.114:0/2214716129 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f466802d050 con 0x55d5bc9c30f0 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f4683d53b80 10 client.16989 fetch_fsmap learned FSMap version 39 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.297+0000 7f4683d53b80 1 -- 172.21.15.114:0/2214716129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d5bc8947e0 con 0x55d5bc9c30f0 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.298+0000 7f4652ff5700 1 -- 172.21.15.114:0/2214716129 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55d5bc8ecd40 con 0x55d5bc9c30f0 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.298+0000 7f4652ff5700 1 -- 172.21.15.114:0/2214716129 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 39) ==== 7725+0+0 (secure 0 0 0) 0x7f4668011070 con 0x55d5bc9c30f0 2024-05-27T07:46:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.298+0000 7f4683d53b80 10 client.16989 fetch_fsmap finished waiting for FSMap version 39 2024-05-27T07:46:24.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.298+0000 7f4683d53b80 10 client.16989 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:24.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.298+0000 7f4683d53b80 20 client.16989 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:24.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.298+0000 7f4683d53b80 1 --2- 172.21.15.114:0/2214716129 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d5bc9d6c30 0x55d5bc9d9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.298+0000 7f4683d53b80 4 client.16989 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:46:24.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.298+0000 7f4683d53b80 1 -- 172.21.15.114:0/2214716129 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5bc8e6520 con 0x55d5bc9d6c30 2024-05-27T07:46:24.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.298+0000 7f466effd700 1 --2- 172.21.15.114:0/2214716129 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d5bc9d6c30 0x55d5bc9d9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.299+0000 7f4652ff5700 10 client.16989 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:24.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.299+0000 7f466effd700 1 --2- 172.21.15.114:0/2214716129 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d5bc9d6c30 0x55d5bc9d9050 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7af9cea9e0f88187 server_cookie=6e47c9ec1d15db95 in_seq=0 out_seq=0 2024-05-27T07:46:24.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4652ff5700 1 -- 172.21.15.114:0/2214716129 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d5bc8e6520 con 0x55d5bc9d6c30 2024-05-27T07:46:24.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4652ff5700 10 client.16989 handle_command_reply: tid=0 2024-05-27T07:46:24.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4683d53b80 1 client.16989 shutdown 2024-05-27T07:46:24.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4683d53b80 1 -- 172.21.15.114:0/2214716129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9c30f0 msgr2=0x55d5bc9c7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:24.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4683d53b80 1 --2- 172.21.15.114:0/2214716129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9c7460 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f466800b0d0 tx=0x7f4668009510 comp rx=0 tx=0).stop 2024-05-27T07:46:24.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4683d53b80 1 -- 172.21.15.114:0/2214716129 shutdown_connections 2024-05-27T07:46:24.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4683d53b80 1 --2- 172.21.15.114:0/2214716129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9c7460 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4683d53b80 1 --2- 172.21.15.114:0/2214716129 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d5bc9d6c30 0x55d5bc9d9050 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4683d53b80 1 --2- 172.21.15.114:0/2214716129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc83dec0 0x55d5bc9c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4683d53b80 1 --2- 172.21.15.114:0/2214716129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4683d53b80 1 -- 172.21.15.114:0/2214716129 >> 172.21.15.114:0/2214716129 conn(0x55d5bc8407f0 msgr2=0x55d5bc9bbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:24.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4652ff5700 0 client.16989 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:24.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4683d53b80 1 -- 172.21.15.114:0/2214716129 shutdown_connections 2024-05-27T07:46:24.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4683d53b80 1 -- 172.21.15.114:0/2214716129 wait complete. 2024-05-27T07:46:24.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.300+0000 7f4683d53b80 20 client.16989 trim_cache size 0 max 16384 2024-05-27T07:46:24.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.325+0000 7f4683d53b80 1 Processor -- start 2024-05-27T07:46:24.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.325+0000 7f4683d53b80 1 -- start start 2024-05-27T07:46:24.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.325+0000 7f4683d53b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc83dec0 0x55d5bc9bcc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.325+0000 7f4683d53b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9c5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.325+0000 7f4683d53b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.325+0000 7f4683d53b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f46701b3720 con 0x55d5bc9c30f0 2024-05-27T07:46:24.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.325+0000 7f4683d53b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f46701b07d0 con 0x55d5bc83dec0 2024-05-27T07:46:24.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.325+0000 7f4683d53b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f46701b2980 con 0x55d5bc9bd1a0 2024-05-27T07:46:24.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.326+0000 7f466f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.326+0000 7f466f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46890/0 (socket says 172.21.15.114:46890) 2024-05-27T07:46:24.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.326+0000 7f466f7fe700 1 -- 172.21.15.114:0/1958340225 learned_addr learned my addr 172.21.15.114:0/1958340225 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:24.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.326+0000 7f466effd700 1 --2- 172.21.15.114:0/1958340225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc83dec0 0x55d5bc9bcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.326+0000 7f466effd700 1 -- 172.21.15.114:0/1958340225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9bd1a0 msgr2=0x55d5bc9c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:24.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.326+0000 7f466effd700 1 --2- 172.21.15.114:0/1958340225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.326+0000 7f466effd700 1 -- 172.21.15.114:0/1958340225 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9c30f0 msgr2=0x55d5bc9c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:24.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.326+0000 7f466effd700 1 --2- 172.21.15.114:0/1958340225 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.326+0000 7f466effd700 1 -- 172.21.15.114:0/1958340225 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5bc9c4f70 con 0x55d5bc83dec0 2024-05-27T07:46:24.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.326+0000 7f466effd700 1 --2- 172.21.15.114:0/1958340225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc83dec0 0x55d5bc9bcc50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x55d5bc9d9750 tx=0x7f466001df90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2e98301358d6e3d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:24.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.326+0000 7f4654ff9700 1 -- 172.21.15.114:0/1958340225 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46600056a0 con 0x55d5bc83dec0 2024-05-27T07:46:24.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.327+0000 7f4654ff9700 1 -- 172.21.15.114:0/1958340225 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4660005160 con 0x55d5bc83dec0 2024-05-27T07:46:24.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.327+0000 7f4654ff9700 1 -- 172.21.15.114:0/1958340225 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466001d840 con 0x55d5bc83dec0 2024-05-27T07:46:24.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.327+0000 7f4683d53b80 1 -- 172.21.15.114:0/1958340225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc83dec0 msgr2=0x55d5bc9bcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:24.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.327+0000 7f4683d53b80 1 --2- 172.21.15.114:0/1958340225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc83dec0 0x55d5bc9bcc50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x55d5bc9d9750 tx=0x7f466001df90 comp rx=0 tx=0).stop 2024-05-27T07:46:24.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.328+0000 7f4683d53b80 1 -- 172.21.15.114:0/1958340225 shutdown_connections 2024-05-27T07:46:24.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.328+0000 7f4683d53b80 1 --2- 172.21.15.114:0/1958340225 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.328+0000 7f4683d53b80 1 --2- 172.21.15.114:0/1958340225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc83dec0 0x55d5bc9bcc50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.328+0000 7f4683d53b80 1 --2- 172.21.15.114:0/1958340225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.328+0000 7f4683d53b80 1 -- 172.21.15.114:0/1958340225 >> 172.21.15.114:0/1958340225 conn(0x55d5bc8407f0 msgr2=0x55d5bc9bc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:24.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.328+0000 7f4683d53b80 1 -- 172.21.15.114:0/1958340225 shutdown_connections 2024-05-27T07:46:24.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.328+0000 7f4683d53b80 1 -- 172.21.15.114:0/1958340225 wait complete. 2024-05-27T07:46:24.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.328+0000 7f4683d53b80 1 -- start start 2024-05-27T07:46:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.328+0000 7f4683d53b80 1 Processor -- start 2024-05-27T07:46:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f4683d53b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc83dec0 0x55d5bc9cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f4683d53b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9de1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f4683d53b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9de860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f4683d53b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f46701b22b0 con 0x55d5bc83dec0 2024-05-27T07:46:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f4683d53b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f46701b0e40 con 0x55d5bc9bd1a0 2024-05-27T07:46:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f4683d53b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f46701b1510 con 0x55d5bc9c30f0 2024-05-27T07:46:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f4683d53b80 10 client.0 _ll_register_callbacks cb 0x55d5bc871630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f466effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc83dec0 0x55d5bc9cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f466effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc83dec0 0x55d5bc9cf6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46898/0 (socket says 172.21.15.114:46898) 2024-05-27T07:46:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f466effd700 1 -- 172.21.15.114:0/311357076 learned_addr learned my addr 172.21.15.114:0/311357076 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f466f7fe700 1 --2- 172.21.15.114:0/311357076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9de860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f466e7fc700 1 --2- 172.21.15.114:0/311357076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9de1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f466effd700 1 -- 172.21.15.114:0/311357076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9c30f0 msgr2=0x55d5bc9de860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f466effd700 1 --2- 172.21.15.114:0/311357076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9de860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f466effd700 1 -- 172.21.15.114:0/311357076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc9bd1a0 msgr2=0x55d5bc9de1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f466effd700 1 --2- 172.21.15.114:0/311357076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9de1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f466effd700 1 -- 172.21.15.114:0/311357076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5bc9defe0 con 0x55d5bc83dec0 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f466effd700 1 --2- 172.21.15.114:0/311357076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc83dec0 0x55d5bc9cf6f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f466001f010 tx=0x7f46600056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d75ccb56bcebee5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.329+0000 7f4652ff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4652ff5700 1 -- 172.21.15.114:0/311357076 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4660004810 con 0x55d5bc83dec0 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4683d53b80 1 -- 172.21.15.114:0/311357076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4660005160 con 0x55d5bc83dec0 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4652ff5700 1 -- 172.21.15.114:0/311357076 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7f46600049b0 con 0x55d5bc83dec0 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4652ff5700 1 -- 172.21.15.114:0/311357076 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f466004e050 con 0x55d5bc83dec0 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4683d53b80 10 client.16995 fetch_fsmap learned FSMap version 39 2024-05-27T07:46:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4683d53b80 1 -- 172.21.15.114:0/311357076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d5bca41780 con 0x55d5bc83dec0 2024-05-27T07:46:24.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4652ff5700 1 -- 172.21.15.114:0/311357076 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4660005160 con 0x55d5bc83dec0 2024-05-27T07:46:24.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4652ff5700 1 -- 172.21.15.114:0/311357076 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 39) ==== 7725+0+0 (secure 0 0 0) 0x7f4660026430 con 0x55d5bc83dec0 2024-05-27T07:46:24.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4683d53b80 10 client.16995 fetch_fsmap finished waiting for FSMap version 39 2024-05-27T07:46:24.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4683d53b80 10 client.16995 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:24.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4683d53b80 20 client.16995 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:24.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4683d53b80 1 --2- 172.21.15.114:0/311357076 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d5bca41c90 0x55d5bca440b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:24.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4683d53b80 4 client.16995 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:46:24.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.330+0000 7f4683d53b80 1 -- 172.21.15.114:0/311357076 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d5bc75d2d0 con 0x55d5bca41c90 2024-05-27T07:46:24.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.336+0000 7f466e7fc700 1 --2- 172.21.15.114:0/311357076 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d5bca41c90 0x55d5bca440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:24.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.342+0000 7f4652ff5700 10 client.16995 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:24.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.342+0000 7f466e7fc700 1 --2- 172.21.15.114:0/311357076 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d5bca41c90 0x55d5bca440b0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb190ea5e7bffb17 server_cookie=535800299f86a495 in_seq=0 out_seq=0 2024-05-27T07:46:24.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.347+0000 7f4652ff5700 1 -- 172.21.15.114:0/311357076 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d5bc75d2d0 con 0x55d5bca41c90 2024-05-27T07:46:24.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.347+0000 7f4652ff5700 10 client.16995 handle_command_reply: tid=0 2024-05-27T07:46:24.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.347+0000 7f4683d53b80 1 client.16995 shutdown 2024-05-27T07:46:24.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.348+0000 7f4683d53b80 1 -- 172.21.15.114:0/311357076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc83dec0 msgr2=0x55d5bc9cf6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:24.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.348+0000 7f4683d53b80 1 --2- 172.21.15.114:0/311357076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc83dec0 0x55d5bc9cf6f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f466001f010 tx=0x7f46600056b0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.349+0000 7f4683d53b80 1 -- 172.21.15.114:0/311357076 shutdown_connections 2024-05-27T07:46:24.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.349+0000 7f4683d53b80 1 --2- 172.21.15.114:0/311357076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bc9c30f0 0x55d5bc9de860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.349+0000 7f4683d53b80 1 --2- 172.21.15.114:0/311357076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5bc83dec0 0x55d5bc9cf6f0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.349+0000 7f4683d53b80 1 --2- 172.21.15.114:0/311357076 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d5bca41c90 0x55d5bca440b0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.349+0000 7f4683d53b80 1 --2- 172.21.15.114:0/311357076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5bc9bd1a0 0x55d5bc9de1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.349+0000 7f4683d53b80 1 -- 172.21.15.114:0/311357076 >> 172.21.15.114:0/311357076 conn(0x55d5bc8407f0 msgr2=0x55d5bc9c0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:24.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.350+0000 7f4652ff5700 0 client.16995 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:24.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.350+0000 7f4683d53b80 1 -- 172.21.15.114:0/311357076 shutdown_connections 2024-05-27T07:46:24.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.351+0000 7f4683d53b80 1 -- 172.21.15.114:0/311357076 wait complete. 2024-05-27T07:46:24.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.351+0000 7f4683d53b80 20 client.16995 trim_cache size 0 max 16384 2024-05-27T07:46:24.352 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:46:24.352 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:46:24.352 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "966f41b4-98a4-4e54-a9c6-a8b4c549e385", 2024-05-27T07:46:24.352 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:46:24.352 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:46:24.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.352+0000 7f467533e700 1 -- 172.21.15.114:0/3946728874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46580b29f0 msgr2=0x7f46580b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:24.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.352+0000 7f467533e700 1 --2- 172.21.15.114:0/3946728874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46580b29f0 0x7f46580b4f00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4660011550 tx=0x7f4660009260 comp rx=0 tx=0).stop 2024-05-27T07:46:24.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.353+0000 7f467533e700 1 -- 172.21.15.114:0/3946728874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4670107de0 msgr2=0x7f467019b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:24.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.353+0000 7f467533e700 1 --2- 172.21.15.114:0/3946728874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4670107de0 0x7f467019b360 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4670076aa0 tx=0x7f466400dc30 comp rx=0 tx=0).stop 2024-05-27T07:46:24.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.353+0000 7f467533e700 1 -- 172.21.15.114:0/3946728874 shutdown_connections 2024-05-27T07:46:24.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.353+0000 7f467533e700 1 --2- 172.21.15.114:0/3946728874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4670108730 0x7f467019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.353+0000 7f467533e700 1 --2- 172.21.15.114:0/3946728874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46701034b0 0x7f467019adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.353+0000 7f467533e700 1 --2- 172.21.15.114:0/3946728874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46580b29f0 0x7f46580b4f00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.353+0000 7f467533e700 1 --2- 172.21.15.114:0/3946728874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4670107de0 0x7f467019b360 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:24.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.353+0000 7f467533e700 1 -- 172.21.15.114:0/3946728874 >> 172.21.15.114:0/3946728874 conn(0x7f46700fedb0 msgr2=0x7f467006b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:24.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.353+0000 7f467533e700 1 -- 172.21.15.114:0/3946728874 shutdown_connections 2024-05-27T07:46:24.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:24.353+0000 7f467533e700 1 -- 172.21.15.114:0/3946728874 wait complete. 2024-05-27T07:46:25.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:46:25.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 669971 12.47 MB/sec execute 157 sec latency 157.427 ms 2024-05-27T07:46:25.505 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:46:25.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:25 smithi114 ceph-mon[32038]: pgmap v213: 129 pgs: 129 active+clean; 81 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 341 op/s 2024-05-27T07:46:25.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:25 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:46:25.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:25 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:25.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:25 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:46:25.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:25 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:25.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:25 smithi163 ceph-mon[36451]: pgmap v213: 129 pgs: 129 active+clean; 81 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 341 op/s 2024-05-27T07:46:25.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:25 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:46:25.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:25 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:25.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:25 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:46:25.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:25 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:25.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:25 smithi186 ceph-mon[36647]: pgmap v213: 129 pgs: 129 active+clean; 81 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 341 op/s 2024-05-27T07:46:25.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:25 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:46:25.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:25 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:25.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:25 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:46:25.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:25 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:26.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 671844 12.47 MB/sec execute 158 sec latency 46.090 ms 2024-05-27T07:46:27.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 674922 12.50 MB/sec execute 159 sec latency 21.565 ms 2024-05-27T07:46:27.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.511+0000 7f8187782700 1 -- 172.21.15.114:0/3621409638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8180108ef0 msgr2=0x7f8180109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:27.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.511+0000 7f8187782700 1 --2- 172.21.15.114:0/3621409638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8180108ef0 0x7f8180109300 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f81700099c0 tx=0x7f8170009cd0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.511+0000 7f8187782700 1 -- 172.21.15.114:0/3621409638 shutdown_connections 2024-05-27T07:46:27.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.511+0000 7f8187782700 1 --2- 172.21.15.114:0/3621409638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8180109840 0x7f81801111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.511+0000 7f8187782700 1 --2- 172.21.15.114:0/3621409638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81801045c0 0x7f81801049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.511+0000 7f8187782700 1 --2- 172.21.15.114:0/3621409638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8180108ef0 0x7f8180109300 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.511+0000 7f8187782700 1 -- 172.21.15.114:0/3621409638 >> 172.21.15.114:0/3621409638 conn(0x7f81800ffe70 msgr2=0x7f81801022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:27.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.511+0000 7f8187782700 1 -- 172.21.15.114:0/3621409638 shutdown_connections 2024-05-27T07:46:27.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:27 smithi114 ceph-mon[32038]: pgmap v214: 129 pgs: 129 active+clean; 71 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.7 MiB/s wr, 422 op/s 2024-05-27T07:46:27.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:27 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:46:27.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.511+0000 7f8187782700 1 -- 172.21.15.114:0/3621409638 wait complete. 2024-05-27T07:46:27.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.512+0000 7f8187782700 1 Processor -- start 2024-05-27T07:46:27.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.512+0000 7f8187782700 1 -- start start 2024-05-27T07:46:27.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f8187782700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81801045c0 0x7f818019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f8187782700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8180108ef0 0x7f818019b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f8187782700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8180109840 0x7f818019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f8187782700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8180114450 con 0x7f8180108ef0 2024-05-27T07:46:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f8187782700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f81801142d0 con 0x7f81801045c0 2024-05-27T07:46:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f8187782700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f81801145d0 con 0x7f8180109840 2024-05-27T07:46:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f8184d1d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8180108ef0 0x7f818019b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f8185d1f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8180109840 0x7f818019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f818551e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81801045c0 0x7f818019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f8185d1f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8180109840 0x7f818019f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:46416/0 (socket says 172.21.15.114:46416) 2024-05-27T07:46:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f8185d1f700 1 -- 172.21.15.114:0/4115271169 learned_addr learned my addr 172.21.15.114:0/4115271169 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f818551e700 1 -- 172.21.15.114:0/4115271169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8180109840 msgr2=0x7f818019f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f818551e700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8180109840 0x7f818019f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f818551e700 1 -- 172.21.15.114:0/4115271169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8180108ef0 msgr2=0x7f818019b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f818551e700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8180108ef0 0x7f818019b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f818551e700 1 -- 172.21.15.114:0/4115271169 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818019fcb0 con 0x7f81801045c0 2024-05-27T07:46:27.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f8185d1f700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8180109840 0x7f818019f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:27.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f8184d1d700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8180108ef0 0x7f818019b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:27.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.513+0000 7f818551e700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81801045c0 0x7f818019ade0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f817c007ae0 tx=0x7f817c007df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:27.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.514+0000 7f81767fc700 1 -- 172.21.15.114:0/4115271169 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c004220 con 0x7f81801045c0 2024-05-27T07:46:27.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.514+0000 7f81767fc700 1 -- 172.21.15.114:0/4115271169 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f817c0043c0 con 0x7f81801045c0 2024-05-27T07:46:27.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.514+0000 7f8187782700 1 -- 172.21.15.114:0/4115271169 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f818019ffa0 con 0x7f81801045c0 2024-05-27T07:46:27.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.514+0000 7f81767fc700 1 -- 172.21.15.114:0/4115271169 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c00b750 con 0x7f81801045c0 2024-05-27T07:46:27.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.514+0000 7f8187782700 1 -- 172.21.15.114:0/4115271169 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8180076af0 con 0x7f81801045c0 2024-05-27T07:46:27.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.516+0000 7f81767fc700 1 -- 172.21.15.114:0/4115271169 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f817c00b8f0 con 0x7f81801045c0 2024-05-27T07:46:27.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.517+0000 7f81767fc700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f816c0b29f0 0x7f816c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.517+0000 7f8184d1d700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f816c0b29f0 0x7f816c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.517+0000 7f81767fc700 1 -- 172.21.15.114:0/4115271169 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7f817c0e2ab0 con 0x7f81801045c0 2024-05-27T07:46:27.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f8184d1d700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f816c0b29f0 0x7f816c0b4f00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8170000c00 tx=0x7f8170024040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f8196197b80 1 Processor -- start 2024-05-27T07:46:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f8196197b80 1 -- start start 2024-05-27T07:46:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f8196197b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x558059416710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f8196197b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x558059410a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f8196197b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059411390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f8196197b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55805933fc90 con 0x558059416c50 2024-05-27T07:46:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f8196197b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55805932f830 con 0x558059292e40 2024-05-27T07:46:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f8196197b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558059333870 con 0x558059410f80 2024-05-27T07:46:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f8184d1d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x558059410a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f818551e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x558059416710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f818551e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x558059416710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51736/0 (socket says 172.21.15.114:51736) 2024-05-27T07:46:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f818551e700 1 -- 172.21.15.114:0/1727410066 learned_addr learned my addr 172.21.15.114:0/1727410066 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f8184d1d700 1 --2- 172.21.15.114:0/1727410066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x558059410a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57646/0 (socket says 172.21.15.114:57646) 2024-05-27T07:46:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.518+0000 7f8185d1f700 1 --2- 172.21.15.114:0/1727410066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059411390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.519+0000 7f818551e700 1 -- 172.21.15.114:0/1727410066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 msgr2=0x558059411390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.519+0000 7f818551e700 1 --2- 172.21.15.114:0/1727410066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059411390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.519+0000 7f818551e700 1 -- 172.21.15.114:0/1727410066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 msgr2=0x558059410a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.519+0000 7f818551e700 1 --2- 172.21.15.114:0/1727410066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x558059410a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.519+0000 7f818551e700 1 -- 172.21.15.114:0/1727410066 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55805927f8d0 con 0x558059292e40 2024-05-27T07:46:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.519+0000 7f818551e700 1 --2- 172.21.15.114:0/1727410066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x558059416710 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f817c0a9970 tx=0x7f817c0a5240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b53e7fccdf799f5c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.519+0000 7f816b7fe700 1 -- 172.21.15.114:0/1727410066 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c01d040 con 0x558059292e40 2024-05-27T07:46:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.519+0000 7f816b7fe700 1 -- 172.21.15.114:0/1727410066 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f817c0a99e0 con 0x558059292e40 2024-05-27T07:46:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.519+0000 7f816b7fe700 1 -- 172.21.15.114:0/1727410066 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c0cc0d0 con 0x558059292e40 2024-05-27T07:46:27.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.520+0000 7f8196197b80 1 -- 172.21.15.114:0/1727410066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 msgr2=0x558059416710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:27.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.520+0000 7f8196197b80 1 --2- 172.21.15.114:0/1727410066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x558059416710 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f817c0a9970 tx=0x7f817c0a5240 comp rx=0 tx=0).stop 2024-05-27T07:46:27.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.520+0000 7f8196197b80 1 -- 172.21.15.114:0/1727410066 shutdown_connections 2024-05-27T07:46:27.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.520+0000 7f8196197b80 1 --2- 172.21.15.114:0/1727410066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059411390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.520+0000 7f8196197b80 1 --2- 172.21.15.114:0/1727410066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x558059416710 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.520+0000 7f8196197b80 1 --2- 172.21.15.114:0/1727410066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x558059410a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.520+0000 7f8196197b80 1 -- 172.21.15.114:0/1727410066 >> 172.21.15.114:0/1727410066 conn(0x5580592edaf0 msgr2=0x55805940b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:27.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.520+0000 7f8196197b80 1 -- 172.21.15.114:0/1727410066 shutdown_connections 2024-05-27T07:46:27.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.520+0000 7f8196197b80 1 -- 172.21.15.114:0/1727410066 wait complete. 2024-05-27T07:46:27.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.520+0000 7f8196197b80 1 -- start start 2024-05-27T07:46:27.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.520+0000 7f8196197b80 1 Processor -- start 2024-05-27T07:46:27.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f8196197b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x55805941a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f8196197b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x55805941a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f8196197b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x55805941ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f8196197b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558059235fa0 con 0x558059416c50 2024-05-27T07:46:27.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f8196197b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5580592d17c0 con 0x558059292e40 2024-05-27T07:46:27.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f8196197b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55805941d740 con 0x558059410f80 2024-05-27T07:46:27.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f8196197b80 10 client.0 _ll_register_callbacks cb 0x5580592ca920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:27.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f8185d1f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x55805941ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f8184d1d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x55805941a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f8185d1f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x55805941ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57652/0 (socket says 172.21.15.114:57652) 2024-05-27T07:46:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f8185d1f700 1 -- 172.21.15.114:0/1942182286 learned_addr learned my addr 172.21.15.114:0/1942182286 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f818551e700 1 --2- 172.21.15.114:0/1942182286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x55805941a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f8185d1f700 1 -- 172.21.15.114:0/1942182286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 msgr2=0x55805941a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.521+0000 7f8185d1f700 1 --2- 172.21.15.114:0/1942182286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x55805941a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.522+0000 7f8185d1f700 1 -- 172.21.15.114:0/1942182286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 msgr2=0x55805941a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.522+0000 7f8185d1f700 1 --2- 172.21.15.114:0/1942182286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x55805941a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.522+0000 7f8185d1f700 1 -- 172.21.15.114:0/1942182286 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580592ce690 con 0x558059416c50 2024-05-27T07:46:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.522+0000 7f818551e700 1 --2- 172.21.15.114:0/1942182286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x55805941a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.522+0000 7f8185d1f700 1 --2- 172.21.15.114:0/1942182286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x55805941ae70 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f81780093a0 tx=0x7f817800e7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467e8eaea399d8d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.522+0000 7f8168ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.522+0000 7f8168ff9700 1 -- 172.21.15.114:0/1942182286 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817801a070 con 0x558059416c50 2024-05-27T07:46:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.522+0000 7f8168ff9700 1 -- 172.21.15.114:0/1942182286 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7f8178004db0 con 0x558059416c50 2024-05-27T07:46:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.522+0000 7f8196197b80 1 -- 172.21.15.114:0/1942182286 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f817c0a99e0 con 0x558059416c50 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.522+0000 7f8168ff9700 1 -- 172.21.15.114:0/1942182286 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f817800ee60 con 0x558059416c50 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.522+0000 7f8196197b80 10 client.17007 fetch_fsmap learned FSMap version 39 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.522+0000 7f8196197b80 1 -- 172.21.15.114:0/1942182286 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5580594282a0 con 0x558059416c50 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.522+0000 7f8168ff9700 1 -- 172.21.15.114:0/1942182286 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f817c0a99e0 con 0x558059416c50 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.523+0000 7f8168ff9700 1 -- 172.21.15.114:0/1942182286 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 39) ==== 7725+0+0 (secure 0 0 0) 0x7f817801e4e0 con 0x558059416c50 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.523+0000 7f8196197b80 10 client.17007 fetch_fsmap finished waiting for FSMap version 39 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.523+0000 7f8196197b80 10 client.17007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.523+0000 7f8196197b80 20 client.17007 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.523+0000 7f8196197b80 1 --2- 172.21.15.114:0/1942182286 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55805942a8e0 0x55805942cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.523+0000 7f8196197b80 4 client.17007 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.523+0000 7f8196197b80 1 -- 172.21.15.114:0/1942182286 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580593dc9f0 con 0x55805942a8e0 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.523+0000 7f818551e700 1 --2- 172.21.15.114:0/1942182286 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55805942a8e0 0x55805942cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.523+0000 7f8168ff9700 10 client.17007 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.523+0000 7f818551e700 1 --2- 172.21.15.114:0/1942182286 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55805942a8e0 0x55805942cd20 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=544135f2b9764e44 server_cookie=73c7775909bcc9bb in_seq=0 out_seq=0 2024-05-27T07:46:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.524+0000 7f8168ff9700 1 -- 172.21.15.114:0/1942182286 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5580593dc9f0 con 0x55805942a8e0 2024-05-27T07:46:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.524+0000 7f8168ff9700 10 client.17007 handle_command_reply: tid=0 2024-05-27T07:46:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.524+0000 7f8196197b80 1 client.17007 shutdown 2024-05-27T07:46:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.524+0000 7f8196197b80 1 -- 172.21.15.114:0/1942182286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 msgr2=0x55805941ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.524+0000 7f8196197b80 1 --2- 172.21.15.114:0/1942182286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x55805941ae70 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f81780093a0 tx=0x7f817800e7d0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.524+0000 7f8196197b80 1 -- 172.21.15.114:0/1942182286 shutdown_connections 2024-05-27T07:46:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.524+0000 7f8196197b80 1 --2- 172.21.15.114:0/1942182286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x55805941ae70 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.524+0000 7f8196197b80 1 --2- 172.21.15.114:0/1942182286 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55805942a8e0 0x55805942cd20 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.524+0000 7f8196197b80 1 --2- 172.21.15.114:0/1942182286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x55805941a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.524+0000 7f8196197b80 1 --2- 172.21.15.114:0/1942182286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x55805941a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.524+0000 7f8196197b80 1 -- 172.21.15.114:0/1942182286 >> 172.21.15.114:0/1942182286 conn(0x5580592edaf0 msgr2=0x55805940fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.524+0000 7f8168ff9700 0 client.17007 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:27.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.525+0000 7f8196197b80 1 -- 172.21.15.114:0/1942182286 shutdown_connections 2024-05-27T07:46:27.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.525+0000 7f8196197b80 1 -- 172.21.15.114:0/1942182286 wait complete. 2024-05-27T07:46:27.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.525+0000 7f8196197b80 20 client.17007 trim_cache size 0 max 16384 2024-05-27T07:46:27.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.546+0000 7f8196197b80 1 Processor -- start 2024-05-27T07:46:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.546+0000 7f8196197b80 1 -- start start 2024-05-27T07:46:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.546+0000 7f8196197b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x558059410a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.546+0000 7f8196197b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059419710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.546+0000 7f8196197b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x55805941c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.546+0000 7f8196197b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f81801af330 con 0x558059416c50 2024-05-27T07:46:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.546+0000 7f8196197b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f81801ac3e0 con 0x558059292e40 2024-05-27T07:46:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.546+0000 7f8196197b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f81801ae590 con 0x558059410f80 2024-05-27T07:46:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f8185d1f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x55805941c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f8184d1d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059419710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f818551e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x558059410a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f8184d1d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059419710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:46454/0 (socket says 172.21.15.114:46454) 2024-05-27T07:46:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f8184d1d700 1 -- 172.21.15.114:0/1860726122 learned_addr learned my addr 172.21.15.114:0/1860726122 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f8185d1f700 1 -- 172.21.15.114:0/1860726122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 msgr2=0x558059419710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:27.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f8185d1f700 1 --2- 172.21.15.114:0/1860726122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059419710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f8185d1f700 1 -- 172.21.15.114:0/1860726122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 msgr2=0x558059410a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:27.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f8185d1f700 1 --2- 172.21.15.114:0/1860726122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x558059410a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f8185d1f700 1 -- 172.21.15.114:0/1860726122 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558059406410 con 0x558059416c50 2024-05-27T07:46:27.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f8185d1f700 1 --2- 172.21.15.114:0/1860726122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x55805941c910 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f817801e9d0 tx=0x7f817800ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df43c828392a260 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:27.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f816affd700 1 -- 172.21.15.114:0/1860726122 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817801a070 con 0x558059416c50 2024-05-27T07:46:27.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f816affd700 1 -- 172.21.15.114:0/1860726122 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f817801e610 con 0x558059416c50 2024-05-27T07:46:27.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.547+0000 7f816affd700 1 -- 172.21.15.114:0/1860726122 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8178005950 con 0x558059416c50 2024-05-27T07:46:27.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.548+0000 7f8196197b80 1 -- 172.21.15.114:0/1860726122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 msgr2=0x55805941c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:27.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.548+0000 7f8196197b80 1 --2- 172.21.15.114:0/1860726122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x55805941c910 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f817801e9d0 tx=0x7f817800ebe0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.548+0000 7f8196197b80 1 -- 172.21.15.114:0/1860726122 shutdown_connections 2024-05-27T07:46:27.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.548+0000 7f8196197b80 1 --2- 172.21.15.114:0/1860726122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059416c50 0x55805941c910 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.548+0000 7f8196197b80 1 --2- 172.21.15.114:0/1860726122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059292e40 0x558059410a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.548+0000 7f8196197b80 1 --2- 172.21.15.114:0/1860726122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059419710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.548+0000 7f8196197b80 1 -- 172.21.15.114:0/1860726122 >> 172.21.15.114:0/1860726122 conn(0x5580592edaf0 msgr2=0x558059414010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:27.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.548+0000 7f8196197b80 1 -- 172.21.15.114:0/1860726122 shutdown_connections 2024-05-27T07:46:27.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.548+0000 7f8196197b80 1 -- 172.21.15.114:0/1860726122 wait complete. 2024-05-27T07:46:27.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.549+0000 7f8196197b80 1 -- start start 2024-05-27T07:46:27.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.549+0000 7f8196197b80 1 Processor -- start 2024-05-27T07:46:27.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.549+0000 7f8196197b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059292e40 0x558059423020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.549+0000 7f8196197b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059431bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.549+0000 7f8196197b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059416c50 0x558059432130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.549+0000 7f8196197b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f81801adec0 con 0x558059292e40 2024-05-27T07:46:27.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.549+0000 7f8196197b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f81801aca50 con 0x558059416c50 2024-05-27T07:46:27.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.549+0000 7f8196197b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f81801ad120 con 0x558059410f80 2024-05-27T07:46:27.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.549+0000 7f8196197b80 10 client.0 _ll_register_callbacks cb 0x5580592ca920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:27.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.549+0000 7f818551e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059292e40 0x558059423020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.549+0000 7f8184d1d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059431bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8184d1d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059431bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:46464/0 (socket says 172.21.15.114:46464) 2024-05-27T07:46:27.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8184d1d700 1 -- 172.21.15.114:0/1329004961 learned_addr learned my addr 172.21.15.114:0/1329004961 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:27.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f818551e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059292e40 0x558059423020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57670/0 (socket says 172.21.15.114:57670) 2024-05-27T07:46:27.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8185d1f700 1 --2- 172.21.15.114:0/1329004961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059416c50 0x558059432130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8184d1d700 1 -- 172.21.15.114:0/1329004961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059416c50 msgr2=0x558059432130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:27.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8184d1d700 1 --2- 172.21.15.114:0/1329004961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059416c50 0x558059432130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8184d1d700 1 -- 172.21.15.114:0/1329004961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059292e40 msgr2=0x558059423020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:27.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8184d1d700 1 --2- 172.21.15.114:0/1329004961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059292e40 0x558059423020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8184d1d700 1 -- 172.21.15.114:0/1329004961 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580594327d0 con 0x558059410f80 2024-05-27T07:46:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8184d1d700 1 --2- 172.21.15.114:0/1329004961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059431bf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8170009990 tx=0x7f8170009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81afe0b9fcd87456 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8168ff9700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:46:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8168ff9700 1 -- 172.21.15.114:0/1329004961 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817002fe30 con 0x558059410f80 2024-05-27T07:46:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8196197b80 1 -- 172.21.15.114:0/1329004961 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f817801e610 con 0x558059410f80 2024-05-27T07:46:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8168ff9700 1 -- 172.21.15.114:0/1329004961 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7f8170005070 con 0x558059410f80 2024-05-27T07:46:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8168ff9700 1 -- 172.21.15.114:0/1329004961 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f8170005270 con 0x558059410f80 2024-05-27T07:46:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8196197b80 10 client.26593 fetch_fsmap learned FSMap version 39 2024-05-27T07:46:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.550+0000 7f8196197b80 1 -- 172.21.15.114:0/1329004961 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558059495220 con 0x558059410f80 2024-05-27T07:46:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.551+0000 7f8168ff9700 1 -- 172.21.15.114:0/1329004961 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f817801e610 con 0x558059410f80 2024-05-27T07:46:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.551+0000 7f8168ff9700 1 -- 172.21.15.114:0/1329004961 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 39) ==== 7725+0+0 (secure 0 0 0) 0x7f81700394f0 con 0x558059410f80 2024-05-27T07:46:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.551+0000 7f8196197b80 10 client.26593 fetch_fsmap finished waiting for FSMap version 39 2024-05-27T07:46:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.551+0000 7f8196197b80 10 client.26593 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.551+0000 7f8196197b80 20 client.26593 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:27.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.551+0000 7f8196197b80 1 --2- 172.21.15.114:0/1329004961 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558059495960 0x558059497d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:27.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.551+0000 7f8196197b80 4 client.26593 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:46:27.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.551+0000 7f8196197b80 1 -- 172.21.15.114:0/1329004961 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558059341be0 con 0x558059495960 2024-05-27T07:46:27.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.551+0000 7f818551e700 1 --2- 172.21.15.114:0/1329004961 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558059495960 0x558059497d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:27.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.552+0000 7f8168ff9700 10 client.26593 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:27.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.552+0000 7f818551e700 1 --2- 172.21.15.114:0/1329004961 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558059495960 0x558059497d80 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a71f57c1d8fa1b0c server_cookie=c5cf74b256b22cd8 in_seq=0 out_seq=0 2024-05-27T07:46:27.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8168ff9700 1 -- 172.21.15.114:0/1329004961 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558059341be0 con 0x558059495960 2024-05-27T07:46:27.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8168ff9700 10 client.26593 handle_command_reply: tid=0 2024-05-27T07:46:27.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8196197b80 1 client.26593 shutdown 2024-05-27T07:46:27.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8196197b80 1 -- 172.21.15.114:0/1329004961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 msgr2=0x558059431bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:27.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8196197b80 1 --2- 172.21.15.114:0/1329004961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059431bf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8170009990 tx=0x7f8170009310 comp rx=0 tx=0).stop 2024-05-27T07:46:27.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8196197b80 1 -- 172.21.15.114:0/1329004961 shutdown_connections 2024-05-27T07:46:27.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8196197b80 1 --2- 172.21.15.114:0/1329004961 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558059495960 0x558059497d80 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8196197b80 1 --2- 172.21.15.114:0/1329004961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558059416c50 0x558059432130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8196197b80 1 --2- 172.21.15.114:0/1329004961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558059292e40 0x558059423020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8168ff9700 0 client.26593 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:27.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8196197b80 1 --2- 172.21.15.114:0/1329004961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558059410f80 0x558059431bf0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8196197b80 1 -- 172.21.15.114:0/1329004961 >> 172.21.15.114:0/1329004961 conn(0x5580592edaf0 msgr2=0x558059419470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:27.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8196197b80 1 -- 172.21.15.114:0/1329004961 shutdown_connections 2024-05-27T07:46:27.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8196197b80 1 -- 172.21.15.114:0/1329004961 wait complete. 2024-05-27T07:46:27.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8196197b80 20 client.26593 trim_cache size 0 max 16384 2024-05-27T07:46:27.557 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:46:27.557 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:46:27.557 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:46:27.557 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:46:27.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8187782700 1 -- 172.21.15.114:0/4115271169 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f816c0b29f0 msgr2=0x7f816c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:27.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8187782700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f816c0b29f0 0x7f816c0b4f00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8170000c00 tx=0x7f8170024040 comp rx=0 tx=0).stop 2024-05-27T07:46:27.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8187782700 1 -- 172.21.15.114:0/4115271169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81801045c0 msgr2=0x7f818019ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:27.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.553+0000 7f8187782700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81801045c0 0x7f818019ade0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f817c007ae0 tx=0x7f817c007df0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.554+0000 7f8187782700 1 -- 172.21.15.114:0/4115271169 shutdown_connections 2024-05-27T07:46:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.554+0000 7f8187782700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8180109840 0x7f818019f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.554+0000 7f8187782700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f816c0b29f0 0x7f816c0b4f00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.554+0000 7f8187782700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81801045c0 0x7f818019ade0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.554+0000 7f8187782700 1 --2- 172.21.15.114:0/4115271169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8180108ef0 0x7f818019b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.554+0000 7f8187782700 1 -- 172.21.15.114:0/4115271169 >> 172.21.15.114:0/4115271169 conn(0x7f81800ffe70 msgr2=0x7f818010a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.554+0000 7f8187782700 1 -- 172.21.15.114:0/4115271169 shutdown_connections 2024-05-27T07:46:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:27.554+0000 7f8187782700 1 -- 172.21.15.114:0/4115271169 wait complete. 2024-05-27T07:46:27.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:27 smithi163 ceph-mon[36451]: pgmap v214: 129 pgs: 129 active+clean; 71 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.7 MiB/s wr, 422 op/s 2024-05-27T07:46:27.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:27 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:46:27.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:27 smithi186 ceph-mon[36647]: pgmap v214: 129 pgs: 129 active+clean; 71 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.7 MiB/s wr, 422 op/s 2024-05-27T07:46:27.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:27 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:46:28.262 INFO:tasks.workunit.client.0.smithi114.stdout: 1 677325 12.48 MB/sec execute 160 sec latency 18.056 ms 2024-05-27T07:46:28.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:28 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:46:28.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:28 smithi186 systemd[1]: Stopped Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:46:28.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:28 smithi186 systemd[1]: Starting Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:46:28.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:28 smithi186 podman[73161]: 2024-05-27 07:46:28.621075739 +0000 UTC m=+0.019270720 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:46:29.086 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:46:29.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:46:29.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 679382 12.48 MB/sec execute 161 sec latency 58.428 ms 2024-05-27T07:46:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:46:29.631 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:29 smithi114 ceph-mon[32038]: pgmap v215: 129 pgs: 129 active+clean; 71 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 386 op/s 2024-05-27T07:46:29.631 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:29 smithi186 ceph-mon[36647]: pgmap v215: 129 pgs: 129 active+clean; 71 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 386 op/s 2024-05-27T07:46:29.631 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:29 smithi186 podman[73161]: 2024-05-27 07:46:29.4971195 +0000 UTC m=+0.895314462 container create 2ca6a9d92e05f765e85641c92951e9c442aeea241d16e66adb1974847ed86404 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True) 2024-05-27T07:46:29.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:29 smithi163 ceph-mon[36451]: pgmap v215: 129 pgs: 129 active+clean; 71 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 386 op/s 2024-05-27T07:46:29.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:29 smithi186 podman[73161]: 2024-05-27 07:46:29.725153361 +0000 UTC m=+1.123348325 container init 2ca6a9d92e05f765e85641c92951e9c442aeea241d16e66adb1974847ed86404 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=) 2024-05-27T07:46:29.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:29 smithi186 podman[73161]: 2024-05-27 07:46:29.728396341 +0000 UTC m=+1.126591312 container start 2ca6a9d92e05f765e85641c92951e9c442aeea241d16e66adb1974847ed86404 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-05-27T07:46:29.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:29 smithi186 ceph-mds[73180]: -- 172.21.15.186:0/1622567276 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56236fa4ba40 con 0x56236fa93180 2024-05-27T07:46:29.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:29 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: starting mds.f at 2024-05-27T07:46:29.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:29 smithi186 bash[73161]: 2ca6a9d92e05f765e85641c92951e9c442aeea241d16e66adb1974847ed86404 2024-05-27T07:46:29.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:29 smithi186 systemd[1]: Started Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:46:30.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 682169 12.51 MB/sec execute 162 sec latency 38.068 ms 2024-05-27T07:46:30.514 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:30 smithi114 ceph-mon[32038]: daemon mds.f restarted 2024-05-27T07:46:30.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:30 smithi163 ceph-mon[36451]: daemon mds.f restarted 2024-05-27T07:46:30.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:30 smithi186 ceph-mon[36647]: daemon mds.f restarted 2024-05-27T07:46:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.208+0000 7fa09ab10700 1 -- 172.21.15.114:0/1135050690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa094102770 msgr2=0x7fa094102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.208+0000 7fa09ab10700 1 --2- 172.21.15.114:0/1135050690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa094102770 0x7fa094102b80 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fa080009990 tx=0x7fa080009ca0 comp rx=0 tx=0).stop 2024-05-27T07:46:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.208+0000 7fa09ab10700 1 -- 172.21.15.114:0/1135050690 shutdown_connections 2024-05-27T07:46:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.208+0000 7fa09ab10700 1 --2- 172.21.15.114:0/1135050690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0941030c0 0x7fa094111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.208+0000 7fa09ab10700 1 --2- 172.21.15.114:0/1135050690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0941092e0 0x7fa0941096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.208+0000 7fa09ab10700 1 --2- 172.21.15.114:0/1135050690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa094102770 0x7fa094102b80 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.208+0000 7fa09ab10700 1 -- 172.21.15.114:0/1135050690 >> 172.21.15.114:0/1135050690 conn(0x7fa0940fe520 msgr2=0x7fa094100970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.208+0000 7fa09ab10700 1 -- 172.21.15.114:0/1135050690 shutdown_connections 2024-05-27T07:46:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.208+0000 7fa09ab10700 1 -- 172.21.15.114:0/1135050690 wait complete. 2024-05-27T07:46:31.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa09ab10700 1 Processor -- start 2024-05-27T07:46:31.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa09ab10700 1 -- start start 2024-05-27T07:46:31.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa09ab10700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa094102770 0x7fa09419aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:31.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa09ab10700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0941030c0 0x7fa09419b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa09ab10700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0941092e0 0x7fa09419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa09ab10700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa094114580 con 0x7fa0941030c0 2024-05-27T07:46:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa09ab10700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa094114400 con 0x7fa0941092e0 2024-05-27T07:46:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa09ab10700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa094114700 con 0x7fa094102770 2024-05-27T07:46:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa093fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0941030c0 0x7fa09419b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa093fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0941030c0 0x7fa09419b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57674/0 (socket says 172.21.15.114:57674) 2024-05-27T07:46:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa093fff700 1 -- 172.21.15.114:0/1651611555 learned_addr learned my addr 172.21.15.114:0/1651611555 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa0988ac700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa094102770 0x7fa09419aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa0990ad700 1 --2- 172.21.15.114:0/1651611555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0941092e0 0x7fa09419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.209+0000 7fa093fff700 1 -- 172.21.15.114:0/1651611555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa094102770 msgr2=0x7fa09419aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.210+0000 7fa093fff700 1 --2- 172.21.15.114:0/1651611555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa094102770 0x7fa09419aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.210+0000 7fa093fff700 1 -- 172.21.15.114:0/1651611555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0941092e0 msgr2=0x7fa09419f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.210+0000 7fa093fff700 1 --2- 172.21.15.114:0/1651611555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0941092e0 0x7fa09419f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.210+0000 7fa093fff700 1 -- 172.21.15.114:0/1651611555 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa09419fd80 con 0x7fa0941030c0 2024-05-27T07:46:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.210+0000 7fa093fff700 1 --2- 172.21.15.114:0/1651611555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0941030c0 0x7fa09419b450 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa080000c00 tx=0x7fa080009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.210+0000 7fa091ffb700 1 -- 172.21.15.114:0/1651611555 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa080027070 con 0x7fa0941030c0 2024-05-27T07:46:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.210+0000 7fa091ffb700 1 -- 172.21.15.114:0/1651611555 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa080022470 con 0x7fa0941030c0 2024-05-27T07:46:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.210+0000 7fa091ffb700 1 -- 172.21.15.114:0/1651611555 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa080017650 con 0x7fa0941030c0 2024-05-27T07:46:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.210+0000 7fa09ab10700 1 -- 172.21.15.114:0/1651611555 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0941a0010 con 0x7fa0941030c0 2024-05-27T07:46:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.210+0000 7fa09ab10700 1 -- 172.21.15.114:0/1651611555 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0941a04f0 con 0x7fa0941030c0 2024-05-27T07:46:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.211+0000 7fa09ab10700 1 -- 172.21.15.114:0/1651611555 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0940512b0 con 0x7fa0941030c0 2024-05-27T07:46:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.213+0000 7fa091ffb700 1 -- 172.21.15.114:0/1651611555 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa08002a070 con 0x7fa0941030c0 2024-05-27T07:46:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.213+0000 7fa091ffb700 1 --2- 172.21.15.114:0/1651611555 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0840b29f0 0x7fa0840b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.213+0000 7fa0988ac700 1 --2- 172.21.15.114:0/1651611555 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0840b29f0 0x7fa0840b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.214+0000 7fa091ffb700 1 -- 172.21.15.114:0/1651611555 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fa0800e3ae0 con 0x7fa0941030c0 2024-05-27T07:46:31.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.214+0000 7fa0988ac700 1 --2- 172.21.15.114:0/1651611555 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0840b29f0 0x7fa0840b4f00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa09406cae0 tx=0x7fa088009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:31.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.217+0000 7fa091ffb700 1 -- 172.21.15.114:0/1651611555 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa080013070 con 0x7fa0941030c0 2024-05-27T07:46:31.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 683819 12.47 MB/sec execute 163 sec latency 68.056 ms 2024-05-27T07:46:31.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.551+0000 7fa09ab10700 1 -- 172.21.15.114:0/1651611555 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa09402e5e0 con 0x7fa0941030c0 2024-05-27T07:46:31.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.552+0000 7fa091ffb700 1 -- 172.21.15.114:0/1651611555 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+11003 (secure 0 0 0) 0x7fa0800ab1a0 con 0x7fa0941030c0 2024-05-27T07:46:31.552 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:46:31.553 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":26227,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2936519309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2936519309},{"type":"v1","addr":"172.21.15.186:6835","nonce":2936519309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:46:30.376549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26599":{"gid":26599,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1833707868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1833707868},{"type":"v1","addr":"172.21.15.186:6837","nonce":1833707868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:46:31.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.554+0000 7fa09ab10700 1 -- 172.21.15.114:0/1651611555 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0840b29f0 msgr2=0x7fa0840b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:31.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.554+0000 7fa09ab10700 1 --2- 172.21.15.114:0/1651611555 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0840b29f0 0x7fa0840b4f00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa09406cae0 tx=0x7fa088009260 comp rx=0 tx=0).stop 2024-05-27T07:46:31.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.554+0000 7fa09ab10700 1 -- 172.21.15.114:0/1651611555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0941030c0 msgr2=0x7fa09419b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:31.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.554+0000 7fa09ab10700 1 --2- 172.21.15.114:0/1651611555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0941030c0 0x7fa09419b450 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa080000c00 tx=0x7fa080009580 comp rx=0 tx=0).stop 2024-05-27T07:46:31.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.555+0000 7fa09ab10700 1 -- 172.21.15.114:0/1651611555 shutdown_connections 2024-05-27T07:46:31.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.555+0000 7fa09ab10700 1 --2- 172.21.15.114:0/1651611555 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0840b29f0 0x7fa0840b4f00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:31.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.555+0000 7fa09ab10700 1 --2- 172.21.15.114:0/1651611555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0941092e0 0x7fa09419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:31.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.555+0000 7fa09ab10700 1 --2- 172.21.15.114:0/1651611555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa094102770 0x7fa09419aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:31.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.555+0000 7fa09ab10700 1 --2- 172.21.15.114:0/1651611555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0941030c0 0x7fa09419b450 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:31.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.555+0000 7fa09ab10700 1 -- 172.21.15.114:0/1651611555 >> 172.21.15.114:0/1651611555 conn(0x7fa0940fe520 msgr2=0x7fa0941071d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:31.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.555+0000 7fa09ab10700 1 -- 172.21.15.114:0/1651611555 shutdown_connections 2024-05-27T07:46:31.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:31.555+0000 7fa09ab10700 1 -- 172.21.15.114:0/1651611555 wait complete. 2024-05-27T07:46:31.555 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 41 2024-05-27T07:46:31.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:31 smithi163 ceph-mon[36451]: pgmap v216: 129 pgs: 129 active+clean; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 392 op/s 2024-05-27T07:46:31.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:31 smithi163 ceph-mon[36451]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T07:46:31.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:31 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6836/1833707868,v1:172.21.15.186:6837/1833707868] up:boot 2024-05-27T07:46:31.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:31 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:46:31.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:31 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:46:31.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:31 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:46:31.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:31 smithi186 ceph-mon[36647]: pgmap v216: 129 pgs: 129 active+clean; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 392 op/s 2024-05-27T07:46:31.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:31 smithi186 ceph-mon[36647]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T07:46:31.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:31 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6836/1833707868,v1:172.21.15.186:6837/1833707868] up:boot 2024-05-27T07:46:31.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:31 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:46:31.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:31 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:46:31.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:31 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:46:31.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:31 smithi114 ceph-mon[32038]: pgmap v216: 129 pgs: 129 active+clean; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 392 op/s 2024-05-27T07:46:31.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:31 smithi114 ceph-mon[32038]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T07:46:31.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:31 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6836/1833707868,v1:172.21.15.186:6837/1833707868] up:boot 2024-05-27T07:46:31.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:31 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:46:31.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:31 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:46:31.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:31 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:46:32.267 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685993 12.48 MB/sec execute 164 sec latency 54.439 ms 2024-05-27T07:46:32.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:46:32.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:32 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1651611555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:46:32.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:32 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1651611555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:46:32.797 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:46:32.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:32 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1651611555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:46:33.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685993 12.40 MB/sec execute 165 sec latency 1054.509 ms 2024-05-27T07:46:33.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:33 smithi114 ceph-mon[32038]: pgmap v218: 129 pgs: 129 active+clean; 77 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 433 op/s 2024-05-27T07:46:33.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:33 smithi163 ceph-mon[36451]: pgmap v218: 129 pgs: 129 active+clean; 77 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 433 op/s 2024-05-27T07:46:33.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:33 smithi186 ceph-mon[36647]: pgmap v218: 129 pgs: 129 active+clean; 77 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 433 op/s 2024-05-27T07:46:34.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685993 12.32 MB/sec execute 166 sec latency 2054.557 ms 2024-05-27T07:46:34.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.541+0000 7fbe496f2700 1 -- 172.21.15.114:0/614337595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe44105de0 msgr2=0x7fbe4410e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:34.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.541+0000 7fbe496f2700 1 --2- 172.21.15.114:0/614337595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe44105de0 0x7fbe4410e330 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fbe38009990 tx=0x7fbe38009ca0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.541+0000 7fbe496f2700 1 -- 172.21.15.114:0/614337595 shutdown_connections 2024-05-27T07:46:34.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.541+0000 7fbe496f2700 1 --2- 172.21.15.114:0/614337595 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe4410e900 0x7fbe44110e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.541+0000 7fbe496f2700 1 --2- 172.21.15.114:0/614337595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe44105490 0x7fbe441058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.541+0000 7fbe496f2700 1 --2- 172.21.15.114:0/614337595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe44105de0 0x7fbe4410e330 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.541+0000 7fbe496f2700 1 -- 172.21.15.114:0/614337595 >> 172.21.15.114:0/614337595 conn(0x7fbe44078e40 msgr2=0x7fbe44079280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:34.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.541+0000 7fbe496f2700 1 -- 172.21.15.114:0/614337595 shutdown_connections 2024-05-27T07:46:34.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.541+0000 7fbe496f2700 1 -- 172.21.15.114:0/614337595 wait complete. 2024-05-27T07:46:34.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.542+0000 7fbe496f2700 1 Processor -- start 2024-05-27T07:46:34.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.542+0000 7fbe496f2700 1 -- start start 2024-05-27T07:46:34.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.542+0000 7fbe496f2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe44105490 0x7fbe4419aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.542+0000 7fbe496f2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe44105de0 0x7fbe4419b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.542+0000 7fbe496f2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe4410e900 0x7fbe4419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.542+0000 7fbe496f2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441144b0 con 0x7fbe44105de0 2024-05-27T07:46:34.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.542+0000 7fbe496f2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe44114330 con 0x7fbe4410e900 2024-05-27T07:46:34.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.542+0000 7fbe496f2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbe44114630 con 0x7fbe44105490 2024-05-27T07:46:34.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.542+0000 7fbe427fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe44105de0 0x7fbe4419b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.542+0000 7fbe427fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe44105de0 0x7fbe4419b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57694/0 (socket says 172.21.15.114:57694) 2024-05-27T07:46:34.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.542+0000 7fbe427fc700 1 -- 172.21.15.114:0/1874463114 learned_addr learned my addr 172.21.15.114:0/1874463114 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:34.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.542+0000 7fbe42ffd700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe44105490 0x7fbe4419aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.542+0000 7fbe437fe700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe4410e900 0x7fbe4419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.543+0000 7fbe427fc700 1 -- 172.21.15.114:0/1874463114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe44105490 msgr2=0x7fbe4419aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:34.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.543+0000 7fbe427fc700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe44105490 0x7fbe4419aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.543+0000 7fbe427fc700 1 -- 172.21.15.114:0/1874463114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe4410e900 msgr2=0x7fbe4419f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:34.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.543+0000 7fbe427fc700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe4410e900 0x7fbe4419f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.543+0000 7fbe427fc700 1 -- 172.21.15.114:0/1874463114 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4419fc90 con 0x7fbe44105de0 2024-05-27T07:46:34.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.543+0000 7fbe437fe700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe4410e900 0x7fbe4419f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:46:34.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.543+0000 7fbe427fc700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe44105de0 0x7fbe4419b460 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fbe38000c00 tx=0x7fbe380047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:34.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.543+0000 7fbe2bfff700 1 -- 172.21.15.114:0/1874463114 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe38027070 con 0x7fbe44105de0 2024-05-27T07:46:34.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.543+0000 7fbe496f2700 1 -- 172.21.15.114:0/1874463114 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe4419ff20 con 0x7fbe44105de0 2024-05-27T07:46:34.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.543+0000 7fbe2bfff700 1 -- 172.21.15.114:0/1874463114 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe38003f20 con 0x7fbe44105de0 2024-05-27T07:46:34.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.543+0000 7fbe2bfff700 1 -- 172.21.15.114:0/1874463114 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe38022cd0 con 0x7fbe44105de0 2024-05-27T07:46:34.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.543+0000 7fbe496f2700 1 -- 172.21.15.114:0/1874463114 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe441a0450 con 0x7fbe44105de0 2024-05-27T07:46:34.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.546+0000 7fbe2bfff700 1 -- 172.21.15.114:0/1874463114 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fbe3802a070 con 0x7fbe44105de0 2024-05-27T07:46:34.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.547+0000 7fbe2bfff700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbe2c0b29f0 0x7fbe2c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.547+0000 7fbe42ffd700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbe2c0b29f0 0x7fbe2c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.547+0000 7fbe58107b80 1 Processor -- start 2024-05-27T07:46:34.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe2bfff700 1 -- 172.21.15.114:0/1874463114 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fbe380e3800 con 0x7fbe44105de0 2024-05-27T07:46:34.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe58107b80 1 -- start start 2024-05-27T07:46:34.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe58107b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa4e8e40 0x55c0aa66c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe58107b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa66cc50 0x55c0aa666a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe58107b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa666f80 0x55c0aa667390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe58107b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c0aa595c90 con 0x55c0aa4e8e40 2024-05-27T07:46:34.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe58107b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c0aa585830 con 0x55c0aa66cc50 2024-05-27T07:46:34.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe58107b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c0aa589870 con 0x55c0aa666f80 2024-05-27T07:46:34.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe42ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa4e8e40 0x55c0aa66c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe42ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa4e8e40 0x55c0aa66c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57698/0 (socket says 172.21.15.114:57698) 2024-05-27T07:46:34.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe42ffd700 1 -- 172.21.15.114:0/3895021331 learned_addr learned my addr 172.21.15.114:0/3895021331 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:34.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe42ffd700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbe2c0b29f0 0x7fbe2c0b4f00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbe4406d120 tx=0x7fbe3400b410 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:34.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe427fc700 1 --2- 172.21.15.114:0/3895021331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa66cc50 0x55c0aa666a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe437fe700 1 --2- 172.21.15.114:0/3895021331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa666f80 0x55c0aa667390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe42ffd700 1 -- 172.21.15.114:0/3895021331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa666f80 msgr2=0x55c0aa667390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:34.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe42ffd700 1 --2- 172.21.15.114:0/3895021331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa666f80 0x55c0aa667390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe42ffd700 1 -- 172.21.15.114:0/3895021331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa66cc50 msgr2=0x55c0aa666a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:34.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe42ffd700 1 --2- 172.21.15.114:0/3895021331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa66cc50 0x55c0aa666a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe42ffd700 1 -- 172.21.15.114:0/3895021331 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0aa4d58d0 con 0x55c0aa4e8e40 2024-05-27T07:46:34.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe437fe700 1 --2- 172.21.15.114:0/3895021331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa666f80 0x55c0aa667390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:46:34.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.548+0000 7fbe427fc700 1 --2- 172.21.15.114:0/3895021331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa66cc50 0x55c0aa666a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:46:34.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.549+0000 7fbe42ffd700 1 --2- 172.21.15.114:0/3895021331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa4e8e40 0x55c0aa66c710 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fbe34010fa0 tx=0x7fbe34013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8169b14ffe4b816a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:34.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.549+0000 7fbe297fa700 1 -- 172.21.15.114:0/3895021331 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe34011e60 con 0x55c0aa4e8e40 2024-05-27T07:46:34.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.549+0000 7fbe297fa700 1 -- 172.21.15.114:0/3895021331 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe3400be40 con 0x55c0aa4e8e40 2024-05-27T07:46:34.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.549+0000 7fbe297fa700 1 -- 172.21.15.114:0/3895021331 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3401d8f0 con 0x55c0aa4e8e40 2024-05-27T07:46:34.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.550+0000 7fbe58107b80 1 -- 172.21.15.114:0/3895021331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa4e8e40 msgr2=0x55c0aa66c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:34.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.550+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3895021331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa4e8e40 0x55c0aa66c710 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fbe34010fa0 tx=0x7fbe34013000 comp rx=0 tx=0).stop 2024-05-27T07:46:34.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.550+0000 7fbe58107b80 1 -- 172.21.15.114:0/3895021331 shutdown_connections 2024-05-27T07:46:34.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.550+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3895021331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa666f80 0x55c0aa667390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.550+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3895021331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa4e8e40 0x55c0aa66c710 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.550+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3895021331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa66cc50 0x55c0aa666a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.550+0000 7fbe58107b80 1 -- 172.21.15.114:0/3895021331 >> 172.21.15.114:0/3895021331 conn(0x55c0aa543af0 msgr2=0x55c0aa661640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.550+0000 7fbe58107b80 1 -- 172.21.15.114:0/3895021331 shutdown_connections 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.550+0000 7fbe58107b80 1 -- 172.21.15.114:0/3895021331 wait complete. 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.550+0000 7fbe58107b80 1 -- start start 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.550+0000 7fbe58107b80 1 Processor -- start 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe58107b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa4e8e40 0x55c0aa670360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe58107b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 0x55c0aa6708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe58107b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa66cc50 0x55c0aa670de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe58107b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c0aa48bfa0 con 0x55c0aa666f80 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe58107b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c0aa5277c0 con 0x55c0aa4e8e40 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe58107b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c0aa673740 con 0x55c0aa66cc50 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe58107b80 10 client.0 _ll_register_callbacks cb 0x55c0aa520920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe427fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 0x55c0aa6708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe427fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 0x55c0aa6708a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57702/0 (socket says 172.21.15.114:57702) 2024-05-27T07:46:34.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe427fc700 1 -- 172.21.15.114:0/3697837113 learned_addr learned my addr 172.21.15.114:0/3697837113 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe42ffd700 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa4e8e40 0x55c0aa670360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe437fe700 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa66cc50 0x55c0aa670de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe427fc700 1 -- 172.21.15.114:0/3697837113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa66cc50 msgr2=0x55c0aa670de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe427fc700 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa66cc50 0x55c0aa670de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe427fc700 1 -- 172.21.15.114:0/3697837113 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa4e8e40 msgr2=0x55c0aa670360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe427fc700 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa4e8e40 0x55c0aa670360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe427fc700 1 -- 172.21.15.114:0/3697837113 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0aa524690 con 0x55c0aa666f80 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe437fe700 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa66cc50 0x55c0aa670de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe427fc700 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 0x55c0aa6708a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fbe38005f50 tx=0x7fbe38009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cefa1e6fd206e4b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe26ff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe26ff5700 1 -- 172.21.15.114:0/3697837113 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe380aa270 con 0x55c0aa666f80 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe26ff5700 1 -- 172.21.15.114:0/3697837113 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fbe380a6d50 con 0x55c0aa666f80 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.551+0000 7fbe58107b80 1 -- 172.21.15.114:0/3697837113 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c0aa6329f0 con 0x55c0aa666f80 2024-05-27T07:46:34.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.552+0000 7fbe26ff5700 1 -- 172.21.15.114:0/3697837113 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fbe380aae60 con 0x55c0aa666f80 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.552+0000 7fbe58107b80 10 client.17037 fetch_fsmap learned FSMap version 41 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.552+0000 7fbe58107b80 1 -- 172.21.15.114:0/3697837113 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c0aa67e130 con 0x55c0aa666f80 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.552+0000 7fbe26ff5700 1 -- 172.21.15.114:0/3697837113 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c0aa6329f0 con 0x55c0aa666f80 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.552+0000 7fbe26ff5700 1 -- 172.21.15.114:0/3697837113 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 41) ==== 7725+0+0 (secure 0 0 0) 0x7fbe380e3410 con 0x55c0aa666f80 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.552+0000 7fbe58107b80 10 client.17037 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.552+0000 7fbe58107b80 10 client.17037 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.552+0000 7fbe58107b80 20 client.17037 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.552+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c0aa680790 0x55c0aa682be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.552+0000 7fbe58107b80 4 client.17037 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.552+0000 7fbe58107b80 1 -- 172.21.15.114:0/3697837113 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0aa597be0 con 0x55c0aa680790 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.552+0000 7fbe42ffd700 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c0aa680790 0x55c0aa682be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.553+0000 7fbe42ffd700 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c0aa680790 0x55c0aa682be0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1deb1267f093ffb server_cookie=738d545cb907b8a8 in_seq=0 out_seq=0 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.553+0000 7fbe26ff5700 10 client.17037 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe26ff5700 1 -- 172.21.15.114:0/3697837113 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c0aa597be0 con 0x55c0aa680790 2024-05-27T07:46:34.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe26ff5700 10 client.17037 handle_command_reply: tid=0 2024-05-27T07:46:34.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe58107b80 1 client.17037 shutdown 2024-05-27T07:46:34.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe58107b80 1 -- 172.21.15.114:0/3697837113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 msgr2=0x55c0aa6708a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:34.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 0x55c0aa6708a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fbe38005f50 tx=0x7fbe38009ca0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe58107b80 1 -- 172.21.15.114:0/3697837113 shutdown_connections 2024-05-27T07:46:34.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa66cc50 0x55c0aa670de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c0aa680790 0x55c0aa682be0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa4e8e40 0x55c0aa670360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3697837113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 0x55c0aa6708a0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe58107b80 1 -- 172.21.15.114:0/3697837113 >> 172.21.15.114:0/3697837113 conn(0x55c0aa543af0 msgr2=0x55c0aa665a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:34.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe26ff5700 0 client.17037 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:34.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe58107b80 1 -- 172.21.15.114:0/3697837113 shutdown_connections 2024-05-27T07:46:34.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe58107b80 1 -- 172.21.15.114:0/3697837113 wait complete. 2024-05-27T07:46:34.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.554+0000 7fbe58107b80 20 client.17037 trim_cache size 0 max 16384 2024-05-27T07:46:34.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.569+0000 7fbe58107b80 1 Processor -- start 2024-05-27T07:46:34.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.569+0000 7fbe58107b80 1 -- start start 2024-05-27T07:46:34.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.569+0000 7fbe58107b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa4e8e40 0x55c0aa666a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.569+0000 7fbe58107b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa666f80 0x55c0aa66bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.569+0000 7fbe58107b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa66cc50 0x55c0aa670c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.569+0000 7fbe58107b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441b1450 con 0x55c0aa66cc50 2024-05-27T07:46:34.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.569+0000 7fbe58107b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441ae500 con 0x55c0aa666f80 2024-05-27T07:46:34.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.569+0000 7fbe58107b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441b06b0 con 0x55c0aa4e8e40 2024-05-27T07:46:34.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.569+0000 7fbe437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa66cc50 0x55c0aa670c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.569+0000 7fbe437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa66cc50 0x55c0aa670c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54052/0 (socket says 172.21.15.114:54052) 2024-05-27T07:46:34.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.569+0000 7fbe437fe700 1 -- 172.21.15.114:0/3467417252 learned_addr learned my addr 172.21.15.114:0/3467417252 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:34.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.569+0000 7fbe42ffd700 1 --2- 172.21.15.114:0/3467417252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa4e8e40 0x55c0aa666a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.569+0000 7fbe427fc700 1 --2- 172.21.15.114:0/3467417252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa666f80 0x55c0aa66bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.570+0000 7fbe437fe700 1 -- 172.21.15.114:0/3467417252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa4e8e40 msgr2=0x55c0aa666a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:34.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.570+0000 7fbe437fe700 1 --2- 172.21.15.114:0/3467417252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa4e8e40 0x55c0aa666a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.570+0000 7fbe437fe700 1 -- 172.21.15.114:0/3467417252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa666f80 msgr2=0x55c0aa66bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:34.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.570+0000 7fbe437fe700 1 --2- 172.21.15.114:0/3467417252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa666f80 0x55c0aa66bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.570+0000 7fbe437fe700 1 -- 172.21.15.114:0/3467417252 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0aa65bfa0 con 0x55c0aa66cc50 2024-05-27T07:46:34.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.570+0000 7fbe427fc700 1 --2- 172.21.15.114:0/3467417252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa666f80 0x55c0aa66bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:46:34.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.570+0000 7fbe437fe700 1 --2- 172.21.15.114:0/3467417252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa66cc50 0x55c0aa670c20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c007ee0 tx=0x7fbe3c0104c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4149ec613355409b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:34.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.570+0000 7fbe28ff9700 1 -- 172.21.15.114:0/3467417252 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c019020 con 0x55c0aa66cc50 2024-05-27T07:46:34.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.570+0000 7fbe28ff9700 1 -- 172.21.15.114:0/3467417252 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe3c010d90 con 0x55c0aa66cc50 2024-05-27T07:46:34.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.570+0000 7fbe28ff9700 1 -- 172.21.15.114:0/3467417252 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c0146f0 con 0x55c0aa66cc50 2024-05-27T07:46:34.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 -- 172.21.15.114:0/3467417252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa66cc50 msgr2=0x55c0aa670c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:34.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3467417252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa66cc50 0x55c0aa670c20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c007ee0 tx=0x7fbe3c0104c0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 -- 172.21.15.114:0/3467417252 shutdown_connections 2024-05-27T07:46:34.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3467417252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa66cc50 0x55c0aa670c20 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3467417252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa4e8e40 0x55c0aa666a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 --2- 172.21.15.114:0/3467417252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa666f80 0x55c0aa66bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 -- 172.21.15.114:0/3467417252 >> 172.21.15.114:0/3467417252 conn(0x55c0aa543af0 msgr2=0x55c0aa66b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:34.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 -- 172.21.15.114:0/3467417252 shutdown_connections 2024-05-27T07:46:34.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 -- 172.21.15.114:0/3467417252 wait complete. 2024-05-27T07:46:34.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 -- start start 2024-05-27T07:46:34.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 Processor -- start 2024-05-27T07:46:34.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa4e8e40 0x55c0aa679070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 0x55c0aa687b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa66cc50 0x55c0aa688070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441affe0 con 0x55c0aa666f80 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441aeb70 con 0x55c0aa4e8e40 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441af240 con 0x55c0aa66cc50 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.571+0000 7fbe58107b80 10 client.0 _ll_register_callbacks cb 0x55c0aa520920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe427fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 0x55c0aa687b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe427fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 0x55c0aa687b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54056/0 (socket says 172.21.15.114:54056) 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe427fc700 1 -- 172.21.15.114:0/2658157231 learned_addr learned my addr 172.21.15.114:0/2658157231 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe437fe700 1 --2- 172.21.15.114:0/2658157231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa66cc50 0x55c0aa688070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe42ffd700 1 --2- 172.21.15.114:0/2658157231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa4e8e40 0x55c0aa679070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe427fc700 1 -- 172.21.15.114:0/2658157231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa66cc50 msgr2=0x55c0aa688070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe427fc700 1 --2- 172.21.15.114:0/2658157231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa66cc50 0x55c0aa688070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe427fc700 1 -- 172.21.15.114:0/2658157231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa4e8e40 msgr2=0x55c0aa679070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe427fc700 1 --2- 172.21.15.114:0/2658157231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa4e8e40 0x55c0aa679070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe427fc700 1 -- 172.21.15.114:0/2658157231 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0aa6886b0 con 0x55c0aa666f80 2024-05-27T07:46:34.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe427fc700 1 --2- 172.21.15.114:0/2658157231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 0x55c0aa687b30 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fbe38092110 tx=0x7fbe38009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef40312901e6d8e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe26ff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe26ff5700 1 -- 172.21.15.114:0/2658157231 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe380e3590 con 0x55c0aa666f80 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe26ff5700 1 -- 172.21.15.114:0/2658157231 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fbe380a6d50 con 0x55c0aa666f80 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe58107b80 1 -- 172.21.15.114:0/2658157231 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe3c010d90 con 0x55c0aa666f80 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe26ff5700 1 -- 172.21.15.114:0/2658157231 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fbe380ca050 con 0x55c0aa666f80 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe58107b80 10 client.17043 fetch_fsmap learned FSMap version 41 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.572+0000 7fbe58107b80 1 -- 172.21.15.114:0/2658157231 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c0aa6eafb0 con 0x55c0aa666f80 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.573+0000 7fbe26ff5700 1 -- 172.21.15.114:0/2658157231 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe3806b050 con 0x55c0aa666f80 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.573+0000 7fbe26ff5700 1 -- 172.21.15.114:0/2658157231 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 41) ==== 7725+0+0 (secure 0 0 0) 0x7fbe380a60d0 con 0x55c0aa666f80 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.573+0000 7fbe58107b80 10 client.17043 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.573+0000 7fbe58107b80 10 client.17043 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.573+0000 7fbe58107b80 20 client.17043 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.573+0000 7fbe58107b80 1 --2- 172.21.15.114:0/2658157231 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c0aa6eb530 0x55c0aa6ed950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.573+0000 7fbe58107b80 4 client.17043 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:46:34.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.573+0000 7fbe58107b80 1 -- 172.21.15.114:0/2658157231 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fbe3c010d90 con 0x55c0aa6eb530 2024-05-27T07:46:34.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.573+0000 7fbe42ffd700 1 --2- 172.21.15.114:0/2658157231 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c0aa6eb530 0x55c0aa6ed950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:34.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe26ff5700 10 client.17043 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:34.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe42ffd700 1 --2- 172.21.15.114:0/2658157231 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c0aa6eb530 0x55c0aa6ed950 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd631ee8f77af9ce server_cookie=9f10fe4f2d0ea030 in_seq=0 out_seq=0 2024-05-27T07:46:34.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe26ff5700 1 -- 172.21.15.114:0/2658157231 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fbe3c010d90 con 0x55c0aa6eb530 2024-05-27T07:46:34.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe26ff5700 10 client.17043 handle_command_reply: tid=0 2024-05-27T07:46:34.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe58107b80 1 client.17043 shutdown 2024-05-27T07:46:34.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe58107b80 1 -- 172.21.15.114:0/2658157231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 msgr2=0x55c0aa687b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:34.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe58107b80 1 --2- 172.21.15.114:0/2658157231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 0x55c0aa687b30 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fbe38092110 tx=0x7fbe38009310 comp rx=0 tx=0).stop 2024-05-27T07:46:34.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe58107b80 1 -- 172.21.15.114:0/2658157231 shutdown_connections 2024-05-27T07:46:34.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe58107b80 1 --2- 172.21.15.114:0/2658157231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0aa66cc50 0x55c0aa688070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe58107b80 1 --2- 172.21.15.114:0/2658157231 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c0aa6eb530 0x55c0aa6ed950 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe26ff5700 0 client.17043 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:34.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe58107b80 1 --2- 172.21.15.114:0/2658157231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c0aa4e8e40 0x55c0aa679070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe58107b80 1 --2- 172.21.15.114:0/2658157231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c0aa666f80 0x55c0aa687b30 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe58107b80 1 -- 172.21.15.114:0/2658157231 >> 172.21.15.114:0/2658157231 conn(0x55c0aa543af0 msgr2=0x55c0aa66a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:34.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe58107b80 1 -- 172.21.15.114:0/2658157231 shutdown_connections 2024-05-27T07:46:34.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe58107b80 1 -- 172.21.15.114:0/2658157231 wait complete. 2024-05-27T07:46:34.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.574+0000 7fbe58107b80 20 client.17043 trim_cache size 0 max 16384 2024-05-27T07:46:34.579 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:46:34.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.575+0000 7fbe496f2700 1 -- 172.21.15.114:0/1874463114 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbe2c0b29f0 msgr2=0x7fbe2c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:34.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.575+0000 7fbe496f2700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbe2c0b29f0 0x7fbe2c0b4f00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbe4406d120 tx=0x7fbe3400b410 comp rx=0 tx=0).stop 2024-05-27T07:46:34.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.575+0000 7fbe496f2700 1 -- 172.21.15.114:0/1874463114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe44105de0 msgr2=0x7fbe4419b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:34.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.575+0000 7fbe496f2700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe44105de0 0x7fbe4419b460 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fbe38000c00 tx=0x7fbe380047b0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.575+0000 7fbe496f2700 1 -- 172.21.15.114:0/1874463114 shutdown_connections 2024-05-27T07:46:34.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.575+0000 7fbe496f2700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbe2c0b29f0 0x7fbe2c0b4f00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.575+0000 7fbe496f2700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe4410e900 0x7fbe4419f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.575+0000 7fbe496f2700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe44105490 0x7fbe4419aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.575+0000 7fbe496f2700 1 --2- 172.21.15.114:0/1874463114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe44105de0 0x7fbe4419b460 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:34.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.575+0000 7fbe496f2700 1 -- 172.21.15.114:0/1874463114 >> 172.21.15.114:0/1874463114 conn(0x7fbe44078e40 msgr2=0x7fbe44100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:34.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.575+0000 7fbe496f2700 1 -- 172.21.15.114:0/1874463114 shutdown_connections 2024-05-27T07:46:34.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:34.575+0000 7fbe496f2700 1 -- 172.21.15.114:0/1874463114 wait complete. 2024-05-27T07:46:35.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685993 12.25 MB/sec execute 167 sec latency 3054.667 ms 2024-05-27T07:46:35.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:35 smithi163 ceph-mon[36451]: pgmap v219: 129 pgs: 129 active+clean; 77 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 433 op/s 2024-05-27T07:46:35.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:35 smithi186 ceph-mon[36647]: pgmap v219: 129 pgs: 129 active+clean; 77 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 433 op/s 2024-05-27T07:46:35.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:35 smithi114 ceph-mon[32038]: pgmap v219: 129 pgs: 129 active+clean; 77 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 433 op/s 2024-05-27T07:46:36.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685993 12.18 MB/sec execute 168 sec latency 4054.719 ms 2024-05-27T07:46:36.380 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:46:36.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:46:36.632 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:46:37.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.011+0000 7fb0ac7b4640 1 -- 172.21.15.114:0/4082318524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0a4106b00 msgr2=0x7fb0a4106f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:37.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.011+0000 7fb0ac7b4640 1 --2- 172.21.15.114:0/4082318524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0a4106b00 0x7fb0a4106f30 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb094009900 tx=0x7fb09402a410 comp rx=0 tx=0).stop 2024-05-27T07:46:37.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.011+0000 7fb0ac7b4640 1 -- 172.21.15.114:0/4082318524 shutdown_connections 2024-05-27T07:46:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.011+0000 7fb0ac7b4640 1 --2- 172.21.15.114:0/4082318524 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0a41075c0 0x7fb0a410ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.011+0000 7fb0ac7b4640 1 --2- 172.21.15.114:0/4082318524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0a4106b00 0x7fb0a4106f30 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.011+0000 7fb0ac7b4640 1 --2- 172.21.15.114:0/4082318524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0a4102570 0x7fb0a4102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.011+0000 7fb0ac7b4640 1 -- 172.21.15.114:0/4082318524 >> 172.21.15.114:0/4082318524 conn(0x7fb0a40779d0 msgr2=0x7fb0a4077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.011+0000 7fb0ac7b4640 1 -- 172.21.15.114:0/4082318524 shutdown_connections 2024-05-27T07:46:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.011+0000 7fb0ac7b4640 1 -- 172.21.15.114:0/4082318524 wait complete. 2024-05-27T07:46:37.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.012+0000 7fb0ac7b4640 1 Processor -- start 2024-05-27T07:46:37.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.012+0000 7fb0ac7b4640 1 -- start start 2024-05-27T07:46:37.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.012+0000 7fb0ac7b4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0a4102570 0x7fb0a41a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.012+0000 7fb0ac7b4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0a4106b00 0x7fb0a41a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.012+0000 7fb0ac7b4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0a41075c0 0x7fb0a419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.012+0000 7fb0ac7b4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a4112520 con 0x7fb0a4102570 2024-05-27T07:46:37.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.012+0000 7fb0ac7b4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a41123a0 con 0x7fb0a41075c0 2024-05-27T07:46:37.014 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.012+0000 7fb0ac7b4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a41126a0 con 0x7fb0a4106b00 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.012+0000 7fb0aa550640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0a4102570 0x7fb0a41a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb0aa550640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0a4102570 0x7fb0a41a05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54086/0 (socket says 172.21.15.114:54086) 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb0aa550640 1 -- 172.21.15.114:0/3307312357 learned_addr learned my addr 172.21.15.114:0/3307312357 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb0aad51640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0a41075c0 0x7fb0a419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb0a9d4f640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0a4106b00 0x7fb0a41a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb0aa550640 1 -- 172.21.15.114:0/3307312357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0a4106b00 msgr2=0x7fb0a41a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb0aa550640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0a4106b00 0x7fb0a41a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb0aa550640 1 -- 172.21.15.114:0/3307312357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0a41075c0 msgr2=0x7fb0a419a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb0aa550640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0a41075c0 0x7fb0a419a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb0aa550640 1 -- 172.21.15.114:0/3307312357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0a419aed0 con 0x7fb0a4102570 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb0aa550640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0a4102570 0x7fb0a41a05d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000b4a0 tx=0x7fb0a000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb09b7fe640 1 -- 172.21.15.114:0/3307312357 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0014070 con 0x7fb0a4102570 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb0ac7b4640 1 -- 172.21.15.114:0/3307312357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0a419b160 con 0x7fb0a4102570 2024-05-27T07:46:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb09b7fe640 1 -- 172.21.15.114:0/3307312357 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0a0010ac0 con 0x7fb0a4102570 2024-05-27T07:46:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb09b7fe640 1 -- 172.21.15.114:0/3307312357 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0017c30 con 0x7fb0a4102570 2024-05-27T07:46:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.013+0000 7fb0ac7b4640 1 -- 172.21.15.114:0/3307312357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0a41a7370 con 0x7fb0a4102570 2024-05-27T07:46:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.016+0000 7fb09b7fe640 1 -- 172.21.15.114:0/3307312357 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fb0a0017070 con 0x7fb0a4102570 2024-05-27T07:46:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.016+0000 7fb0b3662740 1 Processor -- start 2024-05-27T07:46:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.016+0000 7fb0b3662740 1 -- start start 2024-05-27T07:46:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.016+0000 7fb0b3662740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a2f1b0 0x55f494dd03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.016+0000 7fb0b3662740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 0x55f494dd0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.016+0000 7fb0b3662740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f494a0e4d0 0x55f494dd0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.016+0000 7fb0b3662740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f494d42c30 con 0x55f494a062f0 2024-05-27T07:46:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.016+0000 7fb0b3662740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f494d327d0 con 0x55f494a0e4d0 2024-05-27T07:46:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.016+0000 7fb0b3662740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f494d36810 con 0x55f494a2f1b0 2024-05-27T07:46:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.016+0000 7fb0aa550640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 0x55f494dd0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb0aa550640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 0x55f494dd0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54102/0 (socket says 172.21.15.114:54102) 2024-05-27T07:46:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb0aa550640 1 -- 172.21.15.114:0/3129173279 learned_addr learned my addr 172.21.15.114:0/3129173279 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb0a9d4f640 1 --2- 172.21.15.114:0/3129173279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f494a0e4d0 0x55f494dd0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb0aad51640 1 --2- 172.21.15.114:0/3129173279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a2f1b0 0x55f494dd03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb09b7fe640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0900b2950 0x7fb0900b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb0aa550640 1 -- 172.21.15.114:0/3129173279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a2f1b0 msgr2=0x55f494dd03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb0aa550640 1 --2- 172.21.15.114:0/3129173279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a2f1b0 0x55f494dd03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb0aa550640 1 -- 172.21.15.114:0/3129173279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f494a0e4d0 msgr2=0x55f494dd0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb0aa550640 1 --2- 172.21.15.114:0/3129173279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f494a0e4d0 0x55f494dd0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb0aa550640 1 -- 172.21.15.114:0/3129173279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f494a47210 con 0x55f494a062f0 2024-05-27T07:46:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb0aad51640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0900b2950 0x7fb0900b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb09b7fe640 1 -- 172.21.15.114:0/3307312357 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fb0a00e1fc0 con 0x7fb0a4102570 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb0aa550640 1 --2- 172.21.15.114:0/3129173279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 0x55f494dd0910 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb0a00df3e0 tx=0x7fb0a006e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191659d2306d09bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb098ff9640 1 -- 172.21.15.114:0/3129173279 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a00a8950 con 0x55f494a062f0 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb098ff9640 1 -- 172.21.15.114:0/3129173279 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0a0017360 con 0x55f494a062f0 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.017+0000 7fb098ff9640 1 -- 172.21.15.114:0/3129173279 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a00a8770 con 0x55f494a062f0 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.018+0000 7fb0b3662740 1 -- 172.21.15.114:0/3129173279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 msgr2=0x55f494dd0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.018+0000 7fb0b3662740 1 --2- 172.21.15.114:0/3129173279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 0x55f494dd0910 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb0a00df3e0 tx=0x7fb0a006e5a0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.018+0000 7fb0aad51640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0900b2950 0x7fb0900b4e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x55f494dcd540 tx=0x7fb09c004a40 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.018+0000 7fb0b3662740 1 -- 172.21.15.114:0/3129173279 shutdown_connections 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.018+0000 7fb0b3662740 1 --2- 172.21.15.114:0/3129173279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f494a0e4d0 0x55f494dd0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.018+0000 7fb0b3662740 1 --2- 172.21.15.114:0/3129173279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 0x55f494dd0910 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.018+0000 7fb0b3662740 1 --2- 172.21.15.114:0/3129173279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a2f1b0 0x55f494dd03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.018+0000 7fb0b3662740 1 -- 172.21.15.114:0/3129173279 >> 172.21.15.114:0/3129173279 conn(0x55f494a23300 msgr2=0x55f494dccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.018+0000 7fb0b3662740 1 -- 172.21.15.114:0/3129173279 shutdown_connections 2024-05-27T07:46:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.018+0000 7fb0b3662740 1 -- 172.21.15.114:0/3129173279 wait complete. 2024-05-27T07:46:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.019+0000 7fb0b3662740 1 -- start start 2024-05-27T07:46:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.019+0000 7fb0b3662740 1 Processor -- start 2024-05-27T07:46:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.019+0000 7fb0b3662740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 0x55f494ddaa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.019+0000 7fb0aa550640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 0x55f494ddaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.019+0000 7fb0aa550640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 0x55f494ddaa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54112/0 (socket says 172.21.15.114:54112) 2024-05-27T07:46:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.019+0000 7fb0b3662740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a0e4d0 0x55f494ddaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.019+0000 7fb0aa550640 1 -- 172.21.15.114:0/845349101 learned_addr learned my addr 172.21.15.114:0/845349101 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.019+0000 7fb0b3662740 1 --2- 172.21.15.114:0/845349101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f494a2f1b0 0x55f494de0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.019+0000 7fb0b3662740 1 -- 172.21.15.114:0/845349101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f494a18790 con 0x55f494a062f0 2024-05-27T07:46:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.019+0000 7fb0b3662740 1 -- 172.21.15.114:0/845349101 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f494cee890 con 0x55f494a2f1b0 2024-05-27T07:46:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.019+0000 7fb0b3662740 1 -- 172.21.15.114:0/845349101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f494ddd4a0 con 0x55f494a0e4d0 2024-05-27T07:46:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.019+0000 7fb0b3662740 10 client.0 _ll_register_callbacks cb 0x55f494a46a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb0a9d4f640 1 --2- 172.21.15.114:0/845349101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a0e4d0 0x55f494ddaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb0aa550640 1 -- 172.21.15.114:0/845349101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a0e4d0 msgr2=0x55f494ddaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb0aa550640 1 --2- 172.21.15.114:0/845349101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a0e4d0 0x55f494ddaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb0aa550640 1 -- 172.21.15.114:0/845349101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f494a2f1b0 msgr2=0x55f494de0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb0aa550640 1 --2- 172.21.15.114:0/845349101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f494a2f1b0 0x55f494de0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb0aa550640 1 -- 172.21.15.114:0/845349101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f494b13620 con 0x55f494a062f0 2024-05-27T07:46:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb0aa550640 1 --2- 172.21.15.114:0/845349101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 0x55f494ddaa90 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb0a00a6270 tx=0x7fb0a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1c72a45ae40ff5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb08dffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb08dffb640 1 -- 172.21.15.114:0/845349101 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a001e050 con 0x55f494a062f0 2024-05-27T07:46:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb08dffb640 1 -- 172.21.15.114:0/845349101 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fb0a00df550 con 0x55f494a062f0 2024-05-27T07:46:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb0b3662740 1 -- 172.21.15.114:0/845349101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f494d44b80 con 0x55f494a062f0 2024-05-27T07:46:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb08dffb640 1 -- 172.21.15.114:0/845349101 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fb0a00a8770 con 0x55f494a062f0 2024-05-27T07:46:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb0b3662740 10 client.17055 fetch_fsmap learned FSMap version 41 2024-05-27T07:46:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb0b3662740 1 -- 172.21.15.114:0/845349101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f494de95b0 con 0x55f494a062f0 2024-05-27T07:46:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.020+0000 7fb08dffb640 1 -- 172.21.15.114:0/845349101 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f494d44b80 con 0x55f494a062f0 2024-05-27T07:46:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.021+0000 7fb08dffb640 1 -- 172.21.15.114:0/845349101 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 41) ==== 7725+0+0 (secure 0 0 0) 0x7fb0a006e180 con 0x55f494a062f0 2024-05-27T07:46:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.021+0000 7fb0b3662740 10 client.17055 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:46:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.021+0000 7fb0b3662740 10 client.17055 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.021+0000 7fb0b3662740 20 client.17055 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.021+0000 7fb0b3662740 1 --2- 172.21.15.114:0/845349101 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f4949edd80 0x55f494dec910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.021+0000 7fb0b3662740 4 client.17055 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:46:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.021+0000 7fb0b3662740 1 -- 172.21.15.114:0/845349101 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f494d3e360 con 0x55f4949edd80 2024-05-27T07:46:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.021+0000 7fb0a9d4f640 1 --2- 172.21.15.114:0/845349101 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f4949edd80 0x55f494dec910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.021+0000 7fb08dffb640 10 client.17055 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.021+0000 7fb0a9d4f640 1 --2- 172.21.15.114:0/845349101 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f4949edd80 0x55f494dec910 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6c4285f955f425b server_cookie=3b104ace68fe0b13 in_seq=0 out_seq=0 2024-05-27T07:46:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb08dffb640 1 -- 172.21.15.114:0/845349101 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f494d3e360 con 0x55f4949edd80 2024-05-27T07:46:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb08dffb640 10 client.17055 handle_command_reply: tid=0 2024-05-27T07:46:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb0b3662740 1 client.17055 shutdown 2024-05-27T07:46:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb0b3662740 1 -- 172.21.15.114:0/845349101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 msgr2=0x55f494ddaa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb0b3662740 1 --2- 172.21.15.114:0/845349101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 0x55f494ddaa90 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb0a00a6270 tx=0x7fb0a00046c0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb0b3662740 1 -- 172.21.15.114:0/845349101 shutdown_connections 2024-05-27T07:46:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb0b3662740 1 --2- 172.21.15.114:0/845349101 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f4949edd80 0x55f494dec910 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb0b3662740 1 --2- 172.21.15.114:0/845349101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f494a2f1b0 0x55f494de0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb08dffb640 0 client.17055 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb0b3662740 1 --2- 172.21.15.114:0/845349101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a0e4d0 0x55f494ddaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb0b3662740 1 --2- 172.21.15.114:0/845349101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a062f0 0x55f494ddaa90 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb0b3662740 1 -- 172.21.15.114:0/845349101 >> 172.21.15.114:0/845349101 conn(0x55f494a23300 msgr2=0x55f494dd20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb0b3662740 1 -- 172.21.15.114:0/845349101 shutdown_connections 2024-05-27T07:46:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb0b3662740 1 -- 172.21.15.114:0/845349101 wait complete. 2024-05-27T07:46:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.023+0000 7fb0b3662740 20 client.17055 trim_cache size 0 max 16384 2024-05-27T07:46:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.038+0000 7fb0b3662740 1 Processor -- start 2024-05-27T07:46:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.038+0000 7fb0b3662740 1 -- start start 2024-05-27T07:46:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.038+0000 7fb0b3662740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4949edd80 0x55f494ddcd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.038+0000 7fb0b3662740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a062f0 0x55f494dd2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.038+0000 7fb0b3662740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 0x55f494dd4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.038+0000 7fb0b3662740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a41b22f0 con 0x55f494a0e4d0 2024-05-27T07:46:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.038+0000 7fb0b3662740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a41af3a0 con 0x55f4949edd80 2024-05-27T07:46:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.038+0000 7fb0b3662740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a41b1550 con 0x55f494a062f0 2024-05-27T07:46:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.038+0000 7fb0aad51640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 0x55f494dd4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.038+0000 7fb0aad51640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 0x55f494dd4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54118/0 (socket says 172.21.15.114:54118) 2024-05-27T07:46:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.038+0000 7fb0aad51640 1 -- 172.21.15.114:0/1109665243 learned_addr learned my addr 172.21.15.114:0/1109665243 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.038+0000 7fb0aa550640 1 --2- 172.21.15.114:0/1109665243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4949edd80 0x55f494ddcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.039+0000 7fb0aad51640 1 -- 172.21.15.114:0/1109665243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a062f0 msgr2=0x55f494dd2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.039+0000 7fb0aad51640 1 --2- 172.21.15.114:0/1109665243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a062f0 0x55f494dd2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.039+0000 7fb0aad51640 1 -- 172.21.15.114:0/1109665243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4949edd80 msgr2=0x55f494ddcd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.039+0000 7fb0aad51640 1 --2- 172.21.15.114:0/1109665243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4949edd80 0x55f494ddcd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.039+0000 7fb0aad51640 1 -- 172.21.15.114:0/1109665243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f494dd8df0 con 0x55f494a0e4d0 2024-05-27T07:46:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.039+0000 7fb0aa550640 1 --2- 172.21.15.114:0/1109665243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4949edd80 0x55f494ddcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:46:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.039+0000 7fb0aad51640 1 --2- 172.21.15.114:0/1109665243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 0x55f494dd4150 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00afa0 tx=0x7fb09c015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5409bf34c7aac11f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.039+0000 7fb08ffff640 1 -- 172.21.15.114:0/1109665243 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb09c022070 con 0x55f494a0e4d0 2024-05-27T07:46:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.039+0000 7fb08ffff640 1 -- 172.21.15.114:0/1109665243 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb09c003e20 con 0x55f494a0e4d0 2024-05-27T07:46:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.039+0000 7fb08ffff640 1 -- 172.21.15.114:0/1109665243 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb09c01e660 con 0x55f494a0e4d0 2024-05-27T07:46:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 -- 172.21.15.114:0/1109665243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 msgr2=0x55f494dd4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 --2- 172.21.15.114:0/1109665243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 0x55f494dd4150 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00afa0 tx=0x7fb09c015040 comp rx=0 tx=0).stop 2024-05-27T07:46:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 -- 172.21.15.114:0/1109665243 shutdown_connections 2024-05-27T07:46:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 --2- 172.21.15.114:0/1109665243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 0x55f494dd4150 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 --2- 172.21.15.114:0/1109665243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a062f0 0x55f494dd2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 --2- 172.21.15.114:0/1109665243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4949edd80 0x55f494ddcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 -- 172.21.15.114:0/1109665243 >> 172.21.15.114:0/1109665243 conn(0x55f494a23300 msgr2=0x55f494dd2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 -- 172.21.15.114:0/1109665243 shutdown_connections 2024-05-27T07:46:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 -- 172.21.15.114:0/1109665243 wait complete. 2024-05-27T07:46:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 -- start start 2024-05-27T07:46:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 Processor -- start 2024-05-27T07:46:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4949edd80 0x55f494de6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a062f0 0x55f494de21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.040+0000 7fb0b3662740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 0x55f494de2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0b3662740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a41b0e80 con 0x55f494a0e4d0 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0b3662740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a41afa10 con 0x55f4949edd80 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0b3662740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a41b00e0 con 0x55f494a062f0 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0b3662740 10 client.0 _ll_register_callbacks cb 0x7fb094009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0aad51640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 0x55f494de2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0aad51640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 0x55f494de2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54132/0 (socket says 172.21.15.114:54132) 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0aad51640 1 -- 172.21.15.114:0/1953333134 learned_addr learned my addr 172.21.15.114:0/1953333134 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0a9d4f640 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a062f0 0x55f494de21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0aa550640 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4949edd80 0x55f494de6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0aad51640 1 -- 172.21.15.114:0/1953333134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a062f0 msgr2=0x55f494de21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0aad51640 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a062f0 0x55f494de21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0aad51640 1 -- 172.21.15.114:0/1953333134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4949edd80 msgr2=0x55f494de6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0aad51640 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4949edd80 0x55f494de6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0aad51640 1 -- 172.21.15.114:0/1953333134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f494de2d90 con 0x55f494a0e4d0 2024-05-27T07:46:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0aa550640 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4949edd80 0x55f494de6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:46:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0aad51640 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 0x55f494de2720 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00af70 tx=0x7fb09c0048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622dae49c167b7f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb08dffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb08dffb640 1 -- 172.21.15.114:0/1953333134 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb09c022070 con 0x55f494a0e4d0 2024-05-27T07:46:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb08dffb640 1 -- 172.21.15.114:0/1953333134 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fb09c004230 con 0x55f494a0e4d0 2024-05-27T07:46:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0b3662740 1 -- 172.21.15.114:0/1953333134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb09c003e20 con 0x55f494a0e4d0 2024-05-27T07:46:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb08dffb640 1 -- 172.21.15.114:0/1953333134 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fb09c003970 con 0x55f494a0e4d0 2024-05-27T07:46:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0b3662740 10 client.17061 fetch_fsmap learned FSMap version 41 2024-05-27T07:46:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.041+0000 7fb0b3662740 1 -- 172.21.15.114:0/1953333134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f494e54b00 con 0x55f494a0e4d0 2024-05-27T07:46:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.042+0000 7fb08dffb640 1 -- 172.21.15.114:0/1953333134 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb09c003e20 con 0x55f494a0e4d0 2024-05-27T07:46:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.042+0000 7fb08dffb640 1 -- 172.21.15.114:0/1953333134 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 41) ==== 7725+0+0 (secure 0 0 0) 0x7fb09c01b040 con 0x55f494a0e4d0 2024-05-27T07:46:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.042+0000 7fb0b3662740 10 client.17061 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:46:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.042+0000 7fb0b3662740 10 client.17061 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.042+0000 7fb0b3662740 20 client.17061 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.042+0000 7fb0b3662740 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f494a2f1b0 0x55f494e57130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.042+0000 7fb0b3662740 4 client.17061 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:46:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.042+0000 7fb0b3662740 1 -- 172.21.15.114:0/1953333134 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f494cb85f0 con 0x55f494a2f1b0 2024-05-27T07:46:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.042+0000 7fb0aa550640 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f494a2f1b0 0x55f494e57130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.043+0000 7fb08dffb640 10 client.17061 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.043+0000 7fb0aa550640 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f494a2f1b0 0x55f494e57130 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f5b9965cffbe814 server_cookie=7a084d0905d3a505 in_seq=0 out_seq=0 2024-05-27T07:46:37.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685993 12.11 MB/sec execute 169 sec latency 5054.770 ms 2024-05-27T07:46:37.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.332+0000 7fb08dffb640 1 -- 172.21.15.114:0/1953333134 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f494cb85f0 con 0x55f494a2f1b0 2024-05-27T07:46:37.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.332+0000 7fb08dffb640 10 client.17061 handle_command_reply: tid=0 2024-05-27T07:46:37.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.332+0000 7fb0b3662740 1 client.17061 shutdown 2024-05-27T07:46:37.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.332+0000 7fb0b3662740 1 -- 172.21.15.114:0/1953333134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 msgr2=0x55f494de2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:37.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.332+0000 7fb0b3662740 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 0x55f494de2720 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00af70 tx=0x7fb09c0048a0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.333+0000 7fb0b3662740 1 -- 172.21.15.114:0/1953333134 shutdown_connections 2024-05-27T07:46:37.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.333+0000 7fb0b3662740 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f494a2f1b0 0x55f494e57130 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.333+0000 7fb08dffb640 0 client.17061 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.333+0000 7fb0b3662740 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f494a0e4d0 0x55f494de2720 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.333+0000 7fb0b3662740 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f494a062f0 0x55f494de21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.333+0000 7fb0b3662740 1 --2- 172.21.15.114:0/1953333134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f4949edd80 0x55f494de6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.333+0000 7fb0b3662740 1 -- 172.21.15.114:0/1953333134 >> 172.21.15.114:0/1953333134 conn(0x55f494a23300 msgr2=0x55f494ddbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.333+0000 7fb0b3662740 1 -- 172.21.15.114:0/1953333134 shutdown_connections 2024-05-27T07:46:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.333+0000 7fb0b3662740 1 -- 172.21.15.114:0/1953333134 wait complete. 2024-05-27T07:46:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.333+0000 7fb0b3662740 20 client.17061 trim_cache size 0 max 16384 2024-05-27T07:46:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:46:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:46:37.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:46:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.334+0000 7fb0ac7b4640 1 -- 172.21.15.114:0/3307312357 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0900b2950 msgr2=0x7fb0900b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.334+0000 7fb0ac7b4640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0900b2950 0x7fb0900b4e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x55f494dcd540 tx=0x7fb09c004a40 comp rx=0 tx=0).stop 2024-05-27T07:46:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.334+0000 7fb0ac7b4640 1 -- 172.21.15.114:0/3307312357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0a4102570 msgr2=0x7fb0a41a05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.334+0000 7fb0ac7b4640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0a4102570 0x7fb0a41a05d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000b4a0 tx=0x7fb0a000b960 comp rx=0 tx=0).stop 2024-05-27T07:46:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.334+0000 7fb0ac7b4640 1 -- 172.21.15.114:0/3307312357 shutdown_connections 2024-05-27T07:46:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.334+0000 7fb0ac7b4640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0900b2950 0x7fb0900b4e60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.335+0000 7fb0ac7b4640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0a41075c0 0x7fb0a419a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.335+0000 7fb0ac7b4640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0a4106b00 0x7fb0a41a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.335+0000 7fb0ac7b4640 1 --2- 172.21.15.114:0/3307312357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0a4102570 0x7fb0a41a05d0 secure :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000b4a0 tx=0x7fb0a000b960 comp rx=0 tx=0).stop 2024-05-27T07:46:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.335+0000 7fb0ac7b4640 1 -- 172.21.15.114:0/3307312357 >> 172.21.15.114:0/3307312357 conn(0x7fb0a40779d0 msgr2=0x7fb0a406a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.335+0000 7fb0ac7b4640 1 -- 172.21.15.114:0/3307312357 shutdown_connections 2024-05-27T07:46:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:37.335+0000 7fb0ac7b4640 1 -- 172.21.15.114:0/3307312357 wait complete. 2024-05-27T07:46:37.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:37 smithi163 ceph-mon[36451]: pgmap v220: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 215 op/s 2024-05-27T07:46:37.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:37 smithi186 ceph-mon[36647]: pgmap v220: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 215 op/s 2024-05-27T07:46:37.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:37 smithi114 ceph-mon[32038]: pgmap v220: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 215 op/s 2024-05-27T07:46:38.196 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f434f912700 time 2024-05-27T07:46:37.812740+0000 2024-05-27T07:46:38.196 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:46:38.196 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:38.196 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4362157ba6] 2024-05-27T07:46:38.196 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4362157dd1] 2024-05-27T07:46:38.196 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56236d5f923d] 2024-05-27T07:46:38.196 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56236d5e4971] 2024-05-27T07:46:38.196 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 5: (SnapRealm::check_cache() const+0x13e) [0x56236d5e2fa0] 2024-05-27T07:46:38.196 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56236d5e3651] 2024-05-27T07:46:38.196 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56236d578347] 2024-05-27T07:46:38.196 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56236d27487e] 2024-05-27T07:46:38.197 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56236d279f2f] 2024-05-27T07:46:38.197 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56236d284969] 2024-05-27T07:46:38.197 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 11: (MDLog::_replay_thread()+0x1f38) [0x56236d61d48c] 2024-05-27T07:46:38.197 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56236d6269cb] 2024-05-27T07:46:38.197 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 13: (Thread::entry_wrapper()+0x43) [0x7f436212672d] 2024-05-27T07:46:38.197 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4362126749] 2024-05-27T07:46:38.197 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4360cca1ca] 2024-05-27T07:46:38.197 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 16: clone() 2024-05-27T07:46:38.197 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: *** Caught signal (Aborted) ** 2024-05-27T07:46:38.197 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: in thread 7f434f912700 thread_name:md_log_replay 2024-05-27T07:46:38.198 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:37.814+0000 7f434f912700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f434f912700 time 2024-05-27T07:46:37.812740+0000 2024-05-27T07:46:38.198 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:46:38.198 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:38.198 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:38.198 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4362157ba6] 2024-05-27T07:46:38.198 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4362157dd1] 2024-05-27T07:46:38.198 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56236d5f923d] 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56236d5e4971] 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 5: (SnapRealm::check_cache() const+0x13e) [0x56236d5e2fa0] 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56236d5e3651] 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56236d578347] 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56236d27487e] 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56236d279f2f] 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56236d284969] 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 11: (MDLog::_replay_thread()+0x1f38) [0x56236d61d48c] 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56236d6269cb] 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 13: (Thread::entry_wrapper()+0x43) [0x7f436212672d] 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4362126749] 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4360cca1ca] 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 16: clone() 2024-05-27T07:46:38.199 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4360cd4d20] 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2: gsignal() 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 3: abort() 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4362157ced] 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4362157dd1] 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56236d5f923d] 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56236d5e4971] 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 8: (SnapRealm::check_cache() const+0x13e) [0x56236d5e2fa0] 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56236d5e3651] 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56236d578347] 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56236d27487e] 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56236d279f2f] 2024-05-27T07:46:38.200 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56236d284969] 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 14: (MDLog::_replay_thread()+0x1f38) [0x56236d61d48c] 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56236d6269cb] 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 16: (Thread::entry_wrapper()+0x43) [0x7f436212672d] 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4362126749] 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4360cca1ca] 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 19: clone() 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:37.816+0000 7f434f912700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: in thread 7f434f912700 thread_name:md_log_replay 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4360cd4d20] 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2: gsignal() 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 3: abort() 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4362157ced] 2024-05-27T07:46:38.201 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4362157dd1] 2024-05-27T07:46:38.202 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56236d5f923d] 2024-05-27T07:46:38.202 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56236d5e4971] 2024-05-27T07:46:38.202 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 8: (SnapRealm::check_cache() const+0x13e) [0x56236d5e2fa0] 2024-05-27T07:46:38.202 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56236d5e3651] 2024-05-27T07:46:38.202 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56236d578347] 2024-05-27T07:46:38.202 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56236d27487e] 2024-05-27T07:46:38.202 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56236d279f2f] 2024-05-27T07:46:38.202 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56236d284969] 2024-05-27T07:46:38.202 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 14: (MDLog::_replay_thread()+0x1f38) [0x56236d61d48c] 2024-05-27T07:46:38.202 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56236d6269cb] 2024-05-27T07:46:38.202 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 16: (Thread::entry_wrapper()+0x43) [0x7f436212672d] 2024-05-27T07:46:38.202 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4362126749] 2024-05-27T07:46:38.202 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4360cca1ca] 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 19: clone() 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: -1> 2024-05-27T07:46:37.814+0000 7f434f912700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f434f912700 time 2024-05-27T07:46:37.812740+0000 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4362157ba6] 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4362157dd1] 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56236d5f923d] 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56236d5e4971] 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 5: (SnapRealm::check_cache() const+0x13e) [0x56236d5e2fa0] 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56236d5e3651] 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56236d578347] 2024-05-27T07:46:38.203 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56236d27487e] 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56236d279f2f] 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56236d284969] 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 11: (MDLog::_replay_thread()+0x1f38) [0x56236d61d48c] 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56236d6269cb] 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 13: (Thread::entry_wrapper()+0x43) [0x7f436212672d] 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4362126749] 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4360cca1ca] 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 16: clone() 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 0> 2024-05-27T07:46:37.816+0000 7f434f912700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: in thread 7f434f912700 thread_name:md_log_replay 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:38.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4360cd4d20] 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2: gsignal() 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 3: abort() 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4362157ced] 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4362157dd1] 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56236d5f923d] 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56236d5e4971] 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 8: (SnapRealm::check_cache() const+0x13e) [0x56236d5e2fa0] 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56236d5e3651] 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56236d578347] 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56236d27487e] 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56236d279f2f] 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56236d284969] 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 14: (MDLog::_replay_thread()+0x1f38) [0x56236d61d48c] 2024-05-27T07:46:38.205 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56236d6269cb] 2024-05-27T07:46:38.206 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 16: (Thread::entry_wrapper()+0x43) [0x7f436212672d] 2024-05-27T07:46:38.206 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4362126749] 2024-05-27T07:46:38.206 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4360cca1ca] 2024-05-27T07:46:38.206 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 19: clone() 2024-05-27T07:46:38.206 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:46:38.206 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:38.206 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: -1> 2024-05-27T07:46:37.814+0000 7f434f912700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f434f912700 time 2024-05-27T07:46:37.812740+0000 2024-05-27T07:46:38.206 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4362157ba6] 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4362157dd1] 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56236d5f923d] 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56236d5e4971] 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 5: (SnapRealm::check_cache() const+0x13e) [0x56236d5e2fa0] 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56236d5e3651] 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56236d578347] 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56236d27487e] 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56236d279f2f] 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56236d284969] 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 11: (MDLog::_replay_thread()+0x1f38) [0x56236d61d48c] 2024-05-27T07:46:38.207 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56236d6269cb] 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 13: (Thread::entry_wrapper()+0x43) [0x7f436212672d] 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4362126749] 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4360cca1ca] 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 16: clone() 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 0> 2024-05-27T07:46:37.816+0000 7f434f912700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: in thread 7f434f912700 thread_name:md_log_replay 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4360cd4d20] 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2: gsignal() 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 3: abort() 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4362157ced] 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4362157dd1] 2024-05-27T07:46:38.208 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56236d5f923d] 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56236d5e4971] 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 8: (SnapRealm::check_cache() const+0x13e) [0x56236d5e2fa0] 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56236d5e3651] 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56236d578347] 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56236d27487e] 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56236d279f2f] 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56236d284969] 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 14: (MDLog::_replay_thread()+0x1f38) [0x56236d61d48c] 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56236d6269cb] 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 16: (Thread::entry_wrapper()+0x43) [0x7f436212672d] 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4362126749] 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4360cca1ca] 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 19: clone() 2024-05-27T07:46:38.209 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:46:38.210 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73176]: 2024-05-27T07:46:38.210 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:37 smithi186 podman[73214]: 2024-05-27 07:46:37.88735958 +0000 UTC m=+0.019531067 container died 2ca6a9d92e05f765e85641c92951e9c442aeea241d16e66adb1974847ed86404 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-27T07:46:38.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 687841 12.08 MB/sec execute 170 sec latency 5110.501 ms 2024-05-27T07:46:38.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.427+0000 7fb86b34f700 1 -- 172.21.15.114:0/1175021034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8641045e0 msgr2=0x7fb8641049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:38.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.427+0000 7fb86b34f700 1 --2- 172.21.15.114:0/1175021034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8641045e0 0x7fb8641049f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fb8600099c0 tx=0x7fb860009cd0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.428+0000 7fb86b34f700 1 -- 172.21.15.114:0/1175021034 shutdown_connections 2024-05-27T07:46:38.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.428+0000 7fb86b34f700 1 --2- 172.21.15.114:0/1175021034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb864109860 0x7fb8641111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.428+0000 7fb86b34f700 1 --2- 172.21.15.114:0/1175021034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8641045e0 0x7fb8641049f0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.428+0000 7fb86b34f700 1 --2- 172.21.15.114:0/1175021034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb864108f10 0x7fb864109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.428+0000 7fb86b34f700 1 -- 172.21.15.114:0/1175021034 >> 172.21.15.114:0/1175021034 conn(0x7fb8640fffd0 msgr2=0x7fb8641023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:38.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.428+0000 7fb86b34f700 1 -- 172.21.15.114:0/1175021034 shutdown_connections 2024-05-27T07:46:38.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.428+0000 7fb86b34f700 1 -- 172.21.15.114:0/1175021034 wait complete. 2024-05-27T07:46:38.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.428+0000 7fb86b34f700 1 Processor -- start 2024-05-27T07:46:38.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.428+0000 7fb86b34f700 1 -- start start 2024-05-27T07:46:38.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb86b34f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8641045e0 0x7fb86419ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb86b34f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb864108f10 0x7fb86419b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb86b34f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb864109860 0x7fb86419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb86b34f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb864114490 con 0x7fb8641045e0 2024-05-27T07:46:38.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb86b34f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb864114310 con 0x7fb864108f10 2024-05-27T07:46:38.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb86b34f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb864114610 con 0x7fb864109860 2024-05-27T07:46:38.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb8690eb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8641045e0 0x7fb86419ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb8690eb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8641045e0 0x7fb86419ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54140/0 (socket says 172.21.15.114:54140) 2024-05-27T07:46:38.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb8688ea700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb864108f10 0x7fb86419b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb8690eb700 1 -- 172.21.15.114:0/3061147477 learned_addr learned my addr 172.21.15.114:0/3061147477 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:38.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb8698ec700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb864109860 0x7fb86419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb8688ea700 1 -- 172.21.15.114:0/3061147477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb864109860 msgr2=0x7fb86419f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:38.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb8688ea700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb864109860 0x7fb86419f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb8688ea700 1 -- 172.21.15.114:0/3061147477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8641045e0 msgr2=0x7fb86419ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:38.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb8688ea700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8641045e0 0x7fb86419ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb8688ea700 1 -- 172.21.15.114:0/3061147477 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb86419fd10 con 0x7fb864108f10 2024-05-27T07:46:38.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb8690eb700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8641045e0 0x7fb86419ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:38.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.429+0000 7fb8688ea700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb864108f10 0x7fb86419b3e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb85400e9c0 tx=0x7fb85400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:38.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.430+0000 7fb85a7fc700 1 -- 172.21.15.114:0/3061147477 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85400c9c0 con 0x7fb864108f10 2024-05-27T07:46:38.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.430+0000 7fb86b34f700 1 -- 172.21.15.114:0/3061147477 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8641a0000 con 0x7fb864108f10 2024-05-27T07:46:38.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.430+0000 7fb85a7fc700 1 -- 172.21.15.114:0/3061147477 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb85400cb60 con 0x7fb864108f10 2024-05-27T07:46:38.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.430+0000 7fb85a7fc700 1 -- 172.21.15.114:0/3061147477 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb854018680 con 0x7fb864108f10 2024-05-27T07:46:38.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.430+0000 7fb86b34f700 1 -- 172.21.15.114:0/3061147477 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8641a7920 con 0x7fb864108f10 2024-05-27T07:46:38.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.433+0000 7fb85a7fc700 1 -- 172.21.15.114:0/3061147477 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fb854010430 con 0x7fb864108f10 2024-05-27T07:46:38.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.434+0000 7fb85a7fc700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8500b29f0 0x7fb8500b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.434+0000 7fb879d64b80 1 Processor -- start 2024-05-27T07:46:38.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.434+0000 7fb8690eb700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8500b29f0 0x7fb8500b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.434+0000 7fb85a7fc700 1 -- 172.21.15.114:0/3061147477 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fb8540e2b40 con 0x7fb864108f10 2024-05-27T07:46:38.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.434+0000 7fb879d64b80 1 -- start start 2024-05-27T07:46:38.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb879d64b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8adc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb879d64b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8b1bf0 0x5637ff8ae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb879d64b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb879d64b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5637ff7dbdf0 con 0x5637ff72eec0 2024-05-27T07:46:38.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb879d64b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5637ff7cb990 con 0x5637ff8ae7a0 2024-05-27T07:46:38.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb879d64b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5637ff7cf9d0 con 0x5637ff8b1bf0 2024-05-27T07:46:38.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb8690eb700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8500b29f0 0x7fb8500b4f00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb860009990 tx=0x7fb86001a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:38.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb8690eb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb8690eb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8adc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54156/0 (socket says 172.21.15.114:54156) 2024-05-27T07:46:38.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb8690eb700 1 -- 172.21.15.114:0/412338911 learned_addr learned my addr 172.21.15.114:0/412338911 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:38.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb8688ea700 1 --2- 172.21.15.114:0/412338911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8b1bf0 0x5637ff8ae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb8698ec700 1 --2- 172.21.15.114:0/412338911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb8690eb700 1 -- 172.21.15.114:0/412338911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8b1bf0 msgr2=0x5637ff8ae1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:38.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb8690eb700 1 --2- 172.21.15.114:0/412338911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8b1bf0 0x5637ff8ae1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb8690eb700 1 -- 172.21.15.114:0/412338911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 msgr2=0x5637ff8c28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:38.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb8690eb700 1 --2- 172.21.15.114:0/412338911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb8690eb700 1 -- 172.21.15.114:0/412338911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637ff775030 con 0x5637ff72eec0 2024-05-27T07:46:38.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb8688ea700 1 --2- 172.21.15.114:0/412338911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8b1bf0 0x5637ff8ae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:38.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.435+0000 7fb8690eb700 1 --2- 172.21.15.114:0/412338911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8adc60 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb860000c00 tx=0x7fb86001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff829fc5338817d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:38.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.436+0000 7fb84f7fe700 1 -- 172.21.15.114:0/412338911 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8600056b0 con 0x5637ff72eec0 2024-05-27T07:46:38.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.436+0000 7fb84f7fe700 1 -- 172.21.15.114:0/412338911 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb860003c70 con 0x5637ff72eec0 2024-05-27T07:46:38.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.436+0000 7fb84f7fe700 1 -- 172.21.15.114:0/412338911 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86000fbd0 con 0x5637ff72eec0 2024-05-27T07:46:38.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.437+0000 7fb879d64b80 1 -- 172.21.15.114:0/412338911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 msgr2=0x5637ff8adc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:38.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.437+0000 7fb879d64b80 1 --2- 172.21.15.114:0/412338911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8adc60 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb860000c00 tx=0x7fb86001f000 comp rx=0 tx=0).stop 2024-05-27T07:46:38.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.437+0000 7fb879d64b80 1 -- 172.21.15.114:0/412338911 shutdown_connections 2024-05-27T07:46:38.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.437+0000 7fb879d64b80 1 --2- 172.21.15.114:0/412338911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.437+0000 7fb879d64b80 1 --2- 172.21.15.114:0/412338911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8adc60 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.437+0000 7fb879d64b80 1 --2- 172.21.15.114:0/412338911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8b1bf0 0x5637ff8ae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.437+0000 7fb879d64b80 1 -- 172.21.15.114:0/412338911 >> 172.21.15.114:0/412338911 conn(0x5637ff7317f0 msgr2=0x5637ff8a8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:38.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.437+0000 7fb879d64b80 1 -- 172.21.15.114:0/412338911 shutdown_connections 2024-05-27T07:46:38.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.437+0000 7fb879d64b80 1 -- 172.21.15.114:0/412338911 wait complete. 2024-05-27T07:46:38.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.437+0000 7fb879d64b80 1 -- start start 2024-05-27T07:46:38.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.437+0000 7fb879d64b80 1 Processor -- start 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb879d64b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8bff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb879d64b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb879d64b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8b1bf0 0x5637ff8c09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb879d64b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5637ff6d2020 con 0x5637ff72eec0 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb879d64b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5637ff733e20 con 0x5637ff8b1bf0 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb879d64b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5637ff8c33a0 con 0x5637ff8ae7a0 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb879d64b80 10 client.0 _ll_register_callbacks cb 0x5637ff762630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb8690eb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8bff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb8688ea700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb8690eb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8bff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54160/0 (socket says 172.21.15.114:54160) 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb8690eb700 1 -- 172.21.15.114:0/3642362440 learned_addr learned my addr 172.21.15.114:0/3642362440 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb8698ec700 1 --2- 172.21.15.114:0/3642362440 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8b1bf0 0x5637ff8c09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb8688ea700 1 -- 172.21.15.114:0/3642362440 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8b1bf0 msgr2=0x5637ff8c09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb8688ea700 1 --2- 172.21.15.114:0/3642362440 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8b1bf0 0x5637ff8c09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb8688ea700 1 -- 172.21.15.114:0/3642362440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 msgr2=0x5637ff8bff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb8688ea700 1 --2- 172.21.15.114:0/3642362440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8bff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb8688ea700 1 -- 172.21.15.114:0/3642362440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637ff71b950 con 0x5637ff8ae7a0 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb8688ea700 1 --2- 172.21.15.114:0/3642362440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c0480 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb8540dfee0 tx=0x7fb8540a5f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87fcdc8733f32b0e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.438+0000 7fb84cff9700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb84cff9700 1 -- 172.21.15.114:0/3642362440 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8540a9a70 con 0x5637ff8ae7a0 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb879d64b80 1 -- 172.21.15.114:0/3642362440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb860003c70 con 0x5637ff8ae7a0 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb84cff9700 1 -- 172.21.15.114:0/3642362440 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fb8540b2650 con 0x5637ff8ae7a0 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb84cff9700 1 -- 172.21.15.114:0/3642362440 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fb8540ade20 con 0x5637ff8ae7a0 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb879d64b80 10 client.26641 fetch_fsmap learned FSMap version 41 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb879d64b80 1 -- 172.21.15.114:0/3642362440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5637ff7857e0 con 0x5637ff8ae7a0 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb84cff9700 1 -- 172.21.15.114:0/3642362440 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb860003c70 con 0x5637ff8ae7a0 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb84cff9700 1 -- 172.21.15.114:0/3642362440 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 41) ==== 7725+0+0 (secure 0 0 0) 0x7fb8540a8df0 con 0x5637ff8ae7a0 2024-05-27T07:46:38.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb879d64b80 10 client.26641 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:46:38.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb879d64b80 10 client.26641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:38.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb879d64b80 20 client.26641 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:38.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb879d64b80 1 --2- 172.21.15.114:0/3642362440 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5637ff8d0330 0x5637ff8d2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb879d64b80 4 client.26641 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:46:38.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb879d64b80 1 -- 172.21.15.114:0/3642362440 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637ff7ddd40 con 0x5637ff8d0330 2024-05-27T07:46:38.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.439+0000 7fb8690eb700 1 --2- 172.21.15.114:0/3642362440 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5637ff8d0330 0x5637ff8d2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.440+0000 7fb84cff9700 10 client.26641 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:38.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.440+0000 7fb8690eb700 1 --2- 172.21.15.114:0/3642362440 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5637ff8d0330 0x5637ff8d2750 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4d3f65fdc5b707d server_cookie=c43adf2ad93a9b8e in_seq=0 out_seq=0 2024-05-27T07:46:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.440+0000 7fb84cff9700 1 -- 172.21.15.114:0/3642362440 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5637ff7ddd40 con 0x5637ff8d0330 2024-05-27T07:46:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb84cff9700 10 client.26641 handle_command_reply: tid=0 2024-05-27T07:46:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb879d64b80 1 client.26641 shutdown 2024-05-27T07:46:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb879d64b80 1 -- 172.21.15.114:0/3642362440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8ae7a0 msgr2=0x5637ff8c0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb879d64b80 1 --2- 172.21.15.114:0/3642362440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c0480 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb8540dfee0 tx=0x7fb8540a5f80 comp rx=0 tx=0).stop 2024-05-27T07:46:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb879d64b80 1 -- 172.21.15.114:0/3642362440 shutdown_connections 2024-05-27T07:46:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb879d64b80 1 --2- 172.21.15.114:0/3642362440 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5637ff8d0330 0x5637ff8d2750 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb879d64b80 1 --2- 172.21.15.114:0/3642362440 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8b1bf0 0x5637ff8c09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb84cff9700 0 client.26641 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:38.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb879d64b80 1 --2- 172.21.15.114:0/3642362440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8bff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb879d64b80 1 --2- 172.21.15.114:0/3642362440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c0480 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb879d64b80 1 -- 172.21.15.114:0/3642362440 >> 172.21.15.114:0/3642362440 conn(0x5637ff7317f0 msgr2=0x5637ff8aac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:38.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb879d64b80 1 -- 172.21.15.114:0/3642362440 shutdown_connections 2024-05-27T07:46:38.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb879d64b80 1 -- 172.21.15.114:0/3642362440 wait complete. 2024-05-27T07:46:38.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.441+0000 7fb879d64b80 20 client.26641 trim_cache size 0 max 16384 2024-05-27T07:46:38.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.463+0000 7fb879d64b80 1 Processor -- start 2024-05-27T07:46:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.463+0000 7fb879d64b80 1 -- start start 2024-05-27T07:46:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb879d64b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8c0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb879d64b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8bb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb879d64b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8b1bf0 0x5637ff8adc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb879d64b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb8641af310 con 0x5637ff72eec0 2024-05-27T07:46:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb879d64b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb8641ac3c0 con 0x5637ff8ae7a0 2024-05-27T07:46:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb879d64b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8641ae570 con 0x5637ff8b1bf0 2024-05-27T07:46:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb8690eb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8c0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb8690eb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8c0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54162/0 (socket says 172.21.15.114:54162) 2024-05-27T07:46:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb8690eb700 1 -- 172.21.15.114:0/4209118462 learned_addr learned my addr 172.21.15.114:0/4209118462 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb8688ea700 1 --2- 172.21.15.114:0/4209118462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8bb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb8698ec700 1 --2- 172.21.15.114:0/4209118462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8b1bf0 0x5637ff8adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb8690eb700 1 -- 172.21.15.114:0/4209118462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8b1bf0 msgr2=0x5637ff8adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb8690eb700 1 --2- 172.21.15.114:0/4209118462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8b1bf0 0x5637ff8adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb8690eb700 1 -- 172.21.15.114:0/4209118462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 msgr2=0x5637ff8bb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb8690eb700 1 --2- 172.21.15.114:0/4209118462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8bb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb8690eb700 1 -- 172.21.15.114:0/4209118462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637ff8aa550 con 0x5637ff72eec0 2024-05-27T07:46:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb8688ea700 1 --2- 172.21.15.114:0/4209118462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8bb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.464+0000 7fb8690eb700 1 --2- 172.21.15.114:0/4209118462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8c0800 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb86000f7a0 tx=0x7fb86000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae7d1c5275ab2785 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.465+0000 7fb84effd700 1 -- 172.21.15.114:0/4209118462 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86000f460 con 0x5637ff72eec0 2024-05-27T07:46:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.465+0000 7fb84effd700 1 -- 172.21.15.114:0/4209118462 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8600054b0 con 0x5637ff72eec0 2024-05-27T07:46:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.465+0000 7fb84effd700 1 -- 172.21.15.114:0/4209118462 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86003c3d0 con 0x5637ff72eec0 2024-05-27T07:46:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.465+0000 7fb879d64b80 1 -- 172.21.15.114:0/4209118462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 msgr2=0x5637ff8c0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.465+0000 7fb879d64b80 1 --2- 172.21.15.114:0/4209118462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8c0800 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb86000f7a0 tx=0x7fb86000f730 comp rx=0 tx=0).stop 2024-05-27T07:46:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.466+0000 7fb879d64b80 1 -- 172.21.15.114:0/4209118462 shutdown_connections 2024-05-27T07:46:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.466+0000 7fb879d64b80 1 --2- 172.21.15.114:0/4209118462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff8b1bf0 0x5637ff8adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.466+0000 7fb879d64b80 1 --2- 172.21.15.114:0/4209118462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff72eec0 0x5637ff8c0800 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.466+0000 7fb879d64b80 1 --2- 172.21.15.114:0/4209118462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8bb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.466+0000 7fb879d64b80 1 -- 172.21.15.114:0/4209118462 >> 172.21.15.114:0/4209118462 conn(0x5637ff7317f0 msgr2=0x5637ff8c1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.466+0000 7fb879d64b80 1 -- 172.21.15.114:0/4209118462 shutdown_connections 2024-05-27T07:46:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.466+0000 7fb879d64b80 1 -- 172.21.15.114:0/4209118462 wait complete. 2024-05-27T07:46:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.466+0000 7fb879d64b80 1 -- start start 2024-05-27T07:46:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.466+0000 7fb879d64b80 1 Processor -- start 2024-05-27T07:46:38.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb879d64b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff72eec0 0x5637ff8c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb879d64b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb879d64b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff8b1bf0 0x5637ff939d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb879d64b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb8641adea0 con 0x5637ff8b1bf0 2024-05-27T07:46:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb879d64b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb8641aca30 con 0x5637ff8ae7a0 2024-05-27T07:46:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb879d64b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8641ad100 con 0x5637ff72eec0 2024-05-27T07:46:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb879d64b80 10 client.0 _ll_register_callbacks cb 0x5637ff762630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb8698ec700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff8b1bf0 0x5637ff939d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb8690eb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff72eec0 0x5637ff8c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb8698ec700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff8b1bf0 0x5637ff939d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54176/0 (socket says 172.21.15.114:54176) 2024-05-27T07:46:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb8698ec700 1 -- 172.21.15.114:0/1174432120 learned_addr learned my addr 172.21.15.114:0/1174432120 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb8688ea700 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb8688ea700 1 -- 172.21.15.114:0/1174432120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff72eec0 msgr2=0x5637ff8c4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb8688ea700 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff72eec0 0x5637ff8c4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb8688ea700 1 -- 172.21.15.114:0/1174432120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff8b1bf0 msgr2=0x5637ff939d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb8688ea700 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff8b1bf0 0x5637ff939d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb8688ea700 1 -- 172.21.15.114:0/1174432120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637ff93a2b0 con 0x5637ff8ae7a0 2024-05-27T07:46:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb8698ec700 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff8b1bf0 0x5637ff939d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb8690eb700 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff72eec0 0x5637ff8c4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb8688ea700 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c4fa0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb85400c5b0 tx=0x7fb854009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba2fc3cc4302c93 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.467+0000 7fb84cff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:46:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb84cff9700 1 -- 172.21.15.114:0/1174432120 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85406d070 con 0x5637ff8ae7a0 2024-05-27T07:46:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb84cff9700 1 -- 172.21.15.114:0/1174432120 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fb854010930 con 0x5637ff8ae7a0 2024-05-27T07:46:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb879d64b80 1 -- 172.21.15.114:0/1174432120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8600054b0 con 0x5637ff8ae7a0 2024-05-27T07:46:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb84cff9700 1 -- 172.21.15.114:0/1174432120 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fb8540c40d0 con 0x5637ff8ae7a0 2024-05-27T07:46:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb879d64b80 10 client.26552 fetch_fsmap learned FSMap version 41 2024-05-27T07:46:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb879d64b80 1 -- 172.21.15.114:0/1174432120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5637ff93b480 con 0x5637ff8ae7a0 2024-05-27T07:46:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb84cff9700 1 -- 172.21.15.114:0/1174432120 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8600054b0 con 0x5637ff8ae7a0 2024-05-27T07:46:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb84cff9700 1 -- 172.21.15.114:0/1174432120 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 41) ==== 7725+0+0 (secure 0 0 0) 0x7fb8540a8d60 con 0x5637ff8ae7a0 2024-05-27T07:46:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb879d64b80 10 client.26552 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb879d64b80 10 client.26552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb879d64b80 20 client.26552 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb879d64b80 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5637ff93b9f0 0x5637ff93de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb879d64b80 4 client.26552 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.468+0000 7fb879d64b80 1 -- 172.21.15.114:0/1174432120 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5637ff7d7520 con 0x5637ff93b9f0 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.469+0000 7fb8690eb700 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5637ff93b9f0 0x5637ff93de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.469+0000 7fb84cff9700 10 client.26552 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.469+0000 7fb8690eb700 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5637ff93b9f0 0x5637ff93de10 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=621b82df6fed9e56 server_cookie=ba13490827f54204 in_seq=0 out_seq=0 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb84cff9700 1 -- 172.21.15.114:0/1174432120 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5637ff7d7520 con 0x5637ff93b9f0 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb84cff9700 10 client.26552 handle_command_reply: tid=0 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb879d64b80 1 client.26552 shutdown 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb879d64b80 1 -- 172.21.15.114:0/1174432120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 msgr2=0x5637ff8c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb879d64b80 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c4fa0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb85400c5b0 tx=0x7fb854009790 comp rx=0 tx=0).stop 2024-05-27T07:46:38.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb879d64b80 1 -- 172.21.15.114:0/1174432120 shutdown_connections 2024-05-27T07:46:38.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb879d64b80 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5637ff8b1bf0 0x5637ff939d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb879d64b80 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5637ff72eec0 0x5637ff8c4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb879d64b80 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5637ff93b9f0 0x5637ff93de10 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb879d64b80 1 --2- 172.21.15.114:0/1174432120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5637ff8ae7a0 0x5637ff8c4fa0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb879d64b80 1 -- 172.21.15.114:0/1174432120 >> 172.21.15.114:0/1174432120 conn(0x5637ff7317f0 msgr2=0x5637ff8c1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:38.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb84cff9700 0 client.26552 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:38.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb879d64b80 1 -- 172.21.15.114:0/1174432120 shutdown_connections 2024-05-27T07:46:38.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb879d64b80 1 -- 172.21.15.114:0/1174432120 wait complete. 2024-05-27T07:46:38.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.470+0000 7fb879d64b80 20 client.26552 trim_cache size 0 max 16384 2024-05-27T07:46:38.474 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:46:38.475 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:46:38.475 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "d2f63c0b-2d65-4bd8-98fb-b8a56ea1242f", 2024-05-27T07:46:38.475 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:46:38.475 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:46:38.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.471+0000 7fb86b34f700 1 -- 172.21.15.114:0/3061147477 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8500b29f0 msgr2=0x7fb8500b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:38.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.471+0000 7fb86b34f700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8500b29f0 0x7fb8500b4f00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb860009990 tx=0x7fb86001a040 comp rx=0 tx=0).stop 2024-05-27T07:46:38.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.471+0000 7fb86b34f700 1 -- 172.21.15.114:0/3061147477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb864108f10 msgr2=0x7fb86419b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.471+0000 7fb86b34f700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb864108f10 0x7fb86419b3e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb85400e9c0 tx=0x7fb85400ecd0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.471+0000 7fb86b34f700 1 -- 172.21.15.114:0/3061147477 shutdown_connections 2024-05-27T07:46:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.471+0000 7fb86b34f700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb864109860 0x7fb86419f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.471+0000 7fb86b34f700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8641045e0 0x7fb86419ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.471+0000 7fb86b34f700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8500b29f0 0x7fb8500b4f00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.471+0000 7fb86b34f700 1 --2- 172.21.15.114:0/3061147477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb864108f10 0x7fb86419b3e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.471+0000 7fb86b34f700 1 -- 172.21.15.114:0/3061147477 >> 172.21.15.114:0/3061147477 conn(0x7fb8640fffd0 msgr2=0x7fb864107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.471+0000 7fb86b34f700 1 -- 172.21.15.114:0/3061147477 shutdown_connections 2024-05-27T07:46:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:38.471+0000 7fb86b34f700 1 -- 172.21.15.114:0/3061147477 wait complete. 2024-05-27T07:46:38.767 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:38 smithi186 podman[73214]: 2024-05-27 07:46:38.507404128 +0000 UTC m=+0.639575621 container remove 2ca6a9d92e05f765e85641c92951e9c442aeea241d16e66adb1974847ed86404 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:46:38.767 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:38 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:46:39.228 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:38 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:46:39.228 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:38 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.317s CPU time. 2024-05-27T07:46:39.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 689927 12.08 MB/sec execute 171 sec latency 15.176 ms 2024-05-27T07:46:39.380 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:46:39.651 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:46:39.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:39 smithi163 ceph-mon[36451]: pgmap v221: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 215 op/s 2024-05-27T07:46:39.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:39 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:46:39.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:39 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:39.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:39 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:46:39.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:39 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:39.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:39 smithi186 ceph-mon[36647]: pgmap v221: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 215 op/s 2024-05-27T07:46:39.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:39 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:46:39.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:39 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:39.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:39 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:46:39.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:39 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:39.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:39 smithi114 ceph-mon[32038]: pgmap v221: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 215 op/s 2024-05-27T07:46:39.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:39 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:46:39.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:39 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:39.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:39 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:46:39.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:39 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:46:40.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 692632 12.11 MB/sec execute 172 sec latency 18.980 ms 2024-05-27T07:46:41.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 693979 12.08 MB/sec execute 173 sec latency 79.777 ms 2024-05-27T07:46:41.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:41 smithi114 ceph-mon[32038]: pgmap v222: 129 pgs: 129 active+clean; 73 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 264 op/s 2024-05-27T07:46:41.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.646+0000 7fc7997b5700 1 -- 172.21.15.114:0/2445979729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7941034b0 msgr2=0x7fc7941038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:41.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.646+0000 7fc7997b5700 1 --2- 172.21.15.114:0/2445979729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7941034b0 0x7fc7941038c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc7840099c0 tx=0x7fc784009cd0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.646+0000 7fc7997b5700 1 -- 172.21.15.114:0/2445979729 shutdown_connections 2024-05-27T07:46:41.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.646+0000 7fc7997b5700 1 --2- 172.21.15.114:0/2445979729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc794108730 0x7fc794110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.646+0000 7fc7997b5700 1 --2- 172.21.15.114:0/2445979729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7941034b0 0x7fc7941038c0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.646+0000 7fc7997b5700 1 --2- 172.21.15.114:0/2445979729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc794107de0 0x7fc7941081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.646+0000 7fc7997b5700 1 -- 172.21.15.114:0/2445979729 >> 172.21.15.114:0/2445979729 conn(0x7fc7940fedb0 msgr2=0x7fc7941011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:41.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.646+0000 7fc7997b5700 1 -- 172.21.15.114:0/2445979729 shutdown_connections 2024-05-27T07:46:41.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.646+0000 7fc7997b5700 1 -- 172.21.15.114:0/2445979729 wait complete. 2024-05-27T07:46:41.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.647+0000 7fc7997b5700 1 Processor -- start 2024-05-27T07:46:41.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.647+0000 7fc7997b5700 1 -- start start 2024-05-27T07:46:41.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.647+0000 7fc7997b5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7941034b0 0x7fc79419f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.647+0000 7fc7997b5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc794107de0 0x7fc79419fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.647+0000 7fc7997b5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc794108730 0x7fc794199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.647+0000 7fc7997b5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc794113360 con 0x7fc7941034b0 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.647+0000 7fc7997b5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc7941131e0 con 0x7fc794107de0 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.647+0000 7fc7997b5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7941134e0 con 0x7fc794108730 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.647+0000 7fc792ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7941034b0 0x7fc79419f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.647+0000 7fc792ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7941034b0 0x7fc79419f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54194/0 (socket says 172.21.15.114:54194) 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.647+0000 7fc792ffd700 1 -- 172.21.15.114:0/550025693 learned_addr learned my addr 172.21.15.114:0/550025693 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.647+0000 7fc7937fe700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc794108730 0x7fc794199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.647+0000 7fc78bfff700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc794107de0 0x7fc79419fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.648+0000 7fc792ffd700 1 -- 172.21.15.114:0/550025693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc794108730 msgr2=0x7fc794199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.648+0000 7fc792ffd700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc794108730 0x7fc794199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.648+0000 7fc792ffd700 1 -- 172.21.15.114:0/550025693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc794107de0 msgr2=0x7fc79419fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:41.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.648+0000 7fc792ffd700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc794107de0 0x7fc79419fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.648+0000 7fc792ffd700 1 -- 172.21.15.114:0/550025693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79419a270 con 0x7fc7941034b0 2024-05-27T07:46:41.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.648+0000 7fc7937fe700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc794108730 0x7fc794199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:41.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.648+0000 7fc792ffd700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7941034b0 0x7fc79419f9b0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc784006010 tx=0x7fc784004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:41.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.648+0000 7fc790ff9700 1 -- 172.21.15.114:0/550025693 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc784026070 con 0x7fc7941034b0 2024-05-27T07:46:41.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.648+0000 7fc790ff9700 1 -- 172.21.15.114:0/550025693 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc784003c70 con 0x7fc7941034b0 2024-05-27T07:46:41.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.648+0000 7fc7997b5700 1 -- 172.21.15.114:0/550025693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc79419a500 con 0x7fc7941034b0 2024-05-27T07:46:41.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.648+0000 7fc790ff9700 1 -- 172.21.15.114:0/550025693 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78400fbe0 con 0x7fc7941034b0 2024-05-27T07:46:41.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.648+0000 7fc7997b5700 1 -- 172.21.15.114:0/550025693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7941a6760 con 0x7fc7941034b0 2024-05-27T07:46:41.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.651+0000 7fc790ff9700 1 -- 172.21.15.114:0/550025693 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fc784029070 con 0x7fc7941034b0 2024-05-27T07:46:41.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.652+0000 7fc790ff9700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7800b29f0 0x7fc7800b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.652+0000 7fc78bfff700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7800b29f0 0x7fc7800b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.652+0000 7fc790ff9700 1 -- 172.21.15.114:0/550025693 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fc7840e3260 con 0x7fc7941034b0 2024-05-27T07:46:41.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.652+0000 7fc78bfff700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7800b29f0 0x7fc7800b4f00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc79419b270 tx=0x7fc77c009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:41.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.653+0000 7fc7a81cab80 1 Processor -- start 2024-05-27T07:46:41.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.653+0000 7fc7a81cab80 1 -- start start 2024-05-27T07:46:41.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.653+0000 7fc7a81cab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 0x55edd948e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.653+0000 7fc7a81cab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd948ec50 0x55edd9488a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.653+0000 7fc7a81cab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd9488f80 0x55edd9489390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.653+0000 7fc7a81cab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55edd93b7c90 con 0x55edd930ae40 2024-05-27T07:46:41.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.653+0000 7fc7a81cab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55edd93a7830 con 0x55edd9488f80 2024-05-27T07:46:41.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.653+0000 7fc7a81cab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55edd93ab870 con 0x55edd948ec50 2024-05-27T07:46:41.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.653+0000 7fc792ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 0x55edd948e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.653+0000 7fc792ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 0x55edd948e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54198/0 (socket says 172.21.15.114:54198) 2024-05-27T07:46:41.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.653+0000 7fc792ffd700 1 -- 172.21.15.114:0/2553037301 learned_addr learned my addr 172.21.15.114:0/2553037301 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:41.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.653+0000 7fc78bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd948ec50 0x55edd9488a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.653+0000 7fc7937fe700 1 --2- 172.21.15.114:0/2553037301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd9488f80 0x55edd9489390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.654+0000 7fc78bfff700 1 -- 172.21.15.114:0/2553037301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd9488f80 msgr2=0x55edd9489390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:41.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.654+0000 7fc78bfff700 1 --2- 172.21.15.114:0/2553037301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd9488f80 0x55edd9489390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.654+0000 7fc78bfff700 1 -- 172.21.15.114:0/2553037301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 msgr2=0x55edd948e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:41.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.654+0000 7fc78bfff700 1 --2- 172.21.15.114:0/2553037301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 0x55edd948e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.654+0000 7fc78bfff700 1 -- 172.21.15.114:0/2553037301 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edd92f78d0 con 0x55edd948ec50 2024-05-27T07:46:41.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.654+0000 7fc792ffd700 1 --2- 172.21.15.114:0/2553037301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 0x55edd948e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:41.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.654+0000 7fc78bfff700 1 --2- 172.21.15.114:0/2553037301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd948ec50 0x55edd9488a40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc7941024a0 tx=0x7fc77c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9302ffaf07ebed3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:41.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.654+0000 7fc7897fa700 1 -- 172.21.15.114:0/2553037301 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c012ab0 con 0x55edd948ec50 2024-05-27T07:46:41.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.654+0000 7fc7897fa700 1 -- 172.21.15.114:0/2553037301 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc77c012c50 con 0x55edd948ec50 2024-05-27T07:46:41.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.654+0000 7fc7897fa700 1 -- 172.21.15.114:0/2553037301 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c01d5b0 con 0x55edd948ec50 2024-05-27T07:46:41.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.655+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2553037301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd948ec50 msgr2=0x55edd9488a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:41.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.655+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2553037301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd948ec50 0x55edd9488a40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc7941024a0 tx=0x7fc77c014000 comp rx=0 tx=0).stop 2024-05-27T07:46:41.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.655+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2553037301 shutdown_connections 2024-05-27T07:46:41.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.655+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2553037301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd9488f80 0x55edd9489390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.655+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2553037301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 0x55edd948e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.655+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2553037301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd948ec50 0x55edd9488a40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.655+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2553037301 >> 172.21.15.114:0/2553037301 conn(0x55edd9365af0 msgr2=0x55edd9483640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:41.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.655+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2553037301 shutdown_connections 2024-05-27T07:46:41.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.655+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2553037301 wait complete. 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.655+0000 7fc7a81cab80 1 -- start start 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc7a81cab80 1 Processor -- start 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc7a81cab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd930ae40 0x55edd94922e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc7a81cab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd9488f80 0x55edd9492820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc7a81cab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd948ec50 0x55edd9492d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc7a81cab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55edd92adfa0 con 0x55edd9488f80 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc7a81cab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55edd93497c0 con 0x55edd930ae40 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc7a81cab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55edd9497950 con 0x55edd948ec50 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc7a81cab80 10 client.0 _ll_register_callbacks cb 0x55edd9342920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc78bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd9488f80 0x55edd9492820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc78bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd9488f80 0x55edd9492820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54206/0 (socket says 172.21.15.114:54206) 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc78bfff700 1 -- 172.21.15.114:0/2905975840 learned_addr learned my addr 172.21.15.114:0/2905975840 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc7937fe700 1 --2- 172.21.15.114:0/2905975840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd948ec50 0x55edd9492d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc792ffd700 1 --2- 172.21.15.114:0/2905975840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd930ae40 0x55edd94922e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc78bfff700 1 -- 172.21.15.114:0/2905975840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd948ec50 msgr2=0x55edd9492d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc78bfff700 1 --2- 172.21.15.114:0/2905975840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd948ec50 0x55edd9492d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc78bfff700 1 -- 172.21.15.114:0/2905975840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd930ae40 msgr2=0x55edd94922e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc78bfff700 1 --2- 172.21.15.114:0/2905975840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd930ae40 0x55edd94922e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.656+0000 7fc78bfff700 1 -- 172.21.15.114:0/2905975840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edd9346690 con 0x55edd9488f80 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.657+0000 7fc78bfff700 1 --2- 172.21.15.114:0/2905975840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd9488f80 0x55edd9492820 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc77c01f6a0 tx=0x7fc77c01f6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8613abf7d2b1b11d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.657+0000 7fc77a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.657+0000 7fc77a7fc700 1 -- 172.21.15.114:0/2905975840 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c036060 con 0x55edd9488f80 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.657+0000 7fc77a7fc700 1 -- 172.21.15.114:0/2905975840 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fc77c01fb30 con 0x55edd9488f80 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.657+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2905975840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc77c012c50 con 0x55edd9488f80 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.657+0000 7fc77a7fc700 1 -- 172.21.15.114:0/2905975840 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fc77c03d050 con 0x55edd9488f80 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.657+0000 7fc7a81cab80 10 client.17091 fetch_fsmap learned FSMap version 41 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.657+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2905975840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55edd94a2350 con 0x55edd9488f80 2024-05-27T07:46:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.657+0000 7fc77a7fc700 1 -- 172.21.15.114:0/2905975840 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc77c012c50 con 0x55edd9488f80 2024-05-27T07:46:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.657+0000 7fc77a7fc700 1 -- 172.21.15.114:0/2905975840 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 41) ==== 7725+0+0 (secure 0 0 0) 0x7fc77c004c00 con 0x55edd9488f80 2024-05-27T07:46:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.657+0000 7fc7a81cab80 10 client.17091 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:46:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.657+0000 7fc7a81cab80 10 client.17091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.657+0000 7fc7a81cab80 20 client.17091 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.658+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2905975840 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edd94a48d0 0x55edd94a6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.658+0000 7fc7a81cab80 4 client.17091 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:46:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.658+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2905975840 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edd94549f0 con 0x55edd94a48d0 2024-05-27T07:46:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.658+0000 7fc792ffd700 1 --2- 172.21.15.114:0/2905975840 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edd94a48d0 0x55edd94a6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.658+0000 7fc77a7fc700 10 client.17091 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.658+0000 7fc792ffd700 1 --2- 172.21.15.114:0/2905975840 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edd94a48d0 0x55edd94a6cf0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cd77011b4a31eb server_cookie=33ca0d5f5d407487 in_seq=0 out_seq=0 2024-05-27T07:46:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.659+0000 7fc77a7fc700 1 -- 172.21.15.114:0/2905975840 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55edd94549f0 con 0x55edd94a48d0 2024-05-27T07:46:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.659+0000 7fc77a7fc700 10 client.17091 handle_command_reply: tid=0 2024-05-27T07:46:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.660+0000 7fc7a81cab80 1 client.17091 shutdown 2024-05-27T07:46:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.660+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2905975840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd9488f80 msgr2=0x55edd9492820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.660+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2905975840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd9488f80 0x55edd9492820 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc77c01f6a0 tx=0x7fc77c01f6d0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.660+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2905975840 shutdown_connections 2024-05-27T07:46:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.660+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2905975840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd948ec50 0x55edd9492d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.660+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2905975840 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edd94a48d0 0x55edd94a6cf0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.660+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2905975840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd930ae40 0x55edd94922e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.660+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2905975840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd9488f80 0x55edd9492820 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.660+0000 7fc77a7fc700 0 client.17091 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.660+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2905975840 >> 172.21.15.114:0/2905975840 conn(0x55edd9365af0 msgr2=0x55edd9487a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.660+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2905975840 shutdown_connections 2024-05-27T07:46:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.660+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2905975840 wait complete. 2024-05-27T07:46:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.660+0000 7fc7a81cab80 20 client.17091 trim_cache size 0 max 16384 2024-05-27T07:46:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc7a81cab80 1 Processor -- start 2024-05-27T07:46:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc7a81cab80 1 -- start start 2024-05-27T07:46:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc7a81cab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd930ae40 0x55edd9488a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc7a81cab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd9488f80 0x55edd94a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc7a81cab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd948ec50 0x55edd947de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc7a81cab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc7941af270 con 0x55edd948ec50 2024-05-27T07:46:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc7a81cab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc7941ac320 con 0x55edd930ae40 2024-05-27T07:46:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc7a81cab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7941ae4d0 con 0x55edd9488f80 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc7937fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd948ec50 0x55edd947de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc78bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd9488f80 0x55edd94a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc78bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd9488f80 0x55edd94a09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:37452/0 (socket says 172.21.15.114:37452) 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc7937fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd948ec50 0x55edd947de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54222/0 (socket says 172.21.15.114:54222) 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc78bfff700 1 -- 172.21.15.114:0/2878810050 learned_addr learned my addr 172.21.15.114:0/2878810050 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.682+0000 7fc792ffd700 1 --2- 172.21.15.114:0/2878810050 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd930ae40 0x55edd9488a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.683+0000 7fc78bfff700 1 -- 172.21.15.114:0/2878810050 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd930ae40 msgr2=0x55edd9488a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.683+0000 7fc78bfff700 1 --2- 172.21.15.114:0/2878810050 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd930ae40 0x55edd9488a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.683+0000 7fc78bfff700 1 -- 172.21.15.114:0/2878810050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd948ec50 msgr2=0x55edd947de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.683+0000 7fc78bfff700 1 --2- 172.21.15.114:0/2878810050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd948ec50 0x55edd947de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.683+0000 7fc78bfff700 1 -- 172.21.15.114:0/2878810050 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edd9490850 con 0x55edd9488f80 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.683+0000 7fc792ffd700 1 --2- 172.21.15.114:0/2878810050 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd930ae40 0x55edd9488a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.683+0000 7fc7937fe700 1 --2- 172.21.15.114:0/2878810050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd948ec50 0x55edd947de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.683+0000 7fc78bfff700 1 --2- 172.21.15.114:0/2878810050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd9488f80 0x55edd94a09f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc77c005950 tx=0x7fc77c0058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=190f95cd3829d143 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:41.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.683+0000 7fc788ff9700 1 -- 172.21.15.114:0/2878810050 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c046070 con 0x55edd9488f80 2024-05-27T07:46:41.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.683+0000 7fc788ff9700 1 -- 172.21.15.114:0/2878810050 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc77c01fc70 con 0x55edd9488f80 2024-05-27T07:46:41.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.683+0000 7fc788ff9700 1 -- 172.21.15.114:0/2878810050 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c036070 con 0x55edd9488f80 2024-05-27T07:46:41.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.684+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2878810050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd9488f80 msgr2=0x55edd94a09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:41.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.684+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2878810050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd9488f80 0x55edd94a09f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc77c005950 tx=0x7fc77c0058e0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.684+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2878810050 shutdown_connections 2024-05-27T07:46:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.684+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2878810050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd948ec50 0x55edd947de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.684+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2878810050 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd930ae40 0x55edd9488a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.684+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2878810050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd9488f80 0x55edd94a09f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.684+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2878810050 >> 172.21.15.114:0/2878810050 conn(0x55edd9365af0 msgr2=0x55edd947d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.684+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2878810050 shutdown_connections 2024-05-27T07:46:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.684+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2878810050 wait complete. 2024-05-27T07:46:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.684+0000 7fc7a81cab80 1 -- start start 2024-05-27T07:46:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc7a81cab80 1 Processor -- start 2024-05-27T07:46:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc7a81cab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 0x55edd94abe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc7a81cab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd9488f80 0x55edd94ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc7a81cab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd948ec50 0x55edd94ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc7a81cab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc7941ade00 con 0x55edd930ae40 2024-05-27T07:46:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc7a81cab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc7941ac990 con 0x55edd948ec50 2024-05-27T07:46:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc7a81cab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7941ad060 con 0x55edd9488f80 2024-05-27T07:46:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc792ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 0x55edd94abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc792ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 0x55edd94abe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54234/0 (socket says 172.21.15.114:54234) 2024-05-27T07:46:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc792ffd700 1 -- 172.21.15.114:0/2365955218 learned_addr learned my addr 172.21.15.114:0/2365955218 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc7a81cab80 10 client.0 _ll_register_callbacks cb 0x55edd9342920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc78bfff700 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd9488f80 0x55edd94ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc7937fe700 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd948ec50 0x55edd94ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc792ffd700 1 -- 172.21.15.114:0/2365955218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd9488f80 msgr2=0x55edd94ac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc792ffd700 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd9488f80 0x55edd94ac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc792ffd700 1 -- 172.21.15.114:0/2365955218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd948ec50 msgr2=0x55edd94ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc792ffd700 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd948ec50 0x55edd94ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc792ffd700 1 -- 172.21.15.114:0/2365955218 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edd950e160 con 0x55edd930ae40 2024-05-27T07:46:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.685+0000 7fc7937fe700 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd948ec50 0x55edd94ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc792ffd700 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 0x55edd94abe50 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc7840a7430 tx=0x7fc784003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d628f74ade71916 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc77a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc77a7fc700 1 -- 172.21.15.114:0/2365955218 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7840ae570 con 0x55edd930ae40 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc77a7fc700 1 -- 172.21.15.114:0/2365955218 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fc7840ae120 con 0x55edd930ae40 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2365955218 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc77c01fc70 con 0x55edd930ae40 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc77a7fc700 1 -- 172.21.15.114:0/2365955218 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fc7840b2f90 con 0x55edd930ae40 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc7a81cab80 10 client.17097 fetch_fsmap learned FSMap version 41 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2365955218 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55edd950f330 con 0x55edd930ae40 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc77a7fc700 1 -- 172.21.15.114:0/2365955218 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc77c01fc70 con 0x55edd930ae40 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc77a7fc700 1 -- 172.21.15.114:0/2365955218 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 41) ==== 7725+0+0 (secure 0 0 0) 0x7fc7840aa490 con 0x55edd930ae40 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc7a81cab80 10 client.17097 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc7a81cab80 10 client.17097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc7a81cab80 20 client.17097 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edd950fa10 0x55edd9511e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc7a81cab80 4 client.17097 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.686+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2365955218 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55edd93b9be0 con 0x55edd950fa10 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.687+0000 7fc78bfff700 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edd950fa10 0x55edd9511e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.687+0000 7fc77a7fc700 10 client.17097 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.687+0000 7fc78bfff700 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edd950fa10 0x55edd9511e30 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=996be1068d57c84b server_cookie=e5f87a30a05ee83 in_seq=0 out_seq=0 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc77a7fc700 1 -- 172.21.15.114:0/2365955218 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55edd93b9be0 con 0x55edd950fa10 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc77a7fc700 10 client.17097 handle_command_reply: tid=0 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc7a81cab80 1 client.17097 shutdown 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2365955218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 msgr2=0x55edd94abe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 0x55edd94abe50 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc7840a7430 tx=0x7fc784003e20 comp rx=0 tx=0).stop 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2365955218 shutdown_connections 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edd950fa10 0x55edd9511e30 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edd948ec50 0x55edd94ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edd930ae40 0x55edd94abe50 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc77a7fc700 0 client.17097 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:41.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc7a81cab80 1 --2- 172.21.15.114:0/2365955218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edd9488f80 0x55edd94ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2365955218 >> 172.21.15.114:0/2365955218 conn(0x55edd9365af0 msgr2=0x55edd948dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:41.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2365955218 shutdown_connections 2024-05-27T07:46:41.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc7a81cab80 1 -- 172.21.15.114:0/2365955218 wait complete. 2024-05-27T07:46:41.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.688+0000 7fc7a81cab80 20 client.17097 trim_cache size 0 max 16384 2024-05-27T07:46:41.693 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:46:41.693 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:46:41.693 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:46:41.693 INFO:teuthology.orchestra.run.smithi114.stdout: "d2f63c0b-2d65-4bd8-98fb-b8a56ea1242f": { 2024-05-27T07:46:41.694 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:46:41.694 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "d2f63c0b-2d65-4bd8-98fb-b8a56ea1242f", 2024-05-27T07:46:41.694 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:46:41.694 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:46:41.694 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:46:41.694 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:46:41.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.689+0000 7fc7997b5700 1 -- 172.21.15.114:0/550025693 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7800b29f0 msgr2=0x7fc7800b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:41.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.689+0000 7fc7997b5700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7800b29f0 0x7fc7800b4f00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc79419b270 tx=0x7fc77c009260 comp rx=0 tx=0).stop 2024-05-27T07:46:41.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.689+0000 7fc7997b5700 1 -- 172.21.15.114:0/550025693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7941034b0 msgr2=0x7fc79419f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:41.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.689+0000 7fc7997b5700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7941034b0 0x7fc79419f9b0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc784006010 tx=0x7fc784004270 comp rx=0 tx=0).stop 2024-05-27T07:46:41.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.689+0000 7fc7997b5700 1 -- 172.21.15.114:0/550025693 shutdown_connections 2024-05-27T07:46:41.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.689+0000 7fc7997b5700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc794108730 0x7fc794199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.689+0000 7fc7997b5700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7941034b0 0x7fc79419f9b0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.689+0000 7fc7997b5700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7800b29f0 0x7fc7800b4f00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.689+0000 7fc7997b5700 1 --2- 172.21.15.114:0/550025693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc794107de0 0x7fc79419fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:41.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.689+0000 7fc7997b5700 1 -- 172.21.15.114:0/550025693 >> 172.21.15.114:0/550025693 conn(0x7fc7940fedb0 msgr2=0x7fc794109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:41.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.689+0000 7fc7997b5700 1 -- 172.21.15.114:0/550025693 shutdown_connections 2024-05-27T07:46:41.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:46:41.689+0000 7fc7997b5700 1 -- 172.21.15.114:0/550025693 wait complete. 2024-05-27T07:46:41.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:41 smithi163 ceph-mon[36451]: pgmap v222: 129 pgs: 129 active+clean; 73 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 264 op/s 2024-05-27T07:46:41.727 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:41 smithi186 ceph-mon[36647]: pgmap v222: 129 pgs: 129 active+clean; 73 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 264 op/s 2024-05-27T07:46:42.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 696273 12.09 MB/sec execute 174 sec latency 68.148 ms 2024-05-27T07:46:42.690 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:42 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:46:42.719 INFO:tasks.cephfs.filesystem:scrub status for tag:d2f63c0b-2d65-4bd8-98fb-b8a56ea1242f - {'path': '/', 'tag': 'd2f63c0b-2d65-4bd8-98fb-b8a56ea1242f', 'options': 'recursive,force'} 2024-05-27T07:46:42.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:42 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:46:42.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:42 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:46:43.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 698184 12.05 MB/sec execute 175 sec latency 14.035 ms 2024-05-27T07:46:43.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:43 smithi163 ceph-mon[36451]: pgmap v223: 129 pgs: 129 active+clean; 65 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 230 op/s 2024-05-27T07:46:43.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:43 smithi186 ceph-mon[36647]: pgmap v223: 129 pgs: 129 active+clean; 65 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 230 op/s 2024-05-27T07:46:43.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:43 smithi114 ceph-mon[32038]: pgmap v223: 129 pgs: 129 active+clean; 65 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 230 op/s 2024-05-27T07:46:44.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 700392 12.06 MB/sec execute 176 sec latency 15.314 ms 2024-05-27T07:46:45.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 703190 12.09 MB/sec execute 177 sec latency 19.749 ms 2024-05-27T07:46:45.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:45 smithi163 ceph-mon[36451]: pgmap v224: 129 pgs: 129 active+clean; 65 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 216 op/s 2024-05-27T07:46:45.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:45 smithi186 ceph-mon[36647]: pgmap v224: 129 pgs: 129 active+clean; 65 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 216 op/s 2024-05-27T07:46:45.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:45 smithi114 ceph-mon[32038]: pgmap v224: 129 pgs: 129 active+clean; 65 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 216 op/s 2024-05-27T07:46:46.263 INFO:tasks.workunit.client.0.smithi114.stdout: 1 704410 12.04 MB/sec execute 178 sec latency 34.658 ms 2024-05-27T07:46:47.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 706834 12.06 MB/sec execute 179 sec latency 65.033 ms 2024-05-27T07:46:47.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:47 smithi163 ceph-mon[36451]: pgmap v225: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 319 op/s 2024-05-27T07:46:47.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:47 smithi186 ceph-mon[36647]: pgmap v225: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 319 op/s 2024-05-27T07:46:47.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:47 smithi114 ceph-mon[32038]: pgmap v225: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 319 op/s 2024-05-27T07:46:48.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 708449 12.03 MB/sec execute 180 sec latency 17.286 ms 2024-05-27T07:46:49.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:48 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:46:49.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:48 smithi186 systemd[1]: Stopped Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:46:49.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:48 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.317s CPU time. 2024-05-27T07:46:49.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:48 smithi186 systemd[1]: Starting Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:46:49.240 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:49 smithi186 podman[73301]: 2024-05-27 07:46:49.125962448 +0000 UTC m=+0.020807411 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:46:49.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 710784 12.04 MB/sec execute 181 sec latency 18.498 ms 2024-05-27T07:46:49.525 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:49 smithi186 ceph-mon[36647]: pgmap v226: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 2.3 MiB/s wr, 254 op/s 2024-05-27T07:46:49.525 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:49 smithi186 podman[73301]: 2024-05-27 07:46:49.241589624 +0000 UTC m=+0.136434562 container create 779662ab5886efb3866c2b8ffe549a590cbb072cb38a1cf617e4dc5e802dc19e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-05-27T07:46:49.525 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:49 smithi186 podman[73301]: 2024-05-27 07:46:49.486550501 +0000 UTC m=+0.381395438 container init 779662ab5886efb3866c2b8ffe549a590cbb072cb38a1cf617e4dc5e802dc19e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:46:49.525 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:49 smithi186 podman[73301]: 2024-05-27 07:46:49.48951944 +0000 UTC m=+0.384364383 container start 779662ab5886efb3866c2b8ffe549a590cbb072cb38a1cf617e4dc5e802dc19e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-27T07:46:49.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:49 smithi163 ceph-mon[36451]: pgmap v226: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 2.3 MiB/s wr, 254 op/s 2024-05-27T07:46:49.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:49 smithi114 ceph-mon[32038]: pgmap v226: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 2.3 MiB/s wr, 254 op/s 2024-05-27T07:46:49.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:49 smithi186 ceph-mds[73320]: -- 172.21.15.186:0/3665833369 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556b6e3bfa40 con 0x556b6e407a80 2024-05-27T07:46:49.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:49 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: starting mds.f at 2024-05-27T07:46:49.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:49 smithi186 bash[73301]: 779662ab5886efb3866c2b8ffe549a590cbb072cb38a1cf617e4dc5e802dc19e 2024-05-27T07:46:49.978 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:46:49 smithi186 systemd[1]: Started Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:46:50.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 712831 12.02 MB/sec execute 182 sec latency 46.288 ms 2024-05-27T07:46:50.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.443+0000 7fe74aa3b640 1 -- 172.21.15.114:0/760084994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7441021c0 msgr2=0x7fe7441025d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:50.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.443+0000 7fe74aa3b640 1 --2- 172.21.15.114:0/760084994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7441021c0 0x7fe7441025d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe738009860 tx=0x7fe73802a3d0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.443+0000 7fe74aa3b640 1 -- 172.21.15.114:0/760084994 shutdown_connections 2024-05-27T07:46:50.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.443+0000 7fe74aa3b640 1 --2- 172.21.15.114:0/760084994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe744110400 0x7fe7441128e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.443+0000 7fe74aa3b640 1 --2- 172.21.15.114:0/760084994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7441021c0 0x7fe7441025d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.443+0000 7fe74aa3b640 1 --2- 172.21.15.114:0/760084994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe744104b30 0x7fe744101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.444+0000 7fe74aa3b640 1 -- 172.21.15.114:0/760084994 >> 172.21.15.114:0/760084994 conn(0x7fe7440779d0 msgr2=0x7fe744077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:50.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.444+0000 7fe74aa3b640 1 -- 172.21.15.114:0/760084994 shutdown_connections 2024-05-27T07:46:50.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.444+0000 7fe74aa3b640 1 -- 172.21.15.114:0/760084994 wait complete. 2024-05-27T07:46:50.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.444+0000 7fe74aa3b640 1 Processor -- start 2024-05-27T07:46:50.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.444+0000 7fe74aa3b640 1 -- start start 2024-05-27T07:46:50.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe74aa3b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7441021c0 0x7fe7441a2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe74aa3b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe744104b30 0x7fe7441a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe74aa3b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe744110400 0x7fe74419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe74aa3b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe744116060 con 0x7fe744110400 2024-05-27T07:46:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe74aa3b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe744115ee0 con 0x7fe744104b30 2024-05-27T07:46:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe74aa3b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441161e0 con 0x7fe7441021c0 2024-05-27T07:46:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe748fd8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe744110400 0x7fe74419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe7437fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe744104b30 0x7fe7441a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe748fd8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe744110400 0x7fe74419c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60124/0 (socket says 172.21.15.114:60124) 2024-05-27T07:46:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe748fd8640 1 -- 172.21.15.114:0/3264527188 learned_addr learned my addr 172.21.15.114:0/3264527188 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe743fff640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7441021c0 0x7fe7441a2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe7437fe640 1 -- 172.21.15.114:0/3264527188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7441021c0 msgr2=0x7fe7441a2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe7437fe640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7441021c0 0x7fe7441a2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe7437fe640 1 -- 172.21.15.114:0/3264527188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe744110400 msgr2=0x7fe74419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe7437fe640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe744110400 0x7fe74419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.445+0000 7fe7437fe640 1 -- 172.21.15.114:0/3264527188 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74419cf00 con 0x7fe744104b30 2024-05-27T07:46:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.446+0000 7fe748fd8640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe744110400 0x7fe74419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.446+0000 7fe7437fe640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe744104b30 0x7fe7441a2b90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe73802a8f0 tx=0x7fe738030fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.446+0000 7fe7417fa640 1 -- 172.21.15.114:0/3264527188 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738040070 con 0x7fe744104b30 2024-05-27T07:46:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.446+0000 7fe74aa3b640 1 -- 172.21.15.114:0/3264527188 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74419d190 con 0x7fe744104b30 2024-05-27T07:46:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.446+0000 7fe74aa3b640 1 -- 172.21.15.114:0/3264527188 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7441a94d0 con 0x7fe744104b30 2024-05-27T07:46:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.446+0000 7fe7417fa640 1 -- 172.21.15.114:0/3264527188 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe738005570 con 0x7fe744104b30 2024-05-27T07:46:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.446+0000 7fe7417fa640 1 -- 172.21.15.114:0/3264527188 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73804a060 con 0x7fe744104b30 2024-05-27T07:46:50.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.449+0000 7fe7417fa640 1 -- 172.21.15.114:0/3264527188 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fe738043070 con 0x7fe744104b30 2024-05-27T07:46:50.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.449+0000 7fe7519c9740 1 Processor -- start 2024-05-27T07:46:50.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.449+0000 7fe7519c9740 1 -- start start 2024-05-27T07:46:50.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.450+0000 7fe7519c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44c91b0 0x562cb486a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.450+0000 7fe7519c9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44a02f0 0x562cb486a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.450+0000 7fe7519c9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a84d0 0x562cb486ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.450+0000 7fe7519c9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562cb47dcc30 con 0x562cb44a02f0 2024-05-27T07:46:50.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.450+0000 7fe7519c9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562cb47cc7d0 con 0x562cb44a84d0 2024-05-27T07:46:50.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.450+0000 7fe7519c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562cb47d0810 con 0x562cb44c91b0 2024-05-27T07:46:50.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.450+0000 7fe7417fa640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe72c0b29d0 0x7fe72c0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.450+0000 7fe748fd8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44c91b0 0x562cb486a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.450+0000 7fe748fd8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44c91b0 0x562cb486a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:56132/0 (socket says 172.21.15.114:56132) 2024-05-27T07:46:50.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.450+0000 7fe748fd8640 1 -- 172.21.15.114:0/389797481 learned_addr learned my addr 172.21.15.114:0/389797481 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:50.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.450+0000 7fe743fff640 1 --2- 172.21.15.114:0/389797481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44a02f0 0x562cb486a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.450+0000 7fe7437fe640 1 --2- 172.21.15.114:0/389797481 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a84d0 0x562cb486ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.450+0000 7fe7417fa640 1 -- 172.21.15.114:0/3264527188 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9466+0+0 (secure 0 0 0) 0x7fe738103370 con 0x7fe744104b30 2024-05-27T07:46:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.451+0000 7fe748fd8640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe72c0b29d0 0x7fe72c0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.451+0000 7fe7437fe640 1 -- 172.21.15.114:0/389797481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44c91b0 msgr2=0x562cb486a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.451+0000 7fe7437fe640 1 --2- 172.21.15.114:0/389797481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44c91b0 0x562cb486a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.451+0000 7fe7437fe640 1 -- 172.21.15.114:0/389797481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44a02f0 msgr2=0x562cb486a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.451+0000 7fe7437fe640 1 --2- 172.21.15.114:0/389797481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44a02f0 0x562cb486a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.451+0000 7fe7437fe640 1 -- 172.21.15.114:0/389797481 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cb44e1210 con 0x562cb44a84d0 2024-05-27T07:46:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.451+0000 7fe7437fe640 1 --2- 172.21.15.114:0/389797481 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a84d0 0x562cb486ae50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe738100710 tx=0x7fe73808fce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55e68a434d65e0b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.451+0000 7fe748fd8640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe72c0b29d0 0x7fe72c0b4ee0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x562cb4867540 tx=0x7fe73c004ab0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.451+0000 7fe72a7fc640 1 -- 172.21.15.114:0/389797481 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738100e60 con 0x562cb44a84d0 2024-05-27T07:46:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.451+0000 7fe72a7fc640 1 -- 172.21.15.114:0/389797481 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7380ca9a0 con 0x562cb44a84d0 2024-05-27T07:46:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.451+0000 7fe72a7fc640 1 -- 172.21.15.114:0/389797481 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738031400 con 0x562cb44a84d0 2024-05-27T07:46:50.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.453+0000 7fe7519c9740 1 -- 172.21.15.114:0/389797481 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a84d0 msgr2=0x562cb486ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:50.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.453+0000 7fe7519c9740 1 --2- 172.21.15.114:0/389797481 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a84d0 0x562cb486ae50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe738100710 tx=0x7fe73808fce0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.453+0000 7fe7519c9740 1 -- 172.21.15.114:0/389797481 shutdown_connections 2024-05-27T07:46:50.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.453+0000 7fe7519c9740 1 --2- 172.21.15.114:0/389797481 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a84d0 0x562cb486ae50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.453+0000 7fe7519c9740 1 --2- 172.21.15.114:0/389797481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44a02f0 0x562cb486a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.453+0000 7fe7519c9740 1 --2- 172.21.15.114:0/389797481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44c91b0 0x562cb486a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.453+0000 7fe7519c9740 1 -- 172.21.15.114:0/389797481 >> 172.21.15.114:0/389797481 conn(0x562cb44bd300 msgr2=0x562cb4866b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.453+0000 7fe7519c9740 1 -- 172.21.15.114:0/389797481 shutdown_connections 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.453+0000 7fe7519c9740 1 -- 172.21.15.114:0/389797481 wait complete. 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.454+0000 7fe7519c9740 1 -- start start 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.454+0000 7fe7519c9740 1 Processor -- start 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.454+0000 7fe7519c9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a02f0 0x562cb4874a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.454+0000 7fe7519c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb4874fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.454+0000 7fe7519c9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44c91b0 0x562cb487a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.454+0000 7fe7519c9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562cb44b2790 con 0x562cb44c91b0 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.454+0000 7fe7519c9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562cb4788890 con 0x562cb44a02f0 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.454+0000 7fe7519c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562cb48772c0 con 0x562cb44a84d0 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.454+0000 7fe7519c9740 10 client.0 _ll_register_callbacks cb 0x562cb44e0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe7437fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb4874fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe743fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a02f0 0x562cb4874a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe7437fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb4874fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:56138/0 (socket says 172.21.15.114:56138) 2024-05-27T07:46:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe7437fe640 1 -- 172.21.15.114:0/1526016271 learned_addr learned my addr 172.21.15.114:0/1526016271 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe743fff640 1 --2- 172.21.15.114:0/1526016271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a02f0 0x562cb4874a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:46836/0 (socket says 172.21.15.114:46836) 2024-05-27T07:46:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe748fd8640 1 --2- 172.21.15.114:0/1526016271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44c91b0 0x562cb487a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe743fff640 1 -- 172.21.15.114:0/1526016271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 msgr2=0x562cb4874fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe743fff640 1 --2- 172.21.15.114:0/1526016271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb4874fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe743fff640 1 -- 172.21.15.114:0/1526016271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44c91b0 msgr2=0x562cb487a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe743fff640 1 --2- 172.21.15.114:0/1526016271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44c91b0 0x562cb487a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe743fff640 1 -- 172.21.15.114:0/1526016271 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cb45ad620 con 0x562cb44a02f0 2024-05-27T07:46:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe743fff640 1 --2- 172.21.15.114:0/1526016271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a02f0 0x562cb4874a80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x562cb4867b10 tx=0x7fe734009f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=754ea24425c5743a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe7237fe640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:46:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.455+0000 7fe7237fe640 1 -- 172.21.15.114:0/1526016271 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734012650 con 0x562cb44a02f0 2024-05-27T07:46:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.456+0000 7fe7237fe640 1 -- 172.21.15.114:0/1526016271 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9466+0+0 (secure 0 0 0) 0x7fe7340127f0 con 0x562cb44a02f0 2024-05-27T07:46:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.456+0000 7fe7519c9740 1 -- 172.21.15.114:0/1526016271 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7380ca9a0 con 0x562cb44a02f0 2024-05-27T07:46:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.456+0000 7fe7237fe640 1 -- 172.21.15.114:0/1526016271 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fe734025050 con 0x562cb44a02f0 2024-05-27T07:46:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.456+0000 7fe7519c9740 10 client.26585 fetch_fsmap learned FSMap version 43 2024-05-27T07:46:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.456+0000 7fe7519c9740 1 -- 172.21.15.114:0/1526016271 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562cb48835e0 con 0x562cb44a02f0 2024-05-27T07:46:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.456+0000 7fe7237fe640 1 -- 172.21.15.114:0/1526016271 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7380ca9a0 con 0x562cb44a02f0 2024-05-27T07:46:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.456+0000 7fe7237fe640 1 -- 172.21.15.114:0/1526016271 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 43) ==== 7725+0+0 (secure 0 0 0) 0x7fe73401a020 con 0x562cb44a02f0 2024-05-27T07:46:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.456+0000 7fe7519c9740 10 client.26585 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T07:46:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.456+0000 7fe7519c9740 10 client.26585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.456+0000 7fe7519c9740 20 client.26585 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.456+0000 7fe7519c9740 1 --2- 172.21.15.114:0/1526016271 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562cb4487d80 0x562cb4886ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.456+0000 7fe7519c9740 4 client.26585 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:46:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.456+0000 7fe7519c9740 1 -- 172.21.15.114:0/1526016271 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562cb47deb80 con 0x562cb4487d80 2024-05-27T07:46:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.457+0000 7fe7437fe640 1 --2- 172.21.15.114:0/1526016271 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562cb4487d80 0x562cb4886ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.457+0000 7fe7237fe640 10 client.26585 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.457+0000 7fe7437fe640 1 --2- 172.21.15.114:0/1526016271 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562cb4487d80 0x562cb4886ad0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=184b3d2c060d8d4a server_cookie=2ed8994d610fac41 in_seq=0 out_seq=0 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7237fe640 1 -- 172.21.15.114:0/1526016271 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562cb47deb80 con 0x562cb4487d80 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7237fe640 10 client.26585 handle_command_reply: tid=0 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7519c9740 1 client.26585 shutdown 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7519c9740 1 -- 172.21.15.114:0/1526016271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a02f0 msgr2=0x562cb4874a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7519c9740 1 --2- 172.21.15.114:0/1526016271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a02f0 0x562cb4874a80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x562cb4867b10 tx=0x7fe734009f40 comp rx=0 tx=0).stop 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7519c9740 1 -- 172.21.15.114:0/1526016271 shutdown_connections 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7519c9740 1 --2- 172.21.15.114:0/1526016271 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562cb4487d80 0x562cb4886ad0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7519c9740 1 --2- 172.21.15.114:0/1526016271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44c91b0 0x562cb487a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7519c9740 1 --2- 172.21.15.114:0/1526016271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb4874fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7237fe640 0 client.26585 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7519c9740 1 --2- 172.21.15.114:0/1526016271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a02f0 0x562cb4874a80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7519c9740 1 -- 172.21.15.114:0/1526016271 >> 172.21.15.114:0/1526016271 conn(0x562cb44bd300 msgr2=0x562cb486c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7519c9740 1 -- 172.21.15.114:0/1526016271 shutdown_connections 2024-05-27T07:46:50.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7519c9740 1 -- 172.21.15.114:0/1526016271 wait complete. 2024-05-27T07:46:50.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.459+0000 7fe7519c9740 20 client.26585 trim_cache size 0 max 16384 2024-05-27T07:46:50.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.475+0000 7fe7519c9740 1 Processor -- start 2024-05-27T07:46:50.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.475+0000 7fe7519c9740 1 -- start start 2024-05-27T07:46:50.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.475+0000 7fe7519c9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44a02f0 0x562cb487a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.475+0000 7fe7519c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb48812e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.475+0000 7fe7519c9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44c91b0 0x562cb488c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.475+0000 7fe7519c9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe730021e50 con 0x562cb44a02f0 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.475+0000 7fe7519c9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe73001ef00 con 0x562cb44c91b0 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.475+0000 7fe7519c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe7300210b0 con 0x562cb44a84d0 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.475+0000 7fe743fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44a02f0 0x562cb487a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.475+0000 7fe743fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44a02f0 0x562cb487a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60160/0 (socket says 172.21.15.114:60160) 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.475+0000 7fe743fff640 1 -- 172.21.15.114:0/3939732457 learned_addr learned my addr 172.21.15.114:0/3939732457 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.475+0000 7fe7437fe640 1 --2- 172.21.15.114:0/3939732457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb48812e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.475+0000 7fe748fd8640 1 --2- 172.21.15.114:0/3939732457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44c91b0 0x562cb488c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.476+0000 7fe743fff640 1 -- 172.21.15.114:0/3939732457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 msgr2=0x562cb48812e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.476+0000 7fe743fff640 1 --2- 172.21.15.114:0/3939732457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb48812e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.476+0000 7fe743fff640 1 -- 172.21.15.114:0/3939732457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44c91b0 msgr2=0x562cb488c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.476+0000 7fe743fff640 1 --2- 172.21.15.114:0/3939732457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44c91b0 0x562cb488c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.476+0000 7fe743fff640 1 -- 172.21.15.114:0/3939732457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cb4872dc0 con 0x562cb44a02f0 2024-05-27T07:46:50.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.476+0000 7fe743fff640 1 --2- 172.21.15.114:0/3939732457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44a02f0 0x562cb487a4f0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe734005ce0 tx=0x7fe734004010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f54bc0fcad6d1b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:50.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.476+0000 7fe729ffb640 1 -- 172.21.15.114:0/3939732457 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73402d070 con 0x562cb44a02f0 2024-05-27T07:46:50.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.476+0000 7fe729ffb640 1 -- 172.21.15.114:0/3939732457 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe73401c440 con 0x562cb44a02f0 2024-05-27T07:46:50.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.476+0000 7fe729ffb640 1 -- 172.21.15.114:0/3939732457 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734022040 con 0x562cb44a02f0 2024-05-27T07:46:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.477+0000 7fe7519c9740 1 -- 172.21.15.114:0/3939732457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44a02f0 msgr2=0x562cb487a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.477+0000 7fe7519c9740 1 --2- 172.21.15.114:0/3939732457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44a02f0 0x562cb487a4f0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe734005ce0 tx=0x7fe734004010 comp rx=0 tx=0).stop 2024-05-27T07:46:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.477+0000 7fe7519c9740 1 -- 172.21.15.114:0/3939732457 shutdown_connections 2024-05-27T07:46:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.477+0000 7fe7519c9740 1 --2- 172.21.15.114:0/3939732457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44c91b0 0x562cb488c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.477+0000 7fe7519c9740 1 --2- 172.21.15.114:0/3939732457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb48812e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.477+0000 7fe7519c9740 1 --2- 172.21.15.114:0/3939732457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44a02f0 0x562cb487a4f0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.477+0000 7fe7519c9740 1 -- 172.21.15.114:0/3939732457 >> 172.21.15.114:0/3939732457 conn(0x562cb44bd300 msgr2=0x562cb486ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.477+0000 7fe7519c9740 1 -- 172.21.15.114:0/3939732457 shutdown_connections 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.477+0000 7fe7519c9740 1 -- 172.21.15.114:0/3939732457 wait complete. 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.477+0000 7fe7519c9740 1 -- start start 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe7519c9740 1 Processor -- start 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe7519c9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a02f0 0x562cb48809c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe7519c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb4880f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe7519c9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44c91b0 0x562cb487be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe7519c9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe7300209e0 con 0x562cb44c91b0 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe7519c9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe73001f570 con 0x562cb44a02f0 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe7519c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe73001fc40 con 0x562cb44a84d0 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe7519c9740 10 client.0 _ll_register_callbacks cb 0x7fe734005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe748fd8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44c91b0 0x562cb487be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe7437fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb4880f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe748fd8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44c91b0 0x562cb487be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60166/0 (socket says 172.21.15.114:60166) 2024-05-27T07:46:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe748fd8640 1 -- 172.21.15.114:0/3197328958 learned_addr learned my addr 172.21.15.114:0/3197328958 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:46:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe743fff640 1 --2- 172.21.15.114:0/3197328958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a02f0 0x562cb48809c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe748fd8640 1 -- 172.21.15.114:0/3197328958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 msgr2=0x562cb4880f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe748fd8640 1 --2- 172.21.15.114:0/3197328958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb4880f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe748fd8640 1 -- 172.21.15.114:0/3197328958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a02f0 msgr2=0x562cb48809c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:46:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe748fd8640 1 --2- 172.21.15.114:0/3197328958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a02f0 0x562cb48809c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe748fd8640 1 -- 172.21.15.114:0/3197328958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cb487c4c0 con 0x562cb44c91b0 2024-05-27T07:46:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.478+0000 7fe7437fe640 1 --2- 172.21.15.114:0/3197328958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb4880f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:46:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe748fd8640 1 --2- 172.21.15.114:0/3197328958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44c91b0 0x562cb487be80 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe73c010b90 tx=0x7fe73c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c28ee963608e0f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:46:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7237fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:46:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7237fe640 1 -- 172.21.15.114:0/3197328958 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c004b20 con 0x562cb44c91b0 2024-05-27T07:46:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7237fe640 1 -- 172.21.15.114:0/3197328958 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9466+0+0 (secure 0 0 0) 0x7fe73c01c070 con 0x562cb44c91b0 2024-05-27T07:46:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7519c9740 1 -- 172.21.15.114:0/3197328958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe73401c440 con 0x562cb44c91b0 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7237fe640 1 -- 172.21.15.114:0/3197328958 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fe73c004cc0 con 0x562cb44c91b0 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7519c9740 10 client.17121 fetch_fsmap learned FSMap version 43 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7519c9740 1 -- 172.21.15.114:0/3197328958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562cb48eea80 con 0x562cb44c91b0 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7237fe640 1 -- 172.21.15.114:0/3197328958 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe73401c440 con 0x562cb44c91b0 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7237fe640 1 -- 172.21.15.114:0/3197328958 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 43) ==== 7725+0+0 (secure 0 0 0) 0x7fe73c005220 con 0x562cb44c91b0 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7519c9740 10 client.17121 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7519c9740 10 client.17121 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7519c9740 20 client.17121 populate_metadata read hostname 'smithi114' 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7519c9740 1 --2- 172.21.15.114:0/3197328958 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562cb445d1c0 0x562cb48f1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7519c9740 4 client.17121 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.479+0000 7fe7519c9740 1 -- 172.21.15.114:0/3197328958 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562cb47d8360 con 0x562cb445d1c0 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.480+0000 7fe743fff640 1 --2- 172.21.15.114:0/3197328958 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562cb445d1c0 0x562cb48f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.480+0000 7fe7237fe640 10 client.17121 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.480+0000 7fe743fff640 1 --2- 172.21.15.114:0/3197328958 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562cb445d1c0 0x562cb48f1290 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91c3e292df93188e server_cookie=271ddd7216f40460 in_seq=0 out_seq=0 2024-05-27T07:46:50.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:50 smithi163 ceph-mon[36451]: daemon mds.f restarted 2024-05-27T07:46:50.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:50 smithi186 ceph-mon[36647]: daemon mds.f restarted 2024-05-27T07:46:50.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:50 smithi114 ceph-mon[32038]: daemon mds.f restarted 2024-05-27T07:46:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.959+0000 7fe7237fe640 1 -- 172.21.15.114:0/3197328958 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562cb47d8360 con 0x562cb445d1c0 2024-05-27T07:46:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.959+0000 7fe7237fe640 10 client.17121 handle_command_reply: tid=0 2024-05-27T07:46:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.959+0000 7fe7519c9740 1 client.17121 shutdown 2024-05-27T07:46:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.959+0000 7fe7519c9740 1 -- 172.21.15.114:0/3197328958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44c91b0 msgr2=0x562cb487be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.959+0000 7fe7519c9740 1 --2- 172.21.15.114:0/3197328958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44c91b0 0x562cb487be80 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe73c010b90 tx=0x7fe73c004a40 comp rx=0 tx=0).stop 2024-05-27T07:46:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.959+0000 7fe7519c9740 1 -- 172.21.15.114:0/3197328958 shutdown_connections 2024-05-27T07:46:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.959+0000 7fe7519c9740 1 --2- 172.21.15.114:0/3197328958 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562cb445d1c0 0x562cb48f1290 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.959+0000 7fe7237fe640 0 client.17121 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:46:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.959+0000 7fe7519c9740 1 --2- 172.21.15.114:0/3197328958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562cb44c91b0 0x562cb487be80 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.959+0000 7fe7519c9740 1 --2- 172.21.15.114:0/3197328958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562cb44a84d0 0x562cb4880f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.959+0000 7fe7519c9740 1 --2- 172.21.15.114:0/3197328958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562cb44a02f0 0x562cb48809c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.959+0000 7fe7519c9740 1 -- 172.21.15.114:0/3197328958 >> 172.21.15.114:0/3197328958 conn(0x562cb44bd300 msgr2=0x562cb486a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.960+0000 7fe7519c9740 1 -- 172.21.15.114:0/3197328958 shutdown_connections 2024-05-27T07:46:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.960+0000 7fe7519c9740 1 -- 172.21.15.114:0/3197328958 wait complete. 2024-05-27T07:46:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.960+0000 7fe7519c9740 20 client.17121 trim_cache size 0 max 16384 2024-05-27T07:46:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:46:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:46:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:46:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.961+0000 7fe74aa3b640 1 -- 172.21.15.114:0/3264527188 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe72c0b29d0 msgr2=0x7fe72c0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.961+0000 7fe74aa3b640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe72c0b29d0 0x7fe72c0b4ee0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x562cb4867540 tx=0x7fe73c004ab0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.961+0000 7fe74aa3b640 1 -- 172.21.15.114:0/3264527188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe744104b30 msgr2=0x7fe7441a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:46:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.961+0000 7fe74aa3b640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe744104b30 0x7fe7441a2b90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe73802a8f0 tx=0x7fe738030fa0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.961+0000 7fe74aa3b640 1 -- 172.21.15.114:0/3264527188 shutdown_connections 2024-05-27T07:46:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.961+0000 7fe74aa3b640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe72c0b29d0 0x7fe72c0b4ee0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.961+0000 7fe74aa3b640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe744110400 0x7fe74419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.961+0000 7fe74aa3b640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe744104b30 0x7fe7441a2b90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.961+0000 7fe74aa3b640 1 --2- 172.21.15.114:0/3264527188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7441021c0 0x7fe7441a2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:46:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.961+0000 7fe74aa3b640 1 -- 172.21.15.114:0/3264527188 >> 172.21.15.114:0/3264527188 conn(0x7fe7440779d0 msgr2=0x7fe744100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:46:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.961+0000 7fe74aa3b640 1 -- 172.21.15.114:0/3264527188 shutdown_connections 2024-05-27T07:46:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:46:50.961+0000 7fe74aa3b640 1 -- 172.21.15.114:0/3264527188 wait complete. 2024-05-27T07:46:51.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 714092 12.01 MB/sec execute 183 sec latency 264.639 ms 2024-05-27T07:46:51.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:51 smithi163 ceph-mon[36451]: pgmap v227: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 288 op/s 2024-05-27T07:46:51.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:51 smithi163 ceph-mon[36451]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T07:46:51.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:51 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6836/3355377275,v1:172.21.15.186:6837/3355377275] up:boot 2024-05-27T07:46:51.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:51 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:46:51.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:51 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:46:51.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:51 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:46:51.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:51 smithi186 ceph-mon[36647]: pgmap v227: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 288 op/s 2024-05-27T07:46:51.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:51 smithi186 ceph-mon[36647]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T07:46:51.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:51 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6836/3355377275,v1:172.21.15.186:6837/3355377275] up:boot 2024-05-27T07:46:51.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:51 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:46:51.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:51 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:46:51.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:51 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:46:51.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:51 smithi114 ceph-mon[32038]: pgmap v227: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 288 op/s 2024-05-27T07:46:51.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:51 smithi114 ceph-mon[32038]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T07:46:51.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:51 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6836/3355377275,v1:172.21.15.186:6837/3355377275] up:boot 2024-05-27T07:46:51.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:51 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:46:51.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:51 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:46:51.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:51 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:46:52.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 715680 11.98 MB/sec execute 184 sec latency 41.824 ms 2024-05-27T07:46:53.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 717978 11.99 MB/sec execute 185 sec latency 20.791 ms 2024-05-27T07:46:53.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:53 smithi163 ceph-mon[36451]: pgmap v229: 129 pgs: 129 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 237 op/s 2024-05-27T07:46:53.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:53 smithi186 ceph-mon[36647]: pgmap v229: 129 pgs: 129 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 237 op/s 2024-05-27T07:46:53.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:53 smithi114 ceph-mon[32038]: pgmap v229: 129 pgs: 129 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 237 op/s 2024-05-27T07:46:54.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 720293 11.99 MB/sec execute 186 sec latency 28.065 ms 2024-05-27T07:46:55.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 722266 11.99 MB/sec execute 187 sec latency 18.938 ms 2024-05-27T07:46:55.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:55 smithi163 ceph-mon[36451]: pgmap v230: 129 pgs: 129 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 237 op/s 2024-05-27T07:46:55.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:55 smithi186 ceph-mon[36647]: pgmap v230: 129 pgs: 129 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 237 op/s 2024-05-27T07:46:55.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:55 smithi114 ceph-mon[32038]: pgmap v230: 129 pgs: 129 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 237 op/s 2024-05-27T07:46:56.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 724377 11.99 MB/sec execute 188 sec latency 93.830 ms 2024-05-27T07:46:57.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 725857 11.96 MB/sec execute 189 sec latency 37.105 ms 2024-05-27T07:46:57.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:57 smithi163 ceph-mon[36451]: pgmap v231: 129 pgs: 129 active+clean; 68 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 394 op/s 2024-05-27T07:46:57.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:57 smithi186 ceph-mon[36647]: pgmap v231: 129 pgs: 129 active+clean; 68 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 394 op/s 2024-05-27T07:46:57.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:57 smithi114 ceph-mon[32038]: pgmap v231: 129 pgs: 129 active+clean; 68 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 394 op/s 2024-05-27T07:46:58.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 728275 11.97 MB/sec execute 190 sec latency 30.904 ms 2024-05-27T07:46:59.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 730364 11.95 MB/sec execute 191 sec latency 20.459 ms 2024-05-27T07:46:59.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:46:59 smithi163 ceph-mon[36451]: pgmap v232: 129 pgs: 129 active+clean; 68 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 394 op/s 2024-05-27T07:46:59.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:46:59 smithi186 ceph-mon[36647]: pgmap v232: 129 pgs: 129 active+clean; 68 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 394 op/s 2024-05-27T07:46:59.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:46:59 smithi114 ceph-mon[32038]: pgmap v232: 129 pgs: 129 active+clean; 68 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 394 op/s 2024-05-27T07:47:00.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 732296 11.95 MB/sec execute 192 sec latency 68.094 ms 2024-05-27T07:47:01.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 734277 11.95 MB/sec execute 193 sec latency 85.939 ms 2024-05-27T07:47:01.640 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:01 smithi186 ceph-mon[36647]: pgmap v233: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 437 op/s 2024-05-27T07:47:01.641 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1149861700 time 2024-05-27T07:47:01.620265+0000 2024-05-27T07:47:01.641 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:47:01.641 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:01.641 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f115c0a6ba6] 2024-05-27T07:47:01.641 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f115c0a6dd1] 2024-05-27T07:47:01.641 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b6c09623d] 2024-05-27T07:47:01.641 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556b6c081971] 2024-05-27T07:47:01.641 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 5: (SnapRealm::check_cache() const+0x13e) [0x556b6c07ffa0] 2024-05-27T07:47:01.642 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b6c080651] 2024-05-27T07:47:01.642 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b6c015347] 2024-05-27T07:47:01.642 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b6bd1187e] 2024-05-27T07:47:01.642 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b6bd16e94] 2024-05-27T07:47:01.642 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556b6bd21969] 2024-05-27T07:47:01.642 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 11: (MDLog::_replay_thread()+0x1f38) [0x556b6c0ba48c] 2024-05-27T07:47:01.642 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 12: (MDLog::ReplayThread::entry()+0x11) [0x556b6c0c39cb] 2024-05-27T07:47:01.642 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 13: (Thread::entry_wrapper()+0x43) [0x7f115c07572d] 2024-05-27T07:47:01.642 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 14: (Thread::_entry_func(void*)+0xd) [0x7f115c075749] 2024-05-27T07:47:01.642 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f115ac191ca] 2024-05-27T07:47:01.642 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 16: clone() 2024-05-27T07:47:01.642 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.621+0000 7f1149861700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1149861700 time 2024-05-27T07:47:01.620265+0000 2024-05-27T07:47:01.643 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:47:01.643 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.643 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:01.643 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f115c0a6ba6] 2024-05-27T07:47:01.643 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f115c0a6dd1] 2024-05-27T07:47:01.643 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b6c09623d] 2024-05-27T07:47:01.643 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556b6c081971] 2024-05-27T07:47:01.643 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 5: (SnapRealm::check_cache() const+0x13e) [0x556b6c07ffa0] 2024-05-27T07:47:01.643 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b6c080651] 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b6c015347] 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b6bd1187e] 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b6bd16e94] 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556b6bd21969] 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 11: (MDLog::_replay_thread()+0x1f38) [0x556b6c0ba48c] 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 12: (MDLog::ReplayThread::entry()+0x11) [0x556b6c0c39cb] 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 13: (Thread::entry_wrapper()+0x43) [0x7f115c07572d] 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 14: (Thread::_entry_func(void*)+0xd) [0x7f115c075749] 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f115ac191ca] 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 16: clone() 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: *** Caught signal (Aborted) ** 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: in thread 7f1149861700 thread_name:md_log_replay 2024-05-27T07:47:01.644 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f115ac23d20] 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2: gsignal() 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 3: abort() 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f115c0a6ced] 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f115c0a6dd1] 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b6c09623d] 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556b6c081971] 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 8: (SnapRealm::check_cache() const+0x13e) [0x556b6c07ffa0] 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b6c080651] 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b6c015347] 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b6bd1187e] 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b6bd16e94] 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556b6bd21969] 2024-05-27T07:47:01.645 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 14: (MDLog::_replay_thread()+0x1f38) [0x556b6c0ba48c] 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 15: (MDLog::ReplayThread::entry()+0x11) [0x556b6c0c39cb] 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 16: (Thread::entry_wrapper()+0x43) [0x7f115c07572d] 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 17: (Thread::_entry_func(void*)+0xd) [0x7f115c075749] 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f115ac191ca] 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 19: clone() 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.624+0000 7f1149861700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: in thread 7f1149861700 thread_name:md_log_replay 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f115ac23d20] 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2: gsignal() 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 3: abort() 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f115c0a6ced] 2024-05-27T07:47:01.646 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f115c0a6dd1] 2024-05-27T07:47:01.647 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b6c09623d] 2024-05-27T07:47:01.647 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556b6c081971] 2024-05-27T07:47:01.647 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 8: (SnapRealm::check_cache() const+0x13e) [0x556b6c07ffa0] 2024-05-27T07:47:01.647 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b6c080651] 2024-05-27T07:47:01.647 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b6c015347] 2024-05-27T07:47:01.647 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b6bd1187e] 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b6bd16e94] 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556b6bd21969] 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 14: (MDLog::_replay_thread()+0x1f38) [0x556b6c0ba48c] 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 15: (MDLog::ReplayThread::entry()+0x11) [0x556b6c0c39cb] 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 16: (Thread::entry_wrapper()+0x43) [0x7f115c07572d] 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 17: (Thread::_entry_func(void*)+0xd) [0x7f115c075749] 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f115ac191ca] 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 19: clone() 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: -1> 2024-05-27T07:47:01.621+0000 7f1149861700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1149861700 time 2024-05-27T07:47:01.620265+0000 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.648 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f115c0a6ba6] 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f115c0a6dd1] 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b6c09623d] 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556b6c081971] 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 5: (SnapRealm::check_cache() const+0x13e) [0x556b6c07ffa0] 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b6c080651] 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b6c015347] 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b6bd1187e] 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b6bd16e94] 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556b6bd21969] 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 11: (MDLog::_replay_thread()+0x1f38) [0x556b6c0ba48c] 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 12: (MDLog::ReplayThread::entry()+0x11) [0x556b6c0c39cb] 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 13: (Thread::entry_wrapper()+0x43) [0x7f115c07572d] 2024-05-27T07:47:01.649 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 14: (Thread::_entry_func(void*)+0xd) [0x7f115c075749] 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f115ac191ca] 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 16: clone() 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 0> 2024-05-27T07:47:01.624+0000 7f1149861700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: in thread 7f1149861700 thread_name:md_log_replay 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f115ac23d20] 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2: gsignal() 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 3: abort() 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f115c0a6ced] 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f115c0a6dd1] 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b6c09623d] 2024-05-27T07:47:01.650 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556b6c081971] 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 8: (SnapRealm::check_cache() const+0x13e) [0x556b6c07ffa0] 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b6c080651] 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b6c015347] 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b6bd1187e] 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b6bd16e94] 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556b6bd21969] 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 14: (MDLog::_replay_thread()+0x1f38) [0x556b6c0ba48c] 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 15: (MDLog::ReplayThread::entry()+0x11) [0x556b6c0c39cb] 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 16: (Thread::entry_wrapper()+0x43) [0x7f115c07572d] 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 17: (Thread::_entry_func(void*)+0xd) [0x7f115c075749] 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f115ac191ca] 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 19: clone() 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:47:01.651 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:01 smithi163 ceph-mon[36451]: pgmap v233: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 437 op/s 2024-05-27T07:47:01.728 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: -1> 2024-05-27T07:47:01.621+0000 7f1149861700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1149861700 time 2024-05-27T07:47:01.620265+0000 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f115c0a6ba6] 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f115c0a6dd1] 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b6c09623d] 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556b6c081971] 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 5: (SnapRealm::check_cache() const+0x13e) [0x556b6c07ffa0] 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b6c080651] 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b6c015347] 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b6bd1187e] 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b6bd16e94] 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556b6bd21969] 2024-05-27T07:47:01.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 11: (MDLog::_replay_thread()+0x1f38) [0x556b6c0ba48c] 2024-05-27T07:47:01.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 12: (MDLog::ReplayThread::entry()+0x11) [0x556b6c0c39cb] 2024-05-27T07:47:01.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 13: (Thread::entry_wrapper()+0x43) [0x7f115c07572d] 2024-05-27T07:47:01.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 14: (Thread::_entry_func(void*)+0xd) [0x7f115c075749] 2024-05-27T07:47:01.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f115ac191ca] 2024-05-27T07:47:01.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 16: clone() 2024-05-27T07:47:01.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 0> 2024-05-27T07:47:01.624+0000 7f1149861700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:47:01.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: in thread 7f1149861700 thread_name:md_log_replay 2024-05-27T07:47:01.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:01.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f115ac23d20] 2024-05-27T07:47:01.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2: gsignal() 2024-05-27T07:47:01.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 3: abort() 2024-05-27T07:47:01.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f115c0a6ced] 2024-05-27T07:47:01.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f115c0a6dd1] 2024-05-27T07:47:01.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b6c09623d] 2024-05-27T07:47:01.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556b6c081971] 2024-05-27T07:47:01.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 8: (SnapRealm::check_cache() const+0x13e) [0x556b6c07ffa0] 2024-05-27T07:47:01.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b6c080651] 2024-05-27T07:47:01.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b6c015347] 2024-05-27T07:47:01.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b6bd1187e] 2024-05-27T07:47:01.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b6bd16e94] 2024-05-27T07:47:01.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556b6bd21969] 2024-05-27T07:47:01.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 14: (MDLog::_replay_thread()+0x1f38) [0x556b6c0ba48c] 2024-05-27T07:47:01.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 15: (MDLog::ReplayThread::entry()+0x11) [0x556b6c0c39cb] 2024-05-27T07:47:01.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 16: (Thread::entry_wrapper()+0x43) [0x7f115c07572d] 2024-05-27T07:47:01.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 17: (Thread::_entry_func(void*)+0xd) [0x7f115c075749] 2024-05-27T07:47:01.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f115ac191ca] 2024-05-27T07:47:01.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 19: clone() 2024-05-27T07:47:01.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:47:01.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73316]: 2024-05-27T07:47:01.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:01 smithi186 podman[73361]: 2024-05-27 07:47:01.683070086 +0000 UTC m=+0.017512475 container died 779662ab5886efb3866c2b8ffe549a590cbb072cb38a1cf617e4dc5e802dc19e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:47:01.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:01 smithi114 ceph-mon[32038]: pgmap v233: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 437 op/s 2024-05-27T07:47:02.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 734752 11.90 MB/sec execute 194 sec latency 145.635 ms 2024-05-27T07:47:02.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:02 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:47:02.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:02 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:47:02.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:02 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:47:03.264 INFO:tasks.workunit.client.0.smithi114.stdout: 1 734755 11.84 MB/sec execute 195 sec latency 551.362 ms 2024-05-27T07:47:03.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:03 smithi163 ceph-mon[36451]: pgmap v234: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 344 op/s 2024-05-27T07:47:03.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:03 smithi163 ceph-mon[36451]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T07:47:03.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:03 smithi186 ceph-mon[36647]: pgmap v234: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 344 op/s 2024-05-27T07:47:03.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:03 smithi186 ceph-mon[36647]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T07:47:03.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:03 smithi114 ceph-mon[32038]: pgmap v234: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 344 op/s 2024-05-27T07:47:03.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:03 smithi114 ceph-mon[32038]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T07:47:04.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.120+0000 7f6505d22640 1 -- 172.21.15.114:0/3480659243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64f809a390 msgr2=0x7f64f809a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:04.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.120+0000 7f6505d22640 1 --2- 172.21.15.114:0/3480659243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64f809a390 0x7f64f809a7a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009900 tx=0x7f64ec02a4d0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.121+0000 7f6505d22640 1 -- 172.21.15.114:0/3480659243 shutdown_connections 2024-05-27T07:47:04.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.121+0000 7f6505d22640 1 --2- 172.21.15.114:0/3480659243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f809ad30 0x7f64f809b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.121+0000 7f6505d22640 1 --2- 172.21.15.114:0/3480659243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64f809a390 0x7f64f809a7a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.121+0000 7f6505d22640 1 --2- 172.21.15.114:0/3480659243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64f8095940 0x7f64f8095d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.121+0000 7f6505d22640 1 -- 172.21.15.114:0/3480659243 >> 172.21.15.114:0/3480659243 conn(0x7f64f800aff0 msgr2=0x7f64f800b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.124+0000 7f6505d22640 1 -- 172.21.15.114:0/3480659243 shutdown_connections 2024-05-27T07:47:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.126+0000 7f6505d22640 1 -- 172.21.15.114:0/3480659243 wait complete. 2024-05-27T07:47:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.127+0000 7f6505d22640 1 Processor -- start 2024-05-27T07:47:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.127+0000 7f6505d22640 1 -- start start 2024-05-27T07:47:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.128+0000 7f6505d22640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64f8095940 0x7f64f81399c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.128+0000 7f6505d22640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64f809a390 0x7f64f8139f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.128+0000 7f6505d22640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f809ad30 0x7f64f8133b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.128+0000 7f6505d22640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f64f80ad2f0 con 0x7f64f809ad30 2024-05-27T07:47:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.128+0000 7f6505d22640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f64f80ad170 con 0x7f64f8095940 2024-05-27T07:47:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.128+0000 7f6505d22640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64f80ad470 con 0x7f64f809a390 2024-05-27T07:47:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.128+0000 7f6505521640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f809ad30 0x7f64f8133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.128+0000 7f6505521640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f809ad30 0x7f64f8133b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34368/0 (socket says 172.21.15.114:34368) 2024-05-27T07:47:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.128+0000 7f6505521640 1 -- 172.21.15.114:0/3631119080 learned_addr learned my addr 172.21.15.114:0/3631119080 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.129+0000 7f64fffff640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64f809a390 0x7f64f8139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.129+0000 7f6504d20640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64f8095940 0x7f64f81399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.129+0000 7f6505521640 1 -- 172.21.15.114:0/3631119080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64f809a390 msgr2=0x7f64f8139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.129+0000 7f6505521640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64f809a390 0x7f64f8139f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.129+0000 7f6505521640 1 -- 172.21.15.114:0/3631119080 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64f8095940 msgr2=0x7f64f81399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:04.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.129+0000 7f6505521640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64f8095940 0x7f64f81399c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.129+0000 7f6505521640 1 -- 172.21.15.114:0/3631119080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f8134260 con 0x7f64f809ad30 2024-05-27T07:47:04.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.129+0000 7f6505521640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f809ad30 0x7f64f8133b90 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f650002e1b0 tx=0x7f650007a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:04.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.130+0000 7f64fdffb640 1 -- 172.21.15.114:0/3631119080 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650007ab90 con 0x7f64f809ad30 2024-05-27T07:47:04.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.131+0000 7f64fdffb640 1 -- 172.21.15.114:0/3631119080 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f650007ad30 con 0x7f64f809ad30 2024-05-27T07:47:04.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.131+0000 7f64fdffb640 1 -- 172.21.15.114:0/3631119080 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650007f670 con 0x7f64f809ad30 2024-05-27T07:47:04.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.131+0000 7f6505d22640 1 -- 172.21.15.114:0/3631119080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f8134550 con 0x7f64f809ad30 2024-05-27T07:47:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.131+0000 7f6505d22640 1 -- 172.21.15.114:0/3631119080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f809b960 con 0x7f64f809ad30 2024-05-27T07:47:04.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.134+0000 7f650de38740 1 Processor -- start 2024-05-27T07:47:04.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.139+0000 7f650de38740 1 -- start start 2024-05-27T07:47:04.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.139+0000 7f650de38740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d8a71b0 0x562b5dc48590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.139+0000 7f650de38740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d87e2f0 0x562b5dc48ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.139+0000 7f650de38740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc49010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.139+0000 7f650de38740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562b5dbbac30 con 0x562b5d8a71b0 2024-05-27T07:47:04.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.139+0000 7f650de38740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562b5dbaa7d0 con 0x562b5d87e2f0 2024-05-27T07:47:04.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.139+0000 7f650de38740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562b5dbae810 con 0x562b5d8864d0 2024-05-27T07:47:04.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.140+0000 7f64fdffb640 1 -- 172.21.15.114:0/3631119080 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6500080a80 con 0x7f64f809ad30 2024-05-27T07:47:04.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.140+0000 7f64fffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d8a71b0 0x562b5dc48590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.140+0000 7f64fffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d8a71b0 0x562b5dc48590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34374/0 (socket says 172.21.15.114:34374) 2024-05-27T07:47:04.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.140+0000 7f64fffff640 1 -- 172.21.15.114:0/1562950799 learned_addr learned my addr 172.21.15.114:0/1562950799 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:04.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.141+0000 7f6504d20640 1 --2- 172.21.15.114:0/1562950799 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d87e2f0 0x562b5dc48ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.141+0000 7f64fdffb640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64f00b2b30 0x7f64f00b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.141+0000 7f64fdffb640 1 -- 172.21.15.114:0/3631119080 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9642+0+0 (secure 0 0 0) 0x7f650014ed00 con 0x7f64f809ad30 2024-05-27T07:47:04.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.141+0000 7f64fffff640 1 --2- 172.21.15.114:0/1562950799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc49010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.142+0000 7f64fffff640 1 -- 172.21.15.114:0/1562950799 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d87e2f0 msgr2=0x562b5dc48ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.142+0000 7f64fffff640 1 --2- 172.21.15.114:0/1562950799 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d87e2f0 0x562b5dc48ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.142+0000 7f64fffff640 1 -- 172.21.15.114:0/1562950799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d8a71b0 msgr2=0x562b5dc48590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.142+0000 7f64fffff640 1 --2- 172.21.15.114:0/1562950799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d8a71b0 0x562b5dc48590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.142+0000 7f64fffff640 1 -- 172.21.15.114:0/1562950799 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b5d8bf210 con 0x562b5d8864d0 2024-05-27T07:47:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.142+0000 7f64fffff640 1 --2- 172.21.15.114:0/1562950799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc49010 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0096c0 tx=0x7f64ec005a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a138b13c9d17e38d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.143+0000 7f64eaffd640 1 -- 172.21.15.114:0/1562950799 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec041020 con 0x562b5d8864d0 2024-05-27T07:47:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.143+0000 7f64eaffd640 1 -- 172.21.15.114:0/1562950799 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64ec003bb0 con 0x562b5d8864d0 2024-05-27T07:47:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.143+0000 7f64eaffd640 1 -- 172.21.15.114:0/1562950799 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec035040 con 0x562b5d8864d0 2024-05-27T07:47:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.144+0000 7f650de38740 1 -- 172.21.15.114:0/1562950799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 msgr2=0x562b5dc49010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.144+0000 7f650de38740 1 --2- 172.21.15.114:0/1562950799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc49010 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0096c0 tx=0x7f64ec005a00 comp rx=0 tx=0).stop 2024-05-27T07:47:04.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.146+0000 7f6505521640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64f00b2b30 0x7f64f00b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.148+0000 7f650de38740 1 -- 172.21.15.114:0/1562950799 shutdown_connections 2024-05-27T07:47:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.148+0000 7f650de38740 1 --2- 172.21.15.114:0/1562950799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc49010 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.148+0000 7f650de38740 1 --2- 172.21.15.114:0/1562950799 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d87e2f0 0x562b5dc48ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.148+0000 7f650de38740 1 --2- 172.21.15.114:0/1562950799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d8a71b0 0x562b5dc48590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.148+0000 7f650de38740 1 -- 172.21.15.114:0/1562950799 >> 172.21.15.114:0/1562950799 conn(0x562b5d89b300 msgr2=0x562b5dc44d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:04.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.149+0000 7f6505521640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64f00b2b30 0x7f64f00b5040 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f650014c100 tx=0x7f6500077040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.154+0000 7f650de38740 1 -- 172.21.15.114:0/1562950799 shutdown_connections 2024-05-27T07:47:04.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.155+0000 7f650de38740 1 -- 172.21.15.114:0/1562950799 wait complete. 2024-05-27T07:47:04.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.155+0000 7f650de38740 1 -- start start 2024-05-27T07:47:04.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.155+0000 7f650de38740 1 Processor -- start 2024-05-27T07:47:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.159+0000 7f650de38740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d87e2f0 0x562b5dc545f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.159+0000 7f650de38740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc54b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.159+0000 7f650de38740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d8a71b0 0x562b5dc52ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.159+0000 7f650de38740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562b5d890790 con 0x562b5d8a71b0 2024-05-27T07:47:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.159+0000 7f650de38740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562b5db66890 con 0x562b5d87e2f0 2024-05-27T07:47:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.159+0000 7f650de38740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562b5dc55660 con 0x562b5d8864d0 2024-05-27T07:47:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.159+0000 7f650de38740 10 client.0 _ll_register_callbacks cb 0x562b5d8bea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.160+0000 7f6504d20640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc54b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.160+0000 7f6504d20640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc54b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:52680/0 (socket says 172.21.15.114:52680) 2024-05-27T07:47:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.160+0000 7f6504d20640 1 -- 172.21.15.114:0/2290721249 learned_addr learned my addr 172.21.15.114:0/2290721249 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.160+0000 7f64fffff640 1 --2- 172.21.15.114:0/2290721249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d8a71b0 0x562b5dc52ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.160+0000 7f64fffff640 1 --2- 172.21.15.114:0/2290721249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d87e2f0 0x562b5dc545f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.160+0000 7f6504d20640 1 -- 172.21.15.114:0/2290721249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d87e2f0 msgr2=0x562b5dc545f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.160+0000 7f6504d20640 1 --2- 172.21.15.114:0/2290721249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d87e2f0 0x562b5dc545f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.160+0000 7f6504d20640 1 -- 172.21.15.114:0/2290721249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d8a71b0 msgr2=0x562b5dc52ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.160+0000 7f6504d20640 1 --2- 172.21.15.114:0/2290721249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d8a71b0 0x562b5dc52ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.160+0000 7f6504d20640 1 -- 172.21.15.114:0/2290721249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b5d98b620 con 0x562b5d8864d0 2024-05-27T07:47:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.165+0000 7f6504d20640 1 --2- 172.21.15.114:0/2290721249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc54b50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f64f4010500 tx=0x7f64f40109c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78e53f0fc18b2ee1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.165+0000 7f64dffff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:47:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.165+0000 7f64dffff640 1 -- 172.21.15.114:0/2290721249 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f401a020 con 0x562b5d8864d0 2024-05-27T07:47:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.165+0000 7f650de38740 1 -- 172.21.15.114:0/2290721249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64ec003bb0 con 0x562b5d8864d0 2024-05-27T07:47:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.165+0000 7f64dffff640 1 -- 172.21.15.114:0/2290721249 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9642+0+0 (secure 0 0 0) 0x7f64f4016690 con 0x562b5d8864d0 2024-05-27T07:47:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.173+0000 7f64dffff640 1 -- 172.21.15.114:0/2290721249 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f64f4016890 con 0x562b5d8864d0 2024-05-27T07:47:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.174+0000 7f650de38740 10 client.26698 fetch_fsmap learned FSMap version 43 2024-05-27T07:47:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.174+0000 7f650de38740 1 -- 172.21.15.114:0/2290721249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562b5dc61a80 con 0x562b5d8864d0 2024-05-27T07:47:04.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.178+0000 7f64dffff640 1 -- 172.21.15.114:0/2290721249 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64ec003bb0 con 0x562b5d8864d0 2024-05-27T07:47:04.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.178+0000 7f64dffff640 1 -- 172.21.15.114:0/2290721249 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 43) ==== 7725+0+0 (secure 0 0 0) 0x7f64f4027020 con 0x562b5d8864d0 2024-05-27T07:47:04.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.179+0000 7f650de38740 10 client.26698 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T07:47:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.179+0000 7f650de38740 10 client.26698 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.179+0000 7f650de38740 20 client.26698 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.179+0000 7f650de38740 1 --2- 172.21.15.114:0/2290721249 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b5d865d80 0x562b5dc65490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.179+0000 7f650de38740 4 client.26698 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.179+0000 7f650de38740 1 -- 172.21.15.114:0/2290721249 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b5dbbcb80 con 0x562b5d865d80 2024-05-27T07:47:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.183+0000 7f64fffff640 1 --2- 172.21.15.114:0/2290721249 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b5d865d80 0x562b5dc65490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.188+0000 7f64fffff640 1 --2- 172.21.15.114:0/2290721249 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b5d865d80 0x562b5dc65490 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df12f359ae04db53 server_cookie=c3dd820ff7992206 in_seq=0 out_seq=0 2024-05-27T07:47:04.190 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.190+0000 7f64dffff640 10 client.26698 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.191+0000 7f64dffff640 1 -- 172.21.15.114:0/2290721249 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562b5dbbcb80 con 0x562b5d865d80 2024-05-27T07:47:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.191+0000 7f64dffff640 10 client.26698 handle_command_reply: tid=0 2024-05-27T07:47:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.191+0000 7f650de38740 1 client.26698 shutdown 2024-05-27T07:47:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.197+0000 7f650de38740 1 -- 172.21.15.114:0/2290721249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 msgr2=0x562b5dc54b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.197+0000 7f650de38740 1 --2- 172.21.15.114:0/2290721249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc54b50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f64f4010500 tx=0x7f64f40109c0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.200+0000 7f650de38740 1 -- 172.21.15.114:0/2290721249 shutdown_connections 2024-05-27T07:47:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.201+0000 7f650de38740 1 --2- 172.21.15.114:0/2290721249 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b5d865d80 0x562b5dc65490 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.201+0000 7f650de38740 1 --2- 172.21.15.114:0/2290721249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d8a71b0 0x562b5dc52ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.201+0000 7f64dffff640 0 client.26698 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.201+0000 7f650de38740 1 --2- 172.21.15.114:0/2290721249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc54b50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.201+0000 7f650de38740 1 --2- 172.21.15.114:0/2290721249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d87e2f0 0x562b5dc545f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.201+0000 7f650de38740 1 -- 172.21.15.114:0/2290721249 >> 172.21.15.114:0/2290721249 conn(0x562b5d89b300 msgr2=0x562b5dc4a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.206+0000 7f650de38740 1 -- 172.21.15.114:0/2290721249 shutdown_connections 2024-05-27T07:47:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.206+0000 7f650de38740 1 -- 172.21.15.114:0/2290721249 wait complete. 2024-05-27T07:47:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.207+0000 7f650de38740 20 client.26698 trim_cache size 0 max 16384 2024-05-27T07:47:04.228 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:03 smithi186 podman[73361]: 2024-05-27 07:47:03.870425603 +0000 UTC m=+2.204867996 container remove 779662ab5886efb3866c2b8ffe549a590cbb072cb38a1cf617e4dc5e802dc19e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS) 2024-05-27T07:47:04.228 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:03 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:47:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.237+0000 7f650de38740 1 Processor -- start 2024-05-27T07:47:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.237+0000 7f650de38740 1 -- start start 2024-05-27T07:47:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f650de38740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d865d80 0x562b5dc53000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f650de38740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d87e2f0 0x562b5dc54720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f650de38740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc4c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f650de38740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f64f81541a0 con 0x562b5d87e2f0 2024-05-27T07:47:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f650de38740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f64f80063a0 con 0x562b5d865d80 2024-05-27T07:47:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f650de38740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64f8153400 con 0x562b5d8864d0 2024-05-27T07:47:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f64fffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d865d80 0x562b5dc53000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f64fffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc4c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f64fffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d865d80 0x562b5dc53000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:44884/0 (socket says 172.21.15.114:44884) 2024-05-27T07:47:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f64fffff640 1 -- 172.21.15.114:0/3980577236 learned_addr learned my addr 172.21.15.114:0/3980577236 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f64fffff640 1 -- 172.21.15.114:0/3980577236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 msgr2=0x562b5dc4c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f64fffff640 1 --2- 172.21.15.114:0/3980577236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc4c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f64fffff640 1 -- 172.21.15.114:0/3980577236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d87e2f0 msgr2=0x562b5dc54720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:47:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.238+0000 7f64fffff640 1 --2- 172.21.15.114:0/3980577236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d87e2f0 0x562b5dc54720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.239+0000 7f64fffff640 1 -- 172.21.15.114:0/3980577236 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b5dc51000 con 0x562b5d865d80 2024-05-27T07:47:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.239+0000 7f64fffff640 1 --2- 172.21.15.114:0/3980577236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d865d80 0x562b5dc53000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f64ec005d00 tx=0x7f64ec005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22778ac322908c05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.240+0000 7f64ea7fc640 1 -- 172.21.15.114:0/3980577236 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec041040 con 0x562b5d865d80 2024-05-27T07:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.240+0000 7f64ea7fc640 1 -- 172.21.15.114:0/3980577236 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64ec042040 con 0x562b5d865d80 2024-05-27T07:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.240+0000 7f64ea7fc640 1 -- 172.21.15.114:0/3980577236 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec035040 con 0x562b5d865d80 2024-05-27T07:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.241+0000 7f650de38740 1 -- 172.21.15.114:0/3980577236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d865d80 msgr2=0x562b5dc53000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.241+0000 7f650de38740 1 --2- 172.21.15.114:0/3980577236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d865d80 0x562b5dc53000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f64ec005d00 tx=0x7f64ec005c90 comp rx=0 tx=0).stop 2024-05-27T07:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.241+0000 7f650de38740 1 -- 172.21.15.114:0/3980577236 shutdown_connections 2024-05-27T07:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.242+0000 7f650de38740 1 --2- 172.21.15.114:0/3980577236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d8864d0 0x562b5dc4c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.242+0000 7f650de38740 1 --2- 172.21.15.114:0/3980577236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d87e2f0 0x562b5dc54720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.242+0000 7f650de38740 1 --2- 172.21.15.114:0/3980577236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d865d80 0x562b5dc53000 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.242+0000 7f650de38740 1 -- 172.21.15.114:0/3980577236 >> 172.21.15.114:0/3980577236 conn(0x562b5d89b300 msgr2=0x562b5dc54010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.243+0000 7f650de38740 1 -- 172.21.15.114:0/3980577236 shutdown_connections 2024-05-27T07:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.243+0000 7f650de38740 1 -- 172.21.15.114:0/3980577236 wait complete. 2024-05-27T07:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.244+0000 7f650de38740 1 -- start start 2024-05-27T07:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.244+0000 7f650de38740 1 Processor -- start 2024-05-27T07:47:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.244+0000 7f650de38740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d865d80 0x562b5dc559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.244+0000 7f650de38740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d87e2f0 0x562b5dc55ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.244+0000 7f650de38740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d8864d0 0x562b5dccbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.245+0000 7f650de38740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f64f8152d30 con 0x562b5d865d80 2024-05-27T07:47:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.245+0000 7f650de38740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f64f80a4bc0 con 0x562b5d8864d0 2024-05-27T07:47:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.245+0000 7f650de38740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64f8152080 con 0x562b5d87e2f0 2024-05-27T07:47:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.245+0000 7f650de38740 10 client.0 _ll_register_callbacks cb 0x7f64f400b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.246+0000 7f64fffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d8864d0 0x562b5dccbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.246+0000 7f64fffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d865d80 0x562b5dc559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.246+0000 7f64fffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d865d80 0x562b5dc559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34398/0 (socket says 172.21.15.114:34398) 2024-05-27T07:47:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.246+0000 7f64fffff640 1 -- 172.21.15.114:0/1279717684 learned_addr learned my addr 172.21.15.114:0/1279717684 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.246+0000 7f64fffff640 1 -- 172.21.15.114:0/1279717684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d87e2f0 msgr2=0x562b5dc55ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:47:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.246+0000 7f64fffff640 1 --2- 172.21.15.114:0/1279717684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d87e2f0 0x562b5dc55ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.246+0000 7f64fffff640 1 -- 172.21.15.114:0/1279717684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d8864d0 msgr2=0x562b5dccbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.246+0000 7f64fffff640 1 --2- 172.21.15.114:0/1279717684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d8864d0 0x562b5dccbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.246+0000 7f64fffff640 1 -- 172.21.15.114:0/1279717684 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b5dc56720 con 0x562b5d865d80 2024-05-27T07:47:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.247+0000 7f64fffff640 1 --2- 172.21.15.114:0/1279717684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d865d80 0x562b5dc559b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f64ec02a9f0 tx=0x7f64ec000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b34ff79fc48e1a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.247+0000 7f64dffff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.249+0000 7f64dffff640 1 -- 172.21.15.114:0/1279717684 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec035050 con 0x562b5d865d80 2024-05-27T07:47:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.249+0000 7f64dffff640 1 -- 172.21.15.114:0/1279717684 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9642+0+0 (secure 0 0 0) 0x7f64ec03d860 con 0x562b5d865d80 2024-05-27T07:47:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.249+0000 7f650de38740 1 -- 172.21.15.114:0/1279717684 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64ec042040 con 0x562b5d865d80 2024-05-27T07:47:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.250+0000 7f64dffff640 1 -- 172.21.15.114:0/1279717684 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f64ec041040 con 0x562b5d865d80 2024-05-27T07:47:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.250+0000 7f650de38740 10 client.17133 fetch_fsmap learned FSMap version 43 2024-05-27T07:47:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.250+0000 7f650de38740 1 -- 172.21.15.114:0/1279717684 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562b5dccd040 con 0x562b5d865d80 2024-05-27T07:47:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.251+0000 7f64dffff640 1 -- 172.21.15.114:0/1279717684 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64ec042040 con 0x562b5d865d80 2024-05-27T07:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.251+0000 7f64dffff640 1 -- 172.21.15.114:0/1279717684 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 43) ==== 7725+0+0 (secure 0 0 0) 0x7f64ec06c480 con 0x562b5d865d80 2024-05-27T07:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.251+0000 7f650de38740 10 client.17133 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T07:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.251+0000 7f650de38740 10 client.17133 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.251+0000 7f650de38740 20 client.17133 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.252+0000 7f650de38740 1 --2- 172.21.15.114:0/1279717684 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b5d8a71b0 0x562b5dccf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.252+0000 7f650de38740 4 client.17133 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.252+0000 7f650de38740 1 -- 172.21.15.114:0/1279717684 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562b5dbb6360 con 0x562b5d8a71b0 2024-05-27T07:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.252+0000 7f6504d20640 1 --2- 172.21.15.114:0/1279717684 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b5d8a71b0 0x562b5dccf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.253+0000 7f64dffff640 10 client.17133 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.253+0000 7f6504d20640 1 --2- 172.21.15.114:0/1279717684 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b5d8a71b0 0x562b5dccf050 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e245cef72119f2a2 server_cookie=70d7d25c0f099e96 in_seq=0 out_seq=0 2024-05-27T07:47:04.265 INFO:tasks.workunit.client.0.smithi114.stdout: 1 735117 11.80 MB/sec execute 196 sec latency 668.426 ms 2024-05-27T07:47:04.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.685+0000 7f64dffff640 1 -- 172.21.15.114:0/1279717684 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562b5dbb6360 con 0x562b5d8a71b0 2024-05-27T07:47:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.685+0000 7f64dffff640 10 client.17133 handle_command_reply: tid=0 2024-05-27T07:47:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.685+0000 7f650de38740 1 client.17133 shutdown 2024-05-27T07:47:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.685+0000 7f650de38740 1 -- 172.21.15.114:0/1279717684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d865d80 msgr2=0x562b5dc559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.685+0000 7f650de38740 1 --2- 172.21.15.114:0/1279717684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d865d80 0x562b5dc559b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f64ec02a9f0 tx=0x7f64ec000bc0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.685+0000 7f650de38740 1 -- 172.21.15.114:0/1279717684 shutdown_connections 2024-05-27T07:47:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.685+0000 7f650de38740 1 --2- 172.21.15.114:0/1279717684 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562b5d8a71b0 0x562b5dccf050 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.685+0000 7f650de38740 1 --2- 172.21.15.114:0/1279717684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562b5d8864d0 0x562b5dccbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.685+0000 7f650de38740 1 --2- 172.21.15.114:0/1279717684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b5d87e2f0 0x562b5dc55ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.685+0000 7f650de38740 1 --2- 172.21.15.114:0/1279717684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562b5d865d80 0x562b5dc559b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.685+0000 7f650de38740 1 -- 172.21.15.114:0/1279717684 >> 172.21.15.114:0/1279717684 conn(0x562b5d89b300 msgr2=0x562b5dc48320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.685+0000 7f64dffff640 0 client.17133 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.686+0000 7f650de38740 1 -- 172.21.15.114:0/1279717684 shutdown_connections 2024-05-27T07:47:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.686+0000 7f650de38740 1 -- 172.21.15.114:0/1279717684 wait complete. 2024-05-27T07:47:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.686+0000 7f650de38740 20 client.17133 trim_cache size 0 max 16384 2024-05-27T07:47:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:47:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:47:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:47:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.687+0000 7f6505d22640 1 -- 172.21.15.114:0/3631119080 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64f00b2b30 msgr2=0x7f64f00b5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.687+0000 7f6505d22640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64f00b2b30 0x7f64f00b5040 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f650014c100 tx=0x7f6500077040 comp rx=0 tx=0).stop 2024-05-27T07:47:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.687+0000 7f6505d22640 1 -- 172.21.15.114:0/3631119080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f809ad30 msgr2=0x7f64f8133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.687+0000 7f6505d22640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f809ad30 0x7f64f8133b90 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f650002e1b0 tx=0x7f650007a4b0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.688+0000 7f6505d22640 1 -- 172.21.15.114:0/3631119080 shutdown_connections 2024-05-27T07:47:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.688+0000 7f6505d22640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64f00b2b30 0x7f64f00b5040 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.688+0000 7f6505d22640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f809ad30 0x7f64f8133b90 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.688+0000 7f6505d22640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64f809a390 0x7f64f8139f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.688+0000 7f6505d22640 1 --2- 172.21.15.114:0/3631119080 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64f8095940 0x7f64f81399c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.688+0000 7f6505d22640 1 -- 172.21.15.114:0/3631119080 >> 172.21.15.114:0/3631119080 conn(0x7f64f800aff0 msgr2=0x7f64f8098360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.688+0000 7f6505d22640 1 -- 172.21.15.114:0/3631119080 shutdown_connections 2024-05-27T07:47:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:04.688+0000 7f6505d22640 1 -- 172.21.15.114:0/3631119080 wait complete. 2024-05-27T07:47:04.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:04 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:47:04.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:04 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:47:04.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:04 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:47:04.728 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:04 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:47:04.728 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:04 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Consumed 2.531s CPU time. 2024-05-27T07:47:04.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:04 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:47:04.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:04 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:47:04.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:04 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:47:04.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:04 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:47:04.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:04 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:47:04.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:04 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:47:05.265 INFO:tasks.workunit.client.0.smithi114.stdout: 1 735325 11.74 MB/sec execute 197 sec latency 159.648 ms 2024-05-27T07:47:05.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:05 smithi163 ceph-mon[36451]: pgmap v236: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 407 op/s 2024-05-27T07:47:05.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:05 smithi186 ceph-mon[36647]: pgmap v236: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 407 op/s 2024-05-27T07:47:05.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:05 smithi114 ceph-mon[32038]: pgmap v236: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 407 op/s 2024-05-27T07:47:06.265 INFO:tasks.workunit.client.0.smithi114.stdout: 1 735772 11.69 MB/sec execute 198 sec latency 65.974 ms 2024-05-27T07:47:07.265 INFO:tasks.workunit.client.0.smithi114.stdout: 1 736599 11.65 MB/sec execute 199 sec latency 42.941 ms 2024-05-27T07:47:07.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:07 smithi163 ceph-mon[36451]: pgmap v237: 129 pgs: 15 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 83 active+clean; 58 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 363 op/s 2024-05-27T07:47:07.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:07 smithi186 ceph-mon[36647]: pgmap v237: 129 pgs: 15 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 83 active+clean; 58 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 363 op/s 2024-05-27T07:47:07.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:07 smithi114 ceph-mon[32038]: pgmap v237: 129 pgs: 15 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 83 active+clean; 58 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 363 op/s 2024-05-27T07:47:08.265 INFO:tasks.workunit.client.0.smithi114.stdout: 1 738692 11.66 MB/sec execute 200 sec latency 24.538 ms 2024-05-27T07:47:09.265 INFO:tasks.workunit.client.0.smithi114.stdout: 1 740365 11.63 MB/sec execute 201 sec latency 16.731 ms 2024-05-27T07:47:09.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:09 smithi163 ceph-mon[36451]: pgmap v238: 129 pgs: 15 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 83 active+clean; 58 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 363 op/s 2024-05-27T07:47:09.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:09 smithi186 ceph-mon[36647]: pgmap v238: 129 pgs: 15 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 83 active+clean; 58 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 363 op/s 2024-05-27T07:47:09.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:09 smithi114 ceph-mon[32038]: pgmap v238: 129 pgs: 15 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 83 active+clean; 58 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 363 op/s 2024-05-27T07:47:10.265 INFO:tasks.workunit.client.0.smithi114.stdout: 1 742814 11.64 MB/sec execute 202 sec latency 17.037 ms 2024-05-27T07:47:11.265 INFO:tasks.workunit.client.0.smithi114.stdout: 1 744629 11.62 MB/sec execute 203 sec latency 56.780 ms 2024-05-27T07:47:11.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:11 smithi163 ceph-mon[36451]: pgmap v239: 129 pgs: 9 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 101 active+clean; 62 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 2.6 MiB/s wr, 313 op/s 2024-05-27T07:47:11.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:11 smithi186 ceph-mon[36647]: pgmap v239: 129 pgs: 9 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 101 active+clean; 62 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 2.6 MiB/s wr, 313 op/s 2024-05-27T07:47:11.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:11 smithi114 ceph-mon[32038]: pgmap v239: 129 pgs: 9 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 101 active+clean; 62 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 2.6 MiB/s wr, 313 op/s 2024-05-27T07:47:12.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 746428 11.62 MB/sec execute 204 sec latency 36.673 ms 2024-05-27T07:47:12.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:47:12.975 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:47:13.265 INFO:tasks.workunit.client.0.smithi114.stdout: 1 748872 11.63 MB/sec execute 205 sec latency 27.598 ms 2024-05-27T07:47:13.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:13 smithi163 ceph-mon[36451]: pgmap v240: 129 pgs: 129 active+clean; 65 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 2.4 MiB/s wr, 312 op/s 2024-05-27T07:47:13.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:13 smithi186 ceph-mon[36647]: pgmap v240: 129 pgs: 129 active+clean; 65 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 2.4 MiB/s wr, 312 op/s 2024-05-27T07:47:13.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:13 smithi114 ceph-mon[32038]: pgmap v240: 129 pgs: 129 active+clean; 65 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 2.4 MiB/s wr, 312 op/s 2024-05-27T07:47:14.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 750738 11.63 MB/sec execute 206 sec latency 28.800 ms 2024-05-27T07:47:14.869 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:14 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:47:14.869 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:14 smithi186 systemd[1]: Stopped Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:47:14.869 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:14 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Consumed 2.531s CPU time. 2024-05-27T07:47:14.870 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:14 smithi186 systemd[1]: Starting Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:47:14.870 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:14 smithi186 podman[73574]: 2024-05-27 07:47:14.614204341 +0000 UTC m=+0.020588319 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:47:14.870 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:14 smithi186 podman[73574]: 2024-05-27 07:47:14.729939006 +0000 UTC m=+0.136322971 container create 5c6be51a9ce6db53441d7713baf8a9af489096b233bd0805dd72cc1859975ce8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:47:15.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 753035 11.64 MB/sec execute 207 sec latency 22.387 ms 2024-05-27T07:47:15.269 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:14 smithi186 podman[73574]: 2024-05-27 07:47:14.991881447 +0000 UTC m=+0.398265425 container init 5c6be51a9ce6db53441d7713baf8a9af489096b233bd0805dd72cc1859975ce8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:47:15.270 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:14 smithi186 podman[73574]: 2024-05-27 07:47:14.995220301 +0000 UTC m=+0.401604280 container start 5c6be51a9ce6db53441d7713baf8a9af489096b233bd0805dd72cc1859975ce8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-27T07:47:15.270 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-mds[73593]: -- 172.21.15.186:0/1635620218 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e756adba40 con 0x55e756b23180 2024-05-27T07:47:15.270 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: starting mds.f at 2024-05-27T07:47:15.270 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 bash[73574]: 5c6be51a9ce6db53441d7713baf8a9af489096b233bd0805dd72cc1859975ce8 2024-05-27T07:47:15.270 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 systemd[1]: Started Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:47:15.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.440+0000 7f4995cca700 1 -- 172.21.15.114:0/2695681881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49901034b0 msgr2=0x7f49901038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:15.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.440+0000 7f4995cca700 1 --2- 172.21.15.114:0/2695681881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49901034b0 0x7f49901038c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f49800099c0 tx=0x7f4980009cd0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.440+0000 7f4995cca700 1 -- 172.21.15.114:0/2695681881 shutdown_connections 2024-05-27T07:47:15.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.440+0000 7f4995cca700 1 --2- 172.21.15.114:0/2695681881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4990108730 0x7f4990110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.440+0000 7f4995cca700 1 --2- 172.21.15.114:0/2695681881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49901034b0 0x7f49901038c0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.440+0000 7f4995cca700 1 --2- 172.21.15.114:0/2695681881 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4990107de0 0x7f49901081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.440+0000 7f4995cca700 1 -- 172.21.15.114:0/2695681881 >> 172.21.15.114:0/2695681881 conn(0x7f49900fedb0 msgr2=0x7f49901011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:15.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.440+0000 7f4995cca700 1 -- 172.21.15.114:0/2695681881 shutdown_connections 2024-05-27T07:47:15.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.440+0000 7f4995cca700 1 -- 172.21.15.114:0/2695681881 wait complete. 2024-05-27T07:47:15.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:15 smithi114 ceph-mon[32038]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T07:47:15.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:15 smithi114 ceph-mon[32038]: pgmap v242: 129 pgs: 129 active+clean; 65 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 2.4 MiB/s wr, 312 op/s 2024-05-27T07:47:15.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:15 smithi114 ceph-mon[32038]: daemon mds.f restarted 2024-05-27T07:47:15.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.441+0000 7f4995cca700 1 Processor -- start 2024-05-27T07:47:15.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.441+0000 7f4995cca700 1 -- start start 2024-05-27T07:47:15.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f4995cca700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49901034b0 0x7f499019ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f4995cca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4990107de0 0x7f499019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f4995cca700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4990108730 0x7f499019f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f4995cca700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4990113360 con 0x7f4990107de0 2024-05-27T07:47:15.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f4995cca700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f49901131e0 con 0x7f49901034b0 2024-05-27T07:47:15.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f4995cca700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f49901134e0 con 0x7f4990108730 2024-05-27T07:47:15.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f498effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4990107de0 0x7f499019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f498effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4990107de0 0x7f499019b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55064/0 (socket says 172.21.15.114:55064) 2024-05-27T07:47:15.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f498effd700 1 -- 172.21.15.114:0/3639311092 learned_addr learned my addr 172.21.15.114:0/3639311092 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:15.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f498ffff700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4990108730 0x7f499019f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f498f7fe700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49901034b0 0x7f499019ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f498effd700 1 -- 172.21.15.114:0/3639311092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4990108730 msgr2=0x7f499019f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:15.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f498effd700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4990108730 0x7f499019f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f498effd700 1 -- 172.21.15.114:0/3639311092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49901034b0 msgr2=0x7f499019ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:15.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f498effd700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49901034b0 0x7f499019ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f498effd700 1 -- 172.21.15.114:0/3639311092 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499019fc30 con 0x7f4990107de0 2024-05-27T07:47:15.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f498ffff700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4990108730 0x7f499019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:15.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.442+0000 7f498effd700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4990107de0 0x7f499019b360 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f498400c9e0 tx=0x7f498400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:15.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.443+0000 7f498cff9700 1 -- 172.21.15.114:0/3639311092 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4984007aa0 con 0x7f4990107de0 2024-05-27T07:47:15.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.443+0000 7f4995cca700 1 -- 172.21.15.114:0/3639311092 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f499019ff20 con 0x7f4990107de0 2024-05-27T07:47:15.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.443+0000 7f498cff9700 1 -- 172.21.15.114:0/3639311092 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4984007c40 con 0x7f4990107de0 2024-05-27T07:47:15.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.443+0000 7f498cff9700 1 -- 172.21.15.114:0/3639311092 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498400fe60 con 0x7f4990107de0 2024-05-27T07:47:15.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.443+0000 7f4995cca700 1 -- 172.21.15.114:0/3639311092 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49901a78a0 con 0x7f4990107de0 2024-05-27T07:47:15.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.446+0000 7f498cff9700 1 -- 172.21.15.114:0/3639311092 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f498400f450 con 0x7f4990107de0 2024-05-27T07:47:15.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.447+0000 7f498cff9700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f49780b29f0 0x7f49780b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.447+0000 7f498f7fe700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f49780b29f0 0x7f49780b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.447+0000 7f498cff9700 1 -- 172.21.15.114:0/3639311092 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f49840e28f0 con 0x7f4990107de0 2024-05-27T07:47:15.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.447+0000 7f49a46dfb80 1 Processor -- start 2024-05-27T07:47:15.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.447+0000 7f498f7fe700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f49780b29f0 0x7f49780b4f00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4980000c00 tx=0x7f498001a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:15.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.447+0000 7f49a46dfb80 1 -- start start 2024-05-27T07:47:15.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.447+0000 7f49a46dfb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d83dfe40 0x5587d8564b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.447+0000 7f49a46dfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d855ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.447+0000 7f49a46dfb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d855f120 0x5587d855f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f49a46dfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5587d848cd70 con 0x5587d8565070 2024-05-27T07:47:15.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f49a46dfb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5587d847c910 con 0x5587d83dfe40 2024-05-27T07:47:15.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f49a46dfb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5587d8480950 con 0x5587d855f120 2024-05-27T07:47:15.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f498effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d855ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f498effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d855ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55076/0 (socket says 172.21.15.114:55076) 2024-05-27T07:47:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f498effd700 1 -- 172.21.15.114:0/2810781835 learned_addr learned my addr 172.21.15.114:0/2810781835 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f498ffff700 1 --2- 172.21.15.114:0/2810781835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d855f120 0x5587d855f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f498f7fe700 1 --2- 172.21.15.114:0/2810781835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d83dfe40 0x5587d8564b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f498effd700 1 -- 172.21.15.114:0/2810781835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d855f120 msgr2=0x5587d855f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f498effd700 1 --2- 172.21.15.114:0/2810781835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d855f120 0x5587d855f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f498effd700 1 -- 172.21.15.114:0/2810781835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d83dfe40 msgr2=0x5587d8564b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f498effd700 1 --2- 172.21.15.114:0/2810781835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d83dfe40 0x5587d8564b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f498effd700 1 -- 172.21.15.114:0/2810781835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587d8425fb0 con 0x5587d8565070 2024-05-27T07:47:15.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f498f7fe700 1 --2- 172.21.15.114:0/2810781835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d83dfe40 0x5587d8564b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:47:15.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f498effd700 1 --2- 172.21.15.114:0/2810781835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d855ebe0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f49840a58a0 tx=0x7f49840a91f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697423f7e9fc021a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:15.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f4975ffb700 1 -- 172.21.15.114:0/2810781835 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49840a9f20 con 0x5587d8565070 2024-05-27T07:47:15.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f4975ffb700 1 -- 172.21.15.114:0/2810781835 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4984007df0 con 0x5587d8565070 2024-05-27T07:47:15.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.448+0000 7f4975ffb700 1 -- 172.21.15.114:0/2810781835 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49840a5c10 con 0x5587d8565070 2024-05-27T07:47:15.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.449+0000 7f49a46dfb80 1 -- 172.21.15.114:0/2810781835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 msgr2=0x5587d855ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:15.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.449+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/2810781835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d855ebe0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f49840a58a0 tx=0x7f49840a91f0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.449+0000 7f49a46dfb80 1 -- 172.21.15.114:0/2810781835 shutdown_connections 2024-05-27T07:47:15.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.449+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/2810781835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d855f120 0x5587d855f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.449+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/2810781835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d83dfe40 0x5587d8564b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.449+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/2810781835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d855ebe0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.449+0000 7f49a46dfb80 1 -- 172.21.15.114:0/2810781835 >> 172.21.15.114:0/2810781835 conn(0x5587d83e2770 msgr2=0x5587d8559d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:15.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.449+0000 7f49a46dfb80 1 -- 172.21.15.114:0/2810781835 shutdown_connections 2024-05-27T07:47:15.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.450+0000 7f49a46dfb80 1 -- 172.21.15.114:0/2810781835 wait complete. 2024-05-27T07:47:15.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.450+0000 7f49a46dfb80 1 -- start start 2024-05-27T07:47:15.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.450+0000 7f49a46dfb80 1 Processor -- start 2024-05-27T07:47:15.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f49a46dfb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d83dfe40 0x5587d8568920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f49a46dfb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d855f120 0x5587d8568e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f49a46dfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d85693a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f49a46dfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5587d8382fa0 con 0x5587d8565070 2024-05-27T07:47:15.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f49a46dfb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5587d83e4da0 con 0x5587d855f120 2024-05-27T07:47:15.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f49a46dfb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5587d856bb60 con 0x5587d83dfe40 2024-05-27T07:47:15.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f49a46dfb80 10 client.0 _ll_register_callbacks cb 0x5587d84135b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:15.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f498ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d85693a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f498ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d85693a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55088/0 (socket says 172.21.15.114:55088) 2024-05-27T07:47:15.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f498f7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d83dfe40 0x5587d8568920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f498ffff700 1 -- 172.21.15.114:0/2597916407 learned_addr learned my addr 172.21.15.114:0/2597916407 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:15.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f498effd700 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d855f120 0x5587d8568e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f498ffff700 1 -- 172.21.15.114:0/2597916407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d83dfe40 msgr2=0x5587d8568920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:15.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f498ffff700 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d83dfe40 0x5587d8568920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f498ffff700 1 -- 172.21.15.114:0/2597916407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d855f120 msgr2=0x5587d8568e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:15.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f498ffff700 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d855f120 0x5587d8568e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f498ffff700 1 -- 172.21.15.114:0/2597916407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587d83cc8d0 con 0x5587d8565070 2024-05-27T07:47:15.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f498f7fe700 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d83dfe40 0x5587d8568920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:15.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f498effd700 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d855f120 0x5587d8568e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:47:15.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f498ffff700 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d85693a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4988003d50 tx=0x7f498800e7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceb5da004f22aa2b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:15.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.451+0000 7f496effd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:15.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f496effd700 1 -- 172.21.15.114:0/2597916407 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49880092d0 con 0x5587d8565070 2024-05-27T07:47:15.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f496effd700 1 -- 172.21.15.114:0/2597916407 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f4988007500 con 0x5587d8565070 2024-05-27T07:47:15.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f49a46dfb80 1 -- 172.21.15.114:0/2597916407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4984007df0 con 0x5587d8565070 2024-05-27T07:47:15.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f496effd700 1 -- 172.21.15.114:0/2597916407 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f4988027050 con 0x5587d8565070 2024-05-27T07:47:15.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f49a46dfb80 10 client.17151 fetch_fsmap learned FSMap version 45 2024-05-27T07:47:15.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f49a46dfb80 1 -- 172.21.15.114:0/2597916407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5587d8436760 con 0x5587d8565070 2024-05-27T07:47:15.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f496effd700 1 -- 172.21.15.114:0/2597916407 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4984007df0 con 0x5587d8565070 2024-05-27T07:47:15.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f496effd700 1 -- 172.21.15.114:0/2597916407 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 45) ==== 7725+0+0 (secure 0 0 0) 0x7f498801e670 con 0x5587d8565070 2024-05-27T07:47:15.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f49a46dfb80 10 client.17151 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:47:15.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f49a46dfb80 10 client.17151 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:15.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f49a46dfb80 20 client.17151 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:15.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5587d857d030 0x5587d857f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f49a46dfb80 4 client.17151 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:15.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.452+0000 7f49a46dfb80 1 -- 172.21.15.114:0/2597916407 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587d848ecc0 con 0x5587d857d030 2024-05-27T07:47:15.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.453+0000 7f498f7fe700 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5587d857d030 0x5587d857f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.453+0000 7f496effd700 10 client.17151 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:15.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.453+0000 7f498f7fe700 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5587d857d030 0x5587d857f450 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3ce8b69cb0ea255 server_cookie=cf1a1acbb945463f in_seq=0 out_seq=0 2024-05-27T07:47:15.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.454+0000 7f496effd700 1 -- 172.21.15.114:0/2597916407 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5587d848ecc0 con 0x5587d857d030 2024-05-27T07:47:15.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.454+0000 7f496effd700 10 client.17151 handle_command_reply: tid=0 2024-05-27T07:47:15.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.455+0000 7f49a46dfb80 1 client.17151 shutdown 2024-05-27T07:47:15.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.455+0000 7f49a46dfb80 1 -- 172.21.15.114:0/2597916407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 msgr2=0x5587d85693a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:15.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.455+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d85693a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4988003d50 tx=0x7f498800e7d0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.455+0000 7f49a46dfb80 1 -- 172.21.15.114:0/2597916407 shutdown_connections 2024-05-27T07:47:15.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.455+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d85693a0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.455+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d83dfe40 0x5587d8568920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.455+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5587d857d030 0x5587d857f450 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.455+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/2597916407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d855f120 0x5587d8568e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.455+0000 7f49a46dfb80 1 -- 172.21.15.114:0/2597916407 >> 172.21.15.114:0/2597916407 conn(0x5587d83e2770 msgr2=0x5587d855dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:15.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.455+0000 7f496effd700 0 client.17151 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:15.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.455+0000 7f49a46dfb80 1 -- 172.21.15.114:0/2597916407 shutdown_connections 2024-05-27T07:47:15.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.455+0000 7f49a46dfb80 1 -- 172.21.15.114:0/2597916407 wait complete. 2024-05-27T07:47:15.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.455+0000 7f49a46dfb80 20 client.17151 trim_cache size 0 max 16384 2024-05-27T07:47:15.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.478+0000 7f49a46dfb80 1 Processor -- start 2024-05-27T07:47:15.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.478+0000 7f49a46dfb80 1 -- start start 2024-05-27T07:47:15.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.478+0000 7f49a46dfb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d83dfe40 0x5587d8568e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.478+0000 7f49a46dfb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d855f120 0x5587d85693d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.478+0000 7f49a46dfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d8563880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.478+0000 7f49a46dfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f49901af250 con 0x5587d8565070 2024-05-27T07:47:15.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.478+0000 7f49a46dfb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f49901ac300 con 0x5587d83dfe40 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.478+0000 7f49a46dfb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f49901ae4b0 con 0x5587d855f120 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.478+0000 7f498ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d8563880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.478+0000 7f498ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d8563880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55090/0 (socket says 172.21.15.114:55090) 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.478+0000 7f498ffff700 1 -- 172.21.15.114:0/415817433 learned_addr learned my addr 172.21.15.114:0/415817433 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.478+0000 7f498effd700 1 --2- 172.21.15.114:0/415817433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d855f120 0x5587d85693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.478+0000 7f498f7fe700 1 --2- 172.21.15.114:0/415817433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d83dfe40 0x5587d8568e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.479+0000 7f498ffff700 1 -- 172.21.15.114:0/415817433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d855f120 msgr2=0x5587d85693d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.479+0000 7f498ffff700 1 --2- 172.21.15.114:0/415817433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d855f120 0x5587d85693d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.479+0000 7f498ffff700 1 -- 172.21.15.114:0/415817433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d83dfe40 msgr2=0x5587d8568e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.479+0000 7f498ffff700 1 --2- 172.21.15.114:0/415817433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d83dfe40 0x5587d8568e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.479+0000 7f498ffff700 1 -- 172.21.15.114:0/415817433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587d8566e90 con 0x5587d8565070 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.479+0000 7f498ffff700 1 --2- 172.21.15.114:0/415817433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d8563880 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4988000f80 tx=0x7f498800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1edcd0f5ec9000 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.479+0000 7f49757fa700 1 -- 172.21.15.114:0/415817433 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498802f030 con 0x5587d8565070 2024-05-27T07:47:15.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.479+0000 7f49757fa700 1 -- 172.21.15.114:0/415817433 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49880090d0 con 0x5587d8565070 2024-05-27T07:47:15.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.479+0000 7f49757fa700 1 -- 172.21.15.114:0/415817433 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4988004de0 con 0x5587d8565070 2024-05-27T07:47:15.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.480+0000 7f49a46dfb80 1 -- 172.21.15.114:0/415817433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 msgr2=0x5587d8563880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:15.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.480+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/415817433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d8563880 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4988000f80 tx=0x7f498800ec50 comp rx=0 tx=0).stop 2024-05-27T07:47:15.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.480+0000 7f49a46dfb80 1 -- 172.21.15.114:0/415817433 shutdown_connections 2024-05-27T07:47:15.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.480+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/415817433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d8565070 0x5587d8563880 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.480+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/415817433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d83dfe40 0x5587d8568e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.480+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/415817433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d855f120 0x5587d85693d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.480+0000 7f49a46dfb80 1 -- 172.21.15.114:0/415817433 >> 172.21.15.114:0/415817433 conn(0x5587d83e2770 msgr2=0x5587d8567210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:15.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.480+0000 7f49a46dfb80 1 -- 172.21.15.114:0/415817433 shutdown_connections 2024-05-27T07:47:15.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.480+0000 7f49a46dfb80 1 -- 172.21.15.114:0/415817433 wait complete. 2024-05-27T07:47:15.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.480+0000 7f49a46dfb80 1 -- start start 2024-05-27T07:47:15.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.481+0000 7f49a46dfb80 1 Processor -- start 2024-05-27T07:47:15.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.481+0000 7f49a46dfb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d83dfe40 0x5587d8571110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.481+0000 7f49a46dfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d855f120 0x5587d8571650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.481+0000 7f49a46dfb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d8565070 0x5587d8584310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.481+0000 7f49a46dfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f49901adde0 con 0x5587d855f120 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.481+0000 7f49a46dfb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f49901ac970 con 0x5587d8565070 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.481+0000 7f49a46dfb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f49901ad040 con 0x5587d83dfe40 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.481+0000 7f49a46dfb80 10 client.0 _ll_register_callbacks cb 0x5587d84135b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.481+0000 7f498effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d855f120 0x5587d8571650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.481+0000 7f498effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d855f120 0x5587d8571650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55100/0 (socket says 172.21.15.114:55100) 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.481+0000 7f498effd700 1 -- 172.21.15.114:0/1938414757 learned_addr learned my addr 172.21.15.114:0/1938414757 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.481+0000 7f498f7fe700 1 --2- 172.21.15.114:0/1938414757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d83dfe40 0x5587d8571110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.481+0000 7f498ffff700 1 --2- 172.21.15.114:0/1938414757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d8565070 0x5587d8584310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f498effd700 1 -- 172.21.15.114:0/1938414757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d83dfe40 msgr2=0x5587d8571110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f498effd700 1 --2- 172.21.15.114:0/1938414757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d83dfe40 0x5587d8571110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f498effd700 1 -- 172.21.15.114:0/1938414757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d8565070 msgr2=0x5587d8584310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f498effd700 1 --2- 172.21.15.114:0/1938414757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d8565070 0x5587d8584310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f498effd700 1 -- 172.21.15.114:0/1938414757 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587d8584a50 con 0x5587d855f120 2024-05-27T07:47:15.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f498effd700 1 --2- 172.21.15.114:0/1938414757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d855f120 0x5587d8571650 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f49840a92e0 tx=0x7f498400f970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46055c24fbb4e01e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:15.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f496effd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:15.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f496effd700 1 -- 172.21.15.114:0/1938414757 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4984007df0 con 0x5587d855f120 2024-05-27T07:47:15.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f496effd700 1 -- 172.21.15.114:0/1938414757 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f49840a9c90 con 0x5587d855f120 2024-05-27T07:47:15.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f49a46dfb80 1 -- 172.21.15.114:0/1938414757 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49880090d0 con 0x5587d855f120 2024-05-27T07:47:15.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f496effd700 1 -- 172.21.15.114:0/1938414757 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f49840a9f20 con 0x5587d855f120 2024-05-27T07:47:15.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f49a46dfb80 10 client.17157 fetch_fsmap learned FSMap version 45 2024-05-27T07:47:15.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f49a46dfb80 1 -- 172.21.15.114:0/1938414757 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5587d85e75b0 con 0x5587d855f120 2024-05-27T07:47:15.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f496effd700 1 -- 172.21.15.114:0/1938414757 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49880090d0 con 0x5587d855f120 2024-05-27T07:47:15.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f496effd700 1 -- 172.21.15.114:0/1938414757 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 45) ==== 7725+0+0 (secure 0 0 0) 0x7f49840b2280 con 0x5587d855f120 2024-05-27T07:47:15.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f49a46dfb80 10 client.17157 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:47:15.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f49a46dfb80 10 client.17157 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:15.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.482+0000 7f49a46dfb80 20 client.17157 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:15.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.483+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/1938414757 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5587d85e7890 0x5587d85e9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:15.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.483+0000 7f49a46dfb80 4 client.17157 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:47:15.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.483+0000 7f49a46dfb80 1 -- 172.21.15.114:0/1938414757 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5587d84884a0 con 0x5587d85e7890 2024-05-27T07:47:15.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.483+0000 7f498f7fe700 1 --2- 172.21.15.114:0/1938414757 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5587d85e7890 0x5587d85e9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:15.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.483+0000 7f496effd700 10 client.17157 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:15.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.483+0000 7f498f7fe700 1 --2- 172.21.15.114:0/1938414757 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5587d85e7890 0x5587d85e9cb0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e88e0f7c6b00f02b server_cookie=cead13cfcf2e8e9d in_seq=0 out_seq=0 2024-05-27T07:47:15.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f496effd700 1 -- 172.21.15.114:0/1938414757 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5587d84884a0 con 0x5587d85e7890 2024-05-27T07:47:15.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f496effd700 10 client.17157 handle_command_reply: tid=0 2024-05-27T07:47:15.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f49a46dfb80 1 client.17157 shutdown 2024-05-27T07:47:15.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f49a46dfb80 1 -- 172.21.15.114:0/1938414757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d855f120 msgr2=0x5587d8571650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:15.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/1938414757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d855f120 0x5587d8571650 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f49840a92e0 tx=0x7f498400f970 comp rx=0 tx=0).stop 2024-05-27T07:47:15.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f49a46dfb80 1 -- 172.21.15.114:0/1938414757 shutdown_connections 2024-05-27T07:47:15.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/1938414757 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5587d85e7890 0x5587d85e9cb0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/1938414757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587d8565070 0x5587d8584310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/1938414757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587d83dfe40 0x5587d8571110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f49a46dfb80 1 --2- 172.21.15.114:0/1938414757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587d855f120 0x5587d8571650 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f49a46dfb80 1 -- 172.21.15.114:0/1938414757 >> 172.21.15.114:0/1938414757 conn(0x5587d83e2770 msgr2=0x5587d8563880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:15.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f496effd700 0 client.17157 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:15.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f49a46dfb80 1 -- 172.21.15.114:0/1938414757 shutdown_connections 2024-05-27T07:47:15.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f49a46dfb80 1 -- 172.21.15.114:0/1938414757 wait complete. 2024-05-27T07:47:15.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.485+0000 7f49a46dfb80 20 client.17157 trim_cache size 0 max 16384 2024-05-27T07:47:15.489 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:47:15.489 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:47:15.490 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:47:15.490 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:47:15.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.486+0000 7f4995cca700 1 -- 172.21.15.114:0/3639311092 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f49780b29f0 msgr2=0x7f49780b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:15.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.486+0000 7f4995cca700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f49780b29f0 0x7f49780b4f00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4980000c00 tx=0x7f498001a040 comp rx=0 tx=0).stop 2024-05-27T07:47:15.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.486+0000 7f4995cca700 1 -- 172.21.15.114:0/3639311092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4990107de0 msgr2=0x7f499019b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:15.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.486+0000 7f4995cca700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4990107de0 0x7f499019b360 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f498400c9e0 tx=0x7f498400ccf0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.486+0000 7f4995cca700 1 -- 172.21.15.114:0/3639311092 shutdown_connections 2024-05-27T07:47:15.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.486+0000 7f4995cca700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4990108730 0x7f499019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.486+0000 7f4995cca700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f49780b29f0 0x7f49780b4f00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.486+0000 7f4995cca700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49901034b0 0x7f499019ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.486+0000 7f4995cca700 1 --2- 172.21.15.114:0/3639311092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4990107de0 0x7f499019b360 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:15.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.486+0000 7f4995cca700 1 -- 172.21.15.114:0/3639311092 >> 172.21.15.114:0/3639311092 conn(0x7f49900fedb0 msgr2=0x7f49901011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:15.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.486+0000 7f4995cca700 1 -- 172.21.15.114:0/3639311092 shutdown_connections 2024-05-27T07:47:15.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:15.486+0000 7f4995cca700 1 -- 172.21.15.114:0/3639311092 wait complete. 2024-05-27T07:47:15.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:15 smithi163 ceph-mon[36451]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T07:47:15.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:15 smithi163 ceph-mon[36451]: pgmap v242: 129 pgs: 129 active+clean; 65 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 2.4 MiB/s wr, 312 op/s 2024-05-27T07:47:15.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:15 smithi163 ceph-mon[36451]: daemon mds.f restarted 2024-05-27T07:47:15.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:15 smithi186 ceph-mon[36647]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T07:47:15.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:15 smithi186 ceph-mon[36647]: pgmap v242: 129 pgs: 129 active+clean; 65 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 2.4 MiB/s wr, 312 op/s 2024-05-27T07:47:15.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:15 smithi186 ceph-mon[36647]: daemon mds.f restarted 2024-05-27T07:47:15.728 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd8aad6c700 time 2024-05-27T07:47:15.326852+0000 2024-05-27T07:47:15.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:47:15.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:15.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd8bd5b1ba6] 2024-05-27T07:47:15.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8bd5b1dd1] 2024-05-27T07:47:15.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e754ea323d] 2024-05-27T07:47:15.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e754e8e971] 2024-05-27T07:47:15.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e754e8cfa0] 2024-05-27T07:47:15.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e754e8d651] 2024-05-27T07:47:15.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e754e22347] 2024-05-27T07:47:15.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e754b1e87e] 2024-05-27T07:47:15.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e754b23f2f] 2024-05-27T07:47:15.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e754b2e969] 2024-05-27T07:47:15.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e754ec748c] 2024-05-27T07:47:15.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e754ed09cb] 2024-05-27T07:47:15.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 13: (Thread::entry_wrapper()+0x43) [0x7fd8bd58072d] 2024-05-27T07:47:15.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd8bd580749] 2024-05-27T07:47:15.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd8bc1241ca] 2024-05-27T07:47:15.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 16: clone() 2024-05-27T07:47:15.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.328+0000 7fd8aad6c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd8aad6c700 time 2024-05-27T07:47:15.326852+0000 2024-05-27T07:47:15.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:47:15.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.730 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd8bd5b1ba6] 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8bd5b1dd1] 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e754ea323d] 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e754e8e971] 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e754e8cfa0] 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e754e8d651] 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e754e22347] 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e754b1e87e] 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e754b23f2f] 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e754b2e969] 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e754ec748c] 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e754ed09cb] 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 13: (Thread::entry_wrapper()+0x43) [0x7fd8bd58072d] 2024-05-27T07:47:15.731 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd8bd580749] 2024-05-27T07:47:15.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd8bc1241ca] 2024-05-27T07:47:15.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 16: clone() 2024-05-27T07:47:15.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: *** Caught signal (Aborted) ** 2024-05-27T07:47:15.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: in thread 7fd8aad6c700 thread_name:md_log_replay 2024-05-27T07:47:15.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:15.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd8bc12ed20] 2024-05-27T07:47:15.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2: gsignal() 2024-05-27T07:47:15.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 3: abort() 2024-05-27T07:47:15.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd8bd5b1ced] 2024-05-27T07:47:15.732 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8bd5b1dd1] 2024-05-27T07:47:15.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e754ea323d] 2024-05-27T07:47:15.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e754e8e971] 2024-05-27T07:47:15.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e754e8cfa0] 2024-05-27T07:47:15.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e754e8d651] 2024-05-27T07:47:15.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e754e22347] 2024-05-27T07:47:15.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e754b1e87e] 2024-05-27T07:47:15.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e754b23f2f] 2024-05-27T07:47:15.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e754b2e969] 2024-05-27T07:47:15.733 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e754ec748c] 2024-05-27T07:47:15.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e754ed09cb] 2024-05-27T07:47:15.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 16: (Thread::entry_wrapper()+0x43) [0x7fd8bd58072d] 2024-05-27T07:47:15.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd8bd580749] 2024-05-27T07:47:15.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd8bc1241ca] 2024-05-27T07:47:15.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 19: clone() 2024-05-27T07:47:15.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.330+0000 7fd8aad6c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:47:15.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: in thread 7fd8aad6c700 thread_name:md_log_replay 2024-05-27T07:47:15.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:15.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd8bc12ed20] 2024-05-27T07:47:15.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2: gsignal() 2024-05-27T07:47:15.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 3: abort() 2024-05-27T07:47:15.734 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd8bd5b1ced] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8bd5b1dd1] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e754ea323d] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e754e8e971] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e754e8cfa0] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e754e8d651] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e754e22347] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e754b1e87e] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e754b23f2f] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e754b2e969] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e754ec748c] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e754ed09cb] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 16: (Thread::entry_wrapper()+0x43) [0x7fd8bd58072d] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd8bd580749] 2024-05-27T07:47:15.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd8bc1241ca] 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 19: clone() 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: -1> 2024-05-27T07:47:15.328+0000 7fd8aad6c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd8aad6c700 time 2024-05-27T07:47:15.326852+0000 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd8bd5b1ba6] 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8bd5b1dd1] 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e754ea323d] 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e754e8e971] 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e754e8cfa0] 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e754e8d651] 2024-05-27T07:47:15.736 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e754e22347] 2024-05-27T07:47:15.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e754b1e87e] 2024-05-27T07:47:15.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e754b23f2f] 2024-05-27T07:47:15.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e754b2e969] 2024-05-27T07:47:15.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e754ec748c] 2024-05-27T07:47:15.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e754ed09cb] 2024-05-27T07:47:15.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 13: (Thread::entry_wrapper()+0x43) [0x7fd8bd58072d] 2024-05-27T07:47:15.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd8bd580749] 2024-05-27T07:47:15.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd8bc1241ca] 2024-05-27T07:47:15.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 16: clone() 2024-05-27T07:47:15.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 0> 2024-05-27T07:47:15.330+0000 7fd8aad6c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:47:15.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: in thread 7fd8aad6c700 thread_name:md_log_replay 2024-05-27T07:47:15.737 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:15.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd8bc12ed20] 2024-05-27T07:47:15.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2: gsignal() 2024-05-27T07:47:15.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 3: abort() 2024-05-27T07:47:15.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd8bd5b1ced] 2024-05-27T07:47:15.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8bd5b1dd1] 2024-05-27T07:47:15.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e754ea323d] 2024-05-27T07:47:15.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e754e8e971] 2024-05-27T07:47:15.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e754e8cfa0] 2024-05-27T07:47:15.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e754e8d651] 2024-05-27T07:47:15.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e754e22347] 2024-05-27T07:47:15.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e754b1e87e] 2024-05-27T07:47:15.738 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e754b23f2f] 2024-05-27T07:47:15.739 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e754b2e969] 2024-05-27T07:47:15.739 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e754ec748c] 2024-05-27T07:47:15.739 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e754ed09cb] 2024-05-27T07:47:15.739 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 16: (Thread::entry_wrapper()+0x43) [0x7fd8bd58072d] 2024-05-27T07:47:15.739 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd8bd580749] 2024-05-27T07:47:15.739 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd8bc1241ca] 2024-05-27T07:47:15.739 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 19: clone() 2024-05-27T07:47:15.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:47:15.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: -1> 2024-05-27T07:47:15.328+0000 7fd8aad6c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd8aad6c700 time 2024-05-27T07:47:15.326852+0000 2024-05-27T07:47:15.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:47:15.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:15.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd8bd5b1ba6] 2024-05-27T07:47:15.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8bd5b1dd1] 2024-05-27T07:47:15.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e754ea323d] 2024-05-27T07:47:15.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e754e8e971] 2024-05-27T07:47:15.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e754e8cfa0] 2024-05-27T07:47:15.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e754e8d651] 2024-05-27T07:47:15.740 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e754e22347] 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e754b1e87e] 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e754b23f2f] 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e754b2e969] 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e754ec748c] 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e754ed09cb] 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 13: (Thread::entry_wrapper()+0x43) [0x7fd8bd58072d] 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd8bd580749] 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd8bc1241ca] 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 16: clone() 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 0> 2024-05-27T07:47:15.330+0000 7fd8aad6c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: in thread 7fd8aad6c700 thread_name:md_log_replay 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.741 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd8bc12ed20] 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2: gsignal() 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 3: abort() 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd8bd5b1ced] 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8bd5b1dd1] 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e754ea323d] 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e754e8e971] 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e754e8cfa0] 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e754e8d651] 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e754e22347] 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e754b1e87e] 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e754b23f2f] 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e754b2e969] 2024-05-27T07:47:15.742 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e754ec748c] 2024-05-27T07:47:15.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e754ed09cb] 2024-05-27T07:47:15.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 16: (Thread::entry_wrapper()+0x43) [0x7fd8bd58072d] 2024-05-27T07:47:15.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd8bd580749] 2024-05-27T07:47:15.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd8bc1241ca] 2024-05-27T07:47:15.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 19: clone() 2024-05-27T07:47:15.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:47:15.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f[73589]: 2024-05-27T07:47:15.743 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:15 smithi186 podman[73624]: 2024-05-27 07:47:15.383261675 +0000 UTC m=+0.023569495 container died 5c6be51a9ce6db53441d7713baf8a9af489096b233bd0805dd72cc1859975ce8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T07:47:16.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 754860 11.61 MB/sec execute 208 sec latency 29.372 ms 2024-05-27T07:47:16.359 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:16 smithi186 ceph-mon[36647]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T07:47:16.359 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:16 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6836/298307750,v1:172.21.15.186:6837/298307750] up:boot 2024-05-27T07:47:16.359 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:16 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:47:16.359 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:16 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:47:16.359 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:16 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:47:16.360 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:16 smithi186 podman[73624]: 2024-05-27 07:47:16.074262337 +0000 UTC m=+0.714570157 container remove 5c6be51a9ce6db53441d7713baf8a9af489096b233bd0805dd72cc1859975ce8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-f, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-05-27T07:47:16.360 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:16 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:47:16.404 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:47:16.405 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:47:16.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:16 smithi114 ceph-mon[32038]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T07:47:16.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:16 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6836/298307750,v1:172.21.15.186:6837/298307750] up:boot 2024-05-27T07:47:16.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:16 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:47:16.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:16 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:47:16.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:16 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:47:16.717 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:47:16.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:16 smithi163 ceph-mon[36451]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T07:47:16.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:16 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6836/298307750,v1:172.21.15.186:6837/298307750] up:boot 2024-05-27T07:47:16.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:16 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-05-27T07:47:16.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:16 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:47:16.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:16 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-05-27T07:47:16.728 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:16 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:47:17.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 756693 11.62 MB/sec execute 209 sec latency 77.562 ms 2024-05-27T07:47:17.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:17 smithi114 ceph-mon[32038]: pgmap v244: 129 pgs: 129 active+clean; 72 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-05-27T07:47:17.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:17 smithi163 ceph-mon[36451]: pgmap v244: 129 pgs: 129 active+clean; 72 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-05-27T07:47:17.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:17 smithi186 ceph-mon[36647]: pgmap v244: 129 pgs: 129 active+clean; 72 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-05-27T07:47:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.798+0000 7fc2d7376640 1 -- 172.21.15.114:0/841463158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d010b7f0 msgr2=0x7fc2d010dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.798+0000 7fc2d7376640 1 --2- 172.21.15.114:0/841463158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d010b7f0 0x7fc2d010dc10 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4009900 tx=0x7fc2c402a410 comp rx=0 tx=0).stop 2024-05-27T07:47:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.798+0000 7fc2d7376640 1 -- 172.21.15.114:0/841463158 shutdown_connections 2024-05-27T07:47:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.798+0000 7fc2d7376640 1 --2- 172.21.15.114:0/841463158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d010e1e0 0x7fc2d01106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.798+0000 7fc2d7376640 1 --2- 172.21.15.114:0/841463158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d010b7f0 0x7fc2d010dc10 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.798+0000 7fc2d7376640 1 --2- 172.21.15.114:0/841463158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2d0104d90 0x7fc2d01051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.798+0000 7fc2d7376640 1 -- 172.21.15.114:0/841463158 >> 172.21.15.114:0/841463158 conn(0x7fc2d00782d0 msgr2=0x7fc2d0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.799+0000 7fc2d7376640 1 -- 172.21.15.114:0/841463158 shutdown_connections 2024-05-27T07:47:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.799+0000 7fc2d7376640 1 -- 172.21.15.114:0/841463158 wait complete. 2024-05-27T07:47:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.799+0000 7fc2d7376640 1 Processor -- start 2024-05-27T07:47:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.799+0000 7fc2d7376640 1 -- start start 2024-05-27T07:47:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.799+0000 7fc2d7376640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d0104d90 0x7fc2d01a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.799+0000 7fc2d7376640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2d010b7f0 0x7fc2d01a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.799+0000 7fc2d7376640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d010e1e0 0x7fc2d019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.799+0000 7fc2d7376640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0113e40 con 0x7fc2d010e1e0 2024-05-27T07:47:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.799+0000 7fc2d7376640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0113cc0 con 0x7fc2d010b7f0 2024-05-27T07:47:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.799+0000 7fc2d7376640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0113fc0 con 0x7fc2d0104d90 2024-05-27T07:47:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2d5913640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d010e1e0 0x7fc2d019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2d5112640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d0104d90 0x7fc2d01a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2d5913640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d010e1e0 0x7fc2d019a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55110/0 (socket says 172.21.15.114:55110) 2024-05-27T07:47:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2d5913640 1 -- 172.21.15.114:0/3946674528 learned_addr learned my addr 172.21.15.114:0/3946674528 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2d4911640 1 --2- 172.21.15.114:0/3946674528 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2d010b7f0 0x7fc2d01a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2d5913640 1 -- 172.21.15.114:0/3946674528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d0104d90 msgr2=0x7fc2d01a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2d5913640 1 --2- 172.21.15.114:0/3946674528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d0104d90 0x7fc2d01a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2d5913640 1 -- 172.21.15.114:0/3946674528 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2d010b7f0 msgr2=0x7fc2d01a0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2d5913640 1 --2- 172.21.15.114:0/3946674528 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2d010b7f0 0x7fc2d01a0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2d5913640 1 -- 172.21.15.114:0/3946674528 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d019add0 con 0x7fc2d010e1e0 2024-05-27T07:47:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2d5913640 1 --2- 172.21.15.114:0/3946674528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d010e1e0 0x7fc2d019a6a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800eb10 tx=0x7fc2c800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2c27fc640 1 -- 172.21.15.114:0/3946674528 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8004b20 con 0x7fc2d010e1e0 2024-05-27T07:47:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2d7376640 1 -- 172.21.15.114:0/3946674528 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d019b060 con 0x7fc2d010e1e0 2024-05-27T07:47:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2c27fc640 1 -- 172.21.15.114:0/3946674528 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2c8004cc0 con 0x7fc2d010e1e0 2024-05-27T07:47:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2d7376640 1 -- 172.21.15.114:0/3946674528 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d01a72c0 con 0x7fc2d010e1e0 2024-05-27T07:47:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.800+0000 7fc2c27fc640 1 -- 172.21.15.114:0/3946674528 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8010670 con 0x7fc2d010e1e0 2024-05-27T07:47:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.803+0000 7fc2de22c740 1 Processor -- start 2024-05-27T07:47:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.803+0000 7fc2c27fc640 1 -- 172.21.15.114:0/3946674528 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fc2c8010810 con 0x7fc2d010e1e0 2024-05-27T07:47:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.803+0000 7fc2de22c740 1 -- start start 2024-05-27T07:47:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2de22c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 0x56516da42260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2de22c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da427a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2de22c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 0x56516da42ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2de22c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56516d9b4c30 con 0x56516d6782f0 2024-05-27T07:47:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2de22c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56516d9a47d0 con 0x56516d6804d0 2024-05-27T07:47:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2de22c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56516d9a8810 con 0x56516d6a11b0 2024-05-27T07:47:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2d5112640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2d4911640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 0x56516da42260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2d4911640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 0x56516da42260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59666/0 (socket says 172.21.15.114:59666) 2024-05-27T07:47:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2d4911640 1 -- 172.21.15.114:0/2729301169 learned_addr learned my addr 172.21.15.114:0/2729301169 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2d4911640 1 --2- 172.21.15.114:0/2729301169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 0x56516da42ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2c27fc640 1 --2- 172.21.15.114:0/3946674528 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2bc0b2950 0x7fc2bc0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2d5112640 1 -- 172.21.15.114:0/2729301169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 msgr2=0x56516da42260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2d5112640 1 --2- 172.21.15.114:0/2729301169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 0x56516da42260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2d5112640 1 -- 172.21.15.114:0/2729301169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 msgr2=0x56516da42ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2d5112640 1 --2- 172.21.15.114:0/2729301169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 0x56516da42ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.804+0000 7fc2d5112640 1 -- 172.21.15.114:0/2729301169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56516d6b9210 con 0x56516d6782f0 2024-05-27T07:47:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.805+0000 7fc2c27fc640 1 -- 172.21.15.114:0/3946674528 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fc2c80e2f00 con 0x7fc2d010e1e0 2024-05-27T07:47:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.805+0000 7fc2d5913640 1 --2- 172.21.15.114:0/3946674528 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2bc0b2950 0x7fc2bc0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.805+0000 7fc2d5112640 1 --2- 172.21.15.114:0/2729301169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da427a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc009c20 tx=0x7fc2cc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c586c43f108ed2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.805+0000 7fc2bb7fe640 1 -- 172.21.15.114:0/2729301169 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc004050 con 0x56516d6782f0 2024-05-27T07:47:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.805+0000 7fc2bb7fe640 1 -- 172.21.15.114:0/2729301169 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2cc0054d0 con 0x56516d6782f0 2024-05-27T07:47:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.805+0000 7fc2bb7fe640 1 -- 172.21.15.114:0/2729301169 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc004d90 con 0x56516d6782f0 2024-05-27T07:47:17.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.806+0000 7fc2de22c740 1 -- 172.21.15.114:0/2729301169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 msgr2=0x56516da427a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:17.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.806+0000 7fc2de22c740 1 --2- 172.21.15.114:0/2729301169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da427a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc009c20 tx=0x7fc2cc011810 comp rx=0 tx=0).stop 2024-05-27T07:47:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.806+0000 7fc2d5913640 1 --2- 172.21.15.114:0/3946674528 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2bc0b2950 0x7fc2bc0b4e60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80a5eb0 tx=0x7fc2c800cee0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.806+0000 7fc2de22c740 1 -- 172.21.15.114:0/2729301169 shutdown_connections 2024-05-27T07:47:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.806+0000 7fc2de22c740 1 --2- 172.21.15.114:0/2729301169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 0x56516da42ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.806+0000 7fc2de22c740 1 --2- 172.21.15.114:0/2729301169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da427a0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.806+0000 7fc2de22c740 1 --2- 172.21.15.114:0/2729301169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 0x56516da42260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.806+0000 7fc2de22c740 1 -- 172.21.15.114:0/2729301169 >> 172.21.15.114:0/2729301169 conn(0x56516d695300 msgr2=0x56516da3eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.806+0000 7fc2de22c740 1 -- 172.21.15.114:0/2729301169 shutdown_connections 2024-05-27T07:47:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.806+0000 7fc2de22c740 1 -- 172.21.15.114:0/2729301169 wait complete. 2024-05-27T07:47:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.806+0000 7fc2de22c740 1 -- start start 2024-05-27T07:47:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.807+0000 7fc2de22c740 1 Processor -- start 2024-05-27T07:47:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.807+0000 7fc2de22c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da50f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.807+0000 7fc2de22c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 0x56516da51460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.807+0000 7fc2de22c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 0x56516da56ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.807+0000 7fc2de22c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56516d68a790 con 0x56516d6782f0 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.807+0000 7fc2de22c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56516d960890 con 0x56516d6804d0 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.807+0000 7fc2de22c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56516da53970 con 0x56516d6a11b0 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.807+0000 7fc2de22c740 10 client.0 _ll_register_callbacks cb 0x56516d6b8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.807+0000 7fc2d4911640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da50f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.807+0000 7fc2d4911640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da50f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55128/0 (socket says 172.21.15.114:55128) 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.807+0000 7fc2d4911640 1 -- 172.21.15.114:0/2367089550 learned_addr learned my addr 172.21.15.114:0/2367089550 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.807+0000 7fc2d4911640 1 --2- 172.21.15.114:0/2367089550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 0x56516da56ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.807+0000 7fc2d5112640 1 --2- 172.21.15.114:0/2367089550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 0x56516da51460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2d4911640 1 -- 172.21.15.114:0/2367089550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 msgr2=0x56516da56ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2d4911640 1 --2- 172.21.15.114:0/2367089550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 0x56516da56ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2d4911640 1 -- 172.21.15.114:0/2367089550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 msgr2=0x56516da51460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2d4911640 1 --2- 172.21.15.114:0/2367089550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 0x56516da51460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2d4911640 1 -- 172.21.15.114:0/2367089550 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56516d785620 con 0x56516d6782f0 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2d4911640 1 --2- 172.21.15.114:0/2367089550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da50f20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4031440 tx=0x7fc2c4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a262bb25799437 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2b8ff9640 1 -- 172.21.15.114:0/2367089550 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4035030 con 0x56516d6782f0 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2b8ff9640 1 -- 172.21.15.114:0/2367089550 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fc2c4004320 con 0x56516d6782f0 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2de22c740 1 -- 172.21.15.114:0/2367089550 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2cc0054d0 con 0x56516d6782f0 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2b8ff9640 1 -- 172.21.15.114:0/2367089550 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc2c4055050 con 0x56516d6782f0 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2de22c740 10 client.17169 fetch_fsmap learned FSMap version 45 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2de22c740 1 -- 172.21.15.114:0/2367089550 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56516da5fc70 con 0x56516d6782f0 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.808+0000 7fc2b8ff9640 1 -- 172.21.15.114:0/2367089550 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2cc0054d0 con 0x56516d6782f0 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.809+0000 7fc2b8ff9640 1 -- 172.21.15.114:0/2367089550 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 45) ==== 7725+0+0 (secure 0 0 0) 0x7fc2c403d4c0 con 0x56516d6782f0 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.809+0000 7fc2de22c740 10 client.17169 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.809+0000 7fc2de22c740 10 client.17169 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.809+0000 7fc2de22c740 20 client.17169 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.809+0000 7fc2de22c740 1 --2- 172.21.15.114:0/2367089550 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56516d65fd80 0x56516da63160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.809+0000 7fc2de22c740 4 client.17169 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.809+0000 7fc2de22c740 1 -- 172.21.15.114:0/2367089550 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56516d9b6b80 con 0x56516d65fd80 2024-05-27T07:47:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.809+0000 7fc2d5112640 1 --2- 172.21.15.114:0/2367089550 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56516d65fd80 0x56516da63160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.809+0000 7fc2b8ff9640 10 client.17169 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.809+0000 7fc2d5112640 1 --2- 172.21.15.114:0/2367089550 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56516d65fd80 0x56516da63160 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d700d8a89520473c server_cookie=23764cf64994c8c8 in_seq=0 out_seq=0 2024-05-27T07:47:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.810+0000 7fc2b8ff9640 1 -- 172.21.15.114:0/2367089550 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56516d9b6b80 con 0x56516d65fd80 2024-05-27T07:47:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.810+0000 7fc2b8ff9640 10 client.17169 handle_command_reply: tid=0 2024-05-27T07:47:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.810+0000 7fc2de22c740 1 client.17169 shutdown 2024-05-27T07:47:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.810+0000 7fc2de22c740 1 -- 172.21.15.114:0/2367089550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 msgr2=0x56516da50f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.810+0000 7fc2de22c740 1 --2- 172.21.15.114:0/2367089550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da50f20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4031440 tx=0x7fc2c4009260 comp rx=0 tx=0).stop 2024-05-27T07:47:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.810+0000 7fc2de22c740 1 -- 172.21.15.114:0/2367089550 shutdown_connections 2024-05-27T07:47:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.810+0000 7fc2de22c740 1 --2- 172.21.15.114:0/2367089550 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56516d65fd80 0x56516da63160 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.810+0000 7fc2de22c740 1 --2- 172.21.15.114:0/2367089550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 0x56516da56ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.810+0000 7fc2b8ff9640 0 client.17169 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.810+0000 7fc2de22c740 1 --2- 172.21.15.114:0/2367089550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 0x56516da51460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.810+0000 7fc2de22c740 1 --2- 172.21.15.114:0/2367089550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da50f20 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.811+0000 7fc2de22c740 1 -- 172.21.15.114:0/2367089550 >> 172.21.15.114:0/2367089550 conn(0x56516d695300 msgr2=0x56516da44060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.811+0000 7fc2de22c740 1 -- 172.21.15.114:0/2367089550 shutdown_connections 2024-05-27T07:47:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.811+0000 7fc2de22c740 1 -- 172.21.15.114:0/2367089550 wait complete. 2024-05-27T07:47:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.811+0000 7fc2de22c740 20 client.17169 trim_cache size 0 max 16384 2024-05-27T07:47:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.826+0000 7fc2de22c740 1 Processor -- start 2024-05-27T07:47:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.826+0000 7fc2de22c740 1 -- start start 2024-05-27T07:47:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2de22c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da42930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2de22c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 0x56516da5d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2de22c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 0x56516da4a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2de22c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01b00d0 con 0x56516d6782f0 2024-05-27T07:47:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2de22c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01ad180 con 0x56516d6804d0 2024-05-27T07:47:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2de22c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01af330 con 0x56516d6a11b0 2024-05-27T07:47:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2d4911640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da42930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2d4911640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da42930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55134/0 (socket says 172.21.15.114:55134) 2024-05-27T07:47:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2d4911640 1 -- 172.21.15.114:0/1247764394 learned_addr learned my addr 172.21.15.114:0/1247764394 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2d4911640 1 --2- 172.21.15.114:0/1247764394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 0x56516da4a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2d5112640 1 --2- 172.21.15.114:0/1247764394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 0x56516da5d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2d4911640 1 -- 172.21.15.114:0/1247764394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 msgr2=0x56516da4a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2d4911640 1 --2- 172.21.15.114:0/1247764394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 0x56516da4a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2d4911640 1 -- 172.21.15.114:0/1247764394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 msgr2=0x56516da5d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2d4911640 1 --2- 172.21.15.114:0/1247764394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 0x56516da5d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2d4911640 1 -- 172.21.15.114:0/1247764394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56516da53420 con 0x56516d6782f0 2024-05-27T07:47:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2d5112640 1 --2- 172.21.15.114:0/1247764394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 0x56516da5d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:47:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.827+0000 7fc2d4911640 1 --2- 172.21.15.114:0/1247764394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da42930 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4053040 tx=0x7fc2c402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a9354eb930d78ac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.828+0000 7fc2baffd640 1 -- 172.21.15.114:0/1247764394 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4035070 con 0x56516d6782f0 2024-05-27T07:47:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.828+0000 7fc2baffd640 1 -- 172.21.15.114:0/1247764394 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2c403dc10 con 0x56516d6782f0 2024-05-27T07:47:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.828+0000 7fc2baffd640 1 -- 172.21.15.114:0/1247764394 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c40052d0 con 0x56516d6782f0 2024-05-27T07:47:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.829+0000 7fc2de22c740 1 -- 172.21.15.114:0/1247764394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 msgr2=0x56516da42930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:17.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.829+0000 7fc2de22c740 1 --2- 172.21.15.114:0/1247764394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da42930 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4053040 tx=0x7fc2c402a410 comp rx=0 tx=0).stop 2024-05-27T07:47:17.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.829+0000 7fc2de22c740 1 -- 172.21.15.114:0/1247764394 shutdown_connections 2024-05-27T07:47:17.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.829+0000 7fc2de22c740 1 --2- 172.21.15.114:0/1247764394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6a11b0 0x56516da4a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.829+0000 7fc2de22c740 1 --2- 172.21.15.114:0/1247764394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6804d0 0x56516da5d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.829+0000 7fc2de22c740 1 --2- 172.21.15.114:0/1247764394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6782f0 0x56516da42930 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.829+0000 7fc2de22c740 1 -- 172.21.15.114:0/1247764394 >> 172.21.15.114:0/1247764394 conn(0x56516d695300 msgr2=0x56516da44a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:17.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.829+0000 7fc2de22c740 1 -- 172.21.15.114:0/1247764394 shutdown_connections 2024-05-27T07:47:17.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.829+0000 7fc2de22c740 1 -- 172.21.15.114:0/1247764394 wait complete. 2024-05-27T07:47:17.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.829+0000 7fc2de22c740 1 -- start start 2024-05-27T07:47:17.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.829+0000 7fc2de22c740 1 Processor -- start 2024-05-27T07:47:17.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2de22c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6782f0 0x56516da5afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2de22c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6804d0 0x56516da5b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2de22c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6a11b0 0x56516da56500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2de22c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01aec60 con 0x56516d6804d0 2024-05-27T07:47:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2de22c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01ad7f0 con 0x56516d6a11b0 2024-05-27T07:47:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2de22c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01adec0 con 0x56516d6782f0 2024-05-27T07:47:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2de22c740 10 client.0 _ll_register_callbacks cb 0x7fc2c4032ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2d5112640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6804d0 0x56516da5b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2d5112640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6804d0 0x56516da5b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55150/0 (socket says 172.21.15.114:55150) 2024-05-27T07:47:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2d4911640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6782f0 0x56516da5afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2d5112640 1 -- 172.21.15.114:0/849989416 learned_addr learned my addr 172.21.15.114:0/849989416 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2d4911640 1 --2- 172.21.15.114:0/849989416 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6a11b0 0x56516da56500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2d4911640 1 -- 172.21.15.114:0/849989416 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6a11b0 msgr2=0x56516da56500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2d4911640 1 --2- 172.21.15.114:0/849989416 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6a11b0 0x56516da56500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2d4911640 1 -- 172.21.15.114:0/849989416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6804d0 msgr2=0x56516da5b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2d4911640 1 --2- 172.21.15.114:0/849989416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6804d0 0x56516da5b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2d4911640 1 -- 172.21.15.114:0/849989416 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56516da56ac0 con 0x56516d6782f0 2024-05-27T07:47:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2d5112640 1 --2- 172.21.15.114:0/849989416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6804d0 0x56516da5b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2d4911640 1 --2- 172.21.15.114:0/849989416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6782f0 0x56516da5afd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4032d90 tx=0x7fc2c403d6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88ba0600b6da556f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.830+0000 7fc2b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:47:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2b8ff9640 1 -- 172.21.15.114:0/849989416 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4035040 con 0x56516d6782f0 2024-05-27T07:47:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2b8ff9640 1 -- 172.21.15.114:0/849989416 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fc2c40043e0 con 0x56516d6782f0 2024-05-27T07:47:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2de22c740 1 -- 172.21.15.114:0/849989416 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2c403dc10 con 0x56516d6782f0 2024-05-27T07:47:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2b8ff9640 1 -- 172.21.15.114:0/849989416 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc2c4005020 con 0x56516d6782f0 2024-05-27T07:47:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2de22c740 10 client.26743 fetch_fsmap learned FSMap version 45 2024-05-27T07:47:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2de22c740 1 -- 172.21.15.114:0/849989416 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56516da5a550 con 0x56516d6782f0 2024-05-27T07:47:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2b8ff9640 1 -- 172.21.15.114:0/849989416 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2c403dc10 con 0x56516d6782f0 2024-05-27T07:47:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2b8ff9640 1 -- 172.21.15.114:0/849989416 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 45) ==== 7725+0+0 (secure 0 0 0) 0x7fc2c4058040 con 0x56516d6782f0 2024-05-27T07:47:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2de22c740 10 client.26743 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:47:17.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2de22c740 10 client.26743 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:17.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2de22c740 20 client.26743 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:17.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2de22c740 1 --2- 172.21.15.114:0/849989416 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56516d6351c0 0x56516dacd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:17.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2de22c740 4 client.26743 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:47:17.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.831+0000 7fc2de22c740 1 -- 172.21.15.114:0/849989416 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56516d9b0360 con 0x56516d6351c0 2024-05-27T07:47:17.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.832+0000 7fc2d5112640 1 --2- 172.21.15.114:0/849989416 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56516d6351c0 0x56516dacd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.832+0000 7fc2b8ff9640 10 client.26743 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:17.832+0000 7fc2d5112640 1 --2- 172.21.15.114:0/849989416 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56516d6351c0 0x56516dacd840 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e51b978b588f924 server_cookie=7d43456e21fb662d in_seq=0 out_seq=0 2024-05-27T07:47:18.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2b8ff9640 1 -- 172.21.15.114:0/849989416 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56516d9b0360 con 0x56516d6351c0 2024-05-27T07:47:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2b8ff9640 10 client.26743 handle_command_reply: tid=0 2024-05-27T07:47:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2de22c740 1 client.26743 shutdown 2024-05-27T07:47:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2de22c740 1 -- 172.21.15.114:0/849989416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6782f0 msgr2=0x56516da5afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2de22c740 1 --2- 172.21.15.114:0/849989416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6782f0 0x56516da5afd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4032d90 tx=0x7fc2c403d6d0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2de22c740 1 -- 172.21.15.114:0/849989416 shutdown_connections 2024-05-27T07:47:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2de22c740 1 --2- 172.21.15.114:0/849989416 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56516d6351c0 0x56516dacd840 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2de22c740 1 --2- 172.21.15.114:0/849989416 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56516d6a11b0 0x56516da56500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2de22c740 1 --2- 172.21.15.114:0/849989416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56516d6804d0 0x56516da5b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2de22c740 1 --2- 172.21.15.114:0/849989416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56516d6782f0 0x56516da5afd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2de22c740 1 -- 172.21.15.114:0/849989416 >> 172.21.15.114:0/849989416 conn(0x56516d695300 msgr2=0x56516da44000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2b8ff9640 0 client.26743 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2de22c740 1 -- 172.21.15.114:0/849989416 shutdown_connections 2024-05-27T07:47:18.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2de22c740 1 -- 172.21.15.114:0/849989416 wait complete. 2024-05-27T07:47:18.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.142+0000 7fc2de22c740 20 client.26743 trim_cache size 0 max 16384 2024-05-27T07:47:18.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:47:18.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:47:18.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:47:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.143+0000 7fc2d7376640 1 -- 172.21.15.114:0/3946674528 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2bc0b2950 msgr2=0x7fc2bc0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.143+0000 7fc2d7376640 1 --2- 172.21.15.114:0/3946674528 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2bc0b2950 0x7fc2bc0b4e60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80a5eb0 tx=0x7fc2c800cee0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.143+0000 7fc2d7376640 1 -- 172.21.15.114:0/3946674528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d010e1e0 msgr2=0x7fc2d019a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.143+0000 7fc2d7376640 1 --2- 172.21.15.114:0/3946674528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d010e1e0 0x7fc2d019a6a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800eb10 tx=0x7fc2c800c2d0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.143+0000 7fc2d7376640 1 -- 172.21.15.114:0/3946674528 shutdown_connections 2024-05-27T07:47:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.143+0000 7fc2d7376640 1 --2- 172.21.15.114:0/3946674528 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2bc0b2950 0x7fc2bc0b4e60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.143+0000 7fc2d7376640 1 --2- 172.21.15.114:0/3946674528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d010e1e0 0x7fc2d019a6a0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.143+0000 7fc2d7376640 1 --2- 172.21.15.114:0/3946674528 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2d010b7f0 0x7fc2d01a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.143+0000 7fc2d7376640 1 --2- 172.21.15.114:0/3946674528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d0104d90 0x7fc2d01a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.143+0000 7fc2d7376640 1 -- 172.21.15.114:0/3946674528 >> 172.21.15.114:0/3946674528 conn(0x7fc2d00782d0 msgr2=0x7fc2d01012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.143+0000 7fc2d7376640 1 -- 172.21.15.114:0/3946674528 shutdown_connections 2024-05-27T07:47:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:18.143+0000 7fc2d7376640 1 -- 172.21.15.114:0/3946674528 wait complete. 2024-05-27T07:47:18.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 757836 11.59 MB/sec execute 210 sec latency 201.668 ms 2024-05-27T07:47:18.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.703+0000 7f60bbaa2700 1 -- 172.21.15.114:0/4047488193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b4103540 msgr2=0x7f60b4103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:18.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.703+0000 7f60bbaa2700 1 --2- 172.21.15.114:0/4047488193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b4103540 0x7f60b4103950 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f60b00099c0 tx=0x7f60b0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.703+0000 7f60bbaa2700 1 -- 172.21.15.114:0/4047488193 shutdown_connections 2024-05-27T07:47:18.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.703+0000 7f60bbaa2700 1 --2- 172.21.15.114:0/4047488193 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60b4108770 0x7f60b41100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.703+0000 7f60bbaa2700 1 --2- 172.21.15.114:0/4047488193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b4103540 0x7f60b4103950 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.703+0000 7f60bbaa2700 1 --2- 172.21.15.114:0/4047488193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b4107e20 0x7f60b4108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.703+0000 7f60bbaa2700 1 -- 172.21.15.114:0/4047488193 >> 172.21.15.114:0/4047488193 conn(0x7f60b4078df0 msgr2=0x7f60b4079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:18.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.703+0000 7f60bbaa2700 1 -- 172.21.15.114:0/4047488193 shutdown_connections 2024-05-27T07:47:18.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.703+0000 7f60bbaa2700 1 -- 172.21.15.114:0/4047488193 wait complete. 2024-05-27T07:47:18.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.704+0000 7f60bbaa2700 1 Processor -- start 2024-05-27T07:47:18.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.704+0000 7f60bbaa2700 1 -- start start 2024-05-27T07:47:18.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.704+0000 7f60bbaa2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b4103540 0x7f60b419bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:18.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.704+0000 7f60bbaa2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60b4107e20 0x7f60b419c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:18.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.704+0000 7f60bbaa2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b4108770 0x7f60b419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:18.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.704+0000 7f60bbaa2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f60b41133a0 con 0x7f60b4108770 2024-05-27T07:47:18.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.704+0000 7f60bbaa2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f60b4113220 con 0x7f60b4107e20 2024-05-27T07:47:18.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.704+0000 7f60bbaa2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f60b4113520 con 0x7f60b4103540 2024-05-27T07:47:18.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.704+0000 7f60ba03f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b4108770 0x7f60b419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:18.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.704+0000 7f60ba03f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b4108770 0x7f60b419c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55174/0 (socket says 172.21.15.114:55174) 2024-05-27T07:47:18.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.704+0000 7f60ba03f700 1 -- 172.21.15.114:0/3379888694 learned_addr learned my addr 172.21.15.114:0/3379888694 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:18.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.704+0000 7f60b983e700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b4103540 0x7f60b419bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:18.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.704+0000 7f60b903d700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60b4107e20 0x7f60b419c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:18.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.705+0000 7f60ba03f700 1 -- 172.21.15.114:0/3379888694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b4103540 msgr2=0x7f60b419bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:18.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.705+0000 7f60ba03f700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b4103540 0x7f60b419bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.705+0000 7f60ba03f700 1 -- 172.21.15.114:0/3379888694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60b4107e20 msgr2=0x7f60b419c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:18.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.705+0000 7f60ba03f700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60b4107e20 0x7f60b419c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.705+0000 7f60ba03f700 1 -- 172.21.15.114:0/3379888694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b41a7030 con 0x7f60b4108770 2024-05-27T07:47:18.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.705+0000 7f60b903d700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60b4107e20 0x7f60b419c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:47:18.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.705+0000 7f60b983e700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b4103540 0x7f60b419bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:18.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.705+0000 7f60ba03f700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b4108770 0x7f60b419c780 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00ef30 tx=0x7f60ac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:18.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.705+0000 7f60aaffd700 1 -- 172.21.15.114:0/3379888694 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0092e0 con 0x7f60b4108770 2024-05-27T07:47:18.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.705+0000 7f60bbaa2700 1 -- 172.21.15.114:0/3379888694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b41a72c0 con 0x7f60b4108770 2024-05-27T07:47:18.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.705+0000 7f60bbaa2700 1 -- 172.21.15.114:0/3379888694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b41a7800 con 0x7f60b4108770 2024-05-27T07:47:18.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.705+0000 7f60aaffd700 1 -- 172.21.15.114:0/3379888694 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60ac004bb0 con 0x7f60b4108770 2024-05-27T07:47:18.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.705+0000 7f60aaffd700 1 -- 172.21.15.114:0/3379888694 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac004280 con 0x7f60b4108770 2024-05-27T07:47:18.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.706+0000 7f60bbaa2700 1 -- 172.21.15.114:0/3379888694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6098005330 con 0x7f60b4108770 2024-05-27T07:47:18.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.709+0000 7f60aaffd700 1 -- 172.21.15.114:0/3379888694 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f60ac01b030 con 0x7f60b4108770 2024-05-27T07:47:18.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.710+0000 7f60aaffd700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60a00b2a70 0x7f60a00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:18.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.710+0000 7f60b983e700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60a00b2a70 0x7f60a00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:18.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.710+0000 7f60aaffd700 1 -- 172.21.15.114:0/3379888694 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f60ac0e2d20 con 0x7f60b4108770 2024-05-27T07:47:18.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.711+0000 7f60b983e700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60a00b2a70 0x7f60a00b4f80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f60b0000c00 tx=0x7f60b001a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:18.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.714+0000 7f60aaffd700 1 -- 172.21.15.114:0/3379888694 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f60ac022020 con 0x7f60b4108770 2024-05-27T07:47:18.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.994+0000 7f60bbaa2700 1 -- 172.21.15.114:0/3379888694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6098005620 con 0x7f60b4108770 2024-05-27T07:47:18.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.995+0000 7f60aaffd700 1 -- 172.21.15.114:0/3379888694 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11000 (secure 0 0 0) 0x7f60ac010040 con 0x7f60b4108770 2024-05-27T07:47:18.995 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:47:18.995 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":26227,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2936519309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2936519309},{"type":"v1","addr":"172.21.15.186:6835","nonce":2936519309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:47:15.274340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26707":{"gid":26707,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/298307750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":298307750},{"type":"v1","addr":"172.21.15.186:6837","nonce":298307750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:47:18.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.997+0000 7f60bbaa2700 1 -- 172.21.15.114:0/3379888694 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60a00b2a70 msgr2=0x7f60a00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:18.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.997+0000 7f60bbaa2700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60a00b2a70 0x7f60a00b4f80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f60b0000c00 tx=0x7f60b001a040 comp rx=0 tx=0).stop 2024-05-27T07:47:18.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.997+0000 7f60bbaa2700 1 -- 172.21.15.114:0/3379888694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b4108770 msgr2=0x7f60b419c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:18.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.997+0000 7f60bbaa2700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b4108770 0x7f60b419c780 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00ef30 tx=0x7f60ac00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.997+0000 7f60bbaa2700 1 -- 172.21.15.114:0/3379888694 shutdown_connections 2024-05-27T07:47:18.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.997+0000 7f60bbaa2700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b4108770 0x7f60b419c780 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.997+0000 7f60bbaa2700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b4103540 0x7f60b419bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.997+0000 7f60bbaa2700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60a00b2a70 0x7f60a00b4f80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.997+0000 7f60bbaa2700 1 --2- 172.21.15.114:0/3379888694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60b4107e20 0x7f60b419c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:18.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.997+0000 7f60bbaa2700 1 -- 172.21.15.114:0/3379888694 >> 172.21.15.114:0/3379888694 conn(0x7f60b4078df0 msgr2=0x7f60b4101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:18.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.997+0000 7f60bbaa2700 1 -- 172.21.15.114:0/3379888694 shutdown_connections 2024-05-27T07:47:18.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:18.997+0000 7f60bbaa2700 1 -- 172.21.15.114:0/3379888694 wait complete. 2024-05-27T07:47:18.998 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 45 2024-05-27T07:47:19.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 760239 11.60 MB/sec execute 211 sec latency 64.778 ms 2024-05-27T07:47:19.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:19 smithi114 ceph-mon[32038]: pgmap v245: 129 pgs: 129 active+clean; 72 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 186 op/s 2024-05-27T07:47:19.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:19 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3379888694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:47:19.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:47:19.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:19 smithi163 ceph-mon[36451]: pgmap v245: 129 pgs: 129 active+clean; 72 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 186 op/s 2024-05-27T07:47:19.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:19 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3379888694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:47:19.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:19 smithi186 ceph-mon[36647]: pgmap v245: 129 pgs: 129 active+clean; 72 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 186 op/s 2024-05-27T07:47:19.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:19 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3379888694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:47:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:47:20.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 762344 11.58 MB/sec execute 212 sec latency 15.835 ms 2024-05-27T07:47:21.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 764353 11.58 MB/sec execute 213 sec latency 26.192 ms 2024-05-27T07:47:21.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:21 smithi114 ceph-mon[32038]: pgmap v246: 129 pgs: 129 active+clean; 71 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 286 op/s 2024-05-27T07:47:21.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:21 smithi163 ceph-mon[36451]: pgmap v246: 129 pgs: 129 active+clean; 71 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 286 op/s 2024-05-27T07:47:21.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:21 smithi186 ceph-mon[36647]: pgmap v246: 129 pgs: 129 active+clean; 71 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 286 op/s 2024-05-27T07:47:21.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.799+0000 7efc0e7e3700 1 -- 172.21.15.114:0/1288531530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc081045d0 msgr2=0x7efc081049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:21.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.799+0000 7efc0e7e3700 1 --2- 172.21.15.114:0/1288531530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc081045d0 0x7efc081049e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7efbf80099c0 tx=0x7efbf8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.799+0000 7efc0e7e3700 1 -- 172.21.15.114:0/1288531530 shutdown_connections 2024-05-27T07:47:21.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.799+0000 7efc0e7e3700 1 --2- 172.21.15.114:0/1288531530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc08109850 0x7efc081111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.799+0000 7efc0e7e3700 1 --2- 172.21.15.114:0/1288531530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc081045d0 0x7efc081049e0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.799+0000 7efc0e7e3700 1 --2- 172.21.15.114:0/1288531530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc08108f00 0x7efc08109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.799+0000 7efc0e7e3700 1 -- 172.21.15.114:0/1288531530 >> 172.21.15.114:0/1288531530 conn(0x7efc080ffe60 msgr2=0x7efc081022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:21.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.799+0000 7efc0e7e3700 1 -- 172.21.15.114:0/1288531530 shutdown_connections 2024-05-27T07:47:21.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.799+0000 7efc0e7e3700 1 -- 172.21.15.114:0/1288531530 wait complete. 2024-05-27T07:47:21.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.799+0000 7efc0e7e3700 1 Processor -- start 2024-05-27T07:47:21.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.800+0000 7efc0e7e3700 1 -- start start 2024-05-27T07:47:21.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.800+0000 7efc0e7e3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc081045d0 0x7efc0819add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.800+0000 7efc0e7e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc08108f00 0x7efc0819b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.800+0000 7efc0e7e3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc08109850 0x7efc0819f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.800+0000 7efc0e7e3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efc08114480 con 0x7efc08108f00 2024-05-27T07:47:21.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.800+0000 7efc0e7e3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efc08114300 con 0x7efc08109850 2024-05-27T07:47:21.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.800+0000 7efc0e7e3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efc08114600 con 0x7efc081045d0 2024-05-27T07:47:21.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.800+0000 7efc077fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc08108f00 0x7efc0819b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.800+0000 7efc07fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc081045d0 0x7efc0819add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.800+0000 7efc07fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc081045d0 0x7efc0819add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59724/0 (socket says 172.21.15.114:59724) 2024-05-27T07:47:21.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.800+0000 7efc077fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc08108f00 0x7efc0819b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55204/0 (socket says 172.21.15.114:55204) 2024-05-27T07:47:21.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.800+0000 7efc07fff700 1 -- 172.21.15.114:0/3424961270 learned_addr learned my addr 172.21.15.114:0/3424961270 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:21.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.800+0000 7efc0cd80700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc08109850 0x7efc0819f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.801+0000 7efc077fe700 1 -- 172.21.15.114:0/3424961270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc081045d0 msgr2=0x7efc0819add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:21.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.801+0000 7efc077fe700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc081045d0 0x7efc0819add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.801+0000 7efc077fe700 1 -- 172.21.15.114:0/3424961270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc08109850 msgr2=0x7efc0819f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:21.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.801+0000 7efc077fe700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc08109850 0x7efc0819f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.801+0000 7efc077fe700 1 -- 172.21.15.114:0/3424961270 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0819fbf0 con 0x7efc08108f00 2024-05-27T07:47:21.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.801+0000 7efc07fff700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc081045d0 0x7efc0819add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:21.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.801+0000 7efc077fe700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc08108f00 0x7efc0819b370 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00c940 tx=0x7efbfc00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:21.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.801+0000 7efc057fa700 1 -- 172.21.15.114:0/3424961270 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc014070 con 0x7efc08108f00 2024-05-27T07:47:21.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.801+0000 7efc057fa700 1 -- 172.21.15.114:0/3424961270 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbfc004440 con 0x7efc08108f00 2024-05-27T07:47:21.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.801+0000 7efc0e7e3700 1 -- 172.21.15.114:0/3424961270 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc0819fee0 con 0x7efc08108f00 2024-05-27T07:47:21.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.801+0000 7efc057fa700 1 -- 172.21.15.114:0/3424961270 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc007ab0 con 0x7efc08108f00 2024-05-27T07:47:21.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.801+0000 7efc0e7e3700 1 -- 172.21.15.114:0/3424961270 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc081a0420 con 0x7efc08108f00 2024-05-27T07:47:21.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.805+0000 7efc057fa700 1 -- 172.21.15.114:0/3424961270 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7efbfc017070 con 0x7efc08108f00 2024-05-27T07:47:21.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.806+0000 7efc057fa700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efbf00b29f0 0x7efbf00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.806+0000 7efc1d1f8b80 1 Processor -- start 2024-05-27T07:47:21.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.806+0000 7efc07fff700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efbf00b29f0 0x7efbf00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.806+0000 7efc057fa700 1 -- 172.21.15.114:0/3424961270 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7efbfc0e2650 con 0x7efc08108f00 2024-05-27T07:47:21.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.806+0000 7efc1d1f8b80 1 -- start start 2024-05-27T07:47:21.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.806+0000 7efc1d1f8b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f736f4e40 0x560f73879b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.806+0000 7efc1d1f8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f7387a070 0x560f73873be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.806+0000 7efc1d1f8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f73874120 0x560f73874530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.806+0000 7efc1d1f8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560f737a1d70 con 0x560f7387a070 2024-05-27T07:47:21.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.806+0000 7efc1d1f8b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560f73791910 con 0x560f736f4e40 2024-05-27T07:47:21.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.806+0000 7efc1d1f8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560f73795950 con 0x560f73874120 2024-05-27T07:47:21.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.806+0000 7efc07fff700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efbf00b29f0 0x7efbf00b4f00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009990 tx=0x7efbf801a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:21.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.806+0000 7efc07fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f736f4e40 0x560f73879b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efc07fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f736f4e40 0x560f73879b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:53538/0 (socket says 172.21.15.114:53538) 2024-05-27T07:47:21.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efc077fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f7387a070 0x560f73873be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efc07fff700 1 -- 172.21.15.114:0/256021032 learned_addr learned my addr 172.21.15.114:0/256021032 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:21.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efc0cd80700 1 --2- 172.21.15.114:0/256021032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f73874120 0x560f73874530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efc07fff700 1 -- 172.21.15.114:0/256021032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f73874120 msgr2=0x560f73874530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:21.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efc07fff700 1 --2- 172.21.15.114:0/256021032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f73874120 0x560f73874530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efc07fff700 1 -- 172.21.15.114:0/256021032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f7387a070 msgr2=0x560f73873be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:21.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efc07fff700 1 --2- 172.21.15.114:0/256021032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f7387a070 0x560f73873be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efc07fff700 1 -- 172.21.15.114:0/256021032 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f7373afb0 con 0x560f736f4e40 2024-05-27T07:47:21.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efc077fe700 1 --2- 172.21.15.114:0/256021032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f7387a070 0x560f73873be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:21.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efc07fff700 1 --2- 172.21.15.114:0/256021032 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f736f4e40 0x560f73879b30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efbf8005910 tx=0x7efbf801f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5139e8ba1cd5cdae server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:21.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efbee7fc700 1 -- 172.21.15.114:0/256021032 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf80037c0 con 0x560f736f4e40 2024-05-27T07:47:21.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efbee7fc700 1 -- 172.21.15.114:0/256021032 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbf8003e10 con 0x560f736f4e40 2024-05-27T07:47:21.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.807+0000 7efbee7fc700 1 -- 172.21.15.114:0/256021032 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf800fe20 con 0x560f736f4e40 2024-05-27T07:47:21.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.808+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/256021032 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f736f4e40 msgr2=0x560f73879b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:21.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.808+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/256021032 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f736f4e40 0x560f73879b30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efbf8005910 tx=0x7efbf801f000 comp rx=0 tx=0).stop 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.808+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/256021032 shutdown_connections 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.808+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/256021032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f73874120 0x560f73874530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.808+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/256021032 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f736f4e40 0x560f73879b30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.808+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/256021032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f7387a070 0x560f73873be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.808+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/256021032 >> 172.21.15.114:0/256021032 conn(0x560f736f7770 msgr2=0x560f7386ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.809+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/256021032 shutdown_connections 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.809+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/256021032 wait complete. 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.809+0000 7efc1d1f8b80 1 -- start start 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.809+0000 7efc1d1f8b80 1 Processor -- start 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc1d1f8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f736f4e40 0x560f7387d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc1d1f8b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f73874120 0x560f7387dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc1d1f8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f7387a070 0x560f7387e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc1d1f8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560f73697fa0 con 0x560f736f4e40 2024-05-27T07:47:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc1d1f8b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560f736f9da0 con 0x560f73874120 2024-05-27T07:47:21.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc1d1f8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560f73880b00 con 0x560f7387a070 2024-05-27T07:47:21.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc1d1f8b80 10 client.0 _ll_register_callbacks cb 0x560f737285b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:21.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc07fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f736f4e40 0x560f7387d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc07fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f736f4e40 0x560f7387d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55216/0 (socket says 172.21.15.114:55216) 2024-05-27T07:47:21.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc07fff700 1 -- 172.21.15.114:0/1277835026 learned_addr learned my addr 172.21.15.114:0/1277835026 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:21.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc077fe700 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f73874120 0x560f7387dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc0cd80700 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f7387a070 0x560f7387e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc077fe700 1 -- 172.21.15.114:0/1277835026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f7387a070 msgr2=0x560f7387e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:21.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc077fe700 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f7387a070 0x560f7387e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc077fe700 1 -- 172.21.15.114:0/1277835026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f736f4e40 msgr2=0x560f7387d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:21.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc077fe700 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f736f4e40 0x560f7387d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc077fe700 1 -- 172.21.15.114:0/1277835026 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f736e18d0 con 0x560f73874120 2024-05-27T07:47:21.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc07fff700 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f736f4e40 0x560f7387d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:21.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efc077fe700 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f73874120 0x560f7387dda0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0df9f0 tx=0x7efbfc0a91e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cd354d0597f91c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:21.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efbe77fe700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.810+0000 7efbe77fe700 1 -- 172.21.15.114:0/1277835026 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc0a9580 con 0x560f73874120 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1277835026 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efbf8003e10 con 0x560f73874120 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efbe77fe700 1 -- 172.21.15.114:0/1277835026 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7efbfc0a99f0 con 0x560f73874120 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efbe77fe700 1 -- 172.21.15.114:0/1277835026 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7efbfc0c3050 con 0x560f73874120 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efc1d1f8b80 10 client.26663 fetch_fsmap learned FSMap version 45 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1277835026 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x560f7374b760 con 0x560f73874120 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efbe77fe700 1 -- 172.21.15.114:0/1277835026 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbf8003e10 con 0x560f73874120 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efbe77fe700 1 -- 172.21.15.114:0/1277835026 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 45) ==== 7725+0+0 (secure 0 0 0) 0x7efbfc0e21c0 con 0x560f73874120 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efc1d1f8b80 10 client.26663 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efc1d1f8b80 10 client.26663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efc1d1f8b80 20 client.26663 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560f7388da50 0x560f7388fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efc1d1f8b80 4 client.26663 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1277835026 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f737a3cc0 con 0x560f7388da50 2024-05-27T07:47:21.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.811+0000 7efc07fff700 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560f7388da50 0x560f7388fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.812+0000 7efbe77fe700 10 client.26663 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:21.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.812+0000 7efc07fff700 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560f7388da50 0x560f7388fe70 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b9eef54d06a0b8e server_cookie=431afc3034dac14e in_seq=0 out_seq=0 2024-05-27T07:47:21.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.813+0000 7efbe77fe700 1 -- 172.21.15.114:0/1277835026 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560f737a3cc0 con 0x560f7388da50 2024-05-27T07:47:21.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.813+0000 7efbe77fe700 10 client.26663 handle_command_reply: tid=0 2024-05-27T07:47:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.813+0000 7efc1d1f8b80 1 client.26663 shutdown 2024-05-27T07:47:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.813+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1277835026 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f73874120 msgr2=0x560f7387dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.813+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f73874120 0x560f7387dda0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0df9f0 tx=0x7efbfc0a91e0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.814+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1277835026 shutdown_connections 2024-05-27T07:47:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.814+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f7387a070 0x560f7387e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.814+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f736f4e40 0x560f7387d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.814+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560f7388da50 0x560f7388fe70 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.814+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/1277835026 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f73874120 0x560f7387dda0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.814+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1277835026 >> 172.21.15.114:0/1277835026 conn(0x560f736f7770 msgr2=0x560f73872bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.814+0000 7efbe77fe700 0 client.26663 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.814+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1277835026 shutdown_connections 2024-05-27T07:47:21.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.814+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1277835026 wait complete. 2024-05-27T07:47:21.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.814+0000 7efc1d1f8b80 20 client.26663 trim_cache size 0 max 16384 2024-05-27T07:47:21.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.830+0000 7efc1d1f8b80 1 Processor -- start 2024-05-27T07:47:21.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.830+0000 7efc1d1f8b80 1 -- start start 2024-05-27T07:47:21.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.830+0000 7efc1d1f8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f736f4e40 0x560f7387c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.830+0000 7efc1d1f8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f73874120 0x560f73873af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc1d1f8b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f7387a070 0x560f7386e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc1d1f8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efc081b3550 con 0x560f73874120 2024-05-27T07:47:21.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc1d1f8b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efc081b0600 con 0x560f7387a070 2024-05-27T07:47:21.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc1d1f8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efc081b27b0 con 0x560f736f4e40 2024-05-27T07:47:21.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc077fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f73874120 0x560f73873af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc07fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f736f4e40 0x560f7387c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc077fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f73874120 0x560f73873af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55230/0 (socket says 172.21.15.114:55230) 2024-05-27T07:47:21.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc07fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f736f4e40 0x560f7387c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59746/0 (socket says 172.21.15.114:59746) 2024-05-27T07:47:21.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc077fe700 1 -- 172.21.15.114:0/3768408722 learned_addr learned my addr 172.21.15.114:0/3768408722 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:21.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc0cd80700 1 --2- 172.21.15.114:0/3768408722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f7387a070 0x560f7386e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc077fe700 1 -- 172.21.15.114:0/3768408722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f736f4e40 msgr2=0x560f7387c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:21.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc077fe700 1 --2- 172.21.15.114:0/3768408722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f736f4e40 0x560f7387c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc077fe700 1 -- 172.21.15.114:0/3768408722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f7387a070 msgr2=0x560f7386e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:21.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc077fe700 1 --2- 172.21.15.114:0/3768408722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f7387a070 0x560f7386e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc077fe700 1 -- 172.21.15.114:0/3768408722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f7387bdd0 con 0x560f73874120 2024-05-27T07:47:21.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc0cd80700 1 --2- 172.21.15.114:0/3768408722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f7387a070 0x560f7386e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:47:21.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efc077fe700 1 --2- 172.21.15.114:0/3768408722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f73874120 0x560f73873af0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0a67f0 tx=0x7efbfc00a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c72492c447b0156a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:21.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efbedffb700 1 -- 172.21.15.114:0/3768408722 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc0a9580 con 0x560f73874120 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efbedffb700 1 -- 172.21.15.114:0/3768408722 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbfc0c8020 con 0x560f73874120 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.831+0000 7efbedffb700 1 -- 172.21.15.114:0/3768408722 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc0a99f0 con 0x560f73874120 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.832+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/3768408722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f73874120 msgr2=0x560f73873af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.832+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/3768408722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f73874120 0x560f73873af0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0a67f0 tx=0x7efbfc00a410 comp rx=0 tx=0).stop 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.832+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/3768408722 shutdown_connections 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.832+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/3768408722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f7387a070 0x560f7386e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.832+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/3768408722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f736f4e40 0x560f7387c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.832+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/3768408722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f73874120 0x560f73873af0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.832+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/3768408722 >> 172.21.15.114:0/3768408722 conn(0x560f736f7770 msgr2=0x560f738733e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.832+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/3768408722 shutdown_connections 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.832+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/3768408722 wait complete. 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.833+0000 7efc1d1f8b80 1 -- start start 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.833+0000 7efc1d1f8b80 1 Processor -- start 2024-05-27T07:47:21.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.833+0000 7efc1d1f8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f736f4e40 0x560f73885d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.833+0000 7efc1d1f8b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f73874120 0x560f738862f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.833+0000 7efc1d1f8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f7387a070 0x560f7387a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.833+0000 7efc1d1f8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efc081b20e0 con 0x560f7387a070 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.833+0000 7efc1d1f8b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efc081b0c70 con 0x560f73874120 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.833+0000 7efc1d1f8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efc081b1340 con 0x560f736f4e40 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.833+0000 7efc1d1f8b80 10 client.0 _ll_register_callbacks cb 0x560f737285b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.833+0000 7efc07fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f736f4e40 0x560f73885d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.833+0000 7efc0cd80700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f7387a070 0x560f7387a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.833+0000 7efc07fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f736f4e40 0x560f73885d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59750/0 (socket says 172.21.15.114:59750) 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.833+0000 7efc07fff700 1 -- 172.21.15.114:0/1579661921 learned_addr learned my addr 172.21.15.114:0/1579661921 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efc077fe700 1 --2- 172.21.15.114:0/1579661921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f73874120 0x560f738862f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efc0cd80700 1 -- 172.21.15.114:0/1579661921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f736f4e40 msgr2=0x560f73885d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efc0cd80700 1 --2- 172.21.15.114:0/1579661921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f736f4e40 0x560f73885d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efc0cd80700 1 -- 172.21.15.114:0/1579661921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f73874120 msgr2=0x560f738862f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efc0cd80700 1 --2- 172.21.15.114:0/1579661921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f73874120 0x560f738862f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efc0cd80700 1 -- 172.21.15.114:0/1579661921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f7387acb0 con 0x560f7387a070 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efc077fe700 1 --2- 172.21.15.114:0/1579661921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f73874120 0x560f738862f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efc0cd80700 1 --2- 172.21.15.114:0/1579661921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f7387a070 0x560f7387a670 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7efc00007ed0 tx=0x7efc00007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa12dca9f06299ca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efbe77fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efbe77fe700 1 -- 172.21.15.114:0/1579661921 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc00003a00 con 0x560f7387a070 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efbe77fe700 1 -- 172.21.15.114:0/1579661921 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7efc00003ba0 con 0x560f7387a070 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1579661921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efbfc0c8020 con 0x560f7387a070 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efbe77fe700 1 -- 172.21.15.114:0/1579661921 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7efc0002a050 con 0x560f7387a070 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efc1d1f8b80 10 client.17199 fetch_fsmap learned FSMap version 45 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1579661921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x560f738f7c80 con 0x560f7387a070 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.834+0000 7efbe77fe700 1 -- 172.21.15.114:0/1579661921 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbfc0c8020 con 0x560f7387a070 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.835+0000 7efbe77fe700 1 -- 172.21.15.114:0/1579661921 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 45) ==== 7725+0+0 (secure 0 0 0) 0x7efc0001f020 con 0x560f7387a070 2024-05-27T07:47:21.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.835+0000 7efc1d1f8b80 10 client.17199 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:47:21.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.835+0000 7efc1d1f8b80 10 client.17199 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:21.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.835+0000 7efc1d1f8b80 20 client.17199 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:21.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.835+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/1579661921 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560f738f83f0 0x560f738fa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:21.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.835+0000 7efc1d1f8b80 4 client.17199 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:47:21.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.835+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1579661921 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f7379d4a0 con 0x560f738f83f0 2024-05-27T07:47:21.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.835+0000 7efc07fff700 1 --2- 172.21.15.114:0/1579661921 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560f738f83f0 0x560f738fa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:21.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.835+0000 7efbe77fe700 10 client.17199 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:21.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.835+0000 7efc07fff700 1 --2- 172.21.15.114:0/1579661921 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560f738f83f0 0x560f738fa810 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7c21967f11f6056 server_cookie=3dc3e6b899b21f22 in_seq=0 out_seq=0 2024-05-27T07:47:21.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.836+0000 7efbe77fe700 1 -- 172.21.15.114:0/1579661921 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f7379d4a0 con 0x560f738f83f0 2024-05-27T07:47:21.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.836+0000 7efbe77fe700 10 client.17199 handle_command_reply: tid=0 2024-05-27T07:47:21.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.836+0000 7efc1d1f8b80 1 client.17199 shutdown 2024-05-27T07:47:21.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.837+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1579661921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f7387a070 msgr2=0x560f7387a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:21.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.837+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/1579661921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f7387a070 0x560f7387a670 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7efc00007ed0 tx=0x7efc00007f00 comp rx=0 tx=0).stop 2024-05-27T07:47:21.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.837+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1579661921 shutdown_connections 2024-05-27T07:47:21.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.837+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/1579661921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f7387a070 0x560f7387a670 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.837+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/1579661921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f736f4e40 0x560f73885d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.837+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/1579661921 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560f738f83f0 0x560f738fa810 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.837+0000 7efc1d1f8b80 1 --2- 172.21.15.114:0/1579661921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f73874120 0x560f738862f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.837+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1579661921 >> 172.21.15.114:0/1579661921 conn(0x560f736f7770 msgr2=0x560f73877490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:21.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.837+0000 7efbe77fe700 0 client.17199 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:21.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.837+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1579661921 shutdown_connections 2024-05-27T07:47:21.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.837+0000 7efc1d1f8b80 1 -- 172.21.15.114:0/1579661921 wait complete. 2024-05-27T07:47:21.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.837+0000 7efc1d1f8b80 20 client.17199 trim_cache size 0 max 16384 2024-05-27T07:47:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:47:21.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.838+0000 7efc0e7e3700 1 -- 172.21.15.114:0/3424961270 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efbf00b29f0 msgr2=0x7efbf00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:21.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.838+0000 7efc0e7e3700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efbf00b29f0 0x7efbf00b4f00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009990 tx=0x7efbf801a040 comp rx=0 tx=0).stop 2024-05-27T07:47:21.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.838+0000 7efc0e7e3700 1 -- 172.21.15.114:0/3424961270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc08108f00 msgr2=0x7efc0819b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:21.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.838+0000 7efc0e7e3700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc08108f00 0x7efc0819b370 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00c940 tx=0x7efbfc00cc50 comp rx=0 tx=0).stop 2024-05-27T07:47:21.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.838+0000 7efc0e7e3700 1 -- 172.21.15.114:0/3424961270 shutdown_connections 2024-05-27T07:47:21.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.838+0000 7efc0e7e3700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efbf00b29f0 0x7efbf00b4f00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.838+0000 7efc0e7e3700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc08109850 0x7efc0819f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.838+0000 7efc0e7e3700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc081045d0 0x7efc0819add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.838+0000 7efc0e7e3700 1 --2- 172.21.15.114:0/3424961270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc08108f00 0x7efc0819b370 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:21.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.838+0000 7efc0e7e3700 1 -- 172.21.15.114:0/3424961270 >> 172.21.15.114:0/3424961270 conn(0x7efc080ffe60 msgr2=0x7efc081086a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:21.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.838+0000 7efc0e7e3700 1 -- 172.21.15.114:0/3424961270 shutdown_connections 2024-05-27T07:47:21.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:21.838+0000 7efc0e7e3700 1 -- 172.21.15.114:0/3424961270 wait complete. 2024-05-27T07:47:22.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 766421 11.58 MB/sec execute 214 sec latency 103.128 ms 2024-05-27T07:47:23.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 768342 11.58 MB/sec execute 215 sec latency 32.062 ms 2024-05-27T07:47:23.523 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:47:23.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:47:23.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:23 smithi114 ceph-mon[32038]: pgmap v247: 129 pgs: 129 active+clean; 63 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 466 op/s 2024-05-27T07:47:23.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:23 smithi163 ceph-mon[36451]: pgmap v247: 129 pgs: 129 active+clean; 63 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 466 op/s 2024-05-27T07:47:23.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:23 smithi186 ceph-mon[36647]: pgmap v247: 129 pgs: 129 active+clean; 63 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 466 op/s 2024-05-27T07:47:23.785 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:47:24.267 INFO:tasks.workunit.client.0.smithi114.stdout: 1 770744 11.59 MB/sec execute 216 sec latency 37.481 ms 2024-05-27T07:47:25.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 772477 11.57 MB/sec execute 217 sec latency 16.862 ms 2024-05-27T07:47:25.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:25 smithi114 ceph-mon[32038]: pgmap v248: 129 pgs: 129 active+clean; 63 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 373 op/s 2024-05-27T07:47:25.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.555+0000 7eff82596700 1 -- 172.21.15.114:0/932844402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff7c116ec0 msgr2=0x7eff7c1193d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:25.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.555+0000 7eff82596700 1 --2- 172.21.15.114:0/932844402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff7c116ec0 0x7eff7c1193d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7eff7400b0a0 tx=0x7eff7400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.555+0000 7eff82596700 1 -- 172.21.15.114:0/932844402 shutdown_connections 2024-05-27T07:47:25.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.555+0000 7eff82596700 1 --2- 172.21.15.114:0/932844402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff7c116ec0 0x7eff7c1193d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.555+0000 7eff82596700 1 --2- 172.21.15.114:0/932844402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff7c10aa40 0x7eff7c10ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.555+0000 7eff82596700 1 --2- 172.21.15.114:0/932844402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff7c10b390 0x7eff7c116830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.555+0000 7eff82596700 1 -- 172.21.15.114:0/932844402 >> 172.21.15.114:0/932844402 conn(0x7eff7c06f280 msgr2=0x7eff7c06f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:25.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.556+0000 7eff82596700 1 -- 172.21.15.114:0/932844402 shutdown_connections 2024-05-27T07:47:25.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.556+0000 7eff82596700 1 -- 172.21.15.114:0/932844402 wait complete. 2024-05-27T07:47:25.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.557+0000 7eff82596700 1 Processor -- start 2024-05-27T07:47:25.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.557+0000 7eff82596700 1 -- start start 2024-05-27T07:47:25.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.558+0000 7eff82596700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff7c10aa40 0x7eff7c121140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.558+0000 7eff82596700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff7c10b390 0x7eff7c1216a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.558+0000 7eff82596700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff7c116ec0 0x7eff7c11c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.558+0000 7eff82596700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c11a8f0 con 0x7eff7c116ec0 2024-05-27T07:47:25.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.558+0000 7eff82596700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c11a770 con 0x7eff7c10b390 2024-05-27T07:47:25.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.558+0000 7eff82596700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c11aa70 con 0x7eff7c10aa40 2024-05-27T07:47:25.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.558+0000 7eff7b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff7c10b390 0x7eff7c1216a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.558+0000 7eff7b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff7c10b390 0x7eff7c1216a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37322/0 (socket says 172.21.15.114:37322) 2024-05-27T07:47:25.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.558+0000 7eff7b7fe700 1 -- 172.21.15.114:0/853433983 learned_addr learned my addr 172.21.15.114:0/853433983 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:25.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.558+0000 7eff80b33700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff7c116ec0 0x7eff7c11c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.559+0000 7eff7bfff700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff7c10aa40 0x7eff7c121140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.559+0000 7eff7b7fe700 1 -- 172.21.15.114:0/853433983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff7c10aa40 msgr2=0x7eff7c121140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:25.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.559+0000 7eff7b7fe700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff7c10aa40 0x7eff7c121140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.559+0000 7eff7b7fe700 1 -- 172.21.15.114:0/853433983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff7c116ec0 msgr2=0x7eff7c11c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:25.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.559+0000 7eff7b7fe700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff7c116ec0 0x7eff7c11c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.559+0000 7eff7b7fe700 1 -- 172.21.15.114:0/853433983 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7c11ca40 con 0x7eff7c10b390 2024-05-27T07:47:25.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.559+0000 7eff7b7fe700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff7c10b390 0x7eff7c1216a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00e970 tx=0x7eff6c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:25.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.559+0000 7eff797fa700 1 -- 172.21.15.114:0/853433983 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c00cbd0 con 0x7eff7c10b390 2024-05-27T07:47:25.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.560+0000 7eff82596700 1 -- 172.21.15.114:0/853433983 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff7c11cd30 con 0x7eff7c10b390 2024-05-27T07:47:25.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.560+0000 7eff797fa700 1 -- 172.21.15.114:0/853433983 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff6c00cd70 con 0x7eff7c10b390 2024-05-27T07:47:25.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.560+0000 7eff797fa700 1 -- 172.21.15.114:0/853433983 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c005680 con 0x7eff7c10b390 2024-05-27T07:47:25.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.560+0000 7eff82596700 1 -- 172.21.15.114:0/853433983 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff7c1bec90 con 0x7eff7c10b390 2024-05-27T07:47:25.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.562+0000 7eff797fa700 1 -- 172.21.15.114:0/853433983 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7eff6c021440 con 0x7eff7c10b390 2024-05-27T07:47:25.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff797fa700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7eff640b2a20 0x7eff640b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff90fabb80 1 Processor -- start 2024-05-27T07:47:25.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff90fabb80 1 -- start start 2024-05-27T07:47:25.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff7bfff700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7eff640b2a20 0x7eff640b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff797fa700 1 -- 172.21.15.114:0/853433983 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7eff6c0e2c10 con 0x7eff7c10b390 2024-05-27T07:47:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff90fabb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55819645aec0 0x5581965d9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff90fabb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5581965d95d0 0x5581965d99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff90fabb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965e3e60 0x5581965e6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff90fabb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558196507df0 con 0x5581965e3e60 2024-05-27T07:47:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff90fabb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5581964f7990 con 0x55819645aec0 2024-05-27T07:47:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff90fabb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5581964fb9d0 con 0x5581965d95d0 2024-05-27T07:47:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff80b33700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965e3e60 0x5581965e6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff7bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55819645aec0 0x5581965d9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff80b33700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965e3e60 0x5581965e6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48304/0 (socket says 172.21.15.114:48304) 2024-05-27T07:47:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff80b33700 1 -- 172.21.15.114:0/3111069682 learned_addr learned my addr 172.21.15.114:0/3111069682 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff7b7fe700 1 --2- 172.21.15.114:0/3111069682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5581965d95d0 0x5581965d99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.564+0000 7eff7bfff700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7eff640b2a20 0x7eff640b4f30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7eff70009bf0 tx=0x7eff70009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.565+0000 7eff7bfff700 1 -- 172.21.15.114:0/3111069682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5581965d95d0 msgr2=0x5581965d99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:25.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.565+0000 7eff7bfff700 1 --2- 172.21.15.114:0/3111069682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5581965d95d0 0x5581965d99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.565+0000 7eff7bfff700 1 -- 172.21.15.114:0/3111069682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965e3e60 msgr2=0x5581965e6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:25.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.565+0000 7eff7bfff700 1 --2- 172.21.15.114:0/3111069682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965e3e60 0x5581965e6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.565+0000 7eff7bfff700 1 -- 172.21.15.114:0/3111069682 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581964a1030 con 0x55819645aec0 2024-05-27T07:47:25.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.565+0000 7eff7bfff700 1 --2- 172.21.15.114:0/3111069682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55819645aec0 0x5581965d9090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7eff70011770 tx=0x7eff70014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39de9c092432b5e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:25.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.565+0000 7eff627fc700 1 -- 172.21.15.114:0/3111069682 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff70012dc0 con 0x55819645aec0 2024-05-27T07:47:25.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.565+0000 7eff627fc700 1 -- 172.21.15.114:0/3111069682 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff70011e60 con 0x55819645aec0 2024-05-27T07:47:25.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.565+0000 7eff627fc700 1 -- 172.21.15.114:0/3111069682 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff70003df0 con 0x55819645aec0 2024-05-27T07:47:25.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.566+0000 7eff90fabb80 1 -- 172.21.15.114:0/3111069682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55819645aec0 msgr2=0x5581965d9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:25.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.566+0000 7eff90fabb80 1 --2- 172.21.15.114:0/3111069682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55819645aec0 0x5581965d9090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7eff70011770 tx=0x7eff70014000 comp rx=0 tx=0).stop 2024-05-27T07:47:25.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.566+0000 7eff90fabb80 1 -- 172.21.15.114:0/3111069682 shutdown_connections 2024-05-27T07:47:25.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.566+0000 7eff90fabb80 1 --2- 172.21.15.114:0/3111069682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965e3e60 0x5581965e6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.566+0000 7eff90fabb80 1 --2- 172.21.15.114:0/3111069682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55819645aec0 0x5581965d9090 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.566+0000 7eff90fabb80 1 --2- 172.21.15.114:0/3111069682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5581965d95d0 0x5581965d99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.566+0000 7eff90fabb80 1 -- 172.21.15.114:0/3111069682 >> 172.21.15.114:0/3111069682 conn(0x55819645d7f0 msgr2=0x5581965d50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:25.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.566+0000 7eff90fabb80 1 -- 172.21.15.114:0/3111069682 shutdown_connections 2024-05-27T07:47:25.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.566+0000 7eff90fabb80 1 -- 172.21.15.114:0/3111069682 wait complete. 2024-05-27T07:47:25.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.567+0000 7eff90fabb80 1 -- start start 2024-05-27T07:47:25.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.567+0000 7eff90fabb80 1 Processor -- start 2024-05-27T07:47:25.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.567+0000 7eff90fabb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 0x5581965e0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.567+0000 7eff90fabb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965d95d0 0x5581965e12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.567+0000 7eff90fabb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965e3e60 0x5581965e1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.567+0000 7eff90fabb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5581963fe020 con 0x5581965d95d0 2024-05-27T07:47:25.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.567+0000 7eff90fabb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55819645fe20 con 0x5581965e3e60 2024-05-27T07:47:25.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.567+0000 7eff90fabb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5581965e6d80 con 0x55819645aec0 2024-05-27T07:47:25.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.567+0000 7eff7bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 0x5581965e0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff7bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 0x5581965e0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54132/0 (socket says 172.21.15.114:54132) 2024-05-27T07:47:25.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff7bfff700 1 -- 172.21.15.114:0/321204088 learned_addr learned my addr 172.21.15.114:0/321204088 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:25.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff7b7fe700 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965d95d0 0x5581965e12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff80b33700 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965e3e60 0x5581965e1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff90fabb80 10 client.0 _ll_register_callbacks cb 0x55819648e630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:25.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff7bfff700 1 -- 172.21.15.114:0/321204088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965e3e60 msgr2=0x5581965e1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:25.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff7bfff700 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965e3e60 0x5581965e1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff7bfff700 1 -- 172.21.15.114:0/321204088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965d95d0 msgr2=0x5581965e12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:25.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff7bfff700 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965d95d0 0x5581965e12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff7bfff700 1 -- 172.21.15.114:0/321204088 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558196447950 con 0x55819645aec0 2024-05-27T07:47:25.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff7b7fe700 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965d95d0 0x5581965e12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:25.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff7bfff700 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 0x5581965e0da0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7eff70011a20 tx=0x7eff7001faa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21b5a4ae31c74a2c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:25.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff5b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:47:25.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff5b7fe700 1 -- 172.21.15.114:0/321204088 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff70037060 con 0x55819645aec0 2024-05-27T07:47:25.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff5b7fe700 1 -- 172.21.15.114:0/321204088 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7eff70012c40 con 0x55819645aec0 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.568+0000 7eff90fabb80 1 -- 172.21.15.114:0/321204088 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff70011e60 con 0x55819645aec0 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.569+0000 7eff5b7fe700 1 -- 172.21.15.114:0/321204088 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7eff7003e050 con 0x55819645aec0 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.569+0000 7eff90fabb80 10 client.26779 fetch_fsmap learned FSMap version 45 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.569+0000 7eff90fabb80 1 -- 172.21.15.114:0/321204088 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5581964b17e0 con 0x55819645aec0 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.569+0000 7eff5b7fe700 1 -- 172.21.15.114:0/321204088 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff70011e60 con 0x55819645aec0 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.569+0000 7eff5b7fe700 1 -- 172.21.15.114:0/321204088 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 45) ==== 7725+0+0 (secure 0 0 0) 0x7eff700263b0 con 0x55819645aec0 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.569+0000 7eff90fabb80 10 client.26779 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.569+0000 7eff90fabb80 10 client.26779 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.569+0000 7eff90fabb80 20 client.26779 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.569+0000 7eff90fabb80 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5581965f3b60 0x5581965f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.569+0000 7eff90fabb80 4 client.26779 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.569+0000 7eff90fabb80 1 -- 172.21.15.114:0/321204088 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558196509d40 con 0x5581965f3b60 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.569+0000 7eff7b7fe700 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5581965f3b60 0x5581965f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.570+0000 7eff5b7fe700 10 client.26779 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:25.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.570+0000 7eff7b7fe700 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5581965f3b60 0x5581965f5f80 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c68b4ef9adb1dcc4 server_cookie=57b51a2f06c0ff26 in_seq=0 out_seq=0 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.570+0000 7eff5b7fe700 1 -- 172.21.15.114:0/321204088 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558196509d40 con 0x5581965f3b60 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.570+0000 7eff5b7fe700 10 client.26779 handle_command_reply: tid=0 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.570+0000 7eff90fabb80 1 client.26779 shutdown 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.571+0000 7eff90fabb80 1 -- 172.21.15.114:0/321204088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 msgr2=0x5581965e0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.571+0000 7eff90fabb80 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 0x5581965e0da0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7eff70011a20 tx=0x7eff7001faa0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.571+0000 7eff90fabb80 1 -- 172.21.15.114:0/321204088 shutdown_connections 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.571+0000 7eff90fabb80 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5581965f3b60 0x5581965f5f80 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.571+0000 7eff5b7fe700 0 client.26779 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.571+0000 7eff90fabb80 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965e3e60 0x5581965e1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.571+0000 7eff90fabb80 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 0x5581965e0da0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.571+0000 7eff90fabb80 1 --2- 172.21.15.114:0/321204088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965d95d0 0x5581965e12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.571+0000 7eff90fabb80 1 -- 172.21.15.114:0/321204088 >> 172.21.15.114:0/321204088 conn(0x55819645d7f0 msgr2=0x5581965d3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.571+0000 7eff90fabb80 1 -- 172.21.15.114:0/321204088 shutdown_connections 2024-05-27T07:47:25.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.571+0000 7eff90fabb80 1 -- 172.21.15.114:0/321204088 wait complete. 2024-05-27T07:47:25.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.571+0000 7eff90fabb80 20 client.26779 trim_cache size 0 max 16384 2024-05-27T07:47:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.593+0000 7eff90fabb80 1 Processor -- start 2024-05-27T07:47:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.593+0000 7eff90fabb80 1 -- start start 2024-05-27T07:47:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.593+0000 7eff90fabb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 0x5581965efe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.593+0000 7eff90fabb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965d95d0 0x5581965e2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.593+0000 7eff90fabb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965e3e60 0x5581965e2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.593+0000 7eff90fabb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1c8a10 con 0x5581965e3e60 2024-05-27T07:47:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.593+0000 7eff90fabb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1c5ac0 con 0x5581965d95d0 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.593+0000 7eff90fabb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1c7c70 con 0x55819645aec0 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.593+0000 7eff80b33700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965e3e60 0x5581965e2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.593+0000 7eff7bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 0x5581965efe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.593+0000 7eff7bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 0x5581965efe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54140/0 (socket says 172.21.15.114:54140) 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.593+0000 7eff80b33700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965e3e60 0x5581965e2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48336/0 (socket says 172.21.15.114:48336) 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.594+0000 7eff80b33700 1 -- 172.21.15.114:0/1391561861 learned_addr learned my addr 172.21.15.114:0/1391561861 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.594+0000 7eff7b7fe700 1 --2- 172.21.15.114:0/1391561861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965d95d0 0x5581965e2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.594+0000 7eff80b33700 1 -- 172.21.15.114:0/1391561861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 msgr2=0x5581965efe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.594+0000 7eff80b33700 1 --2- 172.21.15.114:0/1391561861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 0x5581965efe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.594+0000 7eff80b33700 1 -- 172.21.15.114:0/1391561861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965d95d0 msgr2=0x5581965e2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.594+0000 7eff80b33700 1 --2- 172.21.15.114:0/1391561861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965d95d0 0x5581965e2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.594+0000 7eff80b33700 1 -- 172.21.15.114:0/1391561861 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581965dc990 con 0x5581965e3e60 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.594+0000 7eff7bfff700 1 --2- 172.21.15.114:0/1391561861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 0x5581965efe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.594+0000 7eff80b33700 1 --2- 172.21.15.114:0/1391561861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965e3e60 0x5581965e2f50 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7eff7400d010 tx=0x7eff74009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c694ea919d7490 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.594+0000 7eff61ffb700 1 -- 172.21.15.114:0/1391561861 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7400bb30 con 0x5581965e3e60 2024-05-27T07:47:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.594+0000 7eff61ffb700 1 -- 172.21.15.114:0/1391561861 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff7400bcd0 con 0x5581965e3e60 2024-05-27T07:47:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.594+0000 7eff61ffb700 1 -- 172.21.15.114:0/1391561861 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff74007d80 con 0x5581965e3e60 2024-05-27T07:47:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.595+0000 7eff90fabb80 1 -- 172.21.15.114:0/1391561861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965e3e60 msgr2=0x5581965e2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.595+0000 7eff90fabb80 1 --2- 172.21.15.114:0/1391561861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965e3e60 0x5581965e2f50 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7eff7400d010 tx=0x7eff74009520 comp rx=0 tx=0).stop 2024-05-27T07:47:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.595+0000 7eff90fabb80 1 -- 172.21.15.114:0/1391561861 shutdown_connections 2024-05-27T07:47:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.595+0000 7eff90fabb80 1 --2- 172.21.15.114:0/1391561861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5581965e3e60 0x5581965e2f50 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.595+0000 7eff90fabb80 1 --2- 172.21.15.114:0/1391561861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55819645aec0 0x5581965efe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.595+0000 7eff90fabb80 1 --2- 172.21.15.114:0/1391561861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965d95d0 0x5581965e2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.595+0000 7eff90fabb80 1 -- 172.21.15.114:0/1391561861 >> 172.21.15.114:0/1391561861 conn(0x55819645d7f0 msgr2=0x5581965e2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.595+0000 7eff90fabb80 1 -- 172.21.15.114:0/1391561861 shutdown_connections 2024-05-27T07:47:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.596+0000 7eff90fabb80 1 -- 172.21.15.114:0/1391561861 wait complete. 2024-05-27T07:47:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.596+0000 7eff90fabb80 1 -- start start 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.596+0000 7eff90fabb80 1 Processor -- start 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.596+0000 7eff90fabb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55819645aec0 0x5581965ec760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.596+0000 7eff90fabb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965d95d0 0x5581965e8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.596+0000 7eff90fabb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5581965e3e60 0x5581965e9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.596+0000 7eff90fabb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1c75a0 con 0x55819645aec0 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.596+0000 7eff90fabb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1c6130 con 0x5581965d95d0 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.596+0000 7eff90fabb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1c6800 con 0x5581965e3e60 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff90fabb80 10 client.0 _ll_register_callbacks cb 0x55819648e630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff7bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55819645aec0 0x5581965ec760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff7bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55819645aec0 0x5581965ec760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48342/0 (socket says 172.21.15.114:48342) 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff7b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965d95d0 0x5581965e8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff7bfff700 1 -- 172.21.15.114:0/1148425178 learned_addr learned my addr 172.21.15.114:0/1148425178 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff80b33700 1 --2- 172.21.15.114:0/1148425178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5581965e3e60 0x5581965e9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff7bfff700 1 -- 172.21.15.114:0/1148425178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5581965e3e60 msgr2=0x5581965e9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff7bfff700 1 --2- 172.21.15.114:0/1148425178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5581965e3e60 0x5581965e9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff7bfff700 1 -- 172.21.15.114:0/1148425178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965d95d0 msgr2=0x5581965e8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff7bfff700 1 --2- 172.21.15.114:0/1148425178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965d95d0 0x5581965e8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff7bfff700 1 -- 172.21.15.114:0/1148425178 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581965e97b0 con 0x55819645aec0 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff7b7fe700 1 --2- 172.21.15.114:0/1148425178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965d95d0 0x5581965e8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff7bfff700 1 --2- 172.21.15.114:0/1148425178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55819645aec0 0x5581965ec760 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7eff70011d70 tx=0x7eff70009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec950baea816849 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff5b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff5b7fe700 1 -- 172.21.15.114:0/1148425178 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff70037040 con 0x55819645aec0 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff5b7fe700 1 -- 172.21.15.114:0/1148425178 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7eff700269f0 con 0x55819645aec0 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.597+0000 7eff90fabb80 1 -- 172.21.15.114:0/1148425178 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff7400bcd0 con 0x55819645aec0 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.598+0000 7eff5b7fe700 1 -- 172.21.15.114:0/1148425178 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7eff70049050 con 0x55819645aec0 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.598+0000 7eff90fabb80 10 client.17217 fetch_fsmap learned FSMap version 45 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.598+0000 7eff90fabb80 1 -- 172.21.15.114:0/1148425178 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55819665f400 con 0x55819645aec0 2024-05-27T07:47:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.598+0000 7eff5b7fe700 1 -- 172.21.15.114:0/1148425178 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff7400bcd0 con 0x55819645aec0 2024-05-27T07:47:25.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.598+0000 7eff5b7fe700 1 -- 172.21.15.114:0/1148425178 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 45) ==== 7725+0+0 (secure 0 0 0) 0x7eff70005380 con 0x55819645aec0 2024-05-27T07:47:25.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.598+0000 7eff90fabb80 10 client.17217 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:47:25.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.598+0000 7eff90fabb80 10 client.17217 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:25.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.598+0000 7eff90fabb80 20 client.17217 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:25.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.598+0000 7eff90fabb80 1 --2- 172.21.15.114:0/1148425178 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5581965f3b60 0x558196661a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:25.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.598+0000 7eff90fabb80 4 client.17217 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.598+0000 7eff90fabb80 1 -- 172.21.15.114:0/1148425178 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558196503520 con 0x5581965f3b60 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.598+0000 7eff7b7fe700 1 --2- 172.21.15.114:0/1148425178 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5581965f3b60 0x558196661a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.599+0000 7eff5b7fe700 10 client.17217 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.599+0000 7eff7b7fe700 1 --2- 172.21.15.114:0/1148425178 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5581965f3b60 0x558196661a20 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b68b4f3e0de7e7b server_cookie=a168ce84c9fc328e in_seq=0 out_seq=0 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.600+0000 7eff5b7fe700 1 -- 172.21.15.114:0/1148425178 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558196503520 con 0x5581965f3b60 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.600+0000 7eff5b7fe700 10 client.17217 handle_command_reply: tid=0 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.600+0000 7eff90fabb80 1 client.17217 shutdown 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.601+0000 7eff90fabb80 1 -- 172.21.15.114:0/1148425178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55819645aec0 msgr2=0x5581965ec760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.601+0000 7eff90fabb80 1 --2- 172.21.15.114:0/1148425178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55819645aec0 0x5581965ec760 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7eff70011d70 tx=0x7eff70009760 comp rx=0 tx=0).stop 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.601+0000 7eff90fabb80 1 -- 172.21.15.114:0/1148425178 shutdown_connections 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.601+0000 7eff90fabb80 1 --2- 172.21.15.114:0/1148425178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5581965e3e60 0x5581965e9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.601+0000 7eff90fabb80 1 --2- 172.21.15.114:0/1148425178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55819645aec0 0x5581965ec760 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.601+0000 7eff90fabb80 1 --2- 172.21.15.114:0/1148425178 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5581965f3b60 0x558196661a20 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.601+0000 7eff90fabb80 1 --2- 172.21.15.114:0/1148425178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581965d95d0 0x5581965e8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.601+0000 7eff90fabb80 1 -- 172.21.15.114:0/1148425178 >> 172.21.15.114:0/1148425178 conn(0x55819645d7f0 msgr2=0x5581965e2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:25.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.601+0000 7eff5b7fe700 0 client.17217 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:25.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.601+0000 7eff90fabb80 1 -- 172.21.15.114:0/1148425178 shutdown_connections 2024-05-27T07:47:25.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.601+0000 7eff90fabb80 1 -- 172.21.15.114:0/1148425178 wait complete. 2024-05-27T07:47:25.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.601+0000 7eff90fabb80 20 client.17217 trim_cache size 0 max 16384 2024-05-27T07:47:25.604 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:47:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:47:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "206db555-2081-4131-97e7-3ccd1673df81", 2024-05-27T07:47:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:47:25.604 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:47:25.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.602+0000 7eff82596700 1 -- 172.21.15.114:0/853433983 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7eff640b2a20 msgr2=0x7eff640b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:25.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.602+0000 7eff82596700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7eff640b2a20 0x7eff640b4f30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7eff70009bf0 tx=0x7eff70009260 comp rx=0 tx=0).stop 2024-05-27T07:47:25.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.602+0000 7eff82596700 1 -- 172.21.15.114:0/853433983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff7c10b390 msgr2=0x7eff7c1216a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:25.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.602+0000 7eff82596700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff7c10b390 0x7eff7c1216a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00e970 tx=0x7eff6c00ec80 comp rx=0 tx=0).stop 2024-05-27T07:47:25.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.602+0000 7eff82596700 1 -- 172.21.15.114:0/853433983 shutdown_connections 2024-05-27T07:47:25.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.602+0000 7eff82596700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff7c116ec0 0x7eff7c11c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.602+0000 7eff82596700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff7c10aa40 0x7eff7c121140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.602+0000 7eff82596700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7eff640b2a20 0x7eff640b4f30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.602+0000 7eff82596700 1 --2- 172.21.15.114:0/853433983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff7c10b390 0x7eff7c1216a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:25.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.602+0000 7eff82596700 1 -- 172.21.15.114:0/853433983 >> 172.21.15.114:0/853433983 conn(0x7eff7c06f280 msgr2=0x7eff7c074510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:25.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.602+0000 7eff82596700 1 -- 172.21.15.114:0/853433983 shutdown_connections 2024-05-27T07:47:25.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:25.602+0000 7eff82596700 1 -- 172.21.15.114:0/853433983 wait complete. 2024-05-27T07:47:25.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:25 smithi163 ceph-mon[36451]: pgmap v248: 129 pgs: 129 active+clean; 63 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 373 op/s 2024-05-27T07:47:25.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:25 smithi186 ceph-mon[36647]: pgmap v248: 129 pgs: 129 active+clean; 63 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 373 op/s 2024-05-27T07:47:26.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 774734 11.58 MB/sec execute 218 sec latency 17.635 ms 2024-05-27T07:47:26.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:47:26.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:26 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:47:26.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:26 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:26.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:26 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:47:26.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:26 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:26.697 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:47:26.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:26 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:47:26.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:26 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:26.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:26 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:47:26.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:26 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:26.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:26 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:47:26.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:26 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:26.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:26 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:47:26.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:26 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:26.728 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:26 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:47:26.728 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:26 smithi186 systemd[1]: Stopped Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:47:26.728 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:26 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Start request repeated too quickly. 2024-05-27T07:47:26.728 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:26 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:47:26.729 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 27 07:47:26 smithi186 systemd[1]: Failed to start Ceph mds.f for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:47:27.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 776478 11.56 MB/sec execute 219 sec latency 85.647 ms 2024-05-27T07:47:27.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:27 smithi114 ceph-mon[32038]: pgmap v249: 129 pgs: 129 active+clean; 69 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 428 op/s 2024-05-27T07:47:27.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:27 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:47:27.625 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:27 smithi163 ceph-mon[36451]: pgmap v249: 129 pgs: 129 active+clean; 69 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 428 op/s 2024-05-27T07:47:27.626 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:27 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:47:27.628 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:27 smithi186 ceph-mon[36647]: pgmap v249: 129 pgs: 129 active+clean; 69 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 428 op/s 2024-05-27T07:47:27.629 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:27 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:47:28.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 778267 11.56 MB/sec execute 220 sec latency 21.145 ms 2024-05-27T07:47:28.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.672+0000 7fd490a02700 1 -- 172.21.15.114:0/1032869869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd48c105300 msgr2=0x7fd48c105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:28.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.672+0000 7fd490a02700 1 --2- 172.21.15.114:0/1032869869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd48c105300 0x7fd48c105710 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd47c009990 tx=0x7fd47c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.672+0000 7fd490a02700 1 -- 172.21.15.114:0/1032869869 shutdown_connections 2024-05-27T07:47:28.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.672+0000 7fd490a02700 1 --2- 172.21.15.114:0/1032869869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd48c10e730 0x7fd48c110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.672+0000 7fd490a02700 1 --2- 172.21.15.114:0/1032869869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd48c105300 0x7fd48c105710 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.672+0000 7fd490a02700 1 --2- 172.21.15.114:0/1032869869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd48c105c50 0x7fd48c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.672+0000 7fd490a02700 1 -- 172.21.15.114:0/1032869869 >> 172.21.15.114:0/1032869869 conn(0x7fd48c078ab0 msgr2=0x7fd48c078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:28.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.673+0000 7fd490a02700 1 -- 172.21.15.114:0/1032869869 shutdown_connections 2024-05-27T07:47:28.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.673+0000 7fd490a02700 1 -- 172.21.15.114:0/1032869869 wait complete. 2024-05-27T07:47:28.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.673+0000 7fd490a02700 1 Processor -- start 2024-05-27T07:47:28.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd490a02700 1 -- start start 2024-05-27T07:47:28.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd490a02700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd48c105300 0x7fd48c1a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd490a02700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd48c105c50 0x7fd48c1a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd490a02700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd48c10e730 0x7fd48c19aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd490a02700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c114240 con 0x7fd48c105300 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd490a02700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c1140c0 con 0x7fd48c10e730 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd490a02700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c1143c0 con 0x7fd48c105c50 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd48ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd48c105300 0x7fd48c1a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd48ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd48c105300 0x7fd48c1a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48360/0 (socket says 172.21.15.114:48360) 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd48ad9d700 1 -- 172.21.15.114:0/3425737943 learned_addr learned my addr 172.21.15.114:0/3425737943 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd48a59c700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd48c105c50 0x7fd48c1a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd48b59e700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd48c10e730 0x7fd48c19aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd48ad9d700 1 -- 172.21.15.114:0/3425737943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd48c105c50 msgr2=0x7fd48c1a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd48c105c50 0x7fd48c1a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd48ad9d700 1 -- 172.21.15.114:0/3425737943 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd48c10e730 msgr2=0x7fd48c19aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd48c10e730 0x7fd48c19aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.674+0000 7fd48ad9d700 1 -- 172.21.15.114:0/3425737943 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd48c19b0b0 con 0x7fd48c105300 2024-05-27T07:47:28.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.675+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd48c105300 0x7fd48c1a07e0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd47c000c00 tx=0x7fd47c00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:28.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.675+0000 7fd473fff700 1 -- 172.21.15.114:0/3425737943 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c027070 con 0x7fd48c105300 2024-05-27T07:47:28.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.675+0000 7fd490a02700 1 -- 172.21.15.114:0/3425737943 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd48c19b340 con 0x7fd48c105300 2024-05-27T07:47:28.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.675+0000 7fd473fff700 1 -- 172.21.15.114:0/3425737943 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd47c022470 con 0x7fd48c105300 2024-05-27T07:47:28.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.675+0000 7fd473fff700 1 -- 172.21.15.114:0/3425737943 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c017650 con 0x7fd48c105300 2024-05-27T07:47:28.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.675+0000 7fd490a02700 1 -- 172.21.15.114:0/3425737943 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd48c1a7670 con 0x7fd48c105300 2024-05-27T07:47:28.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.678+0000 7fd473fff700 1 -- 172.21.15.114:0/3425737943 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fd47c02a070 con 0x7fd48c105300 2024-05-27T07:47:28.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.678+0000 7fd473fff700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd4740b29f0 0x7fd4740b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd48a59c700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd4740b29f0 0x7fd4740b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd473fff700 1 -- 172.21.15.114:0/3425737943 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fd47c0e3a70 con 0x7fd48c105300 2024-05-27T07:47:28.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd49fc18b80 1 Processor -- start 2024-05-27T07:47:28.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd49fc18b80 1 -- start start 2024-05-27T07:47:28.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd48a59c700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd4740b29f0 0x7fd4740b4f00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd48c102580 tx=0x7fd480009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:28.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd49fc18b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 0x559dee721010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd49fc18b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 0x559dee721960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd49fc18b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee72e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd49fc18b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559dee64fd70 con 0x559dee5a2e40 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd49fc18b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559dee63f910 con 0x559dee72bde0 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd49fc18b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559dee643950 con 0x559dee721550 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd48ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 0x559dee721010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd48a59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 0x559dee721960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd48ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 0x559dee721010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48362/0 (socket says 172.21.15.114:48362) 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd48ad9d700 1 -- 172.21.15.114:0/2771674031 learned_addr learned my addr 172.21.15.114:0/2771674031 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.679+0000 7fd48b59e700 1 --2- 172.21.15.114:0/2771674031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee72e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.680+0000 7fd48ad9d700 1 -- 172.21.15.114:0/2771674031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 msgr2=0x559dee721960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.680+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/2771674031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 0x559dee721960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.680+0000 7fd48ad9d700 1 -- 172.21.15.114:0/2771674031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 msgr2=0x559dee72e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.680+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/2771674031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee72e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.680+0000 7fd48ad9d700 1 -- 172.21.15.114:0/2771674031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dee5e8fb0 con 0x559dee5a2e40 2024-05-27T07:47:28.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.680+0000 7fd48b59e700 1 --2- 172.21.15.114:0/2771674031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee72e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:28.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.680+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/2771674031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 0x559dee721010 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd47c0a6ab0 tx=0x7fd47c0aa220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96796ba3212fec02 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:28.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.680+0000 7fd4717fa700 1 -- 172.21.15.114:0/2771674031 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c0ab130 con 0x559dee5a2e40 2024-05-27T07:47:28.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.680+0000 7fd4717fa700 1 -- 172.21.15.114:0/2771674031 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd47c0aaa30 con 0x559dee5a2e40 2024-05-27T07:47:28.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.680+0000 7fd4717fa700 1 -- 172.21.15.114:0/2771674031 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c0cd0b0 con 0x559dee5a2e40 2024-05-27T07:47:28.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.681+0000 7fd49fc18b80 1 -- 172.21.15.114:0/2771674031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 msgr2=0x559dee721010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.681+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/2771674031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 0x559dee721010 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd47c0a6ab0 tx=0x7fd47c0aa220 comp rx=0 tx=0).stop 2024-05-27T07:47:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.681+0000 7fd49fc18b80 1 -- 172.21.15.114:0/2771674031 shutdown_connections 2024-05-27T07:47:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.681+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/2771674031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee72e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.681+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/2771674031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 0x559dee721010 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.681+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/2771674031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 0x559dee721960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.681+0000 7fd49fc18b80 1 -- 172.21.15.114:0/2771674031 >> 172.21.15.114:0/2771674031 conn(0x559dee5a5770 msgr2=0x559dee71d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.681+0000 7fd49fc18b80 1 -- 172.21.15.114:0/2771674031 shutdown_connections 2024-05-27T07:47:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.681+0000 7fd49fc18b80 1 -- 172.21.15.114:0/2771674031 wait complete. 2024-05-27T07:47:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.681+0000 7fd49fc18b80 1 -- start start 2024-05-27T07:47:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd49fc18b80 1 Processor -- start 2024-05-27T07:47:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd49fc18b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 0x559dee728ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd49fc18b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 0x559dee729430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd49fc18b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee729970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd49fc18b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559dee545fa0 con 0x559dee5a2e40 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd49fc18b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559dee5a7da0 con 0x559dee72bde0 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd49fc18b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559dee72eac0 con 0x559dee721550 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd49fc18b80 10 client.0 _ll_register_callbacks cb 0x559dee5d65b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd48ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 0x559dee728ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd48ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 0x559dee728ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48366/0 (socket says 172.21.15.114:48366) 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd48ad9d700 1 -- 172.21.15.114:0/1411558816 learned_addr learned my addr 172.21.15.114:0/1411558816 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd48b59e700 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee729970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd48a59c700 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 0x559dee729430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd48ad9d700 1 -- 172.21.15.114:0/1411558816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 msgr2=0x559dee729430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 0x559dee729430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd48ad9d700 1 -- 172.21.15.114:0/1411558816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 msgr2=0x559dee729970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee729970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd48ad9d700 1 -- 172.21.15.114:0/1411558816 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dee58f8d0 con 0x559dee5a2e40 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.682+0000 7fd48b59e700 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee729970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.683+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 0x559dee728ef0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd47c00b6c0 tx=0x7fd47c022ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9161817eeb75cc61 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.683+0000 7fd46eff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.683+0000 7fd46eff5700 1 -- 172.21.15.114:0/1411558816 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c0c3020 con 0x559dee5a2e40 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.683+0000 7fd46eff5700 1 -- 172.21.15.114:0/1411558816 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fd47c0b10d0 con 0x559dee5a2e40 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.683+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1411558816 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd47c0aaa30 con 0x559dee5a2e40 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.683+0000 7fd46eff5700 1 -- 172.21.15.114:0/1411558816 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fd47c0aede0 con 0x559dee5a2e40 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.683+0000 7fd49fc18b80 10 client.17229 fetch_fsmap learned FSMap version 45 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.683+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1411558816 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x559dee5f9760 con 0x559dee5a2e40 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.683+0000 7fd46eff5700 1 -- 172.21.15.114:0/1411558816 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd47c0aaa30 con 0x559dee5a2e40 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.683+0000 7fd46eff5700 1 -- 172.21.15.114:0/1411558816 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 45) ==== 7725+0+0 (secure 0 0 0) 0x7fd47c0c7070 con 0x559dee5a2e40 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.683+0000 7fd49fc18b80 10 client.17229 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:47:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.683+0000 7fd49fc18b80 10 client.17229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.683+0000 7fd49fc18b80 20 client.17229 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.684+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559dee73b900 0x559dee73dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.684+0000 7fd49fc18b80 4 client.17229 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.684+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1411558816 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559dee651cc0 con 0x559dee73b900 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.684+0000 7fd48a59c700 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559dee73b900 0x559dee73dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.684+0000 7fd46eff5700 10 client.17229 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.684+0000 7fd48a59c700 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559dee73b900 0x559dee73dd20 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=452efc4e8ab267c1 server_cookie=82fb540c78ab0303 in_seq=0 out_seq=0 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.685+0000 7fd46eff5700 1 -- 172.21.15.114:0/1411558816 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559dee651cc0 con 0x559dee73b900 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.685+0000 7fd46eff5700 10 client.17229 handle_command_reply: tid=0 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.685+0000 7fd49fc18b80 1 client.17229 shutdown 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.685+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1411558816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 msgr2=0x559dee728ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.685+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 0x559dee728ef0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd47c00b6c0 tx=0x7fd47c022ce0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.685+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1411558816 shutdown_connections 2024-05-27T07:47:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.685+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559dee73b900 0x559dee73dd20 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.685+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee729970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.685+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee5a2e40 0x559dee728ef0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.685+0000 7fd46eff5700 0 client.17229 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.685+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/1411558816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 0x559dee729430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.685+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1411558816 >> 172.21.15.114:0/1411558816 conn(0x559dee5a5770 msgr2=0x559dee71b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.686+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1411558816 shutdown_connections 2024-05-27T07:47:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.686+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1411558816 wait complete. 2024-05-27T07:47:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.686+0000 7fd49fc18b80 20 client.17229 trim_cache size 0 max 16384 2024-05-27T07:47:28.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.707+0000 7fd49fc18b80 1 Processor -- start 2024-05-27T07:47:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.707+0000 7fd49fc18b80 1 -- start start 2024-05-27T07:47:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd49fc18b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee5a2e40 0x559dee723630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd49fc18b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee721550 0x559dee7433d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd49fc18b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee743910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd49fc18b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c1b1280 con 0x559dee721550 2024-05-27T07:47:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd49fc18b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c1ae330 con 0x559dee72bde0 2024-05-27T07:47:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd49fc18b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c1b04e0 con 0x559dee5a2e40 2024-05-27T07:47:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd48a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee721550 0x559dee7433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd48a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee721550 0x559dee7433d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48372/0 (socket says 172.21.15.114:48372) 2024-05-27T07:47:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd48a59c700 1 -- 172.21.15.114:0/3016927626 learned_addr learned my addr 172.21.15.114:0/3016927626 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/3016927626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee5a2e40 0x559dee723630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd48b59e700 1 --2- 172.21.15.114:0/3016927626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee743910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd48a59c700 1 -- 172.21.15.114:0/3016927626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee5a2e40 msgr2=0x559dee723630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd48a59c700 1 --2- 172.21.15.114:0/3016927626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee5a2e40 0x559dee723630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd48a59c700 1 -- 172.21.15.114:0/3016927626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 msgr2=0x559dee743910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd48a59c700 1 --2- 172.21.15.114:0/3016927626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee743910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd48a59c700 1 -- 172.21.15.114:0/3016927626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dee72b790 con 0x559dee721550 2024-05-27T07:47:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd48a59c700 1 --2- 172.21.15.114:0/3016927626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee721550 0x559dee7433d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd480013010 tx=0x7fd480014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=169aa788ac6fe171 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd470ff9700 1 -- 172.21.15.114:0/3016927626 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd480003a20 con 0x559dee721550 2024-05-27T07:47:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd470ff9700 1 -- 172.21.15.114:0/3016927626 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd480003bc0 con 0x559dee721550 2024-05-27T07:47:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.708+0000 7fd470ff9700 1 -- 172.21.15.114:0/3016927626 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48001d670 con 0x559dee721550 2024-05-27T07:47:28.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.709+0000 7fd49fc18b80 1 -- 172.21.15.114:0/3016927626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee721550 msgr2=0x559dee7433d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:28.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.709+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/3016927626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee721550 0x559dee7433d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd480013010 tx=0x7fd480014000 comp rx=0 tx=0).stop 2024-05-27T07:47:28.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.709+0000 7fd49fc18b80 1 -- 172.21.15.114:0/3016927626 shutdown_connections 2024-05-27T07:47:28.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.709+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/3016927626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee72bde0 0x559dee743910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.709+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/3016927626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee5a2e40 0x559dee723630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.709+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/3016927626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee721550 0x559dee7433d0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.709+0000 7fd49fc18b80 1 -- 172.21.15.114:0/3016927626 >> 172.21.15.114:0/3016927626 conn(0x559dee5a5770 msgr2=0x559dee729220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.709+0000 7fd49fc18b80 1 -- 172.21.15.114:0/3016927626 shutdown_connections 2024-05-27T07:47:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.709+0000 7fd49fc18b80 1 -- 172.21.15.114:0/3016927626 wait complete. 2024-05-27T07:47:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd49fc18b80 1 -- start start 2024-05-27T07:47:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd49fc18b80 1 Processor -- start 2024-05-27T07:47:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd49fc18b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee5a2e40 0x559dee72ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd49fc18b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 0x559dee72d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd49fc18b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee72bde0 0x559dee72d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd49fc18b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c1afe10 con 0x559dee72bde0 2024-05-27T07:47:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd49fc18b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c1ae9a0 con 0x559dee5a2e40 2024-05-27T07:47:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd49fc18b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c1af070 con 0x559dee721550 2024-05-27T07:47:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd49fc18b80 10 client.0 _ll_register_callbacks cb 0x559dee5d65b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd48b59e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee72bde0 0x559dee72d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd48b59e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee72bde0 0x559dee72d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48386/0 (socket says 172.21.15.114:48386) 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd48b59e700 1 -- 172.21.15.114:0/1322872993 learned_addr learned my addr 172.21.15.114:0/1322872993 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd48a59c700 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 0x559dee72d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.710+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee5a2e40 0x559dee72ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd48b59e700 1 -- 172.21.15.114:0/1322872993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 msgr2=0x559dee72d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd48b59e700 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 0x559dee72d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd48b59e700 1 -- 172.21.15.114:0/1322872993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee5a2e40 msgr2=0x559dee72ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd48b59e700 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee5a2e40 0x559dee72ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd48b59e700 1 -- 172.21.15.114:0/1322872993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dee7a52a0 con 0x559dee72bde0 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee5a2e40 0x559dee72ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd48b59e700 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee72bde0 0x559dee72d950 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd484008d70 tx=0x7fd484004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb41d3feb098bd4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd46eff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd46eff5700 1 -- 172.21.15.114:0/1322872993 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd484004bb0 con 0x559dee72bde0 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd46eff5700 1 -- 172.21.15.114:0/1322872993 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fd484004d50 con 0x559dee72bde0 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1322872993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd480003bc0 con 0x559dee72bde0 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd46eff5700 1 -- 172.21.15.114:0/1322872993 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fd48402b050 con 0x559dee72bde0 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd49fc18b80 10 client.17235 fetch_fsmap learned FSMap version 45 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1322872993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x559dee7a64e0 con 0x559dee72bde0 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.711+0000 7fd46eff5700 1 -- 172.21.15.114:0/1322872993 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd480003bc0 con 0x559dee72bde0 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.712+0000 7fd46eff5700 1 -- 172.21.15.114:0/1322872993 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 45) ==== 7725+0+0 (secure 0 0 0) 0x7fd484016400 con 0x559dee72bde0 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.712+0000 7fd49fc18b80 10 client.17235 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.712+0000 7fd49fc18b80 10 client.17235 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.712+0000 7fd49fc18b80 20 client.17235 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.712+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559dee7a6c70 0x559dee7a9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.712+0000 7fd49fc18b80 4 client.17235 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.712+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1322872993 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559dee738280 con 0x559dee7a6c70 2024-05-27T07:47:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.712+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559dee7a6c70 0x559dee7a9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.712+0000 7fd48ad9d700 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559dee7a6c70 0x559dee7a9090 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca425f1c6dde73b2 server_cookie=5ded74cba9698dea in_seq=0 out_seq=0 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.712+0000 7fd46eff5700 10 client.17235 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd46eff5700 1 -- 172.21.15.114:0/1322872993 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559dee738280 con 0x559dee7a6c70 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd46eff5700 10 client.17235 handle_command_reply: tid=0 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd49fc18b80 1 client.17235 shutdown 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1322872993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee72bde0 msgr2=0x559dee72d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee72bde0 0x559dee72d950 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd484008d70 tx=0x7fd484004320 comp rx=0 tx=0).stop 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1322872993 shutdown_connections 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559dee72bde0 0x559dee72d950 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559dee7a6c70 0x559dee7a9090 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559dee5a2e40 0x559dee72ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd49fc18b80 1 --2- 172.21.15.114:0/1322872993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559dee721550 0x559dee72d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1322872993 >> 172.21.15.114:0/1322872993 conn(0x559dee5a5770 msgr2=0x559dee736b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd46eff5700 0 client.17235 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:28.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1322872993 shutdown_connections 2024-05-27T07:47:28.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd49fc18b80 1 -- 172.21.15.114:0/1322872993 wait complete. 2024-05-27T07:47:28.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.713+0000 7fd49fc18b80 20 client.17235 trim_cache size 0 max 16384 2024-05-27T07:47:28.717 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:47:28.717 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:47:28.717 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:47:28.717 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:47:28.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.714+0000 7fd490a02700 1 -- 172.21.15.114:0/3425737943 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd4740b29f0 msgr2=0x7fd4740b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:28.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.714+0000 7fd490a02700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd4740b29f0 0x7fd4740b4f00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd48c102580 tx=0x7fd480009260 comp rx=0 tx=0).stop 2024-05-27T07:47:28.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.714+0000 7fd490a02700 1 -- 172.21.15.114:0/3425737943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd48c105300 msgr2=0x7fd48c1a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:28.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.714+0000 7fd490a02700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd48c105300 0x7fd48c1a07e0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd47c000c00 tx=0x7fd47c00bfd0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.714+0000 7fd490a02700 1 -- 172.21.15.114:0/3425737943 shutdown_connections 2024-05-27T07:47:28.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.714+0000 7fd490a02700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd4740b29f0 0x7fd4740b4f00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.714+0000 7fd490a02700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd48c10e730 0x7fd48c19aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.714+0000 7fd490a02700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd48c105300 0x7fd48c1a07e0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.714+0000 7fd490a02700 1 --2- 172.21.15.114:0/3425737943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd48c105c50 0x7fd48c1a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:28.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.714+0000 7fd490a02700 1 -- 172.21.15.114:0/3425737943 >> 172.21.15.114:0/3425737943 conn(0x7fd48c078ab0 msgr2=0x7fd48c0ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:28.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.714+0000 7fd490a02700 1 -- 172.21.15.114:0/3425737943 shutdown_connections 2024-05-27T07:47:28.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:28.714+0000 7fd490a02700 1 -- 172.21.15.114:0/3425737943 wait complete. 2024-05-27T07:47:29.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 780199 11.54 MB/sec execute 221 sec latency 33.612 ms 2024-05-27T07:47:29.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:29 smithi186 ceph-mon[36647]: pgmap v250: 129 pgs: 129 active+clean; 69 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 321 op/s 2024-05-27T07:47:29.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:29 smithi114 ceph-mon[32038]: pgmap v250: 129 pgs: 129 active+clean; 69 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 321 op/s 2024-05-27T07:47:29.515 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:47:29.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:47:29.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:29 smithi163 ceph-mon[36451]: pgmap v250: 129 pgs: 129 active+clean; 69 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 321 op/s 2024-05-27T07:47:29.775 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:47:30.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 782280 11.55 MB/sec execute 222 sec latency 18.106 ms 2024-05-27T07:47:31.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.250+0000 7fae145e3640 1 -- 172.21.15.114:0/3605365288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0c109030 msgr2=0x7fae0c102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.250+0000 7fae145e3640 1 --2- 172.21.15.114:0/3605365288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0c109030 0x7fae0c102c70 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009900 tx=0x7fadfc02a410 comp rx=0 tx=0).stop 2024-05-27T07:47:31.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.250+0000 7fae145e3640 1 -- 172.21.15.114:0/3605365288 shutdown_connections 2024-05-27T07:47:31.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.250+0000 7fae145e3640 1 --2- 172.21.15.114:0/3605365288 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae0c1032e0 0x7fae0c10f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.250+0000 7fae145e3640 1 --2- 172.21.15.114:0/3605365288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0c109030 0x7fae0c102c70 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.250+0000 7fae145e3640 1 --2- 172.21.15.114:0/3605365288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae0c108b90 0x7fae0c102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.250+0000 7fae145e3640 1 -- 172.21.15.114:0/3605365288 >> 172.21.15.114:0/3605365288 conn(0x7fae0c0782d0 msgr2=0x7fae0c078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:31.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.250+0000 7fae145e3640 1 -- 172.21.15.114:0/3605365288 shutdown_connections 2024-05-27T07:47:31.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.250+0000 7fae145e3640 1 -- 172.21.15.114:0/3605365288 wait complete. 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.250+0000 7fae145e3640 1 Processor -- start 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae145e3640 1 -- start start 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae145e3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae0c1032e0 0x7fae0c0ff370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae145e3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0c108b90 0x7fae0c0ff8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae145e3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae0c109030 0x7fae0c0ffdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae145e3640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c112d80 con 0x7fae0c108b90 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae145e3640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c112c00 con 0x7fae0c109030 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae145e3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c112f00 con 0x7fae0c1032e0 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae11b7e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0c108b90 0x7fae0c0ff8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae11b7e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0c108b90 0x7fae0c0ff8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48402/0 (socket says 172.21.15.114:48402) 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae11b7e640 1 -- 172.21.15.114:0/469055327 learned_addr learned my addr 172.21.15.114:0/469055327 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae1237f640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae0c1032e0 0x7fae0c0ff370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae12b80640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae0c109030 0x7fae0c0ffdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae11b7e640 1 -- 172.21.15.114:0/469055327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae0c1032e0 msgr2=0x7fae0c0ff370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae11b7e640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae0c1032e0 0x7fae0c0ff370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae11b7e640 1 -- 172.21.15.114:0/469055327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae0c109030 msgr2=0x7fae0c0ffdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae11b7e640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae0c109030 0x7fae0c0ffdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae11b7e640 1 -- 172.21.15.114:0/469055327 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae0c1a8c70 con 0x7fae0c108b90 2024-05-27T07:47:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae1237f640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae0c1032e0 0x7fae0c0ff370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:47:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.251+0000 7fae11b7e640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0c108b90 0x7fae0c0ff8b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fadfc006010 tx=0x7fadfc004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.252+0000 7fae037fe640 1 -- 172.21.15.114:0/469055327 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc03f030 con 0x7fae0c108b90 2024-05-27T07:47:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.252+0000 7fae037fe640 1 -- 172.21.15.114:0/469055327 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadfc003aa0 con 0x7fae0c108b90 2024-05-27T07:47:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.252+0000 7fae145e3640 1 -- 172.21.15.114:0/469055327 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae0c1a8f00 con 0x7fae0c108b90 2024-05-27T07:47:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.252+0000 7fae037fe640 1 -- 172.21.15.114:0/469055327 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc035070 con 0x7fae0c108b90 2024-05-27T07:47:31.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.252+0000 7fae145e3640 1 -- 172.21.15.114:0/469055327 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae0c1a9300 con 0x7fae0c108b90 2024-05-27T07:47:31.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.254+0000 7fae037fe640 1 -- 172.21.15.114:0/469055327 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fadfc042030 con 0x7fae0c108b90 2024-05-27T07:47:31.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.255+0000 7fae1b56b740 1 Processor -- start 2024-05-27T07:47:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.255+0000 7fae1b56b740 1 -- start start 2024-05-27T07:47:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.255+0000 7fae037fe640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fadf80b2950 0x7fadf80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae12b80640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fadf80b2950 0x7fadf80b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae1b56b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca20431b0 0x555ca23e43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae1b56b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23e4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae037fe640 1 -- 172.21.15.114:0/469055327 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fadfc102ba0 con 0x7fae0c108b90 2024-05-27T07:47:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae1b56b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca23e4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae1b56b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555ca2356c30 con 0x555ca201a2f0 2024-05-27T07:47:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae1b56b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555ca23467d0 con 0x555ca20431b0 2024-05-27T07:47:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae1b56b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555ca234a810 con 0x555ca20224d0 2024-05-27T07:47:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae11b7e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23e4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae11b7e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23e4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48410/0 (socket says 172.21.15.114:48410) 2024-05-27T07:47:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae11b7e640 1 -- 172.21.15.114:0/296344701 learned_addr learned my addr 172.21.15.114:0/296344701 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae12b80640 1 --2- 172.21.15.114:0/296344701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca23e4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae11b7e640 1 -- 172.21.15.114:0/296344701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 msgr2=0x555ca23e4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae11b7e640 1 --2- 172.21.15.114:0/296344701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca23e4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae11b7e640 1 -- 172.21.15.114:0/296344701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca20431b0 msgr2=0x555ca23e43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae11b7e640 1 --2- 172.21.15.114:0/296344701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca20431b0 0x555ca23e43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae11b7e640 1 -- 172.21.15.114:0/296344701 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ca205b210 con 0x555ca201a2f0 2024-05-27T07:47:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae12b80640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fadf80b2950 0x7fadf80b4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fae04005fd0 tx=0x7fae040107a0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae11b7e640 1 --2- 172.21.15.114:0/296344701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23e4910 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fadfc0c9a60 tx=0x7fadfc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95994526acde2a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae00bfe640 1 -- 172.21.15.114:0/296344701 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc049040 con 0x555ca201a2f0 2024-05-27T07:47:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae00bfe640 1 -- 172.21.15.114:0/296344701 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadfc0ca1d0 con 0x555ca201a2f0 2024-05-27T07:47:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.256+0000 7fae00bfe640 1 -- 172.21.15.114:0/296344701 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc0c5d60 con 0x555ca201a2f0 2024-05-27T07:47:31.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.257+0000 7fae1b56b740 1 -- 172.21.15.114:0/296344701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 msgr2=0x555ca23e4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.257+0000 7fae1b56b740 1 --2- 172.21.15.114:0/296344701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23e4910 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fadfc0c9a60 tx=0x7fadfc009260 comp rx=0 tx=0).stop 2024-05-27T07:47:31.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.257+0000 7fae1b56b740 1 -- 172.21.15.114:0/296344701 shutdown_connections 2024-05-27T07:47:31.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.257+0000 7fae1b56b740 1 --2- 172.21.15.114:0/296344701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca23e4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.257+0000 7fae1b56b740 1 --2- 172.21.15.114:0/296344701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23e4910 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.257+0000 7fae1b56b740 1 --2- 172.21.15.114:0/296344701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca20431b0 0x555ca23e43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.257+0000 7fae1b56b740 1 -- 172.21.15.114:0/296344701 >> 172.21.15.114:0/296344701 conn(0x555ca2037300 msgr2=0x555ca23e0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:31.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 784827 11.57 MB/sec execute 223 sec latency 29.436 ms 2024-05-27T07:47:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1b56b740 1 -- 172.21.15.114:0/296344701 shutdown_connections 2024-05-27T07:47:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1b56b740 1 -- 172.21.15.114:0/296344701 wait complete. 2024-05-27T07:47:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1b56b740 1 -- start start 2024-05-27T07:47:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1b56b740 1 Processor -- start 2024-05-27T07:47:31.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1b56b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1b56b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca23f0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1b56b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca20431b0 0x555ca23eea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1b56b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555ca202c790 con 0x555ca201a2f0 2024-05-27T07:47:31.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1b56b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555ca2302890 con 0x555ca20431b0 2024-05-27T07:47:31.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1b56b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555ca2356c30 con 0x555ca20224d0 2024-05-27T07:47:31.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1237f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1b56b740 10 client.0 _ll_register_callbacks cb 0x555ca205aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:31.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1237f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23f04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48416/0 (socket says 172.21.15.114:48416) 2024-05-27T07:47:31.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.277+0000 7fae1237f640 1 -- 172.21.15.114:0/1506290720 learned_addr learned my addr 172.21.15.114:0/1506290720 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:31.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fae12b80640 1 --2- 172.21.15.114:0/1506290720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca20431b0 0x555ca23eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fae11b7e640 1 --2- 172.21.15.114:0/1506290720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca23f0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fae1237f640 1 -- 172.21.15.114:0/1506290720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 msgr2=0x555ca23f0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:31.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fae1237f640 1 --2- 172.21.15.114:0/1506290720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca23f0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fae1237f640 1 -- 172.21.15.114:0/1506290720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca20431b0 msgr2=0x555ca23eea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:31.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fae1237f640 1 --2- 172.21.15.114:0/1506290720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca20431b0 0x555ca23eea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fae1237f640 1 -- 172.21.15.114:0/1506290720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ca2127620 con 0x555ca201a2f0 2024-05-27T07:47:31.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fae1237f640 1 --2- 172.21.15.114:0/1506290720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23f04f0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fae0800bdc0 tx=0x7fae0800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c30e83d6cfc61e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:31.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fadf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:31.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fadf5ffb640 1 -- 172.21.15.114:0/1506290720 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae08015070 con 0x555ca201a2f0 2024-05-27T07:47:31.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fadf5ffb640 1 -- 172.21.15.114:0/1506290720 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fae0801a660 con 0x555ca201a2f0 2024-05-27T07:47:31.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fae1b56b740 1 -- 172.21.15.114:0/1506290720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadfc0ca1d0 con 0x555ca201a2f0 2024-05-27T07:47:31.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fadf5ffb640 1 -- 172.21.15.114:0/1506290720 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fae0801a880 con 0x555ca201a2f0 2024-05-27T07:47:31.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fae1b56b740 10 client.17247 fetch_fsmap learned FSMap version 45 2024-05-27T07:47:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.278+0000 7fae1b56b740 1 -- 172.21.15.114:0/1506290720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x555ca23fd5b0 con 0x555ca201a2f0 2024-05-27T07:47:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.279+0000 7fadf5ffb640 1 -- 172.21.15.114:0/1506290720 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadfc0ca1d0 con 0x555ca201a2f0 2024-05-27T07:47:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.279+0000 7fadf5ffb640 1 -- 172.21.15.114:0/1506290720 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 45) ==== 7725+0+0 (secure 0 0 0) 0x7fae08011c30 con 0x555ca201a2f0 2024-05-27T07:47:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.279+0000 7fae1b56b740 10 client.17247 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:47:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.279+0000 7fae1b56b740 10 client.17247 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.279+0000 7fae1b56b740 20 client.17247 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.279+0000 7fae1b56b740 1 --2- 172.21.15.114:0/1506290720 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555ca2001d80 0x555ca2400ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.279+0000 7fae1b56b740 4 client.17247 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.279+0000 7fae1b56b740 1 -- 172.21.15.114:0/1506290720 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ca2358b80 con 0x555ca2001d80 2024-05-27T07:47:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.279+0000 7fae11b7e640 1 --2- 172.21.15.114:0/1506290720 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555ca2001d80 0x555ca2400ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.279+0000 7fadf5ffb640 10 client.17247 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.279+0000 7fae11b7e640 1 --2- 172.21.15.114:0/1506290720 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555ca2001d80 0x555ca2400ad0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=127b82f08bb53fa1 server_cookie=2efd4a8b325d60ad in_seq=0 out_seq=0 2024-05-27T07:47:31.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fadf5ffb640 1 -- 172.21.15.114:0/1506290720 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555ca2358b80 con 0x555ca2001d80 2024-05-27T07:47:31.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fadf5ffb640 10 client.17247 handle_command_reply: tid=0 2024-05-27T07:47:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fae1b56b740 1 client.17247 shutdown 2024-05-27T07:47:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fae1b56b740 1 -- 172.21.15.114:0/1506290720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 msgr2=0x555ca23f04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fae1b56b740 1 --2- 172.21.15.114:0/1506290720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23f04f0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fae0800bdc0 tx=0x7fae0800f730 comp rx=0 tx=0).stop 2024-05-27T07:47:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fae1b56b740 1 -- 172.21.15.114:0/1506290720 shutdown_connections 2024-05-27T07:47:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fae1b56b740 1 --2- 172.21.15.114:0/1506290720 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555ca2001d80 0x555ca2400ad0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fae1b56b740 1 --2- 172.21.15.114:0/1506290720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca20431b0 0x555ca23eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fae1b56b740 1 --2- 172.21.15.114:0/1506290720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca23f0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fadf5ffb640 0 client.17247 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fae1b56b740 1 --2- 172.21.15.114:0/1506290720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23f04f0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fae1b56b740 1 -- 172.21.15.114:0/1506290720 >> 172.21.15.114:0/1506290720 conn(0x555ca2037300 msgr2=0x555ca23e60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fae1b56b740 1 -- 172.21.15.114:0/1506290720 shutdown_connections 2024-05-27T07:47:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fae1b56b740 1 -- 172.21.15.114:0/1506290720 wait complete. 2024-05-27T07:47:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.281+0000 7fae1b56b740 20 client.17247 trim_cache size 0 max 16384 2024-05-27T07:47:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.296+0000 7fae1b56b740 1 Processor -- start 2024-05-27T07:47:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.296+0000 7fae1b56b740 1 -- start start 2024-05-27T07:47:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.296+0000 7fae1b56b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca2001d80 0x555ca23f0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.296+0000 7fae1b56b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23e8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.296+0000 7fae1b56b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca23e85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.296+0000 7fae1b56b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c1b44c0 con 0x555ca201a2f0 2024-05-27T07:47:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.296+0000 7fae1b56b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c1b1570 con 0x555ca2001d80 2024-05-27T07:47:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.296+0000 7fae1b56b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c1b3720 con 0x555ca20224d0 2024-05-27T07:47:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.297+0000 7fae11b7e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23e8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.297+0000 7fae11b7e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23e8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48430/0 (socket says 172.21.15.114:48430) 2024-05-27T07:47:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.297+0000 7fae12b80640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca23e85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.297+0000 7fae11b7e640 1 -- 172.21.15.114:0/1155716889 learned_addr learned my addr 172.21.15.114:0/1155716889 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.297+0000 7fae1237f640 1 --2- 172.21.15.114:0/1155716889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca2001d80 0x555ca23f0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.297+0000 7fae11b7e640 1 -- 172.21.15.114:0/1155716889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 msgr2=0x555ca23e85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.297+0000 7fae11b7e640 1 --2- 172.21.15.114:0/1155716889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca23e85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.297+0000 7fae11b7e640 1 -- 172.21.15.114:0/1155716889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca2001d80 msgr2=0x555ca23f0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.297+0000 7fae11b7e640 1 --2- 172.21.15.114:0/1155716889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca2001d80 0x555ca23f0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.297+0000 7fae11b7e640 1 -- 172.21.15.114:0/1155716889 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ca23f0d70 con 0x555ca201a2f0 2024-05-27T07:47:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.297+0000 7fae11b7e640 1 --2- 172.21.15.114:0/1155716889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23e8080 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fadfc100540 tx=0x7fadfc02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1884d621160a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:31.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.303+0000 7fadf7fff640 1 -- 172.21.15.114:0/1155716889 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc049040 con 0x555ca201a2f0 2024-05-27T07:47:31.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.303+0000 7fadf7fff640 1 -- 172.21.15.114:0/1155716889 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadfc0ae070 con 0x555ca201a2f0 2024-05-27T07:47:31.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.303+0000 7fadf7fff640 1 -- 172.21.15.114:0/1155716889 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc0c5d60 con 0x555ca201a2f0 2024-05-27T07:47:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.304+0000 7fae1b56b740 1 -- 172.21.15.114:0/1155716889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 msgr2=0x555ca23e8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.304+0000 7fae1b56b740 1 --2- 172.21.15.114:0/1155716889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23e8080 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fadfc100540 tx=0x7fadfc02a410 comp rx=0 tx=0).stop 2024-05-27T07:47:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.304+0000 7fae1b56b740 1 -- 172.21.15.114:0/1155716889 shutdown_connections 2024-05-27T07:47:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.304+0000 7fae1b56b740 1 --2- 172.21.15.114:0/1155716889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca23e85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.304+0000 7fae1b56b740 1 --2- 172.21.15.114:0/1155716889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca201a2f0 0x555ca23e8080 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.304+0000 7fae1b56b740 1 --2- 172.21.15.114:0/1155716889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca2001d80 0x555ca23f0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.304+0000 7fae1b56b740 1 -- 172.21.15.114:0/1155716889 >> 172.21.15.114:0/1155716889 conn(0x555ca2037300 msgr2=0x555ca23f0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.304+0000 7fae1b56b740 1 -- 172.21.15.114:0/1155716889 shutdown_connections 2024-05-27T07:47:31.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.304+0000 7fae1b56b740 1 -- 172.21.15.114:0/1155716889 wait complete. 2024-05-27T07:47:31.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.304+0000 7fae1b56b740 1 -- start start 2024-05-27T07:47:31.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.304+0000 7fae1b56b740 1 Processor -- start 2024-05-27T07:47:31.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.304+0000 7fae1b56b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca2001d80 0x555ca23f38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1b56b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca201a2f0 0x555ca23f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1b56b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca2467160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1b56b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c1b3050 con 0x555ca2001d80 2024-05-27T07:47:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1b56b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c1b1be0 con 0x555ca201a2f0 2024-05-27T07:47:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1b56b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c1b22b0 con 0x555ca20224d0 2024-05-27T07:47:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1b56b740 10 client.0 _ll_register_callbacks cb 0x7fae0800b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1237f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca2001d80 0x555ca23f38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1237f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca2001d80 0x555ca23f38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48440/0 (socket says 172.21.15.114:48440) 2024-05-27T07:47:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1237f640 1 -- 172.21.15.114:0/252505740 learned_addr learned my addr 172.21.15.114:0/252505740 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae12b80640 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca2467160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae11b7e640 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca201a2f0 0x555ca23f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1237f640 1 -- 172.21.15.114:0/252505740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 msgr2=0x555ca2467160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1237f640 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca2467160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1237f640 1 -- 172.21.15.114:0/252505740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca201a2f0 msgr2=0x555ca23f3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1237f640 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca201a2f0 0x555ca23f3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1237f640 1 -- 172.21.15.114:0/252505740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ca24676a0 con 0x555ca2001d80 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae11b7e640 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca201a2f0 0x555ca23f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1237f640 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca2001d80 0x555ca23f38d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fae08005d80 tx=0x7fae0801ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e18650d66c349953 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fadf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fadf5ffb640 1 -- 172.21.15.114:0/252505740 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae08015040 con 0x555ca2001d80 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fadf5ffb640 1 -- 172.21.15.114:0/252505740 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fae08005450 con 0x555ca2001d80 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.305+0000 7fae1b56b740 1 -- 172.21.15.114:0/252505740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadfc0ae070 con 0x555ca2001d80 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.306+0000 7fadf5ffb640 1 -- 172.21.15.114:0/252505740 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fae08039050 con 0x555ca2001d80 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.306+0000 7fae1b56b740 10 client.17253 fetch_fsmap learned FSMap version 45 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.306+0000 7fae1b56b740 1 -- 172.21.15.114:0/252505740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x555ca24688f0 con 0x555ca2001d80 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.306+0000 7fadf5ffb640 1 -- 172.21.15.114:0/252505740 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadfc0ae070 con 0x555ca2001d80 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.306+0000 7fadf5ffb640 1 -- 172.21.15.114:0/252505740 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 45) ==== 7725+0+0 (secure 0 0 0) 0x7fae08010610 con 0x555ca2001d80 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.306+0000 7fae1b56b740 10 client.17253 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.306+0000 7fae1b56b740 10 client.17253 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.306+0000 7fae1b56b740 20 client.17253 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:31.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.306+0000 7fae1b56b740 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555ca20431b0 0x555ca246b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.306+0000 7fae1b56b740 4 client.17253 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:47:31.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.306+0000 7fae1b56b740 1 -- 172.21.15.114:0/252505740 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555ca2352360 con 0x555ca20431b0 2024-05-27T07:47:31.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.306+0000 7fae11b7e640 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555ca20431b0 0x555ca246b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.307+0000 7fae11b7e640 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555ca20431b0 0x555ca246b080 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50ae9d582cfedccb server_cookie=7abebf0316e3c7aa in_seq=0 out_seq=0 2024-05-27T07:47:31.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.307+0000 7fadf5ffb640 10 client.17253 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:31.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:31 smithi114 ceph-mon[32038]: pgmap v251: 129 pgs: 129 active+clean; 77 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 386 op/s 2024-05-27T07:47:31.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.558+0000 7f22d0a37700 1 -- 172.21.15.114:0/3164069594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc11d920 msgr2=0x7f22cc11fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.558+0000 7f22d0a37700 1 --2- 172.21.15.114:0/3164069594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc11d920 0x7f22cc11fd40 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f22c400b0a0 tx=0x7f22c400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.562+0000 7f22d0a37700 1 -- 172.21.15.114:0/3164069594 shutdown_connections 2024-05-27T07:47:31.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.562+0000 7f22d0a37700 1 --2- 172.21.15.114:0/3164069594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc11d920 0x7f22cc11fd40 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.562+0000 7f22d0a37700 1 --2- 172.21.15.114:0/3164069594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22cc10a920 0x7f22cc10ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.562+0000 7f22d0a37700 1 --2- 172.21.15.114:0/3164069594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc10b270 0x7f22cc11d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.562+0000 7f22d0a37700 1 -- 172.21.15.114:0/3164069594 >> 172.21.15.114:0/3164069594 conn(0x7f22cc06f4c0 msgr2=0x7f22cc06f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:31.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.564+0000 7f22d0a37700 1 -- 172.21.15.114:0/3164069594 shutdown_connections 2024-05-27T07:47:31.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.564+0000 7f22d0a37700 1 -- 172.21.15.114:0/3164069594 wait complete. 2024-05-27T07:47:31.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.565+0000 7f22d0a37700 1 Processor -- start 2024-05-27T07:47:31.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.567+0000 7f22d0a37700 1 -- start start 2024-05-27T07:47:31.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.567+0000 7f22d0a37700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc10a920 0x7f22cc1af880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.567+0000 7f22d0a37700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc10b270 0x7f22cc1afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.567+0000 7f22d0a37700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22cc11d920 0x7f22cc1a9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.567+0000 7f22d0a37700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc123340 con 0x7f22cc10a920 2024-05-27T07:47:31.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.567+0000 7f22d0a37700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc1231c0 con 0x7f22cc11d920 2024-05-27T07:47:31.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.567+0000 7f22d0a37700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc1234c0 con 0x7f22cc10b270 2024-05-27T07:47:31.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.569+0000 7f22cad9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22cc11d920 0x7f22cc1a9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.569+0000 7f22c9d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc10b270 0x7f22cc1afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.569+0000 7f22c9d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc10b270 0x7f22cc1afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54282/0 (socket says 172.21.15.114:54282) 2024-05-27T07:47:31.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.569+0000 7f22c9d9b700 1 -- 172.21.15.114:0/952201691 learned_addr learned my addr 172.21.15.114:0/952201691 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:31.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.571+0000 7f22c9d9b700 1 -- 172.21.15.114:0/952201691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22cc11d920 msgr2=0x7f22cc1a9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.571+0000 7f22c9d9b700 1 --2- 172.21.15.114:0/952201691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22cc11d920 0x7f22cc1a9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.571+0000 7f22c9d9b700 1 -- 172.21.15.114:0/952201691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc10a920 msgr2=0x7f22cc1af880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.571+0000 7f22c9d9b700 1 --2- 172.21.15.114:0/952201691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc10a920 0x7f22cc1af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.571+0000 7f22c9d9b700 1 -- 172.21.15.114:0/952201691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22cc1aa170 con 0x7f22cc10b270 2024-05-27T07:47:31.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.572+0000 7f22c9d9b700 1 --2- 172.21.15.114:0/952201691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc10b270 0x7f22cc1afdc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f22bc00e9a0 tx=0x7f22bc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:31.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.575+0000 7f22c37fe700 1 -- 172.21.15.114:0/952201691 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc00cbd0 con 0x7f22cc10b270 2024-05-27T07:47:31.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.575+0000 7f22c37fe700 1 -- 172.21.15.114:0/952201691 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22bc00cd70 con 0x7f22cc10b270 2024-05-27T07:47:31.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.575+0000 7f22c37fe700 1 -- 172.21.15.114:0/952201691 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc018680 con 0x7f22cc10b270 2024-05-27T07:47:31.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.575+0000 7f22d0a37700 1 -- 172.21.15.114:0/952201691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22cc1aa460 con 0x7f22cc10b270 2024-05-27T07:47:31.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.576+0000 7f22d0a37700 1 -- 172.21.15.114:0/952201691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22cc112bc0 con 0x7f22cc10b270 2024-05-27T07:47:31.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.584+0000 7f22d0a37700 1 -- 172.21.15.114:0/952201691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22ac005330 con 0x7f22cc10b270 2024-05-27T07:47:31.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.592+0000 7f22c37fe700 1 -- 172.21.15.114:0/952201691 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f22bc018820 con 0x7f22cc10b270 2024-05-27T07:47:31.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.593+0000 7f22c37fe700 1 --2- 172.21.15.114:0/952201691 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f22b80b2c00 0x7f22b80b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:31.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.593+0000 7f22c37fe700 1 -- 172.21.15.114:0/952201691 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f22bc0e2ca0 con 0x7f22cc10b270 2024-05-27T07:47:31.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.595+0000 7f22c37fe700 1 -- 172.21.15.114:0/952201691 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f22bc0aa370 con 0x7f22cc10b270 2024-05-27T07:47:31.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.597+0000 7f22ca59c700 1 --2- 172.21.15.114:0/952201691 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f22b80b2c00 0x7f22b80b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:31.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.598+0000 7f22ca59c700 1 --2- 172.21.15.114:0/952201691 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f22b80b2c00 0x7f22b80b5110 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f22cc1aabe0 tx=0x7f22b4006cd0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:31.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:31 smithi163 ceph-mon[36451]: pgmap v251: 129 pgs: 129 active+clean; 77 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 386 op/s 2024-05-27T07:47:31.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:31 smithi186 ceph-mon[36647]: pgmap v251: 129 pgs: 129 active+clean; 77 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 386 op/s 2024-05-27T07:47:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.819+0000 7fadf5ffb640 1 -- 172.21.15.114:0/252505740 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x555ca2352360 con 0x555ca20431b0 2024-05-27T07:47:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.819+0000 7fadf5ffb640 10 client.17253 handle_command_reply: tid=0 2024-05-27T07:47:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.819+0000 7fae1b56b740 1 client.17253 shutdown 2024-05-27T07:47:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.819+0000 7fae1b56b740 1 -- 172.21.15.114:0/252505740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca2001d80 msgr2=0x555ca23f38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.819+0000 7fae1b56b740 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca2001d80 0x555ca23f38d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fae08005d80 tx=0x7fae0801ac50 comp rx=0 tx=0).stop 2024-05-27T07:47:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.819+0000 7fae1b56b740 1 -- 172.21.15.114:0/252505740 shutdown_connections 2024-05-27T07:47:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.819+0000 7fae1b56b740 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555ca20431b0 0x555ca246b080 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.819+0000 7fadf5ffb640 0 client.17253 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.819+0000 7fae1b56b740 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ca20224d0 0x555ca2467160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.819+0000 7fae1b56b740 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555ca201a2f0 0x555ca23f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.819+0000 7fae1b56b740 1 --2- 172.21.15.114:0/252505740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ca2001d80 0x555ca23f38d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.819+0000 7fae1b56b740 1 -- 172.21.15.114:0/252505740 >> 172.21.15.114:0/252505740 conn(0x555ca2037300 msgr2=0x555ca23ee290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.820+0000 7fae1b56b740 1 -- 172.21.15.114:0/252505740 shutdown_connections 2024-05-27T07:47:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.820+0000 7fae1b56b740 1 -- 172.21.15.114:0/252505740 wait complete. 2024-05-27T07:47:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.820+0000 7fae1b56b740 20 client.17253 trim_cache size 0 max 16384 2024-05-27T07:47:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:47:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:47:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:47:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.821+0000 7fae145e3640 1 -- 172.21.15.114:0/469055327 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fadf80b2950 msgr2=0x7fadf80b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.821+0000 7fae145e3640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fadf80b2950 0x7fadf80b4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fae04005fd0 tx=0x7fae040107a0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.821+0000 7fae145e3640 1 -- 172.21.15.114:0/469055327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0c108b90 msgr2=0x7fae0c0ff8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.821+0000 7fae145e3640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0c108b90 0x7fae0c0ff8b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fadfc006010 tx=0x7fadfc004060 comp rx=0 tx=0).stop 2024-05-27T07:47:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.821+0000 7fae145e3640 1 -- 172.21.15.114:0/469055327 shutdown_connections 2024-05-27T07:47:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.821+0000 7fae145e3640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fadf80b2950 0x7fadf80b4e60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.821+0000 7fae145e3640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae0c109030 0x7fae0c0ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.821+0000 7fae145e3640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0c108b90 0x7fae0c0ff8b0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.821+0000 7fae145e3640 1 --2- 172.21.15.114:0/469055327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae0c1032e0 0x7fae0c0ff370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.821+0000 7fae145e3640 1 -- 172.21.15.114:0/469055327 >> 172.21.15.114:0/469055327 conn(0x7fae0c0782d0 msgr2=0x7fae0c106240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.821+0000 7fae145e3640 1 -- 172.21.15.114:0/469055327 shutdown_connections 2024-05-27T07:47:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:31.821+0000 7fae145e3640 1 -- 172.21.15.114:0/469055327 wait complete. 2024-05-27T07:47:31.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.955+0000 7f22d0a37700 1 -- 172.21.15.114:0/952201691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22ac005620 con 0x7f22cc10b270 2024-05-27T07:47:31.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.956+0000 7f22c37fe700 1 -- 172.21.15.114:0/952201691 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11000 (secure 0 0 0) 0x7f22bc0a9c70 con 0x7f22cc10b270 2024-05-27T07:47:31.955 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:47:31.955 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":26227,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2936519309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2936519309},{"type":"v1","addr":"172.21.15.186:6835","nonce":2936519309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:47:15.274340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26707":{"gid":26707,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/298307750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":298307750},{"type":"v1","addr":"172.21.15.186:6837","nonce":298307750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:47:31.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.959+0000 7f22d0a37700 1 -- 172.21.15.114:0/952201691 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f22b80b2c00 msgr2=0x7f22b80b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.959+0000 7f22d0a37700 1 --2- 172.21.15.114:0/952201691 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f22b80b2c00 0x7f22b80b5110 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f22cc1aabe0 tx=0x7f22b4006cd0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.959+0000 7f22d0a37700 1 -- 172.21.15.114:0/952201691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc10b270 msgr2=0x7f22cc1afdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:31.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.959+0000 7f22d0a37700 1 --2- 172.21.15.114:0/952201691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc10b270 0x7f22cc1afdc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f22bc00e9a0 tx=0x7f22bc00ecb0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.959+0000 7f22d0a37700 1 -- 172.21.15.114:0/952201691 shutdown_connections 2024-05-27T07:47:31.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.959+0000 7f22d0a37700 1 --2- 172.21.15.114:0/952201691 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f22b80b2c00 0x7f22b80b5110 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.959+0000 7f22d0a37700 1 --2- 172.21.15.114:0/952201691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22cc11d920 0x7f22cc1a9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.959+0000 7f22d0a37700 1 --2- 172.21.15.114:0/952201691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22cc10a920 0x7f22cc1af880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.959+0000 7f22d0a37700 1 --2- 172.21.15.114:0/952201691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22cc10b270 0x7f22cc1afdc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:31.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.959+0000 7f22d0a37700 1 -- 172.21.15.114:0/952201691 >> 172.21.15.114:0/952201691 conn(0x7f22cc06f4c0 msgr2=0x7f22cc11b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:31.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.959+0000 7f22d0a37700 1 -- 172.21.15.114:0/952201691 shutdown_connections 2024-05-27T07:47:31.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:31.959+0000 7f22d0a37700 1 -- 172.21.15.114:0/952201691 wait complete. 2024-05-27T07:47:31.959 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 45 2024-05-27T07:47:32.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 785482 11.53 MB/sec execute 224 sec latency 245.825 ms 2024-05-27T07:47:32.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:32 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/952201691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:47:32.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:32 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/952201691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:47:32.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:32 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/952201691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:47:33.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:47:33.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 787691 11.53 MB/sec execute 225 sec latency 70.454 ms 2024-05-27T07:47:33.387 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:47:33.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:33 smithi114 ceph-mon[32038]: pgmap v252: 129 pgs: 129 active+clean; 77 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 327 op/s 2024-05-27T07:47:33.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:33 smithi163 ceph-mon[36451]: pgmap v252: 129 pgs: 129 active+clean; 77 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 327 op/s 2024-05-27T07:47:33.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:33 smithi186 ceph-mon[36647]: pgmap v252: 129 pgs: 129 active+clean; 77 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 327 op/s 2024-05-27T07:47:34.267 INFO:tasks.workunit.client.0.smithi114.stdout: 1 789614 11.53 MB/sec execute 226 sec latency 19.665 ms 2024-05-27T07:47:34.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:34 smithi114 ceph-mon[32038]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-27T07:47:34.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:34 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:47:34.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:34 smithi163 ceph-mon[36451]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-27T07:47:34.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:34 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:47:34.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:34 smithi186 ceph-mon[36647]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-27T07:47:34.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:34 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:47:35.267 INFO:tasks.workunit.client.0.smithi114.stdout: 1 791919 11.53 MB/sec execute 227 sec latency 37.485 ms 2024-05-27T07:47:35.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.368+0000 7f0df31fc700 1 -- 172.21.15.114:0/2543611139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dec108f10 msgr2=0x7f0dec109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:35.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.368+0000 7f0df31fc700 1 --2- 172.21.15.114:0/2543611139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dec108f10 0x7f0dec109320 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0dd80099c0 tx=0x7f0dd8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:35 smithi114 ceph-mon[32038]: pgmap v253: 129 pgs: 129 active+clean; 77 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 206 op/s 2024-05-27T07:47:35.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.368+0000 7f0df31fc700 1 -- 172.21.15.114:0/2543611139 shutdown_connections 2024-05-27T07:47:35.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.368+0000 7f0df31fc700 1 --2- 172.21.15.114:0/2543611139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec109860 0x7f0dec1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.368+0000 7f0df31fc700 1 --2- 172.21.15.114:0/2543611139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dec1045e0 0x7f0dec1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.368+0000 7f0df31fc700 1 --2- 172.21.15.114:0/2543611139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dec108f10 0x7f0dec109320 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.368+0000 7f0df31fc700 1 -- 172.21.15.114:0/2543611139 >> 172.21.15.114:0/2543611139 conn(0x7f0dec100030 msgr2=0x7f0dec102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:35.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.368+0000 7f0df31fc700 1 -- 172.21.15.114:0/2543611139 shutdown_connections 2024-05-27T07:47:35.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.368+0000 7f0df31fc700 1 -- 172.21.15.114:0/2543611139 wait complete. 2024-05-27T07:47:35.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.369+0000 7f0df31fc700 1 Processor -- start 2024-05-27T07:47:35.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.369+0000 7f0df31fc700 1 -- start start 2024-05-27T07:47:35.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.369+0000 7f0df31fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec1045e0 0x7f0dec19ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.369+0000 7f0df31fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dec108f10 0x7f0dec19b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.369+0000 7f0df31fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dec109860 0x7f0dec19f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.369+0000 7f0df31fc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec114470 con 0x7f0dec108f10 2024-05-27T07:47:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.369+0000 7f0df31fc700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1142f0 con 0x7f0dec109860 2024-05-27T07:47:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.369+0000 7f0df31fc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1145f0 con 0x7f0dec1045e0 2024-05-27T07:47:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.369+0000 7f0df0f98700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec1045e0 0x7f0dec19ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.369+0000 7f0df0f98700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec1045e0 0x7f0dec19ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47108/0 (socket says 172.21.15.114:47108) 2024-05-27T07:47:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.369+0000 7f0df0f98700 1 -- 172.21.15.114:0/2680443541 learned_addr learned my addr 172.21.15.114:0/2680443541 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.369+0000 7f0debfff700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dec108f10 0x7f0dec19b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.369+0000 7f0df1799700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dec109860 0x7f0dec19f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.370+0000 7f0df0f98700 1 -- 172.21.15.114:0/2680443541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dec109860 msgr2=0x7f0dec19f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.370+0000 7f0df0f98700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dec109860 0x7f0dec19f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.370+0000 7f0df0f98700 1 -- 172.21.15.114:0/2680443541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dec108f10 msgr2=0x7f0dec19b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.370+0000 7f0df0f98700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dec108f10 0x7f0dec19b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.370+0000 7f0df0f98700 1 -- 172.21.15.114:0/2680443541 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dec19fd50 con 0x7f0dec1045e0 2024-05-27T07:47:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.370+0000 7f0debfff700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dec108f10 0x7f0dec19b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.370+0000 7f0df0f98700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec1045e0 0x7f0dec19ae80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0de000e9a0 tx=0x7f0de000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.370+0000 7f0de9ffb700 1 -- 172.21.15.114:0/2680443541 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de000ca10 con 0x7f0dec1045e0 2024-05-27T07:47:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.370+0000 7f0df31fc700 1 -- 172.21.15.114:0/2680443541 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dec1a0040 con 0x7f0dec1045e0 2024-05-27T07:47:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.370+0000 7f0de9ffb700 1 -- 172.21.15.114:0/2680443541 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0de000cbb0 con 0x7f0dec1045e0 2024-05-27T07:47:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.370+0000 7f0de9ffb700 1 -- 172.21.15.114:0/2680443541 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de0018680 con 0x7f0dec1045e0 2024-05-27T07:47:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.370+0000 7f0df31fc700 1 -- 172.21.15.114:0/2680443541 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dec1a7960 con 0x7f0dec1045e0 2024-05-27T07:47:35.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.373+0000 7f0de9ffb700 1 -- 172.21.15.114:0/2680443541 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f0de0018820 con 0x7f0dec1045e0 2024-05-27T07:47:35.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.374+0000 7f0de9ffb700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0ddc0b29f0 0x7f0ddc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.374+0000 7f0de9ffb700 1 -- 172.21.15.114:0/2680443541 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f0de00e2c30 con 0x7f0dec1045e0 2024-05-27T07:47:35.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.374+0000 7f0e01c11b80 1 Processor -- start 2024-05-27T07:47:35.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.374+0000 7f0debfff700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0ddc0b29f0 0x7f0ddc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.374+0000 7f0e01c11b80 1 -- start start 2024-05-27T07:47:35.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0e01c11b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf93710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0e01c11b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 0x55bf3cf8da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0e01c11b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 0x55bf3cf8e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0e01c11b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bf3cebcc90 con 0x55bf3cf8df80 2024-05-27T07:47:35.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0e01c11b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bf3ceac830 con 0x55bf3cf93c50 2024-05-27T07:47:35.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0e01c11b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bf3ceb0870 con 0x55bf3ce0fe40 2024-05-27T07:47:35.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0debfff700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0ddc0b29f0 0x7f0ddc0b4f00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8000c00 tx=0x7f0dd801a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:35.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0df1799700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 0x55bf3cf8e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0df0f98700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf93710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0df1799700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 0x55bf3cf8e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58904/0 (socket says 172.21.15.114:58904) 2024-05-27T07:47:35.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0df1799700 1 -- 172.21.15.114:0/1919208834 learned_addr learned my addr 172.21.15.114:0/1919208834 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:35.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0debfff700 1 --2- 172.21.15.114:0/1919208834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 0x55bf3cf8da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0df1799700 1 -- 172.21.15.114:0/1919208834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3ce0fe40 msgr2=0x55bf3cf93710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:35.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0df1799700 1 --2- 172.21.15.114:0/1919208834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf93710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0df1799700 1 -- 172.21.15.114:0/1919208834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 msgr2=0x55bf3cf8da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:35.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0df1799700 1 --2- 172.21.15.114:0/1919208834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 0x55bf3cf8da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0df1799700 1 -- 172.21.15.114:0/1919208834 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf3cdfc8d0 con 0x55bf3cf8df80 2024-05-27T07:47:35.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.375+0000 7f0df1799700 1 --2- 172.21.15.114:0/1919208834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 0x55bf3cf8e390 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0de4003f30 tx=0x7f0de4009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2103a3bde7d7f270 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:35.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.376+0000 7f0dd6ffd700 1 -- 172.21.15.114:0/1919208834 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de4004670 con 0x55bf3cf8df80 2024-05-27T07:47:35.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.376+0000 7f0dd6ffd700 1 -- 172.21.15.114:0/1919208834 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0de401a070 con 0x55bf3cf8df80 2024-05-27T07:47:35.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.376+0000 7f0dd6ffd700 1 -- 172.21.15.114:0/1919208834 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de40156e0 con 0x55bf3cf8df80 2024-05-27T07:47:35.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.377+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1919208834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 msgr2=0x55bf3cf8e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:35.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.377+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1919208834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 0x55bf3cf8e390 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0de4003f30 tx=0x7f0de4009c20 comp rx=0 tx=0).stop 2024-05-27T07:47:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.377+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1919208834 shutdown_connections 2024-05-27T07:47:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.377+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1919208834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 0x55bf3cf8e390 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.377+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1919208834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf93710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.377+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1919208834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 0x55bf3cf8da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.377+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1919208834 >> 172.21.15.114:0/1919208834 conn(0x55bf3ce6aaf0 msgr2=0x55bf3cf88640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.377+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1919208834 shutdown_connections 2024-05-27T07:47:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.377+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1919208834 wait complete. 2024-05-27T07:47:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0e01c11b80 1 -- start start 2024-05-27T07:47:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0e01c11b80 1 Processor -- start 2024-05-27T07:47:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0e01c11b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf97360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0e01c11b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf8df80 0x55bf3cf978a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0e01c11b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 0x55bf3cf97de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0e01c11b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bf3cdb2fa0 con 0x55bf3ce0fe40 2024-05-27T07:47:35.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0e01c11b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bf3ce4e7c0 con 0x55bf3cf93c50 2024-05-27T07:47:35.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0e01c11b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bf3cf9a740 con 0x55bf3cf8df80 2024-05-27T07:47:35.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0e01c11b80 10 client.0 _ll_register_callbacks cb 0x55bf3ce47920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:35.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0df0f98700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf97360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0df0f98700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf97360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58910/0 (socket says 172.21.15.114:58910) 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0df0f98700 1 -- 172.21.15.114:0/1756793708 learned_addr learned my addr 172.21.15.114:0/1756793708 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0df1799700 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 0x55bf3cf97de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.378+0000 7f0debfff700 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf8df80 0x55bf3cf978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0df0f98700 1 -- 172.21.15.114:0/1756793708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf8df80 msgr2=0x55bf3cf978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0df0f98700 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf8df80 0x55bf3cf978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0df0f98700 1 -- 172.21.15.114:0/1756793708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 msgr2=0x55bf3cf97de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0df0f98700 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 0x55bf3cf97de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0df0f98700 1 -- 172.21.15.114:0/1756793708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf3ce4b690 con 0x55bf3ce0fe40 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0df1799700 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 0x55bf3cf97de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0df0f98700 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf97360 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f0de00a6d70 tx=0x7f0de00a6040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524241592e98c9b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0dcbfff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0dcbfff700 1 -- 172.21.15.114:0/1756793708 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de00a9610 con 0x55bf3ce0fe40 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0dcbfff700 1 -- 172.21.15.114:0/1756793708 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f0de00b26e0 con 0x55bf3ce0fe40 2024-05-27T07:47:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1756793708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0de401a070 con 0x55bf3ce0fe40 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0dcbfff700 1 -- 172.21.15.114:0/1756793708 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f0de00aa200 con 0x55bf3ce0fe40 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0e01c11b80 10 client.17268 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1756793708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55bf3cfa5210 con 0x55bf3ce0fe40 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.379+0000 7f0dcbfff700 1 -- 172.21.15.114:0/1756793708 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0de401a070 con 0x55bf3ce0fe40 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.380+0000 7f0dcbfff700 1 -- 172.21.15.114:0/1756793708 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7f0de006f380 con 0x55bf3ce0fe40 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.380+0000 7f0e01c11b80 10 client.17268 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.380+0000 7f0e01c11b80 10 client.17268 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.380+0000 7f0e01c11b80 20 client.17268 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.380+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bf3cfa77a0 0x55bf3cfa9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.380+0000 7f0e01c11b80 4 client.17268 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.380+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1756793708 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf3cf599f0 con 0x55bf3cfa77a0 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.380+0000 7f0debfff700 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bf3cfa77a0 0x55bf3cfa9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.380+0000 7f0dcbfff700 10 client.17268 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:35.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.380+0000 7f0debfff700 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bf3cfa77a0 0x55bf3cfa9c00 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a491f6a8438fe16 server_cookie=9e1125cbf74ebea6 in_seq=0 out_seq=0 2024-05-27T07:47:35.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0dcbfff700 1 -- 172.21.15.114:0/1756793708 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bf3cf599f0 con 0x55bf3cfa77a0 2024-05-27T07:47:35.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0dcbfff700 10 client.17268 handle_command_reply: tid=0 2024-05-27T07:47:35.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0e01c11b80 1 client.17268 shutdown 2024-05-27T07:47:35.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1756793708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 msgr2=0x55bf3cf97360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:35.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf97360 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f0de00a6d70 tx=0x7f0de00a6040 comp rx=0 tx=0).stop 2024-05-27T07:47:35.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1756793708 shutdown_connections 2024-05-27T07:47:35.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bf3cfa77a0 0x55bf3cfa9c00 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 0x55bf3cf97de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf97360 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0dcbfff700 0 client.17268 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:35.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1756793708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf8df80 0x55bf3cf978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1756793708 >> 172.21.15.114:0/1756793708 conn(0x55bf3ce6aaf0 msgr2=0x55bf3cf8ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:35.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1756793708 shutdown_connections 2024-05-27T07:47:35.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1756793708 wait complete. 2024-05-27T07:47:35.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.383+0000 7f0e01c11b80 20 client.17268 trim_cache size 0 max 16384 2024-05-27T07:47:35.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.399+0000 7f0e01c11b80 1 Processor -- start 2024-05-27T07:47:35.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.399+0000 7f0e01c11b80 1 -- start start 2024-05-27T07:47:35.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0e01c11b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf8da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0e01c11b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 0x55bf3cf96c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0e01c11b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf93c50 0x55bf3cfa37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0e01c11b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1b0780 con 0x55bf3cf8df80 2024-05-27T07:47:35.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0e01c11b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1b0e50 con 0x55bf3ce0fe40 2024-05-27T07:47:35.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0e01c11b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1ae5d0 con 0x55bf3cf93c50 2024-05-27T07:47:35.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0debfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 0x55bf3cf96c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0debfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 0x55bf3cf96c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58920/0 (socket says 172.21.15.114:58920) 2024-05-27T07:47:35.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0debfff700 1 -- 172.21.15.114:0/1066722041 learned_addr learned my addr 172.21.15.114:0/1066722041 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:35.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0df1799700 1 --2- 172.21.15.114:0/1066722041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf93c50 0x55bf3cfa37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0df0f98700 1 --2- 172.21.15.114:0/1066722041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf8da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0debfff700 1 -- 172.21.15.114:0/1066722041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf93c50 msgr2=0x55bf3cfa37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:35.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0debfff700 1 --2- 172.21.15.114:0/1066722041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf93c50 0x55bf3cfa37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0debfff700 1 -- 172.21.15.114:0/1066722041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3ce0fe40 msgr2=0x55bf3cf8da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:35.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0debfff700 1 --2- 172.21.15.114:0/1066722041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf8da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0debfff700 1 -- 172.21.15.114:0/1066722041 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf3cf88890 con 0x55bf3cf8df80 2024-05-27T07:47:35.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.400+0000 7f0debfff700 1 --2- 172.21.15.114:0/1066722041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 0x55bf3cf96c10 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0dd80099c0 tx=0x7f0dd8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4407bec1a1162e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:35.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.401+0000 7f0dd67fc700 1 -- 172.21.15.114:0/1066722041 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd80039c0 con 0x55bf3cf8df80 2024-05-27T07:47:35.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.401+0000 7f0dd67fc700 1 -- 172.21.15.114:0/1066722041 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dd8003b60 con 0x55bf3cf8df80 2024-05-27T07:47:35.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.401+0000 7f0dd67fc700 1 -- 172.21.15.114:0/1066722041 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8031450 con 0x55bf3cf8df80 2024-05-27T07:47:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.401+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1066722041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 msgr2=0x55bf3cf96c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.401+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1066722041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 0x55bf3cf96c10 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0dd80099c0 tx=0x7f0dd8009310 comp rx=0 tx=0).stop 2024-05-27T07:47:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.401+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1066722041 shutdown_connections 2024-05-27T07:47:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.401+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1066722041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf93c50 0x55bf3cfa37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.401+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1066722041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf8da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.401+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/1066722041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3cf8df80 0x55bf3cf96c10 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.401+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1066722041 >> 172.21.15.114:0/1066722041 conn(0x55bf3ce6aaf0 msgr2=0x55bf3cf91010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.402+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1066722041 shutdown_connections 2024-05-27T07:47:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.402+0000 7f0e01c11b80 1 -- 172.21.15.114:0/1066722041 wait complete. 2024-05-27T07:47:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.402+0000 7f0e01c11b80 1 -- start start 2024-05-27T07:47:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.402+0000 7f0e01c11b80 1 Processor -- start 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.402+0000 7f0e01c11b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf9f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.402+0000 7f0e01c11b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf8df80 0x55bf3cf9ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.402+0000 7f0e01c11b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 0x55bf3cfae520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.402+0000 7f0e01c11b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1aec40 con 0x55bf3ce0fe40 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.402+0000 7f0e01c11b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1b00b0 con 0x55bf3cf93c50 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0e01c11b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1af310 con 0x55bf3cf8df80 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0e01c11b80 10 client.0 _ll_register_callbacks cb 0x55bf3ce47920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0df0f98700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf9f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0debfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf8df80 0x55bf3cf9ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0df0f98700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf9f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58932/0 (socket says 172.21.15.114:58932) 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0df0f98700 1 -- 172.21.15.114:0/414703134 learned_addr learned my addr 172.21.15.114:0/414703134 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0df1799700 1 --2- 172.21.15.114:0/414703134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 0x55bf3cfae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0debfff700 1 -- 172.21.15.114:0/414703134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 msgr2=0x55bf3cfae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0debfff700 1 --2- 172.21.15.114:0/414703134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 0x55bf3cfae520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0debfff700 1 -- 172.21.15.114:0/414703134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 msgr2=0x55bf3cf9f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0debfff700 1 --2- 172.21.15.114:0/414703134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf9f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0debfff700 1 -- 172.21.15.114:0/414703134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf3cfaeb10 con 0x55bf3cf8df80 2024-05-27T07:47:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0df0f98700 1 --2- 172.21.15.114:0/414703134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf9f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0debfff700 1 --2- 172.21.15.114:0/414703134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf8df80 0x55bf3cf9ff30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55bf3cfaa200 tx=0x7f0dd8009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=310ab4a6a78a62f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0dcbfff700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:47:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0dcbfff700 1 -- 172.21.15.114:0/414703134 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd803f070 con 0x55bf3cf8df80 2024-05-27T07:47:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0dcbfff700 1 -- 172.21.15.114:0/414703134 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f0dd800fbf0 con 0x55bf3cf8df80 2024-05-27T07:47:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.403+0000 7f0e01c11b80 1 -- 172.21.15.114:0/414703134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dd8003b60 con 0x55bf3cf8df80 2024-05-27T07:47:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.404+0000 7f0dcbfff700 1 -- 172.21.15.114:0/414703134 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f0dd80039c0 con 0x55bf3cf8df80 2024-05-27T07:47:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.404+0000 7f0e01c11b80 10 client.26845 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.404+0000 7f0e01c11b80 1 -- 172.21.15.114:0/414703134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55bf3d015f10 con 0x55bf3cf8df80 2024-05-27T07:47:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.404+0000 7f0dcbfff700 1 -- 172.21.15.114:0/414703134 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dd8003b60 con 0x55bf3cf8df80 2024-05-27T07:47:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.404+0000 7f0dcbfff700 1 -- 172.21.15.114:0/414703134 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7f0dd8031a70 con 0x55bf3cf8df80 2024-05-27T07:47:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.404+0000 7f0e01c11b80 10 client.26845 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.404+0000 7f0e01c11b80 10 client.26845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.404+0000 7f0e01c11b80 20 client.26845 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.404+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/414703134 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bf3d016530 0x55bf3d018950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.404+0000 7f0e01c11b80 4 client.26845 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:47:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.404+0000 7f0e01c11b80 1 -- 172.21.15.114:0/414703134 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bf3cebebe0 con 0x55bf3d016530 2024-05-27T07:47:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.404+0000 7f0df0f98700 1 --2- 172.21.15.114:0/414703134 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bf3d016530 0x55bf3d018950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0dcbfff700 10 client.26845 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0df0f98700 1 --2- 172.21.15.114:0/414703134 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bf3d016530 0x55bf3d018950 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=953dd1edc64ddd6d server_cookie=50b4a53444c2f80f in_seq=0 out_seq=0 2024-05-27T07:47:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0dcbfff700 1 -- 172.21.15.114:0/414703134 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bf3cebebe0 con 0x55bf3d016530 2024-05-27T07:47:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0dcbfff700 10 client.26845 handle_command_reply: tid=0 2024-05-27T07:47:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0e01c11b80 1 client.26845 shutdown 2024-05-27T07:47:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0e01c11b80 1 -- 172.21.15.114:0/414703134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf8df80 msgr2=0x55bf3cf9ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/414703134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf8df80 0x55bf3cf9ff30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55bf3cfaa200 tx=0x7f0dd8009f80 comp rx=0 tx=0).stop 2024-05-27T07:47:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0e01c11b80 1 -- 172.21.15.114:0/414703134 shutdown_connections 2024-05-27T07:47:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/414703134 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bf3d016530 0x55bf3d018950 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/414703134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bf3cf93c50 0x55bf3cfae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0dcbfff700 0 client.26845 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/414703134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bf3ce0fe40 0x55bf3cf9f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0e01c11b80 1 --2- 172.21.15.114:0/414703134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bf3cf8df80 0x55bf3cf9ff30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.405+0000 7f0e01c11b80 1 -- 172.21.15.114:0/414703134 >> 172.21.15.114:0/414703134 conn(0x55bf3ce6aaf0 msgr2=0x55bf3cf95c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.406+0000 7f0e01c11b80 1 -- 172.21.15.114:0/414703134 shutdown_connections 2024-05-27T07:47:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.406+0000 7f0e01c11b80 1 -- 172.21.15.114:0/414703134 wait complete. 2024-05-27T07:47:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.406+0000 7f0e01c11b80 20 client.26845 trim_cache size 0 max 16384 2024-05-27T07:47:35.411 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:47:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.406+0000 7f0df31fc700 1 -- 172.21.15.114:0/2680443541 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0ddc0b29f0 msgr2=0x7f0ddc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.406+0000 7f0df31fc700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0ddc0b29f0 0x7f0ddc0b4f00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8000c00 tx=0x7f0dd801a040 comp rx=0 tx=0).stop 2024-05-27T07:47:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.406+0000 7f0df31fc700 1 -- 172.21.15.114:0/2680443541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec1045e0 msgr2=0x7f0dec19ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.406+0000 7f0df31fc700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec1045e0 0x7f0dec19ae80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0de000e9a0 tx=0x7f0de000ecb0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.407+0000 7f0df31fc700 1 -- 172.21.15.114:0/2680443541 shutdown_connections 2024-05-27T07:47:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.407+0000 7f0df31fc700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0ddc0b29f0 0x7f0ddc0b4f00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.407+0000 7f0df31fc700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dec109860 0x7f0dec19f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.407+0000 7f0df31fc700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec1045e0 0x7f0dec19ae80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.407+0000 7f0df31fc700 1 --2- 172.21.15.114:0/2680443541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dec108f10 0x7f0dec19b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.407+0000 7f0df31fc700 1 -- 172.21.15.114:0/2680443541 >> 172.21.15.114:0/2680443541 conn(0x7f0dec100030 msgr2=0x7f0dec10a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.407+0000 7f0df31fc700 1 -- 172.21.15.114:0/2680443541 shutdown_connections 2024-05-27T07:47:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:35.407+0000 7f0df31fc700 1 -- 172.21.15.114:0/2680443541 wait complete. 2024-05-27T07:47:35.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:35 smithi163 ceph-mon[36451]: pgmap v253: 129 pgs: 129 active+clean; 77 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 206 op/s 2024-05-27T07:47:35.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:35 smithi186 ceph-mon[36647]: pgmap v253: 129 pgs: 129 active+clean; 77 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 206 op/s 2024-05-27T07:47:36.266 INFO:tasks.workunit.client.0.smithi114.stdout: 1 793548 11.51 MB/sec execute 228 sec latency 18.031 ms 2024-05-27T07:47:37.131 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:47:37.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:47:37.267 INFO:tasks.workunit.client.0.smithi114.stdout: 1 795877 11.52 MB/sec execute 229 sec latency 14.344 ms 2024-05-27T07:47:37.394 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:47:37.517 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:37 smithi114 ceph-mon[32038]: pgmap v254: 129 pgs: 129 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 432 op/s 2024-05-27T07:47:37.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:37 smithi163 ceph-mon[36451]: pgmap v254: 129 pgs: 129 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 432 op/s 2024-05-27T07:47:37.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:37 smithi186 ceph-mon[36647]: pgmap v254: 129 pgs: 129 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 432 op/s 2024-05-27T07:47:38.267 INFO:tasks.workunit.client.0.smithi114.stdout: 1 797925 11.51 MB/sec execute 230 sec latency 28.212 ms 2024-05-27T07:47:39.273 INFO:tasks.workunit.client.0.smithi114.stdout: 1 799786 11.51 MB/sec execute 231 sec latency 15.621 ms 2024-05-27T07:47:39.478 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:39 smithi186 ceph-mon[36647]: pgmap v255: 129 pgs: 129 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 332 op/s 2024-05-27T07:47:39.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.490+0000 7fc6453cb700 1 -- 172.21.15.114:0/3981204868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc64010e8c0 msgr2=0x7fc640110da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:39.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.490+0000 7fc6453cb700 1 --2- 172.21.15.114:0/3981204868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc64010e8c0 0x7fc640110da0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fc63800b0a0 tx=0x7fc63800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:39 smithi114 ceph-mon[32038]: pgmap v255: 129 pgs: 129 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 332 op/s 2024-05-27T07:47:39.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.490+0000 7fc6453cb700 1 -- 172.21.15.114:0/3981204868 shutdown_connections 2024-05-27T07:47:39.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.490+0000 7fc6453cb700 1 --2- 172.21.15.114:0/3981204868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc64010e8c0 0x7fc640110da0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.490+0000 7fc6453cb700 1 --2- 172.21.15.114:0/3981204868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc640105450 0x7fc640105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.490+0000 7fc6453cb700 1 --2- 172.21.15.114:0/3981204868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc640105da0 0x7fc64010e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.490+0000 7fc6453cb700 1 -- 172.21.15.114:0/3981204868 >> 172.21.15.114:0/3981204868 conn(0x7fc6400fedb0 msgr2=0x7fc6401011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.490+0000 7fc6453cb700 1 -- 172.21.15.114:0/3981204868 shutdown_connections 2024-05-27T07:47:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.490+0000 7fc6453cb700 1 -- 172.21.15.114:0/3981204868 wait complete. 2024-05-27T07:47:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.491+0000 7fc6453cb700 1 Processor -- start 2024-05-27T07:47:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.491+0000 7fc6453cb700 1 -- start start 2024-05-27T07:47:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.491+0000 7fc6453cb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc640105450 0x7fc64019ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.491+0000 7fc6453cb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc640105da0 0x7fc64019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.491+0000 7fc6453cb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc64010e8c0 0x7fc64019f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.491+0000 7fc6453cb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc640114420 con 0x7fc640105da0 2024-05-27T07:47:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.491+0000 7fc6453cb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401142a0 con 0x7fc64010e8c0 2024-05-27T07:47:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.491+0000 7fc6453cb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401145a0 con 0x7fc640105450 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.491+0000 7fc63effd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc640105450 0x7fc64019ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.491+0000 7fc63e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc640105da0 0x7fc64019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.491+0000 7fc63effd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc640105450 0x7fc64019ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47150/0 (socket says 172.21.15.114:47150) 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc63effd700 1 -- 172.21.15.114:0/4267379530 learned_addr learned my addr 172.21.15.114:0/4267379530 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc63f7fe700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc64010e8c0 0x7fc64019f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc63effd700 1 -- 172.21.15.114:0/4267379530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc64010e8c0 msgr2=0x7fc64019f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc63effd700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc64010e8c0 0x7fc64019f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc63effd700 1 -- 172.21.15.114:0/4267379530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc640105da0 msgr2=0x7fc64019b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc63effd700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc640105da0 0x7fc64019b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc63effd700 1 -- 172.21.15.114:0/4267379530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc64019fc40 con 0x7fc640105450 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc63e7fc700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc640105da0 0x7fc64019b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc63effd700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc640105450 0x7fc64019ad70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc63000d950 tx=0x7fc63000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc627fff700 1 -- 172.21.15.114:0/4267379530 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc630016070 con 0x7fc640105450 2024-05-27T07:47:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc6453cb700 1 -- 172.21.15.114:0/4267379530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc64019ff30 con 0x7fc640105450 2024-05-27T07:47:39.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc627fff700 1 -- 172.21.15.114:0/4267379530 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc63000fd40 con 0x7fc640105450 2024-05-27T07:47:39.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc627fff700 1 -- 172.21.15.114:0/4267379530 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63000e3b0 con 0x7fc640105450 2024-05-27T07:47:39.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.492+0000 7fc6453cb700 1 -- 172.21.15.114:0/4267379530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6401a7850 con 0x7fc640105450 2024-05-27T07:47:39.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.495+0000 7fc627fff700 1 -- 172.21.15.114:0/4267379530 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fc630019070 con 0x7fc640105450 2024-05-27T07:47:39.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.496+0000 7fc627fff700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc6280b29f0 0x7fc6280b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.496+0000 7fc63e7fc700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc6280b29f0 0x7fc6280b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.496+0000 7fc627fff700 1 -- 172.21.15.114:0/4267379530 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fc6300e24a0 con 0x7fc640105450 2024-05-27T07:47:39.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.496+0000 7fc653de0b80 1 Processor -- start 2024-05-27T07:47:39.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.497+0000 7fc63e7fc700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc6280b29f0 0x7fc6280b4f00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc6401022a0 tx=0x7fc634009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:39.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.497+0000 7fc653de0b80 1 -- start start 2024-05-27T07:47:39.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.497+0000 7fc653de0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c121ec0 0x55c47c2a6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.497+0000 7fc653de0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 0x55c47c2a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.497+0000 7fc653de0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a11a0 0x55c47c2a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.497+0000 7fc653de0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c47c1cedf0 con 0x55c47c2a70f0 2024-05-27T07:47:39.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.497+0000 7fc653de0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c47c1be990 con 0x55c47c121ec0 2024-05-27T07:47:39.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.497+0000 7fc653de0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c47c1c29d0 con 0x55c47c2a11a0 2024-05-27T07:47:39.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.497+0000 7fc63e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 0x55c47c2a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.497+0000 7fc63e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 0x55c47c2a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58974/0 (socket says 172.21.15.114:58974) 2024-05-27T07:47:39.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.497+0000 7fc63e7fc700 1 -- 172.21.15.114:0/574474699 learned_addr learned my addr 172.21.15.114:0/574474699 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:39.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.497+0000 7fc63f7fe700 1 --2- 172.21.15.114:0/574474699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a11a0 0x55c47c2a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.497+0000 7fc63effd700 1 --2- 172.21.15.114:0/574474699 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c121ec0 0x55c47c2a6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.498+0000 7fc63e7fc700 1 -- 172.21.15.114:0/574474699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a11a0 msgr2=0x55c47c2a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:39.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.498+0000 7fc63e7fc700 1 --2- 172.21.15.114:0/574474699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a11a0 0x55c47c2a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.498+0000 7fc63e7fc700 1 -- 172.21.15.114:0/574474699 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c121ec0 msgr2=0x55c47c2a6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:39.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.498+0000 7fc63e7fc700 1 --2- 172.21.15.114:0/574474699 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c121ec0 0x55c47c2a6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.498+0000 7fc63e7fc700 1 -- 172.21.15.114:0/574474699 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c47c168030 con 0x55c47c2a70f0 2024-05-27T07:47:39.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.498+0000 7fc63e7fc700 1 --2- 172.21.15.114:0/574474699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 0x55c47c2a0c60 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc63400e720 tx=0x7fc634014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a468cda5278f3ad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:39.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.498+0000 7fc6257fa700 1 -- 172.21.15.114:0/574474699 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634026600 con 0x55c47c2a70f0 2024-05-27T07:47:39.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.498+0000 7fc6257fa700 1 -- 172.21.15.114:0/574474699 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc63401e8a0 con 0x55c47c2a70f0 2024-05-27T07:47:39.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.498+0000 7fc6257fa700 1 -- 172.21.15.114:0/574474699 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63401d840 con 0x55c47c2a70f0 2024-05-27T07:47:39.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.499+0000 7fc653de0b80 1 -- 172.21.15.114:0/574474699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 msgr2=0x55c47c2a0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:39.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.499+0000 7fc653de0b80 1 --2- 172.21.15.114:0/574474699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 0x55c47c2a0c60 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc63400e720 tx=0x7fc634014000 comp rx=0 tx=0).stop 2024-05-27T07:47:39.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.499+0000 7fc653de0b80 1 -- 172.21.15.114:0/574474699 shutdown_connections 2024-05-27T07:47:39.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.499+0000 7fc653de0b80 1 --2- 172.21.15.114:0/574474699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a11a0 0x55c47c2a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.499+0000 7fc653de0b80 1 --2- 172.21.15.114:0/574474699 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c121ec0 0x55c47c2a6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.499+0000 7fc653de0b80 1 --2- 172.21.15.114:0/574474699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 0x55c47c2a0c60 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.499+0000 7fc653de0b80 1 -- 172.21.15.114:0/574474699 >> 172.21.15.114:0/574474699 conn(0x55c47c1247f0 msgr2=0x55c47c29be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:39.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.500+0000 7fc653de0b80 1 -- 172.21.15.114:0/574474699 shutdown_connections 2024-05-27T07:47:39.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.500+0000 7fc653de0b80 1 -- 172.21.15.114:0/574474699 wait complete. 2024-05-27T07:47:39.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.500+0000 7fc653de0b80 1 -- start start 2024-05-27T07:47:39.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.500+0000 7fc653de0b80 1 Processor -- start 2024-05-27T07:47:39.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc653de0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c121ec0 0x55c47c2aa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc653de0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a11a0 0x55c47c2aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc653de0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c2a70f0 0x55c47c2ab420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc653de0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c47c0c5020 con 0x55c47c2a11a0 2024-05-27T07:47:39.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc653de0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c47c126e20 con 0x55c47c2a70f0 2024-05-27T07:47:39.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc653de0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c47c2adbe0 con 0x55c47c121ec0 2024-05-27T07:47:39.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc653de0b80 10 client.0 _ll_register_callbacks cb 0x55c47c155630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:39.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc63e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a11a0 0x55c47c2aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc63effd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c121ec0 0x55c47c2aa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc63effd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c121ec0 0x55c47c2aa9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47168/0 (socket says 172.21.15.114:47168) 2024-05-27T07:47:39.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc63effd700 1 -- 172.21.15.114:0/3643656769 learned_addr learned my addr 172.21.15.114:0/3643656769 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc63f7fe700 1 --2- 172.21.15.114:0/3643656769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c2a70f0 0x55c47c2ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc63effd700 1 -- 172.21.15.114:0/3643656769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c2a70f0 msgr2=0x55c47c2ab420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc63effd700 1 --2- 172.21.15.114:0/3643656769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c2a70f0 0x55c47c2ab420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc63effd700 1 -- 172.21.15.114:0/3643656769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a11a0 msgr2=0x55c47c2aaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc63effd700 1 --2- 172.21.15.114:0/3643656769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a11a0 0x55c47c2aaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc63effd700 1 -- 172.21.15.114:0/3643656769 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c47c10e950 con 0x55c47c121ec0 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc63e7fc700 1 --2- 172.21.15.114:0/3643656769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a11a0 0x55c47c2aaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc63effd700 1 --2- 172.21.15.114:0/3643656769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c121ec0 0x55c47c2aa9a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc6300df8a0 tx=0x7fc630000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=853b155bbf04d68 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.501+0000 7fc622ff5700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc622ff5700 1 -- 172.21.15.114:0/3643656769 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc630020070 con 0x55c47c121ec0 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc653de0b80 1 -- 172.21.15.114:0/3643656769 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc63401e8a0 con 0x55c47c121ec0 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc622ff5700 1 -- 172.21.15.114:0/3643656769 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fc6300a9630 con 0x55c47c121ec0 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc622ff5700 1 -- 172.21.15.114:0/3643656769 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fc6300a93d0 con 0x55c47c121ec0 2024-05-27T07:47:39.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc653de0b80 10 client.26857 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc653de0b80 1 -- 172.21.15.114:0/3643656769 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55c47c1787e0 con 0x55c47c121ec0 2024-05-27T07:47:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc622ff5700 1 -- 172.21.15.114:0/3643656769 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc63401e8a0 con 0x55c47c121ec0 2024-05-27T07:47:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc622ff5700 1 -- 172.21.15.114:0/3643656769 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7fc6300a8630 con 0x55c47c121ec0 2024-05-27T07:47:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc653de0b80 10 client.26857 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc653de0b80 10 client.26857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc653de0b80 20 client.26857 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc653de0b80 1 --2- 172.21.15.114:0/3643656769 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c47c2baa70 0x55c47c2bcf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc653de0b80 4 client.26857 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc653de0b80 1 -- 172.21.15.114:0/3643656769 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c47c1d0d40 con 0x55c47c2baa70 2024-05-27T07:47:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.502+0000 7fc63e7fc700 1 --2- 172.21.15.114:0/3643656769 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c47c2baa70 0x55c47c2bcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.503+0000 7fc622ff5700 10 client.26857 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.503+0000 7fc63e7fc700 1 --2- 172.21.15.114:0/3643656769 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c47c2baa70 0x55c47c2bcf20 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db8d806b5fa19345 server_cookie=2243abcc8879f50f in_seq=0 out_seq=0 2024-05-27T07:47:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc622ff5700 1 -- 172.21.15.114:0/3643656769 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c47c1d0d40 con 0x55c47c2baa70 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc622ff5700 10 client.26857 handle_command_reply: tid=0 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc653de0b80 1 client.26857 shutdown 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc653de0b80 1 -- 172.21.15.114:0/3643656769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c121ec0 msgr2=0x55c47c2aa9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc653de0b80 1 --2- 172.21.15.114:0/3643656769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c121ec0 0x55c47c2aa9a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc6300df8a0 tx=0x7fc630000bc0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc653de0b80 1 -- 172.21.15.114:0/3643656769 shutdown_connections 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc653de0b80 1 --2- 172.21.15.114:0/3643656769 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c47c2baa70 0x55c47c2bcf20 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc653de0b80 1 --2- 172.21.15.114:0/3643656769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c2a70f0 0x55c47c2ab420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc622ff5700 0 client.26857 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc653de0b80 1 --2- 172.21.15.114:0/3643656769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c121ec0 0x55c47c2aa9a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc653de0b80 1 --2- 172.21.15.114:0/3643656769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a11a0 0x55c47c2aaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc653de0b80 1 -- 172.21.15.114:0/3643656769 >> 172.21.15.114:0/3643656769 conn(0x55c47c1247f0 msgr2=0x55c47c29fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc653de0b80 1 -- 172.21.15.114:0/3643656769 shutdown_connections 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc653de0b80 1 -- 172.21.15.114:0/3643656769 wait complete. 2024-05-27T07:47:39.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.504+0000 7fc653de0b80 20 client.26857 trim_cache size 0 max 16384 2024-05-27T07:47:39.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.527+0000 7fc653de0b80 1 Processor -- start 2024-05-27T07:47:39.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.527+0000 7fc653de0b80 1 -- start start 2024-05-27T07:47:39.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc653de0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c121ec0 0x55c47c2ace40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc653de0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c2a11a0 0x55c47c2ab260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc653de0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a70f0 0x55c47c2a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc653de0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401ae4a0 con 0x55c47c121ec0 2024-05-27T07:47:39.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc653de0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401aeb70 con 0x55c47c2a11a0 2024-05-27T07:47:39.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc653de0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401ac2f0 con 0x55c47c2a70f0 2024-05-27T07:47:39.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc63effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c121ec0 0x55c47c2ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc63effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c121ec0 0x55c47c2ace40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58990/0 (socket says 172.21.15.114:58990) 2024-05-27T07:47:39.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc63effd700 1 -- 172.21.15.114:0/2952203380 learned_addr learned my addr 172.21.15.114:0/2952203380 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:39.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc63f7fe700 1 --2- 172.21.15.114:0/2952203380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a70f0 0x55c47c2a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc63e7fc700 1 --2- 172.21.15.114:0/2952203380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c2a11a0 0x55c47c2ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc63effd700 1 -- 172.21.15.114:0/2952203380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a70f0 msgr2=0x55c47c2a4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:39.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc63effd700 1 --2- 172.21.15.114:0/2952203380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a70f0 0x55c47c2a4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc63effd700 1 -- 172.21.15.114:0/2952203380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c2a11a0 msgr2=0x55c47c2ab260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:39.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc63effd700 1 --2- 172.21.15.114:0/2952203380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c2a11a0 0x55c47c2ab260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc63effd700 1 -- 172.21.15.114:0/2952203380 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c47c2ab7a0 con 0x55c47c121ec0 2024-05-27T07:47:39.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc63e7fc700 1 --2- 172.21.15.114:0/2952203380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c2a11a0 0x55c47c2ab260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:47:39.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc63effd700 1 --2- 172.21.15.114:0/2952203380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c121ec0 0x55c47c2ace40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc6300a9360 tx=0x7fc630010a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=983c881dfb291a5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:39.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.528+0000 7fc624ff9700 1 -- 172.21.15.114:0/2952203380 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc630020040 con 0x55c47c121ec0 2024-05-27T07:47:39.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.529+0000 7fc624ff9700 1 -- 172.21.15.114:0/2952203380 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6300ad780 con 0x55c47c121ec0 2024-05-27T07:47:39.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.529+0000 7fc624ff9700 1 -- 172.21.15.114:0/2952203380 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6300a8790 con 0x55c47c121ec0 2024-05-27T07:47:39.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.529+0000 7fc653de0b80 1 -- 172.21.15.114:0/2952203380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c121ec0 msgr2=0x55c47c2ace40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:39.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.529+0000 7fc653de0b80 1 --2- 172.21.15.114:0/2952203380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c121ec0 0x55c47c2ace40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc6300a9360 tx=0x7fc630010a40 comp rx=0 tx=0).stop 2024-05-27T07:47:39.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 -- 172.21.15.114:0/2952203380 shutdown_connections 2024-05-27T07:47:39.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 --2- 172.21.15.114:0/2952203380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a70f0 0x55c47c2a4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 --2- 172.21.15.114:0/2952203380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c121ec0 0x55c47c2ace40 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 --2- 172.21.15.114:0/2952203380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c2a11a0 0x55c47c2ab260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 -- 172.21.15.114:0/2952203380 >> 172.21.15.114:0/2952203380 conn(0x55c47c1247f0 msgr2=0x55c47c2a4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 -- 172.21.15.114:0/2952203380 shutdown_connections 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 -- 172.21.15.114:0/2952203380 wait complete. 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 -- start start 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 Processor -- start 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c121ec0 0x55c47c2b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a11a0 0x55c47c2b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 0x55c47c2afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401ac960 con 0x55c47c2a70f0 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401addd0 con 0x55c47c121ec0 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.530+0000 7fc653de0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401ad030 con 0x55c47c2a11a0 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc653de0b80 10 client.0 _ll_register_callbacks cb 0x55c47c155630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:39.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc63f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 0x55c47c2afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc63e7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a11a0 0x55c47c2b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc63f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 0x55c47c2afe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58996/0 (socket says 172.21.15.114:58996) 2024-05-27T07:47:39.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc63f7fe700 1 -- 172.21.15.114:0/2330714549 learned_addr learned my addr 172.21.15.114:0/2330714549 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:39.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc63effd700 1 --2- 172.21.15.114:0/2330714549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c121ec0 0x55c47c2b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc63f7fe700 1 -- 172.21.15.114:0/2330714549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a11a0 msgr2=0x55c47c2b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:39.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc63f7fe700 1 --2- 172.21.15.114:0/2330714549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a11a0 0x55c47c2b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc63f7fe700 1 -- 172.21.15.114:0/2330714549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c121ec0 msgr2=0x55c47c2b2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:39.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc63f7fe700 1 --2- 172.21.15.114:0/2330714549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c121ec0 0x55c47c2b2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc63f7fe700 1 -- 172.21.15.114:0/2330714549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c47c2c31b0 con 0x55c47c2a70f0 2024-05-27T07:47:39.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc63f7fe700 1 --2- 172.21.15.114:0/2330714549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 0x55c47c2afe40 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc638015000 tx=0x7fc638003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82698342b0ad4a09 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:39.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc622ff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:39.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc622ff5700 1 -- 172.21.15.114:0/2330714549 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6380075c0 con 0x55c47c2a70f0 2024-05-27T07:47:39.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc622ff5700 1 -- 172.21.15.114:0/2330714549 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fc638007760 con 0x55c47c2a70f0 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc653de0b80 1 -- 172.21.15.114:0/2330714549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6300ad780 con 0x55c47c2a70f0 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc622ff5700 1 -- 172.21.15.114:0/2330714549 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fc638007960 con 0x55c47c2a70f0 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc653de0b80 10 client.17292 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.531+0000 7fc653de0b80 1 -- 172.21.15.114:0/2330714549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55c47c325ae0 con 0x55c47c2a70f0 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.532+0000 7fc622ff5700 1 -- 172.21.15.114:0/2330714549 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6300ad780 con 0x55c47c2a70f0 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.532+0000 7fc622ff5700 1 -- 172.21.15.114:0/2330714549 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7fc638025070 con 0x55c47c2a70f0 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.532+0000 7fc653de0b80 10 client.17292 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.532+0000 7fc653de0b80 10 client.17292 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.532+0000 7fc653de0b80 20 client.17292 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.532+0000 7fc653de0b80 1 --2- 172.21.15.114:0/2330714549 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c47c326250 0x55c47c328670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.532+0000 7fc653de0b80 4 client.17292 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.532+0000 7fc653de0b80 1 -- 172.21.15.114:0/2330714549 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c47c1ca520 con 0x55c47c326250 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.532+0000 7fc63effd700 1 --2- 172.21.15.114:0/2330714549 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c47c326250 0x55c47c328670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:39.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.532+0000 7fc63effd700 1 --2- 172.21.15.114:0/2330714549 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c47c326250 0x55c47c328670 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5296d14d0cb44292 server_cookie=7c951f0cd20068cb in_seq=0 out_seq=0 2024-05-27T07:47:39.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.532+0000 7fc622ff5700 10 client.17292 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:39.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.534+0000 7fc622ff5700 1 -- 172.21.15.114:0/2330714549 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c47c1ca520 con 0x55c47c326250 2024-05-27T07:47:39.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.534+0000 7fc622ff5700 10 client.17292 handle_command_reply: tid=0 2024-05-27T07:47:39.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.534+0000 7fc653de0b80 1 client.17292 shutdown 2024-05-27T07:47:39.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.534+0000 7fc653de0b80 1 -- 172.21.15.114:0/2330714549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 msgr2=0x55c47c2afe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:39.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.534+0000 7fc653de0b80 1 --2- 172.21.15.114:0/2330714549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 0x55c47c2afe40 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc638015000 tx=0x7fc638003c30 comp rx=0 tx=0).stop 2024-05-27T07:47:39.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc653de0b80 1 -- 172.21.15.114:0/2330714549 shutdown_connections 2024-05-27T07:47:39.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc653de0b80 1 --2- 172.21.15.114:0/2330714549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c47c2a70f0 0x55c47c2afe40 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc653de0b80 1 --2- 172.21.15.114:0/2330714549 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c47c326250 0x55c47c328670 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc653de0b80 1 --2- 172.21.15.114:0/2330714549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c47c121ec0 0x55c47c2b2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc653de0b80 1 --2- 172.21.15.114:0/2330714549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c47c2a11a0 0x55c47c2b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc622ff5700 0 client.17292 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:39.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc653de0b80 1 -- 172.21.15.114:0/2330714549 >> 172.21.15.114:0/2330714549 conn(0x55c47c1247f0 msgr2=0x55c47c295570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:39.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc653de0b80 1 -- 172.21.15.114:0/2330714549 shutdown_connections 2024-05-27T07:47:39.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc653de0b80 1 -- 172.21.15.114:0/2330714549 wait complete. 2024-05-27T07:47:39.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc653de0b80 20 client.17292 trim_cache size 0 max 16384 2024-05-27T07:47:39.538 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:47:39.538 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:47:39.538 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "a046a8f4-0875-47d3-b658-9e1c90981b17", 2024-05-27T07:47:39.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:47:39.539 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:47:39.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc6453cb700 1 -- 172.21.15.114:0/4267379530 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc6280b29f0 msgr2=0x7fc6280b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:39.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc6453cb700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc6280b29f0 0x7fc6280b4f00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc6401022a0 tx=0x7fc634009260 comp rx=0 tx=0).stop 2024-05-27T07:47:39.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc6453cb700 1 -- 172.21.15.114:0/4267379530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc640105450 msgr2=0x7fc64019ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:39.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc6453cb700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc640105450 0x7fc64019ad70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc63000d950 tx=0x7fc63000dc60 comp rx=0 tx=0).stop 2024-05-27T07:47:39.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc6453cb700 1 -- 172.21.15.114:0/4267379530 shutdown_connections 2024-05-27T07:47:39.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc6453cb700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc6280b29f0 0x7fc6280b4f00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc6453cb700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc64010e8c0 0x7fc64019f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc6453cb700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc640105450 0x7fc64019ad70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc6453cb700 1 --2- 172.21.15.114:0/4267379530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc640105da0 0x7fc64019b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:39.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.535+0000 7fc6453cb700 1 -- 172.21.15.114:0/4267379530 >> 172.21.15.114:0/4267379530 conn(0x7fc6400fedb0 msgr2=0x7fc6400ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:39.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.536+0000 7fc6453cb700 1 -- 172.21.15.114:0/4267379530 shutdown_connections 2024-05-27T07:47:39.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:39.536+0000 7fc6453cb700 1 -- 172.21.15.114:0/4267379530 wait complete. 2024-05-27T07:47:39.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:39 smithi163 ceph-mon[36451]: pgmap v255: 129 pgs: 129 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 332 op/s 2024-05-27T07:47:40.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:47:40.273 INFO:tasks.workunit.client.0.smithi114.stdout: 1 802517 11.53 MB/sec execute 232 sec latency 39.545 ms 2024-05-27T07:47:40.510 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:47:40.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:40 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:47:40.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:40 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:40.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:40 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:47:40.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:40 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:40.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:40 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:47:40.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:40 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:40.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:40 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:47:40.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:40 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:40.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:40 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:47:40.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:40 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:40.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:40 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:47:40.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:40 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:41.273 INFO:tasks.workunit.client.0.smithi114.stdout: 1 804272 11.51 MB/sec execute 233 sec latency 18.487 ms 2024-05-27T07:47:41.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:41 smithi163 ceph-mon[36451]: pgmap v256: 129 pgs: 129 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 341 op/s 2024-05-27T07:47:41.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:41 smithi186 ceph-mon[36647]: pgmap v256: 129 pgs: 129 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 341 op/s 2024-05-27T07:47:41.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:41 smithi114 ceph-mon[32038]: pgmap v256: 129 pgs: 129 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 341 op/s 2024-05-27T07:47:42.273 INFO:tasks.workunit.client.0.smithi114.stdout: 1 806364 11.52 MB/sec execute 234 sec latency 70.715 ms 2024-05-27T07:47:42.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.328+0000 7f5245fb6700 1 -- 172.21.15.114:0/1595705089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f524010a3d0 msgr2=0x7f524010a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:42.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.328+0000 7f5245fb6700 1 --2- 172.21.15.114:0/1595705089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f524010a3d0 0x7f524010a7e0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f5230009990 tx=0x7f5230009ca0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.328+0000 7f5245fb6700 1 -- 172.21.15.114:0/1595705089 shutdown_connections 2024-05-27T07:47:42.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.328+0000 7f5245fb6700 1 --2- 172.21.15.114:0/1595705089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5240103120 0x7f5240111110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.328+0000 7f5245fb6700 1 --2- 172.21.15.114:0/1595705089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f524010a3d0 0x7f524010a7e0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.328+0000 7f5245fb6700 1 --2- 172.21.15.114:0/1595705089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52401027d0 0x7f5240102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.328+0000 7f5245fb6700 1 -- 172.21.15.114:0/1595705089 >> 172.21.15.114:0/1595705089 conn(0x7f52400fe6e0 msgr2=0x7f5240100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:42.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.328+0000 7f5245fb6700 1 -- 172.21.15.114:0/1595705089 shutdown_connections 2024-05-27T07:47:42.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.328+0000 7f5245fb6700 1 -- 172.21.15.114:0/1595705089 wait complete. 2024-05-27T07:47:42.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.329+0000 7f5245fb6700 1 Processor -- start 2024-05-27T07:47:42.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.330+0000 7f5245fb6700 1 -- start start 2024-05-27T07:47:42.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.330+0000 7f5245fb6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52401027d0 0x7f524019ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.330+0000 7f5245fb6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5240103120 0x7f524019b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.330+0000 7f5245fb6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f524010a3d0 0x7f524019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.330+0000 7f5245fb6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f52401144c0 con 0x7f524010a3d0 2024-05-27T07:47:42.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.330+0000 7f5245fb6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5240114340 con 0x7f5240103120 2024-05-27T07:47:42.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.330+0000 7f5245fb6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5240114640 con 0x7f52401027d0 2024-05-27T07:47:42.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.330+0000 7f523ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f524010a3d0 0x7f524019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.330+0000 7f523ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f524010a3d0 0x7f524019f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59020/0 (socket says 172.21.15.114:59020) 2024-05-27T07:47:42.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.330+0000 7f523ffff700 1 -- 172.21.15.114:0/791277370 learned_addr learned my addr 172.21.15.114:0/791277370 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:42.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.330+0000 7f523effd700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5240103120 0x7f524019b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.330+0000 7f523f7fe700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52401027d0 0x7f524019ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.331+0000 7f523ffff700 1 -- 172.21.15.114:0/791277370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52401027d0 msgr2=0x7f524019ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:42.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.331+0000 7f523ffff700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52401027d0 0x7f524019ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.331+0000 7f523ffff700 1 -- 172.21.15.114:0/791277370 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5240103120 msgr2=0x7f524019b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:42.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.331+0000 7f523ffff700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5240103120 0x7f524019b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.331+0000 7f523ffff700 1 -- 172.21.15.114:0/791277370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524019fd20 con 0x7f524010a3d0 2024-05-27T07:47:42.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.331+0000 7f523effd700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5240103120 0x7f524019b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:42.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.331+0000 7f523ffff700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f524010a3d0 0x7f524019f680 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f523800ef30 tx=0x7f523800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:42.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.331+0000 7f523cff9700 1 -- 172.21.15.114:0/791277370 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52380092e0 con 0x7f524010a3d0 2024-05-27T07:47:42.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.331+0000 7f523cff9700 1 -- 172.21.15.114:0/791277370 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5238004bb0 con 0x7f524010a3d0 2024-05-27T07:47:42.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.331+0000 7f5245fb6700 1 -- 172.21.15.114:0/791277370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f524019ffb0 con 0x7f524010a3d0 2024-05-27T07:47:42.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.331+0000 7f523cff9700 1 -- 172.21.15.114:0/791277370 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5238004020 con 0x7f524010a3d0 2024-05-27T07:47:42.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.331+0000 7f5245fb6700 1 -- 172.21.15.114:0/791277370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52401a04a0 con 0x7f524010a3d0 2024-05-27T07:47:42.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.334+0000 7f523cff9700 1 -- 172.21.15.114:0/791277370 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f523801b030 con 0x7f524010a3d0 2024-05-27T07:47:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.335+0000 7f523cff9700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f52280b29f0 0x7f52280b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.335+0000 7f523f7fe700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f52280b29f0 0x7f52280b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.335+0000 7f523cff9700 1 -- 172.21.15.114:0/791277370 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f52380e23f0 con 0x7f524010a3d0 2024-05-27T07:47:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.335+0000 7f52549cbb80 1 Processor -- start 2024-05-27T07:47:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.335+0000 7f52549cbb80 1 -- start start 2024-05-27T07:47:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.335+0000 7f523f7fe700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f52280b29f0 0x7f52280b4f00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f523000b2e0 tx=0x7f5230005fd0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:42.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.335+0000 7f52549cbb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed7e5e40 0x5591ed96ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.335+0000 7f52549cbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed96b070 0x5591ed964be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f52549cbb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed965120 0x5591ed965530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f52549cbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5591ed892d70 con 0x5591ed96b070 2024-05-27T07:47:42.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f52549cbb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5591ed882910 con 0x5591ed965120 2024-05-27T07:47:42.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f52549cbb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5591ed886950 con 0x5591ed7e5e40 2024-05-27T07:47:42.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f523f7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed7e5e40 0x5591ed96ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f523effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed96b070 0x5591ed964be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f523f7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed7e5e40 0x5591ed96ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47214/0 (socket says 172.21.15.114:47214) 2024-05-27T07:47:42.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f523f7fe700 1 -- 172.21.15.114:0/67369593 learned_addr learned my addr 172.21.15.114:0/67369593 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:42.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f523ffff700 1 --2- 172.21.15.114:0/67369593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed965120 0x5591ed965530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f523f7fe700 1 -- 172.21.15.114:0/67369593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed965120 msgr2=0x5591ed965530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f523f7fe700 1 --2- 172.21.15.114:0/67369593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed965120 0x5591ed965530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f523f7fe700 1 -- 172.21.15.114:0/67369593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed96b070 msgr2=0x5591ed964be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f523f7fe700 1 --2- 172.21.15.114:0/67369593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed96b070 0x5591ed964be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f523f7fe700 1 -- 172.21.15.114:0/67369593 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591ed82bfb0 con 0x5591ed7e5e40 2024-05-27T07:47:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f523effd700 1 --2- 172.21.15.114:0/67369593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed96b070 0x5591ed964be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f523ffff700 1 --2- 172.21.15.114:0/67369593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed965120 0x5591ed965530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:47:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f523f7fe700 1 --2- 172.21.15.114:0/67369593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed7e5e40 0x5591ed96ab30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5230015000 tx=0x7f523001e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=855c8637656f453c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.336+0000 7f5225ffb700 1 -- 172.21.15.114:0/67369593 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523000be10 con 0x5591ed7e5e40 2024-05-27T07:47:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.337+0000 7f5225ffb700 1 -- 172.21.15.114:0/67369593 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5230003b30 con 0x5591ed7e5e40 2024-05-27T07:47:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.337+0000 7f5225ffb700 1 -- 172.21.15.114:0/67369593 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523000fa80 con 0x5591ed7e5e40 2024-05-27T07:47:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.337+0000 7f52549cbb80 1 -- 172.21.15.114:0/67369593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed7e5e40 msgr2=0x5591ed96ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.337+0000 7f52549cbb80 1 --2- 172.21.15.114:0/67369593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed7e5e40 0x5591ed96ab30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5230015000 tx=0x7f523001e000 comp rx=0 tx=0).stop 2024-05-27T07:47:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.338+0000 7f52549cbb80 1 -- 172.21.15.114:0/67369593 shutdown_connections 2024-05-27T07:47:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.338+0000 7f52549cbb80 1 --2- 172.21.15.114:0/67369593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed965120 0x5591ed965530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.338+0000 7f52549cbb80 1 --2- 172.21.15.114:0/67369593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed7e5e40 0x5591ed96ab30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.338+0000 7f52549cbb80 1 --2- 172.21.15.114:0/67369593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed96b070 0x5591ed964be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.338+0000 7f52549cbb80 1 -- 172.21.15.114:0/67369593 >> 172.21.15.114:0/67369593 conn(0x5591ed7e8770 msgr2=0x5591ed95fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.338+0000 7f52549cbb80 1 -- 172.21.15.114:0/67369593 shutdown_connections 2024-05-27T07:47:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.338+0000 7f52549cbb80 1 -- 172.21.15.114:0/67369593 wait complete. 2024-05-27T07:47:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.338+0000 7f52549cbb80 1 -- start start 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f52549cbb80 1 Processor -- start 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f52549cbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 0x5591ed96e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f52549cbb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed965120 0x5591ed96ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f52549cbb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 0x5591ed96f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f52549cbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5591ed788fa0 con 0x5591ed7e5e40 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f52549cbb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5591ed7eada0 con 0x5591ed96b070 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f52549cbb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5591ed975eb0 con 0x5591ed965120 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f52549cbb80 10 client.0 _ll_register_callbacks cb 0x5591ed8195b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f523f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 0x5591ed96e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f523f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 0x5591ed96e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59042/0 (socket says 172.21.15.114:59042) 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f523f7fe700 1 -- 172.21.15.114:0/3202502255 learned_addr learned my addr 172.21.15.114:0/3202502255 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f523effd700 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed965120 0x5591ed96ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f523ffff700 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 0x5591ed96f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f523effd700 1 -- 172.21.15.114:0/3202502255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 msgr2=0x5591ed96f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f523effd700 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 0x5591ed96f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f523effd700 1 -- 172.21.15.114:0/3202502255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 msgr2=0x5591ed96e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f523effd700 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 0x5591ed96e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f523effd700 1 -- 172.21.15.114:0/3202502255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591ed7d28d0 con 0x5591ed965120 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.339+0000 7f523f7fe700 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 0x5591ed96e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.340+0000 7f523effd700 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed965120 0x5591ed96ed70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f52401001f0 tx=0x7f5234009e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa7ecf4c2cbe8c0b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.340+0000 7f521effd700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.340+0000 7f521effd700 1 -- 172.21.15.114:0/3202502255 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5234004030 con 0x5591ed965120 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.340+0000 7f521effd700 1 -- 172.21.15.114:0/3202502255 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f52340041d0 con 0x5591ed965120 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.340+0000 7f52549cbb80 1 -- 172.21.15.114:0/3202502255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5230003b30 con 0x5591ed965120 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.340+0000 7f521effd700 1 -- 172.21.15.114:0/3202502255 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f5234025050 con 0x5591ed965120 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.340+0000 7f52549cbb80 10 client.26875 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.340+0000 7f52549cbb80 1 -- 172.21.15.114:0/3202502255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5591ed83c760 con 0x5591ed965120 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.340+0000 7f521effd700 1 -- 172.21.15.114:0/3202502255 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5230003b30 con 0x5591ed965120 2024-05-27T07:47:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.340+0000 7f521effd700 1 -- 172.21.15.114:0/3202502255 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7f52340039b0 con 0x5591ed965120 2024-05-27T07:47:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.340+0000 7f52549cbb80 10 client.26875 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.340+0000 7f52549cbb80 10 client.26875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.340+0000 7f52549cbb80 20 client.26875 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.341+0000 7f52549cbb80 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591ed982d80 0x5591ed9851a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.341+0000 7f52549cbb80 4 client.26875 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.341+0000 7f52549cbb80 1 -- 172.21.15.114:0/3202502255 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591ed894cc0 con 0x5591ed982d80 2024-05-27T07:47:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.341+0000 7f523f7fe700 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591ed982d80 0x5591ed9851a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.341+0000 7f521effd700 10 client.26875 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.341+0000 7f523f7fe700 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591ed982d80 0x5591ed9851a0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a32910542d3c68d5 server_cookie=6d7cdada8a16cb36 in_seq=0 out_seq=0 2024-05-27T07:47:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.342+0000 7f521effd700 1 -- 172.21.15.114:0/3202502255 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5591ed894cc0 con 0x5591ed982d80 2024-05-27T07:47:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.342+0000 7f521effd700 10 client.26875 handle_command_reply: tid=0 2024-05-27T07:47:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.342+0000 7f52549cbb80 1 client.26875 shutdown 2024-05-27T07:47:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.343+0000 7f52549cbb80 1 -- 172.21.15.114:0/3202502255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed965120 msgr2=0x5591ed96ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.343+0000 7f52549cbb80 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed965120 0x5591ed96ed70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f52401001f0 tx=0x7f5234009e20 comp rx=0 tx=0).stop 2024-05-27T07:47:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.343+0000 7f52549cbb80 1 -- 172.21.15.114:0/3202502255 shutdown_connections 2024-05-27T07:47:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.343+0000 7f52549cbb80 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591ed982d80 0x5591ed9851a0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.343+0000 7f52549cbb80 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 0x5591ed96f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.343+0000 7f52549cbb80 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 0x5591ed96e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.343+0000 7f521effd700 0 client.26875 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.343+0000 7f52549cbb80 1 --2- 172.21.15.114:0/3202502255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed965120 0x5591ed96ed70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.343+0000 7f52549cbb80 1 -- 172.21.15.114:0/3202502255 >> 172.21.15.114:0/3202502255 conn(0x5591ed7e8770 msgr2=0x5591ed963bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.343+0000 7f52549cbb80 1 -- 172.21.15.114:0/3202502255 shutdown_connections 2024-05-27T07:47:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.343+0000 7f52549cbb80 1 -- 172.21.15.114:0/3202502255 wait complete. 2024-05-27T07:47:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.343+0000 7f52549cbb80 20 client.26875 trim_cache size 0 max 16384 2024-05-27T07:47:42.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.363+0000 7f52549cbb80 1 Processor -- start 2024-05-27T07:47:42.364 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:42 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:47:42.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.363+0000 7f52549cbb80 1 -- start start 2024-05-27T07:47:42.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.363+0000 7f52549cbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 0x5591ed964be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f52549cbb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed965120 0x5591ed96f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f52549cbb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 0x5591ed97ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f52549cbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f52401ae470 con 0x5591ed7e5e40 2024-05-27T07:47:42.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f52549cbb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f52401aeb40 con 0x5591ed96b070 2024-05-27T07:47:42.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f52549cbb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f52401ac2c0 con 0x5591ed965120 2024-05-27T07:47:42.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f523f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 0x5591ed964be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f523f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 0x5591ed964be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59052/0 (socket says 172.21.15.114:59052) 2024-05-27T07:47:42.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f523f7fe700 1 -- 172.21.15.114:0/2944989355 learned_addr learned my addr 172.21.15.114:0/2944989355 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:42.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f523effd700 1 --2- 172.21.15.114:0/2944989355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed965120 0x5591ed96f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f523ffff700 1 --2- 172.21.15.114:0/2944989355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 0x5591ed97ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f523f7fe700 1 -- 172.21.15.114:0/2944989355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed965120 msgr2=0x5591ed96f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:42.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f523f7fe700 1 --2- 172.21.15.114:0/2944989355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed965120 0x5591ed96f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f523f7fe700 1 -- 172.21.15.114:0/2944989355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 msgr2=0x5591ed97ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:42.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f523f7fe700 1 --2- 172.21.15.114:0/2944989355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 0x5591ed97ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f523f7fe700 1 -- 172.21.15.114:0/2944989355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591ed9592f0 con 0x5591ed7e5e40 2024-05-27T07:47:42.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f523ffff700 1 --2- 172.21.15.114:0/2944989355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 0x5591ed97ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:42.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f523f7fe700 1 --2- 172.21.15.114:0/2944989355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 0x5591ed964be0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f5230009990 tx=0x7f5230021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c96d1e5f1184e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:42.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.364+0000 7f52257fa700 1 -- 172.21.15.114:0/2944989355 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230042070 con 0x5591ed7e5e40 2024-05-27T07:47:42.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.365+0000 7f52257fa700 1 -- 172.21.15.114:0/2944989355 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f523000fd40 con 0x5591ed7e5e40 2024-05-27T07:47:42.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.365+0000 7f52257fa700 1 -- 172.21.15.114:0/2944989355 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230023660 con 0x5591ed7e5e40 2024-05-27T07:47:42.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.365+0000 7f52549cbb80 1 -- 172.21.15.114:0/2944989355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 msgr2=0x5591ed964be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:42.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.365+0000 7f52549cbb80 1 --2- 172.21.15.114:0/2944989355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 0x5591ed964be0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f5230009990 tx=0x7f5230021000 comp rx=0 tx=0).stop 2024-05-27T07:47:42.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.365+0000 7f52549cbb80 1 -- 172.21.15.114:0/2944989355 shutdown_connections 2024-05-27T07:47:42.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.365+0000 7f52549cbb80 1 --2- 172.21.15.114:0/2944989355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 0x5591ed97ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.365+0000 7f52549cbb80 1 --2- 172.21.15.114:0/2944989355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed7e5e40 0x5591ed964be0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.365+0000 7f52549cbb80 1 --2- 172.21.15.114:0/2944989355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed965120 0x5591ed96f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.365+0000 7f52549cbb80 1 -- 172.21.15.114:0/2944989355 >> 172.21.15.114:0/2944989355 conn(0x5591ed7e8770 msgr2=0x5591ed969db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:42.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.366+0000 7f52549cbb80 1 -- 172.21.15.114:0/2944989355 shutdown_connections 2024-05-27T07:47:42.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.366+0000 7f52549cbb80 1 -- 172.21.15.114:0/2944989355 wait complete. 2024-05-27T07:47:42.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.366+0000 7f52549cbb80 1 -- start start 2024-05-27T07:47:42.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.366+0000 7f52549cbb80 1 Processor -- start 2024-05-27T07:47:42.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f52549cbb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed7e5e40 0x5591ed97b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f52549cbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed965120 0x5591ed97b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f52549cbb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 0x5591ed989cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f52549cbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f52401ac930 con 0x5591ed965120 2024-05-27T07:47:42.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f52549cbb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f52401adda0 con 0x5591ed96b070 2024-05-27T07:47:42.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f52549cbb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f52401ad000 con 0x5591ed7e5e40 2024-05-27T07:47:42.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f52549cbb80 10 client.0 _ll_register_callbacks cb 0x5591ed8195b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:42.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f523effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed965120 0x5591ed97b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f523f7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed7e5e40 0x5591ed97b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f523effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed965120 0x5591ed97b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59064/0 (socket says 172.21.15.114:59064) 2024-05-27T07:47:42.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f523effd700 1 -- 172.21.15.114:0/384189394 learned_addr learned my addr 172.21.15.114:0/384189394 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:42.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f523ffff700 1 --2- 172.21.15.114:0/384189394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 0x5591ed989cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f523effd700 1 -- 172.21.15.114:0/384189394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed7e5e40 msgr2=0x5591ed97b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:42.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f523effd700 1 --2- 172.21.15.114:0/384189394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed7e5e40 0x5591ed97b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f523effd700 1 -- 172.21.15.114:0/384189394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 msgr2=0x5591ed989cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:42.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f523effd700 1 --2- 172.21.15.114:0/384189394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 0x5591ed989cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f523effd700 1 -- 172.21.15.114:0/384189394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591ed98a470 con 0x5591ed965120 2024-05-27T07:47:42.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f523f7fe700 1 --2- 172.21.15.114:0/384189394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed7e5e40 0x5591ed97b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:42.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f523effd700 1 --2- 172.21.15.114:0/384189394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed965120 0x5591ed97b900 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x5591ed95bae0 tx=0x7f523400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=840110e7b3f7ac5d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:42.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f521effd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:42.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.367+0000 7f521effd700 1 -- 172.21.15.114:0/384189394 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5234003a10 con 0x5591ed965120 2024-05-27T07:47:42.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f521effd700 1 -- 172.21.15.114:0/384189394 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f5234018070 con 0x5591ed965120 2024-05-27T07:47:42.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f52549cbb80 1 -- 172.21.15.114:0/384189394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f523000fd40 con 0x5591ed965120 2024-05-27T07:47:42.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f521effd700 1 -- 172.21.15.114:0/384189394 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f5234038050 con 0x5591ed965120 2024-05-27T07:47:42.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f52549cbb80 10 client.17310 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f52549cbb80 1 -- 172.21.15.114:0/384189394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5591ed9ed230 con 0x5591ed965120 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f521effd700 1 -- 172.21.15.114:0/384189394 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f523000fd40 con 0x5591ed965120 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f521effd700 1 -- 172.21.15.114:0/384189394 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7f52340054f0 con 0x5591ed965120 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f52549cbb80 10 client.17310 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f52549cbb80 10 client.17310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f52549cbb80 20 client.17310 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f52549cbb80 1 --2- 172.21.15.114:0/384189394 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591ed9ed960 0x5591ed9efd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f52549cbb80 4 client.17310 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f52549cbb80 1 -- 172.21.15.114:0/384189394 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5591ed88e4a0 con 0x5591ed9ed960 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.368+0000 7f523f7fe700 1 --2- 172.21.15.114:0/384189394 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591ed9ed960 0x5591ed9efd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.369+0000 7f521effd700 10 client.17310 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.369+0000 7f523f7fe700 1 --2- 172.21.15.114:0/384189394 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591ed9ed960 0x5591ed9efd80 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bd696539af530f6 server_cookie=d897c1cec40e617f in_seq=0 out_seq=0 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.370+0000 7f521effd700 1 -- 172.21.15.114:0/384189394 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5591ed88e4a0 con 0x5591ed9ed960 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.370+0000 7f521effd700 10 client.17310 handle_command_reply: tid=0 2024-05-27T07:47:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.370+0000 7f52549cbb80 1 client.17310 shutdown 2024-05-27T07:47:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.370+0000 7f52549cbb80 1 -- 172.21.15.114:0/384189394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed965120 msgr2=0x5591ed97b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.370+0000 7f52549cbb80 1 --2- 172.21.15.114:0/384189394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed965120 0x5591ed97b900 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x5591ed95bae0 tx=0x7f523400f730 comp rx=0 tx=0).stop 2024-05-27T07:47:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.370+0000 7f52549cbb80 1 -- 172.21.15.114:0/384189394 shutdown_connections 2024-05-27T07:47:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.370+0000 7f52549cbb80 1 --2- 172.21.15.114:0/384189394 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591ed9ed960 0x5591ed9efd80 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.370+0000 7f52549cbb80 1 --2- 172.21.15.114:0/384189394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591ed96b070 0x5591ed989cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.370+0000 7f52549cbb80 1 --2- 172.21.15.114:0/384189394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ed7e5e40 0x5591ed97b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.370+0000 7f52549cbb80 1 --2- 172.21.15.114:0/384189394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591ed965120 0x5591ed97b900 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.370+0000 7f52549cbb80 1 -- 172.21.15.114:0/384189394 >> 172.21.15.114:0/384189394 conn(0x5591ed7e8770 msgr2=0x5591ed969db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f521effd700 0 client.17310 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f52549cbb80 1 -- 172.21.15.114:0/384189394 shutdown_connections 2024-05-27T07:47:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f52549cbb80 1 -- 172.21.15.114:0/384189394 wait complete. 2024-05-27T07:47:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f52549cbb80 20 client.17310 trim_cache size 0 max 16384 2024-05-27T07:47:42.377 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:47:42.377 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:47:42.377 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:47:42.377 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:47:42.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f5245fb6700 1 -- 172.21.15.114:0/791277370 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f52280b29f0 msgr2=0x7f52280b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:42.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f5245fb6700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f52280b29f0 0x7f52280b4f00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f523000b2e0 tx=0x7f5230005fd0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f5245fb6700 1 -- 172.21.15.114:0/791277370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f524010a3d0 msgr2=0x7f524019f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:42.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f5245fb6700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f524010a3d0 0x7f524019f680 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f523800ef30 tx=0x7f523800c3a0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f5245fb6700 1 -- 172.21.15.114:0/791277370 shutdown_connections 2024-05-27T07:47:42.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f5245fb6700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f524010a3d0 0x7f524019f680 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f5245fb6700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52401027d0 0x7f524019ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f5245fb6700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f52280b29f0 0x7f52280b4f00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f5245fb6700 1 --2- 172.21.15.114:0/791277370 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5240103120 0x7f524019b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:42.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.371+0000 7f5245fb6700 1 -- 172.21.15.114:0/791277370 >> 172.21.15.114:0/791277370 conn(0x7f52400fe6e0 msgr2=0x7f5240100b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:42.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.372+0000 7f5245fb6700 1 -- 172.21.15.114:0/791277370 shutdown_connections 2024-05-27T07:47:42.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:42.372+0000 7f5245fb6700 1 -- 172.21.15.114:0/791277370 wait complete. 2024-05-27T07:47:42.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:42 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:47:42.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:42 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:47:43.273 INFO:tasks.workunit.client.0.smithi114.stdout: 1 808096 11.50 MB/sec execute 235 sec latency 27.643 ms 2024-05-27T07:47:43.390 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:47:43.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:47:43.693 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:47:43.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:43 smithi163 ceph-mon[36451]: pgmap v257: 129 pgs: 129 active+clean; 75 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 347 op/s 2024-05-27T07:47:43.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:43 smithi186 ceph-mon[36647]: pgmap v257: 129 pgs: 129 active+clean; 75 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 347 op/s 2024-05-27T07:47:43.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:43 smithi114 ceph-mon[32038]: pgmap v257: 129 pgs: 129 active+clean; 75 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 347 op/s 2024-05-27T07:47:44.273 INFO:tasks.workunit.client.0.smithi114.stdout: 1 810292 11.51 MB/sec execute 236 sec latency 25.958 ms 2024-05-27T07:47:44.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.930+0000 7fede60e0640 1 -- 172.21.15.114:0/3840862818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede010a110 msgr2=0x7fede0103d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:44.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.930+0000 7fede60e0640 1 --2- 172.21.15.114:0/3840862818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede010a110 0x7fede0103d50 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009900 tx=0x7fedd402a410 comp rx=0 tx=0).stop 2024-05-27T07:47:44.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.930+0000 7fede60e0640 1 -- 172.21.15.114:0/3840862818 shutdown_connections 2024-05-27T07:47:44.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.930+0000 7fede60e0640 1 --2- 172.21.15.114:0/3840862818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fede01043c0 0x7fede0110790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.930+0000 7fede60e0640 1 --2- 172.21.15.114:0/3840862818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede010a110 0x7fede0103d50 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.930+0000 7fede60e0640 1 --2- 172.21.15.114:0/3840862818 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede0109c70 0x7fede0103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.930+0000 7fede60e0640 1 -- 172.21.15.114:0/3840862818 >> 172.21.15.114:0/3840862818 conn(0x7fede00ff350 msgr2=0x7fede01017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:44.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.930+0000 7fede60e0640 1 -- 172.21.15.114:0/3840862818 shutdown_connections 2024-05-27T07:47:44.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.930+0000 7fede60e0640 1 -- 172.21.15.114:0/3840862818 wait complete. 2024-05-27T07:47:44.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.931+0000 7fede60e0640 1 Processor -- start 2024-05-27T07:47:44.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.931+0000 7fede60e0640 1 -- start start 2024-05-27T07:47:44.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.931+0000 7fede60e0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede01043c0 0x7fede01a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.931+0000 7fede60e0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede0109c70 0x7fede01a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.931+0000 7fede60e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fede010a110 0x7fede019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.931+0000 7fede60e0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fede0113e60 con 0x7fede0109c70 2024-05-27T07:47:44.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.931+0000 7fede60e0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fede0113ce0 con 0x7fede01043c0 2024-05-27T07:47:44.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.931+0000 7fede60e0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fede0113fe0 con 0x7fede010a110 2024-05-27T07:47:44.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.931+0000 7feddeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede0109c70 0x7fede01a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.931+0000 7feddf7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede01043c0 0x7fede01a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.931+0000 7feddf7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede01043c0 0x7fede01a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:60092/0 (socket says 172.21.15.114:60092) 2024-05-27T07:47:44.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.931+0000 7feddeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede0109c70 0x7fede01a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46556/0 (socket says 172.21.15.114:46556) 2024-05-27T07:47:44.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.931+0000 7feddf7fe640 1 -- 172.21.15.114:0/205317988 learned_addr learned my addr 172.21.15.114:0/205317988 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:44.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.932+0000 7feddffff640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fede010a110 0x7fede019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.932+0000 7feddf7fe640 1 -- 172.21.15.114:0/205317988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fede010a110 msgr2=0x7fede019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:44.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.932+0000 7feddf7fe640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fede010a110 0x7fede019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.932+0000 7feddf7fe640 1 -- 172.21.15.114:0/205317988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede0109c70 msgr2=0x7fede01a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:44.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.932+0000 7feddf7fe640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede0109c70 0x7fede01a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.932+0000 7feddf7fe640 1 -- 172.21.15.114:0/205317988 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede019ae50 con 0x7fede01043c0 2024-05-27T07:47:44.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.932+0000 7feddeffd640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede0109c70 0x7fede01a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:44.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.932+0000 7feddf7fe640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede01043c0 0x7fede01a05a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fedd000d770 tx=0x7fedd000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:44.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.932+0000 7feddcff9640 1 -- 172.21.15.114:0/205317988 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0017070 con 0x7fede01043c0 2024-05-27T07:47:44.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.932+0000 7fede60e0640 1 -- 172.21.15.114:0/205317988 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede019b0e0 con 0x7fede01043c0 2024-05-27T07:47:44.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.932+0000 7feddcff9640 1 -- 172.21.15.114:0/205317988 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd0009c30 con 0x7fede01043c0 2024-05-27T07:47:44.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.932+0000 7feddcff9640 1 -- 172.21.15.114:0/205317988 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd000f620 con 0x7fede01043c0 2024-05-27T07:47:44.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.932+0000 7fede60e0640 1 -- 172.21.15.114:0/205317988 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede01a7340 con 0x7fede01043c0 2024-05-27T07:47:44.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7fedecf8a740 1 Processor -- start 2024-05-27T07:47:44.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7fedecf8a740 1 -- start start 2024-05-27T07:47:44.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7fedecf8a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb13e03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7fedecf8a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb10162f0 0x560fb13e0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7fedecf8a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb101e4d0 0x560fb13e0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7fedecf8a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560fb1352c30 con 0x560fb103f1b0 2024-05-27T07:47:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7fedecf8a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560fb13427d0 con 0x560fb101e4d0 2024-05-27T07:47:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7fedecf8a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560fb1346810 con 0x560fb10162f0 2024-05-27T07:47:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7feddcff9640 1 -- 172.21.15.114:0/205317988 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fedd001a070 con 0x7fede01043c0 2024-05-27T07:47:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7feddffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb13e03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7feddffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb13e03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46572/0 (socket says 172.21.15.114:46572) 2024-05-27T07:47:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7feddeffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb101e4d0 0x560fb13e0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7feddffff640 1 -- 172.21.15.114:0/1334556056 learned_addr learned my addr 172.21.15.114:0/1334556056 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.935+0000 7feddf7fe640 1 --2- 172.21.15.114:0/1334556056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb10162f0 0x560fb13e0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7feddffff640 1 -- 172.21.15.114:0/1334556056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb10162f0 msgr2=0x560fb13e0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7feddffff640 1 --2- 172.21.15.114:0/1334556056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb10162f0 0x560fb13e0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7feddffff640 1 -- 172.21.15.114:0/1334556056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb101e4d0 msgr2=0x560fb13e0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7feddffff640 1 --2- 172.21.15.114:0/1334556056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb101e4d0 0x560fb13e0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7feddffff640 1 -- 172.21.15.114:0/1334556056 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fb1057210 con 0x560fb103f1b0 2024-05-27T07:47:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7feddf7fe640 1 --2- 172.21.15.114:0/1334556056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb10162f0 0x560fb13e0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:47:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7feddeffd640 1 --2- 172.21.15.114:0/1334556056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb101e4d0 0x560fb13e0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7feddffff640 1 --2- 172.21.15.114:0/1334556056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb13e03d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009fd0 tx=0x7fedd80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c747b2d14b4a4f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7fedc5ffb640 1 -- 172.21.15.114:0/1334556056 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8004d20 con 0x560fb103f1b0 2024-05-27T07:47:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7feddcff9640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fedc80b2950 0x7fedc80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7fedc5ffb640 1 -- 172.21.15.114:0/1334556056 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd8004050 con 0x560fb103f1b0 2024-05-27T07:47:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7fedc5ffb640 1 -- 172.21.15.114:0/1334556056 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8005280 con 0x560fb103f1b0 2024-05-27T07:47:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7feddf7fe640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fedc80b2950 0x7fedc80b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.936+0000 7feddcff9640 1 -- 172.21.15.114:0/205317988 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fedd00e2400 con 0x7fede01043c0 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.937+0000 7fedecf8a740 1 -- 172.21.15.114:0/1334556056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 msgr2=0x560fb13e03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.937+0000 7feddf7fe640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fedc80b2950 0x7fedc80b4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fedd00df850 tx=0x7fedd0091040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.937+0000 7fedecf8a740 1 --2- 172.21.15.114:0/1334556056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb13e03d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009fd0 tx=0x7fedd80118d0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.937+0000 7fedecf8a740 1 -- 172.21.15.114:0/1334556056 shutdown_connections 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.937+0000 7fedecf8a740 1 --2- 172.21.15.114:0/1334556056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb101e4d0 0x560fb13e0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.937+0000 7fedecf8a740 1 --2- 172.21.15.114:0/1334556056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb10162f0 0x560fb13e0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.937+0000 7fedecf8a740 1 --2- 172.21.15.114:0/1334556056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb13e03d0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.937+0000 7fedecf8a740 1 -- 172.21.15.114:0/1334556056 >> 172.21.15.114:0/1334556056 conn(0x560fb1033300 msgr2=0x560fb13dcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.937+0000 7fedecf8a740 1 -- 172.21.15.114:0/1334556056 shutdown_connections 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.937+0000 7fedecf8a740 1 -- 172.21.15.114:0/1334556056 wait complete. 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7fedecf8a740 1 -- start start 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7fedecf8a740 1 Processor -- start 2024-05-27T07:47:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7fedecf8a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 0x560fb13ec4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7fedecf8a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb101e4d0 0x560fb13eca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7fedecf8a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb13ea940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7fedecf8a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560fb1028790 con 0x560fb103f1b0 2024-05-27T07:47:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7fedecf8a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560fb12fe890 con 0x560fb10162f0 2024-05-27T07:47:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7fedecf8a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560fb13ed500 con 0x560fb101e4d0 2024-05-27T07:47:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7fedecf8a740 10 client.0 _ll_register_callbacks cb 0x560fb1056a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7feddffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb13ea940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7feddeffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb101e4d0 0x560fb13eca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7feddffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb13ea940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46580/0 (socket says 172.21.15.114:46580) 2024-05-27T07:47:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7feddffff640 1 -- 172.21.15.114:0/316927526 learned_addr learned my addr 172.21.15.114:0/316927526 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.938+0000 7feddffff640 1 --2- 172.21.15.114:0/316927526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 0x560fb13ec4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7feddeffd640 1 -- 172.21.15.114:0/316927526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 msgr2=0x560fb13ec4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7feddeffd640 1 --2- 172.21.15.114:0/316927526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 0x560fb13ec4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7feddeffd640 1 -- 172.21.15.114:0/316927526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 msgr2=0x560fb13ea940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7feddeffd640 1 --2- 172.21.15.114:0/316927526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb13ea940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7feddeffd640 1 -- 172.21.15.114:0/316927526 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fb1123620 con 0x560fb101e4d0 2024-05-27T07:47:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7feddffff640 1 --2- 172.21.15.114:0/316927526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 0x560fb13ec4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:47:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7feddeffd640 1 --2- 172.21.15.114:0/316927526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb101e4d0 0x560fb13eca00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fedd402a930 tx=0x7fedd4031740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a90c7f3c42bb35ef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7fedbeffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:47:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7fedbeffd640 1 -- 172.21.15.114:0/316927526 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4035040 con 0x560fb101e4d0 2024-05-27T07:47:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7fedecf8a740 1 -- 172.21.15.114:0/316927526 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedd8004050 con 0x560fb101e4d0 2024-05-27T07:47:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7fedbeffd640 1 -- 172.21.15.114:0/316927526 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fedd4004240 con 0x560fb101e4d0 2024-05-27T07:47:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7fedbeffd640 1 -- 172.21.15.114:0/316927526 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fedd404f050 con 0x560fb101e4d0 2024-05-27T07:47:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7fedecf8a740 10 client.26893 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.939+0000 7fedecf8a740 1 -- 172.21.15.114:0/316927526 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x560fb13f98b0 con 0x560fb101e4d0 2024-05-27T07:47:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.940+0000 7fedbeffd640 1 -- 172.21.15.114:0/316927526 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd8004050 con 0x560fb101e4d0 2024-05-27T07:47:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.940+0000 7fedbeffd640 1 -- 172.21.15.114:0/316927526 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7fedd40049d0 con 0x560fb101e4d0 2024-05-27T07:47:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.940+0000 7fedecf8a740 10 client.26893 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.940+0000 7fedecf8a740 10 client.26893 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.940+0000 7fedecf8a740 20 client.26893 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.940+0000 7fedecf8a740 1 --2- 172.21.15.114:0/316927526 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560fb0ffdd80 0x560fb13fcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.940+0000 7fedecf8a740 4 client.26893 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.940+0000 7fedecf8a740 1 -- 172.21.15.114:0/316927526 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fb1354b80 con 0x560fb0ffdd80 2024-05-27T07:47:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.940+0000 7feddffff640 1 --2- 172.21.15.114:0/316927526 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560fb0ffdd80 0x560fb13fcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.940+0000 7fedbeffd640 10 client.26893 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.940+0000 7feddffff640 1 --2- 172.21.15.114:0/316927526 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560fb0ffdd80 0x560fb13fcd80 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c89ca7ea2759fc4 server_cookie=c1125b8034244988 in_seq=0 out_seq=0 2024-05-27T07:47:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedbeffd640 1 -- 172.21.15.114:0/316927526 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560fb1354b80 con 0x560fb0ffdd80 2024-05-27T07:47:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedbeffd640 10 client.26893 handle_command_reply: tid=0 2024-05-27T07:47:44.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedecf8a740 1 client.26893 shutdown 2024-05-27T07:47:44.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedecf8a740 1 -- 172.21.15.114:0/316927526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb101e4d0 msgr2=0x560fb13eca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:44.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedecf8a740 1 --2- 172.21.15.114:0/316927526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb101e4d0 0x560fb13eca00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fedd402a930 tx=0x7fedd4031740 comp rx=0 tx=0).stop 2024-05-27T07:47:44.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedecf8a740 1 -- 172.21.15.114:0/316927526 shutdown_connections 2024-05-27T07:47:44.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedecf8a740 1 --2- 172.21.15.114:0/316927526 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560fb0ffdd80 0x560fb13fcd80 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedecf8a740 1 --2- 172.21.15.114:0/316927526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb13ea940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedecf8a740 1 --2- 172.21.15.114:0/316927526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb101e4d0 0x560fb13eca00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedecf8a740 1 --2- 172.21.15.114:0/316927526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 0x560fb13ec4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedbeffd640 0 client.26893 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:44.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedecf8a740 1 -- 172.21.15.114:0/316927526 >> 172.21.15.114:0/316927526 conn(0x560fb1033300 msgr2=0x560fb13e0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:44.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedecf8a740 1 -- 172.21.15.114:0/316927526 shutdown_connections 2024-05-27T07:47:44.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedecf8a740 1 -- 172.21.15.114:0/316927526 wait complete. 2024-05-27T07:47:44.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.942+0000 7fedecf8a740 20 client.26893 trim_cache size 0 max 16384 2024-05-27T07:47:44.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.958+0000 7fedecf8a740 1 Processor -- start 2024-05-27T07:47:44.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.958+0000 7fedecf8a740 1 -- start start 2024-05-27T07:47:44.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7fedecf8a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 0x560fb13e4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7fedecf8a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb101e4d0 0x560fb13ec920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7fedecf8a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb103f1b0 0x560fb13ea1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7fedecf8a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fede01b14d0 con 0x560fb101e4d0 2024-05-27T07:47:44.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7fedecf8a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fede01b1ba0 con 0x560fb10162f0 2024-05-27T07:47:44.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7fedecf8a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fede01af320 con 0x560fb103f1b0 2024-05-27T07:47:44.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7feddeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb101e4d0 0x560fb13ec920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7feddffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb103f1b0 0x560fb13ea1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7feddeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb101e4d0 0x560fb13ec920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46596/0 (socket says 172.21.15.114:46596) 2024-05-27T07:47:44.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7feddeffd640 1 -- 172.21.15.114:0/1754988627 learned_addr learned my addr 172.21.15.114:0/1754988627 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:44.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7feddffff640 1 --2- 172.21.15.114:0/1754988627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 0x560fb13e4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7feddeffd640 1 -- 172.21.15.114:0/1754988627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb103f1b0 msgr2=0x560fb13ea1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:44.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7feddeffd640 1 --2- 172.21.15.114:0/1754988627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb103f1b0 0x560fb13ea1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7feddeffd640 1 -- 172.21.15.114:0/1754988627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 msgr2=0x560fb13e4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:44.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7feddeffd640 1 --2- 172.21.15.114:0/1754988627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 0x560fb13e4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7feddeffd640 1 -- 172.21.15.114:0/1754988627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fb13e98f0 con 0x560fb101e4d0 2024-05-27T07:47:44.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7feddffff640 1 --2- 172.21.15.114:0/1754988627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb103f1b0 0x560fb13ea1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:44.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7feddeffd640 1 --2- 172.21.15.114:0/1754988627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb101e4d0 0x560fb13ec920 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fedd4006010 tx=0x7fedd40046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e2648a44e5ac99 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.959+0000 7fedc57fa640 1 -- 172.21.15.114:0/1754988627 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4035030 con 0x560fb101e4d0 2024-05-27T07:47:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.960+0000 7fedc57fa640 1 -- 172.21.15.114:0/1754988627 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd4004fc0 con 0x560fb101e4d0 2024-05-27T07:47:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.960+0000 7fedc57fa640 1 -- 172.21.15.114:0/1754988627 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd40466b0 con 0x560fb101e4d0 2024-05-27T07:47:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.960+0000 7fedecf8a740 1 -- 172.21.15.114:0/1754988627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb101e4d0 msgr2=0x560fb13ec920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.960+0000 7fedecf8a740 1 --2- 172.21.15.114:0/1754988627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb101e4d0 0x560fb13ec920 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fedd4006010 tx=0x7fedd40046e0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.960+0000 7fedecf8a740 1 -- 172.21.15.114:0/1754988627 shutdown_connections 2024-05-27T07:47:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.960+0000 7fedecf8a740 1 --2- 172.21.15.114:0/1754988627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb103f1b0 0x560fb13ea1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.960+0000 7fedecf8a740 1 --2- 172.21.15.114:0/1754988627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb101e4d0 0x560fb13ec920 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.960+0000 7fedecf8a740 1 --2- 172.21.15.114:0/1754988627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 0x560fb13e4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.960+0000 7fedecf8a740 1 -- 172.21.15.114:0/1754988627 >> 172.21.15.114:0/1754988627 conn(0x560fb1033300 msgr2=0x560fb13e4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.961+0000 7fedecf8a740 1 -- 172.21.15.114:0/1754988627 shutdown_connections 2024-05-27T07:47:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.961+0000 7fedecf8a740 1 -- 172.21.15.114:0/1754988627 wait complete. 2024-05-27T07:47:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.961+0000 7fedecf8a740 1 -- start start 2024-05-27T07:47:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.961+0000 7fedecf8a740 1 Processor -- start 2024-05-27T07:47:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7fedecf8a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 0x560fb13ef8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7fedecf8a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb101e4d0 0x560fb13efe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7fedecf8a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb14633a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7fedecf8a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fede01af990 con 0x560fb103f1b0 2024-05-27T07:47:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7fedecf8a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fede01b0e00 con 0x560fb10162f0 2024-05-27T07:47:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7fedecf8a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fede01b0060 con 0x560fb101e4d0 2024-05-27T07:47:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7fedecf8a740 10 client.0 _ll_register_callbacks cb 0x7fedd4009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7feddffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb14633a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7feddeffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb101e4d0 0x560fb13efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7feddffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb14633a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46598/0 (socket says 172.21.15.114:46598) 2024-05-27T07:47:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7feddffff640 1 -- 172.21.15.114:0/2135890243 learned_addr learned my addr 172.21.15.114:0/2135890243 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7feddffff640 1 --2- 172.21.15.114:0/2135890243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 0x560fb13ef8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7feddffff640 1 -- 172.21.15.114:0/2135890243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb101e4d0 msgr2=0x560fb13efe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7feddffff640 1 --2- 172.21.15.114:0/2135890243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb101e4d0 0x560fb13efe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7feddffff640 1 -- 172.21.15.114:0/2135890243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 msgr2=0x560fb13ef8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7feddffff640 1 --2- 172.21.15.114:0/2135890243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 0x560fb13ef8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7feddffff640 1 -- 172.21.15.114:0/2135890243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fb13f0710 con 0x560fb103f1b0 2024-05-27T07:47:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7feddffff640 1 --2- 172.21.15.114:0/2135890243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb14633a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009bd0 tx=0x7fedd802a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9a9c93e2d2645e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7fedbeffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7fedbeffd640 1 -- 172.21.15.114:0/2135890243 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd80056c0 con 0x560fb103f1b0 2024-05-27T07:47:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7fedbeffd640 1 -- 172.21.15.114:0/2135890243 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fedd801f040 con 0x560fb103f1b0 2024-05-27T07:47:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.962+0000 7fedecf8a740 1 -- 172.21.15.114:0/2135890243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedd4004fc0 con 0x560fb103f1b0 2024-05-27T07:47:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.963+0000 7fedbeffd640 1 -- 172.21.15.114:0/2135890243 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fedd8005860 con 0x560fb103f1b0 2024-05-27T07:47:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.963+0000 7fedecf8a740 10 client.17328 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.963+0000 7fedecf8a740 1 -- 172.21.15.114:0/2135890243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x560fb1464980 con 0x560fb103f1b0 2024-05-27T07:47:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.963+0000 7fedbeffd640 1 -- 172.21.15.114:0/2135890243 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd4004fc0 con 0x560fb103f1b0 2024-05-27T07:47:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.963+0000 7fedbeffd640 1 -- 172.21.15.114:0/2135890243 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7fedd8027440 con 0x560fb103f1b0 2024-05-27T07:47:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.963+0000 7fedecf8a740 10 client.17328 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.963+0000 7fedecf8a740 10 client.17328 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.963+0000 7fedecf8a740 20 client.17328 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.963+0000 7fedecf8a740 1 --2- 172.21.15.114:0/2135890243 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560fb0fd31c0 0x560fb1466fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.963+0000 7fedecf8a740 4 client.17328 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:47:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.963+0000 7fedecf8a740 1 -- 172.21.15.114:0/2135890243 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560fb134e360 con 0x560fb0fd31c0 2024-05-27T07:47:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.963+0000 7feddeffd640 1 --2- 172.21.15.114:0/2135890243 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560fb0fd31c0 0x560fb1466fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:44.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.964+0000 7fedbeffd640 10 client.17328 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:44.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:44.964+0000 7feddeffd640 1 --2- 172.21.15.114:0/2135890243 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560fb0fd31c0 0x560fb1466fb0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2bc3f665ede3999 server_cookie=5c02c15601bc49d6 in_seq=0 out_seq=0 2024-05-27T07:47:45.273 INFO:tasks.workunit.client.0.smithi114.stdout: 1 811763 11.48 MB/sec execute 237 sec latency 196.276 ms 2024-05-27T07:47:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.296+0000 7fedbeffd640 1 -- 172.21.15.114:0/2135890243 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x560fb134e360 con 0x560fb0fd31c0 2024-05-27T07:47:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.296+0000 7fedbeffd640 10 client.17328 handle_command_reply: tid=0 2024-05-27T07:47:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.296+0000 7fedecf8a740 1 client.17328 shutdown 2024-05-27T07:47:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.297+0000 7fedecf8a740 1 -- 172.21.15.114:0/2135890243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 msgr2=0x560fb14633a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.297+0000 7fedecf8a740 1 --2- 172.21.15.114:0/2135890243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb14633a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009bd0 tx=0x7fedd802a040 comp rx=0 tx=0).stop 2024-05-27T07:47:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.298+0000 7fedecf8a740 1 -- 172.21.15.114:0/2135890243 shutdown_connections 2024-05-27T07:47:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.298+0000 7fedecf8a740 1 --2- 172.21.15.114:0/2135890243 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560fb0fd31c0 0x560fb1466fb0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.298+0000 7fedecf8a740 1 --2- 172.21.15.114:0/2135890243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560fb103f1b0 0x560fb14633a0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.298+0000 7fedecf8a740 1 --2- 172.21.15.114:0/2135890243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560fb101e4d0 0x560fb13efe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.298+0000 7fedecf8a740 1 --2- 172.21.15.114:0/2135890243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560fb10162f0 0x560fb13ef8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.298+0000 7fedbeffd640 0 client.17328 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.298+0000 7fedecf8a740 1 -- 172.21.15.114:0/2135890243 >> 172.21.15.114:0/2135890243 conn(0x560fb1033300 msgr2=0x560fb13ea1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:47:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:47:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:47:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.300+0000 7fedecf8a740 1 -- 172.21.15.114:0/2135890243 shutdown_connections 2024-05-27T07:47:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.300+0000 7fedecf8a740 1 -- 172.21.15.114:0/2135890243 wait complete. 2024-05-27T07:47:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.300+0000 7fedecf8a740 20 client.17328 trim_cache size 0 max 16384 2024-05-27T07:47:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.305+0000 7fede60e0640 1 -- 172.21.15.114:0/205317988 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fedc80b2950 msgr2=0x7fedc80b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.305+0000 7fede60e0640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fedc80b2950 0x7fedc80b4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fedd00df850 tx=0x7fedd0091040 comp rx=0 tx=0).stop 2024-05-27T07:47:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.305+0000 7fede60e0640 1 -- 172.21.15.114:0/205317988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede01043c0 msgr2=0x7fede01a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.305+0000 7fede60e0640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede01043c0 0x7fede01a05a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fedd000d770 tx=0x7fedd000dc30 comp rx=0 tx=0).stop 2024-05-27T07:47:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.305+0000 7fede60e0640 1 -- 172.21.15.114:0/205317988 shutdown_connections 2024-05-27T07:47:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.305+0000 7fede60e0640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fedc80b2950 0x7fedc80b4e60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.305+0000 7fede60e0640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fede010a110 0x7fede019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.305+0000 7fede60e0640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede0109c70 0x7fede01a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.305+0000 7fede60e0640 1 --2- 172.21.15.114:0/205317988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede01043c0 0x7fede01a05a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.305+0000 7fede60e0640 1 -- 172.21.15.114:0/205317988 >> 172.21.15.114:0/205317988 conn(0x7fede00ff350 msgr2=0x7fede010f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.306+0000 7fede60e0640 1 -- 172.21.15.114:0/205317988 shutdown_connections 2024-05-27T07:47:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:45.306+0000 7fede60e0640 1 -- 172.21.15.114:0/205317988 wait complete. 2024-05-27T07:47:45.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:45 smithi114 ceph-mon[32038]: pgmap v258: 129 pgs: 129 active+clean; 75 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 305 op/s 2024-05-27T07:47:45.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:45 smithi163 ceph-mon[36451]: pgmap v258: 129 pgs: 129 active+clean; 75 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 305 op/s 2024-05-27T07:47:45.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:45 smithi186 ceph-mon[36647]: pgmap v258: 129 pgs: 129 active+clean; 75 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 305 op/s 2024-05-27T07:47:45.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.821+0000 7f2e1c19a700 1 -- 172.21.15.114:0/125336422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14107e10 msgr2=0x7f2e14108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:45.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.821+0000 7f2e1c19a700 1 --2- 172.21.15.114:0/125336422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14107e10 0x7f2e14108220 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f2e040099c0 tx=0x7f2e04009cd0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.821+0000 7f2e1c19a700 1 -- 172.21.15.114:0/125336422 shutdown_connections 2024-05-27T07:47:45.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.821+0000 7f2e1c19a700 1 --2- 172.21.15.114:0/125336422 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e14108760 0x7f2e141100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.821+0000 7f2e1c19a700 1 --2- 172.21.15.114:0/125336422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e14103c70 0x7f2e14104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.821+0000 7f2e1c19a700 1 --2- 172.21.15.114:0/125336422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14107e10 0x7f2e14108220 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.821+0000 7f2e1c19a700 1 -- 172.21.15.114:0/125336422 >> 172.21.15.114:0/125336422 conn(0x7f2e140fe6e0 msgr2=0x7f2e14100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:45.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.821+0000 7f2e1c19a700 1 -- 172.21.15.114:0/125336422 shutdown_connections 2024-05-27T07:47:45.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.821+0000 7f2e1c19a700 1 -- 172.21.15.114:0/125336422 wait complete. 2024-05-27T07:47:45.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.822+0000 7f2e1c19a700 1 Processor -- start 2024-05-27T07:47:45.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.822+0000 7f2e1c19a700 1 -- start start 2024-05-27T07:47:45.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.822+0000 7f2e1c19a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e14103c70 0x7f2e1419bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:45.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.822+0000 7f2e1c19a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e14107e10 0x7f2e1419c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:45.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.822+0000 7f2e1c19a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14108760 0x7f2e1419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:45.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.822+0000 7f2e1c19a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2e14113380 con 0x7f2e14108760 2024-05-27T07:47:45.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.822+0000 7f2e1c19a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2e14113200 con 0x7f2e14103c70 2024-05-27T07:47:45.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.822+0000 7f2e1c19a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2e14113500 con 0x7f2e14107e10 2024-05-27T07:47:45.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e1a737700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14108760 0x7f2e1419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:45.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e19735700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e14107e10 0x7f2e1419c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:45.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e1a737700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14108760 0x7f2e1419c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46622/0 (socket says 172.21.15.114:46622) 2024-05-27T07:47:45.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e1a737700 1 -- 172.21.15.114:0/4234900961 learned_addr learned my addr 172.21.15.114:0/4234900961 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:45.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e19f36700 1 --2- 172.21.15.114:0/4234900961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e14103c70 0x7f2e1419bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:45.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e19735700 1 -- 172.21.15.114:0/4234900961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e14103c70 msgr2=0x7f2e1419bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:45.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e19735700 1 --2- 172.21.15.114:0/4234900961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e14103c70 0x7f2e1419bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e19735700 1 -- 172.21.15.114:0/4234900961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14108760 msgr2=0x7f2e1419c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:45.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e19735700 1 --2- 172.21.15.114:0/4234900961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14108760 0x7f2e1419c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:45.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e19735700 1 -- 172.21.15.114:0/4234900961 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e141a7030 con 0x7f2e14107e10 2024-05-27T07:47:45.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e19735700 1 --2- 172.21.15.114:0/4234900961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e14107e10 0x7f2e1419c200 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e040098e0 tx=0x7f2e040045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:45.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e0affd700 1 -- 172.21.15.114:0/4234900961 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e04026070 con 0x7f2e14107e10 2024-05-27T07:47:45.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e1c19a700 1 -- 172.21.15.114:0/4234900961 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e141a72c0 con 0x7f2e14107e10 2024-05-27T07:47:45.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e0affd700 1 -- 172.21.15.114:0/4234900961 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e0400f460 con 0x7f2e14107e10 2024-05-27T07:47:45.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e0affd700 1 -- 172.21.15.114:0/4234900961 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e04009e60 con 0x7f2e14107e10 2024-05-27T07:47:45.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.823+0000 7f2e1c19a700 1 -- 172.21.15.114:0/4234900961 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e141a7770 con 0x7f2e14107e10 2024-05-27T07:47:45.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.824+0000 7f2e1c19a700 1 -- 172.21.15.114:0/4234900961 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e140512b0 con 0x7f2e14107e10 2024-05-27T07:47:45.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.826+0000 7f2e0affd700 1 -- 172.21.15.114:0/4234900961 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f2e04029070 con 0x7f2e14107e10 2024-05-27T07:47:45.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.827+0000 7f2e0affd700 1 --2- 172.21.15.114:0/4234900961 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2e000b29f0 0x7f2e000b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:45.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.827+0000 7f2e19f36700 1 --2- 172.21.15.114:0/4234900961 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2e000b29f0 0x7f2e000b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:45.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.827+0000 7f2e0affd700 1 -- 172.21.15.114:0/4234900961 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f2e040e3480 con 0x7f2e14107e10 2024-05-27T07:47:45.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.827+0000 7f2e19f36700 1 --2- 172.21.15.114:0/4234900961 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2e000b29f0 0x7f2e000b4f00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2e1406b0a0 tx=0x7f2e10009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:45.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:45.830+0000 7f2e0affd700 1 -- 172.21.15.114:0/4234900961 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2e04013070 con 0x7f2e14107e10 2024-05-27T07:47:46.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.111+0000 7f2e1c19a700 1 -- 172.21.15.114:0/4234900961 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e1402e5e0 con 0x7f2e14107e10 2024-05-27T07:47:46.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.111+0000 7f2e0affd700 1 -- 172.21.15.114:0/4234900961 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+10174 (secure 0 0 0) 0x7f2e040aab40 con 0x7f2e14107e10 2024-05-27T07:47:46.112 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:47:46.112 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/869688838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":869688838},{"type":"v1","addr":"172.21.15.163:6837","nonce":869688838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":46,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:47:34.266249+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26227":{"gid":26227,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/2936519309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2936519309},{"type":"v1","addr":"172.21.15.186:6835","nonce":2936519309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:47:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.114+0000 7f2e1c19a700 1 -- 172.21.15.114:0/4234900961 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2e000b29f0 msgr2=0x7f2e000b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.114+0000 7f2e1c19a700 1 --2- 172.21.15.114:0/4234900961 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2e000b29f0 0x7f2e000b4f00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2e1406b0a0 tx=0x7f2e10009260 comp rx=0 tx=0).stop 2024-05-27T07:47:46.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.114+0000 7f2e1c19a700 1 -- 172.21.15.114:0/4234900961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e14107e10 msgr2=0x7f2e1419c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:46.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.114+0000 7f2e1c19a700 1 --2- 172.21.15.114:0/4234900961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e14107e10 0x7f2e1419c200 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e040098e0 tx=0x7f2e040045e0 comp rx=0 tx=0).stop 2024-05-27T07:47:46.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.114+0000 7f2e1c19a700 1 -- 172.21.15.114:0/4234900961 shutdown_connections 2024-05-27T07:47:46.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.114+0000 7f2e1c19a700 1 --2- 172.21.15.114:0/4234900961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14108760 0x7f2e1419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:46.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.114+0000 7f2e1c19a700 1 --2- 172.21.15.114:0/4234900961 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2e000b29f0 0x7f2e000b4f00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:46.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.114+0000 7f2e1c19a700 1 --2- 172.21.15.114:0/4234900961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e14103c70 0x7f2e1419bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:46.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.114+0000 7f2e1c19a700 1 --2- 172.21.15.114:0/4234900961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e14107e10 0x7f2e1419c200 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:46.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.114+0000 7f2e1c19a700 1 -- 172.21.15.114:0/4234900961 >> 172.21.15.114:0/4234900961 conn(0x7f2e140fe6e0 msgr2=0x7f2e14100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:46.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.114+0000 7f2e1c19a700 1 -- 172.21.15.114:0/4234900961 shutdown_connections 2024-05-27T07:47:46.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:46.114+0000 7f2e1c19a700 1 -- 172.21.15.114:0/4234900961 wait complete. 2024-05-27T07:47:46.117 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 46 2024-05-27T07:47:46.273 INFO:tasks.workunit.client.0.smithi114.stdout: 1 813638 11.49 MB/sec execute 238 sec latency 37.910 ms 2024-05-27T07:47:46.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:46 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/4234900961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:47:46.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:46 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/4234900961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:47:46.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:46 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/4234900961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:47:46.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:47:47.239 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:47:47.273 INFO:tasks.workunit.client.0.smithi114.stdout: 1 815433 11.47 MB/sec execute 239 sec latency 159.785 ms 2024-05-27T07:47:47.610 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:47 smithi114 ceph-mon[32038]: pgmap v259: 129 pgs: 129 active+clean; 74 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 516 op/s 2024-05-27T07:47:47.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:47 smithi163 ceph-mon[36451]: pgmap v259: 129 pgs: 129 active+clean; 74 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 516 op/s 2024-05-27T07:47:47.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:47 smithi186 ceph-mon[36647]: pgmap v259: 129 pgs: 129 active+clean; 74 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 516 op/s 2024-05-27T07:47:48.273 INFO:tasks.workunit.client.0.smithi114.stdout: 1 817260 11.47 MB/sec execute 240 sec latency 42.027 ms 2024-05-27T07:47:49.273 INFO:tasks.workunit.client.0.smithi114.stdout: 1 819533 11.48 MB/sec execute 241 sec latency 32.405 ms 2024-05-27T07:47:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.571+0000 7f651aef0700 1 -- 172.21.15.114:0/426853339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6514107de0 msgr2=0x7f65141081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.571+0000 7f651aef0700 1 --2- 172.21.15.114:0/426853339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6514107de0 0x7f65141081f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6500009920 tx=0x7f6500009c30 comp rx=0 tx=0).stop 2024-05-27T07:47:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.572+0000 7f651aef0700 1 -- 172.21.15.114:0/426853339 shutdown_connections 2024-05-27T07:47:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.572+0000 7f651aef0700 1 --2- 172.21.15.114:0/426853339 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6514108730 0x7f6514110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.572+0000 7f651aef0700 1 --2- 172.21.15.114:0/426853339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65141034b0 0x7f65141038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.572+0000 7f651aef0700 1 --2- 172.21.15.114:0/426853339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6514107de0 0x7f65141081f0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.572+0000 7f651aef0700 1 -- 172.21.15.114:0/426853339 >> 172.21.15.114:0/426853339 conn(0x7f65140fedb0 msgr2=0x7f65141011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:49.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.572+0000 7f651aef0700 1 -- 172.21.15.114:0/426853339 shutdown_connections 2024-05-27T07:47:49.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.572+0000 7f651aef0700 1 -- 172.21.15.114:0/426853339 wait complete. 2024-05-27T07:47:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.572+0000 7f651aef0700 1 Processor -- start 2024-05-27T07:47:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.572+0000 7f651aef0700 1 -- start start 2024-05-27T07:47:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f651aef0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65141034b0 0x7f651419ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f651aef0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6514107de0 0x7f651419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f651aef0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6514108730 0x7f651419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f651aef0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6514113340 con 0x7f6514108730 2024-05-27T07:47:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f651aef0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f65141131c0 con 0x7f65141034b0 2024-05-27T07:47:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f651aef0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f65141134c0 con 0x7f6514107de0 2024-05-27T07:47:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f6518c8c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65141034b0 0x7f651419ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f651948d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6514108730 0x7f651419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f6518c8c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65141034b0 0x7f651419ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:60174/0 (socket says 172.21.15.114:60174) 2024-05-27T07:47:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f6518c8c700 1 -- 172.21.15.114:0/4272451845 learned_addr learned my addr 172.21.15.114:0/4272451845 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f6513fff700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6514107de0 0x7f651419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f6518c8c700 1 -- 172.21.15.114:0/4272451845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6514107de0 msgr2=0x7f651419b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f6518c8c700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6514107de0 0x7f651419b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f6518c8c700 1 -- 172.21.15.114:0/4272451845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6514108730 msgr2=0x7f651419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f6518c8c700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6514108730 0x7f651419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f6518c8c700 1 -- 172.21.15.114:0/4272451845 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651419fd30 con 0x7f65141034b0 2024-05-27T07:47:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f651948d700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6514108730 0x7f651419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.573+0000 7f6518c8c700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65141034b0 0x7f651419ae60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f650800e9a0 tx=0x7f650800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.574+0000 7f6511ffb700 1 -- 172.21.15.114:0/4272451845 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650800ca10 con 0x7f65141034b0 2024-05-27T07:47:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.574+0000 7f6511ffb700 1 -- 172.21.15.114:0/4272451845 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f650800cbb0 con 0x7f65141034b0 2024-05-27T07:47:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.574+0000 7f651aef0700 1 -- 172.21.15.114:0/4272451845 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65141a0020 con 0x7f65141034b0 2024-05-27T07:47:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.574+0000 7f6511ffb700 1 -- 172.21.15.114:0/4272451845 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6508018680 con 0x7f65141034b0 2024-05-27T07:47:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.574+0000 7f651aef0700 1 -- 172.21.15.114:0/4272451845 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65141a7940 con 0x7f65141034b0 2024-05-27T07:47:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.576+0000 7f6511ffb700 1 -- 172.21.15.114:0/4272451845 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6508018820 con 0x7f65141034b0 2024-05-27T07:47:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.577+0000 7f6511ffb700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f65040b29f0 0x7f65040b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.578+0000 7f6513fff700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f65040b29f0 0x7f65040b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.578+0000 7f6511ffb700 1 -- 172.21.15.114:0/4272451845 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f65080e2c60 con 0x7f65141034b0 2024-05-27T07:47:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.578+0000 7f6513fff700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f65040b29f0 0x7f65040b4f00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6500009810 tx=0x7f6500019040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.578+0000 7f6529905b80 1 Processor -- start 2024-05-27T07:47:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.578+0000 7f6529905b80 1 -- start start 2024-05-27T07:47:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.578+0000 7f6529905b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669581e40 0x55c669705710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f6529905b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669705c50 0x55c6696ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f6529905b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 0x55c669700390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f6529905b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c66962ec90 con 0x55c6696fff80 2024-05-27T07:47:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f6529905b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c66961e830 con 0x55c669705c50 2024-05-27T07:47:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f6529905b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c669622870 con 0x55c669581e40 2024-05-27T07:47:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f651948d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 0x55c669700390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f6518c8c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669581e40 0x55c669705710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f651948d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 0x55c669700390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46658/0 (socket says 172.21.15.114:46658) 2024-05-27T07:47:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f651948d700 1 -- 172.21.15.114:0/2696878813 learned_addr learned my addr 172.21.15.114:0/2696878813 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f6513fff700 1 --2- 172.21.15.114:0/2696878813 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669705c50 0x55c6696ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f651948d700 1 -- 172.21.15.114:0/2696878813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669581e40 msgr2=0x55c669705710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f651948d700 1 --2- 172.21.15.114:0/2696878813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669581e40 0x55c669705710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f651948d700 1 -- 172.21.15.114:0/2696878813 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669705c50 msgr2=0x55c6696ffa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f651948d700 1 --2- 172.21.15.114:0/2696878813 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669705c50 0x55c6696ffa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f651948d700 1 -- 172.21.15.114:0/2696878813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c66956e8d0 con 0x55c6696fff80 2024-05-27T07:47:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f651948d700 1 --2- 172.21.15.114:0/2696878813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 0x55c669700390 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f650c003eb0 tx=0x7f650c0099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792082ca26553287 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f64feffd700 1 -- 172.21.15.114:0/2696878813 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c019020 con 0x55c6696fff80 2024-05-27T07:47:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f64feffd700 1 -- 172.21.15.114:0/2696878813 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f650c0092d0 con 0x55c6696fff80 2024-05-27T07:47:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.579+0000 7f64feffd700 1 -- 172.21.15.114:0/2696878813 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c00e750 con 0x55c6696fff80 2024-05-27T07:47:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.580+0000 7f6529905b80 1 -- 172.21.15.114:0/2696878813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 msgr2=0x55c669700390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.580+0000 7f6529905b80 1 --2- 172.21.15.114:0/2696878813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 0x55c669700390 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f650c003eb0 tx=0x7f650c0099e0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.580+0000 7f6529905b80 1 -- 172.21.15.114:0/2696878813 shutdown_connections 2024-05-27T07:47:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.580+0000 7f6529905b80 1 --2- 172.21.15.114:0/2696878813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 0x55c669700390 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.580+0000 7f6529905b80 1 --2- 172.21.15.114:0/2696878813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669581e40 0x55c669705710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.580+0000 7f6529905b80 1 --2- 172.21.15.114:0/2696878813 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669705c50 0x55c6696ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.580+0000 7f6529905b80 1 -- 172.21.15.114:0/2696878813 >> 172.21.15.114:0/2696878813 conn(0x55c6695dcaf0 msgr2=0x55c6696fa640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.580+0000 7f6529905b80 1 -- 172.21.15.114:0/2696878813 shutdown_connections 2024-05-27T07:47:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.580+0000 7f6529905b80 1 -- 172.21.15.114:0/2696878813 wait complete. 2024-05-27T07:47:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.581+0000 7f6529905b80 1 -- start start 2024-05-27T07:47:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.581+0000 7f6529905b80 1 Processor -- start 2024-05-27T07:47:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.581+0000 7f6529905b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669581e40 0x55c669709300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.581+0000 7f6529905b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 0x55c669709840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.581+0000 7f6529905b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669705c50 0x55c669709d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.581+0000 7f6529905b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c669524fa0 con 0x55c6696fff80 2024-05-27T07:47:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.581+0000 7f6529905b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c6695c07c0 con 0x55c669581e40 2024-05-27T07:47:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f6513fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 0x55c669709840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f6529905b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c66970c740 con 0x55c669705c50 2024-05-27T07:47:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f6513fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 0x55c669709840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46668/0 (socket says 172.21.15.114:46668) 2024-05-27T07:47:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f6513fff700 1 -- 172.21.15.114:0/1946936863 learned_addr learned my addr 172.21.15.114:0/1946936863 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f6529905b80 10 client.0 _ll_register_callbacks cb 0x55c6695b9920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:49.584 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:49 smithi114 ceph-mon[32038]: pgmap v260: 129 pgs: 129 active+clean; 74 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 290 op/s 2024-05-27T07:47:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f651948d700 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669705c50 0x55c669709d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f6518c8c700 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669581e40 0x55c669709300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f6513fff700 1 -- 172.21.15.114:0/1946936863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669705c50 msgr2=0x55c669709d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f6513fff700 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669705c50 0x55c669709d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f6513fff700 1 -- 172.21.15.114:0/1946936863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669581e40 msgr2=0x55c669709300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f6513fff700 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669581e40 0x55c669709300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f6513fff700 1 -- 172.21.15.114:0/1946936863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6695bd690 con 0x55c6696fff80 2024-05-27T07:47:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f6518c8c700 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669581e40 0x55c669709300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:47:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f6513fff700 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 0x55c669709840 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f6514076aa0 tx=0x7f6500009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf67eb178194865 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.582+0000 7f64f3fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.583+0000 7f64f3fff700 1 -- 172.21.15.114:0/1946936863 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6500005600 con 0x55c6696fff80 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.583+0000 7f6529905b80 1 -- 172.21.15.114:0/1946936863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f650c0092d0 con 0x55c6696fff80 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.583+0000 7f64f3fff700 1 -- 172.21.15.114:0/1946936863 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f6500003680 con 0x55c6696fff80 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.583+0000 7f64f3fff700 1 -- 172.21.15.114:0/1946936863 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f650003d050 con 0x55c6696fff80 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.583+0000 7f6529905b80 10 client.17346 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.583+0000 7f6529905b80 1 -- 172.21.15.114:0/1946936863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55c6697171b0 con 0x55c6696fff80 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.584+0000 7f64f3fff700 1 -- 172.21.15.114:0/1946936863 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f650c0092d0 con 0x55c6696fff80 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.584+0000 7f64f3fff700 1 -- 172.21.15.114:0/1946936863 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7f650000ba10 con 0x55c6696fff80 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.584+0000 7f6529905b80 10 client.17346 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.584+0000 7f6529905b80 10 client.17346 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.584+0000 7f6529905b80 20 client.17346 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.584+0000 7f6529905b80 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c669719890 0x55c66971bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.584+0000 7f6529905b80 4 client.17346 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.584+0000 7f6529905b80 1 -- 172.21.15.114:0/1946936863 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6696cb9f0 con 0x55c669719890 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.584+0000 7f6518c8c700 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c669719890 0x55c66971bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.585+0000 7f64f3fff700 10 client.17346 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.585+0000 7f6518c8c700 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c669719890 0x55c66971bcd0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7dbceb7805460ea server_cookie=d3ab1cc3bf320644 in_seq=0 out_seq=0 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.586+0000 7f64f3fff700 1 -- 172.21.15.114:0/1946936863 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c6696cb9f0 con 0x55c669719890 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.586+0000 7f64f3fff700 10 client.17346 handle_command_reply: tid=0 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.586+0000 7f6529905b80 1 client.17346 shutdown 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.586+0000 7f6529905b80 1 -- 172.21.15.114:0/1946936863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 msgr2=0x55c669709840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.586+0000 7f6529905b80 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 0x55c669709840 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f6514076aa0 tx=0x7f6500009c30 comp rx=0 tx=0).stop 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.586+0000 7f6529905b80 1 -- 172.21.15.114:0/1946936863 shutdown_connections 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.586+0000 7f6529905b80 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669705c50 0x55c669709d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.586+0000 7f6529905b80 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c669719890 0x55c66971bcd0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.586+0000 7f6529905b80 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669581e40 0x55c669709300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.586+0000 7f6529905b80 1 --2- 172.21.15.114:0/1946936863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c6696fff80 0x55c669709840 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.586+0000 7f6529905b80 1 -- 172.21.15.114:0/1946936863 >> 172.21.15.114:0/1946936863 conn(0x55c6695dcaf0 msgr2=0x55c6696fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:49.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.586+0000 7f64f3fff700 0 client.17346 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:49.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.587+0000 7f6529905b80 1 -- 172.21.15.114:0/1946936863 shutdown_connections 2024-05-27T07:47:49.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.587+0000 7f6529905b80 1 -- 172.21.15.114:0/1946936863 wait complete. 2024-05-27T07:47:49.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.587+0000 7f6529905b80 20 client.17346 trim_cache size 0 max 16384 2024-05-27T07:47:49.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.612+0000 7f6529905b80 1 Processor -- start 2024-05-27T07:47:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.612+0000 7f6529905b80 1 -- start start 2024-05-27T07:47:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6529905b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669581e40 0x55c6696ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6529905b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669705c50 0x55c6697157e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6529905b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669707070 0x55c669703010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6529905b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65141b0710 con 0x55c669705c50 2024-05-27T07:47:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6529905b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f65141b0de0 con 0x55c669707070 2024-05-27T07:47:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6529905b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f65141ae560 con 0x55c669581e40 2024-05-27T07:47:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6513fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669705c50 0x55c6697157e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6513fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669705c50 0x55c6697157e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46678/0 (socket says 172.21.15.114:46678) 2024-05-27T07:47:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6513fff700 1 -- 172.21.15.114:0/2323636915 learned_addr learned my addr 172.21.15.114:0/2323636915 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f651948d700 1 --2- 172.21.15.114:0/2323636915 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669707070 0x55c669703010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6518c8c700 1 --2- 172.21.15.114:0/2323636915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669581e40 0x55c6696ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6513fff700 1 -- 172.21.15.114:0/2323636915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669581e40 msgr2=0x55c6696ffa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6513fff700 1 --2- 172.21.15.114:0/2323636915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669581e40 0x55c6696ffa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6513fff700 1 -- 172.21.15.114:0/2323636915 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669707070 msgr2=0x55c669703010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6513fff700 1 --2- 172.21.15.114:0/2323636915 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669707070 0x55c669703010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6513fff700 1 -- 172.21.15.114:0/2323636915 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c669708620 con 0x55c669705c50 2024-05-27T07:47:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.613+0000 7f6513fff700 1 --2- 172.21.15.114:0/2323636915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669705c50 0x55c6697157e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f6500009680 tx=0x7f65000050f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f03ee673a2dab4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.614+0000 7f64fe7fc700 1 -- 172.21.15.114:0/2323636915 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6500045020 con 0x55c669705c50 2024-05-27T07:47:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.614+0000 7f64fe7fc700 1 -- 172.21.15.114:0/2323636915 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6500034070 con 0x55c669705c50 2024-05-27T07:47:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.614+0000 7f64fe7fc700 1 -- 172.21.15.114:0/2323636915 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6500023710 con 0x55c669705c50 2024-05-27T07:47:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.614+0000 7f6529905b80 1 -- 172.21.15.114:0/2323636915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669705c50 msgr2=0x55c6697157e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.614+0000 7f6529905b80 1 --2- 172.21.15.114:0/2323636915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669705c50 0x55c6697157e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f6500009680 tx=0x7f65000050f0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.615+0000 7f6529905b80 1 -- 172.21.15.114:0/2323636915 shutdown_connections 2024-05-27T07:47:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.615+0000 7f6529905b80 1 --2- 172.21.15.114:0/2323636915 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669707070 0x55c669703010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.615+0000 7f6529905b80 1 --2- 172.21.15.114:0/2323636915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669581e40 0x55c6696ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.615+0000 7f6529905b80 1 --2- 172.21.15.114:0/2323636915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669705c50 0x55c6697157e0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.615+0000 7f6529905b80 1 -- 172.21.15.114:0/2323636915 >> 172.21.15.114:0/2323636915 conn(0x55c6695dcaf0 msgr2=0x55c6696fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.615+0000 7f6529905b80 1 -- 172.21.15.114:0/2323636915 shutdown_connections 2024-05-27T07:47:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.615+0000 7f6529905b80 1 -- 172.21.15.114:0/2323636915 wait complete. 2024-05-27T07:47:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.615+0000 7f6529905b80 1 -- start start 2024-05-27T07:47:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.615+0000 7f6529905b80 1 Processor -- start 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6529905b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669581e40 0x55c66977f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6529905b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669705c50 0x55c66977fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6529905b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669707070 0x55c6697119d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6529905b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65141aebd0 con 0x55c669581e40 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6529905b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f65141b0040 con 0x55c669705c50 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6518c8c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669581e40 0x55c66977f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6529905b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f65141af2a0 con 0x55c669707070 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6518c8c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669581e40 0x55c66977f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46694/0 (socket says 172.21.15.114:46694) 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6518c8c700 1 -- 172.21.15.114:0/1625391829 learned_addr learned my addr 172.21.15.114:0/1625391829 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6513fff700 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669705c50 0x55c66977fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f651948d700 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669707070 0x55c6697119d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6529905b80 10 client.0 _ll_register_callbacks cb 0x55c6695b9920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6513fff700 1 -- 172.21.15.114:0/1625391829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669707070 msgr2=0x55c6697119d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6513fff700 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669707070 0x55c6697119d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6513fff700 1 -- 172.21.15.114:0/1625391829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669581e40 msgr2=0x55c66977f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6513fff700 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669581e40 0x55c66977f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6513fff700 1 -- 172.21.15.114:0/1625391829 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c669711fc0 con 0x55c669705c50 2024-05-27T07:47:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6518c8c700 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669581e40 0x55c66977f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f6513fff700 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669705c50 0x55c66977fb10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f65000051c0 tx=0x7f65000056b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9da7fee68aa0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.616+0000 7f64f3fff700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:47:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f64f3fff700 1 -- 172.21.15.114:0/1625391829 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6500045030 con 0x55c669705c50 2024-05-27T07:47:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f64f3fff700 1 -- 172.21.15.114:0/1625391829 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f650000b9f0 con 0x55c669705c50 2024-05-27T07:47:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f6529905b80 1 -- 172.21.15.114:0/1625391829 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6500034070 con 0x55c669705c50 2024-05-27T07:47:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f64f3fff700 1 -- 172.21.15.114:0/1625391829 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f6500057050 con 0x55c669705c50 2024-05-27T07:47:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f6529905b80 10 client.26813 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f6529905b80 1 -- 172.21.15.114:0/1625391829 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55c6697212c0 con 0x55c669705c50 2024-05-27T07:47:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f64f3fff700 1 -- 172.21.15.114:0/1625391829 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6500034070 con 0x55c669705c50 2024-05-27T07:47:49.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f64f3fff700 1 -- 172.21.15.114:0/1625391829 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7f6500052030 con 0x55c669705c50 2024-05-27T07:47:49.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f6529905b80 10 client.26813 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:49.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f6529905b80 10 client.26813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:49.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f6529905b80 20 client.26813 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:49.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f6529905b80 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c669788800 0x55c66978ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:49.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f6529905b80 4 client.26813 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:47:49.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f6529905b80 1 -- 172.21.15.114:0/1625391829 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c669630be0 con 0x55c669788800 2024-05-27T07:47:49.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.617+0000 7f6518c8c700 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c669788800 0x55c66978ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:49.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.618+0000 7f64f3fff700 10 client.26813 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:49.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.618+0000 7f6518c8c700 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c669788800 0x55c66978ac20 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b47c9fb8f1c49ccd server_cookie=3701846f2d361274 in_seq=0 out_seq=0 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f64f3fff700 1 -- 172.21.15.114:0/1625391829 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c669630be0 con 0x55c669788800 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f64f3fff700 10 client.26813 handle_command_reply: tid=0 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f6529905b80 1 client.26813 shutdown 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f6529905b80 1 -- 172.21.15.114:0/1625391829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669705c50 msgr2=0x55c66977fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f6529905b80 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669705c50 0x55c66977fb10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f65000051c0 tx=0x7f65000056b0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f6529905b80 1 -- 172.21.15.114:0/1625391829 shutdown_connections 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f6529905b80 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c669707070 0x55c6697119d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f6529905b80 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c669581e40 0x55c66977f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f6529905b80 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c669788800 0x55c66978ac20 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f64f3fff700 0 client.26813 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f6529905b80 1 --2- 172.21.15.114:0/1625391829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c669705c50 0x55c66977fb10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f6529905b80 1 -- 172.21.15.114:0/1625391829 >> 172.21.15.114:0/1625391829 conn(0x55c6695dcaf0 msgr2=0x55c6696fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f6529905b80 1 -- 172.21.15.114:0/1625391829 shutdown_connections 2024-05-27T07:47:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f6529905b80 1 -- 172.21.15.114:0/1625391829 wait complete. 2024-05-27T07:47:49.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.619+0000 7f6529905b80 20 client.26813 trim_cache size 0 max 16384 2024-05-27T07:47:49.625 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:47:49.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.620+0000 7f651aef0700 1 -- 172.21.15.114:0/4272451845 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f65040b29f0 msgr2=0x7f65040b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.620+0000 7f651aef0700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f65040b29f0 0x7f65040b4f00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6500009810 tx=0x7f6500019040 comp rx=0 tx=0).stop 2024-05-27T07:47:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.620+0000 7f651aef0700 1 -- 172.21.15.114:0/4272451845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65141034b0 msgr2=0x7f651419ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.620+0000 7f651aef0700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65141034b0 0x7f651419ae60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f650800e9a0 tx=0x7f650800ecb0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.620+0000 7f651aef0700 1 -- 172.21.15.114:0/4272451845 shutdown_connections 2024-05-27T07:47:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.620+0000 7f651aef0700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6514108730 0x7f651419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.620+0000 7f651aef0700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f65040b29f0 0x7f65040b4f00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.620+0000 7f651aef0700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65141034b0 0x7f651419ae60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.620+0000 7f651aef0700 1 --2- 172.21.15.114:0/4272451845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6514107de0 0x7f651419b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.620+0000 7f651aef0700 1 -- 172.21.15.114:0/4272451845 >> 172.21.15.114:0/4272451845 conn(0x7f65140fedb0 msgr2=0x7f651406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.620+0000 7f651aef0700 1 -- 172.21.15.114:0/4272451845 shutdown_connections 2024-05-27T07:47:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:49.620+0000 7f651aef0700 1 -- 172.21.15.114:0/4272451845 wait complete. 2024-05-27T07:47:49.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:49 smithi163 ceph-mon[36451]: pgmap v260: 129 pgs: 129 active+clean; 74 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 290 op/s 2024-05-27T07:47:49.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:49 smithi186 ceph-mon[36647]: pgmap v260: 129 pgs: 129 active+clean; 74 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 290 op/s 2024-05-27T07:47:50.273 INFO:tasks.workunit.client.0.smithi114.stdout: 1 821545 11.48 MB/sec execute 242 sec latency 26.706 ms 2024-05-27T07:47:51.273 INFO:tasks.workunit.client.0.smithi114.stdout: 1 824047 11.49 MB/sec execute 243 sec latency 37.974 ms 2024-05-27T07:47:51.396 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:47:51.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:47:51.685 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:47:51.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:51 smithi163 ceph-mon[36451]: pgmap v261: 129 pgs: 129 active+clean; 70 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-05-27T07:47:51.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:51 smithi186 ceph-mon[36647]: pgmap v261: 129 pgs: 129 active+clean; 70 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-05-27T07:47:51.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:51 smithi114 ceph-mon[32038]: pgmap v261: 129 pgs: 129 active+clean; 70 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-05-27T07:47:52.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 825343 11.46 MB/sec execute 244 sec latency 99.809 ms 2024-05-27T07:47:53.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 827554 11.47 MB/sec execute 245 sec latency 136.153 ms 2024-05-27T07:47:53.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:53 smithi114 ceph-mon[32038]: pgmap v262: 129 pgs: 129 active+clean; 72 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 388 op/s 2024-05-27T07:47:53.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:53 smithi163 ceph-mon[36451]: pgmap v262: 129 pgs: 129 active+clean; 72 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 388 op/s 2024-05-27T07:47:53.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:53 smithi186 ceph-mon[36647]: pgmap v262: 129 pgs: 129 active+clean; 72 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 388 op/s 2024-05-27T07:47:53.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.798+0000 7fea4a5a7700 1 -- 172.21.15.114:0/1106517109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea44108770 msgr2=0x7fea441100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:53.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.798+0000 7fea4a5a7700 1 --2- 172.21.15.114:0/1106517109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea44108770 0x7fea441100d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009fc0 tx=0x7fea3c00b330 comp rx=0 tx=0).stop 2024-05-27T07:47:53.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.798+0000 7fea4a5a7700 1 -- 172.21.15.114:0/1106517109 shutdown_connections 2024-05-27T07:47:53.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.798+0000 7fea4a5a7700 1 --2- 172.21.15.114:0/1106517109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea44108770 0x7fea441100d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.798+0000 7fea4a5a7700 1 --2- 172.21.15.114:0/1106517109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea44103540 0x7fea44103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.798+0000 7fea4a5a7700 1 --2- 172.21.15.114:0/1106517109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea44107e20 0x7fea44108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.798+0000 7fea4a5a7700 1 -- 172.21.15.114:0/1106517109 >> 172.21.15.114:0/1106517109 conn(0x7fea44078df0 msgr2=0x7fea44079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:53.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.798+0000 7fea4a5a7700 1 -- 172.21.15.114:0/1106517109 shutdown_connections 2024-05-27T07:47:53.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.798+0000 7fea4a5a7700 1 -- 172.21.15.114:0/1106517109 wait complete. 2024-05-27T07:47:53.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.799+0000 7fea4a5a7700 1 Processor -- start 2024-05-27T07:47:53.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.799+0000 7fea4a5a7700 1 -- start start 2024-05-27T07:47:53.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.799+0000 7fea4a5a7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea44103540 0x7fea4419bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.799+0000 7fea4a5a7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea44107e20 0x7fea4419c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea4a5a7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea44108770 0x7fea4419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea4a5a7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fea441133a0 con 0x7fea44107e20 2024-05-27T07:47:53.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea4a5a7700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fea44113220 con 0x7fea44103540 2024-05-27T07:47:53.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea4a5a7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fea44113520 con 0x7fea44108770 2024-05-27T07:47:53.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea44107e20 0x7fea4419c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea48b44700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea44108770 0x7fea4419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea44107e20 0x7fea4419c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46698/0 (socket says 172.21.15.114:46698) 2024-05-27T07:47:53.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea437fe700 1 -- 172.21.15.114:0/3993919739 learned_addr learned my addr 172.21.15.114:0/3993919739 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:53.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea43fff700 1 --2- 172.21.15.114:0/3993919739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea44103540 0x7fea4419bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea48b44700 1 -- 172.21.15.114:0/3993919739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea44103540 msgr2=0x7fea4419bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:53.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea48b44700 1 --2- 172.21.15.114:0/3993919739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea44103540 0x7fea4419bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea48b44700 1 -- 172.21.15.114:0/3993919739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea44107e20 msgr2=0x7fea4419c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:53.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea48b44700 1 --2- 172.21.15.114:0/3993919739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea44107e20 0x7fea4419c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea48b44700 1 -- 172.21.15.114:0/3993919739 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea441a6fb0 con 0x7fea44108770 2024-05-27T07:47:53.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.800+0000 7fea48b44700 1 --2- 172.21.15.114:0/3993919739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea44108770 0x7fea4419c700 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fea3c015000 tx=0x7fea3c00ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:53.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.801+0000 7fea417fa700 1 -- 172.21.15.114:0/3993919739 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c021070 con 0x7fea44108770 2024-05-27T07:47:53.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.801+0000 7fea417fa700 1 -- 172.21.15.114:0/3993919739 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea3c007ac0 con 0x7fea44108770 2024-05-27T07:47:53.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.801+0000 7fea417fa700 1 -- 172.21.15.114:0/3993919739 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c005040 con 0x7fea44108770 2024-05-27T07:47:53.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.801+0000 7fea4a5a7700 1 -- 172.21.15.114:0/3993919739 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea44199a30 con 0x7fea44108770 2024-05-27T07:47:53.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.801+0000 7fea4a5a7700 1 -- 172.21.15.114:0/3993919739 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea44199f70 con 0x7fea44108770 2024-05-27T07:47:53.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.804+0000 7fea417fa700 1 -- 172.21.15.114:0/3993919739 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fea3c024070 con 0x7fea44108770 2024-05-27T07:47:53.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.804+0000 7fea417fa700 1 --2- 172.21.15.114:0/3993919739 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fea2c0b2a70 0x7fea2c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.804+0000 7fea58fbcb80 1 Processor -- start 2024-05-27T07:47:53.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea43fff700 1 --2- 172.21.15.114:0/3993919739 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fea2c0b2a70 0x7fea2c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea58fbcb80 1 -- start start 2024-05-27T07:47:53.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea417fa700 1 -- 172.21.15.114:0/3993919739 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fea3c00f020 con 0x7fea44108770 2024-05-27T07:47:53.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea58fbcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f19fe5ec0 0x562f1a16abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea58fbcb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a16b0f0 0x562f1a164c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea58fbcb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a1651a0 0x562f1a1655b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea58fbcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562f1a092df0 con 0x562f19fe5ec0 2024-05-27T07:47:53.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea58fbcb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562f1a082990 con 0x562f1a16b0f0 2024-05-27T07:47:53.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea58fbcb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562f1a0869d0 con 0x562f1a1651a0 2024-05-27T07:47:53.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea48b44700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a1651a0 0x562f1a1655b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea43fff700 1 --2- 172.21.15.114:0/3993919739 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fea2c0b2a70 0x7fea2c0b4f80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fea34009750 tx=0x7fea34009310 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea43fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f19fe5ec0 0x562f1a16abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea48b44700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a1651a0 0x562f1a1655b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:60264/0 (socket says 172.21.15.114:60264) 2024-05-27T07:47:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea48b44700 1 -- 172.21.15.114:0/3723363521 learned_addr learned my addr 172.21.15.114:0/3723363521 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea437fe700 1 --2- 172.21.15.114:0/3723363521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a16b0f0 0x562f1a164c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea48b44700 1 -- 172.21.15.114:0/3723363521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a16b0f0 msgr2=0x562f1a164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea48b44700 1 --2- 172.21.15.114:0/3723363521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a16b0f0 0x562f1a164c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea48b44700 1 -- 172.21.15.114:0/3723363521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f19fe5ec0 msgr2=0x562f1a16abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea48b44700 1 --2- 172.21.15.114:0/3723363521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f19fe5ec0 0x562f1a16abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.805+0000 7fea48b44700 1 -- 172.21.15.114:0/3723363521 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f1a02c030 con 0x562f1a1651a0 2024-05-27T07:47:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.806+0000 7fea48b44700 1 --2- 172.21.15.114:0/3723363521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a1651a0 0x562f1a1655b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fea3c0aacc0 tx=0x7fea3c01f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a62509b6fad495a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:53.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.806+0000 7fea2a7fc700 1 -- 172.21.15.114:0/3723363521 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c0b3440 con 0x562f1a1651a0 2024-05-27T07:47:53.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.806+0000 7fea2a7fc700 1 -- 172.21.15.114:0/3723363521 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea3c0aa840 con 0x562f1a1651a0 2024-05-27T07:47:53.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.806+0000 7fea2a7fc700 1 -- 172.21.15.114:0/3723363521 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c0b3b00 con 0x562f1a1651a0 2024-05-27T07:47:53.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.807+0000 7fea58fbcb80 1 -- 172.21.15.114:0/3723363521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a1651a0 msgr2=0x562f1a1655b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:53.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.807+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/3723363521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a1651a0 0x562f1a1655b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fea3c0aacc0 tx=0x7fea3c01f000 comp rx=0 tx=0).stop 2024-05-27T07:47:53.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.807+0000 7fea58fbcb80 1 -- 172.21.15.114:0/3723363521 shutdown_connections 2024-05-27T07:47:53.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.807+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/3723363521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a1651a0 0x562f1a1655b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.807+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/3723363521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f19fe5ec0 0x562f1a16abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.807+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/3723363521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a16b0f0 0x562f1a164c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.807+0000 7fea58fbcb80 1 -- 172.21.15.114:0/3723363521 >> 172.21.15.114:0/3723363521 conn(0x562f19fe87f0 msgr2=0x562f1a15fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.807+0000 7fea58fbcb80 1 -- 172.21.15.114:0/3723363521 shutdown_connections 2024-05-27T07:47:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.807+0000 7fea58fbcb80 1 -- 172.21.15.114:0/3723363521 wait complete. 2024-05-27T07:47:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.807+0000 7fea58fbcb80 1 -- start start 2024-05-27T07:47:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.808+0000 7fea58fbcb80 1 Processor -- start 2024-05-27T07:47:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.808+0000 7fea58fbcb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f19fe5ec0 0x562f1a16e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.808+0000 7fea58fbcb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a1651a0 0x562f1a16ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.808+0000 7fea58fbcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f1a16b0f0 0x562f1a16f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.808+0000 7fea58fbcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562f19f89020 con 0x562f1a16b0f0 2024-05-27T07:47:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.808+0000 7fea58fbcb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562f19feae20 con 0x562f1a1651a0 2024-05-27T07:47:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.808+0000 7fea58fbcb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562f1a1719a0 con 0x562f19fe5ec0 2024-05-27T07:47:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.808+0000 7fea58fbcb80 10 client.0 _ll_register_callbacks cb 0x562f1a019630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.808+0000 7fea43fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f19fe5ec0 0x562f1a16e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.808+0000 7fea43fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f19fe5ec0 0x562f1a16e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:60268/0 (socket says 172.21.15.114:60268) 2024-05-27T07:47:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.808+0000 7fea43fff700 1 -- 172.21.15.114:0/1337409487 learned_addr learned my addr 172.21.15.114:0/1337409487 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.808+0000 7fea48b44700 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f1a16b0f0 0x562f1a16f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.808+0000 7fea43fff700 1 -- 172.21.15.114:0/1337409487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a1651a0 msgr2=0x562f1a16ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea437fe700 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a1651a0 0x562f1a16ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea43fff700 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a1651a0 0x562f1a16ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea43fff700 1 -- 172.21.15.114:0/1337409487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f1a16b0f0 msgr2=0x562f1a16f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea43fff700 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f1a16b0f0 0x562f1a16f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea43fff700 1 -- 172.21.15.114:0/1337409487 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f19fd2950 con 0x562f19fe5ec0 2024-05-27T07:47:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea48b44700 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f1a16b0f0 0x562f1a16f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:47:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea437fe700 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a1651a0 0x562f1a16ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:47:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea43fff700 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f19fe5ec0 0x562f1a16e8e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fea3401dd30 tx=0x7fea340047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76ff01f7e3ba3bfe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea237fe700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:47:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea237fe700 1 -- 172.21.15.114:0/1337409487 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3402b060 con 0x562f19fe5ec0 2024-05-27T07:47:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea58fbcb80 1 -- 172.21.15.114:0/1337409487 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea3c0aa840 con 0x562f19fe5ec0 2024-05-27T07:47:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea237fe700 1 -- 172.21.15.114:0/1337409487 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fea3401cdb0 con 0x562f19fe5ec0 2024-05-27T07:47:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea237fe700 1 -- 172.21.15.114:0/1337409487 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fea34032050 con 0x562f19fe5ec0 2024-05-27T07:47:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea58fbcb80 10 client.26935 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.809+0000 7fea58fbcb80 1 -- 172.21.15.114:0/1337409487 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x562f1a03c7e0 con 0x562f19fe5ec0 2024-05-27T07:47:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.810+0000 7fea237fe700 1 -- 172.21.15.114:0/1337409487 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea3c0aa840 con 0x562f19fe5ec0 2024-05-27T07:47:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.810+0000 7fea237fe700 1 -- 172.21.15.114:0/1337409487 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7fea3400e5b0 con 0x562f19fe5ec0 2024-05-27T07:47:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.810+0000 7fea58fbcb80 10 client.26935 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.810+0000 7fea58fbcb80 10 client.26935 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.810+0000 7fea58fbcb80 20 client.26935 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:53.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.810+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562f1a17e8e0 0x562f1a180d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.810+0000 7fea58fbcb80 4 client.26935 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:53.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.810+0000 7fea58fbcb80 1 -- 172.21.15.114:0/1337409487 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f1a094d40 con 0x562f1a17e8e0 2024-05-27T07:47:53.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.810+0000 7fea437fe700 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562f1a17e8e0 0x562f1a180d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.810+0000 7fea237fe700 10 client.26935 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:53.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.810+0000 7fea437fe700 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562f1a17e8e0 0x562f1a180d70 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44acae3bb7141cbd server_cookie=ab406b1e77724f2b in_seq=0 out_seq=0 2024-05-27T07:47:53.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.812+0000 7fea237fe700 1 -- 172.21.15.114:0/1337409487 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562f1a094d40 con 0x562f1a17e8e0 2024-05-27T07:47:53.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.812+0000 7fea237fe700 10 client.26935 handle_command_reply: tid=0 2024-05-27T07:47:53.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.813+0000 7fea58fbcb80 1 client.26935 shutdown 2024-05-27T07:47:53.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.813+0000 7fea58fbcb80 1 -- 172.21.15.114:0/1337409487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f19fe5ec0 msgr2=0x562f1a16e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:53.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.813+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f19fe5ec0 0x562f1a16e8e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fea3401dd30 tx=0x7fea340047c0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.813+0000 7fea58fbcb80 1 -- 172.21.15.114:0/1337409487 shutdown_connections 2024-05-27T07:47:53.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.813+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f1a16b0f0 0x562f1a16f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.813+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f19fe5ec0 0x562f1a16e8e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.813+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562f1a17e8e0 0x562f1a180d70 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.813+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/1337409487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a1651a0 0x562f1a16ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.813+0000 7fea237fe700 0 client.26935 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:53.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.813+0000 7fea58fbcb80 1 -- 172.21.15.114:0/1337409487 >> 172.21.15.114:0/1337409487 conn(0x562f19fe87f0 msgr2=0x562f1a163c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:53.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.813+0000 7fea58fbcb80 1 -- 172.21.15.114:0/1337409487 shutdown_connections 2024-05-27T07:47:53.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.813+0000 7fea58fbcb80 1 -- 172.21.15.114:0/1337409487 wait complete. 2024-05-27T07:47:53.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.813+0000 7fea58fbcb80 20 client.26935 trim_cache size 0 max 16384 2024-05-27T07:47:53.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.837+0000 7fea58fbcb80 1 Processor -- start 2024-05-27T07:47:53.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.837+0000 7fea58fbcb80 1 -- start start 2024-05-27T07:47:53.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.837+0000 7fea58fbcb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f19fe5ec0 0x562f1a164c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.837+0000 7fea58fbcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f1a1651a0 0x562f1a16f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.837+0000 7fea58fbcb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a16b0f0 0x562f1a170d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.837+0000 7fea58fbcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fea441b2900 con 0x562f1a1651a0 2024-05-27T07:47:53.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.837+0000 7fea58fbcb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fea441b2fd0 con 0x562f19fe5ec0 2024-05-27T07:47:53.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.837+0000 7fea58fbcb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fea441b0750 con 0x562f1a16b0f0 2024-05-27T07:47:53.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.837+0000 7fea437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f1a1651a0 0x562f1a16f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.837+0000 7fea437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f1a1651a0 0x562f1a16f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46716/0 (socket says 172.21.15.114:46716) 2024-05-27T07:47:53.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.837+0000 7fea437fe700 1 -- 172.21.15.114:0/1188208062 learned_addr learned my addr 172.21.15.114:0/1188208062 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:53.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.837+0000 7fea48b44700 1 --2- 172.21.15.114:0/1188208062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a16b0f0 0x562f1a170d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.837+0000 7fea43fff700 1 --2- 172.21.15.114:0/1188208062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f19fe5ec0 0x562f1a164c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.838+0000 7fea437fe700 1 -- 172.21.15.114:0/1188208062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a16b0f0 msgr2=0x562f1a170d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:53.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.838+0000 7fea437fe700 1 --2- 172.21.15.114:0/1188208062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a16b0f0 0x562f1a170d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.838+0000 7fea437fe700 1 -- 172.21.15.114:0/1188208062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f19fe5ec0 msgr2=0x562f1a164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:53.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.838+0000 7fea437fe700 1 --2- 172.21.15.114:0/1188208062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f19fe5ec0 0x562f1a164c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.838+0000 7fea437fe700 1 -- 172.21.15.114:0/1188208062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f1a160b30 con 0x562f1a1651a0 2024-05-27T07:47:53.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.838+0000 7fea437fe700 1 --2- 172.21.15.114:0/1188208062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f1a1651a0 0x562f1a16f2d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fea3800b750 tx=0x7fea3800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91c2f29e64839e60 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:53.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.838+0000 7fea29ffb700 1 -- 172.21.15.114:0/1188208062 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38003b60 con 0x562f1a1651a0 2024-05-27T07:47:53.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.838+0000 7fea29ffb700 1 -- 172.21.15.114:0/1188208062 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea38003d00 con 0x562f1a1651a0 2024-05-27T07:47:53.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.838+0000 7fea29ffb700 1 -- 172.21.15.114:0/1188208062 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3801c660 con 0x562f1a1651a0 2024-05-27T07:47:53.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.839+0000 7fea58fbcb80 1 -- 172.21.15.114:0/1188208062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f1a1651a0 msgr2=0x562f1a16f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:53.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.839+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/1188208062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f1a1651a0 0x562f1a16f2d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fea3800b750 tx=0x7fea3800bc80 comp rx=0 tx=0).stop 2024-05-27T07:47:53.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.839+0000 7fea58fbcb80 1 -- 172.21.15.114:0/1188208062 shutdown_connections 2024-05-27T07:47:53.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.839+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/1188208062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a16b0f0 0x562f1a170d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.839+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/1188208062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f19fe5ec0 0x562f1a164c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.839+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/1188208062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f1a1651a0 0x562f1a16f2d0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.839+0000 7fea58fbcb80 1 -- 172.21.15.114:0/1188208062 >> 172.21.15.114:0/1188208062 conn(0x562f19fe87f0 msgr2=0x562f1a168b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:53.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.839+0000 7fea58fbcb80 1 -- 172.21.15.114:0/1188208062 shutdown_connections 2024-05-27T07:47:53.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.839+0000 7fea58fbcb80 1 -- 172.21.15.114:0/1188208062 wait complete. 2024-05-27T07:47:53.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea58fbcb80 1 -- start start 2024-05-27T07:47:53.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea58fbcb80 1 Processor -- start 2024-05-27T07:47:53.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea58fbcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f19fe5ec0 0x562f1a173bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea58fbcb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a1651a0 0x562f1a174100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea58fbcb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a16b0f0 0x562f1a174640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea58fbcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fea441b0dc0 con 0x562f19fe5ec0 2024-05-27T07:47:53.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea58fbcb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fea441b2230 con 0x562f1a1651a0 2024-05-27T07:47:53.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea58fbcb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fea441b1490 con 0x562f1a16b0f0 2024-05-27T07:47:53.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea58fbcb80 10 client.0 _ll_register_callbacks cb 0x562f1a019630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:53.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea43fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f19fe5ec0 0x562f1a173bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea43fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f19fe5ec0 0x562f1a173bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46730/0 (socket says 172.21.15.114:46730) 2024-05-27T07:47:53.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea43fff700 1 -- 172.21.15.114:0/2074883237 learned_addr learned my addr 172.21.15.114:0/2074883237 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:53.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea437fe700 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a1651a0 0x562f1a174100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.840+0000 7fea48b44700 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a16b0f0 0x562f1a174640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea437fe700 1 -- 172.21.15.114:0/2074883237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a16b0f0 msgr2=0x562f1a174640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:53.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea437fe700 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a16b0f0 0x562f1a174640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea437fe700 1 -- 172.21.15.114:0/2074883237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f19fe5ec0 msgr2=0x562f1a173bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:53.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea437fe700 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f19fe5ec0 0x562f1a173bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea437fe700 1 -- 172.21.15.114:0/2074883237 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f1a1e9410 con 0x562f1a1651a0 2024-05-27T07:47:53.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea43fff700 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f19fe5ec0 0x562f1a173bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:53.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea437fe700 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a1651a0 0x562f1a174100 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x562f1a16b540 tx=0x7fea38003ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaa1f5e3b3a84112 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:53.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea237fe700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:47:53.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea237fe700 1 -- 172.21.15.114:0/2074883237 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3802b020 con 0x562f1a1651a0 2024-05-27T07:47:53.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea237fe700 1 -- 172.21.15.114:0/2074883237 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fea38005040 con 0x562f1a1651a0 2024-05-27T07:47:53.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea58fbcb80 1 -- 172.21.15.114:0/2074883237 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea38003d00 con 0x562f1a1651a0 2024-05-27T07:47:53.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea237fe700 1 -- 172.21.15.114:0/2074883237 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fea38005240 con 0x562f1a1651a0 2024-05-27T07:47:53.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea58fbcb80 10 client.26828 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:53.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.841+0000 7fea58fbcb80 1 -- 172.21.15.114:0/2074883237 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x562f1a1ea630 con 0x562f1a1651a0 2024-05-27T07:47:53.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.842+0000 7fea237fe700 1 -- 172.21.15.114:0/2074883237 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea38003d00 con 0x562f1a1651a0 2024-05-27T07:47:53.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.842+0000 7fea237fe700 1 -- 172.21.15.114:0/2074883237 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7fea38011520 con 0x562f1a1651a0 2024-05-27T07:47:53.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.842+0000 7fea58fbcb80 10 client.26828 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:53.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.842+0000 7fea58fbcb80 10 client.26828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:53.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.842+0000 7fea58fbcb80 20 client.26828 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:53.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.842+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562f1a1ead30 0x562f1a1ed150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:53.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.842+0000 7fea58fbcb80 4 client.26828 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:47:53.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.842+0000 7fea58fbcb80 1 -- 172.21.15.114:0/2074883237 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562f1a08e520 con 0x562f1a1ead30 2024-05-27T07:47:53.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.842+0000 7fea43fff700 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562f1a1ead30 0x562f1a1ed150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:53.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.842+0000 7fea237fe700 10 client.26828 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:53.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.843+0000 7fea43fff700 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562f1a1ead30 0x562f1a1ed150 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb76b719d00740e server_cookie=b8a58bee16780763 in_seq=0 out_seq=0 2024-05-27T07:47:53.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.844+0000 7fea237fe700 1 -- 172.21.15.114:0/2074883237 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562f1a08e520 con 0x562f1a1ead30 2024-05-27T07:47:53.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.844+0000 7fea237fe700 10 client.26828 handle_command_reply: tid=0 2024-05-27T07:47:53.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.844+0000 7fea58fbcb80 1 client.26828 shutdown 2024-05-27T07:47:53.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.844+0000 7fea58fbcb80 1 -- 172.21.15.114:0/2074883237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a1651a0 msgr2=0x562f1a174100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:53.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.844+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a1651a0 0x562f1a174100 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x562f1a16b540 tx=0x7fea38003ed0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.844+0000 7fea58fbcb80 1 -- 172.21.15.114:0/2074883237 shutdown_connections 2024-05-27T07:47:53.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.844+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562f1a16b0f0 0x562f1a174640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.844+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562f19fe5ec0 0x562f1a173bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.844+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562f1a1ead30 0x562f1a1ed150 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.844+0000 7fea58fbcb80 1 --2- 172.21.15.114:0/2074883237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f1a1651a0 0x562f1a174100 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.844+0000 7fea58fbcb80 1 -- 172.21.15.114:0/2074883237 >> 172.21.15.114:0/2074883237 conn(0x562f19fe87f0 msgr2=0x562f1a168b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:53.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.844+0000 7fea237fe700 0 client.26828 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:53.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.845+0000 7fea58fbcb80 1 -- 172.21.15.114:0/2074883237 shutdown_connections 2024-05-27T07:47:53.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.845+0000 7fea58fbcb80 1 -- 172.21.15.114:0/2074883237 wait complete. 2024-05-27T07:47:53.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.845+0000 7fea58fbcb80 20 client.26828 trim_cache size 0 max 16384 2024-05-27T07:47:53.849 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:47:53.849 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:47:53.849 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "f97396ae-03e4-41e5-947e-8684f8765843", 2024-05-27T07:47:53.850 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:47:53.850 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:47:53.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.845+0000 7fea4a5a7700 1 -- 172.21.15.114:0/3993919739 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fea2c0b2a70 msgr2=0x7fea2c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:53.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.845+0000 7fea4a5a7700 1 --2- 172.21.15.114:0/3993919739 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fea2c0b2a70 0x7fea2c0b4f80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fea34009750 tx=0x7fea34009310 comp rx=0 tx=0).stop 2024-05-27T07:47:53.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.845+0000 7fea4a5a7700 1 -- 172.21.15.114:0/3993919739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea44108770 msgr2=0x7fea4419c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:53.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.845+0000 7fea4a5a7700 1 --2- 172.21.15.114:0/3993919739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea44108770 0x7fea4419c700 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fea3c015000 tx=0x7fea3c00ccb0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.846+0000 7fea4a5a7700 1 -- 172.21.15.114:0/3993919739 shutdown_connections 2024-05-27T07:47:53.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.846+0000 7fea4a5a7700 1 --2- 172.21.15.114:0/3993919739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea44108770 0x7fea4419c700 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.846+0000 7fea4a5a7700 1 --2- 172.21.15.114:0/3993919739 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fea2c0b2a70 0x7fea2c0b4f80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.846+0000 7fea4a5a7700 1 --2- 172.21.15.114:0/3993919739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea44103540 0x7fea4419bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.846+0000 7fea4a5a7700 1 --2- 172.21.15.114:0/3993919739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea44107e20 0x7fea4419c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:53.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.846+0000 7fea4a5a7700 1 -- 172.21.15.114:0/3993919739 >> 172.21.15.114:0/3993919739 conn(0x7fea44078df0 msgr2=0x7fea44100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:53.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.846+0000 7fea4a5a7700 1 -- 172.21.15.114:0/3993919739 shutdown_connections 2024-05-27T07:47:53.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:53.846+0000 7fea4a5a7700 1 -- 172.21.15.114:0/3993919739 wait complete. 2024-05-27T07:47:54.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 829563 11.45 MB/sec execute 246 sec latency 16.916 ms 2024-05-27T07:47:54.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:54 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:47:54.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:54 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:54.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:54 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:47:54.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:54 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:54.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:47:54.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:54 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:47:54.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:54 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:54.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:54 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:47:54.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:54 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:54.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:54 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:47:54.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:54 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:54.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:54 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:47:54.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:54 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:47:54.898 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:47:55.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 831520 11.46 MB/sec execute 247 sec latency 19.976 ms 2024-05-27T07:47:55.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:55 smithi114 ceph-mon[32038]: pgmap v263: 129 pgs: 129 active+clean; 72 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 317 op/s 2024-05-27T07:47:55.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:55 smithi163 ceph-mon[36451]: pgmap v263: 129 pgs: 129 active+clean; 72 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 317 op/s 2024-05-27T07:47:55.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:55 smithi186 ceph-mon[36647]: pgmap v263: 129 pgs: 129 active+clean; 72 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 317 op/s 2024-05-27T07:47:56.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 834163 11.47 MB/sec execute 248 sec latency 31.848 ms 2024-05-27T07:47:56.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.705+0000 7f62bd863700 1 -- 172.21.15.114:0/673506326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62b8107de0 msgr2=0x7f62b81081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.705+0000 7f62bd863700 1 --2- 172.21.15.114:0/673506326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62b8107de0 0x7f62b81081f0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0099c0 tx=0x7f62ac009cd0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.705+0000 7f62bd863700 1 -- 172.21.15.114:0/673506326 shutdown_connections 2024-05-27T07:47:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.705+0000 7f62bd863700 1 --2- 172.21.15.114:0/673506326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62b8108730 0x7f62b8110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.705+0000 7f62bd863700 1 --2- 172.21.15.114:0/673506326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62b81034b0 0x7f62b81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.705+0000 7f62bd863700 1 --2- 172.21.15.114:0/673506326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62b8107de0 0x7f62b81081f0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.705+0000 7f62bd863700 1 -- 172.21.15.114:0/673506326 >> 172.21.15.114:0/673506326 conn(0x7f62b80fedb0 msgr2=0x7f62b81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.705+0000 7f62bd863700 1 -- 172.21.15.114:0/673506326 shutdown_connections 2024-05-27T07:47:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.705+0000 7f62bd863700 1 -- 172.21.15.114:0/673506326 wait complete. 2024-05-27T07:47:56.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.706+0000 7f62bd863700 1 Processor -- start 2024-05-27T07:47:56.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.706+0000 7f62bd863700 1 -- start start 2024-05-27T07:47:56.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.706+0000 7f62bd863700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62b81034b0 0x7f62b819bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.706+0000 7f62bd863700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62b8107de0 0x7f62b819c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.706+0000 7f62bd863700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62b8108730 0x7f62b819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62bd863700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f62b8113340 con 0x7f62b8107de0 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62bd863700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81131c0 con 0x7f62b8108730 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62bd863700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81134c0 con 0x7f62b81034b0 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62b67fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62b8107de0 0x7f62b819c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62b6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62b81034b0 0x7f62b819bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62b6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62b81034b0 0x7f62b819bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:57904/0 (socket says 172.21.15.114:57904) 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62b6ffd700 1 -- 172.21.15.114:0/2498682251 learned_addr learned my addr 172.21.15.114:0/2498682251 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62b67fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62b8107de0 0x7f62b819c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34598/0 (socket says 172.21.15.114:34598) 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62b77fe700 1 --2- 172.21.15.114:0/2498682251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62b8108730 0x7f62b819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62b67fc700 1 -- 172.21.15.114:0/2498682251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62b81034b0 msgr2=0x7f62b819bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62b67fc700 1 --2- 172.21.15.114:0/2498682251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62b81034b0 0x7f62b819bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62b67fc700 1 -- 172.21.15.114:0/2498682251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62b8108730 msgr2=0x7f62b819c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62b67fc700 1 --2- 172.21.15.114:0/2498682251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62b8108730 0x7f62b819c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62b67fc700 1 -- 172.21.15.114:0/2498682251 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b81a6ff0 con 0x7f62b8107de0 2024-05-27T07:47:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62b67fc700 1 --2- 172.21.15.114:0/2498682251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62b8107de0 0x7f62b819c1c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0098e0 tx=0x7f62ac0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:56.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62bc861700 1 -- 172.21.15.114:0/2498682251 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac026070 con 0x7f62b8107de0 2024-05-27T07:47:56.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.707+0000 7f62bd863700 1 -- 172.21.15.114:0/2498682251 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b81a7280 con 0x7f62b8107de0 2024-05-27T07:47:56.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.708+0000 7f62bc861700 1 -- 172.21.15.114:0/2498682251 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62ac00f460 con 0x7f62b8107de0 2024-05-27T07:47:56.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.708+0000 7f62bc861700 1 -- 172.21.15.114:0/2498682251 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac009e60 con 0x7f62b8107de0 2024-05-27T07:47:56.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.708+0000 7f62bd863700 1 -- 172.21.15.114:0/2498682251 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b81a7730 con 0x7f62b8107de0 2024-05-27T07:47:56.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.710+0000 7f62bc861700 1 -- 172.21.15.114:0/2498682251 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f62ac029070 con 0x7f62b8107de0 2024-05-27T07:47:56.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.711+0000 7f62bc861700 1 --2- 172.21.15.114:0/2498682251 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f62a00b29f0 0x7f62a00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.711+0000 7f62cc278b80 1 Processor -- start 2024-05-27T07:47:56.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.711+0000 7f62b6ffd700 1 --2- 172.21.15.114:0/2498682251 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f62a00b29f0 0x7f62a00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.711+0000 7f62cc278b80 1 -- start start 2024-05-27T07:47:56.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.711+0000 7f62bc861700 1 -- 172.21.15.114:0/2498682251 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f62ac0e3060 con 0x7f62b8107de0 2024-05-27T07:47:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62cc278b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e90d8e40 0x5573e925a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62cc278b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e9256a40 0x5573e9256e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62cc278b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9257390 0x5573e926b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62cc278b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5573e9185c90 con 0x5573e9257390 2024-05-27T07:47:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62cc278b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5573e9175830 con 0x5573e90d8e40 2024-05-27T07:47:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62cc278b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5573e9179870 con 0x5573e9256a40 2024-05-27T07:47:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62b77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9257390 0x5573e926b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62b67fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e9256a40 0x5573e9256e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62b6ffd700 1 --2- 172.21.15.114:0/2498682251 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f62a00b29f0 0x7f62a00b4f00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009d20 tx=0x7f62a8009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62b77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9257390 0x5573e926b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34612/0 (socket says 172.21.15.114:34612) 2024-05-27T07:47:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62b67fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e9256a40 0x5573e9256e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:57906/0 (socket says 172.21.15.114:57906) 2024-05-27T07:47:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62b77fe700 1 -- 172.21.15.114:0/2331665754 learned_addr learned my addr 172.21.15.114:0/2331665754 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62b6ffd700 1 --2- 172.21.15.114:0/2331665754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e90d8e40 0x5573e925a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62b77fe700 1 -- 172.21.15.114:0/2331665754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e9256a40 msgr2=0x5573e9256e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:56.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62b77fe700 1 --2- 172.21.15.114:0/2331665754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e9256a40 0x5573e9256e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62b77fe700 1 -- 172.21.15.114:0/2331665754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e90d8e40 msgr2=0x5573e925a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:56.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62b77fe700 1 --2- 172.21.15.114:0/2331665754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e90d8e40 0x5573e925a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62b77fe700 1 -- 172.21.15.114:0/2331665754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573e90c58d0 con 0x5573e9257390 2024-05-27T07:47:56.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f62b77fe700 1 --2- 172.21.15.114:0/2331665754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9257390 0x5573e926b6b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f62b000a580 tx=0x7f62b00078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aff6f6243d659ca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:56.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f629dffb700 1 -- 172.21.15.114:0/2331665754 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0003a00 con 0x5573e9257390 2024-05-27T07:47:56.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f629dffb700 1 -- 172.21.15.114:0/2331665754 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62b0003ba0 con 0x5573e9257390 2024-05-27T07:47:56.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.712+0000 7f629dffb700 1 -- 172.21.15.114:0/2331665754 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0014600 con 0x5573e9257390 2024-05-27T07:47:56.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.713+0000 7f62cc278b80 1 -- 172.21.15.114:0/2331665754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9257390 msgr2=0x5573e926b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:56.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.713+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2331665754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9257390 0x5573e926b6b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f62b000a580 tx=0x7f62b00078d0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.713+0000 7f62cc278b80 1 -- 172.21.15.114:0/2331665754 shutdown_connections 2024-05-27T07:47:56.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.713+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2331665754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9257390 0x5573e926b6b0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.713+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2331665754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e90d8e40 0x5573e925a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.713+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2331665754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e9256a40 0x5573e9256e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.713+0000 7f62cc278b80 1 -- 172.21.15.114:0/2331665754 >> 172.21.15.114:0/2331665754 conn(0x5573e9133af0 msgr2=0x5573e9251640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:56.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.713+0000 7f62cc278b80 1 -- 172.21.15.114:0/2331665754 shutdown_connections 2024-05-27T07:47:56.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.713+0000 7f62cc278b80 1 -- 172.21.15.114:0/2331665754 wait complete. 2024-05-27T07:47:56.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.714+0000 7f62cc278b80 1 -- start start 2024-05-27T07:47:56.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.714+0000 7f62cc278b80 1 Processor -- start 2024-05-27T07:47:56.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.714+0000 7f62cc278b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e90d8e40 0x5573e9268ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62cc278b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9256a40 0x5573e92690e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62cc278b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9257390 0x5573e9269620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62cc278b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5573e907bfa0 con 0x5573e9257390 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62cc278b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5573e91177c0 con 0x5573e9256a40 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62cc278b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5573e926be90 con 0x5573e90d8e40 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62cc278b80 10 client.0 _ll_register_callbacks cb 0x5573e9110920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62b77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9257390 0x5573e9269620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62b6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e90d8e40 0x5573e9268ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62b6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e90d8e40 0x5573e9268ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:57920/0 (socket says 172.21.15.114:57920) 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62b6ffd700 1 -- 172.21.15.114:0/2720056714 learned_addr learned my addr 172.21.15.114:0/2720056714 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62b67fc700 1 --2- 172.21.15.114:0/2720056714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9256a40 0x5573e92690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62b77fe700 1 -- 172.21.15.114:0/2720056714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e90d8e40 msgr2=0x5573e9268ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62b77fe700 1 --2- 172.21.15.114:0/2720056714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e90d8e40 0x5573e9268ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62b77fe700 1 -- 172.21.15.114:0/2720056714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9256a40 msgr2=0x5573e92690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:56.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62b77fe700 1 --2- 172.21.15.114:0/2720056714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9256a40 0x5573e92690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62b77fe700 1 -- 172.21.15.114:0/2720056714 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573e9114690 con 0x5573e9257390 2024-05-27T07:47:56.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62b67fc700 1 --2- 172.21.15.114:0/2720056714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9256a40 0x5573e92690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:56.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62b77fe700 1 --2- 172.21.15.114:0/2720056714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9257390 0x5573e9269620 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f62b000afa0 tx=0x7f62b0004460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a9c9e359674461 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:56.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f6296ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:56.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f6296ffd700 1 -- 172.21.15.114:0/2720056714 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0022070 con 0x5573e9257390 2024-05-27T07:47:56.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f6296ffd700 1 -- 172.21.15.114:0/2720056714 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f62b0004ad0 con 0x5573e9257390 2024-05-27T07:47:56.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.715+0000 7f62cc278b80 1 -- 172.21.15.114:0/2720056714 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62b0003ba0 con 0x5573e9257390 2024-05-27T07:47:56.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.716+0000 7f6296ffd700 1 -- 172.21.15.114:0/2720056714 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f62b0003a00 con 0x5573e9257390 2024-05-27T07:47:56.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.716+0000 7f62cc278b80 10 client.17382 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:56.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.716+0000 7f62cc278b80 1 -- 172.21.15.114:0/2720056714 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5573e9276960 con 0x5573e9257390 2024-05-27T07:47:56.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.716+0000 7f6296ffd700 1 -- 172.21.15.114:0/2720056714 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62b0003ba0 con 0x5573e9257390 2024-05-27T07:47:56.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.716+0000 7f6296ffd700 1 -- 172.21.15.114:0/2720056714 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7f62b0014400 con 0x5573e9257390 2024-05-27T07:47:56.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.716+0000 7f62cc278b80 10 client.17382 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:56.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.716+0000 7f62cc278b80 10 client.17382 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:56.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.716+0000 7f62cc278b80 20 client.17382 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:56.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.716+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2720056714 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5573e9278e90 0x5573e927b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.716+0000 7f62cc278b80 4 client.17382 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:56.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.716+0000 7f62cc278b80 1 -- 172.21.15.114:0/2720056714 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573e92229f0 con 0x5573e9278e90 2024-05-27T07:47:56.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.716+0000 7f62b6ffd700 1 --2- 172.21.15.114:0/2720056714 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5573e9278e90 0x5573e927b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.717+0000 7f6296ffd700 10 client.17382 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:56.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.717+0000 7f62b6ffd700 1 --2- 172.21.15.114:0/2720056714 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5573e9278e90 0x5573e927b2b0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bc52b426c22ed74 server_cookie=13dd571d818f9d0e in_seq=0 out_seq=0 2024-05-27T07:47:56.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f6296ffd700 1 -- 172.21.15.114:0/2720056714 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5573e92229f0 con 0x5573e9278e90 2024-05-27T07:47:56.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f6296ffd700 10 client.17382 handle_command_reply: tid=0 2024-05-27T07:47:56.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f62cc278b80 1 client.17382 shutdown 2024-05-27T07:47:56.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f62cc278b80 1 -- 172.21.15.114:0/2720056714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9257390 msgr2=0x5573e9269620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:56.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2720056714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9257390 0x5573e9269620 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f62b000afa0 tx=0x7f62b0004460 comp rx=0 tx=0).stop 2024-05-27T07:47:56.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f62cc278b80 1 -- 172.21.15.114:0/2720056714 shutdown_connections 2024-05-27T07:47:56.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2720056714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9257390 0x5573e9269620 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2720056714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e90d8e40 0x5573e9268ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2720056714 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5573e9278e90 0x5573e927b2b0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2720056714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9256a40 0x5573e92690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f62cc278b80 1 -- 172.21.15.114:0/2720056714 >> 172.21.15.114:0/2720056714 conn(0x5573e9133af0 msgr2=0x5573e9253a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:56.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f6296ffd700 0 client.17382 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:56.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f62cc278b80 1 -- 172.21.15.114:0/2720056714 shutdown_connections 2024-05-27T07:47:56.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f62cc278b80 1 -- 172.21.15.114:0/2720056714 wait complete. 2024-05-27T07:47:56.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.718+0000 7f62cc278b80 20 client.17382 trim_cache size 0 max 16384 2024-05-27T07:47:56.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.738+0000 7f62cc278b80 1 Processor -- start 2024-05-27T07:47:56.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.738+0000 7f62cc278b80 1 -- start start 2024-05-27T07:47:56.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62cc278b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e90d8e40 0x5573e926ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62cc278b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9256a40 0x5573e9269460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62cc278b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9257390 0x5573e9274f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62cc278b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81ae440 con 0x5573e9256a40 2024-05-27T07:47:56.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62cc278b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81aeb10 con 0x5573e9257390 2024-05-27T07:47:56.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62cc278b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81ac290 con 0x5573e90d8e40 2024-05-27T07:47:56.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62b67fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9256a40 0x5573e9269460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62b67fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9256a40 0x5573e9269460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34624/0 (socket says 172.21.15.114:34624) 2024-05-27T07:47:56.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62b67fc700 1 -- 172.21.15.114:0/2704804413 learned_addr learned my addr 172.21.15.114:0/2704804413 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:56.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62b6ffd700 1 --2- 172.21.15.114:0/2704804413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e90d8e40 0x5573e926ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62b67fc700 1 -- 172.21.15.114:0/2704804413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e90d8e40 msgr2=0x5573e926ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:56.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62b77fe700 1 --2- 172.21.15.114:0/2704804413 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9257390 0x5573e9274f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62b67fc700 1 --2- 172.21.15.114:0/2704804413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e90d8e40 0x5573e926ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62b67fc700 1 -- 172.21.15.114:0/2704804413 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9257390 msgr2=0x5573e9274f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:56.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62b67fc700 1 --2- 172.21.15.114:0/2704804413 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9257390 0x5573e9274f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62b67fc700 1 -- 172.21.15.114:0/2704804413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573e92699a0 con 0x5573e9256a40 2024-05-27T07:47:56.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f62b67fc700 1 --2- 172.21.15.114:0/2704804413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9256a40 0x5573e9269460 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0a9d20 tx=0x7f62ac0a9790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa719bd267967e5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:56.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f629d7fa700 1 -- 172.21.15.114:0/2704804413 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac0c4020 con 0x5573e9256a40 2024-05-27T07:47:56.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f629d7fa700 1 -- 172.21.15.114:0/2704804413 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62ac0aa720 con 0x5573e9256a40 2024-05-27T07:47:56.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.739+0000 7f629d7fa700 1 -- 172.21.15.114:0/2704804413 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac0aa4e0 con 0x5573e9256a40 2024-05-27T07:47:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.740+0000 7f62cc278b80 1 -- 172.21.15.114:0/2704804413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9256a40 msgr2=0x5573e9269460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.740+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2704804413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9256a40 0x5573e9269460 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0a9d20 tx=0x7f62ac0a9790 comp rx=0 tx=0).stop 2024-05-27T07:47:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.740+0000 7f62cc278b80 1 -- 172.21.15.114:0/2704804413 shutdown_connections 2024-05-27T07:47:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.740+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2704804413 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9257390 0x5573e9274f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.740+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2704804413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e90d8e40 0x5573e926ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.740+0000 7f62cc278b80 1 --2- 172.21.15.114:0/2704804413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e9256a40 0x5573e9269460 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.740+0000 7f62cc278b80 1 -- 172.21.15.114:0/2704804413 >> 172.21.15.114:0/2704804413 conn(0x5573e9133af0 msgr2=0x5573e92560a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.740+0000 7f62cc278b80 1 -- 172.21.15.114:0/2704804413 shutdown_connections 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.740+0000 7f62cc278b80 1 -- 172.21.15.114:0/2704804413 wait complete. 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.741+0000 7f62cc278b80 1 -- start start 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.741+0000 7f62cc278b80 1 Processor -- start 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.741+0000 7f62cc278b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e90d8e40 0x5573e9271900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.741+0000 7f62cc278b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e9256a40 0x5573e9265850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.741+0000 7f62cc278b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9257390 0x5573e9265d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.741+0000 7f62cc278b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81ac900 con 0x5573e90d8e40 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.741+0000 7f62cc278b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81add70 con 0x5573e9257390 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.741+0000 7f62cc278b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81acfd0 con 0x5573e9256a40 2024-05-27T07:47:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.741+0000 7f62cc278b80 10 client.0 _ll_register_callbacks cb 0x5573e9110920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.741+0000 7f62b6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e90d8e40 0x5573e9271900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.741+0000 7f62b6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e90d8e40 0x5573e9271900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34628/0 (socket says 172.21.15.114:34628) 2024-05-27T07:47:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.741+0000 7f62b6ffd700 1 -- 172.21.15.114:0/1412293875 learned_addr learned my addr 172.21.15.114:0/1412293875 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.741+0000 7f62b67fc700 1 --2- 172.21.15.114:0/1412293875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e9256a40 0x5573e9265850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f62b77fe700 1 --2- 172.21.15.114:0/1412293875 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9257390 0x5573e9265d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f62b6ffd700 1 -- 172.21.15.114:0/1412293875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e9256a40 msgr2=0x5573e9265850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f62b6ffd700 1 --2- 172.21.15.114:0/1412293875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e9256a40 0x5573e9265850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f62b6ffd700 1 -- 172.21.15.114:0/1412293875 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9257390 msgr2=0x5573e9265d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f62b6ffd700 1 --2- 172.21.15.114:0/1412293875 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9257390 0x5573e9265d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f62b6ffd700 1 -- 172.21.15.114:0/1412293875 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573e9266530 con 0x5573e90d8e40 2024-05-27T07:47:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f62b6ffd700 1 --2- 172.21.15.114:0/1412293875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e90d8e40 0x5573e9271900 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x5573e927b9c0 tx=0x7f62a8013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b985714eea5d9d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f6296ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f6296ffd700 1 -- 172.21.15.114:0/1412293875 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a801e020 con 0x5573e90d8e40 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f6296ffd700 1 -- 172.21.15.114:0/1412293875 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7f62a801e1c0 con 0x5573e90d8e40 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f62cc278b80 1 -- 172.21.15.114:0/1412293875 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62ac0aa720 con 0x5573e90d8e40 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f6296ffd700 1 -- 172.21.15.114:0/1412293875 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f62a803d050 con 0x5573e90d8e40 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f62cc278b80 10 client.17388 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f62cc278b80 1 -- 172.21.15.114:0/1412293875 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5573e92e38e0 con 0x5573e90d8e40 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.742+0000 7f6296ffd700 1 -- 172.21.15.114:0/1412293875 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62ac0aa720 con 0x5573e90d8e40 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.743+0000 7f6296ffd700 1 -- 172.21.15.114:0/1412293875 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7f62a8031030 con 0x5573e90d8e40 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.743+0000 7f62cc278b80 10 client.17388 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.743+0000 7f62cc278b80 10 client.17388 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.743+0000 7f62cc278b80 20 client.17388 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.743+0000 7f62cc278b80 1 --2- 172.21.15.114:0/1412293875 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5573e92e4040 0x5573e92e6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.743+0000 7f62cc278b80 4 client.17388 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.743+0000 7f62cc278b80 1 -- 172.21.15.114:0/1412293875 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5573e9187be0 con 0x5573e92e4040 2024-05-27T07:47:56.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.743+0000 7f62b67fc700 1 --2- 172.21.15.114:0/1412293875 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5573e92e4040 0x5573e92e6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:56.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.743+0000 7f62b67fc700 1 --2- 172.21.15.114:0/1412293875 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5573e92e4040 0x5573e92e6460 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4c8fa90ba53b99c server_cookie=629b83071502226a in_seq=0 out_seq=0 2024-05-27T07:47:56.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.743+0000 7f6296ffd700 10 client.17388 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:56.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.744+0000 7f6296ffd700 1 -- 172.21.15.114:0/1412293875 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5573e9187be0 con 0x5573e92e4040 2024-05-27T07:47:56.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.744+0000 7f6296ffd700 10 client.17388 handle_command_reply: tid=0 2024-05-27T07:47:56.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.744+0000 7f62cc278b80 1 client.17388 shutdown 2024-05-27T07:47:56.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.745+0000 7f62cc278b80 1 -- 172.21.15.114:0/1412293875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e90d8e40 msgr2=0x5573e9271900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:56.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.745+0000 7f62cc278b80 1 --2- 172.21.15.114:0/1412293875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e90d8e40 0x5573e9271900 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x5573e927b9c0 tx=0x7f62a8013040 comp rx=0 tx=0).stop 2024-05-27T07:47:56.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.745+0000 7f62cc278b80 1 -- 172.21.15.114:0/1412293875 shutdown_connections 2024-05-27T07:47:56.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.745+0000 7f62cc278b80 1 --2- 172.21.15.114:0/1412293875 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5573e92e4040 0x5573e92e6460 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.745+0000 7f62cc278b80 1 --2- 172.21.15.114:0/1412293875 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5573e9257390 0x5573e9265d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.745+0000 7f62cc278b80 1 --2- 172.21.15.114:0/1412293875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5573e90d8e40 0x5573e9271900 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.745+0000 7f62cc278b80 1 --2- 172.21.15.114:0/1412293875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573e9256a40 0x5573e9265850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.745+0000 7f6296ffd700 0 client.17388 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:56.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.745+0000 7f62cc278b80 1 -- 172.21.15.114:0/1412293875 >> 172.21.15.114:0/1412293875 conn(0x5573e9133af0 msgr2=0x5573e9263660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:56.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.745+0000 7f62cc278b80 1 -- 172.21.15.114:0/1412293875 shutdown_connections 2024-05-27T07:47:56.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.745+0000 7f62cc278b80 1 -- 172.21.15.114:0/1412293875 wait complete. 2024-05-27T07:47:56.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.745+0000 7f62cc278b80 20 client.17388 trim_cache size 0 max 16384 2024-05-27T07:47:56.750 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:47:56.750 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:47:56.750 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:47:56.750 INFO:teuthology.orchestra.run.smithi114.stdout: "f97396ae-03e4-41e5-947e-8684f8765843": { 2024-05-27T07:47:56.750 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:47:56.750 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "f97396ae-03e4-41e5-947e-8684f8765843", 2024-05-27T07:47:56.750 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:47:56.750 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:47:56.751 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:47:56.751 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:47:56.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.746+0000 7f62bd863700 1 -- 172.21.15.114:0/2498682251 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f62a00b29f0 msgr2=0x7f62a00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:56.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.746+0000 7f62bd863700 1 --2- 172.21.15.114:0/2498682251 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f62a00b29f0 0x7f62a00b4f00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009d20 tx=0x7f62a8009260 comp rx=0 tx=0).stop 2024-05-27T07:47:56.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.746+0000 7f62bd863700 1 -- 172.21.15.114:0/2498682251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62b8107de0 msgr2=0x7f62b819c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:56.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.746+0000 7f62bd863700 1 --2- 172.21.15.114:0/2498682251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62b8107de0 0x7f62b819c1c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0098e0 tx=0x7f62ac0045e0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.746+0000 7f62bd863700 1 -- 172.21.15.114:0/2498682251 shutdown_connections 2024-05-27T07:47:56.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.746+0000 7f62bd863700 1 --2- 172.21.15.114:0/2498682251 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f62a00b29f0 0x7f62a00b4f00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.746+0000 7f62bd863700 1 --2- 172.21.15.114:0/2498682251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62b8108730 0x7f62b819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.746+0000 7f62bd863700 1 --2- 172.21.15.114:0/2498682251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62b81034b0 0x7f62b819bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.746+0000 7f62bd863700 1 --2- 172.21.15.114:0/2498682251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62b8107de0 0x7f62b819c1c0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:56.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.746+0000 7f62bd863700 1 -- 172.21.15.114:0/2498682251 >> 172.21.15.114:0/2498682251 conn(0x7f62b80fedb0 msgr2=0x7f62b81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:56.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.746+0000 7f62bd863700 1 -- 172.21.15.114:0/2498682251 shutdown_connections 2024-05-27T07:47:56.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:47:56.746+0000 7f62bd863700 1 -- 172.21.15.114:0/2498682251 wait complete. 2024-05-27T07:47:57.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 835967 11.46 MB/sec execute 249 sec latency 21.091 ms 2024-05-27T07:47:57.470 INFO:tasks.cephfs.filesystem:scrub status for tag:f97396ae-03e4-41e5-947e-8684f8765843 - {'path': '/', 'tag': 'f97396ae-03e4-41e5-947e-8684f8765843', 'options': 'recursive,force'} 2024-05-27T07:47:57.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:57 smithi163 ceph-mon[36451]: pgmap v264: 129 pgs: 129 active+clean; 79 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 426 op/s 2024-05-27T07:47:57.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:57 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:47:57.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:57 smithi186 ceph-mon[36647]: pgmap v264: 129 pgs: 129 active+clean; 79 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 426 op/s 2024-05-27T07:47:57.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:57 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:47:57.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:57 smithi114 ceph-mon[32038]: pgmap v264: 129 pgs: 129 active+clean; 79 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 426 op/s 2024-05-27T07:47:57.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:57 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:47:58.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 838106 11.47 MB/sec execute 250 sec latency 140.655 ms 2024-05-27T07:47:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.411+0000 7fe5e8dc7640 1 -- 172.21.15.114:0/2780221930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e4108ba0 msgr2=0x7fe5e4102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.411+0000 7fe5e8dc7640 1 --2- 172.21.15.114:0/2780221930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e4108ba0 0x7fe5e4102740 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4009900 tx=0x7fe5d402a410 comp rx=0 tx=0).stop 2024-05-27T07:47:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.411+0000 7fe5e8dc7640 1 -- 172.21.15.114:0/2780221930 shutdown_connections 2024-05-27T07:47:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.411+0000 7fe5e8dc7640 1 --2- 172.21.15.114:0/2780221930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5e41032f0 0x7fe5e410f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.411+0000 7fe5e8dc7640 1 --2- 172.21.15.114:0/2780221930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5e4109040 0x7fe5e4102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.411+0000 7fe5e8dc7640 1 --2- 172.21.15.114:0/2780221930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e4108ba0 0x7fe5e4102740 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.411+0000 7fe5e8dc7640 1 -- 172.21.15.114:0/2780221930 >> 172.21.15.114:0/2780221930 conn(0x7fe5e4078270 msgr2=0x7fe5e40786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.411+0000 7fe5e8dc7640 1 -- 172.21.15.114:0/2780221930 shutdown_connections 2024-05-27T07:47:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.411+0000 7fe5e8dc7640 1 -- 172.21.15.114:0/2780221930 wait complete. 2024-05-27T07:47:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.412+0000 7fe5e8dc7640 1 Processor -- start 2024-05-27T07:47:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.412+0000 7fe5e8dc7640 1 -- start start 2024-05-27T07:47:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.412+0000 7fe5e8dc7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5e41032f0 0x7fe5e419b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.412+0000 7fe5e8dc7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5e4108ba0 0x7fe5e419bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.412+0000 7fe5e8dc7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e4109040 0x7fe5e419c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.412+0000 7fe5e8dc7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e4112d90 con 0x7fe5e4109040 2024-05-27T07:47:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.412+0000 7fe5e8dc7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e4112c10 con 0x7fe5e41032f0 2024-05-27T07:47:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.412+0000 7fe5e8dc7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e4112f10 con 0x7fe5e4108ba0 2024-05-27T07:47:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.412+0000 7fe5e2d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e4109040 0x7fe5e419c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.412+0000 7fe5e2d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e4109040 0x7fe5e419c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34644/0 (socket says 172.21.15.114:34644) 2024-05-27T07:47:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.412+0000 7fe5e2d9d640 1 -- 172.21.15.114:0/1808666331 learned_addr learned my addr 172.21.15.114:0/1808666331 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.412+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5e4108ba0 0x7fe5e419bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.412+0000 7fe5e259c640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5e41032f0 0x7fe5e419b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.413+0000 7fe5e2d9d640 1 -- 172.21.15.114:0/1808666331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5e4108ba0 msgr2=0x7fe5e419bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.413+0000 7fe5e2d9d640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5e4108ba0 0x7fe5e419bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.413+0000 7fe5e2d9d640 1 -- 172.21.15.114:0/1808666331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5e41032f0 msgr2=0x7fe5e419b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.413+0000 7fe5e2d9d640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5e41032f0 0x7fe5e419b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.413+0000 7fe5e2d9d640 1 -- 172.21.15.114:0/1808666331 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e41a6a40 con 0x7fe5e4109040 2024-05-27T07:47:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.413+0000 7fe5e2d9d640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e4109040 0x7fe5e419c1b0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00eb10 tx=0x7fe5dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.413+0000 7fe5dbfff640 1 -- 172.21.15.114:0/1808666331 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc004b20 con 0x7fe5e4109040 2024-05-27T07:47:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.413+0000 7fe5dbfff640 1 -- 172.21.15.114:0/1808666331 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5dc004cc0 con 0x7fe5e4109040 2024-05-27T07:47:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.413+0000 7fe5e8dc7640 1 -- 172.21.15.114:0/1808666331 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e41a6d30 con 0x7fe5e4109040 2024-05-27T07:47:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.413+0000 7fe5dbfff640 1 -- 172.21.15.114:0/1808666331 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc010670 con 0x7fe5e4109040 2024-05-27T07:47:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.413+0000 7fe5e8dc7640 1 -- 172.21.15.114:0/1808666331 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5e41a7220 con 0x7fe5e4109040 2024-05-27T07:47:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.415+0000 7fe5efc7c740 1 Processor -- start 2024-05-27T07:47:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.416+0000 7fe5dbfff640 1 -- 172.21.15.114:0/1808666331 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fe5dc010810 con 0x7fe5e4109040 2024-05-27T07:47:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.416+0000 7fe5efc7c740 1 -- start start 2024-05-27T07:47:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.416+0000 7fe5efc7c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 0x55f99bcda3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.416+0000 7fe5efc7c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9102f0 0x55f99bcda910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.416+0000 7fe5efc7c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bcdae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.416+0000 7fe5efc7c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f99bc4cc30 con 0x55f99b9102f0 2024-05-27T07:47:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.416+0000 7fe5efc7c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f99bc3c7d0 con 0x55f99b9184d0 2024-05-27T07:47:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.416+0000 7fe5efc7c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f99bc40810 con 0x55f99b9391b0 2024-05-27T07:47:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5e259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9102f0 0x55f99bcda910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d9d9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bcdae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d9d9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bcdae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:33450/0 (socket says 172.21.15.114:33450) 2024-05-27T07:47:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d9d9b640 1 -- 172.21.15.114:0/2865893921 learned_addr learned my addr 172.21.15.114:0/2865893921 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/2865893921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 0x55f99bcda3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d9d9b640 1 -- 172.21.15.114:0/2865893921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 msgr2=0x55f99bcda3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/2865893921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 0x55f99bcda3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d9d9b640 1 -- 172.21.15.114:0/2865893921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9102f0 msgr2=0x55f99bcda910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/2865893921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9102f0 0x55f99bcda910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d9d9b640 1 -- 172.21.15.114:0/2865893921 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f99b951210 con 0x55f99b9184d0 2024-05-27T07:47:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5dbfff640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe5d00b2950 0x7fe5d00b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe5d00b2950 0x7fe5d00b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5dbfff640 1 -- 172.21.15.114:0/1808666331 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fe5dc0e2f30 con 0x7fe5e4109040 2024-05-27T07:47:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/2865893921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bcdae50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55f99bcd7540 tx=0x7fe5cc0096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3da13a5f517a809f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d8d99640 1 -- 172.21.15.114:0/2865893921 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc004220 con 0x55f99b9184d0 2024-05-27T07:47:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d8d99640 1 -- 172.21.15.114:0/2865893921 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5cc004440 con 0x55f99b9184d0 2024-05-27T07:47:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.417+0000 7fe5d8d99640 1 -- 172.21.15.114:0/2865893921 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc003e00 con 0x55f99b9184d0 2024-05-27T07:47:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.418+0000 7fe5efc7c740 1 -- 172.21.15.114:0/2865893921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 msgr2=0x55f99bcdae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.418+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/2865893921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bcdae50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55f99bcd7540 tx=0x7fe5cc0096c0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.418+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe5d00b2950 0x7fe5d00b4e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc020000 tx=0x7fe5cc020940 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.418+0000 7fe5efc7c740 1 -- 172.21.15.114:0/2865893921 shutdown_connections 2024-05-27T07:47:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.418+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/2865893921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bcdae50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.418+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/2865893921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9102f0 0x55f99bcda910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.418+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/2865893921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 0x55f99bcda3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.418+0000 7fe5efc7c740 1 -- 172.21.15.114:0/2865893921 >> 172.21.15.114:0/2865893921 conn(0x55f99b92d300 msgr2=0x55f99bcd6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.418+0000 7fe5efc7c740 1 -- 172.21.15.114:0/2865893921 shutdown_connections 2024-05-27T07:47:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.418+0000 7fe5efc7c740 1 -- 172.21.15.114:0/2865893921 wait complete. 2024-05-27T07:47:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5efc7c740 1 -- start start 2024-05-27T07:47:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5efc7c740 1 Processor -- start 2024-05-27T07:47:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5efc7c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9102f0 0x55f99bce4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5efc7c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bce4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5efc7c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 0x55f99bcea7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5efc7c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f99b922790 con 0x55f99b9102f0 2024-05-27T07:47:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5efc7c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f99bbf8890 con 0x55f99b9184d0 2024-05-27T07:47:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5efc7c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f99bce74a0 con 0x55f99b9391b0 2024-05-27T07:47:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5e259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9102f0 0x55f99bce4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5e259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9102f0 0x55f99bce4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34670/0 (socket says 172.21.15.114:34670) 2024-05-27T07:47:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5e259c640 1 -- 172.21.15.114:0/2181824029 learned_addr learned my addr 172.21.15.114:0/2181824029 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5efc7c740 10 client.0 _ll_register_callbacks cb 0x55f99b950a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bce4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5e2d9d640 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 0x55f99bcea7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5e259c640 1 -- 172.21.15.114:0/2181824029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 msgr2=0x55f99bcea7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.419+0000 7fe5e259c640 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 0x55f99bcea7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5e259c640 1 -- 172.21.15.114:0/2181824029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 msgr2=0x55f99bce4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5e259c640 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bce4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5e259c640 1 -- 172.21.15.114:0/2181824029 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f99ba1d620 con 0x55f99b9102f0 2024-05-27T07:47:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bce4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5e259c640 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9102f0 0x55f99bce4a60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4103880 tx=0x7fe5d4032eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599198856523c990 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:47:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5c9ffb640 1 -- 172.21.15.114:0/2181824029 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4035040 con 0x55f99b9102f0 2024-05-27T07:47:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5c9ffb640 1 -- 172.21.15.114:0/2181824029 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fe5d40317e0 con 0x55f99b9102f0 2024-05-27T07:47:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5efc7c740 1 -- 172.21.15.114:0/2181824029 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5cc004440 con 0x55f99b9102f0 2024-05-27T07:47:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5c9ffb640 1 -- 172.21.15.114:0/2181824029 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fe5d404f050 con 0x55f99b9102f0 2024-05-27T07:47:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5efc7c740 10 client.17400 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5efc7c740 1 -- 172.21.15.114:0/2181824029 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55f99bcf36e0 con 0x55f99b9102f0 2024-05-27T07:47:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5c9ffb640 1 -- 172.21.15.114:0/2181824029 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5cc004440 con 0x55f99b9102f0 2024-05-27T07:47:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.420+0000 7fe5c9ffb640 1 -- 172.21.15.114:0/2181824029 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7fe5d4004750 con 0x55f99b9102f0 2024-05-27T07:47:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.421+0000 7fe5efc7c740 10 client.17400 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.421+0000 7fe5efc7c740 10 client.17400 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.421+0000 7fe5efc7c740 20 client.17400 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.421+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f99b8f7d80 0x55f99bcf6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.421+0000 7fe5efc7c740 4 client.17400 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:47:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.421+0000 7fe5efc7c740 1 -- 172.21.15.114:0/2181824029 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f99bc4eb80 con 0x55f99b8f7d80 2024-05-27T07:47:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.421+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f99b8f7d80 0x55f99bcf6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.421+0000 7fe5c9ffb640 10 client.17400 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.421+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f99b8f7d80 0x55f99bcf6c40 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e2e6ab77a0826bf server_cookie=ccfebfd09dce8688 in_seq=0 out_seq=0 2024-05-27T07:47:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5c9ffb640 1 -- 172.21.15.114:0/2181824029 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f99bc4eb80 con 0x55f99b8f7d80 2024-05-27T07:47:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5c9ffb640 10 client.17400 handle_command_reply: tid=0 2024-05-27T07:47:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5efc7c740 1 client.17400 shutdown 2024-05-27T07:47:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5efc7c740 1 -- 172.21.15.114:0/2181824029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9102f0 msgr2=0x55f99bce4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9102f0 0x55f99bce4a60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4103880 tx=0x7fe5d4032eb0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5efc7c740 1 -- 172.21.15.114:0/2181824029 shutdown_connections 2024-05-27T07:47:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f99b8f7d80 0x55f99bcf6c40 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 0x55f99bcea7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bce4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/2181824029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9102f0 0x55f99bce4a60 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5c9ffb640 0 client.17400 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5efc7c740 1 -- 172.21.15.114:0/2181824029 >> 172.21.15.114:0/2181824029 conn(0x55f99b92d300 msgr2=0x55f99bcdc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5efc7c740 1 -- 172.21.15.114:0/2181824029 shutdown_connections 2024-05-27T07:47:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5efc7c740 1 -- 172.21.15.114:0/2181824029 wait complete. 2024-05-27T07:47:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.422+0000 7fe5efc7c740 20 client.17400 trim_cache size 0 max 16384 2024-05-27T07:47:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.437+0000 7fe5efc7c740 1 Processor -- start 2024-05-27T07:47:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.437+0000 7fe5efc7c740 1 -- start start 2024-05-27T07:47:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5efc7c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9102f0 0x55f99bce69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5efc7c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bcfc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5efc7c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9391b0 0x55f99bcfc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5efc7c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41b1410 con 0x55f99b9391b0 2024-05-27T07:47:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5efc7c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41b1ae0 con 0x55f99b9184d0 2024-05-27T07:47:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5efc7c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41af260 con 0x55f99b9102f0 2024-05-27T07:47:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5e2d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9391b0 0x55f99bcfc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5e2d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9391b0 0x55f99bcfc980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34682/0 (socket says 172.21.15.114:34682) 2024-05-27T07:47:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5e2d9d640 1 -- 172.21.15.114:0/153181484 learned_addr learned my addr 172.21.15.114:0/153181484 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5e259c640 1 --2- 172.21.15.114:0/153181484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9102f0 0x55f99bce69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/153181484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bcfc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5e2d9d640 1 -- 172.21.15.114:0/153181484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9102f0 msgr2=0x55f99bce69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5e2d9d640 1 --2- 172.21.15.114:0/153181484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9102f0 0x55f99bce69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5e2d9d640 1 -- 172.21.15.114:0/153181484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 msgr2=0x55f99bcfc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5e2d9d640 1 --2- 172.21.15.114:0/153181484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bcfc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5e2d9d640 1 -- 172.21.15.114:0/153181484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f99bcfcec0 con 0x55f99b9391b0 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5e2d9d640 1 --2- 172.21.15.114:0/153181484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9391b0 0x55f99bcfc980 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc0a9b60 tx=0x7fe5dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930ae160927be9c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.438+0000 7fe5cbfff640 1 -- 172.21.15.114:0/153181484 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc0a9e60 con 0x55f99b9391b0 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.439+0000 7fe5cbfff640 1 -- 172.21.15.114:0/153181484 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5dc0ae210 con 0x55f99b9391b0 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.439+0000 7fe5cbfff640 1 -- 172.21.15.114:0/153181484 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc010b00 con 0x55f99b9391b0 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.439+0000 7fe5efc7c740 1 -- 172.21.15.114:0/153181484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9391b0 msgr2=0x55f99bcfc980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.439+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/153181484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9391b0 0x55f99bcfc980 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc0a9b60 tx=0x7fe5dc017000 comp rx=0 tx=0).stop 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.439+0000 7fe5efc7c740 1 -- 172.21.15.114:0/153181484 shutdown_connections 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.439+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/153181484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9391b0 0x55f99bcfc980 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.439+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/153181484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9184d0 0x55f99bcfc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.439+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/153181484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9102f0 0x55f99bce69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5efc7c740 1 -- 172.21.15.114:0/153181484 >> 172.21.15.114:0/153181484 conn(0x55f99b92d300 msgr2=0x55f99bcdac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5efc7c740 1 -- 172.21.15.114:0/153181484 shutdown_connections 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5efc7c740 1 -- 172.21.15.114:0/153181484 wait complete. 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5efc7c740 1 -- start start 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5efc7c740 1 Processor -- start 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5efc7c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9102f0 0x55f99bced0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5efc7c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9184d0 0x55f99bced610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5efc7c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 0x55f99bd5e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5efc7c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41af8d0 con 0x55f99b9184d0 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5efc7c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41b0d40 con 0x55f99b9102f0 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5efc7c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41affa0 con 0x55f99b9391b0 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5efc7c740 10 client.0 _ll_register_callbacks cb 0x7fe5d4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5d9d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9184d0 0x55f99bced610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5d9d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9184d0 0x55f99bced610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34690/0 (socket says 172.21.15.114:34690) 2024-05-27T07:47:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5d9d9b640 1 -- 172.21.15.114:0/3265483328 learned_addr learned my addr 172.21.15.114:0/3265483328 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5e259c640 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9102f0 0x55f99bced0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.440+0000 7fe5e2d9d640 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 0x55f99bd5e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5e259c640 1 -- 172.21.15.114:0/3265483328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 msgr2=0x55f99bd5e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5e259c640 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 0x55f99bd5e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5e259c640 1 -- 172.21.15.114:0/3265483328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9184d0 msgr2=0x55f99bced610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5e259c640 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9184d0 0x55f99bced610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5e259c640 1 -- 172.21.15.114:0/3265483328 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f99bd5e850 con 0x55f99b9102f0 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9184d0 0x55f99bced610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5e259c640 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9102f0 0x55f99bced0d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4033000 tx=0x7fe5d40048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f69249e4a21d189b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5c9ffb640 1 -- 172.21.15.114:0/3265483328 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4058060 con 0x55f99b9102f0 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5efc7c740 1 -- 172.21.15.114:0/3265483328 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5dc0ae210 con 0x55f99b9102f0 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5c9ffb640 1 -- 172.21.15.114:0/3265483328 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9466+0+0 (secure 0 0 0) 0x7fe5d40317f0 con 0x55f99b9102f0 2024-05-27T07:47:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5c9ffb640 1 -- 172.21.15.114:0/3265483328 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fe5d405f050 con 0x55f99b9102f0 2024-05-27T07:47:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5efc7c740 10 client.26858 fetch_fsmap learned FSMap version 46 2024-05-27T07:47:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.441+0000 7fe5efc7c740 1 -- 172.21.15.114:0/3265483328 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55f99bd5fa30 con 0x55f99b9102f0 2024-05-27T07:47:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.442+0000 7fe5c9ffb640 1 -- 172.21.15.114:0/3265483328 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5dc0ae210 con 0x55f99b9102f0 2024-05-27T07:47:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.442+0000 7fe5c9ffb640 1 -- 172.21.15.114:0/3265483328 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7fe5d4005150 con 0x55f99b9102f0 2024-05-27T07:47:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.442+0000 7fe5efc7c740 10 client.26858 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:47:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.442+0000 7fe5efc7c740 10 client.26858 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:47:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.442+0000 7fe5efc7c740 20 client.26858 populate_metadata read hostname 'smithi114' 2024-05-27T07:47:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.442+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f99b8cd1c0 0x55f99bd62160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:47:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.442+0000 7fe5efc7c740 4 client.26858 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:47:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.442+0000 7fe5efc7c740 1 -- 172.21.15.114:0/3265483328 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f99bc48360 con 0x55f99b8cd1c0 2024-05-27T07:47:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.442+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f99b8cd1c0 0x55f99bd62160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:47:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.443+0000 7fe5d9d9b640 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f99b8cd1c0 0x55f99bd62160 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8acc9ba252cb3cbe server_cookie=3957c6e17e259fcd in_seq=0 out_seq=0 2024-05-27T07:47:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.443+0000 7fe5c9ffb640 10 client.26858 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:58.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5c9ffb640 1 -- 172.21.15.114:0/3265483328 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f99bc48360 con 0x55f99b8cd1c0 2024-05-27T07:47:58.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5c9ffb640 10 client.26858 handle_command_reply: tid=0 2024-05-27T07:47:58.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5efc7c740 1 client.26858 shutdown 2024-05-27T07:47:58.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5efc7c740 1 -- 172.21.15.114:0/3265483328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9102f0 msgr2=0x55f99bced0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:58.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9102f0 0x55f99bced0d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4033000 tx=0x7fe5d40048f0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5efc7c740 1 -- 172.21.15.114:0/3265483328 shutdown_connections 2024-05-27T07:47:58.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f99b8cd1c0 0x55f99bd62160 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5c9ffb640 0 client.26858 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:47:58.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f99b9391b0 0x55f99bd5e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f99b9184d0 0x55f99bced610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5efc7c740 1 --2- 172.21.15.114:0/3265483328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f99b9102f0 0x55f99bced0d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5efc7c740 1 -- 172.21.15.114:0/3265483328 >> 172.21.15.114:0/3265483328 conn(0x55f99b92d300 msgr2=0x55f99bcdac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5efc7c740 1 -- 172.21.15.114:0/3265483328 shutdown_connections 2024-05-27T07:47:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5efc7c740 1 -- 172.21.15.114:0/3265483328 wait complete. 2024-05-27T07:47:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.814+0000 7fe5efc7c740 20 client.26858 trim_cache size 0 max 16384 2024-05-27T07:47:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:47:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:47:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:47:58.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.815+0000 7fe5e8dc7640 1 -- 172.21.15.114:0/1808666331 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe5d00b2950 msgr2=0x7fe5d00b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:58.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.815+0000 7fe5e8dc7640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe5d00b2950 0x7fe5d00b4e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc020000 tx=0x7fe5cc020940 comp rx=0 tx=0).stop 2024-05-27T07:47:58.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.815+0000 7fe5e8dc7640 1 -- 172.21.15.114:0/1808666331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e4109040 msgr2=0x7fe5e419c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:47:58.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.815+0000 7fe5e8dc7640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e4109040 0x7fe5e419c1b0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00eb10 tx=0x7fe5dc00c2d0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.815+0000 7fe5e8dc7640 1 -- 172.21.15.114:0/1808666331 shutdown_connections 2024-05-27T07:47:58.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.815+0000 7fe5e8dc7640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe5d00b2950 0x7fe5d00b4e60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.815+0000 7fe5e8dc7640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e4109040 0x7fe5e419c1b0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.815+0000 7fe5e8dc7640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5e4108ba0 0x7fe5e419bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.815+0000 7fe5e8dc7640 1 --2- 172.21.15.114:0/1808666331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5e41032f0 0x7fe5e419b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:47:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.815+0000 7fe5e8dc7640 1 -- 172.21.15.114:0/1808666331 >> 172.21.15.114:0/1808666331 conn(0x7fe5e4078270 msgr2=0x7fe5e4104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:47:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.816+0000 7fe5e8dc7640 1 -- 172.21.15.114:0/1808666331 shutdown_connections 2024-05-27T07:47:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:47:58.816+0000 7fe5e8dc7640 1 -- 172.21.15.114:0/1808666331 wait complete. 2024-05-27T07:47:59.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 839062 11.43 MB/sec execute 251 sec latency 215.896 ms 2024-05-27T07:47:59.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:47:59 smithi163 ceph-mon[36451]: pgmap v265: 129 pgs: 129 active+clean; 79 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-05-27T07:47:59.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:47:59 smithi186 ceph-mon[36647]: pgmap v265: 129 pgs: 129 active+clean; 79 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-05-27T07:47:59.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:47:59 smithi114 ceph-mon[32038]: pgmap v265: 129 pgs: 129 active+clean; 79 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-05-27T07:48:00.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 841602 11.45 MB/sec execute 252 sec latency 39.754 ms 2024-05-27T07:48:01.229 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fc6198d2700 time 2024-05-27T07:48:00.887819+0000 2024-05-27T07:48:01.229 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:01.229 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:01.229 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc62c117ba6] 2024-05-27T07:48:01.229 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc62c117dd1] 2024-05-27T07:48:01.230 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c0162a123d] 2024-05-27T07:48:01.230 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c01628c971] 2024-05-27T07:48:01.230 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c01628afa0] 2024-05-27T07:48:01.230 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c01628b651] 2024-05-27T07:48:01.230 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c016220347] 2024-05-27T07:48:01.230 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c015f1c87e] 2024-05-27T07:48:01.230 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c015f21e94] 2024-05-27T07:48:01.230 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c015f2c969] 2024-05-27T07:48:01.230 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c0162c548c] 2024-05-27T07:48:01.230 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c0162ce9cb] 2024-05-27T07:48:01.231 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 13: (Thread::entry_wrapper()+0x43) [0x7fc62c0e672d] 2024-05-27T07:48:01.231 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 14: (Thread::_entry_func(void*)+0xd) [0x7fc62c0e6749] 2024-05-27T07:48:01.231 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc62ac8a1ca] 2024-05-27T07:48:01.231 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 16: clone() 2024-05-27T07:48:01.231 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:00.889+0000 7fc6198d2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fc6198d2700 time 2024-05-27T07:48:00.887819+0000 2024-05-27T07:48:01.232 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:01.232 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:01.232 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:01.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc62c117ba6] 2024-05-27T07:48:01.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc62c117dd1] 2024-05-27T07:48:01.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c0162a123d] 2024-05-27T07:48:01.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c01628c971] 2024-05-27T07:48:01.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c01628afa0] 2024-05-27T07:48:01.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c01628b651] 2024-05-27T07:48:01.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c016220347] 2024-05-27T07:48:01.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c015f1c87e] 2024-05-27T07:48:01.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c015f21e94] 2024-05-27T07:48:01.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c015f2c969] 2024-05-27T07:48:01.233 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c0162c548c] 2024-05-27T07:48:01.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c0162ce9cb] 2024-05-27T07:48:01.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 13: (Thread::entry_wrapper()+0x43) [0x7fc62c0e672d] 2024-05-27T07:48:01.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 14: (Thread::_entry_func(void*)+0xd) [0x7fc62c0e6749] 2024-05-27T07:48:01.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc62ac8a1ca] 2024-05-27T07:48:01.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 16: clone() 2024-05-27T07:48:01.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:01.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: *** Caught signal (Aborted) ** 2024-05-27T07:48:01.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: in thread 7fc6198d2700 thread_name:md_log_replay 2024-05-27T07:48:01.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:01.234 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc62ac94d20] 2024-05-27T07:48:01.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2: gsignal() 2024-05-27T07:48:01.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 3: abort() 2024-05-27T07:48:01.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc62c117ced] 2024-05-27T07:48:01.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc62c117dd1] 2024-05-27T07:48:01.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c0162a123d] 2024-05-27T07:48:01.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c01628c971] 2024-05-27T07:48:01.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c01628afa0] 2024-05-27T07:48:01.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c01628b651] 2024-05-27T07:48:01.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c016220347] 2024-05-27T07:48:01.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c015f1c87e] 2024-05-27T07:48:01.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c015f21e94] 2024-05-27T07:48:01.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c015f2c969] 2024-05-27T07:48:01.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c0162c548c] 2024-05-27T07:48:01.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c0162ce9cb] 2024-05-27T07:48:01.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 16: (Thread::entry_wrapper()+0x43) [0x7fc62c0e672d] 2024-05-27T07:48:01.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 17: (Thread::_entry_func(void*)+0xd) [0x7fc62c0e6749] 2024-05-27T07:48:01.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc62ac8a1ca] 2024-05-27T07:48:01.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 19: clone() 2024-05-27T07:48:01.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:00.892+0000 7fc6198d2700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:48:01.236 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: in thread 7fc6198d2700 thread_name:md_log_replay 2024-05-27T07:48:01.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:01.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:01.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc62ac94d20] 2024-05-27T07:48:01.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2: gsignal() 2024-05-27T07:48:01.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 3: abort() 2024-05-27T07:48:01.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc62c117ced] 2024-05-27T07:48:01.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc62c117dd1] 2024-05-27T07:48:01.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c0162a123d] 2024-05-27T07:48:01.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c01628c971] 2024-05-27T07:48:01.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c01628afa0] 2024-05-27T07:48:01.237 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c01628b651] 2024-05-27T07:48:01.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c016220347] 2024-05-27T07:48:01.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c015f1c87e] 2024-05-27T07:48:01.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c015f21e94] 2024-05-27T07:48:01.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c015f2c969] 2024-05-27T07:48:01.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c0162c548c] 2024-05-27T07:48:01.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c0162ce9cb] 2024-05-27T07:48:01.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 16: (Thread::entry_wrapper()+0x43) [0x7fc62c0e672d] 2024-05-27T07:48:01.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 17: (Thread::_entry_func(void*)+0xd) [0x7fc62c0e6749] 2024-05-27T07:48:01.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc62ac8a1ca] 2024-05-27T07:48:01.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 19: clone() 2024-05-27T07:48:01.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:48:01.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:01.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: -1> 2024-05-27T07:48:00.889+0000 7fc6198d2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fc6198d2700 time 2024-05-27T07:48:00.887819+0000 2024-05-27T07:48:01.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:01.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:01.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:01.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc62c117ba6] 2024-05-27T07:48:01.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc62c117dd1] 2024-05-27T07:48:01.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c0162a123d] 2024-05-27T07:48:01.239 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c01628c971] 2024-05-27T07:48:01.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c01628afa0] 2024-05-27T07:48:01.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c01628b651] 2024-05-27T07:48:01.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c016220347] 2024-05-27T07:48:01.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c015f1c87e] 2024-05-27T07:48:01.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c015f21e94] 2024-05-27T07:48:01.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c015f2c969] 2024-05-27T07:48:01.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c0162c548c] 2024-05-27T07:48:01.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c0162ce9cb] 2024-05-27T07:48:01.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 13: (Thread::entry_wrapper()+0x43) [0x7fc62c0e672d] 2024-05-27T07:48:01.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 14: (Thread::_entry_func(void*)+0xd) [0x7fc62c0e6749] 2024-05-27T07:48:01.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc62ac8a1ca] 2024-05-27T07:48:01.240 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 16: clone() 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 0> 2024-05-27T07:48:00.892+0000 7fc6198d2700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: in thread 7fc6198d2700 thread_name:md_log_replay 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc62ac94d20] 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2: gsignal() 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 3: abort() 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc62c117ced] 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc62c117dd1] 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c0162a123d] 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c01628c971] 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c01628afa0] 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c01628b651] 2024-05-27T07:48:01.241 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c016220347] 2024-05-27T07:48:01.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c015f1c87e] 2024-05-27T07:48:01.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c015f21e94] 2024-05-27T07:48:01.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c015f2c969] 2024-05-27T07:48:01.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c0162c548c] 2024-05-27T07:48:01.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c0162ce9cb] 2024-05-27T07:48:01.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 16: (Thread::entry_wrapper()+0x43) [0x7fc62c0e672d] 2024-05-27T07:48:01.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 17: (Thread::_entry_func(void*)+0xd) [0x7fc62c0e6749] 2024-05-27T07:48:01.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc62ac8a1ca] 2024-05-27T07:48:01.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 19: clone() 2024-05-27T07:48:01.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:48:01.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:01.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: -1> 2024-05-27T07:48:00.889+0000 7fc6198d2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fc6198d2700 time 2024-05-27T07:48:00.887819+0000 2024-05-27T07:48:01.242 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc62c117ba6] 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc62c117dd1] 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c0162a123d] 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c01628c971] 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c01628afa0] 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c01628b651] 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c016220347] 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c015f1c87e] 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c015f21e94] 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c015f2c969] 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c0162c548c] 2024-05-27T07:48:01.243 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c0162ce9cb] 2024-05-27T07:48:01.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 13: (Thread::entry_wrapper()+0x43) [0x7fc62c0e672d] 2024-05-27T07:48:01.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 14: (Thread::_entry_func(void*)+0xd) [0x7fc62c0e6749] 2024-05-27T07:48:01.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc62ac8a1ca] 2024-05-27T07:48:01.244 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 16: clone() 2024-05-27T07:48:01.245 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:01.245 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 0> 2024-05-27T07:48:00.892+0000 7fc6198d2700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:48:01.245 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: in thread 7fc6198d2700 thread_name:md_log_replay 2024-05-27T07:48:01.245 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:01.246 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:01.246 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc62ac94d20] 2024-05-27T07:48:01.246 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2: gsignal() 2024-05-27T07:48:01.246 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 3: abort() 2024-05-27T07:48:01.246 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc62c117ced] 2024-05-27T07:48:01.246 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc62c117dd1] 2024-05-27T07:48:01.246 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c0162a123d] 2024-05-27T07:48:01.246 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c01628c971] 2024-05-27T07:48:01.246 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c01628afa0] 2024-05-27T07:48:01.246 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c01628b651] 2024-05-27T07:48:01.246 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c016220347] 2024-05-27T07:48:01.246 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c015f1c87e] 2024-05-27T07:48:01.246 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c015f21e94] 2024-05-27T07:48:01.247 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c015f2c969] 2024-05-27T07:48:01.247 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c0162c548c] 2024-05-27T07:48:01.247 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c0162ce9cb] 2024-05-27T07:48:01.247 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 16: (Thread::entry_wrapper()+0x43) [0x7fc62c0e672d] 2024-05-27T07:48:01.247 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 17: (Thread::_entry_func(void*)+0xd) [0x7fc62c0e6749] 2024-05-27T07:48:01.247 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc62ac8a1ca] 2024-05-27T07:48:01.247 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 19: clone() 2024-05-27T07:48:01.247 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:48:01.247 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[72619]: 2024-05-27T07:48:01.247 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:00 smithi186 podman[73715]: 2024-05-27 07:48:00.958441412 +0000 UTC m=+0.021484920 container died e3552dde5a37224eadfd8a9f0f65bcf0ddfed049861957fcfc2497b3c66ea931 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:48:01.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 841644 11.41 MB/sec execute 253 sec latency 280.696 ms 2024-05-27T07:48:01.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:01 smithi163 ceph-mon[36451]: pgmap v266: 129 pgs: 129 active+clean; 79 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 313 op/s 2024-05-27T07:48:01.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:01 smithi186 ceph-mon[36647]: pgmap v266: 129 pgs: 129 active+clean; 79 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 313 op/s 2024-05-27T07:48:01.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:01 smithi114 ceph-mon[32038]: pgmap v266: 129 pgs: 129 active+clean; 79 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 313 op/s 2024-05-27T07:48:02.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 842238 11.37 MB/sec execute 254 sec latency 152.868 ms 2024-05-27T07:48:03.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 842603 11.33 MB/sec execute 255 sec latency 173.836 ms 2024-05-27T07:48:03.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:03 smithi163 ceph-mon[36451]: pgmap v267: 129 pgs: 129 active+clean; 65 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 441 op/s 2024-05-27T07:48:03.802 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:03 smithi163 ceph-mon[36451]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T07:48:03.803 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:03 smithi186 ceph-mon[36647]: pgmap v267: 129 pgs: 129 active+clean; 65 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 441 op/s 2024-05-27T07:48:03.803 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:03 smithi186 ceph-mon[36647]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T07:48:03.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:03 smithi114 ceph-mon[32038]: pgmap v267: 129 pgs: 129 active+clean; 65 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 441 op/s 2024-05-27T07:48:03.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:03 smithi114 ceph-mon[32038]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T07:48:04.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 843140 11.30 MB/sec execute 256 sec latency 53.775 ms 2024-05-27T07:48:05.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 843796 11.26 MB/sec execute 257 sec latency 185.099 ms 2024-05-27T07:48:05.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:05 smithi163 ceph-mon[36451]: pgmap v269: 129 pgs: 129 active+clean; 65 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 426 op/s 2024-05-27T07:48:05.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:05 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:48:05.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:05 smithi186 ceph-mon[36647]: pgmap v269: 129 pgs: 129 active+clean; 65 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 426 op/s 2024-05-27T07:48:05.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:05 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:48:05.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:05 smithi114 ceph-mon[32038]: pgmap v269: 129 pgs: 129 active+clean; 65 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 426 op/s 2024-05-27T07:48:05.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:05 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:48:06.274 INFO:tasks.workunit.client.0.smithi114.stdout: 1 844452 11.24 MB/sec execute 258 sec latency 86.399 ms 2024-05-27T07:48:07.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 845246 11.22 MB/sec execute 259 sec latency 145.107 ms 2024-05-27T07:48:07.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:07 smithi163 ceph-mon[36451]: pgmap v270: 129 pgs: 17 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 75 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 375 op/s 2024-05-27T07:48:07.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:07 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:48:07.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:07 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:48:07.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:07 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:48:07.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:07 smithi186 ceph-mon[36647]: pgmap v270: 129 pgs: 17 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 75 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 375 op/s 2024-05-27T07:48:07.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:07 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:48:07.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:07 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:48:07.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:07 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:48:07.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:07 smithi114 ceph-mon[32038]: pgmap v270: 129 pgs: 17 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 75 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 375 op/s 2024-05-27T07:48:07.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:07 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:48:07.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:07 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:48:07.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:07 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:48:08.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 845617 11.18 MB/sec execute 260 sec latency 161.869 ms 2024-05-27T07:48:08.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:08 smithi186 podman[73715]: 2024-05-27 07:48:08.299183678 +0000 UTC m=+7.362227177 container remove e3552dde5a37224eadfd8a9f0f65bcf0ddfed049861957fcfc2497b3c66ea931 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:48:08.728 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:08 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:48:09.229 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:08 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:48:09.229 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:08 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Consumed 2.847s CPU time. 2024-05-27T07:48:09.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 846430 11.15 MB/sec execute 261 sec latency 48.003 ms 2024-05-27T07:48:09.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:09 smithi163 ceph-mon[36451]: pgmap v271: 129 pgs: 17 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 75 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 375 op/s 2024-05-27T07:48:09.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:09 smithi186 ceph-mon[36647]: pgmap v271: 129 pgs: 17 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 75 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 375 op/s 2024-05-27T07:48:09.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:09 smithi114 ceph-mon[32038]: pgmap v271: 129 pgs: 17 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 75 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 375 op/s 2024-05-27T07:48:10.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 848395 11.15 MB/sec execute 262 sec latency 25.721 ms 2024-05-27T07:48:11.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 850122 11.14 MB/sec execute 263 sec latency 16.201 ms 2024-05-27T07:48:11.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:11 smithi163 ceph-mon[36451]: pgmap v272: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 68 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 2.8 MiB/s wr, 288 op/s 2024-05-27T07:48:11.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:11 smithi186 ceph-mon[36647]: pgmap v272: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 68 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 2.8 MiB/s wr, 288 op/s 2024-05-27T07:48:11.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:11 smithi114 ceph-mon[32038]: pgmap v272: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 68 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 2.8 MiB/s wr, 288 op/s 2024-05-27T07:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.922+0000 7f655b684640 1 -- 172.21.15.114:0/2584690516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65541021c0 msgr2=0x7f65541025d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.922+0000 7f655b684640 1 --2- 172.21.15.114:0/2584690516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65541021c0 0x7f65541025d0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f6548009900 tx=0x7f654802a410 comp rx=0 tx=0).stop 2024-05-27T07:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.922+0000 7f655b684640 1 -- 172.21.15.114:0/2584690516 shutdown_connections 2024-05-27T07:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.922+0000 7f655b684640 1 --2- 172.21.15.114:0/2584690516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f655410e6c0 0x7f6554110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.922+0000 7f655b684640 1 --2- 172.21.15.114:0/2584690516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65541021c0 0x7f65541025d0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.922+0000 7f655b684640 1 --2- 172.21.15.114:0/2584690516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6554109d10 0x7f6554101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.922+0000 7f655b684640 1 -- 172.21.15.114:0/2584690516 >> 172.21.15.114:0/2584690516 conn(0x7f65540779d0 msgr2=0x7f6554077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.922+0000 7f655b684640 1 -- 172.21.15.114:0/2584690516 shutdown_connections 2024-05-27T07:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.922+0000 7f655b684640 1 -- 172.21.15.114:0/2584690516 wait complete. 2024-05-27T07:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f655b684640 1 Processor -- start 2024-05-27T07:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f655b684640 1 -- start start 2024-05-27T07:48:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f655b684640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65541021c0 0x7f65541a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f655b684640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6554109d10 0x7f65541a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f655b684640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f655410e6c0 0x7f655419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f655b684640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6554113e80 con 0x7f65541021c0 2024-05-27T07:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f655b684640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6554113d00 con 0x7f6554109d10 2024-05-27T07:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f655b684640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6554114000 con 0x7f655410e6c0 2024-05-27T07:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f6559420640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65541021c0 0x7f65541a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f6559420640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65541021c0 0x7f65541a0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59992/0 (socket says 172.21.15.114:59992) 2024-05-27T07:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f6559420640 1 -- 172.21.15.114:0/3924288932 learned_addr learned my addr 172.21.15.114:0/3924288932 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f6558c1f640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6554109d10 0x7f65541a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f6559c21640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f655410e6c0 0x7f655419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f6559420640 1 -- 172.21.15.114:0/3924288932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f655410e6c0 msgr2=0x7f655419a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f6559420640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f655410e6c0 0x7f655419a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f6559420640 1 -- 172.21.15.114:0/3924288932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6554109d10 msgr2=0x7f65541a0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f6559420640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6554109d10 0x7f65541a0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.923+0000 7f6559420640 1 -- 172.21.15.114:0/3924288932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655419add0 con 0x7f65541021c0 2024-05-27T07:48:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.924+0000 7f6559420640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65541021c0 0x7f65541a0520 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f654400e9f0 tx=0x7f654400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.924+0000 7f65527fc640 1 -- 172.21.15.114:0/3924288932 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654400ccc0 con 0x7f65541021c0 2024-05-27T07:48:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.924+0000 7f655b684640 1 -- 172.21.15.114:0/3924288932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f655419b060 con 0x7f65541021c0 2024-05-27T07:48:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.924+0000 7f65527fc640 1 -- 172.21.15.114:0/3924288932 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65440035f0 con 0x7f65541021c0 2024-05-27T07:48:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.924+0000 7f65527fc640 1 -- 172.21.15.114:0/3924288932 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654400ce60 con 0x7f65541021c0 2024-05-27T07:48:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.924+0000 7f655b684640 1 -- 172.21.15.114:0/3924288932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65541a72c0 con 0x7f65541021c0 2024-05-27T07:48:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6562606740 1 Processor -- start 2024-05-27T07:48:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6562606740 1 -- start start 2024-05-27T07:48:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f65527fc640 1 -- 172.21.15.114:0/3924288932 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6544003aa0 con 0x7f65541021c0 2024-05-27T07:48:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6562606740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a66061b0 0x5617a69a7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6562606740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65dd2f0 0x5617a69a77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6562606740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 0x5617a69a7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6562606740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5617a6919c30 con 0x5617a65e54d0 2024-05-27T07:48:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6562606740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5617a69097d0 con 0x5617a65dd2f0 2024-05-27T07:48:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6562606740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5617a690d810 con 0x5617a66061b0 2024-05-27T07:48:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6558c1f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 0x5617a69a7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6559c21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a66061b0 0x5617a69a7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6558c1f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 0x5617a69a7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60004/0 (socket says 172.21.15.114:60004) 2024-05-27T07:48:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6559c21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a66061b0 0x5617a69a7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:38986/0 (socket says 172.21.15.114:38986) 2024-05-27T07:48:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6558c1f640 1 -- 172.21.15.114:0/1679491251 learned_addr learned my addr 172.21.15.114:0/1679491251 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6559420640 1 --2- 172.21.15.114:0/1679491251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65dd2f0 0x5617a69a77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6558c1f640 1 -- 172.21.15.114:0/1679491251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a66061b0 msgr2=0x5617a69a7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.927+0000 7f6558c1f640 1 --2- 172.21.15.114:0/1679491251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a66061b0 0x5617a69a7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.928+0000 7f6558c1f640 1 -- 172.21.15.114:0/1679491251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65dd2f0 msgr2=0x5617a69a77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.928+0000 7f6558c1f640 1 --2- 172.21.15.114:0/1679491251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65dd2f0 0x5617a69a77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.928+0000 7f6558c1f640 1 -- 172.21.15.114:0/1679491251 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617a661e210 con 0x5617a65e54d0 2024-05-27T07:48:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.928+0000 7f65527fc640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f653c0b2950 0x7f653c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.928+0000 7f6558c1f640 1 --2- 172.21.15.114:0/1679491251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 0x5617a69a7d00 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f6548009900 tx=0x7f6548032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd221f8ee206bfa7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.928+0000 7f6559c21640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f653c0b2950 0x7f653c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.928+0000 7f653b7fe640 1 -- 172.21.15.114:0/1679491251 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548040070 con 0x5617a65e54d0 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.928+0000 7f65527fc640 1 -- 172.21.15.114:0/3924288932 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9642+0+0 (secure 0 0 0) 0x7f65440e2d60 con 0x7f65541021c0 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.928+0000 7f653b7fe640 1 -- 172.21.15.114:0/1679491251 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6548004030 con 0x5617a65e54d0 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.928+0000 7f653b7fe640 1 -- 172.21.15.114:0/1679491251 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548035040 con 0x5617a65e54d0 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.929+0000 7f6562606740 1 -- 172.21.15.114:0/1679491251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 msgr2=0x5617a69a7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.929+0000 7f6559c21640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f653c0b2950 0x7f653c0b4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6554102c60 tx=0x7f654c009500 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.929+0000 7f6562606740 1 --2- 172.21.15.114:0/1679491251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 0x5617a69a7d00 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f6548009900 tx=0x7f6548032b20 comp rx=0 tx=0).stop 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.929+0000 7f6562606740 1 -- 172.21.15.114:0/1679491251 shutdown_connections 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.929+0000 7f6562606740 1 --2- 172.21.15.114:0/1679491251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 0x5617a69a7d00 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.929+0000 7f6562606740 1 --2- 172.21.15.114:0/1679491251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65dd2f0 0x5617a69a77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.929+0000 7f6562606740 1 --2- 172.21.15.114:0/1679491251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a66061b0 0x5617a69a7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.929+0000 7f6562606740 1 -- 172.21.15.114:0/1679491251 >> 172.21.15.114:0/1679491251 conn(0x5617a65fa300 msgr2=0x5617a69a3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.929+0000 7f6562606740 1 -- 172.21.15.114:0/1679491251 shutdown_connections 2024-05-27T07:48:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.929+0000 7f6562606740 1 -- 172.21.15.114:0/1679491251 wait complete. 2024-05-27T07:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.929+0000 7f6562606740 1 -- start start 2024-05-27T07:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.929+0000 7f6562606740 1 Processor -- start 2024-05-27T07:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6562606740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65dd2f0 0x5617a69b7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6562606740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65e54d0 0x5617a69b7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6562606740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a66061b0 0x5617a69b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6562606740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5617a65ef790 con 0x5617a66061b0 2024-05-27T07:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6562606740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5617a68c5890 con 0x5617a65e54d0 2024-05-27T07:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6562606740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5617a69b88b0 con 0x5617a65dd2f0 2024-05-27T07:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6562606740 10 client.0 _ll_register_callbacks cb 0x5617a661da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6559c21640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a66061b0 0x5617a69b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6559c21640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a66061b0 0x5617a69b5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60010/0 (socket says 172.21.15.114:60010) 2024-05-27T07:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6559420640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65dd2f0 0x5617a69b7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6559c21640 1 -- 172.21.15.114:0/1076994391 learned_addr learned my addr 172.21.15.114:0/1076994391 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6558c1f640 1 --2- 172.21.15.114:0/1076994391 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65e54d0 0x5617a69b7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6559c21640 1 -- 172.21.15.114:0/1076994391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65dd2f0 msgr2=0x5617a69b7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6559c21640 1 --2- 172.21.15.114:0/1076994391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65dd2f0 0x5617a69b7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6559c21640 1 -- 172.21.15.114:0/1076994391 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65e54d0 msgr2=0x5617a69b7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6559c21640 1 --2- 172.21.15.114:0/1076994391 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65e54d0 0x5617a69b7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.930+0000 7f6559c21640 1 -- 172.21.15.114:0/1076994391 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617a66ea620 con 0x5617a66061b0 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6559c21640 1 --2- 172.21.15.114:0/1076994391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a66061b0 0x5617a69b5d30 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f654c004060 tx=0x7f654c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdeb0687aa4a4b50 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6538ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6538ff9640 1 -- 172.21.15.114:0/1076994391 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c00f8b0 con 0x5617a66061b0 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6538ff9640 1 -- 172.21.15.114:0/1076994391 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9642+0+0 (secure 0 0 0) 0x7f654c00fa50 con 0x5617a66061b0 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6562606740 1 -- 172.21.15.114:0/1076994391 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6548004030 con 0x5617a66061b0 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6538ff9640 1 -- 172.21.15.114:0/1076994391 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f654c00fc50 con 0x5617a66061b0 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6562606740 10 client.17418 fetch_fsmap learned FSMap version 46 2024-05-27T07:48:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6562606740 1 -- 172.21.15.114:0/1076994391 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5617a69c4c50 con 0x5617a66061b0 2024-05-27T07:48:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6538ff9640 1 -- 172.21.15.114:0/1076994391 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6548004030 con 0x5617a66061b0 2024-05-27T07:48:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6538ff9640 1 -- 172.21.15.114:0/1076994391 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7f654c026070 con 0x5617a66061b0 2024-05-27T07:48:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6562606740 10 client.17418 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:48:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6562606740 10 client.17418 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6562606740 20 client.17418 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6562606740 1 --2- 172.21.15.114:0/1076994391 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5617a65c4d80 0x5617a69c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6562606740 4 client.17418 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:48:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.931+0000 7f6562606740 1 -- 172.21.15.114:0/1076994391 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617a691bb80 con 0x5617a65c4d80 2024-05-27T07:48:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.932+0000 7f6559420640 1 --2- 172.21.15.114:0/1076994391 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5617a65c4d80 0x5617a69c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.932+0000 7f6538ff9640 10 client.17418 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.932+0000 7f6559420640 1 --2- 172.21.15.114:0/1076994391 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5617a65c4d80 0x5617a69c84c0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b01a1a1b42ea6047 server_cookie=5c4562132d3fb962 in_seq=0 out_seq=0 2024-05-27T07:48:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6538ff9640 1 -- 172.21.15.114:0/1076994391 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5617a691bb80 con 0x5617a65c4d80 2024-05-27T07:48:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6538ff9640 10 client.17418 handle_command_reply: tid=0 2024-05-27T07:48:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6562606740 1 client.17418 shutdown 2024-05-27T07:48:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6562606740 1 -- 172.21.15.114:0/1076994391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a66061b0 msgr2=0x5617a69b5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6562606740 1 --2- 172.21.15.114:0/1076994391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a66061b0 0x5617a69b5d30 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f654c004060 tx=0x7f654c004ab0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6562606740 1 -- 172.21.15.114:0/1076994391 shutdown_connections 2024-05-27T07:48:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6562606740 1 --2- 172.21.15.114:0/1076994391 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5617a65c4d80 0x5617a69c84c0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6562606740 1 --2- 172.21.15.114:0/1076994391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a66061b0 0x5617a69b5d30 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6562606740 1 --2- 172.21.15.114:0/1076994391 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65e54d0 0x5617a69b7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6538ff9640 0 client.17418 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6562606740 1 --2- 172.21.15.114:0/1076994391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65dd2f0 0x5617a69b7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6562606740 1 -- 172.21.15.114:0/1076994391 >> 172.21.15.114:0/1076994391 conn(0x5617a65fa300 msgr2=0x5617a69a9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6562606740 1 -- 172.21.15.114:0/1076994391 shutdown_connections 2024-05-27T07:48:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6562606740 1 -- 172.21.15.114:0/1076994391 wait complete. 2024-05-27T07:48:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.933+0000 7f6562606740 20 client.17418 trim_cache size 0 max 16384 2024-05-27T07:48:11.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.949+0000 7f6562606740 1 Processor -- start 2024-05-27T07:48:11.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.949+0000 7f6562606740 1 -- start start 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6562606740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65fa300 0x5617a69a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6562606740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65dd2f0 0x5617a69a96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6562606740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 0x5617a69af610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6562606740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65541af470 con 0x5617a65e54d0 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6562606740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f65541afb40 con 0x5617a65dd2f0 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6562606740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f65541ad2e0 con 0x5617a65fa300 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6559c21640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 0x5617a69af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6559420640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65fa300 0x5617a69a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6559c21640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 0x5617a69af610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60020/0 (socket says 172.21.15.114:60020) 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6559c21640 1 -- 172.21.15.114:0/3177671849 learned_addr learned my addr 172.21.15.114:0/3177671849 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6558c1f640 1 --2- 172.21.15.114:0/3177671849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65dd2f0 0x5617a69a96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6559c21640 1 -- 172.21.15.114:0/3177671849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65fa300 msgr2=0x5617a69a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6559c21640 1 --2- 172.21.15.114:0/3177671849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65fa300 0x5617a69a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6559c21640 1 -- 172.21.15.114:0/3177671849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65dd2f0 msgr2=0x5617a69a96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6559c21640 1 --2- 172.21.15.114:0/3177671849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65dd2f0 0x5617a69a96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6559c21640 1 -- 172.21.15.114:0/3177671849 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617a69b84f0 con 0x5617a65e54d0 2024-05-27T07:48:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f6559c21640 1 --2- 172.21.15.114:0/3177671849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 0x5617a69af610 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f654c009f50 tx=0x7f654c009320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=240dea6f0afd395e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f653affd640 1 -- 172.21.15.114:0/3177671849 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c00fdf0 con 0x5617a65e54d0 2024-05-27T07:48:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f653affd640 1 -- 172.21.15.114:0/3177671849 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f654c028ca0 con 0x5617a65e54d0 2024-05-27T07:48:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.950+0000 7f653affd640 1 -- 172.21.15.114:0/3177671849 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c01d8d0 con 0x5617a65e54d0 2024-05-27T07:48:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.951+0000 7f6562606740 1 -- 172.21.15.114:0/3177671849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 msgr2=0x5617a69af610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.951+0000 7f6562606740 1 --2- 172.21.15.114:0/3177671849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 0x5617a69af610 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f654c009f50 tx=0x7f654c009320 comp rx=0 tx=0).stop 2024-05-27T07:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.951+0000 7f6562606740 1 -- 172.21.15.114:0/3177671849 shutdown_connections 2024-05-27T07:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.951+0000 7f6562606740 1 --2- 172.21.15.114:0/3177671849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65e54d0 0x5617a69af610 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.951+0000 7f6562606740 1 --2- 172.21.15.114:0/3177671849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65dd2f0 0x5617a69a96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.951+0000 7f6562606740 1 --2- 172.21.15.114:0/3177671849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65fa300 0x5617a69a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.951+0000 7f6562606740 1 -- 172.21.15.114:0/3177671849 >> 172.21.15.114:0/3177671849 conn(0x5617a69b16b0 msgr2=0x5617a69a3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.951+0000 7f6562606740 1 -- 172.21.15.114:0/3177671849 shutdown_connections 2024-05-27T07:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.951+0000 7f6562606740 1 -- 172.21.15.114:0/3177671849 wait complete. 2024-05-27T07:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6562606740 1 -- start start 2024-05-27T07:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6562606740 1 Processor -- start 2024-05-27T07:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6562606740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65dd2f0 0x5617a69b9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6562606740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65e54d0 0x5617a69ba020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6562606740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65fa300 0x5617a69ba560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6562606740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65541ad930 con 0x5617a65dd2f0 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6562606740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f65541aeda0 con 0x5617a65e54d0 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6562606740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f65541ae000 con 0x5617a65fa300 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6562606740 10 client.0 _ll_register_callbacks cb 0x7f654c009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6559420640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65dd2f0 0x5617a69b9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6559420640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65dd2f0 0x5617a69b9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60026/0 (socket says 172.21.15.114:60026) 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6559420640 1 -- 172.21.15.114:0/283493288 learned_addr learned my addr 172.21.15.114:0/283493288 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6559c21640 1 --2- 172.21.15.114:0/283493288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65fa300 0x5617a69ba560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.952+0000 7f6558c1f640 1 --2- 172.21.15.114:0/283493288 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65e54d0 0x5617a69ba020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6559420640 1 -- 172.21.15.114:0/283493288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65fa300 msgr2=0x5617a69ba560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6559420640 1 --2- 172.21.15.114:0/283493288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65fa300 0x5617a69ba560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6559420640 1 -- 172.21.15.114:0/283493288 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65e54d0 msgr2=0x5617a69ba020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6559420640 1 --2- 172.21.15.114:0/283493288 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65e54d0 0x5617a69ba020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6559420640 1 -- 172.21.15.114:0/283493288 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617a6a2fd40 con 0x5617a65dd2f0 2024-05-27T07:48:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6559420640 1 --2- 172.21.15.114:0/283493288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65dd2f0 0x5617a69b9ae0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f65440e0650 tx=0x7f65440e68b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a273adc62779ec8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6538ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:48:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6538ff9640 1 -- 172.21.15.114:0/283493288 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654408d070 con 0x5617a65dd2f0 2024-05-27T07:48:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6538ff9640 1 -- 172.21.15.114:0/283493288 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9642+0+0 (secure 0 0 0) 0x7f65440a5e70 con 0x5617a65dd2f0 2024-05-27T07:48:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6562606740 1 -- 172.21.15.114:0/283493288 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f654c028ca0 con 0x5617a65dd2f0 2024-05-27T07:48:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6538ff9640 1 -- 172.21.15.114:0/283493288 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f65440a5370 con 0x5617a65dd2f0 2024-05-27T07:48:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6562606740 10 client.17424 fetch_fsmap learned FSMap version 46 2024-05-27T07:48:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6562606740 1 -- 172.21.15.114:0/283493288 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5617a6a30fd0 con 0x5617a65dd2f0 2024-05-27T07:48:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6538ff9640 1 -- 172.21.15.114:0/283493288 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f654c028ca0 con 0x5617a65dd2f0 2024-05-27T07:48:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6538ff9640 1 -- 172.21.15.114:0/283493288 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 46) ==== 7120+0+0 (secure 0 0 0) 0x7f65440c4120 con 0x5617a65dd2f0 2024-05-27T07:48:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6562606740 10 client.17424 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:48:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6562606740 10 client.17424 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6562606740 20 client.17424 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.953+0000 7f6562606740 1 --2- 172.21.15.114:0/283493288 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5617a66061b0 0x5617a6a33760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.954+0000 7f6562606740 4 client.17424 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:48:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.954+0000 7f6562606740 1 -- 172.21.15.114:0/283493288 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5617a6915360 con 0x5617a66061b0 2024-05-27T07:48:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.954+0000 7f6558c1f640 1 --2- 172.21.15.114:0/283493288 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5617a66061b0 0x5617a6a33760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.954+0000 7f6538ff9640 10 client.17424 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:11.954+0000 7f6558c1f640 1 --2- 172.21.15.114:0/283493288 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5617a66061b0 0x5617a6a33760 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb6d7d5b94cfd293 server_cookie=219f605e46629af5 in_seq=0 out_seq=0 2024-05-27T07:48:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6538ff9640 1 -- 172.21.15.114:0/283493288 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5617a6915360 con 0x5617a66061b0 2024-05-27T07:48:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6538ff9640 10 client.17424 handle_command_reply: tid=0 2024-05-27T07:48:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6562606740 1 client.17424 shutdown 2024-05-27T07:48:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6562606740 1 -- 172.21.15.114:0/283493288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65dd2f0 msgr2=0x5617a69b9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6562606740 1 --2- 172.21.15.114:0/283493288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65dd2f0 0x5617a69b9ae0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f65440e0650 tx=0x7f65440e68b0 comp rx=0 tx=0).stop 2024-05-27T07:48:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6562606740 1 -- 172.21.15.114:0/283493288 shutdown_connections 2024-05-27T07:48:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6562606740 1 --2- 172.21.15.114:0/283493288 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5617a66061b0 0x5617a6a33760 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6562606740 1 --2- 172.21.15.114:0/283493288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617a65fa300 0x5617a69ba560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6562606740 1 --2- 172.21.15.114:0/283493288 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617a65e54d0 0x5617a69ba020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6562606740 1 --2- 172.21.15.114:0/283493288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617a65dd2f0 0x5617a69b9ae0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6538ff9640 0 client.17424 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6562606740 1 -- 172.21.15.114:0/283493288 >> 172.21.15.114:0/283493288 conn(0x5617a69b16b0 msgr2=0x5617a69a7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6562606740 1 -- 172.21.15.114:0/283493288 shutdown_connections 2024-05-27T07:48:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6562606740 1 -- 172.21.15.114:0/283493288 wait complete. 2024-05-27T07:48:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.199+0000 7f6562606740 20 client.17424 trim_cache size 0 max 16384 2024-05-27T07:48:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:48:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:48:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:48:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.200+0000 7f655b684640 1 -- 172.21.15.114:0/3924288932 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f653c0b2950 msgr2=0x7f653c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.200+0000 7f655b684640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f653c0b2950 0x7f653c0b4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6554102c60 tx=0x7f654c009500 comp rx=0 tx=0).stop 2024-05-27T07:48:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.200+0000 7f655b684640 1 -- 172.21.15.114:0/3924288932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65541021c0 msgr2=0x7f65541a0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.200+0000 7f655b684640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65541021c0 0x7f65541a0520 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f654400e9f0 tx=0x7f654400eeb0 comp rx=0 tx=0).stop 2024-05-27T07:48:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.200+0000 7f655b684640 1 -- 172.21.15.114:0/3924288932 shutdown_connections 2024-05-27T07:48:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.200+0000 7f655b684640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f653c0b2950 0x7f653c0b4e60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.200+0000 7f655b684640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f655410e6c0 0x7f655419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.200+0000 7f655b684640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6554109d10 0x7f65541a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:12.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.200+0000 7f655b684640 1 --2- 172.21.15.114:0/3924288932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65541021c0 0x7f65541a0520 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:12.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.200+0000 7f655b684640 1 -- 172.21.15.114:0/3924288932 >> 172.21.15.114:0/3924288932 conn(0x7f65540779d0 msgr2=0x7f6554105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:12.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.200+0000 7f655b684640 1 -- 172.21.15.114:0/3924288932 shutdown_connections 2024-05-27T07:48:12.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:12.200+0000 7f655b684640 1 -- 172.21.15.114:0/3924288932 wait complete. 2024-05-27T07:48:12.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 852271 11.15 MB/sec execute 264 sec latency 129.009 ms 2024-05-27T07:48:13.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 853564 11.13 MB/sec execute 265 sec latency 132.387 ms 2024-05-27T07:48:13.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:13 smithi163 ceph-mon[36451]: pgmap v273: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 2.6 MiB/s wr, 160 op/s 2024-05-27T07:48:13.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:13 smithi186 ceph-mon[36647]: pgmap v273: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 2.6 MiB/s wr, 160 op/s 2024-05-27T07:48:13.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:13 smithi114 ceph-mon[32038]: pgmap v273: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 2.6 MiB/s wr, 160 op/s 2024-05-27T07:48:14.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 855894 11.14 MB/sec execute 266 sec latency 41.575 ms 2024-05-27T07:48:15.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 857302 11.12 MB/sec execute 267 sec latency 160.336 ms 2024-05-27T07:48:15.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:15 smithi114 ceph-mon[32038]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T07:48:15.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:15 smithi114 ceph-mon[32038]: pgmap v275: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 2.6 MiB/s wr, 160 op/s 2024-05-27T07:48:15.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:15 smithi163 ceph-mon[36451]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T07:48:15.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:15 smithi163 ceph-mon[36451]: pgmap v275: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 2.6 MiB/s wr, 160 op/s 2024-05-27T07:48:15.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:15 smithi186 ceph-mon[36647]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T07:48:15.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:15 smithi186 ceph-mon[36647]: pgmap v275: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 2.6 MiB/s wr, 160 op/s 2024-05-27T07:48:16.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 859648 11.13 MB/sec execute 268 sec latency 18.476 ms 2024-05-27T07:48:17.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 861549 11.11 MB/sec execute 269 sec latency 34.628 ms 2024-05-27T07:48:17.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:17 smithi163 ceph-mon[36451]: pgmap v276: 129 pgs: 129 active+clean; 65 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 3.4 MiB/s wr, 296 op/s 2024-05-27T07:48:17.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:17 smithi186 ceph-mon[36647]: pgmap v276: 129 pgs: 129 active+clean; 65 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 3.4 MiB/s wr, 296 op/s 2024-05-27T07:48:17.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:17 smithi114 ceph-mon[32038]: pgmap v276: 129 pgs: 129 active+clean; 65 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 3.4 MiB/s wr, 296 op/s 2024-05-27T07:48:18.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 863214 11.12 MB/sec execute 270 sec latency 56.942 ms 2024-05-27T07:48:19.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:18 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:48:19.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:18 smithi186 systemd[1]: Stopped Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:48:19.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:18 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Consumed 2.847s CPU time. 2024-05-27T07:48:19.238 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:18 smithi186 systemd[1]: Starting Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:48:19.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 865040 11.10 MB/sec execute 271 sec latency 55.076 ms 2024-05-27T07:48:19.494 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:19 smithi186 ceph-mon[36647]: pgmap v277: 129 pgs: 129 active+clean; 65 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 3.4 MiB/s wr, 296 op/s 2024-05-27T07:48:19.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:19 smithi186 ceph-mon[36647]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:48:19.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:19 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 7 up:standby 2024-05-27T07:48:19.495 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:19 smithi186 ceph-mon[36647]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T07:48:19.495 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:19 smithi186 podman[73928]: 2024-05-27 07:48:19.138137658 +0000 UTC m=+0.017736345 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:48:19.495 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:19 smithi186 podman[73928]: 2024-05-27 07:48:19.250449163 +0000 UTC m=+0.130047846 container create ec06090fac6668da667e87cf03bd2e0b836783c0443d1cd7bfec43477c749b05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-27T07:48:19.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:19 smithi163 ceph-mon[36451]: pgmap v277: 129 pgs: 129 active+clean; 65 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 3.4 MiB/s wr, 296 op/s 2024-05-27T07:48:19.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:19 smithi163 ceph-mon[36451]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:48:19.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:19 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 7 up:standby 2024-05-27T07:48:19.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:19 smithi163 ceph-mon[36451]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T07:48:19.726 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4d4fa58700 time 2024-05-27T07:48:19.317876+0000 2024-05-27T07:48:19.726 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:19.726 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:19.727 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4d6229dba6] 2024-05-27T07:48:19.727 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d6229ddd1] 2024-05-27T07:48:19.727 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56223c0f523d] 2024-05-27T07:48:19.727 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56223c0e0971] 2024-05-27T07:48:19.727 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 5: (SnapRealm::check_cache() const+0x13e) [0x56223c0defa0] 2024-05-27T07:48:19.727 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56223c0df651] 2024-05-27T07:48:19.727 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56223c074347] 2024-05-27T07:48:19.727 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56223bd7087e] 2024-05-27T07:48:19.727 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56223bd75f2f] 2024-05-27T07:48:19.727 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56223bd80969] 2024-05-27T07:48:19.727 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 11: (MDLog::_replay_thread()+0x1f38) [0x56223c11948c] 2024-05-27T07:48:19.727 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56223c1229cb] 2024-05-27T07:48:19.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 13: (Thread::entry_wrapper()+0x43) [0x7f4d6226c72d] 2024-05-27T07:48:19.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4d6226c749] 2024-05-27T07:48:19.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4d60e101ca] 2024-05-27T07:48:19.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 16: clone() 2024-05-27T07:48:19.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.319+0000 7f4d4fa58700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4d4fa58700 time 2024-05-27T07:48:19.317876+0000 2024-05-27T07:48:19.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:19.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:19.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4d6229dba6] 2024-05-27T07:48:19.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d6229ddd1] 2024-05-27T07:48:19.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56223c0f523d] 2024-05-27T07:48:19.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56223c0e0971] 2024-05-27T07:48:19.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 5: (SnapRealm::check_cache() const+0x13e) [0x56223c0defa0] 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56223c0df651] 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56223c074347] 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56223bd7087e] 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56223bd75f2f] 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56223bd80969] 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 11: (MDLog::_replay_thread()+0x1f38) [0x56223c11948c] 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56223c1229cb] 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 13: (Thread::entry_wrapper()+0x43) [0x7f4d6226c72d] 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4d6226c749] 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4d60e101ca] 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 16: clone() 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: *** Caught signal (Aborted) ** 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: in thread 7f4d4fa58700 thread_name:md_log_replay 2024-05-27T07:48:19.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:19.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4d60e1ad20] 2024-05-27T07:48:19.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2: gsignal() 2024-05-27T07:48:19.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 3: abort() 2024-05-27T07:48:19.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4d6229dced] 2024-05-27T07:48:19.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d6229ddd1] 2024-05-27T07:48:19.731 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56223c0f523d] 2024-05-27T07:48:19.731 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56223c0e0971] 2024-05-27T07:48:19.731 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 8: (SnapRealm::check_cache() const+0x13e) [0x56223c0defa0] 2024-05-27T07:48:19.731 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56223c0df651] 2024-05-27T07:48:19.731 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56223c074347] 2024-05-27T07:48:19.731 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56223bd7087e] 2024-05-27T07:48:19.731 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56223bd75f2f] 2024-05-27T07:48:19.731 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56223bd80969] 2024-05-27T07:48:19.731 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 14: (MDLog::_replay_thread()+0x1f38) [0x56223c11948c] 2024-05-27T07:48:19.731 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56223c1229cb] 2024-05-27T07:48:19.732 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 16: (Thread::entry_wrapper()+0x43) [0x7f4d6226c72d] 2024-05-27T07:48:19.732 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4d6226c749] 2024-05-27T07:48:19.732 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4d60e101ca] 2024-05-27T07:48:19.732 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 19: clone() 2024-05-27T07:48:19.732 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.322+0000 7f4d4fa58700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:48:19.732 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: in thread 7f4d4fa58700 thread_name:md_log_replay 2024-05-27T07:48:19.732 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.732 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:19.732 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4d60e1ad20] 2024-05-27T07:48:19.732 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2: gsignal() 2024-05-27T07:48:19.732 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 3: abort() 2024-05-27T07:48:19.732 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4d6229dced] 2024-05-27T07:48:19.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d6229ddd1] 2024-05-27T07:48:19.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56223c0f523d] 2024-05-27T07:48:19.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56223c0e0971] 2024-05-27T07:48:19.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 8: (SnapRealm::check_cache() const+0x13e) [0x56223c0defa0] 2024-05-27T07:48:19.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56223c0df651] 2024-05-27T07:48:19.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56223c074347] 2024-05-27T07:48:19.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56223bd7087e] 2024-05-27T07:48:19.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56223bd75f2f] 2024-05-27T07:48:19.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56223bd80969] 2024-05-27T07:48:19.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 14: (MDLog::_replay_thread()+0x1f38) [0x56223c11948c] 2024-05-27T07:48:19.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56223c1229cb] 2024-05-27T07:48:19.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 16: (Thread::entry_wrapper()+0x43) [0x7f4d6226c72d] 2024-05-27T07:48:19.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4d6226c749] 2024-05-27T07:48:19.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4d60e101ca] 2024-05-27T07:48:19.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 19: clone() 2024-05-27T07:48:19.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:48:19.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: -1> 2024-05-27T07:48:19.319+0000 7f4d4fa58700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4d4fa58700 time 2024-05-27T07:48:19.317876+0000 2024-05-27T07:48:19.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:19.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:19.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4d6229dba6] 2024-05-27T07:48:19.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d6229ddd1] 2024-05-27T07:48:19.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56223c0f523d] 2024-05-27T07:48:19.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56223c0e0971] 2024-05-27T07:48:19.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 5: (SnapRealm::check_cache() const+0x13e) [0x56223c0defa0] 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56223c0df651] 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56223c074347] 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56223bd7087e] 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56223bd75f2f] 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56223bd80969] 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 11: (MDLog::_replay_thread()+0x1f38) [0x56223c11948c] 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56223c1229cb] 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 13: (Thread::entry_wrapper()+0x43) [0x7f4d6226c72d] 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4d6226c749] 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4d60e101ca] 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 16: clone() 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 0> 2024-05-27T07:48:19.322+0000 7f4d4fa58700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:48:19.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: in thread 7f4d4fa58700 thread_name:md_log_replay 2024-05-27T07:48:19.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:19.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4d60e1ad20] 2024-05-27T07:48:19.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2: gsignal() 2024-05-27T07:48:19.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 3: abort() 2024-05-27T07:48:19.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4d6229dced] 2024-05-27T07:48:19.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d6229ddd1] 2024-05-27T07:48:19.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56223c0f523d] 2024-05-27T07:48:19.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56223c0e0971] 2024-05-27T07:48:19.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 8: (SnapRealm::check_cache() const+0x13e) [0x56223c0defa0] 2024-05-27T07:48:19.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56223c0df651] 2024-05-27T07:48:19.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56223c074347] 2024-05-27T07:48:19.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56223bd7087e] 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56223bd75f2f] 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56223bd80969] 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 14: (MDLog::_replay_thread()+0x1f38) [0x56223c11948c] 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56223c1229cb] 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 16: (Thread::entry_wrapper()+0x43) [0x7f4d6226c72d] 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4d6226c749] 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4d60e101ca] 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 19: clone() 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: -1> 2024-05-27T07:48:19.319+0000 7f4d4fa58700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4d4fa58700 time 2024-05-27T07:48:19.317876+0000 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:19.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4d6229dba6] 2024-05-27T07:48:19.738 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d6229ddd1] 2024-05-27T07:48:19.738 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56223c0f523d] 2024-05-27T07:48:19.738 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56223c0e0971] 2024-05-27T07:48:19.738 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 5: (SnapRealm::check_cache() const+0x13e) [0x56223c0defa0] 2024-05-27T07:48:19.738 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56223c0df651] 2024-05-27T07:48:19.738 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56223c074347] 2024-05-27T07:48:19.738 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56223bd7087e] 2024-05-27T07:48:19.738 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56223bd75f2f] 2024-05-27T07:48:19.738 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56223bd80969] 2024-05-27T07:48:19.738 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 11: (MDLog::_replay_thread()+0x1f38) [0x56223c11948c] 2024-05-27T07:48:19.738 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56223c1229cb] 2024-05-27T07:48:19.738 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 13: (Thread::entry_wrapper()+0x43) [0x7f4d6226c72d] 2024-05-27T07:48:19.738 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4d6226c749] 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4d60e101ca] 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 16: clone() 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 0> 2024-05-27T07:48:19.322+0000 7f4d4fa58700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: in thread 7f4d4fa58700 thread_name:md_log_replay 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4d60e1ad20] 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2: gsignal() 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 3: abort() 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4d6229dced] 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d6229ddd1] 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56223c0f523d] 2024-05-27T07:48:19.739 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56223c0e0971] 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 8: (SnapRealm::check_cache() const+0x13e) [0x56223c0defa0] 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56223c0df651] 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56223c074347] 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56223bd7087e] 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56223bd75f2f] 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56223bd80969] 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 14: (MDLog::_replay_thread()+0x1f38) [0x56223c11948c] 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56223c1229cb] 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 16: (Thread::entry_wrapper()+0x43) [0x7f4d6226c72d] 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4d6226c749] 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4d60e101ca] 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 19: clone() 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:48:19.740 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[66024]: 2024-05-27T07:48:19.741 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:19 smithi163 podman[71588]: 2024-05-27 07:48:19.375561112 +0000 UTC m=+0.021273953 container died ea2b072a7ecad398048b1bbed426087e7c26f0cf040561a2993df10d8bad7424 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-27T07:48:19.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:19 smithi114 ceph-mon[32038]: pgmap v277: 129 pgs: 129 active+clean; 65 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 3.4 MiB/s wr, 296 op/s 2024-05-27T07:48:19.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:19 smithi114 ceph-mon[32038]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:48:19.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:19 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 7 up:standby 2024-05-27T07:48:19.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:19 smithi114 ceph-mon[32038]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T07:48:19.979 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:19 smithi186 podman[73928]: 2024-05-27 07:48:19.495308546 +0000 UTC m=+0.374907226 container init ec06090fac6668da667e87cf03bd2e0b836783c0443d1cd7bfec43477c749b05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-27T07:48:19.979 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:19 smithi186 podman[73928]: 2024-05-27 07:48:19.49913591 +0000 UTC m=+0.378734589 container start ec06090fac6668da667e87cf03bd2e0b836783c0443d1cd7bfec43477c749b05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True) 2024-05-27T07:48:19.979 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:19 smithi186 ceph-mds[73947]: -- 172.21.15.186:0/1181223221 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561702ecda40 con 0x561702f15180 2024-05-27T07:48:19.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: starting mds.c at 2024-05-27T07:48:19.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:19 smithi186 bash[73928]: ec06090fac6668da667e87cf03bd2e0b836783c0443d1cd7bfec43477c749b05 2024-05-27T07:48:19.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:48:19 smithi186 systemd[1]: Started Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:48:20.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 866826 11.11 MB/sec execute 272 sec latency 73.982 ms 2024-05-27T07:48:20.780 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:20 smithi163 podman[71588]: 2024-05-27 07:48:20.51850333 +0000 UTC m=+1.164216165 container remove ea2b072a7ecad398048b1bbed426087e7c26f0cf040561a2993df10d8bad7424 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS) 2024-05-27T07:48:20.780 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:20 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:48:21.225 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:20 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-05-27T07:48:21.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 868694 11.09 MB/sec execute 273 sec latency 21.140 ms 2024-05-27T07:48:21.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:21 smithi163 ceph-mon[36451]: pgmap v279: 129 pgs: 129 active+clean; 68 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 331 op/s 2024-05-27T07:48:21.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:21 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6834/2622386718,v1:172.21.15.186:6835/2622386718] up:boot 2024-05-27T07:48:21.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:21 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:48:21.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:21 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:48:21.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:21 smithi186 ceph-mon[36647]: pgmap v279: 129 pgs: 129 active+clean; 68 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 331 op/s 2024-05-27T07:48:21.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:21 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6834/2622386718,v1:172.21.15.186:6835/2622386718] up:boot 2024-05-27T07:48:21.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:21 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:48:21.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:21 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:48:21.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:21 smithi114 ceph-mon[32038]: pgmap v279: 129 pgs: 129 active+clean; 68 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 331 op/s 2024-05-27T07:48:21.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:21 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6834/2622386718,v1:172.21.15.186:6835/2622386718] up:boot 2024-05-27T07:48:21.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:21 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:48:21.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:21 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:48:22.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 870655 11.10 MB/sec execute 274 sec latency 19.647 ms 2024-05-27T07:48:23.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 872714 11.10 MB/sec execute 275 sec latency 140.551 ms 2024-05-27T07:48:23.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:23 smithi163 ceph-mon[36451]: pgmap v280: 129 pgs: 129 active+clean; 72 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.6 MiB/s wr, 413 op/s 2024-05-27T07:48:23.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:23 smithi186 ceph-mon[36647]: pgmap v280: 129 pgs: 129 active+clean; 72 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.6 MiB/s wr, 413 op/s 2024-05-27T07:48:23.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:23 smithi114 ceph-mon[32038]: pgmap v280: 129 pgs: 129 active+clean; 72 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.6 MiB/s wr, 413 op/s 2024-05-27T07:48:24.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 874259 11.09 MB/sec execute 276 sec latency 38.966 ms 2024-05-27T07:48:25.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 876406 11.09 MB/sec execute 277 sec latency 102.515 ms 2024-05-27T07:48:25.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.303+0000 7f1039e69640 1 -- 172.21.15.114:0/101784000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10341073e0 msgr2=0x7f10341077f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:25.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.303+0000 7f1039e69640 1 --2- 172.21.15.114:0/101784000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10341073e0 0x7f10341077f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1028009900 tx=0x7f102802a410 comp rx=0 tx=0).stop 2024-05-27T07:48:25.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.303+0000 7f1039e69640 1 -- 172.21.15.114:0/101784000 shutdown_connections 2024-05-27T07:48:25.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.303+0000 7f1039e69640 1 --2- 172.21.15.114:0/101784000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1034107e60 0x7f103410f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.303+0000 7f1039e69640 1 --2- 172.21.15.114:0/101784000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10341073e0 0x7f10341077f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.303+0000 7f1039e69640 1 --2- 172.21.15.114:0/101784000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1034103520 0x7f1034103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.303+0000 7f1039e69640 1 -- 172.21.15.114:0/101784000 >> 172.21.15.114:0/101784000 conn(0x7f10340fd9c0 msgr2=0x7f10340ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:25.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.303+0000 7f1039e69640 1 -- 172.21.15.114:0/101784000 shutdown_connections 2024-05-27T07:48:25.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.303+0000 7f1039e69640 1 -- 172.21.15.114:0/101784000 wait complete. 2024-05-27T07:48:25.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.304+0000 7f1039e69640 1 Processor -- start 2024-05-27T07:48:25.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.304+0000 7f1039e69640 1 -- start start 2024-05-27T07:48:25.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.304+0000 7f1039e69640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1034103520 0x7f10341a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.304+0000 7f1039e69640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10341073e0 0x7f10341a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.304+0000 7f1039e69640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1034107e60 0x7f103419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.304+0000 7f1039e69640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1034113e80 con 0x7f1034107e60 2024-05-27T07:48:25.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.304+0000 7f1039e69640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1034113d00 con 0x7f10341073e0 2024-05-27T07:48:25.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.304+0000 7f1039e69640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1034114000 con 0x7f1034103520 2024-05-27T07:48:25.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f10337fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1034103520 0x7f10341a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f1033fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1034107e60 0x7f103419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f1032ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10341073e0 0x7f10341a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f10337fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1034103520 0x7f10341a04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33570/0 (socket says 172.21.15.114:33570) 2024-05-27T07:48:25.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f10337fe640 1 -- 172.21.15.114:0/770289371 learned_addr learned my addr 172.21.15.114:0/770289371 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f1033fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1034107e60 0x7f103419a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38332/0 (socket says 172.21.15.114:38332) 2024-05-27T07:48:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f10337fe640 1 -- 172.21.15.114:0/770289371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10341073e0 msgr2=0x7f10341a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f10337fe640 1 --2- 172.21.15.114:0/770289371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10341073e0 0x7f10341a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f10337fe640 1 -- 172.21.15.114:0/770289371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1034107e60 msgr2=0x7f103419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f10337fe640 1 --2- 172.21.15.114:0/770289371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1034107e60 0x7f103419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f10337fe640 1 -- 172.21.15.114:0/770289371 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103419ad90 con 0x7f1034103520 2024-05-27T07:48:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f10337fe640 1 --2- 172.21.15.114:0/770289371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1034103520 0x7f10341a04e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f102400e9c0 tx=0x7f102400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f1030ff9640 1 -- 172.21.15.114:0/770289371 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102400cd80 con 0x7f1034103520 2024-05-27T07:48:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f1030ff9640 1 -- 172.21.15.114:0/770289371 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10240054d0 con 0x7f1034103520 2024-05-27T07:48:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f1039e69640 1 -- 172.21.15.114:0/770289371 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f103419b020 con 0x7f1034103520 2024-05-27T07:48:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.305+0000 7f1030ff9640 1 -- 172.21.15.114:0/770289371 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1024010430 con 0x7f1034103520 2024-05-27T07:48:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.306+0000 7f1039e69640 1 -- 172.21.15.114:0/770289371 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1034110330 con 0x7f1034103520 2024-05-27T07:48:25.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.308+0000 7f1040df4740 1 Processor -- start 2024-05-27T07:48:25.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.308+0000 7f1040df4740 1 -- start start 2024-05-27T07:48:25.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.308+0000 7f1040df4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd2553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.308+0000 7f1040df4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce8b2f0 0x55fddd255910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.308+0000 7f1040df4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce934d0 0x55fddd255e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.308+0000 7f1040df4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fddd1c7c30 con 0x55fddceb41b0 2024-05-27T07:48:25.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.308+0000 7f1040df4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fddd1b77d0 con 0x55fddce8b2f0 2024-05-27T07:48:25.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.308+0000 7f1040df4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fddd1bb810 con 0x55fddce934d0 2024-05-27T07:48:25.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.308+0000 7f1033fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd2553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.308+0000 7f1033fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd2553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38348/0 (socket says 172.21.15.114:38348) 2024-05-27T07:48:25.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.308+0000 7f1033fff640 1 -- 172.21.15.114:0/223851260 learned_addr learned my addr 172.21.15.114:0/223851260 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:25.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.308+0000 7f1032ffd640 1 --2- 172.21.15.114:0/223851260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce934d0 0x55fddd255e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.308+0000 7f1030ff9640 1 -- 172.21.15.114:0/770289371 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f1024005020 con 0x7f1034103520 2024-05-27T07:48:25.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.309+0000 7f1033fff640 1 -- 172.21.15.114:0/223851260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce934d0 msgr2=0x55fddd255e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:25.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.309+0000 7f1033fff640 1 --2- 172.21.15.114:0/223851260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce934d0 0x55fddd255e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.309+0000 7f1033fff640 1 -- 172.21.15.114:0/223851260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce8b2f0 msgr2=0x55fddd255910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:25.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.309+0000 7f1033fff640 1 --2- 172.21.15.114:0/223851260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce8b2f0 0x55fddd255910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.309+0000 7f1033fff640 1 -- 172.21.15.114:0/223851260 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fddcecc210 con 0x55fddceb41b0 2024-05-27T07:48:25.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.309+0000 7f1033fff640 1 --2- 172.21.15.114:0/223851260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd2553d0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f102c009fd0 tx=0x7f102c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d94a425dcf8ae0b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:25.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.309+0000 7f1019ffb640 1 -- 172.21.15.114:0/223851260 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c004d20 con 0x55fddceb41b0 2024-05-27T07:48:25.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.309+0000 7f1019ffb640 1 -- 172.21.15.114:0/223851260 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f102c004050 con 0x55fddceb41b0 2024-05-27T07:48:25.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.309+0000 7f1019ffb640 1 -- 172.21.15.114:0/223851260 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c005280 con 0x55fddceb41b0 2024-05-27T07:48:25.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.309+0000 7f1030ff9640 1 --2- 172.21.15.114:0/770289371 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f101c0b2950 0x7f101c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.309+0000 7f10337fe640 1 --2- 172.21.15.114:0/770289371 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f101c0b2950 0x7f101c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.310+0000 7f1030ff9640 1 -- 172.21.15.114:0/770289371 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9510+0+0 (secure 0 0 0) 0x7f10240e2e00 con 0x7f1034103520 2024-05-27T07:48:25.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.310+0000 7f1040df4740 1 -- 172.21.15.114:0/223851260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 msgr2=0x55fddd2553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:25.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.310+0000 7f10337fe640 1 --2- 172.21.15.114:0/770289371 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f101c0b2950 0x7f101c0b4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x55fddd252ec0 tx=0x7f1024076630 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:25.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.310+0000 7f1040df4740 1 --2- 172.21.15.114:0/223851260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd2553d0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f102c009fd0 tx=0x7f102c0118d0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.310+0000 7f1040df4740 1 -- 172.21.15.114:0/223851260 shutdown_connections 2024-05-27T07:48:25.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.310+0000 7f1040df4740 1 --2- 172.21.15.114:0/223851260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce934d0 0x55fddd255e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.310+0000 7f1040df4740 1 --2- 172.21.15.114:0/223851260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce8b2f0 0x55fddd255910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.310+0000 7f1040df4740 1 --2- 172.21.15.114:0/223851260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd2553d0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.310+0000 7f1040df4740 1 -- 172.21.15.114:0/223851260 >> 172.21.15.114:0/223851260 conn(0x55fddcea8300 msgr2=0x55fddd251b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.310+0000 7f1040df4740 1 -- 172.21.15.114:0/223851260 shutdown_connections 2024-05-27T07:48:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.310+0000 7f1040df4740 1 -- 172.21.15.114:0/223851260 wait complete. 2024-05-27T07:48:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1040df4740 1 -- start start 2024-05-27T07:48:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1040df4740 1 Processor -- start 2024-05-27T07:48:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1040df4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce8b2f0 0x55fddd261480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1040df4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce934d0 0x55fddd2619c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1040df4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd25f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1040df4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fddce9d790 con 0x55fddceb41b0 2024-05-27T07:48:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1040df4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fddd173890 con 0x55fddce934d0 2024-05-27T07:48:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1040df4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fddd262500 con 0x55fddce8b2f0 2024-05-27T07:48:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1040df4740 10 client.0 _ll_register_callbacks cb 0x55fddcecba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1033fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd25f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1033fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd25f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38352/0 (socket says 172.21.15.114:38352) 2024-05-27T07:48:25.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1033fff640 1 -- 172.21.15.114:0/1937958447 learned_addr learned my addr 172.21.15.114:0/1937958447 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:25.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1033fff640 1 --2- 172.21.15.114:0/1937958447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce8b2f0 0x55fddd261480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.311+0000 7f1032ffd640 1 --2- 172.21.15.114:0/1937958447 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce934d0 0x55fddd2619c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1033fff640 1 -- 172.21.15.114:0/1937958447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce8b2f0 msgr2=0x55fddd261480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:25.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1033fff640 1 --2- 172.21.15.114:0/1937958447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce8b2f0 0x55fddd261480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1033fff640 1 -- 172.21.15.114:0/1937958447 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce934d0 msgr2=0x55fddd2619c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:25.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1033fff640 1 --2- 172.21.15.114:0/1937958447 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce934d0 0x55fddd2619c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1033fff640 1 -- 172.21.15.114:0/1937958447 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fddcf98620 con 0x55fddceb41b0 2024-05-27T07:48:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1033fff640 1 --2- 172.21.15.114:0/1937958447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd25f900 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f102c003ea0 tx=0x7f102c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ddd962505e2b5fb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1012ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:48:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1012ffd640 1 -- 172.21.15.114:0/1937958447 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c02d060 con 0x55fddceb41b0 2024-05-27T07:48:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1012ffd640 1 -- 172.21.15.114:0/1937958447 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9510+0+0 (secure 0 0 0) 0x7f102c004640 con 0x55fddceb41b0 2024-05-27T07:48:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1040df4740 1 -- 172.21.15.114:0/1937958447 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f102c004050 con 0x55fddceb41b0 2024-05-27T07:48:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1012ffd640 1 -- 172.21.15.114:0/1937958447 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f102c034050 con 0x55fddceb41b0 2024-05-27T07:48:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1040df4740 10 client.17436 fetch_fsmap learned FSMap version 48 2024-05-27T07:48:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1040df4740 1 -- 172.21.15.114:0/1937958447 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55fddd26e800 con 0x55fddceb41b0 2024-05-27T07:48:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1012ffd640 1 -- 172.21.15.114:0/1937958447 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f102c004050 con 0x55fddceb41b0 2024-05-27T07:48:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1012ffd640 1 -- 172.21.15.114:0/1937958447 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 7120+0+0 (secure 0 0 0) 0x7f102c025040 con 0x55fddceb41b0 2024-05-27T07:48:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1040df4740 10 client.17436 fetch_fsmap finished waiting for FSMap version 48 2024-05-27T07:48:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1040df4740 10 client.17436 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.312+0000 7f1040df4740 20 client.17436 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.313+0000 7f1040df4740 1 --2- 172.21.15.114:0/1937958447 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fddce72d80 0x55fddd271c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.313+0000 7f1040df4740 4 client.17436 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:48:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.313+0000 7f1040df4740 1 -- 172.21.15.114:0/1937958447 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fddd1c9b80 con 0x55fddce72d80 2024-05-27T07:48:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.313+0000 7f1032ffd640 1 --2- 172.21.15.114:0/1937958447 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fddce72d80 0x55fddd271c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.313+0000 7f1032ffd640 1 --2- 172.21.15.114:0/1937958447 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fddce72d80 0x55fddd271c10 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2439b9370277575b server_cookie=32294c1021dce360 in_seq=0 out_seq=0 2024-05-27T07:48:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.313+0000 7f1012ffd640 10 client.17436 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1012ffd640 1 -- 172.21.15.114:0/1937958447 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fddd1c9b80 con 0x55fddce72d80 2024-05-27T07:48:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1012ffd640 10 client.17436 handle_command_reply: tid=0 2024-05-27T07:48:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1040df4740 1 client.17436 shutdown 2024-05-27T07:48:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1040df4740 1 -- 172.21.15.114:0/1937958447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 msgr2=0x55fddd25f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1040df4740 1 --2- 172.21.15.114:0/1937958447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd25f900 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f102c003ea0 tx=0x7f102c005bd0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1040df4740 1 -- 172.21.15.114:0/1937958447 shutdown_connections 2024-05-27T07:48:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1040df4740 1 --2- 172.21.15.114:0/1937958447 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fddce72d80 0x55fddd271c10 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1040df4740 1 --2- 172.21.15.114:0/1937958447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd25f900 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1040df4740 1 --2- 172.21.15.114:0/1937958447 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce934d0 0x55fddd2619c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1012ffd640 0 client.17436 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1040df4740 1 --2- 172.21.15.114:0/1937958447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce8b2f0 0x55fddd261480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1040df4740 1 -- 172.21.15.114:0/1937958447 >> 172.21.15.114:0/1937958447 conn(0x55fddcea8300 msgr2=0x55fddd2551a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:25.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1040df4740 1 -- 172.21.15.114:0/1937958447 shutdown_connections 2024-05-27T07:48:25.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1040df4740 1 -- 172.21.15.114:0/1937958447 wait complete. 2024-05-27T07:48:25.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.316+0000 7f1040df4740 20 client.17436 trim_cache size 0 max 16384 2024-05-27T07:48:25.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1040df4740 1 Processor -- start 2024-05-27T07:48:25.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1040df4740 1 -- start start 2024-05-27T07:48:25.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1040df4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddce8b2f0 0x55fddd2595a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1040df4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce934d0 0x55fddd25f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1040df4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddceb41b0 0x55fddd25f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1040df4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f10341af290 con 0x55fddce8b2f0 2024-05-27T07:48:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1040df4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f10341af960 con 0x55fddce934d0 2024-05-27T07:48:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1040df4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f10341ad100 con 0x55fddceb41b0 2024-05-27T07:48:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1033fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddce8b2f0 0x55fddd2595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1032ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce934d0 0x55fddd25f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1033fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddce8b2f0 0x55fddd2595a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38356/0 (socket says 172.21.15.114:38356) 2024-05-27T07:48:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1033fff640 1 -- 172.21.15.114:0/932786754 learned_addr learned my addr 172.21.15.114:0/932786754 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1033fff640 1 --2- 172.21.15.114:0/932786754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddceb41b0 0x55fddd25f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1032ffd640 1 -- 172.21.15.114:0/932786754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddceb41b0 msgr2=0x55fddd25f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1032ffd640 1 --2- 172.21.15.114:0/932786754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddceb41b0 0x55fddd25f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1032ffd640 1 -- 172.21.15.114:0/932786754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddce8b2f0 msgr2=0x55fddd2595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1032ffd640 1 --2- 172.21.15.114:0/932786754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddce8b2f0 0x55fddd2595a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1032ffd640 1 -- 172.21.15.114:0/932786754 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fddd25de50 con 0x55fddce934d0 2024-05-27T07:48:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.333+0000 7f1033fff640 1 --2- 172.21.15.114:0/932786754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddce8b2f0 0x55fddd2595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:25.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.334+0000 7f1032ffd640 1 --2- 172.21.15.114:0/932786754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce934d0 0x55fddd25f1e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f102802a930 tx=0x7f1028032df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7410f4e03a3769b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:25.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.334+0000 7f10197fa640 1 -- 172.21.15.114:0/932786754 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1028035040 con 0x55fddce934d0 2024-05-27T07:48:25.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.334+0000 7f10197fa640 1 -- 172.21.15.114:0/932786754 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f102803a670 con 0x55fddce934d0 2024-05-27T07:48:25.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.334+0000 7f10197fa640 1 -- 172.21.15.114:0/932786754 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1028003850 con 0x55fddce934d0 2024-05-27T07:48:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.335+0000 7f1040df4740 1 -- 172.21.15.114:0/932786754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce934d0 msgr2=0x55fddd25f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.335+0000 7f1040df4740 1 --2- 172.21.15.114:0/932786754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce934d0 0x55fddd25f1e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f102802a930 tx=0x7f1028032df0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.335+0000 7f1040df4740 1 -- 172.21.15.114:0/932786754 shutdown_connections 2024-05-27T07:48:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.335+0000 7f1040df4740 1 --2- 172.21.15.114:0/932786754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddceb41b0 0x55fddd25f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.335+0000 7f1040df4740 1 --2- 172.21.15.114:0/932786754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce934d0 0x55fddd25f1e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.335+0000 7f1040df4740 1 --2- 172.21.15.114:0/932786754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddce8b2f0 0x55fddd2595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.335+0000 7f1040df4740 1 -- 172.21.15.114:0/932786754 >> 172.21.15.114:0/932786754 conn(0x55fddcea8300 msgr2=0x55fddd258e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.335+0000 7f1040df4740 1 -- 172.21.15.114:0/932786754 shutdown_connections 2024-05-27T07:48:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.335+0000 7f1040df4740 1 -- 172.21.15.114:0/932786754 wait complete. 2024-05-27T07:48:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.336+0000 7f1040df4740 1 -- start start 2024-05-27T07:48:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.336+0000 7f1040df4740 1 Processor -- start 2024-05-27T07:48:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.336+0000 7f1040df4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce8b2f0 0x55fddd2668c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.336+0000 7f1040df4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce934d0 0x55fddd266e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.336+0000 7f1040df4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd2d8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1040df4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f10341ad750 con 0x55fddceb41b0 2024-05-27T07:48:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1040df4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f10341aebc0 con 0x55fddce8b2f0 2024-05-27T07:48:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1040df4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f10341ade20 con 0x55fddce934d0 2024-05-27T07:48:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1040df4740 10 client.0 _ll_register_callbacks cb 0x7f1024005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1033fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd2d8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1032ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce934d0 0x55fddd266e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1033fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd2d8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38372/0 (socket says 172.21.15.114:38372) 2024-05-27T07:48:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1033fff640 1 -- 172.21.15.114:0/2056667031 learned_addr learned my addr 172.21.15.114:0/2056667031 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1033fff640 1 --2- 172.21.15.114:0/2056667031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce8b2f0 0x55fddd2668c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1033fff640 1 -- 172.21.15.114:0/2056667031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce934d0 msgr2=0x55fddd266e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1033fff640 1 --2- 172.21.15.114:0/2056667031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce934d0 0x55fddd266e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1033fff640 1 -- 172.21.15.114:0/2056667031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce8b2f0 msgr2=0x55fddd2668c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1033fff640 1 --2- 172.21.15.114:0/2056667031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce8b2f0 0x55fddd2668c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1033fff640 1 -- 172.21.15.114:0/2056667031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fddd2676f0 con 0x55fddceb41b0 2024-05-27T07:48:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1033fff640 1 --2- 172.21.15.114:0/2056667031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd2d8260 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f102c006010 tx=0x7f102c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5e491006a6fe43 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.337+0000 7f1012ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:48:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1012ffd640 1 -- 172.21.15.114:0/2056667031 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c02d040 con 0x55fddceb41b0 2024-05-27T07:48:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1040df4740 1 -- 172.21.15.114:0/2056667031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f102803a670 con 0x55fddceb41b0 2024-05-27T07:48:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1012ffd640 1 -- 172.21.15.114:0/2056667031 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9510+0+0 (secure 0 0 0) 0x7f102c005640 con 0x55fddceb41b0 2024-05-27T07:48:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1012ffd640 1 -- 172.21.15.114:0/2056667031 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f102c048050 con 0x55fddceb41b0 2024-05-27T07:48:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1040df4740 10 client.17442 fetch_fsmap learned FSMap version 48 2024-05-27T07:48:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1040df4740 1 -- 172.21.15.114:0/2056667031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55fddd2d9860 con 0x55fddceb41b0 2024-05-27T07:48:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1012ffd640 1 -- 172.21.15.114:0/2056667031 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f102803a670 con 0x55fddceb41b0 2024-05-27T07:48:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1012ffd640 1 -- 172.21.15.114:0/2056667031 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 7120+0+0 (secure 0 0 0) 0x7f102c00f4e0 con 0x55fddceb41b0 2024-05-27T07:48:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1040df4740 10 client.17442 fetch_fsmap finished waiting for FSMap version 48 2024-05-27T07:48:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1040df4740 10 client.17442 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1040df4740 20 client.17442 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1040df4740 1 --2- 172.21.15.114:0/2056667031 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fddce481c0 0x55fddd2dbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1040df4740 4 client.17442 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:48:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.338+0000 7f1040df4740 1 -- 172.21.15.114:0/2056667031 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fddd1c3360 con 0x55fddce481c0 2024-05-27T07:48:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.339+0000 7f1032ffd640 1 --2- 172.21.15.114:0/2056667031 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fddce481c0 0x55fddd2dbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.339+0000 7f1012ffd640 10 client.17442 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.339+0000 7f1032ffd640 1 --2- 172.21.15.114:0/2056667031 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fddce481c0 0x55fddd2dbe30 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fd5b1914da38d56 server_cookie=2a2ada3c1e1077e5 in_seq=0 out_seq=0 2024-05-27T07:48:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.710+0000 7f1012ffd640 1 -- 172.21.15.114:0/2056667031 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55fddd1c3360 con 0x55fddce481c0 2024-05-27T07:48:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.710+0000 7f1012ffd640 10 client.17442 handle_command_reply: tid=0 2024-05-27T07:48:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.710+0000 7f1040df4740 1 client.17442 shutdown 2024-05-27T07:48:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.710+0000 7f1040df4740 1 -- 172.21.15.114:0/2056667031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 msgr2=0x55fddd2d8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.710+0000 7f1040df4740 1 --2- 172.21.15.114:0/2056667031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd2d8260 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f102c006010 tx=0x7f102c009500 comp rx=0 tx=0).stop 2024-05-27T07:48:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.710+0000 7f1040df4740 1 -- 172.21.15.114:0/2056667031 shutdown_connections 2024-05-27T07:48:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.710+0000 7f1040df4740 1 --2- 172.21.15.114:0/2056667031 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fddce481c0 0x55fddd2dbe30 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.710+0000 7f1012ffd640 0 client.17442 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.710+0000 7f1040df4740 1 --2- 172.21.15.114:0/2056667031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fddceb41b0 0x55fddd2d8260 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.710+0000 7f1040df4740 1 --2- 172.21.15.114:0/2056667031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fddce934d0 0x55fddd266e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.710+0000 7f1040df4740 1 --2- 172.21.15.114:0/2056667031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddce8b2f0 0x55fddd2668c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.710+0000 7f1040df4740 1 -- 172.21.15.114:0/2056667031 >> 172.21.15.114:0/2056667031 conn(0x55fddcea8300 msgr2=0x55fddd25e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.710+0000 7f1040df4740 1 -- 172.21.15.114:0/2056667031 shutdown_connections 2024-05-27T07:48:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.711+0000 7f1040df4740 1 -- 172.21.15.114:0/2056667031 wait complete. 2024-05-27T07:48:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.711+0000 7f1040df4740 20 client.17442 trim_cache size 0 max 16384 2024-05-27T07:48:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:48:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:48:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:48:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.711+0000 7f1039e69640 1 -- 172.21.15.114:0/770289371 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f101c0b2950 msgr2=0x7f101c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.711+0000 7f1039e69640 1 --2- 172.21.15.114:0/770289371 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f101c0b2950 0x7f101c0b4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x55fddd252ec0 tx=0x7f1024076630 comp rx=0 tx=0).stop 2024-05-27T07:48:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.711+0000 7f1039e69640 1 -- 172.21.15.114:0/770289371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1034103520 msgr2=0x7f10341a04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.711+0000 7f1039e69640 1 --2- 172.21.15.114:0/770289371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1034103520 0x7f10341a04e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f102400e9c0 tx=0x7f102400ee80 comp rx=0 tx=0).stop 2024-05-27T07:48:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.711+0000 7f1039e69640 1 -- 172.21.15.114:0/770289371 shutdown_connections 2024-05-27T07:48:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.711+0000 7f1039e69640 1 --2- 172.21.15.114:0/770289371 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f101c0b2950 0x7f101c0b4e60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.711+0000 7f1039e69640 1 --2- 172.21.15.114:0/770289371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1034107e60 0x7f103419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.711+0000 7f1039e69640 1 --2- 172.21.15.114:0/770289371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10341073e0 0x7f10341a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.711+0000 7f1039e69640 1 --2- 172.21.15.114:0/770289371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1034103520 0x7f10341a04e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.711+0000 7f1039e69640 1 -- 172.21.15.114:0/770289371 >> 172.21.15.114:0/770289371 conn(0x7f10340fd9c0 msgr2=0x7f10341058e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.712+0000 7f1039e69640 1 -- 172.21.15.114:0/770289371 shutdown_connections 2024-05-27T07:48:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:25.712+0000 7f1039e69640 1 -- 172.21.15.114:0/770289371 wait complete. 2024-05-27T07:48:25.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:25 smithi163 ceph-mon[36451]: pgmap v281: 129 pgs: 129 active+clean; 72 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 331 op/s 2024-05-27T07:48:25.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:25 smithi186 ceph-mon[36647]: pgmap v281: 129 pgs: 129 active+clean; 72 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 331 op/s 2024-05-27T07:48:25.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:25 smithi114 ceph-mon[32038]: pgmap v281: 129 pgs: 129 active+clean; 72 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 331 op/s 2024-05-27T07:48:26.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 877547 11.07 MB/sec execute 278 sec latency 226.015 ms 2024-05-27T07:48:27.275 INFO:tasks.workunit.client.0.smithi114.stdout: 1 879827 11.08 MB/sec execute 279 sec latency 28.798 ms 2024-05-27T07:48:27.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:48:27.717 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:48:27.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:27 smithi163 ceph-mon[36451]: pgmap v282: 129 pgs: 129 active+clean; 78 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 262 op/s 2024-05-27T07:48:27.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:27 smithi186 ceph-mon[36647]: pgmap v282: 129 pgs: 129 active+clean; 78 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 262 op/s 2024-05-27T07:48:27.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:27 smithi114 ceph-mon[32038]: pgmap v282: 129 pgs: 129 active+clean; 78 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 262 op/s 2024-05-27T07:48:28.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 881283 11.06 MB/sec execute 280 sec latency 90.782 ms 2024-05-27T07:48:29.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 883485 11.07 MB/sec execute 281 sec latency 52.353 ms 2024-05-27T07:48:29.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:29 smithi114 ceph-mon[32038]: pgmap v283: 129 pgs: 129 active+clean; 78 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 262 op/s 2024-05-27T07:48:29.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.568+0000 7f13908ca700 1 -- 172.21.15.114:0/1231225397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c108730 msgr2=0x7f138c110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:29.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.568+0000 7f13908ca700 1 --2- 172.21.15.114:0/1231225397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c108730 0x7f138c110090 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f138400b0d0 tx=0x7f138400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.568+0000 7f13908ca700 1 -- 172.21.15.114:0/1231225397 shutdown_connections 2024-05-27T07:48:29.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.568+0000 7f13908ca700 1 --2- 172.21.15.114:0/1231225397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c108730 0x7f138c110090 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.568+0000 7f13908ca700 1 --2- 172.21.15.114:0/1231225397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f138c1034b0 0x7f138c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.568+0000 7f13908ca700 1 --2- 172.21.15.114:0/1231225397 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f138c107de0 0x7f138c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.568+0000 7f13908ca700 1 -- 172.21.15.114:0/1231225397 >> 172.21.15.114:0/1231225397 conn(0x7f138c0fede0 msgr2=0x7f138c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:29.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.568+0000 7f13908ca700 1 -- 172.21.15.114:0/1231225397 shutdown_connections 2024-05-27T07:48:29.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.568+0000 7f13908ca700 1 -- 172.21.15.114:0/1231225397 wait complete. 2024-05-27T07:48:29.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.569+0000 7f13908ca700 1 Processor -- start 2024-05-27T07:48:29.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.569+0000 7f13908ca700 1 -- start start 2024-05-27T07:48:29.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.569+0000 7f13908ca700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f138c1034b0 0x7f138c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.569+0000 7f13908ca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c107de0 0x7f138c19b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.569+0000 7f13908ca700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f138c108730 0x7f138c19f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.569+0000 7f13908ca700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f138c113360 con 0x7f138c107de0 2024-05-27T07:48:29.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.569+0000 7f13908ca700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f138c1131e0 con 0x7f138c1034b0 2024-05-27T07:48:29.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.569+0000 7f13908ca700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f138c1134e0 con 0x7f138c108730 2024-05-27T07:48:29.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.569+0000 7f1389d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c107de0 0x7f138c19b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.569+0000 7f138a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f138c1034b0 0x7f138c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.569+0000 7f1389d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c107de0 0x7f138c19b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38384/0 (socket says 172.21.15.114:38384) 2024-05-27T07:48:29.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f1389d9b700 1 -- 172.21.15.114:0/3137071739 learned_addr learned my addr 172.21.15.114:0/3137071739 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:29.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f138ad9d700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f138c108730 0x7f138c19f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f1389d9b700 1 -- 172.21.15.114:0/3137071739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f138c108730 msgr2=0x7f138c19f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:29.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f1389d9b700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f138c108730 0x7f138c19f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f1389d9b700 1 -- 172.21.15.114:0/3137071739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f138c1034b0 msgr2=0x7f138c19adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:29.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f1389d9b700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f138c1034b0 0x7f138c19adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f1389d9b700 1 -- 172.21.15.114:0/3137071739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138c19fc90 con 0x7f138c107de0 2024-05-27T07:48:29.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f138a59c700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f138c1034b0 0x7f138c19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:29.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f1389d9b700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c107de0 0x7f138c19b360 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f138000e970 tx=0x7f138000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:29.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f137b7fe700 1 -- 172.21.15.114:0/3137071739 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1380004a30 con 0x7f138c107de0 2024-05-27T07:48:29.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f137b7fe700 1 -- 172.21.15.114:0/3137071739 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f138000cc90 con 0x7f138c107de0 2024-05-27T07:48:29.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f137b7fe700 1 -- 172.21.15.114:0/3137071739 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1380005610 con 0x7f138c107de0 2024-05-27T07:48:29.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f13908ca700 1 -- 172.21.15.114:0/3137071739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f138c19ff80 con 0x7f138c107de0 2024-05-27T07:48:29.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.570+0000 7f13908ca700 1 -- 172.21.15.114:0/3137071739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f138c1a78a0 con 0x7f138c107de0 2024-05-27T07:48:29.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.573+0000 7f137b7fe700 1 -- 172.21.15.114:0/3137071739 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f1380021440 con 0x7f138c107de0 2024-05-27T07:48:29.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.574+0000 7f137b7fe700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f13740b29f0 0x7f13740b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.574+0000 7f138a59c700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f13740b29f0 0x7f13740b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.574+0000 7f139f2dfb80 1 Processor -- start 2024-05-27T07:48:29.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.574+0000 7f137b7fe700 1 -- 172.21.15.114:0/3137071739 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9510+0+0 (secure 0 0 0) 0x7f13800e2b20 con 0x7f138c107de0 2024-05-27T07:48:29.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.574+0000 7f139f2dfb80 1 -- start start 2024-05-27T07:48:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.574+0000 7f138a59c700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f13740b29f0 0x7f13740b4f00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f137c011490 tx=0x7f137c009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f139f2dfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c425e40 0x55640c5a9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f139f2dfb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a9c50 0x55640c5a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f139f2dfb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c5a3f80 0x55640c5a4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f139f2dfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55640c4d2c90 con 0x55640c425e40 2024-05-27T07:48:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f139f2dfb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55640c4c2830 con 0x55640c5a9c50 2024-05-27T07:48:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f139f2dfb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55640c4c6870 con 0x55640c5a3f80 2024-05-27T07:48:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f138a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c425e40 0x55640c5a9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f138ad9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c5a3f80 0x55640c5a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f138a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c425e40 0x55640c5a9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38390/0 (socket says 172.21.15.114:38390) 2024-05-27T07:48:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f138a59c700 1 -- 172.21.15.114:0/3649550000 learned_addr learned my addr 172.21.15.114:0/3649550000 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f1389d9b700 1 --2- 172.21.15.114:0/3649550000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a9c50 0x55640c5a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f138a59c700 1 -- 172.21.15.114:0/3649550000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c5a3f80 msgr2=0x55640c5a4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f138a59c700 1 --2- 172.21.15.114:0/3649550000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c5a3f80 0x55640c5a4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f138a59c700 1 -- 172.21.15.114:0/3649550000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a9c50 msgr2=0x55640c5a3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f138a59c700 1 --2- 172.21.15.114:0/3649550000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a9c50 0x55640c5a3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f138a59c700 1 -- 172.21.15.114:0/3649550000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55640c4128d0 con 0x55640c425e40 2024-05-27T07:48:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f1389d9b700 1 --2- 172.21.15.114:0/3649550000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a9c50 0x55640c5a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f138a59c700 1 --2- 172.21.15.114:0/3649550000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c425e40 0x55640c5a9710 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f137c0043c0 tx=0x7f137c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6503fcc03db414a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f1378ff9700 1 -- 172.21.15.114:0/3649550000 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c003d10 con 0x55640c425e40 2024-05-27T07:48:29.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f1378ff9700 1 -- 172.21.15.114:0/3649550000 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f137c011e50 con 0x55640c425e40 2024-05-27T07:48:29.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.575+0000 7f1378ff9700 1 -- 172.21.15.114:0/3649550000 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c01e6c0 con 0x55640c425e40 2024-05-27T07:48:29.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.576+0000 7f139f2dfb80 1 -- 172.21.15.114:0/3649550000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c425e40 msgr2=0x55640c5a9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:29.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.576+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/3649550000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c425e40 0x55640c5a9710 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f137c0043c0 tx=0x7f137c014000 comp rx=0 tx=0).stop 2024-05-27T07:48:29.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.576+0000 7f139f2dfb80 1 -- 172.21.15.114:0/3649550000 shutdown_connections 2024-05-27T07:48:29.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.576+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/3649550000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c5a3f80 0x55640c5a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.576+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/3649550000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c425e40 0x55640c5a9710 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.576+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/3649550000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a9c50 0x55640c5a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.576+0000 7f139f2dfb80 1 -- 172.21.15.114:0/3649550000 >> 172.21.15.114:0/3649550000 conn(0x55640c480af0 msgr2=0x55640c59e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:29.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.577+0000 7f139f2dfb80 1 -- 172.21.15.114:0/3649550000 shutdown_connections 2024-05-27T07:48:29.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.577+0000 7f139f2dfb80 1 -- 172.21.15.114:0/3649550000 wait complete. 2024-05-27T07:48:29.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.577+0000 7f139f2dfb80 1 -- start start 2024-05-27T07:48:29.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.577+0000 7f139f2dfb80 1 Processor -- start 2024-05-27T07:48:29.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.577+0000 7f139f2dfb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c425e40 0x55640c5ad390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.577+0000 7f139f2dfb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c5a3f80 0x55640c5ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.577+0000 7f139f2dfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 0x55640c5ade10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.577+0000 7f139f2dfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55640c3c8fa0 con 0x55640c5a9c50 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.577+0000 7f139f2dfb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55640c4647c0 con 0x55640c425e40 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.577+0000 7f139f2dfb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55640c5b0740 con 0x55640c5a3f80 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f138ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 0x55640c5ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f138a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c425e40 0x55640c5ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f1389d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c5a3f80 0x55640c5ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f138ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 0x55640c5ade10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38394/0 (socket says 172.21.15.114:38394) 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f139f2dfb80 10 client.0 _ll_register_callbacks cb 0x55640c45d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f138ad9d700 1 -- 172.21.15.114:0/1132149870 learned_addr learned my addr 172.21.15.114:0/1132149870 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f138a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c425e40 0x55640c5ad390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:57952/0 (socket says 172.21.15.114:57952) 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f1389d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c5a3f80 0x55640c5ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33648/0 (socket says 172.21.15.114:33648) 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f138a59c700 1 -- 172.21.15.114:0/1132149870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c5a3f80 msgr2=0x55640c5ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f138a59c700 1 --2- 172.21.15.114:0/1132149870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c5a3f80 0x55640c5ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f138a59c700 1 -- 172.21.15.114:0/1132149870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 msgr2=0x55640c5ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:29.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f138a59c700 1 --2- 172.21.15.114:0/1132149870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 0x55640c5ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f138a59c700 1 -- 172.21.15.114:0/1132149870 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55640c461690 con 0x55640c425e40 2024-05-27T07:48:29.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f138ad9d700 1 --2- 172.21.15.114:0/1132149870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 0x55640c5ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:29.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f138a59c700 1 --2- 172.21.15.114:0/1132149870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c425e40 0x55640c5ad390 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f137c0043c0 tx=0x7f137c026630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1350118c0e200bee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:29.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f1371ffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:48:29.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f1371ffb700 1 -- 172.21.15.114:0/1132149870 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c005050 con 0x55640c425e40 2024-05-27T07:48:29.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1132149870 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f137c011e50 con 0x55640c425e40 2024-05-27T07:48:29.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f1371ffb700 1 -- 172.21.15.114:0/1132149870 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9510+0+0 (secure 0 0 0) 0x7f137c0051f0 con 0x55640c425e40 2024-05-27T07:48:29.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.578+0000 7f1371ffb700 1 -- 172.21.15.114:0/1132149870 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f137c039050 con 0x55640c425e40 2024-05-27T07:48:29.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.579+0000 7f139f2dfb80 10 client.26909 fetch_fsmap learned FSMap version 48 2024-05-27T07:48:29.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.579+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1132149870 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55640c5bb1f0 con 0x55640c425e40 2024-05-27T07:48:29.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.579+0000 7f1371ffb700 1 -- 172.21.15.114:0/1132149870 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f137c011e50 con 0x55640c425e40 2024-05-27T07:48:29.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.579+0000 7f1371ffb700 1 -- 172.21.15.114:0/1132149870 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 48) ==== 7120+0+0 (secure 0 0 0) 0x7f137c005510 con 0x55640c425e40 2024-05-27T07:48:29.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.579+0000 7f139f2dfb80 10 client.26909 fetch_fsmap finished waiting for FSMap version 48 2024-05-27T07:48:29.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.579+0000 7f139f2dfb80 10 client.26909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:29.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.579+0000 7f139f2dfb80 20 client.26909 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:29.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.579+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/1132149870 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55640c5bd720 0x55640c5bfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.579+0000 7f139f2dfb80 4 client.26909 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:48:29.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.579+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1132149870 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55640c56f9f0 con 0x55640c5bd720 2024-05-27T07:48:29.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.579+0000 7f1389d9b700 1 --2- 172.21.15.114:0/1132149870 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55640c5bd720 0x55640c5bfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.580+0000 7f1371ffb700 10 client.26909 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:29.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.580+0000 7f1389d9b700 1 --2- 172.21.15.114:0/1132149870 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55640c5bd720 0x55640c5bfb40 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=965c73ccc5fd847 server_cookie=ca6b9f9195c6f4ae in_seq=0 out_seq=0 2024-05-27T07:48:29.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.581+0000 7f1371ffb700 1 -- 172.21.15.114:0/1132149870 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55640c56f9f0 con 0x55640c5bd720 2024-05-27T07:48:29.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.581+0000 7f1371ffb700 10 client.26909 handle_command_reply: tid=0 2024-05-27T07:48:29.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.581+0000 7f139f2dfb80 1 client.26909 shutdown 2024-05-27T07:48:29.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.582+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1132149870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c425e40 msgr2=0x55640c5ad390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:29.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.582+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/1132149870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c425e40 0x55640c5ad390 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f137c0043c0 tx=0x7f137c026630 comp rx=0 tx=0).stop 2024-05-27T07:48:29.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.582+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1132149870 shutdown_connections 2024-05-27T07:48:29.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.582+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/1132149870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 0x55640c5ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.582+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/1132149870 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55640c5bd720 0x55640c5bfb40 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.582+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/1132149870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c425e40 0x55640c5ad390 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.582+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/1132149870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c5a3f80 0x55640c5ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.582+0000 7f1371ffb700 0 client.26909 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:29.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.582+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1132149870 >> 172.21.15.114:0/1132149870 conn(0x55640c480af0 msgr2=0x55640c5a2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:29.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.582+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1132149870 shutdown_connections 2024-05-27T07:48:29.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.582+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1132149870 wait complete. 2024-05-27T07:48:29.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.582+0000 7f139f2dfb80 20 client.26909 trim_cache size 0 max 16384 2024-05-27T07:48:29.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.602+0000 7f139f2dfb80 1 Processor -- start 2024-05-27T07:48:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.602+0000 7f139f2dfb80 1 -- start start 2024-05-27T07:48:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f139f2dfb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c425e40 0x55640c5a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f139f2dfb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a3f80 0x55640c5ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f139f2dfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 0x55640c5b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f139f2dfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f138c1ae620 con 0x55640c5a9c50 2024-05-27T07:48:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f139f2dfb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f138c1aecf0 con 0x55640c5a3f80 2024-05-27T07:48:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f139f2dfb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f138c1ac490 con 0x55640c425e40 2024-05-27T07:48:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f138ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 0x55640c5b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f138ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 0x55640c5b97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38400/0 (socket says 172.21.15.114:38400) 2024-05-27T07:48:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f138ad9d700 1 -- 172.21.15.114:0/4209787684 learned_addr learned my addr 172.21.15.114:0/4209787684 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f138a59c700 1 --2- 172.21.15.114:0/4209787684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c425e40 0x55640c5a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f1389d9b700 1 --2- 172.21.15.114:0/4209787684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a3f80 0x55640c5ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f138ad9d700 1 -- 172.21.15.114:0/4209787684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c425e40 msgr2=0x55640c5a3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:29.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f138ad9d700 1 --2- 172.21.15.114:0/4209787684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c425e40 0x55640c5a3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f138ad9d700 1 -- 172.21.15.114:0/4209787684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a3f80 msgr2=0x55640c5ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:29.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f138ad9d700 1 --2- 172.21.15.114:0/4209787684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a3f80 0x55640c5ac6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f138ad9d700 1 -- 172.21.15.114:0/4209787684 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55640c598fa0 con 0x55640c5a9c50 2024-05-27T07:48:29.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f138ad9d700 1 --2- 172.21.15.114:0/4209787684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 0x55640c5b97e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f138400bce0 tx=0x7f138400bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1274191abe686763 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:29.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f1373fff700 1 -- 172.21.15.114:0/4209787684 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384003a00 con 0x55640c5a9c50 2024-05-27T07:48:29.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f1373fff700 1 -- 172.21.15.114:0/4209787684 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1384020cd0 con 0x55640c5a9c50 2024-05-27T07:48:29.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.603+0000 7f1373fff700 1 -- 172.21.15.114:0/4209787684 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384007b70 con 0x55640c5a9c50 2024-05-27T07:48:29.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.604+0000 7f139f2dfb80 1 -- 172.21.15.114:0/4209787684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 msgr2=0x55640c5b97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:29.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.604+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/4209787684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 0x55640c5b97e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f138400bce0 tx=0x7f138400bd10 comp rx=0 tx=0).stop 2024-05-27T07:48:29.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.604+0000 7f139f2dfb80 1 -- 172.21.15.114:0/4209787684 shutdown_connections 2024-05-27T07:48:29.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.604+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/4209787684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a9c50 0x55640c5b97e0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.604+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/4209787684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c425e40 0x55640c5a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.604+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/4209787684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a3f80 0x55640c5ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.604+0000 7f139f2dfb80 1 -- 172.21.15.114:0/4209787684 >> 172.21.15.114:0/4209787684 conn(0x55640c480af0 msgr2=0x55640c5a8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:29.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.604+0000 7f139f2dfb80 1 -- 172.21.15.114:0/4209787684 shutdown_connections 2024-05-27T07:48:29.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.604+0000 7f139f2dfb80 1 -- 172.21.15.114:0/4209787684 wait complete. 2024-05-27T07:48:29.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f139f2dfb80 1 -- start start 2024-05-27T07:48:29.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f139f2dfb80 1 Processor -- start 2024-05-27T07:48:29.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f139f2dfb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c425e40 0x55640c5b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f139f2dfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a3f80 0x55640c5b6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f139f2dfb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a9c50 0x55640c5c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f139f2dfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f138c1acae0 con 0x55640c5a3f80 2024-05-27T07:48:29.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f139f2dfb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f138c1adf50 con 0x55640c5a9c50 2024-05-27T07:48:29.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f139f2dfb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f138c1ad1b0 con 0x55640c425e40 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f139f2dfb80 10 client.0 _ll_register_callbacks cb 0x55640c45d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f1389d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a3f80 0x55640c5b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f1389d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a3f80 0x55640c5b6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38402/0 (socket says 172.21.15.114:38402) 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f1389d9b700 1 -- 172.21.15.114:0/1647298967 learned_addr learned my addr 172.21.15.114:0/1647298967 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f138a59c700 1 --2- 172.21.15.114:0/1647298967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c425e40 0x55640c5b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f138ad9d700 1 --2- 172.21.15.114:0/1647298967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a9c50 0x55640c5c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f1389d9b700 1 -- 172.21.15.114:0/1647298967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c425e40 msgr2=0x55640c5b5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f1389d9b700 1 --2- 172.21.15.114:0/1647298967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c425e40 0x55640c5b5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f1389d9b700 1 -- 172.21.15.114:0/1647298967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a9c50 msgr2=0x55640c5c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f1389d9b700 1 --2- 172.21.15.114:0/1647298967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a9c50 0x55640c5c49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.605+0000 7f1389d9b700 1 -- 172.21.15.114:0/1647298967 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55640c5c51b0 con 0x55640c5a3f80 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f1389d9b700 1 --2- 172.21.15.114:0/1647298967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a3f80 0x55640c5b6280 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f13800a6c90 tx=0x7f13800cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88809480412ac48f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f1371ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:48:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f1371ffb700 1 -- 172.21.15.114:0/1647298967 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138006e060 con 0x55640c5a3f80 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f1371ffb700 1 -- 172.21.15.114:0/1647298967 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9510+0+0 (secure 0 0 0) 0x7f13800a9c80 con 0x55640c5a3f80 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1647298967 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1384020cd0 con 0x55640c5a3f80 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f1371ffb700 1 -- 172.21.15.114:0/1647298967 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f13800a9530 con 0x55640c5a3f80 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f139f2dfb80 10 client.17460 fetch_fsmap learned FSMap version 48 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1647298967 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55640c627d10 con 0x55640c5a3f80 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f1371ffb700 1 -- 172.21.15.114:0/1647298967 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1384020cd0 con 0x55640c5a3f80 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f1371ffb700 1 -- 172.21.15.114:0/1647298967 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 7120+0+0 (secure 0 0 0) 0x7f138006f250 con 0x55640c5a3f80 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f139f2dfb80 10 client.17460 fetch_fsmap finished waiting for FSMap version 48 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f139f2dfb80 10 client.17460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f139f2dfb80 20 client.17460 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/1647298967 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55640c6283f0 0x55640c62a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f139f2dfb80 4 client.17460 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.606+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1647298967 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55640c4d4be0 con 0x55640c6283f0 2024-05-27T07:48:29.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.607+0000 7f138a59c700 1 --2- 172.21.15.114:0/1647298967 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55640c6283f0 0x55640c62a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.607+0000 7f1371ffb700 10 client.17460 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.607+0000 7f138a59c700 1 --2- 172.21.15.114:0/1647298967 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55640c6283f0 0x55640c62a810 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9a9602bb72fd0e3 server_cookie=81fc137610d552fc in_seq=0 out_seq=0 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f1371ffb700 1 -- 172.21.15.114:0/1647298967 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55640c4d4be0 con 0x55640c6283f0 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f1371ffb700 10 client.17460 handle_command_reply: tid=0 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f139f2dfb80 1 client.17460 shutdown 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1647298967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a3f80 msgr2=0x55640c5b6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/1647298967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a3f80 0x55640c5b6280 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f13800a6c90 tx=0x7f13800cc040 comp rx=0 tx=0).stop 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1647298967 shutdown_connections 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/1647298967 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55640c6283f0 0x55640c62a810 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/1647298967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55640c5a9c50 0x55640c5c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/1647298967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55640c425e40 0x55640c5b5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f1371ffb700 0 client.17460 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f139f2dfb80 1 --2- 172.21.15.114:0/1647298967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55640c5a3f80 0x55640c5b6280 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1647298967 >> 172.21.15.114:0/1647298967 conn(0x55640c480af0 msgr2=0x55640c5a8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:29.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1647298967 shutdown_connections 2024-05-27T07:48:29.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f139f2dfb80 1 -- 172.21.15.114:0/1647298967 wait complete. 2024-05-27T07:48:29.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.608+0000 7f139f2dfb80 20 client.17460 trim_cache size 0 max 16384 2024-05-27T07:48:29.614 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:48:29.614 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:48:29.614 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:48:29.614 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:48:29.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.609+0000 7f13908ca700 1 -- 172.21.15.114:0/3137071739 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f13740b29f0 msgr2=0x7f13740b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:29.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.609+0000 7f13908ca700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f13740b29f0 0x7f13740b4f00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f137c011490 tx=0x7f137c009260 comp rx=0 tx=0).stop 2024-05-27T07:48:29.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.609+0000 7f13908ca700 1 -- 172.21.15.114:0/3137071739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c107de0 msgr2=0x7f138c19b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:29.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.609+0000 7f13908ca700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c107de0 0x7f138c19b360 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f138000e970 tx=0x7f138000ec80 comp rx=0 tx=0).stop 2024-05-27T07:48:29.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.609+0000 7f13908ca700 1 -- 172.21.15.114:0/3137071739 shutdown_connections 2024-05-27T07:48:29.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.609+0000 7f13908ca700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f138c108730 0x7f138c19f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.609+0000 7f13908ca700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f13740b29f0 0x7f13740b4f00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.609+0000 7f13908ca700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f138c1034b0 0x7f138c19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.609+0000 7f13908ca700 1 --2- 172.21.15.114:0/3137071739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c107de0 0x7f138c19b360 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:29.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.609+0000 7f13908ca700 1 -- 172.21.15.114:0/3137071739 >> 172.21.15.114:0/3137071739 conn(0x7f138c0fede0 msgr2=0x7f138c075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:29.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.609+0000 7f13908ca700 1 -- 172.21.15.114:0/3137071739 shutdown_connections 2024-05-27T07:48:29.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:29.609+0000 7f13908ca700 1 -- 172.21.15.114:0/3137071739 wait complete. 2024-05-27T07:48:29.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:29 smithi163 ceph-mon[36451]: pgmap v283: 129 pgs: 129 active+clean; 78 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 262 op/s 2024-05-27T07:48:29.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:29 smithi186 ceph-mon[36647]: pgmap v283: 129 pgs: 129 active+clean; 78 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 262 op/s 2024-05-27T07:48:30.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 885181 11.06 MB/sec execute 282 sec latency 19.249 ms 2024-05-27T07:48:30.406 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:48:30.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:48:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:48:31.236 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:30 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:48:31.236 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:30 smithi163 systemd[1]: Stopped Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:48:31.236 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:30 smithi163 systemd[1]: Starting Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:48:31.236 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:31 smithi163 podman[71691]: 2024-05-27 07:48:31.110373537 +0000 UTC m=+0.017301710 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:48:31.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 887783 11.08 MB/sec execute 283 sec latency 33.371 ms 2024-05-27T07:48:31.505 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:31 smithi163 ceph-mon[36451]: pgmap v284: 129 pgs: 129 active+clean; 74 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 334 op/s 2024-05-27T07:48:31.506 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:31 smithi163 podman[71691]: 2024-05-27 07:48:31.236781028 +0000 UTC m=+0.143709188 container create 46c67c1bcf880a570b232be935f174b39273a00e4e871fec3d4bc1f7027819d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-05-27T07:48:31.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:31 smithi186 ceph-mon[36647]: pgmap v284: 129 pgs: 129 active+clean; 74 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 334 op/s 2024-05-27T07:48:31.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:31 smithi114 ceph-mon[32038]: pgmap v284: 129 pgs: 129 active+clean; 74 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 334 op/s 2024-05-27T07:48:31.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:31 smithi163 podman[71691]: 2024-05-27 07:48:31.506858955 +0000 UTC m=+0.413787115 container init 46c67c1bcf880a570b232be935f174b39273a00e4e871fec3d4bc1f7027819d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:48:31.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:31 smithi163 podman[71691]: 2024-05-27 07:48:31.509762345 +0000 UTC m=+0.416690506 container start 46c67c1bcf880a570b232be935f174b39273a00e4e871fec3d4bc1f7027819d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T07:48:31.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:31 smithi163 ceph-mds[71710]: -- 172.21.15.163:0/1159064774 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e52582ba40 con 0x55e525873a80 2024-05-27T07:48:31.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:31 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: starting mds.h at 2024-05-27T07:48:31.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:31 smithi163 bash[71691]: 46c67c1bcf880a570b232be935f174b39273a00e4e871fec3d4bc1f7027819d6 2024-05-27T07:48:31.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:31 smithi163 systemd[1]: Started Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:48:32.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 889234 11.06 MB/sec execute 284 sec latency 37.926 ms 2024-05-27T07:48:32.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:32 smithi163 ceph-mon[36451]: daemon mds.h restarted 2024-05-27T07:48:32.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:32 smithi186 ceph-mon[36647]: daemon mds.h restarted 2024-05-27T07:48:32.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:32 smithi114 ceph-mon[32038]: daemon mds.h restarted 2024-05-27T07:48:33.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.059+0000 7f11f5c37700 1 -- 172.21.15.114:0/4239666689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11f0107e20 msgr2=0x7f11f0108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:33.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.059+0000 7f11f5c37700 1 --2- 172.21.15.114:0/4239666689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11f0107e20 0x7f11f0108230 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f11e40099c0 tx=0x7f11e4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:48:33.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.059+0000 7f11f5c37700 1 -- 172.21.15.114:0/4239666689 shutdown_connections 2024-05-27T07:48:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.059+0000 7f11f5c37700 1 --2- 172.21.15.114:0/4239666689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11f0108770 0x7f11f01100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.059+0000 7f11f5c37700 1 --2- 172.21.15.114:0/4239666689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11f0103540 0x7f11f0103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.059+0000 7f11f5c37700 1 --2- 172.21.15.114:0/4239666689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11f0107e20 0x7f11f0108230 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.059+0000 7f11f5c37700 1 -- 172.21.15.114:0/4239666689 >> 172.21.15.114:0/4239666689 conn(0x7f11f0078df0 msgr2=0x7f11f0079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.059+0000 7f11f5c37700 1 -- 172.21.15.114:0/4239666689 shutdown_connections 2024-05-27T07:48:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.059+0000 7f11f5c37700 1 -- 172.21.15.114:0/4239666689 wait complete. 2024-05-27T07:48:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.060+0000 7f11f5c37700 1 Processor -- start 2024-05-27T07:48:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.060+0000 7f11f5c37700 1 -- start start 2024-05-27T07:48:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.060+0000 7f11f5c37700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11f0103540 0x7f11f019d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.060+0000 7f11f5c37700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11f0107e20 0x7f11f019d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.060+0000 7f11f5c37700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11f0108770 0x7f11f01a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.060+0000 7f11f5c37700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f11f0112300 con 0x7f11f0103540 2024-05-27T07:48:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.060+0000 7f11f5c37700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f11f0112180 con 0x7f11f0107e20 2024-05-27T07:48:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.060+0000 7f11f5c37700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f11f0112480 con 0x7f11f0108770 2024-05-27T07:48:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.060+0000 7f11ef7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11f0103540 0x7f11f019d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.060+0000 7f11ef7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11f0103540 0x7f11f019d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38426/0 (socket says 172.21.15.114:38426) 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.060+0000 7f11ef7fe700 1 -- 172.21.15.114:0/2640838365 learned_addr learned my addr 172.21.15.114:0/2640838365 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.060+0000 7f11effff700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11f0108770 0x7f11f01a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.061+0000 7f11eeffd700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11f0107e20 0x7f11f019d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.061+0000 7f11effff700 1 -- 172.21.15.114:0/2640838365 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11f0107e20 msgr2=0x7f11f019d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.061+0000 7f11effff700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11f0107e20 0x7f11f019d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.061+0000 7f11effff700 1 -- 172.21.15.114:0/2640838365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11f0103540 msgr2=0x7f11f019d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.061+0000 7f11effff700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11f0103540 0x7f11f019d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.061+0000 7f11effff700 1 -- 172.21.15.114:0/2640838365 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f01a1f20 con 0x7f11f0108770 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.061+0000 7f11effff700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11f0108770 0x7f11f01a1930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f11e800cc50 tx=0x7f11e800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.061+0000 7f11ecff9700 1 -- 172.21.15.114:0/2640838365 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e8016020 con 0x7f11f0108770 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.061+0000 7f11ecff9700 1 -- 172.21.15.114:0/2640838365 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11e8012420 con 0x7f11f0108770 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.061+0000 7f11f5c37700 1 -- 172.21.15.114:0/2640838365 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f01a2210 con 0x7f11f0108770 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.061+0000 7f11ecff9700 1 -- 172.21.15.114:0/2640838365 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e80115a0 con 0x7f11f0108770 2024-05-27T07:48:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.062+0000 7f11f5c37700 1 -- 172.21.15.114:0/2640838365 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f01a26d0 con 0x7f11f0108770 2024-05-27T07:48:33.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.063+0000 7f11f5c37700 1 -- 172.21.15.114:0/2640838365 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11f00512b0 con 0x7f11f0108770 2024-05-27T07:48:33.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.064+0000 7f11ecff9700 1 -- 172.21.15.114:0/2640838365 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f11e8019020 con 0x7f11f0108770 2024-05-27T07:48:33.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.065+0000 7f11ecff9700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f11d80b29e0 0x7f11d80b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:33.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.065+0000 7f11ef7fe700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f11d80b29e0 0x7f11d80b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:33.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.065+0000 7f11ecff9700 1 -- 172.21.15.114:0/2640838365 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f11e80e2af0 con 0x7f11f0108770 2024-05-27T07:48:33.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.065+0000 7f11ef7fe700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f11d80b29e0 0x7f11d80b4ef0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009d20 tx=0x7f11e0009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:33.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.069+0000 7f11ecff9700 1 -- 172.21.15.114:0/2640838365 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f11e8020020 con 0x7f11f0108770 2024-05-27T07:48:33.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 891499 11.07 MB/sec execute 285 sec latency 30.502 ms 2024-05-27T07:48:33.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.372+0000 7f11f5c37700 1 -- 172.21.15.114:0/2640838365 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11f019e460 con 0x7f11f0108770 2024-05-27T07:48:33.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.373+0000 7f11ecff9700 1 -- 172.21.15.114:0/2640838365 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+10175 (secure 0 0 0) 0x7f11e80aa1d0 con 0x7f11f0108770 2024-05-27T07:48:33.373 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:48:33.373 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":26882,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2622386718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2622386718},{"type":"v1","addr":"172.21.15.186:6835","nonce":2622386718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":50,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:48:32.376993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26921":{"gid":26921,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/870343693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":870343693},{"type":"v1","addr":"172.21.15.163:6837","nonce":870343693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:48:33.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.375+0000 7f11f5c37700 1 -- 172.21.15.114:0/2640838365 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f11d80b29e0 msgr2=0x7f11d80b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:33.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.375+0000 7f11f5c37700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f11d80b29e0 0x7f11d80b4ef0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009d20 tx=0x7f11e0009260 comp rx=0 tx=0).stop 2024-05-27T07:48:33.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.375+0000 7f11f5c37700 1 -- 172.21.15.114:0/2640838365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11f0108770 msgr2=0x7f11f01a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:33.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.375+0000 7f11f5c37700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11f0108770 0x7f11f01a1930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f11e800cc50 tx=0x7f11e800cf60 comp rx=0 tx=0).stop 2024-05-27T07:48:33.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.375+0000 7f11f5c37700 1 -- 172.21.15.114:0/2640838365 shutdown_connections 2024-05-27T07:48:33.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.375+0000 7f11f5c37700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11f0108770 0x7f11f01a1930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:33.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.375+0000 7f11f5c37700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11f0103540 0x7f11f019d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:33.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.375+0000 7f11f5c37700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f11d80b29e0 0x7f11d80b4ef0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:33.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.375+0000 7f11f5c37700 1 --2- 172.21.15.114:0/2640838365 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11f0107e20 0x7f11f019d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:33.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.375+0000 7f11f5c37700 1 -- 172.21.15.114:0/2640838365 >> 172.21.15.114:0/2640838365 conn(0x7f11f0078df0 msgr2=0x7f11f0101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:33.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.375+0000 7f11f5c37700 1 -- 172.21.15.114:0/2640838365 shutdown_connections 2024-05-27T07:48:33.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:33.375+0000 7f11f5c37700 1 -- 172.21.15.114:0/2640838365 wait complete. 2024-05-27T07:48:33.377 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 50 2024-05-27T07:48:33.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:33 smithi114 ceph-mon[32038]: pgmap v285: 129 pgs: 129 active+clean; 69 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 363 op/s 2024-05-27T07:48:33.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:33 smithi114 ceph-mon[32038]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T07:48:33.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:33 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6836/870343693,v1:172.21.15.163:6837/870343693] up:boot 2024-05-27T07:48:33.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:33 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 9 up:standby 2024-05-27T07:48:33.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:48:33.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:33 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:48:33.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:33 smithi163 ceph-mon[36451]: pgmap v285: 129 pgs: 129 active+clean; 69 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 363 op/s 2024-05-27T07:48:33.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:33 smithi163 ceph-mon[36451]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T07:48:33.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:33 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6836/870343693,v1:172.21.15.163:6837/870343693] up:boot 2024-05-27T07:48:33.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:33 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 9 up:standby 2024-05-27T07:48:33.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:48:33.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:33 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:48:33.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:33 smithi186 ceph-mon[36647]: pgmap v285: 129 pgs: 129 active+clean; 69 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 363 op/s 2024-05-27T07:48:33.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:33 smithi186 ceph-mon[36647]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T07:48:33.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:33 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6836/870343693,v1:172.21.15.163:6837/870343693] up:boot 2024-05-27T07:48:33.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:33 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 9 up:standby 2024-05-27T07:48:33.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:48:33.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:33 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:48:34.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:48:34.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 893092 11.05 MB/sec execute 286 sec latency 26.990 ms 2024-05-27T07:48:34.298 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:48:34.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:34 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2640838365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:48:34.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:34 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2640838365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:48:34.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:34 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2640838365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:48:35.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 895338 11.06 MB/sec execute 287 sec latency 45.355 ms 2024-05-27T07:48:35.721 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:35 smithi114 ceph-mon[32038]: pgmap v287: 129 pgs: 129 active+clean; 69 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 369 op/s 2024-05-27T07:48:35.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:35 smithi163 ceph-mon[36451]: pgmap v287: 129 pgs: 129 active+clean; 69 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 369 op/s 2024-05-27T07:48:35.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:35 smithi186 ceph-mon[36647]: pgmap v287: 129 pgs: 129 active+clean; 69 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 369 op/s 2024-05-27T07:48:36.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.108+0000 7f378ec21700 1 -- 172.21.15.114:0/1038256353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3788107de0 msgr2=0x7f37881081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:36.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.108+0000 7f378ec21700 1 --2- 172.21.15.114:0/1038256353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3788107de0 0x7f37881081f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f37740099c0 tx=0x7f3774009cd0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.108+0000 7f378ec21700 1 -- 172.21.15.114:0/1038256353 shutdown_connections 2024-05-27T07:48:36.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.108+0000 7f378ec21700 1 --2- 172.21.15.114:0/1038256353 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3788108730 0x7f3788110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.108+0000 7f378ec21700 1 --2- 172.21.15.114:0/1038256353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37881034b0 0x7f37881038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.108+0000 7f378ec21700 1 --2- 172.21.15.114:0/1038256353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3788107de0 0x7f37881081f0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.108+0000 7f378ec21700 1 -- 172.21.15.114:0/1038256353 >> 172.21.15.114:0/1038256353 conn(0x7f37880fedb0 msgr2=0x7f37881011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:36.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.108+0000 7f378ec21700 1 -- 172.21.15.114:0/1038256353 shutdown_connections 2024-05-27T07:48:36.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.108+0000 7f378ec21700 1 -- 172.21.15.114:0/1038256353 wait complete. 2024-05-27T07:48:36.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.109+0000 7f378ec21700 1 Processor -- start 2024-05-27T07:48:36.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.109+0000 7f378ec21700 1 -- start start 2024-05-27T07:48:36.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378ec21700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37881034b0 0x7f378819f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378ec21700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3788107de0 0x7f378819fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378ec21700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3788108730 0x7f3788199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378ec21700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3788113340 con 0x7f3788108730 2024-05-27T07:48:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378ec21700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f37881131c0 con 0x7f3788107de0 2024-05-27T07:48:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378ec21700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f37881134c0 con 0x7f37881034b0 2024-05-27T07:48:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378d1be700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3788108730 0x7f3788199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378c9bd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37881034b0 0x7f378819f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378d1be700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3788108730 0x7f3788199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54762/0 (socket says 172.21.15.114:54762) 2024-05-27T07:48:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378d1be700 1 -- 172.21.15.114:0/1852825675 learned_addr learned my addr 172.21.15.114:0/1852825675 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:36.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f3787fff700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3788107de0 0x7f378819fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378d1be700 1 -- 172.21.15.114:0/1852825675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37881034b0 msgr2=0x7f378819f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:36.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378d1be700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37881034b0 0x7f378819f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378d1be700 1 -- 172.21.15.114:0/1852825675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3788107de0 msgr2=0x7f378819fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:36.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378d1be700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3788107de0 0x7f378819fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378d1be700 1 -- 172.21.15.114:0/1852825675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378819a270 con 0x7f3788108730 2024-05-27T07:48:36.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f3787fff700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3788107de0 0x7f378819fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:48:36.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378c9bd700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37881034b0 0x7f378819f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:36.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378d1be700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3788108730 0x7f3788199b40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f378000ef30 tx=0x7f378000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:36.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f3785ffb700 1 -- 172.21.15.114:0/1852825675 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37800092e0 con 0x7f3788108730 2024-05-27T07:48:36.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f3785ffb700 1 -- 172.21.15.114:0/1852825675 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3780004bb0 con 0x7f3788108730 2024-05-27T07:48:36.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f378ec21700 1 -- 172.21.15.114:0/1852825675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f378819a500 con 0x7f3788108730 2024-05-27T07:48:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.110+0000 7f3785ffb700 1 -- 172.21.15.114:0/1852825675 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3780004280 con 0x7f3788108730 2024-05-27T07:48:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.111+0000 7f378ec21700 1 -- 172.21.15.114:0/1852825675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37881a6760 con 0x7f3788108730 2024-05-27T07:48:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.113+0000 7f3785ffb700 1 -- 172.21.15.114:0/1852825675 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f378001b030 con 0x7f3788108730 2024-05-27T07:48:36.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.114+0000 7f3785ffb700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37780b29f0 0x7f37780b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.114+0000 7f378c9bd700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37780b29f0 0x7f37780b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.114+0000 7f3785ffb700 1 -- 172.21.15.114:0/1852825675 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f37800e2480 con 0x7f3788108730 2024-05-27T07:48:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.114+0000 7f378c9bd700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37780b29f0 0x7f37780b4f00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f377c011490 tx=0x7f377c009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.114+0000 7f379d636b80 1 Processor -- start 2024-05-27T07:48:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f379d636b80 1 -- start start 2024-05-27T07:48:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f379d636b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878ddce40 0x55d878f5cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f379d636b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878f5d430 0x55d878f613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f379d636b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f61920 0x55d878f61d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f379d636b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d878e89c90 con 0x55d878f61920 2024-05-27T07:48:36.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f379d636b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d878e79830 con 0x55d878ddce40 2024-05-27T07:48:36.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f379d636b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d878e7d870 con 0x55d878f5d430 2024-05-27T07:48:36.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f378d1be700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f61920 0x55d878f61d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f3787fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878f5d430 0x55d878f613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f378c9bd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878ddce40 0x55d878f5cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f3787fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878f5d430 0x55d878f613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55724/0 (socket says 172.21.15.114:55724) 2024-05-27T07:48:36.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f378d1be700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f61920 0x55d878f61d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54776/0 (socket says 172.21.15.114:54776) 2024-05-27T07:48:36.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f3787fff700 1 -- 172.21.15.114:0/1563888192 learned_addr learned my addr 172.21.15.114:0/1563888192 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:36.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f378c9bd700 1 -- 172.21.15.114:0/1563888192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878f5d430 msgr2=0x55d878f613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:36.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f378c9bd700 1 --2- 172.21.15.114:0/1563888192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878f5d430 0x55d878f613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f378c9bd700 1 -- 172.21.15.114:0/1563888192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f61920 msgr2=0x55d878f61d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:36.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f378c9bd700 1 --2- 172.21.15.114:0/1563888192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f61920 0x55d878f61d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f378c9bd700 1 -- 172.21.15.114:0/1563888192 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d878dc98d0 con 0x55d878ddce40 2024-05-27T07:48:36.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f3787fff700 1 --2- 172.21.15.114:0/1563888192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878f5d430 0x55d878f613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:36.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.115+0000 7f378d1be700 1 --2- 172.21.15.114:0/1563888192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f61920 0x55d878f61d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:36.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.116+0000 7f378c9bd700 1 --2- 172.21.15.114:0/1563888192 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878ddce40 0x55d878f5cef0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f377c0042f0 tx=0x7f377c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0619dd54fff56dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:36.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.116+0000 7f3772ffd700 1 -- 172.21.15.114:0/1563888192 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c02c070 con 0x55d878ddce40 2024-05-27T07:48:36.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.116+0000 7f3772ffd700 1 -- 172.21.15.114:0/1563888192 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f377c011e50 con 0x55d878ddce40 2024-05-27T07:48:36.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.116+0000 7f3772ffd700 1 -- 172.21.15.114:0/1563888192 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c01cae0 con 0x55d878ddce40 2024-05-27T07:48:36.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.117+0000 7f379d636b80 1 -- 172.21.15.114:0/1563888192 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878ddce40 msgr2=0x55d878f5cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:36.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.117+0000 7f379d636b80 1 --2- 172.21.15.114:0/1563888192 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878ddce40 0x55d878f5cef0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f377c0042f0 tx=0x7f377c014000 comp rx=0 tx=0).stop 2024-05-27T07:48:36.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.117+0000 7f379d636b80 1 -- 172.21.15.114:0/1563888192 shutdown_connections 2024-05-27T07:48:36.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.117+0000 7f379d636b80 1 --2- 172.21.15.114:0/1563888192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f61920 0x55d878f61d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.117+0000 7f379d636b80 1 --2- 172.21.15.114:0/1563888192 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878ddce40 0x55d878f5cef0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.117+0000 7f379d636b80 1 --2- 172.21.15.114:0/1563888192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878f5d430 0x55d878f613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.117+0000 7f379d636b80 1 -- 172.21.15.114:0/1563888192 >> 172.21.15.114:0/1563888192 conn(0x55d878e37af0 msgr2=0x55d878f55640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:36.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.117+0000 7f379d636b80 1 -- 172.21.15.114:0/1563888192 shutdown_connections 2024-05-27T07:48:36.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.117+0000 7f379d636b80 1 -- 172.21.15.114:0/1563888192 wait complete. 2024-05-27T07:48:36.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.117+0000 7f379d636b80 1 -- start start 2024-05-27T07:48:36.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.117+0000 7f379d636b80 1 Processor -- start 2024-05-27T07:48:36.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f379d636b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878ddce40 0x55d878f649a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f379d636b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f5d430 0x55d878f64ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f379d636b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f61920 0x55d878f70730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f379d636b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d878d7ffa0 con 0x55d878f61920 2024-05-27T07:48:36.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f379d636b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d878e1b7c0 con 0x55d878f5d430 2024-05-27T07:48:36.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f379d636b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d878f5ac80 con 0x55d878ddce40 2024-05-27T07:48:36.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f379d636b80 10 client.0 _ll_register_callbacks cb 0x55d878e14920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:36.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f378c9bd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878ddce40 0x55d878f649a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f3787fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f5d430 0x55d878f64ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f378c9bd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878ddce40 0x55d878f649a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55740/0 (socket says 172.21.15.114:55740) 2024-05-27T07:48:36.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f378c9bd700 1 -- 172.21.15.114:0/1368193537 learned_addr learned my addr 172.21.15.114:0/1368193537 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:36.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f378d1be700 1 --2- 172.21.15.114:0/1368193537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f61920 0x55d878f70730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f3787fff700 1 -- 172.21.15.114:0/1368193537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878ddce40 msgr2=0x55d878f649a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:36.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f3787fff700 1 --2- 172.21.15.114:0/1368193537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878ddce40 0x55d878f649a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f3787fff700 1 -- 172.21.15.114:0/1368193537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f61920 msgr2=0x55d878f70730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:36.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f3787fff700 1 --2- 172.21.15.114:0/1368193537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f61920 0x55d878f70730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.118+0000 7f3787fff700 1 -- 172.21.15.114:0/1368193537 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d878e18690 con 0x55d878f5d430 2024-05-27T07:48:36.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.119+0000 7f3787fff700 1 --2- 172.21.15.114:0/1368193537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f5d430 0x55d878f64ee0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3774006010 tx=0x7f3774005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c1dfa5877c812c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:36.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.119+0000 7f3767fff700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:48:36.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.119+0000 7f3767fff700 1 -- 172.21.15.114:0/1368193537 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3774003bd0 con 0x55d878f5d430 2024-05-27T07:48:36.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.119+0000 7f379d636b80 1 -- 172.21.15.114:0/1368193537 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f377c011e50 con 0x55d878f5d430 2024-05-27T07:48:36.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.119+0000 7f3767fff700 1 -- 172.21.15.114:0/1368193537 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f3774003d70 con 0x55d878f5d430 2024-05-27T07:48:36.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.119+0000 7f3767fff700 1 -- 172.21.15.114:0/1368193537 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f3774031050 con 0x55d878f5d430 2024-05-27T07:48:36.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.119+0000 7f379d636b80 10 client.26936 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:36.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.119+0000 7f379d636b80 1 -- 172.21.15.114:0/1368193537 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55d878f72220 con 0x55d878f5d430 2024-05-27T07:48:36.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.119+0000 7f3767fff700 1 -- 172.21.15.114:0/1368193537 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f377c011e50 con 0x55d878f5d430 2024-05-27T07:48:36.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.120+0000 7f3767fff700 1 -- 172.21.15.114:0/1368193537 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7f3774023790 con 0x55d878f5d430 2024-05-27T07:48:36.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.120+0000 7f379d636b80 10 client.26936 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:36.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.120+0000 7f379d636b80 10 client.26936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:36.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.120+0000 7f379d636b80 20 client.26936 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:36.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.120+0000 7f379d636b80 1 --2- 172.21.15.114:0/1368193537 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d878f74860 0x55d878f76c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.120+0000 7f379d636b80 4 client.26936 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:48:36.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.120+0000 7f379d636b80 1 -- 172.21.15.114:0/1368193537 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d878f269f0 con 0x55d878f74860 2024-05-27T07:48:36.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.120+0000 7f378c9bd700 1 --2- 172.21.15.114:0/1368193537 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d878f74860 0x55d878f76c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.120+0000 7f3767fff700 10 client.26936 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:36.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.120+0000 7f378c9bd700 1 --2- 172.21.15.114:0/1368193537 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d878f74860 0x55d878f76c80 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6efc24d11bf60b0 server_cookie=e9d4387d30440d1e in_seq=0 out_seq=0 2024-05-27T07:48:36.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.121+0000 7f3767fff700 1 -- 172.21.15.114:0/1368193537 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d878f269f0 con 0x55d878f74860 2024-05-27T07:48:36.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.121+0000 7f3767fff700 10 client.26936 handle_command_reply: tid=0 2024-05-27T07:48:36.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.121+0000 7f379d636b80 1 client.26936 shutdown 2024-05-27T07:48:36.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.121+0000 7f379d636b80 1 -- 172.21.15.114:0/1368193537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f5d430 msgr2=0x55d878f64ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:36.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.121+0000 7f379d636b80 1 --2- 172.21.15.114:0/1368193537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f5d430 0x55d878f64ee0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3774006010 tx=0x7f3774005800 comp rx=0 tx=0).stop 2024-05-27T07:48:36.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.121+0000 7f379d636b80 1 -- 172.21.15.114:0/1368193537 shutdown_connections 2024-05-27T07:48:36.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.121+0000 7f379d636b80 1 --2- 172.21.15.114:0/1368193537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f61920 0x55d878f70730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.121+0000 7f379d636b80 1 --2- 172.21.15.114:0/1368193537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878ddce40 0x55d878f649a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.121+0000 7f379d636b80 1 --2- 172.21.15.114:0/1368193537 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d878f74860 0x55d878f76c80 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.121+0000 7f379d636b80 1 --2- 172.21.15.114:0/1368193537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f5d430 0x55d878f64ee0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.121+0000 7f3767fff700 0 client.26936 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:36.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.121+0000 7f379d636b80 1 -- 172.21.15.114:0/1368193537 >> 172.21.15.114:0/1368193537 conn(0x55d878e37af0 msgr2=0x55d878f59a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:36.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.121+0000 7f379d636b80 1 -- 172.21.15.114:0/1368193537 shutdown_connections 2024-05-27T07:48:36.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.122+0000 7f379d636b80 1 -- 172.21.15.114:0/1368193537 wait complete. 2024-05-27T07:48:36.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.122+0000 7f379d636b80 20 client.26936 trim_cache size 0 max 16384 2024-05-27T07:48:36.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.137+0000 7f379d636b80 1 Processor -- start 2024-05-27T07:48:36.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.137+0000 7f379d636b80 1 -- start start 2024-05-27T07:48:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.137+0000 7f379d636b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878ddce40 0x55d878f63c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.137+0000 7f379d636b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878f5d430 0x55d878f61d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.137+0000 7f379d636b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f61920 0x55d878f5a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.137+0000 7f379d636b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f37881ae480 con 0x55d878ddce40 2024-05-27T07:48:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.137+0000 7f379d636b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f37881aeb50 con 0x55d878f61920 2024-05-27T07:48:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.137+0000 7f379d636b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f37881ac2d0 con 0x55d878f5d430 2024-05-27T07:48:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.137+0000 7f378c9bd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878ddce40 0x55d878f63c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.138+0000 7f378c9bd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878ddce40 0x55d878f63c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54792/0 (socket says 172.21.15.114:54792) 2024-05-27T07:48:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.138+0000 7f378c9bd700 1 -- 172.21.15.114:0/3948871261 learned_addr learned my addr 172.21.15.114:0/3948871261 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.138+0000 7f3787fff700 1 --2- 172.21.15.114:0/3948871261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878f5d430 0x55d878f61d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.138+0000 7f378d1be700 1 --2- 172.21.15.114:0/3948871261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f61920 0x55d878f5a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.138+0000 7f378c9bd700 1 -- 172.21.15.114:0/3948871261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878f5d430 msgr2=0x55d878f61d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.138+0000 7f378c9bd700 1 --2- 172.21.15.114:0/3948871261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878f5d430 0x55d878f61d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.138+0000 7f378c9bd700 1 -- 172.21.15.114:0/3948871261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f61920 msgr2=0x55d878f5a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.138+0000 7f378c9bd700 1 --2- 172.21.15.114:0/3948871261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f61920 0x55d878f5a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.138+0000 7f378c9bd700 1 -- 172.21.15.114:0/3948871261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d878f61250 con 0x55d878ddce40 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.138+0000 7f378c9bd700 1 --2- 172.21.15.114:0/3948871261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878ddce40 0x55d878f63c50 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f377c0097c0 tx=0x7f377c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0cabd3eb9756848 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.138+0000 7f37727fc700 1 -- 172.21.15.114:0/3948871261 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c038020 con 0x55d878ddce40 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.138+0000 7f37727fc700 1 -- 172.21.15.114:0/3948871261 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f377c011e50 con 0x55d878ddce40 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.138+0000 7f37727fc700 1 -- 172.21.15.114:0/3948871261 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c02c030 con 0x55d878ddce40 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.139+0000 7f379d636b80 1 -- 172.21.15.114:0/3948871261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878ddce40 msgr2=0x55d878f63c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.139+0000 7f379d636b80 1 --2- 172.21.15.114:0/3948871261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878ddce40 0x55d878f63c50 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f377c0097c0 tx=0x7f377c017000 comp rx=0 tx=0).stop 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.139+0000 7f379d636b80 1 -- 172.21.15.114:0/3948871261 shutdown_connections 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.139+0000 7f379d636b80 1 --2- 172.21.15.114:0/3948871261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f61920 0x55d878f5a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.139+0000 7f379d636b80 1 --2- 172.21.15.114:0/3948871261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878ddce40 0x55d878f63c50 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.139+0000 7f379d636b80 1 --2- 172.21.15.114:0/3948871261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878f5d430 0x55d878f61d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.139+0000 7f379d636b80 1 -- 172.21.15.114:0/3948871261 >> 172.21.15.114:0/3948871261 conn(0x55d878e37af0 msgr2=0x55d878f59a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.139+0000 7f379d636b80 1 -- 172.21.15.114:0/3948871261 shutdown_connections 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.139+0000 7f379d636b80 1 -- 172.21.15.114:0/3948871261 wait complete. 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f379d636b80 1 -- start start 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f379d636b80 1 Processor -- start 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f379d636b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878ddce40 0x55d878f70740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f379d636b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f5d430 0x55d878f69ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f379d636b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f61920 0x55d878f69ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f379d636b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f37881ac940 con 0x55d878f5d430 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f379d636b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f37881addb0 con 0x55d878f61920 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f379d636b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f37881ad010 con 0x55d878ddce40 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f379d636b80 10 client.0 _ll_register_callbacks cb 0x55d878e14920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f3787fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f5d430 0x55d878f69ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f3787fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f5d430 0x55d878f69ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54802/0 (socket says 172.21.15.114:54802) 2024-05-27T07:48:36.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f3787fff700 1 -- 172.21.15.114:0/349940212 learned_addr learned my addr 172.21.15.114:0/349940212 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f378c9bd700 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878ddce40 0x55d878f70740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.140+0000 7f378d1be700 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f61920 0x55d878f69ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f378c9bd700 1 -- 172.21.15.114:0/349940212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f61920 msgr2=0x55d878f69ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f378c9bd700 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f61920 0x55d878f69ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f378c9bd700 1 -- 172.21.15.114:0/349940212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f5d430 msgr2=0x55d878f69ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f378c9bd700 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f5d430 0x55d878f69ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f378c9bd700 1 -- 172.21.15.114:0/349940212 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d878f70fb0 con 0x55d878ddce40 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f3787fff700 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f5d430 0x55d878f69ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f378c9bd700 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878ddce40 0x55d878f70740 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f377c01ec60 tx=0x7f377c01ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5460283e1864b911 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f3767fff700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f3767fff700 1 -- 172.21.15.114:0/349940212 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c042060 con 0x55d878ddce40 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f379d636b80 1 -- 172.21.15.114:0/349940212 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f377c011e50 con 0x55d878ddce40 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f3767fff700 1 -- 172.21.15.114:0/349940212 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f377c01e070 con 0x55d878ddce40 2024-05-27T07:48:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f3767fff700 1 -- 172.21.15.114:0/349940212 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f377c02c070 con 0x55d878ddce40 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f379d636b80 10 client.27067 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f379d636b80 1 -- 172.21.15.114:0/349940212 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55d878f6ecd0 con 0x55d878ddce40 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.141+0000 7f3767fff700 1 -- 172.21.15.114:0/349940212 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f377c011e50 con 0x55d878ddce40 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.142+0000 7f3767fff700 1 -- 172.21.15.114:0/349940212 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7f377c0049c0 con 0x55d878ddce40 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.142+0000 7f379d636b80 10 client.27067 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.142+0000 7f379d636b80 10 client.27067 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.142+0000 7f379d636b80 20 client.27067 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.142+0000 7f379d636b80 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d878fdf480 0x55d878fe18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.142+0000 7f379d636b80 4 client.27067 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.142+0000 7f379d636b80 1 -- 172.21.15.114:0/349940212 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d878e8bbe0 con 0x55d878fdf480 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.142+0000 7f3787fff700 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d878fdf480 0x55d878fe18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.142+0000 7f3767fff700 10 client.27067 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.142+0000 7f3787fff700 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d878fdf480 0x55d878fe18a0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a3a4fb6eba819a4 server_cookie=f6f4aaf716399fd9 in_seq=0 out_seq=0 2024-05-27T07:48:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f3767fff700 1 -- 172.21.15.114:0/349940212 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d878e8bbe0 con 0x55d878fdf480 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f3767fff700 10 client.27067 handle_command_reply: tid=0 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f379d636b80 1 client.27067 shutdown 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f379d636b80 1 -- 172.21.15.114:0/349940212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878ddce40 msgr2=0x55d878f70740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f379d636b80 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878ddce40 0x55d878f70740 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f377c01ec60 tx=0x7f377c01ccc0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f379d636b80 1 -- 172.21.15.114:0/349940212 shutdown_connections 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f379d636b80 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d878fdf480 0x55d878fe18a0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f379d636b80 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d878f61920 0x55d878f69ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f379d636b80 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d878ddce40 0x55d878f70740 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f3767fff700 0 client.27067 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f379d636b80 1 --2- 172.21.15.114:0/349940212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d878f5d430 0x55d878f69ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f379d636b80 1 -- 172.21.15.114:0/349940212 >> 172.21.15.114:0/349940212 conn(0x55d878e37af0 msgr2=0x55d878f4fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f379d636b80 1 -- 172.21.15.114:0/349940212 shutdown_connections 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f379d636b80 1 -- 172.21.15.114:0/349940212 wait complete. 2024-05-27T07:48:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.143+0000 7f379d636b80 20 client.27067 trim_cache size 0 max 16384 2024-05-27T07:48:36.150 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:48:36.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.144+0000 7f378ec21700 1 -- 172.21.15.114:0/1852825675 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37780b29f0 msgr2=0x7f37780b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.144+0000 7f378ec21700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37780b29f0 0x7f37780b4f00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f377c011490 tx=0x7f377c009260 comp rx=0 tx=0).stop 2024-05-27T07:48:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.144+0000 7f378ec21700 1 -- 172.21.15.114:0/1852825675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3788108730 msgr2=0x7f3788199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.144+0000 7f378ec21700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3788108730 0x7f3788199b40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f378000ef30 tx=0x7f378000c3a0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.144+0000 7f378ec21700 1 -- 172.21.15.114:0/1852825675 shutdown_connections 2024-05-27T07:48:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.144+0000 7f378ec21700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3788108730 0x7f3788199b40 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.144+0000 7f378ec21700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37881034b0 0x7f378819f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.144+0000 7f378ec21700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37780b29f0 0x7f37780b4f00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.144+0000 7f378ec21700 1 --2- 172.21.15.114:0/1852825675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3788107de0 0x7f378819fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.144+0000 7f378ec21700 1 -- 172.21.15.114:0/1852825675 >> 172.21.15.114:0/1852825675 conn(0x7f37880fedb0 msgr2=0x7f37881011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.144+0000 7f378ec21700 1 -- 172.21.15.114:0/1852825675 shutdown_connections 2024-05-27T07:48:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:36.144+0000 7f378ec21700 1 -- 172.21.15.114:0/1852825675 wait complete. 2024-05-27T07:48:36.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 897586 11.06 MB/sec execute 288 sec latency 34.486 ms 2024-05-27T07:48:37.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 899420 11.06 MB/sec execute 289 sec latency 39.980 ms 2024-05-27T07:48:37.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:37 smithi114 ceph-mon[32038]: pgmap v288: 129 pgs: 129 active+clean; 77 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 321 op/s 2024-05-27T07:48:37.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:37 smithi163 ceph-mon[36451]: pgmap v288: 129 pgs: 129 active+clean; 77 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 321 op/s 2024-05-27T07:48:37.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:37 smithi186 ceph-mon[36647]: pgmap v288: 129 pgs: 129 active+clean; 77 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 321 op/s 2024-05-27T07:48:38.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 901903 11.07 MB/sec execute 290 sec latency 31.468 ms 2024-05-27T07:48:38.332 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:48:38.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:48:38.580 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:48:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.852+0000 7fb77ebe7640 1 -- 172.21.15.114:0/2071975012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb778075480 msgr2=0x7fb778075890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.852+0000 7fb77ebe7640 1 --2- 172.21.15.114:0/2071975012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb778075480 0x7fb778075890 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb768009860 tx=0x7fb76802a410 comp rx=0 tx=0).stop 2024-05-27T07:48:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.855+0000 7fb77ebe7640 1 -- 172.21.15.114:0/2071975012 shutdown_connections 2024-05-27T07:48:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.855+0000 7fb77ebe7640 1 --2- 172.21.15.114:0/2071975012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb778114a90 0x7fb778116eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.855+0000 7fb77ebe7640 1 --2- 172.21.15.114:0/2071975012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb778112130 0x7fb778114550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.855+0000 7fb77ebe7640 1 --2- 172.21.15.114:0/2071975012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb778075480 0x7fb778075890 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.855+0000 7fb77ebe7640 1 -- 172.21.15.114:0/2071975012 >> 172.21.15.114:0/2071975012 conn(0x7fb7780ff330 msgr2=0x7fb778101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.858+0000 7fb77ebe7640 1 -- 172.21.15.114:0/2071975012 shutdown_connections 2024-05-27T07:48:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.858+0000 7fb77ebe7640 1 -- 172.21.15.114:0/2071975012 wait complete. 2024-05-27T07:48:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.860+0000 7fb77ebe7640 1 Processor -- start 2024-05-27T07:48:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.860+0000 7fb77ebe7640 1 -- start start 2024-05-27T07:48:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.860+0000 7fb77ebe7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb778075480 0x7fb7781a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.860+0000 7fb77ebe7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb778112130 0x7fb7781a6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.860+0000 7fb77ebe7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb778114a90 0x7fb7781a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.860+0000 7fb77ebe7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb77811a2b0 con 0x7fb778112130 2024-05-27T07:48:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.860+0000 7fb77ebe7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb77811a130 con 0x7fb778075480 2024-05-27T07:48:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.860+0000 7fb77ebe7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb77811a430 con 0x7fb778114a90 2024-05-27T07:48:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.861+0000 7fb77dbe5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb778075480 0x7fb7781a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.861+0000 7fb77dbe5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb778075480 0x7fb7781a6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:48382/0 (socket says 172.21.15.114:48382) 2024-05-27T07:48:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.861+0000 7fb77dbe5640 1 -- 172.21.15.114:0/579491081 learned_addr learned my addr 172.21.15.114:0/579491081 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.861+0000 7fb77d3e4640 1 --2- 172.21.15.114:0/579491081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb778112130 0x7fb7781a6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.861+0000 7fb77dbe5640 1 -- 172.21.15.114:0/579491081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb778114a90 msgr2=0x7fb7781a0a90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:48:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.861+0000 7fb77dbe5640 1 --2- 172.21.15.114:0/579491081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb778114a90 0x7fb7781a0a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.861+0000 7fb77dbe5640 1 -- 172.21.15.114:0/579491081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb778112130 msgr2=0x7fb7781a6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.861+0000 7fb77dbe5640 1 --2- 172.21.15.114:0/579491081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb778112130 0x7fb7781a6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.861+0000 7fb77dbe5640 1 -- 172.21.15.114:0/579491081 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7781a1080 con 0x7fb778075480 2024-05-27T07:48:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.862+0000 7fb77dbe5640 1 --2- 172.21.15.114:0/579491081 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb778075480 0x7fb7781a6880 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb7680058c0 tx=0x7fb768030f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.863+0000 7fb76effd640 1 -- 172.21.15.114:0/579491081 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb768040070 con 0x7fb778075480 2024-05-27T07:48:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.863+0000 7fb76effd640 1 -- 172.21.15.114:0/579491081 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7680055d0 con 0x7fb778075480 2024-05-27T07:48:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.863+0000 7fb76effd640 1 -- 172.21.15.114:0/579491081 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb768004920 con 0x7fb778075480 2024-05-27T07:48:38.865 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.863+0000 7fb77ebe7640 1 -- 172.21.15.114:0/579491081 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7781a1310 con 0x7fb778075480 2024-05-27T07:48:38.865 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.863+0000 7fb77ebe7640 1 -- 172.21.15.114:0/579491081 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7781a17b0 con 0x7fb778075480 2024-05-27T07:48:38.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.872+0000 7fb786dd6740 1 Processor -- start 2024-05-27T07:48:38.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.874+0000 7fb76effd640 1 -- 172.21.15.114:0/579491081 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fb768039040 con 0x7fb778075480 2024-05-27T07:48:38.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.874+0000 7fb786dd6740 1 -- start start 2024-05-27T07:48:38.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.874+0000 7fb786dd6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62281b0 0x561db65c9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.874+0000 7fb786dd6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db61ff2f0 0x561db65c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.875+0000 7fb77e3e6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62281b0 0x561db65c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.875+0000 7fb77e3e6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62281b0 0x561db65c9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:48392/0 (socket says 172.21.15.114:48392) 2024-05-27T07:48:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.879+0000 7fb786dd6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db62074d0 0x561db65ca010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.879+0000 7fb786dd6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561db653bc30 con 0x561db61ff2f0 2024-05-27T07:48:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.879+0000 7fb786dd6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561db652b7d0 con 0x561db62281b0 2024-05-27T07:48:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.879+0000 7fb786dd6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561db652f810 con 0x561db62074d0 2024-05-27T07:48:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.879+0000 7fb77e3e6640 1 -- 172.21.15.114:0/3210333753 learned_addr learned my addr 172.21.15.114:0/3210333753 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.881+0000 7fb77d3e4640 1 --2- 172.21.15.114:0/3210333753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db62074d0 0x561db65ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.886+0000 7fb77dbe5640 1 --2- 172.21.15.114:0/3210333753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db61ff2f0 0x561db65c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.886+0000 7fb77e3e6640 1 -- 172.21.15.114:0/3210333753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db62074d0 msgr2=0x561db65ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:38.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.886+0000 7fb77e3e6640 1 --2- 172.21.15.114:0/3210333753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db62074d0 0x561db65ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.886+0000 7fb77e3e6640 1 -- 172.21.15.114:0/3210333753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db61ff2f0 msgr2=0x561db65c9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:38.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.888+0000 7fb77e3e6640 1 --2- 172.21.15.114:0/3210333753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db61ff2f0 0x561db65c9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.888+0000 7fb77e3e6640 1 -- 172.21.15.114:0/3210333753 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561db6240210 con 0x561db62281b0 2024-05-27T07:48:38.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.888+0000 7fb77dbe5640 1 --2- 172.21.15.114:0/3210333753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db61ff2f0 0x561db65c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:48:38.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.888+0000 7fb76effd640 1 --2- 172.21.15.114:0/579491081 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb7640b2b60 0x7fb7640b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.889+0000 7fb76effd640 1 -- 172.21.15.114:0/579491081 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7fb768103a20 con 0x7fb778075480 2024-05-27T07:48:38.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.889+0000 7fb77e3e6640 1 --2- 172.21.15.114:0/3210333753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62281b0 0x561db65c9590 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb77400b9a0 tx=0x7fb77400be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6038daed68783a73 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:38.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.889+0000 7fb77dbe5640 1 --2- 172.21.15.114:0/579491081 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb7640b2b60 0x7fb7640b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.890+0000 7fb77dbe5640 1 --2- 172.21.15.114:0/579491081 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb7640b2b60 0x7fb7640b5070 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x561db65c7080 tx=0x7fb768005f90 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:38.893 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.892+0000 7fb763fff640 1 -- 172.21.15.114:0/3210333753 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774016070 con 0x561db62281b0 2024-05-27T07:48:38.893 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.892+0000 7fb763fff640 1 -- 172.21.15.114:0/3210333753 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7740054d0 con 0x561db62281b0 2024-05-27T07:48:38.894 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.893+0000 7fb763fff640 1 -- 172.21.15.114:0/3210333753 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774004d40 con 0x561db62281b0 2024-05-27T07:48:38.894 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.893+0000 7fb786dd6740 1 -- 172.21.15.114:0/3210333753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62281b0 msgr2=0x561db65c9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:38.894 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.893+0000 7fb786dd6740 1 --2- 172.21.15.114:0/3210333753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62281b0 0x561db65c9590 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb77400b9a0 tx=0x7fb77400be60 comp rx=0 tx=0).stop 2024-05-27T07:48:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.895+0000 7fb786dd6740 1 -- 172.21.15.114:0/3210333753 shutdown_connections 2024-05-27T07:48:38.896 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.895+0000 7fb786dd6740 1 --2- 172.21.15.114:0/3210333753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db62074d0 0x561db65ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.896 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.895+0000 7fb786dd6740 1 --2- 172.21.15.114:0/3210333753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db61ff2f0 0x561db65c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.896 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.895+0000 7fb786dd6740 1 --2- 172.21.15.114:0/3210333753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62281b0 0x561db65c9590 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.896 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.896+0000 7fb786dd6740 1 -- 172.21.15.114:0/3210333753 >> 172.21.15.114:0/3210333753 conn(0x561db621c300 msgr2=0x561db65c5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.897+0000 7fb786dd6740 1 -- 172.21.15.114:0/3210333753 shutdown_connections 2024-05-27T07:48:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.897+0000 7fb786dd6740 1 -- 172.21.15.114:0/3210333753 wait complete. 2024-05-27T07:48:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.897+0000 7fb786dd6740 1 -- start start 2024-05-27T07:48:38.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.898+0000 7fb786dd6740 1 Processor -- start 2024-05-27T07:48:38.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.898+0000 7fb786dd6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db61ff2f0 0x561db65d3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.898+0000 7fb786dd6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62074d0 0x561db65d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.898+0000 7fb786dd6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db62281b0 0x561db65dfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.898+0000 7fb786dd6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561db6211790 con 0x561db61ff2f0 2024-05-27T07:48:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.898+0000 7fb786dd6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561db64e7890 con 0x561db62074d0 2024-05-27T07:48:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.898+0000 7fb786dd6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561db65d6710 con 0x561db62281b0 2024-05-27T07:48:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.898+0000 7fb786dd6740 10 client.0 _ll_register_callbacks cb 0x561db623fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.899+0000 7fb77d3e4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62074d0 0x561db65d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.899+0000 7fb77e3e6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db62281b0 0x561db65dfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.899+0000 7fb77d3e4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62074d0 0x561db65d40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:48406/0 (socket says 172.21.15.114:48406) 2024-05-27T07:48:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.899+0000 7fb77d3e4640 1 -- 172.21.15.114:0/4072512932 learned_addr learned my addr 172.21.15.114:0/4072512932 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.900+0000 7fb77d3e4640 1 -- 172.21.15.114:0/4072512932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db62281b0 msgr2=0x561db65dfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.900+0000 7fb77d3e4640 1 --2- 172.21.15.114:0/4072512932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db62281b0 0x561db65dfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.900+0000 7fb77d3e4640 1 -- 172.21.15.114:0/4072512932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db61ff2f0 msgr2=0x561db65d3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.900+0000 7fb77d3e4640 1 --2- 172.21.15.114:0/4072512932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db61ff2f0 0x561db65d3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.900+0000 7fb77d3e4640 1 -- 172.21.15.114:0/4072512932 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561db630c620 con 0x561db62074d0 2024-05-27T07:48:38.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.900+0000 7fb77d3e4640 1 --2- 172.21.15.114:0/4072512932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62074d0 0x561db65d40c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb770010c00 tx=0x7fb77000f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2570820b65452ba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:38.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.900+0000 7fb7617fa640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:48:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.902+0000 7fb7617fa640 1 -- 172.21.15.114:0/4072512932 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77000e5c0 con 0x561db62074d0 2024-05-27T07:48:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.902+0000 7fb7617fa640 1 -- 172.21.15.114:0/4072512932 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7fb77000e760 con 0x561db62074d0 2024-05-27T07:48:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.904+0000 7fb786dd6740 1 -- 172.21.15.114:0/4072512932 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7740054d0 con 0x561db62074d0 2024-05-27T07:48:38.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.908+0000 7fb7617fa640 1 -- 172.21.15.114:0/4072512932 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fb77000e960 con 0x561db62074d0 2024-05-27T07:48:38.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.908+0000 7fb786dd6740 10 client.26954 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:38.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.908+0000 7fb786dd6740 1 -- 172.21.15.114:0/4072512932 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x561db65f3960 con 0x561db62074d0 2024-05-27T07:48:38.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.910+0000 7fb7617fa640 1 -- 172.21.15.114:0/4072512932 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7740054d0 con 0x561db62074d0 2024-05-27T07:48:38.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.910+0000 7fb7617fa640 1 -- 172.21.15.114:0/4072512932 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7fb77001a420 con 0x561db62074d0 2024-05-27T07:48:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.911+0000 7fb786dd6740 10 client.26954 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.911+0000 7fb786dd6740 10 client.26954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.911+0000 7fb786dd6740 20 client.26954 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.912+0000 7fb786dd6740 1 --2- 172.21.15.114:0/4072512932 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561db61e6d80 0x561db65f7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.912+0000 7fb786dd6740 4 client.26954 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:48:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.912+0000 7fb786dd6740 1 -- 172.21.15.114:0/4072512932 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561db653db80 con 0x561db61e6d80 2024-05-27T07:48:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.912+0000 7fb77e3e6640 1 --2- 172.21.15.114:0/4072512932 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561db61e6d80 0x561db65f7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.913 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.912+0000 7fb7617fa640 10 client.26954 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:38.913 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.913+0000 7fb77e3e6640 1 --2- 172.21.15.114:0/4072512932 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561db61e6d80 0x561db65f7030 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86ab8570b2df79b3 server_cookie=1d5ff58e1865467a in_seq=0 out_seq=0 2024-05-27T07:48:38.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.914+0000 7fb7617fa640 1 -- 172.21.15.114:0/4072512932 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561db653db80 con 0x561db61e6d80 2024-05-27T07:48:38.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.914+0000 7fb7617fa640 10 client.26954 handle_command_reply: tid=0 2024-05-27T07:48:38.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.914+0000 7fb786dd6740 1 client.26954 shutdown 2024-05-27T07:48:38.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.915+0000 7fb786dd6740 1 -- 172.21.15.114:0/4072512932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62074d0 msgr2=0x561db65d40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.915+0000 7fb786dd6740 1 --2- 172.21.15.114:0/4072512932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62074d0 0x561db65d40c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb770010c00 tx=0x7fb77000f7f0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.916+0000 7fb786dd6740 1 -- 172.21.15.114:0/4072512932 shutdown_connections 2024-05-27T07:48:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.916+0000 7fb786dd6740 1 --2- 172.21.15.114:0/4072512932 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561db61e6d80 0x561db65f7030 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.916+0000 7fb786dd6740 1 --2- 172.21.15.114:0/4072512932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db62281b0 0x561db65dfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.916+0000 7fb786dd6740 1 --2- 172.21.15.114:0/4072512932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62074d0 0x561db65d40c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.916+0000 7fb786dd6740 1 --2- 172.21.15.114:0/4072512932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db61ff2f0 0x561db65d3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.916+0000 7fb786dd6740 1 -- 172.21.15.114:0/4072512932 >> 172.21.15.114:0/4072512932 conn(0x561db621c300 msgr2=0x561db65c93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.916+0000 7fb7617fa640 0 client.26954 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.917+0000 7fb786dd6740 1 -- 172.21.15.114:0/4072512932 shutdown_connections 2024-05-27T07:48:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.917+0000 7fb786dd6740 1 -- 172.21.15.114:0/4072512932 wait complete. 2024-05-27T07:48:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.917+0000 7fb786dd6740 20 client.26954 trim_cache size 0 max 16384 2024-05-27T07:48:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.937+0000 7fb786dd6740 1 Processor -- start 2024-05-27T07:48:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.937+0000 7fb786dd6740 1 -- start start 2024-05-27T07:48:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb786dd6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db61ff2f0 0x561db65d3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb786dd6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62074d0 0x561db65df830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb786dd6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db62281b0 0x561db65c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb786dd6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c0101a0 con 0x561db62281b0 2024-05-27T07:48:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb786dd6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c010870 con 0x561db62074d0 2024-05-27T07:48:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb786dd6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c00dff0 con 0x561db61ff2f0 2024-05-27T07:48:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb77e3e6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db61ff2f0 0x561db65d3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb77e3e6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db61ff2f0 0x561db65d3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55810/0 (socket says 172.21.15.114:55810) 2024-05-27T07:48:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb77e3e6640 1 -- 172.21.15.114:0/3899059148 learned_addr learned my addr 172.21.15.114:0/3899059148 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb77d3e4640 1 --2- 172.21.15.114:0/3899059148 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62074d0 0x561db65df830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb77e3e6640 1 --2- 172.21.15.114:0/3899059148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db62281b0 0x561db65c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb77d3e4640 1 -- 172.21.15.114:0/3899059148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db61ff2f0 msgr2=0x561db65d3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb77d3e4640 1 --2- 172.21.15.114:0/3899059148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db61ff2f0 0x561db65d3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb77d3e4640 1 -- 172.21.15.114:0/3899059148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db62281b0 msgr2=0x561db65c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb77d3e4640 1 --2- 172.21.15.114:0/3899059148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db62281b0 0x561db65c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb77d3e4640 1 -- 172.21.15.114:0/3899059148 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561db61e6fb0 con 0x561db62074d0 2024-05-27T07:48:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.938+0000 7fb77e3e6640 1 --2- 172.21.15.114:0/3899059148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db62281b0 0x561db65c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.939+0000 7fb77d3e4640 1 --2- 172.21.15.114:0/3899059148 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62074d0 0x561db65df830 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb77001a960 tx=0x7fb770007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=665cbe518c34694d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.939+0000 7fb7637fe640 1 -- 172.21.15.114:0/3899059148 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77001a420 con 0x561db62074d0 2024-05-27T07:48:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.939+0000 7fb7637fe640 1 -- 172.21.15.114:0/3899059148 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb77001a5c0 con 0x561db62074d0 2024-05-27T07:48:38.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.939+0000 7fb7637fe640 1 -- 172.21.15.114:0/3899059148 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77001abd0 con 0x561db62074d0 2024-05-27T07:48:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.940+0000 7fb786dd6740 1 -- 172.21.15.114:0/3899059148 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62074d0 msgr2=0x561db65df830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.940+0000 7fb786dd6740 1 --2- 172.21.15.114:0/3899059148 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62074d0 0x561db65df830 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb77001a960 tx=0x7fb770007700 comp rx=0 tx=0).stop 2024-05-27T07:48:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.940+0000 7fb786dd6740 1 -- 172.21.15.114:0/3899059148 shutdown_connections 2024-05-27T07:48:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.940+0000 7fb786dd6740 1 --2- 172.21.15.114:0/3899059148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db62281b0 0x561db65c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.940+0000 7fb786dd6740 1 --2- 172.21.15.114:0/3899059148 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62074d0 0x561db65df830 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.940+0000 7fb786dd6740 1 --2- 172.21.15.114:0/3899059148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db61ff2f0 0x561db65d3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.940+0000 7fb786dd6740 1 -- 172.21.15.114:0/3899059148 >> 172.21.15.114:0/3899059148 conn(0x561db621c300 msgr2=0x561db65c4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.940+0000 7fb786dd6740 1 -- 172.21.15.114:0/3899059148 shutdown_connections 2024-05-27T07:48:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.940+0000 7fb786dd6740 1 -- 172.21.15.114:0/3899059148 wait complete. 2024-05-27T07:48:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.940+0000 7fb786dd6740 1 -- start start 2024-05-27T07:48:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.940+0000 7fb786dd6740 1 Processor -- start 2024-05-27T07:48:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb786dd6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db61ff2f0 0x561db65d7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb786dd6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db62074d0 0x561db65d7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb77d3e4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db62074d0 0x561db65d7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb786dd6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62281b0 0x561db665d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb786dd6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c00e660 con 0x561db62074d0 2024-05-27T07:48:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb786dd6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c00fad0 con 0x561db62281b0 2024-05-27T07:48:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb786dd6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c00ed30 con 0x561db61ff2f0 2024-05-27T07:48:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb77d3e4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db62074d0 0x561db65d7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54850/0 (socket says 172.21.15.114:54850) 2024-05-27T07:48:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb77d3e4640 1 -- 172.21.15.114:0/1058268116 learned_addr learned my addr 172.21.15.114:0/1058268116 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb77e3e6640 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db61ff2f0 0x561db65d7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb786dd6740 10 client.0 _ll_register_callbacks cb 0x7fb770010810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb77e3e6640 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62281b0 0x561db665d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb77e3e6640 1 -- 172.21.15.114:0/1058268116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62281b0 msgr2=0x561db665d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb77e3e6640 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62281b0 0x561db665d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb77e3e6640 1 -- 172.21.15.114:0/1058268116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db62074d0 msgr2=0x561db65d7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb77e3e6640 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db62074d0 0x561db65d7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb77e3e6640 1 -- 172.21.15.114:0/1058268116 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561db665dbf0 con 0x561db61ff2f0 2024-05-27T07:48:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb77d3e4640 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db62074d0 0x561db65d7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb77e3e6640 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db61ff2f0 0x561db65d7000 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb774004950 tx=0x7fb770010570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8117e3fc8a946c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.941+0000 7fb7617fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:48:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb7617fa640 1 -- 172.21.15.114:0/1058268116 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774016040 con 0x561db61ff2f0 2024-05-27T07:48:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb786dd6740 1 -- 172.21.15.114:0/1058268116 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb77001a5c0 con 0x561db61ff2f0 2024-05-27T07:48:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb7617fa640 1 -- 172.21.15.114:0/1058268116 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7fb77400dd90 con 0x561db61ff2f0 2024-05-27T07:48:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb7617fa640 1 -- 172.21.15.114:0/1058268116 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fb774027390 con 0x561db61ff2f0 2024-05-27T07:48:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb786dd6740 10 client.27082 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb786dd6740 1 -- 172.21.15.114:0/1058268116 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x561db665eea0 con 0x561db61ff2f0 2024-05-27T07:48:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb7617fa640 1 -- 172.21.15.114:0/1058268116 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb77001a5c0 con 0x561db61ff2f0 2024-05-27T07:48:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb7617fa640 1 -- 172.21.15.114:0/1058268116 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7fb7740037b0 con 0x561db61ff2f0 2024-05-27T07:48:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb786dd6740 10 client.27082 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb786dd6740 10 client.27082 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb786dd6740 20 client.27082 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb786dd6740 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561db61bc1c0 0x561db6661520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb786dd6740 4 client.27082 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:48:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb786dd6740 1 -- 172.21.15.114:0/1058268116 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561db6537360 con 0x561db61bc1c0 2024-05-27T07:48:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.942+0000 7fb77d3e4640 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561db61bc1c0 0x561db6661520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.943+0000 7fb7617fa640 10 client.27082 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:38.943+0000 7fb77d3e4640 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561db61bc1c0 0x561db6661520 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11574522ab9af6e9 server_cookie=9ebe59394ab0a57f in_seq=0 out_seq=0 2024-05-27T07:48:39.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 902738 11.04 MB/sec execute 291 sec latency 119.938 ms 2024-05-27T07:48:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.350+0000 7fb7617fa640 1 -- 172.21.15.114:0/1058268116 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x561db6537360 con 0x561db61bc1c0 2024-05-27T07:48:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.350+0000 7fb7617fa640 10 client.27082 handle_command_reply: tid=0 2024-05-27T07:48:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.350+0000 7fb786dd6740 1 client.27082 shutdown 2024-05-27T07:48:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.350+0000 7fb786dd6740 1 -- 172.21.15.114:0/1058268116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db61ff2f0 msgr2=0x561db65d7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.350+0000 7fb786dd6740 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db61ff2f0 0x561db65d7000 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb774004950 tx=0x7fb770010570 comp rx=0 tx=0).stop 2024-05-27T07:48:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.351+0000 7fb786dd6740 1 -- 172.21.15.114:0/1058268116 shutdown_connections 2024-05-27T07:48:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.351+0000 7fb786dd6740 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561db61bc1c0 0x561db6661520 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.351+0000 7fb786dd6740 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561db62281b0 0x561db665d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.351+0000 7fb786dd6740 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561db62074d0 0x561db65d7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.351+0000 7fb7617fa640 0 client.27082 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.351+0000 7fb786dd6740 1 --2- 172.21.15.114:0/1058268116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561db61ff2f0 0x561db65d7000 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.351+0000 7fb786dd6740 1 -- 172.21.15.114:0/1058268116 >> 172.21.15.114:0/1058268116 conn(0x561db621c300 msgr2=0x561db65cd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.351+0000 7fb786dd6740 1 -- 172.21.15.114:0/1058268116 shutdown_connections 2024-05-27T07:48:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.351+0000 7fb786dd6740 1 -- 172.21.15.114:0/1058268116 wait complete. 2024-05-27T07:48:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.351+0000 7fb786dd6740 20 client.27082 trim_cache size 0 max 16384 2024-05-27T07:48:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:48:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:48:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:48:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.352+0000 7fb77ebe7640 1 -- 172.21.15.114:0/579491081 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb7640b2b60 msgr2=0x7fb7640b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.352+0000 7fb77ebe7640 1 --2- 172.21.15.114:0/579491081 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb7640b2b60 0x7fb7640b5070 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x561db65c7080 tx=0x7fb768005f90 comp rx=0 tx=0).stop 2024-05-27T07:48:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.352+0000 7fb77ebe7640 1 -- 172.21.15.114:0/579491081 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb778075480 msgr2=0x7fb7781a6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.352+0000 7fb77ebe7640 1 --2- 172.21.15.114:0/579491081 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb778075480 0x7fb7781a6880 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb7680058c0 tx=0x7fb768030f60 comp rx=0 tx=0).stop 2024-05-27T07:48:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.352+0000 7fb77ebe7640 1 -- 172.21.15.114:0/579491081 shutdown_connections 2024-05-27T07:48:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.352+0000 7fb77ebe7640 1 --2- 172.21.15.114:0/579491081 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb7640b2b60 0x7fb7640b5070 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.352+0000 7fb77ebe7640 1 --2- 172.21.15.114:0/579491081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb778114a90 0x7fb7781a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.352+0000 7fb77ebe7640 1 --2- 172.21.15.114:0/579491081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb778112130 0x7fb7781a6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.352+0000 7fb77ebe7640 1 --2- 172.21.15.114:0/579491081 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb778075480 0x7fb7781a6880 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.352+0000 7fb77ebe7640 1 -- 172.21.15.114:0/579491081 >> 172.21.15.114:0/579491081 conn(0x7fb7780ff330 msgr2=0x7fb7781119a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.352+0000 7fb77ebe7640 1 -- 172.21.15.114:0/579491081 shutdown_connections 2024-05-27T07:48:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:39.352+0000 7fb77ebe7640 1 -- 172.21.15.114:0/579491081 wait complete. 2024-05-27T07:48:39.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:39 smithi114 ceph-mon[32038]: pgmap v289: 129 pgs: 129 active+clean; 77 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 321 op/s 2024-05-27T07:48:39.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:39 smithi163 ceph-mon[36451]: pgmap v289: 129 pgs: 129 active+clean; 77 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 321 op/s 2024-05-27T07:48:39.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:39 smithi186 ceph-mon[36647]: pgmap v289: 129 pgs: 129 active+clean; 77 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 321 op/s 2024-05-27T07:48:40.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 905046 11.05 MB/sec execute 292 sec latency 82.946 ms 2024-05-27T07:48:40.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.740+0000 7f6817a5d700 1 -- 172.21.15.114:0/1701984423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6810109860 msgr2=0x7f68101111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:40.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.740+0000 7f6817a5d700 1 --2- 172.21.15.114:0/1701984423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6810109860 0x7f68101111c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f680800b0d0 tx=0x7f680800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.740+0000 7f6817a5d700 1 -- 172.21.15.114:0/1701984423 shutdown_connections 2024-05-27T07:48:40.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.740+0000 7f6817a5d700 1 --2- 172.21.15.114:0/1701984423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6810109860 0x7f68101111c0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.740+0000 7f6817a5d700 1 --2- 172.21.15.114:0/1701984423 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68101045e0 0x7f68101049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.740+0000 7f6817a5d700 1 --2- 172.21.15.114:0/1701984423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6810108f10 0x7f6810109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.740+0000 7f6817a5d700 1 -- 172.21.15.114:0/1701984423 >> 172.21.15.114:0/1701984423 conn(0x7f6810100030 msgr2=0x7f6810102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:40.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.740+0000 7f6817a5d700 1 -- 172.21.15.114:0/1701984423 shutdown_connections 2024-05-27T07:48:40.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.740+0000 7f6817a5d700 1 -- 172.21.15.114:0/1701984423 wait complete. 2024-05-27T07:48:40.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.741+0000 7f6817a5d700 1 Processor -- start 2024-05-27T07:48:40.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.741+0000 7f6817a5d700 1 -- start start 2024-05-27T07:48:40.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.741+0000 7f6817a5d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68101045e0 0x7f681019d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.741+0000 7f6817a5d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6810108f10 0x7f681019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.741+0000 7f6817a5d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6810109860 0x7f68101a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.741+0000 7f6817a5d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f68101144b0 con 0x7f68101045e0 2024-05-27T07:48:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.741+0000 7f6817a5d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6810114330 con 0x7f6810108f10 2024-05-27T07:48:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.741+0000 7f6817a5d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6810114630 con 0x7f6810109860 2024-05-27T07:48:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f68157f9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68101045e0 0x7f681019d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f68157f9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68101045e0 0x7f681019d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54872/0 (socket says 172.21.15.114:54872) 2024-05-27T07:48:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f68157f9700 1 -- 172.21.15.114:0/1146361369 learned_addr learned my addr 172.21.15.114:0/1146361369 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f6815ffa700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6810109860 0x7f68101a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f6814ff8700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6810108f10 0x7f681019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f68157f9700 1 -- 172.21.15.114:0/1146361369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6810109860 msgr2=0x7f68101a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f68157f9700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6810109860 0x7f68101a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f68157f9700 1 -- 172.21.15.114:0/1146361369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6810108f10 msgr2=0x7f681019d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f68157f9700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6810108f10 0x7f681019d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f68157f9700 1 -- 172.21.15.114:0/1146361369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68101a1e20 con 0x7f68101045e0 2024-05-27T07:48:40.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f6815ffa700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6810109860 0x7f68101a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:40.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f6814ff8700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6810108f10 0x7f681019d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:48:40.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f68157f9700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68101045e0 0x7f681019d000 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f680c00c9b0 tx=0x7f680c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:40.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f68067fc700 1 -- 172.21.15.114:0/1146361369 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c007b00 con 0x7f68101045e0 2024-05-27T07:48:40.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.742+0000 7f6817a5d700 1 -- 172.21.15.114:0/1146361369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68101a2110 con 0x7f68101045e0 2024-05-27T07:48:40.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.743+0000 7f6817a5d700 1 -- 172.21.15.114:0/1146361369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68101a9ae0 con 0x7f68101045e0 2024-05-27T07:48:40.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.743+0000 7f68067fc700 1 -- 172.21.15.114:0/1146361369 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f680c007ca0 con 0x7f68101045e0 2024-05-27T07:48:40.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.743+0000 7f68067fc700 1 -- 172.21.15.114:0/1146361369 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c018660 con 0x7f68101045e0 2024-05-27T07:48:40.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.745+0000 7f68067fc700 1 -- 172.21.15.114:0/1146361369 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f680c004830 con 0x7f68101045e0 2024-05-27T07:48:40.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.746+0000 7f68067fc700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67fc0b2a70 0x7f67fc0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.746+0000 7f6814ff8700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67fc0b2a70 0x7f67fc0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.746+0000 7f68067fc700 1 -- 172.21.15.114:0/1146361369 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f680c0e28b0 con 0x7f68101045e0 2024-05-27T07:48:40.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.746+0000 7f6826472b80 1 Processor -- start 2024-05-27T07:48:40.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6826472b80 1 -- start start 2024-05-27T07:48:40.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6826472b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071c99ec0 0x558071e1d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6814ff8700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67fc0b2a70 0x7f67fc0b4f80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6810110510 tx=0x7f6800009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:40.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6826472b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e17ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6826472b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071e18000 0x558071e18410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6826472b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558071d46d10 con 0x558071e1dcd0 2024-05-27T07:48:40.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6826472b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558071d368b0 con 0x558071e18000 2024-05-27T07:48:40.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6826472b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558071d3a8f0 con 0x558071c99ec0 2024-05-27T07:48:40.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6814ff8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e17ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f68157f9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071c99ec0 0x558071e1d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6814ff8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e17ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54878/0 (socket says 172.21.15.114:54878) 2024-05-27T07:48:40.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6814ff8700 1 -- 172.21.15.114:0/4271350326 learned_addr learned my addr 172.21.15.114:0/4271350326 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:40.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f68157f9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071c99ec0 0x558071e1d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55848/0 (socket says 172.21.15.114:55848) 2024-05-27T07:48:40.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6815ffa700 1 --2- 172.21.15.114:0/4271350326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071e18000 0x558071e18410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f68157f9700 1 -- 172.21.15.114:0/4271350326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071e18000 msgr2=0x558071e18410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:40.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f68157f9700 1 --2- 172.21.15.114:0/4271350326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071e18000 0x558071e18410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f68157f9700 1 -- 172.21.15.114:0/4271350326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 msgr2=0x558071e17ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:40.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f68157f9700 1 --2- 172.21.15.114:0/4271350326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e17ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f68157f9700 1 -- 172.21.15.114:0/4271350326 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558071c86950 con 0x558071c99ec0 2024-05-27T07:48:40.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6815ffa700 1 --2- 172.21.15.114:0/4271350326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071e18000 0x558071e18410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:48:40.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.747+0000 7f6814ff8700 1 --2- 172.21.15.114:0/4271350326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e17ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:40.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.748+0000 7f68157f9700 1 --2- 172.21.15.114:0/4271350326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071c99ec0 0x558071e1d790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f680c0dfc50 tx=0x7f680c0a96e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f6543f7050dbe50 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:40.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.748+0000 7f67fb7fe700 1 -- 172.21.15.114:0/4271350326 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c0a9ee0 con 0x558071c99ec0 2024-05-27T07:48:40.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.748+0000 7f67fb7fe700 1 -- 172.21.15.114:0/4271350326 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f680c0a97e0 con 0x558071c99ec0 2024-05-27T07:48:40.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.748+0000 7f67fb7fe700 1 -- 172.21.15.114:0/4271350326 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c0b21b0 con 0x558071c99ec0 2024-05-27T07:48:40.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.749+0000 7f6826472b80 1 -- 172.21.15.114:0/4271350326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071c99ec0 msgr2=0x558071e1d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:40.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.749+0000 7f6826472b80 1 --2- 172.21.15.114:0/4271350326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071c99ec0 0x558071e1d790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f680c0dfc50 tx=0x7f680c0a96e0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.749+0000 7f6826472b80 1 -- 172.21.15.114:0/4271350326 shutdown_connections 2024-05-27T07:48:40.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.749+0000 7f6826472b80 1 --2- 172.21.15.114:0/4271350326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071e18000 0x558071e18410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.749+0000 7f6826472b80 1 --2- 172.21.15.114:0/4271350326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071c99ec0 0x558071e1d790 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.749+0000 7f6826472b80 1 --2- 172.21.15.114:0/4271350326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e17ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.749+0000 7f6826472b80 1 -- 172.21.15.114:0/4271350326 >> 172.21.15.114:0/4271350326 conn(0x558071cf4b70 msgr2=0x558071e126c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.749+0000 7f6826472b80 1 -- 172.21.15.114:0/4271350326 shutdown_connections 2024-05-27T07:48:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.749+0000 7f6826472b80 1 -- 172.21.15.114:0/4271350326 wait complete. 2024-05-27T07:48:40.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.749+0000 7f6826472b80 1 -- start start 2024-05-27T07:48:40.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.749+0000 7f6826472b80 1 Processor -- start 2024-05-27T07:48:40.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6826472b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 0x558071e21400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6826472b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e18000 0x558071e21940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6826472b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e1dcd0 0x558071e21e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6826472b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558071c3d020 con 0x558071e18000 2024-05-27T07:48:40.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6826472b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558071cd8840 con 0x558071c99ec0 2024-05-27T07:48:40.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6826472b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558071e247c0 con 0x558071e1dcd0 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6826472b80 10 client.0 _ll_register_callbacks cb 0x558071cd19a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6814ff8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e18000 0x558071e21940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f68157f9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 0x558071e21400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6814ff8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e18000 0x558071e21940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54892/0 (socket says 172.21.15.114:54892) 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6814ff8700 1 -- 172.21.15.114:0/4286061587 learned_addr learned my addr 172.21.15.114:0/4286061587 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6815ffa700 1 --2- 172.21.15.114:0/4286061587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e1dcd0 0x558071e21e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6814ff8700 1 -- 172.21.15.114:0/4286061587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e1dcd0 msgr2=0x558071e21e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6814ff8700 1 --2- 172.21.15.114:0/4286061587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e1dcd0 0x558071e21e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6814ff8700 1 -- 172.21.15.114:0/4286061587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 msgr2=0x558071e21400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6814ff8700 1 --2- 172.21.15.114:0/4286061587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 0x558071e21400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6814ff8700 1 -- 172.21.15.114:0/4286061587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558071cd5710 con 0x558071e18000 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f68157f9700 1 --2- 172.21.15.114:0/4286061587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 0x558071e21400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f6814ff8700 1 --2- 172.21.15.114:0/4286061587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e18000 0x558071e21940 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f6800011c30 tx=0x7f68000047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40fec9df2b2a9af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:40.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.750+0000 7f67f8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:48:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f67f8ff9700 1 -- 172.21.15.114:0/4286061587 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6800028630 con 0x558071e18000 2024-05-27T07:48:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f67f8ff9700 1 -- 172.21.15.114:0/4286061587 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f680001f430 con 0x558071e18000 2024-05-27T07:48:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f6826472b80 1 -- 172.21.15.114:0/4286061587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f680c0a97e0 con 0x558071e18000 2024-05-27T07:48:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f67f8ff9700 1 -- 172.21.15.114:0/4286061587 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f68000287d0 con 0x558071e18000 2024-05-27T07:48:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f6826472b80 10 client.17511 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f6826472b80 1 -- 172.21.15.114:0/4286061587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x558071e2f350 con 0x558071e18000 2024-05-27T07:48:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f67f8ff9700 1 -- 172.21.15.114:0/4286061587 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f680c0a97e0 con 0x558071e18000 2024-05-27T07:48:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f67f8ff9700 1 -- 172.21.15.114:0/4286061587 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7f680001e440 con 0x558071e18000 2024-05-27T07:48:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f6826472b80 10 client.17511 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f6826472b80 10 client.17511 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f6826472b80 20 client.17511 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f6826472b80 1 --2- 172.21.15.114:0/4286061587 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558071e31ab0 0x558071e33f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f6826472b80 4 client.17511 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:48:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.751+0000 7f6826472b80 1 -- 172.21.15.114:0/4286061587 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558071de3a70 con 0x558071e31ab0 2024-05-27T07:48:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.752+0000 7f68157f9700 1 --2- 172.21.15.114:0/4286061587 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558071e31ab0 0x558071e33f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.752+0000 7f67f8ff9700 10 client.17511 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.752+0000 7f68157f9700 1 --2- 172.21.15.114:0/4286061587 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558071e31ab0 0x558071e33f20 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c52efa5989b3851f server_cookie=55dc2e217d41d8da in_seq=0 out_seq=0 2024-05-27T07:48:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f67f8ff9700 1 -- 172.21.15.114:0/4286061587 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558071de3a70 con 0x558071e31ab0 2024-05-27T07:48:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f67f8ff9700 10 client.17511 handle_command_reply: tid=0 2024-05-27T07:48:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f6826472b80 1 client.17511 shutdown 2024-05-27T07:48:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f6826472b80 1 -- 172.21.15.114:0/4286061587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e18000 msgr2=0x558071e21940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f6826472b80 1 --2- 172.21.15.114:0/4286061587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e18000 0x558071e21940 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f6800011c30 tx=0x7f68000047c0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f6826472b80 1 -- 172.21.15.114:0/4286061587 shutdown_connections 2024-05-27T07:48:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f6826472b80 1 --2- 172.21.15.114:0/4286061587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e1dcd0 0x558071e21e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f6826472b80 1 --2- 172.21.15.114:0/4286061587 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558071e31ab0 0x558071e33f20 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f6826472b80 1 --2- 172.21.15.114:0/4286061587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 0x558071e21400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f6826472b80 1 --2- 172.21.15.114:0/4286061587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e18000 0x558071e21940 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f67f8ff9700 0 client.17511 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:40.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f6826472b80 1 -- 172.21.15.114:0/4286061587 >> 172.21.15.114:0/4286061587 conn(0x558071cf4b70 msgr2=0x558071e16ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:40.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f6826472b80 1 -- 172.21.15.114:0/4286061587 shutdown_connections 2024-05-27T07:48:40.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.755+0000 7f6826472b80 1 -- 172.21.15.114:0/4286061587 wait complete. 2024-05-27T07:48:40.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.756+0000 7f6826472b80 20 client.17511 trim_cache size 0 max 16384 2024-05-27T07:48:40.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.778+0000 7f6826472b80 1 Processor -- start 2024-05-27T07:48:40.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.778+0000 7f6826472b80 1 -- start start 2024-05-27T07:48:40.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6826472b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 0x558071e17ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6826472b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e18000 0x558071e23870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6826472b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e2d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6826472b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f67f400ee70 con 0x558071e1dcd0 2024-05-27T07:48:40.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6826472b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f67f400f540 con 0x558071c99ec0 2024-05-27T07:48:40.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6826472b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f67f400ccc0 con 0x558071e18000 2024-05-27T07:48:40.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6815ffa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e2d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6814ff8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e18000 0x558071e23870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6815ffa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e2d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54898/0 (socket says 172.21.15.114:54898) 2024-05-27T07:48:40.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6815ffa700 1 -- 172.21.15.114:0/2943060091 learned_addr learned my addr 172.21.15.114:0/2943060091 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:40.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6814ff8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e18000 0x558071e23870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55866/0 (socket says 172.21.15.114:55866) 2024-05-27T07:48:40.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f68157f9700 1 --2- 172.21.15.114:0/2943060091 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 0x558071e17ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6814ff8700 1 -- 172.21.15.114:0/2943060091 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 msgr2=0x558071e17ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:40.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6814ff8700 1 --2- 172.21.15.114:0/2943060091 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 0x558071e17ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6814ff8700 1 -- 172.21.15.114:0/2943060091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 msgr2=0x558071e2d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:40.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6814ff8700 1 --2- 172.21.15.114:0/2943060091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e2d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6814ff8700 1 -- 172.21.15.114:0/2943060091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558071e1f970 con 0x558071e18000 2024-05-27T07:48:40.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6815ffa700 1 --2- 172.21.15.114:0/2943060091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e2d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:40.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f6814ff8700 1 --2- 172.21.15.114:0/2943060091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e18000 0x558071e23870 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6800011f40 tx=0x7f6800009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a34844d52e3fe90 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:40.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.779+0000 7f67faffd700 1 -- 172.21.15.114:0/2943060091 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6800028630 con 0x558071e18000 2024-05-27T07:48:40.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.780+0000 7f67faffd700 1 -- 172.21.15.114:0/2943060091 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6800028e20 con 0x558071e18000 2024-05-27T07:48:40.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.780+0000 7f67faffd700 1 -- 172.21.15.114:0/2943060091 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68000366b0 con 0x558071e18000 2024-05-27T07:48:40.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.780+0000 7f6826472b80 1 -- 172.21.15.114:0/2943060091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e18000 msgr2=0x558071e23870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:40.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.780+0000 7f6826472b80 1 --2- 172.21.15.114:0/2943060091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e18000 0x558071e23870 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6800011f40 tx=0x7f6800009f90 comp rx=0 tx=0).stop 2024-05-27T07:48:40.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.780+0000 7f6826472b80 1 -- 172.21.15.114:0/2943060091 shutdown_connections 2024-05-27T07:48:40.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.780+0000 7f6826472b80 1 --2- 172.21.15.114:0/2943060091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e2d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.780+0000 7f6826472b80 1 --2- 172.21.15.114:0/2943060091 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 0x558071e17ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.780+0000 7f6826472b80 1 --2- 172.21.15.114:0/2943060091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e18000 0x558071e23870 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.780+0000 7f6826472b80 1 -- 172.21.15.114:0/2943060091 >> 172.21.15.114:0/2943060091 conn(0x558071cf4b70 msgr2=0x558071e1b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:40.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.780+0000 7f6826472b80 1 -- 172.21.15.114:0/2943060091 shutdown_connections 2024-05-27T07:48:40.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.780+0000 7f6826472b80 1 -- 172.21.15.114:0/2943060091 wait complete. 2024-05-27T07:48:40.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.781+0000 7f6826472b80 1 -- start start 2024-05-27T07:48:40.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.781+0000 7f6826472b80 1 Processor -- start 2024-05-27T07:48:40.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.781+0000 7f6826472b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 0x558071e39400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.781+0000 7f6826472b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e18000 0x558071e39940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.781+0000 7f6826472b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e39e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.781+0000 7f6826472b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f67f400d330 con 0x558071e1dcd0 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.781+0000 7f6826472b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f67f400e7a0 con 0x558071c99ec0 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.781+0000 7f6826472b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f67f400da00 con 0x558071e18000 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f6826472b80 10 client.0 _ll_register_callbacks cb 0x558071cd19a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f6815ffa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e39e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f6814ff8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e18000 0x558071e39940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f6815ffa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e39e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54914/0 (socket says 172.21.15.114:54914) 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f6815ffa700 1 -- 172.21.15.114:0/3791955498 learned_addr learned my addr 172.21.15.114:0/3791955498 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f6814ff8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e18000 0x558071e39940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55878/0 (socket says 172.21.15.114:55878) 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f68157f9700 1 --2- 172.21.15.114:0/3791955498 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 0x558071e39400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f6815ffa700 1 -- 172.21.15.114:0/3791955498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e18000 msgr2=0x558071e39940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f6815ffa700 1 --2- 172.21.15.114:0/3791955498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e18000 0x558071e39940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f6815ffa700 1 -- 172.21.15.114:0/3791955498 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 msgr2=0x558071e39400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f6815ffa700 1 --2- 172.21.15.114:0/3791955498 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 0x558071e39400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f6815ffa700 1 -- 172.21.15.114:0/3791955498 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558071e9b720 con 0x558071e1dcd0 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f6815ffa700 1 --2- 172.21.15.114:0/3791955498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e39e80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f68080095e0 tx=0x7f68080075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=956a1c20fd136958 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f67f8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f67f8ff9700 1 -- 172.21.15.114:0/3791955498 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6808007a30 con 0x558071e1dcd0 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f67f8ff9700 1 -- 172.21.15.114:0/3791955498 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f6808007bd0 con 0x558071e1dcd0 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.782+0000 7f6826472b80 1 -- 172.21.15.114:0/3791955498 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6800028e20 con 0x558071e1dcd0 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.783+0000 7f67f8ff9700 1 -- 172.21.15.114:0/3791955498 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f6808030050 con 0x558071e1dcd0 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.783+0000 7f6826472b80 10 client.17517 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.783+0000 7f6826472b80 1 -- 172.21.15.114:0/3791955498 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x558071e9c9a0 con 0x558071e1dcd0 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.783+0000 7f67f8ff9700 1 -- 172.21.15.114:0/3791955498 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6800028e20 con 0x558071e1dcd0 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.783+0000 7f67f8ff9700 1 -- 172.21.15.114:0/3791955498 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7f6808011070 con 0x558071e1dcd0 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.783+0000 7f6826472b80 10 client.17517 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.783+0000 7f6826472b80 10 client.17517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.783+0000 7f6826472b80 20 client.17517 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:40.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.783+0000 7f6826472b80 1 --2- 172.21.15.114:0/3791955498 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558071e9d080 0x558071e9f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:40.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.783+0000 7f6826472b80 4 client.17517 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:48:40.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.783+0000 7f6826472b80 1 -- 172.21.15.114:0/3791955498 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558071d48c60 con 0x558071e9d080 2024-05-27T07:48:40.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.783+0000 7f68157f9700 1 --2- 172.21.15.114:0/3791955498 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558071e9d080 0x558071e9f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:40.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.784+0000 7f67f8ff9700 10 client.17517 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:40.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.784+0000 7f68157f9700 1 --2- 172.21.15.114:0/3791955498 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558071e9d080 0x558071e9f4a0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7395674e5d0ec6ff server_cookie=2e3cd6de7f8aec4 in_seq=0 out_seq=0 2024-05-27T07:48:40.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f67f8ff9700 1 -- 172.21.15.114:0/3791955498 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558071d48c60 con 0x558071e9d080 2024-05-27T07:48:40.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f67f8ff9700 10 client.17517 handle_command_reply: tid=0 2024-05-27T07:48:40.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f6826472b80 1 client.17517 shutdown 2024-05-27T07:48:40.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f6826472b80 1 -- 172.21.15.114:0/3791955498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 msgr2=0x558071e39e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:40.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f6826472b80 1 --2- 172.21.15.114:0/3791955498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e39e80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f68080095e0 tx=0x7f68080075c0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f6826472b80 1 -- 172.21.15.114:0/3791955498 shutdown_connections 2024-05-27T07:48:40.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f6826472b80 1 --2- 172.21.15.114:0/3791955498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558071e1dcd0 0x558071e39e80 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f6826472b80 1 --2- 172.21.15.114:0/3791955498 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558071e9d080 0x558071e9f4a0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f6826472b80 1 --2- 172.21.15.114:0/3791955498 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558071c99ec0 0x558071e39400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f6826472b80 1 --2- 172.21.15.114:0/3791955498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558071e18000 0x558071e39940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f67f8ff9700 0 client.17517 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:40.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f6826472b80 1 -- 172.21.15.114:0/3791955498 >> 172.21.15.114:0/3791955498 conn(0x558071cf4b70 msgr2=0x558071e1b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:40.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f6826472b80 1 -- 172.21.15.114:0/3791955498 shutdown_connections 2024-05-27T07:48:40.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.786+0000 7f6826472b80 1 -- 172.21.15.114:0/3791955498 wait complete. 2024-05-27T07:48:40.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.787+0000 7f6826472b80 20 client.17517 trim_cache size 0 max 16384 2024-05-27T07:48:40.790 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:48:40.790 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:48:40.790 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "2a7f49e4-c4d0-4376-9444-e364d8756282", 2024-05-27T07:48:40.790 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:48:40.790 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:48:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.787+0000 7f6817a5d700 1 -- 172.21.15.114:0/1146361369 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67fc0b2a70 msgr2=0x7f67fc0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.787+0000 7f6817a5d700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67fc0b2a70 0x7f67fc0b4f80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6810110510 tx=0x7f6800009260 comp rx=0 tx=0).stop 2024-05-27T07:48:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.787+0000 7f6817a5d700 1 -- 172.21.15.114:0/1146361369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68101045e0 msgr2=0x7f681019d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.787+0000 7f6817a5d700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68101045e0 0x7f681019d000 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f680c00c9b0 tx=0x7f680c00ccc0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.787+0000 7f6817a5d700 1 -- 172.21.15.114:0/1146361369 shutdown_connections 2024-05-27T07:48:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.787+0000 7f6817a5d700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6810109860 0x7f68101a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.787+0000 7f6817a5d700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68101045e0 0x7f681019d000 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.787+0000 7f6817a5d700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67fc0b2a70 0x7f67fc0b4f80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.787+0000 7f6817a5d700 1 --2- 172.21.15.114:0/1146361369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6810108f10 0x7f681019d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:40.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.787+0000 7f6817a5d700 1 -- 172.21.15.114:0/1146361369 >> 172.21.15.114:0/1146361369 conn(0x7f6810100030 msgr2=0x7f6810110dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:40.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.787+0000 7f6817a5d700 1 -- 172.21.15.114:0/1146361369 shutdown_connections 2024-05-27T07:48:40.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:40.787+0000 7f6817a5d700 1 -- 172.21.15.114:0/1146361369 wait complete. 2024-05-27T07:48:41.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 906725 11.05 MB/sec execute 293 sec latency 26.823 ms 2024-05-27T07:48:41.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:41 smithi114 ceph-mon[32038]: pgmap v290: 129 pgs: 129 active+clean; 73 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-05-27T07:48:41.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:41 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:48:41.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:41 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:48:41.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:41 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:48:41.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:41 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:48:41.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:48:41.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:41 smithi163 ceph-mon[36451]: pgmap v290: 129 pgs: 129 active+clean; 73 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-05-27T07:48:41.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:41 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:48:41.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:41 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:48:41.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:41 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:48:41.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:41 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:48:41.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:41 smithi186 ceph-mon[36647]: pgmap v290: 129 pgs: 129 active+clean; 73 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-05-27T07:48:41.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:41 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:48:41.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:41 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:48:41.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:41 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:48:41.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:41 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:48:41.908 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:48:42.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 909079 11.05 MB/sec execute 294 sec latency 18.119 ms 2024-05-27T07:48:42.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:42 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:48:42.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:42 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:48:42.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:42 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:48:43.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 910498 11.04 MB/sec execute 295 sec latency 39.523 ms 2024-05-27T07:48:43.589 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:43 smithi114 ceph-mon[32038]: pgmap v291: 129 pgs: 129 active+clean; 67 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 309 op/s 2024-05-27T07:48:43.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.708+0000 7f53b4643700 1 -- 172.21.15.114:0/2427788904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac1054f0 msgr2=0x7f53ac105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:43.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.708+0000 7f53b4643700 1 --2- 172.21.15.114:0/2427788904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac1054f0 0x7f53ac105900 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f539c009990 tx=0x7f539c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.709+0000 7f53b4643700 1 -- 172.21.15.114:0/2427788904 shutdown_connections 2024-05-27T07:48:43.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.709+0000 7f53b4643700 1 --2- 172.21.15.114:0/2427788904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac10e920 0x7f53ac110e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.709+0000 7f53b4643700 1 --2- 172.21.15.114:0/2427788904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac1054f0 0x7f53ac105900 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.709+0000 7f53b4643700 1 --2- 172.21.15.114:0/2427788904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53ac105e40 0x7f53ac10e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.709+0000 7f53b4643700 1 -- 172.21.15.114:0/2427788904 >> 172.21.15.114:0/2427788904 conn(0x7f53ac0fef20 msgr2=0x7f53ac101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:43.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.709+0000 7f53b4643700 1 -- 172.21.15.114:0/2427788904 shutdown_connections 2024-05-27T07:48:43.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.709+0000 7f53b4643700 1 -- 172.21.15.114:0/2427788904 wait complete. 2024-05-27T07:48:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.709+0000 7f53b4643700 1 Processor -- start 2024-05-27T07:48:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.709+0000 7f53b4643700 1 -- start start 2024-05-27T07:48:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b4643700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac1054f0 0x7f53ac1031e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b4643700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53ac105e40 0x7f53ac103720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b4643700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac10e920 0x7f53ac103c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b4643700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac114440 con 0x7f53ac1054f0 2024-05-27T07:48:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b4643700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1142c0 con 0x7f53ac105e40 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b4643700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1145c0 con 0x7f53ac10e920 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b23df700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac1054f0 0x7f53ac1031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b23df700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac1054f0 0x7f53ac1031e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54942/0 (socket says 172.21.15.114:54942) 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b23df700 1 -- 172.21.15.114:0/966307690 learned_addr learned my addr 172.21.15.114:0/966307690 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b2be0700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac10e920 0x7f53ac103c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b1bde700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53ac105e40 0x7f53ac103720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b2be0700 1 -- 172.21.15.114:0/966307690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53ac105e40 msgr2=0x7f53ac103720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b2be0700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53ac105e40 0x7f53ac103720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b2be0700 1 -- 172.21.15.114:0/966307690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac1054f0 msgr2=0x7f53ac1031e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b2be0700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac1054f0 0x7f53ac1031e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b2be0700 1 -- 172.21.15.114:0/966307690 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac1a9130 con 0x7f53ac10e920 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.710+0000 7f53b23df700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac1054f0 0x7f53ac1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.711+0000 7f53b2be0700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac10e920 0x7f53ac103c60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f53a400efc0 tx=0x7f53a400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.711+0000 7f53a37fe700 1 -- 172.21.15.114:0/966307690 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a40092e0 con 0x7f53ac10e920 2024-05-27T07:48:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.711+0000 7f53b4643700 1 -- 172.21.15.114:0/966307690 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ac1a93c0 con 0x7f53ac10e920 2024-05-27T07:48:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.711+0000 7f53a37fe700 1 -- 172.21.15.114:0/966307690 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53a4004bb0 con 0x7f53ac10e920 2024-05-27T07:48:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.711+0000 7f53a37fe700 1 -- 172.21.15.114:0/966307690 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4004020 con 0x7f53ac10e920 2024-05-27T07:48:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.711+0000 7f53b4643700 1 -- 172.21.15.114:0/966307690 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ac1a98b0 con 0x7f53ac10e920 2024-05-27T07:48:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.714+0000 7f53a37fe700 1 -- 172.21.15.114:0/966307690 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f53a401b030 con 0x7f53ac10e920 2024-05-27T07:48:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.715+0000 7f53c3058b80 1 Processor -- start 2024-05-27T07:48:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.715+0000 7f53c3058b80 1 -- start start 2024-05-27T07:48:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.715+0000 7f53c3058b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525c73e40 0x55a525df8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.715+0000 7f53c3058b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df9070 0x55a525df2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.715+0000 7f53c3058b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df3120 0x55a525df3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.715+0000 7f53c3058b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a525d20d70 con 0x55a525df3120 2024-05-27T07:48:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.715+0000 7f53c3058b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a525d10910 con 0x55a525c73e40 2024-05-27T07:48:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.715+0000 7f53c3058b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a525d14950 con 0x55a525df9070 2024-05-27T07:48:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.715+0000 7f53b1bde700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525c73e40 0x55a525df8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53a37fe700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f53980b29f0 0x7f53980b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53b1bde700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525c73e40 0x55a525df8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:48502/0 (socket says 172.21.15.114:48502) 2024-05-27T07:48:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53b1bde700 1 -- 172.21.15.114:0/3906720470 learned_addr learned my addr 172.21.15.114:0/3906720470 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53b1bde700 1 --2- 172.21.15.114:0/3906720470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df3120 0x55a525df3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53a37fe700 1 -- 172.21.15.114:0/966307690 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f53a40e2450 con 0x7f53ac10e920 2024-05-27T07:48:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53b1bde700 1 -- 172.21.15.114:0/3906720470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df9070 msgr2=0x55a525df2be0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:48:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53b1bde700 1 --2- 172.21.15.114:0/3906720470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df9070 0x55a525df2be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53b1bde700 1 -- 172.21.15.114:0/3906720470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df3120 msgr2=0x55a525df3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53b1bde700 1 --2- 172.21.15.114:0/3906720470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df3120 0x55a525df3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53b1bde700 1 -- 172.21.15.114:0/3906720470 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a525cb9fb0 con 0x55a525c73e40 2024-05-27T07:48:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53b2be0700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f53980b29f0 0x7f53980b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53b1bde700 1 --2- 172.21.15.114:0/3906720470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525c73e40 0x55a525df8b30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f53a8011880 tx=0x7f53a80058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5584689079c98996 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53a0ff9700 1 -- 172.21.15.114:0/3906720470 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8004030 con 0x55a525c73e40 2024-05-27T07:48:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53a0ff9700 1 -- 172.21.15.114:0/3906720470 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53a8004830 con 0x55a525c73e40 2024-05-27T07:48:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.716+0000 7f53a0ff9700 1 -- 172.21.15.114:0/3906720470 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8005260 con 0x55a525c73e40 2024-05-27T07:48:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.717+0000 7f53c3058b80 1 -- 172.21.15.114:0/3906720470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525c73e40 msgr2=0x55a525df8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.717+0000 7f53b2be0700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f53980b29f0 0x7f53980b4f00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f53a40df8a0 tx=0x7f53a4009040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.717+0000 7f53c3058b80 1 --2- 172.21.15.114:0/3906720470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525c73e40 0x55a525df8b30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f53a8011880 tx=0x7f53a80058e0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.717+0000 7f53c3058b80 1 -- 172.21.15.114:0/3906720470 shutdown_connections 2024-05-27T07:48:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.717+0000 7f53c3058b80 1 --2- 172.21.15.114:0/3906720470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df3120 0x55a525df3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.717+0000 7f53c3058b80 1 --2- 172.21.15.114:0/3906720470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525c73e40 0x55a525df8b30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.717+0000 7f53c3058b80 1 --2- 172.21.15.114:0/3906720470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df9070 0x55a525df2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.717+0000 7f53c3058b80 1 -- 172.21.15.114:0/3906720470 >> 172.21.15.114:0/3906720470 conn(0x55a525c76770 msgr2=0x55a525dedd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.717+0000 7f53c3058b80 1 -- 172.21.15.114:0/3906720470 shutdown_connections 2024-05-27T07:48:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.717+0000 7f53c3058b80 1 -- 172.21.15.114:0/3906720470 wait complete. 2024-05-27T07:48:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53c3058b80 1 -- start start 2024-05-27T07:48:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53c3058b80 1 Processor -- start 2024-05-27T07:48:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53c3058b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525c73e40 0x55a525dfc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53c3058b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525df3120 0x55a525dfcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53c3058b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df9070 0x55a525dfd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53c3058b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a525c16fa0 con 0x55a525df9070 2024-05-27T07:48:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53c3058b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a525c78da0 con 0x55a525df3120 2024-05-27T07:48:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53c3058b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a525dffb60 con 0x55a525c73e40 2024-05-27T07:48:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53c3058b80 10 client.0 _ll_register_callbacks cb 0x55a525ca75b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53b1bde700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525c73e40 0x55a525dfc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53b1bde700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525c73e40 0x55a525dfc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55922/0 (socket says 172.21.15.114:55922) 2024-05-27T07:48:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53b1bde700 1 -- 172.21.15.114:0/940297957 learned_addr learned my addr 172.21.15.114:0/940297957 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53b23df700 1 --2- 172.21.15.114:0/940297957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525df3120 0x55a525dfcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.718+0000 7f53b1bde700 1 --2- 172.21.15.114:0/940297957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df9070 0x55a525dfd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.719+0000 7f53b23df700 1 -- 172.21.15.114:0/940297957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525c73e40 msgr2=0x55a525dfc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.719+0000 7f53b23df700 1 --2- 172.21.15.114:0/940297957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525c73e40 0x55a525dfc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.719+0000 7f53b23df700 1 -- 172.21.15.114:0/940297957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df9070 msgr2=0x55a525dfd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.719+0000 7f53b23df700 1 --2- 172.21.15.114:0/940297957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df9070 0x55a525dfd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.719+0000 7f53b23df700 1 -- 172.21.15.114:0/940297957 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a525c608d0 con 0x55a525df3120 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.719+0000 7f53b23df700 1 --2- 172.21.15.114:0/940297957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525df3120 0x55a525dfcdd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f539c006010 tx=0x7f539c005ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7813b19149908bf9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.719+0000 7f5395ffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.719+0000 7f5395ffb700 1 -- 172.21.15.114:0/940297957 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c026060 con 0x55a525df3120 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.719+0000 7f53c3058b80 1 -- 172.21.15.114:0/940297957 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a8004830 con 0x55a525df3120 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.719+0000 7f5395ffb700 1 -- 172.21.15.114:0/940297957 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f539c005580 con 0x55a525df3120 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.719+0000 7f5395ffb700 1 -- 172.21.15.114:0/940297957 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f539c02d050 con 0x55a525df3120 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.719+0000 7f53c3058b80 10 client.26987 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.719+0000 7f53c3058b80 1 -- 172.21.15.114:0/940297957 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55a525cca760 con 0x55a525df3120 2024-05-27T07:48:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.720+0000 7f5395ffb700 1 -- 172.21.15.114:0/940297957 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53a8004830 con 0x55a525df3120 2024-05-27T07:48:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.720+0000 7f5395ffb700 1 -- 172.21.15.114:0/940297957 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7f539c003a50 con 0x55a525df3120 2024-05-27T07:48:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.721+0000 7f53c3058b80 10 client.26987 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.721+0000 7f53c3058b80 10 client.26987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.721+0000 7f53c3058b80 20 client.26987 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.721+0000 7f53c3058b80 1 --2- 172.21.15.114:0/940297957 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a525e0ca90 0x55a525e0eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.721+0000 7f53c3058b80 4 client.26987 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:48:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.721+0000 7f53c3058b80 1 -- 172.21.15.114:0/940297957 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a525d22cc0 con 0x55a525e0ca90 2024-05-27T07:48:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.721+0000 7f53b1bde700 1 --2- 172.21.15.114:0/940297957 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a525e0ca90 0x55a525e0eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.721+0000 7f5395ffb700 10 client.26987 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:43.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.721+0000 7f53b1bde700 1 --2- 172.21.15.114:0/940297957 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a525e0ca90 0x55a525e0eeb0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc97dbb259553b73 server_cookie=dcf009e032742f83 in_seq=0 out_seq=0 2024-05-27T07:48:43.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f5395ffb700 1 -- 172.21.15.114:0/940297957 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a525d22cc0 con 0x55a525e0ca90 2024-05-27T07:48:43.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f5395ffb700 10 client.26987 handle_command_reply: tid=0 2024-05-27T07:48:43.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f53c3058b80 1 client.26987 shutdown 2024-05-27T07:48:43.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f53c3058b80 1 -- 172.21.15.114:0/940297957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525df3120 msgr2=0x55a525dfcdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:43.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f53c3058b80 1 --2- 172.21.15.114:0/940297957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525df3120 0x55a525dfcdd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f539c006010 tx=0x7f539c005ee0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f53c3058b80 1 -- 172.21.15.114:0/940297957 shutdown_connections 2024-05-27T07:48:43.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f53c3058b80 1 --2- 172.21.15.114:0/940297957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df9070 0x55a525dfd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f53c3058b80 1 --2- 172.21.15.114:0/940297957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525c73e40 0x55a525dfc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f53c3058b80 1 --2- 172.21.15.114:0/940297957 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a525e0ca90 0x55a525e0eeb0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f53c3058b80 1 --2- 172.21.15.114:0/940297957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525df3120 0x55a525dfcdd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f5395ffb700 0 client.26987 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:43.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f53c3058b80 1 -- 172.21.15.114:0/940297957 >> 172.21.15.114:0/940297957 conn(0x55a525c76770 msgr2=0x55a525df1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:43.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f53c3058b80 1 -- 172.21.15.114:0/940297957 shutdown_connections 2024-05-27T07:48:43.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f53c3058b80 1 -- 172.21.15.114:0/940297957 wait complete. 2024-05-27T07:48:43.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.722+0000 7f53c3058b80 20 client.26987 trim_cache size 0 max 16384 2024-05-27T07:48:43.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:43 smithi163 ceph-mon[36451]: pgmap v291: 129 pgs: 129 active+clean; 67 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 309 op/s 2024-05-27T07:48:43.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:43 smithi186 ceph-mon[36647]: pgmap v291: 129 pgs: 129 active+clean; 67 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 309 op/s 2024-05-27T07:48:43.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.744+0000 7f53c3058b80 1 Processor -- start 2024-05-27T07:48:43.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.744+0000 7f53c3058b80 1 -- start start 2024-05-27T07:48:43.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53c3058b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525c73e40 0x55a525df2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53c3058b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df3120 0x55a525dfd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53c3058b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df9070 0x55a525de74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53c3058b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1b2950 con 0x55a525df9070 2024-05-27T07:48:43.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53c3058b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1b3020 con 0x55a525c73e40 2024-05-27T07:48:43.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53c3058b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1b07a0 con 0x55a525df3120 2024-05-27T07:48:43.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53b1bde700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df9070 0x55a525de74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53b1bde700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df9070 0x55a525de74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54972/0 (socket says 172.21.15.114:54972) 2024-05-27T07:48:43.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53b23df700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df3120 0x55a525dfd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53b1bde700 1 -- 172.21.15.114:0/3106674313 learned_addr learned my addr 172.21.15.114:0/3106674313 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:43.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53b1bde700 1 --2- 172.21.15.114:0/3106674313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525c73e40 0x55a525df2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53b1bde700 1 -- 172.21.15.114:0/3106674313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df3120 msgr2=0x55a525dfd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:43.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53b1bde700 1 --2- 172.21.15.114:0/3106674313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df3120 0x55a525dfd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53b1bde700 1 -- 172.21.15.114:0/3106674313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525c73e40 msgr2=0x55a525df2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:43.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53b1bde700 1 --2- 172.21.15.114:0/3106674313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525c73e40 0x55a525df2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53b1bde700 1 -- 172.21.15.114:0/3106674313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a525de9aa0 con 0x55a525df9070 2024-05-27T07:48:43.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f53b1bde700 1 --2- 172.21.15.114:0/3106674313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df9070 0x55a525de74f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f53a80116c0 tx=0x7f53a8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd71736c710fbdb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:43.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.745+0000 7f5397fff700 1 -- 172.21.15.114:0/3106674313 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8028040 con 0x55a525df9070 2024-05-27T07:48:43.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.746+0000 7f5397fff700 1 -- 172.21.15.114:0/3106674313 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53a8035070 con 0x55a525df9070 2024-05-27T07:48:43.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.746+0000 7f5397fff700 1 -- 172.21.15.114:0/3106674313 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a801f440 con 0x55a525df9070 2024-05-27T07:48:43.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.746+0000 7f53c3058b80 1 -- 172.21.15.114:0/3106674313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df9070 msgr2=0x55a525de74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:43.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.746+0000 7f53c3058b80 1 --2- 172.21.15.114:0/3106674313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df9070 0x55a525de74f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f53a80116c0 tx=0x7f53a8009da0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.746+0000 7f53c3058b80 1 -- 172.21.15.114:0/3106674313 shutdown_connections 2024-05-27T07:48:43.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.746+0000 7f53c3058b80 1 --2- 172.21.15.114:0/3106674313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525df9070 0x55a525de74f0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.746+0000 7f53c3058b80 1 --2- 172.21.15.114:0/3106674313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525c73e40 0x55a525df2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.746+0000 7f53c3058b80 1 --2- 172.21.15.114:0/3106674313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df3120 0x55a525dfd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.746+0000 7f53c3058b80 1 -- 172.21.15.114:0/3106674313 >> 172.21.15.114:0/3106674313 conn(0x55a525c76770 msgr2=0x55a525dfb180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:43.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53c3058b80 1 -- 172.21.15.114:0/3106674313 shutdown_connections 2024-05-27T07:48:43.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53c3058b80 1 -- 172.21.15.114:0/3106674313 wait complete. 2024-05-27T07:48:43.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53c3058b80 1 -- start start 2024-05-27T07:48:43.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53c3058b80 1 Processor -- start 2024-05-27T07:48:43.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53c3058b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525c73e40 0x55a525e14190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53c3058b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df3120 0x55a525e146d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53c3058b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525df9070 0x55a525e14c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53c3058b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1b0e10 con 0x55a525c73e40 2024-05-27T07:48:43.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53c3058b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1b2280 con 0x55a525df9070 2024-05-27T07:48:43.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53c3058b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1b14e0 con 0x55a525df3120 2024-05-27T07:48:43.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53c3058b80 10 client.0 _ll_register_callbacks cb 0x55a525ca75b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:43.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53b1bde700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525c73e40 0x55a525e14190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53b1bde700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525c73e40 0x55a525e14190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54976/0 (socket says 172.21.15.114:54976) 2024-05-27T07:48:43.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53b1bde700 1 -- 172.21.15.114:0/2495665139 learned_addr learned my addr 172.21.15.114:0/2495665139 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:43.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.747+0000 7f53b23df700 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df3120 0x55a525e146d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.748+0000 7f53b1bde700 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525df9070 0x55a525e14c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.748+0000 7f53b23df700 1 -- 172.21.15.114:0/2495665139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525df9070 msgr2=0x55a525e14c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:43.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.748+0000 7f53b23df700 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525df9070 0x55a525e14c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.748+0000 7f53b23df700 1 -- 172.21.15.114:0/2495665139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525c73e40 msgr2=0x55a525e14190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:43.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.748+0000 7f53b23df700 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525c73e40 0x55a525e14190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.748+0000 7f53b23df700 1 -- 172.21.15.114:0/2495665139 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a525e76530 con 0x55a525df3120 2024-05-27T07:48:43.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.748+0000 7f53b1bde700 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525df9070 0x55a525e14c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:48:43.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.748+0000 7f53b23df700 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df3120 0x55a525e146d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f539c003ea0 tx=0x7f539c0055b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1e60c5d44d68143 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:43.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.748+0000 7f5395ffb700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.748+0000 7f5395ffb700 1 -- 172.21.15.114:0/2495665139 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c026070 con 0x55a525df3120 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.748+0000 7f5395ffb700 1 -- 172.21.15.114:0/2495665139 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f539c003890 con 0x55a525df3120 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.748+0000 7f53c3058b80 1 -- 172.21.15.114:0/2495665139 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a8035070 con 0x55a525df3120 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.749+0000 7f5395ffb700 1 -- 172.21.15.114:0/2495665139 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f539c041050 con 0x55a525df3120 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.749+0000 7f53c3058b80 10 client.27115 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.749+0000 7f53c3058b80 1 -- 172.21.15.114:0/2495665139 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55a525e77720 con 0x55a525df3120 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.749+0000 7f5395ffb700 1 -- 172.21.15.114:0/2495665139 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53a8035070 con 0x55a525df3120 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.749+0000 7f5395ffb700 1 -- 172.21.15.114:0/2495665139 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7f539c003a90 con 0x55a525df3120 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.749+0000 7f53c3058b80 10 client.27115 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.749+0000 7f53c3058b80 10 client.27115 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.749+0000 7f53c3058b80 20 client.27115 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.749+0000 7f53c3058b80 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a525e77dc0 0x55a525e7a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.749+0000 7f53c3058b80 4 client.27115 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:48:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.749+0000 7f53c3058b80 1 -- 172.21.15.114:0/2495665139 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a525d1c4a0 con 0x55a525e77dc0 2024-05-27T07:48:43.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.749+0000 7f53b1bde700 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a525e77dc0 0x55a525e7a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:43.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.750+0000 7f5395ffb700 10 client.27115 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:43.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.750+0000 7f53b1bde700 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a525e77dc0 0x55a525e7a1e0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d82b0f268539d85 server_cookie=981e26796669b333 in_seq=0 out_seq=0 2024-05-27T07:48:43.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.751+0000 7f5395ffb700 1 -- 172.21.15.114:0/2495665139 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a525d1c4a0 con 0x55a525e77dc0 2024-05-27T07:48:43.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.751+0000 7f5395ffb700 10 client.27115 handle_command_reply: tid=0 2024-05-27T07:48:43.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.751+0000 7f53c3058b80 1 client.27115 shutdown 2024-05-27T07:48:43.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.751+0000 7f53c3058b80 1 -- 172.21.15.114:0/2495665139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df3120 msgr2=0x55a525e146d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:43.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.751+0000 7f53c3058b80 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df3120 0x55a525e146d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f539c003ea0 tx=0x7f539c0055b0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.751+0000 7f53c3058b80 1 -- 172.21.15.114:0/2495665139 shutdown_connections 2024-05-27T07:48:43.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.751+0000 7f53c3058b80 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a525e77dc0 0x55a525e7a1e0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.751+0000 7f53c3058b80 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a525df9070 0x55a525e14c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.751+0000 7f53c3058b80 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a525c73e40 0x55a525e14190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.751+0000 7f53c3058b80 1 --2- 172.21.15.114:0/2495665139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a525df3120 0x55a525e146d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.751+0000 7f53c3058b80 1 -- 172.21.15.114:0/2495665139 >> 172.21.15.114:0/2495665139 conn(0x55a525c76770 msgr2=0x55a525de74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:43.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.751+0000 7f5395ffb700 0 client.27115 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:43.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.752+0000 7f53c3058b80 1 -- 172.21.15.114:0/2495665139 shutdown_connections 2024-05-27T07:48:43.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.752+0000 7f53c3058b80 1 -- 172.21.15.114:0/2495665139 wait complete. 2024-05-27T07:48:43.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.752+0000 7f53c3058b80 20 client.27115 trim_cache size 0 max 16384 2024-05-27T07:48:43.756 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:48:43.757 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:48:43.757 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:48:43.757 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:48:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.752+0000 7f53b4643700 1 -- 172.21.15.114:0/966307690 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f53980b29f0 msgr2=0x7f53980b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.752+0000 7f53b4643700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f53980b29f0 0x7f53980b4f00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f53a40df8a0 tx=0x7f53a4009040 comp rx=0 tx=0).stop 2024-05-27T07:48:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.752+0000 7f53b4643700 1 -- 172.21.15.114:0/966307690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac10e920 msgr2=0x7f53ac103c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.752+0000 7f53b4643700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac10e920 0x7f53ac103c60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f53a400efc0 tx=0x7f53a400c3a0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.753+0000 7f53b4643700 1 -- 172.21.15.114:0/966307690 shutdown_connections 2024-05-27T07:48:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.753+0000 7f53b4643700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac10e920 0x7f53ac103c60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.753+0000 7f53b4643700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac1054f0 0x7f53ac1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.753+0000 7f53b4643700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f53980b29f0 0x7f53980b4f00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.753+0000 7f53b4643700 1 --2- 172.21.15.114:0/966307690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53ac105e40 0x7f53ac103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.753+0000 7f53b4643700 1 -- 172.21.15.114:0/966307690 >> 172.21.15.114:0/966307690 conn(0x7f53ac0fef20 msgr2=0x7f53ac100cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.753+0000 7f53b4643700 1 -- 172.21.15.114:0/966307690 shutdown_connections 2024-05-27T07:48:43.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:43.753+0000 7f53b4643700 1 -- 172.21.15.114:0/966307690 wait complete. 2024-05-27T07:48:44.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 912592 11.04 MB/sec execute 296 sec latency 76.374 ms 2024-05-27T07:48:44.992 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:48:44.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:48:45.250 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:48:45.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 913990 11.03 MB/sec execute 297 sec latency 86.942 ms 2024-05-27T07:48:45.642 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:45 smithi114 ceph-mon[32038]: pgmap v292: 129 pgs: 129 active+clean; 67 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 259 op/s 2024-05-27T07:48:45.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:45 smithi163 ceph-mon[36451]: pgmap v292: 129 pgs: 129 active+clean; 67 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 259 op/s 2024-05-27T07:48:45.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:45 smithi186 ceph-mon[36647]: pgmap v292: 129 pgs: 129 active+clean; 67 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 259 op/s 2024-05-27T07:48:46.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 916504 11.04 MB/sec execute 298 sec latency 11.801 ms 2024-05-27T07:48:47.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.121+0000 7f674d73b700 1 -- 172.21.15.114:0/3099053585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6748104600 msgr2=0x7f6748104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:47.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.121+0000 7f674d73b700 1 --2- 172.21.15.114:0/3099053585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6748104600 0x7f6748104a10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f67380099c0 tx=0x7f6738009cd0 comp rx=0 tx=0).stop 2024-05-27T07:48:47.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.122+0000 7f674d73b700 1 -- 172.21.15.114:0/3099053585 shutdown_connections 2024-05-27T07:48:47.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.122+0000 7f674d73b700 1 --2- 172.21.15.114:0/3099053585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6748109880 0x7f67481111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:47.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.122+0000 7f674d73b700 1 --2- 172.21.15.114:0/3099053585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6748104600 0x7f6748104a10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:47.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.122+0000 7f674d73b700 1 --2- 172.21.15.114:0/3099053585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6748108f30 0x7f6748109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:47.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.122+0000 7f674d73b700 1 -- 172.21.15.114:0/3099053585 >> 172.21.15.114:0/3099053585 conn(0x7f6748079f00 msgr2=0x7f674807a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:47.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.122+0000 7f674d73b700 1 -- 172.21.15.114:0/3099053585 shutdown_connections 2024-05-27T07:48:47.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.122+0000 7f674d73b700 1 -- 172.21.15.114:0/3099053585 wait complete. 2024-05-27T07:48:47.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.123+0000 7f674d73b700 1 Processor -- start 2024-05-27T07:48:47.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.123+0000 7f674d73b700 1 -- start start 2024-05-27T07:48:47.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.123+0000 7f674d73b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6748104600 0x7f674819aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:47.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.123+0000 7f674d73b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6748108f30 0x7f674819b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:47.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.123+0000 7f674d73b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6748109880 0x7f674819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:47.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.123+0000 7f674d73b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f67481144b0 con 0x7f6748109880 2024-05-27T07:48:47.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.123+0000 7f674d73b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6748114330 con 0x7f6748108f30 2024-05-27T07:48:47.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.123+0000 7f674d73b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6748114630 con 0x7f6748104600 2024-05-27T07:48:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.123+0000 7f67467fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6748108f30 0x7f674819b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.123+0000 7f67477fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6748109880 0x7f674819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.123+0000 7f67467fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6748108f30 0x7f674819b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:57956/0 (socket says 172.21.15.114:57956) 2024-05-27T07:48:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.123+0000 7f67477fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6748109880 0x7f674819f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39500/0 (socket says 172.21.15.114:39500) 2024-05-27T07:48:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.123+0000 7f67467fc700 1 -- 172.21.15.114:0/2603371961 learned_addr learned my addr 172.21.15.114:0/2603371961 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.124+0000 7f6746ffd700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6748104600 0x7f674819aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.124+0000 7f67467fc700 1 -- 172.21.15.114:0/2603371961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6748104600 msgr2=0x7f674819aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.124+0000 7f67467fc700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6748104600 0x7f674819aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.124+0000 7f67467fc700 1 -- 172.21.15.114:0/2603371961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6748109880 msgr2=0x7f674819f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.124+0000 7f67467fc700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6748109880 0x7f674819f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.124+0000 7f67467fc700 1 -- 172.21.15.114:0/2603371961 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674819fd20 con 0x7f6748108f30 2024-05-27T07:48:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.124+0000 7f67477fe700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6748109880 0x7f674819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.124+0000 7f67467fc700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6748108f30 0x7f674819b440 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f673c009d10 tx=0x7f673c00b5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.124+0000 7f672ffff700 1 -- 172.21.15.114:0/2603371961 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c009e00 con 0x7f6748108f30 2024-05-27T07:48:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.124+0000 7f674d73b700 1 -- 172.21.15.114:0/2603371961 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67481a0010 con 0x7f6748108f30 2024-05-27T07:48:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.124+0000 7f674d73b700 1 -- 172.21.15.114:0/2603371961 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67480759e0 con 0x7f6748108f30 2024-05-27T07:48:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.124+0000 7f672ffff700 1 -- 172.21.15.114:0/2603371961 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f673c00bd10 con 0x7f6748108f30 2024-05-27T07:48:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.126+0000 7f672ffff700 1 -- 172.21.15.114:0/2603371961 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c019410 con 0x7f6748108f30 2024-05-27T07:48:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.126+0000 7f674d73b700 1 -- 172.21.15.114:0/2603371961 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6734005330 con 0x7f6748108f30 2024-05-27T07:48:47.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.127+0000 7f672ffff700 1 -- 172.21.15.114:0/2603371961 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f673c04bb30 con 0x7f6748108f30 2024-05-27T07:48:47.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.128+0000 7f672ffff700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67300b28d0 0x7f67300b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:47.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.128+0000 7f6746ffd700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67300b28d0 0x7f67300b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:47.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.128+0000 7f672ffff700 1 -- 172.21.15.114:0/2603371961 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f673c0e62e0 con 0x7f6748108f30 2024-05-27T07:48:47.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.129+0000 7f6746ffd700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67300b28d0 0x7f67300b4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6738000c00 tx=0x7f673801a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:47.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.132+0000 7f672ffff700 1 -- 172.21.15.114:0/2603371961 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f673c0ad910 con 0x7f6748108f30 2024-05-27T07:48:47.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 917851 11.08 MB/sec execute 299 sec latency 65.334 ms 2024-05-27T07:48:47.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.429+0000 7f674d73b700 1 -- 172.21.15.114:0/2603371961 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6734005620 con 0x7f6748108f30 2024-05-27T07:48:47.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.429+0000 7f672ffff700 1 -- 172.21.15.114:0/2603371961 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+10175 (secure 0 0 0) 0x7f673c04be20 con 0x7f6748108f30 2024-05-27T07:48:47.430 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:48:47.430 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":26882,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2622386718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2622386718},{"type":"v1","addr":"172.21.15.186:6835","nonce":2622386718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":50,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:48:32.376993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26921":{"gid":26921,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/870343693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":870343693},{"type":"v1","addr":"172.21.15.163:6837","nonce":870343693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:48:47.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.432+0000 7f674d73b700 1 -- 172.21.15.114:0/2603371961 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67300b28d0 msgr2=0x7f67300b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:47.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.432+0000 7f674d73b700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67300b28d0 0x7f67300b4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6738000c00 tx=0x7f673801a040 comp rx=0 tx=0).stop 2024-05-27T07:48:47.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.432+0000 7f674d73b700 1 -- 172.21.15.114:0/2603371961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6748108f30 msgr2=0x7f674819b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:47.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.432+0000 7f674d73b700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6748108f30 0x7f674819b440 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f673c009d10 tx=0x7f673c00b5a0 comp rx=0 tx=0).stop 2024-05-27T07:48:47.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.432+0000 7f674d73b700 1 -- 172.21.15.114:0/2603371961 shutdown_connections 2024-05-27T07:48:47.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.432+0000 7f674d73b700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6748109880 0x7f674819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:47.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.432+0000 7f674d73b700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6748104600 0x7f674819aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:47.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.432+0000 7f674d73b700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67300b28d0 0x7f67300b4de0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:47.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.432+0000 7f674d73b700 1 --2- 172.21.15.114:0/2603371961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6748108f30 0x7f674819b440 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:47.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.432+0000 7f674d73b700 1 -- 172.21.15.114:0/2603371961 >> 172.21.15.114:0/2603371961 conn(0x7f6748079f00 msgr2=0x7f6748102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:47.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.432+0000 7f674d73b700 1 -- 172.21.15.114:0/2603371961 shutdown_connections 2024-05-27T07:48:47.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:47.432+0000 7f674d73b700 1 -- 172.21.15.114:0/2603371961 wait complete. 2024-05-27T07:48:47.433 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 50 2024-05-27T07:48:47.475 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:47 smithi163 ceph-mon[36451]: pgmap v293: 129 pgs: 129 active+clean; 74 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 358 op/s 2024-05-27T07:48:47.476 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffb8f9e0700 time 2024-05-27T07:48:47.201273+0000 2024-05-27T07:48:47.476 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:47.476 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:47.476 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffba2225ba6] 2024-05-27T07:48:47.476 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffba2225dd1] 2024-05-27T07:48:47.476 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e52378123d] 2024-05-27T07:48:47.476 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e52376c971] 2024-05-27T07:48:47.476 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e52376afa0] 2024-05-27T07:48:47.476 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e52376b651] 2024-05-27T07:48:47.477 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e523700347] 2024-05-27T07:48:47.477 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e5233fc87e] 2024-05-27T07:48:47.477 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e523401f2f] 2024-05-27T07:48:47.477 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e52340c969] 2024-05-27T07:48:47.477 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e5237a548c] 2024-05-27T07:48:47.477 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e5237ae9cb] 2024-05-27T07:48:47.477 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 13: (Thread::entry_wrapper()+0x43) [0x7ffba21f472d] 2024-05-27T07:48:47.477 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffba21f4749] 2024-05-27T07:48:47.477 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffba0d981ca] 2024-05-27T07:48:47.477 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 16: clone() 2024-05-27T07:48:47.477 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: *** Caught signal (Aborted) ** 2024-05-27T07:48:47.477 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: in thread 7ffb8f9e0700 thread_name:md_log_replay 2024-05-27T07:48:47.478 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.202+0000 7ffb8f9e0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffb8f9e0700 time 2024-05-27T07:48:47.201273+0000 2024-05-27T07:48:47.478 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:47.478 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.478 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:47.479 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffba2225ba6] 2024-05-27T07:48:47.479 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffba2225dd1] 2024-05-27T07:48:47.479 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e52378123d] 2024-05-27T07:48:47.479 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e52376c971] 2024-05-27T07:48:47.479 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e52376afa0] 2024-05-27T07:48:47.479 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e52376b651] 2024-05-27T07:48:47.479 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e523700347] 2024-05-27T07:48:47.479 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e5233fc87e] 2024-05-27T07:48:47.479 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e523401f2f] 2024-05-27T07:48:47.479 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e52340c969] 2024-05-27T07:48:47.479 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e5237a548c] 2024-05-27T07:48:47.479 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e5237ae9cb] 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 13: (Thread::entry_wrapper()+0x43) [0x7ffba21f472d] 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffba21f4749] 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffba0d981ca] 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 16: clone() 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffba0da2d20] 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2: gsignal() 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 3: abort() 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffba2225ced] 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffba2225dd1] 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e52378123d] 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e52376c971] 2024-05-27T07:48:47.480 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e52376afa0] 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e52376b651] 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e523700347] 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e5233fc87e] 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e523401f2f] 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e52340c969] 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e5237a548c] 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e5237ae9cb] 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 16: (Thread::entry_wrapper()+0x43) [0x7ffba21f472d] 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffba21f4749] 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffba0d981ca] 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 19: clone() 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.205+0000 7ffb8f9e0700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: in thread 7ffb8f9e0700 thread_name:md_log_replay 2024-05-27T07:48:47.481 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:47.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffba0da2d20] 2024-05-27T07:48:47.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2: gsignal() 2024-05-27T07:48:47.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 3: abort() 2024-05-27T07:48:47.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffba2225ced] 2024-05-27T07:48:47.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffba2225dd1] 2024-05-27T07:48:47.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e52378123d] 2024-05-27T07:48:47.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e52376c971] 2024-05-27T07:48:47.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e52376afa0] 2024-05-27T07:48:47.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e52376b651] 2024-05-27T07:48:47.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e523700347] 2024-05-27T07:48:47.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e5233fc87e] 2024-05-27T07:48:47.483 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e523401f2f] 2024-05-27T07:48:47.483 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e52340c969] 2024-05-27T07:48:47.483 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e5237a548c] 2024-05-27T07:48:47.483 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e5237ae9cb] 2024-05-27T07:48:47.483 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 16: (Thread::entry_wrapper()+0x43) [0x7ffba21f472d] 2024-05-27T07:48:47.483 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffba21f4749] 2024-05-27T07:48:47.483 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffba0d981ca] 2024-05-27T07:48:47.483 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 19: clone() 2024-05-27T07:48:47.483 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:48:47.483 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.483 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: -1> 2024-05-27T07:48:47.202+0000 7ffb8f9e0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffb8f9e0700 time 2024-05-27T07:48:47.201273+0000 2024-05-27T07:48:47.483 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:47.483 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffba2225ba6] 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffba2225dd1] 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e52378123d] 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e52376c971] 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e52376afa0] 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e52376b651] 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e523700347] 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e5233fc87e] 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e523401f2f] 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e52340c969] 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e5237a548c] 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e5237ae9cb] 2024-05-27T07:48:47.484 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 13: (Thread::entry_wrapper()+0x43) [0x7ffba21f472d] 2024-05-27T07:48:47.485 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffba21f4749] 2024-05-27T07:48:47.485 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffba0d981ca] 2024-05-27T07:48:47.485 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 16: clone() 2024-05-27T07:48:47.485 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.485 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 0> 2024-05-27T07:48:47.205+0000 7ffb8f9e0700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:48:47.485 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: in thread 7ffb8f9e0700 thread_name:md_log_replay 2024-05-27T07:48:47.485 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.485 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:47.485 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffba0da2d20] 2024-05-27T07:48:47.485 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2: gsignal() 2024-05-27T07:48:47.485 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 3: abort() 2024-05-27T07:48:47.485 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffba2225ced] 2024-05-27T07:48:47.485 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffba2225dd1] 2024-05-27T07:48:47.486 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e52378123d] 2024-05-27T07:48:47.486 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e52376c971] 2024-05-27T07:48:47.486 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e52376afa0] 2024-05-27T07:48:47.486 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e52376b651] 2024-05-27T07:48:47.486 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e523700347] 2024-05-27T07:48:47.486 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e5233fc87e] 2024-05-27T07:48:47.486 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e523401f2f] 2024-05-27T07:48:47.486 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e52340c969] 2024-05-27T07:48:47.486 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e5237a548c] 2024-05-27T07:48:47.486 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e5237ae9cb] 2024-05-27T07:48:47.486 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 16: (Thread::entry_wrapper()+0x43) [0x7ffba21f472d] 2024-05-27T07:48:47.487 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffba21f4749] 2024-05-27T07:48:47.487 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffba0d981ca] 2024-05-27T07:48:47.487 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 19: clone() 2024-05-27T07:48:47.487 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:48:47.487 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.487 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: -1> 2024-05-27T07:48:47.202+0000 7ffb8f9e0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffb8f9e0700 time 2024-05-27T07:48:47.201273+0000 2024-05-27T07:48:47.487 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:47.487 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.487 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:47.488 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffba2225ba6] 2024-05-27T07:48:47.488 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffba2225dd1] 2024-05-27T07:48:47.488 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e52378123d] 2024-05-27T07:48:47.488 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e52376c971] 2024-05-27T07:48:47.488 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e52376afa0] 2024-05-27T07:48:47.488 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e52376b651] 2024-05-27T07:48:47.488 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e523700347] 2024-05-27T07:48:47.488 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e5233fc87e] 2024-05-27T07:48:47.488 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e523401f2f] 2024-05-27T07:48:47.488 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e52340c969] 2024-05-27T07:48:47.488 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e5237a548c] 2024-05-27T07:48:47.488 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e5237ae9cb] 2024-05-27T07:48:47.488 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 13: (Thread::entry_wrapper()+0x43) [0x7ffba21f472d] 2024-05-27T07:48:47.489 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffba21f4749] 2024-05-27T07:48:47.489 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffba0d981ca] 2024-05-27T07:48:47.489 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 16: clone() 2024-05-27T07:48:47.489 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.489 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 0> 2024-05-27T07:48:47.205+0000 7ffb8f9e0700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:48:47.489 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: in thread 7ffb8f9e0700 thread_name:md_log_replay 2024-05-27T07:48:47.489 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.489 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:47.489 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffba0da2d20] 2024-05-27T07:48:47.489 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2: gsignal() 2024-05-27T07:48:47.489 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 3: abort() 2024-05-27T07:48:47.489 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffba2225ced] 2024-05-27T07:48:47.489 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffba2225dd1] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e52378123d] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e52376c971] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e52376afa0] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e52376b651] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e523700347] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e5233fc87e] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e523401f2f] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e52340c969] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e5237a548c] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e5237ae9cb] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 16: (Thread::entry_wrapper()+0x43) [0x7ffba21f472d] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffba21f4749] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffba0d981ca] 2024-05-27T07:48:47.490 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 19: clone() 2024-05-27T07:48:47.491 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:48:47.491 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71706]: 2024-05-27T07:48:47.491 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 podman[71756]: 2024-05-27 07:48:47.265523773 +0000 UTC m=+0.018962055 container died 46c67c1bcf880a570b232be935f174b39273a00e4e871fec3d4bc1f7027819d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521) 2024-05-27T07:48:47.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:47 smithi186 ceph-mon[36647]: pgmap v293: 129 pgs: 129 active+clean; 74 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 358 op/s 2024-05-27T07:48:47.756 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:47 smithi114 ceph-mon[32038]: pgmap v293: 129 pgs: 129 active+clean; 74 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 358 op/s 2024-05-27T07:48:48.187 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:48:48.194 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 podman[71756]: 2024-05-27 07:48:47.940879758 +0000 UTC m=+0.694318039 container remove 46c67c1bcf880a570b232be935f174b39273a00e4e871fec3d4bc1f7027819d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:48:48.194 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:47 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:48:48.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 920524 11.10 MB/sec execute 300 sec latency 17.198 ms 2024-05-27T07:48:48.451 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:48:48.475 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:48 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2603371961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:48:48.476 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:48 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-05-27T07:48:48.476 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:48 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Consumed 1.872s CPU time. 2024-05-27T07:48:48.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:48 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2603371961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:48:48.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:48 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2603371961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:48:49.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 921468 11.07 MB/sec execute 301 sec latency 69.525 ms 2024-05-27T07:48:49.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:49 smithi163 ceph-mon[36451]: pgmap v294: 129 pgs: 129 active+clean; 74 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 276 op/s 2024-05-27T07:48:49.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:49 smithi186 ceph-mon[36647]: pgmap v294: 129 pgs: 129 active+clean; 74 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 276 op/s 2024-05-27T07:48:49.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:49 smithi114 ceph-mon[32038]: pgmap v294: 129 pgs: 129 active+clean; 74 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 276 op/s 2024-05-27T07:48:50.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 924187 11.09 MB/sec execute 302 sec latency 16.639 ms 2024-05-27T07:48:50.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.517+0000 7f780c171700 1 -- 172.21.15.114:0/1717750851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78041029b0 msgr2=0x7f7804102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.517+0000 7f780c171700 1 --2- 172.21.15.114:0/1717750851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78041029b0 0x7f7804102dc0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f77f400aaa0 tx=0x7f77f400adb0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.518+0000 7f780c171700 1 -- 172.21.15.114:0/1717750851 shutdown_connections 2024-05-27T07:48:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.518+0000 7f780c171700 1 --2- 172.21.15.114:0/1717750851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7804103300 0x7f780410f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.518+0000 7f780c171700 1 --2- 172.21.15.114:0/1717750851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78041089e0 0x7f7804108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.518+0000 7f780c171700 1 --2- 172.21.15.114:0/1717750851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78041029b0 0x7f7804102dc0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.518+0000 7f780c171700 1 -- 172.21.15.114:0/1717750851 >> 172.21.15.114:0/1717750851 conn(0x7f78040fe6e0 msgr2=0x7f7804100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.518+0000 7f780c171700 1 -- 172.21.15.114:0/1717750851 shutdown_connections 2024-05-27T07:48:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.518+0000 7f780c171700 1 -- 172.21.15.114:0/1717750851 wait complete. 2024-05-27T07:48:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.518+0000 7f780c171700 1 Processor -- start 2024-05-27T07:48:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.518+0000 7f780c171700 1 -- start start 2024-05-27T07:48:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780c171700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78041029b0 0x7f780419ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780c171700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7804103300 0x7f780419b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780c171700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78041089e0 0x7f780419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780c171700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78041122d0 con 0x7f7804103300 2024-05-27T07:48:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780c171700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7804112150 con 0x7f78041089e0 2024-05-27T07:48:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780c171700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7804112450 con 0x7f78041029b0 2024-05-27T07:48:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780970c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7804103300 0x7f780419b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f7809f0d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78041029b0 0x7f780419ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780970c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7804103300 0x7f780419b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39518/0 (socket says 172.21.15.114:39518) 2024-05-27T07:48:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f7809f0d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78041029b0 0x7f780419ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47916/0 (socket says 172.21.15.114:47916) 2024-05-27T07:48:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780970c700 1 -- 172.21.15.114:0/3001706108 learned_addr learned my addr 172.21.15.114:0/3001706108 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780a70e700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78041089e0 0x7f780419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780970c700 1 -- 172.21.15.114:0/3001706108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78041029b0 msgr2=0x7f780419ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780970c700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78041029b0 0x7f780419ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780970c700 1 -- 172.21.15.114:0/3001706108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78041089e0 msgr2=0x7f780419f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780970c700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78041089e0 0x7f780419f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780970c700 1 -- 172.21.15.114:0/3001706108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780419fd50 con 0x7f7804103300 2024-05-27T07:48:50.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.519+0000 7f780a70e700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78041089e0 0x7f780419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:50.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.520+0000 7f780970c700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7804103300 0x7f780419b420 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f77f4000c00 tx=0x7f77f4003930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:50.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.520+0000 7f77faffd700 1 -- 172.21.15.114:0/3001706108 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f4023020 con 0x7f7804103300 2024-05-27T07:48:50.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.520+0000 7f780c171700 1 -- 172.21.15.114:0/3001706108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f780419ffe0 con 0x7f7804103300 2024-05-27T07:48:50.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.520+0000 7f77faffd700 1 -- 172.21.15.114:0/3001706108 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77f4004310 con 0x7f7804103300 2024-05-27T07:48:50.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.520+0000 7f77faffd700 1 -- 172.21.15.114:0/3001706108 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f400ee40 con 0x7f7804103300 2024-05-27T07:48:50.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.520+0000 7f780c171700 1 -- 172.21.15.114:0/3001706108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78041a7960 con 0x7f7804103300 2024-05-27T07:48:50.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.523+0000 7f77faffd700 1 -- 172.21.15.114:0/3001706108 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f77f4026020 con 0x7f7804103300 2024-05-27T07:48:50.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.523+0000 7f77faffd700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f77f00b2a70 0x7f77f00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f781ab86b80 1 Processor -- start 2024-05-27T07:48:50.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f7809f0d700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f77f00b2a70 0x7f77f00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f77faffd700 1 -- 172.21.15.114:0/3001706108 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f77f40e3df0 con 0x7f7804103300 2024-05-27T07:48:50.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f781ab86b80 1 -- start start 2024-05-27T07:48:50.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f781ab86b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x56287534db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f781ab86b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56287534e070 0x562875347be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f781ab86b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x562875348530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f781ab86b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562875275d70 con 0x562875348120 2024-05-27T07:48:50.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f781ab86b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562875265910 con 0x5628751c8e40 2024-05-27T07:48:50.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f781ab86b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562875269950 con 0x56287534e070 2024-05-27T07:48:50.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f7809f0d700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f77f00b2a70 0x7f77f00b4f80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7804103af0 tx=0x7f7800009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:50.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f780a70e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x562875348530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f780970c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56287534e070 0x562875347be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f780a70e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x562875348530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39532/0 (socket says 172.21.15.114:39532) 2024-05-27T07:48:50.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f7809f0d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x56287534db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.524+0000 7f780a70e700 1 -- 172.21.15.114:0/1568030140 learned_addr learned my addr 172.21.15.114:0/1568030140 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:50.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.525+0000 7f780970c700 1 -- 172.21.15.114:0/1568030140 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 msgr2=0x56287534db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:50.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.525+0000 7f780970c700 1 --2- 172.21.15.114:0/1568030140 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x56287534db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.525+0000 7f780970c700 1 -- 172.21.15.114:0/1568030140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 msgr2=0x562875348530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:50.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.525+0000 7f780970c700 1 --2- 172.21.15.114:0/1568030140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x562875348530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.525+0000 7f780970c700 1 -- 172.21.15.114:0/1568030140 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56287520efb0 con 0x56287534e070 2024-05-27T07:48:50.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.525+0000 7f780970c700 1 --2- 172.21.15.114:0/1568030140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56287534e070 0x562875347be0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f77f40e1190 tx=0x7f77f40aa780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1202302c537bccf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:50.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.525+0000 7f77effff700 1 -- 172.21.15.114:0/1568030140 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f40c1070 con 0x56287534e070 2024-05-27T07:48:50.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.525+0000 7f77effff700 1 -- 172.21.15.114:0/1568030140 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77f40aadb0 con 0x56287534e070 2024-05-27T07:48:50.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.525+0000 7f77effff700 1 -- 172.21.15.114:0/1568030140 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f400e570 con 0x56287534e070 2024-05-27T07:48:50.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.526+0000 7f781ab86b80 1 -- 172.21.15.114:0/1568030140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56287534e070 msgr2=0x562875347be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:50.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.526+0000 7f781ab86b80 1 --2- 172.21.15.114:0/1568030140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56287534e070 0x562875347be0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f77f40e1190 tx=0x7f77f40aa780 comp rx=0 tx=0).stop 2024-05-27T07:48:50.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.526+0000 7f781ab86b80 1 -- 172.21.15.114:0/1568030140 shutdown_connections 2024-05-27T07:48:50.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.526+0000 7f781ab86b80 1 --2- 172.21.15.114:0/1568030140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x562875348530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.526+0000 7f781ab86b80 1 --2- 172.21.15.114:0/1568030140 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x56287534db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.526+0000 7f781ab86b80 1 --2- 172.21.15.114:0/1568030140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56287534e070 0x562875347be0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.526+0000 7f781ab86b80 1 -- 172.21.15.114:0/1568030140 >> 172.21.15.114:0/1568030140 conn(0x5628751cb770 msgr2=0x562875342d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:50.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.527+0000 7f781ab86b80 1 -- 172.21.15.114:0/1568030140 shutdown_connections 2024-05-27T07:48:50.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.527+0000 7f781ab86b80 1 -- 172.21.15.114:0/1568030140 wait complete. 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.527+0000 7f781ab86b80 1 -- start start 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.527+0000 7f781ab86b80 1 Processor -- start 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f781ab86b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x562875351930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f781ab86b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562875348120 0x562875351e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f781ab86b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56287534e070 0x5628753523b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f781ab86b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56287516bfa0 con 0x56287534e070 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f781ab86b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5628751cdda0 con 0x5628751c8e40 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f781ab86b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562875354920 con 0x562875348120 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f7809f0d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x562875351930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f780a70e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56287534e070 0x5628753523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f7809f0d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x562875351930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:58000/0 (socket says 172.21.15.114:58000) 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f780a70e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56287534e070 0x5628753523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39544/0 (socket says 172.21.15.114:39544) 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f7809f0d700 1 -- 172.21.15.114:0/3511162430 learned_addr learned my addr 172.21.15.114:0/3511162430 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f780970c700 1 --2- 172.21.15.114:0/3511162430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562875348120 0x562875351e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f781ab86b80 10 client.0 _ll_register_callbacks cb 0x5628751fc5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f780970c700 1 -- 172.21.15.114:0/3511162430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 msgr2=0x562875351930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f780970c700 1 --2- 172.21.15.114:0/3511162430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x562875351930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f780970c700 1 -- 172.21.15.114:0/3511162430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56287534e070 msgr2=0x5628753523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f780970c700 1 --2- 172.21.15.114:0/3511162430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56287534e070 0x5628753523b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f780970c700 1 -- 172.21.15.114:0/3511162430 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628751b58d0 con 0x562875348120 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.528+0000 7f7809f0d700 1 --2- 172.21.15.114:0/3511162430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x562875351930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.529+0000 7f780970c700 1 --2- 172.21.15.114:0/3511162430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562875348120 0x562875351e70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f77f40e11c0 tx=0x7f77f400a7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d66201ba1f5b708 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.529+0000 7f77ed7fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.529+0000 7f77ed7fa700 1 -- 172.21.15.114:0/3511162430 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f40c1040 con 0x562875348120 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.529+0000 7f77ed7fa700 1 -- 172.21.15.114:0/3511162430 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f77f400e570 con 0x562875348120 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.529+0000 7f781ab86b80 1 -- 172.21.15.114:0/3511162430 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77f40aadb0 con 0x562875348120 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.529+0000 7f77ed7fa700 1 -- 172.21.15.114:0/3511162430 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f77f40c53f0 con 0x562875348120 2024-05-27T07:48:50.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.529+0000 7f781ab86b80 10 client.27133 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:50.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.529+0000 7f781ab86b80 1 -- 172.21.15.114:0/3511162430 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x56287521f760 con 0x562875348120 2024-05-27T07:48:50.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.529+0000 7f77ed7fa700 1 -- 172.21.15.114:0/3511162430 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77f40aadb0 con 0x562875348120 2024-05-27T07:48:50.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.529+0000 7f77ed7fa700 1 -- 172.21.15.114:0/3511162430 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7f77f40c4070 con 0x562875348120 2024-05-27T07:48:50.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.529+0000 7f781ab86b80 10 client.27133 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:50.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.530+0000 7f781ab86b80 10 client.27133 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:50.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.530+0000 7f781ab86b80 20 client.27133 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:50.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.530+0000 7f781ab86b80 1 --2- 172.21.15.114:0/3511162430 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5628753616d0 0x562875363af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.530+0000 7f781ab86b80 4 client.27133 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:48:50.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.530+0000 7f781ab86b80 1 -- 172.21.15.114:0/3511162430 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562875277cc0 con 0x5628753616d0 2024-05-27T07:48:50.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.530+0000 7f7809f0d700 1 --2- 172.21.15.114:0/3511162430 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5628753616d0 0x562875363af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.530+0000 7f77ed7fa700 10 client.27133 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.531+0000 7f7809f0d700 1 --2- 172.21.15.114:0/3511162430 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5628753616d0 0x562875363af0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=795d10bb34788d6f server_cookie=89b68154b2f1f8ba in_seq=0 out_seq=0 2024-05-27T07:48:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f77ed7fa700 1 -- 172.21.15.114:0/3511162430 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562875277cc0 con 0x5628753616d0 2024-05-27T07:48:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f77ed7fa700 10 client.27133 handle_command_reply: tid=0 2024-05-27T07:48:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f781ab86b80 1 client.27133 shutdown 2024-05-27T07:48:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f781ab86b80 1 -- 172.21.15.114:0/3511162430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562875348120 msgr2=0x562875351e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f781ab86b80 1 --2- 172.21.15.114:0/3511162430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562875348120 0x562875351e70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f77f40e11c0 tx=0x7f77f400a7f0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f781ab86b80 1 -- 172.21.15.114:0/3511162430 shutdown_connections 2024-05-27T07:48:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f781ab86b80 1 --2- 172.21.15.114:0/3511162430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56287534e070 0x5628753523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f781ab86b80 1 --2- 172.21.15.114:0/3511162430 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5628753616d0 0x562875363af0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f781ab86b80 1 --2- 172.21.15.114:0/3511162430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x562875351930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f77ed7fa700 0 client.27133 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f781ab86b80 1 --2- 172.21.15.114:0/3511162430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562875348120 0x562875351e70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f781ab86b80 1 -- 172.21.15.114:0/3511162430 >> 172.21.15.114:0/3511162430 conn(0x5628751cb770 msgr2=0x562875346bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f781ab86b80 1 -- 172.21.15.114:0/3511162430 shutdown_connections 2024-05-27T07:48:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f781ab86b80 1 -- 172.21.15.114:0/3511162430 wait complete. 2024-05-27T07:48:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.532+0000 7f781ab86b80 20 client.27133 trim_cache size 0 max 16384 2024-05-27T07:48:50.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.551+0000 7f781ab86b80 1 Processor -- start 2024-05-27T07:48:50.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.551+0000 7f781ab86b80 1 -- start start 2024-05-27T07:48:50.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f781ab86b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5628751c8e40 0x562875342e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f781ab86b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x562875350220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f781ab86b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56287534e070 0x562875350760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f781ab86b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78041b06b0 con 0x562875348120 2024-05-27T07:48:50.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f781ab86b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f78041b0d80 con 0x56287534e070 2024-05-27T07:48:50.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f781ab86b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f78041ae500 con 0x5628751c8e40 2024-05-27T07:48:50.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f780970c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x562875350220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f780970c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x562875350220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39558/0 (socket says 172.21.15.114:39558) 2024-05-27T07:48:50.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f780970c700 1 -- 172.21.15.114:0/2479062056 learned_addr learned my addr 172.21.15.114:0/2479062056 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:50.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f7809f0d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5628751c8e40 0x562875342e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f780a70e700 1 --2- 172.21.15.114:0/2479062056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56287534e070 0x562875350760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f7809f0d700 1 -- 172.21.15.114:0/2479062056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56287534e070 msgr2=0x562875350760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:50.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f7809f0d700 1 --2- 172.21.15.114:0/2479062056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56287534e070 0x562875350760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f7809f0d700 1 -- 172.21.15.114:0/2479062056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 msgr2=0x562875350220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:50.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f7809f0d700 1 --2- 172.21.15.114:0/2479062056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x562875350220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f7809f0d700 1 -- 172.21.15.114:0/2479062056 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562875351700 con 0x5628751c8e40 2024-05-27T07:48:50.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f780970c700 1 --2- 172.21.15.114:0/2479062056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x562875350220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:50.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.552+0000 7f7809f0d700 1 --2- 172.21.15.114:0/2479062056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5628751c8e40 0x562875342e30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x562875265910 tx=0x7f7800014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57fdbbe3c67f5cc3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:50.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.553+0000 7f77ef7fe700 1 -- 172.21.15.114:0/2479062056 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7800021ad0 con 0x5628751c8e40 2024-05-27T07:48:50.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.553+0000 7f77ef7fe700 1 -- 172.21.15.114:0/2479062056 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78000044e0 con 0x5628751c8e40 2024-05-27T07:48:50.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.553+0000 7f77ef7fe700 1 -- 172.21.15.114:0/2479062056 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780001f7d0 con 0x5628751c8e40 2024-05-27T07:48:50.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.554+0000 7f781ab86b80 1 -- 172.21.15.114:0/2479062056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5628751c8e40 msgr2=0x562875342e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:50.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.554+0000 7f781ab86b80 1 --2- 172.21.15.114:0/2479062056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5628751c8e40 0x562875342e30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x562875265910 tx=0x7f7800014000 comp rx=0 tx=0).stop 2024-05-27T07:48:50.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.554+0000 7f781ab86b80 1 -- 172.21.15.114:0/2479062056 shutdown_connections 2024-05-27T07:48:50.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.554+0000 7f781ab86b80 1 --2- 172.21.15.114:0/2479062056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56287534e070 0x562875350760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.554+0000 7f781ab86b80 1 --2- 172.21.15.114:0/2479062056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5628751c8e40 0x562875342e30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.554+0000 7f781ab86b80 1 --2- 172.21.15.114:0/2479062056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x562875350220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.554+0000 7f781ab86b80 1 -- 172.21.15.114:0/2479062056 >> 172.21.15.114:0/2479062056 conn(0x5628751cb770 msgr2=0x562875342720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:50.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.554+0000 7f781ab86b80 1 -- 172.21.15.114:0/2479062056 shutdown_connections 2024-05-27T07:48:50.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.554+0000 7f781ab86b80 1 -- 172.21.15.114:0/2479062056 wait complete. 2024-05-27T07:48:50.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.554+0000 7f781ab86b80 1 -- start start 2024-05-27T07:48:50.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.554+0000 7f781ab86b80 1 Processor -- start 2024-05-27T07:48:50.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f781ab86b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x562875359b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f781ab86b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x56287535a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f781ab86b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56287534e070 0x5628753c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f781ab86b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78041aeb70 con 0x562875348120 2024-05-27T07:48:50.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f781ab86b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f78041affe0 con 0x5628751c8e40 2024-05-27T07:48:50.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f781ab86b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f78041af240 con 0x56287534e070 2024-05-27T07:48:50.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f781ab86b80 10 client.0 _ll_register_callbacks cb 0x5628751fc5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:50.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f7809f0d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x562875359b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f780a70e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56287534e070 0x5628753c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f7809f0d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x562875359b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:58014/0 (socket says 172.21.15.114:58014) 2024-05-27T07:48:50.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f7809f0d700 1 -- 172.21.15.114:0/2872882764 learned_addr learned my addr 172.21.15.114:0/2872882764 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:50.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f7809f0d700 1 -- 172.21.15.114:0/2872882764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56287534e070 msgr2=0x5628753c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:50.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f7809f0d700 1 --2- 172.21.15.114:0/2872882764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56287534e070 0x5628753c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.555+0000 7f7809f0d700 1 -- 172.21.15.114:0/2872882764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 msgr2=0x56287535a0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:48:50.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.556+0000 7f7809f0d700 1 --2- 172.21.15.114:0/2872882764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x56287535a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.556+0000 7f7809f0d700 1 -- 172.21.15.114:0/2872882764 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56287535e700 con 0x5628751c8e40 2024-05-27T07:48:50.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.556+0000 7f7809f0d700 1 --2- 172.21.15.114:0/2872882764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x562875359b70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f780001f680 tx=0x7f7800009fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72e4df1154502da8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:50.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.556+0000 7f77ed7fa700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:48:50.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.556+0000 7f77ed7fa700 1 -- 172.21.15.114:0/2872882764 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7800021ce0 con 0x5628751c8e40 2024-05-27T07:48:50.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.556+0000 7f77ed7fa700 1 -- 172.21.15.114:0/2872882764 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f78000210d0 con 0x5628751c8e40 2024-05-27T07:48:50.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.556+0000 7f781ab86b80 1 -- 172.21.15.114:0/2872882764 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78000044e0 con 0x5628751c8e40 2024-05-27T07:48:50.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.557+0000 7f77ed7fa700 1 -- 172.21.15.114:0/2872882764 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f78000212d0 con 0x5628751c8e40 2024-05-27T07:48:50.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.557+0000 7f781ab86b80 10 client.27014 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.557+0000 7f781ab86b80 1 -- 172.21.15.114:0/2872882764 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x56287534ebd0 con 0x5628751c8e40 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.557+0000 7f77ed7fa700 1 -- 172.21.15.114:0/2872882764 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78000044e0 con 0x5628751c8e40 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.557+0000 7f77ed7fa700 1 -- 172.21.15.114:0/2872882764 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7f780001f7d0 con 0x5628751c8e40 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.557+0000 7f781ab86b80 10 client.27014 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.557+0000 7f781ab86b80 10 client.27014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.557+0000 7f781ab86b80 20 client.27014 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.557+0000 7f781ab86b80 1 --2- 172.21.15.114:0/2872882764 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5628753d04d0 0x5628753d28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.557+0000 7f781ab86b80 4 client.27014 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.557+0000 7f781ab86b80 1 -- 172.21.15.114:0/2872882764 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628752714a0 con 0x5628753d04d0 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.557+0000 7f780970c700 1 --2- 172.21.15.114:0/2872882764 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5628753d04d0 0x5628753d28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f77ed7fa700 10 client.27014 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f780970c700 1 --2- 172.21.15.114:0/2872882764 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5628753d04d0 0x5628753d28f0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=682d9713717e593e server_cookie=f8914176ae80a281 in_seq=0 out_seq=0 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f77ed7fa700 1 -- 172.21.15.114:0/2872882764 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628752714a0 con 0x5628753d04d0 2024-05-27T07:48:50.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f77ed7fa700 10 client.27014 handle_command_reply: tid=0 2024-05-27T07:48:50.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f781ab86b80 1 client.27014 shutdown 2024-05-27T07:48:50.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f781ab86b80 1 -- 172.21.15.114:0/2872882764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 msgr2=0x562875359b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:50.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f781ab86b80 1 --2- 172.21.15.114:0/2872882764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x562875359b70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f780001f680 tx=0x7f7800009fd0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f781ab86b80 1 -- 172.21.15.114:0/2872882764 shutdown_connections 2024-05-27T07:48:50.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f781ab86b80 1 --2- 172.21.15.114:0/2872882764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56287534e070 0x5628753c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f781ab86b80 1 --2- 172.21.15.114:0/2872882764 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5628753d04d0 0x5628753d28f0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f781ab86b80 1 --2- 172.21.15.114:0/2872882764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628751c8e40 0x562875359b70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f781ab86b80 1 --2- 172.21.15.114:0/2872882764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562875348120 0x56287535a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f77ed7fa700 0 client.27014 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:50.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f781ab86b80 1 -- 172.21.15.114:0/2872882764 >> 172.21.15.114:0/2872882764 conn(0x5628751cb770 msgr2=0x562875350220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:50.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.558+0000 7f781ab86b80 1 -- 172.21.15.114:0/2872882764 shutdown_connections 2024-05-27T07:48:50.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.559+0000 7f781ab86b80 1 -- 172.21.15.114:0/2872882764 wait complete. 2024-05-27T07:48:50.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.559+0000 7f781ab86b80 20 client.27014 trim_cache size 0 max 16384 2024-05-27T07:48:50.563 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:48:50.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.559+0000 7f780c171700 1 -- 172.21.15.114:0/3001706108 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f77f00b2a70 msgr2=0x7f77f00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:50.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.559+0000 7f780c171700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f77f00b2a70 0x7f77f00b4f80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7804103af0 tx=0x7f7800009260 comp rx=0 tx=0).stop 2024-05-27T07:48:50.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.559+0000 7f780c171700 1 -- 172.21.15.114:0/3001706108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7804103300 msgr2=0x7f780419b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:50.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.559+0000 7f780c171700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7804103300 0x7f780419b420 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f77f4000c00 tx=0x7f77f4003930 comp rx=0 tx=0).stop 2024-05-27T07:48:50.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.559+0000 7f780c171700 1 -- 172.21.15.114:0/3001706108 shutdown_connections 2024-05-27T07:48:50.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.559+0000 7f780c171700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f77f00b2a70 0x7f77f00b4f80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.559+0000 7f780c171700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78041089e0 0x7f780419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.559+0000 7f780c171700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78041029b0 0x7f780419ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.559+0000 7f780c171700 1 --2- 172.21.15.114:0/3001706108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7804103300 0x7f780419b420 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:50.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.559+0000 7f780c171700 1 -- 172.21.15.114:0/3001706108 >> 172.21.15.114:0/3001706108 conn(0x7f78040fe6e0 msgr2=0x7f7804101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:50.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.560+0000 7f780c171700 1 -- 172.21.15.114:0/3001706108 shutdown_connections 2024-05-27T07:48:50.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:50.560+0000 7f780c171700 1 -- 172.21.15.114:0/3001706108 wait complete. 2024-05-27T07:48:51.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 926220 11.07 MB/sec execute 303 sec latency 17.601 ms 2024-05-27T07:48:51.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:51 smithi163 ceph-mon[36451]: pgmap v295: 129 pgs: 129 active+clean; 77 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 318 op/s 2024-05-27T07:48:51.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:51 smithi186 ceph-mon[36647]: pgmap v295: 129 pgs: 129 active+clean; 77 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 318 op/s 2024-05-27T07:48:51.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:51 smithi114 ceph-mon[32038]: pgmap v295: 129 pgs: 129 active+clean; 77 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 318 op/s 2024-05-27T07:48:52.275 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:48:52.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:48:52.276 INFO:tasks.workunit.client.0.smithi114.stdout: 1 928239 11.08 MB/sec execute 304 sec latency 85.698 ms 2024-05-27T07:48:52.465 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.464+0000 7f6b66514640 1 -- 172.21.15.114:0/1903200851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b60104b10 msgr2=0x7f6b60101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:52.465 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.464+0000 7f6b66514640 1 --2- 172.21.15.114:0/1903200851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b60104b10 0x7f6b60101d40 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f6b50009900 tx=0x7f6b5002a480 comp rx=0 tx=0).stop 2024-05-27T07:48:52.465 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.465+0000 7f6b66514640 1 -- 172.21.15.114:0/1903200851 shutdown_connections 2024-05-27T07:48:52.465 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.465+0000 7f6b66514640 1 --2- 172.21.15.114:0/1903200851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b601104c0 0x7f6b601128e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.465 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.465+0000 7f6b66514640 1 --2- 172.21.15.114:0/1903200851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b60102280 0x7f6b60102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.466 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.465+0000 7f6b66514640 1 --2- 172.21.15.114:0/1903200851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b60104b10 0x7f6b60101d40 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.466 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.465+0000 7f6b66514640 1 -- 172.21.15.114:0/1903200851 >> 172.21.15.114:0/1903200851 conn(0x7f6b600779d0 msgr2=0x7f6b60077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:52.466 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.465+0000 7f6b66514640 1 -- 172.21.15.114:0/1903200851 shutdown_connections 2024-05-27T07:48:52.466 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.465+0000 7f6b66514640 1 -- 172.21.15.114:0/1903200851 wait complete. 2024-05-27T07:48:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.465+0000 7f6b66514640 1 Processor -- start 2024-05-27T07:48:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.465+0000 7f6b66514640 1 -- start start 2024-05-27T07:48:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b66514640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b60102280 0x7f6b601a2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b66514640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b60104b10 0x7f6b601a2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b66514640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b601104c0 0x7f6b6019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b66514640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6b60116030 con 0x7f6b60104b10 2024-05-27T07:48:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b66514640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6b60115eb0 con 0x7f6b601104c0 2024-05-27T07:48:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b66514640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601161b0 con 0x7f6b60102280 2024-05-27T07:48:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b57fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b60104b10 0x7f6b601a2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b5ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b60102280 0x7f6b601a2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b57fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b60104b10 0x7f6b601a2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39576/0 (socket says 172.21.15.114:39576) 2024-05-27T07:48:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b64ab1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b601104c0 0x7f6b6019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b57fff640 1 -- 172.21.15.114:0/3795290564 learned_addr learned my addr 172.21.15.114:0/3795290564 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b57fff640 1 -- 172.21.15.114:0/3795290564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b60102280 msgr2=0x7f6b601a2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b57fff640 1 --2- 172.21.15.114:0/3795290564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b60102280 0x7f6b601a2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b57fff640 1 -- 172.21.15.114:0/3795290564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b601104c0 msgr2=0x7f6b6019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b57fff640 1 --2- 172.21.15.114:0/3795290564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b601104c0 0x7f6b6019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b57fff640 1 -- 172.21.15.114:0/3795290564 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6019cf30 con 0x7f6b60104b10 2024-05-27T07:48:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.466+0000 7f6b5ffff640 1 --2- 172.21.15.114:0/3795290564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b60102280 0x7f6b601a2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.467+0000 7f6b57fff640 1 --2- 172.21.15.114:0/3795290564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b60104b10 0x7f6b601a2bc0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800e9f0 tx=0x7f6b4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.467+0000 7f6b5dffb640 1 -- 172.21.15.114:0/3795290564 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4800cd80 con 0x7f6b60104b10 2024-05-27T07:48:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.467+0000 7f6b66514640 1 -- 172.21.15.114:0/3795290564 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b6019d220 con 0x7f6b60104b10 2024-05-27T07:48:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.467+0000 7f6b5dffb640 1 -- 172.21.15.114:0/3795290564 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b480054d0 con 0x7f6b60104b10 2024-05-27T07:48:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.467+0000 7f6b5dffb640 1 -- 172.21.15.114:0/3795290564 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b48010690 con 0x7f6b60104b10 2024-05-27T07:48:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.467+0000 7f6b66514640 1 -- 172.21.15.114:0/3795290564 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b601a9500 con 0x7f6b60104b10 2024-05-27T07:48:52.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b6d3ca740 1 Processor -- start 2024-05-27T07:48:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b6d3ca740 1 -- start start 2024-05-27T07:48:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b5dffb640 1 -- 172.21.15.114:0/3795290564 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6b48010830 con 0x7f6b60104b10 2024-05-27T07:48:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b6d3ca740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2e61b0 0x558c4e6873d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b6d3ca740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e687910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b6d3ca740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2c54d0 0x558c4e687e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b6d3ca740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558c4e5f9c30 con 0x558c4e2bd2f0 2024-05-27T07:48:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b6d3ca740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558c4e5e97d0 con 0x558c4e2e61b0 2024-05-27T07:48:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b6d3ca740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558c4e5ed810 con 0x558c4e2c54d0 2024-05-27T07:48:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b5ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e687910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b5ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e687910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39578/0 (socket says 172.21.15.114:39578) 2024-05-27T07:48:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b5ffff640 1 -- 172.21.15.114:0/1296245638 learned_addr learned my addr 172.21.15.114:0/1296245638 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b64ab1640 1 --2- 172.21.15.114:0/1296245638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2e61b0 0x558c4e6873d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.470+0000 7f6b57fff640 1 --2- 172.21.15.114:0/1296245638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2c54d0 0x558c4e687e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.471+0000 7f6b5ffff640 1 -- 172.21.15.114:0/1296245638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2c54d0 msgr2=0x558c4e687e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.471+0000 7f6b5ffff640 1 --2- 172.21.15.114:0/1296245638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2c54d0 0x558c4e687e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.471+0000 7f6b5ffff640 1 -- 172.21.15.114:0/1296245638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2e61b0 msgr2=0x558c4e6873d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.471+0000 7f6b5ffff640 1 --2- 172.21.15.114:0/1296245638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2e61b0 0x558c4e6873d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.471+0000 7f6b5ffff640 1 -- 172.21.15.114:0/1296245638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c4e2fe210 con 0x558c4e2bd2f0 2024-05-27T07:48:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.471+0000 7f6b57fff640 1 --2- 172.21.15.114:0/1296245638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2c54d0 0x558c4e687e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.471+0000 7f6b5ffff640 1 --2- 172.21.15.114:0/1296245638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e687910 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f6b5002a9a0 tx=0x7f6b50005ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf3114dd6214c3d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.471+0000 7f6b5dffb640 1 --2- 172.21.15.114:0/3795290564 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6b4c0b2950 0x7f6b4c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.471+0000 7f6b64ab1640 1 --2- 172.21.15.114:0/3795290564 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6b4c0b2950 0x7f6b4c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.471+0000 7f6b567fc640 1 -- 172.21.15.114:0/1296245638 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50040070 con 0x558c4e2bd2f0 2024-05-27T07:48:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.471+0000 7f6b567fc640 1 -- 172.21.15.114:0/1296245638 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b50031820 con 0x558c4e2bd2f0 2024-05-27T07:48:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.471+0000 7f6b567fc640 1 -- 172.21.15.114:0/1296245638 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50035040 con 0x558c4e2bd2f0 2024-05-27T07:48:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.471+0000 7f6b5dffb640 1 -- 172.21.15.114:0/3795290564 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f6b480e2e10 con 0x7f6b60104b10 2024-05-27T07:48:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.472+0000 7f6b64ab1640 1 --2- 172.21.15.114:0/3795290564 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6b4c0b2950 0x7f6b4c0b4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x558c4e684540 tx=0x7f6b580096a0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.472+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1296245638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 msgr2=0x558c4e687910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.472+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1296245638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e687910 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f6b5002a9a0 tx=0x7f6b50005ed0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.472+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1296245638 shutdown_connections 2024-05-27T07:48:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.472+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1296245638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2c54d0 0x558c4e687e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.472+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1296245638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e687910 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.472+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1296245638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2e61b0 0x558c4e6873d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.472+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1296245638 >> 172.21.15.114:0/1296245638 conn(0x558c4e2da300 msgr2=0x558c4e683b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.473+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1296245638 shutdown_connections 2024-05-27T07:48:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.473+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1296245638 wait complete. 2024-05-27T07:48:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.473+0000 7f6b6d3ca740 1 -- start start 2024-05-27T07:48:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.473+0000 7f6b6d3ca740 1 Processor -- start 2024-05-27T07:48:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.473+0000 7f6b6d3ca740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e691ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.473+0000 7f6b6d3ca740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2c54d0 0x558c4e692030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.473+0000 7f6b6d3ca740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2e61b0 0x558c4e697810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b6d3ca740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558c4e2cf790 con 0x558c4e2bd2f0 2024-05-27T07:48:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b6d3ca740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558c4e5a5890 con 0x558c4e2e61b0 2024-05-27T07:48:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b6d3ca740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558c4e6944a0 con 0x558c4e2c54d0 2024-05-27T07:48:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b6d3ca740 10 client.0 _ll_register_callbacks cb 0x558c4e2fda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b57fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2c54d0 0x558c4e692030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b5ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e691ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b57fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2c54d0 0x558c4e692030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47984/0 (socket says 172.21.15.114:47984) 2024-05-27T07:48:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b64ab1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2e61b0 0x558c4e697810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b57fff640 1 -- 172.21.15.114:0/227833922 learned_addr learned my addr 172.21.15.114:0/227833922 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b64ab1640 1 -- 172.21.15.114:0/227833922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2c54d0 msgr2=0x558c4e692030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b64ab1640 1 --2- 172.21.15.114:0/227833922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2c54d0 0x558c4e692030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b64ab1640 1 -- 172.21.15.114:0/227833922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 msgr2=0x558c4e691ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b64ab1640 1 --2- 172.21.15.114:0/227833922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e691ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b64ab1640 1 -- 172.21.15.114:0/227833922 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c4e3ca620 con 0x558c4e2e61b0 2024-05-27T07:48:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b64ab1640 1 --2- 172.21.15.114:0/227833922 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2e61b0 0x558c4e697810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009e40 tx=0x7f6b58004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8390d18526757 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.474+0000 7f6b3f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:48:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b3f7fe640 1 -- 172.21.15.114:0/227833922 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b58003c40 con 0x558c4e2e61b0 2024-05-27T07:48:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/227833922 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b50031820 con 0x558c4e2e61b0 2024-05-27T07:48:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b3f7fe640 1 -- 172.21.15.114:0/227833922 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f6b58017030 con 0x558c4e2e61b0 2024-05-27T07:48:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b3f7fe640 1 -- 172.21.15.114:0/227833922 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f6b58003de0 con 0x558c4e2e61b0 2024-05-27T07:48:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b6d3ca740 10 client.27026 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/227833922 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x558c4e6a0890 con 0x558c4e2e61b0 2024-05-27T07:48:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b3f7fe640 1 -- 172.21.15.114:0/227833922 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b50031820 con 0x558c4e2e61b0 2024-05-27T07:48:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b3f7fe640 1 -- 172.21.15.114:0/227833922 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7f6b58004030 con 0x558c4e2e61b0 2024-05-27T07:48:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b6d3ca740 10 client.27026 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b6d3ca740 10 client.27026 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b6d3ca740 20 client.27026 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/227833922 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558c4e2a4d80 0x558c4e6a41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b6d3ca740 4 client.27026 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:48:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/227833922 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c4e5fbb80 con 0x558c4e2a4d80 2024-05-27T07:48:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.475+0000 7f6b5ffff640 1 --2- 172.21.15.114:0/227833922 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558c4e2a4d80 0x558c4e6a41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.476+0000 7f6b3f7fe640 10 client.27026 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.476+0000 7f6b5ffff640 1 --2- 172.21.15.114:0/227833922 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558c4e2a4d80 0x558c4e6a41c0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb404e9a88d2956c server_cookie=46f148f8fa17d258 in_seq=0 out_seq=0 2024-05-27T07:48:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b3f7fe640 1 -- 172.21.15.114:0/227833922 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558c4e5fbb80 con 0x558c4e2a4d80 2024-05-27T07:48:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b3f7fe640 10 client.27026 handle_command_reply: tid=0 2024-05-27T07:48:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b6d3ca740 1 client.27026 shutdown 2024-05-27T07:48:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/227833922 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2e61b0 msgr2=0x558c4e697810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/227833922 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2e61b0 0x558c4e697810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009e40 tx=0x7f6b58004ab0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/227833922 shutdown_connections 2024-05-27T07:48:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/227833922 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558c4e2a4d80 0x558c4e6a41c0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/227833922 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2e61b0 0x558c4e697810 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/227833922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2c54d0 0x558c4e692030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/227833922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e691ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b3f7fe640 0 client.27026 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:52.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/227833922 >> 172.21.15.114:0/227833922 conn(0x558c4e2da300 msgr2=0x558c4e687200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:52.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/227833922 shutdown_connections 2024-05-27T07:48:52.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/227833922 wait complete. 2024-05-27T07:48:52.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.477+0000 7f6b6d3ca740 20 client.27026 trim_cache size 0 max 16384 2024-05-27T07:48:52.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.497+0000 7f6b6d3ca740 1 Processor -- start 2024-05-27T07:48:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.497+0000 7f6b6d3ca740 1 -- start start 2024-05-27T07:48:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.497+0000 7f6b6d3ca740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2da300 0x558c4e68b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.497+0000 7f6b6d3ca740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2a4d80 0x558c4e69e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.497+0000 7f6b6d3ca740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2bd2f0 0x558c4e6a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.497+0000 7f6b6d3ca740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601b3680 con 0x558c4e2da300 2024-05-27T07:48:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.497+0000 7f6b6d3ca740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601b3d50 con 0x558c4e2bd2f0 2024-05-27T07:48:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.497+0000 7f6b6d3ca740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601b14d0 con 0x558c4e2a4d80 2024-05-27T07:48:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.497+0000 7f6b5ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2da300 0x558c4e68b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.497+0000 7f6b5ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2da300 0x558c4e68b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39596/0 (socket says 172.21.15.114:39596) 2024-05-27T07:48:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.497+0000 7f6b5ffff640 1 -- 172.21.15.114:0/1135068439 learned_addr learned my addr 172.21.15.114:0/1135068439 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.497+0000 7f6b64ab1640 1 --2- 172.21.15.114:0/1135068439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2bd2f0 0x558c4e6a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.497+0000 7f6b57fff640 1 --2- 172.21.15.114:0/1135068439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2a4d80 0x558c4e69e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.498+0000 7f6b5ffff640 1 -- 172.21.15.114:0/1135068439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2a4d80 msgr2=0x558c4e69e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.498+0000 7f6b5ffff640 1 --2- 172.21.15.114:0/1135068439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2a4d80 0x558c4e69e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.498+0000 7f6b5ffff640 1 -- 172.21.15.114:0/1135068439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2bd2f0 msgr2=0x558c4e6a99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.498+0000 7f6b5ffff640 1 --2- 172.21.15.114:0/1135068439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2bd2f0 0x558c4e6a99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.498+0000 7f6b5ffff640 1 -- 172.21.15.114:0/1135068439 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c4e6923a0 con 0x558c4e2da300 2024-05-27T07:48:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.498+0000 7f6b64ab1640 1 --2- 172.21.15.114:0/1135068439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2bd2f0 0x558c4e6a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.498+0000 7f6b5ffff640 1 --2- 172.21.15.114:0/1135068439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2da300 0x558c4e68b7a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f6b50033000 tx=0x7f6b500054f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e77d15d5edd76f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.498+0000 7f6b55ffb640 1 -- 172.21.15.114:0/1135068439 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50035040 con 0x558c4e2da300 2024-05-27T07:48:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.498+0000 7f6b55ffb640 1 -- 172.21.15.114:0/1135068439 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b500035f0 con 0x558c4e2da300 2024-05-27T07:48:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.498+0000 7f6b55ffb640 1 -- 172.21.15.114:0/1135068439 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50040040 con 0x558c4e2da300 2024-05-27T07:48:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.499+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1135068439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2da300 msgr2=0x558c4e68b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.499+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1135068439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2da300 0x558c4e68b7a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f6b50033000 tx=0x7f6b500054f0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.499+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1135068439 shutdown_connections 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.499+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1135068439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2bd2f0 0x558c4e6a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.499+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1135068439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2a4d80 0x558c4e69e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.499+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1135068439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2da300 0x558c4e68b7a0 secure :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f6b50033000 tx=0x7f6b500054f0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.499+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1135068439 >> 172.21.15.114:0/1135068439 conn(0x558c4e68d240 msgr2=0x558c4e68b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.499+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1135068439 shutdown_connections 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.499+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1135068439 wait complete. 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.500+0000 7f6b6d3ca740 1 -- start start 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.500+0000 7f6b6d3ca740 1 Processor -- start 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.500+0000 7f6b6d3ca740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2a4d80 0x558c4e69adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.500+0000 7f6b6d3ca740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e69b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.500+0000 7f6b6d3ca740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2c54d0 0x558c4e70a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b6d3ca740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601b1b40 con 0x558c4e2bd2f0 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b6d3ca740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601b2fb0 con 0x558c4e2c54d0 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b6d3ca740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601b2210 con 0x558c4e2a4d80 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b5ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2a4d80 0x558c4e69adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b5ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2a4d80 0x558c4e69adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:48010/0 (socket says 172.21.15.114:48010) 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b5ffff640 1 -- 172.21.15.114:0/1591584383 learned_addr learned my addr 172.21.15.114:0/1591584383 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b57fff640 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e69b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b64ab1640 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2c54d0 0x558c4e70a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b6d3ca740 10 client.0 _ll_register_callbacks cb 0x7f6b58009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b64ab1640 1 -- 172.21.15.114:0/1591584383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2a4d80 msgr2=0x558c4e69adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b64ab1640 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2a4d80 0x558c4e69adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b64ab1640 1 -- 172.21.15.114:0/1591584383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 msgr2=0x558c4e69b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b64ab1640 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e69b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b64ab1640 1 -- 172.21.15.114:0/1591584383 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c4e69bb50 con 0x558c4e2c54d0 2024-05-27T07:48:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b5ffff640 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2a4d80 0x558c4e69adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.501+0000 7f6b57fff640 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e69b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.502+0000 7f6b64ab1640 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2c54d0 0x558c4e70a890 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6b58011e20 tx=0x7f6b58004230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64aa9e79686b697a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.502+0000 7f6b3f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:48:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.502+0000 7f6b3f7fe640 1 -- 172.21.15.114:0/1591584383 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5801f670 con 0x558c4e2c54d0 2024-05-27T07:48:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.502+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1591584383 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b500035f0 con 0x558c4e2c54d0 2024-05-27T07:48:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.502+0000 7f6b3f7fe640 1 -- 172.21.15.114:0/1591584383 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f6b5801f810 con 0x558c4e2c54d0 2024-05-27T07:48:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.502+0000 7f6b3f7fe640 1 -- 172.21.15.114:0/1591584383 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f6b58042050 con 0x558c4e2c54d0 2024-05-27T07:48:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.502+0000 7f6b6d3ca740 10 client.27032 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.502+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1591584383 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x558c4e70bdd0 con 0x558c4e2c54d0 2024-05-27T07:48:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.502+0000 7f6b3f7fe640 1 -- 172.21.15.114:0/1591584383 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b500035f0 con 0x558c4e2c54d0 2024-05-27T07:48:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.503+0000 7f6b3f7fe640 1 -- 172.21.15.114:0/1591584383 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7f6b5803d030 con 0x558c4e2c54d0 2024-05-27T07:48:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.503+0000 7f6b6d3ca740 10 client.27032 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.503+0000 7f6b6d3ca740 10 client.27032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.503+0000 7f6b6d3ca740 20 client.27032 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.503+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558c4e2e61b0 0x558c4e70e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.503+0000 7f6b6d3ca740 4 client.27032 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:48:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.503+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1591584383 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558c4e5f5360 con 0x558c4e2e61b0 2024-05-27T07:48:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.503+0000 7f6b5ffff640 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558c4e2e61b0 0x558c4e70e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.503+0000 7f6b3f7fe640 10 client.27032 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:52.504+0000 7f6b5ffff640 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558c4e2e61b0 0x558c4e70e4f0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28b27ccf9976ef86 server_cookie=c097443c13a7c362 in_seq=0 out_seq=0 2024-05-27T07:48:52.561 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:48:53.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.022+0000 7f6b3f7fe640 1 -- 172.21.15.114:0/1591584383 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x558c4e5f5360 con 0x558c4e2e61b0 2024-05-27T07:48:53.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.022+0000 7f6b3f7fe640 10 client.27032 handle_command_reply: tid=0 2024-05-27T07:48:53.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.022+0000 7f6b6d3ca740 1 client.27032 shutdown 2024-05-27T07:48:53.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.023+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1591584383 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2c54d0 msgr2=0x558c4e70a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:53.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.023+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2c54d0 0x558c4e70a890 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6b58011e20 tx=0x7f6b58004230 comp rx=0 tx=0).stop 2024-05-27T07:48:53.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.023+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1591584383 shutdown_connections 2024-05-27T07:48:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.023+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558c4e2e61b0 0x558c4e70e4f0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.023+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558c4e2c54d0 0x558c4e70a890 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.023+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c4e2bd2f0 0x558c4e69b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.023+0000 7f6b6d3ca740 1 --2- 172.21.15.114:0/1591584383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c4e2a4d80 0x558c4e69adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.023+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1591584383 >> 172.21.15.114:0/1591584383 conn(0x558c4e68d240 msgr2=0x558c4e68fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.023+0000 7f6b3f7fe640 0 client.27032 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.023+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1591584383 shutdown_connections 2024-05-27T07:48:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.023+0000 7f6b6d3ca740 1 -- 172.21.15.114:0/1591584383 wait complete. 2024-05-27T07:48:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.023+0000 7f6b6d3ca740 20 client.27032 trim_cache size 0 max 16384 2024-05-27T07:48:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:48:53.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:48:53.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:48:53.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.024+0000 7f6b66514640 1 -- 172.21.15.114:0/3795290564 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6b4c0b2950 msgr2=0x7f6b4c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:53.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.024+0000 7f6b66514640 1 --2- 172.21.15.114:0/3795290564 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6b4c0b2950 0x7f6b4c0b4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x558c4e684540 tx=0x7f6b580096a0 comp rx=0 tx=0).stop 2024-05-27T07:48:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.024+0000 7f6b66514640 1 -- 172.21.15.114:0/3795290564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b60104b10 msgr2=0x7f6b601a2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.024+0000 7f6b66514640 1 --2- 172.21.15.114:0/3795290564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b60104b10 0x7f6b601a2bc0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800e9f0 tx=0x7f6b4800edf0 comp rx=0 tx=0).stop 2024-05-27T07:48:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.024+0000 7f6b66514640 1 -- 172.21.15.114:0/3795290564 shutdown_connections 2024-05-27T07:48:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.024+0000 7f6b66514640 1 --2- 172.21.15.114:0/3795290564 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6b4c0b2950 0x7f6b4c0b4e60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.024+0000 7f6b66514640 1 --2- 172.21.15.114:0/3795290564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b601104c0 0x7f6b6019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.024+0000 7f6b66514640 1 --2- 172.21.15.114:0/3795290564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b60104b10 0x7f6b601a2bc0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.024+0000 7f6b66514640 1 --2- 172.21.15.114:0/3795290564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b60102280 0x7f6b601a2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.024+0000 7f6b66514640 1 -- 172.21.15.114:0/3795290564 >> 172.21.15.114:0/3795290564 conn(0x7f6b600779d0 msgr2=0x7f6b60100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.024+0000 7f6b66514640 1 -- 172.21.15.114:0/3795290564 shutdown_connections 2024-05-27T07:48:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:48:53.025+0000 7f6b66514640 1 -- 172.21.15.114:0/3795290564 wait complete. 2024-05-27T07:48:53.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 929481 11.06 MB/sec execute 305 sec latency 380.064 ms 2024-05-27T07:48:53.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:53 smithi163 ceph-mon[36451]: pgmap v296: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.2 MiB/s wr, 297 op/s 2024-05-27T07:48:53.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:53 smithi186 ceph-mon[36647]: pgmap v296: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.2 MiB/s wr, 297 op/s 2024-05-27T07:48:53.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:53 smithi114 ceph-mon[32038]: pgmap v296: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.2 MiB/s wr, 297 op/s 2024-05-27T07:48:54.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 931119 11.07 MB/sec execute 306 sec latency 59.251 ms 2024-05-27T07:48:54.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.568+0000 7fc48c588700 1 -- 172.21.15.114:0/3253461516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc484108f00 msgr2=0x7fc484109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:54.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.568+0000 7fc48c588700 1 --2- 172.21.15.114:0/3253461516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc484108f00 0x7fc484109310 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc4800099c0 tx=0x7fc480009cd0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.568+0000 7fc48c588700 1 -- 172.21.15.114:0/3253461516 shutdown_connections 2024-05-27T07:48:54.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.568+0000 7fc48c588700 1 --2- 172.21.15.114:0/3253461516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc484109850 0x7fc4841111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.568+0000 7fc48c588700 1 --2- 172.21.15.114:0/3253461516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4841045d0 0x7fc4841049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.568+0000 7fc48c588700 1 --2- 172.21.15.114:0/3253461516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc484108f00 0x7fc484109310 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.568+0000 7fc48c588700 1 -- 172.21.15.114:0/3253461516 >> 172.21.15.114:0/3253461516 conn(0x7fc4840ffec0 msgr2=0x7fc484102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:54.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.568+0000 7fc48c588700 1 -- 172.21.15.114:0/3253461516 shutdown_connections 2024-05-27T07:48:54.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.568+0000 7fc48c588700 1 -- 172.21.15.114:0/3253461516 wait complete. 2024-05-27T07:48:54.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.569+0000 7fc48c588700 1 Processor -- start 2024-05-27T07:48:54.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.569+0000 7fc48c588700 1 -- start start 2024-05-27T07:48:54.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.569+0000 7fc48c588700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4841045d0 0x7fc48419ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48c588700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc484108f00 0x7fc48419b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48c588700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc484109850 0x7fc48419f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48c588700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc484114460 con 0x7fc484108f00 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48c588700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841142e0 con 0x7fc484109850 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48c588700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841145e0 con 0x7fc4841045d0 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc489b23700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc484108f00 0x7fc48419b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48a324700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4841045d0 0x7fc48419ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48a324700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4841045d0 0x7fc48419ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55360/0 (socket says 172.21.15.114:55360) 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc489b23700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc484108f00 0x7fc48419b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58016/0 (socket says 172.21.15.114:58016) 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48a324700 1 -- 172.21.15.114:0/1004693230 learned_addr learned my addr 172.21.15.114:0/1004693230 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48ab25700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc484109850 0x7fc48419f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48a324700 1 -- 172.21.15.114:0/1004693230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc484109850 msgr2=0x7fc48419f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48a324700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc484109850 0x7fc48419f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48a324700 1 -- 172.21.15.114:0/1004693230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc484108f00 msgr2=0x7fc48419b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48a324700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc484108f00 0x7fc48419b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48a324700 1 -- 172.21.15.114:0/1004693230 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc48419fd00 con 0x7fc4841045d0 2024-05-27T07:48:54.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc489b23700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc484108f00 0x7fc48419b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:54.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48a324700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4841045d0 0x7fc48419ae30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc47800e9a0 tx=0x7fc47800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:54.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc4777fe700 1 -- 172.21.15.114:0/1004693230 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47800ca10 con 0x7fc4841045d0 2024-05-27T07:48:54.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc4777fe700 1 -- 172.21.15.114:0/1004693230 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc47800cbb0 con 0x7fc4841045d0 2024-05-27T07:48:54.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc48c588700 1 -- 172.21.15.114:0/1004693230 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc48419fff0 con 0x7fc4841045d0 2024-05-27T07:48:54.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.570+0000 7fc4777fe700 1 -- 172.21.15.114:0/1004693230 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc478018680 con 0x7fc4841045d0 2024-05-27T07:48:54.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.571+0000 7fc48c588700 1 -- 172.21.15.114:0/1004693230 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4841a7910 con 0x7fc4841045d0 2024-05-27T07:48:54.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.574+0000 7fc4777fe700 1 -- 172.21.15.114:0/1004693230 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fc478018820 con 0x7fc4841045d0 2024-05-27T07:48:54.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.575+0000 7fc4777fe700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc4700b29f0 0x7fc4700b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.575+0000 7fc49af9db80 1 Processor -- start 2024-05-27T07:48:54.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.575+0000 7fc489b23700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc4700b29f0 0x7fc4700b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.575+0000 7fc4777fe700 1 -- 172.21.15.114:0/1004693230 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7fc4780e2bf0 con 0x7fc4841045d0 2024-05-27T07:48:54.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.575+0000 7fc49af9db80 1 -- start start 2024-05-27T07:48:54.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.575+0000 7fc49af9db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537ab5ec0 0x55b537c39790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.575+0000 7fc49af9db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537c39cd0 0x55b537c33ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.575+0000 7fc49af9db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c34000 0x55b537c34410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc49af9db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b537b62d10 con 0x55b537ab5ec0 2024-05-27T07:48:54.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc49af9db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b537b528b0 con 0x55b537c39cd0 2024-05-27T07:48:54.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc49af9db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b537b568f0 con 0x55b537c34000 2024-05-27T07:48:54.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc489b23700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc4700b29f0 0x7fc4700b4f00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc480000c00 tx=0x7fc48001a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:54.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc48a324700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537ab5ec0 0x55b537c39790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc48a324700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537ab5ec0 0x55b537c39790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58026/0 (socket says 172.21.15.114:58026) 2024-05-27T07:48:54.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc48a324700 1 -- 172.21.15.114:0/987340708 learned_addr learned my addr 172.21.15.114:0/987340708 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:54.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc48ab25700 1 --2- 172.21.15.114:0/987340708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c34000 0x55b537c34410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc489b23700 1 --2- 172.21.15.114:0/987340708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537c39cd0 0x55b537c33ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc48a324700 1 -- 172.21.15.114:0/987340708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c34000 msgr2=0x55b537c34410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:54.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc48a324700 1 --2- 172.21.15.114:0/987340708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c34000 0x55b537c34410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc48a324700 1 -- 172.21.15.114:0/987340708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537c39cd0 msgr2=0x55b537c33ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:54.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc48a324700 1 --2- 172.21.15.114:0/987340708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537c39cd0 0x55b537c33ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc48a324700 1 -- 172.21.15.114:0/987340708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b537aa2950 con 0x55b537ab5ec0 2024-05-27T07:48:54.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc48a324700 1 --2- 172.21.15.114:0/987340708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537ab5ec0 0x55b537c39790 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc4780a9820 tx=0x7fc4780a5b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3a518854948e116 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:54.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc474ff9700 1 -- 172.21.15.114:0/987340708 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4780a9b20 con 0x55b537ab5ec0 2024-05-27T07:48:54.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc474ff9700 1 -- 172.21.15.114:0/987340708 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4780aae50 con 0x55b537ab5ec0 2024-05-27T07:48:54.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.576+0000 7fc474ff9700 1 -- 172.21.15.114:0/987340708 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4780b24f0 con 0x55b537ab5ec0 2024-05-27T07:48:54.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.577+0000 7fc49af9db80 1 -- 172.21.15.114:0/987340708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537ab5ec0 msgr2=0x55b537c39790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:54.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.577+0000 7fc49af9db80 1 --2- 172.21.15.114:0/987340708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537ab5ec0 0x55b537c39790 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc4780a9820 tx=0x7fc4780a5b10 comp rx=0 tx=0).stop 2024-05-27T07:48:54.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.577+0000 7fc49af9db80 1 -- 172.21.15.114:0/987340708 shutdown_connections 2024-05-27T07:48:54.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.577+0000 7fc49af9db80 1 --2- 172.21.15.114:0/987340708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c34000 0x55b537c34410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.577+0000 7fc49af9db80 1 --2- 172.21.15.114:0/987340708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537ab5ec0 0x55b537c39790 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.577+0000 7fc49af9db80 1 --2- 172.21.15.114:0/987340708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537c39cd0 0x55b537c33ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.577+0000 7fc49af9db80 1 -- 172.21.15.114:0/987340708 >> 172.21.15.114:0/987340708 conn(0x55b537b10b70 msgr2=0x55b537c2e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:54.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.577+0000 7fc49af9db80 1 -- 172.21.15.114:0/987340708 shutdown_connections 2024-05-27T07:48:54.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.577+0000 7fc49af9db80 1 -- 172.21.15.114:0/987340708 wait complete. 2024-05-27T07:48:54.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc49af9db80 1 -- start start 2024-05-27T07:48:54.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc49af9db80 1 Processor -- start 2024-05-27T07:48:54.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc49af9db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537ab5ec0 0x55b537c3d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc49af9db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c34000 0x55b537c3d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc49af9db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c39cd0 0x55b537c3df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc49af9db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b537a59020 con 0x55b537c34000 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc49af9db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b537af4840 con 0x55b537ab5ec0 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc49af9db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b537c40760 con 0x55b537c39cd0 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc49af9db80 10 client.0 _ll_register_callbacks cb 0x55b537aed9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc489b23700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c34000 0x55b537c3d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc48a324700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537ab5ec0 0x55b537c3d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc489b23700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c34000 0x55b537c3d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58028/0 (socket says 172.21.15.114:58028) 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc489b23700 1 -- 172.21.15.114:0/1063809648 learned_addr learned my addr 172.21.15.114:0/1063809648 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.578+0000 7fc48ab25700 1 --2- 172.21.15.114:0/1063809648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c39cd0 0x55b537c3df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc48a324700 1 -- 172.21.15.114:0/1063809648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c39cd0 msgr2=0x55b537c3df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc48a324700 1 --2- 172.21.15.114:0/1063809648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c39cd0 0x55b537c3df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc48a324700 1 -- 172.21.15.114:0/1063809648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c34000 msgr2=0x55b537c3d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc48a324700 1 --2- 172.21.15.114:0/1063809648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c34000 0x55b537c3d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc48a324700 1 -- 172.21.15.114:0/1063809648 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b537af1710 con 0x55b537ab5ec0 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc48a324700 1 --2- 172.21.15.114:0/1063809648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537ab5ec0 0x55b537c3d480 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc4780a94e0 tx=0x7fc478009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c0bbba4dc6b742e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc46dffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc46dffb700 1 -- 172.21.15.114:0/1063809648 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4780aded0 con 0x55b537ab5ec0 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc49af9db80 1 -- 172.21.15.114:0/1063809648 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4780aae50 con 0x55b537ab5ec0 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc46dffb700 1 -- 172.21.15.114:0/1063809648 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7fc4780b24f0 con 0x55b537ab5ec0 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc46dffb700 1 -- 172.21.15.114:0/1063809648 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fc4780a9b20 con 0x55b537ab5ec0 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc49af9db80 10 client.27044 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc49af9db80 1 -- 172.21.15.114:0/1063809648 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55b537c4b280 con 0x55b537ab5ec0 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc46dffb700 1 -- 172.21.15.114:0/1063809648 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4780aae50 con 0x55b537ab5ec0 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.579+0000 7fc46dffb700 1 -- 172.21.15.114:0/1063809648 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7fc4780cc0d0 con 0x55b537ab5ec0 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.580+0000 7fc49af9db80 10 client.27044 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.580+0000 7fc49af9db80 10 client.27044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:54.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.580+0000 7fc49af9db80 20 client.27044 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:54.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.580+0000 7fc49af9db80 1 --2- 172.21.15.114:0/1063809648 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537c4d750 0x55b537c4fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.580+0000 7fc49af9db80 4 client.27044 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:48:54.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.580+0000 7fc49af9db80 1 -- 172.21.15.114:0/1063809648 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b537bffa70 con 0x55b537c4d750 2024-05-27T07:48:54.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.580+0000 7fc489b23700 1 --2- 172.21.15.114:0/1063809648 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537c4d750 0x55b537c4fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.580+0000 7fc46dffb700 10 client.27044 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:54.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.580+0000 7fc489b23700 1 --2- 172.21.15.114:0/1063809648 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537c4d750 0x55b537c4fb70 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ee2e6a9a9a01ce3 server_cookie=1e11569f1057091a in_seq=0 out_seq=0 2024-05-27T07:48:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc46dffb700 1 -- 172.21.15.114:0/1063809648 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b537bffa70 con 0x55b537c4d750 2024-05-27T07:48:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc46dffb700 10 client.27044 handle_command_reply: tid=0 2024-05-27T07:48:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc49af9db80 1 client.27044 shutdown 2024-05-27T07:48:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc49af9db80 1 -- 172.21.15.114:0/1063809648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537ab5ec0 msgr2=0x55b537c3d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc49af9db80 1 --2- 172.21.15.114:0/1063809648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537ab5ec0 0x55b537c3d480 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc4780a94e0 tx=0x7fc478009790 comp rx=0 tx=0).stop 2024-05-27T07:48:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc49af9db80 1 -- 172.21.15.114:0/1063809648 shutdown_connections 2024-05-27T07:48:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc49af9db80 1 --2- 172.21.15.114:0/1063809648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c39cd0 0x55b537c3df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc49af9db80 1 --2- 172.21.15.114:0/1063809648 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537c4d750 0x55b537c4fb70 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc49af9db80 1 --2- 172.21.15.114:0/1063809648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537ab5ec0 0x55b537c3d480 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc49af9db80 1 --2- 172.21.15.114:0/1063809648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c34000 0x55b537c3d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc46dffb700 0 client.27044 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc49af9db80 1 -- 172.21.15.114:0/1063809648 >> 172.21.15.114:0/1063809648 conn(0x55b537b10b70 msgr2=0x55b537c32ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc49af9db80 1 -- 172.21.15.114:0/1063809648 shutdown_connections 2024-05-27T07:48:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc49af9db80 1 -- 172.21.15.114:0/1063809648 wait complete. 2024-05-27T07:48:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.583+0000 7fc49af9db80 20 client.27044 trim_cache size 0 max 16384 2024-05-27T07:48:54.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.605+0000 7fc49af9db80 1 Processor -- start 2024-05-27T07:48:54.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc49af9db80 1 -- start start 2024-05-27T07:48:54.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc49af9db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537ab5ec0 0x55b537c33ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc49af9db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c34000 0x55b537c37ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc49af9db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537c39cd0 0x55b537c3db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc49af9db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841ae480 con 0x55b537c34000 2024-05-27T07:48:54.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc49af9db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841aeb50 con 0x55b537c39cd0 2024-05-27T07:48:54.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc49af9db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841ac2f0 con 0x55b537ab5ec0 2024-05-27T07:48:54.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc489b23700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c34000 0x55b537c37ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc489b23700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c34000 0x55b537c37ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58044/0 (socket says 172.21.15.114:58044) 2024-05-27T07:48:54.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc489b23700 1 -- 172.21.15.114:0/1115760230 learned_addr learned my addr 172.21.15.114:0/1115760230 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:54.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc48a324700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537ab5ec0 0x55b537c33ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc48ab25700 1 --2- 172.21.15.114:0/1115760230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537c39cd0 0x55b537c3db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc489b23700 1 -- 172.21.15.114:0/1115760230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537ab5ec0 msgr2=0x55b537c33ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:54.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc489b23700 1 --2- 172.21.15.114:0/1115760230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537ab5ec0 0x55b537c33ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc489b23700 1 -- 172.21.15.114:0/1115760230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537c39cd0 msgr2=0x55b537c3db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:54.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc489b23700 1 --2- 172.21.15.114:0/1115760230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537c39cd0 0x55b537c3db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.606+0000 7fc489b23700 1 -- 172.21.15.114:0/1115760230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b537c29490 con 0x55b537c34000 2024-05-27T07:48:54.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.607+0000 7fc489b23700 1 --2- 172.21.15.114:0/1115760230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c34000 0x55b537c37ac0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc480009fd0 tx=0x7fc48001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f48b6e940a8ffb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:54.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.607+0000 7fc46ffff700 1 -- 172.21.15.114:0/1115760230 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4800037c0 con 0x55b537c34000 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.607+0000 7fc46ffff700 1 -- 172.21.15.114:0/1115760230 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc480003e10 con 0x55b537c34000 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.607+0000 7fc46ffff700 1 -- 172.21.15.114:0/1115760230 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48000f460 con 0x55b537c34000 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.608+0000 7fc49af9db80 1 -- 172.21.15.114:0/1115760230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c34000 msgr2=0x55b537c37ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.608+0000 7fc49af9db80 1 --2- 172.21.15.114:0/1115760230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c34000 0x55b537c37ac0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc480009fd0 tx=0x7fc48001f000 comp rx=0 tx=0).stop 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.608+0000 7fc49af9db80 1 -- 172.21.15.114:0/1115760230 shutdown_connections 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.608+0000 7fc49af9db80 1 --2- 172.21.15.114:0/1115760230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537c39cd0 0x55b537c3db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.608+0000 7fc49af9db80 1 --2- 172.21.15.114:0/1115760230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537ab5ec0 0x55b537c33ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.608+0000 7fc49af9db80 1 --2- 172.21.15.114:0/1115760230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c34000 0x55b537c37ac0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.608+0000 7fc49af9db80 1 -- 172.21.15.114:0/1115760230 >> 172.21.15.114:0/1115760230 conn(0x55b537b10b70 msgr2=0x55b537c3d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.608+0000 7fc49af9db80 1 -- 172.21.15.114:0/1115760230 shutdown_connections 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.608+0000 7fc49af9db80 1 -- 172.21.15.114:0/1115760230 wait complete. 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.608+0000 7fc49af9db80 1 -- start start 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.608+0000 7fc49af9db80 1 Processor -- start 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc49af9db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537ab5ec0 0x55b537c51a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc49af9db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c34000 0x55b537c51f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc49af9db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c39cd0 0x55b537c524a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc49af9db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841ac940 con 0x55b537c39cd0 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc49af9db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841addb0 con 0x55b537ab5ec0 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc49af9db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841ad010 con 0x55b537c34000 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc49af9db80 10 client.0 _ll_register_callbacks cb 0x55b537aed9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc48a324700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537ab5ec0 0x55b537c51a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc48a324700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537ab5ec0 0x55b537c51a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51688/0 (socket says 172.21.15.114:51688) 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc48a324700 1 -- 172.21.15.114:0/3275988793 learned_addr learned my addr 172.21.15.114:0/3275988793 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc489b23700 1 --2- 172.21.15.114:0/3275988793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c34000 0x55b537c51f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc48a324700 1 -- 172.21.15.114:0/3275988793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c34000 msgr2=0x55b537c51f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc48a324700 1 --2- 172.21.15.114:0/3275988793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c34000 0x55b537c51f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc48a324700 1 -- 172.21.15.114:0/3275988793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c39cd0 msgr2=0x55b537c524a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc48a324700 1 --2- 172.21.15.114:0/3275988793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c39cd0 0x55b537c524a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.609+0000 7fc48a324700 1 -- 172.21.15.114:0/3275988793 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b537cc0810 con 0x55b537ab5ec0 2024-05-27T07:48:54.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.610+0000 7fc48a324700 1 --2- 172.21.15.114:0/3275988793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537ab5ec0 0x55b537c51a20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc4780c4380 tx=0x7fc4780a9470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dda432e1178f9b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:54.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.610+0000 7fc46dffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:48:54.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.610+0000 7fc46dffb700 1 -- 172.21.15.114:0/3275988793 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4780a59e0 con 0x55b537ab5ec0 2024-05-27T07:48:54.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.610+0000 7fc49af9db80 1 -- 172.21.15.114:0/3275988793 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc480003e10 con 0x55b537ab5ec0 2024-05-27T07:48:54.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.610+0000 7fc46dffb700 1 -- 172.21.15.114:0/3275988793 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7fc4780b2700 con 0x55b537ab5ec0 2024-05-27T07:48:54.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.610+0000 7fc46dffb700 1 -- 172.21.15.114:0/3275988793 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fc4780aded0 con 0x55b537ab5ec0 2024-05-27T07:48:54.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.610+0000 7fc49af9db80 10 client.27050 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:54.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.610+0000 7fc49af9db80 1 -- 172.21.15.114:0/3275988793 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55b537cc1ab0 con 0x55b537ab5ec0 2024-05-27T07:48:54.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.610+0000 7fc46dffb700 1 -- 172.21.15.114:0/3275988793 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc480003e10 con 0x55b537ab5ec0 2024-05-27T07:48:54.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.611+0000 7fc46dffb700 1 -- 172.21.15.114:0/3275988793 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7fc4780a55b0 con 0x55b537ab5ec0 2024-05-27T07:48:54.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.611+0000 7fc49af9db80 10 client.27050 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:54.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.611+0000 7fc49af9db80 10 client.27050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.611+0000 7fc49af9db80 20 client.27050 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.611+0000 7fc49af9db80 1 --2- 172.21.15.114:0/3275988793 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537cc1e70 0x55b537cc4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.611+0000 7fc49af9db80 4 client.27050 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.611+0000 7fc49af9db80 1 -- 172.21.15.114:0/3275988793 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b537b64c60 con 0x55b537cc1e70 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.611+0000 7fc489b23700 1 --2- 172.21.15.114:0/3275988793 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537cc1e70 0x55b537cc4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.611+0000 7fc46dffb700 10 client.27050 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.611+0000 7fc489b23700 1 --2- 172.21.15.114:0/3275988793 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537cc1e70 0x55b537cc4320 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb30753267fd854c server_cookie=e5fe13f4ffabe1df in_seq=0 out_seq=0 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.612+0000 7fc46dffb700 1 -- 172.21.15.114:0/3275988793 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b537b64c60 con 0x55b537cc1e70 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.612+0000 7fc46dffb700 10 client.27050 handle_command_reply: tid=0 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.612+0000 7fc49af9db80 1 client.27050 shutdown 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.613+0000 7fc49af9db80 1 -- 172.21.15.114:0/3275988793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537ab5ec0 msgr2=0x55b537c51a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.613+0000 7fc49af9db80 1 --2- 172.21.15.114:0/3275988793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537ab5ec0 0x55b537c51a20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc4780c4380 tx=0x7fc4780a9470 comp rx=0 tx=0).stop 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.613+0000 7fc49af9db80 1 -- 172.21.15.114:0/3275988793 shutdown_connections 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.613+0000 7fc49af9db80 1 --2- 172.21.15.114:0/3275988793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537c39cd0 0x55b537c524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.613+0000 7fc49af9db80 1 --2- 172.21.15.114:0/3275988793 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537cc1e70 0x55b537cc4320 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.613+0000 7fc49af9db80 1 --2- 172.21.15.114:0/3275988793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537ab5ec0 0x55b537c51a20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.613+0000 7fc49af9db80 1 --2- 172.21.15.114:0/3275988793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537c34000 0x55b537c51f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.613+0000 7fc49af9db80 1 -- 172.21.15.114:0/3275988793 >> 172.21.15.114:0/3275988793 conn(0x55b537b10b70 msgr2=0x55b537c37820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:54.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.613+0000 7fc46dffb700 0 client.27050 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:54.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.613+0000 7fc49af9db80 1 -- 172.21.15.114:0/3275988793 shutdown_connections 2024-05-27T07:48:54.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.613+0000 7fc49af9db80 1 -- 172.21.15.114:0/3275988793 wait complete. 2024-05-27T07:48:54.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.613+0000 7fc49af9db80 20 client.27050 trim_cache size 0 max 16384 2024-05-27T07:48:54.620 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:48:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:48:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "d1f40f44-eedf-427c-ac97-8d563fc8aa05", 2024-05-27T07:48:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:48:54.621 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:48:54.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.614+0000 7fc48c588700 1 -- 172.21.15.114:0/1004693230 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc4700b29f0 msgr2=0x7fc4700b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:54.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.614+0000 7fc48c588700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc4700b29f0 0x7fc4700b4f00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc480000c00 tx=0x7fc48001a040 comp rx=0 tx=0).stop 2024-05-27T07:48:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.614+0000 7fc48c588700 1 -- 172.21.15.114:0/1004693230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4841045d0 msgr2=0x7fc48419ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.614+0000 7fc48c588700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4841045d0 0x7fc48419ae30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc47800e9a0 tx=0x7fc47800ecb0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.614+0000 7fc48c588700 1 -- 172.21.15.114:0/1004693230 shutdown_connections 2024-05-27T07:48:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.614+0000 7fc48c588700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc4700b29f0 0x7fc4700b4f00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.614+0000 7fc48c588700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc484109850 0x7fc48419f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.614+0000 7fc48c588700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4841045d0 0x7fc48419ae30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.614+0000 7fc48c588700 1 --2- 172.21.15.114:0/1004693230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc484108f00 0x7fc48419b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.614+0000 7fc48c588700 1 -- 172.21.15.114:0/1004693230 >> 172.21.15.114:0/1004693230 conn(0x7fc4840ffec0 msgr2=0x7fc48410a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.614+0000 7fc48c588700 1 -- 172.21.15.114:0/1004693230 shutdown_connections 2024-05-27T07:48:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:54.614+0000 7fc48c588700 1 -- 172.21.15.114:0/1004693230 wait complete. 2024-05-27T07:48:55.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 933024 11.05 MB/sec execute 307 sec latency 16.707 ms 2024-05-27T07:48:55.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:48:55.710 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:48:55.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:55 smithi163 ceph-mon[36451]: pgmap v297: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 200 op/s 2024-05-27T07:48:55.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:55 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:48:55.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:55 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:48:55.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:55 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:48:55.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:55 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:48:55.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:55 smithi186 ceph-mon[36647]: pgmap v297: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 200 op/s 2024-05-27T07:48:55.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:55 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:48:55.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:55 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:48:55.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:55 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:48:55.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:55 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:48:55.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:55 smithi114 ceph-mon[32038]: pgmap v297: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 200 op/s 2024-05-27T07:48:55.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:55 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:48:55.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:55 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:48:55.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:55 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:48:55.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:55 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:48:56.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 935306 11.06 MB/sec execute 308 sec latency 15.888 ms 2024-05-27T07:48:57.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 938011 11.08 MB/sec execute 309 sec latency 16.879 ms 2024-05-27T07:48:57.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:57 smithi163 ceph-mon[36451]: pgmap v298: 129 pgs: 129 active+clean; 67 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 494 op/s 2024-05-27T07:48:57.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:57 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:48:57.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:57 smithi186 ceph-mon[36647]: pgmap v298: 129 pgs: 129 active+clean; 67 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 494 op/s 2024-05-27T07:48:57.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:57 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:48:57.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:57 smithi114 ceph-mon[32038]: pgmap v298: 129 pgs: 129 active+clean; 67 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 494 op/s 2024-05-27T07:48:57.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:57 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:48:58.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.774+0000 7fca772c8700 1 -- 172.21.15.114:0/1413253252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca70108730 msgr2=0x7fca70110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:58.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.774+0000 7fca772c8700 1 --2- 172.21.15.114:0/1413253252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca70108730 0x7fca70110090 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fca6800b0d0 tx=0x7fca6800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.774+0000 7fca772c8700 1 -- 172.21.15.114:0/1413253252 shutdown_connections 2024-05-27T07:48:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.774+0000 7fca772c8700 1 --2- 172.21.15.114:0/1413253252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca70108730 0x7fca70110090 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.774+0000 7fca772c8700 1 --2- 172.21.15.114:0/1413253252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca701034b0 0x7fca701038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.774+0000 7fca772c8700 1 --2- 172.21.15.114:0/1413253252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca70107de0 0x7fca701081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.774+0000 7fca772c8700 1 -- 172.21.15.114:0/1413253252 >> 172.21.15.114:0/1413253252 conn(0x7fca700fedb0 msgr2=0x7fca701011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.774+0000 7fca772c8700 1 -- 172.21.15.114:0/1413253252 shutdown_connections 2024-05-27T07:48:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.774+0000 7fca772c8700 1 -- 172.21.15.114:0/1413253252 wait complete. 2024-05-27T07:48:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.775+0000 7fca772c8700 1 Processor -- start 2024-05-27T07:48:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.775+0000 7fca772c8700 1 -- start start 2024-05-27T07:48:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.775+0000 7fca772c8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca701034b0 0x7fca7019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.775+0000 7fca772c8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca70107de0 0x7fca7019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.775+0000 7fca772c8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca70108730 0x7fca7019f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.775+0000 7fca772c8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fca70113360 con 0x7fca701034b0 2024-05-27T07:48:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.775+0000 7fca772c8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fca701131e0 con 0x7fca70107de0 2024-05-27T07:48:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.775+0000 7fca772c8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fca701134e0 con 0x7fca70108730 2024-05-27T07:48:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca75064700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca701034b0 0x7fca7019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca75064700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca701034b0 0x7fca7019adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58052/0 (socket says 172.21.15.114:58052) 2024-05-27T07:48:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca75064700 1 -- 172.21.15.114:0/2763825542 learned_addr learned my addr 172.21.15.114:0/2763825542 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca75865700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca70108730 0x7fca7019f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca74863700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca70107de0 0x7fca7019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca75064700 1 -- 172.21.15.114:0/2763825542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca70108730 msgr2=0x7fca7019f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca75064700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca70108730 0x7fca7019f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca75064700 1 -- 172.21.15.114:0/2763825542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca70107de0 msgr2=0x7fca7019b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca75064700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca70107de0 0x7fca7019b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca75064700 1 -- 172.21.15.114:0/2763825542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca7019fc90 con 0x7fca701034b0 2024-05-27T07:48:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca75865700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca70108730 0x7fca7019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca75064700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca701034b0 0x7fca7019adc0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00e9a0 tx=0x7fca6c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca667fc700 1 -- 172.21.15.114:0/2763825542 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c00ca10 con 0x7fca701034b0 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca772c8700 1 -- 172.21.15.114:0/2763825542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca7019ff80 con 0x7fca701034b0 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca667fc700 1 -- 172.21.15.114:0/2763825542 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca6c00cbb0 con 0x7fca701034b0 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca667fc700 1 -- 172.21.15.114:0/2763825542 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c018680 con 0x7fca701034b0 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.776+0000 7fca772c8700 1 -- 172.21.15.114:0/2763825542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca701a78a0 con 0x7fca701034b0 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.779+0000 7fca667fc700 1 -- 172.21.15.114:0/2763825542 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fca6c010430 con 0x7fca701034b0 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.780+0000 7fca667fc700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fca5c0b29f0 0x7fca5c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.780+0000 7fca74863700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fca5c0b29f0 0x7fca5c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.780+0000 7fca667fc700 1 -- 172.21.15.114:0/2763825542 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7fca6c0e2b20 con 0x7fca701034b0 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.780+0000 7fca85cddb80 1 Processor -- start 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.780+0000 7fca85cddb80 1 -- start start 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca74863700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fca5c0b29f0 0x7fca5c0b4f00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fca701097b0 tx=0x7fca600073d0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca85cddb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c44de40 0x55ac5c5d1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca85cddb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca85cddb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5cc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca85cddb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ac5c4fac90 con 0x55ac5c5cbf80 2024-05-27T07:48:58.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca85cddb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ac5c4ea830 con 0x55ac5c44de40 2024-05-27T07:48:58.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca85cddb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ac5c4ee870 con 0x55ac5c5d1c50 2024-05-27T07:48:58.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca75865700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca75865700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58062/0 (socket says 172.21.15.114:58062) 2024-05-27T07:48:58.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca75865700 1 -- 172.21.15.114:0/3555713261 learned_addr learned my addr 172.21.15.114:0/3555713261 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:58.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca74863700 1 --2- 172.21.15.114:0/3555713261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca75064700 1 --2- 172.21.15.114:0/3555713261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c44de40 0x55ac5c5d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca75865700 1 -- 172.21.15.114:0/3555713261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5d1c50 msgr2=0x55ac5c5cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca75865700 1 --2- 172.21.15.114:0/3555713261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca75865700 1 -- 172.21.15.114:0/3555713261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c44de40 msgr2=0x55ac5c5d1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca75865700 1 --2- 172.21.15.114:0/3555713261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c44de40 0x55ac5c5d1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca75865700 1 -- 172.21.15.114:0/3555713261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac5c43a8d0 con 0x55ac5c5cbf80 2024-05-27T07:48:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca74863700 1 --2- 172.21.15.114:0/3555713261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:48:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca75064700 1 --2- 172.21.15.114:0/3555713261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c44de40 0x55ac5c5d1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:48:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca75865700 1 --2- 172.21.15.114:0/3555713261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5cc390 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fca68005ba0 tx=0x7fca68009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d389e25eb4c8f34 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.781+0000 7fca5b7fe700 1 -- 172.21.15.114:0/3555713261 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6800be50 con 0x55ac5c5cbf80 2024-05-27T07:48:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.782+0000 7fca5b7fe700 1 -- 172.21.15.114:0/3555713261 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca680094e0 con 0x55ac5c5cbf80 2024-05-27T07:48:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.782+0000 7fca5b7fe700 1 -- 172.21.15.114:0/3555713261 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca680075c0 con 0x55ac5c5cbf80 2024-05-27T07:48:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.782+0000 7fca85cddb80 1 -- 172.21.15.114:0/3555713261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5cbf80 msgr2=0x55ac5c5cc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.782+0000 7fca85cddb80 1 --2- 172.21.15.114:0/3555713261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5cc390 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fca68005ba0 tx=0x7fca68009b80 comp rx=0 tx=0).stop 2024-05-27T07:48:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.782+0000 7fca85cddb80 1 -- 172.21.15.114:0/3555713261 shutdown_connections 2024-05-27T07:48:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.782+0000 7fca85cddb80 1 --2- 172.21.15.114:0/3555713261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5cc390 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.782+0000 7fca85cddb80 1 --2- 172.21.15.114:0/3555713261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c44de40 0x55ac5c5d1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.782+0000 7fca85cddb80 1 --2- 172.21.15.114:0/3555713261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.782+0000 7fca85cddb80 1 -- 172.21.15.114:0/3555713261 >> 172.21.15.114:0/3555713261 conn(0x55ac5c4a8af0 msgr2=0x55ac5c5c6640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.783+0000 7fca85cddb80 1 -- 172.21.15.114:0/3555713261 shutdown_connections 2024-05-27T07:48:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.783+0000 7fca85cddb80 1 -- 172.21.15.114:0/3555713261 wait complete. 2024-05-27T07:48:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.783+0000 7fca85cddb80 1 -- start start 2024-05-27T07:48:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.783+0000 7fca85cddb80 1 Processor -- start 2024-05-27T07:48:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.783+0000 7fca85cddb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c44de40 0x55ac5c5d5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.783+0000 7fca85cddb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5d5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.783+0000 7fca85cddb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5d5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.783+0000 7fca85cddb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ac5c3f0fa0 con 0x55ac5c5d1c50 2024-05-27T07:48:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.783+0000 7fca85cddb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ac5c48c7c0 con 0x55ac5c5cbf80 2024-05-27T07:48:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.783+0000 7fca85cddb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ac5c5d86e0 con 0x55ac5c44de40 2024-05-27T07:48:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca85cddb80 10 client.0 _ll_register_callbacks cb 0x55ac5c485920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca75865700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5d5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca75064700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c44de40 0x55ac5c5d5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca75865700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5d5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58066/0 (socket says 172.21.15.114:58066) 2024-05-27T07:48:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca75064700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c44de40 0x55ac5c5d5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55426/0 (socket says 172.21.15.114:55426) 2024-05-27T07:48:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca75865700 1 -- 172.21.15.114:0/2927910905 learned_addr learned my addr 172.21.15.114:0/2927910905 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca74863700 1 --2- 172.21.15.114:0/2927910905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5d5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca75865700 1 -- 172.21.15.114:0/2927910905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c44de40 msgr2=0x55ac5c5d5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca75865700 1 --2- 172.21.15.114:0/2927910905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c44de40 0x55ac5c5d5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca75865700 1 -- 172.21.15.114:0/2927910905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5cbf80 msgr2=0x55ac5c5d5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca75865700 1 --2- 172.21.15.114:0/2927910905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5d5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca75865700 1 -- 172.21.15.114:0/2927910905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac5c489690 con 0x55ac5c5d1c50 2024-05-27T07:48:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca75865700 1 --2- 172.21.15.114:0/2927910905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5d5dc0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fca680054c0 tx=0x7fca68005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f442ffcee448ab86 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca58ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:48:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca58ff9700 1 -- 172.21.15.114:0/2927910905 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca68003c00 con 0x55ac5c5d1c50 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca58ff9700 1 -- 172.21.15.114:0/2927910905 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7fca68003da0 con 0x55ac5c5d1c50 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca85cddb80 1 -- 172.21.15.114:0/2927910905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ac5c5979f0 con 0x55ac5c5d1c50 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca58ff9700 1 -- 172.21.15.114:0/2927910905 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fca68032810 con 0x55ac5c5d1c50 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca85cddb80 10 client.17601 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.784+0000 7fca85cddb80 1 -- 172.21.15.114:0/2927910905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55ac5c5e30b0 con 0x55ac5c5d1c50 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.785+0000 7fca58ff9700 1 -- 172.21.15.114:0/2927910905 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ac5c5979f0 con 0x55ac5c5d1c50 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.785+0000 7fca58ff9700 1 -- 172.21.15.114:0/2927910905 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7fca68011070 con 0x55ac5c5d1c50 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.785+0000 7fca85cddb80 10 client.17601 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.785+0000 7fca85cddb80 10 client.17601 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.785+0000 7fca85cddb80 20 client.17601 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.785+0000 7fca85cddb80 1 --2- 172.21.15.114:0/2927910905 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac5c5e5580 0x55ac5c5e79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.785+0000 7fca85cddb80 4 client.17601 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.785+0000 7fca85cddb80 1 -- 172.21.15.114:0/2927910905 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac5c4fcbe0 con 0x55ac5c5e5580 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.785+0000 7fca75064700 1 --2- 172.21.15.114:0/2927910905 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac5c5e5580 0x55ac5c5e79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.786+0000 7fca58ff9700 10 client.17601 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.786+0000 7fca75064700 1 --2- 172.21.15.114:0/2927910905 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac5c5e5580 0x55ac5c5e79a0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed89c39928ed9b04 server_cookie=4fa554251fda99fd in_seq=0 out_seq=0 2024-05-27T07:48:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca58ff9700 1 -- 172.21.15.114:0/2927910905 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ac5c4fcbe0 con 0x55ac5c5e5580 2024-05-27T07:48:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca58ff9700 10 client.17601 handle_command_reply: tid=0 2024-05-27T07:48:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca85cddb80 1 client.17601 shutdown 2024-05-27T07:48:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca85cddb80 1 -- 172.21.15.114:0/2927910905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5d1c50 msgr2=0x55ac5c5d5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca85cddb80 1 --2- 172.21.15.114:0/2927910905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5d5dc0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fca680054c0 tx=0x7fca68005340 comp rx=0 tx=0).stop 2024-05-27T07:48:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca85cddb80 1 -- 172.21.15.114:0/2927910905 shutdown_connections 2024-05-27T07:48:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca85cddb80 1 --2- 172.21.15.114:0/2927910905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5d5dc0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca85cddb80 1 --2- 172.21.15.114:0/2927910905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c44de40 0x55ac5c5d5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca85cddb80 1 --2- 172.21.15.114:0/2927910905 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac5c5e5580 0x55ac5c5e79a0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca85cddb80 1 --2- 172.21.15.114:0/2927910905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5d5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca58ff9700 0 client.17601 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca85cddb80 1 -- 172.21.15.114:0/2927910905 >> 172.21.15.114:0/2927910905 conn(0x55ac5c4a8af0 msgr2=0x55ac5c5caa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca85cddb80 1 -- 172.21.15.114:0/2927910905 shutdown_connections 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca85cddb80 1 -- 172.21.15.114:0/2927910905 wait complete. 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.789+0000 7fca85cddb80 20 client.17601 trim_cache size 0 max 16384 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.810+0000 7fca85cddb80 1 Processor -- start 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.810+0000 7fca85cddb80 1 -- start start 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.810+0000 7fca85cddb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c44de40 0x55ac5c5cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.810+0000 7fca85cddb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5e1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.810+0000 7fca85cddb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5d4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.810+0000 7fca85cddb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fca701b0640 con 0x55ac5c44de40 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.810+0000 7fca85cddb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fca701b0d10 con 0x55ac5c5d1c50 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.810+0000 7fca85cddb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fca701ae490 con 0x55ac5c5cbf80 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.810+0000 7fca75064700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c44de40 0x55ac5c5cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.810+0000 7fca75064700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c44de40 0x55ac5c5cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58068/0 (socket says 172.21.15.114:58068) 2024-05-27T07:48:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.810+0000 7fca75064700 1 -- 172.21.15.114:0/948412689 learned_addr learned my addr 172.21.15.114:0/948412689 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:58.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.810+0000 7fca74863700 1 --2- 172.21.15.114:0/948412689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5e1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.811+0000 7fca75865700 1 --2- 172.21.15.114:0/948412689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5d4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.811+0000 7fca75064700 1 -- 172.21.15.114:0/948412689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5cbf80 msgr2=0x55ac5c5e1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:58.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.811+0000 7fca75064700 1 --2- 172.21.15.114:0/948412689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5e1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.811+0000 7fca75064700 1 -- 172.21.15.114:0/948412689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5d1c50 msgr2=0x55ac5c5d4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:58.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.811+0000 7fca75064700 1 --2- 172.21.15.114:0/948412689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5d4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.811+0000 7fca75064700 1 -- 172.21.15.114:0/948412689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac5c5be970 con 0x55ac5c44de40 2024-05-27T07:48:58.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.811+0000 7fca75865700 1 --2- 172.21.15.114:0/948412689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5d4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:48:58.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.811+0000 7fca75064700 1 --2- 172.21.15.114:0/948412689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c44de40 0x55ac5c5cba40 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0a6cc0 tx=0x7fca6c091040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83efa811c649dc6f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:58.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.811+0000 7fca5affd700 1 -- 172.21.15.114:0/948412689 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0a5040 con 0x55ac5c44de40 2024-05-27T07:48:58.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.811+0000 7fca5affd700 1 -- 172.21.15.114:0/948412689 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca6c0b2040 con 0x55ac5c44de40 2024-05-27T07:48:58.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.811+0000 7fca5affd700 1 -- 172.21.15.114:0/948412689 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0a9f10 con 0x55ac5c44de40 2024-05-27T07:48:58.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.812+0000 7fca85cddb80 1 -- 172.21.15.114:0/948412689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c44de40 msgr2=0x55ac5c5cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.812+0000 7fca85cddb80 1 --2- 172.21.15.114:0/948412689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c44de40 0x55ac5c5cba40 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0a6cc0 tx=0x7fca6c091040 comp rx=0 tx=0).stop 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.812+0000 7fca85cddb80 1 -- 172.21.15.114:0/948412689 shutdown_connections 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.812+0000 7fca85cddb80 1 --2- 172.21.15.114:0/948412689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5d1c50 0x55ac5c5d4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.812+0000 7fca85cddb80 1 --2- 172.21.15.114:0/948412689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c44de40 0x55ac5c5cba40 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.812+0000 7fca85cddb80 1 --2- 172.21.15.114:0/948412689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5cbf80 0x55ac5c5e1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.812+0000 7fca85cddb80 1 -- 172.21.15.114:0/948412689 >> 172.21.15.114:0/948412689 conn(0x55ac5c4a8af0 msgr2=0x55ac5c5c0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.812+0000 7fca85cddb80 1 -- 172.21.15.114:0/948412689 shutdown_connections 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.812+0000 7fca85cddb80 1 -- 172.21.15.114:0/948412689 wait complete. 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.812+0000 7fca85cddb80 1 -- start start 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.812+0000 7fca85cddb80 1 Processor -- start 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca85cddb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5cbf80 0x55ac5c648a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca85cddb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5d1c50 0x55ac5c648fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca85cddb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c6494f0 0x55ac5c5e8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca85cddb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fca701aeb00 con 0x55ac5c6494f0 2024-05-27T07:48:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca85cddb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fca701aff70 con 0x55ac5c5cbf80 2024-05-27T07:48:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca85cddb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fca701af1d0 con 0x55ac5c5d1c50 2024-05-27T07:48:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca85cddb80 10 client.0 _ll_register_callbacks cb 0x55ac5c485920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:48:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca75064700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5cbf80 0x55ac5c648a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca74863700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5d1c50 0x55ac5c648fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca75865700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c6494f0 0x55ac5c5e8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca75064700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5cbf80 0x55ac5c648a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51728/0 (socket says 172.21.15.114:51728) 2024-05-27T07:48:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca75064700 1 -- 172.21.15.114:0/1862358672 learned_addr learned my addr 172.21.15.114:0/1862358672 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:48:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca74863700 1 -- 172.21.15.114:0/1862358672 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5cbf80 msgr2=0x55ac5c648a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca74863700 1 --2- 172.21.15.114:0/1862358672 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5cbf80 0x55ac5c648a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca74863700 1 -- 172.21.15.114:0/1862358672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c6494f0 msgr2=0x55ac5c5e8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:48:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca74863700 1 --2- 172.21.15.114:0/1862358672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c6494f0 0x55ac5c5e8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.813+0000 7fca74863700 1 -- 172.21.15.114:0/1862358672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac5c5eb580 con 0x55ac5c5d1c50 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.814+0000 7fca74863700 1 --2- 172.21.15.114:0/1862358672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5d1c50 0x55ac5c648fb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55ac5c5c7940 tx=0x7fca60013070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=757c8113ac0c4c8a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.814+0000 7fca58ff9700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.814+0000 7fca58ff9700 1 -- 172.21.15.114:0/1862358672 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6002d060 con 0x55ac5c5d1c50 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.814+0000 7fca85cddb80 1 -- 172.21.15.114:0/1862358672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca6c0b2040 con 0x55ac5c5d1c50 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.814+0000 7fca58ff9700 1 -- 172.21.15.114:0/1862358672 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7fca60004230 con 0x55ac5c5d1c50 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.814+0000 7fca58ff9700 1 -- 172.21.15.114:0/1862358672 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fca60034050 con 0x55ac5c5d1c50 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.814+0000 7fca85cddb80 10 client.27193 fetch_fsmap learned FSMap version 50 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.814+0000 7fca85cddb80 1 -- 172.21.15.114:0/1862358672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55ac5c654770 con 0x55ac5c5d1c50 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.814+0000 7fca58ff9700 1 -- 172.21.15.114:0/1862358672 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca6c0b2040 con 0x55ac5c5d1c50 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.814+0000 7fca58ff9700 1 -- 172.21.15.114:0/1862358672 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 50) ==== 7120+0+0 (secure 0 0 0) 0x7fca6001f4e0 con 0x55ac5c5d1c50 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.814+0000 7fca85cddb80 10 client.27193 fetch_fsmap finished waiting for FSMap version 50 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.815+0000 7fca85cddb80 10 client.27193 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.815+0000 7fca85cddb80 20 client.27193 populate_metadata read hostname 'smithi114' 2024-05-27T07:48:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.815+0000 7fca85cddb80 1 --2- 172.21.15.114:0/1862358672 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac5c654dd0 0x55ac5c6571f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:48:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.815+0000 7fca85cddb80 4 client.27193 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:48:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.815+0000 7fca85cddb80 1 -- 172.21.15.114:0/1862358672 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ac5c4f63c0 con 0x55ac5c654dd0 2024-05-27T07:48:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.815+0000 7fca75064700 1 --2- 172.21.15.114:0/1862358672 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac5c654dd0 0x55ac5c6571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:48:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.815+0000 7fca58ff9700 10 client.27193 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.815+0000 7fca75064700 1 --2- 172.21.15.114:0/1862358672 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac5c654dd0 0x55ac5c6571f0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41beafe895241cbe server_cookie=1bbdd21727611f67 in_seq=0 out_seq=0 2024-05-27T07:48:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.816+0000 7fca58ff9700 1 -- 172.21.15.114:0/1862358672 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ac5c4f63c0 con 0x55ac5c654dd0 2024-05-27T07:48:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.816+0000 7fca58ff9700 10 client.27193 handle_command_reply: tid=0 2024-05-27T07:48:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.816+0000 7fca85cddb80 1 client.27193 shutdown 2024-05-27T07:48:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.816+0000 7fca85cddb80 1 -- 172.21.15.114:0/1862358672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5d1c50 msgr2=0x55ac5c648fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.816+0000 7fca85cddb80 1 --2- 172.21.15.114:0/1862358672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5d1c50 0x55ac5c648fb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55ac5c5c7940 tx=0x7fca60013070 comp rx=0 tx=0).stop 2024-05-27T07:48:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.816+0000 7fca85cddb80 1 -- 172.21.15.114:0/1862358672 shutdown_connections 2024-05-27T07:48:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.816+0000 7fca85cddb80 1 --2- 172.21.15.114:0/1862358672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ac5c6494f0 0x55ac5c5e8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.816+0000 7fca85cddb80 1 --2- 172.21.15.114:0/1862358672 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ac5c654dd0 0x55ac5c6571f0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.816+0000 7fca85cddb80 1 --2- 172.21.15.114:0/1862358672 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ac5c5cbf80 0x55ac5c648a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca85cddb80 1 --2- 172.21.15.114:0/1862358672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac5c5d1c50 0x55ac5c648fb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca58ff9700 0 client.27193 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:48:58.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca85cddb80 1 -- 172.21.15.114:0/1862358672 >> 172.21.15.114:0/1862358672 conn(0x55ac5c4a8af0 msgr2=0x55ac5c5d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:58.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca85cddb80 1 -- 172.21.15.114:0/1862358672 shutdown_connections 2024-05-27T07:48:58.109 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:48:58.109 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:48:58.109 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:48:58.110 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:48:58.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca85cddb80 1 -- 172.21.15.114:0/1862358672 wait complete. 2024-05-27T07:48:58.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca85cddb80 20 client.27193 trim_cache size 0 max 16384 2024-05-27T07:48:58.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca772c8700 1 -- 172.21.15.114:0/2763825542 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fca5c0b29f0 msgr2=0x7fca5c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:58.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca772c8700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fca5c0b29f0 0x7fca5c0b4f00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fca701097b0 tx=0x7fca600073d0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca772c8700 1 -- 172.21.15.114:0/2763825542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca701034b0 msgr2=0x7fca7019adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:48:58.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca772c8700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca701034b0 0x7fca7019adc0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00e9a0 tx=0x7fca6c00ecb0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca772c8700 1 -- 172.21.15.114:0/2763825542 shutdown_connections 2024-05-27T07:48:58.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca772c8700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca70108730 0x7fca7019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca772c8700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca701034b0 0x7fca7019adc0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca772c8700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fca5c0b29f0 0x7fca5c0b4f00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca772c8700 1 --2- 172.21.15.114:0/2763825542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca70107de0 0x7fca7019b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:48:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca772c8700 1 -- 172.21.15.114:0/2763825542 >> 172.21.15.114:0/2763825542 conn(0x7fca700fedb0 msgr2=0x7fca7006b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:48:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca772c8700 1 -- 172.21.15.114:0/2763825542 shutdown_connections 2024-05-27T07:48:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:48:57.817+0000 7fca772c8700 1 -- 172.21.15.114:0/2763825542 wait complete. 2024-05-27T07:48:58.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 939415 11.06 MB/sec execute 310 sec latency 39.523 ms 2024-05-27T07:48:58.613 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:58 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:48:58.613 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:58 smithi163 systemd[1]: Stopped Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:48:58.613 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:58 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Consumed 1.872s CPU time. 2024-05-27T07:48:58.613 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:58 smithi163 systemd[1]: Starting Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:48:58.613 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:58 smithi163 podman[71847]: 2024-05-27 07:48:58.354606886 +0000 UTC m=+0.018931302 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:48:58.613 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:58 smithi163 podman[71847]: 2024-05-27 07:48:58.466794035 +0000 UTC m=+0.131118438 container create c27adfcedcb2c15a73ef49f252d4f4d447b9097882812ebb24b2fd352c5c25b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:48:58.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:58 smithi163 podman[71847]: 2024-05-27 07:48:58.745420384 +0000 UTC m=+0.409744808 container init c27adfcedcb2c15a73ef49f252d4f4d447b9097882812ebb24b2fd352c5c25b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-27T07:48:58.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:58 smithi163 podman[71847]: 2024-05-27 07:48:58.749544591 +0000 UTC m=+0.413869009 container start c27adfcedcb2c15a73ef49f252d4f4d447b9097882812ebb24b2fd352c5c25b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T07:48:58.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:58 smithi163 ceph-mds[71867]: -- 172.21.15.163:0/3287439335 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55caae669a40 con 0x55caae6b1180 2024-05-27T07:48:58.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:58 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: starting mds.h at 2024-05-27T07:48:58.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:58 smithi163 bash[71847]: c27adfcedcb2c15a73ef49f252d4f4d447b9097882812ebb24b2fd352c5c25b6 2024-05-27T07:48:58.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:58 smithi163 systemd[1]: Started Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:48:58.983 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:48:58.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:48:59.244 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:48:59.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 941638 11.07 MB/sec execute 311 sec latency 84.845 ms 2024-05-27T07:48:59.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:59 smithi114 ceph-mon[32038]: pgmap v299: 129 pgs: 129 active+clean; 67 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 3.0 MiB/s wr, 394 op/s 2024-05-27T07:48:59.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:59 smithi114 ceph-mon[32038]: daemon mds.h restarted 2024-05-27T07:48:59.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:59 smithi114 ceph-mon[32038]: osdmap e110: 12 total, 12 up, 12 in 2024-05-27T07:48:59.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:59 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6836/1508277928,v1:172.21.15.163:6837/1508277928] up:boot 2024-05-27T07:48:59.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:59 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:48:59.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:48:59 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:48:59.665 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:59 smithi163 ceph-mon[36451]: pgmap v299: 129 pgs: 129 active+clean; 67 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 3.0 MiB/s wr, 394 op/s 2024-05-27T07:48:59.665 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:59 smithi163 ceph-mon[36451]: daemon mds.h restarted 2024-05-27T07:48:59.665 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:59 smithi163 ceph-mon[36451]: osdmap e110: 12 total, 12 up, 12 in 2024-05-27T07:48:59.665 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:59 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6836/1508277928,v1:172.21.15.163:6837/1508277928] up:boot 2024-05-27T07:48:59.666 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:59 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:48:59.666 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:48:59 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:48:59.666 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff77de96700 time 2024-05-27T07:48:59.333644+0000 2024-05-27T07:48:59.667 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:59.667 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:59.667 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff7906dbba6] 2024-05-27T07:48:59.667 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7906dbdd1] 2024-05-27T07:48:59.667 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55caacf3823d] 2024-05-27T07:48:59.667 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55caacf23971] 2024-05-27T07:48:59.667 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 5: (SnapRealm::check_cache() const+0x13e) [0x55caacf21fa0] 2024-05-27T07:48:59.667 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55caacf22651] 2024-05-27T07:48:59.667 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55caaceb7347] 2024-05-27T07:48:59.667 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55caacbb387e] 2024-05-27T07:48:59.667 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55caacbb8f2f] 2024-05-27T07:48:59.668 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55caacbc3969] 2024-05-27T07:48:59.668 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 11: (MDLog::_replay_thread()+0x1f38) [0x55caacf5c48c] 2024-05-27T07:48:59.668 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55caacf659cb] 2024-05-27T07:48:59.668 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 13: (Thread::entry_wrapper()+0x43) [0x7ff7906aa72d] 2024-05-27T07:48:59.668 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff7906aa749] 2024-05-27T07:48:59.668 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff78f24e1ca] 2024-05-27T07:48:59.668 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 16: clone() 2024-05-27T07:48:59.668 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.335+0000 7ff77de96700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff77de96700 time 2024-05-27T07:48:59.333644+0000 2024-05-27T07:48:59.668 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:59.668 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.668 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:59.668 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff7906dbba6] 2024-05-27T07:48:59.668 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7906dbdd1] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55caacf3823d] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55caacf23971] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 5: (SnapRealm::check_cache() const+0x13e) [0x55caacf21fa0] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55caacf22651] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55caaceb7347] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55caacbb387e] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55caacbb8f2f] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55caacbc3969] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 11: (MDLog::_replay_thread()+0x1f38) [0x55caacf5c48c] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55caacf659cb] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 13: (Thread::entry_wrapper()+0x43) [0x7ff7906aa72d] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff7906aa749] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff78f24e1ca] 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 16: clone() 2024-05-27T07:48:59.669 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: *** Caught signal (Aborted) ** 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: in thread 7ff77de96700 thread_name:md_log_replay 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff78f258d20] 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2: gsignal() 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 3: abort() 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff7906dbced] 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7906dbdd1] 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55caacf3823d] 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55caacf23971] 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 8: (SnapRealm::check_cache() const+0x13e) [0x55caacf21fa0] 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55caacf22651] 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55caaceb7347] 2024-05-27T07:48:59.670 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55caacbb387e] 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55caacbb8f2f] 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55caacbc3969] 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 14: (MDLog::_replay_thread()+0x1f38) [0x55caacf5c48c] 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55caacf659cb] 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 16: (Thread::entry_wrapper()+0x43) [0x7ff7906aa72d] 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff7906aa749] 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff78f24e1ca] 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 19: clone() 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.337+0000 7ff77de96700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: in thread 7ff77de96700 thread_name:md_log_replay 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff78f258d20] 2024-05-27T07:48:59.671 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2: gsignal() 2024-05-27T07:48:59.672 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 3: abort() 2024-05-27T07:48:59.672 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff7906dbced] 2024-05-27T07:48:59.672 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7906dbdd1] 2024-05-27T07:48:59.672 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55caacf3823d] 2024-05-27T07:48:59.672 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55caacf23971] 2024-05-27T07:48:59.672 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 8: (SnapRealm::check_cache() const+0x13e) [0x55caacf21fa0] 2024-05-27T07:48:59.672 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55caacf22651] 2024-05-27T07:48:59.672 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55caaceb7347] 2024-05-27T07:48:59.673 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55caacbb387e] 2024-05-27T07:48:59.673 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55caacbb8f2f] 2024-05-27T07:48:59.673 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55caacbc3969] 2024-05-27T07:48:59.673 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 14: (MDLog::_replay_thread()+0x1f38) [0x55caacf5c48c] 2024-05-27T07:48:59.673 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55caacf659cb] 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 16: (Thread::entry_wrapper()+0x43) [0x7ff7906aa72d] 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff7906aa749] 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff78f24e1ca] 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 19: clone() 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: -1> 2024-05-27T07:48:59.335+0000 7ff77de96700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff77de96700 time 2024-05-27T07:48:59.333644+0000 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff7906dbba6] 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7906dbdd1] 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55caacf3823d] 2024-05-27T07:48:59.674 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55caacf23971] 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 5: (SnapRealm::check_cache() const+0x13e) [0x55caacf21fa0] 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55caacf22651] 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55caaceb7347] 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55caacbb387e] 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55caacbb8f2f] 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55caacbc3969] 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 11: (MDLog::_replay_thread()+0x1f38) [0x55caacf5c48c] 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55caacf659cb] 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 13: (Thread::entry_wrapper()+0x43) [0x7ff7906aa72d] 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff7906aa749] 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff78f24e1ca] 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 16: clone() 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.675 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 0> 2024-05-27T07:48:59.337+0000 7ff77de96700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:48:59.676 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: in thread 7ff77de96700 thread_name:md_log_replay 2024-05-27T07:48:59.676 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.676 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:59.676 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff78f258d20] 2024-05-27T07:48:59.676 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2: gsignal() 2024-05-27T07:48:59.676 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 3: abort() 2024-05-27T07:48:59.676 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff7906dbced] 2024-05-27T07:48:59.676 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7906dbdd1] 2024-05-27T07:48:59.676 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55caacf3823d] 2024-05-27T07:48:59.676 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55caacf23971] 2024-05-27T07:48:59.677 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 8: (SnapRealm::check_cache() const+0x13e) [0x55caacf21fa0] 2024-05-27T07:48:59.677 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55caacf22651] 2024-05-27T07:48:59.677 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55caaceb7347] 2024-05-27T07:48:59.677 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55caacbb387e] 2024-05-27T07:48:59.677 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55caacbb8f2f] 2024-05-27T07:48:59.677 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55caacbc3969] 2024-05-27T07:48:59.677 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 14: (MDLog::_replay_thread()+0x1f38) [0x55caacf5c48c] 2024-05-27T07:48:59.677 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55caacf659cb] 2024-05-27T07:48:59.677 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 16: (Thread::entry_wrapper()+0x43) [0x7ff7906aa72d] 2024-05-27T07:48:59.677 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff7906aa749] 2024-05-27T07:48:59.678 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff78f24e1ca] 2024-05-27T07:48:59.678 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 19: clone() 2024-05-27T07:48:59.678 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:48:59.678 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.678 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: -1> 2024-05-27T07:48:59.335+0000 7ff77de96700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff77de96700 time 2024-05-27T07:48:59.333644+0000 2024-05-27T07:48:59.678 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:48:59.678 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.678 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:59.678 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff7906dbba6] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7906dbdd1] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55caacf3823d] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55caacf23971] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 5: (SnapRealm::check_cache() const+0x13e) [0x55caacf21fa0] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55caacf22651] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55caaceb7347] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55caacbb387e] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55caacbb8f2f] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55caacbc3969] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 11: (MDLog::_replay_thread()+0x1f38) [0x55caacf5c48c] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55caacf659cb] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 13: (Thread::entry_wrapper()+0x43) [0x7ff7906aa72d] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff7906aa749] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff78f24e1ca] 2024-05-27T07:48:59.679 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 16: clone() 2024-05-27T07:48:59.680 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.680 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 0> 2024-05-27T07:48:59.337+0000 7ff77de96700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:48:59.680 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: in thread 7ff77de96700 thread_name:md_log_replay 2024-05-27T07:48:59.680 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.680 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:48:59.680 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff78f258d20] 2024-05-27T07:48:59.680 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2: gsignal() 2024-05-27T07:48:59.680 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 3: abort() 2024-05-27T07:48:59.680 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff7906dbced] 2024-05-27T07:48:59.680 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7906dbdd1] 2024-05-27T07:48:59.681 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55caacf3823d] 2024-05-27T07:48:59.681 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55caacf23971] 2024-05-27T07:48:59.681 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 8: (SnapRealm::check_cache() const+0x13e) [0x55caacf21fa0] 2024-05-27T07:48:59.681 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55caacf22651] 2024-05-27T07:48:59.681 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55caaceb7347] 2024-05-27T07:48:59.681 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55caacbb387e] 2024-05-27T07:48:59.681 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55caacbb8f2f] 2024-05-27T07:48:59.681 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55caacbc3969] 2024-05-27T07:48:59.681 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 14: (MDLog::_replay_thread()+0x1f38) [0x55caacf5c48c] 2024-05-27T07:48:59.681 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55caacf659cb] 2024-05-27T07:48:59.681 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 16: (Thread::entry_wrapper()+0x43) [0x7ff7906aa72d] 2024-05-27T07:48:59.681 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff7906aa749] 2024-05-27T07:48:59.681 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff78f24e1ca] 2024-05-27T07:48:59.682 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 19: clone() 2024-05-27T07:48:59.682 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:48:59.682 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[71863]: 2024-05-27T07:48:59.682 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:48:59 smithi163 podman[71898]: 2024-05-27 07:48:59.379182897 +0000 UTC m=+0.018762908 container died c27adfcedcb2c15a73ef49f252d4f4d447b9097882812ebb24b2fd352c5c25b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, io.buildah.version=1.33.5) 2024-05-27T07:48:59.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:59 smithi186 ceph-mon[36647]: pgmap v299: 129 pgs: 129 active+clean; 67 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 3.0 MiB/s wr, 394 op/s 2024-05-27T07:48:59.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:59 smithi186 ceph-mon[36647]: daemon mds.h restarted 2024-05-27T07:48:59.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:59 smithi186 ceph-mon[36647]: osdmap e110: 12 total, 12 up, 12 in 2024-05-27T07:48:59.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:59 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6836/1508277928,v1:172.21.15.163:6837/1508277928] up:boot 2024-05-27T07:48:59.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:59 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:48:59.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:48:59 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:49:00.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 943212 11.05 MB/sec execute 312 sec latency 31.029 ms 2024-05-27T07:49:00.368 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:00 smithi163 podman[71898]: 2024-05-27 07:49:00.118630662 +0000 UTC m=+0.758210661 container remove c27adfcedcb2c15a73ef49f252d4f4d447b9097882812ebb24b2fd352c5c25b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:49:00.368 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:00 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:49:00.725 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:00 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-05-27T07:49:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.266+0000 7efc84c60700 1 -- 172.21.15.114:0/4241218001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc80109840 msgr2=0x7efc801111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.266+0000 7efc84c60700 1 --2- 172.21.15.114:0/4241218001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc80109840 0x7efc801111a0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7efc7800b0d0 tx=0x7efc7800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:49:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.266+0000 7efc84c60700 1 -- 172.21.15.114:0/4241218001 shutdown_connections 2024-05-27T07:49:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.266+0000 7efc84c60700 1 --2- 172.21.15.114:0/4241218001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc80109840 0x7efc801111a0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.266+0000 7efc84c60700 1 --2- 172.21.15.114:0/4241218001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc801045c0 0x7efc801049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.266+0000 7efc84c60700 1 --2- 172.21.15.114:0/4241218001 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc80108ef0 0x7efc80109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.266+0000 7efc84c60700 1 -- 172.21.15.114:0/4241218001 >> 172.21.15.114:0/4241218001 conn(0x7efc800ffe90 msgr2=0x7efc801022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.266+0000 7efc84c60700 1 -- 172.21.15.114:0/4241218001 shutdown_connections 2024-05-27T07:49:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.266+0000 7efc84c60700 1 -- 172.21.15.114:0/4241218001 wait complete. 2024-05-27T07:49:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.267+0000 7efc84c60700 1 Processor -- start 2024-05-27T07:49:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.267+0000 7efc84c60700 1 -- start start 2024-05-27T07:49:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.267+0000 7efc84c60700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc801045c0 0x7efc8019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.267+0000 7efc84c60700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc80108ef0 0x7efc8019b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.267+0000 7efc84c60700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc80109840 0x7efc8019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.267+0000 7efc84c60700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efc80114470 con 0x7efc801045c0 2024-05-27T07:49:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.267+0000 7efc84c60700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efc801142f0 con 0x7efc80109840 2024-05-27T07:49:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.267+0000 7efc84c60700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efc801145f0 con 0x7efc80108ef0 2024-05-27T07:49:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.267+0000 7efc7e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc801045c0 0x7efc8019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.267+0000 7efc7e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc801045c0 0x7efc8019ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58092/0 (socket says 172.21.15.114:58092) 2024-05-27T07:49:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.267+0000 7efc7dd9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc80108ef0 0x7efc8019b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.267+0000 7efc7e59c700 1 -- 172.21.15.114:0/3888877978 learned_addr learned my addr 172.21.15.114:0/3888877978 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.267+0000 7efc7ed9d700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc80109840 0x7efc8019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.268+0000 7efc7e59c700 1 -- 172.21.15.114:0/3888877978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc80108ef0 msgr2=0x7efc8019b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.268+0000 7efc7e59c700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc80108ef0 0x7efc8019b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:01.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.268+0000 7efc7e59c700 1 -- 172.21.15.114:0/3888877978 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc80109840 msgr2=0x7efc8019f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:01.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.268+0000 7efc7e59c700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc80109840 0x7efc8019f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:01.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.268+0000 7efc7e59c700 1 -- 172.21.15.114:0/3888877978 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc8019fcb0 con 0x7efc801045c0 2024-05-27T07:49:01.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.268+0000 7efc7dd9b700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc80108ef0 0x7efc8019b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:01.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.268+0000 7efc7e59c700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc801045c0 0x7efc8019ade0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7efc7000e9a0 tx=0x7efc7000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:01.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.268+0000 7efc777fe700 1 -- 172.21.15.114:0/3888877978 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7000ca10 con 0x7efc801045c0 2024-05-27T07:49:01.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.268+0000 7efc777fe700 1 -- 172.21.15.114:0/3888877978 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc7000cbb0 con 0x7efc801045c0 2024-05-27T07:49:01.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.268+0000 7efc777fe700 1 -- 172.21.15.114:0/3888877978 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70018680 con 0x7efc801045c0 2024-05-27T07:49:01.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.268+0000 7efc84c60700 1 -- 172.21.15.114:0/3888877978 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc8019ffa0 con 0x7efc801045c0 2024-05-27T07:49:01.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.268+0000 7efc84c60700 1 -- 172.21.15.114:0/3888877978 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc801a78c0 con 0x7efc801045c0 2024-05-27T07:49:01.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.269+0000 7efc84c60700 1 -- 172.21.15.114:0/3888877978 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc800512b0 con 0x7efc801045c0 2024-05-27T07:49:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.271+0000 7efc777fe700 1 -- 172.21.15.114:0/3888877978 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7efc70010430 con 0x7efc801045c0 2024-05-27T07:49:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.271+0000 7efc777fe700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efc6c0b2a70 0x7efc6c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.272+0000 7efc7dd9b700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efc6c0b2a70 0x7efc6c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.272+0000 7efc777fe700 1 -- 172.21.15.114:0/3888877978 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9466+0+0 (secure 0 0 0) 0x7efc700e2ac0 con 0x7efc801045c0 2024-05-27T07:49:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.272+0000 7efc7dd9b700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efc6c0b2a70 0x7efc6c0b4f80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efc80103900 tx=0x7efc68009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.276+0000 7efc777fe700 1 -- 172.21.15.114:0/3888877978 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7efc700aa0f0 con 0x7efc801045c0 2024-05-27T07:49:01.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 943850 11.02 MB/sec execute 313 sec latency 118.044 ms 2024-05-27T07:49:01.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.539+0000 7efc84c60700 1 -- 172.21.15.114:0/3888877978 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc8002e5e0 con 0x7efc801045c0 2024-05-27T07:49:01.539 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:01 smithi114 ceph-mon[32038]: pgmap v301: 129 pgs: 129 active+clean; 69 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 3.4 MiB/s wr, 458 op/s 2024-05-27T07:49:01.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.540+0000 7efc777fe700 1 -- 172.21.15.114:0/3888877978 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+10178 (secure 0 0 0) 0x7efc700b2170 con 0x7efc801045c0 2024-05-27T07:49:01.539 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:49:01.540 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":26882,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2622386718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2622386718},{"type":"v1","addr":"172.21.15.186:6835","nonce":2622386718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":51,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:48:59.286821+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27068":{"gid":27068,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/1508277928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":1508277928},{"type":"v1","addr":"172.21.15.163:6837","nonce":1508277928}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:49:01.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.542+0000 7efc84c60700 1 -- 172.21.15.114:0/3888877978 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efc6c0b2a70 msgr2=0x7efc6c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.542+0000 7efc84c60700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efc6c0b2a70 0x7efc6c0b4f80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efc80103900 tx=0x7efc68009260 comp rx=0 tx=0).stop 2024-05-27T07:49:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.542+0000 7efc84c60700 1 -- 172.21.15.114:0/3888877978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc801045c0 msgr2=0x7efc8019ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.542+0000 7efc84c60700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc801045c0 0x7efc8019ade0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7efc7000e9a0 tx=0x7efc7000ecb0 comp rx=0 tx=0).stop 2024-05-27T07:49:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.542+0000 7efc84c60700 1 -- 172.21.15.114:0/3888877978 shutdown_connections 2024-05-27T07:49:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.542+0000 7efc84c60700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efc6c0b2a70 0x7efc6c0b4f80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.542+0000 7efc84c60700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc80109840 0x7efc8019f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.542+0000 7efc84c60700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc801045c0 0x7efc8019ade0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.542+0000 7efc84c60700 1 --2- 172.21.15.114:0/3888877978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc80108ef0 0x7efc8019b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.542+0000 7efc84c60700 1 -- 172.21.15.114:0/3888877978 >> 172.21.15.114:0/3888877978 conn(0x7efc800ffe90 msgr2=0x7efc80108620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.542+0000 7efc84c60700 1 -- 172.21.15.114:0/3888877978 shutdown_connections 2024-05-27T07:49:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:01.542+0000 7efc84c60700 1 -- 172.21.15.114:0/3888877978 wait complete. 2024-05-27T07:49:01.543 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 51 2024-05-27T07:49:01.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:01 smithi163 ceph-mon[36451]: pgmap v301: 129 pgs: 129 active+clean; 69 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 3.4 MiB/s wr, 458 op/s 2024-05-27T07:49:01.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:01 smithi186 ceph-mon[36647]: pgmap v301: 129 pgs: 129 active+clean; 69 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 3.4 MiB/s wr, 458 op/s 2024-05-27T07:49:02.278 INFO:tasks.workunit.client.0.smithi114.stdout: 1 945522 11.03 MB/sec execute 314 sec latency 87.452 ms 2024-05-27T07:49:02.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:49:02.559 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:02 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3888877978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:49:02.694 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:49:02.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:02 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3888877978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:49:02.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:02 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3888877978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:49:03.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 945905 10.99 MB/sec execute 315 sec latency 124.033 ms 2024-05-27T07:49:03.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:03 smithi163 ceph-mon[36451]: pgmap v302: 129 pgs: 129 active+clean; 73 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 3.4 MiB/s wr, 479 op/s 2024-05-27T07:49:03.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:03 smithi163 ceph-mon[36451]: osdmap e111: 12 total, 12 up, 12 in 2024-05-27T07:49:03.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:03 smithi186 ceph-mon[36647]: pgmap v302: 129 pgs: 129 active+clean; 73 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 3.4 MiB/s wr, 479 op/s 2024-05-27T07:49:03.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:03 smithi186 ceph-mon[36647]: osdmap e111: 12 total, 12 up, 12 in 2024-05-27T07:49:03.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:03 smithi114 ceph-mon[32038]: pgmap v302: 129 pgs: 129 active+clean; 73 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 3.4 MiB/s wr, 479 op/s 2024-05-27T07:49:03.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:03 smithi114 ceph-mon[32038]: osdmap e111: 12 total, 12 up, 12 in 2024-05-27T07:49:04.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 946173 10.96 MB/sec execute 316 sec latency 160.129 ms 2024-05-27T07:49:05.277 INFO:tasks.workunit.client.0.smithi114.stdout: 1 946611 10.94 MB/sec execute 317 sec latency 202.816 ms 2024-05-27T07:49:05.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:05 smithi163 ceph-mon[36451]: pgmap v304: 129 pgs: 129 active+clean; 73 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 158 op/s 2024-05-27T07:49:05.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:05 smithi186 ceph-mon[36647]: pgmap v304: 129 pgs: 129 active+clean; 73 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 158 op/s 2024-05-27T07:49:05.794 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:05 smithi114 ceph-mon[32038]: pgmap v304: 129 pgs: 129 active+clean; 73 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 158 op/s 2024-05-27T07:49:06.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.188+0000 7fa08fcb6640 1 -- 172.21.15.114:0/1919095037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa088102b70 msgr2=0x7fa088102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.188+0000 7fa08fcb6640 1 --2- 172.21.15.114:0/1919095037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa088102b70 0x7fa088102f80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa080009ca0 tx=0x7fa08002a8f0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.189+0000 7fa08fcb6640 1 -- 172.21.15.114:0/1919095037 shutdown_connections 2024-05-27T07:49:06.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.189+0000 7fa08fcb6640 1 --2- 172.21.15.114:0/1919095037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa088107ff0 0x7fa088108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.189+0000 7fa08fcb6640 1 --2- 172.21.15.114:0/1919095037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0881075c0 0x7fa0881079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.189+0000 7fa08fcb6640 1 --2- 172.21.15.114:0/1919095037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa088102b70 0x7fa088102f80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.189+0000 7fa08fcb6640 1 -- 172.21.15.114:0/1919095037 >> 172.21.15.114:0/1919095037 conn(0x7fa0880fe380 msgr2=0x7fa0881007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:06.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.190+0000 7fa08fcb6640 1 -- 172.21.15.114:0/1919095037 shutdown_connections 2024-05-27T07:49:06.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.190+0000 7fa08fcb6640 1 -- 172.21.15.114:0/1919095037 wait complete. 2024-05-27T07:49:06.190 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.191+0000 7fa08fcb6640 1 Processor -- start 2024-05-27T07:49:06.190 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.191+0000 7fa08fcb6640 1 -- start start 2024-05-27T07:49:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.191+0000 7fa08fcb6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa088102b70 0x7fa0881a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.191+0000 7fa08fcb6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0881075c0 0x7fa0881a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.191+0000 7fa08fcb6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa088107ff0 0x7fa0881a5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.191+0000 7fa08fcb6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa08811a580 con 0x7fa0881075c0 2024-05-27T07:49:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.191+0000 7fa08fcb6640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa08811a400 con 0x7fa088102b70 2024-05-27T07:49:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.191+0000 7fa08fcb6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa08811a700 con 0x7fa088107ff0 2024-05-27T07:49:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.192+0000 7fa08da52640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa088102b70 0x7fa0881a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.192+0000 7fa08da52640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa088102b70 0x7fa0881a1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39296/0 (socket says 172.21.15.114:39296) 2024-05-27T07:49:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.192+0000 7fa08da52640 1 -- 172.21.15.114:0/2527158118 learned_addr learned my addr 172.21.15.114:0/2527158118 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.192+0000 7fa08e253640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa088107ff0 0x7fa0881a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.192+0000 7fa08d251640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0881075c0 0x7fa0881a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.192+0000 7fa08da52640 1 -- 172.21.15.114:0/2527158118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa088107ff0 msgr2=0x7fa0881a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.192+0000 7fa08da52640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa088107ff0 0x7fa0881a5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.192+0000 7fa08da52640 1 -- 172.21.15.114:0/2527158118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0881075c0 msgr2=0x7fa0881a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.192+0000 7fa08da52640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0881075c0 0x7fa0881a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.192+0000 7fa08da52640 1 -- 172.21.15.114:0/2527158118 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0881a5ea0 con 0x7fa088102b70 2024-05-27T07:49:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.193+0000 7fa08e253640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa088107ff0 0x7fa0881a5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:49:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.193+0000 7fa08d251640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0881075c0 0x7fa0881a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:49:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.193+0000 7fa08da52640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa088102b70 0x7fa0881a1040 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa0800097b0 tx=0x7fa080003ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.193+0000 7fa07effd640 1 -- 172.21.15.114:0/2527158118 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa080004240 con 0x7fa088102b70 2024-05-27T07:49:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.193+0000 7fa08fcb6640 1 -- 172.21.15.114:0/2527158118 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0881a6130 con 0x7fa088102b70 2024-05-27T07:49:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.193+0000 7fa08fcb6640 1 -- 172.21.15.114:0/2527158118 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0881adb10 con 0x7fa088102b70 2024-05-27T07:49:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.194+0000 7fa07effd640 1 -- 172.21.15.114:0/2527158118 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa08002d070 con 0x7fa088102b70 2024-05-27T07:49:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.194+0000 7fa07effd640 1 -- 172.21.15.114:0/2527158118 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa080030b30 con 0x7fa088102b70 2024-05-27T07:49:06.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.196+0000 7fa096b6a740 1 Processor -- start 2024-05-27T07:49:06.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.197+0000 7fa096b6a740 1 -- start start 2024-05-27T07:49:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.197+0000 7fa096b6a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292ef1b0 0x55aa296903d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.197+0000 7fa096b6a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa29690910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.197+0000 7fa096b6a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 0x55aa29690e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.197+0000 7fa096b6a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55aa29602c30 con 0x55aa292ce4d0 2024-05-27T07:49:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.197+0000 7fa096b6a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55aa295f27d0 con 0x55aa292ef1b0 2024-05-27T07:49:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.197+0000 7fa096b6a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55aa295f6810 con 0x55aa292c62f0 2024-05-27T07:49:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.197+0000 7fa08d251640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 0x55aa29690e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.197+0000 7fa08d251640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 0x55aa29690e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40212/0 (socket says 172.21.15.114:40212) 2024-05-27T07:49:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.197+0000 7fa07effd640 1 -- 172.21.15.114:0/2527158118 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa0800043e0 con 0x7fa088102b70 2024-05-27T07:49:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.197+0000 7fa08d251640 1 -- 172.21.15.114:0/2757447514 learned_addr learned my addr 172.21.15.114:0/2757447514 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.197+0000 7fa08e253640 1 --2- 172.21.15.114:0/2757447514 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292ef1b0 0x55aa296903d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa08da52640 1 --2- 172.21.15.114:0/2757447514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa29690910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa08d251640 1 -- 172.21.15.114:0/2757447514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 msgr2=0x55aa29690910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa08d251640 1 --2- 172.21.15.114:0/2757447514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa29690910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa08d251640 1 -- 172.21.15.114:0/2757447514 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292ef1b0 msgr2=0x55aa296903d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa08d251640 1 --2- 172.21.15.114:0/2757447514 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292ef1b0 0x55aa296903d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa08d251640 1 -- 172.21.15.114:0/2757447514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa29307210 con 0x55aa292ce4d0 2024-05-27T07:49:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa08da52640 1 --2- 172.21.15.114:0/2757447514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa29690910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:49:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa08d251640 1 --2- 172.21.15.114:0/2757447514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 0x55aa29690e50 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa084007800 tx=0x7fa084007cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d78f148086f4f59 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa073fff640 1 -- 172.21.15.114:0/2757447514 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa08400d950 con 0x55aa292ce4d0 2024-05-27T07:49:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa073fff640 1 -- 172.21.15.114:0/2757447514 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa084018070 con 0x55aa292ce4d0 2024-05-27T07:49:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa073fff640 1 -- 172.21.15.114:0/2757447514 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa084013e50 con 0x55aa292ce4d0 2024-05-27T07:49:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa07effd640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0740b2a20 0x7fa0740b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa07effd640 1 -- 172.21.15.114:0/2527158118 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7fa080104040 con 0x7fa088102b70 2024-05-27T07:49:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.198+0000 7fa08e253640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0740b2a20 0x7fa0740b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.199+0000 7fa096b6a740 1 -- 172.21.15.114:0/2757447514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 msgr2=0x55aa29690e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.199+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2757447514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 0x55aa29690e50 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa084007800 tx=0x7fa084007cc0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.199+0000 7fa08e253640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0740b2a20 0x7fa0740b4f30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa078006fd0 tx=0x7fa078008040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.199+0000 7fa096b6a740 1 -- 172.21.15.114:0/2757447514 shutdown_connections 2024-05-27T07:49:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.199+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2757447514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 0x55aa29690e50 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.199+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2757447514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa29690910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.199+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2757447514 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292ef1b0 0x55aa296903d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.199+0000 7fa096b6a740 1 -- 172.21.15.114:0/2757447514 >> 172.21.15.114:0/2757447514 conn(0x55aa292e3300 msgr2=0x55aa2968cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.199+0000 7fa096b6a740 1 -- 172.21.15.114:0/2757447514 shutdown_connections 2024-05-27T07:49:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.199+0000 7fa096b6a740 1 -- 172.21.15.114:0/2757447514 wait complete. 2024-05-27T07:49:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.200+0000 7fa096b6a740 1 -- start start 2024-05-27T07:49:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.200+0000 7fa096b6a740 1 Processor -- start 2024-05-27T07:49:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.200+0000 7fa096b6a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa2969a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.200+0000 7fa096b6a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292ce4d0 0x55aa2969aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.200+0000 7fa096b6a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ef1b0 0x55aa296a6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.200+0000 7fa096b6a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55aa292d8790 con 0x55aa292ef1b0 2024-05-27T07:49:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.200+0000 7fa096b6a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55aa295ae890 con 0x55aa292ce4d0 2024-05-27T07:49:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.200+0000 7fa096b6a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55aa2969d500 con 0x55aa292c62f0 2024-05-27T07:49:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.200+0000 7fa096b6a740 10 client.0 _ll_register_callbacks cb 0x55aa29306a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.200+0000 7fa08da52640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa2969a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa08d251640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292ce4d0 0x55aa2969aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa08da52640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa2969a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:49056/0 (socket says 172.21.15.114:49056) 2024-05-27T07:49:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa08da52640 1 -- 172.21.15.114:0/2495758773 learned_addr learned my addr 172.21.15.114:0/2495758773 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa08e253640 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ef1b0 0x55aa296a6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa08d251640 1 -- 172.21.15.114:0/2495758773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 msgr2=0x55aa2969a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa08d251640 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa2969a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa08d251640 1 -- 172.21.15.114:0/2495758773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ef1b0 msgr2=0x55aa296a6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa08d251640 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ef1b0 0x55aa296a6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa08d251640 1 -- 172.21.15.114:0/2495758773 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa293d3620 con 0x55aa292ce4d0 2024-05-27T07:49:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa08da52640 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa2969a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa08e253640 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ef1b0 0x55aa296a6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa08d251640 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292ce4d0 0x55aa2969aee0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa084000c00 tx=0x7fa084013e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c770ab6ece4e7681 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa0717fa640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:49:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa0717fa640 1 -- 172.21.15.114:0/2495758773 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa08400db50 con 0x55aa292ce4d0 2024-05-27T07:49:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa096b6a740 1 -- 172.21.15.114:0/2495758773 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa084018070 con 0x55aa292ce4d0 2024-05-27T07:49:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa0717fa640 1 -- 172.21.15.114:0/2495758773 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7fa08400dcf0 con 0x55aa292ce4d0 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.201+0000 7fa0717fa640 1 -- 172.21.15.114:0/2495758773 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fa084013750 con 0x55aa292ce4d0 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.202+0000 7fa096b6a740 10 client.27095 fetch_fsmap learned FSMap version 51 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.202+0000 7fa096b6a740 1 -- 172.21.15.114:0/2495758773 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55aa296a9620 con 0x55aa292ce4d0 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.202+0000 7fa0717fa640 1 -- 172.21.15.114:0/2495758773 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa084018070 con 0x55aa292ce4d0 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.202+0000 7fa0717fa640 1 -- 172.21.15.114:0/2495758773 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 51) ==== 7120+0+0 (secure 0 0 0) 0x7fa08401c3f0 con 0x55aa292ce4d0 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.202+0000 7fa096b6a740 10 client.27095 fetch_fsmap finished waiting for FSMap version 51 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.202+0000 7fa096b6a740 10 client.27095 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.202+0000 7fa096b6a740 20 client.27095 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.202+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa292add80 0x55aa296acd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.202+0000 7fa096b6a740 4 client.27095 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.202+0000 7fa096b6a740 1 -- 172.21.15.114:0/2495758773 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa29604b80 con 0x55aa292add80 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.202+0000 7fa08da52640 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa292add80 0x55aa296acd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa0717fa640 10 client.27095 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa08da52640 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa292add80 0x55aa296acd30 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb032002a84b96e3 server_cookie=2be0a20a21daa532 in_seq=0 out_seq=0 2024-05-27T07:49:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa0717fa640 1 -- 172.21.15.114:0/2495758773 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55aa29604b80 con 0x55aa292add80 2024-05-27T07:49:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa0717fa640 10 client.27095 handle_command_reply: tid=0 2024-05-27T07:49:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa096b6a740 1 client.27095 shutdown 2024-05-27T07:49:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa096b6a740 1 -- 172.21.15.114:0/2495758773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292ce4d0 msgr2=0x55aa2969aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292ce4d0 0x55aa2969aee0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa084000c00 tx=0x7fa084013e40 comp rx=0 tx=0).stop 2024-05-27T07:49:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa096b6a740 1 -- 172.21.15.114:0/2495758773 shutdown_connections 2024-05-27T07:49:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa292add80 0x55aa296acd30 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa0717fa640 0 client.27095 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ef1b0 0x55aa296a6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292ce4d0 0x55aa2969aee0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2495758773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa2969a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.203+0000 7fa096b6a740 1 -- 172.21.15.114:0/2495758773 >> 172.21.15.114:0/2495758773 conn(0x55aa292e3300 msgr2=0x55aa296920f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.204+0000 7fa096b6a740 1 -- 172.21.15.114:0/2495758773 shutdown_connections 2024-05-27T07:49:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.204+0000 7fa096b6a740 1 -- 172.21.15.114:0/2495758773 wait complete. 2024-05-27T07:49:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.204+0000 7fa096b6a740 20 client.27095 trim_cache size 0 max 16384 2024-05-27T07:49:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.237+0000 7fa096b6a740 1 Processor -- start 2024-05-27T07:49:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.240+0000 7fa096b6a740 1 -- start start 2024-05-27T07:49:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.240+0000 7fa096b6a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292add80 0x55aa2968cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.241+0000 7fa096b6a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa29699640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.241+0000 7fa096b6a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 0x55aa296b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.241+0000 7fa096b6a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c0102b0 con 0x55aa292ce4d0 2024-05-27T07:49:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.241+0000 7fa096b6a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c010980 con 0x55aa292add80 2024-05-27T07:49:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.241+0000 7fa096b6a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c00e120 con 0x55aa292c62f0 2024-05-27T07:49:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.241+0000 7fa08da52640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292add80 0x55aa2968cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.241+0000 7fa08da52640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292add80 0x55aa2968cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39312/0 (socket says 172.21.15.114:39312) 2024-05-27T07:49:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.241+0000 7fa08da52640 1 -- 172.21.15.114:0/2166863596 learned_addr learned my addr 172.21.15.114:0/2166863596 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.242+0000 7fa08da52640 1 -- 172.21.15.114:0/2166863596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 msgr2=0x55aa29699640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:49:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.242+0000 7fa08da52640 1 --2- 172.21.15.114:0/2166863596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa29699640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.242+0000 7fa08da52640 1 -- 172.21.15.114:0/2166863596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 msgr2=0x55aa296b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.242+0000 7fa08da52640 1 --2- 172.21.15.114:0/2166863596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 0x55aa296b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.242+0000 7fa08da52640 1 -- 172.21.15.114:0/2166863596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa29698ce0 con 0x55aa292add80 2024-05-27T07:49:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.243+0000 7fa08da52640 1 --2- 172.21.15.114:0/2166863596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292add80 0x55aa2968cbb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa080101930 tx=0x7fa08002a8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70d3ae416f5d9f69 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.244+0000 7fa0737fe640 1 -- 172.21.15.114:0/2166863596 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0800caae0 con 0x55aa292add80 2024-05-27T07:49:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.244+0000 7fa0737fe640 1 -- 172.21.15.114:0/2166863596 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa080036030 con 0x55aa292add80 2024-05-27T07:49:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.244+0000 7fa0737fe640 1 -- 172.21.15.114:0/2166863596 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa080030d50 con 0x55aa292add80 2024-05-27T07:49:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.246+0000 7fa096b6a740 1 -- 172.21.15.114:0/2166863596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292add80 msgr2=0x55aa2968cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.246+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2166863596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292add80 0x55aa2968cbb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa080101930 tx=0x7fa08002a8f0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.250+0000 7fa096b6a740 1 -- 172.21.15.114:0/2166863596 shutdown_connections 2024-05-27T07:49:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.250+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2166863596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 0x55aa296b24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.250+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2166863596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa29699640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.250+0000 7fa096b6a740 1 --2- 172.21.15.114:0/2166863596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292add80 0x55aa2968cbb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.250+0000 7fa096b6a740 1 -- 172.21.15.114:0/2166863596 >> 172.21.15.114:0/2166863596 conn(0x55aa292e3300 msgr2=0x55aa29698f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.252+0000 7fa096b6a740 1 -- 172.21.15.114:0/2166863596 shutdown_connections 2024-05-27T07:49:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.253+0000 7fa096b6a740 1 -- 172.21.15.114:0/2166863596 wait complete. 2024-05-27T07:49:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.254+0000 7fa096b6a740 1 -- start start 2024-05-27T07:49:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.254+0000 7fa096b6a740 1 Processor -- start 2024-05-27T07:49:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.255+0000 7fa096b6a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292add80 0x55aa2970d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.255+0000 7fa096b6a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa2970daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.255+0000 7fa096b6a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 0x55aa2970dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.255+0000 7fa096b6a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c00e770 con 0x55aa292ce4d0 2024-05-27T07:49:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.255+0000 7fa096b6a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c00fbe0 con 0x55aa292add80 2024-05-27T07:49:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.255+0000 7fa096b6a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c00ee40 con 0x55aa292c62f0 2024-05-27T07:49:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.255+0000 7fa096b6a740 10 client.0 _ll_register_callbacks cb 0x7fa07800b1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.255+0000 7fa08d251640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa2970daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.255+0000 7fa08d251640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa2970daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:49062/0 (socket says 172.21.15.114:49062) 2024-05-27T07:49:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.255+0000 7fa08d251640 1 -- 172.21.15.114:0/112131053 learned_addr learned my addr 172.21.15.114:0/112131053 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.255+0000 7fa08e253640 1 --2- 172.21.15.114:0/112131053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 0x55aa2970dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.255+0000 7fa08da52640 1 --2- 172.21.15.114:0/112131053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292add80 0x55aa2970d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.256+0000 7fa08da52640 1 -- 172.21.15.114:0/112131053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 msgr2=0x55aa2970daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.256+0000 7fa08da52640 1 --2- 172.21.15.114:0/112131053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa2970daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.256+0000 7fa08da52640 1 -- 172.21.15.114:0/112131053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 msgr2=0x55aa2970dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.256+0000 7fa08da52640 1 --2- 172.21.15.114:0/112131053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 0x55aa2970dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.256+0000 7fa08da52640 1 -- 172.21.15.114:0/112131053 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa296a1930 con 0x55aa292add80 2024-05-27T07:49:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.258+0000 7fa08da52640 1 --2- 172.21.15.114:0/112131053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292add80 0x55aa2970d560 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa080009470 tx=0x7fa0800cf3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6126603ab858f64 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.258+0000 7fa0717fa640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:49:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.258+0000 7fa0717fa640 1 -- 172.21.15.114:0/112131053 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0800cb6d0 con 0x55aa292add80 2024-05-27T07:49:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.258+0000 7fa0717fa640 1 -- 172.21.15.114:0/112131053 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7fa0800c75c0 con 0x55aa292add80 2024-05-27T07:49:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.258+0000 7fa096b6a740 1 -- 172.21.15.114:0/112131053 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa080036030 con 0x55aa292add80 2024-05-27T07:49:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.259+0000 7fa0717fa640 1 -- 172.21.15.114:0/112131053 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fa0801015b0 con 0x55aa292add80 2024-05-27T07:49:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.259+0000 7fa096b6a740 10 client.27101 fetch_fsmap learned FSMap version 51 2024-05-27T07:49:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.259+0000 7fa096b6a740 1 -- 172.21.15.114:0/112131053 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55aa296a2bc0 con 0x55aa292add80 2024-05-27T07:49:06.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.260+0000 7fa0717fa640 1 -- 172.21.15.114:0/112131053 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0800ea070 con 0x55aa292add80 2024-05-27T07:49:06.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.260+0000 7fa0717fa640 1 -- 172.21.15.114:0/112131053 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 51) ==== 7120+0+0 (secure 0 0 0) 0x7fa0801060e0 con 0x55aa292add80 2024-05-27T07:49:06.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.260+0000 7fa096b6a740 10 client.27101 fetch_fsmap finished waiting for FSMap version 51 2024-05-27T07:49:06.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.260+0000 7fa096b6a740 10 client.27101 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:06.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.260+0000 7fa096b6a740 20 client.27101 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:06.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.260+0000 7fa096b6a740 1 --2- 172.21.15.114:0/112131053 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa292ef1b0 0x55aa297283e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.260+0000 7fa096b6a740 4 client.27101 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:49:06.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.260+0000 7fa096b6a740 1 -- 172.21.15.114:0/112131053 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aa295fe360 con 0x55aa292ef1b0 2024-05-27T07:49:06.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.267+0000 7fa08d251640 1 --2- 172.21.15.114:0/112131053 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa292ef1b0 0x55aa297283e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.270+0000 7fa08d251640 1 --2- 172.21.15.114:0/112131053 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa292ef1b0 0x55aa297283e0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a873e6bc00b6736 server_cookie=b54d2f3de77f2915 in_seq=0 out_seq=0 2024-05-27T07:49:06.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.270+0000 7fa0717fa640 10 client.27101 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:06.301 INFO:tasks.workunit.client.0.smithi114.stdout: 1 946931 10.91 MB/sec execute 318 sec latency 121.153 ms 2024-05-27T07:49:06.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.795+0000 7f1b3d955700 1 -- 172.21.15.114:0/2082322086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b3810a920 msgr2=0x7f1b3810ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.795+0000 7f1b3d955700 1 --2- 172.21.15.114:0/2082322086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b3810a920 0x7f1b3810ad30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1b2801a9a0 tx=0x7f1b2801acb0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.795+0000 7f1b3d955700 1 -- 172.21.15.114:0/2082322086 shutdown_connections 2024-05-27T07:49:06.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.795+0000 7f1b3d955700 1 --2- 172.21.15.114:0/2082322086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b3811d920 0x7f1b3811fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.795+0000 7f1b3d955700 1 --2- 172.21.15.114:0/2082322086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b3810a920 0x7f1b3810ad30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.795+0000 7f1b3d955700 1 --2- 172.21.15.114:0/2082322086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b3810b270 0x7f1b3811d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.795+0000 7f1b3d955700 1 -- 172.21.15.114:0/2082322086 >> 172.21.15.114:0/2082322086 conn(0x7f1b3806f4c0 msgr2=0x7f1b3806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:06.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.796+0000 7f1b3d955700 1 -- 172.21.15.114:0/2082322086 shutdown_connections 2024-05-27T07:49:06.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.796+0000 7f1b3d955700 1 -- 172.21.15.114:0/2082322086 wait complete. 2024-05-27T07:49:06.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.796+0000 7f1b3d955700 1 Processor -- start 2024-05-27T07:49:06.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.797+0000 7f1b3d955700 1 -- start start 2024-05-27T07:49:06.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.797+0000 7f1b3d955700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b3810a920 0x7f1b3811d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.797+0000 7f1b3d955700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b3810b270 0x7f1b381192b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.797+0000 7f1b3d955700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b3811d920 0x7f1b381197f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.797+0000 7f1b3d955700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1b381153e0 con 0x7f1b3810b270 2024-05-27T07:49:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.797+0000 7f1b3d955700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1b38115260 con 0x7f1b3810a920 2024-05-27T07:49:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.797+0000 7f1b3d955700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1b38115560 con 0x7f1b3811d920 2024-05-27T07:49:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.798+0000 7f1b36ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b3810a920 0x7f1b3811d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.798+0000 7f1b36ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b3810a920 0x7f1b3811d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39276/0 (socket says 172.21.15.114:39276) 2024-05-27T07:49:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.798+0000 7f1b36ffd700 1 -- 172.21.15.114:0/2510530562 learned_addr learned my addr 172.21.15.114:0/2510530562 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.798+0000 7f1b367fc700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b3810b270 0x7f1b381192b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.798+0000 7f1b377fe700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b3811d920 0x7f1b381197f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.799+0000 7f1b367fc700 1 -- 172.21.15.114:0/2510530562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b3811d920 msgr2=0x7f1b381197f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.799+0000 7f1b367fc700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b3811d920 0x7f1b381197f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.799+0000 7f1b367fc700 1 -- 172.21.15.114:0/2510530562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b3810a920 msgr2=0x7f1b3811d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.799+0000 7f1b367fc700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b3810a920 0x7f1b3811d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.799+0000 7f1b367fc700 1 -- 172.21.15.114:0/2510530562 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b38119f30 con 0x7f1b3810b270 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.799+0000 7f1b367fc700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b3810b270 0x7f1b381192b0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00d950 tx=0x7f1b2c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.803+0000 7f1b3c953700 1 -- 172.21.15.114:0/2510530562 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c017070 con 0x7f1b3810b270 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.803+0000 7f1b3c953700 1 -- 172.21.15.114:0/2510530562 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b2c00be10 con 0x7f1b3810b270 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.803+0000 7f1b3c953700 1 -- 172.21.15.114:0/2510530562 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c00f3c0 con 0x7f1b3810b270 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.804+0000 7f1b3d955700 1 -- 172.21.15.114:0/2510530562 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b381ad8b0 con 0x7f1b3810b270 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.804+0000 7f1b3d955700 1 -- 172.21.15.114:0/2510530562 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b381add10 con 0x7f1b3810b270 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.811+0000 7f1b3c953700 1 -- 172.21.15.114:0/2510530562 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f1b2c01a070 con 0x7f1b3810b270 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.812+0000 7f1b4c36ab80 1 Processor -- start 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.813+0000 7f1b3c953700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1b200b2c00 0x7f1b200b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.813+0000 7f1b3c953700 1 -- 172.21.15.114:0/2510530562 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7f1b2c0e22a0 con 0x7f1b3810b270 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.816+0000 7f1b4c36ab80 1 -- start start 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.816+0000 7f1b4c36ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 0x558582aa0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b4c36ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582a9aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b4c36ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558582a9af80 0x558582a9b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b4c36ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5585829c9c90 con 0x558582a9af80 2024-05-27T07:49:06.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b36ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 0x558582aa0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b36ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 0x558582aa0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39278/0 (socket says 172.21.15.114:39278) 2024-05-27T07:49:06.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b36ffd700 1 -- 172.21.15.114:0/1720598298 learned_addr learned my addr 172.21.15.114:0/1720598298 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:06.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/1720598298 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5585829b9830 con 0x55858291ce40 2024-05-27T07:49:06.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/1720598298 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5585829bd870 con 0x558582aa0c50 2024-05-27T07:49:06.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b377fe700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1b200b2c00 0x7f1b200b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b367fc700 1 --2- 172.21.15.114:0/1720598298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582a9aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b377fe700 1 --2- 172.21.15.114:0/1720598298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558582a9af80 0x558582a9b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b36ffd700 1 -- 172.21.15.114:0/1720598298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 msgr2=0x558582a9aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:06.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b36ffd700 1 --2- 172.21.15.114:0/1720598298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582a9aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b36ffd700 1 -- 172.21.15.114:0/1720598298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558582a9af80 msgr2=0x558582a9b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b36ffd700 1 --2- 172.21.15.114:0/1720598298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558582a9af80 0x558582a9b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.817+0000 7f1b36ffd700 1 -- 172.21.15.114:0/1720598298 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585829098d0 con 0x55858291ce40 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.818+0000 7f1b377fe700 1 --2- 172.21.15.114:0/1720598298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558582a9af80 0x558582a9b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.818+0000 7f1b377fe700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1b200b2c00 0x7f1b200b5110 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1b300060b0 tx=0x7f1b3000d040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.819+0000 7f1b36ffd700 1 --2- 172.21.15.114:0/1720598298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 0x558582aa0710 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1b28003720 tx=0x7f1b28005da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0a35720e4d0790a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.819+0000 7f1b1dffb700 1 -- 172.21.15.114:0/1720598298 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28003790 con 0x55858291ce40 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.820+0000 7f1b1dffb700 1 -- 172.21.15.114:0/1720598298 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b28003930 con 0x55858291ce40 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.820+0000 7f1b1dffb700 1 -- 172.21.15.114:0/1720598298 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28033440 con 0x55858291ce40 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.823+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/1720598298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 msgr2=0x558582aa0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.823+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/1720598298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 0x558582aa0710 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1b28003720 tx=0x7f1b28005da0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.825+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/1720598298 shutdown_connections 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.825+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/1720598298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558582a9af80 0x558582a9b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.825+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/1720598298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 0x558582aa0710 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.825+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/1720598298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582a9aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.825+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/1720598298 >> 172.21.15.114:0/1720598298 conn(0x558582977af0 msgr2=0x558582a95640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.826+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/1720598298 shutdown_connections 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.826+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/1720598298 wait complete. 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.827+0000 7f1b4c36ab80 1 -- start start 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.827+0000 7f1b4c36ab80 1 Processor -- start 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b4c36ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 0x558582aa4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b4c36ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558582a9af80 0x558582aa4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b4c36ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582aa4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b36ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 0x558582aa4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b36ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 0x558582aa4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40192/0 (socket says 172.21.15.114:40192) 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b36ffd700 1 -- 172.21.15.114:0/3854483753 learned_addr learned my addr 172.21.15.114:0/3854483753 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3854483753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5585828bffa0 con 0x55858291ce40 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3854483753 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55858295b7c0 con 0x558582a9af80 2024-05-27T07:49:06.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3854483753 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558582aa76e0 con 0x558582aa0c50 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b4c36ab80 10 client.0 _ll_register_callbacks cb 0x558582954920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b36ffd700 1 -- 172.21.15.114:0/3854483753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 msgr2=0x558582aa4dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b36ffd700 1 --2- 172.21.15.114:0/3854483753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582aa4dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b36ffd700 1 -- 172.21.15.114:0/3854483753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558582a9af80 msgr2=0x558582aa4890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b36ffd700 1 --2- 172.21.15.114:0/3854483753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558582a9af80 0x558582aa4890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.828+0000 7f1b36ffd700 1 -- 172.21.15.114:0/3854483753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558582958690 con 0x55858291ce40 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.829+0000 7f1b36ffd700 1 --2- 172.21.15.114:0/3854483753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 0x558582aa4350 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f1b2801a760 tx=0x7f1b28005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86558fd29d942b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.829+0000 7f1b16ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.829+0000 7f1b16ffd700 1 -- 172.21.15.114:0/3854483753 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b280050c0 con 0x55858291ce40 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.829+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3854483753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b28003930 con 0x55858291ce40 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.829+0000 7f1b16ffd700 1 -- 172.21.15.114:0/3854483753 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7f1b28005260 con 0x55858291ce40 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.830+0000 7f1b16ffd700 1 -- 172.21.15.114:0/3854483753 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f1b28005480 con 0x55858291ce40 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.832+0000 7f1b4c36ab80 10 client.17628 fetch_fsmap learned FSMap version 51 2024-05-27T07:49:06.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.832+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3854483753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x558582ab20a0 con 0x55858291ce40 2024-05-27T07:49:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.834+0000 7f1b16ffd700 1 -- 172.21.15.114:0/3854483753 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b28003930 con 0x55858291ce40 2024-05-27T07:49:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.834+0000 7f1b16ffd700 1 -- 172.21.15.114:0/3854483753 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 51) ==== 7120+0+0 (secure 0 0 0) 0x7f1b2803a020 con 0x55858291ce40 2024-05-27T07:49:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.835+0000 7f1b4c36ab80 10 client.17628 fetch_fsmap finished waiting for FSMap version 51 2024-05-27T07:49:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.835+0000 7f1b4c36ab80 10 client.17628 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.835+0000 7f1b4c36ab80 20 client.17628 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.835+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/3854483753 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558582ab4570 0x558582ab6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.835+0000 7f1b4c36ab80 4 client.17628 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:49:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.835+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3854483753 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558582a669f0 con 0x558582ab4570 2024-05-27T07:49:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.837+0000 7f1b367fc700 1 --2- 172.21.15.114:0/3854483753 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558582ab4570 0x558582ab6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.839+0000 7f1b367fc700 1 --2- 172.21.15.114:0/3854483753 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558582ab4570 0x558582ab6990 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf5e97c8b9068c97 server_cookie=112d62c230a82559 in_seq=0 out_seq=0 2024-05-27T07:49:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.839+0000 7f1b16ffd700 10 client.17628 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.839+0000 7f1b16ffd700 1 -- 172.21.15.114:0/3854483753 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558582a669f0 con 0x558582ab4570 2024-05-27T07:49:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.839+0000 7f1b16ffd700 10 client.17628 handle_command_reply: tid=0 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.840+0000 7f1b4c36ab80 1 client.17628 shutdown 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.840+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3854483753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 msgr2=0x558582aa4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.840+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/3854483753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 0x558582aa4350 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f1b2801a760 tx=0x7f1b28005e10 comp rx=0 tx=0).stop 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.842+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3854483753 shutdown_connections 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.842+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/3854483753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582aa4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.842+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/3854483753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 0x558582aa4350 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.842+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/3854483753 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558582ab4570 0x558582ab6990 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.842+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/3854483753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558582a9af80 0x558582aa4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.842+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3854483753 >> 172.21.15.114:0/3854483753 conn(0x558582977af0 msgr2=0x558582a99a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.842+0000 7f1b16ffd700 0 client.17628 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.843+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3854483753 shutdown_connections 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.843+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3854483753 wait complete. 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.844+0000 7f1b4c36ab80 20 client.17628 trim_cache size 0 max 16384 2024-05-27T07:49:06.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.873+0000 7f1b4c36ab80 1 Processor -- start 2024-05-27T07:49:06.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.874+0000 7f1b4c36ab80 1 -- start start 2024-05-27T07:49:06.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.875+0000 7f1b4c36ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 0x558582a9aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.875+0000 7f1b4c36ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558582a9af80 0x558582aa4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.875+0000 7f1b4c36ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582ab0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.876+0000 7f1b4c36ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1b381b6fb0 con 0x55858291ce40 2024-05-27T07:49:06.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.876+0000 7f1b4c36ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1b381b7680 con 0x558582a9af80 2024-05-27T07:49:06.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.876+0000 7f1b367fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558582a9af80 0x558582aa4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.876+0000 7f1b4c36ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1b381b4e20 con 0x558582aa0c50 2024-05-27T07:49:06.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.877+0000 7f1b36ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 0x558582a9aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.877+0000 7f1b36ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 0x558582a9aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40194/0 (socket says 172.21.15.114:40194) 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.877+0000 7f1b36ffd700 1 -- 172.21.15.114:0/735928001 learned_addr learned my addr 172.21.15.114:0/735928001 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.877+0000 7f1b36ffd700 1 -- 172.21.15.114:0/735928001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 msgr2=0x558582ab0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.877+0000 7f1b36ffd700 1 --2- 172.21.15.114:0/735928001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582ab0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.877+0000 7f1b36ffd700 1 -- 172.21.15.114:0/735928001 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558582a9af80 msgr2=0x558582aa4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.877+0000 7f1b36ffd700 1 --2- 172.21.15.114:0/735928001 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558582a9af80 0x558582aa4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.877+0000 7f1b36ffd700 1 -- 172.21.15.114:0/735928001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558582aa28e0 con 0x55858291ce40 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.877+0000 7f1b367fc700 1 --2- 172.21.15.114:0/735928001 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558582a9af80 0x558582aa4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.878+0000 7f1b36ffd700 1 --2- 172.21.15.114:0/735928001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 0x558582a9aa40 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f1b28005a80 tx=0x7f1b280036a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd6f911a59845742 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.880+0000 7f1b1d7fa700 1 -- 172.21.15.114:0/735928001 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28005620 con 0x55858291ce40 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.881+0000 7f1b1d7fa700 1 -- 172.21.15.114:0/735928001 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b28003910 con 0x55858291ce40 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.881+0000 7f1b1d7fa700 1 -- 172.21.15.114:0/735928001 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28005070 con 0x55858291ce40 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.882+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/735928001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 msgr2=0x558582a9aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.882+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/735928001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 0x558582a9aa40 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f1b28005a80 tx=0x7f1b280036a0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.882+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/735928001 shutdown_connections 2024-05-27T07:49:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.882+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/735928001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582ab0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.882+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/735928001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55858291ce40 0x558582a9aa40 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.882+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/735928001 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558582a9af80 0x558582aa4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.882+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/735928001 >> 172.21.15.114:0/735928001 conn(0x558582977af0 msgr2=0x558582a9e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.885+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/735928001 shutdown_connections 2024-05-27T07:49:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.886+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/735928001 wait complete. 2024-05-27T07:49:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.886+0000 7f1b4c36ab80 1 -- start start 2024-05-27T07:49:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.887+0000 7f1b4c36ab80 1 Processor -- start 2024-05-27T07:49:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.887+0000 7f1b4c36ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 0x558582aad080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.888+0000 7f1b4c36ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558582a9af80 0x558582abb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.888+0000 7f1b4c36ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582abbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.888+0000 7f1b4c36ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1b381b5470 con 0x558582a9af80 2024-05-27T07:49:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.888+0000 7f1b4c36ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1b381b68e0 con 0x55858291ce40 2024-05-27T07:49:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.888+0000 7f1b4c36ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1b381b5b40 con 0x558582aa0c50 2024-05-27T07:49:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.888+0000 7f1b4c36ab80 10 client.0 _ll_register_callbacks cb 0x558582954920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.890+0000 7f1b36ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 0x558582aad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.890+0000 7f1b377fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582abbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.890+0000 7f1b36ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 0x558582aad080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39288/0 (socket says 172.21.15.114:39288) 2024-05-27T07:49:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.890+0000 7f1b36ffd700 1 -- 172.21.15.114:0/3237339842 learned_addr learned my addr 172.21.15.114:0/3237339842 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.891+0000 7f1b36ffd700 1 -- 172.21.15.114:0/3237339842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 msgr2=0x558582abbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.891+0000 7f1b36ffd700 1 --2- 172.21.15.114:0/3237339842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582abbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.891+0000 7f1b36ffd700 1 -- 172.21.15.114:0/3237339842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558582a9af80 msgr2=0x558582abb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.891+0000 7f1b36ffd700 1 --2- 172.21.15.114:0/3237339842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558582a9af80 0x558582abb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.891+0000 7f1b36ffd700 1 -- 172.21.15.114:0/3237339842 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558582abc460 con 0x55858291ce40 2024-05-27T07:49:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.893+0000 7f1b36ffd700 1 --2- 172.21.15.114:0/3237339842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 0x558582aad080 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1b28021e60 tx=0x7f1b28021ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1efc9b27386a715b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.893+0000 7f1b16ffd700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.895+0000 7f1b16ffd700 1 -- 172.21.15.114:0/3237339842 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28005280 con 0x55858291ce40 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.895+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3237339842 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b28003910 con 0x55858291ce40 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.895+0000 7f1b16ffd700 1 -- 172.21.15.114:0/3237339842 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7f1b28019070 con 0x55858291ce40 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.896+0000 7f1b16ffd700 1 -- 172.21.15.114:0/3237339842 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f1b28051050 con 0x55858291ce40 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.897+0000 7f1b4c36ab80 10 client.27086 fetch_fsmap learned FSMap version 51 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.897+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3237339842 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x558582b1ec80 con 0x55858291ce40 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.898+0000 7f1b16ffd700 1 -- 172.21.15.114:0/3237339842 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b28003910 con 0x55858291ce40 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.898+0000 7f1b16ffd700 1 -- 172.21.15.114:0/3237339842 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 51) ==== 7120+0+0 (secure 0 0 0) 0x7f1b2803a040 con 0x55858291ce40 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.899+0000 7f1b4c36ab80 10 client.27086 fetch_fsmap finished waiting for FSMap version 51 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.899+0000 7f1b4c36ab80 10 client.27086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.899+0000 7f1b4c36ab80 20 client.27086 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.899+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/3237339842 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558582b1efa0 0x558582b213c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.899+0000 7f1b4c36ab80 4 client.27086 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:49:06.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.899+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3237339842 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5585829cbbe0 con 0x558582b1efa0 2024-05-27T07:49:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.899+0000 7f1b367fc700 1 --2- 172.21.15.114:0/3237339842 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558582b1efa0 0x558582b213c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.900+0000 7f1b16ffd700 10 client.27086 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.900+0000 7f1b367fc700 1 --2- 172.21.15.114:0/3237339842 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558582b1efa0 0x558582b213c0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b950f642ff6e43a5 server_cookie=e3d794fead9308e9 in_seq=0 out_seq=0 2024-05-27T07:49:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.901+0000 7f1b16ffd700 1 -- 172.21.15.114:0/3237339842 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5585829cbbe0 con 0x558582b1efa0 2024-05-27T07:49:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.901+0000 7f1b16ffd700 10 client.27086 handle_command_reply: tid=0 2024-05-27T07:49:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.901+0000 7f1b4c36ab80 1 client.27086 shutdown 2024-05-27T07:49:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.902+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3237339842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 msgr2=0x558582aad080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.902+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/3237339842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 0x558582aad080 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1b28021e60 tx=0x7f1b28021ee0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.365 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:49:06.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.902+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3237339842 shutdown_connections 2024-05-27T07:49:06.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.902+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/3237339842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558582aa0c50 0x558582abbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.902+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/3237339842 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558582b1efa0 0x558582b213c0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.902+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/3237339842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55858291ce40 0x558582aad080 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.902+0000 7f1b4c36ab80 1 --2- 172.21.15.114:0/3237339842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558582a9af80 0x558582abb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.902+0000 7f1b16ffd700 0 client.27086 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:06.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.902+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3237339842 >> 172.21.15.114:0/3237339842 conn(0x558582977af0 msgr2=0x558582a9f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:06.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.902+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3237339842 shutdown_connections 2024-05-27T07:49:06.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.902+0000 7f1b4c36ab80 1 -- 172.21.15.114:0/3237339842 wait complete. 2024-05-27T07:49:06.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.902+0000 7f1b4c36ab80 20 client.27086 trim_cache size 0 max 16384 2024-05-27T07:49:06.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.902+0000 7f1b3d955700 1 -- 172.21.15.114:0/2510530562 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1b200b2c00 msgr2=0x7f1b200b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.903+0000 7f1b3d955700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1b200b2c00 0x7f1b200b5110 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1b300060b0 tx=0x7f1b3000d040 comp rx=0 tx=0).stop 2024-05-27T07:49:06.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.903+0000 7f1b3d955700 1 -- 172.21.15.114:0/2510530562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b3810b270 msgr2=0x7f1b381192b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.903+0000 7f1b3d955700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b3810b270 0x7f1b381192b0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00d950 tx=0x7f1b2c00dd10 comp rx=0 tx=0).stop 2024-05-27T07:49:06.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.903+0000 7f1b3d955700 1 -- 172.21.15.114:0/2510530562 shutdown_connections 2024-05-27T07:49:06.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.903+0000 7f1b3d955700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b3811d920 0x7f1b381197f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.903+0000 7f1b3d955700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1b200b2c00 0x7f1b200b5110 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.903+0000 7f1b3d955700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b3810a920 0x7f1b3811d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.903+0000 7f1b3d955700 1 --2- 172.21.15.114:0/2510530562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b3810b270 0x7f1b381192b0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.903+0000 7f1b3d955700 1 -- 172.21.15.114:0/2510530562 >> 172.21.15.114:0/2510530562 conn(0x7f1b3806f4c0 msgr2=0x7f1b3811ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:06.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.903+0000 7f1b3d955700 1 -- 172.21.15.114:0/2510530562 shutdown_connections 2024-05-27T07:49:06.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:05.903+0000 7f1b3d955700 1 -- 172.21.15.114:0/2510530562 wait complete. 2024-05-27T07:49:06.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.849+0000 7fa0717fa640 1 -- 172.21.15.114:0/112131053 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55aa295fe360 con 0x55aa292ef1b0 2024-05-27T07:49:06.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.849+0000 7fa0717fa640 10 client.27101 handle_command_reply: tid=0 2024-05-27T07:49:06.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.849+0000 7fa096b6a740 1 client.27101 shutdown 2024-05-27T07:49:06.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.849+0000 7fa096b6a740 1 -- 172.21.15.114:0/112131053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292add80 msgr2=0x55aa2970d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.849+0000 7fa096b6a740 1 --2- 172.21.15.114:0/112131053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292add80 0x55aa2970d560 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa080009470 tx=0x7fa0800cf3f0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.849+0000 7fa096b6a740 1 -- 172.21.15.114:0/112131053 shutdown_connections 2024-05-27T07:49:06.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.849+0000 7fa096b6a740 1 --2- 172.21.15.114:0/112131053 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55aa292ef1b0 0x55aa297283e0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.849+0000 7fa096b6a740 1 --2- 172.21.15.114:0/112131053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa292ce4d0 0x55aa2970dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.849+0000 7fa096b6a740 1 --2- 172.21.15.114:0/112131053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa292c62f0 0x55aa2970daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.849+0000 7fa096b6a740 1 --2- 172.21.15.114:0/112131053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55aa292add80 0x55aa2970d560 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.849+0000 7fa096b6a740 1 -- 172.21.15.114:0/112131053 >> 172.21.15.114:0/112131053 conn(0x55aa292e3300 msgr2=0x55aa296996c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.849+0000 7fa0717fa640 0 client.27101 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.849+0000 7fa096b6a740 1 -- 172.21.15.114:0/112131053 shutdown_connections 2024-05-27T07:49:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.850+0000 7fa096b6a740 1 -- 172.21.15.114:0/112131053 wait complete. 2024-05-27T07:49:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.850+0000 7fa096b6a740 20 client.27101 trim_cache size 0 max 16384 2024-05-27T07:49:06.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:49:06.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:49:06.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:49:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.851+0000 7fa08fcb6640 1 -- 172.21.15.114:0/2527158118 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0740b2a20 msgr2=0x7fa0740b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.851+0000 7fa08fcb6640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0740b2a20 0x7fa0740b4f30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa078006fd0 tx=0x7fa078008040 comp rx=0 tx=0).stop 2024-05-27T07:49:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.851+0000 7fa08fcb6640 1 -- 172.21.15.114:0/2527158118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa088102b70 msgr2=0x7fa0881a1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.851+0000 7fa08fcb6640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa088102b70 0x7fa0881a1040 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa0800097b0 tx=0x7fa080003ce0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.852+0000 7fa08fcb6640 1 -- 172.21.15.114:0/2527158118 shutdown_connections 2024-05-27T07:49:06.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.852+0000 7fa08fcb6640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0740b2a20 0x7fa0740b4f30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.852+0000 7fa08fcb6640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa088107ff0 0x7fa0881a5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.852+0000 7fa08fcb6640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0881075c0 0x7fa0881a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.852+0000 7fa08fcb6640 1 --2- 172.21.15.114:0/2527158118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa088102b70 0x7fa0881a1040 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:06.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.852+0000 7fa08fcb6640 1 -- 172.21.15.114:0/2527158118 >> 172.21.15.114:0/2527158118 conn(0x7fa0880fe380 msgr2=0x7fa0881001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:06.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.852+0000 7fa08fcb6640 1 -- 172.21.15.114:0/2527158118 shutdown_connections 2024-05-27T07:49:06.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:06.852+0000 7fa08fcb6640 1 -- 172.21.15.114:0/2527158118 wait complete. 2024-05-27T07:49:07.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 947306 10.87 MB/sec execute 319 sec latency 390.086 ms 2024-05-27T07:49:07.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:07 smithi163 ceph-mon[36451]: pgmap v305: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 76 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.7 MiB/s wr, 273 op/s 2024-05-27T07:49:07.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:07 smithi186 ceph-mon[36647]: pgmap v305: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 76 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.7 MiB/s wr, 273 op/s 2024-05-27T07:49:07.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:07 smithi114 ceph-mon[32038]: pgmap v305: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 76 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.7 MiB/s wr, 273 op/s 2024-05-27T07:49:08.105 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:49:08.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:49:08.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 947871 10.86 MB/sec execute 320 sec latency 81.333 ms 2024-05-27T07:49:08.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:08 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:49:08.571 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:49:08.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:08 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:49:08.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:08 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:49:09.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 948758 10.84 MB/sec execute 321 sec latency 77.454 ms 2024-05-27T07:49:09.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:09 smithi163 ceph-mon[36451]: pgmap v306: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 76 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 3.3 MiB/s wr, 242 op/s 2024-05-27T07:49:09.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:09 smithi186 ceph-mon[36647]: pgmap v306: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 76 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 3.3 MiB/s wr, 242 op/s 2024-05-27T07:49:09.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:09 smithi114 ceph-mon[32038]: pgmap v306: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 76 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 3.3 MiB/s wr, 242 op/s 2024-05-27T07:49:10.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 949329 10.81 MB/sec execute 322 sec latency 49.078 ms 2024-05-27T07:49:10.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.499+0000 7fa8822e4700 1 -- 172.21.15.114:0/3860581217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa87c10a920 msgr2=0x7fa87c10ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:10.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.499+0000 7fa8822e4700 1 --2- 172.21.15.114:0/3860581217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa87c10a920 0x7fa87c10ad30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa87400b0a0 tx=0x7fa87400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.502+0000 7fa8822e4700 1 -- 172.21.15.114:0/3860581217 shutdown_connections 2024-05-27T07:49:10.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.502+0000 7fa8822e4700 1 --2- 172.21.15.114:0/3860581217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa87c11d920 0x7fa87c11fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.502+0000 7fa8822e4700 1 --2- 172.21.15.114:0/3860581217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa87c10a920 0x7fa87c10ad30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.502+0000 7fa8822e4700 1 --2- 172.21.15.114:0/3860581217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa87c10b270 0x7fa87c11d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.502+0000 7fa8822e4700 1 -- 172.21.15.114:0/3860581217 >> 172.21.15.114:0/3860581217 conn(0x7fa87c06f4c0 msgr2=0x7fa87c06f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:10.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.502+0000 7fa8822e4700 1 -- 172.21.15.114:0/3860581217 shutdown_connections 2024-05-27T07:49:10.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.502+0000 7fa8822e4700 1 -- 172.21.15.114:0/3860581217 wait complete. 2024-05-27T07:49:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.503+0000 7fa8822e4700 1 Processor -- start 2024-05-27T07:49:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.503+0000 7fa8822e4700 1 -- start start 2024-05-27T07:49:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.504+0000 7fa8822e4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa87c10a920 0x7fa87c11ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.504+0000 7fa8822e4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa87c10b270 0x7fa87c112c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.504+0000 7fa8822e4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa87c11d920 0x7fa87c1131d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.504+0000 7fa8822e4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c123350 con 0x7fa87c11d920 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.504+0000 7fa8822e4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c1231d0 con 0x7fa87c10a920 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.504+0000 7fa8822e4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c1234d0 con 0x7fa87c10b270 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.504+0000 7fa87bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa87c10a920 0x7fa87c11ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.504+0000 7fa87bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa87c10a920 0x7fa87c11ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39342/0 (socket says 172.21.15.114:39342) 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.504+0000 7fa87bfff700 1 -- 172.21.15.114:0/3273328326 learned_addr learned my addr 172.21.15.114:0/3273328326 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.504+0000 7fa880881700 1 --2- 172.21.15.114:0/3273328326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa87c11d920 0x7fa87c1131d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.505+0000 7fa87bfff700 1 -- 172.21.15.114:0/3273328326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa87c10b270 msgr2=0x7fa87c112c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.505+0000 7fa87bfff700 1 --2- 172.21.15.114:0/3273328326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa87c10b270 0x7fa87c112c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.505+0000 7fa87bfff700 1 -- 172.21.15.114:0/3273328326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa87c11d920 msgr2=0x7fa87c1131d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.505+0000 7fa87bfff700 1 --2- 172.21.15.114:0/3273328326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa87c11d920 0x7fa87c1131d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.505+0000 7fa87bfff700 1 -- 172.21.15.114:0/3273328326 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87c1138d0 con 0x7fa87c10a920 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.506+0000 7fa87bfff700 1 --2- 172.21.15.114:0/3273328326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa87c10a920 0x7fa87c11ac20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa87400d010 tx=0x7fa874004db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.507+0000 7fa8797fa700 1 -- 172.21.15.114:0/3273328326 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa874021070 con 0x7fa87c10a920 2024-05-27T07:49:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.507+0000 7fa8797fa700 1 -- 172.21.15.114:0/3273328326 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa874007d40 con 0x7fa87c10a920 2024-05-27T07:49:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.507+0000 7fa8797fa700 1 -- 172.21.15.114:0/3273328326 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87401d450 con 0x7fa87c10a920 2024-05-27T07:49:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.508+0000 7fa8822e4700 1 -- 172.21.15.114:0/3273328326 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa87c1a9ae0 con 0x7fa87c10a920 2024-05-27T07:49:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.508+0000 7fa8822e4700 1 -- 172.21.15.114:0/3273328326 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa87c1a9fd0 con 0x7fa87c10a920 2024-05-27T07:49:10.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.516+0000 7fa8797fa700 1 -- 172.21.15.114:0/3273328326 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa874004000 con 0x7fa87c10a920 2024-05-27T07:49:10.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.517+0000 7fa890cf9b80 1 Processor -- start 2024-05-27T07:49:10.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.517+0000 7fa890cf9b80 1 -- start start 2024-05-27T07:49:10.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.517+0000 7fa890cf9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908cf3aec0 0x55908d0be790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.517+0000 7fa890cf9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.517+0000 7fa890cf9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908d0b9000 0x55908d0b9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.517+0000 7fa890cf9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55908cfe7d10 con 0x55908d0b9000 2024-05-27T07:49:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.517+0000 7fa890cf9b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55908cfd78b0 con 0x55908d0becd0 2024-05-27T07:49:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.517+0000 7fa890cf9b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55908cfdb8f0 con 0x55908cf3aec0 2024-05-27T07:49:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.518+0000 7fa87bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.518+0000 7fa87bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0b8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39352/0 (socket says 172.21.15.114:39352) 2024-05-27T07:49:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.518+0000 7fa87bfff700 1 -- 172.21.15.114:0/4259416248 learned_addr learned my addr 172.21.15.114:0/4259416248 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.518+0000 7fa880881700 1 --2- 172.21.15.114:0/4259416248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908cf3aec0 0x55908d0be790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.519+0000 7fa87bfff700 1 -- 172.21.15.114:0/4259416248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908cf3aec0 msgr2=0x55908d0be790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.519+0000 7fa8797fa700 1 --2- 172.21.15.114:0/3273328326 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa8640b2a70 0x7fa8640b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.519+0000 7fa87bfff700 1 --2- 172.21.15.114:0/4259416248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908cf3aec0 0x55908d0be790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.519+0000 7fa87bfff700 1 -- 172.21.15.114:0/4259416248 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908d0b9000 msgr2=0x55908d0b9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.519+0000 7fa87bfff700 1 --2- 172.21.15.114:0/4259416248 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908d0b9000 0x55908d0b9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.519+0000 7fa87bfff700 1 -- 172.21.15.114:0/4259416248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55908cf27950 con 0x55908d0becd0 2024-05-27T07:49:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.519+0000 7fa880881700 1 --2- 172.21.15.114:0/4259416248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908cf3aec0 0x55908d0be790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.519+0000 7fa8797fa700 1 -- 172.21.15.114:0/3273328326 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7fa87400f020 con 0x7fa87c10a920 2024-05-27T07:49:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.520+0000 7fa880881700 1 --2- 172.21.15.114:0/3273328326 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa8640b2a70 0x7fa8640b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.520+0000 7fa87bfff700 1 --2- 172.21.15.114:0/4259416248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0b8ac0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa8740e1260 tx=0x7fa874004350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0718187961b9070 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.521+0000 7fa8627fc700 1 -- 172.21.15.114:0/4259416248 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8740a6e00 con 0x55908d0becd0 2024-05-27T07:49:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.521+0000 7fa8627fc700 1 -- 172.21.15.114:0/4259416248 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8740075c0 con 0x55908d0becd0 2024-05-27T07:49:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.521+0000 7fa8627fc700 1 -- 172.21.15.114:0/4259416248 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8740aa6b0 con 0x55908d0becd0 2024-05-27T07:49:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.522+0000 7fa880881700 1 --2- 172.21.15.114:0/3273328326 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa8640b2a70 0x7fa8640b4f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55908d0b9c30 tx=0x7fa86c004750 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.522+0000 7fa890cf9b80 1 -- 172.21.15.114:0/4259416248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 msgr2=0x55908d0b8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.522+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/4259416248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0b8ac0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa8740e1260 tx=0x7fa874004350 comp rx=0 tx=0).stop 2024-05-27T07:49:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.522+0000 7fa890cf9b80 1 -- 172.21.15.114:0/4259416248 shutdown_connections 2024-05-27T07:49:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.522+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/4259416248 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908d0b9000 0x55908d0b9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.522+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/4259416248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908cf3aec0 0x55908d0be790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.522+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/4259416248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0b8ac0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.522+0000 7fa890cf9b80 1 -- 172.21.15.114:0/4259416248 >> 172.21.15.114:0/4259416248 conn(0x55908cf95b70 msgr2=0x55908d0b36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.523+0000 7fa890cf9b80 1 -- 172.21.15.114:0/4259416248 shutdown_connections 2024-05-27T07:49:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.523+0000 7fa890cf9b80 1 -- 172.21.15.114:0/4259416248 wait complete. 2024-05-27T07:49:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.523+0000 7fa890cf9b80 1 -- start start 2024-05-27T07:49:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.524+0000 7fa890cf9b80 1 Processor -- start 2024-05-27T07:49:10.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.525+0000 7fa890cf9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908cf3aec0 0x55908d0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.525+0000 7fa890cf9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908d0b9000 0x55908d0c2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.525+0000 7fa890cf9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.525+0000 7fa87b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908d0b9000 0x55908d0c2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.525+0000 7fa87b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908d0b9000 0x55908d0c2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40272/0 (socket says 172.21.15.114:40272) 2024-05-27T07:49:10.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.526+0000 7fa880881700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.526+0000 7fa87b7fe700 1 -- 172.21.15.114:0/308297692 learned_addr learned my addr 172.21.15.114:0/308297692 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:10.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.526+0000 7fa87bfff700 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908cf3aec0 0x55908d0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.526+0000 7fa890cf9b80 1 -- 172.21.15.114:0/308297692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55908cede020 con 0x55908d0b9000 2024-05-27T07:49:10.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.526+0000 7fa890cf9b80 1 -- 172.21.15.114:0/308297692 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55908cf79840 con 0x55908d0becd0 2024-05-27T07:49:10.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.526+0000 7fa890cf9b80 1 -- 172.21.15.114:0/308297692 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55908d0c5580 con 0x55908cf3aec0 2024-05-27T07:49:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.527+0000 7fa890cf9b80 10 client.0 _ll_register_callbacks cb 0x55908cf729a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.527+0000 7fa87b7fe700 1 -- 172.21.15.114:0/308297692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908cf3aec0 msgr2=0x55908d0c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.527+0000 7fa87b7fe700 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908cf3aec0 0x55908d0c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.527+0000 7fa87b7fe700 1 -- 172.21.15.114:0/308297692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 msgr2=0x55908d0c2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.527+0000 7fa87b7fe700 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0c2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.527+0000 7fa87b7fe700 1 -- 172.21.15.114:0/308297692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55908cf76710 con 0x55908d0b9000 2024-05-27T07:49:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.528+0000 7fa880881700 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:49:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.528+0000 7fa87bfff700 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908cf3aec0 0x55908d0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:49:10.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.528+0000 7fa87b7fe700 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908d0b9000 0x55908d0c2a30 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa870009e90 tx=0x7fa87000b3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773e74b16dc6525e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.528+0000 7fa85b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:49:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.528+0000 7fa85b7fe700 1 -- 172.21.15.114:0/308297692 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870015020 con 0x55908d0b9000 2024-05-27T07:49:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.529+0000 7fa85b7fe700 1 -- 172.21.15.114:0/308297692 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7fa870011420 con 0x55908d0b9000 2024-05-27T07:49:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.529+0000 7fa890cf9b80 1 -- 172.21.15.114:0/308297692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8740075c0 con 0x55908d0b9000 2024-05-27T07:49:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.529+0000 7fa85b7fe700 1 -- 172.21.15.114:0/308297692 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fa870020050 con 0x55908d0b9000 2024-05-27T07:49:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.529+0000 7fa890cf9b80 10 client.17649 fetch_fsmap learned FSMap version 51 2024-05-27T07:49:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.529+0000 7fa890cf9b80 1 -- 172.21.15.114:0/308297692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55908d0d00f0 con 0x55908d0b9000 2024-05-27T07:49:10.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.530+0000 7fa85b7fe700 1 -- 172.21.15.114:0/308297692 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa870011ad0 con 0x55908d0b9000 2024-05-27T07:49:10.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.530+0000 7fa85b7fe700 1 -- 172.21.15.114:0/308297692 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 51) ==== 7120+0+0 (secure 0 0 0) 0x7fa8700198e0 con 0x55908d0b9000 2024-05-27T07:49:10.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.531+0000 7fa890cf9b80 10 client.17649 fetch_fsmap finished waiting for FSMap version 51 2024-05-27T07:49:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.531+0000 7fa890cf9b80 10 client.17649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.531+0000 7fa890cf9b80 20 client.17649 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.531+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55908d0d2870 0x55908d0d4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.531+0000 7fa890cf9b80 4 client.17649 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:49:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.531+0000 7fa890cf9b80 1 -- 172.21.15.114:0/308297692 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa8740075c0 con 0x55908d0d2870 2024-05-27T07:49:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.532+0000 7fa87bfff700 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55908d0d2870 0x55908d0d4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.532+0000 7fa85b7fe700 10 client.17649 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.532+0000 7fa87bfff700 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55908d0d2870 0x55908d0d4d20 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44bab4a56869f3ca server_cookie=9b59738c2dfeb702 in_seq=0 out_seq=0 2024-05-27T07:49:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.533+0000 7fa85b7fe700 1 -- 172.21.15.114:0/308297692 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fa8740075c0 con 0x55908d0d2870 2024-05-27T07:49:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.533+0000 7fa85b7fe700 10 client.17649 handle_command_reply: tid=0 2024-05-27T07:49:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.533+0000 7fa890cf9b80 1 client.17649 shutdown 2024-05-27T07:49:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.534+0000 7fa890cf9b80 1 -- 172.21.15.114:0/308297692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908d0b9000 msgr2=0x55908d0c2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.534+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908d0b9000 0x55908d0c2a30 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa870009e90 tx=0x7fa87000b3c0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.534+0000 7fa890cf9b80 1 -- 172.21.15.114:0/308297692 shutdown_connections 2024-05-27T07:49:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.534+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55908d0d2870 0x55908d0d4d20 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.534+0000 7fa85b7fe700 0 client.17649 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.534+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.534+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908cf3aec0 0x55908d0c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.534+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/308297692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908d0b9000 0x55908d0c2a30 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.535+0000 7fa890cf9b80 1 -- 172.21.15.114:0/308297692 >> 172.21.15.114:0/308297692 conn(0x55908cf95b70 msgr2=0x55908d0b7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.535+0000 7fa890cf9b80 1 -- 172.21.15.114:0/308297692 shutdown_connections 2024-05-27T07:49:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.535+0000 7fa890cf9b80 1 -- 172.21.15.114:0/308297692 wait complete. 2024-05-27T07:49:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.535+0000 7fa890cf9b80 20 client.17649 trim_cache size 0 max 16384 2024-05-27T07:49:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.572+0000 7fa890cf9b80 1 Processor -- start 2024-05-27T07:49:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.572+0000 7fa890cf9b80 1 -- start start 2024-05-27T07:49:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.572+0000 7fa890cf9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908cf3aec0 0x55908d0b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.572+0000 7fa890cf9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908d0b9000 0x55908d0c2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.573+0000 7fa890cf9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.573+0000 7fa890cf9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c1b3020 con 0x55908cf3aec0 2024-05-27T07:49:10.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.573+0000 7fa890cf9b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c1b36f0 con 0x55908d0becd0 2024-05-27T07:49:10.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.573+0000 7fa890cf9b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c1b0e90 con 0x55908d0b9000 2024-05-27T07:49:10.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.573+0000 7fa880881700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.573+0000 7fa880881700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0ce5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39382/0 (socket says 172.21.15.114:39382) 2024-05-27T07:49:10.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.573+0000 7fa880881700 1 -- 172.21.15.114:0/2924901043 learned_addr learned my addr 172.21.15.114:0/2924901043 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:10.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.573+0000 7fa87b7fe700 1 --2- 172.21.15.114:0/2924901043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908d0b9000 0x55908d0c2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.575+0000 7fa87b7fe700 1 -- 172.21.15.114:0/2924901043 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 msgr2=0x55908d0ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:10.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.575+0000 7fa87b7fe700 1 --2- 172.21.15.114:0/2924901043 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.575+0000 7fa87b7fe700 1 -- 172.21.15.114:0/2924901043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908cf3aec0 msgr2=0x55908d0b8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:10.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.575+0000 7fa87b7fe700 1 --2- 172.21.15.114:0/2924901043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908cf3aec0 0x55908d0b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.575+0000 7fa87b7fe700 1 -- 172.21.15.114:0/2924901043 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55908d0bce10 con 0x55908d0b9000 2024-05-27T07:49:10.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.575+0000 7fa87b7fe700 1 --2- 172.21.15.114:0/2924901043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908d0b9000 0x55908d0c2ee0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa8700048a0 tx=0x7fa870019fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e8080e2931ad227 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:10.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.576+0000 7fa861ffb700 1 -- 172.21.15.114:0/2924901043 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8700193a0 con 0x55908d0b9000 2024-05-27T07:49:10.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.576+0000 7fa861ffb700 1 -- 172.21.15.114:0/2924901043 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8700199f0 con 0x55908d0b9000 2024-05-27T07:49:10.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.576+0000 7fa861ffb700 1 -- 172.21.15.114:0/2924901043 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870010e40 con 0x55908d0b9000 2024-05-27T07:49:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.577+0000 7fa890cf9b80 1 -- 172.21.15.114:0/2924901043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908d0b9000 msgr2=0x55908d0c2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.577+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/2924901043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908d0b9000 0x55908d0c2ee0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa8700048a0 tx=0x7fa870019fb0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.577+0000 7fa890cf9b80 1 -- 172.21.15.114:0/2924901043 shutdown_connections 2024-05-27T07:49:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.577+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/2924901043 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d0ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.577+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/2924901043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908cf3aec0 0x55908d0b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.577+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/2924901043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908d0b9000 0x55908d0c2ee0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.577+0000 7fa890cf9b80 1 -- 172.21.15.114:0/2924901043 >> 172.21.15.114:0/2924901043 conn(0x55908cf95b70 msgr2=0x55908d0bc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:10.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.577+0000 7fa890cf9b80 1 -- 172.21.15.114:0/2924901043 shutdown_connections 2024-05-27T07:49:10.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.577+0000 7fa890cf9b80 1 -- 172.21.15.114:0/2924901043 wait complete. 2024-05-27T07:49:10.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.578+0000 7fa890cf9b80 1 -- start start 2024-05-27T07:49:10.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.578+0000 7fa890cf9b80 1 Processor -- start 2024-05-27T07:49:10.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.579+0000 7fa890cf9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908cf3aec0 0x55908d0c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.579+0000 7fa890cf9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908d0b9000 0x55908d0c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.579+0000 7fa890cf9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d13d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.580+0000 7fa890cf9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c1b14e0 con 0x55908cf3aec0 2024-05-27T07:49:10.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.580+0000 7fa890cf9b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c1b2950 con 0x55908d0becd0 2024-05-27T07:49:10.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.580+0000 7fa890cf9b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c1b1bb0 con 0x55908d0b9000 2024-05-27T07:49:10.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.580+0000 7fa890cf9b80 10 client.0 _ll_register_callbacks cb 0x55908cf729a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:10.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.581+0000 7fa880881700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d13d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.581+0000 7fa880881700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d13d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:39392/0 (socket says 172.21.15.114:39392) 2024-05-27T07:49:10.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.581+0000 7fa880881700 1 -- 172.21.15.114:0/3164087580 learned_addr learned my addr 172.21.15.114:0/3164087580 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:10.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.581+0000 7fa87b7fe700 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908d0b9000 0x55908d0c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.581+0000 7fa87bfff700 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908cf3aec0 0x55908d0c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.581+0000 7fa87b7fe700 1 -- 172.21.15.114:0/3164087580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 msgr2=0x55908d13d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.581+0000 7fa87b7fe700 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d13d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.581+0000 7fa87b7fe700 1 -- 172.21.15.114:0/3164087580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908cf3aec0 msgr2=0x55908d0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.581+0000 7fa87b7fe700 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908cf3aec0 0x55908d0c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.581+0000 7fa87b7fe700 1 -- 172.21.15.114:0/3164087580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55908d13d900 con 0x55908d0b9000 2024-05-27T07:49:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.582+0000 7fa880881700 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d13d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.582+0000 7fa87b7fe700 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908d0b9000 0x55908d0c7ec0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa870009e60 tx=0x7fa870000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2836e0a5b4d24ce3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.582+0000 7fa85b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:49:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.582+0000 7fa85b7fe700 1 -- 172.21.15.114:0/3164087580 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870028070 con 0x55908d0b9000 2024-05-27T07:49:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.582+0000 7fa85b7fe700 1 -- 172.21.15.114:0/3164087580 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7fa8700105d0 con 0x55908d0b9000 2024-05-27T07:49:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.582+0000 7fa890cf9b80 1 -- 172.21.15.114:0/3164087580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8700199f0 con 0x55908d0b9000 2024-05-27T07:49:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.583+0000 7fa85b7fe700 1 -- 172.21.15.114:0/3164087580 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fa870015040 con 0x55908d0b9000 2024-05-27T07:49:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.583+0000 7fa890cf9b80 10 client.27232 fetch_fsmap learned FSMap version 51 2024-05-27T07:49:10.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.583+0000 7fa890cf9b80 1 -- 172.21.15.114:0/3164087580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55908d13ea00 con 0x55908d0b9000 2024-05-27T07:49:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.583+0000 7fa85b7fe700 1 -- 172.21.15.114:0/3164087580 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa87002a070 con 0x55908d0b9000 2024-05-27T07:49:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.583+0000 7fa85b7fe700 1 -- 172.21.15.114:0/3164087580 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 51) ==== 7120+0+0 (secure 0 0 0) 0x7fa870024720 con 0x55908d0b9000 2024-05-27T07:49:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.583+0000 7fa890cf9b80 10 client.27232 fetch_fsmap finished waiting for FSMap version 51 2024-05-27T07:49:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.583+0000 7fa890cf9b80 10 client.27232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.583+0000 7fa890cf9b80 20 client.27232 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.583+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55908d0d2870 0x55908d140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.584+0000 7fa890cf9b80 4 client.27232 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:49:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.584+0000 7fa890cf9b80 1 -- 172.21.15.114:0/3164087580 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fa8700199f0 con 0x55908d0d2870 2024-05-27T07:49:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.584+0000 7fa87bfff700 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55908d0d2870 0x55908d140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.585+0000 7fa87bfff700 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55908d0d2870 0x55908d140ff0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=873cf5e4c97557d0 server_cookie=d9c9b760ad2914ed in_seq=0 out_seq=0 2024-05-27T07:49:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.585+0000 7fa85b7fe700 10 client.27232 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.586+0000 7fa85b7fe700 1 -- 172.21.15.114:0/3164087580 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fa8700199f0 con 0x55908d0d2870 2024-05-27T07:49:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.586+0000 7fa85b7fe700 10 client.27232 handle_command_reply: tid=0 2024-05-27T07:49:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.586+0000 7fa890cf9b80 1 client.27232 shutdown 2024-05-27T07:49:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.587+0000 7fa890cf9b80 1 -- 172.21.15.114:0/3164087580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908d0b9000 msgr2=0x55908d0c7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:10.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.587+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908d0b9000 0x55908d0c7ec0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa870009e60 tx=0x7fa870000bc0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.587+0000 7fa890cf9b80 1 -- 172.21.15.114:0/3164087580 shutdown_connections 2024-05-27T07:49:10.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.587+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55908d0d2870 0x55908d140ff0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.587+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55908d0becd0 0x55908d13d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.587+0000 7fa85b7fe700 0 client.27232 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:10.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.587+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55908cf3aec0 0x55908d0c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.587+0000 7fa890cf9b80 1 --2- 172.21.15.114:0/3164087580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55908d0b9000 0x55908d0c7ec0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.587+0000 7fa890cf9b80 1 -- 172.21.15.114:0/3164087580 >> 172.21.15.114:0/3164087580 conn(0x55908cf95b70 msgr2=0x55908d0bc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:10.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.588+0000 7fa890cf9b80 1 -- 172.21.15.114:0/3164087580 shutdown_connections 2024-05-27T07:49:10.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.588+0000 7fa890cf9b80 1 -- 172.21.15.114:0/3164087580 wait complete. 2024-05-27T07:49:10.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.588+0000 7fa890cf9b80 20 client.27232 trim_cache size 0 max 16384 2024-05-27T07:49:10.588 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:49:10.588 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:49:10.589 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "a859ea6f-5ab7-4e22-abfa-2a130fd4c099", 2024-05-27T07:49:10.589 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:49:10.589 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:49:10.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.589+0000 7fa8822e4700 1 -- 172.21.15.114:0/3273328326 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa8640b2a70 msgr2=0x7fa8640b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:10.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.589+0000 7fa8822e4700 1 --2- 172.21.15.114:0/3273328326 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa8640b2a70 0x7fa8640b4f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55908d0b9c30 tx=0x7fa86c004750 comp rx=0 tx=0).stop 2024-05-27T07:49:10.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.589+0000 7fa8822e4700 1 -- 172.21.15.114:0/3273328326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa87c10a920 msgr2=0x7fa87c11ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:10.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.589+0000 7fa8822e4700 1 --2- 172.21.15.114:0/3273328326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa87c10a920 0x7fa87c11ac20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa87400d010 tx=0x7fa874004db0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.590+0000 7fa8822e4700 1 -- 172.21.15.114:0/3273328326 shutdown_connections 2024-05-27T07:49:10.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.590+0000 7fa8822e4700 1 --2- 172.21.15.114:0/3273328326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa87c11d920 0x7fa87c1131d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.590+0000 7fa8822e4700 1 --2- 172.21.15.114:0/3273328326 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa8640b2a70 0x7fa8640b4f80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.590+0000 7fa8822e4700 1 --2- 172.21.15.114:0/3273328326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa87c10a920 0x7fa87c11ac20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.590+0000 7fa8822e4700 1 --2- 172.21.15.114:0/3273328326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa87c10b270 0x7fa87c112c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:10.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.590+0000 7fa8822e4700 1 -- 172.21.15.114:0/3273328326 >> 172.21.15.114:0/3273328326 conn(0x7fa87c06f4c0 msgr2=0x7fa87c11b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:10.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.591+0000 7fa8822e4700 1 -- 172.21.15.114:0/3273328326 shutdown_connections 2024-05-27T07:49:10.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:10.591+0000 7fa8822e4700 1 -- 172.21.15.114:0/3273328326 wait complete. 2024-05-27T07:49:10.748 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:10 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:49:10.749 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:10 smithi163 systemd[1]: Stopped Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:49:10.749 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:10 smithi163 systemd[1]: Starting Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:49:11.047 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:10 smithi163 podman[72096]: 2024-05-27 07:49:10.653855573 +0000 UTC m=+0.024576109 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:49:11.047 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:10 smithi163 podman[72096]: 2024-05-27 07:49:10.777965092 +0000 UTC m=+0.148685608 container create 226a6c46c572dc7112dd4542ecc9069ecbfa1f1ce4d58e0f9298d397160d2e3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-27T07:49:11.047 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:11 smithi163 podman[72096]: 2024-05-27 07:49:11.047783965 +0000 UTC m=+0.418504481 container init 226a6c46c572dc7112dd4542ecc9069ecbfa1f1ce4d58e0f9298d397160d2e3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T07:49:11.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 950542 10.80 MB/sec execute 323 sec latency 84.507 ms 2024-05-27T07:49:11.388 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:11 smithi163 podman[72096]: 2024-05-27 07:49:11.050822628 +0000 UTC m=+0.421543157 container start 226a6c46c572dc7112dd4542ecc9069ecbfa1f1ce4d58e0f9298d397160d2e3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-27T07:49:11.388 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:11 smithi163 ceph-mds[72133]: -- 172.21.15.163:0/4122383925 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5603ff20ba40 con 0x5603ff253180 2024-05-27T07:49:11.388 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:11 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: starting mds.h at 2024-05-27T07:49:11.388 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:11 smithi163 bash[72096]: 226a6c46c572dc7112dd4542ecc9069ecbfa1f1ce4d58e0f9298d397160d2e3a 2024-05-27T07:49:11.388 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:11 smithi163 systemd[1]: Started Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:49:11.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:11 smithi163 ceph-mon[36451]: pgmap v307: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 71 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 2.8 MiB/s wr, 232 op/s 2024-05-27T07:49:11.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:11 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:49:11.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:11 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:49:11.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:11 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:49:11.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:11 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:49:11.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:11 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:49:11.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:11 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:49:11.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:11 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:49:11.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:11 smithi163 ceph-mon[36451]: daemon mds.h restarted 2024-05-27T07:49:11.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:11 smithi186 ceph-mon[36647]: pgmap v307: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 71 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 2.8 MiB/s wr, 232 op/s 2024-05-27T07:49:11.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:11 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:49:11.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:11 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:49:11.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:11 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:49:11.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:11 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:49:11.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:11 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:49:11.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:11 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:49:11.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:11 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:49:11.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:11 smithi186 ceph-mon[36647]: daemon mds.h restarted 2024-05-27T07:49:11.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:11 smithi114 ceph-mon[32038]: pgmap v307: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 71 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 2.8 MiB/s wr, 232 op/s 2024-05-27T07:49:11.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:11 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:49:11.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:11 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:49:11.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:11 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:49:11.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:11 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:49:11.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:11 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:49:11.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:11 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:49:11.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:11 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:49:11.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:11 smithi114 ceph-mon[32038]: daemon mds.h restarted 2024-05-27T07:49:12.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 952736 10.81 MB/sec execute 324 sec latency 52.471 ms 2024-05-27T07:49:12.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:12 smithi163 ceph-mon[36451]: osdmap e112: 12 total, 12 up, 12 in 2024-05-27T07:49:12.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:12 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6836/2455682013,v1:172.21.15.163:6837/2455682013] up:boot 2024-05-27T07:49:12.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:12 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 9 up:standby 2024-05-27T07:49:12.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:12 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:49:12.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:12 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:49:12.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:12 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:49:12.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:12 smithi186 ceph-mon[36647]: osdmap e112: 12 total, 12 up, 12 in 2024-05-27T07:49:12.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:12 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6836/2455682013,v1:172.21.15.163:6837/2455682013] up:boot 2024-05-27T07:49:12.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:12 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 9 up:standby 2024-05-27T07:49:12.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:12 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:49:12.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:12 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:49:12.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:12 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:49:12.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:12 smithi114 ceph-mon[32038]: osdmap e112: 12 total, 12 up, 12 in 2024-05-27T07:49:12.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:12 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6836/2455682013,v1:172.21.15.163:6837/2455682013] up:boot 2024-05-27T07:49:12.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:12 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 9 up:standby 2024-05-27T07:49:12.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:12 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:49:12.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:12 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:49:12.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:12 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:49:13.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:49:13.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 954649 10.79 MB/sec execute 325 sec latency 48.908 ms 2024-05-27T07:49:13.355 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:49:13.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:13 smithi114 ceph-mon[32038]: pgmap v309: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 64 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-05-27T07:49:13.975 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:13 smithi163 ceph-mon[36451]: pgmap v309: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 64 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-05-27T07:49:13.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:13 smithi186 ceph-mon[36647]: pgmap v309: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 64 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-05-27T07:49:14.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 956405 10.80 MB/sec execute 326 sec latency 128.169 ms 2024-05-27T07:49:15.203 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f62c6221700 time 2024-05-27T07:49:14.762770+0000 2024-05-27T07:49:15.204 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:49:15.204 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:15.204 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f62d8a66ba6] 2024-05-27T07:49:15.204 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f62d8a66dd1] 2024-05-27T07:49:15.204 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5603fd7cc23d] 2024-05-27T07:49:15.204 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5603fd7b7971] 2024-05-27T07:49:15.204 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 5: (SnapRealm::check_cache() const+0x13e) [0x5603fd7b5fa0] 2024-05-27T07:49:15.205 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5603fd7b6651] 2024-05-27T07:49:15.205 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5603fd74b347] 2024-05-27T07:49:15.205 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5603fd44787e] 2024-05-27T07:49:15.205 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5603fd44cf2f] 2024-05-27T07:49:15.205 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5603fd457969] 2024-05-27T07:49:15.205 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 11: (MDLog::_replay_thread()+0x1f38) [0x5603fd7f048c] 2024-05-27T07:49:15.206 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5603fd7f99cb] 2024-05-27T07:49:15.206 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 13: (Thread::entry_wrapper()+0x43) [0x7f62d8a3572d] 2024-05-27T07:49:15.206 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 14: (Thread::_entry_func(void*)+0xd) [0x7f62d8a35749] 2024-05-27T07:49:15.206 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62d75d91ca] 2024-05-27T07:49:15.206 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 16: clone() 2024-05-27T07:49:15.206 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:14.764+0000 7f62c6221700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f62c6221700 time 2024-05-27T07:49:14.762770+0000 2024-05-27T07:49:15.207 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:49:15.207 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:15.207 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:15.207 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f62d8a66ba6] 2024-05-27T07:49:15.207 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f62d8a66dd1] 2024-05-27T07:49:15.207 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5603fd7cc23d] 2024-05-27T07:49:15.208 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5603fd7b7971] 2024-05-27T07:49:15.208 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 5: (SnapRealm::check_cache() const+0x13e) [0x5603fd7b5fa0] 2024-05-27T07:49:15.208 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5603fd7b6651] 2024-05-27T07:49:15.208 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5603fd74b347] 2024-05-27T07:49:15.208 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5603fd44787e] 2024-05-27T07:49:15.208 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5603fd44cf2f] 2024-05-27T07:49:15.208 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5603fd457969] 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 11: (MDLog::_replay_thread()+0x1f38) [0x5603fd7f048c] 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5603fd7f99cb] 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 13: (Thread::entry_wrapper()+0x43) [0x7f62d8a3572d] 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 14: (Thread::_entry_func(void*)+0xd) [0x7f62d8a35749] 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62d75d91ca] 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 16: clone() 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: *** Caught signal (Aborted) ** 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: in thread 7f62c6221700 thread_name:md_log_replay 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62d75e3d20] 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2: gsignal() 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 3: abort() 2024-05-27T07:49:15.209 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f62d8a66ced] 2024-05-27T07:49:15.210 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f62d8a66dd1] 2024-05-27T07:49:15.210 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5603fd7cc23d] 2024-05-27T07:49:15.211 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5603fd7b7971] 2024-05-27T07:49:15.211 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 8: (SnapRealm::check_cache() const+0x13e) [0x5603fd7b5fa0] 2024-05-27T07:49:15.211 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5603fd7b6651] 2024-05-27T07:49:15.211 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5603fd74b347] 2024-05-27T07:49:15.211 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5603fd44787e] 2024-05-27T07:49:15.211 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5603fd44cf2f] 2024-05-27T07:49:15.211 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5603fd457969] 2024-05-27T07:49:15.211 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 14: (MDLog::_replay_thread()+0x1f38) [0x5603fd7f048c] 2024-05-27T07:49:15.211 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5603fd7f99cb] 2024-05-27T07:49:15.211 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 16: (Thread::entry_wrapper()+0x43) [0x7f62d8a3572d] 2024-05-27T07:49:15.211 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 17: (Thread::_entry_func(void*)+0xd) [0x7f62d8a35749] 2024-05-27T07:49:15.211 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62d75d91ca] 2024-05-27T07:49:15.211 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 19: clone() 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:14.766+0000 7f62c6221700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: in thread 7f62c6221700 thread_name:md_log_replay 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62d75e3d20] 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2: gsignal() 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 3: abort() 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f62d8a66ced] 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f62d8a66dd1] 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5603fd7cc23d] 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5603fd7b7971] 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 8: (SnapRealm::check_cache() const+0x13e) [0x5603fd7b5fa0] 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5603fd7b6651] 2024-05-27T07:49:15.212 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5603fd74b347] 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5603fd44787e] 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5603fd44cf2f] 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5603fd457969] 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 14: (MDLog::_replay_thread()+0x1f38) [0x5603fd7f048c] 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5603fd7f99cb] 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 16: (Thread::entry_wrapper()+0x43) [0x7f62d8a3572d] 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 17: (Thread::_entry_func(void*)+0xd) [0x7f62d8a35749] 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62d75d91ca] 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 19: clone() 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: -1> 2024-05-27T07:49:14.764+0000 7f62c6221700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f62c6221700 time 2024-05-27T07:49:14.762770+0000 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:15.213 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:15.214 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f62d8a66ba6] 2024-05-27T07:49:15.214 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f62d8a66dd1] 2024-05-27T07:49:15.214 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5603fd7cc23d] 2024-05-27T07:49:15.214 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5603fd7b7971] 2024-05-27T07:49:15.214 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 5: (SnapRealm::check_cache() const+0x13e) [0x5603fd7b5fa0] 2024-05-27T07:49:15.214 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5603fd7b6651] 2024-05-27T07:49:15.214 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5603fd74b347] 2024-05-27T07:49:15.214 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5603fd44787e] 2024-05-27T07:49:15.214 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5603fd44cf2f] 2024-05-27T07:49:15.214 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5603fd457969] 2024-05-27T07:49:15.214 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 11: (MDLog::_replay_thread()+0x1f38) [0x5603fd7f048c] 2024-05-27T07:49:15.214 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5603fd7f99cb] 2024-05-27T07:49:15.214 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 13: (Thread::entry_wrapper()+0x43) [0x7f62d8a3572d] 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 14: (Thread::_entry_func(void*)+0xd) [0x7f62d8a35749] 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62d75d91ca] 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 16: clone() 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 0> 2024-05-27T07:49:14.766+0000 7f62c6221700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: in thread 7f62c6221700 thread_name:md_log_replay 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62d75e3d20] 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2: gsignal() 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 3: abort() 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f62d8a66ced] 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f62d8a66dd1] 2024-05-27T07:49:15.215 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5603fd7cc23d] 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5603fd7b7971] 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 8: (SnapRealm::check_cache() const+0x13e) [0x5603fd7b5fa0] 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5603fd7b6651] 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5603fd74b347] 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5603fd44787e] 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5603fd44cf2f] 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5603fd457969] 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 14: (MDLog::_replay_thread()+0x1f38) [0x5603fd7f048c] 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5603fd7f99cb] 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 16: (Thread::entry_wrapper()+0x43) [0x7f62d8a3572d] 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 17: (Thread::_entry_func(void*)+0xd) [0x7f62d8a35749] 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62d75d91ca] 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 19: clone() 2024-05-27T07:49:15.216 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:49:15.217 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:15.217 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: -1> 2024-05-27T07:49:14.764+0000 7f62c6221700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f62c6221700 time 2024-05-27T07:49:14.762770+0000 2024-05-27T07:49:15.217 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:49:15.217 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:15.217 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:15.217 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f62d8a66ba6] 2024-05-27T07:49:15.217 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f62d8a66dd1] 2024-05-27T07:49:15.217 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5603fd7cc23d] 2024-05-27T07:49:15.217 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5603fd7b7971] 2024-05-27T07:49:15.217 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 5: (SnapRealm::check_cache() const+0x13e) [0x5603fd7b5fa0] 2024-05-27T07:49:15.217 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5603fd7b6651] 2024-05-27T07:49:15.217 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5603fd74b347] 2024-05-27T07:49:15.217 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5603fd44787e] 2024-05-27T07:49:15.218 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5603fd44cf2f] 2024-05-27T07:49:15.218 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5603fd457969] 2024-05-27T07:49:15.218 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 11: (MDLog::_replay_thread()+0x1f38) [0x5603fd7f048c] 2024-05-27T07:49:15.218 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5603fd7f99cb] 2024-05-27T07:49:15.218 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 13: (Thread::entry_wrapper()+0x43) [0x7f62d8a3572d] 2024-05-27T07:49:15.218 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 14: (Thread::_entry_func(void*)+0xd) [0x7f62d8a35749] 2024-05-27T07:49:15.218 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62d75d91ca] 2024-05-27T07:49:15.218 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 16: clone() 2024-05-27T07:49:15.218 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:15.218 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 0> 2024-05-27T07:49:14.766+0000 7f62c6221700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: in thread 7f62c6221700 thread_name:md_log_replay 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62d75e3d20] 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2: gsignal() 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 3: abort() 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f62d8a66ced] 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f62d8a66dd1] 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5603fd7cc23d] 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5603fd7b7971] 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 8: (SnapRealm::check_cache() const+0x13e) [0x5603fd7b5fa0] 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5603fd7b6651] 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5603fd74b347] 2024-05-27T07:49:15.219 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5603fd44787e] 2024-05-27T07:49:15.220 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5603fd44cf2f] 2024-05-27T07:49:15.220 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5603fd457969] 2024-05-27T07:49:15.220 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 14: (MDLog::_replay_thread()+0x1f38) [0x5603fd7f048c] 2024-05-27T07:49:15.220 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5603fd7f99cb] 2024-05-27T07:49:15.220 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 16: (Thread::entry_wrapper()+0x43) [0x7f62d8a3572d] 2024-05-27T07:49:15.220 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 17: (Thread::_entry_func(void*)+0xd) [0x7f62d8a35749] 2024-05-27T07:49:15.220 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62d75d91ca] 2024-05-27T07:49:15.220 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 19: clone() 2024-05-27T07:49:15.220 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:49:15.220 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72129]: 2024-05-27T07:49:15.220 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:14 smithi163 podman[72179]: 2024-05-27 07:49:14.824108439 +0000 UTC m=+0.018854465 container died 226a6c46c572dc7112dd4542ecc9069ecbfa1f1ce4d58e0f9298d397160d2e3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-27T07:49:15.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 958105 10.79 MB/sec execute 327 sec latency 51.254 ms 2024-05-27T07:49:15.475 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:15 smithi163 ceph-mon[36451]: osdmap e113: 12 total, 12 up, 12 in 2024-05-27T07:49:15.475 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:15 smithi163 ceph-mon[36451]: pgmap v311: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 64 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 MiB/s wr, 211 op/s 2024-05-27T07:49:15.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:15 smithi114 ceph-mon[32038]: osdmap e113: 12 total, 12 up, 12 in 2024-05-27T07:49:15.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:15 smithi114 ceph-mon[32038]: pgmap v311: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 64 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 MiB/s wr, 211 op/s 2024-05-27T07:49:15.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:15 smithi186 ceph-mon[36647]: osdmap e113: 12 total, 12 up, 12 in 2024-05-27T07:49:15.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:15 smithi186 ceph-mon[36647]: pgmap v311: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 64 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 MiB/s wr, 211 op/s 2024-05-27T07:49:15.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.806+0000 7f4d822e7700 1 -- 172.21.15.114:0/724690353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d7c1034d0 msgr2=0x7f4d7c1038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:15.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.806+0000 7f4d822e7700 1 --2- 172.21.15.114:0/724690353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d7c1034d0 0x7f4d7c1038e0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0099c0 tx=0x7f4d6c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.806+0000 7f4d822e7700 1 -- 172.21.15.114:0/724690353 shutdown_connections 2024-05-27T07:49:15.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.806+0000 7f4d822e7700 1 --2- 172.21.15.114:0/724690353 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d7c108750 0x7f4d7c1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.806+0000 7f4d822e7700 1 --2- 172.21.15.114:0/724690353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d7c1034d0 0x7f4d7c1038e0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.806+0000 7f4d822e7700 1 --2- 172.21.15.114:0/724690353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d7c107e00 0x7f4d7c108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.806+0000 7f4d822e7700 1 -- 172.21.15.114:0/724690353 >> 172.21.15.114:0/724690353 conn(0x7f4d7c0fef20 msgr2=0x7f4d7c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:15.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.807+0000 7f4d822e7700 1 -- 172.21.15.114:0/724690353 shutdown_connections 2024-05-27T07:49:15.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.807+0000 7f4d822e7700 1 -- 172.21.15.114:0/724690353 wait complete. 2024-05-27T07:49:15.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.807+0000 7f4d822e7700 1 Processor -- start 2024-05-27T07:49:15.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.807+0000 7f4d822e7700 1 -- start start 2024-05-27T07:49:15.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d822e7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d7c1034d0 0x7f4d7c19bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d822e7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d7c107e00 0x7f4d7c19c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d822e7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d7c108750 0x7f4d7c19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d822e7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c113490 con 0x7f4d7c108750 2024-05-27T07:49:15.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d822e7700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c113310 con 0x7f4d7c1034d0 2024-05-27T07:49:15.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d822e7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c113610 con 0x7f4d7c107e00 2024-05-27T07:49:15.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d80884700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d7c108750 0x7f4d7c19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d80884700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d7c108750 0x7f4d7c19c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59238/0 (socket says 172.21.15.114:59238) 2024-05-27T07:49:15.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d80884700 1 -- 172.21.15.114:0/3967107059 learned_addr learned my addr 172.21.15.114:0/3967107059 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:15.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d7bfff700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d7c1034d0 0x7f4d7c19bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d73fff700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d7c107e00 0x7f4d7c19c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d80884700 1 -- 172.21.15.114:0/3967107059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d7c107e00 msgr2=0x7f4d7c19c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:15.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d80884700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d7c107e00 0x7f4d7c19c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d80884700 1 -- 172.21.15.114:0/3967107059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d7c1034d0 msgr2=0x7f4d7c19bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:15.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d80884700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d7c1034d0 0x7f4d7c19bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d80884700 1 -- 172.21.15.114:0/3967107059 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7c1a7140 con 0x7f4d7c108750 2024-05-27T07:49:15.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d80884700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d7c108750 0x7f4d7c19c890 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400b780 tx=0x7f4d7400ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:15.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d79ffb700 1 -- 172.21.15.114:0/3967107059 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d74015020 con 0x7f4d7c108750 2024-05-27T07:49:15.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d79ffb700 1 -- 172.21.15.114:0/3967107059 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d74011cd0 con 0x7f4d7c108750 2024-05-27T07:49:15.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.808+0000 7f4d822e7700 1 -- 172.21.15.114:0/3967107059 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d7c1a7430 con 0x7f4d7c108750 2024-05-27T07:49:15.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.809+0000 7f4d79ffb700 1 -- 172.21.15.114:0/3967107059 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d74010dd0 con 0x7f4d7c108750 2024-05-27T07:49:15.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.809+0000 7f4d822e7700 1 -- 172.21.15.114:0/3967107059 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d7c1a7970 con 0x7f4d7c108750 2024-05-27T07:49:15.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.812+0000 7f4d79ffb700 1 -- 172.21.15.114:0/3967107059 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f4d74018020 con 0x7f4d7c108750 2024-05-27T07:49:15.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.812+0000 7f4d90cfcb80 1 Processor -- start 2024-05-27T07:49:15.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.812+0000 7f4d90cfcb80 1 -- start start 2024-05-27T07:49:15.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d90cfcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 0x55a49e62bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d90cfcb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e62c070 0x55a49e625be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d90cfcb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e626120 0x55a49e626530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d90cfcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a49e553d70 con 0x55a49e4a6e40 2024-05-27T07:49:15.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d90cfcb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a49e543910 con 0x55a49e626120 2024-05-27T07:49:15.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d90cfcb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a49e547950 con 0x55a49e62c070 2024-05-27T07:49:15.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d73fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 0x55a49e62bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d73fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 0x55a49e62bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59240/0 (socket says 172.21.15.114:59240) 2024-05-27T07:49:15.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d73fff700 1 -- 172.21.15.114:0/2889737617 learned_addr learned my addr 172.21.15.114:0/2889737617 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:15.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d7bfff700 1 --2- 172.21.15.114:0/2889737617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e62c070 0x55a49e625be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d73fff700 1 --2- 172.21.15.114:0/2889737617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e626120 0x55a49e626530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d79ffb700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4d680b2a70 0x7f4d680b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d7bfff700 1 -- 172.21.15.114:0/2889737617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e626120 msgr2=0x55a49e626530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:15.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d7bfff700 1 --2- 172.21.15.114:0/2889737617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e626120 0x55a49e626530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d7bfff700 1 -- 172.21.15.114:0/2889737617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 msgr2=0x55a49e62bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:15.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d7bfff700 1 --2- 172.21.15.114:0/2889737617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 0x55a49e62bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d7bfff700 1 -- 172.21.15.114:0/2889737617 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a49e4ecfb0 con 0x55a49e62c070 2024-05-27T07:49:15.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d73fff700 1 --2- 172.21.15.114:0/2889737617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 0x55a49e62bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:15.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d80884700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4d680b2a70 0x7f4d680b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.813+0000 7f4d79ffb700 1 -- 172.21.15.114:0/3967107059 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9566+0+0 (secure 0 0 0) 0x7f4d740e25e0 con 0x7f4d7c108750 2024-05-27T07:49:15.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.814+0000 7f4d7bfff700 1 --2- 172.21.15.114:0/2889737617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e62c070 0x55a49e625be0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0059d0 tx=0x7f4d6c005960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb75085f13201795 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:15.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.814+0000 7f4d80884700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4d680b2a70 0x7f4d680b4f80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4d740df9d0 tx=0x7f4d74077080 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:15.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.814+0000 7f4d727fc700 1 -- 172.21.15.114:0/2889737617 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c003ad0 con 0x55a49e62c070 2024-05-27T07:49:15.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.814+0000 7f4d727fc700 1 -- 172.21.15.114:0/2889737617 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d6c00bd40 con 0x55a49e62c070 2024-05-27T07:49:15.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.814+0000 7f4d727fc700 1 -- 172.21.15.114:0/2889737617 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c005550 con 0x55a49e62c070 2024-05-27T07:49:15.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.815+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/2889737617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e62c070 msgr2=0x55a49e625be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:15.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.815+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/2889737617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e62c070 0x55a49e625be0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0059d0 tx=0x7f4d6c005960 comp rx=0 tx=0).stop 2024-05-27T07:49:15.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.815+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/2889737617 shutdown_connections 2024-05-27T07:49:15.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.815+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/2889737617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e626120 0x55a49e626530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.815+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/2889737617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 0x55a49e62bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.815+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/2889737617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e62c070 0x55a49e625be0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.815+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/2889737617 >> 172.21.15.114:0/2889737617 conn(0x55a49e4a9770 msgr2=0x55a49e620d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:15.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.815+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/2889737617 shutdown_connections 2024-05-27T07:49:15.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.815+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/2889737617 wait complete. 2024-05-27T07:49:15.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.815+0000 7f4d90cfcb80 1 -- start start 2024-05-27T07:49:15.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d90cfcb80 1 Processor -- start 2024-05-27T07:49:15.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d90cfcb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e4a6e40 0x55a49e62f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d90cfcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e626120 0x55a49e62fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d90cfcb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e62c070 0x55a49e630340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d90cfcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a49e449fa0 con 0x55a49e626120 2024-05-27T07:49:15.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d90cfcb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a49e4abda0 con 0x55a49e4a6e40 2024-05-27T07:49:15.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d90cfcb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a49e632b00 con 0x55a49e62c070 2024-05-27T07:49:15.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d90cfcb80 10 client.0 _ll_register_callbacks cb 0x55a49e4da5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d7bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e626120 0x55a49e62fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d7bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e626120 0x55a49e62fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59244/0 (socket says 172.21.15.114:59244) 2024-05-27T07:49:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d7bfff700 1 -- 172.21.15.114:0/2426594767 learned_addr learned my addr 172.21.15.114:0/2426594767 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d73fff700 1 --2- 172.21.15.114:0/2426594767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e4a6e40 0x55a49e62f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d73fff700 1 --2- 172.21.15.114:0/2426594767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e62c070 0x55a49e630340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d7bfff700 1 -- 172.21.15.114:0/2426594767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e62c070 msgr2=0x55a49e630340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d7bfff700 1 --2- 172.21.15.114:0/2426594767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e62c070 0x55a49e630340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d7bfff700 1 -- 172.21.15.114:0/2426594767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e4a6e40 msgr2=0x55a49e62f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d7bfff700 1 --2- 172.21.15.114:0/2426594767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e4a6e40 0x55a49e62f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d7bfff700 1 -- 172.21.15.114:0/2426594767 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a49e4938d0 con 0x55a49e626120 2024-05-27T07:49:15.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d7bfff700 1 --2- 172.21.15.114:0/2426594767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e626120 0x55a49e62fe00 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c000c00 tx=0x7f4d6c003b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f3ec75d2831862 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:15.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.816+0000 7f4d5b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:49:15.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d5b7fe700 1 -- 172.21.15.114:0/2426594767 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c031020 con 0x55a49e626120 2024-05-27T07:49:15.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d5b7fe700 1 -- 172.21.15.114:0/2426594767 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9566+0+0 (secure 0 0 0) 0x7f4d6c0051b0 con 0x55a49e626120 2024-05-27T07:49:15.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/2426594767 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d6c00bd40 con 0x55a49e626120 2024-05-27T07:49:15.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d5b7fe700 1 -- 172.21.15.114:0/2426594767 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f4d6c03c050 con 0x55a49e626120 2024-05-27T07:49:15.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d90cfcb80 10 client.17670 fetch_fsmap learned FSMap version 53 2024-05-27T07:49:15.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/2426594767 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55a49e4fd760 con 0x55a49e626120 2024-05-27T07:49:15.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d5b7fe700 1 -- 172.21.15.114:0/2426594767 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d6c00bd40 con 0x55a49e626120 2024-05-27T07:49:15.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d5b7fe700 1 -- 172.21.15.114:0/2426594767 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 53) ==== 7120+0+0 (secure 0 0 0) 0x7f4d6c021070 con 0x55a49e626120 2024-05-27T07:49:15.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d90cfcb80 10 client.17670 fetch_fsmap finished waiting for FSMap version 53 2024-05-27T07:49:15.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d90cfcb80 10 client.17670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:15.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d90cfcb80 20 client.17670 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:15.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/2426594767 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a49e63f900 0x55a49e641d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d90cfcb80 4 client.17670 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:49:15.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.817+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/2426594767 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a49e555cc0 con 0x55a49e63f900 2024-05-27T07:49:15.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.818+0000 7f4d73fff700 1 --2- 172.21.15.114:0/2426594767 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a49e63f900 0x55a49e641d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.818+0000 7f4d5b7fe700 10 client.17670 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:15.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.818+0000 7f4d73fff700 1 --2- 172.21.15.114:0/2426594767 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a49e63f900 0x55a49e641d20 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34cd9bbafa206e84 server_cookie=21aecb7f2ea9758f in_seq=0 out_seq=0 2024-05-27T07:49:15.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.819+0000 7f4d5b7fe700 1 -- 172.21.15.114:0/2426594767 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a49e555cc0 con 0x55a49e63f900 2024-05-27T07:49:15.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.819+0000 7f4d5b7fe700 10 client.17670 handle_command_reply: tid=0 2024-05-27T07:49:15.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.819+0000 7f4d90cfcb80 1 client.17670 shutdown 2024-05-27T07:49:15.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.819+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/2426594767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e626120 msgr2=0x55a49e62fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:15.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.819+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/2426594767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e626120 0x55a49e62fe00 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c000c00 tx=0x7f4d6c003b80 comp rx=0 tx=0).stop 2024-05-27T07:49:15.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.819+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/2426594767 shutdown_connections 2024-05-27T07:49:15.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.819+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/2426594767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e62c070 0x55a49e630340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.819+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/2426594767 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a49e63f900 0x55a49e641d20 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.819+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/2426594767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e4a6e40 0x55a49e62f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.819+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/2426594767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e626120 0x55a49e62fe00 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.819+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/2426594767 >> 172.21.15.114:0/2426594767 conn(0x55a49e4a9770 msgr2=0x55a49e624bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:15.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.819+0000 7f4d5b7fe700 0 client.17670 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:15.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.819+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/2426594767 shutdown_connections 2024-05-27T07:49:15.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.820+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/2426594767 wait complete. 2024-05-27T07:49:15.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.820+0000 7f4d90cfcb80 20 client.17670 trim_cache size 0 max 16384 2024-05-27T07:49:15.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.840+0000 7f4d90cfcb80 1 Processor -- start 2024-05-27T07:49:15.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.840+0000 7f4d90cfcb80 1 -- start start 2024-05-27T07:49:15.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.840+0000 7f4d90cfcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 0x55a49e625be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.840+0000 7f4d90cfcb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e626120 0x55a49e6302b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.840+0000 7f4d90cfcb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e62c070 0x55a49e63bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.840+0000 7f4d90cfcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c1b06e0 con 0x55a49e4a6e40 2024-05-27T07:49:15.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.840+0000 7f4d90cfcb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c1b0db0 con 0x55a49e62c070 2024-05-27T07:49:15.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.840+0000 7f4d90cfcb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c1ae530 con 0x55a49e626120 2024-05-27T07:49:15.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d73fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 0x55a49e625be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d73fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 0x55a49e625be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59260/0 (socket says 172.21.15.114:59260) 2024-05-27T07:49:15.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d73fff700 1 -- 172.21.15.114:0/1015710033 learned_addr learned my addr 172.21.15.114:0/1015710033 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:15.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d7bfff700 1 --2- 172.21.15.114:0/1015710033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e626120 0x55a49e6302b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d73fff700 1 -- 172.21.15.114:0/1015710033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e626120 msgr2=0x55a49e6302b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:15.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d73fff700 1 --2- 172.21.15.114:0/1015710033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e626120 0x55a49e6302b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d73fff700 1 -- 172.21.15.114:0/1015710033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e62c070 msgr2=0x55a49e63bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:15.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d73fff700 1 --2- 172.21.15.114:0/1015710033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e62c070 0x55a49e63bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d73fff700 1 -- 172.21.15.114:0/1015710033 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a49e62f690 con 0x55a49e4a6e40 2024-05-27T07:49:15.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d7bfff700 1 --2- 172.21.15.114:0/1015710033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e626120 0x55a49e6302b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:49:15.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d73fff700 1 --2- 172.21.15.114:0/1015710033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 0x55a49e625be0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4d64006a90 tx=0x7f4d640047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993790d0392cf67a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:15.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d71ffb700 1 -- 172.21.15.114:0/1015710033 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6402a070 con 0x55a49e4a6e40 2024-05-27T07:49:15.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d71ffb700 1 -- 172.21.15.114:0/1015710033 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d64025410 con 0x55a49e4a6e40 2024-05-27T07:49:15.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.841+0000 7f4d71ffb700 1 -- 172.21.15.114:0/1015710033 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d640165b0 con 0x55a49e4a6e40 2024-05-27T07:49:15.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.842+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/1015710033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 msgr2=0x55a49e625be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:15.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.842+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/1015710033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 0x55a49e625be0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4d64006a90 tx=0x7f4d640047e0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.842+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/1015710033 shutdown_connections 2024-05-27T07:49:15.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.842+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/1015710033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e62c070 0x55a49e63bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.842+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/1015710033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e4a6e40 0x55a49e625be0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.842+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/1015710033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e626120 0x55a49e6302b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.842+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/1015710033 >> 172.21.15.114:0/1015710033 conn(0x55a49e4a9770 msgr2=0x55a49e62e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:15.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.842+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/1015710033 shutdown_connections 2024-05-27T07:49:15.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.842+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/1015710033 wait complete. 2024-05-27T07:49:15.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d90cfcb80 1 -- start start 2024-05-27T07:49:15.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d90cfcb80 1 Processor -- start 2024-05-27T07:49:15.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d90cfcb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e4a6e40 0x55a49e646f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d90cfcb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e626120 0x55a49e647460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d90cfcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e62c070 0x55a49e6479a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d90cfcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c1aeba0 con 0x55a49e62c070 2024-05-27T07:49:15.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d90cfcb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c1b0010 con 0x55a49e4a6e40 2024-05-27T07:49:15.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d90cfcb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c1af270 con 0x55a49e626120 2024-05-27T07:49:15.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d90cfcb80 10 client.0 _ll_register_callbacks cb 0x55a49e4da5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:15.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d73fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e62c070 0x55a49e6479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d7bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e626120 0x55a49e647460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d73fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e62c070 0x55a49e6479a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59274/0 (socket says 172.21.15.114:59274) 2024-05-27T07:49:15.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d73fff700 1 -- 172.21.15.114:0/1375771274 learned_addr learned my addr 172.21.15.114:0/1375771274 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:15.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.843+0000 7f4d73fff700 1 --2- 172.21.15.114:0/1375771274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e4a6e40 0x55a49e646f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d7bfff700 1 -- 172.21.15.114:0/1375771274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e4a6e40 msgr2=0x55a49e646f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:15.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d7bfff700 1 --2- 172.21.15.114:0/1375771274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e4a6e40 0x55a49e646f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d7bfff700 1 -- 172.21.15.114:0/1375771274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e62c070 msgr2=0x55a49e6479a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d7bfff700 1 --2- 172.21.15.114:0/1375771274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e62c070 0x55a49e6479a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d7bfff700 1 -- 172.21.15.114:0/1375771274 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a49e6a9250 con 0x55a49e626120 2024-05-27T07:49:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d73fff700 1 --2- 172.21.15.114:0/1375771274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e62c070 0x55a49e6479a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d7bfff700 1 --2- 172.21.15.114:0/1375771274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e626120 0x55a49e647460 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55a49e6309c0 tx=0x7f4d6c003ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44636516a17708c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d5b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:49:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d5b7fe700 1 -- 172.21.15.114:0/1375771274 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c031040 con 0x55a49e626120 2024-05-27T07:49:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d5b7fe700 1 -- 172.21.15.114:0/1375771274 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9566+0+0 (secure 0 0 0) 0x7f4d6c00bd20 con 0x55a49e626120 2024-05-27T07:49:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/1375771274 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d64025410 con 0x55a49e626120 2024-05-27T07:49:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d5b7fe700 1 -- 172.21.15.114:0/1375771274 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f4d6c028040 con 0x55a49e626120 2024-05-27T07:49:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d90cfcb80 10 client.27250 fetch_fsmap learned FSMap version 53 2024-05-27T07:49:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/1375771274 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55a49e6aa420 con 0x55a49e626120 2024-05-27T07:49:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.844+0000 7f4d5b7fe700 1 -- 172.21.15.114:0/1375771274 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d64025410 con 0x55a49e626120 2024-05-27T07:49:15.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.845+0000 7f4d5b7fe700 1 -- 172.21.15.114:0/1375771274 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 53) ==== 7120+0+0 (secure 0 0 0) 0x7f4d6c021070 con 0x55a49e626120 2024-05-27T07:49:15.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.845+0000 7f4d90cfcb80 10 client.27250 fetch_fsmap finished waiting for FSMap version 53 2024-05-27T07:49:15.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.845+0000 7f4d90cfcb80 10 client.27250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:15.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.845+0000 7f4d90cfcb80 20 client.27250 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:15.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.845+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/1375771274 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a49e6aaa40 0x55a49e6ace60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:15.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.845+0000 7f4d90cfcb80 4 client.27250 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:49:15.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.845+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/1375771274 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a49e54f4a0 con 0x55a49e6aaa40 2024-05-27T07:49:15.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.845+0000 7f4d73fff700 1 --2- 172.21.15.114:0/1375771274 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a49e6aaa40 0x55a49e6ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:15.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.845+0000 7f4d5b7fe700 10 client.27250 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:15.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.845+0000 7f4d73fff700 1 --2- 172.21.15.114:0/1375771274 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a49e6aaa40 0x55a49e6ace60 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77f62f82a77b882a server_cookie=cfd7ed7c92f4c12c in_seq=0 out_seq=0 2024-05-27T07:49:15.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.845+0000 7f4d5b7fe700 1 -- 172.21.15.114:0/1375771274 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a49e54f4a0 con 0x55a49e6aaa40 2024-05-27T07:49:15.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.845+0000 7f4d5b7fe700 10 client.27250 handle_command_reply: tid=0 2024-05-27T07:49:15.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d90cfcb80 1 client.27250 shutdown 2024-05-27T07:49:15.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/1375771274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e626120 msgr2=0x55a49e647460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:15.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/1375771274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e626120 0x55a49e647460 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55a49e6309c0 tx=0x7f4d6c003ea0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/1375771274 shutdown_connections 2024-05-27T07:49:15.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/1375771274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a49e62c070 0x55a49e6479a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/1375771274 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a49e6aaa40 0x55a49e6ace60 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/1375771274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a49e4a6e40 0x55a49e646f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d90cfcb80 1 --2- 172.21.15.114:0/1375771274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a49e626120 0x55a49e647460 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d5b7fe700 0 client.27250 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:15.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/1375771274 >> 172.21.15.114:0/1375771274 conn(0x55a49e4a9770 msgr2=0x55a49e630010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:15.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/1375771274 shutdown_connections 2024-05-27T07:49:15.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d90cfcb80 1 -- 172.21.15.114:0/1375771274 wait complete. 2024-05-27T07:49:15.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d90cfcb80 20 client.27250 trim_cache size 0 max 16384 2024-05-27T07:49:15.853 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:49:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:49:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:49:15.853 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:49:15.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d822e7700 1 -- 172.21.15.114:0/3967107059 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4d680b2a70 msgr2=0x7f4d680b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:15.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d822e7700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4d680b2a70 0x7f4d680b4f80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4d740df9d0 tx=0x7f4d74077080 comp rx=0 tx=0).stop 2024-05-27T07:49:15.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d822e7700 1 -- 172.21.15.114:0/3967107059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d7c108750 msgr2=0x7f4d7c19c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:15.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.846+0000 7f4d822e7700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d7c108750 0x7f4d7c19c890 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400b780 tx=0x7f4d7400ba90 comp rx=0 tx=0).stop 2024-05-27T07:49:15.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.847+0000 7f4d822e7700 1 -- 172.21.15.114:0/3967107059 shutdown_connections 2024-05-27T07:49:15.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.847+0000 7f4d822e7700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d7c108750 0x7f4d7c19c890 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.847+0000 7f4d822e7700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4d680b2a70 0x7f4d680b4f80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.847+0000 7f4d822e7700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d7c1034d0 0x7f4d7c19bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.847+0000 7f4d822e7700 1 --2- 172.21.15.114:0/3967107059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d7c107e00 0x7f4d7c19c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:15.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.847+0000 7f4d822e7700 1 -- 172.21.15.114:0/3967107059 >> 172.21.15.114:0/3967107059 conn(0x7f4d7c0fef20 msgr2=0x7f4d7c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:15.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.847+0000 7f4d822e7700 1 -- 172.21.15.114:0/3967107059 shutdown_connections 2024-05-27T07:49:15.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:15.847+0000 7f4d822e7700 1 -- 172.21.15.114:0/3967107059 wait complete. 2024-05-27T07:49:15.975 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:15 smithi163 podman[72179]: 2024-05-27 07:49:15.606036256 +0000 UTC m=+0.800782257 container remove 226a6c46c572dc7112dd4542ecc9069ecbfa1f1ce4d58e0f9298d397160d2e3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-05-27T07:49:15.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:15 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:49:15.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:15 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-05-27T07:49:16.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 959706 10.79 MB/sec execute 328 sec latency 203.744 ms 2024-05-27T07:49:16.534 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:49:16.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:49:16.790 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:49:17.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 961485 10.78 MB/sec execute 329 sec latency 32.761 ms 2024-05-27T07:49:17.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:17 smithi114 ceph-mon[32038]: pgmap v312: 129 pgs: 129 active+clean; 69 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 3.4 MiB/s wr, 350 op/s 2024-05-27T07:49:17.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:17 smithi163 ceph-mon[36451]: pgmap v312: 129 pgs: 129 active+clean; 69 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 3.4 MiB/s wr, 350 op/s 2024-05-27T07:49:17.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:17 smithi186 ceph-mon[36647]: pgmap v312: 129 pgs: 129 active+clean; 69 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 3.4 MiB/s wr, 350 op/s 2024-05-27T07:49:18.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 963575 10.78 MB/sec execute 330 sec latency 29.371 ms 2024-05-27T07:49:18.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.503+0000 7f3a50b7c700 1 -- 172.21.15.114:0/3966903096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c1089e0 msgr2=0x7f3a4c108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.503+0000 7f3a50b7c700 1 --2- 172.21.15.114:0/3966903096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c1089e0 0x7f3a4c108df0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00aaa0 tx=0x7f3a3c00adb0 comp rx=0 tx=0).stop 2024-05-27T07:49:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.503+0000 7f3a50b7c700 1 -- 172.21.15.114:0/3966903096 shutdown_connections 2024-05-27T07:49:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.503+0000 7f3a50b7c700 1 --2- 172.21.15.114:0/3966903096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c103300 0x7f3a4c10f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.503+0000 7f3a50b7c700 1 --2- 172.21.15.114:0/3966903096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c1089e0 0x7f3a4c108df0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.503+0000 7f3a50b7c700 1 --2- 172.21.15.114:0/3966903096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c1029b0 0x7f3a4c102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.503+0000 7f3a50b7c700 1 -- 172.21.15.114:0/3966903096 >> 172.21.15.114:0/3966903096 conn(0x7f3a4c0fe6e0 msgr2=0x7f3a4c100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.503+0000 7f3a50b7c700 1 -- 172.21.15.114:0/3966903096 shutdown_connections 2024-05-27T07:49:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.503+0000 7f3a50b7c700 1 -- 172.21.15.114:0/3966903096 wait complete. 2024-05-27T07:49:18.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.504+0000 7f3a50b7c700 1 Processor -- start 2024-05-27T07:49:18.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.504+0000 7f3a50b7c700 1 -- start start 2024-05-27T07:49:18.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a50b7c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c1029b0 0x7f3a4c19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:18.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a50b7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c103300 0x7f3a4c19b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:18.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a50b7c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c1089e0 0x7f3a4c19f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:18.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a50b7c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1122d0 con 0x7f3a4c103300 2024-05-27T07:49:18.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a50b7c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c112150 con 0x7f3a4c1029b0 2024-05-27T07:49:18.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a50b7c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c112450 con 0x7f3a4c1089e0 2024-05-27T07:49:18.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a49d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c103300 0x7f3a4c19b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:18.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a49d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c103300 0x7f3a4c19b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59288/0 (socket says 172.21.15.114:59288) 2024-05-27T07:49:18.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a49d9b700 1 -- 172.21.15.114:0/1889505741 learned_addr learned my addr 172.21.15.114:0/1889505741 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:18.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a4ad9d700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c1089e0 0x7f3a4c19f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:18.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a4a59c700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c1029b0 0x7f3a4c19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:18.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a49d9b700 1 -- 172.21.15.114:0/1889505741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c1089e0 msgr2=0x7f3a4c19f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:18.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a49d9b700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c1089e0 0x7f3a4c19f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:18.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a49d9b700 1 -- 172.21.15.114:0/1889505741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c1029b0 msgr2=0x7f3a4c19ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:18.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a49d9b700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c1029b0 0x7f3a4c19ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:18.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a49d9b700 1 -- 172.21.15.114:0/1889505741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4c19fcf0 con 0x7f3a4c103300 2024-05-27T07:49:18.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.505+0000 7f3a4ad9d700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c1089e0 0x7f3a4c19f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:49:18.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.506+0000 7f3a49d9b700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c103300 0x7f3a4c19b3c0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000e9f0 tx=0x7f3a4000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.506+0000 7f3a3b7fe700 1 -- 172.21.15.114:0/1889505741 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4000cbd0 con 0x7f3a4c103300 2024-05-27T07:49:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.506+0000 7f3a3b7fe700 1 -- 172.21.15.114:0/1889505741 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a4000cd70 con 0x7f3a4c103300 2024-05-27T07:49:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.506+0000 7f3a3b7fe700 1 -- 172.21.15.114:0/1889505741 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a40005650 con 0x7f3a4c103300 2024-05-27T07:49:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.506+0000 7f3a50b7c700 1 -- 172.21.15.114:0/1889505741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4c19ffe0 con 0x7f3a4c103300 2024-05-27T07:49:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.506+0000 7f3a50b7c700 1 -- 172.21.15.114:0/1889505741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a4c1a7900 con 0x7f3a4c103300 2024-05-27T07:49:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.507+0000 7f3a50b7c700 1 -- 172.21.15.114:0/1889505741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a4c0512b0 con 0x7f3a4c103300 2024-05-27T07:49:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.509+0000 7f3a3b7fe700 1 -- 172.21.15.114:0/1889505741 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f3a40004750 con 0x7f3a4c103300 2024-05-27T07:49:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.510+0000 7f3a3b7fe700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3a340b2a70 0x7f3a340b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.510+0000 7f3a4a59c700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3a340b2a70 0x7f3a340b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.510+0000 7f3a3b7fe700 1 -- 172.21.15.114:0/1889505741 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9566+0+0 (secure 0 0 0) 0x7f3a400e2f10 con 0x7f3a4c103300 2024-05-27T07:49:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.510+0000 7f3a4a59c700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3a340b2a70 0x7f3a340b4f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c0059d0 tx=0x7f3a3c005960 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.514+0000 7f3a3b7fe700 1 -- 172.21.15.114:0/1889505741 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3a40054df0 con 0x7f3a4c103300 2024-05-27T07:49:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.802+0000 7f3a50b7c700 1 -- 172.21.15.114:0/1889505741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a4c19c180 con 0x7f3a4c103300 2024-05-27T07:49:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.803+0000 7f3a3b7fe700 1 -- 172.21.15.114:0/1889505741 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+10178 (secure 0 0 0) 0x7f3a400aa4e0 con 0x7f3a4c103300 2024-05-27T07:49:18.803 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:49:18.804 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":26882,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2622386718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2622386718},{"type":"v1","addr":"172.21.15.186:6835","nonce":2622386718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:49:11.540602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27238":{"gid":27238,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/2455682013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":2455682013},{"type":"v1","addr":"172.21.15.163:6837","nonce":2455682013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:49:18.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.805+0000 7f3a50b7c700 1 -- 172.21.15.114:0/1889505741 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3a340b2a70 msgr2=0x7f3a340b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:18.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.805+0000 7f3a50b7c700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3a340b2a70 0x7f3a340b4f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c0059d0 tx=0x7f3a3c005960 comp rx=0 tx=0).stop 2024-05-27T07:49:18.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.805+0000 7f3a50b7c700 1 -- 172.21.15.114:0/1889505741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c103300 msgr2=0x7f3a4c19b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:18.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.805+0000 7f3a50b7c700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c103300 0x7f3a4c19b3c0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000e9f0 tx=0x7f3a4000ed00 comp rx=0 tx=0).stop 2024-05-27T07:49:18.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.805+0000 7f3a50b7c700 1 -- 172.21.15.114:0/1889505741 shutdown_connections 2024-05-27T07:49:18.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.805+0000 7f3a50b7c700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c1089e0 0x7f3a4c19f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:18.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.806+0000 7f3a50b7c700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3a340b2a70 0x7f3a340b4f80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:18.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.806+0000 7f3a50b7c700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c1029b0 0x7f3a4c19ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:18.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.806+0000 7f3a50b7c700 1 --2- 172.21.15.114:0/1889505741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c103300 0x7f3a4c19b3c0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:18.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.806+0000 7f3a50b7c700 1 -- 172.21.15.114:0/1889505741 >> 172.21.15.114:0/1889505741 conn(0x7f3a4c0fe6e0 msgr2=0x7f3a4c100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:18.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.806+0000 7f3a50b7c700 1 -- 172.21.15.114:0/1889505741 shutdown_connections 2024-05-27T07:49:18.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:18.806+0000 7f3a50b7c700 1 -- 172.21.15.114:0/1889505741 wait complete. 2024-05-27T07:49:18.807 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 53 2024-05-27T07:49:19.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 965833 10.79 MB/sec execute 331 sec latency 92.535 ms 2024-05-27T07:49:19.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:19 smithi114 ceph-mon[32038]: pgmap v313: 129 pgs: 129 active+clean; 69 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 3.0 MiB/s wr, 290 op/s 2024-05-27T07:49:19.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:19 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1889505741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:49:19.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:19 smithi114 ceph-mon[32038]: osdmap e114: 12 total, 12 up, 12 in 2024-05-27T07:49:19.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:19 smithi163 ceph-mon[36451]: pgmap v313: 129 pgs: 129 active+clean; 69 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 3.0 MiB/s wr, 290 op/s 2024-05-27T07:49:19.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:19 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1889505741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:49:19.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:19 smithi163 ceph-mon[36451]: osdmap e114: 12 total, 12 up, 12 in 2024-05-27T07:49:19.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:19 smithi186 ceph-mon[36647]: pgmap v313: 129 pgs: 129 active+clean; 69 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 3.0 MiB/s wr, 290 op/s 2024-05-27T07:49:19.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:19 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1889505741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:49:19.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:19 smithi186 ceph-mon[36647]: osdmap e114: 12 total, 12 up, 12 in 2024-05-27T07:49:19.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.987+0000 7f619b7fe640 1 -- 172.21.15.114:0/3808905997 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c031810 con 0x7f619c10a1f0 2024-05-27T07:49:19.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.987+0000 7f61a26f9640 1 -- 172.21.15.114:0/3808905997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f619c10a1f0 msgr2=0x7f619c10a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:19.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.987+0000 7f61a26f9640 1 --2- 172.21.15.114:0/3808905997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f619c10a1f0 0x7f619c10a600 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f618c009900 tx=0x7f618c02a4b0 comp rx=0 tx=0).stop 2024-05-27T07:49:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.989+0000 7f61a26f9640 1 -- 172.21.15.114:0/3808905997 shutdown_connections 2024-05-27T07:49:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.989+0000 7f61a26f9640 1 --2- 172.21.15.114:0/3808905997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c11d240 0x7f619c11f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.989+0000 7f61a26f9640 1 --2- 172.21.15.114:0/3808905997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f619c11a8e0 0x7f619c11cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.989+0000 7f61a26f9640 1 --2- 172.21.15.114:0/3808905997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f619c10a1f0 0x7f619c10a600 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.989+0000 7f61a26f9640 1 -- 172.21.15.114:0/3808905997 >> 172.21.15.114:0/3808905997 conn(0x7f619c06e8f0 msgr2=0x7f619c06ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.990+0000 7f61a26f9640 1 -- 172.21.15.114:0/3808905997 shutdown_connections 2024-05-27T07:49:19.992 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.992+0000 7f61a26f9640 1 -- 172.21.15.114:0/3808905997 wait complete. 2024-05-27T07:49:19.995 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.994+0000 7f61a26f9640 1 Processor -- start 2024-05-27T07:49:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.995+0000 7f61a26f9640 1 -- start start 2024-05-27T07:49:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.995+0000 7f61a26f9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c10a1f0 0x7f619c11a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.995+0000 7f61a26f9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f619c11a8e0 0x7f619c1134b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.995+0000 7f61a26f9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f619c11d240 0x7f619c113b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.995+0000 7f61a26f9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f619c122c90 con 0x7f619c11d240 2024-05-27T07:49:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.995+0000 7f61a26f9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f619c122b10 con 0x7f619c11a8e0 2024-05-27T07:49:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.995+0000 7f61a26f9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f619c122e10 con 0x7f619c10a1f0 2024-05-27T07:49:19.998 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.997+0000 7f619bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c10a1f0 0x7f619c11a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:19.998 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.997+0000 7f619bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c10a1f0 0x7f619c11a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54752/0 (socket says 172.21.15.114:54752) 2024-05-27T07:49:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.997+0000 7f619bfff640 1 -- 172.21.15.114:0/1732940488 learned_addr learned my addr 172.21.15.114:0/1732940488 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.997+0000 7f6193fff640 1 --2- 172.21.15.114:0/1732940488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f619c11a8e0 0x7f619c1134b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.998+0000 7f6193fff640 1 -- 172.21.15.114:0/1732940488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c10a1f0 msgr2=0x7f619c11a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.998+0000 7f6193fff640 1 --2- 172.21.15.114:0/1732940488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c10a1f0 0x7f619c11a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.998+0000 7f6193fff640 1 -- 172.21.15.114:0/1732940488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f619c11d240 msgr2=0x7f619c113b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.998+0000 7f6193fff640 1 --2- 172.21.15.114:0/1732940488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f619c11d240 0x7f619c113b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.998+0000 7f6193fff640 1 -- 172.21.15.114:0/1732940488 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f619c117e90 con 0x7f619c11a8e0 2024-05-27T07:49:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.999+0000 7f6193fff640 1 --2- 172.21.15.114:0/1732940488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f619c11a8e0 0x7f619c1134b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f618400b3f0 tx=0x7f618400b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.999+0000 7f6199ffb640 1 -- 172.21.15.114:0/1732940488 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6184015020 con 0x7f619c11a8e0 2024-05-27T07:49:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.999+0000 7f6199ffb640 1 -- 172.21.15.114:0/1732940488 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6184011ce0 con 0x7f619c11a8e0 2024-05-27T07:49:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.999+0000 7f61a26f9640 1 -- 172.21.15.114:0/1732940488 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f619c118180 con 0x7f619c11a8e0 2024-05-27T07:49:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.999+0000 7f61a26f9640 1 -- 172.21.15.114:0/1732940488 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f619c118720 con 0x7f619c11a8e0 2024-05-27T07:49:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:19.999+0000 7f6199ffb640 1 -- 172.21.15.114:0/1732940488 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6184010de0 con 0x7f619c11a8e0 2024-05-27T07:49:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.003+0000 7f61a95a9740 1 Processor -- start 2024-05-27T07:49:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.003+0000 7f61a95a9740 1 -- start start 2024-05-27T07:49:20.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.004+0000 7f61a95a9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfc61b0 0x55dbbc3673d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.004+0000 7f61a95a9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc367910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.004+0000 7f61a95a9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbfa54d0 0x55dbbc367e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.004+0000 7f61a95a9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55dbbc2d9c30 con 0x55dbbbf9d2f0 2024-05-27T07:49:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.004+0000 7f61a95a9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dbbc2c97d0 con 0x55dbbbfc61b0 2024-05-27T07:49:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.004+0000 7f61a95a9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55dbbc2cd810 con 0x55dbbbfa54d0 2024-05-27T07:49:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.005+0000 7f61a0c96640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfc61b0 0x55dbbc3673d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.005+0000 7f61a0c96640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfc61b0 0x55dbbc3673d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:34978/0 (socket says 172.21.15.114:34978) 2024-05-27T07:49:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.005+0000 7f61a0c96640 1 -- 172.21.15.114:0/3752347466 learned_addr learned my addr 172.21.15.114:0/3752347466 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.005+0000 7f6199ffb640 1 -- 172.21.15.114:0/1732940488 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6184018020 con 0x7f619c11a8e0 2024-05-27T07:49:20.006 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.006+0000 7f6199ffb640 1 --2- 172.21.15.114:0/1732940488 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f61880b2aa0 0x7f61880b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.007+0000 7f61a0c96640 1 -- 172.21.15.114:0/3752347466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbfa54d0 msgr2=0x55dbbc367e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:49:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.007+0000 7f61a0c96640 1 --2- 172.21.15.114:0/3752347466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbfa54d0 0x55dbbc367e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.007+0000 7f61a0c96640 1 -- 172.21.15.114:0/3752347466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf9d2f0 msgr2=0x55dbbc367910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.007+0000 7f61a0c96640 1 --2- 172.21.15.114:0/3752347466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc367910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.007+0000 7f61a0c96640 1 -- 172.21.15.114:0/3752347466 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbbbfde210 con 0x55dbbbfc61b0 2024-05-27T07:49:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.007+0000 7f61a0c96640 1 --2- 172.21.15.114:0/1732940488 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f61880b2aa0 0x7f61880b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.007+0000 7f61a0c96640 1 --2- 172.21.15.114:0/3752347466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfc61b0 0x55dbbc3673d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6194009d80 tx=0x7f619400e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3db92db39a1fe9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.007+0000 7f61927fc640 1 -- 172.21.15.114:0/3752347466 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61940119b0 con 0x55dbbbfc61b0 2024-05-27T07:49:20.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.009+0000 7f61927fc640 1 -- 172.21.15.114:0/3752347466 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f619401e920 con 0x55dbbbfc61b0 2024-05-27T07:49:20.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.009+0000 7f61927fc640 1 -- 172.21.15.114:0/3752347466 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619401db40 con 0x55dbbbfc61b0 2024-05-27T07:49:20.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.009+0000 7f61a0c96640 1 --2- 172.21.15.114:0/1732940488 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f61880b2aa0 0x7f61880b4fb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f619401d000 tx=0x7f61940107a0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:20.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.009+0000 7f6199ffb640 1 -- 172.21.15.114:0/1732940488 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9538+0+0 (secure 0 0 0) 0x7f61840e2040 con 0x7f619c11a8e0 2024-05-27T07:49:20.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.011+0000 7f61a95a9740 1 -- 172.21.15.114:0/3752347466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfc61b0 msgr2=0x55dbbc3673d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:20.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.011+0000 7f61a95a9740 1 --2- 172.21.15.114:0/3752347466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfc61b0 0x55dbbc3673d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6194009d80 tx=0x7f619400e6c0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.011+0000 7f61a95a9740 1 -- 172.21.15.114:0/3752347466 shutdown_connections 2024-05-27T07:49:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.011+0000 7f61a95a9740 1 --2- 172.21.15.114:0/3752347466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbfa54d0 0x55dbbc367e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.011+0000 7f61a95a9740 1 --2- 172.21.15.114:0/3752347466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc367910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.011+0000 7f61a95a9740 1 --2- 172.21.15.114:0/3752347466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfc61b0 0x55dbbc3673d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.011+0000 7f61a95a9740 1 -- 172.21.15.114:0/3752347466 >> 172.21.15.114:0/3752347466 conn(0x55dbbbfba300 msgr2=0x55dbbc363b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.015+0000 7f61a95a9740 1 -- 172.21.15.114:0/3752347466 shutdown_connections 2024-05-27T07:49:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.015+0000 7f61a95a9740 1 -- 172.21.15.114:0/3752347466 wait complete. 2024-05-27T07:49:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.016+0000 7f61a95a9740 1 -- start start 2024-05-27T07:49:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.016+0000 7f61a95a9740 1 Processor -- start 2024-05-27T07:49:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.017+0000 7f61a95a9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc371a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.017+0000 7f61a95a9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbfa54d0 0x55dbbc371fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.017+0000 7f619bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc371a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:20.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.017+0000 7f61a95a9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfc61b0 0x55dbbc3778f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.017+0000 7f619bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc371a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54782/0 (socket says 172.21.15.114:54782) 2024-05-27T07:49:20.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.017+0000 7f6193fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbfa54d0 0x55dbbc371fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:20.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.017+0000 7f619bfff640 1 -- 172.21.15.114:0/2077438375 learned_addr learned my addr 172.21.15.114:0/2077438375 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.018+0000 7f61a95a9740 1 -- 172.21.15.114:0/2077438375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55dbbbfaf790 con 0x55dbbbfa54d0 2024-05-27T07:49:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.018+0000 7f61a95a9740 1 -- 172.21.15.114:0/2077438375 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dbbc285890 con 0x55dbbbfc61b0 2024-05-27T07:49:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.018+0000 7f61a95a9740 1 -- 172.21.15.114:0/2077438375 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55dbbc374550 con 0x55dbbbf9d2f0 2024-05-27T07:49:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.018+0000 7f61a0c96640 1 --2- 172.21.15.114:0/2077438375 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfc61b0 0x55dbbc3778f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.018+0000 7f61a95a9740 10 client.0 _ll_register_callbacks cb 0x55dbbbfdda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:20.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.018+0000 7f6193fff640 1 -- 172.21.15.114:0/2077438375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf9d2f0 msgr2=0x55dbbc371a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:20.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.018+0000 7f6193fff640 1 --2- 172.21.15.114:0/2077438375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc371a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.018+0000 7f6193fff640 1 -- 172.21.15.114:0/2077438375 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfc61b0 msgr2=0x55dbbc3778f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:20.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.018+0000 7f6193fff640 1 --2- 172.21.15.114:0/2077438375 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfc61b0 0x55dbbc3778f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.018+0000 7f6193fff640 1 -- 172.21.15.114:0/2077438375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbbc0aa620 con 0x55dbbbfa54d0 2024-05-27T07:49:20.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.018+0000 7f619bfff640 1 --2- 172.21.15.114:0/2077438375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc371a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:20.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.019+0000 7f6193fff640 1 --2- 172.21.15.114:0/2077438375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbfa54d0 0x55dbbc371fb0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f61840df440 tx=0x7f61840a8740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a0a0268a65c89d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:20.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.019+0000 7f617b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:49:20.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.019+0000 7f617b7fe640 1 -- 172.21.15.114:0/2077438375 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61840df7c0 con 0x55dbbbfa54d0 2024-05-27T07:49:20.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.019+0000 7f617b7fe640 1 -- 172.21.15.114:0/2077438375 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9538+0+0 (secure 0 0 0) 0x7f61840a55c0 con 0x55dbbbfa54d0 2024-05-27T07:49:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.019+0000 7f61a95a9740 1 -- 172.21.15.114:0/2077438375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f619401e920 con 0x55dbbbfa54d0 2024-05-27T07:49:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.019+0000 7f617b7fe640 1 -- 172.21.15.114:0/2077438375 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f61840a8fd0 con 0x55dbbbfa54d0 2024-05-27T07:49:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.019+0000 7f61a95a9740 10 client.17685 fetch_fsmap learned FSMap version 53 2024-05-27T07:49:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.019+0000 7f61a95a9740 1 -- 172.21.15.114:0/2077438375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55dbbc380640 con 0x55dbbbfa54d0 2024-05-27T07:49:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.019+0000 7f617b7fe640 1 -- 172.21.15.114:0/2077438375 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61840ad380 con 0x55dbbbfa54d0 2024-05-27T07:49:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.020+0000 7f617b7fe640 1 -- 172.21.15.114:0/2077438375 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 53) ==== 7120+0+0 (secure 0 0 0) 0x7f61840a4940 con 0x55dbbbfa54d0 2024-05-27T07:49:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.020+0000 7f61a95a9740 10 client.17685 fetch_fsmap finished waiting for FSMap version 53 2024-05-27T07:49:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.020+0000 7f61a95a9740 10 client.17685 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.020+0000 7f61a95a9740 20 client.17685 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.020+0000 7f61a95a9740 1 --2- 172.21.15.114:0/2077438375 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dbbbf84d80 0x55dbbc383bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.020+0000 7f61a95a9740 4 client.17685 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:49:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.020+0000 7f61a95a9740 1 -- 172.21.15.114:0/2077438375 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f619401e920 con 0x55dbbbf84d80 2024-05-27T07:49:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.020+0000 7f619bfff640 1 --2- 172.21.15.114:0/2077438375 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dbbbf84d80 0x55dbbc383bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.022+0000 7f619bfff640 1 --2- 172.21.15.114:0/2077438375 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dbbbf84d80 0x55dbbc383bd0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f51afb05a8582e4 server_cookie=763a06445dfc9ed1 in_seq=0 out_seq=0 2024-05-27T07:49:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.022+0000 7f617b7fe640 10 client.17685 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f617b7fe640 1 -- 172.21.15.114:0/2077438375 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f619401e920 con 0x55dbbbf84d80 2024-05-27T07:49:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f617b7fe640 10 client.17685 handle_command_reply: tid=0 2024-05-27T07:49:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f61a95a9740 1 client.17685 shutdown 2024-05-27T07:49:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f61a95a9740 1 -- 172.21.15.114:0/2077438375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbfa54d0 msgr2=0x55dbbc371fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f61a95a9740 1 --2- 172.21.15.114:0/2077438375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbfa54d0 0x55dbbc371fb0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f61840df440 tx=0x7f61840a8740 comp rx=0 tx=0).stop 2024-05-27T07:49:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f61a95a9740 1 -- 172.21.15.114:0/2077438375 shutdown_connections 2024-05-27T07:49:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f61a95a9740 1 --2- 172.21.15.114:0/2077438375 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dbbbf84d80 0x55dbbc383bd0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f61a95a9740 1 --2- 172.21.15.114:0/2077438375 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfc61b0 0x55dbbc3778f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f61a95a9740 1 --2- 172.21.15.114:0/2077438375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbfa54d0 0x55dbbc371fb0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f617b7fe640 0 client.17685 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f61a95a9740 1 --2- 172.21.15.114:0/2077438375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc371a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f61a95a9740 1 -- 172.21.15.114:0/2077438375 >> 172.21.15.114:0/2077438375 conn(0x55dbbbfba300 msgr2=0x55dbbc3690f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f61a95a9740 1 -- 172.21.15.114:0/2077438375 shutdown_connections 2024-05-27T07:49:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f61a95a9740 1 -- 172.21.15.114:0/2077438375 wait complete. 2024-05-27T07:49:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.023+0000 7f61a95a9740 20 client.17685 trim_cache size 0 max 16384 2024-05-27T07:49:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.039+0000 7f61a95a9740 1 Processor -- start 2024-05-27T07:49:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.039+0000 7f61a95a9740 1 -- start start 2024-05-27T07:49:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.039+0000 7f61a95a9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf84d80 0x55dbbc3777f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.039+0000 7f61a95a9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc36c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.039+0000 7f61a95a9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbfa54d0 0x55dbbc367b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.039+0000 7f61a95a9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f617c010080 con 0x55dbbbf84d80 2024-05-27T07:49:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.039+0000 7f61a95a9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f617c010750 con 0x55dbbbf9d2f0 2024-05-27T07:49:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.039+0000 7f61a95a9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f617c00def0 con 0x55dbbbfa54d0 2024-05-27T07:49:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.039+0000 7f619bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf84d80 0x55dbbc3777f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.039+0000 7f619bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf84d80 0x55dbbc3777f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59326/0 (socket says 172.21.15.114:59326) 2024-05-27T07:49:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.039+0000 7f619bfff640 1 -- 172.21.15.114:0/2895068730 learned_addr learned my addr 172.21.15.114:0/2895068730 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.039+0000 7f61a0c96640 1 --2- 172.21.15.114:0/2895068730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbfa54d0 0x55dbbc367b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.039+0000 7f6193fff640 1 --2- 172.21.15.114:0/2895068730 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc36c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.040+0000 7f619bfff640 1 -- 172.21.15.114:0/2895068730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbfa54d0 msgr2=0x55dbbc367b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.040+0000 7f619bfff640 1 --2- 172.21.15.114:0/2895068730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbfa54d0 0x55dbbc367b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.040+0000 7f619bfff640 1 -- 172.21.15.114:0/2895068730 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbf9d2f0 msgr2=0x55dbbc36c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.040+0000 7f619bfff640 1 --2- 172.21.15.114:0/2895068730 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc36c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.040+0000 7f619bfff640 1 -- 172.21.15.114:0/2895068730 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbbc364c60 con 0x55dbbbf84d80 2024-05-27T07:49:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.040+0000 7f619bfff640 1 --2- 172.21.15.114:0/2895068730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf84d80 0x55dbbc3777f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f618c02aaa0 tx=0x7f618c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3da33abe7c45 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.040+0000 7f6191ffb640 1 -- 172.21.15.114:0/2895068730 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c049020 con 0x55dbbbf84d80 2024-05-27T07:49:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.040+0000 7f6191ffb640 1 -- 172.21.15.114:0/2895068730 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f618c004770 con 0x55dbbbf84d80 2024-05-27T07:49:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.040+0000 7f6191ffb640 1 -- 172.21.15.114:0/2895068730 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c035040 con 0x55dbbbf84d80 2024-05-27T07:49:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.041+0000 7f61a95a9740 1 -- 172.21.15.114:0/2895068730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf84d80 msgr2=0x55dbbc3777f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.041+0000 7f61a95a9740 1 --2- 172.21.15.114:0/2895068730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf84d80 0x55dbbc3777f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f618c02aaa0 tx=0x7f618c009260 comp rx=0 tx=0).stop 2024-05-27T07:49:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.041+0000 7f61a95a9740 1 -- 172.21.15.114:0/2895068730 shutdown_connections 2024-05-27T07:49:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.041+0000 7f61a95a9740 1 --2- 172.21.15.114:0/2895068730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbfa54d0 0x55dbbc367b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.041+0000 7f61a95a9740 1 --2- 172.21.15.114:0/2895068730 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc36c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.041+0000 7f61a95a9740 1 --2- 172.21.15.114:0/2895068730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf84d80 0x55dbbc3777f0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.041+0000 7f61a95a9740 1 -- 172.21.15.114:0/2895068730 >> 172.21.15.114:0/2895068730 conn(0x55dbbbfba300 msgr2=0x55dbbc362c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.041+0000 7f61a95a9740 1 -- 172.21.15.114:0/2895068730 shutdown_connections 2024-05-27T07:49:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.041+0000 7f61a95a9740 1 -- 172.21.15.114:0/2895068730 wait complete. 2024-05-27T07:49:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.041+0000 7f61a95a9740 1 -- start start 2024-05-27T07:49:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f61a95a9740 1 Processor -- start 2024-05-27T07:49:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f61a95a9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf84d80 0x55dbbc37e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f61a95a9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc37e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f61a95a9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfa54d0 0x55dbbc379500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f61a95a9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f617c00e540 con 0x55dbbbf9d2f0 2024-05-27T07:49:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f61a95a9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f617c00f9b0 con 0x55dbbbfa54d0 2024-05-27T07:49:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f61a95a9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f617c00ec10 con 0x55dbbbf84d80 2024-05-27T07:49:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f619bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf84d80 0x55dbbc37e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f6193fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc37e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f619bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf84d80 0x55dbbc37e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54788/0 (socket says 172.21.15.114:54788) 2024-05-27T07:49:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f619bfff640 1 -- 172.21.15.114:0/4192666414 learned_addr learned my addr 172.21.15.114:0/4192666414 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f61a95a9740 10 client.0 _ll_register_callbacks cb 0x7f61840a8e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f61a0c96640 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfa54d0 0x55dbbc379500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f6193fff640 1 -- 172.21.15.114:0/4192666414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf84d80 msgr2=0x55dbbc37e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f6193fff640 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf84d80 0x55dbbc37e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f6193fff640 1 -- 172.21.15.114:0/4192666414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfa54d0 msgr2=0x55dbbc379500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f6193fff640 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfa54d0 0x55dbbc379500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f6193fff640 1 -- 172.21.15.114:0/4192666414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbbc379c70 con 0x55dbbbf9d2f0 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f619bfff640 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf84d80 0x55dbbc37e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.042+0000 7f61a0c96640 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfa54d0 0x55dbbc379500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.043+0000 7f6193fff640 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc37e540 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f61840a8f60 tx=0x7f61840a5470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7058fb47024fb302 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.043+0000 7f617b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.043+0000 7f617b7fe640 1 -- 172.21.15.114:0/4192666414 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61840df7c0 con 0x55dbbbf9d2f0 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.043+0000 7f617b7fe640 1 -- 172.21.15.114:0/4192666414 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9538+0+0 (secure 0 0 0) 0x7f61840a55c0 con 0x55dbbbf9d2f0 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.043+0000 7f61a95a9740 1 -- 172.21.15.114:0/4192666414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f618c004770 con 0x55dbbbf9d2f0 2024-05-27T07:49:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.043+0000 7f617b7fe640 1 -- 172.21.15.114:0/4192666414 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f61840b5050 con 0x55dbbbf9d2f0 2024-05-27T07:49:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.043+0000 7f61a95a9740 10 client.17691 fetch_fsmap learned FSMap version 53 2024-05-27T07:49:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.043+0000 7f61a95a9740 1 -- 172.21.15.114:0/4192666414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55dbbc3ebd00 con 0x55dbbbf9d2f0 2024-05-27T07:49:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.043+0000 7f617b7fe640 1 -- 172.21.15.114:0/4192666414 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f618c004770 con 0x55dbbbf9d2f0 2024-05-27T07:49:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.044+0000 7f617b7fe640 1 -- 172.21.15.114:0/4192666414 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 53) ==== 7120+0+0 (secure 0 0 0) 0x7f618408a020 con 0x55dbbbf9d2f0 2024-05-27T07:49:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.044+0000 7f61a95a9740 10 client.17691 fetch_fsmap finished waiting for FSMap version 53 2024-05-27T07:49:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.044+0000 7f61a95a9740 10 client.17691 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.044+0000 7f61a95a9740 20 client.17691 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.044+0000 7f61a95a9740 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dbbbfc61b0 0x55dbbc3ee480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.044+0000 7f61a95a9740 4 client.17691 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:49:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.044+0000 7f61a95a9740 1 -- 172.21.15.114:0/4192666414 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dbbc2dbb80 con 0x55dbbbfc61b0 2024-05-27T07:49:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.044+0000 7f619bfff640 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dbbbfc61b0 0x55dbbc3ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:20.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.044+0000 7f617b7fe640 10 client.17691 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:20.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.044+0000 7f619bfff640 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dbbbfc61b0 0x55dbbc3ee480 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c45ceebe510f2df server_cookie=e97d31790e84f99d in_seq=0 out_seq=0 2024-05-27T07:49:20.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 966818 10.78 MB/sec execute 332 sec latency 180.397 ms 2024-05-27T07:49:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f617b7fe640 1 -- 172.21.15.114:0/4192666414 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55dbbc2dbb80 con 0x55dbbbfc61b0 2024-05-27T07:49:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f617b7fe640 10 client.17691 handle_command_reply: tid=0 2024-05-27T07:49:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f61a95a9740 1 client.17691 shutdown 2024-05-27T07:49:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f61a95a9740 1 -- 172.21.15.114:0/4192666414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf9d2f0 msgr2=0x55dbbc37e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f61a95a9740 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc37e540 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f61840a8f60 tx=0x7f61840a5470 comp rx=0 tx=0).stop 2024-05-27T07:49:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f61a95a9740 1 -- 172.21.15.114:0/4192666414 shutdown_connections 2024-05-27T07:49:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f61a95a9740 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55dbbbfc61b0 0x55dbbc3ee480 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f617b7fe640 0 client.17691 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f61a95a9740 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbbbfa54d0 0x55dbbc379500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f61a95a9740 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dbbbf9d2f0 0x55dbbc37e540 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f61a95a9740 1 --2- 172.21.15.114:0/4192666414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dbbbf84d80 0x55dbbc37e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f61a95a9740 1 -- 172.21.15.114:0/4192666414 >> 172.21.15.114:0/4192666414 conn(0x55dbbbfba300 msgr2=0x55dbbc362c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f61a95a9740 1 -- 172.21.15.114:0/4192666414 shutdown_connections 2024-05-27T07:49:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.626+0000 7f61a95a9740 1 -- 172.21.15.114:0/4192666414 wait complete. 2024-05-27T07:49:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.627+0000 7f61a95a9740 20 client.17691 trim_cache size 0 max 16384 2024-05-27T07:49:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:49:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:49:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:49:20.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.627+0000 7f61a26f9640 1 -- 172.21.15.114:0/1732940488 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f61880b2aa0 msgr2=0x7f61880b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:20.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.627+0000 7f61a26f9640 1 --2- 172.21.15.114:0/1732940488 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f61880b2aa0 0x7f61880b4fb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f619401d000 tx=0x7f61940107a0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.627+0000 7f61a26f9640 1 -- 172.21.15.114:0/1732940488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f619c11a8e0 msgr2=0x7f619c1134b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:20.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.627+0000 7f61a26f9640 1 --2- 172.21.15.114:0/1732940488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f619c11a8e0 0x7f619c1134b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f618400b3f0 tx=0x7f618400b8b0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.627+0000 7f61a26f9640 1 -- 172.21.15.114:0/1732940488 shutdown_connections 2024-05-27T07:49:20.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.627+0000 7f61a26f9640 1 --2- 172.21.15.114:0/1732940488 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f61880b2aa0 0x7f61880b4fb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.627+0000 7f61a26f9640 1 --2- 172.21.15.114:0/1732940488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f619c11d240 0x7f619c113b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.627+0000 7f61a26f9640 1 --2- 172.21.15.114:0/1732940488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f619c11a8e0 0x7f619c1134b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.627+0000 7f61a26f9640 1 --2- 172.21.15.114:0/1732940488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c10a1f0 0x7f619c11a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:20.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.627+0000 7f61a26f9640 1 -- 172.21.15.114:0/1732940488 >> 172.21.15.114:0/1732940488 conn(0x7f619c06e8f0 msgr2=0x7f619c11acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:20.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.627+0000 7f61a26f9640 1 -- 172.21.15.114:0/1732940488 shutdown_connections 2024-05-27T07:49:20.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:20.627+0000 7f61a26f9640 1 -- 172.21.15.114:0/1732940488 wait complete. 2024-05-27T07:49:20.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:49:21.146 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:49:21.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 968082 10.76 MB/sec execute 333 sec latency 232.309 ms 2024-05-27T07:49:21.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:21 smithi114 ceph-mon[32038]: pgmap v315: 129 pgs: 129 active+clean; 79 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 3.3 MiB/s wr, 220 op/s 2024-05-27T07:49:21.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:21 smithi163 ceph-mon[36451]: pgmap v315: 129 pgs: 129 active+clean; 79 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 3.3 MiB/s wr, 220 op/s 2024-05-27T07:49:21.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:21 smithi186 ceph-mon[36647]: pgmap v315: 129 pgs: 129 active+clean; 79 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 3.3 MiB/s wr, 220 op/s 2024-05-27T07:49:22.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 970635 10.77 MB/sec execute 334 sec latency 16.406 ms 2024-05-27T07:49:22.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.910+0000 7f33ae46b700 1 -- 172.21.15.114:0/2798534885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a81081f0 msgr2=0x7f33a8108600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:22.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.910+0000 7f33ae46b700 1 --2- 172.21.15.114:0/2798534885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a81081f0 0x7f33a8108600 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f3398009990 tx=0x7f3398009ca0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.910+0000 7f33ae46b700 1 -- 172.21.15.114:0/2798534885 shutdown_connections 2024-05-27T07:49:22.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.910+0000 7f33ae46b700 1 --2- 172.21.15.114:0/2798534885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a8104230 0x7f33a8110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.910+0000 7f33ae46b700 1 --2- 172.21.15.114:0/2798534885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a81081f0 0x7f33a8108600 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.910+0000 7f33ae46b700 1 --2- 172.21.15.114:0/2798534885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33a81038e0 0x7f33a8103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.910+0000 7f33ae46b700 1 -- 172.21.15.114:0/2798534885 >> 172.21.15.114:0/2798534885 conn(0x7f33a80fe6e0 msgr2=0x7f33a8100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:22.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.910+0000 7f33ae46b700 1 -- 172.21.15.114:0/2798534885 shutdown_connections 2024-05-27T07:49:22.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.910+0000 7f33ae46b700 1 -- 172.21.15.114:0/2798534885 wait complete. 2024-05-27T07:49:22.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.911+0000 7f33ae46b700 1 Processor -- start 2024-05-27T07:49:22.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.911+0000 7f33ae46b700 1 -- start start 2024-05-27T07:49:22.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.911+0000 7f33ae46b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33a81038e0 0x7f33a8199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.911+0000 7f33ae46b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8104230 0x7f33a819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f33ae46b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a81081f0 0x7f33a819e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f33ae46b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f33a8113400 con 0x7f33a8104230 2024-05-27T07:49:22.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f33ae46b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f33a8113280 con 0x7f33a81081f0 2024-05-27T07:49:22.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f33ae46b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f33a8113580 con 0x7f33a81038e0 2024-05-27T07:49:22.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f33a7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33a81038e0 0x7f33a8199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f339f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8104230 0x7f33a819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f33a7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33a81038e0 0x7f33a8199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54810/0 (socket says 172.21.15.114:54810) 2024-05-27T07:49:22.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f33a7fff700 1 -- 172.21.15.114:0/1881175811 learned_addr learned my addr 172.21.15.114:0/1881175811 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:22.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f33aca08700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a81081f0 0x7f33a819e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f339f7fe700 1 -- 172.21.15.114:0/1881175811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33a81038e0 msgr2=0x7f33a8199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:22.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f339f7fe700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33a81038e0 0x7f33a8199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f339f7fe700 1 -- 172.21.15.114:0/1881175811 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a81081f0 msgr2=0x7f33a819e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:22.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f339f7fe700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a81081f0 0x7f33a819e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f339f7fe700 1 -- 172.21.15.114:0/1881175811 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a819ec00 con 0x7f33a8104230 2024-05-27T07:49:22.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f33aca08700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a81081f0 0x7f33a819e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:49:22.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f339f7fe700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8104230 0x7f33a819a2d0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f339000e9f0 tx=0x7f339000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:22.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f33a5ffb700 1 -- 172.21.15.114:0/1881175811 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339000cbd0 con 0x7f33a8104230 2024-05-27T07:49:22.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f33a5ffb700 1 -- 172.21.15.114:0/1881175811 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f339000cd70 con 0x7f33a8104230 2024-05-27T07:49:22.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f33a5ffb700 1 -- 172.21.15.114:0/1881175811 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3390018680 con 0x7f33a8104230 2024-05-27T07:49:22.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.912+0000 7f33ae46b700 1 -- 172.21.15.114:0/1881175811 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a819eef0 con 0x7f33a8104230 2024-05-27T07:49:22.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.913+0000 7f33ae46b700 1 -- 172.21.15.114:0/1881175811 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a81a6810 con 0x7f33a8104230 2024-05-27T07:49:22.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.916+0000 7f33a5ffb700 1 -- 172.21.15.114:0/1881175811 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f3390010430 con 0x7f33a8104230 2024-05-27T07:49:22.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.917+0000 7f33a5ffb700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f33940b2a70 0x7f33940b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.917+0000 7f33bce80b80 1 Processor -- start 2024-05-27T07:49:22.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.917+0000 7f33a7fff700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f33940b2a70 0x7f33940b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.917+0000 7f33a5ffb700 1 -- 172.21.15.114:0/1881175811 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9538+0+0 (secure 0 0 0) 0x7f33900e2ba0 con 0x7f33a8104230 2024-05-27T07:49:22.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.917+0000 7f33bce80b80 1 -- start start 2024-05-27T07:49:22.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.917+0000 7f33bce80b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562382eb1e40 0x562383033be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.917+0000 7f33bce80b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x5623830301e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.917+0000 7f33bce80b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x562383045a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f33bce80b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562382f5ed70 con 0x562383030720 2024-05-27T07:49:22.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f33bce80b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562382f4e910 con 0x562382eb1e40 2024-05-27T07:49:22.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f33bce80b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562382f52950 con 0x56238302fdd0 2024-05-27T07:49:22.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f33aca08700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x562383045a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f339f7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x5623830301e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f339f7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x5623830301e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54818/0 (socket says 172.21.15.114:54818) 2024-05-27T07:49:22.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f339f7fe700 1 -- 172.21.15.114:0/358120551 learned_addr learned my addr 172.21.15.114:0/358120551 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:22.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f33aca08700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x562383045a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59358/0 (socket says 172.21.15.114:59358) 2024-05-27T07:49:22.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f33a7fff700 1 --2- 172.21.15.114:0/358120551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562382eb1e40 0x562383033be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f33a7fff700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f33940b2a70 0x7f33940b4f80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3398009750 tx=0x7f33980051a0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:22.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f339f7fe700 1 -- 172.21.15.114:0/358120551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562382eb1e40 msgr2=0x562383033be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:22.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f339f7fe700 1 --2- 172.21.15.114:0/358120551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562382eb1e40 0x562383033be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f339f7fe700 1 -- 172.21.15.114:0/358120551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 msgr2=0x562383045a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:22.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f339f7fe700 1 --2- 172.21.15.114:0/358120551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x562383045a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f339f7fe700 1 -- 172.21.15.114:0/358120551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562382ef7fb0 con 0x56238302fdd0 2024-05-27T07:49:22.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f33aca08700 1 --2- 172.21.15.114:0/358120551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x562383045a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:22.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f339f7fe700 1 --2- 172.21.15.114:0/358120551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x5623830301e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f33900a5aa0 tx=0x7f33900a9470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40a095b95b0a3b5d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:22.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f339e7fc700 1 -- 172.21.15.114:0/358120551 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33900a9530 con 0x56238302fdd0 2024-05-27T07:49:22.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f339e7fc700 1 -- 172.21.15.114:0/358120551 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33900aa120 con 0x56238302fdd0 2024-05-27T07:49:22.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.918+0000 7f339e7fc700 1 -- 172.21.15.114:0/358120551 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33900b2650 con 0x56238302fdd0 2024-05-27T07:49:22.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.919+0000 7f33bce80b80 1 -- 172.21.15.114:0/358120551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 msgr2=0x5623830301e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:22.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.919+0000 7f33bce80b80 1 --2- 172.21.15.114:0/358120551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x5623830301e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f33900a5aa0 tx=0x7f33900a9470 comp rx=0 tx=0).stop 2024-05-27T07:49:22.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.919+0000 7f33bce80b80 1 -- 172.21.15.114:0/358120551 shutdown_connections 2024-05-27T07:49:22.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.919+0000 7f33bce80b80 1 --2- 172.21.15.114:0/358120551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x562383045a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.919+0000 7f33bce80b80 1 --2- 172.21.15.114:0/358120551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562382eb1e40 0x562383033be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.919+0000 7f33bce80b80 1 --2- 172.21.15.114:0/358120551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x5623830301e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.919+0000 7f33bce80b80 1 -- 172.21.15.114:0/358120551 >> 172.21.15.114:0/358120551 conn(0x562382eb4770 msgr2=0x56238302bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:22.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.919+0000 7f33bce80b80 1 -- 172.21.15.114:0/358120551 shutdown_connections 2024-05-27T07:49:22.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.919+0000 7f33bce80b80 1 -- 172.21.15.114:0/358120551 wait complete. 2024-05-27T07:49:22.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.920+0000 7f33bce80b80 1 -- start start 2024-05-27T07:49:22.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.920+0000 7f33bce80b80 1 Processor -- start 2024-05-27T07:49:22.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.920+0000 7f33bce80b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562382eb1e40 0x562383043060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.920+0000 7f33bce80b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x5623830435a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.920+0000 7f33bce80b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x562383043ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.920+0000 7f33bce80b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562382e54fa0 con 0x562383030720 2024-05-27T07:49:22.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.920+0000 7f33bce80b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562382eb6da0 con 0x562382eb1e40 2024-05-27T07:49:22.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.920+0000 7f33bce80b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562383046300 con 0x56238302fdd0 2024-05-27T07:49:22.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33bce80b80 10 client.0 _ll_register_callbacks cb 0x562382ee55b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:22.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33aca08700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x562383043ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33aca08700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x562383043ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59370/0 (socket says 172.21.15.114:59370) 2024-05-27T07:49:22.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f339f7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x5623830435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33aca08700 1 -- 172.21.15.114:0/3221544679 learned_addr learned my addr 172.21.15.114:0/3221544679 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:22.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33a7fff700 1 --2- 172.21.15.114:0/3221544679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562382eb1e40 0x562383043060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33aca08700 1 -- 172.21.15.114:0/3221544679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 msgr2=0x5623830435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:22.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33aca08700 1 --2- 172.21.15.114:0/3221544679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x5623830435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33aca08700 1 -- 172.21.15.114:0/3221544679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562382eb1e40 msgr2=0x562383043060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:22.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33aca08700 1 --2- 172.21.15.114:0/3221544679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562382eb1e40 0x562383043060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33aca08700 1 -- 172.21.15.114:0/3221544679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562382e9e8d0 con 0x562383030720 2024-05-27T07:49:22.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f339f7fe700 1 --2- 172.21.15.114:0/3221544679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x5623830435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:22.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33aca08700 1 --2- 172.21.15.114:0/3221544679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x562383043ae0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f33a000fc80 tx=0x7f33a000fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926dd76708fc5a84 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:22.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33877fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:49:22.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33877fe700 1 -- 172.21.15.114:0/3221544679 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a000d960 con 0x562383030720 2024-05-27T07:49:22.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33877fe700 1 -- 172.21.15.114:0/3221544679 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9538+0+0 (secure 0 0 0) 0x7f33a000db00 con 0x562383030720 2024-05-27T07:49:22.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.921+0000 7f33bce80b80 1 -- 172.21.15.114:0/3221544679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33900aa120 con 0x562383030720 2024-05-27T07:49:22.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.922+0000 7f33877fe700 1 -- 172.21.15.114:0/3221544679 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f33a0023050 con 0x562383030720 2024-05-27T07:49:22.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.922+0000 7f33bce80b80 10 client.17703 fetch_fsmap learned FSMap version 53 2024-05-27T07:49:22.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.922+0000 7f33bce80b80 1 -- 172.21.15.114:0/3221544679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x562382f08760 con 0x562383030720 2024-05-27T07:49:22.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.922+0000 7f33877fe700 1 -- 172.21.15.114:0/3221544679 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33900aa120 con 0x562383030720 2024-05-27T07:49:22.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.922+0000 7f33877fe700 1 -- 172.21.15.114:0/3221544679 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 53) ==== 7120+0+0 (secure 0 0 0) 0x7f33a00183f0 con 0x562383030720 2024-05-27T07:49:22.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.922+0000 7f33bce80b80 10 client.17703 fetch_fsmap finished waiting for FSMap version 53 2024-05-27T07:49:22.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.922+0000 7f33bce80b80 10 client.17703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:22.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.922+0000 7f33bce80b80 20 client.17703 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:22.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.922+0000 7f33bce80b80 1 --2- 172.21.15.114:0/3221544679 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5623830535b0 0x562383055a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.922+0000 7f33bce80b80 4 client.17703 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:49:22.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.922+0000 7f33bce80b80 1 -- 172.21.15.114:0/3221544679 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562382f60cc0 con 0x5623830535b0 2024-05-27T07:49:22.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.922+0000 7f33a7fff700 1 --2- 172.21.15.114:0/3221544679 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5623830535b0 0x562383055a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.923+0000 7f33877fe700 10 client.17703 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:22.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.923+0000 7f33a7fff700 1 --2- 172.21.15.114:0/3221544679 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5623830535b0 0x562383055a40 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfde797dee7dfe22 server_cookie=4a3a63a155dd8246 in_seq=0 out_seq=0 2024-05-27T07:49:22.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.927+0000 7f33877fe700 1 -- 172.21.15.114:0/3221544679 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562382f60cc0 con 0x5623830535b0 2024-05-27T07:49:22.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.927+0000 7f33877fe700 10 client.17703 handle_command_reply: tid=0 2024-05-27T07:49:22.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.927+0000 7f33bce80b80 1 client.17703 shutdown 2024-05-27T07:49:22.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.927+0000 7f33bce80b80 1 -- 172.21.15.114:0/3221544679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 msgr2=0x562383043ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:22.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.927+0000 7f33bce80b80 1 --2- 172.21.15.114:0/3221544679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x562383043ae0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f33a000fc80 tx=0x7f33a000fcb0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.928+0000 7f33bce80b80 1 -- 172.21.15.114:0/3221544679 shutdown_connections 2024-05-27T07:49:22.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.928+0000 7f33bce80b80 1 --2- 172.21.15.114:0/3221544679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x562383043ae0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.928+0000 7f33bce80b80 1 --2- 172.21.15.114:0/3221544679 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5623830535b0 0x562383055a40 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.928+0000 7f33bce80b80 1 --2- 172.21.15.114:0/3221544679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562382eb1e40 0x562383043060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.928+0000 7f33bce80b80 1 --2- 172.21.15.114:0/3221544679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x5623830435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.928+0000 7f33877fe700 0 client.17703 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:22.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.928+0000 7f33bce80b80 1 -- 172.21.15.114:0/3221544679 >> 172.21.15.114:0/3221544679 conn(0x562382eb4770 msgr2=0x5623830254f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:22.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.928+0000 7f33bce80b80 1 -- 172.21.15.114:0/3221544679 shutdown_connections 2024-05-27T07:49:22.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.928+0000 7f33bce80b80 1 -- 172.21.15.114:0/3221544679 wait complete. 2024-05-27T07:49:22.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.928+0000 7f33bce80b80 20 client.17703 trim_cache size 0 max 16384 2024-05-27T07:49:22.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.950+0000 7f33bce80b80 1 Processor -- start 2024-05-27T07:49:22.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.950+0000 7f33bce80b80 1 -- start start 2024-05-27T07:49:22.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.950+0000 7f33bce80b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562382eb1e40 0x56238303ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.951+0000 7f33bce80b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56238302fdd0 0x562383044fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.951+0000 7f33bce80b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x56238304f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.951+0000 7f33bce80b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81ae760 con 0x562383030720 2024-05-27T07:49:22.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.951+0000 7f33bce80b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81aee30 con 0x56238302fdd0 2024-05-27T07:49:22.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.951+0000 7f33bce80b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81ac5d0 con 0x562382eb1e40 2024-05-27T07:49:22.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.952+0000 7f339f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56238302fdd0 0x562383044fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.952+0000 7f33a7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562382eb1e40 0x56238303ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.952+0000 7f33a7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562382eb1e40 0x56238303ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54838/0 (socket says 172.21.15.114:54838) 2024-05-27T07:49:22.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.952+0000 7f33a7fff700 1 -- 172.21.15.114:0/1305365039 learned_addr learned my addr 172.21.15.114:0/1305365039 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:22.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.953+0000 7f33a7fff700 1 -- 172.21.15.114:0/1305365039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56238302fdd0 msgr2=0x562383044fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:22.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.953+0000 7f33a7fff700 1 --2- 172.21.15.114:0/1305365039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56238302fdd0 0x562383044fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.953+0000 7f33a7fff700 1 -- 172.21.15.114:0/1305365039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 msgr2=0x56238304f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:22.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.953+0000 7f33a7fff700 1 --2- 172.21.15.114:0/1305365039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x56238304f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.953+0000 7f33a7fff700 1 -- 172.21.15.114:0/1305365039 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623830252f0 con 0x562382eb1e40 2024-05-27T07:49:22.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.955+0000 7f33a7fff700 1 --2- 172.21.15.114:0/1305365039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562382eb1e40 0x56238303ea10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f33980098e0 tx=0x7f3398020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76c584a3e7fa057b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:22.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.955+0000 7f339dffb700 1 -- 172.21.15.114:0/1305365039 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339800b860 con 0x562382eb1e40 2024-05-27T07:49:22.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.956+0000 7f339dffb700 1 -- 172.21.15.114:0/1305365039 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f339800f460 con 0x562382eb1e40 2024-05-27T07:49:22.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.956+0000 7f339dffb700 1 -- 172.21.15.114:0/1305365039 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3398026420 con 0x562382eb1e40 2024-05-27T07:49:22.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.957+0000 7f33bce80b80 1 -- 172.21.15.114:0/1305365039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562382eb1e40 msgr2=0x56238303ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:22.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.957+0000 7f33bce80b80 1 --2- 172.21.15.114:0/1305365039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562382eb1e40 0x56238303ea10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f33980098e0 tx=0x7f3398020000 comp rx=0 tx=0).stop 2024-05-27T07:49:22.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.959+0000 7f33bce80b80 1 -- 172.21.15.114:0/1305365039 shutdown_connections 2024-05-27T07:49:22.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.959+0000 7f33bce80b80 1 --2- 172.21.15.114:0/1305365039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562383030720 0x56238304f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.959+0000 7f33bce80b80 1 --2- 172.21.15.114:0/1305365039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562382eb1e40 0x56238303ea10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.959+0000 7f33bce80b80 1 --2- 172.21.15.114:0/1305365039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56238302fdd0 0x562383044fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.959+0000 7f33bce80b80 1 -- 172.21.15.114:0/1305365039 >> 172.21.15.114:0/1305365039 conn(0x562382eb4770 msgr2=0x562383025b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:22.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.962+0000 7f33bce80b80 1 -- 172.21.15.114:0/1305365039 shutdown_connections 2024-05-27T07:49:22.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.966+0000 7f33bce80b80 1 -- 172.21.15.114:0/1305365039 wait complete. 2024-05-27T07:49:22.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.966+0000 7f33bce80b80 1 -- start start 2024-05-27T07:49:22.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.966+0000 7f33bce80b80 1 Processor -- start 2024-05-27T07:49:22.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.967+0000 7f33bce80b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562382eb1e40 0x56238304bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.967+0000 7f33bce80b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x562383048560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.967+0000 7f33bce80b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562383030720 0x562383048aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.967+0000 7f33bce80b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81acc20 con 0x562382eb1e40 2024-05-27T07:49:22.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.967+0000 7f33bce80b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81ae090 con 0x562383030720 2024-05-27T07:49:22.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.967+0000 7f33bce80b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81ad2f0 con 0x56238302fdd0 2024-05-27T07:49:22.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.967+0000 7f33bce80b80 10 client.0 _ll_register_callbacks cb 0x562382ee55b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:22.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.969+0000 7f33aca08700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562383030720 0x562383048aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.969+0000 7f33aca08700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562383030720 0x562383048aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:35042/0 (socket says 172.21.15.114:35042) 2024-05-27T07:49:22.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.969+0000 7f33aca08700 1 -- 172.21.15.114:0/686494469 learned_addr learned my addr 172.21.15.114:0/686494469 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:22.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.969+0000 7f33a7fff700 1 --2- 172.21.15.114:0/686494469 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562382eb1e40 0x56238304bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.970+0000 7f33aca08700 1 -- 172.21.15.114:0/686494469 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 msgr2=0x562383048560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:49:22.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.970+0000 7f33aca08700 1 --2- 172.21.15.114:0/686494469 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x562383048560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.970+0000 7f33aca08700 1 -- 172.21.15.114:0/686494469 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562382eb1e40 msgr2=0x56238304bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:22.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.970+0000 7f33aca08700 1 --2- 172.21.15.114:0/686494469 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562382eb1e40 0x56238304bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.970+0000 7f33aca08700 1 -- 172.21.15.114:0/686494469 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623830491a0 con 0x562383030720 2024-05-27T07:49:22.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.973+0000 7f33aca08700 1 --2- 172.21.15.114:0/686494469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562383030720 0x562383048aa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f33a0000f80 tx=0x7f33a0007270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cff34f159df3e7b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:22.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.973+0000 7f33877fe700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:49:22.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.973+0000 7f33877fe700 1 -- 172.21.15.114:0/686494469 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a00185f0 con 0x562383030720 2024-05-27T07:49:22.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.974+0000 7f33877fe700 1 -- 172.21.15.114:0/686494469 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9538+0+0 (secure 0 0 0) 0x7f33a0003aa0 con 0x562383030720 2024-05-27T07:49:22.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.974+0000 7f33bce80b80 1 -- 172.21.15.114:0/686494469 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f339800f460 con 0x562383030720 2024-05-27T07:49:22.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.975+0000 7f33877fe700 1 -- 172.21.15.114:0/686494469 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f33a0036050 con 0x562383030720 2024-05-27T07:49:22.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.975+0000 7f33bce80b80 10 client.27170 fetch_fsmap learned FSMap version 53 2024-05-27T07:49:22.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.975+0000 7f33bce80b80 1 -- 172.21.15.114:0/686494469 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5623830cdfb0 con 0x562383030720 2024-05-27T07:49:22.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.978+0000 7f33877fe700 1 -- 172.21.15.114:0/686494469 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f339800f460 con 0x562383030720 2024-05-27T07:49:22.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.978+0000 7f33877fe700 1 -- 172.21.15.114:0/686494469 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 53) ==== 7120+0+0 (secure 0 0 0) 0x7f33a0031030 con 0x562383030720 2024-05-27T07:49:22.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.978+0000 7f33bce80b80 10 client.27170 fetch_fsmap finished waiting for FSMap version 53 2024-05-27T07:49:22.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.978+0000 7f33bce80b80 10 client.27170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:22.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.978+0000 7f33bce80b80 20 client.27170 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:22.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.978+0000 7f33bce80b80 1 --2- 172.21.15.114:0/686494469 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5623830ce690 0x5623830d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:22.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.978+0000 7f33bce80b80 4 client.27170 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:49:22.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.978+0000 7f33bce80b80 1 -- 172.21.15.114:0/686494469 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562382f5a4a0 con 0x5623830ce690 2024-05-27T07:49:22.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.979+0000 7f33a7fff700 1 --2- 172.21.15.114:0/686494469 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5623830ce690 0x5623830d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:22.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.989+0000 7f33a7fff700 1 --2- 172.21.15.114:0/686494469 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5623830ce690 0x5623830d0ab0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff51615ef25d9a7d server_cookie=b80b4796f8e7c09f in_seq=0 out_seq=0 2024-05-27T07:49:22.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.989+0000 7f33877fe700 10 client.27170 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:22.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.989+0000 7f33877fe700 1 -- 172.21.15.114:0/686494469 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562382f5a4a0 con 0x5623830ce690 2024-05-27T07:49:22.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.989+0000 7f33877fe700 10 client.27170 handle_command_reply: tid=0 2024-05-27T07:49:22.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.989+0000 7f33bce80b80 1 client.27170 shutdown 2024-05-27T07:49:22.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.993+0000 7f33bce80b80 1 -- 172.21.15.114:0/686494469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562383030720 msgr2=0x562383048aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:22.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.993+0000 7f33bce80b80 1 --2- 172.21.15.114:0/686494469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562383030720 0x562383048aa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f33a0000f80 tx=0x7f33a0007270 comp rx=0 tx=0).stop 2024-05-27T07:49:22.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.993+0000 7f33bce80b80 1 -- 172.21.15.114:0/686494469 shutdown_connections 2024-05-27T07:49:22.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.993+0000 7f33bce80b80 1 --2- 172.21.15.114:0/686494469 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5623830ce690 0x5623830d0ab0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.993+0000 7f33bce80b80 1 --2- 172.21.15.114:0/686494469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562383030720 0x562383048aa0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.993+0000 7f33bce80b80 1 --2- 172.21.15.114:0/686494469 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562382eb1e40 0x56238304bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.993+0000 7f33bce80b80 1 --2- 172.21.15.114:0/686494469 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56238302fdd0 0x562383048560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:22.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.993+0000 7f33bce80b80 1 -- 172.21.15.114:0/686494469 >> 172.21.15.114:0/686494469 conn(0x562382eb4770 msgr2=0x562383025b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:22.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.993+0000 7f33877fe700 0 client.27170 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:22.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.995+0000 7f33bce80b80 1 -- 172.21.15.114:0/686494469 shutdown_connections 2024-05-27T07:49:22.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.995+0000 7f33bce80b80 1 -- 172.21.15.114:0/686494469 wait complete. 2024-05-27T07:49:22.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.996+0000 7f33bce80b80 20 client.27170 trim_cache size 0 max 16384 2024-05-27T07:49:22.997 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:49:23.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.999+0000 7f33ae46b700 1 -- 172.21.15.114:0/1881175811 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f33940b2a70 msgr2=0x7f33940b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:23.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.999+0000 7f33ae46b700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f33940b2a70 0x7f33940b4f80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3398009750 tx=0x7f33980051a0 comp rx=0 tx=0).stop 2024-05-27T07:49:23.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.999+0000 7f33ae46b700 1 -- 172.21.15.114:0/1881175811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8104230 msgr2=0x7f33a819a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:23.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.999+0000 7f33ae46b700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8104230 0x7f33a819a2d0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f339000e9f0 tx=0x7f339000ed00 comp rx=0 tx=0).stop 2024-05-27T07:49:23.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.999+0000 7f33ae46b700 1 -- 172.21.15.114:0/1881175811 shutdown_connections 2024-05-27T07:49:23.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.999+0000 7f33ae46b700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f33940b2a70 0x7f33940b4f80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:23.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.999+0000 7f33ae46b700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a81081f0 0x7f33a819e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:23.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.999+0000 7f33ae46b700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33a81038e0 0x7f33a8199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:23.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.999+0000 7f33ae46b700 1 --2- 172.21.15.114:0/1881175811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8104230 0x7f33a819a2d0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:23.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:22.999+0000 7f33ae46b700 1 -- 172.21.15.114:0/1881175811 >> 172.21.15.114:0/1881175811 conn(0x7f33a80fe6e0 msgr2=0x7f33a81075f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:23.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:23.003+0000 7f33ae46b700 1 -- 172.21.15.114:0/1881175811 shutdown_connections 2024-05-27T07:49:23.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:23.003+0000 7f33ae46b700 1 -- 172.21.15.114:0/1881175811 wait complete. 2024-05-27T07:49:23.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 972533 10.76 MB/sec execute 335 sec latency 58.095 ms 2024-05-27T07:49:23.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:23 smithi163 ceph-mon[36451]: pgmap v316: 129 pgs: 129 active+clean; 94 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.2 MiB/s wr, 377 op/s 2024-05-27T07:49:23.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:23 smithi186 ceph-mon[36647]: pgmap v316: 129 pgs: 129 active+clean; 94 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.2 MiB/s wr, 377 op/s 2024-05-27T07:49:23.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:23 smithi114 ceph-mon[32038]: pgmap v316: 129 pgs: 129 active+clean; 94 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.2 MiB/s wr, 377 op/s 2024-05-27T07:49:24.279 INFO:tasks.workunit.client.0.smithi114.stdout: 1 974632 10.77 MB/sec execute 336 sec latency 20.415 ms 2024-05-27T07:49:25.066 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:49:25.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:49:25.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 976261 10.76 MB/sec execute 337 sec latency 61.072 ms 2024-05-27T07:49:25.364 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:49:25.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:25 smithi114 ceph-mon[32038]: pgmap v317: 129 pgs: 129 active+clean; 94 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 5.0 MiB/s wr, 302 op/s 2024-05-27T07:49:25.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:25 smithi163 ceph-mon[36451]: pgmap v317: 129 pgs: 129 active+clean; 94 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 5.0 MiB/s wr, 302 op/s 2024-05-27T07:49:25.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:25 smithi186 ceph-mon[36647]: pgmap v317: 129 pgs: 129 active+clean; 94 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 5.0 MiB/s wr, 302 op/s 2024-05-27T07:49:26.273 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:25 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:49:26.273 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:25 smithi163 systemd[1]: Stopped Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:49:26.273 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:25 smithi163 systemd[1]: Starting Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:49:26.273 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:26 smithi163 podman[72270]: 2024-05-27 07:49:26.110809432 +0000 UTC m=+0.018077803 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:49:26.273 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:26 smithi163 podman[72270]: 2024-05-27 07:49:26.273600441 +0000 UTC m=+0.180868802 container create 1f7fdb6f6ea09add5e87026129bc479ed94b2c6b6736b0c2c4dc8c9f43611ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-27T07:49:26.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 976956 10.75 MB/sec execute 338 sec latency 809.223 ms 2024-05-27T07:49:26.553 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:26 smithi163 podman[72270]: 2024-05-27 07:49:26.518494452 +0000 UTC m=+0.425762817 container init 1f7fdb6f6ea09add5e87026129bc479ed94b2c6b6736b0c2c4dc8c9f43611ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:49:26.553 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:26 smithi163 podman[72270]: 2024-05-27 07:49:26.521301646 +0000 UTC m=+0.428570015 container start 1f7fdb6f6ea09add5e87026129bc479ed94b2c6b6736b0c2c4dc8c9f43611ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-27T07:49:26.975 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:26 smithi163 ceph-mds[72289]: -- 172.21.15.163:0/3494680300 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c769cfda40 con 0x55c769d45a80 2024-05-27T07:49:26.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: starting mds.h at 2024-05-27T07:49:26.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:26 smithi163 bash[72270]: 1f7fdb6f6ea09add5e87026129bc479ed94b2c6b6736b0c2c4dc8c9f43611ff7 2024-05-27T07:49:26.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:26 smithi163 systemd[1]: Started Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:49:27.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.092+0000 7f9040992700 1 -- 172.21.15.114:0/984310814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f903c1089e0 msgr2=0x7f903c108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:27.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.092+0000 7f9040992700 1 --2- 172.21.15.114:0/984310814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f903c1089e0 0x7f903c108df0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f902c009990 tx=0x7f902c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.092+0000 7f9040992700 1 -- 172.21.15.114:0/984310814 shutdown_connections 2024-05-27T07:49:27.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.092+0000 7f9040992700 1 --2- 172.21.15.114:0/984310814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c103300 0x7f903c10f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.092+0000 7f9040992700 1 --2- 172.21.15.114:0/984310814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f903c1089e0 0x7f903c108df0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.092+0000 7f9040992700 1 --2- 172.21.15.114:0/984310814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f903c1029b0 0x7f903c102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.092+0000 7f9040992700 1 -- 172.21.15.114:0/984310814 >> 172.21.15.114:0/984310814 conn(0x7f903c0fe510 msgr2=0x7f903c100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:27.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.092+0000 7f9040992700 1 -- 172.21.15.114:0/984310814 shutdown_connections 2024-05-27T07:49:27.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.092+0000 7f9040992700 1 -- 172.21.15.114:0/984310814 wait complete. 2024-05-27T07:49:27.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.093+0000 7f9040992700 1 Processor -- start 2024-05-27T07:49:27.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.093+0000 7f9040992700 1 -- start start 2024-05-27T07:49:27.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.093+0000 7f9040992700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f903c1029b0 0x7f903c19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.093+0000 7f9040992700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f903c103300 0x7f903c19b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.093+0000 7f9040992700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c1089e0 0x7f903c19f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.093+0000 7f9040992700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f903c114510 con 0x7f903c1029b0 2024-05-27T07:49:27.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.093+0000 7f9040992700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f903c114390 con 0x7f903c103300 2024-05-27T07:49:27.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.093+0000 7f9040992700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f903c114690 con 0x7f903c1089e0 2024-05-27T07:49:27.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.093+0000 7f903a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f903c1029b0 0x7f903c19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.093+0000 7f903a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f903c1029b0 0x7f903c19ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48122/0 (socket says 172.21.15.114:48122) 2024-05-27T07:49:27.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.093+0000 7f903a59c700 1 -- 172.21.15.114:0/2044570921 learned_addr learned my addr 172.21.15.114:0/2044570921 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:27.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.093+0000 7f903ad9d700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c1089e0 0x7f903c19f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.093+0000 7f9039d9b700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f903c103300 0x7f903c19b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.094+0000 7f903a59c700 1 -- 172.21.15.114:0/2044570921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c1089e0 msgr2=0x7f903c19f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:27.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.094+0000 7f903a59c700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c1089e0 0x7f903c19f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.094+0000 7f903a59c700 1 -- 172.21.15.114:0/2044570921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f903c103300 msgr2=0x7f903c19b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:27.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.094+0000 7f903a59c700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f903c103300 0x7f903c19b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.094+0000 7f903a59c700 1 -- 172.21.15.114:0/2044570921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903c19fcf0 con 0x7f903c1029b0 2024-05-27T07:49:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.094+0000 7f903a59c700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f903c1029b0 0x7f903c19ae20 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f902c000c00 tx=0x7f902c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.094+0000 7f902b7fe700 1 -- 172.21.15.114:0/2044570921 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902c027070 con 0x7f903c1029b0 2024-05-27T07:49:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.094+0000 7f9040992700 1 -- 172.21.15.114:0/2044570921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f903c19ff80 con 0x7f903c1029b0 2024-05-27T07:49:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.094+0000 7f902b7fe700 1 -- 172.21.15.114:0/2044570921 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f902c022470 con 0x7f903c1029b0 2024-05-27T07:49:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.094+0000 7f902b7fe700 1 -- 172.21.15.114:0/2044570921 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902c017650 con 0x7f903c1029b0 2024-05-27T07:49:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.094+0000 7f9040992700 1 -- 172.21.15.114:0/2044570921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f903c1a7900 con 0x7f903c1029b0 2024-05-27T07:49:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.097+0000 7f902b7fe700 1 -- 172.21.15.114:0/2044570921 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f902c02a070 con 0x7f903c1029b0 2024-05-27T07:49:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.098+0000 7f902b7fe700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f90240b29f0 0x7f90240b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.098+0000 7f9039d9b700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f90240b29f0 0x7f90240b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.098+0000 7f902b7fe700 1 -- 172.21.15.114:0/2044570921 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9538+0+0 (secure 0 0 0) 0x7f902c0e3900 con 0x7f903c1029b0 2024-05-27T07:49:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f904f3a7b80 1 Processor -- start 2024-05-27T07:49:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f9039d9b700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f90240b29f0 0x7f90240b4f00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f903c100090 tx=0x7f9030009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f904f3a7b80 1 -- start start 2024-05-27T07:49:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f904f3a7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16b69ec0 0x560b16ced790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f904f3a7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 0x560b16ce7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f904f3a7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 0x560b16ce8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f904f3a7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560b16c16d10 con 0x560b16cedcd0 2024-05-27T07:49:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f904f3a7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560b16c068b0 con 0x560b16b69ec0 2024-05-27T07:49:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f904f3a7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560b16c0a8f0 con 0x560b16ce8000 2024-05-27T07:49:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f9039d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 0x560b16ce7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f9039d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 0x560b16ce7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48136/0 (socket says 172.21.15.114:48136) 2024-05-27T07:49:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f9039d9b700 1 -- 172.21.15.114:0/4046020996 learned_addr learned my addr 172.21.15.114:0/4046020996 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f903a59c700 1 --2- 172.21.15.114:0/4046020996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16b69ec0 0x560b16ced790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.099+0000 7f903ad9d700 1 --2- 172.21.15.114:0/4046020996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 0x560b16ce8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.100+0000 7f9039d9b700 1 -- 172.21.15.114:0/4046020996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 msgr2=0x560b16ce8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.100+0000 7f9039d9b700 1 --2- 172.21.15.114:0/4046020996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 0x560b16ce8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.100+0000 7f9039d9b700 1 -- 172.21.15.114:0/4046020996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16b69ec0 msgr2=0x560b16ced790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.100+0000 7f9039d9b700 1 --2- 172.21.15.114:0/4046020996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16b69ec0 0x560b16ced790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.100+0000 7f9039d9b700 1 -- 172.21.15.114:0/4046020996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b16b56950 con 0x560b16cedcd0 2024-05-27T07:49:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.100+0000 7f903ad9d700 1 --2- 172.21.15.114:0/4046020996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 0x560b16ce8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:49:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.100+0000 7f9039d9b700 1 --2- 172.21.15.114:0/4046020996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 0x560b16ce7ac0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f9030011de0 tx=0x7f9030014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0716a4a2a2fed47 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:27.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.100+0000 7f9028ff9700 1 -- 172.21.15.114:0/4046020996 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9030012bc0 con 0x560b16cedcd0 2024-05-27T07:49:27.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.100+0000 7f9028ff9700 1 -- 172.21.15.114:0/4046020996 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9030012d60 con 0x560b16cedcd0 2024-05-27T07:49:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.100+0000 7f9028ff9700 1 -- 172.21.15.114:0/4046020996 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903001d6e0 con 0x560b16cedcd0 2024-05-27T07:49:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.101+0000 7f904f3a7b80 1 -- 172.21.15.114:0/4046020996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 msgr2=0x560b16ce7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.101+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/4046020996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 0x560b16ce7ac0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f9030011de0 tx=0x7f9030014000 comp rx=0 tx=0).stop 2024-05-27T07:49:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.101+0000 7f904f3a7b80 1 -- 172.21.15.114:0/4046020996 shutdown_connections 2024-05-27T07:49:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.101+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/4046020996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 0x560b16ce8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.101+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/4046020996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16b69ec0 0x560b16ced790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.101+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/4046020996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 0x560b16ce7ac0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.101+0000 7f904f3a7b80 1 -- 172.21.15.114:0/4046020996 >> 172.21.15.114:0/4046020996 conn(0x560b16bc4b70 msgr2=0x560b16ce26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.101+0000 7f904f3a7b80 1 -- 172.21.15.114:0/4046020996 shutdown_connections 2024-05-27T07:49:27.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.101+0000 7f904f3a7b80 1 -- 172.21.15.114:0/4046020996 wait complete. 2024-05-27T07:49:27.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.101+0000 7f904f3a7b80 1 -- start start 2024-05-27T07:49:27.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.102+0000 7f904f3a7b80 1 Processor -- start 2024-05-27T07:49:27.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.102+0000 7f904f3a7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16b69ec0 0x560b16cf13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.102+0000 7f904f3a7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 0x560b16cf18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.102+0000 7f904f3a7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16cedcd0 0x560b16cf1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.102+0000 7f904f3a7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560b16b0d020 con 0x560b16b69ec0 2024-05-27T07:49:27.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.102+0000 7f903a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16b69ec0 0x560b16cf13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.102+0000 7f9039d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 0x560b16cf18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.102+0000 7f903a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16b69ec0 0x560b16cf13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48144/0 (socket says 172.21.15.114:48144) 2024-05-27T07:49:27.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.102+0000 7f903a59c700 1 -- 172.21.15.114:0/527436307 learned_addr learned my addr 172.21.15.114:0/527436307 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:27.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.102+0000 7f904f3a7b80 1 -- 172.21.15.114:0/527436307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560b16ba8840 con 0x560b16cedcd0 2024-05-27T07:49:27.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.102+0000 7f904f3a7b80 1 -- 172.21.15.114:0/527436307 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560b16cf47c0 con 0x560b16ce8000 2024-05-27T07:49:27.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.102+0000 7f903ad9d700 1 --2- 172.21.15.114:0/527436307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16cedcd0 0x560b16cf1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.102+0000 7f904f3a7b80 10 client.0 _ll_register_callbacks cb 0x560b16ba19a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:27.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.103+0000 7f903ad9d700 1 -- 172.21.15.114:0/527436307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 msgr2=0x560b16cf18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:27.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.103+0000 7f903ad9d700 1 --2- 172.21.15.114:0/527436307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 0x560b16cf18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.103+0000 7f903ad9d700 1 -- 172.21.15.114:0/527436307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16b69ec0 msgr2=0x560b16cf13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.103+0000 7f903ad9d700 1 --2- 172.21.15.114:0/527436307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16b69ec0 0x560b16cf13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.103+0000 7f903ad9d700 1 -- 172.21.15.114:0/527436307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b16ba5710 con 0x560b16cedcd0 2024-05-27T07:49:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.103+0000 7f903ad9d700 1 --2- 172.21.15.114:0/527436307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16cedcd0 0x560b16cf1e30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9034009fd0 tx=0x7f9034004320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccba5b780ab03b2d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.103+0000 7f9021ffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:49:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.103+0000 7f9021ffb700 1 -- 172.21.15.114:0/527436307 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903401a020 con 0x560b16cedcd0 2024-05-27T07:49:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.103+0000 7f904f3a7b80 1 -- 172.21.15.114:0/527436307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9030012d60 con 0x560b16cedcd0 2024-05-27T07:49:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.103+0000 7f9021ffb700 1 -- 172.21.15.114:0/527436307 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9538+0+0 (secure 0 0 0) 0x7f9034007910 con 0x560b16cedcd0 2024-05-27T07:49:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.103+0000 7f9021ffb700 1 -- 172.21.15.114:0/527436307 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f9034025050 con 0x560b16cedcd0 2024-05-27T07:49:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.103+0000 7f904f3a7b80 10 client.27188 fetch_fsmap learned FSMap version 53 2024-05-27T07:49:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.103+0000 7f904f3a7b80 1 -- 172.21.15.114:0/527436307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x560b16cff250 con 0x560b16cedcd0 2024-05-27T07:49:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.104+0000 7f9021ffb700 1 -- 172.21.15.114:0/527436307 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9030012d60 con 0x560b16cedcd0 2024-05-27T07:49:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.104+0000 7f9021ffb700 1 -- 172.21.15.114:0/527436307 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 53) ==== 7120+0+0 (secure 0 0 0) 0x7f9034016bc0 con 0x560b16cedcd0 2024-05-27T07:49:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.104+0000 7f904f3a7b80 10 client.27188 fetch_fsmap finished waiting for FSMap version 53 2024-05-27T07:49:27.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.104+0000 7f904f3a7b80 10 client.27188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:27.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.104+0000 7f904f3a7b80 20 client.27188 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:27.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.104+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/527436307 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560b16d01890 0x560b16d03cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.104+0000 7f904f3a7b80 4 client.27188 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:49:27.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.104+0000 7f904f3a7b80 1 -- 172.21.15.114:0/527436307 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b16cb3a70 con 0x560b16d01890 2024-05-27T07:49:27.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.104+0000 7f903a59c700 1 --2- 172.21.15.114:0/527436307 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560b16d01890 0x560b16d03cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.104+0000 7f903a59c700 1 --2- 172.21.15.114:0/527436307 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560b16d01890 0x560b16d03cf0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cbf7eb1ad30c921 server_cookie=8a2a3ed38e5f404 in_seq=0 out_seq=0 2024-05-27T07:49:27.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.104+0000 7f9021ffb700 10 client.27188 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:27.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.105+0000 7f9021ffb700 1 -- 172.21.15.114:0/527436307 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560b16cb3a70 con 0x560b16d01890 2024-05-27T07:49:27.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.105+0000 7f9021ffb700 10 client.27188 handle_command_reply: tid=0 2024-05-27T07:49:27.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.105+0000 7f904f3a7b80 1 client.27188 shutdown 2024-05-27T07:49:27.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.105+0000 7f904f3a7b80 1 -- 172.21.15.114:0/527436307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16cedcd0 msgr2=0x560b16cf1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:27.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.105+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/527436307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16cedcd0 0x560b16cf1e30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9034009fd0 tx=0x7f9034004320 comp rx=0 tx=0).stop 2024-05-27T07:49:27.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.105+0000 7f904f3a7b80 1 -- 172.21.15.114:0/527436307 shutdown_connections 2024-05-27T07:49:27.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.105+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/527436307 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560b16d01890 0x560b16d03cf0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.105+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/527436307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16cedcd0 0x560b16cf1e30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.105+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/527436307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16b69ec0 0x560b16cf13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.105+0000 7f9021ffb700 0 client.27188 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:27.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.105+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/527436307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 0x560b16cf18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.105+0000 7f904f3a7b80 1 -- 172.21.15.114:0/527436307 >> 172.21.15.114:0/527436307 conn(0x560b16bc4b70 msgr2=0x560b16ce6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:27.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.106+0000 7f904f3a7b80 1 -- 172.21.15.114:0/527436307 shutdown_connections 2024-05-27T07:49:27.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.106+0000 7f904f3a7b80 1 -- 172.21.15.114:0/527436307 wait complete. 2024-05-27T07:49:27.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.106+0000 7f904f3a7b80 20 client.27188 trim_cache size 0 max 16384 2024-05-27T07:49:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.127+0000 7f904f3a7b80 1 Processor -- start 2024-05-27T07:49:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.127+0000 7f904f3a7b80 1 -- start start 2024-05-27T07:49:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.127+0000 7f904f3a7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16b69ec0 0x560b16ce7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.127+0000 7f904f3a7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16ce8000 0x560b16cf1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.127+0000 7f904f3a7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16cedcd0 0x560b16cfd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.127+0000 7f904f3a7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f903c1b0770 con 0x560b16ce8000 2024-05-27T07:49:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.127+0000 7f904f3a7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f903c1b0e40 con 0x560b16cedcd0 2024-05-27T07:49:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.127+0000 7f904f3a7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f903c1ae5e0 con 0x560b16b69ec0 2024-05-27T07:49:27.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.127+0000 7f9039d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16ce8000 0x560b16cf1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.127+0000 7f9039d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16ce8000 0x560b16cf1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48150/0 (socket says 172.21.15.114:48150) 2024-05-27T07:49:27.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.127+0000 7f9039d9b700 1 -- 172.21.15.114:0/3004020830 learned_addr learned my addr 172.21.15.114:0/3004020830 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:27.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.128+0000 7f903a59c700 1 --2- 172.21.15.114:0/3004020830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16b69ec0 0x560b16ce7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.128+0000 7f903ad9d700 1 --2- 172.21.15.114:0/3004020830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16cedcd0 0x560b16cfd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.128+0000 7f9039d9b700 1 -- 172.21.15.114:0/3004020830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16b69ec0 msgr2=0x560b16ce7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:27.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.128+0000 7f9039d9b700 1 --2- 172.21.15.114:0/3004020830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16b69ec0 0x560b16ce7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.128+0000 7f9039d9b700 1 -- 172.21.15.114:0/3004020830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16cedcd0 msgr2=0x560b16cfd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:27.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.128+0000 7f9039d9b700 1 --2- 172.21.15.114:0/3004020830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16cedcd0 0x560b16cfd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.128+0000 7f9039d9b700 1 -- 172.21.15.114:0/3004020830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b16cebe10 con 0x560b16ce8000 2024-05-27T07:49:27.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.128+0000 7f903ad9d700 1 --2- 172.21.15.114:0/3004020830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16cedcd0 0x560b16cfd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:49:27.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.128+0000 7f9039d9b700 1 --2- 172.21.15.114:0/3004020830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16ce8000 0x560b16cf1da0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f9030011db0 tx=0x7f9030021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ea257646dbe6e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.128+0000 7f9023fff700 1 -- 172.21.15.114:0/3004020830 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9030038070 con 0x560b16ce8000 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.128+0000 7f9023fff700 1 -- 172.21.15.114:0/3004020830 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90300054f0 con 0x560b16ce8000 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.128+0000 7f9023fff700 1 -- 172.21.15.114:0/3004020830 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9030004db0 con 0x560b16ce8000 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.129+0000 7f904f3a7b80 1 -- 172.21.15.114:0/3004020830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16ce8000 msgr2=0x560b16cf1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.129+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/3004020830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16ce8000 0x560b16cf1da0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f9030011db0 tx=0x7f9030021000 comp rx=0 tx=0).stop 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.129+0000 7f904f3a7b80 1 -- 172.21.15.114:0/3004020830 shutdown_connections 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.129+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/3004020830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16cedcd0 0x560b16cfd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.129+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/3004020830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16b69ec0 0x560b16ce7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.129+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/3004020830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16ce8000 0x560b16cf1da0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.129+0000 7f904f3a7b80 1 -- 172.21.15.114:0/3004020830 >> 172.21.15.114:0/3004020830 conn(0x560b16bc4b70 msgr2=0x560b16ceb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.129+0000 7f904f3a7b80 1 -- 172.21.15.114:0/3004020830 shutdown_connections 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.129+0000 7f904f3a7b80 1 -- 172.21.15.114:0/3004020830 wait complete. 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.129+0000 7f904f3a7b80 1 -- start start 2024-05-27T07:49:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.129+0000 7f904f3a7b80 1 Processor -- start 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f904f3a7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16b69ec0 0x560b16cf6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f904f3a7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 0x560b16cf6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f904f3a7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 0x560b16cf74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f904f3a7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f903c1aec30 con 0x560b16cedcd0 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f904f3a7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f903c1b00a0 con 0x560b16b69ec0 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f904f3a7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f903c1af300 con 0x560b16ce8000 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f904f3a7b80 10 client.0 _ll_register_callbacks cb 0x560b16ba19a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f903ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 0x560b16cf74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f903ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 0x560b16cf74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48160/0 (socket says 172.21.15.114:48160) 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f903ad9d700 1 -- 172.21.15.114:0/2336244381 learned_addr learned my addr 172.21.15.114:0/2336244381 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f903a59c700 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16b69ec0 0x560b16cf6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f9039d9b700 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 0x560b16cf6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f903ad9d700 1 -- 172.21.15.114:0/2336244381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 msgr2=0x560b16cf6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:27.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f903ad9d700 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 0x560b16cf6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f903ad9d700 1 -- 172.21.15.114:0/2336244381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16b69ec0 msgr2=0x560b16cf6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:27.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f903ad9d700 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16b69ec0 0x560b16cf6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f903ad9d700 1 -- 172.21.15.114:0/2336244381 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b16d6c410 con 0x560b16cedcd0 2024-05-27T07:49:27.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f903a59c700 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16b69ec0 0x560b16cf6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:49:27.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f903ad9d700 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 0x560b16cf74a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f90340158b0 tx=0x7f9034004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6141a6523517b5b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:27.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.130+0000 7f9021ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:49:27.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f9021ffb700 1 -- 172.21.15.114:0/2336244381 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903402e070 con 0x560b16cedcd0 2024-05-27T07:49:27.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f9021ffb700 1 -- 172.21.15.114:0/2336244381 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9538+0+0 (secure 0 0 0) 0x7f9034004c10 con 0x560b16cedcd0 2024-05-27T07:49:27.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f904f3a7b80 1 -- 172.21.15.114:0/2336244381 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90300054f0 con 0x560b16cedcd0 2024-05-27T07:49:27.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f9021ffb700 1 -- 172.21.15.114:0/2336244381 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f9034037050 con 0x560b16cedcd0 2024-05-27T07:49:27.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f904f3a7b80 10 client.17727 fetch_fsmap learned FSMap version 53 2024-05-27T07:49:27.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f904f3a7b80 1 -- 172.21.15.114:0/2336244381 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x560b16d6d6b0 con 0x560b16cedcd0 2024-05-27T07:49:27.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f9021ffb700 1 -- 172.21.15.114:0/2336244381 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90300054f0 con 0x560b16cedcd0 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f9021ffb700 1 -- 172.21.15.114:0/2336244381 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 53) ==== 7120+0+0 (secure 0 0 0) 0x7f9034031020 con 0x560b16cedcd0 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f904f3a7b80 10 client.17727 fetch_fsmap finished waiting for FSMap version 53 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f904f3a7b80 10 client.17727 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f904f3a7b80 20 client.17727 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560b16d6de40 0x560b16d70260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f904f3a7b80 4 client.17727 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f904f3a7b80 1 -- 172.21.15.114:0/2336244381 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560b16c18c60 con 0x560b16d6de40 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.131+0000 7f903a59c700 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560b16d6de40 0x560b16d70260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.132+0000 7f9021ffb700 10 client.17727 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.132+0000 7f903a59c700 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560b16d6de40 0x560b16d70260 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53c1409699eb0b83 server_cookie=9d4d656e13494a57 in_seq=0 out_seq=0 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f9021ffb700 1 -- 172.21.15.114:0/2336244381 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560b16c18c60 con 0x560b16d6de40 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f9021ffb700 10 client.17727 handle_command_reply: tid=0 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f904f3a7b80 1 client.17727 shutdown 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f904f3a7b80 1 -- 172.21.15.114:0/2336244381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 msgr2=0x560b16cf74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:27.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 0x560b16cf74a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f90340158b0 tx=0x7f9034004670 comp rx=0 tx=0).stop 2024-05-27T07:49:27.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f904f3a7b80 1 -- 172.21.15.114:0/2336244381 shutdown_connections 2024-05-27T07:49:27.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560b16cedcd0 0x560b16cf74a0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560b16d6de40 0x560b16d70260 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560b16b69ec0 0x560b16cf6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f904f3a7b80 1 --2- 172.21.15.114:0/2336244381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560b16ce8000 0x560b16cf6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f9021ffb700 0 client.17727 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:27.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f904f3a7b80 1 -- 172.21.15.114:0/2336244381 >> 172.21.15.114:0/2336244381 conn(0x560b16bc4b70 msgr2=0x560b16ceb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:27.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f904f3a7b80 1 -- 172.21.15.114:0/2336244381 shutdown_connections 2024-05-27T07:49:27.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.133+0000 7f904f3a7b80 1 -- 172.21.15.114:0/2336244381 wait complete. 2024-05-27T07:49:27.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.134+0000 7f904f3a7b80 20 client.17727 trim_cache size 0 max 16384 2024-05-27T07:49:27.146 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:49:27.146 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:49:27.146 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "9a5af139-bf71-41fa-be17-a9e48921edcd", 2024-05-27T07:49:27.146 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:49:27.146 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:49:27.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.134+0000 7f9040992700 1 -- 172.21.15.114:0/2044570921 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f90240b29f0 msgr2=0x7f90240b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:27.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.134+0000 7f9040992700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f90240b29f0 0x7f90240b4f00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f903c100090 tx=0x7f9030009260 comp rx=0 tx=0).stop 2024-05-27T07:49:27.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.134+0000 7f9040992700 1 -- 172.21.15.114:0/2044570921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f903c1029b0 msgr2=0x7f903c19ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:27.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.134+0000 7f9040992700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f903c1029b0 0x7f903c19ae20 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f902c000c00 tx=0x7f902c009580 comp rx=0 tx=0).stop 2024-05-27T07:49:27.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.134+0000 7f9040992700 1 -- 172.21.15.114:0/2044570921 shutdown_connections 2024-05-27T07:49:27.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.134+0000 7f9040992700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c1089e0 0x7f903c19f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.134+0000 7f9040992700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f903c1029b0 0x7f903c19ae20 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.134+0000 7f9040992700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f90240b29f0 0x7f90240b4f00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.134+0000 7f9040992700 1 --2- 172.21.15.114:0/2044570921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f903c103300 0x7f903c19b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:27.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.134+0000 7f9040992700 1 -- 172.21.15.114:0/2044570921 >> 172.21.15.114:0/2044570921 conn(0x7f903c0fe510 msgr2=0x7f903c100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:27.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.134+0000 7f9040992700 1 -- 172.21.15.114:0/2044570921 shutdown_connections 2024-05-27T07:49:27.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:27.134+0000 7f9040992700 1 -- 172.21.15.114:0/2044570921 wait complete. 2024-05-27T07:49:27.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 976956 10.72 MB/sec execute 339 sec latency 1809.273 ms 2024-05-27T07:49:27.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:27 smithi163 ceph-mon[36451]: pgmap v318: 129 pgs: 129 active+clean; 88 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 5.1 MiB/s wr, 336 op/s 2024-05-27T07:49:27.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:27 smithi163 ceph-mon[36451]: daemon mds.h restarted 2024-05-27T07:49:27.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:27 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:49:27.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:27 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:49:27.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:27 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:49:27.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:27 smithi186 ceph-mon[36647]: pgmap v318: 129 pgs: 129 active+clean; 88 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 5.1 MiB/s wr, 336 op/s 2024-05-27T07:49:27.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:27 smithi186 ceph-mon[36647]: daemon mds.h restarted 2024-05-27T07:49:27.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:27 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:49:27.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:27 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:49:27.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:27 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:49:27.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:27 smithi114 ceph-mon[32038]: pgmap v318: 129 pgs: 129 active+clean; 88 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 5.1 MiB/s wr, 336 op/s 2024-05-27T07:49:27.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:27 smithi114 ceph-mon[32038]: daemon mds.h restarted 2024-05-27T07:49:27.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:27 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:49:27.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:27 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:49:27.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:27 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:49:27.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:49:28.055 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:49:28.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 976956 10.69 MB/sec execute 340 sec latency 2809.403 ms 2024-05-27T07:49:28.442 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:28 smithi114 ceph-mon[32038]: osdmap e115: 12 total, 12 up, 12 in 2024-05-27T07:49:28.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:28 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6836/2596932959,v1:172.21.15.163:6837/2596932959] up:boot 2024-05-27T07:49:28.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:28 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 9 up:standby 2024-05-27T07:49:28.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:28 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:49:28.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:28 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:49:28.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:28 smithi163 ceph-mon[36451]: osdmap e115: 12 total, 12 up, 12 in 2024-05-27T07:49:28.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:28 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6836/2596932959,v1:172.21.15.163:6837/2596932959] up:boot 2024-05-27T07:49:28.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:28 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 9 up:standby 2024-05-27T07:49:28.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:28 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:49:28.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:28 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:49:28.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:28 smithi186 ceph-mon[36647]: osdmap e115: 12 total, 12 up, 12 in 2024-05-27T07:49:28.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:28 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6836/2596932959,v1:172.21.15.163:6837/2596932959] up:boot 2024-05-27T07:49:28.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:28 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 9 up:standby 2024-05-27T07:49:28.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:28 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:49:28.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:28 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:49:29.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 976956 10.65 MB/sec execute 341 sec latency 3809.454 ms 2024-05-27T07:49:29.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:29 smithi114 ceph-mon[32038]: pgmap v320: 129 pgs: 129 active+clean; 88 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 5.7 MiB/s wr, 374 op/s 2024-05-27T07:49:29.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:29 smithi163 ceph-mon[36451]: pgmap v320: 129 pgs: 129 active+clean; 88 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 5.7 MiB/s wr, 374 op/s 2024-05-27T07:49:29.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:29 smithi186 ceph-mon[36647]: pgmap v320: 129 pgs: 129 active+clean; 88 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 5.7 MiB/s wr, 374 op/s 2024-05-27T07:49:29.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.883+0000 7fb8e373f700 1 -- 172.21.15.114:0/3537633909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc108f30 msgr2=0x7fb8dc109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:29.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.883+0000 7fb8e373f700 1 --2- 172.21.15.114:0/3537633909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc108f30 0x7fb8dc109340 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb8d80099c0 tx=0x7fb8d8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.884+0000 7fb8e373f700 1 -- 172.21.15.114:0/3537633909 shutdown_connections 2024-05-27T07:49:29.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.884+0000 7fb8e373f700 1 --2- 172.21.15.114:0/3537633909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8dc109880 0x7fb8dc1111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.884+0000 7fb8e373f700 1 --2- 172.21.15.114:0/3537633909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8dc104600 0x7fb8dc104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.884+0000 7fb8e373f700 1 --2- 172.21.15.114:0/3537633909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc108f30 0x7fb8dc109340 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.884+0000 7fb8e373f700 1 -- 172.21.15.114:0/3537633909 >> 172.21.15.114:0/3537633909 conn(0x7fb8dc07af60 msgr2=0x7fb8dc07b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:29.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.884+0000 7fb8e373f700 1 -- 172.21.15.114:0/3537633909 shutdown_connections 2024-05-27T07:49:29.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.884+0000 7fb8e373f700 1 -- 172.21.15.114:0/3537633909 wait complete. 2024-05-27T07:49:29.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.884+0000 7fb8e373f700 1 Processor -- start 2024-05-27T07:49:29.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.884+0000 7fb8e373f700 1 -- start start 2024-05-27T07:49:29.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e373f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc104600 0x7fb8dc075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e373f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8dc108f30 0x7fb8dc075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e373f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8dc109880 0x7fb8dc076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e373f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc114490 con 0x7fb8dc104600 2024-05-27T07:49:29.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e373f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc114310 con 0x7fb8dc108f30 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e373f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc114610 con 0x7fb8dc109880 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e14db700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc104600 0x7fb8dc075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e14db700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc104600 0x7fb8dc075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48178/0 (socket says 172.21.15.114:48178) 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e14db700 1 -- 172.21.15.114:0/2468033047 learned_addr learned my addr 172.21.15.114:0/2468033047 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e0cda700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8dc108f30 0x7fb8dc075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e1cdc700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8dc109880 0x7fb8dc076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e14db700 1 -- 172.21.15.114:0/2468033047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8dc109880 msgr2=0x7fb8dc076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e14db700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8dc109880 0x7fb8dc076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e14db700 1 -- 172.21.15.114:0/2468033047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8dc108f30 msgr2=0x7fb8dc075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e14db700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8dc108f30 0x7fb8dc075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e14db700 1 -- 172.21.15.114:0/2468033047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8dc1a91c0 con 0x7fb8dc104600 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e1cdc700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8dc109880 0x7fb8dc076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.885+0000 7fb8e14db700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc104600 0x7fb8dc075ab0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00c920 tx=0x7fb8cc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.886+0000 7fb8d27fc700 1 -- 172.21.15.114:0/2468033047 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc007b00 con 0x7fb8dc104600 2024-05-27T07:49:29.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.886+0000 7fb8e373f700 1 -- 172.21.15.114:0/2468033047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8dc1a94b0 con 0x7fb8dc104600 2024-05-27T07:49:29.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.886+0000 7fb8e373f700 1 -- 172.21.15.114:0/2468033047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8dc1a9a10 con 0x7fb8dc104600 2024-05-27T07:49:29.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.886+0000 7fb8d27fc700 1 -- 172.21.15.114:0/2468033047 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8cc004ce0 con 0x7fb8dc104600 2024-05-27T07:49:29.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.886+0000 7fb8d27fc700 1 -- 172.21.15.114:0/2468033047 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc005500 con 0x7fb8dc104600 2024-05-27T07:49:29.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.888+0000 7fb8d27fc700 1 -- 172.21.15.114:0/2468033047 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fb8cc00f450 con 0x7fb8dc104600 2024-05-27T07:49:29.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.889+0000 7fb8f2154b80 1 Processor -- start 2024-05-27T07:49:29.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.889+0000 7fb8d27fc700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8c80b2a70 0x7fb8c80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.889+0000 7fb8f2154b80 1 -- start start 2024-05-27T07:49:29.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8e1cdc700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8c80b2a70 0x7fb8c80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8d27fc700 1 -- 172.21.15.114:0/2468033047 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9466+0+0 (secure 0 0 0) 0x7fb8cc0e2470 con 0x7fb8dc104600 2024-05-27T07:49:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8f2154b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 0x55655082f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8f2154b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55655082fc50 0x556550829a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8f2154b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 0x55655082a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8f2154b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556550758c90 con 0x5565506abe40 2024-05-27T07:49:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8f2154b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556550748830 con 0x55655082fc50 2024-05-27T07:49:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8f2154b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55655074c870 con 0x556550829f80 2024-05-27T07:49:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8e14db700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 0x55655082f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8e14db700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 0x55655082f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48182/0 (socket says 172.21.15.114:48182) 2024-05-27T07:49:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8e14db700 1 -- 172.21.15.114:0/3180985274 learned_addr learned my addr 172.21.15.114:0/3180985274 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8e0cda700 1 --2- 172.21.15.114:0/3180985274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55655082fc50 0x556550829a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8e1cdc700 1 --2- 172.21.15.114:0/3180985274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 0x55655082a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8e1cdc700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8c80b2a70 0x7fb8c80b4f80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400e270 tx=0x7fb8d4009040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:29.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8e14db700 1 -- 172.21.15.114:0/3180985274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 msgr2=0x55655082a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:29.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8e14db700 1 --2- 172.21.15.114:0/3180985274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 0x55655082a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8e14db700 1 -- 172.21.15.114:0/3180985274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55655082fc50 msgr2=0x556550829a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:29.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8e14db700 1 --2- 172.21.15.114:0/3180985274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55655082fc50 0x556550829a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8e14db700 1 -- 172.21.15.114:0/3180985274 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565506988d0 con 0x5565506abe40 2024-05-27T07:49:29.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.890+0000 7fb8e14db700 1 --2- 172.21.15.114:0/3180985274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 0x55655082f710 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc0a9140 tx=0x7fb8cc0a8fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a75f4e7e2a1266 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:29.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.891+0000 7fb8c77fe700 1 -- 172.21.15.114:0/3180985274 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc0ad7f0 con 0x5565506abe40 2024-05-27T07:49:29.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.891+0000 7fb8c77fe700 1 -- 172.21.15.114:0/3180985274 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8cc0b22a0 con 0x5565506abe40 2024-05-27T07:49:29.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.891+0000 7fb8c77fe700 1 -- 172.21.15.114:0/3180985274 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc0a9900 con 0x5565506abe40 2024-05-27T07:49:29.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.891+0000 7fb8f2154b80 1 -- 172.21.15.114:0/3180985274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 msgr2=0x55655082f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:29.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.891+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/3180985274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 0x55655082f710 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc0a9140 tx=0x7fb8cc0a8fd0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.892+0000 7fb8f2154b80 1 -- 172.21.15.114:0/3180985274 shutdown_connections 2024-05-27T07:49:29.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.892+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/3180985274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 0x55655082a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.892+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/3180985274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 0x55655082f710 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.892+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/3180985274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55655082fc50 0x556550829a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.892+0000 7fb8f2154b80 1 -- 172.21.15.114:0/3180985274 >> 172.21.15.114:0/3180985274 conn(0x556550706af0 msgr2=0x556550824640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.892+0000 7fb8f2154b80 1 -- 172.21.15.114:0/3180985274 shutdown_connections 2024-05-27T07:49:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.892+0000 7fb8f2154b80 1 -- 172.21.15.114:0/3180985274 wait complete. 2024-05-27T07:49:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.892+0000 7fb8f2154b80 1 -- start start 2024-05-27T07:49:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.892+0000 7fb8f2154b80 1 Processor -- start 2024-05-27T07:49:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8f2154b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 0x556550833480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8f2154b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556550829f80 0x5565508339c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8f2154b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55655082fc50 0x556550833f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8f2154b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55655064efa0 con 0x5565506abe40 2024-05-27T07:49:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8f2154b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5565506ea7c0 con 0x556550829f80 2024-05-27T07:49:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8f2154b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5565508366e0 con 0x55655082fc50 2024-05-27T07:49:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8f2154b80 10 client.0 _ll_register_callbacks cb 0x5565506e3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8e14db700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 0x556550833480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8e14db700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 0x556550833480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48190/0 (socket says 172.21.15.114:48190) 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8e14db700 1 -- 172.21.15.114:0/1222323646 learned_addr learned my addr 172.21.15.114:0/1222323646 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8e0cda700 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556550829f80 0x5565508339c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8e1cdc700 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55655082fc50 0x556550833f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8e14db700 1 -- 172.21.15.114:0/1222323646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55655082fc50 msgr2=0x556550833f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8e14db700 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55655082fc50 0x556550833f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8e14db700 1 -- 172.21.15.114:0/1222323646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556550829f80 msgr2=0x5565508339c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8e14db700 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556550829f80 0x5565508339c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8e14db700 1 -- 172.21.15.114:0/1222323646 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565506e7690 con 0x5565506abe40 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8e0cda700 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556550829f80 0x5565508339c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8e1cdc700 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55655082fc50 0x556550833f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8e14db700 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 0x556550833480 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc0a93d0 tx=0x7fb8cc0dfde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f3b90bf2265fa0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8c4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8c4ff9700 1 -- 172.21.15.114:0/1222323646 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc0c8060 con 0x5565506abe40 2024-05-27T07:49:29.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8c4ff9700 1 -- 172.21.15.114:0/1222323646 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9466+0+0 (secure 0 0 0) 0x7fb8cc0a96a0 con 0x5565506abe40 2024-05-27T07:49:29.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.893+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1222323646 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8cc0b22a0 con 0x5565506abe40 2024-05-27T07:49:29.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.894+0000 7fb8c4ff9700 1 -- 172.21.15.114:0/1222323646 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fb8cc0c2030 con 0x5565506abe40 2024-05-27T07:49:29.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.894+0000 7fb8f2154b80 10 client.17739 fetch_fsmap learned FSMap version 55 2024-05-27T07:49:29.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.894+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1222323646 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x556550841230 con 0x5565506abe40 2024-05-27T07:49:29.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.894+0000 7fb8c4ff9700 1 -- 172.21.15.114:0/1222323646 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8cc0b22a0 con 0x5565506abe40 2024-05-27T07:49:29.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.894+0000 7fb8c4ff9700 1 -- 172.21.15.114:0/1222323646 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 55) ==== 7120+0+0 (secure 0 0 0) 0x7fb8cc0b0b70 con 0x5565506abe40 2024-05-27T07:49:29.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.894+0000 7fb8f2154b80 10 client.17739 fetch_fsmap finished waiting for FSMap version 55 2024-05-27T07:49:29.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.894+0000 7fb8f2154b80 10 client.17739 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:29.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.894+0000 7fb8f2154b80 20 client.17739 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:29.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.894+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556550843700 0x556550845b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.894+0000 7fb8f2154b80 4 client.17739 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:49:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.894+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1222323646 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565507f59f0 con 0x556550843700 2024-05-27T07:49:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.894+0000 7fb8e0cda700 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556550843700 0x556550845b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.895+0000 7fb8c4ff9700 10 client.17739 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.895+0000 7fb8e0cda700 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556550843700 0x556550845b20 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db76f973520da77e server_cookie=921d5ac541f1af03 in_seq=0 out_seq=0 2024-05-27T07:49:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.895+0000 7fb8c4ff9700 1 -- 172.21.15.114:0/1222323646 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5565507f59f0 con 0x556550843700 2024-05-27T07:49:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.895+0000 7fb8c4ff9700 10 client.17739 handle_command_reply: tid=0 2024-05-27T07:49:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.895+0000 7fb8f2154b80 1 client.17739 shutdown 2024-05-27T07:49:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.896+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1222323646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 msgr2=0x556550833480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.896+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 0x556550833480 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc0a93d0 tx=0x7fb8cc0dfde0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.896+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1222323646 shutdown_connections 2024-05-27T07:49:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.896+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55655082fc50 0x556550833f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.896+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565506abe40 0x556550833480 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.896+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556550843700 0x556550845b20 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.896+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/1222323646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556550829f80 0x5565508339c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.896+0000 7fb8c4ff9700 0 client.17739 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.896+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1222323646 >> 172.21.15.114:0/1222323646 conn(0x556550706af0 msgr2=0x556550828a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.896+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1222323646 shutdown_connections 2024-05-27T07:49:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.896+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1222323646 wait complete. 2024-05-27T07:49:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.896+0000 7fb8f2154b80 20 client.17739 trim_cache size 0 max 16384 2024-05-27T07:49:29.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8f2154b80 1 Processor -- start 2024-05-27T07:49:29.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8f2154b80 1 -- start start 2024-05-27T07:49:29.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8f2154b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565506abe40 0x556550829a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8f2154b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 0x5565508327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8f2154b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 0x55655082d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8f2154b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc1b29c0 con 0x55655082fc50 2024-05-27T07:49:29.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8f2154b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc1b3090 con 0x5565506abe40 2024-05-27T07:49:29.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8f2154b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc1b0810 con 0x556550829f80 2024-05-27T07:49:29.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8e1cdc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 0x55655082d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8e1cdc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 0x55655082d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48200/0 (socket says 172.21.15.114:48200) 2024-05-27T07:49:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8e1cdc700 1 -- 172.21.15.114:0/3831031863 learned_addr learned my addr 172.21.15.114:0/3831031863 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8e0cda700 1 --2- 172.21.15.114:0/3831031863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 0x5565508327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8e14db700 1 --2- 172.21.15.114:0/3831031863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565506abe40 0x556550829a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8e1cdc700 1 -- 172.21.15.114:0/3831031863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 msgr2=0x5565508327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8e1cdc700 1 --2- 172.21.15.114:0/3831031863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 0x5565508327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8e1cdc700 1 -- 172.21.15.114:0/3831031863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565506abe40 msgr2=0x556550829a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8e1cdc700 1 --2- 172.21.15.114:0/3831031863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565506abe40 0x556550829a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8e1cdc700 1 -- 172.21.15.114:0/3831031863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556550825780 con 0x55655082fc50 2024-05-27T07:49:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.916+0000 7fb8e1cdc700 1 --2- 172.21.15.114:0/3831031863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 0x55655082d010 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4003b00 tx=0x7fb8d4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab59fdec96132 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.917+0000 7fb8c6ffd700 1 -- 172.21.15.114:0/3831031863 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d40046e0 con 0x55655082fc50 2024-05-27T07:49:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.917+0000 7fb8c6ffd700 1 -- 172.21.15.114:0/3831031863 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8d4025070 con 0x55655082fc50 2024-05-27T07:49:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.917+0000 7fb8c6ffd700 1 -- 172.21.15.114:0/3831031863 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4020670 con 0x55655082fc50 2024-05-27T07:49:29.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.917+0000 7fb8f2154b80 1 -- 172.21.15.114:0/3831031863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 msgr2=0x55655082d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:29.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.917+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/3831031863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 0x55655082d010 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4003b00 tx=0x7fb8d4011000 comp rx=0 tx=0).stop 2024-05-27T07:49:29.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.917+0000 7fb8f2154b80 1 -- 172.21.15.114:0/3831031863 shutdown_connections 2024-05-27T07:49:29.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.917+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/3831031863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 0x55655082d010 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.917+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/3831031863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565506abe40 0x556550829a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.917+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/3831031863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 0x5565508327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.917+0000 7fb8f2154b80 1 -- 172.21.15.114:0/3831031863 >> 172.21.15.114:0/3831031863 conn(0x556550706af0 msgr2=0x55655081ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.918+0000 7fb8f2154b80 1 -- 172.21.15.114:0/3831031863 shutdown_connections 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.918+0000 7fb8f2154b80 1 -- 172.21.15.114:0/3831031863 wait complete. 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.918+0000 7fb8f2154b80 1 -- start start 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.918+0000 7fb8f2154b80 1 Processor -- start 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.918+0000 7fb8f2154b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565506abe40 0x556550846440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.918+0000 7fb8f2154b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 0x556550846980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.918+0000 7fb8f2154b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 0x556550846ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.918+0000 7fb8f2154b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc1b0e80 con 0x55655082fc50 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.918+0000 7fb8f2154b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc1b22f0 con 0x5565506abe40 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.918+0000 7fb8f2154b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc1b1550 con 0x556550829f80 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.918+0000 7fb8f2154b80 10 client.0 _ll_register_callbacks cb 0x5565506e3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8e1cdc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 0x556550846ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8e14db700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565506abe40 0x556550846440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8e14db700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565506abe40 0x556550846440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:59968/0 (socket says 172.21.15.114:59968) 2024-05-27T07:49:29.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8e14db700 1 -- 172.21.15.114:0/1916234330 learned_addr learned my addr 172.21.15.114:0/1916234330 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:29.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8e1cdc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 0x556550846ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48204/0 (socket says 172.21.15.114:48204) 2024-05-27T07:49:29.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8e0cda700 1 --2- 172.21.15.114:0/1916234330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 0x556550846980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8e1cdc700 1 -- 172.21.15.114:0/1916234330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 msgr2=0x556550846980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:29.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8e1cdc700 1 --2- 172.21.15.114:0/1916234330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 0x556550846980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8e1cdc700 1 -- 172.21.15.114:0/1916234330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565506abe40 msgr2=0x556550846440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:29.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8e1cdc700 1 --2- 172.21.15.114:0/1916234330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565506abe40 0x556550846440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8e1cdc700 1 -- 172.21.15.114:0/1916234330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565508acf70 con 0x55655082fc50 2024-05-27T07:49:29.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8e14db700 1 --2- 172.21.15.114:0/1916234330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565506abe40 0x556550846440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:29.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8e1cdc700 1 --2- 172.21.15.114:0/1916234330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 0x556550846ec0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400efe0 tx=0x7fb8d4004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17fb894ba5f7ee39 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:29.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8c4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:49:29.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8c4ff9700 1 -- 172.21.15.114:0/1916234330 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4015800 con 0x55655082fc50 2024-05-27T07:49:29.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8c4ff9700 1 -- 172.21.15.114:0/1916234330 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9466+0+0 (secure 0 0 0) 0x7fb8d40159a0 con 0x55655082fc50 2024-05-27T07:49:29.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1916234330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8d4025070 con 0x55655082fc50 2024-05-27T07:49:29.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8c4ff9700 1 -- 172.21.15.114:0/1916234330 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fb8d403a050 con 0x55655082fc50 2024-05-27T07:49:29.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8f2154b80 10 client.17745 fetch_fsmap learned FSMap version 55 2024-05-27T07:49:29.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.919+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1916234330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5565508ae0b0 con 0x55655082fc50 2024-05-27T07:49:29.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.920+0000 7fb8c4ff9700 1 -- 172.21.15.114:0/1916234330 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8d4025070 con 0x55655082fc50 2024-05-27T07:49:29.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.920+0000 7fb8c4ff9700 1 -- 172.21.15.114:0/1916234330 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 55) ==== 7120+0+0 (secure 0 0 0) 0x7fb8d402f020 con 0x55655082fc50 2024-05-27T07:49:29.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.920+0000 7fb8f2154b80 10 client.17745 fetch_fsmap finished waiting for FSMap version 55 2024-05-27T07:49:29.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.920+0000 7fb8f2154b80 10 client.17745 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:29.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.920+0000 7fb8f2154b80 20 client.17745 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.920+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/1916234330 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5565508ae6b0 0x5565508b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.920+0000 7fb8f2154b80 4 client.17745 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.920+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1916234330 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55655075abe0 con 0x5565508ae6b0 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.920+0000 7fb8e14db700 1 --2- 172.21.15.114:0/1916234330 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5565508ae6b0 0x5565508b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8c4ff9700 10 client.17745 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8e14db700 1 --2- 172.21.15.114:0/1916234330 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5565508ae6b0 0x5565508b0ad0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f8164b1d1f0a09c server_cookie=411f2202416823f9 in_seq=0 out_seq=0 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8c4ff9700 1 -- 172.21.15.114:0/1916234330 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55655075abe0 con 0x5565508ae6b0 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8c4ff9700 10 client.17745 handle_command_reply: tid=0 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8f2154b80 1 client.17745 shutdown 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1916234330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 msgr2=0x556550846ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/1916234330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 0x556550846ec0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400efe0 tx=0x7fb8d4004990 comp rx=0 tx=0).stop 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1916234330 shutdown_connections 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/1916234330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55655082fc50 0x556550846ec0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/1916234330 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5565508ae6b0 0x5565508b0ad0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/1916234330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565506abe40 0x556550846440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8f2154b80 1 --2- 172.21.15.114:0/1916234330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556550829f80 0x556550846980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8c4ff9700 0 client.17745 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:29.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1916234330 >> 172.21.15.114:0/1916234330 conn(0x556550706af0 msgr2=0x55655081ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:29.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1916234330 shutdown_connections 2024-05-27T07:49:29.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8f2154b80 1 -- 172.21.15.114:0/1916234330 wait complete. 2024-05-27T07:49:29.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.921+0000 7fb8f2154b80 20 client.17745 trim_cache size 0 max 16384 2024-05-27T07:49:29.928 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:49:29.928 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "scrub active (1 inodes in the stack)", 2024-05-27T07:49:29.928 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:49:29.928 INFO:teuthology.orchestra.run.smithi114.stdout: "9a5af139-bf71-41fa-be17-a9e48921edcd": { 2024-05-27T07:49:29.928 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:49:29.928 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "9a5af139-bf71-41fa-be17-a9e48921edcd", 2024-05-27T07:49:29.928 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:49:29.928 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:49:29.929 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:49:29.929 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:49:29.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.922+0000 7fb8e373f700 1 -- 172.21.15.114:0/2468033047 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8c80b2a70 msgr2=0x7fb8c80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:29.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.922+0000 7fb8e373f700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8c80b2a70 0x7fb8c80b4f80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400e270 tx=0x7fb8d4009040 comp rx=0 tx=0).stop 2024-05-27T07:49:29.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.922+0000 7fb8e373f700 1 -- 172.21.15.114:0/2468033047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc104600 msgr2=0x7fb8dc075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:29.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.922+0000 7fb8e373f700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc104600 0x7fb8dc075ab0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00c920 tx=0x7fb8cc00cc30 comp rx=0 tx=0).stop 2024-05-27T07:49:29.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.922+0000 7fb8e373f700 1 -- 172.21.15.114:0/2468033047 shutdown_connections 2024-05-27T07:49:29.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.922+0000 7fb8e373f700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8dc109880 0x7fb8dc076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.922+0000 7fb8e373f700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc104600 0x7fb8dc075ab0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.922+0000 7fb8e373f700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb8c80b2a70 0x7fb8c80b4f80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.922+0000 7fb8e373f700 1 --2- 172.21.15.114:0/2468033047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8dc108f30 0x7fb8dc075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:29.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.922+0000 7fb8e373f700 1 -- 172.21.15.114:0/2468033047 >> 172.21.15.114:0/2468033047 conn(0x7fb8dc07af60 msgr2=0x7fb8dc101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:29.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.922+0000 7fb8e373f700 1 -- 172.21.15.114:0/2468033047 shutdown_connections 2024-05-27T07:49:29.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:49:29.922+0000 7fb8e373f700 1 -- 172.21.15.114:0/2468033047 wait complete. 2024-05-27T07:49:30.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 976956 10.62 MB/sec execute 342 sec latency 4809.499 ms 2024-05-27T07:49:30.686 INFO:tasks.cephfs.filesystem:scrub status for tag:9a5af139-bf71-41fa-be17-a9e48921edcd - {'path': '/', 'tag': '9a5af139-bf71-41fa-be17-a9e48921edcd', 'options': 'recursive,force'} 2024-05-27T07:49:31.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 976956 10.59 MB/sec execute 343 sec latency 5809.574 ms 2024-05-27T07:49:31.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:31 smithi163 ceph-mon[36451]: pgmap v321: 129 pgs: 129 active+clean; 91 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 4.3 MiB/s wr, 281 op/s 2024-05-27T07:49:31.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:31 smithi186 ceph-mon[36647]: pgmap v321: 129 pgs: 129 active+clean; 91 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 4.3 MiB/s wr, 281 op/s 2024-05-27T07:49:31.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:31 smithi114 ceph-mon[32038]: pgmap v321: 129 pgs: 129 active+clean; 91 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 4.3 MiB/s wr, 281 op/s 2024-05-27T07:49:32.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 977009 10.56 MB/sec execute 344 sec latency 6754.269 ms 2024-05-27T07:49:33.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 977822 10.54 MB/sec execute 345 sec latency 52.553 ms 2024-05-27T07:49:33.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:33 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:49:33.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:33 smithi163 ceph-mon[36451]: pgmap v322: 129 pgs: 129 active+clean; 96 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 175 op/s 2024-05-27T07:49:33.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:33 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:49:33.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:33 smithi186 ceph-mon[36647]: pgmap v322: 129 pgs: 129 active+clean; 96 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 175 op/s 2024-05-27T07:49:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.729+0000 7f381c357640 1 -- 172.21.15.114:0/2266998838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3814109030 msgr2=0x7f3814102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.729+0000 7f381c357640 1 --2- 172.21.15.114:0/2266998838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3814109030 0x7f3814102c70 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f3804009900 tx=0x7f380402a410 comp rx=0 tx=0).stop 2024-05-27T07:49:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.729+0000 7f381c357640 1 -- 172.21.15.114:0/2266998838 shutdown_connections 2024-05-27T07:49:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.729+0000 7f381c357640 1 --2- 172.21.15.114:0/2266998838 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38141032e0 0x7f381410f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.729+0000 7f381c357640 1 --2- 172.21.15.114:0/2266998838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3814109030 0x7f3814102c70 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.729+0000 7f381c357640 1 --2- 172.21.15.114:0/2266998838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3814108b90 0x7f3814102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.729+0000 7f381c357640 1 -- 172.21.15.114:0/2266998838 >> 172.21.15.114:0/2266998838 conn(0x7f38140782d0 msgr2=0x7f3814078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.729+0000 7f381c357640 1 -- 172.21.15.114:0/2266998838 shutdown_connections 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.729+0000 7f381c357640 1 -- 172.21.15.114:0/2266998838 wait complete. 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381c357640 1 Processor -- start 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381c357640 1 -- start start 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381c357640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38141032e0 0x7f3814199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381c357640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3814108b90 0x7f3814199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381c357640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3814109030 0x7f381419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381c357640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3814112d80 con 0x7f38141032e0 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381c357640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3814112c00 con 0x7f3814108b90 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381c357640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3814112f00 con 0x7f3814109030 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381a0f3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38141032e0 0x7f3814199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381a0f3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38141032e0 0x7f3814199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48214/0 (socket says 172.21.15.114:48214) 2024-05-27T07:49:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381a0f3640 1 -- 172.21.15.114:0/768613089 learned_addr learned my addr 172.21.15.114:0/768613089 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f38198f2640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3814108b90 0x7f3814199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381a8f4640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3814109030 0x7f381419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381a0f3640 1 -- 172.21.15.114:0/768613089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3814109030 msgr2=0x7f381419e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381a0f3640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3814109030 0x7f381419e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381a0f3640 1 -- 172.21.15.114:0/768613089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3814108b90 msgr2=0x7f3814199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381a0f3640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3814108b90 0x7f3814199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381a0f3640 1 -- 172.21.15.114:0/768613089 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381419e660 con 0x7f38141032e0 2024-05-27T07:49:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.730+0000 7f381a8f4640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3814109030 0x7f381419e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:49:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.731+0000 7f381a0f3640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38141032e0 0x7f3814199800 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f381000ebc0 tx=0x7f381000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.731+0000 7f380b7fe640 1 -- 172.21.15.114:0/768613089 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810004d20 con 0x7f38141032e0 2024-05-27T07:49:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.731+0000 7f380b7fe640 1 -- 172.21.15.114:0/768613089 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38100054d0 con 0x7f38141032e0 2024-05-27T07:49:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.731+0000 7f381c357640 1 -- 172.21.15.114:0/768613089 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f381419e950 con 0x7f38141032e0 2024-05-27T07:49:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.731+0000 7f380b7fe640 1 -- 172.21.15.114:0/768613089 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810010720 con 0x7f38141032e0 2024-05-27T07:49:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.731+0000 7f381c357640 1 -- 172.21.15.114:0/768613089 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f381419ee20 con 0x7f38141032e0 2024-05-27T07:49:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f3823208740 1 Processor -- start 2024-05-27T07:49:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f3823208740 1 -- start start 2024-05-27T07:49:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f380b7fe640 1 -- 172.21.15.114:0/768613089 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f3810005020 con 0x7f38141032e0 2024-05-27T07:49:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f3823208740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 0x5569a80b1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f3823208740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 0x5569a80b1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f3823208740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7cef4d0 0x5569a80b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f3823208740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5569a8023c30 con 0x5569a7d101b0 2024-05-27T07:49:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f3823208740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5569a80137d0 con 0x5569a7cef4d0 2024-05-27T07:49:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f3823208740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5569a8017810 con 0x5569a7ce72f0 2024-05-27T07:49:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f381a8f4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 0x5569a80b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f381a8f4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 0x5569a80b1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48228/0 (socket says 172.21.15.114:48228) 2024-05-27T07:49:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f381a8f4640 1 -- 172.21.15.114:0/488628808 learned_addr learned my addr 172.21.15.114:0/488628808 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f381a0f3640 1 --2- 172.21.15.114:0/488628808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 0x5569a80b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.734+0000 7f38198f2640 1 --2- 172.21.15.114:0/488628808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7cef4d0 0x5569a80b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.735+0000 7f381a8f4640 1 -- 172.21.15.114:0/488628808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 msgr2=0x5569a80b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.735+0000 7f381a8f4640 1 --2- 172.21.15.114:0/488628808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 0x5569a80b1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.735+0000 7f381a8f4640 1 -- 172.21.15.114:0/488628808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7cef4d0 msgr2=0x5569a80b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.735+0000 7f381a8f4640 1 --2- 172.21.15.114:0/488628808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7cef4d0 0x5569a80b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.735+0000 7f381a8f4640 1 -- 172.21.15.114:0/488628808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569a7d28210 con 0x5569a7d101b0 2024-05-27T07:49:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.735+0000 7f380b7fe640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f38000b2950 0x7f38000b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.735+0000 7f381a8f4640 1 --2- 172.21.15.114:0/488628808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 0x5569a80b1220 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f380c009cc0 tx=0x7f380c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7767c81f1f6978af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.735+0000 7f381a0f3640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f38000b2950 0x7f38000b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.735+0000 7f3808bfe640 1 -- 172.21.15.114:0/488628808 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c012450 con 0x5569a7d101b0 2024-05-27T07:49:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.735+0000 7f380b7fe640 1 -- 172.21.15.114:0/768613089 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9466+0+0 (secure 0 0 0) 0x7f38100e3310 con 0x7f38141032e0 2024-05-27T07:49:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.735+0000 7f3808bfe640 1 -- 172.21.15.114:0/488628808 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f380c012aa0 con 0x5569a7d101b0 2024-05-27T07:49:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.735+0000 7f3808bfe640 1 -- 172.21.15.114:0/488628808 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c00fb80 con 0x5569a7d101b0 2024-05-27T07:49:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.736+0000 7f3823208740 1 -- 172.21.15.114:0/488628808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 msgr2=0x5569a80b1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.736+0000 7f381a0f3640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f38000b2950 0x7f38000b4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f38100e0760 tx=0x7f38100e6e60 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.736+0000 7f3823208740 1 --2- 172.21.15.114:0/488628808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 0x5569a80b1220 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f380c009cc0 tx=0x7f380c0117b0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.736+0000 7f3823208740 1 -- 172.21.15.114:0/488628808 shutdown_connections 2024-05-27T07:49:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.736+0000 7f3823208740 1 --2- 172.21.15.114:0/488628808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7cef4d0 0x5569a80b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.736+0000 7f3823208740 1 --2- 172.21.15.114:0/488628808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 0x5569a80b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.736+0000 7f3823208740 1 --2- 172.21.15.114:0/488628808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 0x5569a80b1220 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.736+0000 7f3823208740 1 -- 172.21.15.114:0/488628808 >> 172.21.15.114:0/488628808 conn(0x5569a7d04300 msgr2=0x5569a80adb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.736+0000 7f3823208740 1 -- 172.21.15.114:0/488628808 shutdown_connections 2024-05-27T07:49:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.736+0000 7f3823208740 1 -- 172.21.15.114:0/488628808 wait complete. 2024-05-27T07:49:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.736+0000 7f3823208740 1 -- start start 2024-05-27T07:49:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f3823208740 1 Processor -- start 2024-05-27T07:49:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f3823208740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7ce72f0 0x5569a80bdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f3823208740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7cef4d0 0x5569a80be1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f3823208740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 0x5569a80c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f3823208740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5569a7cf9790 con 0x5569a7d101b0 2024-05-27T07:49:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f3823208740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5569a7fcf890 con 0x5569a7ce72f0 2024-05-27T07:49:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f3823208740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5569a80c0770 con 0x5569a7cef4d0 2024-05-27T07:49:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f3823208740 10 client.0 _ll_register_callbacks cb 0x5569a7d27a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f381a8f4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 0x5569a80c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f381a8f4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 0x5569a80c3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48238/0 (socket says 172.21.15.114:48238) 2024-05-27T07:49:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f381a8f4640 1 -- 172.21.15.114:0/792385175 learned_addr learned my addr 172.21.15.114:0/792385175 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f38198f2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7cef4d0 0x5569a80be1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f381a8f4640 1 --2- 172.21.15.114:0/792385175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7ce72f0 0x5569a80bdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f381a8f4640 1 -- 172.21.15.114:0/792385175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7cef4d0 msgr2=0x5569a80be1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f381a8f4640 1 --2- 172.21.15.114:0/792385175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7cef4d0 0x5569a80be1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f381a8f4640 1 -- 172.21.15.114:0/792385175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7ce72f0 msgr2=0x5569a80bdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f381a8f4640 1 --2- 172.21.15.114:0/792385175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7ce72f0 0x5569a80bdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.737+0000 7f381a8f4640 1 -- 172.21.15.114:0/792385175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569a7df4620 con 0x5569a7d101b0 2024-05-27T07:49:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f381a8f4640 1 --2- 172.21.15.114:0/792385175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 0x5569a80c3aa0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f380c009a50 tx=0x7f380c014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e86afd8bf4f86 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f37fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:49:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f37fdffb640 1 -- 172.21.15.114:0/792385175 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c012860 con 0x5569a7d101b0 2024-05-27T07:49:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f37fdffb640 1 -- 172.21.15.114:0/792385175 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9466+0+0 (secure 0 0 0) 0x7f380c00f920 con 0x5569a7d101b0 2024-05-27T07:49:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f3823208740 1 -- 172.21.15.114:0/792385175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f380c012aa0 con 0x5569a7d101b0 2024-05-27T07:49:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f37fdffb640 1 -- 172.21.15.114:0/792385175 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f380c037050 con 0x5569a7d101b0 2024-05-27T07:49:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f3823208740 10 client.17757 fetch_fsmap learned FSMap version 55 2024-05-27T07:49:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f3823208740 1 -- 172.21.15.114:0/792385175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5569a80cc980 con 0x5569a7d101b0 2024-05-27T07:49:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f37fdffb640 1 -- 172.21.15.114:0/792385175 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f380c012aa0 con 0x5569a7d101b0 2024-05-27T07:49:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f37fdffb640 1 -- 172.21.15.114:0/792385175 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 55) ==== 7120+0+0 (secure 0 0 0) 0x7f380c026040 con 0x5569a7d101b0 2024-05-27T07:49:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f3823208740 10 client.17757 fetch_fsmap finished waiting for FSMap version 55 2024-05-27T07:49:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f3823208740 10 client.17757 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f3823208740 20 client.17757 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f3823208740 1 --2- 172.21.15.114:0/792385175 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5569a7cced80 0x5569a80cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f3823208740 4 client.17757 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:49:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.738+0000 7f3823208740 1 -- 172.21.15.114:0/792385175 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569a8025b80 con 0x5569a7cced80 2024-05-27T07:49:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.739+0000 7f38198f2640 1 --2- 172.21.15.114:0/792385175 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5569a7cced80 0x5569a80cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.739+0000 7f37fdffb640 10 client.17757 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.739+0000 7f38198f2640 1 --2- 172.21.15.114:0/792385175 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5569a7cced80 0x5569a80cfde0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e8de540d155def7 server_cookie=604f51576d8eb32e in_seq=0 out_seq=0 2024-05-27T07:49:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f37fdffb640 1 -- 172.21.15.114:0/792385175 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5569a8025b80 con 0x5569a7cced80 2024-05-27T07:49:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f37fdffb640 10 client.17757 handle_command_reply: tid=0 2024-05-27T07:49:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f3823208740 1 client.17757 shutdown 2024-05-27T07:49:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f3823208740 1 -- 172.21.15.114:0/792385175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 msgr2=0x5569a80c3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f3823208740 1 --2- 172.21.15.114:0/792385175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 0x5569a80c3aa0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f380c009a50 tx=0x7f380c014040 comp rx=0 tx=0).stop 2024-05-27T07:49:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f3823208740 1 -- 172.21.15.114:0/792385175 shutdown_connections 2024-05-27T07:49:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f3823208740 1 --2- 172.21.15.114:0/792385175 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5569a7cced80 0x5569a80cfde0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f3823208740 1 --2- 172.21.15.114:0/792385175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7d101b0 0x5569a80c3aa0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f3823208740 1 --2- 172.21.15.114:0/792385175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7cef4d0 0x5569a80be1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f37fdffb640 0 client.17757 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f3823208740 1 --2- 172.21.15.114:0/792385175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7ce72f0 0x5569a80bdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f3823208740 1 -- 172.21.15.114:0/792385175 >> 172.21.15.114:0/792385175 conn(0x5569a7d04300 msgr2=0x5569a80b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f3823208740 1 -- 172.21.15.114:0/792385175 shutdown_connections 2024-05-27T07:49:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f3823208740 1 -- 172.21.15.114:0/792385175 wait complete. 2024-05-27T07:49:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.741+0000 7f3823208740 20 client.17757 trim_cache size 0 max 16384 2024-05-27T07:49:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.757+0000 7f3823208740 1 Processor -- start 2024-05-27T07:49:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.757+0000 7f3823208740 1 -- start start 2024-05-27T07:49:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f3823208740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 0x5569a80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f3823208740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 0x5569a80ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f3823208740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7d101b0 0x5569a80ac860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f3823208740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f38141af2c0 con 0x5569a7cef4d0 2024-05-27T07:49:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f3823208740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f38141af990 con 0x5569a7d101b0 2024-05-27T07:49:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f3823208740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f38141ad110 con 0x5569a7ce72f0 2024-05-27T07:49:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f38198f2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 0x5569a80ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f38198f2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 0x5569a80ca6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48248/0 (socket says 172.21.15.114:48248) 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f38198f2640 1 -- 172.21.15.114:0/4085350490 learned_addr learned my addr 172.21.15.114:0/4085350490 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f381a8f4640 1 --2- 172.21.15.114:0/4085350490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 0x5569a80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f381a8f4640 1 --2- 172.21.15.114:0/4085350490 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7d101b0 0x5569a80ac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f38198f2640 1 -- 172.21.15.114:0/4085350490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 msgr2=0x5569a80b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f38198f2640 1 --2- 172.21.15.114:0/4085350490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 0x5569a80b2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f38198f2640 1 -- 172.21.15.114:0/4085350490 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7d101b0 msgr2=0x5569a80ac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f38198f2640 1 --2- 172.21.15.114:0/4085350490 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7d101b0 0x5569a80ac860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f38198f2640 1 -- 172.21.15.114:0/4085350490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569a80bbfc0 con 0x5569a7cef4d0 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f381a8f4640 1 --2- 172.21.15.114:0/4085350490 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7d101b0 0x5569a80ac860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f38198f2640 1 --2- 172.21.15.114:0/4085350490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 0x5569a80ca6f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f3804005940 tx=0x7f3804031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4edf31c182fbbdef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f37fffff640 1 -- 172.21.15.114:0/4085350490 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380404b070 con 0x5569a7cef4d0 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f37fffff640 1 -- 172.21.15.114:0/4085350490 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3804004400 con 0x5569a7cef4d0 2024-05-27T07:49:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.758+0000 7f37fffff640 1 -- 172.21.15.114:0/4085350490 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3804035040 con 0x5569a7cef4d0 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.759+0000 7f3823208740 1 -- 172.21.15.114:0/4085350490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 msgr2=0x5569a80ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.759+0000 7f3823208740 1 --2- 172.21.15.114:0/4085350490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 0x5569a80ca6f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f3804005940 tx=0x7f3804031400 comp rx=0 tx=0).stop 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.759+0000 7f3823208740 1 -- 172.21.15.114:0/4085350490 shutdown_connections 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.759+0000 7f3823208740 1 --2- 172.21.15.114:0/4085350490 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7d101b0 0x5569a80ac860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.759+0000 7f3823208740 1 --2- 172.21.15.114:0/4085350490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 0x5569a80ca6f0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.759+0000 7f3823208740 1 --2- 172.21.15.114:0/4085350490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 0x5569a80b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.759+0000 7f3823208740 1 -- 172.21.15.114:0/4085350490 >> 172.21.15.114:0/4085350490 conn(0x5569a7d04300 msgr2=0x5569a80ac150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.759+0000 7f3823208740 1 -- 172.21.15.114:0/4085350490 shutdown_connections 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.759+0000 7f3823208740 1 -- 172.21.15.114:0/4085350490 wait complete. 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f3823208740 1 -- start start 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f3823208740 1 Processor -- start 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f3823208740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 0x5569a80c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f3823208740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 0x5569a80ca450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f3823208740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7d101b0 0x5569a80c3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f3823208740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f38141ad780 con 0x5569a7cef4d0 2024-05-27T07:49:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f3823208740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f38141aebf0 con 0x5569a7d101b0 2024-05-27T07:49:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f3823208740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f38141ade50 con 0x5569a7ce72f0 2024-05-27T07:49:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f3823208740 10 client.0 _ll_register_callbacks cb 0x7f38100093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f38198f2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 0x5569a80ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f38198f2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 0x5569a80ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48250/0 (socket says 172.21.15.114:48250) 2024-05-27T07:49:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f38198f2640 1 -- 172.21.15.114:0/2119243417 learned_addr learned my addr 172.21.15.114:0/2119243417 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f381a8f4640 1 --2- 172.21.15.114:0/2119243417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 0x5569a80c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f38198f2640 1 -- 172.21.15.114:0/2119243417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 msgr2=0x5569a80c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f38198f2640 1 --2- 172.21.15.114:0/2119243417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 0x5569a80c9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f38198f2640 1 -- 172.21.15.114:0/2119243417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7d101b0 msgr2=0x5569a80c3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f38198f2640 1 --2- 172.21.15.114:0/2119243417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7d101b0 0x5569a80c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f38198f2640 1 -- 172.21.15.114:0/2119243417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569a80c3880 con 0x5569a7cef4d0 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f38198f2640 1 --2- 172.21.15.114:0/2119243417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 0x5569a80ca450 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f3804004b20 tx=0x7f3804000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c09fd659f8e71871 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.760+0000 7f37fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f37fdffb640 1 -- 172.21.15.114:0/2119243417 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380404b040 con 0x5569a7cef4d0 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f37fdffb640 1 -- 172.21.15.114:0/2119243417 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9466+0+0 (secure 0 0 0) 0x7f38040463d0 con 0x5569a7cef4d0 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f3823208740 1 -- 172.21.15.114:0/2119243417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3804004400 con 0x5569a7cef4d0 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f37fdffb640 1 -- 172.21.15.114:0/2119243417 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f380405c050 con 0x5569a7cef4d0 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f3823208740 10 client.17763 fetch_fsmap learned FSMap version 55 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f3823208740 1 -- 172.21.15.114:0/2119243417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5569a80c82f0 con 0x5569a7cef4d0 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f37fdffb640 1 -- 172.21.15.114:0/2119243417 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3804004400 con 0x5569a7cef4d0 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f37fdffb640 1 -- 172.21.15.114:0/2119243417 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 55) ==== 7120+0+0 (secure 0 0 0) 0x7f3804057030 con 0x5569a7cef4d0 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f3823208740 10 client.17763 fetch_fsmap finished waiting for FSMap version 55 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f3823208740 10 client.17763 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f3823208740 20 client.17763 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:33.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f3823208740 1 --2- 172.21.15.114:0/2119243417 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5569a7ca41c0 0x5569a813a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:33.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f3823208740 4 client.17763 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:49:33.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f3823208740 1 -- 172.21.15.114:0/2119243417 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5569a801f360 con 0x5569a7ca41c0 2024-05-27T07:49:33.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.761+0000 7f381a8f4640 1 --2- 172.21.15.114:0/2119243417 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5569a7ca41c0 0x5569a813a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:33.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.762+0000 7f37fdffb640 10 client.17763 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:33.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:33.762+0000 7f381a8f4640 1 --2- 172.21.15.114:0/2119243417 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5569a7ca41c0 0x5569a813a480 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9caf7084914d01b6 server_cookie=2da4475441fd4c65 in_seq=0 out_seq=0 2024-05-27T07:49:33.811 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:33 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:49:33.811 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:33 smithi114 ceph-mon[32038]: pgmap v322: 129 pgs: 129 active+clean; 96 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 175 op/s 2024-05-27T07:49:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.030+0000 7f37fdffb640 1 -- 172.21.15.114:0/2119243417 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5569a801f360 con 0x5569a7ca41c0 2024-05-27T07:49:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.030+0000 7f37fdffb640 10 client.17763 handle_command_reply: tid=0 2024-05-27T07:49:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.030+0000 7f3823208740 1 client.17763 shutdown 2024-05-27T07:49:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.030+0000 7f3823208740 1 -- 172.21.15.114:0/2119243417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 msgr2=0x5569a80ca450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.030+0000 7f3823208740 1 --2- 172.21.15.114:0/2119243417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 0x5569a80ca450 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f3804004b20 tx=0x7f3804000bc0 comp rx=0 tx=0).stop 2024-05-27T07:49:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.030+0000 7f3823208740 1 -- 172.21.15.114:0/2119243417 shutdown_connections 2024-05-27T07:49:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.030+0000 7f3823208740 1 --2- 172.21.15.114:0/2119243417 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5569a7ca41c0 0x5569a813a480 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.031+0000 7f3823208740 1 --2- 172.21.15.114:0/2119243417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5569a7d101b0 0x5569a80c3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.031+0000 7f3823208740 1 --2- 172.21.15.114:0/2119243417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569a7cef4d0 0x5569a80ca450 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.031+0000 7f37fdffb640 0 client.17763 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.031+0000 7f3823208740 1 --2- 172.21.15.114:0/2119243417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569a7ce72f0 0x5569a80c9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.031+0000 7f3823208740 1 -- 172.21.15.114:0/2119243417 >> 172.21.15.114:0/2119243417 conn(0x5569a7d04300 msgr2=0x5569a80adb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.031+0000 7f3823208740 1 -- 172.21.15.114:0/2119243417 shutdown_connections 2024-05-27T07:49:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.031+0000 7f3823208740 1 -- 172.21.15.114:0/2119243417 wait complete. 2024-05-27T07:49:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.031+0000 7f3823208740 20 client.17763 trim_cache size 0 max 16384 2024-05-27T07:49:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:49:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:49:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:49:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.032+0000 7f381c357640 1 -- 172.21.15.114:0/768613089 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f38000b2950 msgr2=0x7f38000b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.032+0000 7f381c357640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f38000b2950 0x7f38000b4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f38100e0760 tx=0x7f38100e6e60 comp rx=0 tx=0).stop 2024-05-27T07:49:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.032+0000 7f381c357640 1 -- 172.21.15.114:0/768613089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38141032e0 msgr2=0x7f3814199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.032+0000 7f381c357640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38141032e0 0x7f3814199800 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f381000ebc0 tx=0x7f381000c6a0 comp rx=0 tx=0).stop 2024-05-27T07:49:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.032+0000 7f381c357640 1 -- 172.21.15.114:0/768613089 shutdown_connections 2024-05-27T07:49:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.032+0000 7f381c357640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f38000b2950 0x7f38000b4e60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.032+0000 7f381c357640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3814109030 0x7f381419e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.032+0000 7f381c357640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3814108b90 0x7f3814199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.032+0000 7f381c357640 1 --2- 172.21.15.114:0/768613089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38141032e0 0x7f3814199800 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.032+0000 7f381c357640 1 -- 172.21.15.114:0/768613089 >> 172.21.15.114:0/768613089 conn(0x7f38140782d0 msgr2=0x7f38141012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.032+0000 7f381c357640 1 -- 172.21.15.114:0/768613089 shutdown_connections 2024-05-27T07:49:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:34.032+0000 7f381c357640 1 -- 172.21.15.114:0/768613089 wait complete. 2024-05-27T07:49:34.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 979418 10.53 MB/sec execute 346 sec latency 183.422 ms 2024-05-27T07:49:34.702 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f485b10c700 time 2024-05-27T07:49:34.680029+0000 2024-05-27T07:49:34.703 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:49:34.703 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:34.703 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f486d951ba6] 2024-05-27T07:49:34.703 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f486d951dd1] 2024-05-27T07:49:34.703 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7675ca23d] 2024-05-27T07:49:34.703 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c7675b5971] 2024-05-27T07:49:34.703 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c7675b3fa0] 2024-05-27T07:49:34.703 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7675b4651] 2024-05-27T07:49:34.703 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c767549347] 2024-05-27T07:49:34.703 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c76724587e] 2024-05-27T07:49:34.703 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c76724af2f] 2024-05-27T07:49:34.703 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c767255969] 2024-05-27T07:49:34.704 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c7675ee48c] 2024-05-27T07:49:34.704 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c7675f79cb] 2024-05-27T07:49:34.704 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 13: (Thread::entry_wrapper()+0x43) [0x7f486d92072d] 2024-05-27T07:49:34.704 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 14: (Thread::_entry_func(void*)+0xd) [0x7f486d920749] 2024-05-27T07:49:34.704 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f486c4c41ca] 2024-05-27T07:49:34.704 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 16: clone() 2024-05-27T07:49:34.704 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: *** Caught signal (Aborted) ** 2024-05-27T07:49:34.704 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: in thread 7f485b10c700 thread_name:md_log_replay 2024-05-27T07:49:34.704 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.681+0000 7f485b10c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f485b10c700 time 2024-05-27T07:49:34.680029+0000 2024-05-27T07:49:34.705 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:49:34.705 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.705 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:34.705 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f486d951ba6] 2024-05-27T07:49:34.705 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f486d951dd1] 2024-05-27T07:49:34.705 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7675ca23d] 2024-05-27T07:49:34.705 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c7675b5971] 2024-05-27T07:49:34.705 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c7675b3fa0] 2024-05-27T07:49:34.705 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7675b4651] 2024-05-27T07:49:34.705 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c767549347] 2024-05-27T07:49:34.705 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c76724587e] 2024-05-27T07:49:34.705 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c76724af2f] 2024-05-27T07:49:34.706 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c767255969] 2024-05-27T07:49:34.706 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c7675ee48c] 2024-05-27T07:49:34.706 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c7675f79cb] 2024-05-27T07:49:34.706 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 13: (Thread::entry_wrapper()+0x43) [0x7f486d92072d] 2024-05-27T07:49:34.706 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 14: (Thread::_entry_func(void*)+0xd) [0x7f486d920749] 2024-05-27T07:49:34.706 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f486c4c41ca] 2024-05-27T07:49:34.706 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 16: clone() 2024-05-27T07:49:34.706 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.706 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:34.706 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f486c4ced20] 2024-05-27T07:49:34.706 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2: gsignal() 2024-05-27T07:49:34.706 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 3: abort() 2024-05-27T07:49:34.706 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f486d951ced] 2024-05-27T07:49:34.707 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f486d951dd1] 2024-05-27T07:49:34.707 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7675ca23d] 2024-05-27T07:49:34.707 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c7675b5971] 2024-05-27T07:49:34.707 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c7675b3fa0] 2024-05-27T07:49:34.707 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7675b4651] 2024-05-27T07:49:34.707 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c767549347] 2024-05-27T07:49:34.707 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c76724587e] 2024-05-27T07:49:34.707 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c76724af2f] 2024-05-27T07:49:34.707 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c767255969] 2024-05-27T07:49:34.707 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c7675ee48c] 2024-05-27T07:49:34.707 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c7675f79cb] 2024-05-27T07:49:34.707 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 16: (Thread::entry_wrapper()+0x43) [0x7f486d92072d] 2024-05-27T07:49:34.707 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 17: (Thread::_entry_func(void*)+0xd) [0x7f486d920749] 2024-05-27T07:49:34.708 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f486c4c41ca] 2024-05-27T07:49:34.708 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 19: clone() 2024-05-27T07:49:34.708 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.683+0000 7f485b10c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:49:34.708 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: in thread 7f485b10c700 thread_name:md_log_replay 2024-05-27T07:49:34.708 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.708 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:34.708 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f486c4ced20] 2024-05-27T07:49:34.708 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2: gsignal() 2024-05-27T07:49:34.708 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 3: abort() 2024-05-27T07:49:34.708 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f486d951ced] 2024-05-27T07:49:34.708 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f486d951dd1] 2024-05-27T07:49:34.708 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7675ca23d] 2024-05-27T07:49:34.708 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c7675b5971] 2024-05-27T07:49:34.709 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c7675b3fa0] 2024-05-27T07:49:34.709 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7675b4651] 2024-05-27T07:49:34.709 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c767549347] 2024-05-27T07:49:34.709 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c76724587e] 2024-05-27T07:49:34.709 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c76724af2f] 2024-05-27T07:49:34.710 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c767255969] 2024-05-27T07:49:34.710 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c7675ee48c] 2024-05-27T07:49:34.710 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c7675f79cb] 2024-05-27T07:49:34.710 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 16: (Thread::entry_wrapper()+0x43) [0x7f486d92072d] 2024-05-27T07:49:34.710 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 17: (Thread::_entry_func(void*)+0xd) [0x7f486d920749] 2024-05-27T07:49:34.710 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f486c4c41ca] 2024-05-27T07:49:34.710 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 19: clone() 2024-05-27T07:49:34.710 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:49:34.710 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.710 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: -1> 2024-05-27T07:49:34.681+0000 7f485b10c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f485b10c700 time 2024-05-27T07:49:34.680029+0000 2024-05-27T07:49:34.710 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:49:34.710 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f486d951ba6] 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f486d951dd1] 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7675ca23d] 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c7675b5971] 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c7675b3fa0] 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7675b4651] 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c767549347] 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c76724587e] 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c76724af2f] 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c767255969] 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c7675ee48c] 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c7675f79cb] 2024-05-27T07:49:34.711 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 13: (Thread::entry_wrapper()+0x43) [0x7f486d92072d] 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 14: (Thread::_entry_func(void*)+0xd) [0x7f486d920749] 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f486c4c41ca] 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 16: clone() 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 0> 2024-05-27T07:49:34.683+0000 7f485b10c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: in thread 7f485b10c700 thread_name:md_log_replay 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f486c4ced20] 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2: gsignal() 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 3: abort() 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f486d951ced] 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f486d951dd1] 2024-05-27T07:49:34.712 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7675ca23d] 2024-05-27T07:49:34.713 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c7675b5971] 2024-05-27T07:49:34.713 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c7675b3fa0] 2024-05-27T07:49:34.713 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7675b4651] 2024-05-27T07:49:34.713 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c767549347] 2024-05-27T07:49:34.713 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c76724587e] 2024-05-27T07:49:34.713 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c76724af2f] 2024-05-27T07:49:34.713 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c767255969] 2024-05-27T07:49:34.713 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c7675ee48c] 2024-05-27T07:49:34.713 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c7675f79cb] 2024-05-27T07:49:34.713 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 16: (Thread::entry_wrapper()+0x43) [0x7f486d92072d] 2024-05-27T07:49:34.713 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 17: (Thread::_entry_func(void*)+0xd) [0x7f486d920749] 2024-05-27T07:49:34.713 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f486c4c41ca] 2024-05-27T07:49:34.713 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 19: clone() 2024-05-27T07:49:34.714 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:49:34.714 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.976 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: -1> 2024-05-27T07:49:34.681+0000 7f485b10c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f485b10c700 time 2024-05-27T07:49:34.680029+0000 2024-05-27T07:49:34.977 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:49:34.977 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.977 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:34.977 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f486d951ba6] 2024-05-27T07:49:34.977 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f486d951dd1] 2024-05-27T07:49:34.977 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7675ca23d] 2024-05-27T07:49:34.977 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c7675b5971] 2024-05-27T07:49:34.977 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c7675b3fa0] 2024-05-27T07:49:34.977 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7675b4651] 2024-05-27T07:49:34.977 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c767549347] 2024-05-27T07:49:34.978 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c76724587e] 2024-05-27T07:49:34.978 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c76724af2f] 2024-05-27T07:49:34.978 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c767255969] 2024-05-27T07:49:34.978 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c7675ee48c] 2024-05-27T07:49:34.978 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c7675f79cb] 2024-05-27T07:49:34.978 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 13: (Thread::entry_wrapper()+0x43) [0x7f486d92072d] 2024-05-27T07:49:34.978 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 14: (Thread::_entry_func(void*)+0xd) [0x7f486d920749] 2024-05-27T07:49:34.978 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f486c4c41ca] 2024-05-27T07:49:34.978 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 16: clone() 2024-05-27T07:49:34.978 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.978 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 0> 2024-05-27T07:49:34.683+0000 7f485b10c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:49:34.978 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: in thread 7f485b10c700 thread_name:md_log_replay 2024-05-27T07:49:34.978 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f486c4ced20] 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2: gsignal() 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 3: abort() 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f486d951ced] 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f486d951dd1] 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7675ca23d] 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c7675b5971] 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c7675b3fa0] 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7675b4651] 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c767549347] 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c76724587e] 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c76724af2f] 2024-05-27T07:49:34.979 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c767255969] 2024-05-27T07:49:34.980 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c7675ee48c] 2024-05-27T07:49:34.980 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c7675f79cb] 2024-05-27T07:49:34.980 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 16: (Thread::entry_wrapper()+0x43) [0x7f486d92072d] 2024-05-27T07:49:34.980 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 17: (Thread::_entry_func(void*)+0xd) [0x7f486d920749] 2024-05-27T07:49:34.980 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f486c4c41ca] 2024-05-27T07:49:34.980 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 19: clone() 2024-05-27T07:49:34.980 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:49:34.980 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72285]: 2024-05-27T07:49:34.980 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:34 smithi163 podman[72323]: 2024-05-27 07:49:34.739489472 +0000 UTC m=+0.017707388 container died 1f7fdb6f6ea09add5e87026129bc479ed94b2c6b6736b0c2c4dc8c9f43611ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-27T07:49:35.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 980845 10.53 MB/sec execute 347 sec latency 91.764 ms 2024-05-27T07:49:35.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:35 smithi163 ceph-mon[36451]: pgmap v323: 129 pgs: 129 active+clean; 96 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 175 op/s 2024-05-27T07:49:35.726 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:35 smithi163 podman[72323]: 2024-05-27 07:49:35.399079048 +0000 UTC m=+0.677296958 container remove 1f7fdb6f6ea09add5e87026129bc479ed94b2c6b6736b0c2c4dc8c9f43611ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:49:35.726 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:35 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:49:35.726 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:35 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-05-27T07:49:35.726 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:35 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Consumed 1.080s CPU time. 2024-05-27T07:49:35.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:35 smithi186 ceph-mon[36647]: pgmap v323: 129 pgs: 129 active+clean; 96 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 175 op/s 2024-05-27T07:49:35.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:35 smithi114 ceph-mon[32038]: pgmap v323: 129 pgs: 129 active+clean; 96 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 175 op/s 2024-05-27T07:49:36.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 982419 10.52 MB/sec execute 348 sec latency 47.474 ms 2024-05-27T07:49:37.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 984446 10.52 MB/sec execute 349 sec latency 17.376 ms 2024-05-27T07:49:37.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:37 smithi163 ceph-mon[36451]: pgmap v324: 129 pgs: 129 active+clean; 79 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 225 op/s 2024-05-27T07:49:37.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:37 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:49:37.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:37 smithi186 ceph-mon[36647]: pgmap v324: 129 pgs: 129 active+clean; 79 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 225 op/s 2024-05-27T07:49:37.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:37 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:49:37.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:37 smithi114 ceph-mon[32038]: pgmap v324: 129 pgs: 129 active+clean; 79 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 225 op/s 2024-05-27T07:49:37.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:37 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:49:38.280 INFO:tasks.workunit.client.0.smithi114.stdout: 1 986462 10.51 MB/sec execute 350 sec latency 28.942 ms 2024-05-27T07:49:39.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 988246 10.52 MB/sec execute 351 sec latency 15.029 ms 2024-05-27T07:49:39.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:39 smithi163 ceph-mon[36451]: pgmap v325: 129 pgs: 129 active+clean; 79 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 205 op/s 2024-05-27T07:49:39.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:39 smithi186 ceph-mon[36647]: pgmap v325: 129 pgs: 129 active+clean; 79 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 205 op/s 2024-05-27T07:49:39.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:39 smithi114 ceph-mon[32038]: pgmap v325: 129 pgs: 129 active+clean; 79 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 205 op/s 2024-05-27T07:49:40.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 990138 10.51 MB/sec execute 352 sec latency 66.001 ms 2024-05-27T07:49:41.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 991851 10.51 MB/sec execute 353 sec latency 17.240 ms 2024-05-27T07:49:41.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:41 smithi163 ceph-mon[36451]: pgmap v326: 129 pgs: 129 active+clean; 82 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 211 op/s 2024-05-27T07:49:41.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:41 smithi186 ceph-mon[36647]: pgmap v326: 129 pgs: 129 active+clean; 82 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 211 op/s 2024-05-27T07:49:41.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:41 smithi114 ceph-mon[32038]: pgmap v326: 129 pgs: 129 active+clean; 82 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 211 op/s 2024-05-27T07:49:42.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 994110 10.52 MB/sec execute 354 sec latency 38.145 ms 2024-05-27T07:49:43.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 995512 10.51 MB/sec execute 355 sec latency 27.014 ms 2024-05-27T07:49:43.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:43 smithi163 ceph-mon[36451]: pgmap v327: 129 pgs: 129 active+clean; 87 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 250 op/s 2024-05-27T07:49:43.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:43 smithi186 ceph-mon[36647]: pgmap v327: 129 pgs: 129 active+clean; 87 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 250 op/s 2024-05-27T07:49:43.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:43 smithi114 ceph-mon[32038]: pgmap v327: 129 pgs: 129 active+clean; 87 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 250 op/s 2024-05-27T07:49:44.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 997960 10.52 MB/sec execute 356 sec latency 39.593 ms 2024-05-27T07:49:45.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 998883 10.50 MB/sec execute 357 sec latency 47.503 ms 2024-05-27T07:49:45.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:45 smithi163 ceph-mon[36451]: pgmap v328: 129 pgs: 129 active+clean; 87 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 233 op/s 2024-05-27T07:49:45.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:45 smithi186 ceph-mon[36647]: pgmap v328: 129 pgs: 129 active+clean; 87 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 233 op/s 2024-05-27T07:49:45.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:45 smithi114 ceph-mon[32038]: pgmap v328: 129 pgs: 129 active+clean; 87 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 233 op/s 2024-05-27T07:49:45.982 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:45 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:49:45.983 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:45 smithi163 systemd[1]: Stopped Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:49:45.983 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:45 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Consumed 1.080s CPU time. 2024-05-27T07:49:45.983 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:45 smithi163 systemd[1]: Starting Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:49:45.983 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:45 smithi163 podman[72414]: 2024-05-27 07:49:45.855788139 +0000 UTC m=+0.019330357 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:49:45.983 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:45 smithi163 podman[72414]: 2024-05-27 07:49:45.983321438 +0000 UTC m=+0.146863688 container create 7cf46663b4ceee35193be40aa8f0938e2e36091bc244883bc50bb82bafa40cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T07:49:46.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1001041 10.50 MB/sec execute 358 sec latency 22.458 ms 2024-05-27T07:49:46.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:46 smithi163 ceph-mon[36451]: daemon mds.h restarted 2024-05-27T07:49:46.511 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 podman[72414]: 2024-05-27 07:49:46.244739418 +0000 UTC m=+0.408281609 container init 7cf46663b4ceee35193be40aa8f0938e2e36091bc244883bc50bb82bafa40cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD) 2024-05-27T07:49:46.511 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 podman[72414]: 2024-05-27 07:49:46.247382693 +0000 UTC m=+0.410924901 container start 7cf46663b4ceee35193be40aa8f0938e2e36091bc244883bc50bb82bafa40cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-27T07:49:46.511 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-mds[72434]: -- 172.21.15.163:0/130642509 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562fd5b4ba40 con 0x562fd5b93180 2024-05-27T07:49:46.511 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: starting mds.h at 2024-05-27T07:49:46.511 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 bash[72414]: 7cf46663b4ceee35193be40aa8f0938e2e36091bc244883bc50bb82bafa40cb2 2024-05-27T07:49:46.511 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 systemd[1]: Started Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:49:46.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb39bd87700 time 2024-05-27T07:49:46.463796+0000 2024-05-27T07:49:46.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:49:46.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:46.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb3ae5ccba6] 2024-05-27T07:49:46.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3ae5ccdd1] 2024-05-27T07:49:46.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562fd3c7923d] 2024-05-27T07:49:46.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562fd3c64971] 2024-05-27T07:49:46.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 5: (SnapRealm::check_cache() const+0x13e) [0x562fd3c62fa0] 2024-05-27T07:49:46.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562fd3c63651] 2024-05-27T07:49:46.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562fd3bf8347] 2024-05-27T07:49:46.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562fd38f487e] 2024-05-27T07:49:46.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562fd38f9f2f] 2024-05-27T07:49:46.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562fd3904969] 2024-05-27T07:49:46.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 11: (MDLog::_replay_thread()+0x1f38) [0x562fd3c9d48c] 2024-05-27T07:49:46.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562fd3ca69cb] 2024-05-27T07:49:46.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 13: (Thread::entry_wrapper()+0x43) [0x7fb3ae59b72d] 2024-05-27T07:49:46.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb3ae59b749] 2024-05-27T07:49:46.514 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb3ad13f1ca] 2024-05-27T07:49:46.514 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 16: clone() 2024-05-27T07:49:46.514 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.465+0000 7fb39bd87700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb39bd87700 time 2024-05-27T07:49:46.463796+0000 2024-05-27T07:49:46.514 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:49:46.514 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.514 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:46.514 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb3ae5ccba6] 2024-05-27T07:49:46.514 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3ae5ccdd1] 2024-05-27T07:49:46.514 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562fd3c7923d] 2024-05-27T07:49:46.514 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562fd3c64971] 2024-05-27T07:49:46.514 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 5: (SnapRealm::check_cache() const+0x13e) [0x562fd3c62fa0] 2024-05-27T07:49:46.514 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562fd3c63651] 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562fd3bf8347] 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562fd38f487e] 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562fd38f9f2f] 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562fd3904969] 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 11: (MDLog::_replay_thread()+0x1f38) [0x562fd3c9d48c] 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562fd3ca69cb] 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 13: (Thread::entry_wrapper()+0x43) [0x7fb3ae59b72d] 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb3ae59b749] 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb3ad13f1ca] 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 16: clone() 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: *** Caught signal (Aborted) ** 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: in thread 7fb39bd87700 thread_name:md_log_replay 2024-05-27T07:49:46.515 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb3ad149d20] 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2: gsignal() 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 3: abort() 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb3ae5ccced] 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3ae5ccdd1] 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562fd3c7923d] 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562fd3c64971] 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 8: (SnapRealm::check_cache() const+0x13e) [0x562fd3c62fa0] 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562fd3c63651] 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562fd3bf8347] 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562fd38f487e] 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562fd38f9f2f] 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562fd3904969] 2024-05-27T07:49:46.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 14: (MDLog::_replay_thread()+0x1f38) [0x562fd3c9d48c] 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562fd3ca69cb] 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 16: (Thread::entry_wrapper()+0x43) [0x7fb3ae59b72d] 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb3ae59b749] 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb3ad13f1ca] 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 19: clone() 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.467+0000 7fb39bd87700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: in thread 7fb39bd87700 thread_name:md_log_replay 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb3ad149d20] 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2: gsignal() 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 3: abort() 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb3ae5ccced] 2024-05-27T07:49:46.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3ae5ccdd1] 2024-05-27T07:49:46.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562fd3c7923d] 2024-05-27T07:49:46.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562fd3c64971] 2024-05-27T07:49:46.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 8: (SnapRealm::check_cache() const+0x13e) [0x562fd3c62fa0] 2024-05-27T07:49:46.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562fd3c63651] 2024-05-27T07:49:46.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562fd3bf8347] 2024-05-27T07:49:46.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562fd38f487e] 2024-05-27T07:49:46.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562fd38f9f2f] 2024-05-27T07:49:46.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562fd3904969] 2024-05-27T07:49:46.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 14: (MDLog::_replay_thread()+0x1f38) [0x562fd3c9d48c] 2024-05-27T07:49:46.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562fd3ca69cb] 2024-05-27T07:49:46.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 16: (Thread::entry_wrapper()+0x43) [0x7fb3ae59b72d] 2024-05-27T07:49:46.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb3ae59b749] 2024-05-27T07:49:46.519 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb3ad13f1ca] 2024-05-27T07:49:46.519 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 19: clone() 2024-05-27T07:49:46.519 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:49:46.519 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: -1> 2024-05-27T07:49:46.465+0000 7fb39bd87700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb39bd87700 time 2024-05-27T07:49:46.463796+0000 2024-05-27T07:49:46.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:49:46.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:46.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb3ae5ccba6] 2024-05-27T07:49:46.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3ae5ccdd1] 2024-05-27T07:49:46.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562fd3c7923d] 2024-05-27T07:49:46.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562fd3c64971] 2024-05-27T07:49:46.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 5: (SnapRealm::check_cache() const+0x13e) [0x562fd3c62fa0] 2024-05-27T07:49:46.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562fd3c63651] 2024-05-27T07:49:46.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562fd3bf8347] 2024-05-27T07:49:46.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562fd38f487e] 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562fd38f9f2f] 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562fd3904969] 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 11: (MDLog::_replay_thread()+0x1f38) [0x562fd3c9d48c] 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562fd3ca69cb] 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 13: (Thread::entry_wrapper()+0x43) [0x7fb3ae59b72d] 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb3ae59b749] 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb3ad13f1ca] 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 16: clone() 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 0> 2024-05-27T07:49:46.467+0000 7fb39bd87700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: in thread 7fb39bd87700 thread_name:md_log_replay 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:46.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb3ad149d20] 2024-05-27T07:49:46.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2: gsignal() 2024-05-27T07:49:46.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 3: abort() 2024-05-27T07:49:46.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb3ae5ccced] 2024-05-27T07:49:46.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3ae5ccdd1] 2024-05-27T07:49:46.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562fd3c7923d] 2024-05-27T07:49:46.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562fd3c64971] 2024-05-27T07:49:46.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 8: (SnapRealm::check_cache() const+0x13e) [0x562fd3c62fa0] 2024-05-27T07:49:46.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562fd3c63651] 2024-05-27T07:49:46.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562fd3bf8347] 2024-05-27T07:49:46.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562fd38f487e] 2024-05-27T07:49:46.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562fd38f9f2f] 2024-05-27T07:49:46.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562fd3904969] 2024-05-27T07:49:46.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 14: (MDLog::_replay_thread()+0x1f38) [0x562fd3c9d48c] 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562fd3ca69cb] 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 16: (Thread::entry_wrapper()+0x43) [0x7fb3ae59b72d] 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb3ae59b749] 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb3ad13f1ca] 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 19: clone() 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: -1> 2024-05-27T07:49:46.465+0000 7fb39bd87700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb39bd87700 time 2024-05-27T07:49:46.463796+0000 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb3ae5ccba6] 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3ae5ccdd1] 2024-05-27T07:49:46.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562fd3c7923d] 2024-05-27T07:49:46.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562fd3c64971] 2024-05-27T07:49:46.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 5: (SnapRealm::check_cache() const+0x13e) [0x562fd3c62fa0] 2024-05-27T07:49:46.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562fd3c63651] 2024-05-27T07:49:46.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562fd3bf8347] 2024-05-27T07:49:46.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562fd38f487e] 2024-05-27T07:49:46.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562fd38f9f2f] 2024-05-27T07:49:46.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562fd3904969] 2024-05-27T07:49:46.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 11: (MDLog::_replay_thread()+0x1f38) [0x562fd3c9d48c] 2024-05-27T07:49:46.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562fd3ca69cb] 2024-05-27T07:49:46.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 13: (Thread::entry_wrapper()+0x43) [0x7fb3ae59b72d] 2024-05-27T07:49:46.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb3ae59b749] 2024-05-27T07:49:46.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb3ad13f1ca] 2024-05-27T07:49:46.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 16: clone() 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 0> 2024-05-27T07:49:46.467+0000 7fb39bd87700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: in thread 7fb39bd87700 thread_name:md_log_replay 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb3ad149d20] 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2: gsignal() 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 3: abort() 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb3ae5ccced] 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3ae5ccdd1] 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562fd3c7923d] 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562fd3c64971] 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 8: (SnapRealm::check_cache() const+0x13e) [0x562fd3c62fa0] 2024-05-27T07:49:46.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562fd3c63651] 2024-05-27T07:49:46.526 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562fd3bf8347] 2024-05-27T07:49:46.526 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562fd38f487e] 2024-05-27T07:49:46.526 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562fd38f9f2f] 2024-05-27T07:49:46.526 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562fd3904969] 2024-05-27T07:49:46.526 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 14: (MDLog::_replay_thread()+0x1f38) [0x562fd3c9d48c] 2024-05-27T07:49:46.526 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562fd3ca69cb] 2024-05-27T07:49:46.526 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 16: (Thread::entry_wrapper()+0x43) [0x7fb3ae59b72d] 2024-05-27T07:49:46.526 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb3ae59b749] 2024-05-27T07:49:46.526 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb3ad13f1ca] 2024-05-27T07:49:46.526 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 19: clone() 2024-05-27T07:49:46.526 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:49:46.526 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h[72430]: 2024-05-27T07:49:46.526 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:46 smithi163 podman[72465]: 2024-05-27 07:49:46.511712357 +0000 UTC m=+0.018188130 container died 7cf46663b4ceee35193be40aa8f0938e2e36091bc244883bc50bb82bafa40cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, org.label-schema.build-date=20240521, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:49:46.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:46 smithi186 ceph-mon[36647]: daemon mds.h restarted 2024-05-27T07:49:46.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:46 smithi114 ceph-mon[32038]: daemon mds.h restarted 2024-05-27T07:49:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.135+0000 7f7b34a29640 1 -- 172.21.15.114:0/935595905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b30109040 msgr2=0x7f7b30102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.135+0000 7f7b34a29640 1 --2- 172.21.15.114:0/935595905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b30109040 0x7f7b30102c80 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009900 tx=0x7f7b2402a410 comp rx=0 tx=0).stop 2024-05-27T07:49:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.136+0000 7f7b34a29640 1 -- 172.21.15.114:0/935595905 shutdown_connections 2024-05-27T07:49:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.136+0000 7f7b34a29640 1 --2- 172.21.15.114:0/935595905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b301032f0 0x7f7b3010f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.136+0000 7f7b34a29640 1 --2- 172.21.15.114:0/935595905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b30109040 0x7f7b30102c80 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.136+0000 7f7b34a29640 1 --2- 172.21.15.114:0/935595905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b30108ba0 0x7f7b30102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.136+0000 7f7b34a29640 1 -- 172.21.15.114:0/935595905 >> 172.21.15.114:0/935595905 conn(0x7f7b30078270 msgr2=0x7f7b300786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.136+0000 7f7b34a29640 1 -- 172.21.15.114:0/935595905 shutdown_connections 2024-05-27T07:49:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.136+0000 7f7b34a29640 1 -- 172.21.15.114:0/935595905 wait complete. 2024-05-27T07:49:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.136+0000 7f7b34a29640 1 Processor -- start 2024-05-27T07:49:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b34a29640 1 -- start start 2024-05-27T07:49:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b34a29640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b301032f0 0x7f7b30199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b34a29640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b30108ba0 0x7f7b30199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b34a29640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b30109040 0x7f7b3019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b34a29640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7b30112d90 con 0x7f7b30109040 2024-05-27T07:49:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b34a29640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7b30112c10 con 0x7f7b301032f0 2024-05-27T07:49:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b34a29640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7b30112f10 con 0x7f7b30108ba0 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b2ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b30109040 0x7f7b3019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b2e59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b301032f0 0x7f7b30199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b2ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b30109040 0x7f7b3019dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43860/0 (socket says 172.21.15.114:43860) 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b2ed9d640 1 -- 172.21.15.114:0/1424649652 learned_addr learned my addr 172.21.15.114:0/1424649652 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b2dd9b640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b30108ba0 0x7f7b30199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b2ed9d640 1 -- 172.21.15.114:0/1424649652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b30108ba0 msgr2=0x7f7b30199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b2ed9d640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b30108ba0 0x7f7b30199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b2ed9d640 1 -- 172.21.15.114:0/1424649652 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b301032f0 msgr2=0x7f7b30199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b2ed9d640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b301032f0 0x7f7b30199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b2ed9d640 1 -- 172.21.15.114:0/1424649652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3019e5f0 con 0x7f7b30109040 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b301032f0 0x7f7b30199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.137+0000 7f7b2ed9d640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b30109040 0x7f7b3019dfb0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800eb10 tx=0x7f7b2800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.138+0000 7f7b1f7fe640 1 -- 172.21.15.114:0/1424649652 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b28004b20 con 0x7f7b30109040 2024-05-27T07:49:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.138+0000 7f7b34a29640 1 -- 172.21.15.114:0/1424649652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3019e8e0 con 0x7f7b30109040 2024-05-27T07:49:47.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.138+0000 7f7b1f7fe640 1 -- 172.21.15.114:0/1424649652 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b28004cc0 con 0x7f7b30109040 2024-05-27T07:49:47.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.138+0000 7f7b1f7fe640 1 -- 172.21.15.114:0/1424649652 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b28010670 con 0x7f7b30109040 2024-05-27T07:49:47.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.138+0000 7f7b34a29640 1 -- 172.21.15.114:0/1424649652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b3006b350 con 0x7f7b30109040 2024-05-27T07:49:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.140+0000 7f7b3b9b5740 1 Processor -- start 2024-05-27T07:49:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.140+0000 7f7b1f7fe640 1 -- 172.21.15.114:0/1424649652 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f7b28005020 con 0x7f7b30109040 2024-05-27T07:49:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.141+0000 7f7b3b9b5740 1 -- start start 2024-05-27T07:49:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.141+0000 7f7b3b9b5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c6b41b0 0x55863ca553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.141+0000 7f7b3b9b5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c68b2f0 0x55863ca55910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.141+0000 7f7b3b9b5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c6934d0 0x55863ca55e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.141+0000 7f7b3b9b5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55863c9c7c30 con 0x55863c6b41b0 2024-05-27T07:49:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.141+0000 7f7b3b9b5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55863c9b77d0 con 0x55863c68b2f0 2024-05-27T07:49:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.141+0000 7f7b3b9b5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55863c9bb810 con 0x55863c6934d0 2024-05-27T07:49:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.141+0000 7f7b2dd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c6b41b0 0x55863ca553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.141+0000 7f7b2dd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c6b41b0 0x55863ca553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43874/0 (socket says 172.21.15.114:43874) 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.141+0000 7f7b2dd9b640 1 -- 172.21.15.114:0/800189911 learned_addr learned my addr 172.21.15.114:0/800189911 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.141+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/800189911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c68b2f0 0x55863ca55910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.141+0000 7f7b2dd9b640 1 --2- 172.21.15.114:0/800189911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c6934d0 0x55863ca55e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.142+0000 7f7b2e59c640 1 -- 172.21.15.114:0/800189911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c6934d0 msgr2=0x55863ca55e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.142+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/800189911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c6934d0 0x55863ca55e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.142+0000 7f7b2e59c640 1 -- 172.21.15.114:0/800189911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c6b41b0 msgr2=0x55863ca553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.142+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/800189911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c6b41b0 0x55863ca553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.142+0000 7f7b2e59c640 1 -- 172.21.15.114:0/800189911 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55863c6cc210 con 0x55863c68b2f0 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.142+0000 7f7b1f7fe640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7b180b2950 0x7f7b180b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.142+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/800189911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c68b2f0 0x55863ca55910 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009f80 tx=0x7f7b200118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c4ca2dc257eaa64 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.142+0000 7f7b2dd9b640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7b180b2950 0x7f7b180b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.142+0000 7f7b1f7fe640 1 -- 172.21.15.114:0/1424649652 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9466+0+0 (secure 0 0 0) 0x7f7b280e2fa0 con 0x7f7b30109040 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.142+0000 7f7b1cff9640 1 -- 172.21.15.114:0/800189911 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b20004310 con 0x55863c68b2f0 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.142+0000 7f7b1cff9640 1 -- 172.21.15.114:0/800189911 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b200044b0 con 0x55863c68b2f0 2024-05-27T07:49:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.142+0000 7f7b1cff9640 1 -- 172.21.15.114:0/800189911 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b200035f0 con 0x55863c68b2f0 2024-05-27T07:49:47.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.143+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/800189911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c68b2f0 msgr2=0x55863ca55910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.143+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/800189911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c68b2f0 0x55863ca55910 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009f80 tx=0x7f7b200118d0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.143+0000 7f7b2dd9b640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7b180b2950 0x7f7b180b4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x55863ca52540 tx=0x7f7b24031400 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.143+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/800189911 shutdown_connections 2024-05-27T07:49:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.143+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/800189911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c6934d0 0x55863ca55e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.143+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/800189911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c68b2f0 0x55863ca55910 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.143+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/800189911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c6b41b0 0x55863ca553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.143+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/800189911 >> 172.21.15.114:0/800189911 conn(0x55863c6a8300 msgr2=0x55863ca51b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.143+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/800189911 shutdown_connections 2024-05-27T07:49:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.143+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/800189911 wait complete. 2024-05-27T07:49:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b3b9b5740 1 -- start start 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b3b9b5740 1 Processor -- start 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b3b9b5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c68b2f0 0x55863ca5fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b3b9b5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 0x55863ca5ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b3b9b5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c6b41b0 0x55863ca65870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b3b9b5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55863c69d790 con 0x55863c6b41b0 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b3b9b5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55863c973890 con 0x55863c6934d0 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b3b9b5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55863ca62500 con 0x55863c68b2f0 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b3b9b5740 10 client.0 _ll_register_callbacks cb 0x55863c6cba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b2ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c6b41b0 0x55863ca65870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b2e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c68b2f0 0x55863ca5fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b2e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c68b2f0 0x55863ca5fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:32844/0 (socket says 172.21.15.114:32844) 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b2e59c640 1 -- 172.21.15.114:0/395760054 learned_addr learned my addr 172.21.15.114:0/395760054 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b2dd9b640 1 --2- 172.21.15.114:0/395760054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 0x55863ca5ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b2ed9d640 1 -- 172.21.15.114:0/395760054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c68b2f0 msgr2=0x55863ca5fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b2ed9d640 1 --2- 172.21.15.114:0/395760054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c68b2f0 0x55863ca5fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b2ed9d640 1 -- 172.21.15.114:0/395760054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 msgr2=0x55863ca5ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b2ed9d640 1 --2- 172.21.15.114:0/395760054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 0x55863ca5ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b2ed9d640 1 -- 172.21.15.114:0/395760054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55863c798620 con 0x55863c6b41b0 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.144+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/395760054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c68b2f0 0x55863ca5fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b2ed9d640 1 --2- 172.21.15.114:0/395760054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c6b41b0 0x55863ca65870 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f7b280e0340 tx=0x7f7b280a9820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed575a0e4e8347a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b15ffb640 1 -- 172.21.15.114:0/395760054 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b280a9ed0 con 0x55863c6b41b0 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b15ffb640 1 -- 172.21.15.114:0/395760054 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9466+0+0 (secure 0 0 0) 0x7f7b280aa340 con 0x55863c6b41b0 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/395760054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b200044b0 con 0x55863c6b41b0 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b15ffb640 1 -- 172.21.15.114:0/395760054 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f7b280ae280 con 0x55863c6b41b0 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b3b9b5740 10 client.17781 fetch_fsmap learned FSMap version 57 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/395760054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55863ca6e880 con 0x55863c6b41b0 2024-05-27T07:49:47.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b15ffb640 1 -- 172.21.15.114:0/395760054 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b200044b0 con 0x55863c6b41b0 2024-05-27T07:49:47.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b15ffb640 1 -- 172.21.15.114:0/395760054 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 57) ==== 7120+0+0 (secure 0 0 0) 0x7f7b280a5960 con 0x55863c6b41b0 2024-05-27T07:49:47.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b3b9b5740 10 client.17781 fetch_fsmap finished waiting for FSMap version 57 2024-05-27T07:49:47.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b3b9b5740 10 client.17781 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:47.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b3b9b5740 20 client.17781 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:47.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/395760054 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55863c672d80 0x55863ca71d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b3b9b5740 4 client.17781 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:49:47.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.145+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/395760054 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55863c9c9b80 con 0x55863c672d80 2024-05-27T07:49:47.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.146+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/395760054 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55863c672d80 0x55863ca71d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.146+0000 7f7b15ffb640 10 client.17781 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:47.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.146+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/395760054 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55863c672d80 0x55863ca71d50 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36f4da532e80924f server_cookie=d06d4706aaf241a9 in_seq=0 out_seq=0 2024-05-27T07:49:47.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b15ffb640 1 -- 172.21.15.114:0/395760054 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55863c9c9b80 con 0x55863c672d80 2024-05-27T07:49:47.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b15ffb640 10 client.17781 handle_command_reply: tid=0 2024-05-27T07:49:47.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b3b9b5740 1 client.17781 shutdown 2024-05-27T07:49:47.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/395760054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c6b41b0 msgr2=0x55863ca65870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:47.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/395760054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c6b41b0 0x55863ca65870 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f7b280e0340 tx=0x7f7b280a9820 comp rx=0 tx=0).stop 2024-05-27T07:49:47.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/395760054 shutdown_connections 2024-05-27T07:49:47.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/395760054 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55863c672d80 0x55863ca71d50 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/395760054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c6b41b0 0x55863ca65870 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b15ffb640 0 client.17781 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:47.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/395760054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 0x55863ca5ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/395760054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c68b2f0 0x55863ca5fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/395760054 >> 172.21.15.114:0/395760054 conn(0x55863c6a8300 msgr2=0x55863ca55370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:47.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/395760054 shutdown_connections 2024-05-27T07:49:47.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/395760054 wait complete. 2024-05-27T07:49:47.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.147+0000 7f7b3b9b5740 20 client.17781 trim_cache size 0 max 16384 2024-05-27T07:49:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.162+0000 7f7b3b9b5740 1 Processor -- start 2024-05-27T07:49:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.162+0000 7f7b3b9b5740 1 -- start start 2024-05-27T07:49:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.162+0000 7f7b3b9b5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c672d80 0x55863ca59730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.162+0000 7f7b3b9b5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c68b2f0 0x55863ca6c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.162+0000 7f7b3b9b5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 0x55863ca65770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.162+0000 7f7b3b9b5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7b301af1a0 con 0x55863c672d80 2024-05-27T07:49:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.162+0000 7f7b3b9b5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7b301af870 con 0x55863c6934d0 2024-05-27T07:49:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.162+0000 7f7b3b9b5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7b301acff0 con 0x55863c68b2f0 2024-05-27T07:49:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.162+0000 7f7b2e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c672d80 0x55863ca59730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.162+0000 7f7b2e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c672d80 0x55863ca59730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43884/0 (socket says 172.21.15.114:43884) 2024-05-27T07:49:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.162+0000 7f7b2e59c640 1 -- 172.21.15.114:0/1151307298 learned_addr learned my addr 172.21.15.114:0/1151307298 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.162+0000 7f7b2dd9b640 1 --2- 172.21.15.114:0/1151307298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c68b2f0 0x55863ca6c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.162+0000 7f7b2ed9d640 1 --2- 172.21.15.114:0/1151307298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 0x55863ca65770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.163+0000 7f7b2e59c640 1 -- 172.21.15.114:0/1151307298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c68b2f0 msgr2=0x55863ca6c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:47.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.163+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/1151307298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c68b2f0 0x55863ca6c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.163+0000 7f7b2e59c640 1 -- 172.21.15.114:0/1151307298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 msgr2=0x55863ca65770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:47.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.163+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/1151307298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 0x55863ca65770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.163+0000 7f7b2e59c640 1 -- 172.21.15.114:0/1151307298 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55863ca60780 con 0x55863c672d80 2024-05-27T07:49:47.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.163+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/1151307298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c672d80 0x55863ca59730 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009710 tx=0x7f7b200118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193a082efcf6c61d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:47.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.163+0000 7f7b17fff640 1 -- 172.21.15.114:0/1151307298 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b20031070 con 0x55863c672d80 2024-05-27T07:49:47.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.163+0000 7f7b17fff640 1 -- 172.21.15.114:0/1151307298 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b2001f040 con 0x55863c672d80 2024-05-27T07:49:47.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.163+0000 7f7b17fff640 1 -- 172.21.15.114:0/1151307298 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2001e040 con 0x55863c672d80 2024-05-27T07:49:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.164+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/1151307298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c672d80 msgr2=0x55863ca59730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.164+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/1151307298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c672d80 0x55863ca59730 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009710 tx=0x7f7b200118d0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.164+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/1151307298 shutdown_connections 2024-05-27T07:49:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.164+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/1151307298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 0x55863ca65770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.164+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/1151307298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c68b2f0 0x55863ca6c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.164+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/1151307298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c672d80 0x55863ca59730 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.164+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/1151307298 >> 172.21.15.114:0/1151307298 conn(0x55863c6a8300 msgr2=0x55863ca59020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.164+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/1151307298 shutdown_connections 2024-05-27T07:49:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.164+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/1151307298 wait complete. 2024-05-27T07:49:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.164+0000 7f7b3b9b5740 1 -- start start 2024-05-27T07:49:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.164+0000 7f7b3b9b5740 1 Processor -- start 2024-05-27T07:49:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b3b9b5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c672d80 0x55863ca6bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b3b9b5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c68b2f0 0x55863ca6c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b3b9b5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 0x55863ca672c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b3b9b5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7b301ad660 con 0x55863c68b2f0 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b3b9b5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7b301aead0 con 0x55863c6934d0 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b3b9b5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7b301add30 con 0x55863c672d80 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b3b9b5740 10 client.0 _ll_register_callbacks cb 0x7f7b280a9ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b2e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c672d80 0x55863ca6bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b2dd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c68b2f0 0x55863ca6c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b2e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c672d80 0x55863ca6bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:32864/0 (socket says 172.21.15.114:32864) 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b2e59c640 1 -- 172.21.15.114:0/2876710018 learned_addr learned my addr 172.21.15.114:0/2876710018 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b2ed9d640 1 --2- 172.21.15.114:0/2876710018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 0x55863ca672c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b2dd9b640 1 -- 172.21.15.114:0/2876710018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c672d80 msgr2=0x55863ca6bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b2dd9b640 1 --2- 172.21.15.114:0/2876710018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c672d80 0x55863ca6bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b2dd9b640 1 -- 172.21.15.114:0/2876710018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 msgr2=0x55863ca672c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:49:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b2dd9b640 1 --2- 172.21.15.114:0/2876710018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 0x55863ca672c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b2dd9b640 1 -- 172.21.15.114:0/2876710018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55863ca67a50 con 0x55863c68b2f0 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/2876710018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c672d80 0x55863ca6bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b2dd9b640 1 --2- 172.21.15.114:0/2876710018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c68b2f0 0x55863ca6c450 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f7b24006010 tx=0x7f7b2403a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f85f289e7efa68 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.165+0000 7f7b15ffb640 1 -- 172.21.15.114:0/2876710018 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b24059060 con 0x55863c68b2f0 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b15ffb640 1 -- 172.21.15.114:0/2876710018 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9466+0+0 (secure 0 0 0) 0x7f7b24004750 con 0x55863c68b2f0 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/2876710018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b2001f040 con 0x55863c68b2f0 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b15ffb640 1 -- 172.21.15.114:0/2876710018 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f7b24060050 con 0x55863c68b2f0 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b3b9b5740 10 client.17787 fetch_fsmap learned FSMap version 57 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/2876710018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55863cad9e90 con 0x55863c68b2f0 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b15ffb640 1 -- 172.21.15.114:0/2876710018 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b2001f040 con 0x55863c68b2f0 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b15ffb640 1 -- 172.21.15.114:0/2876710018 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 57) ==== 7120+0+0 (secure 0 0 0) 0x7f7b24053070 con 0x55863c68b2f0 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b3b9b5740 10 client.17787 fetch_fsmap finished waiting for FSMap version 57 2024-05-27T07:49:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b3b9b5740 10 client.17787 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:49:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b3b9b5740 20 client.17787 populate_metadata read hostname 'smithi114' 2024-05-27T07:49:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/2876710018 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55863c6b41b0 0x55863cadc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:49:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b3b9b5740 4 client.17787 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:49:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/2876710018 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55863c9c3360 con 0x55863c6b41b0 2024-05-27T07:49:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.166+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/2876710018 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55863c6b41b0 0x55863cadc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:49:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.167+0000 7f7b15ffb640 10 client.17787 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.167+0000 7f7b2e59c640 1 --2- 172.21.15.114:0/2876710018 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55863c6b41b0 0x55863cadc5c0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69177b15e4c78a67 server_cookie=cf127db251ae11fe in_seq=0 out_seq=0 2024-05-27T07:49:47.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1002597 10.50 MB/sec execute 359 sec latency 69.193 ms 2024-05-27T07:49:47.414 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:47 smithi163 podman[72465]: 2024-05-27 07:49:47.142444085 +0000 UTC m=+0.648919857 container remove 7cf46663b4ceee35193be40aa8f0938e2e36091bc244883bc50bb82bafa40cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-h, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-27T07:49:47.414 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:47 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:49:47.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b15ffb640 1 -- 172.21.15.114:0/2876710018 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55863c9c3360 con 0x55863c6b41b0 2024-05-27T07:49:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b15ffb640 10 client.17787 handle_command_reply: tid=0 2024-05-27T07:49:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b3b9b5740 1 client.17787 shutdown 2024-05-27T07:49:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/2876710018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c68b2f0 msgr2=0x55863ca6c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/2876710018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c68b2f0 0x55863ca6c450 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f7b24006010 tx=0x7f7b2403a040 comp rx=0 tx=0).stop 2024-05-27T07:49:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/2876710018 shutdown_connections 2024-05-27T07:49:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/2876710018 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55863c6b41b0 0x55863cadc5c0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/2876710018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55863c6934d0 0x55863ca672c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b15ffb640 0 client.17787 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:49:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/2876710018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55863c68b2f0 0x55863ca6c450 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b3b9b5740 1 --2- 172.21.15.114:0/2876710018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55863c672d80 0x55863ca6bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/2876710018 >> 172.21.15.114:0/2876710018 conn(0x55863c6a8300 msgr2=0x55863ca55160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/2876710018 shutdown_connections 2024-05-27T07:49:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b3b9b5740 1 -- 172.21.15.114:0/2876710018 wait complete. 2024-05-27T07:49:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.554+0000 7f7b3b9b5740 20 client.17787 trim_cache size 0 max 16384 2024-05-27T07:49:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:49:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:49:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:49:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.555+0000 7f7b34a29640 1 -- 172.21.15.114:0/1424649652 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7b180b2950 msgr2=0x7f7b180b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.555+0000 7f7b34a29640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7b180b2950 0x7f7b180b4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x55863ca52540 tx=0x7f7b24031400 comp rx=0 tx=0).stop 2024-05-27T07:49:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.555+0000 7f7b34a29640 1 -- 172.21.15.114:0/1424649652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b30109040 msgr2=0x7f7b3019dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:49:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.555+0000 7f7b34a29640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b30109040 0x7f7b3019dfb0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800eb10 tx=0x7f7b2800c2d0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.555+0000 7f7b34a29640 1 -- 172.21.15.114:0/1424649652 shutdown_connections 2024-05-27T07:49:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.555+0000 7f7b34a29640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7b180b2950 0x7f7b180b4e60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.555+0000 7f7b34a29640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b30109040 0x7f7b3019dfb0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.555+0000 7f7b34a29640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b30108ba0 0x7f7b30199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.555+0000 7f7b34a29640 1 --2- 172.21.15.114:0/1424649652 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b301032f0 0x7f7b30199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:49:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.555+0000 7f7b34a29640 1 -- 172.21.15.114:0/1424649652 >> 172.21.15.114:0/1424649652 conn(0x7f7b30078270 msgr2=0x7f7b30101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:49:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.555+0000 7f7b34a29640 1 -- 172.21.15.114:0/1424649652 shutdown_connections 2024-05-27T07:49:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:49:47.555+0000 7f7b34a29640 1 -- 172.21.15.114:0/1424649652 wait complete. 2024-05-27T07:49:47.726 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:47 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-05-27T07:49:47.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:47 smithi163 ceph-mon[36451]: pgmap v329: 129 pgs: 129 active+clean; 92 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 315 op/s 2024-05-27T07:49:47.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:47 smithi163 ceph-mon[36451]: osdmap e116: 12 total, 12 up, 12 in 2024-05-27T07:49:47.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:47 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6836/3375158379,v1:172.21.15.163:6837/3375158379] up:boot 2024-05-27T07:49:47.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:47 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 9 up:standby 2024-05-27T07:49:47.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:47 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:49:47.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:47 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:49:47.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:47 smithi186 ceph-mon[36647]: pgmap v329: 129 pgs: 129 active+clean; 92 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 315 op/s 2024-05-27T07:49:47.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:47 smithi186 ceph-mon[36647]: osdmap e116: 12 total, 12 up, 12 in 2024-05-27T07:49:47.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:47 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6836/3375158379,v1:172.21.15.163:6837/3375158379] up:boot 2024-05-27T07:49:47.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:47 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 9 up:standby 2024-05-27T07:49:47.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:47 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:49:47.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:47 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:49:47.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:47 smithi114 ceph-mon[32038]: pgmap v329: 129 pgs: 129 active+clean; 92 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 315 op/s 2024-05-27T07:49:47.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:47 smithi114 ceph-mon[32038]: osdmap e116: 12 total, 12 up, 12 in 2024-05-27T07:49:47.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:47 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6836/3375158379,v1:172.21.15.163:6837/3375158379] up:boot 2024-05-27T07:49:47.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:47 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 9 up:standby 2024-05-27T07:49:47.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:47 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:49:47.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:47 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:49:48.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1004282 10.49 MB/sec execute 360 sec latency 174.256 ms 2024-05-27T07:49:49.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1006151 10.49 MB/sec execute 361 sec latency 14.875 ms 2024-05-27T07:49:49.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:49 smithi163 ceph-mon[36451]: pgmap v331: 129 pgs: 129 active+clean; 92 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 183 op/s 2024-05-27T07:49:49.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:49 smithi186 ceph-mon[36647]: pgmap v331: 129 pgs: 129 active+clean; 92 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 183 op/s 2024-05-27T07:49:49.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:49 smithi114 ceph-mon[32038]: pgmap v331: 129 pgs: 129 active+clean; 92 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 183 op/s 2024-05-27T07:49:50.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1007893 10.49 MB/sec execute 362 sec latency 179.806 ms 2024-05-27T07:49:51.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1009373 10.49 MB/sec execute 363 sec latency 84.577 ms 2024-05-27T07:49:51.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:51 smithi163 ceph-mon[36451]: pgmap v332: 129 pgs: 129 active+clean; 93 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 267 op/s 2024-05-27T07:49:51.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:51 smithi186 ceph-mon[36647]: pgmap v332: 129 pgs: 129 active+clean; 93 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 267 op/s 2024-05-27T07:49:51.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:51 smithi114 ceph-mon[32038]: pgmap v332: 129 pgs: 129 active+clean; 93 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 267 op/s 2024-05-27T07:49:52.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1011294 10.48 MB/sec execute 364 sec latency 24.999 ms 2024-05-27T07:49:53.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1012952 10.48 MB/sec execute 365 sec latency 20.952 ms 2024-05-27T07:49:53.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:53 smithi163 ceph-mon[36451]: pgmap v333: 129 pgs: 129 active+clean; 96 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-05-27T07:49:53.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:53 smithi186 ceph-mon[36647]: pgmap v333: 129 pgs: 129 active+clean; 96 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-05-27T07:49:53.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:53 smithi114 ceph-mon[32038]: pgmap v333: 129 pgs: 129 active+clean; 96 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-05-27T07:49:54.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1014912 10.47 MB/sec execute 366 sec latency 32.967 ms 2024-05-27T07:49:55.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1016367 10.48 MB/sec execute 367 sec latency 102.149 ms 2024-05-27T07:49:55.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:55 smithi163 ceph-mon[36451]: pgmap v334: 129 pgs: 129 active+clean; 96 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-05-27T07:49:55.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:55 smithi186 ceph-mon[36647]: pgmap v334: 129 pgs: 129 active+clean; 96 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-05-27T07:49:55.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:55 smithi114 ceph-mon[32038]: pgmap v334: 129 pgs: 129 active+clean; 96 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-05-27T07:49:56.282 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1017741 10.46 MB/sec execute 368 sec latency 63.907 ms 2024-05-27T07:49:57.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1019907 10.47 MB/sec execute 369 sec latency 14.029 ms 2024-05-27T07:49:57.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:57 smithi163 ceph-mon[36451]: pgmap v335: 129 pgs: 129 active+clean; 99 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 323 op/s 2024-05-27T07:49:57.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:57 smithi186 ceph-mon[36647]: pgmap v335: 129 pgs: 129 active+clean; 99 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 323 op/s 2024-05-27T07:49:57.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:57 smithi114 ceph-mon[32038]: pgmap v335: 129 pgs: 129 active+clean; 99 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 323 op/s 2024-05-27T07:49:58.226 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:57 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:49:58.226 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:57 smithi163 systemd[1]: Stopped Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:49:58.226 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:57 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Start request repeated too quickly. 2024-05-27T07:49:58.226 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:57 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-05-27T07:49:58.226 INFO:journalctl@ceph.mds.h.smithi163.stdout:May 27 07:49:57 smithi163 systemd[1]: Failed to start Ceph mds.h for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:49:58.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1021581 10.46 MB/sec execute 370 sec latency 63.268 ms 2024-05-27T07:49:59.282 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1023575 10.47 MB/sec execute 371 sec latency 18.937 ms 2024-05-27T07:49:59.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:49:59 smithi163 ceph-mon[36451]: pgmap v336: 129 pgs: 129 active+clean; 99 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 271 op/s 2024-05-27T07:49:59.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:49:59 smithi186 ceph-mon[36647]: pgmap v336: 129 pgs: 129 active+clean; 99 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 271 op/s 2024-05-27T07:49:59.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:49:59 smithi114 ceph-mon[32038]: pgmap v336: 129 pgs: 129 active+clean; 99 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 271 op/s 2024-05-27T07:50:00.282 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1025508 10.46 MB/sec execute 372 sec latency 38.232 ms 2024-05-27T07:50:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.660+0000 7f79a78a8640 1 -- 172.21.15.114:0/998238699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a01084c0 msgr2=0x7f79a01088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.660+0000 7f79a78a8640 1 --2- 172.21.15.114:0/998238699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a01084c0 0x7f79a01088d0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f7990009900 tx=0x7f799002a410 comp rx=0 tx=0).stop 2024-05-27T07:50:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.660+0000 7f79a78a8640 1 -- 172.21.15.114:0/998238699 shutdown_connections 2024-05-27T07:50:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.660+0000 7f79a78a8640 1 --2- 172.21.15.114:0/998238699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a0108f60 0x7f79a0110700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.660+0000 7f79a78a8640 1 --2- 172.21.15.114:0/998238699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a01084c0 0x7f79a01088d0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.660+0000 7f79a78a8640 1 --2- 172.21.15.114:0/998238699 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79a0102440 0x7f79a0102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.660+0000 7f79a78a8640 1 -- 172.21.15.114:0/998238699 >> 172.21.15.114:0/998238699 conn(0x7f79a00779d0 msgr2=0x7f79a0077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.661+0000 7f79a78a8640 1 -- 172.21.15.114:0/998238699 shutdown_connections 2024-05-27T07:50:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.661+0000 7f79a78a8640 1 -- 172.21.15.114:0/998238699 wait complete. 2024-05-27T07:50:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.661+0000 7f79a78a8640 1 Processor -- start 2024-05-27T07:50:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.661+0000 7f79a78a8640 1 -- start start 2024-05-27T07:50:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a78a8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a0102440 0x7f79a01a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a78a8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a01084c0 0x7f79a01a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a78a8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79a0108f60 0x7f79a019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a78a8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79a0113e70 con 0x7f79a0102440 2024-05-27T07:50:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a78a8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79a0113cf0 con 0x7f79a0108f60 2024-05-27T07:50:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a78a8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79a0113ff0 con 0x7f79a01084c0 2024-05-27T07:50:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a5644640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a0102440 0x7f79a01a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a5644640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a0102440 0x7f79a01a0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57002/0 (socket says 172.21.15.114:57002) 2024-05-27T07:50:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a5644640 1 -- 172.21.15.114:0/444467454 learned_addr learned my addr 172.21.15.114:0/444467454 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a4e43640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a01084c0 0x7f79a01a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a5e45640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79a0108f60 0x7f79a019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a4e43640 1 -- 172.21.15.114:0/444467454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79a0108f60 msgr2=0x7f79a019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a4e43640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79a0108f60 0x7f79a019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a4e43640 1 -- 172.21.15.114:0/444467454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a0102440 msgr2=0x7f79a01a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a4e43640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a0102440 0x7f79a01a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a4e43640 1 -- 172.21.15.114:0/444467454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a019ae40 con 0x7f79a01084c0 2024-05-27T07:50:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.662+0000 7f79a4e43640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a01084c0 0x7f79a01a0ad0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f799002a930 tx=0x7f7990031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.663+0000 7f79967fc640 1 -- 172.21.15.114:0/444467454 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799003f070 con 0x7f79a01084c0 2024-05-27T07:50:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.663+0000 7f79a78a8640 1 -- 172.21.15.114:0/444467454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a019b070 con 0x7f79a01084c0 2024-05-27T07:50:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.663+0000 7f79967fc640 1 -- 172.21.15.114:0/444467454 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7990004490 con 0x7f79a01084c0 2024-05-27T07:50:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.663+0000 7f79967fc640 1 -- 172.21.15.114:0/444467454 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7990035070 con 0x7f79a01084c0 2024-05-27T07:50:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.663+0000 7f79a78a8640 1 -- 172.21.15.114:0/444467454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a019b560 con 0x7f79a01084c0 2024-05-27T07:50:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79ae838740 1 Processor -- start 2024-05-27T07:50:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79ae838740 1 -- start start 2024-05-27T07:50:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79ae838740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80d31b0 0x5575f84743d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79ae838740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80aa2f0 0x5575f8474910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79ae838740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f8474e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79ae838740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5575f83e6c30 con 0x5575f80d31b0 2024-05-27T07:50:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79ae838740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5575f83d67d0 con 0x5575f80aa2f0 2024-05-27T07:50:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79ae838740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5575f83da810 con 0x5575f80b24d0 2024-05-27T07:50:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79a5e45640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80d31b0 0x5575f84743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79a5644640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80aa2f0 0x5575f8474910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79a5644640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80aa2f0 0x5575f8474910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:36508/0 (socket says 172.21.15.114:36508) 2024-05-27T07:50:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79a5644640 1 -- 172.21.15.114:0/3314205196 learned_addr learned my addr 172.21.15.114:0/3314205196 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79967fc640 1 -- 172.21.15.114:0/444467454 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f7990043400 con 0x7f79a01084c0 2024-05-27T07:50:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.666+0000 7f79a5644640 1 -- 172.21.15.114:0/3314205196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 msgr2=0x5575f8474e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.667+0000 7f79a5644640 1 --2- 172.21.15.114:0/3314205196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f8474e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.667+0000 7f79a5644640 1 -- 172.21.15.114:0/3314205196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80d31b0 msgr2=0x5575f84743d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.667+0000 7f79a5644640 1 --2- 172.21.15.114:0/3314205196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80d31b0 0x5575f84743d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.667+0000 7f79a5644640 1 -- 172.21.15.114:0/3314205196 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575f80eb210 con 0x5575f80aa2f0 2024-05-27T07:50:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.667+0000 7f79a5e45640 1 --2- 172.21.15.114:0/3314205196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80d31b0 0x5575f84743d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.667+0000 7f79a5644640 1 --2- 172.21.15.114:0/3314205196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80aa2f0 0x5575f8474910 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f799c009d60 tx=0x7f799c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebb04275d2f95c01 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.667+0000 7f798b7fe640 1 -- 172.21.15.114:0/3314205196 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c004230 con 0x5575f80aa2f0 2024-05-27T07:50:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.667+0000 7f798b7fe640 1 -- 172.21.15.114:0/3314205196 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f799c0043d0 con 0x5575f80aa2f0 2024-05-27T07:50:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.667+0000 7f798b7fe640 1 -- 172.21.15.114:0/3314205196 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c019880 con 0x5575f80aa2f0 2024-05-27T07:50:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.667+0000 7f79967fc640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f798c0b2950 0x7f798c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.667+0000 7f79a5e45640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f798c0b2950 0x7f798c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.668+0000 7f79967fc640 1 -- 172.21.15.114:0/444467454 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9466+0+0 (secure 0 0 0) 0x7f7990102e20 con 0x7f79a01084c0 2024-05-27T07:50:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.668+0000 7f79ae838740 1 -- 172.21.15.114:0/3314205196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80aa2f0 msgr2=0x5575f8474910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.668+0000 7f79a5e45640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f798c0b2950 0x7f798c0b4e60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7998009fd0 tx=0x7f7998009500 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.668+0000 7f79ae838740 1 --2- 172.21.15.114:0/3314205196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80aa2f0 0x5575f8474910 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f799c009d60 tx=0x7f799c011810 comp rx=0 tx=0).stop 2024-05-27T07:50:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.668+0000 7f79ae838740 1 -- 172.21.15.114:0/3314205196 shutdown_connections 2024-05-27T07:50:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.668+0000 7f79ae838740 1 --2- 172.21.15.114:0/3314205196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f8474e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.668+0000 7f79ae838740 1 --2- 172.21.15.114:0/3314205196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80aa2f0 0x5575f8474910 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.668+0000 7f79ae838740 1 --2- 172.21.15.114:0/3314205196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80d31b0 0x5575f84743d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.668+0000 7f79ae838740 1 -- 172.21.15.114:0/3314205196 >> 172.21.15.114:0/3314205196 conn(0x5575f80c7300 msgr2=0x5575f8470b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.668+0000 7f79ae838740 1 -- 172.21.15.114:0/3314205196 shutdown_connections 2024-05-27T07:50:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.668+0000 7f79ae838740 1 -- 172.21.15.114:0/3314205196 wait complete. 2024-05-27T07:50:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.669+0000 7f79ae838740 1 -- start start 2024-05-27T07:50:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.669+0000 7f79ae838740 1 Processor -- start 2024-05-27T07:50:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.669+0000 7f79ae838740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 0x5575f8480420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.669+0000 7f79ae838740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f8480960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.669+0000 7f79ae838740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80d31b0 0x5575f847e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.669+0000 7f79ae838740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5575f80bc790 con 0x5575f80aa2f0 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.669+0000 7f79ae838740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5575f8392890 con 0x5575f80d31b0 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.669+0000 7f79ae838740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5575f8481500 con 0x5575f80b24d0 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79ae838740 10 client.0 _ll_register_callbacks cb 0x5575f80eaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a5e45640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80d31b0 0x5575f847e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a5644640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 0x5575f8480420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a4e43640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f8480960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a5644640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 0x5575f8480420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57010/0 (socket says 172.21.15.114:57010) 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a5e45640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80d31b0 0x5575f847e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:36514/0 (socket says 172.21.15.114:36514) 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a5644640 1 -- 172.21.15.114:0/1495223992 learned_addr learned my addr 172.21.15.114:0/1495223992 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a5e45640 1 -- 172.21.15.114:0/1495223992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 msgr2=0x5575f8480960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a5e45640 1 --2- 172.21.15.114:0/1495223992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f8480960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a5e45640 1 -- 172.21.15.114:0/1495223992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 msgr2=0x5575f8480420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a5e45640 1 --2- 172.21.15.114:0/1495223992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 0x5575f8480420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a5e45640 1 -- 172.21.15.114:0/1495223992 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575f81b7620 con 0x5575f80d31b0 2024-05-27T07:50:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a5644640 1 --2- 172.21.15.114:0/1495223992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 0x5575f8480420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a4e43640 1 --2- 172.21.15.114:0/1495223992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f8480960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f79a5e45640 1 --2- 172.21.15.114:0/1495223992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80d31b0 0x5575f847e930 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7998005fd0 tx=0x7f7998004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd6dec7dc1ce4a0b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f7988ff9640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:50:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.670+0000 7f7988ff9640 1 -- 172.21.15.114:0/1495223992 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7998004b20 con 0x5575f80d31b0 2024-05-27T07:50:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f79ae838740 1 -- 172.21.15.114:0/1495223992 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f799c0043d0 con 0x5575f80d31b0 2024-05-27T07:50:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f7988ff9640 1 -- 172.21.15.114:0/1495223992 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9466+0+0 (secure 0 0 0) 0x7f7998017030 con 0x5575f80d31b0 2024-05-27T07:50:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f7988ff9640 1 -- 172.21.15.114:0/1495223992 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f7998032050 con 0x5575f80d31b0 2024-05-27T07:50:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f79ae838740 10 client.27254 fetch_fsmap learned FSMap version 57 2024-05-27T07:50:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f79ae838740 1 -- 172.21.15.114:0/1495223992 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5575f848d8b0 con 0x5575f80d31b0 2024-05-27T07:50:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f7988ff9640 1 -- 172.21.15.114:0/1495223992 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f799c0043d0 con 0x5575f80d31b0 2024-05-27T07:50:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f7988ff9640 1 -- 172.21.15.114:0/1495223992 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 57) ==== 7120+0+0 (secure 0 0 0) 0x7f79980052c0 con 0x5575f80d31b0 2024-05-27T07:50:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f79ae838740 10 client.27254 fetch_fsmap finished waiting for FSMap version 57 2024-05-27T07:50:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f79ae838740 10 client.27254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f79ae838740 20 client.27254 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f79ae838740 1 --2- 172.21.15.114:0/1495223992 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575f8091d80 0x5575f8490f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f79ae838740 4 client.27254 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:50:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f79ae838740 1 -- 172.21.15.114:0/1495223992 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575f83e8b80 con 0x5575f8091d80 2024-05-27T07:50:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.671+0000 7f79a5644640 1 --2- 172.21.15.114:0/1495223992 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575f8091d80 0x5575f8490f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.672+0000 7f7988ff9640 10 client.27254 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.672+0000 7f79a5644640 1 --2- 172.21.15.114:0/1495223992 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575f8091d80 0x5575f8490f50 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0b1f66d048db488 server_cookie=3dc75ce4602a8a38 in_seq=0 out_seq=0 2024-05-27T07:50:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f7988ff9640 1 -- 172.21.15.114:0/1495223992 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5575f83e8b80 con 0x5575f8091d80 2024-05-27T07:50:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f7988ff9640 10 client.27254 handle_command_reply: tid=0 2024-05-27T07:50:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f79ae838740 1 client.27254 shutdown 2024-05-27T07:50:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f79ae838740 1 -- 172.21.15.114:0/1495223992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80d31b0 msgr2=0x5575f847e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f79ae838740 1 --2- 172.21.15.114:0/1495223992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80d31b0 0x5575f847e930 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7998005fd0 tx=0x7f7998004ab0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f79ae838740 1 -- 172.21.15.114:0/1495223992 shutdown_connections 2024-05-27T07:50:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f79ae838740 1 --2- 172.21.15.114:0/1495223992 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575f8091d80 0x5575f8490f50 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f79ae838740 1 --2- 172.21.15.114:0/1495223992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80d31b0 0x5575f847e930 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f79ae838740 1 --2- 172.21.15.114:0/1495223992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f8480960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f7988ff9640 0 client.27254 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f79ae838740 1 --2- 172.21.15.114:0/1495223992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 0x5575f8480420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f79ae838740 1 -- 172.21.15.114:0/1495223992 >> 172.21.15.114:0/1495223992 conn(0x5575f80c7300 msgr2=0x5575f84760f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f79ae838740 1 -- 172.21.15.114:0/1495223992 shutdown_connections 2024-05-27T07:50:00.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f79ae838740 1 -- 172.21.15.114:0/1495223992 wait complete. 2024-05-27T07:50:00.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.673+0000 7f79ae838740 20 client.27254 trim_cache size 0 max 16384 2024-05-27T07:50:00.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:50:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.691+0000 7f79ae838740 1 Processor -- start 2024-05-27T07:50:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.691+0000 7f79ae838740 1 -- start start 2024-05-27T07:50:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.691+0000 7f79ae838740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f8091d80 0x5575f8478080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.691+0000 7f79ae838740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 0x5575f84785c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.691+0000 7f79ae838740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f8496750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.691+0000 7f79ae838740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01b1480 con 0x5575f80aa2f0 2024-05-27T07:50:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.691+0000 7f79ae838740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01b1b50 con 0x5575f8091d80 2024-05-27T07:50:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.691+0000 7f79ae838740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01af2d0 con 0x5575f80b24d0 2024-05-27T07:50:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.691+0000 7f79a4e43640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 0x5575f84785c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.691+0000 7f79a5644640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f8091d80 0x5575f8478080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.691+0000 7f79a4e43640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 0x5575f84785c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57020/0 (socket says 172.21.15.114:57020) 2024-05-27T07:50:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.691+0000 7f79a4e43640 1 -- 172.21.15.114:0/2066067866 learned_addr learned my addr 172.21.15.114:0/2066067866 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.691+0000 7f79a5e45640 1 --2- 172.21.15.114:0/2066067866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f8496750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.692+0000 7f79a4e43640 1 -- 172.21.15.114:0/2066067866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 msgr2=0x5575f8496750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.692+0000 7f79a4e43640 1 --2- 172.21.15.114:0/2066067866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f8496750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.692+0000 7f79a4e43640 1 -- 172.21.15.114:0/2066067866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f8091d80 msgr2=0x5575f8478080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.692+0000 7f79a4e43640 1 --2- 172.21.15.114:0/2066067866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f8091d80 0x5575f8478080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.692+0000 7f79a4e43640 1 -- 172.21.15.114:0/2066067866 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575f847ce80 con 0x5575f80aa2f0 2024-05-27T07:50:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.692+0000 7f79a5644640 1 --2- 172.21.15.114:0/2066067866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f8091d80 0x5575f8478080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.692+0000 7f79a5e45640 1 --2- 172.21.15.114:0/2066067866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f8496750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.692+0000 7f79a4e43640 1 --2- 172.21.15.114:0/2066067866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 0x5575f84785c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f79900c5e80 tx=0x7f7990003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87021c7b9b31c43a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.692+0000 7f798affd640 1 -- 172.21.15.114:0/2066067866 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7990048040 con 0x5575f80aa2f0 2024-05-27T07:50:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.693+0000 7f798affd640 1 -- 172.21.15.114:0/2066067866 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79900ce100 con 0x5575f80aa2f0 2024-05-27T07:50:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.693+0000 7f798affd640 1 -- 172.21.15.114:0/2066067866 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79900ca060 con 0x5575f80aa2f0 2024-05-27T07:50:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.694+0000 7f79ae838740 1 -- 172.21.15.114:0/2066067866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 msgr2=0x5575f84785c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.694+0000 7f79ae838740 1 --2- 172.21.15.114:0/2066067866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 0x5575f84785c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f79900c5e80 tx=0x7f7990003f50 comp rx=0 tx=0).stop 2024-05-27T07:50:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.694+0000 7f79ae838740 1 -- 172.21.15.114:0/2066067866 shutdown_connections 2024-05-27T07:50:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.694+0000 7f79ae838740 1 --2- 172.21.15.114:0/2066067866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f8496750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.694+0000 7f79ae838740 1 --2- 172.21.15.114:0/2066067866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f80aa2f0 0x5575f84785c0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.694+0000 7f79ae838740 1 --2- 172.21.15.114:0/2066067866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f8091d80 0x5575f8478080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.694+0000 7f79ae838740 1 -- 172.21.15.114:0/2066067866 >> 172.21.15.114:0/2066067866 conn(0x5575f80c7300 msgr2=0x5575f847d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.694+0000 7f79ae838740 1 -- 172.21.15.114:0/2066067866 shutdown_connections 2024-05-27T07:50:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.694+0000 7f79ae838740 1 -- 172.21.15.114:0/2066067866 wait complete. 2024-05-27T07:50:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.694+0000 7f79ae838740 1 -- start start 2024-05-27T07:50:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.694+0000 7f79ae838740 1 Processor -- start 2024-05-27T07:50:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79ae838740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f8091d80 0x5575f8483ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79ae838740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80aa2f0 0x5575f84840e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79ae838740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f84f7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79ae838740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01af940 con 0x5575f8091d80 2024-05-27T07:50:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79ae838740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01b0db0 con 0x5575f80aa2f0 2024-05-27T07:50:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79ae838740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01b0010 con 0x5575f80b24d0 2024-05-27T07:50:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79ae838740 10 client.0 _ll_register_callbacks cb 0x7f7998009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79a5644640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f8091d80 0x5575f8483ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79a5e45640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f84f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79a5644640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f8091d80 0x5575f8483ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57034/0 (socket says 172.21.15.114:57034) 2024-05-27T07:50:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79a5644640 1 -- 172.21.15.114:0/2917584582 learned_addr learned my addr 172.21.15.114:0/2917584582 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79a4e43640 1 --2- 172.21.15.114:0/2917584582 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80aa2f0 0x5575f84840e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79a5644640 1 -- 172.21.15.114:0/2917584582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 msgr2=0x5575f84f7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79a5644640 1 --2- 172.21.15.114:0/2917584582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f84f7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79a5644640 1 -- 172.21.15.114:0/2917584582 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80aa2f0 msgr2=0x5575f84840e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79a5644640 1 --2- 172.21.15.114:0/2917584582 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80aa2f0 0x5575f84840e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79a5644640 1 -- 172.21.15.114:0/2917584582 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575f84f7b60 con 0x5575f8091d80 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.695+0000 7f79a5e45640 1 --2- 172.21.15.114:0/2917584582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f84f7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.696+0000 7f79a5644640 1 --2- 172.21.15.114:0/2917584582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f8091d80 0x5575f8483ba0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f799c009a20 tx=0x7f799c005330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e1d5688f80bb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.696+0000 7f7988ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.696+0000 7f7988ff9640 1 -- 172.21.15.114:0/2917584582 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c032070 con 0x5575f8091d80 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.696+0000 7f7988ff9640 1 -- 172.21.15.114:0/2917584582 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9466+0+0 (secure 0 0 0) 0x7f799c01f040 con 0x5575f8091d80 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.696+0000 7f79ae838740 1 -- 172.21.15.114:0/2917584582 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79900ce100 con 0x5575f8091d80 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.697+0000 7f7988ff9640 1 -- 172.21.15.114:0/2917584582 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f799c03e050 con 0x5575f8091d80 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.697+0000 7f79ae838740 10 client.17805 fetch_fsmap learned FSMap version 57 2024-05-27T07:50:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.697+0000 7f79ae838740 1 -- 172.21.15.114:0/2917584582 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5575f84f8ca0 con 0x5575f8091d80 2024-05-27T07:50:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.697+0000 7f7988ff9640 1 -- 172.21.15.114:0/2917584582 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79900ce100 con 0x5575f8091d80 2024-05-27T07:50:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.697+0000 7f7988ff9640 1 -- 172.21.15.114:0/2917584582 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 57) ==== 7120+0+0 (secure 0 0 0) 0x7f799c019950 con 0x5575f8091d80 2024-05-27T07:50:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.697+0000 7f79ae838740 10 client.17805 fetch_fsmap finished waiting for FSMap version 57 2024-05-27T07:50:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.697+0000 7f79ae838740 10 client.17805 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.697+0000 7f79ae838740 20 client.17805 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.697+0000 7f79ae838740 1 --2- 172.21.15.114:0/2917584582 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575f80d31b0 0x5575f84fb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.697+0000 7f79ae838740 4 client.17805 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:50:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.697+0000 7f79ae838740 1 -- 172.21.15.114:0/2917584582 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575f83e2360 con 0x5575f80d31b0 2024-05-27T07:50:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.697+0000 7f79a4e43640 1 --2- 172.21.15.114:0/2917584582 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575f80d31b0 0x5575f84fb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.698+0000 7f7988ff9640 10 client.17805 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:00.698+0000 7f79a4e43640 1 --2- 172.21.15.114:0/2917584582 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575f80d31b0 0x5575f84fb3e0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32741a4e7f067d3e server_cookie=b7249ca3acbffc75 in_seq=0 out_seq=0 2024-05-27T07:50:00.715 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:00 smithi114 ceph-mon[32038]: overall HEALTH_OK 2024-05-27T07:50:00.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:00 smithi163 ceph-mon[36451]: overall HEALTH_OK 2024-05-27T07:50:00.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:00 smithi186 ceph-mon[36647]: overall HEALTH_OK 2024-05-27T07:50:01.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.125+0000 7f7988ff9640 1 -- 172.21.15.114:0/2917584582 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5575f83e2360 con 0x5575f80d31b0 2024-05-27T07:50:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.125+0000 7f7988ff9640 10 client.17805 handle_command_reply: tid=0 2024-05-27T07:50:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.125+0000 7f79ae838740 1 client.17805 shutdown 2024-05-27T07:50:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.125+0000 7f79ae838740 1 -- 172.21.15.114:0/2917584582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f8091d80 msgr2=0x5575f8483ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.125+0000 7f79ae838740 1 --2- 172.21.15.114:0/2917584582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f8091d80 0x5575f8483ba0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f799c009a20 tx=0x7f799c005330 comp rx=0 tx=0).stop 2024-05-27T07:50:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.126+0000 7f79ae838740 1 -- 172.21.15.114:0/2917584582 shutdown_connections 2024-05-27T07:50:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.126+0000 7f79ae838740 1 --2- 172.21.15.114:0/2917584582 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5575f80d31b0 0x5575f84fb3e0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.126+0000 7f79ae838740 1 --2- 172.21.15.114:0/2917584582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575f80b24d0 0x5575f84f7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.126+0000 7f79ae838740 1 --2- 172.21.15.114:0/2917584582 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5575f80aa2f0 0x5575f84840e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.126+0000 7f79ae838740 1 --2- 172.21.15.114:0/2917584582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f8091d80 0x5575f8483ba0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.126+0000 7f79ae838740 1 -- 172.21.15.114:0/2917584582 >> 172.21.15.114:0/2917584582 conn(0x5575f80c7300 msgr2=0x5575f8476010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.126+0000 7f7988ff9640 0 client.17805 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.126+0000 7f79ae838740 1 -- 172.21.15.114:0/2917584582 shutdown_connections 2024-05-27T07:50:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.126+0000 7f79ae838740 1 -- 172.21.15.114:0/2917584582 wait complete. 2024-05-27T07:50:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.126+0000 7f79ae838740 20 client.17805 trim_cache size 0 max 16384 2024-05-27T07:50:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:50:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:50:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:50:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.127+0000 7f79a78a8640 1 -- 172.21.15.114:0/444467454 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f798c0b2950 msgr2=0x7f798c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.127+0000 7f79a78a8640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f798c0b2950 0x7f798c0b4e60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7998009fd0 tx=0x7f7998009500 comp rx=0 tx=0).stop 2024-05-27T07:50:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.127+0000 7f79a78a8640 1 -- 172.21.15.114:0/444467454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a01084c0 msgr2=0x7f79a01a0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.127+0000 7f79a78a8640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a01084c0 0x7f79a01a0ad0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f799002a930 tx=0x7f7990031a20 comp rx=0 tx=0).stop 2024-05-27T07:50:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.127+0000 7f79a78a8640 1 -- 172.21.15.114:0/444467454 shutdown_connections 2024-05-27T07:50:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.127+0000 7f79a78a8640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f798c0b2950 0x7f798c0b4e60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.127+0000 7f79a78a8640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79a0108f60 0x7f79a019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.127+0000 7f79a78a8640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79a01084c0 0x7f79a01a0ad0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.127+0000 7f79a78a8640 1 --2- 172.21.15.114:0/444467454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a0102440 0x7f79a01a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.127+0000 7f79a78a8640 1 -- 172.21.15.114:0/444467454 >> 172.21.15.114:0/444467454 conn(0x7f79a00779d0 msgr2=0x7f79a0100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.127+0000 7f79a78a8640 1 -- 172.21.15.114:0/444467454 shutdown_connections 2024-05-27T07:50:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:01.128+0000 7f79a78a8640 1 -- 172.21.15.114:0/444467454 wait complete. 2024-05-27T07:50:01.136 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:50:01.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1026381 10.45 MB/sec execute 373 sec latency 478.111 ms 2024-05-27T07:50:01.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:01 smithi114 ceph-mon[32038]: pgmap v337: 129 pgs: 129 active+clean; 98 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 285 op/s 2024-05-27T07:50:01.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:01 smithi163 ceph-mon[36451]: pgmap v337: 129 pgs: 129 active+clean; 98 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 285 op/s 2024-05-27T07:50:01.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:01 smithi186 ceph-mon[36647]: pgmap v337: 129 pgs: 129 active+clean; 98 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 285 op/s 2024-05-27T07:50:02.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1026512 10.42 MB/sec execute 374 sec latency 94.700 ms 2024-05-27T07:50:03.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.148+0000 7f67427d8700 1 -- 172.21.15.114:0/2436002265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c108750 msgr2=0x7f673c1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:03.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.148+0000 7f67427d8700 1 --2- 172.21.15.114:0/2436002265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c108750 0x7f673c1100b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6734009fc0 tx=0x7f673400b330 comp rx=0 tx=0).stop 2024-05-27T07:50:03.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.151+0000 7f67427d8700 1 -- 172.21.15.114:0/2436002265 shutdown_connections 2024-05-27T07:50:03.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.151+0000 7f67427d8700 1 --2- 172.21.15.114:0/2436002265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c108750 0x7f673c1100b0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.151+0000 7f67427d8700 1 --2- 172.21.15.114:0/2436002265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f673c1034d0 0x7f673c1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.151+0000 7f67427d8700 1 --2- 172.21.15.114:0/2436002265 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f673c107e00 0x7f673c108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.151+0000 7f67427d8700 1 -- 172.21.15.114:0/2436002265 >> 172.21.15.114:0/2436002265 conn(0x7f673c0fef20 msgr2=0x7f673c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:03.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.155+0000 7f67427d8700 1 -- 172.21.15.114:0/2436002265 shutdown_connections 2024-05-27T07:50:03.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.158+0000 7f67427d8700 1 -- 172.21.15.114:0/2436002265 wait complete. 2024-05-27T07:50:03.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.159+0000 7f67427d8700 1 Processor -- start 2024-05-27T07:50:03.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.166+0000 7f67427d8700 1 -- start start 2024-05-27T07:50:03.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.166+0000 7f67427d8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f673c1034d0 0x7f673c19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.166+0000 7f67427d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c107e00 0x7f673c198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.166+0000 7f67427d8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f673c108750 0x7f673c1991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.166+0000 7f67427d8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f673c112330 con 0x7f673c107e00 2024-05-27T07:50:03.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.166+0000 7f67427d8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f673c1121b0 con 0x7f673c108750 2024-05-27T07:50:03.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.166+0000 7f67427d8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f673c1124b0 con 0x7f673c1034d0 2024-05-27T07:50:03.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.168+0000 7f6733fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c107e00 0x7f673c198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.168+0000 7f6733fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c107e00 0x7f673c198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57050/0 (socket says 172.21.15.114:57050) 2024-05-27T07:50:03.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.168+0000 7f6733fff700 1 -- 172.21.15.114:0/3773348882 learned_addr learned my addr 172.21.15.114:0/3773348882 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:03.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.168+0000 7f6740d75700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f673c108750 0x7f673c1991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.170+0000 7f673bfff700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f673c1034d0 0x7f673c19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.173+0000 7f6733fff700 1 -- 172.21.15.114:0/3773348882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f673c1034d0 msgr2=0x7f673c19f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:03.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.173+0000 7f6733fff700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f673c1034d0 0x7f673c19f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.173+0000 7f6733fff700 1 -- 172.21.15.114:0/3773348882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f673c108750 msgr2=0x7f673c1991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:03.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.173+0000 7f6733fff700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f673c108750 0x7f673c1991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.173+0000 7f6733fff700 1 -- 172.21.15.114:0/3773348882 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f673c19d500 con 0x7f673c107e00 2024-05-27T07:50:03.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.174+0000 7f6733fff700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c107e00 0x7f673c198b50 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f672c00d980 tx=0x7f672c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:03.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.174+0000 7f6739ffb700 1 -- 172.21.15.114:0/3773348882 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c017070 con 0x7f673c107e00 2024-05-27T07:50:03.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.174+0000 7f6739ffb700 1 -- 172.21.15.114:0/3773348882 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f672c010460 con 0x7f673c107e00 2024-05-27T07:50:03.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.174+0000 7f6739ffb700 1 -- 172.21.15.114:0/3773348882 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c00de60 con 0x7f673c107e00 2024-05-27T07:50:03.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.183+0000 7f67427d8700 1 -- 172.21.15.114:0/3773348882 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f673c19d7f0 con 0x7f673c107e00 2024-05-27T07:50:03.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.183+0000 7f67427d8700 1 -- 172.21.15.114:0/3773348882 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f673c19dcb0 con 0x7f673c107e00 2024-05-27T07:50:03.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.189+0000 7f67511edb80 1 Processor -- start 2024-05-27T07:50:03.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.190+0000 7f6739ffb700 1 -- 172.21.15.114:0/3773348882 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f672c010610 con 0x7f673c107e00 2024-05-27T07:50:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.195+0000 7f6739ffb700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67280b2bf0 0x7f67280b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.195+0000 7f6739ffb700 1 -- 172.21.15.114:0/3773348882 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7f672c0e2650 con 0x7f673c107e00 2024-05-27T07:50:03.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.196+0000 7f6740d75700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67280b2bf0 0x7f67280b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.196+0000 7f67511edb80 1 -- start start 2024-05-27T07:50:03.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.196+0000 7f67511edb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeae90e40 0x564eeb015b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.196+0000 7f67511edb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb016070 0x564eeb00fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.196+0000 7f67511edb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb010530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.196+0000 7f67511edb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564eeaf3dd70 con 0x564eeae90e40 2024-05-27T07:50:03.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.196+0000 7f67511edb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564eeaf2d910 con 0x564eeb010120 2024-05-27T07:50:03.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.196+0000 7f67511edb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564eeaf31950 con 0x564eeb016070 2024-05-27T07:50:03.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.197+0000 7f6733fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb016070 0x564eeb00fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.197+0000 7f6733fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb016070 0x564eeb00fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:38616/0 (socket says 172.21.15.114:38616) 2024-05-27T07:50:03.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.197+0000 7f6733fff700 1 -- 172.21.15.114:0/2180347969 learned_addr learned my addr 172.21.15.114:0/2180347969 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:03.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.198+0000 7f6740d75700 1 --2- 172.21.15.114:0/2180347969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb010530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.198+0000 7f6733fff700 1 -- 172.21.15.114:0/2180347969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 msgr2=0x564eeb010530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:03.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.198+0000 7f6733fff700 1 --2- 172.21.15.114:0/2180347969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb010530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.198+0000 7f6733fff700 1 -- 172.21.15.114:0/2180347969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeae90e40 msgr2=0x564eeb015b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:03.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.198+0000 7f6733fff700 1 --2- 172.21.15.114:0/2180347969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeae90e40 0x564eeb015b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.198+0000 7f6733fff700 1 -- 172.21.15.114:0/2180347969 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eeaed6fb0 con 0x564eeb016070 2024-05-27T07:50:03.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.199+0000 7f6733fff700 1 --2- 172.21.15.114:0/2180347969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb016070 0x564eeb00fbe0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f672c0a5570 tx=0x7f672c0a9bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9f6ac857b98fb08 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:03.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.201+0000 7f67327fc700 1 -- 172.21.15.114:0/2180347969 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c0ad8a0 con 0x564eeb016070 2024-05-27T07:50:03.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.201+0000 7f67327fc700 1 -- 172.21.15.114:0/2180347969 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f672c0b2350 con 0x564eeb016070 2024-05-27T07:50:03.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.201+0000 7f67327fc700 1 -- 172.21.15.114:0/2180347969 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c0a89b0 con 0x564eeb016070 2024-05-27T07:50:03.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.203+0000 7f67511edb80 1 -- 172.21.15.114:0/2180347969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb016070 msgr2=0x564eeb00fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:03.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.203+0000 7f67511edb80 1 --2- 172.21.15.114:0/2180347969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb016070 0x564eeb00fbe0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f672c0a5570 tx=0x7f672c0a9bf0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.205+0000 7f6740d75700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67280b2bf0 0x7f67280b5100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6734009840 tx=0x7f6734009c00 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:03.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.205+0000 7f67511edb80 1 -- 172.21.15.114:0/2180347969 shutdown_connections 2024-05-27T07:50:03.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.205+0000 7f67511edb80 1 --2- 172.21.15.114:0/2180347969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb010530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.205+0000 7f67511edb80 1 --2- 172.21.15.114:0/2180347969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeae90e40 0x564eeb015b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.205+0000 7f67511edb80 1 --2- 172.21.15.114:0/2180347969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb016070 0x564eeb00fbe0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.205+0000 7f67511edb80 1 -- 172.21.15.114:0/2180347969 >> 172.21.15.114:0/2180347969 conn(0x564eeae93770 msgr2=0x564eeb00ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:03.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.209+0000 7f67511edb80 1 -- 172.21.15.114:0/2180347969 shutdown_connections 2024-05-27T07:50:03.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.213+0000 7f67511edb80 1 -- 172.21.15.114:0/2180347969 wait complete. 2024-05-27T07:50:03.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.213+0000 7f67511edb80 1 -- start start 2024-05-27T07:50:03.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.214+0000 7f67511edb80 1 Processor -- start 2024-05-27T07:50:03.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.219+0000 7f67511edb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeae90e40 0x564eeb019960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.219+0000 7f67511edb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb019ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.219+0000 7f67511edb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb016070 0x564eeb01a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.219+0000 7f67511edb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564eeae33fa0 con 0x564eeb016070 2024-05-27T07:50:03.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.219+0000 7f67511edb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564eeae95da0 con 0x564eeb010120 2024-05-27T07:50:03.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.219+0000 7f67511edb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564eeb01cb00 con 0x564eeae90e40 2024-05-27T07:50:03.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.219+0000 7f67511edb80 10 client.0 _ll_register_callbacks cb 0x564eeaec45b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:03.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.220+0000 7f6733fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb019ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.220+0000 7f673bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeae90e40 0x564eeb019960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.220+0000 7f673bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeae90e40 0x564eeb019960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:38624/0 (socket says 172.21.15.114:38624) 2024-05-27T07:50:03.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.220+0000 7f673bfff700 1 -- 172.21.15.114:0/696416207 learned_addr learned my addr 172.21.15.114:0/696416207 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:03.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.222+0000 7f673bfff700 1 -- 172.21.15.114:0/696416207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 msgr2=0x564eeb019ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:03.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.222+0000 7f673bfff700 1 --2- 172.21.15.114:0/696416207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb019ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.222+0000 7f673bfff700 1 -- 172.21.15.114:0/696416207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb016070 msgr2=0x564eeb01a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:03.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.222+0000 7f673bfff700 1 --2- 172.21.15.114:0/696416207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb016070 0x564eeb01a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.222+0000 7f673bfff700 1 -- 172.21.15.114:0/696416207 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eeae7d8d0 con 0x564eeae90e40 2024-05-27T07:50:03.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.224+0000 7f6733fff700 1 --2- 172.21.15.114:0/696416207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb019ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:50:03.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.224+0000 7f673bfff700 1 --2- 172.21.15.114:0/696416207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeae90e40 0x564eeb019960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6724007fd0 tx=0x7f672400d4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fcac023f0ee58ab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:03.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.224+0000 7f671b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:50:03.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.226+0000 7f671b7fe700 1 -- 172.21.15.114:0/696416207 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672400de30 con 0x564eeae90e40 2024-05-27T07:50:03.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.226+0000 7f67511edb80 1 -- 172.21.15.114:0/696416207 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f672c0b2350 con 0x564eeae90e40 2024-05-27T07:50:03.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.226+0000 7f671b7fe700 1 -- 172.21.15.114:0/696416207 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7f6724013480 con 0x564eeae90e40 2024-05-27T07:50:03.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.227+0000 7f671b7fe700 1 -- 172.21.15.114:0/696416207 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f6724013680 con 0x564eeae90e40 2024-05-27T07:50:03.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.228+0000 7f67511edb80 10 client.27388 fetch_fsmap learned FSMap version 57 2024-05-27T07:50:03.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.228+0000 7f67511edb80 1 -- 172.21.15.114:0/696416207 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x564eeaee7760 con 0x564eeae90e40 2024-05-27T07:50:03.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.234+0000 7f671b7fe700 1 -- 172.21.15.114:0/696416207 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f672c0b2350 con 0x564eeae90e40 2024-05-27T07:50:03.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.234+0000 7f671b7fe700 1 -- 172.21.15.114:0/696416207 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 57) ==== 7120+0+0 (secure 0 0 0) 0x7f672401d980 con 0x564eeae90e40 2024-05-27T07:50:03.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.234+0000 7f67511edb80 10 client.27388 fetch_fsmap finished waiting for FSMap version 57 2024-05-27T07:50:03.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.234+0000 7f67511edb80 10 client.27388 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:03.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.234+0000 7f67511edb80 20 client.27388 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:03.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.234+0000 7f67511edb80 1 --2- 172.21.15.114:0/696416207 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564eeb029d80 0x564eeb02c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.234+0000 7f67511edb80 4 client.27388 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:50:03.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.234+0000 7f67511edb80 1 -- 172.21.15.114:0/696416207 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564eeaf3fcc0 con 0x564eeb029d80 2024-05-27T07:50:03.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.239+0000 7f6733fff700 1 --2- 172.21.15.114:0/696416207 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564eeb029d80 0x564eeb02c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.240+0000 7f6733fff700 1 --2- 172.21.15.114:0/696416207 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564eeb029d80 0x564eeb02c230 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90d2fdba54d62782 server_cookie=1dc3ed8ddd32b53b in_seq=0 out_seq=0 2024-05-27T07:50:03.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.240+0000 7f671b7fe700 10 client.27388 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:03.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.242+0000 7f671b7fe700 1 -- 172.21.15.114:0/696416207 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564eeaf3fcc0 con 0x564eeb029d80 2024-05-27T07:50:03.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.242+0000 7f671b7fe700 10 client.27388 handle_command_reply: tid=0 2024-05-27T07:50:03.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.242+0000 7f67511edb80 1 client.27388 shutdown 2024-05-27T07:50:03.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.243+0000 7f67511edb80 1 -- 172.21.15.114:0/696416207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeae90e40 msgr2=0x564eeb019960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:03.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.243+0000 7f67511edb80 1 --2- 172.21.15.114:0/696416207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeae90e40 0x564eeb019960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6724007fd0 tx=0x7f672400d4f0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.243+0000 7f67511edb80 1 -- 172.21.15.114:0/696416207 shutdown_connections 2024-05-27T07:50:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.243+0000 7f67511edb80 1 --2- 172.21.15.114:0/696416207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb016070 0x564eeb01a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.243+0000 7f67511edb80 1 --2- 172.21.15.114:0/696416207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeae90e40 0x564eeb019960 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.243+0000 7f67511edb80 1 --2- 172.21.15.114:0/696416207 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564eeb029d80 0x564eeb02c230 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.243+0000 7f67511edb80 1 --2- 172.21.15.114:0/696416207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb019ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.243+0000 7f671b7fe700 0 client.27388 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.243+0000 7f67511edb80 1 -- 172.21.15.114:0/696416207 >> 172.21.15.114:0/696416207 conn(0x564eeae93770 msgr2=0x564eeb00ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.243+0000 7f67511edb80 1 -- 172.21.15.114:0/696416207 shutdown_connections 2024-05-27T07:50:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.243+0000 7f67511edb80 1 -- 172.21.15.114:0/696416207 wait complete. 2024-05-27T07:50:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.244+0000 7f67511edb80 20 client.27388 trim_cache size 0 max 16384 2024-05-27T07:50:03.281 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1026902 10.41 MB/sec execute 375 sec latency 68.716 ms 2024-05-27T07:50:03.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.283+0000 7f67511edb80 1 Processor -- start 2024-05-27T07:50:03.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.283+0000 7f67511edb80 1 -- start start 2024-05-27T07:50:03.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.284+0000 7f67511edb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeae90e40 0x564eeb01a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.284+0000 7f67511edb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb010120 0x564eeb014970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.284+0000 7f67511edb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb016070 0x564eeb014eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.284+0000 7f67511edb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f673c1bf9f0 con 0x564eeb010120 2024-05-27T07:50:03.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.284+0000 7f67511edb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f673c1c00c0 con 0x564eeae90e40 2024-05-27T07:50:03.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.284+0000 7f67511edb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f673c1bd860 con 0x564eeb016070 2024-05-27T07:50:03.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.286+0000 7f6740d75700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb016070 0x564eeb014eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.286+0000 7f6733fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb010120 0x564eeb014970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.286+0000 7f6733fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb010120 0x564eeb014970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57078/0 (socket says 172.21.15.114:57078) 2024-05-27T07:50:03.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.286+0000 7f6733fff700 1 -- 172.21.15.114:0/1442314561 learned_addr learned my addr 172.21.15.114:0/1442314561 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:03.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.290+0000 7f673bfff700 1 --2- 172.21.15.114:0/1442314561 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeae90e40 0x564eeb01a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.291+0000 7f6740d75700 1 -- 172.21.15.114:0/1442314561 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeae90e40 msgr2=0x564eeb01a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:03.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.291+0000 7f6740d75700 1 --2- 172.21.15.114:0/1442314561 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeae90e40 0x564eeb01a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.291+0000 7f6740d75700 1 -- 172.21.15.114:0/1442314561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb010120 msgr2=0x564eeb014970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:03.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.291+0000 7f6740d75700 1 --2- 172.21.15.114:0/1442314561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb010120 0x564eeb014970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.291+0000 7f6740d75700 1 -- 172.21.15.114:0/1442314561 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eeb00bab0 con 0x564eeb016070 2024-05-27T07:50:03.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.291+0000 7f6740d75700 1 --2- 172.21.15.114:0/1442314561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb016070 0x564eeb014eb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f673400b770 tx=0x7f673400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de4e2242be21a4e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:03.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.292+0000 7f6731ffb700 1 -- 172.21.15.114:0/1442314561 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6734009660 con 0x564eeb016070 2024-05-27T07:50:03.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.293+0000 7f6731ffb700 1 -- 172.21.15.114:0/1442314561 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f673400bdb0 con 0x564eeb016070 2024-05-27T07:50:03.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.294+0000 7f67511edb80 1 -- 172.21.15.114:0/1442314561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb016070 msgr2=0x564eeb014eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:03.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.294+0000 7f67511edb80 1 --2- 172.21.15.114:0/1442314561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb016070 0x564eeb014eb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f673400b770 tx=0x7f673400b330 comp rx=0 tx=0).stop 2024-05-27T07:50:03.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.294+0000 7f67511edb80 1 -- 172.21.15.114:0/1442314561 shutdown_connections 2024-05-27T07:50:03.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.294+0000 7f67511edb80 1 --2- 172.21.15.114:0/1442314561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb016070 0x564eeb014eb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.294+0000 7f67511edb80 1 --2- 172.21.15.114:0/1442314561 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeae90e40 0x564eeb01a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.294+0000 7f67511edb80 1 --2- 172.21.15.114:0/1442314561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb010120 0x564eeb014970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.294+0000 7f67511edb80 1 -- 172.21.15.114:0/1442314561 >> 172.21.15.114:0/1442314561 conn(0x564eeae93770 msgr2=0x564eeb018250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:03.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.295+0000 7f67511edb80 1 -- 172.21.15.114:0/1442314561 shutdown_connections 2024-05-27T07:50:03.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.295+0000 7f67511edb80 1 -- 172.21.15.114:0/1442314561 wait complete. 2024-05-27T07:50:03.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.295+0000 7f67511edb80 1 -- start start 2024-05-27T07:50:03.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.296+0000 7f67511edb80 1 Processor -- start 2024-05-27T07:50:03.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.296+0000 7f67511edb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb022330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.296+0000 7f67511edb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb016070 0x564eeb0310e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.296+0000 7f67511edb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb031620 0x564eeb031a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.296+0000 7f67511edb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f673c1bdeb0 con 0x564eeb016070 2024-05-27T07:50:03.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.296+0000 7f67511edb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f673c1bf320 con 0x564eeb010120 2024-05-27T07:50:03.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.296+0000 7f67511edb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f673c1be580 con 0x564eeb031620 2024-05-27T07:50:03.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.296+0000 7f67511edb80 10 client.0 _ll_register_callbacks cb 0x564eeaec45b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:03.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.297+0000 7f673bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb022330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.297+0000 7f673bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb022330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:36604/0 (socket says 172.21.15.114:36604) 2024-05-27T07:50:03.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.297+0000 7f673bfff700 1 -- 172.21.15.114:0/128381441 learned_addr learned my addr 172.21.15.114:0/128381441 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:03.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.297+0000 7f6740d75700 1 --2- 172.21.15.114:0/128381441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb031620 0x564eeb031a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.297+0000 7f6733fff700 1 --2- 172.21.15.114:0/128381441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb016070 0x564eeb0310e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.297+0000 7f673bfff700 1 -- 172.21.15.114:0/128381441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb031620 msgr2=0x564eeb031a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:03.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.297+0000 7f673bfff700 1 --2- 172.21.15.114:0/128381441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb031620 0x564eeb031a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.297+0000 7f673bfff700 1 -- 172.21.15.114:0/128381441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb016070 msgr2=0x564eeb0310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:03.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.297+0000 7f673bfff700 1 --2- 172.21.15.114:0/128381441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb016070 0x564eeb0310e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.297+0000 7f673bfff700 1 -- 172.21.15.114:0/128381441 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eeb0938c0 con 0x564eeb010120 2024-05-27T07:50:03.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.298+0000 7f673bfff700 1 --2- 172.21.15.114:0/128381441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb022330 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f672400d8f0 tx=0x7f6724013510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86951cc7b4d6bdc2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:03.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.298+0000 7f671b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:50:03.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.300+0000 7f671b7fe700 1 -- 172.21.15.114:0/128381441 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672402b070 con 0x564eeb010120 2024-05-27T07:50:03.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.300+0000 7f671b7fe700 1 -- 172.21.15.114:0/128381441 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7f6724013af0 con 0x564eeb010120 2024-05-27T07:50:03.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.300+0000 7f67511edb80 1 -- 172.21.15.114:0/128381441 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f673400bdb0 con 0x564eeb010120 2024-05-27T07:50:03.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.304+0000 7f671b7fe700 1 -- 172.21.15.114:0/128381441 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f6724035050 con 0x564eeb010120 2024-05-27T07:50:03.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.304+0000 7f67511edb80 10 client.27269 fetch_fsmap learned FSMap version 57 2024-05-27T07:50:03.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.304+0000 7f67511edb80 1 -- 172.21.15.114:0/128381441 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x564eeb094a90 con 0x564eeb010120 2024-05-27T07:50:03.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.305+0000 7f671b7fe700 1 -- 172.21.15.114:0/128381441 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f673400bdb0 con 0x564eeb010120 2024-05-27T07:50:03.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.305+0000 7f671b7fe700 1 -- 172.21.15.114:0/128381441 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 57) ==== 7120+0+0 (secure 0 0 0) 0x7f6724025040 con 0x564eeb010120 2024-05-27T07:50:03.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.305+0000 7f67511edb80 10 client.27269 fetch_fsmap finished waiting for FSMap version 57 2024-05-27T07:50:03.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.305+0000 7f67511edb80 10 client.27269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:03.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.305+0000 7f67511edb80 20 client.27269 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:03.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.305+0000 7f67511edb80 1 --2- 172.21.15.114:0/128381441 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564eeb095110 0x564eeb097530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:03.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.305+0000 7f67511edb80 4 client.27269 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:50:03.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.305+0000 7f67511edb80 1 -- 172.21.15.114:0/128381441 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564eeaf394a0 con 0x564eeb095110 2024-05-27T07:50:03.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.307+0000 7f6733fff700 1 --2- 172.21.15.114:0/128381441 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564eeb095110 0x564eeb097530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:03.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.311+0000 7f6733fff700 1 --2- 172.21.15.114:0/128381441 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564eeb095110 0x564eeb097530 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27cdcedc52590bb8 server_cookie=f60b443d633100cb in_seq=0 out_seq=0 2024-05-27T07:50:03.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.311+0000 7f671b7fe700 10 client.27269 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:03.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.314+0000 7f671b7fe700 1 -- 172.21.15.114:0/128381441 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564eeaf394a0 con 0x564eeb095110 2024-05-27T07:50:03.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.314+0000 7f671b7fe700 10 client.27269 handle_command_reply: tid=0 2024-05-27T07:50:03.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.314+0000 7f67511edb80 1 client.27269 shutdown 2024-05-27T07:50:03.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.319+0000 7f67511edb80 1 -- 172.21.15.114:0/128381441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 msgr2=0x564eeb022330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:03.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.319+0000 7f67511edb80 1 --2- 172.21.15.114:0/128381441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb022330 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f672400d8f0 tx=0x7f6724013510 comp rx=0 tx=0).stop 2024-05-27T07:50:03.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.320+0000 7f67511edb80 1 -- 172.21.15.114:0/128381441 shutdown_connections 2024-05-27T07:50:03.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.320+0000 7f67511edb80 1 --2- 172.21.15.114:0/128381441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564eeb031620 0x564eeb031a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.320+0000 7f67511edb80 1 --2- 172.21.15.114:0/128381441 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564eeb095110 0x564eeb097530 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.320+0000 7f67511edb80 1 --2- 172.21.15.114:0/128381441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eeb010120 0x564eeb022330 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.320+0000 7f67511edb80 1 --2- 172.21.15.114:0/128381441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564eeb016070 0x564eeb0310e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.320+0000 7f67511edb80 1 -- 172.21.15.114:0/128381441 >> 172.21.15.114:0/128381441 conn(0x564eeae93770 msgr2=0x564eeb014970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:03.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.320+0000 7f671b7fe700 0 client.27269 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:03.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.323+0000 7f67511edb80 1 -- 172.21.15.114:0/128381441 shutdown_connections 2024-05-27T07:50:03.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.324+0000 7f67511edb80 1 -- 172.21.15.114:0/128381441 wait complete. 2024-05-27T07:50:03.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.325+0000 7f67511edb80 20 client.27269 trim_cache size 0 max 16384 2024-05-27T07:50:03.327 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:50:03.327 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:50:03.327 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:50:03.327 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:50:03.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.327+0000 7f67427d8700 1 -- 172.21.15.114:0/3773348882 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67280b2bf0 msgr2=0x7f67280b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:03.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.327+0000 7f67427d8700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67280b2bf0 0x7f67280b5100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6734009840 tx=0x7f6734009c00 comp rx=0 tx=0).stop 2024-05-27T07:50:03.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.327+0000 7f67427d8700 1 -- 172.21.15.114:0/3773348882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c107e00 msgr2=0x7f673c198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:03.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.327+0000 7f67427d8700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c107e00 0x7f673c198b50 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f672c00d980 tx=0x7f672c00dd40 comp rx=0 tx=0).stop 2024-05-27T07:50:03.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.327+0000 7f67427d8700 1 -- 172.21.15.114:0/3773348882 shutdown_connections 2024-05-27T07:50:03.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.327+0000 7f67427d8700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f67280b2bf0 0x7f67280b5100 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.327+0000 7f67427d8700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f673c108750 0x7f673c1991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.327+0000 7f67427d8700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f673c1034d0 0x7f673c19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.327+0000 7f67427d8700 1 --2- 172.21.15.114:0/3773348882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c107e00 0x7f673c198b50 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:03.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.327+0000 7f67427d8700 1 -- 172.21.15.114:0/3773348882 >> 172.21.15.114:0/3773348882 conn(0x7f673c0fef20 msgr2=0x7f673c107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:03.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.328+0000 7f67427d8700 1 -- 172.21.15.114:0/3773348882 shutdown_connections 2024-05-27T07:50:03.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:03.329+0000 7f67427d8700 1 -- 172.21.15.114:0/3773348882 wait complete. 2024-05-27T07:50:03.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:03 smithi114 ceph-mon[32038]: pgmap v338: 129 pgs: 129 active+clean; 102 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 283 op/s 2024-05-27T07:50:03.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:03 smithi114 ceph-mon[32038]: osdmap e117: 12 total, 12 up, 12 in 2024-05-27T07:50:03.725 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:03 smithi163 ceph-mon[36451]: pgmap v338: 129 pgs: 129 active+clean; 102 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 283 op/s 2024-05-27T07:50:03.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:03 smithi163 ceph-mon[36451]: osdmap e117: 12 total, 12 up, 12 in 2024-05-27T07:50:03.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:03 smithi186 ceph-mon[36647]: pgmap v338: 129 pgs: 129 active+clean; 102 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 283 op/s 2024-05-27T07:50:03.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:03 smithi186 ceph-mon[36647]: osdmap e117: 12 total, 12 up, 12 in 2024-05-27T07:50:04.157 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:50:04.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:50:04.282 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1027163 10.38 MB/sec execute 376 sec latency 47.075 ms 2024-05-27T07:50:04.678 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:50:04.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:04 smithi163 ceph-mon[36451]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-05-27T07:50:04.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:04 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 7 up:standby 2024-05-27T07:50:04.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:04 smithi186 ceph-mon[36647]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-05-27T07:50:04.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:04 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 7 up:standby 2024-05-27T07:50:04.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:04 smithi114 ceph-mon[32038]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-05-27T07:50:04.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:04 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 7 up:standby 2024-05-27T07:50:05.282 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1027489 10.35 MB/sec execute 377 sec latency 84.217 ms 2024-05-27T07:50:05.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:05 smithi163 ceph-mon[36451]: pgmap v340: 129 pgs: 129 active+clean; 102 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 235 op/s 2024-05-27T07:50:05.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-mon[36647]: pgmap v340: 129 pgs: 129 active+clean; 102 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 235 op/s 2024-05-27T07:50:05.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4bc59b5700 time 2024-05-27T07:50:05.407818+0000 2024-05-27T07:50:05.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:50:05.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:05.730 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4bd81faba6] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4bd81fadd1] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5617021e123d] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5617021cc971] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 5: (SnapRealm::check_cache() const+0x13e) [0x5617021cafa0] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5617021cb651] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561702160347] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561701e5c87e] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561701e61f2f] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561701e6c969] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 11: (MDLog::_replay_thread()+0x1f38) [0x56170220548c] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56170220e9cb] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 13: (Thread::entry_wrapper()+0x43) [0x7f4bd81c972d] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4bd81c9749] 2024-05-27T07:50:05.731 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4bd6d6d1ca] 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 16: clone() 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: *** Caught signal (Aborted) ** 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: in thread 7f4bc59b5700 thread_name:md_log_replay 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.412+0000 7f4bc59b5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4bc59b5700 time 2024-05-27T07:50:05.407818+0000 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4bd81faba6] 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4bd81fadd1] 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5617021e123d] 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5617021cc971] 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 5: (SnapRealm::check_cache() const+0x13e) [0x5617021cafa0] 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5617021cb651] 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561702160347] 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561701e5c87e] 2024-05-27T07:50:05.732 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561701e61f2f] 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561701e6c969] 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 11: (MDLog::_replay_thread()+0x1f38) [0x56170220548c] 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56170220e9cb] 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 13: (Thread::entry_wrapper()+0x43) [0x7f4bd81c972d] 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4bd81c9749] 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4bd6d6d1ca] 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 16: clone() 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4bd6d77d20] 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2: gsignal() 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 3: abort() 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4bd81faced] 2024-05-27T07:50:05.733 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4bd81fadd1] 2024-05-27T07:50:05.734 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5617021e123d] 2024-05-27T07:50:05.734 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5617021cc971] 2024-05-27T07:50:05.734 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 8: (SnapRealm::check_cache() const+0x13e) [0x5617021cafa0] 2024-05-27T07:50:05.734 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5617021cb651] 2024-05-27T07:50:05.734 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561702160347] 2024-05-27T07:50:05.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561701e5c87e] 2024-05-27T07:50:05.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561701e61f2f] 2024-05-27T07:50:05.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561701e6c969] 2024-05-27T07:50:05.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 14: (MDLog::_replay_thread()+0x1f38) [0x56170220548c] 2024-05-27T07:50:05.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56170220e9cb] 2024-05-27T07:50:05.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 16: (Thread::entry_wrapper()+0x43) [0x7f4bd81c972d] 2024-05-27T07:50:05.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4bd81c9749] 2024-05-27T07:50:05.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4bd6d6d1ca] 2024-05-27T07:50:05.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 19: clone() 2024-05-27T07:50:05.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.415+0000 7f4bc59b5700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:50:05.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: in thread 7f4bc59b5700 thread_name:md_log_replay 2024-05-27T07:50:05.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4bd6d77d20] 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2: gsignal() 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 3: abort() 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4bd81faced] 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4bd81fadd1] 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5617021e123d] 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5617021cc971] 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 8: (SnapRealm::check_cache() const+0x13e) [0x5617021cafa0] 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5617021cb651] 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561702160347] 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561701e5c87e] 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561701e61f2f] 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561701e6c969] 2024-05-27T07:50:05.736 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 14: (MDLog::_replay_thread()+0x1f38) [0x56170220548c] 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56170220e9cb] 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 16: (Thread::entry_wrapper()+0x43) [0x7f4bd81c972d] 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4bd81c9749] 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4bd6d6d1ca] 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 19: clone() 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: -1> 2024-05-27T07:50:05.412+0000 7f4bc59b5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4bc59b5700 time 2024-05-27T07:50:05.407818+0000 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4bd81faba6] 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4bd81fadd1] 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5617021e123d] 2024-05-27T07:50:05.737 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5617021cc971] 2024-05-27T07:50:05.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 5: (SnapRealm::check_cache() const+0x13e) [0x5617021cafa0] 2024-05-27T07:50:05.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5617021cb651] 2024-05-27T07:50:05.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561702160347] 2024-05-27T07:50:05.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561701e5c87e] 2024-05-27T07:50:05.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561701e61f2f] 2024-05-27T07:50:05.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561701e6c969] 2024-05-27T07:50:05.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 11: (MDLog::_replay_thread()+0x1f38) [0x56170220548c] 2024-05-27T07:50:05.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56170220e9cb] 2024-05-27T07:50:05.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 13: (Thread::entry_wrapper()+0x43) [0x7f4bd81c972d] 2024-05-27T07:50:05.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4bd81c9749] 2024-05-27T07:50:05.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4bd6d6d1ca] 2024-05-27T07:50:05.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 16: clone() 2024-05-27T07:50:05.738 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 0> 2024-05-27T07:50:05.415+0000 7f4bc59b5700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: in thread 7f4bc59b5700 thread_name:md_log_replay 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4bd6d77d20] 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2: gsignal() 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 3: abort() 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4bd81faced] 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4bd81fadd1] 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5617021e123d] 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5617021cc971] 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 8: (SnapRealm::check_cache() const+0x13e) [0x5617021cafa0] 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5617021cb651] 2024-05-27T07:50:05.739 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561702160347] 2024-05-27T07:50:05.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561701e5c87e] 2024-05-27T07:50:05.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561701e61f2f] 2024-05-27T07:50:05.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561701e6c969] 2024-05-27T07:50:05.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 14: (MDLog::_replay_thread()+0x1f38) [0x56170220548c] 2024-05-27T07:50:05.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56170220e9cb] 2024-05-27T07:50:05.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 16: (Thread::entry_wrapper()+0x43) [0x7f4bd81c972d] 2024-05-27T07:50:05.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4bd81c9749] 2024-05-27T07:50:05.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4bd6d6d1ca] 2024-05-27T07:50:05.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 19: clone() 2024-05-27T07:50:05.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:50:05.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.740 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: -1> 2024-05-27T07:50:05.412+0000 7f4bc59b5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4bc59b5700 time 2024-05-27T07:50:05.407818+0000 2024-05-27T07:50:05.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:50:05.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:05.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4bd81faba6] 2024-05-27T07:50:05.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4bd81fadd1] 2024-05-27T07:50:05.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5617021e123d] 2024-05-27T07:50:05.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5617021cc971] 2024-05-27T07:50:05.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 5: (SnapRealm::check_cache() const+0x13e) [0x5617021cafa0] 2024-05-27T07:50:05.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5617021cb651] 2024-05-27T07:50:05.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561702160347] 2024-05-27T07:50:05.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561701e5c87e] 2024-05-27T07:50:05.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561701e61f2f] 2024-05-27T07:50:05.741 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561701e6c969] 2024-05-27T07:50:05.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 11: (MDLog::_replay_thread()+0x1f38) [0x56170220548c] 2024-05-27T07:50:05.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56170220e9cb] 2024-05-27T07:50:05.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 13: (Thread::entry_wrapper()+0x43) [0x7f4bd81c972d] 2024-05-27T07:50:05.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4bd81c9749] 2024-05-27T07:50:05.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4bd6d6d1ca] 2024-05-27T07:50:05.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 16: clone() 2024-05-27T07:50:05.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 0> 2024-05-27T07:50:05.415+0000 7f4bc59b5700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:50:05.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: in thread 7f4bc59b5700 thread_name:md_log_replay 2024-05-27T07:50:05.742 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4bd6d77d20] 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2: gsignal() 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 3: abort() 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4bd81faced] 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4bd81fadd1] 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5617021e123d] 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5617021cc971] 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 8: (SnapRealm::check_cache() const+0x13e) [0x5617021cafa0] 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5617021cb651] 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561702160347] 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561701e5c87e] 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561701e61f2f] 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561701e6c969] 2024-05-27T07:50:05.743 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 14: (MDLog::_replay_thread()+0x1f38) [0x56170220548c] 2024-05-27T07:50:05.744 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56170220e9cb] 2024-05-27T07:50:05.744 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 16: (Thread::entry_wrapper()+0x43) [0x7f4bd81c972d] 2024-05-27T07:50:05.744 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4bd81c9749] 2024-05-27T07:50:05.744 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4bd6d6d1ca] 2024-05-27T07:50:05.744 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 19: clone() 2024-05-27T07:50:05.744 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:50:05.744 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[73943]: 2024-05-27T07:50:05.744 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:05 smithi186 podman[74118]: 2024-05-27 07:50:05.586931322 +0000 UTC m=+0.124586157 container died ec06090fac6668da667e87cf03bd2e0b836783c0443d1cd7bfec43477c749b05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-05-27T07:50:05.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:05 smithi114 ceph-mon[32038]: pgmap v340: 129 pgs: 129 active+clean; 102 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 235 op/s 2024-05-27T07:50:06.282 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1027809 10.33 MB/sec execute 378 sec latency 166.479 ms 2024-05-27T07:50:07.229 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:06 smithi186 podman[74118]: 2024-05-27 07:50:06.952478601 +0000 UTC m=+1.490133401 container remove ec06090fac6668da667e87cf03bd2e0b836783c0443d1cd7bfec43477c749b05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:50:07.229 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:06 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:50:07.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1028467 10.31 MB/sec execute 379 sec latency 148.583 ms 2024-05-27T07:50:07.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:07 smithi163 ceph-mon[36451]: pgmap v341: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 81 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.7 MiB/s wr, 300 op/s 2024-05-27T07:50:07.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:07 smithi186 ceph-mon[36647]: pgmap v341: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 81 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.7 MiB/s wr, 300 op/s 2024-05-27T07:50:07.729 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:07 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:50:07.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:07 smithi114 ceph-mon[32038]: pgmap v341: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 81 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.7 MiB/s wr, 300 op/s 2024-05-27T07:50:07.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.875+0000 7f1e59470700 1 -- 172.21.15.114:0/3412305420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e54105270 msgr2=0x7f1e541027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:07.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.876+0000 7f1e59470700 1 --2- 172.21.15.114:0/3412305420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e54105270 0x7f1e541027d0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f1e44009990 tx=0x7f1e44009ca0 comp rx=0 tx=0).stop 2024-05-27T07:50:07.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.878+0000 7f1e59470700 1 -- 172.21.15.114:0/3412305420 shutdown_connections 2024-05-27T07:50:07.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.878+0000 7f1e59470700 1 --2- 172.21.15.114:0/3412305420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e54110f50 0x7f1e54113370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:07.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.878+0000 7f1e59470700 1 --2- 172.21.15.114:0/3412305420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e54105270 0x7f1e541027d0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:07.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.878+0000 7f1e59470700 1 --2- 172.21.15.114:0/3412305420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e54102d10 0x7f1e54103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:07.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.878+0000 7f1e59470700 1 -- 172.21.15.114:0/3412305420 >> 172.21.15.114:0/3412305420 conn(0x7f1e540fe6e0 msgr2=0x7f1e54100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:07.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.878+0000 7f1e59470700 1 -- 172.21.15.114:0/3412305420 shutdown_connections 2024-05-27T07:50:07.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.878+0000 7f1e59470700 1 -- 172.21.15.114:0/3412305420 wait complete. 2024-05-27T07:50:07.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.880+0000 7f1e59470700 1 Processor -- start 2024-05-27T07:50:07.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.880+0000 7f1e59470700 1 -- start start 2024-05-27T07:50:07.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.881+0000 7f1e59470700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e54102d10 0x7f1e5419d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:07.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.881+0000 7f1e59470700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e54105270 0x7f1e5419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:07.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.881+0000 7f1e59470700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e54110f50 0x7f1e541a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:07.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.881+0000 7f1e52ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e54102d10 0x7f1e5419d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:07.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.881+0000 7f1e52ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e54102d10 0x7f1e5419d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:57874/0 (socket says 172.21.15.114:57874) 2024-05-27T07:50:07.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.881+0000 7f1e52ffd700 1 -- 172.21.15.114:0/1338673419 learned_addr learned my addr 172.21.15.114:0/1338673419 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:07.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.881+0000 7f1e59470700 1 -- 172.21.15.114:0/1338673419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1e541167c0 con 0x7f1e54110f50 2024-05-27T07:50:07.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.881+0000 7f1e59470700 1 -- 172.21.15.114:0/1338673419 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1e54116640 con 0x7f1e54102d10 2024-05-27T07:50:07.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.881+0000 7f1e59470700 1 -- 172.21.15.114:0/1338673419 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1e54116940 con 0x7f1e54105270 2024-05-27T07:50:07.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.881+0000 7f1e527fc700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e54105270 0x7f1e5419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:07.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.881+0000 7f1e537fe700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e54110f50 0x7f1e541a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:07.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.882+0000 7f1e537fe700 1 -- 172.21.15.114:0/1338673419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e54105270 msgr2=0x7f1e5419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:07.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.882+0000 7f1e537fe700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e54105270 0x7f1e5419d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:07.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.882+0000 7f1e537fe700 1 -- 172.21.15.114:0/1338673419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e54102d10 msgr2=0x7f1e5419d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:07.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.882+0000 7f1e537fe700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e54102d10 0x7f1e5419d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:07.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.882+0000 7f1e537fe700 1 -- 172.21.15.114:0/1338673419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e541a1f30 con 0x7f1e54110f50 2024-05-27T07:50:07.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.882+0000 7f1e537fe700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e54110f50 0x7f1e541a1890 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c00be20 tx=0x7f1e4c009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:07.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.883+0000 7f1e3bfff700 1 -- 172.21.15.114:0/1338673419 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c019070 con 0x7f1e54110f50 2024-05-27T07:50:07.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.883+0000 7f1e3bfff700 1 -- 172.21.15.114:0/1338673419 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e4c00cc10 con 0x7f1e54110f50 2024-05-27T07:50:07.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.883+0000 7f1e3bfff700 1 -- 172.21.15.114:0/1338673419 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c0125a0 con 0x7f1e54110f50 2024-05-27T07:50:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.885+0000 7f1e59470700 1 -- 172.21.15.114:0/1338673419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e541a2220 con 0x7f1e54110f50 2024-05-27T07:50:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.885+0000 7f1e59470700 1 -- 172.21.15.114:0/1338673419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e541a9b40 con 0x7f1e54110f50 2024-05-27T07:50:07.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.890+0000 7f1e3bfff700 1 -- 172.21.15.114:0/1338673419 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f1e4c014440 con 0x7f1e54110f50 2024-05-27T07:50:07.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.890+0000 7f1e59470700 1 -- 172.21.15.114:0/1338673419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e541039f0 con 0x7f1e54110f50 2024-05-27T07:50:07.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.891+0000 7f1e3bfff700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1e3c0b2b20 0x7f1e3c0b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:07.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.891+0000 7f1e52ffd700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1e3c0b2b20 0x7f1e3c0b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:07.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.892+0000 7f1e3bfff700 1 -- 172.21.15.114:0/1338673419 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7f1e4c0e1ea0 con 0x7f1e54110f50 2024-05-27T07:50:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.895+0000 7f1e52ffd700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1e3c0b2b20 0x7f1e3c0b5030 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1e44003800 tx=0x7f1e44005fb0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:07.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:07.900+0000 7f1e3bfff700 1 -- 172.21.15.114:0/1338673419 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1e4c0a94d0 con 0x7f1e54110f50 2024-05-27T07:50:08.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1029453 10.30 MB/sec execute 380 sec latency 117.232 ms 2024-05-27T07:50:08.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.406+0000 7f1e59470700 1 -- 172.21.15.114:0/1338673419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e541a9d70 con 0x7f1e54110f50 2024-05-27T07:50:08.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.409+0000 7f1e3bfff700 1 -- 172.21.15.114:0/1338673419 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+9349 (secure 0 0 0) 0x7f1e4c0a8dd0 con 0x7f1e54110f50 2024-05-27T07:50:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:50:08.410 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1801956554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1801956554},{"type":"v1","addr":"172.21.15.186:6833","nonce":1801956554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":58,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:50:04.304854+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26882":{"gid":26882,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/2622386718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2622386718},{"type":"v1","addr":"172.21.15.186:6835","nonce":2622386718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:50:08.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.414+0000 7f1e59470700 1 -- 172.21.15.114:0/1338673419 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1e3c0b2b20 msgr2=0x7f1e3c0b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:08.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.414+0000 7f1e59470700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1e3c0b2b20 0x7f1e3c0b5030 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1e44003800 tx=0x7f1e44005fb0 comp rx=0 tx=0).stop 2024-05-27T07:50:08.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.414+0000 7f1e59470700 1 -- 172.21.15.114:0/1338673419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e54110f50 msgr2=0x7f1e541a1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:08.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.414+0000 7f1e59470700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e54110f50 0x7f1e541a1890 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c00be20 tx=0x7f1e4c009ed0 comp rx=0 tx=0).stop 2024-05-27T07:50:08.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.415+0000 7f1e59470700 1 -- 172.21.15.114:0/1338673419 shutdown_connections 2024-05-27T07:50:08.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.415+0000 7f1e59470700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e54110f50 0x7f1e541a1890 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:08.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.415+0000 7f1e59470700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1e3c0b2b20 0x7f1e3c0b5030 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:08.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.415+0000 7f1e59470700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e54102d10 0x7f1e5419d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:08.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.415+0000 7f1e59470700 1 --2- 172.21.15.114:0/1338673419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e54105270 0x7f1e5419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:08.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.415+0000 7f1e59470700 1 -- 172.21.15.114:0/1338673419 >> 172.21.15.114:0/1338673419 conn(0x7f1e540fe6e0 msgr2=0x7f1e54100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:08.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.415+0000 7f1e59470700 1 -- 172.21.15.114:0/1338673419 shutdown_connections 2024-05-27T07:50:08.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:08.415+0000 7f1e59470700 1 -- 172.21.15.114:0/1338673419 wait complete. 2024-05-27T07:50:08.419 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 58 2024-05-27T07:50:08.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:08 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1338673419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:50:08.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:08 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1338673419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:50:08.763 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:08 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1338673419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:50:09.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:50:09.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1030326 10.29 MB/sec execute 381 sec latency 45.179 ms 2024-05-27T07:50:09.465 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:09 smithi114 ceph-mon[32038]: pgmap v342: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 81 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 1.7 MiB/s wr, 300 op/s 2024-05-27T07:50:09.580 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:50:09.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:09 smithi163 ceph-mon[36451]: pgmap v342: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 81 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 1.7 MiB/s wr, 300 op/s 2024-05-27T07:50:09.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:09 smithi186 ceph-mon[36647]: pgmap v342: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 81 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 1.7 MiB/s wr, 300 op/s 2024-05-27T07:50:10.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1030893 10.27 MB/sec execute 382 sec latency 25.604 ms 2024-05-27T07:50:11.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1031641 10.25 MB/sec execute 383 sec latency 221.484 ms 2024-05-27T07:50:11.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:11 smithi163 ceph-mon[36451]: pgmap v343: 129 pgs: 12 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 76 active+clean; 75 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 1.5 MiB/s wr, 287 op/s 2024-05-27T07:50:11.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:11 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:50:11.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:11 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:11.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:11 smithi186 ceph-mon[36647]: pgmap v343: 129 pgs: 12 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 76 active+clean; 75 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 1.5 MiB/s wr, 287 op/s 2024-05-27T07:50:11.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:11 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:50:11.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:11 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:11.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:11 smithi114 ceph-mon[32038]: pgmap v343: 129 pgs: 12 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 76 active+clean; 75 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 1.5 MiB/s wr, 287 op/s 2024-05-27T07:50:11.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:11 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:50:11.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:11 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1033720 10.26 MB/sec execute 384 sec latency 46.680 ms 2024-05-27T07:50:12.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:12 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:12 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:12 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:12 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:12 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:12 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:12 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:12 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:12 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:12 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:12 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:12 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:12 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:12 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:12.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:12 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:13.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.194+0000 7fb10c772700 1 -- 172.21.15.114:0/3868865007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb104107e20 msgr2=0x7fb104108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:13.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.194+0000 7fb10c772700 1 --2- 172.21.15.114:0/3868865007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb104107e20 0x7fb104108230 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fb1000099c0 tx=0x7fb100009cd0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.194+0000 7fb10c772700 1 -- 172.21.15.114:0/3868865007 shutdown_connections 2024-05-27T07:50:13.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.194+0000 7fb10c772700 1 --2- 172.21.15.114:0/3868865007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb104108770 0x7fb1041100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.194+0000 7fb10c772700 1 --2- 172.21.15.114:0/3868865007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb104103540 0x7fb104103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.194+0000 7fb10c772700 1 --2- 172.21.15.114:0/3868865007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb104107e20 0x7fb104108230 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.194+0000 7fb10c772700 1 -- 172.21.15.114:0/3868865007 >> 172.21.15.114:0/3868865007 conn(0x7fb104078df0 msgr2=0x7fb104079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:13.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.195+0000 7fb10c772700 1 -- 172.21.15.114:0/3868865007 shutdown_connections 2024-05-27T07:50:13.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.195+0000 7fb10c772700 1 -- 172.21.15.114:0/3868865007 wait complete. 2024-05-27T07:50:13.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.195+0000 7fb10c772700 1 Processor -- start 2024-05-27T07:50:13.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb10c772700 1 -- start start 2024-05-27T07:50:13.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb10c772700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb104103540 0x7fb10419bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb10c772700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb104107e20 0x7fb10419c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb10c772700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb104108770 0x7fb10419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb10c772700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb104113380 con 0x7fb104108770 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb10c772700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb104113200 con 0x7fb104107e20 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb10c772700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb104113500 con 0x7fb104103540 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb10ad0f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb104108770 0x7fb10419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb10ad0f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb104108770 0x7fb10419c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48316/0 (socket says 172.21.15.114:48316) 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb10ad0f700 1 -- 172.21.15.114:0/259637639 learned_addr learned my addr 172.21.15.114:0/259637639 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb109d0d700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb104107e20 0x7fb10419c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb10a50e700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb104103540 0x7fb10419bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb109d0d700 1 -- 172.21.15.114:0/259637639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb104103540 msgr2=0x7fb10419bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb109d0d700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb104103540 0x7fb10419bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb109d0d700 1 -- 172.21.15.114:0/259637639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb104108770 msgr2=0x7fb10419c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb109d0d700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb104108770 0x7fb10419c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb109d0d700 1 -- 172.21.15.114:0/259637639 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb104199b10 con 0x7fb104107e20 2024-05-27T07:50:13.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.196+0000 7fb10ad0f700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb104108770 0x7fb10419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:13.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.197+0000 7fb109d0d700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb104107e20 0x7fb10419c260 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb1000098e0 tx=0x7fb100004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:13.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.197+0000 7fb0fb7fe700 1 -- 172.21.15.114:0/259637639 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb100026070 con 0x7fb104107e20 2024-05-27T07:50:13.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.197+0000 7fb10c772700 1 -- 172.21.15.114:0/259637639 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb104199da0 con 0x7fb104107e20 2024-05-27T07:50:13.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.197+0000 7fb0fb7fe700 1 -- 172.21.15.114:0/259637639 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb100003c70 con 0x7fb104107e20 2024-05-27T07:50:13.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.197+0000 7fb0fb7fe700 1 -- 172.21.15.114:0/259637639 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10000f9c0 con 0x7fb104107e20 2024-05-27T07:50:13.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.197+0000 7fb10c772700 1 -- 172.21.15.114:0/259637639 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb10419a280 con 0x7fb104107e20 2024-05-27T07:50:13.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.200+0000 7fb0fb7fe700 1 -- 172.21.15.114:0/259637639 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fb100029070 con 0x7fb104107e20 2024-05-27T07:50:13.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.200+0000 7fb0fb7fe700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0f00b29f0 0x7fb0f00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.200+0000 7fb10a50e700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0f00b29f0 0x7fb0f00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.201+0000 7fb0fb7fe700 1 -- 172.21.15.114:0/259637639 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7fb1000e3210 con 0x7fb104107e20 2024-05-27T07:50:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.201+0000 7fb11b187b80 1 Processor -- start 2024-05-27T07:50:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.201+0000 7fb10a50e700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0f00b29f0 0x7fb0f00b4f00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009c60 tx=0x7fb0f4009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.201+0000 7fb11b187b80 1 -- start start 2024-05-27T07:50:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.201+0000 7fb11b187b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 0x55ecfdfd5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.201+0000 7fb11b187b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfcfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.201+0000 7fb11b187b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.201+0000 7fb11b187b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ecfdefec90 con 0x55ecfde51e40 2024-05-27T07:50:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.201+0000 7fb11b187b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ecfdeee830 con 0x55ecfdfcff80 2024-05-27T07:50:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.201+0000 7fb11b187b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ecfdef2870 con 0x55ecfdfd5c50 2024-05-27T07:50:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.201+0000 7fb10a50e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 0x55ecfdfd5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.201+0000 7fb10a50e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 0x55ecfdfd5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48332/0 (socket says 172.21.15.114:48332) 2024-05-27T07:50:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.202+0000 7fb10a50e700 1 -- 172.21.15.114:0/3466049550 learned_addr learned my addr 172.21.15.114:0/3466049550 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.202+0000 7fb109d0d700 1 --2- 172.21.15.114:0/3466049550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfcfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.202+0000 7fb10ad0f700 1 --2- 172.21.15.114:0/3466049550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.202+0000 7fb10a50e700 1 -- 172.21.15.114:0/3466049550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfdfd5c50 msgr2=0x55ecfdfcfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.202+0000 7fb10a50e700 1 --2- 172.21.15.114:0/3466049550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfcfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.202+0000 7fb10a50e700 1 -- 172.21.15.114:0/3466049550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 msgr2=0x55ecfdfd0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.202+0000 7fb10a50e700 1 --2- 172.21.15.114:0/3466049550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.202+0000 7fb10a50e700 1 -- 172.21.15.114:0/3466049550 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecfde3e8d0 con 0x55ecfde51e40 2024-05-27T07:50:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.202+0000 7fb10a50e700 1 --2- 172.21.15.114:0/3466049550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 0x55ecfdfd5710 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400e4e0 tx=0x7fb0f4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c44e175cb65d6a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.202+0000 7fb0f8ff9700 1 -- 172.21.15.114:0/3466049550 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f400ee00 con 0x55ecfde51e40 2024-05-27T07:50:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.202+0000 7fb0f8ff9700 1 -- 172.21.15.114:0/3466049550 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0f401c8c0 con 0x55ecfde51e40 2024-05-27T07:50:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.202+0000 7fb0f8ff9700 1 -- 172.21.15.114:0/3466049550 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f401da10 con 0x55ecfde51e40 2024-05-27T07:50:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.203+0000 7fb11b187b80 1 -- 172.21.15.114:0/3466049550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 msgr2=0x55ecfdfd5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.203+0000 7fb11b187b80 1 --2- 172.21.15.114:0/3466049550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 0x55ecfdfd5710 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400e4e0 tx=0x7fb0f4014000 comp rx=0 tx=0).stop 2024-05-27T07:50:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.203+0000 7fb11b187b80 1 -- 172.21.15.114:0/3466049550 shutdown_connections 2024-05-27T07:50:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.203+0000 7fb11b187b80 1 --2- 172.21.15.114:0/3466049550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.203+0000 7fb11b187b80 1 --2- 172.21.15.114:0/3466049550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 0x55ecfdfd5710 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.203+0000 7fb11b187b80 1 --2- 172.21.15.114:0/3466049550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfcfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.203+0000 7fb11b187b80 1 -- 172.21.15.114:0/3466049550 >> 172.21.15.114:0/3466049550 conn(0x55ecfdeacaf0 msgr2=0x55ecfdfca640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.203+0000 7fb11b187b80 1 -- 172.21.15.114:0/3466049550 shutdown_connections 2024-05-27T07:50:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.203+0000 7fb11b187b80 1 -- 172.21.15.114:0/3466049550 wait complete. 2024-05-27T07:50:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb11b187b80 1 -- start start 2024-05-27T07:50:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb11b187b80 1 Processor -- start 2024-05-27T07:50:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb11b187b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfde51e40 0x55ecfdfd9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb11b187b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb11b187b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfd9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb11b187b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ecfddf4fa0 con 0x55ecfdfd5c50 2024-05-27T07:50:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb11b187b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ecfde907c0 con 0x55ecfdfcff80 2024-05-27T07:50:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb11b187b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ecfdfdc740 con 0x55ecfde51e40 2024-05-27T07:50:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb11b187b80 10 client.0 _ll_register_callbacks cb 0x55ecfde89920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb10ad0f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfd9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb109d0d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb109d0d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:57936/0 (socket says 172.21.15.114:57936) 2024-05-27T07:50:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb10ad0f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfd9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48342/0 (socket says 172.21.15.114:48342) 2024-05-27T07:50:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb109d0d700 1 -- 172.21.15.114:0/4089745314 learned_addr learned my addr 172.21.15.114:0/4089745314 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.204+0000 7fb10a50e700 1 --2- 172.21.15.114:0/4089745314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfde51e40 0x55ecfdfd9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb109d0d700 1 -- 172.21.15.114:0/4089745314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfde51e40 msgr2=0x55ecfdfd9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb109d0d700 1 --2- 172.21.15.114:0/4089745314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfde51e40 0x55ecfdfd9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb109d0d700 1 -- 172.21.15.114:0/4089745314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfdfd5c50 msgr2=0x55ecfdfd9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb109d0d700 1 --2- 172.21.15.114:0/4089745314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfd9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb109d0d700 1 -- 172.21.15.114:0/4089745314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecfde8d690 con 0x55ecfdfcff80 2024-05-27T07:50:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb109d0d700 1 --2- 172.21.15.114:0/4089745314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd9890 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb1000a7390 tx=0x7fb1000a9910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1584d65b321720ad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb0edffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:50:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb0edffb700 1 -- 172.21.15.114:0/4089745314 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb100003e20 con 0x55ecfdfcff80 2024-05-27T07:50:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb11b187b80 1 -- 172.21.15.114:0/4089745314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0f401c8c0 con 0x55ecfdfcff80 2024-05-27T07:50:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb0edffb700 1 -- 172.21.15.114:0/4089745314 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7fb1000b2c70 con 0x55ecfdfcff80 2024-05-27T07:50:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb0edffb700 1 -- 172.21.15.114:0/4089745314 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fb1000aa820 con 0x55ecfdfcff80 2024-05-27T07:50:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb11b187b80 10 client.27284 fetch_fsmap learned FSMap version 58 2024-05-27T07:50:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb11b187b80 1 -- 172.21.15.114:0/4089745314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55ecfdfe71f0 con 0x55ecfdfcff80 2024-05-27T07:50:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb0edffb700 1 -- 172.21.15.114:0/4089745314 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0f401c8c0 con 0x55ecfdfcff80 2024-05-27T07:50:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.205+0000 7fb0edffb700 1 -- 172.21.15.114:0/4089745314 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 58) ==== 6515+0+0 (secure 0 0 0) 0x7fb100069bb0 con 0x55ecfdfcff80 2024-05-27T07:50:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.206+0000 7fb11b187b80 10 client.27284 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:50:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.206+0000 7fb11b187b80 10 client.27284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.206+0000 7fb11b187b80 20 client.27284 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.206+0000 7fb11b187b80 1 --2- 172.21.15.114:0/4089745314 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecfdfe9830 0x55ecfdfebc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.206+0000 7fb11b187b80 4 client.27284 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:50:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.206+0000 7fb11b187b80 1 -- 172.21.15.114:0/4089745314 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecfdf9b9f0 con 0x55ecfdfe9830 2024-05-27T07:50:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.206+0000 7fb10a50e700 1 --2- 172.21.15.114:0/4089745314 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecfdfe9830 0x55ecfdfebc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.206+0000 7fb0edffb700 10 client.27284 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.206+0000 7fb10a50e700 1 --2- 172.21.15.114:0/4089745314 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecfdfe9830 0x55ecfdfebc80 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aec910436974403 server_cookie=47386457f1a75917 in_seq=0 out_seq=0 2024-05-27T07:50:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb0edffb700 1 -- 172.21.15.114:0/4089745314 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ecfdf9b9f0 con 0x55ecfdfe9830 2024-05-27T07:50:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb0edffb700 10 client.27284 handle_command_reply: tid=0 2024-05-27T07:50:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb11b187b80 1 client.27284 shutdown 2024-05-27T07:50:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb11b187b80 1 -- 172.21.15.114:0/4089745314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 msgr2=0x55ecfdfd9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb11b187b80 1 --2- 172.21.15.114:0/4089745314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd9890 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb1000a7390 tx=0x7fb1000a9910 comp rx=0 tx=0).stop 2024-05-27T07:50:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb11b187b80 1 -- 172.21.15.114:0/4089745314 shutdown_connections 2024-05-27T07:50:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb11b187b80 1 --2- 172.21.15.114:0/4089745314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfd9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb11b187b80 1 --2- 172.21.15.114:0/4089745314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfde51e40 0x55ecfdfd9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb11b187b80 1 --2- 172.21.15.114:0/4089745314 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecfdfe9830 0x55ecfdfebc80 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb11b187b80 1 --2- 172.21.15.114:0/4089745314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd9890 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb0edffb700 0 client.27284 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb11b187b80 1 -- 172.21.15.114:0/4089745314 >> 172.21.15.114:0/4089745314 conn(0x55ecfdeacaf0 msgr2=0x55ecfdfcea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb11b187b80 1 -- 172.21.15.114:0/4089745314 shutdown_connections 2024-05-27T07:50:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.208+0000 7fb11b187b80 1 -- 172.21.15.114:0/4089745314 wait complete. 2024-05-27T07:50:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.209+0000 7fb11b187b80 20 client.27284 trim_cache size 0 max 16384 2024-05-27T07:50:13.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb11b187b80 1 Processor -- start 2024-05-27T07:50:13.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb11b187b80 1 -- start start 2024-05-27T07:50:13.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb11b187b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 0x55ecfdfcfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb11b187b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb11b187b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfe5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb11b187b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb1041b31a0 con 0x55ecfde51e40 2024-05-27T07:50:13.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb11b187b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb1041b0940 con 0x55ecfdfd5c50 2024-05-27T07:50:13.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb11b187b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb1041b2ad0 con 0x55ecfdfcff80 2024-05-27T07:50:13.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb10a50e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 0x55ecfdfcfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb10ad0f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfe5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb109d0d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb10ad0f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfe5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:57950/0 (socket says 172.21.15.114:57950) 2024-05-27T07:50:13.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb10a50e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 0x55ecfdfcfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48350/0 (socket says 172.21.15.114:48350) 2024-05-27T07:50:13.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb10ad0f700 1 -- 172.21.15.114:0/1317123604 learned_addr learned my addr 172.21.15.114:0/1317123604 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:13.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.226+0000 7fb109d0d700 1 --2- 172.21.15.114:0/1317123604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35884/0 (socket says 172.21.15.114:35884) 2024-05-27T07:50:13.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.227+0000 7fb10a50e700 1 -- 172.21.15.114:0/1317123604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfdfcff80 msgr2=0x55ecfdfd9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:13.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.227+0000 7fb10a50e700 1 --2- 172.21.15.114:0/1317123604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.227+0000 7fb10a50e700 1 -- 172.21.15.114:0/1317123604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfd5c50 msgr2=0x55ecfdfe5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:13.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.227+0000 7fb10a50e700 1 --2- 172.21.15.114:0/1317123604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfe5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.227+0000 7fb10a50e700 1 -- 172.21.15.114:0/1317123604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecfdfda150 con 0x55ecfde51e40 2024-05-27T07:50:13.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.227+0000 7fb10ad0f700 1 --2- 172.21.15.114:0/1317123604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfe5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:13.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.227+0000 7fb10a50e700 1 --2- 172.21.15.114:0/1317123604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 0x55ecfdfcfa40 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4011ab0 tx=0x7fb0f40092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ce75fab60ffe19 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:13.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.227+0000 7fb0effff700 1 -- 172.21.15.114:0/1317123604 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f401d410 con 0x55ecfde51e40 2024-05-27T07:50:13.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.227+0000 7fb0effff700 1 -- 172.21.15.114:0/1317123604 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0f400ec00 con 0x55ecfde51e40 2024-05-27T07:50:13.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.227+0000 7fb0effff700 1 -- 172.21.15.114:0/1317123604 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4003d40 con 0x55ecfde51e40 2024-05-27T07:50:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.228+0000 7fb11b187b80 1 -- 172.21.15.114:0/1317123604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 msgr2=0x55ecfdfcfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.228+0000 7fb11b187b80 1 --2- 172.21.15.114:0/1317123604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 0x55ecfdfcfa40 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4011ab0 tx=0x7fb0f40092f0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.228+0000 7fb11b187b80 1 -- 172.21.15.114:0/1317123604 shutdown_connections 2024-05-27T07:50:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.228+0000 7fb11b187b80 1 --2- 172.21.15.114:0/1317123604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfd5c50 0x55ecfdfe5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.228+0000 7fb11b187b80 1 --2- 172.21.15.114:0/1317123604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfde51e40 0x55ecfdfcfa40 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.228+0000 7fb11b187b80 1 --2- 172.21.15.114:0/1317123604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfdfcff80 0x55ecfdfd9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.228+0000 7fb11b187b80 1 -- 172.21.15.114:0/1317123604 >> 172.21.15.114:0/1317123604 conn(0x55ecfdeacaf0 msgr2=0x55ecfdfd7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.228+0000 7fb11b187b80 1 -- 172.21.15.114:0/1317123604 shutdown_connections 2024-05-27T07:50:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.228+0000 7fb11b187b80 1 -- 172.21.15.114:0/1317123604 wait complete. 2024-05-27T07:50:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb11b187b80 1 -- start start 2024-05-27T07:50:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb11b187b80 1 Processor -- start 2024-05-27T07:50:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb11b187b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfde51e40 0x55ecfdfe2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb11b187b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdff0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb11b187b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfdfd5c50 0x55ecfdff1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb11b187b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb1041b2400 con 0x55ecfdfd5c50 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb11b187b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb1041b0f90 con 0x55ecfdfcff80 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb11b187b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb1041b1660 con 0x55ecfde51e40 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb11b187b80 10 client.0 _ll_register_callbacks cb 0x55ecfde89920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb10ad0f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfdfd5c50 0x55ecfdff1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb10ad0f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfdfd5c50 0x55ecfdff1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48360/0 (socket says 172.21.15.114:48360) 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb10a50e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfde51e40 0x55ecfdfe2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb10ad0f700 1 -- 172.21.15.114:0/1320810108 learned_addr learned my addr 172.21.15.114:0/1320810108 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb109d0d700 1 --2- 172.21.15.114:0/1320810108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdff0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb10ad0f700 1 -- 172.21.15.114:0/1320810108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfde51e40 msgr2=0x55ecfdfe2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb10ad0f700 1 --2- 172.21.15.114:0/1320810108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfde51e40 0x55ecfdfe2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb10ad0f700 1 -- 172.21.15.114:0/1320810108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 msgr2=0x55ecfdff0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:13.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb10ad0f700 1 --2- 172.21.15.114:0/1320810108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdff0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.229+0000 7fb10ad0f700 1 -- 172.21.15.114:0/1320810108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecfdff16b0 con 0x55ecfdfd5c50 2024-05-27T07:50:13.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb10ad0f700 1 --2- 172.21.15.114:0/1320810108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfdfd5c50 0x55ecfdff1070 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00e770 tx=0x7fb0fc00ea80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7320be9989016335 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:13.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb0edffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:50:13.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb0edffb700 1 -- 172.21.15.114:0/1320810108 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc004bb0 con 0x55ecfdfd5c50 2024-05-27T07:50:13.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb0edffb700 1 -- 172.21.15.114:0/1320810108 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7fb0fc004d50 con 0x55ecfdfd5c50 2024-05-27T07:50:13.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb11b187b80 1 -- 172.21.15.114:0/1320810108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ecfdf00be0 con 0x55ecfdfd5c50 2024-05-27T07:50:13.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb0edffb700 1 -- 172.21.15.114:0/1320810108 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fb0fc017620 con 0x55ecfdfd5c50 2024-05-27T07:50:13.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb11b187b80 10 client.17841 fetch_fsmap learned FSMap version 58 2024-05-27T07:50:13.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb11b187b80 1 -- 172.21.15.114:0/1320810108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55ecfe054140 con 0x55ecfdfd5c50 2024-05-27T07:50:13.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb0edffb700 1 -- 172.21.15.114:0/1320810108 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ecfdf00be0 con 0x55ecfdfd5c50 2024-05-27T07:50:13.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb0edffb700 1 -- 172.21.15.114:0/1320810108 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 58) ==== 6515+0+0 (secure 0 0 0) 0x7fb0fc020980 con 0x55ecfdfd5c50 2024-05-27T07:50:13.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb11b187b80 10 client.17841 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb11b187b80 10 client.17841 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb11b187b80 20 client.17841 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb11b187b80 1 --2- 172.21.15.114:0/1320810108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecfe0548d0 0x55ecfe056cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb11b187b80 4 client.17841 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.230+0000 7fb11b187b80 1 -- 172.21.15.114:0/1320810108 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ecfdefa3c0 con 0x55ecfe0548d0 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.231+0000 7fb10a50e700 1 --2- 172.21.15.114:0/1320810108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecfe0548d0 0x55ecfe056cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.231+0000 7fb0edffb700 10 client.17841 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.231+0000 7fb10a50e700 1 --2- 172.21.15.114:0/1320810108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecfe0548d0 0x55ecfe056cf0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f63c995e300cbae server_cookie=b46006544701d823 in_seq=0 out_seq=0 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.231+0000 7fb0edffb700 1 -- 172.21.15.114:0/1320810108 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ecfdefa3c0 con 0x55ecfe0548d0 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.231+0000 7fb0edffb700 10 client.17841 handle_command_reply: tid=0 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.231+0000 7fb11b187b80 1 client.17841 shutdown 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.231+0000 7fb11b187b80 1 -- 172.21.15.114:0/1320810108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfdfd5c50 msgr2=0x55ecfdff1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.231+0000 7fb11b187b80 1 --2- 172.21.15.114:0/1320810108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfdfd5c50 0x55ecfdff1070 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00e770 tx=0x7fb0fc00ea80 comp rx=0 tx=0).stop 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.231+0000 7fb11b187b80 1 -- 172.21.15.114:0/1320810108 shutdown_connections 2024-05-27T07:50:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb11b187b80 1 --2- 172.21.15.114:0/1320810108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ecfdfd5c50 0x55ecfdff1070 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb11b187b80 1 --2- 172.21.15.114:0/1320810108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecfde51e40 0x55ecfdfe2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb11b187b80 1 --2- 172.21.15.114:0/1320810108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ecfe0548d0 0x55ecfe056cf0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb11b187b80 1 --2- 172.21.15.114:0/1320810108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ecfdfcff80 0x55ecfdff0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb0edffb700 0 client.17841 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:13.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb11b187b80 1 -- 172.21.15.114:0/1320810108 >> 172.21.15.114:0/1320810108 conn(0x55ecfdeacaf0 msgr2=0x55ecfdfd4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:13.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb11b187b80 1 -- 172.21.15.114:0/1320810108 shutdown_connections 2024-05-27T07:50:13.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb11b187b80 1 -- 172.21.15.114:0/1320810108 wait complete. 2024-05-27T07:50:13.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb11b187b80 20 client.17841 trim_cache size 0 max 16384 2024-05-27T07:50:13.237 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:50:13.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb10c772700 1 -- 172.21.15.114:0/259637639 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0f00b29f0 msgr2=0x7fb0f00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:13.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb10c772700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0f00b29f0 0x7fb0f00b4f00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009c60 tx=0x7fb0f4009260 comp rx=0 tx=0).stop 2024-05-27T07:50:13.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb10c772700 1 -- 172.21.15.114:0/259637639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb104107e20 msgr2=0x7fb10419c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:13.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb10c772700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb104107e20 0x7fb10419c260 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb1000098e0 tx=0x7fb100004270 comp rx=0 tx=0).stop 2024-05-27T07:50:13.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb10c772700 1 -- 172.21.15.114:0/259637639 shutdown_connections 2024-05-27T07:50:13.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb10c772700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb104108770 0x7fb10419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb10c772700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb104103540 0x7fb10419bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb10c772700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0f00b29f0 0x7fb0f00b4f00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb10c772700 1 --2- 172.21.15.114:0/259637639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb104107e20 0x7fb10419c260 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:13.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.232+0000 7fb10c772700 1 -- 172.21.15.114:0/259637639 >> 172.21.15.114:0/259637639 conn(0x7fb104078df0 msgr2=0x7fb104100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:13.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.233+0000 7fb10c772700 1 -- 172.21.15.114:0/259637639 shutdown_connections 2024-05-27T07:50:13.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:13.233+0000 7fb10c772700 1 -- 172.21.15.114:0/259637639 wait complete. 2024-05-27T07:50:13.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1034500 10.24 MB/sec execute 385 sec latency 83.806 ms 2024-05-27T07:50:13.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:13 smithi114 ceph-mon[32038]: pgmap v344: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 71 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.3 MiB/s wr, 217 op/s 2024-05-27T07:50:13.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:13 smithi163 ceph-mon[36451]: pgmap v344: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 71 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.3 MiB/s wr, 217 op/s 2024-05-27T07:50:13.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:13 smithi186 ceph-mon[36647]: pgmap v344: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 71 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.3 MiB/s wr, 217 op/s 2024-05-27T07:50:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.243+0000 7f95e1700640 1 -- 172.21.15.114:0/1442207660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95dc10a160 msgr2=0x7f95dc103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.243+0000 7f95e1700640 1 --2- 172.21.15.114:0/1442207660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95dc10a160 0x7f95dc103da0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f95d0009900 tx=0x7f95d002a410 comp rx=0 tx=0).stop 2024-05-27T07:50:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.243+0000 7f95e1700640 1 -- 172.21.15.114:0/1442207660 shutdown_connections 2024-05-27T07:50:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.243+0000 7f95e1700640 1 --2- 172.21.15.114:0/1442207660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95dc104410 0x7f95dc1107e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.243+0000 7f95e1700640 1 --2- 172.21.15.114:0/1442207660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95dc10a160 0x7f95dc103da0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.243+0000 7f95e1700640 1 --2- 172.21.15.114:0/1442207660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95dc109cc0 0x7f95dc103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.243+0000 7f95e1700640 1 -- 172.21.15.114:0/1442207660 >> 172.21.15.114:0/1442207660 conn(0x7f95dc0793c0 msgr2=0x7f95dc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.243+0000 7f95e1700640 1 -- 172.21.15.114:0/1442207660 shutdown_connections 2024-05-27T07:50:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.243+0000 7f95e1700640 1 -- 172.21.15.114:0/1442207660 wait complete. 2024-05-27T07:50:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.244+0000 7f95e1700640 1 Processor -- start 2024-05-27T07:50:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.244+0000 7f95e1700640 1 -- start start 2024-05-27T07:50:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.244+0000 7f95e1700640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95dc104410 0x7f95dc1a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.244+0000 7f95e1700640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95dc109cc0 0x7f95dc1a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.244+0000 7f95e1700640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95dc10a160 0x7f95dc19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.244+0000 7f95e1700640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc113f00 con 0x7f95dc109cc0 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.244+0000 7f95e1700640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc113d80 con 0x7f95dc104410 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.244+0000 7f95e1700640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc114080 con 0x7f95dc10a160 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.244+0000 7f95da7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95dc109cc0 0x7f95dc1a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.244+0000 7f95da7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95dc109cc0 0x7f95dc1a0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48390/0 (socket says 172.21.15.114:48390) 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.244+0000 7f95da7fc640 1 -- 172.21.15.114:0/108215387 learned_addr learned my addr 172.21.15.114:0/108215387 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.245+0000 7f95db7fe640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95dc10a160 0x7f95dc19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.245+0000 7f95daffd640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95dc104410 0x7f95dc1a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.245+0000 7f95da7fc640 1 -- 172.21.15.114:0/108215387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95dc10a160 msgr2=0x7f95dc19a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.245+0000 7f95da7fc640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95dc10a160 0x7f95dc19a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.245+0000 7f95da7fc640 1 -- 172.21.15.114:0/108215387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95dc104410 msgr2=0x7f95dc1a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.245+0000 7f95da7fc640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95dc104410 0x7f95dc1a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.245+0000 7f95da7fc640 1 -- 172.21.15.114:0/108215387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95dc19ae70 con 0x7f95dc109cc0 2024-05-27T07:50:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.245+0000 7f95daffd640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95dc104410 0x7f95dc1a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.245+0000 7f95da7fc640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95dc109cc0 0x7f95dc1a0b00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f95d0005ab0 tx=0x7f95d0003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.245+0000 7f95c3fff640 1 -- 172.21.15.114:0/108215387 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d003f070 con 0x7f95dc109cc0 2024-05-27T07:50:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.245+0000 7f95c3fff640 1 -- 172.21.15.114:0/108215387 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95d0031b50 con 0x7f95dc109cc0 2024-05-27T07:50:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.245+0000 7f95c3fff640 1 -- 172.21.15.114:0/108215387 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d0035030 con 0x7f95dc109cc0 2024-05-27T07:50:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.246+0000 7f95e1700640 1 -- 172.21.15.114:0/108215387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95dc19b0a0 con 0x7f95dc109cc0 2024-05-27T07:50:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.246+0000 7f95e1700640 1 -- 172.21.15.114:0/108215387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95dc19b540 con 0x7f95dc109cc0 2024-05-27T07:50:14.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.249+0000 7f95e85bb740 1 Processor -- start 2024-05-27T07:50:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.249+0000 7f95e85bb740 1 -- start start 2024-05-27T07:50:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95e85bb740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca2758a1b0 0x55ca2792b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95e85bb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca275612f0 0x55ca2792b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95e85bb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca2792be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95e85bb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ca2789dc30 con 0x55ca275694d0 2024-05-27T07:50:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95e85bb740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ca2788d7d0 con 0x55ca2758a1b0 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95e85bb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ca27891810 con 0x55ca275612f0 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95c3fff640 1 -- 172.21.15.114:0/108215387 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f95d0042070 con 0x7f95dc109cc0 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95da7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca2792be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95db7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca2758a1b0 0x55ca2792b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95db7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca2758a1b0 0x55ca2792b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:57978/0 (socket says 172.21.15.114:57978) 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95da7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca2792be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48394/0 (socket says 172.21.15.114:48394) 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95db7fe640 1 -- 172.21.15.114:0/1570056042 learned_addr learned my addr 172.21.15.114:0/1570056042 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95daffd640 1 --2- 172.21.15.114:0/1570056042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca275612f0 0x55ca2792b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95db7fe640 1 -- 172.21.15.114:0/1570056042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca275612f0 msgr2=0x55ca2792b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95db7fe640 1 --2- 172.21.15.114:0/1570056042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca275612f0 0x55ca2792b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.250+0000 7f95db7fe640 1 -- 172.21.15.114:0/1570056042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 msgr2=0x55ca2792be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.251+0000 7f95db7fe640 1 --2- 172.21.15.114:0/1570056042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca2792be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.251+0000 7f95db7fe640 1 -- 172.21.15.114:0/1570056042 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca275a2210 con 0x55ca2758a1b0 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.251+0000 7f95da7fc640 1 --2- 172.21.15.114:0/1570056042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca2792be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.251+0000 7f95db7fe640 1 --2- 172.21.15.114:0/1570056042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca2758a1b0 0x55ca2792b3d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f95d40039f0 tx=0x7f95d4003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87df019fe44daf2a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:14.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.251+0000 7f95c3fff640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95c40b2950 0x7f95c40b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.251+0000 7f95c17fa640 1 -- 172.21.15.114:0/1570056042 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d401a470 con 0x55ca2758a1b0 2024-05-27T07:50:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.251+0000 7f95c17fa640 1 -- 172.21.15.114:0/1570056042 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95d4015070 con 0x55ca2758a1b0 2024-05-27T07:50:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.251+0000 7f95c17fa640 1 -- 172.21.15.114:0/1570056042 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d4010800 con 0x55ca2758a1b0 2024-05-27T07:50:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.251+0000 7f95daffd640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95c40b2950 0x7f95c40b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.251+0000 7f95c3fff640 1 -- 172.21.15.114:0/108215387 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7f95d01030c0 con 0x7f95dc109cc0 2024-05-27T07:50:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.253+0000 7f95e85bb740 1 -- 172.21.15.114:0/1570056042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca2758a1b0 msgr2=0x55ca2792b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.253+0000 7f95daffd640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95c40b2950 0x7f95c40b4e60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55ca2792c780 tx=0x7f95cc005f90 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.253+0000 7f95e85bb740 1 --2- 172.21.15.114:0/1570056042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca2758a1b0 0x55ca2792b3d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f95d40039f0 tx=0x7f95d4003eb0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.253+0000 7f95e85bb740 1 -- 172.21.15.114:0/1570056042 shutdown_connections 2024-05-27T07:50:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.253+0000 7f95e85bb740 1 --2- 172.21.15.114:0/1570056042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca2792be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.253+0000 7f95e85bb740 1 --2- 172.21.15.114:0/1570056042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca275612f0 0x55ca2792b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.253+0000 7f95e85bb740 1 --2- 172.21.15.114:0/1570056042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca2758a1b0 0x55ca2792b3d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.253+0000 7f95e85bb740 1 -- 172.21.15.114:0/1570056042 >> 172.21.15.114:0/1570056042 conn(0x55ca2757e300 msgr2=0x55ca27927b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.253+0000 7f95e85bb740 1 -- 172.21.15.114:0/1570056042 shutdown_connections 2024-05-27T07:50:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.253+0000 7f95e85bb740 1 -- 172.21.15.114:0/1570056042 wait complete. 2024-05-27T07:50:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.253+0000 7f95e85bb740 1 -- start start 2024-05-27T07:50:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.254+0000 7f95e85bb740 1 Processor -- start 2024-05-27T07:50:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.254+0000 7f95e85bb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca275612f0 0x55ca27937480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.254+0000 7f95e85bb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca279379c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.254+0000 7f95e85bb740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca2758a1b0 0x55ca27935900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.254+0000 7f95e85bb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ca27573790 con 0x55ca275694d0 2024-05-27T07:50:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.254+0000 7f95e85bb740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ca27849890 con 0x55ca2758a1b0 2024-05-27T07:50:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.254+0000 7f95e85bb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ca27938500 con 0x55ca275612f0 2024-05-27T07:50:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.254+0000 7f95da7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca279379c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.254+0000 7f95da7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca279379c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48406/0 (socket says 172.21.15.114:48406) 2024-05-27T07:50:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.254+0000 7f95da7fc640 1 -- 172.21.15.114:0/2556431897 learned_addr learned my addr 172.21.15.114:0/2556431897 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.254+0000 7f95daffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca275612f0 0x55ca27937480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.254+0000 7f95db7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca2758a1b0 0x55ca27935900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.255+0000 7f95e85bb740 10 client.0 _ll_register_callbacks cb 0x55ca275a1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.255+0000 7f95db7fe640 1 -- 172.21.15.114:0/2556431897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca275612f0 msgr2=0x55ca27937480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.255+0000 7f95db7fe640 1 --2- 172.21.15.114:0/2556431897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca275612f0 0x55ca27937480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.255+0000 7f95db7fe640 1 -- 172.21.15.114:0/2556431897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 msgr2=0x55ca279379c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.255+0000 7f95db7fe640 1 --2- 172.21.15.114:0/2556431897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca279379c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.255+0000 7f95db7fe640 1 -- 172.21.15.114:0/2556431897 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca2766e620 con 0x55ca2758a1b0 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.255+0000 7f95e85bb740 1 -- 172.21.15.114:0/2556431897 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95d4015070 con 0x55ca2758a1b0 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.255+0000 7f95db7fe640 1 --2- 172.21.15.114:0/2556431897 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca2758a1b0 0x55ca27935900 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f95d4006010 tx=0x7f95d4009950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d73391c1988193aa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.255+0000 7f95beff5640 10 client.27302 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.255+0000 7f95beff5640 1 -- 172.21.15.114:0/2556431897 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d4011650 con 0x55ca2758a1b0 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.255+0000 7f95beff5640 1 -- 172.21.15.114:0/2556431897 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7f95d40117f0 con 0x55ca2758a1b0 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.256+0000 7f95beff5640 1 -- 172.21.15.114:0/2556431897 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f95d4011a30 con 0x55ca2758a1b0 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.256+0000 7f95e85bb740 10 client.27302 fetch_fsmap learned FSMap version 58 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.256+0000 7f95e85bb740 1 -- 172.21.15.114:0/2556431897 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({fsmap=58}) -- 0x55ca27944770 con 0x55ca2758a1b0 2024-05-27T07:50:14.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.256+0000 7f95beff5640 1 -- 172.21.15.114:0/2556431897 <== mon.2 v2:172.21.15.163:3300/0 4 ==== fsmap(e 58) ==== 6515+0+0 (secure 0 0 0) 0x7f95d4010540 con 0x55ca2758a1b0 2024-05-27T07:50:14.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.256+0000 7f95e85bb740 10 client.27302 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:50:14.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.256+0000 7f95e85bb740 10 client.27302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:14.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.256+0000 7f95e85bb740 20 client.27302 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:14.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.256+0000 7f95e85bb740 1 --2- 172.21.15.114:0/2556431897 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca27548d80 0x55ca27947c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.256+0000 7f95e85bb740 4 client.27302 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:50:14.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.256+0000 7f95e85bb740 1 -- 172.21.15.114:0/2556431897 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca2789fb80 con 0x55ca27548d80 2024-05-27T07:50:14.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.257+0000 7f95daffd640 1 --2- 172.21.15.114:0/2556431897 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca27548d80 0x55ca27947c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.257+0000 7f95beff5640 10 client.27302 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:14.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.257+0000 7f95daffd640 1 --2- 172.21.15.114:0/2556431897 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca27548d80 0x55ca27947c30 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44965b8f4451f149 server_cookie=efc146fc353540e8 in_seq=0 out_seq=0 2024-05-27T07:50:14.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95beff5640 1 -- 172.21.15.114:0/2556431897 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ca2789fb80 con 0x55ca27548d80 2024-05-27T07:50:14.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95beff5640 10 client.27302 handle_command_reply: tid=0 2024-05-27T07:50:14.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95e85bb740 1 client.27302 shutdown 2024-05-27T07:50:14.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95e85bb740 1 -- 172.21.15.114:0/2556431897 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca2758a1b0 msgr2=0x55ca27935900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:14.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95e85bb740 1 --2- 172.21.15.114:0/2556431897 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca2758a1b0 0x55ca27935900 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f95d4006010 tx=0x7f95d4009950 comp rx=0 tx=0).stop 2024-05-27T07:50:14.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95e85bb740 1 -- 172.21.15.114:0/2556431897 shutdown_connections 2024-05-27T07:50:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95e85bb740 1 --2- 172.21.15.114:0/2556431897 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca27548d80 0x55ca27947c30 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95e85bb740 1 --2- 172.21.15.114:0/2556431897 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca2758a1b0 0x55ca27935900 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95e85bb740 1 --2- 172.21.15.114:0/2556431897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca279379c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95e85bb740 1 --2- 172.21.15.114:0/2556431897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca275612f0 0x55ca27937480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95e85bb740 1 -- 172.21.15.114:0/2556431897 >> 172.21.15.114:0/2556431897 conn(0x55ca2757e300 msgr2=0x55ca2792b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95beff5640 0 client.27302 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95e85bb740 1 -- 172.21.15.114:0/2556431897 shutdown_connections 2024-05-27T07:50:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95e85bb740 1 -- 172.21.15.114:0/2556431897 wait complete. 2024-05-27T07:50:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.259+0000 7f95e85bb740 20 client.27302 trim_cache size 0 max 16384 2024-05-27T07:50:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.278+0000 7f95e85bb740 1 Processor -- start 2024-05-27T07:50:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.278+0000 7f95e85bb740 1 -- start start 2024-05-27T07:50:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95e85bb740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca275612f0 0x55ca27937b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95e85bb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca2792b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95e85bb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca2758a1b0 0x55ca2792b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95e85bb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc1afb20 con 0x55ca275694d0 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95e85bb740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc1ad2c0 con 0x55ca275612f0 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95e85bb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc1af450 con 0x55ca2758a1b0 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95da7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca2792b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95da7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca2792b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48414/0 (socket says 172.21.15.114:48414) 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95da7fc640 1 -- 172.21.15.114:0/1960883958 learned_addr learned my addr 172.21.15.114:0/1960883958 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95db7fe640 1 --2- 172.21.15.114:0/1960883958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca2758a1b0 0x55ca2792b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95daffd640 1 --2- 172.21.15.114:0/1960883958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca275612f0 0x55ca27937b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95da7fc640 1 -- 172.21.15.114:0/1960883958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca2758a1b0 msgr2=0x55ca2792b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95da7fc640 1 --2- 172.21.15.114:0/1960883958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca2758a1b0 0x55ca2792b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95da7fc640 1 -- 172.21.15.114:0/1960883958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca275612f0 msgr2=0x55ca27937b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95da7fc640 1 --2- 172.21.15.114:0/1960883958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca275612f0 0x55ca27937b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95da7fc640 1 -- 172.21.15.114:0/1960883958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca2792c7b0 con 0x55ca275694d0 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.279+0000 7f95da7fc640 1 --2- 172.21.15.114:0/1960883958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca2792b160 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f95d00c6010 tx=0x7f95d0106b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=983f33ad59b9205d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.280+0000 7f95c0ff9640 1 -- 172.21.15.114:0/1960883958 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d0049030 con 0x55ca275694d0 2024-05-27T07:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.280+0000 7f95c0ff9640 1 -- 172.21.15.114:0/1960883958 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95d00c9aa0 con 0x55ca275694d0 2024-05-27T07:50:14.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.280+0000 7f95c0ff9640 1 -- 172.21.15.114:0/1960883958 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d00d2d40 con 0x55ca275694d0 2024-05-27T07:50:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.280+0000 7f95e85bb740 1 -- 172.21.15.114:0/1960883958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 msgr2=0x55ca2792b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.280+0000 7f95e85bb740 1 --2- 172.21.15.114:0/1960883958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca2792b160 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f95d00c6010 tx=0x7f95d0106b70 comp rx=0 tx=0).stop 2024-05-27T07:50:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95e85bb740 1 -- 172.21.15.114:0/1960883958 shutdown_connections 2024-05-27T07:50:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95e85bb740 1 --2- 172.21.15.114:0/1960883958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca2758a1b0 0x55ca2792b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95e85bb740 1 --2- 172.21.15.114:0/1960883958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275694d0 0x55ca2792b160 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95e85bb740 1 --2- 172.21.15.114:0/1960883958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca275612f0 0x55ca27937b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95e85bb740 1 -- 172.21.15.114:0/1960883958 >> 172.21.15.114:0/1960883958 conn(0x55ca2757e300 msgr2=0x55ca27937480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95e85bb740 1 -- 172.21.15.114:0/1960883958 shutdown_connections 2024-05-27T07:50:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95e85bb740 1 -- 172.21.15.114:0/1960883958 wait complete. 2024-05-27T07:50:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95e85bb740 1 -- start start 2024-05-27T07:50:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95e85bb740 1 Processor -- start 2024-05-27T07:50:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95e85bb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275612f0 0x55ca2793c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95e85bb740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca275694d0 0x55ca2793cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95e85bb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca2758a1b0 0x55ca2793d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95daffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275612f0 0x55ca2793c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95e85bb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc1aed80 con 0x55ca275612f0 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.281+0000 7f95daffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275612f0 0x55ca2793c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48424/0 (socket says 172.21.15.114:48424) 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95daffd640 1 -- 172.21.15.114:0/2572667160 learned_addr learned my addr 172.21.15.114:0/2572667160 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95da7fc640 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca275694d0 0x55ca2793cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95e85bb740 1 -- 172.21.15.114:0/2572667160 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc1ad910 con 0x55ca275694d0 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95e85bb740 1 -- 172.21.15.114:0/2572667160 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc1adfe0 con 0x55ca2758a1b0 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95db7fe640 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca2758a1b0 0x55ca2793d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95e85bb740 10 client.0 _ll_register_callbacks cb 0x7f95d0106a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95daffd640 1 -- 172.21.15.114:0/2572667160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca2758a1b0 msgr2=0x55ca2793d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95daffd640 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca2758a1b0 0x55ca2793d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95daffd640 1 -- 172.21.15.114:0/2572667160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca275694d0 msgr2=0x55ca2793cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95daffd640 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca275694d0 0x55ca2793cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95daffd640 1 -- 172.21.15.114:0/2572667160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca279ae4a0 con 0x55ca275612f0 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95da7fc640 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca275694d0 0x55ca2793cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95daffd640 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275612f0 0x55ca2793c7d0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x55ca27928ec0 tx=0x7f95cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b55a84a3be32fea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95beff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95beff5640 1 -- 172.21.15.114:0/2572667160 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc003d80 con 0x55ca275612f0 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95beff5640 1 -- 172.21.15.114:0/2572667160 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7f95cc02b380 con 0x55ca275612f0 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95e85bb740 1 -- 172.21.15.114:0/2572667160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95d00c9aa0 con 0x55ca275612f0 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95beff5640 1 -- 172.21.15.114:0/2572667160 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f95cc02b580 con 0x55ca275612f0 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95e85bb740 10 client.17859 fetch_fsmap learned FSMap version 58 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.282+0000 7f95e85bb740 1 -- 172.21.15.114:0/2572667160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55ca279af730 con 0x55ca275612f0 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.283+0000 7f95beff5640 1 -- 172.21.15.114:0/2572667160 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95cc039020 con 0x55ca275612f0 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.283+0000 7f95beff5640 1 -- 172.21.15.114:0/2572667160 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 58) ==== 6515+0+0 (secure 0 0 0) 0x7f95cc00f9c0 con 0x55ca275612f0 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.283+0000 7f95e85bb740 10 client.17859 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.283+0000 7f95e85bb740 10 client.17859 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.283+0000 7f95e85bb740 20 client.17859 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.283+0000 7f95e85bb740 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca2751e1c0 0x55ca279b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.283+0000 7f95e85bb740 4 client.17859 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:50:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.283+0000 7f95e85bb740 1 -- 172.21.15.114:0/2572667160 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ca27899360 con 0x55ca2751e1c0 2024-05-27T07:50:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.283+0000 7f95da7fc640 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca2751e1c0 0x55ca279b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.284+0000 7f95beff5640 10 client.17859 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.284+0000 7f95da7fc640 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca2751e1c0 0x55ca279b1e40 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba83285f079514f7 server_cookie=5dc5b19ac5090e97 in_seq=0 out_seq=0 2024-05-27T07:50:14.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1036349 10.23 MB/sec execute 386 sec latency 49.612 ms 2024-05-27T07:50:14.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.533+0000 7f95beff5640 1 -- 172.21.15.114:0/2572667160 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55ca27899360 con 0x55ca2751e1c0 2024-05-27T07:50:14.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.533+0000 7f95beff5640 10 client.17859 handle_command_reply: tid=0 2024-05-27T07:50:14.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.533+0000 7f95e85bb740 1 client.17859 shutdown 2024-05-27T07:50:14.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.533+0000 7f95e85bb740 1 -- 172.21.15.114:0/2572667160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275612f0 msgr2=0x55ca2793c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:14.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.533+0000 7f95e85bb740 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275612f0 0x55ca2793c7d0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x55ca27928ec0 tx=0x7f95cc0046c0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.534+0000 7f95e85bb740 1 -- 172.21.15.114:0/2572667160 shutdown_connections 2024-05-27T07:50:14.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.534+0000 7f95e85bb740 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ca2751e1c0 0x55ca279b1e40 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.534+0000 7f95e85bb740 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca2758a1b0 0x55ca2793d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.534+0000 7f95e85bb740 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ca275694d0 0x55ca2793cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.534+0000 7f95e85bb740 1 --2- 172.21.15.114:0/2572667160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ca275612f0 0x55ca2793c7d0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.534+0000 7f95e85bb740 1 -- 172.21.15.114:0/2572667160 >> 172.21.15.114:0/2572667160 conn(0x55ca2757e300 msgr2=0x55ca2792b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:14.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.534+0000 7f95beff5640 0 client.17859 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:14.536 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.534+0000 7f95e85bb740 1 -- 172.21.15.114:0/2572667160 shutdown_connections 2024-05-27T07:50:14.536 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.534+0000 7f95e85bb740 1 -- 172.21.15.114:0/2572667160 wait complete. 2024-05-27T07:50:14.536 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.534+0000 7f95e85bb740 20 client.17859 trim_cache size 0 max 16384 2024-05-27T07:50:14.536 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:50:14.536 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:50:14.536 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:50:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.535+0000 7f95e1700640 1 -- 172.21.15.114:0/108215387 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95c40b2950 msgr2=0x7f95c40b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.535+0000 7f95e1700640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95c40b2950 0x7f95c40b4e60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55ca2792c780 tx=0x7f95cc005f90 comp rx=0 tx=0).stop 2024-05-27T07:50:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.535+0000 7f95e1700640 1 -- 172.21.15.114:0/108215387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95dc109cc0 msgr2=0x7f95dc1a0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.535+0000 7f95e1700640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95dc109cc0 0x7f95dc1a0b00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f95d0005ab0 tx=0x7f95d0003b90 comp rx=0 tx=0).stop 2024-05-27T07:50:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.535+0000 7f95e1700640 1 -- 172.21.15.114:0/108215387 shutdown_connections 2024-05-27T07:50:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.535+0000 7f95e1700640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f95c40b2950 0x7f95c40b4e60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.535+0000 7f95e1700640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95dc10a160 0x7f95dc19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.535+0000 7f95e1700640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95dc109cc0 0x7f95dc1a0b00 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.535+0000 7f95e1700640 1 --2- 172.21.15.114:0/108215387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95dc104410 0x7f95dc1a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:14.538 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.535+0000 7f95e1700640 1 -- 172.21.15.114:0/108215387 >> 172.21.15.114:0/108215387 conn(0x7f95dc0793c0 msgr2=0x7f95dc1023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:14.538 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.535+0000 7f95e1700640 1 -- 172.21.15.114:0/108215387 shutdown_connections 2024-05-27T07:50:14.538 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:14.535+0000 7f95e1700640 1 -- 172.21.15.114:0/108215387 wait complete. 2024-05-27T07:50:14.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:14 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:14.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:14 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:14.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:14 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:14.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:14 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:14.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:14 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:14.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:14 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:14.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:14 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:14.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:14 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:14.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:14 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:14.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:14 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:14.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:14 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:14.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:14 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:14.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:14 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:14.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:14 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:14.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:14 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:14.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:14 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:14.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:14 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:14 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:14 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:14 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:14 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:14 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:14 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:14 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:15.066 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:50:15.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:50:15.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1037755 10.23 MB/sec execute 387 sec latency 129.241 ms 2024-05-27T07:50:15.378 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:50:15.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:15 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-27T07:50:15.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:15 smithi114 ceph-mon[32038]: Unable to set osd_memory_target on smithi163 to 184781004: error parsing value: Value '184781004' is below minimum 939524096 2024-05-27T07:50:15.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:15 smithi114 ceph-mon[32038]: pgmap v345: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 71 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-05-27T07:50:15.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:15 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:15.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:15 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:15.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:15 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:15.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:15 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:15.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:15 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-27T07:50:15.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:15 smithi114 ceph-mon[32038]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T07:50:15.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:15 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:15.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:15 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:15.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:15 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-27T07:50:15.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:15 smithi163 ceph-mon[36451]: Unable to set osd_memory_target on smithi163 to 184781004: error parsing value: Value '184781004' is below minimum 939524096 2024-05-27T07:50:15.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:15 smithi163 ceph-mon[36451]: pgmap v345: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 71 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-05-27T07:50:15.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:15 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:15.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:15 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:15.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:15 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:15.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:15 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:15.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:15 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-27T07:50:15.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:15 smithi163 ceph-mon[36451]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T07:50:15.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:15 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:15.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:15 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:15.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:15 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-27T07:50:15.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:15 smithi186 ceph-mon[36647]: Unable to set osd_memory_target on smithi163 to 184781004: error parsing value: Value '184781004' is below minimum 939524096 2024-05-27T07:50:15.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:15 smithi186 ceph-mon[36647]: pgmap v345: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 71 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-05-27T07:50:15.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:15 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:15.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:15 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:15.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:15 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:15.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:15 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:15.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:15 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-27T07:50:15.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:15 smithi186 ceph-mon[36647]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T07:50:15.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:15 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:15.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:15 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:16.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1038883 10.22 MB/sec execute 388 sec latency 64.271 ms 2024-05-27T07:50:16.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:16 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:16.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:16 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:16.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:16 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:16.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:16 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:16.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:16 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:50:16.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:16 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:50:16.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:16 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:16.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:16 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:16.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:16 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:16.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:16 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:16.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:16 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:16.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:16 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:50:16.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:16 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:50:16.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:16 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:16.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:16 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:16.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:16 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:16.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:16 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:16.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:16 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:50:16.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:16 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:50:16.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:16 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:50:16.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:16 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:50:17.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1041024 10.23 MB/sec execute 389 sec latency 75.806 ms 2024-05-27T07:50:17.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.379+0000 7fc2c32e6700 1 -- 172.21.15.114:0/2160634205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc108750 msgr2=0x7fc2bc1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:17.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.379+0000 7fc2c32e6700 1 --2- 172.21.15.114:0/2160634205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc108750 0x7fc2bc1100b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0099c0 tx=0x7fc2ac009cd0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.379+0000 7fc2c32e6700 1 -- 172.21.15.114:0/2160634205 shutdown_connections 2024-05-27T07:50:17.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.379+0000 7fc2c32e6700 1 --2- 172.21.15.114:0/2160634205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc108750 0x7fc2bc1100b0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.379+0000 7fc2c32e6700 1 --2- 172.21.15.114:0/2160634205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2bc1034d0 0x7fc2bc1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.379+0000 7fc2c32e6700 1 --2- 172.21.15.114:0/2160634205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2bc107e00 0x7fc2bc108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.379+0000 7fc2c32e6700 1 -- 172.21.15.114:0/2160634205 >> 172.21.15.114:0/2160634205 conn(0x7fc2bc0fef20 msgr2=0x7fc2bc101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:17.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.380+0000 7fc2c32e6700 1 -- 172.21.15.114:0/2160634205 shutdown_connections 2024-05-27T07:50:17.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.380+0000 7fc2c32e6700 1 -- 172.21.15.114:0/2160634205 wait complete. 2024-05-27T07:50:17.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.380+0000 7fc2c32e6700 1 Processor -- start 2024-05-27T07:50:17.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.380+0000 7fc2c32e6700 1 -- start start 2024-05-27T07:50:17.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c32e6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2bc1034d0 0x7fc2bc19ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c32e6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2bc107e00 0x7fc2bc19b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c32e6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc108750 0x7fc2bc19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c32e6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc113380 con 0x7fc2bc108750 2024-05-27T07:50:17.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c32e6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc113200 con 0x7fc2bc1034d0 2024-05-27T07:50:17.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c32e6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc113500 con 0x7fc2bc107e00 2024-05-27T07:50:17.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c1883700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc108750 0x7fc2bc19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c1883700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc108750 0x7fc2bc19f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40674/0 (socket says 172.21.15.114:40674) 2024-05-27T07:50:17.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c1883700 1 -- 172.21.15.114:0/746934565 learned_addr learned my addr 172.21.15.114:0/746934565 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:17.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c1082700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2bc1034d0 0x7fc2bc19ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c0881700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2bc107e00 0x7fc2bc19b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c1883700 1 -- 172.21.15.114:0/746934565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2bc107e00 msgr2=0x7fc2bc19b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:17.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c1883700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2bc107e00 0x7fc2bc19b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c1883700 1 -- 172.21.15.114:0/746934565 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2bc1034d0 msgr2=0x7fc2bc19ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:17.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c1883700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2bc1034d0 0x7fc2bc19ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.381+0000 7fc2c1883700 1 -- 172.21.15.114:0/746934565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2bc19fcb0 con 0x7fc2bc108750 2024-05-27T07:50:17.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.382+0000 7fc2c1883700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc108750 0x7fc2bc19f610 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0059d0 tx=0x7fc2ac004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:17.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.382+0000 7fc2ba7fc700 1 -- 172.21.15.114:0/746934565 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac026070 con 0x7fc2bc108750 2024-05-27T07:50:17.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.382+0000 7fc2c32e6700 1 -- 172.21.15.114:0/746934565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2bc19ff40 con 0x7fc2bc108750 2024-05-27T07:50:17.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.382+0000 7fc2ba7fc700 1 -- 172.21.15.114:0/746934565 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2ac003c70 con 0x7fc2bc108750 2024-05-27T07:50:17.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.382+0000 7fc2ba7fc700 1 -- 172.21.15.114:0/746934565 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac00fbf0 con 0x7fc2bc108750 2024-05-27T07:50:17.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.382+0000 7fc2c32e6700 1 -- 172.21.15.114:0/746934565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2bc0ff330 con 0x7fc2bc108750 2024-05-27T07:50:17.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.384+0000 7fc2ba7fc700 1 -- 172.21.15.114:0/746934565 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fc2ac029070 con 0x7fc2bc108750 2024-05-27T07:50:17.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.386+0000 7fc2ba7fc700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2a40b29f0 0x7fc2a40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.386+0000 7fc2c1082700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2a40b29f0 0x7fc2a40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.386+0000 7fc2ba7fc700 1 -- 172.21.15.114:0/746934565 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7fc2ac0e31e0 con 0x7fc2bc108750 2024-05-27T07:50:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.386+0000 7fc2d1cfbb80 1 Processor -- start 2024-05-27T07:50:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.386+0000 7fc2d1cfbb80 1 -- start start 2024-05-27T07:50:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.386+0000 7fc2c1082700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2a40b29f0 0x7fc2a40b4f00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc2b400f540 tx=0x7fc2b400b040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.386+0000 7fc2d1cfbb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b3a8ec0 0x557f8b52c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.386+0000 7fc2d1cfbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b526ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2d1cfbb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b527000 0x557f8b527410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2d1cfbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557f8b455d10 con 0x557f8b52ccd0 2024-05-27T07:50:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2d1cfbb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557f8b4458b0 con 0x557f8b527000 2024-05-27T07:50:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2d1cfbb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557f8b4498f0 con 0x557f8b3a8ec0 2024-05-27T07:50:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2c1082700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b3a8ec0 0x557f8b52c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2c1082700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b3a8ec0 0x557f8b52c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:57142/0 (socket says 172.21.15.114:57142) 2024-05-27T07:50:17.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2c1082700 1 -- 172.21.15.114:0/2859740182 learned_addr learned my addr 172.21.15.114:0/2859740182 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:17.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2c1883700 1 --2- 172.21.15.114:0/2859740182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b527000 0x557f8b527410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2c0881700 1 --2- 172.21.15.114:0/2859740182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b526ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2c1082700 1 -- 172.21.15.114:0/2859740182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b527000 msgr2=0x557f8b527410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:17.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2c1082700 1 --2- 172.21.15.114:0/2859740182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b527000 0x557f8b527410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2c1082700 1 -- 172.21.15.114:0/2859740182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 msgr2=0x557f8b526ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:17.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2c1082700 1 --2- 172.21.15.114:0/2859740182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b526ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2c1082700 1 -- 172.21.15.114:0/2859740182 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f8b395950 con 0x557f8b3a8ec0 2024-05-27T07:50:17.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.387+0000 7fc2c1082700 1 --2- 172.21.15.114:0/2859740182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b3a8ec0 0x557f8b52c790 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4003d20 tx=0x7fc2b4014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4acbf5a5fde0a4d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:17.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.388+0000 7fc2a37fe700 1 -- 172.21.15.114:0/2859740182 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4027070 con 0x557f8b3a8ec0 2024-05-27T07:50:17.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.388+0000 7fc2a37fe700 1 -- 172.21.15.114:0/2859740182 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2b401e040 con 0x557f8b3a8ec0 2024-05-27T07:50:17.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.388+0000 7fc2a37fe700 1 -- 172.21.15.114:0/2859740182 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b401d050 con 0x557f8b3a8ec0 2024-05-27T07:50:17.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.388+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/2859740182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b3a8ec0 msgr2=0x557f8b52c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:17.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.388+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/2859740182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b3a8ec0 0x557f8b52c790 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4003d20 tx=0x7fc2b4014000 comp rx=0 tx=0).stop 2024-05-27T07:50:17.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.389+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/2859740182 shutdown_connections 2024-05-27T07:50:17.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.389+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/2859740182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b527000 0x557f8b527410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.389+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/2859740182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b3a8ec0 0x557f8b52c790 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.389+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/2859740182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b526ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.389+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/2859740182 >> 172.21.15.114:0/2859740182 conn(0x557f8b403b70 msgr2=0x557f8b5216c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:17.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.389+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/2859740182 shutdown_connections 2024-05-27T07:50:17.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.389+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/2859740182 wait complete. 2024-05-27T07:50:17.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.389+0000 7fc2d1cfbb80 1 -- start start 2024-05-27T07:50:17.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.389+0000 7fc2d1cfbb80 1 Processor -- start 2024-05-27T07:50:17.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2d1cfbb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b3a8ec0 0x557f8b530410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2d1cfbb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b530950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2d1cfbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b530e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2d1cfbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557f8b34c020 con 0x557f8b52ccd0 2024-05-27T07:50:17.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2d1cfbb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557f8b3e7840 con 0x557f8b3a8ec0 2024-05-27T07:50:17.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2d1cfbb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557f8b5337c0 con 0x557f8b527000 2024-05-27T07:50:17.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2d1cfbb80 10 client.0 _ll_register_callbacks cb 0x557f8b3e09a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:17.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2c1883700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b530e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2c1883700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b530e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40696/0 (socket says 172.21.15.114:40696) 2024-05-27T07:50:17.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2c1883700 1 -- 172.21.15.114:0/601538371 learned_addr learned my addr 172.21.15.114:0/601538371 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:17.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2c0881700 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b530950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2c1082700 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b3a8ec0 0x557f8b530410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2c1883700 1 -- 172.21.15.114:0/601538371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 msgr2=0x557f8b530950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2c1883700 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b530950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2c1883700 1 -- 172.21.15.114:0/601538371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b3a8ec0 msgr2=0x557f8b530410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2c1883700 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b3a8ec0 0x557f8b530410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2c1883700 1 -- 172.21.15.114:0/601538371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f8b3e4710 con 0x557f8b52ccd0 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2c0881700 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b530950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2c1883700 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b530e90 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac003fd0 tx=0x7fc2ac009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92e281c25e0f8e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2a0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2a0ff9700 1 -- 172.21.15.114:0/601538371 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac0aa790 con 0x557f8b52ccd0 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2a0ff9700 1 -- 172.21.15.114:0/601538371 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7fc2ac0ae0d0 con 0x557f8b52ccd0 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.390+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/601538371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2b401e040 con 0x557f8b52ccd0 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.391+0000 7fc2a0ff9700 1 -- 172.21.15.114:0/601538371 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fc2ac0aa090 con 0x557f8b52ccd0 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.391+0000 7fc2d1cfbb80 10 client.17871 fetch_fsmap learned FSMap version 58 2024-05-27T07:50:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.391+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/601538371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x557f8b53e270 con 0x557f8b52ccd0 2024-05-27T07:50:17.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.391+0000 7fc2a0ff9700 1 -- 172.21.15.114:0/601538371 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2b401e040 con 0x557f8b52ccd0 2024-05-27T07:50:17.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.391+0000 7fc2a0ff9700 1 -- 172.21.15.114:0/601538371 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 58) ==== 6515+0+0 (secure 0 0 0) 0x7fc2ac0b29d0 con 0x557f8b52ccd0 2024-05-27T07:50:17.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.391+0000 7fc2d1cfbb80 10 client.17871 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:50:17.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.391+0000 7fc2d1cfbb80 10 client.17871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:17.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.391+0000 7fc2d1cfbb80 20 client.17871 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:17.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.391+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f8b540880 0x557f8b542cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.391+0000 7fc2d1cfbb80 4 client.17871 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:50:17.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.391+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/601538371 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f8b4f2a70 con 0x557f8b540880 2024-05-27T07:50:17.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.391+0000 7fc2c1082700 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f8b540880 0x557f8b542cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.392+0000 7fc2a0ff9700 10 client.17871 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:17.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.392+0000 7fc2c1082700 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f8b540880 0x557f8b542cd0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=debb030d94a28101 server_cookie=edfede88b78ff337 in_seq=0 out_seq=0 2024-05-27T07:50:17.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2a0ff9700 1 -- 172.21.15.114:0/601538371 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557f8b4f2a70 con 0x557f8b540880 2024-05-27T07:50:17.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2a0ff9700 10 client.17871 handle_command_reply: tid=0 2024-05-27T07:50:17.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2d1cfbb80 1 client.17871 shutdown 2024-05-27T07:50:17.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/601538371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 msgr2=0x557f8b530e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:17.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b530e90 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac003fd0 tx=0x7fc2ac009ee0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/601538371 shutdown_connections 2024-05-27T07:50:17.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b530e90 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f8b540880 0x557f8b542cd0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b3a8ec0 0x557f8b530410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/601538371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b530950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/601538371 >> 172.21.15.114:0/601538371 conn(0x557f8b403b70 msgr2=0x557f8b525ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:17.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2a0ff9700 0 client.17871 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:17.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/601538371 shutdown_connections 2024-05-27T07:50:17.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/601538371 wait complete. 2024-05-27T07:50:17.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.394+0000 7fc2d1cfbb80 20 client.17871 trim_cache size 0 max 16384 2024-05-27T07:50:17.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.416+0000 7fc2d1cfbb80 1 Processor -- start 2024-05-27T07:50:17.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.416+0000 7fc2d1cfbb80 1 -- start start 2024-05-27T07:50:17.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2d1cfbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b3a8ec0 0x557f8b52aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2d1cfbb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b530cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2d1cfbb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b52ccd0 0x557f8b5328b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2d1cfbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc1b3170 con 0x557f8b3a8ec0 2024-05-27T07:50:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2d1cfbb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc1b0910 con 0x557f8b52ccd0 2024-05-27T07:50:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2d1cfbb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc1b2aa0 con 0x557f8b527000 2024-05-27T07:50:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2c1082700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b3a8ec0 0x557f8b52aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2c1082700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b3a8ec0 0x557f8b52aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40706/0 (socket says 172.21.15.114:40706) 2024-05-27T07:50:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2c1082700 1 -- 172.21.15.114:0/163374566 learned_addr learned my addr 172.21.15.114:0/163374566 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2c0881700 1 --2- 172.21.15.114:0/163374566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b530cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2c1883700 1 --2- 172.21.15.114:0/163374566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b52ccd0 0x557f8b5328b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2c1082700 1 -- 172.21.15.114:0/163374566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 msgr2=0x557f8b530cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2c1082700 1 --2- 172.21.15.114:0/163374566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b530cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2c1082700 1 -- 172.21.15.114:0/163374566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b52ccd0 msgr2=0x557f8b5328b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2c1082700 1 --2- 172.21.15.114:0/163374566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b52ccd0 0x557f8b5328b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2c1082700 1 -- 172.21.15.114:0/163374566 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f8b5432a0 con 0x557f8b3a8ec0 2024-05-27T07:50:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2c0881700 1 --2- 172.21.15.114:0/163374566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b530cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:17.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.417+0000 7fc2c1082700 1 --2- 172.21.15.114:0/163374566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b3a8ec0 0x557f8b52aac0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4003d20 tx=0x7fc2b4009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ef6af682f37832 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:17.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.418+0000 7fc2a2ffd700 1 -- 172.21.15.114:0/163374566 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b401d040 con 0x557f8b3a8ec0 2024-05-27T07:50:17.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.418+0000 7fc2a2ffd700 1 -- 172.21.15.114:0/163374566 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2b4028040 con 0x557f8b3a8ec0 2024-05-27T07:50:17.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.418+0000 7fc2a2ffd700 1 -- 172.21.15.114:0/163374566 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4027040 con 0x557f8b3a8ec0 2024-05-27T07:50:17.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.419+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/163374566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b3a8ec0 msgr2=0x557f8b52aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:17.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.419+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/163374566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b3a8ec0 0x557f8b52aac0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4003d20 tx=0x7fc2b4009f50 comp rx=0 tx=0).stop 2024-05-27T07:50:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.419+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/163374566 shutdown_connections 2024-05-27T07:50:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.419+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/163374566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b52ccd0 0x557f8b5328b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.419+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/163374566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b3a8ec0 0x557f8b52aac0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.419+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/163374566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b530cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.419+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/163374566 >> 172.21.15.114:0/163374566 conn(0x557f8b403b70 msgr2=0x557f8b52f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.419+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/163374566 shutdown_connections 2024-05-27T07:50:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.419+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/163374566 wait complete. 2024-05-27T07:50:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.419+0000 7fc2d1cfbb80 1 -- start start 2024-05-27T07:50:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.419+0000 7fc2d1cfbb80 1 Processor -- start 2024-05-27T07:50:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2d1cfbb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b3a8ec0 0x557f8b5346c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2d1cfbb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b534c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2d1cfbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b535140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2d1cfbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc1b23d0 con 0x557f8b52ccd0 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2d1cfbb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc1b0f60 con 0x557f8b3a8ec0 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2d1cfbb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc1b1630 con 0x557f8b527000 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2d1cfbb80 10 client.0 _ll_register_callbacks cb 0x557f8b3e09a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2c1883700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b535140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2c1883700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b535140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40712/0 (socket says 172.21.15.114:40712) 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2c1883700 1 -- 172.21.15.114:0/2297148403 learned_addr learned my addr 172.21.15.114:0/2297148403 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2c0881700 1 --2- 172.21.15.114:0/2297148403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b534c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2c1082700 1 --2- 172.21.15.114:0/2297148403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b3a8ec0 0x557f8b5346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2c1883700 1 -- 172.21.15.114:0/2297148403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 msgr2=0x557f8b534c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2c1883700 1 --2- 172.21.15.114:0/2297148403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b534c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2c1883700 1 -- 172.21.15.114:0/2297148403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b3a8ec0 msgr2=0x557f8b5346c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2c1883700 1 --2- 172.21.15.114:0/2297148403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b3a8ec0 0x557f8b5346c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2c1883700 1 -- 172.21.15.114:0/2297148403 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f8b5aa3a0 con 0x557f8b52ccd0 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2c1883700 1 --2- 172.21.15.114:0/2297148403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b535140 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0a7300 tx=0x7fc2ac0a9ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd8a7b544727059a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2a0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2a0ff9700 1 -- 172.21.15.114:0/2297148403 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac0aa790 con 0x557f8b52ccd0 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/2297148403 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2b4028040 con 0x557f8b52ccd0 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.420+0000 7fc2a0ff9700 1 -- 172.21.15.114:0/2297148403 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7fc2ac0037c0 con 0x557f8b52ccd0 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.421+0000 7fc2a0ff9700 1 -- 172.21.15.114:0/2297148403 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fc2ac0c9050 con 0x557f8b52ccd0 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.421+0000 7fc2d1cfbb80 10 client.17877 fetch_fsmap learned FSMap version 58 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.421+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/2297148403 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x557f8b5ab4a0 con 0x557f8b52ccd0 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.421+0000 7fc2a0ff9700 1 -- 172.21.15.114:0/2297148403 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2b4028040 con 0x557f8b52ccd0 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.421+0000 7fc2a0ff9700 1 -- 172.21.15.114:0/2297148403 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 58) ==== 6515+0+0 (secure 0 0 0) 0x7fc2ac0e0ce0 con 0x557f8b52ccd0 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.421+0000 7fc2d1cfbb80 10 client.17877 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.421+0000 7fc2d1cfbb80 10 client.17877 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.421+0000 7fc2d1cfbb80 20 client.17877 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.421+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/2297148403 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f8b5ab930 0x557f8b5add50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:17.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.421+0000 7fc2d1cfbb80 4 client.17877 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:50:17.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.421+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/2297148403 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557f8b52c0e0 con 0x557f8b5ab930 2024-05-27T07:50:17.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.421+0000 7fc2c1082700 1 --2- 172.21.15.114:0/2297148403 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f8b5ab930 0x557f8b5add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:17.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.422+0000 7fc2a0ff9700 10 client.17877 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:17.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.422+0000 7fc2c1082700 1 --2- 172.21.15.114:0/2297148403 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f8b5ab930 0x557f8b5add50 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fb5150b541d5a00 server_cookie=45e8b90bcbf7ae21 in_seq=0 out_seq=0 2024-05-27T07:50:17.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.423+0000 7fc2a0ff9700 1 -- 172.21.15.114:0/2297148403 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557f8b52c0e0 con 0x557f8b5ab930 2024-05-27T07:50:17.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.423+0000 7fc2a0ff9700 10 client.17877 handle_command_reply: tid=0 2024-05-27T07:50:17.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.423+0000 7fc2d1cfbb80 1 client.17877 shutdown 2024-05-27T07:50:17.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.423+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/2297148403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 msgr2=0x557f8b535140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:17.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.423+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/2297148403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b535140 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0a7300 tx=0x7fc2ac0a9ab0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.423+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/2297148403 shutdown_connections 2024-05-27T07:50:17.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.423+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/2297148403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f8b52ccd0 0x557f8b535140 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.423+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/2297148403 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557f8b5ab930 0x557f8b5add50 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.423+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/2297148403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f8b3a8ec0 0x557f8b5346c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.423+0000 7fc2d1cfbb80 1 --2- 172.21.15.114:0/2297148403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557f8b527000 0x557f8b534c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.423+0000 7fc2a0ff9700 0 client.17877 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:17.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.423+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/2297148403 >> 172.21.15.114:0/2297148403 conn(0x557f8b403b70 msgr2=0x557f8b52a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:17.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.423+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/2297148403 shutdown_connections 2024-05-27T07:50:17.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2d1cfbb80 1 -- 172.21.15.114:0/2297148403 wait complete. 2024-05-27T07:50:17.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2d1cfbb80 20 client.17877 trim_cache size 0 max 16384 2024-05-27T07:50:17.428 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:50:17.428 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:50:17.428 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "e6012b15-aaa5-42c0-bd88-554dc25ba4c5", 2024-05-27T07:50:17.428 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:50:17.428 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:50:17.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2c32e6700 1 -- 172.21.15.114:0/746934565 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2a40b29f0 msgr2=0x7fc2a40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:17.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2c32e6700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2a40b29f0 0x7fc2a40b4f00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc2b400f540 tx=0x7fc2b400b040 comp rx=0 tx=0).stop 2024-05-27T07:50:17.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2c32e6700 1 -- 172.21.15.114:0/746934565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc108750 msgr2=0x7fc2bc19f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:17.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2c32e6700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc108750 0x7fc2bc19f610 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0059d0 tx=0x7fc2ac004270 comp rx=0 tx=0).stop 2024-05-27T07:50:17.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2c32e6700 1 -- 172.21.15.114:0/746934565 shutdown_connections 2024-05-27T07:50:17.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2c32e6700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc108750 0x7fc2bc19f610 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2c32e6700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc2a40b29f0 0x7fc2a40b4f00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2c32e6700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc2bc1034d0 0x7fc2bc19ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2c32e6700 1 --2- 172.21.15.114:0/746934565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2bc107e00 0x7fc2bc19b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:17.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2c32e6700 1 -- 172.21.15.114:0/746934565 >> 172.21.15.114:0/746934565 conn(0x7fc2bc0fef20 msgr2=0x7fc2bc06c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:17.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2c32e6700 1 -- 172.21.15.114:0/746934565 shutdown_connections 2024-05-27T07:50:17.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:17.424+0000 7fc2c32e6700 1 -- 172.21.15.114:0/746934565 wait complete. 2024-05-27T07:50:17.751 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:17 smithi186 ceph-mon[36647]: pgmap v346: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.2 MiB/s wr, 314 op/s 2024-05-27T07:50:17.751 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:17 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi114 to 176.1M 2024-05-27T07:50:17.751 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:17 smithi186 ceph-mon[36647]: Unable to set osd_memory_target on smithi114 to 184753049: error parsing value: Value '184753049' is below minimum 939524096 2024-05-27T07:50:17.751 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:17 smithi186 ceph-mon[36647]: pgmap v347: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 2.0 MiB/s wr, 205 op/s 2024-05-27T07:50:17.751 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:17 smithi186 ceph-mon[36647]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:50:17.751 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:17 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:50:17.751 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:17 smithi186 systemd[1]: Stopped Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:50:17.751 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:17 smithi186 systemd[1]: Starting Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:50:17.751 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:17 smithi186 podman[74774]: 2024-05-27 07:50:17.626364026 +0000 UTC m=+0.018277436 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:50:17.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:17 smithi163 ceph-mon[36451]: pgmap v346: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.2 MiB/s wr, 314 op/s 2024-05-27T07:50:17.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:17 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi114 to 176.1M 2024-05-27T07:50:17.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:17 smithi163 ceph-mon[36451]: Unable to set osd_memory_target on smithi114 to 184753049: error parsing value: Value '184753049' is below minimum 939524096 2024-05-27T07:50:17.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:17 smithi163 ceph-mon[36451]: pgmap v347: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 2.0 MiB/s wr, 205 op/s 2024-05-27T07:50:17.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:17 smithi163 ceph-mon[36451]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:50:18.004 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:17 smithi186 podman[74774]: 2024-05-27 07:50:17.751482426 +0000 UTC m=+0.143395829 container create d3e73622e99a41ea6a8b26882f1853ae1412dba926e8fdb62bcee39a48e23a1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD) 2024-05-27T07:50:18.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:17 smithi114 ceph-mon[32038]: pgmap v346: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.2 MiB/s wr, 314 op/s 2024-05-27T07:50:18.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:17 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi114 to 176.1M 2024-05-27T07:50:18.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:17 smithi114 ceph-mon[32038]: Unable to set osd_memory_target on smithi114 to 184753049: error parsing value: Value '184753049' is below minimum 939524096 2024-05-27T07:50:18.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:17 smithi114 ceph-mon[32038]: pgmap v347: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 2.0 MiB/s wr, 205 op/s 2024-05-27T07:50:18.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:17 smithi114 ceph-mon[32038]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:50:18.161 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:50:18.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1042404 10.22 MB/sec execute 390 sec latency 33.778 ms 2024-05-27T07:50:18.419 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:50:18.479 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 podman[74774]: 2024-05-27 07:50:18.00521909 +0000 UTC m=+0.397132495 container init d3e73622e99a41ea6a8b26882f1853ae1412dba926e8fdb62bcee39a48e23a1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-27T07:50:18.479 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 podman[74774]: 2024-05-27 07:50:18.008303703 +0000 UTC m=+0.400217108 container start d3e73622e99a41ea6a8b26882f1853ae1412dba926e8fdb62bcee39a48e23a1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-27T07:50:18.479 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-mds[74793]: -- 172.21.15.186:0/491363792 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fb7e8afa40 con 0x55fb7e8f7180 2024-05-27T07:50:18.479 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: starting mds.c at 2024-05-27T07:50:18.479 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 bash[74774]: d3e73622e99a41ea6a8b26882f1853ae1412dba926e8fdb62bcee39a48e23a1e 2024-05-27T07:50:18.480 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 systemd[1]: Started Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:50:18.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:18 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:50:18.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:18 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:50:18.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:18 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:50:18.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:18 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:50:18.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:18 smithi114 ceph-mon[32038]: daemon mds.c restarted 2024-05-27T07:50:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:18 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:50:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:18 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:50:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:18 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:50:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:18 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:50:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:18 smithi163 ceph-mon[36451]: daemon mds.c restarted 2024-05-27T07:50:18.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:50:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:50:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:50:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:50:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-mon[36647]: daemon mds.c restarted 2024-05-27T07:50:18.979 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f864f64f700 time 2024-05-27T07:50:18.775114+0000 2024-05-27T07:50:18.979 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:50:18.979 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:18.979 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8661e94ba6] 2024-05-27T07:50:18.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8661e94dd1] 2024-05-27T07:50:18.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fb7cc3123d] 2024-05-27T07:50:18.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55fb7cc1c971] 2024-05-27T07:50:18.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 5: (SnapRealm::check_cache() const+0x13e) [0x55fb7cc1afa0] 2024-05-27T07:50:18.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fb7cc1b651] 2024-05-27T07:50:18.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fb7cbb0347] 2024-05-27T07:50:18.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fb7c8ac87e] 2024-05-27T07:50:18.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55fb7c8b1f2f] 2024-05-27T07:50:18.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55fb7c8bc969] 2024-05-27T07:50:18.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 11: (MDLog::_replay_thread()+0x1f38) [0x55fb7cc5548c] 2024-05-27T07:50:18.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55fb7cc5e9cb] 2024-05-27T07:50:18.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 13: (Thread::entry_wrapper()+0x43) [0x7f8661e6372d] 2024-05-27T07:50:18.981 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 14: (Thread::_entry_func(void*)+0xd) [0x7f8661e63749] 2024-05-27T07:50:18.981 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8660a071ca] 2024-05-27T07:50:18.981 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 16: clone() 2024-05-27T07:50:18.981 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.776+0000 7f864f64f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f864f64f700 time 2024-05-27T07:50:18.775114+0000 2024-05-27T07:50:18.981 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:50:18.982 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.982 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:18.982 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8661e94ba6] 2024-05-27T07:50:18.982 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8661e94dd1] 2024-05-27T07:50:18.982 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fb7cc3123d] 2024-05-27T07:50:18.982 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55fb7cc1c971] 2024-05-27T07:50:18.982 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 5: (SnapRealm::check_cache() const+0x13e) [0x55fb7cc1afa0] 2024-05-27T07:50:18.982 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fb7cc1b651] 2024-05-27T07:50:18.982 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fb7cbb0347] 2024-05-27T07:50:18.982 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fb7c8ac87e] 2024-05-27T07:50:18.982 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55fb7c8b1f2f] 2024-05-27T07:50:18.982 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55fb7c8bc969] 2024-05-27T07:50:18.983 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 11: (MDLog::_replay_thread()+0x1f38) [0x55fb7cc5548c] 2024-05-27T07:50:18.983 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55fb7cc5e9cb] 2024-05-27T07:50:18.983 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 13: (Thread::entry_wrapper()+0x43) [0x7f8661e6372d] 2024-05-27T07:50:18.983 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 14: (Thread::_entry_func(void*)+0xd) [0x7f8661e63749] 2024-05-27T07:50:18.983 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8660a071ca] 2024-05-27T07:50:18.983 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 16: clone() 2024-05-27T07:50:18.983 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.983 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: *** Caught signal (Aborted) ** 2024-05-27T07:50:18.983 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: in thread 7f864f64f700 thread_name:md_log_replay 2024-05-27T07:50:18.983 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:18.983 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8660a11d20] 2024-05-27T07:50:18.983 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2: gsignal() 2024-05-27T07:50:18.983 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 3: abort() 2024-05-27T07:50:18.984 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8661e94ced] 2024-05-27T07:50:18.984 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8661e94dd1] 2024-05-27T07:50:18.984 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fb7cc3123d] 2024-05-27T07:50:18.984 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55fb7cc1c971] 2024-05-27T07:50:18.984 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 8: (SnapRealm::check_cache() const+0x13e) [0x55fb7cc1afa0] 2024-05-27T07:50:18.984 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fb7cc1b651] 2024-05-27T07:50:18.984 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fb7cbb0347] 2024-05-27T07:50:18.984 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fb7c8ac87e] 2024-05-27T07:50:18.984 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55fb7c8b1f2f] 2024-05-27T07:50:18.984 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55fb7c8bc969] 2024-05-27T07:50:18.984 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 14: (MDLog::_replay_thread()+0x1f38) [0x55fb7cc5548c] 2024-05-27T07:50:18.984 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55fb7cc5e9cb] 2024-05-27T07:50:18.984 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 16: (Thread::entry_wrapper()+0x43) [0x7f8661e6372d] 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 17: (Thread::_entry_func(void*)+0xd) [0x7f8661e63749] 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8660a071ca] 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 19: clone() 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.778+0000 7f864f64f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: in thread 7f864f64f700 thread_name:md_log_replay 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8660a11d20] 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2: gsignal() 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 3: abort() 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8661e94ced] 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8661e94dd1] 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fb7cc3123d] 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55fb7cc1c971] 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 8: (SnapRealm::check_cache() const+0x13e) [0x55fb7cc1afa0] 2024-05-27T07:50:18.985 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fb7cc1b651] 2024-05-27T07:50:18.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fb7cbb0347] 2024-05-27T07:50:18.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fb7c8ac87e] 2024-05-27T07:50:18.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55fb7c8b1f2f] 2024-05-27T07:50:18.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55fb7c8bc969] 2024-05-27T07:50:18.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 14: (MDLog::_replay_thread()+0x1f38) [0x55fb7cc5548c] 2024-05-27T07:50:18.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55fb7cc5e9cb] 2024-05-27T07:50:18.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 16: (Thread::entry_wrapper()+0x43) [0x7f8661e6372d] 2024-05-27T07:50:18.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 17: (Thread::_entry_func(void*)+0xd) [0x7f8661e63749] 2024-05-27T07:50:18.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8660a071ca] 2024-05-27T07:50:18.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 19: clone() 2024-05-27T07:50:18.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:50:18.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.987 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: -1> 2024-05-27T07:50:18.776+0000 7f864f64f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f864f64f700 time 2024-05-27T07:50:18.775114+0000 2024-05-27T07:50:18.987 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:50:18.987 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.987 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:18.987 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8661e94ba6] 2024-05-27T07:50:18.987 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8661e94dd1] 2024-05-27T07:50:18.987 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fb7cc3123d] 2024-05-27T07:50:18.987 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55fb7cc1c971] 2024-05-27T07:50:18.987 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 5: (SnapRealm::check_cache() const+0x13e) [0x55fb7cc1afa0] 2024-05-27T07:50:18.987 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fb7cc1b651] 2024-05-27T07:50:18.987 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fb7cbb0347] 2024-05-27T07:50:18.987 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fb7c8ac87e] 2024-05-27T07:50:18.988 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55fb7c8b1f2f] 2024-05-27T07:50:18.988 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55fb7c8bc969] 2024-05-27T07:50:18.988 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 11: (MDLog::_replay_thread()+0x1f38) [0x55fb7cc5548c] 2024-05-27T07:50:18.988 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55fb7cc5e9cb] 2024-05-27T07:50:18.988 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 13: (Thread::entry_wrapper()+0x43) [0x7f8661e6372d] 2024-05-27T07:50:18.988 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 14: (Thread::_entry_func(void*)+0xd) [0x7f8661e63749] 2024-05-27T07:50:18.988 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8660a071ca] 2024-05-27T07:50:18.988 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 16: clone() 2024-05-27T07:50:18.988 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.988 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 0> 2024-05-27T07:50:18.778+0000 7f864f64f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: in thread 7f864f64f700 thread_name:md_log_replay 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8660a11d20] 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2: gsignal() 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 3: abort() 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8661e94ced] 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8661e94dd1] 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fb7cc3123d] 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55fb7cc1c971] 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 8: (SnapRealm::check_cache() const+0x13e) [0x55fb7cc1afa0] 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fb7cc1b651] 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fb7cbb0347] 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fb7c8ac87e] 2024-05-27T07:50:18.989 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55fb7c8b1f2f] 2024-05-27T07:50:18.990 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55fb7c8bc969] 2024-05-27T07:50:18.990 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 14: (MDLog::_replay_thread()+0x1f38) [0x55fb7cc5548c] 2024-05-27T07:50:18.990 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55fb7cc5e9cb] 2024-05-27T07:50:18.990 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 16: (Thread::entry_wrapper()+0x43) [0x7f8661e6372d] 2024-05-27T07:50:18.990 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 17: (Thread::_entry_func(void*)+0xd) [0x7f8661e63749] 2024-05-27T07:50:18.990 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8660a071ca] 2024-05-27T07:50:18.990 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 19: clone() 2024-05-27T07:50:18.990 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:50:18.990 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.990 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: -1> 2024-05-27T07:50:18.776+0000 7f864f64f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f864f64f700 time 2024-05-27T07:50:18.775114+0000 2024-05-27T07:50:18.990 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:50:18.990 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.990 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:18.991 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8661e94ba6] 2024-05-27T07:50:18.991 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8661e94dd1] 2024-05-27T07:50:18.991 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fb7cc3123d] 2024-05-27T07:50:18.991 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55fb7cc1c971] 2024-05-27T07:50:18.991 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 5: (SnapRealm::check_cache() const+0x13e) [0x55fb7cc1afa0] 2024-05-27T07:50:18.991 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fb7cc1b651] 2024-05-27T07:50:18.991 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fb7cbb0347] 2024-05-27T07:50:18.991 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fb7c8ac87e] 2024-05-27T07:50:18.991 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55fb7c8b1f2f] 2024-05-27T07:50:18.991 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55fb7c8bc969] 2024-05-27T07:50:18.991 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 11: (MDLog::_replay_thread()+0x1f38) [0x55fb7cc5548c] 2024-05-27T07:50:18.991 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55fb7cc5e9cb] 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 13: (Thread::entry_wrapper()+0x43) [0x7f8661e6372d] 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 14: (Thread::_entry_func(void*)+0xd) [0x7f8661e63749] 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8660a071ca] 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 16: clone() 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 0> 2024-05-27T07:50:18.778+0000 7f864f64f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: in thread 7f864f64f700 thread_name:md_log_replay 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8660a11d20] 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2: gsignal() 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 3: abort() 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8661e94ced] 2024-05-27T07:50:18.992 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8661e94dd1] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fb7cc3123d] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55fb7cc1c971] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 8: (SnapRealm::check_cache() const+0x13e) [0x55fb7cc1afa0] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fb7cc1b651] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fb7cbb0347] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fb7c8ac87e] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55fb7c8b1f2f] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55fb7c8bc969] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 14: (MDLog::_replay_thread()+0x1f38) [0x55fb7cc5548c] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55fb7cc5e9cb] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 16: (Thread::entry_wrapper()+0x43) [0x7f8661e6372d] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 17: (Thread::_entry_func(void*)+0xd) [0x7f8661e63749] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8660a071ca] 2024-05-27T07:50:18.993 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 19: clone() 2024-05-27T07:50:18.994 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:50:18.994 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c[74789]: 2024-05-27T07:50:18.994 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:18 smithi186 podman[74825]: 2024-05-27 07:50:18.846231285 +0000 UTC m=+0.020506551 container died d3e73622e99a41ea6a8b26882f1853ae1412dba926e8fdb62bcee39a48e23a1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-27T07:50:19.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1044514 10.22 MB/sec execute 391 sec latency 16.811 ms 2024-05-27T07:50:19.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:19 smithi114 ceph-mon[32038]: pgmap v348: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 2.0 MiB/s wr, 205 op/s 2024-05-27T07:50:19.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:19 smithi114 ceph-mon[32038]: osdmap e118: 12 total, 12 up, 12 in 2024-05-27T07:50:19.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:19 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6834/3070699812,v1:172.21.15.186:6835/3070699812] up:boot 2024-05-27T07:50:19.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:19 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 8 up:standby 2024-05-27T07:50:19.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:19 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:50:19.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:19 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 7 up:standby 2024-05-27T07:50:19.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:19 smithi114 ceph-mon[32038]: osdmap e119: 12 total, 12 up, 12 in 2024-05-27T07:50:19.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:19 smithi163 ceph-mon[36451]: pgmap v348: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 2.0 MiB/s wr, 205 op/s 2024-05-27T07:50:19.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:19 smithi163 ceph-mon[36451]: osdmap e118: 12 total, 12 up, 12 in 2024-05-27T07:50:19.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:19 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6834/3070699812,v1:172.21.15.186:6835/3070699812] up:boot 2024-05-27T07:50:19.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:19 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 8 up:standby 2024-05-27T07:50:19.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:19 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:50:19.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:19 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 7 up:standby 2024-05-27T07:50:19.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:19 smithi163 ceph-mon[36451]: osdmap e119: 12 total, 12 up, 12 in 2024-05-27T07:50:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:19 smithi186 ceph-mon[36647]: pgmap v348: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 2.0 MiB/s wr, 205 op/s 2024-05-27T07:50:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:19 smithi186 ceph-mon[36647]: osdmap e118: 12 total, 12 up, 12 in 2024-05-27T07:50:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:19 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6834/3070699812,v1:172.21.15.186:6835/3070699812] up:boot 2024-05-27T07:50:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:19 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 8 up:standby 2024-05-27T07:50:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:19 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:50:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:19 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 7 up:standby 2024-05-27T07:50:19.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:19 smithi186 ceph-mon[36647]: osdmap e119: 12 total, 12 up, 12 in 2024-05-27T07:50:19.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:19 smithi186 podman[74825]: 2024-05-27 07:50:19.73777524 +0000 UTC m=+0.912050484 container remove d3e73622e99a41ea6a8b26882f1853ae1412dba926e8fdb62bcee39a48e23a1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240521) 2024-05-27T07:50:19.980 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:19 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:50:20.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.237+0000 7fed06f01700 1 -- 172.21.15.114:0/502868741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed001054b0 msgr2=0x7fed001058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:20.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.237+0000 7fed06f01700 1 --2- 172.21.15.114:0/502868741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed001054b0 0x7fed001058c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fecf4009990 tx=0x7fecf4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.237+0000 7fed06f01700 1 -- 172.21.15.114:0/502868741 shutdown_connections 2024-05-27T07:50:20.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.237+0000 7fed06f01700 1 --2- 172.21.15.114:0/502868741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed0010e8e0 0x7fed00110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.237+0000 7fed06f01700 1 --2- 172.21.15.114:0/502868741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed001054b0 0x7fed001058c0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.237+0000 7fed06f01700 1 --2- 172.21.15.114:0/502868741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed00105e00 0x7fed0010e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.237+0000 7fed06f01700 1 -- 172.21.15.114:0/502868741 >> 172.21.15.114:0/502868741 conn(0x7fed000fedb0 msgr2=0x7fed001011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:20.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.237+0000 7fed06f01700 1 -- 172.21.15.114:0/502868741 shutdown_connections 2024-05-27T07:50:20.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.237+0000 7fed06f01700 1 -- 172.21.15.114:0/502868741 wait complete. 2024-05-27T07:50:20.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fed06f01700 1 Processor -- start 2024-05-27T07:50:20.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fed06f01700 1 -- start start 2024-05-27T07:50:20.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fed06f01700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed001054b0 0x7fed0019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fed06f01700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed00105e00 0x7fed0019b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fed06f01700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed0010e8e0 0x7fed0019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fed06f01700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fed00114430 con 0x7fed001054b0 2024-05-27T07:50:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fed06f01700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fed001142b0 con 0x7fed0010e8e0 2024-05-27T07:50:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fed06f01700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fed001145b0 con 0x7fed00105e00 2024-05-27T07:50:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fed04c9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed001054b0 0x7fed0019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fecfffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed00105e00 0x7fed0019b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fed04c9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed001054b0 0x7fed0019ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40732/0 (socket says 172.21.15.114:40732) 2024-05-27T07:50:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fed04c9d700 1 -- 172.21.15.114:0/2764002807 learned_addr learned my addr 172.21.15.114:0/2764002807 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fed0549e700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed0010e8e0 0x7fed0019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fecfffff700 1 -- 172.21.15.114:0/2764002807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed0010e8e0 msgr2=0x7fed0019f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fecfffff700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed0010e8e0 0x7fed0019f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fecfffff700 1 -- 172.21.15.114:0/2764002807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed001054b0 msgr2=0x7fed0019ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fecfffff700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed001054b0 0x7fed0019ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fecfffff700 1 -- 172.21.15.114:0/2764002807 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0019fcb0 con 0x7fed00105e00 2024-05-27T07:50:20.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.238+0000 7fed04c9d700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed001054b0 0x7fed0019ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:20.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.239+0000 7fecfffff700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed00105e00 0x7fed0019b380 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fecec00e9f0 tx=0x7fecec00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:20.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.239+0000 7fecfdffb700 1 -- 172.21.15.114:0/2764002807 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec00c9c0 con 0x7fed00105e00 2024-05-27T07:50:20.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.239+0000 7fecfdffb700 1 -- 172.21.15.114:0/2764002807 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecec00cb60 con 0x7fed00105e00 2024-05-27T07:50:20.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.239+0000 7fed06f01700 1 -- 172.21.15.114:0/2764002807 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0019ffa0 con 0x7fed00105e00 2024-05-27T07:50:20.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.239+0000 7fecfdffb700 1 -- 172.21.15.114:0/2764002807 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec018680 con 0x7fed00105e00 2024-05-27T07:50:20.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.239+0000 7fed06f01700 1 -- 172.21.15.114:0/2764002807 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed001a78c0 con 0x7fed00105e00 2024-05-27T07:50:20.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.242+0000 7fecfdffb700 1 -- 172.21.15.114:0/2764002807 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fecec018820 con 0x7fed00105e00 2024-05-27T07:50:20.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fecfdffb700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fecf00b29f0 0x7fecf00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed15916b80 1 Processor -- start 2024-05-27T07:50:20.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed04c9d700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fecf00b29f0 0x7fecf00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fecfdffb700 1 -- 172.21.15.114:0/2764002807 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fecec0e2c60 con 0x7fed00105e00 2024-05-27T07:50:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed15916b80 1 -- start start 2024-05-27T07:50:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed15916b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e624e40 0x56507e7a8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed15916b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e7a8c50 0x56507e7a2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed15916b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a2f80 0x56507e7a3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed15916b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56507e6d1c90 con 0x56507e624e40 2024-05-27T07:50:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed15916b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56507e6c1830 con 0x56507e7a2f80 2024-05-27T07:50:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed15916b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56507e6c5870 con 0x56507e7a8c50 2024-05-27T07:50:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed04c9d700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fecf00b29f0 0x7fecf00b4f00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fecf400b2e0 tx=0x7fecf4004710 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed04c9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e624e40 0x56507e7a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed04c9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e624e40 0x56507e7a8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40736/0 (socket says 172.21.15.114:40736) 2024-05-27T07:50:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed04c9d700 1 -- 172.21.15.114:0/3878217315 learned_addr learned my addr 172.21.15.114:0/3878217315 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fecfffff700 1 --2- 172.21.15.114:0/3878217315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e7a8c50 0x56507e7a2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.243+0000 7fed0549e700 1 --2- 172.21.15.114:0/3878217315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a2f80 0x56507e7a3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.244+0000 7fed04c9d700 1 -- 172.21.15.114:0/3878217315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e7a8c50 msgr2=0x56507e7a2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.244+0000 7fed04c9d700 1 --2- 172.21.15.114:0/3878217315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e7a8c50 0x56507e7a2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.244+0000 7fed04c9d700 1 -- 172.21.15.114:0/3878217315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a2f80 msgr2=0x56507e7a3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.244+0000 7fed04c9d700 1 --2- 172.21.15.114:0/3878217315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a2f80 0x56507e7a3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.244+0000 7fed04c9d700 1 -- 172.21.15.114:0/3878217315 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56507e6118d0 con 0x56507e624e40 2024-05-27T07:50:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.244+0000 7fed04c9d700 1 --2- 172.21.15.114:0/3878217315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e624e40 0x56507e7a8710 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fecf4004310 tx=0x7fecf401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c53441b4bed27c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.244+0000 7feceaffd700 1 -- 172.21.15.114:0/3878217315 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf400be60 con 0x56507e624e40 2024-05-27T07:50:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.244+0000 7feceaffd700 1 -- 172.21.15.114:0/3878217315 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf4022450 con 0x56507e624e40 2024-05-27T07:50:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.244+0000 7feceaffd700 1 -- 172.21.15.114:0/3878217315 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf400f6c0 con 0x56507e624e40 2024-05-27T07:50:20.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.245+0000 7fed15916b80 1 -- 172.21.15.114:0/3878217315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e624e40 msgr2=0x56507e7a8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:20.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.245+0000 7fed15916b80 1 --2- 172.21.15.114:0/3878217315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e624e40 0x56507e7a8710 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fecf4004310 tx=0x7fecf401e000 comp rx=0 tx=0).stop 2024-05-27T07:50:20.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.245+0000 7fed15916b80 1 -- 172.21.15.114:0/3878217315 shutdown_connections 2024-05-27T07:50:20.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.245+0000 7fed15916b80 1 --2- 172.21.15.114:0/3878217315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a2f80 0x56507e7a3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.245+0000 7fed15916b80 1 --2- 172.21.15.114:0/3878217315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e624e40 0x56507e7a8710 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.245+0000 7fed15916b80 1 --2- 172.21.15.114:0/3878217315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e7a8c50 0x56507e7a2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.245+0000 7fed15916b80 1 -- 172.21.15.114:0/3878217315 >> 172.21.15.114:0/3878217315 conn(0x56507e67faf0 msgr2=0x56507e79d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.245+0000 7fed15916b80 1 -- 172.21.15.114:0/3878217315 shutdown_connections 2024-05-27T07:50:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.245+0000 7fed15916b80 1 -- 172.21.15.114:0/3878217315 wait complete. 2024-05-27T07:50:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed15916b80 1 -- start start 2024-05-27T07:50:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed15916b80 1 Processor -- start 2024-05-27T07:50:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed15916b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7ac340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed15916b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a2f80 0x56507e7ac880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed15916b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a8c50 0x56507e7acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed15916b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56507e5c7fa0 con 0x56507e7a8c50 2024-05-27T07:50:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed15916b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56507e6637c0 con 0x56507e7a2f80 2024-05-27T07:50:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed15916b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56507e7af680 con 0x56507e624e40 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed15916b80 10 client.0 _ll_register_callbacks cb 0x56507e65c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed04c9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7ac340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed0549e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a8c50 0x56507e7acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fecfffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a2f80 0x56507e7ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed0549e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a8c50 0x56507e7acdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40740/0 (socket says 172.21.15.114:40740) 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed04c9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7ac340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:57198/0 (socket says 172.21.15.114:57198) 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.246+0000 7fed0549e700 1 -- 172.21.15.114:0/1713883275 learned_addr learned my addr 172.21.15.114:0/1713883275 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fed0549e700 1 -- 172.21.15.114:0/1713883275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 msgr2=0x56507e7ac340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fed0549e700 1 --2- 172.21.15.114:0/1713883275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7ac340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fed0549e700 1 -- 172.21.15.114:0/1713883275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a2f80 msgr2=0x56507e7ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fed0549e700 1 --2- 172.21.15.114:0/1713883275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a2f80 0x56507e7ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fed0549e700 1 -- 172.21.15.114:0/1713883275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56507e660690 con 0x56507e7a8c50 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fed04c9d700 1 --2- 172.21.15.114:0/1713883275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7ac340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fecfffff700 1 --2- 172.21.15.114:0/1713883275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a2f80 0x56507e7ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fed0549e700 1 --2- 172.21.15.114:0/1713883275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a8c50 0x56507e7acdc0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fecf8011f60 tx=0x7fecf8014600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ffb26f7c38b94bd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fecdffff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:50:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fecdffff700 1 -- 172.21.15.114:0/1713883275 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8014ba0 con 0x56507e7a8c50 2024-05-27T07:50:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fecdffff700 1 -- 172.21.15.114:0/1713883275 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fecf8014d40 con 0x56507e7a8c50 2024-05-27T07:50:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fed15916b80 1 -- 172.21.15.114:0/1713883275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecf4022450 con 0x56507e7a8c50 2024-05-27T07:50:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fecdffff700 1 -- 172.21.15.114:0/1713883275 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7fecf8027050 con 0x56507e7a8c50 2024-05-27T07:50:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fed15916b80 10 client.17895 fetch_fsmap learned FSMap version 60 2024-05-27T07:50:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.247+0000 7fed15916b80 1 -- 172.21.15.114:0/1713883275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x56507e7ba190 con 0x56507e7a8c50 2024-05-27T07:50:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.248+0000 7fecdffff700 1 -- 172.21.15.114:0/1713883275 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf4022450 con 0x56507e7a8c50 2024-05-27T07:50:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.248+0000 7fecdffff700 1 -- 172.21.15.114:0/1713883275 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 60) ==== 6515+0+0 (secure 0 0 0) 0x7fecf8007500 con 0x56507e7a8c50 2024-05-27T07:50:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.248+0000 7fed15916b80 10 client.17895 fetch_fsmap finished waiting for FSMap version 60 2024-05-27T07:50:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.248+0000 7fed15916b80 10 client.17895 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.248+0000 7fed15916b80 20 client.17895 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.248+0000 7fed15916b80 1 --2- 172.21.15.114:0/1713883275 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56507e7bc8c0 0x56507e7bed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.248+0000 7fed15916b80 4 client.17895 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:50:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.248+0000 7fed15916b80 1 -- 172.21.15.114:0/1713883275 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56507e76e9f0 con 0x56507e7bc8c0 2024-05-27T07:50:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.248+0000 7fed04c9d700 1 --2- 172.21.15.114:0/1713883275 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56507e7bc8c0 0x56507e7bed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.248+0000 7fecdffff700 10 client.17895 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.249+0000 7fed04c9d700 1 --2- 172.21.15.114:0/1713883275 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56507e7bc8c0 0x56507e7bed30 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e1344b10dbb1d80 server_cookie=ac134a569ca07e33 in_seq=0 out_seq=0 2024-05-27T07:50:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fecdffff700 1 -- 172.21.15.114:0/1713883275 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56507e76e9f0 con 0x56507e7bc8c0 2024-05-27T07:50:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fecdffff700 10 client.17895 handle_command_reply: tid=0 2024-05-27T07:50:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fed15916b80 1 client.17895 shutdown 2024-05-27T07:50:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fed15916b80 1 -- 172.21.15.114:0/1713883275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a8c50 msgr2=0x56507e7acdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fed15916b80 1 --2- 172.21.15.114:0/1713883275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a8c50 0x56507e7acdc0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fecf8011f60 tx=0x7fecf8014600 comp rx=0 tx=0).stop 2024-05-27T07:50:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fed15916b80 1 -- 172.21.15.114:0/1713883275 shutdown_connections 2024-05-27T07:50:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fed15916b80 1 --2- 172.21.15.114:0/1713883275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a8c50 0x56507e7acdc0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fed15916b80 1 --2- 172.21.15.114:0/1713883275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7ac340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fed15916b80 1 --2- 172.21.15.114:0/1713883275 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56507e7bc8c0 0x56507e7bed30 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fed15916b80 1 --2- 172.21.15.114:0/1713883275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a2f80 0x56507e7ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fecdffff700 0 client.17895 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fed15916b80 1 -- 172.21.15.114:0/1713883275 >> 172.21.15.114:0/1713883275 conn(0x56507e67faf0 msgr2=0x56507e7a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fed15916b80 1 -- 172.21.15.114:0/1713883275 shutdown_connections 2024-05-27T07:50:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fed15916b80 1 -- 172.21.15.114:0/1713883275 wait complete. 2024-05-27T07:50:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.250+0000 7fed15916b80 20 client.17895 trim_cache size 0 max 16384 2024-05-27T07:50:20.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fed15916b80 1 Processor -- start 2024-05-27T07:50:20.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fed15916b80 1 -- start start 2024-05-27T07:50:20.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fed15916b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7a2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fed15916b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a2f80 0x56507e7a7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fed15916b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a8c50 0x56507e7acc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fed15916b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fed001b0d80 con 0x56507e7a2f80 2024-05-27T07:50:20.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fed15916b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fed001ae520 con 0x56507e7a8c50 2024-05-27T07:50:20.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fed15916b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fed001b06b0 con 0x56507e624e40 2024-05-27T07:50:20.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fecfffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a2f80 0x56507e7a7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fed04c9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7a2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fed04c9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7a2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:57200/0 (socket says 172.21.15.114:57200) 2024-05-27T07:50:20.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fecfffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a2f80 0x56507e7a7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40752/0 (socket says 172.21.15.114:40752) 2024-05-27T07:50:20.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fed04c9d700 1 -- 172.21.15.114:0/4175218820 learned_addr learned my addr 172.21.15.114:0/4175218820 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:20.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.272+0000 7fed0549e700 1 --2- 172.21.15.114:0/4175218820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a8c50 0x56507e7acc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.273+0000 7fed04c9d700 1 -- 172.21.15.114:0/4175218820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a8c50 msgr2=0x56507e7acc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:20.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.273+0000 7fed04c9d700 1 --2- 172.21.15.114:0/4175218820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a8c50 0x56507e7acc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.273+0000 7fed04c9d700 1 -- 172.21.15.114:0/4175218820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a2f80 msgr2=0x56507e7a7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:20.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.273+0000 7fed04c9d700 1 --2- 172.21.15.114:0/4175218820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a2f80 0x56507e7a7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.273+0000 7fed04c9d700 1 -- 172.21.15.114:0/4175218820 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56507e797fa0 con 0x56507e624e40 2024-05-27T07:50:20.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.273+0000 7fed04c9d700 1 --2- 172.21.15.114:0/4175218820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7a2a40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fecf40046e0 tx=0x7fecf400bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b26a7d8191045022 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:20.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.273+0000 7fecea7fc700 1 -- 172.21.15.114:0/4175218820 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf40047d0 con 0x56507e624e40 2024-05-27T07:50:20.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.273+0000 7fecea7fc700 1 -- 172.21.15.114:0/4175218820 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf403f070 con 0x56507e624e40 2024-05-27T07:50:20.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.273+0000 7fecea7fc700 1 -- 172.21.15.114:0/4175218820 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf4005660 con 0x56507e624e40 2024-05-27T07:50:20.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.274+0000 7fed15916b80 1 -- 172.21.15.114:0/4175218820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 msgr2=0x56507e7a2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:20.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.274+0000 7fed15916b80 1 --2- 172.21.15.114:0/4175218820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7a2a40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fecf40046e0 tx=0x7fecf400bf40 comp rx=0 tx=0).stop 2024-05-27T07:50:20.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.274+0000 7fed15916b80 1 -- 172.21.15.114:0/4175218820 shutdown_connections 2024-05-27T07:50:20.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.274+0000 7fed15916b80 1 --2- 172.21.15.114:0/4175218820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a8c50 0x56507e7acc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.274+0000 7fed15916b80 1 --2- 172.21.15.114:0/4175218820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7a2a40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.274+0000 7fed15916b80 1 --2- 172.21.15.114:0/4175218820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a2f80 0x56507e7a7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.274+0000 7fed15916b80 1 -- 172.21.15.114:0/4175218820 >> 172.21.15.114:0/4175218820 conn(0x56507e67faf0 msgr2=0x56507e7a7380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:20.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.274+0000 7fed15916b80 1 -- 172.21.15.114:0/4175218820 shutdown_connections 2024-05-27T07:50:20.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.274+0000 7fed15916b80 1 -- 172.21.15.114:0/4175218820 wait complete. 2024-05-27T07:50:20.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.274+0000 7fed15916b80 1 -- start start 2024-05-27T07:50:20.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.274+0000 7fed15916b80 1 Processor -- start 2024-05-27T07:50:20.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed15916b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed15916b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a2f80 0x56507e7b5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed15916b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a8c50 0x56507e7c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed15916b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fed001affe0 con 0x56507e7a2f80 2024-05-27T07:50:20.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed15916b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fed001aeb70 con 0x56507e7a8c50 2024-05-27T07:50:20.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed15916b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fed001af240 con 0x56507e624e40 2024-05-27T07:50:20.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed15916b80 10 client.0 _ll_register_callbacks cb 0x56507e65c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:20.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fecfffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a2f80 0x56507e7b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed04c9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fecfffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a2f80 0x56507e7b5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40754/0 (socket says 172.21.15.114:40754) 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fecfffff700 1 -- 172.21.15.114:0/3647383201 learned_addr learned my addr 172.21.15.114:0/3647383201 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed0549e700 1 --2- 172.21.15.114:0/3647383201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a8c50 0x56507e7c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed04c9d700 1 -- 172.21.15.114:0/3647383201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a8c50 msgr2=0x56507e7c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed04c9d700 1 --2- 172.21.15.114:0/3647383201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a8c50 0x56507e7c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed04c9d700 1 -- 172.21.15.114:0/3647383201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a2f80 msgr2=0x56507e7b5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed04c9d700 1 --2- 172.21.15.114:0/3647383201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a2f80 0x56507e7b5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fed04c9d700 1 -- 172.21.15.114:0/3647383201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56507e7c4080 con 0x56507e624e40 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.275+0000 7fecfffff700 1 --2- 172.21.15.114:0/3647383201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a2f80 0x56507e7b5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fed04c9d700 1 --2- 172.21.15.114:0/3647383201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7b4b30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fecf4006010 tx=0x7fecf40047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71d0b4e6b2eb924 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fecdffff700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fecdffff700 1 -- 172.21.15.114:0/3647383201 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf4004bd0 con 0x56507e624e40 2024-05-27T07:50:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fed15916b80 1 -- 172.21.15.114:0/3647383201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecf403f070 con 0x56507e624e40 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fecdffff700 1 -- 172.21.15.114:0/3647383201 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fecf4004d70 con 0x56507e624e40 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fecdffff700 1 -- 172.21.15.114:0/3647383201 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7fecf4005040 con 0x56507e624e40 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fed15916b80 10 client.27478 fetch_fsmap learned FSMap version 60 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fed15916b80 1 -- 172.21.15.114:0/3647383201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x56507e826db0 con 0x56507e624e40 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fecdffff700 1 -- 172.21.15.114:0/3647383201 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf403f070 con 0x56507e624e40 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fecdffff700 1 -- 172.21.15.114:0/3647383201 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 60) ==== 6515+0+0 (secure 0 0 0) 0x7fecf4022af0 con 0x56507e624e40 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fed15916b80 10 client.27478 fetch_fsmap finished waiting for FSMap version 60 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fed15916b80 10 client.27478 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fed15916b80 20 client.27478 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fed15916b80 1 --2- 172.21.15.114:0/3647383201 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56507e827270 0x56507e829690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fed15916b80 4 client.27478 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.276+0000 7fed15916b80 1 -- 172.21.15.114:0/3647383201 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56507e6d3be0 con 0x56507e827270 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.277+0000 7fecfffff700 1 --2- 172.21.15.114:0/3647383201 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56507e827270 0x56507e829690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:20.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.277+0000 7fecdffff700 10 client.27478 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:20.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.277+0000 7fecfffff700 1 --2- 172.21.15.114:0/3647383201 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56507e827270 0x56507e829690 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3af4330000eb78f server_cookie=2b05b86bdd0742a6 in_seq=0 out_seq=0 2024-05-27T07:50:20.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.280+0000 7fecdffff700 1 -- 172.21.15.114:0/3647383201 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56507e6d3be0 con 0x56507e827270 2024-05-27T07:50:20.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.280+0000 7fecdffff700 10 client.27478 handle_command_reply: tid=0 2024-05-27T07:50:20.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.280+0000 7fed15916b80 1 client.27478 shutdown 2024-05-27T07:50:20.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.280+0000 7fed15916b80 1 -- 172.21.15.114:0/3647383201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 msgr2=0x56507e7b4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:20.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.280+0000 7fed15916b80 1 --2- 172.21.15.114:0/3647383201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7b4b30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fecf4006010 tx=0x7fecf40047d0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.281+0000 7fed15916b80 1 -- 172.21.15.114:0/3647383201 shutdown_connections 2024-05-27T07:50:20.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.281+0000 7fed15916b80 1 --2- 172.21.15.114:0/3647383201 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56507e827270 0x56507e829690 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.281+0000 7fed15916b80 1 --2- 172.21.15.114:0/3647383201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56507e7a8c50 0x56507e7c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.281+0000 7fecdffff700 0 client.27478 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:20.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.281+0000 7fed15916b80 1 --2- 172.21.15.114:0/3647383201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507e624e40 0x56507e7b4b30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.281+0000 7fed15916b80 1 --2- 172.21.15.114:0/3647383201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507e7a2f80 0x56507e7b5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.281+0000 7fed15916b80 1 -- 172.21.15.114:0/3647383201 >> 172.21.15.114:0/3647383201 conn(0x56507e67faf0 msgr2=0x56507e7a6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:20.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.281+0000 7fed15916b80 1 -- 172.21.15.114:0/3647383201 shutdown_connections 2024-05-27T07:50:20.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.281+0000 7fed15916b80 1 -- 172.21.15.114:0/3647383201 wait complete. 2024-05-27T07:50:20.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.281+0000 7fed15916b80 20 client.27478 trim_cache size 0 max 16384 2024-05-27T07:50:20.284 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:50:20.284 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:50:20.284 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:50:20.284 INFO:teuthology.orchestra.run.smithi114.stdout: "e6012b15-aaa5-42c0-bd88-554dc25ba4c5": { 2024-05-27T07:50:20.285 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:50:20.285 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "e6012b15-aaa5-42c0-bd88-554dc25ba4c5", 2024-05-27T07:50:20.285 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:50:20.285 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:50:20.285 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:50:20.285 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:50:20.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.282+0000 7fed06f01700 1 -- 172.21.15.114:0/2764002807 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fecf00b29f0 msgr2=0x7fecf00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:20.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.282+0000 7fed06f01700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fecf00b29f0 0x7fecf00b4f00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fecf400b2e0 tx=0x7fecf4004710 comp rx=0 tx=0).stop 2024-05-27T07:50:20.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.282+0000 7fed06f01700 1 -- 172.21.15.114:0/2764002807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed00105e00 msgr2=0x7fed0019b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:20.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.282+0000 7fed06f01700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed00105e00 0x7fed0019b380 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fecec00e9f0 tx=0x7fecec00ed00 comp rx=0 tx=0).stop 2024-05-27T07:50:20.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.282+0000 7fed06f01700 1 -- 172.21.15.114:0/2764002807 shutdown_connections 2024-05-27T07:50:20.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.282+0000 7fed06f01700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fecf00b29f0 0x7fecf00b4f00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.282+0000 7fed06f01700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed0010e8e0 0x7fed0019f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.282+0000 7fed06f01700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed001054b0 0x7fed0019ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.282+0000 7fed06f01700 1 --2- 172.21.15.114:0/2764002807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed00105e00 0x7fed0019b380 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:20.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.282+0000 7fed06f01700 1 -- 172.21.15.114:0/2764002807 >> 172.21.15.114:0/2764002807 conn(0x7fed000fedb0 msgr2=0x7fed000ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:20.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.282+0000 7fed06f01700 1 -- 172.21.15.114:0/2764002807 shutdown_connections 2024-05-27T07:50:20.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:20.282+0000 7fed06f01700 1 -- 172.21.15.114:0/2764002807 wait complete. 2024-05-27T07:50:20.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1045570 10.21 MB/sec execute 392 sec latency 28.585 ms 2024-05-27T07:50:20.479 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:20 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:50:21.245 INFO:tasks.cephfs.filesystem:scrub status for tag:e6012b15-aaa5-42c0-bd88-554dc25ba4c5 - {'path': '/', 'tag': 'e6012b15-aaa5-42c0-bd88-554dc25ba4c5', 'options': 'recursive,force'} 2024-05-27T07:50:21.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1046914 10.19 MB/sec execute 393 sec latency 186.812 ms 2024-05-27T07:50:21.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:21 smithi163 ceph-mon[36451]: pgmap v351: 129 pgs: 129 active+clean; 71 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 360 op/s 2024-05-27T07:50:21.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:21 smithi186 ceph-mon[36647]: pgmap v351: 129 pgs: 129 active+clean; 71 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 360 op/s 2024-05-27T07:50:22.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:21 smithi114 ceph-mon[32038]: pgmap v351: 129 pgs: 129 active+clean; 71 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 360 op/s 2024-05-27T07:50:22.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1048643 10.20 MB/sec execute 394 sec latency 14.315 ms 2024-05-27T07:50:22.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:22 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:50:22.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:22 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:50:23.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:22 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:50:23.284 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1050944 10.20 MB/sec execute 395 sec latency 38.749 ms 2024-05-27T07:50:23.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:23 smithi163 ceph-mon[36451]: pgmap v352: 129 pgs: 129 active+clean; 72 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 177 op/s 2024-05-27T07:50:23.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:23 smithi186 ceph-mon[36647]: pgmap v352: 129 pgs: 129 active+clean; 72 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 177 op/s 2024-05-27T07:50:24.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:23 smithi114 ceph-mon[32038]: pgmap v352: 129 pgs: 129 active+clean; 72 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 177 op/s 2024-05-27T07:50:24.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1052411 10.20 MB/sec execute 396 sec latency 14.967 ms 2024-05-27T07:50:25.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1055061 10.21 MB/sec execute 397 sec latency 33.984 ms 2024-05-27T07:50:25.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:25 smithi114 ceph-mon[32038]: pgmap v353: 129 pgs: 129 active+clean; 72 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 134 op/s 2024-05-27T07:50:25.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:25 smithi163 ceph-mon[36451]: pgmap v353: 129 pgs: 129 active+clean; 72 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 134 op/s 2024-05-27T07:50:25.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:25 smithi186 ceph-mon[36647]: pgmap v353: 129 pgs: 129 active+clean; 72 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 134 op/s 2024-05-27T07:50:26.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1055766 10.19 MB/sec execute 398 sec latency 46.179 ms 2024-05-27T07:50:27.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1058047 10.20 MB/sec execute 399 sec latency 20.487 ms 2024-05-27T07:50:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.641+0000 7fb1b8a9a640 1 -- 172.21.15.114:0/3046438226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1b4108b90 msgr2=0x7fb1b4102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.641+0000 7fb1b8a9a640 1 --2- 172.21.15.114:0/3046438226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1b4108b90 0x7fb1b4102730 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4009900 tx=0x7fb1a402a410 comp rx=0 tx=0).stop 2024-05-27T07:50:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.641+0000 7fb1b8a9a640 1 -- 172.21.15.114:0/3046438226 shutdown_connections 2024-05-27T07:50:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.641+0000 7fb1b8a9a640 1 --2- 172.21.15.114:0/3046438226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1b41032e0 0x7fb1b410f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.641+0000 7fb1b8a9a640 1 --2- 172.21.15.114:0/3046438226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1b4109030 0x7fb1b4102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.641+0000 7fb1b8a9a640 1 --2- 172.21.15.114:0/3046438226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1b4108b90 0x7fb1b4102730 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.641+0000 7fb1b8a9a640 1 -- 172.21.15.114:0/3046438226 >> 172.21.15.114:0/3046438226 conn(0x7fb1b40782d0 msgr2=0x7fb1b4078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.641+0000 7fb1b8a9a640 1 -- 172.21.15.114:0/3046438226 shutdown_connections 2024-05-27T07:50:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.641+0000 7fb1b8a9a640 1 -- 172.21.15.114:0/3046438226 wait complete. 2024-05-27T07:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.642+0000 7fb1b8a9a640 1 Processor -- start 2024-05-27T07:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.642+0000 7fb1b8a9a640 1 -- start start 2024-05-27T07:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.642+0000 7fb1b8a9a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1b41032e0 0x7fb1b419b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.642+0000 7fb1b8a9a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1b4108b90 0x7fb1b419bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.642+0000 7fb1b8a9a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1b4109030 0x7fb1b41a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b8a9a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b4112d80 con 0x7fb1b4108b90 2024-05-27T07:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b8a9a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b4112c00 con 0x7fb1b4109030 2024-05-27T07:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b8a9a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b4112f00 con 0x7fb1b41032e0 2024-05-27T07:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b1d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1b4108b90 0x7fb1b419bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1b41032e0 0x7fb1b419b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b1d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1b4108b90 0x7fb1b419bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60864/0 (socket says 172.21.15.114:60864) 2024-05-27T07:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b1d9b640 1 -- 172.21.15.114:0/1689968750 learned_addr learned my addr 172.21.15.114:0/1689968750 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b2d9d640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1b4109030 0x7fb1b41a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b1d9b640 1 -- 172.21.15.114:0/1689968750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1b41032e0 msgr2=0x7fb1b419b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b1d9b640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1b41032e0 0x7fb1b419b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b1d9b640 1 -- 172.21.15.114:0/1689968750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1b4109030 msgr2=0x7fb1b41a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b1d9b640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1b4109030 0x7fb1b41a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b1d9b640 1 -- 172.21.15.114:0/1689968750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b419c480 con 0x7fb1b4108b90 2024-05-27T07:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b2d9d640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1b4109030 0x7fb1b41a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b259c640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1b41032e0 0x7fb1b419b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b1d9b640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1b4108b90 0x7fb1b419bc00 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800c970 tx=0x7fb1a800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1a37fe640 1 -- 172.21.15.114:0/1689968750 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8004220 con 0x7fb1b4108b90 2024-05-27T07:50:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1a37fe640 1 -- 172.21.15.114:0/1689968750 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1a80043c0 con 0x7fb1b4108b90 2024-05-27T07:50:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b8a9a640 1 -- 172.21.15.114:0/1689968750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b41a6d40 con 0x7fb1b4108b90 2024-05-27T07:50:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1a37fe640 1 -- 172.21.15.114:0/1689968750 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a800f450 con 0x7fb1b4108b90 2024-05-27T07:50:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.643+0000 7fb1b8a9a640 1 -- 172.21.15.114:0/1689968750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1b41a71d0 con 0x7fb1b4108b90 2024-05-27T07:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.646+0000 7fb1bfa20740 1 Processor -- start 2024-05-27T07:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.646+0000 7fb1a37fe640 1 -- 172.21.15.114:0/1689968750 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fb1a80079b0 con 0x7fb1b4108b90 2024-05-27T07:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.646+0000 7fb1bfa20740 1 -- start start 2024-05-27T07:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1bfa20740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x5648761883d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1bfa20740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dbe2f0 0x564876188910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1bfa20740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dc64d0 0x564876188e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1bfa20740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5648760fac30 con 0x564875dc64d0 2024-05-27T07:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1bfa20740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5648760ea7d0 con 0x564875dbe2f0 2024-05-27T07:50:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1bfa20740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5648760ee810 con 0x564875de71b0 2024-05-27T07:50:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1b1d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dc64d0 0x564876188e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1b2d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x5648761883d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1b2d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x5648761883d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:43376/0 (socket says 172.21.15.114:43376) 2024-05-27T07:50:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1b1d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dc64d0 0x564876188e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60878/0 (socket says 172.21.15.114:60878) 2024-05-27T07:50:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1b2d9d640 1 -- 172.21.15.114:0/2665947582 learned_addr learned my addr 172.21.15.114:0/2665947582 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1b259c640 1 --2- 172.21.15.114:0/2665947582 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dbe2f0 0x564876188910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1b2d9d640 1 -- 172.21.15.114:0/2665947582 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dbe2f0 msgr2=0x564876188910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1b2d9d640 1 --2- 172.21.15.114:0/2665947582 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dbe2f0 0x564876188910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1b2d9d640 1 -- 172.21.15.114:0/2665947582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dc64d0 msgr2=0x564876188e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1b2d9d640 1 --2- 172.21.15.114:0/2665947582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dc64d0 0x564876188e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1b2d9d640 1 -- 172.21.15.114:0/2665947582 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564875dff210 con 0x564875de71b0 2024-05-27T07:50:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1a37fe640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb19c0b2950 0x7fb19c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.647+0000 7fb1b259c640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb19c0b2950 0x7fb19c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.648+0000 7fb1b2d9d640 1 --2- 172.21.15.114:0/2665947582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x5648761883d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac010480 tx=0x7fb1ac009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=708e3e925324c98c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.648+0000 7fb1a37fe640 1 -- 172.21.15.114:0/1689968750 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fb1a80e2b30 con 0x7fb1b4108b90 2024-05-27T07:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.648+0000 7fb1a0ff9640 1 -- 172.21.15.114:0/2665947582 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac011760 con 0x564875de71b0 2024-05-27T07:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.648+0000 7fb1a0ff9640 1 -- 172.21.15.114:0/2665947582 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1ac011db0 con 0x564875de71b0 2024-05-27T07:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.648+0000 7fb1a0ff9640 1 -- 172.21.15.114:0/2665947582 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac01a380 con 0x564875de71b0 2024-05-27T07:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.649+0000 7fb1bfa20740 1 -- 172.21.15.114:0/2665947582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 msgr2=0x5648761883d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.649+0000 7fb1b259c640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb19c0b2950 0x7fb19c0b4e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4000c00 tx=0x7fb1a4031400 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.649+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/2665947582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x5648761883d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac010480 tx=0x7fb1ac009f90 comp rx=0 tx=0).stop 2024-05-27T07:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.649+0000 7fb1bfa20740 1 -- 172.21.15.114:0/2665947582 shutdown_connections 2024-05-27T07:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.649+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/2665947582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dc64d0 0x564876188e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.649+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/2665947582 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dbe2f0 0x564876188910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.649+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/2665947582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x5648761883d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.649+0000 7fb1bfa20740 1 -- 172.21.15.114:0/2665947582 >> 172.21.15.114:0/2665947582 conn(0x564875ddb300 msgr2=0x564876184b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.649+0000 7fb1bfa20740 1 -- 172.21.15.114:0/2665947582 shutdown_connections 2024-05-27T07:50:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.649+0000 7fb1bfa20740 1 -- 172.21.15.114:0/2665947582 wait complete. 2024-05-27T07:50:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.649+0000 7fb1bfa20740 1 -- start start 2024-05-27T07:50:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1bfa20740 1 Processor -- start 2024-05-27T07:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1bfa20740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dbe2f0 0x564876194490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1bfa20740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dc64d0 0x5648761949d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1bfa20740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x5648761929a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1bfa20740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564875dd0790 con 0x564875dbe2f0 2024-05-27T07:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1bfa20740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5648760a6890 con 0x564875dc64d0 2024-05-27T07:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1bfa20740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564876195500 con 0x564875de71b0 2024-05-27T07:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1bfa20740 10 client.0 _ll_register_callbacks cb 0x564875dfea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1b259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dbe2f0 0x564876194490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1b259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dbe2f0 0x564876194490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60894/0 (socket says 172.21.15.114:60894) 2024-05-27T07:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1b259c640 1 -- 172.21.15.114:0/2830598054 learned_addr learned my addr 172.21.15.114:0/2830598054 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1b1d9b640 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dc64d0 0x5648761949d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1b2d9d640 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x5648761929a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1b259c640 1 -- 172.21.15.114:0/2830598054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 msgr2=0x5648761929a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1b259c640 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x5648761929a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1b259c640 1 -- 172.21.15.114:0/2830598054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dc64d0 msgr2=0x5648761949d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1b259c640 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dc64d0 0x5648761949d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1b259c640 1 -- 172.21.15.114:0/2830598054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564875ecb620 con 0x564875dbe2f0 2024-05-27T07:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.650+0000 7fb1b1d9b640 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dc64d0 0x5648761949d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb1b259c640 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dbe2f0 0x564876194490 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4009900 tx=0x7fb1a4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9822fb6860c58d4e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb199ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb199ffb640 1 -- 172.21.15.114:0/2830598054 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a404d070 con 0x564875dbe2f0 2024-05-27T07:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb1bfa20740 1 -- 172.21.15.114:0/2830598054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1ac011db0 con 0x564875dbe2f0 2024-05-27T07:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb199ffb640 1 -- 172.21.15.114:0/2830598054 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fb1a4004250 con 0x564875dbe2f0 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb199ffb640 1 -- 172.21.15.114:0/2830598054 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7fb1a4059050 con 0x564875dbe2f0 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb1bfa20740 10 client.17913 fetch_fsmap learned FSMap version 60 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb1bfa20740 1 -- 172.21.15.114:0/2830598054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x5648761a18a0 con 0x564875dbe2f0 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb199ffb640 1 -- 172.21.15.114:0/2830598054 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1ac011db0 con 0x564875dbe2f0 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb199ffb640 1 -- 172.21.15.114:0/2830598054 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 60) ==== 6515+0+0 (secure 0 0 0) 0x7fb1a4003820 con 0x564875dbe2f0 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb1bfa20740 10 client.17913 fetch_fsmap finished waiting for FSMap version 60 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb1bfa20740 10 client.17913 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb1bfa20740 20 client.17913 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564875da5d80 0x5648761a5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb1bfa20740 4 client.17913 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.651+0000 7fb1bfa20740 1 -- 172.21.15.114:0/2830598054 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648760fcb80 con 0x564875da5d80 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.652+0000 7fb1b1d9b640 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564875da5d80 0x5648761a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.652+0000 7fb199ffb640 10 client.17913 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.652+0000 7fb1b1d9b640 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564875da5d80 0x5648761a5000 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99177ac2449d6c4d server_cookie=662f96fa64054353 in_seq=0 out_seq=0 2024-05-27T07:50:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb199ffb640 1 -- 172.21.15.114:0/2830598054 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5648760fcb80 con 0x564875da5d80 2024-05-27T07:50:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb199ffb640 10 client.17913 handle_command_reply: tid=0 2024-05-27T07:50:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb1bfa20740 1 client.17913 shutdown 2024-05-27T07:50:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb1bfa20740 1 -- 172.21.15.114:0/2830598054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dbe2f0 msgr2=0x564876194490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dbe2f0 0x564876194490 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4009900 tx=0x7fb1a4009260 comp rx=0 tx=0).stop 2024-05-27T07:50:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb1bfa20740 1 -- 172.21.15.114:0/2830598054 shutdown_connections 2024-05-27T07:50:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564875da5d80 0x5648761a5000 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x5648761929a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dc64d0 0x5648761949d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/2830598054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dbe2f0 0x564876194490 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb199ffb640 0 client.17913 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb1bfa20740 1 -- 172.21.15.114:0/2830598054 >> 172.21.15.114:0/2830598054 conn(0x564875ddb300 msgr2=0x564876188200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb1bfa20740 1 -- 172.21.15.114:0/2830598054 shutdown_connections 2024-05-27T07:50:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb1bfa20740 1 -- 172.21.15.114:0/2830598054 wait complete. 2024-05-27T07:50:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.653+0000 7fb1bfa20740 20 client.17913 trim_cache size 0 max 16384 2024-05-27T07:50:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.669+0000 7fb1bfa20740 1 Processor -- start 2024-05-27T07:50:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1bfa20740 1 -- start start 2024-05-27T07:50:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1bfa20740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875dbe2f0 0x56487618c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1bfa20740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dc64d0 0x56487619a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1bfa20740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875de71b0 0x56487619acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1bfa20740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41b1c40 con 0x564875de71b0 2024-05-27T07:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1bfa20740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41af3e0 con 0x564875dc64d0 2024-05-27T07:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1bfa20740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41b1570 con 0x564875dbe2f0 2024-05-27T07:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1b2d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875de71b0 0x56487619acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1b2d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875de71b0 0x56487619acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60904/0 (socket says 172.21.15.114:60904) 2024-05-27T07:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1b2d9d640 1 -- 172.21.15.114:0/136734856 learned_addr learned my addr 172.21.15.114:0/136734856 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1b259c640 1 --2- 172.21.15.114:0/136734856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875dbe2f0 0x56487618c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1b2d9d640 1 -- 172.21.15.114:0/136734856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875dbe2f0 msgr2=0x56487618c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1b2d9d640 1 --2- 172.21.15.114:0/136734856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875dbe2f0 0x56487618c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1b2d9d640 1 -- 172.21.15.114:0/136734856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dc64d0 msgr2=0x56487619a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1b1d9b640 1 --2- 172.21.15.114:0/136734856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dc64d0 0x56487619a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1b2d9d640 1 --2- 172.21.15.114:0/136734856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dc64d0 0x56487619a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1b2d9d640 1 -- 172.21.15.114:0/136734856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564876190ef0 con 0x564875de71b0 2024-05-27T07:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1b1d9b640 1 --2- 172.21.15.114:0/136734856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dc64d0 0x56487619a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:50:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb1b2d9d640 1 --2- 172.21.15.114:0/136734856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875de71b0 0x56487619acd0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac010480 tx=0x7fb1ac01cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149c25a4dc66a77d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.670+0000 7fb19bfff640 1 -- 172.21.15.114:0/136734856 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac02b070 con 0x564875de71b0 2024-05-27T07:50:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.671+0000 7fb19bfff640 1 -- 172.21.15.114:0/136734856 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1ac01c930 con 0x564875de71b0 2024-05-27T07:50:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.671+0000 7fb19bfff640 1 -- 172.21.15.114:0/136734856 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac011c80 con 0x564875de71b0 2024-05-27T07:50:27.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.671+0000 7fb1bfa20740 1 -- 172.21.15.114:0/136734856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875de71b0 msgr2=0x56487619acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:27.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.671+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/136734856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875de71b0 0x56487619acd0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac010480 tx=0x7fb1ac01cb30 comp rx=0 tx=0).stop 2024-05-27T07:50:27.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.671+0000 7fb1bfa20740 1 -- 172.21.15.114:0/136734856 shutdown_connections 2024-05-27T07:50:27.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.671+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/136734856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875de71b0 0x56487619acd0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.671+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/136734856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dc64d0 0x56487619a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.671+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/136734856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875dbe2f0 0x56487618c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.671+0000 7fb1bfa20740 1 -- 172.21.15.114:0/136734856 >> 172.21.15.114:0/136734856 conn(0x564875ddb300 msgr2=0x56487618c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:27.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.671+0000 7fb1bfa20740 1 -- 172.21.15.114:0/136734856 shutdown_connections 2024-05-27T07:50:27.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.671+0000 7fb1bfa20740 1 -- 172.21.15.114:0/136734856 wait complete. 2024-05-27T07:50:27.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1bfa20740 1 -- start start 2024-05-27T07:50:27.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1bfa20740 1 Processor -- start 2024-05-27T07:50:27.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1bfa20740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dbe2f0 0x56487619af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1bfa20740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dc64d0 0x564876199320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1bfa20740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x564876199920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1bfa20740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41b0ea0 con 0x564875dc64d0 2024-05-27T07:50:27.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1bfa20740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41afa30 con 0x564875dbe2f0 2024-05-27T07:50:27.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1bfa20740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41b0100 con 0x564875de71b0 2024-05-27T07:50:27.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1bfa20740 10 client.0 _ll_register_callbacks cb 0x7fb1a4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:27.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1b1d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dc64d0 0x564876199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1b1d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dc64d0 0x564876199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60918/0 (socket says 172.21.15.114:60918) 2024-05-27T07:50:27.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1b1d9b640 1 -- 172.21.15.114:0/884679143 learned_addr learned my addr 172.21.15.114:0/884679143 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:27.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1b2d9d640 1 --2- 172.21.15.114:0/884679143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x564876199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1b259c640 1 --2- 172.21.15.114:0/884679143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dbe2f0 0x56487619af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1b1d9b640 1 -- 172.21.15.114:0/884679143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 msgr2=0x564876199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:27.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1b1d9b640 1 --2- 172.21.15.114:0/884679143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x564876199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1b1d9b640 1 -- 172.21.15.114:0/884679143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dbe2f0 msgr2=0x56487619af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:27.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1b1d9b640 1 --2- 172.21.15.114:0/884679143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dbe2f0 0x56487619af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:27.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.672+0000 7fb1b1d9b640 1 -- 172.21.15.114:0/884679143 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56487619a100 con 0x564875dc64d0 2024-05-27T07:50:27.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb1b1d9b640 1 --2- 172.21.15.114:0/884679143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dc64d0 0x564876199320 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fb1a80a9990 tx=0x7fb1a80e6680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30ddc49f9bee3f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb199ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:50:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb199ffb640 1 -- 172.21.15.114:0/884679143 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a801d040 con 0x564875dc64d0 2024-05-27T07:50:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb199ffb640 1 -- 172.21.15.114:0/884679143 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fb1a80a5c90 con 0x564875dc64d0 2024-05-27T07:50:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb1bfa20740 1 -- 172.21.15.114:0/884679143 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1ac01c930 con 0x564875dc64d0 2024-05-27T07:50:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb199ffb640 1 -- 172.21.15.114:0/884679143 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7fb1a80a9350 con 0x564875dc64d0 2024-05-27T07:50:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb1bfa20740 10 client.17919 fetch_fsmap learned FSMap version 60 2024-05-27T07:50:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb1bfa20740 1 -- 172.21.15.114:0/884679143 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x56487620c670 con 0x564875dc64d0 2024-05-27T07:50:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb199ffb640 1 -- 172.21.15.114:0/884679143 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1ac01c930 con 0x564875dc64d0 2024-05-27T07:50:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb199ffb640 1 -- 172.21.15.114:0/884679143 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 60) ==== 6515+0+0 (secure 0 0 0) 0x7fb1a80b23d0 con 0x564875dc64d0 2024-05-27T07:50:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb1bfa20740 10 client.17919 fetch_fsmap finished waiting for FSMap version 60 2024-05-27T07:50:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb1bfa20740 10 client.17919 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb1bfa20740 20 client.17919 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/884679143 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564875d7b1c0 0x56487620edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb1bfa20740 4 client.17919 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:50:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb1bfa20740 1 -- 172.21.15.114:0/884679143 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5648760f6360 con 0x564875d7b1c0 2024-05-27T07:50:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.673+0000 7fb1b259c640 1 --2- 172.21.15.114:0/884679143 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564875d7b1c0 0x56487620edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.674+0000 7fb1b259c640 1 --2- 172.21.15.114:0/884679143 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564875d7b1c0 0x56487620edb0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=142d2cc672f8db3a server_cookie=37e754d3894947e in_seq=0 out_seq=0 2024-05-27T07:50:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:27.674+0000 7fb199ffb640 10 client.17919 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:27.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:27 smithi163 ceph-mon[36451]: pgmap v354: 129 pgs: 129 active+clean; 77 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 267 op/s 2024-05-27T07:50:27.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:27 smithi186 ceph-mon[36647]: pgmap v354: 129 pgs: 129 active+clean; 77 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 267 op/s 2024-05-27T07:50:28.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:27 smithi114 ceph-mon[32038]: pgmap v354: 129 pgs: 129 active+clean; 77 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 267 op/s 2024-05-27T07:50:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb199ffb640 1 -- 172.21.15.114:0/884679143 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5648760f6360 con 0x564875d7b1c0 2024-05-27T07:50:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb199ffb640 10 client.17919 handle_command_reply: tid=0 2024-05-27T07:50:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb1bfa20740 1 client.17919 shutdown 2024-05-27T07:50:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb1bfa20740 1 -- 172.21.15.114:0/884679143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dc64d0 msgr2=0x564876199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/884679143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dc64d0 0x564876199320 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fb1a80a9990 tx=0x7fb1a80e6680 comp rx=0 tx=0).stop 2024-05-27T07:50:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb1bfa20740 1 -- 172.21.15.114:0/884679143 shutdown_connections 2024-05-27T07:50:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/884679143 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564875d7b1c0 0x56487620edb0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/884679143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564875de71b0 0x564876199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/884679143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564875dc64d0 0x564876199320 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb199ffb640 0 client.17919 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb1bfa20740 1 --2- 172.21.15.114:0/884679143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564875dbe2f0 0x56487619af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb1bfa20740 1 -- 172.21.15.114:0/884679143 >> 172.21.15.114:0/884679143 conn(0x564875ddb300 msgr2=0x564876193e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb1bfa20740 1 -- 172.21.15.114:0/884679143 shutdown_connections 2024-05-27T07:50:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.075+0000 7fb1bfa20740 1 -- 172.21.15.114:0/884679143 wait complete. 2024-05-27T07:50:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.076+0000 7fb1bfa20740 20 client.17919 trim_cache size 0 max 16384 2024-05-27T07:50:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:50:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:50:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:50:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.076+0000 7fb1b8a9a640 1 -- 172.21.15.114:0/1689968750 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb19c0b2950 msgr2=0x7fb19c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.076+0000 7fb1b8a9a640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb19c0b2950 0x7fb19c0b4e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4000c00 tx=0x7fb1a4031400 comp rx=0 tx=0).stop 2024-05-27T07:50:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.076+0000 7fb1b8a9a640 1 -- 172.21.15.114:0/1689968750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1b4108b90 msgr2=0x7fb1b419bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.076+0000 7fb1b8a9a640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1b4108b90 0x7fb1b419bc00 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800c970 tx=0x7fb1a800ce30 comp rx=0 tx=0).stop 2024-05-27T07:50:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.076+0000 7fb1b8a9a640 1 -- 172.21.15.114:0/1689968750 shutdown_connections 2024-05-27T07:50:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.076+0000 7fb1b8a9a640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb19c0b2950 0x7fb19c0b4e60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.076+0000 7fb1b8a9a640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1b4109030 0x7fb1b41a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.076+0000 7fb1b8a9a640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1b4108b90 0x7fb1b419bc00 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.076+0000 7fb1b8a9a640 1 --2- 172.21.15.114:0/1689968750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1b41032e0 0x7fb1b419b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.076+0000 7fb1b8a9a640 1 -- 172.21.15.114:0/1689968750 >> 172.21.15.114:0/1689968750 conn(0x7fb1b40782d0 msgr2=0x7fb1b4100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.077+0000 7fb1b8a9a640 1 -- 172.21.15.114:0/1689968750 shutdown_connections 2024-05-27T07:50:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:28.077+0000 7fb1b8a9a640 1 -- 172.21.15.114:0/1689968750 wait complete. 2024-05-27T07:50:28.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1058939 10.19 MB/sec execute 400 sec latency 111.972 ms 2024-05-27T07:50:29.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1060493 10.18 MB/sec execute 401 sec latency 55.833 ms 2024-05-27T07:50:29.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:29 smithi163 ceph-mon[36451]: pgmap v355: 129 pgs: 129 active+clean; 77 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 219 op/s 2024-05-27T07:50:29.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:29 smithi186 ceph-mon[36647]: pgmap v355: 129 pgs: 129 active+clean; 77 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 219 op/s 2024-05-27T07:50:30.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:29 smithi114 ceph-mon[32038]: pgmap v355: 129 pgs: 129 active+clean; 77 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 219 op/s 2024-05-27T07:50:30.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1062453 10.18 MB/sec execute 402 sec latency 49.933 ms 2024-05-27T07:50:30.479 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:30 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:50:30.479 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:30 smithi186 systemd[1]: Stopped Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:50:30.479 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:30 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Start request repeated too quickly. 2024-05-27T07:50:30.479 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:30 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:50:30.479 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 27 07:50:30 smithi186 systemd[1]: Failed to start Ceph mds.c for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:50:31.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1063635 10.17 MB/sec execute 403 sec latency 80.282 ms 2024-05-27T07:50:31.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:31 smithi163 ceph-mon[36451]: pgmap v356: 129 pgs: 129 active+clean; 70 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 292 op/s 2024-05-27T07:50:31.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:31 smithi186 ceph-mon[36647]: pgmap v356: 129 pgs: 129 active+clean; 70 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 292 op/s 2024-05-27T07:50:32.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:31 smithi114 ceph-mon[32038]: pgmap v356: 129 pgs: 129 active+clean; 70 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 292 op/s 2024-05-27T07:50:32.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1065808 10.18 MB/sec execute 404 sec latency 46.681 ms 2024-05-27T07:50:33.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1067141 10.17 MB/sec execute 405 sec latency 24.414 ms 2024-05-27T07:50:33.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:33 smithi163 ceph-mon[36451]: pgmap v357: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 314 op/s 2024-05-27T07:50:33.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:33 smithi186 ceph-mon[36647]: pgmap v357: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 314 op/s 2024-05-27T07:50:34.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:33 smithi114 ceph-mon[32038]: pgmap v357: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 314 op/s 2024-05-27T07:50:34.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1069355 10.17 MB/sec execute 406 sec latency 13.290 ms 2024-05-27T07:50:34.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:34 smithi163 ceph-mon[36451]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:50:34.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:34 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:50:34.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:34 smithi186 ceph-mon[36647]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:50:34.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:34 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:50:35.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:34 smithi114 ceph-mon[32038]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:50:35.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:34 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:50:35.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1070518 10.16 MB/sec execute 407 sec latency 47.272 ms 2024-05-27T07:50:35.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:35 smithi163 ceph-mon[36451]: pgmap v358: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 2.4 MiB/s wr, 274 op/s 2024-05-27T07:50:35.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:35 smithi186 ceph-mon[36647]: pgmap v358: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 2.4 MiB/s wr, 274 op/s 2024-05-27T07:50:36.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:35 smithi114 ceph-mon[32038]: pgmap v358: 129 pgs: 129 active+clean; 66 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 2.4 MiB/s wr, 274 op/s 2024-05-27T07:50:36.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1072842 10.17 MB/sec execute 408 sec latency 59.934 ms 2024-05-27T07:50:37.286 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1074166 10.16 MB/sec execute 409 sec latency 32.000 ms 2024-05-27T07:50:37.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:37 smithi163 ceph-mon[36451]: pgmap v359: 129 pgs: 129 active+clean; 71 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 364 op/s 2024-05-27T07:50:37.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:37 smithi186 ceph-mon[36647]: pgmap v359: 129 pgs: 129 active+clean; 71 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 364 op/s 2024-05-27T07:50:38.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:37 smithi114 ceph-mon[32038]: pgmap v359: 129 pgs: 129 active+clean; 71 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 364 op/s 2024-05-27T07:50:38.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1076420 10.17 MB/sec execute 410 sec latency 14.149 ms 2024-05-27T07:50:39.286 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1077762 10.16 MB/sec execute 411 sec latency 26.032 ms 2024-05-27T07:50:39.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:39 smithi163 ceph-mon[36451]: pgmap v360: 129 pgs: 129 active+clean; 71 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 275 op/s 2024-05-27T07:50:39.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:39 smithi186 ceph-mon[36647]: pgmap v360: 129 pgs: 129 active+clean; 71 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 275 op/s 2024-05-27T07:50:40.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:39 smithi114 ceph-mon[32038]: pgmap v360: 129 pgs: 129 active+clean; 71 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 275 op/s 2024-05-27T07:50:40.286 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1079951 10.17 MB/sec execute 412 sec latency 18.313 ms 2024-05-27T07:50:41.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.229+0000 7f44f0114640 1 -- 172.21.15.114:0/1961170386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e811a8e0 msgr2=0x7f44e811cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.230+0000 7f44eceae640 1 -- 172.21.15.114:0/1961170386 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e002e320 con 0x7f44e811a8e0 2024-05-27T07:50:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.230+0000 7f44f0114640 1 --2- 172.21.15.114:0/1961170386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e811a8e0 0x7f44e811cd00 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009ca0 tx=0x7f44e002a970 comp rx=0 tx=0).stop 2024-05-27T07:50:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.235+0000 7f44f0114640 1 -- 172.21.15.114:0/1961170386 shutdown_connections 2024-05-27T07:50:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.235+0000 7f44f0114640 1 --2- 172.21.15.114:0/1961170386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44e811d240 0x7f44e811f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.235+0000 7f44f0114640 1 --2- 172.21.15.114:0/1961170386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e811a8e0 0x7f44e811cd00 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.235+0000 7f44f0114640 1 --2- 172.21.15.114:0/1961170386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44e810a1f0 0x7f44e810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.235+0000 7f44f0114640 1 -- 172.21.15.114:0/1961170386 >> 172.21.15.114:0/1961170386 conn(0x7f44e806e8f0 msgr2=0x7f44e806ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.237+0000 7f44f0114640 1 -- 172.21.15.114:0/1961170386 shutdown_connections 2024-05-27T07:50:41.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.239+0000 7f44f0114640 1 -- 172.21.15.114:0/1961170386 wait complete. 2024-05-27T07:50:41.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.239+0000 7f44f0114640 1 Processor -- start 2024-05-27T07:50:41.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.240+0000 7f44f0114640 1 -- start start 2024-05-27T07:50:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.241+0000 7f44f0114640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e810a1f0 0x7f44e81a9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.241+0000 7f44f0114640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44e811a8e0 0x7f44e81a9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.241+0000 7f44f0114640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44e811d240 0x7f44e81ade70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.241+0000 7f44f0114640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44e8122c70 con 0x7f44e810a1f0 2024-05-27T07:50:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.241+0000 7f44f0114640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f44e8122af0 con 0x7f44e811a8e0 2024-05-27T07:50:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.241+0000 7f44f0114640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f44e8122df0 con 0x7f44e811d240 2024-05-27T07:50:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.243+0000 7f44ed6af640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44e811a8e0 0x7f44e81a9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.243+0000 7f44ed6af640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44e811a8e0 0x7f44e81a9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:60230/0 (socket says 172.21.15.114:60230) 2024-05-27T07:50:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.243+0000 7f44ed6af640 1 -- 172.21.15.114:0/174438250 learned_addr learned my addr 172.21.15.114:0/174438250 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.246+0000 7f44edeb0640 1 --2- 172.21.15.114:0/174438250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e810a1f0 0x7f44e81a9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.246+0000 7f44ed6af640 1 -- 172.21.15.114:0/174438250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44e811d240 msgr2=0x7f44e81ade70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.246+0000 7f44ed6af640 1 --2- 172.21.15.114:0/174438250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44e811d240 0x7f44e81ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.246+0000 7f44ed6af640 1 -- 172.21.15.114:0/174438250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e810a1f0 msgr2=0x7f44e81a9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.246+0000 7f44ed6af640 1 --2- 172.21.15.114:0/174438250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e810a1f0 0x7f44e81a9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.246+0000 7f44ed6af640 1 -- 172.21.15.114:0/174438250 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e81ae4b0 con 0x7f44e811a8e0 2024-05-27T07:50:41.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.252+0000 7f44ed6af640 1 --2- 172.21.15.114:0/174438250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44e811a8e0 0x7f44e81a9bb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f44e0003f90 tx=0x7f44e00097e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:41.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.257+0000 7f44deffd640 1 -- 172.21.15.114:0/174438250 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e0047070 con 0x7f44e811a8e0 2024-05-27T07:50:41.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.257+0000 7f44deffd640 1 -- 172.21.15.114:0/174438250 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44e0030b40 con 0x7f44e811a8e0 2024-05-27T07:50:41.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.257+0000 7f44deffd640 1 -- 172.21.15.114:0/174438250 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e0034070 con 0x7f44e811a8e0 2024-05-27T07:50:41.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.257+0000 7f44f0114640 1 -- 172.21.15.114:0/174438250 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44e81ae740 con 0x7f44e811a8e0 2024-05-27T07:50:41.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.257+0000 7f44f0114640 1 -- 172.21.15.114:0/174438250 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44e81aebe0 con 0x7f44e811a8e0 2024-05-27T07:50:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.262+0000 7f44f6fc6740 1 Processor -- start 2024-05-27T07:50:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.263+0000 7f44f6fc6740 1 -- start start 2024-05-27T07:50:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.263+0000 7f44f6fc6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03da1b0 0x55acf077b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.263+0000 7f44f6fc6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf03b12f0 0x55acf077b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.263+0000 7f44f6fc6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf03b94d0 0x55acf077be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.263+0000 7f44f6fc6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55acf06edc30 con 0x55acf03da1b0 2024-05-27T07:50:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.263+0000 7f44f6fc6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55acf06dd7d0 con 0x55acf03b12f0 2024-05-27T07:50:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.263+0000 7f44f6fc6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55acf06e1810 con 0x55acf03b94d0 2024-05-27T07:50:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.264+0000 7f44ee6b1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03da1b0 0x55acf077b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.264+0000 7f44ee6b1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03da1b0 0x55acf077b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34530/0 (socket says 172.21.15.114:34530) 2024-05-27T07:50:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.264+0000 7f44ee6b1640 1 -- 172.21.15.114:0/2880023229 learned_addr learned my addr 172.21.15.114:0/2880023229 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.264+0000 7f44ee6b1640 1 -- 172.21.15.114:0/2880023229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf03b94d0 msgr2=0x55acf077be50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:50:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.264+0000 7f44ee6b1640 1 --2- 172.21.15.114:0/2880023229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf03b94d0 0x55acf077be50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.264+0000 7f44ee6b1640 1 -- 172.21.15.114:0/2880023229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf03b12f0 msgr2=0x55acf077b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.264+0000 7f44ee6b1640 1 --2- 172.21.15.114:0/2880023229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf03b12f0 0x55acf077b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.264+0000 7f44ee6b1640 1 -- 172.21.15.114:0/2880023229 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acf03f2210 con 0x55acf03da1b0 2024-05-27T07:50:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.264+0000 7f44ee6b1640 1 --2- 172.21.15.114:0/2880023229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03da1b0 0x55acf077b3d0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f44e400a8c0 tx=0x7f44e400ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a05ca7b5c4c1352 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.266+0000 7f44d3fff640 1 -- 172.21.15.114:0/2880023229 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e400ccc0 con 0x55acf03da1b0 2024-05-27T07:50:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.266+0000 7f44d3fff640 1 -- 172.21.15.114:0/2880023229 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44e400f900 con 0x55acf03da1b0 2024-05-27T07:50:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.266+0000 7f44d3fff640 1 -- 172.21.15.114:0/2880023229 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4018920 con 0x55acf03da1b0 2024-05-27T07:50:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.267+0000 7f44f6fc6740 1 -- 172.21.15.114:0/2880023229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03da1b0 msgr2=0x55acf077b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.267+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/2880023229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03da1b0 0x55acf077b3d0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f44e400a8c0 tx=0x7f44e400ad80 comp rx=0 tx=0).stop 2024-05-27T07:50:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.267+0000 7f44f6fc6740 1 -- 172.21.15.114:0/2880023229 shutdown_connections 2024-05-27T07:50:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.267+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/2880023229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf03b94d0 0x55acf077be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.267+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/2880023229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf03b12f0 0x55acf077b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.267+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/2880023229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03da1b0 0x55acf077b3d0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.267+0000 7f44f6fc6740 1 -- 172.21.15.114:0/2880023229 >> 172.21.15.114:0/2880023229 conn(0x55acf03ce300 msgr2=0x55acf0777b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:41.269 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.269+0000 7f44deffd640 1 -- 172.21.15.114:0/174438250 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f44e0037020 con 0x7f44e811a8e0 2024-05-27T07:50:41.269 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.270+0000 7f44f6fc6740 1 -- 172.21.15.114:0/2880023229 shutdown_connections 2024-05-27T07:50:41.269 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.270+0000 7f44f6fc6740 1 -- 172.21.15.114:0/2880023229 wait complete. 2024-05-27T07:50:41.269 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.270+0000 7f44deffd640 1 --2- 172.21.15.114:0/174438250 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f44d40b2a80 0x7f44d40b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.271+0000 7f44deffd640 1 -- 172.21.15.114:0/174438250 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7f44e0103270 con 0x7f44e811a8e0 2024-05-27T07:50:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.271+0000 7f44f6fc6740 1 -- start start 2024-05-27T07:50:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.271+0000 7f44f6fc6740 1 Processor -- start 2024-05-27T07:50:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.271+0000 7f44edeb0640 1 --2- 172.21.15.114:0/174438250 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f44d40b2a80 0x7f44d40b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.272+0000 7f44f6fc6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf03b12f0 0x55acf07875e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.272+0000 7f44f6fc6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf03da1b0 0x55acf0787b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.272+0000 7f44f6fc6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf0398d80 0x55acf07859f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.272+0000 7f44f6fc6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55acf03c3790 con 0x55acf0398d80 2024-05-27T07:50:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.272+0000 7f44ed6af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf03da1b0 0x55acf0787b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.272+0000 7f44ed6af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf03da1b0 0x55acf0787b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54846/0 (socket says 172.21.15.114:54846) 2024-05-27T07:50:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.272+0000 7f44ed6af640 1 -- 172.21.15.114:0/3863115832 learned_addr learned my addr 172.21.15.114:0/3863115832 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.272+0000 7f44f6fc6740 1 -- 172.21.15.114:0/3863115832 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55acf0699890 con 0x55acf03b12f0 2024-05-27T07:50:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.272+0000 7f44f6fc6740 1 -- 172.21.15.114:0/3863115832 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55acf07884a0 con 0x55acf03da1b0 2024-05-27T07:50:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.272+0000 7f44edeb0640 1 --2- 172.21.15.114:0/3863115832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf03b12f0 0x55acf07875e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.273+0000 7f44f6fc6740 10 client.0 _ll_register_callbacks cb 0x55acf03f1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.273+0000 7f44ee6b1640 1 --2- 172.21.15.114:0/3863115832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf0398d80 0x55acf07859f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.273+0000 7f44ed6af640 1 -- 172.21.15.114:0/3863115832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf03b12f0 msgr2=0x55acf07875e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.273+0000 7f44ed6af640 1 --2- 172.21.15.114:0/3863115832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf03b12f0 0x55acf07875e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.273+0000 7f44ed6af640 1 -- 172.21.15.114:0/3863115832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf0398d80 msgr2=0x55acf07859f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.273+0000 7f44ed6af640 1 --2- 172.21.15.114:0/3863115832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf0398d80 0x55acf07859f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.273+0000 7f44ed6af640 1 -- 172.21.15.114:0/3863115832 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acf04be620 con 0x55acf03da1b0 2024-05-27T07:50:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.273+0000 7f44edeb0640 1 --2- 172.21.15.114:0/174438250 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f44d40b2a80 0x7f44d40b4f90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x55acf077c780 tx=0x7f44d80046c0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.273+0000 7f44ed6af640 1 --2- 172.21.15.114:0/3863115832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf03da1b0 0x55acf0787b40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009850 tx=0x7f44e0100ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc9bd3134211d363 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.273+0000 7f44d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.273+0000 7f44d17fa640 1 -- 172.21.15.114:0/3863115832 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e00ca7e0 con 0x55acf03da1b0 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.274+0000 7f44d17fa640 1 -- 172.21.15.114:0/3863115832 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7f44e0004d50 con 0x55acf03da1b0 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.274+0000 7f44f6fc6740 1 -- 172.21.15.114:0/3863115832 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44e400f900 con 0x55acf03da1b0 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.274+0000 7f44d17fa640 1 -- 172.21.15.114:0/3863115832 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f44e00ca0e0 con 0x55acf03da1b0 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.274+0000 7f44f6fc6740 10 client.27499 fetch_fsmap learned FSMap version 61 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.274+0000 7f44f6fc6740 1 -- 172.21.15.114:0/3863115832 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x55acf07949c0 con 0x55acf03da1b0 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.274+0000 7f44d17fa640 1 -- 172.21.15.114:0/3863115832 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44e400f900 con 0x55acf03da1b0 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.274+0000 7f44d17fa640 1 -- 172.21.15.114:0/3863115832 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 61) ==== 5910+0+0 (secure 0 0 0) 0x7f44e0041030 con 0x55acf03da1b0 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.274+0000 7f44f6fc6740 10 client.27499 fetch_fsmap finished waiting for FSMap version 61 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.274+0000 7f44f6fc6740 10 client.27499 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.275+0000 7f44f6fc6740 20 client.27499 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.275+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/3863115832 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55acf036e1c0 0x55acf0797f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.275+0000 7f44f6fc6740 4 client.27499 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:50:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.275+0000 7f44f6fc6740 1 -- 172.21.15.114:0/3863115832 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55acf06efb80 con 0x55acf036e1c0 2024-05-27T07:50:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.275+0000 7f44edeb0640 1 --2- 172.21.15.114:0/3863115832 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55acf036e1c0 0x55acf0797f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.276+0000 7f44edeb0640 1 --2- 172.21.15.114:0/3863115832 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55acf036e1c0 0x55acf0797f80 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1a30087c26682b7 server_cookie=5a0bdc2c1919cd54 in_seq=0 out_seq=0 2024-05-27T07:50:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.276+0000 7f44d17fa640 10 client.27499 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44d17fa640 1 -- 172.21.15.114:0/3863115832 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55acf06efb80 con 0x55acf036e1c0 2024-05-27T07:50:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44d17fa640 10 client.27499 handle_command_reply: tid=0 2024-05-27T07:50:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44f6fc6740 1 client.27499 shutdown 2024-05-27T07:50:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44f6fc6740 1 -- 172.21.15.114:0/3863115832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf03da1b0 msgr2=0x55acf0787b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/3863115832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf03da1b0 0x55acf0787b40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009850 tx=0x7f44e0100ad0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44f6fc6740 1 -- 172.21.15.114:0/3863115832 shutdown_connections 2024-05-27T07:50:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/3863115832 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55acf036e1c0 0x55acf0797f80 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/3863115832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf0398d80 0x55acf07859f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44d17fa640 0 client.27499 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/3863115832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf03da1b0 0x55acf0787b40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/3863115832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf03b12f0 0x55acf07875e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44f6fc6740 1 -- 172.21.15.114:0/3863115832 >> 172.21.15.114:0/3863115832 conn(0x55acf03ce300 msgr2=0x55acf077d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44f6fc6740 1 -- 172.21.15.114:0/3863115832 shutdown_connections 2024-05-27T07:50:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44f6fc6740 1 -- 172.21.15.114:0/3863115832 wait complete. 2024-05-27T07:50:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.277+0000 7f44f6fc6740 20 client.27499 trim_cache size 0 max 16384 2024-05-27T07:50:41.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1080930 10.15 MB/sec execute 413 sec latency 44.188 ms 2024-05-27T07:50:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.294+0000 7f44f6fc6740 1 Processor -- start 2024-05-27T07:50:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.294+0000 7f44f6fc6740 1 -- start start 2024-05-27T07:50:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44f6fc6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf036e1c0 0x55acf077d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44f6fc6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 0x55acf077f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44f6fc6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03b12f0 0x55acf077f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44f6fc6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc0339d0 con 0x55acf03b12f0 2024-05-27T07:50:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44f6fc6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc032160 con 0x55acf036e1c0 2024-05-27T07:50:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44f6fc6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc02ffd0 con 0x55acf0398d80 2024-05-27T07:50:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44edeb0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf036e1c0 0x55acf077d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44ed6af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 0x55acf077f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44ee6b1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03b12f0 0x55acf077f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44ed6af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 0x55acf077f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54852/0 (socket says 172.21.15.114:54852) 2024-05-27T07:50:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44ed6af640 1 -- 172.21.15.114:0/4064483394 learned_addr learned my addr 172.21.15.114:0/4064483394 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44ed6af640 1 -- 172.21.15.114:0/4064483394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf036e1c0 msgr2=0x55acf077d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44ed6af640 1 --2- 172.21.15.114:0/4064483394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf036e1c0 0x55acf077d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44ed6af640 1 -- 172.21.15.114:0/4064483394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03b12f0 msgr2=0x55acf077f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44ed6af640 1 --2- 172.21.15.114:0/4064483394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03b12f0 0x55acf077f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44ed6af640 1 -- 172.21.15.114:0/4064483394 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acf077c7e0 con 0x55acf0398d80 2024-05-27T07:50:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.295+0000 7f44ee6b1640 1 --2- 172.21.15.114:0/4064483394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03b12f0 0x55acf077f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.296+0000 7f44ed6af640 1 --2- 172.21.15.114:0/4064483394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 0x55acf077f080 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f44e0005e30 tx=0x7f44e0004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43e10821c636ca68 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.296+0000 7f44d37fe640 1 -- 172.21.15.114:0/4064483394 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e00c9850 con 0x55acf0398d80 2024-05-27T07:50:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.296+0000 7f44d37fe640 1 -- 172.21.15.114:0/4064483394 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44e00ca7e0 con 0x55acf0398d80 2024-05-27T07:50:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.296+0000 7f44d37fe640 1 -- 172.21.15.114:0/4064483394 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e00c5a50 con 0x55acf0398d80 2024-05-27T07:50:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.297+0000 7f44f6fc6740 1 -- 172.21.15.114:0/4064483394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 msgr2=0x55acf077f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.297+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/4064483394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 0x55acf077f080 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f44e0005e30 tx=0x7f44e0004040 comp rx=0 tx=0).stop 2024-05-27T07:50:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.297+0000 7f44f6fc6740 1 -- 172.21.15.114:0/4064483394 shutdown_connections 2024-05-27T07:50:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.297+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/4064483394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03b12f0 0x55acf077f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.297+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/4064483394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 0x55acf077f080 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.297+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/4064483394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf036e1c0 0x55acf077d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.297+0000 7f44f6fc6740 1 -- 172.21.15.114:0/4064483394 >> 172.21.15.114:0/4064483394 conn(0x55acf03ce300 msgr2=0x55acf077d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.297+0000 7f44f6fc6740 1 -- 172.21.15.114:0/4064483394 shutdown_connections 2024-05-27T07:50:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.297+0000 7f44f6fc6740 1 -- 172.21.15.114:0/4064483394 wait complete. 2024-05-27T07:50:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.297+0000 7f44f6fc6740 1 -- start start 2024-05-27T07:50:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.298+0000 7f44f6fc6740 1 Processor -- start 2024-05-27T07:50:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.298+0000 7f44f6fc6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf036e1c0 0x55acf078ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.298+0000 7f44f6fc6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 0x55acf078b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.298+0000 7f44f6fc6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03b12f0 0x55acf07ff1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.298+0000 7f44f6fc6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc030620 con 0x55acf03b12f0 2024-05-27T07:50:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.298+0000 7f44ed6af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 0x55acf078b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.298+0000 7f44f6fc6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc031a90 con 0x55acf036e1c0 2024-05-27T07:50:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.298+0000 7f44f6fc6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc030cf0 con 0x55acf0398d80 2024-05-27T07:50:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.298+0000 7f44ed6af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 0x55acf078b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54866/0 (socket says 172.21.15.114:54866) 2024-05-27T07:50:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.298+0000 7f44ed6af640 1 -- 172.21.15.114:0/837619617 learned_addr learned my addr 172.21.15.114:0/837619617 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.298+0000 7f44edeb0640 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf036e1c0 0x55acf078ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.298+0000 7f44f6fc6740 10 client.0 _ll_register_callbacks cb 0x7f44e0009580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.298+0000 7f44ee6b1640 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03b12f0 0x55acf07ff1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44ed6af640 1 -- 172.21.15.114:0/837619617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf036e1c0 msgr2=0x55acf078ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44ed6af640 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf036e1c0 0x55acf078ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44ed6af640 1 -- 172.21.15.114:0/837619617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03b12f0 msgr2=0x55acf07ff1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44ed6af640 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03b12f0 0x55acf07ff1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44ed6af640 1 -- 172.21.15.114:0/837619617 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acf07ff710 con 0x55acf0398d80 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44edeb0640 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf036e1c0 0x55acf078ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44ed6af640 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 0x55acf078b240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f44e00c7350 tx=0x7f44e00c5d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65657a133cceb2be server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44d17fa640 1 -- 172.21.15.114:0/837619617 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e00db020 con 0x55acf0398d80 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44f6fc6740 1 -- 172.21.15.114:0/837619617 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44e00ca7e0 con 0x55acf0398d80 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44d17fa640 1 -- 172.21.15.114:0/837619617 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7f44e00c8fc0 con 0x55acf0398d80 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44d17fa640 1 -- 172.21.15.114:0/837619617 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f44e00c9850 con 0x55acf0398d80 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44f6fc6740 10 client.27505 fetch_fsmap learned FSMap version 61 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44f6fc6740 1 -- 172.21.15.114:0/837619617 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x55acf0800910 con 0x55acf0398d80 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44d17fa640 1 -- 172.21.15.114:0/837619617 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44e00ca7e0 con 0x55acf0398d80 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.299+0000 7f44d17fa640 1 -- 172.21.15.114:0/837619617 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 61) ==== 5910+0+0 (secure 0 0 0) 0x7f44e0041040 con 0x55acf0398d80 2024-05-27T07:50:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.300+0000 7f44f6fc6740 10 client.27505 fetch_fsmap finished waiting for FSMap version 61 2024-05-27T07:50:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.300+0000 7f44f6fc6740 10 client.27505 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.300+0000 7f44f6fc6740 20 client.27505 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.300+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55acf03da1b0 0x55acf0802cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.300+0000 7f44f6fc6740 4 client.27505 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:50:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.300+0000 7f44f6fc6740 1 -- 172.21.15.114:0/837619617 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55acf06e9360 con 0x55acf03da1b0 2024-05-27T07:50:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.300+0000 7f44edeb0640 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55acf03da1b0 0x55acf0802cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.300+0000 7f44d17fa640 10 client.27505 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.300+0000 7f44edeb0640 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55acf03da1b0 0x55acf0802cb0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9a64b2a028ba7e2 server_cookie=364889804f553ba0 in_seq=0 out_seq=0 2024-05-27T07:50:41.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44d17fa640 1 -- 172.21.15.114:0/837619617 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55acf06e9360 con 0x55acf03da1b0 2024-05-27T07:50:41.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44d17fa640 10 client.27505 handle_command_reply: tid=0 2024-05-27T07:50:41.676 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44f6fc6740 1 client.27505 shutdown 2024-05-27T07:50:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44f6fc6740 1 -- 172.21.15.114:0/837619617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 msgr2=0x55acf078b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 0x55acf078b240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f44e00c7350 tx=0x7f44e00c5d00 comp rx=0 tx=0).stop 2024-05-27T07:50:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44f6fc6740 1 -- 172.21.15.114:0/837619617 shutdown_connections 2024-05-27T07:50:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55acf03da1b0 0x55acf0802cb0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55acf03b12f0 0x55acf07ff1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44d17fa640 0 client.27505 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55acf0398d80 0x55acf078b240 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44f6fc6740 1 --2- 172.21.15.114:0/837619617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55acf036e1c0 0x55acf078ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44f6fc6740 1 -- 172.21.15.114:0/837619617 >> 172.21.15.114:0/837619617 conn(0x55acf03ce300 msgr2=0x55acf0786fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44f6fc6740 1 -- 172.21.15.114:0/837619617 shutdown_connections 2024-05-27T07:50:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.676+0000 7f44f6fc6740 1 -- 172.21.15.114:0/837619617 wait complete. 2024-05-27T07:50:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.677+0000 7f44f6fc6740 20 client.27505 trim_cache size 0 max 16384 2024-05-27T07:50:41.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:50:41.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:50:41.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:50:41.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.677+0000 7f44f0114640 1 -- 172.21.15.114:0/174438250 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f44d40b2a80 msgr2=0x7f44d40b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:41.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.677+0000 7f44f0114640 1 --2- 172.21.15.114:0/174438250 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f44d40b2a80 0x7f44d40b4f90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x55acf077c780 tx=0x7f44d80046c0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.677+0000 7f44f0114640 1 -- 172.21.15.114:0/174438250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44e811a8e0 msgr2=0x7f44e81a9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:41.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.677+0000 7f44f0114640 1 --2- 172.21.15.114:0/174438250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44e811a8e0 0x7f44e81a9bb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f44e0003f90 tx=0x7f44e00097e0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.677+0000 7f44f0114640 1 -- 172.21.15.114:0/174438250 shutdown_connections 2024-05-27T07:50:41.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.677+0000 7f44f0114640 1 --2- 172.21.15.114:0/174438250 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f44d40b2a80 0x7f44d40b4f90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.677+0000 7f44f0114640 1 --2- 172.21.15.114:0/174438250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44e811d240 0x7f44e81ade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.677+0000 7f44f0114640 1 --2- 172.21.15.114:0/174438250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44e811a8e0 0x7f44e81a9bb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.677+0000 7f44f0114640 1 --2- 172.21.15.114:0/174438250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e810a1f0 0x7f44e81a9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:41.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.677+0000 7f44f0114640 1 -- 172.21.15.114:0/174438250 >> 172.21.15.114:0/174438250 conn(0x7f44e806e8f0 msgr2=0x7f44e811acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:41.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.678+0000 7f44f0114640 1 -- 172.21.15.114:0/174438250 shutdown_connections 2024-05-27T07:50:41.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:41.678+0000 7f44f0114640 1 -- 172.21.15.114:0/174438250 wait complete. 2024-05-27T07:50:41.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:41 smithi163 ceph-mon[36451]: pgmap v361: 129 pgs: 129 active+clean; 74 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-05-27T07:50:41.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:41 smithi186 ceph-mon[36647]: pgmap v361: 129 pgs: 129 active+clean; 74 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-05-27T07:50:41.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:50:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: ignoring --setuser ceph since I am not root 2024-05-27T07:50:41.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:50:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: ignoring --setgroup ceph since I am not root 2024-05-27T07:50:41.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:50:41 smithi186 ceph-mds[67088]: -- 172.21.15.186:0/3111950273 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56528fabba40 con 0x56528fb03a80 2024-05-27T07:50:41.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:50:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: starting mds.l at 2024-05-27T07:50:42.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:41 smithi114 ceph-mon[32038]: pgmap v361: 129 pgs: 129 active+clean; 74 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-05-27T07:50:42.285 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1082326 10.14 MB/sec execute 414 sec latency 262.485 ms 2024-05-27T07:50:42.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:42 smithi163 ceph-mon[36451]: daemon mds.l restarted 2024-05-27T07:50:42.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:42 smithi186 ceph-mon[36647]: daemon mds.l restarted 2024-05-27T07:50:43.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:42 smithi114 ceph-mon[32038]: daemon mds.l restarted 2024-05-27T07:50:43.286 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1083893 10.14 MB/sec execute 415 sec latency 23.439 ms 2024-05-27T07:50:43.957 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:43 smithi114 ceph-mon[32038]: pgmap v362: 129 pgs: 129 active+clean; 77 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 241 op/s 2024-05-27T07:50:43.958 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:43 smithi114 ceph-mon[32038]: osdmap e120: 12 total, 12 up, 12 in 2024-05-27T07:50:43.958 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:43 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6832/3045713021,v1:172.21.15.186:6833/3045713021] up:boot 2024-05-27T07:50:43.958 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:43 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:50:43.958 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:43 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:50:43.958 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:43 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:50:43.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:43 smithi163 ceph-mon[36451]: pgmap v362: 129 pgs: 129 active+clean; 77 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 241 op/s 2024-05-27T07:50:43.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:43 smithi163 ceph-mon[36451]: osdmap e120: 12 total, 12 up, 12 in 2024-05-27T07:50:43.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:43 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6832/3045713021,v1:172.21.15.186:6833/3045713021] up:boot 2024-05-27T07:50:43.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:43 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:50:43.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:43 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:50:43.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:43 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:50:43.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:43 smithi186 ceph-mon[36647]: pgmap v362: 129 pgs: 129 active+clean; 77 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 241 op/s 2024-05-27T07:50:43.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:43 smithi186 ceph-mon[36647]: osdmap e120: 12 total, 12 up, 12 in 2024-05-27T07:50:43.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:43 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6832/3045713021,v1:172.21.15.186:6833/3045713021] up:boot 2024-05-27T07:50:43.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:43 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:50:43.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:43 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:50:43.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:43 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:50:44.286 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1085888 10.14 MB/sec execute 416 sec latency 15.772 ms 2024-05-27T07:50:45.286 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1087412 10.14 MB/sec execute 417 sec latency 18.112 ms 2024-05-27T07:50:45.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:45 smithi163 ceph-mon[36451]: pgmap v364: 129 pgs: 129 active+clean; 77 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 178 op/s 2024-05-27T07:50:45.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:45 smithi186 ceph-mon[36647]: pgmap v364: 129 pgs: 129 active+clean; 77 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 178 op/s 2024-05-27T07:50:46.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:45 smithi114 ceph-mon[32038]: pgmap v364: 129 pgs: 129 active+clean; 77 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 178 op/s 2024-05-27T07:50:46.286 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1089297 10.13 MB/sec execute 418 sec latency 54.749 ms 2024-05-27T07:50:47.286 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1090667 10.13 MB/sec execute 419 sec latency 74.664 ms 2024-05-27T07:50:47.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:47 smithi163 ceph-mon[36451]: pgmap v365: 129 pgs: 129 active+clean; 67 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 263 op/s 2024-05-27T07:50:47.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:47 smithi186 ceph-mon[36647]: pgmap v365: 129 pgs: 129 active+clean; 67 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 263 op/s 2024-05-27T07:50:48.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:47 smithi114 ceph-mon[32038]: pgmap v365: 129 pgs: 129 active+clean; 67 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 263 op/s 2024-05-27T07:50:48.286 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1092040 10.12 MB/sec execute 420 sec latency 59.218 ms 2024-05-27T07:50:49.286 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1094161 10.13 MB/sec execute 421 sec latency 17.022 ms 2024-05-27T07:50:49.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:49 smithi163 ceph-mon[36451]: pgmap v366: 129 pgs: 129 active+clean; 67 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 263 op/s 2024-05-27T07:50:49.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:49 smithi186 ceph-mon[36647]: pgmap v366: 129 pgs: 129 active+clean; 67 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 263 op/s 2024-05-27T07:50:50.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:49 smithi114 ceph-mon[32038]: pgmap v366: 129 pgs: 129 active+clean; 67 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 263 op/s 2024-05-27T07:50:50.286 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1095456 10.12 MB/sec execute 422 sec latency 27.491 ms 2024-05-27T07:50:51.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:50:51.286 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1097694 10.13 MB/sec execute 423 sec latency 18.115 ms 2024-05-27T07:50:51.574 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:50:51.958 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:51 smithi114 ceph-mon[32038]: pgmap v367: 129 pgs: 129 active+clean; 71 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 277 op/s 2024-05-27T07:50:51.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:51 smithi163 ceph-mon[36451]: pgmap v367: 129 pgs: 129 active+clean; 71 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 277 op/s 2024-05-27T07:50:51.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:51 smithi186 ceph-mon[36647]: pgmap v367: 129 pgs: 129 active+clean; 71 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 277 op/s 2024-05-27T07:50:52.286 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1098524 10.11 MB/sec execute 424 sec latency 84.851 ms 2024-05-27T07:50:53.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1100796 10.11 MB/sec execute 425 sec latency 45.051 ms 2024-05-27T07:50:53.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.387+0000 7fe670153700 1 -- 172.21.15.114:0/811924955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe668107de0 msgr2=0x7fe6681081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:53.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.387+0000 7fe670153700 1 --2- 172.21.15.114:0/811924955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe668107de0 0x7fe6681081f0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fe6580099c0 tx=0x7fe658009cd0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.387+0000 7fe670153700 1 -- 172.21.15.114:0/811924955 shutdown_connections 2024-05-27T07:50:53.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.387+0000 7fe670153700 1 --2- 172.21.15.114:0/811924955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe668108730 0x7fe668110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.387+0000 7fe670153700 1 --2- 172.21.15.114:0/811924955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6681034b0 0x7fe6681038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.387+0000 7fe670153700 1 --2- 172.21.15.114:0/811924955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe668107de0 0x7fe6681081f0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.387+0000 7fe670153700 1 -- 172.21.15.114:0/811924955 >> 172.21.15.114:0/811924955 conn(0x7fe6680fedb0 msgr2=0x7fe6681011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:53.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.387+0000 7fe670153700 1 -- 172.21.15.114:0/811924955 shutdown_connections 2024-05-27T07:50:53.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.387+0000 7fe670153700 1 -- 172.21.15.114:0/811924955 wait complete. 2024-05-27T07:50:53.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.388+0000 7fe670153700 1 Processor -- start 2024-05-27T07:50:53.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.388+0000 7fe670153700 1 -- start start 2024-05-27T07:50:53.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.388+0000 7fe670153700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6681034b0 0x7fe668199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.388+0000 7fe670153700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe668107de0 0x7fe66819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.388+0000 7fe670153700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe668108730 0x7fe66819e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.388+0000 7fe670153700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe668113340 con 0x7fe6681034b0 2024-05-27T07:50:53.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.388+0000 7fe670153700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe6681131c0 con 0x7fe668108730 2024-05-27T07:50:53.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.388+0000 7fe670153700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe6681134c0 con 0x7fe668107de0 2024-05-27T07:50:53.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.388+0000 7fe66deef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6681034b0 0x7fe668199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.388+0000 7fe66deef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6681034b0 0x7fe668199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60074/0 (socket says 172.21.15.114:60074) 2024-05-27T07:50:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.388+0000 7fe66deef700 1 -- 172.21.15.114:0/3671664927 learned_addr learned my addr 172.21.15.114:0/3671664927 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.388+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe668107de0 0x7fe66819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.389+0000 7fe66e6f0700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe668108730 0x7fe66819e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.389+0000 7fe66deef700 1 -- 172.21.15.114:0/3671664927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe668107de0 msgr2=0x7fe66819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.389+0000 7fe66deef700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe668107de0 0x7fe66819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.389+0000 7fe66deef700 1 -- 172.21.15.114:0/3671664927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe668108730 msgr2=0x7fe66819e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.389+0000 7fe66deef700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe668108730 0x7fe66819e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.389+0000 7fe66deef700 1 -- 172.21.15.114:0/3671664927 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe66819ebe0 con 0x7fe6681034b0 2024-05-27T07:50:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.389+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe668107de0 0x7fe66819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.389+0000 7fe66deef700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6681034b0 0x7fe668199d10 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe66400e9a0 tx=0x7fe66400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.389+0000 7fe65effd700 1 -- 172.21.15.114:0/3671664927 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66400ca10 con 0x7fe6681034b0 2024-05-27T07:50:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.389+0000 7fe65effd700 1 -- 172.21.15.114:0/3671664927 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe66400cbb0 con 0x7fe6681034b0 2024-05-27T07:50:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.389+0000 7fe65effd700 1 -- 172.21.15.114:0/3671664927 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe664018680 con 0x7fe6681034b0 2024-05-27T07:50:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.389+0000 7fe670153700 1 -- 172.21.15.114:0/3671664927 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe66819eed0 con 0x7fe6681034b0 2024-05-27T07:50:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.389+0000 7fe670153700 1 -- 172.21.15.114:0/3671664927 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe66806b600 con 0x7fe6681034b0 2024-05-27T07:50:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.393+0000 7fe65effd700 1 -- 172.21.15.114:0/3671664927 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fe664010430 con 0x7fe6681034b0 2024-05-27T07:50:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.394+0000 7fe65effd700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe6540b2a70 0x7fe6540b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.394+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe6540b2a70 0x7fe6540b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.394+0000 7fe65effd700 1 -- 172.21.15.114:0/3671664927 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9466+0+0 (secure 0 0 0) 0x7fe6640e2ac0 con 0x7fe6681034b0 2024-05-27T07:50:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.394+0000 7fe67eb68b80 1 Processor -- start 2024-05-27T07:50:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.394+0000 7fe67eb68b80 1 -- start start 2024-05-27T07:50:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.394+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe6540b2a70 0x7fe6540b4f80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe658000c00 tx=0x7fe65801a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe67eb68b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 0x55cda4ea0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe67eb68b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4ea0c50 0x55cda4e9aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe67eb68b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4e9af80 0x55cda4e9b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe67eb68b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cda4dc9c90 con 0x55cda4d1ce40 2024-05-27T07:50:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe67eb68b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55cda4db9830 con 0x55cda4e9af80 2024-05-27T07:50:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe67eb68b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cda4dbd870 con 0x55cda4ea0c50 2024-05-27T07:50:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe66deef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 0x55cda4ea0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe66deef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 0x55cda4ea0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60090/0 (socket says 172.21.15.114:60090) 2024-05-27T07:50:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe66deef700 1 -- 172.21.15.114:0/533113888 learned_addr learned my addr 172.21.15.114:0/533113888 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/533113888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4ea0c50 0x55cda4e9aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe66e6f0700 1 --2- 172.21.15.114:0/533113888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4e9af80 0x55cda4e9b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe66deef700 1 -- 172.21.15.114:0/533113888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4ea0c50 msgr2=0x55cda4e9aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe66deef700 1 --2- 172.21.15.114:0/533113888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4ea0c50 0x55cda4e9aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe66deef700 1 -- 172.21.15.114:0/533113888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4e9af80 msgr2=0x55cda4e9b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:53.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe66deef700 1 --2- 172.21.15.114:0/533113888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4e9af80 0x55cda4e9b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.395+0000 7fe66deef700 1 -- 172.21.15.114:0/533113888 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cda4d098d0 con 0x55cda4d1ce40 2024-05-27T07:50:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.396+0000 7fe66deef700 1 --2- 172.21.15.114:0/533113888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 0x55cda4ea0710 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fe6640a96f0 tx=0x7fe6640a9500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e496a3a3a94bb45f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:53.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.396+0000 7fe653fff700 1 -- 172.21.15.114:0/533113888 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6640a99f0 con 0x55cda4d1ce40 2024-05-27T07:50:53.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.396+0000 7fe653fff700 1 -- 172.21.15.114:0/533113888 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6640adda0 con 0x55cda4d1ce40 2024-05-27T07:50:53.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.396+0000 7fe653fff700 1 -- 172.21.15.114:0/533113888 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6640b23c0 con 0x55cda4d1ce40 2024-05-27T07:50:53.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.397+0000 7fe67eb68b80 1 -- 172.21.15.114:0/533113888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 msgr2=0x55cda4ea0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:53.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.397+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/533113888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 0x55cda4ea0710 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fe6640a96f0 tx=0x7fe6640a9500 comp rx=0 tx=0).stop 2024-05-27T07:50:53.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.397+0000 7fe67eb68b80 1 -- 172.21.15.114:0/533113888 shutdown_connections 2024-05-27T07:50:53.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.397+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/533113888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4e9af80 0x55cda4e9b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.397+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/533113888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 0x55cda4ea0710 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.397+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/533113888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4ea0c50 0x55cda4e9aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.397+0000 7fe67eb68b80 1 -- 172.21.15.114:0/533113888 >> 172.21.15.114:0/533113888 conn(0x55cda4d77af0 msgr2=0x55cda4e95640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:53.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.397+0000 7fe67eb68b80 1 -- 172.21.15.114:0/533113888 shutdown_connections 2024-05-27T07:50:53.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.397+0000 7fe67eb68b80 1 -- 172.21.15.114:0/533113888 wait complete. 2024-05-27T07:50:53.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.397+0000 7fe67eb68b80 1 -- start start 2024-05-27T07:50:53.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe67eb68b80 1 Processor -- start 2024-05-27T07:50:53.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe67eb68b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4d1ce40 0x55cda4ea43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe67eb68b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4e9af80 0x55cda4ea4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe67eb68b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4ea0c50 0x55cda4ea4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe67eb68b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cda4cbffa0 con 0x55cda4ea0c50 2024-05-27T07:50:53.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe67eb68b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55cda4d5b7c0 con 0x55cda4e9af80 2024-05-27T07:50:53.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe67eb68b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cda4ea7740 con 0x55cda4d1ce40 2024-05-27T07:50:53.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe67eb68b80 10 client.0 _ll_register_callbacks cb 0x55cda4d54920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:53.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe66deef700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4d1ce40 0x55cda4ea43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe66e6f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4ea0c50 0x55cda4ea4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe66e6f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4ea0c50 0x55cda4ea4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60106/0 (socket says 172.21.15.114:60106) 2024-05-27T07:50:53.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe66deef700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4d1ce40 0x55cda4ea43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36336/0 (socket says 172.21.15.114:36336) 2024-05-27T07:50:53.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe66e6f0700 1 -- 172.21.15.114:0/2761701926 learned_addr learned my addr 172.21.15.114:0/2761701926 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:53.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.398+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/2761701926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4e9af80 0x55cda4ea4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.399+0000 7fe66deef700 1 -- 172.21.15.114:0/2761701926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4e9af80 msgr2=0x55cda4ea4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:53.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.399+0000 7fe66deef700 1 --2- 172.21.15.114:0/2761701926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4e9af80 0x55cda4ea4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.399+0000 7fe66deef700 1 -- 172.21.15.114:0/2761701926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4ea0c50 msgr2=0x55cda4ea4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:53.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.399+0000 7fe66deef700 1 --2- 172.21.15.114:0/2761701926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4ea0c50 0x55cda4ea4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.399+0000 7fe66deef700 1 -- 172.21.15.114:0/2761701926 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cda4d58690 con 0x55cda4d1ce40 2024-05-27T07:50:53.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.399+0000 7fe66deef700 1 --2- 172.21.15.114:0/2761701926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4d1ce40 0x55cda4ea43d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe6640a93b0 tx=0x7fe664010c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0928ff8ea0c793a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:53.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.399+0000 7fe6517fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:50:53.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.399+0000 7fe6517fa700 1 -- 172.21.15.114:0/2761701926 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6640b2850 con 0x55cda4d1ce40 2024-05-27T07:50:53.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.399+0000 7fe67eb68b80 1 -- 172.21.15.114:0/2761701926 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6640adda0 con 0x55cda4d1ce40 2024-05-27T07:50:53.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.399+0000 7fe6517fa700 1 -- 172.21.15.114:0/2761701926 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9466+0+0 (secure 0 0 0) 0x7fe6640b25d0 con 0x55cda4d1ce40 2024-05-27T07:50:53.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.399+0000 7fe6517fa700 1 -- 172.21.15.114:0/2761701926 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fe6640a99f0 con 0x55cda4d1ce40 2024-05-27T07:50:53.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.400+0000 7fe67eb68b80 10 client.27523 fetch_fsmap learned FSMap version 63 2024-05-27T07:50:53.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.400+0000 7fe67eb68b80 1 -- 172.21.15.114:0/2761701926 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x55cda4eb6650 con 0x55cda4d1ce40 2024-05-27T07:50:53.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.400+0000 7fe6517fa700 1 -- 172.21.15.114:0/2761701926 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe664087020 con 0x55cda4d1ce40 2024-05-27T07:50:53.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.400+0000 7fe6517fa700 1 -- 172.21.15.114:0/2761701926 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 63) ==== 5910+0+0 (secure 0 0 0) 0x7fe6640c8b90 con 0x55cda4d1ce40 2024-05-27T07:50:53.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.400+0000 7fe67eb68b80 10 client.27523 fetch_fsmap finished waiting for FSMap version 63 2024-05-27T07:50:53.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.400+0000 7fe67eb68b80 10 client.27523 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:53.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.400+0000 7fe67eb68b80 20 client.27523 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:53.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.400+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/2761701926 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cda4eb8c60 0x55cda4ebb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.400+0000 7fe67eb68b80 4 client.27523 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:50:53.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.400+0000 7fe67eb68b80 1 -- 172.21.15.114:0/2761701926 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cda4e669f0 con 0x55cda4eb8c60 2024-05-27T07:50:53.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.400+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/2761701926 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cda4eb8c60 0x55cda4ebb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.401+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/2761701926 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cda4eb8c60 0x55cda4ebb0d0 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31c1517bce94b82e server_cookie=84765ce4dbc21af7 in_seq=0 out_seq=0 2024-05-27T07:50:53.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.401+0000 7fe6517fa700 10 client.27523 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:53.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe6517fa700 1 -- 172.21.15.114:0/2761701926 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cda4e669f0 con 0x55cda4eb8c60 2024-05-27T07:50:53.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe6517fa700 10 client.27523 handle_command_reply: tid=0 2024-05-27T07:50:53.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe67eb68b80 1 client.27523 shutdown 2024-05-27T07:50:53.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe67eb68b80 1 -- 172.21.15.114:0/2761701926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4d1ce40 msgr2=0x55cda4ea43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:53.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/2761701926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4d1ce40 0x55cda4ea43d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe6640a93b0 tx=0x7fe664010c10 comp rx=0 tx=0).stop 2024-05-27T07:50:53.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe67eb68b80 1 -- 172.21.15.114:0/2761701926 shutdown_connections 2024-05-27T07:50:53.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/2761701926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4ea0c50 0x55cda4ea4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/2761701926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4d1ce40 0x55cda4ea43d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/2761701926 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cda4eb8c60 0x55cda4ebb0d0 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/2761701926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4e9af80 0x55cda4ea4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe6517fa700 0 client.27523 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:53.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe67eb68b80 1 -- 172.21.15.114:0/2761701926 >> 172.21.15.114:0/2761701926 conn(0x55cda4d77af0 msgr2=0x55cda4e99a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:53.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe67eb68b80 1 -- 172.21.15.114:0/2761701926 shutdown_connections 2024-05-27T07:50:53.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe67eb68b80 1 -- 172.21.15.114:0/2761701926 wait complete. 2024-05-27T07:50:53.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.402+0000 7fe67eb68b80 20 client.27523 trim_cache size 0 max 16384 2024-05-27T07:50:53.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.424+0000 7fe67eb68b80 1 Processor -- start 2024-05-27T07:50:53.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.424+0000 7fe67eb68b80 1 -- start start 2024-05-27T07:50:53.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.424+0000 7fe67eb68b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4d1ce40 0x55cda4e9aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.424+0000 7fe67eb68b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4e9af80 0x55cda4e9ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.424+0000 7fe67eb68b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4ea0c50 0x55cda4ea3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.424+0000 7fe67eb68b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6681afcd0 con 0x55cda4e9af80 2024-05-27T07:50:53.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.424+0000 7fe67eb68b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe6681ae440 con 0x55cda4d1ce40 2024-05-27T07:50:53.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.424+0000 7fe67eb68b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe6681ac290 con 0x55cda4ea0c50 2024-05-27T07:50:53.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.424+0000 7fe66d6ee700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4e9af80 0x55cda4e9ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.424+0000 7fe66d6ee700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4e9af80 0x55cda4e9ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60120/0 (socket says 172.21.15.114:60120) 2024-05-27T07:50:53.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.424+0000 7fe66d6ee700 1 -- 172.21.15.114:0/2770716202 learned_addr learned my addr 172.21.15.114:0/2770716202 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:53.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.424+0000 7fe66e6f0700 1 --2- 172.21.15.114:0/2770716202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4ea0c50 0x55cda4ea3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.424+0000 7fe66deef700 1 --2- 172.21.15.114:0/2770716202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4d1ce40 0x55cda4e9aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.425+0000 7fe66d6ee700 1 -- 172.21.15.114:0/2770716202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4ea0c50 msgr2=0x55cda4ea3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:53.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.425+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/2770716202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4ea0c50 0x55cda4ea3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.425+0000 7fe66d6ee700 1 -- 172.21.15.114:0/2770716202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4d1ce40 msgr2=0x55cda4e9aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:53.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.425+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/2770716202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4d1ce40 0x55cda4e9aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.425+0000 7fe66d6ee700 1 -- 172.21.15.114:0/2770716202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cda4e951c0 con 0x55cda4e9af80 2024-05-27T07:50:53.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.425+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/2770716202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4e9af80 0x55cda4e9ea40 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fe668076aa0 tx=0x7fe658022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3f0f918334626c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:53.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.425+0000 7fe6537fe700 1 -- 172.21.15.114:0/2770716202 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe658039070 con 0x55cda4e9af80 2024-05-27T07:50:53.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.425+0000 7fe6537fe700 1 -- 172.21.15.114:0/2770716202 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe65800f460 con 0x55cda4e9af80 2024-05-27T07:50:53.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.425+0000 7fe6537fe700 1 -- 172.21.15.114:0/2770716202 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6580326a0 con 0x55cda4e9af80 2024-05-27T07:50:53.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.426+0000 7fe67eb68b80 1 -- 172.21.15.114:0/2770716202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4e9af80 msgr2=0x55cda4e9ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:53.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.426+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/2770716202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4e9af80 0x55cda4e9ea40 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fe668076aa0 tx=0x7fe658022000 comp rx=0 tx=0).stop 2024-05-27T07:50:53.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.426+0000 7fe67eb68b80 1 -- 172.21.15.114:0/2770716202 shutdown_connections 2024-05-27T07:50:53.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.426+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/2770716202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4ea0c50 0x55cda4ea3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.426+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/2770716202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4d1ce40 0x55cda4e9aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.426+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/2770716202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4e9af80 0x55cda4e9ea40 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.426+0000 7fe67eb68b80 1 -- 172.21.15.114:0/2770716202 >> 172.21.15.114:0/2770716202 conn(0x55cda4d77af0 msgr2=0x55cda4e9fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:53.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.426+0000 7fe67eb68b80 1 -- 172.21.15.114:0/2770716202 shutdown_connections 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.426+0000 7fe67eb68b80 1 -- 172.21.15.114:0/2770716202 wait complete. 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.426+0000 7fe67eb68b80 1 -- start start 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.427+0000 7fe67eb68b80 1 Processor -- start 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.427+0000 7fe67eb68b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 0x55cda4eacd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.427+0000 7fe67eb68b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4e9af80 0x55cda4ead280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.427+0000 7fe67eb68b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4ea0c50 0x55cda4ebffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.427+0000 7fe67eb68b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6681ac900 con 0x55cda4d1ce40 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.427+0000 7fe67eb68b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe6681add70 con 0x55cda4ea0c50 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.427+0000 7fe67eb68b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe6681acfd0 con 0x55cda4e9af80 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.427+0000 7fe66deef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 0x55cda4eacd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.427+0000 7fe67eb68b80 10 client.0 _ll_register_callbacks cb 0x55cda4d54920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.427+0000 7fe66deef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 0x55cda4eacd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60134/0 (socket says 172.21.15.114:60134) 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.427+0000 7fe66deef700 1 -- 172.21.15.114:0/406151302 learned_addr learned my addr 172.21.15.114:0/406151302 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:53.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.427+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4e9af80 0x55cda4ead280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.427+0000 7fe66e6f0700 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4ea0c50 0x55cda4ebffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe66deef700 1 -- 172.21.15.114:0/406151302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4e9af80 msgr2=0x55cda4ead280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:53.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe66deef700 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4e9af80 0x55cda4ead280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe66deef700 1 -- 172.21.15.114:0/406151302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4ea0c50 msgr2=0x55cda4ebffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:53.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe66deef700 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4ea0c50 0x55cda4ebffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe66deef700 1 -- 172.21.15.114:0/406151302 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cda4ec0760 con 0x55cda4d1ce40 2024-05-27T07:50:53.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe66e6f0700 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4ea0c50 0x55cda4ebffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:53.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe66deef700 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 0x55cda4eacd40 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fe6640a55e0 tx=0x7fe6640dff10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22312b158b431b16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:53.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe6517fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:50:53.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe6517fa700 1 -- 172.21.15.114:0/406151302 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6640a58b0 con 0x55cda4d1ce40 2024-05-27T07:50:53.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe6517fa700 1 -- 172.21.15.114:0/406151302 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9466+0+0 (secure 0 0 0) 0x7fe6640a5fe0 con 0x55cda4d1ce40 2024-05-27T07:50:53.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe67eb68b80 1 -- 172.21.15.114:0/406151302 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe65800f460 con 0x55cda4d1ce40 2024-05-27T07:50:53.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe6517fa700 1 -- 172.21.15.114:0/406151302 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fe6640b2850 con 0x55cda4d1ce40 2024-05-27T07:50:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe67eb68b80 10 client.17961 fetch_fsmap learned FSMap version 63 2024-05-27T07:50:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe67eb68b80 1 -- 172.21.15.114:0/406151302 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x55cda4f232d0 con 0x55cda4d1ce40 2024-05-27T07:50:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe6517fa700 1 -- 172.21.15.114:0/406151302 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe65800f460 con 0x55cda4d1ce40 2024-05-27T07:50:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.428+0000 7fe6517fa700 1 -- 172.21.15.114:0/406151302 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 63) ==== 5910+0+0 (secure 0 0 0) 0x7fe6640aef30 con 0x55cda4d1ce40 2024-05-27T07:50:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.429+0000 7fe67eb68b80 10 client.17961 fetch_fsmap finished waiting for FSMap version 63 2024-05-27T07:50:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.429+0000 7fe67eb68b80 10 client.17961 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.429+0000 7fe67eb68b80 20 client.17961 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.429+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cda4f236f0 0x55cda4f25b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.429+0000 7fe67eb68b80 4 client.17961 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:50:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.429+0000 7fe67eb68b80 1 -- 172.21.15.114:0/406151302 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cda4dcbbe0 con 0x55cda4f236f0 2024-05-27T07:50:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.429+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cda4f236f0 0x55cda4f25b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.429+0000 7fe6517fa700 10 client.17961 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.429+0000 7fe66d6ee700 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cda4f236f0 0x55cda4f25b80 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60ad533d23b3f20b server_cookie=f7f5a62a610d985 in_seq=0 out_seq=0 2024-05-27T07:50:53.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.430+0000 7fe6517fa700 1 -- 172.21.15.114:0/406151302 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cda4dcbbe0 con 0x55cda4f236f0 2024-05-27T07:50:53.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.430+0000 7fe6517fa700 10 client.17961 handle_command_reply: tid=0 2024-05-27T07:50:53.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.430+0000 7fe67eb68b80 1 client.17961 shutdown 2024-05-27T07:50:53.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.430+0000 7fe67eb68b80 1 -- 172.21.15.114:0/406151302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 msgr2=0x55cda4eacd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:53.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.430+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 0x55cda4eacd40 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fe6640a55e0 tx=0x7fe6640dff10 comp rx=0 tx=0).stop 2024-05-27T07:50:53.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe67eb68b80 1 -- 172.21.15.114:0/406151302 shutdown_connections 2024-05-27T07:50:53.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cda4f236f0 0x55cda4f25b80 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cda4ea0c50 0x55cda4ebffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cda4d1ce40 0x55cda4eacd40 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe6517fa700 0 client.17961 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:53.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe67eb68b80 1 --2- 172.21.15.114:0/406151302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cda4e9af80 0x55cda4ead280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe67eb68b80 1 -- 172.21.15.114:0/406151302 >> 172.21.15.114:0/406151302 conn(0x55cda4d77af0 msgr2=0x55cda4ea4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:53.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe67eb68b80 1 -- 172.21.15.114:0/406151302 shutdown_connections 2024-05-27T07:50:53.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe67eb68b80 1 -- 172.21.15.114:0/406151302 wait complete. 2024-05-27T07:50:53.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe67eb68b80 20 client.17961 trim_cache size 0 max 16384 2024-05-27T07:50:53.436 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:50:53.436 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:50:53.436 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:50:53.436 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:50:53.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe670153700 1 -- 172.21.15.114:0/3671664927 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe6540b2a70 msgr2=0x7fe6540b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:53.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe670153700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe6540b2a70 0x7fe6540b4f80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe658000c00 tx=0x7fe65801a040 comp rx=0 tx=0).stop 2024-05-27T07:50:53.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe670153700 1 -- 172.21.15.114:0/3671664927 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6681034b0 msgr2=0x7fe668199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:53.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe670153700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6681034b0 0x7fe668199d10 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe66400e9a0 tx=0x7fe66400ecb0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe670153700 1 -- 172.21.15.114:0/3671664927 shutdown_connections 2024-05-27T07:50:53.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe670153700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe6540b2a70 0x7fe6540b4f80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe670153700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe668108730 0x7fe66819e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe670153700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6681034b0 0x7fe668199d10 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe670153700 1 --2- 172.21.15.114:0/3671664927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe668107de0 0x7fe66819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:53.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.431+0000 7fe670153700 1 -- 172.21.15.114:0/3671664927 >> 172.21.15.114:0/3671664927 conn(0x7fe6680fedb0 msgr2=0x7fe668109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:53.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.432+0000 7fe670153700 1 -- 172.21.15.114:0/3671664927 shutdown_connections 2024-05-27T07:50:53.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:53.432+0000 7fe670153700 1 -- 172.21.15.114:0/3671664927 wait complete. 2024-05-27T07:50:53.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:53 smithi163 ceph-mon[36451]: pgmap v368: 129 pgs: 129 active+clean; 73 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 299 op/s 2024-05-27T07:50:53.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:53 smithi186 ceph-mon[36647]: pgmap v368: 129 pgs: 129 active+clean; 73 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 299 op/s 2024-05-27T07:50:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:53 smithi114 ceph-mon[32038]: pgmap v368: 129 pgs: 129 active+clean; 73 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 299 op/s 2024-05-27T07:50:54.124 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:50:54.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:50:54.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1101929 10.10 MB/sec execute 426 sec latency 38.110 ms 2024-05-27T07:50:54.458 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:50:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.783+0000 7f53b3ba8640 1 -- 172.21.15.114:0/605450419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac1021c0 msgr2=0x7f53ac1025d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.783+0000 7f53b3ba8640 1 --2- 172.21.15.114:0/605450419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac1021c0 0x7f53ac1025d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f539c009900 tx=0x7f539c02a410 comp rx=0 tx=0).stop 2024-05-27T07:50:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.783+0000 7f53b3ba8640 1 -- 172.21.15.114:0/605450419 shutdown_connections 2024-05-27T07:50:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.783+0000 7f53b3ba8640 1 --2- 172.21.15.114:0/605450419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53ac10e6c0 0x7f53ac110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.783+0000 7f53b3ba8640 1 --2- 172.21.15.114:0/605450419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac1021c0 0x7f53ac1025d0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.783+0000 7f53b3ba8640 1 --2- 172.21.15.114:0/605450419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac109d10 0x7f53ac101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.783+0000 7f53b3ba8640 1 -- 172.21.15.114:0/605450419 >> 172.21.15.114:0/605450419 conn(0x7f53ac0779d0 msgr2=0x7f53ac077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.783+0000 7f53b3ba8640 1 -- 172.21.15.114:0/605450419 shutdown_connections 2024-05-27T07:50:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.783+0000 7f53b3ba8640 1 -- 172.21.15.114:0/605450419 wait complete. 2024-05-27T07:50:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b3ba8640 1 Processor -- start 2024-05-27T07:50:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b3ba8640 1 -- start start 2024-05-27T07:50:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b3ba8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53ac1021c0 0x7f53ac1a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b3ba8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac109d10 0x7f53ac1a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b3ba8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac10e6c0 0x7f53ac19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b3ba8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac113ed0 con 0x7f53ac109d10 2024-05-27T07:50:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b3ba8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac113d50 con 0x7f53ac1021c0 2024-05-27T07:50:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b3ba8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac114050 con 0x7f53ac10e6c0 2024-05-27T07:50:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b1143640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac109d10 0x7f53ac1a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b1143640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac109d10 0x7f53ac1a0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36458/0 (socket says 172.21.15.114:36458) 2024-05-27T07:50:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b1944640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53ac1021c0 0x7f53ac1a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b1143640 1 -- 172.21.15.114:0/260199792 learned_addr learned my addr 172.21.15.114:0/260199792 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b2145640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac10e6c0 0x7f53ac19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b1944640 1 -- 172.21.15.114:0/260199792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac10e6c0 msgr2=0x7f53ac19a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b1944640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac10e6c0 0x7f53ac19a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b1944640 1 -- 172.21.15.114:0/260199792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac109d10 msgr2=0x7f53ac1a0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.784+0000 7f53b1944640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac109d10 0x7f53ac1a0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.785+0000 7f53b1944640 1 -- 172.21.15.114:0/260199792 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac19ae20 con 0x7f53ac1021c0 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.785+0000 7f53b1143640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac109d10 0x7f53ac1a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.785+0000 7f53b2145640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac10e6c0 0x7f53ac19a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.785+0000 7f53b1944640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53ac1021c0 0x7f53ac1a0570 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f53ac06a240 tx=0x7f53a800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.785+0000 7f53a2ffd640 1 -- 172.21.15.114:0/260199792 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a800cc40 con 0x7f53ac1021c0 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.785+0000 7f53a2ffd640 1 -- 172.21.15.114:0/260199792 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53a800cde0 con 0x7f53ac1021c0 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.785+0000 7f53b3ba8640 1 -- 172.21.15.114:0/260199792 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ac19b0b0 con 0x7f53ac1021c0 2024-05-27T07:50:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.785+0000 7f53a2ffd640 1 -- 172.21.15.114:0/260199792 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8010430 con 0x7f53ac1021c0 2024-05-27T07:50:54.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.785+0000 7f53b3ba8640 1 -- 172.21.15.114:0/260199792 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ac1a7310 con 0x7f53ac1021c0 2024-05-27T07:50:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.788+0000 7f53a2ffd640 1 -- 172.21.15.114:0/260199792 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f53a8003680 con 0x7f53ac1021c0 2024-05-27T07:50:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.788+0000 7f53baa59740 1 Processor -- start 2024-05-27T07:50:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53baa59740 1 -- start start 2024-05-27T07:50:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53baa59740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a09211b0 0x55f2a0cc23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53a2ffd640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f53980b2950 0x7f53980b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53baa59740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cc2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53baa59740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 0x55f2a0cc2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53baa59740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f2a0c34c30 con 0x55f2a09004d0 2024-05-27T07:50:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53baa59740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f2a0c247d0 con 0x55f2a08f82f0 2024-05-27T07:50:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53baa59740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f2a0c28810 con 0x55f2a09211b0 2024-05-27T07:50:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53b2145640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 0x55f2a0cc2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53b1944640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f53980b2950 0x7f53980b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53b2145640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 0x55f2a0cc2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36466/0 (socket says 172.21.15.114:36466) 2024-05-27T07:50:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53b2145640 1 -- 172.21.15.114:0/1097212615 learned_addr learned my addr 172.21.15.114:0/1097212615 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53b1143640 1 --2- 172.21.15.114:0/1097212615 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cc2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53b2145640 1 --2- 172.21.15.114:0/1097212615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a09211b0 0x55f2a0cc23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53a2ffd640 1 -- 172.21.15.114:0/260199792 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9466+0+0 (secure 0 0 0) 0x7f53a80e2d30 con 0x7f53ac1021c0 2024-05-27T07:50:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53b2145640 1 -- 172.21.15.114:0/1097212615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a09211b0 msgr2=0x55f2a0cc23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53b2145640 1 --2- 172.21.15.114:0/1097212615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a09211b0 0x55f2a0cc23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53b2145640 1 -- 172.21.15.114:0/1097212615 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 msgr2=0x55f2a0cc2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53b2145640 1 --2- 172.21.15.114:0/1097212615 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cc2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53b2145640 1 -- 172.21.15.114:0/1097212615 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2a0939210 con 0x55f2a09004d0 2024-05-27T07:50:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.789+0000 7f53b1143640 1 --2- 172.21.15.114:0/1097212615 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cc2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.790+0000 7f53b1944640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f53980b2950 0x7f53980b4e60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f53a80a6ed0 tx=0x7f53a806f6a0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.790+0000 7f53b2145640 1 --2- 172.21.15.114:0/1097212615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 0x55f2a0cc2e50 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f53a4011ea0 tx=0x7f53a4009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfeef76963dec74 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.790+0000 7f5397fff640 1 -- 172.21.15.114:0/1097212615 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a400fc50 con 0x55f2a09004d0 2024-05-27T07:50:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.790+0000 7f5397fff640 1 -- 172.21.15.114:0/1097212615 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53a400fdf0 con 0x55f2a09004d0 2024-05-27T07:50:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.790+0000 7f5397fff640 1 -- 172.21.15.114:0/1097212615 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a400e5b0 con 0x55f2a09004d0 2024-05-27T07:50:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.791+0000 7f53baa59740 1 -- 172.21.15.114:0/1097212615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 msgr2=0x55f2a0cc2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.791+0000 7f53baa59740 1 --2- 172.21.15.114:0/1097212615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 0x55f2a0cc2e50 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f53a4011ea0 tx=0x7f53a4009f30 comp rx=0 tx=0).stop 2024-05-27T07:50:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.791+0000 7f53baa59740 1 -- 172.21.15.114:0/1097212615 shutdown_connections 2024-05-27T07:50:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.791+0000 7f53baa59740 1 --2- 172.21.15.114:0/1097212615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 0x55f2a0cc2e50 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.791+0000 7f53baa59740 1 --2- 172.21.15.114:0/1097212615 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cc2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.791+0000 7f53baa59740 1 --2- 172.21.15.114:0/1097212615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a09211b0 0x55f2a0cc23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.791+0000 7f53baa59740 1 -- 172.21.15.114:0/1097212615 >> 172.21.15.114:0/1097212615 conn(0x55f2a0915300 msgr2=0x55f2a0cbeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.791+0000 7f53baa59740 1 -- 172.21.15.114:0/1097212615 shutdown_connections 2024-05-27T07:50:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.791+0000 7f53baa59740 1 -- 172.21.15.114:0/1097212615 wait complete. 2024-05-27T07:50:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.791+0000 7f53baa59740 1 -- start start 2024-05-27T07:50:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.791+0000 7f53baa59740 1 Processor -- start 2024-05-27T07:50:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53baa59740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cce570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53baa59740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 0x55f2a0cceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53baa59740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a09211b0 0x55f2a0ccc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53baa59740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f2a090a790 con 0x55f2a09004d0 2024-05-27T07:50:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53baa59740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f2a0be0890 con 0x55f2a08f82f0 2024-05-27T07:50:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53baa59740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f2a0ccf500 con 0x55f2a09211b0 2024-05-27T07:50:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53baa59740 10 client.0 _ll_register_callbacks cb 0x55f2a0938a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53b1143640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 0x55f2a0cceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53b1143640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 0x55f2a0cceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36476/0 (socket says 172.21.15.114:36476) 2024-05-27T07:50:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53b1143640 1 -- 172.21.15.114:0/901212240 learned_addr learned my addr 172.21.15.114:0/901212240 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53b2145640 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a09211b0 0x55f2a0ccc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53b2145640 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53b1143640 1 -- 172.21.15.114:0/901212240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a09211b0 msgr2=0x55f2a0ccc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53b1143640 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a09211b0 0x55f2a0ccc980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53b1143640 1 -- 172.21.15.114:0/901212240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 msgr2=0x55f2a0cce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53b1143640 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cce570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53b1143640 1 -- 172.21.15.114:0/901212240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2a0a05620 con 0x55f2a09004d0 2024-05-27T07:50:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53b2145640 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a09211b0 0x55f2a0ccc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:50:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53b1143640 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 0x55f2a0cceab0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f539c02aa00 tx=0x7f539c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5c37f6f750d29 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.792+0000 7f53957fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:50:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53957fa640 1 -- 172.21.15.114:0/901212240 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c035040 con 0x55f2a09004d0 2024-05-27T07:50:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53957fa640 1 -- 172.21.15.114:0/901212240 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9466+0+0 (secure 0 0 0) 0x7f539c003e20 con 0x55f2a09004d0 2024-05-27T07:50:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53baa59740 1 -- 172.21.15.114:0/901212240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a400fdf0 con 0x55f2a09004d0 2024-05-27T07:50:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53957fa640 1 -- 172.21.15.114:0/901212240 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f539c005020 con 0x55f2a09004d0 2024-05-27T07:50:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53baa59740 10 client.17973 fetch_fsmap learned FSMap version 63 2024-05-27T07:50:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53baa59740 1 -- 172.21.15.114:0/901212240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x55f2a0cdb8a0 con 0x55f2a09004d0 2024-05-27T07:50:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53957fa640 1 -- 172.21.15.114:0/901212240 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53a400fdf0 con 0x55f2a09004d0 2024-05-27T07:50:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53957fa640 1 -- 172.21.15.114:0/901212240 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 63) ==== 5910+0+0 (secure 0 0 0) 0x7f539c0314a0 con 0x55f2a09004d0 2024-05-27T07:50:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53baa59740 10 client.17973 fetch_fsmap finished waiting for FSMap version 63 2024-05-27T07:50:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53baa59740 10 client.17973 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53baa59740 20 client.17973 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53baa59740 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f2a08dfd80 0x55f2a0cdf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53baa59740 4 client.17973 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:50:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53baa59740 1 -- 172.21.15.114:0/901212240 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2a0c36b80 con 0x55f2a08dfd80 2024-05-27T07:50:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.793+0000 7f53b2145640 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f2a08dfd80 0x55f2a0cdf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.794+0000 7f53957fa640 10 client.17973 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.794+0000 7f53b2145640 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f2a08dfd80 0x55f2a0cdf050 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28d39cda966fd3d6 server_cookie=fc28e2ae4fd4a6bb in_seq=0 out_seq=0 2024-05-27T07:50:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.795+0000 7f53957fa640 1 -- 172.21.15.114:0/901212240 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f2a0c36b80 con 0x55f2a08dfd80 2024-05-27T07:50:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.795+0000 7f53957fa640 10 client.17973 handle_command_reply: tid=0 2024-05-27T07:50:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.795+0000 7f53baa59740 1 client.17973 shutdown 2024-05-27T07:50:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.795+0000 7f53baa59740 1 -- 172.21.15.114:0/901212240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 msgr2=0x55f2a0cceab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.795+0000 7f53baa59740 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 0x55f2a0cceab0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f539c02aa00 tx=0x7f539c003820 comp rx=0 tx=0).stop 2024-05-27T07:50:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.795+0000 7f53baa59740 1 -- 172.21.15.114:0/901212240 shutdown_connections 2024-05-27T07:50:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.795+0000 7f53baa59740 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f2a08dfd80 0x55f2a0cdf050 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.795+0000 7f53baa59740 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a09211b0 0x55f2a0ccc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.795+0000 7f53baa59740 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a09004d0 0x55f2a0cceab0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.795+0000 7f53957fa640 0 client.17973 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.795+0000 7f53baa59740 1 --2- 172.21.15.114:0/901212240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cce570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.795+0000 7f53baa59740 1 -- 172.21.15.114:0/901212240 >> 172.21.15.114:0/901212240 conn(0x55f2a0915300 msgr2=0x55f2a0cc40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.796+0000 7f53baa59740 1 -- 172.21.15.114:0/901212240 shutdown_connections 2024-05-27T07:50:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.796+0000 7f53baa59740 1 -- 172.21.15.114:0/901212240 wait complete. 2024-05-27T07:50:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.796+0000 7f53baa59740 20 client.17973 trim_cache size 0 max 16384 2024-05-27T07:50:54.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53baa59740 1 Processor -- start 2024-05-27T07:50:54.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53baa59740 1 -- start start 2024-05-27T07:50:54.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53baa59740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 0x55f2a0cc4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.824 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53baa59740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08dfd80 0x55f2a0cc6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.824 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53baa59740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a08f82f0 0x55f2a0cc65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.824 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53baa59740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1b2b80 con 0x55f2a0915300 2024-05-27T07:50:54.824 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53baa59740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1b1310 con 0x55f2a08dfd80 2024-05-27T07:50:54.824 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53baa59740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1af160 con 0x55f2a08f82f0 2024-05-27T07:50:54.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53b2145640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 0x55f2a0cc4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53b2145640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 0x55f2a0cc4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36492/0 (socket says 172.21.15.114:36492) 2024-05-27T07:50:54.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53b2145640 1 -- 172.21.15.114:0/939901368 learned_addr learned my addr 172.21.15.114:0/939901368 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:54.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53b2145640 1 --2- 172.21.15.114:0/939901368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a08f82f0 0x55f2a0cc65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53b1143640 1 --2- 172.21.15.114:0/939901368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08dfd80 0x55f2a0cc6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53b2145640 1 -- 172.21.15.114:0/939901368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a08f82f0 msgr2=0x55f2a0cc65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:54.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53b2145640 1 --2- 172.21.15.114:0/939901368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a08f82f0 0x55f2a0cc65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53b2145640 1 -- 172.21.15.114:0/939901368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08dfd80 msgr2=0x55f2a0cc6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:54.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53b2145640 1 --2- 172.21.15.114:0/939901368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08dfd80 0x55f2a0cc6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53b2145640 1 -- 172.21.15.114:0/939901368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2a0cc4c60 con 0x55f2a0915300 2024-05-27T07:50:54.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.812+0000 7f53b1143640 1 --2- 172.21.15.114:0/939901368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08dfd80 0x55f2a0cc6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:50:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.813+0000 7f53b2145640 1 --2- 172.21.15.114:0/939901368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 0x55f2a0cc4720 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f53a4000f80 tx=0x7f53a40116f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8a14721e60de80 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.813+0000 7f53977fe640 1 -- 172.21.15.114:0/939901368 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a401f7d0 con 0x55f2a0915300 2024-05-27T07:50:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.813+0000 7f53977fe640 1 -- 172.21.15.114:0/939901368 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53a4003e20 con 0x55f2a0915300 2024-05-27T07:50:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.813+0000 7f53977fe640 1 -- 172.21.15.114:0/939901368 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a40052f0 con 0x55f2a0915300 2024-05-27T07:50:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.814+0000 7f53baa59740 1 -- 172.21.15.114:0/939901368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 msgr2=0x55f2a0cc4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.814+0000 7f53baa59740 1 --2- 172.21.15.114:0/939901368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 0x55f2a0cc4720 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f53a4000f80 tx=0x7f53a40116f0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.814+0000 7f53baa59740 1 -- 172.21.15.114:0/939901368 shutdown_connections 2024-05-27T07:50:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.814+0000 7f53baa59740 1 --2- 172.21.15.114:0/939901368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a08f82f0 0x55f2a0cc65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.814+0000 7f53baa59740 1 --2- 172.21.15.114:0/939901368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08dfd80 0x55f2a0cc6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.814+0000 7f53baa59740 1 --2- 172.21.15.114:0/939901368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 0x55f2a0cc4720 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.814+0000 7f53baa59740 1 -- 172.21.15.114:0/939901368 >> 172.21.15.114:0/939901368 conn(0x55f2a0cc8300 msgr2=0x55f2a0cc4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.814+0000 7f53baa59740 1 -- 172.21.15.114:0/939901368 shutdown_connections 2024-05-27T07:50:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.814+0000 7f53baa59740 1 -- 172.21.15.114:0/939901368 wait complete. 2024-05-27T07:50:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.814+0000 7f53baa59740 1 -- start start 2024-05-27T07:50:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.814+0000 7f53baa59740 1 Processor -- start 2024-05-27T07:50:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.814+0000 7f53baa59740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a08dfd80 0x55f2a0cd1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53baa59740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cd2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53baa59740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 0x55f2a0d43b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53baa59740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1af7d0 con 0x55f2a0915300 2024-05-27T07:50:54.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53baa59740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1b0c40 con 0x55f2a08f82f0 2024-05-27T07:50:54.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53baa59740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac1afea0 con 0x55f2a08dfd80 2024-05-27T07:50:54.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53baa59740 10 client.0 _ll_register_callbacks cb 0x7f539c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:50:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53b2145640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 0x55f2a0d43b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53b2145640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 0x55f2a0d43b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36498/0 (socket says 172.21.15.114:36498) 2024-05-27T07:50:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53b2145640 1 -- 172.21.15.114:0/156068829 learned_addr learned my addr 172.21.15.114:0/156068829 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53b2145640 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a08dfd80 0x55f2a0cd1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53b1143640 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cd2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53b2145640 1 -- 172.21.15.114:0/156068829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a08dfd80 msgr2=0x55f2a0cd1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53b2145640 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a08dfd80 0x55f2a0cd1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53b2145640 1 -- 172.21.15.114:0/156068829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 msgr2=0x55f2a0cd2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:50:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53b2145640 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cd2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53b2145640 1 -- 172.21.15.114:0/156068829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2a0d44040 con 0x55f2a0915300 2024-05-27T07:50:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53b1143640 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cd2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:50:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53b2145640 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 0x55f2a0d43b00 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f53a40116c0 tx=0x7f53a40050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659224a92502c448 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53957fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:50:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53957fa640 1 -- 172.21.15.114:0/156068829 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a401f750 con 0x55f2a0915300 2024-05-27T07:50:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53957fa640 1 -- 172.21.15.114:0/156068829 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9466+0+0 (secure 0 0 0) 0x7f53a401f8f0 con 0x55f2a0915300 2024-05-27T07:50:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.815+0000 7f53baa59740 1 -- 172.21.15.114:0/156068829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a4003e20 con 0x55f2a0915300 2024-05-27T07:50:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.816+0000 7f53957fa640 1 -- 172.21.15.114:0/156068829 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f53a401faf0 con 0x55f2a0915300 2024-05-27T07:50:54.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.816+0000 7f53baa59740 10 client.17979 fetch_fsmap learned FSMap version 63 2024-05-27T07:50:54.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.816+0000 7f53baa59740 1 -- 172.21.15.114:0/156068829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x55f2a0d4b290 con 0x55f2a0915300 2024-05-27T07:50:54.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.816+0000 7f53957fa640 1 -- 172.21.15.114:0/156068829 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53a4003e20 con 0x55f2a0915300 2024-05-27T07:50:54.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.816+0000 7f53957fa640 1 -- 172.21.15.114:0/156068829 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 63) ==== 5910+0+0 (secure 0 0 0) 0x7f53a401d020 con 0x55f2a0915300 2024-05-27T07:50:54.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.816+0000 7f53baa59740 10 client.17979 fetch_fsmap finished waiting for FSMap version 63 2024-05-27T07:50:54.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.816+0000 7f53baa59740 10 client.17979 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:50:54.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.816+0000 7f53baa59740 20 client.17979 populate_metadata read hostname 'smithi114' 2024-05-27T07:50:54.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.816+0000 7f53baa59740 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f2a09004d0 0x55f2a0d4d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:54.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.816+0000 7f53baa59740 4 client.17979 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:50:54.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.816+0000 7f53baa59740 1 -- 172.21.15.114:0/156068829 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f2a0c30360 con 0x55f2a09004d0 2024-05-27T07:50:54.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.816+0000 7f53b1143640 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f2a09004d0 0x55f2a0d4d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:54.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.817+0000 7f53957fa640 10 client.17979 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:54.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:54.817+0000 7f53b1143640 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f2a09004d0 0x55f2a0d4d8d0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b173df8996f66f4 server_cookie=42d9b928bcb35c9 in_seq=0 out_seq=0 2024-05-27T07:50:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.189+0000 7f53957fa640 1 -- 172.21.15.114:0/156068829 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f2a0c30360 con 0x55f2a09004d0 2024-05-27T07:50:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.189+0000 7f53957fa640 10 client.17979 handle_command_reply: tid=0 2024-05-27T07:50:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.190+0000 7f53baa59740 1 client.17979 shutdown 2024-05-27T07:50:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.190+0000 7f53baa59740 1 -- 172.21.15.114:0/156068829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 msgr2=0x55f2a0d43b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.190+0000 7f53baa59740 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 0x55f2a0d43b00 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f53a40116c0 tx=0x7f53a40050d0 comp rx=0 tx=0).stop 2024-05-27T07:50:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.190+0000 7f53baa59740 1 -- 172.21.15.114:0/156068829 shutdown_connections 2024-05-27T07:50:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.190+0000 7f53baa59740 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f2a09004d0 0x55f2a0d4d8d0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.190+0000 7f53baa59740 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f2a0915300 0x55f2a0d43b00 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.190+0000 7f53baa59740 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2a08f82f0 0x55f2a0cd2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.190+0000 7f53baa59740 1 --2- 172.21.15.114:0/156068829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2a08dfd80 0x55f2a0cd1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.190+0000 7f53baa59740 1 -- 172.21.15.114:0/156068829 >> 172.21.15.114:0/156068829 conn(0x55f2a0cc8300 msgr2=0x55f2a0ccbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.190+0000 7f53957fa640 0 client.17979 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:50:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.190+0000 7f53baa59740 1 -- 172.21.15.114:0/156068829 shutdown_connections 2024-05-27T07:50:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.190+0000 7f53baa59740 1 -- 172.21.15.114:0/156068829 wait complete. 2024-05-27T07:50:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.190+0000 7f53baa59740 20 client.17979 trim_cache size 0 max 16384 2024-05-27T07:50:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:50:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:50:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:50:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.191+0000 7f53b3ba8640 1 -- 172.21.15.114:0/260199792 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f53980b2950 msgr2=0x7f53980b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.191+0000 7f53b3ba8640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f53980b2950 0x7f53980b4e60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f53a80a6ed0 tx=0x7f53a806f6a0 comp rx=0 tx=0).stop 2024-05-27T07:50:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.191+0000 7f53b3ba8640 1 -- 172.21.15.114:0/260199792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53ac1021c0 msgr2=0x7f53ac1a0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.191+0000 7f53b3ba8640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53ac1021c0 0x7f53ac1a0570 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f53ac06a240 tx=0x7f53a800ee50 comp rx=0 tx=0).stop 2024-05-27T07:50:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.191+0000 7f53b3ba8640 1 -- 172.21.15.114:0/260199792 shutdown_connections 2024-05-27T07:50:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.191+0000 7f53b3ba8640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f53980b2950 0x7f53980b4e60 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.191+0000 7f53b3ba8640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53ac10e6c0 0x7f53ac19a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.191+0000 7f53b3ba8640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac109d10 0x7f53ac1a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.191+0000 7f53b3ba8640 1 --2- 172.21.15.114:0/260199792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53ac1021c0 0x7f53ac1a0570 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.191+0000 7f53b3ba8640 1 -- 172.21.15.114:0/260199792 >> 172.21.15.114:0/260199792 conn(0x7f53ac0779d0 msgr2=0x7f53ac10a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.191+0000 7f53b3ba8640 1 -- 172.21.15.114:0/260199792 shutdown_connections 2024-05-27T07:50:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:50:55.191+0000 7f53b3ba8640 1 -- 172.21.15.114:0/260199792 wait complete. 2024-05-27T07:50:55.290 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1103344 10.09 MB/sec execute 427 sec latency 158.391 ms 2024-05-27T07:50:55.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:55 smithi163 ceph-mon[36451]: pgmap v369: 129 pgs: 129 active+clean; 73 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 255 op/s 2024-05-27T07:50:55.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:55 smithi186 ceph-mon[36647]: pgmap v369: 129 pgs: 129 active+clean; 73 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 255 op/s 2024-05-27T07:50:56.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:55 smithi114 ceph-mon[32038]: pgmap v369: 129 pgs: 129 active+clean; 73 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 255 op/s 2024-05-27T07:50:56.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1105116 10.10 MB/sec execute 428 sec latency 19.292 ms 2024-05-27T07:50:56.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.521+0000 7f139b140700 1 -- 172.21.15.114:0/2469377227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f139410b270 msgr2=0x7f139411d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:56.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.521+0000 7f139b140700 1 --2- 172.21.15.114:0/2469377227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f139410b270 0x7f139411d3e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1384009990 tx=0x7f1384009ca0 comp rx=0 tx=0).stop 2024-05-27T07:50:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.523+0000 7f139b140700 1 -- 172.21.15.114:0/2469377227 shutdown_connections 2024-05-27T07:50:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.523+0000 7f139b140700 1 --2- 172.21.15.114:0/2469377227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f139411d920 0x7f139411fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.523+0000 7f139b140700 1 --2- 172.21.15.114:0/2469377227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139410a920 0x7f139410ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:56.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.523+0000 7f139b140700 1 --2- 172.21.15.114:0/2469377227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f139410b270 0x7f139411d3e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:56.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.524+0000 7f139b140700 1 -- 172.21.15.114:0/2469377227 >> 172.21.15.114:0/2469377227 conn(0x7f139406f4c0 msgr2=0x7f139406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:56.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.525+0000 7f139b140700 1 -- 172.21.15.114:0/2469377227 shutdown_connections 2024-05-27T07:50:56.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.525+0000 7f139b140700 1 -- 172.21.15.114:0/2469377227 wait complete. 2024-05-27T07:50:56.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.526+0000 7f139b140700 1 Processor -- start 2024-05-27T07:50:56.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.527+0000 7f139b140700 1 -- start start 2024-05-27T07:50:56.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.527+0000 7f139b140700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f139410a920 0x7f13941af820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:56.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.527+0000 7f139b140700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f139410b270 0x7f13941afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:56.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.527+0000 7f139b140700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139411d920 0x7f13941a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:56.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.528+0000 7f139b140700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13941232e0 con 0x7f139410b270 2024-05-27T07:50:56.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.528+0000 7f139b140700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1394123160 con 0x7f139411d920 2024-05-27T07:50:56.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.528+0000 7f139b140700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1394123460 con 0x7f139410a920 2024-05-27T07:50:56.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.531+0000 7f13996dd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139411d920 0x7f13941a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:56.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.532+0000 7f1393fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f139410b270 0x7f13941afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:56.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.532+0000 7f1393fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f139410b270 0x7f13941afd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36504/0 (socket says 172.21.15.114:36504) 2024-05-27T07:50:56.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.532+0000 7f1393fff700 1 -- 172.21.15.114:0/2016863920 learned_addr learned my addr 172.21.15.114:0/2016863920 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:50:56.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.534+0000 7f1393fff700 1 -- 172.21.15.114:0/2016863920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f139410a920 msgr2=0x7f13941af820 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:50:56.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.534+0000 7f1393fff700 1 --2- 172.21.15.114:0/2016863920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f139410a920 0x7f13941af820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:56.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.534+0000 7f1393fff700 1 -- 172.21.15.114:0/2016863920 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139411d920 msgr2=0x7f13941a9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:56.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.534+0000 7f1393fff700 1 --2- 172.21.15.114:0/2016863920 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139411d920 0x7f13941a9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:56.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.534+0000 7f1393fff700 1 -- 172.21.15.114:0/2016863920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13941aa0f0 con 0x7f139410b270 2024-05-27T07:50:56.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.535+0000 7f1393fff700 1 --2- 172.21.15.114:0/2016863920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f139410b270 0x7f13941afd60 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f138400b5c0 tx=0x7f13840057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:56.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.535+0000 7f1391ffb700 1 -- 172.21.15.114:0/2016863920 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384027070 con 0x7f139410b270 2024-05-27T07:50:56.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.535+0000 7f139b140700 1 -- 172.21.15.114:0/2016863920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13941aa380 con 0x7f139410b270 2024-05-27T07:50:56.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.536+0000 7f139b140700 1 -- 172.21.15.114:0/2016863920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13941aa860 con 0x7f139410b270 2024-05-27T07:50:56.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.536+0000 7f1391ffb700 1 -- 172.21.15.114:0/2016863920 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1384004060 con 0x7f139410b270 2024-05-27T07:50:56.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.536+0000 7f1391ffb700 1 -- 172.21.15.114:0/2016863920 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384022d50 con 0x7f139410b270 2024-05-27T07:50:56.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.543+0000 7f139b140700 1 -- 172.21.15.114:0/2016863920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1380005360 con 0x7f139410b270 2024-05-27T07:50:56.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.545+0000 7f1391ffb700 1 -- 172.21.15.114:0/2016863920 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f1384022470 con 0x7f139410b270 2024-05-27T07:50:56.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.546+0000 7f1391ffb700 1 --2- 172.21.15.114:0/2016863920 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f137c0b2bc0 0x7f137c0b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:50:56.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.546+0000 7f1391ffb700 1 -- 172.21.15.114:0/2016863920 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9466+0+0 (secure 0 0 0) 0x7f13840e3c60 con 0x7f139410b270 2024-05-27T07:50:56.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.547+0000 7f1398edc700 1 --2- 172.21.15.114:0/2016863920 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f137c0b2bc0 0x7f137c0b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:50:56.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.549+0000 7f1398edc700 1 --2- 172.21.15.114:0/2016863920 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f137c0b2bc0 0x7f137c0b50d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f13941aab80 tx=0x7f138800b3f0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:50:56.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.556+0000 7f1391ffb700 1 -- 172.21.15.114:0/2016863920 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1384013070 con 0x7f139410b270 2024-05-27T07:50:56.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.874+0000 7f139b140700 1 -- 172.21.15.114:0/2016863920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1380005920 con 0x7f139410b270 2024-05-27T07:50:56.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.874+0000 7f1391ffb700 1 -- 172.21.15.114:0/2016863920 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+8521 (secure 0 0 0) 0x7f13840ab2c0 con 0x7f139410b270 2024-05-27T07:50:56.876 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:50:56.876 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":63,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:50:42.702888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27383":{"gid":27383,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/3045713021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":3045713021},{"type":"v1","addr":"172.21.15.186:6833","nonce":3045713021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:50:56.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.877+0000 7f139b140700 1 -- 172.21.15.114:0/2016863920 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f137c0b2bc0 msgr2=0x7f137c0b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:56.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.877+0000 7f139b140700 1 --2- 172.21.15.114:0/2016863920 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f137c0b2bc0 0x7f137c0b50d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f13941aab80 tx=0x7f138800b3f0 comp rx=0 tx=0).stop 2024-05-27T07:50:56.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.877+0000 7f139b140700 1 -- 172.21.15.114:0/2016863920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f139410b270 msgr2=0x7f13941afd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:50:56.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.877+0000 7f139b140700 1 --2- 172.21.15.114:0/2016863920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f139410b270 0x7f13941afd60 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f138400b5c0 tx=0x7f13840057c0 comp rx=0 tx=0).stop 2024-05-27T07:50:56.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.877+0000 7f139b140700 1 -- 172.21.15.114:0/2016863920 shutdown_connections 2024-05-27T07:50:56.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.877+0000 7f139b140700 1 --2- 172.21.15.114:0/2016863920 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f137c0b2bc0 0x7f137c0b50d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:56.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.877+0000 7f139b140700 1 --2- 172.21.15.114:0/2016863920 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139411d920 0x7f13941a9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:56.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.877+0000 7f139b140700 1 --2- 172.21.15.114:0/2016863920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f139410a920 0x7f13941af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:56.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.877+0000 7f139b140700 1 --2- 172.21.15.114:0/2016863920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f139410b270 0x7f13941afd60 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:50:56.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.877+0000 7f139b140700 1 -- 172.21.15.114:0/2016863920 >> 172.21.15.114:0/2016863920 conn(0x7f139406f4c0 msgr2=0x7f139411b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:50:56.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.877+0000 7f139b140700 1 -- 172.21.15.114:0/2016863920 shutdown_connections 2024-05-27T07:50:56.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:50:56.877+0000 7f139b140700 1 -- 172.21.15.114:0/2016863920 wait complete. 2024-05-27T07:50:56.879 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 63 2024-05-27T07:50:57.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1106362 10.09 MB/sec execute 429 sec latency 97.654 ms 2024-05-27T07:50:57.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:50:57.938 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:50:57.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:57 smithi163 ceph-mon[36451]: pgmap v370: 129 pgs: 129 active+clean; 66 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 423 op/s 2024-05-27T07:50:57.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:57 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2016863920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:50:57.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:57 smithi186 ceph-mon[36647]: pgmap v370: 129 pgs: 129 active+clean; 66 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 423 op/s 2024-05-27T07:50:57.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:57 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2016863920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:50:58.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:57 smithi114 ceph-mon[32038]: pgmap v370: 129 pgs: 129 active+clean; 66 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 423 op/s 2024-05-27T07:50:58.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:57 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2016863920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:50:58.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1108358 10.10 MB/sec execute 430 sec latency 67.235 ms 2024-05-27T07:50:59.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:50:58 smithi114 ceph-mon[32038]: pgmap v371: 129 pgs: 129 active+clean; 66 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 262 op/s 2024-05-27T07:50:59.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:50:58 smithi163 ceph-mon[36451]: pgmap v371: 129 pgs: 129 active+clean; 66 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 262 op/s 2024-05-27T07:50:59.228 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:50:58 smithi186 ceph-mon[36647]: pgmap v371: 129 pgs: 129 active+clean; 66 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 262 op/s 2024-05-27T07:50:59.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1109728 10.09 MB/sec execute 431 sec latency 41.825 ms 2024-05-27T07:51:00.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1111897 10.09 MB/sec execute 432 sec latency 27.287 ms 2024-05-27T07:51:00.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.316+0000 7fa2d58c8700 1 -- 172.21.15.114:0/3024641925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d0109850 msgr2=0x7fa2d01111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:00.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.316+0000 7fa2d58c8700 1 --2- 172.21.15.114:0/3024641925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d0109850 0x7fa2d01111b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fa2c800b0d0 tx=0x7fa2c800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.317+0000 7fa2d58c8700 1 -- 172.21.15.114:0/3024641925 shutdown_connections 2024-05-27T07:51:00.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.317+0000 7fa2d58c8700 1 --2- 172.21.15.114:0/3024641925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d0109850 0x7fa2d01111b0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.317+0000 7fa2d58c8700 1 --2- 172.21.15.114:0/3024641925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d01045d0 0x7fa2d01049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.317+0000 7fa2d58c8700 1 --2- 172.21.15.114:0/3024641925 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2d0108f00 0x7fa2d0109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.317+0000 7fa2d58c8700 1 -- 172.21.15.114:0/3024641925 >> 172.21.15.114:0/3024641925 conn(0x7fa2d00ffe60 msgr2=0x7fa2d01022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:00.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.317+0000 7fa2d58c8700 1 -- 172.21.15.114:0/3024641925 shutdown_connections 2024-05-27T07:51:00.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.317+0000 7fa2d58c8700 1 -- 172.21.15.114:0/3024641925 wait complete. 2024-05-27T07:51:00.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.317+0000 7fa2d58c8700 1 Processor -- start 2024-05-27T07:51:00.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2d58c8700 1 -- start start 2024-05-27T07:51:00.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2d58c8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2d01045d0 0x7fa2d0075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2d58c8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d0108f00 0x7fa2d0075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2d58c8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d0109850 0x7fa2d0076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2d58c8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d0114480 con 0x7fa2d0109850 2024-05-27T07:51:00.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2d58c8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d0114300 con 0x7fa2d01045d0 2024-05-27T07:51:00.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2d58c8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d0114600 con 0x7fa2d0108f00 2024-05-27T07:51:00.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2cf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d0109850 0x7fa2d0076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2ce7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d0108f00 0x7fa2d0075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2cf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d0109850 0x7fa2d0076530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36514/0 (socket says 172.21.15.114:36514) 2024-05-27T07:51:00.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2ce7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d0108f00 0x7fa2d0075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47956/0 (socket says 172.21.15.114:47956) 2024-05-27T07:51:00.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2cf7fe700 1 -- 172.21.15.114:0/4176799474 learned_addr learned my addr 172.21.15.114:0/4176799474 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:00.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2d01045d0 0x7fa2d0075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2cf7fe700 1 -- 172.21.15.114:0/4176799474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d0108f00 msgr2=0x7fa2d0075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:00.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2cf7fe700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d0108f00 0x7fa2d0075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2cf7fe700 1 -- 172.21.15.114:0/4176799474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2d01045d0 msgr2=0x7fa2d0075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:00.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2cf7fe700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2d01045d0 0x7fa2d0075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2cf7fe700 1 -- 172.21.15.114:0/4176799474 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d01a91b0 con 0x7fa2d0109850 2024-05-27T07:51:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2d01045d0 0x7fa2d0075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.318+0000 7fa2ce7fc700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d0108f00 0x7fa2d0075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.319+0000 7fa2cf7fe700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d0109850 0x7fa2d0076530 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009850 tx=0x7fa2c8007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.319+0000 7fa2d48c6700 1 -- 172.21.15.114:0/4176799474 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c801f070 con 0x7fa2d0109850 2024-05-27T07:51:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.319+0000 7fa2d58c8700 1 -- 172.21.15.114:0/4176799474 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2d01a9440 con 0x7fa2d0109850 2024-05-27T07:51:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.319+0000 7fa2d48c6700 1 -- 172.21.15.114:0/4176799474 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c8004c50 con 0x7fa2d0109850 2024-05-27T07:51:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.319+0000 7fa2d48c6700 1 -- 172.21.15.114:0/4176799474 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c801b7b0 con 0x7fa2d0109850 2024-05-27T07:51:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.319+0000 7fa2d58c8700 1 -- 172.21.15.114:0/4176799474 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2d01a9890 con 0x7fa2d0109850 2024-05-27T07:51:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.322+0000 7fa2d48c6700 1 -- 172.21.15.114:0/4176799474 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa2c8022070 con 0x7fa2d0109850 2024-05-27T07:51:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.322+0000 7fa2e42ddb80 1 Processor -- start 2024-05-27T07:51:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2e42ddb80 1 -- start start 2024-05-27T07:51:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2d48c6700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa2b80b29f0 0x7fa2b80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2ce7fc700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa2b80b29f0 0x7fa2b80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2e42ddb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603b2ee40 0x555603cb3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2e42ddb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603cb4070 0x555603cadbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2e42ddb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cae120 0x555603cae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2e42ddb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555603bdbd70 con 0x555603cb4070 2024-05-27T07:51:00.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2e42ddb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555603bcb910 con 0x555603cae120 2024-05-27T07:51:00.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2e42ddb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555603bcf950 con 0x555603b2ee40 2024-05-27T07:51:00.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2d48c6700 1 -- 172.21.15.114:0/4176799474 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9466+0+0 (secure 0 0 0) 0x7fa2c800f020 con 0x7fa2d0109850 2024-05-27T07:51:00.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2ce7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603cb4070 0x555603cadbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2ceffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603b2ee40 0x555603cb3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2ce7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603cb4070 0x555603cadbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36524/0 (socket says 172.21.15.114:36524) 2024-05-27T07:51:00.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2ce7fc700 1 -- 172.21.15.114:0/3837477531 learned_addr learned my addr 172.21.15.114:0/3837477531 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:00.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2cf7fe700 1 --2- 172.21.15.114:0/3837477531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cae120 0x555603cae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2ceffd700 1 -- 172.21.15.114:0/3837477531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cae120 msgr2=0x555603cae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:00.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/3837477531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cae120 0x555603cae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2ceffd700 1 -- 172.21.15.114:0/3837477531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603cb4070 msgr2=0x555603cadbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:00.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/3837477531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603cb4070 0x555603cadbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2ceffd700 1 -- 172.21.15.114:0/3837477531 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555603b74fb0 con 0x555603b2ee40 2024-05-27T07:51:00.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.323+0000 7fa2ce7fc700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa2b80b29f0 0x7fa2b80b4f00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4012880 tx=0x7fa2c4012c40 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:00.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.324+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/3837477531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603b2ee40 0x555603cb3b30 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0009cc0 tx=0x7fa2c00113e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=624d3599ecc2c53 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:00.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.324+0000 7fa2b5ffb700 1 -- 172.21.15.114:0/3837477531 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0011c30 con 0x555603b2ee40 2024-05-27T07:51:00.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.324+0000 7fa2b5ffb700 1 -- 172.21.15.114:0/3837477531 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c0011dd0 con 0x555603b2ee40 2024-05-27T07:51:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.324+0000 7fa2b5ffb700 1 -- 172.21.15.114:0/3837477531 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c000f6a0 con 0x555603b2ee40 2024-05-27T07:51:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.325+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/3837477531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603b2ee40 msgr2=0x555603cb3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.325+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/3837477531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603b2ee40 0x555603cb3b30 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0009cc0 tx=0x7fa2c00113e0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.325+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/3837477531 shutdown_connections 2024-05-27T07:51:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.325+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/3837477531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cae120 0x555603cae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.325+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/3837477531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603b2ee40 0x555603cb3b30 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.325+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/3837477531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603cb4070 0x555603cadbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.325+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/3837477531 >> 172.21.15.114:0/3837477531 conn(0x555603b31770 msgr2=0x555603ca8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.325+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/3837477531 shutdown_connections 2024-05-27T07:51:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.325+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/3837477531 wait complete. 2024-05-27T07:51:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.325+0000 7fa2e42ddb80 1 -- start start 2024-05-27T07:51:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2e42ddb80 1 Processor -- start 2024-05-27T07:51:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2e42ddb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603b2ee40 0x555603cb7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2e42ddb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cae120 0x555603cb7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2e42ddb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603cb4070 0x555603cb82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2e42ddb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555603ad1fa0 con 0x555603cb4070 2024-05-27T07:51:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2e42ddb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555603b33da0 con 0x555603cae120 2024-05-27T07:51:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2e42ddb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555603cbab60 con 0x555603b2ee40 2024-05-27T07:51:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2e42ddb80 10 client.0 _ll_register_callbacks cb 0x555603b625b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2cf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603cb4070 0x555603cb82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2cf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603cb4070 0x555603cb82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36532/0 (socket says 172.21.15.114:36532) 2024-05-27T07:51:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2cf7fe700 1 -- 172.21.15.114:0/2628176955 learned_addr learned my addr 172.21.15.114:0/2628176955 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/2628176955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603b2ee40 0x555603cb7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2ce7fc700 1 --2- 172.21.15.114:0/2628176955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cae120 0x555603cb7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2cf7fe700 1 -- 172.21.15.114:0/2628176955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603b2ee40 msgr2=0x555603cb7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2cf7fe700 1 --2- 172.21.15.114:0/2628176955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603b2ee40 0x555603cb7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2cf7fe700 1 -- 172.21.15.114:0/2628176955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cae120 msgr2=0x555603cb7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2cf7fe700 1 --2- 172.21.15.114:0/2628176955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cae120 0x555603cb7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.326+0000 7fa2cf7fe700 1 -- 172.21.15.114:0/2628176955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555603b1b8d0 con 0x555603cb4070 2024-05-27T07:51:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2cf7fe700 1 --2- 172.21.15.114:0/2628176955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603cb4070 0x555603cb82e0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8007f40 tx=0x7fa2c80a7020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a04981cbf2ce6da server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2aeffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2aeffd700 1 -- 172.21.15.114:0/2628176955 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8004e00 con 0x555603cb4070 2024-05-27T07:51:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2aeffd700 1 -- 172.21.15.114:0/2628176955 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9466+0+0 (secure 0 0 0) 0x7fa2c80b3630 con 0x555603cb4070 2024-05-27T07:51:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/2628176955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2c0011dd0 con 0x555603cb4070 2024-05-27T07:51:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2aeffd700 1 -- 172.21.15.114:0/2628176955 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fa2c80aa5f0 con 0x555603cb4070 2024-05-27T07:51:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2e42ddb80 10 client.17997 fetch_fsmap learned FSMap version 63 2024-05-27T07:51:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/2628176955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x555603b85760 con 0x555603cb4070 2024-05-27T07:51:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2aeffd700 1 -- 172.21.15.114:0/2628176955 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c0011dd0 con 0x555603cb4070 2024-05-27T07:51:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2aeffd700 1 -- 172.21.15.114:0/2628176955 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 63) ==== 5910+0+0 (secure 0 0 0) 0x7fa2c8011070 con 0x555603cb4070 2024-05-27T07:51:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2e42ddb80 10 client.17997 fetch_fsmap finished waiting for FSMap version 63 2024-05-27T07:51:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2e42ddb80 10 client.17997 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2e42ddb80 20 client.17997 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/2628176955 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555603cc7a80 0x555603cc9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2e42ddb80 4 client.17997 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.327+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/2628176955 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555603bddcc0 con 0x555603cc7a80 2024-05-27T07:51:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.328+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/2628176955 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555603cc7a80 0x555603cc9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.328+0000 7fa2aeffd700 10 client.17997 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.328+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/2628176955 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555603cc7a80 0x555603cc9f00 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=763424c259e3e937 server_cookie=86cd721a18c9e7a in_seq=0 out_seq=0 2024-05-27T07:51:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2aeffd700 1 -- 172.21.15.114:0/2628176955 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555603bddcc0 con 0x555603cc7a80 2024-05-27T07:51:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2aeffd700 10 client.17997 handle_command_reply: tid=0 2024-05-27T07:51:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2e42ddb80 1 client.17997 shutdown 2024-05-27T07:51:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/2628176955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603cb4070 msgr2=0x555603cb82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/2628176955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603cb4070 0x555603cb82e0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8007f40 tx=0x7fa2c80a7020 comp rx=0 tx=0).stop 2024-05-27T07:51:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/2628176955 shutdown_connections 2024-05-27T07:51:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/2628176955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603cb4070 0x555603cb82e0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/2628176955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603b2ee40 0x555603cb7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/2628176955 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555603cc7a80 0x555603cc9f00 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/2628176955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cae120 0x555603cb7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/2628176955 >> 172.21.15.114:0/2628176955 conn(0x555603b31770 msgr2=0x555603cacbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:00.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2aeffd700 0 client.17997 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:00.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/2628176955 shutdown_connections 2024-05-27T07:51:00.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.329+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/2628176955 wait complete. 2024-05-27T07:51:00.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.330+0000 7fa2e42ddb80 20 client.17997 trim_cache size 0 max 16384 2024-05-27T07:51:00.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.346+0000 7fa2e42ddb80 1 Processor -- start 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.346+0000 7fa2e42ddb80 1 -- start start 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.346+0000 7fa2e42ddb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 0x555603cadbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.346+0000 7fa2e42ddb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603cae120 0x555603cb8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.346+0000 7fa2e42ddb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cb4070 0x555603cb2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.346+0000 7fa2e42ddb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d01b3fc0 con 0x555603b2ee40 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.346+0000 7fa2e42ddb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d01b2730 con 0x555603cb4070 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.346+0000 7fa2e42ddb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d01b0580 con 0x555603cae120 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.346+0000 7fa2ceffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 0x555603cadbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.346+0000 7fa2ceffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 0x555603cadbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36538/0 (socket says 172.21.15.114:36538) 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.346+0000 7fa2ce7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603cae120 0x555603cb8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.346+0000 7fa2ceffd700 1 -- 172.21.15.114:0/3767048006 learned_addr learned my addr 172.21.15.114:0/3767048006 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.346+0000 7fa2cf7fe700 1 --2- 172.21.15.114:0/3767048006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cb4070 0x555603cb2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.347+0000 7fa2ceffd700 1 -- 172.21.15.114:0/3767048006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603cae120 msgr2=0x555603cb8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.347+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/3767048006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603cae120 0x555603cb8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.347+0000 7fa2ceffd700 1 -- 172.21.15.114:0/3767048006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cb4070 msgr2=0x555603cb2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:00.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.347+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/3767048006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cb4070 0x555603cb2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.347+0000 7fa2ceffd700 1 -- 172.21.15.114:0/3767048006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555603cb5dd0 con 0x555603b2ee40 2024-05-27T07:51:00.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.347+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/3767048006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 0x555603cadbe0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0013e30 tx=0x7fa2c00113e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7ef3ad5164fd15 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:00.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.347+0000 7fa2b57fa700 1 -- 172.21.15.114:0/3767048006 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0030030 con 0x555603b2ee40 2024-05-27T07:51:00.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.347+0000 7fa2b57fa700 1 -- 172.21.15.114:0/3767048006 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c0011d70 con 0x555603b2ee40 2024-05-27T07:51:00.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.347+0000 7fa2b57fa700 1 -- 172.21.15.114:0/3767048006 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0013e60 con 0x555603b2ee40 2024-05-27T07:51:00.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.348+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/3767048006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 msgr2=0x555603cadbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:00.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.348+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/3767048006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 0x555603cadbe0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0013e30 tx=0x7fa2c00113e0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.348+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/3767048006 shutdown_connections 2024-05-27T07:51:00.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.348+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/3767048006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cb4070 0x555603cb2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.348+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/3767048006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 0x555603cadbe0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.348+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/3767048006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603cae120 0x555603cb8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.348+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/3767048006 >> 172.21.15.114:0/3767048006 conn(0x555603b31770 msgr2=0x555603cb6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.348+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/3767048006 shutdown_connections 2024-05-27T07:51:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.348+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/3767048006 wait complete. 2024-05-27T07:51:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.348+0000 7fa2e42ddb80 1 -- start start 2024-05-27T07:51:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2e42ddb80 1 Processor -- start 2024-05-27T07:51:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2e42ddb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 0x555603cbfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2e42ddb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603cae120 0x555603cc0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2e42ddb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cb4070 0x555603cce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2e42ddb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d01b0bf0 con 0x555603b2ee40 2024-05-27T07:51:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2e42ddb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d01b2060 con 0x555603cb4070 2024-05-27T07:51:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2e42ddb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d01b12c0 con 0x555603cae120 2024-05-27T07:51:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2ceffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 0x555603cbfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2e42ddb80 10 client.0 _ll_register_callbacks cb 0x555603b625b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2ceffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 0x555603cbfdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36552/0 (socket says 172.21.15.114:36552) 2024-05-27T07:51:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2ceffd700 1 -- 172.21.15.114:0/543195012 learned_addr learned my addr 172.21.15.114:0/543195012 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2ce7fc700 1 --2- 172.21.15.114:0/543195012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603cae120 0x555603cc0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2cf7fe700 1 --2- 172.21.15.114:0/543195012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cb4070 0x555603cce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2ceffd700 1 -- 172.21.15.114:0/543195012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603cae120 msgr2=0x555603cc0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/543195012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603cae120 0x555603cc0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2ceffd700 1 -- 172.21.15.114:0/543195012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cb4070 msgr2=0x555603cce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/543195012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cb4070 0x555603cce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.349+0000 7fa2ceffd700 1 -- 172.21.15.114:0/543195012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555603ccee20 con 0x555603b2ee40 2024-05-27T07:51:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2ceffd700 1 --2- 172.21.15.114:0/543195012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 0x555603cbfdf0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00058c0 tx=0x7fa2c00052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90af2518729ea289 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2aeffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:00.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2aeffd700 1 -- 172.21.15.114:0/543195012 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0039070 con 0x555603b2ee40 2024-05-27T07:51:00.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2aeffd700 1 -- 172.21.15.114:0/543195012 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9466+0+0 (secure 0 0 0) 0x7fa2c0003880 con 0x555603b2ee40 2024-05-27T07:51:00.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/543195012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2c0011d70 con 0x555603b2ee40 2024-05-27T07:51:00.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2aeffd700 1 -- 172.21.15.114:0/543195012 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fa2c0045050 con 0x555603b2ee40 2024-05-27T07:51:00.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2e42ddb80 10 client.18003 fetch_fsmap learned FSMap version 63 2024-05-27T07:51:00.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/543195012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x555603d31df0 con 0x555603b2ee40 2024-05-27T07:51:00.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2aeffd700 1 -- 172.21.15.114:0/543195012 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c0011d70 con 0x555603b2ee40 2024-05-27T07:51:00.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2aeffd700 1 -- 172.21.15.114:0/543195012 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 63) ==== 5910+0+0 (secure 0 0 0) 0x7fa2c003fbe0 con 0x555603b2ee40 2024-05-27T07:51:00.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2e42ddb80 10 client.18003 fetch_fsmap finished waiting for FSMap version 63 2024-05-27T07:51:00.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2e42ddb80 10 client.18003 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:00.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2e42ddb80 20 client.18003 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:00.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/543195012 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555603d32380 0x555603d347a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:00.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2e42ddb80 4 client.18003 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:51:00.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.350+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/543195012 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555603bd74a0 con 0x555603d32380 2024-05-27T07:51:00.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.351+0000 7fa2ce7fc700 1 --2- 172.21.15.114:0/543195012 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555603d32380 0x555603d347a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:00.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.351+0000 7fa2aeffd700 10 client.18003 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:00.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.351+0000 7fa2ce7fc700 1 --2- 172.21.15.114:0/543195012 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555603d32380 0x555603d347a0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2eb5c56884903ceb server_cookie=5478ec24cbcc8ef1 in_seq=0 out_seq=0 2024-05-27T07:51:00.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2aeffd700 1 -- 172.21.15.114:0/543195012 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555603bd74a0 con 0x555603d32380 2024-05-27T07:51:00.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2aeffd700 10 client.18003 handle_command_reply: tid=0 2024-05-27T07:51:00.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2e42ddb80 1 client.18003 shutdown 2024-05-27T07:51:00.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/543195012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 msgr2=0x555603cbfdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:00.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/543195012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 0x555603cbfdf0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00058c0 tx=0x7fa2c00052f0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/543195012 shutdown_connections 2024-05-27T07:51:00.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/543195012 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555603d32380 0x555603d347a0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/543195012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555603cb4070 0x555603cce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/543195012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555603b2ee40 0x555603cbfdf0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2e42ddb80 1 --2- 172.21.15.114:0/543195012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555603cae120 0x555603cc0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/543195012 >> 172.21.15.114:0/543195012 conn(0x555603b31770 msgr2=0x555603cb7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:00.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2aeffd700 0 client.18003 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:00.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/543195012 shutdown_connections 2024-05-27T07:51:00.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2e42ddb80 1 -- 172.21.15.114:0/543195012 wait complete. 2024-05-27T07:51:00.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.353+0000 7fa2e42ddb80 20 client.18003 trim_cache size 0 max 16384 2024-05-27T07:51:00.359 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:51:00.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.354+0000 7fa2d58c8700 1 -- 172.21.15.114:0/4176799474 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa2b80b29f0 msgr2=0x7fa2b80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:00.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.354+0000 7fa2d58c8700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa2b80b29f0 0x7fa2b80b4f00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4012880 tx=0x7fa2c4012c40 comp rx=0 tx=0).stop 2024-05-27T07:51:00.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.354+0000 7fa2d58c8700 1 -- 172.21.15.114:0/4176799474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d0109850 msgr2=0x7fa2d0076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:00.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.354+0000 7fa2d58c8700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d0109850 0x7fa2d0076530 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009850 tx=0x7fa2c8007a20 comp rx=0 tx=0).stop 2024-05-27T07:51:00.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.354+0000 7fa2d58c8700 1 -- 172.21.15.114:0/4176799474 shutdown_connections 2024-05-27T07:51:00.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.354+0000 7fa2d58c8700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d0109850 0x7fa2d0076530 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.354+0000 7fa2d58c8700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa2b80b29f0 0x7fa2b80b4f00 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.354+0000 7fa2d58c8700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2d01045d0 0x7fa2d0075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.354+0000 7fa2d58c8700 1 --2- 172.21.15.114:0/4176799474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d0108f00 0x7fa2d0075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:00.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.354+0000 7fa2d58c8700 1 -- 172.21.15.114:0/4176799474 >> 172.21.15.114:0/4176799474 conn(0x7fa2d00ffe60 msgr2=0x7fa2d0106fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:00.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.354+0000 7fa2d58c8700 1 -- 172.21.15.114:0/4176799474 shutdown_connections 2024-05-27T07:51:00.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:00.354+0000 7fa2d58c8700 1 -- 172.21.15.114:0/4176799474 wait complete. 2024-05-27T07:51:01.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1112799 10.08 MB/sec execute 433 sec latency 112.237 ms 2024-05-27T07:51:01.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:01 smithi163 ceph-mon[36451]: pgmap v372: 129 pgs: 129 active+clean; 71 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2024-05-27T07:51:01.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:01 smithi186 ceph-mon[36647]: pgmap v372: 129 pgs: 129 active+clean; 71 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2024-05-27T07:51:01.729 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f93b3048700 time 2024-05-27T07:51:01.505626+0000 2024-05-27T07:51:01.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:51:01.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:01.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f93c546fba6] 2024-05-27T07:51:01.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93c546fdd1] 2024-05-27T07:51:01.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56528ee5223d] 2024-05-27T07:51:01.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56528ee3d971] 2024-05-27T07:51:01.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 5: (SnapRealm::check_cache() const+0x13e) [0x56528ee3bfa0] 2024-05-27T07:51:01.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56528ee3c651] 2024-05-27T07:51:01.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56528edd1347] 2024-05-27T07:51:01.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56528eacd87e] 2024-05-27T07:51:01.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56528ead2e94] 2024-05-27T07:51:01.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56528eadd969] 2024-05-27T07:51:01.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 11: (MDLog::_replay_thread()+0x1f38) [0x56528ee7648c] 2024-05-27T07:51:01.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56528ee7f9cb] 2024-05-27T07:51:01.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 13: (Thread::entry_wrapper()+0x43) [0x7f93c543e72d] 2024-05-27T07:51:01.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 14: (Thread::_entry_func(void*)+0xd) [0x7f93c543e749] 2024-05-27T07:51:01.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f93c3fe21ca] 2024-05-27T07:51:01.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 16: clone() 2024-05-27T07:51:01.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.507+0000 7f93b3048700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f93b3048700 time 2024-05-27T07:51:01.505626+0000 2024-05-27T07:51:01.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:51:01.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:01.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f93c546fba6] 2024-05-27T07:51:01.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93c546fdd1] 2024-05-27T07:51:01.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56528ee5223d] 2024-05-27T07:51:01.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56528ee3d971] 2024-05-27T07:51:01.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 5: (SnapRealm::check_cache() const+0x13e) [0x56528ee3bfa0] 2024-05-27T07:51:01.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56528ee3c651] 2024-05-27T07:51:01.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56528edd1347] 2024-05-27T07:51:01.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56528eacd87e] 2024-05-27T07:51:01.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56528ead2e94] 2024-05-27T07:51:01.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56528eadd969] 2024-05-27T07:51:01.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 11: (MDLog::_replay_thread()+0x1f38) [0x56528ee7648c] 2024-05-27T07:51:01.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56528ee7f9cb] 2024-05-27T07:51:01.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 13: (Thread::entry_wrapper()+0x43) [0x7f93c543e72d] 2024-05-27T07:51:01.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 14: (Thread::_entry_func(void*)+0xd) [0x7f93c543e749] 2024-05-27T07:51:01.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f93c3fe21ca] 2024-05-27T07:51:01.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 16: clone() 2024-05-27T07:51:01.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: *** Caught signal (Aborted) ** 2024-05-27T07:51:01.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: in thread 7f93b3048700 thread_name:md_log_replay 2024-05-27T07:51:01.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:01.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93c3fecd20] 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2: gsignal() 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 3: abort() 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f93c546fced] 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93c546fdd1] 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56528ee5223d] 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56528ee3d971] 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 8: (SnapRealm::check_cache() const+0x13e) [0x56528ee3bfa0] 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56528ee3c651] 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56528edd1347] 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56528eacd87e] 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56528ead2e94] 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56528eadd969] 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 14: (MDLog::_replay_thread()+0x1f38) [0x56528ee7648c] 2024-05-27T07:51:01.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56528ee7f9cb] 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 16: (Thread::entry_wrapper()+0x43) [0x7f93c543e72d] 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 17: (Thread::_entry_func(void*)+0xd) [0x7f93c543e749] 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f93c3fe21ca] 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 19: clone() 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.509+0000 7f93b3048700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: in thread 7f93b3048700 thread_name:md_log_replay 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93c3fecd20] 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2: gsignal() 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 3: abort() 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f93c546fced] 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93c546fdd1] 2024-05-27T07:51:01.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56528ee5223d] 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56528ee3d971] 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 8: (SnapRealm::check_cache() const+0x13e) [0x56528ee3bfa0] 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56528ee3c651] 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56528edd1347] 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56528eacd87e] 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56528ead2e94] 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56528eadd969] 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 14: (MDLog::_replay_thread()+0x1f38) [0x56528ee7648c] 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56528ee7f9cb] 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 16: (Thread::entry_wrapper()+0x43) [0x7f93c543e72d] 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 17: (Thread::_entry_func(void*)+0xd) [0x7f93c543e749] 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f93c3fe21ca] 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 19: clone() 2024-05-27T07:51:01.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:51:01.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: -1> 2024-05-27T07:51:01.507+0000 7f93b3048700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f93b3048700 time 2024-05-27T07:51:01.505626+0000 2024-05-27T07:51:01.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:51:01.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:01.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f93c546fba6] 2024-05-27T07:51:01.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93c546fdd1] 2024-05-27T07:51:01.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56528ee5223d] 2024-05-27T07:51:01.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56528ee3d971] 2024-05-27T07:51:01.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 5: (SnapRealm::check_cache() const+0x13e) [0x56528ee3bfa0] 2024-05-27T07:51:01.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56528ee3c651] 2024-05-27T07:51:01.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56528edd1347] 2024-05-27T07:51:01.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56528eacd87e] 2024-05-27T07:51:01.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56528ead2e94] 2024-05-27T07:51:01.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56528eadd969] 2024-05-27T07:51:01.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 11: (MDLog::_replay_thread()+0x1f38) [0x56528ee7648c] 2024-05-27T07:51:01.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56528ee7f9cb] 2024-05-27T07:51:01.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 13: (Thread::entry_wrapper()+0x43) [0x7f93c543e72d] 2024-05-27T07:51:01.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 14: (Thread::_entry_func(void*)+0xd) [0x7f93c543e749] 2024-05-27T07:51:01.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f93c3fe21ca] 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 16: clone() 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 0> 2024-05-27T07:51:01.509+0000 7f93b3048700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: in thread 7f93b3048700 thread_name:md_log_replay 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93c3fecd20] 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2: gsignal() 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 3: abort() 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f93c546fced] 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93c546fdd1] 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56528ee5223d] 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56528ee3d971] 2024-05-27T07:51:01.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 8: (SnapRealm::check_cache() const+0x13e) [0x56528ee3bfa0] 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56528ee3c651] 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56528edd1347] 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56528eacd87e] 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56528ead2e94] 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56528eadd969] 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 14: (MDLog::_replay_thread()+0x1f38) [0x56528ee7648c] 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56528ee7f9cb] 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 16: (Thread::entry_wrapper()+0x43) [0x7f93c543e72d] 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 17: (Thread::_entry_func(void*)+0xd) [0x7f93c543e749] 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f93c3fe21ca] 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 19: clone() 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: -1> 2024-05-27T07:51:01.507+0000 7f93b3048700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f93b3048700 time 2024-05-27T07:51:01.505626+0000 2024-05-27T07:51:01.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:51:01.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:01.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f93c546fba6] 2024-05-27T07:51:01.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93c546fdd1] 2024-05-27T07:51:01.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56528ee5223d] 2024-05-27T07:51:01.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56528ee3d971] 2024-05-27T07:51:01.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 5: (SnapRealm::check_cache() const+0x13e) [0x56528ee3bfa0] 2024-05-27T07:51:01.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56528ee3c651] 2024-05-27T07:51:01.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56528edd1347] 2024-05-27T07:51:01.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56528eacd87e] 2024-05-27T07:51:01.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56528ead2e94] 2024-05-27T07:51:01.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56528eadd969] 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 11: (MDLog::_replay_thread()+0x1f38) [0x56528ee7648c] 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56528ee7f9cb] 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 13: (Thread::entry_wrapper()+0x43) [0x7f93c543e72d] 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 14: (Thread::_entry_func(void*)+0xd) [0x7f93c543e749] 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f93c3fe21ca] 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 16: clone() 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 0> 2024-05-27T07:51:01.509+0000 7f93b3048700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: in thread 7f93b3048700 thread_name:md_log_replay 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93c3fecd20] 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2: gsignal() 2024-05-27T07:51:01.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 3: abort() 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f93c546fced] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93c546fdd1] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56528ee5223d] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56528ee3d971] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 8: (SnapRealm::check_cache() const+0x13e) [0x56528ee3bfa0] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56528ee3c651] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56528edd1347] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56528eacd87e] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56528ead2e94] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56528eadd969] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 14: (MDLog::_replay_thread()+0x1f38) [0x56528ee7648c] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56528ee7f9cb] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 16: (Thread::entry_wrapper()+0x43) [0x7f93c543e72d] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 17: (Thread::_entry_func(void*)+0xd) [0x7f93c543e749] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f93c3fe21ca] 2024-05-27T07:51:01.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 19: clone() 2024-05-27T07:51:01.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:51:01.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[67084]: 2024-05-27T07:51:01.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:01 smithi186 podman[74945]: 2024-05-27 07:51:01.624075204 +0000 UTC m=+0.055117508 container died f84b3ce82512db54743ac267d2a282adc7bb82f1858b228e461af939e666bbec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-27T07:51:01.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:01 smithi114 ceph-mon[32038]: pgmap v372: 129 pgs: 129 active+clean; 71 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2024-05-27T07:51:02.101 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:51:02.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:51:02.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1112815 10.05 MB/sec execute 434 sec latency 256.213 ms 2024-05-27T07:51:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:02.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:02 smithi186 podman[74945]: 2024-05-27 07:51:02.649381104 +0000 UTC m=+1.080423389 container remove f84b3ce82512db54743ac267d2a282adc7bb82f1858b228e461af939e666bbec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD) 2024-05-27T07:51:02.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:02 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:51:03.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1112818 10.03 MB/sec execute 435 sec latency 448.313 ms 2024-05-27T07:51:03.466 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:03 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:51:03.466 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:03 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.865s CPU time. 2024-05-27T07:51:03.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:03 smithi114 ceph-mon[32038]: pgmap v373: 129 pgs: 129 active+clean; 73 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 327 op/s 2024-05-27T07:51:03.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:03 smithi114 ceph-mon[32038]: osdmap e121: 12 total, 12 up, 12 in 2024-05-27T07:51:03.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:03 smithi163 ceph-mon[36451]: pgmap v373: 129 pgs: 129 active+clean; 73 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 327 op/s 2024-05-27T07:51:03.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:03 smithi163 ceph-mon[36451]: osdmap e121: 12 total, 12 up, 12 in 2024-05-27T07:51:03.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:03 smithi186 ceph-mon[36647]: pgmap v373: 129 pgs: 129 active+clean; 73 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 327 op/s 2024-05-27T07:51:03.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:03 smithi186 ceph-mon[36647]: osdmap e121: 12 total, 12 up, 12 in 2024-05-27T07:51:04.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1112968 10.01 MB/sec execute 436 sec latency 452.882 ms 2024-05-27T07:51:05.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1113314 9.99 MB/sec execute 437 sec latency 171.926 ms 2024-05-27T07:51:05.616 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:51:05.616 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:51:05.616 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "f57f1ad3-8522-4259-99aa-99efe084dab9", 2024-05-27T07:51:05.616 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:51:05.616 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:51:05.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.831+0000 7f24258ca700 1 -- 172.21.15.114:0/3363218912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24201092f0 msgr2=0x7f2420109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:05.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.831+0000 7f24258ca700 1 --2- 172.21.15.114:0/3363218912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24201092f0 0x7f2420109700 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f2410009990 tx=0x7f2410009ca0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.831+0000 7f24258ca700 1 -- 172.21.15.114:0/3363218912 shutdown_connections 2024-05-27T07:51:05.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.831+0000 7f24258ca700 1 --2- 172.21.15.114:0/3363218912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24201031c0 0x7f2420111160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.831+0000 7f24258ca700 1 --2- 172.21.15.114:0/3363218912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24201092f0 0x7f2420109700 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.831+0000 7f24258ca700 1 --2- 172.21.15.114:0/3363218912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2420102870 0x7f2420102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.831+0000 7f24258ca700 1 -- 172.21.15.114:0/3363218912 >> 172.21.15.114:0/3363218912 conn(0x7f24200fe6e0 msgr2=0x7f2420100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:05.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.831+0000 7f24258ca700 1 -- 172.21.15.114:0/3363218912 shutdown_connections 2024-05-27T07:51:05.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.831+0000 7f24258ca700 1 -- 172.21.15.114:0/3363218912 wait complete. 2024-05-27T07:51:05.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.832+0000 7f24258ca700 1 Processor -- start 2024-05-27T07:51:05.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.832+0000 7f24258ca700 1 -- start start 2024-05-27T07:51:05.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.832+0000 7f24258ca700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2420102870 0x7f242019af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.832+0000 7f24258ca700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24201031c0 0x7f242019b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.832+0000 7f24258ca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24201092f0 0x7f242019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.832+0000 7f24258ca700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2420114580 con 0x7f24201092f0 2024-05-27T07:51:05.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.832+0000 7f24258ca700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2420114400 con 0x7f24201031c0 2024-05-27T07:51:05.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.832+0000 7f24258ca700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2420114700 con 0x7f2420102870 2024-05-27T07:51:05.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f241f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24201092f0 0x7f242019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f241f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24201092f0 0x7f242019f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34064/0 (socket says 172.21.15.114:34064) 2024-05-27T07:51:05.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f241f7fe700 1 -- 172.21.15.114:0/1126363651 learned_addr learned my addr 172.21.15.114:0/1126363651 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:05.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f241e7fc700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24201031c0 0x7f242019b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f241effd700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2420102870 0x7f242019af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f241f7fe700 1 -- 172.21.15.114:0/1126363651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2420102870 msgr2=0x7f242019af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f241f7fe700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2420102870 0x7f242019af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f241f7fe700 1 -- 172.21.15.114:0/1126363651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24201031c0 msgr2=0x7f242019b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f241f7fe700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24201031c0 0x7f242019b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f241f7fe700 1 -- 172.21.15.114:0/1126363651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242019fde0 con 0x7f24201092f0 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f241f7fe700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24201092f0 0x7f242019f740 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f241800ef30 tx=0x7f241800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f24248c8700 1 -- 172.21.15.114:0/1126363651 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24180092e0 con 0x7f24201092f0 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f24258ca700 1 -- 172.21.15.114:0/1126363651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24201a0070 con 0x7f24201092f0 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f24248c8700 1 -- 172.21.15.114:0/1126363651 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2418004bb0 con 0x7f24201092f0 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f24248c8700 1 -- 172.21.15.114:0/1126363651 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418004020 con 0x7f24201092f0 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.833+0000 7f24258ca700 1 -- 172.21.15.114:0/1126363651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24201a79f0 con 0x7f24201092f0 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.836+0000 7f24248c8700 1 -- 172.21.15.114:0/1126363651 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f241801b030 con 0x7f24201092f0 2024-05-27T07:51:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.837+0000 7f24248c8700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f24080b29f0 0x7f24080b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.837+0000 7f24248c8700 1 -- 172.21.15.114:0/1126363651 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9642+0+0 (secure 0 0 0) 0x7f24180e2560 con 0x7f24201092f0 2024-05-27T07:51:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.837+0000 7f241effd700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f24080b29f0 0x7f24080b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.837+0000 7f24342dfb80 1 Processor -- start 2024-05-27T07:51:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.837+0000 7f24342dfb80 1 -- start start 2024-05-27T07:51:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f241effd700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f24080b29f0 0x7f24080b4f00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f24100059d0 tx=0x7f2410005960 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f24342dfb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea676dec0 0x55cea68f1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f24342dfb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea68f1cd0 0x55cea68ebac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f24342dfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 0x55cea68ec410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f24342dfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cea681ad10 con 0x55cea68ec000 2024-05-27T07:51:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f24342dfb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55cea680a8b0 con 0x55cea676dec0 2024-05-27T07:51:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f24342dfb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cea680e8f0 con 0x55cea68f1cd0 2024-05-27T07:51:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f241e7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea68f1cd0 0x55cea68ebac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f241effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea676dec0 0x55cea68f1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f241e7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea68f1cd0 0x55cea68ebac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:48872/0 (socket says 172.21.15.114:48872) 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f241e7fc700 1 -- 172.21.15.114:0/3037294788 learned_addr learned my addr 172.21.15.114:0/3037294788 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f241f7fe700 1 --2- 172.21.15.114:0/3037294788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 0x55cea68ec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f241e7fc700 1 -- 172.21.15.114:0/3037294788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea676dec0 msgr2=0x55cea68f1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f241e7fc700 1 --2- 172.21.15.114:0/3037294788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea676dec0 0x55cea68f1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f241e7fc700 1 -- 172.21.15.114:0/3037294788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 msgr2=0x55cea68ec410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f241e7fc700 1 --2- 172.21.15.114:0/3037294788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 0x55cea68ec410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f241e7fc700 1 -- 172.21.15.114:0/3037294788 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cea675a950 con 0x55cea68f1cd0 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.838+0000 7f241effd700 1 --2- 172.21.15.114:0/3037294788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea676dec0 0x55cea68f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.839+0000 7f241e7fc700 1 --2- 172.21.15.114:0/3037294788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea68f1cd0 0x55cea68ebac0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2414009cf0 tx=0x7f2414011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d17569b5210b94a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.839+0000 7f2405ffb700 1 -- 172.21.15.114:0/3037294788 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414011d80 con 0x55cea68f1cd0 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.839+0000 7f2405ffb700 1 -- 172.21.15.114:0/3037294788 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2414004500 con 0x55cea68f1cd0 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.839+0000 7f2405ffb700 1 -- 172.21.15.114:0/3037294788 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414005280 con 0x55cea68f1cd0 2024-05-27T07:51:05.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.840+0000 7f24342dfb80 1 -- 172.21.15.114:0/3037294788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea68f1cd0 msgr2=0x55cea68ebac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:05.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.840+0000 7f24342dfb80 1 --2- 172.21.15.114:0/3037294788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea68f1cd0 0x55cea68ebac0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2414009cf0 tx=0x7f2414011440 comp rx=0 tx=0).stop 2024-05-27T07:51:05.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.840+0000 7f24342dfb80 1 -- 172.21.15.114:0/3037294788 shutdown_connections 2024-05-27T07:51:05.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.840+0000 7f24342dfb80 1 --2- 172.21.15.114:0/3037294788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 0x55cea68ec410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.840+0000 7f24342dfb80 1 --2- 172.21.15.114:0/3037294788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea676dec0 0x55cea68f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.840+0000 7f24342dfb80 1 --2- 172.21.15.114:0/3037294788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea68f1cd0 0x55cea68ebac0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.840+0000 7f24342dfb80 1 -- 172.21.15.114:0/3037294788 >> 172.21.15.114:0/3037294788 conn(0x55cea67c8b70 msgr2=0x55cea68e66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:05.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.840+0000 7f24342dfb80 1 -- 172.21.15.114:0/3037294788 shutdown_connections 2024-05-27T07:51:05.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.840+0000 7f24342dfb80 1 -- 172.21.15.114:0/3037294788 wait complete. 2024-05-27T07:51:05.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.840+0000 7f24342dfb80 1 -- start start 2024-05-27T07:51:05.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.841+0000 7f24342dfb80 1 Processor -- start 2024-05-27T07:51:05.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.841+0000 7f24342dfb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea676dec0 0x55cea68f5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.841+0000 7f24342dfb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea68ec000 0x55cea68f5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.841+0000 7f24342dfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68f1cd0 0x55cea68f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.841+0000 7f24342dfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cea6711020 con 0x55cea68f1cd0 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.841+0000 7f24342dfb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55cea67ac840 con 0x55cea68ec000 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.841+0000 7f24342dfb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cea68f8810 con 0x55cea676dec0 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.841+0000 7f241effd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea676dec0 0x55cea68f5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.841+0000 7f241f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68f1cd0 0x55cea68f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.841+0000 7f24342dfb80 10 client.0 _ll_register_callbacks cb 0x55cea67a59a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.841+0000 7f241effd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea676dec0 0x55cea68f5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:48888/0 (socket says 172.21.15.114:48888) 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.841+0000 7f241effd700 1 -- 172.21.15.114:0/712925755 learned_addr learned my addr 172.21.15.114:0/712925755 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.841+0000 7f241e7fc700 1 --2- 172.21.15.114:0/712925755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea68ec000 0x55cea68f5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f241f7fe700 1 -- 172.21.15.114:0/712925755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea676dec0 msgr2=0x55cea68f5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f241f7fe700 1 --2- 172.21.15.114:0/712925755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea676dec0 0x55cea68f5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f241f7fe700 1 -- 172.21.15.114:0/712925755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea68ec000 msgr2=0x55cea68f5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f241f7fe700 1 --2- 172.21.15.114:0/712925755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea68ec000 0x55cea68f5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f241f7fe700 1 -- 172.21.15.114:0/712925755 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cea67a9710 con 0x55cea68f1cd0 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f241f7fe700 1 --2- 172.21.15.114:0/712925755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68f1cd0 0x55cea68f5ed0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f24180df850 tx=0x7f2418003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c2d5ab9fa9e190 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f23feffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f23feffd700 1 -- 172.21.15.114:0/712925755 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24180a9b70 con 0x55cea68f1cd0 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f23feffd700 1 -- 172.21.15.114:0/712925755 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9642+0+0 (secure 0 0 0) 0x7f24180e2360 con 0x55cea68f1cd0 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f24342dfb80 1 -- 172.21.15.114:0/712925755 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2414004500 con 0x55cea68f1cd0 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f23feffd700 1 -- 172.21.15.114:0/712925755 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f24180a9470 con 0x55cea68f1cd0 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f24342dfb80 10 client.18015 fetch_fsmap learned FSMap version 63 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f24342dfb80 1 -- 172.21.15.114:0/712925755 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x55cea69032e0 con 0x55cea68f1cd0 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.842+0000 7f23feffd700 1 -- 172.21.15.114:0/712925755 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2414004500 con 0x55cea68f1cd0 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.843+0000 7f23feffd700 1 -- 172.21.15.114:0/712925755 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 63) ==== 5910+0+0 (secure 0 0 0) 0x7f24180a3b90 con 0x55cea68f1cd0 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.843+0000 7f24342dfb80 10 client.18015 fetch_fsmap finished waiting for FSMap version 63 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.843+0000 7f24342dfb80 10 client.18015 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.843+0000 7f24342dfb80 20 client.18015 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:05.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.843+0000 7f24342dfb80 1 --2- 172.21.15.114:0/712925755 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cea6905c70 0x55cea69080d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.843+0000 7f24342dfb80 4 client.18015 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.843+0000 7f24342dfb80 1 -- 172.21.15.114:0/712925755 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cea68b7a70 con 0x55cea6905c70 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.843+0000 7f241effd700 1 --2- 172.21.15.114:0/712925755 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cea6905c70 0x55cea69080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.843+0000 7f23feffd700 10 client.18015 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.843+0000 7f241effd700 1 --2- 172.21.15.114:0/712925755 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cea6905c70 0x55cea69080d0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=524cfa762e6c2320 server_cookie=48df491dd2f68c5f in_seq=0 out_seq=0 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.844+0000 7f23feffd700 1 -- 172.21.15.114:0/712925755 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cea68b7a70 con 0x55cea6905c70 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.844+0000 7f23feffd700 10 client.18015 handle_command_reply: tid=0 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.844+0000 7f24342dfb80 1 client.18015 shutdown 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.844+0000 7f24342dfb80 1 -- 172.21.15.114:0/712925755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68f1cd0 msgr2=0x55cea68f5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.844+0000 7f24342dfb80 1 --2- 172.21.15.114:0/712925755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68f1cd0 0x55cea68f5ed0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f24180df850 tx=0x7f2418003ce0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.844+0000 7f24342dfb80 1 -- 172.21.15.114:0/712925755 shutdown_connections 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.844+0000 7f24342dfb80 1 --2- 172.21.15.114:0/712925755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68f1cd0 0x55cea68f5ed0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.844+0000 7f24342dfb80 1 --2- 172.21.15.114:0/712925755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea676dec0 0x55cea68f5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.844+0000 7f24342dfb80 1 --2- 172.21.15.114:0/712925755 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cea6905c70 0x55cea69080d0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.845+0000 7f24342dfb80 1 --2- 172.21.15.114:0/712925755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea68ec000 0x55cea68f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.845+0000 7f23feffd700 0 client.18015 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.845+0000 7f24342dfb80 1 -- 172.21.15.114:0/712925755 >> 172.21.15.114:0/712925755 conn(0x55cea67c8b70 msgr2=0x55cea68eaab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.845+0000 7f24342dfb80 1 -- 172.21.15.114:0/712925755 shutdown_connections 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.845+0000 7f24342dfb80 1 -- 172.21.15.114:0/712925755 wait complete. 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.845+0000 7f24342dfb80 20 client.18015 trim_cache size 0 max 16384 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.878+0000 7f24342dfb80 1 Processor -- start 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.878+0000 7f24342dfb80 1 -- start start 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.878+0000 7f24342dfb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea676dec0 0x55cea68f74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.878+0000 7f24342dfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 0x55cea69018b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.878+0000 7f24342dfb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea68f1cd0 0x55cea68f0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.878+0000 7f24342dfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f24201b2110 con 0x55cea68ec000 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.878+0000 7f24342dfb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f24201b0880 con 0x55cea676dec0 2024-05-27T07:51:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.878+0000 7f24342dfb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f24201ae6f0 con 0x55cea68f1cd0 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.879+0000 7f241f7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea68f1cd0 0x55cea68f0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.879+0000 7f241e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 0x55cea69018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.879+0000 7f241effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea676dec0 0x55cea68f74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.879+0000 7f241e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 0x55cea69018b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34078/0 (socket says 172.21.15.114:34078) 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.879+0000 7f241e7fc700 1 -- 172.21.15.114:0/492713951 learned_addr learned my addr 172.21.15.114:0/492713951 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.881+0000 7f241f7fe700 1 -- 172.21.15.114:0/492713951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea676dec0 msgr2=0x55cea68f74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.881+0000 7f241f7fe700 1 --2- 172.21.15.114:0/492713951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea676dec0 0x55cea68f74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.881+0000 7f241f7fe700 1 -- 172.21.15.114:0/492713951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 msgr2=0x55cea69018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.881+0000 7f241f7fe700 1 --2- 172.21.15.114:0/492713951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 0x55cea69018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.881+0000 7f241f7fe700 1 -- 172.21.15.114:0/492713951 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cea68e7f50 con 0x55cea68f1cd0 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.881+0000 7f241effd700 1 --2- 172.21.15.114:0/492713951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea676dec0 0x55cea68f74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.881+0000 7f241f7fe700 1 --2- 172.21.15.114:0/492713951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea68f1cd0 0x55cea68f0ae0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f24180a8f10 tx=0x7f2418004f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc359a37156c6cea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.886+0000 7f24057fa700 1 -- 172.21.15.114:0/492713951 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24180b2240 con 0x55cea68f1cd0 2024-05-27T07:51:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.886+0000 7f24057fa700 1 -- 172.21.15.114:0/492713951 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24180ad820 con 0x55cea68f1cd0 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.886+0000 7f24057fa700 1 -- 172.21.15.114:0/492713951 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24180b1db0 con 0x55cea68f1cd0 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.887+0000 7f24342dfb80 1 -- 172.21.15.114:0/492713951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea68f1cd0 msgr2=0x55cea68f0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.887+0000 7f24342dfb80 1 --2- 172.21.15.114:0/492713951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea68f1cd0 0x55cea68f0ae0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f24180a8f10 tx=0x7f2418004f70 comp rx=0 tx=0).stop 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.888+0000 7f24342dfb80 1 -- 172.21.15.114:0/492713951 shutdown_connections 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.889+0000 7f24342dfb80 1 --2- 172.21.15.114:0/492713951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea68f1cd0 0x55cea68f0ae0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.889+0000 7f24342dfb80 1 --2- 172.21.15.114:0/492713951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea676dec0 0x55cea68f74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.889+0000 7f24342dfb80 1 --2- 172.21.15.114:0/492713951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 0x55cea69018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.889+0000 7f24342dfb80 1 -- 172.21.15.114:0/492713951 >> 172.21.15.114:0/492713951 conn(0x55cea67c8b70 msgr2=0x55cea68f03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.892+0000 7f24342dfb80 1 -- 172.21.15.114:0/492713951 shutdown_connections 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.892+0000 7f24342dfb80 1 -- 172.21.15.114:0/492713951 wait complete. 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.892+0000 7f24342dfb80 1 -- start start 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.893+0000 7f24342dfb80 1 Processor -- start 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.894+0000 7f24342dfb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea676dec0 0x55cea68fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.894+0000 7f24342dfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 0x55cea68faa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.894+0000 7f241effd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea676dec0 0x55cea68fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.894+0000 7f241effd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea676dec0 0x55cea68fe0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:48900/0 (socket says 172.21.15.114:48900) 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.894+0000 7f241effd700 1 -- 172.21.15.114:0/2494755122 learned_addr learned my addr 172.21.15.114:0/2494755122 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.894+0000 7f24342dfb80 1 --2- 172.21.15.114:0/2494755122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea68f1cd0 0x55cea68fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.894+0000 7f24342dfb80 1 -- 172.21.15.114:0/2494755122 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f24201aed40 con 0x55cea68ec000 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.894+0000 7f24342dfb80 1 -- 172.21.15.114:0/2494755122 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f24201b01b0 con 0x55cea68f1cd0 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.894+0000 7f24342dfb80 1 -- 172.21.15.114:0/2494755122 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f24201af410 con 0x55cea676dec0 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.894+0000 7f241f7fe700 1 --2- 172.21.15.114:0/2494755122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea68f1cd0 0x55cea68fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.895+0000 7f24342dfb80 10 client.0 _ll_register_callbacks cb 0x55cea67a59a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.895+0000 7f241effd700 1 -- 172.21.15.114:0/2494755122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea68f1cd0 msgr2=0x55cea68fafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.895+0000 7f241effd700 1 --2- 172.21.15.114:0/2494755122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea68f1cd0 0x55cea68fafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.895+0000 7f241effd700 1 -- 172.21.15.114:0/2494755122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 msgr2=0x55cea68faa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.895+0000 7f241effd700 1 --2- 172.21.15.114:0/2494755122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 0x55cea68faa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.895+0000 7f241effd700 1 -- 172.21.15.114:0/2494755122 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cea68fb5a0 con 0x55cea676dec0 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.895+0000 7f241effd700 1 --2- 172.21.15.114:0/2494755122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea676dec0 0x55cea68fe0d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f24100098e0 tx=0x7f2410009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a431cce8f25b46f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.895+0000 7f24342dfb80 1 -- 172.21.15.114:0/2494755122 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24180ad820 con 0x55cea676dec0 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.896+0000 7f23feffd700 10 client.27586 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.896+0000 7f23feffd700 1 -- 172.21.15.114:0/2494755122 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241000f460 con 0x55cea676dec0 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.900+0000 7f23feffd700 1 -- 172.21.15.114:0/2494755122 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9642+0+0 (secure 0 0 0) 0x7f241000f600 con 0x55cea676dec0 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.900+0000 7f23feffd700 1 -- 172.21.15.114:0/2494755122 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f241000f820 con 0x55cea676dec0 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.900+0000 7f24342dfb80 10 client.27586 fetch_fsmap learned FSMap version 63 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.900+0000 7f24342dfb80 1 -- 172.21.15.114:0/2494755122 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({fsmap=63}) -- 0x55cea6982550 con 0x55cea676dec0 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.905+0000 7f23feffd700 1 -- 172.21.15.114:0/2494755122 <== mon.1 v2:172.21.15.186:3300/0 4 ==== fsmap(e 63) ==== 5910+0+0 (secure 0 0 0) 0x7f2410022070 con 0x55cea676dec0 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.905+0000 7f24342dfb80 10 client.27586 fetch_fsmap finished waiting for FSMap version 63 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.905+0000 7f24342dfb80 10 client.27586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.905+0000 7f24342dfb80 20 client.27586 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.905+0000 7f24342dfb80 1 --2- 172.21.15.114:0/2494755122 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cea6982b50 0x55cea6984fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.905+0000 7f24342dfb80 4 client.27586 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:51:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.905+0000 7f24342dfb80 1 -- 172.21.15.114:0/2494755122 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f24180ad820 con 0x55cea6982b50 2024-05-27T07:51:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.907+0000 7f241e7fc700 1 --2- 172.21.15.114:0/2494755122 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cea6982b50 0x55cea6984fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.914+0000 7f23feffd700 10 client.27586 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.914+0000 7f241e7fc700 1 --2- 172.21.15.114:0/2494755122 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cea6982b50 0x55cea6984fd0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45faa6f2fde3e99f server_cookie=56faae2aeb97b93f in_seq=0 out_seq=0 2024-05-27T07:51:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.915+0000 7f23feffd700 1 -- 172.21.15.114:0/2494755122 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f24180ad820 con 0x55cea6982b50 2024-05-27T07:51:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.915+0000 7f23feffd700 10 client.27586 handle_command_reply: tid=0 2024-05-27T07:51:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.915+0000 7f24342dfb80 1 client.27586 shutdown 2024-05-27T07:51:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.915+0000 7f24342dfb80 1 -- 172.21.15.114:0/2494755122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea676dec0 msgr2=0x55cea68fe0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.915+0000 7f24342dfb80 1 --2- 172.21.15.114:0/2494755122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea676dec0 0x55cea68fe0d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f24100098e0 tx=0x7f2410009ca0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.917+0000 7f24342dfb80 1 -- 172.21.15.114:0/2494755122 shutdown_connections 2024-05-27T07:51:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.917+0000 7f24342dfb80 1 --2- 172.21.15.114:0/2494755122 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cea6982b50 0x55cea6984fd0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.918+0000 7f24342dfb80 1 --2- 172.21.15.114:0/2494755122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cea68f1cd0 0x55cea68fafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.918+0000 7f24342dfb80 1 --2- 172.21.15.114:0/2494755122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cea676dec0 0x55cea68fe0d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.918+0000 7f24342dfb80 1 --2- 172.21.15.114:0/2494755122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cea68ec000 0x55cea68faa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.918+0000 7f23feffd700 0 client.27586 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.918+0000 7f24342dfb80 1 -- 172.21.15.114:0/2494755122 >> 172.21.15.114:0/2494755122 conn(0x55cea67c8b70 msgr2=0x55cea68f7290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.920+0000 7f24342dfb80 1 -- 172.21.15.114:0/2494755122 shutdown_connections 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.921+0000 7f24342dfb80 1 -- 172.21.15.114:0/2494755122 wait complete. 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.923+0000 7f24342dfb80 20 client.27586 trim_cache size 0 max 16384 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.928+0000 7f24258ca700 1 -- 172.21.15.114:0/1126363651 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f24080b29f0 msgr2=0x7f24080b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.928+0000 7f24258ca700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f24080b29f0 0x7f24080b4f00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f24100059d0 tx=0x7f2410005960 comp rx=0 tx=0).stop 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.928+0000 7f24258ca700 1 -- 172.21.15.114:0/1126363651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24201092f0 msgr2=0x7f242019f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.928+0000 7f24258ca700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24201092f0 0x7f242019f740 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f241800ef30 tx=0x7f241800c3a0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.928+0000 7f24258ca700 1 -- 172.21.15.114:0/1126363651 shutdown_connections 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.928+0000 7f24258ca700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24201092f0 0x7f242019f740 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.928+0000 7f24258ca700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2420102870 0x7f242019af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.928+0000 7f24258ca700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f24080b29f0 0x7f24080b4f00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.928+0000 7f24258ca700 1 --2- 172.21.15.114:0/1126363651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24201031c0 0x7f242019b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:05.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.928+0000 7f24258ca700 1 -- 172.21.15.114:0/1126363651 >> 172.21.15.114:0/1126363651 conn(0x7f24200fe6e0 msgr2=0x7f242010a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:05.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.928+0000 7f24258ca700 1 -- 172.21.15.114:0/1126363651 shutdown_connections 2024-05-27T07:51:05.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:04.928+0000 7f24258ca700 1 -- 172.21.15.114:0/1126363651 wait complete. 2024-05-27T07:51:05.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:05 smithi163 ceph-mon[36451]: pgmap v375: 129 pgs: 129 active+clean; 73 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 339 op/s 2024-05-27T07:51:05.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:05 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:51:05.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:05 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:05.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:05 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:51:05.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:05 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:05.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:05 smithi186 ceph-mon[36647]: pgmap v375: 129 pgs: 129 active+clean; 73 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 339 op/s 2024-05-27T07:51:05.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:05 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:51:05.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:05 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:05.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:05 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:51:05.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:05 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:05.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:05 smithi114 ceph-mon[32038]: pgmap v375: 129 pgs: 129 active+clean; 73 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 339 op/s 2024-05-27T07:51:05.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:05 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:51:05.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:05 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:05.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:05 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:51:05.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:05 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:06.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1113678 9.97 MB/sec execute 438 sec latency 108.706 ms 2024-05-27T07:51:06.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:51:06.848 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:07.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1114185 9.95 MB/sec execute 439 sec latency 275.605 ms 2024-05-27T07:51:07.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:07 smithi163 ceph-mon[36451]: pgmap v376: 129 pgs: 19 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 60 active+clean; 73 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 209 op/s 2024-05-27T07:51:07.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:07 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:51:07.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:07 smithi186 ceph-mon[36647]: pgmap v376: 129 pgs: 19 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 60 active+clean; 73 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 209 op/s 2024-05-27T07:51:07.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:07 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:51:07.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:07 smithi114 ceph-mon[32038]: pgmap v376: 129 pgs: 19 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 60 active+clean; 73 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 209 op/s 2024-05-27T07:51:07.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:07 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:51:08.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1114711 9.94 MB/sec execute 440 sec latency 113.904 ms 2024-05-27T07:51:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.350+0000 7f3349c8b640 1 -- 172.21.15.114:0/2660814022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f334411d240 msgr2=0x7f334411f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.350+0000 7f3349c8b640 1 --2- 172.21.15.114:0/2660814022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f334411d240 0x7f334411f660 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f333c009ca0 tx=0x7f333c02a940 comp rx=0 tx=0).stop 2024-05-27T07:51:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.350+0000 7f3349c8b640 1 -- 172.21.15.114:0/2660814022 shutdown_connections 2024-05-27T07:51:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.350+0000 7f3349c8b640 1 --2- 172.21.15.114:0/2660814022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f334411d240 0x7f334411f660 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.350+0000 7f3349c8b640 1 --2- 172.21.15.114:0/2660814022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f334411a8e0 0x7f334411cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.350+0000 7f3349c8b640 1 --2- 172.21.15.114:0/2660814022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f334410a1f0 0x7f334410a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.350+0000 7f3349c8b640 1 -- 172.21.15.114:0/2660814022 >> 172.21.15.114:0/2660814022 conn(0x7f334406e8f0 msgr2=0x7f334406ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.353+0000 7f3349c8b640 1 -- 172.21.15.114:0/2660814022 shutdown_connections 2024-05-27T07:51:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.354+0000 7f3349c8b640 1 -- 172.21.15.114:0/2660814022 wait complete. 2024-05-27T07:51:08.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.355+0000 7f3349c8b640 1 Processor -- start 2024-05-27T07:51:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.359+0000 7f3349c8b640 1 -- start start 2024-05-27T07:51:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.359+0000 7f3349c8b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f334410a1f0 0x7f3344113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.359+0000 7f3349c8b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f334411a8e0 0x7f3344114310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.359+0000 7f3349c8b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3344118530 0x7f3344118940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.359+0000 7f3349c8b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3344112580 con 0x7f334411a8e0 2024-05-27T07:51:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.359+0000 7f3349c8b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3344112400 con 0x7f3344118530 2024-05-27T07:51:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.359+0000 7f3349c8b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3344112700 con 0x7f334410a1f0 2024-05-27T07:51:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.360+0000 7f3343fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3344118530 0x7f3344118940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.360+0000 7f3343fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3344118530 0x7f3344118940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51676/0 (socket says 172.21.15.114:51676) 2024-05-27T07:51:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.360+0000 7f3343fff640 1 -- 172.21.15.114:0/2325109948 learned_addr learned my addr 172.21.15.114:0/2325109948 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.360+0000 7f33437fe640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f334410a1f0 0x7f3344113dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.360+0000 7f3342ffd640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f334411a8e0 0x7f3344114310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.360+0000 7f3343fff640 1 -- 172.21.15.114:0/2325109948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f334410a1f0 msgr2=0x7f3344113dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.360+0000 7f3343fff640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f334410a1f0 0x7f3344113dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.360+0000 7f3343fff640 1 -- 172.21.15.114:0/2325109948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f334411a8e0 msgr2=0x7f3344114310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.360+0000 7f3343fff640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f334411a8e0 0x7f3344114310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.360+0000 7f3343fff640 1 -- 172.21.15.114:0/2325109948 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3344119020 con 0x7f3344118530 2024-05-27T07:51:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.360+0000 7f3342ffd640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f334411a8e0 0x7f3344114310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:51:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.360+0000 7f3343fff640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3344118530 0x7f3344118940 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f333c02af30 tx=0x7f333c003e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.361+0000 7f3340ff9640 1 -- 172.21.15.114:0/2325109948 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f333c034070 con 0x7f3344118530 2024-05-27T07:51:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.361+0000 7f3340ff9640 1 -- 172.21.15.114:0/2325109948 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f333c03d070 con 0x7f3344118530 2024-05-27T07:51:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.361+0000 7f3340ff9640 1 -- 172.21.15.114:0/2325109948 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f333c039420 con 0x7f3344118530 2024-05-27T07:51:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.361+0000 7f3349c8b640 1 -- 172.21.15.114:0/2325109948 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33441192b0 con 0x7f3344118530 2024-05-27T07:51:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.361+0000 7f3349c8b640 1 -- 172.21.15.114:0/2325109948 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33441be750 con 0x7f3344118530 2024-05-27T07:51:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.366+0000 7f3350c13740 1 Processor -- start 2024-05-27T07:51:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.368+0000 7f3340ff9640 1 -- 172.21.15.114:0/2325109948 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f333c037020 con 0x7f3344118530 2024-05-27T07:51:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.369+0000 7f3350c13740 1 -- start start 2024-05-27T07:51:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.370+0000 7f3340ff9640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f332c0b2aa0 0x7f332c0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.370+0000 7f3340ff9640 1 -- 172.21.15.114:0/2325109948 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9642+0+0 (secure 0 0 0) 0x7f333c103540 con 0x7f3344118530 2024-05-27T07:51:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.370+0000 7f3342ffd640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f332c0b2aa0 0x7f332c0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.370+0000 7f3350c13740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb978a01b0 0x55eb97c413d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.370+0000 7f3350c13740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb978772f0 0x55eb97c41910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.370+0000 7f3350c13740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9787f4d0 0x55eb97c41e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.370+0000 7f3350c13740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55eb97bb3c30 con 0x55eb9787f4d0 2024-05-27T07:51:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.370+0000 7f3350c13740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55eb97ba37d0 con 0x55eb978a01b0 2024-05-27T07:51:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.370+0000 7f3350c13740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55eb97ba7810 con 0x55eb978772f0 2024-05-27T07:51:08.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.371+0000 7f3342ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb978772f0 0x55eb97c41910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.371+0000 7f3342ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb978772f0 0x55eb97c41910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:48928/0 (socket says 172.21.15.114:48928) 2024-05-27T07:51:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.371+0000 7f3342ffd640 1 -- 172.21.15.114:0/2824753826 learned_addr learned my addr 172.21.15.114:0/2824753826 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.372+0000 7f3342ffd640 1 -- 172.21.15.114:0/2824753826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb978a01b0 msgr2=0x55eb97c413d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:51:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.372+0000 7f3342ffd640 1 --2- 172.21.15.114:0/2824753826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb978a01b0 0x55eb97c413d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.372+0000 7f3342ffd640 1 -- 172.21.15.114:0/2824753826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9787f4d0 msgr2=0x55eb97c41e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:51:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.372+0000 7f3342ffd640 1 --2- 172.21.15.114:0/2824753826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9787f4d0 0x55eb97c41e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.372+0000 7f3342ffd640 1 -- 172.21.15.114:0/2824753826 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb978b8210 con 0x55eb978772f0 2024-05-27T07:51:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.372+0000 7f3342ffd640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f332c0b2aa0 0x7f332c0b4fb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3338015930 tx=0x7f3338009ad0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.372+0000 7f3342ffd640 1 --2- 172.21.15.114:0/2824753826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb978772f0 0x55eb97c41910 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3338015df0 tx=0x7f333801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a4ab8cfb62ced48 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.375+0000 7f3329ffb640 1 -- 172.21.15.114:0/2824753826 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3338004020 con 0x55eb978772f0 2024-05-27T07:51:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.375+0000 7f3329ffb640 1 -- 172.21.15.114:0/2824753826 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33380054d0 con 0x55eb978772f0 2024-05-27T07:51:08.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.375+0000 7f3329ffb640 1 -- 172.21.15.114:0/2824753826 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3338004a90 con 0x55eb978772f0 2024-05-27T07:51:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.379+0000 7f3350c13740 1 -- 172.21.15.114:0/2824753826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb978772f0 msgr2=0x55eb97c41910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.379+0000 7f3350c13740 1 --2- 172.21.15.114:0/2824753826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb978772f0 0x55eb97c41910 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3338015df0 tx=0x7f333801c000 comp rx=0 tx=0).stop 2024-05-27T07:51:08.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.380+0000 7f3350c13740 1 -- 172.21.15.114:0/2824753826 shutdown_connections 2024-05-27T07:51:08.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.380+0000 7f3350c13740 1 --2- 172.21.15.114:0/2824753826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9787f4d0 0x55eb97c41e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.380+0000 7f3350c13740 1 --2- 172.21.15.114:0/2824753826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb978772f0 0x55eb97c41910 secure :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3338015df0 tx=0x7f333801c000 comp rx=0 tx=0).stop 2024-05-27T07:51:08.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.380+0000 7f3350c13740 1 --2- 172.21.15.114:0/2824753826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb978a01b0 0x55eb97c413d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.380+0000 7f3350c13740 1 -- 172.21.15.114:0/2824753826 >> 172.21.15.114:0/2824753826 conn(0x55eb97894300 msgr2=0x55eb97c3db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.387+0000 7f3350c13740 1 -- 172.21.15.114:0/2824753826 shutdown_connections 2024-05-27T07:51:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.389+0000 7f3350c13740 1 -- 172.21.15.114:0/2824753826 wait complete. 2024-05-27T07:51:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.390+0000 7f3350c13740 1 -- start start 2024-05-27T07:51:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.390+0000 7f3350c13740 1 Processor -- start 2024-05-27T07:51:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.391+0000 7f3350c13740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb9787f4d0 0x55eb97c4be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.391+0000 7f3350c13740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb978a01b0 0x55eb97c4c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.391+0000 7f3350c13740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9785ed80 0x55eb97c4d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.391+0000 7f3350c13740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55eb97889790 con 0x55eb978a01b0 2024-05-27T07:51:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.391+0000 7f3350c13740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55eb97b5f890 con 0x55eb9787f4d0 2024-05-27T07:51:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.391+0000 7f3350c13740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55eb97c4e440 con 0x55eb9785ed80 2024-05-27T07:51:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.391+0000 7f3350c13740 10 client.0 _ll_register_callbacks cb 0x55eb978b7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.391+0000 7f3343fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9785ed80 0x55eb97c4d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.391+0000 7f3343fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9785ed80 0x55eb97c4d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:48944/0 (socket says 172.21.15.114:48944) 2024-05-27T07:51:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.391+0000 7f3343fff640 1 -- 172.21.15.114:0/3183221442 learned_addr learned my addr 172.21.15.114:0/3183221442 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.391+0000 7f33437fe640 1 --2- 172.21.15.114:0/3183221442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb9787f4d0 0x55eb97c4be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.392+0000 7f3343fff640 1 -- 172.21.15.114:0/3183221442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb9787f4d0 msgr2=0x55eb97c4be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.392+0000 7f3343fff640 1 --2- 172.21.15.114:0/3183221442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb9787f4d0 0x55eb97c4be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.392+0000 7f3343fff640 1 -- 172.21.15.114:0/3183221442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb978a01b0 msgr2=0x55eb97c4c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.392+0000 7f3343fff640 1 --2- 172.21.15.114:0/3183221442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb978a01b0 0x55eb97c4c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.392+0000 7f3343fff640 1 -- 172.21.15.114:0/3183221442 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb97984620 con 0x55eb9785ed80 2024-05-27T07:51:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.392+0000 7f3343fff640 1 --2- 172.21.15.114:0/3183221442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9785ed80 0x55eb97c4d210 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f333c0c7770 tx=0x7f333c0c9ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ac43d4dc8191011 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.392+0000 7f33437fe640 1 --2- 172.21.15.114:0/3183221442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb9787f4d0 0x55eb97c4be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:51:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.393+0000 7f3322ffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:51:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.393+0000 7f3322ffd640 1 -- 172.21.15.114:0/3183221442 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f333c100c60 con 0x55eb9785ed80 2024-05-27T07:51:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.393+0000 7f3322ffd640 1 -- 172.21.15.114:0/3183221442 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9642+0+0 (secure 0 0 0) 0x7f333c02fa00 con 0x55eb9785ed80 2024-05-27T07:51:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.393+0000 7f3350c13740 1 -- 172.21.15.114:0/3183221442 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33380054d0 con 0x55eb9785ed80 2024-05-27T07:51:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.394+0000 7f3322ffd640 1 -- 172.21.15.114:0/3183221442 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f333c0b20b0 con 0x55eb9785ed80 2024-05-27T07:51:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.394+0000 7f3350c13740 10 client.27595 fetch_fsmap learned FSMap version 63 2024-05-27T07:51:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.394+0000 7f3350c13740 1 -- 172.21.15.114:0/3183221442 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x55eb97c5a9e0 con 0x55eb9785ed80 2024-05-27T07:51:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.397+0000 7f3322ffd640 1 -- 172.21.15.114:0/3183221442 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33380054d0 con 0x55eb9785ed80 2024-05-27T07:51:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.397+0000 7f3322ffd640 1 -- 172.21.15.114:0/3183221442 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 63) ==== 5910+0+0 (secure 0 0 0) 0x7f333c02f070 con 0x55eb9785ed80 2024-05-27T07:51:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.399+0000 7f3350c13740 10 client.27595 fetch_fsmap finished waiting for FSMap version 63 2024-05-27T07:51:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.399+0000 7f3350c13740 10 client.27595 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.399+0000 7f3350c13740 20 client.27595 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.399+0000 7f3350c13740 1 --2- 172.21.15.114:0/3183221442 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55eb978341c0 0x55eb97c5df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.399+0000 7f3350c13740 4 client.27595 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.399+0000 7f3350c13740 1 -- 172.21.15.114:0/3183221442 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb97bb5b80 con 0x55eb978341c0 2024-05-27T07:51:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.400+0000 7f33437fe640 1 --2- 172.21.15.114:0/3183221442 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55eb978341c0 0x55eb97c5df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.405+0000 7f33437fe640 1 --2- 172.21.15.114:0/3183221442 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55eb978341c0 0x55eb97c5df00 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4758476e4c133f93 server_cookie=8b67b29bdc61cfa2 in_seq=0 out_seq=0 2024-05-27T07:51:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.405+0000 7f3322ffd640 10 client.27595 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.407+0000 7f3322ffd640 1 -- 172.21.15.114:0/3183221442 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55eb97bb5b80 con 0x55eb978341c0 2024-05-27T07:51:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.407+0000 7f3322ffd640 10 client.27595 handle_command_reply: tid=0 2024-05-27T07:51:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.407+0000 7f3350c13740 1 client.27595 shutdown 2024-05-27T07:51:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.408+0000 7f3350c13740 1 -- 172.21.15.114:0/3183221442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9785ed80 msgr2=0x55eb97c4d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.408+0000 7f3350c13740 1 --2- 172.21.15.114:0/3183221442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9785ed80 0x55eb97c4d210 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f333c0c7770 tx=0x7f333c0c9ed0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.414+0000 7f3350c13740 1 -- 172.21.15.114:0/3183221442 shutdown_connections 2024-05-27T07:51:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.414+0000 7f3350c13740 1 --2- 172.21.15.114:0/3183221442 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55eb978341c0 0x55eb97c5df00 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.414+0000 7f3350c13740 1 --2- 172.21.15.114:0/3183221442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9785ed80 0x55eb97c4d210 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.414+0000 7f3350c13740 1 --2- 172.21.15.114:0/3183221442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb978a01b0 0x55eb97c4c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.414+0000 7f3350c13740 1 --2- 172.21.15.114:0/3183221442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb9787f4d0 0x55eb97c4be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.414+0000 7f3350c13740 1 -- 172.21.15.114:0/3183221442 >> 172.21.15.114:0/3183221442 conn(0x55eb97894300 msgr2=0x55eb97c430f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.415+0000 7f3322ffd640 0 client.27595 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.415+0000 7f3350c13740 1 -- 172.21.15.114:0/3183221442 shutdown_connections 2024-05-27T07:51:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.419+0000 7f3350c13740 1 -- 172.21.15.114:0/3183221442 wait complete. 2024-05-27T07:51:08.424 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.423+0000 7f3350c13740 20 client.27595 trim_cache size 0 max 16384 2024-05-27T07:51:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.455+0000 7f3350c13740 1 Processor -- start 2024-05-27T07:51:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.455+0000 7f3350c13740 1 -- start start 2024-05-27T07:51:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.455+0000 7f3350c13740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb9785ed80 0x55eb97c43610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.456+0000 7f3350c13740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9787f4d0 0x55eb97c4b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.456+0000 7f33437fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb9785ed80 0x55eb97c43610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.456+0000 7f33437fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb9785ed80 0x55eb97c43610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51700/0 (socket says 172.21.15.114:51700) 2024-05-27T07:51:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.456+0000 7f3342ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9787f4d0 0x55eb97c4b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.456+0000 7f3342ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9787f4d0 0x55eb97c4b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34146/0 (socket says 172.21.15.114:34146) 2024-05-27T07:51:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.456+0000 7f33437fe640 1 -- 172.21.15.114:0/2219923229 learned_addr learned my addr 172.21.15.114:0/2219923229 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.456+0000 7f3350c13740 1 --2- 172.21.15.114:0/2219923229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb978a01b0 0x55eb97c45080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.456+0000 7f3350c13740 1 -- 172.21.15.114:0/2219923229 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3330011ba0 con 0x55eb9787f4d0 2024-05-27T07:51:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.456+0000 7f3350c13740 1 -- 172.21.15.114:0/2219923229 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3330010330 con 0x55eb9785ed80 2024-05-27T07:51:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.456+0000 7f3343fff640 1 --2- 172.21.15.114:0/2219923229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb978a01b0 0x55eb97c45080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.456+0000 7f3350c13740 1 -- 172.21.15.114:0/2219923229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f333000e1a0 con 0x55eb978a01b0 2024-05-27T07:51:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.457+0000 7f33437fe640 1 -- 172.21.15.114:0/2219923229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb978a01b0 msgr2=0x55eb97c45080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.457+0000 7f33437fe640 1 --2- 172.21.15.114:0/2219923229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb978a01b0 0x55eb97c45080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.457+0000 7f33437fe640 1 -- 172.21.15.114:0/2219923229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9787f4d0 msgr2=0x55eb97c4b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.457+0000 7f33437fe640 1 --2- 172.21.15.114:0/2219923229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9787f4d0 0x55eb97c4b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.457+0000 7f33437fe640 1 -- 172.21.15.114:0/2219923229 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb97c49fd0 con 0x55eb9785ed80 2024-05-27T07:51:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.458+0000 7f33437fe640 1 --2- 172.21.15.114:0/2219923229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb9785ed80 0x55eb97c43610 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3344115380 tx=0x7f3334005e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e60c512ecd1e010c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.458+0000 7f33297fa640 1 -- 172.21.15.114:0/2219923229 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f333401d030 con 0x55eb9785ed80 2024-05-27T07:51:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.458+0000 7f33297fa640 1 -- 172.21.15.114:0/2219923229 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3334011db0 con 0x55eb9785ed80 2024-05-27T07:51:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.458+0000 7f33297fa640 1 -- 172.21.15.114:0/2219923229 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33340192d0 con 0x55eb9785ed80 2024-05-27T07:51:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.459+0000 7f3350c13740 1 -- 172.21.15.114:0/2219923229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb9785ed80 msgr2=0x55eb97c43610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.459+0000 7f3350c13740 1 --2- 172.21.15.114:0/2219923229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb9785ed80 0x55eb97c43610 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3344115380 tx=0x7f3334005e90 comp rx=0 tx=0).stop 2024-05-27T07:51:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.460+0000 7f3350c13740 1 -- 172.21.15.114:0/2219923229 shutdown_connections 2024-05-27T07:51:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.460+0000 7f3350c13740 1 --2- 172.21.15.114:0/2219923229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb978a01b0 0x55eb97c45080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.460+0000 7f3350c13740 1 --2- 172.21.15.114:0/2219923229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9787f4d0 0x55eb97c4b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.460+0000 7f3350c13740 1 --2- 172.21.15.114:0/2219923229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb9785ed80 0x55eb97c43610 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.460+0000 7f3350c13740 1 -- 172.21.15.114:0/2219923229 >> 172.21.15.114:0/2219923229 conn(0x55eb97894300 msgr2=0x55eb97c42f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.460+0000 7f3350c13740 1 -- 172.21.15.114:0/2219923229 shutdown_connections 2024-05-27T07:51:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.460+0000 7f3350c13740 1 -- 172.21.15.114:0/2219923229 wait complete. 2024-05-27T07:51:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.461+0000 7f3350c13740 1 -- start start 2024-05-27T07:51:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.461+0000 7f3350c13740 1 Processor -- start 2024-05-27T07:51:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.462+0000 7f3350c13740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9785ed80 0x55eb97c57c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.462+0000 7f3350c13740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9787f4d0 0x55eb97c581c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.462+0000 7f3350c13740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb978a01b0 0x55eb97c53050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.462+0000 7f3342ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9787f4d0 0x55eb97c581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.462+0000 7f3342ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9787f4d0 0x55eb97c581c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:48966/0 (socket says 172.21.15.114:48966) 2024-05-27T07:51:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.462+0000 7f3342ffd640 1 -- 172.21.15.114:0/515049410 learned_addr learned my addr 172.21.15.114:0/515049410 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.462+0000 7f3343fff640 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb978a01b0 0x55eb97c53050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.462+0000 7f3350c13740 1 -- 172.21.15.114:0/515049410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f333000e7f0 con 0x55eb9785ed80 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.463+0000 7f3350c13740 1 -- 172.21.15.114:0/515049410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f333000fc60 con 0x55eb978a01b0 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.463+0000 7f3350c13740 1 -- 172.21.15.114:0/515049410 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f333000eec0 con 0x55eb9787f4d0 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.463+0000 7f3350c13740 10 client.0 _ll_register_callbacks cb 0x7f333c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.463+0000 7f33437fe640 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9785ed80 0x55eb97c57c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.464+0000 7f33437fe640 1 -- 172.21.15.114:0/515049410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9787f4d0 msgr2=0x55eb97c581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.464+0000 7f33437fe640 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9787f4d0 0x55eb97c581c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.464+0000 7f33437fe640 1 -- 172.21.15.114:0/515049410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb978a01b0 msgr2=0x55eb97c53050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.464+0000 7f33437fe640 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb978a01b0 0x55eb97c53050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.464+0000 7f33437fe640 1 -- 172.21.15.114:0/515049410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb97c537e0 con 0x55eb9785ed80 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.464+0000 7f33437fe640 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9785ed80 0x55eb97c57c80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f3334009a20 tx=0x7f3334004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c35025a6790666 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.465+0000 7f3342ffd640 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9787f4d0 0x55eb97c581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.465+0000 7f3343fff640 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb978a01b0 0x55eb97c53050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.465+0000 7f3322ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.465+0000 7f3322ffd640 1 -- 172.21.15.114:0/515049410 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3334005190 con 0x55eb9785ed80 2024-05-27T07:51:08.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.465+0000 7f3322ffd640 1 -- 172.21.15.114:0/515049410 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9642+0+0 (secure 0 0 0) 0x7f3334005330 con 0x55eb9785ed80 2024-05-27T07:51:08.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.466+0000 7f3350c13740 1 -- 172.21.15.114:0/515049410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3334011db0 con 0x55eb9785ed80 2024-05-27T07:51:08.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.466+0000 7f3322ffd640 1 -- 172.21.15.114:0/515049410 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f3334005530 con 0x55eb9785ed80 2024-05-27T07:51:08.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.467+0000 7f3350c13740 10 client.18027 fetch_fsmap learned FSMap version 63 2024-05-27T07:51:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.467+0000 7f3350c13740 1 -- 172.21.15.114:0/515049410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x55eb97cc6020 con 0x55eb9785ed80 2024-05-27T07:51:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.470+0000 7f3322ffd640 1 -- 172.21.15.114:0/515049410 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3334011db0 con 0x55eb9785ed80 2024-05-27T07:51:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.470+0000 7f3322ffd640 1 -- 172.21.15.114:0/515049410 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 63) ==== 5910+0+0 (secure 0 0 0) 0x7f333402b520 con 0x55eb9785ed80 2024-05-27T07:51:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.470+0000 7f3350c13740 10 client.18027 fetch_fsmap finished waiting for FSMap version 63 2024-05-27T07:51:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.470+0000 7f3350c13740 10 client.18027 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.470+0000 7f3350c13740 20 client.18027 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.470+0000 7f3350c13740 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55eb97cc6670 0x55eb97cc8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.470+0000 7f3350c13740 4 client.18027 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:51:08.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.470+0000 7f3350c13740 1 -- 172.21.15.114:0/515049410 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eb97baf360 con 0x55eb97cc6670 2024-05-27T07:51:08.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.470+0000 7f3342ffd640 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55eb97cc6670 0x55eb97cc8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:08.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.471+0000 7f3342ffd640 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55eb97cc6670 0x55eb97cc8ac0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=305e5e4645864a8 server_cookie=4043e3aad121574d in_seq=0 out_seq=0 2024-05-27T07:51:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.474+0000 7f3322ffd640 10 client.18027 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.827+0000 7f3322ffd640 1 -- 172.21.15.114:0/515049410 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55eb97baf360 con 0x55eb97cc6670 2024-05-27T07:51:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.827+0000 7f3322ffd640 10 client.18027 handle_command_reply: tid=0 2024-05-27T07:51:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.828+0000 7f3350c13740 1 client.18027 shutdown 2024-05-27T07:51:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.828+0000 7f3350c13740 1 -- 172.21.15.114:0/515049410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9785ed80 msgr2=0x55eb97c57c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.828+0000 7f3350c13740 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9785ed80 0x55eb97c57c80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f3334009a20 tx=0x7f3334004f20 comp rx=0 tx=0).stop 2024-05-27T07:51:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.828+0000 7f3350c13740 1 -- 172.21.15.114:0/515049410 shutdown_connections 2024-05-27T07:51:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.828+0000 7f3350c13740 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55eb97cc6670 0x55eb97cc8ac0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.828+0000 7f3350c13740 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb978a01b0 0x55eb97c53050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.828+0000 7f3350c13740 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eb9787f4d0 0x55eb97c581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.828+0000 7f3350c13740 1 --2- 172.21.15.114:0/515049410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb9785ed80 0x55eb97c57c80 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.828+0000 7f3350c13740 1 -- 172.21.15.114:0/515049410 >> 172.21.15.114:0/515049410 conn(0x55eb97894300 msgr2=0x55eb97c43690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.829+0000 7f3322ffd640 0 client.18027 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.829+0000 7f3350c13740 1 -- 172.21.15.114:0/515049410 shutdown_connections 2024-05-27T07:51:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.829+0000 7f3350c13740 1 -- 172.21.15.114:0/515049410 wait complete. 2024-05-27T07:51:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.829+0000 7f3350c13740 20 client.18027 trim_cache size 0 max 16384 2024-05-27T07:51:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:51:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:51:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:51:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.830+0000 7f3349c8b640 1 -- 172.21.15.114:0/2325109948 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f332c0b2aa0 msgr2=0x7f332c0b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.830+0000 7f3349c8b640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f332c0b2aa0 0x7f332c0b4fb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3338015930 tx=0x7f3338009ad0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.830+0000 7f3349c8b640 1 -- 172.21.15.114:0/2325109948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3344118530 msgr2=0x7f3344118940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.830+0000 7f3349c8b640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3344118530 0x7f3344118940 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f333c02af30 tx=0x7f333c003e80 comp rx=0 tx=0).stop 2024-05-27T07:51:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.830+0000 7f3349c8b640 1 -- 172.21.15.114:0/2325109948 shutdown_connections 2024-05-27T07:51:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.830+0000 7f3349c8b640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f332c0b2aa0 0x7f332c0b4fb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.830+0000 7f3349c8b640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3344118530 0x7f3344118940 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.830+0000 7f3349c8b640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f334411a8e0 0x7f3344114310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.830+0000 7f3349c8b640 1 --2- 172.21.15.114:0/2325109948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f334410a1f0 0x7f3344113dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.830+0000 7f3349c8b640 1 -- 172.21.15.114:0/2325109948 >> 172.21.15.114:0/2325109948 conn(0x7f334406e8f0 msgr2=0x7f3344072220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.830+0000 7f3349c8b640 1 -- 172.21.15.114:0/2325109948 shutdown_connections 2024-05-27T07:51:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:08.830+0000 7f3349c8b640 1 -- 172.21.15.114:0/2325109948 wait complete. 2024-05-27T07:51:09.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.067+0000 7f8e49494700 1 -- 172.21.15.114:0/2221690374 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3400e3e0 con 0x7f8e4410b270 2024-05-27T07:51:09.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.067+0000 7f8e4c6fa700 1 -- 172.21.15.114:0/2221690374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4410b270 msgr2=0x7f8e4411d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:09.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.067+0000 7f8e4c6fa700 1 --2- 172.21.15.114:0/2221690374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4410b270 0x7f8e4411d3e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8e340098f0 tx=0x7f8e34009c00 comp rx=0 tx=0).stop 2024-05-27T07:51:09.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.068+0000 7f8e4c6fa700 1 -- 172.21.15.114:0/2221690374 shutdown_connections 2024-05-27T07:51:09.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.068+0000 7f8e4c6fa700 1 --2- 172.21.15.114:0/2221690374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e4411d920 0x7f8e4411fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.068+0000 7f8e4c6fa700 1 --2- 172.21.15.114:0/2221690374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e4410a920 0x7f8e4410ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.068+0000 7f8e4c6fa700 1 --2- 172.21.15.114:0/2221690374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4410b270 0x7f8e4411d3e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.068+0000 7f8e4c6fa700 1 -- 172.21.15.114:0/2221690374 >> 172.21.15.114:0/2221690374 conn(0x7f8e4406f4c0 msgr2=0x7f8e4406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:09.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.071+0000 7f8e4c6fa700 1 -- 172.21.15.114:0/2221690374 shutdown_connections 2024-05-27T07:51:09.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.072+0000 7f8e4c6fa700 1 -- 172.21.15.114:0/2221690374 wait complete. 2024-05-27T07:51:09.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.073+0000 7f8e4c6fa700 1 Processor -- start 2024-05-27T07:51:09.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.073+0000 7f8e4c6fa700 1 -- start start 2024-05-27T07:51:09.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.074+0000 7f8e4c6fa700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4410a920 0x7f8e441a9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.074+0000 7f8e4c6fa700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e4410b270 0x7f8e441aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.074+0000 7f8e4c6fa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e4411d920 0x7f8e441ae570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.074+0000 7f8e4c6fa700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e44123330 con 0x7f8e4411d920 2024-05-27T07:51:09.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.074+0000 7f8e4c6fa700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8e441231b0 con 0x7f8e4410a920 2024-05-27T07:51:09.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.074+0000 7f8e4c6fa700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e441234b0 con 0x7f8e4410b270 2024-05-27T07:51:09.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.074+0000 7f8e4ac97700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e4411d920 0x7f8e441ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:09.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.074+0000 7f8e4ac97700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e4411d920 0x7f8e441ae570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34168/0 (socket says 172.21.15.114:34168) 2024-05-27T07:51:09.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.074+0000 7f8e4ac97700 1 -- 172.21.15.114:0/3285207692 learned_addr learned my addr 172.21.15.114:0/3285207692 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:09.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.074+0000 7f8e4a496700 1 --2- 172.21.15.114:0/3285207692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4410a920 0x7f8e441a9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:09.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.075+0000 7f8e4a496700 1 -- 172.21.15.114:0/3285207692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e4410b270 msgr2=0x7f8e441aa2e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:51:09.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.075+0000 7f8e4a496700 1 --2- 172.21.15.114:0/3285207692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e4410b270 0x7f8e441aa2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.075+0000 7f8e4a496700 1 -- 172.21.15.114:0/3285207692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e4411d920 msgr2=0x7f8e441ae570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:09.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.075+0000 7f8e4a496700 1 --2- 172.21.15.114:0/3285207692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e4411d920 0x7f8e441ae570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.075+0000 7f8e4a496700 1 -- 172.21.15.114:0/3285207692 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e441aec10 con 0x7f8e4410a920 2024-05-27T07:51:09.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.075+0000 7f8e4a496700 1 --2- 172.21.15.114:0/3285207692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4410a920 0x7f8e441a9d40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8e4000e9f0 tx=0x7f8e4000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:09.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.076+0000 7f8e3b7fe700 1 -- 172.21.15.114:0/3285207692 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4000cbd0 con 0x7f8e4410a920 2024-05-27T07:51:09.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.077+0000 7f8e4c6fa700 1 -- 172.21.15.114:0/3285207692 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e441aef00 con 0x7f8e4410a920 2024-05-27T07:51:09.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.077+0000 7f8e4c6fa700 1 -- 172.21.15.114:0/3285207692 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e44112bc0 con 0x7f8e4410a920 2024-05-27T07:51:09.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.077+0000 7f8e3b7fe700 1 -- 172.21.15.114:0/3285207692 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e4000cd70 con 0x7f8e4410a920 2024-05-27T07:51:09.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.077+0000 7f8e3b7fe700 1 -- 172.21.15.114:0/3285207692 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e40018710 con 0x7f8e4410a920 2024-05-27T07:51:09.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.085+0000 7f8e5b10fb80 1 Processor -- start 2024-05-27T07:51:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.088+0000 7f8e5b10fb80 1 -- start start 2024-05-27T07:51:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.088+0000 7f8e5b10fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb192e40 0x5609fb317b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.088+0000 7f8e5b10fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb318070 0x5609fb311be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.088+0000 7f8e5b10fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb312120 0x5609fb312530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.088+0000 7f8e5b10fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5609fb23fd70 con 0x5609fb318070 2024-05-27T07:51:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.088+0000 7f8e5b10fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5609fb22f910 con 0x5609fb192e40 2024-05-27T07:51:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.088+0000 7f8e5b10fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5609fb233950 con 0x5609fb312120 2024-05-27T07:51:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.089+0000 7f8e4ac97700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb192e40 0x5609fb317b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.090+0000 7f8e4ac97700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb192e40 0x5609fb317b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51746/0 (socket says 172.21.15.114:51746) 2024-05-27T07:51:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.090+0000 7f8e4ac97700 1 -- 172.21.15.114:0/4263591729 learned_addr learned my addr 172.21.15.114:0/4263591729 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.090+0000 7f8e3b7fe700 1 -- 172.21.15.114:0/3285207692 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f8e400188b0 con 0x7f8e4410a920 2024-05-27T07:51:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.091+0000 7f8e4ac97700 1 -- 172.21.15.114:0/4263591729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb312120 msgr2=0x5609fb312530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:51:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.091+0000 7f8e4ac97700 1 --2- 172.21.15.114:0/4263591729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb312120 0x5609fb312530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.091+0000 7f8e4ac97700 1 -- 172.21.15.114:0/4263591729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb318070 msgr2=0x5609fb311be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:09.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.091+0000 7f8e4ac97700 1 --2- 172.21.15.114:0/4263591729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb318070 0x5609fb311be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.091+0000 7f8e4ac97700 1 -- 172.21.15.114:0/4263591729 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609fb1d8fb0 con 0x5609fb192e40 2024-05-27T07:51:09.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.092+0000 7f8e3b7fe700 1 --2- 172.21.15.114:0/3285207692 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8e300b2bc0 0x7f8e300b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.092+0000 7f8e3b7fe700 1 -- 172.21.15.114:0/3285207692 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9642+0+0 (secure 0 0 0) 0x7f8e400e4aa0 con 0x7f8e4410a920 2024-05-27T07:51:09.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.092+0000 7f8e4a496700 1 --2- 172.21.15.114:0/3285207692 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8e300b2bc0 0x7f8e300b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:09.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.092+0000 7f8e4ac97700 1 --2- 172.21.15.114:0/4263591729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb192e40 0x5609fb317b30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00f6a0 tx=0x7f8e3c0099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfd5f1768e6ad56e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:09.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.092+0000 7f8e38ff9700 1 -- 172.21.15.114:0/4263591729 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c003a00 con 0x5609fb192e40 2024-05-27T07:51:09.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.092+0000 7f8e38ff9700 1 -- 172.21.15.114:0/4263591729 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e3c03b030 con 0x5609fb192e40 2024-05-27T07:51:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.094+0000 7f8e4a496700 1 --2- 172.21.15.114:0/3285207692 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8e300b2bc0 0x7f8e300b50d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5609fb30e2a0 tx=0x7f8e4000ef90 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.094+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/4263591729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb192e40 msgr2=0x5609fb317b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.094+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/4263591729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb192e40 0x5609fb317b30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00f6a0 tx=0x7f8e3c0099e0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.094+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/4263591729 shutdown_connections 2024-05-27T07:51:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.094+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/4263591729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb312120 0x5609fb312530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.094+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/4263591729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb192e40 0x5609fb317b30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.094+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/4263591729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb318070 0x5609fb311be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.094+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/4263591729 >> 172.21.15.114:0/4263591729 conn(0x5609fb195770 msgr2=0x5609fb30cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:09.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.095+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/4263591729 shutdown_connections 2024-05-27T07:51:09.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.096+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/4263591729 wait complete. 2024-05-27T07:51:09.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.096+0000 7f8e5b10fb80 1 -- start start 2024-05-27T07:51:09.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.097+0000 7f8e5b10fb80 1 Processor -- start 2024-05-27T07:51:09.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.097+0000 7f8e5b10fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb312120 0x5609fb31e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.097+0000 7f8e5b10fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb318070 0x5609fb31b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.097+0000 7f8e5b10fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb31b980 0x5609fb31bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.098+0000 7f8e49c95700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb318070 0x5609fb31b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:09.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.098+0000 7f8e4ac97700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb312120 0x5609fb31e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:09.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.098+0000 7f8e49c95700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb318070 0x5609fb31b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51754/0 (socket says 172.21.15.114:51754) 2024-05-27T07:51:09.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.098+0000 7f8e4ac97700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb312120 0x5609fb31e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34192/0 (socket says 172.21.15.114:34192) 2024-05-27T07:51:09.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.098+0000 7f8e49c95700 1 -- 172.21.15.114:0/834143334 learned_addr learned my addr 172.21.15.114:0/834143334 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:09.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.098+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/834143334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5609fb135fa0 con 0x5609fb312120 2024-05-27T07:51:09.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.098+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/834143334 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5609fb197da0 con 0x5609fb318070 2024-05-27T07:51:09.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.098+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/834143334 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5609fb31ec00 con 0x5609fb31b980 2024-05-27T07:51:09.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.098+0000 7f8e5b10fb80 10 client.0 _ll_register_callbacks cb 0x5609fb1c65b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:09.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.099+0000 7f8e49c95700 1 -- 172.21.15.114:0/834143334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb31b980 msgr2=0x5609fb31bd90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:51:09.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.099+0000 7f8e49c95700 1 --2- 172.21.15.114:0/834143334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb31b980 0x5609fb31bd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.099+0000 7f8e49c95700 1 -- 172.21.15.114:0/834143334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb312120 msgr2=0x5609fb31e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:09.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.099+0000 7f8e49c95700 1 --2- 172.21.15.114:0/834143334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb312120 0x5609fb31e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.099+0000 7f8e49c95700 1 -- 172.21.15.114:0/834143334 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609fb17f8d0 con 0x5609fb318070 2024-05-27T07:51:09.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.099+0000 7f8e49c95700 1 --2- 172.21.15.114:0/834143334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb318070 0x5609fb31b440 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400fac0 tx=0x7f8e34005d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0b2f9ab29816ede server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:09.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.099+0000 7f8e2dffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:51:09.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.100+0000 7f8e2dffb700 1 -- 172.21.15.114:0/834143334 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34005520 con 0x5609fb318070 2024-05-27T07:51:09.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.100+0000 7f8e2dffb700 1 -- 172.21.15.114:0/834143334 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9642+0+0 (secure 0 0 0) 0x7f8e34003680 con 0x5609fb318070 2024-05-27T07:51:09.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.100+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/834143334 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5609fb241cc0 con 0x5609fb318070 2024-05-27T07:51:09.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.101+0000 7f8e2dffb700 1 -- 172.21.15.114:0/834143334 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f8e34029050 con 0x5609fb318070 2024-05-27T07:51:09.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.101+0000 7f8e5b10fb80 10 client.27479 fetch_fsmap learned FSMap version 63 2024-05-27T07:51:09.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.101+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/834143334 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x5609fb1e9760 con 0x5609fb318070 2024-05-27T07:51:09.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.105+0000 7f8e2dffb700 1 -- 172.21.15.114:0/834143334 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5609fb241cc0 con 0x5609fb318070 2024-05-27T07:51:09.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.105+0000 7f8e2dffb700 1 -- 172.21.15.114:0/834143334 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 63) ==== 5910+0+0 (secure 0 0 0) 0x7f8e3400ba30 con 0x5609fb318070 2024-05-27T07:51:09.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.106+0000 7f8e5b10fb80 10 client.27479 fetch_fsmap finished waiting for FSMap version 63 2024-05-27T07:51:09.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.106+0000 7f8e5b10fb80 10 client.27479 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:09.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.106+0000 7f8e5b10fb80 20 client.27479 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:09.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.106+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/834143334 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609fb32c0a0 0x5609fb32e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.106+0000 7f8e5b10fb80 4 client.27479 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:09.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.106+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/834143334 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609fb23b4a0 con 0x5609fb32c0a0 2024-05-27T07:51:09.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.107+0000 7f8e4ac97700 1 --2- 172.21.15.114:0/834143334 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609fb32c0a0 0x5609fb32e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:09.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.109+0000 7f8e4ac97700 1 --2- 172.21.15.114:0/834143334 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609fb32c0a0 0x5609fb32e530 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76ebdd5112702ea4 server_cookie=376285d06cf51be1 in_seq=0 out_seq=0 2024-05-27T07:51:09.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.113+0000 7f8e2dffb700 10 client.27479 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:09.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.115+0000 7f8e2dffb700 1 -- 172.21.15.114:0/834143334 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5609fb23b4a0 con 0x5609fb32c0a0 2024-05-27T07:51:09.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.115+0000 7f8e2dffb700 10 client.27479 handle_command_reply: tid=0 2024-05-27T07:51:09.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.116+0000 7f8e5b10fb80 1 client.27479 shutdown 2024-05-27T07:51:09.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.116+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/834143334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb318070 msgr2=0x5609fb31b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:09.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.116+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/834143334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb318070 0x5609fb31b440 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400fac0 tx=0x7f8e34005d00 comp rx=0 tx=0).stop 2024-05-27T07:51:09.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.118+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/834143334 shutdown_connections 2024-05-27T07:51:09.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.118+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/834143334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb31b980 0x5609fb31bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.118+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/834143334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb312120 0x5609fb31e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.118+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/834143334 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609fb32c0a0 0x5609fb32e530 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.118+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/834143334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb318070 0x5609fb31b440 secure :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400fac0 tx=0x7f8e34005d00 comp rx=0 tx=0).stop 2024-05-27T07:51:09.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.118+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/834143334 >> 172.21.15.114:0/834143334 conn(0x5609fb195770 msgr2=0x5609fb310bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:09.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.118+0000 7f8e2dffb700 0 client.27479 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:09.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.123+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/834143334 shutdown_connections 2024-05-27T07:51:09.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.126+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/834143334 wait complete. 2024-05-27T07:51:09.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.127+0000 7f8e5b10fb80 20 client.27479 trim_cache size 0 max 16384 2024-05-27T07:51:09.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.165+0000 7f8e5b10fb80 1 Processor -- start 2024-05-27T07:51:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.168+0000 7f8e5b10fb80 1 -- start start 2024-05-27T07:51:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.168+0000 7f8e5b10fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb195770 0x5609fb311be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.168+0000 7f8e5b10fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb312120 0x5609fb32e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.168+0000 7f8e5b10fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb31b980 0x5609fb318d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.168+0000 7f8e5b10fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e28010a40 con 0x5609fb31b980 2024-05-27T07:51:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.168+0000 7f8e5b10fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2800f1b0 con 0x5609fb195770 2024-05-27T07:51:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.168+0000 7f8e5b10fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2800d020 con 0x5609fb312120 2024-05-27T07:51:09.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.169+0000 7f8e49c95700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb312120 0x5609fb32e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:09.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.169+0000 7f8e49c95700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb312120 0x5609fb32e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:49006/0 (socket says 172.21.15.114:49006) 2024-05-27T07:51:09.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.169+0000 7f8e49c95700 1 -- 172.21.15.114:0/188669317 learned_addr learned my addr 172.21.15.114:0/188669317 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:09.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.170+0000 7f8e4ac97700 1 --2- 172.21.15.114:0/188669317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb31b980 0x5609fb318d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:09.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.170+0000 7f8e4ac97700 1 --2- 172.21.15.114:0/188669317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb195770 0x5609fb311be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:09.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.172+0000 7f8e4ac97700 1 -- 172.21.15.114:0/188669317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb312120 msgr2=0x5609fb32e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:09.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.172+0000 7f8e4ac97700 1 --2- 172.21.15.114:0/188669317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb312120 0x5609fb32e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.172+0000 7f8e4ac97700 1 -- 172.21.15.114:0/188669317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb195770 msgr2=0x5609fb311be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:09.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.172+0000 7f8e4ac97700 1 --2- 172.21.15.114:0/188669317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb195770 0x5609fb311be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.172+0000 7f8e4ac97700 1 -- 172.21.15.114:0/188669317 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609fb31ec90 con 0x5609fb31b980 2024-05-27T07:51:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.174+0000 7f8e4ac97700 1 --2- 172.21.15.114:0/188669317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb31b980 0x5609fb318d80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00fd40 tx=0x7f8e3c00f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a16829b975786d3a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:09.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.176+0000 7f8e2ffff700 1 -- 172.21.15.114:0/188669317 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c003a00 con 0x5609fb31b980 2024-05-27T07:51:09.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.176+0000 7f8e2ffff700 1 -- 172.21.15.114:0/188669317 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e3c003ba0 con 0x5609fb31b980 2024-05-27T07:51:09.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.176+0000 7f8e2ffff700 1 -- 172.21.15.114:0/188669317 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c0045f0 con 0x5609fb31b980 2024-05-27T07:51:09.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.178+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/188669317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb31b980 msgr2=0x5609fb318d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:09.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.178+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/188669317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb31b980 0x5609fb318d80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00fd40 tx=0x7f8e3c00f1b0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.180+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/188669317 shutdown_connections 2024-05-27T07:51:09.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.180+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/188669317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb31b980 0x5609fb318d80 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.180+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/188669317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb195770 0x5609fb311be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.180+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/188669317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb312120 0x5609fb32e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.180+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/188669317 >> 172.21.15.114:0/188669317 conn(0x5609fb31c0f0 msgr2=0x5609fb318670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:09.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.180+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/188669317 shutdown_connections 2024-05-27T07:51:09.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.181+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/188669317 wait complete. 2024-05-27T07:51:09.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.181+0000 7f8e5b10fb80 1 -- start start 2024-05-27T07:51:09.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.181+0000 7f8e5b10fb80 1 Processor -- start 2024-05-27T07:51:09.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.182+0000 7f8e5b10fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb195770 0x5609fb323f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.182+0000 7f8e5b10fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb312120 0x5609fb324480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.182+0000 7f8e5b10fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb31b980 0x5609fb320f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.182+0000 7f8e5b10fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2800d670 con 0x5609fb312120 2024-05-27T07:51:09.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.182+0000 7f8e5b10fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2800eae0 con 0x5609fb31b980 2024-05-27T07:51:09.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.182+0000 7f8e5b10fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2800dd40 con 0x5609fb195770 2024-05-27T07:51:09.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.182+0000 7f8e5b10fb80 10 client.0 _ll_register_callbacks cb 0x5609fb1c65b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:09.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.183+0000 7f8e49c95700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb312120 0x5609fb324480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:09.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.183+0000 7f8e49c95700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb312120 0x5609fb324480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34210/0 (socket says 172.21.15.114:34210) 2024-05-27T07:51:09.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.183+0000 7f8e49c95700 1 -- 172.21.15.114:0/1082436216 learned_addr learned my addr 172.21.15.114:0/1082436216 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:09.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.187+0000 7f8e49c95700 1 -- 172.21.15.114:0/1082436216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb195770 msgr2=0x5609fb323f40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:51:09.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.187+0000 7f8e49c95700 1 --2- 172.21.15.114:0/1082436216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb195770 0x5609fb323f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.187+0000 7f8e49c95700 1 -- 172.21.15.114:0/1082436216 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb31b980 msgr2=0x5609fb320f10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:51:09.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.187+0000 7f8e49c95700 1 --2- 172.21.15.114:0/1082436216 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb31b980 0x5609fb320f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.187+0000 7f8e49c95700 1 -- 172.21.15.114:0/1082436216 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609fb311cc0 con 0x5609fb312120 2024-05-27T07:51:09.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.187+0000 7f8e49c95700 1 --2- 172.21.15.114:0/1082436216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb312120 0x5609fb324480 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x5609fb309190 tx=0x7f8e3400f820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a651dce177b3f57a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:09.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.187+0000 7f8e2dffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:09.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.188+0000 7f8e2dffb700 1 -- 172.21.15.114:0/1082436216 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34032060 con 0x5609fb312120 2024-05-27T07:51:09.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.188+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/1082436216 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e3c003ba0 con 0x5609fb312120 2024-05-27T07:51:09.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.188+0000 7f8e2dffb700 1 -- 172.21.15.114:0/1082436216 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9642+0+0 (secure 0 0 0) 0x7f8e34017880 con 0x5609fb312120 2024-05-27T07:51:09.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.188+0000 7f8e2dffb700 1 -- 172.21.15.114:0/1082436216 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f8e34039050 con 0x5609fb312120 2024-05-27T07:51:09.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.189+0000 7f8e5b10fb80 10 client.18042 fetch_fsmap learned FSMap version 63 2024-05-27T07:51:09.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.189+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/1082436216 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x5609fb396ef0 con 0x5609fb312120 2024-05-27T07:51:09.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.190+0000 7f8e2dffb700 1 -- 172.21.15.114:0/1082436216 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e3c003ba0 con 0x5609fb312120 2024-05-27T07:51:09.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.190+0000 7f8e2dffb700 1 -- 172.21.15.114:0/1082436216 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 63) ==== 5910+0+0 (secure 0 0 0) 0x7f8e3400ba20 con 0x5609fb312120 2024-05-27T07:51:09.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.190+0000 7f8e5b10fb80 10 client.18042 fetch_fsmap finished waiting for FSMap version 63 2024-05-27T07:51:09.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.190+0000 7f8e5b10fb80 10 client.18042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:09.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.190+0000 7f8e5b10fb80 20 client.18042 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:09.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.190+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/1082436216 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609fb3975b0 0x5609fb3999d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:09.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.190+0000 7f8e5b10fb80 4 client.18042 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:51:09.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.190+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/1082436216 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5609fb0b2250 con 0x5609fb3975b0 2024-05-27T07:51:09.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.195+0000 7f8e4ac97700 1 --2- 172.21.15.114:0/1082436216 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609fb3975b0 0x5609fb3999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:09.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.200+0000 7f8e4ac97700 1 --2- 172.21.15.114:0/1082436216 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609fb3975b0 0x5609fb3999d0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d67196181cc038f server_cookie=c7080e9227cc48d9 in_seq=0 out_seq=0 2024-05-27T07:51:09.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.200+0000 7f8e2dffb700 10 client.18042 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:09.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.200+0000 7f8e2dffb700 1 -- 172.21.15.114:0/1082436216 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5609fb0b2250 con 0x5609fb3975b0 2024-05-27T07:51:09.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.200+0000 7f8e2dffb700 10 client.18042 handle_command_reply: tid=0 2024-05-27T07:51:09.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.201+0000 7f8e5b10fb80 1 client.18042 shutdown 2024-05-27T07:51:09.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.201+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/1082436216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb312120 msgr2=0x5609fb324480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:09.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.201+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/1082436216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb312120 0x5609fb324480 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x5609fb309190 tx=0x7f8e3400f820 comp rx=0 tx=0).stop 2024-05-27T07:51:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.202+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/1082436216 shutdown_connections 2024-05-27T07:51:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.202+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/1082436216 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609fb3975b0 0x5609fb3999d0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.202+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/1082436216 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609fb31b980 0x5609fb320f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.202+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/1082436216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609fb195770 0x5609fb323f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.202+0000 7f8e5b10fb80 1 --2- 172.21.15.114:0/1082436216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609fb312120 0x5609fb324480 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.202+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/1082436216 >> 172.21.15.114:0/1082436216 conn(0x5609fb31c0f0 msgr2=0x5609fb3163a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.205+0000 7f8e2dffb700 0 client.18042 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.205+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/1082436216 shutdown_connections 2024-05-27T07:51:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.205+0000 7f8e5b10fb80 1 -- 172.21.15.114:0/1082436216 wait complete. 2024-05-27T07:51:09.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.206+0000 7f8e5b10fb80 20 client.18042 trim_cache size 0 max 16384 2024-05-27T07:51:09.207 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:51:09.207 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:51:09.207 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:51:09.207 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:51:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.207+0000 7f8e4c6fa700 1 -- 172.21.15.114:0/3285207692 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8e300b2bc0 msgr2=0x7f8e300b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.207+0000 7f8e4c6fa700 1 --2- 172.21.15.114:0/3285207692 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8e300b2bc0 0x7f8e300b50d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5609fb30e2a0 tx=0x7f8e4000ef90 comp rx=0 tx=0).stop 2024-05-27T07:51:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.207+0000 7f8e4c6fa700 1 -- 172.21.15.114:0/3285207692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4410a920 msgr2=0x7f8e441a9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.207+0000 7f8e4c6fa700 1 --2- 172.21.15.114:0/3285207692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4410a920 0x7f8e441a9d40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8e4000e9f0 tx=0x7f8e4000ed00 comp rx=0 tx=0).stop 2024-05-27T07:51:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.207+0000 7f8e4c6fa700 1 -- 172.21.15.114:0/3285207692 shutdown_connections 2024-05-27T07:51:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.207+0000 7f8e4c6fa700 1 --2- 172.21.15.114:0/3285207692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e4411d920 0x7f8e441ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.207+0000 7f8e4c6fa700 1 --2- 172.21.15.114:0/3285207692 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8e300b2bc0 0x7f8e300b50d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.208+0000 7f8e4c6fa700 1 --2- 172.21.15.114:0/3285207692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4410a920 0x7f8e441a9d40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.208+0000 7f8e4c6fa700 1 --2- 172.21.15.114:0/3285207692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e4410b270 0x7f8e441aa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.208+0000 7f8e4c6fa700 1 -- 172.21.15.114:0/3285207692 >> 172.21.15.114:0/3285207692 conn(0x7f8e4406f4c0 msgr2=0x7f8e4411b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:09.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.209+0000 7f8e4c6fa700 1 -- 172.21.15.114:0/3285207692 shutdown_connections 2024-05-27T07:51:09.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:09.210+0000 7f8e4c6fa700 1 -- 172.21.15.114:0/3285207692 wait complete. 2024-05-27T07:51:09.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1115291 9.93 MB/sec execute 441 sec latency 167.552 ms 2024-05-27T07:51:09.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:09 smithi186 ceph-mon[36647]: pgmap v377: 129 pgs: 19 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 60 active+clean; 73 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 209 op/s 2024-05-27T07:51:09.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:09 smithi114 ceph-mon[32038]: pgmap v377: 129 pgs: 19 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 60 active+clean; 73 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 209 op/s 2024-05-27T07:51:09.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:09 smithi163 ceph-mon[36451]: pgmap v377: 129 pgs: 19 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 60 active+clean; 73 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 209 op/s 2024-05-27T07:51:10.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1115549 9.91 MB/sec execute 442 sec latency 57.161 ms 2024-05-27T07:51:11.103 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:51:11.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:51:11.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1116730 9.90 MB/sec execute 443 sec latency 85.058 ms 2024-05-27T07:51:11.400 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:11.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:11 smithi186 ceph-mon[36647]: pgmap v378: 129 pgs: 12 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 80 active+clean; 65 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 2.1 MiB/s wr, 199 op/s 2024-05-27T07:51:11.791 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:11 smithi114 ceph-mon[32038]: pgmap v378: 129 pgs: 12 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 80 active+clean; 65 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 2.1 MiB/s wr, 199 op/s 2024-05-27T07:51:11.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:11 smithi163 ceph-mon[36451]: pgmap v378: 129 pgs: 12 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 80 active+clean; 65 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 2.1 MiB/s wr, 199 op/s 2024-05-27T07:51:12.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1118500 9.90 MB/sec execute 444 sec latency 115.237 ms 2024-05-27T07:51:13.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1119409 9.89 MB/sec execute 445 sec latency 88.049 ms 2024-05-27T07:51:13.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.415+0000 7f83e2acd700 1 -- 172.21.15.114:0/4002605554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83dc10a3d0 msgr2=0x7f83dc10a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:13.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.415+0000 7f83e2acd700 1 --2- 172.21.15.114:0/4002605554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83dc10a3d0 0x7f83dc10a7e0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009990 tx=0x7f83d0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:51:13.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.415+0000 7f83e2acd700 1 -- 172.21.15.114:0/4002605554 shutdown_connections 2024-05-27T07:51:13.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.415+0000 7f83e2acd700 1 --2- 172.21.15.114:0/4002605554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83dc103120 0x7f83dc111110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:13.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.415+0000 7f83e2acd700 1 --2- 172.21.15.114:0/4002605554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83dc10a3d0 0x7f83dc10a7e0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:13.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.416+0000 7f83e2acd700 1 --2- 172.21.15.114:0/4002605554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83dc1027d0 0x7f83dc102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:13.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.416+0000 7f83e2acd700 1 -- 172.21.15.114:0/4002605554 >> 172.21.15.114:0/4002605554 conn(0x7f83dc0fe6e0 msgr2=0x7f83dc100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:13.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.416+0000 7f83e2acd700 1 -- 172.21.15.114:0/4002605554 shutdown_connections 2024-05-27T07:51:13.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.416+0000 7f83e2acd700 1 -- 172.21.15.114:0/4002605554 wait complete. 2024-05-27T07:51:13.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.417+0000 7f83e2acd700 1 Processor -- start 2024-05-27T07:51:13.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.417+0000 7f83e2acd700 1 -- start start 2024-05-27T07:51:13.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.417+0000 7f83e2acd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83dc1027d0 0x7f83dc1039b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:13.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.417+0000 7f83e2acd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83dc103120 0x7f83dc103ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:13.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.417+0000 7f83e2acd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83dc10a3d0 0x7f83dc104430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:13.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.417+0000 7f83e2acd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc114490 con 0x7f83dc103120 2024-05-27T07:51:13.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.417+0000 7f83e2acd700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc114310 con 0x7f83dc10a3d0 2024-05-27T07:51:13.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.417+0000 7f83e2acd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc114610 con 0x7f83dc1027d0 2024-05-27T07:51:13.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.417+0000 7f83dbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83dc103120 0x7f83dc103ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:13.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.417+0000 7f83dbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83dc103120 0x7f83dc103ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34226/0 (socket says 172.21.15.114:34226) 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.417+0000 7f83dbfff700 1 -- 172.21.15.114:0/2110941006 learned_addr learned my addr 172.21.15.114:0/2110941006 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.417+0000 7f83e0869700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83dc1027d0 0x7f83dc1039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.418+0000 7f83e106a700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83dc10a3d0 0x7f83dc104430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.418+0000 7f83dbfff700 1 -- 172.21.15.114:0/2110941006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83dc1027d0 msgr2=0x7f83dc1039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.418+0000 7f83dbfff700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83dc1027d0 0x7f83dc1039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.418+0000 7f83dbfff700 1 -- 172.21.15.114:0/2110941006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83dc10a3d0 msgr2=0x7f83dc104430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.418+0000 7f83dbfff700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83dc10a3d0 0x7f83dc104430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.418+0000 7f83dbfff700 1 -- 172.21.15.114:0/2110941006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83dc1a9200 con 0x7f83dc103120 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.418+0000 7f83e106a700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83dc10a3d0 0x7f83dc104430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.418+0000 7f83dbfff700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83dc103120 0x7f83dc103ef0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f83c800e970 tx=0x7f83c800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.419+0000 7f83d9ffb700 1 -- 172.21.15.114:0/2110941006 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c800cbd0 con 0x7f83dc103120 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.419+0000 7f83d9ffb700 1 -- 172.21.15.114:0/2110941006 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83c800cd70 con 0x7f83dc103120 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.419+0000 7f83d9ffb700 1 -- 172.21.15.114:0/2110941006 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8005680 con 0x7f83dc103120 2024-05-27T07:51:13.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.419+0000 7f83e2acd700 1 -- 172.21.15.114:0/2110941006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83dc1a94f0 con 0x7f83dc103120 2024-05-27T07:51:13.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.419+0000 7f83e2acd700 1 -- 172.21.15.114:0/2110941006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83dc1a9a30 con 0x7f83dc103120 2024-05-27T07:51:13.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.420+0000 7f83e2acd700 1 -- 172.21.15.114:0/2110941006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83dc10c840 con 0x7f83dc103120 2024-05-27T07:51:13.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.423+0000 7f83d9ffb700 1 -- 172.21.15.114:0/2110941006 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f83c8004750 con 0x7f83dc103120 2024-05-27T07:51:13.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.424+0000 7f83d9ffb700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83cc0b2b40 0x7f83cc0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:13.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.424+0000 7f83e0869700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83cc0b2b40 0x7f83cc0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:13.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.424+0000 7f83d9ffb700 1 -- 172.21.15.114:0/2110941006 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9642+0+0 (secure 0 0 0) 0x7f83c80e33d0 con 0x7f83dc103120 2024-05-27T07:51:13.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.425+0000 7f83e0869700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83cc0b2b40 0x7f83cc0b5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f83d000b2e0 tx=0x7f83d0005960 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:13.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.429+0000 7f83d9ffb700 1 -- 172.21.15.114:0/2110941006 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f83c80aa9d0 con 0x7f83dc103120 2024-05-27T07:51:13.481 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:13 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:51:13.481 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:13 smithi186 systemd[1]: Stopped Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:51:13.481 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:13 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.865s CPU time. 2024-05-27T07:51:13.481 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:13 smithi186 systemd[1]: Starting Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:51:13.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:13 smithi114 ceph-mon[32038]: pgmap v379: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 64 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.9 MiB/s wr, 208 op/s 2024-05-27T07:51:13.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.730+0000 7f83e2acd700 1 -- 172.21.15.114:0/2110941006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83dc104ee0 con 0x7f83dc103120 2024-05-27T07:51:13.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.731+0000 7f83d9ffb700 1 -- 172.21.15.114:0/2110941006 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+8521 (secure 0 0 0) 0x7f83c80aa2d0 con 0x7f83dc103120 2024-05-27T07:51:13.734 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:51:13.734 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2110370477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2110370477},{"type":"v1","addr":"172.21.15.163:6835","nonce":2110370477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":63,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:50:42.702888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27383":{"gid":27383,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/3045713021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":3045713021},{"type":"v1","addr":"172.21.15.186:6833","nonce":3045713021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:51:13.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.734+0000 7f83e2acd700 1 -- 172.21.15.114:0/2110941006 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83cc0b2b40 msgr2=0x7f83cc0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:13.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.734+0000 7f83e2acd700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83cc0b2b40 0x7f83cc0b5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f83d000b2e0 tx=0x7f83d0005960 comp rx=0 tx=0).stop 2024-05-27T07:51:13.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.734+0000 7f83e2acd700 1 -- 172.21.15.114:0/2110941006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83dc103120 msgr2=0x7f83dc103ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:13.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.734+0000 7f83e2acd700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83dc103120 0x7f83dc103ef0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f83c800e970 tx=0x7f83c800ec80 comp rx=0 tx=0).stop 2024-05-27T07:51:13.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.734+0000 7f83e2acd700 1 -- 172.21.15.114:0/2110941006 shutdown_connections 2024-05-27T07:51:13.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.734+0000 7f83e2acd700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83cc0b2b40 0x7f83cc0b5050 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:13.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.734+0000 7f83e2acd700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83dc10a3d0 0x7f83dc104430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:13.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.734+0000 7f83e2acd700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83dc1027d0 0x7f83dc1039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:13.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.734+0000 7f83e2acd700 1 --2- 172.21.15.114:0/2110941006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83dc103120 0x7f83dc103ef0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:13.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.734+0000 7f83e2acd700 1 -- 172.21.15.114:0/2110941006 >> 172.21.15.114:0/2110941006 conn(0x7f83dc0fe6e0 msgr2=0x7f83dc100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:13.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.734+0000 7f83e2acd700 1 -- 172.21.15.114:0/2110941006 shutdown_connections 2024-05-27T07:51:13.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:13.734+0000 7f83e2acd700 1 -- 172.21.15.114:0/2110941006 wait complete. 2024-05-27T07:51:13.736 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 63 2024-05-27T07:51:13.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:13 smithi163 ceph-mon[36451]: pgmap v379: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 64 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.9 MiB/s wr, 208 op/s 2024-05-27T07:51:13.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:13 smithi186 ceph-mon[36647]: pgmap v379: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 64 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.9 MiB/s wr, 208 op/s 2024-05-27T07:51:13.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:13 smithi186 podman[75034]: 2024-05-27 07:51:13.390366991 +0000 UTC m=+0.018874119 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:51:14.287 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1121213 9.88 MB/sec execute 446 sec latency 82.177 ms 2024-05-27T07:51:14.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:51:14.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:14 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2110941006' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:51:14.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:14 smithi163 ceph-mon[36451]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:51:14.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:14 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:51:14.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:14 smithi163 ceph-mon[36451]: osdmap e122: 12 total, 12 up, 12 in 2024-05-27T07:51:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb225b53700 time 2024-05-27T07:51:14.430073+0000 2024-05-27T07:51:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:51:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb238398ba6] 2024-05-27T07:51:14.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb238398dd1] 2024-05-27T07:51:14.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55deb0ac423d] 2024-05-27T07:51:14.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55deb0aaf971] 2024-05-27T07:51:14.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 5: (SnapRealm::check_cache() const+0x13e) [0x55deb0aadfa0] 2024-05-27T07:51:14.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55deb0aae651] 2024-05-27T07:51:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55deb0a43347] 2024-05-27T07:51:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55deb073f87e] 2024-05-27T07:51:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55deb0744f2f] 2024-05-27T07:51:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55deb074f969] 2024-05-27T07:51:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 11: (MDLog::_replay_thread()+0x1f38) [0x55deb0ae848c] 2024-05-27T07:51:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55deb0af19cb] 2024-05-27T07:51:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 13: (Thread::entry_wrapper()+0x43) [0x7fb23836772d] 2024-05-27T07:51:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb238367749] 2024-05-27T07:51:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb236f0b1ca] 2024-05-27T07:51:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 16: clone() 2024-05-27T07:51:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.431+0000 7fb225b53700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb225b53700 time 2024-05-27T07:51:14.430073+0000 2024-05-27T07:51:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:51:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb238398ba6] 2024-05-27T07:51:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb238398dd1] 2024-05-27T07:51:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55deb0ac423d] 2024-05-27T07:51:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55deb0aaf971] 2024-05-27T07:51:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 5: (SnapRealm::check_cache() const+0x13e) [0x55deb0aadfa0] 2024-05-27T07:51:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55deb0aae651] 2024-05-27T07:51:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55deb0a43347] 2024-05-27T07:51:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55deb073f87e] 2024-05-27T07:51:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55deb0744f2f] 2024-05-27T07:51:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55deb074f969] 2024-05-27T07:51:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 11: (MDLog::_replay_thread()+0x1f38) [0x55deb0ae848c] 2024-05-27T07:51:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55deb0af19cb] 2024-05-27T07:51:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 13: (Thread::entry_wrapper()+0x43) [0x7fb23836772d] 2024-05-27T07:51:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb238367749] 2024-05-27T07:51:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb236f0b1ca] 2024-05-27T07:51:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 16: clone() 2024-05-27T07:51:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: *** Caught signal (Aborted) ** 2024-05-27T07:51:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: in thread 7fb225b53700 thread_name:md_log_replay 2024-05-27T07:51:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb236f15d20] 2024-05-27T07:51:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2: gsignal() 2024-05-27T07:51:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 3: abort() 2024-05-27T07:51:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb238398ced] 2024-05-27T07:51:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb238398dd1] 2024-05-27T07:51:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55deb0ac423d] 2024-05-27T07:51:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55deb0aaf971] 2024-05-27T07:51:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 8: (SnapRealm::check_cache() const+0x13e) [0x55deb0aadfa0] 2024-05-27T07:51:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55deb0aae651] 2024-05-27T07:51:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55deb0a43347] 2024-05-27T07:51:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55deb073f87e] 2024-05-27T07:51:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55deb0744f2f] 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55deb074f969] 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 14: (MDLog::_replay_thread()+0x1f38) [0x55deb0ae848c] 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55deb0af19cb] 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 16: (Thread::entry_wrapper()+0x43) [0x7fb23836772d] 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb238367749] 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb236f0b1ca] 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 19: clone() 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.434+0000 7fb225b53700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: in thread 7fb225b53700 thread_name:md_log_replay 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb236f15d20] 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2: gsignal() 2024-05-27T07:51:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 3: abort() 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb238398ced] 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb238398dd1] 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55deb0ac423d] 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55deb0aaf971] 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 8: (SnapRealm::check_cache() const+0x13e) [0x55deb0aadfa0] 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55deb0aae651] 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55deb0a43347] 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55deb073f87e] 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55deb0744f2f] 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55deb074f969] 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 14: (MDLog::_replay_thread()+0x1f38) [0x55deb0ae848c] 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55deb0af19cb] 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 16: (Thread::entry_wrapper()+0x43) [0x7fb23836772d] 2024-05-27T07:51:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb238367749] 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb236f0b1ca] 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 19: clone() 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: -1> 2024-05-27T07:51:14.431+0000 7fb225b53700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb225b53700 time 2024-05-27T07:51:14.430073+0000 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb238398ba6] 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb238398dd1] 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55deb0ac423d] 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55deb0aaf971] 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 5: (SnapRealm::check_cache() const+0x13e) [0x55deb0aadfa0] 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55deb0aae651] 2024-05-27T07:51:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55deb0a43347] 2024-05-27T07:51:14.742 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55deb073f87e] 2024-05-27T07:51:14.742 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55deb0744f2f] 2024-05-27T07:51:14.742 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55deb074f969] 2024-05-27T07:51:14.742 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 11: (MDLog::_replay_thread()+0x1f38) [0x55deb0ae848c] 2024-05-27T07:51:14.742 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55deb0af19cb] 2024-05-27T07:51:14.743 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 13: (Thread::entry_wrapper()+0x43) [0x7fb23836772d] 2024-05-27T07:51:14.743 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb238367749] 2024-05-27T07:51:14.743 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb236f0b1ca] 2024-05-27T07:51:14.743 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 16: clone() 2024-05-27T07:51:14.743 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.743 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 0> 2024-05-27T07:51:14.434+0000 7fb225b53700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:51:14.743 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: in thread 7fb225b53700 thread_name:md_log_replay 2024-05-27T07:51:14.743 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.743 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:14.743 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb236f15d20] 2024-05-27T07:51:14.743 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2: gsignal() 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 3: abort() 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb238398ced] 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb238398dd1] 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55deb0ac423d] 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55deb0aaf971] 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 8: (SnapRealm::check_cache() const+0x13e) [0x55deb0aadfa0] 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55deb0aae651] 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55deb0a43347] 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55deb073f87e] 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55deb0744f2f] 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55deb074f969] 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 14: (MDLog::_replay_thread()+0x1f38) [0x55deb0ae848c] 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55deb0af19cb] 2024-05-27T07:51:14.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 16: (Thread::entry_wrapper()+0x43) [0x7fb23836772d] 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb238367749] 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb236f0b1ca] 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 19: clone() 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: -1> 2024-05-27T07:51:14.431+0000 7fb225b53700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb225b53700 time 2024-05-27T07:51:14.430073+0000 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb238398ba6] 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb238398dd1] 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55deb0ac423d] 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55deb0aaf971] 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 5: (SnapRealm::check_cache() const+0x13e) [0x55deb0aadfa0] 2024-05-27T07:51:14.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55deb0aae651] 2024-05-27T07:51:14.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55deb0a43347] 2024-05-27T07:51:14.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55deb073f87e] 2024-05-27T07:51:14.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55deb0744f2f] 2024-05-27T07:51:14.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55deb074f969] 2024-05-27T07:51:14.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 11: (MDLog::_replay_thread()+0x1f38) [0x55deb0ae848c] 2024-05-27T07:51:14.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55deb0af19cb] 2024-05-27T07:51:14.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 13: (Thread::entry_wrapper()+0x43) [0x7fb23836772d] 2024-05-27T07:51:14.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb238367749] 2024-05-27T07:51:14.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb236f0b1ca] 2024-05-27T07:51:14.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 16: clone() 2024-05-27T07:51:14.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 0> 2024-05-27T07:51:14.434+0000 7fb225b53700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:51:14.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: in thread 7fb225b53700 thread_name:md_log_replay 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb236f15d20] 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2: gsignal() 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 3: abort() 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb238398ced] 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb238398dd1] 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55deb0ac423d] 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55deb0aaf971] 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 8: (SnapRealm::check_cache() const+0x13e) [0x55deb0aadfa0] 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55deb0aae651] 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55deb0a43347] 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55deb073f87e] 2024-05-27T07:51:14.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55deb0744f2f] 2024-05-27T07:51:14.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55deb074f969] 2024-05-27T07:51:14.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 14: (MDLog::_replay_thread()+0x1f38) [0x55deb0ae848c] 2024-05-27T07:51:14.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55deb0af19cb] 2024-05-27T07:51:14.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 16: (Thread::entry_wrapper()+0x43) [0x7fb23836772d] 2024-05-27T07:51:14.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb238367749] 2024-05-27T07:51:14.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb236f0b1ca] 2024-05-27T07:51:14.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 19: clone() 2024-05-27T07:51:14.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:51:14.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[65717]: 2024-05-27T07:51:14.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:14 smithi163 podman[73114]: 2024-05-27 07:51:14.494871009 +0000 UTC m=+0.019714658 container died 4e04f6774a3a1923d89bb5c20563d34f2fb5a54932f7537f27b6b8aa6378e890 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T07:51:14.757 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:14.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:14 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2110941006' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:51:14.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:14 smithi114 ceph-mon[32038]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:51:14.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:14 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:51:14.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:14 smithi114 ceph-mon[32038]: osdmap e122: 12 total, 12 up, 12 in 2024-05-27T07:51:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:14 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2110941006' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:51:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:14 smithi186 ceph-mon[36647]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:51:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:14 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:51:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:14 smithi186 ceph-mon[36647]: osdmap e122: 12 total, 12 up, 12 in 2024-05-27T07:51:14.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:14 smithi186 podman[75034]: 2024-05-27 07:51:14.600925456 +0000 UTC m=+1.229432574 container create 4a3816c2982f0bad95eb3dedd7ff1867f43bcbb4cb5a86af113f69c6bc32289a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:51:15.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1122693 9.89 MB/sec execute 447 sec latency 77.399 ms 2024-05-27T07:51:15.384 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:15 smithi163 podman[73114]: 2024-05-27 07:51:15.133108171 +0000 UTC m=+0.657951814 container remove 4e04f6774a3a1923d89bb5c20563d34f2fb5a54932f7537f27b6b8aa6378e890 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-27T07:51:15.384 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:15 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:51:15.726 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:15 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-05-27T07:51:15.726 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:15 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Consumed 1.016s CPU time. 2024-05-27T07:51:15.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:15 smithi163 ceph-mon[36451]: pgmap v381: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 64 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.9 MiB/s wr, 208 op/s 2024-05-27T07:51:15.741 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:15 smithi186 ceph-mon[36647]: pgmap v381: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 64 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.9 MiB/s wr, 208 op/s 2024-05-27T07:51:15.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:15 smithi114 ceph-mon[32038]: pgmap v381: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 64 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.9 MiB/s wr, 208 op/s 2024-05-27T07:51:16.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1123926 9.88 MB/sec execute 448 sec latency 57.242 ms 2024-05-27T07:51:16.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.550+0000 7f9a3e3c2700 1 -- 172.21.15.114:0/39130924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a381034b0 msgr2=0x7f9a381038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:16.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.550+0000 7f9a3e3c2700 1 --2- 172.21.15.114:0/39130924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a381034b0 0x7f9a381038c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f9a28009920 tx=0x7f9a28009c30 comp rx=0 tx=0).stop 2024-05-27T07:51:16.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.550+0000 7f9a3e3c2700 1 -- 172.21.15.114:0/39130924 shutdown_connections 2024-05-27T07:51:16.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.550+0000 7f9a3e3c2700 1 --2- 172.21.15.114:0/39130924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a38108730 0x7f9a38110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.550+0000 7f9a3e3c2700 1 --2- 172.21.15.114:0/39130924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a381034b0 0x7f9a381038c0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.550+0000 7f9a3e3c2700 1 --2- 172.21.15.114:0/39130924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a38107de0 0x7f9a381081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.550+0000 7f9a3e3c2700 1 -- 172.21.15.114:0/39130924 >> 172.21.15.114:0/39130924 conn(0x7f9a380fede0 msgr2=0x7f9a38101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:16.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.550+0000 7f9a3e3c2700 1 -- 172.21.15.114:0/39130924 shutdown_connections 2024-05-27T07:51:16.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.550+0000 7f9a3e3c2700 1 -- 172.21.15.114:0/39130924 wait complete. 2024-05-27T07:51:16.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a3e3c2700 1 Processor -- start 2024-05-27T07:51:16.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a3e3c2700 1 -- start start 2024-05-27T07:51:16.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a3e3c2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a381034b0 0x7f9a38199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a3e3c2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a38107de0 0x7f9a3819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a3e3c2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a38108730 0x7f9a3819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a3e3c2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a38113360 con 0x7f9a38107de0 2024-05-27T07:51:16.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a3e3c2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381131e0 con 0x7f9a38108730 2024-05-27T07:51:16.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a3e3c2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381134e0 con 0x7f9a381034b0 2024-05-27T07:51:16.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a2ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a38107de0 0x7f9a3819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a2ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a38107de0 0x7f9a3819a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48858/0 (socket says 172.21.15.114:48858) 2024-05-27T07:51:16.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a2ffff700 1 -- 172.21.15.114:0/3610358187 learned_addr learned my addr 172.21.15.114:0/3610358187 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:16.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a37fff700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a381034b0 0x7f9a38199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a3c95f700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a38108730 0x7f9a3819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a2ffff700 1 -- 172.21.15.114:0/3610358187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a381034b0 msgr2=0x7f9a38199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:16.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a381034b0 0x7f9a38199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a2ffff700 1 -- 172.21.15.114:0/3610358187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a38108730 msgr2=0x7f9a3819e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:16.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a38108730 0x7f9a3819e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a2ffff700 1 -- 172.21.15.114:0/3610358187 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a3819ec30 con 0x7f9a38107de0 2024-05-27T07:51:16.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.551+0000 7f9a3c95f700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a38108730 0x7f9a3819e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:16.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.552+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a38107de0 0x7f9a3819a350 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000eb00 tx=0x7f9a2000ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:16.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.552+0000 7f9a35ffb700 1 -- 172.21.15.114:0/3610358187 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2001c070 con 0x7f9a38107de0 2024-05-27T07:51:16.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.552+0000 7f9a35ffb700 1 -- 172.21.15.114:0/3610358187 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a20017410 con 0x7f9a38107de0 2024-05-27T07:51:16.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.552+0000 7f9a3e3c2700 1 -- 172.21.15.114:0/3610358187 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a3819ef20 con 0x7f9a38107de0 2024-05-27T07:51:16.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.552+0000 7f9a35ffb700 1 -- 172.21.15.114:0/3610358187 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a20016610 con 0x7f9a38107de0 2024-05-27T07:51:16.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.552+0000 7f9a3e3c2700 1 -- 172.21.15.114:0/3610358187 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a380759e0 con 0x7f9a38107de0 2024-05-27T07:51:16.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.555+0000 7f9a35ffb700 1 -- 172.21.15.114:0/3610358187 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f9a2001f070 con 0x7f9a38107de0 2024-05-27T07:51:16.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.556+0000 7f9a35ffb700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a240b2a70 0x7f9a240b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.556+0000 7f9a35ffb700 1 -- 172.21.15.114:0/3610358187 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9566+0+0 (secure 0 0 0) 0x7f9a200e3d10 con 0x7f9a38107de0 2024-05-27T07:51:16.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.556+0000 7f9a37fff700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a240b2a70 0x7f9a240b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.556+0000 7f9a4cdd7b80 1 Processor -- start 2024-05-27T07:51:16.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.556+0000 7f9a4cdd7b80 1 -- start start 2024-05-27T07:51:16.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a37fff700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a240b2a70 0x7f9a240b4f80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9a28005f40 tx=0x7f9a28019040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:16.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a4cdd7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 0x55d71017e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a4cdd7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71017ec50 0x55d710178a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a4cdd7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d710179390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a4cdd7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d7100a7c90 con 0x55d710178f80 2024-05-27T07:51:16.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a4cdd7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d710097830 con 0x55d70fffae40 2024-05-27T07:51:16.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a4cdd7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d71009b870 con 0x55d71017ec50 2024-05-27T07:51:16.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a3c95f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d710179390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a3c95f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d710179390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48874/0 (socket says 172.21.15.114:48874) 2024-05-27T07:51:16.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a2ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71017ec50 0x55d710178a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a3c95f700 1 -- 172.21.15.114:0/1887313196 learned_addr learned my addr 172.21.15.114:0/1887313196 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a37fff700 1 --2- 172.21.15.114:0/1887313196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 0x55d71017e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a2ffff700 1 -- 172.21.15.114:0/1887313196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 msgr2=0x55d71017e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/1887313196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 0x55d71017e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a2ffff700 1 -- 172.21.15.114:0/1887313196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 msgr2=0x55d710179390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/1887313196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d710179390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a2ffff700 1 -- 172.21.15.114:0/1887313196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d70ffe78d0 con 0x55d71017ec50 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.557+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/1887313196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71017ec50 0x55d710178a40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9a200e1050 tx=0x7f9a200aa6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6902533a8b7778d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.558+0000 7f9a2e7fc700 1 -- 172.21.15.114:0/1887313196 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a200aabe0 con 0x55d71017ec50 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.558+0000 7f9a2e7fc700 1 -- 172.21.15.114:0/1887313196 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a200aef90 con 0x55d71017ec50 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.558+0000 7f9a2e7fc700 1 -- 172.21.15.114:0/1887313196 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a200ab060 con 0x55d71017ec50 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.558+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/1887313196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71017ec50 msgr2=0x55d710178a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.558+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/1887313196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71017ec50 0x55d710178a40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9a200e1050 tx=0x7f9a200aa6f0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.558+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/1887313196 shutdown_connections 2024-05-27T07:51:16.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.558+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/1887313196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d710179390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.559+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/1887313196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 0x55d71017e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.559+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/1887313196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71017ec50 0x55d710178a40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.559+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/1887313196 >> 172.21.15.114:0/1887313196 conn(0x55d710055af0 msgr2=0x55d710173640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:16.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.559+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/1887313196 shutdown_connections 2024-05-27T07:51:16.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.559+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/1887313196 wait complete. 2024-05-27T07:51:16.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.559+0000 7f9a4cdd7b80 1 -- start start 2024-05-27T07:51:16.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.559+0000 7f9a4cdd7b80 1 Processor -- start 2024-05-27T07:51:16.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.559+0000 7f9a4cdd7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 0x55d710182400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a4cdd7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d710178f80 0x55d710182940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a4cdd7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d71017ec50 0x55d710182e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a4cdd7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d70ff9dfa0 con 0x55d71017ec50 2024-05-27T07:51:16.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a4cdd7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d7100397c0 con 0x55d70fffae40 2024-05-27T07:51:16.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a4cdd7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d710185680 con 0x55d710178f80 2024-05-27T07:51:16.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a3c95f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d71017ec50 0x55d710182e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a2ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d710178f80 0x55d710182940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a3c95f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d71017ec50 0x55d710182e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48884/0 (socket says 172.21.15.114:48884) 2024-05-27T07:51:16.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a2ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d710178f80 0x55d710182940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59156/0 (socket says 172.21.15.114:59156) 2024-05-27T07:51:16.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a3c95f700 1 -- 172.21.15.114:0/1071416938 learned_addr learned my addr 172.21.15.114:0/1071416938 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:16.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a4cdd7b80 10 client.0 _ll_register_callbacks cb 0x55d710032920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:16.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a2ffff700 1 -- 172.21.15.114:0/1071416938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 msgr2=0x55d710182400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:16.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/1071416938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 0x55d710182400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a2ffff700 1 -- 172.21.15.114:0/1071416938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d71017ec50 msgr2=0x55d710182e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:16.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/1071416938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d71017ec50 0x55d710182e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a2ffff700 1 -- 172.21.15.114:0/1071416938 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d710036690 con 0x55d710178f80 2024-05-27T07:51:16.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a3c95f700 1 --2- 172.21.15.114:0/1071416938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d71017ec50 0x55d710182e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:16.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/1071416938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d710178f80 0x55d710182940 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9a200e1100 tx=0x7f9a20017c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d91c5b6744fbd19c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:16.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a177fe700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:51:16.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a177fe700 1 -- 172.21.15.114:0/1071416938 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a200c3070 con 0x55d710178f80 2024-05-27T07:51:16.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/1071416938 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a200aef90 con 0x55d710178f80 2024-05-27T07:51:16.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.560+0000 7f9a177fe700 1 -- 172.21.15.114:0/1071416938 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9566+0+0 (secure 0 0 0) 0x7f9a200a71d0 con 0x55d710178f80 2024-05-27T07:51:16.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.561+0000 7f9a177fe700 1 -- 172.21.15.114:0/1071416938 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f9a2008c050 con 0x55d710178f80 2024-05-27T07:51:16.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.561+0000 7f9a4cdd7b80 10 client.27622 fetch_fsmap learned FSMap version 64 2024-05-27T07:51:16.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.561+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/1071416938 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55d710194590 con 0x55d710178f80 2024-05-27T07:51:16.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.561+0000 7f9a177fe700 1 -- 172.21.15.114:0/1071416938 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a200aef90 con 0x55d710178f80 2024-05-27T07:51:16.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.561+0000 7f9a177fe700 1 -- 172.21.15.114:0/1071416938 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 64) ==== 5305+0+0 (secure 0 0 0) 0x7f9a200a6550 con 0x55d710178f80 2024-05-27T07:51:16.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.561+0000 7f9a4cdd7b80 10 client.27622 fetch_fsmap finished waiting for FSMap version 64 2024-05-27T07:51:16.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.561+0000 7f9a4cdd7b80 10 client.27622 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:16.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.561+0000 7f9a4cdd7b80 20 client.27622 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:16.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.561+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/1071416938 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d710196c40 0x55d7101990d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.561+0000 7f9a4cdd7b80 4 client.27622 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:16.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.561+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/1071416938 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7101449f0 con 0x55d710196c40 2024-05-27T07:51:16.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.572+0000 7f9a37fff700 1 --2- 172.21.15.114:0/1071416938 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d710196c40 0x55d7101990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.573+0000 7f9a37fff700 1 --2- 172.21.15.114:0/1071416938 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d710196c40 0x55d7101990d0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8bdf035f8d6d27b server_cookie=c3f6773444416a87 in_seq=0 out_seq=0 2024-05-27T07:51:16.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.573+0000 7f9a177fe700 10 client.27622 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:16.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a177fe700 1 -- 172.21.15.114:0/1071416938 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d7101449f0 con 0x55d710196c40 2024-05-27T07:51:16.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a177fe700 10 client.27622 handle_command_reply: tid=0 2024-05-27T07:51:16.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a4cdd7b80 1 client.27622 shutdown 2024-05-27T07:51:16.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/1071416938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d710178f80 msgr2=0x55d710182940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:16.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/1071416938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d710178f80 0x55d710182940 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9a200e1100 tx=0x7f9a20017c00 comp rx=0 tx=0).stop 2024-05-27T07:51:16.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/1071416938 shutdown_connections 2024-05-27T07:51:16.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/1071416938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d71017ec50 0x55d710182e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/1071416938 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d710196c40 0x55d7101990d0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/1071416938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 0x55d710182400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/1071416938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d710178f80 0x55d710182940 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/1071416938 >> 172.21.15.114:0/1071416938 conn(0x55d710055af0 msgr2=0x55d710177a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:16.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a177fe700 0 client.27622 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:16.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/1071416938 shutdown_connections 2024-05-27T07:51:16.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/1071416938 wait complete. 2024-05-27T07:51:16.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.574+0000 7f9a4cdd7b80 20 client.27622 trim_cache size 0 max 16384 2024-05-27T07:51:16.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.591+0000 7f9a4cdd7b80 1 Processor -- start 2024-05-27T07:51:16.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.591+0000 7f9a4cdd7b80 1 -- start start 2024-05-27T07:51:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a4cdd7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 0x55d710178a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a4cdd7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d710182cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a4cdd7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71017ec50 0x55d7101848a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a4cdd7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381af180 con 0x55d710178f80 2024-05-27T07:51:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a4cdd7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381af850 con 0x55d70fffae40 2024-05-27T07:51:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a4cdd7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381ae000 con 0x55d71017ec50 2024-05-27T07:51:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a2ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d710182cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a2ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d710182cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48894/0 (socket says 172.21.15.114:48894) 2024-05-27T07:51:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a2ffff700 1 -- 172.21.15.114:0/2333568950 learned_addr learned my addr 172.21.15.114:0/2333568950 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a3c95f700 1 --2- 172.21.15.114:0/2333568950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71017ec50 0x55d7101848a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a37fff700 1 --2- 172.21.15.114:0/2333568950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 0x55d710178a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a2ffff700 1 -- 172.21.15.114:0/2333568950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71017ec50 msgr2=0x55d7101848a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/2333568950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71017ec50 0x55d7101848a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a2ffff700 1 -- 172.21.15.114:0/2333568950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 msgr2=0x55d710178a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/2333568950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 0x55d710178a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.592+0000 7f9a2ffff700 1 -- 172.21.15.114:0/2333568950 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7101799f0 con 0x55d710178f80 2024-05-27T07:51:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.593+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/2333568950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d710182cc0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000b5a0 tx=0x7f9a200aa6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3616a62e39968fb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.593+0000 7f9a2dffb700 1 -- 172.21.15.114:0/2333568950 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a200c3070 con 0x55d710178f80 2024-05-27T07:51:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.593+0000 7f9a2dffb700 1 -- 172.21.15.114:0/2333568950 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a200c5050 con 0x55d710178f80 2024-05-27T07:51:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.593+0000 7f9a2dffb700 1 -- 172.21.15.114:0/2333568950 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a200b3a40 con 0x55d710178f80 2024-05-27T07:51:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.594+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/2333568950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 msgr2=0x55d710182cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.594+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/2333568950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d710182cc0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000b5a0 tx=0x7f9a200aa6f0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.594+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/2333568950 shutdown_connections 2024-05-27T07:51:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.594+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/2333568950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71017ec50 0x55d7101848a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.594+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/2333568950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d70fffae40 0x55d710178a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.594+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/2333568950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d710182cc0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.594+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/2333568950 >> 172.21.15.114:0/2333568950 conn(0x55d710055af0 msgr2=0x55d710180cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.594+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/2333568950 shutdown_connections 2024-05-27T07:51:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.594+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/2333568950 wait complete. 2024-05-27T07:51:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.594+0000 7f9a4cdd7b80 1 -- start start 2024-05-27T07:51:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a4cdd7b80 1 Processor -- start 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a4cdd7b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70fffae40 0x55d71018a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a4cdd7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d71018ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a4cdd7b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d71017ec50 0x55d71019d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a4cdd7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381ad930 con 0x55d710178f80 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a4cdd7b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381ac520 con 0x55d71017ec50 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a4cdd7b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381acb90 con 0x55d70fffae40 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a2ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d71018ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a4cdd7b80 10 client.0 _ll_register_callbacks cb 0x55d710032920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a2ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d71018ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48904/0 (socket says 172.21.15.114:48904) 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a2ffff700 1 -- 172.21.15.114:0/2560541837 learned_addr learned my addr 172.21.15.114:0/2560541837 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a37fff700 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70fffae40 0x55d71018a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a3c95f700 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d71017ec50 0x55d71019d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a37fff700 1 -- 172.21.15.114:0/2560541837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d71017ec50 msgr2=0x55d71019d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a37fff700 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d71017ec50 0x55d71019d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a37fff700 1 -- 172.21.15.114:0/2560541837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 msgr2=0x55d71018ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:16.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a37fff700 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d71018ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a37fff700 1 -- 172.21.15.114:0/2560541837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d71019de30 con 0x55d70fffae40 2024-05-27T07:51:16.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.595+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d71018ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:16.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.596+0000 7f9a37fff700 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70fffae40 0x55d71018a910 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9a2800c010 tx=0x7f9a28009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9948c3620b06091a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:16.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.596+0000 7f9a177fe700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:51:16.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.596+0000 7f9a177fe700 1 -- 172.21.15.114:0/2560541837 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a28005240 con 0x55d70fffae40 2024-05-27T07:51:16.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.596+0000 7f9a177fe700 1 -- 172.21.15.114:0/2560541837 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9566+0+0 (secure 0 0 0) 0x7f9a280053e0 con 0x55d70fffae40 2024-05-27T07:51:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.596+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/2560541837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a200c5050 con 0x55d70fffae40 2024-05-27T07:51:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.596+0000 7f9a177fe700 1 -- 172.21.15.114:0/2560541837 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f9a28041050 con 0x55d70fffae40 2024-05-27T07:51:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.596+0000 7f9a4cdd7b80 10 client.27628 fetch_fsmap learned FSMap version 64 2024-05-27T07:51:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.596+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/2560541837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55d71019f0d0 con 0x55d70fffae40 2024-05-27T07:51:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.596+0000 7f9a177fe700 1 -- 172.21.15.114:0/2560541837 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a200c5050 con 0x55d70fffae40 2024-05-27T07:51:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.596+0000 7f9a177fe700 1 -- 172.21.15.114:0/2560541837 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 64) ==== 5305+0+0 (secure 0 0 0) 0x7f9a2803c950 con 0x55d70fffae40 2024-05-27T07:51:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.597+0000 7f9a4cdd7b80 10 client.27628 fetch_fsmap finished waiting for FSMap version 64 2024-05-27T07:51:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.597+0000 7f9a4cdd7b80 10 client.27628 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.597+0000 7f9a4cdd7b80 20 client.27628 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.597+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d710200910 0x55d710202d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.597+0000 7f9a4cdd7b80 4 client.27628 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:51:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.597+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/2560541837 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d7100a9be0 con 0x55d710200910 2024-05-27T07:51:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.597+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d710200910 0x55d710202d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.597+0000 7f9a177fe700 10 client.27628 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.597+0000 7f9a2ffff700 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d710200910 0x55d710202d30 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78860f03f0712707 server_cookie=a1d667b57f0e8e29 in_seq=0 out_seq=0 2024-05-27T07:51:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a177fe700 1 -- 172.21.15.114:0/2560541837 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d7100a9be0 con 0x55d710200910 2024-05-27T07:51:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a177fe700 10 client.27628 handle_command_reply: tid=0 2024-05-27T07:51:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a4cdd7b80 1 client.27628 shutdown 2024-05-27T07:51:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/2560541837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70fffae40 msgr2=0x55d71018a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70fffae40 0x55d71018a910 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9a2800c010 tx=0x7f9a28009260 comp rx=0 tx=0).stop 2024-05-27T07:51:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/2560541837 shutdown_connections 2024-05-27T07:51:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d710200910 0x55d710202d30 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d71017ec50 0x55d71019d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70fffae40 0x55d71018a910 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a4cdd7b80 1 --2- 172.21.15.114:0/2560541837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d710178f80 0x55d71018ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a177fe700 0 client.27628 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/2560541837 >> 172.21.15.114:0/2560541837 conn(0x55d710055af0 msgr2=0x55d71017c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/2560541837 shutdown_connections 2024-05-27T07:51:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a4cdd7b80 1 -- 172.21.15.114:0/2560541837 wait complete. 2024-05-27T07:51:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.599+0000 7f9a4cdd7b80 20 client.27628 trim_cache size 0 max 16384 2024-05-27T07:51:16.605 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:51:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.600+0000 7f9a3e3c2700 1 -- 172.21.15.114:0/3610358187 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a240b2a70 msgr2=0x7f9a240b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.600+0000 7f9a3e3c2700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a240b2a70 0x7f9a240b4f80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9a28005f40 tx=0x7f9a28019040 comp rx=0 tx=0).stop 2024-05-27T07:51:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.600+0000 7f9a3e3c2700 1 -- 172.21.15.114:0/3610358187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a38107de0 msgr2=0x7f9a3819a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.600+0000 7f9a3e3c2700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a38107de0 0x7f9a3819a350 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000eb00 tx=0x7f9a2000ee10 comp rx=0 tx=0).stop 2024-05-27T07:51:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.600+0000 7f9a3e3c2700 1 -- 172.21.15.114:0/3610358187 shutdown_connections 2024-05-27T07:51:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.600+0000 7f9a3e3c2700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a240b2a70 0x7f9a240b4f80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.600+0000 7f9a3e3c2700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a38108730 0x7f9a3819e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.600+0000 7f9a3e3c2700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a381034b0 0x7f9a38199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.600+0000 7f9a3e3c2700 1 --2- 172.21.15.114:0/3610358187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a38107de0 0x7f9a3819a350 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.600+0000 7f9a3e3c2700 1 -- 172.21.15.114:0/3610358187 >> 172.21.15.114:0/3610358187 conn(0x7f9a380fede0 msgr2=0x7f9a38101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.600+0000 7f9a3e3c2700 1 -- 172.21.15.114:0/3610358187 shutdown_connections 2024-05-27T07:51:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:16.600+0000 7f9a3e3c2700 1 -- 172.21.15.114:0/3610358187 wait complete. 2024-05-27T07:51:16.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:16 smithi186 podman[75034]: 2024-05-27 07:51:16.697083586 +0000 UTC m=+3.325590723 container init 4a3816c2982f0bad95eb3dedd7ff1867f43bcbb4cb5a86af113f69c6bc32289a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-27T07:51:16.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:16 smithi186 podman[75034]: 2024-05-27 07:51:16.700830477 +0000 UTC m=+3.329337603 container start 4a3816c2982f0bad95eb3dedd7ff1867f43bcbb4cb5a86af113f69c6bc32289a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521) 2024-05-27T07:51:16.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:16 smithi186 ceph-mds[75119]: -- 172.21.15.186:0/3082606491 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561786c3ba40 con 0x561786c83a80 2024-05-27T07:51:16.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: starting mds.l at 2024-05-27T07:51:17.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1125178 9.87 MB/sec execute 449 sec latency 160.027 ms 2024-05-27T07:51:17.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:17 smithi114 ceph-mon[32038]: pgmap v382: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 236 op/s 2024-05-27T07:51:17.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:17 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:51:17.906 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:17 smithi186 ceph-mon[36647]: pgmap v382: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 236 op/s 2024-05-27T07:51:17.906 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:17 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:51:17.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:17 smithi163 ceph-mon[36451]: pgmap v382: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 236 op/s 2024-05-27T07:51:17.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:17 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:51:18.229 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:17 smithi186 bash[75034]: 4a3816c2982f0bad95eb3dedd7ff1867f43bcbb4cb5a86af113f69c6bc32289a 2024-05-27T07:51:18.229 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:51:17 smithi186 systemd[1]: Started Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:51:18.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1126524 9.86 MB/sec execute 450 sec latency 62.943 ms 2024-05-27T07:51:18.382 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:51:18.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:51:18.657 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:18.781 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:18 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6832/3266101988,v1:172.21.15.186:6833/3266101988] up:boot 2024-05-27T07:51:18.781 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:18 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:51:18.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:18 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:51:18.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:18 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:51:18.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:18 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:51:18.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:18 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:51:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:18 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6832/3266101988,v1:172.21.15.186:6833/3266101988] up:boot 2024-05-27T07:51:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:18 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:51:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:18 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:51:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:18 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:51:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:18 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:51:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:18 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:51:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:18 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6832/3266101988,v1:172.21.15.186:6833/3266101988] up:boot 2024-05-27T07:51:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:18 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:51:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:18 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:51:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:18 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:51:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:18 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:51:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:18 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:51:19.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1128445 9.86 MB/sec execute 451 sec latency 25.727 ms 2024-05-27T07:51:19.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:19 smithi114 ceph-mon[32038]: pgmap v383: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 242 op/s 2024-05-27T07:51:19.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:19 smithi114 ceph-mon[32038]: pgmap v384: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 241 op/s 2024-05-27T07:51:19.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:19 smithi114 ceph-mon[32038]: osdmap e123: 12 total, 12 up, 12 in 2024-05-27T07:51:19.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:19 smithi163 ceph-mon[36451]: pgmap v383: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 242 op/s 2024-05-27T07:51:19.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:19 smithi163 ceph-mon[36451]: pgmap v384: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 241 op/s 2024-05-27T07:51:19.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:19 smithi163 ceph-mon[36451]: osdmap e123: 12 total, 12 up, 12 in 2024-05-27T07:51:19.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:19 smithi186 ceph-mon[36647]: pgmap v383: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 242 op/s 2024-05-27T07:51:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:19 smithi186 ceph-mon[36647]: pgmap v384: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 241 op/s 2024-05-27T07:51:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:19 smithi186 ceph-mon[36647]: osdmap e123: 12 total, 12 up, 12 in 2024-05-27T07:51:20.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1130014 9.86 MB/sec execute 452 sec latency 35.074 ms 2024-05-27T07:51:20.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.433+0000 7efdc089a700 1 -- 172.21.15.114:0/440328849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdbc10e8d0 msgr2=0x7efdbc110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:20.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.433+0000 7efdc089a700 1 --2- 172.21.15.114:0/440328849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdbc10e8d0 0x7efdbc110db0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7efdb400b0a0 tx=0x7efdb400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.433+0000 7efdc089a700 1 -- 172.21.15.114:0/440328849 shutdown_connections 2024-05-27T07:51:20.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.433+0000 7efdc089a700 1 --2- 172.21.15.114:0/440328849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdbc10e8d0 0x7efdbc110db0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.433+0000 7efdc089a700 1 --2- 172.21.15.114:0/440328849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdbc1054f0 0x7efdbc105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.433+0000 7efdc089a700 1 --2- 172.21.15.114:0/440328849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdbc105e40 0x7efdbc10e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.433+0000 7efdc089a700 1 -- 172.21.15.114:0/440328849 >> 172.21.15.114:0/440328849 conn(0x7efdbc0fedb0 msgr2=0x7efdbc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:20.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.433+0000 7efdc089a700 1 -- 172.21.15.114:0/440328849 shutdown_connections 2024-05-27T07:51:20.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.433+0000 7efdc089a700 1 -- 172.21.15.114:0/440328849 wait complete. 2024-05-27T07:51:20.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.434+0000 7efdc089a700 1 Processor -- start 2024-05-27T07:51:20.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.434+0000 7efdc089a700 1 -- start start 2024-05-27T07:51:20.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.434+0000 7efdc089a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdbc1054f0 0x7efdbc1031e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.434+0000 7efdc089a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdbc105e40 0x7efdbc103720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.434+0000 7efdc089a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdbc10e8d0 0x7efdbc103c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.434+0000 7efdc089a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc114430 con 0x7efdbc10e8d0 2024-05-27T07:51:20.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.434+0000 7efdc089a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1142b0 con 0x7efdbc1054f0 2024-05-27T07:51:20.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.434+0000 7efdc089a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1145b0 con 0x7efdbc105e40 2024-05-27T07:51:20.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.434+0000 7efdbad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdbc10e8d0 0x7efdbc103c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdb9d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdbc105e40 0x7efdbc103720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdb9d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdbc105e40 0x7efdbc103720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59218/0 (socket says 172.21.15.114:59218) 2024-05-27T07:51:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdbad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdbc10e8d0 0x7efdbc103c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48920/0 (socket says 172.21.15.114:48920) 2024-05-27T07:51:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdb9d9b700 1 -- 172.21.15.114:0/2658599345 learned_addr learned my addr 172.21.15.114:0/2658599345 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdba59c700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdbc1054f0 0x7efdbc1031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdb9d9b700 1 -- 172.21.15.114:0/2658599345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdbc1054f0 msgr2=0x7efdbc1031e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdb9d9b700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdbc1054f0 0x7efdbc1031e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdb9d9b700 1 -- 172.21.15.114:0/2658599345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdbc10e8d0 msgr2=0x7efdbc103c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdb9d9b700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdbc10e8d0 0x7efdbc103c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdb9d9b700 1 -- 172.21.15.114:0/2658599345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdbc1a9030 con 0x7efdbc105e40 2024-05-27T07:51:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdbad9d700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdbc10e8d0 0x7efdbc103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdb9d9b700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdbc105e40 0x7efdbc103720 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efdb000e9a0 tx=0x7efdb000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdab7fe700 1 -- 172.21.15.114:0/2658599345 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb000ca10 con 0x7efdbc105e40 2024-05-27T07:51:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdc089a700 1 -- 172.21.15.114:0/2658599345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdbc1a9320 con 0x7efdbc105e40 2024-05-27T07:51:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdab7fe700 1 -- 172.21.15.114:0/2658599345 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdb000cbb0 con 0x7efdbc105e40 2024-05-27T07:51:20.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.435+0000 7efdab7fe700 1 -- 172.21.15.114:0/2658599345 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb0018680 con 0x7efdbc105e40 2024-05-27T07:51:20.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.436+0000 7efdc089a700 1 -- 172.21.15.114:0/2658599345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdbc1a98b0 con 0x7efdbc105e40 2024-05-27T07:51:20.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.438+0000 7efdab7fe700 1 -- 172.21.15.114:0/2658599345 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7efdb0018820 con 0x7efdbc105e40 2024-05-27T07:51:20.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.439+0000 7efdab7fe700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efda40b29f0 0x7efda40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.439+0000 7efdcf2afb80 1 Processor -- start 2024-05-27T07:51:20.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.440+0000 7efdba59c700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efda40b29f0 0x7efda40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.440+0000 7efdcf2afb80 1 -- start start 2024-05-27T07:51:20.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.440+0000 7efdab7fe700 1 -- 172.21.15.114:0/2658599345 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9538+0+0 (secure 0 0 0) 0x7efdb00e2bf0 con 0x7efdbc105e40 2024-05-27T07:51:20.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.440+0000 7efdcf2afb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e866aec0 0x5652e87ee790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.440+0000 7efdcf2afb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e87eecd0 0x5652e87e8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.440+0000 7efdcf2afb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87e9000 0x5652e87e9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.440+0000 7efdcf2afb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5652e8717d10 con 0x5652e866aec0 2024-05-27T07:51:20.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.440+0000 7efdcf2afb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5652e87078b0 con 0x5652e87eecd0 2024-05-27T07:51:20.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.440+0000 7efdcf2afb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5652e870b8f0 con 0x5652e87e9000 2024-05-27T07:51:20.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.440+0000 7efdbad9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87e9000 0x5652e87e9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.440+0000 7efdbad9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87e9000 0x5652e87e9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59234/0 (socket says 172.21.15.114:59234) 2024-05-27T07:51:20.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.440+0000 7efdbad9d700 1 -- 172.21.15.114:0/2189384438 learned_addr learned my addr 172.21.15.114:0/2189384438 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:20.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.441+0000 7efdba59c700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efda40b29f0 0x7efda40b4f00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efdac00ac50 tx=0x7efdac00a380 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:20.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.441+0000 7efdba59c700 1 --2- 172.21.15.114:0/2189384438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e866aec0 0x5652e87ee790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.441+0000 7efdb9d9b700 1 --2- 172.21.15.114:0/2189384438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e87eecd0 0x5652e87e8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.441+0000 7efdbad9d700 1 -- 172.21.15.114:0/2189384438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e87eecd0 msgr2=0x5652e87e8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:20.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.441+0000 7efdbad9d700 1 --2- 172.21.15.114:0/2189384438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e87eecd0 0x5652e87e8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.441+0000 7efdbad9d700 1 -- 172.21.15.114:0/2189384438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e866aec0 msgr2=0x5652e87ee790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:20.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.441+0000 7efdbad9d700 1 --2- 172.21.15.114:0/2189384438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e866aec0 0x5652e87ee790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.441+0000 7efdbad9d700 1 -- 172.21.15.114:0/2189384438 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652e8657950 con 0x5652e87e9000 2024-05-27T07:51:20.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.441+0000 7efdba59c700 1 --2- 172.21.15.114:0/2189384438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e866aec0 0x5652e87ee790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:20.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.441+0000 7efdbad9d700 1 --2- 172.21.15.114:0/2189384438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87e9000 0x5652e87e9410 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efdb4015000 tx=0x7efdb400bf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b64981d6ceb35552 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:20.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.442+0000 7efda8ff9700 1 -- 172.21.15.114:0/2189384438 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb4021070 con 0x5652e87e9000 2024-05-27T07:51:20.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.442+0000 7efda8ff9700 1 -- 172.21.15.114:0/2189384438 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdb4004c50 con 0x5652e87e9000 2024-05-27T07:51:20.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.442+0000 7efda8ff9700 1 -- 172.21.15.114:0/2189384438 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb40053c0 con 0x5652e87e9000 2024-05-27T07:51:20.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.443+0000 7efdcf2afb80 1 -- 172.21.15.114:0/2189384438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87e9000 msgr2=0x5652e87e9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:20.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.443+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/2189384438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87e9000 0x5652e87e9410 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efdb4015000 tx=0x7efdb400bf00 comp rx=0 tx=0).stop 2024-05-27T07:51:20.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.444+0000 7efdcf2afb80 1 -- 172.21.15.114:0/2189384438 shutdown_connections 2024-05-27T07:51:20.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.444+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/2189384438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87e9000 0x5652e87e9410 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.444+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/2189384438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e866aec0 0x5652e87ee790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.444+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/2189384438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e87eecd0 0x5652e87e8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.444+0000 7efdcf2afb80 1 -- 172.21.15.114:0/2189384438 >> 172.21.15.114:0/2189384438 conn(0x5652e86c5b70 msgr2=0x5652e87e36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:20.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.444+0000 7efdcf2afb80 1 -- 172.21.15.114:0/2189384438 shutdown_connections 2024-05-27T07:51:20.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.444+0000 7efdcf2afb80 1 -- 172.21.15.114:0/2189384438 wait complete. 2024-05-27T07:51:20.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.444+0000 7efdcf2afb80 1 -- start start 2024-05-27T07:51:20.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.445+0000 7efdcf2afb80 1 Processor -- start 2024-05-27T07:51:20.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.445+0000 7efdcf2afb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e866aec0 0x5652e87f2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.445+0000 7efdcf2afb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.445+0000 7efdcf2afb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87eecd0 0x5652e87f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.445+0000 7efdcf2afb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5652e860e020 con 0x5652e87e9000 2024-05-27T07:51:20.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.445+0000 7efdcf2afb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5652e86a9840 con 0x5652e866aec0 2024-05-27T07:51:20.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.445+0000 7efdcf2afb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5652e87f5760 con 0x5652e87eecd0 2024-05-27T07:51:20.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.445+0000 7efdcf2afb80 10 client.0 _ll_register_callbacks cb 0x5652e86a29a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:20.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.445+0000 7efdb9d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.445+0000 7efdb9d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87f28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48942/0 (socket says 172.21.15.114:48942) 2024-05-27T07:51:20.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.445+0000 7efdb9d9b700 1 -- 172.21.15.114:0/3042910852 learned_addr learned my addr 172.21.15.114:0/3042910852 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:20.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.445+0000 7efdbad9d700 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87eecd0 0x5652e87f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.445+0000 7efdba59c700 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e866aec0 0x5652e87f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.446+0000 7efdb9d9b700 1 -- 172.21.15.114:0/3042910852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87eecd0 msgr2=0x5652e87f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.446+0000 7efdb9d9b700 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87eecd0 0x5652e87f2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.446+0000 7efdb9d9b700 1 -- 172.21.15.114:0/3042910852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e866aec0 msgr2=0x5652e87f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.446+0000 7efdb9d9b700 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e866aec0 0x5652e87f2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.446+0000 7efdb9d9b700 1 -- 172.21.15.114:0/3042910852 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652e86a6710 con 0x5652e87e9000 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.446+0000 7efdbad9d700 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87eecd0 0x5652e87f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.446+0000 7efdb9d9b700 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87f28d0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7efdb00a6d40 tx=0x7efdb00a96e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4829262dea6fe3ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.446+0000 7efda1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.446+0000 7efda1ffb700 1 -- 172.21.15.114:0/3042910852 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb00ade80 con 0x5652e87e9000 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.446+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3042910852 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdb4004c50 con 0x5652e87e9000 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.446+0000 7efda1ffb700 1 -- 172.21.15.114:0/3042910852 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9538+0+0 (secure 0 0 0) 0x7efdb00a60c0 con 0x5652e87e9000 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.446+0000 7efda1ffb700 1 -- 172.21.15.114:0/3042910852 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7efdb00b2930 con 0x5652e87e9000 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.447+0000 7efdcf2afb80 10 client.18084 fetch_fsmap learned FSMap version 65 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.447+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3042910852 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x5652e87fffd0 con 0x5652e87e9000 2024-05-27T07:51:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.447+0000 7efda1ffb700 1 -- 172.21.15.114:0/3042910852 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdb4004c50 con 0x5652e87e9000 2024-05-27T07:51:20.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.447+0000 7efda1ffb700 1 -- 172.21.15.114:0/3042910852 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 65) ==== 5910+0+0 (secure 0 0 0) 0x7efdb006ac70 con 0x5652e87e9000 2024-05-27T07:51:20.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.447+0000 7efdcf2afb80 10 client.18084 fetch_fsmap finished waiting for FSMap version 65 2024-05-27T07:51:20.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.447+0000 7efdcf2afb80 10 client.18084 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:20.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.447+0000 7efdcf2afb80 20 client.18084 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:20.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.447+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5652e8802840 0x5652e8804c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.447+0000 7efdcf2afb80 4 client.18084 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:20.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.447+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3042910852 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652e87b4a70 con 0x5652e8802840 2024-05-27T07:51:20.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.447+0000 7efdba59c700 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5652e8802840 0x5652e8804c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.448+0000 7efda1ffb700 10 client.18084 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:20.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.448+0000 7efdba59c700 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5652e8802840 0x5652e8804c90 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fdd1d3f5be35bc9 server_cookie=6c521718d18e2f95 in_seq=0 out_seq=0 2024-05-27T07:51:20.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efda1ffb700 1 -- 172.21.15.114:0/3042910852 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5652e87b4a70 con 0x5652e8802840 2024-05-27T07:51:20.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efda1ffb700 10 client.18084 handle_command_reply: tid=0 2024-05-27T07:51:20.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efdcf2afb80 1 client.18084 shutdown 2024-05-27T07:51:20.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3042910852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 msgr2=0x5652e87f28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:20.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87f28d0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7efdb00a6d40 tx=0x7efdb00a96e0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3042910852 shutdown_connections 2024-05-27T07:51:20.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87eecd0 0x5652e87f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5652e8802840 0x5652e8804c90 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e866aec0 0x5652e87f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3042910852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87f28d0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3042910852 >> 172.21.15.114:0/3042910852 conn(0x5652e86c5b70 msgr2=0x5652e87e7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:20.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efda1ffb700 0 client.18084 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:20.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3042910852 shutdown_connections 2024-05-27T07:51:20.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.449+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3042910852 wait complete. 2024-05-27T07:51:20.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.450+0000 7efdcf2afb80 20 client.18084 trim_cache size 0 max 16384 2024-05-27T07:51:20.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.478+0000 7efdcf2afb80 1 Processor -- start 2024-05-27T07:51:20.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.478+0000 7efdcf2afb80 1 -- start start 2024-05-27T07:51:20.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.478+0000 7efdcf2afb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e866aec0 0x5652e87f4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.478+0000 7efdcf2afb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87fe800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.478+0000 7efdcf2afb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e87eecd0 0x5652e87edae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.478+0000 7efdcf2afb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1b3fd0 con 0x5652e87e9000 2024-05-27T07:51:20.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.478+0000 7efdcf2afb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1b2740 con 0x5652e87eecd0 2024-05-27T07:51:20.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.478+0000 7efdcf2afb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1b05b0 con 0x5652e866aec0 2024-05-27T07:51:20.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.478+0000 7efdb9d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87fe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.478+0000 7efdb9d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87fe800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48954/0 (socket says 172.21.15.114:48954) 2024-05-27T07:51:20.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.478+0000 7efdb9d9b700 1 -- 172.21.15.114:0/3107378376 learned_addr learned my addr 172.21.15.114:0/3107378376 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:20.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.478+0000 7efdba59c700 1 --2- 172.21.15.114:0/3107378376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e866aec0 0x5652e87f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.478+0000 7efdbad9d700 1 --2- 172.21.15.114:0/3107378376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e87eecd0 0x5652e87edae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.479+0000 7efdba59c700 1 -- 172.21.15.114:0/3107378376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e87eecd0 msgr2=0x5652e87edae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:20.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.479+0000 7efdba59c700 1 --2- 172.21.15.114:0/3107378376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e87eecd0 0x5652e87edae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.479+0000 7efdba59c700 1 -- 172.21.15.114:0/3107378376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 msgr2=0x5652e87fe800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:20.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.479+0000 7efdba59c700 1 --2- 172.21.15.114:0/3107378376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87fe800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.479+0000 7efdba59c700 1 -- 172.21.15.114:0/3107378376 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652e87f0900 con 0x5652e866aec0 2024-05-27T07:51:20.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.479+0000 7efdba59c700 1 --2- 172.21.15.114:0/3107378376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e866aec0 0x5652e87f4470 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x5652e88053b0 tx=0x7efdac012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de2f22c3586d8b68 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:20.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.479+0000 7efda3fff700 1 -- 172.21.15.114:0/3107378376 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac016070 con 0x5652e866aec0 2024-05-27T07:51:20.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.479+0000 7efda3fff700 1 -- 172.21.15.114:0/3107378376 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdac014070 con 0x5652e866aec0 2024-05-27T07:51:20.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.479+0000 7efda3fff700 1 -- 172.21.15.114:0/3107378376 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac010690 con 0x5652e866aec0 2024-05-27T07:51:20.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.480+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3107378376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e866aec0 msgr2=0x5652e87f4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:20.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.480+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3107378376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e866aec0 0x5652e87f4470 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x5652e88053b0 tx=0x7efdac012000 comp rx=0 tx=0).stop 2024-05-27T07:51:20.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.480+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3107378376 shutdown_connections 2024-05-27T07:51:20.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.480+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3107378376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e87eecd0 0x5652e87edae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.480+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3107378376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e866aec0 0x5652e87f4470 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.481+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3107378376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87fe800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.481+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3107378376 >> 172.21.15.114:0/3107378376 conn(0x5652e86c5b70 msgr2=0x5652e87ed3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:20.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.481+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3107378376 shutdown_connections 2024-05-27T07:51:20.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.481+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3107378376 wait complete. 2024-05-27T07:51:20.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.481+0000 7efdcf2afb80 1 -- start start 2024-05-27T07:51:20.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.481+0000 7efdcf2afb80 1 Processor -- start 2024-05-27T07:51:20.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.481+0000 7efdcf2afb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e866aec0 0x5652e87fadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.481+0000 7efdcf2afb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.481+0000 7efdcf2afb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87eecd0 0x5652e87f7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdcf2afb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1b0c00 con 0x5652e87e9000 2024-05-27T07:51:20.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdcf2afb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1b2070 con 0x5652e866aec0 2024-05-27T07:51:20.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdcf2afb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1b12d0 con 0x5652e87eecd0 2024-05-27T07:51:20.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdcf2afb80 10 client.0 _ll_register_callbacks cb 0x5652e86a29a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:20.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdb9d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdbad9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87eecd0 0x5652e87f7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdb9d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87f79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48962/0 (socket says 172.21.15.114:48962) 2024-05-27T07:51:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdba59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e866aec0 0x5652e87fadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdb9d9b700 1 -- 172.21.15.114:0/3629509037 learned_addr learned my addr 172.21.15.114:0/3629509037 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdb9d9b700 1 -- 172.21.15.114:0/3629509037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87eecd0 msgr2=0x5652e87f7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdb9d9b700 1 --2- 172.21.15.114:0/3629509037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87eecd0 0x5652e87f7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdb9d9b700 1 -- 172.21.15.114:0/3629509037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e866aec0 msgr2=0x5652e87fadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdb9d9b700 1 --2- 172.21.15.114:0/3629509037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e866aec0 0x5652e87fadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdb9d9b700 1 -- 172.21.15.114:0/3629509037 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652e87f8440 con 0x5652e87e9000 2024-05-27T07:51:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdbad9d700 1 --2- 172.21.15.114:0/3629509037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87eecd0 0x5652e87f7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdb9d9b700 1 --2- 172.21.15.114:0/3629509037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87f79c0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7efdb00a6d40 tx=0x7efdb0009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae7aa32d311b62e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efda1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efda1ffb700 1 -- 172.21.15.114:0/3629509037 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb00ade80 con 0x5652e87e9000 2024-05-27T07:51:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efda1ffb700 1 -- 172.21.15.114:0/3629509037 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9538+0+0 (secure 0 0 0) 0x7efdb00a60c0 con 0x5652e87e9000 2024-05-27T07:51:20.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.482+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3629509037 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdac014070 con 0x5652e87e9000 2024-05-27T07:51:20.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.483+0000 7efda1ffb700 1 -- 172.21.15.114:0/3629509037 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7efdb00a9420 con 0x5652e87e9000 2024-05-27T07:51:20.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.483+0000 7efdcf2afb80 10 client.18090 fetch_fsmap learned FSMap version 65 2024-05-27T07:51:20.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.483+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3629509037 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x5652e886df70 con 0x5652e87e9000 2024-05-27T07:51:20.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.483+0000 7efda1ffb700 1 -- 172.21.15.114:0/3629509037 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdac014070 con 0x5652e87e9000 2024-05-27T07:51:20.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.483+0000 7efda1ffb700 1 -- 172.21.15.114:0/3629509037 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 65) ==== 5910+0+0 (secure 0 0 0) 0x7efdb00a54d0 con 0x5652e87e9000 2024-05-27T07:51:20.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.483+0000 7efdcf2afb80 10 client.18090 fetch_fsmap finished waiting for FSMap version 65 2024-05-27T07:51:20.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.483+0000 7efdcf2afb80 10 client.18090 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:20.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.483+0000 7efdcf2afb80 20 client.18090 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:20.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.483+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3629509037 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5652e886e600 0x5652e8870a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:20.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.483+0000 7efdcf2afb80 4 client.18090 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:51:20.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.483+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3629509037 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5652e8719c60 con 0x5652e886e600 2024-05-27T07:51:20.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.483+0000 7efdba59c700 1 --2- 172.21.15.114:0/3629509037 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5652e886e600 0x5652e8870a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:20.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.484+0000 7efda1ffb700 10 client.18090 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:20.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.484+0000 7efdba59c700 1 --2- 172.21.15.114:0/3629509037 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5652e886e600 0x5652e8870a20 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7cb1f31f1b15a5a server_cookie=df9d9140c4e3b5a7 in_seq=0 out_seq=0 2024-05-27T07:51:20.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.485+0000 7efda1ffb700 1 -- 172.21.15.114:0/3629509037 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5652e8719c60 con 0x5652e886e600 2024-05-27T07:51:20.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.485+0000 7efda1ffb700 10 client.18090 handle_command_reply: tid=0 2024-05-27T07:51:20.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.485+0000 7efdcf2afb80 1 client.18090 shutdown 2024-05-27T07:51:20.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.485+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3629509037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 msgr2=0x5652e87f79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:20.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.485+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3629509037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87f79c0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7efdb00a6d40 tx=0x7efdb0009790 comp rx=0 tx=0).stop 2024-05-27T07:51:20.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.485+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3629509037 shutdown_connections 2024-05-27T07:51:20.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.485+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3629509037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652e87eecd0 0x5652e87f7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.485+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3629509037 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5652e886e600 0x5652e8870a20 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.485+0000 7efda1ffb700 0 client.18090 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:20.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.485+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3629509037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5652e866aec0 0x5652e87fadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.485+0000 7efdcf2afb80 1 --2- 172.21.15.114:0/3629509037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652e87e9000 0x5652e87f79c0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.485+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3629509037 >> 172.21.15.114:0/3629509037 conn(0x5652e86c5b70 msgr2=0x5652e87edb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:20.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.486+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3629509037 shutdown_connections 2024-05-27T07:51:20.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.486+0000 7efdcf2afb80 1 -- 172.21.15.114:0/3629509037 wait complete. 2024-05-27T07:51:20.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.486+0000 7efdcf2afb80 20 client.18090 trim_cache size 0 max 16384 2024-05-27T07:51:20.489 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:51:20.489 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:51:20.489 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "b9c89c8e-2177-4a87-a7af-1ef7b96a19cf", 2024-05-27T07:51:20.489 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:51:20.490 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:51:20.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.487+0000 7efdc089a700 1 -- 172.21.15.114:0/2658599345 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efda40b29f0 msgr2=0x7efda40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:20.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.487+0000 7efdc089a700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efda40b29f0 0x7efda40b4f00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efdac00ac50 tx=0x7efdac00a380 comp rx=0 tx=0).stop 2024-05-27T07:51:20.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.487+0000 7efdc089a700 1 -- 172.21.15.114:0/2658599345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdbc105e40 msgr2=0x7efdbc103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:20.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.487+0000 7efdc089a700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdbc105e40 0x7efdbc103720 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efdb000e9a0 tx=0x7efdb000ecb0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.487+0000 7efdc089a700 1 -- 172.21.15.114:0/2658599345 shutdown_connections 2024-05-27T07:51:20.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.487+0000 7efdc089a700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdbc10e8d0 0x7efdbc103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.487+0000 7efdc089a700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efda40b29f0 0x7efda40b4f00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.487+0000 7efdc089a700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdbc1054f0 0x7efdbc1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.487+0000 7efdc089a700 1 --2- 172.21.15.114:0/2658599345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdbc105e40 0x7efdbc103720 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:20.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.487+0000 7efdc089a700 1 -- 172.21.15.114:0/2658599345 >> 172.21.15.114:0/2658599345 conn(0x7efdbc0fedb0 msgr2=0x7efdbc0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:20.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.487+0000 7efdc089a700 1 -- 172.21.15.114:0/2658599345 shutdown_connections 2024-05-27T07:51:20.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:20.488+0000 7efdc089a700 1 -- 172.21.15.114:0/2658599345 wait complete. 2024-05-27T07:51:21.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1132041 9.86 MB/sec execute 453 sec latency 15.975 ms 2024-05-27T07:51:21.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:51:21.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:21 smithi114 ceph-mon[32038]: pgmap v386: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 185 op/s 2024-05-27T07:51:21.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:21 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:51:21.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:21 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:21.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:21 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:51:21.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:21 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:21.878 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:21.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.942+0000 7f934f53f640 1 -- 172.21.15.114:0/933825292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9348108b90 msgr2=0x7f9348102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:21.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.942+0000 7f934f53f640 1 --2- 172.21.15.114:0/933825292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9348108b90 0x7f9348102730 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f9344009900 tx=0x7f934402a410 comp rx=0 tx=0).stop 2024-05-27T07:51:21.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.942+0000 7f934f53f640 1 -- 172.21.15.114:0/933825292 shutdown_connections 2024-05-27T07:51:21.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.942+0000 7f934f53f640 1 --2- 172.21.15.114:0/933825292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93481032e0 0x7f934810f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.942+0000 7f934f53f640 1 --2- 172.21.15.114:0/933825292 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9348109030 0x7f9348102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.942+0000 7f934f53f640 1 --2- 172.21.15.114:0/933825292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9348108b90 0x7f9348102730 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.942+0000 7f934f53f640 1 -- 172.21.15.114:0/933825292 >> 172.21.15.114:0/933825292 conn(0x7f93480782d0 msgr2=0x7f9348078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:21.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.942+0000 7f934f53f640 1 -- 172.21.15.114:0/933825292 shutdown_connections 2024-05-27T07:51:21.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.942+0000 7f934f53f640 1 -- 172.21.15.114:0/933825292 wait complete. 2024-05-27T07:51:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.943+0000 7f934f53f640 1 Processor -- start 2024-05-27T07:51:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.943+0000 7f934f53f640 1 -- start start 2024-05-27T07:51:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.943+0000 7f934f53f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93481032e0 0x7f934819f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.943+0000 7f934f53f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9348108b90 0x7f934819fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.943+0000 7f934f53f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9348109030 0x7f93481996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.943+0000 7f934f53f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9348112de0 con 0x7f9348109030 2024-05-27T07:51:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.943+0000 7f934f53f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9348112c60 con 0x7f9348108b90 2024-05-27T07:51:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.943+0000 7f934f53f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9348112f60 con 0x7f93481032e0 2024-05-27T07:51:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f934dadc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9348109030 0x7f93481996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f934dadc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9348109030 0x7f93481996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48982/0 (socket says 172.21.15.114:48982) 2024-05-27T07:51:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f934dadc640 1 -- 172.21.15.114:0/2842994430 learned_addr learned my addr 172.21.15.114:0/2842994430 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f934d2db640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93481032e0 0x7f934819f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f934cada640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9348108b90 0x7f934819fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f934dadc640 1 -- 172.21.15.114:0/2842994430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93481032e0 msgr2=0x7f934819f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f934dadc640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93481032e0 0x7f934819f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f934dadc640 1 -- 172.21.15.114:0/2842994430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9348108b90 msgr2=0x7f934819fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f934dadc640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9348108b90 0x7f934819fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f934dadc640 1 -- 172.21.15.114:0/2842994430 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9348199d00 con 0x7f9348109030 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f934d2db640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93481032e0 0x7f934819f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f934dadc640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9348109030 0x7f93481996c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f934000cce0 tx=0x7f934000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f933e7fc640 1 -- 172.21.15.114:0/2842994430 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934000f450 con 0x7f9348109030 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f933e7fc640 1 -- 172.21.15.114:0/2842994430 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f934000faa0 con 0x7f9348109030 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f934f53f640 1 -- 172.21.15.114:0/2842994430 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9348199ff0 con 0x7f9348109030 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.944+0000 7f933e7fc640 1 -- 172.21.15.114:0/2842994430 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340018b70 con 0x7f9348109030 2024-05-27T07:51:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.945+0000 7f934f53f640 1 -- 172.21.15.114:0/2842994430 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f934819a4f0 con 0x7f9348109030 2024-05-27T07:51:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.947+0000 7f93563f8740 1 Processor -- start 2024-05-27T07:51:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.947+0000 7f933e7fc640 1 -- 172.21.15.114:0/2842994430 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f9340018d10 con 0x7f9348109030 2024-05-27T07:51:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f93563f8740 1 -- start start 2024-05-27T07:51:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f93563f8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9771b0 0x5609bed18220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f93563f8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 0x5609bed18760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f93563f8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 0x5609bed18ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f93563f8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5609bec8ac30 con 0x5609be9564d0 2024-05-27T07:51:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f93563f8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5609bec7a7d0 con 0x5609be94e2f0 2024-05-27T07:51:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f93563f8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5609bec7e810 con 0x5609be9771b0 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f934cada640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 0x5609bed18ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f934cada640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 0x5609bed18ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48986/0 (socket says 172.21.15.114:48986) 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f934cada640 1 -- 172.21.15.114:0/2999905872 learned_addr learned my addr 172.21.15.114:0/2999905872 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f934d2db640 1 --2- 172.21.15.114:0/2999905872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 0x5609bed18760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f934cada640 1 --2- 172.21.15.114:0/2999905872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9771b0 0x5609bed18220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f933e7fc640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f93340b2950 0x7f93340b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f934dadc640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f93340b2950 0x7f93340b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f934cada640 1 -- 172.21.15.114:0/2999905872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9771b0 msgr2=0x5609bed18220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f934cada640 1 --2- 172.21.15.114:0/2999905872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9771b0 0x5609bed18220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f934cada640 1 -- 172.21.15.114:0/2999905872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 msgr2=0x5609bed18760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f934cada640 1 --2- 172.21.15.114:0/2999905872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 0x5609bed18760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f934cada640 1 -- 172.21.15.114:0/2999905872 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609be98f210 con 0x5609be9564d0 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.948+0000 7f933e7fc640 1 -- 172.21.15.114:0/2842994430 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9538+0+0 (secure 0 0 0) 0x7f93400e2ef0 con 0x7f9348109030 2024-05-27T07:51:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.949+0000 7f934cada640 1 --2- 172.21.15.114:0/2999905872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 0x5609bed18ca0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f9338016470 tx=0x7f933800a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6a69c5e8e543e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.949+0000 7f93337fe640 1 -- 172.21.15.114:0/2999905872 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9338016df0 con 0x5609be9564d0 2024-05-27T07:51:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.949+0000 7f934dadc640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f93340b2950 0x7f93340b4e60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f93400a9d60 tx=0x7f934000af30 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.949+0000 7f93337fe640 1 -- 172.21.15.114:0/2999905872 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93380188e0 con 0x5609be9564d0 2024-05-27T07:51:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.949+0000 7f93337fe640 1 -- 172.21.15.114:0/2999905872 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9338017800 con 0x5609be9564d0 2024-05-27T07:51:21.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.950+0000 7f93563f8740 1 -- 172.21.15.114:0/2999905872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 msgr2=0x5609bed18ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:21.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.950+0000 7f93563f8740 1 --2- 172.21.15.114:0/2999905872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 0x5609bed18ca0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f9338016470 tx=0x7f933800a230 comp rx=0 tx=0).stop 2024-05-27T07:51:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.950+0000 7f93563f8740 1 -- 172.21.15.114:0/2999905872 shutdown_connections 2024-05-27T07:51:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.950+0000 7f93563f8740 1 --2- 172.21.15.114:0/2999905872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 0x5609bed18ca0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.950+0000 7f93563f8740 1 --2- 172.21.15.114:0/2999905872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 0x5609bed18760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.950+0000 7f93563f8740 1 --2- 172.21.15.114:0/2999905872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9771b0 0x5609bed18220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.950+0000 7f93563f8740 1 -- 172.21.15.114:0/2999905872 >> 172.21.15.114:0/2999905872 conn(0x5609be96b300 msgr2=0x5609bed14b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.950+0000 7f93563f8740 1 -- 172.21.15.114:0/2999905872 shutdown_connections 2024-05-27T07:51:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.950+0000 7f93563f8740 1 -- 172.21.15.114:0/2999905872 wait complete. 2024-05-27T07:51:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.950+0000 7f93563f8740 1 -- start start 2024-05-27T07:51:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.950+0000 7f93563f8740 1 Processor -- start 2024-05-27T07:51:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f93563f8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 0x5609bed26640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f93563f8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9564d0 0x5609bed26b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f93563f8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9771b0 0x5609bed24b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f93563f8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5609be960790 con 0x5609be9771b0 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f93563f8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5609bec36890 con 0x5609be94e2f0 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f93563f8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5609bed27710 con 0x5609be9564d0 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f93563f8740 10 client.0 _ll_register_callbacks cb 0x5609be98ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f934cada640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9771b0 0x5609bed24b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f934d2db640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9564d0 0x5609bed26b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f934cada640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9771b0 0x5609bed24b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48998/0 (socket says 172.21.15.114:48998) 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f934cada640 1 -- 172.21.15.114:0/835065344 learned_addr learned my addr 172.21.15.114:0/835065344 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f934cada640 1 --2- 172.21.15.114:0/835065344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 0x5609bed26640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f934cada640 1 -- 172.21.15.114:0/835065344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9564d0 msgr2=0x5609bed26b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f934cada640 1 --2- 172.21.15.114:0/835065344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9564d0 0x5609bed26b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f934cada640 1 -- 172.21.15.114:0/835065344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 msgr2=0x5609bed26640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f934cada640 1 --2- 172.21.15.114:0/835065344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 0x5609bed26640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f934cada640 1 -- 172.21.15.114:0/835065344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609bea5b620 con 0x5609be9771b0 2024-05-27T07:51:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f934d2db640 1 --2- 172.21.15.114:0/835065344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9564d0 0x5609bed26b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f934cada640 1 --2- 172.21.15.114:0/835065344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9771b0 0x5609bed24b30 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x5609bed24390 tx=0x7f9338006f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424bab1ce2705656 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.951+0000 7f9330ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f9330ff9640 1 -- 172.21.15.114:0/835065344 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933802a060 con 0x5609be9771b0 2024-05-27T07:51:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f9330ff9640 1 -- 172.21.15.114:0/835065344 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9538+0+0 (secure 0 0 0) 0x7f93380175a0 con 0x5609be9771b0 2024-05-27T07:51:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f93563f8740 1 -- 172.21.15.114:0/835065344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93380188e0 con 0x5609be9771b0 2024-05-27T07:51:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f9330ff9640 1 -- 172.21.15.114:0/835065344 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f9338031050 con 0x5609be9771b0 2024-05-27T07:51:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f93563f8740 10 client.18102 fetch_fsmap learned FSMap version 65 2024-05-27T07:51:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f93563f8740 1 -- 172.21.15.114:0/835065344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x5609bed33a10 con 0x5609be9771b0 2024-05-27T07:51:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f9330ff9640 1 -- 172.21.15.114:0/835065344 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93380188e0 con 0x5609be9771b0 2024-05-27T07:51:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f9330ff9640 1 -- 172.21.15.114:0/835065344 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 65) ==== 5910+0+0 (secure 0 0 0) 0x7f9338012050 con 0x5609be9771b0 2024-05-27T07:51:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f93563f8740 10 client.18102 fetch_fsmap finished waiting for FSMap version 65 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f93563f8740 10 client.18102 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f93563f8740 20 client.18102 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f93563f8740 1 --2- 172.21.15.114:0/835065344 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609be935d80 0x5609bed370f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f93563f8740 4 client.18102 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f93563f8740 1 -- 172.21.15.114:0/835065344 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609bec8cb80 con 0x5609be935d80 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.952+0000 7f934d2db640 1 --2- 172.21.15.114:0/835065344 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609be935d80 0x5609bed370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.953+0000 7f9330ff9640 10 client.18102 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.953+0000 7f934d2db640 1 --2- 172.21.15.114:0/835065344 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609be935d80 0x5609bed370f0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26a3bf3ff633b36f server_cookie=ac2c3a82e6bcb0aa in_seq=0 out_seq=0 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.953+0000 7f9330ff9640 1 -- 172.21.15.114:0/835065344 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5609bec8cb80 con 0x5609be935d80 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.953+0000 7f9330ff9640 10 client.18102 handle_command_reply: tid=0 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.954+0000 7f93563f8740 1 client.18102 shutdown 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.954+0000 7f93563f8740 1 -- 172.21.15.114:0/835065344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9771b0 msgr2=0x5609bed24b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.954+0000 7f93563f8740 1 --2- 172.21.15.114:0/835065344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9771b0 0x5609bed24b30 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x5609bed24390 tx=0x7f9338006f10 comp rx=0 tx=0).stop 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.954+0000 7f93563f8740 1 -- 172.21.15.114:0/835065344 shutdown_connections 2024-05-27T07:51:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.954+0000 7f93563f8740 1 --2- 172.21.15.114:0/835065344 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609be935d80 0x5609bed370f0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.954+0000 7f93563f8740 1 --2- 172.21.15.114:0/835065344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9771b0 0x5609bed24b30 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.954+0000 7f93563f8740 1 --2- 172.21.15.114:0/835065344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9564d0 0x5609bed26b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.954+0000 7f9330ff9640 0 client.18102 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.954+0000 7f93563f8740 1 --2- 172.21.15.114:0/835065344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 0x5609bed26640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.954+0000 7f93563f8740 1 -- 172.21.15.114:0/835065344 >> 172.21.15.114:0/835065344 conn(0x5609be96b300 msgr2=0x5609bed1a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.954+0000 7f93563f8740 1 -- 172.21.15.114:0/835065344 shutdown_connections 2024-05-27T07:51:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.954+0000 7f93563f8740 1 -- 172.21.15.114:0/835065344 wait complete. 2024-05-27T07:51:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.954+0000 7f93563f8740 20 client.18102 trim_cache size 0 max 16384 2024-05-27T07:51:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.970+0000 7f93563f8740 1 Processor -- start 2024-05-27T07:51:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.970+0000 7f93563f8740 1 -- start start 2024-05-27T07:51:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f93563f8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be94e2f0 0x5609bed19170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f93563f8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9564d0 0x5609bed14ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f93563f8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be9771b0 0x5609bed1e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f93563f8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f93481b0b30 con 0x5609be94e2f0 2024-05-27T07:51:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f93563f8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f93481af2c0 con 0x5609be9771b0 2024-05-27T07:51:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f93563f8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f93481ad130 con 0x5609be9564d0 2024-05-27T07:51:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f934cada640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be94e2f0 0x5609bed19170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f934cada640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be94e2f0 0x5609bed19170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49000/0 (socket says 172.21.15.114:49000) 2024-05-27T07:51:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f934cada640 1 -- 172.21.15.114:0/3712036619 learned_addr learned my addr 172.21.15.114:0/3712036619 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f934d2db640 1 --2- 172.21.15.114:0/3712036619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9564d0 0x5609bed14ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f934cada640 1 -- 172.21.15.114:0/3712036619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9564d0 msgr2=0x5609bed14ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f934cada640 1 --2- 172.21.15.114:0/3712036619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9564d0 0x5609bed14ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f934cada640 1 -- 172.21.15.114:0/3712036619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be9771b0 msgr2=0x5609bed1e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f934cada640 1 --2- 172.21.15.114:0/3712036619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be9771b0 0x5609bed1e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f934cada640 1 -- 172.21.15.114:0/3712036619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609bed15c10 con 0x5609be94e2f0 2024-05-27T07:51:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f934cada640 1 --2- 172.21.15.114:0/3712036619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be94e2f0 0x5609bed19170 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f933800ac60 tx=0x7f9338006d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a703e6db465c993 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f9332ffd640 1 -- 172.21.15.114:0/3712036619 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933803a070 con 0x5609be94e2f0 2024-05-27T07:51:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f9332ffd640 1 -- 172.21.15.114:0/3712036619 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9338025bd0 con 0x5609be94e2f0 2024-05-27T07:51:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.971+0000 7f9332ffd640 1 -- 172.21.15.114:0/3712036619 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933802a040 con 0x5609be94e2f0 2024-05-27T07:51:21.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.972+0000 7f93563f8740 1 -- 172.21.15.114:0/3712036619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be94e2f0 msgr2=0x5609bed19170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:21.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.972+0000 7f93563f8740 1 --2- 172.21.15.114:0/3712036619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be94e2f0 0x5609bed19170 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f933800ac60 tx=0x7f9338006d60 comp rx=0 tx=0).stop 2024-05-27T07:51:21.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.972+0000 7f93563f8740 1 -- 172.21.15.114:0/3712036619 shutdown_connections 2024-05-27T07:51:21.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.972+0000 7f93563f8740 1 --2- 172.21.15.114:0/3712036619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be9771b0 0x5609bed1e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.972+0000 7f93563f8740 1 --2- 172.21.15.114:0/3712036619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9564d0 0x5609bed14ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.972+0000 7f93563f8740 1 --2- 172.21.15.114:0/3712036619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be94e2f0 0x5609bed19170 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.972+0000 7f93563f8740 1 -- 172.21.15.114:0/3712036619 >> 172.21.15.114:0/3712036619 conn(0x5609be96b300 msgr2=0x5609bed143a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.972+0000 7f93563f8740 1 -- 172.21.15.114:0/3712036619 shutdown_connections 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.972+0000 7f93563f8740 1 -- 172.21.15.114:0/3712036619 wait complete. 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.973+0000 7f93563f8740 1 -- start start 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.973+0000 7f93563f8740 1 Processor -- start 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.973+0000 7f93563f8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 0x5609bed27920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.973+0000 7f93563f8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 0x5609bed27e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.973+0000 7f93563f8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9771b0 0x5609bed283a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.973+0000 7f93563f8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f93481ad780 con 0x5609be9564d0 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.973+0000 7f93563f8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f93481aebf0 con 0x5609be94e2f0 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.973+0000 7f93563f8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f93481ade50 con 0x5609be9771b0 2024-05-27T07:51:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.973+0000 7f93563f8740 10 client.0 _ll_register_callbacks cb 0x7f9344009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.973+0000 7f934d2db640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 0x5609bed27e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.973+0000 7f934d2db640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 0x5609bed27e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49008/0 (socket says 172.21.15.114:49008) 2024-05-27T07:51:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.973+0000 7f934d2db640 1 -- 172.21.15.114:0/2550347846 learned_addr learned my addr 172.21.15.114:0/2550347846 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.973+0000 7f934cada640 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 0x5609bed27920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f934cada640 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9771b0 0x5609bed283a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f934d2db640 1 -- 172.21.15.114:0/2550347846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9771b0 msgr2=0x5609bed283a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f934d2db640 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9771b0 0x5609bed283a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f934d2db640 1 -- 172.21.15.114:0/2550347846 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 msgr2=0x5609bed27920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f934d2db640 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 0x5609bed27920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f934d2db640 1 -- 172.21.15.114:0/2550347846 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609bed9d7b0 con 0x5609be9564d0 2024-05-27T07:51:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f934cada640 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9771b0 0x5609bed283a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:51:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f934d2db640 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 0x5609bed27e60 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f9344009900 tx=0x7f9344000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ea6a9efe54e864 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f9330ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f9330ff9640 1 -- 172.21.15.114:0/2550347846 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344035040 con 0x5609be9564d0 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f9330ff9640 1 -- 172.21.15.114:0/2550347846 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9538+0+0 (secure 0 0 0) 0x7f93440043f0 con 0x5609be9564d0 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f93563f8740 1 -- 172.21.15.114:0/2550347846 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9338025bd0 con 0x5609be9564d0 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f9330ff9640 1 -- 172.21.15.114:0/2550347846 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f9344057050 con 0x5609be9564d0 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f93563f8740 10 client.18108 fetch_fsmap learned FSMap version 65 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f93563f8740 1 -- 172.21.15.114:0/2550347846 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x5609bed9e9b0 con 0x5609be9564d0 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.974+0000 7f9330ff9640 1 -- 172.21.15.114:0/2550347846 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9338025bd0 con 0x5609be9564d0 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.975+0000 7f9330ff9640 1 -- 172.21.15.114:0/2550347846 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 65) ==== 5910+0+0 (secure 0 0 0) 0x7f9344045560 con 0x5609be9564d0 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.975+0000 7f93563f8740 10 client.18108 fetch_fsmap finished waiting for FSMap version 65 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.975+0000 7f93563f8740 10 client.18108 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.975+0000 7f93563f8740 20 client.18108 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.975+0000 7f93563f8740 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609be90b1c0 0x5609beda1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.975+0000 7f93563f8740 4 client.18108 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:51:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.975+0000 7f93563f8740 1 -- 172.21.15.114:0/2550347846 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5609bec86360 con 0x5609be90b1c0 2024-05-27T07:51:21.980 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.975+0000 7f934cada640 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609be90b1c0 0x5609beda1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:21.980 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.975+0000 7f9330ff9640 10 client.18108 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:21.980 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:21.975+0000 7f934cada640 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609be90b1c0 0x5609beda1100 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d001c22c06e20a92 server_cookie=61698c0d2b2eceec in_seq=0 out_seq=0 2024-05-27T07:51:21.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:21 smithi186 ceph-mon[36647]: pgmap v386: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 185 op/s 2024-05-27T07:51:21.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:21 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:51:21.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:21 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:21.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:21 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:51:21.981 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:21 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:21.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:21 smithi163 ceph-mon[36451]: pgmap v386: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 185 op/s 2024-05-27T07:51:21.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:21 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:51:21.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:21 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:21.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:21 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:51:21.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:21 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:22.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1133266 9.86 MB/sec execute 454 sec latency 280.648 ms 2024-05-27T07:51:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f9330ff9640 1 -- 172.21.15.114:0/2550347846 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5609bec86360 con 0x5609be90b1c0 2024-05-27T07:51:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f9330ff9640 10 client.18108 handle_command_reply: tid=0 2024-05-27T07:51:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f93563f8740 1 client.18108 shutdown 2024-05-27T07:51:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f93563f8740 1 -- 172.21.15.114:0/2550347846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 msgr2=0x5609bed27e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f93563f8740 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 0x5609bed27e60 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f9344009900 tx=0x7f9344000bc0 comp rx=0 tx=0).stop 2024-05-27T07:51:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f93563f8740 1 -- 172.21.15.114:0/2550347846 shutdown_connections 2024-05-27T07:51:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f93563f8740 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5609be90b1c0 0x5609beda1100 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f93563f8740 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609be9771b0 0x5609bed283a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f93563f8740 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5609be9564d0 0x5609bed27e60 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f93563f8740 1 --2- 172.21.15.114:0/2550347846 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5609be94e2f0 0x5609bed27920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f93563f8740 1 -- 172.21.15.114:0/2550347846 >> 172.21.15.114:0/2550347846 conn(0x5609be96b300 msgr2=0x5609bed23400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f9330ff9640 0 client.18108 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f93563f8740 1 -- 172.21.15.114:0/2550347846 shutdown_connections 2024-05-27T07:51:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.494+0000 7f93563f8740 1 -- 172.21.15.114:0/2550347846 wait complete. 2024-05-27T07:51:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.495+0000 7f93563f8740 20 client.18108 trim_cache size 0 max 16384 2024-05-27T07:51:22.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:51:22.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:51:22.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:51:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.495+0000 7f934f53f640 1 -- 172.21.15.114:0/2842994430 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f93340b2950 msgr2=0x7f93340b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.495+0000 7f934f53f640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f93340b2950 0x7f93340b4e60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f93400a9d60 tx=0x7f934000af30 comp rx=0 tx=0).stop 2024-05-27T07:51:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.495+0000 7f934f53f640 1 -- 172.21.15.114:0/2842994430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9348109030 msgr2=0x7f93481996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.495+0000 7f934f53f640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9348109030 0x7f93481996c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f934000cce0 tx=0x7f934000a620 comp rx=0 tx=0).stop 2024-05-27T07:51:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.495+0000 7f934f53f640 1 -- 172.21.15.114:0/2842994430 shutdown_connections 2024-05-27T07:51:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.496+0000 7f934f53f640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f93340b2950 0x7f93340b4e60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.496+0000 7f934f53f640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9348109030 0x7f93481996c0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.496+0000 7f934f53f640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9348108b90 0x7f934819fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.496+0000 7f934f53f640 1 --2- 172.21.15.114:0/2842994430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93481032e0 0x7f934819f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.496+0000 7f934f53f640 1 -- 172.21.15.114:0/2842994430 >> 172.21.15.114:0/2842994430 conn(0x7f93480782d0 msgr2=0x7f9348104230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.496+0000 7f934f53f640 1 -- 172.21.15.114:0/2842994430 shutdown_connections 2024-05-27T07:51:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:22.496+0000 7f934f53f640 1 -- 172.21.15.114:0/2842994430 wait complete. 2024-05-27T07:51:22.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:22 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:51:22.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:22 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:51:22.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:22 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:51:23.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1134563 9.85 MB/sec execute 455 sec latency 346.582 ms 2024-05-27T07:51:23.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:23 smithi114 ceph-mon[32038]: pgmap v387: 129 pgs: 129 active+clean; 74 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.0 MiB/s wr, 265 op/s 2024-05-27T07:51:23.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.720+0000 7fab34eff700 1 -- 172.21.15.114:0/4031708413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab301034b0 msgr2=0x7fab301038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:23.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.720+0000 7fab34eff700 1 --2- 172.21.15.114:0/4031708413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab301034b0 0x7fab301038c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fab200099c0 tx=0x7fab20009cd0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.720+0000 7fab34eff700 1 -- 172.21.15.114:0/4031708413 shutdown_connections 2024-05-27T07:51:23.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.720+0000 7fab34eff700 1 --2- 172.21.15.114:0/4031708413 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab30108730 0x7fab30110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.720+0000 7fab34eff700 1 --2- 172.21.15.114:0/4031708413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab301034b0 0x7fab301038c0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.720+0000 7fab34eff700 1 --2- 172.21.15.114:0/4031708413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab30107de0 0x7fab301081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.720+0000 7fab34eff700 1 -- 172.21.15.114:0/4031708413 >> 172.21.15.114:0/4031708413 conn(0x7fab300fedb0 msgr2=0x7fab301011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:23.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.720+0000 7fab34eff700 1 -- 172.21.15.114:0/4031708413 shutdown_connections 2024-05-27T07:51:23.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.720+0000 7fab34eff700 1 -- 172.21.15.114:0/4031708413 wait complete. 2024-05-27T07:51:23.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.721+0000 7fab34eff700 1 Processor -- start 2024-05-27T07:51:23.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.721+0000 7fab34eff700 1 -- start start 2024-05-27T07:51:23.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.721+0000 7fab34eff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab301034b0 0x7fab3019bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.721+0000 7fab34eff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab30107de0 0x7fab3019c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.721+0000 7fab34eff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab30108730 0x7fab3019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.721+0000 7fab34eff700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fab30113360 con 0x7fab301034b0 2024-05-27T07:51:23.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.721+0000 7fab34eff700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fab301131e0 con 0x7fab30107de0 2024-05-27T07:51:23.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.721+0000 7fab34eff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab301134e0 con 0x7fab30108730 2024-05-27T07:51:23.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.722+0000 7fab2e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab301034b0 0x7fab3019bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.722+0000 7fab2e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab301034b0 0x7fab3019bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49020/0 (socket says 172.21.15.114:49020) 2024-05-27T07:51:23.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.722+0000 7fab2e59c700 1 -- 172.21.15.114:0/765467350 learned_addr learned my addr 172.21.15.114:0/765467350 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:23.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.722+0000 7fab2ed9d700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab30108730 0x7fab3019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.722+0000 7fab2dd9b700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab30107de0 0x7fab3019c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.722+0000 7fab2e59c700 1 -- 172.21.15.114:0/765467350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab30108730 msgr2=0x7fab3019c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:23.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.722+0000 7fab2e59c700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab30108730 0x7fab3019c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.722+0000 7fab2e59c700 1 -- 172.21.15.114:0/765467350 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab30107de0 msgr2=0x7fab3019c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:23.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.722+0000 7fab2e59c700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab30107de0 0x7fab3019c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.722+0000 7fab2e59c700 1 -- 172.21.15.114:0/765467350 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab30199a10 con 0x7fab301034b0 2024-05-27T07:51:23.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.722+0000 7fab2e59c700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab301034b0 0x7fab3019bc20 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fab20006010 tx=0x7fab20004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:23.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.723+0000 7fab1f7fe700 1 -- 172.21.15.114:0/765467350 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab20026070 con 0x7fab301034b0 2024-05-27T07:51:23.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.723+0000 7fab1f7fe700 1 -- 172.21.15.114:0/765467350 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab20004630 con 0x7fab301034b0 2024-05-27T07:51:23.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.723+0000 7fab34eff700 1 -- 172.21.15.114:0/765467350 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab30199ca0 con 0x7fab301034b0 2024-05-27T07:51:23.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.723+0000 7fab1f7fe700 1 -- 172.21.15.114:0/765467350 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2000fb20 con 0x7fab301034b0 2024-05-27T07:51:23.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.723+0000 7fab34eff700 1 -- 172.21.15.114:0/765467350 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab3019a1d0 con 0x7fab301034b0 2024-05-27T07:51:23.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.725+0000 7fab1f7fe700 1 -- 172.21.15.114:0/765467350 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fab20029070 con 0x7fab301034b0 2024-05-27T07:51:23.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.726+0000 7fab1f7fe700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fab180b2a70 0x7fab180b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.726+0000 7fab2dd9b700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fab180b2a70 0x7fab180b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.726+0000 7fab1f7fe700 1 -- 172.21.15.114:0/765467350 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9538+0+0 (secure 0 0 0) 0x7fab200e32f0 con 0x7fab301034b0 2024-05-27T07:51:23.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.726+0000 7fab43914b80 1 Processor -- start 2024-05-27T07:51:23.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab2dd9b700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fab180b2a70 0x7fab180b4f80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fab30106540 tx=0x7fab24009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:23.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab43914b80 1 -- start start 2024-05-27T07:51:23.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab43914b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 0x559650848b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab43914b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650849070 0x559650842be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab43914b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650843120 0x559650843530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab43914b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559650770d70 con 0x5596506c3e40 2024-05-27T07:51:23.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab43914b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559650760910 con 0x559650849070 2024-05-27T07:51:23.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab43914b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559650764950 con 0x559650843120 2024-05-27T07:51:23.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab2e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 0x559650848b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab2e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 0x559650848b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49030/0 (socket says 172.21.15.114:49030) 2024-05-27T07:51:23.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab2e59c700 1 -- 172.21.15.114:0/3158393175 learned_addr learned my addr 172.21.15.114:0/3158393175 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:23.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab2ed9d700 1 --2- 172.21.15.114:0/3158393175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650843120 0x559650843530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab2dd9b700 1 --2- 172.21.15.114:0/3158393175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650849070 0x559650842be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab2e59c700 1 -- 172.21.15.114:0/3158393175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650843120 msgr2=0x559650843530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:23.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab2e59c700 1 --2- 172.21.15.114:0/3158393175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650843120 0x559650843530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab2e59c700 1 -- 172.21.15.114:0/3158393175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650849070 msgr2=0x559650842be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:23.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab2e59c700 1 --2- 172.21.15.114:0/3158393175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650849070 0x559650842be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.727+0000 7fab2e59c700 1 -- 172.21.15.114:0/3158393175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559650709fb0 con 0x5596506c3e40 2024-05-27T07:51:23.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.728+0000 7fab2e59c700 1 --2- 172.21.15.114:0/3158393175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 0x559650848b30 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fab200aa1f0 tx=0x7fab200a9ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4f3c0458a17ed2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:23.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.728+0000 7fab1cff9700 1 -- 172.21.15.114:0/3158393175 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab200aa960 con 0x5596506c3e40 2024-05-27T07:51:23.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.728+0000 7fab1cff9700 1 -- 172.21.15.114:0/3158393175 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab200aa260 con 0x5596506c3e40 2024-05-27T07:51:23.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.728+0000 7fab1cff9700 1 -- 172.21.15.114:0/3158393175 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab200b2ba0 con 0x5596506c3e40 2024-05-27T07:51:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.729+0000 7fab43914b80 1 -- 172.21.15.114:0/3158393175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 msgr2=0x559650848b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.729+0000 7fab43914b80 1 --2- 172.21.15.114:0/3158393175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 0x559650848b30 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fab200aa1f0 tx=0x7fab200a9ac0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.729+0000 7fab43914b80 1 -- 172.21.15.114:0/3158393175 shutdown_connections 2024-05-27T07:51:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.729+0000 7fab43914b80 1 --2- 172.21.15.114:0/3158393175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650843120 0x559650843530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.729+0000 7fab43914b80 1 --2- 172.21.15.114:0/3158393175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 0x559650848b30 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.729+0000 7fab43914b80 1 --2- 172.21.15.114:0/3158393175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650849070 0x559650842be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.729+0000 7fab43914b80 1 -- 172.21.15.114:0/3158393175 >> 172.21.15.114:0/3158393175 conn(0x5596506c6770 msgr2=0x55965083dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.729+0000 7fab43914b80 1 -- 172.21.15.114:0/3158393175 shutdown_connections 2024-05-27T07:51:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.729+0000 7fab43914b80 1 -- 172.21.15.114:0/3158393175 wait complete. 2024-05-27T07:51:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.729+0000 7fab43914b80 1 -- start start 2024-05-27T07:51:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.729+0000 7fab43914b80 1 Processor -- start 2024-05-27T07:51:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab43914b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596506c3e40 0x55965084c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab43914b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650843120 0x55965084ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab43914b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650849070 0x55965084d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab43914b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559650666fa0 con 0x559650843120 2024-05-27T07:51:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab43914b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5596506c8da0 con 0x5596506c3e40 2024-05-27T07:51:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab43914b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55965084f920 con 0x559650849070 2024-05-27T07:51:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab43914b80 10 client.0 _ll_register_callbacks cb 0x5596506f75b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab2dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650843120 0x55965084ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab2dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650843120 0x55965084ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49034/0 (socket says 172.21.15.114:49034) 2024-05-27T07:51:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab2dd9b700 1 -- 172.21.15.114:0/746579129 learned_addr learned my addr 172.21.15.114:0/746579129 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab2ed9d700 1 --2- 172.21.15.114:0/746579129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650849070 0x55965084d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab2e59c700 1 --2- 172.21.15.114:0/746579129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596506c3e40 0x55965084c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab2dd9b700 1 -- 172.21.15.114:0/746579129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650849070 msgr2=0x55965084d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab2dd9b700 1 --2- 172.21.15.114:0/746579129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650849070 0x55965084d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab2dd9b700 1 -- 172.21.15.114:0/746579129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596506c3e40 msgr2=0x55965084c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab2dd9b700 1 --2- 172.21.15.114:0/746579129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596506c3e40 0x55965084c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab2dd9b700 1 -- 172.21.15.114:0/746579129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596506b08d0 con 0x559650843120 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab2dd9b700 1 --2- 172.21.15.114:0/746579129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650843120 0x55965084ce90 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fab24004520 tx=0x7fab240047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cfca135c786b200 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.730+0000 7fab15ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab15ffb700 1 -- 172.21.15.114:0/746579129 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24004550 con 0x559650843120 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab43914b80 1 -- 172.21.15.114:0/746579129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab200aa260 con 0x559650843120 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab15ffb700 1 -- 172.21.15.114:0/746579129 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9538+0+0 (secure 0 0 0) 0x7fab24003cf0 con 0x559650843120 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab15ffb700 1 -- 172.21.15.114:0/746579129 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fab2402b610 con 0x559650843120 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab43914b80 10 client.18120 fetch_fsmap learned FSMap version 65 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab43914b80 1 -- 172.21.15.114:0/746579129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55965071a760 con 0x559650843120 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab15ffb700 1 -- 172.21.15.114:0/746579129 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab200aa260 con 0x559650843120 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab15ffb700 1 -- 172.21.15.114:0/746579129 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 65) ==== 5910+0+0 (secure 0 0 0) 0x7fab24028ba0 con 0x559650843120 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab43914b80 10 client.18120 fetch_fsmap finished waiting for FSMap version 65 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab43914b80 10 client.18120 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab43914b80 20 client.18120 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab43914b80 1 --2- 172.21.15.114:0/746579129 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55965085cb50 0x55965085efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab43914b80 4 client.18120 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:23.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab43914b80 1 -- 172.21.15.114:0/746579129 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559650772cc0 con 0x55965085cb50 2024-05-27T07:51:23.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.731+0000 7fab2e59c700 1 --2- 172.21.15.114:0/746579129 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55965085cb50 0x55965085efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.732+0000 7fab15ffb700 10 client.18120 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:23.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.732+0000 7fab2e59c700 1 --2- 172.21.15.114:0/746579129 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55965085cb50 0x55965085efa0 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57fdfff0e965af85 server_cookie=8b959cd268816712 in_seq=0 out_seq=0 2024-05-27T07:51:23.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.733+0000 7fab15ffb700 1 -- 172.21.15.114:0/746579129 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559650772cc0 con 0x55965085cb50 2024-05-27T07:51:23.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.733+0000 7fab15ffb700 10 client.18120 handle_command_reply: tid=0 2024-05-27T07:51:23.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.733+0000 7fab43914b80 1 client.18120 shutdown 2024-05-27T07:51:23.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.733+0000 7fab43914b80 1 -- 172.21.15.114:0/746579129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650843120 msgr2=0x55965084ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:23.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.734+0000 7fab43914b80 1 --2- 172.21.15.114:0/746579129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650843120 0x55965084ce90 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fab24004520 tx=0x7fab240047c0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.734+0000 7fab43914b80 1 -- 172.21.15.114:0/746579129 shutdown_connections 2024-05-27T07:51:23.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.734+0000 7fab43914b80 1 --2- 172.21.15.114:0/746579129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650849070 0x55965084d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.734+0000 7fab43914b80 1 --2- 172.21.15.114:0/746579129 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55965085cb50 0x55965085efa0 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.734+0000 7fab43914b80 1 --2- 172.21.15.114:0/746579129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596506c3e40 0x55965084c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.734+0000 7fab43914b80 1 --2- 172.21.15.114:0/746579129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650843120 0x55965084ce90 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.734+0000 7fab43914b80 1 -- 172.21.15.114:0/746579129 >> 172.21.15.114:0/746579129 conn(0x5596506c6770 msgr2=0x559650841bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:23.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.734+0000 7fab15ffb700 0 client.18120 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:23.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.734+0000 7fab43914b80 1 -- 172.21.15.114:0/746579129 shutdown_connections 2024-05-27T07:51:23.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.734+0000 7fab43914b80 1 -- 172.21.15.114:0/746579129 wait complete. 2024-05-27T07:51:23.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.734+0000 7fab43914b80 20 client.18120 trim_cache size 0 max 16384 2024-05-27T07:51:23.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.758+0000 7fab43914b80 1 Processor -- start 2024-05-27T07:51:23.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.758+0000 7fab43914b80 1 -- start start 2024-05-27T07:51:23.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab43914b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596506c3e40 0x55965084d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab43914b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650843120 0x559650846ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab43914b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650849070 0x559650858980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab43914b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fab301b4200 con 0x559650849070 2024-05-27T07:51:23.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab43914b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fab301b2970 con 0x559650843120 2024-05-27T07:51:23.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab43914b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab301b07e0 con 0x5596506c3e40 2024-05-27T07:51:23.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab2ed9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650849070 0x559650858980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab2e59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596506c3e40 0x55965084d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab2dd9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650843120 0x559650846ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab2ed9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650849070 0x559650858980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49042/0 (socket says 172.21.15.114:49042) 2024-05-27T07:51:23.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab2ed9d700 1 -- 172.21.15.114:0/2244799689 learned_addr learned my addr 172.21.15.114:0/2244799689 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:23.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab2ed9d700 1 -- 172.21.15.114:0/2244799689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596506c3e40 msgr2=0x55965084d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:23.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab2ed9d700 1 --2- 172.21.15.114:0/2244799689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596506c3e40 0x55965084d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab2ed9d700 1 -- 172.21.15.114:0/2244799689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650843120 msgr2=0x559650846ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:23.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab2ed9d700 1 --2- 172.21.15.114:0/2244799689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650843120 0x559650846ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.759+0000 7fab2ed9d700 1 -- 172.21.15.114:0/2244799689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596508372f0 con 0x559650849070 2024-05-27T07:51:23.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.760+0000 7fab2ed9d700 1 --2- 172.21.15.114:0/2244799689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650849070 0x559650858980 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fab28011920 tx=0x7fab28009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a65b893cedb83ec server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:23.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.760+0000 7fab17fff700 1 -- 172.21.15.114:0/2244799689 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab28003c00 con 0x559650849070 2024-05-27T07:51:23.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.760+0000 7fab17fff700 1 -- 172.21.15.114:0/2244799689 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab28003da0 con 0x559650849070 2024-05-27T07:51:23.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.760+0000 7fab17fff700 1 -- 172.21.15.114:0/2244799689 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab28017910 con 0x559650849070 2024-05-27T07:51:23.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.761+0000 7fab43914b80 1 -- 172.21.15.114:0/2244799689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650849070 msgr2=0x559650858980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:23.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.761+0000 7fab43914b80 1 --2- 172.21.15.114:0/2244799689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650849070 0x559650858980 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fab28011920 tx=0x7fab28009920 comp rx=0 tx=0).stop 2024-05-27T07:51:23.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.761+0000 7fab43914b80 1 -- 172.21.15.114:0/2244799689 shutdown_connections 2024-05-27T07:51:23.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.761+0000 7fab43914b80 1 --2- 172.21.15.114:0/2244799689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559650849070 0x559650858980 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.761+0000 7fab43914b80 1 --2- 172.21.15.114:0/2244799689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596506c3e40 0x55965084d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.761+0000 7fab43914b80 1 --2- 172.21.15.114:0/2244799689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650843120 0x559650846ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.761+0000 7fab43914b80 1 -- 172.21.15.114:0/2244799689 >> 172.21.15.114:0/2244799689 conn(0x5596506c6770 msgr2=0x559650846490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:23.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.762+0000 7fab43914b80 1 -- 172.21.15.114:0/2244799689 shutdown_connections 2024-05-27T07:51:23.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.762+0000 7fab43914b80 1 -- 172.21.15.114:0/2244799689 wait complete. 2024-05-27T07:51:23.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.762+0000 7fab43914b80 1 -- start start 2024-05-27T07:51:23.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.762+0000 7fab43914b80 1 Processor -- start 2024-05-27T07:51:23.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab43914b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 0x559650854ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab43914b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650843120 0x559650855410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab43914b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650849070 0x5596508503b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab43914b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fab301b0e30 con 0x5596506c3e40 2024-05-27T07:51:23.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab43914b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fab301b22a0 con 0x559650843120 2024-05-27T07:51:23.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab43914b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab301b1500 con 0x559650849070 2024-05-27T07:51:23.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab43914b80 10 client.0 _ll_register_callbacks cb 0x559650843cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:23.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab2e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 0x559650854ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab2ed9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650849070 0x5596508503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab2dd9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650843120 0x559650855410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab2e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 0x559650854ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49052/0 (socket says 172.21.15.114:49052) 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab2e59c700 1 -- 172.21.15.114:0/4205839947 learned_addr learned my addr 172.21.15.114:0/4205839947 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab2dd9b700 1 -- 172.21.15.114:0/4205839947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650849070 msgr2=0x5596508503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab2dd9b700 1 --2- 172.21.15.114:0/4205839947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650849070 0x5596508503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab2dd9b700 1 -- 172.21.15.114:0/4205839947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 msgr2=0x559650854ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab2dd9b700 1 --2- 172.21.15.114:0/4205839947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 0x559650854ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab2dd9b700 1 -- 172.21.15.114:0/4205839947 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559650850b50 con 0x559650843120 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab2ed9d700 1 --2- 172.21.15.114:0/4205839947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650849070 0x5596508503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab2e59c700 1 --2- 172.21.15.114:0/4205839947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 0x559650854ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab2dd9b700 1 --2- 172.21.15.114:0/4205839947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650843120 0x559650855410 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fab240044f0 tx=0x7fab240117d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49e845cfb8deaee3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.763+0000 7fab15ffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab15ffb700 1 -- 172.21.15.114:0/4205839947 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2402b370 con 0x559650843120 2024-05-27T07:51:23.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab15ffb700 1 -- 172.21.15.114:0/4205839947 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9538+0+0 (secure 0 0 0) 0x7fab2401da80 con 0x559650843120 2024-05-27T07:51:23.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab43914b80 1 -- 172.21.15.114:0/4205839947 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab28003da0 con 0x559650843120 2024-05-27T07:51:23.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab15ffb700 1 -- 172.21.15.114:0/4205839947 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fab24045050 con 0x559650843120 2024-05-27T07:51:23.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab43914b80 10 client.27551 fetch_fsmap learned FSMap version 65 2024-05-27T07:51:23.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab43914b80 1 -- 172.21.15.114:0/4205839947 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x5596508c7150 con 0x559650843120 2024-05-27T07:51:23.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab15ffb700 1 -- 172.21.15.114:0/4205839947 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab28003da0 con 0x559650843120 2024-05-27T07:51:23.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab15ffb700 1 -- 172.21.15.114:0/4205839947 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 65) ==== 5910+0+0 (secure 0 0 0) 0x7fab24041800 con 0x559650843120 2024-05-27T07:51:23.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab43914b80 10 client.27551 fetch_fsmap finished waiting for FSMap version 65 2024-05-27T07:51:23.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab43914b80 10 client.27551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:23.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab43914b80 20 client.27551 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:23.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab43914b80 1 --2- 172.21.15.114:0/4205839947 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5596508c7770 0x5596508c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:23.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab43914b80 4 client.27551 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:51:23.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab43914b80 1 -- 172.21.15.114:0/4205839947 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55965076c4a0 con 0x5596508c7770 2024-05-27T07:51:23.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.764+0000 7fab2e59c700 1 --2- 172.21.15.114:0/4205839947 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5596508c7770 0x5596508c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:23.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.765+0000 7fab15ffb700 10 client.27551 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:23.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.765+0000 7fab2e59c700 1 --2- 172.21.15.114:0/4205839947 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5596508c7770 0x5596508c9b90 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f26904c9cfd28b2e server_cookie=9da43b661be10c60 in_seq=0 out_seq=0 2024-05-27T07:51:23.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.766+0000 7fab15ffb700 1 -- 172.21.15.114:0/4205839947 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55965076c4a0 con 0x5596508c7770 2024-05-27T07:51:23.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.766+0000 7fab15ffb700 10 client.27551 handle_command_reply: tid=0 2024-05-27T07:51:23.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.766+0000 7fab43914b80 1 client.27551 shutdown 2024-05-27T07:51:23.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.766+0000 7fab43914b80 1 -- 172.21.15.114:0/4205839947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650843120 msgr2=0x559650855410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:23.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.766+0000 7fab43914b80 1 --2- 172.21.15.114:0/4205839947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650843120 0x559650855410 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fab240044f0 tx=0x7fab240117d0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.766+0000 7fab43914b80 1 -- 172.21.15.114:0/4205839947 shutdown_connections 2024-05-27T07:51:23.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.766+0000 7fab43914b80 1 --2- 172.21.15.114:0/4205839947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559650849070 0x5596508503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.766+0000 7fab43914b80 1 --2- 172.21.15.114:0/4205839947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5596506c3e40 0x559650854ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.766+0000 7fab43914b80 1 --2- 172.21.15.114:0/4205839947 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5596508c7770 0x5596508c9b90 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.766+0000 7fab43914b80 1 --2- 172.21.15.114:0/4205839947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559650843120 0x559650855410 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.766+0000 7fab15ffb700 0 client.27551 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:23.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.766+0000 7fab43914b80 1 -- 172.21.15.114:0/4205839947 >> 172.21.15.114:0/4205839947 conn(0x5596506c6770 msgr2=0x5596508374f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:23.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab43914b80 1 -- 172.21.15.114:0/4205839947 shutdown_connections 2024-05-27T07:51:23.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab43914b80 1 -- 172.21.15.114:0/4205839947 wait complete. 2024-05-27T07:51:23.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab43914b80 20 client.27551 trim_cache size 0 max 16384 2024-05-27T07:51:23.772 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:51:23.772 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:51:23.772 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:51:23.772 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:51:23.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab34eff700 1 -- 172.21.15.114:0/765467350 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fab180b2a70 msgr2=0x7fab180b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:23.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab34eff700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fab180b2a70 0x7fab180b4f80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fab30106540 tx=0x7fab24009260 comp rx=0 tx=0).stop 2024-05-27T07:51:23.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab34eff700 1 -- 172.21.15.114:0/765467350 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab301034b0 msgr2=0x7fab3019bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:23.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab34eff700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab301034b0 0x7fab3019bc20 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fab20006010 tx=0x7fab20004270 comp rx=0 tx=0).stop 2024-05-27T07:51:23.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab34eff700 1 -- 172.21.15.114:0/765467350 shutdown_connections 2024-05-27T07:51:23.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab34eff700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab30108730 0x7fab3019c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab34eff700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab301034b0 0x7fab3019bc20 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab34eff700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fab180b2a70 0x7fab180b4f80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab34eff700 1 --2- 172.21.15.114:0/765467350 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab30107de0 0x7fab3019c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:23.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab34eff700 1 -- 172.21.15.114:0/765467350 >> 172.21.15.114:0/765467350 conn(0x7fab300fedb0 msgr2=0x7fab30101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:23.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab34eff700 1 -- 172.21.15.114:0/765467350 shutdown_connections 2024-05-27T07:51:23.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:23.767+0000 7fab34eff700 1 -- 172.21.15.114:0/765467350 wait complete. 2024-05-27T07:51:23.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:23 smithi163 ceph-mon[36451]: pgmap v387: 129 pgs: 129 active+clean; 74 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.0 MiB/s wr, 265 op/s 2024-05-27T07:51:23.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:23 smithi186 ceph-mon[36647]: pgmap v387: 129 pgs: 129 active+clean; 74 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.0 MiB/s wr, 265 op/s 2024-05-27T07:51:24.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1136636 9.85 MB/sec execute 456 sec latency 70.452 ms 2024-05-27T07:51:24.641 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:51:24.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:51:24.908 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:25.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1137646 9.84 MB/sec execute 457 sec latency 18.346 ms 2024-05-27T07:51:25.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:25 smithi163 ceph-mon[36451]: pgmap v388: 129 pgs: 129 active+clean; 74 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 1.9 MiB/s wr, 125 op/s 2024-05-27T07:51:25.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:25 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:51:25.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:25 smithi163 systemd[1]: Stopped Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:51:25.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:25 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Consumed 1.016s CPU time. 2024-05-27T07:51:25.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:25 smithi163 systemd[1]: Starting Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:51:25.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:25 smithi163 podman[73331]: 2024-05-27 07:51:25.611346528 +0000 UTC m=+0.017625477 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:51:25.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:25 smithi163 podman[73331]: 2024-05-27 07:51:25.734122166 +0000 UTC m=+0.140401116 container create 5ec1acc48bde00ae0745b5647b4b7ac4c09611a87a75f275552132e579a1c525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, io.buildah.version=1.33.5) 2024-05-27T07:51:25.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:25 smithi114 ceph-mon[32038]: pgmap v388: 129 pgs: 129 active+clean; 74 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 1.9 MiB/s wr, 125 op/s 2024-05-27T07:51:25.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:25 smithi186 ceph-mon[36647]: pgmap v388: 129 pgs: 129 active+clean; 74 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 1.9 MiB/s wr, 125 op/s 2024-05-27T07:51:26.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1139955 9.85 MB/sec execute 458 sec latency 26.432 ms 2024-05-27T07:51:26.403 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:26 smithi163 podman[73331]: 2024-05-27 07:51:26.012296842 +0000 UTC m=+0.418575792 container init 5ec1acc48bde00ae0745b5647b4b7ac4c09611a87a75f275552132e579a1c525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True) 2024-05-27T07:51:26.403 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:26 smithi163 podman[73331]: 2024-05-27 07:51:26.015176404 +0000 UTC m=+0.421455363 container start 5ec1acc48bde00ae0745b5647b4b7ac4c09611a87a75f275552132e579a1c525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-05-27T07:51:26.403 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:26 smithi163 ceph-mds[73350]: -- 172.21.15.163:0/4132058896 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5579236efa40 con 0x557923737600 2024-05-27T07:51:26.404 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: starting mds.e at 2024-05-27T07:51:26.404 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:26 smithi163 bash[73331]: 5ec1acc48bde00ae0745b5647b4b7ac4c09611a87a75f275552132e579a1c525 2024-05-27T07:51:26.404 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:51:26 smithi163 systemd[1]: Started Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:51:26.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.641+0000 7f5c5a122700 1 -- 172.21.15.114:0/3925640109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54108ef0 msgr2=0x7f5c54109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:26.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.641+0000 7f5c5a122700 1 --2- 172.21.15.114:0/3925640109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54108ef0 0x7f5c54109300 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0099c0 tx=0x7f5c3c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:51:26.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.642+0000 7f5c5a122700 1 -- 172.21.15.114:0/3925640109 shutdown_connections 2024-05-27T07:51:26.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.642+0000 7f5c5a122700 1 --2- 172.21.15.114:0/3925640109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c54109840 0x7f5c541111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:26.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.642+0000 7f5c5a122700 1 --2- 172.21.15.114:0/3925640109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c541045c0 0x7f5c541049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:26.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.642+0000 7f5c5a122700 1 --2- 172.21.15.114:0/3925640109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54108ef0 0x7f5c54109300 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:26.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.642+0000 7f5c5a122700 1 -- 172.21.15.114:0/3925640109 >> 172.21.15.114:0/3925640109 conn(0x7f5c540ffe90 msgr2=0x7f5c541022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:26.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.642+0000 7f5c5a122700 1 -- 172.21.15.114:0/3925640109 shutdown_connections 2024-05-27T07:51:26.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.642+0000 7f5c5a122700 1 -- 172.21.15.114:0/3925640109 wait complete. 2024-05-27T07:51:26.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.642+0000 7f5c5a122700 1 Processor -- start 2024-05-27T07:51:26.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.642+0000 7f5c5a122700 1 -- start start 2024-05-27T07:51:26.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c5a122700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c541045c0 0x7f5c5419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:26.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c5a122700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c54108ef0 0x7f5c5419b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:26.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c5a122700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54109840 0x7f5c5419f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:26.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c5a122700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5c54114450 con 0x7f5c54109840 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c5a122700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5c541142d0 con 0x7f5c541045c0 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c5a122700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5c541145d0 con 0x7f5c54108ef0 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c53fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54109840 0x7f5c5419f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c52ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c54108ef0 0x7f5c5419b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c537fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c541045c0 0x7f5c5419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c537fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c541045c0 0x7f5c5419ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:32838/0 (socket says 172.21.15.114:32838) 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c52ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c54108ef0 0x7f5c5419b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:57396/0 (socket says 172.21.15.114:57396) 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c53fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54109840 0x7f5c5419f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48190/0 (socket says 172.21.15.114:48190) 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c537fe700 1 -- 172.21.15.114:0/152539664 learned_addr learned my addr 172.21.15.114:0/152539664 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c52ffd700 1 -- 172.21.15.114:0/152539664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c541045c0 msgr2=0x7f5c5419ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c52ffd700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c541045c0 0x7f5c5419ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c52ffd700 1 -- 172.21.15.114:0/152539664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54109840 msgr2=0x7f5c5419f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c52ffd700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54109840 0x7f5c5419f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:26.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c52ffd700 1 -- 172.21.15.114:0/152539664 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c5419fcb0 con 0x7f5c54108ef0 2024-05-27T07:51:26.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c53fff700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54109840 0x7f5c5419f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:26.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.643+0000 7f5c537fe700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c541045c0 0x7f5c5419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:26.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.644+0000 7f5c52ffd700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c54108ef0 0x7f5c5419b380 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0098e0 tx=0x7f5c3c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:26.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.644+0000 7f5c50ff9700 1 -- 172.21.15.114:0/152539664 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c026070 con 0x7f5c54108ef0 2024-05-27T07:51:26.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.644+0000 7f5c5a122700 1 -- 172.21.15.114:0/152539664 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c5419ff40 con 0x7f5c54108ef0 2024-05-27T07:51:26.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.644+0000 7f5c50ff9700 1 -- 172.21.15.114:0/152539664 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c3c003c70 con 0x7f5c54108ef0 2024-05-27T07:51:26.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.644+0000 7f5c50ff9700 1 -- 172.21.15.114:0/152539664 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c00f9c0 con 0x7f5c54108ef0 2024-05-27T07:51:26.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.644+0000 7f5c5a122700 1 -- 172.21.15.114:0/152539664 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c54076af0 con 0x7f5c54108ef0 2024-05-27T07:51:26.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.645+0000 7f5c5a122700 1 -- 172.21.15.114:0/152539664 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c54105a70 con 0x7f5c54108ef0 2024-05-27T07:51:26.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.646+0000 7f5c50ff9700 1 -- 172.21.15.114:0/152539664 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f5c3c029070 con 0x7f5c54108ef0 2024-05-27T07:51:26.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.647+0000 7f5c50ff9700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5c400b29f0 0x7f5c400b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:26.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.647+0000 7f5c537fe700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5c400b29f0 0x7f5c400b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:26.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.647+0000 7f5c50ff9700 1 -- 172.21.15.114:0/152539664 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f5c3c0e3160 con 0x7f5c54108ef0 2024-05-27T07:51:26.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.647+0000 7f5c537fe700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5c400b29f0 0x7f5c400b4f00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009c10 tx=0x7f5c4c00b040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:26.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.652+0000 7f5c50ff9700 1 -- 172.21.15.114:0/152539664 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5c3c013070 con 0x7f5c54108ef0 2024-05-27T07:51:26.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:26 smithi163 ceph-mon[36451]: daemon mds.e restarted 2024-05-27T07:51:26.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:26 smithi114 ceph-mon[32038]: daemon mds.e restarted 2024-05-27T07:51:26.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.926+0000 7f5c5a122700 1 -- 172.21.15.114:0/152539664 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c5419c140 con 0x7f5c54108ef0 2024-05-27T07:51:26.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.926+0000 7f5c50ff9700 1 -- 172.21.15.114:0/152539664 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+8522 (secure 0 0 0) 0x7f5c3c003e20 con 0x7f5c54108ef0 2024-05-27T07:51:26.926 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:51:26.926 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":27557,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3261223371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":3261223371},{"type":"v1","addr":"172.21.15.163:6835","nonce":3261223371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":67,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:51:26.564380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27634":{"gid":27634,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/3266101988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":3266101988},{"type":"v1","addr":"172.21.15.186:6833","nonce":3266101988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:51:26.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.929+0000 7f5c5a122700 1 -- 172.21.15.114:0/152539664 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5c400b29f0 msgr2=0x7f5c400b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:26.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.929+0000 7f5c5a122700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5c400b29f0 0x7f5c400b4f00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009c10 tx=0x7f5c4c00b040 comp rx=0 tx=0).stop 2024-05-27T07:51:26.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.929+0000 7f5c5a122700 1 -- 172.21.15.114:0/152539664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c54108ef0 msgr2=0x7f5c5419b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:26.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.929+0000 7f5c5a122700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c54108ef0 0x7f5c5419b380 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0098e0 tx=0x7f5c3c004270 comp rx=0 tx=0).stop 2024-05-27T07:51:26.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.929+0000 7f5c5a122700 1 -- 172.21.15.114:0/152539664 shutdown_connections 2024-05-27T07:51:26.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.929+0000 7f5c5a122700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54109840 0x7f5c5419f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:26.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.929+0000 7f5c5a122700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5c400b29f0 0x7f5c400b4f00 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:26.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.929+0000 7f5c5a122700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c541045c0 0x7f5c5419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:26.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.929+0000 7f5c5a122700 1 --2- 172.21.15.114:0/152539664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c54108ef0 0x7f5c5419b380 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:26.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.929+0000 7f5c5a122700 1 -- 172.21.15.114:0/152539664 >> 172.21.15.114:0/152539664 conn(0x7f5c540ffe90 msgr2=0x7f5c541086d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:26.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.929+0000 7f5c5a122700 1 -- 172.21.15.114:0/152539664 shutdown_connections 2024-05-27T07:51:26.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:26.929+0000 7f5c5a122700 1 -- 172.21.15.114:0/152539664 wait complete. 2024-05-27T07:51:26.930 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 67 2024-05-27T07:51:26.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:26 smithi186 ceph-mon[36647]: daemon mds.e restarted 2024-05-27T07:51:27.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1140982 9.84 MB/sec execute 459 sec latency 34.440 ms 2024-05-27T07:51:27.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:27 smithi163 ceph-mon[36451]: pgmap v389: 129 pgs: 129 active+clean; 71 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 353 op/s 2024-05-27T07:51:27.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:27 smithi163 ceph-mon[36451]: osdmap e124: 12 total, 12 up, 12 in 2024-05-27T07:51:27.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:27 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6834/3261223371,v1:172.21.15.163:6835/3261223371] up:boot 2024-05-27T07:51:27.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:27 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:51:27.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:27 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:51:27.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:27 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:51:27.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:27 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/152539664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:51:27.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:27 smithi114 ceph-mon[32038]: pgmap v389: 129 pgs: 129 active+clean; 71 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 353 op/s 2024-05-27T07:51:27.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:27 smithi114 ceph-mon[32038]: osdmap e124: 12 total, 12 up, 12 in 2024-05-27T07:51:27.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:27 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6834/3261223371,v1:172.21.15.163:6835/3261223371] up:boot 2024-05-27T07:51:27.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:27 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:51:27.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:27 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:51:27.804 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:27 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:51:27.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:27 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/152539664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:51:27.880 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:27 smithi186 ceph-mon[36647]: pgmap v389: 129 pgs: 129 active+clean; 71 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 353 op/s 2024-05-27T07:51:27.880 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:27 smithi186 ceph-mon[36647]: osdmap e124: 12 total, 12 up, 12 in 2024-05-27T07:51:27.880 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:27 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6834/3261223371,v1:172.21.15.163:6835/3261223371] up:boot 2024-05-27T07:51:27.880 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:27 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:51:27.880 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:27 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:51:27.880 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:27 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:51:27.881 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:27 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/152539664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:51:28.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:51:28.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1142979 9.84 MB/sec execute 460 sec latency 55.397 ms 2024-05-27T07:51:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:29.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1144349 9.83 MB/sec execute 461 sec latency 60.225 ms 2024-05-27T07:51:29.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:29 smithi114 ceph-mon[32038]: pgmap v391: 129 pgs: 129 active+clean; 63 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 339 op/s 2024-05-27T07:51:29.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:29 smithi163 ceph-mon[36451]: pgmap v391: 129 pgs: 129 active+clean; 63 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 339 op/s 2024-05-27T07:51:29.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:29 smithi186 ceph-mon[36647]: pgmap v391: 129 pgs: 129 active+clean; 63 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 339 op/s 2024-05-27T07:51:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.273+0000 7f10b70bb700 1 -- 172.21.15.114:0/4025246479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b0108f40 msgr2=0x7f10b0109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.273+0000 7f10b70bb700 1 --2- 172.21.15.114:0/4025246479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b0108f40 0x7f10b0109350 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f109c0099c0 tx=0x7f109c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.273+0000 7f10b70bb700 1 -- 172.21.15.114:0/4025246479 shutdown_connections 2024-05-27T07:51:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.273+0000 7f10b70bb700 1 --2- 172.21.15.114:0/4025246479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10b0109890 0x7f10b0111230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.273+0000 7f10b70bb700 1 --2- 172.21.15.114:0/4025246479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b0102b20 0x7f10b0102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.273+0000 7f10b70bb700 1 --2- 172.21.15.114:0/4025246479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b0108f40 0x7f10b0109350 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.273+0000 7f10b70bb700 1 -- 172.21.15.114:0/4025246479 >> 172.21.15.114:0/4025246479 conn(0x7f10b00fe6e0 msgr2=0x7f10b0100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.273+0000 7f10b70bb700 1 -- 172.21.15.114:0/4025246479 shutdown_connections 2024-05-27T07:51:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.273+0000 7f10b70bb700 1 -- 172.21.15.114:0/4025246479 wait complete. 2024-05-27T07:51:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.274+0000 7f10b70bb700 1 Processor -- start 2024-05-27T07:51:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.274+0000 7f10b70bb700 1 -- start start 2024-05-27T07:51:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.274+0000 7f10b70bb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b0102b20 0x7f10b019d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.274+0000 7f10b70bb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b0108f40 0x7f10b019d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.274+0000 7f10b70bb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10b0109890 0x7f10b01a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.274+0000 7f10b70bb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f10b0114520 con 0x7f10b0108f40 2024-05-27T07:51:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.274+0000 7f10b70bb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01143a0 con 0x7f10b0109890 2024-05-27T07:51:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.274+0000 7f10b70bb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01146a0 con 0x7f10b0102b20 2024-05-27T07:51:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10b4e57700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b0102b20 0x7f10b019d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10affff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b0108f40 0x7f10b019d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10b4e57700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b0102b20 0x7f10b019d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:57408/0 (socket says 172.21.15.114:57408) 2024-05-27T07:51:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10b4e57700 1 -- 172.21.15.114:0/885626734 learned_addr learned my addr 172.21.15.114:0/885626734 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10b5658700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10b0109890 0x7f10b01a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10b4e57700 1 -- 172.21.15.114:0/885626734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10b0109890 msgr2=0x7f10b01a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10b4e57700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10b0109890 0x7f10b01a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10b4e57700 1 -- 172.21.15.114:0/885626734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b0108f40 msgr2=0x7f10b019d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10b4e57700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b0108f40 0x7f10b019d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10b4e57700 1 -- 172.21.15.114:0/885626734 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b01a1fa0 con 0x7f10b0102b20 2024-05-27T07:51:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10affff700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b0108f40 0x7f10b019d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10b4e57700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b0102b20 0x7f10b019d0d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f10a400e9a0 tx=0x7f10a400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10adffb700 1 -- 172.21.15.114:0/885626734 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a400ca10 con 0x7f10b0102b20 2024-05-27T07:51:30.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10adffb700 1 -- 172.21.15.114:0/885626734 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10a400cbb0 con 0x7f10b0102b20 2024-05-27T07:51:30.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10b70bb700 1 -- 172.21.15.114:0/885626734 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b01a2290 con 0x7f10b0102b20 2024-05-27T07:51:30.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.275+0000 7f10adffb700 1 -- 172.21.15.114:0/885626734 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a4018680 con 0x7f10b0102b20 2024-05-27T07:51:30.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.276+0000 7f10b70bb700 1 -- 172.21.15.114:0/885626734 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b01a9bb0 con 0x7f10b0102b20 2024-05-27T07:51:30.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.278+0000 7f10adffb700 1 -- 172.21.15.114:0/885626734 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f10a4018820 con 0x7f10b0102b20 2024-05-27T07:51:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10adffb700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f10a00b29f0 0x7f10a00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10affff700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f10a00b29f0 0x7f10a00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10c5ad0b80 1 Processor -- start 2024-05-27T07:51:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10adffb700 1 -- 172.21.15.114:0/885626734 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f10a40e2c30 con 0x7f10b0102b20 2024-05-27T07:51:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10c5ad0b80 1 -- start start 2024-05-27T07:51:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10c5ad0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 0x55e0de35cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10c5ad0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de35d070 0x55e0de356be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10c5ad0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de357530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10c5ad0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e0de284d70 con 0x55e0de35d070 2024-05-27T07:51:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10c5ad0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e0de274910 con 0x55e0de357120 2024-05-27T07:51:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10c5ad0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e0de278950 con 0x55e0de1d7e40 2024-05-27T07:51:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10affff700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f10a00b29f0 0x7f10a00b4f00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f109c000c00 tx=0x7f109c01a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10b4e57700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 0x55e0de35cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.280+0000 7f10affff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de35d070 0x55e0de356be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.281+0000 7f10b4e57700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 0x55e0de35cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:57420/0 (socket says 172.21.15.114:57420) 2024-05-27T07:51:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.281+0000 7f10b4e57700 1 -- 172.21.15.114:0/911014042 learned_addr learned my addr 172.21.15.114:0/911014042 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.281+0000 7f10b5658700 1 --2- 172.21.15.114:0/911014042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de357530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.281+0000 7f10b4e57700 1 -- 172.21.15.114:0/911014042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 msgr2=0x55e0de357530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.281+0000 7f10b4e57700 1 --2- 172.21.15.114:0/911014042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de357530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.281+0000 7f10b4e57700 1 -- 172.21.15.114:0/911014042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de35d070 msgr2=0x55e0de356be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.281+0000 7f10b4e57700 1 --2- 172.21.15.114:0/911014042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de35d070 0x55e0de356be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.281+0000 7f10b4e57700 1 -- 172.21.15.114:0/911014042 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0de21dfb0 con 0x55e0de1d7e40 2024-05-27T07:51:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.281+0000 7f10b5658700 1 --2- 172.21.15.114:0/911014042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de357530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.281+0000 7f10b4e57700 1 --2- 172.21.15.114:0/911014042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 0x55e0de35cb30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f10a40a5b80 tx=0x7f10a40b23f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=310273dfec01d452 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.281+0000 7f109affd700 1 -- 172.21.15.114:0/911014042 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a40a9b00 con 0x55e0de1d7e40 2024-05-27T07:51:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.281+0000 7f109affd700 1 -- 172.21.15.114:0/911014042 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10a40adeb0 con 0x55e0de1d7e40 2024-05-27T07:51:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.281+0000 7f109affd700 1 -- 172.21.15.114:0/911014042 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a40b24d0 con 0x55e0de1d7e40 2024-05-27T07:51:30.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.282+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/911014042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 msgr2=0x55e0de35cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:30.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.282+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/911014042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 0x55e0de35cb30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f10a40a5b80 tx=0x7f10a40b23f0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.282+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/911014042 shutdown_connections 2024-05-27T07:51:30.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.282+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/911014042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de357530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.282+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/911014042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 0x55e0de35cb30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.282+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/911014042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de35d070 0x55e0de356be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.283+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/911014042 >> 172.21.15.114:0/911014042 conn(0x55e0de1da770 msgr2=0x55e0de351d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.283+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/911014042 shutdown_connections 2024-05-27T07:51:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.283+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/911014042 wait complete. 2024-05-27T07:51:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.283+0000 7f10c5ad0b80 1 -- start start 2024-05-27T07:51:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.283+0000 7f10c5ad0b80 1 Processor -- start 2024-05-27T07:51:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.284+0000 7f10c5ad0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 0x55e0de3609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.284+0000 7f10c5ad0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de357120 0x55e0de360f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.284+0000 7f10c5ad0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de35d070 0x55e0de361460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.284+0000 7f10b4e57700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 0x55e0de3609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.284+0000 7f10affff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de357120 0x55e0de360f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.284+0000 7f10b4e57700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 0x55e0de3609e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48222/0 (socket says 172.21.15.114:48222) 2024-05-27T07:51:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.284+0000 7f10affff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de357120 0x55e0de360f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:57430/0 (socket says 172.21.15.114:57430) 2024-05-27T07:51:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.284+0000 7f10b4e57700 1 -- 172.21.15.114:0/372709344 learned_addr learned my addr 172.21.15.114:0/372709344 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.284+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/372709344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e0de17afa0 con 0x55e0de1d7e40 2024-05-27T07:51:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.284+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/372709344 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e0de1dcda0 con 0x55e0de35d070 2024-05-27T07:51:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.284+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/372709344 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e0de363b60 con 0x55e0de357120 2024-05-27T07:51:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.284+0000 7f10c5ad0b80 10 client.0 _ll_register_callbacks cb 0x55e0de20b5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.284+0000 7f10b5658700 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de35d070 0x55e0de361460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.285+0000 7f10b5658700 1 -- 172.21.15.114:0/372709344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de357120 msgr2=0x55e0de360f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.285+0000 7f10b5658700 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de357120 0x55e0de360f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.285+0000 7f10b5658700 1 -- 172.21.15.114:0/372709344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 msgr2=0x55e0de3609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.285+0000 7f10b5658700 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 0x55e0de3609e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.285+0000 7f10b5658700 1 -- 172.21.15.114:0/372709344 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0de1c48d0 con 0x55e0de35d070 2024-05-27T07:51:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.285+0000 7f10affff700 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de357120 0x55e0de360f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.285+0000 7f10b4e57700 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 0x55e0de3609e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.285+0000 7f10b5658700 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de35d070 0x55e0de361460 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f10a8007c90 tx=0x7f10a80116a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47736fb23987af57 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.285+0000 7f108ffff700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.285+0000 7f108ffff700 1 -- 172.21.15.114:0/372709344 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8004bb0 con 0x55e0de35d070 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.285+0000 7f108ffff700 1 -- 172.21.15.114:0/372709344 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f10a8004d50 con 0x55e0de35d070 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.286+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/372709344 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10a40adeb0 con 0x55e0de35d070 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.286+0000 7f108ffff700 1 -- 172.21.15.114:0/372709344 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f10a8028050 con 0x55e0de35d070 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.286+0000 7f10c5ad0b80 10 client.27575 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.286+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/372709344 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55e0de22e760 con 0x55e0de35d070 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.286+0000 7f108ffff700 1 -- 172.21.15.114:0/372709344 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10a40adeb0 con 0x55e0de35d070 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.286+0000 7f108ffff700 1 -- 172.21.15.114:0/372709344 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f10a8016420 con 0x55e0de35d070 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.286+0000 7f10c5ad0b80 10 client.27575 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.286+0000 7f10c5ad0b80 10 client.27575 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.286+0000 7f10c5ad0b80 20 client.27575 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.286+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e0de370ec0 0x55e0de373370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.286+0000 7f10c5ad0b80 4 client.27575 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.286+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/372709344 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0de286cc0 con 0x55e0de370ec0 2024-05-27T07:51:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.286+0000 7f10b4e57700 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e0de370ec0 0x55e0de373370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.287+0000 7f108ffff700 10 client.27575 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.287+0000 7f10b4e57700 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e0de370ec0 0x55e0de373370 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bce904d9d8fb9da server_cookie=8f256c40073868e0 in_seq=0 out_seq=0 2024-05-27T07:51:30.292 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1146128 9.83 MB/sec execute 462 sec latency 65.524 ms 2024-05-27T07:51:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f108ffff700 1 -- 172.21.15.114:0/372709344 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e0de286cc0 con 0x55e0de370ec0 2024-05-27T07:51:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f108ffff700 10 client.27575 handle_command_reply: tid=0 2024-05-27T07:51:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f10c5ad0b80 1 client.27575 shutdown 2024-05-27T07:51:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/372709344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de35d070 msgr2=0x55e0de361460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de35d070 0x55e0de361460 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f10a8007c90 tx=0x7f10a80116a0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/372709344 shutdown_connections 2024-05-27T07:51:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e0de370ec0 0x55e0de373370 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de35d070 0x55e0de361460 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 0x55e0de3609e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/372709344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de357120 0x55e0de360f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/372709344 >> 172.21.15.114:0/372709344 conn(0x55e0de1da770 msgr2=0x55e0de355bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f108ffff700 0 client.27575 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/372709344 shutdown_connections 2024-05-27T07:51:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/372709344 wait complete. 2024-05-27T07:51:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.289+0000 7f10c5ad0b80 20 client.27575 trim_cache size 0 max 16384 2024-05-27T07:51:30.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.306+0000 7f10c5ad0b80 1 Processor -- start 2024-05-27T07:51:30.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.306+0000 7f10c5ad0b80 1 -- start start 2024-05-27T07:51:30.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.306+0000 7f10c5ad0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 0x55e0de356be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.306+0000 7f10c5ad0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de35fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.306+0000 7f10c5ad0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de35d070 0x55e0de35aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10c5ad0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01b41b0 con 0x55e0de35d070 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10c5ad0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01b2920 con 0x55e0de357120 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10c5ad0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01b0770 con 0x55e0de1d7e40 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10b4e57700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 0x55e0de356be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10b4e57700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 0x55e0de356be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:57440/0 (socket says 172.21.15.114:57440) 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10b5658700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de35d070 0x55e0de35aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10b4e57700 1 -- 172.21.15.114:0/386961228 learned_addr learned my addr 172.21.15.114:0/386961228 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10affff700 1 --2- 172.21.15.114:0/386961228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de35fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10b4e57700 1 -- 172.21.15.114:0/386961228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 msgr2=0x55e0de35fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10b4e57700 1 --2- 172.21.15.114:0/386961228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de35fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10b4e57700 1 -- 172.21.15.114:0/386961228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de35d070 msgr2=0x55e0de35aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10b4e57700 1 --2- 172.21.15.114:0/386961228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de35d070 0x55e0de35aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10b4e57700 1 -- 172.21.15.114:0/386961228 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0de35ef50 con 0x55e0de1d7e40 2024-05-27T07:51:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10affff700 1 --2- 172.21.15.114:0/386961228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de35fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:30.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f10b4e57700 1 --2- 172.21.15.114:0/386961228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 0x55e0de356be0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f10a40a9a40 tx=0x7f10a40c0040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cc8657b1ddd5071 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:30.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.307+0000 7f109a7fc700 1 -- 172.21.15.114:0/386961228 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a40c41a0 con 0x55e0de1d7e40 2024-05-27T07:51:30.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.308+0000 7f109a7fc700 1 -- 172.21.15.114:0/386961228 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10a40adeb0 con 0x55e0de1d7e40 2024-05-27T07:51:30.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.308+0000 7f109a7fc700 1 -- 172.21.15.114:0/386961228 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a40b24d0 con 0x55e0de1d7e40 2024-05-27T07:51:30.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.309+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/386961228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 msgr2=0x55e0de356be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:30.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.309+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/386961228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 0x55e0de356be0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f10a40a9a40 tx=0x7f10a40c0040 comp rx=0 tx=0).stop 2024-05-27T07:51:30.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.309+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/386961228 shutdown_connections 2024-05-27T07:51:30.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.309+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/386961228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de35d070 0x55e0de35aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.309+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/386961228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de1d7e40 0x55e0de356be0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.309+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/386961228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de35fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.309+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/386961228 >> 172.21.15.114:0/386961228 conn(0x55e0de1da770 msgr2=0x55e0de35a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:30.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.309+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/386961228 shutdown_connections 2024-05-27T07:51:30.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.309+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/386961228 wait complete. 2024-05-27T07:51:30.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.309+0000 7f10c5ad0b80 1 -- start start 2024-05-27T07:51:30.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.309+0000 7f10c5ad0b80 1 Processor -- start 2024-05-27T07:51:30.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.310+0000 7f10c5ad0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 0x55e0de379280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.310+0000 7f10c5ad0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de368df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.310+0000 7f10c5ad0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de35d070 0x55e0de369330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.310+0000 7f10c5ad0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01b0de0 con 0x55e0de1d7e40 2024-05-27T07:51:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.310+0000 7f10c5ad0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01b2250 con 0x55e0de357120 2024-05-27T07:51:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.310+0000 7f10c5ad0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01b14b0 con 0x55e0de35d070 2024-05-27T07:51:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.310+0000 7f10affff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de368df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.310+0000 7f10b4e57700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 0x55e0de379280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.310+0000 7f10affff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de368df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:32890/0 (socket says 172.21.15.114:32890) 2024-05-27T07:51:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.310+0000 7f10affff700 1 -- 172.21.15.114:0/1920737955 learned_addr learned my addr 172.21.15.114:0/1920737955 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.310+0000 7f10b4e57700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 0x55e0de379280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48244/0 (socket says 172.21.15.114:48244) 2024-05-27T07:51:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.311+0000 7f10c5ad0b80 10 client.0 _ll_register_callbacks cb 0x55e0de373a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.311+0000 7f10b5658700 1 --2- 172.21.15.114:0/1920737955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de35d070 0x55e0de369330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.311+0000 7f10affff700 1 -- 172.21.15.114:0/1920737955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de35d070 msgr2=0x55e0de369330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.311+0000 7f10affff700 1 --2- 172.21.15.114:0/1920737955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de35d070 0x55e0de369330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.311+0000 7f10affff700 1 -- 172.21.15.114:0/1920737955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 msgr2=0x55e0de379280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:30.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.311+0000 7f10affff700 1 --2- 172.21.15.114:0/1920737955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 0x55e0de379280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.311+0000 7f10affff700 1 -- 172.21.15.114:0/1920737955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0de377b60 con 0x55e0de357120 2024-05-27T07:51:30.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.311+0000 7f10b4e57700 1 --2- 172.21.15.114:0/1920737955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 0x55e0de379280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:30.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.312+0000 7f10affff700 1 --2- 172.21.15.114:0/1920737955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de368df0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f109c0099c0 tx=0x7f109c009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=813ea45dea39aba4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:30.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.312+0000 7f108ffff700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:51:30.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.312+0000 7f108ffff700 1 -- 172.21.15.114:0/1920737955 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c036060 con 0x55e0de357120 2024-05-27T07:51:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.312+0000 7f108ffff700 1 -- 172.21.15.114:0/1920737955 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f109c00f660 con 0x55e0de357120 2024-05-27T07:51:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.312+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/1920737955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10a40adeb0 con 0x55e0de357120 2024-05-27T07:51:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.312+0000 7f108ffff700 1 -- 172.21.15.114:0/1920737955 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f109c00f860 con 0x55e0de357120 2024-05-27T07:51:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.312+0000 7f10c5ad0b80 10 client.27581 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.312+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/1920737955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55e0de3dab00 con 0x55e0de357120 2024-05-27T07:51:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.312+0000 7f108ffff700 1 -- 172.21.15.114:0/1920737955 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10a40adeb0 con 0x55e0de357120 2024-05-27T07:51:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.312+0000 7f108ffff700 1 -- 172.21.15.114:0/1920737955 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f109c030070 con 0x55e0de357120 2024-05-27T07:51:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.313+0000 7f10c5ad0b80 10 client.27581 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.313+0000 7f10c5ad0b80 10 client.27581 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.313+0000 7f10c5ad0b80 20 client.27581 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.313+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/1920737955 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e0de3db240 0x55e0de3dd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.313+0000 7f10c5ad0b80 4 client.27581 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:51:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.313+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/1920737955 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e0de2804a0 con 0x55e0de3db240 2024-05-27T07:51:30.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.313+0000 7f10b4e57700 1 --2- 172.21.15.114:0/1920737955 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e0de3db240 0x55e0de3dd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:30.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.313+0000 7f108ffff700 10 client.27581 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:30.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.314+0000 7f10b4e57700 1 --2- 172.21.15.114:0/1920737955 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e0de3db240 0x55e0de3dd660 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27431b4fb0603f0d server_cookie=1be01937bbaf506f in_seq=0 out_seq=0 2024-05-27T07:51:30.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f108ffff700 1 -- 172.21.15.114:0/1920737955 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e0de2804a0 con 0x55e0de3db240 2024-05-27T07:51:30.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f108ffff700 10 client.27581 handle_command_reply: tid=0 2024-05-27T07:51:30.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f10c5ad0b80 1 client.27581 shutdown 2024-05-27T07:51:30.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/1920737955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 msgr2=0x55e0de368df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:30.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/1920737955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de368df0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f109c0099c0 tx=0x7f109c009310 comp rx=0 tx=0).stop 2024-05-27T07:51:30.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/1920737955 shutdown_connections 2024-05-27T07:51:30.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/1920737955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0de35d070 0x55e0de369330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/1920737955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e0de1d7e40 0x55e0de379280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/1920737955 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e0de3db240 0x55e0de3dd660 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f10c5ad0b80 1 --2- 172.21.15.114:0/1920737955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e0de357120 0x55e0de368df0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f108ffff700 0 client.27581 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:30.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/1920737955 >> 172.21.15.114:0/1920737955 conn(0x55e0de1da770 msgr2=0x55e0de35fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:30.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/1920737955 shutdown_connections 2024-05-27T07:51:30.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f10c5ad0b80 1 -- 172.21.15.114:0/1920737955 wait complete. 2024-05-27T07:51:30.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.315+0000 7f10c5ad0b80 20 client.27581 trim_cache size 0 max 16384 2024-05-27T07:51:30.319 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:51:30.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.316+0000 7f10b70bb700 1 -- 172.21.15.114:0/885626734 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f10a00b29f0 msgr2=0x7f10a00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:30.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.316+0000 7f10b70bb700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f10a00b29f0 0x7f10a00b4f00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f109c000c00 tx=0x7f109c01a040 comp rx=0 tx=0).stop 2024-05-27T07:51:30.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.316+0000 7f10b70bb700 1 -- 172.21.15.114:0/885626734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b0102b20 msgr2=0x7f10b019d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:30.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.316+0000 7f10b70bb700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b0102b20 0x7f10b019d0d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f10a400e9a0 tx=0x7f10a400ecb0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.316+0000 7f10b70bb700 1 -- 172.21.15.114:0/885626734 shutdown_connections 2024-05-27T07:51:30.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.316+0000 7f10b70bb700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f10a00b29f0 0x7f10a00b4f00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.316+0000 7f10b70bb700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10b0109890 0x7f10b01a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.316+0000 7f10b70bb700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b0102b20 0x7f10b019d0d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.316+0000 7f10b70bb700 1 --2- 172.21.15.114:0/885626734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b0108f40 0x7f10b019d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:30.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.316+0000 7f10b70bb700 1 -- 172.21.15.114:0/885626734 >> 172.21.15.114:0/885626734 conn(0x7f10b00fe6e0 msgr2=0x7f10b01087c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:30.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.316+0000 7f10b70bb700 1 -- 172.21.15.114:0/885626734 shutdown_connections 2024-05-27T07:51:30.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:30.316+0000 7f10b70bb700 1 -- 172.21.15.114:0/885626734 wait complete. 2024-05-27T07:51:31.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1147796 9.83 MB/sec execute 463 sec latency 15.457 ms 2024-05-27T07:51:31.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:31 smithi114 ceph-mon[32038]: pgmap v392: 129 pgs: 129 active+clean; 63 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 299 op/s 2024-05-27T07:51:31.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:31 smithi163 ceph-mon[36451]: pgmap v392: 129 pgs: 129 active+clean; 63 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 299 op/s 2024-05-27T07:51:31.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:31 smithi186 ceph-mon[36647]: pgmap v392: 129 pgs: 129 active+clean; 63 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 299 op/s 2024-05-27T07:51:32.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1149735 9.83 MB/sec execute 464 sec latency 35.536 ms 2024-05-27T07:51:32.699 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:51:32.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:51:32.955 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:33.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1151157 9.83 MB/sec execute 465 sec latency 48.339 ms 2024-05-27T07:51:33.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:33 smithi163 ceph-mon[36451]: pgmap v393: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-05-27T07:51:33.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:33 smithi186 ceph-mon[36647]: pgmap v393: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-05-27T07:51:34.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:33 smithi114 ceph-mon[32038]: pgmap v393: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-05-27T07:51:34.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1152725 9.82 MB/sec execute 466 sec latency 29.713 ms 2024-05-27T07:51:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.013+0000 7f87fa227700 1 -- 172.21.15.114:0/1855286475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f4108740 msgr2=0x7f87f41100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.013+0000 7f87fa227700 1 --2- 172.21.15.114:0/1855286475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f4108740 0x7f87f41100a0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00b0d0 tx=0x7f87ec00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.013+0000 7f87fa227700 1 -- 172.21.15.114:0/1855286475 shutdown_connections 2024-05-27T07:51:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.013+0000 7f87fa227700 1 --2- 172.21.15.114:0/1855286475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f4108740 0x7f87f41100a0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.013+0000 7f87fa227700 1 --2- 172.21.15.114:0/1855286475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87f4103c50 0x7f87f4104060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.013+0000 7f87fa227700 1 --2- 172.21.15.114:0/1855286475 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87f4107df0 0x7f87f4108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.013+0000 7f87fa227700 1 -- 172.21.15.114:0/1855286475 >> 172.21.15.114:0/1855286475 conn(0x7f87f40fe6e0 msgr2=0x7f87f4100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.014+0000 7f87fa227700 1 -- 172.21.15.114:0/1855286475 shutdown_connections 2024-05-27T07:51:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.014+0000 7f87fa227700 1 -- 172.21.15.114:0/1855286475 wait complete. 2024-05-27T07:51:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.014+0000 7f87fa227700 1 Processor -- start 2024-05-27T07:51:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87fa227700 1 -- start start 2024-05-27T07:51:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87fa227700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87f4103c50 0x7f87f419ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87fa227700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87f4107df0 0x7f87f419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87fa227700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f4108740 0x7f87f419f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87fa227700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f87f4113370 con 0x7f87f4108740 2024-05-27T07:51:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87fa227700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f87f41131f0 con 0x7f87f4107df0 2024-05-27T07:51:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87fa227700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87f41134f0 con 0x7f87f4103c50 2024-05-27T07:51:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87f37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87f4103c50 0x7f87f419ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87f37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87f4103c50 0x7f87f419ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35400/0 (socket says 172.21.15.114:35400) 2024-05-27T07:51:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87f3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f4108740 0x7f87f419f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87f37fe700 1 -- 172.21.15.114:0/847834953 learned_addr learned my addr 172.21.15.114:0/847834953 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87f2ffd700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87f4107df0 0x7f87f419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87f37fe700 1 -- 172.21.15.114:0/847834953 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87f4107df0 msgr2=0x7f87f419b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87f37fe700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87f4107df0 0x7f87f419b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87f37fe700 1 -- 172.21.15.114:0/847834953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f4108740 msgr2=0x7f87f419f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87f37fe700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f4108740 0x7f87f419f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87f37fe700 1 -- 172.21.15.114:0/847834953 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f419fc40 con 0x7f87f4103c50 2024-05-27T07:51:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.015+0000 7f87f2ffd700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87f4107df0 0x7f87f419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.016+0000 7f87f3fff700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f4108740 0x7f87f419f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.016+0000 7f87f37fe700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87f4103c50 0x7f87f419ad70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f87e400e9a0 tx=0x7f87e400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.016+0000 7f87f0ff9700 1 -- 172.21.15.114:0/847834953 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e400ca10 con 0x7f87f4103c50 2024-05-27T07:51:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.016+0000 7f87f0ff9700 1 -- 172.21.15.114:0/847834953 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87e400cbb0 con 0x7f87f4103c50 2024-05-27T07:51:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.016+0000 7f87fa227700 1 -- 172.21.15.114:0/847834953 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87f419ff30 con 0x7f87f4103c50 2024-05-27T07:51:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.016+0000 7f87f0ff9700 1 -- 172.21.15.114:0/847834953 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e4018680 con 0x7f87f4103c50 2024-05-27T07:51:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.016+0000 7f87fa227700 1 -- 172.21.15.114:0/847834953 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87f41a7850 con 0x7f87f4103c50 2024-05-27T07:51:35.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.019+0000 7f87f0ff9700 1 -- 172.21.15.114:0/847834953 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f87e4010430 con 0x7f87f4103c50 2024-05-27T07:51:35.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.020+0000 7f8808c3cb80 1 Processor -- start 2024-05-27T07:51:35.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.020+0000 7f87f0ff9700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87dc0b29f0 0x7f87dc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.020+0000 7f8808c3cb80 1 -- start start 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.020+0000 7f87f2ffd700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87dc0b29f0 0x7f87dc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.020+0000 7f87f0ff9700 1 -- 172.21.15.114:0/847834953 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f87e40e2b60 con 0x7f87f4103c50 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.020+0000 7f8808c3cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf1bdec0 0x5566cf342bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.020+0000 7f8808c3cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf33cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.020+0000 7f8808c3cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf33d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.020+0000 7f8808c3cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5566cf26adf0 con 0x5566cf3430f0 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.020+0000 7f8808c3cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5566cf25a990 con 0x5566cf33d1a0 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.020+0000 7f8808c3cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5566cf25e9d0 con 0x5566cf1bdec0 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf1bdec0 0x5566cf342bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f3fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf33d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f3fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf33d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:56970/0 (socket says 172.21.15.114:56970) 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf1bdec0 0x5566cf342bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35414/0 (socket says 172.21.15.114:35414) 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f3fff700 1 -- 172.21.15.114:0/4109498364 learned_addr learned my addr 172.21.15.114:0/4109498364 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:35.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f2ffd700 1 --2- 172.21.15.114:0/4109498364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf33cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f2ffd700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87dc0b29f0 0x7f87dc0b4f00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f87f41097c0 tx=0x7f87e8009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f3fff700 1 -- 172.21.15.114:0/4109498364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf1bdec0 msgr2=0x5566cf342bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f3fff700 1 --2- 172.21.15.114:0/4109498364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf1bdec0 0x5566cf342bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f3fff700 1 -- 172.21.15.114:0/4109498364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 msgr2=0x5566cf33cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f3fff700 1 --2- 172.21.15.114:0/4109498364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf33cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f3fff700 1 -- 172.21.15.114:0/4109498364 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566cf204030 con 0x5566cf33d1a0 2024-05-27T07:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f37fe700 1 --2- 172.21.15.114:0/4109498364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf1bdec0 0x5566cf342bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87f3fff700 1 --2- 172.21.15.114:0/4109498364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf33d5b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f87ec005af0 tx=0x7f87ec005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91c433bb893247c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87d9ffb700 1 -- 172.21.15.114:0/4109498364 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec00be50 con 0x5566cf33d1a0 2024-05-27T07:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87d9ffb700 1 -- 172.21.15.114:0/4109498364 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87ec0094e0 con 0x5566cf33d1a0 2024-05-27T07:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.021+0000 7f87d9ffb700 1 -- 172.21.15.114:0/4109498364 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0075c0 con 0x5566cf33d1a0 2024-05-27T07:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.022+0000 7f8808c3cb80 1 -- 172.21.15.114:0/4109498364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 msgr2=0x5566cf33d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.022+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/4109498364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf33d5b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f87ec005af0 tx=0x7f87ec005b20 comp rx=0 tx=0).stop 2024-05-27T07:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.022+0000 7f8808c3cb80 1 -- 172.21.15.114:0/4109498364 shutdown_connections 2024-05-27T07:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.022+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/4109498364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf33d5b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.022+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/4109498364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf1bdec0 0x5566cf342bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.022+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/4109498364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf33cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.022+0000 7f8808c3cb80 1 -- 172.21.15.114:0/4109498364 >> 172.21.15.114:0/4109498364 conn(0x5566cf1c07f0 msgr2=0x5566cf337e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.022+0000 7f8808c3cb80 1 -- 172.21.15.114:0/4109498364 shutdown_connections 2024-05-27T07:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.023+0000 7f8808c3cb80 1 -- 172.21.15.114:0/4109498364 wait complete. 2024-05-27T07:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.023+0000 7f8808c3cb80 1 -- start start 2024-05-27T07:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.023+0000 7f8808c3cb80 1 Processor -- start 2024-05-27T07:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.023+0000 7f8808c3cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf1bdec0 0x5566cf3468c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.023+0000 7f8808c3cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf346e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.023+0000 7f8808c3cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf347340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.023+0000 7f8808c3cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5566cf161020 con 0x5566cf3430f0 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.023+0000 7f8808c3cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5566cf1c2e20 con 0x5566cf33d1a0 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.023+0000 7f8808c3cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5566cf34df90 con 0x5566cf1bdec0 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f8808c3cb80 10 client.0 _ll_register_callbacks cb 0x5566cf1f1630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87f3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf347340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87f37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf1bdec0 0x5566cf3468c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87f3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf347340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49190/0 (socket says 172.21.15.114:49190) 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87f3fff700 1 -- 172.21.15.114:0/1726369011 learned_addr learned my addr 172.21.15.114:0/1726369011 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87f2ffd700 1 --2- 172.21.15.114:0/1726369011 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf346e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87f3fff700 1 -- 172.21.15.114:0/1726369011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf1bdec0 msgr2=0x5566cf3468c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87f3fff700 1 --2- 172.21.15.114:0/1726369011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf1bdec0 0x5566cf3468c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87f3fff700 1 -- 172.21.15.114:0/1726369011 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 msgr2=0x5566cf346e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87f3fff700 1 --2- 172.21.15.114:0/1726369011 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf346e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87f3fff700 1 -- 172.21.15.114:0/1726369011 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566cf1aa950 con 0x5566cf3430f0 2024-05-27T07:51:35.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87f3fff700 1 --2- 172.21.15.114:0/1726369011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf347340 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f87ec005a10 tx=0x7f87ec005370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4cb5a9b923e771 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:35.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87d2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:35.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87d2ffd700 1 -- 172.21.15.114:0/1726369011 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec02d070 con 0x5566cf3430f0 2024-05-27T07:51:35.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f87d2ffd700 1 -- 172.21.15.114:0/1726369011 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f87ec003a00 con 0x5566cf3430f0 2024-05-27T07:51:35.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.024+0000 7f8808c3cb80 1 -- 172.21.15.114:0/1726369011 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5566cf26cd40 con 0x5566cf3430f0 2024-05-27T07:51:35.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.025+0000 7f87d2ffd700 1 -- 172.21.15.114:0/1726369011 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f87ec039050 con 0x5566cf3430f0 2024-05-27T07:51:35.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.025+0000 7f8808c3cb80 10 client.18168 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:35.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.025+0000 7f8808c3cb80 1 -- 172.21.15.114:0/1726369011 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5566cf2147e0 con 0x5566cf3430f0 2024-05-27T07:51:35.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.025+0000 7f87d2ffd700 1 -- 172.21.15.114:0/1726369011 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5566cf26cd40 con 0x5566cf3430f0 2024-05-27T07:51:35.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.025+0000 7f87d2ffd700 1 -- 172.21.15.114:0/1726369011 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f87ec011070 con 0x5566cf3430f0 2024-05-27T07:51:35.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.025+0000 7f8808c3cb80 10 client.18168 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:35.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.025+0000 7f8808c3cb80 10 client.18168 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:35.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.025+0000 7f8808c3cb80 20 client.18168 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:35.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.025+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/1726369011 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5566cf35adb0 0x5566cf35d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.025+0000 7f8808c3cb80 4 client.18168 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:35.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.025+0000 7f8808c3cb80 1 -- 172.21.15.114:0/1726369011 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566cf266520 con 0x5566cf35adb0 2024-05-27T07:51:35.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.025+0000 7f87f37fe700 1 --2- 172.21.15.114:0/1726369011 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5566cf35adb0 0x5566cf35d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.026+0000 7f87d2ffd700 10 client.18168 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:35.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.026+0000 7f87f37fe700 1 --2- 172.21.15.114:0/1726369011 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5566cf35adb0 0x5566cf35d210 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2f2346e0ada2621 server_cookie=be5ae9d18bf0dd27 in_seq=0 out_seq=0 2024-05-27T07:51:35.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.027+0000 7f87d2ffd700 1 -- 172.21.15.114:0/1726369011 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5566cf266520 con 0x5566cf35adb0 2024-05-27T07:51:35.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.027+0000 7f87d2ffd700 10 client.18168 handle_command_reply: tid=0 2024-05-27T07:51:35.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.027+0000 7f8808c3cb80 1 client.18168 shutdown 2024-05-27T07:51:35.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.027+0000 7f8808c3cb80 1 -- 172.21.15.114:0/1726369011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 msgr2=0x5566cf347340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.027+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/1726369011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf347340 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f87ec005a10 tx=0x7f87ec005370 comp rx=0 tx=0).stop 2024-05-27T07:51:35.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.028+0000 7f8808c3cb80 1 -- 172.21.15.114:0/1726369011 shutdown_connections 2024-05-27T07:51:35.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.028+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/1726369011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf347340 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.028+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/1726369011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf1bdec0 0x5566cf3468c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.028+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/1726369011 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5566cf35adb0 0x5566cf35d210 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.028+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/1726369011 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf346e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.028+0000 7f8808c3cb80 1 -- 172.21.15.114:0/1726369011 >> 172.21.15.114:0/1726369011 conn(0x5566cf1c07f0 msgr2=0x5566cf33bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:35.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.028+0000 7f87d2ffd700 0 client.18168 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:35.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.028+0000 7f8808c3cb80 1 -- 172.21.15.114:0/1726369011 shutdown_connections 2024-05-27T07:51:35.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.028+0000 7f8808c3cb80 1 -- 172.21.15.114:0/1726369011 wait complete. 2024-05-27T07:51:35.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.028+0000 7f8808c3cb80 20 client.18168 trim_cache size 0 max 16384 2024-05-27T07:51:35.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f8808c3cb80 1 Processor -- start 2024-05-27T07:51:35.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f8808c3cb80 1 -- start start 2024-05-27T07:51:35.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f8808c3cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf1bdec0 0x5566cf33cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f8808c3cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf33d1a0 0x5566cf357030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f8808c3cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf331c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f8808c3cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f87f41afbe0 con 0x5566cf3430f0 2024-05-27T07:51:35.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f8808c3cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f87f41ae350 con 0x5566cf1bdec0 2024-05-27T07:51:35.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f8808c3cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87f41ac1a0 con 0x5566cf33d1a0 2024-05-27T07:51:35.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f87f3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf331c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f87f3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf331c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49202/0 (socket says 172.21.15.114:49202) 2024-05-27T07:51:35.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f87f3fff700 1 -- 172.21.15.114:0/1062872572 learned_addr learned my addr 172.21.15.114:0/1062872572 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:35.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f87f2ffd700 1 --2- 172.21.15.114:0/1062872572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf33d1a0 0x5566cf357030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f87f37fe700 1 --2- 172.21.15.114:0/1062872572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf1bdec0 0x5566cf33cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f87f3fff700 1 -- 172.21.15.114:0/1062872572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf33d1a0 msgr2=0x5566cf357030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f87f3fff700 1 --2- 172.21.15.114:0/1062872572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf33d1a0 0x5566cf357030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f87f3fff700 1 -- 172.21.15.114:0/1062872572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf1bdec0 msgr2=0x5566cf33cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f87f3fff700 1 --2- 172.21.15.114:0/1062872572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf1bdec0 0x5566cf33cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.050+0000 7f87f3fff700 1 -- 172.21.15.114:0/1062872572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566cf344e50 con 0x5566cf3430f0 2024-05-27T07:51:35.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.051+0000 7f87f3fff700 1 --2- 172.21.15.114:0/1062872572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf331c80 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f87ec000f80 tx=0x7f87ec00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d182d29c81bf8d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:35.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.051+0000 7f87d97fa700 1 -- 172.21.15.114:0/1062872572 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec02d040 con 0x5566cf3430f0 2024-05-27T07:51:35.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.051+0000 7f87d97fa700 1 -- 172.21.15.114:0/1062872572 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87ec005c60 con 0x5566cf3430f0 2024-05-27T07:51:35.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.051+0000 7f87d97fa700 1 -- 172.21.15.114:0/1062872572 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0079e0 con 0x5566cf3430f0 2024-05-27T07:51:35.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.052+0000 7f8808c3cb80 1 -- 172.21.15.114:0/1062872572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 msgr2=0x5566cf331c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.052+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/1062872572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf331c80 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f87ec000f80 tx=0x7f87ec00bf10 comp rx=0 tx=0).stop 2024-05-27T07:51:35.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.052+0000 7f8808c3cb80 1 -- 172.21.15.114:0/1062872572 shutdown_connections 2024-05-27T07:51:35.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.052+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/1062872572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf3430f0 0x5566cf331c80 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.052+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/1062872572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf1bdec0 0x5566cf33cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.052+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/1062872572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf33d1a0 0x5566cf357030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.052+0000 7f8808c3cb80 1 -- 172.21.15.114:0/1062872572 >> 172.21.15.114:0/1062872572 conn(0x5566cf1c07f0 msgr2=0x5566cf331570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.052+0000 7f8808c3cb80 1 -- 172.21.15.114:0/1062872572 shutdown_connections 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.052+0000 7f8808c3cb80 1 -- 172.21.15.114:0/1062872572 wait complete. 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.052+0000 7f8808c3cb80 1 -- start start 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.052+0000 7f8808c3cb80 1 Processor -- start 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f8808c3cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf1bdec0 0x5566cf353270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f8808c3cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf3537d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f8808c3cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf3430f0 0x5566cf3507d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f8808c3cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f87f41ac810 con 0x5566cf1bdec0 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f8808c3cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f87f41adc80 con 0x5566cf33d1a0 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f8808c3cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87f41acee0 con 0x5566cf3430f0 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f8808c3cb80 10 client.0 _ll_register_callbacks cb 0x5566cf1f1630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f87f37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf1bdec0 0x5566cf353270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f87f37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf1bdec0 0x5566cf353270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49214/0 (socket says 172.21.15.114:49214) 2024-05-27T07:51:35.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f87f37fe700 1 -- 172.21.15.114:0/897381345 learned_addr learned my addr 172.21.15.114:0/897381345 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:35.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f87f3fff700 1 --2- 172.21.15.114:0/897381345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf3430f0 0x5566cf3507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f87f2ffd700 1 --2- 172.21.15.114:0/897381345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf3537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f87f37fe700 1 -- 172.21.15.114:0/897381345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf3430f0 msgr2=0x5566cf3507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f87f37fe700 1 --2- 172.21.15.114:0/897381345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf3430f0 0x5566cf3507d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f87f37fe700 1 -- 172.21.15.114:0/897381345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 msgr2=0x5566cf3537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f87f37fe700 1 --2- 172.21.15.114:0/897381345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf3537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f87f37fe700 1 -- 172.21.15.114:0/897381345 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566cf350f00 con 0x5566cf1bdec0 2024-05-27T07:51:35.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f87f37fe700 1 --2- 172.21.15.114:0/897381345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf1bdec0 0x5566cf353270 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f87e40a6d00 tx=0x7f87e40cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=243bc2658dc0ca4e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:35.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.053+0000 7f87d2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:35.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f87d2ffd700 1 -- 172.21.15.114:0/897381345 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e40ade40 con 0x5566cf1bdec0 2024-05-27T07:51:35.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f87d2ffd700 1 -- 172.21.15.114:0/897381345 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f87e40a9f00 con 0x5566cf1bdec0 2024-05-27T07:51:35.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f8808c3cb80 1 -- 172.21.15.114:0/897381345 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87ec005c60 con 0x5566cf1bdec0 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f87d2ffd700 1 -- 172.21.15.114:0/897381345 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f87e40b28f0 con 0x5566cf1bdec0 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f8808c3cb80 10 client.18174 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f8808c3cb80 1 -- 172.21.15.114:0/897381345 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5566cf3c5250 con 0x5566cf1bdec0 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f87d2ffd700 1 -- 172.21.15.114:0/897381345 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87ec005c60 con 0x5566cf1bdec0 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f87d2ffd700 1 -- 172.21.15.114:0/897381345 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f87e40cc0b0 con 0x5566cf1bdec0 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f8808c3cb80 10 client.18174 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f8808c3cb80 10 client.18174 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f8808c3cb80 20 client.18174 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/897381345 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5566cf3c5910 0x5566cf3c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f8808c3cb80 4 client.18174 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f8808c3cb80 1 -- 172.21.15.114:0/897381345 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5566cf3476f0 con 0x5566cf3c5910 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.054+0000 7f87f2ffd700 1 --2- 172.21.15.114:0/897381345 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5566cf3c5910 0x5566cf3c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.055+0000 7f87d2ffd700 10 client.18174 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:35.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.055+0000 7f87f2ffd700 1 --2- 172.21.15.114:0/897381345 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5566cf3c5910 0x5566cf3c7d30 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9936462b0f9b772 server_cookie=a6b2f49642e62d8a in_seq=0 out_seq=0 2024-05-27T07:51:35.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.056+0000 7f87d2ffd700 1 -- 172.21.15.114:0/897381345 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5566cf3476f0 con 0x5566cf3c5910 2024-05-27T07:51:35.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.056+0000 7f87d2ffd700 10 client.18174 handle_command_reply: tid=0 2024-05-27T07:51:35.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.056+0000 7f8808c3cb80 1 client.18174 shutdown 2024-05-27T07:51:35.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.057+0000 7f8808c3cb80 1 -- 172.21.15.114:0/897381345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf1bdec0 msgr2=0x5566cf353270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.057+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/897381345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf1bdec0 0x5566cf353270 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f87e40a6d00 tx=0x7f87e40cc040 comp rx=0 tx=0).stop 2024-05-27T07:51:35.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.057+0000 7f8808c3cb80 1 -- 172.21.15.114:0/897381345 shutdown_connections 2024-05-27T07:51:35.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.057+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/897381345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5566cf3430f0 0x5566cf3507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.057+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/897381345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5566cf1bdec0 0x5566cf353270 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.057+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/897381345 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5566cf3c5910 0x5566cf3c7d30 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.057+0000 7f8808c3cb80 1 --2- 172.21.15.114:0/897381345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5566cf33d1a0 0x5566cf3537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.057+0000 7f87d2ffd700 0 client.18174 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:35.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.057+0000 7f8808c3cb80 1 -- 172.21.15.114:0/897381345 >> 172.21.15.114:0/897381345 conn(0x5566cf1c07f0 msgr2=0x5566cf3437d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:35.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.057+0000 7f8808c3cb80 1 -- 172.21.15.114:0/897381345 shutdown_connections 2024-05-27T07:51:35.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.057+0000 7f8808c3cb80 1 -- 172.21.15.114:0/897381345 wait complete. 2024-05-27T07:51:35.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.057+0000 7f8808c3cb80 20 client.18174 trim_cache size 0 max 16384 2024-05-27T07:51:35.062 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:51:35.062 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:51:35.062 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "c391343b-9357-46cb-a874-7427bff7f838", 2024-05-27T07:51:35.062 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:51:35.062 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:51:35.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.058+0000 7f87fa227700 1 -- 172.21.15.114:0/847834953 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87dc0b29f0 msgr2=0x7f87dc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.058+0000 7f87fa227700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87dc0b29f0 0x7f87dc0b4f00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f87f41097c0 tx=0x7f87e8009260 comp rx=0 tx=0).stop 2024-05-27T07:51:35.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.058+0000 7f87fa227700 1 -- 172.21.15.114:0/847834953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87f4103c50 msgr2=0x7f87f419ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.058+0000 7f87fa227700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87f4103c50 0x7f87f419ad70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f87e400e9a0 tx=0x7f87e400ecb0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.058+0000 7f87fa227700 1 -- 172.21.15.114:0/847834953 shutdown_connections 2024-05-27T07:51:35.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.058+0000 7f87fa227700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f4108740 0x7f87f419f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.058+0000 7f87fa227700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87f4103c50 0x7f87f419ad70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.058+0000 7f87fa227700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87dc0b29f0 0x7f87dc0b4f00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.058+0000 7f87fa227700 1 --2- 172.21.15.114:0/847834953 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87f4107df0 0x7f87f419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.058+0000 7f87fa227700 1 -- 172.21.15.114:0/847834953 >> 172.21.15.114:0/847834953 conn(0x7f87f40fe6e0 msgr2=0x7f87f4102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:35.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.058+0000 7f87fa227700 1 -- 172.21.15.114:0/847834953 shutdown_connections 2024-05-27T07:51:35.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:35.058+0000 7f87fa227700 1 -- 172.21.15.114:0/847834953 wait complete. 2024-05-27T07:51:35.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1154617 9.83 MB/sec execute 467 sec latency 36.340 ms 2024-05-27T07:51:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.601+0000 7f866e734640 1 -- 172.21.15.114:0/4027237028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8668109030 msgr2=0x7f8668102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.601+0000 7f866e734640 1 --2- 172.21.15.114:0/4027237028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8668109030 0x7f8668102c70 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f8650009900 tx=0x7f865002a410 comp rx=0 tx=0).stop 2024-05-27T07:51:35.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.601+0000 7f866e734640 1 -- 172.21.15.114:0/4027237028 shutdown_connections 2024-05-27T07:51:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.601+0000 7f866e734640 1 --2- 172.21.15.114:0/4027237028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f86681032e0 0x7f866810f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.601+0000 7f866e734640 1 --2- 172.21.15.114:0/4027237028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8668109030 0x7f8668102c70 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.601+0000 7f866e734640 1 --2- 172.21.15.114:0/4027237028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8668108b90 0x7f8668102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.601+0000 7f866e734640 1 -- 172.21.15.114:0/4027237028 >> 172.21.15.114:0/4027237028 conn(0x7f86680782d0 msgr2=0x7f8668078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.601+0000 7f866e734640 1 -- 172.21.15.114:0/4027237028 shutdown_connections 2024-05-27T07:51:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.601+0000 7f866e734640 1 -- 172.21.15.114:0/4027237028 wait complete. 2024-05-27T07:51:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866e734640 1 Processor -- start 2024-05-27T07:51:35.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866e734640 1 -- start start 2024-05-27T07:51:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866e734640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86681032e0 0x7f86681a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866e734640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8668108b90 0x7f86681a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866e734640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8668109030 0x7f866819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866e734640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8668112d80 con 0x7f8668109030 2024-05-27T07:51:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866e734640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8668112c00 con 0x7f8668108b90 2024-05-27T07:51:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866e734640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8668112f00 con 0x7f86681032e0 2024-05-27T07:51:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866ccd1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8668109030 0x7f866819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866ccd1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8668109030 0x7f866819a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49232/0 (socket says 172.21.15.114:49232) 2024-05-27T07:51:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866ccd1640 1 -- 172.21.15.114:0/1766288813 learned_addr learned my addr 172.21.15.114:0/1766288813 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f8667fff640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86681032e0 0x7f86681a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f865ffff640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8668108b90 0x7f86681a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866ccd1640 1 -- 172.21.15.114:0/1766288813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86681032e0 msgr2=0x7f86681a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866ccd1640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86681032e0 0x7f86681a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866ccd1640 1 -- 172.21.15.114:0/1766288813 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8668108b90 msgr2=0x7f86681a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866ccd1640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8668108b90 0x7f86681a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.602+0000 7f866ccd1640 1 -- 172.21.15.114:0/1766288813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f866819aea0 con 0x7f8668109030 2024-05-27T07:51:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.603+0000 7f866ccd1640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8668109030 0x7f866819a770 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f866000eb10 tx=0x7f866000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.603+0000 7f8665ffb640 1 -- 172.21.15.114:0/1766288813 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8660004b20 con 0x7f8668109030 2024-05-27T07:51:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.603+0000 7f8665ffb640 1 -- 172.21.15.114:0/1766288813 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8660004cc0 con 0x7f8668109030 2024-05-27T07:51:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.603+0000 7f8665ffb640 1 -- 172.21.15.114:0/1766288813 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8660010670 con 0x7f8668109030 2024-05-27T07:51:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.603+0000 7f866e734640 1 -- 172.21.15.114:0/1766288813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f866819b130 con 0x7f8668109030 2024-05-27T07:51:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.603+0000 7f866e734640 1 -- 172.21.15.114:0/1766288813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86681a7390 con 0x7f8668109030 2024-05-27T07:51:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.605+0000 7f8665ffb640 1 -- 172.21.15.114:0/1766288813 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f8660005020 con 0x7f8668109030 2024-05-27T07:51:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f86755e4740 1 Processor -- start 2024-05-27T07:51:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f86755e4740 1 -- start start 2024-05-27T07:51:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f86755e4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c3f1b0 0x5576a2fe03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f86755e4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 0x5576a2fe0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f86755e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a2fe0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f86755e4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5576a2f52c30 con 0x5576a2c162f0 2024-05-27T07:51:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f86755e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5576a2f427d0 con 0x5576a2c1e4d0 2024-05-27T07:51:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f86755e4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5576a2f46810 con 0x5576a2c3f1b0 2024-05-27T07:51:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f8667fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 0x5576a2fe0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f8667fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 0x5576a2fe0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49238/0 (socket says 172.21.15.114:49238) 2024-05-27T07:51:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f8667fff640 1 -- 172.21.15.114:0/1244252318 learned_addr learned my addr 172.21.15.114:0/1244252318 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f865ffff640 1 --2- 172.21.15.114:0/1244252318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a2fe0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f865ffff640 1 --2- 172.21.15.114:0/1244252318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c3f1b0 0x5576a2fe03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f8667fff640 1 -- 172.21.15.114:0/1244252318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c3f1b0 msgr2=0x5576a2fe03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f8667fff640 1 --2- 172.21.15.114:0/1244252318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c3f1b0 0x5576a2fe03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f8667fff640 1 -- 172.21.15.114:0/1244252318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 msgr2=0x5576a2fe0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f8667fff640 1 --2- 172.21.15.114:0/1244252318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a2fe0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f8667fff640 1 -- 172.21.15.114:0/1244252318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576a2c57210 con 0x5576a2c162f0 2024-05-27T07:51:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f8665ffb640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f86540b29d0 0x7f86540b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.606+0000 7f865ffff640 1 --2- 172.21.15.114:0/1244252318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c3f1b0 0x5576a2fe03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.607+0000 7f865ffff640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f86540b29d0 0x7f86540b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.607+0000 7f8667fff640 1 --2- 172.21.15.114:0/1244252318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 0x5576a2fe0910 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f8658009c20 tx=0x7f8658011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ff4242f0b43737 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.607+0000 7f8665ffb640 1 -- 172.21.15.114:0/1766288813 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f86600e2f80 con 0x7f8668109030 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.607+0000 7f865e7fc640 1 -- 172.21.15.114:0/1244252318 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658004230 con 0x5576a2c162f0 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.607+0000 7f865e7fc640 1 -- 172.21.15.114:0/1244252318 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86580043d0 con 0x5576a2c162f0 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.607+0000 7f865e7fc640 1 -- 172.21.15.114:0/1244252318 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658004c80 con 0x5576a2c162f0 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.608+0000 7f86755e4740 1 -- 172.21.15.114:0/1244252318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 msgr2=0x5576a2fe0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.608+0000 7f865ffff640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f86540b29d0 0x7f86540b4ee0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8650009230 tx=0x7f8650040040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.608+0000 7f86755e4740 1 --2- 172.21.15.114:0/1244252318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 0x5576a2fe0910 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f8658009c20 tx=0x7f8658011810 comp rx=0 tx=0).stop 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.608+0000 7f86755e4740 1 -- 172.21.15.114:0/1244252318 shutdown_connections 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.608+0000 7f86755e4740 1 --2- 172.21.15.114:0/1244252318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a2fe0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.608+0000 7f86755e4740 1 --2- 172.21.15.114:0/1244252318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 0x5576a2fe0910 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.608+0000 7f86755e4740 1 --2- 172.21.15.114:0/1244252318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c3f1b0 0x5576a2fe03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.608+0000 7f86755e4740 1 -- 172.21.15.114:0/1244252318 >> 172.21.15.114:0/1244252318 conn(0x5576a2c33300 msgr2=0x5576a2fdcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:35.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.608+0000 7f86755e4740 1 -- 172.21.15.114:0/1244252318 shutdown_connections 2024-05-27T07:51:35.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.608+0000 7f86755e4740 1 -- 172.21.15.114:0/1244252318 wait complete. 2024-05-27T07:51:35.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.608+0000 7f86755e4740 1 -- start start 2024-05-27T07:51:35.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f86755e4740 1 Processor -- start 2024-05-27T07:51:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f86755e4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c162f0 0x5576a2feaa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f86755e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a2feaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f86755e4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c3f1b0 0x5576a2ff0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f86755e4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5576a2c28790 con 0x5576a2c3f1b0 2024-05-27T07:51:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f86755e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5576a2efe890 con 0x5576a2c1e4d0 2024-05-27T07:51:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f86755e4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5576a2fed500 con 0x5576a2c162f0 2024-05-27T07:51:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f86755e4740 10 client.0 _ll_register_callbacks cb 0x5576a2c56a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f866ccd1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c3f1b0 0x5576a2ff0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f866ccd1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c3f1b0 0x5576a2ff0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49248/0 (socket says 172.21.15.114:49248) 2024-05-27T07:51:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f866ccd1640 1 -- 172.21.15.114:0/216197934 learned_addr learned my addr 172.21.15.114:0/216197934 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f8667fff640 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c162f0 0x5576a2feaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f865ffff640 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a2feaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f866ccd1640 1 -- 172.21.15.114:0/216197934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c162f0 msgr2=0x5576a2feaa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f866ccd1640 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c162f0 0x5576a2feaa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f866ccd1640 1 -- 172.21.15.114:0/216197934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 msgr2=0x5576a2feaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f866ccd1640 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a2feaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.609+0000 7f866ccd1640 1 -- 172.21.15.114:0/216197934 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576a2d23620 con 0x5576a2c3f1b0 2024-05-27T07:51:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f865ffff640 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a2feaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f866ccd1640 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c3f1b0 0x5576a2ff0870 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f86600a9e40 tx=0x7f86600a9930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf25d7a0de7d701 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86477fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86477fe640 1 -- 172.21.15.114:0/216197934 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86600a9eb0 con 0x5576a2c3f1b0 2024-05-27T07:51:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86477fe640 1 -- 172.21.15.114:0/216197934 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f86600aa110 con 0x5576a2c3f1b0 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86755e4740 1 -- 172.21.15.114:0/216197934 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86580043d0 con 0x5576a2c3f1b0 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86477fe640 1 -- 172.21.15.114:0/216197934 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f86600ae260 con 0x5576a2c3f1b0 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86755e4740 10 client.18186 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86755e4740 1 -- 172.21.15.114:0/216197934 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5576a2ff98d0 con 0x5576a2c3f1b0 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86477fe640 1 -- 172.21.15.114:0/216197934 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86580043d0 con 0x5576a2c3f1b0 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86477fe640 1 -- 172.21.15.114:0/216197934 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f86600a5820 con 0x5576a2c3f1b0 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86755e4740 10 client.18186 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86755e4740 10 client.18186 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86755e4740 20 client.18186 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86755e4740 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576a2bfdd80 0x5576a2ffd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86755e4740 4 client.18186 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.610+0000 7f86755e4740 1 -- 172.21.15.114:0/216197934 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576a2f54b80 con 0x5576a2bfdd80 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.611+0000 7f8667fff640 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576a2bfdd80 0x5576a2ffd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.611+0000 7f86477fe640 10 client.18186 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.611+0000 7f8667fff640 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576a2bfdd80 0x5576a2ffd010 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f27c5489b752cc7 server_cookie=e2143aec34038641 in_seq=0 out_seq=0 2024-05-27T07:51:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86477fe640 1 -- 172.21.15.114:0/216197934 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5576a2f54b80 con 0x5576a2bfdd80 2024-05-27T07:51:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86477fe640 10 client.18186 handle_command_reply: tid=0 2024-05-27T07:51:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86755e4740 1 client.18186 shutdown 2024-05-27T07:51:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86755e4740 1 -- 172.21.15.114:0/216197934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c3f1b0 msgr2=0x5576a2ff0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86755e4740 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c3f1b0 0x5576a2ff0870 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f86600a9e40 tx=0x7f86600a9930 comp rx=0 tx=0).stop 2024-05-27T07:51:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86755e4740 1 -- 172.21.15.114:0/216197934 shutdown_connections 2024-05-27T07:51:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86755e4740 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576a2bfdd80 0x5576a2ffd010 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86755e4740 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c3f1b0 0x5576a2ff0870 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86755e4740 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a2feaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86477fe640 0 client.18186 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86755e4740 1 --2- 172.21.15.114:0/216197934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c162f0 0x5576a2feaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86755e4740 1 -- 172.21.15.114:0/216197934 >> 172.21.15.114:0/216197934 conn(0x5576a2c33300 msgr2=0x5576a2fe0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86755e4740 1 -- 172.21.15.114:0/216197934 shutdown_connections 2024-05-27T07:51:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86755e4740 1 -- 172.21.15.114:0/216197934 wait complete. 2024-05-27T07:51:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.612+0000 7f86755e4740 20 client.18186 trim_cache size 0 max 16384 2024-05-27T07:51:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.627+0000 7f86755e4740 1 Processor -- start 2024-05-27T07:51:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.627+0000 7f86755e4740 1 -- start start 2024-05-27T07:51:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f86755e4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2bfdd80 0x5576a2fe4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f86755e4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 0x5576a3002810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f86755e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a3002d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f86755e4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f86681b2bc0 con 0x5576a2c162f0 2024-05-27T07:51:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f86755e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f86681b1350 con 0x5576a2c1e4d0 2024-05-27T07:51:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f86755e4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86681af1a0 con 0x5576a2bfdd80 2024-05-27T07:51:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f865ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 0x5576a3002810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f865ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 0x5576a3002810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49252/0 (socket says 172.21.15.114:49252) 2024-05-27T07:51:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f865ffff640 1 -- 172.21.15.114:0/1766015840 learned_addr learned my addr 172.21.15.114:0/1766015840 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f8667fff640 1 --2- 172.21.15.114:0/1766015840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2bfdd80 0x5576a2fe4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f866ccd1640 1 --2- 172.21.15.114:0/1766015840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a3002d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f865ffff640 1 -- 172.21.15.114:0/1766015840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2bfdd80 msgr2=0x5576a2fe4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f865ffff640 1 --2- 172.21.15.114:0/1766015840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2bfdd80 0x5576a2fe4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f865ffff640 1 -- 172.21.15.114:0/1766015840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 msgr2=0x5576a3002d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f865ffff640 1 --2- 172.21.15.114:0/1766015840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a3002d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f865ffff640 1 -- 172.21.15.114:0/1766015840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576a2fe17b0 con 0x5576a2c162f0 2024-05-27T07:51:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f866ccd1640 1 --2- 172.21.15.114:0/1766015840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a3002d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:51:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f865ffff640 1 --2- 172.21.15.114:0/1766015840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 0x5576a3002810 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f8650009900 tx=0x7f8650031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53546c3449c780c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.628+0000 7f865dffb640 1 -- 172.21.15.114:0/1766015840 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8650050070 con 0x5576a2c162f0 2024-05-27T07:51:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.629+0000 7f865dffb640 1 -- 172.21.15.114:0/1766015840 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8650004240 con 0x5576a2c162f0 2024-05-27T07:51:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.629+0000 7f865dffb640 1 -- 172.21.15.114:0/1766015840 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8650045070 con 0x5576a2c162f0 2024-05-27T07:51:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.629+0000 7f86755e4740 1 -- 172.21.15.114:0/1766015840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 msgr2=0x5576a3002810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.629+0000 7f86755e4740 1 --2- 172.21.15.114:0/1766015840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 0x5576a3002810 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f8650009900 tx=0x7f8650031400 comp rx=0 tx=0).stop 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.629+0000 7f86755e4740 1 -- 172.21.15.114:0/1766015840 shutdown_connections 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.629+0000 7f86755e4740 1 --2- 172.21.15.114:0/1766015840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a3002d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.629+0000 7f86755e4740 1 --2- 172.21.15.114:0/1766015840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2c162f0 0x5576a3002810 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.629+0000 7f86755e4740 1 --2- 172.21.15.114:0/1766015840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2bfdd80 0x5576a2fe4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.629+0000 7f86755e4740 1 -- 172.21.15.114:0/1766015840 >> 172.21.15.114:0/1766015840 conn(0x5576a2c33300 msgr2=0x5576a2fe4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.630+0000 7f86755e4740 1 -- 172.21.15.114:0/1766015840 shutdown_connections 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.630+0000 7f86755e4740 1 -- 172.21.15.114:0/1766015840 wait complete. 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.630+0000 7f86755e4740 1 -- start start 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.630+0000 7f86755e4740 1 Processor -- start 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.630+0000 7f86755e4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2bfdd80 0x5576a2ff3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.630+0000 7f86755e4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c162f0 0x5576a2ff4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.630+0000 7f86755e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a30636e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.630+0000 7f86755e4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f86681af810 con 0x5576a2bfdd80 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.630+0000 7f86755e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f86681b0c80 con 0x5576a2c1e4d0 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.630+0000 7f86755e4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86681afee0 con 0x5576a2c162f0 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f86755e4740 10 client.0 _ll_register_callbacks cb 0x7f86600a9cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f8667fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2bfdd80 0x5576a2ff3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f866ccd1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a30636e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f865ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c162f0 0x5576a2ff4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f866ccd1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a30636e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:57038/0 (socket says 172.21.15.114:57038) 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f866ccd1640 1 -- 172.21.15.114:0/3211551058 learned_addr learned my addr 172.21.15.114:0/3211551058 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f866ccd1640 1 -- 172.21.15.114:0/3211551058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c162f0 msgr2=0x5576a2ff4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f866ccd1640 1 --2- 172.21.15.114:0/3211551058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c162f0 0x5576a2ff4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f866ccd1640 1 -- 172.21.15.114:0/3211551058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2bfdd80 msgr2=0x5576a2ff3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f866ccd1640 1 --2- 172.21.15.114:0/3211551058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2bfdd80 0x5576a2ff3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f866ccd1640 1 -- 172.21.15.114:0/3211551058 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576a2ff4ad0 con 0x5576a2c1e4d0 2024-05-27T07:51:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f865ffff640 1 --2- 172.21.15.114:0/3211551058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c162f0 0x5576a2ff4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f866ccd1640 1 --2- 172.21.15.114:0/3211551058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a30636e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f86600049d0 tx=0x7f86600b2ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=342c9581e5657665 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f86477fe640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:51:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f86477fe640 1 -- 172.21.15.114:0/3211551058 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86600a8f00 con 0x5576a2c1e4d0 2024-05-27T07:51:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f86477fe640 1 -- 172.21.15.114:0/3211551058 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f86600aa320 con 0x5576a2c1e4d0 2024-05-27T07:51:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.631+0000 7f86755e4740 1 -- 172.21.15.114:0/3211551058 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8650004240 con 0x5576a2c1e4d0 2024-05-27T07:51:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.632+0000 7f86477fe640 1 -- 172.21.15.114:0/3211551058 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f86600b7050 con 0x5576a2c1e4d0 2024-05-27T07:51:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.632+0000 7f86755e4740 10 client.27614 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.632+0000 7f86755e4740 1 -- 172.21.15.114:0/3211551058 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5576a3064bf0 con 0x5576a2c1e4d0 2024-05-27T07:51:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.632+0000 7f86477fe640 1 -- 172.21.15.114:0/3211551058 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8650004240 con 0x5576a2c1e4d0 2024-05-27T07:51:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.632+0000 7f86477fe640 1 -- 172.21.15.114:0/3211551058 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f86600af3f0 con 0x5576a2c1e4d0 2024-05-27T07:51:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.632+0000 7f86755e4740 10 client.27614 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.632+0000 7f86755e4740 10 client.27614 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.632+0000 7f86755e4740 20 client.27614 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.632+0000 7f86755e4740 1 --2- 172.21.15.114:0/3211551058 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576a2c3f1b0 0x5576a3067320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.632+0000 7f86755e4740 4 client.27614 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:51:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.632+0000 7f86755e4740 1 -- 172.21.15.114:0/3211551058 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5576a2f4e360 con 0x5576a2c3f1b0 2024-05-27T07:51:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.632+0000 7f8667fff640 1 --2- 172.21.15.114:0/3211551058 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576a2c3f1b0 0x5576a3067320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.633+0000 7f86477fe640 10 client.27614 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:35.633+0000 7f8667fff640 1 --2- 172.21.15.114:0/3211551058 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576a2c3f1b0 0x5576a3067320 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7582a4c3d215ad70 server_cookie=98364f1dca31a059 in_seq=0 out_seq=0 2024-05-27T07:51:35.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:35 smithi114 ceph-mon[32038]: pgmap v394: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-05-27T07:51:35.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:35 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:51:35.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:35 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:35.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:35 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:51:35.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:35 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:35.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:51:35.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:35 smithi163 ceph-mon[36451]: pgmap v394: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-05-27T07:51:35.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:35 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:51:35.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:35 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:35.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:35 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:51:35.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:35 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:35.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:35 smithi186 ceph-mon[36647]: pgmap v394: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-05-27T07:51:35.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:35 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:51:35.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:35 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:35.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:35 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:51:35.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:35 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.159+0000 7f86477fe640 1 -- 172.21.15.114:0/3211551058 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5576a2f4e360 con 0x5576a2c3f1b0 2024-05-27T07:51:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.159+0000 7f86477fe640 10 client.27614 handle_command_reply: tid=0 2024-05-27T07:51:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.159+0000 7f86755e4740 1 client.27614 shutdown 2024-05-27T07:51:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.160+0000 7f86755e4740 1 -- 172.21.15.114:0/3211551058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 msgr2=0x5576a30636e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.160+0000 7f86755e4740 1 --2- 172.21.15.114:0/3211551058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a30636e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f86600049d0 tx=0x7f86600b2ab0 comp rx=0 tx=0).stop 2024-05-27T07:51:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.160+0000 7f86755e4740 1 -- 172.21.15.114:0/3211551058 shutdown_connections 2024-05-27T07:51:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.160+0000 7f86755e4740 1 --2- 172.21.15.114:0/3211551058 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5576a2c3f1b0 0x5576a3067320 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.160+0000 7f86755e4740 1 --2- 172.21.15.114:0/3211551058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576a2c1e4d0 0x5576a30636e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.160+0000 7f86755e4740 1 --2- 172.21.15.114:0/3211551058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576a2c162f0 0x5576a2ff4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.160+0000 7f86755e4740 1 --2- 172.21.15.114:0/3211551058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576a2bfdd80 0x5576a2ff3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.160+0000 7f86755e4740 1 -- 172.21.15.114:0/3211551058 >> 172.21.15.114:0/3211551058 conn(0x5576a2c33300 msgr2=0x5576a2fe8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.160+0000 7f86477fe640 0 client.27614 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.160+0000 7f86755e4740 1 -- 172.21.15.114:0/3211551058 shutdown_connections 2024-05-27T07:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.160+0000 7f86755e4740 1 -- 172.21.15.114:0/3211551058 wait complete. 2024-05-27T07:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.160+0000 7f86755e4740 20 client.27614 trim_cache size 0 max 16384 2024-05-27T07:51:36.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.162+0000 7f866e734640 1 -- 172.21.15.114:0/1766288813 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f86540b29d0 msgr2=0x7f86540b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:36.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.162+0000 7f866e734640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f86540b29d0 0x7f86540b4ee0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8650009230 tx=0x7f8650040040 comp rx=0 tx=0).stop 2024-05-27T07:51:36.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.162+0000 7f866e734640 1 -- 172.21.15.114:0/1766288813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8668109030 msgr2=0x7f866819a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:36.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.162+0000 7f866e734640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8668109030 0x7f866819a770 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f866000eb10 tx=0x7f866000c2d0 comp rx=0 tx=0).stop 2024-05-27T07:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.164+0000 7f866e734640 1 -- 172.21.15.114:0/1766288813 shutdown_connections 2024-05-27T07:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.164+0000 7f866e734640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f86540b29d0 0x7f86540b4ee0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.164+0000 7f866e734640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8668109030 0x7f866819a770 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.164+0000 7f866e734640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8668108b90 0x7f86681a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.164+0000 7f866e734640 1 --2- 172.21.15.114:0/1766288813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86681032e0 0x7f86681a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.164+0000 7f866e734640 1 -- 172.21.15.114:0/1766288813 >> 172.21.15.114:0/1766288813 conn(0x7f86680782d0 msgr2=0x7f86681014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.164+0000 7f866e734640 1 -- 172.21.15.114:0/1766288813 shutdown_connections 2024-05-27T07:51:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:36.165+0000 7f866e734640 1 -- 172.21.15.114:0/1766288813 wait complete. 2024-05-27T07:51:36.219 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:36.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1155190 9.81 MB/sec execute 468 sec latency 415.269 ms 2024-05-27T07:51:37.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1157381 9.82 MB/sec execute 469 sec latency 43.076 ms 2024-05-27T07:51:37.765 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:37 smithi114 ceph-mon[32038]: pgmap v395: 129 pgs: 129 active+clean; 75 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 222 op/s 2024-05-27T07:51:37.765 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:37 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:51:37.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:37 smithi163 ceph-mon[36451]: pgmap v395: 129 pgs: 129 active+clean; 75 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 222 op/s 2024-05-27T07:51:37.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:37 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:51:37.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:37 smithi186 ceph-mon[36647]: pgmap v395: 129 pgs: 129 active+clean; 75 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 222 op/s 2024-05-27T07:51:37.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:37 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:51:38.288 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1158608 9.81 MB/sec execute 470 sec latency 69.614 ms 2024-05-27T07:51:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.711+0000 7fa12050f700 1 -- 172.21.15.114:0/3657639070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa118108730 msgr2=0x7fa118110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.711+0000 7fa12050f700 1 --2- 172.21.15.114:0/3657639070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa118108730 0x7fa118110090 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fa11000b0d0 tx=0x7fa11000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.711+0000 7fa12050f700 1 -- 172.21.15.114:0/3657639070 shutdown_connections 2024-05-27T07:51:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.711+0000 7fa12050f700 1 --2- 172.21.15.114:0/3657639070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa118108730 0x7fa118110090 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.711+0000 7fa12050f700 1 --2- 172.21.15.114:0/3657639070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1181034b0 0x7fa1181038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.711+0000 7fa12050f700 1 --2- 172.21.15.114:0/3657639070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa118107de0 0x7fa1181081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.711+0000 7fa12050f700 1 -- 172.21.15.114:0/3657639070 >> 172.21.15.114:0/3657639070 conn(0x7fa1180fede0 msgr2=0x7fa118101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:38.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.711+0000 7fa12050f700 1 -- 172.21.15.114:0/3657639070 shutdown_connections 2024-05-27T07:51:38.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.711+0000 7fa12050f700 1 -- 172.21.15.114:0/3657639070 wait complete. 2024-05-27T07:51:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.712+0000 7fa12050f700 1 Processor -- start 2024-05-27T07:51:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.712+0000 7fa12050f700 1 -- start start 2024-05-27T07:51:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.712+0000 7fa12050f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1181034b0 0x7fa11819bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.712+0000 7fa12050f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa118107de0 0x7fa11819c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.712+0000 7fa12050f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa118108730 0x7fa11819c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.712+0000 7fa12050f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa118113360 con 0x7fa118107de0 2024-05-27T07:51:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa12050f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181131e0 con 0x7fa118108730 2024-05-27T07:51:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa12050f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181134e0 con 0x7fa1181034b0 2024-05-27T07:51:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa11e2ab700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1181034b0 0x7fa11819bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa11daaa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa118107de0 0x7fa11819c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa11e2ab700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1181034b0 0x7fa11819bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35520/0 (socket says 172.21.15.114:35520) 2024-05-27T07:51:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa11daaa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa118107de0 0x7fa11819c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49288/0 (socket says 172.21.15.114:49288) 2024-05-27T07:51:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa11e2ab700 1 -- 172.21.15.114:0/1489657282 learned_addr learned my addr 172.21.15.114:0/1489657282 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa11eaac700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa118108730 0x7fa11819c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa11e2ab700 1 -- 172.21.15.114:0/1489657282 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa118108730 msgr2=0x7fa11819c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa11e2ab700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa118108730 0x7fa11819c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa11e2ab700 1 -- 172.21.15.114:0/1489657282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa118107de0 msgr2=0x7fa11819c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa11e2ab700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa118107de0 0x7fa11819c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa11e2ab700 1 -- 172.21.15.114:0/1489657282 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1181a6f10 con 0x7fa1181034b0 2024-05-27T07:51:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa11eaac700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa118108730 0x7fa11819c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa11e2ab700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1181034b0 0x7fa11819bbc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa11400e9a0 tx=0x7fa11400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa10f7fe700 1 -- 172.21.15.114:0/1489657282 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11400cbd0 con 0x7fa1181034b0 2024-05-27T07:51:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa10f7fe700 1 -- 172.21.15.114:0/1489657282 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11400cd70 con 0x7fa1181034b0 2024-05-27T07:51:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.713+0000 7fa12050f700 1 -- 172.21.15.114:0/1489657282 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1181a7200 con 0x7fa1181034b0 2024-05-27T07:51:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.714+0000 7fa10f7fe700 1 -- 172.21.15.114:0/1489657282 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa114018680 con 0x7fa1181034b0 2024-05-27T07:51:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.714+0000 7fa12050f700 1 -- 172.21.15.114:0/1489657282 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1181a7790 con 0x7fa1181034b0 2024-05-27T07:51:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.716+0000 7fa10f7fe700 1 -- 172.21.15.114:0/1489657282 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa114010430 con 0x7fa1181034b0 2024-05-27T07:51:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.717+0000 7fa10f7fe700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa1040b2a70 0x7fa1040b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.717+0000 7fa11daaa700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa1040b2a70 0x7fa1040b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.717+0000 7fa10f7fe700 1 -- 172.21.15.114:0/1489657282 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7fa1140e2bc0 con 0x7fa1181034b0 2024-05-27T07:51:38.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.717+0000 7fa11daaa700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa1040b2a70 0x7fa1040b4f80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa1181066b0 tx=0x7fa108009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:38.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.718+0000 7fa12ef24b80 1 Processor -- start 2024-05-27T07:51:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.718+0000 7fa12ef24b80 1 -- start start 2024-05-27T07:51:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.718+0000 7fa12ef24b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd95e40 0x55617bf1ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.718+0000 7fa12ef24b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bf1b070 0x55617bf14be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.718+0000 7fa12ef24b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf15530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.718+0000 7fa12ef24b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55617be42d70 con 0x55617bf15120 2024-05-27T07:51:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.718+0000 7fa12ef24b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55617be32910 con 0x55617bf1b070 2024-05-27T07:51:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.718+0000 7fa12ef24b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55617be36950 con 0x55617bd95e40 2024-05-27T07:51:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.718+0000 7fa11eaac700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf15530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.718+0000 7fa11daaa700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bf1b070 0x55617bf14be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.718+0000 7fa11e2ab700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd95e40 0x55617bf1ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa11eaac700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf15530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49300/0 (socket says 172.21.15.114:49300) 2024-05-27T07:51:38.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa11daaa700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bf1b070 0x55617bf14be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:57068/0 (socket says 172.21.15.114:57068) 2024-05-27T07:51:38.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa11e2ab700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd95e40 0x55617bf1ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35534/0 (socket says 172.21.15.114:35534) 2024-05-27T07:51:38.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa11eaac700 1 -- 172.21.15.114:0/3032540657 learned_addr learned my addr 172.21.15.114:0/3032540657 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:38.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa11eaac700 1 -- 172.21.15.114:0/3032540657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd95e40 msgr2=0x55617bf1ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:38.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa11eaac700 1 --2- 172.21.15.114:0/3032540657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd95e40 0x55617bf1ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa11eaac700 1 -- 172.21.15.114:0/3032540657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bf1b070 msgr2=0x55617bf14be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:38.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa11eaac700 1 --2- 172.21.15.114:0/3032540657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bf1b070 0x55617bf14be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa11eaac700 1 -- 172.21.15.114:0/3032540657 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55617bddbfb0 con 0x55617bf15120 2024-05-27T07:51:38.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa11daaa700 1 --2- 172.21.15.114:0/3032540657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bf1b070 0x55617bf14be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:38.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa11eaac700 1 --2- 172.21.15.114:0/3032540657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf15530 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fa110015000 tx=0x7fa110009d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53be862eb3be3316 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:38.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa10cd47700 1 -- 172.21.15.114:0/3032540657 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa110022070 con 0x55617bf15120 2024-05-27T07:51:38.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa10cd47700 1 -- 172.21.15.114:0/3032540657 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa110004a50 con 0x55617bf15120 2024-05-27T07:51:38.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.719+0000 7fa10cd47700 1 -- 172.21.15.114:0/3032540657 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa110007b00 con 0x55617bf15120 2024-05-27T07:51:38.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.720+0000 7fa12ef24b80 1 -- 172.21.15.114:0/3032540657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 msgr2=0x55617bf15530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:38.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.720+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/3032540657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf15530 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fa110015000 tx=0x7fa110009d90 comp rx=0 tx=0).stop 2024-05-27T07:51:38.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.720+0000 7fa12ef24b80 1 -- 172.21.15.114:0/3032540657 shutdown_connections 2024-05-27T07:51:38.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.720+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/3032540657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf15530 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.720+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/3032540657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd95e40 0x55617bf1ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.720+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/3032540657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bf1b070 0x55617bf14be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.720+0000 7fa12ef24b80 1 -- 172.21.15.114:0/3032540657 >> 172.21.15.114:0/3032540657 conn(0x55617bd98770 msgr2=0x55617bf0fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:38.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.721+0000 7fa12ef24b80 1 -- 172.21.15.114:0/3032540657 shutdown_connections 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.721+0000 7fa12ef24b80 1 -- 172.21.15.114:0/3032540657 wait complete. 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.721+0000 7fa12ef24b80 1 -- start start 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.721+0000 7fa12ef24b80 1 Processor -- start 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa12ef24b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bd95e40 0x55617bf28dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa12ef24b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf29300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa12ef24b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bf1b070 0x55617bf231c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa12ef24b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55617bd38fa0 con 0x55617bf15120 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa12ef24b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55617bd9ada0 con 0x55617bd95e40 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa12ef24b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55617bf21b00 con 0x55617bf1b070 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa11daaa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf29300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa11daaa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf29300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49304/0 (socket says 172.21.15.114:49304) 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa11daaa700 1 -- 172.21.15.114:0/946189826 learned_addr learned my addr 172.21.15.114:0/946189826 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa11eaac700 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bf1b070 0x55617bf231c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa11e2ab700 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bd95e40 0x55617bf28dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa12ef24b80 10 client.0 _ll_register_callbacks cb 0x55617bdc95b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa11daaa700 1 -- 172.21.15.114:0/946189826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bf1b070 msgr2=0x55617bf231c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa11daaa700 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bf1b070 0x55617bf231c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa11daaa700 1 -- 172.21.15.114:0/946189826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bd95e40 msgr2=0x55617bf28dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa11daaa700 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bd95e40 0x55617bf28dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa11daaa700 1 -- 172.21.15.114:0/946189826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55617bd828d0 con 0x55617bf15120 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa11e2ab700 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bd95e40 0x55617bf28dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa11daaa700 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf29300 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fa10801f8e0 tx=0x7fa1080047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e16c714fa9cf84 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa101ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.722+0000 7fa101ffb700 1 -- 172.21.15.114:0/946189826 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa108004030 con 0x55617bf15120 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa101ffb700 1 -- 172.21.15.114:0/946189826 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7fa108009040 con 0x55617bf15120 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa12ef24b80 1 -- 172.21.15.114:0/946189826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa110004a50 con 0x55617bf15120 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa101ffb700 1 -- 172.21.15.114:0/946189826 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fa1080041d0 con 0x55617bf15120 2024-05-27T07:51:38.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa12ef24b80 10 client.18204 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa12ef24b80 1 -- 172.21.15.114:0/946189826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55617bdec760 con 0x55617bf15120 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa101ffb700 1 -- 172.21.15.114:0/946189826 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa110004a50 con 0x55617bf15120 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa101ffb700 1 -- 172.21.15.114:0/946189826 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7fa108003820 con 0x55617bf15120 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa12ef24b80 10 client.18204 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa12ef24b80 10 client.18204 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa12ef24b80 20 client.18204 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55617bf33270 0x55617bf35760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa12ef24b80 4 client.18204 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa12ef24b80 1 -- 172.21.15.114:0/946189826 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55617be44cc0 con 0x55617bf33270 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.723+0000 7fa11e2ab700 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55617bf33270 0x55617bf35760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.724+0000 7fa11e2ab700 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55617bf33270 0x55617bf35760 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5140dd89ae1ec030 server_cookie=d7293be8cb96f98b in_seq=0 out_seq=0 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.724+0000 7fa101ffb700 10 client.18204 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.724+0000 7fa101ffb700 1 -- 172.21.15.114:0/946189826 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55617be44cc0 con 0x55617bf33270 2024-05-27T07:51:38.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.724+0000 7fa101ffb700 10 client.18204 handle_command_reply: tid=0 2024-05-27T07:51:38.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.724+0000 7fa12ef24b80 1 client.18204 shutdown 2024-05-27T07:51:38.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.725+0000 7fa12ef24b80 1 -- 172.21.15.114:0/946189826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 msgr2=0x55617bf29300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:38.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.725+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf29300 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fa10801f8e0 tx=0x7fa1080047c0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.725+0000 7fa12ef24b80 1 -- 172.21.15.114:0/946189826 shutdown_connections 2024-05-27T07:51:38.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.725+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bf1b070 0x55617bf231c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.725+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55617bf33270 0x55617bf35760 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.725+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bd95e40 0x55617bf28dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.725+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/946189826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf29300 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.725+0000 7fa101ffb700 0 client.18204 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:38.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.725+0000 7fa12ef24b80 1 -- 172.21.15.114:0/946189826 >> 172.21.15.114:0/946189826 conn(0x55617bd98770 msgr2=0x55617bf13bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:38.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.725+0000 7fa12ef24b80 1 -- 172.21.15.114:0/946189826 shutdown_connections 2024-05-27T07:51:38.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.725+0000 7fa12ef24b80 1 -- 172.21.15.114:0/946189826 wait complete. 2024-05-27T07:51:38.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.725+0000 7fa12ef24b80 20 client.18204 trim_cache size 0 max 16384 2024-05-27T07:51:38.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.746+0000 7fa12ef24b80 1 Processor -- start 2024-05-27T07:51:38.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.746+0000 7fa12ef24b80 1 -- start start 2024-05-27T07:51:38.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa12ef24b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bd98770 0x55617bf17490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa12ef24b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd95e40 0x55617bf23180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa12ef24b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf20b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa12ef24b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181afc20 con 0x55617bf15120 2024-05-27T07:51:38.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa12ef24b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181ae390 con 0x55617bd98770 2024-05-27T07:51:38.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa12ef24b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181ac1e0 con 0x55617bd95e40 2024-05-27T07:51:38.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa11eaac700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf20b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa11daaa700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd95e40 0x55617bf23180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa11eaac700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf20b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49316/0 (socket says 172.21.15.114:49316) 2024-05-27T07:51:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa11eaac700 1 -- 172.21.15.114:0/1619026408 learned_addr learned my addr 172.21.15.114:0/1619026408 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa11e2ab700 1 --2- 172.21.15.114:0/1619026408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bd98770 0x55617bf17490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa11eaac700 1 -- 172.21.15.114:0/1619026408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd95e40 msgr2=0x55617bf23180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa11eaac700 1 --2- 172.21.15.114:0/1619026408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd95e40 0x55617bf23180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa11eaac700 1 -- 172.21.15.114:0/1619026408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bd98770 msgr2=0x55617bf17490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa11eaac700 1 --2- 172.21.15.114:0/1619026408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bd98770 0x55617bf17490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa11eaac700 1 -- 172.21.15.114:0/1619026408 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55617bf1a500 con 0x55617bf15120 2024-05-27T07:51:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa11eaac700 1 --2- 172.21.15.114:0/1619026408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf20b40 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fa110006130 tx=0x7fa110007660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e63a999a8a366e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.747+0000 7fa103fff700 1 -- 172.21.15.114:0/1619026408 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa110022040 con 0x55617bf15120 2024-05-27T07:51:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.748+0000 7fa103fff700 1 -- 172.21.15.114:0/1619026408 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11002a030 con 0x55617bf15120 2024-05-27T07:51:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.748+0000 7fa103fff700 1 -- 172.21.15.114:0/1619026408 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa110007a30 con 0x55617bf15120 2024-05-27T07:51:38.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.748+0000 7fa12ef24b80 1 -- 172.21.15.114:0/1619026408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 msgr2=0x55617bf20b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:38.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.748+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/1619026408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf20b40 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fa110006130 tx=0x7fa110007660 comp rx=0 tx=0).stop 2024-05-27T07:51:38.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.748+0000 7fa12ef24b80 1 -- 172.21.15.114:0/1619026408 shutdown_connections 2024-05-27T07:51:38.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.748+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/1619026408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bf15120 0x55617bf20b40 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.748+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/1619026408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bd98770 0x55617bf17490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.748+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/1619026408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd95e40 0x55617bf23180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.748+0000 7fa12ef24b80 1 -- 172.21.15.114:0/1619026408 >> 172.21.15.114:0/1619026408 conn(0x55617bf23ad0 msgr2=0x55617bf22a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:38.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.749+0000 7fa12ef24b80 1 -- 172.21.15.114:0/1619026408 shutdown_connections 2024-05-27T07:51:38.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.749+0000 7fa12ef24b80 1 -- 172.21.15.114:0/1619026408 wait complete. 2024-05-27T07:51:38.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.749+0000 7fa12ef24b80 1 -- start start 2024-05-27T07:51:38.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.749+0000 7fa12ef24b80 1 Processor -- start 2024-05-27T07:51:38.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.749+0000 7fa12ef24b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bd95e40 0x55617bf2a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.749+0000 7fa12ef24b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd98770 0x55617bf2a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.749+0000 7fa12ef24b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bf15120 0x55617bf9be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.749+0000 7fa12ef24b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181ac850 con 0x55617bd95e40 2024-05-27T07:51:38.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.749+0000 7fa12ef24b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181adcc0 con 0x55617bf15120 2024-05-27T07:51:38.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.749+0000 7fa12ef24b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181acf20 con 0x55617bd98770 2024-05-27T07:51:38.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.749+0000 7fa11e2ab700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bd95e40 0x55617bf2a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.749+0000 7fa12ef24b80 10 client.0 _ll_register_callbacks cb 0x55617bdc95b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:38.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa11e2ab700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bd95e40 0x55617bf2a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49318/0 (socket says 172.21.15.114:49318) 2024-05-27T07:51:38.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa11e2ab700 1 -- 172.21.15.114:0/1776114885 learned_addr learned my addr 172.21.15.114:0/1776114885 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:38.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa11daaa700 1 --2- 172.21.15.114:0/1776114885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd98770 0x55617bf2a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa11eaac700 1 --2- 172.21.15.114:0/1776114885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bf15120 0x55617bf9be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa11e2ab700 1 -- 172.21.15.114:0/1776114885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd98770 msgr2=0x55617bf2a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:38.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa11e2ab700 1 --2- 172.21.15.114:0/1776114885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd98770 0x55617bf2a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa11e2ab700 1 -- 172.21.15.114:0/1776114885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bf15120 msgr2=0x55617bf9be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:38.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa11e2ab700 1 --2- 172.21.15.114:0/1776114885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bf15120 0x55617bf9be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa11e2ab700 1 -- 172.21.15.114:0/1776114885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55617bf2afe0 con 0x55617bd95e40 2024-05-27T07:51:38.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa11e2ab700 1 --2- 172.21.15.114:0/1776114885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bd95e40 0x55617bf2a210 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fa11400f010 tx=0x7fa1140cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc430bfada864991 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:38.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa101ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:38.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa101ffb700 1 -- 172.21.15.114:0/1776114885 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1140aa1f0 con 0x55617bd95e40 2024-05-27T07:51:38.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa101ffb700 1 -- 172.21.15.114:0/1776114885 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7fa1140b26d0 con 0x55617bd95e40 2024-05-27T07:51:38.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa12ef24b80 1 -- 172.21.15.114:0/1776114885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa11002a030 con 0x55617bd95e40 2024-05-27T07:51:38.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa101ffb700 1 -- 172.21.15.114:0/1776114885 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fa1140a9af0 con 0x55617bd95e40 2024-05-27T07:51:38.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa12ef24b80 10 client.18210 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:38.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.750+0000 7fa12ef24b80 1 -- 172.21.15.114:0/1776114885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55617bf9d3f0 con 0x55617bd95e40 2024-05-27T07:51:38.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.751+0000 7fa101ffb700 1 -- 172.21.15.114:0/1776114885 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11002a030 con 0x55617bd95e40 2024-05-27T07:51:38.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.751+0000 7fa101ffb700 1 -- 172.21.15.114:0/1776114885 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7fa1140a5460 con 0x55617bd95e40 2024-05-27T07:51:38.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.751+0000 7fa12ef24b80 10 client.18210 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:38.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.751+0000 7fa12ef24b80 10 client.18210 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:38.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.751+0000 7fa12ef24b80 20 client.18210 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:38.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.751+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/1776114885 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55617bf1b070 0x55617bf9fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:38.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.751+0000 7fa12ef24b80 4 client.18210 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:51:38.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.751+0000 7fa12ef24b80 1 -- 172.21.15.114:0/1776114885 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55617be3e4a0 con 0x55617bf1b070 2024-05-27T07:51:38.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.751+0000 7fa11daaa700 1 --2- 172.21.15.114:0/1776114885 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55617bf1b070 0x55617bf9fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:38.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.751+0000 7fa101ffb700 10 client.18210 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:38.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.751+0000 7fa11daaa700 1 --2- 172.21.15.114:0/1776114885 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55617bf1b070 0x55617bf9fb60 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8be60a85280f4e6 server_cookie=dff70aeca803e5bb in_seq=0 out_seq=0 2024-05-27T07:51:38.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa101ffb700 1 -- 172.21.15.114:0/1776114885 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55617be3e4a0 con 0x55617bf1b070 2024-05-27T07:51:38.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa101ffb700 10 client.18210 handle_command_reply: tid=0 2024-05-27T07:51:38.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa12ef24b80 1 client.18210 shutdown 2024-05-27T07:51:38.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa12ef24b80 1 -- 172.21.15.114:0/1776114885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bd95e40 msgr2=0x55617bf2a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:38.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/1776114885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bd95e40 0x55617bf2a210 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fa11400f010 tx=0x7fa1140cc040 comp rx=0 tx=0).stop 2024-05-27T07:51:38.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa12ef24b80 1 -- 172.21.15.114:0/1776114885 shutdown_connections 2024-05-27T07:51:38.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/1776114885 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55617bf1b070 0x55617bf9fb60 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/1776114885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55617bf15120 0x55617bf9be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa101ffb700 0 client.18210 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:38.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/1776114885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55617bd95e40 0x55617bf2a210 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa12ef24b80 1 --2- 172.21.15.114:0/1776114885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55617bd98770 0x55617bf2a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa12ef24b80 1 -- 172.21.15.114:0/1776114885 >> 172.21.15.114:0/1776114885 conn(0x55617bf23ad0 msgr2=0x55617bf094f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:38.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa12ef24b80 1 -- 172.21.15.114:0/1776114885 shutdown_connections 2024-05-27T07:51:38.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa12ef24b80 1 -- 172.21.15.114:0/1776114885 wait complete. 2024-05-27T07:51:38.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.752+0000 7fa12ef24b80 20 client.18210 trim_cache size 0 max 16384 2024-05-27T07:51:38.760 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:51:38.760 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:51:38.760 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:51:38.760 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:51:38.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.753+0000 7fa12050f700 1 -- 172.21.15.114:0/1489657282 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa1040b2a70 msgr2=0x7fa1040b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:38.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.753+0000 7fa12050f700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa1040b2a70 0x7fa1040b4f80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa1181066b0 tx=0x7fa108009260 comp rx=0 tx=0).stop 2024-05-27T07:51:38.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.753+0000 7fa12050f700 1 -- 172.21.15.114:0/1489657282 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1181034b0 msgr2=0x7fa11819bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:38.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.753+0000 7fa12050f700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1181034b0 0x7fa11819bbc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa11400e9a0 tx=0x7fa11400ecb0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.753+0000 7fa12050f700 1 -- 172.21.15.114:0/1489657282 shutdown_connections 2024-05-27T07:51:38.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.753+0000 7fa12050f700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa1040b2a70 0x7fa1040b4f80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.753+0000 7fa12050f700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa118108730 0x7fa11819c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.753+0000 7fa12050f700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1181034b0 0x7fa11819bbc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.753+0000 7fa12050f700 1 --2- 172.21.15.114:0/1489657282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa118107de0 0x7fa11819c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:38.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.753+0000 7fa12050f700 1 -- 172.21.15.114:0/1489657282 >> 172.21.15.114:0/1489657282 conn(0x7fa1180fede0 msgr2=0x7fa118107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:38.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.753+0000 7fa12050f700 1 -- 172.21.15.114:0/1489657282 shutdown_connections 2024-05-27T07:51:38.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:38.753+0000 7fa12050f700 1 -- 172.21.15.114:0/1489657282 wait complete. 2024-05-27T07:51:39.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1160486 9.81 MB/sec execute 471 sec latency 91.963 ms 2024-05-27T07:51:39.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:39 smithi114 ceph-mon[32038]: pgmap v396: 129 pgs: 129 active+clean; 75 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 196 op/s 2024-05-27T07:51:39.886 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:51:39.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:51:39.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:39 smithi163 ceph-mon[36451]: pgmap v396: 129 pgs: 129 active+clean; 75 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 196 op/s 2024-05-27T07:51:39.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:39 smithi186 ceph-mon[36647]: pgmap v396: 129 pgs: 129 active+clean; 75 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 196 op/s 2024-05-27T07:51:40.135 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:40.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1162075 9.81 MB/sec execute 472 sec latency 14.972 ms 2024-05-27T07:51:41.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1163942 9.80 MB/sec execute 473 sec latency 55.411 ms 2024-05-27T07:51:41.720 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:41 smithi114 ceph-mon[32038]: pgmap v397: 129 pgs: 129 active+clean; 75 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 189 op/s 2024-05-27T07:51:41.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:41 smithi163 ceph-mon[36451]: pgmap v397: 129 pgs: 129 active+clean; 75 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 189 op/s 2024-05-27T07:51:41.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:41 smithi186 ceph-mon[36647]: pgmap v397: 129 pgs: 129 active+clean; 75 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 189 op/s 2024-05-27T07:51:42.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.114+0000 7f331fa4b700 1 -- 172.21.15.114:0/3731004367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f331810a920 msgr2=0x7f331810ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:42.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.114+0000 7f331fa4b700 1 --2- 172.21.15.114:0/3731004367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f331810a920 0x7f331810ad30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f3308009990 tx=0x7f3308009ca0 comp rx=0 tx=0).stop 2024-05-27T07:51:42.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.114+0000 7f331fa4b700 1 -- 172.21.15.114:0/3731004367 shutdown_connections 2024-05-27T07:51:42.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.114+0000 7f331fa4b700 1 --2- 172.21.15.114:0/3731004367 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f331810b270 0x7f33181198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:42.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.114+0000 7f331fa4b700 1 --2- 172.21.15.114:0/3731004367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3318111bc0 0x7f3318111fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:42.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.114+0000 7f331fa4b700 1 --2- 172.21.15.114:0/3731004367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f331810a920 0x7f331810ad30 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:42.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.114+0000 7f331fa4b700 1 -- 172.21.15.114:0/3731004367 >> 172.21.15.114:0/3731004367 conn(0x7f331806f4c0 msgr2=0x7f331806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:42.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.114+0000 7f331fa4b700 1 -- 172.21.15.114:0/3731004367 shutdown_connections 2024-05-27T07:51:42.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.115+0000 7f331fa4b700 1 -- 172.21.15.114:0/3731004367 wait complete. 2024-05-27T07:51:42.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.116+0000 7f331fa4b700 1 Processor -- start 2024-05-27T07:51:42.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.116+0000 7f331fa4b700 1 -- start start 2024-05-27T07:51:42.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.116+0000 7f331fa4b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f331810a920 0x7f331810e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:42.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.116+0000 7f331fa4b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f331810b270 0x7f331810c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:42.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.116+0000 7f331fa4b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3318111bc0 0x7f331810cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:42.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.116+0000 7f331fa4b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f331811cc40 con 0x7f331810a920 2024-05-27T07:51:42.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.116+0000 7f331fa4b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f331811cac0 con 0x7f331810b270 2024-05-27T07:51:42.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.116+0000 7f331fa4b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f331811cdc0 con 0x7f3318111bc0 2024-05-27T07:51:42.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.116+0000 7f331d7e7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f331810a920 0x7f331810e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:42.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.116+0000 7f331d7e7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f331810a920 0x7f331810e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49338/0 (socket says 172.21.15.114:49338) 2024-05-27T07:51:42.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.116+0000 7f331d7e7700 1 -- 172.21.15.114:0/3326181033 learned_addr learned my addr 172.21.15.114:0/3326181033 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:42.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.116+0000 7f331cfe6700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f331810b270 0x7f331810c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:42.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.117+0000 7f331dfe8700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3318111bc0 0x7f331810cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:42.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.117+0000 7f331cfe6700 1 -- 172.21.15.114:0/3326181033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3318111bc0 msgr2=0x7f331810cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.117+0000 7f331cfe6700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3318111bc0 0x7f331810cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.117+0000 7f331cfe6700 1 -- 172.21.15.114:0/3326181033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f331810a920 msgr2=0x7f331810e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.117+0000 7f331cfe6700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f331810a920 0x7f331810e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.117+0000 7f331cfe6700 1 -- 172.21.15.114:0/3326181033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f331810d480 con 0x7f331810b270 2024-05-27T07:51:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.117+0000 7f331cfe6700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f331810b270 0x7f331810c670 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f33080059d0 tx=0x7f33080047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.118+0000 7f330e7fc700 1 -- 172.21.15.114:0/3326181033 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3308027070 con 0x7f331810b270 2024-05-27T07:51:42.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.118+0000 7f330e7fc700 1 -- 172.21.15.114:0/3326181033 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3308022470 con 0x7f331810b270 2024-05-27T07:51:42.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.118+0000 7f330e7fc700 1 -- 172.21.15.114:0/3326181033 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3308017650 con 0x7f331810b270 2024-05-27T07:51:42.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.119+0000 7f331fa4b700 1 -- 172.21.15.114:0/3326181033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33181b5fe0 con 0x7f331810b270 2024-05-27T07:51:42.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.120+0000 7f331fa4b700 1 -- 172.21.15.114:0/3326181033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33181b6380 con 0x7f331810b270 2024-05-27T07:51:42.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.121+0000 7f331fa4b700 1 -- 172.21.15.114:0/3326181033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f331810c060 con 0x7f331810b270 2024-05-27T07:51:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.123+0000 7f330e7fc700 1 -- 172.21.15.114:0/3326181033 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f3308009d10 con 0x7f331810b270 2024-05-27T07:51:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.124+0000 7f330e7fc700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f33040b2b40 0x7f33040b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.125+0000 7f331d7e7700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f33040b2b40 0x7f33040b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.125+0000 7f330e7fc700 1 -- 172.21.15.114:0/3326181033 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f33080e3a10 con 0x7f331810b270 2024-05-27T07:51:42.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.125+0000 7f331d7e7700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f33040b2b40 0x7f33040b5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3314005950 tx=0x7f3314009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:42.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.128+0000 7f330e7fc700 1 -- 172.21.15.114:0/3326181033 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3308030070 con 0x7f331810b270 2024-05-27T07:51:42.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1165409 9.81 MB/sec execute 474 sec latency 65.191 ms 2024-05-27T07:51:42.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.420+0000 7f331fa4b700 1 -- 172.21.15.114:0/3326181033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f331802e700 con 0x7f331810b270 2024-05-27T07:51:42.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.421+0000 7f330e7fc700 1 -- 172.21.15.114:0/3326181033 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+8522 (secure 0 0 0) 0x7f3308013070 con 0x7f331810b270 2024-05-27T07:51:42.422 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:51:42.422 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":27557,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3261223371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":3261223371},{"type":"v1","addr":"172.21.15.163:6835","nonce":3261223371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":67,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:51:26.564380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27634":{"gid":27634,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/3266101988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":3266101988},{"type":"v1","addr":"172.21.15.186:6833","nonce":3266101988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:51:42.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.423+0000 7f331fa4b700 1 -- 172.21.15.114:0/3326181033 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f33040b2b40 msgr2=0x7f33040b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:42.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.423+0000 7f331fa4b700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f33040b2b40 0x7f33040b5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3314005950 tx=0x7f3314009260 comp rx=0 tx=0).stop 2024-05-27T07:51:42.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.423+0000 7f331fa4b700 1 -- 172.21.15.114:0/3326181033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f331810b270 msgr2=0x7f331810c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:42.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.423+0000 7f331fa4b700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f331810b270 0x7f331810c670 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f33080059d0 tx=0x7f33080047b0 comp rx=0 tx=0).stop 2024-05-27T07:51:42.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.424+0000 7f331fa4b700 1 -- 172.21.15.114:0/3326181033 shutdown_connections 2024-05-27T07:51:42.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.424+0000 7f331fa4b700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3318111bc0 0x7f331810cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:42.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.424+0000 7f331fa4b700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f331810a920 0x7f331810e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:42.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.424+0000 7f331fa4b700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f33040b2b40 0x7f33040b5050 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:42.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.424+0000 7f331fa4b700 1 --2- 172.21.15.114:0/3326181033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f331810b270 0x7f331810c670 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:42.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.424+0000 7f331fa4b700 1 -- 172.21.15.114:0/3326181033 >> 172.21.15.114:0/3326181033 conn(0x7f331806f4c0 msgr2=0x7f3318110630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:42.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.424+0000 7f331fa4b700 1 -- 172.21.15.114:0/3326181033 shutdown_connections 2024-05-27T07:51:42.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:42.424+0000 7f331fa4b700 1 -- 172.21.15.114:0/3326181033 wait complete. 2024-05-27T07:51:42.426 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 67 2024-05-27T07:51:42.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:42 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3326181033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:51:42.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:42 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3326181033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:51:42.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:42 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3326181033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:51:43.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:51:43.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1166860 9.80 MB/sec execute 475 sec latency 38.470 ms 2024-05-27T07:51:43.591 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:43.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:43 smithi114 ceph-mon[32038]: pgmap v398: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 345 op/s 2024-05-27T07:51:43.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:43 smithi163 ceph-mon[36451]: pgmap v398: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 345 op/s 2024-05-27T07:51:43.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:43 smithi186 ceph-mon[36647]: pgmap v398: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 345 op/s 2024-05-27T07:51:44.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1168708 9.80 MB/sec execute 476 sec latency 70.605 ms 2024-05-27T07:51:45.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1170272 9.80 MB/sec execute 477 sec latency 16.126 ms 2024-05-27T07:51:45.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.300+0000 7f60a70a5700 1 -- 172.21.15.114:0/3866891402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a01034b0 msgr2=0x7f60a01038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:45.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.300+0000 7f60a70a5700 1 --2- 172.21.15.114:0/3866891402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a01034b0 0x7f60a01038c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f60940098f0 tx=0x7f6094009c00 comp rx=0 tx=0).stop 2024-05-27T07:51:45.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.300+0000 7f60a70a5700 1 -- 172.21.15.114:0/3866891402 shutdown_connections 2024-05-27T07:51:45.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.300+0000 7f60a70a5700 1 --2- 172.21.15.114:0/3866891402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a0108730 0x7f60a0110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.300+0000 7f60a70a5700 1 --2- 172.21.15.114:0/3866891402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a01034b0 0x7f60a01038c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.300+0000 7f60a70a5700 1 --2- 172.21.15.114:0/3866891402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a0107de0 0x7f60a01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.300+0000 7f60a70a5700 1 -- 172.21.15.114:0/3866891402 >> 172.21.15.114:0/3866891402 conn(0x7f60a00fedb0 msgr2=0x7f60a01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:45.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.300+0000 7f60a70a5700 1 -- 172.21.15.114:0/3866891402 shutdown_connections 2024-05-27T07:51:45.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.300+0000 7f60a70a5700 1 -- 172.21.15.114:0/3866891402 wait complete. 2024-05-27T07:51:45.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.301+0000 7f60a70a5700 1 Processor -- start 2024-05-27T07:51:45.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.301+0000 7f60a70a5700 1 -- start start 2024-05-27T07:51:45.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.301+0000 7f60a70a5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a01034b0 0x7f60a019bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a70a5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a0107de0 0x7f60a019c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a70a5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a0108730 0x7f60a019c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a70a5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f60a0113360 con 0x7f60a0108730 2024-05-27T07:51:45.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a70a5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f60a01131e0 con 0x7f60a0107de0 2024-05-27T07:51:45.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a70a5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f60a01134e0 con 0x7f60a01034b0 2024-05-27T07:51:45.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a4e41700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a01034b0 0x7f60a019bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a5642700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a0108730 0x7f60a019c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a5642700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a0108730 0x7f60a019c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58000/0 (socket says 172.21.15.114:58000) 2024-05-27T07:51:45.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a5642700 1 -- 172.21.15.114:0/3489449037 learned_addr learned my addr 172.21.15.114:0/3489449037 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:45.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f609ffff700 1 --2- 172.21.15.114:0/3489449037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a0107de0 0x7f60a019c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a4e41700 1 -- 172.21.15.114:0/3489449037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a0107de0 msgr2=0x7f60a019c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:45.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a4e41700 1 --2- 172.21.15.114:0/3489449037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a0107de0 0x7f60a019c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a4e41700 1 -- 172.21.15.114:0/3489449037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a0108730 msgr2=0x7f60a019c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:45.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a4e41700 1 --2- 172.21.15.114:0/3489449037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a0108730 0x7f60a019c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a4e41700 1 -- 172.21.15.114:0/3489449037 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60a01a7090 con 0x7f60a01034b0 2024-05-27T07:51:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f60a4e41700 1 --2- 172.21.15.114:0/3489449037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a01034b0 0x7f60a019bd20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6094000c00 tx=0x7f609400fa20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.302+0000 7f609dffb700 1 -- 172.21.15.114:0/3489449037 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60940050c0 con 0x7f60a01034b0 2024-05-27T07:51:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.303+0000 7f609dffb700 1 -- 172.21.15.114:0/3489449037 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f609400b560 con 0x7f60a01034b0 2024-05-27T07:51:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.303+0000 7f609dffb700 1 -- 172.21.15.114:0/3489449037 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60940038e0 con 0x7f60a01034b0 2024-05-27T07:51:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.303+0000 7f60a70a5700 1 -- 172.21.15.114:0/3489449037 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60a01a7320 con 0x7f60a01034b0 2024-05-27T07:51:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.303+0000 7f60a70a5700 1 -- 172.21.15.114:0/3489449037 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60a01a7720 con 0x7f60a01034b0 2024-05-27T07:51:45.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.306+0000 7f609dffb700 1 -- 172.21.15.114:0/3489449037 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6094005260 con 0x7f60a01034b0 2024-05-27T07:51:45.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f60b5abab80 1 Processor -- start 2024-05-27T07:51:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f60b5abab80 1 -- start start 2024-05-27T07:51:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f60b5abab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6f9abe40 0x558a6fb2f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f60b5abab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb29a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f60b5abab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6fb29f80 0x558a6fb2a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f60b5abab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558a6fa58c90 con 0x558a6f9abe40 2024-05-27T07:51:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f60b5abab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558a6fa48830 con 0x558a6fb29f80 2024-05-27T07:51:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f60b5abab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558a6fa4c870 con 0x558a6fb2fc50 2024-05-27T07:51:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f609dffb700 1 --2- 172.21.15.114:0/3489449037 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60900b29f0 0x7f60900b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f60a5642700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6f9abe40 0x558a6fb2f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f60a5642700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6f9abe40 0x558a6fb2f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58008/0 (socket says 172.21.15.114:58008) 2024-05-27T07:51:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f60a5642700 1 -- 172.21.15.114:0/1164470653 learned_addr learned my addr 172.21.15.114:0/1164470653 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f60a5642700 1 --2- 172.21.15.114:0/3489449037 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60900b29f0 0x7f60900b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f60a4e41700 1 --2- 172.21.15.114:0/1164470653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb29a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f609ffff700 1 --2- 172.21.15.114:0/1164470653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6fb29f80 0x558a6fb2a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.307+0000 7f609dffb700 1 -- 172.21.15.114:0/3489449037 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f60940e32d0 con 0x7f60a01034b0 2024-05-27T07:51:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.308+0000 7f60a5642700 1 -- 172.21.15.114:0/1164470653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 msgr2=0x558a6fb29a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.308+0000 7f60a5642700 1 --2- 172.21.15.114:0/1164470653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb29a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.308+0000 7f60a5642700 1 -- 172.21.15.114:0/1164470653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6fb29f80 msgr2=0x558a6fb2a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.308+0000 7f60a5642700 1 --2- 172.21.15.114:0/1164470653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6fb29f80 0x558a6fb2a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.308+0000 7f60a5642700 1 -- 172.21.15.114:0/1164470653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a6f9988d0 con 0x558a6f9abe40 2024-05-27T07:51:45.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.308+0000 7f60a5642700 1 --2- 172.21.15.114:0/1164470653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6f9abe40 0x558a6fb2f710 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f609801a490 tx=0x7f609801a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ade7b1a3d58517 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:45.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.308+0000 7f60a5642700 1 --2- 172.21.15.114:0/3489449037 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60900b29f0 0x7f60900b4f00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6098009550 tx=0x7f609801e000 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.308+0000 7f608affd700 1 -- 172.21.15.114:0/1164470653 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60980086d0 con 0x558a6f9abe40 2024-05-27T07:51:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.308+0000 7f608affd700 1 -- 172.21.15.114:0/1164470653 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f609801f070 con 0x558a6f9abe40 2024-05-27T07:51:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.308+0000 7f608affd700 1 -- 172.21.15.114:0/1164470653 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6098004020 con 0x558a6f9abe40 2024-05-27T07:51:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.309+0000 7f60b5abab80 1 -- 172.21.15.114:0/1164470653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6f9abe40 msgr2=0x558a6fb2f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.309+0000 7f60b5abab80 1 --2- 172.21.15.114:0/1164470653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6f9abe40 0x558a6fb2f710 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f609801a490 tx=0x7f609801a7a0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.309+0000 7f60b5abab80 1 -- 172.21.15.114:0/1164470653 shutdown_connections 2024-05-27T07:51:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.309+0000 7f60b5abab80 1 --2- 172.21.15.114:0/1164470653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6fb29f80 0x558a6fb2a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.309+0000 7f60b5abab80 1 --2- 172.21.15.114:0/1164470653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6f9abe40 0x558a6fb2f710 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.309+0000 7f60b5abab80 1 --2- 172.21.15.114:0/1164470653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb29a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.309+0000 7f60b5abab80 1 -- 172.21.15.114:0/1164470653 >> 172.21.15.114:0/1164470653 conn(0x558a6fa06af0 msgr2=0x558a6fb24640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.309+0000 7f60b5abab80 1 -- 172.21.15.114:0/1164470653 shutdown_connections 2024-05-27T07:51:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.309+0000 7f60b5abab80 1 -- 172.21.15.114:0/1164470653 wait complete. 2024-05-27T07:51:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f60b5abab80 1 -- start start 2024-05-27T07:51:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f60b5abab80 1 Processor -- start 2024-05-27T07:51:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f60b5abab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb33440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f60b5abab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb33980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f60b5abab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb33ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f60b5abab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558a6f94efa0 con 0x558a6fb29f80 2024-05-27T07:51:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f60b5abab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558a6f9ea7c0 con 0x558a6f9abe40 2024-05-27T07:51:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f60b5abab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558a6fb366e0 con 0x558a6fb2fc50 2024-05-27T07:51:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f60b5abab80 10 client.0 _ll_register_callbacks cb 0x558a6f9e3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f609ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb33980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f609ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb33980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58020/0 (socket says 172.21.15.114:58020) 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f609ffff700 1 -- 172.21.15.114:0/2986446487 learned_addr learned my addr 172.21.15.114:0/2986446487 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f60a4e41700 1 --2- 172.21.15.114:0/2986446487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb33440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.310+0000 7f60a5642700 1 --2- 172.21.15.114:0/2986446487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb33ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f609ffff700 1 -- 172.21.15.114:0/2986446487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 msgr2=0x558a6fb33ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f609ffff700 1 --2- 172.21.15.114:0/2986446487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb33ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f609ffff700 1 -- 172.21.15.114:0/2986446487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 msgr2=0x558a6fb33440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f609ffff700 1 --2- 172.21.15.114:0/2986446487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb33440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f609ffff700 1 -- 172.21.15.114:0/2986446487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a6f9e7690 con 0x558a6fb29f80 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f609ffff700 1 --2- 172.21.15.114:0/2986446487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb33980 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f608c012700 tx=0x7f608c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41079b07b0c4b01 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f607ffff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f607ffff700 1 -- 172.21.15.114:0/2986446487 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c004050 con 0x558a6fb29f80 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f607ffff700 1 -- 172.21.15.114:0/2986446487 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f608c0041f0 con 0x558a6fb29f80 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f60b5abab80 1 -- 172.21.15.114:0/2986446487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f609801f070 con 0x558a6fb29f80 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f607ffff700 1 -- 172.21.15.114:0/2986446487 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f608c025050 con 0x558a6fb29f80 2024-05-27T07:51:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f60b5abab80 10 client.18228 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f60b5abab80 1 -- 172.21.15.114:0/2986446487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x558a6fb412a0 con 0x558a6fb29f80 2024-05-27T07:51:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.311+0000 7f607ffff700 1 -- 172.21.15.114:0/2986446487 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f609801f070 con 0x558a6fb29f80 2024-05-27T07:51:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.312+0000 7f607ffff700 1 -- 172.21.15.114:0/2986446487 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f608c0104c0 con 0x558a6fb29f80 2024-05-27T07:51:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.312+0000 7f60b5abab80 10 client.18228 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.312+0000 7f60b5abab80 10 client.18228 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.312+0000 7f60b5abab80 20 client.18228 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.312+0000 7f60b5abab80 1 --2- 172.21.15.114:0/2986446487 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558a6fb43af0 0x558a6fb45fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.312+0000 7f60b5abab80 4 client.18228 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.312+0000 7f60b5abab80 1 -- 172.21.15.114:0/2986446487 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a6faf59f0 con 0x558a6fb43af0 2024-05-27T07:51:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.312+0000 7f60a4e41700 1 --2- 172.21.15.114:0/2986446487 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558a6fb43af0 0x558a6fb45fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.312+0000 7f607ffff700 10 client.18228 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.313+0000 7f60a4e41700 1 --2- 172.21.15.114:0/2986446487 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558a6fb43af0 0x558a6fb45fa0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f1cfb2b32e8f32d server_cookie=2beac85cd666323b in_seq=0 out_seq=0 2024-05-27T07:51:45.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f607ffff700 1 -- 172.21.15.114:0/2986446487 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558a6faf59f0 con 0x558a6fb43af0 2024-05-27T07:51:45.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f607ffff700 10 client.18228 handle_command_reply: tid=0 2024-05-27T07:51:45.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f60b5abab80 1 client.18228 shutdown 2024-05-27T07:51:45.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f60b5abab80 1 -- 172.21.15.114:0/2986446487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 msgr2=0x558a6fb33980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f60b5abab80 1 --2- 172.21.15.114:0/2986446487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb33980 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f608c012700 tx=0x7f608c009f90 comp rx=0 tx=0).stop 2024-05-27T07:51:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f60b5abab80 1 -- 172.21.15.114:0/2986446487 shutdown_connections 2024-05-27T07:51:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f60b5abab80 1 --2- 172.21.15.114:0/2986446487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb33ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f60b5abab80 1 --2- 172.21.15.114:0/2986446487 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558a6fb43af0 0x558a6fb45fa0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f607ffff700 0 client.18228 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f60b5abab80 1 --2- 172.21.15.114:0/2986446487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb33440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f60b5abab80 1 --2- 172.21.15.114:0/2986446487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb33980 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f60b5abab80 1 -- 172.21.15.114:0/2986446487 >> 172.21.15.114:0/2986446487 conn(0x558a6fa06af0 msgr2=0x558a6fb28a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f60b5abab80 1 -- 172.21.15.114:0/2986446487 shutdown_connections 2024-05-27T07:51:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.314+0000 7f60b5abab80 1 -- 172.21.15.114:0/2986446487 wait complete. 2024-05-27T07:51:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.315+0000 7f60b5abab80 20 client.18228 trim_cache size 0 max 16384 2024-05-27T07:51:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.330+0000 7f60b5abab80 1 Processor -- start 2024-05-27T07:51:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.330+0000 7f60b5abab80 1 -- start start 2024-05-27T07:51:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.330+0000 7f60b5abab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb29a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.330+0000 7f60b5abab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb31070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.330+0000 7f60b5abab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb2eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.330+0000 7f60b5abab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f60a01afc40 con 0x558a6fb29f80 2024-05-27T07:51:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.330+0000 7f60b5abab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f60a01ae3b0 con 0x558a6f9abe40 2024-05-27T07:51:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.330+0000 7f60b5abab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f60a01ac200 con 0x558a6fb2fc50 2024-05-27T07:51:45.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.330+0000 7f609ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb31070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.330+0000 7f609ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb31070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58026/0 (socket says 172.21.15.114:58026) 2024-05-27T07:51:45.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.330+0000 7f60a4e41700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb29a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.330+0000 7f609ffff700 1 -- 172.21.15.114:0/3208830591 learned_addr learned my addr 172.21.15.114:0/3208830591 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:45.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.331+0000 7f60a5642700 1 --2- 172.21.15.114:0/3208830591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb2eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.331+0000 7f60a4e41700 1 -- 172.21.15.114:0/3208830591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 msgr2=0x558a6fb2eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:45.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.331+0000 7f60a4e41700 1 --2- 172.21.15.114:0/3208830591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb2eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.331+0000 7f60a4e41700 1 -- 172.21.15.114:0/3208830591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 msgr2=0x558a6fb31070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:45.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.331+0000 7f60a4e41700 1 --2- 172.21.15.114:0/3208830591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb31070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.331+0000 7f60a4e41700 1 -- 172.21.15.114:0/3208830591 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a6fb36240 con 0x558a6f9abe40 2024-05-27T07:51:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.331+0000 7f60a4e41700 1 --2- 172.21.15.114:0/3208830591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb29a40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f60940e0610 tx=0x7f6094019040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ce920aba71cdb9f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.331+0000 7f608a7fc700 1 -- 172.21.15.114:0/3208830591 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60940a9d40 con 0x558a6f9abe40 2024-05-27T07:51:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.331+0000 7f608a7fc700 1 -- 172.21.15.114:0/3208830591 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60940aa930 con 0x558a6f9abe40 2024-05-27T07:51:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.331+0000 7f608a7fc700 1 -- 172.21.15.114:0/3208830591 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60940570b0 con 0x558a6f9abe40 2024-05-27T07:51:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.332+0000 7f60b5abab80 1 -- 172.21.15.114:0/3208830591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 msgr2=0x558a6fb29a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.332+0000 7f60b5abab80 1 --2- 172.21.15.114:0/3208830591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb29a40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f60940e0610 tx=0x7f6094019040 comp rx=0 tx=0).stop 2024-05-27T07:51:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.332+0000 7f60b5abab80 1 -- 172.21.15.114:0/3208830591 shutdown_connections 2024-05-27T07:51:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.332+0000 7f60b5abab80 1 --2- 172.21.15.114:0/3208830591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb2eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.332+0000 7f60b5abab80 1 --2- 172.21.15.114:0/3208830591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb29a40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.332+0000 7f60b5abab80 1 --2- 172.21.15.114:0/3208830591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb31070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.332+0000 7f60b5abab80 1 -- 172.21.15.114:0/3208830591 >> 172.21.15.114:0/3208830591 conn(0x558a6fa06af0 msgr2=0x558a6fb2e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:45.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.332+0000 7f60b5abab80 1 -- 172.21.15.114:0/3208830591 shutdown_connections 2024-05-27T07:51:45.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.332+0000 7f60b5abab80 1 -- 172.21.15.114:0/3208830591 wait complete. 2024-05-27T07:51:45.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f60b5abab80 1 -- start start 2024-05-27T07:51:45.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f60b5abab80 1 Processor -- start 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f60b5abab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb3c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f60b5abab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb4af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f60b5abab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb4b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f60b5abab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f60a01ac870 con 0x558a6fb29f80 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f60b5abab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f60a01adce0 con 0x558a6f9abe40 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f60b5abab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f60a01acf40 con 0x558a6fb2fc50 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f60b5abab80 10 client.0 _ll_register_callbacks cb 0x558a6f9e3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f609ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb4af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f60a4e41700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb3c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f609ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb4af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58038/0 (socket says 172.21.15.114:58038) 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f609ffff700 1 -- 172.21.15.114:0/1928835108 learned_addr learned my addr 172.21.15.114:0/1928835108 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.333+0000 7f60a5642700 1 --2- 172.21.15.114:0/1928835108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb4b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.334+0000 7f60a4e41700 1 -- 172.21.15.114:0/1928835108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 msgr2=0x558a6fb4b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.334+0000 7f60a4e41700 1 --2- 172.21.15.114:0/1928835108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb4b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.334+0000 7f60a4e41700 1 -- 172.21.15.114:0/1928835108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 msgr2=0x558a6fb4af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.334+0000 7f60a4e41700 1 --2- 172.21.15.114:0/1928835108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb4af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.334+0000 7f60a4e41700 1 -- 172.21.15.114:0/1928835108 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a6fb4bb50 con 0x558a6f9abe40 2024-05-27T07:51:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.334+0000 7f60a4e41700 1 --2- 172.21.15.114:0/1928835108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb3c040 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f60940aaae0 tx=0x7f60940e31d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d02d6ce7ebdf9917 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.334+0000 7f607ffff700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:51:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.334+0000 7f607ffff700 1 -- 172.21.15.114:0/1928835108 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60940c3070 con 0x558a6f9abe40 2024-05-27T07:51:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.334+0000 7f607ffff700 1 -- 172.21.15.114:0/1928835108 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f60940570d0 con 0x558a6f9abe40 2024-05-27T07:51:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.334+0000 7f60b5abab80 1 -- 172.21.15.114:0/1928835108 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60940aa930 con 0x558a6f9abe40 2024-05-27T07:51:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.334+0000 7f607ffff700 1 -- 172.21.15.114:0/1928835108 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f609406f050 con 0x558a6f9abe40 2024-05-27T07:51:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.334+0000 7f60b5abab80 10 client.27656 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.334+0000 7f60b5abab80 1 -- 172.21.15.114:0/1928835108 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x558a6fbae310 con 0x558a6f9abe40 2024-05-27T07:51:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.335+0000 7f607ffff700 1 -- 172.21.15.114:0/1928835108 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60940aa930 con 0x558a6f9abe40 2024-05-27T07:51:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.335+0000 7f607ffff700 1 -- 172.21.15.114:0/1928835108 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f60940ae170 con 0x558a6f9abe40 2024-05-27T07:51:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.335+0000 7f60b5abab80 10 client.27656 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.335+0000 7f60b5abab80 10 client.27656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.335+0000 7f60b5abab80 20 client.27656 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.335+0000 7f60b5abab80 1 --2- 172.21.15.114:0/1928835108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558a6fb43af0 0x558a6fbb0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.335+0000 7f60b5abab80 4 client.27656 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:51:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.335+0000 7f60b5abab80 1 -- 172.21.15.114:0/1928835108 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558a6fa5abe0 con 0x558a6fb43af0 2024-05-27T07:51:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.335+0000 7f609ffff700 1 --2- 172.21.15.114:0/1928835108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558a6fb43af0 0x558a6fbb0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.336+0000 7f607ffff700 10 client.27656 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.336+0000 7f609ffff700 1 --2- 172.21.15.114:0/1928835108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558a6fb43af0 0x558a6fbb0960 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b49b86d9cb204a1 server_cookie=d5041ae62cf7251f in_seq=0 out_seq=0 2024-05-27T07:51:45.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f607ffff700 1 -- 172.21.15.114:0/1928835108 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558a6fa5abe0 con 0x558a6fb43af0 2024-05-27T07:51:45.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f607ffff700 10 client.27656 handle_command_reply: tid=0 2024-05-27T07:51:45.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f60b5abab80 1 client.27656 shutdown 2024-05-27T07:51:45.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f60b5abab80 1 -- 172.21.15.114:0/1928835108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 msgr2=0x558a6fb3c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:45.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f60b5abab80 1 --2- 172.21.15.114:0/1928835108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb3c040 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f60940aaae0 tx=0x7f60940e31d0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f60b5abab80 1 -- 172.21.15.114:0/1928835108 shutdown_connections 2024-05-27T07:51:45.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f60b5abab80 1 --2- 172.21.15.114:0/1928835108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a6fb2fc50 0x558a6fb4b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f60b5abab80 1 --2- 172.21.15.114:0/1928835108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558a6fb43af0 0x558a6fbb0960 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f60b5abab80 1 --2- 172.21.15.114:0/1928835108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6f9abe40 0x558a6fb3c040 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f607ffff700 0 client.27656 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:45.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f60b5abab80 1 --2- 172.21.15.114:0/1928835108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558a6fb29f80 0x558a6fb4af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f60b5abab80 1 -- 172.21.15.114:0/1928835108 >> 172.21.15.114:0/1928835108 conn(0x558a6fa06af0 msgr2=0x558a6fb333e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:45.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f60b5abab80 1 -- 172.21.15.114:0/1928835108 shutdown_connections 2024-05-27T07:51:45.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f60b5abab80 1 -- 172.21.15.114:0/1928835108 wait complete. 2024-05-27T07:51:45.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.337+0000 7f60b5abab80 20 client.27656 trim_cache size 0 max 16384 2024-05-27T07:51:45.341 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:51:45.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.338+0000 7f60a70a5700 1 -- 172.21.15.114:0/3489449037 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60900b29f0 msgr2=0x7f60900b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:45.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.338+0000 7f60a70a5700 1 --2- 172.21.15.114:0/3489449037 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60900b29f0 0x7f60900b4f00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6098009550 tx=0x7f609801e000 comp rx=0 tx=0).stop 2024-05-27T07:51:45.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.338+0000 7f60a70a5700 1 -- 172.21.15.114:0/3489449037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a01034b0 msgr2=0x7f60a019bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:45.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.338+0000 7f60a70a5700 1 --2- 172.21.15.114:0/3489449037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a01034b0 0x7f60a019bd20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6094000c00 tx=0x7f609400fa20 comp rx=0 tx=0).stop 2024-05-27T07:51:45.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.338+0000 7f60a70a5700 1 -- 172.21.15.114:0/3489449037 shutdown_connections 2024-05-27T07:51:45.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.338+0000 7f60a70a5700 1 --2- 172.21.15.114:0/3489449037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a0108730 0x7f60a019c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.338+0000 7f60a70a5700 1 --2- 172.21.15.114:0/3489449037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a01034b0 0x7f60a019bd20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.338+0000 7f60a70a5700 1 --2- 172.21.15.114:0/3489449037 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f60900b29f0 0x7f60900b4f00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.338+0000 7f60a70a5700 1 --2- 172.21.15.114:0/3489449037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a0107de0 0x7f60a019c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:45.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.338+0000 7f60a70a5700 1 -- 172.21.15.114:0/3489449037 >> 172.21.15.114:0/3489449037 conn(0x7f60a00fedb0 msgr2=0x7f60a01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:45.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.338+0000 7f60a70a5700 1 -- 172.21.15.114:0/3489449037 shutdown_connections 2024-05-27T07:51:45.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:45.338+0000 7f60a70a5700 1 -- 172.21.15.114:0/3489449037 wait complete. 2024-05-27T07:51:45.861 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:45 smithi114 ceph-mon[32038]: pgmap v399: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 257 op/s 2024-05-27T07:51:45.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:45 smithi163 ceph-mon[36451]: pgmap v399: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 257 op/s 2024-05-27T07:51:45.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:45 smithi186 ceph-mon[36647]: pgmap v399: 129 pgs: 129 active+clean; 69 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 257 op/s 2024-05-27T07:51:46.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1172372 9.80 MB/sec execute 478 sec latency 41.789 ms 2024-05-27T07:51:47.019 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:51:47.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:51:47.277 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:47.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1174091 9.80 MB/sec execute 479 sec latency 27.642 ms 2024-05-27T07:51:47.736 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:47 smithi114 ceph-mon[32038]: pgmap v400: 129 pgs: 129 active+clean; 75 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 320 op/s 2024-05-27T07:51:47.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:47 smithi163 ceph-mon[36451]: pgmap v400: 129 pgs: 129 active+clean; 75 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 320 op/s 2024-05-27T07:51:47.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:47 smithi186 ceph-mon[36647]: pgmap v400: 129 pgs: 129 active+clean; 75 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 320 op/s 2024-05-27T07:51:48.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1175795 9.80 MB/sec execute 480 sec latency 50.821 ms 2024-05-27T07:51:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.111+0000 7f0788e0b700 1 -- 172.21.15.114:0/2886945621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07841034d0 msgr2=0x7f07841038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.111+0000 7f0788e0b700 1 --2- 172.21.15.114:0/2886945621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07841034d0 0x7f07841038e0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f077400b0d0 tx=0x7f077400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.111+0000 7f0788e0b700 1 -- 172.21.15.114:0/2886945621 shutdown_connections 2024-05-27T07:51:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.111+0000 7f0788e0b700 1 --2- 172.21.15.114:0/2886945621 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0784108750 0x7f07841100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.111+0000 7f0788e0b700 1 --2- 172.21.15.114:0/2886945621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07841034d0 0x7f07841038e0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.111+0000 7f0788e0b700 1 --2- 172.21.15.114:0/2886945621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0784107e00 0x7f0784108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.111+0000 7f0788e0b700 1 -- 172.21.15.114:0/2886945621 >> 172.21.15.114:0/2886945621 conn(0x7f07840fef20 msgr2=0x7f0784101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.111+0000 7f0788e0b700 1 -- 172.21.15.114:0/2886945621 shutdown_connections 2024-05-27T07:51:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.111+0000 7f0788e0b700 1 -- 172.21.15.114:0/2886945621 wait complete. 2024-05-27T07:51:49.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f0788e0b700 1 Processor -- start 2024-05-27T07:51:49.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f0788e0b700 1 -- start start 2024-05-27T07:51:49.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f0788e0b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07841034d0 0x7f0784199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f0788e0b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0784107e00 0x7f078419a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f0788e0b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0784108750 0x7f078419e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f0788e0b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0784113380 con 0x7f07841034d0 2024-05-27T07:51:49.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f0788e0b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0784113200 con 0x7f0784107e00 2024-05-27T07:51:49.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f0788e0b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0784113500 con 0x7f0784108750 2024-05-27T07:51:49.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f078259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07841034d0 0x7f0784199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f078259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07841034d0 0x7f0784199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58070/0 (socket says 172.21.15.114:58070) 2024-05-27T07:51:49.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f078259c700 1 -- 172.21.15.114:0/1152706457 learned_addr learned my addr 172.21.15.114:0/1152706457 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:49.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f0782d9d700 1 --2- 172.21.15.114:0/1152706457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0784108750 0x7f078419e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f078259c700 1 -- 172.21.15.114:0/1152706457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0784108750 msgr2=0x7f078419e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f078259c700 1 --2- 172.21.15.114:0/1152706457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0784108750 0x7f078419e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.112+0000 7f078259c700 1 -- 172.21.15.114:0/1152706457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0784107e00 msgr2=0x7f078419a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.113+0000 7f078259c700 1 --2- 172.21.15.114:0/1152706457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0784107e00 0x7f078419a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.113+0000 7f078259c700 1 -- 172.21.15.114:0/1152706457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078419eb50 con 0x7f07841034d0 2024-05-27T07:51:49.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.113+0000 7f078259c700 1 --2- 172.21.15.114:0/1152706457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07841034d0 0x7f0784199d30 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f0774007a20 tx=0x7f0774007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:49.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.113+0000 7f077b7fe700 1 -- 172.21.15.114:0/1152706457 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774004c50 con 0x7f07841034d0 2024-05-27T07:51:49.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.113+0000 7f077b7fe700 1 -- 172.21.15.114:0/1152706457 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0774004df0 con 0x7f07841034d0 2024-05-27T07:51:49.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.113+0000 7f0788e0b700 1 -- 172.21.15.114:0/1152706457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078419ede0 con 0x7f07841034d0 2024-05-27T07:51:49.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.113+0000 7f077b7fe700 1 -- 172.21.15.114:0/1152706457 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774005610 con 0x7f07841034d0 2024-05-27T07:51:49.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.113+0000 7f0788e0b700 1 -- 172.21.15.114:0/1152706457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f078419f2c0 con 0x7f07841034d0 2024-05-27T07:51:49.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.116+0000 7f077b7fe700 1 -- 172.21.15.114:0/1152706457 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f07740057b0 con 0x7f07841034d0 2024-05-27T07:51:49.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.117+0000 7f077b7fe700 1 --2- 172.21.15.114:0/1152706457 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f07700b29f0 0x7f07700b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.117+0000 7f0781d9b700 1 --2- 172.21.15.114:0/1152706457 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f07700b29f0 0x7f07700b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.117+0000 7f077b7fe700 1 -- 172.21.15.114:0/1152706457 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f077400f030 con 0x7f07841034d0 2024-05-27T07:51:49.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.117+0000 7f0797820b80 1 Processor -- start 2024-05-27T07:51:49.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.117+0000 7f0797820b80 1 -- start start 2024-05-27T07:51:49.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.117+0000 7f0781d9b700 1 --2- 172.21.15.114:0/1152706457 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f07700b29f0 0x7f07700b4f00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f076c005950 tx=0x7f076c0058e0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f0797820b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a2560ec0 0x5591a26e4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f0797820b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26deac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f0797820b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26df000 0x5591a26df410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f0797820b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5591a260dd10 con 0x5591a26e4cd0 2024-05-27T07:51:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f0797820b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5591a25fd8b0 con 0x5591a26df000 2024-05-27T07:51:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f0797820b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5591a26018f0 con 0x5591a2560ec0 2024-05-27T07:51:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f0781d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26deac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f078259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a2560ec0 0x5591a26e4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f078259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a2560ec0 0x5591a26e4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47190/0 (socket says 172.21.15.114:47190) 2024-05-27T07:51:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f078259c700 1 -- 172.21.15.114:0/2966769848 learned_addr learned my addr 172.21.15.114:0/2966769848 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f0782d9d700 1 --2- 172.21.15.114:0/2966769848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26df000 0x5591a26df410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f078259c700 1 -- 172.21.15.114:0/2966769848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26df000 msgr2=0x5591a26df410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f078259c700 1 --2- 172.21.15.114:0/2966769848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26df000 0x5591a26df410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f078259c700 1 -- 172.21.15.114:0/2966769848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 msgr2=0x5591a26deac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f078259c700 1 --2- 172.21.15.114:0/2966769848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26deac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f078259c700 1 -- 172.21.15.114:0/2966769848 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591a254d950 con 0x5591a2560ec0 2024-05-27T07:51:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f0781d9b700 1 --2- 172.21.15.114:0/2966769848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26deac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f078259c700 1 --2- 172.21.15.114:0/2966769848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a2560ec0 0x5591a26e4790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f07740ac6a0 tx=0x7f07740ac3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18fb50cfd3002767 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.118+0000 7f0778ff9700 1 -- 172.21.15.114:0/2966769848 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774005aa0 con 0x5591a2560ec0 2024-05-27T07:51:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.119+0000 7f0778ff9700 1 -- 172.21.15.114:0/2966769848 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07740ac4b0 con 0x5591a2560ec0 2024-05-27T07:51:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.119+0000 7f0778ff9700 1 -- 172.21.15.114:0/2966769848 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07740b5770 con 0x5591a2560ec0 2024-05-27T07:51:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.119+0000 7f0797820b80 1 -- 172.21.15.114:0/2966769848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a2560ec0 msgr2=0x5591a26e4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.119+0000 7f0797820b80 1 --2- 172.21.15.114:0/2966769848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a2560ec0 0x5591a26e4790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f07740ac6a0 tx=0x7f07740ac3f0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.119+0000 7f0797820b80 1 -- 172.21.15.114:0/2966769848 shutdown_connections 2024-05-27T07:51:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.119+0000 7f0797820b80 1 --2- 172.21.15.114:0/2966769848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26df000 0x5591a26df410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.119+0000 7f0797820b80 1 --2- 172.21.15.114:0/2966769848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a2560ec0 0x5591a26e4790 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.119+0000 7f0797820b80 1 --2- 172.21.15.114:0/2966769848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26deac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.119+0000 7f0797820b80 1 -- 172.21.15.114:0/2966769848 >> 172.21.15.114:0/2966769848 conn(0x5591a25bbb70 msgr2=0x5591a26d96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.120+0000 7f0797820b80 1 -- 172.21.15.114:0/2966769848 shutdown_connections 2024-05-27T07:51:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.120+0000 7f0797820b80 1 -- 172.21.15.114:0/2966769848 wait complete. 2024-05-27T07:51:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.120+0000 7f0797820b80 1 -- start start 2024-05-27T07:51:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.120+0000 7f0797820b80 1 Processor -- start 2024-05-27T07:51:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.120+0000 7f0797820b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a2560ec0 0x5591a26e8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.120+0000 7f0797820b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26df000 0x5591a26e89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.120+0000 7f0797820b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26e8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0797820b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5591a2504020 con 0x5591a26e4cd0 2024-05-27T07:51:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0797820b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5591a259f840 con 0x5591a26df000 2024-05-27T07:51:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0797820b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5591a26018f0 con 0x5591a2560ec0 2024-05-27T07:51:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0782d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26e8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f078259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a2560ec0 0x5591a26e8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0782d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26e8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58086/0 (socket says 172.21.15.114:58086) 2024-05-27T07:51:49.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0782d9d700 1 -- 172.21.15.114:0/2883894066 learned_addr learned my addr 172.21.15.114:0/2883894066 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:49.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0797820b80 10 client.0 _ll_register_callbacks cb 0x5591a25989a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:49.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f078259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a2560ec0 0x5591a26e8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47202/0 (socket says 172.21.15.114:47202) 2024-05-27T07:51:49.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0781d9b700 1 --2- 172.21.15.114:0/2883894066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26df000 0x5591a26e89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0782d9d700 1 -- 172.21.15.114:0/2883894066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a2560ec0 msgr2=0x5591a26e8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0782d9d700 1 --2- 172.21.15.114:0/2883894066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a2560ec0 0x5591a26e8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0782d9d700 1 -- 172.21.15.114:0/2883894066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26df000 msgr2=0x5591a26e89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0782d9d700 1 --2- 172.21.15.114:0/2883894066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26df000 0x5591a26e89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0782d9d700 1 -- 172.21.15.114:0/2883894066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591a259c710 con 0x5591a26e4cd0 2024-05-27T07:51:49.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0782d9d700 1 --2- 172.21.15.114:0/2883894066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26e8ee0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f077c010a20 tx=0x7f077c010d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483de8858fe2d7ca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:49.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0769ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0769ffb700 1 -- 172.21.15.114:0/2883894066 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c00faa0 con 0x5591a26e4cd0 2024-05-27T07:51:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0797820b80 1 -- 172.21.15.114:0/2883894066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07740ac4b0 con 0x5591a26e4cd0 2024-05-27T07:51:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.121+0000 7f0769ffb700 1 -- 172.21.15.114:0/2883894066 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f077c00fc40 con 0x5591a26e4cd0 2024-05-27T07:51:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.122+0000 7f0769ffb700 1 -- 172.21.15.114:0/2883894066 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f077c024050 con 0x5591a26e4cd0 2024-05-27T07:51:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.122+0000 7f0797820b80 10 client.18246 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.122+0000 7f0797820b80 1 -- 172.21.15.114:0/2883894066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5591a26fa300 con 0x5591a26e4cd0 2024-05-27T07:51:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.122+0000 7f0769ffb700 1 -- 172.21.15.114:0/2883894066 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07740ac4b0 con 0x5591a26e4cd0 2024-05-27T07:51:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.122+0000 7f0769ffb700 1 -- 172.21.15.114:0/2883894066 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f077c005480 con 0x5591a26e4cd0 2024-05-27T07:51:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.122+0000 7f0797820b80 10 client.18246 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.122+0000 7f0797820b80 10 client.18246 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.122+0000 7f0797820b80 20 client.18246 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.122+0000 7f0797820b80 1 --2- 172.21.15.114:0/2883894066 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591a26fcba0 0x5591a26feff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.122+0000 7f0797820b80 4 client.18246 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.122+0000 7f0797820b80 1 -- 172.21.15.114:0/2883894066 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591a26aaa70 con 0x5591a26fcba0 2024-05-27T07:51:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.122+0000 7f078259c700 1 --2- 172.21.15.114:0/2883894066 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591a26fcba0 0x5591a26feff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.123+0000 7f0769ffb700 10 client.18246 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.123+0000 7f078259c700 1 --2- 172.21.15.114:0/2883894066 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591a26fcba0 0x5591a26feff0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5a99136e856387e server_cookie=1de4b564acf0c6d in_seq=0 out_seq=0 2024-05-27T07:51:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.123+0000 7f0769ffb700 1 -- 172.21.15.114:0/2883894066 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5591a26aaa70 con 0x5591a26fcba0 2024-05-27T07:51:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.123+0000 7f0769ffb700 10 client.18246 handle_command_reply: tid=0 2024-05-27T07:51:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.123+0000 7f0797820b80 1 client.18246 shutdown 2024-05-27T07:51:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.124+0000 7f0797820b80 1 -- 172.21.15.114:0/2883894066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 msgr2=0x5591a26e8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.124+0000 7f0797820b80 1 --2- 172.21.15.114:0/2883894066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26e8ee0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f077c010a20 tx=0x7f077c010d30 comp rx=0 tx=0).stop 2024-05-27T07:51:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.124+0000 7f0797820b80 1 -- 172.21.15.114:0/2883894066 shutdown_connections 2024-05-27T07:51:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.124+0000 7f0797820b80 1 --2- 172.21.15.114:0/2883894066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26e8ee0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.124+0000 7f0797820b80 1 --2- 172.21.15.114:0/2883894066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a2560ec0 0x5591a26e8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.124+0000 7f0797820b80 1 --2- 172.21.15.114:0/2883894066 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591a26fcba0 0x5591a26feff0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.124+0000 7f0797820b80 1 --2- 172.21.15.114:0/2883894066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26df000 0x5591a26e89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.124+0000 7f0797820b80 1 -- 172.21.15.114:0/2883894066 >> 172.21.15.114:0/2883894066 conn(0x5591a25bbb70 msgr2=0x5591a26ddab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:49.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.124+0000 7f0769ffb700 0 client.18246 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:49.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.124+0000 7f0797820b80 1 -- 172.21.15.114:0/2883894066 shutdown_connections 2024-05-27T07:51:49.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.124+0000 7f0797820b80 1 -- 172.21.15.114:0/2883894066 wait complete. 2024-05-27T07:51:49.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.124+0000 7f0797820b80 20 client.18246 trim_cache size 0 max 16384 2024-05-27T07:51:49.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.145+0000 7f0797820b80 1 Processor -- start 2024-05-27T07:51:49.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.145+0000 7f0797820b80 1 -- start start 2024-05-27T07:51:49.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.145+0000 7f0797820b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a2560ec0 0x5591a26e2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.145+0000 7f0797820b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a26df000 0x5591a26e8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.145+0000 7f0797820b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26e4cd0 0x5591a26f4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.145+0000 7f0797820b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f07841afc10 con 0x5591a2560ec0 2024-05-27T07:51:49.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.145+0000 7f0797820b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f07841ae380 con 0x5591a26e4cd0 2024-05-27T07:51:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.145+0000 7f0797820b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f07841ac1d0 con 0x5591a26df000 2024-05-27T07:51:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.145+0000 7f078259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a2560ec0 0x5591a26e2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.146+0000 7f078259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a2560ec0 0x5591a26e2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58094/0 (socket says 172.21.15.114:58094) 2024-05-27T07:51:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.146+0000 7f078259c700 1 -- 172.21.15.114:0/3494035705 learned_addr learned my addr 172.21.15.114:0/3494035705 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.146+0000 7f0781d9b700 1 --2- 172.21.15.114:0/3494035705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a26df000 0x5591a26e8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.146+0000 7f0782d9d700 1 --2- 172.21.15.114:0/3494035705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26e4cd0 0x5591a26f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.146+0000 7f078259c700 1 -- 172.21.15.114:0/3494035705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a26df000 msgr2=0x5591a26e8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.146+0000 7f078259c700 1 --2- 172.21.15.114:0/3494035705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a26df000 0x5591a26e8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.146+0000 7f078259c700 1 -- 172.21.15.114:0/3494035705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26e4cd0 msgr2=0x5591a26f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.146+0000 7f078259c700 1 --2- 172.21.15.114:0/3494035705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26e4cd0 0x5591a26f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.146+0000 7f078259c700 1 -- 172.21.15.114:0/3494035705 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591a26d9910 con 0x5591a2560ec0 2024-05-27T07:51:49.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.146+0000 7f078259c700 1 --2- 172.21.15.114:0/3494035705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a2560ec0 0x5591a26e2ac0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f07740e5ad0 tx=0x7f07740060a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c24ee5f08c8f7b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:49.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.146+0000 7f076bfff700 1 -- 172.21.15.114:0/3494035705 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07740b50b0 con 0x5591a2560ec0 2024-05-27T07:51:49.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.146+0000 7f076bfff700 1 -- 172.21.15.114:0/3494035705 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07740ac4b0 con 0x5591a2560ec0 2024-05-27T07:51:49.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.146+0000 7f076bfff700 1 -- 172.21.15.114:0/3494035705 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07740b0d50 con 0x5591a2560ec0 2024-05-27T07:51:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.147+0000 7f0797820b80 1 -- 172.21.15.114:0/3494035705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a2560ec0 msgr2=0x5591a26e2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.147+0000 7f0797820b80 1 --2- 172.21.15.114:0/3494035705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a2560ec0 0x5591a26e2ac0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f07740e5ad0 tx=0x7f07740060a0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.147+0000 7f0797820b80 1 -- 172.21.15.114:0/3494035705 shutdown_connections 2024-05-27T07:51:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.147+0000 7f0797820b80 1 --2- 172.21.15.114:0/3494035705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a26e4cd0 0x5591a26f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.147+0000 7f0797820b80 1 --2- 172.21.15.114:0/3494035705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a2560ec0 0x5591a26e2ac0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.147+0000 7f0797820b80 1 --2- 172.21.15.114:0/3494035705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a26df000 0x5591a26e8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.147+0000 7f0797820b80 1 -- 172.21.15.114:0/3494035705 >> 172.21.15.114:0/3494035705 conn(0x5591a25bbb70 msgr2=0x5591a26e98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:49.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.147+0000 7f0797820b80 1 -- 172.21.15.114:0/3494035705 shutdown_connections 2024-05-27T07:51:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.147+0000 7f0797820b80 1 -- 172.21.15.114:0/3494035705 wait complete. 2024-05-27T07:51:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.147+0000 7f0797820b80 1 -- start start 2024-05-27T07:51:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0797820b80 1 Processor -- start 2024-05-27T07:51:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0797820b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a2560ec0 0x5591a26f0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0797820b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a26df000 0x5591a26f0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0797820b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26ed620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0797820b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f07841ac840 con 0x5591a26e4cd0 2024-05-27T07:51:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0797820b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f07841adcb0 con 0x5591a2560ec0 2024-05-27T07:51:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0797820b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f07841acf10 con 0x5591a26df000 2024-05-27T07:51:49.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0797820b80 10 client.0 _ll_register_callbacks cb 0x5591a25989a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0782d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26ed620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0782d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26ed620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58104/0 (socket says 172.21.15.114:58104) 2024-05-27T07:51:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0782d9d700 1 -- 172.21.15.114:0/1320817937 learned_addr learned my addr 172.21.15.114:0/1320817937 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f078259c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a2560ec0 0x5591a26f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0781d9b700 1 --2- 172.21.15.114:0/1320817937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a26df000 0x5591a26f0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0782d9d700 1 -- 172.21.15.114:0/1320817937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a26df000 msgr2=0x5591a26f0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0782d9d700 1 --2- 172.21.15.114:0/1320817937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a26df000 0x5591a26f0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0782d9d700 1 -- 172.21.15.114:0/1320817937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a2560ec0 msgr2=0x5591a26f0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0782d9d700 1 --2- 172.21.15.114:0/1320817937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a2560ec0 0x5591a26f0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0782d9d700 1 -- 172.21.15.114:0/1320817937 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591a26edd60 con 0x5591a26e4cd0 2024-05-27T07:51:49.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f078259c700 1 --2- 172.21.15.114:0/1320817937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a2560ec0 0x5591a26f0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:49.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0782d9d700 1 --2- 172.21.15.114:0/1320817937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26ed620 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f077c010a20 tx=0x7f077c00fd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749614627fd20bce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:49.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.148+0000 7f0769ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:49.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0769ffb700 1 -- 172.21.15.114:0/1320817937 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c00a340 con 0x5591a26e4cd0 2024-05-27T07:51:49.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0769ffb700 1 -- 172.21.15.114:0/1320817937 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f077c0052f0 con 0x5591a26e4cd0 2024-05-27T07:51:49.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0797820b80 1 -- 172.21.15.114:0/1320817937 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07740ac4b0 con 0x5591a26e4cd0 2024-05-27T07:51:49.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0769ffb700 1 -- 172.21.15.114:0/1320817937 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f077c038050 con 0x5591a26e4cd0 2024-05-27T07:51:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0797820b80 10 client.18252 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0797820b80 1 -- 172.21.15.114:0/1320817937 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5591a2768010 con 0x5591a26e4cd0 2024-05-27T07:51:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0769ffb700 1 -- 172.21.15.114:0/1320817937 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07740ac4b0 con 0x5591a26e4cd0 2024-05-27T07:51:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0769ffb700 1 -- 172.21.15.114:0/1320817937 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f077c032b90 con 0x5591a26e4cd0 2024-05-27T07:51:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0797820b80 10 client.18252 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0797820b80 10 client.18252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0797820b80 20 client.18252 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0797820b80 1 --2- 172.21.15.114:0/1320817937 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591a2768660 0x5591a276aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0797820b80 4 client.18252 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:51:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f0797820b80 1 -- 172.21.15.114:0/1320817937 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5591a260fc60 con 0x5591a2768660 2024-05-27T07:51:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.149+0000 7f078259c700 1 --2- 172.21.15.114:0/1320817937 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591a2768660 0x5591a276aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.150+0000 7f0769ffb700 10 client.18252 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.150+0000 7f078259c700 1 --2- 172.21.15.114:0/1320817937 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591a2768660 0x5591a276aa80 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48df842df14110a5 server_cookie=3c4ae1273edd0d10 in_seq=0 out_seq=0 2024-05-27T07:51:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0769ffb700 1 -- 172.21.15.114:0/1320817937 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5591a260fc60 con 0x5591a2768660 2024-05-27T07:51:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0769ffb700 10 client.18252 handle_command_reply: tid=0 2024-05-27T07:51:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0797820b80 1 client.18252 shutdown 2024-05-27T07:51:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0797820b80 1 -- 172.21.15.114:0/1320817937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 msgr2=0x5591a26ed620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0797820b80 1 --2- 172.21.15.114:0/1320817937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26ed620 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f077c010a20 tx=0x7f077c00fd50 comp rx=0 tx=0).stop 2024-05-27T07:51:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0797820b80 1 -- 172.21.15.114:0/1320817937 shutdown_connections 2024-05-27T07:51:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0797820b80 1 --2- 172.21.15.114:0/1320817937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591a26e4cd0 0x5591a26ed620 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0797820b80 1 --2- 172.21.15.114:0/1320817937 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591a2768660 0x5591a276aa80 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0797820b80 1 --2- 172.21.15.114:0/1320817937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591a2560ec0 0x5591a26f0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0797820b80 1 --2- 172.21.15.114:0/1320817937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591a26df000 0x5591a26f0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0797820b80 1 -- 172.21.15.114:0/1320817937 >> 172.21.15.114:0/1320817937 conn(0x5591a25bbb70 msgr2=0x5591a26e2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:49.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0769ffb700 0 client.18252 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:49.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0797820b80 1 -- 172.21.15.114:0/1320817937 shutdown_connections 2024-05-27T07:51:49.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0797820b80 1 -- 172.21.15.114:0/1320817937 wait complete. 2024-05-27T07:51:49.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.152+0000 7f0797820b80 20 client.18252 trim_cache size 0 max 16384 2024-05-27T07:51:49.163 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:51:49.163 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:51:49.164 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "233721d7-24b3-4eb4-ba46-b50b3d2311fe", 2024-05-27T07:51:49.164 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:51:49.164 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:51:49.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.153+0000 7f0788e0b700 1 -- 172.21.15.114:0/1152706457 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f07700b29f0 msgr2=0x7f07700b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.153+0000 7f0788e0b700 1 --2- 172.21.15.114:0/1152706457 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f07700b29f0 0x7f07700b4f00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f076c005950 tx=0x7f076c0058e0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.153+0000 7f0788e0b700 1 -- 172.21.15.114:0/1152706457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07841034d0 msgr2=0x7f0784199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.153+0000 7f0788e0b700 1 --2- 172.21.15.114:0/1152706457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07841034d0 0x7f0784199d30 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f0774007a20 tx=0x7f0774007a50 comp rx=0 tx=0).stop 2024-05-27T07:51:49.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.153+0000 7f0788e0b700 1 -- 172.21.15.114:0/1152706457 shutdown_connections 2024-05-27T07:51:49.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.153+0000 7f0788e0b700 1 --2- 172.21.15.114:0/1152706457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0784108750 0x7f078419e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.153+0000 7f0788e0b700 1 --2- 172.21.15.114:0/1152706457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07841034d0 0x7f0784199d30 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.153+0000 7f0788e0b700 1 --2- 172.21.15.114:0/1152706457 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f07700b29f0 0x7f07700b4f00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.153+0000 7f0788e0b700 1 --2- 172.21.15.114:0/1152706457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0784107e00 0x7f078419a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.153+0000 7f0788e0b700 1 -- 172.21.15.114:0/1152706457 >> 172.21.15.114:0/1152706457 conn(0x7f07840fef20 msgr2=0x7f07841071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:49.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.153+0000 7f0788e0b700 1 -- 172.21.15.114:0/1152706457 shutdown_connections 2024-05-27T07:51:49.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:49.153+0000 7f0788e0b700 1 -- 172.21.15.114:0/1152706457 wait complete. 2024-05-27T07:51:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.272+0000 7f141ae9f640 1 -- 172.21.15.114:0/1304705189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1414103300 msgr2=0x7f141410f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.272+0000 7f141ae9f640 1 --2- 172.21.15.114:0/1304705189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1414103300 0x7f141410f6d0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f140c009ca0 tx=0x7f140c02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.272+0000 7f141ae9f640 1 -- 172.21.15.114:0/1304705189 shutdown_connections 2024-05-27T07:51:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.272+0000 7f141ae9f640 1 --2- 172.21.15.114:0/1304705189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1414103300 0x7f141410f6d0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.272+0000 7f141ae9f640 1 --2- 172.21.15.114:0/1304705189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1414109050 0x7f1414102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.272+0000 7f141ae9f640 1 --2- 172.21.15.114:0/1304705189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1414108bb0 0x7f1414102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.272+0000 7f141ae9f640 1 -- 172.21.15.114:0/1304705189 >> 172.21.15.114:0/1304705189 conn(0x7f14140781a0 msgr2=0x7f1414078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.272+0000 7f141ae9f640 1 -- 172.21.15.114:0/1304705189 shutdown_connections 2024-05-27T07:51:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.272+0000 7f141ae9f640 1 -- 172.21.15.114:0/1304705189 wait complete. 2024-05-27T07:51:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f141ae9f640 1 Processor -- start 2024-05-27T07:51:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f141ae9f640 1 -- start start 2024-05-27T07:51:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f141ae9f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1414103300 0x7f14141a26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f141ae9f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1414108bb0 0x7f14141a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f141ae9f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1414109050 0x7f141419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f141ae9f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1414112dc0 con 0x7f1414103300 2024-05-27T07:51:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f141ae9f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1414112c40 con 0x7f1414109050 2024-05-27T07:51:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f141ae9f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1414112f40 con 0x7f1414108bb0 2024-05-27T07:51:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f1418c3b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1414103300 0x7f14141a26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f1418c3b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1414103300 0x7f14141a26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58134/0 (socket says 172.21.15.114:58134) 2024-05-27T07:51:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f1418c3b640 1 -- 172.21.15.114:0/2416782850 learned_addr learned my addr 172.21.15.114:0/2416782850 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f140bfff640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1414108bb0 0x7f14141a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f141943c640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1414109050 0x7f141419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f1418c3b640 1 -- 172.21.15.114:0/2416782850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1414108bb0 msgr2=0x7f14141a2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f1418c3b640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1414108bb0 0x7f14141a2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f1418c3b640 1 -- 172.21.15.114:0/2416782850 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1414109050 msgr2=0x7f141419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f1418c3b640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1414109050 0x7f141419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.273+0000 7f1418c3b640 1 -- 172.21.15.114:0/2416782850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141419cf70 con 0x7f1414103300 2024-05-27T07:51:49.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.274+0000 7f1418c3b640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1414103300 0x7f14141a26c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f141000e9c0 tx=0x7f141000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:49.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.274+0000 7f1409ffb640 1 -- 172.21.15.114:0/2416782850 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141000cd80 con 0x7f1414103300 2024-05-27T07:51:49.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.274+0000 7f141ae9f640 1 -- 172.21.15.114:0/2416782850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141419d260 con 0x7f1414103300 2024-05-27T07:51:49.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.274+0000 7f1409ffb640 1 -- 172.21.15.114:0/2416782850 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14100054d0 con 0x7f1414103300 2024-05-27T07:51:49.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.274+0000 7f1409ffb640 1 -- 172.21.15.114:0/2416782850 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410004e60 con 0x7f1414103300 2024-05-27T07:51:49.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.274+0000 7f141ae9f640 1 -- 172.21.15.114:0/2416782850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14141a9540 con 0x7f1414103300 2024-05-27T07:51:49.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f1421e2b740 1 Processor -- start 2024-05-27T07:51:49.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f1421e2b740 1 -- start start 2024-05-27T07:51:49.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f1409ffb640 1 -- 172.21.15.114:0/2416782850 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f1410005020 con 0x7f1414103300 2024-05-27T07:51:49.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f1421e2b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 0x55733de883d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f1421e2b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dabe2f0 0x55733de88910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f1421e2b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 0x55733de88e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f1421e2b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55733ddfac30 con 0x55733dac64d0 2024-05-27T07:51:49.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f1421e2b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55733ddea7d0 con 0x55733dabe2f0 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f1421e2b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55733ddee810 con 0x55733dae71b0 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f140bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 0x55733de88e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f140bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 0x55733de88e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58146/0 (socket says 172.21.15.114:58146) 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f140bfff640 1 -- 172.21.15.114:0/3643380098 learned_addr learned my addr 172.21.15.114:0/3643380098 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f141943c640 1 --2- 172.21.15.114:0/3643380098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 0x55733de883d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f1418c3b640 1 --2- 172.21.15.114:0/3643380098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dabe2f0 0x55733de88910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.277+0000 7f140bfff640 1 -- 172.21.15.114:0/3643380098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 msgr2=0x55733de883d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.278+0000 7f140bfff640 1 --2- 172.21.15.114:0/3643380098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 0x55733de883d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.278+0000 7f140bfff640 1 -- 172.21.15.114:0/3643380098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dabe2f0 msgr2=0x55733de88910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.278+0000 7f140bfff640 1 --2- 172.21.15.114:0/3643380098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dabe2f0 0x55733de88910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.278+0000 7f140bfff640 1 -- 172.21.15.114:0/3643380098 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55733daff210 con 0x55733dac64d0 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.278+0000 7f140bfff640 1 --2- 172.21.15.114:0/3643380098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 0x55733de88e50 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f1404009c00 tx=0x7f1404011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0477de22a185f0c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.278+0000 7f13faffd640 1 -- 172.21.15.114:0/3643380098 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140401e070 con 0x55733dac64d0 2024-05-27T07:51:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.278+0000 7f1409ffb640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f13fc0b2950 0x7f13fc0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.278+0000 7f13faffd640 1 -- 172.21.15.114:0/3643380098 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1404009db0 con 0x55733dac64d0 2024-05-27T07:51:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.278+0000 7f13faffd640 1 -- 172.21.15.114:0/3643380098 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1404019450 con 0x55733dac64d0 2024-05-27T07:51:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.278+0000 7f141943c640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f13fc0b2950 0x7f13fc0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.278+0000 7f1409ffb640 1 -- 172.21.15.114:0/2416782850 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f14100e2db0 con 0x7f1414103300 2024-05-27T07:51:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.279+0000 7f1421e2b740 1 -- 172.21.15.114:0/3643380098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 msgr2=0x55733de88e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.279+0000 7f141943c640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f13fc0b2950 0x7f13fc0b4e60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f140c0094e0 tx=0x7f140c02f310 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.279+0000 7f1421e2b740 1 --2- 172.21.15.114:0/3643380098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 0x55733de88e50 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f1404009c00 tx=0x7f1404011810 comp rx=0 tx=0).stop 2024-05-27T07:51:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.279+0000 7f1421e2b740 1 -- 172.21.15.114:0/3643380098 shutdown_connections 2024-05-27T07:51:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.279+0000 7f1421e2b740 1 --2- 172.21.15.114:0/3643380098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 0x55733de88e50 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.279+0000 7f1421e2b740 1 --2- 172.21.15.114:0/3643380098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dabe2f0 0x55733de88910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.279+0000 7f1421e2b740 1 --2- 172.21.15.114:0/3643380098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 0x55733de883d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.279+0000 7f1421e2b740 1 -- 172.21.15.114:0/3643380098 >> 172.21.15.114:0/3643380098 conn(0x55733dadb300 msgr2=0x55733de84b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.279+0000 7f1421e2b740 1 -- 172.21.15.114:0/3643380098 shutdown_connections 2024-05-27T07:51:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.279+0000 7f1421e2b740 1 -- 172.21.15.114:0/3643380098 wait complete. 2024-05-27T07:51:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.279+0000 7f1421e2b740 1 -- start start 2024-05-27T07:51:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.279+0000 7f1421e2b740 1 Processor -- start 2024-05-27T07:51:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1421e2b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 0x55733de92a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1421e2b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dac64d0 0x55733de92fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1421e2b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 0x55733de98870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1421e2b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55733dad0790 con 0x55733dabe2f0 2024-05-27T07:51:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1421e2b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55733dda6890 con 0x55733dac64d0 2024-05-27T07:51:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1421e2b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55733de95500 con 0x55733dae71b0 2024-05-27T07:51:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1418c3b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 0x55733de92a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1421e2b740 10 client.0 _ll_register_callbacks cb 0x55733dafea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1418c3b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 0x55733de92a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58152/0 (socket says 172.21.15.114:58152) 2024-05-27T07:51:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1418c3b640 1 -- 172.21.15.114:0/1730529193 learned_addr learned my addr 172.21.15.114:0/1730529193 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f141943c640 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 0x55733de98870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f140bfff640 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dac64d0 0x55733de92fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1418c3b640 1 -- 172.21.15.114:0/1730529193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 msgr2=0x55733de98870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1418c3b640 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 0x55733de98870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1418c3b640 1 -- 172.21.15.114:0/1730529193 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dac64d0 msgr2=0x55733de92fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1418c3b640 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dac64d0 0x55733de92fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f1418c3b640 1 -- 172.21.15.114:0/1730529193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55733dbcb620 con 0x55733dabe2f0 2024-05-27T07:51:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.280+0000 7f140bfff640 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dac64d0 0x55733de92fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f1418c3b640 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 0x55733de92a90 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f14100a9c70 tx=0x7f14100048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc786a5a393e2e3d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f13f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f13f3fff640 1 -- 172.21.15.114:0/1730529193 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14100a9450 con 0x55733dabe2f0 2024-05-27T07:51:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f13f3fff640 1 -- 172.21.15.114:0/1730529193 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f14100a62d0 con 0x55733dabe2f0 2024-05-27T07:51:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f1421e2b740 1 -- 172.21.15.114:0/1730529193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1404009db0 con 0x55733dabe2f0 2024-05-27T07:51:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f13f3fff640 1 -- 172.21.15.114:0/1730529193 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f1410005310 con 0x55733dabe2f0 2024-05-27T07:51:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f1421e2b740 10 client.18264 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f1421e2b740 1 -- 172.21.15.114:0/1730529193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55733dea1860 con 0x55733dabe2f0 2024-05-27T07:51:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f13f3fff640 1 -- 172.21.15.114:0/1730529193 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1404009db0 con 0x55733dabe2f0 2024-05-27T07:51:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f13f3fff640 1 -- 172.21.15.114:0/1730529193 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f14100c5d50 con 0x55733dabe2f0 2024-05-27T07:51:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f1421e2b740 10 client.18264 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f1421e2b740 10 client.18264 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f1421e2b740 20 client.18264 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f1421e2b740 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55733daa5d80 0x55733dea5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f1421e2b740 4 client.18264 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f1421e2b740 1 -- 172.21.15.114:0/1730529193 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55733ddfcb80 con 0x55733daa5d80 2024-05-27T07:51:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.281+0000 7f140bfff640 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55733daa5d80 0x55733dea5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.282+0000 7f13f3fff640 10 client.18264 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.282+0000 7f140bfff640 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55733daa5d80 0x55733dea5080 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5af5ba3689531b27 server_cookie=3c423eaa51381de5 in_seq=0 out_seq=0 2024-05-27T07:51:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f13f3fff640 1 -- 172.21.15.114:0/1730529193 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55733ddfcb80 con 0x55733daa5d80 2024-05-27T07:51:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f13f3fff640 10 client.18264 handle_command_reply: tid=0 2024-05-27T07:51:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f1421e2b740 1 client.18264 shutdown 2024-05-27T07:51:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f1421e2b740 1 -- 172.21.15.114:0/1730529193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 msgr2=0x55733de92a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f1421e2b740 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 0x55733de92a90 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f14100a9c70 tx=0x7f14100048f0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f1421e2b740 1 -- 172.21.15.114:0/1730529193 shutdown_connections 2024-05-27T07:51:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f1421e2b740 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55733daa5d80 0x55733dea5080 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f1421e2b740 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 0x55733de98870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f1421e2b740 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dac64d0 0x55733de92fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f13f3fff640 0 client.18264 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f1421e2b740 1 --2- 172.21.15.114:0/1730529193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 0x55733de92a90 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f1421e2b740 1 -- 172.21.15.114:0/1730529193 >> 172.21.15.114:0/1730529193 conn(0x55733dadb300 msgr2=0x55733de8a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f1421e2b740 1 -- 172.21.15.114:0/1730529193 shutdown_connections 2024-05-27T07:51:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f1421e2b740 1 -- 172.21.15.114:0/1730529193 wait complete. 2024-05-27T07:51:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.283+0000 7f1421e2b740 20 client.18264 trim_cache size 0 max 16384 2024-05-27T07:51:49.291 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1177145 9.79 MB/sec execute 481 sec latency 24.617 ms 2024-05-27T07:51:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.299+0000 7f1421e2b740 1 Processor -- start 2024-05-27T07:51:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.299+0000 7f1421e2b740 1 -- start start 2024-05-27T07:51:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.299+0000 7f1421e2b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dabe2f0 0x55733de98770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.299+0000 7f1421e2b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 0x55733de9f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.299+0000 7f1421e2b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dae71b0 0x55733de92f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.299+0000 7f1421e2b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f14141b4dc0 con 0x55733dac64d0 2024-05-27T07:51:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.299+0000 7f1421e2b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f14141b3550 con 0x55733dae71b0 2024-05-27T07:51:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.299+0000 7f1421e2b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f14141b13a0 con 0x55733dabe2f0 2024-05-27T07:51:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.299+0000 7f140bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 0x55733de9f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.299+0000 7f140bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 0x55733de9f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58162/0 (socket says 172.21.15.114:58162) 2024-05-27T07:51:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.299+0000 7f140bfff640 1 -- 172.21.15.114:0/4007636851 learned_addr learned my addr 172.21.15.114:0/4007636851 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.299+0000 7f1418c3b640 1 --2- 172.21.15.114:0/4007636851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dabe2f0 0x55733de98770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.299+0000 7f141943c640 1 --2- 172.21.15.114:0/4007636851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dae71b0 0x55733de92f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.300+0000 7f140bfff640 1 -- 172.21.15.114:0/4007636851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dabe2f0 msgr2=0x55733de98770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.300+0000 7f140bfff640 1 --2- 172.21.15.114:0/4007636851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dabe2f0 0x55733de98770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.300+0000 7f140bfff640 1 -- 172.21.15.114:0/4007636851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dae71b0 msgr2=0x55733de92f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.300+0000 7f140bfff640 1 --2- 172.21.15.114:0/4007636851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dae71b0 0x55733de92f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.300+0000 7f140bfff640 1 -- 172.21.15.114:0/4007636851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55733de93790 con 0x55733dac64d0 2024-05-27T07:51:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.300+0000 7f140bfff640 1 --2- 172.21.15.114:0/4007636851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 0x55733de9f560 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f1404011d10 tx=0x7f1404009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efcb38d64ed7f99c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.300+0000 7f13fa7fc640 1 -- 172.21.15.114:0/4007636851 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140401e040 con 0x55733dac64d0 2024-05-27T07:51:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.300+0000 7f13fa7fc640 1 -- 172.21.15.114:0/4007636851 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f140402c070 con 0x55733dac64d0 2024-05-27T07:51:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.300+0000 7f13fa7fc640 1 -- 172.21.15.114:0/4007636851 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1404004b10 con 0x55733dac64d0 2024-05-27T07:51:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.301+0000 7f1421e2b740 1 -- 172.21.15.114:0/4007636851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 msgr2=0x55733de9f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.301+0000 7f1421e2b740 1 --2- 172.21.15.114:0/4007636851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 0x55733de9f560 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f1404011d10 tx=0x7f1404009660 comp rx=0 tx=0).stop 2024-05-27T07:51:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.301+0000 7f1421e2b740 1 -- 172.21.15.114:0/4007636851 shutdown_connections 2024-05-27T07:51:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.301+0000 7f1421e2b740 1 --2- 172.21.15.114:0/4007636851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dae71b0 0x55733de92f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.301+0000 7f1421e2b740 1 --2- 172.21.15.114:0/4007636851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dac64d0 0x55733de9f560 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.301+0000 7f1421e2b740 1 --2- 172.21.15.114:0/4007636851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dabe2f0 0x55733de98770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.301+0000 7f1421e2b740 1 -- 172.21.15.114:0/4007636851 >> 172.21.15.114:0/4007636851 conn(0x55733dadb300 msgr2=0x55733de92840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.301+0000 7f1421e2b740 1 -- 172.21.15.114:0/4007636851 shutdown_connections 2024-05-27T07:51:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.301+0000 7f1421e2b740 1 -- 172.21.15.114:0/4007636851 wait complete. 2024-05-27T07:51:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.301+0000 7f1421e2b740 1 -- start start 2024-05-27T07:51:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.301+0000 7f1421e2b740 1 Processor -- start 2024-05-27T07:51:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1421e2b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 0x55733de9a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1421e2b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dac64d0 0x55733de9a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1421e2b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 0x55733df0b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1421e2b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f14141b1a10 con 0x55733dabe2f0 2024-05-27T07:51:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1421e2b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f14141b2e80 con 0x55733dac64d0 2024-05-27T07:51:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1421e2b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f14141b20e0 con 0x55733dae71b0 2024-05-27T07:51:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1421e2b740 10 client.0 _ll_register_callbacks cb 0x7f141000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1418c3b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 0x55733de9a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1418c3b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 0x55733de9a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58178/0 (socket says 172.21.15.114:58178) 2024-05-27T07:51:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1418c3b640 1 -- 172.21.15.114:0/2435674776 learned_addr learned my addr 172.21.15.114:0/2435674776 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f141943c640 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 0x55733df0b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f140bfff640 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dac64d0 0x55733de9a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1418c3b640 1 -- 172.21.15.114:0/2435674776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 msgr2=0x55733df0b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1418c3b640 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 0x55733df0b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1418c3b640 1 -- 172.21.15.114:0/2435674776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dac64d0 msgr2=0x55733de9a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1418c3b640 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dac64d0 0x55733de9a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1418c3b640 1 -- 172.21.15.114:0/2435674776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55733df0bc80 con 0x55733dabe2f0 2024-05-27T07:51:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f140bfff640 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dac64d0 0x55733de9a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f1418c3b640 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 0x55733de9a050 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f1410000c00 tx=0x7f14100a9150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83bc18e75553c92 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.302+0000 7f13f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f13f3fff640 1 -- 172.21.15.114:0/2435674776 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14100a9450 con 0x55733dabe2f0 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f13f3fff640 1 -- 172.21.15.114:0/2435674776 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7f14100a62d0 con 0x55733dabe2f0 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f1421e2b740 1 -- 172.21.15.114:0/2435674776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f140402c070 con 0x55733dabe2f0 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f13f3fff640 1 -- 172.21.15.114:0/2435674776 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f14100a9ab0 con 0x55733dabe2f0 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f1421e2b740 10 client.18270 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f1421e2b740 1 -- 172.21.15.114:0/2435674776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55733df0cd80 con 0x55733dabe2f0 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f13f3fff640 1 -- 172.21.15.114:0/2435674776 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f140402c070 con 0x55733dabe2f0 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f13f3fff640 1 -- 172.21.15.114:0/2435674776 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f14100a8bc0 con 0x55733dabe2f0 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f1421e2b740 10 client.18270 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f1421e2b740 10 client.18270 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f1421e2b740 20 client.18270 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f1421e2b740 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55733da7b1c0 0x55733df0f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f1421e2b740 4 client.18270 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f1421e2b740 1 -- 172.21.15.114:0/2435674776 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55733ddf6360 con 0x55733da7b1c0 2024-05-27T07:51:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.303+0000 7f140bfff640 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55733da7b1c0 0x55733df0f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.304+0000 7f13f3fff640 10 client.18270 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:49.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.304+0000 7f140bfff640 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55733da7b1c0 0x55733df0f350 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7eb85760f63b623 server_cookie=a063007005da2215 in_seq=0 out_seq=0 2024-05-27T07:51:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.745+0000 7f13f3fff640 1 -- 172.21.15.114:0/2435674776 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55733ddf6360 con 0x55733da7b1c0 2024-05-27T07:51:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.745+0000 7f13f3fff640 10 client.18270 handle_command_reply: tid=0 2024-05-27T07:51:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.745+0000 7f1421e2b740 1 client.18270 shutdown 2024-05-27T07:51:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.745+0000 7f1421e2b740 1 -- 172.21.15.114:0/2435674776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 msgr2=0x55733de9a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.745+0000 7f1421e2b740 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 0x55733de9a050 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f1410000c00 tx=0x7f14100a9150 comp rx=0 tx=0).stop 2024-05-27T07:51:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.745+0000 7f1421e2b740 1 -- 172.21.15.114:0/2435674776 shutdown_connections 2024-05-27T07:51:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.745+0000 7f1421e2b740 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55733da7b1c0 0x55733df0f350 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.745+0000 7f13f3fff640 0 client.18270 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.745+0000 7f1421e2b740 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55733dae71b0 0x55733df0b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.745+0000 7f1421e2b740 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55733dac64d0 0x55733de9a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.745+0000 7f1421e2b740 1 --2- 172.21.15.114:0/2435674776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733dabe2f0 0x55733de9a050 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.745+0000 7f1421e2b740 1 -- 172.21.15.114:0/2435674776 >> 172.21.15.114:0/2435674776 conn(0x55733dadb300 msgr2=0x55733de88350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.746+0000 7f1421e2b740 1 -- 172.21.15.114:0/2435674776 shutdown_connections 2024-05-27T07:51:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.746+0000 7f1421e2b740 1 -- 172.21.15.114:0/2435674776 wait complete. 2024-05-27T07:51:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.746+0000 7f1421e2b740 20 client.18270 trim_cache size 0 max 16384 2024-05-27T07:51:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:51:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:51:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:51:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.747+0000 7f141ae9f640 1 -- 172.21.15.114:0/2416782850 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f13fc0b2950 msgr2=0x7f13fc0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.747+0000 7f141ae9f640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f13fc0b2950 0x7f13fc0b4e60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f140c0094e0 tx=0x7f140c02f310 comp rx=0 tx=0).stop 2024-05-27T07:51:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.747+0000 7f141ae9f640 1 -- 172.21.15.114:0/2416782850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1414103300 msgr2=0x7f14141a26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.747+0000 7f141ae9f640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1414103300 0x7f14141a26c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f141000e9c0 tx=0x7f141000ee80 comp rx=0 tx=0).stop 2024-05-27T07:51:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.747+0000 7f141ae9f640 1 -- 172.21.15.114:0/2416782850 shutdown_connections 2024-05-27T07:51:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.747+0000 7f141ae9f640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f13fc0b2950 0x7f13fc0b4e60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.747+0000 7f141ae9f640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1414109050 0x7f141419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.747+0000 7f141ae9f640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1414108bb0 0x7f14141a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.747+0000 7f141ae9f640 1 --2- 172.21.15.114:0/2416782850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1414103300 0x7f14141a26c0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.747+0000 7f141ae9f640 1 -- 172.21.15.114:0/2416782850 >> 172.21.15.114:0/2416782850 conn(0x7f14140781a0 msgr2=0x7f1414101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.747+0000 7f141ae9f640 1 -- 172.21.15.114:0/2416782850 shutdown_connections 2024-05-27T07:51:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:51:49.747+0000 7f141ae9f640 1 -- 172.21.15.114:0/2416782850 wait complete. 2024-05-27T07:51:49.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:49 smithi114 ceph-mon[32038]: pgmap v401: 129 pgs: 129 active+clean; 75 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 241 op/s 2024-05-27T07:51:49.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:49 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:51:49.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:49 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:49.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:49 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:51:49.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:49 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:49.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:49 smithi163 ceph-mon[36451]: pgmap v401: 129 pgs: 129 active+clean; 75 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 241 op/s 2024-05-27T07:51:49.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:49 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:51:49.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:49 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:49.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:49 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:51:49.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:49 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:49.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:49 smithi186 ceph-mon[36647]: pgmap v401: 129 pgs: 129 active+clean; 75 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 241 op/s 2024-05-27T07:51:49.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:49 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:51:49.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:49 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:49.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:49 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:51:49.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:49 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:51:50.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:51:50.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1178098 9.78 MB/sec execute 482 sec latency 287.042 ms 2024-05-27T07:51:50.501 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:51.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1179820 9.78 MB/sec execute 483 sec latency 21.687 ms 2024-05-27T07:51:51.853 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:51 smithi114 ceph-mon[32038]: pgmap v402: 129 pgs: 129 active+clean; 75 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 223 op/s 2024-05-27T07:51:51.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:51 smithi163 ceph-mon[36451]: pgmap v402: 129 pgs: 129 active+clean; 75 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 223 op/s 2024-05-27T07:51:51.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:51 smithi186 ceph-mon[36647]: pgmap v402: 129 pgs: 129 active+clean; 75 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 223 op/s 2024-05-27T07:51:52.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.194+0000 7fdaafaff700 1 -- 172.21.15.114:0/2871424788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa8105510 msgr2=0x7fdaa8105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:52.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.194+0000 7fdaafaff700 1 --2- 172.21.15.114:0/2871424788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa8105510 0x7fdaa8105920 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009990 tx=0x7fdaa4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.194+0000 7fdaafaff700 1 -- 172.21.15.114:0/2871424788 shutdown_connections 2024-05-27T07:51:52.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.194+0000 7fdaafaff700 1 --2- 172.21.15.114:0/2871424788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdaa810e8f0 0x7fdaa8110dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.194+0000 7fdaafaff700 1 --2- 172.21.15.114:0/2871424788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa8105510 0x7fdaa8105920 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.194+0000 7fdaafaff700 1 --2- 172.21.15.114:0/2871424788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaa8105e60 0x7fdaa810e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.194+0000 7fdaafaff700 1 -- 172.21.15.114:0/2871424788 >> 172.21.15.114:0/2871424788 conn(0x7fdaa80fef20 msgr2=0x7fdaa8101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:52.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.194+0000 7fdaafaff700 1 -- 172.21.15.114:0/2871424788 shutdown_connections 2024-05-27T07:51:52.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.194+0000 7fdaafaff700 1 -- 172.21.15.114:0/2871424788 wait complete. 2024-05-27T07:51:52.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.195+0000 7fdaafaff700 1 Processor -- start 2024-05-27T07:51:52.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.195+0000 7fdaafaff700 1 -- start start 2024-05-27T07:51:52.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.195+0000 7fdaafaff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaa8105510 0x7fdaa819add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.195+0000 7fdaafaff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdaa8105e60 0x7fdaa819b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.195+0000 7fdaafaff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa810e8f0 0x7fdaa819f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.195+0000 7fdaafaff700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa8114440 con 0x7fdaa810e8f0 2024-05-27T07:51:52.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.195+0000 7fdaafaff700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81142c0 con 0x7fdaa8105e60 2024-05-27T07:51:52.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.195+0000 7fdaafaff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81145c0 con 0x7fdaa8105510 2024-05-27T07:51:52.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.195+0000 7fdaad89b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaa8105510 0x7fdaa819add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.195+0000 7fdaae09c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa810e8f0 0x7fdaa819f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.195+0000 7fdaad89b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaa8105510 0x7fdaa819add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47280/0 (socket says 172.21.15.114:47280) 2024-05-27T07:51:52.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.195+0000 7fdaad89b700 1 -- 172.21.15.114:0/3402332085 learned_addr learned my addr 172.21.15.114:0/3402332085 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:52.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.195+0000 7fdaad09a700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdaa8105e60 0x7fdaa819b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.196+0000 7fdaad89b700 1 -- 172.21.15.114:0/3402332085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdaa8105e60 msgr2=0x7fdaa819b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:52.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.196+0000 7fdaad89b700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdaa8105e60 0x7fdaa819b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.196+0000 7fdaad89b700 1 -- 172.21.15.114:0/3402332085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa810e8f0 msgr2=0x7fdaa819f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:52.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.196+0000 7fdaad89b700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa810e8f0 0x7fdaa819f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.196+0000 7fdaad89b700 1 -- 172.21.15.114:0/3402332085 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa819fca0 con 0x7fdaa8105510 2024-05-27T07:51:52.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.196+0000 7fdaae09c700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa810e8f0 0x7fdaa819f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:52.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.196+0000 7fdaad89b700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaa8105510 0x7fdaa819add0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4000c00 tx=0x7fdaa400bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:52.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.197+0000 7fda9effd700 1 -- 172.21.15.114:0/3402332085 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4027070 con 0x7fdaa8105510 2024-05-27T07:51:52.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.197+0000 7fda9effd700 1 -- 172.21.15.114:0/3402332085 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaa4022470 con 0x7fdaa8105510 2024-05-27T07:51:52.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.197+0000 7fdaafaff700 1 -- 172.21.15.114:0/3402332085 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa819ff30 con 0x7fdaa8105510 2024-05-27T07:51:52.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.197+0000 7fda9effd700 1 -- 172.21.15.114:0/3402332085 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4017650 con 0x7fdaa8105510 2024-05-27T07:51:52.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.197+0000 7fdaafaff700 1 -- 172.21.15.114:0/3402332085 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa81a78b0 con 0x7fdaa8105510 2024-05-27T07:51:52.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.199+0000 7fda9effd700 1 -- 172.21.15.114:0/3402332085 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fdaa402a070 con 0x7fdaa8105510 2024-05-27T07:51:52.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.200+0000 7fda9effd700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fda940b2a70 0x7fda940b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.200+0000 7fdaad09a700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fda940b2a70 0x7fda940b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.200+0000 7fda9effd700 1 -- 172.21.15.114:0/3402332085 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7fdaa40e3630 con 0x7fdaa8105510 2024-05-27T07:51:52.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdabe514b80 1 Processor -- start 2024-05-27T07:51:52.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdaad09a700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fda940b2a70 0x7fda940b4f80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8102370 tx=0x7fda98009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:52.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdabe514b80 1 -- start start 2024-05-27T07:51:52.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdabe514b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d223e40 0x55771d3a8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdabe514b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d3a9070 0x55771d3a2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdabe514b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3a3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdabe514b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55771d2d0d70 con 0x55771d3a3120 2024-05-27T07:51:52.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdabe514b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55771d2c0910 con 0x55771d223e40 2024-05-27T07:51:52.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdabe514b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55771d2c4950 con 0x55771d3a9070 2024-05-27T07:51:52.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdaae09c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3a3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdaae09c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3a3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58198/0 (socket says 172.21.15.114:58198) 2024-05-27T07:51:52.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdaae09c700 1 -- 172.21.15.114:0/81182115 learned_addr learned my addr 172.21.15.114:0/81182115 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:52.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdaad09a700 1 --2- 172.21.15.114:0/81182115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d3a9070 0x55771d3a2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.201+0000 7fdaad89b700 1 --2- 172.21.15.114:0/81182115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d223e40 0x55771d3a8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.202+0000 7fdaae09c700 1 -- 172.21.15.114:0/81182115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d3a9070 msgr2=0x55771d3a2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:52.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.202+0000 7fdaae09c700 1 --2- 172.21.15.114:0/81182115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d3a9070 0x55771d3a2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.202+0000 7fdaae09c700 1 -- 172.21.15.114:0/81182115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d223e40 msgr2=0x55771d3a8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:52.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.202+0000 7fdaae09c700 1 --2- 172.21.15.114:0/81182115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d223e40 0x55771d3a8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.202+0000 7fdaae09c700 1 -- 172.21.15.114:0/81182115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55771d269fb0 con 0x55771d3a3120 2024-05-27T07:51:52.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.202+0000 7fdaad09a700 1 --2- 172.21.15.114:0/81182115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d3a9070 0x55771d3a2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:52.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.202+0000 7fdaae09c700 1 --2- 172.21.15.114:0/81182115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3a3530 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0003eb0 tx=0x7fdaa0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29337960bffe24f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:52.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.202+0000 7fda93fff700 1 -- 172.21.15.114:0/81182115 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0019020 con 0x55771d3a3120 2024-05-27T07:51:52.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.202+0000 7fda93fff700 1 -- 172.21.15.114:0/81182115 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaa00092d0 con 0x55771d3a3120 2024-05-27T07:51:52.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.202+0000 7fda93fff700 1 -- 172.21.15.114:0/81182115 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa000e7e0 con 0x55771d3a3120 2024-05-27T07:51:52.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.203+0000 7fdabe514b80 1 -- 172.21.15.114:0/81182115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 msgr2=0x55771d3a3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:52.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.203+0000 7fdabe514b80 1 --2- 172.21.15.114:0/81182115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3a3530 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0003eb0 tx=0x7fdaa0009920 comp rx=0 tx=0).stop 2024-05-27T07:51:52.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.203+0000 7fdabe514b80 1 -- 172.21.15.114:0/81182115 shutdown_connections 2024-05-27T07:51:52.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.203+0000 7fdabe514b80 1 --2- 172.21.15.114:0/81182115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3a3530 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.203+0000 7fdabe514b80 1 --2- 172.21.15.114:0/81182115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d223e40 0x55771d3a8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.203+0000 7fdabe514b80 1 --2- 172.21.15.114:0/81182115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d3a9070 0x55771d3a2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.203+0000 7fdabe514b80 1 -- 172.21.15.114:0/81182115 >> 172.21.15.114:0/81182115 conn(0x55771d226770 msgr2=0x55771d39dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:52.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.203+0000 7fdabe514b80 1 -- 172.21.15.114:0/81182115 shutdown_connections 2024-05-27T07:51:52.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.203+0000 7fdabe514b80 1 -- 172.21.15.114:0/81182115 wait complete. 2024-05-27T07:51:52.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdabe514b80 1 -- start start 2024-05-27T07:51:52.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdabe514b80 1 Processor -- start 2024-05-27T07:51:52.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdabe514b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d223e40 0x55771d3ac810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdabe514b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdabe514b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a9070 0x55771d3ad290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdabe514b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55771d1c6fa0 con 0x55771d3a3120 2024-05-27T07:51:52.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdabe514b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55771d228da0 con 0x55771d3a9070 2024-05-27T07:51:52.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdabe514b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55771d3b3f70 con 0x55771d223e40 2024-05-27T07:51:52.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdabe514b80 10 client.0 _ll_register_callbacks cb 0x55771d2575b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:52.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdaad09a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdaad09a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58206/0 (socket says 172.21.15.114:58206) 2024-05-27T07:51:52.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdaad09a700 1 -- 172.21.15.114:0/3494852065 learned_addr learned my addr 172.21.15.114:0/3494852065 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:52.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdaad89b700 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d223e40 0x55771d3ac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.204+0000 7fdaae09c700 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a9070 0x55771d3ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fdaad09a700 1 -- 172.21.15.114:0/3494852065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d223e40 msgr2=0x55771d3ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:52.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fdaad09a700 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d223e40 0x55771d3ac810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fdaad09a700 1 -- 172.21.15.114:0/3494852065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a9070 msgr2=0x55771d3ad290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:52.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fdaad09a700 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a9070 0x55771d3ad290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fdaad09a700 1 -- 172.21.15.114:0/3494852065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55771d2108d0 con 0x55771d3a3120 2024-05-27T07:51:52.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fdaad89b700 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d223e40 0x55771d3ac810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:52.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fdaad09a700 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3acd50 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fda9801d8a0 tx=0x7fda980047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1bb5b71a007e83 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:52.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fda917fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:51:52.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fda917fa700 1 -- 172.21.15.114:0/3494852065 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda98004240 con 0x55771d3a3120 2024-05-27T07:51:52.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fda917fa700 1 -- 172.21.15.114:0/3494852065 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7fda98009040 con 0x55771d3a3120 2024-05-27T07:51:52.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fdabe514b80 1 -- 172.21.15.114:0/3494852065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdaa00092d0 con 0x55771d3a3120 2024-05-27T07:51:52.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fda917fa700 1 -- 172.21.15.114:0/3494852065 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fda98030050 con 0x55771d3a3120 2024-05-27T07:51:52.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fdabe514b80 10 client.18282 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:52.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fdabe514b80 1 -- 172.21.15.114:0/3494852065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55771d27a760 con 0x55771d3a3120 2024-05-27T07:51:52.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.205+0000 7fda917fa700 1 -- 172.21.15.114:0/3494852065 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaa00092d0 con 0x55771d3a3120 2024-05-27T07:51:52.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.206+0000 7fda917fa700 1 -- 172.21.15.114:0/3494852065 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7fda98028ba0 con 0x55771d3a3120 2024-05-27T07:51:52.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.206+0000 7fdabe514b80 10 client.18282 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:52.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.206+0000 7fdabe514b80 10 client.18282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:52.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.206+0000 7fdabe514b80 20 client.18282 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:52.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.206+0000 7fdabe514b80 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55771d3c10f0 0x55771d3c35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.206+0000 7fdabe514b80 4 client.18282 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:52.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.206+0000 7fdabe514b80 1 -- 172.21.15.114:0/3494852065 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55771d2d2cc0 con 0x55771d3c10f0 2024-05-27T07:51:52.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.206+0000 7fdaad89b700 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55771d3c10f0 0x55771d3c35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.206+0000 7fda917fa700 10 client.18282 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:52.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.206+0000 7fdaad89b700 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55771d3c10f0 0x55771d3c35a0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f888c832d5ffc8d0 server_cookie=8455b8c71e5791cf in_seq=0 out_seq=0 2024-05-27T07:51:52.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.207+0000 7fda917fa700 1 -- 172.21.15.114:0/3494852065 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55771d2d2cc0 con 0x55771d3c10f0 2024-05-27T07:51:52.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.207+0000 7fda917fa700 10 client.18282 handle_command_reply: tid=0 2024-05-27T07:51:52.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.207+0000 7fdabe514b80 1 client.18282 shutdown 2024-05-27T07:51:52.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.207+0000 7fdabe514b80 1 -- 172.21.15.114:0/3494852065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 msgr2=0x55771d3acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:52.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.207+0000 7fdabe514b80 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3acd50 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fda9801d8a0 tx=0x7fda980047c0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.207+0000 7fdabe514b80 1 -- 172.21.15.114:0/3494852065 shutdown_connections 2024-05-27T07:51:52.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.207+0000 7fdabe514b80 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55771d3c10f0 0x55771d3c35a0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.207+0000 7fdabe514b80 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a9070 0x55771d3ad290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.208+0000 7fdabe514b80 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d223e40 0x55771d3ac810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.208+0000 7fda917fa700 0 client.18282 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:52.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.208+0000 7fdabe514b80 1 --2- 172.21.15.114:0/3494852065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3acd50 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.208+0000 7fdabe514b80 1 -- 172.21.15.114:0/3494852065 >> 172.21.15.114:0/3494852065 conn(0x55771d226770 msgr2=0x55771d3a1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:52.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.208+0000 7fdabe514b80 1 -- 172.21.15.114:0/3494852065 shutdown_connections 2024-05-27T07:51:52.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.208+0000 7fdabe514b80 1 -- 172.21.15.114:0/3494852065 wait complete. 2024-05-27T07:51:52.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.208+0000 7fdabe514b80 20 client.18282 trim_cache size 0 max 16384 2024-05-27T07:51:52.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.232+0000 7fdabe514b80 1 Processor -- start 2024-05-27T07:51:52.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.232+0000 7fdabe514b80 1 -- start start 2024-05-27T07:51:52.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.232+0000 7fdabe514b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d223e40 0x55771d3a2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.232+0000 7fdabe514b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a3120 0x55771d3aba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.232+0000 7fdabe514b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d3a9070 0x55771d3ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.232+0000 7fdabe514b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81afca0 con 0x55771d223e40 2024-05-27T07:51:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.232+0000 7fdabe514b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81ae410 con 0x55771d3a3120 2024-05-27T07:51:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.232+0000 7fdabe514b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81ac280 con 0x55771d3a9070 2024-05-27T07:51:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.232+0000 7fdaad89b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d223e40 0x55771d3a2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.232+0000 7fdaad89b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d223e40 0x55771d3a2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58220/0 (socket says 172.21.15.114:58220) 2024-05-27T07:51:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.232+0000 7fdaad89b700 1 -- 172.21.15.114:0/3349631950 learned_addr learned my addr 172.21.15.114:0/3349631950 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.232+0000 7fdaae09c700 1 --2- 172.21.15.114:0/3349631950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d3a9070 0x55771d3ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.233+0000 7fdaad09a700 1 --2- 172.21.15.114:0/3349631950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a3120 0x55771d3aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.233+0000 7fdaad89b700 1 -- 172.21.15.114:0/3349631950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d3a9070 msgr2=0x55771d3ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.233+0000 7fdaad89b700 1 --2- 172.21.15.114:0/3349631950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d3a9070 0x55771d3ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.233+0000 7fdaad89b700 1 -- 172.21.15.114:0/3349631950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a3120 msgr2=0x55771d3aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:52.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.233+0000 7fdaad89b700 1 --2- 172.21.15.114:0/3349631950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a3120 0x55771d3aba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.233+0000 7fdaad89b700 1 -- 172.21.15.114:0/3349631950 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55771d3ad610 con 0x55771d223e40 2024-05-27T07:51:52.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.233+0000 7fdaad09a700 1 --2- 172.21.15.114:0/3349631950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a3120 0x55771d3aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:51:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.233+0000 7fdaad89b700 1 --2- 172.21.15.114:0/3349631950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d223e40 0x55771d3a2be0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fdaa40e0fd0 tx=0x7fdaa401a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1cd79a0eaa603a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.233+0000 7fda937fe700 1 -- 172.21.15.114:0/3349631950 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa40ae9a0 con 0x55771d223e40 2024-05-27T07:51:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.233+0000 7fda937fe700 1 -- 172.21.15.114:0/3349631950 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaa40b33c0 con 0x55771d223e40 2024-05-27T07:51:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.233+0000 7fda937fe700 1 -- 172.21.15.114:0/3349631950 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa40e3200 con 0x55771d223e40 2024-05-27T07:51:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.234+0000 7fdabe514b80 1 -- 172.21.15.114:0/3349631950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d223e40 msgr2=0x55771d3a2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.234+0000 7fdabe514b80 1 --2- 172.21.15.114:0/3349631950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d223e40 0x55771d3a2be0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fdaa40e0fd0 tx=0x7fdaa401a040 comp rx=0 tx=0).stop 2024-05-27T07:51:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.234+0000 7fdabe514b80 1 -- 172.21.15.114:0/3349631950 shutdown_connections 2024-05-27T07:51:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.234+0000 7fdabe514b80 1 --2- 172.21.15.114:0/3349631950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d3a9070 0x55771d3ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.234+0000 7fdabe514b80 1 --2- 172.21.15.114:0/3349631950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d223e40 0x55771d3a2be0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.234+0000 7fdabe514b80 1 --2- 172.21.15.114:0/3349631950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a3120 0x55771d3aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.234+0000 7fdabe514b80 1 -- 172.21.15.114:0/3349631950 >> 172.21.15.114:0/3349631950 conn(0x55771d226770 msgr2=0x55771d3974f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.234+0000 7fdabe514b80 1 -- 172.21.15.114:0/3349631950 shutdown_connections 2024-05-27T07:51:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.234+0000 7fdabe514b80 1 -- 172.21.15.114:0/3349631950 wait complete. 2024-05-27T07:51:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.235+0000 7fdabe514b80 1 -- start start 2024-05-27T07:51:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.235+0000 7fdabe514b80 1 Processor -- start 2024-05-27T07:51:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.235+0000 7fdabe514b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d223e40 0x55771d3b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.235+0000 7fdabe514b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.235+0000 7fdabe514b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a9070 0x55771d3c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.235+0000 7fdabe514b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81ac8d0 con 0x55771d3a3120 2024-05-27T07:51:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.235+0000 7fdabe514b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81add40 con 0x55771d3a9070 2024-05-27T07:51:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.235+0000 7fdabe514b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81acfa0 con 0x55771d223e40 2024-05-27T07:51:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdaad89b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d223e40 0x55771d3b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdaad09a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdabe514b80 10 client.0 _ll_register_callbacks cb 0x55771d2575b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdaad89b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d223e40 0x55771d3b99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47306/0 (socket says 172.21.15.114:47306) 2024-05-27T07:51:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdaad89b700 1 -- 172.21.15.114:0/608817768 learned_addr learned my addr 172.21.15.114:0/608817768 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdaae09c700 1 --2- 172.21.15.114:0/608817768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a9070 0x55771d3c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdaad89b700 1 -- 172.21.15.114:0/608817768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a9070 msgr2=0x55771d3c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdaad89b700 1 --2- 172.21.15.114:0/608817768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a9070 0x55771d3c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdaad89b700 1 -- 172.21.15.114:0/608817768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 msgr2=0x55771d3c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdaad89b700 1 --2- 172.21.15.114:0/608817768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3c85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdaad89b700 1 -- 172.21.15.114:0/608817768 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55771d3c9230 con 0x55771d223e40 2024-05-27T07:51:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdaad09a700 1 --2- 172.21.15.114:0/608817768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdaad89b700 1 --2- 172.21.15.114:0/608817768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d223e40 0x55771d3b99b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400b6c0 tx=0x7fdaa4022b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95736fdc459e9015 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fda917fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:51:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fda917fa700 1 -- 172.21.15.114:0/608817768 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa40ae9a0 con 0x55771d223e40 2024-05-27T07:51:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fda917fa700 1 -- 172.21.15.114:0/608817768 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7fdaa40aa850 con 0x55771d223e40 2024-05-27T07:51:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.236+0000 7fdabe514b80 1 -- 172.21.15.114:0/608817768 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdaa40b33c0 con 0x55771d223e40 2024-05-27T07:51:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.237+0000 7fda917fa700 1 -- 172.21.15.114:0/608817768 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fdaa40aaa50 con 0x55771d223e40 2024-05-27T07:51:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.237+0000 7fdabe514b80 10 client.27844 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.237+0000 7fdabe514b80 1 -- 172.21.15.114:0/608817768 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55771d42bb50 con 0x55771d223e40 2024-05-27T07:51:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.237+0000 7fda917fa700 1 -- 172.21.15.114:0/608817768 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaa40b33c0 con 0x55771d223e40 2024-05-27T07:51:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.237+0000 7fda917fa700 1 -- 172.21.15.114:0/608817768 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7fdaa40cb350 con 0x55771d223e40 2024-05-27T07:51:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.237+0000 7fdabe514b80 10 client.27844 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.237+0000 7fdabe514b80 10 client.27844 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.237+0000 7fdabe514b80 20 client.27844 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.237+0000 7fdabe514b80 1 --2- 172.21.15.114:0/608817768 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55771d42c200 0x55771d42e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.237+0000 7fdabe514b80 4 client.27844 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:51:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.237+0000 7fdabe514b80 1 -- 172.21.15.114:0/608817768 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55771d3a83c0 con 0x55771d42c200 2024-05-27T07:51:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.237+0000 7fdaad09a700 1 --2- 172.21.15.114:0/608817768 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55771d42c200 0x55771d42e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.238+0000 7fda917fa700 10 client.27844 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.238+0000 7fdaad09a700 1 --2- 172.21.15.114:0/608817768 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55771d42c200 0x55771d42e620 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf58a3f64b8c3cd server_cookie=8f452173ca1c6609 in_seq=0 out_seq=0 2024-05-27T07:51:52.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.238+0000 7fda917fa700 1 -- 172.21.15.114:0/608817768 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55771d3a83c0 con 0x55771d42c200 2024-05-27T07:51:52.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.238+0000 7fda917fa700 10 client.27844 handle_command_reply: tid=0 2024-05-27T07:51:52.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.239+0000 7fdabe514b80 1 client.27844 shutdown 2024-05-27T07:51:52.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.239+0000 7fdabe514b80 1 -- 172.21.15.114:0/608817768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d223e40 msgr2=0x55771d3b99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:52.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.239+0000 7fdabe514b80 1 --2- 172.21.15.114:0/608817768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d223e40 0x55771d3b99b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400b6c0 tx=0x7fdaa4022b70 comp rx=0 tx=0).stop 2024-05-27T07:51:52.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.239+0000 7fdabe514b80 1 -- 172.21.15.114:0/608817768 shutdown_connections 2024-05-27T07:51:52.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.239+0000 7fdabe514b80 1 --2- 172.21.15.114:0/608817768 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55771d42c200 0x55771d42e620 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.239+0000 7fdabe514b80 1 --2- 172.21.15.114:0/608817768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55771d3a9070 0x55771d3c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.239+0000 7fdabe514b80 1 --2- 172.21.15.114:0/608817768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55771d223e40 0x55771d3b99b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.239+0000 7fda917fa700 0 client.27844 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:52.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.239+0000 7fdabe514b80 1 --2- 172.21.15.114:0/608817768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55771d3a3120 0x55771d3c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.239+0000 7fdabe514b80 1 -- 172.21.15.114:0/608817768 >> 172.21.15.114:0/608817768 conn(0x55771d226770 msgr2=0x55771d3ab760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:52.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.239+0000 7fdabe514b80 1 -- 172.21.15.114:0/608817768 shutdown_connections 2024-05-27T07:51:52.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.239+0000 7fdabe514b80 1 -- 172.21.15.114:0/608817768 wait complete. 2024-05-27T07:51:52.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.239+0000 7fdabe514b80 20 client.27844 trim_cache size 0 max 16384 2024-05-27T07:51:52.248 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:51:52.248 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:51:52.248 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:51:52.248 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:51:52.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.240+0000 7fdaafaff700 1 -- 172.21.15.114:0/3402332085 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fda940b2a70 msgr2=0x7fda940b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:52.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.240+0000 7fdaafaff700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fda940b2a70 0x7fda940b4f80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8102370 tx=0x7fda98009260 comp rx=0 tx=0).stop 2024-05-27T07:51:52.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.240+0000 7fdaafaff700 1 -- 172.21.15.114:0/3402332085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaa8105510 msgr2=0x7fdaa819add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:52.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.240+0000 7fdaafaff700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaa8105510 0x7fdaa819add0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4000c00 tx=0x7fdaa400bfd0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.240+0000 7fdaafaff700 1 -- 172.21.15.114:0/3402332085 shutdown_connections 2024-05-27T07:51:52.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.240+0000 7fdaafaff700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa810e8f0 0x7fdaa819f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.240+0000 7fdaafaff700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaa8105510 0x7fdaa819add0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.240+0000 7fdaafaff700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fda940b2a70 0x7fda940b4f80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.240+0000 7fdaafaff700 1 --2- 172.21.15.114:0/3402332085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdaa8105e60 0x7fdaa819b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:52.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.240+0000 7fdaafaff700 1 -- 172.21.15.114:0/3402332085 >> 172.21.15.114:0/3402332085 conn(0x7fdaa80fef20 msgr2=0x7fdaa80ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:52.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.240+0000 7fdaafaff700 1 -- 172.21.15.114:0/3402332085 shutdown_connections 2024-05-27T07:51:52.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:52.240+0000 7fdaafaff700 1 -- 172.21.15.114:0/3402332085 wait complete. 2024-05-27T07:51:52.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1181655 9.78 MB/sec execute 484 sec latency 36.580 ms 2024-05-27T07:51:52.948 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:52 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:51:52.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:52 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:51:52.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:52 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:51:52.988 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:51:52.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:51:53.244 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:53.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1183008 9.78 MB/sec execute 485 sec latency 146.883 ms 2024-05-27T07:51:53.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:53 smithi163 ceph-mon[36451]: pgmap v403: 129 pgs: 129 active+clean; 64 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 416 op/s 2024-05-27T07:51:53.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:53 smithi186 ceph-mon[36647]: pgmap v403: 129 pgs: 129 active+clean; 64 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 416 op/s 2024-05-27T07:51:54.018 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:53 smithi114 ceph-mon[32038]: pgmap v403: 129 pgs: 129 active+clean; 64 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 416 op/s 2024-05-27T07:51:54.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1184254 9.77 MB/sec execute 486 sec latency 28.773 ms 2024-05-27T07:51:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.001+0000 7fcba2e53700 1 -- 172.21.15.114:0/98942564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb9c107e00 msgr2=0x7fcb9c108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.001+0000 7fcba2e53700 1 --2- 172.21.15.114:0/98942564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb9c107e00 0x7fcb9c108210 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fcb880099c0 tx=0x7fcb88009cd0 comp rx=0 tx=0).stop 2024-05-27T07:51:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.001+0000 7fcba2e53700 1 -- 172.21.15.114:0/98942564 shutdown_connections 2024-05-27T07:51:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.001+0000 7fcba2e53700 1 --2- 172.21.15.114:0/98942564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb9c108750 0x7fcb9c1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.001+0000 7fcba2e53700 1 --2- 172.21.15.114:0/98942564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb9c1034d0 0x7fcb9c1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.001+0000 7fcba2e53700 1 --2- 172.21.15.114:0/98942564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb9c107e00 0x7fcb9c108210 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.001+0000 7fcba2e53700 1 -- 172.21.15.114:0/98942564 >> 172.21.15.114:0/98942564 conn(0x7fcb9c0fef20 msgr2=0x7fcb9c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.001+0000 7fcba2e53700 1 -- 172.21.15.114:0/98942564 shutdown_connections 2024-05-27T07:51:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.001+0000 7fcba2e53700 1 -- 172.21.15.114:0/98942564 wait complete. 2024-05-27T07:51:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcba2e53700 1 Processor -- start 2024-05-27T07:51:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcba2e53700 1 -- start start 2024-05-27T07:51:55.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcba2e53700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb9c1034d0 0x7fcb9c199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:55.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcba2e53700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb9c107e00 0x7fcb9c19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:55.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcba2e53700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb9c108750 0x7fcb9c19e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:55.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcba2e53700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c113360 con 0x7fcb9c107e00 2024-05-27T07:51:55.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcba2e53700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c1131e0 con 0x7fcb9c108750 2024-05-27T07:51:55.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcba2e53700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c1134e0 con 0x7fcb9c1034d0 2024-05-27T07:51:55.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcb9bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb9c107e00 0x7fcb9c19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:55.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcb9bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb9c107e00 0x7fcb9c19a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36138/0 (socket says 172.21.15.114:36138) 2024-05-27T07:51:55.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcb9bfff700 1 -- 172.21.15.114:0/2993526660 learned_addr learned my addr 172.21.15.114:0/2993526660 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:55.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcba0bef700 1 --2- 172.21.15.114:0/2993526660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb9c1034d0 0x7fcb9c199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:55.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcb9bfff700 1 -- 172.21.15.114:0/2993526660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb9c1034d0 msgr2=0x7fcb9c199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:55.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcb9bfff700 1 --2- 172.21.15.114:0/2993526660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb9c1034d0 0x7fcb9c199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:55.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcb9bfff700 1 -- 172.21.15.114:0/2993526660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb9c108750 msgr2=0x7fcb9c19e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:55.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcb9bfff700 1 --2- 172.21.15.114:0/2993526660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb9c108750 0x7fcb9c19e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:55.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.002+0000 7fcb9bfff700 1 -- 172.21.15.114:0/2993526660 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb9c19ec00 con 0x7fcb9c107e00 2024-05-27T07:51:55.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.003+0000 7fcb9bfff700 1 --2- 172.21.15.114:0/2993526660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb9c107e00 0x7fcb9c19a2d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fcb880045e0 tx=0x7fcb88004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:55.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.003+0000 7fcb99ffb700 1 -- 172.21.15.114:0/2993526660 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88026070 con 0x7fcb9c107e00 2024-05-27T07:51:55.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.003+0000 7fcb99ffb700 1 -- 172.21.15.114:0/2993526660 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb88005940 con 0x7fcb9c107e00 2024-05-27T07:51:55.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.003+0000 7fcba2e53700 1 -- 172.21.15.114:0/2993526660 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb9c19ee90 con 0x7fcb9c107e00 2024-05-27T07:51:55.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.003+0000 7fcb99ffb700 1 -- 172.21.15.114:0/2993526660 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8800fb00 con 0x7fcb9c107e00 2024-05-27T07:51:55.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.003+0000 7fcba2e53700 1 -- 172.21.15.114:0/2993526660 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb9c19f370 con 0x7fcb9c107e00 2024-05-27T07:51:55.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.004+0000 7fcba2e53700 1 -- 172.21.15.114:0/2993526660 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb9c0512b0 con 0x7fcb9c107e00 2024-05-27T07:51:55.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.006+0000 7fcb99ffb700 1 -- 172.21.15.114:0/2993526660 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fcb88029070 con 0x7fcb9c107e00 2024-05-27T07:51:55.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.007+0000 7fcb99ffb700 1 --2- 172.21.15.114:0/2993526660 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcb8c0b29f0 0x7fcb8c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:55.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.007+0000 7fcba0bef700 1 --2- 172.21.15.114:0/2993526660 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcb8c0b29f0 0x7fcb8c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:55.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.007+0000 7fcb99ffb700 1 -- 172.21.15.114:0/2993526660 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7fcb880e3c00 con 0x7fcb9c107e00 2024-05-27T07:51:55.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.008+0000 7fcba0bef700 1 --2- 172.21.15.114:0/2993526660 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcb8c0b29f0 0x7fcb8c0b4f00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcb90007be0 tx=0x7fcb900073d0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:55.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.011+0000 7fcb99ffb700 1 -- 172.21.15.114:0/2993526660 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcb88013070 con 0x7fcb9c107e00 2024-05-27T07:51:55.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1186381 9.78 MB/sec execute 487 sec latency 51.099 ms 2024-05-27T07:51:55.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.304+0000 7fcba2e53700 1 -- 172.21.15.114:0/2993526660 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb9c02e5e0 con 0x7fcb9c107e00 2024-05-27T07:51:55.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.304+0000 7fcb99ffb700 1 -- 172.21.15.114:0/2993526660 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+8522 (secure 0 0 0) 0x7fcb880ab2c0 con 0x7fcb9c107e00 2024-05-27T07:51:55.305 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:51:55.305 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":27557,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3261223371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":3261223371},{"type":"v1","addr":"172.21.15.163:6835","nonce":3261223371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":67,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:51:26.564380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27634":{"gid":27634,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/3266101988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":3266101988},{"type":"v1","addr":"172.21.15.186:6833","nonce":3266101988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:51:55.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.307+0000 7fcba2e53700 1 -- 172.21.15.114:0/2993526660 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcb8c0b29f0 msgr2=0x7fcb8c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:55.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.307+0000 7fcba2e53700 1 --2- 172.21.15.114:0/2993526660 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcb8c0b29f0 0x7fcb8c0b4f00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcb90007be0 tx=0x7fcb900073d0 comp rx=0 tx=0).stop 2024-05-27T07:51:55.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.307+0000 7fcba2e53700 1 -- 172.21.15.114:0/2993526660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb9c107e00 msgr2=0x7fcb9c19a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:55.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.307+0000 7fcba2e53700 1 --2- 172.21.15.114:0/2993526660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb9c107e00 0x7fcb9c19a2d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fcb880045e0 tx=0x7fcb88004610 comp rx=0 tx=0).stop 2024-05-27T07:51:55.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.307+0000 7fcba2e53700 1 -- 172.21.15.114:0/2993526660 shutdown_connections 2024-05-27T07:51:55.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.307+0000 7fcba2e53700 1 --2- 172.21.15.114:0/2993526660 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcb8c0b29f0 0x7fcb8c0b4f00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:55.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.307+0000 7fcba2e53700 1 --2- 172.21.15.114:0/2993526660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb9c108750 0x7fcb9c19e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:55.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.307+0000 7fcba2e53700 1 --2- 172.21.15.114:0/2993526660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb9c1034d0 0x7fcb9c199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:55.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.307+0000 7fcba2e53700 1 --2- 172.21.15.114:0/2993526660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb9c107e00 0x7fcb9c19a2d0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:55.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.307+0000 7fcba2e53700 1 -- 172.21.15.114:0/2993526660 >> 172.21.15.114:0/2993526660 conn(0x7fcb9c0fef20 msgr2=0x7fcb9c1096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:55.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.307+0000 7fcba2e53700 1 -- 172.21.15.114:0/2993526660 shutdown_connections 2024-05-27T07:51:55.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:55.307+0000 7fcba2e53700 1 -- 172.21.15.114:0/2993526660 wait complete. 2024-05-27T07:51:55.309 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 67 2024-05-27T07:51:55.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:55 smithi163 ceph-mon[36451]: pgmap v404: 129 pgs: 129 active+clean; 64 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 260 op/s 2024-05-27T07:51:55.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:55 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2993526660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:51:55.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:55 smithi186 ceph-mon[36647]: pgmap v404: 129 pgs: 129 active+clean; 64 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 260 op/s 2024-05-27T07:51:55.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:55 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2993526660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:51:56.018 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:55 smithi114 ceph-mon[32038]: pgmap v404: 129 pgs: 129 active+clean; 64 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 260 op/s 2024-05-27T07:51:56.018 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:55 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2993526660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:51:56.048 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:51:56.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1187541 9.77 MB/sec execute 488 sec latency 16.864 ms 2024-05-27T07:51:56.307 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:51:57.290 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1189898 9.78 MB/sec execute 489 sec latency 33.449 ms 2024-05-27T07:51:57.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:57 smithi114 ceph-mon[32038]: pgmap v405: 129 pgs: 129 active+clean; 69 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 341 op/s 2024-05-27T07:51:57.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:57 smithi163 ceph-mon[36451]: pgmap v405: 129 pgs: 129 active+clean; 69 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 341 op/s 2024-05-27T07:51:57.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:57 smithi186 ceph-mon[36647]: pgmap v405: 129 pgs: 129 active+clean; 69 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 341 op/s 2024-05-27T07:51:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.096+0000 7ffac7714700 1 -- 172.21.15.114:0/4278143881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffac0107de0 msgr2=0x7ffac01081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.096+0000 7ffac7714700 1 --2- 172.21.15.114:0/4278143881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffac0107de0 0x7ffac01081f0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7ffabc0099c0 tx=0x7ffabc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.096+0000 7ffac7714700 1 -- 172.21.15.114:0/4278143881 shutdown_connections 2024-05-27T07:51:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.096+0000 7ffac7714700 1 --2- 172.21.15.114:0/4278143881 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffac0108730 0x7ffac0110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.096+0000 7ffac7714700 1 --2- 172.21.15.114:0/4278143881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffac01034b0 0x7ffac01038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.096+0000 7ffac7714700 1 --2- 172.21.15.114:0/4278143881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffac0107de0 0x7ffac01081f0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.096+0000 7ffac7714700 1 -- 172.21.15.114:0/4278143881 >> 172.21.15.114:0/4278143881 conn(0x7ffac00fedb0 msgr2=0x7ffac01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.096+0000 7ffac7714700 1 -- 172.21.15.114:0/4278143881 shutdown_connections 2024-05-27T07:51:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.096+0000 7ffac7714700 1 -- 172.21.15.114:0/4278143881 wait complete. 2024-05-27T07:51:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.097+0000 7ffac7714700 1 Processor -- start 2024-05-27T07:51:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.097+0000 7ffac7714700 1 -- start start 2024-05-27T07:51:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.097+0000 7ffac7714700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffac01034b0 0x7ffac019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.097+0000 7ffac7714700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffac0107de0 0x7ffac019b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.097+0000 7ffac7714700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffac0108730 0x7ffac019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.097+0000 7ffac7714700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffac0114470 con 0x7ffac01034b0 2024-05-27T07:51:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.097+0000 7ffac7714700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffac01142f0 con 0x7ffac0107de0 2024-05-27T07:51:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.097+0000 7ffac7714700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffac01145f0 con 0x7ffac0108730 2024-05-27T07:51:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.097+0000 7ffac54b0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffac01034b0 0x7ffac019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.097+0000 7ffac54b0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffac01034b0 0x7ffac019ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36166/0 (socket says 172.21.15.114:36166) 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.097+0000 7ffac54b0700 1 -- 172.21.15.114:0/1239728573 learned_addr learned my addr 172.21.15.114:0/1239728573 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.097+0000 7ffac5cb1700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffac0108730 0x7ffac019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.098+0000 7ffac4caf700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffac0107de0 0x7ffac019b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.098+0000 7ffac54b0700 1 -- 172.21.15.114:0/1239728573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffac0108730 msgr2=0x7ffac019f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.098+0000 7ffac54b0700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffac0108730 0x7ffac019f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.098+0000 7ffac54b0700 1 -- 172.21.15.114:0/1239728573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffac0107de0 msgr2=0x7ffac019b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.098+0000 7ffac54b0700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffac0107de0 0x7ffac019b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.098+0000 7ffac54b0700 1 -- 172.21.15.114:0/1239728573 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac019fcb0 con 0x7ffac01034b0 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.098+0000 7ffac54b0700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffac01034b0 0x7ffac019ade0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ffab000e9a0 tx=0x7ffab000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.098+0000 7ffab67fc700 1 -- 172.21.15.114:0/1239728573 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab000ca10 con 0x7ffac01034b0 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.098+0000 7ffac7714700 1 -- 172.21.15.114:0/1239728573 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac019ffa0 con 0x7ffac01034b0 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.098+0000 7ffab67fc700 1 -- 172.21.15.114:0/1239728573 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab000cbb0 con 0x7ffac01034b0 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.098+0000 7ffab67fc700 1 -- 172.21.15.114:0/1239728573 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0018680 con 0x7ffac01034b0 2024-05-27T07:51:58.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.098+0000 7ffac7714700 1 -- 172.21.15.114:0/1239728573 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac01a78c0 con 0x7ffac01034b0 2024-05-27T07:51:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.101+0000 7ffab67fc700 1 -- 172.21.15.114:0/1239728573 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ffab0018820 con 0x7ffac01034b0 2024-05-27T07:51:58.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.102+0000 7ffab67fc700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffaac0b29f0 0x7ffaac0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.102+0000 7ffac4caf700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffaac0b29f0 0x7ffaac0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.102+0000 7ffab67fc700 1 -- 172.21.15.114:0/1239728573 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7ffab00e2be0 con 0x7ffac01034b0 2024-05-27T07:51:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffad6129b80 1 Processor -- start 2024-05-27T07:51:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffad6129b80 1 -- start start 2024-05-27T07:51:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffac4caf700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffaac0b29f0 0x7ffaac0b4f00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffabc000c00 tx=0x7ffabc01a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffad6129b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d56435eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffad6129b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 0x55d564358be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffad6129b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d564359120 0x55d564359530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffad6129b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d564286d70 con 0x55d564359120 2024-05-27T07:51:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffad6129b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d564276910 con 0x55d5641d9e40 2024-05-27T07:51:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffad6129b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d56427a950 con 0x55d56435f070 2024-05-27T07:51:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffac54b0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d56435eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffac5cb1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d564359120 0x55d564359530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffac54b0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d56435eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:44730/0 (socket says 172.21.15.114:44730) 2024-05-27T07:51:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffac5cb1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d564359120 0x55d564359530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36178/0 (socket says 172.21.15.114:36178) 2024-05-27T07:51:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.103+0000 7ffac54b0700 1 -- 172.21.15.114:0/1780805717 learned_addr learned my addr 172.21.15.114:0/1780805717 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.104+0000 7ffac4caf700 1 --2- 172.21.15.114:0/1780805717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 0x55d564358be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.104+0000 7ffac54b0700 1 -- 172.21.15.114:0/1780805717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 msgr2=0x55d564358be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.104+0000 7ffac54b0700 1 --2- 172.21.15.114:0/1780805717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 0x55d564358be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.104+0000 7ffac54b0700 1 -- 172.21.15.114:0/1780805717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d564359120 msgr2=0x55d564359530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:58.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.104+0000 7ffac54b0700 1 --2- 172.21.15.114:0/1780805717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d564359120 0x55d564359530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.104+0000 7ffac54b0700 1 -- 172.21.15.114:0/1780805717 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d56421ffb0 con 0x55d5641d9e40 2024-05-27T07:51:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.104+0000 7ffac54b0700 1 --2- 172.21.15.114:0/1780805717 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d56435eb30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ffab00a9810 tx=0x7ffab00a9620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8ef699a784267b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.104+0000 7ffaab7fe700 1 -- 172.21.15.114:0/1780805717 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab00a9b10 con 0x55d5641d9e40 2024-05-27T07:51:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.104+0000 7ffaab7fe700 1 -- 172.21.15.114:0/1780805717 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab00adec0 con 0x55d5641d9e40 2024-05-27T07:51:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.104+0000 7ffaab7fe700 1 -- 172.21.15.114:0/1780805717 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab00b24e0 con 0x55d5641d9e40 2024-05-27T07:51:58.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.105+0000 7ffad6129b80 1 -- 172.21.15.114:0/1780805717 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 msgr2=0x55d56435eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:58.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.105+0000 7ffad6129b80 1 --2- 172.21.15.114:0/1780805717 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d56435eb30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ffab00a9810 tx=0x7ffab00a9620 comp rx=0 tx=0).stop 2024-05-27T07:51:58.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.105+0000 7ffad6129b80 1 -- 172.21.15.114:0/1780805717 shutdown_connections 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.105+0000 7ffad6129b80 1 --2- 172.21.15.114:0/1780805717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d564359120 0x55d564359530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.105+0000 7ffad6129b80 1 --2- 172.21.15.114:0/1780805717 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d56435eb30 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.105+0000 7ffad6129b80 1 --2- 172.21.15.114:0/1780805717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 0x55d564358be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.105+0000 7ffad6129b80 1 -- 172.21.15.114:0/1780805717 >> 172.21.15.114:0/1780805717 conn(0x55d5641dc770 msgr2=0x55d564353d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.106+0000 7ffad6129b80 1 -- 172.21.15.114:0/1780805717 shutdown_connections 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.106+0000 7ffad6129b80 1 -- 172.21.15.114:0/1780805717 wait complete. 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.106+0000 7ffad6129b80 1 -- start start 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.106+0000 7ffad6129b80 1 Processor -- start 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.107+0000 7ffad6129b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d5643629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.107+0000 7ffad6129b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d564359120 0x55d564362f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.107+0000 7ffad6129b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d56435f070 0x55d564363440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.107+0000 7ffac5cb1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d56435f070 0x55d564363440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.107+0000 7ffac5cb1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d56435f070 0x55d564363440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36186/0 (socket says 172.21.15.114:36186) 2024-05-27T07:51:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.107+0000 7ffac5cb1700 1 -- 172.21.15.114:0/4050615573 learned_addr learned my addr 172.21.15.114:0/4050615573 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.107+0000 7ffac4caf700 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d564359120 0x55d564362f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.107+0000 7ffad6129b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d56417cfa0 con 0x55d56435f070 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.107+0000 7ffad6129b80 1 -- 172.21.15.114:0/4050615573 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d5641deda0 con 0x55d5641d9e40 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.107+0000 7ffad6129b80 1 -- 172.21.15.114:0/4050615573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d564369f10 con 0x55d564359120 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.107+0000 7ffac54b0700 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d5643629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.107+0000 7ffad6129b80 10 client.0 _ll_register_callbacks cb 0x55d56420d5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffac54b0700 1 -- 172.21.15.114:0/4050615573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d564359120 msgr2=0x55d564362f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffac54b0700 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d564359120 0x55d564362f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffac54b0700 1 -- 172.21.15.114:0/4050615573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d56435f070 msgr2=0x55d564363440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffac54b0700 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d56435f070 0x55d564363440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffac54b0700 1 -- 172.21.15.114:0/4050615573 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5641c68d0 con 0x55d5641d9e40 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffac5cb1700 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d56435f070 0x55d564363440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffac54b0700 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d5643629c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffab000c4f0 tx=0x7ffab0010c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94d5fb730588ec74 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:58.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffaa8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffaa8ff9700 1 -- 172.21.15.114:0/4050615573 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab00a9b10 con 0x55d5641d9e40 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffad6129b80 1 -- 172.21.15.114:0/4050615573 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffab00adec0 con 0x55d5641d9e40 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffaa8ff9700 1 -- 172.21.15.114:0/4050615573 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7ffab00b26f0 con 0x55d5641d9e40 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffaa8ff9700 1 -- 172.21.15.114:0/4050615573 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7ffab00a8d70 con 0x55d5641d9e40 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffad6129b80 10 client.27719 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.108+0000 7ffad6129b80 1 -- 172.21.15.114:0/4050615573 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55d564230760 con 0x55d5641d9e40 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.109+0000 7ffaa8ff9700 1 -- 172.21.15.114:0/4050615573 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab00adec0 con 0x55d5641d9e40 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.109+0000 7ffaa8ff9700 1 -- 172.21.15.114:0/4050615573 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7ffab00cc2b0 con 0x55d5641d9e40 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.109+0000 7ffad6129b80 10 client.27719 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.109+0000 7ffad6129b80 10 client.27719 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.109+0000 7ffad6129b80 20 client.27719 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.109+0000 7ffad6129b80 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d564376df0 0x55d564379250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.109+0000 7ffad6129b80 4 client.27719 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.109+0000 7ffad6129b80 1 -- 172.21.15.114:0/4050615573 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d564288cc0 con 0x55d564376df0 2024-05-27T07:51:58.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.109+0000 7ffac4caf700 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d564376df0 0x55d564379250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.109+0000 7ffaa8ff9700 10 client.27719 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:58.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.110+0000 7ffac4caf700 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d564376df0 0x55d564379250 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=648dc2b11b9c898d server_cookie=4cc397b1472ac180 in_seq=0 out_seq=0 2024-05-27T07:51:58.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.110+0000 7ffaa8ff9700 1 -- 172.21.15.114:0/4050615573 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d564288cc0 con 0x55d564376df0 2024-05-27T07:51:58.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.110+0000 7ffaa8ff9700 10 client.27719 handle_command_reply: tid=0 2024-05-27T07:51:58.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.110+0000 7ffad6129b80 1 client.27719 shutdown 2024-05-27T07:51:58.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.111+0000 7ffad6129b80 1 -- 172.21.15.114:0/4050615573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 msgr2=0x55d5643629c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:58.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.111+0000 7ffad6129b80 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d5643629c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffab000c4f0 tx=0x7ffab0010c10 comp rx=0 tx=0).stop 2024-05-27T07:51:58.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.111+0000 7ffad6129b80 1 -- 172.21.15.114:0/4050615573 shutdown_connections 2024-05-27T07:51:58.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.111+0000 7ffad6129b80 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d56435f070 0x55d564363440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.111+0000 7ffad6129b80 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d564376df0 0x55d564379250 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.111+0000 7ffad6129b80 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d5643629c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.111+0000 7ffad6129b80 1 --2- 172.21.15.114:0/4050615573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d564359120 0x55d564362f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.111+0000 7ffad6129b80 1 -- 172.21.15.114:0/4050615573 >> 172.21.15.114:0/4050615573 conn(0x55d5641dc770 msgr2=0x55d564357bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:58.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.111+0000 7ffaa8ff9700 0 client.27719 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:58.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.111+0000 7ffad6129b80 1 -- 172.21.15.114:0/4050615573 shutdown_connections 2024-05-27T07:51:58.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.111+0000 7ffad6129b80 1 -- 172.21.15.114:0/4050615573 wait complete. 2024-05-27T07:51:58.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.111+0000 7ffad6129b80 20 client.27719 trim_cache size 0 max 16384 2024-05-27T07:51:58.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.127+0000 7ffad6129b80 1 Processor -- start 2024-05-27T07:51:58.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.127+0000 7ffad6129b80 1 -- start start 2024-05-27T07:51:58.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffad6129b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5641d9e40 0x55d564363280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffad6129b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d564359120 0x55d5643612d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffad6129b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 0x55d564361810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffad6129b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffac00754a0 con 0x55d5641d9e40 2024-05-27T07:51:58.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffad6129b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffac0073c10 con 0x55d564359120 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffad6129b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffac0071a60 con 0x55d56435f070 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffac54b0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5641d9e40 0x55d564363280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffac54b0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5641d9e40 0x55d564363280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36190/0 (socket says 172.21.15.114:36190) 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffac54b0700 1 -- 172.21.15.114:0/2319371586 learned_addr learned my addr 172.21.15.114:0/2319371586 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffac5cb1700 1 --2- 172.21.15.114:0/2319371586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 0x55d564361810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffac4caf700 1 --2- 172.21.15.114:0/2319371586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d564359120 0x55d5643612d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffac54b0700 1 -- 172.21.15.114:0/2319371586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 msgr2=0x55d564361810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffac54b0700 1 --2- 172.21.15.114:0/2319371586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 0x55d564361810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffac54b0700 1 -- 172.21.15.114:0/2319371586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d564359120 msgr2=0x55d5643612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffac54b0700 1 --2- 172.21.15.114:0/2319371586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d564359120 0x55d5643612d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffac54b0700 1 -- 172.21.15.114:0/2319371586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d564360f50 con 0x55d5641d9e40 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffac54b0700 1 --2- 172.21.15.114:0/2319371586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5641d9e40 0x55d564363280 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7ffab000c2d0 tx=0x7ffab00a9590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d223938cf52755 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.128+0000 7ffaaaffd700 1 -- 172.21.15.114:0/2319371586 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab00a9b10 con 0x55d5641d9e40 2024-05-27T07:51:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.129+0000 7ffaaaffd700 1 -- 172.21.15.114:0/2319371586 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab006f070 con 0x55d5641d9e40 2024-05-27T07:51:58.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.129+0000 7ffaaaffd700 1 -- 172.21.15.114:0/2319371586 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab00a9010 con 0x55d5641d9e40 2024-05-27T07:51:58.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.129+0000 7ffad6129b80 1 -- 172.21.15.114:0/2319371586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5641d9e40 msgr2=0x55d564363280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:58.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.129+0000 7ffad6129b80 1 --2- 172.21.15.114:0/2319371586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5641d9e40 0x55d564363280 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7ffab000c2d0 tx=0x7ffab00a9590 comp rx=0 tx=0).stop 2024-05-27T07:51:58.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.129+0000 7ffad6129b80 1 -- 172.21.15.114:0/2319371586 shutdown_connections 2024-05-27T07:51:58.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.129+0000 7ffad6129b80 1 --2- 172.21.15.114:0/2319371586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 0x55d564361810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.129+0000 7ffad6129b80 1 --2- 172.21.15.114:0/2319371586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5641d9e40 0x55d564363280 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.129+0000 7ffad6129b80 1 --2- 172.21.15.114:0/2319371586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d564359120 0x55d5643612d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.129+0000 7ffad6129b80 1 -- 172.21.15.114:0/2319371586 >> 172.21.15.114:0/2319371586 conn(0x55d5641dc770 msgr2=0x55d56435d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:58.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.130+0000 7ffad6129b80 1 -- 172.21.15.114:0/2319371586 shutdown_connections 2024-05-27T07:51:58.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.130+0000 7ffad6129b80 1 -- 172.21.15.114:0/2319371586 wait complete. 2024-05-27T07:51:58.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.130+0000 7ffad6129b80 1 -- start start 2024-05-27T07:51:58.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.130+0000 7ffad6129b80 1 Processor -- start 2024-05-27T07:51:58.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.130+0000 7ffad6129b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d56436f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffad6129b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d564359120 0x55d56436a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffad6129b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 0x55d56436a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffad6129b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffac00720d0 con 0x55d564359120 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffad6129b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffac0073540 con 0x55d5641d9e40 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffad6129b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffac00727a0 con 0x55d56435f070 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffad6129b80 10 client.0 _ll_register_callbacks cb 0x55d56420d5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffac54b0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d56436f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffac4caf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d564359120 0x55d56436a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffac5cb1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 0x55d56436a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffac54b0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d56436f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:44776/0 (socket says 172.21.15.114:44776) 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffac54b0700 1 -- 172.21.15.114:0/516035946 learned_addr learned my addr 172.21.15.114:0/516035946 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffac54b0700 1 -- 172.21.15.114:0/516035946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 msgr2=0x55d56436a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffac54b0700 1 --2- 172.21.15.114:0/516035946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 0x55d56436a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffac54b0700 1 -- 172.21.15.114:0/516035946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d564359120 msgr2=0x55d56436a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffac54b0700 1 --2- 172.21.15.114:0/516035946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d564359120 0x55d56436a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffac54b0700 1 -- 172.21.15.114:0/516035946 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d56436abe0 con 0x55d5641d9e40 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffac5cb1700 1 --2- 172.21.15.114:0/516035946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 0x55d56436a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffac54b0700 1 --2- 172.21.15.114:0/516035946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d56436f620 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ffab000e970 tx=0x7ffab0010c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80da1a49da818575 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffaa8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.131+0000 7ffaa8ff9700 1 -- 172.21.15.114:0/516035946 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab00adec0 con 0x55d5641d9e40 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffad6129b80 1 -- 172.21.15.114:0/516035946 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffab006f070 con 0x55d5641d9e40 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffaa8ff9700 1 -- 172.21.15.114:0/516035946 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7ffab00a6100 con 0x55d5641d9e40 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffaa8ff9700 1 -- 172.21.15.114:0/516035946 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7ffab008c050 con 0x55d5641d9e40 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffad6129b80 10 client.27725 fetch_fsmap learned FSMap version 67 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffad6129b80 1 -- 172.21.15.114:0/516035946 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55d56436abe0 con 0x55d5641d9e40 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffaa8ff9700 1 -- 172.21.15.114:0/516035946 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab006f070 con 0x55d5641d9e40 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffaa8ff9700 1 -- 172.21.15.114:0/516035946 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7ffab00af0a0 con 0x55d5641d9e40 2024-05-27T07:51:58.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffad6129b80 10 client.27725 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffad6129b80 10 client.27725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffad6129b80 20 client.27725 populate_metadata read hostname 'smithi114' 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffad6129b80 1 --2- 172.21.15.114:0/516035946 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d5643e07b0 0x55d5643e2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffad6129b80 4 client.27725 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffad6129b80 1 -- 172.21.15.114:0/516035946 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d5642824a0 con 0x55d5643e07b0 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.132+0000 7ffac4caf700 1 --2- 172.21.15.114:0/516035946 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d5643e07b0 0x55d5643e2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffaa8ff9700 10 client.27725 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffac4caf700 1 --2- 172.21.15.114:0/516035946 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d5643e07b0 0x55d5643e2bd0 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6fb1bf223e656f5 server_cookie=86adb3c9513828ca in_seq=0 out_seq=0 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffaa8ff9700 1 -- 172.21.15.114:0/516035946 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d5642824a0 con 0x55d5643e07b0 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffaa8ff9700 10 client.27725 handle_command_reply: tid=0 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffad6129b80 1 client.27725 shutdown 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffad6129b80 1 -- 172.21.15.114:0/516035946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 msgr2=0x55d56436f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffad6129b80 1 --2- 172.21.15.114:0/516035946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d56436f620 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ffab000e970 tx=0x7ffab0010c10 comp rx=0 tx=0).stop 2024-05-27T07:51:58.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffad6129b80 1 -- 172.21.15.114:0/516035946 shutdown_connections 2024-05-27T07:51:58.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffad6129b80 1 --2- 172.21.15.114:0/516035946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d56435f070 0x55d56436a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffad6129b80 1 --2- 172.21.15.114:0/516035946 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d5643e07b0 0x55d5643e2bd0 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffaa8ff9700 0 client.27725 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:51:58.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffad6129b80 1 --2- 172.21.15.114:0/516035946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d5641d9e40 0x55d56436f620 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffad6129b80 1 --2- 172.21.15.114:0/516035946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d564359120 0x55d56436a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffad6129b80 1 -- 172.21.15.114:0/516035946 >> 172.21.15.114:0/516035946 conn(0x55d5641dc770 msgr2=0x55d5643632e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:58.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffad6129b80 1 -- 172.21.15.114:0/516035946 shutdown_connections 2024-05-27T07:51:58.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.133+0000 7ffad6129b80 1 -- 172.21.15.114:0/516035946 wait complete. 2024-05-27T07:51:58.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.134+0000 7ffad6129b80 20 client.27725 trim_cache size 0 max 16384 2024-05-27T07:51:58.139 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:51:58.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.134+0000 7ffac7714700 1 -- 172.21.15.114:0/1239728573 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffaac0b29f0 msgr2=0x7ffaac0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:58.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.134+0000 7ffac7714700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffaac0b29f0 0x7ffaac0b4f00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffabc000c00 tx=0x7ffabc01a040 comp rx=0 tx=0).stop 2024-05-27T07:51:58.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.134+0000 7ffac7714700 1 -- 172.21.15.114:0/1239728573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffac01034b0 msgr2=0x7ffac019ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:51:58.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.134+0000 7ffac7714700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffac01034b0 0x7ffac019ade0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ffab000e9a0 tx=0x7ffab000ecb0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.134+0000 7ffac7714700 1 -- 172.21.15.114:0/1239728573 shutdown_connections 2024-05-27T07:51:58.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.134+0000 7ffac7714700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffac0108730 0x7ffac019f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.134+0000 7ffac7714700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffac01034b0 0x7ffac019ade0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.134+0000 7ffac7714700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffaac0b29f0 0x7ffaac0b4f00 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.134+0000 7ffac7714700 1 --2- 172.21.15.114:0/1239728573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffac0107de0 0x7ffac019b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:51:58.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.134+0000 7ffac7714700 1 -- 172.21.15.114:0/1239728573 >> 172.21.15.114:0/1239728573 conn(0x7ffac00fedb0 msgr2=0x7ffac01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:51:58.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.135+0000 7ffac7714700 1 -- 172.21.15.114:0/1239728573 shutdown_connections 2024-05-27T07:51:58.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:51:58.135+0000 7ffac7714700 1 -- 172.21.15.114:0/1239728573 wait complete. 2024-05-27T07:51:58.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1190726 9.76 MB/sec execute 490 sec latency 90.879 ms 2024-05-27T07:51:59.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1192739 9.76 MB/sec execute 491 sec latency 47.213 ms 2024-05-27T07:51:59.835 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:51:59.835 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:51:59.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:51:59 smithi163 ceph-mon[36451]: pgmap v406: 129 pgs: 129 active+clean; 69 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 283 op/s 2024-05-27T07:51:59.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:51:59 smithi186 ceph-mon[36647]: pgmap v406: 129 pgs: 129 active+clean; 69 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 283 op/s 2024-05-27T07:52:00.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:51:59 smithi114 ceph-mon[32038]: pgmap v406: 129 pgs: 129 active+clean; 69 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 283 op/s 2024-05-27T07:52:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:52:00.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1193974 9.76 MB/sec execute 492 sec latency 12.524 ms 2024-05-27T07:52:01.290 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1194399 9.74 MB/sec execute 493 sec latency 128.544 ms 2024-05-27T07:52:01.479 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4f2fc1a700 time 2024-05-27T07:52:01.167024+0000 2024-05-27T07:52:01.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:52:01.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:01.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4f4245fba6] 2024-05-27T07:52:01.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4f4245fdd1] 2024-05-27T07:52:01.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561784fa223d] 2024-05-27T07:52:01.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 4: (SnapRealm::build_snap_set() const+0x323) [0x561784f8d971] 2024-05-27T07:52:01.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 5: (SnapRealm::check_cache() const+0x13e) [0x561784f8bfa0] 2024-05-27T07:52:01.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561784f8c651] 2024-05-27T07:52:01.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561784f21347] 2024-05-27T07:52:01.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561784c1d87e] 2024-05-27T07:52:01.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561784c22e94] 2024-05-27T07:52:01.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561784c2d969] 2024-05-27T07:52:01.481 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 11: (MDLog::_replay_thread()+0x1f38) [0x561784fc648c] 2024-05-27T07:52:01.481 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 12: (MDLog::ReplayThread::entry()+0x11) [0x561784fcf9cb] 2024-05-27T07:52:01.481 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 13: (Thread::entry_wrapper()+0x43) [0x7f4f4242e72d] 2024-05-27T07:52:01.481 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4f4242e749] 2024-05-27T07:52:01.481 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4f40fd21ca] 2024-05-27T07:52:01.481 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 16: clone() 2024-05-27T07:52:01.481 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: *** Caught signal (Aborted) ** 2024-05-27T07:52:01.481 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: in thread 7f4f2fc1a700 thread_name:md_log_replay 2024-05-27T07:52:01.482 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.168+0000 7f4f2fc1a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4f2fc1a700 time 2024-05-27T07:52:01.167024+0000 2024-05-27T07:52:01.483 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:52:01.483 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.483 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:01.483 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4f4245fba6] 2024-05-27T07:52:01.483 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4f4245fdd1] 2024-05-27T07:52:01.483 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561784fa223d] 2024-05-27T07:52:01.483 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 4: (SnapRealm::build_snap_set() const+0x323) [0x561784f8d971] 2024-05-27T07:52:01.483 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 5: (SnapRealm::check_cache() const+0x13e) [0x561784f8bfa0] 2024-05-27T07:52:01.483 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561784f8c651] 2024-05-27T07:52:01.483 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561784f21347] 2024-05-27T07:52:01.483 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561784c1d87e] 2024-05-27T07:52:01.484 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561784c22e94] 2024-05-27T07:52:01.484 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561784c2d969] 2024-05-27T07:52:01.484 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 11: (MDLog::_replay_thread()+0x1f38) [0x561784fc648c] 2024-05-27T07:52:01.484 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 12: (MDLog::ReplayThread::entry()+0x11) [0x561784fcf9cb] 2024-05-27T07:52:01.484 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 13: (Thread::entry_wrapper()+0x43) [0x7f4f4242e72d] 2024-05-27T07:52:01.484 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4f4242e749] 2024-05-27T07:52:01.484 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4f40fd21ca] 2024-05-27T07:52:01.484 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 16: clone() 2024-05-27T07:52:01.484 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.484 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:01.484 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4f40fdcd20] 2024-05-27T07:52:01.485 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2: gsignal() 2024-05-27T07:52:01.485 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 3: abort() 2024-05-27T07:52:01.485 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4f4245fced] 2024-05-27T07:52:01.485 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4f4245fdd1] 2024-05-27T07:52:01.485 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561784fa223d] 2024-05-27T07:52:01.485 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 7: (SnapRealm::build_snap_set() const+0x323) [0x561784f8d971] 2024-05-27T07:52:01.485 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 8: (SnapRealm::check_cache() const+0x13e) [0x561784f8bfa0] 2024-05-27T07:52:01.485 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561784f8c651] 2024-05-27T07:52:01.485 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561784f21347] 2024-05-27T07:52:01.485 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561784c1d87e] 2024-05-27T07:52:01.485 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561784c22e94] 2024-05-27T07:52:01.485 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561784c2d969] 2024-05-27T07:52:01.485 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 14: (MDLog::_replay_thread()+0x1f38) [0x561784fc648c] 2024-05-27T07:52:01.486 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 15: (MDLog::ReplayThread::entry()+0x11) [0x561784fcf9cb] 2024-05-27T07:52:01.486 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 16: (Thread::entry_wrapper()+0x43) [0x7f4f4242e72d] 2024-05-27T07:52:01.486 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4f4242e749] 2024-05-27T07:52:01.486 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4f40fd21ca] 2024-05-27T07:52:01.486 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 19: clone() 2024-05-27T07:52:01.486 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.171+0000 7f4f2fc1a700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:52:01.486 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: in thread 7f4f2fc1a700 thread_name:md_log_replay 2024-05-27T07:52:01.486 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.486 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:01.486 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4f40fdcd20] 2024-05-27T07:52:01.486 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2: gsignal() 2024-05-27T07:52:01.486 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 3: abort() 2024-05-27T07:52:01.486 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4f4245fced] 2024-05-27T07:52:01.487 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4f4245fdd1] 2024-05-27T07:52:01.487 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561784fa223d] 2024-05-27T07:52:01.487 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 7: (SnapRealm::build_snap_set() const+0x323) [0x561784f8d971] 2024-05-27T07:52:01.487 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 8: (SnapRealm::check_cache() const+0x13e) [0x561784f8bfa0] 2024-05-27T07:52:01.487 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561784f8c651] 2024-05-27T07:52:01.487 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561784f21347] 2024-05-27T07:52:01.487 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561784c1d87e] 2024-05-27T07:52:01.487 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561784c22e94] 2024-05-27T07:52:01.487 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561784c2d969] 2024-05-27T07:52:01.487 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 14: (MDLog::_replay_thread()+0x1f38) [0x561784fc648c] 2024-05-27T07:52:01.487 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 15: (MDLog::ReplayThread::entry()+0x11) [0x561784fcf9cb] 2024-05-27T07:52:01.487 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 16: (Thread::entry_wrapper()+0x43) [0x7f4f4242e72d] 2024-05-27T07:52:01.487 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4f4242e749] 2024-05-27T07:52:01.488 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4f40fd21ca] 2024-05-27T07:52:01.488 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 19: clone() 2024-05-27T07:52:01.488 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:52:01.488 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.488 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: -1> 2024-05-27T07:52:01.168+0000 7f4f2fc1a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4f2fc1a700 time 2024-05-27T07:52:01.167024+0000 2024-05-27T07:52:01.488 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:52:01.488 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.488 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:01.488 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4f4245fba6] 2024-05-27T07:52:01.488 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4f4245fdd1] 2024-05-27T07:52:01.488 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561784fa223d] 2024-05-27T07:52:01.488 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 4: (SnapRealm::build_snap_set() const+0x323) [0x561784f8d971] 2024-05-27T07:52:01.488 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 5: (SnapRealm::check_cache() const+0x13e) [0x561784f8bfa0] 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561784f8c651] 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561784f21347] 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561784c1d87e] 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561784c22e94] 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561784c2d969] 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 11: (MDLog::_replay_thread()+0x1f38) [0x561784fc648c] 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 12: (MDLog::ReplayThread::entry()+0x11) [0x561784fcf9cb] 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 13: (Thread::entry_wrapper()+0x43) [0x7f4f4242e72d] 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4f4242e749] 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4f40fd21ca] 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 16: clone() 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 0> 2024-05-27T07:52:01.171+0000 7f4f2fc1a700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:52:01.489 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: in thread 7f4f2fc1a700 thread_name:md_log_replay 2024-05-27T07:52:01.490 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.490 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:01.490 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4f40fdcd20] 2024-05-27T07:52:01.490 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2: gsignal() 2024-05-27T07:52:01.490 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 3: abort() 2024-05-27T07:52:01.490 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4f4245fced] 2024-05-27T07:52:01.490 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4f4245fdd1] 2024-05-27T07:52:01.490 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561784fa223d] 2024-05-27T07:52:01.490 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 7: (SnapRealm::build_snap_set() const+0x323) [0x561784f8d971] 2024-05-27T07:52:01.490 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 8: (SnapRealm::check_cache() const+0x13e) [0x561784f8bfa0] 2024-05-27T07:52:01.490 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561784f8c651] 2024-05-27T07:52:01.490 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561784f21347] 2024-05-27T07:52:01.490 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561784c1d87e] 2024-05-27T07:52:01.491 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561784c22e94] 2024-05-27T07:52:01.491 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561784c2d969] 2024-05-27T07:52:01.491 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 14: (MDLog::_replay_thread()+0x1f38) [0x561784fc648c] 2024-05-27T07:52:01.491 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 15: (MDLog::ReplayThread::entry()+0x11) [0x561784fcf9cb] 2024-05-27T07:52:01.491 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 16: (Thread::entry_wrapper()+0x43) [0x7f4f4242e72d] 2024-05-27T07:52:01.491 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4f4242e749] 2024-05-27T07:52:01.491 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4f40fd21ca] 2024-05-27T07:52:01.491 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 19: clone() 2024-05-27T07:52:01.491 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:52:01.491 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.492 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: -1> 2024-05-27T07:52:01.168+0000 7f4f2fc1a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4f2fc1a700 time 2024-05-27T07:52:01.167024+0000 2024-05-27T07:52:01.492 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:52:01.492 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.492 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:01.492 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4f4245fba6] 2024-05-27T07:52:01.492 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4f4245fdd1] 2024-05-27T07:52:01.492 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561784fa223d] 2024-05-27T07:52:01.492 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 4: (SnapRealm::build_snap_set() const+0x323) [0x561784f8d971] 2024-05-27T07:52:01.492 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 5: (SnapRealm::check_cache() const+0x13e) [0x561784f8bfa0] 2024-05-27T07:52:01.492 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561784f8c651] 2024-05-27T07:52:01.492 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561784f21347] 2024-05-27T07:52:01.493 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561784c1d87e] 2024-05-27T07:52:01.493 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561784c22e94] 2024-05-27T07:52:01.493 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561784c2d969] 2024-05-27T07:52:01.493 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 11: (MDLog::_replay_thread()+0x1f38) [0x561784fc648c] 2024-05-27T07:52:01.493 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 12: (MDLog::ReplayThread::entry()+0x11) [0x561784fcf9cb] 2024-05-27T07:52:01.493 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 13: (Thread::entry_wrapper()+0x43) [0x7f4f4242e72d] 2024-05-27T07:52:01.493 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4f4242e749] 2024-05-27T07:52:01.493 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4f40fd21ca] 2024-05-27T07:52:01.493 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 16: clone() 2024-05-27T07:52:01.493 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.493 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 0> 2024-05-27T07:52:01.171+0000 7f4f2fc1a700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:52:01.493 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: in thread 7f4f2fc1a700 thread_name:md_log_replay 2024-05-27T07:52:01.493 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.494 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:01.494 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4f40fdcd20] 2024-05-27T07:52:01.494 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2: gsignal() 2024-05-27T07:52:01.494 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 3: abort() 2024-05-27T07:52:01.494 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4f4245fced] 2024-05-27T07:52:01.494 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4f4245fdd1] 2024-05-27T07:52:01.494 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561784fa223d] 2024-05-27T07:52:01.494 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 7: (SnapRealm::build_snap_set() const+0x323) [0x561784f8d971] 2024-05-27T07:52:01.494 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 8: (SnapRealm::check_cache() const+0x13e) [0x561784f8bfa0] 2024-05-27T07:52:01.494 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561784f8c651] 2024-05-27T07:52:01.494 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561784f21347] 2024-05-27T07:52:01.494 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561784c1d87e] 2024-05-27T07:52:01.494 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561784c22e94] 2024-05-27T07:52:01.495 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561784c2d969] 2024-05-27T07:52:01.495 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 14: (MDLog::_replay_thread()+0x1f38) [0x561784fc648c] 2024-05-27T07:52:01.495 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 15: (MDLog::ReplayThread::entry()+0x11) [0x561784fcf9cb] 2024-05-27T07:52:01.495 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 16: (Thread::entry_wrapper()+0x43) [0x7f4f4242e72d] 2024-05-27T07:52:01.495 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4f4242e749] 2024-05-27T07:52:01.495 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4f40fd21ca] 2024-05-27T07:52:01.495 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 19: clone() 2024-05-27T07:52:01.495 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:52:01.495 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75049]: 2024-05-27T07:52:01.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.858+0000 7ff16fed4700 1 -- 172.21.15.114:0/4036455398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff168107de0 msgr2=0x7ff1681081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:01.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.858+0000 7ff16fed4700 1 --2- 172.21.15.114:0/4036455398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff168107de0 0x7ff1681081f0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7ff1580099c0 tx=0x7ff158009cd0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.858+0000 7ff16fed4700 1 -- 172.21.15.114:0/4036455398 shutdown_connections 2024-05-27T07:52:01.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.858+0000 7ff16fed4700 1 --2- 172.21.15.114:0/4036455398 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff168108730 0x7ff168110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.858+0000 7ff16fed4700 1 --2- 172.21.15.114:0/4036455398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1681034b0 0x7ff1681038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.859+0000 7ff16fed4700 1 --2- 172.21.15.114:0/4036455398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff168107de0 0x7ff1681081f0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.859+0000 7ff16fed4700 1 -- 172.21.15.114:0/4036455398 >> 172.21.15.114:0/4036455398 conn(0x7ff1680fedb0 msgr2=0x7ff1681011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:01.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.859+0000 7ff16fed4700 1 -- 172.21.15.114:0/4036455398 shutdown_connections 2024-05-27T07:52:01.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.859+0000 7ff16fed4700 1 -- 172.21.15.114:0/4036455398 wait complete. 2024-05-27T07:52:01.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.859+0000 7ff16fed4700 1 Processor -- start 2024-05-27T07:52:01.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.859+0000 7ff16fed4700 1 -- start start 2024-05-27T07:52:01.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16fed4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1681034b0 0x7ff16819ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16fed4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff168107de0 0x7ff16819b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16fed4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff168108730 0x7ff16819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16fed4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff168114470 con 0x7ff168107de0 2024-05-27T07:52:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16fed4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff1681142f0 con 0x7ff1681034b0 2024-05-27T07:52:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16fed4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1681145f0 con 0x7ff168108730 2024-05-27T07:52:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16d46f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff168107de0 0x7ff16819b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16d46f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff168107de0 0x7ff16819b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36222/0 (socket says 172.21.15.114:36222) 2024-05-27T07:52:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16d46f700 1 -- 172.21.15.114:0/145513712 learned_addr learned my addr 172.21.15.114:0/145513712 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16dc70700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1681034b0 0x7ff16819ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16e471700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff168108730 0x7ff16819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16d46f700 1 -- 172.21.15.114:0/145513712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff168108730 msgr2=0x7ff16819f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16d46f700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff168108730 0x7ff16819f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16d46f700 1 -- 172.21.15.114:0/145513712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1681034b0 msgr2=0x7ff16819ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16d46f700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1681034b0 0x7ff16819ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16d46f700 1 -- 172.21.15.114:0/145513712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff16819fd50 con 0x7ff168107de0 2024-05-27T07:52:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.860+0000 7ff16d46f700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff168107de0 0x7ff16819b420 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff158005f50 tx=0x7ff158004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.861+0000 7ff15effd700 1 -- 172.21.15.114:0/145513712 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff158026070 con 0x7ff168107de0 2024-05-27T07:52:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.861+0000 7ff15effd700 1 -- 172.21.15.114:0/145513712 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff158003c70 con 0x7ff168107de0 2024-05-27T07:52:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.861+0000 7ff16fed4700 1 -- 172.21.15.114:0/145513712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff16819ffe0 con 0x7ff168107de0 2024-05-27T07:52:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.861+0000 7ff15effd700 1 -- 172.21.15.114:0/145513712 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15800f9c0 con 0x7ff168107de0 2024-05-27T07:52:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.861+0000 7ff16fed4700 1 -- 172.21.15.114:0/145513712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1681a04c0 con 0x7ff168107de0 2024-05-27T07:52:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.863+0000 7ff15effd700 1 -- 172.21.15.114:0/145513712 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ff158029070 con 0x7ff168107de0 2024-05-27T07:52:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.864+0000 7ff15effd700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff1540b29f0 0x7ff1540b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.865 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:01 smithi114 ceph-mon[32038]: pgmap v407: 129 pgs: 129 active+clean; 69 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 279 op/s 2024-05-27T07:52:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.864+0000 7ff15effd700 1 -- 172.21.15.114:0/145513712 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7ff1580e3160 con 0x7ff168107de0 2024-05-27T07:52:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.864+0000 7ff16dc70700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff1540b29f0 0x7ff1540b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.864+0000 7ff17e8e9b80 1 Processor -- start 2024-05-27T07:52:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.865+0000 7ff17e8e9b80 1 -- start start 2024-05-27T07:52:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.865+0000 7ff16dc70700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff1540b29f0 0x7ff1540b4f00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff16400f540 tx=0x7ff16400b040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.865+0000 7ff17e8e9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a4941310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.865+0000 7ff17e8e9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a4945af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.865+0000 7ff17e8e9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 0x5605a4946440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.865+0000 7ff17e8e9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5605a486adf0 con 0x5605a4946030 2024-05-27T07:52:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.865+0000 7ff17e8e9b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5605a485a990 con 0x5605a4941850 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.865+0000 7ff17e8e9b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5605a485e9d0 con 0x5605a47bdec0 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.865+0000 7ff16dc70700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a4941310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.865+0000 7ff16e471700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 0x5605a4946440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.865+0000 7ff16dc70700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a4941310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33970/0 (socket says 172.21.15.114:33970) 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.865+0000 7ff16dc70700 1 -- 172.21.15.114:0/229678201 learned_addr learned my addr 172.21.15.114:0/229678201 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.865+0000 7ff16d46f700 1 --2- 172.21.15.114:0/229678201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a4945af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.866+0000 7ff16dc70700 1 -- 172.21.15.114:0/229678201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 msgr2=0x5605a4945af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.866+0000 7ff16dc70700 1 --2- 172.21.15.114:0/229678201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a4945af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.866+0000 7ff16dc70700 1 -- 172.21.15.114:0/229678201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 msgr2=0x5605a4946440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.866+0000 7ff16dc70700 1 --2- 172.21.15.114:0/229678201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 0x5605a4946440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.866+0000 7ff16dc70700 1 -- 172.21.15.114:0/229678201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605a4804030 con 0x5605a47bdec0 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.866+0000 7ff16dc70700 1 --2- 172.21.15.114:0/229678201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a4941310 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff164003d20 tx=0x7ff164014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f50ca7d0cf8c42e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.866+0000 7ff153fff700 1 -- 172.21.15.114:0/229678201 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff164027070 con 0x5605a47bdec0 2024-05-27T07:52:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.866+0000 7ff153fff700 1 -- 172.21.15.114:0/229678201 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff16401e040 con 0x5605a47bdec0 2024-05-27T07:52:01.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.866+0000 7ff153fff700 1 -- 172.21.15.114:0/229678201 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16401d050 con 0x5605a47bdec0 2024-05-27T07:52:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.867+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/229678201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 msgr2=0x5605a4941310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.867+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/229678201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a4941310 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff164003d20 tx=0x7ff164014000 comp rx=0 tx=0).stop 2024-05-27T07:52:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.867+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/229678201 shutdown_connections 2024-05-27T07:52:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.867+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/229678201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 0x5605a4946440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.867+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/229678201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a4941310 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.867+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/229678201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a4945af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.867+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/229678201 >> 172.21.15.114:0/229678201 conn(0x5605a47c07f0 msgr2=0x5605a4937e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.867+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/229678201 shutdown_connections 2024-05-27T07:52:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.867+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/229678201 wait complete. 2024-05-27T07:52:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.868+0000 7ff17e8e9b80 1 -- start start 2024-05-27T07:52:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.868+0000 7ff17e8e9b80 1 Processor -- start 2024-05-27T07:52:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.868+0000 7ff17e8e9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a494ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.868+0000 7ff17e8e9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a494b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.868+0000 7ff17e8e9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 0x5605a494b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.868+0000 7ff17e8e9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5605a4761020 con 0x5605a4946030 2024-05-27T07:52:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.868+0000 7ff17e8e9b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5605a47c2e20 con 0x5605a4941850 2024-05-27T07:52:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.868+0000 7ff17e8e9b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5605a494e040 con 0x5605a47bdec0 2024-05-27T07:52:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff17e8e9b80 10 client.0 _ll_register_callbacks cb 0x5605a47f1630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16e471700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 0x5605a494b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16dc70700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a494ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16d46f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a494b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16dc70700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a494ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33978/0 (socket says 172.21.15.114:33978) 2024-05-27T07:52:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16d46f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a494b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:44840/0 (socket says 172.21.15.114:44840) 2024-05-27T07:52:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16e471700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 0x5605a494b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36228/0 (socket says 172.21.15.114:36228) 2024-05-27T07:52:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16dc70700 1 -- 172.21.15.114:0/1816044866 learned_addr learned my addr 172.21.15.114:0/1816044866 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16dc70700 1 -- 172.21.15.114:0/1816044866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 msgr2=0x5605a494b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16dc70700 1 --2- 172.21.15.114:0/1816044866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a494b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16dc70700 1 -- 172.21.15.114:0/1816044866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 msgr2=0x5605a494b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16dc70700 1 --2- 172.21.15.114:0/1816044866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 0x5605a494b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16dc70700 1 -- 172.21.15.114:0/1816044866 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605a47aa950 con 0x5605a47bdec0 2024-05-27T07:52:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16d46f700 1 --2- 172.21.15.114:0/1816044866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a494b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff16dc70700 1 --2- 172.21.15.114:0/1816044866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a494ada0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff164009960 tx=0x7ff164005aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=537eacb8f501f4b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.869+0000 7ff1517fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:52:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff1517fa700 1 -- 172.21.15.114:0/1816044866 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff164031020 con 0x5605a47bdec0 2024-05-27T07:52:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/1816044866 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff16401e040 con 0x5605a47bdec0 2024-05-27T07:52:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff1517fa700 1 -- 172.21.15.114:0/1816044866 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7ff164008f90 con 0x5605a47bdec0 2024-05-27T07:52:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff1517fa700 1 -- 172.21.15.114:0/1816044866 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7ff16401d020 con 0x5605a47bdec0 2024-05-27T07:52:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff17e8e9b80 10 client.27880 fetch_fsmap learned FSMap version 67 2024-05-27T07:52:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/1816044866 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5605a48147e0 con 0x5605a47bdec0 2024-05-27T07:52:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff1517fa700 1 -- 172.21.15.114:0/1816044866 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff16401e040 con 0x5605a47bdec0 2024-05-27T07:52:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff1517fa700 1 -- 172.21.15.114:0/1816044866 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7ff164037b70 con 0x5605a47bdec0 2024-05-27T07:52:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff17e8e9b80 10 client.27880 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:52:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff17e8e9b80 10 client.27880 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff17e8e9b80 20 client.27880 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/1816044866 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5605a495b020 0x5605a495d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff17e8e9b80 4 client.27880 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:52:01.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.870+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/1816044866 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605a486cd40 con 0x5605a495b020 2024-05-27T07:52:01.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.871+0000 7ff16d46f700 1 --2- 172.21.15.114:0/1816044866 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5605a495b020 0x5605a495d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.871+0000 7ff1517fa700 10 client.27880 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:01.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.871+0000 7ff16d46f700 1 --2- 172.21.15.114:0/1816044866 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5605a495b020 0x5605a495d4a0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebffb8019317b925 server_cookie=3afdd765f29a354b in_seq=0 out_seq=0 2024-05-27T07:52:01.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff1517fa700 1 -- 172.21.15.114:0/1816044866 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5605a486cd40 con 0x5605a495b020 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff1517fa700 10 client.27880 handle_command_reply: tid=0 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff17e8e9b80 1 client.27880 shutdown 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/1816044866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 msgr2=0x5605a494ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/1816044866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a494ada0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff164009960 tx=0x7ff164005aa0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/1816044866 shutdown_connections 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/1816044866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 0x5605a494b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/1816044866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a494ada0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/1816044866 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5605a495b020 0x5605a495d4a0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/1816044866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a494b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff1517fa700 0 client.27880 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/1816044866 >> 172.21.15.114:0/1816044866 conn(0x5605a47c07f0 msgr2=0x5605a493bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/1816044866 shutdown_connections 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.872+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/1816044866 wait complete. 2024-05-27T07:52:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.873+0000 7ff17e8e9b80 20 client.27880 trim_cache size 0 max 16384 2024-05-27T07:52:01.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.893+0000 7ff17e8e9b80 1 Processor -- start 2024-05-27T07:52:01.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.893+0000 7ff17e8e9b80 1 -- start start 2024-05-27T07:52:01.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.893+0000 7ff17e8e9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a49570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.893+0000 7ff17e8e9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a4949da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.893+0000 7ff17e8e9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 0x5605a4962c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.893+0000 7ff17e8e9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1681afca0 con 0x5605a4946030 2024-05-27T07:52:01.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.893+0000 7ff17e8e9b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff1681ae410 con 0x5605a4941850 2024-05-27T07:52:01.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.893+0000 7ff17e8e9b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1681ac280 con 0x5605a47bdec0 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.894+0000 7ff16e471700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 0x5605a4962c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.894+0000 7ff16dc70700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a49570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.894+0000 7ff16dc70700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a49570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33988/0 (socket says 172.21.15.114:33988) 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.894+0000 7ff16dc70700 1 -- 172.21.15.114:0/3828764491 learned_addr learned my addr 172.21.15.114:0/3828764491 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.894+0000 7ff16d46f700 1 --2- 172.21.15.114:0/3828764491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a4949da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.894+0000 7ff16dc70700 1 -- 172.21.15.114:0/3828764491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 msgr2=0x5605a4949da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.894+0000 7ff16dc70700 1 --2- 172.21.15.114:0/3828764491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a4949da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.894+0000 7ff16dc70700 1 -- 172.21.15.114:0/3828764491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 msgr2=0x5605a4962c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.894+0000 7ff16dc70700 1 --2- 172.21.15.114:0/3828764491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 0x5605a4962c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.894+0000 7ff16dc70700 1 -- 172.21.15.114:0/3828764491 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605a4933b20 con 0x5605a47bdec0 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.894+0000 7ff16dc70700 1 --2- 172.21.15.114:0/3828764491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a49570a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff164037f30 tx=0x7ff164005b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acb85e8245aa95ef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.894+0000 7ff1537fe700 1 -- 172.21.15.114:0/3828764491 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff164045070 con 0x5605a47bdec0 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.895+0000 7ff1537fe700 1 -- 172.21.15.114:0/3828764491 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff164028040 con 0x5605a47bdec0 2024-05-27T07:52:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.895+0000 7ff1537fe700 1 -- 172.21.15.114:0/3828764491 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff164031040 con 0x5605a47bdec0 2024-05-27T07:52:01.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.895+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/3828764491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 msgr2=0x5605a49570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.895+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/3828764491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a49570a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff164037f30 tx=0x7ff164005b10 comp rx=0 tx=0).stop 2024-05-27T07:52:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.895+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/3828764491 shutdown_connections 2024-05-27T07:52:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.895+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/3828764491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4946030 0x5605a4962c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.895+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/3828764491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a49570a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.895+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/3828764491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4941850 0x5605a4949da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.895+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/3828764491 >> 172.21.15.114:0/3828764491 conn(0x5605a47c07f0 msgr2=0x5605a4949690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.895+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/3828764491 shutdown_connections 2024-05-27T07:52:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.895+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/3828764491 wait complete. 2024-05-27T07:52:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.896+0000 7ff17e8e9b80 1 -- start start 2024-05-27T07:52:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.896+0000 7ff17e8e9b80 1 Processor -- start 2024-05-27T07:52:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.896+0000 7ff17e8e9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a4953650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.896+0000 7ff17e8e9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4941850 0x5605a49509d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.896+0000 7ff17e8e9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4946030 0x5605a4950f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.896+0000 7ff17e8e9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1681ac8d0 con 0x5605a4941850 2024-05-27T07:52:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.896+0000 7ff17e8e9b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff1681add40 con 0x5605a4946030 2024-05-27T07:52:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.896+0000 7ff17e8e9b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1681acfa0 con 0x5605a47bdec0 2024-05-27T07:52:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.896+0000 7ff17e8e9b80 10 client.0 _ll_register_callbacks cb 0x5605a47f1630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.896+0000 7ff16dc70700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a4953650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.896+0000 7ff16d46f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4941850 0x5605a49509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.896+0000 7ff16dc70700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a4953650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33990/0 (socket says 172.21.15.114:33990) 2024-05-27T07:52:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff16dc70700 1 -- 172.21.15.114:0/3835652377 learned_addr learned my addr 172.21.15.114:0/3835652377 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff16e471700 1 --2- 172.21.15.114:0/3835652377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4946030 0x5605a4950f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff16dc70700 1 -- 172.21.15.114:0/3835652377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4946030 msgr2=0x5605a4950f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff16dc70700 1 --2- 172.21.15.114:0/3835652377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4946030 0x5605a4950f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff16dc70700 1 -- 172.21.15.114:0/3835652377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4941850 msgr2=0x5605a49509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:01.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff16dc70700 1 --2- 172.21.15.114:0/3835652377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4941850 0x5605a49509d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff16dc70700 1 -- 172.21.15.114:0/3835652377 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605a4951450 con 0x5605a47bdec0 2024-05-27T07:52:01.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff16dc70700 1 --2- 172.21.15.114:0/3835652377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a4953650 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff164015500 tx=0x7ff164005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=757d893633824e04 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:01.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff1517fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:52:01.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff1517fa700 1 -- 172.21.15.114:0/3835652377 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff164045070 con 0x5605a47bdec0 2024-05-27T07:52:01.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff1517fa700 1 -- 172.21.15.114:0/3835652377 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9466+0+0 (secure 0 0 0) 0x7ff164008d90 con 0x5605a47bdec0 2024-05-27T07:52:01.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/3835652377 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff164028040 con 0x5605a47bdec0 2024-05-27T07:52:01.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff1517fa700 1 -- 172.21.15.114:0/3835652377 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7ff164031040 con 0x5605a47bdec0 2024-05-27T07:52:01.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff17e8e9b80 10 client.27886 fetch_fsmap learned FSMap version 67 2024-05-27T07:52:01.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.897+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/3835652377 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5605a49c5740 con 0x5605a47bdec0 2024-05-27T07:52:01.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.898+0000 7ff1517fa700 1 -- 172.21.15.114:0/3835652377 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff164028040 con 0x5605a47bdec0 2024-05-27T07:52:01.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.898+0000 7ff1517fa700 1 -- 172.21.15.114:0/3835652377 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7ff164015a80 con 0x5605a47bdec0 2024-05-27T07:52:01.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.898+0000 7ff17e8e9b80 10 client.27886 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:52:01.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.898+0000 7ff17e8e9b80 10 client.27886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:01.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.898+0000 7ff17e8e9b80 20 client.27886 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:01.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.898+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/3835652377 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5605a49c5bf0 0x5605a49c8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:01.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.898+0000 7ff17e8e9b80 4 client.27886 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:52:01.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.898+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/3835652377 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5605a4866520 con 0x5605a49c5bf0 2024-05-27T07:52:01.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.898+0000 7ff16d46f700 1 --2- 172.21.15.114:0/3835652377 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5605a49c5bf0 0x5605a49c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:01.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.898+0000 7ff1517fa700 10 client.27886 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:01.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.899+0000 7ff16d46f700 1 --2- 172.21.15.114:0/3835652377 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5605a49c5bf0 0x5605a49c8010 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cb26531755bf0ac server_cookie=5518d26cccdbf865 in_seq=0 out_seq=0 2024-05-27T07:52:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff1517fa700 1 -- 172.21.15.114:0/3835652377 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5605a4866520 con 0x5605a49c5bf0 2024-05-27T07:52:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff1517fa700 10 client.27886 handle_command_reply: tid=0 2024-05-27T07:52:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff17e8e9b80 1 client.27886 shutdown 2024-05-27T07:52:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/3835652377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 msgr2=0x5605a4953650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/3835652377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a4953650 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff164015500 tx=0x7ff164005f10 comp rx=0 tx=0).stop 2024-05-27T07:52:01.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/3835652377 shutdown_connections 2024-05-27T07:52:01.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/3835652377 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5605a49c5bf0 0x5605a49c8010 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/3835652377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5605a4946030 0x5605a4950f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/3835652377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5605a47bdec0 0x5605a4953650 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff17e8e9b80 1 --2- 172.21.15.114:0/3835652377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605a4941850 0x5605a49509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff1517fa700 0 client.27886 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:01.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/3835652377 >> 172.21.15.114:0/3835652377 conn(0x5605a47c07f0 msgr2=0x5605a4931570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:01.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/3835652377 shutdown_connections 2024-05-27T07:52:01.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff17e8e9b80 1 -- 172.21.15.114:0/3835652377 wait complete. 2024-05-27T07:52:01.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.900+0000 7ff17e8e9b80 20 client.27886 trim_cache size 0 max 16384 2024-05-27T07:52:01.905 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:52:01.906 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:52:01.906 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "9a0c5171-9087-4a12-a63d-f758879762c1", 2024-05-27T07:52:01.906 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:52:01.906 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:52:01.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.901+0000 7ff16fed4700 1 -- 172.21.15.114:0/145513712 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff1540b29f0 msgr2=0x7ff1540b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:01.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.901+0000 7ff16fed4700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff1540b29f0 0x7ff1540b4f00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff16400f540 tx=0x7ff16400b040 comp rx=0 tx=0).stop 2024-05-27T07:52:01.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.901+0000 7ff16fed4700 1 -- 172.21.15.114:0/145513712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff168107de0 msgr2=0x7ff16819b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:01.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.901+0000 7ff16fed4700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff168107de0 0x7ff16819b420 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff158005f50 tx=0x7ff158004270 comp rx=0 tx=0).stop 2024-05-27T07:52:01.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.901+0000 7ff16fed4700 1 -- 172.21.15.114:0/145513712 shutdown_connections 2024-05-27T07:52:01.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.901+0000 7ff16fed4700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff168108730 0x7ff16819f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.901+0000 7ff16fed4700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff1540b29f0 0x7ff1540b4f00 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.901+0000 7ff16fed4700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1681034b0 0x7ff16819ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.901+0000 7ff16fed4700 1 --2- 172.21.15.114:0/145513712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff168107de0 0x7ff16819b420 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:01.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.901+0000 7ff16fed4700 1 -- 172.21.15.114:0/145513712 >> 172.21.15.114:0/145513712 conn(0x7ff1680fedb0 msgr2=0x7ff168109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:01.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.901+0000 7ff16fed4700 1 -- 172.21.15.114:0/145513712 shutdown_connections 2024-05-27T07:52:01.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:01.901+0000 7ff16fed4700 1 -- 172.21.15.114:0/145513712 wait complete. 2024-05-27T07:52:01.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:01 smithi163 ceph-mon[36451]: pgmap v407: 129 pgs: 129 active+clean; 69 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 279 op/s 2024-05-27T07:52:01.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:01 smithi186 ceph-mon[36647]: pgmap v407: 129 pgs: 129 active+clean; 69 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 279 op/s 2024-05-27T07:52:01.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:01 smithi186 podman[75247]: 2024-05-27 07:52:01.562999793 +0000 UTC m=+0.344209339 container died 4a3816c2982f0bad95eb3dedd7ff1867f43bcbb4cb5a86af113f69c6bc32289a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-27T07:52:02.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1194416 9.72 MB/sec execute 494 sec latency 128.255 ms 2024-05-27T07:52:02.907 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:02 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:52:02.907 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:02 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:52:02.907 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:02 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:52:02.908 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:02 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:52:02.908 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:02 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:52:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.908+0000 7ff3feffd640 1 -- 172.21.15.114:0/3310555482 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8030d70 con 0x7ff4001032e0 2024-05-27T07:52:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.908+0000 7ff40664c640 1 -- 172.21.15.114:0/3310555482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4001032e0 msgr2=0x7ff40010f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.908+0000 7ff40664c640 1 --2- 172.21.15.114:0/3310555482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4001032e0 0x7ff40010f6b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009be0 tx=0x7ff3f802a6d0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.908+0000 7ff40664c640 1 -- 172.21.15.114:0/3310555482 shutdown_connections 2024-05-27T07:52:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.908+0000 7ff40664c640 1 --2- 172.21.15.114:0/3310555482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4001032e0 0x7ff40010f6b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.908+0000 7ff40664c640 1 --2- 172.21.15.114:0/3310555482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff400109030 0x7ff400102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.908+0000 7ff40664c640 1 --2- 172.21.15.114:0/3310555482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff400108b90 0x7ff400102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.908+0000 7ff40664c640 1 -- 172.21.15.114:0/3310555482 >> 172.21.15.114:0/3310555482 conn(0x7ff4000782d0 msgr2=0x7ff400078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.908+0000 7ff40664c640 1 -- 172.21.15.114:0/3310555482 shutdown_connections 2024-05-27T07:52:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.908+0000 7ff40664c640 1 -- 172.21.15.114:0/3310555482 wait complete. 2024-05-27T07:52:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.909+0000 7ff40664c640 1 Processor -- start 2024-05-27T07:52:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.909+0000 7ff40664c640 1 -- start start 2024-05-27T07:52:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.910+0000 7ff40664c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4001032e0 0x7ff40010cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.910+0000 7ff40664c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff400108b90 0x7ff40010d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.910+0000 7ff40664c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff400109030 0x7ff40010d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.910+0000 7ff40664c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff400111cc0 con 0x7ff4001032e0 2024-05-27T07:52:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.910+0000 7ff40664c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff400111b40 con 0x7ff400109030 2024-05-27T07:52:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.910+0000 7ff40664c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff4001163c0 con 0x7ff400108b90 2024-05-27T07:52:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.910+0000 7ff404be9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff400109030 0x7ff40010d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.910+0000 7ff404be9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff400109030 0x7ff40010d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:44874/0 (socket says 172.21.15.114:44874) 2024-05-27T07:52:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.910+0000 7ff404be9640 1 -- 172.21.15.114:0/1255016713 learned_addr learned my addr 172.21.15.114:0/1255016713 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.911+0000 7ff404be9640 1 -- 172.21.15.114:0/1255016713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff400108b90 msgr2=0x7ff40010d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.911+0000 7ff404be9640 1 --2- 172.21.15.114:0/1255016713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff400108b90 0x7ff40010d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.911+0000 7ff404be9640 1 -- 172.21.15.114:0/1255016713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4001032e0 msgr2=0x7ff40010cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.911+0000 7ff404be9640 1 --2- 172.21.15.114:0/1255016713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4001032e0 0x7ff40010cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.911+0000 7ff404be9640 1 -- 172.21.15.114:0/1255016713 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4001acf90 con 0x7ff400109030 2024-05-27T07:52:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.911+0000 7ff404be9640 1 --2- 172.21.15.114:0/1255016713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff400109030 0x7ff40010d5a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009b30 tx=0x7ff3f8003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.911+0000 7ff3fd7fa640 1 -- 172.21.15.114:0/1255016713 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8040020 con 0x7ff400109030 2024-05-27T07:52:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.912+0000 7ff3fd7fa640 1 -- 172.21.15.114:0/1255016713 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3f8030e20 con 0x7ff400109030 2024-05-27T07:52:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.913+0000 7ff3fd7fa640 1 -- 172.21.15.114:0/1255016713 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8036040 con 0x7ff400109030 2024-05-27T07:52:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.913+0000 7ff40664c640 1 -- 172.21.15.114:0/1255016713 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4001ad220 con 0x7ff400109030 2024-05-27T07:52:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.913+0000 7ff40664c640 1 -- 172.21.15.114:0/1255016713 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4001ad6e0 con 0x7ff400109030 2024-05-27T07:52:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.918+0000 7ff40d5dd740 1 Processor -- start 2024-05-27T07:52:02.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.921+0000 7ff3fd7fa640 1 -- 172.21.15.114:0/1255016713 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ff3f8043020 con 0x7ff400109030 2024-05-27T07:52:02.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.922+0000 7ff3fd7fa640 1 --2- 172.21.15.114:0/1255016713 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff3ec0b28a0 0x7ff3ec0b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:02.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.922+0000 7ff3fd7fa640 1 -- 172.21.15.114:0/1255016713 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9642+0+0 (secure 0 0 0) 0x7ff3f8103310 con 0x7ff400109030 2024-05-27T07:52:02.923 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.923+0000 7ff40d5dd740 1 -- start start 2024-05-27T07:52:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.923+0000 7ff40d5dd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f016a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.923+0000 7ff40d5dd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda02f0 0x5570f016a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.924+0000 7ff40d5dd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 0x5570f016ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.924+0000 7ff40d5dd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5570f00dcc30 con 0x5570efda02f0 2024-05-27T07:52:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.924+0000 7ff3ff7fe640 1 --2- 172.21.15.114:0/1255016713 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff3ec0b28a0 0x7ff3ec0b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.924+0000 7ff40d5dd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5570f00cc7d0 con 0x5570efdc91b0 2024-05-27T07:52:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.924+0000 7ff3ff7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda02f0 0x5570f016a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.924+0000 7ff3ff7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda02f0 0x5570f016a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36274/0 (socket says 172.21.15.114:36274) 2024-05-27T07:52:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.924+0000 7ff3ff7fe640 1 -- 172.21.15.114:0/1199085342 learned_addr learned my addr 172.21.15.114:0/1199085342 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.924+0000 7ff40d5dd740 1 -- 172.21.15.114:0/1199085342 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5570f00d0810 con 0x5570efda84d0 2024-05-27T07:52:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.924+0000 7ff3ff7fe640 1 -- 172.21.15.114:0/1199085342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 msgr2=0x5570f016ae50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:52:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.924+0000 7ff3ff7fe640 1 --2- 172.21.15.114:0/1199085342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 0x5570f016ae50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.924+0000 7ff3ff7fe640 1 -- 172.21.15.114:0/1199085342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 msgr2=0x5570f016a3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:52:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.924+0000 7ff3ff7fe640 1 --2- 172.21.15.114:0/1199085342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f016a3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.924+0000 7ff3ff7fe640 1 -- 172.21.15.114:0/1199085342 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570efde1210 con 0x5570efda02f0 2024-05-27T07:52:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.925+0000 7ff3ff7fe640 1 --2- 172.21.15.114:0/1199085342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda02f0 0x5570f016a910 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8013480 tx=0x7ff3e8013940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45083f6facd9b223 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.925+0000 7ff3f67fc640 1 -- 172.21.15.114:0/1199085342 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8004220 con 0x5570efda02f0 2024-05-27T07:52:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.925+0000 7ff3f67fc640 1 -- 172.21.15.114:0/1199085342 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3e80043c0 con 0x5570efda02f0 2024-05-27T07:52:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.925+0000 7ff3f67fc640 1 -- 172.21.15.114:0/1199085342 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8011bc0 con 0x5570efda02f0 2024-05-27T07:52:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.926+0000 7ff40d5dd740 1 -- 172.21.15.114:0/1199085342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda02f0 msgr2=0x5570f016a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.926+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/1199085342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda02f0 0x5570f016a910 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8013480 tx=0x7ff3e8013940 comp rx=0 tx=0).stop 2024-05-27T07:52:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.929+0000 7ff3ff7fe640 1 --2- 172.21.15.114:0/1255016713 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff3ec0b28a0 0x7ff3ec0b4db0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x5570f016b780 tx=0x7ff3e8013940 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.935+0000 7ff40d5dd740 1 -- 172.21.15.114:0/1199085342 shutdown_connections 2024-05-27T07:52:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.935+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/1199085342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 0x5570f016ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.935+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/1199085342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda02f0 0x5570f016a910 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.935+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/1199085342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f016a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.935+0000 7ff40d5dd740 1 -- 172.21.15.114:0/1199085342 >> 172.21.15.114:0/1199085342 conn(0x5570efdbd300 msgr2=0x5570f0166b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:02.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.938+0000 7ff40d5dd740 1 -- 172.21.15.114:0/1199085342 shutdown_connections 2024-05-27T07:52:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.942+0000 7ff40d5dd740 1 -- 172.21.15.114:0/1199085342 wait complete. 2024-05-27T07:52:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.943+0000 7ff40d5dd740 1 -- start start 2024-05-27T07:52:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.944+0000 7ff40d5dd740 1 Processor -- start 2024-05-27T07:52:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.944+0000 7ff40d5dd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efda02f0 0x5570f0176480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.944+0000 7ff40d5dd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 0x5570f01769c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.944+0000 7ff40d5dd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efdc91b0 0x5570f0174990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.944+0000 7ff40d5dd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5570efdb2790 con 0x5570efdc91b0 2024-05-27T07:52:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.944+0000 7ff40d5dd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5570f0088890 con 0x5570efda02f0 2024-05-27T07:52:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.944+0000 7ff40d5dd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5570f0177500 con 0x5570efda84d0 2024-05-27T07:52:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.944+0000 7ff40d5dd740 10 client.0 _ll_register_callbacks cb 0x5570efde0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.944+0000 7ff3fffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efda02f0 0x5570f0176480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.945+0000 7ff3fffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efda02f0 0x5570f0176480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:44896/0 (socket says 172.21.15.114:44896) 2024-05-27T07:52:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.945+0000 7ff3fffff640 1 -- 172.21.15.114:0/2386339302 learned_addr learned my addr 172.21.15.114:0/2386339302 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.945+0000 7ff3fffff640 1 -- 172.21.15.114:0/2386339302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 msgr2=0x5570f01769c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:52:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.945+0000 7ff3fffff640 1 --2- 172.21.15.114:0/2386339302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 0x5570f01769c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.945+0000 7ff3fffff640 1 -- 172.21.15.114:0/2386339302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efdc91b0 msgr2=0x5570f0174990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:52:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.945+0000 7ff3fffff640 1 --2- 172.21.15.114:0/2386339302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efdc91b0 0x5570f0174990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.945+0000 7ff3fffff640 1 -- 172.21.15.114:0/2386339302 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570efead620 con 0x5570efda02f0 2024-05-27T07:52:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.952+0000 7ff3fffff640 1 --2- 172.21.15.114:0/2386339302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efda02f0 0x5570f0176480 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000bbf0 tx=0x7ff3f000d6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73a4208e6a1c138e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.952+0000 7ff3e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:52:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.954+0000 7ff3e77fe640 1 -- 172.21.15.114:0/2386339302 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0016070 con 0x5570efda02f0 2024-05-27T07:52:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.954+0000 7ff3e77fe640 1 -- 172.21.15.114:0/2386339302 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9642+0+0 (secure 0 0 0) 0x7ff3f0005220 con 0x5570efda02f0 2024-05-27T07:52:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.954+0000 7ff40d5dd740 1 -- 172.21.15.114:0/2386339302 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3e80043c0 con 0x5570efda02f0 2024-05-27T07:52:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.958+0000 7ff3e77fe640 1 -- 172.21.15.114:0/2386339302 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7ff3f001f050 con 0x5570efda02f0 2024-05-27T07:52:02.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:52:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.961+0000 7ff40d5dd740 10 client.27752 fetch_fsmap learned FSMap version 67 2024-05-27T07:52:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.961+0000 7ff40d5dd740 1 -- 172.21.15.114:0/2386339302 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5570f01948c0 con 0x5570efda02f0 2024-05-27T07:52:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.961+0000 7ff3e77fe640 1 -- 172.21.15.114:0/2386339302 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3e80043c0 con 0x5570efda02f0 2024-05-27T07:52:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.962+0000 7ff3e77fe640 1 -- 172.21.15.114:0/2386339302 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7ff3f0010460 con 0x5570efda02f0 2024-05-27T07:52:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.962+0000 7ff40d5dd740 10 client.27752 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:52:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.962+0000 7ff40d5dd740 10 client.27752 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.962+0000 7ff40d5dd740 20 client.27752 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.962+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/2386339302 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5570efd87d80 0x5570f0198140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.962+0000 7ff40d5dd740 4 client.27752 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:52:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.962+0000 7ff40d5dd740 1 -- 172.21.15.114:0/2386339302 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570f00deb80 con 0x5570efd87d80 2024-05-27T07:52:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.963+0000 7ff3ff7fe640 1 --2- 172.21.15.114:0/2386339302 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5570efd87d80 0x5570f0198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.969+0000 7ff3ff7fe640 1 --2- 172.21.15.114:0/2386339302 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5570efd87d80 0x5570f0198140 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ce9c7cc9cd217fb server_cookie=9d12ae4c7699f446 in_seq=0 out_seq=0 2024-05-27T07:52:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.970+0000 7ff3e77fe640 10 client.27752 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.974+0000 7ff3e77fe640 1 -- 172.21.15.114:0/2386339302 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5570f00deb80 con 0x5570efd87d80 2024-05-27T07:52:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.974+0000 7ff3e77fe640 10 client.27752 handle_command_reply: tid=0 2024-05-27T07:52:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.974+0000 7ff40d5dd740 1 client.27752 shutdown 2024-05-27T07:52:02.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:02 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:52:02.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:02 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:52:02.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:02 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:52:02.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:02 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:52:02.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:02 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:52:02.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:02 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:52:02.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:02 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:52:02.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:02 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:52:02.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:02 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:52:02.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:02 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:52:02.980 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.979+0000 7ff40d5dd740 1 -- 172.21.15.114:0/2386339302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efda02f0 msgr2=0x5570f0176480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.979+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/2386339302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efda02f0 0x5570f0176480 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000bbf0 tx=0x7ff3f000d6b0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.980+0000 7ff40d5dd740 1 -- 172.21.15.114:0/2386339302 shutdown_connections 2024-05-27T07:52:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.980+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/2386339302 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5570efd87d80 0x5570f0198140 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.980+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/2386339302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efdc91b0 0x5570f0174990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.980+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/2386339302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 0x5570f01769c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.980+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/2386339302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efda02f0 0x5570f0176480 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.980+0000 7ff40d5dd740 1 -- 172.21.15.114:0/2386339302 >> 172.21.15.114:0/2386339302 conn(0x5570efdbd300 msgr2=0x5570f016c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.980+0000 7ff3e77fe640 0 client.27752 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.982+0000 7ff40d5dd740 1 -- 172.21.15.114:0/2386339302 shutdown_connections 2024-05-27T07:52:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.986+0000 7ff40d5dd740 1 -- 172.21.15.114:0/2386339302 wait complete. 2024-05-27T07:52:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:02.986+0000 7ff40d5dd740 20 client.27752 trim_cache size 0 max 16384 2024-05-27T07:52:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.015+0000 7ff40d5dd740 1 Processor -- start 2024-05-27T07:52:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.015+0000 7ff40d5dd740 1 -- start start 2024-05-27T07:52:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.015+0000 7ff40d5dd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda02f0 0x5570f01767c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.015+0000 7ff40d5dd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 0x5570f016e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.016+0000 7ff40d5dd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f016e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.016+0000 7ff40d5dd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff4001b09c0 con 0x5570efda02f0 2024-05-27T07:52:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.016+0000 7ff3ff7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 0x5570f016e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.016+0000 7ff40d5dd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff4001af150 con 0x5570efdc91b0 2024-05-27T07:52:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.016+0000 7ff404be9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f016e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.016+0000 7ff404be9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f016e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:44910/0 (socket says 172.21.15.114:44910) 2024-05-27T07:52:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.016+0000 7ff404be9640 1 -- 172.21.15.114:0/1381197185 learned_addr learned my addr 172.21.15.114:0/1381197185 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.016+0000 7ff3ff7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 0x5570f016e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34014/0 (socket says 172.21.15.114:34014) 2024-05-27T07:52:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.016+0000 7ff40d5dd740 1 -- 172.21.15.114:0/1381197185 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff4000744b0 con 0x5570efda84d0 2024-05-27T07:52:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.016+0000 7ff3fffff640 1 --2- 172.21.15.114:0/1381197185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda02f0 0x5570f01767c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.017+0000 7ff404be9640 1 -- 172.21.15.114:0/1381197185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 msgr2=0x5570f016e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.017+0000 7ff404be9640 1 --2- 172.21.15.114:0/1381197185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 0x5570f016e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.017+0000 7ff404be9640 1 -- 172.21.15.114:0/1381197185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda02f0 msgr2=0x5570f01767c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.017+0000 7ff404be9640 1 --2- 172.21.15.114:0/1381197185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda02f0 0x5570f01767c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.017+0000 7ff404be9640 1 -- 172.21.15.114:0/1381197185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570efd87fb0 con 0x5570efdc91b0 2024-05-27T07:52:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.017+0000 7ff404be9640 1 --2- 172.21.15.114:0/1381197185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f016e5c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8100710 tx=0x7ff3f80c9b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b4eb3c54abe75aa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.018+0000 7ff3ff7fe640 1 --2- 172.21.15.114:0/1381197185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 0x5570f016e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:52:03.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.021+0000 7ff3f5ffb640 1 -- 172.21.15.114:0/1381197185 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f80e4070 con 0x5570efdc91b0 2024-05-27T07:52:03.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.021+0000 7ff3f5ffb640 1 -- 172.21.15.114:0/1381197185 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3f80ca940 con 0x5570efdc91b0 2024-05-27T07:52:03.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.021+0000 7ff3f5ffb640 1 -- 172.21.15.114:0/1381197185 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8100820 con 0x5570efdc91b0 2024-05-27T07:52:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.023+0000 7ff40d5dd740 1 -- 172.21.15.114:0/1381197185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 msgr2=0x5570f016e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.023+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/1381197185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f016e5c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8100710 tx=0x7ff3f80c9b90 comp rx=0 tx=0).stop 2024-05-27T07:52:03.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.024+0000 7ff40d5dd740 1 -- 172.21.15.114:0/1381197185 shutdown_connections 2024-05-27T07:52:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.024+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/1381197185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f016e5c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.024+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/1381197185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda84d0 0x5570f016e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.024+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/1381197185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda02f0 0x5570f01767c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.024+0000 7ff40d5dd740 1 -- 172.21.15.114:0/1381197185 >> 172.21.15.114:0/1381197185 conn(0x5570efdbd300 msgr2=0x5570f01760b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:03.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.029+0000 7ff40d5dd740 1 -- 172.21.15.114:0/1381197185 shutdown_connections 2024-05-27T07:52:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.032+0000 7ff40d5dd740 1 -- 172.21.15.114:0/1381197185 wait complete. 2024-05-27T07:52:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.032+0000 7ff40d5dd740 1 -- start start 2024-05-27T07:52:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.033+0000 7ff40d5dd740 1 Processor -- start 2024-05-27T07:52:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.033+0000 7ff40d5dd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda02f0 0x5570f018a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.033+0000 7ff40d5dd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda84d0 0x5570f018aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.033+0000 7ff40d5dd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f018af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.033+0000 7ff40d5dd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff400074b00 con 0x5570efda84d0 2024-05-27T07:52:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.033+0000 7ff40d5dd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff4001aea80 con 0x5570efdc91b0 2024-05-27T07:52:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.033+0000 7ff40d5dd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff4001adce0 con 0x5570efda02f0 2024-05-27T07:52:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.033+0000 7ff40d5dd740 10 client.0 _ll_register_callbacks cb 0x7ff3f000b260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.036+0000 7ff3fffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda02f0 0x5570f018a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.036+0000 7ff3ff7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda84d0 0x5570f018aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.036+0000 7ff3fffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda02f0 0x5570f018a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34016/0 (socket says 172.21.15.114:34016) 2024-05-27T07:52:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.036+0000 7ff3fffff640 1 -- 172.21.15.114:0/925017198 learned_addr learned my addr 172.21.15.114:0/925017198 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.036+0000 7ff404be9640 1 --2- 172.21.15.114:0/925017198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f018af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.038+0000 7ff404be9640 1 -- 172.21.15.114:0/925017198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda02f0 msgr2=0x5570f018a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.038+0000 7ff404be9640 1 --2- 172.21.15.114:0/925017198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda02f0 0x5570f018a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.038+0000 7ff404be9640 1 -- 172.21.15.114:0/925017198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda84d0 msgr2=0x5570f018aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.038+0000 7ff404be9640 1 --2- 172.21.15.114:0/925017198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda84d0 0x5570f018aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.038+0000 7ff404be9640 1 -- 172.21.15.114:0/925017198 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570f01fe7d0 con 0x5570efdc91b0 2024-05-27T07:52:03.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.039+0000 7ff404be9640 1 --2- 172.21.15.114:0/925017198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f018af70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x5570f00debd0 tx=0x7ff3f8009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42b297b72b43cae9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:03.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.040+0000 7ff3e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:52:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.041+0000 7ff3e77fe640 1 -- 172.21.15.114:0/925017198 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f80e8070 con 0x5570efdc91b0 2024-05-27T07:52:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.041+0000 7ff40d5dd740 1 -- 172.21.15.114:0/925017198 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3f80ca940 con 0x5570efdc91b0 2024-05-27T07:52:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.041+0000 7ff3e77fe640 1 -- 172.21.15.114:0/925017198 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9642+0+0 (secure 0 0 0) 0x7ff3f80ce380 con 0x5570efdc91b0 2024-05-27T07:52:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.042+0000 7ff3e77fe640 1 -- 172.21.15.114:0/925017198 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7ff3f80e4040 con 0x5570efdc91b0 2024-05-27T07:52:03.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.043+0000 7ff40d5dd740 10 client.27758 fetch_fsmap learned FSMap version 67 2024-05-27T07:52:03.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.043+0000 7ff40d5dd740 1 -- 172.21.15.114:0/925017198 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5570f01ff890 con 0x5570efdc91b0 2024-05-27T07:52:03.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.044+0000 7ff3e77fe640 1 -- 172.21.15.114:0/925017198 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3f80ca940 con 0x5570efdc91b0 2024-05-27T07:52:03.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.044+0000 7ff3e77fe640 1 -- 172.21.15.114:0/925017198 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7ff3f80eb020 con 0x5570efdc91b0 2024-05-27T07:52:03.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.044+0000 7ff40d5dd740 10 client.27758 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:52:03.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.044+0000 7ff40d5dd740 10 client.27758 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:03.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.044+0000 7ff40d5dd740 20 client.27758 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:03.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.044+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/925017198 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5570efd5d1c0 0x5570f0201f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:03.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.044+0000 7ff40d5dd740 4 client.27758 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:52:03.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.044+0000 7ff40d5dd740 1 -- 172.21.15.114:0/925017198 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5570f00d8360 con 0x5570efd5d1c0 2024-05-27T07:52:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.046+0000 7ff3fffff640 1 --2- 172.21.15.114:0/925017198 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5570efd5d1c0 0x5570f0201f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.048+0000 7ff3fffff640 1 --2- 172.21.15.114:0/925017198 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5570efd5d1c0 0x5570f0201f40 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4201f16c1352bba server_cookie=83d2e9590862517d in_seq=0 out_seq=0 2024-05-27T07:52:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:03.049+0000 7ff3e77fe640 10 client.27758 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:03.289 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1195021 9.71 MB/sec execute 495 sec latency 187.075 ms 2024-05-27T07:52:03.510 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:52:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:03 smithi114 ceph-mon[32038]: pgmap v408: 129 pgs: 129 active+clean; 75 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 381 op/s 2024-05-27T07:52:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:03 smithi114 ceph-mon[32038]: osdmap e125: 12 total, 12 up, 12 in 2024-05-27T07:52:03.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:03 smithi163 ceph-mon[36451]: pgmap v408: 129 pgs: 129 active+clean; 75 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 381 op/s 2024-05-27T07:52:03.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:03 smithi163 ceph-mon[36451]: osdmap e125: 12 total, 12 up, 12 in 2024-05-27T07:52:03.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:03 smithi186 ceph-mon[36647]: pgmap v408: 129 pgs: 129 active+clean; 75 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 381 op/s 2024-05-27T07:52:03.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:03 smithi186 ceph-mon[36647]: osdmap e125: 12 total, 12 up, 12 in 2024-05-27T07:52:03.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:03 smithi186 podman[75247]: 2024-05-27 07:52:03.656269865 +0000 UTC m=+2.437479413 container remove 4a3816c2982f0bad95eb3dedd7ff1867f43bcbb4cb5a86af113f69c6bc32289a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-27T07:52:03.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:03 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:52:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff3e77fe640 1 -- 172.21.15.114:0/925017198 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5570f00d8360 con 0x5570efd5d1c0 2024-05-27T07:52:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff3e77fe640 10 client.27758 handle_command_reply: tid=0 2024-05-27T07:52:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff40d5dd740 1 client.27758 shutdown 2024-05-27T07:52:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff40d5dd740 1 -- 172.21.15.114:0/925017198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 msgr2=0x5570f018af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/925017198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f018af70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x5570f00debd0 tx=0x7ff3f8009500 comp rx=0 tx=0).stop 2024-05-27T07:52:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff40d5dd740 1 -- 172.21.15.114:0/925017198 shutdown_connections 2024-05-27T07:52:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/925017198 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5570efd5d1c0 0x5570f0201f40 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/925017198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570efdc91b0 0x5570f018af70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/925017198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570efda84d0 0x5570f018aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff40d5dd740 1 --2- 172.21.15.114:0/925017198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570efda02f0 0x5570f018a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff40d5dd740 1 -- 172.21.15.114:0/925017198 >> 172.21.15.114:0/925017198 conn(0x5570efdbd300 msgr2=0x5570f016a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff3e77fe640 0 client.27758 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff40d5dd740 1 -- 172.21.15.114:0/925017198 shutdown_connections 2024-05-27T07:52:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff40d5dd740 1 -- 172.21.15.114:0/925017198 wait complete. 2024-05-27T07:52:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.016+0000 7ff40d5dd740 20 client.27758 trim_cache size 0 max 16384 2024-05-27T07:52:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:52:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:52:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:52:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.017+0000 7ff40664c640 1 -- 172.21.15.114:0/1255016713 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff3ec0b28a0 msgr2=0x7ff3ec0b4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.017+0000 7ff40664c640 1 --2- 172.21.15.114:0/1255016713 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff3ec0b28a0 0x7ff3ec0b4db0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x5570f016b780 tx=0x7ff3e8013940 comp rx=0 tx=0).stop 2024-05-27T07:52:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.017+0000 7ff40664c640 1 -- 172.21.15.114:0/1255016713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff400109030 msgr2=0x7ff40010d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.017+0000 7ff40664c640 1 --2- 172.21.15.114:0/1255016713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff400109030 0x7ff40010d5a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009b30 tx=0x7ff3f8003f10 comp rx=0 tx=0).stop 2024-05-27T07:52:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.017+0000 7ff40664c640 1 -- 172.21.15.114:0/1255016713 shutdown_connections 2024-05-27T07:52:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.017+0000 7ff40664c640 1 --2- 172.21.15.114:0/1255016713 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff3ec0b28a0 0x7ff3ec0b4db0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.017+0000 7ff40664c640 1 --2- 172.21.15.114:0/1255016713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff400109030 0x7ff40010d5a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.017+0000 7ff40664c640 1 --2- 172.21.15.114:0/1255016713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff400108b90 0x7ff40010d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.017+0000 7ff40664c640 1 --2- 172.21.15.114:0/1255016713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4001032e0 0x7ff40010cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.018+0000 7ff40664c640 1 -- 172.21.15.114:0/1255016713 >> 172.21.15.114:0/1255016713 conn(0x7ff4000782d0 msgr2=0x7ff4000ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.018+0000 7ff40664c640 1 -- 172.21.15.114:0/1255016713 shutdown_connections 2024-05-27T07:52:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:04.018+0000 7ff40664c640 1 -- 172.21.15.114:0/1255016713 wait complete. 2024-05-27T07:52:04.290 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1195207 9.69 MB/sec execute 496 sec latency 206.386 ms 2024-05-27T07:52:04.729 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:04 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:52:04.729 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:04 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.966s CPU time. 2024-05-27T07:52:05.292 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1195759 9.67 MB/sec execute 497 sec latency 138.532 ms 2024-05-27T07:52:05.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:05 smithi163 ceph-mon[36451]: pgmap v410: 129 pgs: 129 active+clean; 75 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-05-27T07:52:05.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:05 smithi186 ceph-mon[36647]: pgmap v410: 129 pgs: 129 active+clean; 75 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-05-27T07:52:05.999 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:05 smithi114 ceph-mon[32038]: pgmap v410: 129 pgs: 129 active+clean; 75 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-05-27T07:52:06.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.199+0000 7f3c71488700 1 -- 172.21.15.114:0/1999885583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6c115380 msgr2=0x7f3c6c1177a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:06.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.199+0000 7f3c71488700 1 --2- 172.21.15.114:0/1999885583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6c115380 0x7f3c6c1177a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3c640098f0 tx=0x7f3c64009c00 comp rx=0 tx=0).stop 2024-05-27T07:52:06.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.202+0000 7f3c71488700 1 -- 172.21.15.114:0/1999885583 shutdown_connections 2024-05-27T07:52:06.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.202+0000 7f3c71488700 1 --2- 172.21.15.114:0/1999885583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6c115380 0x7f3c6c1177a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.202+0000 7f3c71488700 1 --2- 172.21.15.114:0/1999885583 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c6c075b50 0x7f3c6c075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.202+0000 7f3c71488700 1 --2- 172.21.15.114:0/1999885583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c6c0764a0 0x7f3c6c114e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.202+0000 7f3c71488700 1 -- 172.21.15.114:0/1999885583 >> 172.21.15.114:0/1999885583 conn(0x7f3c6c07af60 msgr2=0x7f3c6c07b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.205+0000 7f3c71488700 1 -- 172.21.15.114:0/1999885583 shutdown_connections 2024-05-27T07:52:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.208+0000 7f3c71488700 1 -- 172.21.15.114:0/1999885583 wait complete. 2024-05-27T07:52:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.209+0000 7f3c71488700 1 Processor -- start 2024-05-27T07:52:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.212+0000 7f3c71488700 1 -- start start 2024-05-27T07:52:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.212+0000 7f3c71488700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6c075b50 0x7f3c6c114cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.213+0000 7f3c71488700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c6c0764a0 0x7f3c6c110e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.213+0000 7f3c71488700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c6c115380 0x7f3c6c111340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.213+0000 7f3c71488700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c10ae80 con 0x7f3c6c115380 2024-05-27T07:52:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.213+0000 7f3c71488700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c10ad00 con 0x7f3c6c0764a0 2024-05-27T07:52:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.213+0000 7f3c71488700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c10b000 con 0x7f3c6c075b50 2024-05-27T07:52:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.214+0000 7f3c627fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c6c0764a0 0x7f3c6c110e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.214+0000 7f3c6b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c6c115380 0x7f3c6c111340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.214+0000 7f3c6b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c6c115380 0x7f3c6c111340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53042/0 (socket says 172.21.15.114:53042) 2024-05-27T07:52:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.214+0000 7f3c6b7fe700 1 -- 172.21.15.114:0/2161828007 learned_addr learned my addr 172.21.15.114:0/2161828007 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.214+0000 7f3c6affd700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6c075b50 0x7f3c6c114cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.214+0000 7f3c6b7fe700 1 -- 172.21.15.114:0/2161828007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6c075b50 msgr2=0x7f3c6c114cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.214+0000 7f3c6b7fe700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6c075b50 0x7f3c6c114cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.214+0000 7f3c6b7fe700 1 -- 172.21.15.114:0/2161828007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c6c0764a0 msgr2=0x7f3c6c110e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.214+0000 7f3c6b7fe700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c6c0764a0 0x7f3c6c110e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.214+0000 7f3c6b7fe700 1 -- 172.21.15.114:0/2161828007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6c111b50 con 0x7f3c6c115380 2024-05-27T07:52:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.215+0000 7f3c6b7fe700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c6c115380 0x7f3c6c111340 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f3c6400b2e0 tx=0x7f3c6400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.216+0000 7f3c6affd700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6c075b50 0x7f3c6c114cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:52:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.218+0000 7f3c68ff9700 1 -- 172.21.15.114:0/2161828007 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c64024020 con 0x7f3c6c115380 2024-05-27T07:52:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.218+0000 7f3c68ff9700 1 -- 172.21.15.114:0/2161828007 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c6400bcb0 con 0x7f3c6c115380 2024-05-27T07:52:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.218+0000 7f3c68ff9700 1 -- 172.21.15.114:0/2161828007 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c64020650 con 0x7f3c6c115380 2024-05-27T07:52:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.218+0000 7f3c71488700 1 -- 172.21.15.114:0/2161828007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c6c1c7110 con 0x7f3c6c115380 2024-05-27T07:52:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.219+0000 7f3c71488700 1 -- 172.21.15.114:0/2161828007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c6c1c7570 con 0x7f3c6c115380 2024-05-27T07:52:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.226+0000 7f3c68ff9700 1 -- 172.21.15.114:0/2161828007 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f3c64017400 con 0x7f3c6c115380 2024-05-27T07:52:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.227+0000 7f3c68ff9700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3c580b2c00 0x7f3c580b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.227+0000 7f3c68ff9700 1 -- 172.21.15.114:0/2161828007 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9642+0+0 (secure 0 0 0) 0x7f3c640e3b80 con 0x7f3c6c115380 2024-05-27T07:52:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.227+0000 7f3c6affd700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3c580b2c00 0x7f3c580b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.228+0000 7f3c6affd700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3c580b2c00 0x7f3c580b5110 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3c54006fd0 tx=0x7f3c54008040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.229+0000 7f3c7fe9db80 1 Processor -- start 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.229+0000 7f3c7fe9db80 1 -- start start 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.229+0000 7f3c7fe9db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1ac2e40 0x561cd1c46710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.229+0000 7f3c7fe9db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1c46c50 0x561cd1c40a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.229+0000 7f3c7fe9db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 0x561cd1c41390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.229+0000 7f3c7fe9db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561cd1b6fc90 con 0x561cd1ac2e40 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.229+0000 7f3c7fe9db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561cd1b5f830 con 0x561cd1c40f80 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.229+0000 7f3c7fe9db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561cd1b63870 con 0x561cd1c46c50 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.230+0000 7f3c6affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1ac2e40 0x561cd1c46710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.230+0000 7f3c6affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1ac2e40 0x561cd1c46710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53044/0 (socket says 172.21.15.114:53044) 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.230+0000 7f3c6affd700 1 -- 172.21.15.114:0/3656670853 learned_addr learned my addr 172.21.15.114:0/3656670853 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.230+0000 7f3c6affd700 1 -- 172.21.15.114:0/3656670853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1c46c50 msgr2=0x561cd1c40a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.230+0000 7f3c6affd700 1 --2- 172.21.15.114:0/3656670853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1c46c50 0x561cd1c40a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.230+0000 7f3c6affd700 1 -- 172.21.15.114:0/3656670853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 msgr2=0x561cd1c41390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:52:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.230+0000 7f3c6affd700 1 --2- 172.21.15.114:0/3656670853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 0x561cd1c41390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.230+0000 7f3c6affd700 1 -- 172.21.15.114:0/3656670853 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cd1aaf8d0 con 0x561cd1ac2e40 2024-05-27T07:52:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.230+0000 7f3c6affd700 1 --2- 172.21.15.114:0/3656670853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1ac2e40 0x561cd1c46710 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400c910 tx=0x7f3c5400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11addd88949f36be server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.232+0000 7f3c617fa700 1 -- 172.21.15.114:0/3656670853 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c54019040 con 0x561cd1ac2e40 2024-05-27T07:52:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.232+0000 7f3c617fa700 1 -- 172.21.15.114:0/3656670853 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c540044d0 con 0x561cd1ac2e40 2024-05-27T07:52:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.232+0000 7f3c617fa700 1 -- 172.21.15.114:0/3656670853 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c540137e0 con 0x561cd1ac2e40 2024-05-27T07:52:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.233+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/3656670853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1ac2e40 msgr2=0x561cd1c46710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.233+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/3656670853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1ac2e40 0x561cd1c46710 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400c910 tx=0x7f3c5400f000 comp rx=0 tx=0).stop 2024-05-27T07:52:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.240+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/3656670853 shutdown_connections 2024-05-27T07:52:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.240+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/3656670853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 0x561cd1c41390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.240+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/3656670853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1ac2e40 0x561cd1c46710 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.240+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/3656670853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1c46c50 0x561cd1c40a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.240+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/3656670853 >> 172.21.15.114:0/3656670853 conn(0x561cd1b1daf0 msgr2=0x561cd1c3b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.241+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/3656670853 shutdown_connections 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.246+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/3656670853 wait complete. 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.246+0000 7f3c7fe9db80 1 -- start start 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.247+0000 7f3c7fe9db80 1 Processor -- start 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.247+0000 7f3c7fe9db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1ac2e40 0x561cd1c4a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.247+0000 7f3c7fe9db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1c40f80 0x561cd1c4a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.247+0000 7f3c7fe9db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1c46c50 0x561cd1c4ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.248+0000 7f3c7fe9db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561cd1a65fa0 con 0x561cd1c46c50 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.248+0000 7f3c6b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1c46c50 0x561cd1c4ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.248+0000 7f3c7fe9db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561cd1b017c0 con 0x561cd1ac2e40 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.248+0000 7f3c7fe9db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561cd1c4d740 con 0x561cd1c40f80 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.248+0000 7f3c7fe9db80 10 client.0 _ll_register_callbacks cb 0x561cd1afa920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.249+0000 7f3c6b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1c46c50 0x561cd1c4ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53060/0 (socket says 172.21.15.114:53060) 2024-05-27T07:52:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.249+0000 7f3c6b7fe700 1 -- 172.21.15.114:0/3668271767 learned_addr learned my addr 172.21.15.114:0/3668271767 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.250+0000 7f3c6affd700 1 --2- 172.21.15.114:0/3668271767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1ac2e40 0x561cd1c4a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.250+0000 7f3c6b7fe700 1 -- 172.21.15.114:0/3668271767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1c40f80 msgr2=0x561cd1c4a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:52:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.250+0000 7f3c6b7fe700 1 --2- 172.21.15.114:0/3668271767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1c40f80 0x561cd1c4a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.250+0000 7f3c6b7fe700 1 -- 172.21.15.114:0/3668271767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1ac2e40 msgr2=0x561cd1c4a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.250+0000 7f3c6b7fe700 1 --2- 172.21.15.114:0/3668271767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1ac2e40 0x561cd1c4a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.250+0000 7f3c6b7fe700 1 -- 172.21.15.114:0/3668271767 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cd1afe690 con 0x561cd1c46c50 2024-05-27T07:52:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.250+0000 7f3c6b7fe700 1 --2- 172.21.15.114:0/3668271767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1c46c50 0x561cd1c4ae20 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f3c640a7d30 tx=0x7f3c640a6a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114fb2ea49980ab2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.250+0000 7f3c527fc700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:52:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.256+0000 7f3c527fc700 1 -- 172.21.15.114:0/3668271767 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c640aa5d0 con 0x561cd1c46c50 2024-05-27T07:52:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.256+0000 7f3c527fc700 1 -- 172.21.15.114:0/3668271767 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9642+0+0 (secure 0 0 0) 0x7f3c640aad20 con 0x561cd1c46c50 2024-05-27T07:52:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.257+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/3668271767 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c540044d0 con 0x561cd1c46c50 2024-05-27T07:52:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.260+0000 7f3c527fc700 1 -- 172.21.15.114:0/3668271767 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f3c640ab1c0 con 0x561cd1c46c50 2024-05-27T07:52:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.263+0000 7f3c7fe9db80 10 client.18351 fetch_fsmap learned FSMap version 67 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.263+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/3668271767 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x561cd1c58020 con 0x561cd1c46c50 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.266+0000 7f3c527fc700 1 -- 172.21.15.114:0/3668271767 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c640aaac0 con 0x561cd1c46c50 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.266+0000 7f3c527fc700 1 -- 172.21.15.114:0/3668271767 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f3c640cbce0 con 0x561cd1c46c50 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.266+0000 7f3c7fe9db80 10 client.18351 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.266+0000 7f3c7fe9db80 10 client.18351 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.266+0000 7f3c7fe9db80 20 client.18351 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.266+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/3668271767 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cd1c5a540 0x561cd1c5c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.266+0000 7f3c7fe9db80 4 client.18351 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.266+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/3668271767 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3c540044d0 con 0x561cd1c5a540 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.269+0000 7f3c6affd700 1 --2- 172.21.15.114:0/3668271767 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cd1c5a540 0x561cd1c5c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.274+0000 7f3c6affd700 1 --2- 172.21.15.114:0/3668271767 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cd1c5a540 0x561cd1c5c960 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=178399b923b4b5e5 server_cookie=574ccdf8899b137a in_seq=0 out_seq=0 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.276+0000 7f3c527fc700 10 client.18351 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.278+0000 7f3c527fc700 1 -- 172.21.15.114:0/3668271767 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f3c540044d0 con 0x561cd1c5a540 2024-05-27T07:52:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.278+0000 7f3c527fc700 10 client.18351 handle_command_reply: tid=0 2024-05-27T07:52:06.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.278+0000 7f3c7fe9db80 1 client.18351 shutdown 2024-05-27T07:52:06.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.281+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/3668271767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1c46c50 msgr2=0x561cd1c4ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:06.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.281+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/3668271767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1c46c50 0x561cd1c4ae20 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f3c640a7d30 tx=0x7f3c640a6a90 comp rx=0 tx=0).stop 2024-05-27T07:52:06.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.281+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/3668271767 shutdown_connections 2024-05-27T07:52:06.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.281+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/3668271767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1c46c50 0x561cd1c4ae20 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.281+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/3668271767 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cd1c5a540 0x561cd1c5c960 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.281+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/3668271767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1ac2e40 0x561cd1c4a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.281+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/3668271767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1c40f80 0x561cd1c4a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.281+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/3668271767 >> 172.21.15.114:0/3668271767 conn(0x561cd1b1daf0 msgr2=0x561cd1c3fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:06.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.281+0000 7f3c527fc700 0 client.18351 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:06.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.283+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/3668271767 shutdown_connections 2024-05-27T07:52:06.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.283+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/3668271767 wait complete. 2024-05-27T07:52:06.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.286+0000 7f3c7fe9db80 20 client.18351 trim_cache size 0 max 16384 2024-05-27T07:52:06.303 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1195849 9.66 MB/sec execute 498 sec latency 177.561 ms 2024-05-27T07:52:06.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.327+0000 7f3c7fe9db80 1 Processor -- start 2024-05-27T07:52:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.328+0000 7f3c7fe9db80 1 -- start start 2024-05-27T07:52:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.328+0000 7f3c7fe9db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1ac2e40 0x561cd1c40a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.328+0000 7f3c7fe9db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 0x561cd1c496c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.328+0000 7f3c7fe9db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1c46c50 0x561cd1c449f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.329+0000 7f3c6affd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1ac2e40 0x561cd1c40a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.329+0000 7f3c6affd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1ac2e40 0x561cd1c40a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:45946/0 (socket says 172.21.15.114:45946) 2024-05-27T07:52:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.329+0000 7f3c6affd700 1 -- 172.21.15.114:0/769091804 learned_addr learned my addr 172.21.15.114:0/769091804 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.329+0000 7f3c627fc700 1 --2- 172.21.15.114:0/769091804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 0x561cd1c496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.329+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/769091804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1d2110 con 0x561cd1c46c50 2024-05-27T07:52:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.329+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/769091804 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1d0880 con 0x561cd1c40f80 2024-05-27T07:52:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.329+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/769091804 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1ce6f0 con 0x561cd1ac2e40 2024-05-27T07:52:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.329+0000 7f3c6b7fe700 1 --2- 172.21.15.114:0/769091804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1c46c50 0x561cd1c449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.329+0000 7f3c6b7fe700 1 -- 172.21.15.114:0/769091804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1ac2e40 msgr2=0x561cd1c40a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.329+0000 7f3c6b7fe700 1 --2- 172.21.15.114:0/769091804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1ac2e40 0x561cd1c40a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.329+0000 7f3c6b7fe700 1 -- 172.21.15.114:0/769091804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 msgr2=0x561cd1c496c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.329+0000 7f3c6b7fe700 1 --2- 172.21.15.114:0/769091804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 0x561cd1c496c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.329+0000 7f3c6b7fe700 1 -- 172.21.15.114:0/769091804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cd1c48910 con 0x561cd1c46c50 2024-05-27T07:52:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.331+0000 7f3c6b7fe700 1 --2- 172.21.15.114:0/769091804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1c46c50 0x561cd1c449f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f3c64019040 tx=0x7f3c6400bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44554e4de7bb9797 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:06.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.333+0000 7f3c60ff9700 1 -- 172.21.15.114:0/769091804 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c640cd360 con 0x561cd1c46c50 2024-05-27T07:52:06.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.333+0000 7f3c60ff9700 1 -- 172.21.15.114:0/769091804 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c640c50f0 con 0x561cd1c46c50 2024-05-27T07:52:06.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.333+0000 7f3c60ff9700 1 -- 172.21.15.114:0/769091804 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c640aaf80 con 0x561cd1c46c50 2024-05-27T07:52:06.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.336+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/769091804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1c46c50 msgr2=0x561cd1c449f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:06.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.336+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/769091804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1c46c50 0x561cd1c449f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f3c64019040 tx=0x7f3c6400bee0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.339+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/769091804 shutdown_connections 2024-05-27T07:52:06.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.339+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/769091804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1c46c50 0x561cd1c449f0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.339+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/769091804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1ac2e40 0x561cd1c40a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.339+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/769091804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 0x561cd1c496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.339+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/769091804 >> 172.21.15.114:0/769091804 conn(0x561cd1b1daf0 msgr2=0x561cd1c4a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:06.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.343+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/769091804 shutdown_connections 2024-05-27T07:52:06.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.343+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/769091804 wait complete. 2024-05-27T07:52:06.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.344+0000 7f3c7fe9db80 1 -- start start 2024-05-27T07:52:06.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.344+0000 7f3c7fe9db80 1 Processor -- start 2024-05-27T07:52:06.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.345+0000 7f3c7fe9db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1ac2e40 0x561cd1c52c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.345+0000 7f3c7fe9db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 0x561cd1c531a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.345+0000 7f3c7fe9db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1c46c50 0x561cd1c4fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.345+0000 7f3c7fe9db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1ced40 con 0x561cd1ac2e40 2024-05-27T07:52:06.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.345+0000 7f3c7fe9db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1d01b0 con 0x561cd1c40f80 2024-05-27T07:52:06.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.346+0000 7f3c7fe9db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1cf410 con 0x561cd1c46c50 2024-05-27T07:52:06.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.346+0000 7f3c7fe9db80 10 client.0 _ll_register_callbacks cb 0x561cd1afa920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:06.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.347+0000 7f3c627fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 0x561cd1c531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.347+0000 7f3c627fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 0x561cd1c531a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:58070/0 (socket says 172.21.15.114:58070) 2024-05-27T07:52:06.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.347+0000 7f3c627fc700 1 -- 172.21.15.114:0/149433373 learned_addr learned my addr 172.21.15.114:0/149433373 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:06.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.347+0000 7f3c6b7fe700 1 --2- 172.21.15.114:0/149433373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1c46c50 0x561cd1c4fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.347+0000 7f3c627fc700 1 -- 172.21.15.114:0/149433373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1c46c50 msgr2=0x561cd1c4fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:06.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.347+0000 7f3c627fc700 1 --2- 172.21.15.114:0/149433373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1c46c50 0x561cd1c4fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.347+0000 7f3c627fc700 1 -- 172.21.15.114:0/149433373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1ac2e40 msgr2=0x561cd1c52c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:06.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.347+0000 7f3c627fc700 1 --2- 172.21.15.114:0/149433373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1ac2e40 0x561cd1c52c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.347+0000 7f3c627fc700 1 -- 172.21.15.114:0/149433373 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cd1c501e0 con 0x561cd1c40f80 2024-05-27T07:52:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.350+0000 7f3c627fc700 1 --2- 172.21.15.114:0/149433373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 0x561cd1c531a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c018850 tx=0x7f3c5c018c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70c7705f2828d463 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:06.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.352+0000 7f3c527fc700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:52:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.353+0000 7f3c527fc700 1 -- 172.21.15.114:0/149433373 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c0090d0 con 0x561cd1c40f80 2024-05-27T07:52:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.353+0000 7f3c527fc700 1 -- 172.21.15.114:0/149433373 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9642+0+0 (secure 0 0 0) 0x7f3c5c009270 con 0x561cd1c40f80 2024-05-27T07:52:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.353+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/149433373 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c640c50f0 con 0x561cd1c40f80 2024-05-27T07:52:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.359+0000 7f3c527fc700 1 -- 172.21.15.114:0/149433373 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f3c5c009930 con 0x561cd1c40f80 2024-05-27T07:52:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.360+0000 7f3c7fe9db80 10 client.27764 fetch_fsmap learned FSMap version 67 2024-05-27T07:52:06.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.360+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/149433373 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x561cd1cc5af0 con 0x561cd1c40f80 2024-05-27T07:52:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.363+0000 7f3c527fc700 1 -- 172.21.15.114:0/149433373 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c640c50f0 con 0x561cd1c40f80 2024-05-27T07:52:06.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.363+0000 7f3c527fc700 1 -- 172.21.15.114:0/149433373 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f3c5c007c40 con 0x561cd1c40f80 2024-05-27T07:52:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.364+0000 7f3c7fe9db80 10 client.27764 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:52:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.364+0000 7f3c7fe9db80 10 client.27764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.365+0000 7f3c7fe9db80 20 client.27764 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.365+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/149433373 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cd1cc6210 0x561cd1cc8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.365+0000 7f3c7fe9db80 4 client.27764 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:52:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.365+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/149433373 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561cd1c0c9f0 con 0x561cd1cc6210 2024-05-27T07:52:06.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.366+0000 7f3c6affd700 1 --2- 172.21.15.114:0/149433373 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cd1cc6210 0x561cd1cc8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:06.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.368+0000 7f3c6affd700 1 --2- 172.21.15.114:0/149433373 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cd1cc6210 0x561cd1cc8630 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a73a8947cb2dfde4 server_cookie=ec0ea269395a719f in_seq=0 out_seq=0 2024-05-27T07:52:06.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.368+0000 7f3c527fc700 10 client.27764 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:06.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.370+0000 7f3c527fc700 1 -- 172.21.15.114:0/149433373 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561cd1c0c9f0 con 0x561cd1cc6210 2024-05-27T07:52:06.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.370+0000 7f3c527fc700 10 client.27764 handle_command_reply: tid=0 2024-05-27T07:52:06.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.370+0000 7f3c7fe9db80 1 client.27764 shutdown 2024-05-27T07:52:06.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.371+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/149433373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 msgr2=0x561cd1c531a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:06.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.371+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/149433373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 0x561cd1c531a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c018850 tx=0x7f3c5c018c10 comp rx=0 tx=0).stop 2024-05-27T07:52:06.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.372+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/149433373 shutdown_connections 2024-05-27T07:52:06.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.372+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/149433373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cd1c46c50 0x561cd1c4fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.372+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/149433373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cd1ac2e40 0x561cd1c52c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.372+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/149433373 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cd1cc6210 0x561cd1cc8630 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.372+0000 7f3c7fe9db80 1 --2- 172.21.15.114:0/149433373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cd1c40f80 0x561cd1c531a0 secure :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c018850 tx=0x7f3c5c018c10 comp rx=0 tx=0).stop 2024-05-27T07:52:06.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.372+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/149433373 >> 172.21.15.114:0/149433373 conn(0x561cd1b1daf0 msgr2=0x561cd1c49420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:06.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.372+0000 7f3c527fc700 0 client.27764 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:06.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.373+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/149433373 shutdown_connections 2024-05-27T07:52:06.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.373+0000 7f3c7fe9db80 1 -- 172.21.15.114:0/149433373 wait complete. 2024-05-27T07:52:06.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.374+0000 7f3c7fe9db80 20 client.27764 trim_cache size 0 max 16384 2024-05-27T07:52:06.374 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:52:06.374 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:52:06.374 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:52:06.374 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:52:06.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.375+0000 7f3c71488700 1 -- 172.21.15.114:0/2161828007 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3c580b2c00 msgr2=0x7f3c580b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:06.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.375+0000 7f3c71488700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3c580b2c00 0x7f3c580b5110 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3c54006fd0 tx=0x7f3c54008040 comp rx=0 tx=0).stop 2024-05-27T07:52:06.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.375+0000 7f3c71488700 1 -- 172.21.15.114:0/2161828007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c6c115380 msgr2=0x7f3c6c111340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:06.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.375+0000 7f3c71488700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c6c115380 0x7f3c6c111340 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f3c6400b2e0 tx=0x7f3c6400b8d0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.375+0000 7f3c71488700 1 -- 172.21.15.114:0/2161828007 shutdown_connections 2024-05-27T07:52:06.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.375+0000 7f3c71488700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c6c115380 0x7f3c6c111340 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.375+0000 7f3c71488700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6c075b50 0x7f3c6c114cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.375+0000 7f3c71488700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3c580b2c00 0x7f3c580b5110 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.375+0000 7f3c71488700 1 --2- 172.21.15.114:0/2161828007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c6c0764a0 0x7f3c6c110e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:06.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.375+0000 7f3c71488700 1 -- 172.21.15.114:0/2161828007 >> 172.21.15.114:0/2161828007 conn(0x7f3c6c07af60 msgr2=0x7f3c6c101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:06.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.377+0000 7f3c71488700 1 -- 172.21.15.114:0/2161828007 shutdown_connections 2024-05-27T07:52:06.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:06.377+0000 7f3c71488700 1 -- 172.21.15.114:0/2161828007 wait complete. 2024-05-27T07:52:07.151 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:52:07.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:52:07.292 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1196275 9.65 MB/sec execute 499 sec latency 112.800 ms 2024-05-27T07:52:07.644 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:52:07.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:07 smithi163 ceph-mon[36451]: pgmap v411: 129 pgs: 17 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 63 active+clean; 63 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 292 op/s 2024-05-27T07:52:07.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:07 smithi186 ceph-mon[36647]: pgmap v411: 129 pgs: 17 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 63 active+clean; 63 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 292 op/s 2024-05-27T07:52:08.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:07 smithi114 ceph-mon[32038]: pgmap v411: 129 pgs: 17 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 63 active+clean; 63 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 292 op/s 2024-05-27T07:52:08.292 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1196403 9.63 MB/sec execute 500 sec latency 155.788 ms 2024-05-27T07:52:09.292 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1196999 9.62 MB/sec execute 501 sec latency 221.907 ms 2024-05-27T07:52:09.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:09 smithi163 ceph-mon[36451]: pgmap v412: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 63 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 302 op/s 2024-05-27T07:52:09.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:09 smithi186 ceph-mon[36647]: pgmap v412: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 63 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 302 op/s 2024-05-27T07:52:10.006 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:09 smithi114 ceph-mon[32038]: pgmap v412: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 63 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 302 op/s 2024-05-27T07:52:10.292 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1197218 9.60 MB/sec execute 502 sec latency 48.509 ms 2024-05-27T07:52:10.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.338+0000 7f77a9d63700 1 -- 172.21.15.114:0/3892864790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77a410e930 msgr2=0x7f77a4110e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:10.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.338+0000 7f77a9d63700 1 --2- 172.21.15.114:0/3892864790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77a410e930 0x7f77a4110e00 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f779c00b0a0 tx=0x7f779c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:52:10.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.339+0000 7f77a9d63700 1 -- 172.21.15.114:0/3892864790 shutdown_connections 2024-05-27T07:52:10.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.339+0000 7f77a9d63700 1 --2- 172.21.15.114:0/3892864790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77a410e930 0x7f77a4110e00 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:10.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.339+0000 7f77a9d63700 1 --2- 172.21.15.114:0/3892864790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77a4105470 0x7f77a4105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:10.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.339+0000 7f77a9d63700 1 --2- 172.21.15.114:0/3892864790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77a4105e50 0x7f77a410e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:10.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.339+0000 7f77a9d63700 1 -- 172.21.15.114:0/3892864790 >> 172.21.15.114:0/3892864790 conn(0x7f77a40fef40 msgr2=0x7f77a4101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:10.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.339+0000 7f77a9d63700 1 -- 172.21.15.114:0/3892864790 shutdown_connections 2024-05-27T07:52:10.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.339+0000 7f77a9d63700 1 -- 172.21.15.114:0/3892864790 wait complete. 2024-05-27T07:52:10.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.339+0000 7f77a9d63700 1 Processor -- start 2024-05-27T07:52:10.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.339+0000 7f77a9d63700 1 -- start start 2024-05-27T07:52:10.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a9d63700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77a4105470 0x7f77a419adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:10.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a9d63700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77a4105e50 0x7f77a419b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:10.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a9d63700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77a410e930 0x7f77a419f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:10.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a9d63700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f77a41144a0 con 0x7f77a4105e50 2024-05-27T07:52:10.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a9d63700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f77a4114320 con 0x7f77a4105470 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a9d63700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f77a4114620 con 0x7f77a410e930 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77a4105e50 0x7f77a419b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a37fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77a4105470 0x7f77a419adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77a4105e50 0x7f77a419b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53094/0 (socket says 172.21.15.114:53094) 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a2ffd700 1 -- 172.21.15.114:0/1857160584 learned_addr learned my addr 172.21.15.114:0/1857160584 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a3fff700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77a410e930 0x7f77a419f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a37fe700 1 -- 172.21.15.114:0/1857160584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77a410e930 msgr2=0x7f77a419f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a37fe700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77a410e930 0x7f77a419f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a37fe700 1 -- 172.21.15.114:0/1857160584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77a4105e50 msgr2=0x7f77a419b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a37fe700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77a4105e50 0x7f77a419b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a37fe700 1 -- 172.21.15.114:0/1857160584 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a419fc80 con 0x7f77a4105470 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a2ffd700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77a4105e50 0x7f77a419b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a37fe700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77a4105470 0x7f77a419adb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f779400e9f0 tx=0x7f779400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.340+0000 7f77a0ff9700 1 -- 172.21.15.114:0/1857160584 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779400c9c0 con 0x7f77a4105470 2024-05-27T07:52:10.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.341+0000 7f77a9d63700 1 -- 172.21.15.114:0/1857160584 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77a419ff70 con 0x7f77a4105470 2024-05-27T07:52:10.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.341+0000 7f77a0ff9700 1 -- 172.21.15.114:0/1857160584 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f779400cb60 con 0x7f77a4105470 2024-05-27T07:52:10.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.341+0000 7f77a0ff9700 1 -- 172.21.15.114:0/1857160584 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7794018680 con 0x7f77a4105470 2024-05-27T07:52:10.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.341+0000 7f77a9d63700 1 -- 172.21.15.114:0/1857160584 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77a41a7890 con 0x7f77a4105470 2024-05-27T07:52:10.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.341+0000 7f77a9d63700 1 -- 172.21.15.114:0/1857160584 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77a40512b0 con 0x7f77a4105470 2024-05-27T07:52:10.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.343+0000 7f77a0ff9700 1 -- 172.21.15.114:0/1857160584 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f7794010430 con 0x7f77a4105470 2024-05-27T07:52:10.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.344+0000 7f77a0ff9700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f778c0b29f0 0x7f778c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:10.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.344+0000 7f77a2ffd700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f778c0b29f0 0x7f778c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:10.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.344+0000 7f77a0ff9700 1 -- 172.21.15.114:0/1857160584 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9642+0+0 (secure 0 0 0) 0x7f77940e2f50 con 0x7f77a4105470 2024-05-27T07:52:10.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.344+0000 7f77a2ffd700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f778c0b29f0 0x7f778c0b4f00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f77a4102380 tx=0x7f7798009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:10.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.348+0000 7f77a0ff9700 1 -- 172.21.15.114:0/1857160584 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f77940aa4d0 con 0x7f77a4105470 2024-05-27T07:52:10.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.829+0000 7f77a9d63700 1 -- 172.21.15.114:0/1857160584 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77a419c110 con 0x7f77a4105470 2024-05-27T07:52:10.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.830+0000 7f77a0ff9700 1 -- 172.21.15.114:0/1857160584 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+8522 (secure 0 0 0) 0x7f77940a9dd0 con 0x7f77a4105470 2024-05-27T07:52:10.830 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:52:10.830 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":27557,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3261223371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":3261223371},{"type":"v1","addr":"172.21.15.163:6835","nonce":3261223371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":67,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:51:26.564380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27634":{"gid":27634,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/3266101988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":3266101988},{"type":"v1","addr":"172.21.15.186:6833","nonce":3266101988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:52:10.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.833+0000 7f77a9d63700 1 -- 172.21.15.114:0/1857160584 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f778c0b29f0 msgr2=0x7f778c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:10.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.833+0000 7f77a9d63700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f778c0b29f0 0x7f778c0b4f00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f77a4102380 tx=0x7f7798009260 comp rx=0 tx=0).stop 2024-05-27T07:52:10.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.833+0000 7f77a9d63700 1 -- 172.21.15.114:0/1857160584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77a4105470 msgr2=0x7f77a419adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:10.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.833+0000 7f77a9d63700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77a4105470 0x7f77a419adb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f779400e9f0 tx=0x7f779400ed00 comp rx=0 tx=0).stop 2024-05-27T07:52:10.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.834+0000 7f77a9d63700 1 -- 172.21.15.114:0/1857160584 shutdown_connections 2024-05-27T07:52:10.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.834+0000 7f77a9d63700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77a410e930 0x7f77a419f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:10.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.834+0000 7f77a9d63700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f778c0b29f0 0x7f778c0b4f00 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:10.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.834+0000 7f77a9d63700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77a4105470 0x7f77a419adb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:10.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.834+0000 7f77a9d63700 1 --2- 172.21.15.114:0/1857160584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77a4105e50 0x7f77a419b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:10.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.834+0000 7f77a9d63700 1 -- 172.21.15.114:0/1857160584 >> 172.21.15.114:0/1857160584 conn(0x7f77a40fef40 msgr2=0x7f77a40ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:10.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.834+0000 7f77a9d63700 1 -- 172.21.15.114:0/1857160584 shutdown_connections 2024-05-27T07:52:10.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:10.834+0000 7f77a9d63700 1 -- 172.21.15.114:0/1857160584 wait complete. 2024-05-27T07:52:10.835 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 67 2024-05-27T07:52:11.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1197731 9.58 MB/sec execute 503 sec latency 31.173 ms 2024-05-27T07:52:11.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:11 smithi163 ceph-mon[36451]: pgmap v413: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 63 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 302 op/s 2024-05-27T07:52:11.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:11 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1857160584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:52:11.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:11 smithi186 ceph-mon[36647]: pgmap v413: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 63 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 302 op/s 2024-05-27T07:52:11.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:11 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1857160584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:52:12.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:11 smithi114 ceph-mon[32038]: pgmap v413: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 63 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 302 op/s 2024-05-27T07:52:12.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:11 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1857160584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:52:12.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:52:12.292 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1199808 9.59 MB/sec execute 504 sec latency 41.242 ms 2024-05-27T07:52:12.520 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:52:13.292 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1201151 9.58 MB/sec execute 505 sec latency 33.342 ms 2024-05-27T07:52:13.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:13 smithi114 ceph-mon[32038]: pgmap v414: 129 pgs: 5 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 113 active+clean; 62 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1.8 MiB/s wr, 222 op/s 2024-05-27T07:52:13.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:13 smithi163 ceph-mon[36451]: pgmap v414: 129 pgs: 5 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 113 active+clean; 62 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1.8 MiB/s wr, 222 op/s 2024-05-27T07:52:13.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:13 smithi186 ceph-mon[36647]: pgmap v414: 129 pgs: 5 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 113 active+clean; 62 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1.8 MiB/s wr, 222 op/s 2024-05-27T07:52:14.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.275+0000 7f8e214ce700 1 -- 172.21.15.114:0/3308148373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e1c1029b0 msgr2=0x7f8e1c102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:14.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.275+0000 7f8e214ce700 1 --2- 172.21.15.114:0/3308148373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e1c1029b0 0x7f8e1c102dc0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f8e1000aaa0 tx=0x7f8e1000adb0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.275+0000 7f8e214ce700 1 -- 172.21.15.114:0/3308148373 shutdown_connections 2024-05-27T07:52:14.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.275+0000 7f8e214ce700 1 --2- 172.21.15.114:0/3308148373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e1c103300 0x7f8e1c10f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.275+0000 7f8e214ce700 1 --2- 172.21.15.114:0/3308148373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e1c1089e0 0x7f8e1c108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.275+0000 7f8e214ce700 1 --2- 172.21.15.114:0/3308148373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e1c1029b0 0x7f8e1c102dc0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.275+0000 7f8e214ce700 1 -- 172.21.15.114:0/3308148373 >> 172.21.15.114:0/3308148373 conn(0x7f8e1c0fe6e0 msgr2=0x7f8e1c100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:14.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.276+0000 7f8e214ce700 1 -- 172.21.15.114:0/3308148373 shutdown_connections 2024-05-27T07:52:14.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.276+0000 7f8e214ce700 1 -- 172.21.15.114:0/3308148373 wait complete. 2024-05-27T07:52:14.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.276+0000 7f8e214ce700 1 Processor -- start 2024-05-27T07:52:14.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.276+0000 7f8e214ce700 1 -- start start 2024-05-27T07:52:14.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e214ce700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e1c1029b0 0x7f8e1c19ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e214ce700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e1c103300 0x7f8e1c19b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e214ce700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e1c1089e0 0x7f8e1c19f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e214ce700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c1122e0 con 0x7f8e1c1029b0 2024-05-27T07:52:14.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e214ce700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c112160 con 0x7f8e1c1089e0 2024-05-27T07:52:14.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e214ce700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c112460 con 0x7f8e1c103300 2024-05-27T07:52:14.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e1affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e1c1029b0 0x7f8e1c19ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e1affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e1c1029b0 0x7f8e1c19ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53114/0 (socket says 172.21.15.114:53114) 2024-05-27T07:52:14.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e1affd700 1 -- 172.21.15.114:0/1441316832 learned_addr learned my addr 172.21.15.114:0/1441316832 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:14.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e1a7fc700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e1c103300 0x7f8e1c19b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e1b7fe700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e1c1089e0 0x7f8e1c19f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e1affd700 1 -- 172.21.15.114:0/1441316832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e1c103300 msgr2=0x7f8e1c19b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:14.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e1affd700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e1c103300 0x7f8e1c19b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e1affd700 1 -- 172.21.15.114:0/1441316832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e1c1089e0 msgr2=0x7f8e1c19f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:14.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e1affd700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e1c1089e0 0x7f8e1c19f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e1affd700 1 -- 172.21.15.114:0/1441316832 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1c19fd50 con 0x7f8e1c1029b0 2024-05-27T07:52:14.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e1a7fc700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e1c103300 0x7f8e1c19b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:14.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e1b7fe700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e1c1089e0 0x7f8e1c19f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:52:14.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.277+0000 7f8e1affd700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e1c1029b0 0x7f8e1c19ae80 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00e970 tx=0x7f8e0c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:14.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.278+0000 7f8e03fff700 1 -- 172.21.15.114:0/1441316832 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c00cbd0 con 0x7f8e1c1029b0 2024-05-27T07:52:14.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.278+0000 7f8e214ce700 1 -- 172.21.15.114:0/1441316832 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e1c1a0040 con 0x7f8e1c1029b0 2024-05-27T07:52:14.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.278+0000 7f8e03fff700 1 -- 172.21.15.114:0/1441316832 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e0c00cd70 con 0x7f8e1c1029b0 2024-05-27T07:52:14.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.278+0000 7f8e214ce700 1 -- 172.21.15.114:0/1441316832 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e1c1a7960 con 0x7f8e1c1029b0 2024-05-27T07:52:14.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.279+0000 7f8e03fff700 1 -- 172.21.15.114:0/1441316832 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c005680 con 0x7f8e1c1029b0 2024-05-27T07:52:14.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.281+0000 7f8e03fff700 1 -- 172.21.15.114:0/1441316832 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f8e0c021440 con 0x7f8e1c1029b0 2024-05-27T07:52:14.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e2fee3b80 1 Processor -- start 2024-05-27T07:52:14.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e2fee3b80 1 -- start start 2024-05-27T07:52:14.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e03fff700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8e040b29f0 0x7f8e040b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e1b7fe700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8e040b29f0 0x7f8e040b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e2fee3b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe475d2e40 0x55fe47756710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e2fee3b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe47750a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e03fff700 1 -- 172.21.15.114:0/1441316832 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9642+0+0 (secure 0 0 0) 0x7f8e0c0e2c50 con 0x7f8e1c1029b0 2024-05-27T07:52:14.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e2fee3b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 0x55fe47751390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e2fee3b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fe4767fc90 con 0x55fe47756c50 2024-05-27T07:52:14.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e2fee3b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fe4766f830 con 0x55fe475d2e40 2024-05-27T07:52:14.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e2fee3b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fe47673870 con 0x55fe47750f80 2024-05-27T07:52:14.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e1a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe47750a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e1a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe47750a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53120/0 (socket says 172.21.15.114:53120) 2024-05-27T07:52:14.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e1a7fc700 1 -- 172.21.15.114:0/634520449 learned_addr learned my addr 172.21.15.114:0/634520449 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:14.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e1b7fe700 1 --2- 172.21.15.114:0/634520449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 0x55fe47751390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.282+0000 7f8e1affd700 1 --2- 172.21.15.114:0/634520449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe475d2e40 0x55fe47756710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.283+0000 7f8e1a7fc700 1 -- 172.21.15.114:0/634520449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 msgr2=0x55fe47751390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:14.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.283+0000 7f8e1a7fc700 1 --2- 172.21.15.114:0/634520449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 0x55fe47751390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.283+0000 7f8e1b7fe700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8e040b29f0 0x7f8e040b4f00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8e14009f10 tx=0x7f8e140077e0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:14.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.283+0000 7f8e1a7fc700 1 -- 172.21.15.114:0/634520449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe475d2e40 msgr2=0x55fe47756710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:14.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.283+0000 7f8e1a7fc700 1 --2- 172.21.15.114:0/634520449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe475d2e40 0x55fe47756710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.283+0000 7f8e1a7fc700 1 -- 172.21.15.114:0/634520449 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe475bf8d0 con 0x55fe47756c50 2024-05-27T07:52:14.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.283+0000 7f8e1a7fc700 1 --2- 172.21.15.114:0/634520449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe47750a40 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c076200 tx=0x7f8e1000cfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e376122f2d58fe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:14.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.283+0000 7f8e017fa700 1 -- 172.21.15.114:0/634520449 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e10003680 con 0x55fe47756c50 2024-05-27T07:52:14.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.283+0000 7f8e017fa700 1 -- 172.21.15.114:0/634520449 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e10005580 con 0x55fe47756c50 2024-05-27T07:52:14.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.283+0000 7f8e017fa700 1 -- 172.21.15.114:0/634520449 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e100217e0 con 0x55fe47756c50 2024-05-27T07:52:14.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.284+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/634520449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 msgr2=0x55fe47750a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:14.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.284+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/634520449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe47750a40 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c076200 tx=0x7f8e1000cfd0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.284+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/634520449 shutdown_connections 2024-05-27T07:52:14.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.284+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/634520449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 0x55fe47751390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.284+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/634520449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe475d2e40 0x55fe47756710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.284+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/634520449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe47750a40 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.284+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/634520449 >> 172.21.15.114:0/634520449 conn(0x55fe4762daf0 msgr2=0x55fe4774b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:14.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.284+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/634520449 shutdown_connections 2024-05-27T07:52:14.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.284+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/634520449 wait complete. 2024-05-27T07:52:14.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e2fee3b80 1 -- start start 2024-05-27T07:52:14.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e2fee3b80 1 Processor -- start 2024-05-27T07:52:14.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e2fee3b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe475d2e40 0x55fe4775a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e2fee3b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 0x55fe4775a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e2fee3b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe4775ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e2fee3b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fe47575fa0 con 0x55fe47756c50 2024-05-27T07:52:14.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e2fee3b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fe476117c0 con 0x55fe475d2e40 2024-05-27T07:52:14.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e2fee3b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fe47761af0 con 0x55fe47750f80 2024-05-27T07:52:14.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e2fee3b80 10 client.0 _ll_register_callbacks cb 0x55fe4760a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:14.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e1affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe475d2e40 0x55fe4775a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e1b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe4775ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e1affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe475d2e40 0x55fe4775a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:58120/0 (socket says 172.21.15.114:58120) 2024-05-27T07:52:14.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e1affd700 1 -- 172.21.15.114:0/3757943769 learned_addr learned my addr 172.21.15.114:0/3757943769 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:14.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.285+0000 7f8e1b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe4775ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53130/0 (socket says 172.21.15.114:53130) 2024-05-27T07:52:14.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.286+0000 7f8e1a7fc700 1 --2- 172.21.15.114:0/3757943769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 0x55fe4775a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.286+0000 7f8e1affd700 1 -- 172.21.15.114:0/3757943769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 msgr2=0x55fe4775a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:14.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.286+0000 7f8e1affd700 1 --2- 172.21.15.114:0/3757943769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 0x55fe4775a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.286+0000 7f8e1affd700 1 -- 172.21.15.114:0/3757943769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 msgr2=0x55fe4775ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:14.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.286+0000 7f8e1affd700 1 --2- 172.21.15.114:0/3757943769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe4775ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.286+0000 7f8e1affd700 1 -- 172.21.15.114:0/3757943769 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe4760e690 con 0x55fe475d2e40 2024-05-27T07:52:14.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.286+0000 7f8e1affd700 1 --2- 172.21.15.114:0/3757943769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe475d2e40 0x55fe4775a300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0e0540 tx=0x7f8e0c0a9fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a03b9d19d2a2276b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:14.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.286+0000 7f8dfeff5700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:52:14.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.286+0000 7f8dfeff5700 1 -- 172.21.15.114:0/3757943769 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c06e020 con 0x55fe475d2e40 2024-05-27T07:52:14.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.286+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/3757943769 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e10005580 con 0x55fe475d2e40 2024-05-27T07:52:14.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.286+0000 7f8dfeff5700 1 -- 172.21.15.114:0/3757943769 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9642+0+0 (secure 0 0 0) 0x7f8e0c0a60c0 con 0x55fe475d2e40 2024-05-27T07:52:14.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.287+0000 7f8dfeff5700 1 -- 172.21.15.114:0/3757943769 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f8e0c0a9240 con 0x55fe475d2e40 2024-05-27T07:52:14.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.287+0000 7f8e2fee3b80 10 client.27782 fetch_fsmap learned FSMap version 67 2024-05-27T07:52:14.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.287+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/3757943769 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55fe4776c5c0 con 0x55fe475d2e40 2024-05-27T07:52:14.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.287+0000 7f8dfeff5700 1 -- 172.21.15.114:0/3757943769 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e10005580 con 0x55fe475d2e40 2024-05-27T07:52:14.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.287+0000 7f8dfeff5700 1 -- 172.21.15.114:0/3757943769 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f8e0c06f220 con 0x55fe475d2e40 2024-05-27T07:52:14.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.287+0000 7f8e2fee3b80 10 client.27782 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:52:14.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.287+0000 7f8e2fee3b80 10 client.27782 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:14.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.287+0000 7f8e2fee3b80 20 client.27782 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:14.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.287+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/3757943769 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fe4776ef10 0x55fe47771390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.287+0000 7f8e2fee3b80 4 client.27782 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:52:14.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.287+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/3757943769 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe4771c9f0 con 0x55fe4776ef10 2024-05-27T07:52:14.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.287+0000 7f8e1a7fc700 1 --2- 172.21.15.114:0/3757943769 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fe4776ef10 0x55fe47771390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.288+0000 7f8dfeff5700 10 client.27782 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:14.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.288+0000 7f8e1a7fc700 1 --2- 172.21.15.114:0/3757943769 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fe4776ef10 0x55fe47771390 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae5826ab01476e74 server_cookie=ace52329bb3d2508 in_seq=0 out_seq=0 2024-05-27T07:52:14.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.289+0000 7f8dfeff5700 1 -- 172.21.15.114:0/3757943769 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fe4771c9f0 con 0x55fe4776ef10 2024-05-27T07:52:14.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.289+0000 7f8dfeff5700 10 client.27782 handle_command_reply: tid=0 2024-05-27T07:52:14.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.289+0000 7f8e2fee3b80 1 client.27782 shutdown 2024-05-27T07:52:14.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.290+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/3757943769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe475d2e40 msgr2=0x55fe4775a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:14.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.290+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/3757943769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe475d2e40 0x55fe4775a300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0e0540 tx=0x7f8e0c0a9fe0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.290+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/3757943769 shutdown_connections 2024-05-27T07:52:14.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.290+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/3757943769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe4775ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.290+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/3757943769 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fe4776ef10 0x55fe47771390 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.290+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/3757943769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe475d2e40 0x55fe4775a300 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.290+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/3757943769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 0x55fe4775a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.290+0000 7f8dfeff5700 0 client.27782 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:14.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.290+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/3757943769 >> 172.21.15.114:0/3757943769 conn(0x55fe4762daf0 msgr2=0x55fe4774fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:14.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.290+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/3757943769 shutdown_connections 2024-05-27T07:52:14.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.290+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/3757943769 wait complete. 2024-05-27T07:52:14.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.290+0000 7f8e2fee3b80 20 client.27782 trim_cache size 0 max 16384 2024-05-27T07:52:14.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1202248 9.58 MB/sec execute 506 sec latency 146.319 ms 2024-05-27T07:52:14.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.305+0000 7f8e2fee3b80 1 Processor -- start 2024-05-27T07:52:14.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.305+0000 7f8e2fee3b80 1 -- start start 2024-05-27T07:52:14.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.305+0000 7f8e2fee3b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe475d2e40 0x55fe47750a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.305+0000 7f8e2fee3b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 0x55fe4775abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.305+0000 7f8e2fee3b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe47756c50 0x55fe4776ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.305+0000 7f8e2fee3b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c1aff50 con 0x55fe475d2e40 2024-05-27T07:52:14.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.305+0000 7f8e2fee3b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c1ae6c0 con 0x55fe47756c50 2024-05-27T07:52:14.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.305+0000 7f8e2fee3b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c1ac530 con 0x55fe47750f80 2024-05-27T07:52:14.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.305+0000 7f8e1affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe475d2e40 0x55fe47750a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.305+0000 7f8e1b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe47756c50 0x55fe4776ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.305+0000 7f8e1a7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 0x55fe4775abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.305+0000 7f8e1b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe47756c50 0x55fe4776ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:58134/0 (socket says 172.21.15.114:58134) 2024-05-27T07:52:14.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.306+0000 7f8e1b7fe700 1 -- 172.21.15.114:0/68682309 learned_addr learned my addr 172.21.15.114:0/68682309 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:14.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.306+0000 7f8e1affd700 1 -- 172.21.15.114:0/68682309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 msgr2=0x55fe4775abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:14.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.306+0000 7f8e1affd700 1 --2- 172.21.15.114:0/68682309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 0x55fe4775abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.306+0000 7f8e1affd700 1 -- 172.21.15.114:0/68682309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe47756c50 msgr2=0x55fe4776ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:14.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.306+0000 7f8e1affd700 1 --2- 172.21.15.114:0/68682309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe47756c50 0x55fe4776ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.306+0000 7f8e1affd700 1 -- 172.21.15.114:0/68682309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe477519f0 con 0x55fe475d2e40 2024-05-27T07:52:14.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.306+0000 7f8e1affd700 1 --2- 172.21.15.114:0/68682309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe475d2e40 0x55fe47750a40 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0a9a60 tx=0x7f8e0c005120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d6317b613e404a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:14.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.306+0000 7f8e00ff9700 1 -- 172.21.15.114:0/68682309 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c06e040 con 0x55fe475d2e40 2024-05-27T07:52:14.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.306+0000 7f8e00ff9700 1 -- 172.21.15.114:0/68682309 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e0c0b2270 con 0x55fe475d2e40 2024-05-27T07:52:14.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.306+0000 7f8e00ff9700 1 -- 172.21.15.114:0/68682309 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c0ca080 con 0x55fe475d2e40 2024-05-27T07:52:14.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.307+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/68682309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe475d2e40 msgr2=0x55fe47750a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:14.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.307+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/68682309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe475d2e40 0x55fe47750a40 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0a9a60 tx=0x7f8e0c005120 comp rx=0 tx=0).stop 2024-05-27T07:52:14.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.307+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/68682309 shutdown_connections 2024-05-27T07:52:14.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.307+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/68682309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe47756c50 0x55fe4776ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.307+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/68682309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe475d2e40 0x55fe47750a40 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.307+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/68682309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe47750f80 0x55fe4775abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.307+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/68682309 >> 172.21.15.114:0/68682309 conn(0x55fe4762daf0 msgr2=0x55fe4774b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:14.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.307+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/68682309 shutdown_connections 2024-05-27T07:52:14.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.308+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/68682309 wait complete. 2024-05-27T07:52:14.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.308+0000 7f8e2fee3b80 1 -- start start 2024-05-27T07:52:14.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.308+0000 7f8e2fee3b80 1 Processor -- start 2024-05-27T07:52:14.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.308+0000 7f8e2fee3b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe475d2e40 0x55fe47777300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.308+0000 7f8e2fee3b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe47750f80 0x55fe47766d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.308+0000 7f8e2fee3b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe47767300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.308+0000 7f8e2fee3b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c1acb80 con 0x55fe47756c50 2024-05-27T07:52:14.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.308+0000 7f8e2fee3b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c1adff0 con 0x55fe47750f80 2024-05-27T07:52:14.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.308+0000 7f8e2fee3b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c1ad250 con 0x55fe475d2e40 2024-05-27T07:52:14.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.308+0000 7f8e2fee3b80 10 client.0 _ll_register_callbacks cb 0x55fe477518b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:14.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.308+0000 7f8e1affd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe475d2e40 0x55fe47777300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8e1affd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe475d2e40 0x55fe47777300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:46026/0 (socket says 172.21.15.114:46026) 2024-05-27T07:52:14.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8e1affd700 1 -- 172.21.15.114:0/296588798 learned_addr learned my addr 172.21.15.114:0/296588798 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:14.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8e1b7fe700 1 --2- 172.21.15.114:0/296588798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe47767300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8e1a7fc700 1 --2- 172.21.15.114:0/296588798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe47750f80 0x55fe47766d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8e1affd700 1 -- 172.21.15.114:0/296588798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe47750f80 msgr2=0x55fe47766d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:14.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8e1affd700 1 --2- 172.21.15.114:0/296588798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe47750f80 0x55fe47766d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8e1affd700 1 -- 172.21.15.114:0/296588798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 msgr2=0x55fe47767300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:14.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8e1affd700 1 --2- 172.21.15.114:0/296588798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe47767300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8e1affd700 1 -- 172.21.15.114:0/296588798 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe47775b80 con 0x55fe475d2e40 2024-05-27T07:52:14.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8e1affd700 1 --2- 172.21.15.114:0/296588798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe475d2e40 0x55fe47777300 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0a5b50 tx=0x7f8e0c0b2780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b80512fbca313acb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:14.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8dfeff5700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:52:14.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8dfeff5700 1 -- 172.21.15.114:0/296588798 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c06e070 con 0x55fe475d2e40 2024-05-27T07:52:14.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/296588798 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fe47681be0 con 0x55fe475d2e40 2024-05-27T07:52:14.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.309+0000 7f8dfeff5700 1 -- 172.21.15.114:0/296588798 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9642+0+0 (secure 0 0 0) 0x7f8e0c0a60c0 con 0x55fe475d2e40 2024-05-27T07:52:14.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.310+0000 7f8dfeff5700 1 -- 172.21.15.114:0/296588798 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f8e0c0bb050 con 0x55fe475d2e40 2024-05-27T07:52:14.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.310+0000 7f8e2fee3b80 10 client.27922 fetch_fsmap learned FSMap version 67 2024-05-27T07:52:14.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.310+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/296588798 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55fe47775b80 con 0x55fe475d2e40 2024-05-27T07:52:14.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.310+0000 7f8dfeff5700 1 -- 172.21.15.114:0/296588798 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55fe47681be0 con 0x55fe475d2e40 2024-05-27T07:52:14.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.310+0000 7f8dfeff5700 1 -- 172.21.15.114:0/296588798 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 67) ==== 5910+0+0 (secure 0 0 0) 0x7f8e0c0b2b80 con 0x55fe475d2e40 2024-05-27T07:52:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.310+0000 7f8e2fee3b80 10 client.27922 fetch_fsmap finished waiting for FSMap version 67 2024-05-27T07:52:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.310+0000 7f8e2fee3b80 10 client.27922 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.310+0000 7f8e2fee3b80 20 client.27922 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.310+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/296588798 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fe477d8e60 0x55fe477db280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.310+0000 7f8e2fee3b80 4 client.27922 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:52:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.310+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/296588798 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe4767b3c0 con 0x55fe477d8e60 2024-05-27T07:52:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.310+0000 7f8e1a7fc700 1 --2- 172.21.15.114:0/296588798 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fe477d8e60 0x55fe477db280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.311+0000 7f8dfeff5700 10 client.27922 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.311+0000 7f8e1a7fc700 1 --2- 172.21.15.114:0/296588798 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fe477d8e60 0x55fe477db280 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=703565b2427b7871 server_cookie=3a07d704494c33a0 in_seq=0 out_seq=0 2024-05-27T07:52:14.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8dfeff5700 1 -- 172.21.15.114:0/296588798 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe4767b3c0 con 0x55fe477d8e60 2024-05-27T07:52:14.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8dfeff5700 10 client.27922 handle_command_reply: tid=0 2024-05-27T07:52:14.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8e2fee3b80 1 client.27922 shutdown 2024-05-27T07:52:14.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/296588798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe475d2e40 msgr2=0x55fe47777300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:14.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/296588798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe475d2e40 0x55fe47777300 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0a5b50 tx=0x7f8e0c0b2780 comp rx=0 tx=0).stop 2024-05-27T07:52:14.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/296588798 shutdown_connections 2024-05-27T07:52:14.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/296588798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fe47756c50 0x55fe47767300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/296588798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fe475d2e40 0x55fe47777300 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/296588798 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fe477d8e60 0x55fe477db280 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8e2fee3b80 1 --2- 172.21.15.114:0/296588798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe47750f80 0x55fe47766d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/296588798 >> 172.21.15.114:0/296588798 conn(0x55fe4762daf0 msgr2=0x55fe47754010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:14.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8dfeff5700 0 client.27922 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:14.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/296588798 shutdown_connections 2024-05-27T07:52:14.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8e2fee3b80 1 -- 172.21.15.114:0/296588798 wait complete. 2024-05-27T07:52:14.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.312+0000 7f8e2fee3b80 20 client.27922 trim_cache size 0 max 16384 2024-05-27T07:52:14.319 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:52:14.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.313+0000 7f8e214ce700 1 -- 172.21.15.114:0/1441316832 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8e040b29f0 msgr2=0x7f8e040b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:14.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.313+0000 7f8e214ce700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8e040b29f0 0x7f8e040b4f00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8e14009f10 tx=0x7f8e140077e0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.313+0000 7f8e214ce700 1 -- 172.21.15.114:0/1441316832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e1c1029b0 msgr2=0x7f8e1c19ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:14.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.313+0000 7f8e214ce700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e1c1029b0 0x7f8e1c19ae80 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00e970 tx=0x7f8e0c00ec80 comp rx=0 tx=0).stop 2024-05-27T07:52:14.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.313+0000 7f8e214ce700 1 -- 172.21.15.114:0/1441316832 shutdown_connections 2024-05-27T07:52:14.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.313+0000 7f8e214ce700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8e040b29f0 0x7f8e040b4f00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.313+0000 7f8e214ce700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e1c1089e0 0x7f8e1c19f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.313+0000 7f8e214ce700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e1c1029b0 0x7f8e1c19ae80 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.313+0000 7f8e214ce700 1 --2- 172.21.15.114:0/1441316832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e1c103300 0x7f8e1c19b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:14.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.313+0000 7f8e214ce700 1 -- 172.21.15.114:0/1441316832 >> 172.21.15.114:0/1441316832 conn(0x7f8e1c0fe6e0 msgr2=0x7f8e1c107c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:14.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.313+0000 7f8e214ce700 1 -- 172.21.15.114:0/1441316832 shutdown_connections 2024-05-27T07:52:14.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:14.313+0000 7f8e214ce700 1 -- 172.21.15.114:0/1441316832 wait complete. 2024-05-27T07:52:14.926 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:14 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:52:14.926 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:14 smithi186 systemd[1]: Stopped Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:52:14.926 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:14 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.966s CPU time. 2024-05-27T07:52:14.926 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:14 smithi186 systemd[1]: Starting Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:52:15.202 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:14 smithi186 podman[75333]: 2024-05-27 07:52:14.830945738 +0000 UTC m=+0.021885024 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:52:15.202 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:14 smithi186 podman[75333]: 2024-05-27 07:52:14.948903464 +0000 UTC m=+0.139842741 container create 5e80a163146534888574da2055e3754dec7a67ad4ec7f8d6b74e6b1e68078703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:52:15.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1204163 9.58 MB/sec execute 507 sec latency 42.426 ms 2024-05-27T07:52:15.479 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 podman[75333]: 2024-05-27 07:52:15.202091091 +0000 UTC m=+0.393030377 container init 5e80a163146534888574da2055e3754dec7a67ad4ec7f8d6b74e6b1e68078703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-27T07:52:15.479 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 podman[75333]: 2024-05-27 07:52:15.205729246 +0000 UTC m=+0.396668529 container start 5e80a163146534888574da2055e3754dec7a67ad4ec7f8d6b74e6b1e68078703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-27T07:52:15.479 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-mds[75351]: -- 172.21.15.186:0/88518904 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55df00d5fa40 con 0x55df00da7180 2024-05-27T07:52:15.479 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: starting mds.l at 2024-05-27T07:52:15.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 bash[75333]: 5e80a163146534888574da2055e3754dec7a67ad4ec7f8d6b74e6b1e68078703 2024-05-27T07:52:15.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 systemd[1]: Started Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:52:15.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:15 smithi163 ceph-mon[36451]: pgmap v415: 129 pgs: 5 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 113 active+clean; 62 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-05-27T07:52:15.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:15 smithi163 ceph-mon[36451]: daemon mds.l restarted 2024-05-27T07:52:15.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:15 smithi186 ceph-mon[36647]: pgmap v415: 129 pgs: 5 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 113 active+clean; 62 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-05-27T07:52:15.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:15 smithi186 ceph-mon[36647]: daemon mds.l restarted 2024-05-27T07:52:15.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6bf14af700 time 2024-05-27T07:52:15.815934+0000 2024-05-27T07:52:15.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:52:15.980 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:15.980 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6c03cf4ba6] 2024-05-27T07:52:15.980 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6c03cf4dd1] 2024-05-27T07:52:15.980 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55defe28723d] 2024-05-27T07:52:15.980 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55defe272971] 2024-05-27T07:52:15.980 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 5: (SnapRealm::check_cache() const+0x13e) [0x55defe270fa0] 2024-05-27T07:52:15.980 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55defe271651] 2024-05-27T07:52:15.980 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55defe206347] 2024-05-27T07:52:15.980 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55defdf0287e] 2024-05-27T07:52:15.981 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55defdf07f2f] 2024-05-27T07:52:15.981 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55defdf12969] 2024-05-27T07:52:15.981 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 11: (MDLog::_replay_thread()+0x1f38) [0x55defe2ab48c] 2024-05-27T07:52:15.981 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55defe2b49cb] 2024-05-27T07:52:15.981 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 13: (Thread::entry_wrapper()+0x43) [0x7f6c03cc372d] 2024-05-27T07:52:15.981 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6c03cc3749] 2024-05-27T07:52:15.981 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6c028671ca] 2024-05-27T07:52:15.981 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 16: clone() 2024-05-27T07:52:15.981 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: *** Caught signal (Aborted) ** 2024-05-27T07:52:15.982 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: in thread 7f6bf14af700 thread_name:md_log_replay 2024-05-27T07:52:15.982 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.817+0000 7f6bf14af700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6bf14af700 time 2024-05-27T07:52:15.815934+0000 2024-05-27T07:52:15.983 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:52:15.983 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.983 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:15.983 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6c03cf4ba6] 2024-05-27T07:52:15.983 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6c03cf4dd1] 2024-05-27T07:52:15.983 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55defe28723d] 2024-05-27T07:52:15.983 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55defe272971] 2024-05-27T07:52:15.983 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 5: (SnapRealm::check_cache() const+0x13e) [0x55defe270fa0] 2024-05-27T07:52:15.983 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55defe271651] 2024-05-27T07:52:15.983 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55defe206347] 2024-05-27T07:52:15.984 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55defdf0287e] 2024-05-27T07:52:15.984 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55defdf07f2f] 2024-05-27T07:52:15.984 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55defdf12969] 2024-05-27T07:52:15.984 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 11: (MDLog::_replay_thread()+0x1f38) [0x55defe2ab48c] 2024-05-27T07:52:15.984 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55defe2b49cb] 2024-05-27T07:52:15.984 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 13: (Thread::entry_wrapper()+0x43) [0x7f6c03cc372d] 2024-05-27T07:52:15.984 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6c03cc3749] 2024-05-27T07:52:15.984 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6c028671ca] 2024-05-27T07:52:15.984 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 16: clone() 2024-05-27T07:52:15.984 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.984 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:15.985 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6c02871d20] 2024-05-27T07:52:15.985 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2: gsignal() 2024-05-27T07:52:15.985 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 3: abort() 2024-05-27T07:52:15.985 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6c03cf4ced] 2024-05-27T07:52:15.985 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6c03cf4dd1] 2024-05-27T07:52:15.985 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55defe28723d] 2024-05-27T07:52:15.985 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55defe272971] 2024-05-27T07:52:15.985 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 8: (SnapRealm::check_cache() const+0x13e) [0x55defe270fa0] 2024-05-27T07:52:15.985 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55defe271651] 2024-05-27T07:52:15.985 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55defe206347] 2024-05-27T07:52:15.985 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55defdf0287e] 2024-05-27T07:52:15.985 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55defdf07f2f] 2024-05-27T07:52:15.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55defdf12969] 2024-05-27T07:52:15.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 14: (MDLog::_replay_thread()+0x1f38) [0x55defe2ab48c] 2024-05-27T07:52:15.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55defe2b49cb] 2024-05-27T07:52:15.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 16: (Thread::entry_wrapper()+0x43) [0x7f6c03cc372d] 2024-05-27T07:52:15.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6c03cc3749] 2024-05-27T07:52:15.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6c028671ca] 2024-05-27T07:52:15.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 19: clone() 2024-05-27T07:52:15.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.820+0000 7f6bf14af700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:52:15.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: in thread 7f6bf14af700 thread_name:md_log_replay 2024-05-27T07:52:15.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:15.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6c02871d20] 2024-05-27T07:52:15.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2: gsignal() 2024-05-27T07:52:15.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 3: abort() 2024-05-27T07:52:15.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6c03cf4ced] 2024-05-27T07:52:15.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6c03cf4dd1] 2024-05-27T07:52:15.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55defe28723d] 2024-05-27T07:52:15.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55defe272971] 2024-05-27T07:52:15.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 8: (SnapRealm::check_cache() const+0x13e) [0x55defe270fa0] 2024-05-27T07:52:15.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55defe271651] 2024-05-27T07:52:15.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55defe206347] 2024-05-27T07:52:15.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55defdf0287e] 2024-05-27T07:52:15.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55defdf07f2f] 2024-05-27T07:52:15.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55defdf12969] 2024-05-27T07:52:15.988 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 14: (MDLog::_replay_thread()+0x1f38) [0x55defe2ab48c] 2024-05-27T07:52:15.988 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55defe2b49cb] 2024-05-27T07:52:15.988 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 16: (Thread::entry_wrapper()+0x43) [0x7f6c03cc372d] 2024-05-27T07:52:15.988 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6c03cc3749] 2024-05-27T07:52:15.988 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6c028671ca] 2024-05-27T07:52:15.988 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 19: clone() 2024-05-27T07:52:15.988 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:52:15.989 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.989 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: -1> 2024-05-27T07:52:15.817+0000 7f6bf14af700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6bf14af700 time 2024-05-27T07:52:15.815934+0000 2024-05-27T07:52:15.989 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:52:15.989 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.989 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:15.989 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6c03cf4ba6] 2024-05-27T07:52:15.989 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6c03cf4dd1] 2024-05-27T07:52:15.989 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55defe28723d] 2024-05-27T07:52:15.990 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55defe272971] 2024-05-27T07:52:15.990 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 5: (SnapRealm::check_cache() const+0x13e) [0x55defe270fa0] 2024-05-27T07:52:15.990 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55defe271651] 2024-05-27T07:52:15.990 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55defe206347] 2024-05-27T07:52:15.990 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55defdf0287e] 2024-05-27T07:52:15.991 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55defdf07f2f] 2024-05-27T07:52:15.991 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55defdf12969] 2024-05-27T07:52:15.991 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 11: (MDLog::_replay_thread()+0x1f38) [0x55defe2ab48c] 2024-05-27T07:52:15.991 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55defe2b49cb] 2024-05-27T07:52:15.992 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 13: (Thread::entry_wrapper()+0x43) [0x7f6c03cc372d] 2024-05-27T07:52:15.992 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6c03cc3749] 2024-05-27T07:52:15.992 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6c028671ca] 2024-05-27T07:52:15.992 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 16: clone() 2024-05-27T07:52:15.992 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.992 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 0> 2024-05-27T07:52:15.820+0000 7f6bf14af700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:52:15.992 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: in thread 7f6bf14af700 thread_name:md_log_replay 2024-05-27T07:52:15.992 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.992 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:15.992 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6c02871d20] 2024-05-27T07:52:15.992 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2: gsignal() 2024-05-27T07:52:15.993 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 3: abort() 2024-05-27T07:52:15.993 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6c03cf4ced] 2024-05-27T07:52:15.993 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6c03cf4dd1] 2024-05-27T07:52:15.993 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55defe28723d] 2024-05-27T07:52:15.993 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55defe272971] 2024-05-27T07:52:15.993 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 8: (SnapRealm::check_cache() const+0x13e) [0x55defe270fa0] 2024-05-27T07:52:15.993 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55defe271651] 2024-05-27T07:52:15.993 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55defe206347] 2024-05-27T07:52:15.993 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55defdf0287e] 2024-05-27T07:52:15.993 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55defdf07f2f] 2024-05-27T07:52:15.993 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55defdf12969] 2024-05-27T07:52:15.993 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 14: (MDLog::_replay_thread()+0x1f38) [0x55defe2ab48c] 2024-05-27T07:52:15.994 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55defe2b49cb] 2024-05-27T07:52:15.994 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 16: (Thread::entry_wrapper()+0x43) [0x7f6c03cc372d] 2024-05-27T07:52:15.994 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6c03cc3749] 2024-05-27T07:52:15.994 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6c028671ca] 2024-05-27T07:52:15.994 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 19: clone() 2024-05-27T07:52:15.994 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:52:15.994 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.994 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: -1> 2024-05-27T07:52:15.817+0000 7f6bf14af700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6bf14af700 time 2024-05-27T07:52:15.815934+0000 2024-05-27T07:52:15.994 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:52:15.994 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.994 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:15.995 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6c03cf4ba6] 2024-05-27T07:52:15.995 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6c03cf4dd1] 2024-05-27T07:52:15.995 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55defe28723d] 2024-05-27T07:52:15.995 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55defe272971] 2024-05-27T07:52:15.995 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 5: (SnapRealm::check_cache() const+0x13e) [0x55defe270fa0] 2024-05-27T07:52:15.995 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55defe271651] 2024-05-27T07:52:15.995 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55defe206347] 2024-05-27T07:52:15.995 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55defdf0287e] 2024-05-27T07:52:15.995 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55defdf07f2f] 2024-05-27T07:52:15.995 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55defdf12969] 2024-05-27T07:52:15.995 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 11: (MDLog::_replay_thread()+0x1f38) [0x55defe2ab48c] 2024-05-27T07:52:15.995 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55defe2b49cb] 2024-05-27T07:52:15.996 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 13: (Thread::entry_wrapper()+0x43) [0x7f6c03cc372d] 2024-05-27T07:52:15.996 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6c03cc3749] 2024-05-27T07:52:15.996 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6c028671ca] 2024-05-27T07:52:15.996 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 16: clone() 2024-05-27T07:52:15.996 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.996 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 0> 2024-05-27T07:52:15.820+0000 7f6bf14af700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:52:15.996 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: in thread 7f6bf14af700 thread_name:md_log_replay 2024-05-27T07:52:15.996 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.996 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:15.996 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6c02871d20] 2024-05-27T07:52:15.996 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2: gsignal() 2024-05-27T07:52:15.996 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 3: abort() 2024-05-27T07:52:15.997 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6c03cf4ced] 2024-05-27T07:52:15.997 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6c03cf4dd1] 2024-05-27T07:52:15.997 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55defe28723d] 2024-05-27T07:52:15.997 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55defe272971] 2024-05-27T07:52:15.997 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 8: (SnapRealm::check_cache() const+0x13e) [0x55defe270fa0] 2024-05-27T07:52:15.997 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55defe271651] 2024-05-27T07:52:15.997 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55defe206347] 2024-05-27T07:52:15.997 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55defdf0287e] 2024-05-27T07:52:15.997 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55defdf07f2f] 2024-05-27T07:52:15.997 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55defdf12969] 2024-05-27T07:52:15.997 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 14: (MDLog::_replay_thread()+0x1f38) [0x55defe2ab48c] 2024-05-27T07:52:15.997 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55defe2b49cb] 2024-05-27T07:52:15.998 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 16: (Thread::entry_wrapper()+0x43) [0x7f6c03cc372d] 2024-05-27T07:52:15.998 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6c03cc3749] 2024-05-27T07:52:15.998 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6c028671ca] 2024-05-27T07:52:15.998 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 19: clone() 2024-05-27T07:52:15.998 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:52:15.998 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75347]: 2024-05-27T07:52:15.998 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:15 smithi186 podman[75383]: 2024-05-27 07:52:15.886021099 +0000 UTC m=+0.021244600 container died 5e80a163146534888574da2055e3754dec7a67ad4ec7f8d6b74e6b1e68078703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-05-27T07:52:16.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:15 smithi114 ceph-mon[32038]: pgmap v415: 129 pgs: 5 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 113 active+clean; 62 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-05-27T07:52:16.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:15 smithi114 ceph-mon[32038]: daemon mds.l restarted 2024-05-27T07:52:16.126 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:52:16.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:52:16.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1205445 9.57 MB/sec execute 508 sec latency 21.896 ms 2024-05-27T07:52:16.379 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:52:16.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:16 smithi114 ceph-mon[32038]: osdmap e126: 12 total, 12 up, 12 in 2024-05-27T07:52:16.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:16 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6832/379725181,v1:172.21.15.186:6833/379725181] up:boot 2024-05-27T07:52:16.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:16 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:52:16.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:16 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:52:16.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:16 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:52:16.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:16 smithi163 ceph-mon[36451]: osdmap e126: 12 total, 12 up, 12 in 2024-05-27T07:52:16.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:16 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6832/379725181,v1:172.21.15.186:6833/379725181] up:boot 2024-05-27T07:52:16.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:16 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:52:16.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:16 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:52:16.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:16 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:52:16.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:16 smithi186 ceph-mon[36647]: osdmap e126: 12 total, 12 up, 12 in 2024-05-27T07:52:16.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:16 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6832/379725181,v1:172.21.15.186:6833/379725181] up:boot 2024-05-27T07:52:16.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:16 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:52:16.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:16 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:52:16.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:16 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:52:16.980 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:16 smithi186 podman[75383]: 2024-05-27 07:52:16.710604896 +0000 UTC m=+0.845828388 container remove 5e80a163146534888574da2055e3754dec7a67ad4ec7f8d6b74e6b1e68078703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-27T07:52:16.980 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:16 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:52:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.131+0000 7f6bdbed9640 1 -- 172.21.15.114:0/511121178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bd41032d0 msgr2=0x7f6bd410f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.131+0000 7f6bdbed9640 1 --2- 172.21.15.114:0/511121178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bd41032d0 0x7f6bd410f6a0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009ca0 tx=0x7f6bcc02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.131+0000 7f6bdbed9640 1 -- 172.21.15.114:0/511121178 shutdown_connections 2024-05-27T07:52:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.131+0000 7f6bdbed9640 1 --2- 172.21.15.114:0/511121178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bd41032d0 0x7f6bd410f6a0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.132+0000 7f6bdbed9640 1 --2- 172.21.15.114:0/511121178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bd4109020 0x7f6bd4102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.132+0000 7f6bdbed9640 1 --2- 172.21.15.114:0/511121178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bd4108b80 0x7f6bd4102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.132+0000 7f6bdbed9640 1 -- 172.21.15.114:0/511121178 >> 172.21.15.114:0/511121178 conn(0x7f6bd40fe380 msgr2=0x7f6bd41007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.132+0000 7f6bdbed9640 1 -- 172.21.15.114:0/511121178 shutdown_connections 2024-05-27T07:52:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.132+0000 7f6bdbed9640 1 -- 172.21.15.114:0/511121178 wait complete. 2024-05-27T07:52:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.132+0000 7f6bdbed9640 1 Processor -- start 2024-05-27T07:52:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.132+0000 7f6bdbed9640 1 -- start start 2024-05-27T07:52:17.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bdbed9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bd41032d0 0x7f6bd419b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bdbed9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bd4108b80 0x7f6bd419bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bdbed9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bd4109020 0x7f6bd419c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bdbed9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd4112d90 con 0x7f6bd4108b80 2024-05-27T07:52:17.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bdbed9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd4112c10 con 0x7f6bd4109020 2024-05-27T07:52:17.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bdbed9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd4112f10 con 0x7f6bd41032d0 2024-05-27T07:52:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bd9474640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bd4108b80 0x7f6bd419bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bd9474640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bd4108b80 0x7f6bd419bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52528/0 (socket says 172.21.15.114:52528) 2024-05-27T07:52:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bd9474640 1 -- 172.21.15.114:0/3447502213 learned_addr learned my addr 172.21.15.114:0/3447502213 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bd9c75640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bd41032d0 0x7f6bd419b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bda476640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bd4109020 0x7f6bd419c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bd9474640 1 -- 172.21.15.114:0/3447502213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bd41032d0 msgr2=0x7f6bd419b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bd9474640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bd41032d0 0x7f6bd419b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bd9474640 1 -- 172.21.15.114:0/3447502213 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bd4109020 msgr2=0x7f6bd419c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bd9474640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bd4109020 0x7f6bd419c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bd9474640 1 -- 172.21.15.114:0/3447502213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd41a69f0 con 0x7f6bd4108b80 2024-05-27T07:52:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bd9474640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bd4108b80 0x7f6bd419bbf0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400e9f0 tx=0x7f6bc400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bcaffd640 1 -- 172.21.15.114:0/3447502213 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc400ccc0 con 0x7f6bd4108b80 2024-05-27T07:52:17.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bcaffd640 1 -- 172.21.15.114:0/3447502213 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bc40035f0 con 0x7f6bd4108b80 2024-05-27T07:52:17.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bcaffd640 1 -- 172.21.15.114:0/3447502213 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc400ce60 con 0x7f6bd4108b80 2024-05-27T07:52:17.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.133+0000 7f6bdbed9640 1 -- 172.21.15.114:0/3447502213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd41a6ce0 con 0x7f6bd4108b80 2024-05-27T07:52:17.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.134+0000 7f6bdbed9640 1 -- 172.21.15.114:0/3447502213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd41a71d0 con 0x7f6bd4108b80 2024-05-27T07:52:17.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.136+0000 7f6be2d8e740 1 Processor -- start 2024-05-27T07:52:17.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.136+0000 7f6be2d8e740 1 -- start start 2024-05-27T07:52:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.136+0000 7f6bcaffd640 1 -- 172.21.15.114:0/3447502213 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6bc4003aa0 con 0x7f6bd4108b80 2024-05-27T07:52:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.136+0000 7f6be2d8e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda192a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.136+0000 7f6be2d8e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda197e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.136+0000 7f6be2d8e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 0x5627eda19d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.136+0000 7f6be2d8e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5627ed98bc30 con 0x5627ed6781b0 2024-05-27T07:52:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.136+0000 7f6be2d8e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5627ed97b7d0 con 0x5627ed64f2f0 2024-05-27T07:52:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.136+0000 7f6be2d8e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5627ed97f810 con 0x5627ed6574d0 2024-05-27T07:52:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.136+0000 7f6bda476640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda192a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bd9c75640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda197e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bd9474640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 0x5627eda19d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bda476640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda192a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52532/0 (socket says 172.21.15.114:52532) 2024-05-27T07:52:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bda476640 1 -- 172.21.15.114:0/1229793380 learned_addr learned my addr 172.21.15.114:0/1229793380 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bd9c75640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda197e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37710/0 (socket says 172.21.15.114:37710) 2024-05-27T07:52:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bda476640 1 -- 172.21.15.114:0/1229793380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 msgr2=0x5627eda19d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bda476640 1 --2- 172.21.15.114:0/1229793380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 0x5627eda19d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bda476640 1 -- 172.21.15.114:0/1229793380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 msgr2=0x5627eda197e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bda476640 1 --2- 172.21.15.114:0/1229793380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda197e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bda476640 1 -- 172.21.15.114:0/1229793380 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627ed690210 con 0x5627ed6781b0 2024-05-27T07:52:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bda476640 1 --2- 172.21.15.114:0/1229793380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda192a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009ca0 tx=0x7f6bcc02f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68553228414caad3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bcaffd640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6bc00b2950 0x7f6bc00b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bd9c75640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6bc00b2950 0x7f6bc00b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bbffff640 1 -- 172.21.15.114:0/1229793380 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc031cf0 con 0x5627ed6781b0 2024-05-27T07:52:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bcaffd640 1 -- 172.21.15.114:0/3447502213 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7f6bc40e2d60 con 0x7f6bd4108b80 2024-05-27T07:52:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bbffff640 1 -- 172.21.15.114:0/1229793380 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bcc040070 con 0x5627ed6781b0 2024-05-27T07:52:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.137+0000 7f6bbffff640 1 -- 172.21.15.114:0/1229793380 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc004040 con 0x5627ed6781b0 2024-05-27T07:52:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.138+0000 7f6be2d8e740 1 -- 172.21.15.114:0/1229793380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 msgr2=0x5627eda192a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.138+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/1229793380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda192a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009ca0 tx=0x7f6bcc02f420 comp rx=0 tx=0).stop 2024-05-27T07:52:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.138+0000 7f6bd9c75640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6bc00b2950 0x7f6bc00b4e60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x5627eda16c30 tx=0x7f6bd0005f90 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.138+0000 7f6be2d8e740 1 -- 172.21.15.114:0/1229793380 shutdown_connections 2024-05-27T07:52:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.138+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/1229793380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 0x5627eda19d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.138+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/1229793380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda197e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.138+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/1229793380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda192a0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.138+0000 7f6be2d8e740 1 -- 172.21.15.114:0/1229793380 >> 172.21.15.114:0/1229793380 conn(0x5627ed66c300 msgr2=0x5627eda15b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.138+0000 7f6be2d8e740 1 -- 172.21.15.114:0/1229793380 shutdown_connections 2024-05-27T07:52:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.138+0000 7f6be2d8e740 1 -- 172.21.15.114:0/1229793380 wait complete. 2024-05-27T07:52:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6be2d8e740 1 -- start start 2024-05-27T07:52:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6be2d8e740 1 Processor -- start 2024-05-27T07:52:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6be2d8e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda29890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6be2d8e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 0x5627eda29dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6be2d8e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda27d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6be2d8e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5627ed661790 con 0x5627ed6781b0 2024-05-27T07:52:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6be2d8e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5627ed937890 con 0x5627ed64f2f0 2024-05-27T07:52:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6be2d8e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5627eda2a960 con 0x5627ed6574d0 2024-05-27T07:52:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6be2d8e740 10 client.0 _ll_register_callbacks cb 0x5627ed68fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6bda476640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda27d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6bda476640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda27d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52542/0 (socket says 172.21.15.114:52542) 2024-05-27T07:52:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6bda476640 1 -- 172.21.15.114:0/1696371279 learned_addr learned my addr 172.21.15.114:0/1696371279 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6bd9c75640 1 --2- 172.21.15.114:0/1696371279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda29890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.139+0000 7f6bd9474640 1 --2- 172.21.15.114:0/1696371279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 0x5627eda29dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6bda476640 1 -- 172.21.15.114:0/1696371279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 msgr2=0x5627eda29dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6bda476640 1 --2- 172.21.15.114:0/1696371279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 0x5627eda29dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6bda476640 1 -- 172.21.15.114:0/1696371279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 msgr2=0x5627eda29890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6bda476640 1 --2- 172.21.15.114:0/1696371279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda29890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6bda476640 1 -- 172.21.15.114:0/1696371279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627ed75c620 con 0x5627ed6781b0 2024-05-27T07:52:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6bda476640 1 --2- 172.21.15.114:0/1696371279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda27d10 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc02ade0 tx=0x7f6bcc031680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3123b2ef78e2fd6f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6bbd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:52:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6bbd7fa640 1 -- 172.21.15.114:0/1696371279 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc04c070 con 0x5627ed6781b0 2024-05-27T07:52:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6bbd7fa640 1 -- 172.21.15.114:0/1696371279 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7f6bcc02f860 con 0x5627ed6781b0 2024-05-27T07:52:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6be2d8e740 1 -- 172.21.15.114:0/1696371279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bcc040070 con 0x5627ed6781b0 2024-05-27T07:52:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6bbd7fa640 1 -- 172.21.15.114:0/1696371279 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f6bcc009360 con 0x5627ed6781b0 2024-05-27T07:52:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6be2d8e740 10 client.18396 fetch_fsmap learned FSMap version 69 2024-05-27T07:52:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6be2d8e740 1 -- 172.21.15.114:0/1696371279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x5627eda36c90 con 0x5627ed6781b0 2024-05-27T07:52:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6bbd7fa640 1 -- 172.21.15.114:0/1696371279 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bcc040070 con 0x5627ed6781b0 2024-05-27T07:52:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.140+0000 7f6bbd7fa640 1 -- 172.21.15.114:0/1696371279 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 69) ==== 5910+0+0 (secure 0 0 0) 0x7f6bcc052bc0 con 0x5627ed6781b0 2024-05-27T07:52:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.141+0000 7f6be2d8e740 10 client.18396 fetch_fsmap finished waiting for FSMap version 69 2024-05-27T07:52:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.141+0000 7f6be2d8e740 10 client.18396 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.141+0000 7f6be2d8e740 20 client.18396 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.141+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/1696371279 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5627ed636d80 0x5627eda39ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.141+0000 7f6be2d8e740 4 client.18396 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:52:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.141+0000 7f6be2d8e740 1 -- 172.21.15.114:0/1696371279 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627ed98db80 con 0x5627ed636d80 2024-05-27T07:52:17.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.141+0000 7f6bd9c75640 1 --2- 172.21.15.114:0/1696371279 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5627ed636d80 0x5627eda39ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.141+0000 7f6bbd7fa640 10 client.18396 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:17.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.141+0000 7f6bd9c75640 1 --2- 172.21.15.114:0/1696371279 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5627ed636d80 0x5627eda39ff0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb98b1d4fd26c7e6 server_cookie=25e49c7ef02ae3e1 in_seq=0 out_seq=0 2024-05-27T07:52:17.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6bbd7fa640 1 -- 172.21.15.114:0/1696371279 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5627ed98db80 con 0x5627ed636d80 2024-05-27T07:52:17.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6bbd7fa640 10 client.18396 handle_command_reply: tid=0 2024-05-27T07:52:17.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6be2d8e740 1 client.18396 shutdown 2024-05-27T07:52:17.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6be2d8e740 1 -- 172.21.15.114:0/1696371279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 msgr2=0x5627eda27d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:17.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/1696371279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda27d10 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc02ade0 tx=0x7f6bcc031680 comp rx=0 tx=0).stop 2024-05-27T07:52:17.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6be2d8e740 1 -- 172.21.15.114:0/1696371279 shutdown_connections 2024-05-27T07:52:17.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/1696371279 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5627ed636d80 0x5627eda39ff0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/1696371279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda27d10 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6bbd7fa640 0 client.18396 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:17.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/1696371279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 0x5627eda29dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/1696371279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda29890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6be2d8e740 1 -- 172.21.15.114:0/1696371279 >> 172.21.15.114:0/1696371279 conn(0x5627ed66c300 msgr2=0x5627eda1bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:17.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6be2d8e740 1 -- 172.21.15.114:0/1696371279 shutdown_connections 2024-05-27T07:52:17.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6be2d8e740 1 -- 172.21.15.114:0/1696371279 wait complete. 2024-05-27T07:52:17.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.142+0000 7f6be2d8e740 20 client.18396 trim_cache size 0 max 16384 2024-05-27T07:52:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6be2d8e740 1 Processor -- start 2024-05-27T07:52:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6be2d8e740 1 -- start start 2024-05-27T07:52:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6be2d8e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda27010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6be2d8e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6574d0 0x5627eda19870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6be2d8e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6781b0 0x5627eda21610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6be2d8e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd41b0bd0 con 0x5627ed6574d0 2024-05-27T07:52:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6be2d8e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd41af360 con 0x5627ed64f2f0 2024-05-27T07:52:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6be2d8e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd41ad1b0 con 0x5627ed6781b0 2024-05-27T07:52:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6bd9474640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6574d0 0x5627eda19870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6bd9474640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6574d0 0x5627eda19870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52554/0 (socket says 172.21.15.114:52554) 2024-05-27T07:52:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6bd9474640 1 -- 172.21.15.114:0/542714414 learned_addr learned my addr 172.21.15.114:0/542714414 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6bd9c75640 1 --2- 172.21.15.114:0/542714414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda27010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6bda476640 1 --2- 172.21.15.114:0/542714414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6781b0 0x5627eda21610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6bd9474640 1 -- 172.21.15.114:0/542714414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6781b0 msgr2=0x5627eda21610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6bd9474640 1 --2- 172.21.15.114:0/542714414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6781b0 0x5627eda21610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6bd9474640 1 -- 172.21.15.114:0/542714414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 msgr2=0x5627eda27010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6bd9474640 1 --2- 172.21.15.114:0/542714414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda27010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.158+0000 7f6bd9474640 1 -- 172.21.15.114:0/542714414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627eda1a580 con 0x5627ed6574d0 2024-05-27T07:52:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.159+0000 7f6bd9474640 1 --2- 172.21.15.114:0/542714414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6574d0 0x5627eda19870 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40a5cb0 tx=0x7f6bc40e68b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e5d4b77c4c18da server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.159+0000 7f6bbf7fe640 1 -- 172.21.15.114:0/542714414 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc40a9a00 con 0x5627ed6574d0 2024-05-27T07:52:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.159+0000 7f6bbf7fe640 1 -- 172.21.15.114:0/542714414 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bc40b23e0 con 0x5627ed6574d0 2024-05-27T07:52:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.159+0000 7f6bbf7fe640 1 -- 172.21.15.114:0/542714414 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc40a53c0 con 0x5627ed6574d0 2024-05-27T07:52:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.160+0000 7f6be2d8e740 1 -- 172.21.15.114:0/542714414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6574d0 msgr2=0x5627eda19870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.160+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/542714414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6574d0 0x5627eda19870 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40a5cb0 tx=0x7f6bc40e68b0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.160+0000 7f6be2d8e740 1 -- 172.21.15.114:0/542714414 shutdown_connections 2024-05-27T07:52:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.160+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/542714414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6781b0 0x5627eda21610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.160+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/542714414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6574d0 0x5627eda19870 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.160+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/542714414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda27010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.160+0000 7f6be2d8e740 1 -- 172.21.15.114:0/542714414 >> 172.21.15.114:0/542714414 conn(0x5627ed66c300 msgr2=0x5627eda19160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.160+0000 7f6be2d8e740 1 -- 172.21.15.114:0/542714414 shutdown_connections 2024-05-27T07:52:17.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.160+0000 7f6be2d8e740 1 -- 172.21.15.114:0/542714414 wait complete. 2024-05-27T07:52:17.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.160+0000 7f6be2d8e740 1 -- start start 2024-05-27T07:52:17.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.160+0000 7f6be2d8e740 1 Processor -- start 2024-05-27T07:52:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6be2d8e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda2ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6be2d8e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 0x5627eda2bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6be2d8e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda2c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6be2d8e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd41ad820 con 0x5627ed6781b0 2024-05-27T07:52:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6be2d8e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd41aec90 con 0x5627ed64f2f0 2024-05-27T07:52:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6be2d8e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd41adef0 con 0x5627ed6574d0 2024-05-27T07:52:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6be2d8e740 10 client.0 _ll_register_callbacks cb 0x7f6bd0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bda476640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda2c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bda476640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda2c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52564/0 (socket says 172.21.15.114:52564) 2024-05-27T07:52:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bda476640 1 -- 172.21.15.114:0/4068951797 learned_addr learned my addr 172.21.15.114:0/4068951797 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bd9474640 1 --2- 172.21.15.114:0/4068951797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 0x5627eda2bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bd9c75640 1 --2- 172.21.15.114:0/4068951797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda2ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bda476640 1 -- 172.21.15.114:0/4068951797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 msgr2=0x5627eda2bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bda476640 1 --2- 172.21.15.114:0/4068951797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 0x5627eda2bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bda476640 1 -- 172.21.15.114:0/4068951797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 msgr2=0x5627eda2ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bda476640 1 --2- 172.21.15.114:0/4068951797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda2ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bda476640 1 -- 172.21.15.114:0/4068951797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627edaa1610 con 0x5627ed6781b0 2024-05-27T07:52:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bda476640 1 --2- 172.21.15.114:0/4068951797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda2c490 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009ca0 tx=0x7f6bcc031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7461fcfdcc3bba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bbd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:52:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bbd7fa640 1 -- 172.21.15.114:0/4068951797 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc030b70 con 0x5627ed6781b0 2024-05-27T07:52:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6bbd7fa640 1 -- 172.21.15.114:0/4068951797 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7f6bcc030d10 con 0x5627ed6781b0 2024-05-27T07:52:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.161+0000 7f6be2d8e740 1 -- 172.21.15.114:0/4068951797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bc40b23e0 con 0x5627ed6781b0 2024-05-27T07:52:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.162+0000 7f6bbd7fa640 1 -- 172.21.15.114:0/4068951797 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f6bcc064050 con 0x5627ed6781b0 2024-05-27T07:52:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.162+0000 7f6be2d8e740 10 client.18402 fetch_fsmap learned FSMap version 69 2024-05-27T07:52:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.162+0000 7f6be2d8e740 1 -- 172.21.15.114:0/4068951797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x5627edaa2850 con 0x5627ed6781b0 2024-05-27T07:52:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.162+0000 7f6bbd7fa640 1 -- 172.21.15.114:0/4068951797 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bc40b23e0 con 0x5627ed6781b0 2024-05-27T07:52:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.162+0000 7f6bbd7fa640 1 -- 172.21.15.114:0/4068951797 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 69) ==== 5910+0+0 (secure 0 0 0) 0x7f6bcc004430 con 0x5627ed6781b0 2024-05-27T07:52:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.162+0000 7f6be2d8e740 10 client.18402 fetch_fsmap finished waiting for FSMap version 69 2024-05-27T07:52:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.162+0000 7f6be2d8e740 10 client.18402 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.162+0000 7f6be2d8e740 20 client.18402 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.162+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/4068951797 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5627ed60c1c0 0x5627edaa4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.162+0000 7f6be2d8e740 4 client.18402 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:52:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.162+0000 7f6be2d8e740 1 -- 172.21.15.114:0/4068951797 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5627ed987360 con 0x5627ed60c1c0 2024-05-27T07:52:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.162+0000 7f6bd9c75640 1 --2- 172.21.15.114:0/4068951797 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5627ed60c1c0 0x5627edaa4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.163+0000 7f6bbd7fa640 10 client.18402 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.163+0000 7f6bd9c75640 1 --2- 172.21.15.114:0/4068951797 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5627ed60c1c0 0x5627edaa4f10 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d136e76d9274010 server_cookie=74107ddffd2fc20c in_seq=0 out_seq=0 2024-05-27T07:52:17.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1207042 9.57 MB/sec execute 509 sec latency 83.988 ms 2024-05-27T07:52:17.479 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:17 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:52:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.490+0000 7f6bbd7fa640 1 -- 172.21.15.114:0/4068951797 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5627ed987360 con 0x5627ed60c1c0 2024-05-27T07:52:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.490+0000 7f6bbd7fa640 10 client.18402 handle_command_reply: tid=0 2024-05-27T07:52:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.490+0000 7f6be2d8e740 1 client.18402 shutdown 2024-05-27T07:52:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.491+0000 7f6be2d8e740 1 -- 172.21.15.114:0/4068951797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 msgr2=0x5627eda2c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.491+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/4068951797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda2c490 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009ca0 tx=0x7f6bcc031770 comp rx=0 tx=0).stop 2024-05-27T07:52:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.491+0000 7f6be2d8e740 1 -- 172.21.15.114:0/4068951797 shutdown_connections 2024-05-27T07:52:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.491+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/4068951797 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5627ed60c1c0 0x5627edaa4f10 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.491+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/4068951797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5627ed6781b0 0x5627eda2c490 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.491+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/4068951797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627ed6574d0 0x5627eda2bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.491+0000 7f6bbd7fa640 0 client.18402 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.491+0000 7f6be2d8e740 1 --2- 172.21.15.114:0/4068951797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627ed64f2f0 0x5627eda2ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.491+0000 7f6be2d8e740 1 -- 172.21.15.114:0/4068951797 >> 172.21.15.114:0/4068951797 conn(0x5627ed66c300 msgr2=0x5627eda26d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.491+0000 7f6be2d8e740 1 -- 172.21.15.114:0/4068951797 shutdown_connections 2024-05-27T07:52:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.491+0000 7f6be2d8e740 1 -- 172.21.15.114:0/4068951797 wait complete. 2024-05-27T07:52:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.491+0000 7f6be2d8e740 20 client.18402 trim_cache size 0 max 16384 2024-05-27T07:52:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:52:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:52:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:52:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.492+0000 7f6bdbed9640 1 -- 172.21.15.114:0/3447502213 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6bc00b2950 msgr2=0x7f6bc00b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.492+0000 7f6bdbed9640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6bc00b2950 0x7f6bc00b4e60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x5627eda16c30 tx=0x7f6bd0005f90 comp rx=0 tx=0).stop 2024-05-27T07:52:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.492+0000 7f6bdbed9640 1 -- 172.21.15.114:0/3447502213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bd4108b80 msgr2=0x7f6bd419bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.492+0000 7f6bdbed9640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bd4108b80 0x7f6bd419bbf0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400e9f0 tx=0x7f6bc400eeb0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.492+0000 7f6bdbed9640 1 -- 172.21.15.114:0/3447502213 shutdown_connections 2024-05-27T07:52:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.492+0000 7f6bdbed9640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6bc00b2950 0x7f6bc00b4e60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.492+0000 7f6bdbed9640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bd4109020 0x7f6bd419c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.492+0000 7f6bdbed9640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bd4108b80 0x7f6bd419bbf0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.492+0000 7f6bdbed9640 1 --2- 172.21.15.114:0/3447502213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bd41032d0 0x7f6bd419b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.492+0000 7f6bdbed9640 1 -- 172.21.15.114:0/3447502213 >> 172.21.15.114:0/3447502213 conn(0x7f6bd40fe380 msgr2=0x7f6bd4105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.492+0000 7f6bdbed9640 1 -- 172.21.15.114:0/3447502213 shutdown_connections 2024-05-27T07:52:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:17.492+0000 7f6bdbed9640 1 -- 172.21.15.114:0/3447502213 wait complete. 2024-05-27T07:52:17.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:17 smithi114 ceph-mon[32038]: pgmap v417: 129 pgs: 129 active+clean; 67 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-05-27T07:52:17.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:17 smithi163 ceph-mon[36451]: pgmap v417: 129 pgs: 129 active+clean; 67 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-05-27T07:52:17.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:17 smithi186 ceph-mon[36647]: pgmap v417: 129 pgs: 129 active+clean; 67 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-05-27T07:52:18.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.162+0000 7fa471824700 1 -- 172.21.15.114:0/2975018584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c1034b0 msgr2=0x7fa46c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:18.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.162+0000 7fa471824700 1 --2- 172.21.15.114:0/2975018584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c1034b0 0x7fa46c1038c0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fa45c0099c0 tx=0x7fa45c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.162+0000 7fa471824700 1 -- 172.21.15.114:0/2975018584 shutdown_connections 2024-05-27T07:52:18.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.162+0000 7fa471824700 1 --2- 172.21.15.114:0/2975018584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa46c108730 0x7fa46c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.162+0000 7fa471824700 1 --2- 172.21.15.114:0/2975018584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c1034b0 0x7fa46c1038c0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.162+0000 7fa471824700 1 --2- 172.21.15.114:0/2975018584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa46c107de0 0x7fa46c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.162+0000 7fa471824700 1 -- 172.21.15.114:0/2975018584 >> 172.21.15.114:0/2975018584 conn(0x7fa46c0fedb0 msgr2=0x7fa46c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:18.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.162+0000 7fa471824700 1 -- 172.21.15.114:0/2975018584 shutdown_connections 2024-05-27T07:52:18.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.162+0000 7fa471824700 1 -- 172.21.15.114:0/2975018584 wait complete. 2024-05-27T07:52:18.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.163+0000 7fa471824700 1 Processor -- start 2024-05-27T07:52:18.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.163+0000 7fa471824700 1 -- start start 2024-05-27T07:52:18.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.163+0000 7fa471824700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa46c1034b0 0x7fa46c19ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.163+0000 7fa471824700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa46c107de0 0x7fa46c19b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.163+0000 7fa471824700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c108730 0x7fa46c19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.163+0000 7fa471824700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1144b0 con 0x7fa46c108730 2024-05-27T07:52:18.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.163+0000 7fa471824700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c114330 con 0x7fa46c107de0 2024-05-27T07:52:18.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.163+0000 7fa471824700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c114630 con 0x7fa46c1034b0 2024-05-27T07:52:18.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa46affd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa46c1034b0 0x7fa46c19ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa46b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c108730 0x7fa46c19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa46affd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa46c1034b0 0x7fa46c19ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:52774/0 (socket says 172.21.15.114:52774) 2024-05-27T07:52:18.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa46b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c108730 0x7fa46c19f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52580/0 (socket says 172.21.15.114:52580) 2024-05-27T07:52:18.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa46affd700 1 -- 172.21.15.114:0/3553417455 learned_addr learned my addr 172.21.15.114:0/3553417455 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:18.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa463fff700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa46c107de0 0x7fa46c19b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa46affd700 1 -- 172.21.15.114:0/3553417455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa46c107de0 msgr2=0x7fa46c19b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa46affd700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa46c107de0 0x7fa46c19b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa46affd700 1 -- 172.21.15.114:0/3553417455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c108730 msgr2=0x7fa46c19f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa46affd700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c108730 0x7fa46c19f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa46affd700 1 -- 172.21.15.114:0/3553417455 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46c19fcb0 con 0x7fa46c1034b0 2024-05-27T07:52:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa46b7fe700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c108730 0x7fa46c19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa46affd700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa46c1034b0 0x7fa46c19ade0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa45c006010 tx=0x7fa45c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa470822700 1 -- 172.21.15.114:0/3553417455 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c026070 con 0x7fa46c1034b0 2024-05-27T07:52:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa471824700 1 -- 172.21.15.114:0/3553417455 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46c19ff40 con 0x7fa46c1034b0 2024-05-27T07:52:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa470822700 1 -- 172.21.15.114:0/3553417455 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa45c003c70 con 0x7fa46c1034b0 2024-05-27T07:52:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa470822700 1 -- 172.21.15.114:0/3553417455 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c00fbe0 con 0x7fa46c1034b0 2024-05-27T07:52:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.164+0000 7fa471824700 1 -- 172.21.15.114:0/3553417455 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa46c1a78c0 con 0x7fa46c1034b0 2024-05-27T07:52:18.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.167+0000 7fa470822700 1 -- 172.21.15.114:0/3553417455 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa45c029070 con 0x7fa46c1034b0 2024-05-27T07:52:18.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa480239b80 1 Processor -- start 2024-05-27T07:52:18.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa470822700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa4580b29f0 0x7fa4580b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa480239b80 1 -- start start 2024-05-27T07:52:18.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa46b7fe700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa4580b29f0 0x7fa4580b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa470822700 1 -- 172.21.15.114:0/3553417455 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7fa45c0e3270 con 0x7fa46c1034b0 2024-05-27T07:52:18.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa480239b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd4d8ec0 0x5565cd65dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa480239b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd65e0f0 0x5565cd657c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa480239b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 0x5565cd6585b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa480239b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5565cd585df0 con 0x5565cd65e0f0 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa480239b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5565cd575990 con 0x5565cd4d8ec0 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa480239b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5565cd5799d0 con 0x5565cd6581a0 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa46affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd4d8ec0 0x5565cd65dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa46affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd4d8ec0 0x5565cd65dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37770/0 (socket says 172.21.15.114:37770) 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa46affd700 1 -- 172.21.15.114:0/3051196991 learned_addr learned my addr 172.21.15.114:0/3051196991 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.168+0000 7fa46b7fe700 1 --2- 172.21.15.114:0/3051196991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 0x5565cd6585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.169+0000 7fa463fff700 1 --2- 172.21.15.114:0/3051196991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd65e0f0 0x5565cd657c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.169+0000 7fa46b7fe700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa4580b29f0 0x7fa4580b4f00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa464003e80 tx=0x7fa464009920 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.169+0000 7fa46affd700 1 -- 172.21.15.114:0/3051196991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 msgr2=0x5565cd6585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.169+0000 7fa46affd700 1 --2- 172.21.15.114:0/3051196991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 0x5565cd6585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.169+0000 7fa46affd700 1 -- 172.21.15.114:0/3051196991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd65e0f0 msgr2=0x5565cd657c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:18.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.169+0000 7fa46affd700 1 --2- 172.21.15.114:0/3051196991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd65e0f0 0x5565cd657c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.169+0000 7fa46affd700 1 -- 172.21.15.114:0/3051196991 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565cd51f030 con 0x5565cd4d8ec0 2024-05-27T07:52:18.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.169+0000 7fa46affd700 1 --2- 172.21.15.114:0/3051196991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd4d8ec0 0x5565cd65dbb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa45c0aa100 tx=0x7fa45c0a9960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8851494e8004b390 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:18.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.169+0000 7fa461ffb700 1 -- 172.21.15.114:0/3051196991 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0aa870 con 0x5565cd4d8ec0 2024-05-27T07:52:18.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.169+0000 7fa461ffb700 1 -- 172.21.15.114:0/3051196991 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa45c0aa170 con 0x5565cd4d8ec0 2024-05-27T07:52:18.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.169+0000 7fa461ffb700 1 -- 172.21.15.114:0/3051196991 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0aa3d0 con 0x5565cd4d8ec0 2024-05-27T07:52:18.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.170+0000 7fa480239b80 1 -- 172.21.15.114:0/3051196991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd4d8ec0 msgr2=0x5565cd65dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:18.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.170+0000 7fa480239b80 1 --2- 172.21.15.114:0/3051196991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd4d8ec0 0x5565cd65dbb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa45c0aa100 tx=0x7fa45c0a9960 comp rx=0 tx=0).stop 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.170+0000 7fa480239b80 1 -- 172.21.15.114:0/3051196991 shutdown_connections 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.170+0000 7fa480239b80 1 --2- 172.21.15.114:0/3051196991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 0x5565cd6585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.170+0000 7fa480239b80 1 --2- 172.21.15.114:0/3051196991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd4d8ec0 0x5565cd65dbb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.170+0000 7fa480239b80 1 --2- 172.21.15.114:0/3051196991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd65e0f0 0x5565cd657c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.170+0000 7fa480239b80 1 -- 172.21.15.114:0/3051196991 >> 172.21.15.114:0/3051196991 conn(0x5565cd4db7f0 msgr2=0x5565cd652e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.170+0000 7fa480239b80 1 -- 172.21.15.114:0/3051196991 shutdown_connections 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.170+0000 7fa480239b80 1 -- 172.21.15.114:0/3051196991 wait complete. 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.170+0000 7fa480239b80 1 -- start start 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa480239b80 1 Processor -- start 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa480239b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 0x5565cd661a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa480239b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 0x5565cd661f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa480239b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd65e0f0 0x5565cd6624b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa480239b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5565cd47c020 con 0x5565cd4d8ec0 2024-05-27T07:52:18.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa480239b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5565cd4dde20 con 0x5565cd65e0f0 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa480239b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5565cd6649a0 con 0x5565cd6581a0 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa480239b80 10 client.0 _ll_register_callbacks cb 0x5565cd50c630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa46affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 0x5565cd661a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa46affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 0x5565cd661a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52604/0 (socket says 172.21.15.114:52604) 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa46affd700 1 -- 172.21.15.114:0/2074517941 learned_addr learned my addr 172.21.15.114:0/2074517941 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa463fff700 1 --2- 172.21.15.114:0/2074517941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 0x5565cd661f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa46b7fe700 1 --2- 172.21.15.114:0/2074517941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd65e0f0 0x5565cd6624b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa46affd700 1 -- 172.21.15.114:0/2074517941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 msgr2=0x5565cd661f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.171+0000 7fa46affd700 1 --2- 172.21.15.114:0/2074517941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 0x5565cd661f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa46affd700 1 -- 172.21.15.114:0/2074517941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd65e0f0 msgr2=0x5565cd6624b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa46affd700 1 --2- 172.21.15.114:0/2074517941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd65e0f0 0x5565cd6624b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa46affd700 1 -- 172.21.15.114:0/2074517941 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565cd4c5950 con 0x5565cd4d8ec0 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa46affd700 1 --2- 172.21.15.114:0/2074517941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 0x5565cd661a30 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fa45c0a73e0 tx=0x7fa45c0a9de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8a948ae0147c45 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa44affd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa44affd700 1 -- 172.21.15.114:0/2074517941 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0aa870 con 0x5565cd4d8ec0 2024-05-27T07:52:18.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa44affd700 1 -- 172.21.15.114:0/2074517941 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7fa45c0aa3d0 con 0x5565cd4d8ec0 2024-05-27T07:52:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa480239b80 1 -- 172.21.15.114:0/2074517941 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa45c0aa170 con 0x5565cd4d8ec0 2024-05-27T07:52:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa44affd700 1 -- 172.21.15.114:0/2074517941 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7fa45c0b2f40 con 0x5565cd4d8ec0 2024-05-27T07:52:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa480239b80 10 client.18414 fetch_fsmap learned FSMap version 69 2024-05-27T07:52:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa480239b80 1 -- 172.21.15.114:0/2074517941 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x5565cd52f7e0 con 0x5565cd4d8ec0 2024-05-27T07:52:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa44affd700 1 -- 172.21.15.114:0/2074517941 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa45c0aa170 con 0x5565cd4d8ec0 2024-05-27T07:52:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa44affd700 1 -- 172.21.15.114:0/2074517941 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 69) ==== 5910+0+0 (secure 0 0 0) 0x7fa45c0a94d0 con 0x5565cd4d8ec0 2024-05-27T07:52:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa480239b80 10 client.18414 fetch_fsmap finished waiting for FSMap version 69 2024-05-27T07:52:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa480239b80 10 client.18414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.172+0000 7fa480239b80 20 client.18414 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.173+0000 7fa480239b80 1 --2- 172.21.15.114:0/2074517941 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5565cd671820 0x5565cd673c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.173+0000 7fa480239b80 4 client.18414 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:52:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.173+0000 7fa480239b80 1 -- 172.21.15.114:0/2074517941 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565cd587d40 con 0x5565cd671820 2024-05-27T07:52:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.173+0000 7fa463fff700 1 --2- 172.21.15.114:0/2074517941 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5565cd671820 0x5565cd673c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.173+0000 7fa44affd700 10 client.18414 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.173+0000 7fa463fff700 1 --2- 172.21.15.114:0/2074517941 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5565cd671820 0x5565cd673c40 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90172d0502bef1ca server_cookie=fb2b4db043499a4b in_seq=0 out_seq=0 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa44affd700 1 -- 172.21.15.114:0/2074517941 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5565cd587d40 con 0x5565cd671820 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa44affd700 10 client.18414 handle_command_reply: tid=0 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa480239b80 1 client.18414 shutdown 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa480239b80 1 -- 172.21.15.114:0/2074517941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 msgr2=0x5565cd661a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa480239b80 1 --2- 172.21.15.114:0/2074517941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 0x5565cd661a30 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fa45c0a73e0 tx=0x7fa45c0a9de0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa480239b80 1 -- 172.21.15.114:0/2074517941 shutdown_connections 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa480239b80 1 --2- 172.21.15.114:0/2074517941 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5565cd671820 0x5565cd673c40 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa480239b80 1 --2- 172.21.15.114:0/2074517941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd65e0f0 0x5565cd6624b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa480239b80 1 --2- 172.21.15.114:0/2074517941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 0x5565cd661a30 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa44affd700 0 client.18414 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa480239b80 1 --2- 172.21.15.114:0/2074517941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 0x5565cd661f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa480239b80 1 -- 172.21.15.114:0/2074517941 >> 172.21.15.114:0/2074517941 conn(0x5565cd4db7f0 msgr2=0x5565cd656c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:18.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa480239b80 1 -- 172.21.15.114:0/2074517941 shutdown_connections 2024-05-27T07:52:18.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa480239b80 1 -- 172.21.15.114:0/2074517941 wait complete. 2024-05-27T07:52:18.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.174+0000 7fa480239b80 20 client.18414 trim_cache size 0 max 16384 2024-05-27T07:52:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.196+0000 7fa480239b80 1 Processor -- start 2024-05-27T07:52:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.196+0000 7fa480239b80 1 -- start start 2024-05-27T07:52:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.196+0000 7fa480239b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 0x5565cd657c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.196+0000 7fa480239b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd6581a0 0x5565cd65bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.196+0000 7fa480239b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd65e0f0 0x5565cd6792c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.196+0000 7fa480239b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1b1f20 con 0x5565cd4d8ec0 2024-05-27T07:52:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.196+0000 7fa480239b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1b0690 con 0x5565cd6581a0 2024-05-27T07:52:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.196+0000 7fa480239b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1ae4e0 con 0x5565cd65e0f0 2024-05-27T07:52:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.196+0000 7fa46affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 0x5565cd657c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.196+0000 7fa46affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 0x5565cd657c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52608/0 (socket says 172.21.15.114:52608) 2024-05-27T07:52:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.196+0000 7fa46affd700 1 -- 172.21.15.114:0/1401509490 learned_addr learned my addr 172.21.15.114:0/1401509490 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:18.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.196+0000 7fa463fff700 1 --2- 172.21.15.114:0/1401509490 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd6581a0 0x5565cd65bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.196+0000 7fa46b7fe700 1 --2- 172.21.15.114:0/1401509490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd65e0f0 0x5565cd6792c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.197+0000 7fa46affd700 1 -- 172.21.15.114:0/1401509490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd65e0f0 msgr2=0x5565cd6792c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:18.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.197+0000 7fa46affd700 1 --2- 172.21.15.114:0/1401509490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd65e0f0 0x5565cd6792c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.197+0000 7fa46affd700 1 -- 172.21.15.114:0/1401509490 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd6581a0 msgr2=0x5565cd65bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:18.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.197+0000 7fa46affd700 1 --2- 172.21.15.114:0/1401509490 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd6581a0 0x5565cd65bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.197+0000 7fa46affd700 1 -- 172.21.15.114:0/1401509490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565cd65d720 con 0x5565cd4d8ec0 2024-05-27T07:52:18.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.197+0000 7fa463fff700 1 --2- 172.21.15.114:0/1401509490 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd6581a0 0x5565cd65bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:18.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.197+0000 7fa46affd700 1 --2- 172.21.15.114:0/1401509490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 0x5565cd657c60 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fa45c0a9e70 tx=0x7fa45c009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec855c0e9ccfb34 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:18.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.197+0000 7fa4617fa700 1 -- 172.21.15.114:0/1401509490 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0aa170 con 0x5565cd4d8ec0 2024-05-27T07:52:18.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.197+0000 7fa4617fa700 1 -- 172.21.15.114:0/1401509490 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa45c0a5c00 con 0x5565cd4d8ec0 2024-05-27T07:52:18.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.197+0000 7fa4617fa700 1 -- 172.21.15.114:0/1401509490 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0aa3d0 con 0x5565cd4d8ec0 2024-05-27T07:52:18.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.198+0000 7fa480239b80 1 -- 172.21.15.114:0/1401509490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 msgr2=0x5565cd657c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:18.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.198+0000 7fa480239b80 1 --2- 172.21.15.114:0/1401509490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 0x5565cd657c60 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fa45c0a9e70 tx=0x7fa45c009670 comp rx=0 tx=0).stop 2024-05-27T07:52:18.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.198+0000 7fa480239b80 1 -- 172.21.15.114:0/1401509490 shutdown_connections 2024-05-27T07:52:18.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.198+0000 7fa480239b80 1 --2- 172.21.15.114:0/1401509490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd65e0f0 0x5565cd6792c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.198+0000 7fa480239b80 1 --2- 172.21.15.114:0/1401509490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd4d8ec0 0x5565cd657c60 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.198+0000 7fa480239b80 1 --2- 172.21.15.114:0/1401509490 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd6581a0 0x5565cd65bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.198+0000 7fa480239b80 1 -- 172.21.15.114:0/1401509490 >> 172.21.15.114:0/1401509490 conn(0x5565cd4db7f0 msgr2=0x5565cd660990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.198+0000 7fa480239b80 1 -- 172.21.15.114:0/1401509490 shutdown_connections 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.198+0000 7fa480239b80 1 -- 172.21.15.114:0/1401509490 wait complete. 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.198+0000 7fa480239b80 1 -- start start 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa480239b80 1 Processor -- start 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa480239b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd4d8ec0 0x5565cd678f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa480239b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 0x5565cd679450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa480239b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd65e0f0 0x5565cd679990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa480239b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1aeb50 con 0x5565cd65e0f0 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa480239b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1affc0 con 0x5565cd4d8ec0 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa480239b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1af220 con 0x5565cd6581a0 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa480239b80 10 client.0 _ll_register_callbacks cb 0x5565cd50c630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa46b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd65e0f0 0x5565cd679990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa46b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd65e0f0 0x5565cd679990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52624/0 (socket says 172.21.15.114:52624) 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa46b7fe700 1 -- 172.21.15.114:0/418315807 learned_addr learned my addr 172.21.15.114:0/418315807 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa46affd700 1 --2- 172.21.15.114:0/418315807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd4d8ec0 0x5565cd678f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa463fff700 1 --2- 172.21.15.114:0/418315807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 0x5565cd679450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa46b7fe700 1 -- 172.21.15.114:0/418315807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 msgr2=0x5565cd679450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa46b7fe700 1 --2- 172.21.15.114:0/418315807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 0x5565cd679450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa46b7fe700 1 -- 172.21.15.114:0/418315807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd4d8ec0 msgr2=0x5565cd678f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa46b7fe700 1 --2- 172.21.15.114:0/418315807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd4d8ec0 0x5565cd678f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.199+0000 7fa46b7fe700 1 -- 172.21.15.114:0/418315807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565cd6db210 con 0x5565cd65e0f0 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa46b7fe700 1 --2- 172.21.15.114:0/418315807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd65e0f0 0x5565cd679990 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fa464019500 tx=0x7fa464004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a628747d3b505ac2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa44affd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa44affd700 1 -- 172.21.15.114:0/418315807 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46401d070 con 0x5565cd65e0f0 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa44affd700 1 -- 172.21.15.114:0/418315807 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7fa46402d480 con 0x5565cd65e0f0 2024-05-27T07:52:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa480239b80 1 -- 172.21.15.114:0/418315807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa45c0a5c00 con 0x5565cd65e0f0 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa44affd700 1 -- 172.21.15.114:0/418315807 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7fa464036050 con 0x5565cd65e0f0 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa480239b80 10 client.18420 fetch_fsmap learned FSMap version 69 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa480239b80 1 -- 172.21.15.114:0/418315807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x5565cd6e0920 con 0x5565cd65e0f0 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa44affd700 1 -- 172.21.15.114:0/418315807 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa45c0a5c00 con 0x5565cd65e0f0 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa44affd700 1 -- 172.21.15.114:0/418315807 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 69) ==== 5910+0+0 (secure 0 0 0) 0x7fa4640176e0 con 0x5565cd65e0f0 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa480239b80 10 client.18420 fetch_fsmap finished waiting for FSMap version 69 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa480239b80 10 client.18420 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa480239b80 20 client.18420 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa480239b80 1 --2- 172.21.15.114:0/418315807 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5565cd6e1090 0x5565cd6e34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa480239b80 4 client.18420 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.200+0000 7fa480239b80 1 -- 172.21.15.114:0/418315807 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5565cd581520 con 0x5565cd6e1090 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.201+0000 7fa46affd700 1 --2- 172.21.15.114:0/418315807 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5565cd6e1090 0x5565cd6e34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.201+0000 7fa44affd700 10 client.18420 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.201+0000 7fa46affd700 1 --2- 172.21.15.114:0/418315807 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5565cd6e1090 0x5565cd6e34b0 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ebaa102c25a36d server_cookie=be31205a9dc4bb96 in_seq=0 out_seq=0 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.203+0000 7fa44affd700 1 -- 172.21.15.114:0/418315807 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5565cd581520 con 0x5565cd6e1090 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.203+0000 7fa44affd700 10 client.18420 handle_command_reply: tid=0 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.203+0000 7fa480239b80 1 client.18420 shutdown 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.204+0000 7fa480239b80 1 -- 172.21.15.114:0/418315807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd65e0f0 msgr2=0x5565cd679990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.204+0000 7fa480239b80 1 --2- 172.21.15.114:0/418315807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd65e0f0 0x5565cd679990 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fa464019500 tx=0x7fa464004b40 comp rx=0 tx=0).stop 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.204+0000 7fa480239b80 1 -- 172.21.15.114:0/418315807 shutdown_connections 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.204+0000 7fa480239b80 1 --2- 172.21.15.114:0/418315807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565cd65e0f0 0x5565cd679990 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.204+0000 7fa480239b80 1 --2- 172.21.15.114:0/418315807 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5565cd6e1090 0x5565cd6e34b0 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.204+0000 7fa480239b80 1 --2- 172.21.15.114:0/418315807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565cd4d8ec0 0x5565cd678f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.204+0000 7fa480239b80 1 --2- 172.21.15.114:0/418315807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565cd6581a0 0x5565cd679450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.204+0000 7fa480239b80 1 -- 172.21.15.114:0/418315807 >> 172.21.15.114:0/418315807 conn(0x5565cd4db7f0 msgr2=0x5565cd6619f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.204+0000 7fa44affd700 0 client.18420 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.204+0000 7fa480239b80 1 -- 172.21.15.114:0/418315807 shutdown_connections 2024-05-27T07:52:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.204+0000 7fa480239b80 1 -- 172.21.15.114:0/418315807 wait complete. 2024-05-27T07:52:18.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.204+0000 7fa480239b80 20 client.18420 trim_cache size 0 max 16384 2024-05-27T07:52:18.208 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:52:18.208 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:52:18.208 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "abae1d54-3c5b-4c75-9550-e12e348cf7c1", 2024-05-27T07:52:18.208 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:52:18.208 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:52:18.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.205+0000 7fa471824700 1 -- 172.21.15.114:0/3553417455 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa4580b29f0 msgr2=0x7fa4580b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:18.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.205+0000 7fa471824700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa4580b29f0 0x7fa4580b4f00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa464003e80 tx=0x7fa464009920 comp rx=0 tx=0).stop 2024-05-27T07:52:18.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.205+0000 7fa471824700 1 -- 172.21.15.114:0/3553417455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa46c1034b0 msgr2=0x7fa46c19ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:18.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.205+0000 7fa471824700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa46c1034b0 0x7fa46c19ade0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa45c006010 tx=0x7fa45c004270 comp rx=0 tx=0).stop 2024-05-27T07:52:18.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.205+0000 7fa471824700 1 -- 172.21.15.114:0/3553417455 shutdown_connections 2024-05-27T07:52:18.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.205+0000 7fa471824700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c108730 0x7fa46c19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.205+0000 7fa471824700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa46c1034b0 0x7fa46c19ade0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.205+0000 7fa471824700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa4580b29f0 0x7fa4580b4f00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.205+0000 7fa471824700 1 --2- 172.21.15.114:0/3553417455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa46c107de0 0x7fa46c19b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:18.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.205+0000 7fa471824700 1 -- 172.21.15.114:0/3553417455 >> 172.21.15.114:0/3553417455 conn(0x7fa46c0fedb0 msgr2=0x7fa46c109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:18.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.205+0000 7fa471824700 1 -- 172.21.15.114:0/3553417455 shutdown_connections 2024-05-27T07:52:18.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:18.205+0000 7fa471824700 1 -- 172.21.15.114:0/3553417455 wait complete. 2024-05-27T07:52:18.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1208121 9.56 MB/sec execute 510 sec latency 160.635 ms 2024-05-27T07:52:18.934 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:18 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:52:18.934 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:18 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:52:18.934 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:18 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:52:18.934 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:18 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:52:18.934 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:18 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:52:18.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:52:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:18 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:52:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:18 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:52:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:18 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:52:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:18 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:52:18.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:18 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:52:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:18 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:52:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:18 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:52:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:18 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:52:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:18 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:52:18.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:18 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:52:19.222 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:52:19.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1209222 9.56 MB/sec execute 511 sec latency 73.880 ms 2024-05-27T07:52:19.972 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:19 smithi114 ceph-mon[32038]: pgmap v418: 129 pgs: 129 active+clean; 67 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.0 MiB/s wr, 134 op/s 2024-05-27T07:52:19.972 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:19 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:52:19.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:19 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:52:19.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:19 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:52:19.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:19 smithi114 ceph-mon[32038]: osdmap e127: 12 total, 12 up, 12 in 2024-05-27T07:52:19.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:19 smithi163 ceph-mon[36451]: pgmap v418: 129 pgs: 129 active+clean; 67 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.0 MiB/s wr, 134 op/s 2024-05-27T07:52:19.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:19 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:52:19.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:19 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:52:19.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:19 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:52:19.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:19 smithi163 ceph-mon[36451]: osdmap e127: 12 total, 12 up, 12 in 2024-05-27T07:52:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:19 smithi186 ceph-mon[36647]: pgmap v418: 129 pgs: 129 active+clean; 67 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.0 MiB/s wr, 134 op/s 2024-05-27T07:52:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:19 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:52:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:19 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:52:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:19 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:52:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:19 smithi186 ceph-mon[36647]: osdmap e127: 12 total, 12 up, 12 in 2024-05-27T07:52:20.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1211247 9.56 MB/sec execute 512 sec latency 40.206 ms 2024-05-27T07:52:20.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:20 smithi163 ceph-mon[36451]: pgmap v419: 129 pgs: 129 active+clean; 67 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 2.2 MiB/s wr, 149 op/s 2024-05-27T07:52:20.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:20 smithi163 ceph-mon[36451]: pgmap v420: 129 pgs: 129 active+clean; 67 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-05-27T07:52:20.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:20 smithi186 ceph-mon[36647]: pgmap v419: 129 pgs: 129 active+clean; 67 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 2.2 MiB/s wr, 149 op/s 2024-05-27T07:52:20.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:20 smithi186 ceph-mon[36647]: pgmap v420: 129 pgs: 129 active+clean; 67 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-05-27T07:52:21.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:20 smithi114 ceph-mon[32038]: pgmap v419: 129 pgs: 129 active+clean; 67 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 2.2 MiB/s wr, 149 op/s 2024-05-27T07:52:21.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:20 smithi114 ceph-mon[32038]: pgmap v420: 129 pgs: 129 active+clean; 67 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-05-27T07:52:21.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1212675 9.56 MB/sec execute 513 sec latency 19.266 ms 2024-05-27T07:52:21.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.340+0000 7f4525439700 1 -- 172.21.15.114:0/2268120265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4520108010 msgr2=0x7f4520108420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:21.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.340+0000 7f4525439700 1 --2- 172.21.15.114:0/2268120265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4520108010 0x7f4520108420 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f45080099c0 tx=0x7f4508009cd0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.340+0000 7f4525439700 1 -- 172.21.15.114:0/2268120265 shutdown_connections 2024-05-27T07:52:21.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.340+0000 7f4525439700 1 --2- 172.21.15.114:0/2268120265 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4520108960 0x7f45201102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.340+0000 7f4525439700 1 --2- 172.21.15.114:0/2268120265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4520103c00 0x7f4520104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.340+0000 7f4525439700 1 --2- 172.21.15.114:0/2268120265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4520108010 0x7f4520108420 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.340+0000 7f4525439700 1 -- 172.21.15.114:0/2268120265 >> 172.21.15.114:0/2268120265 conn(0x7f45200fe510 msgr2=0x7f4520100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:21.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.340+0000 7f4525439700 1 -- 172.21.15.114:0/2268120265 shutdown_connections 2024-05-27T07:52:21.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.340+0000 7f4525439700 1 -- 172.21.15.114:0/2268120265 wait complete. 2024-05-27T07:52:21.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.341+0000 7f4525439700 1 Processor -- start 2024-05-27T07:52:21.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.341+0000 7f4525439700 1 -- start start 2024-05-27T07:52:21.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.341+0000 7f4525439700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4520103c00 0x7f452019bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.341+0000 7f4525439700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4520108010 0x7f452019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.341+0000 7f4525439700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4520108960 0x7f452019c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.341+0000 7f4525439700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4520113570 con 0x7f4520108010 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.341+0000 7f4525439700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f45201133f0 con 0x7f4520103c00 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.341+0000 7f4525439700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45201136f0 con 0x7f4520108960 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.341+0000 7f451e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4520108010 0x7f452019c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.341+0000 7f451e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4520108010 0x7f452019c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52652/0 (socket says 172.21.15.114:52652) 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.341+0000 7f451e7fc700 1 -- 172.21.15.114:0/2841489997 learned_addr learned my addr 172.21.15.114:0/2841489997 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.341+0000 7f451f7fe700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4520108960 0x7f452019c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.341+0000 7f451effd700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4520103c00 0x7f452019bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.342+0000 7f451f7fe700 1 -- 172.21.15.114:0/2841489997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4520103c00 msgr2=0x7f452019bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.342+0000 7f451f7fe700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4520103c00 0x7f452019bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.342+0000 7f451f7fe700 1 -- 172.21.15.114:0/2841489997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4520108010 msgr2=0x7f452019c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:21.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.342+0000 7f451f7fe700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4520108010 0x7f452019c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.342+0000 7f451f7fe700 1 -- 172.21.15.114:0/2841489997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45201a7260 con 0x7f4520108960 2024-05-27T07:52:21.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.342+0000 7f451e7fc700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4520108010 0x7f452019c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:21.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.342+0000 7f451f7fe700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4520108960 0x7f452019c9b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f451800e970 tx=0x7f451800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:21.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.342+0000 7f4517fff700 1 -- 172.21.15.114:0/2841489997 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518004220 con 0x7f4520108960 2024-05-27T07:52:21.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.342+0000 7f4517fff700 1 -- 172.21.15.114:0/2841489997 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45180043c0 con 0x7f4520108960 2024-05-27T07:52:21.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.342+0000 7f4525439700 1 -- 172.21.15.114:0/2841489997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4520199d40 con 0x7f4520108960 2024-05-27T07:52:21.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.342+0000 7f4517fff700 1 -- 172.21.15.114:0/2841489997 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518010700 con 0x7f4520108960 2024-05-27T07:52:21.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.342+0000 7f4525439700 1 -- 172.21.15.114:0/2841489997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f452019a280 con 0x7f4520108960 2024-05-27T07:52:21.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.345+0000 7f4517fff700 1 -- 172.21.15.114:0/2841489997 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f45180108a0 con 0x7f4520108960 2024-05-27T07:52:21.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.347+0000 7f4517fff700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f450c0b2a70 0x7f450c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.347+0000 7f451effd700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f450c0b2a70 0x7f450c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.347+0000 7f4517fff700 1 -- 172.21.15.114:0/2841489997 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9554+0+0 (secure 0 0 0) 0x7f45180e27a0 con 0x7f4520108960 2024-05-27T07:52:21.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.347+0000 7f451effd700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f450c0b2a70 0x7f450c0b4f80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f451000f540 tx=0x7f451000b040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:21.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.347+0000 7f4533e4eb80 1 Processor -- start 2024-05-27T07:52:21.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.347+0000 7f4533e4eb80 1 -- start start 2024-05-27T07:52:21.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f4533e4eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fb3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f4533e4eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fada40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f4533e4eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fadf80 0x55c5c3fae390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f4533e4eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c5c3edcc90 con 0x55c5c3fb3c50 2024-05-27T07:52:21.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f4533e4eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c5c3ecc830 con 0x55c5c3e2fe40 2024-05-27T07:52:21.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f4533e4eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c5c3ed0870 con 0x55c5c3fadf80 2024-05-27T07:52:21.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f451e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f451e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fada40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52662/0 (socket says 172.21.15.114:52662) 2024-05-27T07:52:21.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f451e7fc700 1 -- 172.21.15.114:0/3618169883 learned_addr learned my addr 172.21.15.114:0/3618169883 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:21.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f451f7fe700 1 --2- 172.21.15.114:0/3618169883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fadf80 0x55c5c3fae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f451effd700 1 --2- 172.21.15.114:0/3618169883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fb3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f451e7fc700 1 -- 172.21.15.114:0/3618169883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fadf80 msgr2=0x55c5c3fae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:21.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f451e7fc700 1 --2- 172.21.15.114:0/3618169883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fadf80 0x55c5c3fae390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f451e7fc700 1 -- 172.21.15.114:0/3618169883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 msgr2=0x55c5c3fb3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f451e7fc700 1 --2- 172.21.15.114:0/3618169883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fb3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f451e7fc700 1 -- 172.21.15.114:0/3618169883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5c3e1c8d0 con 0x55c5c3fb3c50 2024-05-27T07:52:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.348+0000 7f451e7fc700 1 --2- 172.21.15.114:0/3618169883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fada40 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f45080036b0 tx=0x7f4508005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1919d5ed7a1ddcb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.349+0000 7f45157fa700 1 -- 172.21.15.114:0/3618169883 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4508027070 con 0x55c5c3fb3c50 2024-05-27T07:52:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.349+0000 7f45157fa700 1 -- 172.21.15.114:0/3618169883 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4508005550 con 0x55c5c3fb3c50 2024-05-27T07:52:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.349+0000 7f45157fa700 1 -- 172.21.15.114:0/3618169883 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4508003cf0 con 0x55c5c3fb3c50 2024-05-27T07:52:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.350+0000 7f4533e4eb80 1 -- 172.21.15.114:0/3618169883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 msgr2=0x55c5c3fada40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.350+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/3618169883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fada40 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f45080036b0 tx=0x7f4508005fb0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.350+0000 7f4533e4eb80 1 -- 172.21.15.114:0/3618169883 shutdown_connections 2024-05-27T07:52:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.350+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/3618169883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fadf80 0x55c5c3fae390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.350+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/3618169883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fb3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.350+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/3618169883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fada40 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.350+0000 7f4533e4eb80 1 -- 172.21.15.114:0/3618169883 >> 172.21.15.114:0/3618169883 conn(0x55c5c3e8aaf0 msgr2=0x55c5c3fa8640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.350+0000 7f4533e4eb80 1 -- 172.21.15.114:0/3618169883 shutdown_connections 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.350+0000 7f4533e4eb80 1 -- 172.21.15.114:0/3618169883 wait complete. 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.350+0000 7f4533e4eb80 1 -- start start 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.350+0000 7f4533e4eb80 1 Processor -- start 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f4533e4eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fb72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f4533e4eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fadf80 0x55c5c3fb7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f4533e4eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fb7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f4533e4eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c5c3dd2fa0 con 0x55c5c3fb3c50 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f4533e4eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c5c3e6e7c0 con 0x55c5c3e2fe40 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f4533e4eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c5c3fba6e0 con 0x55c5c3fadf80 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f4533e4eb80 10 client.0 _ll_register_callbacks cb 0x55c5c3e67920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f451f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fb7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f451effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fb72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f451f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fb7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52676/0 (socket says 172.21.15.114:52676) 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f451f7fe700 1 -- 172.21.15.114:0/4008280051 learned_addr learned my addr 172.21.15.114:0/4008280051 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f451effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fb72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37846/0 (socket says 172.21.15.114:37846) 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f451e7fc700 1 --2- 172.21.15.114:0/4008280051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fadf80 0x55c5c3fb7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f451effd700 1 -- 172.21.15.114:0/4008280051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fadf80 msgr2=0x55c5c3fb7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f451effd700 1 --2- 172.21.15.114:0/4008280051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fadf80 0x55c5c3fb7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f451effd700 1 -- 172.21.15.114:0/4008280051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 msgr2=0x55c5c3fb7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f451effd700 1 --2- 172.21.15.114:0/4008280051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fb7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f451effd700 1 -- 172.21.15.114:0/4008280051 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5c3e6b690 con 0x55c5c3e2fe40 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f451f7fe700 1 --2- 172.21.15.114:0/4008280051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fb7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f451effd700 1 --2- 172.21.15.114:0/4008280051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fb72e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55c5c3faebb0 tx=0x7f4510004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f5a2a7a33cd58b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f45067fc700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.351+0000 7f45067fc700 1 -- 172.21.15.114:0/4008280051 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4510027070 con 0x55c5c3e2fe40 2024-05-27T07:52:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f45067fc700 1 -- 172.21.15.114:0/4008280051 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9554+0+0 (secure 0 0 0) 0x7f4510015cc0 con 0x55c5c3e2fe40 2024-05-27T07:52:21.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f4533e4eb80 1 -- 172.21.15.114:0/4008280051 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4508005550 con 0x55c5c3e2fe40 2024-05-27T07:52:21.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f45067fc700 1 -- 172.21.15.114:0/4008280051 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f4510033050 con 0x55c5c3e2fe40 2024-05-27T07:52:21.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f4533e4eb80 10 client.27842 fetch_fsmap learned FSMap version 69 2024-05-27T07:52:21.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f4533e4eb80 1 -- 172.21.15.114:0/4008280051 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x55c5c3fc5170 con 0x55c5c3e2fe40 2024-05-27T07:52:21.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f45067fc700 1 -- 172.21.15.114:0/4008280051 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4508005550 con 0x55c5c3e2fe40 2024-05-27T07:52:21.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f45067fc700 1 -- 172.21.15.114:0/4008280051 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 69) ==== 5910+0+0 (secure 0 0 0) 0x7f451002dbc0 con 0x55c5c3e2fe40 2024-05-27T07:52:21.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f4533e4eb80 10 client.27842 fetch_fsmap finished waiting for FSMap version 69 2024-05-27T07:52:21.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f4533e4eb80 10 client.27842 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:21.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f4533e4eb80 20 client.27842 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:21.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/4008280051 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c5c3fc7960 0x55c5c3fc9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f4533e4eb80 4 client.27842 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:52:21.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f4533e4eb80 1 -- 172.21.15.114:0/4008280051 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5c3f799f0 con 0x55c5c3fc7960 2024-05-27T07:52:21.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.352+0000 7f451e7fc700 1 --2- 172.21.15.114:0/4008280051 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c5c3fc7960 0x55c5c3fc9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.353+0000 7f451e7fc700 1 --2- 172.21.15.114:0/4008280051 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c5c3fc7960 0x55c5c3fc9e10 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6da6cba00c6b74f8 server_cookie=d0307df4fe7f41b in_seq=0 out_seq=0 2024-05-27T07:52:21.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.353+0000 7f45067fc700 10 client.27842 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:21.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f45067fc700 1 -- 172.21.15.114:0/4008280051 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c5c3f799f0 con 0x55c5c3fc7960 2024-05-27T07:52:21.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f45067fc700 10 client.27842 handle_command_reply: tid=0 2024-05-27T07:52:21.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f4533e4eb80 1 client.27842 shutdown 2024-05-27T07:52:21.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f4533e4eb80 1 -- 172.21.15.114:0/4008280051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 msgr2=0x55c5c3fb72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:21.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/4008280051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fb72e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55c5c3faebb0 tx=0x7f4510004ad0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f4533e4eb80 1 -- 172.21.15.114:0/4008280051 shutdown_connections 2024-05-27T07:52:21.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/4008280051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fb7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/4008280051 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c5c3fc7960 0x55c5c3fc9e10 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/4008280051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fb72e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/4008280051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fadf80 0x55c5c3fb7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f45067fc700 0 client.27842 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:21.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f4533e4eb80 1 -- 172.21.15.114:0/4008280051 >> 172.21.15.114:0/4008280051 conn(0x55c5c3e8aaf0 msgr2=0x55c5c3faca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:21.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f4533e4eb80 1 -- 172.21.15.114:0/4008280051 shutdown_connections 2024-05-27T07:52:21.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f4533e4eb80 1 -- 172.21.15.114:0/4008280051 wait complete. 2024-05-27T07:52:21.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.354+0000 7f4533e4eb80 20 client.27842 trim_cache size 0 max 16384 2024-05-27T07:52:21.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.375+0000 7f4533e4eb80 1 Processor -- start 2024-05-27T07:52:21.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.375+0000 7f4533e4eb80 1 -- start start 2024-05-27T07:52:21.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f4533e4eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fada40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f4533e4eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fadf80 0x55c5c3fb2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f4533e4eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fc37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f4533e4eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f45201b4240 con 0x55c5c3fadf80 2024-05-27T07:52:21.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f4533e4eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f45201b29b0 con 0x55c5c3e2fe40 2024-05-27T07:52:21.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f4533e4eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45201b0820 con 0x55c5c3fb3c50 2024-05-27T07:52:21.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f451e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fadf80 0x55c5c3fb2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f451e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fadf80 0x55c5c3fb2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52692/0 (socket says 172.21.15.114:52692) 2024-05-27T07:52:21.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f451e7fc700 1 -- 172.21.15.114:0/2779045836 learned_addr learned my addr 172.21.15.114:0/2779045836 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:21.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f451f7fe700 1 --2- 172.21.15.114:0/2779045836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fc37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f451effd700 1 --2- 172.21.15.114:0/2779045836 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f451e7fc700 1 -- 172.21.15.114:0/2779045836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fb3c50 msgr2=0x55c5c3fc37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:21.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f451e7fc700 1 --2- 172.21.15.114:0/2779045836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fc37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f451e7fc700 1 -- 172.21.15.114:0/2779045836 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 msgr2=0x55c5c3fada40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:21.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f451e7fc700 1 --2- 172.21.15.114:0/2779045836 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fada40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f451e7fc700 1 -- 172.21.15.114:0/2779045836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5c3fa9ed0 con 0x55c5c3fadf80 2024-05-27T07:52:21.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.376+0000 7f451effd700 1 --2- 172.21.15.114:0/2779045836 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:52:21.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.377+0000 7f451e7fc700 1 --2- 172.21.15.114:0/2779045836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fadf80 0x55c5c3fb2ac0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f4508005b40 tx=0x7f450801a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7273878133bdebee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:21.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.377+0000 7f4514ff9700 1 -- 172.21.15.114:0/2779045836 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4508027040 con 0x55c5c3fadf80 2024-05-27T07:52:21.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.377+0000 7f4514ff9700 1 -- 172.21.15.114:0/2779045836 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4508028040 con 0x55c5c3fadf80 2024-05-27T07:52:21.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.377+0000 7f4514ff9700 1 -- 172.21.15.114:0/2779045836 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450800f6c0 con 0x55c5c3fadf80 2024-05-27T07:52:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.378+0000 7f4533e4eb80 1 -- 172.21.15.114:0/2779045836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fadf80 msgr2=0x55c5c3fb2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.378+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/2779045836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fadf80 0x55c5c3fb2ac0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f4508005b40 tx=0x7f450801a040 comp rx=0 tx=0).stop 2024-05-27T07:52:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.378+0000 7f4533e4eb80 1 -- 172.21.15.114:0/2779045836 shutdown_connections 2024-05-27T07:52:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.378+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/2779045836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fc37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.378+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/2779045836 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.378+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/2779045836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fadf80 0x55c5c3fb2ac0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.378+0000 7f4533e4eb80 1 -- 172.21.15.114:0/2779045836 >> 172.21.15.114:0/2779045836 conn(0x55c5c3e8aaf0 msgr2=0x55c5c3fb23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.378+0000 7f4533e4eb80 1 -- 172.21.15.114:0/2779045836 shutdown_connections 2024-05-27T07:52:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.378+0000 7f4533e4eb80 1 -- 172.21.15.114:0/2779045836 wait complete. 2024-05-27T07:52:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.378+0000 7f4533e4eb80 1 -- start start 2024-05-27T07:52:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.378+0000 7f4533e4eb80 1 Processor -- start 2024-05-27T07:52:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f4533e4eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fbff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f4533e4eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fadf80 0x55c5c3fcec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f4533e4eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fcf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f4533e4eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f45201b0e70 con 0x55c5c3fadf80 2024-05-27T07:52:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f4533e4eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f45201b22e0 con 0x55c5c3e2fe40 2024-05-27T07:52:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f4533e4eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45201b1540 con 0x55c5c3fb3c50 2024-05-27T07:52:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f4533e4eb80 10 client.0 _ll_register_callbacks cb 0x55c5c3e67920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f451effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fbff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f451e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fadf80 0x55c5c3fcec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f451effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fbff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37868/0 (socket says 172.21.15.114:37868) 2024-05-27T07:52:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f451effd700 1 -- 172.21.15.114:0/191578263 learned_addr learned my addr 172.21.15.114:0/191578263 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f451f7fe700 1 --2- 172.21.15.114:0/191578263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fcf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f451e7fc700 1 -- 172.21.15.114:0/191578263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fb3c50 msgr2=0x55c5c3fcf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f451e7fc700 1 --2- 172.21.15.114:0/191578263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fcf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f451e7fc700 1 -- 172.21.15.114:0/191578263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 msgr2=0x55c5c3fbff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f451e7fc700 1 --2- 172.21.15.114:0/191578263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fbff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f451e7fc700 1 -- 172.21.15.114:0/191578263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5c3fcf6e0 con 0x55c5c3fadf80 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f451e7fc700 1 --2- 172.21.15.114:0/191578263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fadf80 0x55c5c3fcec60 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f4508005a80 tx=0x7f4508009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b7fe9ec67e5e92 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f45067fc700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.379+0000 7f45067fc700 1 -- 172.21.15.114:0/191578263 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4508027040 con 0x55c5c3fadf80 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f4533e4eb80 1 -- 172.21.15.114:0/191578263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4508028040 con 0x55c5c3fadf80 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f45067fc700 1 -- 172.21.15.114:0/191578263 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9554+0+0 (secure 0 0 0) 0x7f4508003880 con 0x55c5c3fadf80 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f45067fc700 1 -- 172.21.15.114:0/191578263 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f4508005530 con 0x55c5c3fadf80 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f4533e4eb80 10 client.18438 fetch_fsmap learned FSMap version 69 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f4533e4eb80 1 -- 172.21.15.114:0/191578263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x55c5c4032130 con 0x55c5c3fadf80 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f45067fc700 1 -- 172.21.15.114:0/191578263 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4508028040 con 0x55c5c3fadf80 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f45067fc700 1 -- 172.21.15.114:0/191578263 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 69) ==== 5910+0+0 (secure 0 0 0) 0x7f4508021070 con 0x55c5c3fadf80 2024-05-27T07:52:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f4533e4eb80 10 client.18438 fetch_fsmap finished waiting for FSMap version 69 2024-05-27T07:52:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f4533e4eb80 10 client.18438 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f4533e4eb80 20 client.18438 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/191578263 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c5c4032750 0x55c5c4034b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f4533e4eb80 4 client.18438 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:52:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f4533e4eb80 1 -- 172.21.15.114:0/191578263 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c5c3edebe0 con 0x55c5c4032750 2024-05-27T07:52:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.380+0000 7f451effd700 1 --2- 172.21.15.114:0/191578263 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c5c4032750 0x55c5c4034b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f45067fc700 10 client.18438 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f451effd700 1 --2- 172.21.15.114:0/191578263 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c5c4032750 0x55c5c4034b70 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee4427503fd513a9 server_cookie=3bd11eadcd26a8f4 in_seq=0 out_seq=0 2024-05-27T07:52:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f45067fc700 1 -- 172.21.15.114:0/191578263 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c5c3edebe0 con 0x55c5c4032750 2024-05-27T07:52:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f45067fc700 10 client.18438 handle_command_reply: tid=0 2024-05-27T07:52:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f4533e4eb80 1 client.18438 shutdown 2024-05-27T07:52:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f4533e4eb80 1 -- 172.21.15.114:0/191578263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fadf80 msgr2=0x55c5c3fcec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/191578263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fadf80 0x55c5c3fcec60 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f4508005a80 tx=0x7f4508009cd0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f4533e4eb80 1 -- 172.21.15.114:0/191578263 shutdown_connections 2024-05-27T07:52:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/191578263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c5c3fb3c50 0x55c5c3fcf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/191578263 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55c5c4032750 0x55c5c4034b70 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f45067fc700 0 client.18438 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/191578263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c5c3e2fe40 0x55c5c3fbff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f4533e4eb80 1 --2- 172.21.15.114:0/191578263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5c3fadf80 0x55c5c3fcec60 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.381+0000 7f4533e4eb80 1 -- 172.21.15.114:0/191578263 >> 172.21.15.114:0/191578263 conn(0x55c5c3e8aaf0 msgr2=0x55c5c3fb1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.382+0000 7f4533e4eb80 1 -- 172.21.15.114:0/191578263 shutdown_connections 2024-05-27T07:52:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.382+0000 7f4533e4eb80 1 -- 172.21.15.114:0/191578263 wait complete. 2024-05-27T07:52:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.382+0000 7f4533e4eb80 20 client.18438 trim_cache size 0 max 16384 2024-05-27T07:52:21.387 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:52:21.388 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:52:21.388 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:52:21.388 INFO:teuthology.orchestra.run.smithi114.stdout: "abae1d54-3c5b-4c75-9550-e12e348cf7c1": { 2024-05-27T07:52:21.388 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:52:21.388 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "abae1d54-3c5b-4c75-9550-e12e348cf7c1", 2024-05-27T07:52:21.388 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:52:21.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:52:21.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:52:21.388 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:52:21.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.382+0000 7f4525439700 1 -- 172.21.15.114:0/2841489997 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f450c0b2a70 msgr2=0x7f450c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:21.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.382+0000 7f4525439700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f450c0b2a70 0x7f450c0b4f80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f451000f540 tx=0x7f451000b040 comp rx=0 tx=0).stop 2024-05-27T07:52:21.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.382+0000 7f4525439700 1 -- 172.21.15.114:0/2841489997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4520108960 msgr2=0x7f452019c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:21.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.382+0000 7f4525439700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4520108960 0x7f452019c9b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f451800e970 tx=0x7f451800ec80 comp rx=0 tx=0).stop 2024-05-27T07:52:21.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.382+0000 7f4525439700 1 -- 172.21.15.114:0/2841489997 shutdown_connections 2024-05-27T07:52:21.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.382+0000 7f4525439700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4520108960 0x7f452019c9b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.382+0000 7f4525439700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f450c0b2a70 0x7f450c0b4f80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.382+0000 7f4525439700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4520103c00 0x7f452019bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.382+0000 7f4525439700 1 --2- 172.21.15.114:0/2841489997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4520108010 0x7f452019c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:21.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.382+0000 7f4525439700 1 -- 172.21.15.114:0/2841489997 >> 172.21.15.114:0/2841489997 conn(0x7f45200fe510 msgr2=0x7f45201098c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:21.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.383+0000 7f4525439700 1 -- 172.21.15.114:0/2841489997 shutdown_connections 2024-05-27T07:52:21.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:21.383+0000 7f4525439700 1 -- 172.21.15.114:0/2841489997 wait complete. 2024-05-27T07:52:22.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:21 smithi114 ceph-mon[32038]: pgmap v422: 129 pgs: 129 active+clean; 65 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 353 op/s 2024-05-27T07:52:22.150 INFO:tasks.cephfs.filesystem:scrub status for tag:abae1d54-3c5b-4c75-9550-e12e348cf7c1 - {'path': '/', 'tag': 'abae1d54-3c5b-4c75-9550-e12e348cf7c1', 'options': 'recursive,force'} 2024-05-27T07:52:22.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:21 smithi163 ceph-mon[36451]: pgmap v422: 129 pgs: 129 active+clean; 65 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 353 op/s 2024-05-27T07:52:22.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:21 smithi186 ceph-mon[36647]: pgmap v422: 129 pgs: 129 active+clean; 65 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 353 op/s 2024-05-27T07:52:22.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1214780 9.56 MB/sec execute 514 sec latency 40.890 ms 2024-05-27T07:52:23.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:22 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:52:23.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:22 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:52:23.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:22 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:52:23.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1216149 9.56 MB/sec execute 515 sec latency 25.768 ms 2024-05-27T07:52:24.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:23 smithi114 ceph-mon[32038]: pgmap v423: 129 pgs: 129 active+clean; 65 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 278 op/s 2024-05-27T07:52:24.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:23 smithi163 ceph-mon[36451]: pgmap v423: 129 pgs: 129 active+clean; 65 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 278 op/s 2024-05-27T07:52:24.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:23 smithi186 ceph-mon[36647]: pgmap v423: 129 pgs: 129 active+clean; 65 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 278 op/s 2024-05-27T07:52:24.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1218215 9.56 MB/sec execute 516 sec latency 81.399 ms 2024-05-27T07:52:25.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1219097 9.55 MB/sec execute 517 sec latency 55.904 ms 2024-05-27T07:52:26.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1221559 9.56 MB/sec execute 518 sec latency 37.244 ms 2024-05-27T07:52:26.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:26 smithi114 ceph-mon[32038]: pgmap v424: 129 pgs: 129 active+clean; 65 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 269 op/s 2024-05-27T07:52:26.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:26 smithi163 ceph-mon[36451]: pgmap v424: 129 pgs: 129 active+clean; 65 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 269 op/s 2024-05-27T07:52:26.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:26 smithi186 ceph-mon[36647]: pgmap v424: 129 pgs: 129 active+clean; 65 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 269 op/s 2024-05-27T07:52:27.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1222627 9.55 MB/sec execute 519 sec latency 20.158 ms 2024-05-27T07:52:27.515 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:27 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:52:27.515 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:27 smithi186 systemd[1]: Stopped Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:52:27.515 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:27 smithi186 systemd[1]: Starting Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:52:27.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:27 smithi186 podman[75616]: 2024-05-27 07:52:27.417631701 +0000 UTC m=+0.018938927 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:52:28.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1224672 9.55 MB/sec execute 520 sec latency 42.591 ms 2024-05-27T07:52:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:28 smithi186 ceph-mon[36647]: pgmap v425: 129 pgs: 129 active+clean; 71 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.8 MiB/s wr, 352 op/s 2024-05-27T07:52:28.528 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:28 smithi163 ceph-mon[36451]: pgmap v425: 129 pgs: 129 active+clean; 71 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.8 MiB/s wr, 352 op/s 2024-05-27T07:52:28.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:28 smithi114 ceph-mon[32038]: pgmap v425: 129 pgs: 129 active+clean; 71 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.8 MiB/s wr, 352 op/s 2024-05-27T07:52:28.729 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:28 smithi186 podman[75616]: 2024-05-27 07:52:28.429520066 +0000 UTC m=+1.030827289 container create d28f19d0d8332a64ed907be6c135730c2c9c99c5ed9b6e622583bd24220dfdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-27T07:52:29.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1225775 9.55 MB/sec execute 521 sec latency 165.513 ms 2024-05-27T07:52:30.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1226757 9.54 MB/sec execute 522 sec latency 186.739 ms 2024-05-27T07:52:30.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:30 smithi186 ceph-mon[36647]: pgmap v426: 129 pgs: 129 active+clean; 71 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 281 op/s 2024-05-27T07:52:30.479 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:30 smithi186 podman[75616]: 2024-05-27 07:52:30.192014303 +0000 UTC m=+2.793321524 container init d28f19d0d8332a64ed907be6c135730c2c9c99c5ed9b6e622583bd24220dfdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:52:30.479 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:30 smithi186 podman[75616]: 2024-05-27 07:52:30.195153208 +0000 UTC m=+2.796460431 container start d28f19d0d8332a64ed907be6c135730c2c9c99c5ed9b6e622583bd24220dfdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-27T07:52:30.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:30 smithi186 ceph-mds[75635]: -- 172.21.15.186:0/621160661 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55e04fb8c4e0 con 0x55e04fbd7180 2024-05-27T07:52:30.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:30 smithi186 ceph-mds[75635]: -- 172.21.15.186:0/621160661 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e04fb8fa40 con 0x55e04fbd7180 2024-05-27T07:52:30.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:30 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: starting mds.l at 2024-05-27T07:52:30.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:30 smithi186 bash[75616]: d28f19d0d8332a64ed907be6c135730c2c9c99c5ed9b6e622583bd24220dfdca 2024-05-27T07:52:30.480 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:30 smithi186 systemd[1]: Started Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:52:30.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:30 smithi114 ceph-mon[32038]: pgmap v426: 129 pgs: 129 active+clean; 71 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 281 op/s 2024-05-27T07:52:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.618+0000 7f3645ba8640 1 -- 172.21.15.114:0/1597164371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3640102400 msgr2=0x7f3640102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.618+0000 7f3645ba8640 1 --2- 172.21.15.114:0/1597164371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3640102400 0x7f3640102810 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f3630009900 tx=0x7f363002a410 comp rx=0 tx=0).stop 2024-05-27T07:52:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.618+0000 7f3645ba8640 1 -- 172.21.15.114:0/1597164371 shutdown_connections 2024-05-27T07:52:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.618+0000 7f3645ba8640 1 --2- 172.21.15.114:0/1597164371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3640108f40 0x7f36401106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.618+0000 7f3645ba8640 1 --2- 172.21.15.114:0/1597164371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3640108480 0x7f36401088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.618+0000 7f3645ba8640 1 --2- 172.21.15.114:0/1597164371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3640102400 0x7f3640102810 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.618+0000 7f3645ba8640 1 -- 172.21.15.114:0/1597164371 >> 172.21.15.114:0/1597164371 conn(0x7f36400779d0 msgr2=0x7f3640077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.619+0000 7f3645ba8640 1 -- 172.21.15.114:0/1597164371 shutdown_connections 2024-05-27T07:52:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.619+0000 7f3645ba8640 1 -- 172.21.15.114:0/1597164371 wait complete. 2024-05-27T07:52:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.619+0000 7f3645ba8640 1 Processor -- start 2024-05-27T07:52:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.619+0000 7f3645ba8640 1 -- start start 2024-05-27T07:52:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f3645ba8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3640102400 0x7f36401a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f3645ba8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3640108480 0x7f36401a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f3645ba8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3640108f40 0x7f364019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f3645ba8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3640113e50 con 0x7f3640108f40 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f3645ba8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3640113cd0 con 0x7f3640102400 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f3645ba8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3640113fd0 con 0x7f3640108480 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f363ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3640108f40 0x7f364019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f363ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3640108f40 0x7f364019a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42992/0 (socket says 172.21.15.114:42992) 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f363ffff640 1 -- 172.21.15.114:0/2845444608 learned_addr learned my addr 172.21.15.114:0/2845444608 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f363effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3640108480 0x7f36401a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f363f7fe640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3640102400 0x7f36401a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f363ffff640 1 -- 172.21.15.114:0/2845444608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3640108480 msgr2=0x7f36401a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f363ffff640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3640108480 0x7f36401a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f363ffff640 1 -- 172.21.15.114:0/2845444608 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3640102400 msgr2=0x7f36401a0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f363ffff640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3640102400 0x7f36401a0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f363ffff640 1 -- 172.21.15.114:0/2845444608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f364019ade0 con 0x7f3640108f40 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f363f7fe640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3640102400 0x7f36401a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:52:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f363ffff640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3640108f40 0x7f364019a6b0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f363800eb10 tx=0x7f363800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f363cff9640 1 -- 172.21.15.114:0/2845444608 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3638004b20 con 0x7f3640108f40 2024-05-27T07:52:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.620+0000 7f3645ba8640 1 -- 172.21.15.114:0/2845444608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f364019b070 con 0x7f3640108f40 2024-05-27T07:52:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.621+0000 7f363cff9640 1 -- 172.21.15.114:0/2845444608 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3638004cc0 con 0x7f3640108f40 2024-05-27T07:52:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.621+0000 7f3645ba8640 1 -- 172.21.15.114:0/2845444608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36401a72d0 con 0x7f3640108f40 2024-05-27T07:52:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.621+0000 7f363cff9640 1 -- 172.21.15.114:0/2845444608 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3638010690 con 0x7f3640108f40 2024-05-27T07:52:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.623+0000 7f363cff9640 1 -- 172.21.15.114:0/2845444608 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f36380108f0 con 0x7f3640108f40 2024-05-27T07:52:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.623+0000 7f364ca59740 1 Processor -- start 2024-05-27T07:52:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.623+0000 7f364ca59740 1 -- start start 2024-05-27T07:52:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.623+0000 7f364ca59740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310531b0 0x5633313f4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.623+0000 7f364ca59740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56333102a2f0 0x5633313f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.623+0000 7f364ca59740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633310324d0 0x5633313f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.623+0000 7f364ca59740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563331366c30 con 0x5633310324d0 2024-05-27T07:52:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.623+0000 7f364ca59740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5633313567d0 con 0x56333102a2f0 2024-05-27T07:52:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.623+0000 7f364ca59740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56333135a810 con 0x5633310531b0 2024-05-27T07:52:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633310324d0 0x5633313f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633310324d0 0x5633313f4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42996/0 (socket says 172.21.15.114:42996) 2024-05-27T07:52:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363effd640 1 -- 172.21.15.114:0/3622907537 learned_addr learned my addr 172.21.15.114:0/3622907537 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363f7fe640 1 --2- 172.21.15.114:0/3622907537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56333102a2f0 0x5633313f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363effd640 1 --2- 172.21.15.114:0/3622907537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310531b0 0x5633313f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363effd640 1 -- 172.21.15.114:0/3622907537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310531b0 msgr2=0x5633313f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363effd640 1 --2- 172.21.15.114:0/3622907537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310531b0 0x5633313f4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363effd640 1 -- 172.21.15.114:0/3622907537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56333102a2f0 msgr2=0x5633313f47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363effd640 1 --2- 172.21.15.114:0/3622907537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56333102a2f0 0x5633313f47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363effd640 1 -- 172.21.15.114:0/3622907537 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56333106b210 con 0x5633310324d0 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363cff9640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f36280b29d0 0x7f36280b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363f7fe640 1 --2- 172.21.15.114:0/3622907537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56333102a2f0 0x5633313f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363f7fe640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f36280b29d0 0x7f36280b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363effd640 1 --2- 172.21.15.114:0/3622907537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633310324d0 0x5633313f4ce0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f36340058c0 tx=0x7f3634005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224f2e8eefb7072e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f363cff9640 1 -- 172.21.15.114:0/2845444608 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9554+0+0 (secure 0 0 0) 0x7f36380e2f30 con 0x7f3640108f40 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f3625ffb640 1 -- 172.21.15.114:0/3622907537 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363400f840 con 0x5633310324d0 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f3625ffb640 1 -- 172.21.15.114:0/3622907537 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f363400fe90 con 0x5633310324d0 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.624+0000 7f3625ffb640 1 -- 172.21.15.114:0/3622907537 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363401fca0 con 0x5633310324d0 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.625+0000 7f364ca59740 1 -- 172.21.15.114:0/3622907537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633310324d0 msgr2=0x5633313f4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.625+0000 7f364ca59740 1 --2- 172.21.15.114:0/3622907537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633310324d0 0x5633313f4ce0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f36340058c0 tx=0x7f3634005850 comp rx=0 tx=0).stop 2024-05-27T07:52:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.625+0000 7f363f7fe640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f36280b29d0 0x7f36280b4ee0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3630031440 tx=0x7f36300058b0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.625+0000 7f364ca59740 1 -- 172.21.15.114:0/3622907537 shutdown_connections 2024-05-27T07:52:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.625+0000 7f364ca59740 1 --2- 172.21.15.114:0/3622907537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633310324d0 0x5633313f4ce0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.625+0000 7f364ca59740 1 --2- 172.21.15.114:0/3622907537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56333102a2f0 0x5633313f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.625+0000 7f364ca59740 1 --2- 172.21.15.114:0/3622907537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310531b0 0x5633313f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.625+0000 7f364ca59740 1 -- 172.21.15.114:0/3622907537 >> 172.21.15.114:0/3622907537 conn(0x563331047300 msgr2=0x5633313f0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.625+0000 7f364ca59740 1 -- 172.21.15.114:0/3622907537 shutdown_connections 2024-05-27T07:52:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.625+0000 7f364ca59740 1 -- 172.21.15.114:0/3622907537 wait complete. 2024-05-27T07:52:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.626+0000 7f364ca59740 1 -- start start 2024-05-27T07:52:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.626+0000 7f364ca59740 1 Processor -- start 2024-05-27T07:52:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.626+0000 7f364ca59740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x5633314048e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.626+0000 7f364ca59740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5633310324d0 0x563331404e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.626+0000 7f364ca59740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310531b0 0x563331402d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.626+0000 7f364ca59740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56333103c790 con 0x56333102a2f0 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.626+0000 7f364ca59740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563331312890 con 0x5633310324d0 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.626+0000 7f364ca59740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563331405970 con 0x5633310531b0 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.626+0000 7f363f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x5633314048e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.626+0000 7f363f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x5633314048e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43010/0 (socket says 172.21.15.114:43010) 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.626+0000 7f363f7fe640 1 -- 172.21.15.114:0/338970823 learned_addr learned my addr 172.21.15.114:0/338970823 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.626+0000 7f364ca59740 10 client.0 _ll_register_callbacks cb 0x56333106aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f363ffff640 1 --2- 172.21.15.114:0/338970823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310531b0 0x563331402d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f363effd640 1 --2- 172.21.15.114:0/338970823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5633310324d0 0x563331404e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f363f7fe640 1 -- 172.21.15.114:0/338970823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310531b0 msgr2=0x563331402d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f363f7fe640 1 --2- 172.21.15.114:0/338970823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310531b0 0x563331402d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f363f7fe640 1 -- 172.21.15.114:0/338970823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5633310324d0 msgr2=0x563331404e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f363f7fe640 1 --2- 172.21.15.114:0/338970823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5633310324d0 0x563331404e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f363f7fe640 1 -- 172.21.15.114:0/338970823 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563331137620 con 0x56333102a2f0 2024-05-27T07:52:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f363f7fe640 1 --2- 172.21.15.114:0/338970823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x5633314048e0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f3630009900 tx=0x7f3630009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d287b26c4e55b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f361effd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f361effd640 1 -- 172.21.15.114:0/338970823 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363004c070 con 0x56333102a2f0 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f364ca59740 1 -- 172.21.15.114:0/338970823 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f363400fe90 con 0x56333102a2f0 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f361effd640 1 -- 172.21.15.114:0/338970823 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9554+0+0 (secure 0 0 0) 0x7f3630005250 con 0x56333102a2f0 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f361effd640 1 -- 172.21.15.114:0/338970823 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f3630058050 con 0x56333102a2f0 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f364ca59740 10 client.18456 fetch_fsmap learned FSMap version 69 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.627+0000 7f364ca59740 1 -- 172.21.15.114:0/338970823 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x563331411cd0 con 0x56333102a2f0 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.628+0000 7f361effd640 1 -- 172.21.15.114:0/338970823 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f363400fe90 con 0x56333102a2f0 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.628+0000 7f361effd640 1 -- 172.21.15.114:0/338970823 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 69) ==== 5910+0+0 (secure 0 0 0) 0x7f3630052bc0 con 0x56333102a2f0 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.628+0000 7f364ca59740 10 client.18456 fetch_fsmap finished waiting for FSMap version 69 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.628+0000 7f364ca59740 10 client.18456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.628+0000 7f364ca59740 20 client.18456 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.628+0000 7f364ca59740 1 --2- 172.21.15.114:0/338970823 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563331011d80 0x5633314154c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.628+0000 7f364ca59740 4 client.18456 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:52:30.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.628+0000 7f364ca59740 1 -- 172.21.15.114:0/338970823 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563331368b80 con 0x563331011d80 2024-05-27T07:52:30.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.628+0000 7f363effd640 1 --2- 172.21.15.114:0/338970823 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563331011d80 0x5633314154c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.628+0000 7f361effd640 10 client.18456 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:30.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.629+0000 7f363effd640 1 --2- 172.21.15.114:0/338970823 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563331011d80 0x5633314154c0 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c62bba19a903cba server_cookie=712c43175d2227a0 in_seq=0 out_seq=0 2024-05-27T07:52:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.629+0000 7f361effd640 1 -- 172.21.15.114:0/338970823 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f363402ae00 con 0x563331011d80 2024-05-27T07:52:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.629+0000 7f361effd640 10 client.18456 handle_command_reply: tid=0 2024-05-27T07:52:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.629+0000 7f364ca59740 1 client.18456 shutdown 2024-05-27T07:52:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.630+0000 7f364ca59740 1 -- 172.21.15.114:0/338970823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 msgr2=0x5633314048e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.630+0000 7f364ca59740 1 --2- 172.21.15.114:0/338970823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x5633314048e0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f3630009900 tx=0x7f3630009260 comp rx=0 tx=0).stop 2024-05-27T07:52:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.630+0000 7f364ca59740 1 -- 172.21.15.114:0/338970823 shutdown_connections 2024-05-27T07:52:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.630+0000 7f364ca59740 1 --2- 172.21.15.114:0/338970823 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563331011d80 0x5633314154c0 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.630+0000 7f364ca59740 1 --2- 172.21.15.114:0/338970823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310531b0 0x563331402d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.630+0000 7f361effd640 0 client.18456 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.630+0000 7f364ca59740 1 --2- 172.21.15.114:0/338970823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5633310324d0 0x563331404e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.630+0000 7f364ca59740 1 --2- 172.21.15.114:0/338970823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x5633314048e0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.630+0000 7f364ca59740 1 -- 172.21.15.114:0/338970823 >> 172.21.15.114:0/338970823 conn(0x563331047300 msgr2=0x5633313f60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.630+0000 7f364ca59740 1 -- 172.21.15.114:0/338970823 shutdown_connections 2024-05-27T07:52:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.630+0000 7f364ca59740 1 -- 172.21.15.114:0/338970823 wait complete. 2024-05-27T07:52:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.630+0000 7f364ca59740 20 client.18456 trim_cache size 0 max 16384 2024-05-27T07:52:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f364ca59740 1 Processor -- start 2024-05-27T07:52:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f364ca59740 1 -- start start 2024-05-27T07:52:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f364ca59740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563331011d80 0x563331404b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f364ca59740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x5633313fc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f364ca59740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310324d0 0x5633313fcb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f364ca59740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f36401b0be0 con 0x56333102a2f0 2024-05-27T07:52:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f364ca59740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f36401af370 con 0x563331011d80 2024-05-27T07:52:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f364ca59740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36401ad1e0 con 0x5633310324d0 2024-05-27T07:52:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f363effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x5633313fc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f363effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x5633313fc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43018/0 (socket says 172.21.15.114:43018) 2024-05-27T07:52:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f363effd640 1 -- 172.21.15.114:0/2021922062 learned_addr learned my addr 172.21.15.114:0/2021922062 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f363ffff640 1 --2- 172.21.15.114:0/2021922062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310324d0 0x5633313fcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f363f7fe640 1 --2- 172.21.15.114:0/2021922062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563331011d80 0x563331404b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f363effd640 1 -- 172.21.15.114:0/2021922062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310324d0 msgr2=0x5633313fcb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f363effd640 1 --2- 172.21.15.114:0/2021922062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310324d0 0x5633313fcb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f363effd640 1 -- 172.21.15.114:0/2021922062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563331011d80 msgr2=0x563331404b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f363effd640 1 --2- 172.21.15.114:0/2021922062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563331011d80 0x563331404b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f363effd640 1 -- 172.21.15.114:0/2021922062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633314012b0 con 0x56333102a2f0 2024-05-27T07:52:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f363f7fe640 1 --2- 172.21.15.114:0/2021922062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563331011d80 0x563331404b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:52:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.647+0000 7f363effd640 1 --2- 172.21.15.114:0/2021922062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x5633313fc610 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f363400f790 tx=0x7f3634011500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1883881c64e42ff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.648+0000 7f36257fa640 1 -- 172.21.15.114:0/2021922062 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363401f920 con 0x56333102a2f0 2024-05-27T07:52:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.648+0000 7f36257fa640 1 -- 172.21.15.114:0/2021922062 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f363402f070 con 0x56333102a2f0 2024-05-27T07:52:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.648+0000 7f36257fa640 1 -- 172.21.15.114:0/2021922062 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363400f840 con 0x56333102a2f0 2024-05-27T07:52:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.648+0000 7f364ca59740 1 -- 172.21.15.114:0/2021922062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 msgr2=0x5633313fc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.648+0000 7f364ca59740 1 --2- 172.21.15.114:0/2021922062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x5633313fc610 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f363400f790 tx=0x7f3634011500 comp rx=0 tx=0).stop 2024-05-27T07:52:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.649+0000 7f364ca59740 1 -- 172.21.15.114:0/2021922062 shutdown_connections 2024-05-27T07:52:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.649+0000 7f364ca59740 1 --2- 172.21.15.114:0/2021922062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633310324d0 0x5633313fcb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.649+0000 7f364ca59740 1 --2- 172.21.15.114:0/2021922062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x5633313fc610 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.649+0000 7f364ca59740 1 --2- 172.21.15.114:0/2021922062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563331011d80 0x563331404b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.649+0000 7f364ca59740 1 -- 172.21.15.114:0/2021922062 >> 172.21.15.114:0/2021922062 conn(0x563331047300 msgr2=0x563331404480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.649+0000 7f364ca59740 1 -- 172.21.15.114:0/2021922062 shutdown_connections 2024-05-27T07:52:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.649+0000 7f364ca59740 1 -- 172.21.15.114:0/2021922062 wait complete. 2024-05-27T07:52:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.649+0000 7f364ca59740 1 -- start start 2024-05-27T07:52:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.649+0000 7f364ca59740 1 Processor -- start 2024-05-27T07:52:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.649+0000 7f364ca59740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563331011d80 0x56333140ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.649+0000 7f364ca59740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x56333140b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f364ca59740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5633310324d0 0x563331408010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f364ca59740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f36401ad830 con 0x56333102a2f0 2024-05-27T07:52:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f364ca59740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f36401aeca0 con 0x5633310324d0 2024-05-27T07:52:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f364ca59740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36401adf00 con 0x563331011d80 2024-05-27T07:52:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f363effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x56333140b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f364ca59740 10 client.0 _ll_register_callbacks cb 0x7f36380093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f363effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x56333140b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43032/0 (socket says 172.21.15.114:43032) 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f363effd640 1 -- 172.21.15.114:0/1287513747 learned_addr learned my addr 172.21.15.114:0/1287513747 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f363f7fe640 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563331011d80 0x56333140ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f363ffff640 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5633310324d0 0x563331408010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f363effd640 1 -- 172.21.15.114:0/1287513747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563331011d80 msgr2=0x56333140ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f363effd640 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563331011d80 0x56333140ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f363effd640 1 -- 172.21.15.114:0/1287513747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5633310324d0 msgr2=0x563331408010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f363effd640 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5633310324d0 0x563331408010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f363effd640 1 -- 172.21.15.114:0/1287513747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563331408650 con 0x56333102a2f0 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f363ffff640 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5633310324d0 0x563331408010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f363f7fe640 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563331011d80 0x56333140ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f363effd640 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x56333140b070 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f3634009b10 tx=0x7f3634009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79a245673298e06 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f361effd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:52:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f361effd640 1 -- 172.21.15.114:0/1287513747 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3634005520 con 0x56333102a2f0 2024-05-27T07:52:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f364ca59740 1 -- 172.21.15.114:0/1287513747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f363402f070 con 0x56333102a2f0 2024-05-27T07:52:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.650+0000 7f361effd640 1 -- 172.21.15.114:0/1287513747 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9554+0+0 (secure 0 0 0) 0x7f36340050b0 con 0x56333102a2f0 2024-05-27T07:52:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.651+0000 7f361effd640 1 -- 172.21.15.114:0/1287513747 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f363403d950 con 0x56333102a2f0 2024-05-27T07:52:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.651+0000 7f364ca59740 10 client.18462 fetch_fsmap learned FSMap version 69 2024-05-27T07:52:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.651+0000 7f364ca59740 1 -- 172.21.15.114:0/1287513747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x56333147d3e0 con 0x56333102a2f0 2024-05-27T07:52:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.651+0000 7f361effd640 1 -- 172.21.15.114:0/1287513747 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f363402f070 con 0x56333102a2f0 2024-05-27T07:52:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.651+0000 7f361effd640 1 -- 172.21.15.114:0/1287513747 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 69) ==== 5910+0+0 (secure 0 0 0) 0x7f363401d020 con 0x56333102a2f0 2024-05-27T07:52:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.651+0000 7f364ca59740 10 client.18462 fetch_fsmap finished waiting for FSMap version 69 2024-05-27T07:52:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.651+0000 7f364ca59740 10 client.18462 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.651+0000 7f364ca59740 20 client.18462 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.651+0000 7f364ca59740 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5633310531b0 0x56333147fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.651+0000 7f364ca59740 4 client.18462 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:52:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.651+0000 7f364ca59740 1 -- 172.21.15.114:0/1287513747 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563331362360 con 0x5633310531b0 2024-05-27T07:52:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.651+0000 7f363f7fe640 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5633310531b0 0x56333147fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.652+0000 7f361effd640 10 client.18462 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:30.652+0000 7f363f7fe640 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5633310531b0 0x56333147fa10 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=824dff5a943970dd server_cookie=78b3f175f894f22a in_seq=0 out_seq=0 2024-05-27T07:52:30.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:30 smithi163 ceph-mon[36451]: pgmap v426: 129 pgs: 129 active+clean; 71 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 281 op/s 2024-05-27T07:52:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f361effd640 1 -- 172.21.15.114:0/1287513747 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x563331362360 con 0x5633310531b0 2024-05-27T07:52:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f361effd640 10 client.18462 handle_command_reply: tid=0 2024-05-27T07:52:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f364ca59740 1 client.18462 shutdown 2024-05-27T07:52:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f364ca59740 1 -- 172.21.15.114:0/1287513747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 msgr2=0x56333140b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f364ca59740 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x56333140b070 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f3634009b10 tx=0x7f3634009cc0 comp rx=0 tx=0).stop 2024-05-27T07:52:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f364ca59740 1 -- 172.21.15.114:0/1287513747 shutdown_connections 2024-05-27T07:52:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f364ca59740 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5633310531b0 0x56333147fa10 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f364ca59740 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5633310324d0 0x563331408010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f364ca59740 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56333102a2f0 0x56333140b070 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f364ca59740 1 --2- 172.21.15.114:0/1287513747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563331011d80 0x56333140ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f361effd640 0 client.18462 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f364ca59740 1 -- 172.21.15.114:0/1287513747 >> 172.21.15.114:0/1287513747 conn(0x563331047300 msgr2=0x5633314022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f364ca59740 1 -- 172.21.15.114:0/1287513747 shutdown_connections 2024-05-27T07:52:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f364ca59740 1 -- 172.21.15.114:0/1287513747 wait complete. 2024-05-27T07:52:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.050+0000 7f364ca59740 20 client.18462 trim_cache size 0 max 16384 2024-05-27T07:52:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:52:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:52:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:52:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.051+0000 7f3645ba8640 1 -- 172.21.15.114:0/2845444608 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f36280b29d0 msgr2=0x7f36280b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.051+0000 7f3645ba8640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f36280b29d0 0x7f36280b4ee0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3630031440 tx=0x7f36300058b0 comp rx=0 tx=0).stop 2024-05-27T07:52:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.051+0000 7f3645ba8640 1 -- 172.21.15.114:0/2845444608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3640108f40 msgr2=0x7f364019a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.051+0000 7f3645ba8640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3640108f40 0x7f364019a6b0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f363800eb10 tx=0x7f363800c2d0 comp rx=0 tx=0).stop 2024-05-27T07:52:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.051+0000 7f3645ba8640 1 -- 172.21.15.114:0/2845444608 shutdown_connections 2024-05-27T07:52:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.051+0000 7f3645ba8640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f36280b29d0 0x7f36280b4ee0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.051+0000 7f3645ba8640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3640108f40 0x7f364019a6b0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.051+0000 7f3645ba8640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3640108480 0x7f36401a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.051+0000 7f3645ba8640 1 --2- 172.21.15.114:0/2845444608 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3640102400 0x7f36401a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.051+0000 7f3645ba8640 1 -- 172.21.15.114:0/2845444608 >> 172.21.15.114:0/2845444608 conn(0x7f36400779d0 msgr2=0x7f3640100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.051+0000 7f3645ba8640 1 -- 172.21.15.114:0/2845444608 shutdown_connections 2024-05-27T07:52:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:31.051+0000 7f3645ba8640 1 -- 172.21.15.114:0/2845444608 wait complete. 2024-05-27T07:52:31.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1228281 9.53 MB/sec execute 523 sec latency 152.314 ms 2024-05-27T07:52:31.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:31 smithi114 ceph-mon[32038]: daemon mds.l restarted 2024-05-27T07:52:31.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:31 smithi163 ceph-mon[36451]: daemon mds.l restarted 2024-05-27T07:52:31.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:31 smithi186 ceph-mon[36647]: daemon mds.l restarted 2024-05-27T07:52:32.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1229566 9.53 MB/sec execute 524 sec latency 15.085 ms 2024-05-27T07:52:32.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:32 smithi114 ceph-mon[32038]: pgmap v427: 129 pgs: 129 active+clean; 76 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 327 op/s 2024-05-27T07:52:32.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:32 smithi114 ceph-mon[32038]: osdmap e128: 12 total, 12 up, 12 in 2024-05-27T07:52:32.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:32 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6832/1217442030,v1:172.21.15.186:6833/1217442030] up:boot 2024-05-27T07:52:32.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:32 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:52:32.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:52:32.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:32 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:52:32.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:32 smithi163 ceph-mon[36451]: pgmap v427: 129 pgs: 129 active+clean; 76 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 327 op/s 2024-05-27T07:52:32.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:32 smithi163 ceph-mon[36451]: osdmap e128: 12 total, 12 up, 12 in 2024-05-27T07:52:32.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:32 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6832/1217442030,v1:172.21.15.186:6833/1217442030] up:boot 2024-05-27T07:52:32.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:32 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:52:32.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:52:32.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:32 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:52:32.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:32 smithi186 ceph-mon[36647]: pgmap v427: 129 pgs: 129 active+clean; 76 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 327 op/s 2024-05-27T07:52:32.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:32 smithi186 ceph-mon[36647]: osdmap e128: 12 total, 12 up, 12 in 2024-05-27T07:52:32.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:32 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6832/1217442030,v1:172.21.15.186:6833/1217442030] up:boot 2024-05-27T07:52:32.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:32 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:52:32.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:52:32.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:32 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:52:33.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1231451 9.53 MB/sec execute 525 sec latency 23.570 ms 2024-05-27T07:52:34.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1233059 9.53 MB/sec execute 526 sec latency 40.803 ms 2024-05-27T07:52:34.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:34 smithi114 ceph-mon[32038]: pgmap v429: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 216 op/s 2024-05-27T07:52:34.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:34 smithi163 ceph-mon[36451]: pgmap v429: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 216 op/s 2024-05-27T07:52:34.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:34 smithi186 ceph-mon[36647]: pgmap v429: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 216 op/s 2024-05-27T07:52:34.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f37f4c99700 time 2024-05-27T07:52:34.379932+0000 2024-05-27T07:52:34.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:52:34.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:34.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f38074deba6] 2024-05-27T07:52:34.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f38074dedd1] 2024-05-27T07:52:34.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e04ea0623d] 2024-05-27T07:52:34.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e04e9f1971] 2024-05-27T07:52:34.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e04e9effa0] 2024-05-27T07:52:34.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e04e9f0651] 2024-05-27T07:52:34.730 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e04e985347] 2024-05-27T07:52:34.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e04e68187e] 2024-05-27T07:52:34.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e04e686f2f] 2024-05-27T07:52:34.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e04e691969] 2024-05-27T07:52:34.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e04ea2a48c] 2024-05-27T07:52:34.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e04ea339cb] 2024-05-27T07:52:34.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 13: (Thread::entry_wrapper()+0x43) [0x7f38074ad72d] 2024-05-27T07:52:34.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 14: (Thread::_entry_func(void*)+0xd) [0x7f38074ad749] 2024-05-27T07:52:34.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f38060511ca] 2024-05-27T07:52:34.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 16: clone() 2024-05-27T07:52:34.731 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.381+0000 7f37f4c99700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f37f4c99700 time 2024-05-27T07:52:34.379932+0000 2024-05-27T07:52:34.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:52:34.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:34.732 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f38074deba6] 2024-05-27T07:52:34.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f38074dedd1] 2024-05-27T07:52:34.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e04ea0623d] 2024-05-27T07:52:34.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e04e9f1971] 2024-05-27T07:52:34.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e04e9effa0] 2024-05-27T07:52:34.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e04e9f0651] 2024-05-27T07:52:34.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e04e985347] 2024-05-27T07:52:34.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e04e68187e] 2024-05-27T07:52:34.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e04e686f2f] 2024-05-27T07:52:34.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e04e691969] 2024-05-27T07:52:34.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e04ea2a48c] 2024-05-27T07:52:34.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e04ea339cb] 2024-05-27T07:52:34.733 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 13: (Thread::entry_wrapper()+0x43) [0x7f38074ad72d] 2024-05-27T07:52:34.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 14: (Thread::_entry_func(void*)+0xd) [0x7f38074ad749] 2024-05-27T07:52:34.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f38060511ca] 2024-05-27T07:52:34.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 16: clone() 2024-05-27T07:52:34.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: *** Caught signal (Aborted) ** 2024-05-27T07:52:34.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: in thread 7f37f4c99700 thread_name:md_log_replay 2024-05-27T07:52:34.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:34.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f380605bd20] 2024-05-27T07:52:34.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2: gsignal() 2024-05-27T07:52:34.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 3: abort() 2024-05-27T07:52:34.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f38074deced] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f38074dedd1] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e04ea0623d] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e04e9f1971] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e04e9effa0] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e04e9f0651] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e04e985347] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e04e68187e] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e04e686f2f] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e04e691969] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e04ea2a48c] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e04ea339cb] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 16: (Thread::entry_wrapper()+0x43) [0x7f38074ad72d] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 17: (Thread::_entry_func(void*)+0xd) [0x7f38074ad749] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f38060511ca] 2024-05-27T07:52:34.735 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 19: clone() 2024-05-27T07:52:34.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.384+0000 7f37f4c99700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:52:34.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: in thread 7f37f4c99700 thread_name:md_log_replay 2024-05-27T07:52:34.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:34.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f380605bd20] 2024-05-27T07:52:34.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2: gsignal() 2024-05-27T07:52:34.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 3: abort() 2024-05-27T07:52:34.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f38074deced] 2024-05-27T07:52:34.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f38074dedd1] 2024-05-27T07:52:34.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e04ea0623d] 2024-05-27T07:52:34.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e04e9f1971] 2024-05-27T07:52:34.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e04e9effa0] 2024-05-27T07:52:34.736 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e04e9f0651] 2024-05-27T07:52:34.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e04e985347] 2024-05-27T07:52:34.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e04e68187e] 2024-05-27T07:52:34.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e04e686f2f] 2024-05-27T07:52:34.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e04e691969] 2024-05-27T07:52:34.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e04ea2a48c] 2024-05-27T07:52:34.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e04ea339cb] 2024-05-27T07:52:34.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 16: (Thread::entry_wrapper()+0x43) [0x7f38074ad72d] 2024-05-27T07:52:34.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 17: (Thread::_entry_func(void*)+0xd) [0x7f38074ad749] 2024-05-27T07:52:34.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f38060511ca] 2024-05-27T07:52:34.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 19: clone() 2024-05-27T07:52:34.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:52:34.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.737 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: -1> 2024-05-27T07:52:34.381+0000 7f37f4c99700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f37f4c99700 time 2024-05-27T07:52:34.379932+0000 2024-05-27T07:52:34.738 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:52:34.738 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.738 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:34.738 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f38074deba6] 2024-05-27T07:52:34.738 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f38074dedd1] 2024-05-27T07:52:34.738 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e04ea0623d] 2024-05-27T07:52:34.738 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e04e9f1971] 2024-05-27T07:52:34.738 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e04e9effa0] 2024-05-27T07:52:34.738 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e04e9f0651] 2024-05-27T07:52:34.738 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e04e985347] 2024-05-27T07:52:34.738 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e04e68187e] 2024-05-27T07:52:34.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e04e686f2f] 2024-05-27T07:52:34.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e04e691969] 2024-05-27T07:52:34.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e04ea2a48c] 2024-05-27T07:52:34.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e04ea339cb] 2024-05-27T07:52:34.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 13: (Thread::entry_wrapper()+0x43) [0x7f38074ad72d] 2024-05-27T07:52:34.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 14: (Thread::_entry_func(void*)+0xd) [0x7f38074ad749] 2024-05-27T07:52:34.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f38060511ca] 2024-05-27T07:52:34.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 16: clone() 2024-05-27T07:52:34.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 0> 2024-05-27T07:52:34.384+0000 7f37f4c99700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:52:34.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: in thread 7f37f4c99700 thread_name:md_log_replay 2024-05-27T07:52:34.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:34.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f380605bd20] 2024-05-27T07:52:34.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2: gsignal() 2024-05-27T07:52:34.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 3: abort() 2024-05-27T07:52:34.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f38074deced] 2024-05-27T07:52:34.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f38074dedd1] 2024-05-27T07:52:34.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e04ea0623d] 2024-05-27T07:52:34.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e04e9f1971] 2024-05-27T07:52:34.740 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e04e9effa0] 2024-05-27T07:52:34.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e04e9f0651] 2024-05-27T07:52:34.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e04e985347] 2024-05-27T07:52:34.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e04e68187e] 2024-05-27T07:52:34.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e04e686f2f] 2024-05-27T07:52:34.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e04e691969] 2024-05-27T07:52:34.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e04ea2a48c] 2024-05-27T07:52:34.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e04ea339cb] 2024-05-27T07:52:34.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 16: (Thread::entry_wrapper()+0x43) [0x7f38074ad72d] 2024-05-27T07:52:34.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 17: (Thread::_entry_func(void*)+0xd) [0x7f38074ad749] 2024-05-27T07:52:34.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f38060511ca] 2024-05-27T07:52:34.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 19: clone() 2024-05-27T07:52:34.741 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:52:34.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: -1> 2024-05-27T07:52:34.381+0000 7f37f4c99700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f37f4c99700 time 2024-05-27T07:52:34.379932+0000 2024-05-27T07:52:34.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:52:34.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:34.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f38074deba6] 2024-05-27T07:52:34.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f38074dedd1] 2024-05-27T07:52:34.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e04ea0623d] 2024-05-27T07:52:34.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e04e9f1971] 2024-05-27T07:52:34.742 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e04e9effa0] 2024-05-27T07:52:34.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e04e9f0651] 2024-05-27T07:52:34.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e04e985347] 2024-05-27T07:52:34.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e04e68187e] 2024-05-27T07:52:34.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e04e686f2f] 2024-05-27T07:52:34.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e04e691969] 2024-05-27T07:52:34.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e04ea2a48c] 2024-05-27T07:52:34.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e04ea339cb] 2024-05-27T07:52:34.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 13: (Thread::entry_wrapper()+0x43) [0x7f38074ad72d] 2024-05-27T07:52:34.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 14: (Thread::_entry_func(void*)+0xd) [0x7f38074ad749] 2024-05-27T07:52:34.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f38060511ca] 2024-05-27T07:52:34.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 16: clone() 2024-05-27T07:52:34.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.743 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 0> 2024-05-27T07:52:34.384+0000 7f37f4c99700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:52:34.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: in thread 7f37f4c99700 thread_name:md_log_replay 2024-05-27T07:52:34.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:52:34.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f380605bd20] 2024-05-27T07:52:34.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2: gsignal() 2024-05-27T07:52:34.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 3: abort() 2024-05-27T07:52:34.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f38074deced] 2024-05-27T07:52:34.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f38074dedd1] 2024-05-27T07:52:34.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e04ea0623d] 2024-05-27T07:52:34.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e04e9f1971] 2024-05-27T07:52:34.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e04e9effa0] 2024-05-27T07:52:34.744 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e04e9f0651] 2024-05-27T07:52:34.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e04e985347] 2024-05-27T07:52:34.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e04e68187e] 2024-05-27T07:52:34.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e04e686f2f] 2024-05-27T07:52:34.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e04e691969] 2024-05-27T07:52:34.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e04ea2a48c] 2024-05-27T07:52:34.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e04ea339cb] 2024-05-27T07:52:34.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 16: (Thread::entry_wrapper()+0x43) [0x7f38074ad72d] 2024-05-27T07:52:34.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 17: (Thread::_entry_func(void*)+0xd) [0x7f38074ad749] 2024-05-27T07:52:34.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f38060511ca] 2024-05-27T07:52:34.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 19: clone() 2024-05-27T07:52:34.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:52:34.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75631]: 2024-05-27T07:52:34.745 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:34 smithi186 podman[75669]: 2024-05-27 07:52:34.460532771 +0000 UTC m=+0.022165082 container died d28f19d0d8332a64ed907be6c135730c2c9c99c5ed9b6e622583bd24220dfdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-27T07:52:35.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1234372 9.52 MB/sec execute 527 sec latency 103.242 ms 2024-05-27T07:52:36.294 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1236046 9.53 MB/sec execute 528 sec latency 144.776 ms 2024-05-27T07:52:36.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:36 smithi114 ceph-mon[32038]: pgmap v430: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 216 op/s 2024-05-27T07:52:36.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:36 smithi163 ceph-mon[36451]: pgmap v430: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 216 op/s 2024-05-27T07:52:36.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:36 smithi186 ceph-mon[36647]: pgmap v430: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 216 op/s 2024-05-27T07:52:37.293 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1236821 9.51 MB/sec execute 529 sec latency 63.183 ms 2024-05-27T07:52:37.729 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:37 smithi186 podman[75669]: 2024-05-27 07:52:37.274824721 +0000 UTC m=+2.836457030 container remove d28f19d0d8332a64ed907be6c135730c2c9c99c5ed9b6e622583bd24220dfdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-27T07:52:37.729 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:37 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:52:38.294 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1238947 9.52 MB/sec execute 530 sec latency 53.963 ms 2024-05-27T07:52:38.479 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:38 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:52:38.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:38 smithi186 ceph-mon[36647]: pgmap v431: 129 pgs: 129 active+clean; 66 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-05-27T07:52:38.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:38 smithi114 ceph-mon[32038]: pgmap v431: 129 pgs: 129 active+clean; 66 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-05-27T07:52:38.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:38 smithi163 ceph-mon[36451]: pgmap v431: 129 pgs: 129 active+clean; 66 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-05-27T07:52:39.294 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1240215 9.51 MB/sec execute 531 sec latency 27.601 ms 2024-05-27T07:52:40.294 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1242356 9.51 MB/sec execute 532 sec latency 29.070 ms 2024-05-27T07:52:40.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:40 smithi114 ceph-mon[32038]: pgmap v432: 129 pgs: 129 active+clean; 66 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-05-27T07:52:40.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:40 smithi163 ceph-mon[36451]: pgmap v432: 129 pgs: 129 active+clean; 66 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-05-27T07:52:40.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:40 smithi186 ceph-mon[36647]: pgmap v432: 129 pgs: 129 active+clean; 66 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-05-27T07:52:41.294 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1243758 9.51 MB/sec execute 533 sec latency 13.832 ms 2024-05-27T07:52:42.294 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1246213 9.52 MB/sec execute 534 sec latency 52.290 ms 2024-05-27T07:52:42.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:42 smithi114 ceph-mon[32038]: pgmap v433: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 234 op/s 2024-05-27T07:52:42.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:42 smithi163 ceph-mon[36451]: pgmap v433: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 234 op/s 2024-05-27T07:52:42.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:42 smithi186 ceph-mon[36647]: pgmap v433: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 234 op/s 2024-05-27T07:52:43.294 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1247374 9.51 MB/sec execute 535 sec latency 14.495 ms 2024-05-27T07:52:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.159+0000 7f4798e6e640 1 -- 172.21.15.114:0/2779810746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4794108b90 msgr2=0x7f4794102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.159+0000 7f4798e6e640 1 --2- 172.21.15.114:0/2779810746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4794108b90 0x7f4794102730 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f4784009900 tx=0x7f478402a410 comp rx=0 tx=0).stop 2024-05-27T07:52:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.159+0000 7f4798e6e640 1 -- 172.21.15.114:0/2779810746 shutdown_connections 2024-05-27T07:52:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.159+0000 7f4798e6e640 1 --2- 172.21.15.114:0/2779810746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47941032e0 0x7f479410f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.159+0000 7f4798e6e640 1 --2- 172.21.15.114:0/2779810746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4794109030 0x7f4794102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.159+0000 7f4798e6e640 1 --2- 172.21.15.114:0/2779810746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4794108b90 0x7f4794102730 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.159+0000 7f4798e6e640 1 -- 172.21.15.114:0/2779810746 >> 172.21.15.114:0/2779810746 conn(0x7f47940782d0 msgr2=0x7f4794078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:44.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.159+0000 7f4798e6e640 1 -- 172.21.15.114:0/2779810746 shutdown_connections 2024-05-27T07:52:44.161 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.159+0000 7f4798e6e640 1 -- 172.21.15.114:0/2779810746 wait complete. 2024-05-27T07:52:44.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.160+0000 7f4798e6e640 1 Processor -- start 2024-05-27T07:52:44.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.160+0000 7f4798e6e640 1 -- start start 2024-05-27T07:52:44.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.160+0000 7f4798e6e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47941032e0 0x7f47941997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.160+0000 7f4798e6e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4794108b90 0x7f4794199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.160+0000 7f4798e6e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4794109030 0x7f479419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.160+0000 7f4798e6e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4794112d80 con 0x7f47941032e0 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.160+0000 7f4798e6e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4794112c00 con 0x7f4794109030 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.160+0000 7f4798e6e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4794112f00 con 0x7f4794108b90 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.160+0000 7f479259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47941032e0 0x7f47941997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.160+0000 7f479259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47941032e0 0x7f47941997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42612/0 (socket says 172.21.15.114:42612) 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.160+0000 7f479259c640 1 -- 172.21.15.114:0/3852738485 learned_addr learned my addr 172.21.15.114:0/3852738485 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.160+0000 7f4791d9b640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4794108b90 0x7f4794199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.160+0000 7f4792d9d640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4794109030 0x7f479419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.161+0000 7f479259c640 1 -- 172.21.15.114:0/3852738485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4794108b90 msgr2=0x7f4794199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.161+0000 7f479259c640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4794108b90 0x7f4794199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.161+0000 7f479259c640 1 -- 172.21.15.114:0/3852738485 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4794109030 msgr2=0x7f479419dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.161+0000 7f479259c640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4794109030 0x7f479419dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.161+0000 7f479259c640 1 -- 172.21.15.114:0/3852738485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f479419e620 con 0x7f47941032e0 2024-05-27T07:52:44.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.161+0000 7f4792d9d640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4794109030 0x7f479419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:44.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.161+0000 7f479259c640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47941032e0 0x7f47941997c0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f4784032fd0 tx=0x7f4784031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:44.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.161+0000 7f478b7fe640 1 -- 172.21.15.114:0/3852738485 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478403f030 con 0x7f47941032e0 2024-05-27T07:52:44.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.161+0000 7f4798e6e640 1 -- 172.21.15.114:0/3852738485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f479419e8b0 con 0x7f47941032e0 2024-05-27T07:52:44.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.161+0000 7f478b7fe640 1 -- 172.21.15.114:0/3852738485 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4784004490 con 0x7f47941032e0 2024-05-27T07:52:44.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.161+0000 7f478b7fe640 1 -- 172.21.15.114:0/3852738485 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784035070 con 0x7f47941032e0 2024-05-27T07:52:44.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.161+0000 7f4798e6e640 1 -- 172.21.15.114:0/3852738485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f479419edf0 con 0x7f47941032e0 2024-05-27T07:52:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f478b7fe640 1 -- 172.21.15.114:0/3852738485 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f4784042030 con 0x7f47941032e0 2024-05-27T07:52:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f479fe04740 1 Processor -- start 2024-05-27T07:52:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f479fe04740 1 -- start start 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f479fe04740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069da1b0 0x564106d7b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f479fe04740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069b12f0 0x564106d7b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f479fe04740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 0x564106d7bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f479fe04740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564106cedc30 con 0x5641069b94d0 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f479fe04740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564106cdd7d0 con 0x5641069da1b0 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f479fe04740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564106ce1810 con 0x5641069b12f0 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f4791d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 0x564106d7bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f4792d9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069da1b0 0x564106d7b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f479259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069b12f0 0x564106d7b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f4791d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 0x564106d7bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42628/0 (socket says 172.21.15.114:42628) 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f4792d9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069da1b0 0x564106d7b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:38064/0 (socket says 172.21.15.114:38064) 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.164+0000 7f4791d9b640 1 -- 172.21.15.114:0/807270825 learned_addr learned my addr 172.21.15.114:0/807270825 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.165+0000 7f4792d9d640 1 -- 172.21.15.114:0/807270825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069b12f0 msgr2=0x564106d7b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.165+0000 7f4792d9d640 1 --2- 172.21.15.114:0/807270825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069b12f0 0x564106d7b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.165+0000 7f4792d9d640 1 -- 172.21.15.114:0/807270825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 msgr2=0x564106d7bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.165+0000 7f4792d9d640 1 --2- 172.21.15.114:0/807270825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 0x564106d7bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.165+0000 7f4792d9d640 1 -- 172.21.15.114:0/807270825 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641069f2210 con 0x5641069da1b0 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.165+0000 7f478b7fe640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f47800b2950 0x7f47800b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.165+0000 7f4791d9b640 1 --2- 172.21.15.114:0/807270825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 0x564106d7bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.165+0000 7f479259c640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f47800b2950 0x7f47800b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.165+0000 7f4792d9d640 1 --2- 172.21.15.114:0/807270825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069da1b0 0x564106d7b1e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f478c005fd0 tx=0x7f478c011bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18c9ba40649f3339 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.165+0000 7f478b7fe640 1 -- 172.21.15.114:0/3852738485 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9466+0+0 (secure 0 0 0) 0x7f4784102cd0 con 0x7f47941032e0 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.165+0000 7f4788ff9640 1 -- 172.21.15.114:0/807270825 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c004d20 con 0x5641069da1b0 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.165+0000 7f4788ff9640 1 -- 172.21.15.114:0/807270825 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f478c00f9a0 con 0x5641069da1b0 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.165+0000 7f4788ff9640 1 -- 172.21.15.114:0/807270825 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c004470 con 0x5641069da1b0 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.166+0000 7f479fe04740 1 -- 172.21.15.114:0/807270825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069da1b0 msgr2=0x564106d7b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.166+0000 7f479259c640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f47800b2950 0x7f47800b4e60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4784100120 tx=0x7f4784106820 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.166+0000 7f479fe04740 1 --2- 172.21.15.114:0/807270825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069da1b0 0x564106d7b1e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f478c005fd0 tx=0x7f478c011bf0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.166+0000 7f479fe04740 1 -- 172.21.15.114:0/807270825 shutdown_connections 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.166+0000 7f479fe04740 1 --2- 172.21.15.114:0/807270825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 0x564106d7bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.166+0000 7f479fe04740 1 --2- 172.21.15.114:0/807270825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069b12f0 0x564106d7b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.166+0000 7f479fe04740 1 --2- 172.21.15.114:0/807270825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069da1b0 0x564106d7b1e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.166+0000 7f479fe04740 1 -- 172.21.15.114:0/807270825 >> 172.21.15.114:0/807270825 conn(0x5641069ce300 msgr2=0x564106d77b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.166+0000 7f479fe04740 1 -- 172.21.15.114:0/807270825 shutdown_connections 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.166+0000 7f479fe04740 1 -- 172.21.15.114:0/807270825 wait complete. 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f479fe04740 1 -- start start 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f479fe04740 1 Processor -- start 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f479fe04740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b12f0 0x564106d87cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f479fe04740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b94d0 0x564106d88210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f479fe04740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069da1b0 0x564106d8dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f479fe04740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5641069c3790 con 0x5641069b12f0 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f479fe04740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564106c99890 con 0x5641069b94d0 2024-05-27T07:52:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f479fe04740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564106d8a770 con 0x5641069da1b0 2024-05-27T07:52:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f479fe04740 10 client.0 _ll_register_callbacks cb 0x5641069f1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f4792d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b12f0 0x564106d87cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f4792d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b12f0 0x564106d87cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42634/0 (socket says 172.21.15.114:42634) 2024-05-27T07:52:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f4792d9d640 1 -- 172.21.15.114:0/1320394334 learned_addr learned my addr 172.21.15.114:0/1320394334 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f4791d9b640 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b94d0 0x564106d88210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f4792d9d640 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069da1b0 0x564106d8dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f4792d9d640 1 -- 172.21.15.114:0/1320394334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069da1b0 msgr2=0x564106d8dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f4792d9d640 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069da1b0 0x564106d8dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f4792d9d640 1 -- 172.21.15.114:0/1320394334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b94d0 msgr2=0x564106d88210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f4792d9d640 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b94d0 0x564106d88210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f4792d9d640 1 -- 172.21.15.114:0/1320394334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564106abe620 con 0x5641069b12f0 2024-05-27T07:52:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.167+0000 7f4791d9b640 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b94d0 0x564106d88210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f4792d9d640 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b12f0 0x564106d87cd0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f478c004150 tx=0x7f478c003c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8874f3e14ad72fc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f4779ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f4779ffb640 1 -- 172.21.15.114:0/1320394334 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c011220 con 0x5641069b12f0 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f4779ffb640 1 -- 172.21.15.114:0/1320394334 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9466+0+0 (secure 0 0 0) 0x7f478c0041f0 con 0x5641069b12f0 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f479fe04740 1 -- 172.21.15.114:0/1320394334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f478c00f9a0 con 0x5641069b12f0 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f4779ffb640 1 -- 172.21.15.114:0/1320394334 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f478c0043f0 con 0x5641069b12f0 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f479fe04740 10 client.18474 fetch_fsmap learned FSMap version 71 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f479fe04740 1 -- 172.21.15.114:0/1320394334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x564106d96a10 con 0x5641069b12f0 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f4779ffb640 1 -- 172.21.15.114:0/1320394334 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f478c00f9a0 con 0x5641069b12f0 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f4779ffb640 1 -- 172.21.15.114:0/1320394334 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 71) ==== 5910+0+0 (secure 0 0 0) 0x7f478c022030 con 0x5641069b12f0 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f479fe04740 10 client.18474 fetch_fsmap finished waiting for FSMap version 71 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f479fe04740 10 client.18474 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f479fe04740 20 client.18474 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f479fe04740 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564106998d80 0x564106d9a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f479fe04740 4 client.18474 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.168+0000 7f479fe04740 1 -- 172.21.15.114:0/1320394334 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564106cefb80 con 0x564106998d80 2024-05-27T07:52:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.169+0000 7f4791d9b640 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564106998d80 0x564106d9a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.169+0000 7f4779ffb640 10 client.18474 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.169+0000 7f4791d9b640 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564106998d80 0x564106d9a120 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=407c24322a6b627 server_cookie=5f1c4e7e7a5ad439 in_seq=0 out_seq=0 2024-05-27T07:52:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f4779ffb640 1 -- 172.21.15.114:0/1320394334 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564106cefb80 con 0x564106998d80 2024-05-27T07:52:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f4779ffb640 10 client.18474 handle_command_reply: tid=0 2024-05-27T07:52:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f479fe04740 1 client.18474 shutdown 2024-05-27T07:52:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f479fe04740 1 -- 172.21.15.114:0/1320394334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b12f0 msgr2=0x564106d87cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f479fe04740 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b12f0 0x564106d87cd0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f478c004150 tx=0x7f478c003c60 comp rx=0 tx=0).stop 2024-05-27T07:52:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f479fe04740 1 -- 172.21.15.114:0/1320394334 shutdown_connections 2024-05-27T07:52:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f479fe04740 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564106998d80 0x564106d9a120 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f479fe04740 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069da1b0 0x564106d8dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f4779ffb640 0 client.18474 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f479fe04740 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b94d0 0x564106d88210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f479fe04740 1 --2- 172.21.15.114:0/1320394334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b12f0 0x564106d87cd0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f479fe04740 1 -- 172.21.15.114:0/1320394334 >> 172.21.15.114:0/1320394334 conn(0x5641069ce300 msgr2=0x564106d7d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:44.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f479fe04740 1 -- 172.21.15.114:0/1320394334 shutdown_connections 2024-05-27T07:52:44.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f479fe04740 1 -- 172.21.15.114:0/1320394334 wait complete. 2024-05-27T07:52:44.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.172+0000 7f479fe04740 20 client.18474 trim_cache size 0 max 16384 2024-05-27T07:52:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.187+0000 7f479fe04740 1 Processor -- start 2024-05-27T07:52:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.187+0000 7f479fe04740 1 -- start start 2024-05-27T07:52:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f479fe04740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b12f0 0x564106d86a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f479fe04740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 0x564106d94780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f479fe04740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069da1b0 0x564106d77c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f479fe04740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f47941b0a00 con 0x5641069b94d0 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f479fe04740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f47941af190 con 0x5641069b12f0 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f479fe04740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f47941acfe0 con 0x5641069da1b0 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f4791d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 0x564106d94780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f4791d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 0x564106d94780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42644/0 (socket says 172.21.15.114:42644) 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f4791d9b640 1 -- 172.21.15.114:0/3554849237 learned_addr learned my addr 172.21.15.114:0/3554849237 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f4792d9d640 1 --2- 172.21.15.114:0/3554849237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069da1b0 0x564106d77c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f4792d9d640 1 --2- 172.21.15.114:0/3554849237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b12f0 0x564106d86a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f4791d9b640 1 -- 172.21.15.114:0/3554849237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069da1b0 msgr2=0x564106d77c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f4791d9b640 1 --2- 172.21.15.114:0/3554849237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069da1b0 0x564106d77c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f4791d9b640 1 -- 172.21.15.114:0/3554849237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b12f0 msgr2=0x564106d86a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f4791d9b640 1 --2- 172.21.15.114:0/3554849237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b12f0 0x564106d86a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f4791d9b640 1 -- 172.21.15.114:0/3554849237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564106d97e70 con 0x5641069b94d0 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f4791d9b640 1 --2- 172.21.15.114:0/3554849237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 0x564106d94780 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f477c009730 tx=0x7f477c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba50b2cb26f0915 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f477bfff640 1 -- 172.21.15.114:0/3554849237 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477c005020 con 0x5641069b94d0 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f477bfff640 1 -- 172.21.15.114:0/3554849237 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f477c00f7f0 con 0x5641069b94d0 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.188+0000 7f477bfff640 1 -- 172.21.15.114:0/3554849237 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477c004a00 con 0x5641069b94d0 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.189+0000 7f479fe04740 1 -- 172.21.15.114:0/3554849237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 msgr2=0x564106d94780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.189+0000 7f479fe04740 1 --2- 172.21.15.114:0/3554849237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 0x564106d94780 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f477c009730 tx=0x7f477c005f90 comp rx=0 tx=0).stop 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.189+0000 7f479fe04740 1 -- 172.21.15.114:0/3554849237 shutdown_connections 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.189+0000 7f479fe04740 1 --2- 172.21.15.114:0/3554849237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069da1b0 0x564106d77c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.189+0000 7f479fe04740 1 --2- 172.21.15.114:0/3554849237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069b94d0 0x564106d94780 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.189+0000 7f479fe04740 1 --2- 172.21.15.114:0/3554849237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b12f0 0x564106d86a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.189+0000 7f479fe04740 1 -- 172.21.15.114:0/3554849237 >> 172.21.15.114:0/3554849237 conn(0x5641069ce300 msgr2=0x564106d77540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.189+0000 7f479fe04740 1 -- 172.21.15.114:0/3554849237 shutdown_connections 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.189+0000 7f479fe04740 1 -- 172.21.15.114:0/3554849237 wait complete. 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f479fe04740 1 -- start start 2024-05-27T07:52:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f479fe04740 1 Processor -- start 2024-05-27T07:52:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f479fe04740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069b12f0 0x564106d940d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f479fe04740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b94d0 0x564106d94610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f479fe04740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069da1b0 0x564106d8c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f479fe04740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f47941ad650 con 0x5641069da1b0 2024-05-27T07:52:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f479fe04740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f47941aeac0 con 0x5641069b94d0 2024-05-27T07:52:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f479fe04740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f47941add20 con 0x5641069b12f0 2024-05-27T07:52:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f479fe04740 10 client.0 _ll_register_callbacks cb 0x7f477c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f4792d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069da1b0 0x564106d8c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f4792d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069da1b0 0x564106d8c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42656/0 (socket says 172.21.15.114:42656) 2024-05-27T07:52:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f4792d9d640 1 -- 172.21.15.114:0/1217499510 learned_addr learned my addr 172.21.15.114:0/1217499510 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f4791d9b640 1 --2- 172.21.15.114:0/1217499510 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b94d0 0x564106d94610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f4792d9d640 1 --2- 172.21.15.114:0/1217499510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069b12f0 0x564106d940d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f4792d9d640 1 -- 172.21.15.114:0/1217499510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069b12f0 msgr2=0x564106d940d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f4792d9d640 1 --2- 172.21.15.114:0/1217499510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069b12f0 0x564106d940d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f4792d9d640 1 -- 172.21.15.114:0/1217499510 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b94d0 msgr2=0x564106d94610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f4792d9d640 1 --2- 172.21.15.114:0/1217499510 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b94d0 0x564106d94610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f4792d9d640 1 -- 172.21.15.114:0/1217499510 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564106d8d030 con 0x5641069da1b0 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f4792d9d640 1 --2- 172.21.15.114:0/1217499510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069da1b0 0x564106d8c990 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f478c00fea0 tx=0x7f478c004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8426c2d20b5b19 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.190+0000 7f4779ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f4779ffb640 1 -- 172.21.15.114:0/1217499510 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c003ee0 con 0x5641069da1b0 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f4779ffb640 1 -- 172.21.15.114:0/1217499510 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9466+0+0 (secure 0 0 0) 0x7f478c005830 con 0x5641069da1b0 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f479fe04740 1 -- 172.21.15.114:0/1217499510 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f477c00f7f0 con 0x5641069da1b0 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f4779ffb640 1 -- 172.21.15.114:0/1217499510 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f478c04b050 con 0x5641069da1b0 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f479fe04740 10 client.18480 fetch_fsmap learned FSMap version 71 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f479fe04740 1 -- 172.21.15.114:0/1217499510 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x564106e01110 con 0x5641069da1b0 2024-05-27T07:52:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f4779ffb640 1 -- 172.21.15.114:0/1217499510 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f477c00f7f0 con 0x5641069da1b0 2024-05-27T07:52:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f4779ffb640 1 -- 172.21.15.114:0/1217499510 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 71) ==== 5910+0+0 (secure 0 0 0) 0x7f478c022070 con 0x5641069da1b0 2024-05-27T07:52:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f479fe04740 10 client.18480 fetch_fsmap finished waiting for FSMap version 71 2024-05-27T07:52:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f479fe04740 10 client.18480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f479fe04740 20 client.18480 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f479fe04740 1 --2- 172.21.15.114:0/1217499510 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56410696e1c0 0x564106e03780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f479fe04740 4 client.18480 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:52:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f479fe04740 1 -- 172.21.15.114:0/1217499510 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564106ce9360 con 0x56410696e1c0 2024-05-27T07:52:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.191+0000 7f4791d9b640 1 --2- 172.21.15.114:0/1217499510 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56410696e1c0 0x564106e03780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.192+0000 7f4791d9b640 1 --2- 172.21.15.114:0/1217499510 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56410696e1c0 0x564106e03780 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34a6e9a021cd218e server_cookie=f97616ad89c128ac in_seq=0 out_seq=0 2024-05-27T07:52:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.192+0000 7f4779ffb640 10 client.18480 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:44.294 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1249169 9.51 MB/sec execute 536 sec latency 46.942 ms 2024-05-27T07:52:44.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:44 smithi114 ceph-mon[32038]: pgmap v434: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 199 op/s 2024-05-27T07:52:44.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.701+0000 7f4779ffb640 1 -- 172.21.15.114:0/1217499510 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x564106ce9360 con 0x56410696e1c0 2024-05-27T07:52:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.701+0000 7f4779ffb640 10 client.18480 handle_command_reply: tid=0 2024-05-27T07:52:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.701+0000 7f479fe04740 1 client.18480 shutdown 2024-05-27T07:52:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.701+0000 7f479fe04740 1 -- 172.21.15.114:0/1217499510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069da1b0 msgr2=0x564106d8c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.701+0000 7f479fe04740 1 --2- 172.21.15.114:0/1217499510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069da1b0 0x564106d8c990 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f478c00fea0 tx=0x7f478c004b00 comp rx=0 tx=0).stop 2024-05-27T07:52:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.702+0000 7f479fe04740 1 -- 172.21.15.114:0/1217499510 shutdown_connections 2024-05-27T07:52:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.702+0000 7f479fe04740 1 --2- 172.21.15.114:0/1217499510 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56410696e1c0 0x564106e03780 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.702+0000 7f4779ffb640 0 client.18480 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.702+0000 7f479fe04740 1 --2- 172.21.15.114:0/1217499510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641069da1b0 0x564106d8c990 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.702+0000 7f479fe04740 1 --2- 172.21.15.114:0/1217499510 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641069b94d0 0x564106d94610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.702+0000 7f479fe04740 1 --2- 172.21.15.114:0/1217499510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641069b12f0 0x564106d940d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.702+0000 7f479fe04740 1 -- 172.21.15.114:0/1217499510 >> 172.21.15.114:0/1217499510 conn(0x5641069ce300 msgr2=0x564106d7b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.702+0000 7f479fe04740 1 -- 172.21.15.114:0/1217499510 shutdown_connections 2024-05-27T07:52:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.702+0000 7f479fe04740 1 -- 172.21.15.114:0/1217499510 wait complete. 2024-05-27T07:52:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.702+0000 7f479fe04740 20 client.18480 trim_cache size 0 max 16384 2024-05-27T07:52:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:52:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:52:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:52:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.703+0000 7f4798e6e640 1 -- 172.21.15.114:0/3852738485 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f47800b2950 msgr2=0x7f47800b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.703+0000 7f4798e6e640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f47800b2950 0x7f47800b4e60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4784100120 tx=0x7f4784106820 comp rx=0 tx=0).stop 2024-05-27T07:52:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.703+0000 7f4798e6e640 1 -- 172.21.15.114:0/3852738485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47941032e0 msgr2=0x7f47941997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.703+0000 7f4798e6e640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47941032e0 0x7f47941997c0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f4784032fd0 tx=0x7f4784031a20 comp rx=0 tx=0).stop 2024-05-27T07:52:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.703+0000 7f4798e6e640 1 -- 172.21.15.114:0/3852738485 shutdown_connections 2024-05-27T07:52:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.703+0000 7f4798e6e640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f47800b2950 0x7f47800b4e60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.703+0000 7f4798e6e640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4794109030 0x7f479419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.703+0000 7f4798e6e640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4794108b90 0x7f4794199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.703+0000 7f4798e6e640 1 --2- 172.21.15.114:0/3852738485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47941032e0 0x7f47941997c0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.703+0000 7f4798e6e640 1 -- 172.21.15.114:0/3852738485 >> 172.21.15.114:0/3852738485 conn(0x7f47940782d0 msgr2=0x7f47940ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:44.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.704+0000 7f4798e6e640 1 -- 172.21.15.114:0/3852738485 shutdown_connections 2024-05-27T07:52:44.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:44.704+0000 7f4798e6e640 1 -- 172.21.15.114:0/3852738485 wait complete. 2024-05-27T07:52:44.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:44 smithi163 ceph-mon[36451]: pgmap v434: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 199 op/s 2024-05-27T07:52:44.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:44 smithi186 ceph-mon[36647]: pgmap v434: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 199 op/s 2024-05-27T07:52:45.294 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1250380 9.51 MB/sec execute 537 sec latency 219.354 ms 2024-05-27T07:52:46.294 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1252095 9.50 MB/sec execute 538 sec latency 25.018 ms 2024-05-27T07:52:46.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:46 smithi114 ceph-mon[32038]: pgmap v435: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 183 op/s 2024-05-27T07:52:46.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:46 smithi163 ceph-mon[36451]: pgmap v435: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 183 op/s 2024-05-27T07:52:46.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:46 smithi186 ceph-mon[36647]: pgmap v435: 129 pgs: 129 active+clean; 72 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 183 op/s 2024-05-27T07:52:47.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1253901 9.51 MB/sec execute 539 sec latency 36.794 ms 2024-05-27T07:52:48.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1255581 9.50 MB/sec execute 540 sec latency 19.672 ms 2024-05-27T07:52:48.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:48 smithi114 ceph-mon[32038]: pgmap v436: 129 pgs: 129 active+clean; 67 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 352 op/s 2024-05-27T07:52:48.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:48 smithi163 ceph-mon[36451]: pgmap v436: 129 pgs: 129 active+clean; 67 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 352 op/s 2024-05-27T07:52:48.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:48 smithi186 ceph-mon[36647]: pgmap v436: 129 pgs: 129 active+clean; 67 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 352 op/s 2024-05-27T07:52:48.729 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:48 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:52:48.729 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:48 smithi186 systemd[1]: Stopped Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:52:48.729 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:48 smithi186 systemd[1]: Starting Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:52:48.729 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:48 smithi186 podman[75760]: 2024-05-27 07:52:48.360961907 +0000 UTC m=+0.020090318 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:52:49.071 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:48 smithi186 podman[75760]: 2024-05-27 07:52:48.77592467 +0000 UTC m=+0.435053054 container create d1bed78545fcf854aa9306e11efbfe67a1d0886838089797a0fe271bc6661695 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-27T07:52:49.071 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:49 smithi186 podman[75760]: 2024-05-27 07:52:49.020558377 +0000 UTC m=+0.679686762 container init d1bed78545fcf854aa9306e11efbfe67a1d0886838089797a0fe271bc6661695 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:52:49.071 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:49 smithi186 podman[75760]: 2024-05-27 07:52:49.023833052 +0000 UTC m=+0.682961437 container start d1bed78545fcf854aa9306e11efbfe67a1d0886838089797a0fe271bc6661695 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:52:49.294 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1257340 9.51 MB/sec execute 541 sec latency 128.109 ms 2024-05-27T07:52:49.479 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:49 smithi186 ceph-mon[36647]: daemon mds.l restarted 2024-05-27T07:52:49.479 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:49 smithi186 ceph-mds[75779]: -- 172.21.15.186:0/1341406 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5562bb863a40 con 0x5562bb8ab600 2024-05-27T07:52:49.479 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:49 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: starting mds.l at 2024-05-27T07:52:49.479 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:49 smithi186 bash[75760]: d1bed78545fcf854aa9306e11efbfe67a1d0886838089797a0fe271bc6661695 2024-05-27T07:52:49.479 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:52:49 smithi186 systemd[1]: Started Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:52:49.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:49 smithi114 ceph-mon[32038]: daemon mds.l restarted 2024-05-27T07:52:49.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:49 smithi163 ceph-mon[36451]: daemon mds.l restarted 2024-05-27T07:52:50.294 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1258465 9.50 MB/sec execute 542 sec latency 38.967 ms 2024-05-27T07:52:50.480 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:50 smithi186 ceph-mon[36647]: pgmap v437: 129 pgs: 129 active+clean; 67 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.9 MiB/s wr, 224 op/s 2024-05-27T07:52:50.480 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:50 smithi186 ceph-mon[36647]: osdmap e129: 12 total, 12 up, 12 in 2024-05-27T07:52:50.480 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:50 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6832/1524985663,v1:172.21.15.186:6833/1524985663] up:boot 2024-05-27T07:52:50.480 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:50 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:52:50.480 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:50 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:52:50.480 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:50 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:52:50.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:50 smithi114 ceph-mon[32038]: pgmap v437: 129 pgs: 129 active+clean; 67 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.9 MiB/s wr, 224 op/s 2024-05-27T07:52:50.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:50 smithi114 ceph-mon[32038]: osdmap e129: 12 total, 12 up, 12 in 2024-05-27T07:52:50.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:50 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6832/1524985663,v1:172.21.15.186:6833/1524985663] up:boot 2024-05-27T07:52:50.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:50 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:52:50.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:50 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:52:50.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:50 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:52:50.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:50 smithi163 ceph-mon[36451]: pgmap v437: 129 pgs: 129 active+clean; 67 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.9 MiB/s wr, 224 op/s 2024-05-27T07:52:50.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:50 smithi163 ceph-mon[36451]: osdmap e129: 12 total, 12 up, 12 in 2024-05-27T07:52:50.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:50 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6832/1524985663,v1:172.21.15.186:6833/1524985663] up:boot 2024-05-27T07:52:50.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:50 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:52:50.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:50 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:52:50.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:50 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:52:51.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1260760 9.51 MB/sec execute 543 sec latency 20.796 ms 2024-05-27T07:52:52.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:52:52.294 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1261897 9.50 MB/sec execute 544 sec latency 14.469 ms 2024-05-27T07:52:52.410 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:52:52.533 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:52 smithi114 ceph-mon[32038]: pgmap v439: 129 pgs: 129 active+clean; 72 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 298 op/s 2024-05-27T07:52:52.660 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:52 smithi186 ceph-mon[36647]: pgmap v439: 129 pgs: 129 active+clean; 72 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 298 op/s 2024-05-27T07:52:52.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:52 smithi163 ceph-mon[36451]: pgmap v439: 129 pgs: 129 active+clean; 72 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 298 op/s 2024-05-27T07:52:53.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1264307 9.51 MB/sec execute 545 sec latency 78.365 ms 2024-05-27T07:52:54.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1265222 9.50 MB/sec execute 546 sec latency 40.559 ms 2024-05-27T07:52:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.374+0000 7f06dcbb3700 1 -- 172.21.15.114:0/4011931814 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4017400 con 0x7f06d810a920 2024-05-27T07:52:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.375+0000 7f06dfe19700 1 -- 172.21.15.114:0/4011931814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06d810a920 msgr2=0x7f06d810ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.375+0000 7f06dfe19700 1 --2- 172.21.15.114:0/4011931814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06d810a920 0x7f06d810ad30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f06d40098f0 tx=0x7f06d4009c00 comp rx=0 tx=0).stop 2024-05-27T07:52:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.375+0000 7f06dfe19700 1 -- 172.21.15.114:0/4011931814 shutdown_connections 2024-05-27T07:52:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.375+0000 7f06dfe19700 1 --2- 172.21.15.114:0/4011931814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06d811d920 0x7f06d811fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.375+0000 7f06dfe19700 1 --2- 172.21.15.114:0/4011931814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06d810a920 0x7f06d810ad30 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.375+0000 7f06dfe19700 1 --2- 172.21.15.114:0/4011931814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06d810b270 0x7f06d811d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.375+0000 7f06dfe19700 1 -- 172.21.15.114:0/4011931814 >> 172.21.15.114:0/4011931814 conn(0x7f06d806f4c0 msgr2=0x7f06d806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:54.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.381+0000 7f06dfe19700 1 -- 172.21.15.114:0/4011931814 shutdown_connections 2024-05-27T07:52:54.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.382+0000 7f06dfe19700 1 -- 172.21.15.114:0/4011931814 wait complete. 2024-05-27T07:52:54.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.383+0000 7f06dfe19700 1 Processor -- start 2024-05-27T07:52:54.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.383+0000 7f06dfe19700 1 -- start start 2024-05-27T07:52:54.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.384+0000 7f06dfe19700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06d810a920 0x7f06d811bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.384+0000 7f06dfe19700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06d810b270 0x7f06d8114f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.384+0000 7f06dfe19700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06d811d920 0x7f06d8115570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.384+0000 7f06dfe19700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f06d8123320 con 0x7f06d810b270 2024-05-27T07:52:54.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.384+0000 7f06dfe19700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f06d81231a0 con 0x7f06d810a920 2024-05-27T07:52:54.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.384+0000 7f06dfe19700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f06d81234a0 con 0x7f06d811d920 2024-05-27T07:52:54.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.384+0000 7f06dd3b4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06d810b270 0x7f06d8114f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.384+0000 7f06ddbb5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06d810a920 0x7f06d811bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.384+0000 7f06dd3b4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06d810b270 0x7f06d8114f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35344/0 (socket says 172.21.15.114:35344) 2024-05-27T07:52:54.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.384+0000 7f06dd3b4700 1 -- 172.21.15.114:0/1353592390 learned_addr learned my addr 172.21.15.114:0/1353592390 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:54.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.384+0000 7f06de3b6700 1 --2- 172.21.15.114:0/1353592390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06d811d920 0x7f06d8115570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.387+0000 7f06dd3b4700 1 -- 172.21.15.114:0/1353592390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06d811d920 msgr2=0x7f06d8115570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.387+0000 7f06dd3b4700 1 --2- 172.21.15.114:0/1353592390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06d811d920 0x7f06d8115570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.387+0000 7f06dd3b4700 1 -- 172.21.15.114:0/1353592390 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06d810a920 msgr2=0x7f06d811bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.387+0000 7f06dd3b4700 1 --2- 172.21.15.114:0/1353592390 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06d810a920 0x7f06d811bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.387+0000 7f06dd3b4700 1 -- 172.21.15.114:0/1353592390 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d81198d0 con 0x7f06d810b270 2024-05-27T07:52:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.388+0000 7f06dd3b4700 1 --2- 172.21.15.114:0/1353592390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06d810b270 0x7f06d8114f20 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f06c800e9c0 tx=0x7f06c800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:54.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.388+0000 7f06ceffd700 1 -- 172.21.15.114:0/1353592390 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c800cbd0 con 0x7f06d810b270 2024-05-27T07:52:54.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.389+0000 7f06ceffd700 1 -- 172.21.15.114:0/1353592390 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06c800cd70 con 0x7f06d810b270 2024-05-27T07:52:54.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.389+0000 7f06ceffd700 1 -- 172.21.15.114:0/1353592390 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8005580 con 0x7f06d810b270 2024-05-27T07:52:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.390+0000 7f06dfe19700 1 -- 172.21.15.114:0/1353592390 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d8119bc0 con 0x7f06d810b270 2024-05-27T07:52:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.390+0000 7f06dfe19700 1 -- 172.21.15.114:0/1353592390 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d811a0b0 con 0x7f06d810b270 2024-05-27T07:52:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.396+0000 7f06ee82eb80 1 Processor -- start 2024-05-27T07:52:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.397+0000 7f06ee82eb80 1 -- start start 2024-05-27T07:52:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.397+0000 7f06ee82eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be13de40 0x5590be2c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.398+0000 7f06ee82eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2bcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.398+0000 7f06de3b6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be13de40 0x5590be2c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.398+0000 7f06de3b6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be13de40 0x5590be2c2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:35736/0 (socket says 172.21.15.114:35736) 2024-05-27T07:52:54.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.399+0000 7f06de3b6700 1 -- 172.21.15.114:0/4001056820 learned_addr learned my addr 172.21.15.114:0/4001056820 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:54.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.399+0000 7f06ddbb5700 1 --2- 172.21.15.114:0/4001056820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2bcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.399+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/4001056820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be2bd120 0x5590be2bd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.399+0000 7f06ee82eb80 1 -- 172.21.15.114:0/4001056820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5590be1ead70 con 0x5590be2c3070 2024-05-27T07:52:54.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.399+0000 7f06ee82eb80 1 -- 172.21.15.114:0/4001056820 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5590be1da910 con 0x5590be13de40 2024-05-27T07:52:54.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.399+0000 7f06ee82eb80 1 -- 172.21.15.114:0/4001056820 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5590be1de950 con 0x5590be2bd120 2024-05-27T07:52:54.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.400+0000 7f06ddbb5700 1 -- 172.21.15.114:0/4001056820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be2bd120 msgr2=0x5590be2bd530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:52:54.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.400+0000 7f06ddbb5700 1 --2- 172.21.15.114:0/4001056820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be2bd120 0x5590be2bd530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.400+0000 7f06ddbb5700 1 -- 172.21.15.114:0/4001056820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be13de40 msgr2=0x5590be2c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:54.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.400+0000 7f06ddbb5700 1 --2- 172.21.15.114:0/4001056820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be13de40 0x5590be2c2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.400+0000 7f06ddbb5700 1 -- 172.21.15.114:0/4001056820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590be183fb0 con 0x5590be2c3070 2024-05-27T07:52:54.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.401+0000 7f06ceffd700 1 -- 172.21.15.114:0/1353592390 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f06c8004750 con 0x7f06d810b270 2024-05-27T07:52:54.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.401+0000 7f06ddbb5700 1 --2- 172.21.15.114:0/4001056820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2bcbe0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f06d8116720 tx=0x7f06d4009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=605553c9828bffce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:54.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.402+0000 7f06c3fff700 1 -- 172.21.15.114:0/4001056820 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4026070 con 0x5590be2c3070 2024-05-27T07:52:54.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.402+0000 7f06c3fff700 1 -- 172.21.15.114:0/4001056820 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d400f910 con 0x5590be2c3070 2024-05-27T07:52:54.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.402+0000 7f06c3fff700 1 -- 172.21.15.114:0/4001056820 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d40038e0 con 0x5590be2c3070 2024-05-27T07:52:54.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.402+0000 7f06ceffd700 1 --2- 172.21.15.114:0/1353592390 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f06c40b2b80 0x7f06c40b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.402+0000 7f06ceffd700 1 -- 172.21.15.114:0/1353592390 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9466+0+0 (secure 0 0 0) 0x7f06c80e2b50 con 0x7f06d810b270 2024-05-27T07:52:54.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.402+0000 7f06de3b6700 1 --2- 172.21.15.114:0/1353592390 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f06c40b2b80 0x7f06c40b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.403+0000 7f06de3b6700 1 --2- 172.21.15.114:0/1353592390 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f06c40b2b80 0x7f06c40b5090 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009df0 tx=0x7f06d0009040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.403+0000 7f06ee82eb80 1 -- 172.21.15.114:0/4001056820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 msgr2=0x5590be2bcbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.403+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/4001056820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2bcbe0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f06d8116720 tx=0x7f06d4009f50 comp rx=0 tx=0).stop 2024-05-27T07:52:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.404+0000 7f06ee82eb80 1 -- 172.21.15.114:0/4001056820 shutdown_connections 2024-05-27T07:52:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.404+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/4001056820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be2bd120 0x5590be2bd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.404+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/4001056820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be13de40 0x5590be2c2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.404+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/4001056820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2bcbe0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.404+0000 7f06ee82eb80 1 -- 172.21.15.114:0/4001056820 >> 172.21.15.114:0/4001056820 conn(0x5590be140770 msgr2=0x5590be2b7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.404+0000 7f06ee82eb80 1 -- 172.21.15.114:0/4001056820 shutdown_connections 2024-05-27T07:52:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.404+0000 7f06ee82eb80 1 -- 172.21.15.114:0/4001056820 wait complete. 2024-05-27T07:52:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.405+0000 7f06ee82eb80 1 -- start start 2024-05-27T07:52:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.405+0000 7f06ee82eb80 1 Processor -- start 2024-05-27T07:52:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.406+0000 7f06ee82eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.406+0000 7f06ee82eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2bd120 0x5590be2c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.407+0000 7f06ee82eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2c3070 0x5590be2c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.407+0000 7f06ee82eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5590be0e0fa0 con 0x5590be2bd120 2024-05-27T07:52:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.407+0000 7f06ee82eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5590be142da0 con 0x5590be2c3070 2024-05-27T07:52:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.407+0000 7f06ddbb5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.407+0000 7f06ddbb5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2c69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54230/0 (socket says 172.21.15.114:54230) 2024-05-27T07:52:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.407+0000 7f06ddbb5700 1 -- 172.21.15.114:0/2807830177 learned_addr learned my addr 172.21.15.114:0/2807830177 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.407+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2807830177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5590be2c9b00 con 0x5590be13de40 2024-05-27T07:52:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.407+0000 7f06ee82eb80 10 client.0 _ll_register_callbacks cb 0x5590be1715b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.407+0000 7f06dd3b4700 1 --2- 172.21.15.114:0/2807830177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2bd120 0x5590be2c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.408+0000 7f06ddbb5700 1 -- 172.21.15.114:0/2807830177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2c3070 msgr2=0x5590be2c7440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:52:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.408+0000 7f06ddbb5700 1 --2- 172.21.15.114:0/2807830177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2c3070 0x5590be2c7440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.408+0000 7f06ddbb5700 1 -- 172.21.15.114:0/2807830177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2bd120 msgr2=0x5590be2c6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.408+0000 7f06ddbb5700 1 --2- 172.21.15.114:0/2807830177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2bd120 0x5590be2c6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.408+0000 7f06ddbb5700 1 -- 172.21.15.114:0/2807830177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590be12a8d0 con 0x5590be13de40 2024-05-27T07:52:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.409+0000 7f06ddbb5700 1 --2- 172.21.15.114:0/2807830177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2c69c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f06d400f690 tx=0x7f06d400f770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47e50259f8680c5c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.410+0000 7f06c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:52:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.412+0000 7f06c17fa700 1 -- 172.21.15.114:0/2807830177 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4026070 con 0x5590be13de40 2024-05-27T07:52:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.412+0000 7f06c17fa700 1 -- 172.21.15.114:0/2807830177 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9466+0+0 (secure 0 0 0) 0x7f06d4003680 con 0x5590be13de40 2024-05-27T07:52:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.413+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2807830177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d400f910 con 0x5590be13de40 2024-05-27T07:52:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.413+0000 7f06c17fa700 1 -- 172.21.15.114:0/2807830177 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f06d4032050 con 0x5590be13de40 2024-05-27T07:52:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.413+0000 7f06ee82eb80 10 client.28039 fetch_fsmap learned FSMap version 73 2024-05-27T07:52:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.413+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2807830177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x5590be194760 con 0x5590be13de40 2024-05-27T07:52:54.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.414+0000 7f06c17fa700 1 -- 172.21.15.114:0/2807830177 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d4005690 con 0x5590be13de40 2024-05-27T07:52:54.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.414+0000 7f06c17fa700 1 -- 172.21.15.114:0/2807830177 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7f06d402cbc0 con 0x5590be13de40 2024-05-27T07:52:54.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.415+0000 7f06ee82eb80 10 client.28039 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:52:54.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.415+0000 7f06ee82eb80 10 client.28039 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:54.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.415+0000 7f06ee82eb80 20 client.28039 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:54.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.415+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/2807830177 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5590be2d67f0 0x5590be2d8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.415+0000 7f06ee82eb80 4 client.28039 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:52:54.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.415+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2807830177 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f06d400f910 con 0x5590be2d67f0 2024-05-27T07:52:54.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.421+0000 7f06dd3b4700 1 --2- 172.21.15.114:0/2807830177 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5590be2d67f0 0x5590be2d8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.423+0000 7f06dd3b4700 1 --2- 172.21.15.114:0/2807830177 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5590be2d67f0 0x5590be2d8c40 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70beaad8a88be7f9 server_cookie=a2cd0a6f7b9ec6ea in_seq=0 out_seq=0 2024-05-27T07:52:54.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.424+0000 7f06c17fa700 10 client.28039 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:54.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.427+0000 7f06c17fa700 1 -- 172.21.15.114:0/2807830177 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f06d400f910 con 0x5590be2d67f0 2024-05-27T07:52:54.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.427+0000 7f06c17fa700 10 client.28039 handle_command_reply: tid=0 2024-05-27T07:52:54.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.427+0000 7f06ee82eb80 1 client.28039 shutdown 2024-05-27T07:52:54.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.430+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2807830177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 msgr2=0x5590be2c69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:54.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.430+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/2807830177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2c69c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f06d400f690 tx=0x7f06d400f770 comp rx=0 tx=0).stop 2024-05-27T07:52:54.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.430+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2807830177 shutdown_connections 2024-05-27T07:52:54.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.430+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/2807830177 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5590be2d67f0 0x5590be2d8c40 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.430+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/2807830177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2c3070 0x5590be2c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.430+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/2807830177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2c69c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.430+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/2807830177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2bd120 0x5590be2c6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.430+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2807830177 >> 172.21.15.114:0/2807830177 conn(0x5590be140770 msgr2=0x5590be2bbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:54.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.430+0000 7f06c17fa700 0 client.28039 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:54.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.431+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2807830177 shutdown_connections 2024-05-27T07:52:54.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.431+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2807830177 wait complete. 2024-05-27T07:52:54.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.431+0000 7f06ee82eb80 20 client.28039 trim_cache size 0 max 16384 2024-05-27T07:52:54.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.456+0000 7f06ee82eb80 1 Processor -- start 2024-05-27T07:52:54.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.456+0000 7f06ee82eb80 1 -- start start 2024-05-27T07:52:54.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:54 smithi114 ceph-mon[32038]: pgmap v440: 129 pgs: 129 active+clean; 72 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-05-27T07:52:54.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06ee82eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2bcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06ee82eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2bd120 0x5590be2c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06ee82eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2c0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06ee82eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f06d81c94e0 con 0x5590be2c3070 2024-05-27T07:52:54.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06de3b6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2c0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06ee82eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f06d81c7c50 con 0x5590be2bd120 2024-05-27T07:52:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06ee82eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f06d81c5aa0 con 0x5590be13de40 2024-05-27T07:52:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06ddbb5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2bcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06de3b6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2c0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35378/0 (socket says 172.21.15.114:35378) 2024-05-27T07:52:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06ddbb5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2bcbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54236/0 (socket says 172.21.15.114:54236) 2024-05-27T07:52:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06dd3b4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2bd120 0x5590be2c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06de3b6700 1 -- 172.21.15.114:0/2634838622 learned_addr learned my addr 172.21.15.114:0/2634838622 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06de3b6700 1 -- 172.21.15.114:0/2634838622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 msgr2=0x5590be2bcbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06de3b6700 1 --2- 172.21.15.114:0/2634838622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2bcbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06de3b6700 1 -- 172.21.15.114:0/2634838622 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2bd120 msgr2=0x5590be2c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06de3b6700 1 --2- 172.21.15.114:0/2634838622 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2bd120 0x5590be2c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.457+0000 7f06de3b6700 1 -- 172.21.15.114:0/2634838622 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590be2c4f30 con 0x5590be2c3070 2024-05-27T07:52:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.458+0000 7f06dd3b4700 1 --2- 172.21.15.114:0/2634838622 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2bd120 0x5590be2c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.458+0000 7f06de3b6700 1 --2- 172.21.15.114:0/2634838622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2c0ba0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f06d00185a0 tx=0x7f06d0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9fee4d592cb285f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.458+0000 7f06c37fe700 1 -- 172.21.15.114:0/2634838622 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0028070 con 0x5590be2c3070 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.458+0000 7f06c37fe700 1 -- 172.21.15.114:0/2634838622 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d001f040 con 0x5590be2c3070 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.458+0000 7f06c37fe700 1 -- 172.21.15.114:0/2634838622 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d001e050 con 0x5590be2c3070 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.459+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2634838622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 msgr2=0x5590be2c0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.459+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/2634838622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2c0ba0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f06d00185a0 tx=0x7f06d0015000 comp rx=0 tx=0).stop 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.459+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2634838622 shutdown_connections 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.459+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/2634838622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2c0ba0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.459+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/2634838622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2bcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.459+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/2634838622 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2bd120 0x5590be2c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.459+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2634838622 >> 172.21.15.114:0/2634838622 conn(0x5590be140770 msgr2=0x5590be2c0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.459+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2634838622 shutdown_connections 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.459+0000 7f06ee82eb80 1 -- 172.21.15.114:0/2634838622 wait complete. 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.459+0000 7f06ee82eb80 1 -- start start 2024-05-27T07:52:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.459+0000 7f06ee82eb80 1 Processor -- start 2024-05-27T07:52:54.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ee82eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2cf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ee82eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2bd120 0x5590be2cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ee82eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2dd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ee82eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f06d81c6110 con 0x5590be2c3070 2024-05-27T07:52:54.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ee82eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f06d81c7580 con 0x5590be2bd120 2024-05-27T07:52:54.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ee82eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f06d81c67e0 con 0x5590be13de40 2024-05-27T07:52:54.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ee82eb80 10 client.0 _ll_register_callbacks cb 0x5590be1715b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:54.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ddbb5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ddbb5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2cf070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54238/0 (socket says 172.21.15.114:54238) 2024-05-27T07:52:54.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ddbb5700 1 -- 172.21.15.114:0/3476754793 learned_addr learned my addr 172.21.15.114:0/3476754793 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:54.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06de3b6700 1 --2- 172.21.15.114:0/3476754793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2dd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06dd3b4700 1 --2- 172.21.15.114:0/3476754793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2bd120 0x5590be2cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ddbb5700 1 -- 172.21.15.114:0/3476754793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2bd120 msgr2=0x5590be2cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:54.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ddbb5700 1 --2- 172.21.15.114:0/3476754793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2bd120 0x5590be2cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ddbb5700 1 -- 172.21.15.114:0/3476754793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 msgr2=0x5590be2dd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:54.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ddbb5700 1 --2- 172.21.15.114:0/3476754793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2dd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.460+0000 7f06ddbb5700 1 -- 172.21.15.114:0/3476754793 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590be2de0f0 con 0x5590be13de40 2024-05-27T07:52:54.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.461+0000 7f06ddbb5700 1 --2- 172.21.15.114:0/3476754793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2cf070 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f06d4001eb0 tx=0x7f06d400be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7288f6f531f87f8c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.461+0000 7f06c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.461+0000 7f06c17fa700 1 -- 172.21.15.114:0/3476754793 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4026040 con 0x5590be13de40 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.461+0000 7f06c17fa700 1 -- 172.21.15.114:0/3476754793 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9466+0+0 (secure 0 0 0) 0x7f06d400b760 con 0x5590be13de40 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.461+0000 7f06ee82eb80 1 -- 172.21.15.114:0/3476754793 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d001f040 con 0x5590be13de40 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.461+0000 7f06c17fa700 1 -- 172.21.15.114:0/3476754793 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f06d400b960 con 0x5590be13de40 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.461+0000 7f06ee82eb80 10 client.28045 fetch_fsmap learned FSMap version 73 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.461+0000 7f06ee82eb80 1 -- 172.21.15.114:0/3476754793 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x5590be340e30 con 0x5590be13de40 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.461+0000 7f06c17fa700 1 -- 172.21.15.114:0/3476754793 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d001f040 con 0x5590be13de40 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.462+0000 7f06c17fa700 1 -- 172.21.15.114:0/3476754793 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7f06d4003680 con 0x5590be13de40 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.462+0000 7f06ee82eb80 10 client.28045 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.462+0000 7f06ee82eb80 10 client.28045 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.462+0000 7f06ee82eb80 20 client.28045 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.462+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/3476754793 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5590be3412c0 0x5590be3436e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.462+0000 7f06ee82eb80 4 client.28045 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.462+0000 7f06ee82eb80 1 -- 172.21.15.114:0/3476754793 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5590be1eccc0 con 0x5590be3412c0 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.462+0000 7f06dd3b4700 1 --2- 172.21.15.114:0/3476754793 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5590be3412c0 0x5590be3436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.462+0000 7f06c17fa700 10 client.28045 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.462+0000 7f06dd3b4700 1 --2- 172.21.15.114:0/3476754793 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5590be3412c0 0x5590be3436e0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c045b816c15e0fb7 server_cookie=6675abb3d4e3346 in_seq=0 out_seq=0 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.463+0000 7f06c17fa700 1 -- 172.21.15.114:0/3476754793 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5590be1eccc0 con 0x5590be3412c0 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.463+0000 7f06c17fa700 10 client.28045 handle_command_reply: tid=0 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.463+0000 7f06ee82eb80 1 client.28045 shutdown 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.463+0000 7f06ee82eb80 1 -- 172.21.15.114:0/3476754793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 msgr2=0x5590be2cf070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.463+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/3476754793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2cf070 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f06d4001eb0 tx=0x7f06d400be90 comp rx=0 tx=0).stop 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.463+0000 7f06ee82eb80 1 -- 172.21.15.114:0/3476754793 shutdown_connections 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.463+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/3476754793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5590be2c3070 0x5590be2dd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.463+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/3476754793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590be13de40 0x5590be2cf070 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.463+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/3476754793 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5590be3412c0 0x5590be3436e0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.463+0000 7f06ee82eb80 1 --2- 172.21.15.114:0/3476754793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5590be2bd120 0x5590be2cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.463+0000 7f06ee82eb80 1 -- 172.21.15.114:0/3476754793 >> 172.21.15.114:0/3476754793 conn(0x5590be140770 msgr2=0x5590be2b14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:54.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.463+0000 7f06c17fa700 0 client.28045 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:54.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06ee82eb80 1 -- 172.21.15.114:0/3476754793 shutdown_connections 2024-05-27T07:52:54.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06ee82eb80 1 -- 172.21.15.114:0/3476754793 wait complete. 2024-05-27T07:52:54.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06ee82eb80 20 client.28045 trim_cache size 0 max 16384 2024-05-27T07:52:54.469 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:52:54.469 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:52:54.469 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:52:54.469 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:52:54.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06dfe19700 1 -- 172.21.15.114:0/1353592390 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f06c40b2b80 msgr2=0x7f06c40b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:54.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06dfe19700 1 --2- 172.21.15.114:0/1353592390 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f06c40b2b80 0x7f06c40b5090 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009df0 tx=0x7f06d0009040 comp rx=0 tx=0).stop 2024-05-27T07:52:54.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06dfe19700 1 -- 172.21.15.114:0/1353592390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06d810b270 msgr2=0x7f06d8114f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:54.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06dfe19700 1 --2- 172.21.15.114:0/1353592390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06d810b270 0x7f06d8114f20 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f06c800e9c0 tx=0x7f06c800ecd0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06dfe19700 1 -- 172.21.15.114:0/1353592390 shutdown_connections 2024-05-27T07:52:54.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06dfe19700 1 --2- 172.21.15.114:0/1353592390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06d811d920 0x7f06d8115570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06dfe19700 1 --2- 172.21.15.114:0/1353592390 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f06c40b2b80 0x7f06c40b5090 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06dfe19700 1 --2- 172.21.15.114:0/1353592390 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06d810a920 0x7f06d811bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06dfe19700 1 --2- 172.21.15.114:0/1353592390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06d810b270 0x7f06d8114f20 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:54.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06dfe19700 1 -- 172.21.15.114:0/1353592390 >> 172.21.15.114:0/1353592390 conn(0x7f06d806f4c0 msgr2=0x7f06d8072780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:54.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06dfe19700 1 -- 172.21.15.114:0/1353592390 shutdown_connections 2024-05-27T07:52:54.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:54.464+0000 7f06dfe19700 1 -- 172.21.15.114:0/1353592390 wait complete. 2024-05-27T07:52:54.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:54 smithi163 ceph-mon[36451]: pgmap v440: 129 pgs: 129 active+clean; 72 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-05-27T07:52:54.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:54 smithi186 ceph-mon[36647]: pgmap v440: 129 pgs: 129 active+clean; 72 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-05-27T07:52:55.136 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:52:55.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:52:55.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1266707 9.49 MB/sec execute 547 sec latency 150.415 ms 2024-05-27T07:52:55.394 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:52:56.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1268304 9.49 MB/sec execute 548 sec latency 17.017 ms 2024-05-27T07:52:56.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:56 smithi114 ceph-mon[32038]: pgmap v441: 129 pgs: 129 active+clean; 72 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-05-27T07:52:56.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:56 smithi163 ceph-mon[36451]: pgmap v441: 129 pgs: 129 active+clean; 72 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-05-27T07:52:56.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:56 smithi186 ceph-mon[36647]: pgmap v441: 129 pgs: 129 active+clean; 72 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-05-27T07:52:57.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.228+0000 7f282d156700 1 -- 172.21.15.114:0/2050618814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2828108750 msgr2=0x7f282806c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:57.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.228+0000 7f282d156700 1 --2- 172.21.15.114:0/2050618814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2828108750 0x7f282806c030 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f282000b0a0 tx=0x7f282000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.228+0000 7f282d156700 1 -- 172.21.15.114:0/2050618814 shutdown_connections 2024-05-27T07:52:57.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.228+0000 7f282d156700 1 --2- 172.21.15.114:0/2050618814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2828108750 0x7f282806c030 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.228+0000 7f282d156700 1 --2- 172.21.15.114:0/2050618814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28281034d0 0x7f28281038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.228+0000 7f282d156700 1 --2- 172.21.15.114:0/2050618814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2828107e00 0x7f2828108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.228+0000 7f282d156700 1 -- 172.21.15.114:0/2050618814 >> 172.21.15.114:0/2050618814 conn(0x7f28280fef20 msgr2=0x7f2828101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:57.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.228+0000 7f282d156700 1 -- 172.21.15.114:0/2050618814 shutdown_connections 2024-05-27T07:52:57.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.228+0000 7f282d156700 1 -- 172.21.15.114:0/2050618814 wait complete. 2024-05-27T07:52:57.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.229+0000 7f282d156700 1 Processor -- start 2024-05-27T07:52:57.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.230+0000 7f282d156700 1 -- start start 2024-05-27T07:52:57.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.230+0000 7f282d156700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28281034d0 0x7f282810fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.230+0000 7f282d156700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2828107e00 0x7f282810e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.230+0000 7f282d156700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2828108750 0x7f282810e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.230+0000 7f282d156700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f28281144d0 con 0x7f2828107e00 2024-05-27T07:52:57.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.230+0000 7f282d156700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2828114350 con 0x7f28281034d0 2024-05-27T07:52:57.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.230+0000 7f282d156700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2828114650 con 0x7f2828108750 2024-05-27T07:52:57.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.230+0000 7f282659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2828107e00 0x7f282810e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.230+0000 7f282659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2828107e00 0x7f282810e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45460/0 (socket says 172.21.15.114:45460) 2024-05-27T07:52:57.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.230+0000 7f282659c700 1 -- 172.21.15.114:0/2304663137 learned_addr learned my addr 172.21.15.114:0/2304663137 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:57.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.230+0000 7f2826d9d700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28281034d0 0x7f282810fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.230+0000 7f282759e700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2828108750 0x7f282810e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.231+0000 7f282659c700 1 -- 172.21.15.114:0/2304663137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2828108750 msgr2=0x7f282810e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:57.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.231+0000 7f282659c700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2828108750 0x7f282810e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.231+0000 7f282659c700 1 -- 172.21.15.114:0/2304663137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28281034d0 msgr2=0x7f282810fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:57.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.231+0000 7f282659c700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28281034d0 0x7f282810fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.231+0000 7f282659c700 1 -- 172.21.15.114:0/2304663137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f282810ef80 con 0x7f2828107e00 2024-05-27T07:52:57.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.231+0000 7f282659c700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2828107e00 0x7f282810e170 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f281c00d980 tx=0x7f281c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:57.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.231+0000 7f2826d9d700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28281034d0 0x7f282810fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:52:57.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.231+0000 7f280ffff700 1 -- 172.21.15.114:0/2304663137 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281c017070 con 0x7f2828107e00 2024-05-27T07:52:57.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.231+0000 7f282d156700 1 -- 172.21.15.114:0/2304663137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28281ad890 con 0x7f2828107e00 2024-05-27T07:52:57.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.231+0000 7f282d156700 1 -- 172.21.15.114:0/2304663137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28281addf0 con 0x7f2828107e00 2024-05-27T07:52:57.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.232+0000 7f280ffff700 1 -- 172.21.15.114:0/2304663137 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f281c010460 con 0x7f2828107e00 2024-05-27T07:52:57.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.232+0000 7f280ffff700 1 -- 172.21.15.114:0/2304663137 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281c00de60 con 0x7f2828107e00 2024-05-27T07:52:57.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.233+0000 7f282d156700 1 -- 172.21.15.114:0/2304663137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2814005330 con 0x7f2828107e00 2024-05-27T07:52:57.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.234+0000 7f280ffff700 1 -- 172.21.15.114:0/2304663137 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f281c009b40 con 0x7f2828107e00 2024-05-27T07:52:57.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.235+0000 7f280ffff700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f28100b2a70 0x7f28100b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.235+0000 7f2826d9d700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f28100b2a70 0x7f28100b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.235+0000 7f280ffff700 1 -- 172.21.15.114:0/2304663137 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9466+0+0 (secure 0 0 0) 0x7f281c0e22f0 con 0x7f2828107e00 2024-05-27T07:52:57.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.236+0000 7f2826d9d700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f28100b2a70 0x7f28100b4f80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2818009cb0 tx=0x7f2818009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:57.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.239+0000 7f280ffff700 1 -- 172.21.15.114:0/2304663137 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f281c0a99a0 con 0x7f2828107e00 2024-05-27T07:52:57.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1269990 9.49 MB/sec execute 549 sec latency 55.340 ms 2024-05-27T07:52:57.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.538+0000 7f282d156700 1 -- 172.21.15.114:0/2304663137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2814005620 con 0x7f2828107e00 2024-05-27T07:52:57.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.539+0000 7f280ffff700 1 -- 172.21.15.114:0/2304663137 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+8522 (secure 0 0 0) 0x7f281c0a92a0 con 0x7f2828107e00 2024-05-27T07:52:57.539 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:52:57.539 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":27557,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3261223371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":3261223371},{"type":"v1","addr":"172.21.15.163:6835","nonce":3261223371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":73,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:52:49.276264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_28030":{"gid":28030,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1524985663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1524985663},{"type":"v1","addr":"172.21.15.186:6833","nonce":1524985663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:52:57.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.542+0000 7f282d156700 1 -- 172.21.15.114:0/2304663137 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f28100b2a70 msgr2=0x7f28100b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:57.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.542+0000 7f282d156700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f28100b2a70 0x7f28100b4f80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2818009cb0 tx=0x7f2818009260 comp rx=0 tx=0).stop 2024-05-27T07:52:57.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.542+0000 7f282d156700 1 -- 172.21.15.114:0/2304663137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2828107e00 msgr2=0x7f282810e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:57.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.542+0000 7f282d156700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2828107e00 0x7f282810e170 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f281c00d980 tx=0x7f281c00dd40 comp rx=0 tx=0).stop 2024-05-27T07:52:57.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.542+0000 7f282d156700 1 -- 172.21.15.114:0/2304663137 shutdown_connections 2024-05-27T07:52:57.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.542+0000 7f282d156700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2828108750 0x7f282810e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.542+0000 7f282d156700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f28100b2a70 0x7f28100b4f80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.542+0000 7f282d156700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28281034d0 0x7f282810fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.542+0000 7f282d156700 1 --2- 172.21.15.114:0/2304663137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2828107e00 0x7f282810e170 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.542+0000 7f282d156700 1 -- 172.21.15.114:0/2304663137 >> 172.21.15.114:0/2304663137 conn(0x7f28280fef20 msgr2=0x7f2828104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:57.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.542+0000 7f282d156700 1 -- 172.21.15.114:0/2304663137 shutdown_connections 2024-05-27T07:52:57.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:52:57.542+0000 7f282d156700 1 -- 172.21.15.114:0/2304663137 wait complete. 2024-05-27T07:52:57.543 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 73 2024-05-27T07:52:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.812+0000 7fa51d261640 1 -- 172.21.15.114:0/3627625280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa518109030 msgr2=0x7fa518102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.812+0000 7fa51d261640 1 --2- 172.21.15.114:0/3627625280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa518109030 0x7fa518102c70 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009900 tx=0x7fa50c02a410 comp rx=0 tx=0).stop 2024-05-27T07:52:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.812+0000 7fa51d261640 1 -- 172.21.15.114:0/3627625280 shutdown_connections 2024-05-27T07:52:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.812+0000 7fa51d261640 1 --2- 172.21.15.114:0/3627625280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5181032e0 0x7fa51810f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.812+0000 7fa51d261640 1 --2- 172.21.15.114:0/3627625280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa518109030 0x7fa518102c70 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.812+0000 7fa51d261640 1 --2- 172.21.15.114:0/3627625280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa518108b90 0x7fa518102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.812+0000 7fa51d261640 1 -- 172.21.15.114:0/3627625280 >> 172.21.15.114:0/3627625280 conn(0x7fa5180782d0 msgr2=0x7fa518078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.813+0000 7fa51d261640 1 -- 172.21.15.114:0/3627625280 shutdown_connections 2024-05-27T07:52:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.813+0000 7fa51d261640 1 -- 172.21.15.114:0/3627625280 wait complete. 2024-05-27T07:52:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.813+0000 7fa51d261640 1 Processor -- start 2024-05-27T07:52:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.813+0000 7fa51d261640 1 -- start start 2024-05-27T07:52:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa51d261640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5181032e0 0x7fa5181a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa51d261640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa518108b90 0x7fa5181a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa51d261640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa518109030 0x7fa51819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa51d261640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa518112d80 con 0x7fa5181032e0 2024-05-27T07:52:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa51d261640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa518112c00 con 0x7fa518108b90 2024-05-27T07:52:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa51d261640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa518112f00 con 0x7fa518109030 2024-05-27T07:52:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa516ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5181032e0 0x7fa5181a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa5167fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa518108b90 0x7fa5181a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa516ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5181032e0 0x7fa5181a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45480/0 (socket says 172.21.15.114:45480) 2024-05-27T07:52:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa516ffd640 1 -- 172.21.15.114:0/4264981942 learned_addr learned my addr 172.21.15.114:0/4264981942 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa5177fe640 1 --2- 172.21.15.114:0/4264981942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa518109030 0x7fa51819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa516ffd640 1 -- 172.21.15.114:0/4264981942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa518109030 msgr2=0x7fa51819a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa516ffd640 1 --2- 172.21.15.114:0/4264981942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa518109030 0x7fa51819a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa516ffd640 1 -- 172.21.15.114:0/4264981942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa518108b90 msgr2=0x7fa5181a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa516ffd640 1 --2- 172.21.15.114:0/4264981942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa518108b90 0x7fa5181a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.814+0000 7fa516ffd640 1 -- 172.21.15.114:0/4264981942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51819aea0 con 0x7fa5181032e0 2024-05-27T07:52:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.815+0000 7fa516ffd640 1 --2- 172.21.15.114:0/4264981942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5181032e0 0x7fa5181a05f0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fa508005fd0 tx=0x7fa50800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.815+0000 7fa4fffff640 1 -- 172.21.15.114:0/4264981942 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa508017070 con 0x7fa5181032e0 2024-05-27T07:52:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.815+0000 7fa51d261640 1 -- 172.21.15.114:0/4264981942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa51819b130 con 0x7fa5181032e0 2024-05-27T07:52:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.815+0000 7fa4fffff640 1 -- 172.21.15.114:0/4264981942 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa508009c30 con 0x7fa5181032e0 2024-05-27T07:52:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.815+0000 7fa4fffff640 1 -- 172.21.15.114:0/4264981942 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa508003d30 con 0x7fa5181032e0 2024-05-27T07:52:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.815+0000 7fa51d261640 1 -- 172.21.15.114:0/4264981942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5181a7390 con 0x7fa5181032e0 2024-05-27T07:52:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.818+0000 7fa4fffff640 1 -- 172.21.15.114:0/4264981942 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa508003ed0 con 0x7fa5181032e0 2024-05-27T07:52:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.818+0000 7fa5241ef740 1 Processor -- start 2024-05-27T07:52:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.818+0000 7fa5241ef740 1 -- start start 2024-05-27T07:52:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5241ef740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241991b0 0x561a2453e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5241ef740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241702f0 0x561a2453ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5241ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241784d0 0x561a2453f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5241ef740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561a244acc30 con 0x561a241991b0 2024-05-27T07:52:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5241ef740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561a2449c7d0 con 0x561a241702f0 2024-05-27T07:52:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5241ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561a244a0810 con 0x561a241784d0 2024-05-27T07:52:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5177fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241991b0 0x561a2453e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa4fffff640 1 --2- 172.21.15.114:0/4264981942 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa5000b29d0 0x7fa5000b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5177fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241991b0 0x561a2453e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45488/0 (socket says 172.21.15.114:45488) 2024-05-27T07:52:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5177fe640 1 -- 172.21.15.114:0/3031384070 learned_addr learned my addr 172.21.15.114:0/3031384070 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5167fc640 1 --2- 172.21.15.114:0/3031384070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241784d0 0x561a2453f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5177fe640 1 --2- 172.21.15.114:0/4264981942 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa5000b29d0 0x7fa5000b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa4fffff640 1 -- 172.21.15.114:0/4264981942 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9466+0+0 (secure 0 0 0) 0x7fa5080e25c0 con 0x7fa5181032e0 2024-05-27T07:52:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa516ffd640 1 --2- 172.21.15.114:0/3031384070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241702f0 0x561a2453ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5177fe640 1 -- 172.21.15.114:0/3031384070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241784d0 msgr2=0x561a2453f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5177fe640 1 --2- 172.21.15.114:0/3031384070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241784d0 0x561a2453f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5177fe640 1 -- 172.21.15.114:0/3031384070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241702f0 msgr2=0x561a2453ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5177fe640 1 --2- 172.21.15.114:0/3031384070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241702f0 0x561a2453ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5177fe640 1 -- 172.21.15.114:0/3031384070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a241b1210 con 0x561a241991b0 2024-05-27T07:52:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.819+0000 7fa5167fc640 1 --2- 172.21.15.114:0/3031384070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241784d0 0x561a2453f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:52:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.820+0000 7fa5177fe640 1 --2- 172.21.15.114:0/3031384070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241991b0 0x561a2453e7e0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fa510005c40 tx=0x7fa51000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ec65e9427d00b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.820+0000 7fa4fd7fa640 1 -- 172.21.15.114:0/3031384070 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa510022020 con 0x561a241991b0 2024-05-27T07:52:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.820+0000 7fa4fd7fa640 1 -- 172.21.15.114:0/3031384070 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa510013170 con 0x561a241991b0 2024-05-27T07:52:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.820+0000 7fa4fd7fa640 1 -- 172.21.15.114:0/3031384070 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa510010720 con 0x561a241991b0 2024-05-27T07:52:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.820+0000 7fa5177fe640 1 --2- 172.21.15.114:0/4264981942 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa5000b29d0 0x7fa5000b4ee0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa510023000 tx=0x7fa510021000 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.821+0000 7fa5241ef740 1 -- 172.21.15.114:0/3031384070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241991b0 msgr2=0x561a2453e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.821+0000 7fa5241ef740 1 --2- 172.21.15.114:0/3031384070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241991b0 0x561a2453e7e0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fa510005c40 tx=0x7fa51000f3c0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.821+0000 7fa5241ef740 1 -- 172.21.15.114:0/3031384070 shutdown_connections 2024-05-27T07:52:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.821+0000 7fa5241ef740 1 --2- 172.21.15.114:0/3031384070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241784d0 0x561a2453f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.821+0000 7fa5241ef740 1 --2- 172.21.15.114:0/3031384070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241702f0 0x561a2453ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.821+0000 7fa5241ef740 1 --2- 172.21.15.114:0/3031384070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241991b0 0x561a2453e7e0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.821+0000 7fa5241ef740 1 -- 172.21.15.114:0/3031384070 >> 172.21.15.114:0/3031384070 conn(0x561a2418d300 msgr2=0x561a2453af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.821+0000 7fa5241ef740 1 -- 172.21.15.114:0/3031384070 shutdown_connections 2024-05-27T07:52:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.821+0000 7fa5241ef740 1 -- 172.21.15.114:0/3031384070 wait complete. 2024-05-27T07:52:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.821+0000 7fa5241ef740 1 -- start start 2024-05-27T07:52:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.821+0000 7fa5241ef740 1 Processor -- start 2024-05-27T07:52:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa5241ef740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 0x561a2454a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa5241ef740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241784d0 0x561a2454ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa5241ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241991b0 0x561a24548d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa5241ef740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561a24182790 con 0x561a241702f0 2024-05-27T07:52:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa5241ef740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561a24458890 con 0x561a241784d0 2024-05-27T07:52:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa5241ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561a2454b910 con 0x561a241991b0 2024-05-27T07:52:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa5241ef740 10 client.0 _ll_register_callbacks cb 0x561a241b0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa516ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 0x561a2454a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa516ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 0x561a2454a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45490/0 (socket says 172.21.15.114:45490) 2024-05-27T07:52:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa516ffd640 1 -- 172.21.15.114:0/3430782545 learned_addr learned my addr 172.21.15.114:0/3430782545 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa5177fe640 1 --2- 172.21.15.114:0/3430782545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241991b0 0x561a24548d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa5167fc640 1 --2- 172.21.15.114:0/3430782545 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241784d0 0x561a2454ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa516ffd640 1 -- 172.21.15.114:0/3430782545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241991b0 msgr2=0x561a24548d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa516ffd640 1 --2- 172.21.15.114:0/3430782545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241991b0 0x561a24548d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa516ffd640 1 -- 172.21.15.114:0/3430782545 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241784d0 msgr2=0x561a2454ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa516ffd640 1 --2- 172.21.15.114:0/3430782545 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241784d0 0x561a2454ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa516ffd640 1 -- 172.21.15.114:0/3430782545 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a2427d620 con 0x561a241702f0 2024-05-27T07:52:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa516ffd640 1 --2- 172.21.15.114:0/3430782545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 0x561a2454a880 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fa5080a6700 tx=0x7fa5080dfe00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77bb37df69eb988 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.822+0000 7fa4faff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:52:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa4faff5640 1 -- 172.21.15.114:0/3430782545 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5080a9490 con 0x561a241702f0 2024-05-27T07:52:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa4faff5640 1 -- 172.21.15.114:0/3430782545 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9466+0+0 (secure 0 0 0) 0x7fa5080e2130 con 0x561a241702f0 2024-05-27T07:52:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa5241ef740 1 -- 172.21.15.114:0/3430782545 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa510013170 con 0x561a241702f0 2024-05-27T07:52:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa4faff5640 1 -- 172.21.15.114:0/3430782545 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fa5080ad840 con 0x561a241702f0 2024-05-27T07:52:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa5241ef740 10 client.18519 fetch_fsmap learned FSMap version 73 2024-05-27T07:52:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa5241ef740 1 -- 172.21.15.114:0/3430782545 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x561a24557d10 con 0x561a241702f0 2024-05-27T07:52:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa4faff5640 1 -- 172.21.15.114:0/3430782545 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa510013170 con 0x561a241702f0 2024-05-27T07:52:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa4faff5640 1 -- 172.21.15.114:0/3430782545 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7fa5080b0ce0 con 0x561a241702f0 2024-05-27T07:52:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa5241ef740 10 client.18519 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:52:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa5241ef740 10 client.18519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa5241ef740 20 client.18519 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa5241ef740 1 --2- 172.21.15.114:0/3430782545 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561a24157d80 0x561a2455b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa5241ef740 4 client.18519 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:52:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa5241ef740 1 -- 172.21.15.114:0/3430782545 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a244aeb80 con 0x561a24157d80 2024-05-27T07:52:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.823+0000 7fa5167fc640 1 --2- 172.21.15.114:0/3430782545 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561a24157d80 0x561a2455b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.824+0000 7fa4faff5640 10 client.18519 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.824+0000 7fa5167fc640 1 --2- 172.21.15.114:0/3430782545 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561a24157d80 0x561a2455b670 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a73e5d03c1a26640 server_cookie=f711764d5a9c3b0 in_seq=0 out_seq=0 2024-05-27T07:52:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa4faff5640 1 -- 172.21.15.114:0/3430782545 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561a244aeb80 con 0x561a24157d80 2024-05-27T07:52:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa4faff5640 10 client.18519 handle_command_reply: tid=0 2024-05-27T07:52:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa5241ef740 1 client.18519 shutdown 2024-05-27T07:52:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa5241ef740 1 -- 172.21.15.114:0/3430782545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 msgr2=0x561a2454a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa5241ef740 1 --2- 172.21.15.114:0/3430782545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 0x561a2454a880 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fa5080a6700 tx=0x7fa5080dfe00 comp rx=0 tx=0).stop 2024-05-27T07:52:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa5241ef740 1 -- 172.21.15.114:0/3430782545 shutdown_connections 2024-05-27T07:52:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa5241ef740 1 --2- 172.21.15.114:0/3430782545 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561a24157d80 0x561a2455b670 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa5241ef740 1 --2- 172.21.15.114:0/3430782545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241991b0 0x561a24548d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa5241ef740 1 --2- 172.21.15.114:0/3430782545 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241784d0 0x561a2454ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa4faff5640 0 client.18519 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa5241ef740 1 --2- 172.21.15.114:0/3430782545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 0x561a2454a880 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa5241ef740 1 -- 172.21.15.114:0/3430782545 >> 172.21.15.114:0/3430782545 conn(0x561a2418d300 msgr2=0x561a2453e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa5241ef740 1 -- 172.21.15.114:0/3430782545 shutdown_connections 2024-05-27T07:52:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa5241ef740 1 -- 172.21.15.114:0/3430782545 wait complete. 2024-05-27T07:52:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.825+0000 7fa5241ef740 20 client.18519 trim_cache size 0 max 16384 2024-05-27T07:52:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.840+0000 7fa5241ef740 1 Processor -- start 2024-05-27T07:52:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.840+0000 7fa5241ef740 1 -- start start 2024-05-27T07:52:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.840+0000 7fa5241ef740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a24157d80 0x561a245424a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.840+0000 7fa5241ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241702f0 0x561a245429e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.840+0000 7fa5241ef740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241784d0 0x561a24560df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.840+0000 7fa5241ef740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181b09d0 con 0x561a24157d80 2024-05-27T07:52:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.840+0000 7fa5241ef740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181af160 con 0x561a241784d0 2024-05-27T07:52:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.840+0000 7fa5241ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181acfb0 con 0x561a241702f0 2024-05-27T07:52:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.840+0000 7fa516ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a24157d80 0x561a245424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.840+0000 7fa516ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a24157d80 0x561a245424a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45492/0 (socket says 172.21.15.114:45492) 2024-05-27T07:52:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.840+0000 7fa516ffd640 1 -- 172.21.15.114:0/14138661 learned_addr learned my addr 172.21.15.114:0/14138661 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.840+0000 7fa5167fc640 1 --2- 172.21.15.114:0/14138661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241702f0 0x561a245429e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.841+0000 7fa5177fe640 1 --2- 172.21.15.114:0/14138661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241784d0 0x561a24560df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.841+0000 7fa516ffd640 1 -- 172.21.15.114:0/14138661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241702f0 msgr2=0x561a245429e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.841+0000 7fa516ffd640 1 --2- 172.21.15.114:0/14138661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241702f0 0x561a245429e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.841+0000 7fa516ffd640 1 -- 172.21.15.114:0/14138661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241784d0 msgr2=0x561a24560df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.841+0000 7fa516ffd640 1 --2- 172.21.15.114:0/14138661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241784d0 0x561a24560df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.841+0000 7fa516ffd640 1 -- 172.21.15.114:0/14138661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a2453e170 con 0x561a24157d80 2024-05-27T07:52:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.841+0000 7fa516ffd640 1 --2- 172.21.15.114:0/14138661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a24157d80 0x561a245424a0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fa508000c00 tx=0x7fa508009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24418a5e84c1034e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.841+0000 7fa4fcff9640 1 -- 172.21.15.114:0/14138661 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5080a9b90 con 0x561a24157d80 2024-05-27T07:52:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.841+0000 7fa4fcff9640 1 -- 172.21.15.114:0/14138661 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5080a9490 con 0x561a24157d80 2024-05-27T07:52:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.841+0000 7fa4fcff9640 1 -- 172.21.15.114:0/14138661 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5080b0bc0 con 0x561a24157d80 2024-05-27T07:52:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.842+0000 7fa5241ef740 1 -- 172.21.15.114:0/14138661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a24157d80 msgr2=0x561a245424a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.842+0000 7fa5241ef740 1 --2- 172.21.15.114:0/14138661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a24157d80 0x561a245424a0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fa508000c00 tx=0x7fa508009ea0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.842+0000 7fa5241ef740 1 -- 172.21.15.114:0/14138661 shutdown_connections 2024-05-27T07:52:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.842+0000 7fa5241ef740 1 --2- 172.21.15.114:0/14138661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a241784d0 0x561a24560df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.842+0000 7fa5241ef740 1 --2- 172.21.15.114:0/14138661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241702f0 0x561a245429e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.842+0000 7fa5241ef740 1 --2- 172.21.15.114:0/14138661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a24157d80 0x561a245424a0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.842+0000 7fa5241ef740 1 -- 172.21.15.114:0/14138661 >> 172.21.15.114:0/14138661 conn(0x561a2418d300 msgr2=0x561a245475f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.842+0000 7fa5241ef740 1 -- 172.21.15.114:0/14138661 shutdown_connections 2024-05-27T07:52:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.842+0000 7fa5241ef740 1 -- 172.21.15.114:0/14138661 wait complete. 2024-05-27T07:52:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.842+0000 7fa5241ef740 1 -- start start 2024-05-27T07:52:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.842+0000 7fa5241ef740 1 Processor -- start 2024-05-27T07:52:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5241ef740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a24157d80 0x561a24551050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5241ef740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 0x561a2454e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5241ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241784d0 0x561a2454e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5241ef740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181ad620 con 0x561a241702f0 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5241ef740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181aea90 con 0x561a24157d80 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5241ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181adcf0 con 0x561a241784d0 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5241ef740 10 client.0 _ll_register_callbacks cb 0x7fa508005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5167fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 0x561a2454e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5167fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 0x561a2454e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45508/0 (socket says 172.21.15.114:45508) 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5167fc640 1 -- 172.21.15.114:0/2124077603 learned_addr learned my addr 172.21.15.114:0/2124077603 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5177fe640 1 --2- 172.21.15.114:0/2124077603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241784d0 0x561a2454e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa516ffd640 1 --2- 172.21.15.114:0/2124077603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a24157d80 0x561a24551050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5167fc640 1 -- 172.21.15.114:0/2124077603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241784d0 msgr2=0x561a2454e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5167fc640 1 --2- 172.21.15.114:0/2124077603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241784d0 0x561a2454e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5167fc640 1 -- 172.21.15.114:0/2124077603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a24157d80 msgr2=0x561a24551050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5167fc640 1 --2- 172.21.15.114:0/2124077603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a24157d80 0x561a24551050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5167fc640 1 -- 172.21.15.114:0/2124077603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a2454ee40 con 0x561a241702f0 2024-05-27T07:52:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5167fc640 1 --2- 172.21.15.114:0/2124077603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 0x561a2454e3c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009900 tx=0x7fa50c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d20de19fd294a1c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:52:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa4faff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:52:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa4faff5640 1 -- 172.21.15.114:0/2124077603 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50c035040 con 0x561a241702f0 2024-05-27T07:52:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa4faff5640 1 -- 172.21.15.114:0/2124077603 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9466+0+0 (secure 0 0 0) 0x7fa50c005040 con 0x561a241702f0 2024-05-27T07:52:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.843+0000 7fa5241ef740 1 -- 172.21.15.114:0/2124077603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5080a9490 con 0x561a241702f0 2024-05-27T07:52:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.844+0000 7fa4faff5640 1 -- 172.21.15.114:0/2124077603 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fa50c054050 con 0x561a241702f0 2024-05-27T07:52:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.844+0000 7fa5241ef740 10 client.18525 fetch_fsmap learned FSMap version 73 2024-05-27T07:52:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.844+0000 7fa5241ef740 1 -- 172.21.15.114:0/2124077603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x561a245c3970 con 0x561a241702f0 2024-05-27T07:52:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.844+0000 7fa4faff5640 1 -- 172.21.15.114:0/2124077603 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5080a9490 con 0x561a241702f0 2024-05-27T07:52:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.844+0000 7fa4faff5640 1 -- 172.21.15.114:0/2124077603 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7fa50c04fbb0 con 0x561a241702f0 2024-05-27T07:52:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.844+0000 7fa5241ef740 10 client.18525 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:52:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.844+0000 7fa5241ef740 10 client.18525 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:52:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.844+0000 7fa5241ef740 20 client.18525 populate_metadata read hostname 'smithi114' 2024-05-27T07:52:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.844+0000 7fa5241ef740 1 --2- 172.21.15.114:0/2124077603 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561a241991b0 0x561a245c6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:52:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.844+0000 7fa5241ef740 4 client.18525 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:52:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.844+0000 7fa5241ef740 1 -- 172.21.15.114:0/2124077603 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561a244a8360 con 0x561a241991b0 2024-05-27T07:52:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.844+0000 7fa516ffd640 1 --2- 172.21.15.114:0/2124077603 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561a241991b0 0x561a245c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:52:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.845+0000 7fa4faff5640 10 client.18525 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:57.845+0000 7fa516ffd640 1 --2- 172.21.15.114:0/2124077603 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561a241991b0 0x561a245c6180 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db36b4e4dc64ea8b server_cookie=67ce811c2dc458d1 in_seq=0 out_seq=0 2024-05-27T07:52:58.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:52:58.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1271462 9.49 MB/sec execute 550 sec latency 220.131 ms 2024-05-27T07:52:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa4faff5640 1 -- 172.21.15.114:0/2124077603 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x561a244a8360 con 0x561a241991b0 2024-05-27T07:52:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa4faff5640 10 client.18525 handle_command_reply: tid=0 2024-05-27T07:52:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa5241ef740 1 client.18525 shutdown 2024-05-27T07:52:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa5241ef740 1 -- 172.21.15.114:0/2124077603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 msgr2=0x561a2454e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa5241ef740 1 --2- 172.21.15.114:0/2124077603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 0x561a2454e3c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009900 tx=0x7fa50c005ef0 comp rx=0 tx=0).stop 2024-05-27T07:52:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa5241ef740 1 -- 172.21.15.114:0/2124077603 shutdown_connections 2024-05-27T07:52:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa5241ef740 1 --2- 172.21.15.114:0/2124077603 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561a241991b0 0x561a245c6180 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa5241ef740 1 --2- 172.21.15.114:0/2124077603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561a241784d0 0x561a2454e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa4faff5640 0 client.18525 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa5241ef740 1 --2- 172.21.15.114:0/2124077603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561a241702f0 0x561a2454e3c0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa5241ef740 1 --2- 172.21.15.114:0/2124077603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a24157d80 0x561a24551050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa5241ef740 1 -- 172.21.15.114:0/2124077603 >> 172.21.15.114:0/2124077603 conn(0x561a2418d300 msgr2=0x561a24547530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa5241ef740 1 -- 172.21.15.114:0/2124077603 shutdown_connections 2024-05-27T07:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa5241ef740 1 -- 172.21.15.114:0/2124077603 wait complete. 2024-05-27T07:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.308+0000 7fa5241ef740 20 client.18525 trim_cache size 0 max 16384 2024-05-27T07:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:52:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.309+0000 7fa51d261640 1 -- 172.21.15.114:0/4264981942 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa5000b29d0 msgr2=0x7fa5000b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.309+0000 7fa51d261640 1 --2- 172.21.15.114:0/4264981942 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa5000b29d0 0x7fa5000b4ee0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa510023000 tx=0x7fa510021000 comp rx=0 tx=0).stop 2024-05-27T07:52:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.309+0000 7fa51d261640 1 -- 172.21.15.114:0/4264981942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5181032e0 msgr2=0x7fa5181a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:52:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.309+0000 7fa51d261640 1 --2- 172.21.15.114:0/4264981942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5181032e0 0x7fa5181a05f0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fa508005fd0 tx=0x7fa50800dde0 comp rx=0 tx=0).stop 2024-05-27T07:52:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.309+0000 7fa51d261640 1 -- 172.21.15.114:0/4264981942 shutdown_connections 2024-05-27T07:52:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.309+0000 7fa51d261640 1 --2- 172.21.15.114:0/4264981942 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa5000b29d0 0x7fa5000b4ee0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.309+0000 7fa51d261640 1 --2- 172.21.15.114:0/4264981942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa518109030 0x7fa51819a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.309+0000 7fa51d261640 1 --2- 172.21.15.114:0/4264981942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa518108b90 0x7fa5181a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.309+0000 7fa51d261640 1 --2- 172.21.15.114:0/4264981942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5181032e0 0x7fa5181a05f0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:52:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.309+0000 7fa51d261640 1 -- 172.21.15.114:0/4264981942 >> 172.21.15.114:0/4264981942 conn(0x7fa5180782d0 msgr2=0x7fa5181014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:52:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.309+0000 7fa51d261640 1 -- 172.21.15.114:0/4264981942 shutdown_connections 2024-05-27T07:52:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:52:58.309+0000 7fa51d261640 1 -- 172.21.15.114:0/4264981942 wait complete. 2024-05-27T07:52:58.530 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:58 smithi186 ceph-mon[36647]: pgmap v442: 129 pgs: 129 active+clean; 77 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 195 op/s 2024-05-27T07:52:58.531 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:52:58 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2304663137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:52:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:52:58.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:58 smithi114 ceph-mon[32038]: pgmap v442: 129 pgs: 129 active+clean; 77 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 195 op/s 2024-05-27T07:52:58.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:52:58 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2304663137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:52:58.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:58 smithi163 ceph-mon[36451]: pgmap v442: 129 pgs: 129 active+clean; 77 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 195 op/s 2024-05-27T07:52:58.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:52:58 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2304663137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:52:59.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1272456 9.48 MB/sec execute 551 sec latency 37.854 ms 2024-05-27T07:53:00.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.292+0000 7f4bd5217700 1 -- 172.21.15.114:0/3425781143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd01027d0 msgr2=0x7f4bd0102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:00.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.292+0000 7f4bd5217700 1 --2- 172.21.15.114:0/3425781143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd01027d0 0x7f4bd0102be0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009990 tx=0x7f4bc4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.293+0000 7f4bd5217700 1 -- 172.21.15.114:0/3425781143 shutdown_connections 2024-05-27T07:53:00.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.293+0000 7f4bd5217700 1 --2- 172.21.15.114:0/3425781143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bd0103120 0x7f4bd0111110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.293+0000 7f4bd5217700 1 --2- 172.21.15.114:0/3425781143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bd010a3d0 0x7f4bd010a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.293+0000 7f4bd5217700 1 --2- 172.21.15.114:0/3425781143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd01027d0 0x7f4bd0102be0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.293+0000 7f4bd5217700 1 -- 172.21.15.114:0/3425781143 >> 172.21.15.114:0/3425781143 conn(0x7f4bd00fe6e0 msgr2=0x7f4bd0100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:00.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.293+0000 7f4bd5217700 1 -- 172.21.15.114:0/3425781143 shutdown_connections 2024-05-27T07:53:00.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.293+0000 7f4bd5217700 1 -- 172.21.15.114:0/3425781143 wait complete. 2024-05-27T07:53:00.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.293+0000 7f4bd5217700 1 Processor -- start 2024-05-27T07:53:00.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.294+0000 7f4bd5217700 1 -- start start 2024-05-27T07:53:00.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.294+0000 7f4bd5217700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bd01027d0 0x7f4bd019ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.294+0000 7f4bd5217700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd0103120 0x7f4bd019b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.294+0000 7f4bd5217700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bd010a3d0 0x7f4bd019f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.294+0000 7f4bd5217700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01144a0 con 0x7f4bd0103120 2024-05-27T07:53:00.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.294+0000 7f4bd5217700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd0114320 con 0x7f4bd010a3d0 2024-05-27T07:53:00.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.294+0000 7f4bd5217700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd0114620 con 0x7f4bd01027d0 2024-05-27T07:53:00.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.294+0000 7f4bce59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd0103120 0x7f4bd019b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.294+0000 7f4bced9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bd01027d0 0x7f4bd019ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.294+0000 7f4bce59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd0103120 0x7f4bd019b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45526/0 (socket says 172.21.15.114:45526) 2024-05-27T07:53:00.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.294+0000 7f4bce59c700 1 -- 172.21.15.114:0/2799809541 learned_addr learned my addr 172.21.15.114:0/2799809541 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:00.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.294+0000 7f4bcf59e700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bd010a3d0 0x7f4bd019f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.295+0000 7f4bced9d700 1 -- 172.21.15.114:0/2799809541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bd010a3d0 msgr2=0x7f4bd019f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:00.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.295+0000 7f4bced9d700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bd010a3d0 0x7f4bd019f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.295+0000 7f4bced9d700 1 -- 172.21.15.114:0/2799809541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd0103120 msgr2=0x7f4bd019b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:00.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.295+0000 7f4bced9d700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd0103120 0x7f4bd019b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.295+0000 7f4bced9d700 1 -- 172.21.15.114:0/2799809541 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd019fd60 con 0x7f4bd01027d0 2024-05-27T07:53:00.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.295+0000 7f4bce59c700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd0103120 0x7f4bd019b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:00.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.295+0000 7f4bced9d700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bd01027d0 0x7f4bd019ae90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000e9f0 tx=0x7f4bc000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:00.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.295+0000 7f4bb7fff700 1 -- 172.21.15.114:0/2799809541 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc000c9c0 con 0x7f4bd01027d0 2024-05-27T07:53:00.298 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1273826 9.47 MB/sec execute 552 sec latency 70.587 ms 2024-05-27T07:53:00.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.295+0000 7f4bb7fff700 1 -- 172.21.15.114:0/2799809541 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc000cb60 con 0x7f4bd01027d0 2024-05-27T07:53:00.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.295+0000 7f4bd5217700 1 -- 172.21.15.114:0/2799809541 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd01a0050 con 0x7f4bd01027d0 2024-05-27T07:53:00.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.295+0000 7f4bb7fff700 1 -- 172.21.15.114:0/2799809541 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0018680 con 0x7f4bd01027d0 2024-05-27T07:53:00.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.295+0000 7f4bd5217700 1 -- 172.21.15.114:0/2799809541 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd01a7970 con 0x7f4bd01027d0 2024-05-27T07:53:00.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.298+0000 7f4bb7fff700 1 -- 172.21.15.114:0/2799809541 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f4bc0018820 con 0x7f4bd01027d0 2024-05-27T07:53:00.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.298+0000 7f4bb7fff700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4bb80b29f0 0x7f4bb80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.298+0000 7f4bce59c700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4bb80b29f0 0x7f4bb80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.298+0000 7f4bb7fff700 1 -- 172.21.15.114:0/2799809541 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9466+0+0 (secure 0 0 0) 0x7f4bc00e2ae0 con 0x7f4bd01027d0 2024-05-27T07:53:00.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4bce59c700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4bb80b29f0 0x7f4bb80b4f00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0100260 tx=0x7f4bc4005fd0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:00.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4be3c2cb80 1 Processor -- start 2024-05-27T07:53:00.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4be3c2cb80 1 -- start start 2024-05-27T07:53:00.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4be3c2cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 0x55bb28c8b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4be3c2cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c8bc50 0x55bb28c85a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4be3c2cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28c85f80 0x55bb28c86390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4be3c2cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bb28bb4c90 con 0x55bb28c85f80 2024-05-27T07:53:00.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4be3c2cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bb28ba4830 con 0x55bb28c8bc50 2024-05-27T07:53:00.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4be3c2cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bb28ba8870 con 0x55bb28b07e40 2024-05-27T07:53:00.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4bce59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c8bc50 0x55bb28c85a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4bced9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 0x55bb28c8b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4bce59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c8bc50 0x55bb28c85a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:46904/0 (socket says 172.21.15.114:46904) 2024-05-27T07:53:00.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4bce59c700 1 -- 172.21.15.114:0/3184095538 learned_addr learned my addr 172.21.15.114:0/3184095538 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:00.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.299+0000 7f4bced9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 0x55bb28c8b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33832/0 (socket says 172.21.15.114:33832) 2024-05-27T07:53:00.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.300+0000 7f4bcf59e700 1 --2- 172.21.15.114:0/3184095538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28c85f80 0x55bb28c86390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.300+0000 7f4bced9d700 1 -- 172.21.15.114:0/3184095538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c8bc50 msgr2=0x55bb28c85a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:00.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.300+0000 7f4bced9d700 1 --2- 172.21.15.114:0/3184095538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c8bc50 0x55bb28c85a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.300+0000 7f4bced9d700 1 -- 172.21.15.114:0/3184095538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28c85f80 msgr2=0x55bb28c86390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:00.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.300+0000 7f4bced9d700 1 --2- 172.21.15.114:0/3184095538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28c85f80 0x55bb28c86390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.300+0000 7f4bced9d700 1 -- 172.21.15.114:0/3184095538 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb28af48d0 con 0x55bb28b07e40 2024-05-27T07:53:00.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.300+0000 7f4bced9d700 1 --2- 172.21.15.114:0/3184095538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 0x55bb28c8b710 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00dfe80 tx=0x7f4bc00a9290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a615f5353ee59e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:00.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.300+0000 7f4bb57fa700 1 -- 172.21.15.114:0/3184095538 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0090e60 con 0x55bb28b07e40 2024-05-27T07:53:00.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.300+0000 7f4bb57fa700 1 -- 172.21.15.114:0/3184095538 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc00b21c0 con 0x55bb28b07e40 2024-05-27T07:53:00.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.300+0000 7f4bb57fa700 1 -- 172.21.15.114:0/3184095538 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc00a9ed0 con 0x55bb28b07e40 2024-05-27T07:53:00.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.301+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/3184095538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 msgr2=0x55bb28c8b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:00.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.301+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/3184095538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 0x55bb28c8b710 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00dfe80 tx=0x7f4bc00a9290 comp rx=0 tx=0).stop 2024-05-27T07:53:00.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.301+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/3184095538 shutdown_connections 2024-05-27T07:53:00.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.301+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/3184095538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28c85f80 0x55bb28c86390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.301+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/3184095538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 0x55bb28c8b710 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.301+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/3184095538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c8bc50 0x55bb28c85a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.301+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/3184095538 >> 172.21.15.114:0/3184095538 conn(0x55bb28b62af0 msgr2=0x55bb28c80640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:00.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.301+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/3184095538 shutdown_connections 2024-05-27T07:53:00.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.301+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/3184095538 wait complete. 2024-05-27T07:53:00.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4be3c2cb80 1 -- start start 2024-05-27T07:53:00.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4be3c2cb80 1 Processor -- start 2024-05-27T07:53:00.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4be3c2cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 0x55bb28c8f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4be3c2cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 0x55bb28c8f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4be3c2cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28c8bc50 0x55bb28c8fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4be3c2cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bb28aaafa0 con 0x55bb28b07e40 2024-05-27T07:53:00.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4be3c2cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bb28b467c0 con 0x55bb28c85f80 2024-05-27T07:53:00.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4be3c2cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bb28c92740 con 0x55bb28c8bc50 2024-05-27T07:53:00.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4be3c2cb80 10 client.0 _ll_register_callbacks cb 0x55bb28b3f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:00.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4bced9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 0x55bb28c8f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4bced9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 0x55bb28c8f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45546/0 (socket says 172.21.15.114:45546) 2024-05-27T07:53:00.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4bced9d700 1 -- 172.21.15.114:0/3438165158 learned_addr learned my addr 172.21.15.114:0/3438165158 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:00.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4bce59c700 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 0x55bb28c8f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.302+0000 7f4bcf59e700 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28c8bc50 0x55bb28c8fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4bced9d700 1 -- 172.21.15.114:0/3438165158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28c8bc50 msgr2=0x55bb28c8fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:00.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4bced9d700 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28c8bc50 0x55bb28c8fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4bced9d700 1 -- 172.21.15.114:0/3438165158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 msgr2=0x55bb28c8f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:00.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4bced9d700 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 0x55bb28c8f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4bced9d700 1 -- 172.21.15.114:0/3438165158 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb28b43690 con 0x55bb28b07e40 2024-05-27T07:53:00.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4bce59c700 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 0x55bb28c8f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:00.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4bced9d700 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 0x55bb28c8f440 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00a6c80 tx=0x7f4bc00a9f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8824f048442e34 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:00.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4bb2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:00.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4bb2ff5700 1 -- 172.21.15.114:0/3438165158 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc00c2070 con 0x55bb28b07e40 2024-05-27T07:53:00.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/3438165158 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bc00b21c0 con 0x55bb28b07e40 2024-05-27T07:53:00.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4bb2ff5700 1 -- 172.21.15.114:0/3438165158 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9466+0+0 (secure 0 0 0) 0x7f4bc00b25f0 con 0x55bb28b07e40 2024-05-27T07:53:00.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4bb2ff5700 1 -- 172.21.15.114:0/3438165158 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f4bc00bee60 con 0x55bb28b07e40 2024-05-27T07:53:00.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4be3c2cb80 10 client.18537 fetch_fsmap learned FSMap version 73 2024-05-27T07:53:00.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/3438165158 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55bb28c9d2a0 con 0x55bb28b07e40 2024-05-27T07:53:00.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.303+0000 7f4bb2ff5700 1 -- 172.21.15.114:0/3438165158 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc00b21c0 con 0x55bb28b07e40 2024-05-27T07:53:00.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.304+0000 7f4bb2ff5700 1 -- 172.21.15.114:0/3438165158 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7f4bc00a5380 con 0x55bb28b07e40 2024-05-27T07:53:00.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.304+0000 7f4be3c2cb80 10 client.18537 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:53:00.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.304+0000 7f4be3c2cb80 10 client.18537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:00.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.304+0000 7f4be3c2cb80 20 client.18537 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:00.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.304+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bb28c9f770 0x55bb28ca1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.304+0000 7f4be3c2cb80 4 client.18537 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:00.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.304+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/3438165158 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb28c519f0 con 0x55bb28c9f770 2024-05-27T07:53:00.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.304+0000 7f4bce59c700 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bb28c9f770 0x55bb28ca1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.304+0000 7f4bb2ff5700 10 client.18537 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:00.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.304+0000 7f4bce59c700 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bb28c9f770 0x55bb28ca1b90 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76d4f9755538fe9a server_cookie=c80947159211ea64 in_seq=0 out_seq=0 2024-05-27T07:53:00.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.305+0000 7f4bb2ff5700 1 -- 172.21.15.114:0/3438165158 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bb28c519f0 con 0x55bb28c9f770 2024-05-27T07:53:00.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.305+0000 7f4bb2ff5700 10 client.18537 handle_command_reply: tid=0 2024-05-27T07:53:00.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.306+0000 7f4be3c2cb80 1 client.18537 shutdown 2024-05-27T07:53:00.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.306+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/3438165158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 msgr2=0x55bb28c8f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:00.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.306+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 0x55bb28c8f440 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00a6c80 tx=0x7f4bc00a9f20 comp rx=0 tx=0).stop 2024-05-27T07:53:00.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.306+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/3438165158 shutdown_connections 2024-05-27T07:53:00.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.306+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28c8bc50 0x55bb28c8fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.306+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 0x55bb28c8f440 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.306+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bb28c9f770 0x55bb28ca1b90 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.306+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/3438165158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 0x55bb28c8f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.306+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/3438165158 >> 172.21.15.114:0/3438165158 conn(0x55bb28b62af0 msgr2=0x55bb28c84a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:00.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.306+0000 7f4bb2ff5700 0 client.18537 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:00.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.306+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/3438165158 shutdown_connections 2024-05-27T07:53:00.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.306+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/3438165158 wait complete. 2024-05-27T07:53:00.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.306+0000 7f4be3c2cb80 20 client.18537 trim_cache size 0 max 16384 2024-05-27T07:53:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.321+0000 7f4be3c2cb80 1 Processor -- start 2024-05-27T07:53:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4be3c2cb80 1 -- start start 2024-05-27T07:53:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4be3c2cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 0x55bb28c85a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4be3c2cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 0x55bb28ca1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4be3c2cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28c8bc50 0x55bb28c8ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4be3c2cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01b2070 con 0x55bb28b07e40 2024-05-27T07:53:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4be3c2cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01b07e0 con 0x55bb28c85f80 2024-05-27T07:53:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4be3c2cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01ae630 con 0x55bb28c8bc50 2024-05-27T07:53:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4bced9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 0x55bb28c85a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4bced9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 0x55bb28c85a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45560/0 (socket says 172.21.15.114:45560) 2024-05-27T07:53:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4bced9d700 1 -- 172.21.15.114:0/1414480436 learned_addr learned my addr 172.21.15.114:0/1414480436 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:00.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4bcf59e700 1 --2- 172.21.15.114:0/1414480436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28c8bc50 0x55bb28c8ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4bce59c700 1 --2- 172.21.15.114:0/1414480436 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 0x55bb28ca1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4bced9d700 1 -- 172.21.15.114:0/1414480436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28c8bc50 msgr2=0x55bb28c8ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4bced9d700 1 --2- 172.21.15.114:0/1414480436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28c8bc50 0x55bb28c8ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4bced9d700 1 -- 172.21.15.114:0/1414480436 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 msgr2=0x55bb28ca1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4bced9d700 1 --2- 172.21.15.114:0/1414480436 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 0x55bb28ca1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.322+0000 7f4bced9d700 1 -- 172.21.15.114:0/1414480436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb28c80890 con 0x55bb28b07e40 2024-05-27T07:53:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.323+0000 7f4bced9d700 1 --2- 172.21.15.114:0/1414480436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 0x55bb28c85a40 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00a5a90 tx=0x7f4bc0009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427af9c2255bc2de server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.323+0000 7f4bb4ff9700 1 -- 172.21.15.114:0/1414480436 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc00c2030 con 0x55bb28b07e40 2024-05-27T07:53:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.323+0000 7f4bb4ff9700 1 -- 172.21.15.114:0/1414480436 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc00b21c0 con 0x55bb28b07e40 2024-05-27T07:53:00.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.323+0000 7f4bb4ff9700 1 -- 172.21.15.114:0/1414480436 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0090e60 con 0x55bb28b07e40 2024-05-27T07:53:00.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.324+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/1414480436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 msgr2=0x55bb28c85a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:00.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.324+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/1414480436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 0x55bb28c85a40 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00a5a90 tx=0x7f4bc0009ef0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.324+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/1414480436 shutdown_connections 2024-05-27T07:53:00.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.324+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/1414480436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28c8bc50 0x55bb28c8ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.324+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/1414480436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28b07e40 0x55bb28c85a40 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.324+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/1414480436 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 0x55bb28ca1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.324+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/1414480436 >> 172.21.15.114:0/1414480436 conn(0x55bb28b62af0 msgr2=0x55bb28c8a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:00.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.324+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/1414480436 shutdown_connections 2024-05-27T07:53:00.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.324+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/1414480436 wait complete. 2024-05-27T07:53:00.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.324+0000 7f4be3c2cb80 1 -- start start 2024-05-27T07:53:00.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.324+0000 7f4be3c2cb80 1 Processor -- start 2024-05-27T07:53:00.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4be3c2cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 0x55bb28c979d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4be3c2cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 0x55bb28c97f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4be3c2cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28c8bc50 0x55bb28ca62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4be3c2cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01aeca0 con 0x55bb28c8bc50 2024-05-27T07:53:00.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4be3c2cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01b0110 con 0x55bb28c85f80 2024-05-27T07:53:00.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4be3c2cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01af370 con 0x55bb28b07e40 2024-05-27T07:53:00.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4be3c2cb80 10 client.0 _ll_register_callbacks cb 0x55bb28b3f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:00.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4bcf59e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28c8bc50 0x55bb28ca62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4bced9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 0x55bb28c979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4bcf59e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28c8bc50 0x55bb28ca62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45574/0 (socket says 172.21.15.114:45574) 2024-05-27T07:53:00.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4bcf59e700 1 -- 172.21.15.114:0/2252107220 learned_addr learned my addr 172.21.15.114:0/2252107220 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:00.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4bced9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 0x55bb28c979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33852/0 (socket says 172.21.15.114:33852) 2024-05-27T07:53:00.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4bce59c700 1 --2- 172.21.15.114:0/2252107220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 0x55bb28c97f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4bced9d700 1 -- 172.21.15.114:0/2252107220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 msgr2=0x55bb28c97f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4bced9d700 1 --2- 172.21.15.114:0/2252107220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 0x55bb28c97f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4bced9d700 1 -- 172.21.15.114:0/2252107220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28c8bc50 msgr2=0x55bb28ca62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4bced9d700 1 --2- 172.21.15.114:0/2252107220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28c8bc50 0x55bb28ca62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4bced9d700 1 -- 172.21.15.114:0/2252107220 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb28c985f0 con 0x55bb28b07e40 2024-05-27T07:53:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4bced9d700 1 --2- 172.21.15.114:0/2252107220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 0x55bb28c979d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00e0400 tx=0x7f4bc00a92b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1ea7f218cda3b95 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.325+0000 7f4bb2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:53:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4bb2ff5700 1 -- 172.21.15.114:0/2252107220 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc00c2030 con 0x55bb28b07e40 2024-05-27T07:53:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/2252107220 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bc00b21c0 con 0x55bb28b07e40 2024-05-27T07:53:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4bb2ff5700 1 -- 172.21.15.114:0/2252107220 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9466+0+0 (secure 0 0 0) 0x7f4bc00cc0b0 con 0x55bb28b07e40 2024-05-27T07:53:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4bb2ff5700 1 -- 172.21.15.114:0/2252107220 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f4bc0090e60 con 0x55bb28b07e40 2024-05-27T07:53:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4be3c2cb80 10 client.28087 fetch_fsmap learned FSMap version 73 2024-05-27T07:53:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/2252107220 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55bb28d09740 con 0x55bb28b07e40 2024-05-27T07:53:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4bb2ff5700 1 -- 172.21.15.114:0/2252107220 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc00b21c0 con 0x55bb28b07e40 2024-05-27T07:53:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4bb2ff5700 1 -- 172.21.15.114:0/2252107220 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7f4bc00a8870 con 0x55bb28b07e40 2024-05-27T07:53:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4be3c2cb80 10 client.28087 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:53:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4be3c2cb80 10 client.28087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4be3c2cb80 20 client.28087 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/2252107220 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bb28d09a40 0x55bb28d0be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4be3c2cb80 4 client.28087 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:53:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/2252107220 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb28bb6be0 con 0x55bb28d09a40 2024-05-27T07:53:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.326+0000 7f4bce59c700 1 --2- 172.21.15.114:0/2252107220 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bb28d09a40 0x55bb28d0be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4bb2ff5700 10 client.28087 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4bce59c700 1 --2- 172.21.15.114:0/2252107220 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bb28d09a40 0x55bb28d0be60 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9875e1006438b393 server_cookie=a6ba9429abc2c8b5 in_seq=0 out_seq=0 2024-05-27T07:53:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4bb2ff5700 1 -- 172.21.15.114:0/2252107220 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb28bb6be0 con 0x55bb28d09a40 2024-05-27T07:53:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4bb2ff5700 10 client.28087 handle_command_reply: tid=0 2024-05-27T07:53:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4be3c2cb80 1 client.28087 shutdown 2024-05-27T07:53:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/2252107220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 msgr2=0x55bb28c979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/2252107220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 0x55bb28c979d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00e0400 tx=0x7f4bc00a92b0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/2252107220 shutdown_connections 2024-05-27T07:53:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/2252107220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bb28c8bc50 0x55bb28ca62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/2252107220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb28b07e40 0x55bb28c979d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/2252107220 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bb28d09a40 0x55bb28d0be60 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4be3c2cb80 1 --2- 172.21.15.114:0/2252107220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb28c85f80 0x55bb28c97f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4bb2ff5700 0 client.28087 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/2252107220 >> 172.21.15.114:0/2252107220 conn(0x55bb28b62af0 msgr2=0x55bb28c8f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/2252107220 shutdown_connections 2024-05-27T07:53:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.327+0000 7f4be3c2cb80 1 -- 172.21.15.114:0/2252107220 wait complete. 2024-05-27T07:53:00.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.328+0000 7f4be3c2cb80 20 client.28087 trim_cache size 0 max 16384 2024-05-27T07:53:00.334 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:53:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.328+0000 7f4bd5217700 1 -- 172.21.15.114:0/2799809541 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4bb80b29f0 msgr2=0x7f4bb80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.328+0000 7f4bd5217700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4bb80b29f0 0x7f4bb80b4f00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0100260 tx=0x7f4bc4005fd0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.328+0000 7f4bd5217700 1 -- 172.21.15.114:0/2799809541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bd01027d0 msgr2=0x7f4bd019ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.328+0000 7f4bd5217700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bd01027d0 0x7f4bd019ae90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000e9f0 tx=0x7f4bc000ed00 comp rx=0 tx=0).stop 2024-05-27T07:53:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.328+0000 7f4bd5217700 1 -- 172.21.15.114:0/2799809541 shutdown_connections 2024-05-27T07:53:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.328+0000 7f4bd5217700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4bb80b29f0 0x7f4bb80b4f00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.328+0000 7f4bd5217700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bd010a3d0 0x7f4bd019f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.328+0000 7f4bd5217700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bd01027d0 0x7f4bd019ae90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.328+0000 7f4bd5217700 1 --2- 172.21.15.114:0/2799809541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd0103120 0x7f4bd019b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:00.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.328+0000 7f4bd5217700 1 -- 172.21.15.114:0/2799809541 >> 172.21.15.114:0/2799809541 conn(0x7f4bd00fe6e0 msgr2=0x7f4bd0100c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:00.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.328+0000 7f4bd5217700 1 -- 172.21.15.114:0/2799809541 shutdown_connections 2024-05-27T07:53:00.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:00.329+0000 7f4bd5217700 1 -- 172.21.15.114:0/2799809541 wait complete. 2024-05-27T07:53:00.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:00 smithi114 ceph-mon[32038]: pgmap v443: 129 pgs: 129 active+clean; 77 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 195 op/s 2024-05-27T07:53:00.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:00 smithi163 ceph-mon[36451]: pgmap v443: 129 pgs: 129 active+clean; 77 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 195 op/s 2024-05-27T07:53:00.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:00 smithi186 ceph-mon[36647]: pgmap v443: 129 pgs: 129 active+clean; 77 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 195 op/s 2024-05-27T07:53:01.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1275111 9.48 MB/sec execute 553 sec latency 108.629 ms 2024-05-27T07:53:02.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1275128 9.46 MB/sec execute 554 sec latency 134.639 ms 2024-05-27T07:53:02.412 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:53:02.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:53:02.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:02 smithi114 ceph-mon[32038]: pgmap v444: 129 pgs: 129 active+clean; 66 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 331 op/s 2024-05-27T07:53:02.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:02 smithi163 ceph-mon[36451]: pgmap v444: 129 pgs: 129 active+clean; 66 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 331 op/s 2024-05-27T07:53:02.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:02 smithi186 ceph-mon[36647]: pgmap v444: 129 pgs: 129 active+clean; 66 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 331 op/s 2024-05-27T07:53:02.864 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:03.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1275130 9.44 MB/sec execute 555 sec latency 436.260 ms 2024-05-27T07:53:03.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:03 smithi114 ceph-mon[32038]: osdmap e130: 12 total, 12 up, 12 in 2024-05-27T07:53:03.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:03 smithi163 ceph-mon[36451]: osdmap e130: 12 total, 12 up, 12 in 2024-05-27T07:53:03.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:03 smithi186 ceph-mon[36647]: osdmap e130: 12 total, 12 up, 12 in 2024-05-27T07:53:04.295 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1275190 9.43 MB/sec execute 556 sec latency 472.468 ms 2024-05-27T07:53:04.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:04 smithi114 ceph-mon[32038]: pgmap v446: 129 pgs: 129 active+clean; 66 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 335 op/s 2024-05-27T07:53:04.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:04 smithi163 ceph-mon[36451]: pgmap v446: 129 pgs: 129 active+clean; 66 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 335 op/s 2024-05-27T07:53:04.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:04 smithi186 ceph-mon[36647]: pgmap v446: 129 pgs: 129 active+clean; 66 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 335 op/s 2024-05-27T07:53:05.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1275353 9.41 MB/sec execute 557 sec latency 184.272 ms 2024-05-27T07:53:05.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.441+0000 7fa050f3c700 1 -- 172.21.15.114:0/18525991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04c107c00 msgr2=0x7fa04c108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:05.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.441+0000 7fa050f3c700 1 --2- 172.21.15.114:0/18525991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04c107c00 0x7fa04c108010 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa03c009990 tx=0x7fa03c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.445+0000 7fa050f3c700 1 -- 172.21.15.114:0/18525991 shutdown_connections 2024-05-27T07:53:05.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.445+0000 7fa050f3c700 1 --2- 172.21.15.114:0/18525991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa04c108550 0x7fa04c10feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.445+0000 7fa050f3c700 1 --2- 172.21.15.114:0/18525991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa04c1032d0 0x7fa04c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.445+0000 7fa050f3c700 1 --2- 172.21.15.114:0/18525991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04c107c00 0x7fa04c108010 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.445+0000 7fa050f3c700 1 -- 172.21.15.114:0/18525991 >> 172.21.15.114:0/18525991 conn(0x7fa04c078ab0 msgr2=0x7fa04c078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:05.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.447+0000 7fa050f3c700 1 -- 172.21.15.114:0/18525991 shutdown_connections 2024-05-27T07:53:05.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.449+0000 7fa050f3c700 1 -- 172.21.15.114:0/18525991 wait complete. 2024-05-27T07:53:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.450+0000 7fa050f3c700 1 Processor -- start 2024-05-27T07:53:05.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.456+0000 7fa050f3c700 1 -- start start 2024-05-27T07:53:05.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.456+0000 7fa050f3c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa04c1032d0 0x7fa04c071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.456+0000 7fa050f3c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa04c107c00 0x7fa04c071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.456+0000 7fa050f3c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04c108550 0x7fa04c0721e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.456+0000 7fa050f3c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c06dab0 con 0x7fa04c107c00 2024-05-27T07:53:05.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.456+0000 7fa050f3c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c06d930 con 0x7fa04c1032d0 2024-05-27T07:53:05.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.457+0000 7fa050f3c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c06dc30 con 0x7fa04c108550 2024-05-27T07:53:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.458+0000 7fa04b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa04c1032d0 0x7fa04c071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.458+0000 7fa04b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa04c1032d0 0x7fa04c071760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37964/0 (socket says 172.21.15.114:37964) 2024-05-27T07:53:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.458+0000 7fa04b7fe700 1 -- 172.21.15.114:0/3528169540 learned_addr learned my addr 172.21.15.114:0/3528169540 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.459+0000 7fa04affd700 1 --2- 172.21.15.114:0/3528169540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa04c107c00 0x7fa04c071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.459+0000 7fa04b7fe700 1 -- 172.21.15.114:0/3528169540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04c108550 msgr2=0x7fa04c0721e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:53:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.459+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/3528169540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04c108550 0x7fa04c0721e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.459+0000 7fa04b7fe700 1 -- 172.21.15.114:0/3528169540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa04c107c00 msgr2=0x7fa04c071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.459+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/3528169540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa04c107c00 0x7fa04c071ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.459+0000 7fa04b7fe700 1 -- 172.21.15.114:0/3528169540 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04c1b1870 con 0x7fa04c1032d0 2024-05-27T07:53:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.460+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/3528169540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa04c1032d0 0x7fa04c071760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa04000e9d0 tx=0x7fa04000ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.462+0000 7fa048ff9700 1 -- 172.21.15.114:0/3528169540 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04000cbd0 con 0x7fa04c1032d0 2024-05-27T07:53:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.462+0000 7fa050f3c700 1 -- 172.21.15.114:0/3528169540 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa04c1b1b60 con 0x7fa04c1032d0 2024-05-27T07:53:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.462+0000 7fa050f3c700 1 -- 172.21.15.114:0/3528169540 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa04c1b20f0 con 0x7fa04c1032d0 2024-05-27T07:53:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.462+0000 7fa048ff9700 1 -- 172.21.15.114:0/3528169540 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa04000cd70 con 0x7fa04c1032d0 2024-05-27T07:53:05.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.462+0000 7fa048ff9700 1 -- 172.21.15.114:0/3528169540 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040018710 con 0x7fa04c1032d0 2024-05-27T07:53:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.473+0000 7fa060bb3b80 1 Processor -- start 2024-05-27T07:53:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.475+0000 7fa048ff9700 1 -- 172.21.15.114:0/3528169540 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa040010970 con 0x7fa04c1032d0 2024-05-27T07:53:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.476+0000 7fa048ff9700 1 --2- 172.21.15.114:0/3528169540 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0340b2aa0 0x7fa0340b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.476+0000 7fa04bfff700 1 --2- 172.21.15.114:0/3528169540 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0340b2aa0 0x7fa0340b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.477+0000 7fa060bb3b80 1 -- start start 2024-05-27T07:53:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.477+0000 7fa04bfff700 1 --2- 172.21.15.114:0/3528169540 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0340b2aa0 0x7fa0340b4fb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa04400ab30 tx=0x7fa044009250 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.477+0000 7fa048ff9700 1 -- 172.21.15.114:0/3528169540 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9642+0+0 (secure 0 0 0) 0x7fa0400e2d00 con 0x7fa04c1032d0 2024-05-27T07:53:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.480+0000 7fa060bb3b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5bf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.480+0000 7fa060bb3b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae5bf7b0 0x55edae5c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.480+0000 7fa060bb3b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bfbc0 0x55edae5cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.480+0000 7fa060bb3b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55edae4eddf0 con 0x55edae5bf7b0 2024-05-27T07:53:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.480+0000 7fa060bb3b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55edae4dd990 con 0x55edae5bfbc0 2024-05-27T07:53:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.480+0000 7fa060bb3b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55edae4e19d0 con 0x55edae440ec0 2024-05-27T07:53:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.480+0000 7fa04b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5bf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.480+0000 7fa04b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5bf270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35262/0 (socket says 172.21.15.114:35262) 2024-05-27T07:53:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.480+0000 7fa04b7fe700 1 -- 172.21.15.114:0/1974356971 learned_addr learned my addr 172.21.15.114:0/1974356971 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.482+0000 7fa04affd700 1 --2- 172.21.15.114:0/1974356971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae5bf7b0 0x55edae5c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.483+0000 7fa04bfff700 1 --2- 172.21.15.114:0/1974356971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bfbc0 0x55edae5cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.484+0000 7fa04b7fe700 1 -- 172.21.15.114:0/1974356971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bfbc0 msgr2=0x55edae5cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:05.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.484+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/1974356971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bfbc0 0x55edae5cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.484+0000 7fa04b7fe700 1 -- 172.21.15.114:0/1974356971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae5bf7b0 msgr2=0x55edae5c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:05.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.484+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/1974356971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae5bf7b0 0x55edae5c9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.484+0000 7fa04b7fe700 1 -- 172.21.15.114:0/1974356971 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edae487030 con 0x55edae440ec0 2024-05-27T07:53:05.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.484+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/1974356971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5bf270 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa0400a9820 tx=0x7fa0400a5a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8e7ab21ff8ca8af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:05.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.488+0000 7fa031ffb700 1 -- 172.21.15.114:0/1974356971 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0400a9b20 con 0x55edae440ec0 2024-05-27T07:53:05.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.488+0000 7fa031ffb700 1 -- 172.21.15.114:0/1974356971 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0400aae50 con 0x55edae440ec0 2024-05-27T07:53:05.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.490+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1974356971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 msgr2=0x55edae5bf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:05.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.490+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1974356971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5bf270 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa0400a9820 tx=0x7fa0400a5a60 comp rx=0 tx=0).stop 2024-05-27T07:53:05.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.490+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1974356971 shutdown_connections 2024-05-27T07:53:05.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.491+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1974356971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bfbc0 0x55edae5cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.491+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1974356971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5bf270 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.491+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1974356971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae5bf7b0 0x55edae5c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.491+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1974356971 >> 172.21.15.114:0/1974356971 conn(0x55edae4437f0 msgr2=0x55edae5bb140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:05.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.492+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1974356971 shutdown_connections 2024-05-27T07:53:05.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.495+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1974356971 wait complete. 2024-05-27T07:53:05.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.495+0000 7fa060bb3b80 1 -- start start 2024-05-27T07:53:05.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.496+0000 7fa060bb3b80 1 Processor -- start 2024-05-27T07:53:05.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.497+0000 7fa060bb3b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.497+0000 7fa060bb3b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae5bf7b0 0x55edae5c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.497+0000 7fa060bb3b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bfbc0 0x55edae5ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.497+0000 7fa04b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.497+0000 7fa04b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5c99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35268/0 (socket says 172.21.15.114:35268) 2024-05-27T07:53:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.497+0000 7fa04b7fe700 1 -- 172.21.15.114:0/1432012845 learned_addr learned my addr 172.21.15.114:0/1432012845 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.497+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1432012845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55edae3e4020 con 0x55edae5bf7b0 2024-05-27T07:53:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.497+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1432012845 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55edae445e20 con 0x55edae5bfbc0 2024-05-27T07:53:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.497+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1432012845 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55edae5d55c0 con 0x55edae440ec0 2024-05-27T07:53:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.497+0000 7fa060bb3b80 10 client.0 _ll_register_callbacks cb 0x55edae474630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.497+0000 7fa04bfff700 1 --2- 172.21.15.114:0/1432012845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bfbc0 0x55edae5ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.498+0000 7fa04bfff700 1 -- 172.21.15.114:0/1432012845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 msgr2=0x55edae5c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.498+0000 7fa04bfff700 1 --2- 172.21.15.114:0/1432012845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5c99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.498+0000 7fa04bfff700 1 -- 172.21.15.114:0/1432012845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae5bf7b0 msgr2=0x55edae5c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.498+0000 7fa04bfff700 1 --2- 172.21.15.114:0/1432012845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae5bf7b0 0x55edae5c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.498+0000 7fa04bfff700 1 -- 172.21.15.114:0/1432012845 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edae42d950 con 0x55edae5bfbc0 2024-05-27T07:53:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.498+0000 7fa04bfff700 1 --2- 172.21.15.114:0/1432012845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bfbc0 0x55edae5ca420 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa04400c690 tx=0x7fa044004b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4375b062b948afc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.500+0000 7fa02affd700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:53:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.500+0000 7fa02affd700 1 -- 172.21.15.114:0/1432012845 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04401a780 con 0x55edae5bfbc0 2024-05-27T07:53:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.500+0000 7fa02affd700 1 -- 172.21.15.114:0/1432012845 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9642+0+0 (secure 0 0 0) 0x7fa04401a920 con 0x55edae5bfbc0 2024-05-27T07:53:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.501+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1432012845 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0400aae50 con 0x55edae5bfbc0 2024-05-27T07:53:05.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.503+0000 7fa02affd700 1 -- 172.21.15.114:0/1432012845 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fa04401ab20 con 0x55edae5bfbc0 2024-05-27T07:53:05.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.504+0000 7fa060bb3b80 10 client.27953 fetch_fsmap learned FSMap version 73 2024-05-27T07:53:05.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.504+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1432012845 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55edae4977e0 con 0x55edae5bfbc0 2024-05-27T07:53:05.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.505+0000 7fa02affd700 1 -- 172.21.15.114:0/1432012845 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0400aae50 con 0x55edae5bfbc0 2024-05-27T07:53:05.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.505+0000 7fa02affd700 1 -- 172.21.15.114:0/1432012845 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7fa044024430 con 0x55edae5bfbc0 2024-05-27T07:53:05.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.505+0000 7fa060bb3b80 10 client.27953 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:53:05.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.505+0000 7fa060bb3b80 10 client.27953 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:05.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.505+0000 7fa060bb3b80 20 client.27953 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:05.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.505+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1432012845 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edae5e29b0 0x55edae5e4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.505+0000 7fa060bb3b80 4 client.27953 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:05.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.505+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1432012845 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edae4efd40 con 0x55edae5e29b0 2024-05-27T07:53:05.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.506+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/1432012845 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edae5e29b0 0x55edae5e4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.508+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/1432012845 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edae5e29b0 0x55edae5e4ea0 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=473eb078ac6dfc02 server_cookie=2a37db98c985322f in_seq=0 out_seq=0 2024-05-27T07:53:05.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.508+0000 7fa02affd700 10 client.27953 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:05.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.509+0000 7fa02affd700 1 -- 172.21.15.114:0/1432012845 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55edae4efd40 con 0x55edae5e29b0 2024-05-27T07:53:05.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.509+0000 7fa02affd700 10 client.27953 handle_command_reply: tid=0 2024-05-27T07:53:05.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.509+0000 7fa060bb3b80 1 client.27953 shutdown 2024-05-27T07:53:05.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.511+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1432012845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bfbc0 msgr2=0x55edae5ca420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:05.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.511+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1432012845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bfbc0 0x55edae5ca420 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa04400c690 tx=0x7fa044004b60 comp rx=0 tx=0).stop 2024-05-27T07:53:05.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.511+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1432012845 shutdown_connections 2024-05-27T07:53:05.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.511+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1432012845 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edae5e29b0 0x55edae5e4ea0 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.511+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1432012845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bfbc0 0x55edae5ca420 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.511+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1432012845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.511+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1432012845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae5bf7b0 0x55edae5c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.511+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1432012845 >> 172.21.15.114:0/1432012845 conn(0x55edae4437f0 msgr2=0x55edae5c0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:05.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.512+0000 7fa02affd700 0 client.27953 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:05.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.512+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1432012845 shutdown_connections 2024-05-27T07:53:05.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.512+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1432012845 wait complete. 2024-05-27T07:53:05.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.512+0000 7fa060bb3b80 20 client.27953 trim_cache size 0 max 16384 2024-05-27T07:53:05.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.551+0000 7fa060bb3b80 1 Processor -- start 2024-05-27T07:53:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.551+0000 7fa060bb3b80 1 -- start start 2024-05-27T07:53:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.552+0000 7fa060bb3b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae440ec0 0x55edae5c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.552+0000 7fa060bb3b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bf7b0 0x55edae5ca260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.552+0000 7fa04b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae440ec0 0x55edae5c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.552+0000 7fa04b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae440ec0 0x55edae5c3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57698/0 (socket says 172.21.15.114:57698) 2024-05-27T07:53:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.552+0000 7fa04b7fe700 1 -- 172.21.15.114:0/1991571074 learned_addr learned my addr 172.21.15.114:0/1991571074 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.552+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1991571074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae5bfbc0 0x55edae5dddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.552+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1991571074 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa038010a30 con 0x55edae440ec0 2024-05-27T07:53:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.552+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1991571074 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa03800f1a0 con 0x55edae5bf7b0 2024-05-27T07:53:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.552+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1991571074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa03800d010 con 0x55edae5bfbc0 2024-05-27T07:53:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.552+0000 7fa04affd700 1 --2- 172.21.15.114:0/1991571074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bf7b0 0x55edae5ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.554+0000 7fa04b7fe700 1 -- 172.21.15.114:0/1991571074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae5bfbc0 msgr2=0x55edae5dddb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:53:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.554+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/1991571074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae5bfbc0 0x55edae5dddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.554+0000 7fa04b7fe700 1 -- 172.21.15.114:0/1991571074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bf7b0 msgr2=0x55edae5ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.554+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/1991571074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bf7b0 0x55edae5ca260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.554+0000 7fa04b7fe700 1 -- 172.21.15.114:0/1991571074 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edae5bb390 con 0x55edae440ec0 2024-05-27T07:53:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.554+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/1991571074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae440ec0 0x55edae5c3b60 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fa04000cfd0 tx=0x7fa040091040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2e91a0cffbe97c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.555+0000 7fa0317fa700 1 -- 172.21.15.114:0/1991571074 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0400aae50 con 0x55edae440ec0 2024-05-27T07:53:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.555+0000 7fa0317fa700 1 -- 172.21.15.114:0/1991571074 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0400a9b20 con 0x55edae440ec0 2024-05-27T07:53:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.555+0000 7fa0317fa700 1 -- 172.21.15.114:0/1991571074 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040010430 con 0x55edae440ec0 2024-05-27T07:53:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.556+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1991571074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae440ec0 msgr2=0x55edae5c3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.556+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1991571074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae440ec0 0x55edae5c3b60 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fa04000cfd0 tx=0x7fa040091040 comp rx=0 tx=0).stop 2024-05-27T07:53:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.556+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1991571074 shutdown_connections 2024-05-27T07:53:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.556+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1991571074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae5bfbc0 0x55edae5dddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.556+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1991571074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae440ec0 0x55edae5c3b60 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.556+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/1991571074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bf7b0 0x55edae5ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.556+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1991571074 >> 172.21.15.114:0/1991571074 conn(0x55edae4437f0 msgr2=0x55edae5dd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:05.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.561+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1991571074 shutdown_connections 2024-05-27T07:53:05.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.561+0000 7fa060bb3b80 1 -- 172.21.15.114:0/1991571074 wait complete. 2024-05-27T07:53:05.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.561+0000 7fa060bb3b80 1 -- start start 2024-05-27T07:53:05.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.562+0000 7fa060bb3b80 1 Processor -- start 2024-05-27T07:53:05.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.562+0000 7fa060bb3b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5d6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.563+0000 7fa060bb3b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bf7b0 0x55edae5d72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.563+0000 7fa060bb3b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae5bfbc0 0x55edae64c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.563+0000 7fa060bb3b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa03800d660 con 0x55edae5bfbc0 2024-05-27T07:53:05.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.563+0000 7fa060bb3b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa03800ead0 con 0x55edae5bf7b0 2024-05-27T07:53:05.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.563+0000 7fa060bb3b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa03800dd30 con 0x55edae440ec0 2024-05-27T07:53:05.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.563+0000 7fa060bb3b80 10 client.0 _ll_register_callbacks cb 0x55edae474630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:05.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.563+0000 7fa04affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bf7b0 0x55edae5d72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.564+0000 7fa04affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bf7b0 0x55edae5d72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37998/0 (socket says 172.21.15.114:37998) 2024-05-27T07:53:05.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.564+0000 7fa04affd700 1 -- 172.21.15.114:0/3243659015 learned_addr learned my addr 172.21.15.114:0/3243659015 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:05.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.563+0000 7fa04b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5d6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.565+0000 7fa04b7fe700 1 -- 172.21.15.114:0/3243659015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bf7b0 msgr2=0x55edae5d72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:05.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.565+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/3243659015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bf7b0 0x55edae5d72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.565+0000 7fa04b7fe700 1 -- 172.21.15.114:0/3243659015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae5bfbc0 msgr2=0x55edae64c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:05.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.565+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/3243659015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae5bfbc0 0x55edae64c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.565+0000 7fa04b7fe700 1 -- 172.21.15.114:0/3243659015 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edae64cab0 con 0x55edae440ec0 2024-05-27T07:53:05.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.570+0000 7fa04b7fe700 1 --2- 172.21.15.114:0/3243659015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5d6d60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa0400a9820 tx=0x7fa0400a9790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41a2cd02c7f8d425 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:05.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.571+0000 7fa02affd700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:53:05.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.575+0000 7fa02affd700 1 -- 172.21.15.114:0/3243659015 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0400c2020 con 0x55edae440ec0 2024-05-27T07:53:05.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.575+0000 7fa060bb3b80 1 -- 172.21.15.114:0/3243659015 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0400a9b20 con 0x55edae440ec0 2024-05-27T07:53:05.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.575+0000 7fa02affd700 1 -- 172.21.15.114:0/3243659015 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9642+0+0 (secure 0 0 0) 0x7fa0400a5630 con 0x55edae440ec0 2024-05-27T07:53:05.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.581+0000 7fa02affd700 1 -- 172.21.15.114:0/3243659015 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fa0400b2980 con 0x55edae440ec0 2024-05-27T07:53:05.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.582+0000 7fa060bb3b80 10 client.28099 fetch_fsmap learned FSMap version 73 2024-05-27T07:53:05.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.582+0000 7fa060bb3b80 1 -- 172.21.15.114:0/3243659015 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55edae64dd20 con 0x55edae440ec0 2024-05-27T07:53:05.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.584+0000 7fa02affd700 1 -- 172.21.15.114:0/3243659015 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0400a8f90 con 0x55edae440ec0 2024-05-27T07:53:05.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.586+0000 7fa02affd700 1 -- 172.21.15.114:0/3243659015 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7fa0400a8960 con 0x55edae440ec0 2024-05-27T07:53:05.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.586+0000 7fa060bb3b80 10 client.28099 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:53:05.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.586+0000 7fa060bb3b80 10 client.28099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:05.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.586+0000 7fa060bb3b80 20 client.28099 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.586+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/3243659015 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edae64e2d0 0x55edae6506f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.586+0000 7fa060bb3b80 4 client.28099 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:53:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.586+0000 7fa060bb3b80 1 -- 172.21.15.114:0/3243659015 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fa0400a9b20 con 0x55edae64e2d0 2024-05-27T07:53:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.586+0000 7fa04affd700 1 --2- 172.21.15.114:0/3243659015 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edae64e2d0 0x55edae6506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.587+0000 7fa04affd700 1 --2- 172.21.15.114:0/3243659015 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edae64e2d0 0x55edae6506f0 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77e0af70a0bb4ce0 server_cookie=7b8540abe49e7caa in_seq=0 out_seq=0 2024-05-27T07:53:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.587+0000 7fa02affd700 10 client.28099 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:05.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.592+0000 7fa02affd700 1 -- 172.21.15.114:0/3243659015 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fa0400a9b20 con 0x55edae64e2d0 2024-05-27T07:53:05.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.592+0000 7fa02affd700 10 client.28099 handle_command_reply: tid=0 2024-05-27T07:53:05.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.592+0000 7fa060bb3b80 1 client.28099 shutdown 2024-05-27T07:53:05.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.592+0000 7fa060bb3b80 1 -- 172.21.15.114:0/3243659015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 msgr2=0x55edae5d6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:05.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.592+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/3243659015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5d6d60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa0400a9820 tx=0x7fa0400a9790 comp rx=0 tx=0).stop 2024-05-27T07:53:05.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.592+0000 7fa060bb3b80 1 -- 172.21.15.114:0/3243659015 shutdown_connections 2024-05-27T07:53:05.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.592+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/3243659015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55edae5bfbc0 0x55edae64c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.592+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/3243659015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edae440ec0 0x55edae5d6d60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.592+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/3243659015 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55edae64e2d0 0x55edae6506f0 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.592+0000 7fa060bb3b80 1 --2- 172.21.15.114:0/3243659015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55edae5bf7b0 0x55edae5d72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.592+0000 7fa060bb3b80 1 -- 172.21.15.114:0/3243659015 >> 172.21.15.114:0/3243659015 conn(0x55edae4437f0 msgr2=0x55edae5de0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:05.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.593+0000 7fa02affd700 0 client.28099 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:05.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.594+0000 7fa060bb3b80 1 -- 172.21.15.114:0/3243659015 shutdown_connections 2024-05-27T07:53:05.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.594+0000 7fa060bb3b80 1 -- 172.21.15.114:0/3243659015 wait complete. 2024-05-27T07:53:05.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.595+0000 7fa060bb3b80 20 client.28099 trim_cache size 0 max 16384 2024-05-27T07:53:05.596 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:53:05.597 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:53:05.597 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "22158187-21e2-4cc0-ba5b-0ac277762669", 2024-05-27T07:53:05.597 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:53:05.597 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:53:05.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.597+0000 7fa050f3c700 1 -- 172.21.15.114:0/3528169540 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0340b2aa0 msgr2=0x7fa0340b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:05.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.597+0000 7fa050f3c700 1 --2- 172.21.15.114:0/3528169540 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0340b2aa0 0x7fa0340b4fb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa04400ab30 tx=0x7fa044009250 comp rx=0 tx=0).stop 2024-05-27T07:53:05.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.597+0000 7fa050f3c700 1 -- 172.21.15.114:0/3528169540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa04c1032d0 msgr2=0x7fa04c071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:05.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.597+0000 7fa050f3c700 1 --2- 172.21.15.114:0/3528169540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa04c1032d0 0x7fa04c071760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa04000e9d0 tx=0x7fa04000ed90 comp rx=0 tx=0).stop 2024-05-27T07:53:05.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.600+0000 7fa050f3c700 1 -- 172.21.15.114:0/3528169540 shutdown_connections 2024-05-27T07:53:05.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.600+0000 7fa050f3c700 1 --2- 172.21.15.114:0/3528169540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04c108550 0x7fa04c0721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.600+0000 7fa050f3c700 1 --2- 172.21.15.114:0/3528169540 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa0340b2aa0 0x7fa0340b4fb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.600+0000 7fa050f3c700 1 --2- 172.21.15.114:0/3528169540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa04c1032d0 0x7fa04c071760 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.600+0000 7fa050f3c700 1 --2- 172.21.15.114:0/3528169540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa04c107c00 0x7fa04c071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:05.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.600+0000 7fa050f3c700 1 -- 172.21.15.114:0/3528169540 >> 172.21.15.114:0/3528169540 conn(0x7fa04c078ab0 msgr2=0x7fa04c1005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:05.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.604+0000 7fa050f3c700 1 -- 172.21.15.114:0/3528169540 shutdown_connections 2024-05-27T07:53:05.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:05.604+0000 7fa050f3c700 1 -- 172.21.15.114:0/3528169540 wait complete. 2024-05-27T07:53:06.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1275672 9.39 MB/sec execute 558 sec latency 97.734 ms 2024-05-27T07:53:06.312 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f35aa997700 time 2024-05-27T07:53:06.044267+0000 2024-05-27T07:53:06.312 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:53:06.312 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:06.312 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f35bd1dcba6] 2024-05-27T07:53:06.313 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f35bd1dcdd1] 2024-05-27T07:53:06.313 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5562ba58f23d] 2024-05-27T07:53:06.313 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5562ba57a971] 2024-05-27T07:53:06.313 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 5: (SnapRealm::check_cache() const+0x13e) [0x5562ba578fa0] 2024-05-27T07:53:06.313 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562ba579651] 2024-05-27T07:53:06.313 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5562ba50e347] 2024-05-27T07:53:06.313 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5562ba20a87e] 2024-05-27T07:53:06.313 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5562ba20fe94] 2024-05-27T07:53:06.313 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5562ba21a969] 2024-05-27T07:53:06.313 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 11: (MDLog::_replay_thread()+0x1f38) [0x5562ba5b348c] 2024-05-27T07:53:06.313 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5562ba5bc9cb] 2024-05-27T07:53:06.313 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 13: (Thread::entry_wrapper()+0x43) [0x7f35bd1ab72d] 2024-05-27T07:53:06.314 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 14: (Thread::_entry_func(void*)+0xd) [0x7f35bd1ab749] 2024-05-27T07:53:06.314 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f35bbd4f1ca] 2024-05-27T07:53:06.314 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 16: clone() 2024-05-27T07:53:06.314 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: *** Caught signal (Aborted) ** 2024-05-27T07:53:06.314 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: in thread 7f35aa997700 thread_name:md_log_replay 2024-05-27T07:53:06.314 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.046+0000 7f35aa997700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f35aa997700 time 2024-05-27T07:53:06.044267+0000 2024-05-27T07:53:06.315 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:53:06.315 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.315 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:06.315 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f35bd1dcba6] 2024-05-27T07:53:06.315 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f35bd1dcdd1] 2024-05-27T07:53:06.315 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5562ba58f23d] 2024-05-27T07:53:06.315 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5562ba57a971] 2024-05-27T07:53:06.315 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 5: (SnapRealm::check_cache() const+0x13e) [0x5562ba578fa0] 2024-05-27T07:53:06.315 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562ba579651] 2024-05-27T07:53:06.315 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5562ba50e347] 2024-05-27T07:53:06.315 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5562ba20a87e] 2024-05-27T07:53:06.315 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5562ba20fe94] 2024-05-27T07:53:06.316 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5562ba21a969] 2024-05-27T07:53:06.316 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 11: (MDLog::_replay_thread()+0x1f38) [0x5562ba5b348c] 2024-05-27T07:53:06.316 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5562ba5bc9cb] 2024-05-27T07:53:06.316 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 13: (Thread::entry_wrapper()+0x43) [0x7f35bd1ab72d] 2024-05-27T07:53:06.316 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 14: (Thread::_entry_func(void*)+0xd) [0x7f35bd1ab749] 2024-05-27T07:53:06.316 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f35bbd4f1ca] 2024-05-27T07:53:06.316 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 16: clone() 2024-05-27T07:53:06.316 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.316 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:06.316 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f35bbd59d20] 2024-05-27T07:53:06.316 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2: gsignal() 2024-05-27T07:53:06.316 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 3: abort() 2024-05-27T07:53:06.316 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f35bd1dcced] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f35bd1dcdd1] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5562ba58f23d] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5562ba57a971] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 8: (SnapRealm::check_cache() const+0x13e) [0x5562ba578fa0] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562ba579651] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5562ba50e347] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5562ba20a87e] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5562ba20fe94] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5562ba21a969] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 14: (MDLog::_replay_thread()+0x1f38) [0x5562ba5b348c] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5562ba5bc9cb] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 16: (Thread::entry_wrapper()+0x43) [0x7f35bd1ab72d] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 17: (Thread::_entry_func(void*)+0xd) [0x7f35bd1ab749] 2024-05-27T07:53:06.317 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f35bbd4f1ca] 2024-05-27T07:53:06.318 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 19: clone() 2024-05-27T07:53:06.318 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.050+0000 7f35aa997700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:53:06.318 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: in thread 7f35aa997700 thread_name:md_log_replay 2024-05-27T07:53:06.318 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.318 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:06.318 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f35bbd59d20] 2024-05-27T07:53:06.318 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2: gsignal() 2024-05-27T07:53:06.318 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 3: abort() 2024-05-27T07:53:06.318 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f35bd1dcced] 2024-05-27T07:53:06.318 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f35bd1dcdd1] 2024-05-27T07:53:06.318 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5562ba58f23d] 2024-05-27T07:53:06.318 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5562ba57a971] 2024-05-27T07:53:06.318 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 8: (SnapRealm::check_cache() const+0x13e) [0x5562ba578fa0] 2024-05-27T07:53:06.319 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562ba579651] 2024-05-27T07:53:06.319 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5562ba50e347] 2024-05-27T07:53:06.319 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5562ba20a87e] 2024-05-27T07:53:06.320 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5562ba20fe94] 2024-05-27T07:53:06.320 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5562ba21a969] 2024-05-27T07:53:06.320 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 14: (MDLog::_replay_thread()+0x1f38) [0x5562ba5b348c] 2024-05-27T07:53:06.320 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5562ba5bc9cb] 2024-05-27T07:53:06.320 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 16: (Thread::entry_wrapper()+0x43) [0x7f35bd1ab72d] 2024-05-27T07:53:06.320 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 17: (Thread::_entry_func(void*)+0xd) [0x7f35bd1ab749] 2024-05-27T07:53:06.320 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f35bbd4f1ca] 2024-05-27T07:53:06.320 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 19: clone() 2024-05-27T07:53:06.320 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:53:06.320 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.320 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: -1> 2024-05-27T07:53:06.046+0000 7f35aa997700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f35aa997700 time 2024-05-27T07:53:06.044267+0000 2024-05-27T07:53:06.320 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:53:06.321 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.321 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:06.321 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f35bd1dcba6] 2024-05-27T07:53:06.321 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f35bd1dcdd1] 2024-05-27T07:53:06.321 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5562ba58f23d] 2024-05-27T07:53:06.321 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5562ba57a971] 2024-05-27T07:53:06.321 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 5: (SnapRealm::check_cache() const+0x13e) [0x5562ba578fa0] 2024-05-27T07:53:06.321 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562ba579651] 2024-05-27T07:53:06.321 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5562ba50e347] 2024-05-27T07:53:06.321 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5562ba20a87e] 2024-05-27T07:53:06.321 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5562ba20fe94] 2024-05-27T07:53:06.321 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5562ba21a969] 2024-05-27T07:53:06.321 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 11: (MDLog::_replay_thread()+0x1f38) [0x5562ba5b348c] 2024-05-27T07:53:06.322 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5562ba5bc9cb] 2024-05-27T07:53:06.322 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 13: (Thread::entry_wrapper()+0x43) [0x7f35bd1ab72d] 2024-05-27T07:53:06.322 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 14: (Thread::_entry_func(void*)+0xd) [0x7f35bd1ab749] 2024-05-27T07:53:06.322 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f35bbd4f1ca] 2024-05-27T07:53:06.322 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 16: clone() 2024-05-27T07:53:06.322 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.322 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 0> 2024-05-27T07:53:06.050+0000 7f35aa997700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:53:06.322 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: in thread 7f35aa997700 thread_name:md_log_replay 2024-05-27T07:53:06.322 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.322 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:06.322 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f35bbd59d20] 2024-05-27T07:53:06.322 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2: gsignal() 2024-05-27T07:53:06.322 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 3: abort() 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f35bd1dcced] 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f35bd1dcdd1] 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5562ba58f23d] 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5562ba57a971] 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 8: (SnapRealm::check_cache() const+0x13e) [0x5562ba578fa0] 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562ba579651] 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5562ba50e347] 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5562ba20a87e] 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5562ba20fe94] 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5562ba21a969] 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 14: (MDLog::_replay_thread()+0x1f38) [0x5562ba5b348c] 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5562ba5bc9cb] 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 16: (Thread::entry_wrapper()+0x43) [0x7f35bd1ab72d] 2024-05-27T07:53:06.323 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 17: (Thread::_entry_func(void*)+0xd) [0x7f35bd1ab749] 2024-05-27T07:53:06.324 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f35bbd4f1ca] 2024-05-27T07:53:06.324 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 19: clone() 2024-05-27T07:53:06.324 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:53:06.324 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.324 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: -1> 2024-05-27T07:53:06.046+0000 7f35aa997700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f35aa997700 time 2024-05-27T07:53:06.044267+0000 2024-05-27T07:53:06.324 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:53:06.324 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.324 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:06.324 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f35bd1dcba6] 2024-05-27T07:53:06.324 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f35bd1dcdd1] 2024-05-27T07:53:06.324 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5562ba58f23d] 2024-05-27T07:53:06.324 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5562ba57a971] 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 5: (SnapRealm::check_cache() const+0x13e) [0x5562ba578fa0] 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562ba579651] 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5562ba50e347] 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5562ba20a87e] 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5562ba20fe94] 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5562ba21a969] 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 11: (MDLog::_replay_thread()+0x1f38) [0x5562ba5b348c] 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5562ba5bc9cb] 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 13: (Thread::entry_wrapper()+0x43) [0x7f35bd1ab72d] 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 14: (Thread::_entry_func(void*)+0xd) [0x7f35bd1ab749] 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f35bbd4f1ca] 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 16: clone() 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 0> 2024-05-27T07:53:06.050+0000 7f35aa997700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:53:06.325 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: in thread 7f35aa997700 thread_name:md_log_replay 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f35bbd59d20] 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2: gsignal() 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 3: abort() 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f35bd1dcced] 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f35bd1dcdd1] 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5562ba58f23d] 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5562ba57a971] 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 8: (SnapRealm::check_cache() const+0x13e) [0x5562ba578fa0] 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562ba579651] 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5562ba50e347] 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5562ba20a87e] 2024-05-27T07:53:06.326 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5562ba20fe94] 2024-05-27T07:53:06.327 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5562ba21a969] 2024-05-27T07:53:06.327 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 14: (MDLog::_replay_thread()+0x1f38) [0x5562ba5b348c] 2024-05-27T07:53:06.327 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5562ba5bc9cb] 2024-05-27T07:53:06.327 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 16: (Thread::entry_wrapper()+0x43) [0x7f35bd1ab72d] 2024-05-27T07:53:06.327 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 17: (Thread::_entry_func(void*)+0xd) [0x7f35bd1ab749] 2024-05-27T07:53:06.327 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f35bbd4f1ca] 2024-05-27T07:53:06.327 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 19: clone() 2024-05-27T07:53:06.327 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:53:06.327 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75775]: 2024-05-27T07:53:06.327 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:06 smithi186 podman[75840]: 2024-05-27 07:53:06.126607042 +0000 UTC m=+0.019462624 container died d1bed78545fcf854aa9306e11efbfe67a1d0886838089797a0fe271bc6661695 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-27T07:53:06.632 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:06 smithi114 ceph-mon[32038]: pgmap v447: 129 pgs: 129 active+clean; 66 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 335 op/s 2024-05-27T07:53:06.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:53:06.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:06 smithi163 ceph-mon[36451]: pgmap v447: 129 pgs: 129 active+clean; 66 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 335 op/s 2024-05-27T07:53:06.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:06 smithi186 ceph-mon[36647]: pgmap v447: 129 pgs: 129 active+clean; 66 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 335 op/s 2024-05-27T07:53:07.115 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:07.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1275794 9.38 MB/sec execute 559 sec latency 158.710 ms 2024-05-27T07:53:07.462 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:07 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:53:07.462 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:07 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:07.462 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:07 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:53:07.462 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:07 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:07.463 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:07 smithi186 podman[75840]: 2024-05-27 07:53:07.078736306 +0000 UTC m=+0.971591893 container remove d1bed78545fcf854aa9306e11efbfe67a1d0886838089797a0fe271bc6661695 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-27T07:53:07.463 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:07 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:53:07.521 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:07 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:53:07.521 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:07 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:07.521 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:07 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:53:07.521 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:07 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:07.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:07 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:53:07.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:07 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:07.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:07 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:53:07.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:07 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:07.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:07 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:53:07.734 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:07 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.002s CPU time. 2024-05-27T07:53:08.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1276372 9.37 MB/sec execute 560 sec latency 121.047 ms 2024-05-27T07:53:08.717 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:08 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:53:08.717 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:08 smithi114 ceph-mon[32038]: pgmap v448: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 70 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 292 op/s 2024-05-27T07:53:08.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:08 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:53:08.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:08 smithi163 ceph-mon[36451]: pgmap v448: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 70 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 292 op/s 2024-05-27T07:53:08.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:08 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:53:08.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:08 smithi186 ceph-mon[36647]: pgmap v448: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 70 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 292 op/s 2024-05-27T07:53:09.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1276917 9.35 MB/sec execute 561 sec latency 114.264 ms 2024-05-27T07:53:10.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1278143 9.35 MB/sec execute 562 sec latency 217.792 ms 2024-05-27T07:53:10.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.392+0000 7fd99aed9700 1 -- 172.21.15.114:0/1406717988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd994108730 msgr2=0x7fd994110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:10.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.392+0000 7fd99aed9700 1 --2- 172.21.15.114:0/1406717988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd994108730 0x7fd994110090 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00b0d0 tx=0x7fd98c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.392+0000 7fd99aed9700 1 -- 172.21.15.114:0/1406717988 shutdown_connections 2024-05-27T07:53:10.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.392+0000 7fd99aed9700 1 --2- 172.21.15.114:0/1406717988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd994108730 0x7fd994110090 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.392+0000 7fd99aed9700 1 --2- 172.21.15.114:0/1406717988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9941034b0 0x7fd9941038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.392+0000 7fd99aed9700 1 --2- 172.21.15.114:0/1406717988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd994107de0 0x7fd9941081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.392+0000 7fd99aed9700 1 -- 172.21.15.114:0/1406717988 >> 172.21.15.114:0/1406717988 conn(0x7fd9940fedb0 msgr2=0x7fd9941011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:10.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.392+0000 7fd99aed9700 1 -- 172.21.15.114:0/1406717988 shutdown_connections 2024-05-27T07:53:10.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.392+0000 7fd99aed9700 1 -- 172.21.15.114:0/1406717988 wait complete. 2024-05-27T07:53:10.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.393+0000 7fd99aed9700 1 Processor -- start 2024-05-27T07:53:10.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.393+0000 7fd99aed9700 1 -- start start 2024-05-27T07:53:10.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd99aed9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9941034b0 0x7fd99419b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd99aed9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd994107de0 0x7fd99419bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd99aed9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd994108730 0x7fd99419c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd99aed9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd994113360 con 0x7fd9941034b0 2024-05-27T07:53:10.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd99aed9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd9941131e0 con 0x7fd994107de0 2024-05-27T07:53:10.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd99aed9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9941134e0 con 0x7fd994108730 2024-05-27T07:53:10.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd998c75700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9941034b0 0x7fd99419b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd998c75700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9941034b0 0x7fd99419b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57728/0 (socket says 172.21.15.114:57728) 2024-05-27T07:53:10.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd998c75700 1 -- 172.21.15.114:0/105394724 learned_addr learned my addr 172.21.15.114:0/105394724 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:10.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd999476700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd994108730 0x7fd99419c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd993fff700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd994107de0 0x7fd99419bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd998c75700 1 -- 172.21.15.114:0/105394724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd994108730 msgr2=0x7fd99419c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:10.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd998c75700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd994108730 0x7fd99419c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd998c75700 1 -- 172.21.15.114:0/105394724 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd994107de0 msgr2=0x7fd99419bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:10.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd998c75700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd994107de0 0x7fd99419bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd998c75700 1 -- 172.21.15.114:0/105394724 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9941a6d10 con 0x7fd9941034b0 2024-05-27T07:53:10.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd999476700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd994108730 0x7fd99419c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:10.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.394+0000 7fd998c75700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9941034b0 0x7fd99419b9a0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fd98800e9a0 tx=0x7fd98800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:10.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.395+0000 7fd991ffb700 1 -- 172.21.15.114:0/105394724 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98800cbd0 con 0x7fd9941034b0 2024-05-27T07:53:10.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.395+0000 7fd99aed9700 1 -- 172.21.15.114:0/105394724 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9941a6fa0 con 0x7fd9941034b0 2024-05-27T07:53:10.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.395+0000 7fd991ffb700 1 -- 172.21.15.114:0/105394724 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd98800cd70 con 0x7fd9941034b0 2024-05-27T07:53:10.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.395+0000 7fd99aed9700 1 -- 172.21.15.114:0/105394724 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9941a7500 con 0x7fd9941034b0 2024-05-27T07:53:10.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.395+0000 7fd991ffb700 1 -- 172.21.15.114:0/105394724 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988018620 con 0x7fd9941034b0 2024-05-27T07:53:10.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.398+0000 7fd991ffb700 1 -- 172.21.15.114:0/105394724 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fd988010430 con 0x7fd9941034b0 2024-05-27T07:53:10.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.399+0000 7fd991ffb700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd9840b29f0 0x7fd9840b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.399+0000 7fd993fff700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd9840b29f0 0x7fd9840b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.399+0000 7fd991ffb700 1 -- 172.21.15.114:0/105394724 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9642+0+0 (secure 0 0 0) 0x7fd9880e2ca0 con 0x7fd9941034b0 2024-05-27T07:53:10.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.400+0000 7fd9a98eeb80 1 Processor -- start 2024-05-27T07:53:10.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.400+0000 7fd9a98eeb80 1 -- start start 2024-05-27T07:53:10.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.400+0000 7fd993fff700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd9840b29f0 0x7fd9840b4f00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd9941027f0 tx=0x7fd980009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:10.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.400+0000 7fd9a98eeb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f847ae40 0x55d6f85fe710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.400+0000 7fd9a98eeb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f85fec50 0x55d6f85f8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.400+0000 7fd9a98eeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85f8f80 0x55d6f85f9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.400+0000 7fd9a98eeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d6f8527c90 con 0x55d6f85f8f80 2024-05-27T07:53:10.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.400+0000 7fd9a98eeb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d6f8517830 con 0x55d6f85fec50 2024-05-27T07:53:10.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.400+0000 7fd9a98eeb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d6f851b870 con 0x55d6f847ae40 2024-05-27T07:53:10.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.400+0000 7fd999476700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85f8f80 0x55d6f85f9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.400+0000 7fd999476700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85f8f80 0x55d6f85f9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57744/0 (socket says 172.21.15.114:57744) 2024-05-27T07:53:10.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.400+0000 7fd999476700 1 -- 172.21.15.114:0/2405887078 learned_addr learned my addr 172.21.15.114:0/2405887078 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:10.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.400+0000 7fd993fff700 1 --2- 172.21.15.114:0/2405887078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f85fec50 0x55d6f85f8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.401+0000 7fd999476700 1 -- 172.21.15.114:0/2405887078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f847ae40 msgr2=0x55d6f85fe710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:53:10.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.401+0000 7fd999476700 1 --2- 172.21.15.114:0/2405887078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f847ae40 0x55d6f85fe710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.401+0000 7fd999476700 1 -- 172.21.15.114:0/2405887078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f85fec50 msgr2=0x55d6f85f8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:10.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.401+0000 7fd999476700 1 --2- 172.21.15.114:0/2405887078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f85fec50 0x55d6f85f8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.401+0000 7fd999476700 1 -- 172.21.15.114:0/2405887078 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6f84678d0 con 0x55d6f85f8f80 2024-05-27T07:53:10.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.401+0000 7fd993fff700 1 --2- 172.21.15.114:0/2405887078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f85fec50 0x55d6f85f8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:10.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.401+0000 7fd999476700 1 --2- 172.21.15.114:0/2405887078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85f8f80 0x55d6f85f9390 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009bb0 tx=0x7fd98c0099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f6d8af083f9ea0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:10.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.401+0000 7fd97effd700 1 -- 172.21.15.114:0/2405887078 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c006040 con 0x55d6f85f8f80 2024-05-27T07:53:10.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.401+0000 7fd97effd700 1 -- 172.21.15.114:0/2405887078 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd98c0075c0 con 0x55d6f85f8f80 2024-05-27T07:53:10.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.401+0000 7fd97effd700 1 -- 172.21.15.114:0/2405887078 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c0256d0 con 0x55d6f85f8f80 2024-05-27T07:53:10.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.402+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/2405887078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85f8f80 msgr2=0x55d6f85f9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:10.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.402+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/2405887078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85f8f80 0x55d6f85f9390 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009bb0 tx=0x7fd98c0099e0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.402+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/2405887078 shutdown_connections 2024-05-27T07:53:10.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.402+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/2405887078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85f8f80 0x55d6f85f9390 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.402+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/2405887078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f847ae40 0x55d6f85fe710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.402+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/2405887078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f85fec50 0x55d6f85f8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.402+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/2405887078 >> 172.21.15.114:0/2405887078 conn(0x55d6f84d5af0 msgr2=0x55d6f85f3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:10.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.402+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/2405887078 shutdown_connections 2024-05-27T07:53:10.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.403+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/2405887078 wait complete. 2024-05-27T07:53:10.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.403+0000 7fd9a98eeb80 1 -- start start 2024-05-27T07:53:10.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.403+0000 7fd9a98eeb80 1 Processor -- start 2024-05-27T07:53:10.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd9a98eeb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f847ae40 0x55d6f86023b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd9a98eeb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f85f8f80 0x55d6f86028f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd9a98eeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f8602e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd9a98eeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d6f841dfa0 con 0x55d6f85fec50 2024-05-27T07:53:10.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd9a98eeb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d6f84b97c0 con 0x55d6f847ae40 2024-05-27T07:53:10.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd9a98eeb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d6f8609af0 con 0x55d6f85f8f80 2024-05-27T07:53:10.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd993fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f85f8f80 0x55d6f86028f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd9a98eeb80 10 client.0 _ll_register_callbacks cb 0x55d6f84b2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:10.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd993fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f85f8f80 0x55d6f86028f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35314/0 (socket says 172.21.15.114:35314) 2024-05-27T07:53:10.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd993fff700 1 -- 172.21.15.114:0/913536322 learned_addr learned my addr 172.21.15.114:0/913536322 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:10.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd999476700 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f8602e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd998c75700 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f847ae40 0x55d6f86023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd999476700 1 -- 172.21.15.114:0/913536322 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f85f8f80 msgr2=0x55d6f86028f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:10.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd999476700 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f85f8f80 0x55d6f86028f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd999476700 1 -- 172.21.15.114:0/913536322 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f847ae40 msgr2=0x55d6f86023b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:10.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd999476700 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f847ae40 0x55d6f86023b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd999476700 1 -- 172.21.15.114:0/913536322 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6f84b6690 con 0x55d6f85fec50 2024-05-27T07:53:10.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd993fff700 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f85f8f80 0x55d6f86028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:10.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd999476700 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f8602e30 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fd98c024000 tx=0x7fd98c007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91c326c4ff691df server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:10.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.404+0000 7fd973fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:10.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd973fff700 1 -- 172.21.15.114:0/913536322 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c005a50 con 0x55d6f85fec50 2024-05-27T07:53:10.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd973fff700 1 -- 172.21.15.114:0/913536322 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9642+0+0 (secure 0 0 0) 0x7fd98c00f040 con 0x55d6f85fec50 2024-05-27T07:53:10.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/913536322 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd98c0075c0 con 0x55d6f85fec50 2024-05-27T07:53:10.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd973fff700 1 -- 172.21.15.114:0/913536322 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fd98c035050 con 0x55d6f85fec50 2024-05-27T07:53:10.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd9a98eeb80 10 client.18567 fetch_fsmap learned FSMap version 73 2024-05-27T07:53:10.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/913536322 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55d6f86143e0 con 0x55d6f85fec50 2024-05-27T07:53:10.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd973fff700 1 -- 172.21.15.114:0/913536322 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd98c0075c0 con 0x55d6f85fec50 2024-05-27T07:53:10.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd973fff700 1 -- 172.21.15.114:0/913536322 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7fd98c011070 con 0x55d6f85fec50 2024-05-27T07:53:10.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd9a98eeb80 10 client.18567 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:53:10.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd9a98eeb80 10 client.18567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:10.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd9a98eeb80 20 client.18567 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:10.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6f86168b0 0x55d6f8618cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd9a98eeb80 4 client.18567 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:10.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.405+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/913536322 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6f85c49f0 con 0x55d6f86168b0 2024-05-27T07:53:10.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.406+0000 7fd998c75700 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6f86168b0 0x55d6f8618cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.407+0000 7fd998c75700 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6f86168b0 0x55d6f8618cd0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e25c395da00727da server_cookie=57bc1bce9e9bd43 in_seq=0 out_seq=0 2024-05-27T07:53:10.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.407+0000 7fd973fff700 10 client.18567 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:10.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd973fff700 1 -- 172.21.15.114:0/913536322 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d6f85c49f0 con 0x55d6f86168b0 2024-05-27T07:53:10.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd973fff700 10 client.18567 handle_command_reply: tid=0 2024-05-27T07:53:10.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd9a98eeb80 1 client.18567 shutdown 2024-05-27T07:53:10.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/913536322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 msgr2=0x55d6f8602e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:10.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f8602e30 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fd98c024000 tx=0x7fd98c007ca0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/913536322 shutdown_connections 2024-05-27T07:53:10.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f8602e30 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6f86168b0 0x55d6f8618cd0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f847ae40 0x55d6f86023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/913536322 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f85f8f80 0x55d6f86028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd973fff700 0 client.18567 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:10.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/913536322 >> 172.21.15.114:0/913536322 conn(0x55d6f84d5af0 msgr2=0x55d6f85f7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:10.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/913536322 shutdown_connections 2024-05-27T07:53:10.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.408+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/913536322 wait complete. 2024-05-27T07:53:10.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.409+0000 7fd9a98eeb80 20 client.18567 trim_cache size 0 max 16384 2024-05-27T07:53:10.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.432+0000 7fd9a98eeb80 1 Processor -- start 2024-05-27T07:53:10.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.432+0000 7fd9a98eeb80 1 -- start start 2024-05-27T07:53:10.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd9a98eeb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f847ae40 0x55d6f85f8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd9a98eeb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f85f8f80 0x55d6f8602c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd9a98eeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f8612bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd9a98eeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd9941afcd0 con 0x55d6f85fec50 2024-05-27T07:53:10.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd9a98eeb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd9941ae440 con 0x55d6f847ae40 2024-05-27T07:53:10.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd9a98eeb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9941ac2b0 con 0x55d6f85f8f80 2024-05-27T07:53:10.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd999476700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f8612bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd993fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f85f8f80 0x55d6f8602c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd999476700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f8612bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57752/0 (socket says 172.21.15.114:57752) 2024-05-27T07:53:10.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd999476700 1 -- 172.21.15.114:0/1194227062 learned_addr learned my addr 172.21.15.114:0/1194227062 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:10.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd998c75700 1 --2- 172.21.15.114:0/1194227062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f847ae40 0x55d6f85f8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd999476700 1 -- 172.21.15.114:0/1194227062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f85f8f80 msgr2=0x55d6f8602c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:10.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd999476700 1 --2- 172.21.15.114:0/1194227062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f85f8f80 0x55d6f8602c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd999476700 1 -- 172.21.15.114:0/1194227062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f847ae40 msgr2=0x55d6f85f8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:10.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd999476700 1 --2- 172.21.15.114:0/1194227062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f847ae40 0x55d6f85f8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd999476700 1 -- 172.21.15.114:0/1194227062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6f8603470 con 0x55d6f85fec50 2024-05-27T07:53:10.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd999476700 1 --2- 172.21.15.114:0/1194227062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f8612bc0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009610 tx=0x7fd98c00b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a200e61d4aadc5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:10.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.433+0000 7fd97e7fc700 1 -- 172.21.15.114:0/1194227062 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c006080 con 0x55d6f85fec50 2024-05-27T07:53:10.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.434+0000 7fd97e7fc700 1 -- 172.21.15.114:0/1194227062 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd98c02ace0 con 0x55d6f85fec50 2024-05-27T07:53:10.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.434+0000 7fd97e7fc700 1 -- 172.21.15.114:0/1194227062 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c005960 con 0x55d6f85fec50 2024-05-27T07:53:10.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.435+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/1194227062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 msgr2=0x55d6f8612bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:10.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.435+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/1194227062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f8612bc0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009610 tx=0x7fd98c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.435+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/1194227062 shutdown_connections 2024-05-27T07:53:10.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.435+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/1194227062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f8612bc0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.435+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/1194227062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f847ae40 0x55d6f85f8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.435+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/1194227062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f85f8f80 0x55d6f8602c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.435+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/1194227062 >> 172.21.15.114:0/1194227062 conn(0x55d6f84d5af0 msgr2=0x55d6f85fc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:10.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.435+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/1194227062 shutdown_connections 2024-05-27T07:53:10.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.435+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/1194227062 wait complete. 2024-05-27T07:53:10.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.435+0000 7fd9a98eeb80 1 -- start start 2024-05-27T07:53:10.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.436+0000 7fd9a98eeb80 1 Processor -- start 2024-05-27T07:53:10.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.436+0000 7fd9a98eeb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f847ae40 0x55d6f8609d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.436+0000 7fd9a98eeb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f85f8f80 0x55d6f860a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.436+0000 7fd9a98eeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f860a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.436+0000 7fd9a98eeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd9941ac900 con 0x55d6f85fec50 2024-05-27T07:53:10.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.436+0000 7fd9a98eeb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd9941add70 con 0x55d6f85f8f80 2024-05-27T07:53:10.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.436+0000 7fd9a98eeb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9941acfd0 con 0x55d6f847ae40 2024-05-27T07:53:10.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.436+0000 7fd998c75700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f847ae40 0x55d6f8609d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.436+0000 7fd993fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f85f8f80 0x55d6f860a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.436+0000 7fd9a98eeb80 10 client.0 _ll_register_callbacks cb 0x55d6f84b2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:10.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.436+0000 7fd999476700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f860a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.436+0000 7fd993fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f85f8f80 0x55d6f860a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:38052/0 (socket says 172.21.15.114:38052) 2024-05-27T07:53:10.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.436+0000 7fd993fff700 1 -- 172.21.15.114:0/2865381360 learned_addr learned my addr 172.21.15.114:0/2865381360 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:10.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.437+0000 7fd999476700 1 -- 172.21.15.114:0/2865381360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f847ae40 msgr2=0x55d6f8609d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:10.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.437+0000 7fd999476700 1 --2- 172.21.15.114:0/2865381360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f847ae40 0x55d6f8609d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.437+0000 7fd999476700 1 -- 172.21.15.114:0/2865381360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f85f8f80 msgr2=0x55d6f860a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:10.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.437+0000 7fd999476700 1 --2- 172.21.15.114:0/2865381360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f85f8f80 0x55d6f860a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.437+0000 7fd999476700 1 -- 172.21.15.114:0/2865381360 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6f867f410 con 0x55d6f85fec50 2024-05-27T07:53:10.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.437+0000 7fd999476700 1 --2- 172.21.15.114:0/2865381360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f860a7f0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fd98c02a9d0 tx=0x7fd98c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ebefa393d64199 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:10.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.437+0000 7fd973fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:10.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.437+0000 7fd973fff700 1 -- 172.21.15.114:0/2865381360 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c006080 con 0x55d6f85fec50 2024-05-27T07:53:10.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.437+0000 7fd973fff700 1 -- 172.21.15.114:0/2865381360 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9642+0+0 (secure 0 0 0) 0x7fd98c02d040 con 0x55d6f85fec50 2024-05-27T07:53:10.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.437+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/2865381360 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd98c02ace0 con 0x55d6f85fec50 2024-05-27T07:53:10.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.437+0000 7fd973fff700 1 -- 172.21.15.114:0/2865381360 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fd98c005410 con 0x55d6f85fec50 2024-05-27T07:53:10.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.437+0000 7fd9a98eeb80 10 client.18573 fetch_fsmap learned FSMap version 73 2024-05-27T07:53:10.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.437+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/2865381360 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55d6f8680680 con 0x55d6f85fec50 2024-05-27T07:53:10.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.438+0000 7fd973fff700 1 -- 172.21.15.114:0/2865381360 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd98c02ace0 con 0x55d6f85fec50 2024-05-27T07:53:10.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.438+0000 7fd973fff700 1 -- 172.21.15.114:0/2865381360 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7fd98c042070 con 0x55d6f85fec50 2024-05-27T07:53:10.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.438+0000 7fd9a98eeb80 10 client.18573 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:53:10.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.438+0000 7fd9a98eeb80 10 client.18573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:10.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.438+0000 7fd9a98eeb80 20 client.18573 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:10.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.438+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/2865381360 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6f8680ae0 0x55d6f8682f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:10.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.438+0000 7fd9a98eeb80 4 client.18573 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:53:10.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.438+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/2865381360 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d6f8529be0 con 0x55d6f8680ae0 2024-05-27T07:53:10.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.438+0000 7fd998c75700 1 --2- 172.21.15.114:0/2865381360 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6f8680ae0 0x55d6f8682f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:10.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.438+0000 7fd973fff700 10 client.18573 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:10.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.439+0000 7fd998c75700 1 --2- 172.21.15.114:0/2865381360 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6f8680ae0 0x55d6f8682f00 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1693721bfa05748 server_cookie=c6f5d2294d9e1e1 in_seq=0 out_seq=0 2024-05-27T07:53:10.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.439+0000 7fd973fff700 1 -- 172.21.15.114:0/2865381360 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d6f8529be0 con 0x55d6f8680ae0 2024-05-27T07:53:10.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.439+0000 7fd973fff700 10 client.18573 handle_command_reply: tid=0 2024-05-27T07:53:10.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.439+0000 7fd9a98eeb80 1 client.18573 shutdown 2024-05-27T07:53:10.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.439+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/2865381360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 msgr2=0x55d6f860a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:10.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.439+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/2865381360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f860a7f0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fd98c02a9d0 tx=0x7fd98c000f40 comp rx=0 tx=0).stop 2024-05-27T07:53:10.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.440+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/2865381360 shutdown_connections 2024-05-27T07:53:10.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.440+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/2865381360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6f85fec50 0x55d6f860a7f0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.440+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/2865381360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6f847ae40 0x55d6f8609d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.440+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/2865381360 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d6f8680ae0 0x55d6f8682f00 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.440+0000 7fd9a98eeb80 1 --2- 172.21.15.114:0/2865381360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d6f85f8f80 0x55d6f860a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.440+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/2865381360 >> 172.21.15.114:0/2865381360 conn(0x55d6f84d5af0 msgr2=0x55d6f85fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:10.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.440+0000 7fd973fff700 0 client.18573 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:10.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.440+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/2865381360 shutdown_connections 2024-05-27T07:53:10.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.440+0000 7fd9a98eeb80 1 -- 172.21.15.114:0/2865381360 wait complete. 2024-05-27T07:53:10.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.440+0000 7fd9a98eeb80 20 client.18573 trim_cache size 0 max 16384 2024-05-27T07:53:10.448 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:53:10.448 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:53:10.448 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:53:10.449 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:53:10.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.441+0000 7fd99aed9700 1 -- 172.21.15.114:0/105394724 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd9840b29f0 msgr2=0x7fd9840b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:10.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.441+0000 7fd99aed9700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd9840b29f0 0x7fd9840b4f00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd9941027f0 tx=0x7fd980009260 comp rx=0 tx=0).stop 2024-05-27T07:53:10.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.441+0000 7fd99aed9700 1 -- 172.21.15.114:0/105394724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9941034b0 msgr2=0x7fd99419b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:10.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.441+0000 7fd99aed9700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9941034b0 0x7fd99419b9a0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fd98800e9a0 tx=0x7fd98800ecb0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.441+0000 7fd99aed9700 1 -- 172.21.15.114:0/105394724 shutdown_connections 2024-05-27T07:53:10.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.441+0000 7fd99aed9700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd994108730 0x7fd99419c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.441+0000 7fd99aed9700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9941034b0 0x7fd99419b9a0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.441+0000 7fd99aed9700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd9840b29f0 0x7fd9840b4f00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.441+0000 7fd99aed9700 1 --2- 172.21.15.114:0/105394724 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd994107de0 0x7fd99419bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:10.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.441+0000 7fd99aed9700 1 -- 172.21.15.114:0/105394724 >> 172.21.15.114:0/105394724 conn(0x7fd9940fedb0 msgr2=0x7fd9941073e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:10.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.441+0000 7fd99aed9700 1 -- 172.21.15.114:0/105394724 shutdown_connections 2024-05-27T07:53:10.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:10.441+0000 7fd99aed9700 1 -- 172.21.15.114:0/105394724 wait complete. 2024-05-27T07:53:10.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:10 smithi114 ceph-mon[32038]: pgmap v449: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 70 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 292 op/s 2024-05-27T07:53:10.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:10 smithi163 ceph-mon[36451]: pgmap v449: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 70 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 292 op/s 2024-05-27T07:53:10.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:10 smithi186 ceph-mon[36647]: pgmap v449: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 70 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 292 op/s 2024-05-27T07:53:11.229 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:53:11.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:53:11.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1279055 9.34 MB/sec execute 563 sec latency 82.200 ms 2024-05-27T07:53:11.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.415+0000 7fc093f57640 1 -- 172.21.15.114:0/3020001820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc08c10b7f0 msgr2=0x7fc08c10dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:11.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.415+0000 7fc093f57640 1 --2- 172.21.15.114:0/3020001820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc08c10b7f0 0x7fc08c10dc10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc07c009900 tx=0x7fc07c02a410 comp rx=0 tx=0).stop 2024-05-27T07:53:11.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.415+0000 7fc093f57640 1 -- 172.21.15.114:0/3020001820 shutdown_connections 2024-05-27T07:53:11.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.415+0000 7fc093f57640 1 --2- 172.21.15.114:0/3020001820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc08c10e1e0 0x7fc08c1106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.415+0000 7fc093f57640 1 --2- 172.21.15.114:0/3020001820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc08c10b7f0 0x7fc08c10dc10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.415+0000 7fc093f57640 1 --2- 172.21.15.114:0/3020001820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc08c104d90 0x7fc08c1051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.415+0000 7fc093f57640 1 -- 172.21.15.114:0/3020001820 >> 172.21.15.114:0/3020001820 conn(0x7fc08c0782d0 msgr2=0x7fc08c078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:11.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.415+0000 7fc093f57640 1 -- 172.21.15.114:0/3020001820 shutdown_connections 2024-05-27T07:53:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.415+0000 7fc093f57640 1 -- 172.21.15.114:0/3020001820 wait complete. 2024-05-27T07:53:11.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.415+0000 7fc093f57640 1 Processor -- start 2024-05-27T07:53:11.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.415+0000 7fc093f57640 1 -- start start 2024-05-27T07:53:11.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc093f57640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc08c104d90 0x7fc08c1a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc093f57640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc08c10b7f0 0x7fc08c1a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc093f57640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc08c10e1e0 0x7fc08c19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc093f57640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c113e40 con 0x7fc08c10e1e0 2024-05-27T07:53:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc093f57640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c113cc0 con 0x7fc08c104d90 2024-05-27T07:53:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc093f57640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c113fc0 con 0x7fc08c10b7f0 2024-05-27T07:53:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc0924f4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc08c10e1e0 0x7fc08c19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc0924f4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc08c10e1e0 0x7fc08c19a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57764/0 (socket says 172.21.15.114:57764) 2024-05-27T07:53:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc0924f4640 1 -- 172.21.15.114:0/4217380377 learned_addr learned my addr 172.21.15.114:0/4217380377 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc0914f2640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc08c10b7f0 0x7fc08c1a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc091cf3640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc08c104d90 0x7fc08c1a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc0924f4640 1 -- 172.21.15.114:0/4217380377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc08c10b7f0 msgr2=0x7fc08c1a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc0924f4640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc08c10b7f0 0x7fc08c1a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc0924f4640 1 -- 172.21.15.114:0/4217380377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc08c104d90 msgr2=0x7fc08c1a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc0924f4640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc08c104d90 0x7fc08c1a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc0924f4640 1 -- 172.21.15.114:0/4217380377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc08c19ae70 con 0x7fc08c10e1e0 2024-05-27T07:53:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc0914f2640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc08c10b7f0 0x7fc08c1a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.416+0000 7fc0924f4640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc08c10e1e0 0x7fc08c19a740 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fc08400efd0 tx=0x7fc08400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.417+0000 7fc082ffd640 1 -- 172.21.15.114:0/4217380377 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc084003b70 con 0x7fc08c10e1e0 2024-05-27T07:53:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.417+0000 7fc093f57640 1 -- 172.21.15.114:0/4217380377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc08c19b100 con 0x7fc08c10e1e0 2024-05-27T07:53:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.417+0000 7fc093f57640 1 -- 172.21.15.114:0/4217380377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc08c1a7360 con 0x7fc08c10e1e0 2024-05-27T07:53:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.417+0000 7fc082ffd640 1 -- 172.21.15.114:0/4217380377 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc084003d10 con 0x7fc08c10e1e0 2024-05-27T07:53:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.417+0000 7fc082ffd640 1 -- 172.21.15.114:0/4217380377 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0840108a0 con 0x7fc08c10e1e0 2024-05-27T07:53:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc09ae09740 1 Processor -- start 2024-05-27T07:53:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc082ffd640 1 -- 172.21.15.114:0/4217380377 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fc084010ae0 con 0x7fc08c10e1e0 2024-05-27T07:53:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc09ae09740 1 -- start start 2024-05-27T07:53:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc09ae09740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99ea71b0 0x555a9a2483d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc09ae09740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e7e2f0 0x555a9a248910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc09ae09740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e864d0 0x555a9a248e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc09ae09740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555a9a1bac30 con 0x555a99ea71b0 2024-05-27T07:53:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc09ae09740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555a9a1aa7d0 con 0x555a99e864d0 2024-05-27T07:53:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc09ae09740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555a9a1ae810 con 0x555a99e7e2f0 2024-05-27T07:53:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc0914f2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99ea71b0 0x555a9a2483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc091cf3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e7e2f0 0x555a9a248910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc0914f2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99ea71b0 0x555a9a2483d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57772/0 (socket says 172.21.15.114:57772) 2024-05-27T07:53:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc0914f2640 1 -- 172.21.15.114:0/688756748 learned_addr learned my addr 172.21.15.114:0/688756748 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.420+0000 7fc0914f2640 1 --2- 172.21.15.114:0/688756748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e864d0 0x555a9a248e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.421+0000 7fc082ffd640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc0780b2950 0x7fc0780b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.421+0000 7fc091cf3640 1 -- 172.21.15.114:0/688756748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e864d0 msgr2=0x555a9a248e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.421+0000 7fc091cf3640 1 --2- 172.21.15.114:0/688756748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e864d0 0x555a9a248e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.421+0000 7fc091cf3640 1 -- 172.21.15.114:0/688756748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99ea71b0 msgr2=0x555a9a2483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.421+0000 7fc091cf3640 1 --2- 172.21.15.114:0/688756748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99ea71b0 0x555a9a2483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.421+0000 7fc091cf3640 1 -- 172.21.15.114:0/688756748 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a99ebf210 con 0x555a99e7e2f0 2024-05-27T07:53:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.421+0000 7fc0924f4640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc0780b2950 0x7fc0780b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.421+0000 7fc082ffd640 1 -- 172.21.15.114:0/4217380377 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9642+0+0 (secure 0 0 0) 0x7fc0840e2e70 con 0x7fc08c10e1e0 2024-05-27T07:53:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.421+0000 7fc091cf3640 1 --2- 172.21.15.114:0/688756748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e7e2f0 0x555a9a248910 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc088009c00 tx=0x7fc088011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9d3894036651e12 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.421+0000 7fc077fff640 1 -- 172.21.15.114:0/688756748 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc088004230 con 0x555a99e7e2f0 2024-05-27T07:53:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.421+0000 7fc0924f4640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc0780b2950 0x7fc0780b4e60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc0840a6f60 tx=0x7fc08401bce0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.421+0000 7fc077fff640 1 -- 172.21.15.114:0/688756748 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0880043d0 con 0x555a99e7e2f0 2024-05-27T07:53:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.421+0000 7fc077fff640 1 -- 172.21.15.114:0/688756748 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc088004c00 con 0x555a99e7e2f0 2024-05-27T07:53:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.422+0000 7fc09ae09740 1 -- 172.21.15.114:0/688756748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e7e2f0 msgr2=0x555a9a248910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.422+0000 7fc09ae09740 1 --2- 172.21.15.114:0/688756748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e7e2f0 0x555a9a248910 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc088009c00 tx=0x7fc088011810 comp rx=0 tx=0).stop 2024-05-27T07:53:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.422+0000 7fc09ae09740 1 -- 172.21.15.114:0/688756748 shutdown_connections 2024-05-27T07:53:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.422+0000 7fc09ae09740 1 --2- 172.21.15.114:0/688756748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e864d0 0x555a9a248e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.422+0000 7fc09ae09740 1 --2- 172.21.15.114:0/688756748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e7e2f0 0x555a9a248910 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.422+0000 7fc09ae09740 1 --2- 172.21.15.114:0/688756748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99ea71b0 0x555a9a2483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.422+0000 7fc09ae09740 1 -- 172.21.15.114:0/688756748 >> 172.21.15.114:0/688756748 conn(0x555a99e9b300 msgr2=0x555a9a244b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.422+0000 7fc09ae09740 1 -- 172.21.15.114:0/688756748 shutdown_connections 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.422+0000 7fc09ae09740 1 -- 172.21.15.114:0/688756748 wait complete. 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.423+0000 7fc09ae09740 1 -- start start 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.423+0000 7fc09ae09740 1 Processor -- start 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.423+0000 7fc09ae09740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e7e2f0 0x555a9a252a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.423+0000 7fc09ae09740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e864d0 0x555a9a252fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.423+0000 7fc09ae09740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99ea71b0 0x555a9a258830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.423+0000 7fc09ae09740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555a99e90790 con 0x555a99ea71b0 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.423+0000 7fc09ae09740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555a9a166890 con 0x555a99e864d0 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.423+0000 7fc09ae09740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555a9a255500 con 0x555a99e7e2f0 2024-05-27T07:53:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.423+0000 7fc09ae09740 10 client.0 _ll_register_callbacks cb 0x555a99ebea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0914f2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e7e2f0 0x555a9a252a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc091cf3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e864d0 0x555a9a252fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0914f2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e7e2f0 0x555a9a252a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35364/0 (socket says 172.21.15.114:35364) 2024-05-27T07:53:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc091cf3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e864d0 0x555a9a252fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:38100/0 (socket says 172.21.15.114:38100) 2024-05-27T07:53:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0914f2640 1 -- 172.21.15.114:0/3898574318 learned_addr learned my addr 172.21.15.114:0/3898574318 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0914f2640 1 --2- 172.21.15.114:0/3898574318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99ea71b0 0x555a9a258830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0914f2640 1 -- 172.21.15.114:0/3898574318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e864d0 msgr2=0x555a9a252fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0914f2640 1 --2- 172.21.15.114:0/3898574318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e864d0 0x555a9a252fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0914f2640 1 -- 172.21.15.114:0/3898574318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99ea71b0 msgr2=0x555a9a258830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0914f2640 1 --2- 172.21.15.114:0/3898574318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99ea71b0 0x555a9a258830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0914f2640 1 -- 172.21.15.114:0/3898574318 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a99f8b620 con 0x555a99e7e2f0 2024-05-27T07:53:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0914f2640 1 --2- 172.21.15.114:0/3898574318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e7e2f0 0x555a9a252a60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0057c0 tx=0x7fc07c032ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ba4a5f8de87150b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0757fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:53:11.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0757fa640 1 -- 172.21.15.114:0/3898574318 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc07c041070 con 0x555a99e7e2f0 2024-05-27T07:53:11.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc09ae09740 1 -- 172.21.15.114:0/3898574318 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0880043d0 con 0x555a99e7e2f0 2024-05-27T07:53:11.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0757fa640 1 -- 172.21.15.114:0/3898574318 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9642+0+0 (secure 0 0 0) 0x7fc07c005220 con 0x555a99e7e2f0 2024-05-27T07:53:11.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc0757fa640 1 -- 172.21.15.114:0/3898574318 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fc07c052050 con 0x555a99e7e2f0 2024-05-27T07:53:11.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc09ae09740 10 client.28126 fetch_fsmap learned FSMap version 73 2024-05-27T07:53:11.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.424+0000 7fc09ae09740 1 -- 172.21.15.114:0/3898574318 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x555a9a2617e0 con 0x555a99e7e2f0 2024-05-27T07:53:11.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.425+0000 7fc0757fa640 1 -- 172.21.15.114:0/3898574318 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0880043d0 con 0x555a99e7e2f0 2024-05-27T07:53:11.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.425+0000 7fc0757fa640 1 -- 172.21.15.114:0/3898574318 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7fc07c04cb90 con 0x555a99e7e2f0 2024-05-27T07:53:11.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.425+0000 7fc09ae09740 10 client.28126 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:53:11.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.425+0000 7fc09ae09740 10 client.28126 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:11.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.425+0000 7fc09ae09740 20 client.28126 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:11.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.425+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3898574318 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555a99e65d80 0x555a9a264f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.425+0000 7fc09ae09740 4 client.28126 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.425+0000 7fc09ae09740 1 -- 172.21.15.114:0/3898574318 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a9a1bcb80 con 0x555a99e65d80 2024-05-27T07:53:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.425+0000 7fc091cf3640 1 --2- 172.21.15.114:0/3898574318 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555a99e65d80 0x555a9a264f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.426+0000 7fc0757fa640 10 client.28126 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.426+0000 7fc091cf3640 1 --2- 172.21.15.114:0/3898574318 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555a99e65d80 0x555a9a264f30 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=117b5d8e2e8908a1 server_cookie=52744eb359e5f590 in_seq=0 out_seq=0 2024-05-27T07:53:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.426+0000 7fc0757fa640 1 -- 172.21.15.114:0/3898574318 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555a9a1bcb80 con 0x555a99e65d80 2024-05-27T07:53:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.426+0000 7fc0757fa640 10 client.28126 handle_command_reply: tid=0 2024-05-27T07:53:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.426+0000 7fc09ae09740 1 client.28126 shutdown 2024-05-27T07:53:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.426+0000 7fc09ae09740 1 -- 172.21.15.114:0/3898574318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e7e2f0 msgr2=0x555a9a252a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.426+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3898574318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e7e2f0 0x555a9a252a60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0057c0 tx=0x7fc07c032ed0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.427+0000 7fc09ae09740 1 -- 172.21.15.114:0/3898574318 shutdown_connections 2024-05-27T07:53:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.427+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3898574318 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555a99e65d80 0x555a9a264f30 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.427+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3898574318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99ea71b0 0x555a9a258830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.427+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3898574318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e864d0 0x555a9a252fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.427+0000 7fc0757fa640 0 client.28126 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.427+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3898574318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e7e2f0 0x555a9a252a60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.427+0000 7fc09ae09740 1 -- 172.21.15.114:0/3898574318 >> 172.21.15.114:0/3898574318 conn(0x555a99e9b300 msgr2=0x555a9a24a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.427+0000 7fc09ae09740 1 -- 172.21.15.114:0/3898574318 shutdown_connections 2024-05-27T07:53:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.427+0000 7fc09ae09740 1 -- 172.21.15.114:0/3898574318 wait complete. 2024-05-27T07:53:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.427+0000 7fc09ae09740 20 client.28126 trim_cache size 0 max 16384 2024-05-27T07:53:11.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.444+0000 7fc09ae09740 1 Processor -- start 2024-05-27T07:53:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.444+0000 7fc09ae09740 1 -- start start 2024-05-27T07:53:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc09ae09740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99e65d80 0x555a9a24a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc09ae09740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e7e2f0 0x555a9a25f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc09ae09740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e864d0 0x555a9a24c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc09ae09740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c1b2f10 con 0x555a99e65d80 2024-05-27T07:53:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc09ae09740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c1b16a0 con 0x555a99e7e2f0 2024-05-27T07:53:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc09ae09740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c1af510 con 0x555a99e864d0 2024-05-27T07:53:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc0914f2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99e65d80 0x555a9a24a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc0914f2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99e65d80 0x555a9a24a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57782/0 (socket says 172.21.15.114:57782) 2024-05-27T07:53:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc091cf3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e7e2f0 0x555a9a25f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc0914f2640 1 -- 172.21.15.114:0/3258851637 learned_addr learned my addr 172.21.15.114:0/3258851637 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc0914f2640 1 --2- 172.21.15.114:0/3258851637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e864d0 0x555a9a24c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc0914f2640 1 -- 172.21.15.114:0/3258851637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e864d0 msgr2=0x555a9a24c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc0914f2640 1 --2- 172.21.15.114:0/3258851637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e864d0 0x555a9a24c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc0914f2640 1 -- 172.21.15.114:0/3258851637 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e7e2f0 msgr2=0x555a9a25f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc0914f2640 1 --2- 172.21.15.114:0/3258851637 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e7e2f0 0x555a9a25f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc0914f2640 1 -- 172.21.15.114:0/3258851637 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a9a245730 con 0x555a99e65d80 2024-05-27T07:53:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.445+0000 7fc091cf3640 1 --2- 172.21.15.114:0/3258851637 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e7e2f0 0x555a9a25f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.446+0000 7fc0914f2640 1 --2- 172.21.15.114:0/3258851637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99e65d80 0x555a9a24a010 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc07c004290 tx=0x7fc07c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9362e9ea384d0003 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.446+0000 7fc0777fe640 1 -- 172.21.15.114:0/3258851637 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc07c041030 con 0x555a99e65d80 2024-05-27T07:53:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.446+0000 7fc0777fe640 1 -- 172.21.15.114:0/3258851637 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc07c003990 con 0x555a99e65d80 2024-05-27T07:53:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.446+0000 7fc0777fe640 1 -- 172.21.15.114:0/3258851637 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc07c035040 con 0x555a99e65d80 2024-05-27T07:53:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.447+0000 7fc09ae09740 1 -- 172.21.15.114:0/3258851637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99e65d80 msgr2=0x555a9a24a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.447+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3258851637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99e65d80 0x555a9a24a010 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc07c004290 tx=0x7fc07c000bc0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.447+0000 7fc09ae09740 1 -- 172.21.15.114:0/3258851637 shutdown_connections 2024-05-27T07:53:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.447+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3258851637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e864d0 0x555a9a24c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.447+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3258851637 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e7e2f0 0x555a9a25f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.447+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3258851637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99e65d80 0x555a9a24a010 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.447+0000 7fc09ae09740 1 -- 172.21.15.114:0/3258851637 >> 172.21.15.114:0/3258851637 conn(0x555a99e9b300 msgr2=0x555a9a2549a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.447+0000 7fc09ae09740 1 -- 172.21.15.114:0/3258851637 shutdown_connections 2024-05-27T07:53:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.447+0000 7fc09ae09740 1 -- 172.21.15.114:0/3258851637 wait complete. 2024-05-27T07:53:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.447+0000 7fc09ae09740 1 -- start start 2024-05-27T07:53:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.447+0000 7fc09ae09740 1 Processor -- start 2024-05-27T07:53:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc09ae09740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e65d80 0x555a9a257ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc09ae09740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e7e2f0 0x555a9a2580e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc09ae09740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99e864d0 0x555a9a258620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc09ae09740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c1afb60 con 0x555a99e864d0 2024-05-27T07:53:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc09ae09740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c1b0fd0 con 0x555a99e7e2f0 2024-05-27T07:53:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc09ae09740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c1b0230 con 0x555a99e65d80 2024-05-27T07:53:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc09ae09740 10 client.0 _ll_register_callbacks cb 0x7fc07c004180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc091cf3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e7e2f0 0x555a9a2580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc0914f2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99e864d0 0x555a9a258620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc0914f2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e65d80 0x555a9a257ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc091cf3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e7e2f0 0x555a9a2580e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:38114/0 (socket says 172.21.15.114:38114) 2024-05-27T07:53:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc091cf3640 1 -- 172.21.15.114:0/3923703105 learned_addr learned my addr 172.21.15.114:0/3923703105 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc091cf3640 1 -- 172.21.15.114:0/3923703105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e65d80 msgr2=0x555a9a257ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc091cf3640 1 --2- 172.21.15.114:0/3923703105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e65d80 0x555a9a257ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc091cf3640 1 -- 172.21.15.114:0/3923703105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99e864d0 msgr2=0x555a9a258620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.448+0000 7fc091cf3640 1 --2- 172.21.15.114:0/3923703105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99e864d0 0x555a9a258620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.449+0000 7fc091cf3640 1 -- 172.21.15.114:0/3923703105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a9a2cb640 con 0x555a99e7e2f0 2024-05-27T07:53:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.449+0000 7fc091cf3640 1 --2- 172.21.15.114:0/3923703105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e7e2f0 0x555a9a2580e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc088009b20 tx=0x7fc088009e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=329a3f4e0efbedc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.449+0000 7fc0757fa640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:53:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.449+0000 7fc0757fa640 1 -- 172.21.15.114:0/3923703105 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08801e040 con 0x555a99e7e2f0 2024-05-27T07:53:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.449+0000 7fc0757fa640 1 -- 172.21.15.114:0/3923703105 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9642+0+0 (secure 0 0 0) 0x7fc08802ce00 con 0x555a99e7e2f0 2024-05-27T07:53:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.449+0000 7fc09ae09740 1 -- 172.21.15.114:0/3923703105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc07c003990 con 0x555a99e7e2f0 2024-05-27T07:53:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.449+0000 7fc0757fa640 1 -- 172.21.15.114:0/3923703105 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fc08803a050 con 0x555a99e7e2f0 2024-05-27T07:53:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.449+0000 7fc09ae09740 10 client.27995 fetch_fsmap learned FSMap version 73 2024-05-27T07:53:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.449+0000 7fc09ae09740 1 -- 172.21.15.114:0/3923703105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x555a9a2cc900 con 0x555a99e7e2f0 2024-05-27T07:53:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.449+0000 7fc0757fa640 1 -- 172.21.15.114:0/3923703105 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc07c003990 con 0x555a99e7e2f0 2024-05-27T07:53:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.450+0000 7fc0757fa640 1 -- 172.21.15.114:0/3923703105 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7fc088024040 con 0x555a99e7e2f0 2024-05-27T07:53:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.450+0000 7fc09ae09740 10 client.27995 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:53:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.450+0000 7fc09ae09740 10 client.27995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.450+0000 7fc09ae09740 20 client.27995 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.450+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3923703105 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555a99ea71b0 0x555a9a2cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.450+0000 7fc09ae09740 4 client.27995 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:53:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.450+0000 7fc09ae09740 1 -- 172.21.15.114:0/3923703105 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555a9a1b6360 con 0x555a99ea71b0 2024-05-27T07:53:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.450+0000 7fc0914f2640 1 --2- 172.21.15.114:0/3923703105 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555a99ea71b0 0x555a9a2cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.450+0000 7fc0757fa640 10 client.27995 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.450+0000 7fc0914f2640 1 --2- 172.21.15.114:0/3923703105 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555a99ea71b0 0x555a9a2cf0f0 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60de64a74dcc38e1 server_cookie=db84b67bddc1a581 in_seq=0 out_seq=0 2024-05-27T07:53:11.506 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:11.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.764+0000 7fc0757fa640 1 -- 172.21.15.114:0/3923703105 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x555a9a1b6360 con 0x555a99ea71b0 2024-05-27T07:53:11.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.764+0000 7fc0757fa640 10 client.27995 handle_command_reply: tid=0 2024-05-27T07:53:11.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.765+0000 7fc09ae09740 1 client.27995 shutdown 2024-05-27T07:53:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.765+0000 7fc09ae09740 1 -- 172.21.15.114:0/3923703105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e7e2f0 msgr2=0x555a9a2580e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.765+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3923703105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e7e2f0 0x555a9a2580e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc088009b20 tx=0x7fc088009e70 comp rx=0 tx=0).stop 2024-05-27T07:53:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.765+0000 7fc09ae09740 1 -- 172.21.15.114:0/3923703105 shutdown_connections 2024-05-27T07:53:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.765+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3923703105 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555a99ea71b0 0x555a9a2cf0f0 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.765+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3923703105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a99e864d0 0x555a9a258620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.765+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3923703105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a99e7e2f0 0x555a9a2580e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.765+0000 7fc09ae09740 1 --2- 172.21.15.114:0/3923703105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a99e65d80 0x555a9a257ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.765+0000 7fc09ae09740 1 -- 172.21.15.114:0/3923703105 >> 172.21.15.114:0/3923703105 conn(0x555a99e9b300 msgr2=0x555a9a2482e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.765+0000 7fc0757fa640 0 client.27995 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.765+0000 7fc09ae09740 1 -- 172.21.15.114:0/3923703105 shutdown_connections 2024-05-27T07:53:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.765+0000 7fc09ae09740 1 -- 172.21.15.114:0/3923703105 wait complete. 2024-05-27T07:53:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.765+0000 7fc09ae09740 20 client.27995 trim_cache size 0 max 16384 2024-05-27T07:53:11.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:53:11.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:53:11.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:53:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.766+0000 7fc093f57640 1 -- 172.21.15.114:0/4217380377 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc0780b2950 msgr2=0x7fc0780b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.766+0000 7fc093f57640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc0780b2950 0x7fc0780b4e60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc0840a6f60 tx=0x7fc08401bce0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.766+0000 7fc093f57640 1 -- 172.21.15.114:0/4217380377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc08c10e1e0 msgr2=0x7fc08c19a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.766+0000 7fc093f57640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc08c10e1e0 0x7fc08c19a740 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fc08400efd0 tx=0x7fc08400c6a0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.766+0000 7fc093f57640 1 -- 172.21.15.114:0/4217380377 shutdown_connections 2024-05-27T07:53:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.766+0000 7fc093f57640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc0780b2950 0x7fc0780b4e60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.766+0000 7fc093f57640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc08c10e1e0 0x7fc08c19a740 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.766+0000 7fc093f57640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc08c10b7f0 0x7fc08c1a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.766+0000 7fc093f57640 1 --2- 172.21.15.114:0/4217380377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc08c104d90 0x7fc08c1a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.766+0000 7fc093f57640 1 -- 172.21.15.114:0/4217380377 >> 172.21.15.114:0/4217380377 conn(0x7fc08c0782d0 msgr2=0x7fc08c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.766+0000 7fc093f57640 1 -- 172.21.15.114:0/4217380377 shutdown_connections 2024-05-27T07:53:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:11.766+0000 7fc093f57640 1 -- 172.21.15.114:0/4217380377 wait complete. 2024-05-27T07:53:12.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1280147 9.34 MB/sec execute 564 sec latency 151.130 ms 2024-05-27T07:53:12.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:12 smithi163 ceph-mon[36451]: pgmap v450: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 71 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 1.8 MiB/s wr, 155 op/s 2024-05-27T07:53:12.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:12 smithi186 ceph-mon[36647]: pgmap v450: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 71 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 1.8 MiB/s wr, 155 op/s 2024-05-27T07:53:12.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:12 smithi114 ceph-mon[32038]: pgmap v450: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 71 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 1.8 MiB/s wr, 155 op/s 2024-05-27T07:53:13.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1282142 9.34 MB/sec execute 565 sec latency 44.762 ms 2024-05-27T07:53:13.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.984+0000 7f6a5805c700 1 -- 172.21.15.114:0/2581759019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a501045e0 msgr2=0x7f6a501049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:13.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.984+0000 7f6a5805c700 1 --2- 172.21.15.114:0/2581759019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a501045e0 0x7f6a501049f0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c0099c0 tx=0x7f6a4c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:53:13.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.984+0000 7f6a5805c700 1 -- 172.21.15.114:0/2581759019 shutdown_connections 2024-05-27T07:53:13.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.984+0000 7f6a5805c700 1 --2- 172.21.15.114:0/2581759019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a50109860 0x7f6a501111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:13.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.984+0000 7f6a5805c700 1 --2- 172.21.15.114:0/2581759019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a501045e0 0x7f6a501049f0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:13.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.984+0000 7f6a5805c700 1 --2- 172.21.15.114:0/2581759019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a50108f10 0x7f6a50109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:13.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.984+0000 7f6a5805c700 1 -- 172.21.15.114:0/2581759019 >> 172.21.15.114:0/2581759019 conn(0x7f6a50100030 msgr2=0x7f6a50102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:13.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.984+0000 7f6a5805c700 1 -- 172.21.15.114:0/2581759019 shutdown_connections 2024-05-27T07:53:13.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.984+0000 7f6a5805c700 1 -- 172.21.15.114:0/2581759019 wait complete. 2024-05-27T07:53:13.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a5805c700 1 Processor -- start 2024-05-27T07:53:13.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a5805c700 1 -- start start 2024-05-27T07:53:13.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a5805c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a501045e0 0x7f6a5019e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:13.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a5805c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a50108f10 0x7f6a5019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:13.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a5805c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a50109860 0x7f6a50198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:13.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a5805c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6a50112330 con 0x7f6a50109860 2024-05-27T07:53:13.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a5805c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6a501121b0 con 0x7f6a50108f10 2024-05-27T07:53:13.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a5805c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6a501124b0 con 0x7f6a501045e0 2024-05-27T07:53:13.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a565f9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a50109860 0x7f6a50198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:13.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a555f7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a50108f10 0x7f6a5019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:13.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a55df8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a501045e0 0x7f6a5019e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:13.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a555f7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a50108f10 0x7f6a5019ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:38120/0 (socket says 172.21.15.114:38120) 2024-05-27T07:53:13.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a565f9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a50109860 0x7f6a50198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57820/0 (socket says 172.21.15.114:57820) 2024-05-27T07:53:13.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a555f7700 1 -- 172.21.15.114:0/2393610935 learned_addr learned my addr 172.21.15.114:0/2393610935 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:13.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.985+0000 7f6a55df8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a501045e0 0x7f6a5019e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35400/0 (socket says 172.21.15.114:35400) 2024-05-27T07:53:13.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.986+0000 7f6a565f9700 1 -- 172.21.15.114:0/2393610935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a501045e0 msgr2=0x7f6a5019e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:13.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.986+0000 7f6a565f9700 1 --2- 172.21.15.114:0/2393610935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a501045e0 0x7f6a5019e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:13.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.986+0000 7f6a565f9700 1 -- 172.21.15.114:0/2393610935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a50108f10 msgr2=0x7f6a5019ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:13.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.986+0000 7f6a565f9700 1 --2- 172.21.15.114:0/2393610935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a50108f10 0x7f6a5019ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:13.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.986+0000 7f6a565f9700 1 -- 172.21.15.114:0/2393610935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a50199160 con 0x7f6a50109860 2024-05-27T07:53:13.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.986+0000 7f6a55df8700 1 --2- 172.21.15.114:0/2393610935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a501045e0 0x7f6a5019e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:13.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.986+0000 7f6a565f9700 1 --2- 172.21.15.114:0/2393610935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a50109860 0x7f6a50198ab0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800b6f0 tx=0x7f6a48009720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:13.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.986+0000 7f6a46ffd700 1 -- 172.21.15.114:0/2393610935 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a48003c00 con 0x7f6a50109860 2024-05-27T07:53:13.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.986+0000 7f6a46ffd700 1 -- 172.21.15.114:0/2393610935 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a4800bdc0 con 0x7f6a50109860 2024-05-27T07:53:13.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.986+0000 7f6a46ffd700 1 -- 172.21.15.114:0/2393610935 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a48004280 con 0x7f6a50109860 2024-05-27T07:53:13.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.986+0000 7f6a5805c700 1 -- 172.21.15.114:0/2393610935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a50199450 con 0x7f6a50109860 2024-05-27T07:53:13.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.987+0000 7f6a5805c700 1 -- 172.21.15.114:0/2393610935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a501998e0 con 0x7f6a50109860 2024-05-27T07:53:13.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.988+0000 7f6a5805c700 1 -- 172.21.15.114:0/2393610935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a500512b0 con 0x7f6a50109860 2024-05-27T07:53:13.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.990+0000 7f6a46ffd700 1 -- 172.21.15.114:0/2393610935 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6a4800b910 con 0x7f6a50109860 2024-05-27T07:53:13.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.991+0000 7f6a46ffd700 1 --2- 172.21.15.114:0/2393610935 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6a3c0b2b40 0x7f6a3c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:13.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.991+0000 7f6a55df8700 1 --2- 172.21.15.114:0/2393610935 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6a3c0b2b40 0x7f6a3c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:13.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.991+0000 7f6a46ffd700 1 -- 172.21.15.114:0/2393610935 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9642+0+0 (secure 0 0 0) 0x7f6a480e2c80 con 0x7f6a50109860 2024-05-27T07:53:13.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.991+0000 7f6a55df8700 1 --2- 172.21.15.114:0/2393610935 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6a3c0b2b40 0x7f6a3c0b5050 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c000c00 tx=0x7f6a4c01a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:13.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:13.995+0000 7f6a46ffd700 1 -- 172.21.15.114:0/2393610935 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6a48052df0 con 0x7f6a50109860 2024-05-27T07:53:14.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.285+0000 7f6a5805c700 1 -- 172.21.15.114:0/2393610935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a5002e5e0 con 0x7f6a50109860 2024-05-27T07:53:14.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.286+0000 7f6a46ffd700 1 -- 172.21.15.114:0/2393610935 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+8522 (secure 0 0 0) 0x7f6a501998e0 con 0x7f6a50109860 2024-05-27T07:53:14.288 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:53:14.288 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":27557,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3261223371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":3261223371},{"type":"v1","addr":"172.21.15.163:6835","nonce":3261223371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":73,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:52:49.276264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_28030":{"gid":28030,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1524985663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1524985663},{"type":"v1","addr":"172.21.15.186:6833","nonce":1524985663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:53:14.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.289+0000 7f6a5805c700 1 -- 172.21.15.114:0/2393610935 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6a3c0b2b40 msgr2=0x7f6a3c0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:14.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.289+0000 7f6a5805c700 1 --2- 172.21.15.114:0/2393610935 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6a3c0b2b40 0x7f6a3c0b5050 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c000c00 tx=0x7f6a4c01a040 comp rx=0 tx=0).stop 2024-05-27T07:53:14.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.289+0000 7f6a5805c700 1 -- 172.21.15.114:0/2393610935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a50109860 msgr2=0x7f6a50198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:14.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.289+0000 7f6a5805c700 1 --2- 172.21.15.114:0/2393610935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a50109860 0x7f6a50198ab0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800b6f0 tx=0x7f6a48009720 comp rx=0 tx=0).stop 2024-05-27T07:53:14.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.289+0000 7f6a5805c700 1 -- 172.21.15.114:0/2393610935 shutdown_connections 2024-05-27T07:53:14.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.289+0000 7f6a5805c700 1 --2- 172.21.15.114:0/2393610935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a50109860 0x7f6a50198ab0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:14.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.289+0000 7f6a5805c700 1 --2- 172.21.15.114:0/2393610935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a501045e0 0x7f6a5019e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:14.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.289+0000 7f6a5805c700 1 --2- 172.21.15.114:0/2393610935 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6a3c0b2b40 0x7f6a3c0b5050 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:14.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.289+0000 7f6a5805c700 1 --2- 172.21.15.114:0/2393610935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a50108f10 0x7f6a5019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:14.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.289+0000 7f6a5805c700 1 -- 172.21.15.114:0/2393610935 >> 172.21.15.114:0/2393610935 conn(0x7f6a50100030 msgr2=0x7f6a50102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:14.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.289+0000 7f6a5805c700 1 -- 172.21.15.114:0/2393610935 shutdown_connections 2024-05-27T07:53:14.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:14.290+0000 7f6a5805c700 1 -- 172.21.15.114:0/2393610935 wait complete. 2024-05-27T07:53:14.292 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 73 2024-05-27T07:53:14.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1283463 9.34 MB/sec execute 566 sec latency 14.531 ms 2024-05-27T07:53:14.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:14 smithi114 ceph-mon[32038]: pgmap v451: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 71 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-05-27T07:53:14.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:14 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2393610935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:53:14.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:14 smithi163 ceph-mon[36451]: pgmap v451: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 71 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-05-27T07:53:14.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:14 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2393610935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:53:14.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:14 smithi186 ceph-mon[36647]: pgmap v451: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 71 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-05-27T07:53:14.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:14 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2393610935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:53:15.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1285223 9.33 MB/sec execute 567 sec latency 121.137 ms 2024-05-27T07:53:15.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:53:15.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:15 smithi114 ceph-mon[32038]: osdmap e131: 12 total, 12 up, 12 in 2024-05-27T07:53:15.598 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:15.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:15 smithi163 ceph-mon[36451]: osdmap e131: 12 total, 12 up, 12 in 2024-05-27T07:53:15.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:15 smithi186 ceph-mon[36647]: osdmap e131: 12 total, 12 up, 12 in 2024-05-27T07:53:16.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1286236 9.33 MB/sec execute 568 sec latency 60.405 ms 2024-05-27T07:53:16.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:16 smithi163 ceph-mon[36451]: pgmap v453: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 71 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 1.8 MiB/s wr, 124 op/s 2024-05-27T07:53:16.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:16 smithi186 ceph-mon[36647]: pgmap v453: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 71 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 1.8 MiB/s wr, 124 op/s 2024-05-27T07:53:16.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:16 smithi114 ceph-mon[32038]: pgmap v453: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 71 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 1.8 MiB/s wr, 124 op/s 2024-05-27T07:53:17.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1287980 9.32 MB/sec execute 569 sec latency 72.029 ms 2024-05-27T07:53:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.538+0000 7f1920048700 1 -- 172.21.15.114:0/1245568696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1918108760 msgr2=0x7f19181100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.538+0000 7f1920048700 1 --2- 172.21.15.114:0/1245568696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1918108760 0x7f19181100c0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f191000b0d0 tx=0x7f191000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.538+0000 7f1920048700 1 -- 172.21.15.114:0/1245568696 shutdown_connections 2024-05-27T07:53:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.538+0000 7f1920048700 1 --2- 172.21.15.114:0/1245568696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1918108760 0x7f19181100c0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.538+0000 7f1920048700 1 --2- 172.21.15.114:0/1245568696 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1918103530 0x7f1918103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.538+0000 7f1920048700 1 --2- 172.21.15.114:0/1245568696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1918107e10 0x7f1918108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.538+0000 7f1920048700 1 -- 172.21.15.114:0/1245568696 >> 172.21.15.114:0/1245568696 conn(0x7f1918078e40 msgr2=0x7f1918079280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.538+0000 7f1920048700 1 -- 172.21.15.114:0/1245568696 shutdown_connections 2024-05-27T07:53:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.538+0000 7f1920048700 1 -- 172.21.15.114:0/1245568696 wait complete. 2024-05-27T07:53:17.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.539+0000 7f1920048700 1 Processor -- start 2024-05-27T07:53:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.539+0000 7f1920048700 1 -- start start 2024-05-27T07:53:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f1920048700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1918103530 0x7f191819f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f1920048700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1918107e10 0x7f191819fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f1920048700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1918108760 0x7f1918199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f1920048700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1918113390 con 0x7f1918107e10 2024-05-27T07:53:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f1920048700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1918113210 con 0x7f1918103530 2024-05-27T07:53:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f1920048700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1918113510 con 0x7f1918108760 2024-05-27T07:53:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f191d5e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1918107e10 0x7f191819fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f191dde4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1918103530 0x7f191819f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f191d5e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1918107e10 0x7f191819fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36970/0 (socket says 172.21.15.114:36970) 2024-05-27T07:53:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f191d5e3700 1 -- 172.21.15.114:0/73568942 learned_addr learned my addr 172.21.15.114:0/73568942 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f191e5e5700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1918108760 0x7f1918199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f191dde4700 1 -- 172.21.15.114:0/73568942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1918108760 msgr2=0x7f1918199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f191dde4700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1918108760 0x7f1918199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f191dde4700 1 -- 172.21.15.114:0/73568942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1918107e10 msgr2=0x7f191819fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f191dde4700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1918107e10 0x7f191819fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f191dde4700 1 -- 172.21.15.114:0/73568942 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f191819a1b0 con 0x7f1918103530 2024-05-27T07:53:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f191d5e3700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1918107e10 0x7f191819fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.540+0000 7f191dde4700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1918103530 0x7f191819f890 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f191400e9f0 tx=0x7f191400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.541+0000 7f190effd700 1 -- 172.21.15.114:0/73568942 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191400c9c0 con 0x7f1918103530 2024-05-27T07:53:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.541+0000 7f1920048700 1 -- 172.21.15.114:0/73568942 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f191819a4a0 con 0x7f1918103530 2024-05-27T07:53:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.541+0000 7f190effd700 1 -- 172.21.15.114:0/73568942 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f191400cb60 con 0x7f1918103530 2024-05-27T07:53:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.541+0000 7f190effd700 1 -- 172.21.15.114:0/73568942 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1914018680 con 0x7f1918103530 2024-05-27T07:53:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.541+0000 7f1920048700 1 -- 172.21.15.114:0/73568942 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19180759e0 con 0x7f1918103530 2024-05-27T07:53:17.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.544+0000 7f190effd700 1 -- 172.21.15.114:0/73568942 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f1914010430 con 0x7f1918103530 2024-05-27T07:53:17.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.544+0000 7f192ea5db80 1 Processor -- start 2024-05-27T07:53:17.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f192ea5db80 1 -- start start 2024-05-27T07:53:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f192ea5db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 0x559da3122710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f192ea5db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da3122c50 0x559da311ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f192ea5db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da311cf80 0x559da311d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f192ea5db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559da304bc90 con 0x559da3122c50 2024-05-27T07:53:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f192ea5db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559da303b830 con 0x559da311cf80 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f192ea5db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559da303f870 con 0x559da2f9ee40 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f190effd700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f19040b29f0 0x7f19040b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f191dde4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da3122c50 0x559da311ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f191d5e3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da311cf80 0x559da311d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f191dde4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da3122c50 0x559da311ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36972/0 (socket says 172.21.15.114:36972) 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f191d5e3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da311cf80 0x559da311d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:45250/0 (socket says 172.21.15.114:45250) 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f191dde4700 1 -- 172.21.15.114:0/272168119 learned_addr learned my addr 172.21.15.114:0/272168119 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f191e5e5700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f19040b29f0 0x7f19040b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.545+0000 7f191e5e5700 1 --2- 172.21.15.114:0/272168119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 0x559da3122710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.546+0000 7f191dde4700 1 -- 172.21.15.114:0/272168119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 msgr2=0x559da3122710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.546+0000 7f190effd700 1 -- 172.21.15.114:0/73568942 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9566+0+0 (secure 0 0 0) 0x7f19140e2be0 con 0x7f1918103530 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.546+0000 7f191dde4700 1 --2- 172.21.15.114:0/272168119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 0x559da3122710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.546+0000 7f191dde4700 1 -- 172.21.15.114:0/272168119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da311cf80 msgr2=0x559da311d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.546+0000 7f191dde4700 1 --2- 172.21.15.114:0/272168119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da311cf80 0x559da311d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.546+0000 7f191dde4700 1 -- 172.21.15.114:0/272168119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559da2f8b8d0 con 0x559da3122c50 2024-05-27T07:53:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.546+0000 7f191d5e3700 1 --2- 172.21.15.114:0/272168119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da311cf80 0x559da311d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.546+0000 7f191e5e5700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f19040b29f0 0x7f19040b4f00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x559da311dbb0 tx=0x7f19100099e0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.546+0000 7f191dde4700 1 --2- 172.21.15.114:0/272168119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da3122c50 0x559da311ca40 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f19140a5ad0 tx=0x7f1914009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ad5fa8ca14eadd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.546+0000 7f1903fff700 1 -- 172.21.15.114:0/272168119 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19140a5070 con 0x559da3122c50 2024-05-27T07:53:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.546+0000 7f1903fff700 1 -- 172.21.15.114:0/272168119 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19140b2030 con 0x559da3122c50 2024-05-27T07:53:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.546+0000 7f1903fff700 1 -- 172.21.15.114:0/272168119 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19140a9f10 con 0x559da3122c50 2024-05-27T07:53:17.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.547+0000 7f192ea5db80 1 -- 172.21.15.114:0/272168119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da3122c50 msgr2=0x559da311ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:17.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.547+0000 7f192ea5db80 1 --2- 172.21.15.114:0/272168119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da3122c50 0x559da311ca40 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f19140a5ad0 tx=0x7f1914009790 comp rx=0 tx=0).stop 2024-05-27T07:53:17.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.547+0000 7f192ea5db80 1 -- 172.21.15.114:0/272168119 shutdown_connections 2024-05-27T07:53:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.547+0000 7f192ea5db80 1 --2- 172.21.15.114:0/272168119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da311cf80 0x559da311d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.547+0000 7f192ea5db80 1 --2- 172.21.15.114:0/272168119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 0x559da3122710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.547+0000 7f192ea5db80 1 --2- 172.21.15.114:0/272168119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da3122c50 0x559da311ca40 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.547+0000 7f192ea5db80 1 -- 172.21.15.114:0/272168119 >> 172.21.15.114:0/272168119 conn(0x559da2ff9af0 msgr2=0x559da3117640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.547+0000 7f192ea5db80 1 -- 172.21.15.114:0/272168119 shutdown_connections 2024-05-27T07:53:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.547+0000 7f192ea5db80 1 -- 172.21.15.114:0/272168119 wait complete. 2024-05-27T07:53:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.548+0000 7f192ea5db80 1 -- start start 2024-05-27T07:53:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.548+0000 7f192ea5db80 1 Processor -- start 2024-05-27T07:53:17.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.548+0000 7f192ea5db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 0x559da3126360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.548+0000 7f192ea5db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da31268a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.548+0000 7f192ea5db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da3122c50 0x559da3126de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.548+0000 7f192ea5db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559da2f41fa0 con 0x559da311cf80 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.548+0000 7f192ea5db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559da2fdd7c0 con 0x559da3122c50 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.548+0000 7f192ea5db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559da31296e0 con 0x559da2f9ee40 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.548+0000 7f192ea5db80 10 client.0 _ll_register_callbacks cb 0x559da2fd6920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.548+0000 7f191d5e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da31268a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.548+0000 7f191e5e5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da3122c50 0x559da3126de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f191d5e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da31268a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36976/0 (socket says 172.21.15.114:36976) 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f191d5e3700 1 -- 172.21.15.114:0/1438248479 learned_addr learned my addr 172.21.15.114:0/1438248479 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f191dde4700 1 --2- 172.21.15.114:0/1438248479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 0x559da3126360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f191e5e5700 1 -- 172.21.15.114:0/1438248479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 msgr2=0x559da3126360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f191e5e5700 1 --2- 172.21.15.114:0/1438248479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 0x559da3126360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f191e5e5700 1 -- 172.21.15.114:0/1438248479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 msgr2=0x559da31268a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:17.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f191e5e5700 1 --2- 172.21.15.114:0/1438248479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da31268a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f191e5e5700 1 -- 172.21.15.114:0/1438248479 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559da2fda690 con 0x559da3122c50 2024-05-27T07:53:17.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f191d5e3700 1 --2- 172.21.15.114:0/1438248479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da31268a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:17.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f191e5e5700 1 --2- 172.21.15.114:0/1438248479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da3122c50 0x559da3126de0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f191000b770 tx=0x7f191000b3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4ff15ea39f5ca13 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:17.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f19017fa700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:53:17.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f19017fa700 1 -- 172.21.15.114:0/1438248479 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191000be60 con 0x559da3122c50 2024-05-27T07:53:17.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f192ea5db80 1 -- 172.21.15.114:0/1438248479 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19140b2030 con 0x559da3122c50 2024-05-27T07:53:17.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f19017fa700 1 -- 172.21.15.114:0/1438248479 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9566+0+0 (secure 0 0 0) 0x7f1910005310 con 0x559da3122c50 2024-05-27T07:53:17.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f19017fa700 1 -- 172.21.15.114:0/1438248479 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f191003a050 con 0x559da3122c50 2024-05-27T07:53:17.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f192ea5db80 10 client.28013 fetch_fsmap learned FSMap version 73 2024-05-27T07:53:17.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.549+0000 7f192ea5db80 1 -- 172.21.15.114:0/1438248479 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x559da3134190 con 0x559da3122c50 2024-05-27T07:53:17.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.550+0000 7f19017fa700 1 -- 172.21.15.114:0/1438248479 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19140b2030 con 0x559da3122c50 2024-05-27T07:53:17.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.550+0000 7f19017fa700 1 -- 172.21.15.114:0/1438248479 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7f1910011070 con 0x559da3122c50 2024-05-27T07:53:17.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.550+0000 7f192ea5db80 10 client.28013 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:53:17.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.550+0000 7f192ea5db80 10 client.28013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:17.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.550+0000 7f192ea5db80 20 client.28013 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:17.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.550+0000 7f192ea5db80 1 --2- 172.21.15.114:0/1438248479 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559da3136a80 0x559da3138ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.550+0000 7f192ea5db80 4 client.28013 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:17.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.550+0000 7f192ea5db80 1 -- 172.21.15.114:0/1438248479 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559da30e89f0 con 0x559da3136a80 2024-05-27T07:53:17.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.550+0000 7f191dde4700 1 --2- 172.21.15.114:0/1438248479 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559da3136a80 0x559da3138ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.550+0000 7f19017fa700 10 client.28013 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:17.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.550+0000 7f191dde4700 1 --2- 172.21.15.114:0/1438248479 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559da3136a80 0x559da3138ed0 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73b729bf2d508d26 server_cookie=bf7fa8f62e5fcc60 in_seq=0 out_seq=0 2024-05-27T07:53:17.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f19017fa700 1 -- 172.21.15.114:0/1438248479 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559da30e89f0 con 0x559da3136a80 2024-05-27T07:53:17.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f19017fa700 10 client.28013 handle_command_reply: tid=0 2024-05-27T07:53:17.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f192ea5db80 1 client.28013 shutdown 2024-05-27T07:53:17.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f192ea5db80 1 -- 172.21.15.114:0/1438248479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da3122c50 msgr2=0x559da3126de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:17.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f192ea5db80 1 --2- 172.21.15.114:0/1438248479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da3122c50 0x559da3126de0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f191000b770 tx=0x7f191000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f192ea5db80 1 -- 172.21.15.114:0/1438248479 shutdown_connections 2024-05-27T07:53:17.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f192ea5db80 1 --2- 172.21.15.114:0/1438248479 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559da3136a80 0x559da3138ed0 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f192ea5db80 1 --2- 172.21.15.114:0/1438248479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da3122c50 0x559da3126de0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f192ea5db80 1 --2- 172.21.15.114:0/1438248479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 0x559da3126360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f19017fa700 0 client.28013 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f192ea5db80 1 --2- 172.21.15.114:0/1438248479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da31268a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f192ea5db80 1 -- 172.21.15.114:0/1438248479 >> 172.21.15.114:0/1438248479 conn(0x559da2ff9af0 msgr2=0x559da311ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f192ea5db80 1 -- 172.21.15.114:0/1438248479 shutdown_connections 2024-05-27T07:53:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f192ea5db80 1 -- 172.21.15.114:0/1438248479 wait complete. 2024-05-27T07:53:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.552+0000 7f192ea5db80 20 client.28013 trim_cache size 0 max 16384 2024-05-27T07:53:17.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.568+0000 7f192ea5db80 1 Processor -- start 2024-05-27T07:53:17.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.568+0000 7f192ea5db80 1 -- start start 2024-05-27T07:53:17.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.568+0000 7f192ea5db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 0x559da311ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.568+0000 7f192ea5db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da3120a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.568+0000 7f192ea5db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da3122c50 0x559da3126c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.568+0000 7f192ea5db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f19181b20d0 con 0x559da311cf80 2024-05-27T07:53:17.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.568+0000 7f192ea5db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f19181b0840 con 0x559da3122c50 2024-05-27T07:53:17.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.568+0000 7f192ea5db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f19181ae690 con 0x559da2f9ee40 2024-05-27T07:53:17.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.568+0000 7f191d5e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da3120a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.568+0000 7f191d5e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da3120a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36984/0 (socket says 172.21.15.114:36984) 2024-05-27T07:53:17.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.568+0000 7f191d5e3700 1 -- 172.21.15.114:0/1750753788 learned_addr learned my addr 172.21.15.114:0/1750753788 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:17.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.568+0000 7f191dde4700 1 --2- 172.21.15.114:0/1750753788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 0x559da311ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.568+0000 7f191e5e5700 1 --2- 172.21.15.114:0/1750753788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da3122c50 0x559da3126c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.569+0000 7f191d5e3700 1 -- 172.21.15.114:0/1750753788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 msgr2=0x559da311ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:17.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.569+0000 7f191d5e3700 1 --2- 172.21.15.114:0/1750753788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 0x559da311ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.569+0000 7f191d5e3700 1 -- 172.21.15.114:0/1750753788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da3122c50 msgr2=0x559da3126c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:17.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.569+0000 7f191d5e3700 1 --2- 172.21.15.114:0/1750753788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da3122c50 0x559da3126c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.569+0000 7f191d5e3700 1 -- 172.21.15.114:0/1750753788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559da3111fa0 con 0x559da311cf80 2024-05-27T07:53:17.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.569+0000 7f191d5e3700 1 --2- 172.21.15.114:0/1750753788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da3120a40 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f1908004550 tx=0x7f1908011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11154448d2532b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:17.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.569+0000 7f19037fe700 1 -- 172.21.15.114:0/1750753788 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908011310 con 0x559da311cf80 2024-05-27T07:53:17.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.569+0000 7f19037fe700 1 -- 172.21.15.114:0/1750753788 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1908020070 con 0x559da311cf80 2024-05-27T07:53:17.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.569+0000 7f19037fe700 1 -- 172.21.15.114:0/1750753788 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908010b30 con 0x559da311cf80 2024-05-27T07:53:17.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.570+0000 7f192ea5db80 1 -- 172.21.15.114:0/1750753788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 msgr2=0x559da3120a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:17.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.570+0000 7f192ea5db80 1 --2- 172.21.15.114:0/1750753788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da3120a40 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f1908004550 tx=0x7f1908011ad0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.570+0000 7f192ea5db80 1 -- 172.21.15.114:0/1750753788 shutdown_connections 2024-05-27T07:53:17.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.570+0000 7f192ea5db80 1 --2- 172.21.15.114:0/1750753788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da3122c50 0x559da3126c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.570+0000 7f192ea5db80 1 --2- 172.21.15.114:0/1750753788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da2f9ee40 0x559da311ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.570+0000 7f192ea5db80 1 --2- 172.21.15.114:0/1750753788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da3120a40 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.570+0000 7f192ea5db80 1 -- 172.21.15.114:0/1750753788 >> 172.21.15.114:0/1750753788 conn(0x559da2ff9af0 msgr2=0x559da3124c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:17.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.570+0000 7f192ea5db80 1 -- 172.21.15.114:0/1750753788 shutdown_connections 2024-05-27T07:53:17.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.570+0000 7f192ea5db80 1 -- 172.21.15.114:0/1750753788 wait complete. 2024-05-27T07:53:17.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.570+0000 7f192ea5db80 1 -- start start 2024-05-27T07:53:17.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.570+0000 7f192ea5db80 1 Processor -- start 2024-05-27T07:53:17.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f192ea5db80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da2f9ee40 0x559da312e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f192ea5db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da312eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f192ea5db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da3122c50 0x559da313d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f192ea5db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f19181aed00 con 0x559da311cf80 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f192ea5db80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f19181b0170 con 0x559da2f9ee40 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f192ea5db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f19181af3d0 con 0x559da3122c50 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f192ea5db80 10 client.0 _ll_register_callbacks cb 0x559da2fd6920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f191d5e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da312eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f191d5e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da312eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36990/0 (socket says 172.21.15.114:36990) 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f191d5e3700 1 -- 172.21.15.114:0/3242403273 learned_addr learned my addr 172.21.15.114:0/3242403273 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f191dde4700 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da2f9ee40 0x559da312e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f191e5e5700 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da3122c50 0x559da313d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f191d5e3700 1 -- 172.21.15.114:0/3242403273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da3122c50 msgr2=0x559da313d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f191d5e3700 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da3122c50 0x559da313d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f191d5e3700 1 -- 172.21.15.114:0/3242403273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da2f9ee40 msgr2=0x559da312e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f191d5e3700 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da2f9ee40 0x559da312e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f191d5e3700 1 -- 172.21.15.114:0/3242403273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559da313dc40 con 0x559da311cf80 2024-05-27T07:53:17.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.571+0000 7f191dde4700 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da2f9ee40 0x559da312e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:17.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f191d5e3700 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da312eee0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f1908009690 tx=0x7f1908010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd59117e8fc9fc9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:17.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f19017fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:17.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f19017fa700 1 -- 172.21.15.114:0/3242403273 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908005030 con 0x559da311cf80 2024-05-27T07:53:17.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f19017fa700 1 -- 172.21.15.114:0/3242403273 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9566+0+0 (secure 0 0 0) 0x7f19080051d0 con 0x559da311cf80 2024-05-27T07:53:17.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f192ea5db80 1 -- 172.21.15.114:0/3242403273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1908020070 con 0x559da311cf80 2024-05-27T07:53:17.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f19017fa700 1 -- 172.21.15.114:0/3242403273 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f1908036050 con 0x559da311cf80 2024-05-27T07:53:17.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f192ea5db80 10 client.18615 fetch_fsmap learned FSMap version 73 2024-05-27T07:53:17.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f192ea5db80 1 -- 172.21.15.114:0/3242403273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x559da31a0c60 con 0x559da311cf80 2024-05-27T07:53:17.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f19017fa700 1 -- 172.21.15.114:0/3242403273 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1908020070 con 0x559da311cf80 2024-05-27T07:53:17.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f19017fa700 1 -- 172.21.15.114:0/3242403273 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 73) ==== 5910+0+0 (secure 0 0 0) 0x7f1908003680 con 0x559da311cf80 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f192ea5db80 10 client.18615 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f192ea5db80 10 client.18615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f192ea5db80 20 client.18615 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f192ea5db80 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559da31a1290 0x559da31a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f192ea5db80 4 client.18615 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.572+0000 7f192ea5db80 1 -- 172.21.15.114:0/3242403273 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559da304dbe0 con 0x559da31a1290 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.573+0000 7f191dde4700 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559da31a1290 0x559da31a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.573+0000 7f19017fa700 10 client.18615 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.573+0000 7f191dde4700 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559da31a1290 0x559da31a36b0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20d3a4e5ca9084d server_cookie=585852f63bfd7ff9 in_seq=0 out_seq=0 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.573+0000 7f19017fa700 1 -- 172.21.15.114:0/3242403273 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559da304dbe0 con 0x559da31a1290 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.573+0000 7f19017fa700 10 client.18615 handle_command_reply: tid=0 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.573+0000 7f192ea5db80 1 client.18615 shutdown 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.573+0000 7f192ea5db80 1 -- 172.21.15.114:0/3242403273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 msgr2=0x559da312eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:17.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.573+0000 7f192ea5db80 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da312eee0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f1908009690 tx=0x7f1908010430 comp rx=0 tx=0).stop 2024-05-27T07:53:17.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f192ea5db80 1 -- 172.21.15.114:0/3242403273 shutdown_connections 2024-05-27T07:53:17.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f192ea5db80 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559da3122c50 0x559da313d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f192ea5db80 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559da31a1290 0x559da31a36b0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f192ea5db80 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559da2f9ee40 0x559da312e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f192ea5db80 1 --2- 172.21.15.114:0/3242403273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559da311cf80 0x559da312eee0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f19017fa700 0 client.18615 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:17.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f192ea5db80 1 -- 172.21.15.114:0/3242403273 >> 172.21.15.114:0/3242403273 conn(0x559da2ff9af0 msgr2=0x559da31213f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:17.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f192ea5db80 1 -- 172.21.15.114:0/3242403273 shutdown_connections 2024-05-27T07:53:17.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f192ea5db80 1 -- 172.21.15.114:0/3242403273 wait complete. 2024-05-27T07:53:17.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f192ea5db80 20 client.18615 trim_cache size 0 max 16384 2024-05-27T07:53:17.581 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:53:17.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f1920048700 1 -- 172.21.15.114:0/73568942 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f19040b29f0 msgr2=0x7f19040b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:17.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f1920048700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f19040b29f0 0x7f19040b4f00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x559da311dbb0 tx=0x7f19100099e0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f1920048700 1 -- 172.21.15.114:0/73568942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1918103530 msgr2=0x7f191819f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:17.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f1920048700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1918103530 0x7f191819f890 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f191400e9f0 tx=0x7f191400ed00 comp rx=0 tx=0).stop 2024-05-27T07:53:17.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f1920048700 1 -- 172.21.15.114:0/73568942 shutdown_connections 2024-05-27T07:53:17.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.574+0000 7f1920048700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1918108760 0x7f1918199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.575+0000 7f1920048700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f19040b29f0 0x7f19040b4f00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.575+0000 7f1920048700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1918103530 0x7f191819f890 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.575+0000 7f1920048700 1 --2- 172.21.15.114:0/73568942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1918107e10 0x7f191819fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:17.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.575+0000 7f1920048700 1 -- 172.21.15.114:0/73568942 >> 172.21.15.114:0/73568942 conn(0x7f1918078e40 msgr2=0x7f1918106df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:17.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.575+0000 7f1920048700 1 -- 172.21.15.114:0/73568942 shutdown_connections 2024-05-27T07:53:17.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:17.575+0000 7f1920048700 1 -- 172.21.15.114:0/73568942 wait complete. 2024-05-27T07:53:17.747 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:17 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:53:17.747 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:17 smithi186 systemd[1]: Stopped Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:53:17.747 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:17 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.002s CPU time. 2024-05-27T07:53:17.747 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:17 smithi186 systemd[1]: Starting Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:53:17.747 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:17 smithi186 podman[75924]: 2024-05-27 07:53:17.618819408 +0000 UTC m=+0.018093634 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:53:17.748 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:17 smithi186 podman[75924]: 2024-05-27 07:53:17.746849745 +0000 UTC m=+0.146124006 container create b690abb7ad1442d6273cb9760c6fb252755f5012b8a17ddbbfb905802b16e838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-27T07:53:18.037 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:17 smithi186 podman[75924]: 2024-05-27 07:53:17.978389415 +0000 UTC m=+0.377663623 container init b690abb7ad1442d6273cb9760c6fb252755f5012b8a17ddbbfb905802b16e838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:53:18.037 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:17 smithi186 podman[75924]: 2024-05-27 07:53:17.981792188 +0000 UTC m=+0.381066401 container start b690abb7ad1442d6273cb9760c6fb252755f5012b8a17ddbbfb905802b16e838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-27T07:53:18.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1289523 9.33 MB/sec execute 570 sec latency 30.926 ms 2024-05-27T07:53:18.367 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-mds[75943]: -- 172.21.15.186:0/4142027724 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5566b03d5a40 con 0x5566b041da80 2024-05-27T07:53:18.367 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: starting mds.l at 2024-05-27T07:53:18.367 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 bash[75924]: b690abb7ad1442d6273cb9760c6fb252755f5012b8a17ddbbfb905802b16e838 2024-05-27T07:53:18.367 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 systemd[1]: Started Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:53:18.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:18 smithi114 ceph-mon[32038]: pgmap v454: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 243 op/s 2024-05-27T07:53:18.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:18 smithi114 ceph-mon[32038]: daemon mds.l restarted 2024-05-27T07:53:18.620 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:18 smithi186 ceph-mon[36647]: pgmap v454: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 243 op/s 2024-05-27T07:53:18.620 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:18 smithi186 ceph-mon[36647]: daemon mds.l restarted 2024-05-27T07:53:18.621 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff4a9952700 time 2024-05-27T07:53:18.552184+0000 2024-05-27T07:53:18.621 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:53:18.621 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:18.621 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff4bc197ba6] 2024-05-27T07:53:18.621 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4bc197dd1] 2024-05-27T07:53:18.621 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5566adc8223d] 2024-05-27T07:53:18.621 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5566adc6d971] 2024-05-27T07:53:18.621 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 5: (SnapRealm::check_cache() const+0x13e) [0x5566adc6bfa0] 2024-05-27T07:53:18.622 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5566adc6c651] 2024-05-27T07:53:18.622 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5566adc01347] 2024-05-27T07:53:18.622 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5566ad8fd87e] 2024-05-27T07:53:18.622 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5566ad902f2f] 2024-05-27T07:53:18.622 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5566ad90d969] 2024-05-27T07:53:18.622 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 11: (MDLog::_replay_thread()+0x1f38) [0x5566adca648c] 2024-05-27T07:53:18.622 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5566adcaf9cb] 2024-05-27T07:53:18.622 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 13: (Thread::entry_wrapper()+0x43) [0x7ff4bc16672d] 2024-05-27T07:53:18.622 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff4bc166749] 2024-05-27T07:53:18.622 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff4bad0a1ca] 2024-05-27T07:53:18.622 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 16: clone() 2024-05-27T07:53:18.622 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.553+0000 7ff4a9952700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff4a9952700 time 2024-05-27T07:53:18.552184+0000 2024-05-27T07:53:18.623 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:53:18.623 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.623 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:18.623 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff4bc197ba6] 2024-05-27T07:53:18.623 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4bc197dd1] 2024-05-27T07:53:18.623 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5566adc8223d] 2024-05-27T07:53:18.623 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5566adc6d971] 2024-05-27T07:53:18.623 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 5: (SnapRealm::check_cache() const+0x13e) [0x5566adc6bfa0] 2024-05-27T07:53:18.624 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5566adc6c651] 2024-05-27T07:53:18.624 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5566adc01347] 2024-05-27T07:53:18.624 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5566ad8fd87e] 2024-05-27T07:53:18.624 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5566ad902f2f] 2024-05-27T07:53:18.624 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5566ad90d969] 2024-05-27T07:53:18.624 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 11: (MDLog::_replay_thread()+0x1f38) [0x5566adca648c] 2024-05-27T07:53:18.624 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5566adcaf9cb] 2024-05-27T07:53:18.624 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 13: (Thread::entry_wrapper()+0x43) [0x7ff4bc16672d] 2024-05-27T07:53:18.624 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff4bc166749] 2024-05-27T07:53:18.624 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff4bad0a1ca] 2024-05-27T07:53:18.624 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 16: clone() 2024-05-27T07:53:18.624 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.625 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: *** Caught signal (Aborted) ** 2024-05-27T07:53:18.625 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: in thread 7ff4a9952700 thread_name:md_log_replay 2024-05-27T07:53:18.625 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:18.625 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff4bad14d20] 2024-05-27T07:53:18.625 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2: gsignal() 2024-05-27T07:53:18.625 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 3: abort() 2024-05-27T07:53:18.625 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff4bc197ced] 2024-05-27T07:53:18.625 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4bc197dd1] 2024-05-27T07:53:18.625 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5566adc8223d] 2024-05-27T07:53:18.625 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5566adc6d971] 2024-05-27T07:53:18.625 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 8: (SnapRealm::check_cache() const+0x13e) [0x5566adc6bfa0] 2024-05-27T07:53:18.625 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5566adc6c651] 2024-05-27T07:53:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5566adc01347] 2024-05-27T07:53:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5566ad8fd87e] 2024-05-27T07:53:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5566ad902f2f] 2024-05-27T07:53:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5566ad90d969] 2024-05-27T07:53:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 14: (MDLog::_replay_thread()+0x1f38) [0x5566adca648c] 2024-05-27T07:53:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5566adcaf9cb] 2024-05-27T07:53:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 16: (Thread::entry_wrapper()+0x43) [0x7ff4bc16672d] 2024-05-27T07:53:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff4bc166749] 2024-05-27T07:53:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff4bad0a1ca] 2024-05-27T07:53:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 19: clone() 2024-05-27T07:53:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.556+0000 7ff4a9952700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:53:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: in thread 7ff4a9952700 thread_name:md_log_replay 2024-05-27T07:53:18.627 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.627 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:18.627 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff4bad14d20] 2024-05-27T07:53:18.627 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2: gsignal() 2024-05-27T07:53:18.627 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 3: abort() 2024-05-27T07:53:18.627 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff4bc197ced] 2024-05-27T07:53:18.627 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4bc197dd1] 2024-05-27T07:53:18.627 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5566adc8223d] 2024-05-27T07:53:18.627 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5566adc6d971] 2024-05-27T07:53:18.628 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 8: (SnapRealm::check_cache() const+0x13e) [0x5566adc6bfa0] 2024-05-27T07:53:18.628 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5566adc6c651] 2024-05-27T07:53:18.628 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5566adc01347] 2024-05-27T07:53:18.629 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5566ad8fd87e] 2024-05-27T07:53:18.629 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5566ad902f2f] 2024-05-27T07:53:18.629 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5566ad90d969] 2024-05-27T07:53:18.629 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 14: (MDLog::_replay_thread()+0x1f38) [0x5566adca648c] 2024-05-27T07:53:18.629 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5566adcaf9cb] 2024-05-27T07:53:18.629 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 16: (Thread::entry_wrapper()+0x43) [0x7ff4bc16672d] 2024-05-27T07:53:18.630 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff4bc166749] 2024-05-27T07:53:18.630 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff4bad0a1ca] 2024-05-27T07:53:18.630 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 19: clone() 2024-05-27T07:53:18.630 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:53:18.630 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.630 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: -1> 2024-05-27T07:53:18.553+0000 7ff4a9952700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff4a9952700 time 2024-05-27T07:53:18.552184+0000 2024-05-27T07:53:18.630 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:53:18.630 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.630 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:18.630 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff4bc197ba6] 2024-05-27T07:53:18.630 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4bc197dd1] 2024-05-27T07:53:18.631 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5566adc8223d] 2024-05-27T07:53:18.631 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5566adc6d971] 2024-05-27T07:53:18.631 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 5: (SnapRealm::check_cache() const+0x13e) [0x5566adc6bfa0] 2024-05-27T07:53:18.631 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5566adc6c651] 2024-05-27T07:53:18.631 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5566adc01347] 2024-05-27T07:53:18.631 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5566ad8fd87e] 2024-05-27T07:53:18.631 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5566ad902f2f] 2024-05-27T07:53:18.631 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5566ad90d969] 2024-05-27T07:53:18.631 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 11: (MDLog::_replay_thread()+0x1f38) [0x5566adca648c] 2024-05-27T07:53:18.631 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5566adcaf9cb] 2024-05-27T07:53:18.631 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 13: (Thread::entry_wrapper()+0x43) [0x7ff4bc16672d] 2024-05-27T07:53:18.631 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff4bc166749] 2024-05-27T07:53:18.632 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff4bad0a1ca] 2024-05-27T07:53:18.632 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 16: clone() 2024-05-27T07:53:18.632 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.632 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 0> 2024-05-27T07:53:18.556+0000 7ff4a9952700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:53:18.632 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: in thread 7ff4a9952700 thread_name:md_log_replay 2024-05-27T07:53:18.632 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.632 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:18.632 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff4bad14d20] 2024-05-27T07:53:18.632 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2: gsignal() 2024-05-27T07:53:18.632 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 3: abort() 2024-05-27T07:53:18.632 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff4bc197ced] 2024-05-27T07:53:18.632 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4bc197dd1] 2024-05-27T07:53:18.632 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5566adc8223d] 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5566adc6d971] 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 8: (SnapRealm::check_cache() const+0x13e) [0x5566adc6bfa0] 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5566adc6c651] 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5566adc01347] 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5566ad8fd87e] 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5566ad902f2f] 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5566ad90d969] 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 14: (MDLog::_replay_thread()+0x1f38) [0x5566adca648c] 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5566adcaf9cb] 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 16: (Thread::entry_wrapper()+0x43) [0x7ff4bc16672d] 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff4bc166749] 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff4bad0a1ca] 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 19: clone() 2024-05-27T07:53:18.633 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:53:18.634 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.634 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: -1> 2024-05-27T07:53:18.553+0000 7ff4a9952700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff4a9952700 time 2024-05-27T07:53:18.552184+0000 2024-05-27T07:53:18.634 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:53:18.634 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.634 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:18.634 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff4bc197ba6] 2024-05-27T07:53:18.634 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4bc197dd1] 2024-05-27T07:53:18.634 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5566adc8223d] 2024-05-27T07:53:18.634 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5566adc6d971] 2024-05-27T07:53:18.634 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 5: (SnapRealm::check_cache() const+0x13e) [0x5566adc6bfa0] 2024-05-27T07:53:18.634 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5566adc6c651] 2024-05-27T07:53:18.635 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5566adc01347] 2024-05-27T07:53:18.635 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5566ad8fd87e] 2024-05-27T07:53:18.635 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5566ad902f2f] 2024-05-27T07:53:18.635 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5566ad90d969] 2024-05-27T07:53:18.635 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 11: (MDLog::_replay_thread()+0x1f38) [0x5566adca648c] 2024-05-27T07:53:18.635 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5566adcaf9cb] 2024-05-27T07:53:18.635 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 13: (Thread::entry_wrapper()+0x43) [0x7ff4bc16672d] 2024-05-27T07:53:18.635 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff4bc166749] 2024-05-27T07:53:18.635 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff4bad0a1ca] 2024-05-27T07:53:18.635 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 16: clone() 2024-05-27T07:53:18.635 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.635 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 0> 2024-05-27T07:53:18.556+0000 7ff4a9952700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:53:18.635 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: in thread 7ff4a9952700 thread_name:md_log_replay 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff4bad14d20] 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2: gsignal() 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 3: abort() 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff4bc197ced] 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4bc197dd1] 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5566adc8223d] 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5566adc6d971] 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 8: (SnapRealm::check_cache() const+0x13e) [0x5566adc6bfa0] 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5566adc6c651] 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5566adc01347] 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5566ad8fd87e] 2024-05-27T07:53:18.636 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5566ad902f2f] 2024-05-27T07:53:18.637 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5566ad90d969] 2024-05-27T07:53:18.637 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 14: (MDLog::_replay_thread()+0x1f38) [0x5566adca648c] 2024-05-27T07:53:18.637 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5566adcaf9cb] 2024-05-27T07:53:18.637 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 16: (Thread::entry_wrapper()+0x43) [0x7ff4bc16672d] 2024-05-27T07:53:18.637 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff4bc166749] 2024-05-27T07:53:18.637 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff4bad0a1ca] 2024-05-27T07:53:18.637 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 19: clone() 2024-05-27T07:53:18.637 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:53:18.637 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l[75939]: 2024-05-27T07:53:18.637 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:18 smithi186 podman[75974]: 2024-05-27 07:53:18.620740259 +0000 UTC m=+0.020678620 container died b690abb7ad1442d6273cb9760c6fb252755f5012b8a17ddbbfb905802b16e838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-27T07:53:18.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:18 smithi163 ceph-mon[36451]: pgmap v454: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 243 op/s 2024-05-27T07:53:18.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:18 smithi163 ceph-mon[36451]: daemon mds.l restarted 2024-05-27T07:53:19.268 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:53:19.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:53:19.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1291034 9.32 MB/sec execute 571 sec latency 62.707 ms 2024-05-27T07:53:19.547 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:19.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:19 smithi114 ceph-mon[32038]: osdmap e132: 12 total, 12 up, 12 in 2024-05-27T07:53:19.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:19 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6832/2682173878,v1:172.21.15.186:6833/2682173878] up:boot 2024-05-27T07:53:19.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:19 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:53:19.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:19 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:53:19.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:19 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:53:19.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:19 smithi114 ceph-mon[32038]: osdmap e133: 12 total, 12 up, 12 in 2024-05-27T07:53:19.564 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:19 smithi186 ceph-mon[36647]: osdmap e132: 12 total, 12 up, 12 in 2024-05-27T07:53:19.564 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:19 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6832/2682173878,v1:172.21.15.186:6833/2682173878] up:boot 2024-05-27T07:53:19.565 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:19 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:53:19.565 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:19 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:53:19.565 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:19 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:53:19.565 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:19 smithi186 ceph-mon[36647]: osdmap e133: 12 total, 12 up, 12 in 2024-05-27T07:53:19.565 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:19 smithi186 podman[75974]: 2024-05-27 07:53:19.285285959 +0000 UTC m=+0.685224310 container remove b690abb7ad1442d6273cb9760c6fb252755f5012b8a17ddbbfb905802b16e838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:53:19.565 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:19 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:53:19.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:19 smithi163 ceph-mon[36451]: osdmap e132: 12 total, 12 up, 12 in 2024-05-27T07:53:19.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:19 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6832/2682173878,v1:172.21.15.186:6833/2682173878] up:boot 2024-05-27T07:53:19.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:19 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 7 up:standby 2024-05-27T07:53:19.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:19 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:53:19.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:19 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 6 up:standby 2024-05-27T07:53:19.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:19 smithi163 ceph-mon[36451]: osdmap e133: 12 total, 12 up, 12 in 2024-05-27T07:53:19.979 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:19 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:53:20.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1292796 9.33 MB/sec execute 572 sec latency 61.331 ms 2024-05-27T07:53:20.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:20 smithi114 ceph-mon[32038]: pgmap v456: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 2.0 MiB/s wr, 225 op/s 2024-05-27T07:53:20.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:20 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:53:20.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:20 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:53:20.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:20 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:53:20.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:20 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:53:20.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:20 smithi163 ceph-mon[36451]: pgmap v456: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 2.0 MiB/s wr, 225 op/s 2024-05-27T07:53:20.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:20 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:53:20.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:20 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:53:20.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:20 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:53:20.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:20 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:53:20.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:20 smithi186 ceph-mon[36647]: pgmap v456: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 2.0 MiB/s wr, 225 op/s 2024-05-27T07:53:20.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:20 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:53:20.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:20 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:53:20.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:20 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:53:20.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:20 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:53:21.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1293841 9.32 MB/sec execute 573 sec latency 45.963 ms 2024-05-27T07:53:21.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.370+0000 7f9397806700 1 -- 172.21.15.114:0/97372677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9390108730 msgr2=0x7f9390110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:21.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.370+0000 7f9397806700 1 --2- 172.21.15.114:0/97372677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9390108730 0x7f9390110090 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f938800b0d0 tx=0x7f938800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.371+0000 7f9397806700 1 -- 172.21.15.114:0/97372677 shutdown_connections 2024-05-27T07:53:21.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.371+0000 7f9397806700 1 --2- 172.21.15.114:0/97372677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9390108730 0x7f9390110090 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.371+0000 7f9397806700 1 --2- 172.21.15.114:0/97372677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93901034b0 0x7f93901038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.371+0000 7f9397806700 1 --2- 172.21.15.114:0/97372677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9390107de0 0x7f93901081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.371+0000 7f9397806700 1 -- 172.21.15.114:0/97372677 >> 172.21.15.114:0/97372677 conn(0x7f93900fedb0 msgr2=0x7f93901011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:21.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.371+0000 7f9397806700 1 -- 172.21.15.114:0/97372677 shutdown_connections 2024-05-27T07:53:21.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.371+0000 7f9397806700 1 -- 172.21.15.114:0/97372677 wait complete. 2024-05-27T07:53:21.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.371+0000 7f9397806700 1 Processor -- start 2024-05-27T07:53:21.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.371+0000 7f9397806700 1 -- start start 2024-05-27T07:53:21.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9397806700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93901034b0 0x7f939019bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9397806700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9390107de0 0x7f939019c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9397806700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9390108730 0x7f939019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9397806700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9390113360 con 0x7f9390108730 2024-05-27T07:53:21.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9397806700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f93901131e0 con 0x7f93901034b0 2024-05-27T07:53:21.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9397806700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f93901134e0 con 0x7f9390107de0 2024-05-27T07:53:21.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9395da3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9390108730 0x7f939019c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9394da1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9390107de0 0x7f939019c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9395da3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9390108730 0x7f939019c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37012/0 (socket says 172.21.15.114:37012) 2024-05-27T07:53:21.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9394da1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9390107de0 0x7f939019c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:42880/0 (socket says 172.21.15.114:42880) 2024-05-27T07:53:21.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9395da3700 1 -- 172.21.15.114:0/953366721 learned_addr learned my addr 172.21.15.114:0/953366721 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:21.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f93955a2700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93901034b0 0x7f939019bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9395da3700 1 -- 172.21.15.114:0/953366721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9390107de0 msgr2=0x7f939019c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:21.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9395da3700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9390107de0 0x7f939019c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9395da3700 1 -- 172.21.15.114:0/953366721 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93901034b0 msgr2=0x7f939019bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:21.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9395da3700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93901034b0 0x7f939019bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9395da3700 1 -- 172.21.15.114:0/953366721 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93901a6ff0 con 0x7f9390108730 2024-05-27T07:53:21.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9394da1700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9390107de0 0x7f939019c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:21.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.372+0000 7f9395da3700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9390108730 0x7f939019c740 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f9388009850 tx=0x7f9388007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:21.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.373+0000 7f93867fc700 1 -- 172.21.15.114:0/953366721 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938801f070 con 0x7f9390108730 2024-05-27T07:53:21.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.373+0000 7f9397806700 1 -- 172.21.15.114:0/953366721 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93901a7280 con 0x7f9390108730 2024-05-27T07:53:21.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.373+0000 7f93867fc700 1 -- 172.21.15.114:0/953366721 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9388004c50 con 0x7f9390108730 2024-05-27T07:53:21.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.373+0000 7f93867fc700 1 -- 172.21.15.114:0/953366721 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938801b690 con 0x7f9390108730 2024-05-27T07:53:21.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.373+0000 7f9397806700 1 -- 172.21.15.114:0/953366721 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93901a7730 con 0x7f9390108730 2024-05-27T07:53:21.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.376+0000 7f93867fc700 1 -- 172.21.15.114:0/953366721 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f9388022070 con 0x7f9390108730 2024-05-27T07:53:21.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.377+0000 7f93867fc700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f937c0b29f0 0x7f937c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.377+0000 7f93a621bb80 1 Processor -- start 2024-05-27T07:53:21.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.377+0000 7f93a621bb80 1 -- start start 2024-05-27T07:53:21.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.377+0000 7f93955a2700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f937c0b29f0 0x7f937c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.377+0000 7f93867fc700 1 -- 172.21.15.114:0/953366721 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f938800f020 con 0x7f9390108730 2024-05-27T07:53:21.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.377+0000 7f93a621bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 0x55be63456790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f93a621bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63456cd0 0x55be63450ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f93a621bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63451000 0x55be63451410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f93a621bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55be6337fd10 con 0x55be632d2ec0 2024-05-27T07:53:21.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f93a621bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55be6336f8b0 con 0x55be63456cd0 2024-05-27T07:53:21.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f93a621bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55be633738f0 con 0x55be63451000 2024-05-27T07:53:21.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f9395da3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63451000 0x55be63451410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f93955a2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 0x55be63456790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f93955a2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 0x55be63456790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37020/0 (socket says 172.21.15.114:37020) 2024-05-27T07:53:21.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f9395da3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63451000 0x55be63451410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:42888/0 (socket says 172.21.15.114:42888) 2024-05-27T07:53:21.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f93955a2700 1 -- 172.21.15.114:0/1119949482 learned_addr learned my addr 172.21.15.114:0/1119949482 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:21.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f9394da1700 1 --2- 172.21.15.114:0/1119949482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63456cd0 0x55be63450ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f93955a2700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f937c0b29f0 0x7f937c0b4f00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f938c009cb0 tx=0x7f938c011720 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:21.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f9395da3700 1 -- 172.21.15.114:0/1119949482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63456cd0 msgr2=0x55be63450ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:21.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f9395da3700 1 --2- 172.21.15.114:0/1119949482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63456cd0 0x55be63450ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f9395da3700 1 -- 172.21.15.114:0/1119949482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 msgr2=0x55be63456790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:21.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f9395da3700 1 --2- 172.21.15.114:0/1119949482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 0x55be63456790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f9395da3700 1 -- 172.21.15.114:0/1119949482 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be632bf950 con 0x55be63451000 2024-05-27T07:53:21.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f93955a2700 1 --2- 172.21.15.114:0/1119949482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 0x55be63456790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:21.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.378+0000 7f9395da3700 1 --2- 172.21.15.114:0/1119949482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63451000 0x55be63451410 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f93880e0e60 tx=0x7f93880aa1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1ef7f401d5bdfeb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:21.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.379+0000 7f937b7fe700 1 -- 172.21.15.114:0/1119949482 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93880b31a0 con 0x55be63451000 2024-05-27T07:53:21.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.379+0000 7f937b7fe700 1 -- 172.21.15.114:0/1119949482 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9388004e00 con 0x55be63451000 2024-05-27T07:53:21.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.379+0000 7f937b7fe700 1 -- 172.21.15.114:0/1119949482 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93880aaa80 con 0x55be63451000 2024-05-27T07:53:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.379+0000 7f93a621bb80 1 -- 172.21.15.114:0/1119949482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63451000 msgr2=0x55be63451410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.379+0000 7f93a621bb80 1 --2- 172.21.15.114:0/1119949482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63451000 0x55be63451410 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f93880e0e60 tx=0x7f93880aa1f0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.380+0000 7f93a621bb80 1 -- 172.21.15.114:0/1119949482 shutdown_connections 2024-05-27T07:53:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.380+0000 7f93a621bb80 1 --2- 172.21.15.114:0/1119949482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63451000 0x55be63451410 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.380+0000 7f93a621bb80 1 --2- 172.21.15.114:0/1119949482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 0x55be63456790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.380+0000 7f93a621bb80 1 --2- 172.21.15.114:0/1119949482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63456cd0 0x55be63450ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.380+0000 7f93a621bb80 1 -- 172.21.15.114:0/1119949482 >> 172.21.15.114:0/1119949482 conn(0x55be6332db70 msgr2=0x55be6344b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.380+0000 7f93a621bb80 1 -- 172.21.15.114:0/1119949482 shutdown_connections 2024-05-27T07:53:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.380+0000 7f93a621bb80 1 -- 172.21.15.114:0/1119949482 wait complete. 2024-05-27T07:53:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.380+0000 7f93a621bb80 1 -- start start 2024-05-27T07:53:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.380+0000 7f93a621bb80 1 Processor -- start 2024-05-27T07:53:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f93a621bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be632d2ec0 0x55be6345a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f93a621bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63451000 0x55be6345a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f93a621bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be63456cd0 0x55be6345ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f93a621bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55be63276020 con 0x55be63456cd0 2024-05-27T07:53:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f93a621bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55be63311840 con 0x55be63451000 2024-05-27T07:53:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f93a621bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55be633738f0 con 0x55be632d2ec0 2024-05-27T07:53:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f93a621bb80 10 client.0 _ll_register_callbacks cb 0x55be6330a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f93955a2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be632d2ec0 0x55be6345a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f9395da3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be63456cd0 0x55be6345ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f93955a2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be632d2ec0 0x55be6345a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:42894/0 (socket says 172.21.15.114:42894) 2024-05-27T07:53:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f93955a2700 1 -- 172.21.15.114:0/1931733077 learned_addr learned my addr 172.21.15.114:0/1931733077 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f9394da1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63451000 0x55be6345a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f9394da1700 1 -- 172.21.15.114:0/1931733077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be632d2ec0 msgr2=0x55be6345a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:21.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f9394da1700 1 --2- 172.21.15.114:0/1931733077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be632d2ec0 0x55be6345a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f9394da1700 1 -- 172.21.15.114:0/1931733077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be63456cd0 msgr2=0x55be6345ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:21.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f9394da1700 1 --2- 172.21.15.114:0/1931733077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be63456cd0 0x55be6345ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f9394da1700 1 -- 172.21.15.114:0/1931733077 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be6330e710 con 0x55be63451000 2024-05-27T07:53:21.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f93955a2700 1 --2- 172.21.15.114:0/1931733077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be632d2ec0 0x55be6345a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:21.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f9395da3700 1 --2- 172.21.15.114:0/1931733077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be63456cd0 0x55be6345ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:21.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f9394da1700 1 --2- 172.21.15.114:0/1931733077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63451000 0x55be6345a930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f93901066b0 tx=0x7f9380011ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=198406dfefaef893 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:21.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.381+0000 7f9378ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:53:21.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.382+0000 7f9378ff9700 1 -- 172.21.15.114:0/1931733077 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9380004020 con 0x55be63451000 2024-05-27T07:53:21.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.382+0000 7f9378ff9700 1 -- 172.21.15.114:0/1931733077 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f93800041c0 con 0x55be63451000 2024-05-27T07:53:21.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.382+0000 7f93a621bb80 1 -- 172.21.15.114:0/1931733077 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55be6341ca70 con 0x55be63451000 2024-05-27T07:53:21.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.388+0000 7f9378ff9700 1 -- 172.21.15.114:0/1931733077 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f93800043c0 con 0x55be63451000 2024-05-27T07:53:21.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.388+0000 7f93a621bb80 10 client.28037 fetch_fsmap learned FSMap version 75 2024-05-27T07:53:21.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.388+0000 7f93a621bb80 1 -- 172.21.15.114:0/1931733077 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x55be63467eb0 con 0x55be63451000 2024-05-27T07:53:21.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.388+0000 7f9378ff9700 1 -- 172.21.15.114:0/1931733077 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55be6341ca70 con 0x55be63451000 2024-05-27T07:53:21.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.389+0000 7f9378ff9700 1 -- 172.21.15.114:0/1931733077 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 75) ==== 5910+0+0 (secure 0 0 0) 0x7f9380005420 con 0x55be63451000 2024-05-27T07:53:21.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.389+0000 7f93a621bb80 10 client.28037 fetch_fsmap finished waiting for FSMap version 75 2024-05-27T07:53:21.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.389+0000 7f93a621bb80 10 client.28037 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:21.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.389+0000 7f93a621bb80 20 client.28037 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:21.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.389+0000 7f93a621bb80 1 --2- 172.21.15.114:0/1931733077 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be6346a800 0x55be6346cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.389+0000 7f93a621bb80 4 client.28037 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:21.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.389+0000 7f93a621bb80 1 -- 172.21.15.114:0/1931733077 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be63381c60 con 0x55be6346a800 2024-05-27T07:53:21.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.389+0000 7f93955a2700 1 --2- 172.21.15.114:0/1931733077 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be6346a800 0x55be6346cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.389+0000 7f9378ff9700 10 client.28037 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:21.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.389+0000 7f93955a2700 1 --2- 172.21.15.114:0/1931733077 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be6346a800 0x55be6346cc50 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85010c8ffbbda599 server_cookie=bd01ce6c8f2a4193 in_seq=0 out_seq=0 2024-05-27T07:53:21.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.390+0000 7f9378ff9700 1 -- 172.21.15.114:0/1931733077 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55be63381c60 con 0x55be6346a800 2024-05-27T07:53:21.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.390+0000 7f9378ff9700 10 client.28037 handle_command_reply: tid=0 2024-05-27T07:53:21.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.390+0000 7f93a621bb80 1 client.28037 shutdown 2024-05-27T07:53:21.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.390+0000 7f93a621bb80 1 -- 172.21.15.114:0/1931733077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63451000 msgr2=0x55be6345a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:21.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.390+0000 7f93a621bb80 1 --2- 172.21.15.114:0/1931733077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63451000 0x55be6345a930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f93901066b0 tx=0x7f9380011ae0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.390+0000 7f93a621bb80 1 -- 172.21.15.114:0/1931733077 shutdown_connections 2024-05-27T07:53:21.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.391+0000 7f93a621bb80 1 --2- 172.21.15.114:0/1931733077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be63456cd0 0x55be6345ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.391+0000 7f93a621bb80 1 --2- 172.21.15.114:0/1931733077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be632d2ec0 0x55be6345a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.391+0000 7f93a621bb80 1 --2- 172.21.15.114:0/1931733077 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be6346a800 0x55be6346cc50 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.391+0000 7f93a621bb80 1 --2- 172.21.15.114:0/1931733077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63451000 0x55be6345a930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.391+0000 7f9378ff9700 0 client.28037 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:21.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.391+0000 7f93a621bb80 1 -- 172.21.15.114:0/1931733077 >> 172.21.15.114:0/1931733077 conn(0x55be6332db70 msgr2=0x55be6344fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:21.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.391+0000 7f93a621bb80 1 -- 172.21.15.114:0/1931733077 shutdown_connections 2024-05-27T07:53:21.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.391+0000 7f93a621bb80 1 -- 172.21.15.114:0/1931733077 wait complete. 2024-05-27T07:53:21.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.391+0000 7f93a621bb80 20 client.28037 trim_cache size 0 max 16384 2024-05-27T07:53:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.412+0000 7f93a621bb80 1 Processor -- start 2024-05-27T07:53:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93a621bb80 1 -- start start 2024-05-27T07:53:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93a621bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 0x55be634664a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93a621bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63451000 0x55be63457ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93a621bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63456cd0 0x55be63457ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93a621bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f93901b1fa0 con 0x55be632d2ec0 2024-05-27T07:53:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93a621bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f93901b0710 con 0x55be63451000 2024-05-27T07:53:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93a621bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f93901ae580 con 0x55be63456cd0 2024-05-27T07:53:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93955a2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 0x55be634664a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93955a2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 0x55be634664a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37034/0 (socket says 172.21.15.114:37034) 2024-05-27T07:53:21.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93955a2700 1 -- 172.21.15.114:0/3810864284 learned_addr learned my addr 172.21.15.114:0/3810864284 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:21.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f9395da3700 1 --2- 172.21.15.114:0/3810864284 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63456cd0 0x55be63457ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f9394da1700 1 --2- 172.21.15.114:0/3810864284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63451000 0x55be63457ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93955a2700 1 -- 172.21.15.114:0/3810864284 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63456cd0 msgr2=0x55be63457ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:21.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93955a2700 1 --2- 172.21.15.114:0/3810864284 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63456cd0 0x55be63457ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93955a2700 1 -- 172.21.15.114:0/3810864284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63451000 msgr2=0x55be63457ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:21.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93955a2700 1 --2- 172.21.15.114:0/3810864284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63451000 0x55be63457ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93955a2700 1 -- 172.21.15.114:0/3810864284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be6344b910 con 0x55be632d2ec0 2024-05-27T07:53:21.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f9395da3700 1 --2- 172.21.15.114:0/3810864284 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63456cd0 0x55be63457ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:53:21.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.413+0000 7f93955a2700 1 --2- 172.21.15.114:0/3810864284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 0x55be634664a0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f938c005c50 tx=0x7f938c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b7dc8e11031716 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:21.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.414+0000 7f937affd700 1 -- 172.21.15.114:0/3810864284 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c02e070 con 0x55be632d2ec0 2024-05-27T07:53:21.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.414+0000 7f937affd700 1 -- 172.21.15.114:0/3810864284 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f938c019050 con 0x55be632d2ec0 2024-05-27T07:53:21.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.414+0000 7f937affd700 1 -- 172.21.15.114:0/3810864284 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c017040 con 0x55be632d2ec0 2024-05-27T07:53:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.414+0000 7f93a621bb80 1 -- 172.21.15.114:0/3810864284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 msgr2=0x55be634664a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.414+0000 7f93a621bb80 1 --2- 172.21.15.114:0/3810864284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 0x55be634664a0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f938c005c50 tx=0x7f938c01d000 comp rx=0 tx=0).stop 2024-05-27T07:53:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.415+0000 7f93a621bb80 1 -- 172.21.15.114:0/3810864284 shutdown_connections 2024-05-27T07:53:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.415+0000 7f93a621bb80 1 --2- 172.21.15.114:0/3810864284 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63456cd0 0x55be63457ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.415+0000 7f93a621bb80 1 --2- 172.21.15.114:0/3810864284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be632d2ec0 0x55be634664a0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.415+0000 7f93a621bb80 1 --2- 172.21.15.114:0/3810864284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be63451000 0x55be63457ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.415+0000 7f93a621bb80 1 -- 172.21.15.114:0/3810864284 >> 172.21.15.114:0/3810864284 conn(0x55be6332db70 msgr2=0x55be634573a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.415+0000 7f93a621bb80 1 -- 172.21.15.114:0/3810864284 shutdown_connections 2024-05-27T07:53:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.415+0000 7f93a621bb80 1 -- 172.21.15.114:0/3810864284 wait complete. 2024-05-27T07:53:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.415+0000 7f93a621bb80 1 -- start start 2024-05-27T07:53:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.415+0000 7f93a621bb80 1 Processor -- start 2024-05-27T07:53:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f93a621bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be632d2ec0 0x55be63460610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f93a621bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63451000 0x55be63460b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f93a621bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be63456cd0 0x55be63461090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f93a621bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f93901aebd0 con 0x55be63456cd0 2024-05-27T07:53:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f93a621bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f93901b0040 con 0x55be632d2ec0 2024-05-27T07:53:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f93a621bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f93901af2a0 con 0x55be63451000 2024-05-27T07:53:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f93a621bb80 10 client.0 _ll_register_callbacks cb 0x55be6330a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f9394da1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63451000 0x55be63460b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f9395da3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be63456cd0 0x55be63461090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f9394da1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63451000 0x55be63460b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:42904/0 (socket says 172.21.15.114:42904) 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f9394da1700 1 -- 172.21.15.114:0/2332383591 learned_addr learned my addr 172.21.15.114:0/2332383591 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f93955a2700 1 --2- 172.21.15.114:0/2332383591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be632d2ec0 0x55be63460610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f9395da3700 1 -- 172.21.15.114:0/2332383591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63451000 msgr2=0x55be63460b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f9395da3700 1 --2- 172.21.15.114:0/2332383591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63451000 0x55be63460b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f9395da3700 1 -- 172.21.15.114:0/2332383591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be632d2ec0 msgr2=0x55be63460610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f9395da3700 1 --2- 172.21.15.114:0/2332383591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be632d2ec0 0x55be63460610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f9395da3700 1 -- 172.21.15.114:0/2332383591 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be634d44f0 con 0x55be63456cd0 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f9395da3700 1 --2- 172.21.15.114:0/2332383591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be63456cd0 0x55be63461090 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f9388009fd0 tx=0x7f9388009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba4857d3ddddab8d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.416+0000 7f9378ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f9378ff9700 1 -- 172.21.15.114:0/2332383591 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938806e070 con 0x55be63456cd0 2024-05-27T07:53:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f9378ff9700 1 -- 172.21.15.114:0/2332383591 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f93880b35d0 con 0x55be63456cd0 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f93a621bb80 1 -- 172.21.15.114:0/2332383591 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f938c019050 con 0x55be63456cd0 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f9378ff9700 1 -- 172.21.15.114:0/2332383591 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f93880b31a0 con 0x55be63456cd0 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f93a621bb80 10 client.18639 fetch_fsmap learned FSMap version 75 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f93a621bb80 1 -- 172.21.15.114:0/2332383591 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x55be634d5770 con 0x55be63456cd0 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f9378ff9700 1 -- 172.21.15.114:0/2332383591 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f938c019050 con 0x55be63456cd0 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f9378ff9700 1 -- 172.21.15.114:0/2332383591 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 75) ==== 5910+0+0 (secure 0 0 0) 0x7f9388011070 con 0x55be63456cd0 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f93a621bb80 10 client.18639 fetch_fsmap finished waiting for FSMap version 75 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f93a621bb80 10 client.18639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f93a621bb80 20 client.18639 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f93a621bb80 1 --2- 172.21.15.114:0/2332383591 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be634d5b00 0x55be634d7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f93a621bb80 4 client.18639 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f93a621bb80 1 -- 172.21.15.114:0/2332383591 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55be6337b440 con 0x55be634d5b00 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.417+0000 7f93955a2700 1 --2- 172.21.15.114:0/2332383591 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be634d5b00 0x55be634d7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.418+0000 7f9378ff9700 10 client.18639 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.418+0000 7f93955a2700 1 --2- 172.21.15.114:0/2332383591 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be634d5b00 0x55be634d7f20 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1826e738e3bcb28 server_cookie=6ed37c6bc1ceb37f in_seq=0 out_seq=0 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f9378ff9700 1 -- 172.21.15.114:0/2332383591 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55be6337b440 con 0x55be634d5b00 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f9378ff9700 10 client.18639 handle_command_reply: tid=0 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f93a621bb80 1 client.18639 shutdown 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f93a621bb80 1 -- 172.21.15.114:0/2332383591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be63456cd0 msgr2=0x55be63461090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f93a621bb80 1 --2- 172.21.15.114:0/2332383591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be63456cd0 0x55be63461090 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f9388009fd0 tx=0x7f9388009650 comp rx=0 tx=0).stop 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f93a621bb80 1 -- 172.21.15.114:0/2332383591 shutdown_connections 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f93a621bb80 1 --2- 172.21.15.114:0/2332383591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be63456cd0 0x55be63461090 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f93a621bb80 1 --2- 172.21.15.114:0/2332383591 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be634d5b00 0x55be634d7f20 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f93a621bb80 1 --2- 172.21.15.114:0/2332383591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be632d2ec0 0x55be63460610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f93a621bb80 1 --2- 172.21.15.114:0/2332383591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be63451000 0x55be63460b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f93a621bb80 1 -- 172.21.15.114:0/2332383591 >> 172.21.15.114:0/2332383591 conn(0x55be6332db70 msgr2=0x55be63454820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f9378ff9700 0 client.18639 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f93a621bb80 1 -- 172.21.15.114:0/2332383591 shutdown_connections 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f93a621bb80 1 -- 172.21.15.114:0/2332383591 wait complete. 2024-05-27T07:53:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.420+0000 7f93a621bb80 20 client.18639 trim_cache size 0 max 16384 2024-05-27T07:53:21.428 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:53:21.428 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:53:21.428 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "2f3831b3-73ee-468c-a3b0-6822d0b7cde2", 2024-05-27T07:53:21.428 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:53:21.428 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:53:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.421+0000 7f9397806700 1 -- 172.21.15.114:0/953366721 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f937c0b29f0 msgr2=0x7f937c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.421+0000 7f9397806700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f937c0b29f0 0x7f937c0b4f00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f938c009cb0 tx=0x7f938c011720 comp rx=0 tx=0).stop 2024-05-27T07:53:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.421+0000 7f9397806700 1 -- 172.21.15.114:0/953366721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9390108730 msgr2=0x7f939019c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.421+0000 7f9397806700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9390108730 0x7f939019c740 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f9388009850 tx=0x7f9388007a20 comp rx=0 tx=0).stop 2024-05-27T07:53:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.421+0000 7f9397806700 1 -- 172.21.15.114:0/953366721 shutdown_connections 2024-05-27T07:53:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.421+0000 7f9397806700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9390108730 0x7f939019c740 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.421+0000 7f9397806700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f937c0b29f0 0x7f937c0b4f00 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.421+0000 7f9397806700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93901034b0 0x7f939019bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.421+0000 7f9397806700 1 --2- 172.21.15.114:0/953366721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9390107de0 0x7f939019c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.421+0000 7f9397806700 1 -- 172.21.15.114:0/953366721 >> 172.21.15.114:0/953366721 conn(0x7f93900fedb0 msgr2=0x7f9390107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.421+0000 7f9397806700 1 -- 172.21.15.114:0/953366721 shutdown_connections 2024-05-27T07:53:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:21.421+0000 7f9397806700 1 -- 172.21.15.114:0/953366721 wait complete. 2024-05-27T07:53:21.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:21 smithi114 ceph-mon[32038]: pgmap v458: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.5 MiB/s wr, 300 op/s 2024-05-27T07:53:21.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:21 smithi114 ceph-mon[32038]: pgmap v459: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 352 op/s 2024-05-27T07:53:21.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:21 smithi163 ceph-mon[36451]: pgmap v458: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.5 MiB/s wr, 300 op/s 2024-05-27T07:53:21.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:21 smithi163 ceph-mon[36451]: pgmap v459: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 352 op/s 2024-05-27T07:53:21.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:21 smithi186 ceph-mon[36647]: pgmap v458: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.5 MiB/s wr, 300 op/s 2024-05-27T07:53:21.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:21 smithi186 ceph-mon[36647]: pgmap v459: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 352 op/s 2024-05-27T07:53:22.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:53:22.296 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1296229 9.33 MB/sec execute 574 sec latency 53.510 ms 2024-05-27T07:53:22.360 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:22.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:22 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:53:22.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:22 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:22.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:22 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:53:22.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:22 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:22.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:22 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:53:22.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:22 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:53:22.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:22 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:22.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:22 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:53:22.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:22 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:22.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:22 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:53:22.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:22 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:53:22.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:22 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:22.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:22 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:53:22.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:22 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:22.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:22 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:53:23.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1297231 9.32 MB/sec execute 575 sec latency 14.608 ms 2024-05-27T07:53:23.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:23 smithi114 ceph-mon[32038]: pgmap v460: 129 pgs: 129 active+clean; 76 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-05-27T07:53:23.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:23 smithi163 ceph-mon[36451]: pgmap v460: 129 pgs: 129 active+clean; 76 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-05-27T07:53:23.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:23 smithi186 ceph-mon[36647]: pgmap v460: 129 pgs: 129 active+clean; 76 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-05-27T07:53:24.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.096+0000 7f5d3255a700 1 -- 172.21.15.114:0/325511411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2c103990 msgr2=0x7f5d2c103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.096+0000 7f5d3255a700 1 --2- 172.21.15.114:0/325511411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2c103990 0x7f5d2c103da0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009990 tx=0x7f5d20009ca0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.096+0000 7f5d3255a700 1 -- 172.21.15.114:0/325511411 shutdown_connections 2024-05-27T07:53:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.096+0000 7f5d3255a700 1 --2- 172.21.15.114:0/325511411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c1042e0 0x7f5d2c110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.096+0000 7f5d3255a700 1 --2- 172.21.15.114:0/325511411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d2c1081f0 0x7f5d2c108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.096+0000 7f5d3255a700 1 --2- 172.21.15.114:0/325511411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2c103990 0x7f5d2c103da0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.096+0000 7f5d3255a700 1 -- 172.21.15.114:0/325511411 >> 172.21.15.114:0/325511411 conn(0x7f5d2c0fe4d0 msgr2=0x7f5d2c100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.096+0000 7f5d3255a700 1 -- 172.21.15.114:0/325511411 shutdown_connections 2024-05-27T07:53:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.096+0000 7f5d3255a700 1 -- 172.21.15.114:0/325511411 wait complete. 2024-05-27T07:53:24.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.097+0000 7f5d3255a700 1 Processor -- start 2024-05-27T07:53:24.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.097+0000 7f5d3255a700 1 -- start start 2024-05-27T07:53:24.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.098+0000 7f5d3255a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2c103990 0x7f5d2c19d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.098+0000 7f5d3255a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c1042e0 0x7f5d2c19d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.098+0000 7f5d3255a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d2c1081f0 0x7f5d2c1a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.098+0000 7f5d3255a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c1122c0 con 0x7f5d2c103990 2024-05-27T07:53:24.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.098+0000 7f5d3255a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c112140 con 0x7f5d2c1081f0 2024-05-27T07:53:24.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.098+0000 7f5d3255a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c112440 con 0x7f5d2c1042e0 2024-05-27T07:53:24.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.098+0000 7f5d2b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c1042e0 0x7f5d2c19d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.098+0000 7f5d2b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c1042e0 0x7f5d2c19d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:42932/0 (socket says 172.21.15.114:42932) 2024-05-27T07:53:24.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.098+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/17910822 learned_addr learned my addr 172.21.15.114:0/17910822 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:24.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.098+0000 7f5d30af7700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d2c1081f0 0x7f5d2c1a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.099+0000 7f5d2bfff700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2c103990 0x7f5d2c19d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.099+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/17910822 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d2c1081f0 msgr2=0x7f5d2c1a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.099+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d2c1081f0 0x7f5d2c1a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.099+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/17910822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2c103990 msgr2=0x7f5d2c19d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.099+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2c103990 0x7f5d2c19d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.099+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/17910822 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2c1a1f20 con 0x7f5d2c1042e0 2024-05-27T07:53:24.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.099+0000 7f5d30af7700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d2c1081f0 0x7f5d2c1a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:53:24.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.099+0000 7f5d2bfff700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2c103990 0x7f5d2c19d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:53:24.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.099+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c1042e0 0x7f5d2c19d5f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5d20000c00 tx=0x7f5d2000ddc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:24.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.100+0000 7f5d297fa700 1 -- 172.21.15.114:0/17910822 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20022070 con 0x7f5d2c1042e0 2024-05-27T07:53:24.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.100+0000 7f5d3255a700 1 -- 172.21.15.114:0/17910822 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d2c1a21b0 con 0x7f5d2c1042e0 2024-05-27T07:53:24.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.100+0000 7f5d3255a700 1 -- 172.21.15.114:0/17910822 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d2c1a9b30 con 0x7f5d2c1042e0 2024-05-27T07:53:24.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.100+0000 7f5d297fa700 1 -- 172.21.15.114:0/17910822 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d200054e0 con 0x7f5d2c1042e0 2024-05-27T07:53:24.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.100+0000 7f5d297fa700 1 -- 172.21.15.114:0/17910822 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2001a070 con 0x7f5d2c1042e0 2024-05-27T07:53:24.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.104+0000 7f5d297fa700 1 -- 172.21.15.114:0/17910822 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f5d20025070 con 0x7f5d2c1042e0 2024-05-27T07:53:24.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.104+0000 7f5d40f6fb80 1 Processor -- start 2024-05-27T07:53:24.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d40f6fb80 1 -- start start 2024-05-27T07:53:24.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d297fa700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5d140b2a70 0x7f5d140b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d40f6fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f4fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d40f6fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f50070 0x556cc6f49be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d30af7700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5d140b2a70 0x7f5d140b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d2bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f4fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d297fa700 1 -- 172.21.15.114:0/17910822 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f5d200e2a50 con 0x7f5d2c1042e0 2024-05-27T07:53:24.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d2bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f4fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:42940/0 (socket says 172.21.15.114:42940) 2024-05-27T07:53:24.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d2b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f50070 0x556cc6f49be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d40f6fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f4a120 0x556cc6f4a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d2bfff700 1 -- 172.21.15.114:0/3037198825 learned_addr learned my addr 172.21.15.114:0/3037198825 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:24.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3037198825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556cc6e77d70 con 0x556cc6f50070 2024-05-27T07:53:24.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3037198825 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556cc6e67910 con 0x556cc6f4a120 2024-05-27T07:53:24.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3037198825 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556cc6e6b950 con 0x556cc6dcae40 2024-05-27T07:53:24.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.105+0000 7f5d30af7700 1 --2- 172.21.15.114:0/3037198825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f4a120 0x556cc6f4a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.106+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/3037198825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 msgr2=0x556cc6f4fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.106+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/3037198825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f4fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.106+0000 7f5d30af7700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5d140b2a70 0x7f5d140b4f80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5d24007770 tx=0x7f5d24007b30 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:24.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.106+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/3037198825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f4a120 msgr2=0x556cc6f4a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.106+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/3037198825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f4a120 0x556cc6f4a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.106+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/3037198825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cc6e10fb0 con 0x556cc6f50070 2024-05-27T07:53:24.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.106+0000 7f5d2bfff700 1 --2- 172.21.15.114:0/3037198825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f4fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:24.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.106+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/3037198825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f50070 0x556cc6f49be0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f5d200dfda0 tx=0x7f5d200a59b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d7bfff9fa04bd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:24.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.106+0000 7f5d127fc700 1 -- 172.21.15.114:0/3037198825 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2001f260 con 0x556cc6f50070 2024-05-27T07:53:24.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.106+0000 7f5d127fc700 1 -- 172.21.15.114:0/3037198825 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d200095d0 con 0x556cc6f50070 2024-05-27T07:53:24.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.106+0000 7f5d127fc700 1 -- 172.21.15.114:0/3037198825 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2001f080 con 0x556cc6f50070 2024-05-27T07:53:24.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.107+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3037198825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f50070 msgr2=0x556cc6f49be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.107+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/3037198825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f50070 0x556cc6f49be0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f5d200dfda0 tx=0x7f5d200a59b0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.107+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3037198825 shutdown_connections 2024-05-27T07:53:24.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.107+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/3037198825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f4a120 0x556cc6f4a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.107+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/3037198825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f4fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.107+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/3037198825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f50070 0x556cc6f49be0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.107+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3037198825 >> 172.21.15.114:0/3037198825 conn(0x556cc6dcd770 msgr2=0x556cc6f44d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.107+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3037198825 shutdown_connections 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.107+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3037198825 wait complete. 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.108+0000 7f5d40f6fb80 1 -- start start 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.108+0000 7f5d40f6fb80 1 Processor -- start 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.108+0000 7f5d40f6fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f53950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.108+0000 7f5d40f6fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 0x556cc6f53e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.108+0000 7f5d40f6fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f543d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.108+0000 7f5d40f6fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556cc6d6dfa0 con 0x556cc6f4a120 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.108+0000 7f5d40f6fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556cc6dcfda0 con 0x556cc6f50070 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.108+0000 7f5d40f6fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556cc6f56920 con 0x556cc6dcae40 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d40f6fb80 10 client.0 _ll_register_callbacks cb 0x556cc6dfe5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d2b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 0x556cc6f53e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d30af7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d30af7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f543d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:45356/0 (socket says 172.21.15.114:45356) 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d2b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 0x556cc6f53e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37084/0 (socket says 172.21.15.114:37084) 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d30af7700 1 -- 172.21.15.114:0/2164652940 learned_addr learned my addr 172.21.15.114:0/2164652940 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d2bfff700 1 --2- 172.21.15.114:0/2164652940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f53950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/2164652940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 msgr2=0x556cc6f53950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/2164652940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f53950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/2164652940 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 msgr2=0x556cc6f543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/2164652940 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/2164652940 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cc6db78d0 con 0x556cc6f4a120 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d30af7700 1 --2- 172.21.15.114:0/2164652940 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/2164652940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 0x556cc6f53e90 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f5d200dfda0 tx=0x7f5d20029f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42da90e1ab5290db server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.109+0000 7f5d0b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d0b7fe700 1 -- 172.21.15.114:0/2164652940 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d200c3060 con 0x556cc6f4a120 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d0b7fe700 1 -- 172.21.15.114:0/2164652940 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f5d200a5c50 con 0x556cc6f4a120 2024-05-27T07:53:24.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/2164652940 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d200095d0 con 0x556cc6f4a120 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d0b7fe700 1 -- 172.21.15.114:0/2164652940 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f5d2001f080 con 0x556cc6f4a120 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d40f6fb80 10 client.18648 fetch_fsmap learned FSMap version 75 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/2164652940 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x556cc6e21760 con 0x556cc6f4a120 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d0b7fe700 1 -- 172.21.15.114:0/2164652940 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d200095d0 con 0x556cc6f4a120 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d0b7fe700 1 -- 172.21.15.114:0/2164652940 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 75) ==== 5910+0+0 (secure 0 0 0) 0x7f5d200b0110 con 0x556cc6f4a120 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d40f6fb80 10 client.18648 fetch_fsmap finished waiting for FSMap version 75 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d40f6fb80 10 client.18648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d40f6fb80 20 client.18648 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/2164652940 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556cc6f63720 0x556cc6f65b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d40f6fb80 4 client.18648 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.110+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/2164652940 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cc6e79cc0 con 0x556cc6f63720 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.111+0000 7f5d2bfff700 1 --2- 172.21.15.114:0/2164652940 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556cc6f63720 0x556cc6f65b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.111+0000 7f5d0b7fe700 10 client.18648 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.111+0000 7f5d2bfff700 1 --2- 172.21.15.114:0/2164652940 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556cc6f63720 0x556cc6f65b40 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30fd2bccb6990429 server_cookie=cf3e068ea54c35c2 in_seq=0 out_seq=0 2024-05-27T07:53:24.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d0b7fe700 1 -- 172.21.15.114:0/2164652940 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556cc6e79cc0 con 0x556cc6f63720 2024-05-27T07:53:24.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d0b7fe700 10 client.18648 handle_command_reply: tid=0 2024-05-27T07:53:24.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d40f6fb80 1 client.18648 shutdown 2024-05-27T07:53:24.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/2164652940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 msgr2=0x556cc6f53e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/2164652940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 0x556cc6f53e90 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f5d200dfda0 tx=0x7f5d20029f50 comp rx=0 tx=0).stop 2024-05-27T07:53:24.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/2164652940 shutdown_connections 2024-05-27T07:53:24.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/2164652940 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556cc6f63720 0x556cc6f65b40 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d0b7fe700 0 client.18648 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:24.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/2164652940 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/2164652940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f53950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/2164652940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 0x556cc6f53e90 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/2164652940 >> 172.21.15.114:0/2164652940 conn(0x556cc6dcd770 msgr2=0x556cc6f48bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:24.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/2164652940 shutdown_connections 2024-05-27T07:53:24.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/2164652940 wait complete. 2024-05-27T07:53:24.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.113+0000 7f5d40f6fb80 20 client.18648 trim_cache size 0 max 16384 2024-05-27T07:53:24.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.133+0000 7f5d40f6fb80 1 Processor -- start 2024-05-27T07:53:24.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d40f6fb80 1 -- start start 2024-05-27T07:53:24.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d40f6fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f49be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d40f6fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 0x556cc6f4eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d40f6fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f54210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d40f6fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5d18010740 con 0x556cc6f4a120 2024-05-27T07:53:24.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d40f6fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1800eeb0 con 0x556cc6f50070 2024-05-27T07:53:24.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d40f6fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1800cd20 con 0x556cc6dcae40 2024-05-27T07:53:24.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d2b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 0x556cc6f4eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d2b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 0x556cc6f4eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37098/0 (socket says 172.21.15.114:37098) 2024-05-27T07:53:24.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/1937695903 learned_addr learned my addr 172.21.15.114:0/1937695903 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:24.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d2bfff700 1 --2- 172.21.15.114:0/1937695903 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f49be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d30af7700 1 --2- 172.21.15.114:0/1937695903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f54210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/1937695903 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 msgr2=0x556cc6f49be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/1937695903 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f49be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/1937695903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 msgr2=0x556cc6f54210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/1937695903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f54210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d2b7fe700 1 -- 172.21.15.114:0/1937695903 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cc6f44df0 con 0x556cc6f4a120 2024-05-27T07:53:24.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.134+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/1937695903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 0x556cc6f4eed0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f5d200a6c30 tx=0x7f5d200a5970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d7bacd9cf52ec6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:24.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.135+0000 7f5d11ffb700 1 -- 172.21.15.114:0/1937695903 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d200c3030 con 0x556cc6f4a120 2024-05-27T07:53:24.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.135+0000 7f5d11ffb700 1 -- 172.21.15.114:0/1937695903 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d200095d0 con 0x556cc6f4a120 2024-05-27T07:53:24.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.135+0000 7f5d11ffb700 1 -- 172.21.15.114:0/1937695903 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d200b0110 con 0x556cc6f4a120 2024-05-27T07:53:24.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.135+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/1937695903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 msgr2=0x556cc6f4eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.135+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/1937695903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 0x556cc6f4eed0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f5d200a6c30 tx=0x7f5d200a5970 comp rx=0 tx=0).stop 2024-05-27T07:53:24.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.136+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/1937695903 shutdown_connections 2024-05-27T07:53:24.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.136+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/1937695903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f54210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.136+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/1937695903 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6dcae40 0x556cc6f49be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.136+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/1937695903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6f4a120 0x556cc6f4eed0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.136+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/1937695903 >> 172.21.15.114:0/1937695903 conn(0x556cc6dcd770 msgr2=0x556cc6f5eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:24.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.136+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/1937695903 shutdown_connections 2024-05-27T07:53:24.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.136+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/1937695903 wait complete. 2024-05-27T07:53:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.136+0000 7f5d40f6fb80 1 -- start start 2024-05-27T07:53:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.136+0000 7f5d40f6fb80 1 Processor -- start 2024-05-27T07:53:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d40f6fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6dcae40 0x556cc6f69090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d40f6fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6f4a120 0x556cc6f66460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d40f6fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f669a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d40f6fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1800d370 con 0x556cc6dcae40 2024-05-27T07:53:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d40f6fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1800e7e0 con 0x556cc6f50070 2024-05-27T07:53:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d40f6fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1800da40 con 0x556cc6f4a120 2024-05-27T07:53:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d40f6fb80 10 client.0 _ll_register_callbacks cb 0x556cc6f601a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d2bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6dcae40 0x556cc6f69090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d2bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6dcae40 0x556cc6f69090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37100/0 (socket says 172.21.15.114:37100) 2024-05-27T07:53:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d2bfff700 1 -- 172.21.15.114:0/3797247199 learned_addr learned my addr 172.21.15.114:0/3797247199 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:24.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d30af7700 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6f4a120 0x556cc6f66460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d2bfff700 1 -- 172.21.15.114:0/3797247199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6f4a120 msgr2=0x556cc6f66460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d2bfff700 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6f4a120 0x556cc6f66460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d2bfff700 1 -- 172.21.15.114:0/3797247199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 msgr2=0x556cc6f669a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d2bfff700 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f669a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d2bfff700 1 -- 172.21.15.114:0/3797247199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cc6f66f30 con 0x556cc6dcae40 2024-05-27T07:53:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d30af7700 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f669a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d2bfff700 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6dcae40 0x556cc6f69090 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x556cc6e217f0 tx=0x7f5d1c01e7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e52eaf09255b58 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d0b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d0b7fe700 1 -- 172.21.15.114:0/3797247199 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c012860 con 0x556cc6dcae40 2024-05-27T07:53:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3797247199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d200095d0 con 0x556cc6dcae40 2024-05-27T07:53:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.137+0000 7f5d0b7fe700 1 -- 172.21.15.114:0/3797247199 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f5d1c012a00 con 0x556cc6dcae40 2024-05-27T07:53:24.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.138+0000 7f5d0b7fe700 1 -- 172.21.15.114:0/3797247199 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f5d1c030050 con 0x556cc6dcae40 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.138+0000 7f5d40f6fb80 10 client.18654 fetch_fsmap learned FSMap version 75 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.138+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3797247199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x556cc6fcdd70 con 0x556cc6dcae40 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.138+0000 7f5d0b7fe700 1 -- 172.21.15.114:0/3797247199 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d200095d0 con 0x556cc6dcae40 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.138+0000 7f5d0b7fe700 1 -- 172.21.15.114:0/3797247199 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 75) ==== 5910+0+0 (secure 0 0 0) 0x7f5d1c00f040 con 0x556cc6dcae40 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.138+0000 7f5d40f6fb80 10 client.18654 fetch_fsmap finished waiting for FSMap version 75 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.138+0000 7f5d40f6fb80 10 client.18654 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.138+0000 7f5d40f6fb80 20 client.18654 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.138+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556cc6fce400 0x556cc6fd0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.138+0000 7f5d40f6fb80 4 client.18654 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.138+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3797247199 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556cc6e734a0 con 0x556cc6fce400 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.138+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556cc6fce400 0x556cc6fd0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.138+0000 7f5d0b7fe700 10 client.18654 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d2b7fe700 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556cc6fce400 0x556cc6fd0820 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33d6265991879ad7 server_cookie=72d7b9be9669ea9c in_seq=0 out_seq=0 2024-05-27T07:53:24.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d0b7fe700 1 -- 172.21.15.114:0/3797247199 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556cc6e734a0 con 0x556cc6fce400 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d0b7fe700 10 client.18654 handle_command_reply: tid=0 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d40f6fb80 1 client.18654 shutdown 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3797247199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6dcae40 msgr2=0x556cc6f69090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6dcae40 0x556cc6f69090 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x556cc6e217f0 tx=0x7f5d1c01e7c0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3797247199 shutdown_connections 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556cc6fce400 0x556cc6fd0820 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556cc6f50070 0x556cc6f669a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556cc6dcae40 0x556cc6f69090 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d40f6fb80 1 --2- 172.21.15.114:0/3797247199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556cc6f4a120 0x556cc6f66460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3797247199 >> 172.21.15.114:0/3797247199 conn(0x556cc6dcd770 msgr2=0x556cc6f4dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d0b7fe700 0 client.18654 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3797247199 shutdown_connections 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d40f6fb80 1 -- 172.21.15.114:0/3797247199 wait complete. 2024-05-27T07:53:24.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.139+0000 7f5d40f6fb80 20 client.18654 trim_cache size 0 max 16384 2024-05-27T07:53:24.146 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:53:24.146 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:53:24.146 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:53:24.146 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:53:24.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.140+0000 7f5d3255a700 1 -- 172.21.15.114:0/17910822 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5d140b2a70 msgr2=0x7f5d140b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.140+0000 7f5d3255a700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5d140b2a70 0x7f5d140b4f80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5d24007770 tx=0x7f5d24007b30 comp rx=0 tx=0).stop 2024-05-27T07:53:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.140+0000 7f5d3255a700 1 -- 172.21.15.114:0/17910822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c1042e0 msgr2=0x7f5d2c19d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.140+0000 7f5d3255a700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c1042e0 0x7f5d2c19d5f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5d20000c00 tx=0x7f5d2000ddc0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.140+0000 7f5d3255a700 1 -- 172.21.15.114:0/17910822 shutdown_connections 2024-05-27T07:53:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.140+0000 7f5d3255a700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5d140b2a70 0x7f5d140b4f80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.140+0000 7f5d3255a700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d2c1081f0 0x7f5d2c1a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.140+0000 7f5d3255a700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2c103990 0x7f5d2c19d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.140+0000 7f5d3255a700 1 --2- 172.21.15.114:0/17910822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c1042e0 0x7f5d2c19d5f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.140+0000 7f5d3255a700 1 -- 172.21.15.114:0/17910822 >> 172.21.15.114:0/17910822 conn(0x7f5d2c0fe4d0 msgr2=0x7f5d2c109680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.140+0000 7f5d3255a700 1 -- 172.21.15.114:0/17910822 shutdown_connections 2024-05-27T07:53:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:24.140+0000 7f5d3255a700 1 -- 172.21.15.114:0/17910822 wait complete. 2024-05-27T07:53:24.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1299576 9.32 MB/sec execute 576 sec latency 40.439 ms 2024-05-27T07:53:24.798 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:53:24.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:53:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.871+0000 7fc74f0d1640 1 -- 172.21.15.114:0/2254497320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc748102570 msgr2=0x7fc748102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.871+0000 7fc74f0d1640 1 --2- 172.21.15.114:0/2254497320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc748102570 0x7fc748102980 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fc740009900 tx=0x7fc74002a410 comp rx=0 tx=0).stop 2024-05-27T07:53:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.871+0000 7fc74f0d1640 1 -- 172.21.15.114:0/2254497320 shutdown_connections 2024-05-27T07:53:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.871+0000 7fc74f0d1640 1 --2- 172.21.15.114:0/2254497320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7481075c0 0x7fc74810ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.871+0000 7fc74f0d1640 1 --2- 172.21.15.114:0/2254497320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc748106b00 0x7fc748106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.871+0000 7fc74f0d1640 1 --2- 172.21.15.114:0/2254497320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc748102570 0x7fc748102980 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.871+0000 7fc74f0d1640 1 -- 172.21.15.114:0/2254497320 >> 172.21.15.114:0/2254497320 conn(0x7fc7480779d0 msgr2=0x7fc748077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.872+0000 7fc74f0d1640 1 -- 172.21.15.114:0/2254497320 shutdown_connections 2024-05-27T07:53:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.872+0000 7fc74f0d1640 1 -- 172.21.15.114:0/2254497320 wait complete. 2024-05-27T07:53:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.872+0000 7fc74f0d1640 1 Processor -- start 2024-05-27T07:53:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.872+0000 7fc74f0d1640 1 -- start start 2024-05-27T07:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.872+0000 7fc74f0d1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc748102570 0x7fc748103b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.872+0000 7fc74f0d1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc748106b00 0x7fc748104050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc74f0d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7481075c0 0x7fc748104590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc74f0d1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481124f0 con 0x7fc748106b00 2024-05-27T07:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc74f0d1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc748112370 con 0x7fc748102570 2024-05-27T07:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc74f0d1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc748112670 con 0x7fc7481075c0 2024-05-27T07:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc747fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc748106b00 0x7fc748104050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc747fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc748106b00 0x7fc748104050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51964/0 (socket says 172.21.15.114:51964) 2024-05-27T07:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc747fff640 1 -- 172.21.15.114:0/3158514484 learned_addr learned my addr 172.21.15.114:0/3158514484 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc74d66e640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7481075c0 0x7fc748104590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc74ce6d640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc748102570 0x7fc748103b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc747fff640 1 -- 172.21.15.114:0/3158514484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7481075c0 msgr2=0x7fc748104590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc747fff640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7481075c0 0x7fc748104590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc747fff640 1 -- 172.21.15.114:0/3158514484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc748102570 msgr2=0x7fc748103b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc747fff640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc748102570 0x7fc748103b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc747fff640 1 -- 172.21.15.114:0/3158514484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7481a8c50 con 0x7fc748106b00 2024-05-27T07:53:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc74ce6d640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc748102570 0x7fc748103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:53:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc747fff640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc748106b00 0x7fc748104050 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fc73800ebc0 tx=0x7fc73800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc745ffb640 1 -- 172.21.15.114:0/3158514484 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738004b20 con 0x7fc748106b00 2024-05-27T07:53:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc745ffb640 1 -- 172.21.15.114:0/3158514484 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc738004cc0 con 0x7fc748106b00 2024-05-27T07:53:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc74f0d1640 1 -- 172.21.15.114:0/3158514484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7481a8f40 con 0x7fc748106b00 2024-05-27T07:53:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.873+0000 7fc745ffb640 1 -- 172.21.15.114:0/3158514484 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738010760 con 0x7fc748106b00 2024-05-27T07:53:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.874+0000 7fc74f0d1640 1 -- 172.21.15.114:0/3158514484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7481a93a0 con 0x7fc748106b00 2024-05-27T07:53:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.876+0000 7fc745ffb640 1 -- 172.21.15.114:0/3158514484 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fc738005020 con 0x7fc748106b00 2024-05-27T07:53:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.876+0000 7fc755f81740 1 Processor -- start 2024-05-27T07:53:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc755f81740 1 -- start start 2024-05-27T07:53:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc755f81740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 0x561cb054e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc755f81740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb01842f0 0x561cb054e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc755f81740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb018c4d0 0x561cb054ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc755f81740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561cb04c0c30 con 0x561cb01ad1b0 2024-05-27T07:53:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc755f81740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561cb04b07d0 con 0x561cb018c4d0 2024-05-27T07:53:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc755f81740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561cb04b4810 con 0x561cb01842f0 2024-05-27T07:53:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc74d66e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 0x561cb054e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc74d66e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 0x561cb054e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51966/0 (socket says 172.21.15.114:51966) 2024-05-27T07:53:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc74d66e640 1 -- 172.21.15.114:0/4132515268 learned_addr learned my addr 172.21.15.114:0/4132515268 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc74ce6d640 1 --2- 172.21.15.114:0/4132515268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb01842f0 0x561cb054e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc747fff640 1 --2- 172.21.15.114:0/4132515268 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb018c4d0 0x561cb054ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc74d66e640 1 -- 172.21.15.114:0/4132515268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb01842f0 msgr2=0x561cb054e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc74d66e640 1 --2- 172.21.15.114:0/4132515268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb01842f0 0x561cb054e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc74d66e640 1 -- 172.21.15.114:0/4132515268 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb018c4d0 msgr2=0x561cb054ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc74d66e640 1 --2- 172.21.15.114:0/4132515268 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb018c4d0 0x561cb054ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc745ffb640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7340b2950 0x7fc7340b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc74d66e640 1 -- 172.21.15.114:0/4132515268 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cb01c5210 con 0x561cb01ad1b0 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc747fff640 1 --2- 172.21.15.114:0/4132515268 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb018c4d0 0x561cb054ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.877+0000 7fc74ce6d640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7340b2950 0x7fc7340b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.878+0000 7fc74d66e640 1 --2- 172.21.15.114:0/4132515268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 0x561cb054e3d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fc730009c10 tx=0x7fc730011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b579b40dbf0239 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.878+0000 7fc745ffb640 1 -- 172.21.15.114:0/3158514484 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7fc7380e3330 con 0x7fc748106b00 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.878+0000 7fc73effd640 1 -- 172.21.15.114:0/4132515268 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730004230 con 0x561cb01ad1b0 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.878+0000 7fc73effd640 1 -- 172.21.15.114:0/4132515268 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7300043d0 con 0x561cb01ad1b0 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.878+0000 7fc73effd640 1 -- 172.21.15.114:0/4132515268 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730019640 con 0x561cb01ad1b0 2024-05-27T07:53:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.879+0000 7fc755f81740 1 -- 172.21.15.114:0/4132515268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 msgr2=0x561cb054e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.879+0000 7fc755f81740 1 --2- 172.21.15.114:0/4132515268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 0x561cb054e3d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fc730009c10 tx=0x7fc730011810 comp rx=0 tx=0).stop 2024-05-27T07:53:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.879+0000 7fc74ce6d640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7340b2950 0x7fc7340b4e60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc740031440 tx=0x7fc7400058b0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.879+0000 7fc755f81740 1 -- 172.21.15.114:0/4132515268 shutdown_connections 2024-05-27T07:53:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.879+0000 7fc755f81740 1 --2- 172.21.15.114:0/4132515268 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb018c4d0 0x561cb054ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.879+0000 7fc755f81740 1 --2- 172.21.15.114:0/4132515268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb01842f0 0x561cb054e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.879+0000 7fc755f81740 1 --2- 172.21.15.114:0/4132515268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 0x561cb054e3d0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.879+0000 7fc755f81740 1 -- 172.21.15.114:0/4132515268 >> 172.21.15.114:0/4132515268 conn(0x561cb01a1300 msgr2=0x561cb054ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.879+0000 7fc755f81740 1 -- 172.21.15.114:0/4132515268 shutdown_connections 2024-05-27T07:53:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.879+0000 7fc755f81740 1 -- 172.21.15.114:0/4132515268 wait complete. 2024-05-27T07:53:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.879+0000 7fc755f81740 1 -- start start 2024-05-27T07:53:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.879+0000 7fc755f81740 1 Processor -- start 2024-05-27T07:53:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc755f81740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01842f0 0x561cb0558ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc755f81740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb018c4d0 0x561cb0559030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc755f81740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 0x561cb055e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc755f81740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561cb0196790 con 0x561cb01ad1b0 2024-05-27T07:53:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc755f81740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561cb046c890 con 0x561cb01842f0 2024-05-27T07:53:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc755f81740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561cb055b500 con 0x561cb018c4d0 2024-05-27T07:53:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc755f81740 10 client.0 _ll_register_callbacks cb 0x561cb01c4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc74d66e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 0x561cb055e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc74d66e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 0x561cb055e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51968/0 (socket says 172.21.15.114:51968) 2024-05-27T07:53:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc74d66e640 1 -- 172.21.15.114:0/4240334407 learned_addr learned my addr 172.21.15.114:0/4240334407 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc747fff640 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb018c4d0 0x561cb0559030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc74ce6d640 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01842f0 0x561cb0558ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc74d66e640 1 -- 172.21.15.114:0/4240334407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb018c4d0 msgr2=0x561cb0559030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc74d66e640 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb018c4d0 0x561cb0559030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc74d66e640 1 -- 172.21.15.114:0/4240334407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01842f0 msgr2=0x561cb0558ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc74d66e640 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01842f0 0x561cb0558ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc74d66e640 1 -- 172.21.15.114:0/4240334407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cb0291620 con 0x561cb01ad1b0 2024-05-27T07:53:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc74ce6d640 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01842f0 0x561cb0558ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:53:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc74d66e640 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 0x561cb055e870 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fc730005760 tx=0x7fc730009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63e67cc7d2b64ec server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.880+0000 7fc727fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc727fff640 1 -- 172.21.15.114:0/4240334407 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730027070 con 0x561cb01ad1b0 2024-05-27T07:53:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc727fff640 1 -- 172.21.15.114:0/4240334407 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7fc730003800 con 0x561cb01ad1b0 2024-05-27T07:53:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc755f81740 1 -- 172.21.15.114:0/4240334407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7300043d0 con 0x561cb01ad1b0 2024-05-27T07:53:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc727fff640 1 -- 172.21.15.114:0/4240334407 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7fc730033050 con 0x561cb01ad1b0 2024-05-27T07:53:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc755f81740 10 client.18666 fetch_fsmap learned FSMap version 75 2024-05-27T07:53:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc755f81740 1 -- 172.21.15.114:0/4240334407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x561cb0567840 con 0x561cb01ad1b0 2024-05-27T07:53:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc727fff640 1 -- 172.21.15.114:0/4240334407 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7300043d0 con 0x561cb01ad1b0 2024-05-27T07:53:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc727fff640 1 -- 172.21.15.114:0/4240334407 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 75) ==== 5910+0+0 (secure 0 0 0) 0x7fc730018070 con 0x561cb01ad1b0 2024-05-27T07:53:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc755f81740 10 client.18666 fetch_fsmap finished waiting for FSMap version 75 2024-05-27T07:53:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc755f81740 10 client.18666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc755f81740 20 client.18666 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc755f81740 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cb016bd80 0x561cb056aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc755f81740 4 client.18666 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc755f81740 1 -- 172.21.15.114:0/4240334407 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cb04c2b80 con 0x561cb016bd80 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.881+0000 7fc74ce6d640 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cb016bd80 0x561cb056aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.882+0000 7fc727fff640 10 client.18666 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.882+0000 7fc74ce6d640 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cb016bd80 0x561cb056aed0 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b92f641bc7f00f3d server_cookie=8967b7664ad68df8 in_seq=0 out_seq=0 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc727fff640 1 -- 172.21.15.114:0/4240334407 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561cb04c2b80 con 0x561cb016bd80 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc727fff640 10 client.18666 handle_command_reply: tid=0 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc755f81740 1 client.18666 shutdown 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc755f81740 1 -- 172.21.15.114:0/4240334407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 msgr2=0x561cb055e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc755f81740 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 0x561cb055e870 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fc730005760 tx=0x7fc730009e80 comp rx=0 tx=0).stop 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc755f81740 1 -- 172.21.15.114:0/4240334407 shutdown_connections 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc755f81740 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cb016bd80 0x561cb056aed0 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc755f81740 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01ad1b0 0x561cb055e870 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc755f81740 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb018c4d0 0x561cb0559030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc755f81740 1 --2- 172.21.15.114:0/4240334407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01842f0 0x561cb0558ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc727fff640 0 client.18666 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc755f81740 1 -- 172.21.15.114:0/4240334407 >> 172.21.15.114:0/4240334407 conn(0x561cb01a1300 msgr2=0x561cb054e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc755f81740 1 -- 172.21.15.114:0/4240334407 shutdown_connections 2024-05-27T07:53:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc755f81740 1 -- 172.21.15.114:0/4240334407 wait complete. 2024-05-27T07:53:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.883+0000 7fc755f81740 20 client.18666 trim_cache size 0 max 16384 2024-05-27T07:53:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc755f81740 1 Processor -- start 2024-05-27T07:53:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc755f81740 1 -- start start 2024-05-27T07:53:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc755f81740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01a1300 0x561cb05706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc755f81740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb016bd80 0x561cb0570c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc755f81740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb01842f0 0x561cb054e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc755f81740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481b4f60 con 0x561cb016bd80 2024-05-27T07:53:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc755f81740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481b36f0 con 0x561cb01a1300 2024-05-27T07:53:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc755f81740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481b1560 con 0x561cb01842f0 2024-05-27T07:53:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc747fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb016bd80 0x561cb0570c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc74d66e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb01842f0 0x561cb054e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc74ce6d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01a1300 0x561cb05706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc747fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb016bd80 0x561cb0570c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51972/0 (socket says 172.21.15.114:51972) 2024-05-27T07:53:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc74ce6d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01a1300 0x561cb05706d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:57080/0 (socket says 172.21.15.114:57080) 2024-05-27T07:53:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc74d66e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb01842f0 0x561cb054e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54972/0 (socket says 172.21.15.114:54972) 2024-05-27T07:53:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc747fff640 1 -- 172.21.15.114:0/3230377232 learned_addr learned my addr 172.21.15.114:0/3230377232 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc747fff640 1 -- 172.21.15.114:0/3230377232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb01842f0 msgr2=0x561cb054e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc747fff640 1 --2- 172.21.15.114:0/3230377232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb01842f0 0x561cb054e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc747fff640 1 -- 172.21.15.114:0/3230377232 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01a1300 msgr2=0x561cb05706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc747fff640 1 --2- 172.21.15.114:0/3230377232 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01a1300 0x561cb05706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc747fff640 1 -- 172.21.15.114:0/3230377232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cb054f780 con 0x561cb016bd80 2024-05-27T07:53:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc74ce6d640 1 --2- 172.21.15.114:0/3230377232 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01a1300 0x561cb05706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.900+0000 7fc74d66e640 1 --2- 172.21.15.114:0/3230377232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb01842f0 0x561cb054e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.901+0000 7fc747fff640 1 --2- 172.21.15.114:0/3230377232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb016bd80 0x561cb0570c10 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fc7380e0c80 tx=0x7fc7380a9c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b65f680589ec977 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.901+0000 7fc73e7fc640 1 -- 172.21.15.114:0/3230377232 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7380aa910 con 0x561cb016bd80 2024-05-27T07:53:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.901+0000 7fc73e7fc640 1 -- 172.21.15.114:0/3230377232 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7380aa210 con 0x561cb016bd80 2024-05-27T07:53:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.901+0000 7fc73e7fc640 1 -- 172.21.15.114:0/3230377232 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7380b2e40 con 0x561cb016bd80 2024-05-27T07:53:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.902+0000 7fc755f81740 1 -- 172.21.15.114:0/3230377232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb016bd80 msgr2=0x561cb0570c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.902+0000 7fc755f81740 1 --2- 172.21.15.114:0/3230377232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb016bd80 0x561cb0570c10 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fc7380e0c80 tx=0x7fc7380a9c20 comp rx=0 tx=0).stop 2024-05-27T07:53:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.902+0000 7fc755f81740 1 -- 172.21.15.114:0/3230377232 shutdown_connections 2024-05-27T07:53:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.902+0000 7fc755f81740 1 --2- 172.21.15.114:0/3230377232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb01842f0 0x561cb054e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.902+0000 7fc755f81740 1 --2- 172.21.15.114:0/3230377232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb016bd80 0x561cb0570c10 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.902+0000 7fc755f81740 1 --2- 172.21.15.114:0/3230377232 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01a1300 0x561cb05706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.902+0000 7fc755f81740 1 -- 172.21.15.114:0/3230377232 >> 172.21.15.114:0/3230377232 conn(0x561cb0552890 msgr2=0x561cb0557190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.902+0000 7fc755f81740 1 -- 172.21.15.114:0/3230377232 shutdown_connections 2024-05-27T07:53:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.902+0000 7fc755f81740 1 -- 172.21.15.114:0/3230377232 wait complete. 2024-05-27T07:53:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.902+0000 7fc755f81740 1 -- start start 2024-05-27T07:53:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.902+0000 7fc755f81740 1 Processor -- start 2024-05-27T07:53:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc755f81740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb016bd80 0x561cb05cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc755f81740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01842f0 0x561cb05cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc755f81740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01a1300 0x561cb0561c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc755f81740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481b1bb0 con 0x561cb01a1300 2024-05-27T07:53:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc755f81740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481b3020 con 0x561cb01842f0 2024-05-27T07:53:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc755f81740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481b2280 con 0x561cb016bd80 2024-05-27T07:53:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc755f81740 10 client.0 _ll_register_callbacks cb 0x7fc730005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc74d66e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01a1300 0x561cb0561c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc74ce6d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb016bd80 0x561cb05cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc74d66e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01a1300 0x561cb0561c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51982/0 (socket says 172.21.15.114:51982) 2024-05-27T07:53:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc74d66e640 1 -- 172.21.15.114:0/2377639018 learned_addr learned my addr 172.21.15.114:0/2377639018 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc747fff640 1 --2- 172.21.15.114:0/2377639018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01842f0 0x561cb05cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc74d66e640 1 -- 172.21.15.114:0/2377639018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb016bd80 msgr2=0x561cb05cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc74d66e640 1 --2- 172.21.15.114:0/2377639018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb016bd80 0x561cb05cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc74d66e640 1 -- 172.21.15.114:0/2377639018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01842f0 msgr2=0x561cb05cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc74d66e640 1 --2- 172.21.15.114:0/2377639018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01842f0 0x561cb05cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc74d66e640 1 -- 172.21.15.114:0/2377639018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cb0562210 con 0x561cb01a1300 2024-05-27T07:53:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc74ce6d640 1 --2- 172.21.15.114:0/2377639018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb016bd80 0x561cb05cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc74d66e640 1 --2- 172.21.15.114:0/2377639018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01a1300 0x561cb0561c00 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fc73001afd0 tx=0x7fc730009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec22172db3bac000 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc727fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.903+0000 7fc727fff640 1 -- 172.21.15.114:0/2377639018 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73003c060 con 0x561cb01a1300 2024-05-27T07:53:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc727fff640 1 -- 172.21.15.114:0/2377639018 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7fc730005410 con 0x561cb01a1300 2024-05-27T07:53:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc755f81740 1 -- 172.21.15.114:0/2377639018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7380aa210 con 0x561cb01a1300 2024-05-27T07:53:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc727fff640 1 -- 172.21.15.114:0/2377639018 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7fc730043050 con 0x561cb01a1300 2024-05-27T07:53:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc755f81740 10 client.18672 fetch_fsmap learned FSMap version 75 2024-05-27T07:53:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc755f81740 1 -- 172.21.15.114:0/2377639018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x561cb05d3060 con 0x561cb01a1300 2024-05-27T07:53:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc727fff640 1 -- 172.21.15.114:0/2377639018 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7380aa210 con 0x561cb01a1300 2024-05-27T07:53:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc727fff640 1 -- 172.21.15.114:0/2377639018 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 75) ==== 5910+0+0 (secure 0 0 0) 0x7fc730018030 con 0x561cb01a1300 2024-05-27T07:53:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc755f81740 10 client.18672 fetch_fsmap finished waiting for FSMap version 75 2024-05-27T07:53:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc755f81740 10 client.18672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc755f81740 20 client.18672 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc755f81740 1 --2- 172.21.15.114:0/2377639018 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cb018c4d0 0x561cb05d5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc755f81740 4 client.18672 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:53:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc755f81740 1 -- 172.21.15.114:0/2377639018 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561cb04bc360 con 0x561cb018c4d0 2024-05-27T07:53:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.904+0000 7fc74ce6d640 1 --2- 172.21.15.114:0/2377639018 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cb018c4d0 0x561cb05d5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.905+0000 7fc727fff640 10 client.18672 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:24.905+0000 7fc74ce6d640 1 --2- 172.21.15.114:0/2377639018 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cb018c4d0 0x561cb05d5730 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=995b48398a5ab5f8 server_cookie=7c62cfce84e88506 in_seq=0 out_seq=0 2024-05-27T07:53:25.298 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1300219 9.32 MB/sec execute 577 sec latency 305.583 ms 2024-05-27T07:53:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc727fff640 1 -- 172.21.15.114:0/2377639018 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x561cb04bc360 con 0x561cb018c4d0 2024-05-27T07:53:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc727fff640 10 client.18672 handle_command_reply: tid=0 2024-05-27T07:53:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc755f81740 1 client.18672 shutdown 2024-05-27T07:53:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc755f81740 1 -- 172.21.15.114:0/2377639018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01a1300 msgr2=0x561cb0561c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc755f81740 1 --2- 172.21.15.114:0/2377639018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01a1300 0x561cb0561c00 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fc73001afd0 tx=0x7fc730009ef0 comp rx=0 tx=0).stop 2024-05-27T07:53:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc755f81740 1 -- 172.21.15.114:0/2377639018 shutdown_connections 2024-05-27T07:53:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc755f81740 1 --2- 172.21.15.114:0/2377639018 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561cb018c4d0 0x561cb05d5730 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc755f81740 1 --2- 172.21.15.114:0/2377639018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cb01a1300 0x561cb0561c00 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc755f81740 1 --2- 172.21.15.114:0/2377639018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561cb01842f0 0x561cb05cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc755f81740 1 --2- 172.21.15.114:0/2377639018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561cb016bd80 0x561cb05cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc755f81740 1 -- 172.21.15.114:0/2377639018 >> 172.21.15.114:0/2377639018 conn(0x561cb0552890 msgr2=0x561cb054e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc727fff640 0 client.18672 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc755f81740 1 -- 172.21.15.114:0/2377639018 shutdown_connections 2024-05-27T07:53:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.597+0000 7fc755f81740 1 -- 172.21.15.114:0/2377639018 wait complete. 2024-05-27T07:53:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.598+0000 7fc755f81740 20 client.18672 trim_cache size 0 max 16384 2024-05-27T07:53:25.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:53:25.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:53:25.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:53:25.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.598+0000 7fc74f0d1640 1 -- 172.21.15.114:0/3158514484 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7340b2950 msgr2=0x7fc7340b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:25.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.598+0000 7fc74f0d1640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7340b2950 0x7fc7340b4e60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc740031440 tx=0x7fc7400058b0 comp rx=0 tx=0).stop 2024-05-27T07:53:25.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.598+0000 7fc74f0d1640 1 -- 172.21.15.114:0/3158514484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc748106b00 msgr2=0x7fc748104050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:25.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.598+0000 7fc74f0d1640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc748106b00 0x7fc748104050 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fc73800ebc0 tx=0x7fc73800c2d0 comp rx=0 tx=0).stop 2024-05-27T07:53:25.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.598+0000 7fc74f0d1640 1 -- 172.21.15.114:0/3158514484 shutdown_connections 2024-05-27T07:53:25.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.598+0000 7fc74f0d1640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7340b2950 0x7fc7340b4e60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:25.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.598+0000 7fc74f0d1640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7481075c0 0x7fc748104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:25.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.598+0000 7fc74f0d1640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc748106b00 0x7fc748104050 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:25.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.598+0000 7fc74f0d1640 1 --2- 172.21.15.114:0/3158514484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc748102570 0x7fc748103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:25.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.598+0000 7fc74f0d1640 1 -- 172.21.15.114:0/3158514484 >> 172.21.15.114:0/3158514484 conn(0x7fc7480779d0 msgr2=0x7fc748069c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:25.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.598+0000 7fc74f0d1640 1 -- 172.21.15.114:0/3158514484 shutdown_connections 2024-05-27T07:53:25.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:25.598+0000 7fc74f0d1640 1 -- 172.21.15.114:0/3158514484 wait complete. 2024-05-27T07:53:25.662 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:25 smithi114 ceph-mon[32038]: pgmap v461: 129 pgs: 129 active+clean; 76 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 174 op/s 2024-05-27T07:53:25.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:25 smithi163 ceph-mon[36451]: pgmap v461: 129 pgs: 129 active+clean; 76 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 174 op/s 2024-05-27T07:53:25.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:25 smithi186 ceph-mon[36647]: pgmap v461: 129 pgs: 129 active+clean; 76 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 174 op/s 2024-05-27T07:53:26.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1300861 9.30 MB/sec execute 578 sec latency 483.903 ms 2024-05-27T07:53:27.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1303372 9.31 MB/sec execute 579 sec latency 18.000 ms 2024-05-27T07:53:27.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.383+0000 7f281679e700 1 -- 172.21.15.114:0/1616844543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2810108560 msgr2=0x7f281010fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:27.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.383+0000 7f281679e700 1 --2- 172.21.15.114:0/1616844543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2810108560 0x7f281010fec0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f280800b0d0 tx=0x7f280800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:53:27.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.383+0000 7f281679e700 1 -- 172.21.15.114:0/1616844543 shutdown_connections 2024-05-27T07:53:27.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.383+0000 7f281679e700 1 --2- 172.21.15.114:0/1616844543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2810108560 0x7f281010fec0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:27.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.383+0000 7f281679e700 1 --2- 172.21.15.114:0/1616844543 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28101032e0 0x7f28101036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:27.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.383+0000 7f281679e700 1 --2- 172.21.15.114:0/1616844543 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2810107c10 0x7f2810108020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:27.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.383+0000 7f281679e700 1 -- 172.21.15.114:0/1616844543 >> 172.21.15.114:0/1616844543 conn(0x7f2810078b40 msgr2=0x7f2810078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:27.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.384+0000 7f281679e700 1 -- 172.21.15.114:0/1616844543 shutdown_connections 2024-05-27T07:53:27.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.384+0000 7f281679e700 1 -- 172.21.15.114:0/1616844543 wait complete. 2024-05-27T07:53:27.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.384+0000 7f281679e700 1 Processor -- start 2024-05-27T07:53:27.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281679e700 1 -- start start 2024-05-27T07:53:27.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281679e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28101032e0 0x7f2810113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:27.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281679e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2810107c10 0x7f2810113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:27.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281679e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2810108560 0x7f2810114410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:27.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281679e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2810113190 con 0x7f2810108560 2024-05-27T07:53:27.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281679e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2810113010 con 0x7f2810107c10 2024-05-27T07:53:27.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281679e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2810113310 con 0x7f28101032e0 2024-05-27T07:53:27.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281579c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28101032e0 0x7f2810113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:27.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f2815f9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2810108560 0x7f2810114410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:27.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281579c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28101032e0 0x7f2810113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54998/0 (socket says 172.21.15.114:54998) 2024-05-27T07:53:27.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281579c700 1 -- 172.21.15.114:0/589101466 learned_addr learned my addr 172.21.15.114:0/589101466 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:27.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f2814f9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2810107c10 0x7f2810113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:27.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281579c700 1 -- 172.21.15.114:0/589101466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2810107c10 msgr2=0x7f2810113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:27.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281579c700 1 --2- 172.21.15.114:0/589101466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2810107c10 0x7f2810113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:27.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281579c700 1 -- 172.21.15.114:0/589101466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2810108560 msgr2=0x7f2810114410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:27.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281579c700 1 --2- 172.21.15.114:0/589101466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2810108560 0x7f2810114410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:27.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.385+0000 7f281579c700 1 -- 172.21.15.114:0/589101466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28101a8fd0 con 0x7f28101032e0 2024-05-27T07:53:27.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.386+0000 7f281579c700 1 --2- 172.21.15.114:0/589101466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28101032e0 0x7f2810113990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f280c00e9a0 tx=0x7f280c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:27.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.386+0000 7f28067fc700 1 -- 172.21.15.114:0/589101466 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c00ca10 con 0x7f28101032e0 2024-05-27T07:53:27.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.386+0000 7f281679e700 1 -- 172.21.15.114:0/589101466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28101a92c0 con 0x7f28101032e0 2024-05-27T07:53:27.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.386+0000 7f28067fc700 1 -- 172.21.15.114:0/589101466 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f280c00cbb0 con 0x7f28101032e0 2024-05-27T07:53:27.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.386+0000 7f28067fc700 1 -- 172.21.15.114:0/589101466 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c018680 con 0x7f28101032e0 2024-05-27T07:53:27.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.386+0000 7f281679e700 1 -- 172.21.15.114:0/589101466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28101a9850 con 0x7f28101032e0 2024-05-27T07:53:27.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.387+0000 7f281679e700 1 -- 172.21.15.114:0/589101466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2810104790 con 0x7f28101032e0 2024-05-27T07:53:27.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.389+0000 7f28067fc700 1 -- 172.21.15.114:0/589101466 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f280c010430 con 0x7f28101032e0 2024-05-27T07:53:27.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.390+0000 7f28067fc700 1 --2- 172.21.15.114:0/589101466 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f27fc0b29f0 0x7f27fc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:27.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.390+0000 7f2814f9b700 1 --2- 172.21.15.114:0/589101466 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f27fc0b29f0 0x7f27fc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:27.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.390+0000 7f28067fc700 1 -- 172.21.15.114:0/589101466 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f280c0e2eb0 con 0x7f28101032e0 2024-05-27T07:53:27.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.390+0000 7f2814f9b700 1 --2- 172.21.15.114:0/589101466 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f27fc0b29f0 0x7f27fc0b4f00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2810101d20 tx=0x7f2800009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:27.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.393+0000 7f28067fc700 1 -- 172.21.15.114:0/589101466 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f280c0aa490 con 0x7f28101032e0 2024-05-27T07:53:27.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:27 smithi114 ceph-mon[32038]: pgmap v462: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.2 MiB/s wr, 437 op/s 2024-05-27T07:53:27.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.670+0000 7f281679e700 1 -- 172.21.15.114:0/589101466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28100512b0 con 0x7f28101032e0 2024-05-27T07:53:27.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.670+0000 7f28067fc700 1 -- 172.21.15.114:0/589101466 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+8522 (secure 0 0 0) 0x7f280c0a9d90 con 0x7f28101032e0 2024-05-27T07:53:27.672 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:53:27.672 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":27557,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3261223371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":3261223371},{"type":"v1","addr":"172.21.15.163:6835","nonce":3261223371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":75,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:53:18.374687+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_28162":{"gid":28162,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/2682173878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2682173878},{"type":"v1","addr":"172.21.15.186:6833","nonce":2682173878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:53:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.673+0000 7f281679e700 1 -- 172.21.15.114:0/589101466 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f27fc0b29f0 msgr2=0x7f27fc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.673+0000 7f281679e700 1 --2- 172.21.15.114:0/589101466 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f27fc0b29f0 0x7f27fc0b4f00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2810101d20 tx=0x7f2800009260 comp rx=0 tx=0).stop 2024-05-27T07:53:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.673+0000 7f281679e700 1 -- 172.21.15.114:0/589101466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28101032e0 msgr2=0x7f2810113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.673+0000 7f281679e700 1 --2- 172.21.15.114:0/589101466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28101032e0 0x7f2810113990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f280c00e9a0 tx=0x7f280c00ecb0 comp rx=0 tx=0).stop 2024-05-27T07:53:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.673+0000 7f281679e700 1 -- 172.21.15.114:0/589101466 shutdown_connections 2024-05-27T07:53:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.673+0000 7f281679e700 1 --2- 172.21.15.114:0/589101466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2810108560 0x7f2810114410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.673+0000 7f281679e700 1 --2- 172.21.15.114:0/589101466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28101032e0 0x7f2810113990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.673+0000 7f281679e700 1 --2- 172.21.15.114:0/589101466 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f27fc0b29f0 0x7f27fc0b4f00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.673+0000 7f281679e700 1 --2- 172.21.15.114:0/589101466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2810107c10 0x7f2810113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.673+0000 7f281679e700 1 -- 172.21.15.114:0/589101466 >> 172.21.15.114:0/589101466 conn(0x7f2810078b40 msgr2=0x7f2810106bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.673+0000 7f281679e700 1 -- 172.21.15.114:0/589101466 shutdown_connections 2024-05-27T07:53:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:27.673+0000 7f281679e700 1 -- 172.21.15.114:0/589101466 wait complete. 2024-05-27T07:53:27.676 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 75 2024-05-27T07:53:27.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:27 smithi163 ceph-mon[36451]: pgmap v462: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.2 MiB/s wr, 437 op/s 2024-05-27T07:53:27.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:27 smithi186 ceph-mon[36647]: pgmap v462: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.2 MiB/s wr, 437 op/s 2024-05-27T07:53:28.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1304500 9.30 MB/sec execute 580 sec latency 24.498 ms 2024-05-27T07:53:28.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:53:28.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:28 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/589101466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:53:28.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:28 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/589101466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:53:28.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:28 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/589101466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:53:28.812 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:29.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1306915 9.31 MB/sec execute 581 sec latency 46.714 ms 2024-05-27T07:53:29.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:29 smithi114 ceph-mon[32038]: pgmap v463: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 367 op/s 2024-05-27T07:53:29.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:29 smithi163 ceph-mon[36451]: pgmap v463: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 367 op/s 2024-05-27T07:53:29.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:29 smithi186 ceph-mon[36647]: pgmap v463: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 367 op/s 2024-05-27T07:53:30.229 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:29 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:53:30.229 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:29 smithi186 systemd[1]: Stopped Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:53:30.230 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:29 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Start request repeated too quickly. 2024-05-27T07:53:30.230 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:29 smithi186 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:53:30.230 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 27 07:53:29 smithi186 systemd[1]: Failed to start Ceph mds.l for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:53:30.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1307720 9.30 MB/sec execute 582 sec latency 82.077 ms 2024-05-27T07:53:30.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.760+0000 7f9b493b9700 1 -- 172.21.15.114:0/4099493340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b4410b2d0 msgr2=0x7f9b44117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:30.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.760+0000 7f9b493b9700 1 --2- 172.21.15.114:0/4099493340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b4410b2d0 0x7f9b44117800 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00b0a0 tx=0x7f9b3c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.762+0000 7f9b493b9700 1 -- 172.21.15.114:0/4099493340 shutdown_connections 2024-05-27T07:53:30.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.762+0000 7f9b493b9700 1 --2- 172.21.15.114:0/4099493340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b4410b2d0 0x7f9b44117800 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.762+0000 7f9b493b9700 1 --2- 172.21.15.114:0/4099493340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b44110960 0x7f9b44110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.762+0000 7f9b493b9700 1 --2- 172.21.15.114:0/4099493340 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b4410a980 0x7f9b4410ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.762+0000 7f9b493b9700 1 -- 172.21.15.114:0/4099493340 >> 172.21.15.114:0/4099493340 conn(0x7f9b44077c70 msgr2=0x7f9b440780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.764+0000 7f9b493b9700 1 -- 172.21.15.114:0/4099493340 shutdown_connections 2024-05-27T07:53:30.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.765+0000 7f9b493b9700 1 -- 172.21.15.114:0/4099493340 wait complete. 2024-05-27T07:53:30.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.766+0000 7f9b493b9700 1 Processor -- start 2024-05-27T07:53:30.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.766+0000 7f9b493b9700 1 -- start start 2024-05-27T07:53:30.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.766+0000 7f9b493b9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b4410a980 0x7f9b44075230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.766+0000 7f9b493b9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b4410b2d0 0x7f9b4406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.766+0000 7f9b493b9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b44110960 0x7f9b4406d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.766+0000 7f9b493b9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4411ab50 con 0x7f9b4410a980 2024-05-27T07:53:30.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.766+0000 7f9b493b9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4411a9d0 con 0x7f9b4410b2d0 2024-05-27T07:53:30.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.766+0000 7f9b493b9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4411acd0 con 0x7f9b44110960 2024-05-27T07:53:30.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.769+0000 7f9b42ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b4410a980 0x7f9b44075230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.769+0000 7f9b42ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b4410a980 0x7f9b44075230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52020/0 (socket says 172.21.15.114:52020) 2024-05-27T07:53:30.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.769+0000 7f9b42ffd700 1 -- 172.21.15.114:0/2066416833 learned_addr learned my addr 172.21.15.114:0/2066416833 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:30.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.770+0000 7f9b427fc700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b4410b2d0 0x7f9b4406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.771+0000 7f9b437fe700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b44110960 0x7f9b4406d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.771+0000 7f9b42ffd700 1 -- 172.21.15.114:0/2066416833 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b44110960 msgr2=0x7f9b4406d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:30.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.771+0000 7f9b42ffd700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b44110960 0x7f9b4406d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.771+0000 7f9b42ffd700 1 -- 172.21.15.114:0/2066416833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b4410b2d0 msgr2=0x7f9b4406d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:30.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.771+0000 7f9b42ffd700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b4410b2d0 0x7f9b4406d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.771+0000 7f9b42ffd700 1 -- 172.21.15.114:0/2066416833 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4406dee0 con 0x7f9b4410a980 2024-05-27T07:53:30.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.772+0000 7f9b42ffd700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b4410a980 0x7f9b44075230 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800b550 tx=0x7f9b3800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:30.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.772+0000 7f9b2bfff700 1 -- 172.21.15.114:0/2066416833 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38015020 con 0x7f9b4410a980 2024-05-27T07:53:30.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.772+0000 7f9b2bfff700 1 -- 172.21.15.114:0/2066416833 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b38011ce0 con 0x7f9b4410a980 2024-05-27T07:53:30.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.772+0000 7f9b2bfff700 1 -- 172.21.15.114:0/2066416833 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38010d80 con 0x7f9b4410a980 2024-05-27T07:53:30.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.772+0000 7f9b493b9700 1 -- 172.21.15.114:0/2066416833 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b441b6210 con 0x7f9b4410a980 2024-05-27T07:53:30.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.772+0000 7f9b493b9700 1 -- 172.21.15.114:0/2066416833 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b441b6670 con 0x7f9b4410a980 2024-05-27T07:53:30.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.780+0000 7f9b57dceb80 1 Processor -- start 2024-05-27T07:53:30.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.780+0000 7f9b2bfff700 1 -- 172.21.15.114:0/2066416833 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f9b38018020 con 0x7f9b4410a980 2024-05-27T07:53:30.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.781+0000 7f9b2bfff700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9b2c0b2aa0 0x7f9b2c0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.782+0000 7f9b2bfff700 1 -- 172.21.15.114:0/2066416833 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f9b380e20c0 con 0x7f9b4410a980 2024-05-27T07:53:30.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.782+0000 7f9b57dceb80 1 -- start start 2024-05-27T07:53:30.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.783+0000 7f9b57dceb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba36710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.783+0000 7f9b57dceb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 0x55fdbba30a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.783+0000 7f9b57dceb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba30f80 0x55fdbba31390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.783+0000 7f9b57dceb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fdbb95fc90 con 0x55fdbb8b2e40 2024-05-27T07:53:30.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.783+0000 7f9b57dceb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fdbb94f830 con 0x55fdbba30f80 2024-05-27T07:53:30.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.783+0000 7f9b57dceb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fdbb953870 con 0x55fdbba36c50 2024-05-27T07:53:30.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.784+0000 7f9b42ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba36710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.784+0000 7f9b437fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba30f80 0x55fdbba31390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.784+0000 7f9b42ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba36710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52036/0 (socket says 172.21.15.114:52036) 2024-05-27T07:53:30.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.784+0000 7f9b42ffd700 1 -- 172.21.15.114:0/117028665 learned_addr learned my addr 172.21.15.114:0/117028665 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:30.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.784+0000 7f9b437fe700 1 -- 172.21.15.114:0/117028665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 msgr2=0x55fdbba30a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:53:30.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.784+0000 7f9b437fe700 1 --2- 172.21.15.114:0/117028665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 0x55fdbba30a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.785+0000 7f9b437fe700 1 -- 172.21.15.114:0/117028665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 msgr2=0x55fdbba36710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:30.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.785+0000 7f9b437fe700 1 --2- 172.21.15.114:0/117028665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba36710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.785+0000 7f9b437fe700 1 -- 172.21.15.114:0/117028665 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdbb89f8d0 con 0x55fdbba30f80 2024-05-27T07:53:30.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.785+0000 7f9b437fe700 1 --2- 172.21.15.114:0/117028665 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba30f80 0x55fdbba31390 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00b070 tx=0x7f9b3c009280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9be0e61524c1ee27 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:30.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.787+0000 7f9b437fe700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9b2c0b2aa0 0x7f9b2c0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.789+0000 7f9b297fa700 1 -- 172.21.15.114:0/117028665 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c004c50 con 0x55fdbba30f80 2024-05-27T07:53:30.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.789+0000 7f9b297fa700 1 -- 172.21.15.114:0/117028665 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b3c004df0 con 0x55fdbba30f80 2024-05-27T07:53:30.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.789+0000 7f9b297fa700 1 -- 172.21.15.114:0/117028665 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c005530 con 0x55fdbba30f80 2024-05-27T07:53:30.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.790+0000 7f9b57dceb80 1 -- 172.21.15.114:0/117028665 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba30f80 msgr2=0x55fdbba31390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:30.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.790+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/117028665 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba30f80 0x55fdbba31390 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00b070 tx=0x7f9b3c009280 comp rx=0 tx=0).stop 2024-05-27T07:53:30.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.791+0000 7f9b437fe700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9b2c0b2aa0 0x7f9b2c0b4fb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c023000 tx=0x7f9b3c027040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:30.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.792+0000 7f9b57dceb80 1 -- 172.21.15.114:0/117028665 shutdown_connections 2024-05-27T07:53:30.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.792+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/117028665 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba30f80 0x55fdbba31390 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.792+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/117028665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba36710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.792+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/117028665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 0x55fdbba30a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.792+0000 7f9b57dceb80 1 -- 172.21.15.114:0/117028665 >> 172.21.15.114:0/117028665 conn(0x55fdbb90daf0 msgr2=0x55fdbba2b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:30.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.798+0000 7f9b57dceb80 1 -- 172.21.15.114:0/117028665 shutdown_connections 2024-05-27T07:53:30.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.799+0000 7f9b57dceb80 1 -- 172.21.15.114:0/117028665 wait complete. 2024-05-27T07:53:30.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.799+0000 7f9b57dceb80 1 -- start start 2024-05-27T07:53:30.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.800+0000 7f9b57dceb80 1 Processor -- start 2024-05-27T07:53:30.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.800+0000 7f9b57dceb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbb8b2e40 0x55fdbba3a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.801+0000 7f9b57dceb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba30f80 0x55fdbba3a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.801+0000 7f9b57dceb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbba36c50 0x55fdbba3ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.801+0000 7f9b57dceb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fdbb855fa0 con 0x55fdbba36c50 2024-05-27T07:53:30.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.801+0000 7f9b42ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbb8b2e40 0x55fdbba3a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.801+0000 7f9b42ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbb8b2e40 0x55fdbba3a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:57158/0 (socket says 172.21.15.114:57158) 2024-05-27T07:53:30.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.801+0000 7f9b42ffd700 1 -- 172.21.15.114:0/1314622841 learned_addr learned my addr 172.21.15.114:0/1314622841 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:30.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.801+0000 7f9b427fc700 1 --2- 172.21.15.114:0/1314622841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba30f80 0x55fdbba3a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.801+0000 7f9b57dceb80 1 -- 172.21.15.114:0/1314622841 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fdbb8f17c0 con 0x55fdbb8b2e40 2024-05-27T07:53:30.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.801+0000 7f9b57dceb80 1 -- 172.21.15.114:0/1314622841 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fdbba3d780 con 0x55fdbba30f80 2024-05-27T07:53:30.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.801+0000 7f9b57dceb80 10 client.0 _ll_register_callbacks cb 0x55fdbb8ea920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:30.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.802+0000 7f9b427fc700 1 -- 172.21.15.114:0/1314622841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbb8b2e40 msgr2=0x55fdbba3a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:30.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.802+0000 7f9b427fc700 1 --2- 172.21.15.114:0/1314622841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbb8b2e40 0x55fdbba3a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.802+0000 7f9b427fc700 1 -- 172.21.15.114:0/1314622841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbba36c50 msgr2=0x55fdbba3ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:30.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.802+0000 7f9b427fc700 1 --2- 172.21.15.114:0/1314622841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbba36c50 0x55fdbba3ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.802+0000 7f9b427fc700 1 -- 172.21.15.114:0/1314622841 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdbb8ee690 con 0x55fdbba30f80 2024-05-27T07:53:30.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.803+0000 7f9b427fc700 1 --2- 172.21.15.114:0/1314622841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba30f80 0x55fdbba3a7f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9b441098e0 tx=0x7f9b340106b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7528837799333e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:30.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.803+0000 7f9b26ff5700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:53:30.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.803+0000 7f9b26ff5700 1 -- 172.21.15.114:0/1314622841 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34004030 con 0x55fdbba30f80 2024-05-27T07:53:30.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.803+0000 7f9b26ff5700 1 -- 172.21.15.114:0/1314622841 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f9b340041d0 con 0x55fdbba30f80 2024-05-27T07:53:30.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.804+0000 7f9b57dceb80 1 -- 172.21.15.114:0/1314622841 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b3c004df0 con 0x55fdbba30f80 2024-05-27T07:53:30.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.804+0000 7f9b26ff5700 1 -- 172.21.15.114:0/1314622841 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f9b340043d0 con 0x55fdbba30f80 2024-05-27T07:53:30.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.804+0000 7f9b57dceb80 10 client.28228 fetch_fsmap learned FSMap version 75 2024-05-27T07:53:30.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.804+0000 7f9b57dceb80 1 -- 172.21.15.114:0/1314622841 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x55fdbba481f0 con 0x55fdbba30f80 2024-05-27T07:53:30.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.808+0000 7f9b26ff5700 1 -- 172.21.15.114:0/1314622841 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b34004ce0 con 0x55fdbba30f80 2024-05-27T07:53:30.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.808+0000 7f9b26ff5700 1 -- 172.21.15.114:0/1314622841 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 75) ==== 5910+0+0 (secure 0 0 0) 0x7f9b34003680 con 0x55fdbba30f80 2024-05-27T07:53:30.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.808+0000 7f9b57dceb80 10 client.28228 fetch_fsmap finished waiting for FSMap version 75 2024-05-27T07:53:30.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.809+0000 7f9b57dceb80 10 client.28228 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:30.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.811+0000 7f9b57dceb80 20 client.28228 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:30.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.811+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/1314622841 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fdbba4aa40 0x55fdbba4cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.811+0000 7f9b57dceb80 4 client.28228 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:30.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.811+0000 7f9b57dceb80 1 -- 172.21.15.114:0/1314622841 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9b3c004df0 con 0x55fdbba4aa40 2024-05-27T07:53:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.818+0000 7f9b42ffd700 1 --2- 172.21.15.114:0/1314622841 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fdbba4aa40 0x55fdbba4cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.822+0000 7f9b42ffd700 1 --2- 172.21.15.114:0/1314622841 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fdbba4aa40 0x55fdbba4cf30 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d7cffd3bd0c31a7 server_cookie=5fc62d4018727efc in_seq=0 out_seq=0 2024-05-27T07:53:30.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.823+0000 7f9b26ff5700 10 client.28228 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:30.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.824+0000 7f9b26ff5700 1 -- 172.21.15.114:0/1314622841 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f9b3c004df0 con 0x55fdbba4aa40 2024-05-27T07:53:30.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.824+0000 7f9b26ff5700 10 client.28228 handle_command_reply: tid=0 2024-05-27T07:53:30.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.825+0000 7f9b57dceb80 1 client.28228 shutdown 2024-05-27T07:53:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.828+0000 7f9b57dceb80 1 -- 172.21.15.114:0/1314622841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba30f80 msgr2=0x55fdbba3a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.828+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/1314622841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba30f80 0x55fdbba3a7f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9b441098e0 tx=0x7f9b340106b0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.828+0000 7f9b57dceb80 1 -- 172.21.15.114:0/1314622841 shutdown_connections 2024-05-27T07:53:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.828+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/1314622841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbba36c50 0x55fdbba3ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.828+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/1314622841 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fdbba4aa40 0x55fdbba4cf30 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.828+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/1314622841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbb8b2e40 0x55fdbba3a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.828+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/1314622841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba30f80 0x55fdbba3a7f0 secure :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9b441098e0 tx=0x7f9b340106b0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.828+0000 7f9b26ff5700 0 client.28228 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.828+0000 7f9b57dceb80 1 -- 172.21.15.114:0/1314622841 >> 172.21.15.114:0/1314622841 conn(0x55fdbb90daf0 msgr2=0x55fdbba2fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.828+0000 7f9b57dceb80 1 -- 172.21.15.114:0/1314622841 shutdown_connections 2024-05-27T07:53:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.829+0000 7f9b57dceb80 1 -- 172.21.15.114:0/1314622841 wait complete. 2024-05-27T07:53:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.830+0000 7f9b57dceb80 20 client.28228 trim_cache size 0 max 16384 2024-05-27T07:53:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.858+0000 7f9b57dceb80 1 Processor -- start 2024-05-27T07:53:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.858+0000 7f9b57dceb80 1 -- start start 2024-05-27T07:53:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.859+0000 7f9b57dceb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba30a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.859+0000 7f9b57dceb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 0x55fdbba34720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.859+0000 7f9b57dceb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba39b60 0x55fdbba46850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.859+0000 7f9b57dceb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441c0f50 con 0x55fdbb8b2e40 2024-05-27T07:53:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.859+0000 7f9b57dceb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441bf6c0 con 0x55fdbba39b60 2024-05-27T07:53:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.859+0000 7f9b57dceb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441bd530 con 0x55fdbba36c50 2024-05-27T07:53:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.859+0000 7f9b42ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba30a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.860+0000 7f9b42ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba30a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52050/0 (socket says 172.21.15.114:52050) 2024-05-27T07:53:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.860+0000 7f9b42ffd700 1 -- 172.21.15.114:0/3924621499 learned_addr learned my addr 172.21.15.114:0/3924621499 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.860+0000 7f9b42ffd700 1 -- 172.21.15.114:0/3924621499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 msgr2=0x55fdbba34720 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:53:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.860+0000 7f9b42ffd700 1 --2- 172.21.15.114:0/3924621499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 0x55fdbba34720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.860+0000 7f9b42ffd700 1 -- 172.21.15.114:0/3924621499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba39b60 msgr2=0x55fdbba46850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:53:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.860+0000 7f9b42ffd700 1 --2- 172.21.15.114:0/3924621499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba39b60 0x55fdbba46850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.860+0000 7f9b42ffd700 1 -- 172.21.15.114:0/3924621499 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdbba2ced0 con 0x55fdbb8b2e40 2024-05-27T07:53:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.861+0000 7f9b42ffd700 1 --2- 172.21.15.114:0/3924621499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba30a40 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f9b38000c00 tx=0x7f9b380093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f7e3531efb8520 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.863+0000 7f9b28ff9700 1 -- 172.21.15.114:0/3924621499 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b380c2020 con 0x55fdbb8b2e40 2024-05-27T07:53:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.863+0000 7f9b28ff9700 1 -- 172.21.15.114:0/3924621499 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b380ad400 con 0x55fdbb8b2e40 2024-05-27T07:53:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.863+0000 7f9b28ff9700 1 -- 172.21.15.114:0/3924621499 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b380a9510 con 0x55fdbb8b2e40 2024-05-27T07:53:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.865+0000 7f9b57dceb80 1 -- 172.21.15.114:0/3924621499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 msgr2=0x55fdbba30a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.865+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/3924621499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba30a40 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f9b38000c00 tx=0x7f9b380093a0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.865+0000 7f9b57dceb80 1 -- 172.21.15.114:0/3924621499 shutdown_connections 2024-05-27T07:53:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.865+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/3924621499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba39b60 0x55fdbba46850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.865+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/3924621499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba30a40 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.865+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/3924621499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 0x55fdbba34720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.865+0000 7f9b57dceb80 1 -- 172.21.15.114:0/3924621499 >> 172.21.15.114:0/3924621499 conn(0x55fdbb90daf0 msgr2=0x55fdbba34010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:30.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.867+0000 7f9b57dceb80 1 -- 172.21.15.114:0/3924621499 shutdown_connections 2024-05-27T07:53:30.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.867+0000 7f9b57dceb80 1 -- 172.21.15.114:0/3924621499 wait complete. 2024-05-27T07:53:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.868+0000 7f9b57dceb80 1 -- start start 2024-05-27T07:53:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.868+0000 7f9b57dceb80 1 Processor -- start 2024-05-27T07:53:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.869+0000 7f9b57dceb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba432d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.870+0000 7f9b57dceb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 0x55fdbba52120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.870+0000 7f9b57dceb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba39b60 0x55fdbba527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.870+0000 7f9b57dceb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441bdb80 con 0x55fdbb8b2e40 2024-05-27T07:53:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.870+0000 7f9b57dceb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441beff0 con 0x55fdbba39b60 2024-05-27T07:53:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.870+0000 7f9b57dceb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441be250 con 0x55fdbba36c50 2024-05-27T07:53:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.870+0000 7f9b57dceb80 10 client.0 _ll_register_callbacks cb 0x55fdbb8ea920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:30.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.871+0000 7f9b427fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 0x55fdbba52120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.871+0000 7f9b437fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba39b60 0x55fdbba527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.871+0000 7f9b427fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 0x55fdbba52120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55050/0 (socket says 172.21.15.114:55050) 2024-05-27T07:53:30.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.871+0000 7f9b427fc700 1 -- 172.21.15.114:0/3124293186 learned_addr learned my addr 172.21.15.114:0/3124293186 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:30.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.872+0000 7f9b42ffd700 1 --2- 172.21.15.114:0/3124293186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.872+0000 7f9b42ffd700 1 -- 172.21.15.114:0/3124293186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 msgr2=0x55fdbba52120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:30.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.872+0000 7f9b42ffd700 1 --2- 172.21.15.114:0/3124293186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 0x55fdbba52120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.872+0000 7f9b42ffd700 1 -- 172.21.15.114:0/3124293186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba39b60 msgr2=0x55fdbba527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:30.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.872+0000 7f9b42ffd700 1 --2- 172.21.15.114:0/3124293186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba39b60 0x55fdbba527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.872+0000 7f9b42ffd700 1 -- 172.21.15.114:0/3124293186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdbba52f40 con 0x55fdbb8b2e40 2024-05-27T07:53:30.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.873+0000 7f9b42ffd700 1 --2- 172.21.15.114:0/3124293186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba432d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f9b380a62c0 tx=0x7f9b380107e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c4b9b715e90b20 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:30.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.873+0000 7f9b26ff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:30.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.874+0000 7f9b26ff5700 1 -- 172.21.15.114:0/3124293186 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b380c2070 con 0x55fdbb8b2e40 2024-05-27T07:53:30.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.874+0000 7f9b57dceb80 1 -- 172.21.15.114:0/3124293186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b380ad400 con 0x55fdbb8b2e40 2024-05-27T07:53:30.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.874+0000 7f9b26ff5700 1 -- 172.21.15.114:0/3124293186 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f9b380b1a20 con 0x55fdbb8b2e40 2024-05-27T07:53:30.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.874+0000 7f9b26ff5700 1 -- 172.21.15.114:0/3124293186 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f9b380b1c20 con 0x55fdbb8b2e40 2024-05-27T07:53:30.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.874+0000 7f9b57dceb80 10 client.18693 fetch_fsmap learned FSMap version 75 2024-05-27T07:53:30.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.874+0000 7f9b57dceb80 1 -- 172.21.15.114:0/3124293186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x55fdbbab5690 con 0x55fdbb8b2e40 2024-05-27T07:53:30.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.875+0000 7f9b26ff5700 1 -- 172.21.15.114:0/3124293186 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b380a82b0 con 0x55fdbb8b2e40 2024-05-27T07:53:30.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.875+0000 7f9b26ff5700 1 -- 172.21.15.114:0/3124293186 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 75) ==== 5910+0+0 (secure 0 0 0) 0x7f9b380a49c0 con 0x55fdbb8b2e40 2024-05-27T07:53:30.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.875+0000 7f9b57dceb80 10 client.18693 fetch_fsmap finished waiting for FSMap version 75 2024-05-27T07:53:30.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.875+0000 7f9b57dceb80 10 client.18693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:30.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.875+0000 7f9b57dceb80 20 client.18693 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:30.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.875+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/3124293186 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fdbbab5c60 0x55fdbbab8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:30.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.875+0000 7f9b57dceb80 4 client.18693 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:53:30.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.875+0000 7f9b57dceb80 1 -- 172.21.15.114:0/3124293186 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f9b380ad400 con 0x55fdbbab5c60 2024-05-27T07:53:30.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.879+0000 7f9b427fc700 1 --2- 172.21.15.114:0/3124293186 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fdbbab5c60 0x55fdbbab8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:30.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.898+0000 7f9b427fc700 1 --2- 172.21.15.114:0/3124293186 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fdbbab5c60 0x55fdbbab8080 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee7082327f2940d0 server_cookie=a84c9bf559f62da4 in_seq=0 out_seq=0 2024-05-27T07:53:30.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.898+0000 7f9b26ff5700 10 client.18693 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:30.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.903+0000 7f9b26ff5700 1 -- 172.21.15.114:0/3124293186 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f9b380ad400 con 0x55fdbbab5c60 2024-05-27T07:53:30.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.903+0000 7f9b26ff5700 10 client.18693 handle_command_reply: tid=0 2024-05-27T07:53:30.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.903+0000 7f9b57dceb80 1 client.18693 shutdown 2024-05-27T07:53:30.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.906+0000 7f9b57dceb80 1 -- 172.21.15.114:0/3124293186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 msgr2=0x55fdbba432d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:30.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.906+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/3124293186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba432d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f9b380a62c0 tx=0x7f9b380107e0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.910+0000 7f9b57dceb80 1 -- 172.21.15.114:0/3124293186 shutdown_connections 2024-05-27T07:53:30.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.910+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/3124293186 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fdbbab5c60 0x55fdbbab8080 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.910+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/3124293186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdbba39b60 0x55fdbba527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.910+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/3124293186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdbb8b2e40 0x55fdbba432d0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.910+0000 7f9b57dceb80 1 --2- 172.21.15.114:0/3124293186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fdbba36c50 0x55fdbba52120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.910+0000 7f9b57dceb80 1 -- 172.21.15.114:0/3124293186 >> 172.21.15.114:0/3124293186 conn(0x55fdbb90daf0 msgr2=0x55fdbba256f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:30.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.910+0000 7f9b26ff5700 0 client.18693 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:30.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.915+0000 7f9b57dceb80 1 -- 172.21.15.114:0/3124293186 shutdown_connections 2024-05-27T07:53:30.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.915+0000 7f9b57dceb80 1 -- 172.21.15.114:0/3124293186 wait complete. 2024-05-27T07:53:30.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.915+0000 7f9b57dceb80 20 client.18693 trim_cache size 0 max 16384 2024-05-27T07:53:30.918 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:53:30.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.919+0000 7f9b493b9700 1 -- 172.21.15.114:0/2066416833 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9b2c0b2aa0 msgr2=0x7f9b2c0b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:30.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.919+0000 7f9b493b9700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9b2c0b2aa0 0x7f9b2c0b4fb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c023000 tx=0x7f9b3c027040 comp rx=0 tx=0).stop 2024-05-27T07:53:30.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.919+0000 7f9b493b9700 1 -- 172.21.15.114:0/2066416833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b4410a980 msgr2=0x7f9b44075230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:30.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.919+0000 7f9b493b9700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b4410a980 0x7f9b44075230 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800b550 tx=0x7f9b3800b860 comp rx=0 tx=0).stop 2024-05-27T07:53:30.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.919+0000 7f9b493b9700 1 -- 172.21.15.114:0/2066416833 shutdown_connections 2024-05-27T07:53:30.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.919+0000 7f9b493b9700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b44110960 0x7f9b4406d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.919+0000 7f9b493b9700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b4410a980 0x7f9b44075230 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.919+0000 7f9b493b9700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9b2c0b2aa0 0x7f9b2c0b4fb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.919+0000 7f9b493b9700 1 --2- 172.21.15.114:0/2066416833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b4410b2d0 0x7f9b4406d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:30.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.919+0000 7f9b493b9700 1 -- 172.21.15.114:0/2066416833 >> 172.21.15.114:0/2066416833 conn(0x7f9b44077c70 msgr2=0x7f9b4407b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:30.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.921+0000 7f9b493b9700 1 -- 172.21.15.114:0/2066416833 shutdown_connections 2024-05-27T07:53:30.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:30.923+0000 7f9b493b9700 1 -- 172.21.15.114:0/2066416833 wait complete. 2024-05-27T07:53:31.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1309666 9.30 MB/sec execute 583 sec latency 67.559 ms 2024-05-27T07:53:31.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:31 smithi163 ceph-mon[36451]: pgmap v464: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 323 op/s 2024-05-27T07:53:31.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:31 smithi186 ceph-mon[36647]: pgmap v464: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 323 op/s 2024-05-27T07:53:31.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:31 smithi114 ceph-mon[32038]: pgmap v464: 129 pgs: 129 active+clean; 69 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 323 op/s 2024-05-27T07:53:32.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1311047 9.30 MB/sec execute 584 sec latency 16.292 ms 2024-05-27T07:53:32.864 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:53:32.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:53:33.129 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:33.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1312911 9.30 MB/sec execute 585 sec latency 55.755 ms 2024-05-27T07:53:33.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:33 smithi114 ceph-mon[32038]: pgmap v465: 129 pgs: 129 active+clean; 70 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 353 op/s 2024-05-27T07:53:33.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:33 smithi163 ceph-mon[36451]: pgmap v465: 129 pgs: 129 active+clean; 70 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 353 op/s 2024-05-27T07:53:33.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:33 smithi186 ceph-mon[36647]: pgmap v465: 129 pgs: 129 active+clean; 70 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 353 op/s 2024-05-27T07:53:34.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1314591 9.30 MB/sec execute 586 sec latency 26.964 ms 2024-05-27T07:53:34.705 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:34 smithi114 ceph-mon[32038]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:53:34.705 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:34 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:53:34.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:34 smithi163 ceph-mon[36451]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:53:34.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:34 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:53:34.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:34 smithi186 ceph-mon[36647]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:53:34.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:34 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:53:34.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.970+0000 7f965ca01700 1 -- 172.21.15.114:0/791448815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9658103300 msgr2=0x7f9658103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:34.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.970+0000 7f965ca01700 1 --2- 172.21.15.114:0/791448815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9658103300 0x7f9658103710 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f96480099c0 tx=0x7f9648009cd0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.971+0000 7f965ca01700 1 -- 172.21.15.114:0/791448815 shutdown_connections 2024-05-27T07:53:34.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.971+0000 7f965ca01700 1 --2- 172.21.15.114:0/791448815 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9658108580 0x7f965810fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.971+0000 7f965ca01700 1 --2- 172.21.15.114:0/791448815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9658103300 0x7f9658103710 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.971+0000 7f965ca01700 1 --2- 172.21.15.114:0/791448815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9658107c30 0x7f9658108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.971+0000 7f965ca01700 1 -- 172.21.15.114:0/791448815 >> 172.21.15.114:0/791448815 conn(0x7f9658078c30 msgr2=0x7f9658079070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:34.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.971+0000 7f965ca01700 1 -- 172.21.15.114:0/791448815 shutdown_connections 2024-05-27T07:53:34.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.971+0000 7f965ca01700 1 -- 172.21.15.114:0/791448815 wait complete. 2024-05-27T07:53:34.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.972+0000 7f965ca01700 1 Processor -- start 2024-05-27T07:53:34.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.972+0000 7f965ca01700 1 -- start start 2024-05-27T07:53:34.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.972+0000 7f965ca01700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9658103300 0x7f96581a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:34.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.972+0000 7f965ca01700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9658107c30 0x7f96581a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:34.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.972+0000 7f965ca01700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9658108580 0x7f965819aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:34.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.972+0000 7f965ca01700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9658113200 con 0x7f9658108580 2024-05-27T07:53:34.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.972+0000 7f965ca01700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9658113080 con 0x7f9658107c30 2024-05-27T07:53:34.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.972+0000 7f965ca01700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9658113380 con 0x7f9658103300 2024-05-27T07:53:34.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.972+0000 7f9656d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9658103300 0x7f96581a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:34.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.972+0000 7f9656d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9658103300 0x7f96581a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59002/0 (socket says 172.21.15.114:59002) 2024-05-27T07:53:34.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.972+0000 7f965759e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9658108580 0x7f965819aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:34.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.972+0000 7f9656d9d700 1 -- 172.21.15.114:0/2273120778 learned_addr learned my addr 172.21.15.114:0/2273120778 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:34.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.972+0000 7f965659c700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9658107c30 0x7f96581a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:34.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.973+0000 7f9656d9d700 1 -- 172.21.15.114:0/2273120778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9658107c30 msgr2=0x7f96581a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:34.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.973+0000 7f9656d9d700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9658107c30 0x7f96581a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.973+0000 7f9656d9d700 1 -- 172.21.15.114:0/2273120778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9658108580 msgr2=0x7f965819aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:34.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.973+0000 7f9656d9d700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9658108580 0x7f965819aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.973+0000 7f9656d9d700 1 -- 172.21.15.114:0/2273120778 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965819b1d0 con 0x7f9658103300 2024-05-27T07:53:34.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.973+0000 7f965659c700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9658107c30 0x7f96581a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:53:34.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.973+0000 7f9656d9d700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9658103300 0x7f96581a08b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9648005910 tx=0x7f96480045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:34.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.973+0000 7f963ffff700 1 -- 172.21.15.114:0/2273120778 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648026070 con 0x7f9658103300 2024-05-27T07:53:34.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.973+0000 7f963ffff700 1 -- 172.21.15.114:0/2273120778 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f964800f460 con 0x7f9658103300 2024-05-27T07:53:34.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.973+0000 7f965ca01700 1 -- 172.21.15.114:0/2273120778 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965819b460 con 0x7f9658103300 2024-05-27T07:53:34.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.973+0000 7f963ffff700 1 -- 172.21.15.114:0/2273120778 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648009e60 con 0x7f9658103300 2024-05-27T07:53:34.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.973+0000 7f965ca01700 1 -- 172.21.15.114:0/2273120778 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96581a7760 con 0x7f9658103300 2024-05-27T07:53:34.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.977+0000 7f963ffff700 1 -- 172.21.15.114:0/2273120778 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f9648029070 con 0x7f9658103300 2024-05-27T07:53:34.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.977+0000 7f963ffff700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f96400b29f0 0x7f96400b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:34.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.977+0000 7f966bc17b80 1 Processor -- start 2024-05-27T07:53:34.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.977+0000 7f965659c700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f96400b29f0 0x7f96400b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:34.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f963ffff700 1 -- 172.21.15.114:0/2273120778 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f96480e3330 con 0x7f9658103300 2024-05-27T07:53:34.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f966bc17b80 1 -- start start 2024-05-27T07:53:34.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f966bc17b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d7c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:34.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f966bc17b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d7ce40 0x55a310d76ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:34.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f966bc17b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d77000 0x55a310d77410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:34.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f966bc17b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a310ca5d10 con 0x55a310bf8ec0 2024-05-27T07:53:34.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f966bc17b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a310c958b0 con 0x55a310d77000 2024-05-27T07:53:34.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f966bc17b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a310c998f0 con 0x55a310d7ce40 2024-05-27T07:53:34.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f965659c700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f96400b29f0 0x7f96400b4f00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f96580ffd50 tx=0x7f964c009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f9656d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d7c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f965659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d7ce40 0x55a310d76ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f965659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d7ce40 0x55a310d76ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59018/0 (socket says 172.21.15.114:59018) 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f965659c700 1 -- 172.21.15.114:0/2788468441 learned_addr learned my addr 172.21.15.114:0/2788468441 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f965759e700 1 --2- 172.21.15.114:0/2788468441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d77000 0x55a310d77410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f9656d9d700 1 -- 172.21.15.114:0/2788468441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d7ce40 msgr2=0x55a310d76ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f9656d9d700 1 --2- 172.21.15.114:0/2788468441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d7ce40 0x55a310d76ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f9656d9d700 1 -- 172.21.15.114:0/2788468441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d77000 msgr2=0x55a310d77410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f9656d9d700 1 --2- 172.21.15.114:0/2788468441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d77000 0x55a310d77410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f9656d9d700 1 -- 172.21.15.114:0/2788468441 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a310be5950 con 0x55a310bf8ec0 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.978+0000 7f965659c700 1 --2- 172.21.15.114:0/2788468441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d7ce40 0x55a310d76ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.979+0000 7f9656d9d700 1 --2- 172.21.15.114:0/2788468441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d7c900 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f96480e0680 tx=0x7f96480a9c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90210ce8132ad272 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.979+0000 7f963d7fa700 1 -- 172.21.15.114:0/2788468441 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96480aa9a0 con 0x55a310bf8ec0 2024-05-27T07:53:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.979+0000 7f963d7fa700 1 -- 172.21.15.114:0/2788468441 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96480aa2a0 con 0x55a310bf8ec0 2024-05-27T07:53:34.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.979+0000 7f963d7fa700 1 -- 172.21.15.114:0/2788468441 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96480aa500 con 0x55a310bf8ec0 2024-05-27T07:53:34.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.980+0000 7f966bc17b80 1 -- 172.21.15.114:0/2788468441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 msgr2=0x55a310d7c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:34.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.980+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2788468441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d7c900 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f96480e0680 tx=0x7f96480a9c70 comp rx=0 tx=0).stop 2024-05-27T07:53:34.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.980+0000 7f966bc17b80 1 -- 172.21.15.114:0/2788468441 shutdown_connections 2024-05-27T07:53:34.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.980+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2788468441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d77000 0x55a310d77410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.980+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2788468441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d7c900 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.980+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2788468441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d7ce40 0x55a310d76ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.980+0000 7f966bc17b80 1 -- 172.21.15.114:0/2788468441 >> 172.21.15.114:0/2788468441 conn(0x55a310c53b70 msgr2=0x55a310d717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:34.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.980+0000 7f966bc17b80 1 -- 172.21.15.114:0/2788468441 shutdown_connections 2024-05-27T07:53:34.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.980+0000 7f966bc17b80 1 -- 172.21.15.114:0/2788468441 wait complete. 2024-05-27T07:53:34.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.980+0000 7f966bc17b80 1 -- start start 2024-05-27T07:53:34.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.980+0000 7f966bc17b80 1 Processor -- start 2024-05-27T07:53:34.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f966bc17b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d805f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:34.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f966bc17b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d77000 0x55a310d80b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:34.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f966bc17b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d7ce40 0x55a310d81070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f966bc17b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a310b9c020 con 0x55a310bf8ec0 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f966bc17b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a310c37840 con 0x55a310d7ce40 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f966bc17b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a310d87ae0 con 0x55a310d77000 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f966bc17b80 10 client.0 _ll_register_callbacks cb 0x55a310c309a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f9656d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d805f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f9656d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d805f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53606/0 (socket says 172.21.15.114:53606) 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f965659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d77000 0x55a310d80b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f9656d9d700 1 -- 172.21.15.114:0/2751520035 learned_addr learned my addr 172.21.15.114:0/2751520035 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f965759e700 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d7ce40 0x55a310d81070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f9656d9d700 1 -- 172.21.15.114:0/2751520035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d77000 msgr2=0x55a310d80b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f9656d9d700 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d77000 0x55a310d80b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f9656d9d700 1 -- 172.21.15.114:0/2751520035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d7ce40 msgr2=0x55a310d81070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f9656d9d700 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d7ce40 0x55a310d81070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f9656d9d700 1 -- 172.21.15.114:0/2751520035 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a310c34710 con 0x55a310bf8ec0 2024-05-27T07:53:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f965659c700 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d77000 0x55a310d80b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f965759e700 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d7ce40 0x55a310d81070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:53:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f9656d9d700 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d805f0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f96480e06e0 tx=0x7f964800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade695b0c3435616 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.981+0000 7f963aff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f963aff5700 1 -- 172.21.15.114:0/2751520035 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96480ae650 con 0x55a310bf8ec0 2024-05-27T07:53:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f963aff5700 1 -- 172.21.15.114:0/2751520035 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f96480aa500 con 0x55a310bf8ec0 2024-05-27T07:53:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f966bc17b80 1 -- 172.21.15.114:0/2751520035 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96480aa2a0 con 0x55a310bf8ec0 2024-05-27T07:53:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f963aff5700 1 -- 172.21.15.114:0/2751520035 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f96480b3070 con 0x55a310bf8ec0 2024-05-27T07:53:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f966bc17b80 10 client.18705 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f966bc17b80 1 -- 172.21.15.114:0/2751520035 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55a310d924e0 con 0x55a310bf8ec0 2024-05-27T07:53:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f963aff5700 1 -- 172.21.15.114:0/2751520035 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96480aa2a0 con 0x55a310bf8ec0 2024-05-27T07:53:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f963aff5700 1 -- 172.21.15.114:0/2751520035 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f96480ae1f0 con 0x55a310bf8ec0 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f966bc17b80 10 client.18705 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f966bc17b80 10 client.18705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f966bc17b80 20 client.18705 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a310d94a00 0x55a310d96e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f966bc17b80 4 client.18705 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.982+0000 7f966bc17b80 1 -- 172.21.15.114:0/2751520035 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a310d42a70 con 0x55a310d94a00 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.983+0000 7f965659c700 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a310d94a00 0x55a310d96e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.983+0000 7f963aff5700 10 client.18705 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.983+0000 7f965659c700 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a310d94a00 0x55a310d96e60 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=961985f972f1cc21 server_cookie=7fe577d464a82c5c in_seq=0 out_seq=0 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f963aff5700 1 -- 172.21.15.114:0/2751520035 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a310d42a70 con 0x55a310d94a00 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f963aff5700 10 client.18705 handle_command_reply: tid=0 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f966bc17b80 1 client.18705 shutdown 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f966bc17b80 1 -- 172.21.15.114:0/2751520035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 msgr2=0x55a310d805f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:34.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d805f0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f96480e06e0 tx=0x7f964800fcb0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f966bc17b80 1 -- 172.21.15.114:0/2751520035 shutdown_connections 2024-05-27T07:53:34.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a310d94a00 0x55a310d96e60 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d7ce40 0x55a310d81070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d805f0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2751520035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d77000 0x55a310d80b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:34.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f966bc17b80 1 -- 172.21.15.114:0/2751520035 >> 172.21.15.114:0/2751520035 conn(0x55a310c53b70 msgr2=0x55a310d75ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:34.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f963aff5700 0 client.18705 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:34.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f966bc17b80 1 -- 172.21.15.114:0/2751520035 shutdown_connections 2024-05-27T07:53:34.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f966bc17b80 1 -- 172.21.15.114:0/2751520035 wait complete. 2024-05-27T07:53:34.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:34.984+0000 7f966bc17b80 20 client.18705 trim_cache size 0 max 16384 2024-05-27T07:53:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.009+0000 7f966bc17b80 1 Processor -- start 2024-05-27T07:53:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.009+0000 7f966bc17b80 1 -- start start 2024-05-27T07:53:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.009+0000 7f966bc17b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d7e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.009+0000 7f966bc17b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d77000 0x55a310d826d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.009+0000 7f966bc17b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d7ce40 0x55a310d80eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.009+0000 7f966bc17b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f96581aece0 con 0x55a310bf8ec0 2024-05-27T07:53:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.009+0000 7f966bc17b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f96581af3b0 con 0x55a310d77000 2024-05-27T07:53:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.009+0000 7f966bc17b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f96581adb60 con 0x55a310d7ce40 2024-05-27T07:53:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.009+0000 7f9656d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d7e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.009+0000 7f9656d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d7e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53620/0 (socket says 172.21.15.114:53620) 2024-05-27T07:53:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.009+0000 7f9656d9d700 1 -- 172.21.15.114:0/3231963409 learned_addr learned my addr 172.21.15.114:0/3231963409 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.009+0000 7f965759e700 1 --2- 172.21.15.114:0/3231963409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d7ce40 0x55a310d80eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.009+0000 7f965659c700 1 --2- 172.21.15.114:0/3231963409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d77000 0x55a310d826d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.010+0000 7f9656d9d700 1 -- 172.21.15.114:0/3231963409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d7ce40 msgr2=0x55a310d80eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.010+0000 7f9656d9d700 1 --2- 172.21.15.114:0/3231963409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d7ce40 0x55a310d80eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.010+0000 7f9656d9d700 1 -- 172.21.15.114:0/3231963409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d77000 msgr2=0x55a310d826d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.010+0000 7f9656d9d700 1 --2- 172.21.15.114:0/3231963409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d77000 0x55a310d826d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.010+0000 7f9656d9d700 1 -- 172.21.15.114:0/3231963409 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a310d73100 con 0x55a310bf8ec0 2024-05-27T07:53:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.010+0000 7f965759e700 1 --2- 172.21.15.114:0/3231963409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d7ce40 0x55a310d80eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.010+0000 7f9656d9d700 1 --2- 172.21.15.114:0/3231963409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d7e260 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f96480aa230 tx=0x7f96480a9b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69907a44f2565b79 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.010+0000 7f963cff9700 1 -- 172.21.15.114:0/3231963409 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96480ae650 con 0x55a310bf8ec0 2024-05-27T07:53:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.010+0000 7f963cff9700 1 -- 172.21.15.114:0/3231963409 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96480ae160 con 0x55a310bf8ec0 2024-05-27T07:53:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.010+0000 7f963cff9700 1 -- 172.21.15.114:0/3231963409 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96480a5f60 con 0x55a310bf8ec0 2024-05-27T07:53:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.011+0000 7f966bc17b80 1 -- 172.21.15.114:0/3231963409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 msgr2=0x55a310d7e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.011+0000 7f966bc17b80 1 --2- 172.21.15.114:0/3231963409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d7e260 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f96480aa230 tx=0x7f96480a9b10 comp rx=0 tx=0).stop 2024-05-27T07:53:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.011+0000 7f966bc17b80 1 -- 172.21.15.114:0/3231963409 shutdown_connections 2024-05-27T07:53:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.011+0000 7f966bc17b80 1 --2- 172.21.15.114:0/3231963409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d7ce40 0x55a310d80eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.011+0000 7f966bc17b80 1 --2- 172.21.15.114:0/3231963409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d7e260 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.011+0000 7f966bc17b80 1 --2- 172.21.15.114:0/3231963409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d77000 0x55a310d826d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.011+0000 7f966bc17b80 1 -- 172.21.15.114:0/3231963409 >> 172.21.15.114:0/3231963409 conn(0x55a310c53b70 msgr2=0x55a310d7f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.011+0000 7f966bc17b80 1 -- 172.21.15.114:0/3231963409 shutdown_connections 2024-05-27T07:53:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.011+0000 7f966bc17b80 1 -- 172.21.15.114:0/3231963409 wait complete. 2024-05-27T07:53:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.011+0000 7f966bc17b80 1 -- start start 2024-05-27T07:53:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f966bc17b80 1 Processor -- start 2024-05-27T07:53:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f966bc17b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d8d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f966bc17b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d77000 0x55a310d888b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f966bc17b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d7ce40 0x55a310d88f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f966bc17b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f96581ad490 con 0x55a310bf8ec0 2024-05-27T07:53:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f9656d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d8d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f966bc17b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f96581ac0a0 con 0x55a310d7ce40 2024-05-27T07:53:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f966bc17b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f96581ac6f0 con 0x55a310d77000 2024-05-27T07:53:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f9656d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d8d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53632/0 (socket says 172.21.15.114:53632) 2024-05-27T07:53:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f9656d9d700 1 -- 172.21.15.114:0/2945070780 learned_addr learned my addr 172.21.15.114:0/2945070780 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f965659c700 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d77000 0x55a310d888b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f966bc17b80 10 client.0 _ll_register_callbacks cb 0x55a310c309a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f965759e700 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d7ce40 0x55a310d88f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f965659c700 1 -- 172.21.15.114:0/2945070780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d7ce40 msgr2=0x55a310d88f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f965659c700 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d7ce40 0x55a310d88f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f965659c700 1 -- 172.21.15.114:0/2945070780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 msgr2=0x55a310d8d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f965659c700 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d8d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f965659c700 1 -- 172.21.15.114:0/2945070780 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a310d89680 con 0x55a310d77000 2024-05-27T07:53:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.012+0000 7f9656d9d700 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d8d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f965659c700 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d77000 0x55a310d888b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f964c009a00 tx=0x7f964c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31040ad53cb717cb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f963aff5700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:53:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f963aff5700 1 -- 172.21.15.114:0/2945070780 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c02c580 con 0x55a310d77000 2024-05-27T07:53:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f966bc17b80 1 -- 172.21.15.114:0/2945070780 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96480ae160 con 0x55a310d77000 2024-05-27T07:53:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f963aff5700 1 -- 172.21.15.114:0/2945070780 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f964c02c720 con 0x55a310d77000 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f963aff5700 1 -- 172.21.15.114:0/2945070780 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f964c03d050 con 0x55a310d77000 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f966bc17b80 10 client.28249 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f966bc17b80 1 -- 172.21.15.114:0/2945070780 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55a310dff5b0 con 0x55a310d77000 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f963aff5700 1 -- 172.21.15.114:0/2945070780 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96480ae160 con 0x55a310d77000 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f963aff5700 1 -- 172.21.15.114:0/2945070780 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f964c012b70 con 0x55a310d77000 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f966bc17b80 10 client.28249 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f966bc17b80 10 client.28249 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f966bc17b80 20 client.28249 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a310dffc90 0x55a310e020b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f966bc17b80 4 client.28249 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.013+0000 7f966bc17b80 1 -- 172.21.15.114:0/2945070780 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a310ca7c60 con 0x55a310dffc90 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.014+0000 7f9656d9d700 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a310dffc90 0x55a310e020b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.014+0000 7f963aff5700 10 client.28249 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:35.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.014+0000 7f9656d9d700 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a310dffc90 0x55a310e020b0 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7caaccff704adc5 server_cookie=415ba9e54c7094d5 in_seq=0 out_seq=0 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.015+0000 7f963aff5700 1 -- 172.21.15.114:0/2945070780 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a310ca7c60 con 0x55a310dffc90 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.015+0000 7f963aff5700 10 client.28249 handle_command_reply: tid=0 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.015+0000 7f966bc17b80 1 client.28249 shutdown 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.015+0000 7f966bc17b80 1 -- 172.21.15.114:0/2945070780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d77000 msgr2=0x55a310d888b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.015+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d77000 0x55a310d888b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f964c009a00 tx=0x7f964c0047c0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f966bc17b80 1 -- 172.21.15.114:0/2945070780 shutdown_connections 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a310dffc90 0x55a310e020b0 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a310d7ce40 0x55a310d88f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a310bf8ec0 0x55a310d8d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f966bc17b80 1 --2- 172.21.15.114:0/2945070780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a310d77000 0x55a310d888b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f966bc17b80 1 -- 172.21.15.114:0/2945070780 >> 172.21.15.114:0/2945070780 conn(0x55a310c53b70 msgr2=0x55a310d75ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f963aff5700 0 client.28249 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f966bc17b80 1 -- 172.21.15.114:0/2945070780 shutdown_connections 2024-05-27T07:53:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f966bc17b80 1 -- 172.21.15.114:0/2945070780 wait complete. 2024-05-27T07:53:35.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f966bc17b80 20 client.28249 trim_cache size 0 max 16384 2024-05-27T07:53:35.022 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:53:35.022 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:53:35.022 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "21231218-e5d9-4dce-8693-3a6a51a54cf0", 2024-05-27T07:53:35.022 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:53:35.022 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:53:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f965ca01700 1 -- 172.21.15.114:0/2273120778 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f96400b29f0 msgr2=0x7f96400b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f965ca01700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f96400b29f0 0x7f96400b4f00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f96580ffd50 tx=0x7f964c009260 comp rx=0 tx=0).stop 2024-05-27T07:53:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f965ca01700 1 -- 172.21.15.114:0/2273120778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9658103300 msgr2=0x7f96581a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f965ca01700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9658103300 0x7f96581a08b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9648005910 tx=0x7f96480045e0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f965ca01700 1 -- 172.21.15.114:0/2273120778 shutdown_connections 2024-05-27T07:53:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f965ca01700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9658108580 0x7f965819aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.016+0000 7f965ca01700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9658103300 0x7f96581a08b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.017+0000 7f965ca01700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f96400b29f0 0x7f96400b4f00 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.017+0000 7f965ca01700 1 --2- 172.21.15.114:0/2273120778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9658107c30 0x7f96581a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:35.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.017+0000 7f965ca01700 1 -- 172.21.15.114:0/2273120778 >> 172.21.15.114:0/2273120778 conn(0x7f9658078c30 msgr2=0x7f965806c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:35.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.017+0000 7f965ca01700 1 -- 172.21.15.114:0/2273120778 shutdown_connections 2024-05-27T07:53:35.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:35.017+0000 7f965ca01700 1 -- 172.21.15.114:0/2273120778 wait complete. 2024-05-27T07:53:35.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1316344 9.30 MB/sec execute 587 sec latency 37.149 ms 2024-05-27T07:53:35.686 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:35 smithi114 ceph-mon[32038]: pgmap v466: 129 pgs: 129 active+clean; 70 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 2.3 MiB/s wr, 269 op/s 2024-05-27T07:53:35.686 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:35 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:53:35.686 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:35 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:35.686 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:35 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:53:35.686 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:35 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:35.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:53:35.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:35 smithi163 ceph-mon[36451]: pgmap v466: 129 pgs: 129 active+clean; 70 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 2.3 MiB/s wr, 269 op/s 2024-05-27T07:53:35.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:35 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:53:35.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:35 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:35.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:35 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:53:35.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:35 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:35.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:35 smithi186 ceph-mon[36647]: pgmap v466: 129 pgs: 129 active+clean; 70 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 2.3 MiB/s wr, 269 op/s 2024-05-27T07:53:35.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:35 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:53:35.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:35 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:35.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:35 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:53:35.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:35 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:35.977 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:36.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1317984 9.30 MB/sec execute 588 sec latency 21.545 ms 2024-05-27T07:53:37.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1319747 9.30 MB/sec execute 589 sec latency 34.700 ms 2024-05-27T07:53:37.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:37 smithi114 ceph-mon[32038]: pgmap v467: 129 pgs: 129 active+clean; 70 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 314 op/s 2024-05-27T07:53:37.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:37 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:53:37.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:37 smithi163 ceph-mon[36451]: pgmap v467: 129 pgs: 129 active+clean; 70 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 314 op/s 2024-05-27T07:53:37.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:37 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:53:37.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:37 smithi186 ceph-mon[36647]: pgmap v467: 129 pgs: 129 active+clean; 70 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 314 op/s 2024-05-27T07:53:37.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:37 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:53:38.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.254+0000 7f1a3c4f6700 1 -- 172.21.15.114:0/1649295056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a341030d0 msgr2=0x7f1a34111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.254+0000 7f1a3c4f6700 1 --2- 172.21.15.114:0/1649295056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a341030d0 0x7f1a34111160 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c00b0a0 tx=0x7f1a2c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.254+0000 7f1a3c4f6700 1 -- 172.21.15.114:0/1649295056 shutdown_connections 2024-05-27T07:53:38.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.254+0000 7f1a3c4f6700 1 --2- 172.21.15.114:0/1649295056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a341030d0 0x7f1a34111160 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.254+0000 7f1a3c4f6700 1 --2- 172.21.15.114:0/1649295056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a341092f0 0x7f1a34109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.254+0000 7f1a3c4f6700 1 --2- 172.21.15.114:0/1649295056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a34102780 0x7f1a34102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.255+0000 7f1a3c4f6700 1 -- 172.21.15.114:0/1649295056 >> 172.21.15.114:0/1649295056 conn(0x7f1a340fe530 msgr2=0x7f1a34100980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:38.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.255+0000 7f1a3c4f6700 1 -- 172.21.15.114:0/1649295056 shutdown_connections 2024-05-27T07:53:38.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.255+0000 7f1a3c4f6700 1 -- 172.21.15.114:0/1649295056 wait complete. 2024-05-27T07:53:38.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.255+0000 7f1a3c4f6700 1 Processor -- start 2024-05-27T07:53:38.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.255+0000 7f1a3c4f6700 1 -- start start 2024-05-27T07:53:38.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3c4f6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a34102780 0x7f1a3419ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3c4f6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a341030d0 0x7f1a3419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3c4f6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a341092f0 0x7f1a3419f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3c4f6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1a34114590 con 0x7f1a341030d0 2024-05-27T07:53:38.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3c4f6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1a34114410 con 0x7f1a341092f0 2024-05-27T07:53:38.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3c4f6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1a34114710 con 0x7f1a34102780 2024-05-27T07:53:38.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3a292700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a34102780 0x7f1a3419ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3a292700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a34102780 0x7f1a3419ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59070/0 (socket says 172.21.15.114:59070) 2024-05-27T07:53:38.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3a292700 1 -- 172.21.15.114:0/4068017174 learned_addr learned my addr 172.21.15.114:0/4068017174 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:38.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a39a91700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a341030d0 0x7f1a3419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3aa93700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a341092f0 0x7f1a3419f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3a292700 1 -- 172.21.15.114:0/4068017174 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a341092f0 msgr2=0x7f1a3419f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3a292700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a341092f0 0x7f1a3419f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3a292700 1 -- 172.21.15.114:0/4068017174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a341030d0 msgr2=0x7f1a3419b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3a292700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a341030d0 0x7f1a3419b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3a292700 1 -- 172.21.15.114:0/4068017174 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3419fce0 con 0x7f1a34102780 2024-05-27T07:53:38.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.256+0000 7f1a3a292700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a34102780 0x7f1a3419ae60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000e940 tx=0x7f1a3000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:38.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.257+0000 7f1a2b7fe700 1 -- 172.21.15.114:0/4068017174 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a30004220 con 0x7f1a34102780 2024-05-27T07:53:38.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.257+0000 7f1a3c4f6700 1 -- 172.21.15.114:0/4068017174 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a3419ffd0 con 0x7f1a34102780 2024-05-27T07:53:38.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.257+0000 7f1a2b7fe700 1 -- 172.21.15.114:0/4068017174 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a300043c0 con 0x7f1a34102780 2024-05-27T07:53:38.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.257+0000 7f1a2b7fe700 1 -- 172.21.15.114:0/4068017174 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a30003c60 con 0x7f1a34102780 2024-05-27T07:53:38.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.257+0000 7f1a3c4f6700 1 -- 172.21.15.114:0/4068017174 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a341a78f0 con 0x7f1a34102780 2024-05-27T07:53:38.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.260+0000 7f1a2b7fe700 1 -- 172.21.15.114:0/4068017174 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f1a30018430 con 0x7f1a34102780 2024-05-27T07:53:38.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.260+0000 7f1a4af0bb80 1 Processor -- start 2024-05-27T07:53:38.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.261+0000 7f1a4af0bb80 1 -- start start 2024-05-27T07:53:38.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.261+0000 7f1a4af0bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 0x56188bc1ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.261+0000 7f1a4af0bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1b070 0x56188bc14be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.261+0000 7f1a4af0bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc15120 0x56188bc15530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.261+0000 7f1a4af0bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56188bb42d70 con 0x56188ba95e40 2024-05-27T07:53:38.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.261+0000 7f1a4af0bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56188bb32910 con 0x56188bc1b070 2024-05-27T07:53:38.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.261+0000 7f1a4af0bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56188bb36950 con 0x56188bc15120 2024-05-27T07:53:38.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.261+0000 7f1a3aa93700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 0x56188bc1ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.261+0000 7f1a3aa93700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 0x56188bc1ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53670/0 (socket says 172.21.15.114:53670) 2024-05-27T07:53:38.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.261+0000 7f1a3aa93700 1 -- 172.21.15.114:0/3670175979 learned_addr learned my addr 172.21.15.114:0/3670175979 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:38.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.261+0000 7f1a39a91700 1 --2- 172.21.15.114:0/3670175979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc15120 0x56188bc15530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.261+0000 7f1a3a292700 1 --2- 172.21.15.114:0/3670175979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1b070 0x56188bc14be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.261+0000 7f1a2b7fe700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1a200b29f0 0x7f1a200b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.262+0000 7f1a39a91700 1 -- 172.21.15.114:0/3670175979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1b070 msgr2=0x56188bc14be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:38.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.262+0000 7f1a39a91700 1 --2- 172.21.15.114:0/3670175979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1b070 0x56188bc14be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.262+0000 7f1a39a91700 1 -- 172.21.15.114:0/3670175979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 msgr2=0x56188bc1ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:38.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.262+0000 7f1a39a91700 1 --2- 172.21.15.114:0/3670175979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 0x56188bc1ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.262+0000 7f1a39a91700 1 -- 172.21.15.114:0/3670175979 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56188badbfb0 con 0x56188bc15120 2024-05-27T07:53:38.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.262+0000 7f1a3aa93700 1 --2- 172.21.15.114:0/3670175979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 0x56188bc1ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:38.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.262+0000 7f1a3aa93700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1a200b29f0 0x7f1a200b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.262+0000 7f1a2b7fe700 1 -- 172.21.15.114:0/4068017174 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f1a300e2be0 con 0x7f1a34102780 2024-05-27T07:53:38.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.262+0000 7f1a39a91700 1 --2- 172.21.15.114:0/3670175979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc15120 0x56188bc15530 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1a3406cce0 tx=0x7f1a24009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dd9d7361a197987 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:38.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.262+0000 7f1a3aa93700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1a200b29f0 0x7f1a200b4f00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c009900 tx=0x7f1a2c0099e0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:38.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.262+0000 7f1a28d47700 1 -- 172.21.15.114:0/3670175979 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24011c80 con 0x56188bc15120 2024-05-27T07:53:38.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.262+0000 7f1a28d47700 1 -- 172.21.15.114:0/3670175979 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a24011e20 con 0x56188bc15120 2024-05-27T07:53:38.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.262+0000 7f1a28d47700 1 -- 172.21.15.114:0/3670175979 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24005280 con 0x56188bc15120 2024-05-27T07:53:38.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.263+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/3670175979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc15120 msgr2=0x56188bc15530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.263+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/3670175979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc15120 0x56188bc15530 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1a3406cce0 tx=0x7f1a24009f60 comp rx=0 tx=0).stop 2024-05-27T07:53:38.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.263+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/3670175979 shutdown_connections 2024-05-27T07:53:38.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.263+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/3670175979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc15120 0x56188bc15530 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.263+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/3670175979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 0x56188bc1ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.263+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/3670175979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1b070 0x56188bc14be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.263+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/3670175979 >> 172.21.15.114:0/3670175979 conn(0x56188ba98770 msgr2=0x56188bc0fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:38.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.263+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/3670175979 shutdown_connections 2024-05-27T07:53:38.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.263+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/3670175979 wait complete. 2024-05-27T07:53:38.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.264+0000 7f1a4af0bb80 1 -- start start 2024-05-27T07:53:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.264+0000 7f1a4af0bb80 1 Processor -- start 2024-05-27T07:53:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.264+0000 7f1a4af0bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188ba95e40 0x56188bc1e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.264+0000 7f1a4af0bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc15120 0x56188bc1ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.264+0000 7f1a4af0bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc1b070 0x56188bc1f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.264+0000 7f1a4af0bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56188ba38fa0 con 0x56188bc15120 2024-05-27T07:53:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.264+0000 7f1a4af0bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56188ba9ada0 con 0x56188ba95e40 2024-05-27T07:53:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.264+0000 7f1a4af0bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56188bc21b60 con 0x56188bc1b070 2024-05-27T07:53:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.264+0000 7f1a4af0bb80 10 client.0 _ll_register_callbacks cb 0x56188bac95b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.264+0000 7f1a39a91700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc15120 0x56188bc1ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.264+0000 7f1a39a91700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc15120 0x56188bc1ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53682/0 (socket says 172.21.15.114:53682) 2024-05-27T07:53:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.264+0000 7f1a39a91700 1 -- 172.21.15.114:0/618247633 learned_addr learned my addr 172.21.15.114:0/618247633 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.264+0000 7f1a3aa93700 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc1b070 0x56188bc1f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a3a292700 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188ba95e40 0x56188bc1e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a39a91700 1 -- 172.21.15.114:0/618247633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc1b070 msgr2=0x56188bc1f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a39a91700 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc1b070 0x56188bc1f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a39a91700 1 -- 172.21.15.114:0/618247633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188ba95e40 msgr2=0x56188bc1e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a39a91700 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188ba95e40 0x56188bc1e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a39a91700 1 -- 172.21.15.114:0/618247633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56188ba828d0 con 0x56188bc15120 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a3a292700 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188ba95e40 0x56188bc1e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a39a91700 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc15120 0x56188bc1ede0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f1a24004900 tx=0x7f1a24004930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192b3e39e758063c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a1dffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a1dffb700 1 -- 172.21.15.114:0/618247633 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24004260 con 0x56188bc15120 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a1dffb700 1 -- 172.21.15.114:0/618247633 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f1a24004400 con 0x56188bc15120 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/618247633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a24011e20 con 0x56188bc15120 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a1dffb700 1 -- 172.21.15.114:0/618247633 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f1a24030050 con 0x56188bc15120 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a4af0bb80 10 client.18723 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/618247633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x56188baec760 con 0x56188bc15120 2024-05-27T07:53:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.265+0000 7f1a1dffb700 1 -- 172.21.15.114:0/618247633 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a24011e20 con 0x56188bc15120 2024-05-27T07:53:38.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.267+0000 7f1a1dffb700 1 -- 172.21.15.114:0/618247633 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f1a24018070 con 0x56188bc15120 2024-05-27T07:53:38.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.267+0000 7f1a4af0bb80 10 client.18723 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:38.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.267+0000 7f1a4af0bb80 10 client.18723 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:38.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.267+0000 7f1a4af0bb80 20 client.18723 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:38.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.267+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56188bc2ec50 0x56188bc31120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.267+0000 7f1a4af0bb80 4 client.18723 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:38.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.267+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/618247633 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56188bb44cc0 con 0x56188bc2ec50 2024-05-27T07:53:38.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.267+0000 7f1a3a292700 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56188bc2ec50 0x56188bc31120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.267+0000 7f1a1dffb700 10 client.18723 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:38.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.267+0000 7f1a3a292700 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56188bc2ec50 0x56188bc31120 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=423c34c9746b67a3 server_cookie=25dc18b29245742 in_seq=0 out_seq=0 2024-05-27T07:53:38.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.267+0000 7f1a1dffb700 1 -- 172.21.15.114:0/618247633 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f1a3008ae00 con 0x56188bc2ec50 2024-05-27T07:53:38.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.267+0000 7f1a1dffb700 10 client.18723 handle_command_reply: tid=0 2024-05-27T07:53:38.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.267+0000 7f1a4af0bb80 1 client.18723 shutdown 2024-05-27T07:53:38.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.268+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/618247633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc15120 msgr2=0x56188bc1ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.268+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc15120 0x56188bc1ede0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f1a24004900 tx=0x7f1a24004930 comp rx=0 tx=0).stop 2024-05-27T07:53:38.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.268+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/618247633 shutdown_connections 2024-05-27T07:53:38.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.268+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc1b070 0x56188bc1f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.268+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56188bc2ec50 0x56188bc31120 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.268+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188ba95e40 0x56188bc1e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.268+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/618247633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc15120 0x56188bc1ede0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.268+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/618247633 >> 172.21.15.114:0/618247633 conn(0x56188ba98770 msgr2=0x56188bc13bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:38.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.268+0000 7f1a1dffb700 0 client.18723 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:38.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.268+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/618247633 shutdown_connections 2024-05-27T07:53:38.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.268+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/618247633 wait complete. 2024-05-27T07:53:38.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.268+0000 7f1a4af0bb80 20 client.18723 trim_cache size 0 max 16384 2024-05-27T07:53:38.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a4af0bb80 1 Processor -- start 2024-05-27T07:53:38.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a4af0bb80 1 -- start start 2024-05-27T07:53:38.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a4af0bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 0x56188bc14be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a4af0bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1b070 0x56188bc18ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a4af0bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc1c680 0x56188bc2ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a4af0bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1a341af070 con 0x56188ba95e40 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a4af0bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1a341af740 con 0x56188bc1b070 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a4af0bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1a341adef0 con 0x56188bc1c680 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a3a292700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 0x56188bc14be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a3aa93700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc1c680 0x56188bc2ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a3a292700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 0x56188bc14be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53698/0 (socket says 172.21.15.114:53698) 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a3a292700 1 -- 172.21.15.114:0/801501974 learned_addr learned my addr 172.21.15.114:0/801501974 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a39a91700 1 --2- 172.21.15.114:0/801501974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1b070 0x56188bc18ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a3a292700 1 -- 172.21.15.114:0/801501974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc1c680 msgr2=0x56188bc2ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a3a292700 1 --2- 172.21.15.114:0/801501974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc1c680 0x56188bc2ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a3a292700 1 -- 172.21.15.114:0/801501974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1b070 msgr2=0x56188bc18ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:38.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a3a292700 1 --2- 172.21.15.114:0/801501974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1b070 0x56188bc18ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a3a292700 1 -- 172.21.15.114:0/801501974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56188bc15a90 con 0x56188ba95e40 2024-05-27T07:53:38.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.291+0000 7f1a3a292700 1 --2- 172.21.15.114:0/801501974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 0x56188bc14be0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f1a300a6d30 tx=0x7f1a300cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4340d42a9d505f5e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:38.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.292+0000 7f1a1ffff700 1 -- 172.21.15.114:0/801501974 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a300a9410 con 0x56188ba95e40 2024-05-27T07:53:38.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.292+0000 7f1a1ffff700 1 -- 172.21.15.114:0/801501974 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56188bb44cc0 con 0x56188ba95e40 2024-05-27T07:53:38.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.292+0000 7f1a1ffff700 1 -- 172.21.15.114:0/801501974 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a30004570 con 0x56188ba95e40 2024-05-27T07:53:38.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.293+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/801501974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 msgr2=0x56188bc14be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.293+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/801501974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 0x56188bc14be0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f1a300a6d30 tx=0x7f1a300cc040 comp rx=0 tx=0).stop 2024-05-27T07:53:38.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.293+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/801501974 shutdown_connections 2024-05-27T07:53:38.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.293+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/801501974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188bc1c680 0x56188bc2ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.293+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/801501974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188ba95e40 0x56188bc14be0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.293+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/801501974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1b070 0x56188bc18ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.293+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/801501974 >> 172.21.15.114:0/801501974 conn(0x56188ba98770 msgr2=0x56188bc18490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:38.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.293+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/801501974 shutdown_connections 2024-05-27T07:53:38.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.293+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/801501974 wait complete. 2024-05-27T07:53:38.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a4af0bb80 1 -- start start 2024-05-27T07:53:38.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a4af0bb80 1 Processor -- start 2024-05-27T07:53:38.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a4af0bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188ba95e40 0x56188bc26ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a4af0bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc1b070 0x56188bc27530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a4af0bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1c680 0x56188bc35ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a4af0bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1a341ad820 con 0x56188bc1b070 2024-05-27T07:53:38.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a4af0bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1a341ac430 con 0x56188bc1c680 2024-05-27T07:53:38.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a4af0bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1a341aca80 con 0x56188ba95e40 2024-05-27T07:53:38.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a4af0bb80 10 client.0 _ll_register_callbacks cb 0x56188bac95b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:38.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a39a91700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc1b070 0x56188bc27530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a3a292700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188ba95e40 0x56188bc26ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a39a91700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc1b070 0x56188bc27530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53710/0 (socket says 172.21.15.114:53710) 2024-05-27T07:53:38.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a39a91700 1 -- 172.21.15.114:0/3312292097 learned_addr learned my addr 172.21.15.114:0/3312292097 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.294+0000 7f1a3aa93700 1 --2- 172.21.15.114:0/3312292097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1c680 0x56188bc35ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a39a91700 1 -- 172.21.15.114:0/3312292097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188ba95e40 msgr2=0x56188bc26ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a39a91700 1 --2- 172.21.15.114:0/3312292097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188ba95e40 0x56188bc26ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a39a91700 1 -- 172.21.15.114:0/3312292097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1c680 msgr2=0x56188bc35ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a39a91700 1 --2- 172.21.15.114:0/3312292097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1c680 0x56188bc35ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a39a91700 1 -- 172.21.15.114:0/3312292097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56188bc360a0 con 0x56188bc1b070 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a3a292700 1 --2- 172.21.15.114:0/3312292097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188ba95e40 0x56188bc26ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a39a91700 1 --2- 172.21.15.114:0/3312292097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc1b070 0x56188bc27530 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009bd0 tx=0x7f1a24009300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905e65fd897c9368 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a1dffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a1dffb700 1 -- 172.21.15.114:0/3312292097 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24004a50 con 0x56188bc1b070 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a1dffb700 1 -- 172.21.15.114:0/3312292097 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f1a24004470 con 0x56188bc1b070 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/3312292097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56188bb44cc0 con 0x56188bc1b070 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a1dffb700 1 -- 172.21.15.114:0/3312292097 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f1a24042050 con 0x56188bc1b070 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a4af0bb80 10 client.18729 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/3312292097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x56188bc99020 con 0x56188bc1b070 2024-05-27T07:53:38.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.295+0000 7f1a1dffb700 1 -- 172.21.15.114:0/3312292097 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56188bb44cc0 con 0x56188bc1b070 2024-05-27T07:53:38.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.296+0000 7f1a1dffb700 1 -- 172.21.15.114:0/3312292097 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f1a24018040 con 0x56188bc1b070 2024-05-27T07:53:38.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.296+0000 7f1a4af0bb80 10 client.18729 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:38.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.296+0000 7f1a4af0bb80 10 client.18729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:38.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.296+0000 7f1a4af0bb80 20 client.18729 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:38.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.296+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/3312292097 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56188bc99600 0x56188bc9ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.296+0000 7f1a4af0bb80 4 client.18729 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:53:38.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.296+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/3312292097 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56188bb3e4a0 con 0x56188bc99600 2024-05-27T07:53:38.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.296+0000 7f1a3a292700 1 --2- 172.21.15.114:0/3312292097 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56188bc99600 0x56188bc9ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.302 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1321487 9.30 MB/sec execute 590 sec latency 14.716 ms 2024-05-27T07:53:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.296+0000 7f1a1dffb700 10 client.18729 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.296+0000 7f1a3a292700 1 --2- 172.21.15.114:0/3312292097 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56188bc99600 0x56188bc9ba20 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=651870283a769381 server_cookie=ab1840e3de3a2a99 in_seq=0 out_seq=0 2024-05-27T07:53:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.297+0000 7f1a1dffb700 1 -- 172.21.15.114:0/3312292097 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56188bb3e4a0 con 0x56188bc99600 2024-05-27T07:53:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.297+0000 7f1a1dffb700 10 client.18729 handle_command_reply: tid=0 2024-05-27T07:53:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.297+0000 7f1a4af0bb80 1 client.18729 shutdown 2024-05-27T07:53:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.298+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/3312292097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc1b070 msgr2=0x56188bc27530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.298+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/3312292097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc1b070 0x56188bc27530 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009bd0 tx=0x7f1a24009300 comp rx=0 tx=0).stop 2024-05-27T07:53:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.298+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/3312292097 shutdown_connections 2024-05-27T07:53:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.298+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/3312292097 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56188bc99600 0x56188bc9ba20 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.298+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/3312292097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56188bc1c680 0x56188bc35ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.298+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/3312292097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56188ba95e40 0x56188bc26ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.298+0000 7f1a4af0bb80 1 --2- 172.21.15.114:0/3312292097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56188bc1b070 0x56188bc27530 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.298+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/3312292097 >> 172.21.15.114:0/3312292097 conn(0x56188ba98770 msgr2=0x56188bc1b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.298+0000 7f1a1dffb700 0 client.18729 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.298+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/3312292097 shutdown_connections 2024-05-27T07:53:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.298+0000 7f1a4af0bb80 1 -- 172.21.15.114:0/3312292097 wait complete. 2024-05-27T07:53:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.298+0000 7f1a4af0bb80 20 client.18729 trim_cache size 0 max 16384 2024-05-27T07:53:38.304 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:53:38.304 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:53:38.304 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:53:38.305 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:53:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.299+0000 7f1a3c4f6700 1 -- 172.21.15.114:0/4068017174 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1a200b29f0 msgr2=0x7f1a200b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.299+0000 7f1a3c4f6700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1a200b29f0 0x7f1a200b4f00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c009900 tx=0x7f1a2c0099e0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.299+0000 7f1a3c4f6700 1 -- 172.21.15.114:0/4068017174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a34102780 msgr2=0x7f1a3419ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.299+0000 7f1a3c4f6700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a34102780 0x7f1a3419ae60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000e940 tx=0x7f1a3000ec50 comp rx=0 tx=0).stop 2024-05-27T07:53:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.299+0000 7f1a3c4f6700 1 -- 172.21.15.114:0/4068017174 shutdown_connections 2024-05-27T07:53:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.299+0000 7f1a3c4f6700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f1a200b29f0 0x7f1a200b4f00 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.299+0000 7f1a3c4f6700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a341092f0 0x7f1a3419f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.299+0000 7f1a3c4f6700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a34102780 0x7f1a3419ae60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.299+0000 7f1a3c4f6700 1 --2- 172.21.15.114:0/4068017174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a341030d0 0x7f1a3419b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.299+0000 7f1a3c4f6700 1 -- 172.21.15.114:0/4068017174 >> 172.21.15.114:0/4068017174 conn(0x7f1a340fe530 msgr2=0x7f1a34107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:38.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.299+0000 7f1a3c4f6700 1 -- 172.21.15.114:0/4068017174 shutdown_connections 2024-05-27T07:53:38.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:38.299+0000 7f1a3c4f6700 1 -- 172.21.15.114:0/4068017174 wait complete. 2024-05-27T07:53:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.713+0000 7f37889c7640 1 -- 172.21.15.114:0/1894096027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f378410e6c0 msgr2=0x7f3784110ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.713+0000 7f37889c7640 1 --2- 172.21.15.114:0/1894096027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f378410e6c0 0x7f3784110ae0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f377c009ca0 tx=0x7f377c02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.713+0000 7f37889c7640 1 -- 172.21.15.114:0/1894096027 shutdown_connections 2024-05-27T07:53:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.713+0000 7f37889c7640 1 --2- 172.21.15.114:0/1894096027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f378410e6c0 0x7f3784110ae0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.713+0000 7f37889c7640 1 --2- 172.21.15.114:0/1894096027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37841021c0 0x7f37841025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.713+0000 7f37889c7640 1 --2- 172.21.15.114:0/1894096027 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3784109d10 0x7f3784101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.713+0000 7f37889c7640 1 -- 172.21.15.114:0/1894096027 >> 172.21.15.114:0/1894096027 conn(0x7f37840779d0 msgr2=0x7f3784077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.713+0000 7f37889c7640 1 -- 172.21.15.114:0/1894096027 shutdown_connections 2024-05-27T07:53:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.713+0000 7f37889c7640 1 -- 172.21.15.114:0/1894096027 wait complete. 2024-05-27T07:53:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f37889c7640 1 Processor -- start 2024-05-27T07:53:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f37889c7640 1 -- start start 2024-05-27T07:53:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f37889c7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37841021c0 0x7f37841a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f37889c7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3784109d10 0x7f37841a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f37889c7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f378410e6c0 0x7f378419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f37889c7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3784113e80 con 0x7f37841021c0 2024-05-27T07:53:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f37889c7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3784113d00 con 0x7f3784109d10 2024-05-27T07:53:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f37889c7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3784114000 con 0x7f378410e6c0 2024-05-27T07:53:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f378259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37841021c0 0x7f37841a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f378259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37841021c0 0x7f37841a0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53736/0 (socket says 172.21.15.114:53736) 2024-05-27T07:53:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f378259c640 1 -- 172.21.15.114:0/3184399123 learned_addr learned my addr 172.21.15.114:0/3184399123 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f3782d9d640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f378410e6c0 0x7f378419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f377bfff640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3784109d10 0x7f37841a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f378259c640 1 -- 172.21.15.114:0/3184399123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f378410e6c0 msgr2=0x7f378419a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f378259c640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f378410e6c0 0x7f378419a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f378259c640 1 -- 172.21.15.114:0/3184399123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3784109d10 msgr2=0x7f37841a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f378259c640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3784109d10 0x7f37841a0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f378259c640 1 -- 172.21.15.114:0/3184399123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378419adb0 con 0x7f37841021c0 2024-05-27T07:53:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.714+0000 7f377bfff640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3784109d10 0x7f37841a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:53:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.715+0000 7f378259c640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37841021c0 0x7f37841a0500 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f377400e9f0 tx=0x7f377400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.715+0000 7f377b7fe640 1 -- 172.21.15.114:0/3184399123 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377400ccc0 con 0x7f37841021c0 2024-05-27T07:53:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.715+0000 7f377b7fe640 1 -- 172.21.15.114:0/3184399123 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37740035f0 con 0x7f37841021c0 2024-05-27T07:53:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.715+0000 7f37889c7640 1 -- 172.21.15.114:0/3184399123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f378419b040 con 0x7f37841021c0 2024-05-27T07:53:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.715+0000 7f377b7fe640 1 -- 172.21.15.114:0/3184399123 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377400ce60 con 0x7f37841021c0 2024-05-27T07:53:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.715+0000 7f37889c7640 1 -- 172.21.15.114:0/3184399123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37841a72a0 con 0x7f37841021c0 2024-05-27T07:53:38.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.718+0000 7f377b7fe640 1 -- 172.21.15.114:0/3184399123 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f3774003aa0 con 0x7f37841021c0 2024-05-27T07:53:38.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.718+0000 7f378f87a740 1 Processor -- start 2024-05-27T07:53:38.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.718+0000 7f378f87a740 1 -- start start 2024-05-27T07:53:38.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.718+0000 7f378f87a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d887a081b0 0x55d887da9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.718+0000 7f378f87a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879df2f0 0x55d887da97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.718+0000 7f378f87a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879e74d0 0x55d887da9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.718+0000 7f378f87a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d887d1bc30 con 0x55d8879e74d0 2024-05-27T07:53:38.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.718+0000 7f378f87a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d887d0b7d0 con 0x55d8879df2f0 2024-05-27T07:53:38.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.718+0000 7f378f87a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d887d0f810 con 0x55d887a081b0 2024-05-27T07:53:38.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.719+0000 7f3782d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d887a081b0 0x55d887da9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.719+0000 7f3782d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d887a081b0 0x55d887da9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59114/0 (socket says 172.21.15.114:59114) 2024-05-27T07:53:38.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.719+0000 7f3782d9d640 1 -- 172.21.15.114:0/489569121 learned_addr learned my addr 172.21.15.114:0/489569121 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:38.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.719+0000 7f378259c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879df2f0 0x55d887da97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.719+0000 7f377bfff640 1 --2- 172.21.15.114:0/489569121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879e74d0 0x55d887da9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.719+0000 7f377b7fe640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37700b2950 0x7f37700b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.719+0000 7f3782d9d640 1 -- 172.21.15.114:0/489569121 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879df2f0 msgr2=0x55d887da97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:38.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.719+0000 7f377b7fe640 1 -- 172.21.15.114:0/3184399123 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f37740e2e10 con 0x7f37841021c0 2024-05-27T07:53:38.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.720+0000 7f3782d9d640 1 --2- 172.21.15.114:0/489569121 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879df2f0 0x55d887da97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.720+0000 7f3782d9d640 1 -- 172.21.15.114:0/489569121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879e74d0 msgr2=0x55d887da9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:38.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.720+0000 7f3782d9d640 1 --2- 172.21.15.114:0/489569121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879e74d0 0x55d887da9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.720+0000 7f3782d9d640 1 -- 172.21.15.114:0/489569121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d887a20210 con 0x55d887a081b0 2024-05-27T07:53:38.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.720+0000 7f377bfff640 1 --2- 172.21.15.114:0/489569121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879e74d0 0x55d887da9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:38.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.720+0000 7f378259c640 1 --2- 172.21.15.114:0/489569121 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879df2f0 0x55d887da97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:53:38.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.720+0000 7f3782d9d640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37700b2950 0x7f37700b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.720+0000 7f3782d9d640 1 --2- 172.21.15.114:0/489569121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d887a081b0 0x55d887da9280 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f377c035000 tx=0x7f377c02f410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76f48843846bb046 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:38.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.720+0000 7f3778ff9640 1 -- 172.21.15.114:0/489569121 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c031d50 con 0x55d887a081b0 2024-05-27T07:53:38.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.720+0000 7f3778ff9640 1 -- 172.21.15.114:0/489569121 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f377c036030 con 0x55d887a081b0 2024-05-27T07:53:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.720+0000 7f3778ff9640 1 -- 172.21.15.114:0/489569121 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c0047d0 con 0x55d887a081b0 2024-05-27T07:53:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.722+0000 7f378f87a740 1 -- 172.21.15.114:0/489569121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d887a081b0 msgr2=0x55d887da9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.722+0000 7f378f87a740 1 --2- 172.21.15.114:0/489569121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d887a081b0 0x55d887da9280 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f377c035000 tx=0x7f377c02f410 comp rx=0 tx=0).stop 2024-05-27T07:53:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.722+0000 7f3782d9d640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37700b2950 0x7f37700b4e60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f377c04a000 tx=0x7f377c04b000 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.722+0000 7f378f87a740 1 -- 172.21.15.114:0/489569121 shutdown_connections 2024-05-27T07:53:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.722+0000 7f378f87a740 1 --2- 172.21.15.114:0/489569121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879e74d0 0x55d887da9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.722+0000 7f378f87a740 1 --2- 172.21.15.114:0/489569121 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879df2f0 0x55d887da97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.722+0000 7f378f87a740 1 --2- 172.21.15.114:0/489569121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d887a081b0 0x55d887da9280 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.722+0000 7f378f87a740 1 -- 172.21.15.114:0/489569121 >> 172.21.15.114:0/489569121 conn(0x55d8879fc300 msgr2=0x55d887da5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.722+0000 7f378f87a740 1 -- 172.21.15.114:0/489569121 shutdown_connections 2024-05-27T07:53:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.722+0000 7f378f87a740 1 -- 172.21.15.114:0/489569121 wait complete. 2024-05-27T07:53:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.722+0000 7f378f87a740 1 -- start start 2024-05-27T07:53:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.722+0000 7f378f87a740 1 Processor -- start 2024-05-27T07:53:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f378f87a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d8879df2f0 0x55d887db9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f378f87a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879e74d0 0x55d887db9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f378f87a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d887a081b0 0x55d887db7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f378f87a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d8879f1790 con 0x55d887a081b0 2024-05-27T07:53:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f378f87a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d887cc7890 con 0x55d8879e74d0 2024-05-27T07:53:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f378f87a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d887dba790 con 0x55d8879df2f0 2024-05-27T07:53:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f378f87a740 10 client.0 _ll_register_callbacks cb 0x55d887a1fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f3782d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d887a081b0 0x55d887db7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f378259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d8879df2f0 0x55d887db9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f3782d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d887a081b0 0x55d887db7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53742/0 (socket says 172.21.15.114:53742) 2024-05-27T07:53:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f3782d9d640 1 -- 172.21.15.114:0/3552665706 learned_addr learned my addr 172.21.15.114:0/3552665706 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f377bfff640 1 --2- 172.21.15.114:0/3552665706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879e74d0 0x55d887db9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f3782d9d640 1 -- 172.21.15.114:0/3552665706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d8879df2f0 msgr2=0x55d887db9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f3782d9d640 1 --2- 172.21.15.114:0/3552665706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d8879df2f0 0x55d887db9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f3782d9d640 1 -- 172.21.15.114:0/3552665706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879e74d0 msgr2=0x55d887db9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f3782d9d640 1 --2- 172.21.15.114:0/3552665706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879e74d0 0x55d887db9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f3782d9d640 1 -- 172.21.15.114:0/3552665706 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d887aec620 con 0x55d887a081b0 2024-05-27T07:53:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f3782d9d640 1 --2- 172.21.15.114:0/3552665706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d887a081b0 0x55d887db7d90 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f377c009ca0 tx=0x7f377c0044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac8d1c6016658f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f3769ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f3769ffb640 1 -- 172.21.15.114:0/3552665706 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c05e060 con 0x55d887a081b0 2024-05-27T07:53:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f3769ffb640 1 -- 172.21.15.114:0/3552665706 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f377c004b40 con 0x55d887a081b0 2024-05-27T07:53:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.723+0000 7f378f87a740 1 -- 172.21.15.114:0/3552665706 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f377c036030 con 0x55d887a081b0 2024-05-27T07:53:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.724+0000 7f3769ffb640 1 -- 172.21.15.114:0/3552665706 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f377c065050 con 0x55d887a081b0 2024-05-27T07:53:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.724+0000 7f378f87a740 10 client.18741 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.724+0000 7f378f87a740 1 -- 172.21.15.114:0/3552665706 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55d887dc6a80 con 0x55d887a081b0 2024-05-27T07:53:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.724+0000 7f3769ffb640 1 -- 172.21.15.114:0/3552665706 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f377c036030 con 0x55d887a081b0 2024-05-27T07:53:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.724+0000 7f3769ffb640 1 -- 172.21.15.114:0/3552665706 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f377c05a960 con 0x55d887a081b0 2024-05-27T07:53:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.724+0000 7f378f87a740 10 client.18741 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.724+0000 7f378f87a740 10 client.18741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.724+0000 7f378f87a740 20 client.18741 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.724+0000 7f378f87a740 1 --2- 172.21.15.114:0/3552665706 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d8879c6d80 0x55d887dc9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.724+0000 7f378f87a740 4 client.18741 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.724+0000 7f378f87a740 1 -- 172.21.15.114:0/3552665706 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d887d1db80 con 0x55d8879c6d80 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.724+0000 7f378259c640 1 --2- 172.21.15.114:0/3552665706 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d8879c6d80 0x55d887dc9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.725+0000 7f3769ffb640 10 client.18741 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.725+0000 7f378259c640 1 --2- 172.21.15.114:0/3552665706 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d8879c6d80 0x55d887dc9ea0 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ad119cc60a6d06d server_cookie=fa4a990705e2a3e5 in_seq=0 out_seq=0 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f3769ffb640 1 -- 172.21.15.114:0/3552665706 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d887d1db80 con 0x55d8879c6d80 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f3769ffb640 10 client.18741 handle_command_reply: tid=0 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f378f87a740 1 client.18741 shutdown 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f378f87a740 1 -- 172.21.15.114:0/3552665706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d887a081b0 msgr2=0x55d887db7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f378f87a740 1 --2- 172.21.15.114:0/3552665706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d887a081b0 0x55d887db7d90 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f377c009ca0 tx=0x7f377c0044f0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f378f87a740 1 -- 172.21.15.114:0/3552665706 shutdown_connections 2024-05-27T07:53:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f378f87a740 1 --2- 172.21.15.114:0/3552665706 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d8879c6d80 0x55d887dc9ea0 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f378f87a740 1 --2- 172.21.15.114:0/3552665706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d887a081b0 0x55d887db7d90 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f378f87a740 1 --2- 172.21.15.114:0/3552665706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879e74d0 0x55d887db9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f378f87a740 1 --2- 172.21.15.114:0/3552665706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d8879df2f0 0x55d887db9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f3769ffb640 0 client.18741 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f378f87a740 1 -- 172.21.15.114:0/3552665706 >> 172.21.15.114:0/3552665706 conn(0x55d8879fc300 msgr2=0x55d887dab080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f378f87a740 1 -- 172.21.15.114:0/3552665706 shutdown_connections 2024-05-27T07:53:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f378f87a740 1 -- 172.21.15.114:0/3552665706 wait complete. 2024-05-27T07:53:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.728+0000 7f378f87a740 20 client.18741 trim_cache size 0 max 16384 2024-05-27T07:53:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.755+0000 7f378f87a740 1 Processor -- start 2024-05-27T07:53:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.756+0000 7f378f87a740 1 -- start start 2024-05-27T07:53:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.757+0000 7f378f87a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d8879df2f0 0x55d887da5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.757+0000 7f378f87a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879e74d0 0x55d887db82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.757+0000 7f378f87a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d887a081b0 0x55d887dcf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.757+0000 7f378f87a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f37841afd40 con 0x55d8879e74d0 2024-05-27T07:53:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.757+0000 7f378f87a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f37841b0410 con 0x55d887a081b0 2024-05-27T07:53:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.757+0000 7f378f87a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f37841aebe0 con 0x55d8879df2f0 2024-05-27T07:53:38.758 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.757+0000 7f378259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d8879df2f0 0x55d887da5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.757+0000 7f378259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d8879df2f0 0x55d887da5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59126/0 (socket says 172.21.15.114:59126) 2024-05-27T07:53:38.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.757+0000 7f378259c640 1 -- 172.21.15.114:0/357683218 learned_addr learned my addr 172.21.15.114:0/357683218 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:38.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.758+0000 7f377bfff640 1 --2- 172.21.15.114:0/357683218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879e74d0 0x55d887db82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.758+0000 7f378259c640 1 -- 172.21.15.114:0/357683218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d887a081b0 msgr2=0x55d887dcf650 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:53:38.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.758+0000 7f378259c640 1 --2- 172.21.15.114:0/357683218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d887a081b0 0x55d887dcf650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.758+0000 7f378259c640 1 -- 172.21.15.114:0/357683218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879e74d0 msgr2=0x55d887db82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:38.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.758+0000 7f378259c640 1 --2- 172.21.15.114:0/357683218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879e74d0 0x55d887db82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.758+0000 7f378259c640 1 -- 172.21.15.114:0/357683218 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d887db7090 con 0x55d8879df2f0 2024-05-27T07:53:38.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.759+0000 7f378259c640 1 --2- 172.21.15.114:0/357683218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d8879df2f0 0x55d887da5c50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f37740a99f0 tx=0x7f37740a9b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cabf44bb7b793128 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:38.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.760+0000 7f376bfff640 1 -- 172.21.15.114:0/357683218 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37740c43d0 con 0x55d8879df2f0 2024-05-27T07:53:38.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.760+0000 7f376bfff640 1 -- 172.21.15.114:0/357683218 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37740a9cf0 con 0x55d8879df2f0 2024-05-27T07:53:38.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.760+0000 7f376bfff640 1 -- 172.21.15.114:0/357683218 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37740a5f80 con 0x55d8879df2f0 2024-05-27T07:53:38.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.761+0000 7f378f87a740 1 -- 172.21.15.114:0/357683218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d8879df2f0 msgr2=0x55d887da5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:38.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.761+0000 7f378f87a740 1 --2- 172.21.15.114:0/357683218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d8879df2f0 0x55d887da5c50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f37740a99f0 tx=0x7f37740a9b90 comp rx=0 tx=0).stop 2024-05-27T07:53:38.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.763+0000 7f378f87a740 1 -- 172.21.15.114:0/357683218 shutdown_connections 2024-05-27T07:53:38.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.763+0000 7f378f87a740 1 --2- 172.21.15.114:0/357683218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d887a081b0 0x55d887dcf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.763+0000 7f378f87a740 1 --2- 172.21.15.114:0/357683218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879e74d0 0x55d887db82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.763+0000 7f378f87a740 1 --2- 172.21.15.114:0/357683218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d8879df2f0 0x55d887da5c50 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.763+0000 7f378f87a740 1 -- 172.21.15.114:0/357683218 >> 172.21.15.114:0/357683218 conn(0x55d8879fc300 msgr2=0x55d887da5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.766+0000 7f378f87a740 1 -- 172.21.15.114:0/357683218 shutdown_connections 2024-05-27T07:53:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.768+0000 7f378f87a740 1 -- 172.21.15.114:0/357683218 wait complete. 2024-05-27T07:53:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.768+0000 7f378f87a740 1 -- start start 2024-05-27T07:53:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.768+0000 7f378f87a740 1 Processor -- start 2024-05-27T07:53:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.769+0000 7f378f87a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879df2f0 0x55d887dbca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.769+0000 7f378f87a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879e74d0 0x55d887dbcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.769+0000 7f378f87a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d887a081b0 0x55d887dbd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.769+0000 7f378f87a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f37841ae510 con 0x55d8879df2f0 2024-05-27T07:53:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.769+0000 7f378f87a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f37841ad120 con 0x55d8879e74d0 2024-05-27T07:53:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.769+0000 7f378f87a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f37841ad770 con 0x55d887a081b0 2024-05-27T07:53:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.769+0000 7f378f87a740 10 client.0 _ll_register_callbacks cb 0x7f377c049020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.769+0000 7f378259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879df2f0 0x55d887dbca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.769+0000 7f378259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879df2f0 0x55d887dbca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53758/0 (socket says 172.21.15.114:53758) 2024-05-27T07:53:38.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.769+0000 7f378259c640 1 -- 172.21.15.114:0/3488787094 learned_addr learned my addr 172.21.15.114:0/3488787094 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:38.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.771+0000 7f378259c640 1 -- 172.21.15.114:0/3488787094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d887a081b0 msgr2=0x55d887dbd510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:53:38.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.771+0000 7f378259c640 1 --2- 172.21.15.114:0/3488787094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d887a081b0 0x55d887dbd510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.771+0000 7f378259c640 1 -- 172.21.15.114:0/3488787094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879e74d0 msgr2=0x55d887dbcfd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:53:38.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.771+0000 7f378259c640 1 --2- 172.21.15.114:0/3488787094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879e74d0 0x55d887dbcfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:38.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.771+0000 7f378259c640 1 -- 172.21.15.114:0/3488787094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d887e2c570 con 0x55d8879df2f0 2024-05-27T07:53:38.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.771+0000 7f378259c640 1 --2- 172.21.15.114:0/3488787094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879df2f0 0x55d887dbca90 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f37740a9c80 tx=0x7f37740c4460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce155eb281cc50a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:38.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.771+0000 7f3769ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:38.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.772+0000 7f3769ffb640 1 -- 172.21.15.114:0/3488787094 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377406d020 con 0x55d8879df2f0 2024-05-27T07:53:38.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.772+0000 7f378f87a740 1 -- 172.21.15.114:0/3488787094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37740a9cf0 con 0x55d8879df2f0 2024-05-27T07:53:38.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.772+0000 7f3769ffb640 1 -- 172.21.15.114:0/3488787094 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f37740037a0 con 0x55d8879df2f0 2024-05-27T07:53:38.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.772+0000 7f3769ffb640 1 -- 172.21.15.114:0/3488787094 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f37740ae0a0 con 0x55d8879df2f0 2024-05-27T07:53:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.776+0000 7f378f87a740 10 client.18747 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.776+0000 7f378f87a740 1 -- 172.21.15.114:0/3488787094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55d887e2d7c0 con 0x55d8879df2f0 2024-05-27T07:53:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.777+0000 7f3769ffb640 1 -- 172.21.15.114:0/3488787094 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37740a9cf0 con 0x55d8879df2f0 2024-05-27T07:53:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.777+0000 7f3769ffb640 1 -- 172.21.15.114:0/3488787094 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f37740a5660 con 0x55d8879df2f0 2024-05-27T07:53:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.777+0000 7f378f87a740 10 client.18747 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.777+0000 7f378f87a740 10 client.18747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.777+0000 7f378f87a740 20 client.18747 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.777+0000 7f378f87a740 1 --2- 172.21.15.114:0/3488787094 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d88799c1c0 0x55d887e2ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.777+0000 7f378f87a740 4 client.18747 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:53:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.777+0000 7f378f87a740 1 -- 172.21.15.114:0/3488787094 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d887d17360 con 0x55d88799c1c0 2024-05-27T07:53:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.781+0000 7f377bfff640 1 --2- 172.21.15.114:0/3488787094 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d88799c1c0 0x55d887e2ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:38.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.783+0000 7f377bfff640 1 --2- 172.21.15.114:0/3488787094 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d88799c1c0 0x55d887e2ff60 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b04289927fa47dd6 server_cookie=9d1ebd01a59d154a in_seq=0 out_seq=0 2024-05-27T07:53:38.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:38.783+0000 7f3769ffb640 10 client.18747 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f3769ffb640 1 -- 172.21.15.114:0/3488787094 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55d887d17360 con 0x55d88799c1c0 2024-05-27T07:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f3769ffb640 10 client.18747 handle_command_reply: tid=0 2024-05-27T07:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f378f87a740 1 client.18747 shutdown 2024-05-27T07:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f378f87a740 1 -- 172.21.15.114:0/3488787094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879df2f0 msgr2=0x55d887dbca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f378f87a740 1 --2- 172.21.15.114:0/3488787094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879df2f0 0x55d887dbca90 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f37740a9c80 tx=0x7f37740c4460 comp rx=0 tx=0).stop 2024-05-27T07:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f378f87a740 1 -- 172.21.15.114:0/3488787094 shutdown_connections 2024-05-27T07:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f378f87a740 1 --2- 172.21.15.114:0/3488787094 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d88799c1c0 0x55d887e2ff60 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f378f87a740 1 --2- 172.21.15.114:0/3488787094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d887a081b0 0x55d887dbd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f378f87a740 1 --2- 172.21.15.114:0/3488787094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d8879e74d0 0x55d887dbcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f378f87a740 1 --2- 172.21.15.114:0/3488787094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8879df2f0 0x55d887dbca90 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f378f87a740 1 -- 172.21.15.114:0/3488787094 >> 172.21.15.114:0/3488787094 conn(0x55d8879fc300 msgr2=0x55d887da9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f3769ffb640 0 client.18747 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f378f87a740 1 -- 172.21.15.114:0/3488787094 shutdown_connections 2024-05-27T07:53:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f378f87a740 1 -- 172.21.15.114:0/3488787094 wait complete. 2024-05-27T07:53:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.275+0000 7f378f87a740 20 client.18747 trim_cache size 0 max 16384 2024-05-27T07:53:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:53:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:53:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.276+0000 7f37889c7640 1 -- 172.21.15.114:0/3184399123 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37700b2950 msgr2=0x7f37700b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.276+0000 7f37889c7640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37700b2950 0x7f37700b4e60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f377c04a000 tx=0x7f377c04b000 comp rx=0 tx=0).stop 2024-05-27T07:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.276+0000 7f37889c7640 1 -- 172.21.15.114:0/3184399123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37841021c0 msgr2=0x7f37841a0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.276+0000 7f37889c7640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37841021c0 0x7f37841a0500 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f377400e9f0 tx=0x7f377400eeb0 comp rx=0 tx=0).stop 2024-05-27T07:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.277+0000 7f37889c7640 1 -- 172.21.15.114:0/3184399123 shutdown_connections 2024-05-27T07:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.277+0000 7f37889c7640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37700b2950 0x7f37700b4e60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.277+0000 7f37889c7640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f378410e6c0 0x7f378419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.277+0000 7f37889c7640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3784109d10 0x7f37841a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.277+0000 7f37889c7640 1 --2- 172.21.15.114:0/3184399123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37841021c0 0x7f37841a0500 secure :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f377400e9f0 tx=0x7f377400eeb0 comp rx=0 tx=0).stop 2024-05-27T07:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.277+0000 7f37889c7640 1 -- 172.21.15.114:0/3184399123 >> 172.21.15.114:0/3184399123 conn(0x7f37840779d0 msgr2=0x7f378410a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.277+0000 7f37889c7640 1 -- 172.21.15.114:0/3184399123 shutdown_connections 2024-05-27T07:53:39.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:39.277+0000 7f37889c7640 1 -- 172.21.15.114:0/3184399123 wait complete. 2024-05-27T07:53:39.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1322147 9.29 MB/sec execute 591 sec latency 247.175 ms 2024-05-27T07:53:39.342 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:53:39.342 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:53:39.601 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:39.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:39 smithi114 ceph-mon[32038]: pgmap v468: 129 pgs: 129 active+clean; 75 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-05-27T07:53:39.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:39 smithi163 ceph-mon[36451]: pgmap v468: 129 pgs: 129 active+clean; 75 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-05-27T07:53:39.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:39 smithi186 ceph-mon[36647]: pgmap v468: 129 pgs: 129 active+clean; 75 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-05-27T07:53:40.298 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1324415 9.30 MB/sec execute 592 sec latency 41.933 ms 2024-05-27T07:53:41.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1325313 9.29 MB/sec execute 593 sec latency 60.952 ms 2024-05-27T07:53:41.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:41 smithi163 ceph-mon[36451]: pgmap v469: 129 pgs: 129 active+clean; 75 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 163 op/s 2024-05-27T07:53:41.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:41 smithi186 ceph-mon[36647]: pgmap v469: 129 pgs: 129 active+clean; 75 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 163 op/s 2024-05-27T07:53:41.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:41 smithi114 ceph-mon[32038]: pgmap v469: 129 pgs: 129 active+clean; 75 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 163 op/s 2024-05-27T07:53:42.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.173+0000 7f2236cfe700 1 -- 172.21.15.114:0/96945772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22301034b0 msgr2=0x7f22301038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.173+0000 7f2236cfe700 1 --2- 172.21.15.114:0/96945772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22301034b0 0x7f22301038c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f22280099c0 tx=0x7f2228009cd0 comp rx=0 tx=0).stop 2024-05-27T07:53:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.174+0000 7f2236cfe700 1 -- 172.21.15.114:0/96945772 shutdown_connections 2024-05-27T07:53:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.174+0000 7f2236cfe700 1 --2- 172.21.15.114:0/96945772 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2230108730 0x7f2230110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.174+0000 7f2236cfe700 1 --2- 172.21.15.114:0/96945772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22301034b0 0x7f22301038c0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.174+0000 7f2236cfe700 1 --2- 172.21.15.114:0/96945772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2230107de0 0x7f22301081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.174+0000 7f2236cfe700 1 -- 172.21.15.114:0/96945772 >> 172.21.15.114:0/96945772 conn(0x7f22300fedb0 msgr2=0x7f22301011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.174+0000 7f2236cfe700 1 -- 172.21.15.114:0/96945772 shutdown_connections 2024-05-27T07:53:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.174+0000 7f2236cfe700 1 -- 172.21.15.114:0/96945772 wait complete. 2024-05-27T07:53:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.174+0000 7f2236cfe700 1 Processor -- start 2024-05-27T07:53:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.174+0000 7f2236cfe700 1 -- start start 2024-05-27T07:53:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2236cfe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22301034b0 0x7f223019ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2236cfe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2230107de0 0x7f223019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2236cfe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2230108730 0x7f223019f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2236cfe700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2230113360 con 0x7f2230107de0 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2236cfe700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f22301131e0 con 0x7f2230108730 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2236cfe700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22301134e0 con 0x7f22301034b0 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f222ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2230107de0 0x7f223019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2234a9a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22301034b0 0x7f223019ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2234a9a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22301034b0 0x7f223019ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:59144/0 (socket says 172.21.15.114:59144) 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f222ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2230107de0 0x7f223019b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53776/0 (socket says 172.21.15.114:53776) 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2234a9a700 1 -- 172.21.15.114:0/1618932729 learned_addr learned my addr 172.21.15.114:0/1618932729 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f223529b700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2230108730 0x7f223019f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2234a9a700 1 -- 172.21.15.114:0/1618932729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2230108730 msgr2=0x7f223019f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2234a9a700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2230108730 0x7f223019f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2234a9a700 1 -- 172.21.15.114:0/1618932729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2230107de0 msgr2=0x7f223019b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2234a9a700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2230107de0 0x7f223019b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2234a9a700 1 -- 172.21.15.114:0/1618932729 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223019fc70 con 0x7f22301034b0 2024-05-27T07:53:42.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f222ffff700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2230107de0 0x7f223019b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:42.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.175+0000 7f2234a9a700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22301034b0 0x7f223019ada0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2228006010 tx=0x7f22280045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:42.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.176+0000 7f222dffb700 1 -- 172.21.15.114:0/1618932729 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228026070 con 0x7f22301034b0 2024-05-27T07:53:42.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.176+0000 7f222dffb700 1 -- 172.21.15.114:0/1618932729 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f222800f460 con 0x7f22301034b0 2024-05-27T07:53:42.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.176+0000 7f2236cfe700 1 -- 172.21.15.114:0/1618932729 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223019ff00 con 0x7f22301034b0 2024-05-27T07:53:42.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.176+0000 7f222dffb700 1 -- 172.21.15.114:0/1618932729 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228009e60 con 0x7f22301034b0 2024-05-27T07:53:42.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.176+0000 7f2236cfe700 1 -- 172.21.15.114:0/1618932729 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22301a03e0 con 0x7f22301034b0 2024-05-27T07:53:42.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.177+0000 7f2236cfe700 1 -- 172.21.15.114:0/1618932729 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22300512b0 con 0x7f22301034b0 2024-05-27T07:53:42.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.179+0000 7f222dffb700 1 -- 172.21.15.114:0/1618932729 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f2228029070 con 0x7f22301034b0 2024-05-27T07:53:42.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.179+0000 7f222dffb700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f221c0b29f0 0x7f221c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:42.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.179+0000 7f222dffb700 1 -- 172.21.15.114:0/1618932729 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f22280e3540 con 0x7f22301034b0 2024-05-27T07:53:42.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.179+0000 7f222ffff700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f221c0b29f0 0x7f221c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:42.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.180+0000 7f222ffff700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f221c0b29f0 0x7f221c0b4f00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2230109a40 tx=0x7f222000b040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:42.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.183+0000 7f222dffb700 1 -- 172.21.15.114:0/1618932729 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2228013070 con 0x7f22301034b0 2024-05-27T07:53:42.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1327037 9.29 MB/sec execute 594 sec latency 53.099 ms 2024-05-27T07:53:42.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.458+0000 7f2236cfe700 1 -- 172.21.15.114:0/1618932729 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f223002e5e0 con 0x7f22301034b0 2024-05-27T07:53:42.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.458+0000 7f222dffb700 1 -- 172.21.15.114:0/1618932729 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+7693 (secure 0 0 0) 0x7f22280aabb0 con 0x7f22301034b0 2024-05-27T07:53:42.459 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:53:42.459 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:53:34.360222+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27557":{"gid":27557,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/3261223371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":3261223371},{"type":"v1","addr":"172.21.15.163:6835","nonce":3261223371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:53:42.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.460+0000 7f2236cfe700 1 -- 172.21.15.114:0/1618932729 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f221c0b29f0 msgr2=0x7f221c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:42.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.460+0000 7f2236cfe700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f221c0b29f0 0x7f221c0b4f00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2230109a40 tx=0x7f222000b040 comp rx=0 tx=0).stop 2024-05-27T07:53:42.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.460+0000 7f2236cfe700 1 -- 172.21.15.114:0/1618932729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22301034b0 msgr2=0x7f223019ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:42.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.460+0000 7f2236cfe700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22301034b0 0x7f223019ada0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2228006010 tx=0x7f22280045e0 comp rx=0 tx=0).stop 2024-05-27T07:53:42.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.461+0000 7f2236cfe700 1 -- 172.21.15.114:0/1618932729 shutdown_connections 2024-05-27T07:53:42.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.461+0000 7f2236cfe700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f221c0b29f0 0x7f221c0b4f00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:42.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.461+0000 7f2236cfe700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2230108730 0x7f223019f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:42.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.461+0000 7f2236cfe700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22301034b0 0x7f223019ada0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:42.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.461+0000 7f2236cfe700 1 --2- 172.21.15.114:0/1618932729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2230107de0 0x7f223019b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:42.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.461+0000 7f2236cfe700 1 -- 172.21.15.114:0/1618932729 >> 172.21.15.114:0/1618932729 conn(0x7f22300fedb0 msgr2=0x7f223006b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:42.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.461+0000 7f2236cfe700 1 -- 172.21.15.114:0/1618932729 shutdown_connections 2024-05-27T07:53:42.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:42.461+0000 7f2236cfe700 1 -- 172.21.15.114:0/1618932729 wait complete. 2024-05-27T07:53:42.462 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 76 2024-05-27T07:53:43.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:53:43.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1328643 9.29 MB/sec execute 595 sec latency 15.749 ms 2024-05-27T07:53:43.523 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:43.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:43 smithi114 ceph-mon[32038]: pgmap v470: 129 pgs: 129 active+clean; 68 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 324 op/s 2024-05-27T07:53:43.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:43 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1618932729' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:53:43.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:43 smithi163 ceph-mon[36451]: pgmap v470: 129 pgs: 129 active+clean; 68 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 324 op/s 2024-05-27T07:53:43.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:43 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1618932729' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:53:43.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:43 smithi186 ceph-mon[36647]: pgmap v470: 129 pgs: 129 active+clean; 68 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 324 op/s 2024-05-27T07:53:43.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:43 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1618932729' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:53:44.297 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1330479 9.29 MB/sec execute 596 sec latency 23.684 ms 2024-05-27T07:53:45.298 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1332183 9.29 MB/sec execute 597 sec latency 16.964 ms 2024-05-27T07:53:45.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.532+0000 7f04afc7f700 1 -- 172.21.15.114:0/2018471848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04a810a920 msgr2=0x7f04a810ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:45.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.532+0000 7f04afc7f700 1 --2- 172.21.15.114:0/2018471848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04a810a920 0x7f04a810ad30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f04a40098f0 tx=0x7f04a4009c00 comp rx=0 tx=0).stop 2024-05-27T07:53:45.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.532+0000 7f04afc7f700 1 -- 172.21.15.114:0/2018471848 shutdown_connections 2024-05-27T07:53:45.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.532+0000 7f04afc7f700 1 --2- 172.21.15.114:0/2018471848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a811d920 0x7f04a811fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.532+0000 7f04afc7f700 1 --2- 172.21.15.114:0/2018471848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04a810a920 0x7f04a810ad30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.532+0000 7f04afc7f700 1 --2- 172.21.15.114:0/2018471848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04a810b270 0x7f04a811d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.532+0000 7f04afc7f700 1 -- 172.21.15.114:0/2018471848 >> 172.21.15.114:0/2018471848 conn(0x7f04a806f4c0 msgr2=0x7f04a806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:45.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.533+0000 7f04afc7f700 1 -- 172.21.15.114:0/2018471848 shutdown_connections 2024-05-27T07:53:45.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.534+0000 7f04afc7f700 1 -- 172.21.15.114:0/2018471848 wait complete. 2024-05-27T07:53:45.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.535+0000 7f04afc7f700 1 Processor -- start 2024-05-27T07:53:45.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.535+0000 7f04afc7f700 1 -- start start 2024-05-27T07:53:45.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.535+0000 7f04afc7f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04a810a920 0x7f04a81af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.536+0000 7f04afc7f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a810b270 0x7f04a81afe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.536+0000 7f04afc7f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04a811d920 0x7f04a81a9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.536+0000 7f04afc7f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04a81232d0 con 0x7f04a811d920 2024-05-27T07:53:45.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.536+0000 7f04afc7f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04a8123150 con 0x7f04a810a920 2024-05-27T07:53:45.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.536+0000 7f04afc7f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04a8123450 con 0x7f04a810b270 2024-05-27T07:53:45.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.538+0000 7f04ae21c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04a811d920 0x7f04a81a9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:45.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.538+0000 7f04ae21c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04a811d920 0x7f04a81a9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58102/0 (socket says 172.21.15.114:58102) 2024-05-27T07:53:45.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.538+0000 7f04ae21c700 1 -- 172.21.15.114:0/3303760764 learned_addr learned my addr 172.21.15.114:0/3303760764 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:45.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.539+0000 7f04ada1b700 1 --2- 172.21.15.114:0/3303760764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04a810a920 0x7f04a81af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:45.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.539+0000 7f04ada1b700 1 -- 172.21.15.114:0/3303760764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a810b270 msgr2=0x7f04a81afe10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:53:45.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.539+0000 7f04ada1b700 1 --2- 172.21.15.114:0/3303760764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a810b270 0x7f04a81afe10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.539+0000 7f04ada1b700 1 -- 172.21.15.114:0/3303760764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04a811d920 msgr2=0x7f04a81a9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:45.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.539+0000 7f04ada1b700 1 --2- 172.21.15.114:0/3303760764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04a811d920 0x7f04a81a9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.539+0000 7f04ada1b700 1 -- 172.21.15.114:0/3303760764 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a81aa1d0 con 0x7f04a810a920 2024-05-27T07:53:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.540+0000 7f04ada1b700 1 --2- 172.21.15.114:0/3303760764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04a810a920 0x7f04a81af8b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f04a4000c00 tx=0x7f04a4005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.541+0000 7f049effd700 1 -- 172.21.15.114:0/3303760764 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a400b560 con 0x7f04a810a920 2024-05-27T07:53:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.541+0000 7f049effd700 1 -- 172.21.15.114:0/3303760764 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04a400bbb0 con 0x7f04a810a920 2024-05-27T07:53:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.541+0000 7f04afc7f700 1 -- 172.21.15.114:0/3303760764 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a81aa460 con 0x7f04a810a920 2024-05-27T07:53:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.541+0000 7f049effd700 1 -- 172.21.15.114:0/3303760764 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4021440 con 0x7f04a810a920 2024-05-27T07:53:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.541+0000 7f04afc7f700 1 -- 172.21.15.114:0/3303760764 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a81b6760 con 0x7f04a810a920 2024-05-27T07:53:45.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.546+0000 7f049effd700 1 -- 172.21.15.114:0/3303760764 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f04a4003680 con 0x7f04a810a920 2024-05-27T07:53:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.547+0000 7f049effd700 1 --2- 172.21.15.114:0/3303760764 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04940b2bc0 0x7f04940b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.547+0000 7f049effd700 1 -- 172.21.15.114:0/3303760764 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f04a40e3730 con 0x7f04a810a920 2024-05-27T07:53:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.548+0000 7f04ad21a700 1 --2- 172.21.15.114:0/3303760764 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04940b2bc0 0x7f04940b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.548+0000 7f04be694b80 1 Processor -- start 2024-05-27T07:53:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.548+0000 7f04be694b80 1 -- start start 2024-05-27T07:53:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.549+0000 7f04ad21a700 1 --2- 172.21.15.114:0/3303760764 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04940b2bc0 0x7f04940b50d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f04a81200c0 tx=0x7f0498009310 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.549+0000 7f04be694b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd3aae40 0x5653cd52fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.549+0000 7f04be694b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd530070 0x5653cd529be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.549+0000 7f04be694b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd52a120 0x5653cd52a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.549+0000 7f04be694b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5653cd457d70 con 0x5653cd52a120 2024-05-27T07:53:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.549+0000 7f04be694b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5653cd447910 con 0x5653cd530070 2024-05-27T07:53:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.549+0000 7f04be694b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5653cd44b950 con 0x5653cd3aae40 2024-05-27T07:53:45.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.550+0000 7f04ae21c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd52a120 0x5653cd52a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:45.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.550+0000 7f04ae21c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd52a120 0x5653cd52a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58118/0 (socket says 172.21.15.114:58118) 2024-05-27T07:53:45.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.550+0000 7f04ae21c700 1 -- 172.21.15.114:0/3718089568 learned_addr learned my addr 172.21.15.114:0/3718089568 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:45.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.550+0000 7f04ae21c700 1 -- 172.21.15.114:0/3718089568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd3aae40 msgr2=0x5653cd52fb30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:53:45.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.550+0000 7f04ae21c700 1 --2- 172.21.15.114:0/3718089568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd3aae40 0x5653cd52fb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.550+0000 7f04ae21c700 1 -- 172.21.15.114:0/3718089568 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd530070 msgr2=0x5653cd529be0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:53:45.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.550+0000 7f04ae21c700 1 --2- 172.21.15.114:0/3718089568 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd530070 0x5653cd529be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.550+0000 7f04ae21c700 1 -- 172.21.15.114:0/3718089568 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653cd3f0fb0 con 0x5653cd52a120 2024-05-27T07:53:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.550+0000 7f04ae21c700 1 --2- 172.21.15.114:0/3718089568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd52a120 0x5653cd52a530 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009e90 tx=0x7f04a0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90c00bee1dd6356 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.552+0000 7f0493fff700 1 -- 172.21.15.114:0/3718089568 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0004bb0 con 0x5653cd52a120 2024-05-27T07:53:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.552+0000 7f0493fff700 1 -- 172.21.15.114:0/3718089568 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04a0004d50 con 0x5653cd52a120 2024-05-27T07:53:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.552+0000 7f0493fff700 1 -- 172.21.15.114:0/3718089568 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a00053a0 con 0x5653cd52a120 2024-05-27T07:53:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.554+0000 7f04be694b80 1 -- 172.21.15.114:0/3718089568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd52a120 msgr2=0x5653cd52a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.554+0000 7f04be694b80 1 --2- 172.21.15.114:0/3718089568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd52a120 0x5653cd52a530 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009e90 tx=0x7f04a0009190 comp rx=0 tx=0).stop 2024-05-27T07:53:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.561+0000 7f04be694b80 1 -- 172.21.15.114:0/3718089568 shutdown_connections 2024-05-27T07:53:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.561+0000 7f04be694b80 1 --2- 172.21.15.114:0/3718089568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd52a120 0x5653cd52a530 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.561+0000 7f04be694b80 1 --2- 172.21.15.114:0/3718089568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd3aae40 0x5653cd52fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.561+0000 7f04be694b80 1 --2- 172.21.15.114:0/3718089568 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd530070 0x5653cd529be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.561+0000 7f04be694b80 1 -- 172.21.15.114:0/3718089568 >> 172.21.15.114:0/3718089568 conn(0x5653cd3ad770 msgr2=0x5653cd524d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:45.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.562+0000 7f04be694b80 1 -- 172.21.15.114:0/3718089568 shutdown_connections 2024-05-27T07:53:45.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.566+0000 7f04be694b80 1 -- 172.21.15.114:0/3718089568 wait complete. 2024-05-27T07:53:45.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.566+0000 7f04be694b80 1 -- start start 2024-05-27T07:53:45.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.567+0000 7f04be694b80 1 Processor -- start 2024-05-27T07:53:45.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.568+0000 7f04be694b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd3aae40 0x5653cd5339e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.569+0000 7f04be694b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 0x5653cd533f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.569+0000 7f04be694b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd530070 0x5653cd534460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.569+0000 7f04be694b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5653cd34dfa0 con 0x5653cd530070 2024-05-27T07:53:45.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.569+0000 7f04be694b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5653cd3afda0 con 0x5653cd52a120 2024-05-27T07:53:45.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.569+0000 7f04be694b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5653cd536b60 con 0x5653cd3aae40 2024-05-27T07:53:45.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.569+0000 7f04be694b80 10 client.0 _ll_register_callbacks cb 0x5653cd3de5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:45.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.570+0000 7f04ada1b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd3aae40 0x5653cd5339e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:45.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.570+0000 7f04ada1b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd3aae40 0x5653cd5339e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:41230/0 (socket says 172.21.15.114:41230) 2024-05-27T07:53:45.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.570+0000 7f04ada1b700 1 -- 172.21.15.114:0/1294625843 learned_addr learned my addr 172.21.15.114:0/1294625843 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:45.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.570+0000 7f04ae21c700 1 --2- 172.21.15.114:0/1294625843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd530070 0x5653cd534460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:45.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.571+0000 7f04ad21a700 1 --2- 172.21.15.114:0/1294625843 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 0x5653cd533f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:45.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.571+0000 7f04ada1b700 1 -- 172.21.15.114:0/1294625843 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 msgr2=0x5653cd533f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:45.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.571+0000 7f04ada1b700 1 --2- 172.21.15.114:0/1294625843 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 0x5653cd533f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.571+0000 7f04ada1b700 1 -- 172.21.15.114:0/1294625843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd530070 msgr2=0x5653cd534460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:45.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.571+0000 7f04ada1b700 1 --2- 172.21.15.114:0/1294625843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd530070 0x5653cd534460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.571+0000 7f04ada1b700 1 -- 172.21.15.114:0/1294625843 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653cd3978d0 con 0x5653cd3aae40 2024-05-27T07:53:45.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.571+0000 7f04ada1b700 1 --2- 172.21.15.114:0/1294625843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd3aae40 0x5653cd5339e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f04a40a6770 tx=0x7f04a40aac50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=497ee1512bfd76f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:45.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.572+0000 7f04917fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:53:45.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.572+0000 7f04917fa700 1 -- 172.21.15.114:0/1294625843 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a40e2070 con 0x5653cd3aae40 2024-05-27T07:53:45.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.572+0000 7f04917fa700 1 -- 172.21.15.114:0/1294625843 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f04a400bd80 con 0x5653cd3aae40 2024-05-27T07:53:45.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.572+0000 7f04be694b80 1 -- 172.21.15.114:0/1294625843 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04a0004d50 con 0x5653cd3aae40 2024-05-27T07:53:45.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.573+0000 7f04917fa700 1 -- 172.21.15.114:0/1294625843 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f04a40a9e60 con 0x5653cd3aae40 2024-05-27T07:53:45.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.574+0000 7f04be694b80 10 client.28294 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:45.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.574+0000 7f04be694b80 1 -- 172.21.15.114:0/1294625843 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x5653cd401760 con 0x5653cd3aae40 2024-05-27T07:53:45.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.575+0000 7f04917fa700 1 -- 172.21.15.114:0/1294625843 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04a0004d50 con 0x5653cd3aae40 2024-05-27T07:53:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.578+0000 7f04917fa700 1 -- 172.21.15.114:0/1294625843 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f04a40b1ce0 con 0x5653cd3aae40 2024-05-27T07:53:45.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.579+0000 7f04be694b80 10 client.28294 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:45.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.579+0000 7f04be694b80 10 client.28294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:45.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.579+0000 7f04be694b80 20 client.28294 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:45.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.580+0000 7f04be694b80 1 --2- 172.21.15.114:0/1294625843 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5653cd5439a0 0x5653cd545dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.580+0000 7f04be694b80 4 client.28294 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:45.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.580+0000 7f04be694b80 1 -- 172.21.15.114:0/1294625843 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653cd459cc0 con 0x5653cd5439a0 2024-05-27T07:53:45.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.582+0000 7f04ad21a700 1 --2- 172.21.15.114:0/1294625843 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5653cd5439a0 0x5653cd545dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:45.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.584+0000 7f04ad21a700 1 --2- 172.21.15.114:0/1294625843 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5653cd5439a0 0x5653cd545dc0 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8da6f79c62fb0595 server_cookie=7bb4ef88d8e21fbd in_seq=0 out_seq=0 2024-05-27T07:53:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.584+0000 7f04917fa700 10 client.28294 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:45.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.589+0000 7f04917fa700 1 -- 172.21.15.114:0/1294625843 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5653cd459cc0 con 0x5653cd5439a0 2024-05-27T07:53:45.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.589+0000 7f04917fa700 10 client.28294 handle_command_reply: tid=0 2024-05-27T07:53:45.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.590+0000 7f04be694b80 1 client.28294 shutdown 2024-05-27T07:53:45.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.591+0000 7f04be694b80 1 -- 172.21.15.114:0/1294625843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd3aae40 msgr2=0x5653cd5339e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:45.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.591+0000 7f04be694b80 1 --2- 172.21.15.114:0/1294625843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd3aae40 0x5653cd5339e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f04a40a6770 tx=0x7f04a40aac50 comp rx=0 tx=0).stop 2024-05-27T07:53:45.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.593+0000 7f04be694b80 1 -- 172.21.15.114:0/1294625843 shutdown_connections 2024-05-27T07:53:45.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.593+0000 7f04be694b80 1 --2- 172.21.15.114:0/1294625843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd530070 0x5653cd534460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.593+0000 7f04be694b80 1 --2- 172.21.15.114:0/1294625843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd3aae40 0x5653cd5339e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.593+0000 7f04be694b80 1 --2- 172.21.15.114:0/1294625843 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5653cd5439a0 0x5653cd545dc0 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.593+0000 7f04be694b80 1 --2- 172.21.15.114:0/1294625843 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 0x5653cd533f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.593+0000 7f04be694b80 1 -- 172.21.15.114:0/1294625843 >> 172.21.15.114:0/1294625843 conn(0x5653cd3ad770 msgr2=0x5653cd528bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:45.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.593+0000 7f04917fa700 0 client.28294 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:45.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.593+0000 7f04be694b80 1 -- 172.21.15.114:0/1294625843 shutdown_connections 2024-05-27T07:53:45.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.594+0000 7f04be694b80 1 -- 172.21.15.114:0/1294625843 wait complete. 2024-05-27T07:53:45.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.595+0000 7f04be694b80 20 client.28294 trim_cache size 0 max 16384 2024-05-27T07:53:45.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.627+0000 7f04be694b80 1 Processor -- start 2024-05-27T07:53:45.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.627+0000 7f04be694b80 1 -- start start 2024-05-27T07:53:45.628 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:45 smithi114 ceph-mon[32038]: pgmap v471: 129 pgs: 129 active+clean; 68 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 239 op/s 2024-05-27T07:53:45.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.628+0000 7f04be694b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 0x5653cd529be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.628+0000 7f04ada1b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 0x5653cd529be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:45.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.628+0000 7f04ada1b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 0x5653cd529be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58148/0 (socket says 172.21.15.114:58148) 2024-05-27T07:53:45.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.628+0000 7f04ada1b700 1 -- 172.21.15.114:0/3625810472 learned_addr learned my addr 172.21.15.114:0/3625810472 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:45.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.628+0000 7f04be694b80 1 --2- 172.21.15.114:0/3625810472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 0x5653cd52d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.628+0000 7f04be694b80 1 --2- 172.21.15.114:0/3625810472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd530070 0x5653cd52d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.628+0000 7f04be694b80 1 -- 172.21.15.114:0/3625810472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04a81c00f0 con 0x5653cd3aae40 2024-05-27T07:53:45.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.628+0000 7f04be694b80 1 -- 172.21.15.114:0/3625810472 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04a81c07c0 con 0x5653cd52a120 2024-05-27T07:53:45.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.628+0000 7f04be694b80 1 -- 172.21.15.114:0/3625810472 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04a81bef70 con 0x5653cd530070 2024-05-27T07:53:45.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.628+0000 7f04ae21c700 1 --2- 172.21.15.114:0/3625810472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd530070 0x5653cd52d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:45.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.628+0000 7f04ad21a700 1 --2- 172.21.15.114:0/3625810472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 0x5653cd52d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:45.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.629+0000 7f04ada1b700 1 -- 172.21.15.114:0/3625810472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd530070 msgr2=0x5653cd52d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:45.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.629+0000 7f04ada1b700 1 --2- 172.21.15.114:0/3625810472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd530070 0x5653cd52d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.629+0000 7f04ada1b700 1 -- 172.21.15.114:0/3625810472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 msgr2=0x5653cd52d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:45.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.629+0000 7f04ada1b700 1 --2- 172.21.15.114:0/3625810472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 0x5653cd52d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.629+0000 7f04ada1b700 1 -- 172.21.15.114:0/3625810472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653cd531f50 con 0x5653cd3aae40 2024-05-27T07:53:45.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.629+0000 7f04ada1b700 1 --2- 172.21.15.114:0/3625810472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 0x5653cd529be0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f04a40a7960 tx=0x7f04a40b3410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b76c6f438260498 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:45.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.629+0000 7f04937fe700 1 -- 172.21.15.114:0/3625810472 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a40e2030 con 0x5653cd3aae40 2024-05-27T07:53:45.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.629+0000 7f04937fe700 1 -- 172.21.15.114:0/3625810472 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04a40aa6f0 con 0x5653cd3aae40 2024-05-27T07:53:45.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.629+0000 7f04937fe700 1 -- 172.21.15.114:0/3625810472 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a40afd80 con 0x5653cd3aae40 2024-05-27T07:53:45.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.631+0000 7f04be694b80 1 -- 172.21.15.114:0/3625810472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 msgr2=0x5653cd529be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:45.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.631+0000 7f04be694b80 1 --2- 172.21.15.114:0/3625810472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 0x5653cd529be0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f04a40a7960 tx=0x7f04a40b3410 comp rx=0 tx=0).stop 2024-05-27T07:53:45.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.631+0000 7f04be694b80 1 -- 172.21.15.114:0/3625810472 shutdown_connections 2024-05-27T07:53:45.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.631+0000 7f04be694b80 1 --2- 172.21.15.114:0/3625810472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd530070 0x5653cd52d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.631+0000 7f04be694b80 1 --2- 172.21.15.114:0/3625810472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 0x5653cd529be0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.631+0000 7f04be694b80 1 --2- 172.21.15.114:0/3625810472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 0x5653cd52d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.631+0000 7f04be694b80 1 -- 172.21.15.114:0/3625810472 >> 172.21.15.114:0/3625810472 conn(0x5653cd3ad770 msgr2=0x5653cd5322d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:45.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.631+0000 7f04be694b80 1 -- 172.21.15.114:0/3625810472 shutdown_connections 2024-05-27T07:53:45.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.631+0000 7f04be694b80 1 -- 172.21.15.114:0/3625810472 wait complete. 2024-05-27T07:53:45.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.632+0000 7f04be694b80 1 -- start start 2024-05-27T07:53:45.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.632+0000 7f04be694b80 1 Processor -- start 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.633+0000 7f04be694b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 0x5653cd53c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.633+0000 7f04be694b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 0x5653cd53c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.633+0000 7f04be694b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd530070 0x5653cd528c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.633+0000 7f04ada1b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 0x5653cd53c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.633+0000 7f04ada1b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 0x5653cd53c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58150/0 (socket says 172.21.15.114:58150) 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.633+0000 7f04ada1b700 1 -- 172.21.15.114:0/1865825982 learned_addr learned my addr 172.21.15.114:0/1865825982 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.634+0000 7f04be694b80 1 -- 172.21.15.114:0/1865825982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04a81be8a0 con 0x5653cd3aae40 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.634+0000 7f04be694b80 1 -- 172.21.15.114:0/1865825982 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04a81bd4b0 con 0x5653cd52a120 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.634+0000 7f04be694b80 1 -- 172.21.15.114:0/1865825982 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04a81bdb00 con 0x5653cd530070 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.634+0000 7f04be694b80 10 client.0 _ll_register_callbacks cb 0x5653cd3de5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.634+0000 7f04ada1b700 1 -- 172.21.15.114:0/1865825982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd530070 msgr2=0x5653cd528c50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.634+0000 7f04ada1b700 1 --2- 172.21.15.114:0/1865825982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd530070 0x5653cd528c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.634+0000 7f04ada1b700 1 -- 172.21.15.114:0/1865825982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 msgr2=0x5653cd53c620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:53:45.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.634+0000 7f04ada1b700 1 --2- 172.21.15.114:0/1865825982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 0x5653cd53c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.634+0000 7f04ada1b700 1 -- 172.21.15.114:0/1865825982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653cd529390 con 0x5653cd3aae40 2024-05-27T07:53:45.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.635+0000 7f04ada1b700 1 --2- 172.21.15.114:0/1865825982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 0x5653cd53c0e0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f04a40e3040 tx=0x7f04a40aff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e073afc41485e12 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:45.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.635+0000 7f04917fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:45.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.635+0000 7f04917fa700 1 -- 172.21.15.114:0/1865825982 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a40b7020 con 0x5653cd3aae40 2024-05-27T07:53:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.635+0000 7f04917fa700 1 -- 172.21.15.114:0/1865825982 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f04a400bd80 con 0x5653cd3aae40 2024-05-27T07:53:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.635+0000 7f04be694b80 1 -- 172.21.15.114:0/1865825982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04a40aa6f0 con 0x5653cd3aae40 2024-05-27T07:53:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.636+0000 7f04917fa700 1 -- 172.21.15.114:0/1865825982 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f04a40ae9f0 con 0x5653cd3aae40 2024-05-27T07:53:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.636+0000 7f04be694b80 10 client.18771 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.636+0000 7f04be694b80 1 -- 172.21.15.114:0/1865825982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x5653cd529390 con 0x5653cd3aae40 2024-05-27T07:53:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.636+0000 7f04917fa700 1 -- 172.21.15.114:0/1865825982 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04a40cb550 con 0x5653cd3aae40 2024-05-27T07:53:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.636+0000 7f04917fa700 1 -- 172.21.15.114:0/1865825982 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f04a40b1ce0 con 0x5653cd3aae40 2024-05-27T07:53:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.636+0000 7f04be694b80 10 client.18771 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.636+0000 7f04be694b80 10 client.18771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.636+0000 7f04be694b80 20 client.18771 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.636+0000 7f04be694b80 1 --2- 172.21.15.114:0/1865825982 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5653cd5ae060 0x5653cd5b0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.637+0000 7f04be694b80 4 client.18771 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:53:45.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.637+0000 7f04be694b80 1 -- 172.21.15.114:0/1865825982 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f04a40aa6f0 con 0x5653cd5ae060 2024-05-27T07:53:45.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.637+0000 7f04ad21a700 1 --2- 172.21.15.114:0/1865825982 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5653cd5ae060 0x5653cd5b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:45.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.637+0000 7f04917fa700 10 client.18771 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:45.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.638+0000 7f04ad21a700 1 --2- 172.21.15.114:0/1865825982 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5653cd5ae060 0x5653cd5b0480 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdfdb7ffc08306ab server_cookie=6ac8661d6a3052ce in_seq=0 out_seq=0 2024-05-27T07:53:45.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.640+0000 7f04917fa700 1 -- 172.21.15.114:0/1865825982 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f04a40aa6f0 con 0x5653cd5ae060 2024-05-27T07:53:45.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.640+0000 7f04917fa700 10 client.18771 handle_command_reply: tid=0 2024-05-27T07:53:45.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.640+0000 7f04be694b80 1 client.18771 shutdown 2024-05-27T07:53:45.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.640+0000 7f04be694b80 1 -- 172.21.15.114:0/1865825982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 msgr2=0x5653cd53c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:45.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.640+0000 7f04be694b80 1 --2- 172.21.15.114:0/1865825982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 0x5653cd53c0e0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f04a40e3040 tx=0x7f04a40aff90 comp rx=0 tx=0).stop 2024-05-27T07:53:45.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.641+0000 7f04be694b80 1 -- 172.21.15.114:0/1865825982 shutdown_connections 2024-05-27T07:53:45.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.641+0000 7f04be694b80 1 --2- 172.21.15.114:0/1865825982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5653cd530070 0x5653cd528c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.641+0000 7f04be694b80 1 --2- 172.21.15.114:0/1865825982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653cd3aae40 0x5653cd53c0e0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.641+0000 7f04be694b80 1 --2- 172.21.15.114:0/1865825982 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5653cd5ae060 0x5653cd5b0480 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.641+0000 7f04be694b80 1 --2- 172.21.15.114:0/1865825982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653cd52a120 0x5653cd53c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.641+0000 7f04917fa700 0 client.18771 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:45.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.641+0000 7f04be694b80 1 -- 172.21.15.114:0/1865825982 >> 172.21.15.114:0/1865825982 conn(0x5653cd3ad770 msgr2=0x5653cd52d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:45.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.641+0000 7f04be694b80 1 -- 172.21.15.114:0/1865825982 shutdown_connections 2024-05-27T07:53:45.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.641+0000 7f04be694b80 1 -- 172.21.15.114:0/1865825982 wait complete. 2024-05-27T07:53:45.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.641+0000 7f04be694b80 20 client.18771 trim_cache size 0 max 16384 2024-05-27T07:53:45.644 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:53:45.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.644+0000 7f04afc7f700 1 -- 172.21.15.114:0/3303760764 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04940b2bc0 msgr2=0x7f04940b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:45.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.644+0000 7f04afc7f700 1 --2- 172.21.15.114:0/3303760764 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04940b2bc0 0x7f04940b50d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f04a81200c0 tx=0x7f0498009310 comp rx=0 tx=0).stop 2024-05-27T07:53:45.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.644+0000 7f04afc7f700 1 -- 172.21.15.114:0/3303760764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04a810a920 msgr2=0x7f04a81af8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:45.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.644+0000 7f04afc7f700 1 --2- 172.21.15.114:0/3303760764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04a810a920 0x7f04a81af8b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f04a4000c00 tx=0x7f04a4005040 comp rx=0 tx=0).stop 2024-05-27T07:53:45.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.644+0000 7f04afc7f700 1 -- 172.21.15.114:0/3303760764 shutdown_connections 2024-05-27T07:53:45.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.644+0000 7f04afc7f700 1 --2- 172.21.15.114:0/3303760764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04a811d920 0x7f04a81a9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.644+0000 7f04afc7f700 1 --2- 172.21.15.114:0/3303760764 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f04940b2bc0 0x7f04940b50d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.644+0000 7f04afc7f700 1 --2- 172.21.15.114:0/3303760764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04a810a920 0x7f04a81af8b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.644+0000 7f04afc7f700 1 --2- 172.21.15.114:0/3303760764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a810b270 0x7f04a81afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:45.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.644+0000 7f04afc7f700 1 -- 172.21.15.114:0/3303760764 >> 172.21.15.114:0/3303760764 conn(0x7f04a806f4c0 msgr2=0x7f04a811b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:45.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.644+0000 7f04afc7f700 1 -- 172.21.15.114:0/3303760764 shutdown_connections 2024-05-27T07:53:45.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:45.644+0000 7f04afc7f700 1 -- 172.21.15.114:0/3303760764 wait complete. 2024-05-27T07:53:45.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:45 smithi163 ceph-mon[36451]: pgmap v471: 129 pgs: 129 active+clean; 68 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 239 op/s 2024-05-27T07:53:45.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:45 smithi186 ceph-mon[36647]: pgmap v471: 129 pgs: 129 active+clean; 68 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 239 op/s 2024-05-27T07:53:46.298 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1333138 9.28 MB/sec execute 598 sec latency 67.248 ms 2024-05-27T07:53:47.298 INFO:tasks.workunit.client.0.smithi114.stdout: 1 1335360 9.29 MB/sec execute 599 sec latency 15.694 ms 2024-05-27T07:53:47.395 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:53:47.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:53:47.652 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:47.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:47 smithi163 ceph-mon[36451]: pgmap v472: 129 pgs: 129 active+clean; 74 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 303 op/s 2024-05-27T07:53:47.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:47 smithi186 ceph-mon[36647]: pgmap v472: 129 pgs: 129 active+clean; 74 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 303 op/s 2024-05-27T07:53:47.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:47 smithi114 ceph-mon[32038]: pgmap v472: 129 pgs: 129 active+clean; 74 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 303 op/s 2024-05-27T07:53:48.298 INFO:tasks.workunit.client.0.smithi114.stdout: 1 cleanup 600 sec 2024-05-27T07:53:48.485 INFO:tasks.workunit.client.0.smithi114.stdout: 0 cleanup 600 sec 2024-05-27T07:53:48.485 INFO:tasks.workunit.client.0.smithi114.stdout: 2024-05-27T07:53:48.485 INFO:tasks.workunit.client.0.smithi114.stdout: Operation Count AvgLat MaxLat 2024-05-27T07:53:48.485 INFO:tasks.workunit.client.0.smithi114.stdout: ---------------------------------------- 2024-05-27T07:53:48.485 INFO:tasks.workunit.client.0.smithi114.stdout: NTCreateX 177498 1.086 483.891 2024-05-27T07:53:48.485 INFO:tasks.workunit.client.0.smithi114.stdout: Close 130379 0.002 0.111 2024-05-27T07:53:48.485 INFO:tasks.workunit.client.0.smithi114.stdout: Rename 7517 5.339 247.164 2024-05-27T07:53:48.485 INFO:tasks.workunit.client.0.smithi114.stdout: Unlink 35847 1.665 186.734 2024-05-27T07:53:48.485 INFO:tasks.workunit.client.0.smithi114.stdout: Deltree 4 85.378 101.041 2024-05-27T07:53:48.485 INFO:tasks.workunit.client.0.smithi114.stdout: Mkdir 2 0.009 0.012 2024-05-27T07:53:48.485 INFO:tasks.workunit.client.0.smithi114.stdout: Qpathinfo 160885 1.066 6753.826 2024-05-27T07:53:48.486 INFO:tasks.workunit.client.0.smithi114.stdout: Qfileinfo 28180 0.001 0.063 2024-05-27T07:53:48.486 INFO:tasks.workunit.client.0.smithi114.stdout: Qfsinfo 29503 0.197 27.326 2024-05-27T07:53:48.486 INFO:tasks.workunit.client.0.smithi114.stdout: Sfileinfo 14462 0.027 46.449 2024-05-27T07:53:48.486 INFO:tasks.workunit.client.0.smithi114.stdout: Find 62205 0.201 5110.493 2024-05-27T07:53:48.486 INFO:tasks.workunit.client.0.smithi114.stdout: WriteX 88402 0.236 668.418 2024-05-27T07:53:48.486 INFO:tasks.workunit.client.0.smithi114.stdout: ReadX 278338 0.022 94.898 2024-05-27T07:53:48.486 INFO:tasks.workunit.client.0.smithi114.stdout: LockX 578 0.886 14.903 2024-05-27T07:53:48.486 INFO:tasks.workunit.client.0.smithi114.stdout: UnlockX 578 0.776 15.994 2024-05-27T07:53:48.486 INFO:tasks.workunit.client.0.smithi114.stdout: Flush 12439 6.962 415.261 2024-05-27T07:53:48.486 INFO:tasks.workunit.client.0.smithi114.stdout: 2024-05-27T07:53:48.486 INFO:tasks.workunit.client.0.smithi114.stdout:Throughput 9.29026 MB/sec 1 clients 1 procs max_latency=6754.269 ms 2024-05-27T07:53:48.487 INFO:tasks.workunit.client.0.smithi114.stderr:+ dbench 10 2024-05-27T07:53:48.487 INFO:tasks.workunit.client.0.smithi114.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-05-27T07:53:48.487 INFO:tasks.workunit.client.0.smithi114.stdout: 2024-05-27T07:53:48.487 INFO:tasks.workunit.client.0.smithi114.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-05-27T07:53:48.487 INFO:tasks.workunit.client.0.smithi114.stdout:8 of 10 processes prepared for launch 0 sec 2024-05-27T07:53:48.586 INFO:tasks.workunit.client.0.smithi114.stdout:10 of 10 processes prepared for launch 0 sec 2024-05-27T07:53:48.587 INFO:tasks.workunit.client.0.smithi114.stdout:releasing clients 2024-05-27T07:53:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.450+0000 7f2b8a115700 1 -- 172.21.15.114:0/443548172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b841039d0 msgr2=0x7f2b84103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.450+0000 7f2b8a115700 1 --2- 172.21.15.114:0/443548172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b841039d0 0x7f2b84103de0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0098f0 tx=0x7f2b6c009c00 comp rx=0 tx=0).stop 2024-05-27T07:53:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.450+0000 7f2b8a115700 1 -- 172.21.15.114:0/443548172 shutdown_connections 2024-05-27T07:53:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.450+0000 7f2b8a115700 1 --2- 172.21.15.114:0/443548172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b84104320 0x7f2b84110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.450+0000 7f2b8a115700 1 --2- 172.21.15.114:0/443548172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b841092c0 0x7f2b841096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.450+0000 7f2b8a115700 1 --2- 172.21.15.114:0/443548172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b841039d0 0x7f2b84103de0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.450+0000 7f2b8a115700 1 -- 172.21.15.114:0/443548172 >> 172.21.15.114:0/443548172 conn(0x7f2b840fe4f0 msgr2=0x7f2b84100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.450+0000 7f2b8a115700 1 -- 172.21.15.114:0/443548172 shutdown_connections 2024-05-27T07:53:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.450+0000 7f2b8a115700 1 -- 172.21.15.114:0/443548172 wait complete. 2024-05-27T07:53:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b8a115700 1 Processor -- start 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b8a115700 1 -- start start 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b8a115700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b841039d0 0x7f2b84104d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b8a115700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b84104320 0x7f2b84105290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b8a115700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b841092c0 0x7f2b841057d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b8a115700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b84113340 con 0x7f2b84104320 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b8a115700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2b841131c0 con 0x7f2b841039d0 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b8a115700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b841134c0 con 0x7f2b841092c0 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b7bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b84104320 0x7f2b84105290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b7bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b84104320 0x7f2b84105290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58170/0 (socket says 172.21.15.114:58170) 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b7bfff700 1 -- 172.21.15.114:0/4036079734 learned_addr learned my addr 172.21.15.114:0/4036079734 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b83fff700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b841092c0 0x7f2b841057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b837fe700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b841039d0 0x7f2b84104d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b7bfff700 1 -- 172.21.15.114:0/4036079734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b841092c0 msgr2=0x7f2b841057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b7bfff700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b841092c0 0x7f2b841057d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b7bfff700 1 -- 172.21.15.114:0/4036079734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b841039d0 msgr2=0x7f2b84104d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b7bfff700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b841039d0 0x7f2b84104d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b7bfff700 1 -- 172.21.15.114:0/4036079734 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b841a92e0 con 0x7f2b84104320 2024-05-27T07:53:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.451+0000 7f2b83fff700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b841092c0 0x7f2b841057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.452+0000 7f2b7bfff700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b84104320 0x7f2b84105290 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c009230 tx=0x7f2b6c00f990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.452+0000 7f2b817fa700 1 -- 172.21.15.114:0/4036079734 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c0050c0 con 0x7f2b84104320 2024-05-27T07:53:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.452+0000 7f2b817fa700 1 -- 172.21.15.114:0/4036079734 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b6c00b560 con 0x7f2b84104320 2024-05-27T07:53:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.452+0000 7f2b8a115700 1 -- 172.21.15.114:0/4036079734 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b841a9570 con 0x7f2b84104320 2024-05-27T07:53:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.452+0000 7f2b817fa700 1 -- 172.21.15.114:0/4036079734 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c003a40 con 0x7f2b84104320 2024-05-27T07:53:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.452+0000 7f2b8a115700 1 -- 172.21.15.114:0/4036079734 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b841a9a20 con 0x7f2b84104320 2024-05-27T07:53:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.456+0000 7f2b817fa700 1 -- 172.21.15.114:0/4036079734 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f2b6c00ba10 con 0x7f2b84104320 2024-05-27T07:53:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.456+0000 7f2b98b2ab80 1 Processor -- start 2024-05-27T07:53:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.456+0000 7f2b817fa700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b700b2a70 0x7f2b700b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.456+0000 7f2b98b2ab80 1 -- start start 2024-05-27T07:53:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b817fa700 1 -- 172.21.15.114:0/4036079734 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f2b6c0e38d0 con 0x7f2b84104320 2024-05-27T07:53:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b837fe700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b700b2a70 0x7f2b700b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b98b2ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d041ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b98b2ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d041f0f0 0x5640d0418c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b98b2ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d04191a0 0x5640d04195b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b98b2ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5640d0346df0 con 0x5640d041f0f0 2024-05-27T07:53:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b98b2ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5640d0336990 con 0x5640d04191a0 2024-05-27T07:53:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b98b2ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5640d033a9d0 con 0x5640d0299ec0 2024-05-27T07:53:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b837fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d041ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b837fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d041ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:41282/0 (socket says 172.21.15.114:41282) 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b7bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d041f0f0 0x5640d0418c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b837fe700 1 -- 172.21.15.114:0/2832861008 learned_addr learned my addr 172.21.15.114:0/2832861008 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b837fe700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b700b2a70 0x7f2b700b4f80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2b74009c30 tx=0x7f2b74011440 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b83fff700 1 --2- 172.21.15.114:0/2832861008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d04191a0 0x5640d04195b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b837fe700 1 -- 172.21.15.114:0/2832861008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d04191a0 msgr2=0x5640d04195b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b837fe700 1 --2- 172.21.15.114:0/2832861008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d04191a0 0x5640d04195b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b837fe700 1 -- 172.21.15.114:0/2832861008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d041f0f0 msgr2=0x5640d0418c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b837fe700 1 --2- 172.21.15.114:0/2832861008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d041f0f0 0x5640d0418c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b837fe700 1 -- 172.21.15.114:0/2832861008 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640d02e0030 con 0x5640d0299ec0 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.457+0000 7f2b83fff700 1 --2- 172.21.15.114:0/2832861008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d04191a0 0x5640d04195b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.458+0000 7f2b837fe700 1 --2- 172.21.15.114:0/2832861008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d041ebb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2b74011e80 tx=0x7f2b7401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f620dc8df668b01 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.458+0000 7f2b79ffb700 1 -- 172.21.15.114:0/2832861008 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74017040 con 0x5640d0299ec0 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.458+0000 7f2b79ffb700 1 -- 172.21.15.114:0/2832861008 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b74019050 con 0x5640d0299ec0 2024-05-27T07:53:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.458+0000 7f2b79ffb700 1 -- 172.21.15.114:0/2832861008 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74005600 con 0x5640d0299ec0 2024-05-27T07:53:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.459+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/2832861008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 msgr2=0x5640d041ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.459+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/2832861008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d041ebb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2b74011e80 tx=0x7f2b7401c000 comp rx=0 tx=0).stop 2024-05-27T07:53:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.459+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/2832861008 shutdown_connections 2024-05-27T07:53:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.459+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/2832861008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d04191a0 0x5640d04195b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.459+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/2832861008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d041ebb0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.459+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/2832861008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d041f0f0 0x5640d0418c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.459+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/2832861008 >> 172.21.15.114:0/2832861008 conn(0x5640d029c7f0 msgr2=0x5640d0413e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.459+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/2832861008 shutdown_connections 2024-05-27T07:53:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.459+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/2832861008 wait complete. 2024-05-27T07:53:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.459+0000 7f2b98b2ab80 1 -- start start 2024-05-27T07:53:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.459+0000 7f2b98b2ab80 1 Processor -- start 2024-05-27T07:53:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b98b2ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d04228e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b98b2ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d04191a0 0x5640d0422e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b98b2ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d041f0f0 0x5640d0423360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b98b2ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5640d023d020 con 0x5640d041f0f0 2024-05-27T07:53:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b98b2ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5640d029ee20 con 0x5640d04191a0 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b98b2ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5640d0425be0 con 0x5640d0299ec0 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b98b2ab80 10 client.0 _ll_register_callbacks cb 0x5640d02cd630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b83fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d041f0f0 0x5640d0423360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b837fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d04228e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b83fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d041f0f0 0x5640d0423360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58182/0 (socket says 172.21.15.114:58182) 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b837fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d04228e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:41290/0 (socket says 172.21.15.114:41290) 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b83fff700 1 -- 172.21.15.114:0/895008131 learned_addr learned my addr 172.21.15.114:0/895008131 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b7bfff700 1 --2- 172.21.15.114:0/895008131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d04191a0 0x5640d0422e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b837fe700 1 -- 172.21.15.114:0/895008131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d04191a0 msgr2=0x5640d0422e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b837fe700 1 --2- 172.21.15.114:0/895008131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d04191a0 0x5640d0422e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b837fe700 1 -- 172.21.15.114:0/895008131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d041f0f0 msgr2=0x5640d0423360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b837fe700 1 --2- 172.21.15.114:0/895008131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d041f0f0 0x5640d0423360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b837fe700 1 -- 172.21.15.114:0/895008131 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640d0286950 con 0x5640d0299ec0 2024-05-27T07:53:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b837fe700 1 --2- 172.21.15.114:0/895008131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d04228e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2b74011840 tx=0x7f2b74005080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=113e788073b8e9fc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.460+0000 7f2b62ffd700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b62ffd700 1 -- 172.21.15.114:0/895008131 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74017050 con 0x5640d0299ec0 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b62ffd700 1 -- 172.21.15.114:0/895008131 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f2b740048f0 con 0x5640d0299ec0 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/895008131 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b74019050 con 0x5640d0299ec0 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b62ffd700 1 -- 172.21.15.114:0/895008131 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f2b74004af0 con 0x5640d0299ec0 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b98b2ab80 10 client.28309 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/895008131 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x5640d02f07e0 con 0x5640d0299ec0 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b62ffd700 1 -- 172.21.15.114:0/895008131 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b74019050 con 0x5640d0299ec0 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b62ffd700 1 -- 172.21.15.114:0/895008131 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f2b74024020 con 0x5640d0299ec0 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b98b2ab80 10 client.28309 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b98b2ab80 10 client.28309 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b98b2ab80 20 client.28309 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/895008131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5640d0432ca0 0x5640d0435130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b98b2ab80 4 client.28309 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.461+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/895008131 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640d0348d40 con 0x5640d0432ca0 2024-05-27T07:53:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.462+0000 7f2b7bfff700 1 --2- 172.21.15.114:0/895008131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5640d0432ca0 0x5640d0435130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.462+0000 7f2b62ffd700 10 client.28309 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.462+0000 7f2b7bfff700 1 --2- 172.21.15.114:0/895008131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5640d0432ca0 0x5640d0435130 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebc21faafc556c65 server_cookie=ba0b98673659d072 in_seq=0 out_seq=0 2024-05-27T07:53:49.467 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcb1e450700 time 2024-05-27T07:53:49.113440+0000 2024-05-27T07:53:49.467 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:53:49.467 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:49.467 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcb30c95ba6] 2024-05-27T07:53:49.467 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb30c95dd1] 2024-05-27T07:53:49.467 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55792149d23d] 2024-05-27T07:53:49.467 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557921488971] 2024-05-27T07:53:49.467 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 5: (SnapRealm::check_cache() const+0x13e) [0x557921486fa0] 2024-05-27T07:53:49.467 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557921487651] 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55792141c347] 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55792111887e] 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55792111df2f] 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557921128969] 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 11: (MDLog::_replay_thread()+0x1f38) [0x5579214c148c] 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5579214ca9cb] 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 13: (Thread::entry_wrapper()+0x43) [0x7fcb30c6472d] 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcb30c64749] 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcb2f8081ca] 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 16: clone() 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.115+0000 7fcb1e450700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcb1e450700 time 2024-05-27T07:53:49.113440+0000 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.468 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcb30c95ba6] 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb30c95dd1] 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55792149d23d] 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557921488971] 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 5: (SnapRealm::check_cache() const+0x13e) [0x557921486fa0] 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557921487651] 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55792141c347] 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55792111887e] 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55792111df2f] 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557921128969] 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 11: (MDLog::_replay_thread()+0x1f38) [0x5579214c148c] 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5579214ca9cb] 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 13: (Thread::entry_wrapper()+0x43) [0x7fcb30c6472d] 2024-05-27T07:53:49.469 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcb30c64749] 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcb2f8081ca] 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 16: clone() 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: *** Caught signal (Aborted) ** 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: in thread 7fcb1e450700 thread_name:md_log_replay 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcb2f812d20] 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2: gsignal() 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 3: abort() 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcb30c95ced] 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb30c95dd1] 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55792149d23d] 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557921488971] 2024-05-27T07:53:49.470 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 8: (SnapRealm::check_cache() const+0x13e) [0x557921486fa0] 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557921487651] 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55792141c347] 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55792111887e] 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55792111df2f] 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557921128969] 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 14: (MDLog::_replay_thread()+0x1f38) [0x5579214c148c] 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5579214ca9cb] 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 16: (Thread::entry_wrapper()+0x43) [0x7fcb30c6472d] 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcb30c64749] 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcb2f8081ca] 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 19: clone() 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.117+0000 7fcb1e450700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: in thread 7fcb1e450700 thread_name:md_log_replay 2024-05-27T07:53:49.471 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.472 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:49.472 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcb2f812d20] 2024-05-27T07:53:49.472 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2: gsignal() 2024-05-27T07:53:49.472 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 3: abort() 2024-05-27T07:53:49.472 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcb30c95ced] 2024-05-27T07:53:49.472 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb30c95dd1] 2024-05-27T07:53:49.472 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55792149d23d] 2024-05-27T07:53:49.472 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557921488971] 2024-05-27T07:53:49.472 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 8: (SnapRealm::check_cache() const+0x13e) [0x557921486fa0] 2024-05-27T07:53:49.472 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557921487651] 2024-05-27T07:53:49.472 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55792141c347] 2024-05-27T07:53:49.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b62ffd700 1 -- 172.21.15.114:0/895008131 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5640d0348d40 con 0x5640d0432ca0 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b62ffd700 10 client.28309 handle_command_reply: tid=0 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b98b2ab80 1 client.28309 shutdown 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/895008131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 msgr2=0x5640d04228e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/895008131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d04228e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2b74011840 tx=0x7f2b74005080 comp rx=0 tx=0).stop 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/895008131 shutdown_connections 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/895008131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d041f0f0 0x5640d0423360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/895008131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d04228e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/895008131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5640d0432ca0 0x5640d0435130 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/895008131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d04191a0 0x5640d0422e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b62ffd700 0 client.28309 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/895008131 >> 172.21.15.114:0/895008131 conn(0x5640d029c7f0 msgr2=0x5640d0417c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/895008131 shutdown_connections 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.464+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/895008131 wait complete. 2024-05-27T07:53:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.465+0000 7f2b98b2ab80 20 client.28309 trim_cache size 0 max 16384 2024-05-27T07:53:49.474 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55792111887e] 2024-05-27T07:53:49.474 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55792111df2f] 2024-05-27T07:53:49.474 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557921128969] 2024-05-27T07:53:49.474 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 14: (MDLog::_replay_thread()+0x1f38) [0x5579214c148c] 2024-05-27T07:53:49.475 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5579214ca9cb] 2024-05-27T07:53:49.475 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 16: (Thread::entry_wrapper()+0x43) [0x7fcb30c6472d] 2024-05-27T07:53:49.475 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcb30c64749] 2024-05-27T07:53:49.475 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcb2f8081ca] 2024-05-27T07:53:49.475 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 19: clone() 2024-05-27T07:53:49.475 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:53:49.475 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.475 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: -1> 2024-05-27T07:53:49.115+0000 7fcb1e450700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcb1e450700 time 2024-05-27T07:53:49.113440+0000 2024-05-27T07:53:49.475 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:53:49.475 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.475 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:49.475 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcb30c95ba6] 2024-05-27T07:53:49.475 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb30c95dd1] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55792149d23d] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557921488971] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 5: (SnapRealm::check_cache() const+0x13e) [0x557921486fa0] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557921487651] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55792141c347] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55792111887e] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55792111df2f] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557921128969] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 11: (MDLog::_replay_thread()+0x1f38) [0x5579214c148c] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5579214ca9cb] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 13: (Thread::entry_wrapper()+0x43) [0x7fcb30c6472d] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcb30c64749] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcb2f8081ca] 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 16: clone() 2024-05-27T07:53:49.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 0> 2024-05-27T07:53:49.117+0000 7fcb1e450700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:53:49.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: in thread 7fcb1e450700 thread_name:md_log_replay 2024-05-27T07:53:49.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:49.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcb2f812d20] 2024-05-27T07:53:49.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2: gsignal() 2024-05-27T07:53:49.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 3: abort() 2024-05-27T07:53:49.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcb30c95ced] 2024-05-27T07:53:49.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb30c95dd1] 2024-05-27T07:53:49.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55792149d23d] 2024-05-27T07:53:49.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557921488971] 2024-05-27T07:53:49.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 8: (SnapRealm::check_cache() const+0x13e) [0x557921486fa0] 2024-05-27T07:53:49.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557921487651] 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55792141c347] 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55792111887e] 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55792111df2f] 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557921128969] 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 14: (MDLog::_replay_thread()+0x1f38) [0x5579214c148c] 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5579214ca9cb] 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 16: (Thread::entry_wrapper()+0x43) [0x7fcb30c6472d] 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcb30c64749] 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcb2f8081ca] 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 19: clone() 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: -1> 2024-05-27T07:53:49.115+0000 7fcb1e450700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcb1e450700 time 2024-05-27T07:53:49.113440+0000 2024-05-27T07:53:49.478 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcb30c95ba6] 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb30c95dd1] 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55792149d23d] 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557921488971] 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 5: (SnapRealm::check_cache() const+0x13e) [0x557921486fa0] 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557921487651] 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55792141c347] 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55792111887e] 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55792111df2f] 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557921128969] 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 11: (MDLog::_replay_thread()+0x1f38) [0x5579214c148c] 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5579214ca9cb] 2024-05-27T07:53:49.479 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 13: (Thread::entry_wrapper()+0x43) [0x7fcb30c6472d] 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcb30c64749] 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcb2f8081ca] 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 16: clone() 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 0> 2024-05-27T07:53:49.117+0000 7fcb1e450700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: in thread 7fcb1e450700 thread_name:md_log_replay 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcb2f812d20] 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2: gsignal() 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 3: abort() 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcb30c95ced] 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb30c95dd1] 2024-05-27T07:53:49.480 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55792149d23d] 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557921488971] 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 8: (SnapRealm::check_cache() const+0x13e) [0x557921486fa0] 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557921487651] 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55792141c347] 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55792111887e] 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55792111df2f] 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557921128969] 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 14: (MDLog::_replay_thread()+0x1f38) [0x5579214c148c] 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5579214ca9cb] 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 16: (Thread::entry_wrapper()+0x43) [0x7fcb30c6472d] 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcb30c64749] 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcb2f8081ca] 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 19: clone() 2024-05-27T07:53:49.481 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:53:49.482 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73346]: 2024-05-27T07:53:49.482 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:49 smithi163 podman[73679]: 2024-05-27 07:53:49.177128682 +0000 UTC m=+0.020992723 container died 5ec1acc48bde00ae0745b5647b4b7ac4c09611a87a75f275552132e579a1c525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS) 2024-05-27T07:53:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.487+0000 7f2b98b2ab80 1 Processor -- start 2024-05-27T07:53:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.487+0000 7f2b98b2ab80 1 -- start start 2024-05-27T07:53:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.487+0000 7f2b98b2ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d0418c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.487+0000 7f2b98b2ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 0x5640d041dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.487+0000 7f2b98b2ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d041f0f0 0x5640d042ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.487+0000 7f2b98b2ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b841b3370 con 0x5640d04191a0 2024-05-27T07:53:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.487+0000 7f2b98b2ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2b841b3a40 con 0x5640d041f0f0 2024-05-27T07:53:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.487+0000 7f2b98b2ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b841b21f0 con 0x5640d0299ec0 2024-05-27T07:53:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.487+0000 7f2b7bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 0x5640d041dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.487+0000 7f2b7bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 0x5640d041dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58184/0 (socket says 172.21.15.114:58184) 2024-05-27T07:53:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.487+0000 7f2b7bfff700 1 -- 172.21.15.114:0/3511943839 learned_addr learned my addr 172.21.15.114:0/3511943839 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.487+0000 7f2b837fe700 1 --2- 172.21.15.114:0/3511943839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d0418c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.487+0000 7f2b83fff700 1 --2- 172.21.15.114:0/3511943839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d041f0f0 0x5640d042ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.488+0000 7f2b7bfff700 1 -- 172.21.15.114:0/3511943839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 msgr2=0x5640d0418c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.488+0000 7f2b7bfff700 1 --2- 172.21.15.114:0/3511943839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d0418c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.488+0000 7f2b7bfff700 1 -- 172.21.15.114:0/3511943839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d041f0f0 msgr2=0x5640d042ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.488+0000 7f2b7bfff700 1 --2- 172.21.15.114:0/3511943839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d041f0f0 0x5640d042ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.488+0000 7f2b7bfff700 1 -- 172.21.15.114:0/3511943839 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640d0420e50 con 0x5640d04191a0 2024-05-27T07:53:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.488+0000 7f2b7bfff700 1 --2- 172.21.15.114:0/3511943839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 0x5640d041dec0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0aa540 tx=0x7f2b6c0b1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe6725dbd2345caf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.488+0000 7f2b797fa700 1 -- 172.21.15.114:0/3511943839 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c0c3070 con 0x5640d04191a0 2024-05-27T07:53:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.488+0000 7f2b797fa700 1 -- 172.21.15.114:0/3511943839 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b6c0aa350 con 0x5640d04191a0 2024-05-27T07:53:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.488+0000 7f2b797fa700 1 -- 172.21.15.114:0/3511943839 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c0b10b0 con 0x5640d04191a0 2024-05-27T07:53:49.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.489+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/3511943839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 msgr2=0x5640d041dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:49.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.489+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/3511943839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 0x5640d041dec0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0aa540 tx=0x7f2b6c0b1040 comp rx=0 tx=0).stop 2024-05-27T07:53:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.489+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/3511943839 shutdown_connections 2024-05-27T07:53:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.489+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/3511943839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d041f0f0 0x5640d042ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.489+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/3511943839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d0418c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.489+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/3511943839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 0x5640d041dec0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.489+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/3511943839 >> 172.21.15.114:0/3511943839 conn(0x5640d029c7f0 msgr2=0x5640d040d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.489+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/3511943839 shutdown_connections 2024-05-27T07:53:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.489+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/3511943839 wait complete. 2024-05-27T07:53:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.489+0000 7f2b98b2ab80 1 -- start start 2024-05-27T07:53:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b98b2ab80 1 Processor -- start 2024-05-27T07:53:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b98b2ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d043a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b98b2ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 0x5640d043abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b98b2ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d041f0f0 0x5640d043b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b98b2ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b841b1b20 con 0x5640d04191a0 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b98b2ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2b841b0730 con 0x5640d041f0f0 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b98b2ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b841b0d80 con 0x5640d0299ec0 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b98b2ab80 10 client.0 _ll_register_callbacks cb 0x5640d02cd630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b7bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 0x5640d043abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b837fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d043a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b7bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 0x5640d043abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58190/0 (socket says 172.21.15.114:58190) 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b7bfff700 1 -- 172.21.15.114:0/1999721811 learned_addr learned my addr 172.21.15.114:0/1999721811 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b83fff700 1 --2- 172.21.15.114:0/1999721811 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d041f0f0 0x5640d043b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b7bfff700 1 -- 172.21.15.114:0/1999721811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 msgr2=0x5640d043a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b7bfff700 1 --2- 172.21.15.114:0/1999721811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d043a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b7bfff700 1 -- 172.21.15.114:0/1999721811 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d041f0f0 msgr2=0x5640d043b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b7bfff700 1 --2- 172.21.15.114:0/1999721811 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d041f0f0 0x5640d043b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b7bfff700 1 -- 172.21.15.114:0/1999721811 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640d049c990 con 0x5640d04191a0 2024-05-27T07:53:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.490+0000 7f2b837fe700 1 --2- 172.21.15.114:0/1999721811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d043a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b7bfff700 1 --2- 172.21.15.114:0/1999721811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 0x5640d043abc0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0e0cd0 tx=0x7f2b6c0aa2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd4cf1bf1819d7b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b62ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b62ffd700 1 -- 172.21.15.114:0/1999721811 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c088020 con 0x5640d04191a0 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b62ffd700 1 -- 172.21.15.114:0/1999721811 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f2b6c0a6e30 con 0x5640d04191a0 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/1999721811 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b6c0aa350 con 0x5640d04191a0 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b62ffd700 1 -- 172.21.15.114:0/1999721811 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f2b6c0c3040 con 0x5640d04191a0 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b98b2ab80 10 client.18789 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/1999721811 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x5640d049dc30 con 0x5640d04191a0 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b62ffd700 1 -- 172.21.15.114:0/1999721811 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b6c0aa350 con 0x5640d04191a0 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b62ffd700 1 -- 172.21.15.114:0/1999721811 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f2b6c0a6ad0 con 0x5640d04191a0 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b98b2ab80 10 client.18789 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b98b2ab80 10 client.18789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b98b2ab80 20 client.18789 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/1999721811 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5640d049e170 0x5640d04a0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b98b2ab80 4 client.18789 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:53:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.491+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/1999721811 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5640d0342520 con 0x5640d049e170 2024-05-27T07:53:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.492+0000 7f2b837fe700 1 --2- 172.21.15.114:0/1999721811 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5640d049e170 0x5640d04a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.492+0000 7f2b62ffd700 10 client.18789 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.492+0000 7f2b837fe700 1 --2- 172.21.15.114:0/1999721811 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5640d049e170 0x5640d04a0590 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a75719ff717dfb32 server_cookie=218a2b39e60f90d2 in_seq=0 out_seq=0 2024-05-27T07:53:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.508+0000 7f2b62ffd700 1 -- 172.21.15.114:0/1999721811 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5640d0342520 con 0x5640d049e170 2024-05-27T07:53:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.508+0000 7f2b62ffd700 10 client.18789 handle_command_reply: tid=0 2024-05-27T07:53:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.508+0000 7f2b98b2ab80 1 client.18789 shutdown 2024-05-27T07:53:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/1999721811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 msgr2=0x5640d043abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/1999721811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 0x5640d043abc0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0e0cd0 tx=0x7f2b6c0aa2b0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/1999721811 shutdown_connections 2024-05-27T07:53:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/1999721811 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5640d049e170 0x5640d04a0590 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/1999721811 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d041f0f0 0x5640d043b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/1999721811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640d0299ec0 0x5640d043a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b98b2ab80 1 --2- 172.21.15.114:0/1999721811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5640d04191a0 0x5640d043abc0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b62ffd700 0 client.18789 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/1999721811 >> 172.21.15.114:0/1999721811 conn(0x5640d029c7f0 msgr2=0x5640d041cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/1999721811 shutdown_connections 2024-05-27T07:53:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b98b2ab80 1 -- 172.21.15.114:0/1999721811 wait complete. 2024-05-27T07:53:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b98b2ab80 20 client.18789 trim_cache size 0 max 16384 2024-05-27T07:53:49.510 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:53:49.510 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:53:49.511 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "d9e1dfce-f2e3-44a2-b243-1b4abd951048", 2024-05-27T07:53:49.511 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:53:49.511 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:53:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b8a115700 1 -- 172.21.15.114:0/4036079734 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b700b2a70 msgr2=0x7f2b700b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b8a115700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b700b2a70 0x7f2b700b4f80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2b74009c30 tx=0x7f2b74011440 comp rx=0 tx=0).stop 2024-05-27T07:53:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b8a115700 1 -- 172.21.15.114:0/4036079734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b84104320 msgr2=0x7f2b84105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.509+0000 7f2b8a115700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b84104320 0x7f2b84105290 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c009230 tx=0x7f2b6c00f990 comp rx=0 tx=0).stop 2024-05-27T07:53:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.510+0000 7f2b8a115700 1 -- 172.21.15.114:0/4036079734 shutdown_connections 2024-05-27T07:53:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.510+0000 7f2b8a115700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b841092c0 0x7f2b841057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.510+0000 7f2b8a115700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2b700b2a70 0x7f2b700b4f80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.510+0000 7f2b8a115700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b841039d0 0x7f2b84104d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.510+0000 7f2b8a115700 1 --2- 172.21.15.114:0/4036079734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b84104320 0x7f2b84105290 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.510+0000 7f2b8a115700 1 -- 172.21.15.114:0/4036079734 >> 172.21.15.114:0/4036079734 conn(0x7f2b840fe4f0 msgr2=0x7f2b84102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.510+0000 7f2b8a115700 1 -- 172.21.15.114:0/4036079734 shutdown_connections 2024-05-27T07:53:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:49.510+0000 7f2b8a115700 1 -- 172.21.15.114:0/4036079734 wait complete. 2024-05-27T07:53:49.586 INFO:tasks.workunit.client.0.smithi114.stdout: 10 346 124.47 MB/sec warmup 1 sec latency 112.970 ms 2024-05-27T07:53:49.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:49 smithi163 ceph-mon[36451]: pgmap v473: 129 pgs: 129 active+clean; 74 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 265 op/s 2024-05-27T07:53:49.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:49 smithi186 ceph-mon[36647]: pgmap v473: 129 pgs: 129 active+clean; 74 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 265 op/s 2024-05-27T07:53:49.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:49 smithi114 ceph-mon[32038]: pgmap v473: 129 pgs: 129 active+clean; 74 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 265 op/s 2024-05-27T07:53:50.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:53:50.586 INFO:tasks.workunit.client.0.smithi114.stdout: 10 682 101.87 MB/sec warmup 2 sec latency 155.741 ms 2024-05-27T07:53:50.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:50 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:53:50.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:50 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:50.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:50 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:53:50.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:50 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:50.661 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:50 smithi163 podman[73679]: 2024-05-27 07:53:50.312292553 +0000 UTC m=+1.156156639 container remove 5ec1acc48bde00ae0745b5647b4b7ac4c09611a87a75f275552132e579a1c525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:53:50.661 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:50 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:53:50.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:50 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:53:50.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:50 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:50.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:50 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:53:50.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:50 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:50.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:50 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:53:50.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:50 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:50.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:50 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:53:50.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:50 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:53:50.917 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:50.976 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:50 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-05-27T07:53:50.976 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:53:50 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Consumed 1.838s CPU time. 2024-05-27T07:53:51.586 INFO:tasks.workunit.client.0.smithi114.stdout: 10 694 68.22 MB/sec warmup 3 sec latency 1026.023 ms 2024-05-27T07:53:51.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:51 smithi114 ceph-mon[32038]: pgmap v474: 129 pgs: 129 active+clean; 74 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 232 op/s 2024-05-27T07:53:51.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:51 smithi163 ceph-mon[36451]: pgmap v474: 129 pgs: 129 active+clean; 74 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 232 op/s 2024-05-27T07:53:51.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:51 smithi186 ceph-mon[36647]: pgmap v474: 129 pgs: 129 active+clean; 74 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 232 op/s 2024-05-27T07:53:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.451+0000 7f3304ee2640 1 -- 172.21.15.114:0/435509809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3300108020 msgr2=0x7f33001084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.451+0000 7f3304ee2640 1 --2- 172.21.15.114:0/435509809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3300108020 0x7f33001084b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f32f8009ca0 tx=0x7f32f802a8f0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.457+0000 7f3304ee2640 1 -- 172.21.15.114:0/435509809 shutdown_connections 2024-05-27T07:53:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.457+0000 7f3304ee2640 1 --2- 172.21.15.114:0/435509809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3300108020 0x7f33001084b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.457+0000 7f3304ee2640 1 --2- 172.21.15.114:0/435509809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33001075f0 0x7f3300107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.457+0000 7f3304ee2640 1 --2- 172.21.15.114:0/435509809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3300102ba0 0x7f3300102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.457+0000 7f3304ee2640 1 -- 172.21.15.114:0/435509809 >> 172.21.15.114:0/435509809 conn(0x7f33000781a0 msgr2=0x7f3300078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:52.466 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.465+0000 7f3304ee2640 1 -- 172.21.15.114:0/435509809 shutdown_connections 2024-05-27T07:53:52.467 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.466+0000 7f3304ee2640 1 -- 172.21.15.114:0/435509809 wait complete. 2024-05-27T07:53:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.467+0000 7f3304ee2640 1 Processor -- start 2024-05-27T07:53:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.467+0000 7f3304ee2640 1 -- start start 2024-05-27T07:53:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.467+0000 7f3304ee2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3300102ba0 0x7f3300116e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.467+0000 7f3304ee2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33001075f0 0x7f3300110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.467+0000 7f3304ee2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3300108020 0x7f33001106e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.467+0000 7f3304ee2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f330011a4a0 con 0x7f33001075f0 2024-05-27T07:53:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.467+0000 7f3304ee2640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f330011a320 con 0x7f3300102ba0 2024-05-27T07:53:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.468+0000 7f3304ee2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f330011a620 con 0x7f3300108020 2024-05-27T07:53:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.468+0000 7f32fed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3300108020 0x7f33001106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.468+0000 7f32fed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3300108020 0x7f33001106e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:41326/0 (socket says 172.21.15.114:41326) 2024-05-27T07:53:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.468+0000 7f32fed9d640 1 -- 172.21.15.114:0/2179503598 learned_addr learned my addr 172.21.15.114:0/2179503598 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.468+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33001075f0 0x7f3300110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.468+0000 7f32fed9d640 1 -- 172.21.15.114:0/2179503598 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3300102ba0 msgr2=0x7f3300116e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:53:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.468+0000 7f32fed9d640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3300102ba0 0x7f3300116e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.468+0000 7f32fed9d640 1 -- 172.21.15.114:0/2179503598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33001075f0 msgr2=0x7f3300110020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.468+0000 7f32fed9d640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33001075f0 0x7f3300110020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.468+0000 7f32fed9d640 1 -- 172.21.15.114:0/2179503598 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33001149b0 con 0x7f3300108020 2024-05-27T07:53:52.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.469+0000 7f32fed9d640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3300108020 0x7f33001106e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f32f802aee0 tx=0x7f32f802f520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:52.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.469+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33001075f0 0x7f3300110020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:53:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.469+0000 7f32ef7fe640 1 -- 172.21.15.114:0/2179503598 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f8003c70 con 0x7f3300108020 2024-05-27T07:53:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.470+0000 7f3304ee2640 1 -- 172.21.15.114:0/2179503598 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3300114c40 con 0x7f3300108020 2024-05-27T07:53:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.470+0000 7f3304ee2640 1 -- 172.21.15.114:0/2179503598 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33001151e0 con 0x7f3300108020 2024-05-27T07:53:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.470+0000 7f32ef7fe640 1 -- 172.21.15.114:0/2179503598 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32f802d070 con 0x7f3300108020 2024-05-27T07:53:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.470+0000 7f32ef7fe640 1 -- 172.21.15.114:0/2179503598 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f802fa30 con 0x7f3300108020 2024-05-27T07:53:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.474+0000 7f330bd93740 1 Processor -- start 2024-05-27T07:53:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.474+0000 7f330bd93740 1 -- start start 2024-05-27T07:53:52.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.476+0000 7f330bd93740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97e51b0 0x5632b9b86260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.476+0000 7f330bd93740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97bc2f0 0x5632b9b867a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.476+0000 7f330bd93740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9b86ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.476+0000 7f330bd93740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5632b9af8c30 con 0x5632b97c44d0 2024-05-27T07:53:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.476+0000 7f330bd93740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5632b9ae87d0 con 0x5632b97bc2f0 2024-05-27T07:53:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.476+0000 7f330bd93740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5632b9aec810 con 0x5632b97e51b0 2024-05-27T07:53:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.477+0000 7f32fdd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97e51b0 0x5632b9b86260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.477+0000 7f32fdd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97e51b0 0x5632b9b86260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:41338/0 (socket says 172.21.15.114:41338) 2024-05-27T07:53:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.477+0000 7f32fdd9b640 1 -- 172.21.15.114:0/2062186769 learned_addr learned my addr 172.21.15.114:0/2062186769 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.477+0000 7f32ef7fe640 1 -- 172.21.15.114:0/2179503598 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f32f802fbd0 con 0x7f3300108020 2024-05-27T07:53:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.477+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/2062186769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9b86ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.480+0000 7f32ef7fe640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f32e80b2a20 0x7f32e80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.481+0000 7f32ef7fe640 1 -- 172.21.15.114:0/2179503598 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f32f8103fc0 con 0x7f3300108020 2024-05-27T07:53:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.481+0000 7f32fe59c640 1 --2- 172.21.15.114:0/2062186769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97bc2f0 0x5632b9b867a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.482+0000 7f32fed9d640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f32e80b2a20 0x7f32e80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.482+0000 7f32fdd9b640 1 -- 172.21.15.114:0/2062186769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97bc2f0 msgr2=0x5632b9b867a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.482+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/2062186769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97bc2f0 0x5632b9b867a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.482+0000 7f32fdd9b640 1 -- 172.21.15.114:0/2062186769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 msgr2=0x5632b9b86ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.482+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/2062186769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9b86ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.482+0000 7f32fdd9b640 1 -- 172.21.15.114:0/2062186769 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632b97fd210 con 0x5632b97e51b0 2024-05-27T07:53:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.482+0000 7f32fed9d640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f32e80b2a20 0x7f32e80b4f30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f32f8009dc0 tx=0x7f32f8009400 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.483+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/2062186769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97e51b0 0x5632b9b86260 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f32f0013a30 tx=0x7f32f000a700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=271c528bcc6dea3d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.484+0000 7f32ecff9640 1 -- 172.21.15.114:0/2062186769 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f000ea20 con 0x5632b97e51b0 2024-05-27T07:53:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.484+0000 7f32ecff9640 1 -- 172.21.15.114:0/2062186769 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32f00054d0 con 0x5632b97e51b0 2024-05-27T07:53:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.484+0000 7f32ecff9640 1 -- 172.21.15.114:0/2062186769 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f0004a00 con 0x5632b97e51b0 2024-05-27T07:53:52.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.486+0000 7f330bd93740 1 -- 172.21.15.114:0/2062186769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97e51b0 msgr2=0x5632b9b86260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:52.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.486+0000 7f330bd93740 1 --2- 172.21.15.114:0/2062186769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97e51b0 0x5632b9b86260 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f32f0013a30 tx=0x7f32f000a700 comp rx=0 tx=0).stop 2024-05-27T07:53:52.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.486+0000 7f330bd93740 1 -- 172.21.15.114:0/2062186769 shutdown_connections 2024-05-27T07:53:52.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.486+0000 7f330bd93740 1 --2- 172.21.15.114:0/2062186769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9b86ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.486+0000 7f330bd93740 1 --2- 172.21.15.114:0/2062186769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97bc2f0 0x5632b9b867a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.486+0000 7f330bd93740 1 --2- 172.21.15.114:0/2062186769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97e51b0 0x5632b9b86260 secure :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f32f0013a30 tx=0x7f32f000a700 comp rx=0 tx=0).stop 2024-05-27T07:53:52.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.486+0000 7f330bd93740 1 -- 172.21.15.114:0/2062186769 >> 172.21.15.114:0/2062186769 conn(0x5632b97d9300 msgr2=0x5632b9b82b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:52.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.487+0000 7f330bd93740 1 -- 172.21.15.114:0/2062186769 shutdown_connections 2024-05-27T07:53:52.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.489+0000 7f330bd93740 1 -- 172.21.15.114:0/2062186769 wait complete. 2024-05-27T07:53:52.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.489+0000 7f330bd93740 1 -- start start 2024-05-27T07:53:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.489+0000 7f330bd93740 1 Processor -- start 2024-05-27T07:53:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.490+0000 7f330bd93740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97bc2f0 0x5632b9ba35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.490+0000 7f330bd93740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9ba3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.490+0000 7f330bd93740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9ba1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.490+0000 7f330bd93740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5632b97ce790 con 0x5632b97c44d0 2024-05-27T07:53:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.490+0000 7f330bd93740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5632b9aa4890 con 0x5632b97a3d80 2024-05-27T07:53:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.490+0000 7f330bd93740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5632b9ba4570 con 0x5632b97bc2f0 2024-05-27T07:53:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.490+0000 7f330bd93740 10 client.0 _ll_register_callbacks cb 0x5632b97fca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:52.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.491+0000 7f32fe59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9ba3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.491+0000 7f32fe59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9ba3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58224/0 (socket says 172.21.15.114:58224) 2024-05-27T07:53:52.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.491+0000 7f32fe59c640 1 -- 172.21.15.114:0/3117226131 learned_addr learned my addr 172.21.15.114:0/3117226131 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:52.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.491+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/3117226131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9ba1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.491+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/3117226131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97bc2f0 0x5632b9ba35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.492+0000 7f32fe59c640 1 -- 172.21.15.114:0/3117226131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97bc2f0 msgr2=0x5632b9ba35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:52.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.492+0000 7f32fe59c640 1 --2- 172.21.15.114:0/3117226131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97bc2f0 0x5632b9ba35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.492+0000 7f32fe59c640 1 -- 172.21.15.114:0/3117226131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 msgr2=0x5632b9ba1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:52.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.492+0000 7f32fe59c640 1 --2- 172.21.15.114:0/3117226131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9ba1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.492+0000 7f32fe59c640 1 -- 172.21.15.114:0/3117226131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632b98c9620 con 0x5632b97c44d0 2024-05-27T07:53:52.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.494+0000 7f32fe59c640 1 --2- 172.21.15.114:0/3117226131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9ba3b30 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f32f4006aa0 tx=0x7f32f400d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0fc6eb195c7fa3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:52.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.494+0000 7f32e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.499+0000 7f32e5ffb640 1 -- 172.21.15.114:0/3117226131 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f4014430 con 0x5632b97c44d0 2024-05-27T07:53:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.500+0000 7f32e5ffb640 1 -- 172.21.15.114:0/3117226131 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f32f40145d0 con 0x5632b97c44d0 2024-05-27T07:53:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.500+0000 7f330bd93740 1 -- 172.21.15.114:0/3117226131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32f00054d0 con 0x5632b97c44d0 2024-05-27T07:53:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.501+0000 7f32e5ffb640 1 -- 172.21.15.114:0/3117226131 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f32f40147d0 con 0x5632b97c44d0 2024-05-27T07:53:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.502+0000 7f330bd93740 10 client.18798 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:52.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.502+0000 7f330bd93740 1 -- 172.21.15.114:0/3117226131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x5632b9bb0a40 con 0x5632b97c44d0 2024-05-27T07:53:52.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.502+0000 7f32e5ffb640 1 -- 172.21.15.114:0/3117226131 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32f00054d0 con 0x5632b97c44d0 2024-05-27T07:53:52.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.502+0000 7f32e5ffb640 1 -- 172.21.15.114:0/3117226131 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f32f4013070 con 0x5632b97c44d0 2024-05-27T07:53:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.503+0000 7f330bd93740 10 client.18798 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.503+0000 7f330bd93740 10 client.18798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.503+0000 7f330bd93740 20 client.18798 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.503+0000 7f330bd93740 1 --2- 172.21.15.114:0/3117226131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5632b97791c0 0x5632b9bb4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.503+0000 7f330bd93740 4 client.18798 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.503+0000 7f330bd93740 1 -- 172.21.15.114:0/3117226131 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632b9afab80 con 0x5632b97791c0 2024-05-27T07:53:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.504+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/3117226131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5632b97791c0 0x5632b9bb4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.506+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/3117226131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5632b97791c0 0x5632b9bb4420 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8803beab3be017f server_cookie=f35333cf6993f435 in_seq=0 out_seq=0 2024-05-27T07:53:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.506+0000 7f32e5ffb640 10 client.18798 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.509+0000 7f32e5ffb640 1 -- 172.21.15.114:0/3117226131 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5632b9afab80 con 0x5632b97791c0 2024-05-27T07:53:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.509+0000 7f32e5ffb640 10 client.18798 handle_command_reply: tid=0 2024-05-27T07:53:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.509+0000 7f330bd93740 1 client.18798 shutdown 2024-05-27T07:53:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.510+0000 7f330bd93740 1 -- 172.21.15.114:0/3117226131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 msgr2=0x5632b9ba3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.510+0000 7f330bd93740 1 --2- 172.21.15.114:0/3117226131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9ba3b30 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f32f4006aa0 tx=0x7f32f400d5f0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.512+0000 7f330bd93740 1 -- 172.21.15.114:0/3117226131 shutdown_connections 2024-05-27T07:53:52.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.512+0000 7f330bd93740 1 --2- 172.21.15.114:0/3117226131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5632b97791c0 0x5632b9bb4420 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.512+0000 7f330bd93740 1 --2- 172.21.15.114:0/3117226131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9ba1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.512+0000 7f330bd93740 1 --2- 172.21.15.114:0/3117226131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9ba3b30 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.512+0000 7f330bd93740 1 --2- 172.21.15.114:0/3117226131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97bc2f0 0x5632b9ba35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.512+0000 7f330bd93740 1 -- 172.21.15.114:0/3117226131 >> 172.21.15.114:0/3117226131 conn(0x5632b97d9300 msgr2=0x5632b9b880d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.513+0000 7f32e5ffb640 0 client.18798 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.513+0000 7f330bd93740 1 -- 172.21.15.114:0/3117226131 shutdown_connections 2024-05-27T07:53:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.513+0000 7f330bd93740 1 -- 172.21.15.114:0/3117226131 wait complete. 2024-05-27T07:53:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.514+0000 7f330bd93740 20 client.18798 trim_cache size 0 max 16384 2024-05-27T07:53:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.547+0000 7f330bd93740 1 Processor -- start 2024-05-27T07:53:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.549+0000 7f330bd93740 1 -- start start 2024-05-27T07:53:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.550+0000 7f330bd93740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9b887e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.550+0000 7f330bd93740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97bc2f0 0x5632b9b9b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.550+0000 7f330bd93740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9b9b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.550+0000 7f330bd93740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f32e0010ba0 con 0x5632b97c44d0 2024-05-27T07:53:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.550+0000 7f330bd93740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f32e0011270 con 0x5632b97a3d80 2024-05-27T07:53:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.550+0000 7f330bd93740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f32e000fa40 con 0x5632b97bc2f0 2024-05-27T07:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.552+0000 7f32fdd9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9b887e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.552+0000 7f32fdd9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9b887e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:46228/0 (socket says 172.21.15.114:46228) 2024-05-27T07:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.552+0000 7f32fdd9b640 1 -- 172.21.15.114:0/2151280374 learned_addr learned my addr 172.21.15.114:0/2151280374 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.554+0000 7f32fe59c640 1 --2- 172.21.15.114:0/2151280374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97bc2f0 0x5632b9b9b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.555+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/2151280374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9b9b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.555+0000 7f32fdd9b640 1 -- 172.21.15.114:0/2151280374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97bc2f0 msgr2=0x5632b9b9b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.555+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/2151280374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97bc2f0 0x5632b9b9b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.555+0000 7f32fdd9b640 1 -- 172.21.15.114:0/2151280374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 msgr2=0x5632b9b9b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.555+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/2151280374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9b9b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.555+0000 7f32fdd9b640 1 -- 172.21.15.114:0/2151280374 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632b9aaa6c0 con 0x5632b97a3d80 2024-05-27T07:53:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.558+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/2151280374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9b887e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f32f0005670 tx=0x7f32f0013600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e40d8ab0d9ed52c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.558+0000 7f32e7fff640 1 -- 172.21.15.114:0/2151280374 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f0033030 con 0x5632b97a3d80 2024-05-27T07:53:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.558+0000 7f32e7fff640 1 -- 172.21.15.114:0/2151280374 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32f0013110 con 0x5632b97a3d80 2024-05-27T07:53:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.558+0000 7f32e7fff640 1 -- 172.21.15.114:0/2151280374 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f002f860 con 0x5632b97a3d80 2024-05-27T07:53:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.560+0000 7f330bd93740 1 -- 172.21.15.114:0/2151280374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 msgr2=0x5632b9b887e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.560+0000 7f330bd93740 1 --2- 172.21.15.114:0/2151280374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9b887e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f32f0005670 tx=0x7f32f0013600 comp rx=0 tx=0).stop 2024-05-27T07:53:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.564+0000 7f330bd93740 1 -- 172.21.15.114:0/2151280374 shutdown_connections 2024-05-27T07:53:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.564+0000 7f330bd93740 1 --2- 172.21.15.114:0/2151280374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97c44d0 0x5632b9b9b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.564+0000 7f330bd93740 1 --2- 172.21.15.114:0/2151280374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97bc2f0 0x5632b9b9b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.564+0000 7f330bd93740 1 --2- 172.21.15.114:0/2151280374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9b887e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.564+0000 7f330bd93740 1 -- 172.21.15.114:0/2151280374 >> 172.21.15.114:0/2151280374 conn(0x5632b97d9300 msgr2=0x5632b9b880d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.569+0000 7f330bd93740 1 -- 172.21.15.114:0/2151280374 shutdown_connections 2024-05-27T07:53:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.570+0000 7f330bd93740 1 -- 172.21.15.114:0/2151280374 wait complete. 2024-05-27T07:53:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.571+0000 7f330bd93740 1 -- start start 2024-05-27T07:53:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.571+0000 7f330bd93740 1 Processor -- start 2024-05-27T07:53:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.574+0000 7f330bd93740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9ba4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.574+0000 7f330bd93740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97bc2f0 0x5632b9ba4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.574+0000 7f330bd93740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97c44d0 0x5632b9ba52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.574+0000 7f330bd93740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f32e000f370 con 0x5632b97bc2f0 2024-05-27T07:53:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.574+0000 7f330bd93740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f32e000df80 con 0x5632b97a3d80 2024-05-27T07:53:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.574+0000 7f330bd93740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f32e000e5d0 con 0x5632b97c44d0 2024-05-27T07:53:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.574+0000 7f330bd93740 10 client.0 _ll_register_callbacks cb 0x7f32f400b0e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.574+0000 7f32fe59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97bc2f0 0x5632b9ba4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.575+0000 7f32fe59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97bc2f0 0x5632b9ba4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58256/0 (socket says 172.21.15.114:58256) 2024-05-27T07:53:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.575+0000 7f32fe59c640 1 -- 172.21.15.114:0/1480612075 learned_addr learned my addr 172.21.15.114:0/1480612075 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.575+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/1480612075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9ba4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.575+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/1480612075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97c44d0 0x5632b9ba52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.576+0000 7f32fdd9b640 1 -- 172.21.15.114:0/1480612075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97c44d0 msgr2=0x5632b9ba52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.576+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/1480612075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97c44d0 0x5632b9ba52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.576+0000 7f32fdd9b640 1 -- 172.21.15.114:0/1480612075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97bc2f0 msgr2=0x5632b9ba4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.576+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/1480612075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97bc2f0 0x5632b9ba4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.576+0000 7f32fdd9b640 1 -- 172.21.15.114:0/1480612075 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632b9c1aaa0 con 0x5632b97a3d80 2024-05-27T07:53:52.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.585+0000 7f32fdd9b640 1 --2- 172.21.15.114:0/1480612075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9ba4820 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x5632b9b89050 tx=0x7f32f0005da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=495467dbe38ceb69 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:52.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.585+0000 7f32e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:53:52.588 INFO:tasks.workunit.client.0.smithi114.stdout: 10 728 51.25 MB/sec warmup 4 sec latency 1889.582 ms 2024-05-27T07:53:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.591+0000 7f32e5ffb640 1 -- 172.21.15.114:0/1480612075 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f0033070 con 0x5632b97a3d80 2024-05-27T07:53:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.591+0000 7f32e5ffb640 1 -- 172.21.15.114:0/1480612075 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f32f000ea20 con 0x5632b97a3d80 2024-05-27T07:53:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.591+0000 7f330bd93740 1 -- 172.21.15.114:0/1480612075 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32f0013110 con 0x5632b97a3d80 2024-05-27T07:53:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.593+0000 7f32e5ffb640 1 -- 172.21.15.114:0/1480612075 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f32f0040050 con 0x5632b97a3d80 2024-05-27T07:53:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.595+0000 7f330bd93740 10 client.28187 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.595+0000 7f330bd93740 1 -- 172.21.15.114:0/1480612075 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x5632b9c1bc00 con 0x5632b97a3d80 2024-05-27T07:53:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.597+0000 7f32e5ffb640 1 -- 172.21.15.114:0/1480612075 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32f0013110 con 0x5632b97a3d80 2024-05-27T07:53:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.597+0000 7f32e5ffb640 1 -- 172.21.15.114:0/1480612075 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f32f003b940 con 0x5632b97a3d80 2024-05-27T07:53:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.597+0000 7f330bd93740 10 client.28187 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.597+0000 7f330bd93740 10 client.28187 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.597+0000 7f330bd93740 20 client.28187 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.597+0000 7f330bd93740 1 --2- 172.21.15.114:0/1480612075 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5632b9c1c160 0x5632b9c1e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.597+0000 7f330bd93740 4 client.28187 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:53:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.597+0000 7f330bd93740 1 -- 172.21.15.114:0/1480612075 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5632b9af4360 con 0x5632b9c1c160 2024-05-27T07:53:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.598+0000 7f32fe59c640 1 --2- 172.21.15.114:0/1480612075 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5632b9c1c160 0x5632b9c1e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.604+0000 7f32fe59c640 1 --2- 172.21.15.114:0/1480612075 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5632b9c1c160 0x5632b9c1e580 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63a560096df40c0e server_cookie=cd789bf01cac46a0 in_seq=0 out_seq=0 2024-05-27T07:53:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:52.607+0000 7f32e5ffb640 10 client.28187 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:52.660 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:52 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:53:52.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.858+0000 7f37c58ef700 1 -- 172.21.15.114:0/1778045626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c0103300 msgr2=0x7f37c010f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:52.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.858+0000 7f37c58ef700 1 --2- 172.21.15.114:0/1778045626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c0103300 0x7f37c010f880 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f37b000aaa0 tx=0x7f37b000adb0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.858+0000 7f37c58ef700 1 -- 172.21.15.114:0/1778045626 shutdown_connections 2024-05-27T07:53:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.858+0000 7f37c58ef700 1 --2- 172.21.15.114:0/1778045626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c0103300 0x7f37c010f880 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.858+0000 7f37c58ef700 1 --2- 172.21.15.114:0/1778045626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37c01089e0 0x7f37c0108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.858+0000 7f37c58ef700 1 --2- 172.21.15.114:0/1778045626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37c01029b0 0x7f37c0102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.858+0000 7f37c58ef700 1 -- 172.21.15.114:0/1778045626 >> 172.21.15.114:0/1778045626 conn(0x7f37c00fe6e0 msgr2=0x7f37c0100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.859+0000 7f37c58ef700 1 -- 172.21.15.114:0/1778045626 shutdown_connections 2024-05-27T07:53:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.859+0000 7f37c58ef700 1 -- 172.21.15.114:0/1778045626 wait complete. 2024-05-27T07:53:52.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.860+0000 7f37c58ef700 1 Processor -- start 2024-05-27T07:53:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.861+0000 7f37c58ef700 1 -- start start 2024-05-27T07:53:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.861+0000 7f37c58ef700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37c01029b0 0x7f37c019e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.861+0000 7f37c58ef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c0103300 0x7f37c019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.861+0000 7f37c58ef700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37c01089e0 0x7f37c0198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.861+0000 7f37c58ef700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f37c0112330 con 0x7f37c0103300 2024-05-27T07:53:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.861+0000 7f37c58ef700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f37c01121b0 con 0x7f37c01029b0 2024-05-27T07:53:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.861+0000 7f37c58ef700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f37c01124b0 con 0x7f37c01089e0 2024-05-27T07:53:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.862+0000 7f37beffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37c01029b0 0x7f37c019e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.862+0000 7f37beffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37c01029b0 0x7f37c019e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:46256/0 (socket says 172.21.15.114:46256) 2024-05-27T07:53:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.862+0000 7f37beffd700 1 -- 172.21.15.114:0/3291501964 learned_addr learned my addr 172.21.15.114:0/3291501964 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.862+0000 7f37bf7fe700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37c01089e0 0x7f37c0198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.862+0000 7f37be7fc700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c0103300 0x7f37c019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.862+0000 7f37beffd700 1 -- 172.21.15.114:0/3291501964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37c01089e0 msgr2=0x7f37c0198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.862+0000 7f37beffd700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37c01089e0 0x7f37c0198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.862+0000 7f37beffd700 1 -- 172.21.15.114:0/3291501964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c0103300 msgr2=0x7f37c019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.862+0000 7f37beffd700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c0103300 0x7f37c019edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.862+0000 7f37beffd700 1 -- 172.21.15.114:0/3291501964 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c01991b0 con 0x7f37c01029b0 2024-05-27T07:53:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.863+0000 7f37beffd700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37c01029b0 0x7f37c019e890 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f37b400d980 tx=0x7f37b400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.863+0000 7f37c48ed700 1 -- 172.21.15.114:0/3291501964 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b4017070 con 0x7f37c01029b0 2024-05-27T07:53:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.863+0000 7f37c58ef700 1 -- 172.21.15.114:0/3291501964 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37c01994a0 con 0x7f37c01029b0 2024-05-27T07:53:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.863+0000 7f37c58ef700 1 -- 172.21.15.114:0/3291501964 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37c01a5740 con 0x7f37c01029b0 2024-05-27T07:53:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.863+0000 7f37c48ed700 1 -- 172.21.15.114:0/3291501964 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37b400fd40 con 0x7f37c01029b0 2024-05-27T07:53:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.863+0000 7f37c48ed700 1 -- 172.21.15.114:0/3291501964 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b400e3b0 con 0x7f37c01029b0 2024-05-27T07:53:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.869+0000 7f37c48ed700 1 -- 172.21.15.114:0/3291501964 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f37b400e550 con 0x7f37c01029b0 2024-05-27T07:53:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.869+0000 7f37d4304b80 1 Processor -- start 2024-05-27T07:53:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.869+0000 7f37d4304b80 1 -- start start 2024-05-27T07:53:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.869+0000 7f37c48ed700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37a80b6f30 0x7f37a80b9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.869+0000 7f37c48ed700 1 -- 172.21.15.114:0/3291501964 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f37b40e1ed0 con 0x7f37c01029b0 2024-05-27T07:53:52.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.869+0000 7f37bf7fe700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37a80b6f30 0x7f37a80b9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.870+0000 7f37d4304b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c6d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.870+0000 7f37d4304b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 0x564cd5c67a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.870+0000 7f37bf7fe700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37a80b6f30 0x7f37a80b9440 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f37b000a860 tx=0x7f37b0005fd0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:52.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.870+0000 7f37beffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c6d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.870+0000 7f37beffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c6d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58288/0 (socket says 172.21.15.114:58288) 2024-05-27T07:53:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.870+0000 7f37beffd700 1 -- 172.21.15.114:0/536759562 learned_addr learned my addr 172.21.15.114:0/536759562 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.871+0000 7f37d4304b80 1 --2- 172.21.15.114:0/536759562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c68390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.871+0000 7f37d4304b80 1 -- 172.21.15.114:0/536759562 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564cd5b96c90 con 0x564cd5ae9e40 2024-05-27T07:53:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.871+0000 7f37d4304b80 1 -- 172.21.15.114:0/536759562 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564cd5b86830 con 0x564cd5c67f80 2024-05-27T07:53:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.871+0000 7f37bf7fe700 1 --2- 172.21.15.114:0/536759562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c68390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.871+0000 7f37d4304b80 1 -- 172.21.15.114:0/536759562 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564cd5b8a870 con 0x564cd5c6dc50 2024-05-27T07:53:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.872+0000 7f37be7fc700 1 --2- 172.21.15.114:0/536759562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 0x564cd5c67a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.872+0000 7f37bf7fe700 1 -- 172.21.15.114:0/536759562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 msgr2=0x564cd5c67a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.872+0000 7f37bf7fe700 1 --2- 172.21.15.114:0/536759562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 0x564cd5c67a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.872+0000 7f37bf7fe700 1 -- 172.21.15.114:0/536759562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 msgr2=0x564cd5c6d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:52.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.872+0000 7f37bf7fe700 1 --2- 172.21.15.114:0/536759562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c6d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.872+0000 7f37bf7fe700 1 -- 172.21.15.114:0/536759562 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cd5ad68d0 con 0x564cd5c67f80 2024-05-27T07:53:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.879+0000 7f37bf7fe700 1 --2- 172.21.15.114:0/536759562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c68390 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f37b000a9f0 tx=0x7f37b0020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ebe0b2fa2aa33a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:52.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.889+0000 7f37a5ffb700 1 -- 172.21.15.114:0/536759562 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b0032070 con 0x564cd5c67f80 2024-05-27T07:53:52.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.889+0000 7f37a5ffb700 1 -- 172.21.15.114:0/536759562 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37b000e900 con 0x564cd5c67f80 2024-05-27T07:53:52.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.891+0000 7f37d4304b80 1 -- 172.21.15.114:0/536759562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 msgr2=0x564cd5c68390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:52.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.891+0000 7f37d4304b80 1 --2- 172.21.15.114:0/536759562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c68390 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f37b000a9f0 tx=0x7f37b0020000 comp rx=0 tx=0).stop 2024-05-27T07:53:52.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.895+0000 7f37d4304b80 1 -- 172.21.15.114:0/536759562 shutdown_connections 2024-05-27T07:53:52.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.895+0000 7f37d4304b80 1 --2- 172.21.15.114:0/536759562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c68390 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.895+0000 7f37d4304b80 1 --2- 172.21.15.114:0/536759562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c6d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.895+0000 7f37d4304b80 1 --2- 172.21.15.114:0/536759562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 0x564cd5c67a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.895+0000 7f37d4304b80 1 -- 172.21.15.114:0/536759562 >> 172.21.15.114:0/536759562 conn(0x564cd5b44af0 msgr2=0x564cd5c62640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:52.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.902+0000 7f37d4304b80 1 -- 172.21.15.114:0/536759562 shutdown_connections 2024-05-27T07:53:52.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.904+0000 7f37d4304b80 1 -- 172.21.15.114:0/536759562 wait complete. 2024-05-27T07:53:52.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.904+0000 7f37d4304b80 1 -- start start 2024-05-27T07:53:52.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.905+0000 7f37d4304b80 1 Processor -- start 2024-05-27T07:53:52.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.906+0000 7f37d4304b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c712e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.906+0000 7f37d4304b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c71820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.906+0000 7f37d4304b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 0x564cd5c71d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.906+0000 7f37d4304b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564cd5a8cfa0 con 0x564cd5ae9e40 2024-05-27T07:53:52.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.906+0000 7f37d4304b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564cd5b287c0 con 0x564cd5c67f80 2024-05-27T07:53:52.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.906+0000 7f37d4304b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564cd5c74730 con 0x564cd5c6dc50 2024-05-27T07:53:52.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.907+0000 7f37bf7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 0x564cd5c71d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.907+0000 7f37d4304b80 10 client.0 _ll_register_callbacks cb 0x564cd5b21920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:52.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.907+0000 7f37bf7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 0x564cd5c71d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:41396/0 (socket says 172.21.15.114:41396) 2024-05-27T07:53:52.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.907+0000 7f37bf7fe700 1 -- 172.21.15.114:0/2168523417 learned_addr learned my addr 172.21.15.114:0/2168523417 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:52.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.907+0000 7f37beffd700 1 --2- 172.21.15.114:0/2168523417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.908+0000 7f37beffd700 1 -- 172.21.15.114:0/2168523417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 msgr2=0x564cd5c71d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:52.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.908+0000 7f37beffd700 1 --2- 172.21.15.114:0/2168523417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 0x564cd5c71d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.908+0000 7f37beffd700 1 -- 172.21.15.114:0/2168523417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 msgr2=0x564cd5c71820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:53:52.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.908+0000 7f37beffd700 1 --2- 172.21.15.114:0/2168523417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c71820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.908+0000 7f37beffd700 1 -- 172.21.15.114:0/2168523417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cd5b25690 con 0x564cd5ae9e40 2024-05-27T07:53:52.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.908+0000 7f37beffd700 1 --2- 172.21.15.114:0/2168523417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c712e0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f37b40a4dd0 tx=0x7f37b40a9290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eaf892f91f24d8a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:52.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.908+0000 7f379effd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:53:52.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.910+0000 7f379effd700 1 -- 172.21.15.114:0/2168523417 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b4020040 con 0x564cd5ae9e40 2024-05-27T07:53:52.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.910+0000 7f37d4304b80 1 -- 172.21.15.114:0/2168523417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37b000e900 con 0x564cd5ae9e40 2024-05-27T07:53:52.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.910+0000 7f379effd700 1 -- 172.21.15.114:0/2168523417 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f37b400faa0 con 0x564cd5ae9e40 2024-05-27T07:53:52.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.913+0000 7f379effd700 1 -- 172.21.15.114:0/2168523417 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f37b40a9450 con 0x564cd5ae9e40 2024-05-27T07:53:52.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.916+0000 7f37d4304b80 10 client.18816 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:52.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.916+0000 7f37d4304b80 1 -- 172.21.15.114:0/2168523417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x564cd5c7f270 con 0x564cd5ae9e40 2024-05-27T07:53:52.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.921+0000 7f379effd700 1 -- 172.21.15.114:0/2168523417 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37b000e900 con 0x564cd5ae9e40 2024-05-27T07:53:52.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.921+0000 7f379effd700 1 -- 172.21.15.114:0/2168523417 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f37b40c2020 con 0x564cd5ae9e40 2024-05-27T07:53:52.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.921+0000 7f37d4304b80 10 client.18816 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:52.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.921+0000 7f37d4304b80 10 client.18816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:52.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.921+0000 7f37d4304b80 20 client.18816 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:52.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.921+0000 7f37d4304b80 1 --2- 172.21.15.114:0/2168523417 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564cd5c81e00 0x564cd5c842b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.921+0000 7f37d4304b80 4 client.18816 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:53:52.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.921+0000 7f37d4304b80 1 -- 172.21.15.114:0/2168523417 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cd5c339f0 con 0x564cd5c81e00 2024-05-27T07:53:52.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.925+0000 7f37be7fc700 1 --2- 172.21.15.114:0/2168523417 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564cd5c81e00 0x564cd5c842b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.931+0000 7f37be7fc700 1 --2- 172.21.15.114:0/2168523417 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564cd5c81e00 0x564cd5c842b0 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe94a98129ba1b35 server_cookie=2b691b89d57a43b8 in_seq=0 out_seq=0 2024-05-27T07:53:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.931+0000 7f379effd700 10 client.18816 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.939+0000 7f379effd700 1 -- 172.21.15.114:0/2168523417 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564cd5c339f0 con 0x564cd5c81e00 2024-05-27T07:53:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.939+0000 7f379effd700 10 client.18816 handle_command_reply: tid=0 2024-05-27T07:53:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.939+0000 7f37d4304b80 1 client.18816 shutdown 2024-05-27T07:53:52.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.947+0000 7f37d4304b80 1 -- 172.21.15.114:0/2168523417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 msgr2=0x564cd5c712e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:52.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.947+0000 7f37d4304b80 1 --2- 172.21.15.114:0/2168523417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c712e0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f37b40a4dd0 tx=0x7f37b40a9290 comp rx=0 tx=0).stop 2024-05-27T07:53:52.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.953+0000 7f37d4304b80 1 -- 172.21.15.114:0/2168523417 shutdown_connections 2024-05-27T07:53:52.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.953+0000 7f37d4304b80 1 --2- 172.21.15.114:0/2168523417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 0x564cd5c71d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.953+0000 7f37d4304b80 1 --2- 172.21.15.114:0/2168523417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c712e0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.953+0000 7f37d4304b80 1 --2- 172.21.15.114:0/2168523417 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564cd5c81e00 0x564cd5c842b0 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.953+0000 7f37d4304b80 1 --2- 172.21.15.114:0/2168523417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c71820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:52.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.953+0000 7f37d4304b80 1 -- 172.21.15.114:0/2168523417 >> 172.21.15.114:0/2168523417 conn(0x564cd5b44af0 msgr2=0x564cd5c66a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:52.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.953+0000 7f379effd700 0 client.18816 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:52.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.953+0000 7f37d4304b80 1 -- 172.21.15.114:0/2168523417 shutdown_connections 2024-05-27T07:53:52.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.955+0000 7f37d4304b80 1 -- 172.21.15.114:0/2168523417 wait complete. 2024-05-27T07:53:52.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.955+0000 7f37d4304b80 20 client.18816 trim_cache size 0 max 16384 2024-05-27T07:53:52.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:52 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:53:52.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:52 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:53:52.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.994+0000 7f37d4304b80 1 Processor -- start 2024-05-27T07:53:52.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.994+0000 7f37d4304b80 1 -- start start 2024-05-27T07:53:52.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.994+0000 7f37d4304b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c67a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.994+0000 7f37d4304b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c67f80 0x564cd5c84480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.994+0000 7f37d4304b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c6dc50 0x564cd5c7d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.994+0000 7f37d4304b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac00fae0 con 0x564cd5ae9e40 2024-05-27T07:53:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.994+0000 7f37d4304b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac0101b0 con 0x564cd5c6dc50 2024-05-27T07:53:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.994+0000 7f37d4304b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac00e960 con 0x564cd5c67f80 2024-05-27T07:53:52.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.997+0000 7f37be7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c67f80 0x564cd5c84480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.997+0000 7f37beffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c67a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:52.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.997+0000 7f37beffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c67a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58294/0 (socket says 172.21.15.114:58294) 2024-05-27T07:53:52.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.997+0000 7f37beffd700 1 -- 172.21.15.114:0/4081340218 learned_addr learned my addr 172.21.15.114:0/4081340218 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:52.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.998+0000 7f37bf7fe700 1 --2- 172.21.15.114:0/4081340218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c6dc50 0x564cd5c7d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:53.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.999+0000 7f37beffd700 1 -- 172.21.15.114:0/4081340218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c67f80 msgr2=0x564cd5c84480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:53.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.999+0000 7f37beffd700 1 --2- 172.21.15.114:0/4081340218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c67f80 0x564cd5c84480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.999+0000 7f37beffd700 1 -- 172.21.15.114:0/4081340218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c6dc50 msgr2=0x564cd5c7d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:53.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.999+0000 7f37beffd700 1 --2- 172.21.15.114:0/4081340218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c6dc50 0x564cd5c7d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:52.999+0000 7f37beffd700 1 -- 172.21.15.114:0/4081340218 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cd5c5a970 con 0x564cd5ae9e40 2024-05-27T07:53:53.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.004+0000 7f37beffd700 1 --2- 172.21.15.114:0/4081340218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c67a40 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f37b400be80 tx=0x7f37b40a8570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265af2662054ee20 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:53.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.008+0000 7f37a57fa700 1 -- 172.21.15.114:0/4081340218 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b40c7020 con 0x564cd5ae9e40 2024-05-27T07:53:53.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.008+0000 7f37a57fa700 1 -- 172.21.15.114:0/4081340218 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37b40ad100 con 0x564cd5ae9e40 2024-05-27T07:53:53.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.008+0000 7f37a57fa700 1 -- 172.21.15.114:0/4081340218 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b4020070 con 0x564cd5ae9e40 2024-05-27T07:53:53.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.010+0000 7f37d4304b80 1 -- 172.21.15.114:0/4081340218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 msgr2=0x564cd5c67a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:53.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.010+0000 7f37d4304b80 1 --2- 172.21.15.114:0/4081340218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c67a40 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f37b400be80 tx=0x7f37b40a8570 comp rx=0 tx=0).stop 2024-05-27T07:53:53.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.014+0000 7f37d4304b80 1 -- 172.21.15.114:0/4081340218 shutdown_connections 2024-05-27T07:53:53.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.014+0000 7f37d4304b80 1 --2- 172.21.15.114:0/4081340218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c6dc50 0x564cd5c7d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.014+0000 7f37d4304b80 1 --2- 172.21.15.114:0/4081340218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c67a40 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.014+0000 7f37d4304b80 1 --2- 172.21.15.114:0/4081340218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c67f80 0x564cd5c84480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.014+0000 7f37d4304b80 1 -- 172.21.15.114:0/4081340218 >> 172.21.15.114:0/4081340218 conn(0x564cd5b44af0 msgr2=0x564cd5c6b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:53.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.021+0000 7f37d4304b80 1 -- 172.21.15.114:0/4081340218 shutdown_connections 2024-05-27T07:53:53.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.021+0000 7f37d4304b80 1 -- 172.21.15.114:0/4081340218 wait complete. 2024-05-27T07:53:53.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.021+0000 7f37d4304b80 1 -- start start 2024-05-27T07:53:53.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.022+0000 7f37d4304b80 1 Processor -- start 2024-05-27T07:53:53.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.023+0000 7f37d4304b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c897b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:53.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.023+0000 7f37d4304b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c89cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:53.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.023+0000 7f37d4304b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 0x564cd5c8a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:53.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.023+0000 7f37d4304b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac00e290 con 0x564cd5ae9e40 2024-05-27T07:53:53.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.024+0000 7f37d4304b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac00cea0 con 0x564cd5c67f80 2024-05-27T07:53:53.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.024+0000 7f37d4304b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac00d4f0 con 0x564cd5c6dc50 2024-05-27T07:53:53.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.024+0000 7f37d4304b80 10 client.0 _ll_register_callbacks cb 0x564cd5b21920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:53:53.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.026+0000 7f37be7fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c89cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:53.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.026+0000 7f37bf7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 0x564cd5c8a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:53.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.026+0000 7f37be7fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c89cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:46286/0 (socket says 172.21.15.114:46286) 2024-05-27T07:53:53.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.026+0000 7f37be7fc700 1 -- 172.21.15.114:0/2978593895 learned_addr learned my addr 172.21.15.114:0/2978593895 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:53.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.029+0000 7f37beffd700 1 --2- 172.21.15.114:0/2978593895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c897b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:53.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.029+0000 7f37be7fc700 1 -- 172.21.15.114:0/2978593895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 msgr2=0x564cd5c8a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:53.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.029+0000 7f37be7fc700 1 --2- 172.21.15.114:0/2978593895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 0x564cd5c8a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.029+0000 7f37be7fc700 1 -- 172.21.15.114:0/2978593895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 msgr2=0x564cd5c897b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:53:53.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.029+0000 7f37be7fc700 1 --2- 172.21.15.114:0/2978593895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c897b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.029+0000 7f37be7fc700 1 -- 172.21.15.114:0/2978593895 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cd5cebab0 con 0x564cd5c67f80 2024-05-27T07:53:53.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.035+0000 7f37be7fc700 1 --2- 172.21.15.114:0/2978593895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c89cf0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f37c0102100 tx=0x7f37b8003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1698ea34c83a8ef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:53.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.035+0000 7f379effd700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:53:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.039+0000 7f379effd700 1 -- 172.21.15.114:0/2978593895 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b8004610 con 0x564cd5c67f80 2024-05-27T07:53:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.039+0000 7f379effd700 1 -- 172.21.15.114:0/2978593895 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f37b80047b0 con 0x564cd5c67f80 2024-05-27T07:53:53.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.039+0000 7f37d4304b80 1 -- 172.21.15.114:0/2978593895 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37b40ad100 con 0x564cd5c67f80 2024-05-27T07:53:53.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.044+0000 7f379effd700 1 -- 172.21.15.114:0/2978593895 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f37b8004b40 con 0x564cd5c67f80 2024-05-27T07:53:53.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.046+0000 7f37d4304b80 10 client.28199 fetch_fsmap learned FSMap version 76 2024-05-27T07:53:53.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.046+0000 7f37d4304b80 1 -- 172.21.15.114:0/2978593895 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x564cd5cecdb0 con 0x564cd5c67f80 2024-05-27T07:53:53.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.049+0000 7f379effd700 1 -- 172.21.15.114:0/2978593895 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37b40ad100 con 0x564cd5c67f80 2024-05-27T07:53:53.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.049+0000 7f379effd700 1 -- 172.21.15.114:0/2978593895 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f37b8007500 con 0x564cd5c67f80 2024-05-27T07:53:53.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.049+0000 7f37d4304b80 10 client.28199 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:53:53.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.049+0000 7f37d4304b80 10 client.28199 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:53:53.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.049+0000 7f37d4304b80 20 client.28199 populate_metadata read hostname 'smithi114' 2024-05-27T07:53:53.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.049+0000 7f37d4304b80 1 --2- 172.21.15.114:0/2978593895 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564cd5ced4f0 0x564cd5cef910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:53.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.049+0000 7f37d4304b80 4 client.28199 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:53:53.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.049+0000 7f37d4304b80 1 -- 172.21.15.114:0/2978593895 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564cd5b98be0 con 0x564cd5ced4f0 2024-05-27T07:53:53.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.049+0000 7f37beffd700 1 --2- 172.21.15.114:0/2978593895 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564cd5ced4f0 0x564cd5cef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:53.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.057+0000 7f37beffd700 1 --2- 172.21.15.114:0/2978593895 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564cd5ced4f0 0x564cd5cef910 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6290eceb27226a47 server_cookie=dbc3dc58c062aea4 in_seq=0 out_seq=0 2024-05-27T07:53:53.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.059+0000 7f379effd700 10 client.28199 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:53.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.059+0000 7f379effd700 1 -- 172.21.15.114:0/2978593895 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564cd5b98be0 con 0x564cd5ced4f0 2024-05-27T07:53:53.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.059+0000 7f379effd700 10 client.28199 handle_command_reply: tid=0 2024-05-27T07:53:53.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.059+0000 7f37d4304b80 1 client.28199 shutdown 2024-05-27T07:53:53.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.063+0000 7f37d4304b80 1 -- 172.21.15.114:0/2978593895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 msgr2=0x564cd5c89cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:53.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.063+0000 7f37d4304b80 1 --2- 172.21.15.114:0/2978593895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c89cf0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f37c0102100 tx=0x7f37b8003c80 comp rx=0 tx=0).stop 2024-05-27T07:53:53.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.066+0000 7f37d4304b80 1 -- 172.21.15.114:0/2978593895 shutdown_connections 2024-05-27T07:53:53.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.066+0000 7f37d4304b80 1 --2- 172.21.15.114:0/2978593895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564cd5c6dc50 0x564cd5c8a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.066+0000 7f37d4304b80 1 --2- 172.21.15.114:0/2978593895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564cd5ae9e40 0x564cd5c897b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.066+0000 7f37d4304b80 1 --2- 172.21.15.114:0/2978593895 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564cd5ced4f0 0x564cd5cef910 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.066+0000 7f37d4304b80 1 --2- 172.21.15.114:0/2978593895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564cd5c67f80 0x564cd5c89cf0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.066+0000 7f379effd700 0 client.28199 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:53.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.066+0000 7f37d4304b80 1 -- 172.21.15.114:0/2978593895 >> 172.21.15.114:0/2978593895 conn(0x564cd5b44af0 msgr2=0x564cd5c6e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:53.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.066+0000 7f37d4304b80 1 -- 172.21.15.114:0/2978593895 shutdown_connections 2024-05-27T07:53:53.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.066+0000 7f37d4304b80 1 -- 172.21.15.114:0/2978593895 wait complete. 2024-05-27T07:53:53.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.066+0000 7f37d4304b80 20 client.28199 trim_cache size 0 max 16384 2024-05-27T07:53:53.068 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:53:53.068 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:53:53.068 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:53:53.068 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:53:53.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.070+0000 7f37c58ef700 1 -- 172.21.15.114:0/3291501964 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37a80b6f30 msgr2=0x7f37a80b9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:53.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.070+0000 7f37c58ef700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37a80b6f30 0x7f37a80b9440 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f37b000a860 tx=0x7f37b0005fd0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.070+0000 7f37c58ef700 1 -- 172.21.15.114:0/3291501964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37c01029b0 msgr2=0x7f37c019e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:53.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.070+0000 7f37c58ef700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37c01029b0 0x7f37c019e890 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f37b400d980 tx=0x7f37b400dc90 comp rx=0 tx=0).stop 2024-05-27T07:53:53.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.070+0000 7f37c58ef700 1 -- 172.21.15.114:0/3291501964 shutdown_connections 2024-05-27T07:53:53.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.070+0000 7f37c58ef700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37c01089e0 0x7f37c0198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.070+0000 7f37c58ef700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f37a80b6f30 0x7f37a80b9440 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.070+0000 7f37c58ef700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37c01029b0 0x7f37c019e890 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.070+0000 7f37c58ef700 1 --2- 172.21.15.114:0/3291501964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c0103300 0x7f37c019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:53.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.070+0000 7f37c58ef700 1 -- 172.21.15.114:0/3291501964 >> 172.21.15.114:0/3291501964 conn(0x7f37c00fe6e0 msgr2=0x7f37c0106c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:53.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.071+0000 7f37c58ef700 1 -- 172.21.15.114:0/3291501964 shutdown_connections 2024-05-27T07:53:53.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:53.071+0000 7f37c58ef700 1 -- 172.21.15.114:0/3291501964 wait complete. 2024-05-27T07:53:53.589 INFO:tasks.workunit.client.0.smithi114.stdout: 10 760 41.37 MB/sec warmup 5 sec latency 1897.671 ms 2024-05-27T07:53:53.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:53 smithi114 ceph-mon[32038]: pgmap v475: 129 pgs: 129 active+clean; 118 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.8 MiB/s wr, 330 op/s 2024-05-27T07:53:53.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:53 smithi163 ceph-mon[36451]: pgmap v475: 129 pgs: 129 active+clean; 118 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.8 MiB/s wr, 330 op/s 2024-05-27T07:53:53.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:53 smithi186 ceph-mon[36647]: pgmap v475: 129 pgs: 129 active+clean; 118 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.8 MiB/s wr, 330 op/s 2024-05-27T07:53:54.190 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:53:54.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:53:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.225+0000 7f32e5ffb640 1 -- 172.21.15.114:0/1480612075 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5632b9af4360 con 0x5632b9c1c160 2024-05-27T07:53:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.225+0000 7f32e5ffb640 10 client.28187 handle_command_reply: tid=0 2024-05-27T07:53:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.225+0000 7f330bd93740 1 client.28187 shutdown 2024-05-27T07:53:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.226+0000 7f330bd93740 1 -- 172.21.15.114:0/1480612075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 msgr2=0x5632b9ba4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.226+0000 7f330bd93740 1 --2- 172.21.15.114:0/1480612075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9ba4820 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x5632b9b89050 tx=0x7f32f0005da0 comp rx=0 tx=0).stop 2024-05-27T07:53:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.226+0000 7f330bd93740 1 -- 172.21.15.114:0/1480612075 shutdown_connections 2024-05-27T07:53:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.226+0000 7f330bd93740 1 --2- 172.21.15.114:0/1480612075 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5632b9c1c160 0x5632b9c1e580 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.226+0000 7f330bd93740 1 --2- 172.21.15.114:0/1480612075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5632b97c44d0 0x5632b9ba52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.226+0000 7f330bd93740 1 --2- 172.21.15.114:0/1480612075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5632b97bc2f0 0x5632b9ba4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.226+0000 7f330bd93740 1 --2- 172.21.15.114:0/1480612075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5632b97a3d80 0x5632b9ba4820 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.226+0000 7f330bd93740 1 -- 172.21.15.114:0/1480612075 >> 172.21.15.114:0/1480612075 conn(0x5632b97d9300 msgr2=0x5632b9b86160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.226+0000 7f32e5ffb640 0 client.28187 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:53:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.226+0000 7f330bd93740 1 -- 172.21.15.114:0/1480612075 shutdown_connections 2024-05-27T07:53:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.226+0000 7f330bd93740 1 -- 172.21.15.114:0/1480612075 wait complete. 2024-05-27T07:53:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.226+0000 7f330bd93740 20 client.28187 trim_cache size 0 max 16384 2024-05-27T07:53:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:53:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:53:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:53:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.227+0000 7f3304ee2640 1 -- 172.21.15.114:0/2179503598 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f32e80b2a20 msgr2=0x7f32e80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.227+0000 7f3304ee2640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f32e80b2a20 0x7f32e80b4f30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f32f8009dc0 tx=0x7f32f8009400 comp rx=0 tx=0).stop 2024-05-27T07:53:54.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.227+0000 7f3304ee2640 1 -- 172.21.15.114:0/2179503598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3300108020 msgr2=0x7f33001106e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:54.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.227+0000 7f3304ee2640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3300108020 0x7f33001106e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f32f802aee0 tx=0x7f32f802f520 comp rx=0 tx=0).stop 2024-05-27T07:53:54.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.227+0000 7f3304ee2640 1 -- 172.21.15.114:0/2179503598 shutdown_connections 2024-05-27T07:53:54.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.227+0000 7f3304ee2640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f32e80b2a20 0x7f32e80b4f30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:54.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.227+0000 7f3304ee2640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3300108020 0x7f33001106e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:54.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.227+0000 7f3304ee2640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33001075f0 0x7f3300110020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:54.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.227+0000 7f3304ee2640 1 --2- 172.21.15.114:0/2179503598 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3300102ba0 0x7f3300116e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:54.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.227+0000 7f3304ee2640 1 -- 172.21.15.114:0/2179503598 >> 172.21.15.114:0/2179503598 conn(0x7f33000781a0 msgr2=0x7f3300105180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:54.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.227+0000 7f3304ee2640 1 -- 172.21.15.114:0/2179503598 shutdown_connections 2024-05-27T07:53:54.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:53:54.228+0000 7f3304ee2640 1 -- 172.21.15.114:0/2179503598 wait complete. 2024-05-27T07:53:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:54.588 INFO:tasks.workunit.client.0.smithi114.stdout: 10 978 36.25 MB/sec warmup 6 sec latency 814.622 ms 2024-05-27T07:53:55.589 INFO:tasks.workunit.client.0.smithi114.stdout: 10 1466 36.60 MB/sec warmup 7 sec latency 176.549 ms 2024-05-27T07:53:55.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:55 smithi114 ceph-mon[32038]: pgmap v476: 129 pgs: 129 active+clean; 118 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 168 op/s 2024-05-27T07:53:55.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:55 smithi163 ceph-mon[36451]: pgmap v476: 129 pgs: 129 active+clean; 118 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 168 op/s 2024-05-27T07:53:55.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:55 smithi186 ceph-mon[36647]: pgmap v476: 129 pgs: 129 active+clean; 118 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 168 op/s 2024-05-27T07:53:56.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.521+0000 7ffb497d9700 1 -- 172.21.15.114:0/93315437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb44107de0 msgr2=0x7ffb441081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:56.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.521+0000 7ffb497d9700 1 --2- 172.21.15.114:0/93315437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb44107de0 0x7ffb441081f0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7ffb380099c0 tx=0x7ffb38009cd0 comp rx=0 tx=0).stop 2024-05-27T07:53:56.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.521+0000 7ffb497d9700 1 -- 172.21.15.114:0/93315437 shutdown_connections 2024-05-27T07:53:56.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.521+0000 7ffb497d9700 1 --2- 172.21.15.114:0/93315437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffb44108730 0x7ffb44110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:56.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.521+0000 7ffb497d9700 1 --2- 172.21.15.114:0/93315437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb441034b0 0x7ffb441038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:56.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.521+0000 7ffb497d9700 1 --2- 172.21.15.114:0/93315437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb44107de0 0x7ffb441081f0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:56.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.521+0000 7ffb497d9700 1 -- 172.21.15.114:0/93315437 >> 172.21.15.114:0/93315437 conn(0x7ffb440fedb0 msgr2=0x7ffb441011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:56.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.521+0000 7ffb497d9700 1 -- 172.21.15.114:0/93315437 shutdown_connections 2024-05-27T07:53:56.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.521+0000 7ffb497d9700 1 -- 172.21.15.114:0/93315437 wait complete. 2024-05-27T07:53:56.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.522+0000 7ffb497d9700 1 Processor -- start 2024-05-27T07:53:56.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.522+0000 7ffb497d9700 1 -- start start 2024-05-27T07:53:56.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.522+0000 7ffb497d9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffb441034b0 0x7ffb4419bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:56.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.522+0000 7ffb497d9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb44107de0 0x7ffb4419c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:56.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.522+0000 7ffb497d9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb44108730 0x7ffb4419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:56.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.522+0000 7ffb497d9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffb44113340 con 0x7ffb44108730 2024-05-27T07:53:56.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.522+0000 7ffb497d9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffb441131c0 con 0x7ffb441034b0 2024-05-27T07:53:56.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.522+0000 7ffb497d9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffb441134c0 con 0x7ffb44107de0 2024-05-27T07:53:56.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.522+0000 7ffb437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb44108730 0x7ffb4419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:56.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.522+0000 7ffb427fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb44107de0 0x7ffb4419c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:56.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.522+0000 7ffb437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb44108730 0x7ffb4419c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53748/0 (socket says 172.21.15.114:53748) 2024-05-27T07:53:56.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.522+0000 7ffb437fe700 1 -- 172.21.15.114:0/561955283 learned_addr learned my addr 172.21.15.114:0/561955283 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:53:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.522+0000 7ffb42ffd700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffb441034b0 0x7ffb4419bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.523+0000 7ffb427fc700 1 -- 172.21.15.114:0/561955283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffb441034b0 msgr2=0x7ffb4419bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.523+0000 7ffb427fc700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffb441034b0 0x7ffb4419bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.523+0000 7ffb427fc700 1 -- 172.21.15.114:0/561955283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb44108730 msgr2=0x7ffb4419c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.523+0000 7ffb427fc700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb44108730 0x7ffb4419c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.523+0000 7ffb427fc700 1 -- 172.21.15.114:0/561955283 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb441a6fb0 con 0x7ffb44107de0 2024-05-27T07:53:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.523+0000 7ffb437fe700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb44108730 0x7ffb4419c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:53:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.523+0000 7ffb427fc700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb44107de0 0x7ffb4419c180 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffb380098e0 tx=0x7ffb380045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.523+0000 7ffb2bfff700 1 -- 172.21.15.114:0/561955283 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb38026070 con 0x7ffb44107de0 2024-05-27T07:53:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.523+0000 7ffb497d9700 1 -- 172.21.15.114:0/561955283 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb441a7240 con 0x7ffb44107de0 2024-05-27T07:53:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.523+0000 7ffb2bfff700 1 -- 172.21.15.114:0/561955283 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb3800f460 con 0x7ffb44107de0 2024-05-27T07:53:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.523+0000 7ffb2bfff700 1 -- 172.21.15.114:0/561955283 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb38009e60 con 0x7ffb44107de0 2024-05-27T07:53:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.523+0000 7ffb497d9700 1 -- 172.21.15.114:0/561955283 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb441a76f0 con 0x7ffb44107de0 2024-05-27T07:53:56.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.524+0000 7ffb497d9700 1 -- 172.21.15.114:0/561955283 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb440512b0 con 0x7ffb44107de0 2024-05-27T07:53:56.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.526+0000 7ffb2bfff700 1 -- 172.21.15.114:0/561955283 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ffb38029070 con 0x7ffb44107de0 2024-05-27T07:53:56.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.527+0000 7ffb2bfff700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffb2c0b29f0 0x7ffb2c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:53:56.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.527+0000 7ffb42ffd700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffb2c0b29f0 0x7ffb2c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:53:56.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.527+0000 7ffb2bfff700 1 -- 172.21.15.114:0/561955283 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7ffb380e3510 con 0x7ffb44107de0 2024-05-27T07:53:56.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.527+0000 7ffb42ffd700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffb2c0b29f0 0x7ffb2c0b4f00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009c60 tx=0x7ffb34009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:53:56.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.531+0000 7ffb2bfff700 1 -- 172.21.15.114:0/561955283 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ffb38013070 con 0x7ffb44107de0 2024-05-27T07:53:56.589 INFO:tasks.workunit.client.0.smithi114.stdout: 10 2144 33.65 MB/sec warmup 8 sec latency 78.089 ms 2024-05-27T07:53:56.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.818+0000 7ffb497d9700 1 -- 172.21.15.114:0/561955283 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb4402e5e0 con 0x7ffb44107de0 2024-05-27T07:53:56.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.819+0000 7ffb2bfff700 1 -- 172.21.15.114:0/561955283 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+7693 (secure 0 0 0) 0x7ffb380aab80 con 0x7ffb44107de0 2024-05-27T07:53:56.820 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:53:56.820 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:53:34.360222+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27557":{"gid":27557,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/3261223371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":3261223371},{"type":"v1","addr":"172.21.15.163:6835","nonce":3261223371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:53:56.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.821+0000 7ffb497d9700 1 -- 172.21.15.114:0/561955283 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffb2c0b29f0 msgr2=0x7ffb2c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:56.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.821+0000 7ffb497d9700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffb2c0b29f0 0x7ffb2c0b4f00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009c60 tx=0x7ffb34009260 comp rx=0 tx=0).stop 2024-05-27T07:53:56.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.821+0000 7ffb497d9700 1 -- 172.21.15.114:0/561955283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb44107de0 msgr2=0x7ffb4419c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:53:56.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.821+0000 7ffb497d9700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb44107de0 0x7ffb4419c180 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffb380098e0 tx=0x7ffb380045e0 comp rx=0 tx=0).stop 2024-05-27T07:53:56.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.822+0000 7ffb497d9700 1 -- 172.21.15.114:0/561955283 shutdown_connections 2024-05-27T07:53:56.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.822+0000 7ffb497d9700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb44108730 0x7ffb4419c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:56.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.822+0000 7ffb497d9700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffb2c0b29f0 0x7ffb2c0b4f00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:56.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.822+0000 7ffb497d9700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffb441034b0 0x7ffb4419bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:56.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.822+0000 7ffb497d9700 1 --2- 172.21.15.114:0/561955283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb44107de0 0x7ffb4419c180 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:53:56.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.822+0000 7ffb497d9700 1 -- 172.21.15.114:0/561955283 >> 172.21.15.114:0/561955283 conn(0x7ffb440fedb0 msgr2=0x7ffb441011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:53:56.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.822+0000 7ffb497d9700 1 -- 172.21.15.114:0/561955283 shutdown_connections 2024-05-27T07:53:56.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:56.822+0000 7ffb497d9700 1 -- 172.21.15.114:0/561955283 wait complete. 2024-05-27T07:53:56.824 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 76 2024-05-27T07:53:57.589 INFO:tasks.workunit.client.0.smithi114.stdout: 10 2648 34.62 MB/sec warmup 9 sec latency 161.253 ms 2024-05-27T07:53:57.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:53:57.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:57 smithi114 ceph-mon[32038]: pgmap v477: 129 pgs: 129 active+clean; 191 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 14 MiB/s wr, 482 op/s 2024-05-27T07:53:57.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:57 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/561955283' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:53:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:53:57.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:57 smithi163 ceph-mon[36451]: pgmap v477: 129 pgs: 129 active+clean; 191 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 14 MiB/s wr, 482 op/s 2024-05-27T07:53:57.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:57 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/561955283' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:53:57.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:57 smithi186 ceph-mon[36647]: pgmap v477: 129 pgs: 129 active+clean; 191 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 14 MiB/s wr, 482 op/s 2024-05-27T07:53:57.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:57 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/561955283' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:53:58.589 INFO:tasks.workunit.client.0.smithi114.stdout: 10 3231 35.38 MB/sec warmup 10 sec latency 158.013 ms 2024-05-27T07:53:59.589 INFO:tasks.workunit.client.0.smithi114.stdout: 10 3547 34.55 MB/sec warmup 11 sec latency 195.281 ms 2024-05-27T07:53:59.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:53:59 smithi114 ceph-mon[32038]: pgmap v478: 129 pgs: 129 active+clean; 200 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 17 MiB/s wr, 578 op/s 2024-05-27T07:53:59.778 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:53:59 smithi163 ceph-mon[36451]: pgmap v478: 129 pgs: 129 active+clean; 200 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 17 MiB/s wr, 578 op/s 2024-05-27T07:53:59.779 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:53:59 smithi186 ceph-mon[36647]: pgmap v478: 129 pgs: 129 active+clean; 200 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 17 MiB/s wr, 578 op/s 2024-05-27T07:54:00.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.876+0000 7f72faf20700 1 -- 172.21.15.114:0/705541192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72f4103c50 msgr2=0x7f72f4104060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:00.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.876+0000 7f72faf20700 1 --2- 172.21.15.114:0/705541192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72f4103c50 0x7f72f4104060 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f72e80099c0 tx=0x7f72e8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.876+0000 7f72faf20700 1 -- 172.21.15.114:0/705541192 shutdown_connections 2024-05-27T07:54:00.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.876+0000 7f72faf20700 1 --2- 172.21.15.114:0/705541192 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72f4108740 0x7f72f41100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.876+0000 7f72faf20700 1 --2- 172.21.15.114:0/705541192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72f4103c50 0x7f72f4104060 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.876+0000 7f72faf20700 1 --2- 172.21.15.114:0/705541192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4107df0 0x7f72f4108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.876+0000 7f72faf20700 1 -- 172.21.15.114:0/705541192 >> 172.21.15.114:0/705541192 conn(0x7f72f40fe6e0 msgr2=0x7f72f4100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:00.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.876+0000 7f72faf20700 1 -- 172.21.15.114:0/705541192 shutdown_connections 2024-05-27T07:54:00.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.877+0000 7f72faf20700 1 -- 172.21.15.114:0/705541192 wait complete. 2024-05-27T07:54:00.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.877+0000 7f72faf20700 1 Processor -- start 2024-05-27T07:54:00.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.877+0000 7f72faf20700 1 -- start start 2024-05-27T07:54:00.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.877+0000 7f72faf20700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72f4103c50 0x7f72f40febc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.877+0000 7f72faf20700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72f4107df0 0x7f72f40ff100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72faf20700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4108740 0x7f72f40ff640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72faf20700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f72f4113370 con 0x7f72f4103c50 2024-05-27T07:54:00.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72faf20700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41131f0 con 0x7f72f4107df0 2024-05-27T07:54:00.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72faf20700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41134f0 con 0x7f72f4108740 2024-05-27T07:54:00.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f8cbc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72f4103c50 0x7f72f40febc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f8cbc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72f4103c50 0x7f72f40febc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53764/0 (socket says 172.21.15.114:53764) 2024-05-27T07:54:00.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f8cbc700 1 -- 172.21.15.114:0/267476164 learned_addr learned my addr 172.21.15.114:0/267476164 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:00.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f3fff700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72f4107df0 0x7f72f40ff100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f94bd700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4108740 0x7f72f40ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f8cbc700 1 -- 172.21.15.114:0/267476164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4108740 msgr2=0x7f72f40ff640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:00.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f8cbc700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4108740 0x7f72f40ff640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f8cbc700 1 -- 172.21.15.114:0/267476164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72f4107df0 msgr2=0x7f72f40ff100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:00.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f8cbc700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72f4107df0 0x7f72f40ff100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f8cbc700 1 -- 172.21.15.114:0/267476164 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f41a91b0 con 0x7f72f4103c50 2024-05-27T07:54:00.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f8cbc700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72f4103c50 0x7f72f40febc0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f72e8006010 tx=0x7f72e8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:00.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f1ffb700 1 -- 172.21.15.114:0/267476164 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e8026070 con 0x7f72f4103c50 2024-05-27T07:54:00.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72faf20700 1 -- 172.21.15.114:0/267476164 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72f41a9440 con 0x7f72f4103c50 2024-05-27T07:54:00.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f1ffb700 1 -- 172.21.15.114:0/267476164 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72e8004630 con 0x7f72f4103c50 2024-05-27T07:54:00.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.878+0000 7f72f1ffb700 1 -- 172.21.15.114:0/267476164 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e800fb20 con 0x7f72f4103c50 2024-05-27T07:54:00.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.879+0000 7f72faf20700 1 -- 172.21.15.114:0/267476164 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72f41a98f0 con 0x7f72f4103c50 2024-05-27T07:54:00.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.881+0000 7f72f1ffb700 1 -- 172.21.15.114:0/267476164 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f72e8029070 con 0x7f72f4103c50 2024-05-27T07:54:00.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.882+0000 7f72f1ffb700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f72e40b29f0 0x7f72e40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.882+0000 7f72f3fff700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f72e40b29f0 0x7f72e40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.882+0000 7f7309935b80 1 Processor -- start 2024-05-27T07:54:00.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.882+0000 7f72f1ffb700 1 -- 172.21.15.114:0/267476164 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f72e80e3120 con 0x7f72f4103c50 2024-05-27T07:54:00.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f7309935b80 1 -- start start 2024-05-27T07:54:00.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f7309935b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e437732e40 0x55e4378b6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f7309935b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b6c50 0x55e4378b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f72f3fff700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f72e40b29f0 0x7f72e40b4f00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f72f4109ac0 tx=0x7f72e0009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:00.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f7309935b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b0f80 0x55e4378b1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f7309935b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e4377dfc90 con 0x55e4378b6c50 2024-05-27T07:54:00.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f7309935b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e4377cf830 con 0x55e437732e40 2024-05-27T07:54:00.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f7309935b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e4377d3870 con 0x55e4378b0f80 2024-05-27T07:54:00.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f72f3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b6c50 0x55e4378b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f72f3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b6c50 0x55e4378b0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53770/0 (socket says 172.21.15.114:53770) 2024-05-27T07:54:00.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f72f3fff700 1 -- 172.21.15.114:0/3323430418 learned_addr learned my addr 172.21.15.114:0/3323430418 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:00.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f72f94bd700 1 --2- 172.21.15.114:0/3323430418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b0f80 0x55e4378b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f72f8cbc700 1 --2- 172.21.15.114:0/3323430418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e437732e40 0x55e4378b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f72f3fff700 1 -- 172.21.15.114:0/3323430418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b0f80 msgr2=0x55e4378b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:00.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f72f3fff700 1 --2- 172.21.15.114:0/3323430418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b0f80 0x55e4378b1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f72f3fff700 1 -- 172.21.15.114:0/3323430418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e437732e40 msgr2=0x55e4378b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:00.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f72f3fff700 1 --2- 172.21.15.114:0/3323430418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e437732e40 0x55e4378b6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f72f3fff700 1 -- 172.21.15.114:0/3323430418 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e43771f8d0 con 0x55e4378b6c50 2024-05-27T07:54:00.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.883+0000 7f72f3fff700 1 --2- 172.21.15.114:0/3323430418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b6c50 0x55e4378b0a40 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f72e0011d20 tx=0x7f72e0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4e7cd896953e9f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:00.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.884+0000 7f72deffd700 1 -- 172.21.15.114:0/3323430418 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e000eb20 con 0x55e4378b6c50 2024-05-27T07:54:00.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.884+0000 7f72deffd700 1 -- 172.21.15.114:0/3323430418 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72e000ecc0 con 0x55e4378b6c50 2024-05-27T07:54:00.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.884+0000 7f72deffd700 1 -- 172.21.15.114:0/3323430418 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e001d670 con 0x55e4378b6c50 2024-05-27T07:54:00.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.884+0000 7f7309935b80 1 -- 172.21.15.114:0/3323430418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b6c50 msgr2=0x55e4378b0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:00.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.884+0000 7f7309935b80 1 --2- 172.21.15.114:0/3323430418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b6c50 0x55e4378b0a40 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f72e0011d20 tx=0x7f72e0014000 comp rx=0 tx=0).stop 2024-05-27T07:54:00.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 -- 172.21.15.114:0/3323430418 shutdown_connections 2024-05-27T07:54:00.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 --2- 172.21.15.114:0/3323430418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b0f80 0x55e4378b1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 --2- 172.21.15.114:0/3323430418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e437732e40 0x55e4378b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 --2- 172.21.15.114:0/3323430418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b6c50 0x55e4378b0a40 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 -- 172.21.15.114:0/3323430418 >> 172.21.15.114:0/3323430418 conn(0x55e43778daf0 msgr2=0x55e4378ab640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:00.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 -- 172.21.15.114:0/3323430418 shutdown_connections 2024-05-27T07:54:00.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 -- 172.21.15.114:0/3323430418 wait complete. 2024-05-27T07:54:00.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 -- start start 2024-05-27T07:54:00.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 Processor -- start 2024-05-27T07:54:00.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 0x55e4378ba390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b0f80 0x55e4378ba8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b6c50 0x55e4378bae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e4376d5fa0 con 0x55e437732e40 2024-05-27T07:54:00.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.885+0000 7f7309935b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e4377717c0 con 0x55e4378b6c50 2024-05-27T07:54:00.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f7309935b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e4378bd740 con 0x55e4378b0f80 2024-05-27T07:54:00.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f7309935b80 10 client.0 _ll_register_callbacks cb 0x55e43776a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:00.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72f8cbc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 0x55e4378ba390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72f8cbc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 0x55e4378ba390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53778/0 (socket says 172.21.15.114:53778) 2024-05-27T07:54:00.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72f8cbc700 1 -- 172.21.15.114:0/835855094 learned_addr learned my addr 172.21.15.114:0/835855094 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:00.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72f3fff700 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b0f80 0x55e4378ba8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72f94bd700 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b6c50 0x55e4378bae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72f8cbc700 1 -- 172.21.15.114:0/835855094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b0f80 msgr2=0x55e4378ba8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:00.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72f8cbc700 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b0f80 0x55e4378ba8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72f8cbc700 1 -- 172.21.15.114:0/835855094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b6c50 msgr2=0x55e4378bae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:00.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72f8cbc700 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b6c50 0x55e4378bae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72f8cbc700 1 -- 172.21.15.114:0/835855094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e43776e690 con 0x55e437732e40 2024-05-27T07:54:00.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72f3fff700 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b0f80 0x55e4378ba8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:00.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72f94bd700 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b6c50 0x55e4378bae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:00.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72f8cbc700 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 0x55e4378ba390 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f72e80059d0 tx=0x7f72e8003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31deab81dc21d008 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:00.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72d3fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:54:00.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72d3fff700 1 -- 172.21.15.114:0/835855094 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e80aa790 con 0x55e437732e40 2024-05-27T07:54:00.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f7309935b80 1 -- 172.21.15.114:0/835855094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72e000ecc0 con 0x55e437732e40 2024-05-27T07:54:00.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.886+0000 7f72d3fff700 1 -- 172.21.15.114:0/835855094 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f72e80ae0c0 con 0x55e437732e40 2024-05-27T07:54:00.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.887+0000 7f72d3fff700 1 -- 172.21.15.114:0/835855094 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f72e80aa090 con 0x55e437732e40 2024-05-27T07:54:00.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.887+0000 7f7309935b80 10 client.18840 fetch_fsmap learned FSMap version 76 2024-05-27T07:54:00.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.887+0000 7f7309935b80 1 -- 172.21.15.114:0/835855094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55e4378c81f0 con 0x55e437732e40 2024-05-27T07:54:00.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.887+0000 7f72d3fff700 1 -- 172.21.15.114:0/835855094 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72e000ecc0 con 0x55e437732e40 2024-05-27T07:54:00.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.887+0000 7f72d3fff700 1 -- 172.21.15.114:0/835855094 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f72e80e0bc0 con 0x55e437732e40 2024-05-27T07:54:00.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.887+0000 7f7309935b80 10 client.18840 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:54:00.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.887+0000 7f7309935b80 10 client.18840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:00.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.887+0000 7f7309935b80 20 client.18840 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:00.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.887+0000 7f7309935b80 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e4378ca940 0x55e4378ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.887+0000 7f7309935b80 4 client.18840 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:54:00.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.887+0000 7f7309935b80 1 -- 172.21.15.114:0/835855094 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e43787c9f0 con 0x55e4378ca940 2024-05-27T07:54:00.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.887+0000 7f72f3fff700 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e4378ca940 0x55e4378ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.888+0000 7f72d3fff700 10 client.18840 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:00.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.888+0000 7f72f3fff700 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e4378ca940 0x55e4378ccda0 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=282dbbce8453a1b4 server_cookie=9daacc867a0a5fa6 in_seq=0 out_seq=0 2024-05-27T07:54:00.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f72d3fff700 1 -- 172.21.15.114:0/835855094 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e43787c9f0 con 0x55e4378ca940 2024-05-27T07:54:00.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f72d3fff700 10 client.18840 handle_command_reply: tid=0 2024-05-27T07:54:00.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f7309935b80 1 client.18840 shutdown 2024-05-27T07:54:00.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f7309935b80 1 -- 172.21.15.114:0/835855094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 msgr2=0x55e4378ba390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:00.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f7309935b80 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 0x55e4378ba390 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f72e80059d0 tx=0x7f72e8003e20 comp rx=0 tx=0).stop 2024-05-27T07:54:00.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f7309935b80 1 -- 172.21.15.114:0/835855094 shutdown_connections 2024-05-27T07:54:00.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f7309935b80 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e4378ca940 0x55e4378ccda0 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f7309935b80 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b6c50 0x55e4378bae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f7309935b80 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 0x55e4378ba390 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f72d3fff700 0 client.18840 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:00.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f7309935b80 1 --2- 172.21.15.114:0/835855094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b0f80 0x55e4378ba8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f7309935b80 1 -- 172.21.15.114:0/835855094 >> 172.21.15.114:0/835855094 conn(0x55e43778daf0 msgr2=0x55e4378afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:00.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f7309935b80 1 -- 172.21.15.114:0/835855094 shutdown_connections 2024-05-27T07:54:00.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f7309935b80 1 -- 172.21.15.114:0/835855094 wait complete. 2024-05-27T07:54:00.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.891+0000 7f7309935b80 20 client.18840 trim_cache size 0 max 16384 2024-05-27T07:54:00.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f7309935b80 1 Processor -- start 2024-05-27T07:54:00.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f7309935b80 1 -- start start 2024-05-27T07:54:00.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f7309935b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 0x55e4378b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f7309935b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b0f80 0x55e4378c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f7309935b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b6c50 0x55e4378bc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f7309935b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41b33c0 con 0x55e437732e40 2024-05-27T07:54:00.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f7309935b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41b3a90 con 0x55e4378b0f80 2024-05-27T07:54:00.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f7309935b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41b2240 con 0x55e4378b6c50 2024-05-27T07:54:00.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f72f8cbc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 0x55e4378b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f72f8cbc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 0x55e4378b5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53792/0 (socket says 172.21.15.114:53792) 2024-05-27T07:54:00.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f72f8cbc700 1 -- 172.21.15.114:0/4251151437 learned_addr learned my addr 172.21.15.114:0/4251151437 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:00.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f72f94bd700 1 --2- 172.21.15.114:0/4251151437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b6c50 0x55e4378bc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f72f3fff700 1 --2- 172.21.15.114:0/4251151437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b0f80 0x55e4378c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f72f8cbc700 1 -- 172.21.15.114:0/4251151437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b6c50 msgr2=0x55e4378bc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:00.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f72f8cbc700 1 --2- 172.21.15.114:0/4251151437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b6c50 0x55e4378bc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f72f8cbc700 1 -- 172.21.15.114:0/4251151437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b0f80 msgr2=0x55e4378c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:00.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f72f8cbc700 1 --2- 172.21.15.114:0/4251151437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b0f80 0x55e4378c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f72f8cbc700 1 -- 172.21.15.114:0/4251151437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4378a5fa0 con 0x55e437732e40 2024-05-27T07:54:00.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f72f94bd700 1 --2- 172.21.15.114:0/4251151437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b6c50 0x55e4378bc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.907+0000 7f72f8cbc700 1 --2- 172.21.15.114:0/4251151437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 0x55e4378b5af0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f72e80a9d90 tx=0x7f72e8009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eff9e78c3fc1b6d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.908+0000 7f72de7fc700 1 -- 172.21.15.114:0/4251151437 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e806d030 con 0x55e437732e40 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.908+0000 7f72de7fc700 1 -- 172.21.15.114:0/4251151437 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72e80aa790 con 0x55e437732e40 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.908+0000 7f72de7fc700 1 -- 172.21.15.114:0/4251151437 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e80ae0c0 con 0x55e437732e40 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.908+0000 7f7309935b80 1 -- 172.21.15.114:0/4251151437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 msgr2=0x55e4378b5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.908+0000 7f7309935b80 1 --2- 172.21.15.114:0/4251151437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 0x55e4378b5af0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f72e80a9d90 tx=0x7f72e8009660 comp rx=0 tx=0).stop 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.908+0000 7f7309935b80 1 -- 172.21.15.114:0/4251151437 shutdown_connections 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.908+0000 7f7309935b80 1 --2- 172.21.15.114:0/4251151437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e4378b6c50 0x55e4378bc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.908+0000 7f7309935b80 1 --2- 172.21.15.114:0/4251151437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e437732e40 0x55e4378b5af0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.908+0000 7f7309935b80 1 --2- 172.21.15.114:0/4251151437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b0f80 0x55e4378c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.908+0000 7f7309935b80 1 -- 172.21.15.114:0/4251151437 >> 172.21.15.114:0/4251151437 conn(0x55e43778daf0 msgr2=0x55e4378b53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.908+0000 7f7309935b80 1 -- 172.21.15.114:0/4251151437 shutdown_connections 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.908+0000 7f7309935b80 1 -- 172.21.15.114:0/4251151437 wait complete. 2024-05-27T07:54:00.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.909+0000 7f7309935b80 1 -- start start 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.909+0000 7f7309935b80 1 Processor -- start 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.909+0000 7f7309935b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e437732e40 0x55e4378c29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.909+0000 7f7309935b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b0f80 0x55e4378c2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.909+0000 7f7309935b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b6c50 0x55e4378d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f7309935b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41b1b70 con 0x55e4378b0f80 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f7309935b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41b0780 con 0x55e4378b6c50 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f7309935b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41b0dd0 con 0x55e437732e40 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f7309935b80 10 client.0 _ll_register_callbacks cb 0x55e43776a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72f3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b0f80 0x55e4378c2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72f8cbc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e437732e40 0x55e4378c29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72f3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b0f80 0x55e4378c2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53800/0 (socket says 172.21.15.114:53800) 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72f3fff700 1 -- 172.21.15.114:0/656776569 learned_addr learned my addr 172.21.15.114:0/656776569 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72f94bd700 1 --2- 172.21.15.114:0/656776569 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b6c50 0x55e4378d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72f3fff700 1 -- 172.21.15.114:0/656776569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e437732e40 msgr2=0x55e4378c29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72f3fff700 1 --2- 172.21.15.114:0/656776569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e437732e40 0x55e4378c29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72f3fff700 1 -- 172.21.15.114:0/656776569 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b6c50 msgr2=0x55e4378d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72f3fff700 1 --2- 172.21.15.114:0/656776569 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b6c50 0x55e4378d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72f3fff700 1 -- 172.21.15.114:0/656776569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4378d2ff0 con 0x55e4378b0f80 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72f3fff700 1 --2- 172.21.15.114:0/656776569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b0f80 0x55e4378c2f10 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x55e4378ad0a0 tx=0x7f72e00058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49dec28357ee953c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72d3fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72d3fff700 1 -- 172.21.15.114:0/656776569 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e000ec20 con 0x55e4378b0f80 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f7309935b80 1 -- 172.21.15.114:0/656776569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72e80aa790 con 0x55e4378b0f80 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72d3fff700 1 -- 172.21.15.114:0/656776569 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9538+0+0 (secure 0 0 0) 0x7f72e0005040 con 0x55e4378b0f80 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f72d3fff700 1 -- 172.21.15.114:0/656776569 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f72e0042050 con 0x55e4378b0f80 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f7309935b80 10 client.18846 fetch_fsmap learned FSMap version 76 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.910+0000 7f7309935b80 1 -- 172.21.15.114:0/656776569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55e437934910 con 0x55e4378b0f80 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.911+0000 7f72d3fff700 1 -- 172.21.15.114:0/656776569 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72e80aa790 con 0x55e4378b0f80 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.911+0000 7f72d3fff700 1 -- 172.21.15.114:0/656776569 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 76) ==== 5305+0+0 (secure 0 0 0) 0x7f72e003c980 con 0x55e4378b0f80 2024-05-27T07:54:00.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.911+0000 7f7309935b80 10 client.18846 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.911+0000 7f7309935b80 10 client.18846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.911+0000 7f7309935b80 20 client.18846 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.911+0000 7f7309935b80 1 --2- 172.21.15.114:0/656776569 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e437934f60 0x55e437937380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.911+0000 7f7309935b80 4 client.18846 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.911+0000 7f7309935b80 1 -- 172.21.15.114:0/656776569 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e4377e1be0 con 0x55e437934f60 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.911+0000 7f72f8cbc700 1 --2- 172.21.15.114:0/656776569 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e437934f60 0x55e437937380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.911+0000 7f72d3fff700 10 client.18846 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.911+0000 7f72f8cbc700 1 --2- 172.21.15.114:0/656776569 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e437934f60 0x55e437937380 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac49d98897ac78df server_cookie=54fc7178bf0b15c0 in_seq=0 out_seq=0 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f72d3fff700 1 -- 172.21.15.114:0/656776569 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e4377e1be0 con 0x55e437934f60 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f72d3fff700 10 client.18846 handle_command_reply: tid=0 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f7309935b80 1 client.18846 shutdown 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f7309935b80 1 -- 172.21.15.114:0/656776569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b0f80 msgr2=0x55e4378c2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f7309935b80 1 --2- 172.21.15.114:0/656776569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b0f80 0x55e4378c2f10 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x55e4378ad0a0 tx=0x7f72e00058e0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f7309935b80 1 -- 172.21.15.114:0/656776569 shutdown_connections 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f7309935b80 1 --2- 172.21.15.114:0/656776569 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e437934f60 0x55e437937380 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f7309935b80 1 --2- 172.21.15.114:0/656776569 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e4378b6c50 0x55e4378d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f7309935b80 1 --2- 172.21.15.114:0/656776569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e437732e40 0x55e4378c29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f72d3fff700 0 client.18846 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:00.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f7309935b80 1 --2- 172.21.15.114:0/656776569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e4378b0f80 0x55e4378c2f10 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f7309935b80 1 -- 172.21.15.114:0/656776569 >> 172.21.15.114:0/656776569 conn(0x55e43778daf0 msgr2=0x55e4378b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:00.236 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:54:00.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f7309935b80 1 -- 172.21.15.114:0/656776569 shutdown_connections 2024-05-27T07:54:00.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f7309935b80 1 -- 172.21.15.114:0/656776569 wait complete. 2024-05-27T07:54:00.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.916+0000 7f7309935b80 20 client.18846 trim_cache size 0 max 16384 2024-05-27T07:54:00.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.917+0000 7f72faf20700 1 -- 172.21.15.114:0/267476164 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f72e40b29f0 msgr2=0x7f72e40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:00.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.917+0000 7f72faf20700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f72e40b29f0 0x7f72e40b4f00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f72f4109ac0 tx=0x7f72e0009260 comp rx=0 tx=0).stop 2024-05-27T07:54:00.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.917+0000 7f72faf20700 1 -- 172.21.15.114:0/267476164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72f4103c50 msgr2=0x7f72f40febc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:00.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.917+0000 7f72faf20700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72f4103c50 0x7f72f40febc0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f72e8006010 tx=0x7f72e8004270 comp rx=0 tx=0).stop 2024-05-27T07:54:00.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.917+0000 7f72faf20700 1 -- 172.21.15.114:0/267476164 shutdown_connections 2024-05-27T07:54:00.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.917+0000 7f72faf20700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4108740 0x7f72f40ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.917+0000 7f72faf20700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72f4103c50 0x7f72f40febc0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.917+0000 7f72faf20700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f72e40b29f0 0x7f72e40b4f00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.917+0000 7f72faf20700 1 --2- 172.21.15.114:0/267476164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72f4107df0 0x7f72f40ff100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:00.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.917+0000 7f72faf20700 1 -- 172.21.15.114:0/267476164 >> 172.21.15.114:0/267476164 conn(0x7f72f40fe6e0 msgr2=0x7f72f4102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:00.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.917+0000 7f72faf20700 1 -- 172.21.15.114:0/267476164 shutdown_connections 2024-05-27T07:54:00.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:53:59.917+0000 7f72faf20700 1 -- 172.21.15.114:0/267476164 wait complete. 2024-05-27T07:54:00.589 INFO:tasks.workunit.client.0.smithi114.stdout: 10 3714 32.14 MB/sec warmup 12 sec latency 175.745 ms 2024-05-27T07:54:00.988 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:00 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:54:00.988 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:00 smithi163 systemd[1]: Stopped Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:54:00.988 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:00 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Consumed 1.838s CPU time. 2024-05-27T07:54:00.989 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:00 smithi163 systemd[1]: Starting Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:54:00.989 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:00 smithi163 podman[73769]: 2024-05-27 07:54:00.857466669 +0000 UTC m=+0.017830722 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:54:00.989 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:00 smithi163 podman[73769]: 2024-05-27 07:54:00.988296029 +0000 UTC m=+0.148660071 container create 60a2b494e2edde4a2cd3f49f1d161ddd95f885158091630739a06fce5971a65c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-05-27T07:54:01.514 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 podman[73769]: 2024-05-27 07:54:01.250248278 +0000 UTC m=+0.410612326 container init 60a2b494e2edde4a2cd3f49f1d161ddd95f885158091630739a06fce5971a65c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-27T07:54:01.515 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 podman[73769]: 2024-05-27 07:54:01.253397235 +0000 UTC m=+0.413761283 container start 60a2b494e2edde4a2cd3f49f1d161ddd95f885158091630739a06fce5971a65c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS) 2024-05-27T07:54:01.515 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-mds[73786]: -- 172.21.15.163:0/766185928 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fe409efa40 con 0x55fe40a37a80 2024-05-27T07:54:01.515 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: starting mds.e at 2024-05-27T07:54:01.515 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 bash[73769]: 60a2b494e2edde4a2cd3f49f1d161ddd95f885158091630739a06fce5971a65c 2024-05-27T07:54:01.515 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 systemd[1]: Started Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:54:01.589 INFO:tasks.workunit.client.0.smithi114.stdout: 10 3846 29.77 MB/sec warmup 13 sec latency 279.576 ms 2024-05-27T07:54:01.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:01 smithi114 ceph-mon[32038]: pgmap v479: 129 pgs: 129 active+clean; 200 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 16 MiB/s wr, 571 op/s 2024-05-27T07:54:01.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:01 smithi114 ceph-mon[32038]: daemon mds.e restarted 2024-05-27T07:54:01.892 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:01 smithi163 ceph-mon[36451]: pgmap v479: 129 pgs: 129 active+clean; 200 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 16 MiB/s wr, 571 op/s 2024-05-27T07:54:01.892 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:01 smithi163 ceph-mon[36451]: daemon mds.e restarted 2024-05-27T07:54:01.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:01 smithi186 ceph-mon[36647]: pgmap v479: 129 pgs: 129 active+clean; 200 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 16 MiB/s wr, 571 op/s 2024-05-27T07:54:01.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:01 smithi186 ceph-mon[36647]: daemon mds.e restarted 2024-05-27T07:54:02.110 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:54:02.110 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:54:02.227 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5af6d06700 time 2024-05-27T07:54:01.892473+0000 2024-05-27T07:54:02.227 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:02.227 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:02.227 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5b0954bba6] 2024-05-27T07:54:02.228 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b0954bdd1] 2024-05-27T07:54:02.228 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fe3e09723d] 2024-05-27T07:54:02.228 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55fe3e082971] 2024-05-27T07:54:02.228 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 5: (SnapRealm::check_cache() const+0x13e) [0x55fe3e080fa0] 2024-05-27T07:54:02.228 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fe3e081651] 2024-05-27T07:54:02.228 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fe3e016347] 2024-05-27T07:54:02.228 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fe3dd1287e] 2024-05-27T07:54:02.228 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55fe3dd17e94] 2024-05-27T07:54:02.228 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55fe3dd22969] 2024-05-27T07:54:02.228 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 11: (MDLog::_replay_thread()+0x1f38) [0x55fe3e0bb48c] 2024-05-27T07:54:02.228 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55fe3e0c49cb] 2024-05-27T07:54:02.229 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 13: (Thread::entry_wrapper()+0x43) [0x7f5b0951a72d] 2024-05-27T07:54:02.229 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5b0951a749] 2024-05-27T07:54:02.229 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5b080be1ca] 2024-05-27T07:54:02.229 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 16: clone() 2024-05-27T07:54:02.229 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:01.893+0000 7f5af6d06700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5af6d06700 time 2024-05-27T07:54:01.892473+0000 2024-05-27T07:54:02.230 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:02.230 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:02.230 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:02.230 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5b0954bba6] 2024-05-27T07:54:02.230 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b0954bdd1] 2024-05-27T07:54:02.230 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fe3e09723d] 2024-05-27T07:54:02.230 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55fe3e082971] 2024-05-27T07:54:02.230 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 5: (SnapRealm::check_cache() const+0x13e) [0x55fe3e080fa0] 2024-05-27T07:54:02.230 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fe3e081651] 2024-05-27T07:54:02.231 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fe3e016347] 2024-05-27T07:54:02.231 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fe3dd1287e] 2024-05-27T07:54:02.231 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55fe3dd17e94] 2024-05-27T07:54:02.231 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55fe3dd22969] 2024-05-27T07:54:02.231 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 11: (MDLog::_replay_thread()+0x1f38) [0x55fe3e0bb48c] 2024-05-27T07:54:02.231 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55fe3e0c49cb] 2024-05-27T07:54:02.231 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 13: (Thread::entry_wrapper()+0x43) [0x7f5b0951a72d] 2024-05-27T07:54:02.231 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5b0951a749] 2024-05-27T07:54:02.231 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5b080be1ca] 2024-05-27T07:54:02.231 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 16: clone() 2024-05-27T07:54:02.231 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:02.231 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: *** Caught signal (Aborted) ** 2024-05-27T07:54:02.231 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: in thread 7f5af6d06700 thread_name:md_log_replay 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5b080c8d20] 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2: gsignal() 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 3: abort() 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5b0954bced] 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b0954bdd1] 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fe3e09723d] 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55fe3e082971] 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 8: (SnapRealm::check_cache() const+0x13e) [0x55fe3e080fa0] 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fe3e081651] 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fe3e016347] 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fe3dd1287e] 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55fe3dd17e94] 2024-05-27T07:54:02.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55fe3dd22969] 2024-05-27T07:54:02.233 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 14: (MDLog::_replay_thread()+0x1f38) [0x55fe3e0bb48c] 2024-05-27T07:54:02.233 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55fe3e0c49cb] 2024-05-27T07:54:02.233 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 16: (Thread::entry_wrapper()+0x43) [0x7f5b0951a72d] 2024-05-27T07:54:02.233 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5b0951a749] 2024-05-27T07:54:02.233 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5b080be1ca] 2024-05-27T07:54:02.233 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 19: clone() 2024-05-27T07:54:02.233 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:01.896+0000 7f5af6d06700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:54:02.233 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: in thread 7f5af6d06700 thread_name:md_log_replay 2024-05-27T07:54:02.233 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:02.233 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:02.233 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5b080c8d20] 2024-05-27T07:54:02.234 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2: gsignal() 2024-05-27T07:54:02.234 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 3: abort() 2024-05-27T07:54:02.234 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5b0954bced] 2024-05-27T07:54:02.234 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b0954bdd1] 2024-05-27T07:54:02.234 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fe3e09723d] 2024-05-27T07:54:02.234 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55fe3e082971] 2024-05-27T07:54:02.234 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 8: (SnapRealm::check_cache() const+0x13e) [0x55fe3e080fa0] 2024-05-27T07:54:02.234 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fe3e081651] 2024-05-27T07:54:02.234 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fe3e016347] 2024-05-27T07:54:02.235 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fe3dd1287e] 2024-05-27T07:54:02.235 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55fe3dd17e94] 2024-05-27T07:54:02.235 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55fe3dd22969] 2024-05-27T07:54:02.235 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 14: (MDLog::_replay_thread()+0x1f38) [0x55fe3e0bb48c] 2024-05-27T07:54:02.235 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55fe3e0c49cb] 2024-05-27T07:54:02.235 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 16: (Thread::entry_wrapper()+0x43) [0x7f5b0951a72d] 2024-05-27T07:54:02.235 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5b0951a749] 2024-05-27T07:54:02.235 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5b080be1ca] 2024-05-27T07:54:02.235 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 19: clone() 2024-05-27T07:54:02.235 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:54:02.236 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:02.236 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: -1> 2024-05-27T07:54:01.893+0000 7f5af6d06700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5af6d06700 time 2024-05-27T07:54:01.892473+0000 2024-05-27T07:54:02.236 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:02.236 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:02.236 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:02.236 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5b0954bba6] 2024-05-27T07:54:02.236 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b0954bdd1] 2024-05-27T07:54:02.236 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fe3e09723d] 2024-05-27T07:54:02.236 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55fe3e082971] 2024-05-27T07:54:02.237 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 5: (SnapRealm::check_cache() const+0x13e) [0x55fe3e080fa0] 2024-05-27T07:54:02.237 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fe3e081651] 2024-05-27T07:54:02.237 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fe3e016347] 2024-05-27T07:54:02.237 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fe3dd1287e] 2024-05-27T07:54:02.237 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55fe3dd17e94] 2024-05-27T07:54:02.237 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55fe3dd22969] 2024-05-27T07:54:02.237 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 11: (MDLog::_replay_thread()+0x1f38) [0x55fe3e0bb48c] 2024-05-27T07:54:02.237 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55fe3e0c49cb] 2024-05-27T07:54:02.237 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 13: (Thread::entry_wrapper()+0x43) [0x7f5b0951a72d] 2024-05-27T07:54:02.238 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5b0951a749] 2024-05-27T07:54:02.238 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5b080be1ca] 2024-05-27T07:54:02.238 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 16: clone() 2024-05-27T07:54:02.238 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:02.238 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 0> 2024-05-27T07:54:01.896+0000 7f5af6d06700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:54:02.238 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: in thread 7f5af6d06700 thread_name:md_log_replay 2024-05-27T07:54:02.238 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:02.238 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:02.238 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5b080c8d20] 2024-05-27T07:54:02.238 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2: gsignal() 2024-05-27T07:54:02.238 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 3: abort() 2024-05-27T07:54:02.238 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5b0954bced] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b0954bdd1] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fe3e09723d] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55fe3e082971] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 8: (SnapRealm::check_cache() const+0x13e) [0x55fe3e080fa0] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fe3e081651] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fe3e016347] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fe3dd1287e] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55fe3dd17e94] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55fe3dd22969] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 14: (MDLog::_replay_thread()+0x1f38) [0x55fe3e0bb48c] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55fe3e0c49cb] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 16: (Thread::entry_wrapper()+0x43) [0x7f5b0951a72d] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5b0951a749] 2024-05-27T07:54:02.239 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5b080be1ca] 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 19: clone() 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: -1> 2024-05-27T07:54:01.893+0000 7f5af6d06700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5af6d06700 time 2024-05-27T07:54:01.892473+0000 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5b0954bba6] 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b0954bdd1] 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fe3e09723d] 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55fe3e082971] 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 5: (SnapRealm::check_cache() const+0x13e) [0x55fe3e080fa0] 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fe3e081651] 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fe3e016347] 2024-05-27T07:54:02.240 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fe3dd1287e] 2024-05-27T07:54:02.241 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55fe3dd17e94] 2024-05-27T07:54:02.241 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55fe3dd22969] 2024-05-27T07:54:02.241 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 11: (MDLog::_replay_thread()+0x1f38) [0x55fe3e0bb48c] 2024-05-27T07:54:02.241 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55fe3e0c49cb] 2024-05-27T07:54:02.241 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 13: (Thread::entry_wrapper()+0x43) [0x7f5b0951a72d] 2024-05-27T07:54:02.241 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5b0951a749] 2024-05-27T07:54:02.241 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5b080be1ca] 2024-05-27T07:54:02.241 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 16: clone() 2024-05-27T07:54:02.241 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:02.241 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 0> 2024-05-27T07:54:01.896+0000 7f5af6d06700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: in thread 7f5af6d06700 thread_name:md_log_replay 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5b080c8d20] 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2: gsignal() 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 3: abort() 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5b0954bced] 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b0954bdd1] 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55fe3e09723d] 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55fe3e082971] 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 8: (SnapRealm::check_cache() const+0x13e) [0x55fe3e080fa0] 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55fe3e081651] 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55fe3e016347] 2024-05-27T07:54:02.242 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55fe3dd1287e] 2024-05-27T07:54:02.243 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55fe3dd17e94] 2024-05-27T07:54:02.243 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55fe3dd22969] 2024-05-27T07:54:02.243 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 14: (MDLog::_replay_thread()+0x1f38) [0x55fe3e0bb48c] 2024-05-27T07:54:02.243 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55fe3e0c49cb] 2024-05-27T07:54:02.243 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 16: (Thread::entry_wrapper()+0x43) [0x7f5b0951a72d] 2024-05-27T07:54:02.243 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5b0951a749] 2024-05-27T07:54:02.243 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5b080be1ca] 2024-05-27T07:54:02.243 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 19: clone() 2024-05-27T07:54:02.243 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:54:02.243 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73782]: 2024-05-27T07:54:02.243 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:01 smithi163 podman[73817]: 2024-05-27 07:54:01.953574258 +0000 UTC m=+0.018984703 container died 60a2b494e2edde4a2cd3f49f1d161ddd95f885158091630739a06fce5971a65c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:54:02.401 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:54:02.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:02 smithi114 ceph-mon[32038]: osdmap e134: 12 total, 12 up, 12 in 2024-05-27T07:54:02.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:02 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6834/927041449,v1:172.21.15.163:6835/927041449] up:boot 2024-05-27T07:54:02.589 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4021 27.79 MB/sec warmup 14 sec latency 358.332 ms 2024-05-27T07:54:02.793 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:02 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 6 up:standby 2024-05-27T07:54:02.793 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:02 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:54:02.793 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:02 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:54:02.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:02 smithi163 ceph-mon[36451]: osdmap e134: 12 total, 12 up, 12 in 2024-05-27T07:54:02.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:02 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6834/927041449,v1:172.21.15.163:6835/927041449] up:boot 2024-05-27T07:54:02.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:02 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 6 up:standby 2024-05-27T07:54:02.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:02 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:54:02.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:02 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:54:02.977 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:02 smithi163 podman[73817]: 2024-05-27 07:54:02.742513301 +0000 UTC m=+0.807923760 container remove 60a2b494e2edde4a2cd3f49f1d161ddd95f885158091630739a06fce5971a65c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, ceph=True) 2024-05-27T07:54:02.977 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:02 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:54:02.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:02 smithi186 ceph-mon[36647]: osdmap e134: 12 total, 12 up, 12 in 2024-05-27T07:54:02.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:02 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6834/927041449,v1:172.21.15.163:6835/927041449] up:boot 2024-05-27T07:54:02.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:02 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 6 up:standby 2024-05-27T07:54:02.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:02 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:54:02.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:02 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:54:03.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:03 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-05-27T07:54:03.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:03 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Consumed 1.422s CPU time. 2024-05-27T07:54:03.589 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4118 26.07 MB/sec warmup 15 sec latency 695.384 ms 2024-05-27T07:54:03.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:03 smithi114 ceph-mon[32038]: pgmap v481: 129 pgs: 129 active+clean; 213 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 16 MiB/s wr, 680 op/s 2024-05-27T07:54:03.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:03 smithi114 ceph-mon[32038]: osdmap e135: 12 total, 12 up, 12 in 2024-05-27T07:54:03.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:03 smithi163 ceph-mon[36451]: pgmap v481: 129 pgs: 129 active+clean; 213 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 16 MiB/s wr, 680 op/s 2024-05-27T07:54:03.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:03 smithi163 ceph-mon[36451]: osdmap e135: 12 total, 12 up, 12 in 2024-05-27T07:54:03.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:03 smithi186 ceph-mon[36647]: pgmap v481: 129 pgs: 129 active+clean; 213 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 16 MiB/s wr, 680 op/s 2024-05-27T07:54:03.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:03 smithi186 ceph-mon[36647]: osdmap e135: 12 total, 12 up, 12 in 2024-05-27T07:54:04.589 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4231 24.60 MB/sec warmup 16 sec latency 819.873 ms 2024-05-27T07:54:05.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.012+0000 7f351dbb1700 1 -- 172.21.15.114:0/1717784257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351810b270 msgr2=0x7f351811d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:05.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.012+0000 7f351dbb1700 1 --2- 172.21.15.114:0/1717784257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351810b270 0x7f351811d3e0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f350c009990 tx=0x7f350c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.017+0000 7f351dbb1700 1 -- 172.21.15.114:0/1717784257 shutdown_connections 2024-05-27T07:54:05.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.017+0000 7f351dbb1700 1 --2- 172.21.15.114:0/1717784257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f351811d920 0x7f351811fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.017+0000 7f351dbb1700 1 --2- 172.21.15.114:0/1717784257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351810a920 0x7f351810ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.017+0000 7f351dbb1700 1 --2- 172.21.15.114:0/1717784257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351810b270 0x7f351811d3e0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.017+0000 7f351dbb1700 1 -- 172.21.15.114:0/1717784257 >> 172.21.15.114:0/1717784257 conn(0x7f351806f4c0 msgr2=0x7f351806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:05.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.022+0000 7f351dbb1700 1 -- 172.21.15.114:0/1717784257 shutdown_connections 2024-05-27T07:54:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.025+0000 7f351dbb1700 1 -- 172.21.15.114:0/1717784257 wait complete. 2024-05-27T07:54:05.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.026+0000 7f351dbb1700 1 Processor -- start 2024-05-27T07:54:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.031+0000 7f351dbb1700 1 -- start start 2024-05-27T07:54:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.032+0000 7f351dbb1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f351810a920 0x7f35181af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.032+0000 7f351dbb1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351810b270 0x7f35181afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.032+0000 7f351dbb1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351811d920 0x7f35181a9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.032+0000 7f351dbb1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f35181233a0 con 0x7f351810b270 2024-05-27T07:54:05.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.032+0000 7f351dbb1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3518123220 con 0x7f351811d920 2024-05-27T07:54:05.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.032+0000 7f351dbb1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3518123520 con 0x7f351810a920 2024-05-27T07:54:05.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.033+0000 7f3517fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351811d920 0x7f35181a9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:05.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.033+0000 7f3517fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351811d920 0x7f35181a9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:44258/0 (socket says 172.21.15.114:44258) 2024-05-27T07:54:05.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.033+0000 7f3517fff700 1 -- 172.21.15.114:0/3024021019 learned_addr learned my addr 172.21.15.114:0/3024021019 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.035+0000 7f3516ffd700 1 --2- 172.21.15.114:0/3024021019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351810b270 0x7f35181afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.036+0000 7f3517fff700 1 -- 172.21.15.114:0/3024021019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f351810a920 msgr2=0x7f35181af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.036+0000 7f3517fff700 1 --2- 172.21.15.114:0/3024021019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f351810a920 0x7f35181af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.036+0000 7f3517fff700 1 -- 172.21.15.114:0/3024021019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351810b270 msgr2=0x7f35181afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.036+0000 7f3517fff700 1 --2- 172.21.15.114:0/3024021019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351810b270 0x7f35181afef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.036+0000 7f3517fff700 1 -- 172.21.15.114:0/3024021019 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35181aa1e0 con 0x7f351811d920 2024-05-27T07:54:05.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.040+0000 7f3517fff700 1 --2- 172.21.15.114:0/3024021019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351811d920 0x7f35181a9b40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f351000b9e0 tx=0x7f351000bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:05.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.045+0000 7f3514ff9700 1 -- 172.21.15.114:0/3024021019 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510019070 con 0x7f351811d920 2024-05-27T07:54:05.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.045+0000 7f351dbb1700 1 -- 172.21.15.114:0/3024021019 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35181aa4d0 con 0x7f351811d920 2024-05-27T07:54:05.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.045+0000 7f351dbb1700 1 -- 172.21.15.114:0/3024021019 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3518112bc0 con 0x7f351811d920 2024-05-27T07:54:05.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.045+0000 7f3514ff9700 1 -- 172.21.15.114:0/3024021019 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3510014440 con 0x7f351811d920 2024-05-27T07:54:05.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.045+0000 7f3514ff9700 1 -- 172.21.15.114:0/3024021019 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351000ce60 con 0x7f351811d920 2024-05-27T07:54:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.053+0000 7f3514ff9700 1 -- 172.21.15.114:0/3024021019 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f351001c070 con 0x7f351811d920 2024-05-27T07:54:05.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.060+0000 7f352c5c6b80 1 Processor -- start 2024-05-27T07:54:05.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.060+0000 7f352c5c6b80 1 -- start start 2024-05-27T07:54:05.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.060+0000 7f352c5c6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a57526ec0 0x556a576aa790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.060+0000 7f352c5c6b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576aacd0 0x556a576a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.060+0000 7f352c5c6b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 0x556a576a5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.060+0000 7f352c5c6b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556a575d3d10 con 0x556a576aacd0 2024-05-27T07:54:05.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.060+0000 7f352c5c6b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556a575c38b0 con 0x556a576a5000 2024-05-27T07:54:05.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.060+0000 7f352c5c6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556a575c78f0 con 0x556a57526ec0 2024-05-27T07:54:05.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.061+0000 7f3514ff9700 1 --2- 172.21.15.114:0/3024021019 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35000b2c00 0x7f35000b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.062+0000 7f3514ff9700 1 -- 172.21.15.114:0/3024021019 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9642+0+0 (secure 0 0 0) 0x7f35100e1bb0 con 0x7f351811d920 2024-05-27T07:54:05.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.062+0000 7f35177fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576aacd0 0x556a576a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:05.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.062+0000 7f35177fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576aacd0 0x556a576a4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42662/0 (socket says 172.21.15.114:42662) 2024-05-27T07:54:05.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.062+0000 7f35177fe700 1 -- 172.21.15.114:0/2483426906 learned_addr learned my addr 172.21.15.114:0/2483426906 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:05.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.062+0000 7f3517fff700 1 --2- 172.21.15.114:0/3024021019 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35000b2c00 0x7f35000b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:05.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.062+0000 7f35177fe700 1 -- 172.21.15.114:0/2483426906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a57526ec0 msgr2=0x556a576aa790 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:54:05.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.062+0000 7f35177fe700 1 --2- 172.21.15.114:0/2483426906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a57526ec0 0x556a576aa790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.062+0000 7f35177fe700 1 -- 172.21.15.114:0/2483426906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 msgr2=0x556a576a5410 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:54:05.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.062+0000 7f35177fe700 1 --2- 172.21.15.114:0/2483426906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 0x556a576a5410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.062+0000 7f35177fe700 1 -- 172.21.15.114:0/2483426906 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a57513950 con 0x556a576aacd0 2024-05-27T07:54:05.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.063+0000 7f35177fe700 1 --2- 172.21.15.114:0/2483426906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576aacd0 0x556a576a4ac0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f350800ab20 tx=0x7f350800ae30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3cbc6d9dfde223 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:05.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.064+0000 7f3517fff700 1 --2- 172.21.15.114:0/3024021019 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35000b2c00 0x7f35000b5110 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f35100a5d50 tx=0x7f351000e040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:05.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.067+0000 7f34fdffb700 1 -- 172.21.15.114:0/2483426906 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350800ccd0 con 0x556a576aacd0 2024-05-27T07:54:05.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.067+0000 7f34fdffb700 1 -- 172.21.15.114:0/2483426906 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f350800f900 con 0x556a576aacd0 2024-05-27T07:54:05.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.067+0000 7f34fdffb700 1 -- 172.21.15.114:0/2483426906 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3508018940 con 0x556a576aacd0 2024-05-27T07:54:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.070+0000 7f352c5c6b80 1 -- 172.21.15.114:0/2483426906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576aacd0 msgr2=0x556a576a4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.070+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/2483426906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576aacd0 0x556a576a4ac0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f350800ab20 tx=0x7f350800ae30 comp rx=0 tx=0).stop 2024-05-27T07:54:05.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.071+0000 7f352c5c6b80 1 -- 172.21.15.114:0/2483426906 shutdown_connections 2024-05-27T07:54:05.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.071+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/2483426906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 0x556a576a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.071+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/2483426906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a57526ec0 0x556a576aa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.071+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/2483426906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576aacd0 0x556a576a4ac0 secure :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f350800ab20 tx=0x7f350800ae30 comp rx=0 tx=0).stop 2024-05-27T07:54:05.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.071+0000 7f352c5c6b80 1 -- 172.21.15.114:0/2483426906 >> 172.21.15.114:0/2483426906 conn(0x556a57581b70 msgr2=0x556a5769f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.075+0000 7f352c5c6b80 1 -- 172.21.15.114:0/2483426906 shutdown_connections 2024-05-27T07:54:05.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.077+0000 7f352c5c6b80 1 -- 172.21.15.114:0/2483426906 wait complete. 2024-05-27T07:54:05.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.078+0000 7f352c5c6b80 1 -- start start 2024-05-27T07:54:05.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.078+0000 7f352c5c6b80 1 Processor -- start 2024-05-27T07:54:05.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.081+0000 7f352c5c6b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a57526ec0 0x556a576ae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.082+0000 7f352c5c6b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 0x556a576aea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.082+0000 7f352c5c6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a576aef60 0x556a576bab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.082+0000 7f352c5c6b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556a574ca020 con 0x556a57526ec0 2024-05-27T07:54:05.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.082+0000 7f352c5c6b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556a57565840 con 0x556a576a5000 2024-05-27T07:54:05.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.082+0000 7f3516ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a57526ec0 0x556a576ae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:05.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.082+0000 7f3516ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a57526ec0 0x556a576ae4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42672/0 (socket says 172.21.15.114:42672) 2024-05-27T07:54:05.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.082+0000 7f3516ffd700 1 -- 172.21.15.114:0/261702385 learned_addr learned my addr 172.21.15.114:0/261702385 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:05.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.082+0000 7f35177fe700 1 --2- 172.21.15.114:0/261702385 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 0x556a576aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:05.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.082+0000 7f352c5c6b80 1 -- 172.21.15.114:0/261702385 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556a576b17c0 con 0x556a576aef60 2024-05-27T07:54:05.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.082+0000 7f352c5c6b80 10 client.0 _ll_register_callbacks cb 0x556a5755e9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.089+0000 7f35177fe700 1 -- 172.21.15.114:0/261702385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a576aef60 msgr2=0x556a576bab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.089+0000 7f35177fe700 1 --2- 172.21.15.114:0/261702385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a576aef60 0x556a576bab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.089+0000 7f35177fe700 1 -- 172.21.15.114:0/261702385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a57526ec0 msgr2=0x556a576ae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.089+0000 7f35177fe700 1 --2- 172.21.15.114:0/261702385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a57526ec0 0x556a576ae4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.089+0000 7f35177fe700 1 -- 172.21.15.114:0/261702385 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a57562710 con 0x556a576a5000 2024-05-27T07:54:05.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.092+0000 7f35177fe700 1 --2- 172.21.15.114:0/261702385 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 0x556a576aea20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x556a576a1720 tx=0x7f350800f680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2553f39bbfeda709 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:05.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.092+0000 7f34f6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:54:05.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.097+0000 7f34f6ffd700 1 -- 172.21.15.114:0/261702385 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3508026060 con 0x556a576a5000 2024-05-27T07:54:05.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.097+0000 7f352c5c6b80 1 -- 172.21.15.114:0/261702385 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f350800f900 con 0x556a576a5000 2024-05-27T07:54:05.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.097+0000 7f34f6ffd700 1 -- 172.21.15.114:0/261702385 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9642+0+0 (secure 0 0 0) 0x7f3508004020 con 0x556a576a5000 2024-05-27T07:54:05.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.104+0000 7f34f6ffd700 1 -- 172.21.15.114:0/261702385 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f3508004220 con 0x556a576a5000 2024-05-27T07:54:05.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.106+0000 7f352c5c6b80 10 client.28235 fetch_fsmap learned FSMap version 78 2024-05-27T07:54:05.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.106+0000 7f352c5c6b80 1 -- 172.21.15.114:0/261702385 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x556a576bc8b0 con 0x556a576a5000 2024-05-27T07:54:05.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.111+0000 7f34f6ffd700 1 -- 172.21.15.114:0/261702385 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f350800f900 con 0x556a576a5000 2024-05-27T07:54:05.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.111+0000 7f34f6ffd700 1 -- 172.21.15.114:0/261702385 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 78) ==== 5305+0+0 (secure 0 0 0) 0x7f3508029050 con 0x556a576a5000 2024-05-27T07:54:05.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.112+0000 7f352c5c6b80 10 client.28235 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:54:05.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.112+0000 7f352c5c6b80 10 client.28235 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:05.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.112+0000 7f352c5c6b80 20 client.28235 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:05.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.112+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/261702385 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a576beff0 0x556a576c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.112+0000 7f352c5c6b80 4 client.28235 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:54:05.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.112+0000 7f352c5c6b80 1 -- 172.21.15.114:0/261702385 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a57670a70 con 0x556a576beff0 2024-05-27T07:54:05.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.113+0000 7f3516ffd700 1 --2- 172.21.15.114:0/261702385 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a576beff0 0x556a576c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:05.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.115+0000 7f3516ffd700 1 --2- 172.21.15.114:0/261702385 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a576beff0 0x556a576c1480 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7603a5cdfae818d8 server_cookie=255f1e0bf9df7d1a in_seq=0 out_seq=0 2024-05-27T07:54:05.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.116+0000 7f34f6ffd700 10 client.28235 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:05.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.119+0000 7f34f6ffd700 1 -- 172.21.15.114:0/261702385 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556a57670a70 con 0x556a576beff0 2024-05-27T07:54:05.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.119+0000 7f34f6ffd700 10 client.28235 handle_command_reply: tid=0 2024-05-27T07:54:05.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.119+0000 7f352c5c6b80 1 client.28235 shutdown 2024-05-27T07:54:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.124+0000 7f352c5c6b80 1 -- 172.21.15.114:0/261702385 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 msgr2=0x556a576aea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.124+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/261702385 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 0x556a576aea20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x556a576a1720 tx=0x7f350800f680 comp rx=0 tx=0).stop 2024-05-27T07:54:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.124+0000 7f352c5c6b80 1 -- 172.21.15.114:0/261702385 shutdown_connections 2024-05-27T07:54:05.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.124+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/261702385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a576aef60 0x556a576bab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.124+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/261702385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a57526ec0 0x556a576ae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.124+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/261702385 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a576beff0 0x556a576c1480 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.124+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/261702385 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 0x556a576aea20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.124+0000 7f352c5c6b80 1 -- 172.21.15.114:0/261702385 >> 172.21.15.114:0/261702385 conn(0x556a57581b70 msgr2=0x556a576a3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:05.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.128+0000 7f34f6ffd700 0 client.28235 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:05.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.128+0000 7f352c5c6b80 1 -- 172.21.15.114:0/261702385 shutdown_connections 2024-05-27T07:54:05.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.128+0000 7f352c5c6b80 1 -- 172.21.15.114:0/261702385 wait complete. 2024-05-27T07:54:05.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.128+0000 7f352c5c6b80 20 client.28235 trim_cache size 0 max 16384 2024-05-27T07:54:05.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.166+0000 7f352c5c6b80 1 Processor -- start 2024-05-27T07:54:05.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.166+0000 7f352c5c6b80 1 -- start start 2024-05-27T07:54:05.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.167+0000 7f352c5c6b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a57526ec0 0x556a576c1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.167+0000 7f352c5c6b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576a5000 0x556a576b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.167+0000 7f3516ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a57526ec0 0x556a576c1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:05.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.167+0000 7f3516ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a57526ec0 0x556a576c1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:44268/0 (socket says 172.21.15.114:44268) 2024-05-27T07:54:05.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.167+0000 7f35177fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576a5000 0x556a576b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:05.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.168+0000 7f35177fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576a5000 0x556a576b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42688/0 (socket says 172.21.15.114:42688) 2024-05-27T07:54:05.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.168+0000 7f3516ffd700 1 -- 172.21.15.114:0/770214604 learned_addr learned my addr 172.21.15.114:0/770214604 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:05.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.168+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/770214604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a576aef60 0x556a576ba8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.168+0000 7f352c5c6b80 1 -- 172.21.15.114:0/770214604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f350400fbd0 con 0x556a576a5000 2024-05-27T07:54:05.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.168+0000 7f352c5c6b80 1 -- 172.21.15.114:0/770214604 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f35040102a0 con 0x556a57526ec0 2024-05-27T07:54:05.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.168+0000 7f352c5c6b80 1 -- 172.21.15.114:0/770214604 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f350400ea50 con 0x556a576aef60 2024-05-27T07:54:05.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.168+0000 7f3516ffd700 1 --2- 172.21.15.114:0/770214604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a576aef60 0x556a576ba8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:05.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.169+0000 7f3516ffd700 1 -- 172.21.15.114:0/770214604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a576aef60 msgr2=0x556a576ba8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:05.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.169+0000 7f3516ffd700 1 --2- 172.21.15.114:0/770214604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a576aef60 0x556a576ba8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.169+0000 7f3516ffd700 1 -- 172.21.15.114:0/770214604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576a5000 msgr2=0x556a576b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:05.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.169+0000 7f3516ffd700 1 --2- 172.21.15.114:0/770214604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576a5000 0x556a576b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.169+0000 7f3516ffd700 1 -- 172.21.15.114:0/770214604 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a576aca50 con 0x556a57526ec0 2024-05-27T07:54:05.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.169+0000 7f3516ffd700 1 --2- 172.21.15.114:0/770214604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a57526ec0 0x556a576c1670 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f350c009fd0 tx=0x7f350c009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8c736059a3ddcf0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:05.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.175+0000 7f34fd7fa700 1 -- 172.21.15.114:0/770214604 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c0038f0 con 0x556a57526ec0 2024-05-27T07:54:05.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.175+0000 7f34fd7fa700 1 -- 172.21.15.114:0/770214604 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f350c005110 con 0x556a57526ec0 2024-05-27T07:54:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.176+0000 7f352c5c6b80 1 -- 172.21.15.114:0/770214604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a57526ec0 msgr2=0x556a576c1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.176+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/770214604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a57526ec0 0x556a576c1670 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f350c009fd0 tx=0x7f350c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.178+0000 7f352c5c6b80 1 -- 172.21.15.114:0/770214604 shutdown_connections 2024-05-27T07:54:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.178+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/770214604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a576aef60 0x556a576ba8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.178+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/770214604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a57526ec0 0x556a576c1670 secure :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f350c009fd0 tx=0x7f350c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.178+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/770214604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576a5000 0x556a576b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.178+0000 7f352c5c6b80 1 -- 172.21.15.114:0/770214604 >> 172.21.15.114:0/770214604 conn(0x556a57581b70 msgr2=0x556a576acdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:05.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.182+0000 7f352c5c6b80 1 -- 172.21.15.114:0/770214604 shutdown_connections 2024-05-27T07:54:05.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.182+0000 7f352c5c6b80 1 -- 172.21.15.114:0/770214604 wait complete. 2024-05-27T07:54:05.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.182+0000 7f352c5c6b80 1 -- start start 2024-05-27T07:54:05.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.183+0000 7f352c5c6b80 1 Processor -- start 2024-05-27T07:54:05.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.183+0000 7f352c5c6b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 0x556a576c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.183+0000 7f352c5c6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a576aef60 0x556a576c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.183+0000 7f352c5c6b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576c7450 0x556a57728b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.183+0000 7f352c5c6b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f350400e380 con 0x556a576c7450 2024-05-27T07:54:05.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.183+0000 7f352c5c6b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f350400cf90 con 0x556a576a5000 2024-05-27T07:54:05.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.183+0000 7f352c5c6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f350400d5e0 con 0x556a576aef60 2024-05-27T07:54:05.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.183+0000 7f352c5c6b80 10 client.0 _ll_register_callbacks cb 0x556a5755e9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.185+0000 7f3516ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576c7450 0x556a57728b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.185+0000 7f3516ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576c7450 0x556a57728b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42690/0 (socket says 172.21.15.114:42690) 2024-05-27T07:54:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.185+0000 7f3516ffd700 1 -- 172.21.15.114:0/2607672280 learned_addr learned my addr 172.21.15.114:0/2607672280 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.185+0000 7f3516ffd700 1 --2- 172.21.15.114:0/2607672280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 0x556a576c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.185+0000 7f3516ffd700 1 -- 172.21.15.114:0/2607672280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a576aef60 msgr2=0x556a576c6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.185+0000 7f3516ffd700 1 --2- 172.21.15.114:0/2607672280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a576aef60 0x556a576c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.185+0000 7f3516ffd700 1 -- 172.21.15.114:0/2607672280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 msgr2=0x556a576c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.185+0000 7f3516ffd700 1 --2- 172.21.15.114:0/2607672280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 0x556a576c69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.185+0000 7f3516ffd700 1 -- 172.21.15.114:0/2607672280 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a57729090 con 0x556a576c7450 2024-05-27T07:54:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.186+0000 7f3516ffd700 1 --2- 172.21.15.114:0/2607672280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576c7450 0x556a57728b50 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f350c001eb0 tx=0x7f350c00baa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595d58e09977f4a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.186+0000 7f34f6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:54:05.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.187+0000 7f34f6ffd700 1 -- 172.21.15.114:0/2607672280 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c0038f0 con 0x556a576c7450 2024-05-27T07:54:05.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.187+0000 7f352c5c6b80 1 -- 172.21.15.114:0/2607672280 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556a575d5c60 con 0x556a576c7450 2024-05-27T07:54:05.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.187+0000 7f34f6ffd700 1 -- 172.21.15.114:0/2607672280 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9642+0+0 (secure 0 0 0) 0x7f350c003a90 con 0x556a576c7450 2024-05-27T07:54:05.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.191+0000 7f34f6ffd700 1 -- 172.21.15.114:0/2607672280 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f350c003c90 con 0x556a576c7450 2024-05-27T07:54:05.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.195+0000 7f352c5c6b80 10 client.18870 fetch_fsmap learned FSMap version 78 2024-05-27T07:54:05.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.195+0000 7f352c5c6b80 1 -- 172.21.15.114:0/2607672280 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x556a5772a3a0 con 0x556a576c7450 2024-05-27T07:54:05.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.198+0000 7f34f6ffd700 1 -- 172.21.15.114:0/2607672280 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556a575d5c60 con 0x556a576c7450 2024-05-27T07:54:05.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.198+0000 7f34f6ffd700 1 -- 172.21.15.114:0/2607672280 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 78) ==== 5305+0+0 (secure 0 0 0) 0x7f350c02f040 con 0x556a576c7450 2024-05-27T07:54:05.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.199+0000 7f352c5c6b80 10 client.18870 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:54:05.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.199+0000 7f352c5c6b80 10 client.18870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:05.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.199+0000 7f352c5c6b80 20 client.18870 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:05.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.199+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/2607672280 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a5772a750 0x556a5772cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:05.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.199+0000 7f352c5c6b80 4 client.18870 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:54:05.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.199+0000 7f352c5c6b80 1 -- 172.21.15.114:0/2607672280 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556a575cf440 con 0x556a5772a750 2024-05-27T07:54:05.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.204+0000 7f35177fe700 1 --2- 172.21.15.114:0/2607672280 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a5772a750 0x556a5772cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:05.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.229+0000 7f34f6ffd700 10 client.18870 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:05.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.229+0000 7f35177fe700 1 --2- 172.21.15.114:0/2607672280 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a5772a750 0x556a5772cb70 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cb4995bf6b030c1 server_cookie=cd592bfc7d4c4f9d in_seq=0 out_seq=0 2024-05-27T07:54:05.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.230+0000 7f34f6ffd700 1 -- 172.21.15.114:0/2607672280 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556a575cf440 con 0x556a5772a750 2024-05-27T07:54:05.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.230+0000 7f34f6ffd700 10 client.18870 handle_command_reply: tid=0 2024-05-27T07:54:05.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.230+0000 7f352c5c6b80 1 client.18870 shutdown 2024-05-27T07:54:05.234 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:54:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:54:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "8d9ec0de-bd1e-4db0-8770-c1ca85cb9698", 2024-05-27T07:54:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:54:05.235 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:54:05.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.231+0000 7f352c5c6b80 1 -- 172.21.15.114:0/2607672280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576c7450 msgr2=0x556a57728b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.231+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/2607672280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576c7450 0x556a57728b50 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f350c001eb0 tx=0x7f350c00baa0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.231+0000 7f352c5c6b80 1 -- 172.21.15.114:0/2607672280 shutdown_connections 2024-05-27T07:54:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.231+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/2607672280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a576c7450 0x556a57728b50 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.231+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/2607672280 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a5772a750 0x556a5772cb70 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.231+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/2607672280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a576a5000 0x556a576c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.231+0000 7f352c5c6b80 1 --2- 172.21.15.114:0/2607672280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a576aef60 0x556a576c6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.231+0000 7f34f6ffd700 0 client.18870 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.231+0000 7f352c5c6b80 1 -- 172.21.15.114:0/2607672280 >> 172.21.15.114:0/2607672280 conn(0x556a57581b70 msgr2=0x556a576a8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.233+0000 7f352c5c6b80 1 -- 172.21.15.114:0/2607672280 shutdown_connections 2024-05-27T07:54:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.233+0000 7f352c5c6b80 1 -- 172.21.15.114:0/2607672280 wait complete. 2024-05-27T07:54:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.233+0000 7f352c5c6b80 20 client.18870 trim_cache size 0 max 16384 2024-05-27T07:54:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.234+0000 7f351dbb1700 1 -- 172.21.15.114:0/3024021019 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35000b2c00 msgr2=0x7f35000b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.234+0000 7f351dbb1700 1 --2- 172.21.15.114:0/3024021019 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35000b2c00 0x7f35000b5110 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f35100a5d50 tx=0x7f351000e040 comp rx=0 tx=0).stop 2024-05-27T07:54:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.234+0000 7f351dbb1700 1 -- 172.21.15.114:0/3024021019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351811d920 msgr2=0x7f35181a9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.234+0000 7f351dbb1700 1 --2- 172.21.15.114:0/3024021019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351811d920 0x7f35181a9b40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f351000b9e0 tx=0x7f351000bda0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.234+0000 7f351dbb1700 1 -- 172.21.15.114:0/3024021019 shutdown_connections 2024-05-27T07:54:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.234+0000 7f351dbb1700 1 --2- 172.21.15.114:0/3024021019 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35000b2c00 0x7f35000b5110 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.234+0000 7f351dbb1700 1 --2- 172.21.15.114:0/3024021019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351811d920 0x7f35181a9b40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.234+0000 7f351dbb1700 1 --2- 172.21.15.114:0/3024021019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f351810a920 0x7f35181af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.234+0000 7f351dbb1700 1 --2- 172.21.15.114:0/3024021019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f351810b270 0x7f35181afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.234+0000 7f351dbb1700 1 -- 172.21.15.114:0/3024021019 >> 172.21.15.114:0/3024021019 conn(0x7f351806f4c0 msgr2=0x7f351811b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.234+0000 7f351dbb1700 1 -- 172.21.15.114:0/3024021019 shutdown_connections 2024-05-27T07:54:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:05.235+0000 7f351dbb1700 1 -- 172.21.15.114:0/3024021019 wait complete. 2024-05-27T07:54:05.589 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4307 23.28 MB/sec warmup 17 sec latency 854.521 ms 2024-05-27T07:54:05.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:05 smithi114 ceph-mon[32038]: pgmap v483: 129 pgs: 129 active+clean; 213 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.3 MiB/s wr, 379 op/s 2024-05-27T07:54:05.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:05 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:54:05.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:05 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:05.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:05 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:54:05.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:05 smithi163 ceph-mon[36451]: pgmap v483: 129 pgs: 129 active+clean; 213 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.3 MiB/s wr, 379 op/s 2024-05-27T07:54:05.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:05 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:54:05.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:05 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:05.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:05 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:54:05.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:05 smithi186 ceph-mon[36647]: pgmap v483: 129 pgs: 129 active+clean; 213 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.3 MiB/s wr, 379 op/s 2024-05-27T07:54:05.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:05 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:54:05.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:05 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:05.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:05 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:54:06.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:54:06.589 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4339 22.06 MB/sec warmup 18 sec latency 1451.782 ms 2024-05-27T07:54:06.966 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:54:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.403+0000 7f17ee3b4640 1 -- 172.21.15.114:0/885817285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e81075c0 msgr2=0x7f17e810ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.403+0000 7f17ee3b4640 1 --2- 172.21.15.114:0/885817285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e81075c0 0x7f17e810ed60 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009be0 tx=0x7f17e002a6d0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.408+0000 7f17ee3b4640 1 -- 172.21.15.114:0/885817285 shutdown_connections 2024-05-27T07:54:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.408+0000 7f17ee3b4640 1 --2- 172.21.15.114:0/885817285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e81075c0 0x7f17e810ed60 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.408+0000 7f17ee3b4640 1 --2- 172.21.15.114:0/885817285 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17e8106b00 0x7f17e8106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.408+0000 7f17ee3b4640 1 --2- 172.21.15.114:0/885817285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17e8102570 0x7f17e8102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.408+0000 7f17ee3b4640 1 -- 172.21.15.114:0/885817285 >> 172.21.15.114:0/885817285 conn(0x7f17e80779d0 msgr2=0x7f17e8077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:07.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.412+0000 7f17ee3b4640 1 -- 172.21.15.114:0/885817285 shutdown_connections 2024-05-27T07:54:07.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.413+0000 7f17ee3b4640 1 -- 172.21.15.114:0/885817285 wait complete. 2024-05-27T07:54:07.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.414+0000 7f17ee3b4640 1 Processor -- start 2024-05-27T07:54:07.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.414+0000 7f17ee3b4640 1 -- start start 2024-05-27T07:54:07.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.414+0000 7f17ee3b4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e8102570 0x7f17e81a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.414+0000 7f17ee3b4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17e8106b00 0x7f17e81a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.415+0000 7f17e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e8102570 0x7f17e81a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:07.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.415+0000 7f17e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e8102570 0x7f17e81a0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42698/0 (socket says 172.21.15.114:42698) 2024-05-27T07:54:07.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.420+0000 7f17ee3b4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17e81075c0 0x7f17e819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.420+0000 7f17ee3b4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f17e8111c50 con 0x7f17e8102570 2024-05-27T07:54:07.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.420+0000 7f17ee3b4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f17e8111ad0 con 0x7f17e81075c0 2024-05-27T07:54:07.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.420+0000 7f17ee3b4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17e8111dd0 con 0x7f17e8106b00 2024-05-27T07:54:07.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.420+0000 7f17e7fff640 1 -- 172.21.15.114:0/987584928 learned_addr learned my addr 172.21.15.114:0/987584928 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:07.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.420+0000 7f17e77fe640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17e8106b00 0x7f17e81a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:07.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.421+0000 7f17ec951640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17e81075c0 0x7f17e819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.422+0000 7f17ec951640 1 -- 172.21.15.114:0/987584928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17e8106b00 msgr2=0x7f17e81a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.422+0000 7f17ec951640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17e8106b00 0x7f17e81a0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.422+0000 7f17ec951640 1 -- 172.21.15.114:0/987584928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e8102570 msgr2=0x7f17e81a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.422+0000 7f17ec951640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e8102570 0x7f17e81a0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.422+0000 7f17ec951640 1 -- 172.21.15.114:0/987584928 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e819adb0 con 0x7f17e81075c0 2024-05-27T07:54:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.422+0000 7f17ec951640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17e81075c0 0x7f17e819a680 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f17e0034040 tx=0x7f17e0003cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.423+0000 7f17e57fa640 1 -- 172.21.15.114:0/987584928 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0040020 con 0x7f17e81075c0 2024-05-27T07:54:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.423+0000 7f17ee3b4640 1 -- 172.21.15.114:0/987584928 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17e819afe0 con 0x7f17e81075c0 2024-05-27T07:54:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.423+0000 7f17ee3b4640 1 -- 172.21.15.114:0/987584928 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17e8198380 con 0x7f17e81075c0 2024-05-27T07:54:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.423+0000 7f17e57fa640 1 -- 172.21.15.114:0/987584928 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e0031af0 con 0x7f17e81075c0 2024-05-27T07:54:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.423+0000 7f17e57fa640 1 -- 172.21.15.114:0/987584928 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0036040 con 0x7f17e81075c0 2024-05-27T07:54:07.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.427+0000 7f17e57fa640 1 -- 172.21.15.114:0/987584928 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f17e0043020 con 0x7f17e81075c0 2024-05-27T07:54:07.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.428+0000 7f17e57fa640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f17d00c3ab0 0x7f17d00c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.429+0000 7f17e57fa640 1 -- 172.21.15.114:0/987584928 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9642+0+0 (secure 0 0 0) 0x7f17e01044a0 con 0x7f17e81075c0 2024-05-27T07:54:07.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.429+0000 7f17e7fff640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f17d00c3ab0 0x7f17d00c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.433+0000 7f17e7fff640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f17d00c3ab0 0x7f17d00c5fc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f17d80044f0 tx=0x7f17d8014040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.433+0000 7f17f5261740 1 Processor -- start 2024-05-27T07:54:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.434+0000 7f17f5261740 1 -- start start 2024-05-27T07:54:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.434+0000 7f17f5261740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239bbe1b0 0x55a239f5f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.434+0000 7f17f5261740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b952f0 0x55a239f5f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.434+0000 7f17f5261740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239b9d4d0 0x55a239f5fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.434+0000 7f17f5261740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a239ed1c30 con 0x55a239bbe1b0 2024-05-27T07:54:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.434+0000 7f17f5261740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a239ec17d0 con 0x55a239b952f0 2024-05-27T07:54:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.434+0000 7f17f5261740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a239ec5810 con 0x55a239b9d4d0 2024-05-27T07:54:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.435+0000 7f17ec951640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239b9d4d0 0x55a239f5fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.435+0000 7f17ec951640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239b9d4d0 0x55a239f5fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40722/0 (socket says 172.21.15.114:40722) 2024-05-27T07:54:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.435+0000 7f17ec951640 1 -- 172.21.15.114:0/2470094346 learned_addr learned my addr 172.21.15.114:0/2470094346 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.435+0000 7f17ec951640 1 -- 172.21.15.114:0/2470094346 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b952f0 msgr2=0x55a239f5f910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:54:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.435+0000 7f17ec951640 1 --2- 172.21.15.114:0/2470094346 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b952f0 0x55a239f5f910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.435+0000 7f17ec951640 1 -- 172.21.15.114:0/2470094346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239bbe1b0 msgr2=0x55a239f5f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.435+0000 7f17ec951640 1 --2- 172.21.15.114:0/2470094346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239bbe1b0 0x55a239f5f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.435+0000 7f17ec951640 1 -- 172.21.15.114:0/2470094346 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a239bd6210 con 0x55a239b9d4d0 2024-05-27T07:54:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.435+0000 7f17ec951640 1 --2- 172.21.15.114:0/2470094346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239b9d4d0 0x55a239f5fe50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009b30 tx=0x7f17e00cb260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fda21f0546d49f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.436+0000 7f17ce7fc640 1 -- 172.21.15.114:0/2470094346 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e00caee0 con 0x55a239b9d4d0 2024-05-27T07:54:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.436+0000 7f17ce7fc640 1 -- 172.21.15.114:0/2470094346 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e00cbad0 con 0x55a239b9d4d0 2024-05-27T07:54:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.437+0000 7f17f5261740 1 -- 172.21.15.114:0/2470094346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239b9d4d0 msgr2=0x55a239f5fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.437+0000 7f17f5261740 1 --2- 172.21.15.114:0/2470094346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239b9d4d0 0x55a239f5fe50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009b30 tx=0x7f17e00cb260 comp rx=0 tx=0).stop 2024-05-27T07:54:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.437+0000 7f17f5261740 1 -- 172.21.15.114:0/2470094346 shutdown_connections 2024-05-27T07:54:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.437+0000 7f17f5261740 1 --2- 172.21.15.114:0/2470094346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239b9d4d0 0x55a239f5fe50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.437+0000 7f17f5261740 1 --2- 172.21.15.114:0/2470094346 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b952f0 0x55a239f5f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.437+0000 7f17f5261740 1 --2- 172.21.15.114:0/2470094346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239bbe1b0 0x55a239f5f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.437+0000 7f17f5261740 1 -- 172.21.15.114:0/2470094346 >> 172.21.15.114:0/2470094346 conn(0x55a239bb2300 msgr2=0x55a239f5bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.449+0000 7f17f5261740 1 -- 172.21.15.114:0/2470094346 shutdown_connections 2024-05-27T07:54:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.451+0000 7f17f5261740 1 -- 172.21.15.114:0/2470094346 wait complete. 2024-05-27T07:54:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.452+0000 7f17f5261740 1 -- start start 2024-05-27T07:54:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.452+0000 7f17f5261740 1 Processor -- start 2024-05-27T07:54:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.453+0000 7f17f5261740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239f695d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.453+0000 7f17f5261740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b7cd80 0x55a239f69b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.453+0000 7f17f5261740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b521c0 0x55a239f7ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.453+0000 7f17f5261740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a239ba7790 con 0x55a239b7cd80 2024-05-27T07:54:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.453+0000 7f17f5261740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a239e7d890 con 0x55a239b521c0 2024-05-27T07:54:07.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.453+0000 7f17f5261740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a239f6c2c0 con 0x55a239bbe1b0 2024-05-27T07:54:07.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.453+0000 7f17f5261740 10 client.0 _ll_register_callbacks cb 0x55a239bd5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:07.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.453+0000 7f17e7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239f695d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:07.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.453+0000 7f17e7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239f695d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40730/0 (socket says 172.21.15.114:40730) 2024-05-27T07:54:07.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.453+0000 7f17e7fff640 1 -- 172.21.15.114:0/1873083842 learned_addr learned my addr 172.21.15.114:0/1873083842 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:07.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.454+0000 7f17ec951640 1 --2- 172.21.15.114:0/1873083842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b521c0 0x55a239f7ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.458+0000 7f17ec951640 1 -- 172.21.15.114:0/1873083842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 msgr2=0x55a239f695d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.458+0000 7f17ec951640 1 --2- 172.21.15.114:0/1873083842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239f695d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.458+0000 7f17ec951640 1 -- 172.21.15.114:0/1873083842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b7cd80 msgr2=0x55a239f69b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.458+0000 7f17ec951640 1 --2- 172.21.15.114:0/1873083842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b7cd80 0x55a239f69b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.458+0000 7f17ec951640 1 -- 172.21.15.114:0/1873083842 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a239ca2620 con 0x55a239b521c0 2024-05-27T07:54:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.460+0000 7f17ec951640 1 --2- 172.21.15.114:0/1873083842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b521c0 0x55a239f7ddc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f17e00cbc80 tx=0x7f17e00c7910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f959e321134a8af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.460+0000 7f17c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:54:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.461+0000 7f17c77fe640 1 -- 172.21.15.114:0/1873083842 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e00caee0 con 0x55a239b521c0 2024-05-27T07:54:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.461+0000 7f17f5261740 1 -- 172.21.15.114:0/1873083842 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a239ed3b80 con 0x55a239b521c0 2024-05-27T07:54:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.461+0000 7f17c77fe640 1 -- 172.21.15.114:0/1873083842 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9642+0+0 (secure 0 0 0) 0x7f17e00d3da0 con 0x55a239b521c0 2024-05-27T07:54:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.463+0000 7f17c77fe640 1 -- 172.21.15.114:0/1873083842 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f17e0101bc0 con 0x55a239b521c0 2024-05-27T07:54:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.467+0000 7f17f5261740 10 client.28250 fetch_fsmap learned FSMap version 78 2024-05-27T07:54:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.467+0000 7f17f5261740 1 -- 172.21.15.114:0/1873083842 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55a239f6ed50 con 0x55a239b521c0 2024-05-27T07:54:07.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.476+0000 7f17c77fe640 1 -- 172.21.15.114:0/1873083842 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a239ed3b80 con 0x55a239b521c0 2024-05-27T07:54:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.477+0000 7f17c77fe640 1 -- 172.21.15.114:0/1873083842 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 78) ==== 5305+0+0 (secure 0 0 0) 0x7f17e00edba0 con 0x55a239b521c0 2024-05-27T07:54:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.477+0000 7f17f5261740 10 client.28250 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:54:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.477+0000 7f17f5261740 10 client.28250 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.477+0000 7f17f5261740 20 client.28250 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.477+0000 7f17f5261740 1 --2- 172.21.15.114:0/1873083842 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a239f70260 0x55a239f726a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.477+0000 7f17f5261740 4 client.28250 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:54:07.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.477+0000 7f17f5261740 1 -- 172.21.15.114:0/1873083842 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a239ecd360 con 0x55a239f70260 2024-05-27T07:54:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.479+0000 7f17e7fff640 1 --2- 172.21.15.114:0/1873083842 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a239f70260 0x55a239f726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.480+0000 7f17e7fff640 1 --2- 172.21.15.114:0/1873083842 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a239f70260 0x55a239f726a0 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ef59a0100cf6579 server_cookie=1a635b0ff851df58 in_seq=0 out_seq=0 2024-05-27T07:54:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.480+0000 7f17c77fe640 10 client.28250 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.481+0000 7f17c77fe640 1 -- 172.21.15.114:0/1873083842 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a239ecd360 con 0x55a239f70260 2024-05-27T07:54:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.481+0000 7f17c77fe640 10 client.28250 handle_command_reply: tid=0 2024-05-27T07:54:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.481+0000 7f17f5261740 1 client.28250 shutdown 2024-05-27T07:54:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.482+0000 7f17f5261740 1 -- 172.21.15.114:0/1873083842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b521c0 msgr2=0x55a239f7ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.482+0000 7f17f5261740 1 --2- 172.21.15.114:0/1873083842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b521c0 0x55a239f7ddc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f17e00cbc80 tx=0x7f17e00c7910 comp rx=0 tx=0).stop 2024-05-27T07:54:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.487+0000 7f17f5261740 1 -- 172.21.15.114:0/1873083842 shutdown_connections 2024-05-27T07:54:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.487+0000 7f17f5261740 1 --2- 172.21.15.114:0/1873083842 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a239f70260 0x55a239f726a0 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.487+0000 7f17f5261740 1 --2- 172.21.15.114:0/1873083842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b521c0 0x55a239f7ddc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.487+0000 7f17f5261740 1 --2- 172.21.15.114:0/1873083842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b7cd80 0x55a239f69b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.487+0000 7f17f5261740 1 --2- 172.21.15.114:0/1873083842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239f695d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.487+0000 7f17f5261740 1 -- 172.21.15.114:0/1873083842 >> 172.21.15.114:0/1873083842 conn(0x55a239bb2300 msgr2=0x55a239f610f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.487+0000 7f17c77fe640 0 client.28250 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:07.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.490+0000 7f17f5261740 1 -- 172.21.15.114:0/1873083842 shutdown_connections 2024-05-27T07:54:07.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.492+0000 7f17f5261740 1 -- 172.21.15.114:0/1873083842 wait complete. 2024-05-27T07:54:07.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.493+0000 7f17f5261740 20 client.28250 trim_cache size 0 max 16384 2024-05-27T07:54:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.523+0000 7f17f5261740 1 Processor -- start 2024-05-27T07:54:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.523+0000 7f17f5261740 1 -- start start 2024-05-27T07:54:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.523+0000 7f17f5261740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b521c0 0x55a239f6bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.523+0000 7f17f5261740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b7cd80 0x55a239f7dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.523+0000 7f17f5261740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239f69570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.524+0000 7f17f5261740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f17d4010c90 con 0x55a239b521c0 2024-05-27T07:54:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.524+0000 7f17f5261740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f17d4011360 con 0x55a239b7cd80 2024-05-27T07:54:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.524+0000 7f17f5261740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17d400fb30 con 0x55a239bbe1b0 2024-05-27T07:54:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.524+0000 7f17e77fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b7cd80 0x55a239f7dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.524+0000 7f17ec951640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239f69570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.524+0000 7f17ec951640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239f69570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40732/0 (socket says 172.21.15.114:40732) 2024-05-27T07:54:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.524+0000 7f17ec951640 1 -- 172.21.15.114:0/2326309886 learned_addr learned my addr 172.21.15.114:0/2326309886 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.525+0000 7f17e77fe640 1 -- 172.21.15.114:0/2326309886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 msgr2=0x55a239f69570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.525+0000 7f17e77fe640 1 --2- 172.21.15.114:0/2326309886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239f69570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.525+0000 7f17e77fe640 1 -- 172.21.15.114:0/2326309886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b521c0 msgr2=0x55a239f6bd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:54:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.525+0000 7f17e77fe640 1 --2- 172.21.15.114:0/2326309886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b521c0 0x55a239f6bd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.525+0000 7f17e77fe640 1 -- 172.21.15.114:0/2326309886 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a239f607b0 con 0x55a239b7cd80 2024-05-27T07:54:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.525+0000 7f17e77fe640 1 --2- 172.21.15.114:0/2326309886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b7cd80 0x55a239f7dd30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55a239f5cb10 tx=0x7f17dc00ba60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9292b501710a78d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.526+0000 7f17cdffb640 1 -- 172.21.15.114:0/2326309886 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc003910 con 0x55a239b7cd80 2024-05-27T07:54:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.527+0000 7f17cdffb640 1 -- 172.21.15.114:0/2326309886 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17dc015070 con 0x55a239b7cd80 2024-05-27T07:54:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.527+0000 7f17cdffb640 1 -- 172.21.15.114:0/2326309886 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc011440 con 0x55a239b7cd80 2024-05-27T07:54:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.529+0000 7f17f5261740 1 -- 172.21.15.114:0/2326309886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b7cd80 msgr2=0x55a239f7dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.530+0000 7f17f5261740 1 --2- 172.21.15.114:0/2326309886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b7cd80 0x55a239f7dd30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55a239f5cb10 tx=0x7f17dc00ba60 comp rx=0 tx=0).stop 2024-05-27T07:54:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.531+0000 7f17f5261740 1 -- 172.21.15.114:0/2326309886 shutdown_connections 2024-05-27T07:54:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.531+0000 7f17f5261740 1 --2- 172.21.15.114:0/2326309886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239f69570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.531+0000 7f17f5261740 1 --2- 172.21.15.114:0/2326309886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b7cd80 0x55a239f7dd30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.531+0000 7f17f5261740 1 --2- 172.21.15.114:0/2326309886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b521c0 0x55a239f6bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.531+0000 7f17f5261740 1 -- 172.21.15.114:0/2326309886 >> 172.21.15.114:0/2326309886 conn(0x55a239bb2300 msgr2=0x55a239f68e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.536+0000 7f17f5261740 1 -- 172.21.15.114:0/2326309886 shutdown_connections 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.537+0000 7f17f5261740 1 -- 172.21.15.114:0/2326309886 wait complete. 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.537+0000 7f17f5261740 1 -- start start 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.538+0000 7f17f5261740 1 Processor -- start 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.538+0000 7f17f5261740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b521c0 0x55a239f73530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.538+0000 7f17f5261740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b7cd80 0x55a239f73a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.538+0000 7f17f5261740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239fed1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.538+0000 7f17f5261740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f17d400f460 con 0x55a239b521c0 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.538+0000 7f17f5261740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f17d400e100 con 0x55a239b7cd80 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.538+0000 7f17f5261740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17d400e6c0 con 0x55a239bbe1b0 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.538+0000 7f17f5261740 10 client.0 _ll_register_callbacks cb 0x7f17e00c7290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.539+0000 7f17e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b521c0 0x55a239f73530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.539+0000 7f17e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b521c0 0x55a239f73530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42728/0 (socket says 172.21.15.114:42728) 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.539+0000 7f17e7fff640 1 -- 172.21.15.114:0/1293732599 learned_addr learned my addr 172.21.15.114:0/1293732599 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.539+0000 7f17ec951640 1 --2- 172.21.15.114:0/1293732599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239fed1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.540+0000 7f17e7fff640 1 -- 172.21.15.114:0/1293732599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 msgr2=0x55a239fed1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.540+0000 7f17e7fff640 1 --2- 172.21.15.114:0/1293732599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239fed1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.540+0000 7f17e7fff640 1 -- 172.21.15.114:0/1293732599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b7cd80 msgr2=0x55a239f73a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:54:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.540+0000 7f17e7fff640 1 --2- 172.21.15.114:0/1293732599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b7cd80 0x55a239f73a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.540+0000 7f17e7fff640 1 -- 172.21.15.114:0/1293732599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a239fed700 con 0x55a239b521c0 2024-05-27T07:54:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.540+0000 7f17e7fff640 1 --2- 172.21.15.114:0/1293732599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b521c0 0x55a239f73530 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x55a239f68e30 tx=0x7f17d8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e708a0dc32f2e0d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.540+0000 7f17c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:54:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.556+0000 7f17c77fe640 1 -- 172.21.15.114:0/1293732599 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d802e060 con 0x55a239b521c0 2024-05-27T07:54:07.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.556+0000 7f17f5261740 1 -- 172.21.15.114:0/1293732599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17dc015070 con 0x55a239b521c0 2024-05-27T07:54:07.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.556+0000 7f17c77fe640 1 -- 172.21.15.114:0/1293732599 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9642+0+0 (secure 0 0 0) 0x7f17d8003800 con 0x55a239b521c0 2024-05-27T07:54:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.561+0000 7f17c77fe640 1 -- 172.21.15.114:0/1293732599 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f17d8003a00 con 0x55a239b521c0 2024-05-27T07:54:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.561+0000 7f17f5261740 10 client.18879 fetch_fsmap learned FSMap version 78 2024-05-27T07:54:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.561+0000 7f17f5261740 1 -- 172.21.15.114:0/1293732599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55a239f7e5c0 con 0x55a239b521c0 2024-05-27T07:54:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.562+0000 7f17c77fe640 1 -- 172.21.15.114:0/1293732599 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17dc015070 con 0x55a239b521c0 2024-05-27T07:54:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.562+0000 7f17c77fe640 1 -- 172.21.15.114:0/1293732599 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 78) ==== 5305+0+0 (secure 0 0 0) 0x7f17d801c070 con 0x55a239b521c0 2024-05-27T07:54:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.563+0000 7f17f5261740 10 client.18879 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:54:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.563+0000 7f17f5261740 10 client.18879 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:07.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.563+0000 7f17f5261740 20 client.18879 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:07.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.563+0000 7f17f5261740 1 --2- 172.21.15.114:0/1293732599 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a239f7ec80 0x55a239f810d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:07.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.563+0000 7f17f5261740 4 client.18879 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:54:07.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.563+0000 7f17f5261740 1 -- 172.21.15.114:0/1293732599 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a239e475f0 con 0x55a239f7ec80 2024-05-27T07:54:07.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.564+0000 7f17e77fe640 1 --2- 172.21.15.114:0/1293732599 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a239f7ec80 0x55a239f810d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.571+0000 7f17e77fe640 1 --2- 172.21.15.114:0/1293732599 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a239f7ec80 0x55a239f810d0 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c54e28a78768d1cb server_cookie=3c2328f5e0207a18 in_seq=0 out_seq=0 2024-05-27T07:54:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:07.571+0000 7f17c77fe640 10 client.18879 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:07.590 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4342 20.94 MB/sec warmup 19 sec latency 1273.176 ms 2024-05-27T07:54:07.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:07 smithi114 ceph-mon[32038]: pgmap v484: 129 pgs: 11 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 93 active+clean; 211 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 178 op/s 2024-05-27T07:54:07.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:07 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:07.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:07 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:54:07.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:07 smithi163 ceph-mon[36451]: pgmap v484: 129 pgs: 11 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 93 active+clean; 211 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 178 op/s 2024-05-27T07:54:07.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:07 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:07.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:07 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:54:07.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:07 smithi186 ceph-mon[36647]: pgmap v484: 129 pgs: 11 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 93 active+clean; 211 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 178 op/s 2024-05-27T07:54:07.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:07 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:07.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:07 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:54:08.591 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4345 19.92 MB/sec warmup 20 sec latency 1638.977 ms 2024-05-27T07:54:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.927+0000 7f17c77fe640 1 -- 172.21.15.114:0/1293732599 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a239e475f0 con 0x55a239f7ec80 2024-05-27T07:54:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.927+0000 7f17c77fe640 10 client.18879 handle_command_reply: tid=0 2024-05-27T07:54:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.927+0000 7f17f5261740 1 client.18879 shutdown 2024-05-27T07:54:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.928+0000 7f17f5261740 1 -- 172.21.15.114:0/1293732599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b521c0 msgr2=0x55a239f73530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.928+0000 7f17f5261740 1 --2- 172.21.15.114:0/1293732599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b521c0 0x55a239f73530 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x55a239f68e30 tx=0x7f17d8005f90 comp rx=0 tx=0).stop 2024-05-27T07:54:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.928+0000 7f17f5261740 1 -- 172.21.15.114:0/1293732599 shutdown_connections 2024-05-27T07:54:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.928+0000 7f17f5261740 1 --2- 172.21.15.114:0/1293732599 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a239f7ec80 0x55a239f810d0 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.928+0000 7f17f5261740 1 --2- 172.21.15.114:0/1293732599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a239bbe1b0 0x55a239fed1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.928+0000 7f17f5261740 1 --2- 172.21.15.114:0/1293732599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a239b7cd80 0x55a239f73a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.928+0000 7f17f5261740 1 --2- 172.21.15.114:0/1293732599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a239b521c0 0x55a239f73530 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.928+0000 7f17f5261740 1 -- 172.21.15.114:0/1293732599 >> 172.21.15.114:0/1293732599 conn(0x55a239bb2300 msgr2=0x55a239f69890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.928+0000 7f17c77fe640 0 client.18879 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.928+0000 7f17f5261740 1 -- 172.21.15.114:0/1293732599 shutdown_connections 2024-05-27T07:54:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.928+0000 7f17f5261740 1 -- 172.21.15.114:0/1293732599 wait complete. 2024-05-27T07:54:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:54:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:54:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:54:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.928+0000 7f17f5261740 20 client.18879 trim_cache size 0 max 16384 2024-05-27T07:54:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.930+0000 7f17ee3b4640 1 -- 172.21.15.114:0/987584928 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f17d00c3ab0 msgr2=0x7f17d00c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.930+0000 7f17ee3b4640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f17d00c3ab0 0x7f17d00c5fc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f17d80044f0 tx=0x7f17d8014040 comp rx=0 tx=0).stop 2024-05-27T07:54:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.930+0000 7f17ee3b4640 1 -- 172.21.15.114:0/987584928 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17e81075c0 msgr2=0x7f17e819a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.930+0000 7f17ee3b4640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17e81075c0 0x7f17e819a680 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f17e0034040 tx=0x7f17e0003cc0 comp rx=0 tx=0).stop 2024-05-27T07:54:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.930+0000 7f17ee3b4640 1 -- 172.21.15.114:0/987584928 shutdown_connections 2024-05-27T07:54:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.930+0000 7f17ee3b4640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f17d00c3ab0 0x7f17d00c5fc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.930+0000 7f17ee3b4640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17e81075c0 0x7f17e819a680 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.930+0000 7f17ee3b4640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17e8106b00 0x7f17e81a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.930+0000 7f17ee3b4640 1 --2- 172.21.15.114:0/987584928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e8102570 0x7f17e81a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.931+0000 7f17ee3b4640 1 -- 172.21.15.114:0/987584928 >> 172.21.15.114:0/987584928 conn(0x7f17e80779d0 msgr2=0x7f17e80ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.932+0000 7f17ee3b4640 1 -- 172.21.15.114:0/987584928 shutdown_connections 2024-05-27T07:54:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:08.932+0000 7f17ee3b4640 1 -- 172.21.15.114:0/987584928 wait complete. 2024-05-27T07:54:09.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.399+0000 7f3d4e13e700 1 -- 172.21.15.114:0/2931127243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4811d920 msgr2=0x7f3d4811fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:09.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.399+0000 7f3d4e13e700 1 --2- 172.21.15.114:0/2931127243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4811d920 0x7f3d4811fd40 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000b0a0 tx=0x7f3d4000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.399+0000 7f3d4e13e700 1 -- 172.21.15.114:0/2931127243 shutdown_connections 2024-05-27T07:54:09.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.399+0000 7f3d4e13e700 1 --2- 172.21.15.114:0/2931127243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4811d920 0x7f3d4811fd40 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.399+0000 7f3d4e13e700 1 --2- 172.21.15.114:0/2931127243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d4810a920 0x7f3d4810ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.399+0000 7f3d4e13e700 1 --2- 172.21.15.114:0/2931127243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d4810b270 0x7f3d4811d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.399+0000 7f3d4e13e700 1 -- 172.21.15.114:0/2931127243 >> 172.21.15.114:0/2931127243 conn(0x7f3d4806f4c0 msgr2=0x7f3d4806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:09.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.399+0000 7f3d4e13e700 1 -- 172.21.15.114:0/2931127243 shutdown_connections 2024-05-27T07:54:09.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.399+0000 7f3d4e13e700 1 -- 172.21.15.114:0/2931127243 wait complete. 2024-05-27T07:54:09.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.400+0000 7f3d4e13e700 1 Processor -- start 2024-05-27T07:54:09.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.400+0000 7f3d4e13e700 1 -- start start 2024-05-27T07:54:09.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.401+0000 7f3d4e13e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4810a920 0x7f3d481a9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.401+0000 7f3d4e13e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d4810b270 0x7f3d481aa370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.401+0000 7f3d4e13e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d4811d920 0x7f3d481ae600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.404+0000 7f3d4e13e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3d481233e0 con 0x7f3d4810a920 2024-05-27T07:54:09.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.404+0000 7f3d4e13e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3d48123260 con 0x7f3d4810b270 2024-05-27T07:54:09.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.404+0000 7f3d4e13e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3d48123560 con 0x7f3d4811d920 2024-05-27T07:54:09.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.404+0000 7f3d477fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4810a920 0x7f3d481a9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:09.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.404+0000 7f3d477fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4810a920 0x7f3d481a9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42750/0 (socket says 172.21.15.114:42750) 2024-05-27T07:54:09.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.404+0000 7f3d477fe700 1 -- 172.21.15.114:0/3333790545 learned_addr learned my addr 172.21.15.114:0/3333790545 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:09.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.405+0000 7f3d477fe700 1 -- 172.21.15.114:0/3333790545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d4811d920 msgr2=0x7f3d481ae600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:54:09.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.405+0000 7f3d477fe700 1 --2- 172.21.15.114:0/3333790545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d4811d920 0x7f3d481ae600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.405+0000 7f3d477fe700 1 -- 172.21.15.114:0/3333790545 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d4810b270 msgr2=0x7f3d481aa370 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:54:09.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.405+0000 7f3d477fe700 1 --2- 172.21.15.114:0/3333790545 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d4810b270 0x7f3d481aa370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.405+0000 7f3d477fe700 1 -- 172.21.15.114:0/3333790545 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d481aeca0 con 0x7f3d4810a920 2024-05-27T07:54:09.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.405+0000 7f3d477fe700 1 --2- 172.21.15.114:0/3333790545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4810a920 0x7f3d481a9dd0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800b550 tx=0x7f3d3800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:09.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.407+0000 7f3d44ff9700 1 -- 172.21.15.114:0/3333790545 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38015020 con 0x7f3d4810a920 2024-05-27T07:54:09.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.407+0000 7f3d44ff9700 1 -- 172.21.15.114:0/3333790545 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d38011ce0 con 0x7f3d4810a920 2024-05-27T07:54:09.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.408+0000 7f3d44ff9700 1 -- 172.21.15.114:0/3333790545 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38010d40 con 0x7f3d4810a920 2024-05-27T07:54:09.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.415+0000 7f3d4e13e700 1 -- 172.21.15.114:0/3333790545 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d481aef90 con 0x7f3d4810a920 2024-05-27T07:54:09.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.415+0000 7f3d4e13e700 1 -- 172.21.15.114:0/3333790545 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d481b68b0 con 0x7f3d4810a920 2024-05-27T07:54:09.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.423+0000 7f3d5cb53b80 1 Processor -- start 2024-05-27T07:54:09.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.423+0000 7f3d5cb53b80 1 -- start start 2024-05-27T07:54:09.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.424+0000 7f3d5cb53b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d778d3e40 0x562d77a57710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.424+0000 7f3d5cb53b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d77a57c50 0x562d77a51a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.424+0000 7f3d5cb53b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a51f80 0x562d77a52390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.424+0000 7f3d5cb53b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562d77980c90 con 0x562d778d3e40 2024-05-27T07:54:09.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.424+0000 7f3d5cb53b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562d77970830 con 0x562d77a51f80 2024-05-27T07:54:09.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.424+0000 7f3d5cb53b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562d77974870 con 0x562d77a57c50 2024-05-27T07:54:09.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.424+0000 7f3d47fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d778d3e40 0x562d77a57710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:09.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.424+0000 7f3d47fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d778d3e40 0x562d77a57710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42754/0 (socket says 172.21.15.114:42754) 2024-05-27T07:54:09.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.424+0000 7f3d47fff700 1 -- 172.21.15.114:0/1575369575 learned_addr learned my addr 172.21.15.114:0/1575369575 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:09.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.424+0000 7f3d46ffd700 1 --2- 172.21.15.114:0/1575369575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a51f80 0x562d77a52390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:09.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.425+0000 7f3d47fff700 1 -- 172.21.15.114:0/1575369575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d77a57c50 msgr2=0x562d77a51a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:54:09.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.425+0000 7f3d47fff700 1 --2- 172.21.15.114:0/1575369575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d77a57c50 0x562d77a51a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.425+0000 7f3d47fff700 1 -- 172.21.15.114:0/1575369575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a51f80 msgr2=0x562d77a52390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:09.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.425+0000 7f3d47fff700 1 --2- 172.21.15.114:0/1575369575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a51f80 0x562d77a52390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.425+0000 7f3d47fff700 1 -- 172.21.15.114:0/1575369575 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d778c08d0 con 0x562d778d3e40 2024-05-27T07:54:09.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.426+0000 7f3d44ff9700 1 -- 172.21.15.114:0/3333790545 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f3d38018020 con 0x7f3d4810a920 2024-05-27T07:54:09.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.426+0000 7f3d47fff700 1 --2- 172.21.15.114:0/1575369575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d778d3e40 0x562d77a57710 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009bd0 tx=0x7f3d4000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3122a940f0ccfa7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:09.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.426+0000 7f3d2dffb700 1 -- 172.21.15.114:0/1575369575 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d400092d0 con 0x562d778d3e40 2024-05-27T07:54:09.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.426+0000 7f3d2dffb700 1 -- 172.21.15.114:0/1575369575 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d400075c0 con 0x562d778d3e40 2024-05-27T07:54:09.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.426+0000 7f3d2dffb700 1 -- 172.21.15.114:0/1575369575 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4003d6d0 con 0x562d778d3e40 2024-05-27T07:54:09.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.428+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/1575369575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d778d3e40 msgr2=0x562d77a57710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:09.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.428+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/1575369575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d778d3e40 0x562d77a57710 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009bd0 tx=0x7f3d4000cd80 comp rx=0 tx=0).stop 2024-05-27T07:54:09.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.428+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/1575369575 shutdown_connections 2024-05-27T07:54:09.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.429+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/1575369575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a51f80 0x562d77a52390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.429+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/1575369575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d778d3e40 0x562d77a57710 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.429+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/1575369575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d77a57c50 0x562d77a51a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.429+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/1575369575 >> 172.21.15.114:0/1575369575 conn(0x562d7792eaf0 msgr2=0x562d77a4c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:09.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.429+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/1575369575 shutdown_connections 2024-05-27T07:54:09.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.429+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/1575369575 wait complete. 2024-05-27T07:54:09.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.429+0000 7f3d5cb53b80 1 -- start start 2024-05-27T07:54:09.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.430+0000 7f3d44ff9700 1 --2- 172.21.15.114:0/3333790545 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3d300b2960 0x7f3d300b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.430+0000 7f3d5cb53b80 1 Processor -- start 2024-05-27T07:54:09.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.430+0000 7f3d47fff700 1 --2- 172.21.15.114:0/3333790545 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3d300b2960 0x7f3d300b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:09.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.431+0000 7f3d44ff9700 1 -- 172.21.15.114:0/3333790545 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9642+0+0 (secure 0 0 0) 0x7f3d380e18b0 con 0x7f3d4810a920 2024-05-27T07:54:09.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.431+0000 7f3d47fff700 1 --2- 172.21.15.114:0/3333790545 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3d300b2960 0x7f3d300b4e70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009ba0 tx=0x7f3d400099e0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.432+0000 7f3d5cb53b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a5b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.432+0000 7f3d5cb53b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a5b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.432+0000 7f3d5cb53b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 0x562d77a5bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.432+0000 7f3d5cb53b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562d77876fa0 con 0x562d77a51f80 2024-05-27T07:54:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.432+0000 7f3d5cb53b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562d779127c0 con 0x562d77a57c50 2024-05-27T07:54:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.432+0000 7f3d5cb53b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562d77a5e6e0 con 0x562d778d3e40 2024-05-27T07:54:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.432+0000 7f3d5cb53b80 10 client.0 _ll_register_callbacks cb 0x562d7790b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.433+0000 7f3d46ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a5b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.433+0000 7f3d46ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a5b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42770/0 (socket says 172.21.15.114:42770) 2024-05-27T07:54:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.433+0000 7f3d46ffd700 1 -- 172.21.15.114:0/3684336876 learned_addr learned my addr 172.21.15.114:0/3684336876 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.433+0000 7f3d47fff700 1 --2- 172.21.15.114:0/3684336876 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 0x562d77a5bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.433+0000 7f3d46ffd700 1 -- 172.21.15.114:0/3684336876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 msgr2=0x562d77a5b330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:54:09.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.433+0000 7f3d46ffd700 1 --2- 172.21.15.114:0/3684336876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a5b330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.433+0000 7f3d46ffd700 1 -- 172.21.15.114:0/3684336876 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 msgr2=0x562d77a5bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:09.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.433+0000 7f3d46ffd700 1 --2- 172.21.15.114:0/3684336876 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 0x562d77a5bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.433+0000 7f3d46ffd700 1 -- 172.21.15.114:0/3684336876 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d7790f690 con 0x562d77a51f80 2024-05-27T07:54:09.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.434+0000 7f3d46ffd700 1 --2- 172.21.15.114:0/3684336876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a5b870 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00e480 tx=0x7f3d3c00e840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84d41634ecea218 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:09.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.434+0000 7f3d26ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:54:09.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.434+0000 7f3d26ffd700 1 -- 172.21.15.114:0/3684336876 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c017070 con 0x562d77a51f80 2024-05-27T07:54:09.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.434+0000 7f3d26ffd700 1 -- 172.21.15.114:0/3684336876 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9642+0+0 (secure 0 0 0) 0x7f3d3c013680 con 0x562d77a51f80 2024-05-27T07:54:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.435+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/3684336876 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d400075c0 con 0x562d77a51f80 2024-05-27T07:54:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.435+0000 7f3d26ffd700 1 -- 172.21.15.114:0/3684336876 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f3d3c021050 con 0x562d77a51f80 2024-05-27T07:54:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.435+0000 7f3d5cb53b80 10 client.18891 fetch_fsmap learned FSMap version 78 2024-05-27T07:54:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.435+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/3684336876 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x562d77a691f0 con 0x562d77a51f80 2024-05-27T07:54:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.436+0000 7f3d26ffd700 1 -- 172.21.15.114:0/3684336876 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d400075c0 con 0x562d77a51f80 2024-05-27T07:54:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.436+0000 7f3d26ffd700 1 -- 172.21.15.114:0/3684336876 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 78) ==== 5305+0+0 (secure 0 0 0) 0x7f3d3c01c8e0 con 0x562d77a51f80 2024-05-27T07:54:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.436+0000 7f3d5cb53b80 10 client.18891 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:54:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.436+0000 7f3d5cb53b80 10 client.18891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.436+0000 7f3d5cb53b80 20 client.18891 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.436+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/3684336876 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562d77a6bc30 0x562d77a6e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.436+0000 7f3d5cb53b80 4 client.18891 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:54:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.436+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/3684336876 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d77a1d9f0 con 0x562d77a6bc30 2024-05-27T07:54:09.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.438+0000 7f3d477fe700 1 --2- 172.21.15.114:0/3684336876 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562d77a6bc30 0x562d77a6e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:09.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.439+0000 7f3d26ffd700 10 client.18891 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:09.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.439+0000 7f3d477fe700 1 --2- 172.21.15.114:0/3684336876 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562d77a6bc30 0x562d77a6e100 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87c1b833df43f02e server_cookie=2f555790da7b36c4 in_seq=0 out_seq=0 2024-05-27T07:54:09.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.440+0000 7f3d26ffd700 1 -- 172.21.15.114:0/3684336876 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f3d380a82f0 con 0x562d77a6bc30 2024-05-27T07:54:09.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.440+0000 7f3d26ffd700 10 client.18891 handle_command_reply: tid=0 2024-05-27T07:54:09.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.440+0000 7f3d5cb53b80 1 client.18891 shutdown 2024-05-27T07:54:09.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.441+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/3684336876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 msgr2=0x562d77a5b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:09.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.441+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/3684336876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a5b870 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00e480 tx=0x7f3d3c00e840 comp rx=0 tx=0).stop 2024-05-27T07:54:09.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.441+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/3684336876 shutdown_connections 2024-05-27T07:54:09.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.441+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/3684336876 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562d77a6bc30 0x562d77a6e100 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.441+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/3684336876 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 0x562d77a5bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.441+0000 7f3d26ffd700 0 client.18891 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:09.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.441+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/3684336876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a5b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.441+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/3684336876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a5b870 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.441+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/3684336876 >> 172.21.15.114:0/3684336876 conn(0x562d7792eaf0 msgr2=0x562d77a50a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:09.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.442+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/3684336876 shutdown_connections 2024-05-27T07:54:09.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.442+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/3684336876 wait complete. 2024-05-27T07:54:09.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.442+0000 7f3d5cb53b80 20 client.18891 trim_cache size 0 max 16384 2024-05-27T07:54:09.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.479+0000 7f3d5cb53b80 1 Processor -- start 2024-05-27T07:54:09.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.480+0000 7f3d5cb53b80 1 -- start start 2024-05-27T07:54:09.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.480+0000 7f3d5cb53b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a51a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.480+0000 7f3d5cb53b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a5bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.480+0000 7f3d5cb53b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 0x562d77a55720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.480+0000 7f3d5cb53b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4807eef0 con 0x562d77a51f80 2024-05-27T07:54:09.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.480+0000 7f3d5cb53b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4807f5c0 con 0x562d77a57c50 2024-05-27T07:54:09.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.480+0000 7f3d5cb53b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4807ddb0 con 0x562d778d3e40 2024-05-27T07:54:09.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.482+0000 7f3d477fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a51a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:09.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.482+0000 7f3d477fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a51a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40772/0 (socket says 172.21.15.114:40772) 2024-05-27T07:54:09.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.482+0000 7f3d477fe700 1 -- 172.21.15.114:0/520981454 learned_addr learned my addr 172.21.15.114:0/520981454 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:09.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.482+0000 7f3d46ffd700 1 --2- 172.21.15.114:0/520981454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a5bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:09.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.483+0000 7f3d477fe700 1 -- 172.21.15.114:0/520981454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 msgr2=0x562d77a55720 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:54:09.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.483+0000 7f3d477fe700 1 --2- 172.21.15.114:0/520981454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 0x562d77a55720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.483+0000 7f3d477fe700 1 -- 172.21.15.114:0/520981454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 msgr2=0x562d77a5bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:09.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.483+0000 7f3d477fe700 1 --2- 172.21.15.114:0/520981454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a5bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.483+0000 7f3d477fe700 1 -- 172.21.15.114:0/520981454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d77a4e0a0 con 0x562d778d3e40 2024-05-27T07:54:09.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.484+0000 7f3d477fe700 1 --2- 172.21.15.114:0/520981454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a51a40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3d380a4860 tx=0x7f3d380b1040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=793a665f6d463ab3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:09.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.485+0000 7f3d2d7fa700 1 -- 172.21.15.114:0/520981454 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d380a87e0 con 0x562d778d3e40 2024-05-27T07:54:09.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.485+0000 7f3d2d7fa700 1 -- 172.21.15.114:0/520981454 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d380c2070 con 0x562d778d3e40 2024-05-27T07:54:09.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.485+0000 7f3d2d7fa700 1 -- 172.21.15.114:0/520981454 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d380a8a40 con 0x562d778d3e40 2024-05-27T07:54:09.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.488+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/520981454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 msgr2=0x562d77a51a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:09.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.488+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/520981454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a51a40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3d380a4860 tx=0x7f3d380b1040 comp rx=0 tx=0).stop 2024-05-27T07:54:09.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.490+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/520981454 shutdown_connections 2024-05-27T07:54:09.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.490+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/520981454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 0x562d77a55720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.490+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/520981454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a51a40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.490+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/520981454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a5bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.490+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/520981454 >> 172.21.15.114:0/520981454 conn(0x562d7792eaf0 msgr2=0x562d77a55010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:09.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.494+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/520981454 shutdown_connections 2024-05-27T07:54:09.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.495+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/520981454 wait complete. 2024-05-27T07:54:09.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.495+0000 7f3d5cb53b80 1 -- start start 2024-05-27T07:54:09.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.495+0000 7f3d5cb53b80 1 Processor -- start 2024-05-27T07:54:09.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.496+0000 7f3d5cb53b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a639d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.496+0000 7f3d5cb53b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a63f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.496+0000 7f3d5cb53b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 0x562d77a609a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.496+0000 7f3d5cb53b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4807d6e0 con 0x562d77a51f80 2024-05-27T07:54:09.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.496+0000 7f3d5cb53b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4807c2f0 con 0x562d77a57c50 2024-05-27T07:54:09.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.496+0000 7f3d5cb53b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4807c940 con 0x562d778d3e40 2024-05-27T07:54:09.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.496+0000 7f3d5cb53b80 10 client.0 _ll_register_callbacks cb 0x562d7790b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:09.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.496+0000 7f3d46ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a63f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:09.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.496+0000 7f3d46ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a63f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42798/0 (socket says 172.21.15.114:42798) 2024-05-27T07:54:09.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.496+0000 7f3d46ffd700 1 -- 172.21.15.114:0/1399518457 learned_addr learned my addr 172.21.15.114:0/1399518457 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:09.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.497+0000 7f3d47fff700 1 --2- 172.21.15.114:0/1399518457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 0x562d77a609a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:09.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.497+0000 7f3d477fe700 1 --2- 172.21.15.114:0/1399518457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a639d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:09.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.499+0000 7f3d477fe700 1 -- 172.21.15.114:0/1399518457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 msgr2=0x562d77a609a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:09.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.499+0000 7f3d477fe700 1 --2- 172.21.15.114:0/1399518457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 0x562d77a609a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.499+0000 7f3d477fe700 1 -- 172.21.15.114:0/1399518457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 msgr2=0x562d77a63f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:09.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.499+0000 7f3d477fe700 1 --2- 172.21.15.114:0/1399518457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a63f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.499+0000 7f3d477fe700 1 -- 172.21.15.114:0/1399518457 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d77a64500 con 0x562d778d3e40 2024-05-27T07:54:09.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.500+0000 7f3d477fe700 1 --2- 172.21.15.114:0/1399518457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a639d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x562d77912910 tx=0x7f3d380debf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9a6b4e058c2aa8d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:09.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.500+0000 7f3d26ffd700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:54:09.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.501+0000 7f3d26ffd700 1 -- 172.21.15.114:0/1399518457 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d380b5070 con 0x562d778d3e40 2024-05-27T07:54:09.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.501+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/1399518457 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d380c2070 con 0x562d778d3e40 2024-05-27T07:54:09.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.501+0000 7f3d26ffd700 1 -- 172.21.15.114:0/1399518457 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9642+0+0 (secure 0 0 0) 0x7f3d380a8a40 con 0x562d778d3e40 2024-05-27T07:54:09.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.502+0000 7f3d26ffd700 1 -- 172.21.15.114:0/1399518457 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f3d380b1640 con 0x562d778d3e40 2024-05-27T07:54:09.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.502+0000 7f3d5cb53b80 10 client.28402 fetch_fsmap learned FSMap version 78 2024-05-27T07:54:09.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.502+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/1399518457 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x562d77ad6ca0 con 0x562d778d3e40 2024-05-27T07:54:09.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.502+0000 7f3d26ffd700 1 -- 172.21.15.114:0/1399518457 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d380c2070 con 0x562d778d3e40 2024-05-27T07:54:09.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.503+0000 7f3d26ffd700 1 -- 172.21.15.114:0/1399518457 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 78) ==== 5305+0+0 (secure 0 0 0) 0x7f3d380a7a40 con 0x562d778d3e40 2024-05-27T07:54:09.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.503+0000 7f3d5cb53b80 10 client.28402 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:54:09.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.503+0000 7f3d5cb53b80 10 client.28402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:09.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.503+0000 7f3d5cb53b80 20 client.28402 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:09.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.503+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/1399518457 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562d77a6bc30 0x562d77ad9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:09.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.503+0000 7f3d5cb53b80 4 client.28402 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:54:09.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.503+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/1399518457 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562d77982be0 con 0x562d77a6bc30 2024-05-27T07:54:09.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.505+0000 7f3d46ffd700 1 --2- 172.21.15.114:0/1399518457 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562d77a6bc30 0x562d77ad9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:09.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.519+0000 7f3d46ffd700 1 --2- 172.21.15.114:0/1399518457 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562d77a6bc30 0x562d77ad9440 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8570ccb6cb5ee4ac server_cookie=6f198f2e31a9a780 in_seq=0 out_seq=0 2024-05-27T07:54:09.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.519+0000 7f3d26ffd700 10 client.28402 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:09.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.523+0000 7f3d26ffd700 1 -- 172.21.15.114:0/1399518457 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562d77982be0 con 0x562d77a6bc30 2024-05-27T07:54:09.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.523+0000 7f3d26ffd700 10 client.28402 handle_command_reply: tid=0 2024-05-27T07:54:09.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.523+0000 7f3d5cb53b80 1 client.28402 shutdown 2024-05-27T07:54:09.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.526+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/1399518457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 msgr2=0x562d77a639d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:09.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.526+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/1399518457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a639d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x562d77912910 tx=0x7f3d380debf0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.527+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/1399518457 shutdown_connections 2024-05-27T07:54:09.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.527+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/1399518457 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562d77a6bc30 0x562d77ad9440 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.527+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/1399518457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562d77a57c50 0x562d77a609a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.527+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/1399518457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d778d3e40 0x562d77a639d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.527+0000 7f3d5cb53b80 1 --2- 172.21.15.114:0/1399518457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562d77a51f80 0x562d77a63f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.527+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/1399518457 >> 172.21.15.114:0/1399518457 conn(0x562d7792eaf0 msgr2=0x562d77a5c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:09.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.527+0000 7f3d26ffd700 0 client.28402 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:09.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.531+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/1399518457 shutdown_connections 2024-05-27T07:54:09.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.531+0000 7f3d5cb53b80 1 -- 172.21.15.114:0/1399518457 wait complete. 2024-05-27T07:54:09.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.531+0000 7f3d5cb53b80 20 client.28402 trim_cache size 0 max 16384 2024-05-27T07:54:09.532 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:54:09.532 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:54:09.532 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:54:09.532 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:54:09.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.534+0000 7f3d4e13e700 1 -- 172.21.15.114:0/3333790545 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3d300b2960 msgr2=0x7f3d300b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:09.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.534+0000 7f3d4e13e700 1 --2- 172.21.15.114:0/3333790545 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3d300b2960 0x7f3d300b4e70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009ba0 tx=0x7f3d400099e0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.534+0000 7f3d4e13e700 1 -- 172.21.15.114:0/3333790545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4810a920 msgr2=0x7f3d481a9dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:09.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.534+0000 7f3d4e13e700 1 --2- 172.21.15.114:0/3333790545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4810a920 0x7f3d481a9dd0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800b550 tx=0x7f3d3800b910 comp rx=0 tx=0).stop 2024-05-27T07:54:09.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.534+0000 7f3d4e13e700 1 -- 172.21.15.114:0/3333790545 shutdown_connections 2024-05-27T07:54:09.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.534+0000 7f3d4e13e700 1 --2- 172.21.15.114:0/3333790545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d4811d920 0x7f3d481ae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.534+0000 7f3d4e13e700 1 --2- 172.21.15.114:0/3333790545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4810a920 0x7f3d481a9dd0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.534+0000 7f3d4e13e700 1 --2- 172.21.15.114:0/3333790545 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3d300b2960 0x7f3d300b4e70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.534+0000 7f3d4e13e700 1 --2- 172.21.15.114:0/3333790545 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d4810b270 0x7f3d481aa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:09.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.534+0000 7f3d4e13e700 1 -- 172.21.15.114:0/3333790545 >> 172.21.15.114:0/3333790545 conn(0x7f3d4806f4c0 msgr2=0x7f3d4811b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:09.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.538+0000 7f3d4e13e700 1 -- 172.21.15.114:0/3333790545 shutdown_connections 2024-05-27T07:54:09.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:09.538+0000 7f3d4e13e700 1 -- 172.21.15.114:0/3333790545 wait complete. 2024-05-27T07:54:09.591 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4349 19.02 MB/sec warmup 21 sec latency 2057.216 ms 2024-05-27T07:54:09.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:09 smithi114 ceph-mon[32038]: pgmap v485: 129 pgs: 19 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 60 active+clean; 214 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 273 op/s 2024-05-27T07:54:09.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:09 smithi163 ceph-mon[36451]: pgmap v485: 129 pgs: 19 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 60 active+clean; 214 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 273 op/s 2024-05-27T07:54:09.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:09 smithi186 ceph-mon[36647]: pgmap v485: 129 pgs: 19 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 60 active+clean; 214 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 273 op/s 2024-05-27T07:54:10.591 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4354 18.22 MB/sec warmup 22 sec latency 2007.302 ms 2024-05-27T07:54:10.630 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:54:10.630 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:54:11.120 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:54:11.591 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4374 17.49 MB/sec warmup 23 sec latency 1207.294 ms 2024-05-27T07:54:11.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:11 smithi186 ceph-mon[36647]: pgmap v486: 129 pgs: 19 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 60 active+clean; 214 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 189 op/s 2024-05-27T07:54:11.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:11 smithi163 ceph-mon[36451]: pgmap v486: 129 pgs: 19 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 60 active+clean; 214 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 189 op/s 2024-05-27T07:54:12.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:11 smithi114 ceph-mon[32038]: pgmap v486: 129 pgs: 19 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 60 active+clean; 214 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 189 op/s 2024-05-27T07:54:12.591 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4445 16.93 MB/sec warmup 24 sec latency 1557.088 ms 2024-05-27T07:54:13.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.072+0000 7f752f2ea700 1 -- 172.21.15.114:0/4130687937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528107c00 msgr2=0x7f7528108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:13.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.072+0000 7f752f2ea700 1 --2- 172.21.15.114:0/4130687937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528107c00 0x7f7528108010 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f752401a9d0 tx=0x7f752401ace0 comp rx=0 tx=0).stop 2024-05-27T07:54:13.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.076+0000 7f752f2ea700 1 -- 172.21.15.114:0/4130687937 shutdown_connections 2024-05-27T07:54:13.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.076+0000 7f752f2ea700 1 --2- 172.21.15.114:0/4130687937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7528108550 0x7f752810feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:13.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.076+0000 7f752f2ea700 1 --2- 172.21.15.114:0/4130687937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75281032d0 0x7f75281036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:13.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.076+0000 7f752f2ea700 1 --2- 172.21.15.114:0/4130687937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528107c00 0x7f7528108010 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:13.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.076+0000 7f752f2ea700 1 -- 172.21.15.114:0/4130687937 >> 172.21.15.114:0/4130687937 conn(0x7f7528078ab0 msgr2=0x7f7528078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:13.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.080+0000 7f752f2ea700 1 -- 172.21.15.114:0/4130687937 shutdown_connections 2024-05-27T07:54:13.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.081+0000 7f752f2ea700 1 -- 172.21.15.114:0/4130687937 wait complete. 2024-05-27T07:54:13.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.082+0000 7f752f2ea700 1 Processor -- start 2024-05-27T07:54:13.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.083+0000 7f752f2ea700 1 -- start start 2024-05-27T07:54:13.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.084+0000 7f752f2ea700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75281032d0 0x7f7528074e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:13.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.084+0000 7f752f2ea700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528107c00 0x7f75280753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:13.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.084+0000 7f752e2e8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75281032d0 0x7f7528074e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:13.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.084+0000 7f752e2e8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75281032d0 0x7f7528074e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:44352/0 (socket says 172.21.15.114:44352) 2024-05-27T07:54:13.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.084+0000 7f752e2e8700 1 -- 172.21.15.114:0/504196648 learned_addr learned my addr 172.21.15.114:0/504196648 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:13.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.084+0000 7f752f2ea700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7528108550 0x7f7528070020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:13.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.084+0000 7f752f2ea700 1 -- 172.21.15.114:0/504196648 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f752806e3d0 con 0x7f7528107c00 2024-05-27T07:54:13.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.084+0000 7f752dae7700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528107c00 0x7f75280753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:13.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.085+0000 7f752f2ea700 1 -- 172.21.15.114:0/504196648 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f752806e250 con 0x7f75281032d0 2024-05-27T07:54:13.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.085+0000 7f752f2ea700 1 -- 172.21.15.114:0/504196648 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f752806e550 con 0x7f7528108550 2024-05-27T07:54:13.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.086+0000 7f752dae7700 1 -- 172.21.15.114:0/504196648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7528108550 msgr2=0x7f7528070020 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:54:13.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.086+0000 7f752dae7700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7528108550 0x7f7528070020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:13.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.086+0000 7f752dae7700 1 -- 172.21.15.114:0/504196648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75281032d0 msgr2=0x7f7528074e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:13.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.086+0000 7f752dae7700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75281032d0 0x7f7528074e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:13.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.086+0000 7f752dae7700 1 -- 172.21.15.114:0/504196648 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7528070850 con 0x7f7528107c00 2024-05-27T07:54:13.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.089+0000 7f752dae7700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528107c00 0x7f75280753b0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f7524006010 tx=0x7f7524005530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:13.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.091+0000 7f751f7fe700 1 -- 172.21.15.114:0/504196648 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75240036f0 con 0x7f7528107c00 2024-05-27T07:54:13.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.091+0000 7f751f7fe700 1 -- 172.21.15.114:0/504196648 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f752401cbc0 con 0x7f7528107c00 2024-05-27T07:54:13.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.091+0000 7f751f7fe700 1 -- 172.21.15.114:0/504196648 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752402fda0 con 0x7f7528107c00 2024-05-27T07:54:13.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.091+0000 7f752f2ea700 1 -- 172.21.15.114:0/504196648 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7528070ae0 con 0x7f7528107c00 2024-05-27T07:54:13.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.092+0000 7f752f2ea700 1 -- 172.21.15.114:0/504196648 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75281add80 con 0x7f7528107c00 2024-05-27T07:54:13.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.096+0000 7f752f2ea700 1 -- 172.21.15.114:0/504196648 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f750c005330 con 0x7f7528107c00 2024-05-27T07:54:13.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.099+0000 7f751f7fe700 1 -- 172.21.15.114:0/504196648 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f7524036020 con 0x7f7528107c00 2024-05-27T07:54:13.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.100+0000 7f751f7fe700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75140b2bf0 0x7f75140b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:13.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.101+0000 7f751f7fe700 1 -- 172.21.15.114:0/504196648 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9642+0+0 (secure 0 0 0) 0x7f75240f46c0 con 0x7f7528107c00 2024-05-27T07:54:13.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.104+0000 7f752e2e8700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75140b2bf0 0x7f75140b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:13.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.105+0000 7f752e2e8700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75140b2bf0 0x7f75140b5100 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f75180099d0 tx=0x7f7518008040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:13.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.111+0000 7f751f7fe700 1 -- 172.21.15.114:0/504196648 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f75240bbd80 con 0x7f7528107c00 2024-05-27T07:54:13.591 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4577 16.58 MB/sec warmup 25 sec latency 1118.245 ms 2024-05-27T07:54:13.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.601+0000 7f752f2ea700 1 -- 172.21.15.114:0/504196648 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f750c0058f0 con 0x7f7528107c00 2024-05-27T07:54:13.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.605+0000 7f751f7fe700 1 -- 172.21.15.114:0/504196648 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) ==== 94+0+7690 (secure 0 0 0) 0x7f75240bb680 con 0x7f7528107c00 2024-05-27T07:54:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:54:13.606 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":78,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:54:01.512457+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_28229":{"gid":28229,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/927041449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":927041449},{"type":"v1","addr":"172.21.15.163:6835","nonce":927041449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:54:13.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.610+0000 7f751d7fa700 1 -- 172.21.15.114:0/504196648 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75140b2bf0 msgr2=0x7f75140b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:13.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.610+0000 7f751d7fa700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75140b2bf0 0x7f75140b5100 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f75180099d0 tx=0x7f7518008040 comp rx=0 tx=0).stop 2024-05-27T07:54:13.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.610+0000 7f751d7fa700 1 -- 172.21.15.114:0/504196648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528107c00 msgr2=0x7f75280753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:13.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.610+0000 7f751d7fa700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528107c00 0x7f75280753b0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f7524006010 tx=0x7f7524005530 comp rx=0 tx=0).stop 2024-05-27T07:54:13.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.611+0000 7f751d7fa700 1 -- 172.21.15.114:0/504196648 shutdown_connections 2024-05-27T07:54:13.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.611+0000 7f751d7fa700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7528108550 0x7f7528070020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:13.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.611+0000 7f751d7fa700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75140b2bf0 0x7f75140b5100 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:13.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.611+0000 7f751d7fa700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75281032d0 0x7f7528074e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:13.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.611+0000 7f751d7fa700 1 --2- 172.21.15.114:0/504196648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528107c00 0x7f75280753b0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:13.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.611+0000 7f751d7fa700 1 -- 172.21.15.114:0/504196648 >> 172.21.15.114:0/504196648 conn(0x7f7528078ab0 msgr2=0x7f7528100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:13.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.612+0000 7f751d7fa700 1 -- 172.21.15.114:0/504196648 shutdown_connections 2024-05-27T07:54:13.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:13.612+0000 7f751d7fa700 1 -- 172.21.15.114:0/504196648 wait complete. 2024-05-27T07:54:13.614 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 78 2024-05-27T07:54:13.738 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:13 smithi163 ceph-mon[36451]: pgmap v487: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 192 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.5 MiB/s wr, 470 op/s 2024-05-27T07:54:13.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:13 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:54:13.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:13 smithi163 systemd[1]: Stopped Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:54:13.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:13 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Consumed 1.422s CPU time. 2024-05-27T07:54:13.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:13 smithi163 systemd[1]: Starting Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:54:13.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:13 smithi186 ceph-mon[36647]: pgmap v487: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 192 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.5 MiB/s wr, 470 op/s 2024-05-27T07:54:13.983 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:13 smithi114 ceph-mon[32038]: pgmap v487: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 192 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.5 MiB/s wr, 470 op/s 2024-05-27T07:54:14.020 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:13 smithi163 podman[73899]: 2024-05-27 07:54:13.639522272 +0000 UTC m=+0.023096056 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:54:14.020 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:13 smithi163 podman[73899]: 2024-05-27 07:54:13.775776224 +0000 UTC m=+0.159349982 container create 9cedcca6917062d5106074c74f5f3f599d85c264b6cd30e3abfad557e52608ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:54:14.020 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 podman[73899]: 2024-05-27 07:54:14.020327036 +0000 UTC m=+0.403900806 container init 9cedcca6917062d5106074c74f5f3f599d85c264b6cd30e3abfad557e52608ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-27T07:54:14.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:54:14.454 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 podman[73899]: 2024-05-27 07:54:14.024648456 +0000 UTC m=+0.408222223 container start 9cedcca6917062d5106074c74f5f3f599d85c264b6cd30e3abfad557e52608ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T07:54:14.454 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: starting mds.e at 2024-05-27T07:54:14.454 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 bash[73899]: 9cedcca6917062d5106074c74f5f3f599d85c264b6cd30e3abfad557e52608ed 2024-05-27T07:54:14.454 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 systemd[1]: Started Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:54:14.454 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f5294e700 time 2024-05-27T07:54:14.454325+0000 2024-05-27T07:54:14.455 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:14.591 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4914 16.59 MB/sec warmup 26 sec latency 1198.827 ms 2024-05-27T07:54:14.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:14 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/504196648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:54:14.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:14 smithi163 ceph-mon[36451]: daemon mds.e restarted 2024-05-27T07:54:14.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:14 smithi163 ceph-mon[36451]: osdmap e136: 12 total, 12 up, 12 in 2024-05-27T07:54:14.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:14 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6834/4190527329,v1:172.21.15.163:6835/4190527329] up:boot 2024-05-27T07:54:14.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:14 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:54:14.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:14 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:54:14.727 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:14.727 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f65193ba6] 2024-05-27T07:54:14.727 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f65193dd1] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6fb54023d] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c6fb52b971] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c6fb529fa0] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6fb52a651] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6fb4bf347] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6fb1bb87e] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6fb1c0f2f] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c6fb1cb969] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c6fb56448c] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c6fb56d9cb] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f6516272d] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f65162749] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f63d061ca] 2024-05-27T07:54:14.728 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 16: clone() 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.455+0000 7f6f5294e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f5294e700 time 2024-05-27T07:54:14.454325+0000 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f65193ba6] 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f65193dd1] 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6fb54023d] 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c6fb52b971] 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c6fb529fa0] 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6fb52a651] 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6fb4bf347] 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6fb1bb87e] 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6fb1c0f2f] 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c6fb1cb969] 2024-05-27T07:54:14.729 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c6fb56448c] 2024-05-27T07:54:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c6fb56d9cb] 2024-05-27T07:54:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f6516272d] 2024-05-27T07:54:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f65162749] 2024-05-27T07:54:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f63d061ca] 2024-05-27T07:54:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 16: clone() 2024-05-27T07:54:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: *** Caught signal (Aborted) ** 2024-05-27T07:54:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: in thread 7f6f5294e700 thread_name:md_log_replay 2024-05-27T07:54:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f63d10d20] 2024-05-27T07:54:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2: gsignal() 2024-05-27T07:54:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 3: abort() 2024-05-27T07:54:14.730 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f65193ced] 2024-05-27T07:54:14.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f65193dd1] 2024-05-27T07:54:14.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6fb54023d] 2024-05-27T07:54:14.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c6fb52b971] 2024-05-27T07:54:14.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c6fb529fa0] 2024-05-27T07:54:14.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6fb52a651] 2024-05-27T07:54:14.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6fb4bf347] 2024-05-27T07:54:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6fb1bb87e] 2024-05-27T07:54:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6fb1c0f2f] 2024-05-27T07:54:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c6fb1cb969] 2024-05-27T07:54:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c6fb56448c] 2024-05-27T07:54:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c6fb56d9cb] 2024-05-27T07:54:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f6516272d] 2024-05-27T07:54:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f65162749] 2024-05-27T07:54:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f63d061ca] 2024-05-27T07:54:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 19: clone() 2024-05-27T07:54:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.458+0000 7f6f5294e700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:54:14.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: in thread 7f6f5294e700 thread_name:md_log_replay 2024-05-27T07:54:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f63d10d20] 2024-05-27T07:54:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2: gsignal() 2024-05-27T07:54:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 3: abort() 2024-05-27T07:54:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f65193ced] 2024-05-27T07:54:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f65193dd1] 2024-05-27T07:54:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6fb54023d] 2024-05-27T07:54:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c6fb52b971] 2024-05-27T07:54:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c6fb529fa0] 2024-05-27T07:54:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6fb52a651] 2024-05-27T07:54:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6fb4bf347] 2024-05-27T07:54:14.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6fb1bb87e] 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6fb1c0f2f] 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c6fb1cb969] 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c6fb56448c] 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c6fb56d9cb] 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f6516272d] 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f65162749] 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f63d061ca] 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 19: clone() 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: -1> 2024-05-27T07:54:14.455+0000 7f6f5294e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f5294e700 time 2024-05-27T07:54:14.454325+0000 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.734 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f65193ba6] 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f65193dd1] 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6fb54023d] 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c6fb52b971] 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c6fb529fa0] 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6fb52a651] 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6fb4bf347] 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6fb1bb87e] 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6fb1c0f2f] 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c6fb1cb969] 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c6fb56448c] 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c6fb56d9cb] 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f6516272d] 2024-05-27T07:54:14.735 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f65162749] 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f63d061ca] 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 16: clone() 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 0> 2024-05-27T07:54:14.458+0000 7f6f5294e700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: in thread 7f6f5294e700 thread_name:md_log_replay 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f63d10d20] 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2: gsignal() 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 3: abort() 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f65193ced] 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f65193dd1] 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6fb54023d] 2024-05-27T07:54:14.736 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c6fb52b971] 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c6fb529fa0] 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6fb52a651] 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6fb4bf347] 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6fb1bb87e] 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6fb1c0f2f] 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c6fb1cb969] 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c6fb56448c] 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c6fb56d9cb] 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f6516272d] 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f65162749] 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f63d061ca] 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 19: clone() 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.737 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: -1> 2024-05-27T07:54:14.455+0000 7f6f5294e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f5294e700 time 2024-05-27T07:54:14.454325+0000 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f65193ba6] 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f65193dd1] 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6fb54023d] 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c6fb52b971] 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c6fb529fa0] 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6fb52a651] 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6fb4bf347] 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6fb1bb87e] 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6fb1c0f2f] 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c6fb1cb969] 2024-05-27T07:54:14.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c6fb56448c] 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c6fb56d9cb] 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f6516272d] 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f65162749] 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f63d061ca] 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 16: clone() 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 0> 2024-05-27T07:54:14.458+0000 7f6f5294e700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: in thread 7f6f5294e700 thread_name:md_log_replay 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f63d10d20] 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2: gsignal() 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 3: abort() 2024-05-27T07:54:14.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f65193ced] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f65193dd1] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6fb54023d] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c6fb52b971] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c6fb529fa0] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6fb52a651] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6fb4bf347] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6fb1bb87e] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6fb1c0f2f] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c6fb1cb969] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c6fb56448c] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c6fb56d9cb] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f6516272d] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f65162749] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f63d061ca] 2024-05-27T07:54:14.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 19: clone() 2024-05-27T07:54:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:54:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[73912]: 2024-05-27T07:54:14.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:14 smithi163 podman[73947]: 2024-05-27 07:54:14.528483462 +0000 UTC m=+0.027681524 container died 9cedcca6917062d5106074c74f5f3f599d85c264b6cd30e3abfad557e52608ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:54:14.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:14 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/504196648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:54:14.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:14 smithi114 ceph-mon[32038]: daemon mds.e restarted 2024-05-27T07:54:14.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:14 smithi114 ceph-mon[32038]: osdmap e136: 12 total, 12 up, 12 in 2024-05-27T07:54:14.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:14 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6834/4190527329,v1:172.21.15.163:6835/4190527329] up:boot 2024-05-27T07:54:14.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:14 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:54:14.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:14 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:54:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:54:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:14 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/504196648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:54:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:14 smithi186 ceph-mon[36647]: daemon mds.e restarted 2024-05-27T07:54:14.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:14 smithi186 ceph-mon[36647]: osdmap e136: 12 total, 12 up, 12 in 2024-05-27T07:54:14.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:14 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6834/4190527329,v1:172.21.15.163:6835/4190527329] up:boot 2024-05-27T07:54:14.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:14 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:54:14.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:14 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:54:15.579 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:15 smithi163 podman[73947]: 2024-05-27 07:54:15.278328175 +0000 UTC m=+0.777526217 container remove 9cedcca6917062d5106074c74f5f3f599d85c264b6cd30e3abfad557e52608ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:54:15.579 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:15 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:54:15.591 INFO:tasks.workunit.client.0.smithi114.stdout: 10 5230 16.62 MB/sec warmup 27 sec latency 666.722 ms 2024-05-27T07:54:15.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:15 smithi114 ceph-mon[32038]: pgmap v488: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 192 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.3 MiB/s wr, 404 op/s 2024-05-27T07:54:15.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:15 smithi163 ceph-mon[36451]: pgmap v488: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 192 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.3 MiB/s wr, 404 op/s 2024-05-27T07:54:15.977 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:15 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-05-27T07:54:15.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:15 smithi186 ceph-mon[36647]: pgmap v488: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 192 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.3 MiB/s wr, 404 op/s 2024-05-27T07:54:16.591 INFO:tasks.workunit.client.0.smithi114.stdout: 10 5682 16.99 MB/sec warmup 28 sec latency 286.161 ms 2024-05-27T07:54:17.591 INFO:tasks.workunit.client.0.smithi114.stdout: 10 6067 17.07 MB/sec warmup 29 sec latency 200.129 ms 2024-05-27T07:54:17.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:17 smithi114 ceph-mon[32038]: pgmap v490: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 201 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 2.4 MiB/s wr, 566 op/s 2024-05-27T07:54:17.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.748+0000 7f720733b700 1 -- 172.21.15.114:0/2000488585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7200109860 msgr2=0x7f72001111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:17.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.748+0000 7f720733b700 1 --2- 172.21.15.114:0/2000488585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7200109860 0x7f72001111c0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f71f800b0d0 tx=0x7f71f800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.748+0000 7f720733b700 1 -- 172.21.15.114:0/2000488585 shutdown_connections 2024-05-27T07:54:17.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.748+0000 7f720733b700 1 --2- 172.21.15.114:0/2000488585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7200109860 0x7f72001111c0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.748+0000 7f720733b700 1 --2- 172.21.15.114:0/2000488585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72001045e0 0x7f72001049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.748+0000 7f720733b700 1 --2- 172.21.15.114:0/2000488585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7200108f10 0x7f7200109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.748+0000 7f720733b700 1 -- 172.21.15.114:0/2000488585 >> 172.21.15.114:0/2000488585 conn(0x7f72000fffd0 msgr2=0x7f72001023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:17.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.748+0000 7f720733b700 1 -- 172.21.15.114:0/2000488585 shutdown_connections 2024-05-27T07:54:17.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.748+0000 7f720733b700 1 -- 172.21.15.114:0/2000488585 wait complete. 2024-05-27T07:54:17.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.748+0000 7f720733b700 1 Processor -- start 2024-05-27T07:54:17.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.748+0000 7f720733b700 1 -- start start 2024-05-27T07:54:17.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f720733b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72001045e0 0x7f720019ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f720733b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7200108f10 0x7f720019b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f720733b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7200109860 0x7f720019f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f720733b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7200114490 con 0x7f72001045e0 2024-05-27T07:54:17.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f720733b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7200114310 con 0x7f7200109860 2024-05-27T07:54:17.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f720733b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7200114610 con 0x7f7200108f10 2024-05-27T07:54:17.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f72050d7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72001045e0 0x7f720019ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f72050d7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72001045e0 0x7f720019ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45348/0 (socket says 172.21.15.114:45348) 2024-05-27T07:54:17.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f72050d7700 1 -- 172.21.15.114:0/958396494 learned_addr learned my addr 172.21.15.114:0/958396494 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:17.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f72048d6700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7200108f10 0x7f720019b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f72058d8700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7200109860 0x7f720019f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f72048d6700 1 -- 172.21.15.114:0/958396494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7200109860 msgr2=0x7f720019f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:17.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f72048d6700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7200109860 0x7f720019f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f72048d6700 1 -- 172.21.15.114:0/958396494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72001045e0 msgr2=0x7f720019ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:17.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f72048d6700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72001045e0 0x7f720019ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f72048d6700 1 -- 172.21.15.114:0/958396494 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f720019fc50 con 0x7f7200108f10 2024-05-27T07:54:17.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f72050d7700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72001045e0 0x7f720019ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:17.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.749+0000 7f72048d6700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7200108f10 0x7f720019b3c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f71f000e9c0 tx=0x7f71f000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:17.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.750+0000 7f71f67fc700 1 -- 172.21.15.114:0/958396494 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f000cbd0 con 0x7f7200108f10 2024-05-27T07:54:17.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.750+0000 7f720733b700 1 -- 172.21.15.114:0/958396494 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f720019ff40 con 0x7f7200108f10 2024-05-27T07:54:17.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.750+0000 7f71f67fc700 1 -- 172.21.15.114:0/958396494 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71f000cd70 con 0x7f7200108f10 2024-05-27T07:54:17.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.750+0000 7f71f67fc700 1 -- 172.21.15.114:0/958396494 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f0018680 con 0x7f7200108f10 2024-05-27T07:54:17.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.750+0000 7f720733b700 1 -- 172.21.15.114:0/958396494 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72001a7860 con 0x7f7200108f10 2024-05-27T07:54:17.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.753+0000 7f71f67fc700 1 -- 172.21.15.114:0/958396494 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f71f0010430 con 0x7f7200108f10 2024-05-27T07:54:17.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.754+0000 7f7215d50b80 1 Processor -- start 2024-05-27T07:54:17.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.754+0000 7f71f67fc700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f71ec0b29f0 0x7f71ec0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.754+0000 7f7215d50b80 1 -- start start 2024-05-27T07:54:17.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.754+0000 7f72050d7700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f71ec0b29f0 0x7f71ec0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.754+0000 7f71f67fc700 1 -- 172.21.15.114:0/958396494 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 9554+0+0 (secure 0 0 0) 0x7f71f00e2ba0 con 0x7f7200108f10 2024-05-27T07:54:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f7215d50b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392b7ee40 0x561392d02710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f7215d50b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392d02c50 0x561392cfca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f7215d50b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392cfcf80 0x561392cfd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f7215d50b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561392c2bc90 con 0x561392d02c50 2024-05-27T07:54:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f7215d50b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561392c1b830 con 0x561392b7ee40 2024-05-27T07:54:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f7215d50b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561392c1f870 con 0x561392cfcf80 2024-05-27T07:54:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f72048d6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392d02c50 0x561392cfca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f72048d6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392d02c50 0x561392cfca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45354/0 (socket says 172.21.15.114:45354) 2024-05-27T07:54:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f72048d6700 1 -- 172.21.15.114:0/3798167626 learned_addr learned my addr 172.21.15.114:0/3798167626 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f72058d8700 1 --2- 172.21.15.114:0/3798167626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392cfcf80 0x561392cfd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f72050d7700 1 --2- 172.21.15.114:0/3798167626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392b7ee40 0x561392d02710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f72050d7700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f71ec0b29f0 0x7f71ec0b4f00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0115a0 tx=0x7f71fc009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f72048d6700 1 -- 172.21.15.114:0/3798167626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392cfcf80 msgr2=0x561392cfd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f72048d6700 1 --2- 172.21.15.114:0/3798167626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392cfcf80 0x561392cfd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f72048d6700 1 -- 172.21.15.114:0/3798167626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392b7ee40 msgr2=0x561392d02710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f72048d6700 1 --2- 172.21.15.114:0/3798167626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392b7ee40 0x561392d02710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f72048d6700 1 -- 172.21.15.114:0/3798167626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561392b6b8d0 con 0x561392d02c50 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f72058d8700 1 --2- 172.21.15.114:0/3798167626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392cfcf80 0x561392cfd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.755+0000 7f72048d6700 1 --2- 172.21.15.114:0/3798167626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392d02c50 0x561392cfca40 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f71f00a5a90 tx=0x7f71f00a9180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0eb895f8b7e6e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.756+0000 7f71eb7fe700 1 -- 172.21.15.114:0/3798167626 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f00b21c0 con 0x561392d02c50 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.756+0000 7f71eb7fe700 1 -- 172.21.15.114:0/3798167626 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71f00a9520 con 0x561392d02c50 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.756+0000 7f71eb7fe700 1 -- 172.21.15.114:0/3798167626 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f00a9eb0 con 0x561392d02c50 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.757+0000 7f7215d50b80 1 -- 172.21.15.114:0/3798167626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392d02c50 msgr2=0x561392cfca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.757+0000 7f7215d50b80 1 --2- 172.21.15.114:0/3798167626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392d02c50 0x561392cfca40 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f71f00a5a90 tx=0x7f71f00a9180 comp rx=0 tx=0).stop 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.757+0000 7f7215d50b80 1 -- 172.21.15.114:0/3798167626 shutdown_connections 2024-05-27T07:54:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.757+0000 7f7215d50b80 1 --2- 172.21.15.114:0/3798167626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392cfcf80 0x561392cfd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.757+0000 7f7215d50b80 1 --2- 172.21.15.114:0/3798167626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392b7ee40 0x561392d02710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.757+0000 7f7215d50b80 1 --2- 172.21.15.114:0/3798167626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392d02c50 0x561392cfca40 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.757+0000 7f7215d50b80 1 -- 172.21.15.114:0/3798167626 >> 172.21.15.114:0/3798167626 conn(0x561392bd9af0 msgr2=0x561392cf7640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:17.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.757+0000 7f7215d50b80 1 -- 172.21.15.114:0/3798167626 shutdown_connections 2024-05-27T07:54:17.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.757+0000 7f7215d50b80 1 -- 172.21.15.114:0/3798167626 wait complete. 2024-05-27T07:54:17.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.757+0000 7f7215d50b80 1 -- start start 2024-05-27T07:54:17.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.757+0000 7f7215d50b80 1 Processor -- start 2024-05-27T07:54:17.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f7215d50b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392b7ee40 0x561392d063c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f7215d50b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392cfcf80 0x561392d06900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f7215d50b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392d02c50 0x561392d06e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f7215d50b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561392b21fa0 con 0x561392b7ee40 2024-05-27T07:54:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f7215d50b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561392bbd7c0 con 0x561392cfcf80 2024-05-27T07:54:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f7215d50b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561392d09740 con 0x561392d02c50 2024-05-27T07:54:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f7215d50b80 10 client.0 _ll_register_callbacks cb 0x561392bb6920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f72050d7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392b7ee40 0x561392d063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f72050d7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392b7ee40 0x561392d063c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45366/0 (socket says 172.21.15.114:45366) 2024-05-27T07:54:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f72050d7700 1 -- 172.21.15.114:0/1036839293 learned_addr learned my addr 172.21.15.114:0/1036839293 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f72058d8700 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392d02c50 0x561392d06e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f72048d6700 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392cfcf80 0x561392d06900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f72050d7700 1 -- 172.21.15.114:0/1036839293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392d02c50 msgr2=0x561392d06e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f72050d7700 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392d02c50 0x561392d06e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f72050d7700 1 -- 172.21.15.114:0/1036839293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392cfcf80 msgr2=0x561392d06900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f72050d7700 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392cfcf80 0x561392d06900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f72050d7700 1 -- 172.21.15.114:0/1036839293 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561392bba690 con 0x561392b7ee40 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f72058d8700 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392d02c50 0x561392d06e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f72048d6700 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392cfcf80 0x561392d06900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f72050d7700 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392b7ee40 0x561392d063c0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f71fc01cb90 tx=0x7f71fc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2db0419a66bf6b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.758+0000 7f71e8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.759+0000 7f71e8ff9700 1 -- 172.21.15.114:0/1036839293 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc01ba20 con 0x561392b7ee40 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.759+0000 7f71e8ff9700 1 -- 172.21.15.114:0/1036839293 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 9554+0+0 (secure 0 0 0) 0x7f71fc01bbc0 con 0x561392b7ee40 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.759+0000 7f7215d50b80 1 -- 172.21.15.114:0/1036839293 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71f00a9520 con 0x561392b7ee40 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.759+0000 7f71e8ff9700 1 -- 172.21.15.114:0/1036839293 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f71fc030050 con 0x561392b7ee40 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.759+0000 7f7215d50b80 10 client.18918 fetch_fsmap learned FSMap version 79 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.759+0000 7f7215d50b80 1 -- 172.21.15.114:0/1036839293 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x561392d14270 con 0x561392b7ee40 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.759+0000 7f71e8ff9700 1 -- 172.21.15.114:0/1036839293 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71fc01bdc0 con 0x561392b7ee40 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.759+0000 7f71e8ff9700 1 -- 172.21.15.114:0/1036839293 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 79) ==== 5305+0+0 (secure 0 0 0) 0x7f71fc024070 con 0x561392b7ee40 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.759+0000 7f7215d50b80 10 client.18918 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:54:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.759+0000 7f7215d50b80 10 client.18918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:17.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.760+0000 7f7215d50b80 20 client.18918 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:17.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.760+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561392d16c60 0x561392d190d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.760+0000 7f7215d50b80 4 client.18918 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:54:17.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.760+0000 7f7215d50b80 1 -- 172.21.15.114:0/1036839293 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f71f00a9520 con 0x561392d16c60 2024-05-27T07:54:17.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.760+0000 7f72048d6700 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561392d16c60 0x561392d190d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.760+0000 7f71e8ff9700 10 client.18918 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:17.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.761+0000 7f72048d6700 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561392d16c60 0x561392d190d0 crc :-1 s=READY pgs=412 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1df0ca76df5d080 server_cookie=f87d0e0b5259c2d9 in_seq=0 out_seq=0 2024-05-27T07:54:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f71e8ff9700 1 -- 172.21.15.114:0/1036839293 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f71f00a9520 con 0x561392d16c60 2024-05-27T07:54:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f71e8ff9700 10 client.18918 handle_command_reply: tid=0 2024-05-27T07:54:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f7215d50b80 1 client.18918 shutdown 2024-05-27T07:54:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f7215d50b80 1 -- 172.21.15.114:0/1036839293 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392b7ee40 msgr2=0x561392d063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392b7ee40 0x561392d063c0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f71fc01cb90 tx=0x7f71fc0047c0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f7215d50b80 1 -- 172.21.15.114:0/1036839293 shutdown_connections 2024-05-27T07:54:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392d02c50 0x561392d06e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392b7ee40 0x561392d063c0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561392d16c60 0x561392d190d0 crc :-1 s=READY pgs=412 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1036839293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392cfcf80 0x561392d06900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f7215d50b80 1 -- 172.21.15.114:0/1036839293 >> 172.21.15.114:0/1036839293 conn(0x561392bd9af0 msgr2=0x561392cfba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f71e8ff9700 0 client.18918 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f7215d50b80 1 -- 172.21.15.114:0/1036839293 shutdown_connections 2024-05-27T07:54:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f7215d50b80 1 -- 172.21.15.114:0/1036839293 wait complete. 2024-05-27T07:54:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.764+0000 7f7215d50b80 20 client.18918 trim_cache size 0 max 16384 2024-05-27T07:54:17.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.782+0000 7f7215d50b80 1 Processor -- start 2024-05-27T07:54:17.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.782+0000 7f7215d50b80 1 -- start start 2024-05-27T07:54:17.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f7215d50b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392b7ee40 0x561392d04070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f7215d50b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392cfcf80 0x561392d01b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f7215d50b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392d02c50 0x561392d127e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f7215d50b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f72001b1190 con 0x561392cfcf80 2024-05-27T07:54:17.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f7215d50b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72001b1860 con 0x561392d02c50 2024-05-27T07:54:17.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f7215d50b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72001b0010 con 0x561392b7ee40 2024-05-27T07:54:17.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f72048d6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392cfcf80 0x561392d01b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f72050d7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392b7ee40 0x561392d04070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f72048d6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392cfcf80 0x561392d01b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45376/0 (socket says 172.21.15.114:45376) 2024-05-27T07:54:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f72048d6700 1 -- 172.21.15.114:0/1235647619 learned_addr learned my addr 172.21.15.114:0/1235647619 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f72058d8700 1 --2- 172.21.15.114:0/1235647619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392d02c50 0x561392d127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f72048d6700 1 -- 172.21.15.114:0/1235647619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392b7ee40 msgr2=0x561392d04070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f72048d6700 1 --2- 172.21.15.114:0/1235647619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392b7ee40 0x561392d04070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f72048d6700 1 -- 172.21.15.114:0/1235647619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392d02c50 msgr2=0x561392d127e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f72048d6700 1 --2- 172.21.15.114:0/1235647619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392d02c50 0x561392d127e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f72048d6700 1 -- 172.21.15.114:0/1235647619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561392cf87c0 con 0x561392cfcf80 2024-05-27T07:54:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.783+0000 7f72048d6700 1 --2- 172.21.15.114:0/1235647619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392cfcf80 0x561392d01b80 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f71f00a6c80 tx=0x7f71f00c0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7ef97c82602680 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.784+0000 7f71eaffd700 1 -- 172.21.15.114:0/1235647619 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f00bee60 con 0x561392cfcf80 2024-05-27T07:54:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.784+0000 7f71eaffd700 1 -- 172.21.15.114:0/1235647619 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71f00aa110 con 0x561392cfcf80 2024-05-27T07:54:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.784+0000 7f71eaffd700 1 -- 172.21.15.114:0/1235647619 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f00b2640 con 0x561392cfcf80 2024-05-27T07:54:17.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.785+0000 7f7215d50b80 1 -- 172.21.15.114:0/1235647619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392cfcf80 msgr2=0x561392d01b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:17.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.785+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1235647619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392cfcf80 0x561392d01b80 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f71f00a6c80 tx=0x7f71f00c0040 comp rx=0 tx=0).stop 2024-05-27T07:54:17.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.785+0000 7f7215d50b80 1 -- 172.21.15.114:0/1235647619 shutdown_connections 2024-05-27T07:54:17.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.785+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1235647619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392d02c50 0x561392d127e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.785+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1235647619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392b7ee40 0x561392d04070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.785+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1235647619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392cfcf80 0x561392d01b80 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.785+0000 7f7215d50b80 1 -- 172.21.15.114:0/1235647619 >> 172.21.15.114:0/1235647619 conn(0x561392bd9af0 msgr2=0x561392d01470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:17.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.785+0000 7f7215d50b80 1 -- 172.21.15.114:0/1235647619 shutdown_connections 2024-05-27T07:54:17.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.785+0000 7f7215d50b80 1 -- 172.21.15.114:0/1235647619 wait complete. 2024-05-27T07:54:17.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.785+0000 7f7215d50b80 1 -- start start 2024-05-27T07:54:17.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.786+0000 7f7215d50b80 1 Processor -- start 2024-05-27T07:54:17.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.786+0000 7f7215d50b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392b7ee40 0x561392d0e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.786+0000 7f7215d50b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392cfcf80 0x561392d0ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.786+0000 7f7215d50b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392d02c50 0x561392d1d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.786+0000 7f72050d7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392b7ee40 0x561392d0e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.786+0000 7f72058d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392d02c50 0x561392d1d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.786+0000 7f72050d7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392b7ee40 0x561392d0e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33576/0 (socket says 172.21.15.114:33576) 2024-05-27T07:54:17.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.786+0000 7f72048d6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392cfcf80 0x561392d0ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.786+0000 7f72050d7700 1 -- 172.21.15.114:0/1699202687 learned_addr learned my addr 172.21.15.114:0/1699202687 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:17.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.786+0000 7f7215d50b80 1 -- 172.21.15.114:0/1699202687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f72001af940 con 0x561392d02c50 2024-05-27T07:54:17.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.786+0000 7f7215d50b80 1 -- 172.21.15.114:0/1699202687 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72001ae530 con 0x561392cfcf80 2024-05-27T07:54:17.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.786+0000 7f7215d50b80 1 -- 172.21.15.114:0/1699202687 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72001aeba0 con 0x561392b7ee40 2024-05-27T07:54:17.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f7215d50b80 10 client.0 _ll_register_callbacks cb 0x561392bb6920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:17.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f72048d6700 1 -- 172.21.15.114:0/1699202687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392b7ee40 msgr2=0x561392d0e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:17.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f72048d6700 1 --2- 172.21.15.114:0/1699202687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392b7ee40 0x561392d0e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f72048d6700 1 -- 172.21.15.114:0/1699202687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392d02c50 msgr2=0x561392d1d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:17.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f72048d6700 1 --2- 172.21.15.114:0/1699202687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392d02c50 0x561392d1d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f72048d6700 1 -- 172.21.15.114:0/1699202687 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561392d1de00 con 0x561392cfcf80 2024-05-27T07:54:17.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f72048d6700 1 --2- 172.21.15.114:0/1699202687 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392cfcf80 0x561392d0ef10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f71f00a93a0 tx=0x7f71f000efd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92ef67ab7b431c56 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:17.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f71e8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f71e8ff9700 1 -- 172.21.15.114:0/1699202687 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f00bee60 con 0x561392cfcf80 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f71e8ff9700 1 -- 172.21.15.114:0/1699202687 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 9554+0+0 (secure 0 0 0) 0x7f71f00b25f0 con 0x561392cfcf80 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f7215d50b80 1 -- 172.21.15.114:0/1699202687 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71f00aa110 con 0x561392cfcf80 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f71e8ff9700 1 -- 172.21.15.114:0/1699202687 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f71f0010930 con 0x561392cfcf80 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f7215d50b80 10 client.28286 fetch_fsmap learned FSMap version 79 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.787+0000 7f7215d50b80 1 -- 172.21.15.114:0/1699202687 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x561392d80d60 con 0x561392cfcf80 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.788+0000 7f71e8ff9700 1 -- 172.21.15.114:0/1699202687 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71f00aa110 con 0x561392cfcf80 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.788+0000 7f71e8ff9700 1 -- 172.21.15.114:0/1699202687 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 79) ==== 5305+0+0 (secure 0 0 0) 0x7f71f008a030 con 0x561392cfcf80 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.788+0000 7f7215d50b80 10 client.28286 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.788+0000 7f7215d50b80 10 client.28286 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.788+0000 7f7215d50b80 20 client.28286 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.788+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1699202687 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561392d812d0 0x561392d836f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.788+0000 7f7215d50b80 4 client.28286 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:54:17.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.788+0000 7f7215d50b80 1 -- 172.21.15.114:0/1699202687 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561392cc89f0 con 0x561392d812d0 2024-05-27T07:54:17.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.788+0000 7f72050d7700 1 --2- 172.21.15.114:0/1699202687 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561392d812d0 0x561392d836f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:17.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.788+0000 7f71e8ff9700 10 client.28286 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:17.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.789+0000 7f72050d7700 1 --2- 172.21.15.114:0/1699202687 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561392d812d0 0x561392d836f0 crc :-1 s=READY pgs=413 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b03ad21d0a8df9a server_cookie=8e333f35d42b4504 in_seq=0 out_seq=0 2024-05-27T07:54:17.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f71e8ff9700 1 -- 172.21.15.114:0/1699202687 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561392cc89f0 con 0x561392d812d0 2024-05-27T07:54:17.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f71e8ff9700 10 client.28286 handle_command_reply: tid=0 2024-05-27T07:54:17.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f7215d50b80 1 client.28286 shutdown 2024-05-27T07:54:17.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f7215d50b80 1 -- 172.21.15.114:0/1699202687 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392cfcf80 msgr2=0x561392d0ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:17.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1699202687 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392cfcf80 0x561392d0ef10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f71f00a93a0 tx=0x7f71f000efd0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f7215d50b80 1 -- 172.21.15.114:0/1699202687 shutdown_connections 2024-05-27T07:54:17.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1699202687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561392d02c50 0x561392d1d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1699202687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561392b7ee40 0x561392d0e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1699202687 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561392d812d0 0x561392d836f0 crc :-1 s=READY pgs=413 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f7215d50b80 1 --2- 172.21.15.114:0/1699202687 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561392cfcf80 0x561392d0ef10 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f71e8ff9700 0 client.28286 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:17.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f7215d50b80 1 -- 172.21.15.114:0/1699202687 >> 172.21.15.114:0/1699202687 conn(0x561392bd9af0 msgr2=0x561392d00010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:17.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f7215d50b80 1 -- 172.21.15.114:0/1699202687 shutdown_connections 2024-05-27T07:54:17.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f7215d50b80 1 -- 172.21.15.114:0/1699202687 wait complete. 2024-05-27T07:54:17.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.793+0000 7f7215d50b80 20 client.28286 trim_cache size 0 max 16384 2024-05-27T07:54:17.796 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:54:17.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.794+0000 7f720733b700 1 -- 172.21.15.114:0/958396494 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f71ec0b29f0 msgr2=0x7f71ec0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:17.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.794+0000 7f720733b700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f71ec0b29f0 0x7f71ec0b4f00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0115a0 tx=0x7f71fc009260 comp rx=0 tx=0).stop 2024-05-27T07:54:17.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.794+0000 7f720733b700 1 -- 172.21.15.114:0/958396494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7200108f10 msgr2=0x7f720019b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:17.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.794+0000 7f720733b700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7200108f10 0x7f720019b3c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f71f000e9c0 tx=0x7f71f000ecd0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.794+0000 7f720733b700 1 -- 172.21.15.114:0/958396494 shutdown_connections 2024-05-27T07:54:17.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.794+0000 7f720733b700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f71ec0b29f0 0x7f71ec0b4f00 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.794+0000 7f720733b700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7200109860 0x7f720019f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.794+0000 7f720733b700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72001045e0 0x7f720019ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.794+0000 7f720733b700 1 --2- 172.21.15.114:0/958396494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7200108f10 0x7f720019b3c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:17.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.794+0000 7f720733b700 1 -- 172.21.15.114:0/958396494 >> 172.21.15.114:0/958396494 conn(0x7f72000fffd0 msgr2=0x7f7200106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:17.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.794+0000 7f720733b700 1 -- 172.21.15.114:0/958396494 shutdown_connections 2024-05-27T07:54:17.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:17.794+0000 7f720733b700 1 -- 172.21.15.114:0/958396494 wait complete. 2024-05-27T07:54:17.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:17 smithi163 ceph-mon[36451]: pgmap v490: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 201 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 2.4 MiB/s wr, 566 op/s 2024-05-27T07:54:17.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:17 smithi186 ceph-mon[36647]: pgmap v490: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 201 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 2.4 MiB/s wr, 566 op/s 2024-05-27T07:54:18.591 INFO:tasks.workunit.client.0.smithi114.stdout: 10 6492 17.35 MB/sec warmup 30 sec latency 108.657 ms 2024-05-27T07:54:19.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 6919 17.78 MB/sec warmup 31 sec latency 164.101 ms 2024-05-27T07:54:19.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:19 smithi163 ceph-mon[36451]: pgmap v491: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 210 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 568 op/s 2024-05-27T07:54:19.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:19 smithi163 ceph-mon[36451]: osdmap e137: 12 total, 12 up, 12 in 2024-05-27T07:54:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:19 smithi186 ceph-mon[36647]: pgmap v491: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 210 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 568 op/s 2024-05-27T07:54:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:19 smithi186 ceph-mon[36647]: osdmap e137: 12 total, 12 up, 12 in 2024-05-27T07:54:20.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:19 smithi114 ceph-mon[32038]: pgmap v491: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 210 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 568 op/s 2024-05-27T07:54:20.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:19 smithi114 ceph-mon[32038]: osdmap e137: 12 total, 12 up, 12 in 2024-05-27T07:54:20.140 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:54:20.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:54:20.535 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:54:20.591 INFO:tasks.workunit.client.0.smithi114.stdout: 10 7246 17.92 MB/sec warmup 32 sec latency 160.191 ms 2024-05-27T07:54:20.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:20 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:54:20.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:20 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:54:20.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:20 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:54:21.591 INFO:tasks.workunit.client.0.smithi114.stdout: 10 7434 17.66 MB/sec warmup 33 sec latency 331.171 ms 2024-05-27T07:54:21.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:21 smithi114 ceph-mon[32038]: pgmap v493: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 210 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 255 op/s 2024-05-27T07:54:21.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:21 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:54:21.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:21 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:54:21.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:21 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:54:21.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:21 smithi163 ceph-mon[36451]: pgmap v493: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 210 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 255 op/s 2024-05-27T07:54:21.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:21 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:54:21.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:21 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:54:21.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:21 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:54:21.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:21 smithi186 ceph-mon[36647]: pgmap v493: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 210 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 255 op/s 2024-05-27T07:54:21.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:21 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:54:21.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:21 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:54:21.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:21 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:54:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.045+0000 7fa57638d640 1 -- 172.21.15.114:0/4071607865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa570108b90 msgr2=0x7fa570102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.045+0000 7fa57638d640 1 --2- 172.21.15.114:0/4071607865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa570108b90 0x7fa570102730 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fa560009900 tx=0x7fa56002a410 comp rx=0 tx=0).stop 2024-05-27T07:54:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.045+0000 7fa57638d640 1 -- 172.21.15.114:0/4071607865 shutdown_connections 2024-05-27T07:54:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.045+0000 7fa57638d640 1 --2- 172.21.15.114:0/4071607865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5701032e0 0x7fa57010f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.045+0000 7fa57638d640 1 --2- 172.21.15.114:0/4071607865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa570109030 0x7fa570102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.045+0000 7fa57638d640 1 --2- 172.21.15.114:0/4071607865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa570108b90 0x7fa570102730 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.045+0000 7fa57638d640 1 -- 172.21.15.114:0/4071607865 >> 172.21.15.114:0/4071607865 conn(0x7fa5700782d0 msgr2=0x7fa570078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.045+0000 7fa57638d640 1 -- 172.21.15.114:0/4071607865 shutdown_connections 2024-05-27T07:54:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.045+0000 7fa57638d640 1 -- 172.21.15.114:0/4071607865 wait complete. 2024-05-27T07:54:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.046+0000 7fa57638d640 1 Processor -- start 2024-05-27T07:54:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.046+0000 7fa57638d640 1 -- start start 2024-05-27T07:54:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.046+0000 7fa57638d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5701032e0 0x7fa57019b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.046+0000 7fa57638d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa570108b90 0x7fa57019bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.046+0000 7fa57638d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa570109030 0x7fa57019c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.046+0000 7fa57638d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa570112d80 con 0x7fa570108b90 2024-05-27T07:54:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.046+0000 7fa57638d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa570112c00 con 0x7fa570109030 2024-05-27T07:54:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.046+0000 7fa57638d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa570112f00 con 0x7fa5701032e0 2024-05-27T07:54:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.046+0000 7fa567fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa570108b90 0x7fa57019bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.046+0000 7fa56ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5701032e0 0x7fa57019b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.046+0000 7fa567fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa570108b90 0x7fa57019bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45398/0 (socket says 172.21.15.114:45398) 2024-05-27T07:54:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.046+0000 7fa567fff640 1 -- 172.21.15.114:0/1154132129 learned_addr learned my addr 172.21.15.114:0/1154132129 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.046+0000 7fa57492a640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa570109030 0x7fa57019c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.047+0000 7fa567fff640 1 -- 172.21.15.114:0/1154132129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5701032e0 msgr2=0x7fa57019b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.047+0000 7fa567fff640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5701032e0 0x7fa57019b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.047+0000 7fa567fff640 1 -- 172.21.15.114:0/1154132129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa570109030 msgr2=0x7fa57019c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.047+0000 7fa567fff640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa570109030 0x7fa57019c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.047+0000 7fa567fff640 1 -- 172.21.15.114:0/1154132129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5701a6a30 con 0x7fa570108b90 2024-05-27T07:54:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.047+0000 7fa57492a640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa570109030 0x7fa57019c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.047+0000 7fa567fff640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa570108b90 0x7fa57019bc60 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fa55800e9f0 tx=0x7fa55800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.047+0000 7fa56dffb640 1 -- 172.21.15.114:0/1154132129 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55800ccc0 con 0x7fa570108b90 2024-05-27T07:54:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.047+0000 7fa56dffb640 1 -- 172.21.15.114:0/1154132129 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5580035f0 con 0x7fa570108b90 2024-05-27T07:54:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.047+0000 7fa57638d640 1 -- 172.21.15.114:0/1154132129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5701a6d20 con 0x7fa570108b90 2024-05-27T07:54:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.047+0000 7fa56dffb640 1 -- 172.21.15.114:0/1154132129 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55800ce60 con 0x7fa570108b90 2024-05-27T07:54:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.047+0000 7fa57638d640 1 -- 172.21.15.114:0/1154132129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5701a7210 con 0x7fa570108b90 2024-05-27T07:54:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa57d24a740 1 Processor -- start 2024-05-27T07:54:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa56dffb640 1 -- 172.21.15.114:0/1154132129 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa558003aa0 con 0x7fa570108b90 2024-05-27T07:54:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa57d24a740 1 -- start start 2024-05-27T07:54:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa57d24a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a6541b0 0x56273a9f5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa57d24a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a62b2f0 0x56273a9f57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa57d24a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6334d0 0x56273a9f5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa57d24a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56273a967c30 con 0x56273a62b2f0 2024-05-27T07:54:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa57d24a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56273a9577d0 con 0x56273a6334d0 2024-05-27T07:54:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa57d24a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56273a95b810 con 0x56273a6541b0 2024-05-27T07:54:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa56ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a62b2f0 0x56273a9f57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa56ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a62b2f0 0x56273a9f57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45404/0 (socket says 172.21.15.114:45404) 2024-05-27T07:54:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa56ffff640 1 -- 172.21.15.114:0/2892633764 learned_addr learned my addr 172.21.15.114:0/2892633764 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa57492a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a6541b0 0x56273a9f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa567fff640 1 --2- 172.21.15.114:0/2892633764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6334d0 0x56273a9f5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa56ffff640 1 -- 172.21.15.114:0/2892633764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a6541b0 msgr2=0x56273a9f5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa56ffff640 1 --2- 172.21.15.114:0/2892633764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a6541b0 0x56273a9f5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa56ffff640 1 -- 172.21.15.114:0/2892633764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6334d0 msgr2=0x56273a9f5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa56ffff640 1 --2- 172.21.15.114:0/2892633764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6334d0 0x56273a9f5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.050+0000 7fa56ffff640 1 -- 172.21.15.114:0/2892633764 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56273a66c210 con 0x56273a62b2f0 2024-05-27T07:54:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.051+0000 7fa56dffb640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa55c0b2950 0x7fa55c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.051+0000 7fa56ffff640 1 --2- 172.21.15.114:0/2892633764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a62b2f0 0x56273a9f57a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fa560009900 tx=0x7fa560032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f32e817917cc730 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.051+0000 7fa57492a640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa55c0b2950 0x7fa55c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.051+0000 7fa56dffb640 1 -- 172.21.15.114:0/1154132129 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fa5580e2cc0 con 0x7fa570108b90 2024-05-27T07:54:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.051+0000 7fa5667fc640 1 -- 172.21.15.114:0/2892633764 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa560040070 con 0x56273a62b2f0 2024-05-27T07:54:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.051+0000 7fa5667fc640 1 -- 172.21.15.114:0/2892633764 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa560004030 con 0x56273a62b2f0 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.051+0000 7fa5667fc640 1 -- 172.21.15.114:0/2892633764 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa560035040 con 0x56273a62b2f0 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.052+0000 7fa57d24a740 1 -- 172.21.15.114:0/2892633764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a62b2f0 msgr2=0x56273a9f57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.052+0000 7fa57492a640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa55c0b2950 0x7fa55c0b4e60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x56273a9f2570 tx=0x7fa568009500 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.052+0000 7fa57d24a740 1 --2- 172.21.15.114:0/2892633764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a62b2f0 0x56273a9f57a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fa560009900 tx=0x7fa560032b20 comp rx=0 tx=0).stop 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.052+0000 7fa57d24a740 1 -- 172.21.15.114:0/2892633764 shutdown_connections 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.052+0000 7fa57d24a740 1 --2- 172.21.15.114:0/2892633764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6334d0 0x56273a9f5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.052+0000 7fa57d24a740 1 --2- 172.21.15.114:0/2892633764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a62b2f0 0x56273a9f57a0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.052+0000 7fa57d24a740 1 --2- 172.21.15.114:0/2892633764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a6541b0 0x56273a9f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.052+0000 7fa57d24a740 1 -- 172.21.15.114:0/2892633764 >> 172.21.15.114:0/2892633764 conn(0x56273a648300 msgr2=0x56273a9f1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.052+0000 7fa57d24a740 1 -- 172.21.15.114:0/2892633764 shutdown_connections 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.052+0000 7fa57d24a740 1 -- 172.21.15.114:0/2892633764 wait complete. 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.052+0000 7fa57d24a740 1 -- start start 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.052+0000 7fa57d24a740 1 Processor -- start 2024-05-27T07:54:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57d24a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a62b2f0 0x56273aa05980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57d24a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6334d0 0x56273aa05ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57d24a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a6541b0 0x56273aa03d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57d24a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56273a63d790 con 0x56273a6541b0 2024-05-27T07:54:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57d24a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56273a913890 con 0x56273a6334d0 2024-05-27T07:54:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57d24a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56273aa06910 con 0x56273a62b2f0 2024-05-27T07:54:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57d24a740 10 client.0 _ll_register_callbacks cb 0x56273a66ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57492a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a6541b0 0x56273aa03d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57492a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a6541b0 0x56273aa03d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45406/0 (socket says 172.21.15.114:45406) 2024-05-27T07:54:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa56ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a62b2f0 0x56273aa05980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57492a640 1 -- 172.21.15.114:0/2040569997 learned_addr learned my addr 172.21.15.114:0/2040569997 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa567fff640 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6334d0 0x56273aa05ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57492a640 1 -- 172.21.15.114:0/2040569997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a62b2f0 msgr2=0x56273aa05980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57492a640 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a62b2f0 0x56273aa05980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57492a640 1 -- 172.21.15.114:0/2040569997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6334d0 msgr2=0x56273aa05ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57492a640 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6334d0 0x56273aa05ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57492a640 1 -- 172.21.15.114:0/2040569997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56273a738620 con 0x56273a6541b0 2024-05-27T07:54:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa567fff640 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6334d0 0x56273aa05ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa56ffff640 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a62b2f0 0x56273aa05980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa57492a640 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a6541b0 0x56273aa03d90 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fa56800fa50 tx=0x7fa568004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a69e88d94aaafc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.053+0000 7fa54f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:54:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa54f7fe640 1 -- 172.21.15.114:0/2040569997 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568011bd0 con 0x56273a6541b0 2024-05-27T07:54:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa54f7fe640 1 -- 172.21.15.114:0/2040569997 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fa568011d70 con 0x56273a6541b0 2024-05-27T07:54:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa57d24a740 1 -- 172.21.15.114:0/2040569997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa560004030 con 0x56273a6541b0 2024-05-27T07:54:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa54f7fe640 1 -- 172.21.15.114:0/2040569997 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fa568029440 con 0x56273a6541b0 2024-05-27T07:54:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa57d24a740 10 client.18936 fetch_fsmap learned FSMap version 79 2024-05-27T07:54:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa57d24a740 1 -- 172.21.15.114:0/2040569997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x56273aa12d10 con 0x56273a6541b0 2024-05-27T07:54:22.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa54f7fe640 1 -- 172.21.15.114:0/2040569997 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa560004030 con 0x56273a6541b0 2024-05-27T07:54:22.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa54f7fe640 1 -- 172.21.15.114:0/2040569997 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 79) ==== 5305+0+0 (secure 0 0 0) 0x7fa568027070 con 0x56273a6541b0 2024-05-27T07:54:22.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa57d24a740 10 client.18936 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:54:22.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa57d24a740 10 client.18936 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:22.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa57d24a740 20 client.18936 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:22.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa57d24a740 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56273a612d80 0x56273aa166e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa57d24a740 4 client.18936 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:54:22.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa57d24a740 1 -- 172.21.15.114:0/2040569997 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56273a969b80 con 0x56273a612d80 2024-05-27T07:54:22.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.054+0000 7fa56ffff640 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56273a612d80 0x56273aa166e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.055+0000 7fa54f7fe640 10 client.18936 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:22.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.055+0000 7fa56ffff640 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56273a612d80 0x56273aa166e0 crc :-1 s=READY pgs=414 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16dfb42f95debd58 server_cookie=841f5127b796fefc in_seq=0 out_seq=0 2024-05-27T07:54:22.063 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.058+0000 7fa54f7fe640 1 -- 172.21.15.114:0/2040569997 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56273a969b80 con 0x56273a612d80 2024-05-27T07:54:22.063 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.058+0000 7fa54f7fe640 10 client.18936 handle_command_reply: tid=0 2024-05-27T07:54:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.058+0000 7fa57d24a740 1 client.18936 shutdown 2024-05-27T07:54:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.058+0000 7fa57d24a740 1 -- 172.21.15.114:0/2040569997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a6541b0 msgr2=0x56273aa03d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.058+0000 7fa57d24a740 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a6541b0 0x56273aa03d90 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fa56800fa50 tx=0x7fa568004ab0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.058+0000 7fa57d24a740 1 -- 172.21.15.114:0/2040569997 shutdown_connections 2024-05-27T07:54:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.058+0000 7fa57d24a740 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56273a612d80 0x56273aa166e0 crc :-1 s=READY pgs=414 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.058+0000 7fa57d24a740 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a6541b0 0x56273aa03d90 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.058+0000 7fa57d24a740 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6334d0 0x56273aa05ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.058+0000 7fa54f7fe640 0 client.18936 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.058+0000 7fa57d24a740 1 --2- 172.21.15.114:0/2040569997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a62b2f0 0x56273aa05980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.058+0000 7fa57d24a740 1 -- 172.21.15.114:0/2040569997 >> 172.21.15.114:0/2040569997 conn(0x56273a648300 msgr2=0x56273a9f70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.058+0000 7fa57d24a740 1 -- 172.21.15.114:0/2040569997 shutdown_connections 2024-05-27T07:54:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.059+0000 7fa57d24a740 1 -- 172.21.15.114:0/2040569997 wait complete. 2024-05-27T07:54:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.059+0000 7fa57d24a740 20 client.18936 trim_cache size 0 max 16384 2024-05-27T07:54:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.076+0000 7fa57d24a740 1 Processor -- start 2024-05-27T07:54:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.076+0000 7fa57d24a740 1 -- start start 2024-05-27T07:54:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.076+0000 7fa57d24a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a62b2f0 0x56273a9fd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.076+0000 7fa57d24a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a6334d0 0x56273a9fdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.076+0000 7fa57d24a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6541b0 0x56273aa1bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.076+0000 7fa57d24a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa5701afda0 con 0x56273a6334d0 2024-05-27T07:54:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.076+0000 7fa57d24a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa5701b0470 con 0x56273a6541b0 2024-05-27T07:54:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.076+0000 7fa57d24a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5701aec40 con 0x56273a62b2f0 2024-05-27T07:54:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.076+0000 7fa57492a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6541b0 0x56273aa1bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.076+0000 7fa57492a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6541b0 0x56273aa1bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:53468/0 (socket says 172.21.15.114:53468) 2024-05-27T07:54:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.076+0000 7fa567fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a6334d0 0x56273a9fdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.076+0000 7fa57492a640 1 -- 172.21.15.114:0/965394245 learned_addr learned my addr 172.21.15.114:0/965394245 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.077+0000 7fa56ffff640 1 --2- 172.21.15.114:0/965394245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a62b2f0 0x56273a9fd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.077+0000 7fa57492a640 1 -- 172.21.15.114:0/965394245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a62b2f0 msgr2=0x56273a9fd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.077+0000 7fa57492a640 1 --2- 172.21.15.114:0/965394245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a62b2f0 0x56273a9fd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.077+0000 7fa57492a640 1 -- 172.21.15.114:0/965394245 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a6334d0 msgr2=0x56273a9fdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.077+0000 7fa57492a640 1 --2- 172.21.15.114:0/965394245 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a6334d0 0x56273a9fdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.077+0000 7fa57492a640 1 -- 172.21.15.114:0/965394245 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56273aa022e0 con 0x56273a6541b0 2024-05-27T07:54:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.077+0000 7fa57492a640 1 --2- 172.21.15.114:0/965394245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6541b0 0x56273aa1bee0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa56801f940 tx=0x7fa568005fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2040e3a28772244 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.077+0000 7fa565ffb640 1 -- 172.21.15.114:0/965394245 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56803c070 con 0x56273a6541b0 2024-05-27T07:54:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.077+0000 7fa565ffb640 1 -- 172.21.15.114:0/965394245 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa56801f9c0 con 0x56273a6541b0 2024-05-27T07:54:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.077+0000 7fa565ffb640 1 -- 172.21.15.114:0/965394245 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568029d70 con 0x56273a6541b0 2024-05-27T07:54:22.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.079+0000 7fa57d24a740 1 -- 172.21.15.114:0/965394245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6541b0 msgr2=0x56273aa1bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:22.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.079+0000 7fa57d24a740 1 --2- 172.21.15.114:0/965394245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6541b0 0x56273aa1bee0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa56801f940 tx=0x7fa568005fc0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.088+0000 7fa57d24a740 1 -- 172.21.15.114:0/965394245 shutdown_connections 2024-05-27T07:54:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.088+0000 7fa57d24a740 1 --2- 172.21.15.114:0/965394245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6541b0 0x56273aa1bee0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.088+0000 7fa57d24a740 1 --2- 172.21.15.114:0/965394245 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a6334d0 0x56273a9fdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.088+0000 7fa57d24a740 1 --2- 172.21.15.114:0/965394245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a62b2f0 0x56273a9fd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.088+0000 7fa57d24a740 1 -- 172.21.15.114:0/965394245 >> 172.21.15.114:0/965394245 conn(0x56273a648300 msgr2=0x56273aa02660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.092+0000 7fa57d24a740 1 -- 172.21.15.114:0/965394245 shutdown_connections 2024-05-27T07:54:22.095 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.093+0000 7fa57d24a740 1 -- 172.21.15.114:0/965394245 wait complete. 2024-05-27T07:54:22.095 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.093+0000 7fa57d24a740 1 -- start start 2024-05-27T07:54:22.095 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.094+0000 7fa57d24a740 1 Processor -- start 2024-05-27T07:54:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.095+0000 7fa57d24a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a62b2f0 0x56273aa0a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.095+0000 7fa57d24a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a6334d0 0x56273aa0aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.095+0000 7fa57d24a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6541b0 0x56273aa0b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.095+0000 7fa57d24a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa5701ae570 con 0x56273a62b2f0 2024-05-27T07:54:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.095+0000 7fa57d24a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa5701ad180 con 0x56273a6541b0 2024-05-27T07:54:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.095+0000 7fa57d24a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5701ad7d0 con 0x56273a6334d0 2024-05-27T07:54:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.095+0000 7fa57d24a740 10 client.0 _ll_register_callbacks cb 0x7fa568012f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.098+0000 7fa567fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a6334d0 0x56273aa0aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.099+0000 7fa57492a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6541b0 0x56273aa0b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.099+0000 7fa567fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a6334d0 0x56273aa0aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33620/0 (socket says 172.21.15.114:33620) 2024-05-27T07:54:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.099+0000 7fa567fff640 1 -- 172.21.15.114:0/487043497 learned_addr learned my addr 172.21.15.114:0/487043497 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.100+0000 7fa567fff640 1 -- 172.21.15.114:0/487043497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6541b0 msgr2=0x56273aa0b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.100+0000 7fa567fff640 1 --2- 172.21.15.114:0/487043497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6541b0 0x56273aa0b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.102 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.100+0000 7fa567fff640 1 -- 172.21.15.114:0/487043497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a62b2f0 msgr2=0x56273aa0a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:22.102 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.100+0000 7fa567fff640 1 --2- 172.21.15.114:0/487043497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a62b2f0 0x56273aa0a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:22.102 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.100+0000 7fa567fff640 1 -- 172.21.15.114:0/487043497 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56273a9fc260 con 0x56273a6334d0 2024-05-27T07:54:22.102 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.100+0000 7fa567fff640 1 --2- 172.21.15.114:0/487043497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a6334d0 0x56273aa0aed0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa55800e9c0 tx=0x7fa5580aa130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c03d514af26cc77 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:22.102 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.100+0000 7fa54f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:54:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.107+0000 7fa54f7fe640 1 -- 172.21.15.114:0/487043497 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5580e4040 con 0x56273a6334d0 2024-05-27T07:54:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.107+0000 7fa54f7fe640 1 -- 172.21.15.114:0/487043497 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fa5580a9000 con 0x56273a6334d0 2024-05-27T07:54:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.108+0000 7fa57d24a740 1 -- 172.21.15.114:0/487043497 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa56801f9c0 con 0x56273a6334d0 2024-05-27T07:54:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.109+0000 7fa54f7fe640 1 -- 172.21.15.114:0/487043497 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fa5580a94e0 con 0x56273a6334d0 2024-05-27T07:54:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.112+0000 7fa57d24a740 10 client.28444 fetch_fsmap learned FSMap version 79 2024-05-27T07:54:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.112+0000 7fa57d24a740 1 -- 172.21.15.114:0/487043497 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x56273a9fd4b0 con 0x56273a6334d0 2024-05-27T07:54:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.115+0000 7fa54f7fe640 1 -- 172.21.15.114:0/487043497 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa56801f9c0 con 0x56273a6334d0 2024-05-27T07:54:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.115+0000 7fa54f7fe640 1 -- 172.21.15.114:0/487043497 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 79) ==== 5305+0+0 (secure 0 0 0) 0x7fa558003fb0 con 0x56273a6334d0 2024-05-27T07:54:22.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.115+0000 7fa57d24a740 10 client.28444 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:54:22.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.115+0000 7fa57d24a740 10 client.28444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:22.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.115+0000 7fa57d24a740 20 client.28444 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:22.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.115+0000 7fa57d24a740 1 --2- 172.21.15.114:0/487043497 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56273a5e81c0 0x56273aa965c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:22.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.115+0000 7fa57d24a740 4 client.28444 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:54:22.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.115+0000 7fa57d24a740 1 -- 172.21.15.114:0/487043497 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56273a963360 con 0x56273a5e81c0 2024-05-27T07:54:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.118+0000 7fa56ffff640 1 --2- 172.21.15.114:0/487043497 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56273a5e81c0 0x56273aa965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:22.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.129+0000 7fa56ffff640 1 --2- 172.21.15.114:0/487043497 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56273a5e81c0 0x56273aa965c0 crc :-1 s=READY pgs=415 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=904896865a04b5ec server_cookie=52a6c5813b47b750 in_seq=0 out_seq=0 2024-05-27T07:54:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:22.129+0000 7fa54f7fe640 10 client.28444 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:22.592 INFO:tasks.workunit.client.0.smithi114.stdout: 10 7608 17.54 MB/sec warmup 34 sec latency 352.192 ms 2024-05-27T07:54:22.727 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:22 smithi114 ceph-mon[32038]: pgmap v494: 129 pgs: 129 active+clean; 227 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 355 op/s 2024-05-27T07:54:22.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:22 smithi114 ceph-mon[32038]: pgmap v495: 129 pgs: 129 active+clean; 227 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.8 MiB/s wr, 365 op/s 2024-05-27T07:54:22.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:22 smithi163 ceph-mon[36451]: pgmap v494: 129 pgs: 129 active+clean; 227 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 355 op/s 2024-05-27T07:54:22.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:22 smithi163 ceph-mon[36451]: pgmap v495: 129 pgs: 129 active+clean; 227 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.8 MiB/s wr, 365 op/s 2024-05-27T07:54:22.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:22 smithi186 ceph-mon[36647]: pgmap v494: 129 pgs: 129 active+clean; 227 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 355 op/s 2024-05-27T07:54:22.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:22 smithi186 ceph-mon[36647]: pgmap v495: 129 pgs: 129 active+clean; 227 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.8 MiB/s wr, 365 op/s 2024-05-27T07:54:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.942+0000 7fbbe8c53700 1 -- 172.21.15.114:0/2229096919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbe4107e00 msgr2=0x7fbbe4108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.942+0000 7fbbe159a700 1 -- 172.21.15.114:0/2229096919 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd4012c60 con 0x7fbbe4107e00 2024-05-27T07:54:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.942+0000 7fbbe8c53700 1 --2- 172.21.15.114:0/2229096919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbe4107e00 0x7fbbe4108210 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4013ef0 tx=0x7fbbd40097f0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.944+0000 7fbbe8c53700 1 -- 172.21.15.114:0/2229096919 shutdown_connections 2024-05-27T07:54:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.944+0000 7fbbe8c53700 1 --2- 172.21.15.114:0/2229096919 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbe4108750 0x7fbbe406c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.944+0000 7fbbe8c53700 1 --2- 172.21.15.114:0/2229096919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbe41034d0 0x7fbbe41038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.944+0000 7fbbe8c53700 1 --2- 172.21.15.114:0/2229096919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbe4107e00 0x7fbbe4108210 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.944+0000 7fbbe8c53700 1 -- 172.21.15.114:0/2229096919 >> 172.21.15.114:0/2229096919 conn(0x7fbbe40fef20 msgr2=0x7fbbe4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.951+0000 7fbbe8c53700 1 -- 172.21.15.114:0/2229096919 shutdown_connections 2024-05-27T07:54:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.952+0000 7fbbe8c53700 1 -- 172.21.15.114:0/2229096919 wait complete. 2024-05-27T07:54:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.953+0000 7fbbe8c53700 1 Processor -- start 2024-05-27T07:54:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.953+0000 7fbbe8c53700 1 -- start start 2024-05-27T07:54:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.953+0000 7fbbe8c53700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbe41034d0 0x7fbbe41a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.953+0000 7fbbe8c53700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbe4107e00 0x7fbbe41aa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.953+0000 7fbbe8c53700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbe4108750 0x7fbbe41ae560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.953+0000 7fbbe8c53700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe4123450 con 0x7fbbe41034d0 2024-05-27T07:54:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.953+0000 7fbbe8c53700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe41232d0 con 0x7fbbe4108750 2024-05-27T07:54:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.953+0000 7fbbe8c53700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe41235d0 con 0x7fbbe4107e00 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.954+0000 7fbbe1d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbe4107e00 0x7fbbe41aa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.954+0000 7fbbe1d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbe4107e00 0x7fbbe41aa2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33626/0 (socket says 172.21.15.114:33626) 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.954+0000 7fbbe1d9b700 1 -- 172.21.15.114:0/3045421559 learned_addr learned my addr 172.21.15.114:0/3045421559 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.954+0000 7fbbe1d9b700 1 -- 172.21.15.114:0/3045421559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbe4108750 msgr2=0x7fbbe41ae560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.954+0000 7fbbe1d9b700 1 --2- 172.21.15.114:0/3045421559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbe4108750 0x7fbbe41ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.954+0000 7fbbe1d9b700 1 -- 172.21.15.114:0/3045421559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbe41034d0 msgr2=0x7fbbe41a9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.954+0000 7fbbe1d9b700 1 --2- 172.21.15.114:0/3045421559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbe41034d0 0x7fbbe41a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.954+0000 7fbbe1d9b700 1 -- 172.21.15.114:0/3045421559 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe41aec00 con 0x7fbbe4107e00 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.954+0000 7fbbe1d9b700 1 --2- 172.21.15.114:0/3045421559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbe4107e00 0x7fbbe41aa2d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4000c00 tx=0x7fbbd4004540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.958+0000 7fbbd37fe700 1 -- 172.21.15.114:0/3045421559 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd4013460 con 0x7fbbe4107e00 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.958+0000 7fbbd37fe700 1 -- 172.21.15.114:0/3045421559 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbd4005da0 con 0x7fbbe4107e00 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.958+0000 7fbbd37fe700 1 -- 172.21.15.114:0/3045421559 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd401b420 con 0x7fbbe4107e00 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.958+0000 7fbbe8c53700 1 -- 172.21.15.114:0/3045421559 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbe41aee90 con 0x7fbbe4107e00 2024-05-27T07:54:23.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.960+0000 7fbbe8c53700 1 -- 172.21.15.114:0/3045421559 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbe41af380 con 0x7fbbe4107e00 2024-05-27T07:54:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.971+0000 7fbbd37fe700 1 -- 172.21.15.114:0/3045421559 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fbbd4022020 con 0x7fbbe4107e00 2024-05-27T07:54:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.973+0000 7fbbf7668b80 1 Processor -- start 2024-05-27T07:54:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.978+0000 7fbbd37fe700 1 --2- 172.21.15.114:0/3045421559 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbbcc0b2bf0 0x7fbbcc0b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.979+0000 7fbbd37fe700 1 -- 172.21.15.114:0/3045421559 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fbbd40e33c0 con 0x7fbbe4107e00 2024-05-27T07:54:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.980+0000 7fbbf7668b80 1 -- start start 2024-05-27T07:54:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.980+0000 7fbbf7668b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad700abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.980+0000 7fbbf7668b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad700b0f0 0x556ad7004c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.980+0000 7fbbf7668b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad70051a0 0x556ad70055b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.980+0000 7fbbf7668b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556ad6f32df0 con 0x556ad700b0f0 2024-05-27T07:54:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.980+0000 7fbbf7668b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556ad6f22990 con 0x556ad70051a0 2024-05-27T07:54:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.980+0000 7fbbf7668b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556ad6f269d0 con 0x556ad6e85ec0 2024-05-27T07:54:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.981+0000 7fbbe1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad700b0f0 0x556ad7004c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.981+0000 7fbbe1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad700b0f0 0x556ad7004c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45446/0 (socket says 172.21.15.114:45446) 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.981+0000 7fbbe1d9b700 1 -- 172.21.15.114:0/233139607 learned_addr learned my addr 172.21.15.114:0/233139607 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.981+0000 7fbbe2d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad70051a0 0x556ad70055b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.981+0000 7fbbe2d9d700 1 --2- 172.21.15.114:0/3045421559 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbbcc0b2bf0 0x7fbbcc0b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.981+0000 7fbbe259c700 1 --2- 172.21.15.114:0/233139607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad700abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.985+0000 7fbbe2d9d700 1 -- 172.21.15.114:0/233139607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 msgr2=0x556ad700abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.985+0000 7fbbe2d9d700 1 --2- 172.21.15.114:0/233139607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad700abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.985+0000 7fbbe2d9d700 1 -- 172.21.15.114:0/233139607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad700b0f0 msgr2=0x556ad7004c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.985+0000 7fbbe2d9d700 1 --2- 172.21.15.114:0/233139607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad700b0f0 0x556ad7004c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.985+0000 7fbbe2d9d700 1 -- 172.21.15.114:0/233139607 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ad6ecc030 con 0x556ad70051a0 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.985+0000 7fbbe2d9d700 1 --2- 172.21.15.114:0/233139607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad70051a0 0x556ad70055b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800d650 tx=0x7fbbd800d960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49eddde0974d56d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.985+0000 7fbbe2d9d700 1 --2- 172.21.15.114:0/3045421559 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbbcc0b2bf0 0x7fbbcc0b5100 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800dcb0 tx=0x7fbbd8018000 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.986+0000 7fbbd0ff9700 1 -- 172.21.15.114:0/233139607 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8006600 con 0x556ad70051a0 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.986+0000 7fbbd0ff9700 1 -- 172.21.15.114:0/233139607 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbd80188b0 con 0x556ad70051a0 2024-05-27T07:54:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.986+0000 7fbbd0ff9700 1 -- 172.21.15.114:0/233139607 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd80175c0 con 0x556ad70051a0 2024-05-27T07:54:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.990+0000 7fbbf7668b80 1 -- 172.21.15.114:0/233139607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad70051a0 msgr2=0x556ad70055b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.990+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/233139607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad70051a0 0x556ad70055b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800d650 tx=0x7fbbd800d960 comp rx=0 tx=0).stop 2024-05-27T07:54:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.994+0000 7fbbf7668b80 1 -- 172.21.15.114:0/233139607 shutdown_connections 2024-05-27T07:54:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.994+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/233139607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad70051a0 0x556ad70055b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.994+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/233139607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad700abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.994+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/233139607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad700b0f0 0x556ad7004c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.994+0000 7fbbf7668b80 1 -- 172.21.15.114:0/233139607 >> 172.21.15.114:0/233139607 conn(0x556ad6e887f0 msgr2=0x556ad6fffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:22.998+0000 7fbbf7668b80 1 -- 172.21.15.114:0/233139607 shutdown_connections 2024-05-27T07:54:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.003+0000 7fbbf7668b80 1 -- 172.21.15.114:0/233139607 wait complete. 2024-05-27T07:54:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.003+0000 7fbbf7668b80 1 -- start start 2024-05-27T07:54:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.003+0000 7fbbf7668b80 1 Processor -- start 2024-05-27T07:54:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.006+0000 7fbbf7668b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad6e85ec0 0x556ad700e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.006+0000 7fbbf7668b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad70051a0 0x556ad700eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.006+0000 7fbbf7668b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad700b0f0 0x556ad700f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.006+0000 7fbbf7668b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556ad6e29020 con 0x556ad70051a0 2024-05-27T07:54:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.006+0000 7fbbf7668b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556ad6e8ae20 con 0x556ad6e85ec0 2024-05-27T07:54:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.006+0000 7fbbf7668b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556ad7011b80 con 0x556ad700b0f0 2024-05-27T07:54:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.006+0000 7fbbf7668b80 10 client.0 _ll_register_callbacks cb 0x556ad6eb9630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.007+0000 7fbbe2d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad700b0f0 0x556ad700f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.007+0000 7fbbe2d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad700b0f0 0x556ad700f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33636/0 (socket says 172.21.15.114:33636) 2024-05-27T07:54:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.007+0000 7fbbe2d9d700 1 -- 172.21.15.114:0/2283227224 learned_addr learned my addr 172.21.15.114:0/2283227224 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.009+0000 7fbbe2d9d700 1 -- 172.21.15.114:0/2283227224 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad6e85ec0 msgr2=0x556ad700e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.009+0000 7fbbe2d9d700 1 --2- 172.21.15.114:0/2283227224 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad6e85ec0 0x556ad700e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.009+0000 7fbbe2d9d700 1 -- 172.21.15.114:0/2283227224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad70051a0 msgr2=0x556ad700eec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:54:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.009+0000 7fbbe2d9d700 1 --2- 172.21.15.114:0/2283227224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad70051a0 0x556ad700eec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.009+0000 7fbbe2d9d700 1 -- 172.21.15.114:0/2283227224 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ad6e72950 con 0x556ad700b0f0 2024-05-27T07:54:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.012+0000 7fbbe2d9d700 1 --2- 172.21.15.114:0/2283227224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad700b0f0 0x556ad700f400 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800b4b0 tx=0x7fbbd8018750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a4737fc0a3e8ca3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.012+0000 7fbbc9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:54:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.014+0000 7fbbc9ffb700 1 -- 172.21.15.114:0/2283227224 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8018e30 con 0x556ad700b0f0 2024-05-27T07:54:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.014+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2283227224 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbd80188b0 con 0x556ad700b0f0 2024-05-27T07:54:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.014+0000 7fbbc9ffb700 1 -- 172.21.15.114:0/2283227224 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fbbd801d040 con 0x556ad700b0f0 2024-05-27T07:54:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.014+0000 7fbbc9ffb700 1 -- 172.21.15.114:0/2283227224 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fbbd8004020 con 0x556ad700b0f0 2024-05-27T07:54:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.016+0000 7fbbf7668b80 10 client.28450 fetch_fsmap learned FSMap version 79 2024-05-27T07:54:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.016+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2283227224 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x556ad6edc7e0 con 0x556ad700b0f0 2024-05-27T07:54:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.018+0000 7fbbc9ffb700 1 -- 172.21.15.114:0/2283227224 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbd80188b0 con 0x556ad700b0f0 2024-05-27T07:54:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.018+0000 7fbbc9ffb700 1 -- 172.21.15.114:0/2283227224 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 79) ==== 5305+0+0 (secure 0 0 0) 0x7fbbd8021460 con 0x556ad700b0f0 2024-05-27T07:54:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.018+0000 7fbbf7668b80 10 client.28450 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:54:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.018+0000 7fbbf7668b80 10 client.28450 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.018+0000 7fbbf7668b80 20 client.28450 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.018+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/2283227224 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ad701ed10 0x556ad70211b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.018+0000 7fbbf7668b80 4 client.28450 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.018+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2283227224 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ad6f34d40 con 0x556ad701ed10 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.020+0000 7fbbe259c700 1 --2- 172.21.15.114:0/2283227224 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ad701ed10 0x556ad70211b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.029+0000 7fbbe259c700 1 --2- 172.21.15.114:0/2283227224 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ad701ed10 0x556ad70211b0 crc :-1 s=READY pgs=416 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d49d600ffdb2174 server_cookie=f2ba7fef19641333 in_seq=0 out_seq=0 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.029+0000 7fbbc9ffb700 10 client.28450 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.031+0000 7fbbc9ffb700 1 -- 172.21.15.114:0/2283227224 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556ad6f34d40 con 0x556ad701ed10 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.031+0000 7fbbc9ffb700 10 client.28450 handle_command_reply: tid=0 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.031+0000 7fbbf7668b80 1 client.28450 shutdown 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.033+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2283227224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad700b0f0 msgr2=0x556ad700f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.033+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/2283227224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad700b0f0 0x556ad700f400 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800b4b0 tx=0x7fbbd8018750 comp rx=0 tx=0).stop 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.034+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2283227224 shutdown_connections 2024-05-27T07:54:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.034+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/2283227224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad700b0f0 0x556ad700f400 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.034+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/2283227224 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ad701ed10 0x556ad70211b0 crc :-1 s=READY pgs=416 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.034+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/2283227224 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad6e85ec0 0x556ad700e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.034+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/2283227224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad70051a0 0x556ad700eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.034+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2283227224 >> 172.21.15.114:0/2283227224 conn(0x556ad6e887f0 msgr2=0x556ad7003c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:23.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.034+0000 7fbbc9ffb700 0 client.28450 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:23.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.035+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2283227224 shutdown_connections 2024-05-27T07:54:23.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.035+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2283227224 wait complete. 2024-05-27T07:54:23.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.035+0000 7fbbf7668b80 20 client.28450 trim_cache size 0 max 16384 2024-05-27T07:54:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.077+0000 7fbbf7668b80 1 Processor -- start 2024-05-27T07:54:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.078+0000 7fbbf7668b80 1 -- start start 2024-05-27T07:54:23.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.078+0000 7fbbf7668b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad7004c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.078+0000 7fbbf7668b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad70051a0 0x556ad7010e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.078+0000 7fbbf7668b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad700b0f0 0x556ad7008f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.078+0000 7fbbe259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad7004c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:23.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.078+0000 7fbbe259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad7004c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33642/0 (socket says 172.21.15.114:33642) 2024-05-27T07:54:23.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.079+0000 7fbbe259c700 1 -- 172.21.15.114:0/1458008427 learned_addr learned my addr 172.21.15.114:0/1458008427 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:23.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.079+0000 7fbbf7668b80 1 -- 172.21.15.114:0/1458008427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe41c8950 con 0x556ad70051a0 2024-05-27T07:54:23.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.079+0000 7fbbe2d9d700 1 --2- 172.21.15.114:0/1458008427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad700b0f0 0x556ad7008f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:23.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.079+0000 7fbbf7668b80 1 -- 172.21.15.114:0/1458008427 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe41c9020 con 0x556ad700b0f0 2024-05-27T07:54:23.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.079+0000 7fbbf7668b80 1 -- 172.21.15.114:0/1458008427 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe41c7820 con 0x556ad6e85ec0 2024-05-27T07:54:23.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.079+0000 7fbbe1d9b700 1 --2- 172.21.15.114:0/1458008427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad70051a0 0x556ad7010e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:23.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.079+0000 7fbbe2d9d700 1 -- 172.21.15.114:0/1458008427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 msgr2=0x556ad7004c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:23.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.079+0000 7fbbe2d9d700 1 --2- 172.21.15.114:0/1458008427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad7004c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.079+0000 7fbbe2d9d700 1 -- 172.21.15.114:0/1458008427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad70051a0 msgr2=0x556ad7010e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:23.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.079+0000 7fbbe2d9d700 1 --2- 172.21.15.114:0/1458008427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad70051a0 0x556ad7010e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.080+0000 7fbbe2d9d700 1 -- 172.21.15.114:0/1458008427 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ad701eaf0 con 0x556ad700b0f0 2024-05-27T07:54:23.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.080+0000 7fbbe2d9d700 1 --2- 172.21.15.114:0/1458008427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad700b0f0 0x556ad7008f40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800d570 tx=0x7fbbd800d960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28eaf4158ed23bfc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:23.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.080+0000 7fbbcbfff700 1 -- 172.21.15.114:0/1458008427 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8018870 con 0x556ad700b0f0 2024-05-27T07:54:23.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.080+0000 7fbbcbfff700 1 -- 172.21.15.114:0/1458008427 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbd8037020 con 0x556ad700b0f0 2024-05-27T07:54:23.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.080+0000 7fbbcbfff700 1 -- 172.21.15.114:0/1458008427 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8021a90 con 0x556ad700b0f0 2024-05-27T07:54:23.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.082+0000 7fbbf7668b80 1 -- 172.21.15.114:0/1458008427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad700b0f0 msgr2=0x556ad7008f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:23.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.082+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/1458008427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad700b0f0 0x556ad7008f40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800d570 tx=0x7fbbd800d960 comp rx=0 tx=0).stop 2024-05-27T07:54:23.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.085+0000 7fbbf7668b80 1 -- 172.21.15.114:0/1458008427 shutdown_connections 2024-05-27T07:54:23.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.085+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/1458008427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad700b0f0 0x556ad7008f40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.085+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/1458008427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad7004c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.085+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/1458008427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad70051a0 0x556ad7010e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.085+0000 7fbbf7668b80 1 -- 172.21.15.114:0/1458008427 >> 172.21.15.114:0/1458008427 conn(0x556ad6e887f0 msgr2=0x556ad6ff9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:23.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.085+0000 7fbbf7668b80 1 -- 172.21.15.114:0/1458008427 shutdown_connections 2024-05-27T07:54:23.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.085+0000 7fbbf7668b80 1 -- 172.21.15.114:0/1458008427 wait complete. 2024-05-27T07:54:23.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.086+0000 7fbbf7668b80 1 -- start start 2024-05-27T07:54:23.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.086+0000 7fbbf7668b80 1 Processor -- start 2024-05-27T07:54:23.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.086+0000 7fbbf7668b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad7017210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.086+0000 7fbbf7668b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad70051a0 0x556ad7025f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.086+0000 7fbbf7668b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad700b0f0 0x556ad70264c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.086+0000 7fbbf7668b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe40752c0 con 0x556ad700b0f0 2024-05-27T07:54:23.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.086+0000 7fbbf7668b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe4073ed0 con 0x556ad70051a0 2024-05-27T07:54:23.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.086+0000 7fbbf7668b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe4074520 con 0x556ad6e85ec0 2024-05-27T07:54:23.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.086+0000 7fbbf7668b80 10 client.0 _ll_register_callbacks cb 0x556ad6eb9630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:23.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.087+0000 7fbbe259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad7017210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:23.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.087+0000 7fbbe259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad7017210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:33656/0 (socket says 172.21.15.114:33656) 2024-05-27T07:54:23.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.087+0000 7fbbe259c700 1 -- 172.21.15.114:0/2103659134 learned_addr learned my addr 172.21.15.114:0/2103659134 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:23.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.087+0000 7fbbe1d9b700 1 --2- 172.21.15.114:0/2103659134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad70051a0 0x556ad7025f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:23.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.087+0000 7fbbe259c700 1 -- 172.21.15.114:0/2103659134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad70051a0 msgr2=0x556ad7025f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:23.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.087+0000 7fbbe259c700 1 --2- 172.21.15.114:0/2103659134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad70051a0 0x556ad7025f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.087+0000 7fbbe259c700 1 -- 172.21.15.114:0/2103659134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad700b0f0 msgr2=0x556ad70264c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:54:23.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.087+0000 7fbbe259c700 1 --2- 172.21.15.114:0/2103659134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad700b0f0 0x556ad70264c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.087+0000 7fbbe259c700 1 -- 172.21.15.114:0/2103659134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ad7026a00 con 0x556ad6e85ec0 2024-05-27T07:54:23.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.089+0000 7fbbe259c700 1 --2- 172.21.15.114:0/2103659134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad7017210 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0158d0 tx=0x7fbbdc015900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=589556db869ecf69 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:23.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.089+0000 7fbbc9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:54:23.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.089+0000 7fbbc9ffb700 1 -- 172.21.15.114:0/2103659134 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc003ac0 con 0x556ad6e85ec0 2024-05-27T07:54:23.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.089+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2103659134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbd8037020 con 0x556ad6e85ec0 2024-05-27T07:54:23.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.089+0000 7fbbc9ffb700 1 -- 172.21.15.114:0/2103659134 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fbbdc003c60 con 0x556ad6e85ec0 2024-05-27T07:54:23.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.090+0000 7fbbc9ffb700 1 -- 172.21.15.114:0/2103659134 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fbbdc028050 con 0x556ad6e85ec0 2024-05-27T07:54:23.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.090+0000 7fbbf7668b80 10 client.28456 fetch_fsmap learned FSMap version 79 2024-05-27T07:54:23.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.090+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2103659134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x556ad70894f0 con 0x556ad6e85ec0 2024-05-27T07:54:23.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.090+0000 7fbbc9ffb700 1 -- 172.21.15.114:0/2103659134 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbd8037020 con 0x556ad6e85ec0 2024-05-27T07:54:23.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.090+0000 7fbbc9ffb700 1 -- 172.21.15.114:0/2103659134 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 79) ==== 5305+0+0 (secure 0 0 0) 0x7fbbd8006600 con 0x556ad6e85ec0 2024-05-27T07:54:23.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.090+0000 7fbbf7668b80 10 client.28456 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:54:23.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.090+0000 7fbbf7668b80 10 client.28456 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:23.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.090+0000 7fbbf7668b80 20 client.28456 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:23.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.090+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/2103659134 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ad7089c00 0x556ad708c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:23.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.090+0000 7fbbf7668b80 4 client.28456 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:54:23.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.090+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2103659134 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556ad6f2e520 con 0x556ad7089c00 2024-05-27T07:54:23.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.091+0000 7fbbe1d9b700 1 --2- 172.21.15.114:0/2103659134 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ad7089c00 0x556ad708c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:23.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.095+0000 7fbbe1d9b700 1 --2- 172.21.15.114:0/2103659134 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ad7089c00 0x556ad708c020 crc :-1 s=READY pgs=417 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=868a55b7adde93a server_cookie=ad3e6a9d049be747 in_seq=0 out_seq=0 2024-05-27T07:54:23.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.095+0000 7fbbc9ffb700 10 client.28456 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:23.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.107+0000 7fbbc9ffb700 1 -- 172.21.15.114:0/2103659134 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556ad6f2e520 con 0x556ad7089c00 2024-05-27T07:54:23.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.107+0000 7fbbc9ffb700 10 client.28456 handle_command_reply: tid=0 2024-05-27T07:54:23.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.107+0000 7fbbf7668b80 1 client.28456 shutdown 2024-05-27T07:54:23.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.107+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2103659134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 msgr2=0x556ad7017210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:23.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.107+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/2103659134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad7017210 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0158d0 tx=0x7fbbdc015900 comp rx=0 tx=0).stop 2024-05-27T07:54:23.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.107+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2103659134 shutdown_connections 2024-05-27T07:54:23.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.107+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/2103659134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ad700b0f0 0x556ad70264c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.107+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/2103659134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ad6e85ec0 0x556ad7017210 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.107+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/2103659134 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ad7089c00 0x556ad708c020 crc :-1 s=READY pgs=417 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.107+0000 7fbbf7668b80 1 --2- 172.21.15.114:0/2103659134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ad70051a0 0x556ad7025f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.107+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2103659134 >> 172.21.15.114:0/2103659134 conn(0x556ad6e887f0 msgr2=0x556ad7008ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:23.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.108+0000 7fbbc9ffb700 0 client.28456 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:23.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.108+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2103659134 shutdown_connections 2024-05-27T07:54:23.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.108+0000 7fbbf7668b80 1 -- 172.21.15.114:0/2103659134 wait complete. 2024-05-27T07:54:23.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.108+0000 7fbbf7668b80 20 client.28456 trim_cache size 0 max 16384 2024-05-27T07:54:23.110 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:54:23.110 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:54:23.111 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "2db79a72-e4b7-4b11-908c-d6b0cb869395", 2024-05-27T07:54:23.111 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:54:23.111 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:54:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.108+0000 7fbbe8c53700 1 -- 172.21.15.114:0/3045421559 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbbcc0b2bf0 msgr2=0x7fbbcc0b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.108+0000 7fbbe8c53700 1 --2- 172.21.15.114:0/3045421559 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbbcc0b2bf0 0x7fbbcc0b5100 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800dcb0 tx=0x7fbbd8018000 comp rx=0 tx=0).stop 2024-05-27T07:54:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.108+0000 7fbbe8c53700 1 -- 172.21.15.114:0/3045421559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbe4107e00 msgr2=0x7fbbe41aa2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.108+0000 7fbbe8c53700 1 --2- 172.21.15.114:0/3045421559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbe4107e00 0x7fbbe41aa2d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4000c00 tx=0x7fbbd4004540 comp rx=0 tx=0).stop 2024-05-27T07:54:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.109+0000 7fbbe8c53700 1 -- 172.21.15.114:0/3045421559 shutdown_connections 2024-05-27T07:54:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.109+0000 7fbbe8c53700 1 --2- 172.21.15.114:0/3045421559 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbbcc0b2bf0 0x7fbbcc0b5100 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.109+0000 7fbbe8c53700 1 --2- 172.21.15.114:0/3045421559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbe4108750 0x7fbbe41ae560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.109+0000 7fbbe8c53700 1 --2- 172.21.15.114:0/3045421559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbe41034d0 0x7fbbe41a9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.109+0000 7fbbe8c53700 1 --2- 172.21.15.114:0/3045421559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbe4107e00 0x7fbbe41aa2d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.109+0000 7fbbe8c53700 1 -- 172.21.15.114:0/3045421559 >> 172.21.15.114:0/3045421559 conn(0x7fbbe40fef20 msgr2=0x7fbbe4110b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.109+0000 7fbbe8c53700 1 -- 172.21.15.114:0/3045421559 shutdown_connections 2024-05-27T07:54:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:23.109+0000 7fbbe8c53700 1 -- 172.21.15.114:0/3045421559 wait complete. 2024-05-27T07:54:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa54f7fe640 1 -- 172.21.15.114:0/487043497 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56273a963360 con 0x56273a5e81c0 2024-05-27T07:54:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa54f7fe640 10 client.28444 handle_command_reply: tid=0 2024-05-27T07:54:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa57d24a740 1 client.28444 shutdown 2024-05-27T07:54:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa57d24a740 1 -- 172.21.15.114:0/487043497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a6334d0 msgr2=0x56273aa0aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa57d24a740 1 --2- 172.21.15.114:0/487043497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a6334d0 0x56273aa0aed0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa55800e9c0 tx=0x7fa5580aa130 comp rx=0 tx=0).stop 2024-05-27T07:54:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa57d24a740 1 -- 172.21.15.114:0/487043497 shutdown_connections 2024-05-27T07:54:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa57d24a740 1 --2- 172.21.15.114:0/487043497 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56273a5e81c0 0x56273aa965c0 crc :-1 s=READY pgs=415 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa57d24a740 1 --2- 172.21.15.114:0/487043497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56273a6541b0 0x56273aa0b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa57d24a740 1 --2- 172.21.15.114:0/487043497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56273a6334d0 0x56273aa0aed0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa57d24a740 1 --2- 172.21.15.114:0/487043497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56273a62b2f0 0x56273aa0a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa54f7fe640 0 client.28444 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa57d24a740 1 -- 172.21.15.114:0/487043497 >> 172.21.15.114:0/487043497 conn(0x56273a648300 msgr2=0x56273a9f5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa57d24a740 1 -- 172.21.15.114:0/487043497 shutdown_connections 2024-05-27T07:54:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa57d24a740 1 -- 172.21.15.114:0/487043497 wait complete. 2024-05-27T07:54:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.213+0000 7fa57d24a740 20 client.28444 trim_cache size 0 max 16384 2024-05-27T07:54:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:54:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:54:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:54:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.214+0000 7fa57638d640 1 -- 172.21.15.114:0/1154132129 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa55c0b2950 msgr2=0x7fa55c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.214+0000 7fa57638d640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa55c0b2950 0x7fa55c0b4e60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x56273a9f2570 tx=0x7fa568009500 comp rx=0 tx=0).stop 2024-05-27T07:54:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.214+0000 7fa57638d640 1 -- 172.21.15.114:0/1154132129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa570108b90 msgr2=0x7fa57019bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.214+0000 7fa57638d640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa570108b90 0x7fa57019bc60 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fa55800e9f0 tx=0x7fa55800eeb0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.214+0000 7fa57638d640 1 -- 172.21.15.114:0/1154132129 shutdown_connections 2024-05-27T07:54:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.214+0000 7fa57638d640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa55c0b2950 0x7fa55c0b4e60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.214+0000 7fa57638d640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa570109030 0x7fa57019c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.214+0000 7fa57638d640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa570108b90 0x7fa57019bc60 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.214+0000 7fa57638d640 1 --2- 172.21.15.114:0/1154132129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5701032e0 0x7fa57019b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:23.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.214+0000 7fa57638d640 1 -- 172.21.15.114:0/1154132129 >> 172.21.15.114:0/1154132129 conn(0x7fa5700782d0 msgr2=0x7fa5701008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:23.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.214+0000 7fa57638d640 1 -- 172.21.15.114:0/1154132129 shutdown_connections 2024-05-27T07:54:23.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:23.214+0000 7fa57638d640 1 -- 172.21.15.114:0/1154132129 wait complete. 2024-05-27T07:54:23.592 INFO:tasks.workunit.client.0.smithi114.stdout: 10 7759 17.17 MB/sec warmup 35 sec latency 608.910 ms 2024-05-27T07:54:23.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:23 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:54:23.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:23 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:23.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:23 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:54:23.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:23 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:54:23.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:23 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:23.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:23 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:54:23.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:23 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:54:23.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:23 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:23.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:23 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:54:24.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:54:24.408 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:54:24.592 INFO:tasks.workunit.client.0.smithi114.stdout: 10 8020 16.86 MB/sec warmup 36 sec latency 101.812 ms 2024-05-27T07:54:24.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:24 smithi114 ceph-mon[32038]: pgmap v496: 129 pgs: 129 active+clean; 244 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 275 op/s 2024-05-27T07:54:24.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:24 smithi114 ceph-mon[32038]: osdmap e138: 12 total, 12 up, 12 in 2024-05-27T07:54:24.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:24 smithi163 ceph-mon[36451]: pgmap v496: 129 pgs: 129 active+clean; 244 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 275 op/s 2024-05-27T07:54:24.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:24 smithi163 ceph-mon[36451]: osdmap e138: 12 total, 12 up, 12 in 2024-05-27T07:54:24.982 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:24 smithi186 ceph-mon[36647]: pgmap v496: 129 pgs: 129 active+clean; 244 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 275 op/s 2024-05-27T07:54:24.982 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:24 smithi186 ceph-mon[36647]: osdmap e138: 12 total, 12 up, 12 in 2024-05-27T07:54:25.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 8281 16.66 MB/sec warmup 37 sec latency 130.962 ms 2024-05-27T07:54:25.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:25 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:25.956 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:25 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:25.956 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:25 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:54:25.956 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:25 smithi163 systemd[1]: Stopped Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:54:25.956 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:25 smithi163 systemd[1]: Starting Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:54:25.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:25 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:26.226 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:25 smithi163 podman[74170]: 2024-05-27 07:54:25.858472433 +0000 UTC m=+0.019240102 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:54:26.227 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:25 smithi163 podman[74170]: 2024-05-27 07:54:25.97961256 +0000 UTC m=+0.140380219 container create 5ac410326d261981c0ca5abbc3399e714e72a85215f038849ffac44aba6ba608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:54:26.592 INFO:tasks.workunit.client.0.smithi114.stdout: 10 8555 16.60 MB/sec warmup 38 sec latency 190.875 ms 2024-05-27T07:54:26.644 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 podman[74170]: 2024-05-27 07:54:26.249387394 +0000 UTC m=+0.410155075 container init 5ac410326d261981c0ca5abbc3399e714e72a85215f038849ffac44aba6ba608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-27T07:54:26.644 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 podman[74170]: 2024-05-27 07:54:26.252965622 +0000 UTC m=+0.413733298 container start 5ac410326d261981c0ca5abbc3399e714e72a85215f038849ffac44aba6ba608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T07:54:26.644 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-mds[74187]: -- 172.21.15.163:0/2081809165 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c42182fa40 con 0x55c421877600 2024-05-27T07:54:26.645 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: starting mds.e at 2024-05-27T07:54:26.645 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 bash[74170]: 5ac410326d261981c0ca5abbc3399e714e72a85215f038849ffac44aba6ba608 2024-05-27T07:54:26.645 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 systemd[1]: Started Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:54:26.916 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:26 smithi163 ceph-mon[36451]: pgmap v498: 129 pgs: 129 active+clean; 261 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.7 MiB/s wr, 423 op/s 2024-05-27T07:54:26.916 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:26 smithi163 ceph-mon[36451]: daemon mds.e restarted 2024-05-27T07:54:26.917 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2e47417700 time 2024-05-27T07:54:26.856924+0000 2024-05-27T07:54:26.917 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:26.917 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:26.917 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2e59c5cba6] 2024-05-27T07:54:26.917 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2e59c5cdd1] 2024-05-27T07:54:26.917 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c41f7dc23d] 2024-05-27T07:54:26.917 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c41f7c7971] 2024-05-27T07:54:26.917 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c41f7c5fa0] 2024-05-27T07:54:26.917 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c41f7c6651] 2024-05-27T07:54:26.917 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c41f75b347] 2024-05-27T07:54:26.917 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c41f45787e] 2024-05-27T07:54:26.918 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c41f45cf2f] 2024-05-27T07:54:26.918 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c41f467969] 2024-05-27T07:54:26.918 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c41f80048c] 2024-05-27T07:54:26.918 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c41f8099cb] 2024-05-27T07:54:26.918 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 13: (Thread::entry_wrapper()+0x43) [0x7f2e59c2b72d] 2024-05-27T07:54:26.918 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2e59c2b749] 2024-05-27T07:54:26.918 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2e587cf1ca] 2024-05-27T07:54:26.918 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 16: clone() 2024-05-27T07:54:26.918 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.858+0000 7f2e47417700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2e47417700 time 2024-05-27T07:54:26.856924+0000 2024-05-27T07:54:26.918 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:26.918 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.918 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2e59c5cba6] 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2e59c5cdd1] 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c41f7dc23d] 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c41f7c7971] 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c41f7c5fa0] 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c41f7c6651] 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c41f75b347] 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c41f45787e] 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c41f45cf2f] 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c41f467969] 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c41f80048c] 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c41f8099cb] 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 13: (Thread::entry_wrapper()+0x43) [0x7f2e59c2b72d] 2024-05-27T07:54:26.919 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2e59c2b749] 2024-05-27T07:54:26.920 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2e587cf1ca] 2024-05-27T07:54:26.920 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 16: clone() 2024-05-27T07:54:26.920 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.920 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: *** Caught signal (Aborted) ** 2024-05-27T07:54:26.920 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: in thread 7f2e47417700 thread_name:md_log_replay 2024-05-27T07:54:26.920 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:26.920 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2e587d9d20] 2024-05-27T07:54:26.920 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2: gsignal() 2024-05-27T07:54:26.920 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 3: abort() 2024-05-27T07:54:26.920 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2e59c5cced] 2024-05-27T07:54:26.920 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2e59c5cdd1] 2024-05-27T07:54:26.921 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c41f7dc23d] 2024-05-27T07:54:26.921 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c41f7c7971] 2024-05-27T07:54:26.921 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c41f7c5fa0] 2024-05-27T07:54:26.921 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c41f7c6651] 2024-05-27T07:54:26.921 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c41f75b347] 2024-05-27T07:54:26.922 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c41f45787e] 2024-05-27T07:54:26.922 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c41f45cf2f] 2024-05-27T07:54:26.922 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c41f467969] 2024-05-27T07:54:26.922 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c41f80048c] 2024-05-27T07:54:26.922 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c41f8099cb] 2024-05-27T07:54:26.922 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 16: (Thread::entry_wrapper()+0x43) [0x7f2e59c2b72d] 2024-05-27T07:54:26.922 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2e59c2b749] 2024-05-27T07:54:26.922 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2e587cf1ca] 2024-05-27T07:54:26.922 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 19: clone() 2024-05-27T07:54:26.923 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.860+0000 7f2e47417700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:54:26.923 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: in thread 7f2e47417700 thread_name:md_log_replay 2024-05-27T07:54:26.923 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.923 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:26.923 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2e587d9d20] 2024-05-27T07:54:26.923 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2: gsignal() 2024-05-27T07:54:26.923 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 3: abort() 2024-05-27T07:54:26.923 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2e59c5cced] 2024-05-27T07:54:26.923 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2e59c5cdd1] 2024-05-27T07:54:26.923 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c41f7dc23d] 2024-05-27T07:54:26.923 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c41f7c7971] 2024-05-27T07:54:26.924 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c41f7c5fa0] 2024-05-27T07:54:26.924 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c41f7c6651] 2024-05-27T07:54:26.924 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c41f75b347] 2024-05-27T07:54:26.924 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c41f45787e] 2024-05-27T07:54:26.924 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c41f45cf2f] 2024-05-27T07:54:26.924 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c41f467969] 2024-05-27T07:54:26.924 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c41f80048c] 2024-05-27T07:54:26.924 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c41f8099cb] 2024-05-27T07:54:26.924 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 16: (Thread::entry_wrapper()+0x43) [0x7f2e59c2b72d] 2024-05-27T07:54:26.924 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2e59c2b749] 2024-05-27T07:54:26.924 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2e587cf1ca] 2024-05-27T07:54:26.924 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 19: clone() 2024-05-27T07:54:26.925 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:54:26.925 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.925 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: -1> 2024-05-27T07:54:26.858+0000 7f2e47417700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2e47417700 time 2024-05-27T07:54:26.856924+0000 2024-05-27T07:54:26.925 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:26.925 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.925 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:26.925 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2e59c5cba6] 2024-05-27T07:54:26.925 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2e59c5cdd1] 2024-05-27T07:54:26.925 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c41f7dc23d] 2024-05-27T07:54:26.925 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c41f7c7971] 2024-05-27T07:54:26.925 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c41f7c5fa0] 2024-05-27T07:54:26.925 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c41f7c6651] 2024-05-27T07:54:26.926 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c41f75b347] 2024-05-27T07:54:26.926 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c41f45787e] 2024-05-27T07:54:26.926 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c41f45cf2f] 2024-05-27T07:54:26.926 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c41f467969] 2024-05-27T07:54:26.926 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c41f80048c] 2024-05-27T07:54:26.926 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c41f8099cb] 2024-05-27T07:54:26.926 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 13: (Thread::entry_wrapper()+0x43) [0x7f2e59c2b72d] 2024-05-27T07:54:26.926 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2e59c2b749] 2024-05-27T07:54:26.926 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2e587cf1ca] 2024-05-27T07:54:26.926 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 16: clone() 2024-05-27T07:54:26.926 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.927 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 0> 2024-05-27T07:54:26.860+0000 7f2e47417700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:54:26.927 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: in thread 7f2e47417700 thread_name:md_log_replay 2024-05-27T07:54:26.927 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.927 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:26.927 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2e587d9d20] 2024-05-27T07:54:26.927 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2: gsignal() 2024-05-27T07:54:26.927 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 3: abort() 2024-05-27T07:54:26.927 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2e59c5cced] 2024-05-27T07:54:26.927 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2e59c5cdd1] 2024-05-27T07:54:26.927 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c41f7dc23d] 2024-05-27T07:54:26.927 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c41f7c7971] 2024-05-27T07:54:26.928 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c41f7c5fa0] 2024-05-27T07:54:26.928 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c41f7c6651] 2024-05-27T07:54:26.928 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c41f75b347] 2024-05-27T07:54:26.928 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c41f45787e] 2024-05-27T07:54:26.928 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c41f45cf2f] 2024-05-27T07:54:26.928 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c41f467969] 2024-05-27T07:54:26.928 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c41f80048c] 2024-05-27T07:54:26.928 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c41f8099cb] 2024-05-27T07:54:26.928 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 16: (Thread::entry_wrapper()+0x43) [0x7f2e59c2b72d] 2024-05-27T07:54:26.928 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2e59c2b749] 2024-05-27T07:54:26.928 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2e587cf1ca] 2024-05-27T07:54:26.928 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 19: clone() 2024-05-27T07:54:26.929 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:54:26.929 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.929 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: -1> 2024-05-27T07:54:26.858+0000 7f2e47417700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2e47417700 time 2024-05-27T07:54:26.856924+0000 2024-05-27T07:54:26.929 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:26.929 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.929 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:26.929 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2e59c5cba6] 2024-05-27T07:54:26.929 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2e59c5cdd1] 2024-05-27T07:54:26.929 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c41f7dc23d] 2024-05-27T07:54:26.929 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c41f7c7971] 2024-05-27T07:54:26.929 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c41f7c5fa0] 2024-05-27T07:54:26.929 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c41f7c6651] 2024-05-27T07:54:26.930 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c41f75b347] 2024-05-27T07:54:26.930 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c41f45787e] 2024-05-27T07:54:26.930 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c41f45cf2f] 2024-05-27T07:54:26.930 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c41f467969] 2024-05-27T07:54:26.930 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c41f80048c] 2024-05-27T07:54:26.930 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c41f8099cb] 2024-05-27T07:54:26.930 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 13: (Thread::entry_wrapper()+0x43) [0x7f2e59c2b72d] 2024-05-27T07:54:26.930 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2e59c2b749] 2024-05-27T07:54:26.930 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2e587cf1ca] 2024-05-27T07:54:26.930 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 16: clone() 2024-05-27T07:54:26.930 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.930 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 0> 2024-05-27T07:54:26.860+0000 7f2e47417700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:54:26.931 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: in thread 7f2e47417700 thread_name:md_log_replay 2024-05-27T07:54:26.931 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.931 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:26.931 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2e587d9d20] 2024-05-27T07:54:26.931 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2: gsignal() 2024-05-27T07:54:26.931 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 3: abort() 2024-05-27T07:54:26.931 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2e59c5cced] 2024-05-27T07:54:26.931 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2e59c5cdd1] 2024-05-27T07:54:26.931 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c41f7dc23d] 2024-05-27T07:54:26.931 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c41f7c7971] 2024-05-27T07:54:26.931 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c41f7c5fa0] 2024-05-27T07:54:26.931 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c41f7c6651] 2024-05-27T07:54:26.932 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c41f75b347] 2024-05-27T07:54:26.932 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c41f45787e] 2024-05-27T07:54:26.932 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c41f45cf2f] 2024-05-27T07:54:26.932 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c41f467969] 2024-05-27T07:54:26.932 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c41f80048c] 2024-05-27T07:54:26.932 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c41f8099cb] 2024-05-27T07:54:26.932 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 16: (Thread::entry_wrapper()+0x43) [0x7f2e59c2b72d] 2024-05-27T07:54:26.932 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2e59c2b749] 2024-05-27T07:54:26.932 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2e587cf1ca] 2024-05-27T07:54:26.932 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 19: clone() 2024-05-27T07:54:26.932 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:54:26.932 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74183]: 2024-05-27T07:54:26.933 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:26 smithi163 podman[74218]: 2024-05-27 07:54:26.915829977 +0000 UTC m=+0.017727997 container died 5ac410326d261981c0ca5abbc3399e714e72a85215f038849ffac44aba6ba608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True) 2024-05-27T07:54:26.933 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:26 smithi114 ceph-mon[32038]: pgmap v498: 129 pgs: 129 active+clean; 261 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.7 MiB/s wr, 423 op/s 2024-05-27T07:54:26.933 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:26 smithi114 ceph-mon[32038]: daemon mds.e restarted 2024-05-27T07:54:26.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:26 smithi186 ceph-mon[36647]: pgmap v498: 129 pgs: 129 active+clean; 261 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.7 MiB/s wr, 423 op/s 2024-05-27T07:54:26.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:26 smithi186 ceph-mon[36647]: daemon mds.e restarted 2024-05-27T07:54:27.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.400+0000 7fa2f2922700 1 -- 172.21.15.114:0/3715612688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec10abb0 msgr2=0x7fa2ec10afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:27.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.400+0000 7fa2f2922700 1 --2- 172.21.15.114:0/3715612688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec10abb0 0x7fa2ec10afc0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400b0a0 tx=0x7fa2e400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.402+0000 7fa2f2922700 1 -- 172.21.15.114:0/3715612688 shutdown_connections 2024-05-27T07:54:27.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.402+0000 7fa2f2922700 1 --2- 172.21.15.114:0/3715612688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2ec1192b0 0x7fa2ec11b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.402+0000 7fa2f2922700 1 --2- 172.21.15.114:0/3715612688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec10abb0 0x7fa2ec10afc0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.402+0000 7fa2f2922700 1 --2- 172.21.15.114:0/3715612688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ec10b500 0x7fa2ec118c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.402+0000 7fa2f2922700 1 -- 172.21.15.114:0/3715612688 >> 172.21.15.114:0/3715612688 conn(0x7fa2ec06f4c0 msgr2=0x7fa2ec06f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:27.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.405+0000 7fa2f2922700 1 -- 172.21.15.114:0/3715612688 shutdown_connections 2024-05-27T07:54:27.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.405+0000 7fa2f2922700 1 -- 172.21.15.114:0/3715612688 wait complete. 2024-05-27T07:54:27.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.406+0000 7fa2f2922700 1 Processor -- start 2024-05-27T07:54:27.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.407+0000 7fa2f2922700 1 -- start start 2024-05-27T07:54:27.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.407+0000 7fa2f2922700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec10abb0 0x7fa2ec117920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.407+0000 7fa2f2922700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ec10b500 0x7fa2ec117e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.408+0000 7fa2f2922700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2ec1192b0 0x7fa2ec1ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.408+0000 7fa2f2922700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec10f9b0 con 0x7fa2ec10abb0 2024-05-27T07:54:27.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.408+0000 7fa2f2922700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec10f830 con 0x7fa2ec1192b0 2024-05-27T07:54:27.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.408+0000 7fa2f2922700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec10fb30 con 0x7fa2ec10b500 2024-05-27T07:54:27.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.409+0000 7fa2eb7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ec10b500 0x7fa2ec117e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.410+0000 7fa2eb7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ec10b500 0x7fa2ec117e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36214/0 (socket says 172.21.15.114:36214) 2024-05-27T07:54:27.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.410+0000 7fa2eb7fe700 1 -- 172.21.15.114:0/4201626768 learned_addr learned my addr 172.21.15.114:0/4201626768 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:27.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.410+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2ec1192b0 0x7fa2ec1ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.412+0000 7fa2ebfff700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec10abb0 0x7fa2ec117920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.412+0000 7fa2eb7fe700 1 -- 172.21.15.114:0/4201626768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2ec1192b0 msgr2=0x7fa2ec1ae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:27.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.412+0000 7fa2eb7fe700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2ec1192b0 0x7fa2ec1ae2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.412+0000 7fa2eb7fe700 1 -- 172.21.15.114:0/4201626768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec10abb0 msgr2=0x7fa2ec117920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:27.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.412+0000 7fa2eb7fe700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec10abb0 0x7fa2ec117920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.412+0000 7fa2eb7fe700 1 -- 172.21.15.114:0/4201626768 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ec1ae950 con 0x7fa2ec10b500 2024-05-27T07:54:27.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.419+0000 7fa2eb7fe700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ec10b500 0x7fa2ec117e80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00e970 tx=0x7fa2dc00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:27.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.419+0000 7fa2e97fa700 1 -- 172.21.15.114:0/4201626768 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc00cbd0 con 0x7fa2ec10b500 2024-05-27T07:54:27.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.419+0000 7fa2e97fa700 1 -- 172.21.15.114:0/4201626768 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2dc00cd70 con 0x7fa2ec10b500 2024-05-27T07:54:27.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.419+0000 7fa2e97fa700 1 -- 172.21.15.114:0/4201626768 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc010430 con 0x7fa2ec10b500 2024-05-27T07:54:27.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.419+0000 7fa2f2922700 1 -- 172.21.15.114:0/4201626768 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2ec1aec40 con 0x7fa2ec10b500 2024-05-27T07:54:27.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.419+0000 7fa2f2922700 1 -- 172.21.15.114:0/4201626768 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2ec1b6590 con 0x7fa2ec10b500 2024-05-27T07:54:27.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.425+0000 7fa301337b80 1 Processor -- start 2024-05-27T07:54:27.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.425+0000 7fa301337b80 1 -- start start 2024-05-27T07:54:27.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.426+0000 7fa301337b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 0x55cf5c09a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.426+0000 7fa301337b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c099680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.426+0000 7fa301337b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5c099bc0 0x55cf5c099fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.426+0000 7fa301337b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cf5bfc2d70 con 0x55cf5c09af00 2024-05-27T07:54:27.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.426+0000 7fa301337b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55cf5bfb2910 con 0x55cf5c099bc0 2024-05-27T07:54:27.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.426+0000 7fa301337b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cf5bfb6950 con 0x55cf5bf15e40 2024-05-27T07:54:27.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.428+0000 7fa2f0ebf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 0x55cf5c09a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.429+0000 7fa2f0ebf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 0x55cf5c09a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36224/0 (socket says 172.21.15.114:36224) 2024-05-27T07:54:27.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.429+0000 7fa2f0ebf700 1 -- 172.21.15.114:0/3455965316 learned_addr learned my addr 172.21.15.114:0/3455965316 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:27.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.429+0000 7fa2e97fa700 1 -- 172.21.15.114:0/4201626768 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa2dc004d70 con 0x7fa2ec10b500 2024-05-27T07:54:27.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.430+0000 7fa2e97fa700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa2d40b2c00 0x7fa2d40b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.431+0000 7fa2e97fa700 1 -- 172.21.15.114:0/4201626768 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9466+0+0 (secure 0 0 0) 0x7fa2dc0e2c30 con 0x7fa2ec10b500 2024-05-27T07:54:27.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.431+0000 7fa2eb7fe700 1 --2- 172.21.15.114:0/3455965316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5c099bc0 0x55cf5c099fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.431+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa2d40b2c00 0x7fa2d40b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.432+0000 7fa2f0ebf700 1 -- 172.21.15.114:0/3455965316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5c099bc0 msgr2=0x55cf5c099fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:27.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.432+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/3455965316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5c099bc0 0x55cf5c099fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.432+0000 7fa2f0ebf700 1 -- 172.21.15.114:0/3455965316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 msgr2=0x55cf5c099680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:27.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.432+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/3455965316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c099680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.432+0000 7fa2f0ebf700 1 -- 172.21.15.114:0/3455965316 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf5bf5bfb0 con 0x55cf5bf15e40 2024-05-27T07:54:27.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.433+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/3455965316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 0x55cf5c09a9c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0016880 tx=0x7fa2e0016b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aca08b490db15c5d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:27.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.433+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa2d40b2c00 0x7fa2d40b5110 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa2e001a000 tx=0x7fa2e001d000 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:27.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.433+0000 7fa2f2922700 1 -- 172.21.15.114:0/3455965316 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0022660 con 0x55cf5bf15e40 2024-05-27T07:54:27.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.433+0000 7fa2f2922700 1 -- 172.21.15.114:0/3455965316 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2e0010040 con 0x55cf5bf15e40 2024-05-27T07:54:27.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.435+0000 7fa301337b80 1 -- 172.21.15.114:0/3455965316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 msgr2=0x55cf5c09a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:27.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.435+0000 7fa301337b80 1 --2- 172.21.15.114:0/3455965316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 0x55cf5c09a9c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0016880 tx=0x7fa2e0016b90 comp rx=0 tx=0).stop 2024-05-27T07:54:27.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.439+0000 7fa301337b80 1 -- 172.21.15.114:0/3455965316 shutdown_connections 2024-05-27T07:54:27.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.439+0000 7fa301337b80 1 --2- 172.21.15.114:0/3455965316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5c099bc0 0x55cf5c099fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.439+0000 7fa301337b80 1 --2- 172.21.15.114:0/3455965316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 0x55cf5c09a9c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.439+0000 7fa301337b80 1 --2- 172.21.15.114:0/3455965316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c099680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.439+0000 7fa301337b80 1 -- 172.21.15.114:0/3455965316 >> 172.21.15.114:0/3455965316 conn(0x55cf5bf18770 msgr2=0x55cf5c090240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:27.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.443+0000 7fa301337b80 1 -- 172.21.15.114:0/3455965316 shutdown_connections 2024-05-27T07:54:27.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.444+0000 7fa301337b80 1 -- 172.21.15.114:0/3455965316 wait complete. 2024-05-27T07:54:27.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.444+0000 7fa301337b80 1 -- start start 2024-05-27T07:54:27.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.445+0000 7fa301337b80 1 Processor -- start 2024-05-27T07:54:27.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.445+0000 7fa301337b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5bf15e40 0x55cf5c09e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.445+0000 7fa301337b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5c099bc0 0x55cf5c09eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.445+0000 7fa301337b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c09f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.445+0000 7fa301337b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cf5beb8fa0 con 0x55cf5c09af00 2024-05-27T07:54:27.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.445+0000 7fa301337b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55cf5bf1ada0 con 0x55cf5bf15e40 2024-05-27T07:54:27.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.445+0000 7fa301337b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cf5c0b2a30 con 0x55cf5c099bc0 2024-05-27T07:54:27.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.445+0000 7fa301337b80 10 client.0 _ll_register_callbacks cb 0x55cf5bf495b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:27.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.447+0000 7fa2f0ebf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c09f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.447+0000 7fa2f0ebf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c09f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59808/0 (socket says 172.21.15.114:59808) 2024-05-27T07:54:27.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.447+0000 7fa2f0ebf700 1 -- 172.21.15.114:0/3344507974 learned_addr learned my addr 172.21.15.114:0/3344507974 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:27.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.452+0000 7fa2f0ebf700 1 -- 172.21.15.114:0/3344507974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5c099bc0 msgr2=0x55cf5c09eeb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:54:27.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.452+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/3344507974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5c099bc0 0x55cf5c09eeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.452+0000 7fa2f0ebf700 1 -- 172.21.15.114:0/3344507974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5bf15e40 msgr2=0x55cf5c09e970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:54:27.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.452+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/3344507974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5bf15e40 0x55cf5c09e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.452+0000 7fa2f0ebf700 1 -- 172.21.15.114:0/3344507974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf5bf028d0 con 0x55cf5c09af00 2024-05-27T07:54:27.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.453+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/3344507974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c09f3f0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0018e50 tx=0x7fa2e0018f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b41adb0fcde25c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:27.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.454+0000 7fa2cb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:54:27.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.455+0000 7fa2cb7fe700 1 -- 172.21.15.114:0/3344507974 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e002e070 con 0x55cf5c09af00 2024-05-27T07:54:27.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.455+0000 7fa301337b80 1 -- 172.21.15.114:0/3344507974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2e0010040 con 0x55cf5c09af00 2024-05-27T07:54:27.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.455+0000 7fa2cb7fe700 1 -- 172.21.15.114:0/3344507974 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9466+0+0 (secure 0 0 0) 0x7fa2e0022870 con 0x55cf5c09af00 2024-05-27T07:54:27.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.456+0000 7fa2cb7fe700 1 -- 172.21.15.114:0/3344507974 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fa2e003a050 con 0x55cf5c09af00 2024-05-27T07:54:27.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.461+0000 7fa301337b80 10 client.18963 fetch_fsmap learned FSMap version 81 2024-05-27T07:54:27.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.461+0000 7fa301337b80 1 -- 172.21.15.114:0/3344507974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=4+}) -- 0x55cf5bf6c760 con 0x55cf5c09af00 2024-05-27T07:54:27.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.462+0000 7fa2cb7fe700 1 -- 172.21.15.114:0/3344507974 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2e0010040 con 0x55cf5c09af00 2024-05-27T07:54:27.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.463+0000 7fa2cb7fe700 1 -- 172.21.15.114:0/3344507974 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 81) ==== 5305+0+0 (secure 0 0 0) 0x7fa2e0034960 con 0x55cf5c09af00 2024-05-27T07:54:27.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.463+0000 7fa301337b80 10 client.18963 fetch_fsmap finished waiting for FSMap version 81 2024-05-27T07:54:27.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.463+0000 7fa301337b80 10 client.18963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:27.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.463+0000 7fa301337b80 20 client.18963 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:27.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.463+0000 7fa301337b80 1 --2- 172.21.15.114:0/3344507974 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cf5c0bfe40 0x55cf5c0c2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.463+0000 7fa301337b80 4 client.18963 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:54:27.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.463+0000 7fa301337b80 1 -- 172.21.15.114:0/3344507974 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf5bfc4cc0 con 0x55cf5c0bfe40 2024-05-27T07:54:27.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.464+0000 7fa2ebfff700 1 --2- 172.21.15.114:0/3344507974 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cf5c0bfe40 0x55cf5c0c2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.477+0000 7fa2ebfff700 1 --2- 172.21.15.114:0/3344507974 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cf5c0bfe40 0x55cf5c0c2300 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23446b37b4a50c04 server_cookie=258af79f76e2dc6a in_seq=0 out_seq=0 2024-05-27T07:54:27.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.477+0000 7fa2cb7fe700 10 client.18963 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:27.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.484+0000 7fa2cb7fe700 1 -- 172.21.15.114:0/3344507974 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cf5bfc4cc0 con 0x55cf5c0bfe40 2024-05-27T07:54:27.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.484+0000 7fa2cb7fe700 10 client.18963 handle_command_reply: tid=0 2024-05-27T07:54:27.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.484+0000 7fa301337b80 1 client.18963 shutdown 2024-05-27T07:54:27.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.484+0000 7fa301337b80 1 -- 172.21.15.114:0/3344507974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 msgr2=0x55cf5c09f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:27.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.484+0000 7fa301337b80 1 --2- 172.21.15.114:0/3344507974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c09f3f0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0018e50 tx=0x7fa2e0018f30 comp rx=0 tx=0).stop 2024-05-27T07:54:27.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.486+0000 7fa301337b80 1 -- 172.21.15.114:0/3344507974 shutdown_connections 2024-05-27T07:54:27.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.486+0000 7fa301337b80 1 --2- 172.21.15.114:0/3344507974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c09f3f0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.486+0000 7fa301337b80 1 --2- 172.21.15.114:0/3344507974 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cf5c0bfe40 0x55cf5c0c2300 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.486+0000 7fa301337b80 1 --2- 172.21.15.114:0/3344507974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5bf15e40 0x55cf5c09e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.486+0000 7fa301337b80 1 --2- 172.21.15.114:0/3344507974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5c099bc0 0x55cf5c09eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.486+0000 7fa301337b80 1 -- 172.21.15.114:0/3344507974 >> 172.21.15.114:0/3344507974 conn(0x55cf5bf18770 msgr2=0x55cf5c093bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:27.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.486+0000 7fa2cb7fe700 0 client.18963 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:27.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.490+0000 7fa301337b80 1 -- 172.21.15.114:0/3344507974 shutdown_connections 2024-05-27T07:54:27.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.490+0000 7fa301337b80 1 -- 172.21.15.114:0/3344507974 wait complete. 2024-05-27T07:54:27.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.491+0000 7fa301337b80 20 client.18963 trim_cache size 0 max 16384 2024-05-27T07:54:27.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.525+0000 7fa301337b80 1 Processor -- start 2024-05-27T07:54:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.525+0000 7fa301337b80 1 -- start start 2024-05-27T07:54:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.525+0000 7fa301337b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5bf15e40 0x55cf5c09f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.525+0000 7fa301337b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5c099bc0 0x55cf5c0bbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.525+0000 7fa301337b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c09d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.525+0000 7fa301337b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec1b76e0 con 0x55cf5c09af00 2024-05-27T07:54:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.525+0000 7fa301337b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec1b7db0 con 0x55cf5bf15e40 2024-05-27T07:54:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.525+0000 7fa301337b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec126380 con 0x55cf5c099bc0 2024-05-27T07:54:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.525+0000 7fa2eb7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5c099bc0 0x55cf5c0bbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.525+0000 7fa2eb7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5c099bc0 0x55cf5c0bbc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36244/0 (socket says 172.21.15.114:36244) 2024-05-27T07:54:27.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.525+0000 7fa2eb7fe700 1 -- 172.21.15.114:0/2696609227 learned_addr learned my addr 172.21.15.114:0/2696609227 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.526+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/2696609227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c09d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.526+0000 7fa2f0ebf700 1 -- 172.21.15.114:0/2696609227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5c099bc0 msgr2=0x55cf5c0bbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.526+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/2696609227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5c099bc0 0x55cf5c0bbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.526+0000 7fa2f0ebf700 1 -- 172.21.15.114:0/2696609227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5bf15e40 msgr2=0x55cf5c09f230 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:54:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.526+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/2696609227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5bf15e40 0x55cf5c09f230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.526+0000 7fa2f0ebf700 1 -- 172.21.15.114:0/2696609227 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf5c0916c0 con 0x55cf5c09af00 2024-05-27T07:54:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.526+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/2696609227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c09d260 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000b190 tx=0x7fa2e0016b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2800eed20aef98 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:27.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.527+0000 7fa2d1ffb700 1 -- 172.21.15.114:0/2696609227 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e002e040 con 0x55cf5c09af00 2024-05-27T07:54:27.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.527+0000 7fa2d1ffb700 1 -- 172.21.15.114:0/2696609227 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2e002a900 con 0x55cf5c09af00 2024-05-27T07:54:27.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.527+0000 7fa2d1ffb700 1 -- 172.21.15.114:0/2696609227 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0026040 con 0x55cf5c09af00 2024-05-27T07:54:27.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.528+0000 7fa301337b80 1 -- 172.21.15.114:0/2696609227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 msgr2=0x55cf5c09d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:27.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.528+0000 7fa301337b80 1 --2- 172.21.15.114:0/2696609227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c09d260 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000b190 tx=0x7fa2e0016b90 comp rx=0 tx=0).stop 2024-05-27T07:54:27.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.528+0000 7fa301337b80 1 -- 172.21.15.114:0/2696609227 shutdown_connections 2024-05-27T07:54:27.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.528+0000 7fa301337b80 1 --2- 172.21.15.114:0/2696609227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c09af00 0x55cf5c09d260 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.528+0000 7fa301337b80 1 --2- 172.21.15.114:0/2696609227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5bf15e40 0x55cf5c09f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.528+0000 7fa301337b80 1 --2- 172.21.15.114:0/2696609227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5c099bc0 0x55cf5c0bbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.529+0000 7fa301337b80 1 -- 172.21.15.114:0/2696609227 >> 172.21.15.114:0/2696609227 conn(0x55cf5bf18770 msgr2=0x55cf5c090150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:27.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.529+0000 7fa301337b80 1 -- 172.21.15.114:0/2696609227 shutdown_connections 2024-05-27T07:54:27.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.529+0000 7fa301337b80 1 -- 172.21.15.114:0/2696609227 wait complete. 2024-05-27T07:54:27.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.530+0000 7fa301337b80 1 -- start start 2024-05-27T07:54:27.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.530+0000 7fa301337b80 1 Processor -- start 2024-05-27T07:54:27.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.531+0000 7fa301337b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 0x55cf5c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.531+0000 7fa301337b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c099bc0 0x55cf5c0b5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.531+0000 7fa2ebfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 0x55cf5c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.531+0000 7fa2ebfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 0x55cf5c0b4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36258/0 (socket says 172.21.15.114:36258) 2024-05-27T07:54:27.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.531+0000 7fa2ebfff700 1 -- 172.21.15.114:0/1672918122 learned_addr learned my addr 172.21.15.114:0/1672918122 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:27.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.531+0000 7fa301337b80 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5c09af00 0x55cf5c0b5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.531+0000 7fa2eb7fe700 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c099bc0 0x55cf5c0b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.531+0000 7fa301337b80 1 -- 172.21.15.114:0/1672918122 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec125cb0 con 0x55cf5c099bc0 2024-05-27T07:54:27.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.531+0000 7fa301337b80 1 -- 172.21.15.114:0/1672918122 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec1248a0 con 0x55cf5c09af00 2024-05-27T07:54:27.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.531+0000 7fa2f0ebf700 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5c09af00 0x55cf5c0b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.532+0000 7fa301337b80 1 -- 172.21.15.114:0/1672918122 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec124f10 con 0x55cf5bf15e40 2024-05-27T07:54:27.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.532+0000 7fa301337b80 10 client.0 _ll_register_callbacks cb 0x55cf5bf495b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:27.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.532+0000 7fa2ebfff700 1 -- 172.21.15.114:0/1672918122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5c09af00 msgr2=0x55cf5c0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:27.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.532+0000 7fa2ebfff700 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5c09af00 0x55cf5c0b5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.532+0000 7fa2ebfff700 1 -- 172.21.15.114:0/1672918122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c099bc0 msgr2=0x55cf5c0b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:27.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.532+0000 7fa2ebfff700 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c099bc0 0x55cf5c0b5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.532+0000 7fa2ebfff700 1 -- 172.21.15.114:0/1672918122 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf5c1289d0 con 0x55cf5bf15e40 2024-05-27T07:54:27.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.532+0000 7fa2ebfff700 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 0x55cf5c0b4dd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009900 tx=0x7fa2e400bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9cdbc0e9cb7272e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:27.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.533+0000 7fa2cb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:54:27.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.533+0000 7fa2cb7fe700 1 -- 172.21.15.114:0/1672918122 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4009470 con 0x55cf5bf15e40 2024-05-27T07:54:27.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.533+0000 7fa301337b80 1 -- 172.21.15.114:0/1672918122 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2e002a900 con 0x55cf5bf15e40 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.533+0000 7fa2cb7fe700 1 -- 172.21.15.114:0/1672918122 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9466+0+0 (secure 0 0 0) 0x7fa2e4004c60 con 0x55cf5bf15e40 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.533+0000 7fa2cb7fe700 1 -- 172.21.15.114:0/1672918122 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fa2e40286c0 con 0x55cf5bf15e40 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.533+0000 7fa301337b80 10 client.28477 fetch_fsmap learned FSMap version 81 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.533+0000 7fa301337b80 1 -- 172.21.15.114:0/1672918122 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=4+}) -- 0x55cf5c129b80 con 0x55cf5bf15e40 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.533+0000 7fa2cb7fe700 1 -- 172.21.15.114:0/1672918122 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2e4028d10 con 0x55cf5bf15e40 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.533+0000 7fa2cb7fe700 1 -- 172.21.15.114:0/1672918122 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 81) ==== 5305+0+0 (secure 0 0 0) 0x7fa2e4011070 con 0x55cf5bf15e40 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.534+0000 7fa301337b80 10 client.28477 fetch_fsmap finished waiting for FSMap version 81 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.534+0000 7fa301337b80 10 client.28477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.534+0000 7fa301337b80 20 client.28477 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.534+0000 7fa301337b80 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cf5c12a300 0x55cf5c12c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.534+0000 7fa301337b80 4 client.28477 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.534+0000 7fa301337b80 1 -- 172.21.15.114:0/1672918122 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fa2e002a900 con 0x55cf5c12a300 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.534+0000 7fa2eb7fe700 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cf5c12a300 0x55cf5c12c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.534+0000 7fa2cb7fe700 10 client.28477 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:27.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.534+0000 7fa2eb7fe700 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cf5c12a300 0x55cf5c12c720 crc :-1 s=READY pgs=419 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eea39242af2415a4 server_cookie=81455621592018ae in_seq=0 out_seq=0 2024-05-27T07:54:27.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.538+0000 7fa2cb7fe700 1 -- 172.21.15.114:0/1672918122 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fa2e002a900 con 0x55cf5c12a300 2024-05-27T07:54:27.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.538+0000 7fa2cb7fe700 10 client.28477 handle_command_reply: tid=0 2024-05-27T07:54:27.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.539+0000 7fa301337b80 1 client.28477 shutdown 2024-05-27T07:54:27.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.539+0000 7fa301337b80 1 -- 172.21.15.114:0/1672918122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 msgr2=0x55cf5c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:27.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.539+0000 7fa301337b80 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 0x55cf5c0b4dd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009900 tx=0x7fa2e400bce0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.540+0000 7fa301337b80 1 -- 172.21.15.114:0/1672918122 shutdown_connections 2024-05-27T07:54:27.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.540+0000 7fa301337b80 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55cf5c12a300 0x55cf5c12c720 crc :-1 s=READY pgs=419 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.540+0000 7fa301337b80 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf5c09af00 0x55cf5c0b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.540+0000 7fa301337b80 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cf5bf15e40 0x55cf5c0b4dd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.540+0000 7fa2cb7fe700 0 client.28477 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:27.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.540+0000 7fa301337b80 1 --2- 172.21.15.114:0/1672918122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf5c099bc0 0x55cf5c0b5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.540+0000 7fa301337b80 1 -- 172.21.15.114:0/1672918122 >> 172.21.15.114:0/1672918122 conn(0x55cf5bf18770 msgr2=0x55cf5c090150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:27.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.540+0000 7fa301337b80 1 -- 172.21.15.114:0/1672918122 shutdown_connections 2024-05-27T07:54:27.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.540+0000 7fa301337b80 1 -- 172.21.15.114:0/1672918122 wait complete. 2024-05-27T07:54:27.542 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:54:27.542 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:54:27.542 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:54:27.542 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:54:27.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.541+0000 7fa301337b80 20 client.28477 trim_cache size 0 max 16384 2024-05-27T07:54:27.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.542+0000 7fa2d2ffd700 1 -- 172.21.15.114:0/4201626768 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa2d40b2c00 msgr2=0x7fa2d40b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:27.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.542+0000 7fa2d2ffd700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa2d40b2c00 0x7fa2d40b5110 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa2e001a000 tx=0x7fa2e001d000 comp rx=0 tx=0).stop 2024-05-27T07:54:27.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.542+0000 7fa2d2ffd700 1 -- 172.21.15.114:0/4201626768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ec10b500 msgr2=0x7fa2ec117e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:27.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.542+0000 7fa2d2ffd700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ec10b500 0x7fa2ec117e80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00e970 tx=0x7fa2dc00ed30 comp rx=0 tx=0).stop 2024-05-27T07:54:27.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.542+0000 7fa2d2ffd700 1 -- 172.21.15.114:0/4201626768 shutdown_connections 2024-05-27T07:54:27.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.542+0000 7fa2d2ffd700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa2d40b2c00 0x7fa2d40b5110 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.542+0000 7fa2d2ffd700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2ec1192b0 0x7fa2ec1ae2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.542+0000 7fa2d2ffd700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec10abb0 0x7fa2ec117920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.542+0000 7fa2d2ffd700 1 --2- 172.21.15.114:0/4201626768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ec10b500 0x7fa2ec117e80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:27.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.542+0000 7fa2d2ffd700 1 -- 172.21.15.114:0/4201626768 >> 172.21.15.114:0/4201626768 conn(0x7fa2ec06f4c0 msgr2=0x7fa2ec1197d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:27.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.542+0000 7fa2d2ffd700 1 -- 172.21.15.114:0/4201626768 shutdown_connections 2024-05-27T07:54:27.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:27.542+0000 7fa2d2ffd700 1 -- 172.21.15.114:0/4201626768 wait complete. 2024-05-27T07:54:27.592 INFO:tasks.workunit.client.0.smithi114.stdout: 10 8793 16.45 MB/sec warmup 39 sec latency 369.896 ms 2024-05-27T07:54:27.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:27 smithi114 ceph-mon[32038]: osdmap e139: 12 total, 12 up, 12 in 2024-05-27T07:54:27.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:27 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6834/3343072720,v1:172.21.15.163:6835/3343072720] up:boot 2024-05-27T07:54:27.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:27 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 6 up:standby 2024-05-27T07:54:27.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:27 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:54:27.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:27 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:54:27.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:27 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:54:27.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:27 smithi163 ceph-mon[36451]: osdmap e139: 12 total, 12 up, 12 in 2024-05-27T07:54:27.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:27 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6834/3343072720,v1:172.21.15.163:6835/3343072720] up:boot 2024-05-27T07:54:27.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:27 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 6 up:standby 2024-05-27T07:54:27.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:27 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:54:27.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:27 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:54:27.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:27 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:54:27.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:27 smithi186 ceph-mon[36647]: osdmap e139: 12 total, 12 up, 12 in 2024-05-27T07:54:27.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:27 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6834/3343072720,v1:172.21.15.163:6835/3343072720] up:boot 2024-05-27T07:54:27.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:27 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 6 up:standby 2024-05-27T07:54:27.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:27 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:54:27.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:27 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:54:27.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:27 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:54:28.387 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:54:28.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:54:28.476 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:28 smithi163 podman[74218]: 2024-05-27 07:54:28.181788562 +0000 UTC m=+1.283686574 container remove 5ac410326d261981c0ca5abbc3399e714e72a85215f038849ffac44aba6ba608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-27T07:54:28.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:28 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:54:28.477 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:28 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-05-27T07:54:28.592 INFO:tasks.workunit.client.0.smithi114.stdout: 10 9355 16.90 MB/sec warmup 40 sec latency 121.272 ms 2024-05-27T07:54:28.676 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:54:28.932 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:28 smithi114 ceph-mon[32038]: pgmap v500: 129 pgs: 129 active+clean; 274 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 11 MiB/s wr, 801 op/s 2024-05-27T07:54:28.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:28 smithi163 ceph-mon[36451]: pgmap v500: 129 pgs: 129 active+clean; 274 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 11 MiB/s wr, 801 op/s 2024-05-27T07:54:28.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:28 smithi186 ceph-mon[36647]: pgmap v500: 129 pgs: 129 active+clean; 274 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 11 MiB/s wr, 801 op/s 2024-05-27T07:54:29.592 INFO:tasks.workunit.client.0.smithi114.stdout: 10 9840 17.04 MB/sec warmup 41 sec latency 181.886 ms 2024-05-27T07:54:30.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.502+0000 7f349baca700 1 -- 172.21.15.114:0/324183992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3494108730 msgr2=0x7f3494110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:30.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.502+0000 7f349baca700 1 --2- 172.21.15.114:0/324183992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3494108730 0x7f3494110090 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f348c00b0d0 tx=0x7f348c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:54:30.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.502+0000 7f349baca700 1 -- 172.21.15.114:0/324183992 shutdown_connections 2024-05-27T07:54:30.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.502+0000 7f349baca700 1 --2- 172.21.15.114:0/324183992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3494108730 0x7f3494110090 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:30.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.502+0000 7f349baca700 1 --2- 172.21.15.114:0/324183992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34941034b0 0x7f34941038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:30.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.502+0000 7f349baca700 1 --2- 172.21.15.114:0/324183992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3494107de0 0x7f34941081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:30.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.502+0000 7f349baca700 1 -- 172.21.15.114:0/324183992 >> 172.21.15.114:0/324183992 conn(0x7f34940fedb0 msgr2=0x7f34941011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:30 smithi114 ceph-mon[32038]: pgmap v501: 129 pgs: 129 active+clean; 274 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 8.0 MiB/s wr, 601 op/s 2024-05-27T07:54:30.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.503+0000 7f349baca700 1 -- 172.21.15.114:0/324183992 shutdown_connections 2024-05-27T07:54:30.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.503+0000 7f349baca700 1 -- 172.21.15.114:0/324183992 wait complete. 2024-05-27T07:54:30.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.504+0000 7f349baca700 1 Processor -- start 2024-05-27T07:54:30.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.504+0000 7f349baca700 1 -- start start 2024-05-27T07:54:30.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.504+0000 7f349baca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34941034b0 0x7f349419f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:30.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.504+0000 7f349baca700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3494107de0 0x7f349419fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:30.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.504+0000 7f349baca700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494108730 0x7f3494199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:30.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.504+0000 7f349baca700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3494113360 con 0x7f34941034b0 2024-05-27T07:54:30.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.504+0000 7f349baca700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f34941131e0 con 0x7f3494107de0 2024-05-27T07:54:30.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.504+0000 7f349baca700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f34941134e0 con 0x7f3494108730 2024-05-27T07:54:30.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.504+0000 7f3499866700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34941034b0 0x7f349419f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.504+0000 7f3499065700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3494107de0 0x7f349419fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.504+0000 7f349a067700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494108730 0x7f3494199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.504+0000 7f3499866700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34941034b0 0x7f349419f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59852/0 (socket says 172.21.15.114:59852) 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.504+0000 7f3499866700 1 -- 172.21.15.114:0/2463502111 learned_addr learned my addr 172.21.15.114:0/2463502111 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.505+0000 7f3499065700 1 -- 172.21.15.114:0/2463502111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494108730 msgr2=0x7f3494199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.505+0000 7f3499065700 1 --2- 172.21.15.114:0/2463502111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494108730 0x7f3494199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.505+0000 7f3499065700 1 -- 172.21.15.114:0/2463502111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34941034b0 msgr2=0x7f349419f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.505+0000 7f3499065700 1 --2- 172.21.15.114:0/2463502111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34941034b0 0x7f349419f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.505+0000 7f3499065700 1 -- 172.21.15.114:0/2463502111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349419a1e0 con 0x7f3494107de0 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.505+0000 7f3499866700 1 --2- 172.21.15.114:0/2463502111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34941034b0 0x7f349419f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.505+0000 7f3499065700 1 --2- 172.21.15.114:0/2463502111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3494107de0 0x7f349419fe20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f348400e4b0 tx=0x7f348400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.505+0000 7f348affd700 1 -- 172.21.15.114:0/2463502111 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348401f070 con 0x7f3494107de0 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.505+0000 7f348affd700 1 -- 172.21.15.114:0/2463502111 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f348400cd30 con 0x7f3494107de0 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.505+0000 7f348affd700 1 -- 172.21.15.114:0/2463502111 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348400d760 con 0x7f3494107de0 2024-05-27T07:54:30.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.505+0000 7f349baca700 1 -- 172.21.15.114:0/2463502111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f349419a4d0 con 0x7f3494107de0 2024-05-27T07:54:30.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.505+0000 7f349baca700 1 -- 172.21.15.114:0/2463502111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f349406b600 con 0x7f3494107de0 2024-05-27T07:54:30.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.507+0000 7f349baca700 1 -- 172.21.15.114:0/2463502111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3494104960 con 0x7f3494107de0 2024-05-27T07:54:30.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.509+0000 7f348affd700 1 -- 172.21.15.114:0/2463502111 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f3484022070 con 0x7f3494107de0 2024-05-27T07:54:30.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.510+0000 7f348affd700 1 --2- 172.21.15.114:0/2463502111 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f34800b2a70 0x7f34800b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:30.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.510+0000 7f3499866700 1 --2- 172.21.15.114:0/2463502111 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f34800b2a70 0x7f34800b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:30.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.510+0000 7f348affd700 1 -- 172.21.15.114:0/2463502111 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9466+0+0 (secure 0 0 0) 0x7f34840e6b60 con 0x7f3494107de0 2024-05-27T07:54:30.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.510+0000 7f3499866700 1 --2- 172.21.15.114:0/2463502111 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f34800b2a70 0x7f34800b4f80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f349419ac40 tx=0x7f3490009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:30.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.514+0000 7f348affd700 1 -- 172.21.15.114:0/2463502111 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f34840ae190 con 0x7f3494107de0 2024-05-27T07:54:30.592 INFO:tasks.workunit.client.0.smithi114.stdout: 10 10232 17.29 MB/sec warmup 42 sec latency 176.447 ms 2024-05-27T07:54:30.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:30 smithi163 ceph-mon[36451]: pgmap v501: 129 pgs: 129 active+clean; 274 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 8.0 MiB/s wr, 601 op/s 2024-05-27T07:54:30.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:30 smithi186 ceph-mon[36647]: pgmap v501: 129 pgs: 129 active+clean; 274 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 8.0 MiB/s wr, 601 op/s 2024-05-27T07:54:30.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.801+0000 7f349baca700 1 -- 172.21.15.114:0/2463502111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34940512b0 con 0x7f3494107de0 2024-05-27T07:54:30.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.802+0000 7f348affd700 1 -- 172.21.15.114:0/2463502111 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+7693 (secure 0 0 0) 0x7f34840a9030 con 0x7f3494107de0 2024-05-27T07:54:30.802 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:54:30.802 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/1125403618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1125403618},{"type":"v1","addr":"172.21.15.163:6833","nonce":1125403618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":81,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:54:26.661795+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_28316":{"gid":28316,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/3343072720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":3343072720},{"type":"v1","addr":"172.21.15.163:6835","nonce":3343072720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:54:30.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.805+0000 7f349baca700 1 -- 172.21.15.114:0/2463502111 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f34800b2a70 msgr2=0x7f34800b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:30.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.805+0000 7f349baca700 1 --2- 172.21.15.114:0/2463502111 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f34800b2a70 0x7f34800b4f80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f349419ac40 tx=0x7f3490009260 comp rx=0 tx=0).stop 2024-05-27T07:54:30.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.805+0000 7f349baca700 1 -- 172.21.15.114:0/2463502111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3494107de0 msgr2=0x7f349419fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:30.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.805+0000 7f349baca700 1 --2- 172.21.15.114:0/2463502111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3494107de0 0x7f349419fe20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f348400e4b0 tx=0x7f348400c2d0 comp rx=0 tx=0).stop 2024-05-27T07:54:30.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.805+0000 7f349baca700 1 -- 172.21.15.114:0/2463502111 shutdown_connections 2024-05-27T07:54:30.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.805+0000 7f349baca700 1 --2- 172.21.15.114:0/2463502111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494108730 0x7f3494199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:30.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.805+0000 7f349baca700 1 --2- 172.21.15.114:0/2463502111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34941034b0 0x7f349419f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:30.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.805+0000 7f349baca700 1 --2- 172.21.15.114:0/2463502111 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f34800b2a70 0x7f34800b4f80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:30.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.805+0000 7f349baca700 1 --2- 172.21.15.114:0/2463502111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3494107de0 0x7f349419fe20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:30.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.805+0000 7f349baca700 1 -- 172.21.15.114:0/2463502111 >> 172.21.15.114:0/2463502111 conn(0x7f34940fedb0 msgr2=0x7f3494107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:30.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.805+0000 7f349baca700 1 -- 172.21.15.114:0/2463502111 shutdown_connections 2024-05-27T07:54:30.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:30.805+0000 7f349baca700 1 -- 172.21.15.114:0/2463502111 wait complete. 2024-05-27T07:54:30.807 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 81 2024-05-27T07:54:31.592 INFO:tasks.workunit.client.0.smithi114.stdout: 10 10571 17.46 MB/sec warmup 43 sec latency 193.253 ms 2024-05-27T07:54:31.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:31 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2463502111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:54:31.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:31 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2463502111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:54:31.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:31 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2463502111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:54:31.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:54:32.056 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:54:32.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:32 smithi114 ceph-mon[32038]: pgmap v502: 129 pgs: 129 active+clean; 279 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.7 MiB/s wr, 597 op/s 2024-05-27T07:54:32.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 10752 17.29 MB/sec warmup 44 sec latency 293.038 ms 2024-05-27T07:54:32.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:32 smithi163 ceph-mon[36451]: pgmap v502: 129 pgs: 129 active+clean; 279 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.7 MiB/s wr, 597 op/s 2024-05-27T07:54:32.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:32 smithi186 ceph-mon[36647]: pgmap v502: 129 pgs: 129 active+clean; 279 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.7 MiB/s wr, 597 op/s 2024-05-27T07:54:33.592 INFO:tasks.workunit.client.0.smithi114.stdout: 10 11032 17.19 MB/sec warmup 45 sec latency 181.943 ms 2024-05-27T07:54:33.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.788+0000 7f2cf3367700 1 -- 172.21.15.114:0/3195936145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cec108750 msgr2=0x7f2cec1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:33.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.788+0000 7f2cf3367700 1 --2- 172.21.15.114:0/3195936145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cec108750 0x7f2cec1100b0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009fc0 tx=0x7f2ce400b330 comp rx=0 tx=0).stop 2024-05-27T07:54:33.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.789+0000 7f2cf3367700 1 -- 172.21.15.114:0/3195936145 shutdown_connections 2024-05-27T07:54:33.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.789+0000 7f2cf3367700 1 --2- 172.21.15.114:0/3195936145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cec108750 0x7f2cec1100b0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.789+0000 7f2cf3367700 1 --2- 172.21.15.114:0/3195936145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cec1034d0 0x7f2cec1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.789+0000 7f2cf3367700 1 --2- 172.21.15.114:0/3195936145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cec107e00 0x7f2cec108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.789+0000 7f2cf3367700 1 -- 172.21.15.114:0/3195936145 >> 172.21.15.114:0/3195936145 conn(0x7f2cec0fef20 msgr2=0x7f2cec101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:33.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.789+0000 7f2cf3367700 1 -- 172.21.15.114:0/3195936145 shutdown_connections 2024-05-27T07:54:33.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.789+0000 7f2cf3367700 1 -- 172.21.15.114:0/3195936145 wait complete. 2024-05-27T07:54:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.789+0000 7f2cf3367700 1 Processor -- start 2024-05-27T07:54:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.790+0000 7f2cf3367700 1 -- start start 2024-05-27T07:54:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.790+0000 7f2cf3367700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cec1034d0 0x7f2cec199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.790+0000 7f2cf3367700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cec107e00 0x7f2cec19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.790+0000 7f2cf3367700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cec108750 0x7f2cec19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.790+0000 7f2cf3367700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec113380 con 0x7f2cec108750 2024-05-27T07:54:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.790+0000 7f2cf3367700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec113200 con 0x7f2cec1034d0 2024-05-27T07:54:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.790+0000 7f2cf3367700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec113500 con 0x7f2cec107e00 2024-05-27T07:54:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.790+0000 7f2cf1103700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cec1034d0 0x7f2cec199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.790+0000 7f2cf1904700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cec108750 0x7f2cec19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.790+0000 7f2cf1103700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cec1034d0 0x7f2cec199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:48216/0 (socket says 172.21.15.114:48216) 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.790+0000 7f2cf1103700 1 -- 172.21.15.114:0/1004576263 learned_addr learned my addr 172.21.15.114:0/1004576263 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.790+0000 7f2cf0902700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cec107e00 0x7f2cec19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.791+0000 7f2cf1103700 1 -- 172.21.15.114:0/1004576263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cec107e00 msgr2=0x7f2cec19a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.791+0000 7f2cf1103700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cec107e00 0x7f2cec19a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.791+0000 7f2cf1103700 1 -- 172.21.15.114:0/1004576263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cec108750 msgr2=0x7f2cec19e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.791+0000 7f2cf1103700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cec108750 0x7f2cec19e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.791+0000 7f2cf1103700 1 -- 172.21.15.114:0/1004576263 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cec19ec40 con 0x7f2cec1034d0 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.791+0000 7f2cf1904700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cec108750 0x7f2cec19e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.791+0000 7f2cf1103700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cec1034d0 0x7f2cec199d70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2ce800e9c0 tx=0x7f2ce800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.791+0000 7f2ce27fc700 1 -- 172.21.15.114:0/1004576263 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce800ee00 con 0x7f2cec1034d0 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.791+0000 7f2cf3367700 1 -- 172.21.15.114:0/1004576263 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cec19ef30 con 0x7f2cec1034d0 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.791+0000 7f2ce27fc700 1 -- 172.21.15.114:0/1004576263 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ce800de90 con 0x7f2cec1034d0 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.791+0000 7f2ce27fc700 1 -- 172.21.15.114:0/1004576263 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce80053a0 con 0x7f2cec1034d0 2024-05-27T07:54:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.791+0000 7f2cf3367700 1 -- 172.21.15.114:0/1004576263 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cec06c030 con 0x7f2cec1034d0 2024-05-27T07:54:33.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.795+0000 7f2ce27fc700 1 -- 172.21.15.114:0/1004576263 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f2ce8005540 con 0x7f2cec1034d0 2024-05-27T07:54:33.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.796+0000 7f2d01d7cb80 1 Processor -- start 2024-05-27T07:54:33.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.796+0000 7f2d01d7cb80 1 -- start start 2024-05-27T07:54:33.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2d01d7cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5393e40 0x5650c5518b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2d01d7cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5519070 0x5650c5512be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2ce27fc700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2cd80b29f0 0x7f2cd80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2d01d7cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5513120 0x5650c5513530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2d01d7cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5650c5440d70 con 0x5650c5519070 2024-05-27T07:54:33.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2d01d7cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5650c5430910 con 0x5650c5513120 2024-05-27T07:54:33.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2d01d7cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5650c5434950 con 0x5650c5393e40 2024-05-27T07:54:33.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2cf0902700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5519070 0x5650c5512be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2cf0902700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5519070 0x5650c5512be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59882/0 (socket says 172.21.15.114:59882) 2024-05-27T07:54:33.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2cf0902700 1 -- 172.21.15.114:0/853130124 learned_addr learned my addr 172.21.15.114:0/853130124 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:33.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2cf1904700 1 --2- 172.21.15.114:0/853130124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5393e40 0x5650c5518b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2cf1904700 1 --2- 172.21.15.114:0/853130124 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5513120 0x5650c5513530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2ce27fc700 1 -- 172.21.15.114:0/1004576263 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9466+0+0 (secure 0 0 0) 0x7f2ce80e2080 con 0x7f2cec1034d0 2024-05-27T07:54:33.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2cf1103700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2cd80b29f0 0x7f2cd80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2cf1904700 1 -- 172.21.15.114:0/853130124 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5513120 msgr2=0x5650c5513530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:33.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2cf1904700 1 --2- 172.21.15.114:0/853130124 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5513120 0x5650c5513530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2cf1904700 1 -- 172.21.15.114:0/853130124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5519070 msgr2=0x5650c5512be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2cf1904700 1 --2- 172.21.15.114:0/853130124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5519070 0x5650c5512be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.797+0000 7f2cf1904700 1 -- 172.21.15.114:0/853130124 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650c53d9fb0 con 0x5650c5393e40 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.798+0000 7f2cf1904700 1 --2- 172.21.15.114:0/853130124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5393e40 0x5650c5518b30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4015000 tx=0x7f2ce4009f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c49019008e7824a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.798+0000 7f2cf1103700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2cd80b29f0 0x7f2cd80b4f00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8005950 tx=0x7f2ce8004df0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.798+0000 7f2cd77fe700 1 -- 172.21.15.114:0/853130124 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4023030 con 0x5650c5393e40 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.798+0000 7f2cd77fe700 1 -- 172.21.15.114:0/853130124 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ce400bd20 con 0x5650c5393e40 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.798+0000 7f2cd77fe700 1 -- 172.21.15.114:0/853130124 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4026040 con 0x5650c5393e40 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.799+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/853130124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5393e40 msgr2=0x5650c5518b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.799+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/853130124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5393e40 0x5650c5518b30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4015000 tx=0x7f2ce4009f10 comp rx=0 tx=0).stop 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.799+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/853130124 shutdown_connections 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.799+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/853130124 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5513120 0x5650c5513530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.799+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/853130124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5393e40 0x5650c5518b30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.799+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/853130124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5519070 0x5650c5512be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.799+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/853130124 >> 172.21.15.114:0/853130124 conn(0x5650c5396770 msgr2=0x5650c550dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:33.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.799+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/853130124 shutdown_connections 2024-05-27T07:54:33.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.799+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/853130124 wait complete. 2024-05-27T07:54:33.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.800+0000 7f2d01d7cb80 1 -- start start 2024-05-27T07:54:33.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.800+0000 7f2d01d7cb80 1 Processor -- start 2024-05-27T07:54:33.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.800+0000 7f2d01d7cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5393e40 0x5650c551c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.800+0000 7f2d01d7cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5513120 0x5650c551ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.800+0000 7f2d01d7cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 0x5650c551d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.800+0000 7f2d01d7cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5650c5336fa0 con 0x5650c5513120 2024-05-27T07:54:33.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.800+0000 7f2d01d7cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5650c5398da0 con 0x5650c5519070 2024-05-27T07:54:33.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.800+0000 7f2d01d7cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5650c551fa70 con 0x5650c5393e40 2024-05-27T07:54:33.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.800+0000 7f2d01d7cb80 10 client.0 _ll_register_callbacks cb 0x5650c53c75b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:33.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cf0902700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5513120 0x5650c551ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cf1904700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5393e40 0x5650c551c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cf1904700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5393e40 0x5650c551c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36302/0 (socket says 172.21.15.114:36302) 2024-05-27T07:54:33.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cf0902700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5513120 0x5650c551ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59884/0 (socket says 172.21.15.114:59884) 2024-05-27T07:54:33.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cf1904700 1 -- 172.21.15.114:0/1520628900 learned_addr learned my addr 172.21.15.114:0/1520628900 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:33.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cf1904700 1 --2- 172.21.15.114:0/1520628900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 0x5650c551d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cf0902700 1 -- 172.21.15.114:0/1520628900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5393e40 msgr2=0x5650c551c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:33.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cf0902700 1 --2- 172.21.15.114:0/1520628900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5393e40 0x5650c551c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cf0902700 1 -- 172.21.15.114:0/1520628900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 msgr2=0x5650c551d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:33.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cf0902700 1 --2- 172.21.15.114:0/1520628900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 0x5650c551d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cf0902700 1 -- 172.21.15.114:0/1520628900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650c53808d0 con 0x5650c5513120 2024-05-27T07:54:33.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cf0902700 1 --2- 172.21.15.114:0/1520628900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5513120 0x5650c551ce90 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0126d0 tx=0x7f2cdc0129e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c5c515490551ba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:33.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cd4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:54:33.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cd4ff9700 1 -- 172.21.15.114:0/1520628900 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc004050 con 0x5650c5513120 2024-05-27T07:54:33.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cd4ff9700 1 -- 172.21.15.114:0/1520628900 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9466+0+0 (secure 0 0 0) 0x7f2cdc0041f0 con 0x5650c5513120 2024-05-27T07:54:33.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/1520628900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ce400bd20 con 0x5650c5513120 2024-05-27T07:54:33.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.801+0000 7f2cd4ff9700 1 -- 172.21.15.114:0/1520628900 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f2cdc025050 con 0x5650c5513120 2024-05-27T07:54:33.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.802+0000 7f2d01d7cb80 10 client.18987 fetch_fsmap learned FSMap version 81 2024-05-27T07:54:33.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.802+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/1520628900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=4+}) -- 0x5650c53ea760 con 0x5650c5513120 2024-05-27T07:54:33.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.802+0000 7f2cd4ff9700 1 -- 172.21.15.114:0/1520628900 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ce400bd20 con 0x5650c5513120 2024-05-27T07:54:33.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.802+0000 7f2cd4ff9700 1 -- 172.21.15.114:0/1520628900 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 81) ==== 5305+0+0 (secure 0 0 0) 0x7f2cdc0104c0 con 0x5650c5513120 2024-05-27T07:54:33.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.802+0000 7f2d01d7cb80 10 client.18987 fetch_fsmap finished waiting for FSMap version 81 2024-05-27T07:54:33.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.802+0000 7f2d01d7cb80 10 client.18987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:33.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.802+0000 7f2d01d7cb80 20 client.18987 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:33.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.802+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/1520628900 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5650c552ced0 0x5650c552f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.802+0000 7f2d01d7cb80 4 client.18987 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:54:33.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.802+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/1520628900 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650c5442cc0 con 0x5650c552ced0 2024-05-27T07:54:33.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.802+0000 7f2cf1904700 1 --2- 172.21.15.114:0/1520628900 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5650c552ced0 0x5650c552f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.803+0000 7f2cd4ff9700 10 client.18987 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:33.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.803+0000 7f2cf1904700 1 --2- 172.21.15.114:0/1520628900 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5650c552ced0 0x5650c552f320 crc :-1 s=READY pgs=420 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8c53a6435e6c736 server_cookie=f6b8b3d7d5709bf in_seq=0 out_seq=0 2024-05-27T07:54:33.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2cd4ff9700 1 -- 172.21.15.114:0/1520628900 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5650c5442cc0 con 0x5650c552ced0 2024-05-27T07:54:33.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2cd4ff9700 10 client.18987 handle_command_reply: tid=0 2024-05-27T07:54:33.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2d01d7cb80 1 client.18987 shutdown 2024-05-27T07:54:33.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/1520628900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5513120 msgr2=0x5650c551ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:33.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/1520628900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5513120 0x5650c551ce90 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0126d0 tx=0x7f2cdc0129e0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/1520628900 shutdown_connections 2024-05-27T07:54:33.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/1520628900 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5650c552ced0 0x5650c552f320 crc :-1 s=READY pgs=420 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/1520628900 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 0x5650c551d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/1520628900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5393e40 0x5650c551c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/1520628900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5513120 0x5650c551ce90 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/1520628900 >> 172.21.15.114:0/1520628900 conn(0x5650c5396770 msgr2=0x5650c5511bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:33.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2cd4ff9700 0 client.18987 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:33.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/1520628900 shutdown_connections 2024-05-27T07:54:33.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/1520628900 wait complete. 2024-05-27T07:54:33.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.805+0000 7f2d01d7cb80 20 client.18987 trim_cache size 0 max 16384 2024-05-27T07:54:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.824+0000 7f2d01d7cb80 1 Processor -- start 2024-05-27T07:54:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2d01d7cb80 1 -- start start 2024-05-27T07:54:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2d01d7cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 0x5650c5512b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2d01d7cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5513120 0x5650c551d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2d01d7cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 0x5650c5528b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2d01d7cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec1aee70 con 0x5650c5393e40 2024-05-27T07:54:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2d01d7cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec1af540 con 0x5650c5519070 2024-05-27T07:54:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2d01d7cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec1adcf0 con 0x5650c5513120 2024-05-27T07:54:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2cf1904700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 0x5650c5512b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2cf1904700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 0x5650c5512b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59890/0 (socket says 172.21.15.114:59890) 2024-05-27T07:54:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2cf1904700 1 -- 172.21.15.114:0/3700896880 learned_addr learned my addr 172.21.15.114:0/3700896880 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2cf0902700 1 --2- 172.21.15.114:0/3700896880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5513120 0x5650c551d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2cf1904700 1 --2- 172.21.15.114:0/3700896880 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 0x5650c5528b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2cf1904700 1 -- 172.21.15.114:0/3700896880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5513120 msgr2=0x5650c551d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2cf1904700 1 --2- 172.21.15.114:0/3700896880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5513120 0x5650c551d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2cf1904700 1 -- 172.21.15.114:0/3700896880 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 msgr2=0x5650c5528b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2cf1904700 1 --2- 172.21.15.114:0/3700896880 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 0x5650c5528b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2cf1904700 1 -- 172.21.15.114:0/3700896880 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650c551d750 con 0x5650c5393e40 2024-05-27T07:54:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.825+0000 7f2cf0902700 1 --2- 172.21.15.114:0/3700896880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5513120 0x5650c551d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.826+0000 7f2cf1904700 1 --2- 172.21.15.114:0/3700896880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 0x5650c5512b90 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009050 tx=0x7f2ce40096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4581c6fd808a09e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.826+0000 7f2cd6ffd700 1 -- 172.21.15.114:0/3700896880 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4023040 con 0x5650c5393e40 2024-05-27T07:54:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.826+0000 7f2cd6ffd700 1 -- 172.21.15.114:0/3700896880 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ce4021070 con 0x5650c5393e40 2024-05-27T07:54:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.826+0000 7f2cd6ffd700 1 -- 172.21.15.114:0/3700896880 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce40079b0 con 0x5650c5393e40 2024-05-27T07:54:33.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.827+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/3700896880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 msgr2=0x5650c5512b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:33.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.827+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/3700896880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 0x5650c5512b90 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009050 tx=0x7f2ce40096c0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.827+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/3700896880 shutdown_connections 2024-05-27T07:54:33.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.827+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/3700896880 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 0x5650c5528b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.827+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/3700896880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 0x5650c5512b90 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.827+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/3700896880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5513120 0x5650c551d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.827+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/3700896880 >> 172.21.15.114:0/3700896880 conn(0x5650c5396770 msgr2=0x5650c5512480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:33.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.827+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/3700896880 shutdown_connections 2024-05-27T07:54:33.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.827+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/3700896880 wait complete. 2024-05-27T07:54:33.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.827+0000 7f2d01d7cb80 1 -- start start 2024-05-27T07:54:33.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.828+0000 7f2d01d7cb80 1 Processor -- start 2024-05-27T07:54:33.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.828+0000 7f2d01d7cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 0x5650c5535180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.828+0000 7f2d01d7cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5513120 0x5650c5524d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.828+0000 7f2d01d7cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 0x5650c5525240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.828+0000 7f2d01d7cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec1ad620 con 0x5650c5393e40 2024-05-27T07:54:33.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.828+0000 7f2d01d7cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec1ac210 con 0x5650c5519070 2024-05-27T07:54:33.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.828+0000 7f2d01d7cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec1ac880 con 0x5650c5513120 2024-05-27T07:54:33.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.828+0000 7f2cf1904700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 0x5650c5535180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.828+0000 7f2cf1904700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 0x5650c5535180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59900/0 (socket says 172.21.15.114:59900) 2024-05-27T07:54:33.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.828+0000 7f2cf1904700 1 -- 172.21.15.114:0/335388108 learned_addr learned my addr 172.21.15.114:0/335388108 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:33.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.828+0000 7f2cf0902700 1 --2- 172.21.15.114:0/335388108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5513120 0x5650c5524d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.828+0000 7f2cf1904700 1 --2- 172.21.15.114:0/335388108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 0x5650c5525240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.828+0000 7f2d01d7cb80 10 client.0 _ll_register_callbacks cb 0x5650c53c75b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:33.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.829+0000 7f2cf1904700 1 -- 172.21.15.114:0/335388108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5513120 msgr2=0x5650c5524d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:33.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.829+0000 7f2cf1904700 1 --2- 172.21.15.114:0/335388108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5513120 0x5650c5524d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.829+0000 7f2cf1904700 1 -- 172.21.15.114:0/335388108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 msgr2=0x5650c5525240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:33.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.829+0000 7f2cf1904700 1 --2- 172.21.15.114:0/335388108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 0x5650c5525240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.829+0000 7f2cf1904700 1 -- 172.21.15.114:0/335388108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650c5533af0 con 0x5650c5393e40 2024-05-27T07:54:33.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.829+0000 7f2cf1904700 1 --2- 172.21.15.114:0/335388108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 0x5650c5535180 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009f70 tx=0x7f2ce400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c83130bf0a45d05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:33.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.829+0000 7f2cd4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:54:33.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.829+0000 7f2cd4ff9700 1 -- 172.21.15.114:0/335388108 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4023040 con 0x5650c5393e40 2024-05-27T07:54:33.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.829+0000 7f2cd4ff9700 1 -- 172.21.15.114:0/335388108 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9466+0+0 (secure 0 0 0) 0x7f2ce403ae00 con 0x5650c5393e40 2024-05-27T07:54:33.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.829+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/335388108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ce4021070 con 0x5650c5393e40 2024-05-27T07:54:33.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.829+0000 7f2cd4ff9700 1 -- 172.21.15.114:0/335388108 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f2ce401ca20 con 0x5650c5393e40 2024-05-27T07:54:33.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.829+0000 7f2d01d7cb80 10 client.18993 fetch_fsmap learned FSMap version 81 2024-05-27T07:54:33.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.829+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/335388108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=4+}) -- 0x5650c5596a90 con 0x5650c5393e40 2024-05-27T07:54:33.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.830+0000 7f2cd4ff9700 1 -- 172.21.15.114:0/335388108 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ce4021070 con 0x5650c5393e40 2024-05-27T07:54:33.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.830+0000 7f2cd4ff9700 1 -- 172.21.15.114:0/335388108 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 81) ==== 5305+0+0 (secure 0 0 0) 0x7f2ce4038070 con 0x5650c5393e40 2024-05-27T07:54:33.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.830+0000 7f2d01d7cb80 10 client.18993 fetch_fsmap finished waiting for FSMap version 81 2024-05-27T07:54:33.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.830+0000 7f2d01d7cb80 10 client.18993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:33.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.830+0000 7f2d01d7cb80 20 client.18993 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:33.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.830+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/335388108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5650c5597040 0x5650c5599460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:33.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.830+0000 7f2d01d7cb80 4 client.18993 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:54:33.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.830+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/335388108 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5650c543c4a0 con 0x5650c5597040 2024-05-27T07:54:33.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.830+0000 7f2cf0902700 1 --2- 172.21.15.114:0/335388108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5650c5597040 0x5650c5599460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:33.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.831+0000 7f2cd4ff9700 10 client.18993 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:33.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.831+0000 7f2cf0902700 1 --2- 172.21.15.114:0/335388108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5650c5597040 0x5650c5599460 crc :-1 s=READY pgs=421 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4bc308d2b449de server_cookie=e8af78890d6ab36a in_seq=0 out_seq=0 2024-05-27T07:54:33.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2cd4ff9700 1 -- 172.21.15.114:0/335388108 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5650c543c4a0 con 0x5650c5597040 2024-05-27T07:54:33.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2cd4ff9700 10 client.18993 handle_command_reply: tid=0 2024-05-27T07:54:33.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2d01d7cb80 1 client.18993 shutdown 2024-05-27T07:54:33.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/335388108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 msgr2=0x5650c5535180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:33.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/335388108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 0x5650c5535180 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009f70 tx=0x7f2ce400b330 comp rx=0 tx=0).stop 2024-05-27T07:54:33.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/335388108 shutdown_connections 2024-05-27T07:54:33.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/335388108 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5650c5597040 0x5650c5599460 crc :-1 s=READY pgs=421 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/335388108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5650c5519070 0x5650c5525240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/335388108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650c5393e40 0x5650c5535180 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2d01d7cb80 1 --2- 172.21.15.114:0/335388108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5650c5513120 0x5650c5524d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/335388108 >> 172.21.15.114:0/335388108 conn(0x5650c5396770 msgr2=0x5650c5512420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:33.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2cd4ff9700 0 client.18993 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:33.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/335388108 shutdown_connections 2024-05-27T07:54:33.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2d01d7cb80 1 -- 172.21.15.114:0/335388108 wait complete. 2024-05-27T07:54:33.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.833+0000 7f2d01d7cb80 20 client.18993 trim_cache size 0 max 16384 2024-05-27T07:54:33.838 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:54:33.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.834+0000 7f2cf3367700 1 -- 172.21.15.114:0/1004576263 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2cd80b29f0 msgr2=0x7f2cd80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:33.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.834+0000 7f2cf3367700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2cd80b29f0 0x7f2cd80b4f00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8005950 tx=0x7f2ce8004df0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.834+0000 7f2cf3367700 1 -- 172.21.15.114:0/1004576263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cec1034d0 msgr2=0x7f2cec199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:33.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.834+0000 7f2cf3367700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cec1034d0 0x7f2cec199d70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2ce800e9c0 tx=0x7f2ce800ecd0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.834+0000 7f2cf3367700 1 -- 172.21.15.114:0/1004576263 shutdown_connections 2024-05-27T07:54:33.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.834+0000 7f2cf3367700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cec108750 0x7f2cec19e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.834+0000 7f2cf3367700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f2cd80b29f0 0x7f2cd80b4f00 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.834+0000 7f2cf3367700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cec1034d0 0x7f2cec199d70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.834+0000 7f2cf3367700 1 --2- 172.21.15.114:0/1004576263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cec107e00 0x7f2cec19a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:33.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.834+0000 7f2cf3367700 1 -- 172.21.15.114:0/1004576263 >> 172.21.15.114:0/1004576263 conn(0x7f2cec0fef20 msgr2=0x7f2cec107540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:33.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.834+0000 7f2cf3367700 1 -- 172.21.15.114:0/1004576263 shutdown_connections 2024-05-27T07:54:33.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:33.834+0000 7f2cf3367700 1 -- 172.21.15.114:0/1004576263 wait complete. 2024-05-27T07:54:34.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 11281 17.02 MB/sec warmup 46 sec latency 89.898 ms 2024-05-27T07:54:34.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:34 smithi114 ceph-mon[32038]: pgmap v503: 129 pgs: 129 active+clean; 288 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 621 op/s 2024-05-27T07:54:34.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:34 smithi163 ceph-mon[36451]: pgmap v503: 129 pgs: 129 active+clean; 288 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 621 op/s 2024-05-27T07:54:34.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:34 smithi186 ceph-mon[36647]: pgmap v503: 129 pgs: 129 active+clean; 288 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 621 op/s 2024-05-27T07:54:35.546 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:54:35.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:54:35.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 11640 16.93 MB/sec warmup 47 sec latency 93.813 ms 2024-05-27T07:54:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:54:36.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.322+0000 7fa7c131c640 1 -- 172.21.15.114:0/1927984098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7bc1075c0 msgr2=0x7fa7bc1079d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:36.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.322+0000 7fa7c131c640 1 --2- 172.21.15.114:0/1927984098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7bc1075c0 0x7fa7bc1079d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac01a910 tx=0x7fa7ac03b450 comp rx=0 tx=0).stop 2024-05-27T07:54:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.323+0000 7fa7c131c640 1 -- 172.21.15.114:0/1927984098 shutdown_connections 2024-05-27T07:54:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.323+0000 7fa7c131c640 1 --2- 172.21.15.114:0/1927984098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7bc107ff0 0x7fa7bc108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.323+0000 7fa7c131c640 1 --2- 172.21.15.114:0/1927984098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7bc1075c0 0x7fa7bc1079d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.323+0000 7fa7c131c640 1 --2- 172.21.15.114:0/1927984098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7bc102b70 0x7fa7bc102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.323+0000 7fa7c131c640 1 -- 172.21.15.114:0/1927984098 >> 172.21.15.114:0/1927984098 conn(0x7fa7bc0fe380 msgr2=0x7fa7bc1007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.323+0000 7fa7c131c640 1 -- 172.21.15.114:0/1927984098 shutdown_connections 2024-05-27T07:54:36.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.323+0000 7fa7c131c640 1 -- 172.21.15.114:0/1927984098 wait complete. 2024-05-27T07:54:36.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.324+0000 7fa7c131c640 1 Processor -- start 2024-05-27T07:54:36.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.324+0000 7fa7c131c640 1 -- start start 2024-05-27T07:54:36.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.324+0000 7fa7c131c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7bc102b70 0x7fa7bc113140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.324+0000 7fa7c131c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7bc1075c0 0x7fa7bc113680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.324+0000 7fa7c131c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7bc107ff0 0x7fa7bc113bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.324+0000 7fa7c131c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc10b490 con 0x7fa7bc107ff0 2024-05-27T07:54:36.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.324+0000 7fa7c131c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc10b310 con 0x7fa7bc1075c0 2024-05-27T07:54:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.324+0000 7fa7c131c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc10b610 con 0x7fa7bc102b70 2024-05-27T07:54:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.325+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7bc107ff0 0x7fa7bc113bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.325+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7bc107ff0 0x7fa7bc113bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41412/0 (socket says 172.21.15.114:41412) 2024-05-27T07:54:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.325+0000 7fa7bb7fe640 1 -- 172.21.15.114:0/621427120 learned_addr learned my addr 172.21.15.114:0/621427120 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.325+0000 7fa7bb7fe640 1 -- 172.21.15.114:0/621427120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7bc102b70 msgr2=0x7fa7bc113140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:54:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.325+0000 7fa7bb7fe640 1 --2- 172.21.15.114:0/621427120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7bc102b70 0x7fa7bc113140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.325+0000 7fa7bb7fe640 1 -- 172.21.15.114:0/621427120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7bc1075c0 msgr2=0x7fa7bc113680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:54:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.325+0000 7fa7bb7fe640 1 --2- 172.21.15.114:0/621427120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7bc1075c0 0x7fa7bc113680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.325+0000 7fa7bb7fe640 1 -- 172.21.15.114:0/621427120 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7bc1a4970 con 0x7fa7bc107ff0 2024-05-27T07:54:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.326+0000 7fa7bb7fe640 1 --2- 172.21.15.114:0/621427120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7bc107ff0 0x7fa7bc113bc0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400b970 tx=0x7fa7b400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.326+0000 7fa7b3fff640 1 -- 172.21.15.114:0/621427120 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4019070 con 0x7fa7bc107ff0 2024-05-27T07:54:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.326+0000 7fa7c131c640 1 -- 172.21.15.114:0/621427120 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7bc1a4c60 con 0x7fa7bc107ff0 2024-05-27T07:54:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.326+0000 7fa7c131c640 1 -- 172.21.15.114:0/621427120 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7bc1a51b0 con 0x7fa7bc107ff0 2024-05-27T07:54:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.326+0000 7fa7b3fff640 1 -- 172.21.15.114:0/621427120 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7b4014440 con 0x7fa7bc107ff0 2024-05-27T07:54:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.326+0000 7fa7b3fff640 1 -- 172.21.15.114:0/621427120 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b40075d0 con 0x7fa7bc107ff0 2024-05-27T07:54:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.330+0000 7fa7c81cc740 1 Processor -- start 2024-05-27T07:54:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.331+0000 7fa7b3fff640 1 -- 172.21.15.114:0/621427120 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa7b401c070 con 0x7fa7bc107ff0 2024-05-27T07:54:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.331+0000 7fa7c81cc740 1 -- start start 2024-05-27T07:54:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.331+0000 7fa7c81cc740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edfd1b0 0x55f29f19e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.332+0000 7fa7c81cc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edd42f0 0x55f29f19e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.332+0000 7fa7c81cc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.332+0000 7fa7c81cc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f29f110c30 con 0x55f29eddc4d0 2024-05-27T07:54:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.332+0000 7fa7c81cc740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f29f1007d0 con 0x55f29edfd1b0 2024-05-27T07:54:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.332+0000 7fa7c81cc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f29f104810 con 0x55f29edd42f0 2024-05-27T07:54:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.332+0000 7fa7ba7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.332+0000 7fa7ba7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f19ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41426/0 (socket says 172.21.15.114:41426) 2024-05-27T07:54:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.332+0000 7fa7ba7fc640 1 -- 172.21.15.114:0/2928526962 learned_addr learned my addr 172.21.15.114:0/2928526962 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.332+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/2928526962 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edfd1b0 0x55f29f19e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.333+0000 7fa7ba7fc640 1 -- 172.21.15.114:0/2928526962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edd42f0 msgr2=0x55f29f19e910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:54:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.333+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/2928526962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edd42f0 0x55f29f19e910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.333+0000 7fa7ba7fc640 1 -- 172.21.15.114:0/2928526962 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edfd1b0 msgr2=0x55f29f19e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.333+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/2928526962 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edfd1b0 0x55f29f19e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.333+0000 7fa7ba7fc640 1 -- 172.21.15.114:0/2928526962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f29ee15210 con 0x55f29eddc4d0 2024-05-27T07:54:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.333+0000 7fa7b3fff640 1 --2- 172.21.15.114:0/621427120 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7a80b29d0 0x7fa7a80b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.333+0000 7fa7bb7fe640 1 --2- 172.21.15.114:0/621427120 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7a80b29d0 0x7fa7a80b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.333+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/2928526962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f19ee50 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac040920 tx=0x7fa7ac004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6413d1971603ae server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:36.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.333+0000 7fa7b3fff640 1 -- 172.21.15.114:0/621427120 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9466+0+0 (secure 0 0 0) 0x7fa7b40e1e90 con 0x7fa7bc107ff0 2024-05-27T07:54:36.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.333+0000 7fa7b17fa640 1 -- 172.21.15.114:0/2928526962 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac046030 con 0x55f29eddc4d0 2024-05-27T07:54:36.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.333+0000 7fa7b17fa640 1 -- 172.21.15.114:0/2928526962 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7ac040d30 con 0x55f29eddc4d0 2024-05-27T07:54:36.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.333+0000 7fa7b17fa640 1 -- 172.21.15.114:0/2928526962 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac004fc0 con 0x55f29eddc4d0 2024-05-27T07:54:36.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.335+0000 7fa7c81cc740 1 -- 172.21.15.114:0/2928526962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 msgr2=0x55f29f19ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:36.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.335+0000 7fa7bb7fe640 1 --2- 172.21.15.114:0/621427120 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7a80b29d0 0x7fa7a80b4ee0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa7b409d150 tx=0x7fa7b40a8c40 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:36.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.335+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/2928526962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f19ee50 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac040920 tx=0x7fa7ac004e20 comp rx=0 tx=0).stop 2024-05-27T07:54:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.335+0000 7fa7c81cc740 1 -- 172.21.15.114:0/2928526962 shutdown_connections 2024-05-27T07:54:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.335+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/2928526962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f19ee50 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.335+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/2928526962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edd42f0 0x55f29f19e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.335+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/2928526962 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edfd1b0 0x55f29f19e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.335+0000 7fa7c81cc740 1 -- 172.21.15.114:0/2928526962 >> 172.21.15.114:0/2928526962 conn(0x55f29edf1300 msgr2=0x55f29f19ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.335+0000 7fa7c81cc740 1 -- 172.21.15.114:0/2928526962 shutdown_connections 2024-05-27T07:54:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.335+0000 7fa7c81cc740 1 -- 172.21.15.114:0/2928526962 wait complete. 2024-05-27T07:54:36.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.336+0000 7fa7c81cc740 1 -- start start 2024-05-27T07:54:36.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.336+0000 7fa7c81cc740 1 Processor -- start 2024-05-27T07:54:36.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.336+0000 7fa7c81cc740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1a8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.336+0000 7fa7c81cc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f1a8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.336+0000 7fa7c81cc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edfd1b0 0x55f29f1ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.336+0000 7fa7c81cc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f29ede6790 con 0x55f29eddc4d0 2024-05-27T07:54:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.336+0000 7fa7c81cc740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f29f0bc890 con 0x55f29edd42f0 2024-05-27T07:54:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.336+0000 7fa7c81cc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f29f1ab4a0 con 0x55f29edfd1b0 2024-05-27T07:54:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.336+0000 7fa7c81cc740 10 client.0 _ll_register_callbacks cb 0x55f29ee14a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.336+0000 7fa7ba7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1a8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.337+0000 7fa7ba7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edfd1b0 0x55f29f1ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.337+0000 7fa7ba7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1a8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:52866/0 (socket says 172.21.15.114:52866) 2024-05-27T07:54:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.337+0000 7fa7ba7fc640 1 -- 172.21.15.114:0/3032323938 learned_addr learned my addr 172.21.15.114:0/3032323938 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.337+0000 7fa7baffd640 1 --2- 172.21.15.114:0/3032323938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f1a8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.337+0000 7fa7ba7fc640 1 -- 172.21.15.114:0/3032323938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edfd1b0 msgr2=0x55f29f1ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.337+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/3032323938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edfd1b0 0x55f29f1ae8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.337+0000 7fa7ba7fc640 1 -- 172.21.15.114:0/3032323938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 msgr2=0x55f29f1a8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:36.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.337+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/3032323938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f1a8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.337+0000 7fa7ba7fc640 1 -- 172.21.15.114:0/3032323938 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f29eee1620 con 0x55f29edd42f0 2024-05-27T07:54:36.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.337+0000 7fa7baffd640 1 --2- 172.21.15.114:0/3032323938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f1a8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:36.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.337+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/3032323938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1a8a80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac041010 tx=0x7fa7ac0048b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7b35ada8159b1b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:36.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.337+0000 7fa7a27fc640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:54:36.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.338+0000 7fa7a27fc640 1 -- 172.21.15.114:0/3032323938 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac046030 con 0x55f29edd42f0 2024-05-27T07:54:36.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.338+0000 7fa7a27fc640 1 -- 172.21.15.114:0/3032323938 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9466+0+0 (secure 0 0 0) 0x7fa7ac0051d0 con 0x55f29edd42f0 2024-05-27T07:54:36.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.338+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3032323938 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7ac040d30 con 0x55f29edd42f0 2024-05-27T07:54:36.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.338+0000 7fa7a27fc640 1 -- 172.21.15.114:0/3032323938 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fa7ac063050 con 0x55f29edd42f0 2024-05-27T07:54:36.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.338+0000 7fa7c81cc740 10 client.28352 fetch_fsmap learned FSMap version 81 2024-05-27T07:54:36.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.338+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3032323938 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=4+}) -- 0x55f29f1b7830 con 0x55f29edd42f0 2024-05-27T07:54:36.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.338+0000 7fa7a27fc640 1 -- 172.21.15.114:0/3032323938 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7ac040d30 con 0x55f29edd42f0 2024-05-27T07:54:36.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.339+0000 7fa7a27fc640 1 -- 172.21.15.114:0/3032323938 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 81) ==== 5305+0+0 (secure 0 0 0) 0x7fa7ac04b040 con 0x55f29edd42f0 2024-05-27T07:54:36.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.339+0000 7fa7c81cc740 10 client.28352 fetch_fsmap finished waiting for FSMap version 81 2024-05-27T07:54:36.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.339+0000 7fa7c81cc740 10 client.28352 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:36.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.339+0000 7fa7c81cc740 20 client.28352 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:36.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.339+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/3032323938 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f29edbbd80 0x55f29f1bae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.339+0000 7fa7c81cc740 4 client.28352 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:54:36.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.339+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3032323938 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f29f112b80 con 0x55f29edbbd80 2024-05-27T07:54:36.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.339+0000 7fa7baffd640 1 --2- 172.21.15.114:0/3032323938 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f29edbbd80 0x55f29f1bae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.340+0000 7fa7a27fc640 10 client.28352 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:36.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.340+0000 7fa7baffd640 1 --2- 172.21.15.114:0/3032323938 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f29edbbd80 0x55f29f1bae70 crc :-1 s=READY pgs=422 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2baa6562d14db61 server_cookie=af6bee45f023735d in_seq=0 out_seq=0 2024-05-27T07:54:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.343+0000 7fa7a27fc640 1 -- 172.21.15.114:0/3032323938 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f29f112b80 con 0x55f29edbbd80 2024-05-27T07:54:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.343+0000 7fa7a27fc640 10 client.28352 handle_command_reply: tid=0 2024-05-27T07:54:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.343+0000 7fa7c81cc740 1 client.28352 shutdown 2024-05-27T07:54:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.343+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3032323938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 msgr2=0x55f29f1a8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.343+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/3032323938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1a8a80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac041010 tx=0x7fa7ac0048b0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.343+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3032323938 shutdown_connections 2024-05-27T07:54:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.343+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/3032323938 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f29edbbd80 0x55f29f1bae70 crc :-1 s=READY pgs=422 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.343+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/3032323938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edfd1b0 0x55f29f1ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.343+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/3032323938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f1a8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.343+0000 7fa7a27fc640 0 client.28352 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.343+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/3032323938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1a8a80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.343+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3032323938 >> 172.21.15.114:0/3032323938 conn(0x55f29edf1300 msgr2=0x55f29f1a00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.344+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3032323938 shutdown_connections 2024-05-27T07:54:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.344+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3032323938 wait complete. 2024-05-27T07:54:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.344+0000 7fa7c81cc740 20 client.28352 trim_cache size 0 max 16384 2024-05-27T07:54:36.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.361+0000 7fa7c81cc740 1 Processor -- start 2024-05-27T07:54:36.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.361+0000 7fa7c81cc740 1 -- start start 2024-05-27T07:54:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7c81cc740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1aad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7c81cc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f1b5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7c81cc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edfd1b0 0x55f29f19e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7c81cc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c010b00 con 0x55f29eddc4d0 2024-05-27T07:54:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7c81cc740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c0111d0 con 0x55f29edd42f0 2024-05-27T07:54:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7c81cc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c00f9a0 con 0x55f29edfd1b0 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7baffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f1b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7baffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f1b5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41458/0 (socket says 172.21.15.114:41458) 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7baffd640 1 -- 172.21.15.114:0/177203501 learned_addr learned my addr 172.21.15.114:0/177203501 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/177203501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/177203501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edfd1b0 0x55f29f19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7baffd640 1 -- 172.21.15.114:0/177203501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edfd1b0 msgr2=0x55f29f19e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7baffd640 1 --2- 172.21.15.114:0/177203501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edfd1b0 0x55f29f19e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7baffd640 1 -- 172.21.15.114:0/177203501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 msgr2=0x55f29f1aad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7baffd640 1 --2- 172.21.15.114:0/177203501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1aad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7baffd640 1 -- 172.21.15.114:0/177203501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f29f1a9660 con 0x55f29eddc4d0 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/177203501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.362+0000 7fa7baffd640 1 --2- 172.21.15.114:0/177203501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f1b5590 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4000c00 tx=0x7fa7a4006ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b900ed90f5913a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.363+0000 7fa7b0ff9640 1 -- 172.21.15.114:0/177203501 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4024070 con 0x55f29eddc4d0 2024-05-27T07:54:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.363+0000 7fa7b0ff9640 1 -- 172.21.15.114:0/177203501 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7a4016d90 con 0x55f29eddc4d0 2024-05-27T07:54:36.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.363+0000 7fa7b0ff9640 1 -- 172.21.15.114:0/177203501 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a401fc70 con 0x55f29eddc4d0 2024-05-27T07:54:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.364+0000 7fa7c81cc740 1 -- 172.21.15.114:0/177203501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 msgr2=0x55f29f1b5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.364+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/177203501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f1b5590 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4000c00 tx=0x7fa7a4006ed0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.364+0000 7fa7c81cc740 1 -- 172.21.15.114:0/177203501 shutdown_connections 2024-05-27T07:54:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.364+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/177203501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29edfd1b0 0x55f29f19e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.364+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/177203501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29eddc4d0 0x55f29f1b5590 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.364+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/177203501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.364+0000 7fa7c81cc740 1 -- 172.21.15.114:0/177203501 >> 172.21.15.114:0/177203501 conn(0x55f29edf1300 msgr2=0x55f29f19e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.364+0000 7fa7c81cc740 1 -- 172.21.15.114:0/177203501 shutdown_connections 2024-05-27T07:54:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.364+0000 7fa7c81cc740 1 -- 172.21.15.114:0/177203501 wait complete. 2024-05-27T07:54:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.364+0000 7fa7c81cc740 1 -- start start 2024-05-27T07:54:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7c81cc740 1 Processor -- start 2024-05-27T07:54:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7c81cc740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7c81cc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29eddc4d0 0x55f29f1af520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7c81cc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29edfd1b0 0x55f29f1afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7c81cc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c00f2d0 con 0x55f29edfd1b0 2024-05-27T07:54:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7c81cc740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c00dec0 con 0x55f29edd42f0 2024-05-27T07:54:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7c81cc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c00e530 con 0x55f29eddc4d0 2024-05-27T07:54:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7c81cc740 10 client.0 _ll_register_callbacks cb 0x7fa7ac051070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7ba7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29edfd1b0 0x55f29f1afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7ba7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29edfd1b0 0x55f29f1afa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41466/0 (socket says 172.21.15.114:41466) 2024-05-27T07:54:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7ba7fc640 1 -- 172.21.15.114:0/3782922326 learned_addr learned my addr 172.21.15.114:0/3782922326 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7baffd640 1 --2- 172.21.15.114:0/3782922326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29eddc4d0 0x55f29f1af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/3782922326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7ba7fc640 1 -- 172.21.15.114:0/3782922326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29eddc4d0 msgr2=0x55f29f1af520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/3782922326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29eddc4d0 0x55f29f1af520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7ba7fc640 1 -- 172.21.15.114:0/3782922326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 msgr2=0x55f29f1b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/3782922326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7ba7fc640 1 -- 172.21.15.114:0/3782922326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f29f1affa0 con 0x55f29edfd1b0 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7ba7fc640 1 --2- 172.21.15.114:0/3782922326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29edfd1b0 0x55f29f1afa60 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x55f29f1aed80 tx=0x7fa7ac0048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c7bac21e51550c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.365+0000 7fa7a27fc640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7a27fc640 1 -- 172.21.15.114:0/3782922326 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac046040 con 0x55f29edfd1b0 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7a27fc640 1 -- 172.21.15.114:0/3782922326 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9466+0+0 (secure 0 0 0) 0x7fa7ac003d40 con 0x55f29edfd1b0 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3782922326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7a4016d90 con 0x55f29edfd1b0 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7a27fc640 1 -- 172.21.15.114:0/3782922326 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fa7ac077050 con 0x55f29edfd1b0 2024-05-27T07:54:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7c81cc740 10 client.19008 fetch_fsmap learned FSMap version 81 2024-05-27T07:54:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3782922326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=4+}) -- 0x55f29f2221f0 con 0x55f29edfd1b0 2024-05-27T07:54:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7a27fc640 1 -- 172.21.15.114:0/3782922326 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7a4016d90 con 0x55f29edfd1b0 2024-05-27T07:54:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7a27fc640 1 -- 172.21.15.114:0/3782922326 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 81) ==== 5305+0+0 (secure 0 0 0) 0x7fa7ac05e450 con 0x55f29edfd1b0 2024-05-27T07:54:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7c81cc740 10 client.19008 fetch_fsmap finished waiting for FSMap version 81 2024-05-27T07:54:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7c81cc740 10 client.19008 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7c81cc740 20 client.19008 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/3782922326 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f29ed911c0 0x55f29f224920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7c81cc740 4 client.19008 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:54:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3782922326 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f29f10c360 con 0x55f29ed911c0 2024-05-27T07:54:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.366+0000 7fa7baffd640 1 --2- 172.21.15.114:0/3782922326 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f29ed911c0 0x55f29f224920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.367+0000 7fa7a27fc640 10 client.19008 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:36.367+0000 7fa7baffd640 1 --2- 172.21.15.114:0/3782922326 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f29ed911c0 0x55f29f224920 crc :-1 s=READY pgs=423 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=913baf2585e331db server_cookie=120960a30329ac45 in_seq=0 out_seq=0 2024-05-27T07:54:36.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 11947 17.00 MB/sec warmup 48 sec latency 142.641 ms 2024-05-27T07:54:36.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:36 smithi163 ceph-mon[36451]: pgmap v504: 129 pgs: 129 active+clean; 294 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.3 MiB/s wr, 478 op/s 2024-05-27T07:54:36.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:36 smithi186 ceph-mon[36647]: pgmap v504: 129 pgs: 129 active+clean; 294 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.3 MiB/s wr, 478 op/s 2024-05-27T07:54:37.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:36 smithi114 ceph-mon[32038]: pgmap v504: 129 pgs: 129 active+clean; 294 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.3 MiB/s wr, 478 op/s 2024-05-27T07:54:37.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 11979 16.66 MB/sec warmup 49 sec latency 918.485 ms 2024-05-27T07:54:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.101+0000 7fa7a27fc640 1 -- 172.21.15.114:0/3782922326 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f29f10c360 con 0x55f29ed911c0 2024-05-27T07:54:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.101+0000 7fa7a27fc640 10 client.19008 handle_command_reply: tid=0 2024-05-27T07:54:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.101+0000 7fa7c81cc740 1 client.19008 shutdown 2024-05-27T07:54:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.101+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3782922326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29edfd1b0 msgr2=0x55f29f1afa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.101+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/3782922326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29edfd1b0 0x55f29f1afa60 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x55f29f1aed80 tx=0x7fa7ac0048b0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.102+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3782922326 shutdown_connections 2024-05-27T07:54:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.102+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/3782922326 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55f29ed911c0 0x55f29f224920 crc :-1 s=READY pgs=423 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.102+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/3782922326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29edfd1b0 0x55f29f1afa60 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.102+0000 7fa7a27fc640 0 client.19008 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.102+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/3782922326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f29eddc4d0 0x55f29f1af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.102+0000 7fa7c81cc740 1 --2- 172.21.15.114:0/3782922326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f29edd42f0 0x55f29f1b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.102+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3782922326 >> 172.21.15.114:0/3782922326 conn(0x55f29edf1300 msgr2=0x55f29f19eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.102+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3782922326 shutdown_connections 2024-05-27T07:54:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.102+0000 7fa7c81cc740 1 -- 172.21.15.114:0/3782922326 wait complete. 2024-05-27T07:54:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.103+0000 7fa7c81cc740 20 client.19008 trim_cache size 0 max 16384 2024-05-27T07:54:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:54:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:54:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:54:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.103+0000 7fa7c131c640 1 -- 172.21.15.114:0/621427120 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7a80b29d0 msgr2=0x7fa7a80b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.103+0000 7fa7c131c640 1 --2- 172.21.15.114:0/621427120 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7a80b29d0 0x7fa7a80b4ee0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa7b409d150 tx=0x7fa7b40a8c40 comp rx=0 tx=0).stop 2024-05-27T07:54:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.103+0000 7fa7c131c640 1 -- 172.21.15.114:0/621427120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7bc107ff0 msgr2=0x7fa7bc113bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.103+0000 7fa7c131c640 1 --2- 172.21.15.114:0/621427120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7bc107ff0 0x7fa7bc113bc0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400b970 tx=0x7fa7b400be30 comp rx=0 tx=0).stop 2024-05-27T07:54:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.103+0000 7fa7c131c640 1 -- 172.21.15.114:0/621427120 shutdown_connections 2024-05-27T07:54:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.103+0000 7fa7c131c640 1 --2- 172.21.15.114:0/621427120 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa7a80b29d0 0x7fa7a80b4ee0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.103+0000 7fa7c131c640 1 --2- 172.21.15.114:0/621427120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7bc107ff0 0x7fa7bc113bc0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.103+0000 7fa7c131c640 1 --2- 172.21.15.114:0/621427120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7bc1075c0 0x7fa7bc113680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.103+0000 7fa7c131c640 1 --2- 172.21.15.114:0/621427120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7bc102b70 0x7fa7bc113140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.104+0000 7fa7c131c640 1 -- 172.21.15.114:0/621427120 >> 172.21.15.114:0/621427120 conn(0x7fa7bc0fe380 msgr2=0x7fa7bc1002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.104+0000 7fa7c131c640 1 -- 172.21.15.114:0/621427120 shutdown_connections 2024-05-27T07:54:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:38.104+0000 7fa7c131c640 1 -- 172.21.15.114:0/621427120 wait complete. 2024-05-27T07:54:38.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 12256 16.61 MB/sec warmup 50 sec latency 280.804 ms 2024-05-27T07:54:38.722 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:38 smithi114 ceph-mon[32038]: pgmap v505: 129 pgs: 129 active+clean; 297 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 275 op/s 2024-05-27T07:54:38.741 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:38 smithi163 ceph-mon[36451]: pgmap v505: 129 pgs: 129 active+clean; 297 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 275 op/s 2024-05-27T07:54:38.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:38 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:54:38.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:38 smithi163 systemd[1]: Stopped Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:54:38.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:38 smithi163 systemd[1]: Starting Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:54:38.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:38 smithi163 podman[74323]: 2024-05-27 07:54:38.616647307 +0000 UTC m=+0.016993430 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:54:38.741 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:38 smithi163 podman[74323]: 2024-05-27 07:54:38.741142217 +0000 UTC m=+0.141488328 container create 41d2c56949a23225ea96ae80cef29e98f0f817d1956a065b36102d93345851dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:54:38.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.860+0000 7f05374ef700 1 -- 172.21.15.114:0/725882696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0530107de0 msgr2=0x7f05301081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:38.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.860+0000 7f05374ef700 1 --2- 172.21.15.114:0/725882696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0530107de0 0x7f05301081f0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f05200099c0 tx=0x7f0520009cd0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.860+0000 7f05374ef700 1 -- 172.21.15.114:0/725882696 shutdown_connections 2024-05-27T07:54:38.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.860+0000 7f05374ef700 1 --2- 172.21.15.114:0/725882696 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0530108730 0x7f0530110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.860+0000 7f05374ef700 1 --2- 172.21.15.114:0/725882696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05301034b0 0x7f05301038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.860+0000 7f05374ef700 1 --2- 172.21.15.114:0/725882696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0530107de0 0x7f05301081f0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.860+0000 7f05374ef700 1 -- 172.21.15.114:0/725882696 >> 172.21.15.114:0/725882696 conn(0x7f05300fedb0 msgr2=0x7f05301011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:38.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.861+0000 7f05374ef700 1 -- 172.21.15.114:0/725882696 shutdown_connections 2024-05-27T07:54:38.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.862+0000 7f05374ef700 1 -- 172.21.15.114:0/725882696 wait complete. 2024-05-27T07:54:38.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.862+0000 7f05374ef700 1 Processor -- start 2024-05-27T07:54:38.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.862+0000 7f05374ef700 1 -- start start 2024-05-27T07:54:38.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f05374ef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05301034b0 0x7f053019d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f05374ef700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0530107de0 0x7f053019d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f05374ef700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0530108730 0x7f05301a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f05374ef700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0530112290 con 0x7f05301034b0 2024-05-27T07:54:38.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f05374ef700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0530112110 con 0x7f0530108730 2024-05-27T07:54:38.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f05374ef700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0530112410 con 0x7f0530107de0 2024-05-27T07:54:38.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f0534a8a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0530107de0 0x7f053019d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f053528b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05301034b0 0x7f053019d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f0534a8a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0530107de0 0x7f053019d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54878/0 (socket says 172.21.15.114:54878) 2024-05-27T07:54:38.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f0534a8a700 1 -- 172.21.15.114:0/1832334899 learned_addr learned my addr 172.21.15.114:0/1832334899 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:38.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f0535a8c700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0530108730 0x7f05301a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f0534a8a700 1 -- 172.21.15.114:0/1832334899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0530108730 msgr2=0x7f05301a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:38.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f0534a8a700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0530108730 0x7f05301a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f0534a8a700 1 -- 172.21.15.114:0/1832334899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05301034b0 msgr2=0x7f053019d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:38.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.863+0000 7f0534a8a700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05301034b0 0x7f053019d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.864+0000 7f0534a8a700 1 -- 172.21.15.114:0/1832334899 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05301a1fb0 con 0x7f0530107de0 2024-05-27T07:54:38.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.864+0000 7f053528b700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05301034b0 0x7f053019d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:38.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.864+0000 7f0535a8c700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0530108730 0x7f05301a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:38.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.864+0000 7f0534a8a700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0530107de0 0x7f053019d690 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f05200055f0 tx=0x7f0520005620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:38.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.864+0000 7f05267fc700 1 -- 172.21.15.114:0/1832334899 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0520026070 con 0x7f0530107de0 2024-05-27T07:54:38.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.864+0000 7f05374ef700 1 -- 172.21.15.114:0/1832334899 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05301a2240 con 0x7f0530107de0 2024-05-27T07:54:38.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.864+0000 7f05374ef700 1 -- 172.21.15.114:0/1832334899 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05301a9bc0 con 0x7f0530107de0 2024-05-27T07:54:38.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.864+0000 7f05267fc700 1 -- 172.21.15.114:0/1832334899 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f052000b790 con 0x7f0530107de0 2024-05-27T07:54:38.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.864+0000 7f05267fc700 1 -- 172.21.15.114:0/1832334899 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052000f7a0 con 0x7f0530107de0 2024-05-27T07:54:38.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.867+0000 7f05267fc700 1 -- 172.21.15.114:0/1832334899 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f052000f940 con 0x7f0530107de0 2024-05-27T07:54:38.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.868+0000 7f05267fc700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f051c0b29f0 0x7f051c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.868+0000 7f053528b700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f051c0b29f0 0x7f051c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.868+0000 7f05267fc700 1 -- 172.21.15.114:0/1832334899 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9466+0+0 (secure 0 0 0) 0x7f05200e3240 con 0x7f0530107de0 2024-05-27T07:54:38.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f053528b700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f051c0b29f0 0x7f051c0b4f00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f052c009680 tx=0x7f052c009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:38.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f0545f04b80 1 Processor -- start 2024-05-27T07:54:38.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f0545f04b80 1 -- start start 2024-05-27T07:54:38.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f0545f04b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e57f7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f0545f04b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 0x5584e57f1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f0545f04b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57f2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f0545f04b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5584e5720d10 con 0x5584e57f2000 2024-05-27T07:54:38.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f0545f04b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5584e57108b0 con 0x5584e57f7cd0 2024-05-27T07:54:38.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f0545f04b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5584e57148f0 con 0x5584e5673ec0 2024-05-27T07:54:38.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f0535a8c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f053528b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e57f7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f0534a8a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 0x5584e57f1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f0535a8c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57f2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41498/0 (socket says 172.21.15.114:41498) 2024-05-27T07:54:38.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.869+0000 7f0535a8c700 1 -- 172.21.15.114:0/827788212 learned_addr learned my addr 172.21.15.114:0/827788212 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:38.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.870+0000 7f0534a8a700 1 -- 172.21.15.114:0/827788212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 msgr2=0x5584e57f7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:38.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.870+0000 7f0534a8a700 1 --2- 172.21.15.114:0/827788212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e57f7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.870+0000 7f0534a8a700 1 -- 172.21.15.114:0/827788212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 msgr2=0x5584e57f2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:38.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.870+0000 7f0534a8a700 1 --2- 172.21.15.114:0/827788212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57f2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.870+0000 7f0534a8a700 1 -- 172.21.15.114:0/827788212 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584e5660950 con 0x5584e57f7cd0 2024-05-27T07:54:38.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.870+0000 7f0535a8c700 1 --2- 172.21.15.114:0/827788212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57f2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:38.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.870+0000 7f0534a8a700 1 --2- 172.21.15.114:0/827788212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 0x5584e57f1ac0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f05200e0580 tx=0x7f05200a9910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40745fec2ebac6ee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:38.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.870+0000 7f051b7fe700 1 -- 172.21.15.114:0/827788212 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05200aa1a0 con 0x5584e57f7cd0 2024-05-27T07:54:38.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.870+0000 7f051b7fe700 1 -- 172.21.15.114:0/827788212 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05200ae550 con 0x5584e57f7cd0 2024-05-27T07:54:38.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.870+0000 7f051b7fe700 1 -- 172.21.15.114:0/827788212 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05200b2ae0 con 0x5584e57f7cd0 2024-05-27T07:54:38.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.871+0000 7f0545f04b80 1 -- 172.21.15.114:0/827788212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 msgr2=0x5584e57f1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:38.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.871+0000 7f0545f04b80 1 --2- 172.21.15.114:0/827788212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 0x5584e57f1ac0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f05200e0580 tx=0x7f05200a9910 comp rx=0 tx=0).stop 2024-05-27T07:54:38.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.871+0000 7f0545f04b80 1 -- 172.21.15.114:0/827788212 shutdown_connections 2024-05-27T07:54:38.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.871+0000 7f0545f04b80 1 --2- 172.21.15.114:0/827788212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57f2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.871+0000 7f0545f04b80 1 --2- 172.21.15.114:0/827788212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e57f7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.871+0000 7f0545f04b80 1 --2- 172.21.15.114:0/827788212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 0x5584e57f1ac0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.871+0000 7f0545f04b80 1 -- 172.21.15.114:0/827788212 >> 172.21.15.114:0/827788212 conn(0x5584e56ceb70 msgr2=0x5584e57ec6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:38.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.871+0000 7f0545f04b80 1 -- 172.21.15.114:0/827788212 shutdown_connections 2024-05-27T07:54:38.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.871+0000 7f0545f04b80 1 -- 172.21.15.114:0/827788212 wait complete. 2024-05-27T07:54:38.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.872+0000 7f0545f04b80 1 -- start start 2024-05-27T07:54:38.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.872+0000 7f0545f04b80 1 Processor -- start 2024-05-27T07:54:38.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f0545f04b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e57fb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f0545f04b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57fb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f0545f04b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 0x5584e57fbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f0545f04b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5584e5617020 con 0x5584e57f2000 2024-05-27T07:54:38.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f0545f04b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5584e56b2840 con 0x5584e57f7cd0 2024-05-27T07:54:38.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f0545f04b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5584e57fe580 con 0x5584e5673ec0 2024-05-27T07:54:38.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f0545f04b80 10 client.0 _ll_register_callbacks cb 0x5584e56ab9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:38.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f053528b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e57fb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f053528b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e57fb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54886/0 (socket says 172.21.15.114:54886) 2024-05-27T07:54:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f0534a8a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f053528b700 1 -- 172.21.15.114:0/271226813 learned_addr learned my addr 172.21.15.114:0/271226813 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f0535a8c700 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 0x5584e57fbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f053528b700 1 -- 172.21.15.114:0/271226813 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 msgr2=0x5584e57fbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f053528b700 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 0x5584e57fbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f053528b700 1 -- 172.21.15.114:0/271226813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 msgr2=0x5584e57fb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f053528b700 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57fb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f053528b700 1 -- 172.21.15.114:0/271226813 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584e56af710 con 0x5584e5673ec0 2024-05-27T07:54:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f0535a8c700 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 0x5584e57fbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.873+0000 7f0534a8a700 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f053528b700 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e57fb450 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f052c00ee30 tx=0x7f052c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d4657f2b5588dfd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f0518ff9700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f0518ff9700 1 -- 172.21.15.114:0/271226813 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c02b060 con 0x5584e5673ec0 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f0518ff9700 1 -- 172.21.15.114:0/271226813 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9466+0+0 (secure 0 0 0) 0x7f052c01dd10 con 0x5584e5673ec0 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f0545f04b80 1 -- 172.21.15.114:0/271226813 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05200ae550 con 0x5584e5673ec0 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f0518ff9700 1 -- 172.21.15.114:0/271226813 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f052c032050 con 0x5584e5673ec0 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f0545f04b80 10 client.28525 fetch_fsmap learned FSMap version 81 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f0545f04b80 1 -- 172.21.15.114:0/271226813 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=4+}) -- 0x5584e5809030 con 0x5584e5673ec0 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f0518ff9700 1 -- 172.21.15.114:0/271226813 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05200ae550 con 0x5584e5673ec0 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f0518ff9700 1 -- 172.21.15.114:0/271226813 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 81) ==== 5305+0+0 (secure 0 0 0) 0x7f052c01c400 con 0x5584e5673ec0 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f0545f04b80 10 client.28525 fetch_fsmap finished waiting for FSMap version 81 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f0545f04b80 10 client.28525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f0545f04b80 20 client.28525 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.874+0000 7f0545f04b80 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5584e580bad0 0x5584e580df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.875+0000 7f0545f04b80 4 client.28525 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:54:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.875+0000 7f0545f04b80 1 -- 172.21.15.114:0/271226813 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584e57bda70 con 0x5584e580bad0 2024-05-27T07:54:38.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.875+0000 7f0534a8a700 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5584e580bad0 0x5584e580df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.875+0000 7f0518ff9700 10 client.28525 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:38.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.876+0000 7f0534a8a700 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5584e580bad0 0x5584e580df20 crc :-1 s=READY pgs=424 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a4c868722f8516d server_cookie=89622ed0f458d3e1 in_seq=0 out_seq=0 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.879+0000 7f0518ff9700 1 -- 172.21.15.114:0/271226813 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5584e57bda70 con 0x5584e580bad0 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.879+0000 7f0518ff9700 10 client.28525 handle_command_reply: tid=0 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.879+0000 7f0545f04b80 1 client.28525 shutdown 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.879+0000 7f0545f04b80 1 -- 172.21.15.114:0/271226813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 msgr2=0x5584e57fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.879+0000 7f0545f04b80 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e57fb450 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f052c00ee30 tx=0x7f052c0047c0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.879+0000 7f0545f04b80 1 -- 172.21.15.114:0/271226813 shutdown_connections 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.879+0000 7f0545f04b80 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5584e580bad0 0x5584e580df20 crc :-1 s=READY pgs=424 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.879+0000 7f0545f04b80 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 0x5584e57fbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.879+0000 7f0545f04b80 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e57fb450 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.879+0000 7f0545f04b80 1 --2- 172.21.15.114:0/271226813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.879+0000 7f0518ff9700 0 client.28525 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.879+0000 7f0545f04b80 1 -- 172.21.15.114:0/271226813 >> 172.21.15.114:0/271226813 conn(0x5584e56ceb70 msgr2=0x5584e57f0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.880+0000 7f0545f04b80 1 -- 172.21.15.114:0/271226813 shutdown_connections 2024-05-27T07:54:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.880+0000 7f0545f04b80 1 -- 172.21.15.114:0/271226813 wait complete. 2024-05-27T07:54:38.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.880+0000 7f0545f04b80 20 client.28525 trim_cache size 0 max 16384 2024-05-27T07:54:38.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0545f04b80 1 Processor -- start 2024-05-27T07:54:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0545f04b80 1 -- start start 2024-05-27T07:54:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0545f04b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e58075e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0545f04b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57fbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0545f04b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 0x5584e57fd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0545f04b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f051400f840 con 0x5584e57f2000 2024-05-27T07:54:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0545f04b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f051400ff10 con 0x5584e57f7cd0 2024-05-27T07:54:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0545f04b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f051400e6c0 con 0x5584e5673ec0 2024-05-27T07:54:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0534a8a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57fbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0534a8a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57fbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41506/0 (socket says 172.21.15.114:41506) 2024-05-27T07:54:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0534a8a700 1 -- 172.21.15.114:0/3883901455 learned_addr learned my addr 172.21.15.114:0/3883901455 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f053528b700 1 --2- 172.21.15.114:0/3883901455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e58075e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0534a8a700 1 -- 172.21.15.114:0/3883901455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 msgr2=0x5584e58075e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:38.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0534a8a700 1 --2- 172.21.15.114:0/3883901455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e58075e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0534a8a700 1 -- 172.21.15.114:0/3883901455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 msgr2=0x5584e57fd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:38.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0534a8a700 1 --2- 172.21.15.114:0/3883901455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 0x5584e57fd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.908+0000 7f0534a8a700 1 -- 172.21.15.114:0/3883901455 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584e57f99c0 con 0x5584e57f2000 2024-05-27T07:54:38.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.909+0000 7f0534a8a700 1 --2- 172.21.15.114:0/3883901455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57fbb20 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f05200a6220 tx=0x7f05200e3170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276fbe041489a4e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.909+0000 7f051affd700 1 -- 172.21.15.114:0/3883901455 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05200b2f70 con 0x5584e57f2000 2024-05-27T07:54:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.909+0000 7f051affd700 1 -- 172.21.15.114:0/3883901455 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05200aa400 con 0x5584e57f2000 2024-05-27T07:54:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.909+0000 7f051affd700 1 -- 172.21.15.114:0/3883901455 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05200a5d70 con 0x5584e57f2000 2024-05-27T07:54:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.910+0000 7f0545f04b80 1 -- 172.21.15.114:0/3883901455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 msgr2=0x5584e57fbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:38.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.910+0000 7f0545f04b80 1 --2- 172.21.15.114:0/3883901455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57fbb20 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f05200a6220 tx=0x7f05200e3170 comp rx=0 tx=0).stop 2024-05-27T07:54:38.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.910+0000 7f0545f04b80 1 -- 172.21.15.114:0/3883901455 shutdown_connections 2024-05-27T07:54:38.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.910+0000 7f0545f04b80 1 --2- 172.21.15.114:0/3883901455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e57f7cd0 0x5584e57fd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.910+0000 7f0545f04b80 1 --2- 172.21.15.114:0/3883901455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e5673ec0 0x5584e58075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.910+0000 7f0545f04b80 1 --2- 172.21.15.114:0/3883901455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e57fbb20 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.910+0000 7f0545f04b80 1 -- 172.21.15.114:0/3883901455 >> 172.21.15.114:0/3883901455 conn(0x5584e56ceb70 msgr2=0x5584e57fb410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:38.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.910+0000 7f0545f04b80 1 -- 172.21.15.114:0/3883901455 shutdown_connections 2024-05-27T07:54:38.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.910+0000 7f0545f04b80 1 -- 172.21.15.114:0/3883901455 wait complete. 2024-05-27T07:54:38.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f0545f04b80 1 -- start start 2024-05-27T07:54:38.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f0545f04b80 1 Processor -- start 2024-05-27T07:54:38.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f0545f04b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e5673ec0 0x5584e5803ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f0545f04b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e5803ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f0545f04b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e57f7cd0 0x5584e5800820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f0545f04b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f051400dff0 con 0x5584e57f2000 2024-05-27T07:54:38.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f0545f04b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f051400cbe0 con 0x5584e5673ec0 2024-05-27T07:54:38.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f0545f04b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f051400d250 con 0x5584e57f7cd0 2024-05-27T07:54:38.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f0545f04b80 10 client.0 _ll_register_callbacks cb 0x5584e56ab9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:38.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f0534a8a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e5803ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f053528b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e5673ec0 0x5584e5803ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f0534a8a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e5803ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41514/0 (socket says 172.21.15.114:41514) 2024-05-27T07:54:38.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f0534a8a700 1 -- 172.21.15.114:0/2698553848 learned_addr learned my addr 172.21.15.114:0/2698553848 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:38.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.911+0000 7f0535a8c700 1 --2- 172.21.15.114:0/2698553848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e57f7cd0 0x5584e5800820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f053528b700 1 -- 172.21.15.114:0/2698553848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e57f7cd0 msgr2=0x5584e5800820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:38.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f053528b700 1 --2- 172.21.15.114:0/2698553848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e57f7cd0 0x5584e5800820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f053528b700 1 -- 172.21.15.114:0/2698553848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 msgr2=0x5584e5803ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:38.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f053528b700 1 --2- 172.21.15.114:0/2698553848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e5803ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f053528b700 1 -- 172.21.15.114:0/2698553848 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584e5800fc0 con 0x5584e5673ec0 2024-05-27T07:54:38.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f0534a8a700 1 --2- 172.21.15.114:0/2698553848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e5803ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:38.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f053528b700 1 --2- 172.21.15.114:0/2698553848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e5673ec0 0x5584e5803ab0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f052c016040 tx=0x7f052c011c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78f78e53a565e49c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:38.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f0518ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:54:38.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f0518ff9700 1 -- 172.21.15.114:0/2698553848 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c03c060 con 0x5584e5673ec0 2024-05-27T07:54:38.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f0518ff9700 1 -- 172.21.15.114:0/2698553848 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9466+0+0 (secure 0 0 0) 0x7f052c01ca40 con 0x5584e5673ec0 2024-05-27T07:54:38.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f0545f04b80 1 -- 172.21.15.114:0/2698553848 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05200aa400 con 0x5584e5673ec0 2024-05-27T07:54:38.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f0518ff9700 1 -- 172.21.15.114:0/2698553848 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f052c043050 con 0x5584e5673ec0 2024-05-27T07:54:38.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f0545f04b80 10 client.28373 fetch_fsmap learned FSMap version 81 2024-05-27T07:54:38.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.912+0000 7f0545f04b80 1 -- 172.21.15.114:0/2698553848 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=4+}) -- 0x5584e58770c0 con 0x5584e5673ec0 2024-05-27T07:54:38.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.913+0000 7f0518ff9700 1 -- 172.21.15.114:0/2698553848 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05200aa400 con 0x5584e5673ec0 2024-05-27T07:54:38.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.913+0000 7f0518ff9700 1 -- 172.21.15.114:0/2698553848 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 81) ==== 5305+0+0 (secure 0 0 0) 0x7f052c038440 con 0x5584e5673ec0 2024-05-27T07:54:38.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.913+0000 7f0545f04b80 10 client.28373 fetch_fsmap finished waiting for FSMap version 81 2024-05-27T07:54:38.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.913+0000 7f0545f04b80 10 client.28373 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:38.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.913+0000 7f0545f04b80 20 client.28373 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:38.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.913+0000 7f0545f04b80 1 --2- 172.21.15.114:0/2698553848 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5584e5877740 0x5584e5879b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:38.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.913+0000 7f0545f04b80 4 client.28373 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:54:38.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.913+0000 7f0545f04b80 1 -- 172.21.15.114:0/2698553848 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5584e5722c60 con 0x5584e5877740 2024-05-27T07:54:38.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.913+0000 7f0534a8a700 1 --2- 172.21.15.114:0/2698553848 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5584e5877740 0x5584e5879b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:38.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.913+0000 7f0518ff9700 10 client.28373 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:38.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.914+0000 7f0534a8a700 1 --2- 172.21.15.114:0/2698553848 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5584e5877740 0x5584e5879b60 crc :-1 s=READY pgs=425 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffdd44162fd10a98 server_cookie=7d66bf20d382eb6a in_seq=0 out_seq=0 2024-05-27T07:54:38.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0518ff9700 1 -- 172.21.15.114:0/2698553848 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5584e5722c60 con 0x5584e5877740 2024-05-27T07:54:38.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0518ff9700 10 client.28373 handle_command_reply: tid=0 2024-05-27T07:54:38.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0545f04b80 1 client.28373 shutdown 2024-05-27T07:54:38.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0545f04b80 1 -- 172.21.15.114:0/2698553848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e5673ec0 msgr2=0x5584e5803ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:38.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0545f04b80 1 --2- 172.21.15.114:0/2698553848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e5673ec0 0x5584e5803ab0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f052c016040 tx=0x7f052c011c40 comp rx=0 tx=0).stop 2024-05-27T07:54:38.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0545f04b80 1 -- 172.21.15.114:0/2698553848 shutdown_connections 2024-05-27T07:54:38.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0545f04b80 1 --2- 172.21.15.114:0/2698553848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e57f7cd0 0x5584e5800820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0545f04b80 1 --2- 172.21.15.114:0/2698553848 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5584e5877740 0x5584e5879b60 crc :-1 s=READY pgs=425 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0545f04b80 1 --2- 172.21.15.114:0/2698553848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5584e5673ec0 0x5584e5803ab0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0545f04b80 1 --2- 172.21.15.114:0/2698553848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584e57f2000 0x5584e5803ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0518ff9700 0 client.28373 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:38.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0545f04b80 1 -- 172.21.15.114:0/2698553848 >> 172.21.15.114:0/2698553848 conn(0x5584e56ceb70 msgr2=0x5584e57f5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:38.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0545f04b80 1 -- 172.21.15.114:0/2698553848 shutdown_connections 2024-05-27T07:54:38.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0545f04b80 1 -- 172.21.15.114:0/2698553848 wait complete. 2024-05-27T07:54:38.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.917+0000 7f0545f04b80 20 client.28373 trim_cache size 0 max 16384 2024-05-27T07:54:38.921 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:54:38.921 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:54:38.921 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "732d7839-9f93-4aff-80d2-86cfafbf6bc3", 2024-05-27T07:54:38.921 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:54:38.921 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:54:38.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.918+0000 7f05374ef700 1 -- 172.21.15.114:0/1832334899 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f051c0b29f0 msgr2=0x7f051c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:38.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.918+0000 7f05374ef700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f051c0b29f0 0x7f051c0b4f00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f052c009680 tx=0x7f052c009260 comp rx=0 tx=0).stop 2024-05-27T07:54:38.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.918+0000 7f05374ef700 1 -- 172.21.15.114:0/1832334899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0530107de0 msgr2=0x7f053019d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:38.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.918+0000 7f05374ef700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0530107de0 0x7f053019d690 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f05200055f0 tx=0x7f0520005620 comp rx=0 tx=0).stop 2024-05-27T07:54:38.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.918+0000 7f05374ef700 1 -- 172.21.15.114:0/1832334899 shutdown_connections 2024-05-27T07:54:38.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.918+0000 7f05374ef700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f051c0b29f0 0x7f051c0b4f00 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.918+0000 7f05374ef700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0530108730 0x7f05301a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.918+0000 7f05374ef700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05301034b0 0x7f053019d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.918+0000 7f05374ef700 1 --2- 172.21.15.114:0/1832334899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0530107de0 0x7f053019d690 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:38.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.918+0000 7f05374ef700 1 -- 172.21.15.114:0/1832334899 >> 172.21.15.114:0/1832334899 conn(0x7f05300fedb0 msgr2=0x7f0530104db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:38.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.919+0000 7f05374ef700 1 -- 172.21.15.114:0/1832334899 shutdown_connections 2024-05-27T07:54:38.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:38.919+0000 7f05374ef700 1 -- 172.21.15.114:0/1832334899 wait complete. 2024-05-27T07:54:38.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:38 smithi186 ceph-mon[36647]: pgmap v505: 129 pgs: 129 active+clean; 297 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 275 op/s 2024-05-27T07:54:39.353 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 podman[74323]: 2024-05-27 07:54:39.078022964 +0000 UTC m=+0.478369089 container init 41d2c56949a23225ea96ae80cef29e98f0f817d1956a065b36102d93345851dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-27T07:54:39.353 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 podman[74323]: 2024-05-27 07:54:39.080788894 +0000 UTC m=+0.481135014 container start 41d2c56949a23225ea96ae80cef29e98f0f817d1956a065b36102d93345851dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-27T07:54:39.354 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-mds[74342]: -- 172.21.15.163:0/3916332085 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5594c0a0ba40 con 0x5594c0a53a80 2024-05-27T07:54:39.354 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: starting mds.e at 2024-05-27T07:54:39.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 12743 16.80 MB/sec warmup 51 sec latency 197.136 ms 2024-05-27T07:54:39.623 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 bash[74323]: 41d2c56949a23225ea96ae80cef29e98f0f817d1956a065b36102d93345851dc 2024-05-27T07:54:39.623 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 systemd[1]: Started Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:54:39.623 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5f91ce7700 time 2024-05-27T07:54:39.441094+0000 2024-05-27T07:54:39.623 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:39.623 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:39.623 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5fa452cba6] 2024-05-27T07:54:39.623 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fa452cdd1] 2024-05-27T07:54:39.624 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5594bde6f23d] 2024-05-27T07:54:39.624 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5594bde5a971] 2024-05-27T07:54:39.624 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 5: (SnapRealm::check_cache() const+0x13e) [0x5594bde58fa0] 2024-05-27T07:54:39.624 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5594bde59651] 2024-05-27T07:54:39.624 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5594bddee347] 2024-05-27T07:54:39.624 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5594bdaea87e] 2024-05-27T07:54:39.624 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5594bdaeff2f] 2024-05-27T07:54:39.624 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5594bdafa969] 2024-05-27T07:54:39.624 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 11: (MDLog::_replay_thread()+0x1f38) [0x5594bde9348c] 2024-05-27T07:54:39.624 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5594bde9c9cb] 2024-05-27T07:54:39.624 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 13: (Thread::entry_wrapper()+0x43) [0x7f5fa44fb72d] 2024-05-27T07:54:39.624 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5fa44fb749] 2024-05-27T07:54:39.624 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5fa309f1ca] 2024-05-27T07:54:39.625 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 16: clone() 2024-05-27T07:54:39.625 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.442+0000 7f5f91ce7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5f91ce7700 time 2024-05-27T07:54:39.441094+0000 2024-05-27T07:54:39.626 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:39.626 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.626 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:39.626 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5fa452cba6] 2024-05-27T07:54:39.626 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fa452cdd1] 2024-05-27T07:54:39.626 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5594bde6f23d] 2024-05-27T07:54:39.626 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5594bde5a971] 2024-05-27T07:54:39.626 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 5: (SnapRealm::check_cache() const+0x13e) [0x5594bde58fa0] 2024-05-27T07:54:39.626 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5594bde59651] 2024-05-27T07:54:39.626 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5594bddee347] 2024-05-27T07:54:39.626 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5594bdaea87e] 2024-05-27T07:54:39.627 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5594bdaeff2f] 2024-05-27T07:54:39.627 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5594bdafa969] 2024-05-27T07:54:39.627 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 11: (MDLog::_replay_thread()+0x1f38) [0x5594bde9348c] 2024-05-27T07:54:39.627 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5594bde9c9cb] 2024-05-27T07:54:39.627 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 13: (Thread::entry_wrapper()+0x43) [0x7f5fa44fb72d] 2024-05-27T07:54:39.627 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5fa44fb749] 2024-05-27T07:54:39.627 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5fa309f1ca] 2024-05-27T07:54:39.627 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 16: clone() 2024-05-27T07:54:39.627 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.627 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: *** Caught signal (Aborted) ** 2024-05-27T07:54:39.627 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: in thread 7f5f91ce7700 thread_name:md_log_replay 2024-05-27T07:54:39.627 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:39.627 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5fa30a9d20] 2024-05-27T07:54:39.628 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2: gsignal() 2024-05-27T07:54:39.628 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 3: abort() 2024-05-27T07:54:39.628 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5fa452cced] 2024-05-27T07:54:39.628 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fa452cdd1] 2024-05-27T07:54:39.628 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5594bde6f23d] 2024-05-27T07:54:39.628 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5594bde5a971] 2024-05-27T07:54:39.628 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 8: (SnapRealm::check_cache() const+0x13e) [0x5594bde58fa0] 2024-05-27T07:54:39.628 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5594bde59651] 2024-05-27T07:54:39.628 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5594bddee347] 2024-05-27T07:54:39.628 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5594bdaea87e] 2024-05-27T07:54:39.628 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5594bdaeff2f] 2024-05-27T07:54:39.628 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5594bdafa969] 2024-05-27T07:54:39.629 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 14: (MDLog::_replay_thread()+0x1f38) [0x5594bde9348c] 2024-05-27T07:54:39.629 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5594bde9c9cb] 2024-05-27T07:54:39.629 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 16: (Thread::entry_wrapper()+0x43) [0x7f5fa44fb72d] 2024-05-27T07:54:39.629 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5fa44fb749] 2024-05-27T07:54:39.629 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5fa309f1ca] 2024-05-27T07:54:39.629 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 19: clone() 2024-05-27T07:54:39.629 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.444+0000 7f5f91ce7700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:54:39.629 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: in thread 7f5f91ce7700 thread_name:md_log_replay 2024-05-27T07:54:39.629 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.629 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:39.629 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5fa30a9d20] 2024-05-27T07:54:39.629 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2: gsignal() 2024-05-27T07:54:39.629 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 3: abort() 2024-05-27T07:54:39.630 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5fa452cced] 2024-05-27T07:54:39.630 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fa452cdd1] 2024-05-27T07:54:39.630 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5594bde6f23d] 2024-05-27T07:54:39.630 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5594bde5a971] 2024-05-27T07:54:39.630 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 8: (SnapRealm::check_cache() const+0x13e) [0x5594bde58fa0] 2024-05-27T07:54:39.630 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5594bde59651] 2024-05-27T07:54:39.630 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5594bddee347] 2024-05-27T07:54:39.630 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5594bdaea87e] 2024-05-27T07:54:39.630 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5594bdaeff2f] 2024-05-27T07:54:39.630 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5594bdafa969] 2024-05-27T07:54:39.630 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 14: (MDLog::_replay_thread()+0x1f38) [0x5594bde9348c] 2024-05-27T07:54:39.631 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5594bde9c9cb] 2024-05-27T07:54:39.631 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 16: (Thread::entry_wrapper()+0x43) [0x7f5fa44fb72d] 2024-05-27T07:54:39.631 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5fa44fb749] 2024-05-27T07:54:39.631 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5fa309f1ca] 2024-05-27T07:54:39.631 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 19: clone() 2024-05-27T07:54:39.631 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:54:39.631 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.631 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: -1> 2024-05-27T07:54:39.442+0000 7f5f91ce7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5f91ce7700 time 2024-05-27T07:54:39.441094+0000 2024-05-27T07:54:39.631 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:39.631 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.631 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:39.631 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5fa452cba6] 2024-05-27T07:54:39.631 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fa452cdd1] 2024-05-27T07:54:39.632 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5594bde6f23d] 2024-05-27T07:54:39.632 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5594bde5a971] 2024-05-27T07:54:39.632 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 5: (SnapRealm::check_cache() const+0x13e) [0x5594bde58fa0] 2024-05-27T07:54:39.632 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5594bde59651] 2024-05-27T07:54:39.632 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5594bddee347] 2024-05-27T07:54:39.632 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5594bdaea87e] 2024-05-27T07:54:39.632 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5594bdaeff2f] 2024-05-27T07:54:39.632 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5594bdafa969] 2024-05-27T07:54:39.632 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 11: (MDLog::_replay_thread()+0x1f38) [0x5594bde9348c] 2024-05-27T07:54:39.632 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5594bde9c9cb] 2024-05-27T07:54:39.632 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 13: (Thread::entry_wrapper()+0x43) [0x7f5fa44fb72d] 2024-05-27T07:54:39.632 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5fa44fb749] 2024-05-27T07:54:39.633 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5fa309f1ca] 2024-05-27T07:54:39.633 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 16: clone() 2024-05-27T07:54:39.633 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.633 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 0> 2024-05-27T07:54:39.444+0000 7f5f91ce7700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:54:39.633 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: in thread 7f5f91ce7700 thread_name:md_log_replay 2024-05-27T07:54:39.633 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.633 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:39.633 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5fa30a9d20] 2024-05-27T07:54:39.633 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2: gsignal() 2024-05-27T07:54:39.633 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 3: abort() 2024-05-27T07:54:39.633 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5fa452cced] 2024-05-27T07:54:39.633 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fa452cdd1] 2024-05-27T07:54:39.633 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5594bde6f23d] 2024-05-27T07:54:39.634 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5594bde5a971] 2024-05-27T07:54:39.634 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 8: (SnapRealm::check_cache() const+0x13e) [0x5594bde58fa0] 2024-05-27T07:54:39.634 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5594bde59651] 2024-05-27T07:54:39.634 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5594bddee347] 2024-05-27T07:54:39.634 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5594bdaea87e] 2024-05-27T07:54:39.634 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5594bdaeff2f] 2024-05-27T07:54:39.634 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5594bdafa969] 2024-05-27T07:54:39.634 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 14: (MDLog::_replay_thread()+0x1f38) [0x5594bde9348c] 2024-05-27T07:54:39.634 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5594bde9c9cb] 2024-05-27T07:54:39.634 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 16: (Thread::entry_wrapper()+0x43) [0x7f5fa44fb72d] 2024-05-27T07:54:39.634 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5fa44fb749] 2024-05-27T07:54:39.634 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5fa309f1ca] 2024-05-27T07:54:39.635 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 19: clone() 2024-05-27T07:54:39.635 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:54:39.635 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.635 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: -1> 2024-05-27T07:54:39.442+0000 7f5f91ce7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5f91ce7700 time 2024-05-27T07:54:39.441094+0000 2024-05-27T07:54:39.635 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:54:39.635 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.635 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:39.635 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5fa452cba6] 2024-05-27T07:54:39.635 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fa452cdd1] 2024-05-27T07:54:39.635 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5594bde6f23d] 2024-05-27T07:54:39.635 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5594bde5a971] 2024-05-27T07:54:39.635 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 5: (SnapRealm::check_cache() const+0x13e) [0x5594bde58fa0] 2024-05-27T07:54:39.635 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5594bde59651] 2024-05-27T07:54:39.636 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5594bddee347] 2024-05-27T07:54:39.636 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5594bdaea87e] 2024-05-27T07:54:39.636 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5594bdaeff2f] 2024-05-27T07:54:39.636 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5594bdafa969] 2024-05-27T07:54:39.636 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 11: (MDLog::_replay_thread()+0x1f38) [0x5594bde9348c] 2024-05-27T07:54:39.636 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5594bde9c9cb] 2024-05-27T07:54:39.636 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 13: (Thread::entry_wrapper()+0x43) [0x7f5fa44fb72d] 2024-05-27T07:54:39.636 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5fa44fb749] 2024-05-27T07:54:39.636 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5fa309f1ca] 2024-05-27T07:54:39.637 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 16: clone() 2024-05-27T07:54:39.637 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.637 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 0> 2024-05-27T07:54:39.444+0000 7f5f91ce7700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:54:39.637 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: in thread 7f5f91ce7700 thread_name:md_log_replay 2024-05-27T07:54:39.637 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.637 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:54:39.637 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5fa30a9d20] 2024-05-27T07:54:39.637 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2: gsignal() 2024-05-27T07:54:39.637 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 3: abort() 2024-05-27T07:54:39.637 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5fa452cced] 2024-05-27T07:54:39.637 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fa452cdd1] 2024-05-27T07:54:39.637 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5594bde6f23d] 2024-05-27T07:54:39.637 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5594bde5a971] 2024-05-27T07:54:39.638 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 8: (SnapRealm::check_cache() const+0x13e) [0x5594bde58fa0] 2024-05-27T07:54:39.638 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5594bde59651] 2024-05-27T07:54:39.638 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5594bddee347] 2024-05-27T07:54:39.638 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5594bdaea87e] 2024-05-27T07:54:39.638 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5594bdaeff2f] 2024-05-27T07:54:39.638 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5594bdafa969] 2024-05-27T07:54:39.638 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 14: (MDLog::_replay_thread()+0x1f38) [0x5594bde9348c] 2024-05-27T07:54:39.638 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5594bde9c9cb] 2024-05-27T07:54:39.638 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 16: (Thread::entry_wrapper()+0x43) [0x7f5fa44fb72d] 2024-05-27T07:54:39.638 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5fa44fb749] 2024-05-27T07:54:39.638 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5fa309f1ca] 2024-05-27T07:54:39.638 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 19: clone() 2024-05-27T07:54:39.638 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:54:39.639 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e[74338]: 2024-05-27T07:54:39.639 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:39 smithi163 podman[74373]: 2024-05-27 07:54:39.496185613 +0000 UTC m=+0.018868352 container died 41d2c56949a23225ea96ae80cef29e98f0f817d1956a065b36102d93345851dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-27T07:54:39.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:39 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:54:39.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:39 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:39.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:39 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:54:39.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:39 smithi114 ceph-mon[32038]: daemon mds.e restarted 2024-05-27T07:54:39.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:39 smithi114 ceph-mon[32038]: osdmap e140: 12 total, 12 up, 12 in 2024-05-27T07:54:39.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:39 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6834/1904281067,v1:172.21.15.163:6835/1904281067] up:boot 2024-05-27T07:54:39.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:39 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:54:39.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:39 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:54:39.883 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:39 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:54:39.884 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:39 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:39.884 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:39 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:54:39.884 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:39 smithi163 ceph-mon[36451]: daemon mds.e restarted 2024-05-27T07:54:39.884 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:39 smithi163 ceph-mon[36451]: osdmap e140: 12 total, 12 up, 12 in 2024-05-27T07:54:39.884 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:39 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6834/1904281067,v1:172.21.15.163:6835/1904281067] up:boot 2024-05-27T07:54:39.884 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:39 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:54:39.884 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:39 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:54:39.885 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:54:39.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:39 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:54:39.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:39 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:39.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:39 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:54:39.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:39 smithi186 ceph-mon[36647]: daemon mds.e restarted 2024-05-27T07:54:39.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:39 smithi186 ceph-mon[36647]: osdmap e140: 12 total, 12 up, 12 in 2024-05-27T07:54:39.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:39 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6834/1904281067,v1:172.21.15.163:6835/1904281067] up:boot 2024-05-27T07:54:39.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:39 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 5 up:standby 2024-05-27T07:54:39.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:39 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:54:40.170 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:54:40.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 13145 16.89 MB/sec warmup 52 sec latency 180.278 ms 2024-05-27T07:54:40.623 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:40 smithi163 podman[74373]: 2024-05-27 07:54:40.30258139 +0000 UTC m=+0.825264129 container remove 41d2c56949a23225ea96ae80cef29e98f0f817d1956a065b36102d93345851dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:54:40.623 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:40 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:54:40.623 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:40 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-05-27T07:54:40.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:40 smithi163 ceph-mon[36451]: pgmap v507: 129 pgs: 129 active+clean; 298 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 297 op/s 2024-05-27T07:54:40.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:40 smithi186 ceph-mon[36647]: pgmap v507: 129 pgs: 129 active+clean; 298 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 297 op/s 2024-05-27T07:54:41.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:40 smithi114 ceph-mon[32038]: pgmap v507: 129 pgs: 129 active+clean; 298 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 297 op/s 2024-05-27T07:54:41.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 13605 17.12 MB/sec warmup 53 sec latency 39.258 ms 2024-05-27T07:54:41.726 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:41 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:41.880 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:41 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:41.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:41 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:54:42.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.001+0000 7f8e1768f700 1 -- 172.21.15.114:0/2146066122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e101045d0 msgr2=0x7f8e101049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:42.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.001+0000 7f8e1768f700 1 --2- 172.21.15.114:0/2146066122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e101045d0 0x7f8e101049e0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0099c0 tx=0x7f8e0c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.002+0000 7f8e1768f700 1 -- 172.21.15.114:0/2146066122 shutdown_connections 2024-05-27T07:54:42.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.002+0000 7f8e1768f700 1 --2- 172.21.15.114:0/2146066122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e10109850 0x7f8e101111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.002+0000 7f8e1768f700 1 --2- 172.21.15.114:0/2146066122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e101045d0 0x7f8e101049e0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.002+0000 7f8e1768f700 1 --2- 172.21.15.114:0/2146066122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e10108f00 0x7f8e10109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.002+0000 7f8e1768f700 1 -- 172.21.15.114:0/2146066122 >> 172.21.15.114:0/2146066122 conn(0x7f8e100ffe60 msgr2=0x7f8e101022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:42.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.002+0000 7f8e1768f700 1 -- 172.21.15.114:0/2146066122 shutdown_connections 2024-05-27T07:54:42.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.002+0000 7f8e1768f700 1 -- 172.21.15.114:0/2146066122 wait complete. 2024-05-27T07:54:42.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.002+0000 7f8e1768f700 1 Processor -- start 2024-05-27T07:54:42.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.002+0000 7f8e1768f700 1 -- start start 2024-05-27T07:54:42.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e1768f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e101045d0 0x7f8e1019adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e1768f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e10108f00 0x7f8e1019b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e1768f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e10109850 0x7f8e1019f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e1768f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e10114480 con 0x7f8e10108f00 2024-05-27T07:54:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e1768f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8e10114300 con 0x7f8e101045d0 2024-05-27T07:54:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e1768f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e10114600 con 0x7f8e10109850 2024-05-27T07:54:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e14c2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e10108f00 0x7f8e1019b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e14c2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e10108f00 0x7f8e1019b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41540/0 (socket says 172.21.15.114:41540) 2024-05-27T07:54:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e14c2a700 1 -- 172.21.15.114:0/2264403064 learned_addr learned my addr 172.21.15.114:0/2264403064 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e1542b700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e101045d0 0x7f8e1019adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e15c2c700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e10109850 0x7f8e1019f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e14c2a700 1 -- 172.21.15.114:0/2264403064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e10109850 msgr2=0x7f8e1019f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e14c2a700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e10109850 0x7f8e1019f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e14c2a700 1 -- 172.21.15.114:0/2264403064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e101045d0 msgr2=0x7f8e1019adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:42.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e14c2a700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e101045d0 0x7f8e1019adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e14c2a700 1 -- 172.21.15.114:0/2264403064 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1019fcc0 con 0x7f8e10108f00 2024-05-27T07:54:42.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.003+0000 7f8e14c2a700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e10108f00 0x7f8e1019b390 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000e970 tx=0x7f8e0000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:42.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.004+0000 7f8e067fc700 1 -- 172.21.15.114:0/2264403064 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00004a30 con 0x7f8e10108f00 2024-05-27T07:54:42.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.004+0000 7f8e067fc700 1 -- 172.21.15.114:0/2264403064 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e0000cc90 con 0x7f8e10108f00 2024-05-27T07:54:42.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.004+0000 7f8e1768f700 1 -- 172.21.15.114:0/2264403064 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e1019ffb0 con 0x7f8e10108f00 2024-05-27T07:54:42.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.004+0000 7f8e067fc700 1 -- 172.21.15.114:0/2264403064 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00005610 con 0x7f8e10108f00 2024-05-27T07:54:42.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.004+0000 7f8e1768f700 1 -- 172.21.15.114:0/2264403064 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e101a78d0 con 0x7f8e10108f00 2024-05-27T07:54:42.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.007+0000 7f8e067fc700 1 -- 172.21.15.114:0/2264403064 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f8e0000c7e0 con 0x7f8e10108f00 2024-05-27T07:54:42.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.008+0000 7f8e067fc700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8dfc0b29f0 0x7f8dfc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.008+0000 7f8e1542b700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8dfc0b29f0 0x7f8dfc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.008+0000 7f8e260a4b80 1 Processor -- start 2024-05-27T07:54:42.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.008+0000 7f8e067fc700 1 -- 172.21.15.114:0/2264403064 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 9466+0+0 (secure 0 0 0) 0x7f8e000e2b20 con 0x7f8e10108f00 2024-05-27T07:54:42.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.008+0000 7f8e260a4b80 1 -- start start 2024-05-27T07:54:42.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.008+0000 7f8e260a4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52a2ee40 0x556a52bb3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.008+0000 7f8e260a4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bb4070 0x556a52badbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.008+0000 7f8e260a4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 0x556a52bae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.008+0000 7f8e1542b700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8dfc0b29f0 0x7f8dfc0b4f00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c000c00 tx=0x7f8e0c01a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:42.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.008+0000 7f8e260a4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556a52adbd70 con 0x556a52bb4070 2024-05-27T07:54:42.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.008+0000 7f8e260a4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556a52acb910 con 0x556a52a2ee40 2024-05-27T07:54:42.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.008+0000 7f8e260a4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556a52acf950 con 0x556a52bae120 2024-05-27T07:54:42.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8e15c2c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 0x556a52bae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8e14c2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bb4070 0x556a52badbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8e1542b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52a2ee40 0x556a52bb3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8e15c2c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 0x556a52bae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54924/0 (socket says 172.21.15.114:54924) 2024-05-27T07:54:42.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8e15c2c700 1 -- 172.21.15.114:0/2861955685 learned_addr learned my addr 172.21.15.114:0/2861955685 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:42.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8e14c2a700 1 --2- 172.21.15.114:0/2861955685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bb4070 0x556a52badbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41550/0 (socket says 172.21.15.114:41550) 2024-05-27T07:54:42.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8e15c2c700 1 -- 172.21.15.114:0/2861955685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52a2ee40 msgr2=0x556a52bb3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:42.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8e15c2c700 1 --2- 172.21.15.114:0/2861955685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52a2ee40 0x556a52bb3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8e15c2c700 1 -- 172.21.15.114:0/2861955685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bb4070 msgr2=0x556a52badbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:42.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8e15c2c700 1 --2- 172.21.15.114:0/2861955685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bb4070 0x556a52badbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8e15c2c700 1 -- 172.21.15.114:0/2861955685 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a52a74fb0 con 0x556a52bae120 2024-05-27T07:54:42.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8e15c2c700 1 --2- 172.21.15.114:0/2861955685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 0x556a52bae530 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8e08011680 tx=0x7f8e08009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff7e40bbf5a2c28d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:42.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8dfb7fe700 1 -- 172.21.15.114:0/2861955685 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e08019030 con 0x556a52bae120 2024-05-27T07:54:42.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8dfb7fe700 1 -- 172.21.15.114:0/2861955685 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e080092e0 con 0x556a52bae120 2024-05-27T07:54:42.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.009+0000 7f8dfb7fe700 1 -- 172.21.15.114:0/2861955685 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e08007500 con 0x556a52bae120 2024-05-27T07:54:42.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.010+0000 7f8e260a4b80 1 -- 172.21.15.114:0/2861955685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 msgr2=0x556a52bae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.010+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/2861955685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 0x556a52bae530 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8e08011680 tx=0x7f8e08009920 comp rx=0 tx=0).stop 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.010+0000 7f8e260a4b80 1 -- 172.21.15.114:0/2861955685 shutdown_connections 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.010+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/2861955685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 0x556a52bae530 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.010+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/2861955685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52a2ee40 0x556a52bb3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.010+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/2861955685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bb4070 0x556a52badbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.010+0000 7f8e260a4b80 1 -- 172.21.15.114:0/2861955685 >> 172.21.15.114:0/2861955685 conn(0x556a52a31770 msgr2=0x556a52ba8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.010+0000 7f8e260a4b80 1 -- 172.21.15.114:0/2861955685 shutdown_connections 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.010+0000 7f8e260a4b80 1 -- 172.21.15.114:0/2861955685 wait complete. 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.011+0000 7f8e260a4b80 1 -- start start 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.011+0000 7f8e260a4b80 1 Processor -- start 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e260a4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52a2ee40 0x556a52bb7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e260a4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bae120 0x556a52bb7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e260a4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bb4070 0x556a52bb8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e260a4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556a529d1fa0 con 0x556a52bae120 2024-05-27T07:54:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e260a4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556a52a33da0 con 0x556a52a2ee40 2024-05-27T07:54:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e260a4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556a52bbab60 con 0x556a52bb4070 2024-05-27T07:54:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e260a4b80 10 client.0 _ll_register_callbacks cb 0x556a52a625b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e14c2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bae120 0x556a52bb7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e1542b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52a2ee40 0x556a52bb7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e1542b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52a2ee40 0x556a52bb7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:52976/0 (socket says 172.21.15.114:52976) 2024-05-27T07:54:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e14c2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bae120 0x556a52bb7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41564/0 (socket says 172.21.15.114:41564) 2024-05-27T07:54:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e1542b700 1 -- 172.21.15.114:0/918026599 learned_addr learned my addr 172.21.15.114:0/918026599 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e15c2c700 1 --2- 172.21.15.114:0/918026599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bb4070 0x556a52bb8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e1542b700 1 -- 172.21.15.114:0/918026599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bb4070 msgr2=0x556a52bb8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e1542b700 1 --2- 172.21.15.114:0/918026599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bb4070 0x556a52bb8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e1542b700 1 -- 172.21.15.114:0/918026599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bae120 msgr2=0x556a52bb7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e1542b700 1 --2- 172.21.15.114:0/918026599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bae120 0x556a52bb7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e1542b700 1 -- 172.21.15.114:0/918026599 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a52a1b8d0 con 0x556a52a2ee40 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8e1542b700 1 --2- 172.21.15.114:0/918026599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52a2ee40 0x556a52bb7800 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c005f50 tx=0x7f8e0c005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaf6a0e11fa16730 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8df8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8df8ff9700 1 -- 172.21.15.114:0/918026599 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c00bdf0 con 0x556a52a2ee40 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.012+0000 7f8df8ff9700 1 -- 172.21.15.114:0/918026599 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(140..140 src has 1..140) ==== 9466+0+0 (secure 0 0 0) 0x7f8e0c00f460 con 0x556a52a2ee40 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.013+0000 7f8e260a4b80 1 -- 172.21.15.114:0/918026599 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e080092e0 con 0x556a52a2ee40 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.013+0000 7f8df8ff9700 1 -- 172.21.15.114:0/918026599 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f8e0c00f660 con 0x556a52a2ee40 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.013+0000 7f8e260a4b80 10 client.28391 fetch_fsmap learned FSMap version 82 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.013+0000 7f8e260a4b80 1 -- 172.21.15.114:0/918026599 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x556a52a85760 con 0x556a52a2ee40 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.013+0000 7f8df8ff9700 1 -- 172.21.15.114:0/918026599 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e080092e0 con 0x556a52a2ee40 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.013+0000 7f8df8ff9700 1 -- 172.21.15.114:0/918026599 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 82) ==== 5305+0+0 (secure 0 0 0) 0x7f8e0c02f020 con 0x556a52a2ee40 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.013+0000 7f8e260a4b80 10 client.28391 fetch_fsmap finished waiting for FSMap version 82 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.013+0000 7f8e260a4b80 10 client.28391 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.013+0000 7f8e260a4b80 20 client.28391 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:42.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.013+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/918026599 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a52bc7dd0 0x556a52bca280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.013+0000 7f8e260a4b80 4 client.28391 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:54:42.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.013+0000 7f8e260a4b80 1 -- 172.21.15.114:0/918026599 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a52addcc0 con 0x556a52bc7dd0 2024-05-27T07:54:42.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.013+0000 7f8e14c2a700 1 --2- 172.21.15.114:0/918026599 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a52bc7dd0 0x556a52bca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.014+0000 7f8df8ff9700 10 client.28391 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:42.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.014+0000 7f8e14c2a700 1 --2- 172.21.15.114:0/918026599 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a52bc7dd0 0x556a52bca280 crc :-1 s=READY pgs=426 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad6d9c273899d75b server_cookie=a5a560a982476499 in_seq=0 out_seq=0 2024-05-27T07:54:42.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.015+0000 7f8df8ff9700 1 -- 172.21.15.114:0/918026599 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556a52addcc0 con 0x556a52bc7dd0 2024-05-27T07:54:42.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.015+0000 7f8df8ff9700 10 client.28391 handle_command_reply: tid=0 2024-05-27T07:54:42.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.015+0000 7f8e260a4b80 1 client.28391 shutdown 2024-05-27T07:54:42.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.016+0000 7f8e260a4b80 1 -- 172.21.15.114:0/918026599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52a2ee40 msgr2=0x556a52bb7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:42.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.016+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/918026599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52a2ee40 0x556a52bb7800 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c005f50 tx=0x7f8e0c005040 comp rx=0 tx=0).stop 2024-05-27T07:54:42.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.016+0000 7f8e260a4b80 1 -- 172.21.15.114:0/918026599 shutdown_connections 2024-05-27T07:54:42.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.016+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/918026599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bb4070 0x556a52bb8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.016+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/918026599 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a52bc7dd0 0x556a52bca280 crc :-1 s=READY pgs=426 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.016+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/918026599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52a2ee40 0x556a52bb7800 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.016+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/918026599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bae120 0x556a52bb7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.016+0000 7f8df8ff9700 0 client.28391 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.016+0000 7f8e260a4b80 1 -- 172.21.15.114:0/918026599 >> 172.21.15.114:0/918026599 conn(0x556a52a31770 msgr2=0x556a52bacbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.016+0000 7f8e260a4b80 1 -- 172.21.15.114:0/918026599 shutdown_connections 2024-05-27T07:54:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.016+0000 7f8e260a4b80 1 -- 172.21.15.114:0/918026599 wait complete. 2024-05-27T07:54:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.016+0000 7f8e260a4b80 20 client.28391 trim_cache size 0 max 16384 2024-05-27T07:54:42.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e260a4b80 1 Processor -- start 2024-05-27T07:54:42.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e260a4b80 1 -- start start 2024-05-27T07:54:42.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e260a4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52a2ee40 0x556a52badbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e260a4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 0x556a52bb6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e260a4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52bb4070 0x556a52bc3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e260a4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e101b11d0 con 0x556a52a2ee40 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e260a4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8e101b18a0 con 0x556a52bb4070 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e260a4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e101b0050 con 0x556a52bae120 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e1542b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52a2ee40 0x556a52badbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e14c2a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 0x556a52bb6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e15c2c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52bb4070 0x556a52bc3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e1542b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52a2ee40 0x556a52badbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41574/0 (socket says 172.21.15.114:41574) 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e1542b700 1 -- 172.21.15.114:0/736541712 learned_addr learned my addr 172.21.15.114:0/736541712 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.038+0000 7f8e14c2a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 0x556a52bb6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54942/0 (socket says 172.21.15.114:54942) 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.039+0000 7f8e14c2a700 1 -- 172.21.15.114:0/736541712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52bb4070 msgr2=0x556a52bc3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.039+0000 7f8e14c2a700 1 --2- 172.21.15.114:0/736541712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52bb4070 0x556a52bc3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.039+0000 7f8e14c2a700 1 -- 172.21.15.114:0/736541712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52a2ee40 msgr2=0x556a52badbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.039+0000 7f8e14c2a700 1 --2- 172.21.15.114:0/736541712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52a2ee40 0x556a52badbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.039+0000 7f8e14c2a700 1 -- 172.21.15.114:0/736541712 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a52bb5d70 con 0x556a52bae120 2024-05-27T07:54:42.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.039+0000 7f8e15c2c700 1 --2- 172.21.15.114:0/736541712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52bb4070 0x556a52bc3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:42.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.039+0000 7f8e14c2a700 1 --2- 172.21.15.114:0/736541712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 0x556a52bb6760 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8e000dfec0 tx=0x7f8e000050b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5799e35011cf0594 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:42.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.039+0000 7f8dfaffd700 1 -- 172.21.15.114:0/736541712 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e000a5040 con 0x556a52bae120 2024-05-27T07:54:42.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.039+0000 7f8dfaffd700 1 -- 172.21.15.114:0/736541712 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e000c4070 con 0x556a52bae120 2024-05-27T07:54:42.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.039+0000 7f8dfaffd700 1 -- 172.21.15.114:0/736541712 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e000aa150 con 0x556a52bae120 2024-05-27T07:54:42.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.040+0000 7f8e260a4b80 1 -- 172.21.15.114:0/736541712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 msgr2=0x556a52bb6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:42.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.040+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/736541712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 0x556a52bb6760 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8e000dfec0 tx=0x7f8e000050b0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.040+0000 7f8e260a4b80 1 -- 172.21.15.114:0/736541712 shutdown_connections 2024-05-27T07:54:42.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.040+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/736541712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52bb4070 0x556a52bc3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.040+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/736541712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52a2ee40 0x556a52badbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.040+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/736541712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52bae120 0x556a52bb6760 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.040+0000 7f8e260a4b80 1 -- 172.21.15.114:0/736541712 >> 172.21.15.114:0/736541712 conn(0x556a52a31770 msgr2=0x556a52bb1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:42.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.040+0000 7f8e260a4b80 1 -- 172.21.15.114:0/736541712 shutdown_connections 2024-05-27T07:54:42.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.040+0000 7f8e260a4b80 1 -- 172.21.15.114:0/736541712 wait complete. 2024-05-27T07:54:42.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.040+0000 7f8e260a4b80 1 -- start start 2024-05-27T07:54:42.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.041+0000 7f8e260a4b80 1 Processor -- start 2024-05-27T07:54:42.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.041+0000 7f8e260a4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52a2ee40 0x556a52bc0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.041+0000 7f8e260a4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bae120 0x556a52bc0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.041+0000 7f8e260a4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52bb4070 0x556a52bcee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.041+0000 7f8e260a4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e101af980 con 0x556a52bae120 2024-05-27T07:54:42.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.041+0000 7f8e260a4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8e101ae570 con 0x556a52bb4070 2024-05-27T07:54:42.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.041+0000 7f8e260a4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e101aebe0 con 0x556a52a2ee40 2024-05-27T07:54:42.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.041+0000 7f8e260a4b80 10 client.0 _ll_register_callbacks cb 0x556a52a625b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:42.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.041+0000 7f8e14c2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bae120 0x556a52bc0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.041+0000 7f8e1542b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52a2ee40 0x556a52bc0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.041+0000 7f8e14c2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bae120 0x556a52bc0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41580/0 (socket says 172.21.15.114:41580) 2024-05-27T07:54:42.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.041+0000 7f8e14c2a700 1 -- 172.21.15.114:0/1746757182 learned_addr learned my addr 172.21.15.114:0/1746757182 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:42.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.041+0000 7f8e15c2c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52bb4070 0x556a52bcee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8e14c2a700 1 -- 172.21.15.114:0/1746757182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52a2ee40 msgr2=0x556a52bc0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8e14c2a700 1 --2- 172.21.15.114:0/1746757182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52a2ee40 0x556a52bc0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8e14c2a700 1 -- 172.21.15.114:0/1746757182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52bb4070 msgr2=0x556a52bcee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8e14c2a700 1 --2- 172.21.15.114:0/1746757182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52bb4070 0x556a52bcee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8e14c2a700 1 -- 172.21.15.114:0/1746757182 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a52bcf640 con 0x556a52bae120 2024-05-27T07:54:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8e15c2c700 1 --2- 172.21.15.114:0/1746757182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52bb4070 0x556a52bcee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:54:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8e14c2a700 1 --2- 172.21.15.114:0/1746757182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bae120 0x556a52bc0550 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f8e000e0440 tx=0x7f8e00004e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37f9237aab67672 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8df8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:54:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8df8ff9700 1 -- 172.21.15.114:0/1746757182 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e000a5040 con 0x556a52bae120 2024-05-27T07:54:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8df8ff9700 1 -- 172.21.15.114:0/1746757182 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(140..140 src has 1..140) ==== 9466+0+0 (secure 0 0 0) 0x7f8e0006f2c0 con 0x556a52bae120 2024-05-27T07:54:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8e260a4b80 1 -- 172.21.15.114:0/1746757182 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e000c4070 con 0x556a52bae120 2024-05-27T07:54:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8df8ff9700 1 -- 172.21.15.114:0/1746757182 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f8e0000ce40 con 0x556a52bae120 2024-05-27T07:54:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8e260a4b80 10 client.19050 fetch_fsmap learned FSMap version 82 2024-05-27T07:54:42.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8e260a4b80 1 -- 172.21.15.114:0/1746757182 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x556a52c32340 con 0x556a52bae120 2024-05-27T07:54:42.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8df8ff9700 1 -- 172.21.15.114:0/1746757182 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e000c4070 con 0x556a52bae120 2024-05-27T07:54:42.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.042+0000 7f8df8ff9700 1 -- 172.21.15.114:0/1746757182 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 82) ==== 5305+0+0 (secure 0 0 0) 0x7f8e00016330 con 0x556a52bae120 2024-05-27T07:54:42.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.043+0000 7f8e260a4b80 10 client.19050 fetch_fsmap finished waiting for FSMap version 82 2024-05-27T07:54:42.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.043+0000 7f8e260a4b80 10 client.19050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:42.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.043+0000 7f8e260a4b80 20 client.19050 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:42.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.043+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/1746757182 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a52c329f0 0x556a52c34e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:42.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.043+0000 7f8e260a4b80 4 client.19050 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:54:42.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.043+0000 7f8e260a4b80 1 -- 172.21.15.114:0/1746757182 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556a52ad74a0 con 0x556a52c329f0 2024-05-27T07:54:42.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.043+0000 7f8e1542b700 1 --2- 172.21.15.114:0/1746757182 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a52c329f0 0x556a52c34e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:42.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.043+0000 7f8df8ff9700 10 client.19050 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:42.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.043+0000 7f8e1542b700 1 --2- 172.21.15.114:0/1746757182 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a52c329f0 0x556a52c34e10 crc :-1 s=READY pgs=427 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7e08e5f8fc9607 server_cookie=ef53bfac894e74fd in_seq=0 out_seq=0 2024-05-27T07:54:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.049+0000 7f8df8ff9700 1 -- 172.21.15.114:0/1746757182 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556a52ad74a0 con 0x556a52c329f0 2024-05-27T07:54:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.049+0000 7f8df8ff9700 10 client.19050 handle_command_reply: tid=0 2024-05-27T07:54:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.049+0000 7f8e260a4b80 1 client.19050 shutdown 2024-05-27T07:54:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.049+0000 7f8e260a4b80 1 -- 172.21.15.114:0/1746757182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bae120 msgr2=0x556a52bc0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.049+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/1746757182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bae120 0x556a52bc0550 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f8e000e0440 tx=0x7f8e00004e90 comp rx=0 tx=0).stop 2024-05-27T07:54:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.049+0000 7f8e260a4b80 1 -- 172.21.15.114:0/1746757182 shutdown_connections 2024-05-27T07:54:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.049+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/1746757182 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556a52c329f0 0x556a52c34e10 crc :-1 s=READY pgs=427 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.049+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/1746757182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556a52bb4070 0x556a52bcee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.049+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/1746757182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a52a2ee40 0x556a52bc0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.049+0000 7f8df8ff9700 0 client.19050 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.049+0000 7f8e260a4b80 1 --2- 172.21.15.114:0/1746757182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a52bae120 0x556a52bc0550 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.049+0000 7f8e260a4b80 1 -- 172.21.15.114:0/1746757182 >> 172.21.15.114:0/1746757182 conn(0x556a52a31770 msgr2=0x556a52ba24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e260a4b80 1 -- 172.21.15.114:0/1746757182 shutdown_connections 2024-05-27T07:54:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e260a4b80 1 -- 172.21.15.114:0/1746757182 wait complete. 2024-05-27T07:54:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e260a4b80 20 client.19050 trim_cache size 0 max 16384 2024-05-27T07:54:42.052 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:54:42.052 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:54:42.052 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:54:42.052 INFO:teuthology.orchestra.run.smithi114.stdout: "732d7839-9f93-4aff-80d2-86cfafbf6bc3": { 2024-05-27T07:54:42.053 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:54:42.053 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "732d7839-9f93-4aff-80d2-86cfafbf6bc3", 2024-05-27T07:54:42.053 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:54:42.053 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:54:42.053 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:54:42.053 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:54:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e1768f700 1 -- 172.21.15.114:0/2264403064 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8dfc0b29f0 msgr2=0x7f8dfc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e1768f700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8dfc0b29f0 0x7f8dfc0b4f00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c000c00 tx=0x7f8e0c01a040 comp rx=0 tx=0).stop 2024-05-27T07:54:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e1768f700 1 -- 172.21.15.114:0/2264403064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e10108f00 msgr2=0x7f8e1019b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e1768f700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e10108f00 0x7f8e1019b390 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000e970 tx=0x7f8e0000ec80 comp rx=0 tx=0).stop 2024-05-27T07:54:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e1768f700 1 -- 172.21.15.114:0/2264403064 shutdown_connections 2024-05-27T07:54:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e1768f700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e10109850 0x7f8e1019f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e1768f700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f8dfc0b29f0 0x7f8dfc0b4f00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e1768f700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e101045d0 0x7f8e1019adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e1768f700 1 --2- 172.21.15.114:0/2264403064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e10108f00 0x7f8e1019b390 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e1768f700 1 -- 172.21.15.114:0/2264403064 >> 172.21.15.114:0/2264403064 conn(0x7f8e100ffe60 msgr2=0x7f8e101086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e1768f700 1 -- 172.21.15.114:0/2264403064 shutdown_connections 2024-05-27T07:54:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:54:42.050+0000 7f8e1768f700 1 -- 172.21.15.114:0/2264403064 wait complete. 2024-05-27T07:54:42.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 13885 17.24 MB/sec warmup 54 sec latency 229.248 ms 2024-05-27T07:54:42.699 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:42 smithi114 ceph-mon[32038]: pgmap v508: 129 pgs: 129 active+clean; 302 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.7 MiB/s wr, 583 op/s 2024-05-27T07:54:42.699 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:42 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:54:42.725 INFO:tasks.cephfs.filesystem:scrub status for tag:732d7839-9f93-4aff-80d2-86cfafbf6bc3 - {'path': '/', 'tag': '732d7839-9f93-4aff-80d2-86cfafbf6bc3', 'options': 'recursive,force'} 2024-05-27T07:54:42.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:42 smithi163 ceph-mon[36451]: pgmap v508: 129 pgs: 129 active+clean; 302 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.7 MiB/s wr, 583 op/s 2024-05-27T07:54:42.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:42 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:54:42.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:42 smithi186 ceph-mon[36647]: pgmap v508: 129 pgs: 129 active+clean; 302 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.7 MiB/s wr, 583 op/s 2024-05-27T07:54:42.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:42 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:54:43.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 14168 17.18 MB/sec warmup 55 sec latency 156.066 ms 2024-05-27T07:54:44.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 14438 17.13 MB/sec warmup 56 sec latency 88.800 ms 2024-05-27T07:54:44.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:44 smithi163 ceph-mon[36451]: pgmap v509: 129 pgs: 129 active+clean; 286 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 580 op/s 2024-05-27T07:54:44.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:44 smithi186 ceph-mon[36647]: pgmap v509: 129 pgs: 129 active+clean; 286 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 580 op/s 2024-05-27T07:54:45.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:44 smithi114 ceph-mon[32038]: pgmap v509: 129 pgs: 129 active+clean; 286 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 580 op/s 2024-05-27T07:54:45.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 14698 17.04 MB/sec warmup 57 sec latency 209.533 ms 2024-05-27T07:54:46.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 15023 16.89 MB/sec warmup 58 sec latency 178.565 ms 2024-05-27T07:54:46.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:46 smithi163 ceph-mon[36451]: pgmap v510: 129 pgs: 129 active+clean; 291 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 573 op/s 2024-05-27T07:54:46.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:46 smithi186 ceph-mon[36647]: pgmap v510: 129 pgs: 129 active+clean; 291 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 573 op/s 2024-05-27T07:54:47.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:46 smithi114 ceph-mon[32038]: pgmap v510: 129 pgs: 129 active+clean; 291 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 573 op/s 2024-05-27T07:54:47.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 15207 16.73 MB/sec warmup 59 sec latency 552.445 ms 2024-05-27T07:54:48.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 15637 16.87 MB/sec warmup 60 sec latency 175.820 ms 2024-05-27T07:54:48.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:48 smithi163 ceph-mon[36451]: pgmap v511: 129 pgs: 129 active+clean; 300 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 591 op/s 2024-05-27T07:54:48.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:48 smithi186 ceph-mon[36647]: pgmap v511: 129 pgs: 129 active+clean; 300 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 591 op/s 2024-05-27T07:54:49.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:48 smithi114 ceph-mon[32038]: pgmap v511: 129 pgs: 129 active+clean; 300 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 591 op/s 2024-05-27T07:54:49.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 16192 17.02 MB/sec warmup 61 sec latency 167.326 ms 2024-05-27T07:54:50.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 16652 17.18 MB/sec warmup 62 sec latency 201.461 ms 2024-05-27T07:54:50.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:50 smithi163 ceph-mon[36451]: pgmap v512: 129 pgs: 129 active+clean; 301 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 550 op/s 2024-05-27T07:54:50.977 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:50 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:54:50.977 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:50 smithi163 systemd[1]: Stopped Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:54:50.977 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:50 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Start request repeated too quickly. 2024-05-27T07:54:50.977 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:50 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-05-27T07:54:50.977 INFO:journalctl@ceph.mds.e.smithi163.stdout:May 27 07:54:50 smithi163 systemd[1]: Failed to start Ceph mds.e for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:54:50.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:50 smithi186 ceph-mon[36647]: pgmap v512: 129 pgs: 129 active+clean; 301 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 550 op/s 2024-05-27T07:54:51.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:50 smithi114 ceph-mon[32038]: pgmap v512: 129 pgs: 129 active+clean; 301 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 550 op/s 2024-05-27T07:54:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.209+0000 7f3e547f3640 1 -- 172.21.15.114:0/3514841108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e4c103a30 msgr2=0x7f3e4c10f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:51.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.209+0000 7f3e547f3640 1 --2- 172.21.15.114:0/3514841108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e4c103a30 0x7f3e4c10f8e0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009ca0 tx=0x7f3e4402a8c0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.210+0000 7f3e547f3640 1 -- 172.21.15.114:0/3514841108 shutdown_connections 2024-05-27T07:54:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.210+0000 7f3e547f3640 1 --2- 172.21.15.114:0/3514841108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e4c103a30 0x7f3e4c10f8e0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.210+0000 7f3e547f3640 1 --2- 172.21.15.114:0/3514841108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e4c109260 0x7f3e4c1033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.210+0000 7f3e547f3640 1 --2- 172.21.15.114:0/3514841108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e4c108dc0 0x7f3e4c102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.210+0000 7f3e547f3640 1 -- 172.21.15.114:0/3514841108 >> 172.21.15.114:0/3514841108 conn(0x7f3e4c0fd9c0 msgr2=0x7f3e4c0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.213+0000 7f3e547f3640 1 -- 172.21.15.114:0/3514841108 shutdown_connections 2024-05-27T07:54:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.213+0000 7f3e547f3640 1 -- 172.21.15.114:0/3514841108 wait complete. 2024-05-27T07:54:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.214+0000 7f3e547f3640 1 Processor -- start 2024-05-27T07:54:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.214+0000 7f3e547f3640 1 -- start start 2024-05-27T07:54:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.215+0000 7f3e547f3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e4c103a30 0x7f3e4c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.215+0000 7f3e547f3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e4c108dc0 0x7f3e4c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.215+0000 7f3e547f3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e4c109260 0x7f3e4c06f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.215+0000 7f3e5258f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e4c103a30 0x7f3e4c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.215+0000 7f3e5258f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e4c103a30 0x7f3e4c06eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:53112/0 (socket says 172.21.15.114:53112) 2024-05-27T07:54:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.215+0000 7f3e5258f640 1 -- 172.21.15.114:0/1305394360 learned_addr learned my addr 172.21.15.114:0/1305394360 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.215+0000 7f3e547f3640 1 -- 172.21.15.114:0/1305394360 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c112fd0 con 0x7f3e4c109260 2024-05-27T07:54:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.215+0000 7f3e547f3640 1 -- 172.21.15.114:0/1305394360 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c112e50 con 0x7f3e4c108dc0 2024-05-27T07:54:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.215+0000 7f3e547f3640 1 -- 172.21.15.114:0/1305394360 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c113150 con 0x7f3e4c103a30 2024-05-27T07:54:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.215+0000 7f3e51d8e640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e4c108dc0 0x7f3e4c06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.218+0000 7f3e52d90640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e4c109260 0x7f3e4c06f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.220+0000 7f3e5258f640 1 -- 172.21.15.114:0/1305394360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e4c108dc0 msgr2=0x7f3e4c06f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.220+0000 7f3e5258f640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e4c108dc0 0x7f3e4c06f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.220+0000 7f3e5258f640 1 -- 172.21.15.114:0/1305394360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e4c109260 msgr2=0x7f3e4c06f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.220+0000 7f3e5258f640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e4c109260 0x7f3e4c06f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.220+0000 7f3e5258f640 1 -- 172.21.15.114:0/1305394360 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4c19e4d0 con 0x7f3e4c103a30 2024-05-27T07:54:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.225+0000 7f3e5258f640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e4c103a30 0x7f3e4c06eb30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800d770 tx=0x7f3e4800dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.228+0000 7f3e437fe640 1 -- 172.21.15.114:0/1305394360 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4801c070 con 0x7f3e4c103a30 2024-05-27T07:54:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.228+0000 7f3e437fe640 1 -- 172.21.15.114:0/1305394360 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e48009cd0 con 0x7f3e4c103a30 2024-05-27T07:54:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.228+0000 7f3e437fe640 1 -- 172.21.15.114:0/1305394360 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48020c20 con 0x7f3e4c103a30 2024-05-27T07:54:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.231+0000 7f3e547f3640 1 -- 172.21.15.114:0/1305394360 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4c10d870 con 0x7f3e4c103a30 2024-05-27T07:54:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.231+0000 7f3e547f3640 1 -- 172.21.15.114:0/1305394360 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e4c10dd40 con 0x7f3e4c103a30 2024-05-27T07:54:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.236+0000 7f3e5b780740 1 Processor -- start 2024-05-27T07:54:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.237+0000 7f3e5b780740 1 -- start start 2024-05-27T07:54:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.239+0000 7f3e5b780740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 0x5602ca0f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.239+0000 7f3e5b780740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d292f0 0x5602ca0f3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.239+0000 7f3e5b780740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d314d0 0x5602ca0f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.239+0000 7f3e5b780740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5602ca065c30 con 0x5602c9d521b0 2024-05-27T07:54:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.239+0000 7f3e5b780740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5602ca0557d0 con 0x5602c9d292f0 2024-05-27T07:54:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.239+0000 7f3e5b780740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5602ca059810 con 0x5602c9d314d0 2024-05-27T07:54:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.241+0000 7f3e437fe640 1 -- 172.21.15.114:0/1305394360 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f3e48009820 con 0x7f3e4c103a30 2024-05-27T07:54:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.241+0000 7f3e52d90640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 0x5602ca0f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.241+0000 7f3e52d90640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 0x5602ca0f33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37044/0 (socket says 172.21.15.114:37044) 2024-05-27T07:54:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.241+0000 7f3e52d90640 1 -- 172.21.15.114:0/304846227 learned_addr learned my addr 172.21.15.114:0/304846227 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.242+0000 7f3e52d90640 1 -- 172.21.15.114:0/304846227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d314d0 msgr2=0x5602ca0f3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.242+0000 7f3e52d90640 1 --2- 172.21.15.114:0/304846227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d314d0 0x5602ca0f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.242+0000 7f3e52d90640 1 -- 172.21.15.114:0/304846227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d292f0 msgr2=0x5602ca0f3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.243+0000 7f3e437fe640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3e380b2b20 0x7f3e380b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.243+0000 7f3e5258f640 1 --2- 172.21.15.114:0/304846227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d292f0 0x5602ca0f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.243+0000 7f3e52d90640 1 --2- 172.21.15.114:0/304846227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d292f0 0x5602ca0f3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.243+0000 7f3e52d90640 1 -- 172.21.15.114:0/304846227 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602c9d6a210 con 0x5602c9d521b0 2024-05-27T07:54:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.243+0000 7f3e437fe640 1 -- 172.21.15.114:0/1305394360 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 9466+0+0 (secure 0 0 0) 0x7f3e480ea8a0 con 0x7f3e4c103a30 2024-05-27T07:54:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.244+0000 7f3e51d8e640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3e380b2b20 0x7f3e380b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.245+0000 7f3e52d90640 1 --2- 172.21.15.114:0/304846227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 0x5602ca0f33d0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f3e44005cc0 tx=0x7f3e4402f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f894a6b9ef3312 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.245+0000 7f3e51d8e640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3e380b2b20 0x7f3e380b5030 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c106ad0 tx=0x7f3e3c0128c0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.246+0000 7f3e40ff9640 1 -- 172.21.15.114:0/304846227 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e44031cf0 con 0x5602c9d521b0 2024-05-27T07:54:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.247+0000 7f3e40ff9640 1 -- 172.21.15.114:0/304846227 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e44036040 con 0x5602c9d521b0 2024-05-27T07:54:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.248+0000 7f3e5b780740 1 -- 172.21.15.114:0/304846227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 msgr2=0x5602ca0f33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.248+0000 7f3e5b780740 1 --2- 172.21.15.114:0/304846227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 0x5602ca0f33d0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f3e44005cc0 tx=0x7f3e4402f420 comp rx=0 tx=0).stop 2024-05-27T07:54:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.250+0000 7f3e5b780740 1 -- 172.21.15.114:0/304846227 shutdown_connections 2024-05-27T07:54:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.250+0000 7f3e5b780740 1 --2- 172.21.15.114:0/304846227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d314d0 0x5602ca0f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.250+0000 7f3e5b780740 1 --2- 172.21.15.114:0/304846227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d292f0 0x5602ca0f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.250+0000 7f3e5b780740 1 --2- 172.21.15.114:0/304846227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 0x5602ca0f33d0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.250+0000 7f3e5b780740 1 -- 172.21.15.114:0/304846227 >> 172.21.15.114:0/304846227 conn(0x5602c9d46300 msgr2=0x5602ca0efb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.250+0000 7f3e5b780740 1 -- 172.21.15.114:0/304846227 shutdown_connections 2024-05-27T07:54:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.250+0000 7f3e5b780740 1 -- 172.21.15.114:0/304846227 wait complete. 2024-05-27T07:54:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.251+0000 7f3e5b780740 1 -- start start 2024-05-27T07:54:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.251+0000 7f3e5b780740 1 Processor -- start 2024-05-27T07:54:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.253+0000 7f3e5b780740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d292f0 0x5602ca0fda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.253+0000 7f3e5b780740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d314d0 0x5602ca0fdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.253+0000 7f3e5b780740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d521b0 0x5602ca1038b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.253+0000 7f3e5b780740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5602c9d3b790 con 0x5602c9d314d0 2024-05-27T07:54:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.253+0000 7f3e5b780740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5602ca011890 con 0x5602c9d521b0 2024-05-27T07:54:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.253+0000 7f3e5b780740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5602ca1004f0 con 0x5602c9d292f0 2024-05-27T07:54:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.253+0000 7f3e5b780740 10 client.0 _ll_register_callbacks cb 0x5602c9d69a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.254+0000 7f3e52d90640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d521b0 0x5602ca1038b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.254+0000 7f3e52d90640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d521b0 0x5602ca1038b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:38500/0 (socket says 172.21.15.114:38500) 2024-05-27T07:54:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.254+0000 7f3e52d90640 1 -- 172.21.15.114:0/3756820886 learned_addr learned my addr 172.21.15.114:0/3756820886 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.254+0000 7f3e5258f640 1 --2- 172.21.15.114:0/3756820886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d292f0 0x5602ca0fda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.254+0000 7f3e51d8e640 1 --2- 172.21.15.114:0/3756820886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d314d0 0x5602ca0fdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.255+0000 7f3e5258f640 1 -- 172.21.15.114:0/3756820886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d521b0 msgr2=0x5602ca1038b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.255+0000 7f3e5258f640 1 --2- 172.21.15.114:0/3756820886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d521b0 0x5602ca1038b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.255+0000 7f3e5258f640 1 -- 172.21.15.114:0/3756820886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d314d0 msgr2=0x5602ca0fdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.255+0000 7f3e5258f640 1 --2- 172.21.15.114:0/3756820886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d314d0 0x5602ca0fdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.255+0000 7f3e5258f640 1 -- 172.21.15.114:0/3756820886 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602c9e36620 con 0x5602c9d292f0 2024-05-27T07:54:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.256+0000 7f3e5258f640 1 --2- 172.21.15.114:0/3756820886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d292f0 0x5602ca0fda90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3e480e7cf0 tx=0x7f3e4800de80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4425c982090b1214 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.256+0000 7f3e35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:54:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.256+0000 7f3e35ffb640 1 -- 172.21.15.114:0/3756820886 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e480b1390 con 0x5602c9d292f0 2024-05-27T07:54:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.256+0000 7f3e5b780740 1 -- 172.21.15.114:0/3756820886 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5602ca067b80 con 0x5602c9d292f0 2024-05-27T07:54:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.257+0000 7f3e35ffb640 1 -- 172.21.15.114:0/3756820886 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(140..140 src has 1..140) ==== 9466+0+0 (secure 0 0 0) 0x7f3e480adb10 con 0x5602c9d292f0 2024-05-27T07:54:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.257+0000 7f3e35ffb640 1 -- 172.21.15.114:0/3756820886 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f3e480b1f80 con 0x5602c9d292f0 2024-05-27T07:54:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.259+0000 7f3e5b780740 10 client.28564 fetch_fsmap learned FSMap version 82 2024-05-27T07:54:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.259+0000 7f3e5b780740 1 -- 172.21.15.114:0/3756820886 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x5602ca10c710 con 0x5602c9d292f0 2024-05-27T07:54:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.260+0000 7f3e35ffb640 1 -- 172.21.15.114:0/3756820886 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5602ca067b80 con 0x5602c9d292f0 2024-05-27T07:54:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.260+0000 7f3e35ffb640 1 -- 172.21.15.114:0/3756820886 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 82) ==== 5305+0+0 (secure 0 0 0) 0x7f3e480ea430 con 0x5602c9d292f0 2024-05-27T07:54:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.260+0000 7f3e5b780740 10 client.28564 fetch_fsmap finished waiting for FSMap version 82 2024-05-27T07:54:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.260+0000 7f3e5b780740 10 client.28564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.260+0000 7f3e5b780740 20 client.28564 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.260+0000 7f3e5b780740 1 --2- 172.21.15.114:0/3756820886 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602c9d10d80 0x5602ca10ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.260+0000 7f3e5b780740 4 client.28564 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:54:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.260+0000 7f3e5b780740 1 -- 172.21.15.114:0/3756820886 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602ca061360 con 0x5602c9d10d80 2024-05-27T07:54:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.272+0000 7f3e51d8e640 1 --2- 172.21.15.114:0/3756820886 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602c9d10d80 0x5602ca10ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.278+0000 7f3e35ffb640 10 client.28564 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.278+0000 7f3e51d8e640 1 --2- 172.21.15.114:0/3756820886 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602c9d10d80 0x5602ca10ff30 crc :-1 s=READY pgs=428 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e67cebbe8cc572f server_cookie=1fdd5292773b382f in_seq=0 out_seq=0 2024-05-27T07:54:51.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.283+0000 7f3e35ffb640 1 -- 172.21.15.114:0/3756820886 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5602ca061360 con 0x5602c9d10d80 2024-05-27T07:54:51.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.283+0000 7f3e35ffb640 10 client.28564 handle_command_reply: tid=0 2024-05-27T07:54:51.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.283+0000 7f3e5b780740 1 client.28564 shutdown 2024-05-27T07:54:51.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.284+0000 7f3e5b780740 1 -- 172.21.15.114:0/3756820886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d292f0 msgr2=0x5602ca0fda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.284+0000 7f3e5b780740 1 --2- 172.21.15.114:0/3756820886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d292f0 0x5602ca0fda90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3e480e7cf0 tx=0x7f3e4800de80 comp rx=0 tx=0).stop 2024-05-27T07:54:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.284+0000 7f3e5b780740 1 -- 172.21.15.114:0/3756820886 shutdown_connections 2024-05-27T07:54:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.284+0000 7f3e5b780740 1 --2- 172.21.15.114:0/3756820886 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602c9d10d80 0x5602ca10ff30 crc :-1 s=READY pgs=428 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.284+0000 7f3e5b780740 1 --2- 172.21.15.114:0/3756820886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d521b0 0x5602ca1038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.284+0000 7f3e5b780740 1 --2- 172.21.15.114:0/3756820886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d314d0 0x5602ca0fdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.284+0000 7f3e5b780740 1 --2- 172.21.15.114:0/3756820886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d292f0 0x5602ca0fda90 secure :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3e480e7cf0 tx=0x7f3e4800de80 comp rx=0 tx=0).stop 2024-05-27T07:54:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.284+0000 7f3e5b780740 1 -- 172.21.15.114:0/3756820886 >> 172.21.15.114:0/3756820886 conn(0x5602c9d46300 msgr2=0x5602ca0f3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.284+0000 7f3e35ffb640 0 client.28564 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.285+0000 7f3e5b780740 1 -- 172.21.15.114:0/3756820886 shutdown_connections 2024-05-27T07:54:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.286+0000 7f3e5b780740 1 -- 172.21.15.114:0/3756820886 wait complete. 2024-05-27T07:54:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.287+0000 7f3e5b780740 20 client.28564 trim_cache size 0 max 16384 2024-05-27T07:54:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.320+0000 7f3e5b780740 1 Processor -- start 2024-05-27T07:54:51.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.321+0000 7f3e5b780740 1 -- start start 2024-05-27T07:54:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.321+0000 7f3e5b780740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d314d0 0x5602ca0f7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.321+0000 7f3e5b780740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 0x5602ca0fd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.321+0000 7f3e5b780740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9ce61c0 0x5602ca0fdd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.322+0000 7f3e5b780740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1a7540 con 0x5602c9d521b0 2024-05-27T07:54:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.322+0000 7f3e5b780740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1a7c10 con 0x5602c9d314d0 2024-05-27T07:54:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.322+0000 7f3e5b780740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1a63e0 con 0x5602c9ce61c0 2024-05-27T07:54:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.322+0000 7f3e51d8e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 0x5602ca0fd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.322+0000 7f3e51d8e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 0x5602ca0fd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37058/0 (socket says 172.21.15.114:37058) 2024-05-27T07:54:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.322+0000 7f3e51d8e640 1 -- 172.21.15.114:0/3769883856 learned_addr learned my addr 172.21.15.114:0/3769883856 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.323+0000 7f3e51d8e640 1 -- 172.21.15.114:0/3769883856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9ce61c0 msgr2=0x5602ca0fdd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:54:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.323+0000 7f3e51d8e640 1 --2- 172.21.15.114:0/3769883856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9ce61c0 0x5602ca0fdd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.323+0000 7f3e51d8e640 1 -- 172.21.15.114:0/3769883856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d314d0 msgr2=0x5602ca0f7730 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:54:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.323+0000 7f3e51d8e640 1 --2- 172.21.15.114:0/3769883856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d314d0 0x5602ca0f7730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.323+0000 7f3e51d8e640 1 -- 172.21.15.114:0/3769883856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602ca0fbdd0 con 0x5602c9d521b0 2024-05-27T07:54:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.323+0000 7f3e51d8e640 1 --2- 172.21.15.114:0/3769883856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 0x5602ca0fd840 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c0702a0 tx=0x7f3e3c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4395e4e53d9e230 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.324+0000 7f3e37fff640 1 -- 172.21.15.114:0/3769883856 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0210d0 con 0x5602c9d521b0 2024-05-27T07:54:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.324+0000 7f3e37fff640 1 -- 172.21.15.114:0/3769883856 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e3c021270 con 0x5602c9d521b0 2024-05-27T07:54:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.324+0000 7f3e37fff640 1 -- 172.21.15.114:0/3769883856 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c005280 con 0x5602c9d521b0 2024-05-27T07:54:51.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.325+0000 7f3e5b780740 1 -- 172.21.15.114:0/3769883856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 msgr2=0x5602ca0fd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:51.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.325+0000 7f3e5b780740 1 --2- 172.21.15.114:0/3769883856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 0x5602ca0fd840 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c0702a0 tx=0x7f3e3c015000 comp rx=0 tx=0).stop 2024-05-27T07:54:51.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.330+0000 7f3e5b780740 1 -- 172.21.15.114:0/3769883856 shutdown_connections 2024-05-27T07:54:51.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.330+0000 7f3e5b780740 1 --2- 172.21.15.114:0/3769883856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9ce61c0 0x5602ca0fdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.330+0000 7f3e5b780740 1 --2- 172.21.15.114:0/3769883856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d521b0 0x5602ca0fd840 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.330+0000 7f3e5b780740 1 --2- 172.21.15.114:0/3769883856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9d314d0 0x5602ca0f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.330+0000 7f3e5b780740 1 -- 172.21.15.114:0/3769883856 >> 172.21.15.114:0/3769883856 conn(0x5602c9d46300 msgr2=0x5602ca0f7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.334+0000 7f3e5b780740 1 -- 172.21.15.114:0/3769883856 shutdown_connections 2024-05-27T07:54:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.334+0000 7f3e5b780740 1 -- 172.21.15.114:0/3769883856 wait complete. 2024-05-27T07:54:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.334+0000 7f3e5b780740 1 -- start start 2024-05-27T07:54:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.335+0000 7f3e5b780740 1 Processor -- start 2024-05-27T07:54:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.335+0000 7f3e5b780740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9ce61c0 0x5602ca107060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.335+0000 7f3e5b780740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d314d0 0x5602ca1075a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.336+0000 7f3e5b780740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d521b0 0x5602ca176600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.336+0000 7f3e5b780740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1a5d10 con 0x5602c9d314d0 2024-05-27T07:54:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.336+0000 7f3e51d8e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d314d0 0x5602ca1075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.336+0000 7f3e5258f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9ce61c0 0x5602ca107060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.336+0000 7f3e5b780740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1a4900 con 0x5602c9ce61c0 2024-05-27T07:54:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.336+0000 7f3e5b780740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1a4f70 con 0x5602c9d521b0 2024-05-27T07:54:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.336+0000 7f3e51d8e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d314d0 0x5602ca1075a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37064/0 (socket says 172.21.15.114:37064) 2024-05-27T07:54:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.336+0000 7f3e51d8e640 1 -- 172.21.15.114:0/1562410860 learned_addr learned my addr 172.21.15.114:0/1562410860 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:54:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.336+0000 7f3e5b780740 10 client.0 _ll_register_callbacks cb 0x7f3e480b16e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:54:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.336+0000 7f3e52d90640 1 --2- 172.21.15.114:0/1562410860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d521b0 0x5602ca176600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.337+0000 7f3e51d8e640 1 -- 172.21.15.114:0/1562410860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d521b0 msgr2=0x5602ca176600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.337+0000 7f3e51d8e640 1 --2- 172.21.15.114:0/1562410860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d521b0 0x5602ca176600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.337+0000 7f3e51d8e640 1 -- 172.21.15.114:0/1562410860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9ce61c0 msgr2=0x5602ca107060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:54:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.337+0000 7f3e51d8e640 1 --2- 172.21.15.114:0/1562410860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9ce61c0 0x5602ca107060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.337+0000 7f3e51d8e640 1 -- 172.21.15.114:0/1562410860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602ca176b40 con 0x5602c9d314d0 2024-05-27T07:54:51.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.338+0000 7f3e51d8e640 1 --2- 172.21.15.114:0/1562410860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d314d0 0x5602ca1075a0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c021b40 tx=0x7f3e3c021b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7904dc11780655f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:54:51.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.338+0000 7f3e35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:54:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.338+0000 7f3e35ffb640 1 -- 172.21.15.114:0/1562410860 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c036020 con 0x5602c9d314d0 2024-05-27T07:54:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.339+0000 7f3e5b780740 1 -- 172.21.15.114:0/1562410860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e3c021270 con 0x5602c9d314d0 2024-05-27T07:54:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.339+0000 7f3e35ffb640 1 -- 172.21.15.114:0/1562410860 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(140..140 src has 1..140) ==== 9466+0+0 (secure 0 0 0) 0x7f3e3c005230 con 0x5602c9d314d0 2024-05-27T07:54:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.340+0000 7f3e35ffb640 1 -- 172.21.15.114:0/1562410860 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f3e3c005470 con 0x5602c9d314d0 2024-05-27T07:54:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.340+0000 7f3e5b780740 10 client.19068 fetch_fsmap learned FSMap version 82 2024-05-27T07:54:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.340+0000 7f3e5b780740 1 -- 172.21.15.114:0/1562410860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x5602ca177c50 con 0x5602c9d314d0 2024-05-27T07:54:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.340+0000 7f3e35ffb640 1 -- 172.21.15.114:0/1562410860 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e3c005610 con 0x5602c9d314d0 2024-05-27T07:54:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.342+0000 7f3e35ffb640 1 -- 172.21.15.114:0/1562410860 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 82) ==== 5305+0+0 (secure 0 0 0) 0x7f3e3c0035f0 con 0x5602c9d314d0 2024-05-27T07:54:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.342+0000 7f3e5b780740 10 client.19068 fetch_fsmap finished waiting for FSMap version 82 2024-05-27T07:54:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.342+0000 7f3e5b780740 10 client.19068 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:54:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.342+0000 7f3e5b780740 20 client.19068 populate_metadata read hostname 'smithi114' 2024-05-27T07:54:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.342+0000 7f3e5b780740 1 --2- 172.21.15.114:0/1562410860 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602ca178300 0x5602ca17a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:54:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.342+0000 7f3e5b780740 4 client.19068 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:54:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.342+0000 7f3e5b780740 1 -- 172.21.15.114:0/1562410860 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f3e3c021270 con 0x5602ca178300 2024-05-27T07:54:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.342+0000 7f3e5258f640 1 --2- 172.21.15.114:0/1562410860 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602ca178300 0x5602ca17a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:54:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.343+0000 7f3e35ffb640 10 client.19068 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:51.343+0000 7f3e5258f640 1 --2- 172.21.15.114:0/1562410860 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602ca178300 0x5602ca17a720 crc :-1 s=READY pgs=429 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ac620bf00088782 server_cookie=b53e79f75c7baf25 in_seq=0 out_seq=0 2024-05-27T07:54:51.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 16976 17.26 MB/sec warmup 63 sec latency 229.779 ms 2024-05-27T07:54:52.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 16978 16.99 MB/sec warmup 64 sec latency 1148.695 ms 2024-05-27T07:54:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e35ffb640 1 -- 172.21.15.114:0/1562410860 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7f3e3c021270 con 0x5602ca178300 2024-05-27T07:54:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e35ffb640 10 client.19068 handle_command_reply: tid=0 2024-05-27T07:54:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e5b780740 1 client.19068 shutdown 2024-05-27T07:54:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e5b780740 1 -- 172.21.15.114:0/1562410860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d314d0 msgr2=0x5602ca1075a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e5b780740 1 --2- 172.21.15.114:0/1562410860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d314d0 0x5602ca1075a0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c021b40 tx=0x7f3e3c021b70 comp rx=0 tx=0).stop 2024-05-27T07:54:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e5b780740 1 -- 172.21.15.114:0/1562410860 shutdown_connections 2024-05-27T07:54:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e5b780740 1 --2- 172.21.15.114:0/1562410860 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602ca178300 0x5602ca17a720 crc :-1 s=READY pgs=429 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e5b780740 1 --2- 172.21.15.114:0/1562410860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602c9d521b0 0x5602ca176600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e5b780740 1 --2- 172.21.15.114:0/1562410860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602c9d314d0 0x5602ca1075a0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e5b780740 1 --2- 172.21.15.114:0/1562410860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602c9ce61c0 0x5602ca107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e35ffb640 0 client.19068 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:54:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e5b780740 1 -- 172.21.15.114:0/1562410860 >> 172.21.15.114:0/1562410860 conn(0x5602c9d46300 msgr2=0x5602ca0f7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e5b780740 1 -- 172.21.15.114:0/1562410860 shutdown_connections 2024-05-27T07:54:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e5b780740 1 -- 172.21.15.114:0/1562410860 wait complete. 2024-05-27T07:54:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.966+0000 7f3e5b780740 20 client.19068 trim_cache size 0 max 16384 2024-05-27T07:54:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:54:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:54:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:54:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.967+0000 7f3e547f3640 1 -- 172.21.15.114:0/1305394360 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3e380b2b20 msgr2=0x7f3e380b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.967+0000 7f3e547f3640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3e380b2b20 0x7f3e380b5030 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c106ad0 tx=0x7f3e3c0128c0 comp rx=0 tx=0).stop 2024-05-27T07:54:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.967+0000 7f3e547f3640 1 -- 172.21.15.114:0/1305394360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e4c103a30 msgr2=0x7f3e4c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:54:52.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.967+0000 7f3e547f3640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e4c103a30 0x7f3e4c06eb30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800d770 tx=0x7f3e4800dc30 comp rx=0 tx=0).stop 2024-05-27T07:54:52.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.967+0000 7f3e547f3640 1 -- 172.21.15.114:0/1305394360 shutdown_connections 2024-05-27T07:54:52.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.967+0000 7f3e547f3640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3e380b2b20 0x7f3e380b5030 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:52.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.967+0000 7f3e547f3640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e4c109260 0x7f3e4c06f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:52.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.967+0000 7f3e547f3640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e4c108dc0 0x7f3e4c06f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:52.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.967+0000 7f3e547f3640 1 --2- 172.21.15.114:0/1305394360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e4c103a30 0x7f3e4c06eb30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:54:52.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.967+0000 7f3e547f3640 1 -- 172.21.15.114:0/1305394360 >> 172.21.15.114:0/1305394360 conn(0x7f3e4c0fd9c0 msgr2=0x7f3e4c104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:54:52.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.967+0000 7f3e547f3640 1 -- 172.21.15.114:0/1305394360 shutdown_connections 2024-05-27T07:54:52.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:54:52.967+0000 7f3e547f3640 1 -- 172.21.15.114:0/1305394360 wait complete. 2024-05-27T07:54:52.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:52 smithi163 ceph-mon[36451]: pgmap v513: 129 pgs: 129 active+clean; 303 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 531 op/s 2024-05-27T07:54:52.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:52 smithi186 ceph-mon[36647]: pgmap v513: 129 pgs: 129 active+clean; 303 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 531 op/s 2024-05-27T07:54:53.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:52 smithi114 ceph-mon[32038]: pgmap v513: 129 pgs: 129 active+clean; 303 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 531 op/s 2024-05-27T07:54:53.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 17172 16.96 MB/sec warmup 65 sec latency 1338.237 ms 2024-05-27T07:54:54.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 17480 17.05 MB/sec warmup 66 sec latency 105.217 ms 2024-05-27T07:54:54.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:54 smithi163 ceph-mon[36451]: pgmap v514: 129 pgs: 129 active+clean; 312 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 365 op/s 2024-05-27T07:54:54.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:54 smithi163 ceph-mon[36451]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-27T07:54:54.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:54 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:54:54.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:54 smithi186 ceph-mon[36647]: pgmap v514: 129 pgs: 129 active+clean; 312 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 365 op/s 2024-05-27T07:54:54.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:54 smithi186 ceph-mon[36647]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-27T07:54:54.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:54 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:54:55.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:54 smithi114 ceph-mon[32038]: pgmap v514: 129 pgs: 129 active+clean; 312 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 365 op/s 2024-05-27T07:54:55.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:54 smithi114 ceph-mon[32038]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-27T07:54:55.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:54 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:54:55.593 INFO:tasks.workunit.client.0.smithi114.stdout: 10 17763 17.05 MB/sec warmup 67 sec latency 172.303 ms 2024-05-27T07:54:56.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 18007 16.98 MB/sec warmup 68 sec latency 206.696 ms 2024-05-27T07:54:56.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:56 smithi114 ceph-mon[32038]: pgmap v515: 129 pgs: 129 active+clean; 304 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 434 op/s 2024-05-27T07:54:56.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:56 smithi163 ceph-mon[36451]: pgmap v515: 129 pgs: 129 active+clean; 304 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 434 op/s 2024-05-27T07:54:56.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:56 smithi186 ceph-mon[36647]: pgmap v515: 129 pgs: 129 active+clean; 304 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 434 op/s 2024-05-27T07:54:57.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 18166 16.76 MB/sec warmup 69 sec latency 396.166 ms 2024-05-27T07:54:58.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 18459 16.65 MB/sec warmup 70 sec latency 195.041 ms 2024-05-27T07:54:58.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:54:58 smithi163 ceph-mon[36451]: pgmap v516: 129 pgs: 129 active+clean; 299 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 510 op/s 2024-05-27T07:54:58.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:54:58 smithi186 ceph-mon[36647]: pgmap v516: 129 pgs: 129 active+clean; 299 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 510 op/s 2024-05-27T07:54:59.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:54:58 smithi114 ceph-mon[32038]: pgmap v516: 129 pgs: 129 active+clean; 299 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 510 op/s 2024-05-27T07:54:59.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 18736 16.67 MB/sec warmup 71 sec latency 121.813 ms 2024-05-27T07:55:00.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19210 16.72 MB/sec warmup 72 sec latency 107.658 ms 2024-05-27T07:55:00.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:00 smithi163 ceph-mon[36451]: pgmap v517: 129 pgs: 129 active+clean; 294 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 531 op/s 2024-05-27T07:55:00.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:00 smithi186 ceph-mon[36647]: pgmap v517: 129 pgs: 129 active+clean; 294 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 531 op/s 2024-05-27T07:55:01.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:00 smithi114 ceph-mon[32038]: pgmap v517: 129 pgs: 129 active+clean; 294 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 531 op/s 2024-05-27T07:55:01.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19571 16.72 MB/sec warmup 73 sec latency 257.628 ms 2024-05-27T07:55:02.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2d60d94700 time 2024-05-27T07:55:01.940050+0000 2024-05-27T07:55:02.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:02.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:02.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2d735d9ba6] 2024-05-27T07:55:02.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2d735d9dd1] 2024-05-27T07:55:02.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f0dff4b23d] 2024-05-27T07:55:02.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55f0dff36971] 2024-05-27T07:55:02.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 5: (SnapRealm::check_cache() const+0x13e) [0x55f0dff34fa0] 2024-05-27T07:55:02.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f0dff35651] 2024-05-27T07:55:02.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0dfeca347] 2024-05-27T07:55:02.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f0dfbc687e] 2024-05-27T07:55:02.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f0dfbcbe94] 2024-05-27T07:55:02.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55f0dfbd6969] 2024-05-27T07:55:02.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 11: (MDLog::_replay_thread()+0x1f38) [0x55f0dff6f48c] 2024-05-27T07:55:02.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55f0dff789cb] 2024-05-27T07:55:02.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 13: (Thread::entry_wrapper()+0x43) [0x7f2d735a872d] 2024-05-27T07:55:02.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2d735a8749] 2024-05-27T07:55:02.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2d7214c1ca] 2024-05-27T07:55:02.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 16: clone() 2024-05-27T07:55:02.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:01.941+0000 7f2d60d94700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2d60d94700 time 2024-05-27T07:55:01.940050+0000 2024-05-27T07:55:02.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:02.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:02.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:02.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2d735d9ba6] 2024-05-27T07:55:02.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2d735d9dd1] 2024-05-27T07:55:02.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f0dff4b23d] 2024-05-27T07:55:02.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55f0dff36971] 2024-05-27T07:55:02.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 5: (SnapRealm::check_cache() const+0x13e) [0x55f0dff34fa0] 2024-05-27T07:55:02.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f0dff35651] 2024-05-27T07:55:02.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0dfeca347] 2024-05-27T07:55:02.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f0dfbc687e] 2024-05-27T07:55:02.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f0dfbcbe94] 2024-05-27T07:55:02.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55f0dfbd6969] 2024-05-27T07:55:02.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 11: (MDLog::_replay_thread()+0x1f38) [0x55f0dff6f48c] 2024-05-27T07:55:02.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55f0dff789cb] 2024-05-27T07:55:02.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 13: (Thread::entry_wrapper()+0x43) [0x7f2d735a872d] 2024-05-27T07:55:02.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2d735a8749] 2024-05-27T07:55:02.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2d7214c1ca] 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 16: clone() 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: *** Caught signal (Aborted) ** 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: in thread 7f2d60d94700 thread_name:md_log_replay 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2d72156d20] 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2: gsignal() 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 3: abort() 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2d735d9ced] 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2d735d9dd1] 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f0dff4b23d] 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55f0dff36971] 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 8: (SnapRealm::check_cache() const+0x13e) [0x55f0dff34fa0] 2024-05-27T07:55:02.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f0dff35651] 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0dfeca347] 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f0dfbc687e] 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f0dfbcbe94] 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55f0dfbd6969] 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 14: (MDLog::_replay_thread()+0x1f38) [0x55f0dff6f48c] 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55f0dff789cb] 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 16: (Thread::entry_wrapper()+0x43) [0x7f2d735a872d] 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2d735a8749] 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2d7214c1ca] 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 19: clone() 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:01.944+0000 7f2d60d94700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: in thread 7f2d60d94700 thread_name:md_log_replay 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:02.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:02.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2d72156d20] 2024-05-27T07:55:02.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2: gsignal() 2024-05-27T07:55:02.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 3: abort() 2024-05-27T07:55:02.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2d735d9ced] 2024-05-27T07:55:02.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2d735d9dd1] 2024-05-27T07:55:02.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f0dff4b23d] 2024-05-27T07:55:02.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55f0dff36971] 2024-05-27T07:55:02.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 8: (SnapRealm::check_cache() const+0x13e) [0x55f0dff34fa0] 2024-05-27T07:55:02.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f0dff35651] 2024-05-27T07:55:02.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0dfeca347] 2024-05-27T07:55:02.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f0dfbc687e] 2024-05-27T07:55:02.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f0dfbcbe94] 2024-05-27T07:55:02.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55f0dfbd6969] 2024-05-27T07:55:02.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 14: (MDLog::_replay_thread()+0x1f38) [0x55f0dff6f48c] 2024-05-27T07:55:02.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55f0dff789cb] 2024-05-27T07:55:02.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 16: (Thread::entry_wrapper()+0x43) [0x7f2d735a872d] 2024-05-27T07:55:02.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2d735a8749] 2024-05-27T07:55:02.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2d7214c1ca] 2024-05-27T07:55:02.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 19: clone() 2024-05-27T07:55:02.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:55:02.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:02.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: -1> 2024-05-27T07:55:01.941+0000 7f2d60d94700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2d60d94700 time 2024-05-27T07:55:01.940050+0000 2024-05-27T07:55:02.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:02.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:02.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:02.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2d735d9ba6] 2024-05-27T07:55:02.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2d735d9dd1] 2024-05-27T07:55:02.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f0dff4b23d] 2024-05-27T07:55:02.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55f0dff36971] 2024-05-27T07:55:02.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 5: (SnapRealm::check_cache() const+0x13e) [0x55f0dff34fa0] 2024-05-27T07:55:02.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f0dff35651] 2024-05-27T07:55:02.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0dfeca347] 2024-05-27T07:55:02.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f0dfbc687e] 2024-05-27T07:55:02.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f0dfbcbe94] 2024-05-27T07:55:02.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55f0dfbd6969] 2024-05-27T07:55:02.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 11: (MDLog::_replay_thread()+0x1f38) [0x55f0dff6f48c] 2024-05-27T07:55:02.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55f0dff789cb] 2024-05-27T07:55:02.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 13: (Thread::entry_wrapper()+0x43) [0x7f2d735a872d] 2024-05-27T07:55:02.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2d735a8749] 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2d7214c1ca] 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 16: clone() 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 0> 2024-05-27T07:55:01.944+0000 7f2d60d94700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: in thread 7f2d60d94700 thread_name:md_log_replay 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2d72156d20] 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2: gsignal() 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 3: abort() 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2d735d9ced] 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2d735d9dd1] 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f0dff4b23d] 2024-05-27T07:55:02.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55f0dff36971] 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 8: (SnapRealm::check_cache() const+0x13e) [0x55f0dff34fa0] 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f0dff35651] 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0dfeca347] 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f0dfbc687e] 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f0dfbcbe94] 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55f0dfbd6969] 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 14: (MDLog::_replay_thread()+0x1f38) [0x55f0dff6f48c] 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55f0dff789cb] 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 16: (Thread::entry_wrapper()+0x43) [0x7f2d735a872d] 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2d735a8749] 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2d7214c1ca] 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 19: clone() 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:55:02.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:02.242 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: -1> 2024-05-27T07:55:01.941+0000 7f2d60d94700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2d60d94700 time 2024-05-27T07:55:01.940050+0000 2024-05-27T07:55:02.242 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:02.242 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:02.242 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:02.242 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2d735d9ba6] 2024-05-27T07:55:02.242 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2d735d9dd1] 2024-05-27T07:55:02.242 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f0dff4b23d] 2024-05-27T07:55:02.242 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55f0dff36971] 2024-05-27T07:55:02.242 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 5: (SnapRealm::check_cache() const+0x13e) [0x55f0dff34fa0] 2024-05-27T07:55:02.242 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f0dff35651] 2024-05-27T07:55:02.242 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0dfeca347] 2024-05-27T07:55:02.242 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f0dfbc687e] 2024-05-27T07:55:02.242 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f0dfbcbe94] 2024-05-27T07:55:02.243 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55f0dfbd6969] 2024-05-27T07:55:02.243 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 11: (MDLog::_replay_thread()+0x1f38) [0x55f0dff6f48c] 2024-05-27T07:55:02.243 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55f0dff789cb] 2024-05-27T07:55:02.243 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 13: (Thread::entry_wrapper()+0x43) [0x7f2d735a872d] 2024-05-27T07:55:02.243 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2d735a8749] 2024-05-27T07:55:02.243 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2d7214c1ca] 2024-05-27T07:55:02.243 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 16: clone() 2024-05-27T07:55:02.243 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:02.243 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 0> 2024-05-27T07:55:01.944+0000 7f2d60d94700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:55:02.243 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: in thread 7f2d60d94700 thread_name:md_log_replay 2024-05-27T07:55:02.243 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:02.243 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:02.243 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2d72156d20] 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2: gsignal() 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 3: abort() 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2d735d9ced] 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2d735d9dd1] 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f0dff4b23d] 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55f0dff36971] 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 8: (SnapRealm::check_cache() const+0x13e) [0x55f0dff34fa0] 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f0dff35651] 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0dfeca347] 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f0dfbc687e] 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f0dfbcbe94] 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55f0dfbd6969] 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 14: (MDLog::_replay_thread()+0x1f38) [0x55f0dff6f48c] 2024-05-27T07:55:02.244 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55f0dff789cb] 2024-05-27T07:55:02.245 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 16: (Thread::entry_wrapper()+0x43) [0x7f2d735a872d] 2024-05-27T07:55:02.245 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2d735a8749] 2024-05-27T07:55:02.245 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2d7214c1ca] 2024-05-27T07:55:02.245 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 19: clone() 2024-05-27T07:55:02.245 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:55:02.245 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[65397]: 2024-05-27T07:55:02.245 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:02 smithi163 podman[74437]: 2024-05-27 07:55:02.025636111 +0000 UTC m=+0.024727138 container died aa7f7804209788d8785ca4a1416d69cb401668ef25110f2865cc272871660b1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:55:02.597 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19650 16.53 MB/sec warmup 74 sec latency 734.807 ms 2024-05-27T07:55:02.942 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:02 smithi163 ceph-mon[36451]: pgmap v518: 129 pgs: 129 active+clean; 297 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.8 MiB/s wr, 596 op/s 2024-05-27T07:55:02.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:02 smithi186 ceph-mon[36647]: pgmap v518: 129 pgs: 129 active+clean; 297 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.8 MiB/s wr, 596 op/s 2024-05-27T07:55:03.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:02 smithi114 ceph-mon[32038]: pgmap v518: 129 pgs: 129 active+clean; 297 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.8 MiB/s wr, 596 op/s 2024-05-27T07:55:03.227 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:02 smithi163 podman[74437]: 2024-05-27 07:55:02.942601164 +0000 UTC m=+0.941692174 container remove aa7f7804209788d8785ca4a1416d69cb401668ef25110f2865cc272871660b1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-27T07:55:03.227 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:02 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:55:03.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19739 16.39 MB/sec warmup 75 sec latency 1684.907 ms 2024-05-27T07:55:03.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:03 smithi163 ceph-mon[36451]: osdmap e141: 12 total, 12 up, 12 in 2024-05-27T07:55:03.977 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:03 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:55:03.977 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:03 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.929s CPU time. 2024-05-27T07:55:03.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:03 smithi186 ceph-mon[36647]: osdmap e141: 12 total, 12 up, 12 in 2024-05-27T07:55:04.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:03 smithi114 ceph-mon[32038]: osdmap e141: 12 total, 12 up, 12 in 2024-05-27T07:55:04.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19784 16.20 MB/sec warmup 76 sec latency 1723.109 ms 2024-05-27T07:55:04.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:04 smithi163 ceph-mon[36451]: pgmap v520: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 300 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 547 op/s 2024-05-27T07:55:04.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:04 smithi186 ceph-mon[36647]: pgmap v520: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 300 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 547 op/s 2024-05-27T07:55:05.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:04 smithi114 ceph-mon[32038]: pgmap v520: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 300 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 547 op/s 2024-05-27T07:55:05.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19800 16.00 MB/sec warmup 77 sec latency 1290.928 ms 2024-05-27T07:55:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.130+0000 7fd7d83b4640 1 -- 172.21.15.114:0/4089129848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d0108f80 msgr2=0x7fd7d0102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:06.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.130+0000 7fd7d83b4640 1 --2- 172.21.15.114:0/4089129848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d0108f80 0x7fd7d0102200 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8009ca0 tx=0x7fd7c802a980 comp rx=0 tx=0).stop 2024-05-27T07:55:06.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.134+0000 7fd7d83b4640 1 -- 172.21.15.114:0/4089129848 shutdown_connections 2024-05-27T07:55:06.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.134+0000 7fd7d83b4640 1 --2- 172.21.15.114:0/4089129848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7d0107410 0x7fd7d0107820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.134+0000 7fd7d83b4640 1 --2- 172.21.15.114:0/4089129848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d0108f80 0x7fd7d0102200 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.134+0000 7fd7d83b4640 1 --2- 172.21.15.114:0/4089129848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7d0108b70 0x7fd7d0101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.134+0000 7fd7d83b4640 1 -- 172.21.15.114:0/4089129848 >> 172.21.15.114:0/4089129848 conn(0x7fd7d00779d0 msgr2=0x7fd7d0077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:06.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.139+0000 7fd7d83b4640 1 -- 172.21.15.114:0/4089129848 shutdown_connections 2024-05-27T07:55:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.142+0000 7fd7d83b4640 1 -- 172.21.15.114:0/4089129848 wait complete. 2024-05-27T07:55:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.143+0000 7fd7d83b4640 1 Processor -- start 2024-05-27T07:55:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.143+0000 7fd7d83b4640 1 -- start start 2024-05-27T07:55:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.143+0000 7fd7d83b4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7d0107410 0x7fd7d010de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.143+0000 7fd7d83b4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7d0108b70 0x7fd7d010e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.143+0000 7fd7d83b4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d0108f80 0x7fd7d01093b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.143+0000 7fd7d83b4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d006cfe0 con 0x7fd7d0108f80 2024-05-27T07:55:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.143+0000 7fd7d83b4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d006ce60 con 0x7fd7d0107410 2024-05-27T07:55:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.143+0000 7fd7d83b4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d006d160 con 0x7fd7d0108b70 2024-05-27T07:55:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.145+0000 7fd7d6150640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7d0107410 0x7fd7d010de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.145+0000 7fd7d6150640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7d0107410 0x7fd7d010de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:54450/0 (socket says 172.21.15.114:54450) 2024-05-27T07:55:06.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.145+0000 7fd7d6150640 1 -- 172.21.15.114:0/320573563 learned_addr learned my addr 172.21.15.114:0/320573563 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:06.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.145+0000 7fd7d6150640 1 -- 172.21.15.114:0/320573563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7d0108b70 msgr2=0x7fd7d010e370 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:55:06.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.145+0000 7fd7d6150640 1 --2- 172.21.15.114:0/320573563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7d0108b70 0x7fd7d010e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.145+0000 7fd7d6150640 1 -- 172.21.15.114:0/320573563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d0108f80 msgr2=0x7fd7d01093b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:55:06.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.145+0000 7fd7d6150640 1 --2- 172.21.15.114:0/320573563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d0108f80 0x7fd7d01093b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.145+0000 7fd7d6150640 1 -- 172.21.15.114:0/320573563 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d0109bb0 con 0x7fd7d0107410 2024-05-27T07:55:06.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.146+0000 7fd7d6150640 1 --2- 172.21.15.114:0/320573563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7d0107410 0x7fd7d010de10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc00b470 tx=0x7fd7cc00b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:06.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.146+0000 7fd7c37fe640 1 -- 172.21.15.114:0/320573563 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc015020 con 0x7fd7d0107410 2024-05-27T07:55:06.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.146+0000 7fd7c37fe640 1 -- 172.21.15.114:0/320573563 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7cc011d40 con 0x7fd7d0107410 2024-05-27T07:55:06.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.146+0000 7fd7c37fe640 1 -- 172.21.15.114:0/320573563 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc00f340 con 0x7fd7d0107410 2024-05-27T07:55:06.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.146+0000 7fd7d83b4640 1 -- 172.21.15.114:0/320573563 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7d0109ea0 con 0x7fd7d0107410 2024-05-27T07:55:06.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.146+0000 7fd7d83b4640 1 -- 172.21.15.114:0/320573563 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7d0115b30 con 0x7fd7d0107410 2024-05-27T07:55:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.151+0000 7fd7df262740 1 Processor -- start 2024-05-27T07:55:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.152+0000 7fd7df262740 1 -- start start 2024-05-27T07:55:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.152+0000 7fd7df262740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c95b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.152+0000 7fd7df262740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91ea2f0 0x5614c95b4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.152+0000 7fd7df262740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91f24d0 0x5614c95b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.152+0000 7fd7df262740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5614c9526c30 con 0x5614c91ea2f0 2024-05-27T07:55:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.152+0000 7fd7df262740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5614c95167d0 con 0x5614c92131b0 2024-05-27T07:55:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.152+0000 7fd7df262740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5614c951a810 con 0x5614c91f24d0 2024-05-27T07:55:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.153+0000 7fd7d594f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91f24d0 0x5614c95b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.153+0000 7fd7d6951640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c95b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.153+0000 7fd7d6951640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c95b43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:54452/0 (socket says 172.21.15.114:54452) 2024-05-27T07:55:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.153+0000 7fd7d6951640 1 -- 172.21.15.114:0/539346155 learned_addr learned my addr 172.21.15.114:0/539346155 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.154+0000 7fd7d6951640 1 -- 172.21.15.114:0/539346155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91f24d0 msgr2=0x5614c95b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.154+0000 7fd7d6951640 1 --2- 172.21.15.114:0/539346155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91f24d0 0x5614c95b4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.154+0000 7fd7d6951640 1 -- 172.21.15.114:0/539346155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91ea2f0 msgr2=0x5614c95b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.154+0000 7fd7d6951640 1 --2- 172.21.15.114:0/539346155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91ea2f0 0x5614c95b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.154+0000 7fd7d6951640 1 -- 172.21.15.114:0/539346155 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614c922b210 con 0x5614c92131b0 2024-05-27T07:55:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.156+0000 7fd7d6951640 1 --2- 172.21.15.114:0/539346155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c95b43d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd7c402c8c0 tx=0x7fd7c402cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebab8fcaea30358e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.160+0000 7fd7c0bfe640 1 -- 172.21.15.114:0/539346155 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c402ecd0 con 0x5614c92131b0 2024-05-27T07:55:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.161+0000 7fd7c0bfe640 1 -- 172.21.15.114:0/539346155 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7c4031910 con 0x5614c92131b0 2024-05-27T07:55:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.162+0000 7fd7c37fe640 1 -- 172.21.15.114:0/320573563 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fd7cc018020 con 0x7fd7d0107410 2024-05-27T07:55:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.162+0000 7fd7df262740 1 -- 172.21.15.114:0/539346155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 msgr2=0x5614c95b43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.162+0000 7fd7df262740 1 --2- 172.21.15.114:0/539346155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c95b43d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd7c402c8c0 tx=0x7fd7c402cd80 comp rx=0 tx=0).stop 2024-05-27T07:55:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.163+0000 7fd7c37fe640 1 --2- 172.21.15.114:0/320573563 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7bc0b2aa0 0x7fd7bc0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.163+0000 7fd7c37fe640 1 -- 172.21.15.114:0/320573563 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 9642+0+0 (secure 0 0 0) 0x7fd7cc0e44e0 con 0x7fd7d0107410 2024-05-27T07:55:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.164+0000 7fd7d6951640 1 --2- 172.21.15.114:0/320573563 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7bc0b2aa0 0x7fd7bc0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.164+0000 7fd7df262740 1 -- 172.21.15.114:0/539346155 shutdown_connections 2024-05-27T07:55:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.164+0000 7fd7df262740 1 --2- 172.21.15.114:0/539346155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91f24d0 0x5614c95b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.164+0000 7fd7df262740 1 --2- 172.21.15.114:0/539346155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91ea2f0 0x5614c95b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.164+0000 7fd7df262740 1 --2- 172.21.15.114:0/539346155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c95b43d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.164+0000 7fd7df262740 1 -- 172.21.15.114:0/539346155 >> 172.21.15.114:0/539346155 conn(0x5614c9207300 msgr2=0x5614c95b0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.165+0000 7fd7df262740 1 -- 172.21.15.114:0/539346155 shutdown_connections 2024-05-27T07:55:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.166+0000 7fd7d6951640 1 --2- 172.21.15.114:0/320573563 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7bc0b2aa0 0x7fd7bc0b4fb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd7c403a440 tx=0x7fd7c402c550 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.169+0000 7fd7df262740 1 -- 172.21.15.114:0/539346155 wait complete. 2024-05-27T07:55:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.169+0000 7fd7df262740 1 -- start start 2024-05-27T07:55:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.169+0000 7fd7df262740 1 Processor -- start 2024-05-27T07:55:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.171+0000 7fd7df262740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c91ea2f0 0x5614c95bea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.171+0000 7fd7df262740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91f24d0 0x5614c95befd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.171+0000 7fd7df262740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c92131b0 0x5614c95c4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.171+0000 7fd7df262740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5614c91fc790 con 0x5614c91f24d0 2024-05-27T07:55:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.171+0000 7fd7df262740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5614c94d2890 con 0x5614c91ea2f0 2024-05-27T07:55:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.171+0000 7fd7df262740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5614c95c1550 con 0x5614c92131b0 2024-05-27T07:55:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.171+0000 7fd7df262740 10 client.0 _ll_register_callbacks cb 0x5614c922aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.171+0000 7fd7d6150640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c91ea2f0 0x5614c95bea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.172+0000 7fd7d6150640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c91ea2f0 0x5614c95bea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:54456/0 (socket says 172.21.15.114:54456) 2024-05-27T07:55:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.172+0000 7fd7d6150640 1 -- 172.21.15.114:0/3676957345 learned_addr learned my addr 172.21.15.114:0/3676957345 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.172+0000 7fd7d6150640 1 -- 172.21.15.114:0/3676957345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c92131b0 msgr2=0x5614c95c4940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:55:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.172+0000 7fd7d6150640 1 --2- 172.21.15.114:0/3676957345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c92131b0 0x5614c95c4940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.172+0000 7fd7d6150640 1 -- 172.21.15.114:0/3676957345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91f24d0 msgr2=0x5614c95befd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:55:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.172+0000 7fd7d6150640 1 --2- 172.21.15.114:0/3676957345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91f24d0 0x5614c95befd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.172+0000 7fd7d6150640 1 -- 172.21.15.114:0/3676957345 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614c92f7620 con 0x5614c91ea2f0 2024-05-27T07:55:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.172+0000 7fd7d6150640 1 --2- 172.21.15.114:0/3676957345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c91ea2f0 0x5614c95bea90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc006e70 tx=0x7fd7cc0e1cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88455de2631a624c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:06.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.173+0000 7fd7b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:55:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.173+0000 7fd7b9ffb640 1 -- 172.21.15.114:0/3676957345 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc0ab410 con 0x5614c91ea2f0 2024-05-27T07:55:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.174+0000 7fd7df262740 1 -- 172.21.15.114:0/3676957345 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7c4031910 con 0x5614c91ea2f0 2024-05-27T07:55:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.177+0000 7fd7b9ffb640 1 -- 172.21.15.114:0/3676957345 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9642+0+0 (secure 0 0 0) 0x7fd7cc011890 con 0x5614c91ea2f0 2024-05-27T07:55:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.177+0000 7fd7b9ffb640 1 -- 172.21.15.114:0/3676957345 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fd7cc0af7c0 con 0x5614c91ea2f0 2024-05-27T07:55:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.179+0000 7fd7df262740 10 client.28418 fetch_fsmap learned FSMap version 83 2024-05-27T07:55:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.179+0000 7fd7df262740 1 -- 172.21.15.114:0/3676957345 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=4+}) -- 0x5614c95cd8c0 con 0x5614c91ea2f0 2024-05-27T07:55:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.180+0000 7fd7b9ffb640 1 -- 172.21.15.114:0/3676957345 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7c4031910 con 0x5614c91ea2f0 2024-05-27T07:55:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.180+0000 7fd7b9ffb640 1 -- 172.21.15.114:0/3676957345 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 83) ==== 4700+0+0 (secure 0 0 0) 0x7fd7cc0c1b20 con 0x5614c91ea2f0 2024-05-27T07:55:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.180+0000 7fd7df262740 10 client.28418 fetch_fsmap finished waiting for FSMap version 83 2024-05-27T07:55:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.180+0000 7fd7df262740 10 client.28418 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.180+0000 7fd7df262740 20 client.28418 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.180+0000 7fd7df262740 1 --2- 172.21.15.114:0/3676957345 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5614c91d1d80 0x5614c95d1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.180+0000 7fd7df262740 4 client.28418 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:55:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.180+0000 7fd7df262740 1 -- 172.21.15.114:0/3676957345 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614c9528b80 con 0x5614c91d1d80 2024-05-27T07:55:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.182+0000 7fd7d594f640 1 --2- 172.21.15.114:0/3676957345 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5614c91d1d80 0x5614c95d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.193+0000 7fd7d594f640 1 --2- 172.21.15.114:0/3676957345 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5614c91d1d80 0x5614c95d1290 crc :-1 s=READY pgs=430 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b8a634366ccebf9 server_cookie=67a5cfbbcf3cb142 in_seq=0 out_seq=0 2024-05-27T07:55:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.193+0000 7fd7b9ffb640 10 client.28418 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.194+0000 7fd7b9ffb640 1 -- 172.21.15.114:0/3676957345 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5614c9528b80 con 0x5614c91d1d80 2024-05-27T07:55:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.194+0000 7fd7b9ffb640 10 client.28418 handle_command_reply: tid=0 2024-05-27T07:55:06.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.195+0000 7fd7df262740 1 client.28418 shutdown 2024-05-27T07:55:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.198+0000 7fd7df262740 1 -- 172.21.15.114:0/3676957345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c91ea2f0 msgr2=0x5614c95bea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.198+0000 7fd7df262740 1 --2- 172.21.15.114:0/3676957345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c91ea2f0 0x5614c95bea90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc006e70 tx=0x7fd7cc0e1cf0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.201+0000 7fd7df262740 1 -- 172.21.15.114:0/3676957345 shutdown_connections 2024-05-27T07:55:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.201+0000 7fd7df262740 1 --2- 172.21.15.114:0/3676957345 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5614c91d1d80 0x5614c95d1290 crc :-1 s=READY pgs=430 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.201+0000 7fd7df262740 1 --2- 172.21.15.114:0/3676957345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c92131b0 0x5614c95c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.201+0000 7fd7df262740 1 --2- 172.21.15.114:0/3676957345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91f24d0 0x5614c95befd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.201+0000 7fd7df262740 1 --2- 172.21.15.114:0/3676957345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c91ea2f0 0x5614c95bea90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.201+0000 7fd7df262740 1 -- 172.21.15.114:0/3676957345 >> 172.21.15.114:0/3676957345 conn(0x5614c9207300 msgr2=0x5614c95b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.202+0000 7fd7b9ffb640 0 client.28418 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.207+0000 7fd7df262740 1 -- 172.21.15.114:0/3676957345 shutdown_connections 2024-05-27T07:55:06.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.211+0000 7fd7df262740 1 -- 172.21.15.114:0/3676957345 wait complete. 2024-05-27T07:55:06.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.213+0000 7fd7df262740 20 client.28418 trim_cache size 0 max 16384 2024-05-27T07:55:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.244+0000 7fd7df262740 1 Processor -- start 2024-05-27T07:55:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.246+0000 7fd7df262740 1 -- start start 2024-05-27T07:55:06.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.246+0000 7fd7df262740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91ea2f0 0x5614c95bef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.246+0000 7fd7df262740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91f24d0 0x5614c95d6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.246+0000 7fd7df262740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c95d6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.246+0000 7fd7df262740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b40105e0 con 0x5614c91f24d0 2024-05-27T07:55:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.246+0000 7fd7df262740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b4010cb0 con 0x5614c92131b0 2024-05-27T07:55:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.246+0000 7fd7df262740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b400f500 con 0x5614c91ea2f0 2024-05-27T07:55:06.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.247+0000 7fd7d6150640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91ea2f0 0x5614c95bef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:06.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.247+0000 7fd7d6951640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c95d6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:06.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.247+0000 7fd7d6951640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c95d6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:54460/0 (socket says 172.21.15.114:54460) 2024-05-27T07:55:06.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.247+0000 7fd7d6951640 1 -- 172.21.15.114:0/3887086547 learned_addr learned my addr 172.21.15.114:0/3887086547 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:06.251 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.250+0000 7fd7d594f640 1 --2- 172.21.15.114:0/3887086547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91f24d0 0x5614c95d6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.252+0000 7fd7d6951640 1 -- 172.21.15.114:0/3887086547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91ea2f0 msgr2=0x5614c95bef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:06.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.252+0000 7fd7d6951640 1 --2- 172.21.15.114:0/3887086547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91ea2f0 0x5614c95bef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.252+0000 7fd7d6951640 1 -- 172.21.15.114:0/3887086547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91f24d0 msgr2=0x5614c95d6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:06.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.252+0000 7fd7d6951640 1 --2- 172.21.15.114:0/3887086547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91f24d0 0x5614c95d6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.256 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.252+0000 7fd7d6951640 1 -- 172.21.15.114:0/3887086547 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614c95ac3d0 con 0x5614c92131b0 2024-05-27T07:55:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.257+0000 7fd7d6951640 1 --2- 172.21.15.114:0/3887086547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c95d6fd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd7c402cfd0 tx=0x7fd7c4044000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba32ff21eb4ef203 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.262+0000 7fd7bbfff640 1 -- 172.21.15.114:0/3887086547 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c402ed10 con 0x5614c92131b0 2024-05-27T07:55:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.262+0000 7fd7bbfff640 1 -- 172.21.15.114:0/3887086547 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7c40054d0 con 0x5614c92131b0 2024-05-27T07:55:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.262+0000 7fd7bbfff640 1 -- 172.21.15.114:0/3887086547 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4004b60 con 0x5614c92131b0 2024-05-27T07:55:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.265+0000 7fd7df262740 1 -- 172.21.15.114:0/3887086547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 msgr2=0x5614c95d6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.265+0000 7fd7df262740 1 --2- 172.21.15.114:0/3887086547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c95d6fd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd7c402cfd0 tx=0x7fd7c4044000 comp rx=0 tx=0).stop 2024-05-27T07:55:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.268+0000 7fd7df262740 1 -- 172.21.15.114:0/3887086547 shutdown_connections 2024-05-27T07:55:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.268+0000 7fd7df262740 1 --2- 172.21.15.114:0/3887086547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c95d6fd0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.268+0000 7fd7df262740 1 --2- 172.21.15.114:0/3887086547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91f24d0 0x5614c95d6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.268+0000 7fd7df262740 1 --2- 172.21.15.114:0/3887086547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91ea2f0 0x5614c95bef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.268+0000 7fd7df262740 1 -- 172.21.15.114:0/3887086547 >> 172.21.15.114:0/3887086547 conn(0x5614c9207300 msgr2=0x5614c95be860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:06.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.273+0000 7fd7df262740 1 -- 172.21.15.114:0/3887086547 shutdown_connections 2024-05-27T07:55:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.275+0000 7fd7df262740 1 -- 172.21.15.114:0/3887086547 wait complete. 2024-05-27T07:55:06.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.275+0000 7fd7df262740 1 -- start start 2024-05-27T07:55:06.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.275+0000 7fd7df262740 1 Processor -- start 2024-05-27T07:55:06.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.276+0000 7fd7df262740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91ea2f0 0x5614c95c8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.276+0000 7fd7df262740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91f24d0 0x5614c95c8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.276+0000 7fd7df262740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c9631930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.276+0000 7fd7df262740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b400e760 con 0x5614c91ea2f0 2024-05-27T07:55:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.276+0000 7fd7df262740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b400e110 con 0x5614c92131b0 2024-05-27T07:55:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.276+0000 7fd7df262740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b400ee30 con 0x5614c91f24d0 2024-05-27T07:55:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.276+0000 7fd7df262740 10 client.0 _ll_register_callbacks cb 0x7fd7cc012f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.276+0000 7fd7d6951640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c9631930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.276+0000 7fd7d6150640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91ea2f0 0x5614c95c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.276+0000 7fd7d6150640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91ea2f0 0x5614c95c8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58360/0 (socket says 172.21.15.114:58360) 2024-05-27T07:55:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.276+0000 7fd7d6951640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c9631930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:54466/0 (socket says 172.21.15.114:54466) 2024-05-27T07:55:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.276+0000 7fd7d6150640 1 -- 172.21.15.114:0/2651505635 learned_addr learned my addr 172.21.15.114:0/2651505635 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.277+0000 7fd7d6150640 1 -- 172.21.15.114:0/2651505635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91f24d0 msgr2=0x5614c95c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.277+0000 7fd7d6150640 1 --2- 172.21.15.114:0/2651505635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91f24d0 0x5614c95c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.277+0000 7fd7d6150640 1 -- 172.21.15.114:0/2651505635 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 msgr2=0x5614c9631930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.277+0000 7fd7d6150640 1 --2- 172.21.15.114:0/2651505635 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c9631930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.277+0000 7fd7d6150640 1 -- 172.21.15.114:0/2651505635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614c9631e70 con 0x5614c91ea2f0 2024-05-27T07:55:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.277+0000 7fd7d6150640 1 --2- 172.21.15.114:0/2651505635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91ea2f0 0x5614c95c8330 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0a6da0 tx=0x7fd7cc0ab310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28484e4924f6a8b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.277+0000 7fd7b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:55:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.278+0000 7fd7b9ffb640 1 -- 172.21.15.114:0/2651505635 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc0ab410 con 0x5614c91ea2f0 2024-05-27T07:55:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.278+0000 7fd7df262740 1 -- 172.21.15.114:0/2651505635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7c40054d0 con 0x5614c91ea2f0 2024-05-27T07:55:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.278+0000 7fd7b9ffb640 1 -- 172.21.15.114:0/2651505635 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9642+0+0 (secure 0 0 0) 0x7fd7cc0a5b30 con 0x5614c91ea2f0 2024-05-27T07:55:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.278+0000 7fd7b9ffb640 1 -- 172.21.15.114:0/2651505635 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fd7cc0b4270 con 0x5614c91ea2f0 2024-05-27T07:55:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.278+0000 7fd7df262740 10 client.19077 fetch_fsmap learned FSMap version 83 2024-05-27T07:55:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.278+0000 7fd7df262740 1 -- 172.21.15.114:0/2651505635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=4+}) -- 0x5614c9639170 con 0x5614c91ea2f0 2024-05-27T07:55:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.279+0000 7fd7b9ffb640 1 -- 172.21.15.114:0/2651505635 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7c40054d0 con 0x5614c91ea2f0 2024-05-27T07:55:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.279+0000 7fd7b9ffb640 1 -- 172.21.15.114:0/2651505635 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 83) ==== 4700+0+0 (secure 0 0 0) 0x7fd7cc085c70 con 0x5614c91ea2f0 2024-05-27T07:55:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.279+0000 7fd7df262740 10 client.19077 fetch_fsmap finished waiting for FSMap version 83 2024-05-27T07:55:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.279+0000 7fd7df262740 10 client.19077 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.279+0000 7fd7df262740 20 client.19077 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.279+0000 7fd7df262740 1 --2- 172.21.15.114:0/2651505635 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5614c91a71c0 0x5614c963b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.279+0000 7fd7df262740 4 client.19077 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:55:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.279+0000 7fd7df262740 1 -- 172.21.15.114:0/2651505635 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5614c9522360 con 0x5614c91a71c0 2024-05-27T07:55:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.280+0000 7fd7d594f640 1 --2- 172.21.15.114:0/2651505635 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5614c91a71c0 0x5614c963b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.287+0000 7fd7d594f640 1 --2- 172.21.15.114:0/2651505635 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5614c91a71c0 0x5614c963b760 crc :-1 s=READY pgs=431 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9ddf024ac0cd37b server_cookie=9c4aa8e6620531fd in_seq=0 out_seq=0 2024-05-27T07:55:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:06.287+0000 7fd7b9ffb640 10 client.19077 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:06.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19803 15.81 MB/sec warmup 78 sec latency 1538.923 ms 2024-05-27T07:55:06.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:06 smithi163 ceph-mon[36451]: pgmap v521: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 301 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.1 MiB/s wr, 456 op/s 2024-05-27T07:55:06.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:06 smithi186 ceph-mon[36647]: pgmap v521: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 301 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.1 MiB/s wr, 456 op/s 2024-05-27T07:55:07.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:06 smithi114 ceph-mon[32038]: pgmap v521: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 301 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.1 MiB/s wr, 456 op/s 2024-05-27T07:55:07.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19805 15.61 MB/sec warmup 79 sec latency 1922.801 ms 2024-05-27T07:55:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.694+0000 7fd7b9ffb640 1 -- 172.21.15.114:0/2651505635 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5614c9522360 con 0x5614c91a71c0 2024-05-27T07:55:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.694+0000 7fd7b9ffb640 10 client.19077 handle_command_reply: tid=0 2024-05-27T07:55:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.694+0000 7fd7df262740 1 client.19077 shutdown 2024-05-27T07:55:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.694+0000 7fd7df262740 1 -- 172.21.15.114:0/2651505635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91ea2f0 msgr2=0x5614c95c8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.694+0000 7fd7df262740 1 --2- 172.21.15.114:0/2651505635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91ea2f0 0x5614c95c8330 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0a6da0 tx=0x7fd7cc0ab310 comp rx=0 tx=0).stop 2024-05-27T07:55:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.695+0000 7fd7df262740 1 -- 172.21.15.114:0/2651505635 shutdown_connections 2024-05-27T07:55:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.695+0000 7fd7df262740 1 --2- 172.21.15.114:0/2651505635 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5614c91a71c0 0x5614c963b760 crc :-1 s=READY pgs=431 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.695+0000 7fd7df262740 1 --2- 172.21.15.114:0/2651505635 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5614c92131b0 0x5614c9631930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.695+0000 7fd7b9ffb640 0 client.19077 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.695+0000 7fd7df262740 1 --2- 172.21.15.114:0/2651505635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614c91f24d0 0x5614c95c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.695+0000 7fd7df262740 1 --2- 172.21.15.114:0/2651505635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5614c91ea2f0 0x5614c95c8330 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.695+0000 7fd7df262740 1 -- 172.21.15.114:0/2651505635 >> 172.21.15.114:0/2651505635 conn(0x5614c9207300 msgr2=0x5614c95b4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.695+0000 7fd7df262740 1 -- 172.21.15.114:0/2651505635 shutdown_connections 2024-05-27T07:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.695+0000 7fd7df262740 1 -- 172.21.15.114:0/2651505635 wait complete. 2024-05-27T07:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.695+0000 7fd7df262740 20 client.19077 trim_cache size 0 max 16384 2024-05-27T07:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:55:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:55:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.696+0000 7fd7d83b4640 1 -- 172.21.15.114:0/320573563 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7bc0b2aa0 msgr2=0x7fd7bc0b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.696+0000 7fd7d83b4640 1 --2- 172.21.15.114:0/320573563 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7bc0b2aa0 0x7fd7bc0b4fb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd7c403a440 tx=0x7fd7c402c550 comp rx=0 tx=0).stop 2024-05-27T07:55:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.696+0000 7fd7d83b4640 1 -- 172.21.15.114:0/320573563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7d0107410 msgr2=0x7fd7d010de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.696+0000 7fd7d83b4640 1 --2- 172.21.15.114:0/320573563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7d0107410 0x7fd7d010de10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc00b470 tx=0x7fd7cc00b930 comp rx=0 tx=0).stop 2024-05-27T07:55:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.696+0000 7fd7d83b4640 1 -- 172.21.15.114:0/320573563 shutdown_connections 2024-05-27T07:55:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.696+0000 7fd7d83b4640 1 --2- 172.21.15.114:0/320573563 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7bc0b2aa0 0x7fd7bc0b4fb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.696+0000 7fd7d83b4640 1 --2- 172.21.15.114:0/320573563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d0108f80 0x7fd7d01093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.696+0000 7fd7d83b4640 1 --2- 172.21.15.114:0/320573563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7d0108b70 0x7fd7d010e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.696+0000 7fd7d83b4640 1 --2- 172.21.15.114:0/320573563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7d0107410 0x7fd7d010de10 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.696+0000 7fd7d83b4640 1 -- 172.21.15.114:0/320573563 >> 172.21.15.114:0/320573563 conn(0x7fd7d00779d0 msgr2=0x7fd7d01053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.696+0000 7fd7d83b4640 1 -- 172.21.15.114:0/320573563 shutdown_connections 2024-05-27T07:55:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:07.696+0000 7fd7d83b4640 1 -- 172.21.15.114:0/320573563 wait complete. 2024-05-27T07:55:08.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19807 15.42 MB/sec warmup 80 sec latency 2616.188 ms 2024-05-27T07:55:08.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:08 smithi163 ceph-mon[36451]: pgmap v522: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 304 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.2 MiB/s wr, 421 op/s 2024-05-27T07:55:08.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:08 smithi186 ceph-mon[36647]: pgmap v522: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 304 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.2 MiB/s wr, 421 op/s 2024-05-27T07:55:09.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:08 smithi114 ceph-mon[32038]: pgmap v522: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 304 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.2 MiB/s wr, 421 op/s 2024-05-27T07:55:09.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19809 15.25 MB/sec warmup 81 sec latency 3072.985 ms 2024-05-27T07:55:10.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19827 15.07 MB/sec warmup 82 sec latency 1738.963 ms 2024-05-27T07:55:10.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:10 smithi163 ceph-mon[36451]: pgmap v523: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 293 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.3 MiB/s wr, 399 op/s 2024-05-27T07:55:10.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:10 smithi186 ceph-mon[36647]: pgmap v523: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 293 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.3 MiB/s wr, 399 op/s 2024-05-27T07:55:11.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:10 smithi114 ceph-mon[32038]: pgmap v523: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 293 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.3 MiB/s wr, 399 op/s 2024-05-27T07:55:11.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19870 14.94 MB/sec warmup 83 sec latency 1687.853 ms 2024-05-27T07:55:12.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19903 14.77 MB/sec warmup 84 sec latency 1391.525 ms 2024-05-27T07:55:12.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:55:12.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:12 smithi163 ceph-mon[36451]: pgmap v524: 129 pgs: 13 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 77 active+clean; 287 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 590 op/s 2024-05-27T07:55:12.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:12 smithi186 ceph-mon[36647]: pgmap v524: 129 pgs: 13 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 77 active+clean; 287 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 590 op/s 2024-05-27T07:55:13.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:12 smithi114 ceph-mon[32038]: pgmap v524: 129 pgs: 13 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 77 active+clean; 287 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 590 op/s 2024-05-27T07:55:13.224 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:55:13.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19954 14.62 MB/sec warmup 85 sec latency 1630.396 ms 2024-05-27T07:55:14.077 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:13 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:55:14.077 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:13 smithi163 systemd[1]: Stopped Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:55:14.077 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:13 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.929s CPU time. 2024-05-27T07:55:14.077 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:13 smithi163 systemd[1]: Starting Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:55:14.371 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 podman[74513]: 2024-05-27 07:55:13.980273051 +0000 UTC m=+0.023263863 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:55:14.371 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 podman[74513]: 2024-05-27 07:55:14.101639592 +0000 UTC m=+0.144630452 container create f5b8da4ac32960f75a74a28eefcf16d7aaadeae5e91e05a8aede6cc94faf1588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:55:14.372 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 podman[74513]: 2024-05-27 07:55:14.371437903 +0000 UTC m=+0.414428711 container init f5b8da4ac32960f75a74a28eefcf16d7aaadeae5e91e05a8aede6cc94faf1588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:55:14.595 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19994 14.46 MB/sec warmup 86 sec latency 1695.349 ms 2024-05-27T07:55:14.727 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 podman[74513]: 2024-05-27 07:55:14.374462605 +0000 UTC m=+0.417453404 container start f5b8da4ac32960f75a74a28eefcf16d7aaadeae5e91e05a8aede6cc94faf1588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-05-27T07:55:14.727 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: starting mds.k at 2024-05-27T07:55:14.727 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 bash[74513]: f5b8da4ac32960f75a74a28eefcf16d7aaadeae5e91e05a8aede6cc94faf1588 2024-05-27T07:55:14.727 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 systemd[1]: Started Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:55:15.006 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f004a312700 time 2024-05-27T07:55:14.969768+0000 2024-05-27T07:55:15.006 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:15.006 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:15.006 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f005cb57ba6] 2024-05-27T07:55:15.006 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f005cb57dd1] 2024-05-27T07:55:15.006 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562f801cc23d] 2024-05-27T07:55:15.006 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562f801b7971] 2024-05-27T07:55:15.006 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 5: (SnapRealm::check_cache() const+0x13e) [0x562f801b5fa0] 2024-05-27T07:55:15.006 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562f801b6651] 2024-05-27T07:55:15.007 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562f8014b347] 2024-05-27T07:55:15.007 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562f7fe4787e] 2024-05-27T07:55:15.007 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562f7fe4cf2f] 2024-05-27T07:55:15.007 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562f7fe57969] 2024-05-27T07:55:15.007 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 11: (MDLog::_replay_thread()+0x1f38) [0x562f801f048c] 2024-05-27T07:55:15.007 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562f801f99cb] 2024-05-27T07:55:15.007 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 13: (Thread::entry_wrapper()+0x43) [0x7f005cb2672d] 2024-05-27T07:55:15.007 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 14: (Thread::_entry_func(void*)+0xd) [0x7f005cb26749] 2024-05-27T07:55:15.007 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f005b6ca1ca] 2024-05-27T07:55:15.007 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 16: clone() 2024-05-27T07:55:15.007 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:14.972+0000 7f004a312700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f004a312700 time 2024-05-27T07:55:14.969768+0000 2024-05-27T07:55:15.008 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:15.008 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:15.008 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:15.008 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f005cb57ba6] 2024-05-27T07:55:15.008 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f005cb57dd1] 2024-05-27T07:55:15.008 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562f801cc23d] 2024-05-27T07:55:15.008 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562f801b7971] 2024-05-27T07:55:15.008 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 5: (SnapRealm::check_cache() const+0x13e) [0x562f801b5fa0] 2024-05-27T07:55:15.009 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562f801b6651] 2024-05-27T07:55:15.009 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562f8014b347] 2024-05-27T07:55:15.009 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562f7fe4787e] 2024-05-27T07:55:15.009 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562f7fe4cf2f] 2024-05-27T07:55:15.009 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562f7fe57969] 2024-05-27T07:55:15.009 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 11: (MDLog::_replay_thread()+0x1f38) [0x562f801f048c] 2024-05-27T07:55:15.009 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562f801f99cb] 2024-05-27T07:55:15.009 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 13: (Thread::entry_wrapper()+0x43) [0x7f005cb2672d] 2024-05-27T07:55:15.009 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 14: (Thread::_entry_func(void*)+0xd) [0x7f005cb26749] 2024-05-27T07:55:15.009 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f005b6ca1ca] 2024-05-27T07:55:15.009 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 16: clone() 2024-05-27T07:55:15.009 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:15.010 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: *** Caught signal (Aborted) ** 2024-05-27T07:55:15.010 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: in thread 7f004a312700 thread_name:md_log_replay 2024-05-27T07:55:15.010 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:15.010 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f005b6d4d20] 2024-05-27T07:55:15.010 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2: gsignal() 2024-05-27T07:55:15.010 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 3: abort() 2024-05-27T07:55:15.010 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f005cb57ced] 2024-05-27T07:55:15.010 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f005cb57dd1] 2024-05-27T07:55:15.010 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562f801cc23d] 2024-05-27T07:55:15.010 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562f801b7971] 2024-05-27T07:55:15.010 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 8: (SnapRealm::check_cache() const+0x13e) [0x562f801b5fa0] 2024-05-27T07:55:15.010 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562f801b6651] 2024-05-27T07:55:15.010 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562f8014b347] 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562f7fe4787e] 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562f7fe4cf2f] 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562f7fe57969] 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 14: (MDLog::_replay_thread()+0x1f38) [0x562f801f048c] 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562f801f99cb] 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 16: (Thread::entry_wrapper()+0x43) [0x7f005cb2672d] 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 17: (Thread::_entry_func(void*)+0xd) [0x7f005cb26749] 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f005b6ca1ca] 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 19: clone() 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:14.976+0000 7f004a312700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: in thread 7f004a312700 thread_name:md_log_replay 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:15.011 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f005b6d4d20] 2024-05-27T07:55:15.012 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2: gsignal() 2024-05-27T07:55:15.012 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 3: abort() 2024-05-27T07:55:15.012 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f005cb57ced] 2024-05-27T07:55:15.012 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f005cb57dd1] 2024-05-27T07:55:15.012 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562f801cc23d] 2024-05-27T07:55:15.012 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562f801b7971] 2024-05-27T07:55:15.012 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 8: (SnapRealm::check_cache() const+0x13e) [0x562f801b5fa0] 2024-05-27T07:55:15.012 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562f801b6651] 2024-05-27T07:55:15.012 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562f8014b347] 2024-05-27T07:55:15.013 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562f7fe4787e] 2024-05-27T07:55:15.013 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562f7fe4cf2f] 2024-05-27T07:55:15.013 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562f7fe57969] 2024-05-27T07:55:15.013 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 14: (MDLog::_replay_thread()+0x1f38) [0x562f801f048c] 2024-05-27T07:55:15.013 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562f801f99cb] 2024-05-27T07:55:15.013 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 16: (Thread::entry_wrapper()+0x43) [0x7f005cb2672d] 2024-05-27T07:55:15.013 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 17: (Thread::_entry_func(void*)+0xd) [0x7f005cb26749] 2024-05-27T07:55:15.013 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f005b6ca1ca] 2024-05-27T07:55:15.013 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 19: clone() 2024-05-27T07:55:15.014 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:55:15.014 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:15.014 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: -1> 2024-05-27T07:55:14.972+0000 7f004a312700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f004a312700 time 2024-05-27T07:55:14.969768+0000 2024-05-27T07:55:15.014 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:15.014 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:15.014 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:15.014 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f005cb57ba6] 2024-05-27T07:55:15.014 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f005cb57dd1] 2024-05-27T07:55:15.014 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562f801cc23d] 2024-05-27T07:55:15.014 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562f801b7971] 2024-05-27T07:55:15.014 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 5: (SnapRealm::check_cache() const+0x13e) [0x562f801b5fa0] 2024-05-27T07:55:15.014 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562f801b6651] 2024-05-27T07:55:15.014 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562f8014b347] 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562f7fe4787e] 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562f7fe4cf2f] 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562f7fe57969] 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 11: (MDLog::_replay_thread()+0x1f38) [0x562f801f048c] 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562f801f99cb] 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 13: (Thread::entry_wrapper()+0x43) [0x7f005cb2672d] 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 14: (Thread::_entry_func(void*)+0xd) [0x7f005cb26749] 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f005b6ca1ca] 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 16: clone() 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 0> 2024-05-27T07:55:14.976+0000 7f004a312700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: in thread 7f004a312700 thread_name:md_log_replay 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:15.015 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f005b6d4d20] 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2: gsignal() 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 3: abort() 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f005cb57ced] 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f005cb57dd1] 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562f801cc23d] 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562f801b7971] 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 8: (SnapRealm::check_cache() const+0x13e) [0x562f801b5fa0] 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562f801b6651] 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562f8014b347] 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562f7fe4787e] 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562f7fe4cf2f] 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562f7fe57969] 2024-05-27T07:55:15.016 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 14: (MDLog::_replay_thread()+0x1f38) [0x562f801f048c] 2024-05-27T07:55:15.017 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562f801f99cb] 2024-05-27T07:55:15.017 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 16: (Thread::entry_wrapper()+0x43) [0x7f005cb2672d] 2024-05-27T07:55:15.017 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 17: (Thread::_entry_func(void*)+0xd) [0x7f005cb26749] 2024-05-27T07:55:15.017 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f005b6ca1ca] 2024-05-27T07:55:15.017 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 19: clone() 2024-05-27T07:55:15.017 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:55:15.017 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:15.017 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:14 smithi163 ceph-mon[36451]: pgmap v525: 129 pgs: 13 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 77 active+clean; 288 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 573 op/s 2024-05-27T07:55:15.017 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:14 smithi163 ceph-mon[36451]: daemon mds.k restarted 2024-05-27T07:55:15.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:14 smithi114 ceph-mon[32038]: pgmap v525: 129 pgs: 13 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 77 active+clean; 288 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 573 op/s 2024-05-27T07:55:15.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:14 smithi114 ceph-mon[32038]: daemon mds.k restarted 2024-05-27T07:55:15.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:14 smithi186 ceph-mon[36647]: pgmap v525: 129 pgs: 13 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 77 active+clean; 288 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 573 op/s 2024-05-27T07:55:15.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:14 smithi186 ceph-mon[36647]: daemon mds.k restarted 2024-05-27T07:55:15.382 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: -1> 2024-05-27T07:55:14.972+0000 7f004a312700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f004a312700 time 2024-05-27T07:55:14.969768+0000 2024-05-27T07:55:15.382 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:15.382 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f005cb57ba6] 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f005cb57dd1] 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562f801cc23d] 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562f801b7971] 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 5: (SnapRealm::check_cache() const+0x13e) [0x562f801b5fa0] 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562f801b6651] 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562f8014b347] 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562f7fe4787e] 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562f7fe4cf2f] 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562f7fe57969] 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 11: (MDLog::_replay_thread()+0x1f38) [0x562f801f048c] 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562f801f99cb] 2024-05-27T07:55:15.383 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 13: (Thread::entry_wrapper()+0x43) [0x7f005cb2672d] 2024-05-27T07:55:15.384 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 14: (Thread::_entry_func(void*)+0xd) [0x7f005cb26749] 2024-05-27T07:55:15.384 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f005b6ca1ca] 2024-05-27T07:55:15.384 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 16: clone() 2024-05-27T07:55:15.384 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:15.384 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 0> 2024-05-27T07:55:14.976+0000 7f004a312700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:55:15.384 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: in thread 7f004a312700 thread_name:md_log_replay 2024-05-27T07:55:15.384 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:15.384 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:15.384 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f005b6d4d20] 2024-05-27T07:55:15.384 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2: gsignal() 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 3: abort() 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f005cb57ced] 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f005cb57dd1] 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562f801cc23d] 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562f801b7971] 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 8: (SnapRealm::check_cache() const+0x13e) [0x562f801b5fa0] 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562f801b6651] 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562f8014b347] 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562f7fe4787e] 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x562f7fe4cf2f] 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562f7fe57969] 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 14: (MDLog::_replay_thread()+0x1f38) [0x562f801f048c] 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562f801f99cb] 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 16: (Thread::entry_wrapper()+0x43) [0x7f005cb2672d] 2024-05-27T07:55:15.385 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 17: (Thread::_entry_func(void*)+0xd) [0x7f005cb26749] 2024-05-27T07:55:15.386 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f005b6ca1ca] 2024-05-27T07:55:15.386 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 19: clone() 2024-05-27T07:55:15.386 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:55:15.386 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74528]: 2024-05-27T07:55:15.386 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 podman[74563]: 2024-05-27 07:55:15.080303716 +0000 UTC m=+0.035894600 container died f5b8da4ac32960f75a74a28eefcf16d7aaadeae5e91e05a8aede6cc94faf1588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:55:15.594 INFO:tasks.workunit.client.0.smithi114.stdout: 10 20086 14.36 MB/sec warmup 87 sec latency 1478.355 ms 2024-05-27T07:55:15.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:15 smithi163 ceph-mon[36451]: osdmap e142: 12 total, 12 up, 12 in 2024-05-27T07:55:15.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:15 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6832/3235541320,v1:172.21.15.163:6833/3235541320] up:boot 2024-05-27T07:55:15.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:15 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:55:15.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:15 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:55:15.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:15 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:55:15.977 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 podman[74563]: 2024-05-27 07:55:15.712297435 +0000 UTC m=+0.667888302 container remove f5b8da4ac32960f75a74a28eefcf16d7aaadeae5e91e05a8aede6cc94faf1588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-27T07:55:15.977 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:15 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:55:16.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:15 smithi114 ceph-mon[32038]: osdmap e142: 12 total, 12 up, 12 in 2024-05-27T07:55:16.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:15 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6832/3235541320,v1:172.21.15.163:6833/3235541320] up:boot 2024-05-27T07:55:16.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:15 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:55:16.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:15 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:55:16.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:15 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:55:16.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:15 smithi186 ceph-mon[36647]: osdmap e142: 12 total, 12 up, 12 in 2024-05-27T07:55:16.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:15 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6832/3235541320,v1:172.21.15.163:6833/3235541320] up:boot 2024-05-27T07:55:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:15 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:55:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:15 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:55:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:15 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:55:16.476 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:16 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:55:16.595 INFO:tasks.workunit.client.0.smithi114.stdout: 10 20248 14.29 MB/sec warmup 88 sec latency 746.962 ms 2024-05-27T07:55:16.861 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:16 smithi114 ceph-mon[32038]: pgmap v527: 129 pgs: 9 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 94 active+clean; 288 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 3.7 MiB/s wr, 556 op/s 2024-05-27T07:55:17.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.028+0000 7f36060d0700 1 -- 172.21.15.114:0/260557049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3600107de0 msgr2=0x7f36001081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:17.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.028+0000 7f36060d0700 1 --2- 172.21.15.114:0/260557049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3600107de0 0x7f36001081f0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f35f40099c0 tx=0x7f35f4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.028+0000 7f36060d0700 1 -- 172.21.15.114:0/260557049 shutdown_connections 2024-05-27T07:55:17.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.028+0000 7f36060d0700 1 --2- 172.21.15.114:0/260557049 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3600108730 0x7f3600110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.028+0000 7f36060d0700 1 --2- 172.21.15.114:0/260557049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f36001034b0 0x7f36001038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.028+0000 7f36060d0700 1 --2- 172.21.15.114:0/260557049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3600107de0 0x7f36001081f0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.028+0000 7f36060d0700 1 -- 172.21.15.114:0/260557049 >> 172.21.15.114:0/260557049 conn(0x7f36000fede0 msgr2=0x7f3600101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:17.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.028+0000 7f36060d0700 1 -- 172.21.15.114:0/260557049 shutdown_connections 2024-05-27T07:55:17.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.028+0000 7f36060d0700 1 -- 172.21.15.114:0/260557049 wait complete. 2024-05-27T07:55:17.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.029+0000 7f36060d0700 1 Processor -- start 2024-05-27T07:55:17.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.029+0000 7f36060d0700 1 -- start start 2024-05-27T07:55:17.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.029+0000 7f36060d0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36001034b0 0x7f360019f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.029+0000 7f36060d0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3600107de0 0x7f360019fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.029+0000 7f36060d0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3600108730 0x7f3600199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.029+0000 7f36060d0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3600113340 con 0x7f36001034b0 2024-05-27T07:55:17.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.029+0000 7f36060d0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f36001131c0 con 0x7f3600108730 2024-05-27T07:55:17.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.029+0000 7f36060d0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36001134c0 con 0x7f3600107de0 2024-05-27T07:55:17.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.030+0000 7f35fffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3600108730 0x7f3600199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.030+0000 7f35fffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3600108730 0x7f3600199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:60414/0 (socket says 172.21.15.114:60414) 2024-05-27T07:55:17.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.030+0000 7f35fffff700 1 -- 172.21.15.114:0/390424243 learned_addr learned my addr 172.21.15.114:0/390424243 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:17.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.030+0000 7f35feffd700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3600107de0 0x7f360019fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.030+0000 7f35ff7fe700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36001034b0 0x7f360019f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.030+0000 7f35fffff700 1 -- 172.21.15.114:0/390424243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3600107de0 msgr2=0x7f360019fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:17.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.030+0000 7f35fffff700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3600107de0 0x7f360019fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.030+0000 7f35fffff700 1 -- 172.21.15.114:0/390424243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36001034b0 msgr2=0x7f360019f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:17.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.030+0000 7f35fffff700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36001034b0 0x7f360019f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.030+0000 7f35fffff700 1 -- 172.21.15.114:0/390424243 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f360019a270 con 0x7f3600108730 2024-05-27T07:55:17.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.030+0000 7f35feffd700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3600107de0 0x7f360019fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:55:17.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.030+0000 7f35fffff700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3600108730 0x7f3600199b40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f35f800ef30 tx=0x7f35f800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:17.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.031+0000 7f35fcff9700 1 -- 172.21.15.114:0/390424243 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f80092e0 con 0x7f3600108730 2024-05-27T07:55:17.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.031+0000 7f36060d0700 1 -- 172.21.15.114:0/390424243 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f360019a500 con 0x7f3600108730 2024-05-27T07:55:17.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.031+0000 7f35fcff9700 1 -- 172.21.15.114:0/390424243 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35f8004bb0 con 0x7f3600108730 2024-05-27T07:55:17.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.031+0000 7f35fcff9700 1 -- 172.21.15.114:0/390424243 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f8004020 con 0x7f3600108730 2024-05-27T07:55:17.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.031+0000 7f36060d0700 1 -- 172.21.15.114:0/390424243 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36000759e0 con 0x7f3600108730 2024-05-27T07:55:17.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.034+0000 7f35fcff9700 1 -- 172.21.15.114:0/390424243 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f35f801b030 con 0x7f3600108730 2024-05-27T07:55:17.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.034+0000 7f35fcff9700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35e80b29f0 0x7f35e80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.034+0000 7f35fcff9700 1 -- 172.21.15.114:0/390424243 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 9554+0+0 (secure 0 0 0) 0x7f35f80e2480 con 0x7f3600108730 2024-05-27T07:55:17.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.035+0000 7f35ff7fe700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35e80b29f0 0x7f35e80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.035+0000 7f35ff7fe700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35e80b29f0 0x7f35e80b4f00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009c60 tx=0x7f35f0009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:17.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.036+0000 7f3614ae5b80 1 Processor -- start 2024-05-27T07:55:17.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.036+0000 7f3614ae5b80 1 -- start start 2024-05-27T07:55:17.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.036+0000 7f3614ae5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc25efe40 0x55ccc2773710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.036+0000 7f3614ae5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc2773c50 0x55ccc276da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f3614ae5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc276df80 0x55ccc276e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f3614ae5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ccc269cc90 con 0x55ccc2773c50 2024-05-27T07:55:17.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f35feffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc2773c50 0x55ccc276da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f35ff7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc25efe40 0x55ccc2773710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f35feffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc2773c50 0x55ccc276da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51886/0 (socket says 172.21.15.114:51886) 2024-05-27T07:55:17.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f35ff7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc25efe40 0x55ccc2773710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:41780/0 (socket says 172.21.15.114:41780) 2024-05-27T07:55:17.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f35feffd700 1 -- 172.21.15.114:0/3969704127 learned_addr learned my addr 172.21.15.114:0/3969704127 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:17.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f3614ae5b80 1 -- 172.21.15.114:0/3969704127 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ccc268c830 con 0x55ccc276df80 2024-05-27T07:55:17.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f3614ae5b80 1 -- 172.21.15.114:0/3969704127 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ccc2690870 con 0x55ccc25efe40 2024-05-27T07:55:17.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f35fffff700 1 --2- 172.21.15.114:0/3969704127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc276df80 0x55ccc276e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f35feffd700 1 -- 172.21.15.114:0/3969704127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc25efe40 msgr2=0x55ccc2773710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:17.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f35feffd700 1 --2- 172.21.15.114:0/3969704127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc25efe40 0x55ccc2773710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f35feffd700 1 -- 172.21.15.114:0/3969704127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc276df80 msgr2=0x55ccc276e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:17.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f35feffd700 1 --2- 172.21.15.114:0/3969704127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc276df80 0x55ccc276e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f35feffd700 1 -- 172.21.15.114:0/3969704127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccc25dc8d0 con 0x55ccc2773c50 2024-05-27T07:55:17.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f35fffff700 1 --2- 172.21.15.114:0/3969704127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc276df80 0x55ccc276e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:55:17.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.037+0000 7f35feffd700 1 --2- 172.21.15.114:0/3969704127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc2773c50 0x55ccc276da40 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f3600076b30 tx=0x7f35f4005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d647bdc6a6e030a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:17.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.038+0000 7f35e5ffb700 1 -- 172.21.15.114:0/3969704127 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4027070 con 0x55ccc2773c50 2024-05-27T07:55:17.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.038+0000 7f35e5ffb700 1 -- 172.21.15.114:0/3969704127 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35f4005550 con 0x55ccc2773c50 2024-05-27T07:55:17.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.038+0000 7f35e5ffb700 1 -- 172.21.15.114:0/3969704127 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4003d00 con 0x55ccc2773c50 2024-05-27T07:55:17.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.039+0000 7f3614ae5b80 1 -- 172.21.15.114:0/3969704127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc2773c50 msgr2=0x55ccc276da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:17.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.039+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/3969704127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc2773c50 0x55ccc276da40 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f3600076b30 tx=0x7f35f4005fb0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.039+0000 7f3614ae5b80 1 -- 172.21.15.114:0/3969704127 shutdown_connections 2024-05-27T07:55:17.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.039+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/3969704127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc276df80 0x55ccc276e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.039+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/3969704127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc25efe40 0x55ccc2773710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.039+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/3969704127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc2773c50 0x55ccc276da40 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.039+0000 7f3614ae5b80 1 -- 172.21.15.114:0/3969704127 >> 172.21.15.114:0/3969704127 conn(0x55ccc264aaf0 msgr2=0x55ccc2768640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:17.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.039+0000 7f3614ae5b80 1 -- 172.21.15.114:0/3969704127 shutdown_connections 2024-05-27T07:55:17.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.040+0000 7f3614ae5b80 1 -- 172.21.15.114:0/3969704127 wait complete. 2024-05-27T07:55:17.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.040+0000 7f3614ae5b80 1 -- start start 2024-05-27T07:55:17.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.041+0000 7f3614ae5b80 1 Processor -- start 2024-05-27T07:55:17.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.041+0000 7f3614ae5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc25efe40 0x55ccc2777300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.041+0000 7f3614ae5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc276df80 0x55ccc2777840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.041+0000 7f3614ae5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc2773c50 0x55ccc2777d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.041+0000 7f3614ae5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ccc2592fa0 con 0x55ccc25efe40 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.041+0000 7f3614ae5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ccc262e7c0 con 0x55ccc2773c50 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.041+0000 7f3614ae5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ccc277a6e0 con 0x55ccc276df80 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.041+0000 7f35feffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc276df80 0x55ccc2777840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.042+0000 7f35feffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc276df80 0x55ccc2777840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:41786/0 (socket says 172.21.15.114:41786) 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.042+0000 7f35fffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc2773c50 0x55ccc2777d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.042+0000 7f35feffd700 1 -- 172.21.15.114:0/1886278379 learned_addr learned my addr 172.21.15.114:0/1886278379 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.042+0000 7f35ff7fe700 1 --2- 172.21.15.114:0/1886278379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc25efe40 0x55ccc2777300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.042+0000 7f3614ae5b80 10 client.0 _ll_register_callbacks cb 0x55ccc2627920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.042+0000 7f35fffff700 1 -- 172.21.15.114:0/1886278379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc276df80 msgr2=0x55ccc2777840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.042+0000 7f35fffff700 1 --2- 172.21.15.114:0/1886278379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc276df80 0x55ccc2777840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.042+0000 7f35fffff700 1 -- 172.21.15.114:0/1886278379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc25efe40 msgr2=0x55ccc2777300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:17.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.042+0000 7f35fffff700 1 --2- 172.21.15.114:0/1886278379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc25efe40 0x55ccc2777300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.042+0000 7f35fffff700 1 -- 172.21.15.114:0/1886278379 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccc262b690 con 0x55ccc2773c50 2024-05-27T07:55:17.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.042+0000 7f35fffff700 1 --2- 172.21.15.114:0/1886278379 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc2773c50 0x55ccc2777d80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f35f80df7c0 tx=0x7f35f80e23a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9aab795274a8b38b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:17.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.042+0000 7f35deffd700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:55:17.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.042+0000 7f35deffd700 1 -- 172.21.15.114:0/1886278379 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f80ad790 con 0x55ccc2773c50 2024-05-27T07:55:17.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.043+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1886278379 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35f4005550 con 0x55ccc2773c50 2024-05-27T07:55:17.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.043+0000 7f35deffd700 1 -- 172.21.15.114:0/1886278379 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 9554+0+0 (secure 0 0 0) 0x7f35f80b1f30 con 0x55ccc2773c50 2024-05-27T07:55:17.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.043+0000 7f35deffd700 1 -- 172.21.15.114:0/1886278379 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f35f80b21b0 con 0x55ccc2773c50 2024-05-27T07:55:17.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.043+0000 7f3614ae5b80 10 client.28442 fetch_fsmap learned FSMap version 85 2024-05-27T07:55:17.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.043+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1886278379 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=4+}) -- 0x55ccc27851f0 con 0x55ccc2773c50 2024-05-27T07:55:17.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.043+0000 7f35deffd700 1 -- 172.21.15.114:0/1886278379 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35f80a3b00 con 0x55ccc2773c50 2024-05-27T07:55:17.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.043+0000 7f35deffd700 1 -- 172.21.15.114:0/1886278379 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 85) ==== 4700+0+0 (secure 0 0 0) 0x7f35f80a80b0 con 0x55ccc2773c50 2024-05-27T07:55:17.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.043+0000 7f3614ae5b80 10 client.28442 fetch_fsmap finished waiting for FSMap version 85 2024-05-27T07:55:17.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.043+0000 7f3614ae5b80 10 client.28442 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:17.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.043+0000 7f3614ae5b80 20 client.28442 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:17.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.044+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1886278379 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ccc2787b80 0x55ccc278a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.044+0000 7f3614ae5b80 4 client.28442 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:55:17.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.044+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1886278379 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f35f4005550 con 0x55ccc2787b80 2024-05-27T07:55:17.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.044+0000 7f35ff7fe700 1 --2- 172.21.15.114:0/1886278379 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ccc2787b80 0x55ccc278a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.044+0000 7f35deffd700 10 client.28442 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:17.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.044+0000 7f35ff7fe700 1 --2- 172.21.15.114:0/1886278379 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ccc2787b80 0x55ccc278a050 crc :-1 s=READY pgs=432 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e151962be1716454 server_cookie=8cb76b3288e22ffb in_seq=0 out_seq=0 2024-05-27T07:55:17.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.046+0000 7f35deffd700 1 -- 172.21.15.114:0/1886278379 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f35f4005550 con 0x55ccc2787b80 2024-05-27T07:55:17.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.046+0000 7f35deffd700 10 client.28442 handle_command_reply: tid=0 2024-05-27T07:55:17.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.046+0000 7f3614ae5b80 1 client.28442 shutdown 2024-05-27T07:55:17.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.046+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1886278379 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc2773c50 msgr2=0x55ccc2777d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:17.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.047+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1886278379 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc2773c50 0x55ccc2777d80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f35f80df7c0 tx=0x7f35f80e23a0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.047+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1886278379 shutdown_connections 2024-05-27T07:55:17.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.047+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1886278379 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ccc2787b80 0x55ccc278a050 crc :-1 s=READY pgs=432 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.047+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1886278379 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc2773c50 0x55ccc2777d80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.047+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1886278379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc25efe40 0x55ccc2777300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.047+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1886278379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc276df80 0x55ccc2777840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.047+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1886278379 >> 172.21.15.114:0/1886278379 conn(0x55ccc264aaf0 msgr2=0x55ccc276ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:17.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.047+0000 7f35deffd700 0 client.28442 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:17.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.047+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1886278379 shutdown_connections 2024-05-27T07:55:17.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.047+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1886278379 wait complete. 2024-05-27T07:55:17.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.047+0000 7f3614ae5b80 20 client.28442 trim_cache size 0 max 16384 2024-05-27T07:55:17.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.078+0000 7f3614ae5b80 1 Processor -- start 2024-05-27T07:55:17.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.078+0000 7f3614ae5b80 1 -- start start 2024-05-27T07:55:17.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.079+0000 7f3614ae5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc25efe40 0x55ccc276da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.079+0000 7f3614ae5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc276df80 0x55ccc2789f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.079+0000 7f3614ae5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc2773c50 0x55ccc2772ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.079+0000 7f3614ae5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f36001b2c30 con 0x55ccc276df80 2024-05-27T07:55:17.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.079+0000 7f3614ae5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f36001b3300 con 0x55ccc25efe40 2024-05-27T07:55:17.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.079+0000 7f3614ae5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36001b1b30 con 0x55ccc2773c50 2024-05-27T07:55:17.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.079+0000 7f35feffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc276df80 0x55ccc2789f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.079+0000 7f35feffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc276df80 0x55ccc2789f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51910/0 (socket says 172.21.15.114:51910) 2024-05-27T07:55:17.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.079+0000 7f35feffd700 1 -- 172.21.15.114:0/1579142419 learned_addr learned my addr 172.21.15.114:0/1579142419 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:17.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.079+0000 7f35fffff700 1 --2- 172.21.15.114:0/1579142419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc2773c50 0x55ccc2772ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.079+0000 7f35ff7fe700 1 --2- 172.21.15.114:0/1579142419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc25efe40 0x55ccc276da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.080+0000 7f35feffd700 1 -- 172.21.15.114:0/1579142419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc2773c50 msgr2=0x55ccc2772ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:17.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.080+0000 7f35feffd700 1 --2- 172.21.15.114:0/1579142419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc2773c50 0x55ccc2772ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.080+0000 7f35feffd700 1 -- 172.21.15.114:0/1579142419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc25efe40 msgr2=0x55ccc276da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:17.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.080+0000 7f35feffd700 1 --2- 172.21.15.114:0/1579142419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc25efe40 0x55ccc276da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.080+0000 7f35feffd700 1 -- 172.21.15.114:0/1579142419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccc27697c0 con 0x55ccc276df80 2024-05-27T07:55:17.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.080+0000 7f35ff7fe700 1 --2- 172.21.15.114:0/1579142419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc25efe40 0x55ccc276da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:55:17.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.080+0000 7f35feffd700 1 --2- 172.21.15.114:0/1579142419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc276df80 0x55ccc2789f00 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f35f4003830 tx=0x7f35f4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a786aef4588db21 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:17.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.080+0000 7f35e57fa700 1 -- 172.21.15.114:0/1579142419 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4032070 con 0x55ccc276df80 2024-05-27T07:55:17.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.080+0000 7f35e57fa700 1 -- 172.21.15.114:0/1579142419 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35f4003d80 con 0x55ccc276df80 2024-05-27T07:55:17.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.080+0000 7f35e57fa700 1 -- 172.21.15.114:0/1579142419 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4027040 con 0x55ccc276df80 2024-05-27T07:55:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.081+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1579142419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc276df80 msgr2=0x55ccc2789f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.081+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1579142419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc276df80 0x55ccc2789f00 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f35f4003830 tx=0x7f35f4009310 comp rx=0 tx=0).stop 2024-05-27T07:55:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.081+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1579142419 shutdown_connections 2024-05-27T07:55:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.081+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1579142419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc2773c50 0x55ccc2772ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.082+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1579142419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc25efe40 0x55ccc276da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.082+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1579142419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc276df80 0x55ccc2789f00 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.082+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1579142419 >> 172.21.15.114:0/1579142419 conn(0x55ccc264aaf0 msgr2=0x55ccc27723b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.082+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1579142419 shutdown_connections 2024-05-27T07:55:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.082+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1579142419 wait complete. 2024-05-27T07:55:17.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.083+0000 7f3614ae5b80 1 -- start start 2024-05-27T07:55:17.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.083+0000 7f3614ae5b80 1 Processor -- start 2024-05-27T07:55:17.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.083+0000 7f3614ae5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc25efe40 0x55ccc277fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f3614ae5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc276df80 0x55ccc2780110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f3614ae5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc2773c50 0x55ccc278e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f3614ae5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f36001b0d90 con 0x55ccc25efe40 2024-05-27T07:55:17.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f3614ae5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f36001b0720 con 0x55ccc276df80 2024-05-27T07:55:17.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f3614ae5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36001b1460 con 0x55ccc2773c50 2024-05-27T07:55:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f3614ae5b80 10 client.0 _ll_register_callbacks cb 0x55ccc2627920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f35ff7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc25efe40 0x55ccc277fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f35feffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc276df80 0x55ccc2780110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f35ff7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc25efe40 0x55ccc277fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51926/0 (socket says 172.21.15.114:51926) 2024-05-27T07:55:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f35feffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc276df80 0x55ccc2780110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:60450/0 (socket says 172.21.15.114:60450) 2024-05-27T07:55:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f35ff7fe700 1 -- 172.21.15.114:0/1340962542 learned_addr learned my addr 172.21.15.114:0/1340962542 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f35fffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc2773c50 0x55ccc278e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f35feffd700 1 -- 172.21.15.114:0/1340962542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc2773c50 msgr2=0x55ccc278e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:17.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f35feffd700 1 --2- 172.21.15.114:0/1340962542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc2773c50 0x55ccc278e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f35feffd700 1 -- 172.21.15.114:0/1340962542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc25efe40 msgr2=0x55ccc277fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:17.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f35feffd700 1 --2- 172.21.15.114:0/1340962542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc25efe40 0x55ccc277fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.084+0000 7f35feffd700 1 -- 172.21.15.114:0/1340962542 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccc278ef90 con 0x55ccc276df80 2024-05-27T07:55:17.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.085+0000 7f35feffd700 1 --2- 172.21.15.114:0/1340962542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc276df80 0x55ccc2780110 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f35f4030040 tx=0x7f35f400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20402a02280e5f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:17.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.085+0000 7f35deffd700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:55:17.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.085+0000 7f35deffd700 1 -- 172.21.15.114:0/1340962542 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4027070 con 0x55ccc276df80 2024-05-27T07:55:17.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.085+0000 7f35deffd700 1 -- 172.21.15.114:0/1340962542 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 9554+0+0 (secure 0 0 0) 0x7f35f40050d0 con 0x55ccc276df80 2024-05-27T07:55:17.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.085+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1340962542 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35f4003d80 con 0x55ccc276df80 2024-05-27T07:55:17.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.085+0000 7f35deffd700 1 -- 172.21.15.114:0/1340962542 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f35f4032040 con 0x55ccc276df80 2024-05-27T07:55:17.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.086+0000 7f3614ae5b80 10 client.28448 fetch_fsmap learned FSMap version 85 2024-05-27T07:55:17.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.086+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1340962542 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=4+}) -- 0x55ccc27fa720 con 0x55ccc276df80 2024-05-27T07:55:17.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.086+0000 7f35deffd700 1 -- 172.21.15.114:0/1340962542 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35f4003d80 con 0x55ccc276df80 2024-05-27T07:55:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.087+0000 7f35deffd700 1 -- 172.21.15.114:0/1340962542 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 85) ==== 4700+0+0 (secure 0 0 0) 0x7f35f4021070 con 0x55ccc276df80 2024-05-27T07:55:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.087+0000 7f3614ae5b80 10 client.28448 fetch_fsmap finished waiting for FSMap version 85 2024-05-27T07:55:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.087+0000 7f3614ae5b80 10 client.28448 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.087+0000 7f3614ae5b80 20 client.28448 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.087+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1340962542 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ccc27fac10 0x55ccc27fd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.087+0000 7f3614ae5b80 4 client.28448 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:55:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.087+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1340962542 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ccc27399f0 con 0x55ccc27fac10 2024-05-27T07:55:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.087+0000 7f35ff7fe700 1 --2- 172.21.15.114:0/1340962542 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ccc27fac10 0x55ccc27fd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.088+0000 7f35ff7fe700 1 --2- 172.21.15.114:0/1340962542 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ccc27fac10 0x55ccc27fd030 crc :-1 s=READY pgs=433 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a35fb1a93197149 server_cookie=516fa56d2bedf2a3 in_seq=0 out_seq=0 2024-05-27T07:55:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.088+0000 7f35deffd700 10 client.28448 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:17.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f35deffd700 1 -- 172.21.15.114:0/1340962542 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ccc27399f0 con 0x55ccc27fac10 2024-05-27T07:55:17.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f35deffd700 10 client.28448 handle_command_reply: tid=0 2024-05-27T07:55:17.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f3614ae5b80 1 client.28448 shutdown 2024-05-27T07:55:17.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1340962542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc276df80 msgr2=0x55ccc2780110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:17.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1340962542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc276df80 0x55ccc2780110 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f35f4030040 tx=0x7f35f400b8d0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1340962542 shutdown_connections 2024-05-27T07:55:17.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1340962542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc2773c50 0x55ccc278e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1340962542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccc25efe40 0x55ccc277fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1340962542 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ccc27fac10 0x55ccc27fd030 crc :-1 s=READY pgs=433 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f3614ae5b80 1 --2- 172.21.15.114:0/1340962542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ccc276df80 0x55ccc2780110 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1340962542 >> 172.21.15.114:0/1340962542 conn(0x55ccc264aaf0 msgr2=0x55ccc2770e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:17.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f35deffd700 0 client.28448 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:17.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1340962542 shutdown_connections 2024-05-27T07:55:17.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f3614ae5b80 1 -- 172.21.15.114:0/1340962542 wait complete. 2024-05-27T07:55:17.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.091+0000 7f3614ae5b80 20 client.28448 trim_cache size 0 max 16384 2024-05-27T07:55:17.093 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:55:17.093 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:55:17.094 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:55:17.094 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:55:17.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.092+0000 7f36060d0700 1 -- 172.21.15.114:0/390424243 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35e80b29f0 msgr2=0x7f35e80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:17.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.092+0000 7f36060d0700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35e80b29f0 0x7f35e80b4f00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009c60 tx=0x7f35f0009260 comp rx=0 tx=0).stop 2024-05-27T07:55:17.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.092+0000 7f36060d0700 1 -- 172.21.15.114:0/390424243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3600108730 msgr2=0x7f3600199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:17.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.092+0000 7f36060d0700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3600108730 0x7f3600199b40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f35f800ef30 tx=0x7f35f800c3a0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.092+0000 7f36060d0700 1 -- 172.21.15.114:0/390424243 shutdown_connections 2024-05-27T07:55:17.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.092+0000 7f36060d0700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f35e80b29f0 0x7f35e80b4f00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.092+0000 7f36060d0700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3600108730 0x7f3600199b40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.092+0000 7f36060d0700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36001034b0 0x7f360019f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.092+0000 7f36060d0700 1 --2- 172.21.15.114:0/390424243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3600107de0 0x7f360019fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:17.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.092+0000 7f36060d0700 1 -- 172.21.15.114:0/390424243 >> 172.21.15.114:0/390424243 conn(0x7f36000fede0 msgr2=0x7f3600109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:17.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.092+0000 7f36060d0700 1 -- 172.21.15.114:0/390424243 shutdown_connections 2024-05-27T07:55:17.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:17.092+0000 7f36060d0700 1 -- 172.21.15.114:0/390424243 wait complete. 2024-05-27T07:55:17.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:16 smithi163 ceph-mon[36451]: pgmap v527: 129 pgs: 9 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 94 active+clean; 288 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 3.7 MiB/s wr, 556 op/s 2024-05-27T07:55:17.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:16 smithi186 ceph-mon[36647]: pgmap v527: 129 pgs: 9 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 94 active+clean; 288 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 3.7 MiB/s wr, 556 op/s 2024-05-27T07:55:17.595 INFO:tasks.workunit.client.0.smithi114.stdout: 10 20543 14.37 MB/sec warmup 89 sec latency 590.995 ms 2024-05-27T07:55:18.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:17 smithi114 ceph-mon[32038]: pgmap v528: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 293 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 3.9 MiB/s wr, 554 op/s 2024-05-27T07:55:18.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:17 smithi163 ceph-mon[36451]: pgmap v528: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 293 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 3.9 MiB/s wr, 554 op/s 2024-05-27T07:55:18.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:17 smithi186 ceph-mon[36647]: pgmap v528: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 293 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 3.9 MiB/s wr, 554 op/s 2024-05-27T07:55:18.277 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:55:18.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:55:18.597 INFO:tasks.workunit.client.0.smithi114.stdout: 10 20776 14.41 MB/sec warmup 90 sec latency 258.308 ms 2024-05-27T07:55:18.677 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:55:19.597 INFO:tasks.workunit.client.0.smithi114.stdout: 10 21153 14.46 MB/sec warmup 91 sec latency 174.568 ms 2024-05-27T07:55:20.597 INFO:tasks.workunit.client.0.smithi114.stdout: 10 21468 14.47 MB/sec warmup 92 sec latency 163.204 ms 2024-05-27T07:55:20.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.602+0000 7f6d5359e700 1 -- 172.21.15.114:0/243520046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d5410a7b0 msgr2=0x7f6d5410abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:20.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.602+0000 7f6d5359e700 1 --2- 172.21.15.114:0/243520046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d5410a7b0 0x7f6d5410abc0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009990 tx=0x7f6d44009ca0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.608+0000 7f6d5359e700 1 -- 172.21.15.114:0/243520046 shutdown_connections 2024-05-27T07:55:20.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.608+0000 7f6d5359e700 1 --2- 172.21.15.114:0/243520046 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5411d920 0x7f6d5411fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.608+0000 7f6d5359e700 1 --2- 172.21.15.114:0/243520046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d5410a7b0 0x7f6d5410abc0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.608+0000 7f6d5359e700 1 --2- 172.21.15.114:0/243520046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d5410b100 0x7f6d5411d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.608+0000 7f6d5359e700 1 -- 172.21.15.114:0/243520046 >> 172.21.15.114:0/243520046 conn(0x7f6d5406f280 msgr2=0x7f6d5406f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:20.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.612+0000 7f6d5359e700 1 -- 172.21.15.114:0/243520046 shutdown_connections 2024-05-27T07:55:20.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.612+0000 7f6d5359e700 1 -- 172.21.15.114:0/243520046 wait complete. 2024-05-27T07:55:20.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.613+0000 7f6d5359e700 1 Processor -- start 2024-05-27T07:55:20.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.613+0000 7f6d5359e700 1 -- start start 2024-05-27T07:55:20.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.613+0000 7f6d5359e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5410a7b0 0x7f6d5411d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.613+0000 7f6d5359e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d5410b100 0x7f6d54117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.613+0000 7f6d5359e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d5411d920 0x7f6d54117b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.613+0000 7f6d5359e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541134c0 con 0x7f6d5410b100 2024-05-27T07:55:20.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.613+0000 7f6d5359e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6d54113340 con 0x7f6d5410a7b0 2024-05-27T07:55:20.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.613+0000 7f6d5359e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d54113640 con 0x7f6d5411d920 2024-05-27T07:55:20.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.614+0000 7f6d5259c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5410a7b0 0x7f6d5411d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.614+0000 7f6d5259c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5410a7b0 0x7f6d5411d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:60472/0 (socket says 172.21.15.114:60472) 2024-05-27T07:55:20.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.614+0000 7f6d5259c700 1 -- 172.21.15.114:0/1510031153 learned_addr learned my addr 172.21.15.114:0/1510031153 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:20.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.614+0000 7f6d52d9d700 1 --2- 172.21.15.114:0/1510031153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d5411d920 0x7f6d54117b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.614+0000 7f6d5259c700 1 -- 172.21.15.114:0/1510031153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d5411d920 msgr2=0x7f6d54117b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:20.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.614+0000 7f6d5259c700 1 --2- 172.21.15.114:0/1510031153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d5411d920 0x7f6d54117b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.614+0000 7f6d5259c700 1 -- 172.21.15.114:0/1510031153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d5410b100 msgr2=0x7f6d54117480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:55:20.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.614+0000 7f6d5259c700 1 --2- 172.21.15.114:0/1510031153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d5410b100 0x7f6d54117480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.614+0000 7f6d5259c700 1 -- 172.21.15.114:0/1510031153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d541be5d0 con 0x7f6d5410a7b0 2024-05-27T07:55:20.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.615+0000 7f6d5259c700 1 --2- 172.21.15.114:0/1510031153 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5410a7b0 0x7f6d5411d360 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6d440057f0 tx=0x7f6d4400b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:20.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.617+0000 7f6d437fe700 1 -- 172.21.15.114:0/1510031153 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d44026070 con 0x7f6d5410a7b0 2024-05-27T07:55:20.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.617+0000 7f6d5359e700 1 -- 172.21.15.114:0/1510031153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d541be7a0 con 0x7f6d5410a7b0 2024-05-27T07:55:20.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.617+0000 7f6d5359e700 1 -- 172.21.15.114:0/1510031153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d541bec50 con 0x7f6d5410a7b0 2024-05-27T07:55:20.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.617+0000 7f6d437fe700 1 -- 172.21.15.114:0/1510031153 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d44021410 con 0x7f6d5410a7b0 2024-05-27T07:55:20.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.617+0000 7f6d437fe700 1 -- 172.21.15.114:0/1510031153 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d44017650 con 0x7f6d5410a7b0 2024-05-27T07:55:20.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.621+0000 7f6d5359e700 1 -- 172.21.15.114:0/1510031153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d34005360 con 0x7f6d5410a7b0 2024-05-27T07:55:20.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.623+0000 7f6d437fe700 1 -- 172.21.15.114:0/1510031153 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6d44009d10 con 0x7f6d5410a7b0 2024-05-27T07:55:20.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.624+0000 7f6d437fe700 1 --2- 172.21.15.114:0/1510031153 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6d3c0b2c00 0x7f6d3c0b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.624+0000 7f6d437fe700 1 -- 172.21.15.114:0/1510031153 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 9554+0+0 (secure 0 0 0) 0x7f6d440e3410 con 0x7f6d5410a7b0 2024-05-27T07:55:20.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.625+0000 7f6d51d9b700 1 --2- 172.21.15.114:0/1510031153 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6d3c0b2c00 0x7f6d3c0b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.625+0000 7f6d51d9b700 1 --2- 172.21.15.114:0/1510031153 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6d3c0b2c00 0x7f6d3c0b5110 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6d5411fcc0 tx=0x7f6d48006c60 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:20.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.636+0000 7f6d437fe700 1 -- 172.21.15.114:0/1510031153 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6d44013070 con 0x7f6d5410a7b0 2024-05-27T07:55:20.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:20 smithi114 ceph-mon[32038]: pgmap v529: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 291 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 511 op/s 2024-05-27T07:55:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.811+0000 7f4bff11f640 1 -- 172.21.15.114:0/4194626839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bf81032f0 msgr2=0x7f4bf810f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.811+0000 7f4bff11f640 1 --2- 172.21.15.114:0/4194626839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bf81032f0 0x7f4bf810f6c0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0009ca0 tx=0x7f4bf002a8c0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.811+0000 7f4bff11f640 1 -- 172.21.15.114:0/4194626839 shutdown_connections 2024-05-27T07:55:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.811+0000 7f4bff11f640 1 --2- 172.21.15.114:0/4194626839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bf81032f0 0x7f4bf810f6c0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.811+0000 7f4bff11f640 1 --2- 172.21.15.114:0/4194626839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bf8109040 0x7f4bf8102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.811+0000 7f4bff11f640 1 --2- 172.21.15.114:0/4194626839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bf8108ba0 0x7f4bf8102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.811+0000 7f4bff11f640 1 -- 172.21.15.114:0/4194626839 >> 172.21.15.114:0/4194626839 conn(0x7f4bf8078270 msgr2=0x7f4bf80786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.811+0000 7f4bff11f640 1 -- 172.21.15.114:0/4194626839 shutdown_connections 2024-05-27T07:55:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.811+0000 7f4bff11f640 1 -- 172.21.15.114:0/4194626839 wait complete. 2024-05-27T07:55:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.812+0000 7f4bff11f640 1 Processor -- start 2024-05-27T07:55:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.812+0000 7f4bff11f640 1 -- start start 2024-05-27T07:55:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bff11f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bf81032f0 0x7f4bf8199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bff11f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bf8108ba0 0x7f4bf8199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bff11f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bf8109040 0x7f4bf819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bff11f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf8112db0 con 0x7f4bf81032f0 2024-05-27T07:55:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bff11f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf8112c30 con 0x7f4bf8108ba0 2024-05-27T07:55:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bff11f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf8112f30 con 0x7f4bf8109040 2024-05-27T07:55:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bfcebb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bf81032f0 0x7f4bf8199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bfcebb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bf81032f0 0x7f4bf8199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51950/0 (socket says 172.21.15.114:51950) 2024-05-27T07:55:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bfcebb640 1 -- 172.21.15.114:0/4088192116 learned_addr learned my addr 172.21.15.114:0/4088192116 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bf8109040 0x7f4bf819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bf7fff640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bf8108ba0 0x7f4bf8199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bfcebb640 1 -- 172.21.15.114:0/4088192116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bf8109040 msgr2=0x7f4bf819df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bfcebb640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bf8109040 0x7f4bf819df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bfcebb640 1 -- 172.21.15.114:0/4088192116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bf8108ba0 msgr2=0x7f4bf8199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bfcebb640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bf8108ba0 0x7f4bf8199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bfcebb640 1 -- 172.21.15.114:0/4088192116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf819e5d0 con 0x7f4bf81032f0 2024-05-27T07:55:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.813+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bf8109040 0x7f4bf819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:55:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.814+0000 7f4bfcebb640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bf81032f0 0x7f4bf8199770 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00e9f0 tx=0x7f4bec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.814+0000 7f4bf5ffb640 1 -- 172.21.15.114:0/4088192116 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec00ccc0 con 0x7f4bf81032f0 2024-05-27T07:55:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.814+0000 7f4bf5ffb640 1 -- 172.21.15.114:0/4088192116 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bec0054d0 con 0x7f4bf81032f0 2024-05-27T07:55:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.814+0000 7f4bf5ffb640 1 -- 172.21.15.114:0/4088192116 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec010430 con 0x7f4bf81032f0 2024-05-27T07:55:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.814+0000 7f4bff11f640 1 -- 172.21.15.114:0/4088192116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf819e8c0 con 0x7f4bf81032f0 2024-05-27T07:55:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.814+0000 7f4bff11f640 1 -- 172.21.15.114:0/4088192116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf81a6240 con 0x7f4bf81032f0 2024-05-27T07:55:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.817+0000 7f4c05fd5740 1 Processor -- start 2024-05-27T07:55:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.817+0000 7f4c05fd5740 1 -- start start 2024-05-27T07:55:20.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.817+0000 7f4c05fd5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965491b0 0x55be968f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.817+0000 7f4c05fd5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965202f0 0x55be968f5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.817+0000 7f4c05fd5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968efab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.817+0000 7f4c05fd5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55be9685cc30 con 0x55be965284d0 2024-05-27T07:55:20.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.817+0000 7f4c05fd5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55be9684c7d0 con 0x55be965202f0 2024-05-27T07:55:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4c05fd5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55be96850810 con 0x55be965491b0 2024-05-27T07:55:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4bf5ffb640 1 -- 172.21.15.114:0/4088192116 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f4bec0046c0 con 0x7f4bf81032f0 2024-05-27T07:55:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4bfd6bc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965491b0 0x55be968f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4bfd6bc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965491b0 0x55be968f50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:41838/0 (socket says 172.21.15.114:41838) 2024-05-27T07:55:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4bf7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968efab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/640640823 learned_addr learned my addr 172.21.15.114:0/640640823 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4bfcebb640 1 --2- 172.21.15.114:0/640640823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965202f0 0x55be968f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/640640823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965202f0 msgr2=0x55be968f5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/640640823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965202f0 0x55be968f5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/640640823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 msgr2=0x55be968efab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/640640823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968efab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/640640823 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be96561210 con 0x55be965491b0 2024-05-27T07:55:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/640640823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965491b0 0x55be968f50e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0033040 tx=0x7f4bf002f3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7db950fa2c36279 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:20.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.818+0000 7f4be2ffd640 1 -- 172.21.15.114:0/640640823 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0031650 con 0x55be965491b0 2024-05-27T07:55:20.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.819+0000 7f4be2ffd640 1 -- 172.21.15.114:0/640640823 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bf0036040 con 0x55be965491b0 2024-05-27T07:55:20.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.819+0000 7f4be2ffd640 1 -- 172.21.15.114:0/640640823 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf00044e0 con 0x55be965491b0 2024-05-27T07:55:20.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.819+0000 7f4bf5ffb640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4be80b29d0 0x7f4be80b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.819+0000 7f4bfcebb640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4be80b29d0 0x7f4be80b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.819+0000 7f4bf5ffb640 1 -- 172.21.15.114:0/4088192116 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 9554+0+0 (secure 0 0 0) 0x7f4bec0e2ce0 con 0x7f4bf81032f0 2024-05-27T07:55:20.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.820+0000 7f4c05fd5740 1 -- 172.21.15.114:0/640640823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965491b0 msgr2=0x55be968f50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:20.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.820+0000 7f4bfcebb640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4be80b29d0 0x7f4be80b4ee0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0e00d0 tx=0x7f4bec0e6560 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:20.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.820+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/640640823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965491b0 0x55be968f50e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0033040 tx=0x7f4bf002f3c0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.820+0000 7f4c05fd5740 1 -- 172.21.15.114:0/640640823 shutdown_connections 2024-05-27T07:55:20.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.820+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/640640823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968efab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.820+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/640640823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965202f0 0x55be968f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.820+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/640640823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965491b0 0x55be968f50e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.820+0000 7f4c05fd5740 1 -- 172.21.15.114:0/640640823 >> 172.21.15.114:0/640640823 conn(0x55be9653d300 msgr2=0x55be968e6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:20.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.820+0000 7f4c05fd5740 1 -- 172.21.15.114:0/640640823 shutdown_connections 2024-05-27T07:55:20.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.820+0000 7f4c05fd5740 1 -- 172.21.15.114:0/640640823 wait complete. 2024-05-27T07:55:20.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.820+0000 7f4c05fd5740 1 -- start start 2024-05-27T07:55:20.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.820+0000 7f4c05fd5740 1 Processor -- start 2024-05-27T07:55:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4c05fd5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965202f0 0x55be968fec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4c05fd5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968ff170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4c05fd5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965491b0 0x55be968f9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4c05fd5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55be96532790 con 0x55be965284d0 2024-05-27T07:55:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4c05fd5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55be96808890 con 0x55be965491b0 2024-05-27T07:55:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4c05fd5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55be968f0320 con 0x55be965202f0 2024-05-27T07:55:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4c05fd5740 10 client.0 _ll_register_callbacks cb 0x55be96560a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4bf7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968ff170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4bfd6bc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965202f0 0x55be968fec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4bfd6bc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965202f0 0x55be968fec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:41840/0 (socket says 172.21.15.114:41840) 2024-05-27T07:55:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/1495034412 learned_addr learned my addr 172.21.15.114:0/1495034412 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/1495034412 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965491b0 0x55be968f9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/1495034412 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965491b0 msgr2=0x55be968f9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/1495034412 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965491b0 0x55be968f9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/1495034412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 msgr2=0x55be968ff170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/1495034412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968ff170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/1495034412 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be9662d620 con 0x55be965202f0 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4bf7fff640 1 --2- 172.21.15.114:0/1495034412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.821+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/1495034412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965202f0 0x55be968fec30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0033010 tx=0x7f4bf0004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0f84afed62b8e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4bd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4bd7fff640 1 -- 172.21.15.114:0/1495034412 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0059060 con 0x55be965202f0 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4bd7fff640 1 -- 172.21.15.114:0/1495034412 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 9554+0+0 (secure 0 0 0) 0x7f4bf0031660 con 0x55be965202f0 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1495034412 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bf0036040 con 0x55be965202f0 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4bd7fff640 1 -- 172.21.15.114:0/1495034412 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f4bf0060050 con 0x55be965202f0 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4c05fd5740 10 client.28606 fetch_fsmap learned FSMap version 85 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1495034412 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=4+}) -- 0x55be969079d0 con 0x55be965202f0 2024-05-27T07:55:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4bd7fff640 1 -- 172.21.15.114:0/1495034412 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bf0036040 con 0x55be965202f0 2024-05-27T07:55:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4bd7fff640 1 -- 172.21.15.114:0/1495034412 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 85) ==== 4700+0+0 (secure 0 0 0) 0x7f4bf004e030 con 0x55be965202f0 2024-05-27T07:55:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4c05fd5740 10 client.28606 fetch_fsmap finished waiting for FSMap version 85 2024-05-27T07:55:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4c05fd5740 10 client.28606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4c05fd5740 20 client.28606 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/1495034412 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be96507d80 0x55be9690ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4c05fd5740 4 client.28606 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:55:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1495034412 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be9685eb80 con 0x55be96507d80 2024-05-27T07:55:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.822+0000 7f4bf7fff640 1 --2- 172.21.15.114:0/1495034412 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be96507d80 0x55be9690ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.823+0000 7f4bd7fff640 10 client.28606 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.823+0000 7f4bf7fff640 1 --2- 172.21.15.114:0/1495034412 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be96507d80 0x55be9690ada0 crc :-1 s=READY pgs=434 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4400b06c02b3fa4 server_cookie=bb964cb922b7e7bb in_seq=0 out_seq=0 2024-05-27T07:55:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4bd7fff640 1 -- 172.21.15.114:0/1495034412 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55be9685eb80 con 0x55be96507d80 2024-05-27T07:55:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4bd7fff640 10 client.28606 handle_command_reply: tid=0 2024-05-27T07:55:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4c05fd5740 1 client.28606 shutdown 2024-05-27T07:55:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1495034412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965202f0 msgr2=0x55be968fec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/1495034412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965202f0 0x55be968fec30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0033010 tx=0x7f4bf0004040 comp rx=0 tx=0).stop 2024-05-27T07:55:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1495034412 shutdown_connections 2024-05-27T07:55:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/1495034412 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be96507d80 0x55be9690ada0 crc :-1 s=READY pgs=434 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/1495034412 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965491b0 0x55be968f9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/1495034412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4bd7fff640 0 client.28606 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/1495034412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965202f0 0x55be968fec30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1495034412 >> 172.21.15.114:0/1495034412 conn(0x55be9653d300 msgr2=0x55be968e5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:20.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1495034412 shutdown_connections 2024-05-27T07:55:20.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1495034412 wait complete. 2024-05-27T07:55:20.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.830+0000 7f4c05fd5740 20 client.28606 trim_cache size 0 max 16384 2024-05-27T07:55:20.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.850+0000 7f4c05fd5740 1 Processor -- start 2024-05-27T07:55:20.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.850+0000 7f4c05fd5740 1 -- start start 2024-05-27T07:55:20.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4c05fd5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be96507d80 0x55be968f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4c05fd5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965202f0 0x55be968f8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4c05fd5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968e6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4c05fd5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf81af550 con 0x55be965284d0 2024-05-27T07:55:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4c05fd5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf81afc20 con 0x55be965202f0 2024-05-27T07:55:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4c05fd5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf81ae470 con 0x55be96507d80 2024-05-27T07:55:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4bfd6bc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968e6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4bfd6bc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be96507d80 0x55be968f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4bf7fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965202f0 0x55be968f8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4bfd6bc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be96507d80 0x55be968f6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:41846/0 (socket says 172.21.15.114:41846) 2024-05-27T07:55:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/3276755648 learned_addr learned my addr 172.21.15.114:0/3276755648 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/3276755648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be96507d80 msgr2=0x55be968f6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/3276755648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be96507d80 0x55be968f6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/3276755648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965202f0 msgr2=0x55be968f8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/3276755648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965202f0 0x55be968f8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.851+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/3276755648 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be968e7c00 con 0x55be965284d0 2024-05-27T07:55:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.852+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/3276755648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968e6160 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x55be968e1880 tx=0x7f4bf0005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826f8c1cc8f3672 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.852+0000 7f4be27fc640 1 -- 172.21.15.114:0/3276755648 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0059040 con 0x55be965284d0 2024-05-27T07:55:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.852+0000 7f4be27fc640 1 -- 172.21.15.114:0/3276755648 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bf0036070 con 0x55be965284d0 2024-05-27T07:55:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.852+0000 7f4be27fc640 1 -- 172.21.15.114:0/3276755648 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0031750 con 0x55be965284d0 2024-05-27T07:55:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.853+0000 7f4c05fd5740 1 -- 172.21.15.114:0/3276755648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 msgr2=0x55be968e6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.853+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/3276755648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968e6160 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x55be968e1880 tx=0x7f4bf0005d80 comp rx=0 tx=0).stop 2024-05-27T07:55:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.853+0000 7f4c05fd5740 1 -- 172.21.15.114:0/3276755648 shutdown_connections 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.853+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/3276755648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be965284d0 0x55be968e6160 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.853+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/3276755648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965202f0 0x55be968f8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.853+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/3276755648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be96507d80 0x55be968f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.853+0000 7f4c05fd5740 1 -- 172.21.15.114:0/3276755648 >> 172.21.15.114:0/3276755648 conn(0x55be9653d300 msgr2=0x55be968f88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.853+0000 7f4c05fd5740 1 -- 172.21.15.114:0/3276755648 shutdown_connections 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.853+0000 7f4c05fd5740 1 -- 172.21.15.114:0/3276755648 wait complete. 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.853+0000 7f4c05fd5740 1 -- start start 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.853+0000 7f4c05fd5740 1 Processor -- start 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4c05fd5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be96507d80 0x55be9690fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4c05fd5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965202f0 0x55be969103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4c05fd5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965284d0 0x55be96971410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4c05fd5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf81ad6d0 con 0x55be96507d80 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4c05fd5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf81ad060 con 0x55be965284d0 2024-05-27T07:55:20.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4c05fd5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf81adda0 con 0x55be965202f0 2024-05-27T07:55:20.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4bfd6bc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be96507d80 0x55be9690fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4bfd6bc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be96507d80 0x55be9690fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51996/0 (socket says 172.21.15.114:51996) 2024-05-27T07:55:20.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/1348196578 learned_addr learned my addr 172.21.15.114:0/1348196578 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:20.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4c05fd5740 10 client.0 _ll_register_callbacks cb 0x7f4be4004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:20.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4bf7fff640 1 --2- 172.21.15.114:0/1348196578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965202f0 0x55be969103d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/1348196578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965284d0 0x55be96971410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/1348196578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965202f0 msgr2=0x55be969103d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:20.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/1348196578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965202f0 0x55be969103d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/1348196578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965284d0 msgr2=0x55be96971410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:20.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/1348196578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965284d0 0x55be96971410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4bfd6bc640 1 -- 172.21.15.114:0/1348196578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be96971950 con 0x55be96507d80 2024-05-27T07:55:20.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4bfd6bc640 1 --2- 172.21.15.114:0/1348196578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be96507d80 0x55be9690fe90 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0009620 tx=0x7f4bf002b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9c81b65ba15509 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.854+0000 7f4bd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:55:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4bd7fff640 1 -- 172.21.15.114:0/1348196578 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0059070 con 0x55be96507d80 2024-05-27T07:55:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1348196578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bf0036070 con 0x55be96507d80 2024-05-27T07:55:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4bd7fff640 1 -- 172.21.15.114:0/1348196578 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 9554+0+0 (secure 0 0 0) 0x7f4bf002f3c0 con 0x55be96507d80 2024-05-27T07:55:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4bd7fff640 1 -- 172.21.15.114:0/1348196578 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f4bf00047f0 con 0x55be96507d80 2024-05-27T07:55:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4c05fd5740 10 client.19122 fetch_fsmap learned FSMap version 85 2024-05-27T07:55:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1348196578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=4+}) -- 0x55be96972b40 con 0x55be96507d80 2024-05-27T07:55:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4bd7fff640 1 -- 172.21.15.114:0/1348196578 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bf0036070 con 0x55be96507d80 2024-05-27T07:55:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4bd7fff640 1 -- 172.21.15.114:0/1348196578 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 85) ==== 4700+0+0 (secure 0 0 0) 0x7f4bf0063070 con 0x55be96507d80 2024-05-27T07:55:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4c05fd5740 10 client.19122 fetch_fsmap finished waiting for FSMap version 85 2024-05-27T07:55:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4c05fd5740 10 client.19122 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4c05fd5740 20 client.19122 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/1348196578 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be965491b0 0x55be96975220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4c05fd5740 4 client.19122 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:55:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.855+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1348196578 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55be96858360 con 0x55be965491b0 2024-05-27T07:55:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.856+0000 7f4bf7fff640 1 --2- 172.21.15.114:0/1348196578 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be965491b0 0x55be96975220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.856+0000 7f4bd7fff640 10 client.19122 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:20.856+0000 7f4bf7fff640 1 --2- 172.21.15.114:0/1348196578 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be965491b0 0x55be96975220 crc :-1 s=READY pgs=435 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3267e735bd81ac8 server_cookie=5d64a3f67effef26 in_seq=0 out_seq=0 2024-05-27T07:55:20.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:20 smithi163 ceph-mon[36451]: pgmap v529: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 291 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 511 op/s 2024-05-27T07:55:20.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:20 smithi186 ceph-mon[36647]: pgmap v529: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 291 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 511 op/s 2024-05-27T07:55:20.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.976+0000 7f6d5359e700 1 -- 172.21.15.114:0/1510031153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d34005920 con 0x7f6d5410a7b0 2024-05-27T07:55:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:55:20.981 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":85,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:55:14.728570+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_28430":{"gid":28430,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/3235541320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3235541320},{"type":"v1","addr":"172.21.15.163:6833","nonce":3235541320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:55:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.977+0000 7f6d437fe700 1 -- 172.21.15.114:0/1510031153 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+6865 (secure 0 0 0) 0x7f6d440aaa10 con 0x7f6d5410a7b0 2024-05-27T07:55:20.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.985+0000 7f6d5359e700 1 -- 172.21.15.114:0/1510031153 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6d3c0b2c00 msgr2=0x7f6d3c0b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:20.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.985+0000 7f6d5359e700 1 --2- 172.21.15.114:0/1510031153 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6d3c0b2c00 0x7f6d3c0b5110 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6d5411fcc0 tx=0x7f6d48006c60 comp rx=0 tx=0).stop 2024-05-27T07:55:20.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.985+0000 7f6d5359e700 1 -- 172.21.15.114:0/1510031153 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5410a7b0 msgr2=0x7f6d5411d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:20.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.985+0000 7f6d5359e700 1 --2- 172.21.15.114:0/1510031153 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5410a7b0 0x7f6d5411d360 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6d440057f0 tx=0x7f6d4400b610 comp rx=0 tx=0).stop 2024-05-27T07:55:20.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.986+0000 7f6d5359e700 1 -- 172.21.15.114:0/1510031153 shutdown_connections 2024-05-27T07:55:20.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.986+0000 7f6d5359e700 1 --2- 172.21.15.114:0/1510031153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d5411d920 0x7f6d54117b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.986+0000 7f6d5359e700 1 --2- 172.21.15.114:0/1510031153 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6d3c0b2c00 0x7f6d3c0b5110 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.986+0000 7f6d5359e700 1 --2- 172.21.15.114:0/1510031153 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5410a7b0 0x7f6d5411d360 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.986+0000 7f6d5359e700 1 --2- 172.21.15.114:0/1510031153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d5410b100 0x7f6d54117480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:20.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.986+0000 7f6d5359e700 1 -- 172.21.15.114:0/1510031153 >> 172.21.15.114:0/1510031153 conn(0x7f6d5406f280 msgr2=0x7f6d5411df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:20.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.986+0000 7f6d5359e700 1 -- 172.21.15.114:0/1510031153 shutdown_connections 2024-05-27T07:55:20.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:20.986+0000 7f6d5359e700 1 -- 172.21.15.114:0/1510031153 wait complete. 2024-05-27T07:55:20.992 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 85 2024-05-27T07:55:21.597 INFO:tasks.workunit.client.0.smithi114.stdout: 10 21611 14.39 MB/sec warmup 93 sec latency 341.682 ms 2024-05-27T07:55:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.604+0000 7f4bd7fff640 1 -- 172.21.15.114:0/1348196578 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55be96858360 con 0x55be965491b0 2024-05-27T07:55:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.604+0000 7f4bd7fff640 10 client.19122 handle_command_reply: tid=0 2024-05-27T07:55:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.604+0000 7f4c05fd5740 1 client.19122 shutdown 2024-05-27T07:55:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.604+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1348196578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be96507d80 msgr2=0x55be9690fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.604+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/1348196578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be96507d80 0x55be9690fe90 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0009620 tx=0x7f4bf002b040 comp rx=0 tx=0).stop 2024-05-27T07:55:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.604+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1348196578 shutdown_connections 2024-05-27T07:55:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.604+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/1348196578 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55be965491b0 0x55be96975220 crc :-1 s=READY pgs=435 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.605+0000 7f4bd7fff640 0 client.19122 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.605+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/1348196578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55be965284d0 0x55be96971410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.605+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/1348196578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be965202f0 0x55be969103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.605+0000 7f4c05fd5740 1 --2- 172.21.15.114:0/1348196578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55be96507d80 0x55be9690fe90 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.605+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1348196578 >> 172.21.15.114:0/1348196578 conn(0x55be9653d300 msgr2=0x55be968f3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.605+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1348196578 shutdown_connections 2024-05-27T07:55:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.605+0000 7f4c05fd5740 1 -- 172.21.15.114:0/1348196578 wait complete. 2024-05-27T07:55:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.605+0000 7f4c05fd5740 20 client.19122 trim_cache size 0 max 16384 2024-05-27T07:55:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:55:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:55:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:55:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.606+0000 7f4bff11f640 1 -- 172.21.15.114:0/4088192116 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4be80b29d0 msgr2=0x7f4be80b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.606+0000 7f4bff11f640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4be80b29d0 0x7f4be80b4ee0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0e00d0 tx=0x7f4bec0e6560 comp rx=0 tx=0).stop 2024-05-27T07:55:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.606+0000 7f4bff11f640 1 -- 172.21.15.114:0/4088192116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bf81032f0 msgr2=0x7f4bf8199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.606+0000 7f4bff11f640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bf81032f0 0x7f4bf8199770 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00e9f0 tx=0x7f4bec00eeb0 comp rx=0 tx=0).stop 2024-05-27T07:55:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.606+0000 7f4bff11f640 1 -- 172.21.15.114:0/4088192116 shutdown_connections 2024-05-27T07:55:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.606+0000 7f4bff11f640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4be80b29d0 0x7f4be80b4ee0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.606+0000 7f4bff11f640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bf8109040 0x7f4bf819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.606+0000 7f4bff11f640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bf8108ba0 0x7f4bf8199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.606+0000 7f4bff11f640 1 --2- 172.21.15.114:0/4088192116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bf81032f0 0x7f4bf8199770 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.606+0000 7f4bff11f640 1 -- 172.21.15.114:0/4088192116 >> 172.21.15.114:0/4088192116 conn(0x7f4bf8078270 msgr2=0x7f4bf80ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.606+0000 7f4bff11f640 1 -- 172.21.15.114:0/4088192116 shutdown_connections 2024-05-27T07:55:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:21.606+0000 7f4bff11f640 1 -- 172.21.15.114:0/4088192116 wait complete. 2024-05-27T07:55:21.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:55:21.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:21 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1510031153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:55:21.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:21 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1510031153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:55:22.040 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:55:22.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:21 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1510031153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:55:22.597 INFO:tasks.workunit.client.0.smithi114.stdout: 10 21897 14.41 MB/sec warmup 94 sec latency 213.788 ms 2024-05-27T07:55:22.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:22 smithi114 ceph-mon[32038]: pgmap v530: 129 pgs: 129 active+clean; 293 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 331 op/s 2024-05-27T07:55:22.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:22 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:55:22.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:22 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:55:22.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:22 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:55:22.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:22 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:55:22.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:22 smithi163 ceph-mon[36451]: pgmap v530: 129 pgs: 129 active+clean; 293 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 331 op/s 2024-05-27T07:55:22.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:22 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:55:22.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:22 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:55:22.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:22 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:55:22.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:22 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:55:22.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:22 smithi186 ceph-mon[36647]: pgmap v530: 129 pgs: 129 active+clean; 293 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 331 op/s 2024-05-27T07:55:22.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:22 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:55:22.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:22 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:55:22.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:22 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:55:22.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:22 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:55:23.597 INFO:tasks.workunit.client.0.smithi114.stdout: 10 22078 14.34 MB/sec warmup 95 sec latency 422.652 ms 2024-05-27T07:55:23.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:23 smithi114 ceph-mon[32038]: pgmap v531: 129 pgs: 129 active+clean; 288 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.5 MiB/s wr, 368 op/s 2024-05-27T07:55:23.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:23 smithi114 ceph-mon[32038]: pgmap v532: 129 pgs: 129 active+clean; 288 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.2 MiB/s wr, 422 op/s 2024-05-27T07:55:23.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:23 smithi163 ceph-mon[36451]: pgmap v531: 129 pgs: 129 active+clean; 288 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.5 MiB/s wr, 368 op/s 2024-05-27T07:55:23.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:23 smithi163 ceph-mon[36451]: pgmap v532: 129 pgs: 129 active+clean; 288 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.2 MiB/s wr, 422 op/s 2024-05-27T07:55:23.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:23 smithi186 ceph-mon[36647]: pgmap v531: 129 pgs: 129 active+clean; 288 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.5 MiB/s wr, 368 op/s 2024-05-27T07:55:23.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:23 smithi186 ceph-mon[36647]: pgmap v532: 129 pgs: 129 active+clean; 288 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.2 MiB/s wr, 422 op/s 2024-05-27T07:55:24.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 22300 14.37 MB/sec warmup 96 sec latency 141.680 ms 2024-05-27T07:55:24.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.615+0000 7ff23302d700 1 -- 172.21.15.114:0/2273228354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff22c108740 msgr2=0x7ff22c1100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:24.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.615+0000 7ff23302d700 1 --2- 172.21.15.114:0/2273228354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff22c108740 0x7ff22c1100a0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7ff2140099c0 tx=0x7ff214009cd0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.615+0000 7ff23302d700 1 -- 172.21.15.114:0/2273228354 shutdown_connections 2024-05-27T07:55:24.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.615+0000 7ff23302d700 1 --2- 172.21.15.114:0/2273228354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff22c108740 0x7ff22c1100a0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.615+0000 7ff23302d700 1 --2- 172.21.15.114:0/2273228354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff22c103bb0 0x7ff22c103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.615+0000 7ff23302d700 1 --2- 172.21.15.114:0/2273228354 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff22c107df0 0x7ff22c108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.615+0000 7ff23302d700 1 -- 172.21.15.114:0/2273228354 >> 172.21.15.114:0/2273228354 conn(0x7ff22c0fe6e0 msgr2=0x7ff22c100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:24.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.615+0000 7ff23302d700 1 -- 172.21.15.114:0/2273228354 shutdown_connections 2024-05-27T07:55:24.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.615+0000 7ff23302d700 1 -- 172.21.15.114:0/2273228354 wait complete. 2024-05-27T07:55:24.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.616+0000 7ff23302d700 1 Processor -- start 2024-05-27T07:55:24.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.616+0000 7ff23302d700 1 -- start start 2024-05-27T07:55:24.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff23302d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff22c103bb0 0x7ff22c0febc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff23302d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff22c107df0 0x7ff22c0ff100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff23302d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff22c108740 0x7ff22c0ff640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff23302d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c113370 con 0x7ff22c107df0 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff23302d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1131f0 con 0x7ff22c103bb0 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff23302d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1134f0 con 0x7ff22c108740 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff22bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff22c107df0 0x7ff22c0ff100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff230dc9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff22c103bb0 0x7ff22c0febc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff230dc9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff22c103bb0 0x7ff22c0febc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:54108/0 (socket says 172.21.15.114:54108) 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff22bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff22c107df0 0x7ff22c0ff100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51978/0 (socket says 172.21.15.114:51978) 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff230dc9700 1 -- 172.21.15.114:0/624921676 learned_addr learned my addr 172.21.15.114:0/624921676 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff2315ca700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff22c108740 0x7ff22c0ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff2315ca700 1 -- 172.21.15.114:0/624921676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff22c103bb0 msgr2=0x7ff22c0febc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff2315ca700 1 --2- 172.21.15.114:0/624921676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff22c103bb0 0x7ff22c0febc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff2315ca700 1 -- 172.21.15.114:0/624921676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff22c107df0 msgr2=0x7ff22c0ff100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff2315ca700 1 --2- 172.21.15.114:0/624921676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff22c107df0 0x7ff22c0ff100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff2315ca700 1 -- 172.21.15.114:0/624921676 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff22c1a9080 con 0x7ff22c108740 2024-05-27T07:55:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.617+0000 7ff22bfff700 1 --2- 172.21.15.114:0/624921676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff22c107df0 0x7ff22c0ff100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:55:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.618+0000 7ff2315ca700 1 --2- 172.21.15.114:0/624921676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff22c108740 0x7ff22c0ff640 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff2140098e0 tx=0x7ff2140045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.618+0000 7ff229ffb700 1 -- 172.21.15.114:0/624921676 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff214009e60 con 0x7ff22c108740 2024-05-27T07:55:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.618+0000 7ff229ffb700 1 -- 172.21.15.114:0/624921676 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff214004810 con 0x7ff22c108740 2024-05-27T07:55:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.618+0000 7ff23302d700 1 -- 172.21.15.114:0/624921676 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff22c1a9310 con 0x7ff22c108740 2024-05-27T07:55:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.618+0000 7ff229ffb700 1 -- 172.21.15.114:0/624921676 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff214005250 con 0x7ff22c108740 2024-05-27T07:55:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.618+0000 7ff23302d700 1 -- 172.21.15.114:0/624921676 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff22c1a9700 con 0x7ff22c108740 2024-05-27T07:55:24.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.622+0000 7ff229ffb700 1 -- 172.21.15.114:0/624921676 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ff2140053f0 con 0x7ff22c108740 2024-05-27T07:55:24.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.623+0000 7ff241a42b80 1 Processor -- start 2024-05-27T07:55:24.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.623+0000 7ff241a42b80 1 -- start start 2024-05-27T07:55:24.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.623+0000 7ff241a42b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ed07e40 0x555f3ee8b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.623+0000 7ff241a42b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ee8bc50 0x555f3ee85a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.623+0000 7ff229ffb700 1 --2- 172.21.15.114:0/624921676 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2180b29f0 0x7ff2180b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.623+0000 7ff241a42b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee86390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.623+0000 7ff241a42b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555f3edb4c90 con 0x555f3ee85f80 2024-05-27T07:55:24.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.623+0000 7ff241a42b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555f3eda4830 con 0x555f3ee8bc50 2024-05-27T07:55:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.623+0000 7ff241a42b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555f3eda8870 con 0x555f3ed07e40 2024-05-27T07:55:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.623+0000 7ff2315ca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee86390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.624+0000 7ff230dc9700 1 --2- 172.21.15.114:0/624921676 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2180b29f0 0x7ff2180b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.624+0000 7ff2315ca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee86390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51984/0 (socket says 172.21.15.114:51984) 2024-05-27T07:55:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.624+0000 7ff2315ca700 1 -- 172.21.15.114:0/1004861463 learned_addr learned my addr 172.21.15.114:0/1004861463 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.624+0000 7ff229ffb700 1 -- 172.21.15.114:0/624921676 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7ff2140e3410 con 0x7ff22c108740 2024-05-27T07:55:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.624+0000 7ff22bfff700 1 --2- 172.21.15.114:0/1004861463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ed07e40 0x555f3ee8b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.624+0000 7ff22bfff700 1 --2- 172.21.15.114:0/1004861463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ee8bc50 0x555f3ee85a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.624+0000 7ff2315ca700 1 -- 172.21.15.114:0/1004861463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ed07e40 msgr2=0x555f3ee8b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.624+0000 7ff2315ca700 1 --2- 172.21.15.114:0/1004861463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ed07e40 0x555f3ee8b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.624+0000 7ff2315ca700 1 -- 172.21.15.114:0/1004861463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ee8bc50 msgr2=0x555f3ee85a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.624+0000 7ff2315ca700 1 --2- 172.21.15.114:0/1004861463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ee8bc50 0x555f3ee85a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.624+0000 7ff2315ca700 1 -- 172.21.15.114:0/1004861463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f3ecf48d0 con 0x555f3ee85f80 2024-05-27T07:55:24.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.624+0000 7ff22bfff700 1 --2- 172.21.15.114:0/1004861463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ed07e40 0x555f3ee8b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:55:24.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.624+0000 7ff2315ca700 1 --2- 172.21.15.114:0/1004861463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee86390 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7ff21400b580 tx=0x7ff21409ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b70f5f67cbad74 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:24.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.625+0000 7ff222ffd700 1 -- 172.21.15.114:0/1004861463 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21402f040 con 0x555f3ee85f80 2024-05-27T07:55:24.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.625+0000 7ff222ffd700 1 -- 172.21.15.114:0/1004861463 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2140aaa70 con 0x555f3ee85f80 2024-05-27T07:55:24.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.625+0000 7ff222ffd700 1 -- 172.21.15.114:0/1004861463 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff214014d60 con 0x555f3ee85f80 2024-05-27T07:55:24.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.626+0000 7ff241a42b80 1 -- 172.21.15.114:0/1004861463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 msgr2=0x555f3ee86390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:24.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.626+0000 7ff241a42b80 1 --2- 172.21.15.114:0/1004861463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee86390 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7ff21400b580 tx=0x7ff21409ebf0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.626+0000 7ff230dc9700 1 --2- 172.21.15.114:0/624921676 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2180b29f0 0x7ff2180b4f00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff224009d20 tx=0x7ff224009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:24.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.626+0000 7ff241a42b80 1 -- 172.21.15.114:0/1004861463 shutdown_connections 2024-05-27T07:55:24.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.626+0000 7ff241a42b80 1 --2- 172.21.15.114:0/1004861463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee86390 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.626+0000 7ff241a42b80 1 --2- 172.21.15.114:0/1004861463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ed07e40 0x555f3ee8b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.626+0000 7ff241a42b80 1 --2- 172.21.15.114:0/1004861463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ee8bc50 0x555f3ee85a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.626+0000 7ff241a42b80 1 -- 172.21.15.114:0/1004861463 >> 172.21.15.114:0/1004861463 conn(0x555f3ed62af0 msgr2=0x555f3ee80640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.626+0000 7ff241a42b80 1 -- 172.21.15.114:0/1004861463 shutdown_connections 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.626+0000 7ff241a42b80 1 -- 172.21.15.114:0/1004861463 wait complete. 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.626+0000 7ff241a42b80 1 -- start start 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.627+0000 7ff241a42b80 1 Processor -- start 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.627+0000 7ff241a42b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ed07e40 0x555f3ee8f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.627+0000 7ff241a42b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee8f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.627+0000 7ff241a42b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ee8bc50 0x555f3ee8fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.627+0000 7ff241a42b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555f3ecaafa0 con 0x555f3ee85f80 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.627+0000 7ff241a42b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555f3ed467c0 con 0x555f3ed07e40 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.627+0000 7ff241a42b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555f3ee92500 con 0x555f3ee8bc50 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.627+0000 7ff241a42b80 10 client.0 _ll_register_callbacks cb 0x555f3ed3f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.627+0000 7ff22bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee8f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.627+0000 7ff22bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee8f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51988/0 (socket says 172.21.15.114:51988) 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.627+0000 7ff22bfff700 1 -- 172.21.15.114:0/3626280131 learned_addr learned my addr 172.21.15.114:0/3626280131 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.627+0000 7ff2315ca700 1 --2- 172.21.15.114:0/3626280131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ee8bc50 0x555f3ee8fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff230dc9700 1 --2- 172.21.15.114:0/3626280131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ed07e40 0x555f3ee8f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff22bfff700 1 -- 172.21.15.114:0/3626280131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ee8bc50 msgr2=0x555f3ee8fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff22bfff700 1 --2- 172.21.15.114:0/3626280131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ee8bc50 0x555f3ee8fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff22bfff700 1 -- 172.21.15.114:0/3626280131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ed07e40 msgr2=0x555f3ee8f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff22bfff700 1 --2- 172.21.15.114:0/3626280131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ed07e40 0x555f3ee8f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff22bfff700 1 -- 172.21.15.114:0/3626280131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f3ed43690 con 0x555f3ee85f80 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff22bfff700 1 --2- 172.21.15.114:0/3626280131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee8f9a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00ca60 tx=0x7ff21c00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25205b20c9fca246 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff20bfff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff20bfff700 1 -- 172.21.15.114:0/3626280131 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c00ce90 con 0x555f3ee85f80 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff241a42b80 1 -- 172.21.15.114:0/3626280131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2140aaa70 con 0x555f3ee85f80 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff20bfff700 1 -- 172.21.15.114:0/3626280131 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7ff21c004bb0 con 0x555f3ee85f80 2024-05-27T07:55:24.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff20bfff700 1 -- 172.21.15.114:0/3626280131 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7ff21c02b050 con 0x555f3ee85f80 2024-05-27T07:55:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff241a42b80 10 client.19134 fetch_fsmap learned FSMap version 85 2024-05-27T07:55:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.628+0000 7ff241a42b80 1 -- 172.21.15.114:0/3626280131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=4+}) -- 0x555f3ee9cf80 con 0x555f3ee85f80 2024-05-27T07:55:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.629+0000 7ff20bfff700 1 -- 172.21.15.114:0/3626280131 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2140aaa70 con 0x555f3ee85f80 2024-05-27T07:55:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.629+0000 7ff20bfff700 1 -- 172.21.15.114:0/3626280131 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 85) ==== 4700+0+0 (secure 0 0 0) 0x7ff21c020ae0 con 0x555f3ee85f80 2024-05-27T07:55:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.629+0000 7ff241a42b80 10 client.19134 fetch_fsmap finished waiting for FSMap version 85 2024-05-27T07:55:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.629+0000 7ff241a42b80 10 client.19134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.629+0000 7ff241a42b80 20 client.19134 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.629+0000 7ff241a42b80 1 --2- 172.21.15.114:0/3626280131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555f3ee9f9c0 0x555f3eea1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.629+0000 7ff241a42b80 4 client.19134 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:55:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.629+0000 7ff241a42b80 1 -- 172.21.15.114:0/3626280131 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f3ee519f0 con 0x555f3ee9f9c0 2024-05-27T07:55:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.629+0000 7ff230dc9700 1 --2- 172.21.15.114:0/3626280131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555f3ee9f9c0 0x555f3eea1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.630+0000 7ff20bfff700 10 client.19134 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.630+0000 7ff230dc9700 1 --2- 172.21.15.114:0/3626280131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555f3ee9f9c0 0x555f3eea1e50 crc :-1 s=READY pgs=436 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27ba4b82d07e4745 server_cookie=b5aef29a0ab9b611 in_seq=0 out_seq=0 2024-05-27T07:55:24.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff20bfff700 1 -- 172.21.15.114:0/3626280131 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555f3ee519f0 con 0x555f3ee9f9c0 2024-05-27T07:55:24.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff20bfff700 10 client.19134 handle_command_reply: tid=0 2024-05-27T07:55:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff241a42b80 1 client.19134 shutdown 2024-05-27T07:55:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff241a42b80 1 -- 172.21.15.114:0/3626280131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 msgr2=0x555f3ee8f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff241a42b80 1 --2- 172.21.15.114:0/3626280131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee8f9a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00ca60 tx=0x7ff21c00cd70 comp rx=0 tx=0).stop 2024-05-27T07:55:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff241a42b80 1 -- 172.21.15.114:0/3626280131 shutdown_connections 2024-05-27T07:55:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff241a42b80 1 --2- 172.21.15.114:0/3626280131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ee8bc50 0x555f3ee8fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff241a42b80 1 --2- 172.21.15.114:0/3626280131 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555f3ee9f9c0 0x555f3eea1e50 crc :-1 s=READY pgs=436 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff241a42b80 1 --2- 172.21.15.114:0/3626280131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ed07e40 0x555f3ee8f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff241a42b80 1 --2- 172.21.15.114:0/3626280131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee8f9a0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff241a42b80 1 -- 172.21.15.114:0/3626280131 >> 172.21.15.114:0/3626280131 conn(0x555f3ed62af0 msgr2=0x555f3ee84a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff20bfff700 0 client.19134 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff241a42b80 1 -- 172.21.15.114:0/3626280131 shutdown_connections 2024-05-27T07:55:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff241a42b80 1 -- 172.21.15.114:0/3626280131 wait complete. 2024-05-27T07:55:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.635+0000 7ff241a42b80 20 client.19134 trim_cache size 0 max 16384 2024-05-27T07:55:24.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.650+0000 7ff241a42b80 1 Processor -- start 2024-05-27T07:55:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.650+0000 7ff241a42b80 1 -- start start 2024-05-27T07:55:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.650+0000 7ff241a42b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ed07e40 0x555f3ee85a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.650+0000 7ff241a42b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ee85f80 0x555f3ee8ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.650+0000 7ff241a42b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee8bc50 0x555f3eea7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff241a42b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1ae6d0 con 0x555f3ee8bc50 2024-05-27T07:55:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff241a42b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1aeda0 con 0x555f3ee85f80 2024-05-27T07:55:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff241a42b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1ad5d0 con 0x555f3ed07e40 2024-05-27T07:55:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff2315ca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee8bc50 0x555f3eea7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff2315ca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee8bc50 0x555f3eea7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51998/0 (socket says 172.21.15.114:51998) 2024-05-27T07:55:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff2315ca700 1 -- 172.21.15.114:0/997431100 learned_addr learned my addr 172.21.15.114:0/997431100 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff2315ca700 1 -- 172.21.15.114:0/997431100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ed07e40 msgr2=0x555f3ee85a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff2315ca700 1 --2- 172.21.15.114:0/997431100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ed07e40 0x555f3ee85a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff2315ca700 1 -- 172.21.15.114:0/997431100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ee85f80 msgr2=0x555f3ee8ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:24.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff2315ca700 1 --2- 172.21.15.114:0/997431100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ee85f80 0x555f3ee8ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff2315ca700 1 -- 172.21.15.114:0/997431100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f3ee78970 con 0x555f3ee8bc50 2024-05-27T07:55:24.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff2315ca700 1 --2- 172.21.15.114:0/997431100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee8bc50 0x555f3eea7650 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff2140e0800 tx=0x7ff2140a6360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d3d357c261725a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:24.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff2227fc700 1 -- 172.21.15.114:0/997431100 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2140aa370 con 0x555f3ee8bc50 2024-05-27T07:55:24.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff2227fc700 1 -- 172.21.15.114:0/997431100 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2140ae720 con 0x555f3ee8bc50 2024-05-27T07:55:24.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.651+0000 7ff2227fc700 1 -- 172.21.15.114:0/997431100 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2140aa5d0 con 0x555f3ee8bc50 2024-05-27T07:55:24.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.652+0000 7ff241a42b80 1 -- 172.21.15.114:0/997431100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee8bc50 msgr2=0x555f3eea7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:24.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.652+0000 7ff241a42b80 1 --2- 172.21.15.114:0/997431100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee8bc50 0x555f3eea7650 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff2140e0800 tx=0x7ff2140a6360 comp rx=0 tx=0).stop 2024-05-27T07:55:24.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.652+0000 7ff241a42b80 1 -- 172.21.15.114:0/997431100 shutdown_connections 2024-05-27T07:55:24.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.652+0000 7ff241a42b80 1 --2- 172.21.15.114:0/997431100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee8bc50 0x555f3eea7650 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.652+0000 7ff241a42b80 1 --2- 172.21.15.114:0/997431100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ed07e40 0x555f3ee85a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.652+0000 7ff241a42b80 1 --2- 172.21.15.114:0/997431100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ee85f80 0x555f3ee8ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.652+0000 7ff241a42b80 1 -- 172.21.15.114:0/997431100 >> 172.21.15.114:0/997431100 conn(0x555f3ed62af0 msgr2=0x555f3ee8a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:24.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.652+0000 7ff241a42b80 1 -- 172.21.15.114:0/997431100 shutdown_connections 2024-05-27T07:55:24.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.652+0000 7ff241a42b80 1 -- 172.21.15.114:0/997431100 wait complete. 2024-05-27T07:55:24.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.653+0000 7ff241a42b80 1 -- start start 2024-05-27T07:55:24.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.653+0000 7ff241a42b80 1 Processor -- start 2024-05-27T07:55:24.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.653+0000 7ff241a42b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ed07e40 0x555f3ee97750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.653+0000 7ff241a42b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee97c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.653+0000 7ff241a42b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ee8bc50 0x555f3ee84a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.653+0000 7ff241a42b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1ac830 con 0x555f3ee85f80 2024-05-27T07:55:24.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.653+0000 7ff241a42b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1ac1e0 con 0x555f3ed07e40 2024-05-27T07:55:24.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.653+0000 7ff241a42b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1acf00 con 0x555f3ee8bc50 2024-05-27T07:55:24.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.653+0000 7ff241a42b80 10 client.0 _ll_register_callbacks cb 0x555f3ed3f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:24.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.653+0000 7ff22bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee97c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.653+0000 7ff22bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee97c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52000/0 (socket says 172.21.15.114:52000) 2024-05-27T07:55:24.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.653+0000 7ff22bfff700 1 -- 172.21.15.114:0/748970516 learned_addr learned my addr 172.21.15.114:0/748970516 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:24.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff2315ca700 1 --2- 172.21.15.114:0/748970516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ee8bc50 0x555f3ee84a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff230dc9700 1 --2- 172.21.15.114:0/748970516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ed07e40 0x555f3ee97750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff22bfff700 1 -- 172.21.15.114:0/748970516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ee8bc50 msgr2=0x555f3ee84a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:24.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff22bfff700 1 --2- 172.21.15.114:0/748970516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ee8bc50 0x555f3ee84a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff22bfff700 1 -- 172.21.15.114:0/748970516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ed07e40 msgr2=0x555f3ee97750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff22bfff700 1 --2- 172.21.15.114:0/748970516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ed07e40 0x555f3ee97750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff22bfff700 1 -- 172.21.15.114:0/748970516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f3ee98340 con 0x555f3ee85f80 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff22bfff700 1 --2- 172.21.15.114:0/748970516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee97c90 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff21c020f00 tx=0x7ff21c01f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3cbca0da1510c33 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff20bfff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff20bfff700 1 -- 172.21.15.114:0/748970516 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c00e4a0 con 0x555f3ee85f80 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff20bfff700 1 -- 172.21.15.114:0/748970516 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7ff21c004dc0 con 0x555f3ee85f80 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff241a42b80 1 -- 172.21.15.114:0/748970516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2140ae720 con 0x555f3ee85f80 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff20bfff700 1 -- 172.21.15.114:0/748970516 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7ff21c00e8c0 con 0x555f3ee85f80 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff241a42b80 10 client.19140 fetch_fsmap learned FSMap version 85 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.654+0000 7ff241a42b80 1 -- 172.21.15.114:0/748970516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=4+}) -- 0x555f3ef09ab0 con 0x555f3ee85f80 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.655+0000 7ff20bfff700 1 -- 172.21.15.114:0/748970516 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2140ae720 con 0x555f3ee85f80 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.655+0000 7ff20bfff700 1 -- 172.21.15.114:0/748970516 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 85) ==== 4700+0+0 (secure 0 0 0) 0x7ff21c0306b0 con 0x555f3ee85f80 2024-05-27T07:55:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.655+0000 7ff241a42b80 10 client.19140 fetch_fsmap finished waiting for FSMap version 85 2024-05-27T07:55:24.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.655+0000 7ff241a42b80 10 client.19140 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:24.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.655+0000 7ff241a42b80 20 client.19140 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:24.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.655+0000 7ff241a42b80 1 --2- 172.21.15.114:0/748970516 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555f3ef0a240 0x555f3ef0c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:24.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.655+0000 7ff241a42b80 4 client.19140 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:55:24.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.655+0000 7ff241a42b80 1 -- 172.21.15.114:0/748970516 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555f3edb6be0 con 0x555f3ef0a240 2024-05-27T07:55:24.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.655+0000 7ff230dc9700 1 --2- 172.21.15.114:0/748970516 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555f3ef0a240 0x555f3ef0c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:24.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.655+0000 7ff20bfff700 10 client.19140 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:24.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.656+0000 7ff230dc9700 1 --2- 172.21.15.114:0/748970516 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555f3ef0a240 0x555f3ef0c660 crc :-1 s=READY pgs=437 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc184d13ee7ae59f server_cookie=db4fa88b28498c46 in_seq=0 out_seq=0 2024-05-27T07:55:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff20bfff700 1 -- 172.21.15.114:0/748970516 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555f3edb6be0 con 0x555f3ef0a240 2024-05-27T07:55:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff20bfff700 10 client.19140 handle_command_reply: tid=0 2024-05-27T07:55:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff241a42b80 1 client.19140 shutdown 2024-05-27T07:55:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff241a42b80 1 -- 172.21.15.114:0/748970516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 msgr2=0x555f3ee97c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff241a42b80 1 --2- 172.21.15.114:0/748970516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee97c90 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff21c020f00 tx=0x7ff21c01f770 comp rx=0 tx=0).stop 2024-05-27T07:55:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff241a42b80 1 -- 172.21.15.114:0/748970516 shutdown_connections 2024-05-27T07:55:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff241a42b80 1 --2- 172.21.15.114:0/748970516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555f3ee8bc50 0x555f3ee84a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff241a42b80 1 --2- 172.21.15.114:0/748970516 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555f3ef0a240 0x555f3ef0c660 crc :-1 s=READY pgs=437 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff20bfff700 0 client.19140 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff241a42b80 1 --2- 172.21.15.114:0/748970516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555f3ed07e40 0x555f3ee97750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff241a42b80 1 --2- 172.21.15.114:0/748970516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555f3ee85f80 0x555f3ee97c90 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff241a42b80 1 -- 172.21.15.114:0/748970516 >> 172.21.15.114:0/748970516 conn(0x555f3ed62af0 msgr2=0x555f3ee8dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff241a42b80 1 -- 172.21.15.114:0/748970516 shutdown_connections 2024-05-27T07:55:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.658+0000 7ff241a42b80 1 -- 172.21.15.114:0/748970516 wait complete. 2024-05-27T07:55:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.659+0000 7ff241a42b80 20 client.19140 trim_cache size 0 max 16384 2024-05-27T07:55:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:55:24.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.659+0000 7ff23302d700 1 -- 172.21.15.114:0/624921676 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2180b29f0 msgr2=0x7ff2180b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.659+0000 7ff23302d700 1 --2- 172.21.15.114:0/624921676 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2180b29f0 0x7ff2180b4f00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff224009d20 tx=0x7ff224009260 comp rx=0 tx=0).stop 2024-05-27T07:55:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.659+0000 7ff23302d700 1 -- 172.21.15.114:0/624921676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff22c108740 msgr2=0x7ff22c0ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.659+0000 7ff23302d700 1 --2- 172.21.15.114:0/624921676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff22c108740 0x7ff22c0ff640 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff2140098e0 tx=0x7ff2140045e0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.659+0000 7ff23302d700 1 -- 172.21.15.114:0/624921676 shutdown_connections 2024-05-27T07:55:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.659+0000 7ff23302d700 1 --2- 172.21.15.114:0/624921676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff22c108740 0x7ff22c0ff640 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.659+0000 7ff23302d700 1 --2- 172.21.15.114:0/624921676 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2180b29f0 0x7ff2180b4f00 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.659+0000 7ff23302d700 1 --2- 172.21.15.114:0/624921676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff22c103bb0 0x7ff22c0febc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.659+0000 7ff23302d700 1 --2- 172.21.15.114:0/624921676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff22c107df0 0x7ff22c0ff100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:24.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.659+0000 7ff23302d700 1 -- 172.21.15.114:0/624921676 >> 172.21.15.114:0/624921676 conn(0x7ff22c0fe6e0 msgr2=0x7ff22c102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:24.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.659+0000 7ff23302d700 1 -- 172.21.15.114:0/624921676 shutdown_connections 2024-05-27T07:55:24.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:24.659+0000 7ff23302d700 1 -- 172.21.15.114:0/624921676 wait complete. 2024-05-27T07:55:25.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 22750 14.39 MB/sec warmup 97 sec latency 70.561 ms 2024-05-27T07:55:25.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:25 smithi163 ceph-mon[36451]: osdmap e143: 12 total, 12 up, 12 in 2024-05-27T07:55:25.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:25 smithi163 ceph-mon[36451]: pgmap v534: 129 pgs: 129 active+clean; 291 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 330 op/s 2024-05-27T07:55:25.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:25 smithi186 ceph-mon[36647]: osdmap e143: 12 total, 12 up, 12 in 2024-05-27T07:55:25.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:25 smithi186 ceph-mon[36647]: pgmap v534: 129 pgs: 129 active+clean; 291 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 330 op/s 2024-05-27T07:55:25.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:25 smithi114 ceph-mon[32038]: osdmap e143: 12 total, 12 up, 12 in 2024-05-27T07:55:25.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:25 smithi114 ceph-mon[32038]: pgmap v534: 129 pgs: 129 active+clean; 291 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 330 op/s 2024-05-27T07:55:26.386 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:55:26.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:55:26.516 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:26 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:55:26.516 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:26 smithi163 systemd[1]: Stopped Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:55:26.516 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:26 smithi163 systemd[1]: Starting Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:55:26.516 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:26 smithi163 podman[74779]: 2024-05-27 07:55:26.366797524 +0000 UTC m=+0.017446764 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:55:26.597 INFO:tasks.workunit.client.0.smithi114.stdout: 10 23096 14.43 MB/sec warmup 98 sec latency 193.078 ms 2024-05-27T07:55:26.693 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:55:26.767 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:26 smithi163 podman[74779]: 2024-05-27 07:55:26.515733122 +0000 UTC m=+0.166382341 container create a3e28185b9fa2bdfee926f9f8b25a08ef23a1c91bc3a8624dc3cd06e61896cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:55:27.227 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:26 smithi163 podman[74779]: 2024-05-27 07:55:26.766799789 +0000 UTC m=+0.417449005 container init a3e28185b9fa2bdfee926f9f8b25a08ef23a1c91bc3a8624dc3cd06e61896cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:55:27.227 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:26 smithi163 podman[74779]: 2024-05-27 07:55:26.76956236 +0000 UTC m=+0.420211583 container start a3e28185b9fa2bdfee926f9f8b25a08ef23a1c91bc3a8624dc3cd06e61896cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:55:27.227 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:26 smithi163 ceph-mds[74798]: -- 172.21.15.163:0/2418985042 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bc8977da40 con 0x55bc897c5180 2024-05-27T07:55:27.227 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: starting mds.k at 2024-05-27T07:55:27.227 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:26 smithi163 bash[74779]: a3e28185b9fa2bdfee926f9f8b25a08ef23a1c91bc3a8624dc3cd06e61896cb9 2024-05-27T07:55:27.227 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:26 smithi163 systemd[1]: Started Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:55:27.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 23424 14.48 MB/sec warmup 99 sec latency 226.129 ms 2024-05-27T07:55:27.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:27 smithi163 ceph-mon[36451]: pgmap v535: 129 pgs: 129 active+clean; 294 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 629 op/s 2024-05-27T07:55:27.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:27 smithi163 ceph-mon[36451]: daemon mds.k restarted 2024-05-27T07:55:27.979 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2a00887700 time 2024-05-27T07:55:27.714532+0000 2024-05-27T07:55:27.979 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:27.979 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:27.979 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2a130ccba6] 2024-05-27T07:55:27.979 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2a130ccdd1] 2024-05-27T07:55:27.979 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bc86ddc23d] 2024-05-27T07:55:27.979 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bc86dc7971] 2024-05-27T07:55:27.979 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bc86dc5fa0] 2024-05-27T07:55:27.979 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bc86dc6651] 2024-05-27T07:55:27.980 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bc86d5b347] 2024-05-27T07:55:27.980 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bc86a5787e] 2024-05-27T07:55:27.980 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bc86a5ce94] 2024-05-27T07:55:27.980 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bc86a67969] 2024-05-27T07:55:27.980 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bc86e0048c] 2024-05-27T07:55:27.980 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bc86e099cb] 2024-05-27T07:55:27.980 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 13: (Thread::entry_wrapper()+0x43) [0x7f2a1309b72d] 2024-05-27T07:55:27.980 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2a1309b749] 2024-05-27T07:55:27.980 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2a11c3f1ca] 2024-05-27T07:55:27.980 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 16: clone() 2024-05-27T07:55:27.980 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.715+0000 7f2a00887700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2a00887700 time 2024-05-27T07:55:27.714532+0000 2024-05-27T07:55:27.980 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:27.980 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2a130ccba6] 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2a130ccdd1] 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bc86ddc23d] 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bc86dc7971] 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bc86dc5fa0] 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bc86dc6651] 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bc86d5b347] 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bc86a5787e] 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bc86a5ce94] 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bc86a67969] 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bc86e0048c] 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bc86e099cb] 2024-05-27T07:55:27.981 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 13: (Thread::entry_wrapper()+0x43) [0x7f2a1309b72d] 2024-05-27T07:55:27.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2a1309b749] 2024-05-27T07:55:27.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2a11c3f1ca] 2024-05-27T07:55:27.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 16: clone() 2024-05-27T07:55:27.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: *** Caught signal (Aborted) ** 2024-05-27T07:55:27.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: in thread 7f2a00887700 thread_name:md_log_replay 2024-05-27T07:55:27.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:27.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2a11c49d20] 2024-05-27T07:55:27.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2: gsignal() 2024-05-27T07:55:27.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 3: abort() 2024-05-27T07:55:27.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2a130ccced] 2024-05-27T07:55:27.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2a130ccdd1] 2024-05-27T07:55:27.983 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:27 smithi186 ceph-mon[36647]: pgmap v535: 129 pgs: 129 active+clean; 294 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 629 op/s 2024-05-27T07:55:27.983 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:27 smithi186 ceph-mon[36647]: daemon mds.k restarted 2024-05-27T07:55:27.984 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bc86ddc23d] 2024-05-27T07:55:27.984 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bc86dc7971] 2024-05-27T07:55:27.984 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bc86dc5fa0] 2024-05-27T07:55:27.984 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bc86dc6651] 2024-05-27T07:55:27.984 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bc86d5b347] 2024-05-27T07:55:27.984 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bc86a5787e] 2024-05-27T07:55:27.984 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bc86a5ce94] 2024-05-27T07:55:27.984 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bc86a67969] 2024-05-27T07:55:27.984 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bc86e0048c] 2024-05-27T07:55:27.984 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bc86e099cb] 2024-05-27T07:55:27.984 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 16: (Thread::entry_wrapper()+0x43) [0x7f2a1309b72d] 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2a1309b749] 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2a11c3f1ca] 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 19: clone() 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.718+0000 7f2a00887700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: in thread 7f2a00887700 thread_name:md_log_replay 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2a11c49d20] 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2: gsignal() 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 3: abort() 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2a130ccced] 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2a130ccdd1] 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bc86ddc23d] 2024-05-27T07:55:27.985 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bc86dc7971] 2024-05-27T07:55:27.986 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bc86dc5fa0] 2024-05-27T07:55:27.986 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bc86dc6651] 2024-05-27T07:55:27.986 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bc86d5b347] 2024-05-27T07:55:27.986 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bc86a5787e] 2024-05-27T07:55:27.986 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bc86a5ce94] 2024-05-27T07:55:27.986 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bc86a67969] 2024-05-27T07:55:27.986 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bc86e0048c] 2024-05-27T07:55:27.986 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bc86e099cb] 2024-05-27T07:55:27.986 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 16: (Thread::entry_wrapper()+0x43) [0x7f2a1309b72d] 2024-05-27T07:55:27.986 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2a1309b749] 2024-05-27T07:55:27.986 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2a11c3f1ca] 2024-05-27T07:55:27.986 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 19: clone() 2024-05-27T07:55:27.986 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: -1> 2024-05-27T07:55:27.715+0000 7f2a00887700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2a00887700 time 2024-05-27T07:55:27.714532+0000 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2a130ccba6] 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2a130ccdd1] 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bc86ddc23d] 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bc86dc7971] 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bc86dc5fa0] 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bc86dc6651] 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bc86d5b347] 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bc86a5787e] 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bc86a5ce94] 2024-05-27T07:55:27.987 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bc86a67969] 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bc86e0048c] 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bc86e099cb] 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 13: (Thread::entry_wrapper()+0x43) [0x7f2a1309b72d] 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2a1309b749] 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2a11c3f1ca] 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 16: clone() 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 0> 2024-05-27T07:55:27.718+0000 7f2a00887700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: in thread 7f2a00887700 thread_name:md_log_replay 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2a11c49d20] 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2: gsignal() 2024-05-27T07:55:27.988 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 3: abort() 2024-05-27T07:55:27.989 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2a130ccced] 2024-05-27T07:55:27.989 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2a130ccdd1] 2024-05-27T07:55:27.989 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bc86ddc23d] 2024-05-27T07:55:27.989 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bc86dc7971] 2024-05-27T07:55:27.989 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bc86dc5fa0] 2024-05-27T07:55:27.989 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bc86dc6651] 2024-05-27T07:55:27.989 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bc86d5b347] 2024-05-27T07:55:27.989 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bc86a5787e] 2024-05-27T07:55:27.989 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bc86a5ce94] 2024-05-27T07:55:27.989 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bc86a67969] 2024-05-27T07:55:27.989 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bc86e0048c] 2024-05-27T07:55:27.989 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bc86e099cb] 2024-05-27T07:55:27.989 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 16: (Thread::entry_wrapper()+0x43) [0x7f2a1309b72d] 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2a1309b749] 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2a11c3f1ca] 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 19: clone() 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: -1> 2024-05-27T07:55:27.715+0000 7f2a00887700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2a00887700 time 2024-05-27T07:55:27.714532+0000 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2a130ccba6] 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2a130ccdd1] 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bc86ddc23d] 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bc86dc7971] 2024-05-27T07:55:27.990 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bc86dc5fa0] 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bc86dc6651] 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bc86d5b347] 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bc86a5787e] 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bc86a5ce94] 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bc86a67969] 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bc86e0048c] 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bc86e099cb] 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 13: (Thread::entry_wrapper()+0x43) [0x7f2a1309b72d] 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2a1309b749] 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2a11c3f1ca] 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 16: clone() 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 0> 2024-05-27T07:55:27.718+0000 7f2a00887700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: in thread 7f2a00887700 thread_name:md_log_replay 2024-05-27T07:55:27.991 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.992 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:27.992 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2a11c49d20] 2024-05-27T07:55:27.992 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2: gsignal() 2024-05-27T07:55:27.992 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 3: abort() 2024-05-27T07:55:27.992 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2a130ccced] 2024-05-27T07:55:27.992 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2a130ccdd1] 2024-05-27T07:55:27.992 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bc86ddc23d] 2024-05-27T07:55:27.992 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bc86dc7971] 2024-05-27T07:55:27.992 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bc86dc5fa0] 2024-05-27T07:55:27.992 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bc86dc6651] 2024-05-27T07:55:27.992 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bc86d5b347] 2024-05-27T07:55:27.992 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bc86a5787e] 2024-05-27T07:55:27.992 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bc86a5ce94] 2024-05-27T07:55:27.993 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bc86a67969] 2024-05-27T07:55:27.993 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bc86e0048c] 2024-05-27T07:55:27.993 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bc86e099cb] 2024-05-27T07:55:27.993 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 16: (Thread::entry_wrapper()+0x43) [0x7f2a1309b72d] 2024-05-27T07:55:27.993 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2a1309b749] 2024-05-27T07:55:27.993 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2a11c3f1ca] 2024-05-27T07:55:27.993 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 19: clone() 2024-05-27T07:55:27.993 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:55:27.993 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74794]: 2024-05-27T07:55:27.993 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:27 smithi163 podman[74829]: 2024-05-27 07:55:27.774933477 +0000 UTC m=+0.018229010 container died a3e28185b9fa2bdfee926f9f8b25a08ef23a1c91bc3a8624dc3cd06e61896cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-27T07:55:28.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:27 smithi114 ceph-mon[32038]: pgmap v535: 129 pgs: 129 active+clean; 294 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 629 op/s 2024-05-27T07:55:28.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:27 smithi114 ceph-mon[32038]: daemon mds.k restarted 2024-05-27T07:55:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.586+0000 7f4a9b04b700 1 -- 172.21.15.114:0/610247583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a94108ef0 msgr2=0x7f4a94109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.586+0000 7f4a9b04b700 1 --2- 172.21.15.114:0/610247583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a94108ef0 0x7f4a94109300 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f4a800099c0 tx=0x7f4a80009cd0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.589+0000 7f4a9b04b700 1 -- 172.21.15.114:0/610247583 shutdown_connections 2024-05-27T07:55:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.589+0000 7f4a9b04b700 1 --2- 172.21.15.114:0/610247583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a94109840 0x7f4a941111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.589+0000 7f4a9b04b700 1 --2- 172.21.15.114:0/610247583 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a941045c0 0x7f4a941049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.589+0000 7f4a9b04b700 1 --2- 172.21.15.114:0/610247583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a94108ef0 0x7f4a94109300 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.589+0000 7f4a9b04b700 1 -- 172.21.15.114:0/610247583 >> 172.21.15.114:0/610247583 conn(0x7f4a940ffe90 msgr2=0x7f4a941022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:28.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.590+0000 7f4a9b04b700 1 -- 172.21.15.114:0/610247583 shutdown_connections 2024-05-27T07:55:28.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.594+0000 7f4a9b04b700 1 -- 172.21.15.114:0/610247583 wait complete. 2024-05-27T07:55:28.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.595+0000 7f4a9b04b700 1 Processor -- start 2024-05-27T07:55:28.597 INFO:tasks.workunit.client.0.smithi114.stdout: 10 23673 14.48 MB/sec warmup 100 sec latency 259.203 ms 2024-05-27T07:55:28.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.597+0000 7f4a9b04b700 1 -- start start 2024-05-27T07:55:28.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.597+0000 7f4a9b04b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a941045c0 0x7f4a940752f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.597+0000 7f4a9b04b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a94108ef0 0x7f4a9406e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.597+0000 7f4a9b04b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a94109840 0x7f4a9406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.597+0000 7f4a9b04b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9406dad0 con 0x7f4a941045c0 2024-05-27T07:55:28.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.597+0000 7f4a9b04b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9406d950 con 0x7f4a94109840 2024-05-27T07:55:28.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.597+0000 7f4a9b04b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9406dc50 con 0x7f4a94108ef0 2024-05-27T07:55:28.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.598+0000 7f4a98de7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a941045c0 0x7f4a940752f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.598+0000 7f4a98de7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a941045c0 0x7f4a940752f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52012/0 (socket says 172.21.15.114:52012) 2024-05-27T07:55:28.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.598+0000 7f4a98de7700 1 -- 172.21.15.114:0/3164579408 learned_addr learned my addr 172.21.15.114:0/3164579408 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:28.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.598+0000 7f4a93fff700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a94108ef0 0x7f4a9406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.598+0000 7f4a995e8700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a94109840 0x7f4a9406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.599+0000 7f4a93fff700 1 -- 172.21.15.114:0/3164579408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a94109840 msgr2=0x7f4a9406e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:28.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.599+0000 7f4a93fff700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a94109840 0x7f4a9406e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.599+0000 7f4a93fff700 1 -- 172.21.15.114:0/3164579408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a941045c0 msgr2=0x7f4a940752f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:28.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.599+0000 7f4a93fff700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a941045c0 0x7f4a940752f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.599+0000 7f4a93fff700 1 -- 172.21.15.114:0/3164579408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9406ef10 con 0x7f4a94108ef0 2024-05-27T07:55:28.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.601+0000 7f4a93fff700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a94108ef0 0x7f4a9406e290 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000b580 tx=0x7f4a8000b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:28.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.602+0000 7f4a91ffb700 1 -- 172.21.15.114:0/3164579408 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80026070 con 0x7f4a94108ef0 2024-05-27T07:55:28.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.602+0000 7f4a91ffb700 1 -- 172.21.15.114:0/3164579408 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a80003680 con 0x7f4a94108ef0 2024-05-27T07:55:28.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.602+0000 7f4a91ffb700 1 -- 172.21.15.114:0/3164579408 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80021670 con 0x7f4a94108ef0 2024-05-27T07:55:28.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.602+0000 7f4a9b04b700 1 -- 172.21.15.114:0/3164579408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a941ad9c0 con 0x7f4a94108ef0 2024-05-27T07:55:28.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.603+0000 7f4a9b04b700 1 -- 172.21.15.114:0/3164579408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a941ade50 con 0x7f4a94108ef0 2024-05-27T07:55:28.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.611+0000 7f4a91ffb700 1 -- 172.21.15.114:0/3164579408 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f4a8000f460 con 0x7f4a94108ef0 2024-05-27T07:55:28.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.613+0000 7f4a91ffb700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4a840b2c00 0x7f4a840b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.613+0000 7f4a91ffb700 1 -- 172.21.15.114:0/3164579408 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 9466+0+0 (secure 0 0 0) 0x7f4a800e3680 con 0x7f4a94108ef0 2024-05-27T07:55:28.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.614+0000 7f4a98de7700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4a840b2c00 0x7f4a840b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.617+0000 7f4aa9a60b80 1 Processor -- start 2024-05-27T07:55:28.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.617+0000 7f4a98de7700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4a840b2c00 0x7f4a840b5110 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4a88009ad0 tx=0x7f4a88009210 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:28.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.622+0000 7f4aa9a60b80 1 -- start start 2024-05-27T07:55:28.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.623+0000 7f4aa9a60b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55984680fec0 0x559846994bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.623+0000 7f4aa9a60b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5598469950f0 0x55984698ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.623+0000 7f4aa9a60b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x55984698f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.623+0000 7f4aa9a60b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5598468bcdf0 con 0x55984680fec0 2024-05-27T07:55:28.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.623+0000 7f4aa9a60b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5598468ac990 con 0x5598469950f0 2024-05-27T07:55:28.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.623+0000 7f4aa9a60b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5598468b09d0 con 0x55984698f1a0 2024-05-27T07:55:28.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.626+0000 7f4a93fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5598469950f0 0x55984698ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.627+0000 7f4a98de7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55984680fec0 0x559846994bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.627+0000 7f4a98de7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55984680fec0 0x559846994bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52026/0 (socket says 172.21.15.114:52026) 2024-05-27T07:55:28.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.627+0000 7f4a98de7700 1 -- 172.21.15.114:0/4049531339 learned_addr learned my addr 172.21.15.114:0/4049531339 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:28.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.627+0000 7f4a98de7700 1 -- 172.21.15.114:0/4049531339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 msgr2=0x55984698f5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:55:28.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.627+0000 7f4a98de7700 1 --2- 172.21.15.114:0/4049531339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x55984698f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.627+0000 7f4a98de7700 1 -- 172.21.15.114:0/4049531339 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5598469950f0 msgr2=0x55984698ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:28.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.627+0000 7f4a98de7700 1 --2- 172.21.15.114:0/4049531339 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5598469950f0 0x55984698ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.627+0000 7f4a98de7700 1 -- 172.21.15.114:0/4049531339 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559846856030 con 0x55984680fec0 2024-05-27T07:55:28.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.628+0000 7f4a98de7700 1 --2- 172.21.15.114:0/4049531339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55984680fec0 0x559846994bb0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800eae0 tx=0x7f4a88011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252af7d63b3ecf48 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:28.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.628+0000 7f4a7effd700 1 -- 172.21.15.114:0/4049531339 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8801b040 con 0x55984680fec0 2024-05-27T07:55:28.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.628+0000 7f4a7effd700 1 -- 172.21.15.114:0/4049531339 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a88024070 con 0x55984680fec0 2024-05-27T07:55:28.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.629+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/4049531339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55984680fec0 msgr2=0x559846994bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:28.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.629+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/4049531339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55984680fec0 0x559846994bb0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800eae0 tx=0x7f4a88011000 comp rx=0 tx=0).stop 2024-05-27T07:55:28.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.631+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/4049531339 shutdown_connections 2024-05-27T07:55:28.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.631+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/4049531339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x55984698f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.631+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/4049531339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55984680fec0 0x559846994bb0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.631+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/4049531339 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5598469950f0 0x55984698ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.631+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/4049531339 >> 172.21.15.114:0/4049531339 conn(0x5598468127f0 msgr2=0x559846989e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:28.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.637+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/4049531339 shutdown_connections 2024-05-27T07:55:28.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.638+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/4049531339 wait complete. 2024-05-27T07:55:28.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.639+0000 7f4aa9a60b80 1 -- start start 2024-05-27T07:55:28.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.639+0000 7f4aa9a60b80 1 Processor -- start 2024-05-27T07:55:28.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.640+0000 7f4aa9a60b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55984680fec0 0x559846998880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.640+0000 7f4aa9a60b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x559846998dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.640+0000 7f4aa9a60b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5598469950f0 0x559846999300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.640+0000 7f4aa9a60b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5598467b3020 con 0x5598469950f0 2024-05-27T07:55:28.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.640+0000 7f4aa9a60b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559846814e20 con 0x55984680fec0 2024-05-27T07:55:28.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.640+0000 7f4aa9a60b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55984699bc30 con 0x55984698f1a0 2024-05-27T07:55:28.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.640+0000 7f4aa9a60b80 10 client.0 _ll_register_callbacks cb 0x559846843630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:28.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.641+0000 7f4a995e8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5598469950f0 0x559846999300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.641+0000 7f4a995e8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5598469950f0 0x559846999300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52030/0 (socket says 172.21.15.114:52030) 2024-05-27T07:55:28.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.641+0000 7f4a995e8700 1 -- 172.21.15.114:0/347814117 learned_addr learned my addr 172.21.15.114:0/347814117 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:28.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.642+0000 7f4a98de7700 1 --2- 172.21.15.114:0/347814117 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55984680fec0 0x559846998880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.643+0000 7f4a93fff700 1 --2- 172.21.15.114:0/347814117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x559846998dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.644+0000 7f4a93fff700 1 -- 172.21.15.114:0/347814117 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55984680fec0 msgr2=0x559846998880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:28.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.644+0000 7f4a93fff700 1 --2- 172.21.15.114:0/347814117 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55984680fec0 0x559846998880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.644+0000 7f4a93fff700 1 -- 172.21.15.114:0/347814117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5598469950f0 msgr2=0x559846999300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:28.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.644+0000 7f4a93fff700 1 --2- 172.21.15.114:0/347814117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5598469950f0 0x559846999300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.644+0000 7f4a93fff700 1 -- 172.21.15.114:0/347814117 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598467fc950 con 0x55984698f1a0 2024-05-27T07:55:28.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.650+0000 7f4a93fff700 1 --2- 172.21.15.114:0/347814117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x559846998dc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4a800e09c0 tx=0x7f4a800a9fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d1b18c2e781a2ef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:28.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.650+0000 7f4a73fff700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:55:28.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.650+0000 7f4a73fff700 1 -- 172.21.15.114:0/347814117 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a800c3020 con 0x55984698f1a0 2024-05-27T07:55:28.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.650+0000 7f4a73fff700 1 -- 172.21.15.114:0/347814117 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(144..144 src has 1..144) ==== 9466+0+0 (secure 0 0 0) 0x7f4a800aa840 con 0x55984698f1a0 2024-05-27T07:55:28.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.656+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/347814117 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5598468bed40 con 0x55984698f1a0 2024-05-27T07:55:28.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.659+0000 7f4a73fff700 1 -- 172.21.15.114:0/347814117 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f4a800c9050 con 0x55984698f1a0 2024-05-27T07:55:28.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.662+0000 7f4aa9a60b80 10 client.28636 fetch_fsmap learned FSMap version 87 2024-05-27T07:55:28.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.662+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/347814117 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x5598468667e0 con 0x55984698f1a0 2024-05-27T07:55:28.663 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:28 smithi114 ceph-mon[32038]: osdmap e144: 12 total, 12 up, 12 in 2024-05-27T07:55:28.663 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:28 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6832/1527496858,v1:172.21.15.163:6833/1527496858] up:boot 2024-05-27T07:55:28.663 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:28 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:55:28.663 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:28 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:55:28.664 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:28 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:55:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.663+0000 7f4a73fff700 1 -- 172.21.15.114:0/347814117 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5598468bed40 con 0x55984698f1a0 2024-05-27T07:55:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.664+0000 7f4a73fff700 1 -- 172.21.15.114:0/347814117 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 87) ==== 4700+0+0 (secure 0 0 0) 0x7f4a800acbd0 con 0x55984698f1a0 2024-05-27T07:55:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.665+0000 7f4aa9a60b80 10 client.28636 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:55:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.665+0000 7f4aa9a60b80 10 client.28636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.665+0000 7f4aa9a60b80 20 client.28636 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.665+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/347814117 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5598469a8b40 0x5598469aaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.665+0000 7f4aa9a60b80 4 client.28636 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:55:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.665+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/347814117 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598468b8520 con 0x5598469a8b40 2024-05-27T07:55:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.673+0000 7f4a98de7700 1 --2- 172.21.15.114:0/347814117 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5598469a8b40 0x5598469aaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.685+0000 7f4a73fff700 10 client.28636 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.685+0000 7f4a98de7700 1 --2- 172.21.15.114:0/347814117 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5598469a8b40 0x5598469aaf60 crc :-1 s=READY pgs=438 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69e6d699bc8c49be server_cookie=40c8eaf18a4ec75c in_seq=0 out_seq=0 2024-05-27T07:55:28.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.693+0000 7f4a73fff700 1 -- 172.21.15.114:0/347814117 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5598468b8520 con 0x5598469a8b40 2024-05-27T07:55:28.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.693+0000 7f4a73fff700 10 client.28636 handle_command_reply: tid=0 2024-05-27T07:55:28.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.693+0000 7f4aa9a60b80 1 client.28636 shutdown 2024-05-27T07:55:28.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.693+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/347814117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 msgr2=0x559846998dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:28.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.693+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/347814117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x559846998dc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4a800e09c0 tx=0x7f4a800a9fb0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.693+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/347814117 shutdown_connections 2024-05-27T07:55:28.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.693+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/347814117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5598469950f0 0x559846999300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.693+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/347814117 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5598469a8b40 0x5598469aaf60 crc :-1 s=READY pgs=438 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.693+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/347814117 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55984680fec0 0x559846998880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.693+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/347814117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x559846998dc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.693+0000 7f4a73fff700 0 client.28636 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:28.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.693+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/347814117 >> 172.21.15.114:0/347814117 conn(0x5598468127f0 msgr2=0x55984698dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:28.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.694+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/347814117 shutdown_connections 2024-05-27T07:55:28.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.694+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/347814117 wait complete. 2024-05-27T07:55:28.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.694+0000 7f4aa9a60b80 20 client.28636 trim_cache size 0 max 16384 2024-05-27T07:55:28.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:28 smithi163 ceph-mon[36451]: osdmap e144: 12 total, 12 up, 12 in 2024-05-27T07:55:28.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:28 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6832/1527496858,v1:172.21.15.163:6833/1527496858] up:boot 2024-05-27T07:55:28.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:28 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:55:28.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:28 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:55:28.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:28 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:55:28.728 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:28 smithi163 podman[74829]: 2024-05-27 07:55:28.472250558 +0000 UTC m=+0.715546083 container remove a3e28185b9fa2bdfee926f9f8b25a08ef23a1c91bc3a8624dc3cd06e61896cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-27T07:55:28.728 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:28 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:55:28.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.733+0000 7f4aa9a60b80 1 Processor -- start 2024-05-27T07:55:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.734+0000 7f4aa9a60b80 1 -- start start 2024-05-27T07:55:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.734+0000 7f4aa9a60b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55984680fec0 0x559846999140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.735+0000 7f4aa9a60b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x5598469a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.735+0000 7f4aa9a60b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5598469950f0 0x5598469b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.735+0000 7f4aa9a60b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941c7d10 con 0x55984680fec0 2024-05-27T07:55:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.735+0000 7f4aa9a60b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941c83e0 con 0x5598469950f0 2024-05-27T07:55:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.735+0000 7f4aa9a60b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941c6c10 con 0x55984698f1a0 2024-05-27T07:55:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.735+0000 7f4a93fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x5598469a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.735+0000 7f4a93fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x5598469a4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36374/0 (socket says 172.21.15.114:36374) 2024-05-27T07:55:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.735+0000 7f4a93fff700 1 -- 172.21.15.114:0/3681379187 learned_addr learned my addr 172.21.15.114:0/3681379187 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.735+0000 7f4a93fff700 1 -- 172.21.15.114:0/3681379187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5598469950f0 msgr2=0x5598469b06d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:55:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.735+0000 7f4a93fff700 1 --2- 172.21.15.114:0/3681379187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5598469950f0 0x5598469b06d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.735+0000 7f4a93fff700 1 -- 172.21.15.114:0/3681379187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55984680fec0 msgr2=0x559846999140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.735+0000 7f4a93fff700 1 --2- 172.21.15.114:0/3681379187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55984680fec0 0x559846999140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.735+0000 7f4a93fff700 1 -- 172.21.15.114:0/3681379187 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559846996df0 con 0x55984698f1a0 2024-05-27T07:55:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.736+0000 7f4a93fff700 1 --2- 172.21.15.114:0/3681379187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x5598469a4d20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4a800a7850 tx=0x7f4a8000fcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deba98e1de2f4c48 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.736+0000 7f4a7e7fc700 1 -- 172.21.15.114:0/3681379187 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a800b7070 con 0x55984698f1a0 2024-05-27T07:55:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.737+0000 7f4a7e7fc700 1 -- 172.21.15.114:0/3681379187 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a800e3070 con 0x55984698f1a0 2024-05-27T07:55:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.737+0000 7f4a7e7fc700 1 -- 172.21.15.114:0/3681379187 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a800c3070 con 0x55984698f1a0 2024-05-27T07:55:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.738+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/3681379187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 msgr2=0x5598469a4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.738+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/3681379187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x5598469a4d20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4a800a7850 tx=0x7f4a8000fcb0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.739+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/3681379187 shutdown_connections 2024-05-27T07:55:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.739+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/3681379187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5598469950f0 0x5598469b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.740+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/3681379187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55984680fec0 0x559846999140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.740+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/3681379187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x5598469a4d20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.740+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/3681379187 >> 172.21.15.114:0/3681379187 conn(0x5598468127f0 msgr2=0x559846992b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.741+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/3681379187 shutdown_connections 2024-05-27T07:55:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.741+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/3681379187 wait complete. 2024-05-27T07:55:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.741+0000 7f4aa9a60b80 1 -- start start 2024-05-27T07:55:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.742+0000 7f4aa9a60b80 1 Processor -- start 2024-05-27T07:55:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.742+0000 7f4aa9a60b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55984680fec0 0x5598469a1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.743+0000 7f4aa9a60b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x55984699dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.743+0000 7f4a98de7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55984680fec0 0x5598469a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.743+0000 7f4a98de7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55984680fec0 0x5598469a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:54184/0 (socket says 172.21.15.114:54184) 2024-05-27T07:55:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.743+0000 7f4a93fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x55984699dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.743+0000 7f4aa9a60b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5598469950f0 0x55984699e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.743+0000 7f4a98de7700 1 -- 172.21.15.114:0/2965985541 learned_addr learned my addr 172.21.15.114:0/2965985541 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.743+0000 7f4a995e8700 1 --2- 172.21.15.114:0/2965985541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5598469950f0 0x55984699e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.744+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/2965985541 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941c5ec0 con 0x5598469950f0 2024-05-27T07:55:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.744+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/2965985541 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941c5900 con 0x55984680fec0 2024-05-27T07:55:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.744+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/2965985541 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941c6540 con 0x55984698f1a0 2024-05-27T07:55:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.744+0000 7f4aa9a60b80 10 client.0 _ll_register_callbacks cb 0x559846843630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.747+0000 7f4a995e8700 1 -- 172.21.15.114:0/2965985541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 msgr2=0x55984699dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.747+0000 7f4a995e8700 1 --2- 172.21.15.114:0/2965985541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x55984699dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.747+0000 7f4a995e8700 1 -- 172.21.15.114:0/2965985541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55984680fec0 msgr2=0x5598469a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.747+0000 7f4a995e8700 1 --2- 172.21.15.114:0/2965985541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55984680fec0 0x5598469a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.747+0000 7f4a995e8700 1 -- 172.21.15.114:0/2965985541 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55984699e960 con 0x5598469950f0 2024-05-27T07:55:28.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.749+0000 7f4a93fff700 1 --2- 172.21.15.114:0/2965985541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x55984699dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:55:28.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.750+0000 7f4a995e8700 1 --2- 172.21.15.114:0/2965985541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5598469950f0 0x55984699e420 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00ae80 tx=0x7f4a8c012510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4874b3cda9bf72d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.752+0000 7f4a73fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:55:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.752+0000 7f4a73fff700 1 -- 172.21.15.114:0/2965985541 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c01a070 con 0x5598469950f0 2024-05-27T07:55:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.752+0000 7f4a73fff700 1 -- 172.21.15.114:0/2965985541 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(144..144 src has 1..144) ==== 9466+0+0 (secure 0 0 0) 0x7f4a8c003c00 con 0x5598469950f0 2024-05-27T07:55:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.753+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/2965985541 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a800e3070 con 0x5598469950f0 2024-05-27T07:55:28.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.757+0000 7f4a73fff700 1 -- 172.21.15.114:0/2965985541 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f4a8c025050 con 0x5598469950f0 2024-05-27T07:55:28.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.758+0000 7f4aa9a60b80 10 client.19161 fetch_fsmap learned FSMap version 87 2024-05-27T07:55:28.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.758+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/2965985541 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x559846a14220 con 0x5598469950f0 2024-05-27T07:55:28.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.762+0000 7f4a73fff700 1 -- 172.21.15.114:0/2965985541 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a8c003e40 con 0x5598469950f0 2024-05-27T07:55:28.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.763+0000 7f4a73fff700 1 -- 172.21.15.114:0/2965985541 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 87) ==== 4700+0+0 (secure 0 0 0) 0x7f4a8c004680 con 0x5598469950f0 2024-05-27T07:55:28.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.763+0000 7f4aa9a60b80 10 client.19161 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:55:28.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.763+0000 7f4aa9a60b80 10 client.19161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:28.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.763+0000 7f4aa9a60b80 20 client.19161 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:28.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.763+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/2965985541 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559846a14910 0x559846a16d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:28.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.763+0000 7f4aa9a60b80 4 client.19161 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:55:28.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.763+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/2965985541 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f4a800e3070 con 0x559846a14910 2024-05-27T07:55:28.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.763+0000 7f4a98de7700 1 --2- 172.21.15.114:0/2965985541 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559846a14910 0x559846a16d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.766+0000 7f4a98de7700 1 --2- 172.21.15.114:0/2965985541 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559846a14910 0x559846a16d30 crc :-1 s=READY pgs=439 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e683a1f7fdf27846 server_cookie=dd2ee95370f8a001 in_seq=0 out_seq=0 2024-05-27T07:55:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.767+0000 7f4a73fff700 10 client.19161 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.768+0000 7f4a73fff700 1 -- 172.21.15.114:0/2965985541 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f4a800e3070 con 0x559846a14910 2024-05-27T07:55:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.768+0000 7f4a73fff700 10 client.19161 handle_command_reply: tid=0 2024-05-27T07:55:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.769+0000 7f4aa9a60b80 1 client.19161 shutdown 2024-05-27T07:55:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.770+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/2965985541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5598469950f0 msgr2=0x55984699e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.770+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/2965985541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5598469950f0 0x55984699e420 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00ae80 tx=0x7f4a8c012510 comp rx=0 tx=0).stop 2024-05-27T07:55:28.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.770+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/2965985541 shutdown_connections 2024-05-27T07:55:28.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.770+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/2965985541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5598469950f0 0x55984699e420 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.770+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/2965985541 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559846a14910 0x559846a16d30 crc :-1 s=READY pgs=439 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.770+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/2965985541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55984680fec0 0x5598469a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.770+0000 7f4aa9a60b80 1 --2- 172.21.15.114:0/2965985541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55984698f1a0 0x55984699dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.770+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/2965985541 >> 172.21.15.114:0/2965985541 conn(0x5598468127f0 msgr2=0x559846992b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:28.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.770+0000 7f4a73fff700 0 client.19161 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:28.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.771+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/2965985541 shutdown_connections 2024-05-27T07:55:28.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.771+0000 7f4aa9a60b80 1 -- 172.21.15.114:0/2965985541 wait complete. 2024-05-27T07:55:28.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.771+0000 7f4aa9a60b80 20 client.19161 trim_cache size 0 max 16384 2024-05-27T07:55:28.776 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:55:28.776 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:55:28.776 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "3da1fe97-0bf2-4de8-b4eb-d168c195edd8", 2024-05-27T07:55:28.776 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:55:28.776 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:55:28.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.775+0000 7f4a9b04b700 1 -- 172.21.15.114:0/3164579408 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4a840b2c00 msgr2=0x7f4a840b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:28.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.775+0000 7f4a9b04b700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4a840b2c00 0x7f4a840b5110 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4a88009ad0 tx=0x7f4a88009210 comp rx=0 tx=0).stop 2024-05-27T07:55:28.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.775+0000 7f4a9b04b700 1 -- 172.21.15.114:0/3164579408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a94108ef0 msgr2=0x7f4a9406e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:28.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.775+0000 7f4a9b04b700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a94108ef0 0x7f4a9406e290 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000b580 tx=0x7f4a8000b9c0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.776+0000 7f4a9b04b700 1 -- 172.21.15.114:0/3164579408 shutdown_connections 2024-05-27T07:55:28.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.776+0000 7f4a9b04b700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4a840b2c00 0x7f4a840b5110 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.776+0000 7f4a9b04b700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a94109840 0x7f4a9406e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.776+0000 7f4a9b04b700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a941045c0 0x7f4a940752f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.776+0000 7f4a9b04b700 1 --2- 172.21.15.114:0/3164579408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a94108ef0 0x7f4a9406e290 secure :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000b580 tx=0x7f4a8000b9c0 comp rx=0 tx=0).stop 2024-05-27T07:55:28.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.776+0000 7f4a9b04b700 1 -- 172.21.15.114:0/3164579408 >> 172.21.15.114:0/3164579408 conn(0x7f4a940ffe90 msgr2=0x7f4a9410a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:28.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.778+0000 7f4a9b04b700 1 -- 172.21.15.114:0/3164579408 shutdown_connections 2024-05-27T07:55:28.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:28.778+0000 7f4a9b04b700 1 -- 172.21.15.114:0/3164579408 wait complete. 2024-05-27T07:55:28.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:28 smithi186 ceph-mon[36647]: osdmap e144: 12 total, 12 up, 12 in 2024-05-27T07:55:28.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:28 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6832/1527496858,v1:172.21.15.163:6833/1527496858] up:boot 2024-05-27T07:55:28.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:28 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:55:28.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:28 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:55:28.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:28 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:55:29.226 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:28 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:55:29.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:55:29.597 INFO:tasks.workunit.client.0.smithi114.stdout: 10 24029 14.57 MB/sec warmup 101 sec latency 209.105 ms 2024-05-27T07:55:29.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:29 smithi114 ceph-mon[32038]: pgmap v537: 129 pgs: 129 active+clean; 295 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 574 op/s 2024-05-27T07:55:29.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:29 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:55:29.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:29 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:55:29.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:29 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:55:29.883 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:55:29.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:29 smithi163 ceph-mon[36451]: pgmap v537: 129 pgs: 129 active+clean; 295 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 574 op/s 2024-05-27T07:55:29.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:29 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:55:29.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:29 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:55:29.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:29 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:55:29.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:29 smithi186 ceph-mon[36647]: pgmap v537: 129 pgs: 129 active+clean; 295 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 574 op/s 2024-05-27T07:55:29.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:29 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:55:29.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:29 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:55:29.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:29 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:55:30.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 24377 14.60 MB/sec warmup 102 sec latency 77.870 ms 2024-05-27T07:55:31.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 24700 14.60 MB/sec warmup 103 sec latency 100.917 ms 2024-05-27T07:55:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.663+0000 7f293f220700 1 -- 172.21.15.114:0/2062254320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29381092c0 msgr2=0x7f29381096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.663+0000 7f293f220700 1 --2- 172.21.15.114:0/2062254320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29381092c0 0x7f29381096d0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f292c00b0a0 tx=0x7f292c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.663+0000 7f293f220700 1 -- 172.21.15.114:0/2062254320 shutdown_connections 2024-05-27T07:55:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.663+0000 7f293f220700 1 --2- 172.21.15.114:0/2062254320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2938104320 0x7f2938110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.663+0000 7f293f220700 1 --2- 172.21.15.114:0/2062254320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29381092c0 0x7f29381096d0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.663+0000 7f293f220700 1 --2- 172.21.15.114:0/2062254320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29381039d0 0x7f2938103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.663+0000 7f293f220700 1 -- 172.21.15.114:0/2062254320 >> 172.21.15.114:0/2062254320 conn(0x7f29380fe510 msgr2=0x7f2938100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.663+0000 7f293f220700 1 -- 172.21.15.114:0/2062254320 shutdown_connections 2024-05-27T07:55:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.663+0000 7f293f220700 1 -- 172.21.15.114:0/2062254320 wait complete. 2024-05-27T07:55:31.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.663+0000 7f293f220700 1 Processor -- start 2024-05-27T07:55:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293f220700 1 -- start start 2024-05-27T07:55:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293f220700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29381039d0 0x7f2938199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293f220700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2938104320 0x7f293819a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293f220700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29381092c0 0x7f293819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293f220700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2938113450 con 0x7f29381092c0 2024-05-27T07:55:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293f220700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f29381132d0 con 0x7f29381039d0 2024-05-27T07:55:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293f220700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f29381135d0 con 0x7f2938104320 2024-05-27T07:55:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293d7bd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29381092c0 0x7f293819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f2937fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2938104320 0x7f293819a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293d7bd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29381092c0 0x7f293819eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52070/0 (socket says 172.21.15.114:52070) 2024-05-27T07:55:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293d7bd700 1 -- 172.21.15.114:0/2863078124 learned_addr learned my addr 172.21.15.114:0/2863078124 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293cfbc700 1 --2- 172.21.15.114:0/2863078124 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29381039d0 0x7f2938199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293d7bd700 1 -- 172.21.15.114:0/2863078124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2938104320 msgr2=0x7f293819a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293d7bd700 1 --2- 172.21.15.114:0/2863078124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2938104320 0x7f293819a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293d7bd700 1 -- 172.21.15.114:0/2863078124 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29381039d0 msgr2=0x7f2938199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293d7bd700 1 --2- 172.21.15.114:0/2863078124 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29381039d0 0x7f2938199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.664+0000 7f293d7bd700 1 -- 172.21.15.114:0/2863078124 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293819a930 con 0x7f29381092c0 2024-05-27T07:55:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.665+0000 7f293d7bd700 1 --2- 172.21.15.114:0/2863078124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29381092c0 0x7f293819eb90 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f293000b550 tx=0x7f293000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.665+0000 7f2935ffb700 1 -- 172.21.15.114:0/2863078124 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930015020 con 0x7f29381092c0 2024-05-27T07:55:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.665+0000 7f293f220700 1 -- 172.21.15.114:0/2863078124 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f293819f1f0 con 0x7f29381092c0 2024-05-27T07:55:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.665+0000 7f2935ffb700 1 -- 172.21.15.114:0/2863078124 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2930011ce0 con 0x7f29381092c0 2024-05-27T07:55:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.665+0000 7f2935ffb700 1 -- 172.21.15.114:0/2863078124 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930010dd0 con 0x7f29381092c0 2024-05-27T07:55:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.665+0000 7f293f220700 1 -- 172.21.15.114:0/2863078124 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f293819f6e0 con 0x7f29381092c0 2024-05-27T07:55:31.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.670+0000 7f2935ffb700 1 -- 172.21.15.114:0/2863078124 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f2930018020 con 0x7f29381092c0 2024-05-27T07:55:31.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.670+0000 7f294dc35b80 1 Processor -- start 2024-05-27T07:55:31.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.670+0000 7f294dc35b80 1 -- start start 2024-05-27T07:55:31.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.670+0000 7f294dc35b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 0x55e48bd4a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.670+0000 7f294dc35b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd44a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.670+0000 7f294dc35b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd45390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.670+0000 7f294dc35b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e48bc73c90 con 0x55e48bbc6e40 2024-05-27T07:55:31.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.670+0000 7f294dc35b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e48bc63830 con 0x55e48bd4ac50 2024-05-27T07:55:31.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.670+0000 7f294dc35b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e48bc67870 con 0x55e48bd44f80 2024-05-27T07:55:31.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.670+0000 7f293cfbc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd44a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f293cfbc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd44a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:54222/0 (socket says 172.21.15.114:54222) 2024-05-27T07:55:31.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f293cfbc700 1 -- 172.21.15.114:0/1676371315 learned_addr learned my addr 172.21.15.114:0/1676371315 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:31.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f2937fff700 1 --2- 172.21.15.114:0/1676371315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd45390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f2937fff700 1 --2- 172.21.15.114:0/1676371315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 0x55e48bd4a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f293cfbc700 1 -- 172.21.15.114:0/1676371315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 msgr2=0x55e48bd45390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f293cfbc700 1 --2- 172.21.15.114:0/1676371315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd45390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f293cfbc700 1 -- 172.21.15.114:0/1676371315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 msgr2=0x55e48bd4a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f293cfbc700 1 --2- 172.21.15.114:0/1676371315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 0x55e48bd4a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f293cfbc700 1 -- 172.21.15.114:0/1676371315 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e48bbb38d0 con 0x55e48bd4ac50 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f2935ffb700 1 --2- 172.21.15.114:0/2863078124 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f29280b2a70 0x7f29280b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f2937fff700 1 --2- 172.21.15.114:0/1676371315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd45390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f2937fff700 1 --2- 172.21.15.114:0/2863078124 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f29280b2a70 0x7f29280b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f293cfbc700 1 --2- 172.21.15.114:0/1676371315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd44a40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f292c015000 tx=0x7f292c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cbd115bce89da32 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.671+0000 7f2935ffb700 1 -- 172.21.15.114:0/2863078124 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 9466+0+0 (secure 0 0 0) 0x7f29300e1f40 con 0x7f29381092c0 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.672+0000 7f2937fff700 1 --2- 172.21.15.114:0/2863078124 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f29280b2a70 0x7f29280b4f80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55e48bd45bb0 tx=0x7f2924009190 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.672+0000 7f2922ffd700 1 -- 172.21.15.114:0/1676371315 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292c004c50 con 0x55e48bd4ac50 2024-05-27T07:55:31.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.672+0000 7f2922ffd700 1 -- 172.21.15.114:0/1676371315 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f292c004df0 con 0x55e48bd4ac50 2024-05-27T07:55:31.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.672+0000 7f2922ffd700 1 -- 172.21.15.114:0/1676371315 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292c01c8b0 con 0x55e48bd4ac50 2024-05-27T07:55:31.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.673+0000 7f294dc35b80 1 -- 172.21.15.114:0/1676371315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 msgr2=0x55e48bd44a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:31.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.673+0000 7f294dc35b80 1 --2- 172.21.15.114:0/1676371315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd44a40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f292c015000 tx=0x7f292c009260 comp rx=0 tx=0).stop 2024-05-27T07:55:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.673+0000 7f294dc35b80 1 -- 172.21.15.114:0/1676371315 shutdown_connections 2024-05-27T07:55:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.673+0000 7f294dc35b80 1 --2- 172.21.15.114:0/1676371315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd45390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.673+0000 7f294dc35b80 1 --2- 172.21.15.114:0/1676371315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 0x55e48bd4a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.673+0000 7f294dc35b80 1 --2- 172.21.15.114:0/1676371315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd44a40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.673+0000 7f294dc35b80 1 -- 172.21.15.114:0/1676371315 >> 172.21.15.114:0/1676371315 conn(0x55e48bc21af0 msgr2=0x55e48bd3f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.673+0000 7f294dc35b80 1 -- 172.21.15.114:0/1676371315 shutdown_connections 2024-05-27T07:55:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.673+0000 7f294dc35b80 1 -- 172.21.15.114:0/1676371315 wait complete. 2024-05-27T07:55:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.673+0000 7f294dc35b80 1 -- start start 2024-05-27T07:55:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.673+0000 7f294dc35b80 1 Processor -- start 2024-05-27T07:55:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.674+0000 7f294dc35b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 0x55e48bd4e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.674+0000 7f294dc35b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd4e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.674+0000 7f294dc35b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd4ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.674+0000 7f294dc35b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e48bb69fa0 con 0x55e48bbc6e40 2024-05-27T07:55:31.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.674+0000 7f294dc35b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e48bc057c0 con 0x55e48bd4ac50 2024-05-27T07:55:31.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.674+0000 7f294dc35b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e48bd516e0 con 0x55e48bd44f80 2024-05-27T07:55:31.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.674+0000 7f294dc35b80 10 client.0 _ll_register_callbacks cb 0x55e48bbfe920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:31.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.674+0000 7f293cfbc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 0x55e48bd4e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.674+0000 7f2937fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd4e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.674+0000 7f2937fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd4e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36418/0 (socket says 172.21.15.114:36418) 2024-05-27T07:55:31.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.674+0000 7f293d7bd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd4ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.674+0000 7f2937fff700 1 -- 172.21.15.114:0/1645439241 learned_addr learned my addr 172.21.15.114:0/1645439241 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:31.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.674+0000 7f293cfbc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 0x55e48bd4e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52086/0 (socket says 172.21.15.114:52086) 2024-05-27T07:55:31.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.675+0000 7f293cfbc700 1 -- 172.21.15.114:0/1645439241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 msgr2=0x55e48bd4e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:31.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.675+0000 7f293cfbc700 1 --2- 172.21.15.114:0/1645439241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd4e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.675+0000 7f293cfbc700 1 -- 172.21.15.114:0/1645439241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 msgr2=0x55e48bd4ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:31.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.675+0000 7f293cfbc700 1 --2- 172.21.15.114:0/1645439241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd4ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.675+0000 7f293cfbc700 1 -- 172.21.15.114:0/1645439241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e48bc02690 con 0x55e48bbc6e40 2024-05-27T07:55:31.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.675+0000 7f293d7bd700 1 --2- 172.21.15.114:0/1645439241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd4ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:55:31.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.675+0000 7f293cfbc700 1 --2- 172.21.15.114:0/1645439241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 0x55e48bd4e3d0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f292c009fd0 tx=0x7f292c0077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f95f1b75c3fc6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:31.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.675+0000 7f2917fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:55:31.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.675+0000 7f2917fff700 1 -- 172.21.15.114:0/1645439241 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292c02a070 con 0x55e48bbc6e40 2024-05-27T07:55:31.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.675+0000 7f2917fff700 1 -- 172.21.15.114:0/1645439241 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(144..144 src has 1..144) ==== 9466+0+0 (secure 0 0 0) 0x7f292c01c440 con 0x55e48bbc6e40 2024-05-27T07:55:31.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.675+0000 7f294dc35b80 1 -- 172.21.15.114:0/1645439241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f292c004df0 con 0x55e48bbc6e40 2024-05-27T07:55:31.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.676+0000 7f2917fff700 1 -- 172.21.15.114:0/1645439241 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f292c037050 con 0x55e48bbc6e40 2024-05-27T07:55:31.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.676+0000 7f294dc35b80 10 client.19173 fetch_fsmap learned FSMap version 87 2024-05-27T07:55:31.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.676+0000 7f294dc35b80 1 -- 172.21.15.114:0/1645439241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x55e48bd5c180 con 0x55e48bbc6e40 2024-05-27T07:55:31.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.676+0000 7f2917fff700 1 -- 172.21.15.114:0/1645439241 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f292c004df0 con 0x55e48bbc6e40 2024-05-27T07:55:31.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.676+0000 7f2917fff700 1 -- 172.21.15.114:0/1645439241 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 87) ==== 4700+0+0 (secure 0 0 0) 0x7f292c011070 con 0x55e48bbc6e40 2024-05-27T07:55:31.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.676+0000 7f294dc35b80 10 client.19173 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:55:31.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.676+0000 7f294dc35b80 10 client.19173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:31.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.676+0000 7f294dc35b80 20 client.19173 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:31.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.676+0000 7f294dc35b80 1 --2- 172.21.15.114:0/1645439241 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e48bd5e7e0 0x55e48bd60cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.676+0000 7f294dc35b80 4 client.19173 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:55:31.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.676+0000 7f294dc35b80 1 -- 172.21.15.114:0/1645439241 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e48bd109f0 con 0x55e48bd5e7e0 2024-05-27T07:55:31.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.677+0000 7f2937fff700 1 --2- 172.21.15.114:0/1645439241 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e48bd5e7e0 0x55e48bd60cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.677+0000 7f2937fff700 1 --2- 172.21.15.114:0/1645439241 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e48bd5e7e0 0x55e48bd60cb0 crc :-1 s=READY pgs=440 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef3b4f9100dffa71 server_cookie=66ef54c4b990c021 in_seq=0 out_seq=0 2024-05-27T07:55:31.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.677+0000 7f2917fff700 10 client.19173 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:31.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.681+0000 7f2917fff700 1 -- 172.21.15.114:0/1645439241 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e48bd109f0 con 0x55e48bd5e7e0 2024-05-27T07:55:31.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.681+0000 7f2917fff700 10 client.19173 handle_command_reply: tid=0 2024-05-27T07:55:31.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.681+0000 7f294dc35b80 1 client.19173 shutdown 2024-05-27T07:55:31.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.682+0000 7f294dc35b80 1 -- 172.21.15.114:0/1645439241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 msgr2=0x55e48bd4e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:31.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.682+0000 7f294dc35b80 1 --2- 172.21.15.114:0/1645439241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 0x55e48bd4e3d0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f292c009fd0 tx=0x7f292c0077f0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.682+0000 7f294dc35b80 1 -- 172.21.15.114:0/1645439241 shutdown_connections 2024-05-27T07:55:31.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.682+0000 7f294dc35b80 1 --2- 172.21.15.114:0/1645439241 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e48bd5e7e0 0x55e48bd60cb0 crc :-1 s=READY pgs=440 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.682+0000 7f294dc35b80 1 --2- 172.21.15.114:0/1645439241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd4ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.682+0000 7f294dc35b80 1 --2- 172.21.15.114:0/1645439241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 0x55e48bd4e3d0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.682+0000 7f2917fff700 0 client.19173 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:31.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.682+0000 7f294dc35b80 1 --2- 172.21.15.114:0/1645439241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd4e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.682+0000 7f294dc35b80 1 -- 172.21.15.114:0/1645439241 >> 172.21.15.114:0/1645439241 conn(0x55e48bc21af0 msgr2=0x55e48bd43a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:31.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.682+0000 7f294dc35b80 1 -- 172.21.15.114:0/1645439241 shutdown_connections 2024-05-27T07:55:31.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.682+0000 7f294dc35b80 1 -- 172.21.15.114:0/1645439241 wait complete. 2024-05-27T07:55:31.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.682+0000 7f294dc35b80 20 client.19173 trim_cache size 0 max 16384 2024-05-27T07:55:31.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.702+0000 7f294dc35b80 1 Processor -- start 2024-05-27T07:55:31.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.702+0000 7f294dc35b80 1 -- start start 2024-05-27T07:55:31.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.702+0000 7f294dc35b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bbc6e40 0x55e48bd44a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.702+0000 7f294dc35b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bd44f80 0x55e48bd4ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.702+0000 7f294dc35b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd5a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.702+0000 7f294dc35b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f29381b6f40 con 0x55e48bd44f80 2024-05-27T07:55:31.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.702+0000 7f294dc35b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f29381b7610 con 0x55e48bd4ac50 2024-05-27T07:55:31.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.702+0000 7f294dc35b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f29381b5e40 con 0x55e48bbc6e40 2024-05-27T07:55:31.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.703+0000 7f2937fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bd44f80 0x55e48bd4ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.703+0000 7f293cfbc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bbc6e40 0x55e48bd44a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.703+0000 7f293cfbc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bbc6e40 0x55e48bd44a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36430/0 (socket says 172.21.15.114:36430) 2024-05-27T07:55:31.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.703+0000 7f293cfbc700 1 -- 172.21.15.114:0/579136446 learned_addr learned my addr 172.21.15.114:0/579136446 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:31.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.703+0000 7f2937fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bd44f80 0x55e48bd4ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52088/0 (socket says 172.21.15.114:52088) 2024-05-27T07:55:31.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.703+0000 7f293d7bd700 1 --2- 172.21.15.114:0/579136446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd5a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.703+0000 7f293cfbc700 1 -- 172.21.15.114:0/579136446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 msgr2=0x55e48bd5a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:31.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.703+0000 7f293cfbc700 1 --2- 172.21.15.114:0/579136446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd5a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.703+0000 7f293cfbc700 1 -- 172.21.15.114:0/579136446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bd44f80 msgr2=0x55e48bd4ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:31.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.703+0000 7f293cfbc700 1 --2- 172.21.15.114:0/579136446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bd44f80 0x55e48bd4ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.703+0000 7f293cfbc700 1 -- 172.21.15.114:0/579136446 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e48bd3a3d0 con 0x55e48bbc6e40 2024-05-27T07:55:31.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.703+0000 7f2937fff700 1 --2- 172.21.15.114:0/579136446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bd44f80 0x55e48bd4ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:55:31.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.703+0000 7f293cfbc700 1 --2- 172.21.15.114:0/579136446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bbc6e40 0x55e48bd44a40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f292c009900 tx=0x7f292c0099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26197381097710ab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:31.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.704+0000 7f29227fc700 1 -- 172.21.15.114:0/579136446 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292c02a040 con 0x55e48bbc6e40 2024-05-27T07:55:31.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.704+0000 7f29227fc700 1 -- 172.21.15.114:0/579136446 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f292c02fb40 con 0x55e48bbc6e40 2024-05-27T07:55:31.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.704+0000 7f29227fc700 1 -- 172.21.15.114:0/579136446 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292c01c6a0 con 0x55e48bbc6e40 2024-05-27T07:55:31.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.705+0000 7f294dc35b80 1 -- 172.21.15.114:0/579136446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bbc6e40 msgr2=0x55e48bd44a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:31.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.705+0000 7f294dc35b80 1 --2- 172.21.15.114:0/579136446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bbc6e40 0x55e48bd44a40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f292c009900 tx=0x7f292c0099e0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.705+0000 7f294dc35b80 1 -- 172.21.15.114:0/579136446 shutdown_connections 2024-05-27T07:55:31.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.705+0000 7f294dc35b80 1 --2- 172.21.15.114:0/579136446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd5a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.705+0000 7f294dc35b80 1 --2- 172.21.15.114:0/579136446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bbc6e40 0x55e48bd44a40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.705+0000 7f294dc35b80 1 --2- 172.21.15.114:0/579136446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bd44f80 0x55e48bd4ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.705+0000 7f294dc35b80 1 -- 172.21.15.114:0/579136446 >> 172.21.15.114:0/579136446 conn(0x55e48bc21af0 msgr2=0x55e48bd4ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:31.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.705+0000 7f294dc35b80 1 -- 172.21.15.114:0/579136446 shutdown_connections 2024-05-27T07:55:31.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.705+0000 7f294dc35b80 1 -- 172.21.15.114:0/579136446 wait complete. 2024-05-27T07:55:31.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.705+0000 7f294dc35b80 1 -- start start 2024-05-27T07:55:31.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.705+0000 7f294dc35b80 1 Processor -- start 2024-05-27T07:55:31.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f294dc35b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 0x55e48bd528d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f294dc35b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd52e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f294dc35b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd53350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f294dc35b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f29381b50a0 con 0x55e48bbc6e40 2024-05-27T07:55:31.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f294dc35b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f29381b4a30 con 0x55e48bd4ac50 2024-05-27T07:55:31.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f294dc35b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f29381b5770 con 0x55e48bd44f80 2024-05-27T07:55:31.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f294dc35b80 10 client.0 _ll_register_callbacks cb 0x55e48bbfe920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:31.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f2937fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd52e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f2937fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd52e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36436/0 (socket says 172.21.15.114:36436) 2024-05-27T07:55:31.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f2937fff700 1 -- 172.21.15.114:0/2783658396 learned_addr learned my addr 172.21.15.114:0/2783658396 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:31.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f293d7bd700 1 --2- 172.21.15.114:0/2783658396 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd53350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f2937fff700 1 -- 172.21.15.114:0/2783658396 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 msgr2=0x55e48bd53350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:31.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f2937fff700 1 --2- 172.21.15.114:0/2783658396 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd53350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f2937fff700 1 -- 172.21.15.114:0/2783658396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 msgr2=0x55e48bd528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:31.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f2937fff700 1 --2- 172.21.15.114:0/2783658396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 0x55e48bd528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.706+0000 7f2937fff700 1 -- 172.21.15.114:0/2783658396 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e48bdc8500 con 0x55e48bd44f80 2024-05-27T07:55:31.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.707+0000 7f2937fff700 1 --2- 172.21.15.114:0/2783658396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd52e10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2924006b30 tx=0x7f2924004860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c17facf3c46befdb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:31.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.707+0000 7f2917fff700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:55:31.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.707+0000 7f2917fff700 1 -- 172.21.15.114:0/2783658396 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2924025c40 con 0x55e48bd44f80 2024-05-27T07:55:31.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.707+0000 7f2917fff700 1 -- 172.21.15.114:0/2783658396 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(144..144 src has 1..144) ==== 9466+0+0 (secure 0 0 0) 0x7f2924025de0 con 0x55e48bd44f80 2024-05-27T07:55:31.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.707+0000 7f294dc35b80 1 -- 172.21.15.114:0/2783658396 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f292c02fb40 con 0x55e48bd44f80 2024-05-27T07:55:31.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.707+0000 7f2917fff700 1 -- 172.21.15.114:0/2783658396 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f292403f050 con 0x55e48bd44f80 2024-05-27T07:55:31.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.708+0000 7f294dc35b80 10 client.28657 fetch_fsmap learned FSMap version 87 2024-05-27T07:55:31.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.708+0000 7f294dc35b80 1 -- 172.21.15.114:0/2783658396 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x55e48bdc97e0 con 0x55e48bd44f80 2024-05-27T07:55:31.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.708+0000 7f2917fff700 1 -- 172.21.15.114:0/2783658396 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f292c02fb40 con 0x55e48bd44f80 2024-05-27T07:55:31.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.708+0000 7f2917fff700 1 -- 172.21.15.114:0/2783658396 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 87) ==== 4700+0+0 (secure 0 0 0) 0x7f2924022040 con 0x55e48bd44f80 2024-05-27T07:55:31.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.708+0000 7f294dc35b80 10 client.28657 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:55:31.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.708+0000 7f294dc35b80 10 client.28657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:31.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.708+0000 7f294dc35b80 20 client.28657 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:31.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.708+0000 7f294dc35b80 1 --2- 172.21.15.114:0/2783658396 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e48bdc9ed0 0x55e48bdcc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:31.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.708+0000 7f294dc35b80 4 client.28657 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:55:31.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.708+0000 7f294dc35b80 1 -- 172.21.15.114:0/2783658396 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e48bc75be0 con 0x55e48bdc9ed0 2024-05-27T07:55:31.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.708+0000 7f293cfbc700 1 --2- 172.21.15.114:0/2783658396 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e48bdc9ed0 0x55e48bdcc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:31.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.709+0000 7f2917fff700 10 client.28657 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:31.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.709+0000 7f293cfbc700 1 --2- 172.21.15.114:0/2783658396 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e48bdc9ed0 0x55e48bdcc2f0 crc :-1 s=READY pgs=441 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53e603a6f45cdfee server_cookie=68116e460079653a in_seq=0 out_seq=0 2024-05-27T07:55:31.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.711+0000 7f2917fff700 1 -- 172.21.15.114:0/2783658396 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e48bc75be0 con 0x55e48bdc9ed0 2024-05-27T07:55:31.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.711+0000 7f2917fff700 10 client.28657 handle_command_reply: tid=0 2024-05-27T07:55:31.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.711+0000 7f294dc35b80 1 client.28657 shutdown 2024-05-27T07:55:31.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.711+0000 7f294dc35b80 1 -- 172.21.15.114:0/2783658396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 msgr2=0x55e48bd52e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:31.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.711+0000 7f294dc35b80 1 --2- 172.21.15.114:0/2783658396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd52e10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2924006b30 tx=0x7f2924004860 comp rx=0 tx=0).stop 2024-05-27T07:55:31.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.712+0000 7f294dc35b80 1 -- 172.21.15.114:0/2783658396 shutdown_connections 2024-05-27T07:55:31.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.712+0000 7f294dc35b80 1 --2- 172.21.15.114:0/2783658396 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e48bdc9ed0 0x55e48bdcc2f0 crc :-1 s=READY pgs=441 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.712+0000 7f294dc35b80 1 --2- 172.21.15.114:0/2783658396 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e48bd4ac50 0x55e48bd53350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.712+0000 7f294dc35b80 1 --2- 172.21.15.114:0/2783658396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e48bbc6e40 0x55e48bd528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.712+0000 7f294dc35b80 1 --2- 172.21.15.114:0/2783658396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e48bd44f80 0x55e48bd52e10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.712+0000 7f294dc35b80 1 -- 172.21.15.114:0/2783658396 >> 172.21.15.114:0/2783658396 conn(0x55e48bc21af0 msgr2=0x55e48bd50100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:31.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.712+0000 7f2917fff700 0 client.28657 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:31.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.712+0000 7f294dc35b80 1 -- 172.21.15.114:0/2783658396 shutdown_connections 2024-05-27T07:55:31.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.712+0000 7f294dc35b80 1 -- 172.21.15.114:0/2783658396 wait complete. 2024-05-27T07:55:31.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.712+0000 7f294dc35b80 20 client.28657 trim_cache size 0 max 16384 2024-05-27T07:55:31.716 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:55:31.716 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:55:31.716 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:55:31.716 INFO:teuthology.orchestra.run.smithi114.stdout: "3da1fe97-0bf2-4de8-b4eb-d168c195edd8": { 2024-05-27T07:55:31.716 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:55:31.717 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "3da1fe97-0bf2-4de8-b4eb-d168c195edd8", 2024-05-27T07:55:31.717 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:55:31.717 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:55:31.717 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:55:31.717 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:55:31.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.713+0000 7f293f220700 1 -- 172.21.15.114:0/2863078124 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f29280b2a70 msgr2=0x7f29280b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:31.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.713+0000 7f293f220700 1 --2- 172.21.15.114:0/2863078124 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f29280b2a70 0x7f29280b4f80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55e48bd45bb0 tx=0x7f2924009190 comp rx=0 tx=0).stop 2024-05-27T07:55:31.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.713+0000 7f293f220700 1 -- 172.21.15.114:0/2863078124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29381092c0 msgr2=0x7f293819eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:31.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.713+0000 7f293f220700 1 --2- 172.21.15.114:0/2863078124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29381092c0 0x7f293819eb90 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f293000b550 tx=0x7f293000b860 comp rx=0 tx=0).stop 2024-05-27T07:55:31.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.713+0000 7f293f220700 1 -- 172.21.15.114:0/2863078124 shutdown_connections 2024-05-27T07:55:31.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.713+0000 7f293f220700 1 --2- 172.21.15.114:0/2863078124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29381092c0 0x7f293819eb90 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.713+0000 7f293f220700 1 --2- 172.21.15.114:0/2863078124 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f29280b2a70 0x7f29280b4f80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.713+0000 7f293f220700 1 --2- 172.21.15.114:0/2863078124 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29381039d0 0x7f2938199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.713+0000 7f293f220700 1 --2- 172.21.15.114:0/2863078124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2938104320 0x7f293819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:31.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.713+0000 7f293f220700 1 -- 172.21.15.114:0/2863078124 >> 172.21.15.114:0/2863078124 conn(0x7f29380fe510 msgr2=0x7f2938100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:31.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.714+0000 7f293f220700 1 -- 172.21.15.114:0/2863078124 shutdown_connections 2024-05-27T07:55:31.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:55:31.714+0000 7f293f220700 1 -- 172.21.15.114:0/2863078124 wait complete. 2024-05-27T07:55:31.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:31 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:55:31.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:31 smithi163 ceph-mon[36451]: pgmap v538: 129 pgs: 129 active+clean; 303 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.5 MiB/s wr, 527 op/s 2024-05-27T07:55:31.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:31 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:55:31.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:31 smithi186 ceph-mon[36647]: pgmap v538: 129 pgs: 129 active+clean; 303 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.5 MiB/s wr, 527 op/s 2024-05-27T07:55:32.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:31 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:55:32.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:31 smithi114 ceph-mon[32038]: pgmap v538: 129 pgs: 129 active+clean; 303 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.5 MiB/s wr, 527 op/s 2024-05-27T07:55:32.493 INFO:tasks.cephfs.filesystem:scrub status for tag:3da1fe97-0bf2-4de8-b4eb-d168c195edd8 - {'path': '/', 'tag': '3da1fe97-0bf2-4de8-b4eb-d168c195edd8', 'options': 'recursive,force'} 2024-05-27T07:55:32.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 24985 14.62 MB/sec warmup 104 sec latency 233.558 ms 2024-05-27T07:55:32.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:32 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:55:32.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:32 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:55:32.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:32 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:55:33.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 25174 14.56 MB/sec warmup 105 sec latency 216.234 ms 2024-05-27T07:55:33.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:33 smithi163 ceph-mon[36451]: pgmap v539: 129 pgs: 129 active+clean; 316 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.0 MiB/s wr, 592 op/s 2024-05-27T07:55:33.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:33 smithi186 ceph-mon[36647]: pgmap v539: 129 pgs: 129 active+clean; 316 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.0 MiB/s wr, 592 op/s 2024-05-27T07:55:34.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:33 smithi114 ceph-mon[32038]: pgmap v539: 129 pgs: 129 active+clean; 316 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.0 MiB/s wr, 592 op/s 2024-05-27T07:55:34.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 25509 14.62 MB/sec warmup 106 sec latency 164.266 ms 2024-05-27T07:55:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.717+0000 7f0bc3d0d640 1 -- 172.21.15.114:0/1166995492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc108b80 msgr2=0x7f0bbc102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.717+0000 7f0bc3d0d640 1 --2- 172.21.15.114:0/1166995492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc108b80 0x7f0bbc102720 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009900 tx=0x7f0bb802a410 comp rx=0 tx=0).stop 2024-05-27T07:55:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.717+0000 7f0bc3d0d640 1 -- 172.21.15.114:0/1166995492 shutdown_connections 2024-05-27T07:55:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.717+0000 7f0bc3d0d640 1 --2- 172.21.15.114:0/1166995492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0bbc1032d0 0x7f0bbc10f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.717+0000 7f0bc3d0d640 1 --2- 172.21.15.114:0/1166995492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0bbc109020 0x7f0bbc102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.717+0000 7f0bc3d0d640 1 --2- 172.21.15.114:0/1166995492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc108b80 0x7f0bbc102720 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.717+0000 7f0bc3d0d640 1 -- 172.21.15.114:0/1166995492 >> 172.21.15.114:0/1166995492 conn(0x7f0bbc0fe380 msgr2=0x7f0bbc1007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.718+0000 7f0bc3d0d640 1 -- 172.21.15.114:0/1166995492 shutdown_connections 2024-05-27T07:55:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.718+0000 7f0bc3d0d640 1 -- 172.21.15.114:0/1166995492 wait complete. 2024-05-27T07:55:34.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc3d0d640 1 Processor -- start 2024-05-27T07:55:34.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc3d0d640 1 -- start start 2024-05-27T07:55:34.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc3d0d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0bbc1032d0 0x7f0bbc1a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc3d0d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc108b80 0x7f0bbc1a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc3d0d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0bbc109020 0x7f0bbc19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc3d0d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc112d70 con 0x7f0bbc108b80 2024-05-27T07:55:34.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc3d0d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc112bf0 con 0x7f0bbc1032d0 2024-05-27T07:55:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc3d0d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc112ef0 con 0x7f0bbc109020 2024-05-27T07:55:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc12a8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc108b80 0x7f0bbc1a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc12a8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc108b80 0x7f0bbc1a0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35186/0 (socket says 172.21.15.114:35186) 2024-05-27T07:55:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc12a8640 1 -- 172.21.15.114:0/4246017556 learned_addr learned my addr 172.21.15.114:0/4246017556 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc1aa9640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0bbc1032d0 0x7f0bbc1a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc22aa640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0bbc109020 0x7f0bbc19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc12a8640 1 -- 172.21.15.114:0/4246017556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0bbc109020 msgr2=0x7f0bbc19a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc12a8640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0bbc109020 0x7f0bbc19a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc12a8640 1 -- 172.21.15.114:0/4246017556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0bbc1032d0 msgr2=0x7f0bbc1a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc12a8640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0bbc1032d0 0x7f0bbc1a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.719+0000 7f0bc12a8640 1 -- 172.21.15.114:0/4246017556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bbc19ae70 con 0x7f0bbc108b80 2024-05-27T07:55:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.720+0000 7f0bc12a8640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc108b80 0x7f0bbc1a0b00 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00e9f0 tx=0x7f0bac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.720+0000 7f0bb2ffd640 1 -- 172.21.15.114:0/4246017556 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bac00ccc0 con 0x7f0bbc108b80 2024-05-27T07:55:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.720+0000 7f0bc3d0d640 1 -- 172.21.15.114:0/4246017556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bbc19b100 con 0x7f0bbc108b80 2024-05-27T07:55:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.720+0000 7f0bb2ffd640 1 -- 172.21.15.114:0/4246017556 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bac0035f0 con 0x7f0bbc108b80 2024-05-27T07:55:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.720+0000 7f0bc3d0d640 1 -- 172.21.15.114:0/4246017556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bbc1a7360 con 0x7f0bbc108b80 2024-05-27T07:55:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.720+0000 7f0bb2ffd640 1 -- 172.21.15.114:0/4246017556 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bac00ce60 con 0x7f0bbc108b80 2024-05-27T07:55:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.724+0000 7f0bb2ffd640 1 -- 172.21.15.114:0/4246017556 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f0bac010690 con 0x7f0bbc108b80 2024-05-27T07:55:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.724+0000 7f0bcabc3740 1 Processor -- start 2024-05-27T07:55:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.724+0000 7f0bcabc3740 1 -- start start 2024-05-27T07:55:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.724+0000 7f0bcabc3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 0x5622bc1bc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.724+0000 7f0bcabc3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbdf22f0 0x5622bc1bc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.724+0000 7f0bcabc3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 0x5622bc1bcd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.724+0000 7f0bcabc3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5622bc12ec30 con 0x5622bbe1b1b0 2024-05-27T07:55:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.724+0000 7f0bcabc3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5622bc11e7d0 con 0x5622bbdfa4d0 2024-05-27T07:55:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.724+0000 7f0bcabc3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5622bc122810 con 0x5622bbdf22f0 2024-05-27T07:55:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc22aa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 0x5622bc1bc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc1aa9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbdf22f0 0x5622bc1bc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc22aa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 0x5622bc1bc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35192/0 (socket says 172.21.15.114:35192) 2024-05-27T07:55:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc22aa640 1 -- 172.21.15.114:0/1860629188 learned_addr learned my addr 172.21.15.114:0/1860629188 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc12a8640 1 --2- 172.21.15.114:0/1860629188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 0x5622bc1bcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc1aa9640 1 -- 172.21.15.114:0/1860629188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 msgr2=0x5622bc1bcd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc1aa9640 1 --2- 172.21.15.114:0/1860629188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 0x5622bc1bcd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc1aa9640 1 -- 172.21.15.114:0/1860629188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 msgr2=0x5622bc1bc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc1aa9640 1 --2- 172.21.15.114:0/1860629188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 0x5622bc1bc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc1aa9640 1 -- 172.21.15.114:0/1860629188 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622bbe33210 con 0x5622bbdf22f0 2024-05-27T07:55:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc12a8640 1 --2- 172.21.15.114:0/1860629188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 0x5622bc1bcd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:55:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc22aa640 1 --2- 172.21.15.114:0/1860629188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 0x5622bc1bc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:55:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bb2ffd640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0ba80b2950 0x7f0ba80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc1aa9640 1 --2- 172.21.15.114:0/1860629188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbdf22f0 0x5622bc1bc7e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8006010 tx=0x7f0bb8032b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe909711131d76d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bc22aa640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0ba80b2950 0x7f0ba80b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.725+0000 7f0bb2ffd640 1 -- 172.21.15.114:0/4246017556 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 9466+0+0 (secure 0 0 0) 0x7f0bac0e2c90 con 0x7f0bbc108b80 2024-05-27T07:55:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.726+0000 7f0ba7fff640 1 -- 172.21.15.114:0/1860629188 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8040070 con 0x5622bbdf22f0 2024-05-27T07:55:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.726+0000 7f0ba7fff640 1 -- 172.21.15.114:0/1860629188 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bb8004030 con 0x5622bbdf22f0 2024-05-27T07:55:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.726+0000 7f0ba7fff640 1 -- 172.21.15.114:0/1860629188 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8035040 con 0x5622bbdf22f0 2024-05-27T07:55:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.727+0000 7f0bcabc3740 1 -- 172.21.15.114:0/1860629188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbdf22f0 msgr2=0x5622bc1bc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.727+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/1860629188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbdf22f0 0x5622bc1bc7e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8006010 tx=0x7f0bb8032b20 comp rx=0 tx=0).stop 2024-05-27T07:55:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.727+0000 7f0bc22aa640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0ba80b2950 0x7f0ba80b4e60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4011890 tx=0x7f0bb4009500 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.727+0000 7f0bcabc3740 1 -- 172.21.15.114:0/1860629188 shutdown_connections 2024-05-27T07:55:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.727+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/1860629188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 0x5622bc1bcd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.727+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/1860629188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbdf22f0 0x5622bc1bc7e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.727+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/1860629188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 0x5622bc1bc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.727+0000 7f0bcabc3740 1 -- 172.21.15.114:0/1860629188 >> 172.21.15.114:0/1860629188 conn(0x5622bbe0f300 msgr2=0x5622bc1b8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.727+0000 7f0bcabc3740 1 -- 172.21.15.114:0/1860629188 shutdown_connections 2024-05-27T07:55:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.727+0000 7f0bcabc3740 1 -- 172.21.15.114:0/1860629188 wait complete. 2024-05-27T07:55:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.727+0000 7f0bcabc3740 1 -- start start 2024-05-27T07:55:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.727+0000 7f0bcabc3740 1 Processor -- start 2024-05-27T07:55:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.728+0000 7f0bcabc3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdf22f0 0x5622bc1cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.728+0000 7f0bcabc3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdfa4d0 0x5622bc1cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.728+0000 7f0bcabc3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.728+0000 7f0bcabc3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5622bbe04790 con 0x5622bbdfa4d0 2024-05-27T07:55:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.728+0000 7f0bcabc3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5622bc0da890 con 0x5622bbdf22f0 2024-05-27T07:55:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.728+0000 7f0bcabc3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5622bc1cd960 con 0x5622bbe1b1b0 2024-05-27T07:55:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.728+0000 7f0bc12a8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdfa4d0 0x5622bc1cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.728+0000 7f0bc22aa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.728+0000 7f0bc22aa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36208/0 (socket says 172.21.15.114:36208) 2024-05-27T07:55:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.728+0000 7f0bc12a8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdfa4d0 0x5622bc1cb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35208/0 (socket says 172.21.15.114:35208) 2024-05-27T07:55:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.728+0000 7f0bc22aa640 1 -- 172.21.15.114:0/3854702655 learned_addr learned my addr 172.21.15.114:0/3854702655 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.728+0000 7f0bcabc3740 10 client.0 _ll_register_callbacks cb 0x5622bbe32a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0bc1aa9640 1 --2- 172.21.15.114:0/3854702655 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdf22f0 0x5622bc1cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0bc22aa640 1 -- 172.21.15.114:0/3854702655 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdf22f0 msgr2=0x5622bc1cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0bc22aa640 1 --2- 172.21.15.114:0/3854702655 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdf22f0 0x5622bc1cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0bc22aa640 1 -- 172.21.15.114:0/3854702655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdfa4d0 msgr2=0x5622bc1cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0bc22aa640 1 --2- 172.21.15.114:0/3854702655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdfa4d0 0x5622bc1cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0bc22aa640 1 -- 172.21.15.114:0/3854702655 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622bbeff620 con 0x5622bbe1b1b0 2024-05-27T07:55:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0bc12a8640 1 --2- 172.21.15.114:0/3854702655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdfa4d0 0x5622bc1cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:55:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0bc22aa640 1 --2- 172.21.15.114:0/3854702655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d0cd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400fb50 tx=0x7f0bb4004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6cb9b82e771b526 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0ba57fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:55:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0ba57fa640 1 -- 172.21.15.114:0/3854702655 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb400fdc0 con 0x5622bbe1b1b0 2024-05-27T07:55:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0ba57fa640 1 -- 172.21.15.114:0/3854702655 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(144..144 src has 1..144) ==== 9466+0+0 (secure 0 0 0) 0x7f0bb4017030 con 0x5622bbe1b1b0 2024-05-27T07:55:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0bcabc3740 1 -- 172.21.15.114:0/3854702655 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bb8004030 con 0x5622bbe1b1b0 2024-05-27T07:55:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0ba57fa640 1 -- 172.21.15.114:0/3854702655 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f0bb4004b20 con 0x5622bbe1b1b0 2024-05-27T07:55:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0bcabc3740 10 client.28669 fetch_fsmap learned FSMap version 87 2024-05-27T07:55:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.729+0000 7f0bcabc3740 1 -- 172.21.15.114:0/3854702655 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x5622bc1d9c90 con 0x5622bbe1b1b0 2024-05-27T07:55:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.730+0000 7f0ba57fa640 1 -- 172.21.15.114:0/3854702655 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bb8004030 con 0x5622bbe1b1b0 2024-05-27T07:55:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.730+0000 7f0ba57fa640 1 -- 172.21.15.114:0/3854702655 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 87) ==== 4700+0+0 (secure 0 0 0) 0x7f0bb401dbb0 con 0x5622bbe1b1b0 2024-05-27T07:55:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.730+0000 7f0bcabc3740 10 client.28669 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:55:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.730+0000 7f0bcabc3740 10 client.28669 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.730+0000 7f0bcabc3740 20 client.28669 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:34.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.730+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/3854702655 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622bbdd9d80 0x5622bc1dd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.730+0000 7f0bcabc3740 4 client.28669 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:55:34.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.730+0000 7f0bcabc3740 1 -- 172.21.15.114:0/3854702655 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622bc130b80 con 0x5622bbdd9d80 2024-05-27T07:55:34.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.730+0000 7f0bc1aa9640 1 --2- 172.21.15.114:0/3854702655 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622bbdd9d80 0x5622bc1dd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.731+0000 7f0ba57fa640 10 client.28669 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:34.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.731+0000 7f0bc1aa9640 1 --2- 172.21.15.114:0/3854702655 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622bbdd9d80 0x5622bc1dd6d0 crc :-1 s=READY pgs=442 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7e6e2e1eeda98de server_cookie=c1551998eeb5c661 in_seq=0 out_seq=0 2024-05-27T07:55:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.733+0000 7f0ba57fa640 1 -- 172.21.15.114:0/3854702655 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5622bc130b80 con 0x5622bbdd9d80 2024-05-27T07:55:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.733+0000 7f0ba57fa640 10 client.28669 handle_command_reply: tid=0 2024-05-27T07:55:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.734+0000 7f0bcabc3740 1 client.28669 shutdown 2024-05-27T07:55:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.734+0000 7f0bcabc3740 1 -- 172.21.15.114:0/3854702655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbe1b1b0 msgr2=0x5622bc1d0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.734+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/3854702655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d0cd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400fb50 tx=0x7f0bb4004ab0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.734+0000 7f0bcabc3740 1 -- 172.21.15.114:0/3854702655 shutdown_connections 2024-05-27T07:55:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.734+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/3854702655 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622bbdd9d80 0x5622bc1dd6d0 crc :-1 s=READY pgs=442 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.734+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/3854702655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d0cd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.734+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/3854702655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdfa4d0 0x5622bc1cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.734+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/3854702655 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdf22f0 0x5622bc1cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.734+0000 7f0bcabc3740 1 -- 172.21.15.114:0/3854702655 >> 172.21.15.114:0/3854702655 conn(0x5622bbe0f300 msgr2=0x5622bc1bea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.734+0000 7f0ba57fa640 0 client.28669 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.734+0000 7f0bcabc3740 1 -- 172.21.15.114:0/3854702655 shutdown_connections 2024-05-27T07:55:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.734+0000 7f0bcabc3740 1 -- 172.21.15.114:0/3854702655 wait complete. 2024-05-27T07:55:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.734+0000 7f0bcabc3740 20 client.28669 trim_cache size 0 max 16384 2024-05-27T07:55:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.752+0000 7f0bcabc3740 1 Processor -- start 2024-05-27T07:55:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.752+0000 7f0bcabc3740 1 -- start start 2024-05-27T07:55:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.752+0000 7f0bcabc3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbdf22f0 0x5622bc1c4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.752+0000 7f0bcabc3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 0x5622bc1c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.752+0000 7f0bcabc3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.752+0000 7f0bcabc3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc1af5a0 con 0x5622bbe1b1b0 2024-05-27T07:55:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.752+0000 7f0bcabc3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc1afc70 con 0x5622bbdfa4d0 2024-05-27T07:55:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.752+0000 7f0bcabc3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc1ae4c0 con 0x5622bbdf22f0 2024-05-27T07:55:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0bc22aa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0bc22aa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35210/0 (socket says 172.21.15.114:35210) 2024-05-27T07:55:34.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0bc22aa640 1 -- 172.21.15.114:0/1138297640 learned_addr learned my addr 172.21.15.114:0/1138297640 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:34.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0bc1aa9640 1 --2- 172.21.15.114:0/1138297640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbdf22f0 0x5622bc1c4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0bc12a8640 1 --2- 172.21.15.114:0/1138297640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 0x5622bc1c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0bc22aa640 1 -- 172.21.15.114:0/1138297640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbdf22f0 msgr2=0x5622bc1c4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:34.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0bc22aa640 1 --2- 172.21.15.114:0/1138297640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbdf22f0 0x5622bc1c4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0bc22aa640 1 -- 172.21.15.114:0/1138297640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 msgr2=0x5622bc1c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:34.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0bc22aa640 1 --2- 172.21.15.114:0/1138297640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 0x5622bc1c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0bc22aa640 1 -- 172.21.15.114:0/1138297640 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622bbdd9fb0 con 0x5622bbe1b1b0 2024-05-27T07:55:34.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0bc22aa640 1 --2- 172.21.15.114:0/1138297640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d79a0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4000f80 tx=0x7f0bb4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17a50b4a4ab1909 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:34.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0ba77fe640 1 -- 172.21.15.114:0/1138297640 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb403a020 con 0x5622bbe1b1b0 2024-05-27T07:55:34.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0ba77fe640 1 -- 172.21.15.114:0/1138297640 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bb4003e20 con 0x5622bbe1b1b0 2024-05-27T07:55:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.753+0000 7f0ba77fe640 1 -- 172.21.15.114:0/1138297640 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb401da50 con 0x5622bbe1b1b0 2024-05-27T07:55:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.754+0000 7f0bcabc3740 1 -- 172.21.15.114:0/1138297640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 msgr2=0x5622bc1d79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.754+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/1138297640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d79a0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4000f80 tx=0x7f0bb4009f90 comp rx=0 tx=0).stop 2024-05-27T07:55:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.754+0000 7f0bcabc3740 1 -- 172.21.15.114:0/1138297640 shutdown_connections 2024-05-27T07:55:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.754+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/1138297640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d79a0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.754+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/1138297640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 0x5622bc1c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.754+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/1138297640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbdf22f0 0x5622bc1c4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.754+0000 7f0bcabc3740 1 -- 172.21.15.114:0/1138297640 >> 172.21.15.114:0/1138297640 conn(0x5622bbe0f300 msgr2=0x5622bc1c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.754+0000 7f0bcabc3740 1 -- 172.21.15.114:0/1138297640 shutdown_connections 2024-05-27T07:55:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.754+0000 7f0bcabc3740 1 -- 172.21.15.114:0/1138297640 wait complete. 2024-05-27T07:55:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bcabc3740 1 -- start start 2024-05-27T07:55:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bcabc3740 1 Processor -- start 2024-05-27T07:55:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bcabc3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdf22f0 0x5622bc1cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bcabc3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 0x5622bc1d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bcabc3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bcabc3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc1ad720 con 0x5622bbdf22f0 2024-05-27T07:55:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bcabc3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc1ad0b0 con 0x5622bbdfa4d0 2024-05-27T07:55:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bcabc3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc1addf0 con 0x5622bbe1b1b0 2024-05-27T07:55:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bcabc3740 10 client.0 _ll_register_callbacks cb 0x7f0bb4011e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bc1aa9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdf22f0 0x5622bc1cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bc1aa9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdf22f0 0x5622bc1cfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35224/0 (socket says 172.21.15.114:35224) 2024-05-27T07:55:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bc1aa9640 1 -- 172.21.15.114:0/532617467 learned_addr learned my addr 172.21.15.114:0/532617467 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bc12a8640 1 --2- 172.21.15.114:0/532617467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 0x5622bc1d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.755+0000 7f0bc22aa640 1 --2- 172.21.15.114:0/532617467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0bc1aa9640 1 -- 172.21.15.114:0/532617467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbe1b1b0 msgr2=0x5622bc1d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0bc1aa9640 1 --2- 172.21.15.114:0/532617467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0bc1aa9640 1 -- 172.21.15.114:0/532617467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 msgr2=0x5622bc1d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0bc1aa9640 1 --2- 172.21.15.114:0/532617467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 0x5622bc1d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0bc1aa9640 1 -- 172.21.15.114:0/532617467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622bc243d50 con 0x5622bbdf22f0 2024-05-27T07:55:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0bc1aa9640 1 --2- 172.21.15.114:0/532617467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdf22f0 0x5622bc1cfca0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8033000 tx=0x7f0bb8031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf08d8213f2a6f54 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0ba57fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:55:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0ba57fa640 1 -- 172.21.15.114:0/532617467 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8035040 con 0x5622bbdf22f0 2024-05-27T07:55:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0bcabc3740 1 -- 172.21.15.114:0/532617467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bb4003e20 con 0x5622bbdf22f0 2024-05-27T07:55:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0ba57fa640 1 -- 172.21.15.114:0/532617467 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(144..144 src has 1..144) ==== 9466+0+0 (secure 0 0 0) 0x7f0bb8004040 con 0x5622bbdf22f0 2024-05-27T07:55:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0ba57fa640 1 -- 172.21.15.114:0/532617467 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f0bb8040040 con 0x5622bbdf22f0 2024-05-27T07:55:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0bcabc3740 10 client.19194 fetch_fsmap learned FSMap version 87 2024-05-27T07:55:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0bcabc3740 1 -- 172.21.15.114:0/532617467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x5622bc244e80 con 0x5622bbdf22f0 2024-05-27T07:55:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0ba57fa640 1 -- 172.21.15.114:0/532617467 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bb4003e20 con 0x5622bbdf22f0 2024-05-27T07:55:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0ba57fa640 1 -- 172.21.15.114:0/532617467 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 87) ==== 4700+0+0 (secure 0 0 0) 0x7f0bb8003680 con 0x5622bbdf22f0 2024-05-27T07:55:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0bcabc3740 10 client.19194 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:55:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0bcabc3740 10 client.19194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0bcabc3740 20 client.19194 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.756+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/532617467 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622bbdaf1c0 0x5622bc2475f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.757+0000 7f0bcabc3740 4 client.19194 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:55:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.757+0000 7f0bcabc3740 1 -- 172.21.15.114:0/532617467 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5622bc12a360 con 0x5622bbdaf1c0 2024-05-27T07:55:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.757+0000 7f0bc12a8640 1 --2- 172.21.15.114:0/532617467 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622bbdaf1c0 0x5622bc2475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.757+0000 7f0ba57fa640 10 client.19194 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:34.757+0000 7f0bc12a8640 1 --2- 172.21.15.114:0/532617467 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622bbdaf1c0 0x5622bc2475f0 crc :-1 s=READY pgs=443 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cc6622d3597c3f3 server_cookie=8004487132511285 in_seq=0 out_seq=0 2024-05-27T07:55:35.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 25598 14.53 MB/sec warmup 107 sec latency 735.834 ms 2024-05-27T07:55:35.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.973+0000 7f0ba57fa640 1 -- 172.21.15.114:0/532617467 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5622bc12a360 con 0x5622bbdaf1c0 2024-05-27T07:55:35.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.973+0000 7f0ba57fa640 10 client.19194 handle_command_reply: tid=0 2024-05-27T07:55:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.973+0000 7f0bcabc3740 1 client.19194 shutdown 2024-05-27T07:55:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.973+0000 7f0bcabc3740 1 -- 172.21.15.114:0/532617467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdf22f0 msgr2=0x5622bc1cfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.973+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/532617467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdf22f0 0x5622bc1cfca0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8033000 tx=0x7f0bb8031400 comp rx=0 tx=0).stop 2024-05-27T07:55:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.974+0000 7f0bcabc3740 1 -- 172.21.15.114:0/532617467 shutdown_connections 2024-05-27T07:55:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.974+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/532617467 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622bbdaf1c0 0x5622bc2475f0 crc :-1 s=READY pgs=443 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.974+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/532617467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622bbe1b1b0 0x5622bc1d0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.974+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/532617467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622bbdfa4d0 0x5622bc1d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.974+0000 7f0bcabc3740 1 --2- 172.21.15.114:0/532617467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622bbdf22f0 0x5622bc1cfca0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.974+0000 7f0ba57fa640 0 client.19194 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.974+0000 7f0bcabc3740 1 -- 172.21.15.114:0/532617467 >> 172.21.15.114:0/532617467 conn(0x5622bbe0f300 msgr2=0x5622bc1bdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.974+0000 7f0bcabc3740 1 -- 172.21.15.114:0/532617467 shutdown_connections 2024-05-27T07:55:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.974+0000 7f0bcabc3740 1 -- 172.21.15.114:0/532617467 wait complete. 2024-05-27T07:55:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.974+0000 7f0bcabc3740 20 client.19194 trim_cache size 0 max 16384 2024-05-27T07:55:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:55:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:55:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:55:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.975+0000 7f0bc3d0d640 1 -- 172.21.15.114:0/4246017556 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0ba80b2950 msgr2=0x7f0ba80b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.975+0000 7f0bc3d0d640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0ba80b2950 0x7f0ba80b4e60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4011890 tx=0x7f0bb4009500 comp rx=0 tx=0).stop 2024-05-27T07:55:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.975+0000 7f0bc3d0d640 1 -- 172.21.15.114:0/4246017556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc108b80 msgr2=0x7f0bbc1a0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.975+0000 7f0bc3d0d640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc108b80 0x7f0bbc1a0b00 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00e9f0 tx=0x7f0bac00eeb0 comp rx=0 tx=0).stop 2024-05-27T07:55:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.975+0000 7f0bc3d0d640 1 -- 172.21.15.114:0/4246017556 shutdown_connections 2024-05-27T07:55:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.975+0000 7f0bc3d0d640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0ba80b2950 0x7f0ba80b4e60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.975+0000 7f0bc3d0d640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0bbc109020 0x7f0bbc19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.975+0000 7f0bc3d0d640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc108b80 0x7f0bbc1a0b00 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.975+0000 7f0bc3d0d640 1 --2- 172.21.15.114:0/4246017556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0bbc1032d0 0x7f0bbc1a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.975+0000 7f0bc3d0d640 1 -- 172.21.15.114:0/4246017556 >> 172.21.15.114:0/4246017556 conn(0x7f0bbc0fe380 msgr2=0x7f0bbc100c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.975+0000 7f0bc3d0d640 1 -- 172.21.15.114:0/4246017556 shutdown_connections 2024-05-27T07:55:35.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:35.975+0000 7f0bc3d0d640 1 -- 172.21.15.114:0/4246017556 wait complete. 2024-05-27T07:55:35.979 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:35 smithi163 ceph-mon[36451]: pgmap v540: 129 pgs: 129 active+clean; 315 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 518 op/s 2024-05-27T07:55:35.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:35 smithi186 ceph-mon[36647]: pgmap v540: 129 pgs: 129 active+clean; 315 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 518 op/s 2024-05-27T07:55:36.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:35 smithi114 ceph-mon[32038]: pgmap v540: 129 pgs: 129 active+clean; 315 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 518 op/s 2024-05-27T07:55:36.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 25860 14.51 MB/sec warmup 108 sec latency 967.153 ms 2024-05-27T07:55:37.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 26216 14.50 MB/sec warmup 109 sec latency 159.467 ms 2024-05-27T07:55:37.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:37 smithi163 ceph-mon[36451]: pgmap v541: 129 pgs: 129 active+clean; 320 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.0 MiB/s wr, 552 op/s 2024-05-27T07:55:37.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:37 smithi186 ceph-mon[36647]: pgmap v541: 129 pgs: 129 active+clean; 320 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.0 MiB/s wr, 552 op/s 2024-05-27T07:55:38.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:37 smithi114 ceph-mon[32038]: pgmap v541: 129 pgs: 129 active+clean; 320 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.0 MiB/s wr, 552 op/s 2024-05-27T07:55:38.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 26590 14.58 MB/sec warmup 110 sec latency 175.810 ms 2024-05-27T07:55:39.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:38 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:55:39.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:38 smithi163 systemd[1]: Stopped Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:55:39.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:38 smithi163 systemd[1]: Starting Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:55:39.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 podman[74926]: 2024-05-27 07:55:39.116583063 +0000 UTC m=+0.018187715 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:55:39.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 podman[74926]: 2024-05-27 07:55:39.231692739 +0000 UTC m=+0.133297379 container create 2a4673cdcf4f1050084004c76636da15e946448a780ab684ba6d392d551bce78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:55:39.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 26994 14.65 MB/sec warmup 111 sec latency 213.787 ms 2024-05-27T07:55:39.797 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:39 smithi163 ceph-mon[36451]: pgmap v542: 129 pgs: 129 active+clean; 317 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.2 MiB/s wr, 624 op/s 2024-05-27T07:55:39.797 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:39 smithi163 ceph-mon[36451]: daemon mds.k restarted 2024-05-27T07:55:39.797 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 podman[74926]: 2024-05-27 07:55:39.501977065 +0000 UTC m=+0.403581716 container init 2a4673cdcf4f1050084004c76636da15e946448a780ab684ba6d392d551bce78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-27T07:55:39.798 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 podman[74926]: 2024-05-27 07:55:39.508536517 +0000 UTC m=+0.410141169 container start 2a4673cdcf4f1050084004c76636da15e946448a780ab684ba6d392d551bce78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521) 2024-05-27T07:55:39.798 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-mds[74944]: -- 172.21.15.163:0/1838590081 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d7d22da4e0 con 0x55d7d2325a80 2024-05-27T07:55:39.798 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-mds[74944]: -- 172.21.15.163:0/1838590081 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d7d22dda40 con 0x55d7d2325a80 2024-05-27T07:55:39.798 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: starting mds.k at 2024-05-27T07:55:39.798 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 bash[74926]: 2a4673cdcf4f1050084004c76636da15e946448a780ab684ba6d392d551bce78 2024-05-27T07:55:39.798 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 systemd[1]: Started Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:55:39.798 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe500e5f700 time 2024-05-27T07:55:39.796978+0000 2024-05-27T07:55:39.798 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:39.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:39 smithi186 ceph-mon[36647]: pgmap v542: 129 pgs: 129 active+clean; 317 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.2 MiB/s wr, 624 op/s 2024-05-27T07:55:39.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:39 smithi186 ceph-mon[36647]: daemon mds.k restarted 2024-05-27T07:55:40.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:39 smithi114 ceph-mon[32038]: pgmap v542: 129 pgs: 129 active+clean; 317 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.2 MiB/s wr, 624 op/s 2024-05-27T07:55:40.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:39 smithi114 ceph-mon[32038]: daemon mds.k restarted 2024-05-27T07:55:40.227 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:40.227 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5136a4ba6] 2024-05-27T07:55:40.227 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5136a4dd1] 2024-05-27T07:55:40.227 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d7cfce223d] 2024-05-27T07:55:40.227 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d7cfccd971] 2024-05-27T07:55:40.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d7cfccbfa0] 2024-05-27T07:55:40.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d7cfccc651] 2024-05-27T07:55:40.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d7cfc61347] 2024-05-27T07:55:40.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d7cf95d87e] 2024-05-27T07:55:40.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55d7cf962f2f] 2024-05-27T07:55:40.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d7cf96d969] 2024-05-27T07:55:40.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d7cfd0648c] 2024-05-27T07:55:40.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d7cfd0f9cb] 2024-05-27T07:55:40.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 13: (Thread::entry_wrapper()+0x43) [0x7fe51367372d] 2024-05-27T07:55:40.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe513673749] 2024-05-27T07:55:40.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5122171ca] 2024-05-27T07:55:40.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 16: clone() 2024-05-27T07:55:40.228 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:39.797+0000 7fe500e5f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe500e5f700 time 2024-05-27T07:55:39.796978+0000 2024-05-27T07:55:40.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:40.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:40.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:40.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5136a4ba6] 2024-05-27T07:55:40.229 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5136a4dd1] 2024-05-27T07:55:40.230 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d7cfce223d] 2024-05-27T07:55:40.230 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d7cfccd971] 2024-05-27T07:55:40.230 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d7cfccbfa0] 2024-05-27T07:55:40.230 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d7cfccc651] 2024-05-27T07:55:40.230 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d7cfc61347] 2024-05-27T07:55:40.230 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d7cf95d87e] 2024-05-27T07:55:40.230 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55d7cf962f2f] 2024-05-27T07:55:40.230 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d7cf96d969] 2024-05-27T07:55:40.230 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d7cfd0648c] 2024-05-27T07:55:40.230 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d7cfd0f9cb] 2024-05-27T07:55:40.230 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 13: (Thread::entry_wrapper()+0x43) [0x7fe51367372d] 2024-05-27T07:55:40.231 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe513673749] 2024-05-27T07:55:40.231 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5122171ca] 2024-05-27T07:55:40.231 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 16: clone() 2024-05-27T07:55:40.231 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:40.231 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: *** Caught signal (Aborted) ** 2024-05-27T07:55:40.231 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: in thread 7fe500e5f700 thread_name:md_log_replay 2024-05-27T07:55:40.231 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:40.231 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe512221d20] 2024-05-27T07:55:40.231 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2: gsignal() 2024-05-27T07:55:40.231 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 3: abort() 2024-05-27T07:55:40.231 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5136a4ced] 2024-05-27T07:55:40.231 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5136a4dd1] 2024-05-27T07:55:40.231 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d7cfce223d] 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d7cfccd971] 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d7cfccbfa0] 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d7cfccc651] 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d7cfc61347] 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d7cf95d87e] 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55d7cf962f2f] 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d7cf96d969] 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d7cfd0648c] 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d7cfd0f9cb] 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 16: (Thread::entry_wrapper()+0x43) [0x7fe51367372d] 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe513673749] 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5122171ca] 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 19: clone() 2024-05-27T07:55:40.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:39.799+0000 7fe500e5f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: in thread 7fe500e5f700 thread_name:md_log_replay 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe512221d20] 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2: gsignal() 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 3: abort() 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5136a4ced] 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5136a4dd1] 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d7cfce223d] 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d7cfccd971] 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d7cfccbfa0] 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d7cfccc651] 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d7cfc61347] 2024-05-27T07:55:40.233 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d7cf95d87e] 2024-05-27T07:55:40.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55d7cf962f2f] 2024-05-27T07:55:40.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d7cf96d969] 2024-05-27T07:55:40.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d7cfd0648c] 2024-05-27T07:55:40.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d7cfd0f9cb] 2024-05-27T07:55:40.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 16: (Thread::entry_wrapper()+0x43) [0x7fe51367372d] 2024-05-27T07:55:40.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe513673749] 2024-05-27T07:55:40.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5122171ca] 2024-05-27T07:55:40.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 19: clone() 2024-05-27T07:55:40.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:55:40.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:40.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: -1> 2024-05-27T07:55:39.797+0000 7fe500e5f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe500e5f700 time 2024-05-27T07:55:39.796978+0000 2024-05-27T07:55:40.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:40.234 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5136a4ba6] 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5136a4dd1] 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d7cfce223d] 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d7cfccd971] 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d7cfccbfa0] 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d7cfccc651] 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d7cfc61347] 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d7cf95d87e] 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55d7cf962f2f] 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d7cf96d969] 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d7cfd0648c] 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d7cfd0f9cb] 2024-05-27T07:55:40.235 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 13: (Thread::entry_wrapper()+0x43) [0x7fe51367372d] 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe513673749] 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5122171ca] 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 16: clone() 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 0> 2024-05-27T07:55:39.799+0000 7fe500e5f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: in thread 7fe500e5f700 thread_name:md_log_replay 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe512221d20] 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2: gsignal() 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 3: abort() 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5136a4ced] 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5136a4dd1] 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d7cfce223d] 2024-05-27T07:55:40.236 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d7cfccd971] 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d7cfccbfa0] 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d7cfccc651] 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d7cfc61347] 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d7cf95d87e] 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55d7cf962f2f] 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d7cf96d969] 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d7cfd0648c] 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d7cfd0f9cb] 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 16: (Thread::entry_wrapper()+0x43) [0x7fe51367372d] 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe513673749] 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5122171ca] 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 19: clone() 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:55:40.237 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:40.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: -1> 2024-05-27T07:55:39.797+0000 7fe500e5f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe500e5f700 time 2024-05-27T07:55:39.796978+0000 2024-05-27T07:55:40.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:55:40.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:40.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:40.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5136a4ba6] 2024-05-27T07:55:40.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5136a4dd1] 2024-05-27T07:55:40.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d7cfce223d] 2024-05-27T07:55:40.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d7cfccd971] 2024-05-27T07:55:40.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d7cfccbfa0] 2024-05-27T07:55:40.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d7cfccc651] 2024-05-27T07:55:40.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d7cfc61347] 2024-05-27T07:55:40.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d7cf95d87e] 2024-05-27T07:55:40.238 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55d7cf962f2f] 2024-05-27T07:55:40.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d7cf96d969] 2024-05-27T07:55:40.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d7cfd0648c] 2024-05-27T07:55:40.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d7cfd0f9cb] 2024-05-27T07:55:40.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 13: (Thread::entry_wrapper()+0x43) [0x7fe51367372d] 2024-05-27T07:55:40.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe513673749] 2024-05-27T07:55:40.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5122171ca] 2024-05-27T07:55:40.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 16: clone() 2024-05-27T07:55:40.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:40.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 0> 2024-05-27T07:55:39.799+0000 7fe500e5f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:55:40.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: in thread 7fe500e5f700 thread_name:md_log_replay 2024-05-27T07:55:40.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:40.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:55:40.239 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe512221d20] 2024-05-27T07:55:40.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2: gsignal() 2024-05-27T07:55:40.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 3: abort() 2024-05-27T07:55:40.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5136a4ced] 2024-05-27T07:55:40.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5136a4dd1] 2024-05-27T07:55:40.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d7cfce223d] 2024-05-27T07:55:40.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d7cfccd971] 2024-05-27T07:55:40.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d7cfccbfa0] 2024-05-27T07:55:40.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d7cfccc651] 2024-05-27T07:55:40.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d7cfc61347] 2024-05-27T07:55:40.240 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d7cf95d87e] 2024-05-27T07:55:40.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55d7cf962f2f] 2024-05-27T07:55:40.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d7cf96d969] 2024-05-27T07:55:40.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d7cfd0648c] 2024-05-27T07:55:40.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d7cfd0f9cb] 2024-05-27T07:55:40.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 16: (Thread::entry_wrapper()+0x43) [0x7fe51367372d] 2024-05-27T07:55:40.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe513673749] 2024-05-27T07:55:40.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5122171ca] 2024-05-27T07:55:40.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 19: clone() 2024-05-27T07:55:40.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:55:40.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[74940]: 2024-05-27T07:55:40.241 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:39 smithi163 podman[74975]: 2024-05-27 07:55:39.864962819 +0000 UTC m=+0.024630761 container died 2a4673cdcf4f1050084004c76636da15e946448a780ab684ba6d392d551bce78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T07:55:40.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 27406 14.77 MB/sec warmup 112 sec latency 180.103 ms 2024-05-27T07:55:40.777 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:40 smithi163 ceph-mon[36451]: osdmap e145: 12 total, 12 up, 12 in 2024-05-27T07:55:40.777 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:40 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6832/1565612474,v1:172.21.15.163:6833/1565612474] up:boot 2024-05-27T07:55:40.777 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:40 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:55:40.777 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:40 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:55:40.777 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:40 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:55:40.778 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:40 smithi163 podman[74975]: 2024-05-27 07:55:40.51673664 +0000 UTC m=+0.676404553 container remove 2a4673cdcf4f1050084004c76636da15e946448a780ab684ba6d392d551bce78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:55:40.778 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:40 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:55:40.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:40 smithi186 ceph-mon[36647]: osdmap e145: 12 total, 12 up, 12 in 2024-05-27T07:55:40.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:40 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6832/1565612474,v1:172.21.15.163:6833/1565612474] up:boot 2024-05-27T07:55:40.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:40 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:55:40.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:40 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:55:40.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:40 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:55:41.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:40 smithi114 ceph-mon[32038]: osdmap e145: 12 total, 12 up, 12 in 2024-05-27T07:55:41.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:40 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6832/1565612474,v1:172.21.15.163:6833/1565612474] up:boot 2024-05-27T07:55:41.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:40 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:55:41.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:40 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:55:41.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:40 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:55:41.226 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:40 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:55:41.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 27847 14.83 MB/sec warmup 113 sec latency 169.851 ms 2024-05-27T07:55:41.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:41 smithi163 ceph-mon[36451]: pgmap v544: 129 pgs: 129 active+clean; 306 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 596 op/s 2024-05-27T07:55:41.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:41 smithi186 ceph-mon[36647]: pgmap v544: 129 pgs: 129 active+clean; 306 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 596 op/s 2024-05-27T07:55:42.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:41 smithi114 ceph-mon[32038]: pgmap v544: 129 pgs: 129 active+clean; 306 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 596 op/s 2024-05-27T07:55:42.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 28145 14.78 MB/sec warmup 114 sec latency 250.461 ms 2024-05-27T07:55:43.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 28426 14.81 MB/sec warmup 115 sec latency 197.792 ms 2024-05-27T07:55:43.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:43 smithi163 ceph-mon[36451]: pgmap v545: 129 pgs: 129 active+clean; 311 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 539 op/s 2024-05-27T07:55:43.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:43 smithi186 ceph-mon[36647]: pgmap v545: 129 pgs: 129 active+clean; 311 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 539 op/s 2024-05-27T07:55:44.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:43 smithi114 ceph-mon[32038]: pgmap v545: 129 pgs: 129 active+clean; 311 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 539 op/s 2024-05-27T07:55:44.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 28800 14.90 MB/sec warmup 116 sec latency 67.928 ms 2024-05-27T07:55:45.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 29222 14.98 MB/sec warmup 117 sec latency 146.691 ms 2024-05-27T07:55:45.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:45 smithi163 ceph-mon[36451]: pgmap v546: 129 pgs: 129 active+clean; 311 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 519 op/s 2024-05-27T07:55:45.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:45 smithi186 ceph-mon[36647]: pgmap v546: 129 pgs: 129 active+clean; 311 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 519 op/s 2024-05-27T07:55:46.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:45 smithi114 ceph-mon[32038]: pgmap v546: 129 pgs: 129 active+clean; 311 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 519 op/s 2024-05-27T07:55:46.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 29598 14.97 MB/sec warmup 118 sec latency 158.457 ms 2024-05-27T07:55:47.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 29981 15.01 MB/sec warmup 119 sec latency 184.629 ms 2024-05-27T07:55:47.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:47 smithi163 ceph-mon[36451]: pgmap v547: 129 pgs: 129 active+clean; 320 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 328 op/s 2024-05-27T07:55:47.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:47 smithi186 ceph-mon[36647]: pgmap v547: 129 pgs: 129 active+clean; 320 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 328 op/s 2024-05-27T07:55:48.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:47 smithi114 ceph-mon[32038]: pgmap v547: 129 pgs: 129 active+clean; 320 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 328 op/s 2024-05-27T07:55:49.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.098+0000 7f1564092640 1 -- 172.21.15.114:0/299923226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c10e1f0 msgr2=0x7f155c1106e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:49.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.098+0000 7f1564092640 1 --2- 172.21.15.114:0/299923226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c10e1f0 0x7f155c1106e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1554009be0 tx=0x7f155402a6c0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.099+0000 7f1564092640 1 -- 172.21.15.114:0/299923226 shutdown_connections 2024-05-27T07:55:49.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.099+0000 7f1564092640 1 --2- 172.21.15.114:0/299923226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c10e1f0 0x7f155c1106e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.099+0000 7f1564092640 1 --2- 172.21.15.114:0/299923226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155c10b800 0x7f155c10dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.099+0000 7f1564092640 1 --2- 172.21.15.114:0/299923226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f155c104da0 0x7f155c1051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.099+0000 7f1564092640 1 -- 172.21.15.114:0/299923226 >> 172.21.15.114:0/299923226 conn(0x7f155c078270 msgr2=0x7f155c0786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:49.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.099+0000 7f1564092640 1 -- 172.21.15.114:0/299923226 shutdown_connections 2024-05-27T07:55:49.101 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.099+0000 7f1564092640 1 -- 172.21.15.114:0/299923226 wait complete. 2024-05-27T07:55:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.099+0000 7f1564092640 1 Processor -- start 2024-05-27T07:55:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1564092640 1 -- start start 2024-05-27T07:55:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1564092640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c104da0 0x7f155c1a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1564092640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155c10b800 0x7f155c1a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1564092640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f155c10e1f0 0x7f155c19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1564092640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f155c113e50 con 0x7f155c10b800 2024-05-27T07:55:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1564092640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f155c113cd0 con 0x7f155c10e1f0 2024-05-27T07:55:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1564092640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f155c113fd0 con 0x7f155c104da0 2024-05-27T07:55:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1561e2e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c104da0 0x7f155c1a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f156162d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155c10b800 0x7f155c1a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1561e2e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c104da0 0x7f155c1a04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:56332/0 (socket says 172.21.15.114:56332) 2024-05-27T07:55:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f156162d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155c10b800 0x7f155c1a0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54312/0 (socket says 172.21.15.114:54312) 2024-05-27T07:55:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1561e2e640 1 -- 172.21.15.114:0/2179815878 learned_addr learned my addr 172.21.15.114:0/2179815878 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f156262f640 1 --2- 172.21.15.114:0/2179815878 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f155c10e1f0 0x7f155c19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1561e2e640 1 -- 172.21.15.114:0/2179815878 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f155c10e1f0 msgr2=0x7f155c19a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1561e2e640 1 --2- 172.21.15.114:0/2179815878 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f155c10e1f0 0x7f155c19a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1561e2e640 1 -- 172.21.15.114:0/2179815878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155c10b800 msgr2=0x7f155c1a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1561e2e640 1 --2- 172.21.15.114:0/2179815878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155c10b800 0x7f155c1a0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.100+0000 7f1561e2e640 1 -- 172.21.15.114:0/2179815878 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f155c19ada0 con 0x7f155c104da0 2024-05-27T07:55:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.101+0000 7f1561e2e640 1 --2- 172.21.15.114:0/2179815878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c104da0 0x7f155c1a04f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f155800e9f0 tx=0x7f155800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.101+0000 7f1552ffd640 1 -- 172.21.15.114:0/2179815878 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155800cd80 con 0x7f155c104da0 2024-05-27T07:55:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.101+0000 7f1552ffd640 1 -- 172.21.15.114:0/2179815878 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15580054d0 con 0x7f155c104da0 2024-05-27T07:55:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.101+0000 7f1564092640 1 -- 172.21.15.114:0/2179815878 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f155c19b030 con 0x7f155c104da0 2024-05-27T07:55:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.101+0000 7f1552ffd640 1 -- 172.21.15.114:0/2179815878 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1558010690 con 0x7f155c104da0 2024-05-27T07:55:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.101+0000 7f1564092640 1 -- 172.21.15.114:0/2179815878 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f155c1a7290 con 0x7f155c104da0 2024-05-27T07:55:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.104+0000 7f156b01d740 1 Processor -- start 2024-05-27T07:55:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.104+0000 7f156b01d740 1 -- start start 2024-05-27T07:55:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.104+0000 7f156b01d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f7d1b0 0x557b9231e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156b01d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f542f0 0x557b9231e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156b01d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f5c4d0 0x557b9231ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156b01d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557b92290c30 con 0x557b91f542f0 2024-05-27T07:55:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156b01d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557b922807d0 con 0x557b91f7d1b0 2024-05-27T07:55:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156b01d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557b92284810 con 0x557b91f5c4d0 2024-05-27T07:55:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156262f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f7d1b0 0x557b9231e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156262f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f7d1b0 0x557b9231e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:33740/0 (socket says 172.21.15.114:33740) 2024-05-27T07:55:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156262f640 1 -- 172.21.15.114:0/1529852347 learned_addr learned my addr 172.21.15.114:0/1529852347 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156162d640 1 --2- 172.21.15.114:0/1529852347 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f5c4d0 0x557b9231ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f1552ffd640 1 -- 172.21.15.114:0/2179815878 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f1558004e90 con 0x7f155c104da0 2024-05-27T07:55:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f1561e2e640 1 --2- 172.21.15.114:0/1529852347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f542f0 0x557b9231e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156262f640 1 -- 172.21.15.114:0/1529852347 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f5c4d0 msgr2=0x557b9231ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156262f640 1 --2- 172.21.15.114:0/1529852347 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f5c4d0 0x557b9231ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156262f640 1 -- 172.21.15.114:0/1529852347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f542f0 msgr2=0x557b9231e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156262f640 1 --2- 172.21.15.114:0/1529852347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f542f0 0x557b9231e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156262f640 1 -- 172.21.15.114:0/1529852347 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b91f95210 con 0x557b91f7d1b0 2024-05-27T07:55:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f1561e2e640 1 --2- 172.21.15.114:0/1529852347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f542f0 0x557b9231e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:55:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.105+0000 7f156262f640 1 --2- 172.21.15.114:0/1529852347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f7d1b0 0x557b9231e3d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f155402acb0 tx=0x7f1554030420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73e5fe6831acc2d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.106+0000 7f1547fff640 1 -- 172.21.15.114:0/1529852347 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554041030 con 0x557b91f7d1b0 2024-05-27T07:55:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.106+0000 7f1552ffd640 1 --2- 172.21.15.114:0/2179815878 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f15480b2a50 0x7f15480b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.106+0000 7f1547fff640 1 -- 172.21.15.114:0/1529852347 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1554003e20 con 0x557b91f7d1b0 2024-05-27T07:55:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.106+0000 7f1547fff640 1 -- 172.21.15.114:0/1529852347 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554036040 con 0x557b91f7d1b0 2024-05-27T07:55:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.106+0000 7f1552ffd640 1 -- 172.21.15.114:0/2179815878 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 9466+0+0 (secure 0 0 0) 0x7f15580e2da0 con 0x7f155c104da0 2024-05-27T07:55:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.106+0000 7f1561e2e640 1 --2- 172.21.15.114:0/2179815878 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f15480b2a50 0x7f15480b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.107+0000 7f156b01d740 1 -- 172.21.15.114:0/1529852347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f7d1b0 msgr2=0x557b9231e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.107+0000 7f1561e2e640 1 --2- 172.21.15.114:0/2179815878 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f15480b2a50 0x7f15480b4f60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x557b9231bec0 tx=0x7f1558004a90 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.107+0000 7f156b01d740 1 --2- 172.21.15.114:0/1529852347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f7d1b0 0x557b9231e3d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f155402acb0 tx=0x7f1554030420 comp rx=0 tx=0).stop 2024-05-27T07:55:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.107+0000 7f156b01d740 1 -- 172.21.15.114:0/1529852347 shutdown_connections 2024-05-27T07:55:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.107+0000 7f156b01d740 1 --2- 172.21.15.114:0/1529852347 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f5c4d0 0x557b9231ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.107+0000 7f156b01d740 1 --2- 172.21.15.114:0/1529852347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f542f0 0x557b9231e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.107+0000 7f156b01d740 1 --2- 172.21.15.114:0/1529852347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f7d1b0 0x557b9231e3d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.107+0000 7f156b01d740 1 -- 172.21.15.114:0/1529852347 >> 172.21.15.114:0/1529852347 conn(0x557b91f71300 msgr2=0x557b9231ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.108+0000 7f156b01d740 1 -- 172.21.15.114:0/1529852347 shutdown_connections 2024-05-27T07:55:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.108+0000 7f156b01d740 1 -- 172.21.15.114:0/1529852347 wait complete. 2024-05-27T07:55:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.108+0000 7f156b01d740 1 -- start start 2024-05-27T07:55:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.108+0000 7f156b01d740 1 Processor -- start 2024-05-27T07:55:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.109+0000 7f156b01d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 0x557b92328a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.109+0000 7f156b01d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f5c4d0 0x557b92328fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.109+0000 7f156b01d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f7d1b0 0x557b9232e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.109+0000 7f156262f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 0x557b92328a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.109+0000 7f156262f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 0x557b92328a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:33748/0 (socket says 172.21.15.114:33748) 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.110+0000 7f156162d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f5c4d0 0x557b92328fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.110+0000 7f156162d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f5c4d0 0x557b92328fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:56350/0 (socket says 172.21.15.114:56350) 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.110+0000 7f156262f640 1 -- 172.21.15.114:0/498304078 learned_addr learned my addr 172.21.15.114:0/498304078 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.110+0000 7f156b01d740 1 -- 172.21.15.114:0/498304078 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557b91f66790 con 0x557b91f7d1b0 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.110+0000 7f156b01d740 1 -- 172.21.15.114:0/498304078 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557b9223c890 con 0x557b91f542f0 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.110+0000 7f156b01d740 1 -- 172.21.15.114:0/498304078 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557b9232b4f0 con 0x557b91f5c4d0 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.110+0000 7f156262f640 1 --2- 172.21.15.114:0/498304078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f7d1b0 0x557b9232e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.110+0000 7f156262f640 1 -- 172.21.15.114:0/498304078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f5c4d0 msgr2=0x557b92328fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.110+0000 7f156262f640 1 --2- 172.21.15.114:0/498304078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f5c4d0 0x557b92328fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.110+0000 7f156262f640 1 -- 172.21.15.114:0/498304078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f7d1b0 msgr2=0x557b9232e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.110+0000 7f156262f640 1 --2- 172.21.15.114:0/498304078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f7d1b0 0x557b9232e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.110+0000 7f156262f640 1 -- 172.21.15.114:0/498304078 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b92061620 con 0x557b91f542f0 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.110+0000 7f156162d640 1 --2- 172.21.15.114:0/498304078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f5c4d0 0x557b92328fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.111+0000 7f156262f640 1 --2- 172.21.15.114:0/498304078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 0x557b92328a80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f15540099a0 tx=0x7f155402a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf32f356f743412c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.111+0000 7f156b01d740 10 client.0 _ll_register_callbacks cb 0x557b91f94a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.111+0000 7f15457fa640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.111+0000 7f156b01d740 1 -- 172.21.15.114:0/498304078 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1554003e20 con 0x557b91f542f0 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.111+0000 7f15457fa640 1 -- 172.21.15.114:0/498304078 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554041040 con 0x557b91f542f0 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.111+0000 7f15457fa640 1 -- 172.21.15.114:0/498304078 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9466+0+0 (secure 0 0 0) 0x7f1554030970 con 0x557b91f542f0 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.111+0000 7f15457fa640 1 -- 172.21.15.114:0/498304078 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f155405e050 con 0x557b91f542f0 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.111+0000 7f156b01d740 10 client.28544 fetch_fsmap learned FSMap version 89 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.111+0000 7f156b01d740 1 -- 172.21.15.114:0/498304078 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({fsmap=89}) -- 0x557b92337340 con 0x557b91f542f0 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.112+0000 7f15457fa640 1 -- 172.21.15.114:0/498304078 <== mon.2 v2:172.21.15.163:3300/0 4 ==== fsmap(e 89) ==== 4700+0+0 (secure 0 0 0) 0x7f155404a820 con 0x557b91f542f0 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.112+0000 7f156b01d740 10 client.28544 fetch_fsmap finished waiting for FSMap version 89 2024-05-27T07:55:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.112+0000 7f156b01d740 10 client.28544 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.112+0000 7f156b01d740 20 client.28544 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.112+0000 7f156b01d740 1 --2- 172.21.15.114:0/498304078 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b91f3bd80 0x557b9233aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.112+0000 7f156b01d740 4 client.28544 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:55:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.112+0000 7f156b01d740 1 -- 172.21.15.114:0/498304078 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b92292b80 con 0x557b91f3bd80 2024-05-27T07:55:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.112+0000 7f156162d640 1 --2- 172.21.15.114:0/498304078 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b91f3bd80 0x557b9233aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.113+0000 7f156162d640 1 --2- 172.21.15.114:0/498304078 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b91f3bd80 0x557b9233aaa0 crc :-1 s=READY pgs=444 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9703387e4d2873 server_cookie=c0521a109b155008 in_seq=0 out_seq=0 2024-05-27T07:55:49.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.113+0000 7f15457fa640 10 client.28544 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:49.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.116+0000 7f15457fa640 1 -- 172.21.15.114:0/498304078 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557b92292b80 con 0x557b91f3bd80 2024-05-27T07:55:49.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.116+0000 7f15457fa640 10 client.28544 handle_command_reply: tid=0 2024-05-27T07:55:49.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.116+0000 7f156b01d740 1 client.28544 shutdown 2024-05-27T07:55:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.117+0000 7f156b01d740 1 -- 172.21.15.114:0/498304078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 msgr2=0x557b92328a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.117+0000 7f156b01d740 1 --2- 172.21.15.114:0/498304078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 0x557b92328a80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f15540099a0 tx=0x7f155402a6c0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.117+0000 7f156b01d740 1 -- 172.21.15.114:0/498304078 shutdown_connections 2024-05-27T07:55:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.117+0000 7f156b01d740 1 --2- 172.21.15.114:0/498304078 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b91f3bd80 0x557b9233aaa0 crc :-1 s=READY pgs=444 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.117+0000 7f15457fa640 0 client.28544 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.117+0000 7f156b01d740 1 --2- 172.21.15.114:0/498304078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f7d1b0 0x557b9232e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.117+0000 7f156b01d740 1 --2- 172.21.15.114:0/498304078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f5c4d0 0x557b92328fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.117+0000 7f156b01d740 1 --2- 172.21.15.114:0/498304078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 0x557b92328a80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.117+0000 7f156b01d740 1 -- 172.21.15.114:0/498304078 >> 172.21.15.114:0/498304078 conn(0x557b91f71300 msgr2=0x557b9231e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.117+0000 7f156b01d740 1 -- 172.21.15.114:0/498304078 shutdown_connections 2024-05-27T07:55:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.117+0000 7f156b01d740 1 -- 172.21.15.114:0/498304078 wait complete. 2024-05-27T07:55:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.117+0000 7f156b01d740 20 client.28544 trim_cache size 0 max 16384 2024-05-27T07:55:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156b01d740 1 Processor -- start 2024-05-27T07:55:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156b01d740 1 -- start start 2024-05-27T07:55:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156b01d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f3bd80 0x557b9231e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156b01d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f542f0 0x557b9232e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156b01d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f5c4d0 0x557b9232ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156b01d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f155c1af400 con 0x557b91f3bd80 2024-05-27T07:55:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156b01d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f155c1afad0 con 0x557b91f5c4d0 2024-05-27T07:55:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156b01d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f155c1ae320 con 0x557b91f542f0 2024-05-27T07:55:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156262f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f3bd80 0x557b9231e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156262f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f3bd80 0x557b9231e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54324/0 (socket says 172.21.15.114:54324) 2024-05-27T07:55:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156262f640 1 -- 172.21.15.114:0/3452812404 learned_addr learned my addr 172.21.15.114:0/3452812404 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156162d640 1 --2- 172.21.15.114:0/3452812404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f542f0 0x557b9232e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156262f640 1 --2- 172.21.15.114:0/3452812404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f5c4d0 0x557b9232ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156262f640 1 -- 172.21.15.114:0/3452812404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f542f0 msgr2=0x557b9232e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156262f640 1 --2- 172.21.15.114:0/3452812404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f542f0 0x557b9232e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156262f640 1 -- 172.21.15.114:0/3452812404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f5c4d0 msgr2=0x557b9232ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156262f640 1 --2- 172.21.15.114:0/3452812404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f5c4d0 0x557b9232ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.135+0000 7f156262f640 1 -- 172.21.15.114:0/3452812404 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b92329260 con 0x557b91f3bd80 2024-05-27T07:55:49.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.136+0000 7f156262f640 1 --2- 172.21.15.114:0/3452812404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f3bd80 0x557b9231e870 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f155402a700 tx=0x7f155404f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7490b5a2bec8a9ee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:49.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.136+0000 7f15477fe640 1 -- 172.21.15.114:0/3452812404 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554041040 con 0x557b91f3bd80 2024-05-27T07:55:49.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.136+0000 7f15477fe640 1 -- 172.21.15.114:0/3452812404 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1554003e20 con 0x557b91f3bd80 2024-05-27T07:55:49.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.136+0000 7f15477fe640 1 -- 172.21.15.114:0/3452812404 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554036040 con 0x557b91f3bd80 2024-05-27T07:55:49.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.137+0000 7f156b01d740 1 -- 172.21.15.114:0/3452812404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f3bd80 msgr2=0x557b9231e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:49.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.137+0000 7f156b01d740 1 --2- 172.21.15.114:0/3452812404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f3bd80 0x557b9231e870 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f155402a700 tx=0x7f155404f070 comp rx=0 tx=0).stop 2024-05-27T07:55:49.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.137+0000 7f156b01d740 1 -- 172.21.15.114:0/3452812404 shutdown_connections 2024-05-27T07:55:49.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.137+0000 7f156b01d740 1 --2- 172.21.15.114:0/3452812404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f5c4d0 0x557b9232ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.137+0000 7f156b01d740 1 --2- 172.21.15.114:0/3452812404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f542f0 0x557b9232e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.137+0000 7f156b01d740 1 --2- 172.21.15.114:0/3452812404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f3bd80 0x557b9231e870 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.137+0000 7f156b01d740 1 -- 172.21.15.114:0/3452812404 >> 172.21.15.114:0/3452812404 conn(0x557b91f71300 msgr2=0x557b9231e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.137+0000 7f156b01d740 1 -- 172.21.15.114:0/3452812404 shutdown_connections 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.137+0000 7f156b01d740 1 -- 172.21.15.114:0/3452812404 wait complete. 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.137+0000 7f156b01d740 1 -- start start 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.137+0000 7f156b01d740 1 Processor -- start 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156b01d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f3bd80 0x557b92335180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156b01d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 0x557b923314e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156b01d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f5c4d0 0x557b92331a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156b01d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f155c1ad580 con 0x557b91f5c4d0 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156b01d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f155c1acf10 con 0x557b91f542f0 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156b01d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f155c1adc50 con 0x557b91f3bd80 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156b01d740 10 client.0 _ll_register_callbacks cb 0x7f154c009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:55:49.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156262f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f5c4d0 0x557b92331a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156162d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 0x557b923314e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156262f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f3bd80 0x557b92335180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156162d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 0x557b923314e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:33776/0 (socket says 172.21.15.114:33776) 2024-05-27T07:55:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156162d640 1 -- 172.21.15.114:0/1537633242 learned_addr learned my addr 172.21.15.114:0/1537633242 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:55:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156262f640 1 -- 172.21.15.114:0/1537633242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 msgr2=0x557b923314e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156262f640 1 --2- 172.21.15.114:0/1537633242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 0x557b923314e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156262f640 1 -- 172.21.15.114:0/1537633242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f5c4d0 msgr2=0x557b92331a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:55:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156262f640 1 --2- 172.21.15.114:0/1537633242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f5c4d0 0x557b92331a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156262f640 1 -- 172.21.15.114:0/1537633242 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b92332060 con 0x557b91f3bd80 2024-05-27T07:55:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.138+0000 7f156162d640 1 --2- 172.21.15.114:0/1537633242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 0x557b923314e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:55:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f156262f640 1 --2- 172.21.15.114:0/1537633242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f3bd80 0x557b92335180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1554002230 tx=0x7f1554000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7801c99f9adcb85 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:55:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f15457fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:55:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f15457fa640 1 -- 172.21.15.114:0/1537633242 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554041040 con 0x557b91f3bd80 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f156b01d740 1 -- 172.21.15.114:0/1537633242 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1554003e20 con 0x557b91f3bd80 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f15457fa640 1 -- 172.21.15.114:0/1537633242 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9466+0+0 (secure 0 0 0) 0x7f1554004b40 con 0x557b91f3bd80 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f15457fa640 1 -- 172.21.15.114:0/1537633242 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f1554036040 con 0x557b91f3bd80 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f156b01d740 10 client.28696 fetch_fsmap learned FSMap version 89 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f156b01d740 1 -- 172.21.15.114:0/1537633242 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=4+}) -- 0x557b923a1f70 con 0x557b91f3bd80 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f15457fa640 1 -- 172.21.15.114:0/1537633242 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1554003e20 con 0x557b91f3bd80 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f15457fa640 1 -- 172.21.15.114:0/1537633242 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 89) ==== 4700+0+0 (secure 0 0 0) 0x7f1554063040 con 0x557b91f3bd80 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f156b01d740 10 client.28696 fetch_fsmap finished waiting for FSMap version 89 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f156b01d740 10 client.28696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f156b01d740 20 client.28696 populate_metadata read hostname 'smithi114' 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f156b01d740 1 --2- 172.21.15.114:0/1537633242 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b91f7d1b0 0x557b923a4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f156b01d740 4 client.28696 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.139+0000 7f156b01d740 1 -- 172.21.15.114:0/1537633242 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557b92324450 con 0x557b91f7d1b0 2024-05-27T07:55:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.140+0000 7f156162d640 1 --2- 172.21.15.114:0/1537633242 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b91f7d1b0 0x557b923a4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:55:49.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.140+0000 7f15457fa640 10 client.28696 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:49.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:49.140+0000 7f156162d640 1 --2- 172.21.15.114:0/1537633242 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b91f7d1b0 0x557b923a4650 crc :-1 s=READY pgs=445 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2aa46fab78adb91d server_cookie=f5ce9ad8c6f99c00 in_seq=0 out_seq=0 2024-05-27T07:55:49.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 30523 8.92 MB/sec execute 1 sec latency 351.427 ms 2024-05-27T07:55:49.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:49 smithi163 ceph-mon[36451]: pgmap v548: 129 pgs: 129 active+clean; 322 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 200 op/s 2024-05-27T07:55:49.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:49 smithi186 ceph-mon[36647]: pgmap v548: 129 pgs: 129 active+clean; 322 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 200 op/s 2024-05-27T07:55:50.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:49 smithi114 ceph-mon[32038]: pgmap v548: 129 pgs: 129 active+clean; 322 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 200 op/s 2024-05-27T07:55:50.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 30533 4.58 MB/sec execute 2 sec latency 1293.590 ms 2024-05-27T07:55:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.609+0000 7f15457fa640 1 -- 172.21.15.114:0/1537633242 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x557b92324450 con 0x557b91f7d1b0 2024-05-27T07:55:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.609+0000 7f15457fa640 10 client.28696 handle_command_reply: tid=0 2024-05-27T07:55:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.609+0000 7f156b01d740 1 client.28696 shutdown 2024-05-27T07:55:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.609+0000 7f156b01d740 1 -- 172.21.15.114:0/1537633242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f3bd80 msgr2=0x557b92335180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.609+0000 7f156b01d740 1 --2- 172.21.15.114:0/1537633242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f3bd80 0x557b92335180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1554002230 tx=0x7f1554000f40 comp rx=0 tx=0).stop 2024-05-27T07:55:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.610+0000 7f156b01d740 1 -- 172.21.15.114:0/1537633242 shutdown_connections 2024-05-27T07:55:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.610+0000 7f156b01d740 1 --2- 172.21.15.114:0/1537633242 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557b91f7d1b0 0x557b923a4650 crc :-1 s=READY pgs=445 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.610+0000 7f15457fa640 0 client.28696 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:55:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.610+0000 7f156b01d740 1 --2- 172.21.15.114:0/1537633242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557b91f5c4d0 0x557b92331a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.610+0000 7f156b01d740 1 --2- 172.21.15.114:0/1537633242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557b91f542f0 0x557b923314e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.610+0000 7f156b01d740 1 --2- 172.21.15.114:0/1537633242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557b91f3bd80 0x557b92335180 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.610+0000 7f156b01d740 1 -- 172.21.15.114:0/1537633242 >> 172.21.15.114:0/1537633242 conn(0x557b91f71300 msgr2=0x557b92327010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.610+0000 7f156b01d740 1 -- 172.21.15.114:0/1537633242 shutdown_connections 2024-05-27T07:55:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.610+0000 7f156b01d740 1 -- 172.21.15.114:0/1537633242 wait complete. 2024-05-27T07:55:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.610+0000 7f156b01d740 20 client.28696 trim_cache size 0 max 16384 2024-05-27T07:55:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:55:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:55:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:55:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.612+0000 7f1564092640 1 -- 172.21.15.114:0/2179815878 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f15480b2a50 msgr2=0x7f15480b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.612+0000 7f1564092640 1 --2- 172.21.15.114:0/2179815878 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f15480b2a50 0x7f15480b4f60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x557b9231bec0 tx=0x7f1558004a90 comp rx=0 tx=0).stop 2024-05-27T07:55:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.612+0000 7f1564092640 1 -- 172.21.15.114:0/2179815878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c104da0 msgr2=0x7f155c1a04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:55:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.612+0000 7f1564092640 1 --2- 172.21.15.114:0/2179815878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c104da0 0x7f155c1a04f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f155800e9f0 tx=0x7f155800edf0 comp rx=0 tx=0).stop 2024-05-27T07:55:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.612+0000 7f1564092640 1 -- 172.21.15.114:0/2179815878 shutdown_connections 2024-05-27T07:55:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.612+0000 7f1564092640 1 --2- 172.21.15.114:0/2179815878 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f15480b2a50 0x7f15480b4f60 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.612+0000 7f1564092640 1 --2- 172.21.15.114:0/2179815878 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f155c10e1f0 0x7f155c19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.612+0000 7f1564092640 1 --2- 172.21.15.114:0/2179815878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155c10b800 0x7f155c1a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.612+0000 7f1564092640 1 --2- 172.21.15.114:0/2179815878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c104da0 0x7f155c1a04f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:55:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.612+0000 7f1564092640 1 -- 172.21.15.114:0/2179815878 >> 172.21.15.114:0/2179815878 conn(0x7f155c078270 msgr2=0x7f155c101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:55:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.612+0000 7f1564092640 1 -- 172.21.15.114:0/2179815878 shutdown_connections 2024-05-27T07:55:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:55:50.612+0000 7f1564092640 1 -- 172.21.15.114:0/2179815878 wait complete. 2024-05-27T07:55:51.349 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:50 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:55:51.349 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:50 smithi163 systemd[1]: Stopped Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:55:51.349 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:50 smithi163 systemd[1]: Starting Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:55:51.349 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:51 smithi163 podman[75068]: 2024-05-27 07:55:51.106258478 +0000 UTC m=+0.018285665 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:55:51.350 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:51 smithi163 podman[75068]: 2024-05-27 07:55:51.225976792 +0000 UTC m=+0.138003982 container create dc1745274d236c785761b5d488cc69a8b7a6c81675634a946fa1b2ae7e76c376 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True) 2024-05-27T07:55:51.598 INFO:tasks.workunit.client.0.smithi114.stdout: 10 30946 12.38 MB/sec execute 3 sec latency 1253.206 ms 2024-05-27T07:55:51.727 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:51 smithi163 podman[75068]: 2024-05-27 07:55:51.47948976 +0000 UTC m=+0.391516940 container init dc1745274d236c785761b5d488cc69a8b7a6c81675634a946fa1b2ae7e76c376 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:55:51.727 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:51 smithi163 podman[75068]: 2024-05-27 07:55:51.48290966 +0000 UTC m=+0.394936848 container start dc1745274d236c785761b5d488cc69a8b7a6c81675634a946fa1b2ae7e76c376 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-27T07:55:51.727 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:51 smithi163 ceph-mds[75088]: -- 172.21.15.163:0/291833627 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560fee5e5a40 con 0x560fee62d600 2024-05-27T07:55:51.727 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:51 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: starting mds.k at 2024-05-27T07:55:51.727 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:51 smithi163 bash[75068]: dc1745274d236c785761b5d488cc69a8b7a6c81675634a946fa1b2ae7e76c376 2024-05-27T07:55:51.727 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:55:51 smithi163 systemd[1]: Started Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:55:52.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:51 smithi114 ceph-mon[32038]: pgmap v549: 129 pgs: 129 active+clean; 327 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 312 op/s 2024-05-27T07:55:52.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:51 smithi114 ceph-mon[32038]: daemon mds.k restarted 2024-05-27T07:55:52.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:51 smithi163 ceph-mon[36451]: pgmap v549: 129 pgs: 129 active+clean; 327 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 312 op/s 2024-05-27T07:55:52.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:51 smithi163 ceph-mon[36451]: daemon mds.k restarted 2024-05-27T07:55:52.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:51 smithi186 ceph-mon[36647]: pgmap v549: 129 pgs: 129 active+clean; 327 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 312 op/s 2024-05-27T07:55:52.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:51 smithi186 ceph-mon[36647]: daemon mds.k restarted 2024-05-27T07:55:52.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 31318 13.38 MB/sec execute 4 sec latency 138.190 ms 2024-05-27T07:55:53.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:52 smithi114 ceph-mon[32038]: osdmap e146: 12 total, 12 up, 12 in 2024-05-27T07:55:53.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:52 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6832/2578326696,v1:172.21.15.163:6833/2578326696] up:boot 2024-05-27T07:55:53.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:52 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:55:53.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:52 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:55:53.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:52 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:55:53.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:52 smithi163 ceph-mon[36451]: osdmap e146: 12 total, 12 up, 12 in 2024-05-27T07:55:53.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:52 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6832/2578326696,v1:172.21.15.163:6833/2578326696] up:boot 2024-05-27T07:55:53.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:52 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:55:53.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:52 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:55:53.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:52 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:55:53.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:52 smithi186 ceph-mon[36647]: osdmap e146: 12 total, 12 up, 12 in 2024-05-27T07:55:53.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:52 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6832/2578326696,v1:172.21.15.163:6833/2578326696] up:boot 2024-05-27T07:55:53.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:52 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:55:53.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:52 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:55:53.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:52 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:55:53.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 31592 12.89 MB/sec execute 5 sec latency 172.686 ms 2024-05-27T07:55:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:53 smithi114 ceph-mon[32038]: pgmap v551: 129 pgs: 129 active+clean; 321 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 454 op/s 2024-05-27T07:55:54.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:53 smithi163 ceph-mon[36451]: pgmap v551: 129 pgs: 129 active+clean; 321 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 454 op/s 2024-05-27T07:55:54.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:53 smithi186 ceph-mon[36647]: pgmap v551: 129 pgs: 129 active+clean; 321 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 454 op/s 2024-05-27T07:55:54.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 31789 12.60 MB/sec execute 6 sec latency 406.120 ms 2024-05-27T07:55:55.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 32026 12.66 MB/sec execute 7 sec latency 169.849 ms 2024-05-27T07:55:56.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:55 smithi114 ceph-mon[32038]: pgmap v552: 129 pgs: 129 active+clean; 317 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 530 op/s 2024-05-27T07:55:56.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:55 smithi163 ceph-mon[36451]: pgmap v552: 129 pgs: 129 active+clean; 317 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 530 op/s 2024-05-27T07:55:56.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:55 smithi186 ceph-mon[36647]: pgmap v552: 129 pgs: 129 active+clean; 317 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 530 op/s 2024-05-27T07:55:56.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 32435 14.09 MB/sec execute 8 sec latency 185.556 ms 2024-05-27T07:55:57.600 INFO:tasks.workunit.client.0.smithi114.stdout: 10 32748 14.08 MB/sec execute 9 sec latency 295.386 ms 2024-05-27T07:55:58.059 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:57 smithi114 ceph-mon[32038]: pgmap v553: 129 pgs: 129 active+clean; 312 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 565 op/s 2024-05-27T07:55:58.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:57 smithi163 ceph-mon[36451]: pgmap v553: 129 pgs: 129 active+clean; 312 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 565 op/s 2024-05-27T07:55:58.231 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:57 smithi186 ceph-mon[36647]: pgmap v553: 129 pgs: 129 active+clean; 312 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 565 op/s 2024-05-27T07:55:58.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 33057 14.14 MB/sec execute 10 sec latency 159.700 ms 2024-05-27T07:55:59.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 33311 14.39 MB/sec execute 11 sec latency 530.172 ms 2024-05-27T07:56:00.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:55:59 smithi114 ceph-mon[32038]: pgmap v554: 129 pgs: 129 active+clean; 313 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.3 MiB/s wr, 580 op/s 2024-05-27T07:56:00.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:55:59 smithi163 ceph-mon[36451]: pgmap v554: 129 pgs: 129 active+clean; 313 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.3 MiB/s wr, 580 op/s 2024-05-27T07:56:00.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:55:59 smithi186 ceph-mon[36647]: pgmap v554: 129 pgs: 129 active+clean; 313 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.3 MiB/s wr, 580 op/s 2024-05-27T07:56:00.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 33499 13.68 MB/sec execute 12 sec latency 231.273 ms 2024-05-27T07:56:00.859 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9233f1c700 time 2024-05-27T07:56:00.488988+0000 2024-05-27T07:56:00.859 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:56:00.859 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:00.859 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9246761ba6] 2024-05-27T07:56:00.859 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9246761dd1] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560fec17023d] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 4: (SnapRealm::build_snap_set() const+0x323) [0x560fec15b971] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 5: (SnapRealm::check_cache() const+0x13e) [0x560fec159fa0] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560fec15a651] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560fec0ef347] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560febdeb87e] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560febdf0e94] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560febdfb969] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 11: (MDLog::_replay_thread()+0x1f38) [0x560fec19448c] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 12: (MDLog::ReplayThread::entry()+0x11) [0x560fec19d9cb] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 13: (Thread::entry_wrapper()+0x43) [0x7f924673072d] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9246730749] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f92452d41ca] 2024-05-27T07:56:00.860 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 16: clone() 2024-05-27T07:56:00.861 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.490+0000 7f9233f1c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9233f1c700 time 2024-05-27T07:56:00.488988+0000 2024-05-27T07:56:00.861 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:56:00.862 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.862 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:00.862 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9246761ba6] 2024-05-27T07:56:00.862 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9246761dd1] 2024-05-27T07:56:00.862 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560fec17023d] 2024-05-27T07:56:00.862 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 4: (SnapRealm::build_snap_set() const+0x323) [0x560fec15b971] 2024-05-27T07:56:00.862 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 5: (SnapRealm::check_cache() const+0x13e) [0x560fec159fa0] 2024-05-27T07:56:00.862 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560fec15a651] 2024-05-27T07:56:00.862 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560fec0ef347] 2024-05-27T07:56:00.862 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560febdeb87e] 2024-05-27T07:56:00.862 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560febdf0e94] 2024-05-27T07:56:00.862 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560febdfb969] 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 11: (MDLog::_replay_thread()+0x1f38) [0x560fec19448c] 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 12: (MDLog::ReplayThread::entry()+0x11) [0x560fec19d9cb] 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 13: (Thread::entry_wrapper()+0x43) [0x7f924673072d] 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9246730749] 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f92452d41ca] 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 16: clone() 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: *** Caught signal (Aborted) ** 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: in thread 7f9233f1c700 thread_name:md_log_replay 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f92452ded20] 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2: gsignal() 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 3: abort() 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9246761ced] 2024-05-27T07:56:00.863 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9246761dd1] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560fec17023d] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 7: (SnapRealm::build_snap_set() const+0x323) [0x560fec15b971] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 8: (SnapRealm::check_cache() const+0x13e) [0x560fec159fa0] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560fec15a651] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560fec0ef347] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560febdeb87e] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560febdf0e94] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560febdfb969] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 14: (MDLog::_replay_thread()+0x1f38) [0x560fec19448c] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 15: (MDLog::ReplayThread::entry()+0x11) [0x560fec19d9cb] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 16: (Thread::entry_wrapper()+0x43) [0x7f924673072d] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9246730749] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f92452d41ca] 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 19: clone() 2024-05-27T07:56:00.864 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.493+0000 7f9233f1c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: in thread 7f9233f1c700 thread_name:md_log_replay 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f92452ded20] 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2: gsignal() 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 3: abort() 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9246761ced] 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9246761dd1] 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560fec17023d] 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 7: (SnapRealm::build_snap_set() const+0x323) [0x560fec15b971] 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 8: (SnapRealm::check_cache() const+0x13e) [0x560fec159fa0] 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560fec15a651] 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560fec0ef347] 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560febdeb87e] 2024-05-27T07:56:00.865 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560febdf0e94] 2024-05-27T07:56:00.866 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560febdfb969] 2024-05-27T07:56:00.866 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 14: (MDLog::_replay_thread()+0x1f38) [0x560fec19448c] 2024-05-27T07:56:00.866 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 15: (MDLog::ReplayThread::entry()+0x11) [0x560fec19d9cb] 2024-05-27T07:56:00.866 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 16: (Thread::entry_wrapper()+0x43) [0x7f924673072d] 2024-05-27T07:56:00.866 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9246730749] 2024-05-27T07:56:00.866 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f92452d41ca] 2024-05-27T07:56:00.866 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 19: clone() 2024-05-27T07:56:00.866 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:56:00.866 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.866 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: -1> 2024-05-27T07:56:00.490+0000 7f9233f1c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9233f1c700 time 2024-05-27T07:56:00.488988+0000 2024-05-27T07:56:00.866 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:56:00.866 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.866 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9246761ba6] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9246761dd1] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560fec17023d] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 4: (SnapRealm::build_snap_set() const+0x323) [0x560fec15b971] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 5: (SnapRealm::check_cache() const+0x13e) [0x560fec159fa0] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560fec15a651] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560fec0ef347] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560febdeb87e] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560febdf0e94] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560febdfb969] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 11: (MDLog::_replay_thread()+0x1f38) [0x560fec19448c] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 12: (MDLog::ReplayThread::entry()+0x11) [0x560fec19d9cb] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 13: (Thread::entry_wrapper()+0x43) [0x7f924673072d] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9246730749] 2024-05-27T07:56:00.867 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f92452d41ca] 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 16: clone() 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 0> 2024-05-27T07:56:00.493+0000 7f9233f1c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: in thread 7f9233f1c700 thread_name:md_log_replay 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f92452ded20] 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2: gsignal() 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 3: abort() 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9246761ced] 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9246761dd1] 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560fec17023d] 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 7: (SnapRealm::build_snap_set() const+0x323) [0x560fec15b971] 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 8: (SnapRealm::check_cache() const+0x13e) [0x560fec159fa0] 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560fec15a651] 2024-05-27T07:56:00.868 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560fec0ef347] 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560febdeb87e] 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560febdf0e94] 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560febdfb969] 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 14: (MDLog::_replay_thread()+0x1f38) [0x560fec19448c] 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 15: (MDLog::ReplayThread::entry()+0x11) [0x560fec19d9cb] 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 16: (Thread::entry_wrapper()+0x43) [0x7f924673072d] 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9246730749] 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f92452d41ca] 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 19: clone() 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: -1> 2024-05-27T07:56:00.490+0000 7f9233f1c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9233f1c700 time 2024-05-27T07:56:00.488988+0000 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.869 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9246761ba6] 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9246761dd1] 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560fec17023d] 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 4: (SnapRealm::build_snap_set() const+0x323) [0x560fec15b971] 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 5: (SnapRealm::check_cache() const+0x13e) [0x560fec159fa0] 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560fec15a651] 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560fec0ef347] 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560febdeb87e] 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560febdf0e94] 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560febdfb969] 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 11: (MDLog::_replay_thread()+0x1f38) [0x560fec19448c] 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 12: (MDLog::ReplayThread::entry()+0x11) [0x560fec19d9cb] 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 13: (Thread::entry_wrapper()+0x43) [0x7f924673072d] 2024-05-27T07:56:00.870 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9246730749] 2024-05-27T07:56:00.871 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f92452d41ca] 2024-05-27T07:56:00.871 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 16: clone() 2024-05-27T07:56:00.871 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.871 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 0> 2024-05-27T07:56:00.493+0000 7f9233f1c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:56:00.871 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: in thread 7f9233f1c700 thread_name:md_log_replay 2024-05-27T07:56:00.871 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.871 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:00.871 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f92452ded20] 2024-05-27T07:56:00.871 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2: gsignal() 2024-05-27T07:56:00.871 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 3: abort() 2024-05-27T07:56:00.871 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9246761ced] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9246761dd1] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560fec17023d] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 7: (SnapRealm::build_snap_set() const+0x323) [0x560fec15b971] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 8: (SnapRealm::check_cache() const+0x13e) [0x560fec159fa0] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560fec15a651] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560fec0ef347] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560febdeb87e] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560febdf0e94] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560febdfb969] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 14: (MDLog::_replay_thread()+0x1f38) [0x560fec19448c] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 15: (MDLog::ReplayThread::entry()+0x11) [0x560fec19d9cb] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 16: (Thread::entry_wrapper()+0x43) [0x7f924673072d] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9246730749] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f92452d41ca] 2024-05-27T07:56:00.872 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 19: clone() 2024-05-27T07:56:00.873 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:56:00.873 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75084]: 2024-05-27T07:56:00.873 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:00 smithi163 podman[75127]: 2024-05-27 07:56:00.553969744 +0000 UTC m=+0.020140560 container died dc1745274d236c785761b5d488cc69a8b7a6c81675634a946fa1b2ae7e76c376 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-27T07:56:01.496 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:01 smithi163 podman[75127]: 2024-05-27 07:56:01.244805518 +0000 UTC m=+0.710976314 container remove dc1745274d236c785761b5d488cc69a8b7a6c81675634a946fa1b2ae7e76c376 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True) 2024-05-27T07:56:01.496 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:01 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:56:01.602 INFO:tasks.workunit.client.0.smithi114.stdout: 10 33839 14.35 MB/sec execute 13 sec latency 230.897 ms 2024-05-27T07:56:01.761 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:01 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:56:01.762 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:01 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.702s CPU time. 2024-05-27T07:56:02.031 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:01 smithi186 ceph-mon[36647]: pgmap v555: 129 pgs: 129 active+clean; 319 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.6 MiB/s wr, 511 op/s 2024-05-27T07:56:02.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:01 smithi114 ceph-mon[32038]: pgmap v555: 129 pgs: 129 active+clean; 319 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.6 MiB/s wr, 511 op/s 2024-05-27T07:56:02.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:01 smithi163 ceph-mon[36451]: pgmap v555: 129 pgs: 129 active+clean; 319 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.6 MiB/s wr, 511 op/s 2024-05-27T07:56:02.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:56:02.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 34068 13.91 MB/sec execute 14 sec latency 426.939 ms 2024-05-27T07:56:02.865 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:56:03.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 34176 13.11 MB/sec execute 15 sec latency 895.643 ms 2024-05-27T07:56:03.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:03 smithi114 ceph-mon[32038]: pgmap v556: 129 pgs: 129 active+clean; 323 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.8 MiB/s wr, 396 op/s 2024-05-27T07:56:03.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:03 smithi114 ceph-mon[32038]: osdmap e147: 12 total, 12 up, 12 in 2024-05-27T07:56:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.777+0000 7f66021eb640 1 -- 172.21.15.114:0/1426154911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc114c80 msgr2=0x7f65fc1170a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.777+0000 7f66021eb640 1 --2- 172.21.15.114:0/1426154911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc114c80 0x7f65fc1170a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009ca0 tx=0x7f65f402a980 comp rx=0 tx=0).stop 2024-05-27T07:56:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.782+0000 7f66021eb640 1 -- 172.21.15.114:0/1426154911 shutdown_connections 2024-05-27T07:56:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.782+0000 7f66021eb640 1 --2- 172.21.15.114:0/1426154911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc114c80 0x7f65fc1170a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.782+0000 7f66021eb640 1 --2- 172.21.15.114:0/1426154911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65fc112320 0x7f65fc114740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.782+0000 7f66021eb640 1 --2- 172.21.15.114:0/1426154911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65fc075480 0x7f65fc075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.782+0000 7f66021eb640 1 -- 172.21.15.114:0/1426154911 >> 172.21.15.114:0/1426154911 conn(0x7f65fc0793c0 msgr2=0x7f65fc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.787+0000 7f66021eb640 1 -- 172.21.15.114:0/1426154911 shutdown_connections 2024-05-27T07:56:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.787+0000 7f66021eb640 1 -- 172.21.15.114:0/1426154911 wait complete. 2024-05-27T07:56:03.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.788+0000 7f66021eb640 1 Processor -- start 2024-05-27T07:56:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.791+0000 7f66021eb640 1 -- start start 2024-05-27T07:56:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.791+0000 7f66021eb640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc075480 0x7f65fc1a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.792+0000 7f66021eb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65fc112320 0x7f65fc1a1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.792+0000 7f65fb7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc075480 0x7f65fc1a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.792+0000 7f65fb7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc075480 0x7f65fc1a0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:50928/0 (socket says 172.21.15.114:50928) 2024-05-27T07:56:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.792+0000 7f65fb7fe640 1 -- 172.21.15.114:0/1582580957 learned_addr learned my addr 172.21.15.114:0/1582580957 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.792+0000 7f66021eb640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65fc114c80 0x7f65fc1a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.792+0000 7f66021eb640 1 -- 172.21.15.114:0/1582580957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc11a4b0 con 0x7f65fc112320 2024-05-27T07:56:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.792+0000 7f66021eb640 1 -- 172.21.15.114:0/1582580957 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc11a330 con 0x7f65fc075480 2024-05-27T07:56:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.792+0000 7f65faffd640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65fc112320 0x7f65fc1a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.792+0000 7f66021eb640 1 -- 172.21.15.114:0/1582580957 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc11a630 con 0x7f65fc114c80 2024-05-27T07:56:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.794+0000 7f65faffd640 1 -- 172.21.15.114:0/1582580957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65fc114c80 msgr2=0x7f65fc1a5610 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:56:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.794+0000 7f65faffd640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65fc114c80 0x7f65fc1a5610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.794+0000 7f65faffd640 1 -- 172.21.15.114:0/1582580957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc075480 msgr2=0x7f65fc1a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.794+0000 7f65faffd640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc075480 0x7f65fc1a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.794+0000 7f65faffd640 1 -- 172.21.15.114:0/1582580957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc1a5c50 con 0x7f65fc112320 2024-05-27T07:56:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.795+0000 7f65faffd640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65fc112320 0x7f65fc1a1350 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f65f000e9f0 tx=0x7f65f000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.796+0000 7f65f8ff9640 1 -- 172.21.15.114:0/1582580957 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f000ccc0 con 0x7f65fc112320 2024-05-27T07:56:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.796+0000 7f65f8ff9640 1 -- 172.21.15.114:0/1582580957 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65f00108e0 con 0x7f65fc112320 2024-05-27T07:56:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.796+0000 7f66021eb640 1 -- 172.21.15.114:0/1582580957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65fc1a5f40 con 0x7f65fc112320 2024-05-27T07:56:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.796+0000 7f66021eb640 1 -- 172.21.15.114:0/1582580957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65fc1ad8c0 con 0x7f65fc112320 2024-05-27T07:56:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.796+0000 7f65f8ff9640 1 -- 172.21.15.114:0/1582580957 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f00188b0 con 0x7f65fc112320 2024-05-27T07:56:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.805+0000 7f6609170740 1 Processor -- start 2024-05-27T07:56:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.811+0000 7f6609170740 1 -- start start 2024-05-27T07:56:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.811+0000 7f6609170740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c47e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.811+0000 7f6609170740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0b42f0 0x55ab7c47e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.811+0000 7f6609170740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c47ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.811+0000 7f6609170740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ab7c3f0c30 con 0x55ab7c0bc4d0 2024-05-27T07:56:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.811+0000 7f6609170740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ab7c3e07d0 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.811+0000 7f6609170740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ab7c3e4810 con 0x55ab7c0b42f0 2024-05-27T07:56:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.812+0000 7f65fbfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c47e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.814+0000 7f65fbfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c47e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:50930/0 (socket says 172.21.15.114:50930) 2024-05-27T07:56:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.814+0000 7f65fbfff640 1 -- 172.21.15.114:0/3592277540 learned_addr learned my addr 172.21.15.114:0/3592277540 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.815+0000 7f65fb7fe640 1 --2- 172.21.15.114:0/3592277540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0b42f0 0x55ab7c47e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.822+0000 7f65f8ff9640 1 -- 172.21.15.114:0/1582580957 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f65f0010430 con 0x7f65fc112320 2024-05-27T07:56:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.823+0000 7f65faffd640 1 --2- 172.21.15.114:0/3592277540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c47ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.826+0000 7f65f8ff9640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f65e40b2aa0 0x7f65e40b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.826+0000 7f65f8ff9640 1 -- 172.21.15.114:0/1582580957 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 9642+0+0 (secure 0 0 0) 0x7f65f00e2db0 con 0x7f65fc112320 2024-05-27T07:56:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.828+0000 7f65fbfff640 1 -- 172.21.15.114:0/3592277540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0b42f0 msgr2=0x55ab7c47e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.828+0000 7f65fbfff640 1 --2- 172.21.15.114:0/3592277540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0b42f0 0x55ab7c47e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.828+0000 7f65fbfff640 1 -- 172.21.15.114:0/3592277540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0bc4d0 msgr2=0x55ab7c47ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.828+0000 7f65fbfff640 1 --2- 172.21.15.114:0/3592277540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c47ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.828+0000 7f65fbfff640 1 -- 172.21.15.114:0/3592277540 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab7c0f5210 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.828+0000 7f65fb7fe640 1 --2- 172.21.15.114:0/3592277540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0b42f0 0x55ab7c47e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:56:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.828+0000 7f65fbfff640 1 --2- 172.21.15.114:0/3592277540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c47e3d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f65f40094d0 tx=0x7f65f402f3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65e6bf4288958b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.828+0000 7f65faffd640 1 --2- 172.21.15.114:0/3592277540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c47ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:56:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.829+0000 7f65e1ffb640 1 -- 172.21.15.114:0/3592277540 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4040030 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.830+0000 7f65e1ffb640 1 -- 172.21.15.114:0/3592277540 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65f402f630 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.830+0000 7f65e1ffb640 1 -- 172.21.15.114:0/3592277540 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4034070 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.831+0000 7f6609170740 1 -- 172.21.15.114:0/3592277540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0dd1b0 msgr2=0x55ab7c47e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.831+0000 7f6609170740 1 --2- 172.21.15.114:0/3592277540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c47e3d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f65f40094d0 tx=0x7f65f402f3d0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.832+0000 7f65fbfff640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f65e40b2aa0 0x7f65e40b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.836+0000 7f65fbfff640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f65e40b2aa0 0x7f65e40b4fb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f65f4046040 tx=0x7f65f402f3d0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.839+0000 7f6609170740 1 -- 172.21.15.114:0/3592277540 shutdown_connections 2024-05-27T07:56:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.839+0000 7f6609170740 1 --2- 172.21.15.114:0/3592277540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c47ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.839+0000 7f6609170740 1 --2- 172.21.15.114:0/3592277540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0b42f0 0x55ab7c47e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.839+0000 7f6609170740 1 --2- 172.21.15.114:0/3592277540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c47e3d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.839+0000 7f6609170740 1 -- 172.21.15.114:0/3592277540 >> 172.21.15.114:0/3592277540 conn(0x55ab7c0d1300 msgr2=0x55ab7c47ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.844+0000 7f6609170740 1 -- 172.21.15.114:0/3592277540 shutdown_connections 2024-05-27T07:56:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.845+0000 7f6609170740 1 -- 172.21.15.114:0/3592277540 wait complete. 2024-05-27T07:56:03.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.846+0000 7f6609170740 1 -- start start 2024-05-27T07:56:03.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.846+0000 7f6609170740 1 Processor -- start 2024-05-27T07:56:03.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.853+0000 7f6609170740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0b42f0 0x55ab7c48a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.853+0000 7f6609170740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c48a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.853+0000 7f6609170740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c488930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.853+0000 7f6609170740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ab7c0c6790 con 0x55ab7c0bc4d0 2024-05-27T07:56:03.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.853+0000 7f6609170740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ab7c39c890 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.853+0000 7f6609170740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ab7c49c450 con 0x55ab7c0b42f0 2024-05-27T07:56:03.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.853+0000 7f6609170740 10 client.0 _ll_register_callbacks cb 0x55ab7c0f4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.854+0000 7f65fbfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c488930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.856+0000 7f65faffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c48a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.856+0000 7f65faffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c48a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45138/0 (socket says 172.21.15.114:45138) 2024-05-27T07:56:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.856+0000 7f65faffd640 1 -- 172.21.15.114:0/3156002401 learned_addr learned my addr 172.21.15.114:0/3156002401 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:03.857 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.856+0000 7f65fb7fe640 1 --2- 172.21.15.114:0/3156002401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0b42f0 0x55ab7c48a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.857+0000 7f65faffd640 1 -- 172.21.15.114:0/3156002401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0b42f0 msgr2=0x55ab7c48a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.857+0000 7f65faffd640 1 --2- 172.21.15.114:0/3156002401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0b42f0 0x55ab7c48a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.857+0000 7f65faffd640 1 -- 172.21.15.114:0/3156002401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0dd1b0 msgr2=0x55ab7c488930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.857+0000 7f65faffd640 1 --2- 172.21.15.114:0/3156002401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c488930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.857+0000 7f65faffd640 1 -- 172.21.15.114:0/3156002401 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab7c1c1620 con 0x55ab7c0bc4d0 2024-05-27T07:56:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.858+0000 7f65faffd640 1 --2- 172.21.15.114:0/3156002401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c48a960 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f65f000e9c0 tx=0x7f65f0010720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b161964db621de0a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.858+0000 7f65daffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:56:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.858+0000 7f65daffd640 1 -- 172.21.15.114:0/3156002401 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f00e04d0 con 0x55ab7c0bc4d0 2024-05-27T07:56:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.858+0000 7f6609170740 1 -- 172.21.15.114:0/3156002401 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65f402f630 con 0x55ab7c0bc4d0 2024-05-27T07:56:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.858+0000 7f65daffd640 1 -- 172.21.15.114:0/3156002401 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9642+0+0 (secure 0 0 0) 0x7f65f00a8e30 con 0x55ab7c0bc4d0 2024-05-27T07:56:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.858+0000 7f65daffd640 1 -- 172.21.15.114:0/3156002401 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f65f00a96e0 con 0x55ab7c0bc4d0 2024-05-27T07:56:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.860+0000 7f6609170740 10 client.19230 fetch_fsmap learned FSMap version 91 2024-05-27T07:56:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.860+0000 7f6609170740 1 -- 172.21.15.114:0/3156002401 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=4+}) -- 0x55ab7c4b97d0 con 0x55ab7c0bc4d0 2024-05-27T07:56:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.860+0000 7f65daffd640 1 -- 172.21.15.114:0/3156002401 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65f402f630 con 0x55ab7c0bc4d0 2024-05-27T07:56:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.861+0000 7f65daffd640 1 -- 172.21.15.114:0/3156002401 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 91) ==== 4700+0+0 (secure 0 0 0) 0x7f65f00a5050 con 0x55ab7c0bc4d0 2024-05-27T07:56:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.861+0000 7f6609170740 10 client.19230 fetch_fsmap finished waiting for FSMap version 91 2024-05-27T07:56:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.861+0000 7f6609170740 10 client.19230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.861+0000 7f6609170740 20 client.19230 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.861+0000 7f6609170740 1 --2- 172.21.15.114:0/3156002401 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ab7c09bd80 0x55ab7c4bd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.861+0000 7f6609170740 4 client.19230 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:56:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.861+0000 7f6609170740 1 -- 172.21.15.114:0/3156002401 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab7c3f2b80 con 0x55ab7c09bd80 2024-05-27T07:56:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.862+0000 7f65fb7fe640 1 --2- 172.21.15.114:0/3156002401 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ab7c09bd80 0x55ab7c4bd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.867+0000 7f65fb7fe640 1 --2- 172.21.15.114:0/3156002401 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ab7c09bd80 0x55ab7c4bd1c0 crc :-1 s=READY pgs=446 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c44ce73d9892aa1f server_cookie=852b6e3e37f9ec10 in_seq=0 out_seq=0 2024-05-27T07:56:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.867+0000 7f65daffd640 10 client.19230 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.870+0000 7f65daffd640 1 -- 172.21.15.114:0/3156002401 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ab7c3f2b80 con 0x55ab7c09bd80 2024-05-27T07:56:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.870+0000 7f65daffd640 10 client.19230 handle_command_reply: tid=0 2024-05-27T07:56:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.870+0000 7f6609170740 1 client.19230 shutdown 2024-05-27T07:56:03.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.879+0000 7f6609170740 1 -- 172.21.15.114:0/3156002401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0bc4d0 msgr2=0x55ab7c48a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:03.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.879+0000 7f6609170740 1 --2- 172.21.15.114:0/3156002401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c48a960 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f65f000e9c0 tx=0x7f65f0010720 comp rx=0 tx=0).stop 2024-05-27T07:56:03.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.883+0000 7f6609170740 1 -- 172.21.15.114:0/3156002401 shutdown_connections 2024-05-27T07:56:03.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.883+0000 7f6609170740 1 --2- 172.21.15.114:0/3156002401 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ab7c09bd80 0x55ab7c4bd1c0 crc :-1 s=READY pgs=446 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.883+0000 7f6609170740 1 --2- 172.21.15.114:0/3156002401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c488930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.883+0000 7f6609170740 1 --2- 172.21.15.114:0/3156002401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c48a960 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.883+0000 7f6609170740 1 --2- 172.21.15.114:0/3156002401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0b42f0 0x55ab7c48a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.883+0000 7f6609170740 1 -- 172.21.15.114:0/3156002401 >> 172.21.15.114:0/3156002401 conn(0x55ab7c0d1300 msgr2=0x55ab7c47e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:03.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.884+0000 7f65daffd640 0 client.19230 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:03.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.889+0000 7f6609170740 1 -- 172.21.15.114:0/3156002401 shutdown_connections 2024-05-27T07:56:03.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.889+0000 7f6609170740 1 -- 172.21.15.114:0/3156002401 wait complete. 2024-05-27T07:56:03.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.889+0000 7f6609170740 20 client.19230 trim_cache size 0 max 16384 2024-05-27T07:56:03.924 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.924+0000 7f6609170740 1 Processor -- start 2024-05-27T07:56:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.928+0000 7f6609170740 1 -- start start 2024-05-27T07:56:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.928+0000 7f6609170740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0b42f0 0x55ab7c4826a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.928+0000 7f6609170740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c4a1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.929+0000 7f65fb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0b42f0 0x55ab7c4826a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.929+0000 7f65fb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0b42f0 0x55ab7c4826a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45148/0 (socket says 172.21.15.114:45148) 2024-05-27T07:56:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.929+0000 7f6609170740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c4a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.929+0000 7f6609170740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc06e230 con 0x55ab7c0b42f0 2024-05-27T07:56:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.929+0000 7f6609170740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc06e900 con 0x55ab7c0bc4d0 2024-05-27T07:56:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.929+0000 7f6609170740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc06d1d0 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.929+0000 7f65fb7fe640 1 -- 172.21.15.114:0/71902415 learned_addr learned my addr 172.21.15.114:0/71902415 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.929+0000 7f65fb7fe640 1 -- 172.21.15.114:0/71902415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0dd1b0 msgr2=0x55ab7c4a1c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:56:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.929+0000 7f65fb7fe640 1 --2- 172.21.15.114:0/71902415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c4a1c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.929+0000 7f65fb7fe640 1 -- 172.21.15.114:0/71902415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0bc4d0 msgr2=0x55ab7c4a1740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:56:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.929+0000 7f65fb7fe640 1 --2- 172.21.15.114:0/71902415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c4a1740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.929+0000 7f65fb7fe640 1 -- 172.21.15.114:0/71902415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab7c489770 con 0x55ab7c0b42f0 2024-05-27T07:56:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.929+0000 7f65fb7fe640 1 --2- 172.21.15.114:0/71902415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0b42f0 0x55ab7c4826a0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f65ec000c00 tx=0x7f65ec005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32e36bff162d87c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.932+0000 7f65e17fa640 1 -- 172.21.15.114:0/71902415 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0140d0 con 0x55ab7c0b42f0 2024-05-27T07:56:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.932+0000 7f65e17fa640 1 -- 172.21.15.114:0/71902415 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65ec014270 con 0x55ab7c0b42f0 2024-05-27T07:56:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.932+0000 7f65e17fa640 1 -- 172.21.15.114:0/71902415 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec012450 con 0x55ab7c0b42f0 2024-05-27T07:56:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.933+0000 7f6609170740 1 -- 172.21.15.114:0/71902415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0b42f0 msgr2=0x55ab7c4826a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.934+0000 7f6609170740 1 --2- 172.21.15.114:0/71902415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0b42f0 0x55ab7c4826a0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f65ec000c00 tx=0x7f65ec005f90 comp rx=0 tx=0).stop 2024-05-27T07:56:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.937+0000 7f6609170740 1 -- 172.21.15.114:0/71902415 shutdown_connections 2024-05-27T07:56:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.937+0000 7f6609170740 1 --2- 172.21.15.114:0/71902415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c4a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.937+0000 7f6609170740 1 --2- 172.21.15.114:0/71902415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c4a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.937+0000 7f6609170740 1 --2- 172.21.15.114:0/71902415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0b42f0 0x55ab7c4826a0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.937+0000 7f6609170740 1 -- 172.21.15.114:0/71902415 >> 172.21.15.114:0/71902415 conn(0x55ab7c0d1300 msgr2=0x55ab7c481f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:03.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.944+0000 7f6609170740 1 -- 172.21.15.114:0/71902415 shutdown_connections 2024-05-27T07:56:03.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.944+0000 7f6609170740 1 -- 172.21.15.114:0/71902415 wait complete. 2024-05-27T07:56:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.945+0000 7f6609170740 1 -- start start 2024-05-27T07:56:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.945+0000 7f6609170740 1 Processor -- start 2024-05-27T07:56:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.949+0000 7f6609170740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0b42f0 0x55ab7c4a0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.949+0000 7f6609170740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c4a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.949+0000 7f6609170740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c4a1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.949+0000 7f6609170740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc1100d0 con 0x55ab7c0b42f0 2024-05-27T07:56:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.949+0000 7f6609170740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc050860 con 0x55ab7c0bc4d0 2024-05-27T07:56:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.949+0000 7f6609170740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc06cb00 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.949+0000 7f6609170740 10 client.0 _ll_register_callbacks cb 0x7f65f00a5a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.952+0000 7f65fbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c4a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.952+0000 7f65fbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c4a1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54940/0 (socket says 172.21.15.114:54940) 2024-05-27T07:56:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.952+0000 7f65fbfff640 1 -- 172.21.15.114:0/4072966387 learned_addr learned my addr 172.21.15.114:0/4072966387 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.953+0000 7f65fb7fe640 1 --2- 172.21.15.114:0/4072966387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0b42f0 0x55ab7c4a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.953+0000 7f65fbfff640 1 -- 172.21.15.114:0/4072966387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0bc4d0 msgr2=0x55ab7c4a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.953+0000 7f65fbfff640 1 --2- 172.21.15.114:0/4072966387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c4a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.953+0000 7f65fbfff640 1 -- 172.21.15.114:0/4072966387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0b42f0 msgr2=0x55ab7c4a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.953+0000 7f65fbfff640 1 --2- 172.21.15.114:0/4072966387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0b42f0 0x55ab7c4a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.953+0000 7f65fbfff640 1 -- 172.21.15.114:0/4072966387 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab7c523aa0 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.954+0000 7f65fbfff640 1 --2- 172.21.15.114:0/4072966387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c4a1180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f65f4004af0 tx=0x7f65f4009420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8d28aec09f5a694 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.954+0000 7f65daffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:56:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.955+0000 7f65daffd640 1 -- 172.21.15.114:0/4072966387 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4040040 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.955+0000 7f65daffd640 1 -- 172.21.15.114:0/4072966387 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9642+0+0 (secure 0 0 0) 0x7f65f4030680 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.955+0000 7f6609170740 1 -- 172.21.15.114:0/4072966387 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65ec014270 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.959+0000 7f65daffd640 1 -- 172.21.15.114:0/4072966387 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f65f4034040 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.960+0000 7f6609170740 10 client.28711 fetch_fsmap learned FSMap version 91 2024-05-27T07:56:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.960+0000 7f6609170740 1 -- 172.21.15.114:0/4072966387 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=4+}) -- 0x55ab7c524d60 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.963+0000 7f65daffd640 1 -- 172.21.15.114:0/4072966387 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65f4041040 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.963+0000 7f65daffd640 1 -- 172.21.15.114:0/4072966387 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 91) ==== 4700+0+0 (secure 0 0 0) 0x7f65f4058460 con 0x55ab7c0dd1b0 2024-05-27T07:56:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.964+0000 7f6609170740 10 client.28711 fetch_fsmap finished waiting for FSMap version 91 2024-05-27T07:56:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.964+0000 7f6609170740 10 client.28711 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.964+0000 7f6609170740 20 client.28711 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.965+0000 7f6609170740 1 --2- 172.21.15.114:0/4072966387 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ab7c0711c0 0x55ab7c5274d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.965+0000 7f6609170740 4 client.28711 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:56:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.965+0000 7f6609170740 1 -- 172.21.15.114:0/4072966387 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f65ec014270 con 0x55ab7c0711c0 2024-05-27T07:56:03.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.972+0000 7f65fb7fe640 1 --2- 172.21.15.114:0/4072966387 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ab7c0711c0 0x55ab7c5274d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:03.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.988+0000 7f65fb7fe640 1 --2- 172.21.15.114:0/4072966387 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ab7c0711c0 0x55ab7c5274d0 crc :-1 s=READY pgs=447 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6916580acfc2958 server_cookie=1556a2228002dd03 in_seq=0 out_seq=0 2024-05-27T07:56:03.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:03.989+0000 7f65daffd640 10 client.28711 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:04.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:03 smithi163 ceph-mon[36451]: pgmap v556: 129 pgs: 129 active+clean; 323 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.8 MiB/s wr, 396 op/s 2024-05-27T07:56:04.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:03 smithi163 ceph-mon[36451]: osdmap e147: 12 total, 12 up, 12 in 2024-05-27T07:56:04.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:03 smithi186 ceph-mon[36647]: pgmap v556: 129 pgs: 129 active+clean; 323 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.8 MiB/s wr, 396 op/s 2024-05-27T07:56:04.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:03 smithi186 ceph-mon[36647]: osdmap e147: 12 total, 12 up, 12 in 2024-05-27T07:56:04.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 34238 12.67 MB/sec execute 16 sec latency 1386.801 ms 2024-05-27T07:56:05.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.084+0000 7f83c8831700 1 -- 172.21.15.114:0/2464464718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c411d920 msgr2=0x7f83c411fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:05.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.084+0000 7f83c8831700 1 --2- 172.21.15.114:0/2464464718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c411d920 0x7f83c411fd40 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00b0a0 tx=0x7f83bc00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.084+0000 7f83c8831700 1 -- 172.21.15.114:0/2464464718 shutdown_connections 2024-05-27T07:56:05.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.084+0000 7f83c8831700 1 --2- 172.21.15.114:0/2464464718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c411d920 0x7f83c411fd40 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.084+0000 7f83c8831700 1 --2- 172.21.15.114:0/2464464718 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83c410a920 0x7f83c410ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.084+0000 7f83c8831700 1 --2- 172.21.15.114:0/2464464718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83c410b270 0x7f83c411d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.084+0000 7f83c8831700 1 -- 172.21.15.114:0/2464464718 >> 172.21.15.114:0/2464464718 conn(0x7f83c406f4c0 msgr2=0x7f83c406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:05.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.084+0000 7f83c8831700 1 -- 172.21.15.114:0/2464464718 shutdown_connections 2024-05-27T07:56:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.084+0000 7f83c8831700 1 -- 172.21.15.114:0/2464464718 wait complete. 2024-05-27T07:56:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.085+0000 7f83c8831700 1 Processor -- start 2024-05-27T07:56:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.085+0000 7f83c8831700 1 -- start start 2024-05-27T07:56:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.086+0000 7f83c8831700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83c410a920 0x7f83c411d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.086+0000 7f83c8831700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83c410b270 0x7f83c41174a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.086+0000 7f83c8831700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c411d920 0x7f83c4117b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.086+0000 7f83c8831700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83c41134c0 con 0x7f83c411d920 2024-05-27T07:56:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.086+0000 7f83c8831700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f83c4113340 con 0x7f83c410b270 2024-05-27T07:56:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.086+0000 7f83c8831700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83c4113640 con 0x7f83c410a920 2024-05-27T07:56:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.088+0000 7f83c2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c411d920 0x7f83c4117b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.088+0000 7f83c2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c411d920 0x7f83c4117b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57246/0 (socket says 172.21.15.114:57246) 2024-05-27T07:56:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.088+0000 7f83c2d9d700 1 -- 172.21.15.114:0/3804205132 learned_addr learned my addr 172.21.15.114:0/3804205132 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.089+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/3804205132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83c410b270 0x7f83c41174a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.090+0000 7f83c2d9d700 1 -- 172.21.15.114:0/3804205132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83c410a920 msgr2=0x7f83c411d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:05.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.090+0000 7f83c2d9d700 1 --2- 172.21.15.114:0/3804205132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83c410a920 0x7f83c411d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.090+0000 7f83c2d9d700 1 -- 172.21.15.114:0/3804205132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83c410b270 msgr2=0x7f83c41174a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:05.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.090+0000 7f83c2d9d700 1 --2- 172.21.15.114:0/3804205132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83c410b270 0x7f83c41174a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.090+0000 7f83c2d9d700 1 -- 172.21.15.114:0/3804205132 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c4128a60 con 0x7f83c411d920 2024-05-27T07:56:05.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.094+0000 7f83c2d9d700 1 --2- 172.21.15.114:0/3804205132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c411d920 0x7f83c4117b40 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0062a0 tx=0x7f83bc004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:05.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.099+0000 7f83b37fe700 1 -- 172.21.15.114:0/3804205132 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc021070 con 0x7f83c411d920 2024-05-27T07:56:05.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.099+0000 7f83c8831700 1 -- 172.21.15.114:0/3804205132 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83c4128cf0 con 0x7f83c411d920 2024-05-27T07:56:05.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.099+0000 7f83c8831700 1 -- 172.21.15.114:0/3804205132 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83c4129230 con 0x7f83c411d920 2024-05-27T07:56:05.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.099+0000 7f83b37fe700 1 -- 172.21.15.114:0/3804205132 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83bc007b80 con 0x7f83c411d920 2024-05-27T07:56:05.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.099+0000 7f83b37fe700 1 -- 172.21.15.114:0/3804205132 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0052a0 con 0x7f83c411d920 2024-05-27T07:56:05.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.109+0000 7f83d7246b80 1 Processor -- start 2024-05-27T07:56:05.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.109+0000 7f83d7246b80 1 -- start start 2024-05-27T07:56:05.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.110+0000 7f83d7246b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e3373e40 0x5555e34f8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.110+0000 7f83d7246b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f9070 0x5555e34f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.110+0000 7f83d7246b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f3120 0x5555e34f3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.110+0000 7f83d7246b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5555e3420d70 con 0x5555e3373e40 2024-05-27T07:56:05.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.110+0000 7f83d7246b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5555e3410910 con 0x5555e34f3120 2024-05-27T07:56:05.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.110+0000 7f83d7246b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5555e3414950 con 0x5555e34f9070 2024-05-27T07:56:05.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.115+0000 7f83b37fe700 1 -- 172.21.15.114:0/3804205132 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f83bc024070 con 0x7f83c411d920 2024-05-27T07:56:05.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.116+0000 7f83b37fe700 1 --2- 172.21.15.114:0/3804205132 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83ac0b2c00 0x7f83ac0b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.116+0000 7f83c1d9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f3120 0x5555e34f3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.116+0000 7f83c1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e3373e40 0x5555e34f8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.116+0000 7f83c1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e3373e40 0x5555e34f8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57260/0 (socket says 172.21.15.114:57260) 2024-05-27T07:56:05.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.116+0000 7f83c1d9b700 1 -- 172.21.15.114:0/2614593466 learned_addr learned my addr 172.21.15.114:0/2614593466 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:05.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.116+0000 7f83c2d9d700 1 --2- 172.21.15.114:0/3804205132 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83ac0b2c00 0x7f83ac0b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.117+0000 7f83b37fe700 1 -- 172.21.15.114:0/3804205132 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 9642+0+0 (secure 0 0 0) 0x7f83bc00f020 con 0x7f83c411d920 2024-05-27T07:56:05.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.118+0000 7f83c259c700 1 --2- 172.21.15.114:0/2614593466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f9070 0x5555e34f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.119+0000 7f83c2d9d700 1 --2- 172.21.15.114:0/3804205132 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83ac0b2c00 0x7f83ac0b5110 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0e6fd0 tx=0x7f83bc0a6550 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:05.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.119+0000 7f83c1d9b700 1 -- 172.21.15.114:0/2614593466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f9070 msgr2=0x5555e34f2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:05.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.119+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/2614593466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f9070 0x5555e34f2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.119+0000 7f83c1d9b700 1 -- 172.21.15.114:0/2614593466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e3373e40 msgr2=0x5555e34f8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:05.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.119+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/2614593466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e3373e40 0x5555e34f8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.119+0000 7f83c1d9b700 1 -- 172.21.15.114:0/2614593466 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555e33b9fb0 con 0x5555e34f3120 2024-05-27T07:56:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.125+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/2614593466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f3120 0x5555e34f3530 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f83b80143f0 tx=0x7f83b8006f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaab1343e05219e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:05.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.129+0000 7f83b0ff9700 1 -- 172.21.15.114:0/2614593466 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b801c070 con 0x5555e34f3120 2024-05-27T07:56:05.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.129+0000 7f83b0ff9700 1 -- 172.21.15.114:0/2614593466 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83b8014dc0 con 0x5555e34f3120 2024-05-27T07:56:05.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.129+0000 7f83b0ff9700 1 -- 172.21.15.114:0/2614593466 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b80126f0 con 0x5555e34f3120 2024-05-27T07:56:05.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.130+0000 7f83d7246b80 1 -- 172.21.15.114:0/2614593466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f3120 msgr2=0x5555e34f3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:05.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.130+0000 7f83d7246b80 1 --2- 172.21.15.114:0/2614593466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f3120 0x5555e34f3530 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f83b80143f0 tx=0x7f83b8006f70 comp rx=0 tx=0).stop 2024-05-27T07:56:05.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.134+0000 7f83d7246b80 1 -- 172.21.15.114:0/2614593466 shutdown_connections 2024-05-27T07:56:05.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.134+0000 7f83d7246b80 1 --2- 172.21.15.114:0/2614593466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f3120 0x5555e34f3530 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.134+0000 7f83d7246b80 1 --2- 172.21.15.114:0/2614593466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e3373e40 0x5555e34f8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.134+0000 7f83d7246b80 1 --2- 172.21.15.114:0/2614593466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f9070 0x5555e34f2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.134+0000 7f83d7246b80 1 -- 172.21.15.114:0/2614593466 >> 172.21.15.114:0/2614593466 conn(0x5555e3376770 msgr2=0x5555e34edd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:05.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.138+0000 7f83d7246b80 1 -- 172.21.15.114:0/2614593466 shutdown_connections 2024-05-27T07:56:05.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.139+0000 7f83d7246b80 1 -- 172.21.15.114:0/2614593466 wait complete. 2024-05-27T07:56:05.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.139+0000 7f83d7246b80 1 -- start start 2024-05-27T07:56:05.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.139+0000 7f83d7246b80 1 Processor -- start 2024-05-27T07:56:05.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.140+0000 7f83d7246b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e3373e40 0x5555e34fc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.140+0000 7f83d7246b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34f3120 0x5555e34fce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.140+0000 7f83d7246b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f9070 0x5555e34fd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.140+0000 7f83d7246b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5555e3316fa0 con 0x5555e34f3120 2024-05-27T07:56:05.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.140+0000 7f83d7246b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5555e3378da0 con 0x5555e34f9070 2024-05-27T07:56:05.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.140+0000 7f83d7246b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5555e34ffb60 con 0x5555e3373e40 2024-05-27T07:56:05.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.140+0000 7f83d7246b80 10 client.0 _ll_register_callbacks cb 0x5555e33a75b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:05.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.142+0000 7f83c259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34f3120 0x5555e34fce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.142+0000 7f83c259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34f3120 0x5555e34fce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57272/0 (socket says 172.21.15.114:57272) 2024-05-27T07:56:05.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.142+0000 7f83c259c700 1 -- 172.21.15.114:0/4191627855 learned_addr learned my addr 172.21.15.114:0/4191627855 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:05.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.142+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/4191627855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e3373e40 0x5555e34fc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.142+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/4191627855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f9070 0x5555e34fd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.146+0000 7f83c1d9b700 1 -- 172.21.15.114:0/4191627855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f9070 msgr2=0x5555e34fd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.146+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/4191627855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f9070 0x5555e34fd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.146+0000 7f83c1d9b700 1 -- 172.21.15.114:0/4191627855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34f3120 msgr2=0x5555e34fce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:05.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.146+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/4191627855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34f3120 0x5555e34fce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.146+0000 7f83c1d9b700 1 -- 172.21.15.114:0/4191627855 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555e33608d0 con 0x5555e3373e40 2024-05-27T07:56:05.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.146+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/4191627855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e3373e40 0x5555e34fc930 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f83b8001eb0 tx=0x7f83b8004860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3be3605daa0ab5e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:05.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.147+0000 7f83a9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:56:05.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.150+0000 7f83a9ffb700 1 -- 172.21.15.114:0/4191627855 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b802b060 con 0x5555e3373e40 2024-05-27T07:56:05.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.150+0000 7f83d7246b80 1 -- 172.21.15.114:0/4191627855 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83b8014dc0 con 0x5555e3373e40 2024-05-27T07:56:05.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.150+0000 7f83a9ffb700 1 -- 172.21.15.114:0/4191627855 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9642+0+0 (secure 0 0 0) 0x7f83b8012e00 con 0x5555e3373e40 2024-05-27T07:56:05.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.153+0000 7f83a9ffb700 1 -- 172.21.15.114:0/4191627855 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f83b801c040 con 0x5555e3373e40 2024-05-27T07:56:05.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.158+0000 7f83d7246b80 10 client.28717 fetch_fsmap learned FSMap version 91 2024-05-27T07:56:05.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.158+0000 7f83d7246b80 1 -- 172.21.15.114:0/4191627855 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=4+}) -- 0x5555e33ca760 con 0x5555e3373e40 2024-05-27T07:56:05.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.163+0000 7f83a9ffb700 1 -- 172.21.15.114:0/4191627855 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83b8014dc0 con 0x5555e3373e40 2024-05-27T07:56:05.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.163+0000 7f83a9ffb700 1 -- 172.21.15.114:0/4191627855 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 91) ==== 4700+0+0 (secure 0 0 0) 0x7f83b8012070 con 0x5555e3373e40 2024-05-27T07:56:05.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.164+0000 7f83d7246b80 10 client.28717 fetch_fsmap finished waiting for FSMap version 91 2024-05-27T07:56:05.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.164+0000 7f83d7246b80 10 client.28717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:05.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.164+0000 7f83d7246b80 20 client.28717 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:05.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.164+0000 7f83d7246b80 1 --2- 172.21.15.114:0/4191627855 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5555e352ee40 0x5555e35312b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.164+0000 7f83d7246b80 4 client.28717 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:56:05.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.164+0000 7f83d7246b80 1 -- 172.21.15.114:0/4191627855 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555e3422cc0 con 0x5555e352ee40 2024-05-27T07:56:05.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.165+0000 7f83c259c700 1 --2- 172.21.15.114:0/4191627855 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5555e352ee40 0x5555e35312b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.177+0000 7f83a9ffb700 10 client.28717 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:05.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.177+0000 7f83c259c700 1 --2- 172.21.15.114:0/4191627855 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5555e352ee40 0x5555e35312b0 crc :-1 s=READY pgs=448 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69455f63eddf7322 server_cookie=e3f6aaa31ea27816 in_seq=0 out_seq=0 2024-05-27T07:56:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.181+0000 7f83a9ffb700 1 -- 172.21.15.114:0/4191627855 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5555e3422cc0 con 0x5555e352ee40 2024-05-27T07:56:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.181+0000 7f83a9ffb700 10 client.28717 handle_command_reply: tid=0 2024-05-27T07:56:05.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.182+0000 7f83d7246b80 1 client.28717 shutdown 2024-05-27T07:56:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.186+0000 7f83d7246b80 1 -- 172.21.15.114:0/4191627855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e3373e40 msgr2=0x5555e34fc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.186+0000 7f83d7246b80 1 --2- 172.21.15.114:0/4191627855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e3373e40 0x5555e34fc930 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f83b8001eb0 tx=0x7f83b8004860 comp rx=0 tx=0).stop 2024-05-27T07:56:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.186+0000 7f83d7246b80 1 -- 172.21.15.114:0/4191627855 shutdown_connections 2024-05-27T07:56:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.186+0000 7f83d7246b80 1 --2- 172.21.15.114:0/4191627855 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5555e352ee40 0x5555e35312b0 crc :-1 s=READY pgs=448 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.186+0000 7f83d7246b80 1 --2- 172.21.15.114:0/4191627855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f9070 0x5555e34fd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.186+0000 7f83d7246b80 1 --2- 172.21.15.114:0/4191627855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e3373e40 0x5555e34fc930 secure :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f83b8001eb0 tx=0x7f83b8004860 comp rx=0 tx=0).stop 2024-05-27T07:56:05.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.186+0000 7f83d7246b80 1 --2- 172.21.15.114:0/4191627855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34f3120 0x5555e34fce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.186+0000 7f83d7246b80 1 -- 172.21.15.114:0/4191627855 >> 172.21.15.114:0/4191627855 conn(0x5555e3376770 msgr2=0x5555e34f1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:05.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.186+0000 7f83a9ffb700 0 client.28717 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:05.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.190+0000 7f83d7246b80 1 -- 172.21.15.114:0/4191627855 shutdown_connections 2024-05-27T07:56:05.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.190+0000 7f83d7246b80 1 -- 172.21.15.114:0/4191627855 wait complete. 2024-05-27T07:56:05.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.190+0000 7f83d7246b80 20 client.28717 trim_cache size 0 max 16384 2024-05-27T07:56:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.234+0000 7f83d7246b80 1 Processor -- start 2024-05-27T07:56:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.235+0000 7f83d7246b80 1 -- start start 2024-05-27T07:56:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.235+0000 7f83d7246b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f3120 0x5555e34f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.235+0000 7f83d7246b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34f9070 0x5555e34f6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.235+0000 7f83d7246b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34fece0 0x5555e3536a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.235+0000 7f83c1d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f3120 0x5555e34f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.235+0000 7f83d7246b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83a400f420 con 0x5555e34f9070 2024-05-27T07:56:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.235+0000 7f83d7246b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f83a400faf0 con 0x5555e34fece0 2024-05-27T07:56:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.235+0000 7f83d7246b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83a400e320 con 0x5555e34f3120 2024-05-27T07:56:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.235+0000 7f83c1d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f3120 0x5555e34f2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:49632/0 (socket says 172.21.15.114:49632) 2024-05-27T07:56:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.235+0000 7f83c1d9b700 1 -- 172.21.15.114:0/3456833996 learned_addr learned my addr 172.21.15.114:0/3456833996 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.235+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/3456833996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34fece0 0x5555e3536a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.236+0000 7f83c1d9b700 1 -- 172.21.15.114:0/3456833996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34fece0 msgr2=0x5555e3536a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:05.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.236+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/3456833996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34fece0 0x5555e3536a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.236+0000 7f83c1d9b700 1 -- 172.21.15.114:0/3456833996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34f9070 msgr2=0x5555e34f6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:05.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.236+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/3456833996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34f9070 0x5555e34f6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.236+0000 7f83c1d9b700 1 -- 172.21.15.114:0/3456833996 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555e34fbc50 con 0x5555e34f3120 2024-05-27T07:56:05.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.236+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/3456833996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f3120 0x5555e34f2be0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f83b8000c00 tx=0x7f83b8012300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f80f365b7d3a5988 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:05.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.239+0000 7f83abfff700 1 -- 172.21.15.114:0/3456833996 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b802b070 con 0x5555e34f3120 2024-05-27T07:56:05.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.239+0000 7f83abfff700 1 -- 172.21.15.114:0/3456833996 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83b8031520 con 0x5555e34f3120 2024-05-27T07:56:05.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.239+0000 7f83abfff700 1 -- 172.21.15.114:0/3456833996 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b8035080 con 0x5555e34f3120 2024-05-27T07:56:05.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.240+0000 7f83d7246b80 1 -- 172.21.15.114:0/3456833996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f3120 msgr2=0x5555e34f2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:05.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.240+0000 7f83d7246b80 1 --2- 172.21.15.114:0/3456833996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f3120 0x5555e34f2be0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f83b8000c00 tx=0x7f83b8012300 comp rx=0 tx=0).stop 2024-05-27T07:56:05.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.245+0000 7f83d7246b80 1 -- 172.21.15.114:0/3456833996 shutdown_connections 2024-05-27T07:56:05.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.245+0000 7f83d7246b80 1 --2- 172.21.15.114:0/3456833996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34fece0 0x5555e3536a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.245+0000 7f83d7246b80 1 --2- 172.21.15.114:0/3456833996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f3120 0x5555e34f2be0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.245+0000 7f83d7246b80 1 --2- 172.21.15.114:0/3456833996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34f9070 0x5555e34f6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.245+0000 7f83d7246b80 1 -- 172.21.15.114:0/3456833996 >> 172.21.15.114:0/3456833996 conn(0x5555e3376770 msgr2=0x5555e34f6490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:05.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.246+0000 7f83d7246b80 1 -- 172.21.15.114:0/3456833996 shutdown_connections 2024-05-27T07:56:05.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.246+0000 7f83d7246b80 1 -- 172.21.15.114:0/3456833996 wait complete. 2024-05-27T07:56:05.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.246+0000 7f83d7246b80 1 -- start start 2024-05-27T07:56:05.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.247+0000 7f83d7246b80 1 Processor -- start 2024-05-27T07:56:05.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.247+0000 7f83d7246b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f3120 0x5555e34f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.248+0000 7f83d7246b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f9070 0x5555e34f2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.248+0000 7f83d7246b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34fece0 0x5555e34f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.248+0000 7f83c1d9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f3120 0x5555e34f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.248+0000 7f83c1d9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f3120 0x5555e34f1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:48800/0 (socket says 172.21.15.114:48800) 2024-05-27T07:56:05.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.248+0000 7f83c1d9b700 1 -- 172.21.15.114:0/2122238633 learned_addr learned my addr 172.21.15.114:0/2122238633 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:05.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.248+0000 7f83d7246b80 1 -- 172.21.15.114:0/2122238633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83a400d580 con 0x5555e34fece0 2024-05-27T07:56:05.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.248+0000 7f83d7246b80 1 -- 172.21.15.114:0/2122238633 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f83a400cf30 con 0x5555e34f3120 2024-05-27T07:56:05.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.248+0000 7f83d7246b80 1 -- 172.21.15.114:0/2122238633 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83a400dc50 con 0x5555e34f9070 2024-05-27T07:56:05.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.249+0000 7f83c1d9b700 1 -- 172.21.15.114:0/2122238633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f9070 msgr2=0x5555e34f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:05.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.249+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/2122238633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f9070 0x5555e34f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.249+0000 7f83c1d9b700 1 -- 172.21.15.114:0/2122238633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34fece0 msgr2=0x5555e34f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:05.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.249+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/2122238633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34fece0 0x5555e34f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.249+0000 7f83c1d9b700 1 -- 172.21.15.114:0/2122238633 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555e3598940 con 0x5555e34f3120 2024-05-27T07:56:05.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.250+0000 7f83c1d9b700 1 --2- 172.21.15.114:0/2122238633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f3120 0x5555e34f1ff0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f83b8015010 tx=0x7f83b8006ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18c7b260da018e1c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:05.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.250+0000 7f83d7246b80 10 client.0 _ll_register_callbacks cb 0x5555e33a75b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:05.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.250+0000 7f83d7246b80 1 -- 172.21.15.114:0/2122238633 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5555e341c4a0 con 0x5555e34f3120 2024-05-27T07:56:05.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.252+0000 7f83a9ffb700 10 client.28586 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:56:05.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.257+0000 7f83a9ffb700 1 -- 172.21.15.114:0/2122238633 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b8035070 con 0x5555e34f3120 2024-05-27T07:56:05.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.257+0000 7f83a9ffb700 1 -- 172.21.15.114:0/2122238633 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9642+0+0 (secure 0 0 0) 0x7f83b8031ad0 con 0x5555e34f3120 2024-05-27T07:56:05.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.262+0000 7f83a9ffb700 1 -- 172.21.15.114:0/2122238633 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f83b8049050 con 0x5555e34f3120 2024-05-27T07:56:05.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.262+0000 7f83d7246b80 10 client.28586 fetch_fsmap learned FSMap version 91 2024-05-27T07:56:05.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.262+0000 7f83d7246b80 1 -- 172.21.15.114:0/2122238633 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({fsmap=91}) -- 0x5555e3599890 con 0x5555e34f3120 2024-05-27T07:56:05.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.263+0000 7f83a9ffb700 1 -- 172.21.15.114:0/2122238633 <== mon.2 v2:172.21.15.163:3300/0 4 ==== fsmap(e 91) ==== 4700+0+0 (secure 0 0 0) 0x7f83b8015040 con 0x5555e34f3120 2024-05-27T07:56:05.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.263+0000 7f83d7246b80 10 client.28586 fetch_fsmap finished waiting for FSMap version 91 2024-05-27T07:56:05.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.263+0000 7f83d7246b80 10 client.28586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:05.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.263+0000 7f83d7246b80 20 client.28586 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:05.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.263+0000 7f83d7246b80 1 --2- 172.21.15.114:0/2122238633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5555e3599eb0 0x5555e359c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:05.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.263+0000 7f83d7246b80 4 client.28586 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:56:05.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.263+0000 7f83d7246b80 1 -- 172.21.15.114:0/2122238633 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5555e3293250 con 0x5555e3599eb0 2024-05-27T07:56:05.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.267+0000 7f83c259c700 1 --2- 172.21.15.114:0/2122238633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5555e3599eb0 0x5555e359c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:05.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.279+0000 7f83c259c700 1 --2- 172.21.15.114:0/2122238633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5555e3599eb0 0x5555e359c3c0 crc :-1 s=READY pgs=449 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b752c59e594935 server_cookie=11737d51a483419c in_seq=0 out_seq=0 2024-05-27T07:56:05.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.281+0000 7f83a9ffb700 10 client.28586 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:05.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.281+0000 7f83a9ffb700 1 -- 172.21.15.114:0/2122238633 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5555e3293250 con 0x5555e3599eb0 2024-05-27T07:56:05.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.281+0000 7f83a9ffb700 10 client.28586 handle_command_reply: tid=0 2024-05-27T07:56:05.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.283+0000 7f83d7246b80 1 client.28586 shutdown 2024-05-27T07:56:05.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.288+0000 7f83d7246b80 1 -- 172.21.15.114:0/2122238633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f3120 msgr2=0x5555e34f1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:05.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.288+0000 7f83d7246b80 1 --2- 172.21.15.114:0/2122238633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f3120 0x5555e34f1ff0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f83b8015010 tx=0x7f83b8006ac0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.293+0000 7f83d7246b80 1 -- 172.21.15.114:0/2122238633 shutdown_connections 2024-05-27T07:56:05.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.293+0000 7f83d7246b80 1 --2- 172.21.15.114:0/2122238633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5555e34fece0 0x5555e34f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.293+0000 7f83d7246b80 1 --2- 172.21.15.114:0/2122238633 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5555e3599eb0 0x5555e359c3c0 crc :-1 s=READY pgs=449 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.293+0000 7f83d7246b80 1 --2- 172.21.15.114:0/2122238633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5555e34f3120 0x5555e34f1ff0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.293+0000 7f83d7246b80 1 --2- 172.21.15.114:0/2122238633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5555e34f9070 0x5555e34f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.293+0000 7f83d7246b80 1 -- 172.21.15.114:0/2122238633 >> 172.21.15.114:0/2122238633 conn(0x5555e3376770 msgr2=0x5555e34e74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:05.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.295+0000 7f83a9ffb700 0 client.28586 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:05.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.295+0000 7f83d7246b80 1 -- 172.21.15.114:0/2122238633 shutdown_connections 2024-05-27T07:56:05.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.297+0000 7f83d7246b80 1 -- 172.21.15.114:0/2122238633 wait complete. 2024-05-27T07:56:05.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.298+0000 7f83d7246b80 20 client.28586 trim_cache size 0 max 16384 2024-05-27T07:56:05.299 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:56:05.299 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:56:05.299 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:56:05.299 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:56:05.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.299+0000 7f83c8831700 1 -- 172.21.15.114:0/3804205132 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83ac0b2c00 msgr2=0x7f83ac0b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:05.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.299+0000 7f83c8831700 1 --2- 172.21.15.114:0/3804205132 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83ac0b2c00 0x7f83ac0b5110 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0e6fd0 tx=0x7f83bc0a6550 comp rx=0 tx=0).stop 2024-05-27T07:56:05.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.299+0000 7f83c8831700 1 -- 172.21.15.114:0/3804205132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c411d920 msgr2=0x7f83c4117b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:05.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.299+0000 7f83c8831700 1 --2- 172.21.15.114:0/3804205132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c411d920 0x7f83c4117b40 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0062a0 tx=0x7f83bc004da0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.299+0000 7f83c8831700 1 -- 172.21.15.114:0/3804205132 shutdown_connections 2024-05-27T07:56:05.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.299+0000 7f83c8831700 1 --2- 172.21.15.114:0/3804205132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c411d920 0x7f83c4117b40 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.299+0000 7f83c8831700 1 --2- 172.21.15.114:0/3804205132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83c410a920 0x7f83c411d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.299+0000 7f83c8831700 1 --2- 172.21.15.114:0/3804205132 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f83ac0b2c00 0x7f83ac0b5110 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.299+0000 7f83c8831700 1 --2- 172.21.15.114:0/3804205132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83c410b270 0x7f83c41174a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.299+0000 7f83c8831700 1 -- 172.21.15.114:0/3804205132 >> 172.21.15.114:0/3804205132 conn(0x7f83c406f4c0 msgr2=0x7f83c411dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.300+0000 7f83c8831700 1 -- 172.21.15.114:0/3804205132 shutdown_connections 2024-05-27T07:56:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:05.300+0000 7f83c8831700 1 -- 172.21.15.114:0/3804205132 wait complete. 2024-05-27T07:56:05.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 34250 11.94 MB/sec execute 17 sec latency 1280.028 ms 2024-05-27T07:56:05.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:05 smithi114 ceph-mon[32038]: pgmap v558: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 323 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.4 MiB/s wr, 364 op/s 2024-05-27T07:56:06.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:05 smithi163 ceph-mon[36451]: pgmap v558: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 323 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.4 MiB/s wr, 364 op/s 2024-05-27T07:56:06.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:05 smithi186 ceph-mon[36647]: pgmap v558: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 323 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.4 MiB/s wr, 364 op/s 2024-05-27T07:56:06.454 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:56:06.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:56:06.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 34301 11.47 MB/sec execute 18 sec latency 1593.259 ms 2024-05-27T07:56:07.003 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:56:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.337+0000 7f65daffd640 1 -- 172.21.15.114:0/4072966387 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7f65ec014270 con 0x55ab7c0711c0 2024-05-27T07:56:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.337+0000 7f65daffd640 10 client.28711 handle_command_reply: tid=0 2024-05-27T07:56:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.337+0000 7f6609170740 1 client.28711 shutdown 2024-05-27T07:56:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.338+0000 7f6609170740 1 -- 172.21.15.114:0/4072966387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0dd1b0 msgr2=0x55ab7c4a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.338+0000 7f6609170740 1 --2- 172.21.15.114:0/4072966387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c4a1180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f65f4004af0 tx=0x7f65f4009420 comp rx=0 tx=0).stop 2024-05-27T07:56:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.338+0000 7f6609170740 1 -- 172.21.15.114:0/4072966387 shutdown_connections 2024-05-27T07:56:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.338+0000 7f6609170740 1 --2- 172.21.15.114:0/4072966387 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ab7c0711c0 0x55ab7c5274d0 crc :-1 s=READY pgs=447 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.338+0000 7f6609170740 1 --2- 172.21.15.114:0/4072966387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab7c0dd1b0 0x55ab7c4a1180 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.338+0000 7f6609170740 1 --2- 172.21.15.114:0/4072966387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ab7c0bc4d0 0x55ab7c4a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.338+0000 7f6609170740 1 --2- 172.21.15.114:0/4072966387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ab7c0b42f0 0x55ab7c4a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.338+0000 7f6609170740 1 -- 172.21.15.114:0/4072966387 >> 172.21.15.114:0/4072966387 conn(0x55ab7c0d1300 msgr2=0x55ab7c489fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.338+0000 7f65daffd640 0 client.28711 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.338+0000 7f6609170740 1 -- 172.21.15.114:0/4072966387 shutdown_connections 2024-05-27T07:56:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.338+0000 7f6609170740 1 -- 172.21.15.114:0/4072966387 wait complete. 2024-05-27T07:56:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.338+0000 7f6609170740 20 client.28711 trim_cache size 0 max 16384 2024-05-27T07:56:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:56:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:56:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:56:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.339+0000 7f66021eb640 1 -- 172.21.15.114:0/1582580957 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f65e40b2aa0 msgr2=0x7f65e40b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.339+0000 7f66021eb640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f65e40b2aa0 0x7f65e40b4fb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f65f4046040 tx=0x7f65f402f3d0 comp rx=0 tx=0).stop 2024-05-27T07:56:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.339+0000 7f66021eb640 1 -- 172.21.15.114:0/1582580957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65fc112320 msgr2=0x7f65fc1a1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.339+0000 7f66021eb640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65fc112320 0x7f65fc1a1350 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f65f000e9f0 tx=0x7f65f000eeb0 comp rx=0 tx=0).stop 2024-05-27T07:56:07.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.343+0000 7f66021eb640 1 -- 172.21.15.114:0/1582580957 shutdown_connections 2024-05-27T07:56:07.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.343+0000 7f66021eb640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f65e40b2aa0 0x7f65e40b4fb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:07.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.343+0000 7f66021eb640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65fc114c80 0x7f65fc1a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:07.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.343+0000 7f66021eb640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65fc112320 0x7f65fc1a1350 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:07.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.343+0000 7f66021eb640 1 --2- 172.21.15.114:0/1582580957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc075480 0x7f65fc1a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.343+0000 7f66021eb640 1 -- 172.21.15.114:0/1582580957 >> 172.21.15.114:0/1582580957 conn(0x7f65fc0793c0 msgr2=0x7f65fc112730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.344+0000 7f66021eb640 1 -- 172.21.15.114:0/1582580957 shutdown_connections 2024-05-27T07:56:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:07.344+0000 7f66021eb640 1 -- 172.21.15.114:0/1582580957 wait complete. 2024-05-27T07:56:07.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 34328 10.90 MB/sec execute 19 sec latency 2151.088 ms 2024-05-27T07:56:08.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:07 smithi114 ceph-mon[32038]: pgmap v559: 129 pgs: 15 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 73 active+clean; 330 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.3 MiB/s wr, 474 op/s 2024-05-27T07:56:08.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:07 smithi163 ceph-mon[36451]: pgmap v559: 129 pgs: 15 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 73 active+clean; 330 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.3 MiB/s wr, 474 op/s 2024-05-27T07:56:08.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:07 smithi186 ceph-mon[36647]: pgmap v559: 129 pgs: 15 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 73 active+clean; 330 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.3 MiB/s wr, 474 op/s 2024-05-27T07:56:08.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 34429 10.78 MB/sec execute 20 sec latency 2589.570 ms 2024-05-27T07:56:09.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:08 smithi163 ceph-mon[36451]: pgmap v560: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 331 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.2 MiB/s wr, 545 op/s 2024-05-27T07:56:09.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:08 smithi186 ceph-mon[36647]: pgmap v560: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 331 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.2 MiB/s wr, 545 op/s 2024-05-27T07:56:09.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:08 smithi114 ceph-mon[32038]: pgmap v560: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 331 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.2 MiB/s wr, 545 op/s 2024-05-27T07:56:09.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 34483 10.39 MB/sec execute 21 sec latency 1561.602 ms 2024-05-27T07:56:10.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 34645 10.54 MB/sec execute 22 sec latency 2002.812 ms 2024-05-27T07:56:11.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.052+0000 7ff9413cb700 1 -- 172.21.15.114:0/784526128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c10a7b0 msgr2=0x7ff93c10abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:11.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.052+0000 7ff9413cb700 1 --2- 172.21.15.114:0/784526128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c10a7b0 0x7ff93c10abc0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7ff92c009990 tx=0x7ff92c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:56:11.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.053+0000 7ff9413cb700 1 -- 172.21.15.114:0/784526128 shutdown_connections 2024-05-27T07:56:11.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.053+0000 7ff9413cb700 1 --2- 172.21.15.114:0/784526128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff93c11d920 0x7ff93c11fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:11.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.053+0000 7ff9413cb700 1 --2- 172.21.15.114:0/784526128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c10a7b0 0x7ff93c10abc0 secure :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7ff92c009990 tx=0x7ff92c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:56:11.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.053+0000 7ff9413cb700 1 --2- 172.21.15.114:0/784526128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff93c10b100 0x7ff93c11d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:11.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.053+0000 7ff9413cb700 1 -- 172.21.15.114:0/784526128 >> 172.21.15.114:0/784526128 conn(0x7ff93c06f280 msgr2=0x7ff93c06f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:11.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.056+0000 7ff9413cb700 1 -- 172.21.15.114:0/784526128 shutdown_connections 2024-05-27T07:56:11.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.059+0000 7ff9413cb700 1 -- 172.21.15.114:0/784526128 wait complete. 2024-05-27T07:56:11.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.060+0000 7ff9413cb700 1 Processor -- start 2024-05-27T07:56:11.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.060+0000 7ff9413cb700 1 -- start start 2024-05-27T07:56:11.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.060+0000 7ff9413cb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c10b100 0x7ff93c114f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:11.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.060+0000 7ff9413cb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff93c11d920 0x7ff93c1154c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:11.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.060+0000 7ff9413cb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff93c115a00 0x7ff93c119980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:11.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.060+0000 7ff9413cb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff93c123100 con 0x7ff93c10b100 2024-05-27T07:56:11.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.060+0000 7ff9413cb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff93c122f80 con 0x7ff93c11d920 2024-05-27T07:56:11.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.060+0000 7ff9413cb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff93c123280 con 0x7ff93c115a00 2024-05-27T07:56:11.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.061+0000 7ff93b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff93c11d920 0x7ff93c1154c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:11.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.061+0000 7ff93b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff93c11d920 0x7ff93c1154c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:48820/0 (socket says 172.21.15.114:48820) 2024-05-27T07:56:11.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.061+0000 7ff93b7fe700 1 -- 172.21.15.114:0/686613072 learned_addr learned my addr 172.21.15.114:0/686613072 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:11.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.061+0000 7ff940bca700 1 --2- 172.21.15.114:0/686613072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff93c115a00 0x7ff93c119980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:11.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.063+0000 7ff940bca700 1 -- 172.21.15.114:0/686613072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff93c11d920 msgr2=0x7ff93c1154c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:11.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.063+0000 7ff940bca700 1 --2- 172.21.15.114:0/686613072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff93c11d920 0x7ff93c1154c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:11.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.063+0000 7ff940bca700 1 -- 172.21.15.114:0/686613072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c10b100 msgr2=0x7ff93c114f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:56:11.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.063+0000 7ff940bca700 1 --2- 172.21.15.114:0/686613072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c10b100 0x7ff93c114f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:11.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.063+0000 7ff940bca700 1 -- 172.21.15.114:0/686613072 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff93c119ec0 con 0x7ff93c115a00 2024-05-27T07:56:11.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.063+0000 7ff940bca700 1 --2- 172.21.15.114:0/686613072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff93c115a00 0x7ff93c119980 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff93400ab40 tx=0x7ff93400ae50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:11.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.066+0000 7ff9397fa700 1 -- 172.21.15.114:0/686613072 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934017020 con 0x7ff93c115a00 2024-05-27T07:56:11.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.066+0000 7ff9413cb700 1 -- 172.21.15.114:0/686613072 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff93c11a1b0 con 0x7ff93c115a00 2024-05-27T07:56:11.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.066+0000 7ff9413cb700 1 -- 172.21.15.114:0/686613072 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff93c11a6f0 con 0x7ff93c115a00 2024-05-27T07:56:11.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.066+0000 7ff9397fa700 1 -- 172.21.15.114:0/686613072 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff934004bb0 con 0x7ff93c115a00 2024-05-27T07:56:11.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.066+0000 7ff9397fa700 1 -- 172.21.15.114:0/686613072 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934005310 con 0x7ff93c115a00 2024-05-27T07:56:11.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.069+0000 7ff9413cb700 1 -- 172.21.15.114:0/686613072 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff93c0753a0 con 0x7ff93c115a00 2024-05-27T07:56:11.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.076+0000 7ff9397fa700 1 -- 172.21.15.114:0/686613072 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ff93400c640 con 0x7ff93c115a00 2024-05-27T07:56:11.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.077+0000 7ff9397fa700 1 --2- 172.21.15.114:0/686613072 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff9240b2bc0 0x7ff9240b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:11.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.077+0000 7ff9397fa700 1 -- 172.21.15.114:0/686613072 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 9642+0+0 (secure 0 0 0) 0x7ff9340e2d40 con 0x7ff93c115a00 2024-05-27T07:56:11.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.078+0000 7ff93bfff700 1 --2- 172.21.15.114:0/686613072 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff9240b2bc0 0x7ff9240b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:11.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.079+0000 7ff93bfff700 1 --2- 172.21.15.114:0/686613072 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff9240b2bc0 0x7ff9240b50d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff92c005950 tx=0x7ff92c019040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:11.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.085+0000 7ff9397fa700 1 -- 172.21.15.114:0/686613072 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff934021020 con 0x7ff93c115a00 2024-05-27T07:56:11.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.579+0000 7ff9413cb700 1 -- 172.21.15.114:0/686613072 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff93c10be40 con 0x7ff93c115a00 2024-05-27T07:56:11.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.581+0000 7ff9397fa700 1 -- 172.21.15.114:0/686613072 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6865 (secure 0 0 0) 0x7ff9340b0020 con 0x7ff93c115a00 2024-05-27T07:56:11.583 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:56:11.583 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3861955816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3861955816},{"type":"v1","addr":"172.21.15.114:6841","nonce":3861955816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":91,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:55:51.732668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_28556":{"gid":28556,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/2578326696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":2578326696},{"type":"v1","addr":"172.21.15.163:6833","nonce":2578326696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:56:11.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.585+0000 7ff9413cb700 1 -- 172.21.15.114:0/686613072 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff9240b2bc0 msgr2=0x7ff9240b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:11.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.585+0000 7ff9413cb700 1 --2- 172.21.15.114:0/686613072 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff9240b2bc0 0x7ff9240b50d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff92c005950 tx=0x7ff92c019040 comp rx=0 tx=0).stop 2024-05-27T07:56:11.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.585+0000 7ff9413cb700 1 -- 172.21.15.114:0/686613072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff93c115a00 msgr2=0x7ff93c119980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:11.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.585+0000 7ff9413cb700 1 --2- 172.21.15.114:0/686613072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff93c115a00 0x7ff93c119980 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff93400ab40 tx=0x7ff93400ae50 comp rx=0 tx=0).stop 2024-05-27T07:56:11.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.587+0000 7ff9413cb700 1 -- 172.21.15.114:0/686613072 shutdown_connections 2024-05-27T07:56:11.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.587+0000 7ff9413cb700 1 --2- 172.21.15.114:0/686613072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff93c115a00 0x7ff93c119980 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:11.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.588+0000 7ff9413cb700 1 --2- 172.21.15.114:0/686613072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c10b100 0x7ff93c114f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:11.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.588+0000 7ff9413cb700 1 --2- 172.21.15.114:0/686613072 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff9240b2bc0 0x7ff9240b50d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:11.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.588+0000 7ff9413cb700 1 --2- 172.21.15.114:0/686613072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff93c11d920 0x7ff93c1154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:11.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.588+0000 7ff9413cb700 1 -- 172.21.15.114:0/686613072 >> 172.21.15.114:0/686613072 conn(0x7ff93c06f280 msgr2=0x7ff93c0724e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:11.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.590+0000 7ff9413cb700 1 -- 172.21.15.114:0/686613072 shutdown_connections 2024-05-27T07:56:11.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:11.590+0000 7ff9413cb700 1 -- 172.21.15.114:0/686613072 wait complete. 2024-05-27T07:56:11.593 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 91 2024-05-27T07:56:11.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 34966 10.61 MB/sec execute 23 sec latency 1529.487 ms 2024-05-27T07:56:11.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:11 smithi163 ceph-mon[36451]: pgmap v561: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 322 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 564 op/s 2024-05-27T07:56:11.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:11 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/686613072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:56:11.977 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:11 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:56:11.977 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:11 smithi163 systemd[1]: Stopped Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:56:11.977 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:11 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.702s CPU time. 2024-05-27T07:56:11.977 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:11 smithi163 systemd[1]: Starting Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:56:11.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:11 smithi186 ceph-mon[36647]: pgmap v561: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 322 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 564 op/s 2024-05-27T07:56:11.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:11 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/686613072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:56:12.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:11 smithi114 ceph-mon[32038]: pgmap v561: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 322 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 564 op/s 2024-05-27T07:56:12.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:11 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/686613072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:56:12.324 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:11 smithi163 podman[75213]: 2024-05-27 07:56:11.898490427 +0000 UTC m=+0.031986067 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:56:12.324 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:12 smithi163 podman[75213]: 2024-05-27 07:56:12.029092053 +0000 UTC m=+0.162587681 container create ea5ac6f2e6d84bc13f414c956ccc01a08fb11d165669b84702731b26aa773379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True) 2024-05-27T07:56:12.324 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:12 smithi163 podman[75213]: 2024-05-27 07:56:12.323673251 +0000 UTC m=+0.457168874 container init ea5ac6f2e6d84bc13f414c956ccc01a08fb11d165669b84702731b26aa773379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521) 2024-05-27T07:56:12.599 INFO:tasks.workunit.client.0.smithi114.stdout: 10 35118 10.56 MB/sec execute 24 sec latency 1029.427 ms 2024-05-27T07:56:12.691 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:12 smithi163 podman[75213]: 2024-05-27 07:56:12.328779297 +0000 UTC m=+0.462274932 container start ea5ac6f2e6d84bc13f414c956ccc01a08fb11d165669b84702731b26aa773379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-27T07:56:12.691 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:12 smithi163 ceph-mds[75232]: -- 172.21.15.163:0/2314642270 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563ff41f1a40 con 0x563ff4239180 2024-05-27T07:56:12.691 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:12 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: starting mds.k at 2024-05-27T07:56:12.691 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:12 smithi163 bash[75213]: ea5ac6f2e6d84bc13f414c956ccc01a08fb11d165669b84702731b26aa773379 2024-05-27T07:56:12.691 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:12 smithi163 systemd[1]: Started Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:56:12.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:56:12.948 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:12 smithi114 ceph-mon[32038]: daemon mds.k restarted 2024-05-27T07:56:12.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:12 smithi163 ceph-mon[36451]: daemon mds.k restarted 2024-05-27T07:56:13.000 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:12 smithi186 ceph-mon[36647]: daemon mds.k restarted 2024-05-27T07:56:13.432 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:56:13.438 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff487763700 time 2024-05-27T07:56:13.032728+0000 2024-05-27T07:56:13.438 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:56:13.438 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:13.438 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff499fa8ba6] 2024-05-27T07:56:13.438 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff499fa8dd1] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563ff304423d] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563ff302f971] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 5: (SnapRealm::check_cache() const+0x13e) [0x563ff302dfa0] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563ff302e651] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563ff2fc3347] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563ff2cbf87e] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563ff2cc4f2f] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563ff2ccf969] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 11: (MDLog::_replay_thread()+0x1f38) [0x563ff306848c] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563ff30719cb] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 13: (Thread::entry_wrapper()+0x43) [0x7ff499f7772d] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff499f77749] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff498b1b1ca] 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 16: clone() 2024-05-27T07:56:13.439 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.034+0000 7ff487763700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff487763700 time 2024-05-27T07:56:13.032728+0000 2024-05-27T07:56:13.440 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:56:13.440 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.440 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:13.440 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff499fa8ba6] 2024-05-27T07:56:13.440 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff499fa8dd1] 2024-05-27T07:56:13.440 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563ff304423d] 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563ff302f971] 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 5: (SnapRealm::check_cache() const+0x13e) [0x563ff302dfa0] 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563ff302e651] 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563ff2fc3347] 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563ff2cbf87e] 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563ff2cc4f2f] 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563ff2ccf969] 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 11: (MDLog::_replay_thread()+0x1f38) [0x563ff306848c] 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563ff30719cb] 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 13: (Thread::entry_wrapper()+0x43) [0x7ff499f7772d] 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff499f77749] 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff498b1b1ca] 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 16: clone() 2024-05-27T07:56:13.441 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.442 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: *** Caught signal (Aborted) ** 2024-05-27T07:56:13.442 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: in thread 7ff487763700 thread_name:md_log_replay 2024-05-27T07:56:13.442 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:13.442 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff498b25d20] 2024-05-27T07:56:13.442 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2: gsignal() 2024-05-27T07:56:13.442 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 3: abort() 2024-05-27T07:56:13.442 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff499fa8ced] 2024-05-27T07:56:13.442 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff499fa8dd1] 2024-05-27T07:56:13.442 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563ff304423d] 2024-05-27T07:56:13.442 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563ff302f971] 2024-05-27T07:56:13.442 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 8: (SnapRealm::check_cache() const+0x13e) [0x563ff302dfa0] 2024-05-27T07:56:13.442 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563ff302e651] 2024-05-27T07:56:13.442 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563ff2fc3347] 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563ff2cbf87e] 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563ff2cc4f2f] 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563ff2ccf969] 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 14: (MDLog::_replay_thread()+0x1f38) [0x563ff306848c] 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563ff30719cb] 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 16: (Thread::entry_wrapper()+0x43) [0x7ff499f7772d] 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff499f77749] 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff498b1b1ca] 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 19: clone() 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.036+0000 7ff487763700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: in thread 7ff487763700 thread_name:md_log_replay 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:13.443 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff498b25d20] 2024-05-27T07:56:13.444 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2: gsignal() 2024-05-27T07:56:13.444 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 3: abort() 2024-05-27T07:56:13.444 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff499fa8ced] 2024-05-27T07:56:13.444 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff499fa8dd1] 2024-05-27T07:56:13.444 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563ff304423d] 2024-05-27T07:56:13.444 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563ff302f971] 2024-05-27T07:56:13.444 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 8: (SnapRealm::check_cache() const+0x13e) [0x563ff302dfa0] 2024-05-27T07:56:13.444 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563ff302e651] 2024-05-27T07:56:13.444 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563ff2fc3347] 2024-05-27T07:56:13.445 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563ff2cbf87e] 2024-05-27T07:56:13.445 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563ff2cc4f2f] 2024-05-27T07:56:13.445 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563ff2ccf969] 2024-05-27T07:56:13.445 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 14: (MDLog::_replay_thread()+0x1f38) [0x563ff306848c] 2024-05-27T07:56:13.445 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563ff30719cb] 2024-05-27T07:56:13.445 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 16: (Thread::entry_wrapper()+0x43) [0x7ff499f7772d] 2024-05-27T07:56:13.445 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff499f77749] 2024-05-27T07:56:13.445 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff498b1b1ca] 2024-05-27T07:56:13.445 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 19: clone() 2024-05-27T07:56:13.445 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: -1> 2024-05-27T07:56:13.034+0000 7ff487763700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff487763700 time 2024-05-27T07:56:13.032728+0000 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff499fa8ba6] 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff499fa8dd1] 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563ff304423d] 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563ff302f971] 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 5: (SnapRealm::check_cache() const+0x13e) [0x563ff302dfa0] 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563ff302e651] 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563ff2fc3347] 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563ff2cbf87e] 2024-05-27T07:56:13.446 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563ff2cc4f2f] 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563ff2ccf969] 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 11: (MDLog::_replay_thread()+0x1f38) [0x563ff306848c] 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563ff30719cb] 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 13: (Thread::entry_wrapper()+0x43) [0x7ff499f7772d] 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff499f77749] 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff498b1b1ca] 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 16: clone() 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 0> 2024-05-27T07:56:13.036+0000 7ff487763700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: in thread 7ff487763700 thread_name:md_log_replay 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff498b25d20] 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2: gsignal() 2024-05-27T07:56:13.447 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 3: abort() 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff499fa8ced] 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff499fa8dd1] 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563ff304423d] 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563ff302f971] 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 8: (SnapRealm::check_cache() const+0x13e) [0x563ff302dfa0] 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563ff302e651] 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563ff2fc3347] 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563ff2cbf87e] 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563ff2cc4f2f] 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563ff2ccf969] 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 14: (MDLog::_replay_thread()+0x1f38) [0x563ff306848c] 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563ff30719cb] 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 16: (Thread::entry_wrapper()+0x43) [0x7ff499f7772d] 2024-05-27T07:56:13.448 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff499f77749] 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff498b1b1ca] 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 19: clone() 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: -2> 2024-05-27T07:56:13.034+0000 7ff487763700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff487763700 time 2024-05-27T07:56:13.032728+0000 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff499fa8ba6] 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff499fa8dd1] 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563ff304423d] 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563ff302f971] 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 5: (SnapRealm::check_cache() const+0x13e) [0x563ff302dfa0] 2024-05-27T07:56:13.449 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563ff302e651] 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563ff2fc3347] 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563ff2cbf87e] 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563ff2cc4f2f] 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563ff2ccf969] 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 11: (MDLog::_replay_thread()+0x1f38) [0x563ff306848c] 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563ff30719cb] 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 13: (Thread::entry_wrapper()+0x43) [0x7ff499f7772d] 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff499f77749] 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff498b1b1ca] 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 16: clone() 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: -1> 2024-05-27T07:56:13.036+0000 7ff487763700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: in thread 7ff487763700 thread_name:md_log_replay 2024-05-27T07:56:13.450 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff498b25d20] 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2: gsignal() 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 3: abort() 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff499fa8ced] 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff499fa8dd1] 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563ff304423d] 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563ff302f971] 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 8: (SnapRealm::check_cache() const+0x13e) [0x563ff302dfa0] 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563ff302e651] 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563ff2fc3347] 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563ff2cbf87e] 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563ff2cc4f2f] 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563ff2ccf969] 2024-05-27T07:56:13.451 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 14: (MDLog::_replay_thread()+0x1f38) [0x563ff306848c] 2024-05-27T07:56:13.452 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563ff30719cb] 2024-05-27T07:56:13.452 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 16: (Thread::entry_wrapper()+0x43) [0x7ff499f7772d] 2024-05-27T07:56:13.452 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff499f77749] 2024-05-27T07:56:13.452 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff498b1b1ca] 2024-05-27T07:56:13.452 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 19: clone() 2024-05-27T07:56:13.452 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:56:13.452 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k[75228]: 2024-05-27T07:56:13.452 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 podman[75263]: 2024-05-27 07:56:13.117780193 +0000 UTC m=+0.024465621 container died ea5ac6f2e6d84bc13f414c956ccc01a08fb11d165669b84702731b26aa773379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-27T07:56:13.600 INFO:tasks.workunit.client.0.smithi114.stdout: 10 35230 10.50 MB/sec execute 25 sec latency 1473.639 ms 2024-05-27T07:56:13.877 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:13 smithi114 ceph-mon[32038]: pgmap v562: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 309 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 538 op/s 2024-05-27T07:56:13.877 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:13 smithi114 ceph-mon[32038]: osdmap e148: 12 total, 12 up, 12 in 2024-05-27T07:56:13.877 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:13 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.163:6832/4080983086,v1:172.21.15.163:6833/4080983086] up:boot 2024-05-27T07:56:13.877 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:13 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:56:13.877 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:13 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:56:13.877 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:13 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:56:13.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:13 smithi163 ceph-mon[36451]: pgmap v562: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 309 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 538 op/s 2024-05-27T07:56:13.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:13 smithi163 ceph-mon[36451]: osdmap e148: 12 total, 12 up, 12 in 2024-05-27T07:56:13.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:13 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.163:6832/4080983086,v1:172.21.15.163:6833/4080983086] up:boot 2024-05-27T07:56:13.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:13 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:56:13.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:13 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:56:13.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:13 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:56:13.978 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 podman[75263]: 2024-05-27 07:56:13.823050253 +0000 UTC m=+0.729735670 container remove ea5ac6f2e6d84bc13f414c956ccc01a08fb11d165669b84702731b26aa773379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-k, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T07:56:13.978 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:13 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:56:13.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:13 smithi186 ceph-mon[36647]: pgmap v562: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 309 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 538 op/s 2024-05-27T07:56:13.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:13 smithi186 ceph-mon[36647]: osdmap e148: 12 total, 12 up, 12 in 2024-05-27T07:56:13.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:13 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.163:6832/4080983086,v1:172.21.15.163:6833/4080983086] up:boot 2024-05-27T07:56:13.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:13 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 5 up:standby 2024-05-27T07:56:13.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:13 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:56:13.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:13 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 4 up:standby 2024-05-27T07:56:14.600 INFO:tasks.workunit.client.0.smithi114.stdout: 10 35320 10.39 MB/sec execute 26 sec latency 1075.319 ms 2024-05-27T07:56:14.726 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:14 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:56:15.600 INFO:tasks.workunit.client.0.smithi114.stdout: 10 35434 10.37 MB/sec execute 27 sec latency 1480.598 ms 2024-05-27T07:56:15.761 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:15 smithi114 ceph-mon[32038]: pgmap v564: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 312 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 524 op/s 2024-05-27T07:56:15.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:15 smithi163 ceph-mon[36451]: pgmap v564: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 312 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 524 op/s 2024-05-27T07:56:15.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:15 smithi186 ceph-mon[36647]: pgmap v564: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 312 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 524 op/s 2024-05-27T07:56:16.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.252+0000 7f0104b34700 1 -- 172.21.15.114:0/1036829496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f010010a3d0 msgr2=0x7f010010a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:16.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.252+0000 7f0104b34700 1 --2- 172.21.15.114:0/1036829496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f010010a3d0 0x7f010010a7e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f00f40098f0 tx=0x7f00f4009c00 comp rx=0 tx=0).stop 2024-05-27T07:56:16.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.252+0000 7f0104b34700 1 -- 172.21.15.114:0/1036829496 shutdown_connections 2024-05-27T07:56:16.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.252+0000 7f0104b34700 1 --2- 172.21.15.114:0/1036829496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0100103120 0x7f0100111110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.252+0000 7f0104b34700 1 --2- 172.21.15.114:0/1036829496 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f010010a3d0 0x7f010010a7e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.252+0000 7f0104b34700 1 --2- 172.21.15.114:0/1036829496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01001027d0 0x7f0100102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.252+0000 7f0104b34700 1 -- 172.21.15.114:0/1036829496 >> 172.21.15.114:0/1036829496 conn(0x7f01000fe6e0 msgr2=0x7f0100100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:16.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.253+0000 7f0104b34700 1 -- 172.21.15.114:0/1036829496 shutdown_connections 2024-05-27T07:56:16.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.253+0000 7f0104b34700 1 -- 172.21.15.114:0/1036829496 wait complete. 2024-05-27T07:56:16.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.254+0000 7f0104b34700 1 Processor -- start 2024-05-27T07:56:16.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.254+0000 7f0104b34700 1 -- start start 2024-05-27T07:56:16.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.255+0000 7f0104b34700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01001027d0 0x7f010010e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.255+0000 7f0104b34700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0100103120 0x7f010010ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.255+0000 7f0104b34700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f010010a3d0 0x7f01001ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.255+0000 7f0104b34700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0100105440 con 0x7f01001027d0 2024-05-27T07:56:16.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.255+0000 7f0104b34700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f01001052c0 con 0x7f0100103120 2024-05-27T07:56:16.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.255+0000 7f0104b34700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f01001055c0 con 0x7f010010a3d0 2024-05-27T07:56:16.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.255+0000 7f00fdd9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0100103120 0x7f010010ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.255+0000 7f00fe59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01001027d0 0x7f010010e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.255+0000 7f00fdd9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0100103120 0x7f010010ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:59442/0 (socket says 172.21.15.114:59442) 2024-05-27T07:56:16.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.255+0000 7f00fdd9b700 1 -- 172.21.15.114:0/1702160905 learned_addr learned my addr 172.21.15.114:0/1702160905 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:16.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.256+0000 7f00fdd9b700 1 -- 172.21.15.114:0/1702160905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f010010a3d0 msgr2=0x7f01001ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:16.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.256+0000 7f00fdd9b700 1 --2- 172.21.15.114:0/1702160905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f010010a3d0 0x7f01001ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.256+0000 7f00fdd9b700 1 -- 172.21.15.114:0/1702160905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01001027d0 msgr2=0x7f010010e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:16.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.256+0000 7f00fdd9b700 1 --2- 172.21.15.114:0/1702160905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01001027d0 0x7f010010e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.256+0000 7f00fdd9b700 1 -- 172.21.15.114:0/1702160905 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010010f460 con 0x7f0100103120 2024-05-27T07:56:16.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.256+0000 7f00fdd9b700 1 --2- 172.21.15.114:0/1702160905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0100103120 0x7f010010ebd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f00f000e9f0 tx=0x7f00f000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:16.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.257+0000 7f00ef7fe700 1 -- 172.21.15.114:0/1702160905 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f000c9c0 con 0x7f0100103120 2024-05-27T07:56:16.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.257+0000 7f00ef7fe700 1 -- 172.21.15.114:0/1702160905 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00f000cb60 con 0x7f0100103120 2024-05-27T07:56:16.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.257+0000 7f00ef7fe700 1 -- 172.21.15.114:0/1702160905 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f0018680 con 0x7f0100103120 2024-05-27T07:56:16.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.258+0000 7f0104b34700 1 -- 172.21.15.114:0/1702160905 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01001ad840 con 0x7f0100103120 2024-05-27T07:56:16.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.258+0000 7f0104b34700 1 -- 172.21.15.114:0/1702160905 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01001add30 con 0x7f0100103120 2024-05-27T07:56:16.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.263+0000 7f0113549b80 1 Processor -- start 2024-05-27T07:56:16.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.265+0000 7f00ef7fe700 1 -- 172.21.15.114:0/1702160905 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f00f0010430 con 0x7f0100103120 2024-05-27T07:56:16.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.266+0000 7f00ef7fe700 1 --2- 172.21.15.114:0/1702160905 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f00e80b2bc0 0x7f00e80b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.266+0000 7f00ef7fe700 1 -- 172.21.15.114:0/1702160905 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 9554+0+0 (secure 0 0 0) 0x7f00f00e2bf0 con 0x7f0100103120 2024-05-27T07:56:16.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.267+0000 7f0113549b80 1 -- start start 2024-05-27T07:56:16.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.267+0000 7f0113549b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a689e40 0x556c7a80d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.267+0000 7f0113549b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a80dc50 0x556c7a807a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.267+0000 7f00fed9d700 1 --2- 172.21.15.114:0/1702160905 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f00e80b2bc0 0x7f00e80b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.267+0000 7f0113549b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a807f80 0x556c7a808390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.267+0000 7f0113549b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556c7a736c90 con 0x556c7a80dc50 2024-05-27T07:56:16.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.267+0000 7f0113549b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556c7a726830 con 0x556c7a807f80 2024-05-27T07:56:16.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.267+0000 7f0113549b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556c7a72a870 con 0x556c7a689e40 2024-05-27T07:56:16.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.267+0000 7f00fe59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a689e40 0x556c7a80d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.268+0000 7f00fe59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a689e40 0x556c7a80d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40834/0 (socket says 172.21.15.114:40834) 2024-05-27T07:56:16.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.268+0000 7f00fe59c700 1 -- 172.21.15.114:0/2109479732 learned_addr learned my addr 172.21.15.114:0/2109479732 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:16.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.268+0000 7f00fed9d700 1 --2- 172.21.15.114:0/2109479732 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a807f80 0x556c7a808390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.268+0000 7f00fdd9b700 1 --2- 172.21.15.114:0/2109479732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a80dc50 0x556c7a807a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.268+0000 7f00fdd9b700 1 -- 172.21.15.114:0/2109479732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a689e40 msgr2=0x556c7a80d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.268+0000 7f00fdd9b700 1 --2- 172.21.15.114:0/2109479732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a689e40 0x556c7a80d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.268+0000 7f00fdd9b700 1 -- 172.21.15.114:0/2109479732 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a807f80 msgr2=0x556c7a808390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.268+0000 7f00fdd9b700 1 --2- 172.21.15.114:0/2109479732 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a807f80 0x556c7a808390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.268+0000 7f00fdd9b700 1 -- 172.21.15.114:0/2109479732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c7a6768d0 con 0x556c7a80dc50 2024-05-27T07:56:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.269+0000 7f00fdd9b700 1 --2- 172.21.15.114:0/2109479732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a80dc50 0x556c7a807a40 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f00f00dfed0 tx=0x7f00f00a94e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc53f03de450e76 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.269+0000 7f00ecff9700 1 -- 172.21.15.114:0/2109479732 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f00e0360 con 0x556c7a80dc50 2024-05-27T07:56:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.269+0000 7f00ecff9700 1 -- 172.21.15.114:0/2109479732 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00f00a5040 con 0x556c7a80dc50 2024-05-27T07:56:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.269+0000 7f00ecff9700 1 -- 172.21.15.114:0/2109479732 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f00b2430 con 0x556c7a80dc50 2024-05-27T07:56:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.270+0000 7f0113549b80 1 -- 172.21.15.114:0/2109479732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a80dc50 msgr2=0x556c7a807a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:16.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.270+0000 7f0113549b80 1 --2- 172.21.15.114:0/2109479732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a80dc50 0x556c7a807a40 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f00f00dfed0 tx=0x7f00f00a94e0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.271+0000 7f00fed9d700 1 --2- 172.21.15.114:0/1702160905 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f00e80b2bc0 0x7f00e80b50d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f00f8009710 tx=0x7f00f80092c0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:16.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.273+0000 7f0113549b80 1 -- 172.21.15.114:0/2109479732 shutdown_connections 2024-05-27T07:56:16.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.273+0000 7f0113549b80 1 --2- 172.21.15.114:0/2109479732 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a807f80 0x556c7a808390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.273+0000 7f0113549b80 1 --2- 172.21.15.114:0/2109479732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a689e40 0x556c7a80d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.273+0000 7f0113549b80 1 --2- 172.21.15.114:0/2109479732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a80dc50 0x556c7a807a40 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.273+0000 7f0113549b80 1 -- 172.21.15.114:0/2109479732 >> 172.21.15.114:0/2109479732 conn(0x556c7a6e4af0 msgr2=0x556c7a802640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:16.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.274+0000 7f0113549b80 1 -- 172.21.15.114:0/2109479732 shutdown_connections 2024-05-27T07:56:16.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.275+0000 7f0113549b80 1 -- 172.21.15.114:0/2109479732 wait complete. 2024-05-27T07:56:16.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.276+0000 7f0113549b80 1 -- start start 2024-05-27T07:56:16.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.276+0000 7f0113549b80 1 Processor -- start 2024-05-27T07:56:16.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.277+0000 7f0113549b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a689e40 0x556c7a811350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.277+0000 7f0113549b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a807f80 0x556c7a811890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.277+0000 7f0113549b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a80dc50 0x556c7a811dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.277+0000 7f0113549b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556c7a62cfa0 con 0x556c7a689e40 2024-05-27T07:56:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.277+0000 7f0113549b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556c7a6c87c0 con 0x556c7a80dc50 2024-05-27T07:56:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.277+0000 7f0113549b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556c7a814740 con 0x556c7a807f80 2024-05-27T07:56:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.277+0000 7f0113549b80 10 client.0 _ll_register_callbacks cb 0x556c7a6c1920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.278+0000 7f00fed9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a80dc50 0x556c7a811dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.278+0000 7f00fed9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a80dc50 0x556c7a811dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:59456/0 (socket says 172.21.15.114:59456) 2024-05-27T07:56:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.278+0000 7f00fed9d700 1 -- 172.21.15.114:0/4259056729 learned_addr learned my addr 172.21.15.114:0/4259056729 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.278+0000 7f00fe59c700 1 --2- 172.21.15.114:0/4259056729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a689e40 0x556c7a811350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.278+0000 7f00fe59c700 1 -- 172.21.15.114:0/4259056729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a807f80 msgr2=0x556c7a811890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:56:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.278+0000 7f00fe59c700 1 --2- 172.21.15.114:0/4259056729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a807f80 0x556c7a811890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.278+0000 7f00fe59c700 1 -- 172.21.15.114:0/4259056729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a80dc50 msgr2=0x556c7a811dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:16.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.278+0000 7f00fe59c700 1 --2- 172.21.15.114:0/4259056729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a80dc50 0x556c7a811dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.278+0000 7f00fe59c700 1 -- 172.21.15.114:0/4259056729 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c7a6c5690 con 0x556c7a689e40 2024-05-27T07:56:16.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.278+0000 7f00fe59c700 1 --2- 172.21.15.114:0/4259056729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a689e40 0x556c7a811350 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f00f40098c0 tx=0x7f00f400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd40bcead58d2a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:16.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.279+0000 7f00e5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:56:16.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.279+0000 7f00e5ffb700 1 -- 172.21.15.114:0/4259056729 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f400b760 con 0x556c7a689e40 2024-05-27T07:56:16.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.279+0000 7f00e5ffb700 1 -- 172.21.15.114:0/4259056729 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(148..148 src has 1..148) ==== 9554+0+0 (secure 0 0 0) 0x7f00f400f7f0 con 0x556c7a689e40 2024-05-27T07:56:16.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.279+0000 7f0113549b80 1 -- 172.21.15.114:0/4259056729 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00f00a5040 con 0x556c7a689e40 2024-05-27T07:56:16.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.279+0000 7f00e5ffb700 1 -- 172.21.15.114:0/4259056729 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f00f40050b0 con 0x556c7a689e40 2024-05-27T07:56:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.280+0000 7f0113549b80 10 client.19266 fetch_fsmap learned FSMap version 93 2024-05-27T07:56:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.280+0000 7f0113549b80 1 -- 172.21.15.114:0/4259056729 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=4+}) -- 0x556c7a81f000 con 0x556c7a689e40 2024-05-27T07:56:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.280+0000 7f00e5ffb700 1 -- 172.21.15.114:0/4259056729 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00f4003680 con 0x556c7a689e40 2024-05-27T07:56:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.280+0000 7f00e5ffb700 1 -- 172.21.15.114:0/4259056729 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 93) ==== 4700+0+0 (secure 0 0 0) 0x7f00f402a030 con 0x556c7a689e40 2024-05-27T07:56:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.280+0000 7f0113549b80 10 client.19266 fetch_fsmap finished waiting for FSMap version 93 2024-05-27T07:56:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.280+0000 7f0113549b80 10 client.19266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.280+0000 7f0113549b80 20 client.19266 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.280+0000 7f0113549b80 1 --2- 172.21.15.114:0/4259056729 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556c7a8219a0 0x556c7a823e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.280+0000 7f0113549b80 4 client.19266 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:56:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.280+0000 7f0113549b80 1 -- 172.21.15.114:0/4259056729 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f00f00a5040 con 0x556c7a8219a0 2024-05-27T07:56:16.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.284+0000 7f00fdd9b700 1 --2- 172.21.15.114:0/4259056729 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556c7a8219a0 0x556c7a823e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.289+0000 7f00fdd9b700 1 --2- 172.21.15.114:0/4259056729 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556c7a8219a0 0x556c7a823e50 crc :-1 s=READY pgs=450 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2bb924ca6183423 server_cookie=e861650502dc82f2 in_seq=0 out_seq=0 2024-05-27T07:56:16.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.289+0000 7f00e5ffb700 10 client.19266 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:16.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.294+0000 7f00e5ffb700 1 -- 172.21.15.114:0/4259056729 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f00f00a5040 con 0x556c7a8219a0 2024-05-27T07:56:16.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.294+0000 7f00e5ffb700 10 client.19266 handle_command_reply: tid=0 2024-05-27T07:56:16.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.294+0000 7f0113549b80 1 client.19266 shutdown 2024-05-27T07:56:16.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.294+0000 7f0113549b80 1 -- 172.21.15.114:0/4259056729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a689e40 msgr2=0x556c7a811350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:16.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.294+0000 7f0113549b80 1 --2- 172.21.15.114:0/4259056729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a689e40 0x556c7a811350 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f00f40098c0 tx=0x7f00f400be20 comp rx=0 tx=0).stop 2024-05-27T07:56:16.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.295+0000 7f0113549b80 1 -- 172.21.15.114:0/4259056729 shutdown_connections 2024-05-27T07:56:16.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.295+0000 7f0113549b80 1 --2- 172.21.15.114:0/4259056729 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556c7a8219a0 0x556c7a823e50 crc :-1 s=READY pgs=450 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.295+0000 7f0113549b80 1 --2- 172.21.15.114:0/4259056729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a80dc50 0x556c7a811dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.295+0000 7f00e5ffb700 0 client.19266 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:16.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.295+0000 7f0113549b80 1 --2- 172.21.15.114:0/4259056729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a689e40 0x556c7a811350 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.295+0000 7f0113549b80 1 --2- 172.21.15.114:0/4259056729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a807f80 0x556c7a811890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.295+0000 7f0113549b80 1 -- 172.21.15.114:0/4259056729 >> 172.21.15.114:0/4259056729 conn(0x556c7a6e4af0 msgr2=0x556c7a806a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:16.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.296+0000 7f0113549b80 1 -- 172.21.15.114:0/4259056729 shutdown_connections 2024-05-27T07:56:16.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.296+0000 7f0113549b80 1 -- 172.21.15.114:0/4259056729 wait complete. 2024-05-27T07:56:16.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.299+0000 7f0113549b80 20 client.19266 trim_cache size 0 max 16384 2024-05-27T07:56:16.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.329+0000 7f0113549b80 1 Processor -- start 2024-05-27T07:56:16.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.329+0000 7f0113549b80 1 -- start start 2024-05-27T07:56:16.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.330+0000 7f0113549b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a689e40 0x556c7a807a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.330+0000 7f0113549b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a807f80 0x556c7a803c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.330+0000 7f0113549b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a80dc50 0x556c7a811c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.330+0000 7f0113549b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f01001b7240 con 0x556c7a807f80 2024-05-27T07:56:16.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.330+0000 7f0113549b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f01001b7910 con 0x556c7a80dc50 2024-05-27T07:56:16.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.330+0000 7f0113549b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f01001b6140 con 0x556c7a689e40 2024-05-27T07:56:16.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.330+0000 7f00fed9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a80dc50 0x556c7a811c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.330+0000 7f00fed9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a80dc50 0x556c7a811c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:59472/0 (socket says 172.21.15.114:59472) 2024-05-27T07:56:16.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.330+0000 7f00fed9d700 1 -- 172.21.15.114:0/1350529895 learned_addr learned my addr 172.21.15.114:0/1350529895 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:16.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.330+0000 7f00fe59c700 1 --2- 172.21.15.114:0/1350529895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a689e40 0x556c7a807a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.331+0000 7f00fe59c700 1 -- 172.21.15.114:0/1350529895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a80dc50 msgr2=0x556c7a811c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:16.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.331+0000 7f00fe59c700 1 --2- 172.21.15.114:0/1350529895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a80dc50 0x556c7a811c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.331+0000 7f00fe59c700 1 -- 172.21.15.114:0/1350529895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a807f80 msgr2=0x556c7a803c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:16.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.331+0000 7f00fe59c700 1 --2- 172.21.15.114:0/1350529895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a807f80 0x556c7a803c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.331+0000 7f00fe59c700 1 -- 172.21.15.114:0/1350529895 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c7a80f8c0 con 0x556c7a689e40 2024-05-27T07:56:16.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.332+0000 7f00fe59c700 1 --2- 172.21.15.114:0/1350529895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a689e40 0x556c7a807a40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f00f4009810 tx=0x7f00f4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=824913a80e30f7bf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:16.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.334+0000 7f00e7fff700 1 -- 172.21.15.114:0/1350529895 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f400b760 con 0x556c7a689e40 2024-05-27T07:56:16.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.334+0000 7f00e7fff700 1 -- 172.21.15.114:0/1350529895 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00f4022a20 con 0x556c7a689e40 2024-05-27T07:56:16.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.334+0000 7f00e7fff700 1 -- 172.21.15.114:0/1350529895 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f4003990 con 0x556c7a689e40 2024-05-27T07:56:16.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.336+0000 7f0113549b80 1 -- 172.21.15.114:0/1350529895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a689e40 msgr2=0x556c7a807a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:16.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.336+0000 7f0113549b80 1 --2- 172.21.15.114:0/1350529895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a689e40 0x556c7a807a40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f00f4009810 tx=0x7f00f4009c00 comp rx=0 tx=0).stop 2024-05-27T07:56:16.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.339+0000 7f0113549b80 1 -- 172.21.15.114:0/1350529895 shutdown_connections 2024-05-27T07:56:16.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.339+0000 7f0113549b80 1 --2- 172.21.15.114:0/1350529895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a80dc50 0x556c7a811c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.339+0000 7f0113549b80 1 --2- 172.21.15.114:0/1350529895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a689e40 0x556c7a807a40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.339+0000 7f0113549b80 1 --2- 172.21.15.114:0/1350529895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a807f80 0x556c7a803c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.339+0000 7f0113549b80 1 -- 172.21.15.114:0/1350529895 >> 172.21.15.114:0/1350529895 conn(0x556c7a6e4af0 msgr2=0x556c7a80c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:16.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.341+0000 7f0113549b80 1 -- 172.21.15.114:0/1350529895 shutdown_connections 2024-05-27T07:56:16.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.341+0000 7f0113549b80 1 -- 172.21.15.114:0/1350529895 wait complete. 2024-05-27T07:56:16.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.342+0000 7f0113549b80 1 -- start start 2024-05-27T07:56:16.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.342+0000 7f0113549b80 1 Processor -- start 2024-05-27T07:56:16.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.343+0000 7f0113549b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a689e40 0x556c7a819bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.343+0000 7f0113549b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a807f80 0x556c7a81a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.343+0000 7f0113549b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a80dc50 0x556c7a828a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.343+0000 7f0113549b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f01001b53a0 con 0x556c7a807f80 2024-05-27T07:56:16.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.343+0000 7f0113549b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f01001b4d30 con 0x556c7a689e40 2024-05-27T07:56:16.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.343+0000 7f0113549b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f01001b5a70 con 0x556c7a80dc50 2024-05-27T07:56:16.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.343+0000 7f0113549b80 10 client.0 _ll_register_callbacks cb 0x556c7a6c1920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:16.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.345+0000 7f00fe59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a689e40 0x556c7a819bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.345+0000 7f00fe59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a689e40 0x556c7a819bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:59474/0 (socket says 172.21.15.114:59474) 2024-05-27T07:56:16.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.345+0000 7f00fe59c700 1 -- 172.21.15.114:0/1337088194 learned_addr learned my addr 172.21.15.114:0/1337088194 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:16.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.345+0000 7f00fed9d700 1 --2- 172.21.15.114:0/1337088194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a80dc50 0x556c7a828a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.345+0000 7f00fdd9b700 1 --2- 172.21.15.114:0/1337088194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a807f80 0x556c7a81a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.345+0000 7f00fe59c700 1 -- 172.21.15.114:0/1337088194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a80dc50 msgr2=0x556c7a828a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:16.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.345+0000 7f00fe59c700 1 --2- 172.21.15.114:0/1337088194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a80dc50 0x556c7a828a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.345+0000 7f00fe59c700 1 -- 172.21.15.114:0/1337088194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a807f80 msgr2=0x556c7a81a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:16.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.345+0000 7f00fe59c700 1 --2- 172.21.15.114:0/1337088194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a807f80 0x556c7a81a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.345+0000 7f00fe59c700 1 -- 172.21.15.114:0/1337088194 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c7a8291e0 con 0x556c7a689e40 2024-05-27T07:56:16.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.345+0000 7f00fe59c700 1 --2- 172.21.15.114:0/1337088194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a689e40 0x556c7a819bd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f00f4022e00 tx=0x7f00f4005a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a40e45c4b9d66dc5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:16.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.346+0000 7f00e5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:56:16.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.346+0000 7f00e5ffb700 1 -- 172.21.15.114:0/1337088194 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f4005520 con 0x556c7a689e40 2024-05-27T07:56:16.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.346+0000 7f0113549b80 1 -- 172.21.15.114:0/1337088194 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00f4022a20 con 0x556c7a689e40 2024-05-27T07:56:16.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.346+0000 7f00e5ffb700 1 -- 172.21.15.114:0/1337088194 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(148..148 src has 1..148) ==== 9554+0+0 (secure 0 0 0) 0x7f00f4003680 con 0x556c7a689e40 2024-05-27T07:56:16.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.346+0000 7f00e5ffb700 1 -- 172.21.15.114:0/1337088194 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f00f4047050 con 0x556c7a689e40 2024-05-27T07:56:16.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.348+0000 7f0113549b80 10 client.28613 fetch_fsmap learned FSMap version 93 2024-05-27T07:56:16.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.348+0000 7f0113549b80 1 -- 172.21.15.114:0/1337088194 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=4+}) -- 0x556c7a89cf30 con 0x556c7a689e40 2024-05-27T07:56:16.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.351+0000 7f00e5ffb700 1 -- 172.21.15.114:0/1337088194 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00f4022a20 con 0x556c7a689e40 2024-05-27T07:56:16.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.351+0000 7f00e5ffb700 1 -- 172.21.15.114:0/1337088194 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 93) ==== 4700+0+0 (secure 0 0 0) 0x7f00f4033070 con 0x556c7a689e40 2024-05-27T07:56:16.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.351+0000 7f0113549b80 10 client.28613 fetch_fsmap finished waiting for FSMap version 93 2024-05-27T07:56:16.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.351+0000 7f0113549b80 10 client.28613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:16.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.351+0000 7f0113549b80 20 client.28613 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:16.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.351+0000 7f0113549b80 1 --2- 172.21.15.114:0/1337088194 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556c7a8219a0 0x556c7a89f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:16.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.351+0000 7f0113549b80 4 client.28613 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:56:16.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.351+0000 7f0113549b80 1 -- 172.21.15.114:0/1337088194 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c7a7d39f0 con 0x556c7a8219a0 2024-05-27T07:56:16.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.353+0000 7f00fdd9b700 1 --2- 172.21.15.114:0/1337088194 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556c7a8219a0 0x556c7a89f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:16.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.357+0000 7f00fdd9b700 1 --2- 172.21.15.114:0/1337088194 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556c7a8219a0 0x556c7a89f490 crc :-1 s=READY pgs=451 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5e5c37967751984 server_cookie=4489b4b8c749f3bb in_seq=0 out_seq=0 2024-05-27T07:56:16.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.357+0000 7f00e5ffb700 10 client.28613 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:16.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.370+0000 7f00e5ffb700 1 -- 172.21.15.114:0/1337088194 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c7a7d39f0 con 0x556c7a8219a0 2024-05-27T07:56:16.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.370+0000 7f00e5ffb700 10 client.28613 handle_command_reply: tid=0 2024-05-27T07:56:16.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.370+0000 7f0113549b80 1 client.28613 shutdown 2024-05-27T07:56:16.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.372+0000 7f0113549b80 1 -- 172.21.15.114:0/1337088194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a689e40 msgr2=0x556c7a819bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:16.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.372+0000 7f0113549b80 1 --2- 172.21.15.114:0/1337088194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a689e40 0x556c7a819bd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f00f4022e00 tx=0x7f00f4005a40 comp rx=0 tx=0).stop 2024-05-27T07:56:16.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.372+0000 7f0113549b80 1 -- 172.21.15.114:0/1337088194 shutdown_connections 2024-05-27T07:56:16.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.372+0000 7f0113549b80 1 --2- 172.21.15.114:0/1337088194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c7a80dc50 0x556c7a828a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.372+0000 7f0113549b80 1 --2- 172.21.15.114:0/1337088194 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556c7a8219a0 0x556c7a89f490 crc :-1 s=READY pgs=451 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.372+0000 7f0113549b80 1 --2- 172.21.15.114:0/1337088194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c7a689e40 0x556c7a819bd0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.372+0000 7f0113549b80 1 --2- 172.21.15.114:0/1337088194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556c7a807f80 0x556c7a81a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.372+0000 7f0113549b80 1 -- 172.21.15.114:0/1337088194 >> 172.21.15.114:0/1337088194 conn(0x556c7a6e4af0 msgr2=0x556c7a80b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:16.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.372+0000 7f00e5ffb700 0 client.28613 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:16.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.373+0000 7f0113549b80 1 -- 172.21.15.114:0/1337088194 shutdown_connections 2024-05-27T07:56:16.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.374+0000 7f0113549b80 1 -- 172.21.15.114:0/1337088194 wait complete. 2024-05-27T07:56:16.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.374+0000 7f0113549b80 20 client.28613 trim_cache size 0 max 16384 2024-05-27T07:56:16.377 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:56:16.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.376+0000 7f0104b34700 1 -- 172.21.15.114:0/1702160905 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f00e80b2bc0 msgr2=0x7f00e80b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:16.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.376+0000 7f0104b34700 1 --2- 172.21.15.114:0/1702160905 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f00e80b2bc0 0x7f00e80b50d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f00f8009710 tx=0x7f00f80092c0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.376+0000 7f0104b34700 1 -- 172.21.15.114:0/1702160905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0100103120 msgr2=0x7f010010ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:16.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.376+0000 7f0104b34700 1 --2- 172.21.15.114:0/1702160905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0100103120 0x7f010010ebd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f00f000e9f0 tx=0x7f00f000ed00 comp rx=0 tx=0).stop 2024-05-27T07:56:16.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.376+0000 7f0104b34700 1 -- 172.21.15.114:0/1702160905 shutdown_connections 2024-05-27T07:56:16.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.376+0000 7f0104b34700 1 --2- 172.21.15.114:0/1702160905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f010010a3d0 0x7f01001ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.376+0000 7f0104b34700 1 --2- 172.21.15.114:0/1702160905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01001027d0 0x7f010010e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.376+0000 7f0104b34700 1 --2- 172.21.15.114:0/1702160905 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f00e80b2bc0 0x7f00e80b50d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.376+0000 7f0104b34700 1 --2- 172.21.15.114:0/1702160905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0100103120 0x7f010010ebd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:16.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.376+0000 7f0104b34700 1 -- 172.21.15.114:0/1702160905 >> 172.21.15.114:0/1702160905 conn(0x7f01000fe6e0 msgr2=0x7f0100100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:16.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.376+0000 7f0104b34700 1 -- 172.21.15.114:0/1702160905 shutdown_connections 2024-05-27T07:56:16.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:16.376+0000 7f0104b34700 1 -- 172.21.15.114:0/1702160905 wait complete. 2024-05-27T07:56:16.600 INFO:tasks.workunit.client.0.smithi114.stdout: 10 35617 10.26 MB/sec execute 28 sec latency 1714.986 ms 2024-05-27T07:56:17.600 INFO:tasks.workunit.client.0.smithi114.stdout: 10 35786 10.20 MB/sec execute 29 sec latency 373.230 ms 2024-05-27T07:56:17.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:17 smithi163 ceph-mon[36451]: pgmap v565: 129 pgs: 13 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 77 active+clean; 314 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.1 MiB/s wr, 378 op/s 2024-05-27T07:56:17.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:17 smithi186 ceph-mon[36647]: pgmap v565: 129 pgs: 13 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 77 active+clean; 314 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.1 MiB/s wr, 378 op/s 2024-05-27T07:56:18.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:17 smithi114 ceph-mon[32038]: pgmap v565: 129 pgs: 13 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 77 active+clean; 314 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.1 MiB/s wr, 378 op/s 2024-05-27T07:56:18.192 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:56:18.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:56:18.469 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:56:18.600 INFO:tasks.workunit.client.0.smithi114.stdout: 10 35953 10.25 MB/sec execute 30 sec latency 309.135 ms 2024-05-27T07:56:19.600 INFO:tasks.workunit.client.0.smithi114.stdout: 10 36122 10.13 MB/sec execute 31 sec latency 308.187 ms 2024-05-27T07:56:19.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:19 smithi163 ceph-mon[36451]: pgmap v566: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 313 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 311 op/s 2024-05-27T07:56:19.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:19 smithi186 ceph-mon[36647]: pgmap v566: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 313 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 311 op/s 2024-05-27T07:56:20.012 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:19 smithi114 ceph-mon[32038]: pgmap v566: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 313 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 311 op/s 2024-05-27T07:56:20.527 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.525+0000 7fe887c27640 1 -- 172.21.15.114:0/2708783229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe88010a1f0 msgr2=0x7fe88010a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.527 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.525+0000 7fe887c27640 1 --2- 172.21.15.114:0/2708783229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe88010a1f0 0x7fe88010a600 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe874009900 tx=0x7fe87402a4d0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.527+0000 7fe887c27640 1 -- 172.21.15.114:0/2708783229 shutdown_connections 2024-05-27T07:56:20.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.527+0000 7fe887c27640 1 --2- 172.21.15.114:0/2708783229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe88011d240 0x7fe88011f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.527+0000 7fe887c27640 1 --2- 172.21.15.114:0/2708783229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe88011a8e0 0x7fe88011cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.527+0000 7fe887c27640 1 --2- 172.21.15.114:0/2708783229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe88010a1f0 0x7fe88010a600 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.527+0000 7fe887c27640 1 -- 172.21.15.114:0/2708783229 >> 172.21.15.114:0/2708783229 conn(0x7fe88006e8f0 msgr2=0x7fe88006ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.534+0000 7fe887c27640 1 -- 172.21.15.114:0/2708783229 shutdown_connections 2024-05-27T07:56:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.537+0000 7fe887c27640 1 -- 172.21.15.114:0/2708783229 wait complete. 2024-05-27T07:56:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.538+0000 7fe887c27640 1 Processor -- start 2024-05-27T07:56:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.542+0000 7fe887c27640 1 -- start start 2024-05-27T07:56:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.542+0000 7fe887c27640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe88010a1f0 0x7fe88011a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.542+0000 7fe887c27640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe88011a8e0 0x7fe880113420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.542+0000 7fe887c27640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe88011d240 0x7fe880113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.542+0000 7fe887c27640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe880122cb0 con 0x7fe88011d240 2024-05-27T07:56:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.542+0000 7fe887c27640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe880122b30 con 0x7fe88011a8e0 2024-05-27T07:56:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.542+0000 7fe887c27640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe880122e30 con 0x7fe88010a1f0 2024-05-27T07:56:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.543+0000 7fe8851c2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe88011a8e0 0x7fe880113420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.543+0000 7fe8851c2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe88011a8e0 0x7fe880113420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:59496/0 (socket says 172.21.15.114:59496) 2024-05-27T07:56:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.543+0000 7fe8851c2640 1 -- 172.21.15.114:0/3588953672 learned_addr learned my addr 172.21.15.114:0/3588953672 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.544+0000 7fe8859c3640 1 --2- 172.21.15.114:0/3588953672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe88010a1f0 0x7fe88011a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.544+0000 7fe8851c2640 1 -- 172.21.15.114:0/3588953672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe88010a1f0 msgr2=0x7fe88011a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.544+0000 7fe8851c2640 1 --2- 172.21.15.114:0/3588953672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe88010a1f0 0x7fe88011a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.544+0000 7fe8851c2640 1 -- 172.21.15.114:0/3588953672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe88011d240 msgr2=0x7fe880113aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:56:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.544+0000 7fe8851c2640 1 --2- 172.21.15.114:0/3588953672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe88011d240 0x7fe880113aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.545+0000 7fe8851c2640 1 -- 172.21.15.114:0/3588953672 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe880117e20 con 0x7fe88011a8e0 2024-05-27T07:56:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.548+0000 7fe8851c2640 1 --2- 172.21.15.114:0/3588953672 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe88011a8e0 0x7fe880113420 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00b450 tx=0x7fe87c00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:20.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.553+0000 7fc71e2db700 1 -- 172.21.15.114:0/3684413118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718106eb0 msgr2=0x7fc718113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.553+0000 7fc71e2db700 1 --2- 172.21.15.114:0/3684413118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718106eb0 0x7fc718113340 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fc710013930 tx=0x7fc710013c40 comp rx=0 tx=0).stop 2024-05-27T07:56:20.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.555+0000 7fc71e2db700 1 -- 172.21.15.114:0/3684413118 shutdown_connections 2024-05-27T07:56:20.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.555+0000 7fc71e2db700 1 --2- 172.21.15.114:0/3684413118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718106eb0 0x7fc718113340 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.555+0000 7fc71e2db700 1 --2- 172.21.15.114:0/3684413118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc71810c5a0 0x7fc71810c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.555+0000 7fc71e2db700 1 --2- 172.21.15.114:0/3684413118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc718106560 0x7fc718106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.555+0000 7fc71e2db700 1 -- 172.21.15.114:0/3684413118 >> 172.21.15.114:0/3684413118 conn(0x7fc718102040 msgr2=0x7fc718104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.554+0000 7fe872ffd640 1 -- 172.21.15.114:0/3588953672 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c015020 con 0x7fe88011a8e0 2024-05-27T07:56:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.554+0000 7fe872ffd640 1 -- 172.21.15.114:0/3588953672 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe87c011d40 con 0x7fe88011a8e0 2024-05-27T07:56:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.554+0000 7fe872ffd640 1 -- 172.21.15.114:0/3588953672 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c00f340 con 0x7fe88011a8e0 2024-05-27T07:56:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.554+0000 7fe887c27640 1 -- 172.21.15.114:0/3588953672 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe880118110 con 0x7fe88011a8e0 2024-05-27T07:56:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.554+0000 7fe887c27640 1 -- 172.21.15.114:0/3588953672 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe880118610 con 0x7fe88011a8e0 2024-05-27T07:56:20.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.559+0000 7fc71e2db700 1 -- 172.21.15.114:0/3684413118 shutdown_connections 2024-05-27T07:56:20.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.561+0000 7fc71e2db700 1 -- 172.21.15.114:0/3684413118 wait complete. 2024-05-27T07:56:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.561+0000 7fe88ebbc740 1 Processor -- start 2024-05-27T07:56:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.561+0000 7fe88ebbc740 1 -- start start 2024-05-27T07:56:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.562+0000 7fe88ebbc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 0x55adc1d6e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.562+0000 7fe88ebbc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc19a42f0 0x55adc1d6e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.562+0000 7fe88ebbc740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19ac4d0 0x55adc1d6ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.562+0000 7fe88ebbc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55adc1ce0c30 con 0x55adc19a42f0 2024-05-27T07:56:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.562+0000 7fe88ebbc740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55adc1cd07d0 con 0x55adc19ac4d0 2024-05-27T07:56:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.562+0000 7fe88ebbc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55adc1cd4810 con 0x55adc19cd1b0 2024-05-27T07:56:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.563+0000 7fe8861c4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 0x55adc1d6e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.563+0000 7fe8861c4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 0x55adc1d6e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40882/0 (socket says 172.21.15.114:40882) 2024-05-27T07:56:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.563+0000 7fe8861c4640 1 -- 172.21.15.114:0/645310100 learned_addr learned my addr 172.21.15.114:0/645310100 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.563+0000 7fe8859c3640 1 --2- 172.21.15.114:0/645310100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc19a42f0 0x55adc1d6e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.563+0000 7fe8861c4640 1 -- 172.21.15.114:0/645310100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19ac4d0 msgr2=0x55adc1d6ee50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:56:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.563+0000 7fe8861c4640 1 --2- 172.21.15.114:0/645310100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19ac4d0 0x55adc1d6ee50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.563+0000 7fe8861c4640 1 -- 172.21.15.114:0/645310100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc19a42f0 msgr2=0x55adc1d6e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.563+0000 7fe8861c4640 1 --2- 172.21.15.114:0/645310100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc19a42f0 0x55adc1d6e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.562+0000 7fc71e2db700 1 Processor -- start 2024-05-27T07:56:20.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.563+0000 7fc71e2db700 1 -- start start 2024-05-27T07:56:20.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.564+0000 7fc71e2db700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc718106560 0x7fc718198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.564+0000 7fc71e2db700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718106eb0 0x7fc7181993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.564+0000 7fc71e2db700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc71810c5a0 0x7fc71819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.564+0000 7fc71e2db700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc71807a450 con 0x7fc718106eb0 2024-05-27T07:56:20.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.564+0000 7fc71e2db700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc71807a2d0 con 0x7fc718106560 2024-05-27T07:56:20.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.564+0000 7fc71e2db700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc71807a5d0 con 0x7fc71810c5a0 2024-05-27T07:56:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.563+0000 7fe8861c4640 1 -- 172.21.15.114:0/645310100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adc19e5210 con 0x55adc19cd1b0 2024-05-27T07:56:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.563+0000 7fe8861c4640 1 --2- 172.21.15.114:0/645310100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 0x55adc1d6e3d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe878007bd0 tx=0x7fe87800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=564419f7c8fac8c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.563+0000 7fe86bfff640 1 -- 172.21.15.114:0/645310100 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878014070 con 0x55adc19cd1b0 2024-05-27T07:56:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.564+0000 7fe86bfff640 1 -- 172.21.15.114:0/645310100 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe87800fb00 con 0x55adc19cd1b0 2024-05-27T07:56:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.565+0000 7fe88ebbc740 1 -- 172.21.15.114:0/645310100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 msgr2=0x55adc1d6e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.565+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/645310100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 0x55adc1d6e3d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe878007bd0 tx=0x7fe87800c6a0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.565+0000 7fe88ebbc740 1 -- 172.21.15.114:0/645310100 shutdown_connections 2024-05-27T07:56:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.565+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/645310100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19ac4d0 0x55adc1d6ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.565+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/645310100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc19a42f0 0x55adc1d6e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.565+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/645310100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 0x55adc1d6e3d0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.565+0000 7fe88ebbc740 1 -- 172.21.15.114:0/645310100 >> 172.21.15.114:0/645310100 conn(0x55adc19c1300 msgr2=0x55adc1d6ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.567+0000 7fe872ffd640 1 -- 172.21.15.114:0/3588953672 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fe87c018020 con 0x7fe88011a8e0 2024-05-27T07:56:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.567+0000 7fe88ebbc740 1 -- 172.21.15.114:0/645310100 shutdown_connections 2024-05-27T07:56:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.566+0000 7fc717fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc718106560 0x7fc718198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.566+0000 7fc717fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc718106560 0x7fc718198e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:59498/0 (socket says 172.21.15.114:59498) 2024-05-27T07:56:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.566+0000 7fc717fff700 1 -- 172.21.15.114:0/3848112766 learned_addr learned my addr 172.21.15.114:0/3848112766 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.567+0000 7fc71c878700 1 --2- 172.21.15.114:0/3848112766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc71810c5a0 0x7fc71819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.568+0000 7fc71c878700 1 -- 172.21.15.114:0/3848112766 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc718106560 msgr2=0x7fc718198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.568+0000 7fc71c878700 1 --2- 172.21.15.114:0/3848112766 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc718106560 0x7fc718198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.568+0000 7fc71c878700 1 -- 172.21.15.114:0/3848112766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718106eb0 msgr2=0x7fc7181993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.568+0000 7fc71c878700 1 --2- 172.21.15.114:0/3848112766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718106eb0 0x7fc7181993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.568+0000 7fc71c878700 1 -- 172.21.15.114:0/3848112766 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc71819dbe0 con 0x7fc71810c5a0 2024-05-27T07:56:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.569+0000 7fe88ebbc740 1 -- 172.21.15.114:0/645310100 wait complete. 2024-05-27T07:56:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.570+0000 7fe88ebbc740 1 -- start start 2024-05-27T07:56:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.570+0000 7fe88ebbc740 1 Processor -- start 2024-05-27T07:56:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.571+0000 7fe88ebbc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc19cd1b0 0x55adc1d78990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.571+0000 7fe88ebbc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc198bd80 0x55adc1d78ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.571+0000 7fe88ebbc740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19611c0 0x55adc1d84990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.571+0000 7fe88ebbc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55adc19b6790 con 0x55adc19cd1b0 2024-05-27T07:56:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.571+0000 7fe88ebbc740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55adc1c8c890 con 0x55adc19611c0 2024-05-27T07:56:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.571+0000 7fe88ebbc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55adc1d7b5a0 con 0x55adc198bd80 2024-05-27T07:56:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.571+0000 7fe88ebbc740 10 client.0 _ll_register_callbacks cb 0x55adc19e4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.571+0000 7fe8851c2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19611c0 0x55adc1d84990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.571+0000 7fe8851c2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19611c0 0x55adc1d84990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:59506/0 (socket says 172.21.15.114:59506) 2024-05-27T07:56:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.571+0000 7fe8851c2640 1 -- 172.21.15.114:0/2632302167 learned_addr learned my addr 172.21.15.114:0/2632302167 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.572+0000 7fe8859c3640 1 --2- 172.21.15.114:0/2632302167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc198bd80 0x55adc1d78ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.572+0000 7fe872ffd640 1 --2- 172.21.15.114:0/3588953672 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe86c0b29b0 0x7fe86c0b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.573+0000 7fc71c878700 1 --2- 172.21.15.114:0/3848112766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc71810c5a0 0x7fc71819d570 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc710013930 tx=0x7fc7100077d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.574+0000 7fe8851c2640 1 -- 172.21.15.114:0/2632302167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc198bd80 msgr2=0x55adc1d78ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.574+0000 7fe8851c2640 1 --2- 172.21.15.114:0/2632302167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc198bd80 0x55adc1d78ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.574+0000 7fe8851c2640 1 -- 172.21.15.114:0/2632302167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc19cd1b0 msgr2=0x55adc1d78990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.574+0000 7fe8851c2640 1 --2- 172.21.15.114:0/2632302167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc19cd1b0 0x55adc1d78990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.574+0000 7fe8851c2640 1 -- 172.21.15.114:0/2632302167 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adc1ab1620 con 0x55adc19611c0 2024-05-27T07:56:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.574+0000 7fe872ffd640 1 -- 172.21.15.114:0/3588953672 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 9554+0+0 (secure 0 0 0) 0x7fe87c0e23c0 con 0x7fe88011a8e0 2024-05-27T07:56:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.576+0000 7fe8861c4640 1 --2- 172.21.15.114:0/3588953672 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe86c0b29b0 0x7fe86c0b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.579+0000 7fe8861c4640 1 --2- 172.21.15.114:0/3588953672 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe86c0b29b0 0x7fe86c0b4ec0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe87800d010 tx=0x7fe87801b040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.574+0000 7fc7157fa700 1 -- 172.21.15.114:0/3848112766 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71001b070 con 0x7fc71810c5a0 2024-05-27T07:56:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.574+0000 7fc71e2db700 1 -- 172.21.15.114:0/3848112766 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc71819de70 con 0x7fc71810c5a0 2024-05-27T07:56:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.574+0000 7fc71e2db700 1 -- 172.21.15.114:0/3848112766 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7181b6830 con 0x7fc71810c5a0 2024-05-27T07:56:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.574+0000 7fc7157fa700 1 -- 172.21.15.114:0/3848112766 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc710009480 con 0x7fc71810c5a0 2024-05-27T07:56:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.574+0000 7fc7157fa700 1 -- 172.21.15.114:0/3848112766 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710005040 con 0x7fc71810c5a0 2024-05-27T07:56:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.581+0000 7fe8851c2640 1 --2- 172.21.15.114:0/2632302167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19611c0 0x55adc1d84990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0df780 tx=0x7fe87c010890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=703281e7c7150065 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.582+0000 7fe8697fa640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:56:20.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.581+0000 7fc7157fa700 1 -- 172.21.15.114:0/3848112766 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fc71001e070 con 0x7fc71810c5a0 2024-05-27T07:56:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.586+0000 7fe8697fa640 1 -- 172.21.15.114:0/2632302167 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0a9170 con 0x55adc19611c0 2024-05-27T07:56:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.586+0000 7fe8697fa640 1 -- 172.21.15.114:0/2632302167 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(148..148 src has 1..148) ==== 9554+0+0 (secure 0 0 0) 0x7fe87c017d80 con 0x55adc19611c0 2024-05-27T07:56:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.586+0000 7fe88ebbc740 1 -- 172.21.15.114:0/2632302167 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe87800fb00 con 0x55adc19611c0 2024-05-27T07:56:20.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.585+0000 7fc72ccf0b80 1 Processor -- start 2024-05-27T07:56:20.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.585+0000 7fc72ccf0b80 1 -- start start 2024-05-27T07:56:20.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.586+0000 7fc72ccf0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56381533aec0 0x5638154bfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.586+0000 7fc72ccf0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638154c00f0 0x5638154b9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.586+0000 7fc72ccf0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154ba5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.586+0000 7fc72ccf0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5638153e7df0 con 0x56381533aec0 2024-05-27T07:56:20.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.586+0000 7fc72ccf0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5638153d7990 con 0x5638154c00f0 2024-05-27T07:56:20.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.586+0000 7fc72ccf0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5638153db9d0 con 0x5638154ba1a0 2024-05-27T07:56:20.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.586+0000 7fc7157fa700 1 --2- 172.21.15.114:0/3848112766 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7000b2c00 0x7fc7000b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.590+0000 7fc7177fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154ba5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.590+0000 7fc7177fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56381533aec0 0x5638154bfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.590+0000 7fc7177fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56381533aec0 0x5638154bfbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36930/0 (socket says 172.21.15.114:36930) 2024-05-27T07:56:20.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.590+0000 7fc7177fe700 1 -- 172.21.15.114:0/866701701 learned_addr learned my addr 172.21.15.114:0/866701701 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:20.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.591+0000 7fc717fff700 1 --2- 172.21.15.114:0/866701701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638154c00f0 0x5638154b9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.591+0000 7fc7157fa700 1 -- 172.21.15.114:0/3848112766 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 9554+0+0 (secure 0 0 0) 0x7fc7100e45d0 con 0x7fc71810c5a0 2024-05-27T07:56:20.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.592+0000 7fc71c878700 1 --2- 172.21.15.114:0/3848112766 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7000b2c00 0x7fc7000b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.597+0000 7fe8697fa640 1 -- 172.21.15.114:0/2632302167 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fe87c0a8b10 con 0x55adc19611c0 2024-05-27T07:56:20.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.594+0000 7fc7177fe700 1 -- 172.21.15.114:0/866701701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638154c00f0 msgr2=0x5638154b9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.594+0000 7fc7177fe700 1 --2- 172.21.15.114:0/866701701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638154c00f0 0x5638154b9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.594+0000 7fc7177fe700 1 -- 172.21.15.114:0/866701701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56381533aec0 msgr2=0x5638154bfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.594+0000 7fc7177fe700 1 --2- 172.21.15.114:0/866701701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56381533aec0 0x5638154bfbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.594+0000 7fc7177fe700 1 -- 172.21.15.114:0/866701701 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563815381030 con 0x5638154ba1a0 2024-05-27T07:56:20.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.594+0000 7fc71c878700 1 --2- 172.21.15.114:0/3848112766 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7000b2c00 0x7fc7000b5110 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc710009e60 tx=0x7fc710013c40 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.598+0000 7fe88ebbc740 10 client.28625 fetch_fsmap learned FSMap version 93 2024-05-27T07:56:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.598+0000 7fe88ebbc740 1 -- 172.21.15.114:0/2632302167 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=4+}) -- 0x55adc1d87830 con 0x55adc19611c0 2024-05-27T07:56:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.602+0000 7fe8697fa640 1 -- 172.21.15.114:0/2632302167 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe87800fb00 con 0x55adc19611c0 2024-05-27T07:56:20.603 INFO:tasks.workunit.client.0.smithi114.stdout: 10 36286 9.98 MB/sec execute 32 sec latency 136.988 ms 2024-05-27T07:56:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.602+0000 7fe8697fa640 1 -- 172.21.15.114:0/2632302167 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 93) ==== 4700+0+0 (secure 0 0 0) 0x7fe87c0a8600 con 0x55adc19611c0 2024-05-27T07:56:20.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.600+0000 7fc7177fe700 1 --2- 172.21.15.114:0/866701701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154ba5b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00af00 tx=0x7fc70c0090e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=728ca779165d11b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:20.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.604+0000 7fc6fe7fc700 1 -- 172.21.15.114:0/866701701 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70c004020 con 0x5638154ba1a0 2024-05-27T07:56:20.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.604+0000 7fc6fe7fc700 1 -- 172.21.15.114:0/866701701 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc70c004920 con 0x5638154ba1a0 2024-05-27T07:56:20.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.604+0000 7fc6fe7fc700 1 -- 172.21.15.114:0/866701701 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70c0065d0 con 0x5638154ba1a0 2024-05-27T07:56:20.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.607+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/866701701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 msgr2=0x5638154ba5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.607+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/866701701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154ba5b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00af00 tx=0x7fc70c0090e0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.606+0000 7fe88ebbc740 10 client.28625 fetch_fsmap finished waiting for FSMap version 93 2024-05-27T07:56:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.606+0000 7fe88ebbc740 10 client.28625 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.606+0000 7fe88ebbc740 20 client.28625 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.606+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/2632302167 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55adc1d89030 0x55adc1d8b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.606+0000 7fe88ebbc740 4 client.28625 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:56:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.606+0000 7fe88ebbc740 1 -- 172.21.15.114:0/2632302167 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adc1ce2b80 con 0x55adc1d89030 2024-05-27T07:56:20.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.608+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/866701701 shutdown_connections 2024-05-27T07:56:20.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.608+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/866701701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154ba5b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.608+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/866701701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56381533aec0 0x5638154bfbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.608+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/866701701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638154c00f0 0x5638154b9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.609+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/866701701 >> 172.21.15.114:0/866701701 conn(0x56381533d7f0 msgr2=0x5638154b4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:20.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.612+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/866701701 shutdown_connections 2024-05-27T07:56:20.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.615+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/866701701 wait complete. 2024-05-27T07:56:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.610+0000 7fe8859c3640 1 --2- 172.21.15.114:0/2632302167 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55adc1d89030 0x55adc1d8b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.615+0000 7fe8859c3640 1 --2- 172.21.15.114:0/2632302167 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55adc1d89030 0x55adc1d8b480 crc :-1 s=READY pgs=452 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c119a5a7f1548e1 server_cookie=84555f2d70641ef0 in_seq=0 out_seq=0 2024-05-27T07:56:20.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.616+0000 7fc72ccf0b80 1 -- start start 2024-05-27T07:56:20.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.616+0000 7fc72ccf0b80 1 Processor -- start 2024-05-27T07:56:20.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.619+0000 7fe8697fa640 10 client.28625 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:20.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.617+0000 7fc72ccf0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56381533aec0 0x5638154c3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.617+0000 7fc72ccf0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.617+0000 7fc72ccf0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638154c00f0 0x5638154c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.617+0000 7fc72ccf0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5638152de020 con 0x5638154c00f0 2024-05-27T07:56:20.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.617+0000 7fc72ccf0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56381533fe20 con 0x56381533aec0 2024-05-27T07:56:20.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.617+0000 7fc72ccf0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5638154c6b80 con 0x5638154ba1a0 2024-05-27T07:56:20.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.617+0000 7fc72ccf0b80 10 client.0 _ll_register_callbacks cb 0x56381536e630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.623+0000 7fe8697fa640 1 -- 172.21.15.114:0/2632302167 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55adc1ce2b80 con 0x55adc1d89030 2024-05-27T07:56:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.623+0000 7fe8697fa640 10 client.28625 handle_command_reply: tid=0 2024-05-27T07:56:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.623+0000 7fe88ebbc740 1 client.28625 shutdown 2024-05-27T07:56:20.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.619+0000 7fc717fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.619+0000 7fc7177fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56381533aec0 0x5638154c3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.619+0000 7fc7177fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56381533aec0 0x5638154c3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:59510/0 (socket says 172.21.15.114:59510) 2024-05-27T07:56:20.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.619+0000 7fc7177fe700 1 -- 172.21.15.114:0/1020130178 learned_addr learned my addr 172.21.15.114:0/1020130178 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:20.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.619+0000 7fc7177fe700 1 --2- 172.21.15.114:0/1020130178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638154c00f0 0x5638154c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.622+0000 7fc717fff700 1 -- 172.21.15.114:0/1020130178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56381533aec0 msgr2=0x5638154c3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.622+0000 7fc717fff700 1 --2- 172.21.15.114:0/1020130178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56381533aec0 0x5638154c3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.622+0000 7fc717fff700 1 -- 172.21.15.114:0/1020130178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638154c00f0 msgr2=0x5638154c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.622+0000 7fc717fff700 1 --2- 172.21.15.114:0/1020130178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638154c00f0 0x5638154c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.622+0000 7fc717fff700 1 -- 172.21.15.114:0/1020130178 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563815327950 con 0x5638154ba1a0 2024-05-27T07:56:20.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.623+0000 7fc717fff700 1 --2- 172.21.15.114:0/1020130178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154c3e90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc718104b70 tx=0x7fc708011840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca2817a973b5f359 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:20.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.623+0000 7fc6f77fe700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:56:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.626+0000 7fe88ebbc740 1 -- 172.21.15.114:0/2632302167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19611c0 msgr2=0x55adc1d84990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.626+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/2632302167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19611c0 0x55adc1d84990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0df780 tx=0x7fe87c010890 comp rx=0 tx=0).stop 2024-05-27T07:56:20.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.628+0000 7fc6f77fe700 1 -- 172.21.15.114:0/1020130178 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc708014650 con 0x5638154ba1a0 2024-05-27T07:56:20.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.628+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/1020130178 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc70c004920 con 0x5638154ba1a0 2024-05-27T07:56:20.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.628+0000 7fc6f77fe700 1 -- 172.21.15.114:0/1020130178 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(148..148 src has 1..148) ==== 9554+0+0 (secure 0 0 0) 0x7fc7080147f0 con 0x5638154ba1a0 2024-05-27T07:56:20.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.630+0000 7fe88ebbc740 1 -- 172.21.15.114:0/2632302167 shutdown_connections 2024-05-27T07:56:20.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.630+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/2632302167 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55adc1d89030 0x55adc1d8b480 crc :-1 s=READY pgs=452 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.630+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/2632302167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19611c0 0x55adc1d84990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.630+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/2632302167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc198bd80 0x55adc1d78ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.631+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/2632302167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc19cd1b0 0x55adc1d78990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.631+0000 7fe88ebbc740 1 -- 172.21.15.114:0/2632302167 >> 172.21.15.114:0/2632302167 conn(0x55adc19c1300 msgr2=0x55adc1d6e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:20.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.631+0000 7fe8697fa640 0 client.28625 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:20.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.632+0000 7fc6f77fe700 1 -- 172.21.15.114:0/1020130178 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fc7080149f0 con 0x5638154ba1a0 2024-05-27T07:56:20.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.634+0000 7fc72ccf0b80 10 client.28759 fetch_fsmap learned FSMap version 93 2024-05-27T07:56:20.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.634+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/1020130178 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=4+}) -- 0x5638153917e0 con 0x5638154ba1a0 2024-05-27T07:56:20.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.635+0000 7fe88ebbc740 1 -- 172.21.15.114:0/2632302167 shutdown_connections 2024-05-27T07:56:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.639+0000 7fe88ebbc740 1 -- 172.21.15.114:0/2632302167 wait complete. 2024-05-27T07:56:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.639+0000 7fe88ebbc740 20 client.28625 trim_cache size 0 max 16384 2024-05-27T07:56:20.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.639+0000 7fc6f77fe700 1 -- 172.21.15.114:0/1020130178 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc70c004920 con 0x5638154ba1a0 2024-05-27T07:56:20.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.639+0000 7fc6f77fe700 1 -- 172.21.15.114:0/1020130178 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 93) ==== 4700+0+0 (secure 0 0 0) 0x7fc708027c80 con 0x5638154ba1a0 2024-05-27T07:56:20.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.639+0000 7fc72ccf0b80 10 client.28759 fetch_fsmap finished waiting for FSMap version 93 2024-05-27T07:56:20.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.639+0000 7fc72ccf0b80 10 client.28759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:20.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.639+0000 7fc72ccf0b80 20 client.28759 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:20.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.639+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/1020130178 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5638154d3d60 0x5638154d61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.639+0000 7fc72ccf0b80 4 client.28759 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:56:20.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.639+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/1020130178 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638153e9d40 con 0x5638154d3d60 2024-05-27T07:56:20.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.643+0000 7fc7177fe700 1 --2- 172.21.15.114:0/1020130178 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5638154d3d60 0x5638154d61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.655+0000 7fc7177fe700 1 --2- 172.21.15.114:0/1020130178 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5638154d3d60 0x5638154d61b0 crc :-1 s=READY pgs=453 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2bf8c009006745e server_cookie=985788aad43f6a3d in_seq=0 out_seq=0 2024-05-27T07:56:20.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.655+0000 7fc6f77fe700 10 client.28759 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:20.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.657+0000 7fc6f77fe700 1 -- 172.21.15.114:0/1020130178 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5638153e9d40 con 0x5638154d3d60 2024-05-27T07:56:20.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.657+0000 7fc6f77fe700 10 client.28759 handle_command_reply: tid=0 2024-05-27T07:56:20.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.657+0000 7fc72ccf0b80 1 client.28759 shutdown 2024-05-27T07:56:20.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.660+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/1020130178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 msgr2=0x5638154c3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.660+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/1020130178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154c3e90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc718104b70 tx=0x7fc708011840 comp rx=0 tx=0).stop 2024-05-27T07:56:20.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.660+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/1020130178 shutdown_connections 2024-05-27T07:56:20.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.660+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/1020130178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638154c00f0 0x5638154c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.660+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/1020130178 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5638154d3d60 0x5638154d61b0 crc :-1 s=READY pgs=453 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.660+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/1020130178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56381533aec0 0x5638154c3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.660+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/1020130178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154c3e90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.660+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/1020130178 >> 172.21.15.114:0/1020130178 conn(0x56381533d7f0 msgr2=0x5638154b8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:20.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.661+0000 7fc6f77fe700 0 client.28759 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:20.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.661+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/1020130178 shutdown_connections 2024-05-27T07:56:20.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.661+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/1020130178 wait complete. 2024-05-27T07:56:20.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.661+0000 7fc72ccf0b80 20 client.28759 trim_cache size 0 max 16384 2024-05-27T07:56:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.673+0000 7fe88ebbc740 1 Processor -- start 2024-05-27T07:56:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.675+0000 7fe88ebbc740 1 -- start start 2024-05-27T07:56:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.676+0000 7fe88ebbc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19611c0 0x55adc1d73030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.676+0000 7fe88ebbc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc198bd80 0x55adc1d6e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.676+0000 7fe88ebbc740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19cd1b0 0x55adc1d84700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.676+0000 7fe88ebbc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe8801c8ea0 con 0x55adc198bd80 2024-05-27T07:56:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.676+0000 7fe88ebbc740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe8801c9570 con 0x55adc19cd1b0 2024-05-27T07:56:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.676+0000 7fe88ebbc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8801c7dc0 con 0x55adc19611c0 2024-05-27T07:56:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.676+0000 7fe8859c3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19611c0 0x55adc1d73030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.676+0000 7fe8861c4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19cd1b0 0x55adc1d84700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.676+0000 7fe8861c4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19cd1b0 0x55adc1d84700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:59516/0 (socket says 172.21.15.114:59516) 2024-05-27T07:56:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.676+0000 7fe8861c4640 1 -- 172.21.15.114:0/976387670 learned_addr learned my addr 172.21.15.114:0/976387670 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.676+0000 7fe8859c3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19611c0 0x55adc1d73030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40916/0 (socket says 172.21.15.114:40916) 2024-05-27T07:56:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.677+0000 7fe8859c3640 1 -- 172.21.15.114:0/976387670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19cd1b0 msgr2=0x55adc1d84700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.677+0000 7fe8859c3640 1 --2- 172.21.15.114:0/976387670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19cd1b0 0x55adc1d84700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.677+0000 7fe8859c3640 1 -- 172.21.15.114:0/976387670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc198bd80 msgr2=0x55adc1d6e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.677+0000 7fe8859c3640 1 --2- 172.21.15.114:0/976387670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc198bd80 0x55adc1d6e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.677+0000 7fe8859c3640 1 -- 172.21.15.114:0/976387670 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adc1d76cd0 con 0x55adc19611c0 2024-05-27T07:56:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.677+0000 7fe8859c3640 1 --2- 172.21.15.114:0/976387670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19611c0 0x55adc1d73030 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe880114c70 tx=0x7fe8740092d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3013ff6fa17e9958 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.677+0000 7fe86b7fe640 1 -- 172.21.15.114:0/976387670 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874035040 con 0x55adc19611c0 2024-05-27T07:56:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.678+0000 7fe86b7fe640 1 -- 172.21.15.114:0/976387670 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe874004040 con 0x55adc19611c0 2024-05-27T07:56:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.679+0000 7fe88ebbc740 1 -- 172.21.15.114:0/976387670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19611c0 msgr2=0x55adc1d73030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.679+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/976387670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19611c0 0x55adc1d73030 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe880114c70 tx=0x7fe8740092d0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.681+0000 7fe88ebbc740 1 -- 172.21.15.114:0/976387670 shutdown_connections 2024-05-27T07:56:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.681+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/976387670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19cd1b0 0x55adc1d84700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.681+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/976387670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc198bd80 0x55adc1d6e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.681+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/976387670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19611c0 0x55adc1d73030 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.681+0000 7fe88ebbc740 1 -- 172.21.15.114:0/976387670 >> 172.21.15.114:0/976387670 conn(0x55adc19c1300 msgr2=0x55adc1d69b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.684+0000 7fe88ebbc740 1 -- 172.21.15.114:0/976387670 shutdown_connections 2024-05-27T07:56:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.685+0000 7fe88ebbc740 1 -- 172.21.15.114:0/976387670 wait complete. 2024-05-27T07:56:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.685+0000 7fe88ebbc740 1 -- start start 2024-05-27T07:56:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.686+0000 7fe88ebbc740 1 Processor -- start 2024-05-27T07:56:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.686+0000 7fe88ebbc740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19611c0 0x55adc1d81130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.686+0000 7fe88ebbc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc198bd80 0x55adc1d7c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.686+0000 7fe88ebbc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 0x55adc1d7cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.686+0000 7fe88ebbc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe8801c7020 con 0x55adc198bd80 2024-05-27T07:56:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.686+0000 7fe88ebbc740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe8801c69b0 con 0x55adc19611c0 2024-05-27T07:56:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.686+0000 7fe88ebbc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8801c76f0 con 0x55adc19cd1b0 2024-05-27T07:56:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.686+0000 7fe88ebbc740 10 client.0 _ll_register_callbacks cb 0x7fe87c0e5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.687+0000 7fe8861c4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 0x55adc1d7cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.687+0000 7fe8861c4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 0x55adc1d7cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40930/0 (socket says 172.21.15.114:40930) 2024-05-27T07:56:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.687+0000 7fe8861c4640 1 -- 172.21.15.114:0/252580805 learned_addr learned my addr 172.21.15.114:0/252580805 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.687+0000 7fe8861c4640 1 -- 172.21.15.114:0/252580805 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19611c0 msgr2=0x55adc1d81130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.687+0000 7fe8861c4640 1 --2- 172.21.15.114:0/252580805 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19611c0 0x55adc1d81130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.687+0000 7fe8861c4640 1 -- 172.21.15.114:0/252580805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc198bd80 msgr2=0x55adc1d7c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.687+0000 7fe8861c4640 1 --2- 172.21.15.114:0/252580805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc198bd80 0x55adc1d7c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.687+0000 7fe8861c4640 1 -- 172.21.15.114:0/252580805 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adc1d7d220 con 0x55adc19cd1b0 2024-05-27T07:56:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.688+0000 7fe8861c4640 1 --2- 172.21.15.114:0/252580805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 0x55adc1d7cab0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe87800c6e0 tx=0x7fe8780074e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f37f58fe8d5dc0f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.690+0000 7fe8697fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:56:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.691+0000 7fe8697fa640 1 -- 172.21.15.114:0/252580805 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878034060 con 0x55adc19cd1b0 2024-05-27T07:56:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.691+0000 7fe8697fa640 1 -- 172.21.15.114:0/252580805 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(148..148 src has 1..148) ==== 9554+0+0 (secure 0 0 0) 0x7fe878005260 con 0x55adc19cd1b0 2024-05-27T07:56:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.692+0000 7fe88ebbc740 1 -- 172.21.15.114:0/252580805 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe874004040 con 0x55adc19cd1b0 2024-05-27T07:56:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.700+0000 7fe8697fa640 1 -- 172.21.15.114:0/252580805 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fe878014040 con 0x55adc19cd1b0 2024-05-27T07:56:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.701+0000 7fe88ebbc740 10 client.28765 fetch_fsmap learned FSMap version 93 2024-05-27T07:56:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.701+0000 7fe88ebbc740 1 -- 172.21.15.114:0/252580805 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=4+}) -- 0x55adc1df3c10 con 0x55adc19cd1b0 2024-05-27T07:56:20.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.700+0000 7fc72ccf0b80 1 Processor -- start 2024-05-27T07:56:20.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.701+0000 7fc72ccf0b80 1 -- start start 2024-05-27T07:56:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.704+0000 7fe8697fa640 1 -- 172.21.15.114:0/252580805 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe874004040 con 0x55adc19cd1b0 2024-05-27T07:56:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.704+0000 7fe8697fa640 1 -- 172.21.15.114:0/252580805 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 93) ==== 4700+0+0 (secure 0 0 0) 0x7fe878024a10 con 0x55adc19cd1b0 2024-05-27T07:56:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.706+0000 7fe88ebbc740 10 client.28765 fetch_fsmap finished waiting for FSMap version 93 2024-05-27T07:56:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.706+0000 7fe88ebbc740 10 client.28765 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.706+0000 7fe88ebbc740 20 client.28765 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.706+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/252580805 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55adc1df4210 0x55adc1df66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.706+0000 7fe88ebbc740 4 client.28765 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:56:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.706+0000 7fe88ebbc740 1 -- 172.21.15.114:0/252580805 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55adc1cdc360 con 0x55adc1df4210 2024-05-27T07:56:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.701+0000 7fc72ccf0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56381533aec0 0x5638154b9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.701+0000 7fc72ccf0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.701+0000 7fc72ccf0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638154c00f0 0x5638154c4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.701+0000 7fc72ccf0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc70400f350 con 0x56381533aec0 2024-05-27T07:56:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.701+0000 7fc72ccf0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc70400fa20 con 0x5638154c00f0 2024-05-27T07:56:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.701+0000 7fc72ccf0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc70400e250 con 0x5638154ba1a0 2024-05-27T07:56:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.703+0000 7fc7177fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638154c00f0 0x5638154c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.703+0000 7fc7177fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56381533aec0 0x5638154b9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.703+0000 7fc7177fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56381533aec0 0x5638154b9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36960/0 (socket says 172.21.15.114:36960) 2024-05-27T07:56:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.703+0000 7fc7177fe700 1 -- 172.21.15.114:0/108533803 learned_addr learned my addr 172.21.15.114:0/108533803 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.704+0000 7fc7177fe700 1 -- 172.21.15.114:0/108533803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 msgr2=0x5638154aefa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:56:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.704+0000 7fc7177fe700 1 --2- 172.21.15.114:0/108533803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154aefa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.704+0000 7fc7177fe700 1 -- 172.21.15.114:0/108533803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638154c00f0 msgr2=0x5638154c4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.704+0000 7fc7177fe700 1 --2- 172.21.15.114:0/108533803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638154c00f0 0x5638154c4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.704+0000 7fc7177fe700 1 -- 172.21.15.114:0/108533803 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638154ae370 con 0x56381533aec0 2024-05-27T07:56:20.716 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.713+0000 7fe8859c3640 1 --2- 172.21.15.114:0/252580805 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55adc1df4210 0x55adc1df66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.712+0000 7fc7177fe700 1 --2- 172.21.15.114:0/108533803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56381533aec0 0x5638154b9c60 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fc70c000c00 tx=0x7fc70c004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ac8bd5e57268e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:20.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.714+0000 7fc6fdffb700 1 -- 172.21.15.114:0/108533803 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70c02f020 con 0x56381533aec0 2024-05-27T07:56:20.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.714+0000 7fc6fdffb700 1 -- 172.21.15.114:0/108533803 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc70c0044e0 con 0x56381533aec0 2024-05-27T07:56:20.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.715+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/108533803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56381533aec0 msgr2=0x5638154b9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.715+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/108533803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56381533aec0 0x5638154b9c60 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fc70c000c00 tx=0x7fc70c004860 comp rx=0 tx=0).stop 2024-05-27T07:56:20.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.718+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/108533803 shutdown_connections 2024-05-27T07:56:20.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.718+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/108533803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638154c00f0 0x5638154c4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.718+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/108533803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56381533aec0 0x5638154b9c60 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.718+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/108533803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.718+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/108533803 >> 172.21.15.114:0/108533803 conn(0x56381533d7f0 msgr2=0x5638154bdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:20.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.718+0000 7fe8859c3640 1 --2- 172.21.15.114:0/252580805 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55adc1df4210 0x55adc1df66e0 crc :-1 s=READY pgs=454 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e52996f88671852 server_cookie=3b852a4659214df in_seq=0 out_seq=0 2024-05-27T07:56:20.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:20.721+0000 7fe8697fa640 10 client.28765 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:20.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.720+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/108533803 shutdown_connections 2024-05-27T07:56:20.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.721+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/108533803 wait complete. 2024-05-27T07:56:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.721+0000 7fc72ccf0b80 1 -- start start 2024-05-27T07:56:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.722+0000 7fc72ccf0b80 1 Processor -- start 2024-05-27T07:56:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.722+0000 7fc72ccf0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56381533aec0 0x5638154db710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.722+0000 7fc72ccf0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154dbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.722+0000 7fc72ccf0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638154c00f0 0x5638154dc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.722+0000 7fc72ccf0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc70400d4b0 con 0x5638154c00f0 2024-05-27T07:56:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.722+0000 7fc72ccf0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc70400ce40 con 0x56381533aec0 2024-05-27T07:56:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.722+0000 7fc72ccf0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc70400db80 con 0x5638154ba1a0 2024-05-27T07:56:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.722+0000 7fc72ccf0b80 10 client.0 _ll_register_callbacks cb 0x56381536e630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.724+0000 7fc7177fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638154c00f0 0x5638154dc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.724+0000 7fc7177fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638154c00f0 0x5638154dc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36962/0 (socket says 172.21.15.114:36962) 2024-05-27T07:56:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.724+0000 7fc7177fe700 1 -- 172.21.15.114:0/3497851534 learned_addr learned my addr 172.21.15.114:0/3497851534 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.724+0000 7fc7177fe700 1 --2- 172.21.15.114:0/3497851534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56381533aec0 0x5638154db710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.725+0000 7fc717fff700 1 --2- 172.21.15.114:0/3497851534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154dbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.726+0000 7fc7177fe700 1 -- 172.21.15.114:0/3497851534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 msgr2=0x5638154dbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.726+0000 7fc7177fe700 1 --2- 172.21.15.114:0/3497851534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154dbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.726+0000 7fc7177fe700 1 -- 172.21.15.114:0/3497851534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638154c00f0 msgr2=0x5638154dc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.726+0000 7fc7177fe700 1 --2- 172.21.15.114:0/3497851534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638154c00f0 0x5638154dc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.726+0000 7fc7177fe700 1 -- 172.21.15.114:0/3497851534 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56381554ea40 con 0x56381533aec0 2024-05-27T07:56:20.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.727+0000 7fc7177fe700 1 --2- 172.21.15.114:0/3497851534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56381533aec0 0x5638154db710 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00ab90 tx=0x7fc70c00a2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ac60c15b56f2de3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:20.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.727+0000 7fc6f77fe700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:56:20.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.729+0000 7fc6f77fe700 1 -- 172.21.15.114:0/3497851534 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70c02f070 con 0x56381533aec0 2024-05-27T07:56:20.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.729+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/3497851534 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc70c0044e0 con 0x56381533aec0 2024-05-27T07:56:20.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.729+0000 7fc6f77fe700 1 -- 172.21.15.114:0/3497851534 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(148..148 src has 1..148) ==== 9554+0+0 (secure 0 0 0) 0x7fc70c014680 con 0x56381533aec0 2024-05-27T07:56:20.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.733+0000 7fc6f77fe700 1 -- 172.21.15.114:0/3497851534 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fc70c00a020 con 0x56381533aec0 2024-05-27T07:56:20.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.734+0000 7fc72ccf0b80 10 client.28637 fetch_fsmap learned FSMap version 93 2024-05-27T07:56:20.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.734+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/3497851534 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=4+}) -- 0x56381554fcc0 con 0x56381533aec0 2024-05-27T07:56:20.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.734+0000 7fc6f77fe700 1 -- 172.21.15.114:0/3497851534 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc70c0044e0 con 0x56381533aec0 2024-05-27T07:56:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.734+0000 7fc6f77fe700 1 -- 172.21.15.114:0/3497851534 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 93) ==== 4700+0+0 (secure 0 0 0) 0x7fc70c034040 con 0x56381533aec0 2024-05-27T07:56:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.734+0000 7fc72ccf0b80 10 client.28637 fetch_fsmap finished waiting for FSMap version 93 2024-05-27T07:56:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.734+0000 7fc72ccf0b80 10 client.28637 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.734+0000 7fc72ccf0b80 20 client.28637 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.734+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/3497851534 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5638155502a0 0x5638155526c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.734+0000 7fc72ccf0b80 4 client.28637 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:56:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.734+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/3497851534 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5638153e3520 con 0x5638155502a0 2024-05-27T07:56:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.735+0000 7fc717fff700 1 --2- 172.21.15.114:0/3497851534 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5638155502a0 0x5638155526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.743+0000 7fc6f77fe700 10 client.28637 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.743+0000 7fc717fff700 1 --2- 172.21.15.114:0/3497851534 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5638155502a0 0x5638155526c0 crc :-1 s=READY pgs=455 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df6415d74a1f0c6d server_cookie=852ba7f3d5927b07 in_seq=0 out_seq=0 2024-05-27T07:56:20.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.753+0000 7fc6f77fe700 1 -- 172.21.15.114:0/3497851534 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5638153e3520 con 0x5638155502a0 2024-05-27T07:56:20.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.753+0000 7fc6f77fe700 10 client.28637 handle_command_reply: tid=0 2024-05-27T07:56:20.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.753+0000 7fc72ccf0b80 1 client.28637 shutdown 2024-05-27T07:56:20.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.755+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/3497851534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56381533aec0 msgr2=0x5638154db710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.755+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/3497851534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56381533aec0 0x5638154db710 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00ab90 tx=0x7fc70c00a2f0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.756+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/3497851534 shutdown_connections 2024-05-27T07:56:20.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.756+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/3497851534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638154c00f0 0x5638154dc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.756+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/3497851534 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5638155502a0 0x5638155526c0 crc :-1 s=READY pgs=455 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.756+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/3497851534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56381533aec0 0x5638154db710 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.756+0000 7fc72ccf0b80 1 --2- 172.21.15.114:0/3497851534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638154ba1a0 0x5638154dbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.756+0000 7fc6f77fe700 0 client.28637 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:20.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.756+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/3497851534 >> 172.21.15.114:0/3497851534 conn(0x56381533d7f0 msgr2=0x5638154c06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:20.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.759+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/3497851534 shutdown_connections 2024-05-27T07:56:20.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.759+0000 7fc72ccf0b80 1 -- 172.21.15.114:0/3497851534 wait complete. 2024-05-27T07:56:20.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.760+0000 7fc72ccf0b80 20 client.28637 trim_cache size 0 max 16384 2024-05-27T07:56:20.764 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:56:20.765 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:56:20.765 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "b3cd9c24-f7f0-4371-9b36-07866e5f3fc9", 2024-05-27T07:56:20.765 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:56:20.765 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:56:20.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.764+0000 7fc71e2db700 1 -- 172.21.15.114:0/3848112766 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7000b2c00 msgr2=0x7fc7000b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.764+0000 7fc71e2db700 1 --2- 172.21.15.114:0/3848112766 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7000b2c00 0x7fc7000b5110 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc710009e60 tx=0x7fc710013c40 comp rx=0 tx=0).stop 2024-05-27T07:56:20.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.764+0000 7fc71e2db700 1 -- 172.21.15.114:0/3848112766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc71810c5a0 msgr2=0x7fc71819d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:20.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.764+0000 7fc71e2db700 1 --2- 172.21.15.114:0/3848112766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc71810c5a0 0x7fc71819d570 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc710013930 tx=0x7fc7100077d0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.765+0000 7fc71e2db700 1 -- 172.21.15.114:0/3848112766 shutdown_connections 2024-05-27T07:56:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.765+0000 7fc71e2db700 1 --2- 172.21.15.114:0/3848112766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc71810c5a0 0x7fc71819d570 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.765+0000 7fc71e2db700 1 --2- 172.21.15.114:0/3848112766 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7000b2c00 0x7fc7000b5110 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.765+0000 7fc71e2db700 1 --2- 172.21.15.114:0/3848112766 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc718106560 0x7fc718198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.765+0000 7fc71e2db700 1 --2- 172.21.15.114:0/3848112766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718106eb0 0x7fc7181993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.765+0000 7fc71e2db700 1 -- 172.21.15.114:0/3848112766 >> 172.21.15.114:0/3848112766 conn(0x7fc718102040 msgr2=0x7fc718102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:20.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.766+0000 7fc71e2db700 1 -- 172.21.15.114:0/3848112766 shutdown_connections 2024-05-27T07:56:20.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:20.768+0000 7fc71e2db700 1 -- 172.21.15.114:0/3848112766 wait complete. 2024-05-27T07:56:21.603 INFO:tasks.workunit.client.0.smithi114.stdout: 10 36347 9.72 MB/sec execute 33 sec latency 487.982 ms 2024-05-27T07:56:21.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:56:21.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.842+0000 7fe8697fa640 1 -- 172.21.15.114:0/252580805 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55adc1cdc360 con 0x55adc1df4210 2024-05-27T07:56:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.842+0000 7fe8697fa640 10 client.28765 handle_command_reply: tid=0 2024-05-27T07:56:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.842+0000 7fe88ebbc740 1 client.28765 shutdown 2024-05-27T07:56:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.842+0000 7fe88ebbc740 1 -- 172.21.15.114:0/252580805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 msgr2=0x55adc1d7cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.842+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/252580805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 0x55adc1d7cab0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe87800c6e0 tx=0x7fe8780074e0 comp rx=0 tx=0).stop 2024-05-27T07:56:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.842+0000 7fe88ebbc740 1 -- 172.21.15.114:0/252580805 shutdown_connections 2024-05-27T07:56:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.842+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/252580805 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55adc1df4210 0x55adc1df66e0 crc :-1 s=READY pgs=454 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.842+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/252580805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55adc19cd1b0 0x55adc1d7cab0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.842+0000 7fe8697fa640 0 client.28765 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.842+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/252580805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adc198bd80 0x55adc1d7c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.842+0000 7fe88ebbc740 1 --2- 172.21.15.114:0/252580805 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55adc19611c0 0x55adc1d81130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.842+0000 7fe88ebbc740 1 -- 172.21.15.114:0/252580805 >> 172.21.15.114:0/252580805 conn(0x55adc19c1300 msgr2=0x55adc1d698e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.843+0000 7fe88ebbc740 1 -- 172.21.15.114:0/252580805 shutdown_connections 2024-05-27T07:56:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.843+0000 7fe88ebbc740 1 -- 172.21.15.114:0/252580805 wait complete. 2024-05-27T07:56:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.843+0000 7fe88ebbc740 20 client.28765 trim_cache size 0 max 16384 2024-05-27T07:56:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:56:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:56:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:56:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.843+0000 7fe887c27640 1 -- 172.21.15.114:0/3588953672 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe86c0b29b0 msgr2=0x7fe86c0b4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.843+0000 7fe887c27640 1 --2- 172.21.15.114:0/3588953672 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe86c0b29b0 0x7fe86c0b4ec0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe87800d010 tx=0x7fe87801b040 comp rx=0 tx=0).stop 2024-05-27T07:56:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.843+0000 7fe887c27640 1 -- 172.21.15.114:0/3588953672 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe88011a8e0 msgr2=0x7fe880113420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.843+0000 7fe887c27640 1 --2- 172.21.15.114:0/3588953672 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe88011a8e0 0x7fe880113420 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00b450 tx=0x7fe87c00b910 comp rx=0 tx=0).stop 2024-05-27T07:56:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.844+0000 7fe887c27640 1 -- 172.21.15.114:0/3588953672 shutdown_connections 2024-05-27T07:56:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.844+0000 7fe887c27640 1 --2- 172.21.15.114:0/3588953672 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe86c0b29b0 0x7fe86c0b4ec0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.844+0000 7fe887c27640 1 --2- 172.21.15.114:0/3588953672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe88011d240 0x7fe880113aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.844+0000 7fe887c27640 1 --2- 172.21.15.114:0/3588953672 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe88011a8e0 0x7fe880113420 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.844+0000 7fe887c27640 1 --2- 172.21.15.114:0/3588953672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe88010a1f0 0x7fe88011a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:21.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.844+0000 7fe887c27640 1 -- 172.21.15.114:0/3588953672 >> 172.21.15.114:0/3588953672 conn(0x7fe88006e8f0 msgr2=0x7fe88011acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:21.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.844+0000 7fe887c27640 1 -- 172.21.15.114:0/3588953672 shutdown_connections 2024-05-27T07:56:21.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:21.844+0000 7fe887c27640 1 -- 172.21.15.114:0/3588953672 wait complete. 2024-05-27T07:56:21.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:21 smithi163 ceph-mon[36451]: pgmap v567: 129 pgs: 5 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 112 active+clean; 321 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.8 MiB/s wr, 294 op/s 2024-05-27T07:56:21.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:21 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:56:21.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:21 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:56:21.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:21 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:56:21.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:21 smithi186 ceph-mon[36647]: pgmap v567: 129 pgs: 5 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 112 active+clean; 321 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.8 MiB/s wr, 294 op/s 2024-05-27T07:56:21.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:21 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:56:21.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:21 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:56:21.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:21 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:56:22.011 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:56:22.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:21 smithi114 ceph-mon[32038]: pgmap v567: 129 pgs: 5 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 112 active+clean; 321 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.8 MiB/s wr, 294 op/s 2024-05-27T07:56:22.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:21 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:56:22.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:21 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:56:22.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:21 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:56:22.603 INFO:tasks.workunit.client.0.smithi114.stdout: 10 36603 9.90 MB/sec execute 34 sec latency 441.590 ms 2024-05-27T07:56:23.603 INFO:tasks.workunit.client.0.smithi114.stdout: 10 36838 9.97 MB/sec execute 35 sec latency 150.858 ms 2024-05-27T07:56:23.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.855+0000 7f4662209700 1 -- 172.21.15.114:0/3636767383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f465c108730 msgr2=0x7f465c110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:23.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.855+0000 7f4662209700 1 --2- 172.21.15.114:0/3636767383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f465c108730 0x7f465c110090 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f465400b0d0 tx=0x7f465400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.855+0000 7f4662209700 1 -- 172.21.15.114:0/3636767383 shutdown_connections 2024-05-27T07:56:23.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.855+0000 7f4662209700 1 --2- 172.21.15.114:0/3636767383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f465c108730 0x7f465c110090 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.855+0000 7f4662209700 1 --2- 172.21.15.114:0/3636767383 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f465c1034b0 0x7f465c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.855+0000 7f4662209700 1 --2- 172.21.15.114:0/3636767383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f465c107de0 0x7f465c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.855+0000 7f4662209700 1 -- 172.21.15.114:0/3636767383 >> 172.21.15.114:0/3636767383 conn(0x7f465c0fedb0 msgr2=0x7f465c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:23.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.855+0000 7f4662209700 1 -- 172.21.15.114:0/3636767383 shutdown_connections 2024-05-27T07:56:23.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.855+0000 7f4662209700 1 -- 172.21.15.114:0/3636767383 wait complete. 2024-05-27T07:56:23.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.856+0000 7f4662209700 1 Processor -- start 2024-05-27T07:56:23.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.856+0000 7f4662209700 1 -- start start 2024-05-27T07:56:23.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.857+0000 7f4662209700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f465c1034b0 0x7f465c19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.857+0000 7f4662209700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f465c107de0 0x7f465c19d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.857+0000 7f4662209700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f465c108730 0x7f465c1a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.857+0000 7f4662209700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f465c113360 con 0x7f465c107de0 2024-05-27T07:56:23.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.857+0000 7f4662209700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f465c1131e0 con 0x7f465c1034b0 2024-05-27T07:56:23.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.857+0000 7f4662209700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f465c1134e0 con 0x7f465c108730 2024-05-27T07:56:23.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.857+0000 7f465b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f465c1034b0 0x7f465c19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.857+0000 7f465b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f465c1034b0 0x7f465c19cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:59552/0 (socket says 172.21.15.114:59552) 2024-05-27T07:56:23.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.857+0000 7f465b7fe700 1 -- 172.21.15.114:0/488162874 learned_addr learned my addr 172.21.15.114:0/488162874 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:23.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.857+0000 7f465bfff700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f465c108730 0x7f465c1a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.857+0000 7f465b7fe700 1 -- 172.21.15.114:0/488162874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f465c108730 msgr2=0x7f465c1a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:23.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.857+0000 7f465b7fe700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f465c108730 0x7f465c1a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.857+0000 7f465b7fe700 1 -- 172.21.15.114:0/488162874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f465c107de0 msgr2=0x7f465c19d4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:56:23.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.858+0000 7f465affd700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f465c107de0 0x7f465c19d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.858+0000 7f465b7fe700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f465c107de0 0x7f465c19d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.858+0000 7f465b7fe700 1 -- 172.21.15.114:0/488162874 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f465c1a1e20 con 0x7f465c1034b0 2024-05-27T07:56:23.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.858+0000 7f465affd700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f465c107de0 0x7f465c19d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:56:23.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.858+0000 7f465b7fe700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f465c1034b0 0x7f465c19cf50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f464c00eb30 tx=0x7f464c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:23.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.858+0000 7f4658ff9700 1 -- 172.21.15.114:0/488162874 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464c00cc90 con 0x7f465c1034b0 2024-05-27T07:56:23.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.858+0000 7f4662209700 1 -- 172.21.15.114:0/488162874 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f465c1a2110 con 0x7f465c1034b0 2024-05-27T07:56:23.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.858+0000 7f4662209700 1 -- 172.21.15.114:0/488162874 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f465c1a9a30 con 0x7f465c1034b0 2024-05-27T07:56:23.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.858+0000 7f4658ff9700 1 -- 172.21.15.114:0/488162874 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f464c0044d0 con 0x7f465c1034b0 2024-05-27T07:56:23.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.858+0000 7f4658ff9700 1 -- 172.21.15.114:0/488162874 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464c010430 con 0x7f465c1034b0 2024-05-27T07:56:23.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.862+0000 7f4658ff9700 1 -- 172.21.15.114:0/488162874 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f464c0105d0 con 0x7f465c1034b0 2024-05-27T07:56:23.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.862+0000 7f4670c1eb80 1 Processor -- start 2024-05-27T07:56:23.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.862+0000 7f4670c1eb80 1 -- start start 2024-05-27T07:56:23.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.863+0000 7f4670c1eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 0x557965921b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.863+0000 7f4670c1eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557965922070 0x55796591bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.863+0000 7f4670c1eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55796591c120 0x55796591c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.863+0000 7f4670c1eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557965849d70 con 0x55796591c120 2024-05-27T07:56:23.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.863+0000 7f4670c1eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557965839910 con 0x557965922070 2024-05-27T07:56:23.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.863+0000 7f4670c1eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55796583d950 con 0x55796579ce40 2024-05-27T07:56:23.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.863+0000 7f465bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 0x557965921b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.863+0000 7f465bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 0x557965921b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40982/0 (socket says 172.21.15.114:40982) 2024-05-27T07:56:23.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.863+0000 7f465bfff700 1 -- 172.21.15.114:0/2177970163 learned_addr learned my addr 172.21.15.114:0/2177970163 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:23.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.863+0000 7f465affd700 1 --2- 172.21.15.114:0/2177970163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55796591c120 0x55796591c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.863+0000 7f465b7fe700 1 --2- 172.21.15.114:0/2177970163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557965922070 0x55796591bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.864+0000 7f465bfff700 1 -- 172.21.15.114:0/2177970163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557965922070 msgr2=0x55796591bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:23.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.864+0000 7f465bfff700 1 --2- 172.21.15.114:0/2177970163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557965922070 0x55796591bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.864+0000 7f465bfff700 1 -- 172.21.15.114:0/2177970163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55796591c120 msgr2=0x55796591c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:23.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.864+0000 7f465bfff700 1 --2- 172.21.15.114:0/2177970163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55796591c120 0x55796591c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.864+0000 7f465bfff700 1 -- 172.21.15.114:0/2177970163 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579657e2fb0 con 0x55796579ce40 2024-05-27T07:56:23.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.864+0000 7f465affd700 1 --2- 172.21.15.114:0/2177970163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55796591c120 0x55796591c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:56:23.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.864+0000 7f4658ff9700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46440b2a70 0x7f46440b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.864+0000 7f465bfff700 1 --2- 172.21.15.114:0/2177970163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 0x557965921b30 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f46540098f0 tx=0x7f4654009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=909b03531f84cedd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:23.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.864+0000 7f465b7fe700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46440b2a70 0x7f46440b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.864+0000 7f4641ffb700 1 -- 172.21.15.114:0/2177970163 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46540092e0 con 0x55796579ce40 2024-05-27T07:56:23.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.864+0000 7f4641ffb700 1 -- 172.21.15.114:0/2177970163 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4654020070 con 0x55796579ce40 2024-05-27T07:56:23.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.864+0000 7f4641ffb700 1 -- 172.21.15.114:0/2177970163 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46540053e0 con 0x55796579ce40 2024-05-27T07:56:23.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.864+0000 7f4658ff9700 1 -- 172.21.15.114:0/488162874 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 9554+0+0 (secure 0 0 0) 0x7f464c0e2b50 con 0x7f465c1034b0 2024-05-27T07:56:23.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.865+0000 7f4670c1eb80 1 -- 172.21.15.114:0/2177970163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 msgr2=0x557965921b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:23.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.865+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/2177970163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 0x557965921b30 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f46540098f0 tx=0x7f4654009b80 comp rx=0 tx=0).stop 2024-05-27T07:56:23.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.865+0000 7f465b7fe700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46440b2a70 0x7f46440b4f80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f464c0dff40 tx=0x7f464c0e66a0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:23.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.865+0000 7f4670c1eb80 1 -- 172.21.15.114:0/2177970163 shutdown_connections 2024-05-27T07:56:23.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.865+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/2177970163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55796591c120 0x55796591c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.865+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/2177970163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 0x557965921b30 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.865+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/2177970163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557965922070 0x55796591bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.865+0000 7f4670c1eb80 1 -- 172.21.15.114:0/2177970163 >> 172.21.15.114:0/2177970163 conn(0x55796579f770 msgr2=0x557965916d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.865+0000 7f4670c1eb80 1 -- 172.21.15.114:0/2177970163 shutdown_connections 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.865+0000 7f4670c1eb80 1 -- 172.21.15.114:0/2177970163 wait complete. 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.866+0000 7f4670c1eb80 1 -- start start 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.866+0000 7f4670c1eb80 1 Processor -- start 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.866+0000 7f4670c1eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55796579ce40 0x557965925840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.866+0000 7f4670c1eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796591c120 0x557965925d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.866+0000 7f4670c1eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557965922070 0x5579659262c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.866+0000 7f4670c1eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55796573ffa0 con 0x557965922070 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.866+0000 7f4670c1eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5579657a1da0 con 0x55796579ce40 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.866+0000 7f4670c1eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557965928aa0 con 0x55796591c120 2024-05-27T07:56:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f4670c1eb80 10 client.0 _ll_register_callbacks cb 0x5579657d05b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f465bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55796579ce40 0x557965925840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f465bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55796579ce40 0x557965925840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:59580/0 (socket says 172.21.15.114:59580) 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f465bfff700 1 -- 172.21.15.114:0/881980572 learned_addr learned my addr 172.21.15.114:0/881980572 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f465bfff700 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557965922070 0x5579659262c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f465affd700 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796591c120 0x557965925d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f465bfff700 1 -- 172.21.15.114:0/881980572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796591c120 msgr2=0x557965925d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f465bfff700 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796591c120 0x557965925d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f465bfff700 1 -- 172.21.15.114:0/881980572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557965922070 msgr2=0x5579659262c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f465bfff700 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557965922070 0x5579659262c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f465bfff700 1 -- 172.21.15.114:0/881980572 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579657898d0 con 0x55796579ce40 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f465affd700 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796591c120 0x557965925d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f465bfff700 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55796579ce40 0x557965925840 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4654005780 tx=0x7f4654003ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa852309eb6cfafe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.867+0000 7f463affd700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:56:23.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f463affd700 1 -- 172.21.15.114:0/881980572 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4654005390 con 0x55796579ce40 2024-05-27T07:56:23.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f463affd700 1 -- 172.21.15.114:0/881980572 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(148..148 src has 1..148) ==== 9554+0+0 (secure 0 0 0) 0x7f465400baa0 con 0x55796579ce40 2024-05-27T07:56:23.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f4670c1eb80 1 -- 172.21.15.114:0/881980572 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4654020070 con 0x55796579ce40 2024-05-27T07:56:23.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f463affd700 1 -- 172.21.15.114:0/881980572 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f4654038050 con 0x55796579ce40 2024-05-27T07:56:23.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f4670c1eb80 10 client.28649 fetch_fsmap learned FSMap version 93 2024-05-27T07:56:23.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f4670c1eb80 1 -- 172.21.15.114:0/881980572 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=4+}) -- 0x5579657f3760 con 0x55796579ce40 2024-05-27T07:56:23.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f463affd700 1 -- 172.21.15.114:0/881980572 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4654020070 con 0x55796579ce40 2024-05-27T07:56:23.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f463affd700 1 -- 172.21.15.114:0/881980572 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 93) ==== 4700+0+0 (secure 0 0 0) 0x7f4654011070 con 0x55796579ce40 2024-05-27T07:56:23.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f4670c1eb80 10 client.28649 fetch_fsmap finished waiting for FSMap version 93 2024-05-27T07:56:23.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f4670c1eb80 10 client.28649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:23.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f4670c1eb80 20 client.28649 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:23.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557965935990 0x557965937de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f4670c1eb80 4 client.28649 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:56:23.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f4670c1eb80 1 -- 172.21.15.114:0/881980572 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55796584bcc0 con 0x557965935990 2024-05-27T07:56:23.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.868+0000 7f465affd700 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557965935990 0x557965937de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.869+0000 7f463affd700 10 client.28649 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:23.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.869+0000 7f465affd700 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557965935990 0x557965937de0 crc :-1 s=READY pgs=456 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e343615b2c6ae0f server_cookie=cd0c2a90ff4fe3bb in_seq=0 out_seq=0 2024-05-27T07:56:23.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.873+0000 7f463affd700 1 -- 172.21.15.114:0/881980572 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55796584bcc0 con 0x557965935990 2024-05-27T07:56:23.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.873+0000 7f463affd700 10 client.28649 handle_command_reply: tid=0 2024-05-27T07:56:23.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.873+0000 7f4670c1eb80 1 client.28649 shutdown 2024-05-27T07:56:23.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.873+0000 7f4670c1eb80 1 -- 172.21.15.114:0/881980572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55796579ce40 msgr2=0x557965925840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.873+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55796579ce40 0x557965925840 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4654005780 tx=0x7f4654003ee0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.873+0000 7f4670c1eb80 1 -- 172.21.15.114:0/881980572 shutdown_connections 2024-05-27T07:56:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.873+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557965922070 0x5579659262c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.873+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557965935990 0x557965937de0 crc :-1 s=READY pgs=456 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.873+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55796579ce40 0x557965925840 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.873+0000 7f463affd700 0 client.28649 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.873+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/881980572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796591c120 0x557965925d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.873+0000 7f4670c1eb80 1 -- 172.21.15.114:0/881980572 >> 172.21.15.114:0/881980572 conn(0x55796579f770 msgr2=0x55796591abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.874+0000 7f4670c1eb80 1 -- 172.21.15.114:0/881980572 shutdown_connections 2024-05-27T07:56:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.874+0000 7f4670c1eb80 1 -- 172.21.15.114:0/881980572 wait complete. 2024-05-27T07:56:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.874+0000 7f4670c1eb80 20 client.28649 trim_cache size 0 max 16384 2024-05-27T07:56:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.896+0000 7f4670c1eb80 1 Processor -- start 2024-05-27T07:56:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.896+0000 7f4670c1eb80 1 -- start start 2024-05-27T07:56:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.896+0000 7f4670c1eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 0x55796591bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.896+0000 7f4670c1eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55796591c120 0x557965931b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.896+0000 7f4670c1eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557965922070 0x557965925f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.896+0000 7f4670c1eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f464800f230 con 0x557965922070 2024-05-27T07:56:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.896+0000 7f4670c1eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f464800f900 con 0x55796591c120 2024-05-27T07:56:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.896+0000 7f4670c1eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f464800e130 con 0x55796579ce40 2024-05-27T07:56:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.896+0000 7f465bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 0x55796591bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.896+0000 7f465bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 0x55796591bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40998/0 (socket says 172.21.15.114:40998) 2024-05-27T07:56:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.896+0000 7f465bfff700 1 -- 172.21.15.114:0/972703830 learned_addr learned my addr 172.21.15.114:0/972703830 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.896+0000 7f465bfff700 1 --2- 172.21.15.114:0/972703830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557965922070 0x557965925f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.896+0000 7f465affd700 1 --2- 172.21.15.114:0/972703830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55796591c120 0x557965931b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.897+0000 7f465bfff700 1 -- 172.21.15.114:0/972703830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55796591c120 msgr2=0x557965931b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.897+0000 7f465bfff700 1 --2- 172.21.15.114:0/972703830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55796591c120 0x557965931b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.897+0000 7f465bfff700 1 -- 172.21.15.114:0/972703830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557965922070 msgr2=0x557965925f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.897+0000 7f465bfff700 1 --2- 172.21.15.114:0/972703830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557965922070 0x557965925f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.897+0000 7f465bfff700 1 -- 172.21.15.114:0/972703830 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557965925610 con 0x55796579ce40 2024-05-27T07:56:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.897+0000 7f465bfff700 1 --2- 172.21.15.114:0/972703830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 0x55796591bbe0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f46540021d0 tx=0x7f465400b3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34f20a1ca9243b58 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.897+0000 7f46417fa700 1 -- 172.21.15.114:0/972703830 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4654009b80 con 0x55796579ce40 2024-05-27T07:56:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.897+0000 7f46417fa700 1 -- 172.21.15.114:0/972703830 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4654020020 con 0x55796579ce40 2024-05-27T07:56:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.897+0000 7f46417fa700 1 -- 172.21.15.114:0/972703830 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4654005dd0 con 0x55796579ce40 2024-05-27T07:56:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.898+0000 7f4670c1eb80 1 -- 172.21.15.114:0/972703830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 msgr2=0x55796591bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.898+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/972703830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 0x55796591bbe0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f46540021d0 tx=0x7f465400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.898+0000 7f4670c1eb80 1 -- 172.21.15.114:0/972703830 shutdown_connections 2024-05-27T07:56:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.898+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/972703830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557965922070 0x557965925f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.898+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/972703830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796579ce40 0x55796591bbe0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.898+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/972703830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55796591c120 0x557965931b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.898+0000 7f4670c1eb80 1 -- 172.21.15.114:0/972703830 >> 172.21.15.114:0/972703830 conn(0x55796579f770 msgr2=0x557965925800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.898+0000 7f4670c1eb80 1 -- 172.21.15.114:0/972703830 shutdown_connections 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.899+0000 7f4670c1eb80 1 -- 172.21.15.114:0/972703830 wait complete. 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.899+0000 7f4670c1eb80 1 -- start start 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.899+0000 7f4670c1eb80 1 Processor -- start 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f4670c1eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55796579ce40 0x55796592e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f4670c1eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796591c120 0x55796593cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f4670c1eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557965922070 0x55796593d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f4670c1eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f464800d390 con 0x55796579ce40 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f4670c1eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f464800cd20 con 0x557965922070 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f4670c1eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f464800da60 con 0x55796591c120 2024-05-27T07:56:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f4670c1eb80 10 client.0 _ll_register_callbacks cb 0x5579657d05b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f465affd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796591c120 0x55796593cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f465affd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796591c120 0x55796593cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:41000/0 (socket says 172.21.15.114:41000) 2024-05-27T07:56:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f465affd700 1 -- 172.21.15.114:0/3341128692 learned_addr learned my addr 172.21.15.114:0/3341128692 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f465bfff700 1 --2- 172.21.15.114:0/3341128692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55796579ce40 0x55796592e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f465bfff700 1 --2- 172.21.15.114:0/3341128692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557965922070 0x55796593d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f465bfff700 1 -- 172.21.15.114:0/3341128692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796591c120 msgr2=0x55796593cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f465bfff700 1 --2- 172.21.15.114:0/3341128692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796591c120 0x55796593cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f465bfff700 1 -- 172.21.15.114:0/3341128692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557965922070 msgr2=0x55796593d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f465bfff700 1 --2- 172.21.15.114:0/3341128692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557965922070 0x55796593d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.900+0000 7f465bfff700 1 -- 172.21.15.114:0/3341128692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55796593d5f0 con 0x55796579ce40 2024-05-27T07:56:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.901+0000 7f465bfff700 1 --2- 172.21.15.114:0/3341128692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55796579ce40 0x55796592e0d0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f4654024520 tx=0x7f4654005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22bc60ae701a421 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.901+0000 7f463affd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:56:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.901+0000 7f463affd700 1 -- 172.21.15.114:0/3341128692 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465400bcb0 con 0x55796579ce40 2024-05-27T07:56:23.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.901+0000 7f4670c1eb80 1 -- 172.21.15.114:0/3341128692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4654020020 con 0x55796579ce40 2024-05-27T07:56:23.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.901+0000 7f463affd700 1 -- 172.21.15.114:0/3341128692 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(148..148 src has 1..148) ==== 9554+0+0 (secure 0 0 0) 0x7f4654005c50 con 0x55796579ce40 2024-05-27T07:56:23.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.901+0000 7f463affd700 1 -- 172.21.15.114:0/3341128692 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f4654005ed0 con 0x55796579ce40 2024-05-27T07:56:23.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.901+0000 7f4670c1eb80 10 client.19302 fetch_fsmap learned FSMap version 93 2024-05-27T07:56:23.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.901+0000 7f4670c1eb80 1 -- 172.21.15.114:0/3341128692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=4+}) -- 0x5579659a0040 con 0x55796579ce40 2024-05-27T07:56:23.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.902+0000 7f463affd700 1 -- 172.21.15.114:0/3341128692 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4654020020 con 0x55796579ce40 2024-05-27T07:56:23.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.902+0000 7f463affd700 1 -- 172.21.15.114:0/3341128692 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 93) ==== 4700+0+0 (secure 0 0 0) 0x7f4654011070 con 0x55796579ce40 2024-05-27T07:56:23.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.902+0000 7f4670c1eb80 10 client.19302 fetch_fsmap finished waiting for FSMap version 93 2024-05-27T07:56:23.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.902+0000 7f4670c1eb80 10 client.19302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:23.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.902+0000 7f4670c1eb80 20 client.19302 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:23.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.902+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/3341128692 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557965935990 0x5579659a24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:23.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.902+0000 7f4670c1eb80 4 client.19302 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:56:23.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.902+0000 7f4670c1eb80 1 -- 172.21.15.114:0/3341128692 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5579658454a0 con 0x557965935990 2024-05-27T07:56:23.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.902+0000 7f465affd700 1 --2- 172.21.15.114:0/3341128692 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557965935990 0x5579659a24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:23.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.903+0000 7f463affd700 10 client.19302 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:23.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.903+0000 7f465affd700 1 --2- 172.21.15.114:0/3341128692 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557965935990 0x5579659a24b0 crc :-1 s=READY pgs=457 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=424bf06885fa424f server_cookie=88163d5c37d627ba in_seq=0 out_seq=0 2024-05-27T07:56:23.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.905+0000 7f463affd700 1 -- 172.21.15.114:0/3341128692 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5579658454a0 con 0x557965935990 2024-05-27T07:56:23.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.905+0000 7f463affd700 10 client.19302 handle_command_reply: tid=0 2024-05-27T07:56:23.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.905+0000 7f4670c1eb80 1 client.19302 shutdown 2024-05-27T07:56:23.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.906+0000 7f4670c1eb80 1 -- 172.21.15.114:0/3341128692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55796579ce40 msgr2=0x55796592e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:23.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.906+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/3341128692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55796579ce40 0x55796592e0d0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f4654024520 tx=0x7f4654005bb0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.906+0000 7f4670c1eb80 1 -- 172.21.15.114:0/3341128692 shutdown_connections 2024-05-27T07:56:23.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.906+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/3341128692 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x557965935990 0x5579659a24b0 crc :-1 s=READY pgs=457 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.906+0000 7f463affd700 0 client.19302 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:23.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.906+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/3341128692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557965922070 0x55796593d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.906+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/3341128692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55796579ce40 0x55796592e0d0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.906+0000 7f4670c1eb80 1 --2- 172.21.15.114:0/3341128692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55796591c120 0x55796593cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.906+0000 7f4670c1eb80 1 -- 172.21.15.114:0/3341128692 >> 172.21.15.114:0/3341128692 conn(0x55796579f770 msgr2=0x5579659104f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:23.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.906+0000 7f4670c1eb80 1 -- 172.21.15.114:0/3341128692 shutdown_connections 2024-05-27T07:56:23.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.906+0000 7f4670c1eb80 1 -- 172.21.15.114:0/3341128692 wait complete. 2024-05-27T07:56:23.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.906+0000 7f4670c1eb80 20 client.19302 trim_cache size 0 max 16384 2024-05-27T07:56:23.909 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:56:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:56:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:56:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: "b3cd9c24-f7f0-4371-9b36-07866e5f3fc9": { 2024-05-27T07:56:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:56:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "b3cd9c24-f7f0-4371-9b36-07866e5f3fc9", 2024-05-27T07:56:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:56:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:56:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:56:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:56:23.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.907+0000 7f4662209700 1 -- 172.21.15.114:0/488162874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46440b2a70 msgr2=0x7f46440b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:23.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.907+0000 7f4662209700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46440b2a70 0x7f46440b4f80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f464c0dff40 tx=0x7f464c0e66a0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.907+0000 7f4662209700 1 -- 172.21.15.114:0/488162874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f465c1034b0 msgr2=0x7f465c19cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:23.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.907+0000 7f4662209700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f465c1034b0 0x7f465c19cf50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f464c00eb30 tx=0x7f464c00ee40 comp rx=0 tx=0).stop 2024-05-27T07:56:23.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.907+0000 7f4662209700 1 -- 172.21.15.114:0/488162874 shutdown_connections 2024-05-27T07:56:23.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.907+0000 7f4662209700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f465c108730 0x7f465c1a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.907+0000 7f4662209700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f46440b2a70 0x7f46440b4f80 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.907+0000 7f4662209700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f465c1034b0 0x7f465c19cf50 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.907+0000 7f4662209700 1 --2- 172.21.15.114:0/488162874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f465c107de0 0x7f465c19d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.907+0000 7f4662209700 1 -- 172.21.15.114:0/488162874 >> 172.21.15.114:0/488162874 conn(0x7f465c0fedb0 msgr2=0x7f465c10fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.907+0000 7f4662209700 1 -- 172.21.15.114:0/488162874 shutdown_connections 2024-05-27T07:56:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:23.907+0000 7f4662209700 1 -- 172.21.15.114:0/488162874 wait complete. 2024-05-27T07:56:23.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:23 smithi163 ceph-mon[36451]: pgmap v568: 129 pgs: 129 active+clean; 323 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.4 MiB/s wr, 328 op/s 2024-05-27T07:56:23.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:23 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:56:23.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:23 smithi186 ceph-mon[36647]: pgmap v568: 129 pgs: 129 active+clean; 323 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.4 MiB/s wr, 328 op/s 2024-05-27T07:56:23.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:23 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:56:24.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:23 smithi114 ceph-mon[32038]: pgmap v568: 129 pgs: 129 active+clean; 323 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.4 MiB/s wr, 328 op/s 2024-05-27T07:56:24.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:23 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:56:24.603 INFO:tasks.workunit.client.0.smithi114.stdout: 10 37120 10.14 MB/sec execute 36 sec latency 349.261 ms 2024-05-27T07:56:24.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:24 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:56:24.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:24 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:56:24.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:24 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:56:24.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:24 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:56:24.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:24 smithi163 ceph-mon[36451]: osdmap e149: 12 total, 12 up, 12 in 2024-05-27T07:56:24.978 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:24 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:56:24.978 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:24 smithi163 systemd[1]: Stopped Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:56:24.978 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:24 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Start request repeated too quickly. 2024-05-27T07:56:24.978 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:24 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:56:24.978 INFO:journalctl@ceph.mds.k.smithi163.stdout:May 27 07:56:24 smithi163 systemd[1]: Failed to start Ceph mds.k for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:56:24.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:24 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:56:24.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:24 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:56:24.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:24 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:56:24.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:24 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:56:24.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:24 smithi186 ceph-mon[36647]: osdmap e149: 12 total, 12 up, 12 in 2024-05-27T07:56:25.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:24 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:56:25.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:24 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:56:25.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:24 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:56:25.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:24 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:56:25.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:24 smithi114 ceph-mon[32038]: osdmap e149: 12 total, 12 up, 12 in 2024-05-27T07:56:25.203 INFO:tasks.cephfs.filesystem:scrub status for tag:b3cd9c24-f7f0-4371-9b36-07866e5f3fc9 - {'path': '/', 'tag': 'b3cd9c24-f7f0-4371-9b36-07866e5f3fc9', 'options': 'recursive,force'} 2024-05-27T07:56:25.603 INFO:tasks.workunit.client.0.smithi114.stdout: 10 37499 10.20 MB/sec execute 37 sec latency 159.258 ms 2024-05-27T07:56:25.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:25 smithi163 ceph-mon[36451]: pgmap v569: 129 pgs: 129 active+clean; 325 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 332 op/s 2024-05-27T07:56:25.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:25 smithi163 ceph-mon[36451]: pgmap v570: 129 pgs: 129 active+clean; 325 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.3 MiB/s wr, 359 op/s 2024-05-27T07:56:25.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:25 smithi186 ceph-mon[36647]: pgmap v569: 129 pgs: 129 active+clean; 325 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 332 op/s 2024-05-27T07:56:25.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:25 smithi186 ceph-mon[36647]: pgmap v570: 129 pgs: 129 active+clean; 325 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.3 MiB/s wr, 359 op/s 2024-05-27T07:56:26.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:25 smithi114 ceph-mon[32038]: pgmap v569: 129 pgs: 129 active+clean; 325 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 332 op/s 2024-05-27T07:56:26.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:25 smithi114 ceph-mon[32038]: pgmap v570: 129 pgs: 129 active+clean; 325 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.3 MiB/s wr, 359 op/s 2024-05-27T07:56:26.603 INFO:tasks.workunit.client.0.smithi114.stdout: 10 37873 10.64 MB/sec execute 38 sec latency 157.952 ms 2024-05-27T07:56:27.603 INFO:tasks.workunit.client.0.smithi114.stdout: 10 38335 10.98 MB/sec execute 39 sec latency 139.345 ms 2024-05-27T07:56:27.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:27 smithi186 ceph-mon[36647]: pgmap v572: 129 pgs: 129 active+clean; 311 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 689 op/s 2024-05-27T07:56:27.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:27 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:56:28.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:27 smithi114 ceph-mon[32038]: pgmap v572: 129 pgs: 129 active+clean; 311 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 689 op/s 2024-05-27T07:56:28.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:27 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:56:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:27 smithi163 ceph-mon[36451]: pgmap v572: 129 pgs: 129 active+clean; 311 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 689 op/s 2024-05-27T07:56:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:27 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:56:28.603 INFO:tasks.workunit.client.0.smithi114.stdout: 10 38636 11.28 MB/sec execute 40 sec latency 247.771 ms 2024-05-27T07:56:29.603 INFO:tasks.workunit.client.0.smithi114.stdout: 10 38874 11.29 MB/sec execute 41 sec latency 281.723 ms 2024-05-27T07:56:29.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-mon[32038]: pgmap v573: 129 pgs: 129 active+clean; 313 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 650 op/s 2024-05-27T07:56:29.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-mon[32038]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:56:29.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:56:29.808 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7324526700 time 2024-05-27T07:56:29.527873+0000 2024-05-27T07:56:29.808 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:56:29.808 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:29.808 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7336d6bba6] 2024-05-27T07:56:29.808 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7336d6bdd1] 2024-05-27T07:56:29.808 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561dd9fd423d] 2024-05-27T07:56:29.808 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 4: (SnapRealm::build_snap_set() const+0x323) [0x561dd9fbf971] 2024-05-27T07:56:29.808 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 5: (SnapRealm::check_cache() const+0x13e) [0x561dd9fbdfa0] 2024-05-27T07:56:29.808 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561dd9fbe651] 2024-05-27T07:56:29.808 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561dd9f53347] 2024-05-27T07:56:29.808 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561dd9c4f87e] 2024-05-27T07:56:29.809 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561dd9c54f2f] 2024-05-27T07:56:29.809 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561dd9c5f969] 2024-05-27T07:56:29.809 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 11: (MDLog::_replay_thread()+0x1f38) [0x561dd9ff848c] 2024-05-27T07:56:29.809 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 12: (MDLog::ReplayThread::entry()+0x11) [0x561dda0019cb] 2024-05-27T07:56:29.809 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 13: (Thread::entry_wrapper()+0x43) [0x7f7336d3a72d] 2024-05-27T07:56:29.809 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7336d3a749] 2024-05-27T07:56:29.809 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f73358de1ca] 2024-05-27T07:56:29.809 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 16: clone() 2024-05-27T07:56:29.809 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.530+0000 7f7324526700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7324526700 time 2024-05-27T07:56:29.527873+0000 2024-05-27T07:56:29.810 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:56:29.810 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.810 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:29.810 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7336d6bba6] 2024-05-27T07:56:29.810 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7336d6bdd1] 2024-05-27T07:56:29.810 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561dd9fd423d] 2024-05-27T07:56:29.810 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 4: (SnapRealm::build_snap_set() const+0x323) [0x561dd9fbf971] 2024-05-27T07:56:29.811 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 5: (SnapRealm::check_cache() const+0x13e) [0x561dd9fbdfa0] 2024-05-27T07:56:29.811 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561dd9fbe651] 2024-05-27T07:56:29.811 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561dd9f53347] 2024-05-27T07:56:29.811 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561dd9c4f87e] 2024-05-27T07:56:29.811 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561dd9c54f2f] 2024-05-27T07:56:29.811 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561dd9c5f969] 2024-05-27T07:56:29.811 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 11: (MDLog::_replay_thread()+0x1f38) [0x561dd9ff848c] 2024-05-27T07:56:29.811 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 12: (MDLog::ReplayThread::entry()+0x11) [0x561dda0019cb] 2024-05-27T07:56:29.811 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 13: (Thread::entry_wrapper()+0x43) [0x7f7336d3a72d] 2024-05-27T07:56:29.811 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7336d3a749] 2024-05-27T07:56:29.811 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f73358de1ca] 2024-05-27T07:56:29.811 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 16: clone() 2024-05-27T07:56:29.811 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: *** Caught signal (Aborted) ** 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: in thread 7f7324526700 thread_name:md_log_replay 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f73358e8d20] 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2: gsignal() 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 3: abort() 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7336d6bced] 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7336d6bdd1] 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561dd9fd423d] 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 7: (SnapRealm::build_snap_set() const+0x323) [0x561dd9fbf971] 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 8: (SnapRealm::check_cache() const+0x13e) [0x561dd9fbdfa0] 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561dd9fbe651] 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561dd9f53347] 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561dd9c4f87e] 2024-05-27T07:56:29.812 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561dd9c54f2f] 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561dd9c5f969] 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 14: (MDLog::_replay_thread()+0x1f38) [0x561dd9ff848c] 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 15: (MDLog::ReplayThread::entry()+0x11) [0x561dda0019cb] 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 16: (Thread::entry_wrapper()+0x43) [0x7f7336d3a72d] 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7336d3a749] 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f73358de1ca] 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 19: clone() 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.533+0000 7f7324526700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: in thread 7f7324526700 thread_name:md_log_replay 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f73358e8d20] 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2: gsignal() 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 3: abort() 2024-05-27T07:56:29.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7336d6bced] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7336d6bdd1] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561dd9fd423d] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 7: (SnapRealm::build_snap_set() const+0x323) [0x561dd9fbf971] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 8: (SnapRealm::check_cache() const+0x13e) [0x561dd9fbdfa0] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561dd9fbe651] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561dd9f53347] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561dd9c4f87e] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561dd9c54f2f] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561dd9c5f969] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 14: (MDLog::_replay_thread()+0x1f38) [0x561dd9ff848c] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 15: (MDLog::ReplayThread::entry()+0x11) [0x561dda0019cb] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 16: (Thread::entry_wrapper()+0x43) [0x7f7336d3a72d] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7336d3a749] 2024-05-27T07:56:29.814 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f73358de1ca] 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 19: clone() 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: -1> 2024-05-27T07:56:29.530+0000 7f7324526700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7324526700 time 2024-05-27T07:56:29.527873+0000 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7336d6bba6] 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7336d6bdd1] 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561dd9fd423d] 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 4: (SnapRealm::build_snap_set() const+0x323) [0x561dd9fbf971] 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 5: (SnapRealm::check_cache() const+0x13e) [0x561dd9fbdfa0] 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561dd9fbe651] 2024-05-27T07:56:29.815 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561dd9f53347] 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561dd9c4f87e] 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561dd9c54f2f] 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561dd9c5f969] 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 11: (MDLog::_replay_thread()+0x1f38) [0x561dd9ff848c] 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 12: (MDLog::ReplayThread::entry()+0x11) [0x561dda0019cb] 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 13: (Thread::entry_wrapper()+0x43) [0x7f7336d3a72d] 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7336d3a749] 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f73358de1ca] 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 16: clone() 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 0> 2024-05-27T07:56:29.533+0000 7f7324526700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: in thread 7f7324526700 thread_name:md_log_replay 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.816 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f73358e8d20] 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2: gsignal() 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 3: abort() 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7336d6bced] 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7336d6bdd1] 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561dd9fd423d] 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 7: (SnapRealm::build_snap_set() const+0x323) [0x561dd9fbf971] 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 8: (SnapRealm::check_cache() const+0x13e) [0x561dd9fbdfa0] 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561dd9fbe651] 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561dd9f53347] 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561dd9c4f87e] 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561dd9c54f2f] 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561dd9c5f969] 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 14: (MDLog::_replay_thread()+0x1f38) [0x561dd9ff848c] 2024-05-27T07:56:29.817 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 15: (MDLog::ReplayThread::entry()+0x11) [0x561dda0019cb] 2024-05-27T07:56:29.818 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 16: (Thread::entry_wrapper()+0x43) [0x7f7336d3a72d] 2024-05-27T07:56:29.818 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7336d3a749] 2024-05-27T07:56:29.818 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f73358de1ca] 2024-05-27T07:56:29.818 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 19: clone() 2024-05-27T07:56:29.818 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:56:29.818 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.818 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: -6> 2024-05-27T07:56:29.530+0000 7f7324526700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7324526700 time 2024-05-27T07:56:29.527873+0000 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7336d6bba6] 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7336d6bdd1] 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561dd9fd423d] 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 4: (SnapRealm::build_snap_set() const+0x323) [0x561dd9fbf971] 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 5: (SnapRealm::check_cache() const+0x13e) [0x561dd9fbdfa0] 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561dd9fbe651] 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561dd9f53347] 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561dd9c4f87e] 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561dd9c54f2f] 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561dd9c5f969] 2024-05-27T07:56:29.819 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 11: (MDLog::_replay_thread()+0x1f38) [0x561dd9ff848c] 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 12: (MDLog::ReplayThread::entry()+0x11) [0x561dda0019cb] 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 13: (Thread::entry_wrapper()+0x43) [0x7f7336d3a72d] 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7336d3a749] 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f73358de1ca] 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 16: clone() 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: -5> 2024-05-27T07:56:29.533+0000 7f7324526700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: in thread 7f7324526700 thread_name:md_log_replay 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f73358e8d20] 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2: gsignal() 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 3: abort() 2024-05-27T07:56:29.820 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7336d6bced] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7336d6bdd1] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561dd9fd423d] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 7: (SnapRealm::build_snap_set() const+0x323) [0x561dd9fbf971] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 8: (SnapRealm::check_cache() const+0x13e) [0x561dd9fbdfa0] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561dd9fbe651] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561dd9f53347] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561dd9c4f87e] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x561dd9c54f2f] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561dd9c5f969] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 14: (MDLog::_replay_thread()+0x1f38) [0x561dd9ff848c] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 15: (MDLog::ReplayThread::entry()+0x11) [0x561dda0019cb] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 16: (Thread::entry_wrapper()+0x43) [0x7f7336d3a72d] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7336d3a749] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f73358de1ca] 2024-05-27T07:56:29.821 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 19: clone() 2024-05-27T07:56:29.822 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:56:29.822 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[71409]: 2024-05-27T07:56:29.822 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:29 smithi114 podman[127700]: 2024-05-27 07:56:29.606178246 +0000 UTC m=+0.023756084 container died cb013cfd6fb63a78fc49d747c6ae4d802a84bc79139bcf89f86a6b9920e00c11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-27T07:56:30.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:29 smithi163 ceph-mon[36451]: pgmap v573: 129 pgs: 129 active+clean; 313 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 650 op/s 2024-05-27T07:56:30.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:29 smithi163 ceph-mon[36451]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:56:30.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:29 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:56:30.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:29 smithi186 ceph-mon[36647]: pgmap v573: 129 pgs: 129 active+clean; 313 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 650 op/s 2024-05-27T07:56:30.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:29 smithi186 ceph-mon[36647]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:56:30.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:29 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:56:30.603 INFO:tasks.workunit.client.0.smithi114.stdout: 10 39115 11.30 MB/sec execute 42 sec latency 183.190 ms 2024-05-27T07:56:31.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 39429 11.40 MB/sec execute 43 sec latency 171.791 ms 2024-05-27T07:56:32.028 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:31 smithi163 ceph-mon[36451]: pgmap v574: 129 pgs: 129 active+clean; 318 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 599 op/s 2024-05-27T07:56:32.031 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:31 smithi186 ceph-mon[36647]: pgmap v574: 129 pgs: 129 active+clean; 318 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 599 op/s 2024-05-27T07:56:32.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:31 smithi114 ceph-mon[32038]: pgmap v574: 129 pgs: 129 active+clean; 318 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 599 op/s 2024-05-27T07:56:32.493 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:32 smithi114 podman[127700]: 2024-05-27 07:56:32.236896146 +0000 UTC m=+2.654473968 container remove cb013cfd6fb63a78fc49d747c6ae4d802a84bc79139bcf89f86a6b9920e00c11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:56:32.493 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:32 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:56:32.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 39720 11.34 MB/sec execute 44 sec latency 309.284 ms 2024-05-27T07:56:32.807 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:32 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:56:33.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 40080 11.56 MB/sec execute 45 sec latency 178.508 ms 2024-05-27T07:56:34.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:33 smithi114 ceph-mon[32038]: pgmap v575: 129 pgs: 129 active+clean; 327 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.3 MiB/s wr, 621 op/s 2024-05-27T07:56:34.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:33 smithi163 ceph-mon[36451]: pgmap v575: 129 pgs: 129 active+clean; 327 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.3 MiB/s wr, 621 op/s 2024-05-27T07:56:34.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:33 smithi186 ceph-mon[36647]: pgmap v575: 129 pgs: 129 active+clean; 327 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.3 MiB/s wr, 621 op/s 2024-05-27T07:56:34.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 40395 11.75 MB/sec execute 46 sec latency 207.235 ms 2024-05-27T07:56:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:34.998+0000 7ff356738640 1 -- 172.21.15.114:0/35983087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff350112900 msgr2=0x7ff350118eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:34.998+0000 7ff356738640 1 --2- 172.21.15.114:0/35983087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff350112900 0x7ff350118eb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff340009ca0 tx=0x7ff34002a970 comp rx=0 tx=0).stop 2024-05-27T07:56:35.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:34.999+0000 7ff356738640 1 -- 172.21.15.114:0/35983087 shutdown_connections 2024-05-27T07:56:35.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:34.999+0000 7ff356738640 1 --2- 172.21.15.114:0/35983087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff350112900 0x7ff350118eb0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:34.999+0000 7ff356738640 1 --2- 172.21.15.114:0/35983087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff350111de0 0x7ff350112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:34.999+0000 7ff356738640 1 --2- 172.21.15.114:0/35983087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff35010a490 0x7ff35010a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:34.999+0000 7ff356738640 1 -- 172.21.15.114:0/35983087 >> 172.21.15.114:0/35983087 conn(0x7ff35006e8f0 msgr2=0x7ff35006ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.007+0000 7ff356738640 1 -- 172.21.15.114:0/35983087 shutdown_connections 2024-05-27T07:56:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.011+0000 7ff356738640 1 -- 172.21.15.114:0/35983087 wait complete. 2024-05-27T07:56:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.011+0000 7ff356738640 1 Processor -- start 2024-05-27T07:56:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.014+0000 7ff356738640 1 -- start start 2024-05-27T07:56:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.015+0000 7ff356738640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff35010a490 0x7ff350121e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.015+0000 7ff356738640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff350111de0 0x7ff35011bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.015+0000 7ff356738640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff350112900 0x7ff35011c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.015+0000 7ff356738640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff35011a450 con 0x7ff350111de0 2024-05-27T07:56:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.015+0000 7ff356738640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff35011a2d0 con 0x7ff35010a490 2024-05-27T07:56:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.015+0000 7ff356738640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff35011a5d0 con 0x7ff350112900 2024-05-27T07:56:35.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.016+0000 7ff354cd5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff350112900 0x7ff35011c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:35.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.019+0000 7ff354cd5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff350112900 0x7ff35011c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35042/0 (socket says 172.21.15.114:35042) 2024-05-27T07:56:35.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.019+0000 7ff354cd5640 1 -- 172.21.15.114:0/3200080696 learned_addr learned my addr 172.21.15.114:0/3200080696 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:35.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.020+0000 7ff34f7fe640 1 --2- 172.21.15.114:0/3200080696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff350111de0 0x7ff35011bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.023+0000 7ff354cd5640 1 -- 172.21.15.114:0/3200080696 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff35010a490 msgr2=0x7ff350121e90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:56:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.023+0000 7ff354cd5640 1 --2- 172.21.15.114:0/3200080696 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff35010a490 0x7ff350121e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.023+0000 7ff354cd5640 1 -- 172.21.15.114:0/3200080696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff350111de0 msgr2=0x7ff35011bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.023+0000 7ff354cd5640 1 --2- 172.21.15.114:0/3200080696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff350111de0 0x7ff35011bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.023+0000 7ff354cd5640 1 -- 172.21.15.114:0/3200080696 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff35011cc80 con 0x7ff350112900 2024-05-27T07:56:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.027+0000 7ff354cd5640 1 --2- 172.21.15.114:0/3200080696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff350112900 0x7ff35011c4e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff340009bf0 tx=0x7ff34002f9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.028+0000 7ff34d7fa640 1 -- 172.21.15.114:0/3200080696 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff340034050 con 0x7ff350112900 2024-05-27T07:56:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.028+0000 7ff34d7fa640 1 -- 172.21.15.114:0/3200080696 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3400398d0 con 0x7ff350112900 2024-05-27T07:56:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.028+0000 7ff34d7fa640 1 -- 172.21.15.114:0/3200080696 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff340042740 con 0x7ff350112900 2024-05-27T07:56:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.029+0000 7ff356738640 1 -- 172.21.15.114:0/3200080696 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3501be340 con 0x7ff350112900 2024-05-27T07:56:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.029+0000 7ff356738640 1 -- 172.21.15.114:0/3200080696 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3501be750 con 0x7ff350112900 2024-05-27T07:56:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.034+0000 7ff35d5ed740 1 Processor -- start 2024-05-27T07:56:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.037+0000 7ff34d7fa640 1 -- 172.21.15.114:0/3200080696 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ff340037020 con 0x7ff350112900 2024-05-27T07:56:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.038+0000 7ff34d7fa640 1 --2- 172.21.15.114:0/3200080696 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff33c0b2b60 0x7ff33c0b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.038+0000 7ff35d5ed740 1 -- start start 2024-05-27T07:56:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.038+0000 7ff34d7fa640 1 -- 172.21.15.114:0/3200080696 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 9554+0+0 (secure 0 0 0) 0x7ff340103950 con 0x7ff350112900 2024-05-27T07:56:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.038+0000 7ff35d5ed740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff531b0 0x55b4702f43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.038+0000 7ff35d5ed740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 0x55b4702f4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.038+0000 7ff35d5ed740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff324d0 0x55b4702f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.038+0000 7ff35d5ed740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b470266c30 con 0x55b46ff531b0 2024-05-27T07:56:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.038+0000 7ff35d5ed740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b4702567d0 con 0x55b46ff324d0 2024-05-27T07:56:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.038+0000 7ff35d5ed740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b47025a810 con 0x55b46ff2a2f0 2024-05-27T07:56:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.042+0000 7ff34f7fe640 1 --2- 172.21.15.114:0/3200080696 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff33c0b2b60 0x7ff33c0b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.045+0000 7ff34f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 0x55b4702f4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.045+0000 7ff34f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 0x55b4702f4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35048/0 (socket says 172.21.15.114:35048) 2024-05-27T07:56:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.045+0000 7ff34f7fe640 1 -- 172.21.15.114:0/3785837958 learned_addr learned my addr 172.21.15.114:0/3785837958 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.045+0000 7ff354cd5640 1 --2- 172.21.15.114:0/3785837958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff324d0 0x55b4702f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.049+0000 7ff34f7fe640 1 --2- 172.21.15.114:0/3200080696 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff33c0b2b60 0x7ff33c0b5070 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff338005fd0 tx=0x7ff33800c2d0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.049+0000 7ff34f7fe640 1 -- 172.21.15.114:0/3785837958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff324d0 msgr2=0x55b4702f4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.049+0000 7ff34f7fe640 1 --2- 172.21.15.114:0/3785837958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff324d0 0x55b4702f4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.049+0000 7ff34f7fe640 1 -- 172.21.15.114:0/3785837958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff531b0 msgr2=0x55b4702f43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.049+0000 7ff34f7fe640 1 --2- 172.21.15.114:0/3785837958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff531b0 0x55b4702f43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.049+0000 7ff34f7fe640 1 -- 172.21.15.114:0/3785837958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b46ff6b210 con 0x55b46ff2a2f0 2024-05-27T07:56:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.052+0000 7ff34f7fe640 1 --2- 172.21.15.114:0/3785837958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 0x55b4702f4910 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff33800ce90 tx=0x7ff33801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94137f0bc7a840e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.057+0000 7ff3467fc640 1 -- 172.21.15.114:0/3785837958 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338012e60 con 0x55b46ff2a2f0 2024-05-27T07:56:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.057+0000 7ff3467fc640 1 -- 172.21.15.114:0/3785837958 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff338020070 con 0x55b46ff2a2f0 2024-05-27T07:56:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.057+0000 7ff3467fc640 1 -- 172.21.15.114:0/3785837958 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338024420 con 0x55b46ff2a2f0 2024-05-27T07:56:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.059+0000 7ff35d5ed740 1 -- 172.21.15.114:0/3785837958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 msgr2=0x55b4702f4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.059+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/3785837958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 0x55b4702f4910 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff33800ce90 tx=0x7ff33801a000 comp rx=0 tx=0).stop 2024-05-27T07:56:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.060+0000 7ff35d5ed740 1 -- 172.21.15.114:0/3785837958 shutdown_connections 2024-05-27T07:56:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.060+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/3785837958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff324d0 0x55b4702f4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.060+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/3785837958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 0x55b4702f4910 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.060+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/3785837958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff531b0 0x55b4702f43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.060+0000 7ff35d5ed740 1 -- 172.21.15.114:0/3785837958 >> 172.21.15.114:0/3785837958 conn(0x55b46ff47300 msgr2=0x55b4702f0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.066+0000 7ff35d5ed740 1 -- 172.21.15.114:0/3785837958 shutdown_connections 2024-05-27T07:56:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.068+0000 7ff35d5ed740 1 -- 172.21.15.114:0/3785837958 wait complete. 2024-05-27T07:56:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.068+0000 7ff35d5ed740 1 -- start start 2024-05-27T07:56:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.069+0000 7ff35d5ed740 1 Processor -- start 2024-05-27T07:56:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.073+0000 7ff35d5ed740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff2a2f0 0x55b470300480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.073+0000 7ff35d5ed740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff324d0 0x55b4703009c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.073+0000 7ff35d5ed740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff531b0 0x55b4702fe990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.073+0000 7ff35d5ed740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b46ff3c790 con 0x55b46ff531b0 2024-05-27T07:56:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.073+0000 7ff35d5ed740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b470212890 con 0x55b46ff2a2f0 2024-05-27T07:56:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.073+0000 7ff35d5ed740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b470301500 con 0x55b46ff324d0 2024-05-27T07:56:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.073+0000 7ff35d5ed740 10 client.0 _ll_register_callbacks cb 0x55b46ff6aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.074+0000 7ff34f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff324d0 0x55b4703009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.074+0000 7ff34f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff324d0 0x55b4703009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35064/0 (socket says 172.21.15.114:35064) 2024-05-27T07:56:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.074+0000 7ff34f7fe640 1 -- 172.21.15.114:0/3647923688 learned_addr learned my addr 172.21.15.114:0/3647923688 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.075+0000 7ff34f7fe640 1 -- 172.21.15.114:0/3647923688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff2a2f0 msgr2=0x55b470300480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:56:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.075+0000 7ff34f7fe640 1 --2- 172.21.15.114:0/3647923688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff2a2f0 0x55b470300480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.075+0000 7ff34f7fe640 1 -- 172.21.15.114:0/3647923688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff531b0 msgr2=0x55b4702fe990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:56:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.075+0000 7ff34f7fe640 1 --2- 172.21.15.114:0/3647923688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff531b0 0x55b4702fe990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.075+0000 7ff34f7fe640 1 -- 172.21.15.114:0/3647923688 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b470037620 con 0x55b46ff324d0 2024-05-27T07:56:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.076+0000 7ff34f7fe640 1 --2- 172.21.15.114:0/3647923688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff324d0 0x55b4703009c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff338023000 tx=0x7ff338005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9637a8e56786a878 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.076+0000 7ff32f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:56:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.076+0000 7ff32f7fe640 1 -- 172.21.15.114:0/3647923688 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338012e60 con 0x55b46ff324d0 2024-05-27T07:56:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.076+0000 7ff35d5ed740 1 -- 172.21.15.114:0/3647923688 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff338020070 con 0x55b46ff324d0 2024-05-27T07:56:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.076+0000 7ff32f7fe640 1 -- 172.21.15.114:0/3647923688 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9554+0+0 (secure 0 0 0) 0x7ff3380040c0 con 0x55b46ff324d0 2024-05-27T07:56:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.076+0000 7ff32f7fe640 1 -- 172.21.15.114:0/3647923688 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7ff338038050 con 0x55b46ff324d0 2024-05-27T07:56:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.080+0000 7ff35d5ed740 10 client.28795 fetch_fsmap learned FSMap version 94 2024-05-27T07:56:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.080+0000 7ff35d5ed740 1 -- 172.21.15.114:0/3647923688 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=4+}) -- 0x55b47030d930 con 0x55b46ff324d0 2024-05-27T07:56:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.083+0000 7ff32f7fe640 1 -- 172.21.15.114:0/3647923688 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff338020070 con 0x55b46ff324d0 2024-05-27T07:56:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.083+0000 7ff32f7fe640 1 -- 172.21.15.114:0/3647923688 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 94) ==== 4095+0+0 (secure 0 0 0) 0x7ff33802db80 con 0x55b46ff324d0 2024-05-27T07:56:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.084+0000 7ff35d5ed740 10 client.28795 fetch_fsmap finished waiting for FSMap version 94 2024-05-27T07:56:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.084+0000 7ff35d5ed740 10 client.28795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.084+0000 7ff35d5ed740 20 client.28795 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:35.086 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.084+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/3647923688 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b46ff11d80 0x55b470311280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.086 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.084+0000 7ff35d5ed740 4 client.28795 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:56:35.086 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.084+0000 7ff35d5ed740 1 -- 172.21.15.114:0/3647923688 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b470268b80 con 0x55b46ff11d80 2024-05-27T07:56:35.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.087+0000 7ff34ffff640 1 --2- 172.21.15.114:0/3647923688 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b46ff11d80 0x55b470311280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.092+0000 7ff34ffff640 1 --2- 172.21.15.114:0/3647923688 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b46ff11d80 0x55b470311280 crc :-1 s=READY pgs=458 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5de1ffe1c97f758 server_cookie=a6f29b32678e6c89 in_seq=0 out_seq=0 2024-05-27T07:56:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.094+0000 7ff32f7fe640 10 client.28795 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.097+0000 7ff32f7fe640 1 -- 172.21.15.114:0/3647923688 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b470268b80 con 0x55b46ff11d80 2024-05-27T07:56:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.097+0000 7ff32f7fe640 10 client.28795 handle_command_reply: tid=0 2024-05-27T07:56:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.097+0000 7ff35d5ed740 1 client.28795 shutdown 2024-05-27T07:56:35.102 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.101+0000 7ff35d5ed740 1 -- 172.21.15.114:0/3647923688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff324d0 msgr2=0x55b4703009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:35.102 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.101+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/3647923688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff324d0 0x55b4703009c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff338023000 tx=0x7ff338005850 comp rx=0 tx=0).stop 2024-05-27T07:56:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.103+0000 7ff35d5ed740 1 -- 172.21.15.114:0/3647923688 shutdown_connections 2024-05-27T07:56:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.103+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/3647923688 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b46ff11d80 0x55b470311280 crc :-1 s=READY pgs=458 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.103+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/3647923688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff531b0 0x55b4702fe990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.103+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/3647923688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff324d0 0x55b4703009c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.103+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/3647923688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff2a2f0 0x55b470300480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.103+0000 7ff35d5ed740 1 -- 172.21.15.114:0/3647923688 >> 172.21.15.114:0/3647923688 conn(0x55b46ff47300 msgr2=0x55b4702f60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.103+0000 7ff32f7fe640 0 client.28795 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.109+0000 7ff35d5ed740 1 -- 172.21.15.114:0/3647923688 shutdown_connections 2024-05-27T07:56:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.111+0000 7ff35d5ed740 1 -- 172.21.15.114:0/3647923688 wait complete. 2024-05-27T07:56:35.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.115+0000 7ff35d5ed740 20 client.28795 trim_cache size 0 max 16384 2024-05-27T07:56:35.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.151+0000 7ff35d5ed740 1 Processor -- start 2024-05-27T07:56:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.155+0000 7ff35d5ed740 1 -- start start 2024-05-27T07:56:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.155+0000 7ff35d5ed740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 0x55b4702fd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.155+0000 7ff35d5ed740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff324d0 0x55b4702fd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.155+0000 7ff34ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 0x55b4702fd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.155+0000 7ff34f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff324d0 0x55b4702fd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.155+0000 7ff34ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 0x55b4702fd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35078/0 (socket says 172.21.15.114:35078) 2024-05-27T07:56:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.156+0000 7ff34ffff640 1 -- 172.21.15.114:0/609958427 learned_addr learned my addr 172.21.15.114:0/609958427 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.156+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/609958427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff531b0 0x55b470316a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.156+0000 7ff35d5ed740 1 -- 172.21.15.114:0/609958427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff33000fcc0 con 0x55b46ff324d0 2024-05-27T07:56:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.156+0000 7ff35d5ed740 1 -- 172.21.15.114:0/609958427 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff330010390 con 0x55b46ff531b0 2024-05-27T07:56:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.156+0000 7ff35d5ed740 1 -- 172.21.15.114:0/609958427 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff33000ec60 con 0x55b46ff2a2f0 2024-05-27T07:56:35.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.161+0000 7ff354cd5640 1 --2- 172.21.15.114:0/609958427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff531b0 0x55b470316a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:35.164 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.163+0000 7ff34ffff640 1 -- 172.21.15.114:0/609958427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff531b0 msgr2=0x55b470316a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:35.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.163+0000 7ff34ffff640 1 --2- 172.21.15.114:0/609958427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff531b0 0x55b470316a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.163+0000 7ff34ffff640 1 -- 172.21.15.114:0/609958427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff324d0 msgr2=0x55b4702fd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:35.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.163+0000 7ff34ffff640 1 --2- 172.21.15.114:0/609958427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff324d0 0x55b4702fd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.163+0000 7ff34ffff640 1 -- 172.21.15.114:0/609958427 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4702f1c60 con 0x55b46ff2a2f0 2024-05-27T07:56:35.165 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.163+0000 7ff34f7fe640 1 --2- 172.21.15.114:0/609958427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff324d0 0x55b4702fd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:56:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.170+0000 7ff34ffff640 1 --2- 172.21.15.114:0/609958427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 0x55b4702fd260 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff3480094d0 tx=0x7ff348009dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5135e4098d462125 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.172+0000 7ff345ffb640 1 -- 172.21.15.114:0/609958427 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3480127f0 con 0x55b46ff2a2f0 2024-05-27T07:56:35.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.174+0000 7ff345ffb640 1 -- 172.21.15.114:0/609958427 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff34801a070 con 0x55b46ff2a2f0 2024-05-27T07:56:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.174+0000 7ff345ffb640 1 -- 172.21.15.114:0/609958427 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348015c70 con 0x55b46ff2a2f0 2024-05-27T07:56:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.177+0000 7ff35d5ed740 1 -- 172.21.15.114:0/609958427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 msgr2=0x55b4702fd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.177+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/609958427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 0x55b4702fd260 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff3480094d0 tx=0x7ff348009dc0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.183+0000 7ff35d5ed740 1 -- 172.21.15.114:0/609958427 shutdown_connections 2024-05-27T07:56:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.183+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/609958427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff531b0 0x55b470316a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.183+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/609958427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff324d0 0x55b4702fd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.183+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/609958427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff2a2f0 0x55b4702fd260 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.183+0000 7ff35d5ed740 1 -- 172.21.15.114:0/609958427 >> 172.21.15.114:0/609958427 conn(0x55b46ff47300 msgr2=0x55b4702fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:35.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.186+0000 7ff35d5ed740 1 -- 172.21.15.114:0/609958427 shutdown_connections 2024-05-27T07:56:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.187+0000 7ff35d5ed740 1 -- 172.21.15.114:0/609958427 wait complete. 2024-05-27T07:56:35.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.188+0000 7ff35d5ed740 1 -- start start 2024-05-27T07:56:35.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.188+0000 7ff35d5ed740 1 Processor -- start 2024-05-27T07:56:35.191 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.190+0000 7ff35d5ed740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff2a2f0 0x55b470303720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.190+0000 7ff35d5ed740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff324d0 0x55b470303c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.190+0000 7ff35d5ed740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff531b0 0x55b4703041a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.190+0000 7ff35d5ed740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff33000e590 con 0x55b46ff2a2f0 2024-05-27T07:56:35.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.190+0000 7ff35d5ed740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff33000df40 con 0x55b46ff531b0 2024-05-27T07:56:35.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.190+0000 7ff35d5ed740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b4702fe780 con 0x55b46ff324d0 2024-05-27T07:56:35.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.190+0000 7ff35d5ed740 10 client.0 _ll_register_callbacks cb 0x7ff3400c66a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:35.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.191+0000 7ff34ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff2a2f0 0x55b470303720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:35.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.191+0000 7ff34ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff2a2f0 0x55b470303720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39462/0 (socket says 172.21.15.114:39462) 2024-05-27T07:56:35.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.191+0000 7ff34ffff640 1 -- 172.21.15.114:0/2534967318 learned_addr learned my addr 172.21.15.114:0/2534967318 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:35.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.192+0000 7ff34ffff640 1 -- 172.21.15.114:0/2534967318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff324d0 msgr2=0x55b470303c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:35.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.192+0000 7ff34ffff640 1 --2- 172.21.15.114:0/2534967318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff324d0 0x55b470303c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.192+0000 7ff34ffff640 1 -- 172.21.15.114:0/2534967318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff531b0 msgr2=0x55b4703041a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:56:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.192+0000 7ff34ffff640 1 --2- 172.21.15.114:0/2534967318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff531b0 0x55b4703041a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.192+0000 7ff34ffff640 1 -- 172.21.15.114:0/2534967318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b470377a40 con 0x55b46ff2a2f0 2024-05-27T07:56:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.192+0000 7ff34ffff640 1 --2- 172.21.15.114:0/2534967318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff2a2f0 0x55b470303720 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7ff3480046d0 tx=0x7ff348015670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8259172fc4a0537a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.192+0000 7ff32f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:56:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.193+0000 7ff32f7fe640 1 -- 172.21.15.114:0/2534967318 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34802b020 con 0x55b46ff2a2f0 2024-05-27T07:56:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.193+0000 7ff32f7fe640 1 -- 172.21.15.114:0/2534967318 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9554+0+0 (secure 0 0 0) 0x7ff348015d00 con 0x55b46ff2a2f0 2024-05-27T07:56:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.193+0000 7ff35d5ed740 1 -- 172.21.15.114:0/2534967318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff34801a070 con 0x55b46ff2a2f0 2024-05-27T07:56:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.193+0000 7ff32f7fe640 1 -- 172.21.15.114:0/2534967318 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7ff3480059c0 con 0x55b46ff2a2f0 2024-05-27T07:56:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.195+0000 7ff35d5ed740 10 client.19311 fetch_fsmap learned FSMap version 94 2024-05-27T07:56:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.195+0000 7ff35d5ed740 1 -- 172.21.15.114:0/2534967318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=4+}) -- 0x55b470378c90 con 0x55b46ff2a2f0 2024-05-27T07:56:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.198+0000 7ff32f7fe640 1 -- 172.21.15.114:0/2534967318 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff34801a070 con 0x55b46ff2a2f0 2024-05-27T07:56:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.198+0000 7ff32f7fe640 1 -- 172.21.15.114:0/2534967318 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 94) ==== 4095+0+0 (secure 0 0 0) 0x7ff348006260 con 0x55b46ff2a2f0 2024-05-27T07:56:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.199+0000 7ff35d5ed740 10 client.19311 fetch_fsmap finished waiting for FSMap version 94 2024-05-27T07:56:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.199+0000 7ff35d5ed740 10 client.19311 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.199+0000 7ff35d5ed740 20 client.19311 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.199+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/2534967318 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b46fee71c0 0x55b47037b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.199+0000 7ff35d5ed740 4 client.19311 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:56:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.199+0000 7ff35d5ed740 1 -- 172.21.15.114:0/2534967318 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b470262360 con 0x55b46fee71c0 2024-05-27T07:56:35.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.203+0000 7ff34f7fe640 1 --2- 172.21.15.114:0/2534967318 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b46fee71c0 0x55b47037b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.217+0000 7ff34f7fe640 1 --2- 172.21.15.114:0/2534967318 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b46fee71c0 0x55b47037b420 crc :-1 s=READY pgs=459 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93ac985401d9674d server_cookie=a60933e234385cd4 in_seq=0 out_seq=0 2024-05-27T07:56:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:35.217+0000 7ff32f7fe640 10 client.19311 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:35.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 40675 11.81 MB/sec execute 47 sec latency 261.547 ms 2024-05-27T07:56:36.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:35 smithi114 ceph-mon[32038]: pgmap v576: 129 pgs: 129 active+clean; 331 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.9 MiB/s wr, 539 op/s 2024-05-27T07:56:36.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:35 smithi163 ceph-mon[36451]: pgmap v576: 129 pgs: 129 active+clean; 331 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.9 MiB/s wr, 539 op/s 2024-05-27T07:56:36.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:35 smithi186 ceph-mon[36647]: pgmap v576: 129 pgs: 129 active+clean; 331 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.9 MiB/s wr, 539 op/s 2024-05-27T07:56:36.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 40782 11.64 MB/sec execute 48 sec latency 651.685 ms 2024-05-27T07:56:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.607+0000 7ff32f7fe640 1 -- 172.21.15.114:0/2534967318 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b470262360 con 0x55b46fee71c0 2024-05-27T07:56:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.607+0000 7ff32f7fe640 10 client.19311 handle_command_reply: tid=0 2024-05-27T07:56:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.608+0000 7ff35d5ed740 1 client.19311 shutdown 2024-05-27T07:56:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.608+0000 7ff35d5ed740 1 -- 172.21.15.114:0/2534967318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff2a2f0 msgr2=0x55b470303720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.608+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/2534967318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff2a2f0 0x55b470303720 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7ff3480046d0 tx=0x7ff348015670 comp rx=0 tx=0).stop 2024-05-27T07:56:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.608+0000 7ff35d5ed740 1 -- 172.21.15.114:0/2534967318 shutdown_connections 2024-05-27T07:56:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.608+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/2534967318 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b46fee71c0 0x55b47037b420 crc :-1 s=READY pgs=459 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.608+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/2534967318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b46ff531b0 0x55b4703041a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.608+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/2534967318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b46ff324d0 0x55b470303c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.608+0000 7ff35d5ed740 1 --2- 172.21.15.114:0/2534967318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b46ff2a2f0 0x55b470303720 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.608+0000 7ff35d5ed740 1 -- 172.21.15.114:0/2534967318 >> 172.21.15.114:0/2534967318 conn(0x55b46ff47300 msgr2=0x55b4702fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.608+0000 7ff32f7fe640 0 client.19311 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.608+0000 7ff35d5ed740 1 -- 172.21.15.114:0/2534967318 shutdown_connections 2024-05-27T07:56:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.608+0000 7ff35d5ed740 1 -- 172.21.15.114:0/2534967318 wait complete. 2024-05-27T07:56:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.608+0000 7ff35d5ed740 20 client.19311 trim_cache size 0 max 16384 2024-05-27T07:56:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:56:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:56:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:56:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.609+0000 7ff356738640 1 -- 172.21.15.114:0/3200080696 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff33c0b2b60 msgr2=0x7ff33c0b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.609+0000 7ff356738640 1 --2- 172.21.15.114:0/3200080696 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff33c0b2b60 0x7ff33c0b5070 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff338005fd0 tx=0x7ff33800c2d0 comp rx=0 tx=0).stop 2024-05-27T07:56:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.609+0000 7ff356738640 1 -- 172.21.15.114:0/3200080696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff350112900 msgr2=0x7ff35011c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.609+0000 7ff356738640 1 --2- 172.21.15.114:0/3200080696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff350112900 0x7ff35011c4e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff340009bf0 tx=0x7ff34002f9d0 comp rx=0 tx=0).stop 2024-05-27T07:56:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.609+0000 7ff356738640 1 -- 172.21.15.114:0/3200080696 shutdown_connections 2024-05-27T07:56:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.609+0000 7ff356738640 1 --2- 172.21.15.114:0/3200080696 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff33c0b2b60 0x7ff33c0b5070 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.609+0000 7ff356738640 1 --2- 172.21.15.114:0/3200080696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff350112900 0x7ff35011c4e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.609+0000 7ff356738640 1 --2- 172.21.15.114:0/3200080696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff350111de0 0x7ff35011bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.609+0000 7ff356738640 1 --2- 172.21.15.114:0/3200080696 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff35010a490 0x7ff350121e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.609+0000 7ff356738640 1 -- 172.21.15.114:0/3200080696 >> 172.21.15.114:0/3200080696 conn(0x7ff35006e8f0 msgr2=0x7ff350073860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.609+0000 7ff356738640 1 -- 172.21.15.114:0/3200080696 shutdown_connections 2024-05-27T07:56:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:36.609+0000 7ff356738640 1 -- 172.21.15.114:0/3200080696 wait complete. 2024-05-27T07:56:37.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 41225 11.88 MB/sec execute 49 sec latency 156.975 ms 2024-05-27T07:56:38.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:37 smithi114 ceph-mon[32038]: pgmap v577: 129 pgs: 129 active+clean; 332 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.5 MiB/s wr, 513 op/s 2024-05-27T07:56:38.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:37 smithi163 ceph-mon[36451]: pgmap v577: 129 pgs: 129 active+clean; 332 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.5 MiB/s wr, 513 op/s 2024-05-27T07:56:38.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:37 smithi186 ceph-mon[36647]: pgmap v577: 129 pgs: 129 active+clean; 332 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.5 MiB/s wr, 513 op/s 2024-05-27T07:56:38.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 41601 12.10 MB/sec execute 50 sec latency 189.249 ms 2024-05-27T07:56:39.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 41960 12.31 MB/sec execute 51 sec latency 181.034 ms 2024-05-27T07:56:40.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:39 smithi114 ceph-mon[32038]: pgmap v578: 129 pgs: 129 active+clean; 339 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.7 MiB/s wr, 424 op/s 2024-05-27T07:56:40.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:39 smithi163 ceph-mon[36451]: pgmap v578: 129 pgs: 129 active+clean; 339 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.7 MiB/s wr, 424 op/s 2024-05-27T07:56:40.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:39 smithi186 ceph-mon[36647]: pgmap v578: 129 pgs: 129 active+clean; 339 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.7 MiB/s wr, 424 op/s 2024-05-27T07:56:40.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 42287 12.50 MB/sec execute 52 sec latency 243.329 ms 2024-05-27T07:56:41.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 42631 12.59 MB/sec execute 53 sec latency 66.243 ms 2024-05-27T07:56:42.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:41 smithi114 ceph-mon[32038]: pgmap v579: 129 pgs: 129 active+clean; 333 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-05-27T07:56:42.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:41 smithi163 ceph-mon[36451]: pgmap v579: 129 pgs: 129 active+clean; 333 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-05-27T07:56:42.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:41 smithi186 ceph-mon[36647]: pgmap v579: 129 pgs: 129 active+clean; 333 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-05-27T07:56:42.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 42870 12.51 MB/sec execute 54 sec latency 163.082 ms 2024-05-27T07:56:42.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:42 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:56:42.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:42 smithi114 systemd[1]: Stopped Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:56:42.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:42 smithi114 systemd[1]: Starting Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:56:42.813 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:42 smithi114 podman[127838]: 2024-05-27 07:56:42.689797799 +0000 UTC m=+0.017670647 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:56:43.098 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:42 smithi114 podman[127838]: 2024-05-27 07:56:42.81304278 +0000 UTC m=+0.140915618 container create b8604cfea005a835e2a9b99f75ad906ce13bd7d2fbc40a1ead3f9b3181090b62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-27T07:56:43.099 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:43 smithi114 podman[127838]: 2024-05-27 07:56:43.058045914 +0000 UTC m=+0.385918760 container init b8604cfea005a835e2a9b99f75ad906ce13bd7d2fbc40a1ead3f9b3181090b62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:56:43.099 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:43 smithi114 podman[127838]: 2024-05-27 07:56:43.061469162 +0000 UTC m=+0.389342008 container start b8604cfea005a835e2a9b99f75ad906ce13bd7d2fbc40a1ead3f9b3181090b62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:56:43.464 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:43 smithi114 ceph-mds[127857]: -- 172.21.15.114:0/833934080 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56137fae3a40 con 0x56137fb2ba80 2024-05-27T07:56:43.464 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:43 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: starting mds.a at 2024-05-27T07:56:43.464 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:43 smithi114 bash[127838]: b8604cfea005a835e2a9b99f75ad906ce13bd7d2fbc40a1ead3f9b3181090b62 2024-05-27T07:56:43.464 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:56:43 smithi114 systemd[1]: Started Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:56:43.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 43175 12.49 MB/sec execute 55 sec latency 172.177 ms 2024-05-27T07:56:44.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:43 smithi114 ceph-mon[32038]: pgmap v580: 129 pgs: 129 active+clean; 322 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 517 op/s 2024-05-27T07:56:44.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:43 smithi114 ceph-mon[32038]: daemon mds.a restarted 2024-05-27T07:56:44.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:43 smithi163 ceph-mon[36451]: pgmap v580: 129 pgs: 129 active+clean; 322 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 517 op/s 2024-05-27T07:56:44.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:43 smithi163 ceph-mon[36451]: daemon mds.a restarted 2024-05-27T07:56:44.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:43 smithi186 ceph-mon[36647]: pgmap v580: 129 pgs: 129 active+clean; 322 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 517 op/s 2024-05-27T07:56:44.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:43 smithi186 ceph-mon[36647]: daemon mds.a restarted 2024-05-27T07:56:44.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 43506 12.63 MB/sec execute 56 sec latency 184.038 ms 2024-05-27T07:56:45.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:44 smithi114 ceph-mon[32038]: osdmap e150: 12 total, 12 up, 12 in 2024-05-27T07:56:45.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:44 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6840/1351506174,v1:172.21.15.114:6841/1351506174] up:boot 2024-05-27T07:56:45.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:44 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:56:45.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:56:45.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:44 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:56:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:44 smithi163 ceph-mon[36451]: osdmap e150: 12 total, 12 up, 12 in 2024-05-27T07:56:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:44 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6840/1351506174,v1:172.21.15.114:6841/1351506174] up:boot 2024-05-27T07:56:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:44 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:56:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:56:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:44 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:56:45.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:44 smithi186 ceph-mon[36647]: osdmap e150: 12 total, 12 up, 12 in 2024-05-27T07:56:45.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:44 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6840/1351506174,v1:172.21.15.114:6841/1351506174] up:boot 2024-05-27T07:56:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:44 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:56:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:56:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:44 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:56:45.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 43757 12.65 MB/sec execute 57 sec latency 346.012 ms 2024-05-27T07:56:46.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:45 smithi114 ceph-mon[32038]: pgmap v582: 129 pgs: 129 active+clean; 327 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 508 op/s 2024-05-27T07:56:46.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:45 smithi163 ceph-mon[36451]: pgmap v582: 129 pgs: 129 active+clean; 327 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 508 op/s 2024-05-27T07:56:46.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:45 smithi186 ceph-mon[36647]: pgmap v582: 129 pgs: 129 active+clean; 327 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 508 op/s 2024-05-27T07:56:46.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 44106 12.72 MB/sec execute 58 sec latency 166.897 ms 2024-05-27T07:56:47.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 44477 12.82 MB/sec execute 59 sec latency 214.714 ms 2024-05-27T07:56:48.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:47 smithi114 ceph-mon[32038]: pgmap v583: 129 pgs: 129 active+clean; 332 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 499 op/s 2024-05-27T07:56:48.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:47 smithi163 ceph-mon[36451]: pgmap v583: 129 pgs: 129 active+clean; 332 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 499 op/s 2024-05-27T07:56:48.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:47 smithi186 ceph-mon[36647]: pgmap v583: 129 pgs: 129 active+clean; 332 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 499 op/s 2024-05-27T07:56:48.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 44911 13.08 MB/sec execute 60 sec latency 190.358 ms 2024-05-27T07:56:49.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 45245 13.12 MB/sec execute 61 sec latency 204.296 ms 2024-05-27T07:56:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.720+0000 7fcec899d640 1 -- 172.21.15.114:0/592394368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcec4102ab0 msgr2=0x7fcec4102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.720+0000 7fcec899d640 1 --2- 172.21.15.114:0/592394368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcec4102ab0 0x7fcec4102ec0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fceac009900 tx=0x7fceac02a410 comp rx=0 tx=0).stop 2024-05-27T07:56:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.720+0000 7fcec899d640 1 -- 172.21.15.114:0/592394368 shutdown_connections 2024-05-27T07:56:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.720+0000 7fcec899d640 1 --2- 172.21.15.114:0/592394368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec410d340 0x7fcec410f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.721+0000 7fcec899d640 1 --2- 172.21.15.114:0/592394368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcec4102ab0 0x7fcec4102ec0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.721+0000 7fcec899d640 1 --2- 172.21.15.114:0/592394368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec41089e0 0x7fcec4102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.721+0000 7fcec899d640 1 -- 172.21.15.114:0/592394368 >> 172.21.15.114:0/592394368 conn(0x7fcec40fe210 msgr2=0x7fcec4100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.721+0000 7fcec899d640 1 -- 172.21.15.114:0/592394368 shutdown_connections 2024-05-27T07:56:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.721+0000 7fcec899d640 1 -- 172.21.15.114:0/592394368 wait complete. 2024-05-27T07:56:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.721+0000 7fcec899d640 1 Processor -- start 2024-05-27T07:56:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.721+0000 7fcec899d640 1 -- start start 2024-05-27T07:56:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec899d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec4102ab0 0x7fcec419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec899d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec41089e0 0x7fcec419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec899d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcec410d340 0x7fcec419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec899d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcec4112ba0 con 0x7fcec410d340 2024-05-27T07:56:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec899d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcec4112a20 con 0x7fcec4102ab0 2024-05-27T07:56:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec899d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcec4112d20 con 0x7fcec41089e0 2024-05-27T07:56:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcec410d340 0x7fcec419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcec410d340 0x7fcec419ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44112/0 (socket says 172.21.15.114:44112) 2024-05-27T07:56:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec41089e0 0x7fcec419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec3fff640 1 -- 172.21.15.114:0/1692114984 learned_addr learned my addr 172.21.15.114:0/1692114984 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec37fe640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec4102ab0 0x7fcec419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec3fff640 1 -- 172.21.15.114:0/1692114984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec41089e0 msgr2=0x7fcec419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec3fff640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec41089e0 0x7fcec419ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec3fff640 1 -- 172.21.15.114:0/1692114984 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec4102ab0 msgr2=0x7fcec419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:49.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec3fff640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec4102ab0 0x7fcec419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec3fff640 1 -- 172.21.15.114:0/1692114984 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec419f550 con 0x7fcec410d340 2024-05-27T07:56:49.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcebbfff640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec41089e0 0x7fcec419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:56:49.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec3fff640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcec410d340 0x7fcec419ef10 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00eb10 tx=0x7fcebc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:49.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec17fa640 1 -- 172.21.15.114:0/1692114984 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc004b20 con 0x7fcec410d340 2024-05-27T07:56:49.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec17fa640 1 -- 172.21.15.114:0/1692114984 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcebc004cc0 con 0x7fcec410d340 2024-05-27T07:56:49.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.722+0000 7fcec899d640 1 -- 172.21.15.114:0/1692114984 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec419f840 con 0x7fcec410d340 2024-05-27T07:56:49.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.723+0000 7fcec17fa640 1 -- 172.21.15.114:0/1692114984 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc010690 con 0x7fcec410d340 2024-05-27T07:56:49.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.723+0000 7fcec899d640 1 -- 172.21.15.114:0/1692114984 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec41a71c0 con 0x7fcec410d340 2024-05-27T07:56:49.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.725+0000 7fcec17fa640 1 -- 172.21.15.114:0/1692114984 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fcebc005020 con 0x7fcec410d340 2024-05-27T07:56:49.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.725+0000 7fced0b8d740 1 Processor -- start 2024-05-27T07:56:49.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.725+0000 7fced0b8d740 1 -- start start 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.725+0000 7fced0b8d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee884c1b0 0x556ee8bed590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.725+0000 7fced0b8d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee88232f0 0x556ee8bedad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.725+0000 7fced0b8d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee882b4d0 0x556ee8bee010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.725+0000 7fced0b8d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556ee8b5fc30 con 0x556ee882b4d0 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.725+0000 7fced0b8d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556ee8b4f7d0 con 0x556ee88232f0 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.725+0000 7fced0b8d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556ee8b53810 con 0x556ee884c1b0 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee884c1b0 0x556ee8bed590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee884c1b0 0x556ee8bed590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:44404/0 (socket says 172.21.15.114:44404) 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcebbfff640 1 -- 172.21.15.114:0/1554107141 learned_addr learned my addr 172.21.15.114:0/1554107141 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcebbfff640 1 --2- 172.21.15.114:0/1554107141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee882b4d0 0x556ee8bee010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcec37fe640 1 --2- 172.21.15.114:0/1554107141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee88232f0 0x556ee8bedad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcec17fa640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fceb00b2950 0x7fceb00b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcebbfff640 1 -- 172.21.15.114:0/1554107141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee88232f0 msgr2=0x556ee8bedad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcebbfff640 1 --2- 172.21.15.114:0/1554107141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee88232f0 0x556ee8bedad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcec3fff640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fceb00b2950 0x7fceb00b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcebbfff640 1 -- 172.21.15.114:0/1554107141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee882b4d0 msgr2=0x556ee8bee010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcebbfff640 1 --2- 172.21.15.114:0/1554107141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee882b4d0 0x556ee8bee010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcebbfff640 1 -- 172.21.15.114:0/1554107141 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ee8864210 con 0x556ee884c1b0 2024-05-27T07:56:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcec37fe640 1 --2- 172.21.15.114:0/1554107141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee88232f0 0x556ee8bedad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:56:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcec17fa640 1 -- 172.21.15.114:0/1692114984 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 9466+0+0 (secure 0 0 0) 0x7fcebc0e2ec0 con 0x7fcec410d340 2024-05-27T07:56:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcebbfff640 1 --2- 172.21.15.114:0/1554107141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee884c1b0 0x556ee8bed590 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fceac0057c0 tx=0x7fceac000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ecb7f4088f273a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fceb9ffb640 1 -- 172.21.15.114:0/1554107141 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac035070 con 0x556ee884c1b0 2024-05-27T07:56:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fcec3fff640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fceb00b2950 0x7fceb00b4e60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0a7060 tx=0x7fcebc076720 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fceb9ffb640 1 -- 172.21.15.114:0/1554107141 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceac0054d0 con 0x556ee884c1b0 2024-05-27T07:56:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.726+0000 7fceb9ffb640 1 -- 172.21.15.114:0/1554107141 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac004a40 con 0x556ee884c1b0 2024-05-27T07:56:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.727+0000 7fced0b8d740 1 -- 172.21.15.114:0/1554107141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee884c1b0 msgr2=0x556ee8bed590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.727+0000 7fced0b8d740 1 --2- 172.21.15.114:0/1554107141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee884c1b0 0x556ee8bed590 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fceac0057c0 tx=0x7fceac000bc0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.727+0000 7fced0b8d740 1 -- 172.21.15.114:0/1554107141 shutdown_connections 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.727+0000 7fced0b8d740 1 --2- 172.21.15.114:0/1554107141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee882b4d0 0x556ee8bee010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.727+0000 7fced0b8d740 1 --2- 172.21.15.114:0/1554107141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee88232f0 0x556ee8bedad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.727+0000 7fced0b8d740 1 --2- 172.21.15.114:0/1554107141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee884c1b0 0x556ee8bed590 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.727+0000 7fced0b8d740 1 -- 172.21.15.114:0/1554107141 >> 172.21.15.114:0/1554107141 conn(0x556ee8840300 msgr2=0x556ee8be9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.727+0000 7fced0b8d740 1 -- 172.21.15.114:0/1554107141 shutdown_connections 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.727+0000 7fced0b8d740 1 -- 172.21.15.114:0/1554107141 wait complete. 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fced0b8d740 1 -- start start 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fced0b8d740 1 Processor -- start 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fced0b8d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee88232f0 0x556ee8bf9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fced0b8d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee882b4d0 0x556ee8bf9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fced0b8d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee884c1b0 0x556ee8bf7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fced0b8d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556ee8835790 con 0x556ee884c1b0 2024-05-27T07:56:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fced0b8d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556ee8b0b890 con 0x556ee882b4d0 2024-05-27T07:56:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fced0b8d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556ee8bfa660 con 0x556ee88232f0 2024-05-27T07:56:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fced0b8d740 10 client.0 _ll_register_callbacks cb 0x556ee8863a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee884c1b0 0x556ee8bf7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee884c1b0 0x556ee8bf7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44130/0 (socket says 172.21.15.114:44130) 2024-05-27T07:56:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fcebbfff640 1 -- 172.21.15.114:0/3054650380 learned_addr learned my addr 172.21.15.114:0/3054650380 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fcebbfff640 1 --2- 172.21.15.114:0/3054650380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee88232f0 0x556ee8bf9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.728+0000 7fcec37fe640 1 --2- 172.21.15.114:0/3054650380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee882b4d0 0x556ee8bf9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fcebbfff640 1 -- 172.21.15.114:0/3054650380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee88232f0 msgr2=0x556ee8bf9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fcebbfff640 1 --2- 172.21.15.114:0/3054650380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee88232f0 0x556ee8bf9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fcebbfff640 1 -- 172.21.15.114:0/3054650380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee882b4d0 msgr2=0x556ee8bf9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fcebbfff640 1 --2- 172.21.15.114:0/3054650380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee882b4d0 0x556ee8bf9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fcebbfff640 1 -- 172.21.15.114:0/3054650380 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ee8930620 con 0x556ee884c1b0 2024-05-27T07:56:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fcebbfff640 1 --2- 172.21.15.114:0/3054650380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee884c1b0 0x556ee8bf7b60 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fceac02a930 tx=0x7fceac041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e58c19d50bc42de server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fcea2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:56:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fcea2ffd640 1 -- 172.21.15.114:0/3054650380 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac04e030 con 0x556ee884c1b0 2024-05-27T07:56:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fcea2ffd640 1 -- 172.21.15.114:0/3054650380 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(150..150 src has 1..150) ==== 9466+0+0 (secure 0 0 0) 0x7fceac0049e0 con 0x556ee884c1b0 2024-05-27T07:56:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fced0b8d740 1 -- 172.21.15.114:0/3054650380 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fceac0054d0 con 0x556ee884c1b0 2024-05-27T07:56:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fcea2ffd640 1 -- 172.21.15.114:0/3054650380 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fceac035070 con 0x556ee884c1b0 2024-05-27T07:56:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fced0b8d740 10 client.19329 fetch_fsmap learned FSMap version 96 2024-05-27T07:56:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fced0b8d740 1 -- 172.21.15.114:0/3054650380 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x556ee8c06a00 con 0x556ee884c1b0 2024-05-27T07:56:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.729+0000 7fcea2ffd640 1 -- 172.21.15.114:0/3054650380 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceac0054d0 con 0x556ee884c1b0 2024-05-27T07:56:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.730+0000 7fcea2ffd640 1 -- 172.21.15.114:0/3054650380 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 96) ==== 4095+0+0 (secure 0 0 0) 0x7fceac03da20 con 0x556ee884c1b0 2024-05-27T07:56:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.730+0000 7fced0b8d740 10 client.19329 fetch_fsmap finished waiting for FSMap version 96 2024-05-27T07:56:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.730+0000 7fced0b8d740 10 client.19329 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.730+0000 7fced0b8d740 20 client.19329 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.730+0000 7fced0b8d740 1 --2- 172.21.15.114:0/3054650380 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ee880ad80 0x556ee8c0a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.730+0000 7fced0b8d740 4 client.19329 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:56:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.730+0000 7fced0b8d740 1 -- 172.21.15.114:0/3054650380 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ee8b61b80 con 0x556ee880ad80 2024-05-27T07:56:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.730+0000 7fcec37fe640 1 --2- 172.21.15.114:0/3054650380 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ee880ad80 0x556ee8c0a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.730+0000 7fcea2ffd640 10 client.19329 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.730+0000 7fcec37fe640 1 --2- 172.21.15.114:0/3054650380 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ee880ad80 0x556ee8c0a0e0 crc :-1 s=READY pgs=460 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6bdbf42cbbc0f8a server_cookie=44125ee113e06d0 in_seq=0 out_seq=0 2024-05-27T07:56:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fcea2ffd640 1 -- 172.21.15.114:0/3054650380 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556ee8b61b80 con 0x556ee880ad80 2024-05-27T07:56:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fcea2ffd640 10 client.19329 handle_command_reply: tid=0 2024-05-27T07:56:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fced0b8d740 1 client.19329 shutdown 2024-05-27T07:56:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fced0b8d740 1 -- 172.21.15.114:0/3054650380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee884c1b0 msgr2=0x556ee8bf7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fced0b8d740 1 --2- 172.21.15.114:0/3054650380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee884c1b0 0x556ee8bf7b60 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fceac02a930 tx=0x7fceac041040 comp rx=0 tx=0).stop 2024-05-27T07:56:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fced0b8d740 1 -- 172.21.15.114:0/3054650380 shutdown_connections 2024-05-27T07:56:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fced0b8d740 1 --2- 172.21.15.114:0/3054650380 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ee880ad80 0x556ee8c0a0e0 crc :-1 s=READY pgs=460 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fced0b8d740 1 --2- 172.21.15.114:0/3054650380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee884c1b0 0x556ee8bf7b60 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fced0b8d740 1 --2- 172.21.15.114:0/3054650380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee882b4d0 0x556ee8bf9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fcea2ffd640 0 client.19329 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fced0b8d740 1 --2- 172.21.15.114:0/3054650380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee88232f0 0x556ee8bf9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fced0b8d740 1 -- 172.21.15.114:0/3054650380 >> 172.21.15.114:0/3054650380 conn(0x556ee8840300 msgr2=0x556ee8be8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fced0b8d740 1 -- 172.21.15.114:0/3054650380 shutdown_connections 2024-05-27T07:56:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fced0b8d740 1 -- 172.21.15.114:0/3054650380 wait complete. 2024-05-27T07:56:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.734+0000 7fced0b8d740 20 client.19329 trim_cache size 0 max 16384 2024-05-27T07:56:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.765+0000 7fced0b8d740 1 Processor -- start 2024-05-27T07:56:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.766+0000 7fced0b8d740 1 -- start start 2024-05-27T07:56:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.766+0000 7fced0b8d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee8840300 0x556ee8bf80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.766+0000 7fced0b8d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee880ad80 0x556ee8bf1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.766+0000 7fced0b8d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee88232f0 0x556ee8c0f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.766+0000 7fced0b8d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcec41b0cd0 con 0x556ee88232f0 2024-05-27T07:56:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.766+0000 7fced0b8d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcec41b13a0 con 0x556ee880ad80 2024-05-27T07:56:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.766+0000 7fced0b8d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcec41afc70 con 0x556ee8840300 2024-05-27T07:56:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.769+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee8840300 0x556ee8bf80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.769+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee8840300 0x556ee8bf80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:44428/0 (socket says 172.21.15.114:44428) 2024-05-27T07:56:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.769+0000 7fcebbfff640 1 -- 172.21.15.114:0/2135900501 learned_addr learned my addr 172.21.15.114:0/2135900501 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.769+0000 7fcebbfff640 1 --2- 172.21.15.114:0/2135900501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee88232f0 0x556ee8c0f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.769+0000 7fcebbfff640 1 -- 172.21.15.114:0/2135900501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee880ad80 msgr2=0x556ee8bf1760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:56:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.770+0000 7fcec37fe640 1 --2- 172.21.15.114:0/2135900501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee880ad80 0x556ee8bf1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.770+0000 7fcebbfff640 1 --2- 172.21.15.114:0/2135900501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee880ad80 0x556ee8bf1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.770+0000 7fcebbfff640 1 -- 172.21.15.114:0/2135900501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee88232f0 msgr2=0x556ee8c0f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.770+0000 7fcebbfff640 1 --2- 172.21.15.114:0/2135900501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee88232f0 0x556ee8c0f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.770+0000 7fcebbfff640 1 -- 172.21.15.114:0/2135900501 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ee8bf60b0 con 0x556ee8840300 2024-05-27T07:56:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.770+0000 7fcebbfff640 1 --2- 172.21.15.114:0/2135900501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee8840300 0x556ee8bf80b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fceac0057c0 tx=0x7fceac005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c1dbe3ef8f4f25b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.773+0000 7fceb97fa640 1 -- 172.21.15.114:0/2135900501 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac04e040 con 0x556ee8840300 2024-05-27T07:56:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.773+0000 7fceb97fa640 1 -- 172.21.15.114:0/2135900501 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceac0054c0 con 0x556ee8840300 2024-05-27T07:56:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.773+0000 7fceb97fa640 1 -- 172.21.15.114:0/2135900501 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac04ae20 con 0x556ee8840300 2024-05-27T07:56:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.775+0000 7fced0b8d740 1 -- 172.21.15.114:0/2135900501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee8840300 msgr2=0x556ee8bf80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.775+0000 7fced0b8d740 1 --2- 172.21.15.114:0/2135900501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee8840300 0x556ee8bf80b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fceac0057c0 tx=0x7fceac005d90 comp rx=0 tx=0).stop 2024-05-27T07:56:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.781+0000 7fced0b8d740 1 -- 172.21.15.114:0/2135900501 shutdown_connections 2024-05-27T07:56:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.781+0000 7fced0b8d740 1 --2- 172.21.15.114:0/2135900501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee88232f0 0x556ee8c0f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.781+0000 7fced0b8d740 1 --2- 172.21.15.114:0/2135900501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee880ad80 0x556ee8bf1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.781+0000 7fced0b8d740 1 --2- 172.21.15.114:0/2135900501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee8840300 0x556ee8bf80b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.781+0000 7fced0b8d740 1 -- 172.21.15.114:0/2135900501 >> 172.21.15.114:0/2135900501 conn(0x556ee8bf34e0 msgr2=0x556ee8bf1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.791+0000 7fced0b8d740 1 -- 172.21.15.114:0/2135900501 shutdown_connections 2024-05-27T07:56:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.794+0000 7fced0b8d740 1 -- 172.21.15.114:0/2135900501 wait complete. 2024-05-27T07:56:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.794+0000 7fced0b8d740 1 -- start start 2024-05-27T07:56:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.795+0000 7fced0b8d740 1 Processor -- start 2024-05-27T07:56:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.799+0000 7fced0b8d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee880ad80 0x556ee8bfccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.799+0000 7fced0b8d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee88232f0 0x556ee8bfd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.799+0000 7fced0b8d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee8840300 0x556ee8c78fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.799+0000 7fced0b8d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcec41af5a0 con 0x556ee880ad80 2024-05-27T07:56:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.799+0000 7fced0b8d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcec41aef30 con 0x556ee8840300 2024-05-27T07:56:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.799+0000 7fced0b8d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556ee8be4100 con 0x556ee88232f0 2024-05-27T07:56:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.799+0000 7fced0b8d740 10 client.0 _ll_register_callbacks cb 0x7fceac031520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.802+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee880ad80 0x556ee8bfccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.802+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee880ad80 0x556ee8bfccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44138/0 (socket says 172.21.15.114:44138) 2024-05-27T07:56:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.802+0000 7fcebbfff640 1 -- 172.21.15.114:0/3556583780 learned_addr learned my addr 172.21.15.114:0/3556583780 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.802+0000 7fcebbfff640 1 --2- 172.21.15.114:0/3556583780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee8840300 0x556ee8c78fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.803+0000 7fcebbfff640 1 -- 172.21.15.114:0/3556583780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee88232f0 msgr2=0x556ee8bfd200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:56:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.803+0000 7fcebbfff640 1 --2- 172.21.15.114:0/3556583780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee88232f0 0x556ee8bfd200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.803+0000 7fcebbfff640 1 -- 172.21.15.114:0/3556583780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee8840300 msgr2=0x556ee8c78fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.803+0000 7fcebbfff640 1 --2- 172.21.15.114:0/3556583780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee8840300 0x556ee8c78fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.803+0000 7fcebbfff640 1 -- 172.21.15.114:0/3556583780 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ee8c8a510 con 0x556ee880ad80 2024-05-27T07:56:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.804+0000 7fcebbfff640 1 --2- 172.21.15.114:0/3556583780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee880ad80 0x556ee8bfccc0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x556ee8ada5d0 tx=0x7fceac005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e4765a329a8279a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.806+0000 7fcea2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:56:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.808+0000 7fcea2ffd640 1 -- 172.21.15.114:0/3556583780 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac04e040 con 0x556ee880ad80 2024-05-27T07:56:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.808+0000 7fced0b8d740 1 -- 172.21.15.114:0/3556583780 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556ee8b5b360 con 0x556ee880ad80 2024-05-27T07:56:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.808+0000 7fcea2ffd640 1 -- 172.21.15.114:0/3556583780 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(150..150 src has 1..150) ==== 9466+0+0 (secure 0 0 0) 0x7fceac031960 con 0x556ee880ad80 2024-05-27T07:56:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.811+0000 7fcea2ffd640 1 -- 172.21.15.114:0/3556583780 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fceac031b60 con 0x556ee880ad80 2024-05-27T07:56:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.819+0000 7fced0b8d740 10 client.19335 fetch_fsmap learned FSMap version 96 2024-05-27T07:56:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.819+0000 7fced0b8d740 1 -- 172.21.15.114:0/3556583780 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x556ee8c8b620 con 0x556ee880ad80 2024-05-27T07:56:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.821+0000 7fcea2ffd640 1 -- 172.21.15.114:0/3556583780 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556ee8b5b360 con 0x556ee880ad80 2024-05-27T07:56:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.822+0000 7fcea2ffd640 1 -- 172.21.15.114:0/3556583780 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 96) ==== 4095+0+0 (secure 0 0 0) 0x7fceac03d6b0 con 0x556ee880ad80 2024-05-27T07:56:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.823+0000 7fced0b8d740 10 client.19335 fetch_fsmap finished waiting for FSMap version 96 2024-05-27T07:56:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.823+0000 7fced0b8d740 10 client.19335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.823+0000 7fced0b8d740 20 client.19335 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.823+0000 7fced0b8d740 1 --2- 172.21.15.114:0/3556583780 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ee882b4d0 0x556ee8c8dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.823+0000 7fced0b8d740 4 client.19335 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:56:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.823+0000 7fced0b8d740 1 -- 172.21.15.114:0/3556583780 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556ee8ad55f0 con 0x556ee882b4d0 2024-05-27T07:56:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.826+0000 7fcec37fe640 1 --2- 172.21.15.114:0/3556583780 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ee882b4d0 0x556ee8c8dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.838+0000 7fcec37fe640 1 --2- 172.21.15.114:0/3556583780 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ee882b4d0 0x556ee8c8dc80 crc :-1 s=READY pgs=461 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7763e996a3a646e server_cookie=b169dc7af46f6fba in_seq=0 out_seq=0 2024-05-27T07:56:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:49.838+0000 7fcea2ffd640 10 client.19335 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:50.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:49 smithi114 ceph-mon[32038]: pgmap v584: 129 pgs: 129 active+clean; 340 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 357 op/s 2024-05-27T07:56:50.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:49 smithi163 ceph-mon[36451]: pgmap v584: 129 pgs: 129 active+clean; 340 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 357 op/s 2024-05-27T07:56:50.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:49 smithi186 ceph-mon[36647]: pgmap v584: 129 pgs: 129 active+clean; 340 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 357 op/s 2024-05-27T07:56:50.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 45357 13.09 MB/sec execute 62 sec latency 518.004 ms 2024-05-27T07:56:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fcea2ffd640 1 -- 172.21.15.114:0/3556583780 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x556ee8ad55f0 con 0x556ee882b4d0 2024-05-27T07:56:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fcea2ffd640 10 client.19335 handle_command_reply: tid=0 2024-05-27T07:56:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fced0b8d740 1 client.19335 shutdown 2024-05-27T07:56:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fced0b8d740 1 -- 172.21.15.114:0/3556583780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee880ad80 msgr2=0x556ee8bfccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fced0b8d740 1 --2- 172.21.15.114:0/3556583780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee880ad80 0x556ee8bfccc0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x556ee8ada5d0 tx=0x7fceac005c90 comp rx=0 tx=0).stop 2024-05-27T07:56:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fced0b8d740 1 -- 172.21.15.114:0/3556583780 shutdown_connections 2024-05-27T07:56:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fced0b8d740 1 --2- 172.21.15.114:0/3556583780 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x556ee882b4d0 0x556ee8c8dc80 crc :-1 s=READY pgs=461 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fced0b8d740 1 --2- 172.21.15.114:0/3556583780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556ee8840300 0x556ee8c78fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fced0b8d740 1 --2- 172.21.15.114:0/3556583780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ee88232f0 0x556ee8bfd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fced0b8d740 1 --2- 172.21.15.114:0/3556583780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ee880ad80 0x556ee8bfccc0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fced0b8d740 1 -- 172.21.15.114:0/3556583780 >> 172.21.15.114:0/3556583780 conn(0x556ee8bf34e0 msgr2=0x556ee8be92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fcea2ffd640 0 client.19335 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fced0b8d740 1 -- 172.21.15.114:0/3556583780 shutdown_connections 2024-05-27T07:56:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.587+0000 7fced0b8d740 1 -- 172.21.15.114:0/3556583780 wait complete. 2024-05-27T07:56:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.588+0000 7fced0b8d740 20 client.19335 trim_cache size 0 max 16384 2024-05-27T07:56:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:56:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:56:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:56:51.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.588+0000 7fcec899d640 1 -- 172.21.15.114:0/1692114984 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fceb00b2950 msgr2=0x7fceb00b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:51.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.588+0000 7fcec899d640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fceb00b2950 0x7fceb00b4e60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0a7060 tx=0x7fcebc076720 comp rx=0 tx=0).stop 2024-05-27T07:56:51.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.588+0000 7fcec899d640 1 -- 172.21.15.114:0/1692114984 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcec410d340 msgr2=0x7fcec419ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:51.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.588+0000 7fcec899d640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcec410d340 0x7fcec419ef10 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00eb10 tx=0x7fcebc00c2d0 comp rx=0 tx=0).stop 2024-05-27T07:56:51.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.588+0000 7fcec899d640 1 -- 172.21.15.114:0/1692114984 shutdown_connections 2024-05-27T07:56:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.589+0000 7fcec899d640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fceb00b2950 0x7fceb00b4e60 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.589+0000 7fcec899d640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcec410d340 0x7fcec419ef10 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.589+0000 7fcec899d640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec41089e0 0x7fcec419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.589+0000 7fcec899d640 1 --2- 172.21.15.114:0/1692114984 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec4102ab0 0x7fcec419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.589+0000 7fcec899d640 1 -- 172.21.15.114:0/1692114984 >> 172.21.15.114:0/1692114984 conn(0x7fcec40fe210 msgr2=0x7fcec41009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.589+0000 7fcec899d640 1 -- 172.21.15.114:0/1692114984 shutdown_connections 2024-05-27T07:56:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:56:51.589+0000 7fcec899d640 1 -- 172.21.15.114:0/1692114984 wait complete. 2024-05-27T07:56:51.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 45476 13.01 MB/sec execute 63 sec latency 690.299 ms 2024-05-27T07:56:52.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:51 smithi114 ceph-mon[32038]: pgmap v585: 129 pgs: 129 active+clean; 341 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 284 op/s 2024-05-27T07:56:52.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:51 smithi163 ceph-mon[36451]: pgmap v585: 129 pgs: 129 active+clean; 341 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 284 op/s 2024-05-27T07:56:52.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:51 smithi186 ceph-mon[36647]: pgmap v585: 129 pgs: 129 active+clean; 341 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 284 op/s 2024-05-27T07:56:52.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 45825 13.04 MB/sec execute 64 sec latency 204.806 ms 2024-05-27T07:56:53.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 46124 13.10 MB/sec execute 65 sec latency 204.120 ms 2024-05-27T07:56:54.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:53 smithi114 ceph-mon[32038]: pgmap v586: 129 pgs: 129 active+clean; 350 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.8 MiB/s wr, 434 op/s 2024-05-27T07:56:54.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:53 smithi163 ceph-mon[36451]: pgmap v586: 129 pgs: 129 active+clean; 350 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.8 MiB/s wr, 434 op/s 2024-05-27T07:56:54.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:53 smithi186 ceph-mon[36647]: pgmap v586: 129 pgs: 129 active+clean; 350 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.8 MiB/s wr, 434 op/s 2024-05-27T07:56:54.605 INFO:tasks.workunit.client.0.smithi114.stdout: 10 46469 13.07 MB/sec execute 66 sec latency 158.854 ms 2024-05-27T07:56:55.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:56:55.523 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:56:55.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 46724 13.13 MB/sec execute 67 sec latency 382.191 ms 2024-05-27T07:56:55.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:55 smithi114 ceph-mon[32038]: pgmap v587: 129 pgs: 129 active+clean; 342 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.4 MiB/s wr, 466 op/s 2024-05-27T07:56:56.226 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:55 smithi163 ceph-mon[36451]: pgmap v587: 129 pgs: 129 active+clean; 342 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.4 MiB/s wr, 466 op/s 2024-05-27T07:56:56.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:55 smithi186 ceph-mon[36647]: pgmap v587: 129 pgs: 129 active+clean; 342 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.4 MiB/s wr, 466 op/s 2024-05-27T07:56:56.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 47051 13.20 MB/sec execute 68 sec latency 123.112 ms 2024-05-27T07:56:57.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 47520 13.43 MB/sec execute 69 sec latency 195.111 ms 2024-05-27T07:56:57.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.873+0000 7fadfe5bf700 1 -- 172.21.15.114:0/2848645822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadf8108750 msgr2=0x7fadf81100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:57.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.873+0000 7fadfe5bf700 1 --2- 172.21.15.114:0/2848645822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadf8108750 0x7fadf81100b0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fadf000b0d0 tx=0x7fadf000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.873+0000 7fadfe5bf700 1 -- 172.21.15.114:0/2848645822 shutdown_connections 2024-05-27T07:56:57.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.873+0000 7fadfe5bf700 1 --2- 172.21.15.114:0/2848645822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadf8108750 0x7fadf81100b0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.873+0000 7fadfe5bf700 1 --2- 172.21.15.114:0/2848645822 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf81034d0 0x7fadf81038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.873+0000 7fadfe5bf700 1 --2- 172.21.15.114:0/2848645822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadf8107e00 0x7fadf8108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.873+0000 7fadfe5bf700 1 -- 172.21.15.114:0/2848645822 >> 172.21.15.114:0/2848645822 conn(0x7fadf80fef20 msgr2=0x7fadf8101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:57.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.873+0000 7fadfe5bf700 1 -- 172.21.15.114:0/2848645822 shutdown_connections 2024-05-27T07:56:57.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.873+0000 7fadfe5bf700 1 -- 172.21.15.114:0/2848645822 wait complete. 2024-05-27T07:56:57.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.874+0000 7fadfe5bf700 1 Processor -- start 2024-05-27T07:56:57.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.875+0000 7fadfe5bf700 1 -- start start 2024-05-27T07:56:57.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.875+0000 7fadfe5bf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadf81034d0 0x7fadf8199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.875+0000 7fadfe5bf700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf8107e00 0x7fadf819a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.875+0000 7fadfe5bf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadf8108750 0x7fadf819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.875+0000 7fadfe5bf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fadf8113380 con 0x7fadf8108750 2024-05-27T07:56:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.875+0000 7fadfe5bf700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fadf8113200 con 0x7fadf8107e00 2024-05-27T07:56:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.875+0000 7fadfe5bf700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fadf8113500 con 0x7fadf81034d0 2024-05-27T07:56:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.875+0000 7fadf7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadf81034d0 0x7fadf8199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadf77fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf8107e00 0x7fadf819a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadf7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadf81034d0 0x7fadf8199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:39758/0 (socket says 172.21.15.114:39758) 2024-05-27T07:56:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadf7fff700 1 -- 172.21.15.114:0/1900585970 learned_addr learned my addr 172.21.15.114:0/1900585970 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadfcb5c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadf8108750 0x7fadf819e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadf7fff700 1 -- 172.21.15.114:0/1900585970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf8107e00 msgr2=0x7fadf819a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadf7fff700 1 --2- 172.21.15.114:0/1900585970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf8107e00 0x7fadf819a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadf7fff700 1 -- 172.21.15.114:0/1900585970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadf8108750 msgr2=0x7fadf819e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadf7fff700 1 --2- 172.21.15.114:0/1900585970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadf8108750 0x7fadf819e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadf7fff700 1 -- 172.21.15.114:0/1900585970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf819ec40 con 0x7fadf81034d0 2024-05-27T07:56:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadf7fff700 1 --2- 172.21.15.114:0/1900585970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadf81034d0 0x7fadf8199d70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fade800d980 tx=0x7fade800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadf57fa700 1 -- 172.21.15.114:0/1900585970 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8016070 con 0x7fadf81034d0 2024-05-27T07:56:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadfe5bf700 1 -- 172.21.15.114:0/1900585970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadf819ef30 con 0x7fadf81034d0 2024-05-27T07:56:57.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadf57fa700 1 -- 172.21.15.114:0/1900585970 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fade8004830 con 0x7fadf81034d0 2024-05-27T07:56:57.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.877+0000 7fadfe5bf700 1 -- 172.21.15.114:0/1900585970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadf81a6850 con 0x7fadf81034d0 2024-05-27T07:56:57.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.876+0000 7fadf57fa700 1 -- 172.21.15.114:0/1900585970 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8005540 con 0x7fadf81034d0 2024-05-27T07:56:57.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.879+0000 7fadf57fa700 1 -- 172.21.15.114:0/1900585970 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fade800f690 con 0x7fadf81034d0 2024-05-27T07:56:57.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.880+0000 7fae0cfd4b80 1 Processor -- start 2024-05-27T07:56:57.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.880+0000 7fae0cfd4b80 1 -- start start 2024-05-27T07:56:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fae0cfd4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487915e40 0x560487a99710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fae0cfd4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a99c50 0x560487a93a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fae0cfd4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a93f80 0x560487a94390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fae0cfd4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5604879c2c90 con 0x560487a93f80 2024-05-27T07:56:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fae0cfd4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5604879b2830 con 0x560487915e40 2024-05-27T07:56:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fae0cfd4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5604879b6870 con 0x560487a99c50 2024-05-27T07:56:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadf57fa700 1 --2- 172.21.15.114:0/1900585970 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fade00b29f0 0x7fade00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadfcb5c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487915e40 0x560487a99710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadf77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a93f80 0x560487a94390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadfcb5c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487915e40 0x560487a99710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:44990/0 (socket says 172.21.15.114:44990) 2024-05-27T07:56:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadfcb5c700 1 -- 172.21.15.114:0/2644981099 learned_addr learned my addr 172.21.15.114:0/2644981099 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadfcb5c700 1 --2- 172.21.15.114:0/1900585970 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fade00b29f0 0x7fade00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadf57fa700 1 -- 172.21.15.114:0/1900585970 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 9466+0+0 (secure 0 0 0) 0x7fade80e27a0 con 0x7fadf81034d0 2024-05-27T07:56:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadf7fff700 1 --2- 172.21.15.114:0/2644981099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a99c50 0x560487a93a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadfcb5c700 1 -- 172.21.15.114:0/2644981099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a99c50 msgr2=0x560487a93a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadfcb5c700 1 --2- 172.21.15.114:0/2644981099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a99c50 0x560487a93a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadfcb5c700 1 -- 172.21.15.114:0/2644981099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a93f80 msgr2=0x560487a94390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadfcb5c700 1 --2- 172.21.15.114:0/2644981099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a93f80 0x560487a94390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadfcb5c700 1 -- 172.21.15.114:0/2644981099 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604879028d0 con 0x560487915e40 2024-05-27T07:56:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.881+0000 7fadf77fe700 1 --2- 172.21.15.114:0/2644981099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a93f80 0x560487a94390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:56:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.882+0000 7fadfcb5c700 1 --2- 172.21.15.114:0/2644981099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487915e40 0x560487a99710 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fadf000bb50 tx=0x7fadf000bb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a67de4ac8c6b1755 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.882+0000 7fadfcb5c700 1 --2- 172.21.15.114:0/1900585970 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fade00b29f0 0x7fade00b4f00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fadf0005000 tx=0x7fadf0028000 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:57.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.882+0000 7fadde7fc700 1 -- 172.21.15.114:0/2644981099 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf00050a0 con 0x560487915e40 2024-05-27T07:56:57.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.882+0000 7fadde7fc700 1 -- 172.21.15.114:0/2644981099 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadf00056f0 con 0x560487915e40 2024-05-27T07:56:57.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.882+0000 7fadde7fc700 1 -- 172.21.15.114:0/2644981099 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf001c440 con 0x560487915e40 2024-05-27T07:56:57.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.883+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/2644981099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487915e40 msgr2=0x560487a99710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:57.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.883+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/2644981099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487915e40 0x560487a99710 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fadf000bb50 tx=0x7fadf000bb80 comp rx=0 tx=0).stop 2024-05-27T07:56:57.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.883+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/2644981099 shutdown_connections 2024-05-27T07:56:57.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.883+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/2644981099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a93f80 0x560487a94390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.883+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/2644981099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487915e40 0x560487a99710 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.884+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/2644981099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a99c50 0x560487a93a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.884+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/2644981099 >> 172.21.15.114:0/2644981099 conn(0x560487970af0 msgr2=0x560487a8e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.884+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/2644981099 shutdown_connections 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.884+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/2644981099 wait complete. 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.884+0000 7fae0cfd4b80 1 -- start start 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.885+0000 7fae0cfd4b80 1 Processor -- start 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.885+0000 7fae0cfd4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487915e40 0x560487a9d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fae0cfd4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a93f80 0x560487a9d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fae0cfd4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487a99c50 0x560487a9de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fae0cfd4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5604878b8fa0 con 0x560487915e40 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fae0cfd4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5604879547c0 con 0x560487a99c50 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fae0cfd4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560487aa0650 con 0x560487a93f80 2024-05-27T07:56:57.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fae0cfd4b80 10 client.0 _ll_register_callbacks cb 0x56048794d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:57.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fadf77fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a93f80 0x560487a9d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fadf77fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a93f80 0x560487a9d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:39786/0 (socket says 172.21.15.114:39786) 2024-05-27T07:56:57.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fadf77fe700 1 -- 172.21.15.114:0/377072898 learned_addr learned my addr 172.21.15.114:0/377072898 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:57.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fadf77fe700 1 -- 172.21.15.114:0/377072898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487a99c50 msgr2=0x560487a9de80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:56:57.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fadf77fe700 1 --2- 172.21.15.114:0/377072898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487a99c50 0x560487a9de80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fadf7fff700 1 --2- 172.21.15.114:0/377072898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487915e40 0x560487a9d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fadf77fe700 1 -- 172.21.15.114:0/377072898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487915e40 msgr2=0x560487a9d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:57.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fadf77fe700 1 --2- 172.21.15.114:0/377072898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487915e40 0x560487a9d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.886+0000 7fadf77fe700 1 -- 172.21.15.114:0/377072898 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560487951690 con 0x560487a93f80 2024-05-27T07:56:57.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.887+0000 7fadf7fff700 1 --2- 172.21.15.114:0/377072898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487915e40 0x560487a9d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:56:57.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.887+0000 7fadd77fe700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:56:57.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.887+0000 7fadf77fe700 1 --2- 172.21.15.114:0/377072898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a93f80 0x560487a9d940 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fadec013b80 tx=0x7fadec013f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66a51bac28e57a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:57.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.887+0000 7fadd77fe700 1 -- 172.21.15.114:0/377072898 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadec020020 con 0x560487a93f80 2024-05-27T07:56:57.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.887+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/377072898 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadf00056f0 con 0x560487a93f80 2024-05-27T07:56:57.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.887+0000 7fadd77fe700 1 -- 172.21.15.114:0/377072898 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(150..150 src has 1..150) ==== 9466+0+0 (secure 0 0 0) 0x7fadec00fc70 con 0x560487a93f80 2024-05-27T07:56:57.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.888+0000 7fadd77fe700 1 -- 172.21.15.114:0/377072898 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fadec028050 con 0x560487a93f80 2024-05-27T07:56:57.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.888+0000 7fae0cfd4b80 10 client.28831 fetch_fsmap learned FSMap version 96 2024-05-27T07:56:57.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.888+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/377072898 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x560487aab200 con 0x560487a93f80 2024-05-27T07:56:57.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.888+0000 7fadd77fe700 1 -- 172.21.15.114:0/377072898 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadf00056f0 con 0x560487a93f80 2024-05-27T07:56:57.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.888+0000 7fadd77fe700 1 -- 172.21.15.114:0/377072898 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 96) ==== 4095+0+0 (secure 0 0 0) 0x7fadec01b070 con 0x560487a93f80 2024-05-27T07:56:57.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.888+0000 7fae0cfd4b80 10 client.28831 fetch_fsmap finished waiting for FSMap version 96 2024-05-27T07:56:57.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.888+0000 7fae0cfd4b80 10 client.28831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:57.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.888+0000 7fae0cfd4b80 20 client.28831 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:57.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.888+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/377072898 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560487aadcc0 0x560487ab0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.888+0000 7fae0cfd4b80 4 client.28831 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:56:57.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.888+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/377072898 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560487a5f9f0 con 0x560487aadcc0 2024-05-27T07:56:57.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.889+0000 7fadf7fff700 1 --2- 172.21.15.114:0/377072898 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560487aadcc0 0x560487ab0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.889+0000 7fadd77fe700 10 client.28831 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:57.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.890+0000 7fadf7fff700 1 --2- 172.21.15.114:0/377072898 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560487aadcc0 0x560487ab0130 crc :-1 s=READY pgs=462 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bf12b731cf41f35 server_cookie=87b835eefedb3ad8 in_seq=0 out_seq=0 2024-05-27T07:56:57.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fadd77fe700 1 -- 172.21.15.114:0/377072898 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560487a5f9f0 con 0x560487aadcc0 2024-05-27T07:56:57.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fadd77fe700 10 client.28831 handle_command_reply: tid=0 2024-05-27T07:56:57.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fae0cfd4b80 1 client.28831 shutdown 2024-05-27T07:56:57.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/377072898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a93f80 msgr2=0x560487a9d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:57.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/377072898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a93f80 0x560487a9d940 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fadec013b80 tx=0x7fadec013f40 comp rx=0 tx=0).stop 2024-05-27T07:56:57.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/377072898 shutdown_connections 2024-05-27T07:56:57.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/377072898 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560487aadcc0 0x560487ab0130 crc :-1 s=READY pgs=462 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/377072898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487a99c50 0x560487a9de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/377072898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487915e40 0x560487a9d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/377072898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a93f80 0x560487a9d940 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/377072898 >> 172.21.15.114:0/377072898 conn(0x560487970af0 msgr2=0x560487a92a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:57.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fadd77fe700 0 client.28831 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:57.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/377072898 shutdown_connections 2024-05-27T07:56:57.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/377072898 wait complete. 2024-05-27T07:56:57.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.896+0000 7fae0cfd4b80 20 client.28831 trim_cache size 0 max 16384 2024-05-27T07:56:57.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.917+0000 7fae0cfd4b80 1 Processor -- start 2024-05-27T07:56:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.917+0000 7fae0cfd4b80 1 -- start start 2024-05-27T07:56:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.917+0000 7fae0cfd4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487915e40 0x560487a93a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.917+0000 7fae0cfd4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a93f80 0x560487ab0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.917+0000 7fae0cfd4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 0x560487a9dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.917+0000 7fae0cfd4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560487a8e8f0 con 0x560487a99c50 2024-05-27T07:56:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.917+0000 7fae0cfd4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fadf81adfa0 con 0x560487915e40 2024-05-27T07:56:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.917+0000 7fae0cfd4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fadf81ae670 con 0x560487a93f80 2024-05-27T07:56:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.917+0000 7fadfcb5c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 0x560487a9dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.917+0000 7fadfcb5c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 0x560487a9dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50528/0 (socket says 172.21.15.114:50528) 2024-05-27T07:56:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.917+0000 7fadfcb5c700 1 -- 172.21.15.114:0/3675889316 learned_addr learned my addr 172.21.15.114:0/3675889316 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.917+0000 7fadf7fff700 1 --2- 172.21.15.114:0/3675889316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487915e40 0x560487a93a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.918+0000 7fadf77fe700 1 --2- 172.21.15.114:0/3675889316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a93f80 0x560487ab0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.918+0000 7fadfcb5c700 1 -- 172.21.15.114:0/3675889316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a93f80 msgr2=0x560487ab0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.918+0000 7fadfcb5c700 1 --2- 172.21.15.114:0/3675889316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a93f80 0x560487ab0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.918+0000 7fadfcb5c700 1 -- 172.21.15.114:0/3675889316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487915e40 msgr2=0x560487a93a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.918+0000 7fadfcb5c700 1 --2- 172.21.15.114:0/3675889316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487915e40 0x560487a93a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.918+0000 7fadfcb5c700 1 -- 172.21.15.114:0/3675889316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560487a9e4c0 con 0x560487a99c50 2024-05-27T07:56:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.918+0000 7fadf7fff700 1 --2- 172.21.15.114:0/3675889316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487915e40 0x560487a93a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:56:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.918+0000 7fadfcb5c700 1 --2- 172.21.15.114:0/3675889316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 0x560487a9dcc0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009610 tx=0x7fadf0009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b06b638bf34d29 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.918+0000 7fadddffb700 1 -- 172.21.15.114:0/3675889316 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf0036070 con 0x560487a99c50 2024-05-27T07:56:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.918+0000 7fadddffb700 1 -- 172.21.15.114:0/3675889316 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadf0009c90 con 0x560487a99c50 2024-05-27T07:56:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.918+0000 7fadddffb700 1 -- 172.21.15.114:0/3675889316 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf0005530 con 0x560487a99c50 2024-05-27T07:56:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.919+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/3675889316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 msgr2=0x560487a9dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.919+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/3675889316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 0x560487a9dcc0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009610 tx=0x7fadf0009960 comp rx=0 tx=0).stop 2024-05-27T07:56:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.919+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/3675889316 shutdown_connections 2024-05-27T07:56:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.919+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/3675889316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 0x560487a9dcc0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.919+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/3675889316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487915e40 0x560487a93a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.919+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/3675889316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487a93f80 0x560487ab0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.919+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/3675889316 >> 172.21.15.114:0/3675889316 conn(0x560487970af0 msgr2=0x560487a97010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.919+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/3675889316 shutdown_connections 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.919+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/3675889316 wait complete. 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.919+0000 7fae0cfd4b80 1 -- start start 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.920+0000 7fae0cfd4b80 1 Processor -- start 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.920+0000 7fae0cfd4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487915e40 0x560487aa58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.920+0000 7fae0cfd4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487a93f80 0x560487aa5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.920+0000 7fae0cfd4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 0x560487ab4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.920+0000 7fae0cfd4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fadf81ac1e0 con 0x560487a99c50 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.920+0000 7fae0cfd4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fadf81acf20 con 0x560487a93f80 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.920+0000 7fae0cfd4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fadf81ac850 con 0x560487915e40 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.920+0000 7fadfcb5c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 0x560487ab4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.920+0000 7fae0cfd4b80 10 client.0 _ll_register_callbacks cb 0x56048794d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:56:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.920+0000 7fadfcb5c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 0x560487ab4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50544/0 (socket says 172.21.15.114:50544) 2024-05-27T07:56:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.920+0000 7fadfcb5c700 1 -- 172.21.15.114:0/3511952830 learned_addr learned my addr 172.21.15.114:0/3511952830 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:56:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.920+0000 7fadf7fff700 1 --2- 172.21.15.114:0/3511952830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487915e40 0x560487aa58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.920+0000 7fadf77fe700 1 --2- 172.21.15.114:0/3511952830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487a93f80 0x560487aa5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fadfcb5c700 1 -- 172.21.15.114:0/3511952830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487915e40 msgr2=0x560487aa58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fadfcb5c700 1 --2- 172.21.15.114:0/3511952830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487915e40 0x560487aa58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fadfcb5c700 1 -- 172.21.15.114:0/3511952830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487a93f80 msgr2=0x560487aa5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:56:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fadfcb5c700 1 --2- 172.21.15.114:0/3511952830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487a93f80 0x560487aa5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fadfcb5c700 1 -- 172.21.15.114:0/3511952830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560487ab5090 con 0x560487a99c50 2024-05-27T07:56:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fadfcb5c700 1 --2- 172.21.15.114:0/3511952830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 0x560487ab4b00 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009e60 tx=0x7fadf0005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265ecef41b4f5930 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:56:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fadd77fe700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:56:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fadd77fe700 1 -- 172.21.15.114:0/3511952830 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf0042070 con 0x560487a99c50 2024-05-27T07:56:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fadd77fe700 1 -- 172.21.15.114:0/3511952830 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(150..150 src has 1..150) ==== 9466+0+0 (secure 0 0 0) 0x7fadf00077d0 con 0x560487a99c50 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/3511952830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadf0009c90 con 0x560487a99c50 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fadd77fe700 1 -- 172.21.15.114:0/3511952830 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fadf0036070 con 0x560487a99c50 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fae0cfd4b80 10 client.19350 fetch_fsmap learned FSMap version 96 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/3511952830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x560487b18010 con 0x560487a99c50 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.921+0000 7fadd77fe700 1 -- 172.21.15.114:0/3511952830 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadf0009c90 con 0x560487a99c50 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.922+0000 7fadd77fe700 1 -- 172.21.15.114:0/3511952830 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 96) ==== 4095+0+0 (secure 0 0 0) 0x7fadf0003ce0 con 0x560487a99c50 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.922+0000 7fae0cfd4b80 10 client.19350 fetch_fsmap finished waiting for FSMap version 96 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.922+0000 7fae0cfd4b80 10 client.19350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.922+0000 7fae0cfd4b80 20 client.19350 populate_metadata read hostname 'smithi114' 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.922+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/3511952830 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560487b18690 0x560487b1aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.922+0000 7fae0cfd4b80 4 client.19350 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.922+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/3511952830 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5604879c4be0 con 0x560487b18690 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.922+0000 7fadf7fff700 1 --2- 172.21.15.114:0/3511952830 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560487b18690 0x560487b1aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:56:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.922+0000 7fadd77fe700 10 client.19350 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:57.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.922+0000 7fadf7fff700 1 --2- 172.21.15.114:0/3511952830 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560487b18690 0x560487b1aab0 crc :-1 s=READY pgs=463 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c753bc16f7778099 server_cookie=764b11a49bbc08a2 in_seq=0 out_seq=0 2024-05-27T07:56:57.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fadd77fe700 1 -- 172.21.15.114:0/3511952830 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5604879c4be0 con 0x560487b18690 2024-05-27T07:56:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fadd77fe700 10 client.19350 handle_command_reply: tid=0 2024-05-27T07:56:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fae0cfd4b80 1 client.19350 shutdown 2024-05-27T07:56:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/3511952830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 msgr2=0x560487ab4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/3511952830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 0x560487ab4b00 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009e60 tx=0x7fadf0005680 comp rx=0 tx=0).stop 2024-05-27T07:56:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/3511952830 shutdown_connections 2024-05-27T07:56:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/3511952830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560487a99c50 0x560487ab4b00 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/3511952830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560487915e40 0x560487aa58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/3511952830 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560487b18690 0x560487b1aab0 crc :-1 s=READY pgs=463 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fae0cfd4b80 1 --2- 172.21.15.114:0/3511952830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560487a93f80 0x560487aa5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/3511952830 >> 172.21.15.114:0/3511952830 conn(0x560487970af0 msgr2=0x560487a9e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fadd77fe700 0 client.19350 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:56:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/3511952830 shutdown_connections 2024-05-27T07:56:57.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fae0cfd4b80 1 -- 172.21.15.114:0/3511952830 wait complete. 2024-05-27T07:56:57.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.928+0000 7fae0cfd4b80 20 client.19350 trim_cache size 0 max 16384 2024-05-27T07:56:57.931 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:56:57.931 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:56:57.931 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:56:57.931 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:56:57.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.929+0000 7fadfe5bf700 1 -- 172.21.15.114:0/1900585970 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fade00b29f0 msgr2=0x7fade00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:57.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.929+0000 7fadfe5bf700 1 --2- 172.21.15.114:0/1900585970 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fade00b29f0 0x7fade00b4f00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fadf0005000 tx=0x7fadf0028000 comp rx=0 tx=0).stop 2024-05-27T07:56:57.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.929+0000 7fadfe5bf700 1 -- 172.21.15.114:0/1900585970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadf81034d0 msgr2=0x7fadf8199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:56:57.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.929+0000 7fadfe5bf700 1 --2- 172.21.15.114:0/1900585970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadf81034d0 0x7fadf8199d70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fade800d980 tx=0x7fade800dc90 comp rx=0 tx=0).stop 2024-05-27T07:56:57.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.929+0000 7fadfe5bf700 1 -- 172.21.15.114:0/1900585970 shutdown_connections 2024-05-27T07:56:57.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.929+0000 7fadfe5bf700 1 --2- 172.21.15.114:0/1900585970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadf8108750 0x7fadf819e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.929+0000 7fadfe5bf700 1 --2- 172.21.15.114:0/1900585970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadf81034d0 0x7fadf8199d70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.929+0000 7fadfe5bf700 1 --2- 172.21.15.114:0/1900585970 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fade00b29f0 0x7fade00b4f00 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.929+0000 7fadfe5bf700 1 --2- 172.21.15.114:0/1900585970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf8107e00 0x7fadf819a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:56:57.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.929+0000 7fadfe5bf700 1 -- 172.21.15.114:0/1900585970 >> 172.21.15.114:0/1900585970 conn(0x7fadf80fef20 msgr2=0x7fadf8107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:56:57.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.929+0000 7fadfe5bf700 1 -- 172.21.15.114:0/1900585970 shutdown_connections 2024-05-27T07:56:57.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:56:57.929+0000 7fadfe5bf700 1 -- 172.21.15.114:0/1900585970 wait complete. 2024-05-27T07:56:58.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:57 smithi114 ceph-mon[32038]: pgmap v588: 129 pgs: 129 active+clean; 343 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 468 op/s 2024-05-27T07:56:58.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:57 smithi163 ceph-mon[36451]: pgmap v588: 129 pgs: 129 active+clean; 343 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 468 op/s 2024-05-27T07:56:58.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:57 smithi186 ceph-mon[36647]: pgmap v588: 129 pgs: 129 active+clean; 343 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 468 op/s 2024-05-27T07:56:58.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 47902 13.48 MB/sec execute 70 sec latency 180.428 ms 2024-05-27T07:56:59.344 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:56:59.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:56:59.604 INFO:tasks.workunit.client.0.smithi114.stdout: 10 48317 13.59 MB/sec execute 71 sec latency 248.068 ms 2024-05-27T07:56:59.606 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:56:59.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:56:59 smithi114 ceph-mon[32038]: pgmap v589: 129 pgs: 129 active+clean; 341 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 506 op/s 2024-05-27T07:57:00.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:56:59 smithi163 ceph-mon[36451]: pgmap v589: 129 pgs: 129 active+clean; 341 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 506 op/s 2024-05-27T07:57:00.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:56:59 smithi186 ceph-mon[36647]: pgmap v589: 129 pgs: 129 active+clean; 341 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 506 op/s 2024-05-27T07:57:00.605 INFO:tasks.workunit.client.0.smithi114.stdout: 10 48480 13.53 MB/sec execute 72 sec latency 604.833 ms 2024-05-27T07:57:01.605 INFO:tasks.workunit.client.0.smithi114.stdout: 10 48636 13.49 MB/sec execute 73 sec latency 730.795 ms 2024-05-27T07:57:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.860+0000 7efdd455a700 1 -- 172.21.15.114:0/887193005 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdcc11d920 msgr2=0x7efdcc11fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:01.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.860+0000 7efdd455a700 1 --2- 172.21.15.114:0/887193005 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdcc11d920 0x7efdcc11fd40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7efdc4009f90 tx=0x7efdc400b330 comp rx=0 tx=0).stop 2024-05-27T07:57:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.862+0000 7efdd455a700 1 -- 172.21.15.114:0/887193005 shutdown_connections 2024-05-27T07:57:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.862+0000 7efdd455a700 1 --2- 172.21.15.114:0/887193005 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdcc11d920 0x7efdcc11fd40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.862+0000 7efdd455a700 1 --2- 172.21.15.114:0/887193005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdcc10a920 0x7efdcc10ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.862+0000 7efdd455a700 1 --2- 172.21.15.114:0/887193005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdcc10b270 0x7efdcc11d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.862+0000 7efdd455a700 1 -- 172.21.15.114:0/887193005 >> 172.21.15.114:0/887193005 conn(0x7efdcc06f4c0 msgr2=0x7efdcc06f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.866+0000 7efdd455a700 1 -- 172.21.15.114:0/887193005 shutdown_connections 2024-05-27T07:57:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.867+0000 7efdd455a700 1 -- 172.21.15.114:0/887193005 wait complete. 2024-05-27T07:57:01.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.868+0000 7efdd455a700 1 Processor -- start 2024-05-27T07:57:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.872+0000 7efdd455a700 1 -- start start 2024-05-27T07:57:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.872+0000 7efdd455a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdcc10a920 0x7efdcc1af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.872+0000 7efdd455a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdcc10b270 0x7efdcc1afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.872+0000 7efdd455a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdcc11d920 0x7efdcc1a9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.872+0000 7efdd455a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc1232d0 con 0x7efdcc10b270 2024-05-27T07:57:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.872+0000 7efdd455a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc123150 con 0x7efdcc10a920 2024-05-27T07:57:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.872+0000 7efdd455a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc123450 con 0x7efdcc11d920 2024-05-27T07:57:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.874+0000 7efdd2af7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdcc11d920 0x7efdcc1a9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.874+0000 7efdd1af5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdcc10b270 0x7efdcc1afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.874+0000 7efdd1af5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdcc10b270 0x7efdcc1afd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50572/0 (socket says 172.21.15.114:50572) 2024-05-27T07:57:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.874+0000 7efdd1af5700 1 -- 172.21.15.114:0/1843771033 learned_addr learned my addr 172.21.15.114:0/1843771033 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.874+0000 7efdd22f6700 1 --2- 172.21.15.114:0/1843771033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdcc10a920 0x7efdcc1af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.877+0000 7efdd22f6700 1 -- 172.21.15.114:0/1843771033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdcc11d920 msgr2=0x7efdcc1a9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:01.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.877+0000 7efdd22f6700 1 --2- 172.21.15.114:0/1843771033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdcc11d920 0x7efdcc1a9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:01.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.877+0000 7efdd22f6700 1 -- 172.21.15.114:0/1843771033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdcc10b270 msgr2=0x7efdcc1afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:01.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.877+0000 7efdd22f6700 1 --2- 172.21.15.114:0/1843771033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdcc10b270 0x7efdcc1afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:01.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.877+0000 7efdd22f6700 1 -- 172.21.15.114:0/1843771033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdcc1aa0e0 con 0x7efdcc10a920 2024-05-27T07:57:01.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.884+0000 7efdd22f6700 1 --2- 172.21.15.114:0/1843771033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdcc10a920 0x7efdcc1af810 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7efdc800d750 tx=0x7efdc800da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:01.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.889+0000 7efdc37fe700 1 -- 172.21.15.114:0/1843771033 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc8016070 con 0x7efdcc10a920 2024-05-27T07:57:01.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.889+0000 7efdc37fe700 1 -- 172.21.15.114:0/1843771033 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdc8005020 con 0x7efdcc10a920 2024-05-27T07:57:01.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.889+0000 7efdc37fe700 1 -- 172.21.15.114:0/1843771033 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc800fa10 con 0x7efdcc10a920 2024-05-27T07:57:01.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.889+0000 7efdd455a700 1 -- 172.21.15.114:0/1843771033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdcc1aa3d0 con 0x7efdcc10a920 2024-05-27T07:57:01.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.889+0000 7efdd455a700 1 -- 172.21.15.114:0/1843771033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdcc1b66a0 con 0x7efdcc10a920 2024-05-27T07:57:01.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.895+0000 7efdd455a700 1 -- 172.21.15.114:0/1843771033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdb0005330 con 0x7efdcc10a920 2024-05-27T07:57:01.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.895+0000 7efdc37fe700 1 -- 172.21.15.114:0/1843771033 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7efdc800fbb0 con 0x7efdcc10a920 2024-05-27T07:57:01.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.896+0000 7efdc37fe700 1 --2- 172.21.15.114:0/1843771033 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efdb80b2c00 0x7efdb80b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:01.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.896+0000 7efdc37fe700 1 -- 172.21.15.114:0/1843771033 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 9466+0+0 (secure 0 0 0) 0x7efdc80e2270 con 0x7efdcc10a920 2024-05-27T07:57:01.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.897+0000 7efdd1af5700 1 --2- 172.21.15.114:0/1843771033 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efdb80b2c00 0x7efdb80b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:01.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.898+0000 7efdd1af5700 1 --2- 172.21.15.114:0/1843771033 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efdb80b2c00 0x7efdb80b5110 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7efdcc06ff50 tx=0x7efdbc009310 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:01.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:01.904+0000 7efdc37fe700 1 -- 172.21.15.114:0/1843771033 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7efdc80a9840 con 0x7efdcc10a920 2024-05-27T07:57:02.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-mon[32038]: pgmap v590: 129 pgs: 129 active+clean; 342 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 461 op/s 2024-05-27T07:57:02.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe11ed41700 time 2024-05-27T07:57:01.672321+0000 2024-05-27T07:57:02.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:57:02.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:02.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe131586ba6] 2024-05-27T07:57:02.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe131586dd1] 2024-05-27T07:57:02.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56137e38323d] 2024-05-27T07:57:02.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56137e36e971] 2024-05-27T07:57:02.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 5: (SnapRealm::check_cache() const+0x13e) [0x56137e36cfa0] 2024-05-27T07:57:02.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56137e36d651] 2024-05-27T07:57:02.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56137e302347] 2024-05-27T07:57:02.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56137dffe87e] 2024-05-27T07:57:02.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56137e003e94] 2024-05-27T07:57:02.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56137e00e969] 2024-05-27T07:57:02.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 11: (MDLog::_replay_thread()+0x1f38) [0x56137e3a748c] 2024-05-27T07:57:02.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56137e3b09cb] 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 13: (Thread::entry_wrapper()+0x43) [0x7fe13155572d] 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe131555749] 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe1300f91ca] 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 16: clone() 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:01.675+0000 7fe11ed41700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe11ed41700 time 2024-05-27T07:57:01.672321+0000 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe131586ba6] 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe131586dd1] 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56137e38323d] 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56137e36e971] 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 5: (SnapRealm::check_cache() const+0x13e) [0x56137e36cfa0] 2024-05-27T07:57:02.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56137e36d651] 2024-05-27T07:57:02.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56137e302347] 2024-05-27T07:57:02.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56137dffe87e] 2024-05-27T07:57:02.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56137e003e94] 2024-05-27T07:57:02.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56137e00e969] 2024-05-27T07:57:02.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 11: (MDLog::_replay_thread()+0x1f38) [0x56137e3a748c] 2024-05-27T07:57:02.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56137e3b09cb] 2024-05-27T07:57:02.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 13: (Thread::entry_wrapper()+0x43) [0x7fe13155572d] 2024-05-27T07:57:02.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe131555749] 2024-05-27T07:57:02.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe1300f91ca] 2024-05-27T07:57:02.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 16: clone() 2024-05-27T07:57:02.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:02.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: *** Caught signal (Aborted) ** 2024-05-27T07:57:02.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: in thread 7fe11ed41700 thread_name:md_log_replay 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe130103d20] 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2: gsignal() 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 3: abort() 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe131586ced] 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe131586dd1] 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56137e38323d] 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56137e36e971] 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 8: (SnapRealm::check_cache() const+0x13e) [0x56137e36cfa0] 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56137e36d651] 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56137e302347] 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56137dffe87e] 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56137e003e94] 2024-05-27T07:57:02.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56137e00e969] 2024-05-27T07:57:02.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 14: (MDLog::_replay_thread()+0x1f38) [0x56137e3a748c] 2024-05-27T07:57:02.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56137e3b09cb] 2024-05-27T07:57:02.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 16: (Thread::entry_wrapper()+0x43) [0x7fe13155572d] 2024-05-27T07:57:02.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe131555749] 2024-05-27T07:57:02.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe1300f91ca] 2024-05-27T07:57:02.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 19: clone() 2024-05-27T07:57:02.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:01.679+0000 7fe11ed41700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:57:02.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: in thread 7fe11ed41700 thread_name:md_log_replay 2024-05-27T07:57:02.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:02.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:02.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe130103d20] 2024-05-27T07:57:02.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2: gsignal() 2024-05-27T07:57:02.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 3: abort() 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe131586ced] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe131586dd1] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56137e38323d] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56137e36e971] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 8: (SnapRealm::check_cache() const+0x13e) [0x56137e36cfa0] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56137e36d651] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56137e302347] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56137dffe87e] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56137e003e94] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56137e00e969] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 14: (MDLog::_replay_thread()+0x1f38) [0x56137e3a748c] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56137e3b09cb] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 16: (Thread::entry_wrapper()+0x43) [0x7fe13155572d] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe131555749] 2024-05-27T07:57:02.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe1300f91ca] 2024-05-27T07:57:02.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 19: clone() 2024-05-27T07:57:02.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:57:02.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:02.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: -1> 2024-05-27T07:57:01.675+0000 7fe11ed41700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe11ed41700 time 2024-05-27T07:57:01.672321+0000 2024-05-27T07:57:02.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:57:02.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:02.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:02.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe131586ba6] 2024-05-27T07:57:02.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe131586dd1] 2024-05-27T07:57:02.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56137e38323d] 2024-05-27T07:57:02.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56137e36e971] 2024-05-27T07:57:02.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 5: (SnapRealm::check_cache() const+0x13e) [0x56137e36cfa0] 2024-05-27T07:57:02.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56137e36d651] 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56137e302347] 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56137dffe87e] 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56137e003e94] 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56137e00e969] 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 11: (MDLog::_replay_thread()+0x1f38) [0x56137e3a748c] 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56137e3b09cb] 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 13: (Thread::entry_wrapper()+0x43) [0x7fe13155572d] 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe131555749] 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe1300f91ca] 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 16: clone() 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 0> 2024-05-27T07:57:01.679+0000 7fe11ed41700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: in thread 7fe11ed41700 thread_name:md_log_replay 2024-05-27T07:57:02.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:02.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:02.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe130103d20] 2024-05-27T07:57:02.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2: gsignal() 2024-05-27T07:57:02.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 3: abort() 2024-05-27T07:57:02.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe131586ced] 2024-05-27T07:57:02.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe131586dd1] 2024-05-27T07:57:02.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56137e38323d] 2024-05-27T07:57:02.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56137e36e971] 2024-05-27T07:57:02.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 8: (SnapRealm::check_cache() const+0x13e) [0x56137e36cfa0] 2024-05-27T07:57:02.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56137e36d651] 2024-05-27T07:57:02.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56137e302347] 2024-05-27T07:57:02.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56137dffe87e] 2024-05-27T07:57:02.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56137e003e94] 2024-05-27T07:57:02.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56137e00e969] 2024-05-27T07:57:02.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 14: (MDLog::_replay_thread()+0x1f38) [0x56137e3a748c] 2024-05-27T07:57:02.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56137e3b09cb] 2024-05-27T07:57:02.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 16: (Thread::entry_wrapper()+0x43) [0x7fe13155572d] 2024-05-27T07:57:02.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe131555749] 2024-05-27T07:57:02.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe1300f91ca] 2024-05-27T07:57:02.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 19: clone() 2024-05-27T07:57:02.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:57:02.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:02.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: -1> 2024-05-27T07:57:01.675+0000 7fe11ed41700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe11ed41700 time 2024-05-27T07:57:01.672321+0000 2024-05-27T07:57:02.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:57:02.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:02.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:02.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe131586ba6] 2024-05-27T07:57:02.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe131586dd1] 2024-05-27T07:57:02.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56137e38323d] 2024-05-27T07:57:02.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56137e36e971] 2024-05-27T07:57:02.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 5: (SnapRealm::check_cache() const+0x13e) [0x56137e36cfa0] 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56137e36d651] 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56137e302347] 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56137dffe87e] 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56137e003e94] 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56137e00e969] 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 11: (MDLog::_replay_thread()+0x1f38) [0x56137e3a748c] 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56137e3b09cb] 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 13: (Thread::entry_wrapper()+0x43) [0x7fe13155572d] 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe131555749] 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe1300f91ca] 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 16: clone() 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 0> 2024-05-27T07:57:01.679+0000 7fe11ed41700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:57:02.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: in thread 7fe11ed41700 thread_name:md_log_replay 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe130103d20] 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2: gsignal() 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 3: abort() 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe131586ced] 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe131586dd1] 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56137e38323d] 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56137e36e971] 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 8: (SnapRealm::check_cache() const+0x13e) [0x56137e36cfa0] 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56137e36d651] 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56137e302347] 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56137dffe87e] 2024-05-27T07:57:02.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56137e003e94] 2024-05-27T07:57:02.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56137e00e969] 2024-05-27T07:57:02.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 14: (MDLog::_replay_thread()+0x1f38) [0x56137e3a748c] 2024-05-27T07:57:02.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56137e3b09cb] 2024-05-27T07:57:02.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 16: (Thread::entry_wrapper()+0x43) [0x7fe13155572d] 2024-05-27T07:57:02.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe131555749] 2024-05-27T07:57:02.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe1300f91ca] 2024-05-27T07:57:02.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 19: clone() 2024-05-27T07:57:02.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:57:02.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[127853]: 2024-05-27T07:57:02.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:01 smithi114 podman[128257]: 2024-05-27 07:57:01.833367785 +0000 UTC m=+0.066012913 container died b8604cfea005a835e2a9b99f75ad906ce13bd7d2fbc40a1ead3f9b3181090b62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5) 2024-05-27T07:57:02.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:01 smithi163 ceph-mon[36451]: pgmap v590: 129 pgs: 129 active+clean; 342 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 461 op/s 2024-05-27T07:57:02.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:01 smithi186 ceph-mon[36647]: pgmap v590: 129 pgs: 129 active+clean; 342 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 461 op/s 2024-05-27T07:57:02.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.405+0000 7efdd455a700 1 -- 172.21.15.114:0/1843771033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efdb00058f0 con 0x7efdcc10a920 2024-05-27T07:57:02.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.406+0000 7efdc37fe700 1 -- 172.21.15.114:0/1843771033 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+6036 (secure 0 0 0) 0x7efdc80a9140 con 0x7efdcc10a920 2024-05-27T07:57:02.409 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:57:02.409 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":96,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:56:43.753407+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_19317":{"gid":19317,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6841/1351506174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":1351506174},{"type":"v1","addr":"172.21.15.114:6841","nonce":1351506174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:57:02.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.413+0000 7efdd455a700 1 -- 172.21.15.114:0/1843771033 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efdb80b2c00 msgr2=0x7efdb80b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:02.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.413+0000 7efdd455a700 1 --2- 172.21.15.114:0/1843771033 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efdb80b2c00 0x7efdb80b5110 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7efdcc06ff50 tx=0x7efdbc009310 comp rx=0 tx=0).stop 2024-05-27T07:57:02.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.413+0000 7efdd455a700 1 -- 172.21.15.114:0/1843771033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdcc10a920 msgr2=0x7efdcc1af810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:02.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.413+0000 7efdd455a700 1 --2- 172.21.15.114:0/1843771033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdcc10a920 0x7efdcc1af810 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7efdc800d750 tx=0x7efdc800da60 comp rx=0 tx=0).stop 2024-05-27T07:57:02.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.413+0000 7efdd455a700 1 -- 172.21.15.114:0/1843771033 shutdown_connections 2024-05-27T07:57:02.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.413+0000 7efdd455a700 1 --2- 172.21.15.114:0/1843771033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdcc11d920 0x7efdcc1a9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:02.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.413+0000 7efdd455a700 1 --2- 172.21.15.114:0/1843771033 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efdb80b2c00 0x7efdb80b5110 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:02.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.413+0000 7efdd455a700 1 --2- 172.21.15.114:0/1843771033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdcc10a920 0x7efdcc1af810 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:02.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.413+0000 7efdd455a700 1 --2- 172.21.15.114:0/1843771033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdcc10b270 0x7efdcc1afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:02.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.413+0000 7efdd455a700 1 -- 172.21.15.114:0/1843771033 >> 172.21.15.114:0/1843771033 conn(0x7efdcc06f4c0 msgr2=0x7efdcc11b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:02.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.416+0000 7efdd455a700 1 -- 172.21.15.114:0/1843771033 shutdown_connections 2024-05-27T07:57:02.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:02.416+0000 7efdd455a700 1 -- 172.21.15.114:0/1843771033 wait complete. 2024-05-27T07:57:02.421 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 96 2024-05-27T07:57:02.605 INFO:tasks.workunit.client.0.smithi114.stdout: 10 48807 13.41 MB/sec execute 74 sec latency 881.025 ms 2024-05-27T07:57:03.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:02 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1843771033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:57:03.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:02 smithi114 podman[128257]: 2024-05-27 07:57:02.937486759 +0000 UTC m=+1.170131881 container remove b8604cfea005a835e2a9b99f75ad906ce13bd7d2fbc40a1ead3f9b3181090b62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:57:03.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:02 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:57:03.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:02 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1843771033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:57:03.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:02 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1843771033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:57:03.605 INFO:tasks.workunit.client.0.smithi114.stdout: 10 48813 13.24 MB/sec execute 75 sec latency 950.717 ms 2024-05-27T07:57:03.896 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:03 smithi114 ceph-mon[32038]: pgmap v591: 129 pgs: 129 active+clean; 352 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.1 MiB/s wr, 545 op/s 2024-05-27T07:57:03.896 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:03 smithi114 ceph-mon[32038]: osdmap e151: 12 total, 12 up, 12 in 2024-05-27T07:57:03.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:57:04.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:03 smithi163 ceph-mon[36451]: pgmap v591: 129 pgs: 129 active+clean; 352 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.1 MiB/s wr, 545 op/s 2024-05-27T07:57:04.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:03 smithi163 ceph-mon[36451]: osdmap e151: 12 total, 12 up, 12 in 2024-05-27T07:57:04.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:03 smithi186 ceph-mon[36647]: pgmap v591: 129 pgs: 129 active+clean; 352 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.1 MiB/s wr, 545 op/s 2024-05-27T07:57:04.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:03 smithi186 ceph-mon[36647]: osdmap e151: 12 total, 12 up, 12 in 2024-05-27T07:57:04.465 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:57:04.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 48818 13.08 MB/sec execute 76 sec latency 1395.475 ms 2024-05-27T07:57:04.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.766+0000 7f5f62784640 1 -- 172.21.15.114:0/2613495563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f5c109030 msgr2=0x7f5f5c102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:04.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.767+0000 7f5f62784640 1 --2- 172.21.15.114:0/2613495563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f5c109030 0x7f5f5c102c70 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c01a910 tx=0x7f5f4c03b470 comp rx=0 tx=0).stop 2024-05-27T07:57:04.775 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.773+0000 7f5f62784640 1 -- 172.21.15.114:0/2613495563 shutdown_connections 2024-05-27T07:57:04.775 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.773+0000 7f5f62784640 1 --2- 172.21.15.114:0/2613495563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5c1032e0 0x7f5f5c10f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.775 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.773+0000 7f5f62784640 1 --2- 172.21.15.114:0/2613495563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f5c109030 0x7f5f5c102c70 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.775 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.773+0000 7f5f62784640 1 --2- 172.21.15.114:0/2613495563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f5c108b90 0x7f5f5c102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.775 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.773+0000 7f5f62784640 1 -- 172.21.15.114:0/2613495563 >> 172.21.15.114:0/2613495563 conn(0x7f5f5c0782d0 msgr2=0x7f5f5c078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:04.775 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.774+0000 7f5f62784640 1 -- 172.21.15.114:0/2613495563 shutdown_connections 2024-05-27T07:57:04.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.776+0000 7f5f62784640 1 -- 172.21.15.114:0/2613495563 wait complete. 2024-05-27T07:57:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.777+0000 7f5f62784640 1 Processor -- start 2024-05-27T07:57:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.778+0000 7f5f62784640 1 -- start start 2024-05-27T07:57:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.778+0000 7f5f62784640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5c1032e0 0x7f5f5c0747f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.778+0000 7f5f62784640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f5c108b90 0x7f5f5c06f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.778+0000 7f5f62784640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f5c109030 0x7f5f5c06fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.779+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5c1032e0 0x7f5f5c0747f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.779+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5c1032e0 0x7f5f5c0747f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:53608/0 (socket says 172.21.15.114:53608) 2024-05-27T07:57:04.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.779+0000 7f5f60d21640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f5c109030 0x7f5f5c06fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.779+0000 7f5f60d21640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f5c109030 0x7f5f5c06fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:54602/0 (socket says 172.21.15.114:54602) 2024-05-27T07:57:04.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.779+0000 7f5f5bfff640 1 -- 172.21.15.114:0/3488042198 learned_addr learned my addr 172.21.15.114:0/3488042198 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:04.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.779+0000 7f5f62784640 1 -- 172.21.15.114:0/3488042198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5c06dd80 con 0x7f5f5c108b90 2024-05-27T07:57:04.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.779+0000 7f5f62784640 1 -- 172.21.15.114:0/3488042198 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5c06dc00 con 0x7f5f5c109030 2024-05-27T07:57:04.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.779+0000 7f5f62784640 1 -- 172.21.15.114:0/3488042198 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5c06df00 con 0x7f5f5c1032e0 2024-05-27T07:57:04.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.780+0000 7f5f60d21640 1 -- 172.21.15.114:0/3488042198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5c1032e0 msgr2=0x7f5f5c0747f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:04.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.780+0000 7f5f60d21640 1 --2- 172.21.15.114:0/3488042198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5c1032e0 0x7f5f5c0747f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.780+0000 7f5f60d21640 1 -- 172.21.15.114:0/3488042198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f5c108b90 msgr2=0x7f5f5c06f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:04.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.780+0000 7f5f60d21640 1 --2- 172.21.15.114:0/3488042198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f5c108b90 0x7f5f5c06f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.780+0000 7f5f60d21640 1 -- 172.21.15.114:0/3488042198 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f5c0705f0 con 0x7f5f5c109030 2024-05-27T07:57:04.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.780+0000 7f5f60d21640 1 --2- 172.21.15.114:0/3488042198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f5c109030 0x7f5f5c06fe50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000d680 tx=0x7f5f5000db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.781+0000 7f5f597fa640 1 -- 172.21.15.114:0/3488042198 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f50017070 con 0x7f5f5c109030 2024-05-27T07:57:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.781+0000 7f5f62784640 1 -- 172.21.15.114:0/3488042198 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f5c1a4b40 con 0x7f5f5c109030 2024-05-27T07:57:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.781+0000 7f5f62784640 1 -- 172.21.15.114:0/3488042198 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f5c1a5060 con 0x7f5f5c109030 2024-05-27T07:57:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.785+0000 7f5f597fa640 1 -- 172.21.15.114:0/3488042198 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f50009820 con 0x7f5f5c109030 2024-05-27T07:57:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.785+0000 7f5f597fa640 1 -- 172.21.15.114:0/3488042198 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f50004ce0 con 0x7f5f5c109030 2024-05-27T07:57:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.791+0000 7f5f69636740 1 Processor -- start 2024-05-27T07:57:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.791+0000 7f5f69636740 1 -- start start 2024-05-27T07:57:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.791+0000 7f5f597fa640 1 -- 172.21.15.114:0/3488042198 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f5f50005220 con 0x7f5f5c109030 2024-05-27T07:57:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.791+0000 7f5f69636740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7c21b0 0x56078bb633d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.791+0000 7f5f69636740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7992f0 0x56078bb63910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.791+0000 7f5f69636740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7a14d0 0x56078bb63e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.791+0000 7f5f69636740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56078bad5c30 con 0x56078b7c21b0 2024-05-27T07:57:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.791+0000 7f5f69636740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56078bac57d0 con 0x56078b7a14d0 2024-05-27T07:57:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.791+0000 7f5f69636740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56078bac9810 con 0x56078b7992f0 2024-05-27T07:57:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.793+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7992f0 0x56078bb63910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.793+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7992f0 0x56078bb63910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:53616/0 (socket says 172.21.15.114:53616) 2024-05-27T07:57:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.793+0000 7f5f5bfff640 1 -- 172.21.15.114:0/1996977271 learned_addr learned my addr 172.21.15.114:0/1996977271 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:04.795 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:04 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:57:04.796 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:04 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Consumed 3.402s CPU time. 2024-05-27T07:57:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.794+0000 7f5f5bfff640 1 -- 172.21.15.114:0/1996977271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7a14d0 msgr2=0x56078bb63e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:57:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.794+0000 7f5f5bfff640 1 --2- 172.21.15.114:0/1996977271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7a14d0 0x56078bb63e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.794+0000 7f5f5bfff640 1 -- 172.21.15.114:0/1996977271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7c21b0 msgr2=0x56078bb633d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.794+0000 7f5f5bfff640 1 --2- 172.21.15.114:0/1996977271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7c21b0 0x56078bb633d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.794+0000 7f5f5bfff640 1 -- 172.21.15.114:0/1996977271 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56078b7da210 con 0x56078b7992f0 2024-05-27T07:57:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.794+0000 7f5f5bfff640 1 --2- 172.21.15.114:0/1996977271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7992f0 0x56078bb63910 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5f54007f20 tx=0x7f5f5400f850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b0e79363a3857d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.795+0000 7f5f597fa640 1 --2- 172.21.15.114:0/3488042198 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5f440b2b10 0x7f5f440b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.795+0000 7f5f597fa640 1 -- 172.21.15.114:0/3488042198 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7f5f500e2190 con 0x7f5f5c109030 2024-05-27T07:57:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.795+0000 7f5f427fc640 1 -- 172.21.15.114:0/1996977271 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f54010900 con 0x56078b7992f0 2024-05-27T07:57:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.796+0000 7f5f427fc640 1 -- 172.21.15.114:0/1996977271 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f540183f0 con 0x56078b7992f0 2024-05-27T07:57:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.796+0000 7f5f5b7fe640 1 --2- 172.21.15.114:0/3488042198 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5f440b2b10 0x7f5f440b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.797+0000 7f5f69636740 1 -- 172.21.15.114:0/1996977271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7992f0 msgr2=0x56078bb63910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.797+0000 7f5f69636740 1 --2- 172.21.15.114:0/1996977271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7992f0 0x56078bb63910 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5f54007f20 tx=0x7f5f5400f850 comp rx=0 tx=0).stop 2024-05-27T07:57:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.797+0000 7f5f5b7fe640 1 --2- 172.21.15.114:0/3488042198 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5f440b2b10 0x7f5f440b5020 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x56078bb60540 tx=0x7f5f4c000bc0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.800+0000 7f5f69636740 1 -- 172.21.15.114:0/1996977271 shutdown_connections 2024-05-27T07:57:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.800+0000 7f5f69636740 1 --2- 172.21.15.114:0/1996977271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7a14d0 0x56078bb63e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.800+0000 7f5f69636740 1 --2- 172.21.15.114:0/1996977271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7992f0 0x56078bb63910 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.800+0000 7f5f69636740 1 --2- 172.21.15.114:0/1996977271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7c21b0 0x56078bb633d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.800+0000 7f5f69636740 1 -- 172.21.15.114:0/1996977271 >> 172.21.15.114:0/1996977271 conn(0x56078b7b6300 msgr2=0x56078bb5fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.805+0000 7f5f69636740 1 -- 172.21.15.114:0/1996977271 shutdown_connections 2024-05-27T07:57:04.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.809+0000 7f5f69636740 1 -- 172.21.15.114:0/1996977271 wait complete. 2024-05-27T07:57:04.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.809+0000 7f5f69636740 1 -- start start 2024-05-27T07:57:04.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.810+0000 7f5f69636740 1 Processor -- start 2024-05-27T07:57:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.810+0000 7f5f69636740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7992f0 0x56078bb6d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.810+0000 7f5f69636740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7a14d0 0x56078bb6de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.810+0000 7f5f69636740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7c21b0 0x56078bb79960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.810+0000 7f5f69636740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56078b7ab790 con 0x56078b7992f0 2024-05-27T07:57:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.810+0000 7f5f69636740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56078ba81890 con 0x56078b7a14d0 2024-05-27T07:57:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.810+0000 7f5f69636740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56078bb70550 con 0x56078b7c21b0 2024-05-27T07:57:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.810+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7992f0 0x56078bb6d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.810+0000 7f5f60d21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7c21b0 0x56078bb79960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.811+0000 7f5f60d21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7c21b0 0x56078bb79960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:53624/0 (socket says 172.21.15.114:53624) 2024-05-27T07:57:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.811+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7992f0 0x56078bb6d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33762/0 (socket says 172.21.15.114:33762) 2024-05-27T07:57:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.811+0000 7f5f60d21640 1 -- 172.21.15.114:0/7080721 learned_addr learned my addr 172.21.15.114:0/7080721 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.811+0000 7f5f69636740 10 client.0 _ll_register_callbacks cb 0x56078b7d9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.811+0000 7f5f5b7fe640 1 --2- 172.21.15.114:0/7080721 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7a14d0 0x56078bb6de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.812+0000 7f5f60d21640 1 -- 172.21.15.114:0/7080721 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7a14d0 msgr2=0x56078bb6de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.812+0000 7f5f60d21640 1 --2- 172.21.15.114:0/7080721 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7a14d0 0x56078bb6de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.812+0000 7f5f60d21640 1 -- 172.21.15.114:0/7080721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7992f0 msgr2=0x56078bb6d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.812+0000 7f5f60d21640 1 --2- 172.21.15.114:0/7080721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7992f0 0x56078bb6d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.812+0000 7f5f60d21640 1 -- 172.21.15.114:0/7080721 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56078b8a6620 con 0x56078b7c21b0 2024-05-27T07:57:04.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.819+0000 7f5f60d21640 1 --2- 172.21.15.114:0/7080721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7c21b0 0x56078bb79960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5f500a8f40 tx=0x7f5f50004f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17dab4e64fca9840 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:04.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.819+0000 7f5f3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:57:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.821+0000 7f5f3b7fe640 1 -- 172.21.15.114:0/7080721 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f50020040 con 0x56078b7c21b0 2024-05-27T07:57:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.822+0000 7f5f69636740 1 -- 172.21.15.114:0/7080721 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f540183f0 con 0x56078b7c21b0 2024-05-27T07:57:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.822+0000 7f5f3b7fe640 1 -- 172.21.15.114:0/7080721 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7f5f500dfb70 con 0x56078b7c21b0 2024-05-27T07:57:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.824+0000 7f5f3b7fe640 1 -- 172.21.15.114:0/7080721 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f5f500df8b0 con 0x56078b7c21b0 2024-05-27T07:57:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.828+0000 7f5f69636740 10 client.28855 fetch_fsmap learned FSMap version 96 2024-05-27T07:57:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.828+0000 7f5f69636740 1 -- 172.21.15.114:0/7080721 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x56078bb7c540 con 0x56078b7c21b0 2024-05-27T07:57:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.828+0000 7f5f3b7fe640 1 -- 172.21.15.114:0/7080721 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f500a8fb0 con 0x56078b7c21b0 2024-05-27T07:57:04.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.830+0000 7f5f3b7fe640 1 -- 172.21.15.114:0/7080721 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 96) ==== 4095+0+0 (secure 0 0 0) 0x7f5f500a4b20 con 0x56078b7c21b0 2024-05-27T07:57:04.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.830+0000 7f5f69636740 10 client.28855 fetch_fsmap finished waiting for FSMap version 96 2024-05-27T07:57:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.830+0000 7f5f69636740 10 client.28855 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.830+0000 7f5f69636740 20 client.28855 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.830+0000 7f5f69636740 1 --2- 172.21.15.114:0/7080721 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56078b780d80 0x56078bb7fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.830+0000 7f5f69636740 4 client.28855 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:57:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.830+0000 7f5f69636740 1 -- 172.21.15.114:0/7080721 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5f540183f0 con 0x56078b780d80 2024-05-27T07:57:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.836+0000 7f5f5bfff640 1 --2- 172.21.15.114:0/7080721 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56078b780d80 0x56078bb7fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.852+0000 7f5f5bfff640 1 --2- 172.21.15.114:0/7080721 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56078b780d80 0x56078bb7fec0 crc :-1 s=READY pgs=464 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6571529e500a2a6 server_cookie=8d4136d933ed4229 in_seq=0 out_seq=0 2024-05-27T07:57:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.853+0000 7f5f3b7fe640 10 client.28855 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:04.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.856+0000 7f5f3b7fe640 1 -- 172.21.15.114:0/7080721 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f5f540183f0 con 0x56078b780d80 2024-05-27T07:57:04.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.856+0000 7f5f3b7fe640 10 client.28855 handle_command_reply: tid=0 2024-05-27T07:57:04.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.858+0000 7f5f69636740 1 client.28855 shutdown 2024-05-27T07:57:04.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.860+0000 7f5f69636740 1 -- 172.21.15.114:0/7080721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7c21b0 msgr2=0x56078bb79960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:04.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.860+0000 7f5f69636740 1 --2- 172.21.15.114:0/7080721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7c21b0 0x56078bb79960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5f500a8f40 tx=0x7f5f50004f80 comp rx=0 tx=0).stop 2024-05-27T07:57:04.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.860+0000 7f5f69636740 1 -- 172.21.15.114:0/7080721 shutdown_connections 2024-05-27T07:57:04.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.860+0000 7f5f69636740 1 --2- 172.21.15.114:0/7080721 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56078b780d80 0x56078bb7fec0 crc :-1 s=READY pgs=464 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.860+0000 7f5f69636740 1 --2- 172.21.15.114:0/7080721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7c21b0 0x56078bb79960 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.860+0000 7f5f69636740 1 --2- 172.21.15.114:0/7080721 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7a14d0 0x56078bb6de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.860+0000 7f5f69636740 1 --2- 172.21.15.114:0/7080721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7992f0 0x56078bb6d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.861+0000 7f5f69636740 1 -- 172.21.15.114:0/7080721 >> 172.21.15.114:0/7080721 conn(0x56078b7b6300 msgr2=0x56078bb63370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:04.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.861+0000 7f5f3b7fe640 0 client.28855 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:04.866 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.865+0000 7f5f69636740 1 -- 172.21.15.114:0/7080721 shutdown_connections 2024-05-27T07:57:04.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.870+0000 7f5f69636740 1 -- 172.21.15.114:0/7080721 wait complete. 2024-05-27T07:57:04.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.870+0000 7f5f69636740 20 client.28855 trim_cache size 0 max 16384 2024-05-27T07:57:04.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.900+0000 7f5f69636740 1 Processor -- start 2024-05-27T07:57:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.900+0000 7f5f69636740 1 -- start start 2024-05-27T07:57:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.900+0000 7f5f69636740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7992f0 0x56078bb5fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.900+0000 7f5f69636740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7a14d0 0x56078bb796d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.900+0000 7f5f69636740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7c21b0 0x56078bb79c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.901+0000 7f5f69636740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4800fd50 con 0x56078b7992f0 2024-05-27T07:57:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.901+0000 7f5f69636740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5f48010420 con 0x56078b7c21b0 2024-05-27T07:57:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.901+0000 7f5f69636740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4800ecf0 con 0x56078b7a14d0 2024-05-27T07:57:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.901+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7992f0 0x56078bb5fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.904+0000 7f5f60d21640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7c21b0 0x56078bb79c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.904+0000 7f5f5b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7a14d0 0x56078bb796d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.904+0000 7f5f60d21640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7c21b0 0x56078bb79c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:54640/0 (socket says 172.21.15.114:54640) 2024-05-27T07:57:04.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.904+0000 7f5f60d21640 1 -- 172.21.15.114:0/1539687406 learned_addr learned my addr 172.21.15.114:0/1539687406 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:04.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.906+0000 7f5f5b7fe640 1 -- 172.21.15.114:0/1539687406 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7c21b0 msgr2=0x56078bb79c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:04.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.906+0000 7f5f5b7fe640 1 --2- 172.21.15.114:0/1539687406 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7c21b0 0x56078bb79c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.906+0000 7f5f5b7fe640 1 -- 172.21.15.114:0/1539687406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7992f0 msgr2=0x56078bb5fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:04.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.906+0000 7f5f5b7fe640 1 --2- 172.21.15.114:0/1539687406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7992f0 0x56078bb5fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.906+0000 7f5f5b7fe640 1 -- 172.21.15.114:0/1539687406 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56078bb6e1a0 con 0x56078b7a14d0 2024-05-27T07:57:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.909+0000 7f5f5b7fe640 1 --2- 172.21.15.114:0/1539687406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7a14d0 0x56078bb796d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0710d0 tx=0x7f5f4c04c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fca3c88556f81c5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.911+0000 7f5f41ffb640 1 -- 172.21.15.114:0/1539687406 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c05e070 con 0x56078b7a14d0 2024-05-27T07:57:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.911+0000 7f5f41ffb640 1 -- 172.21.15.114:0/1539687406 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f4c004030 con 0x56078b7a14d0 2024-05-27T07:57:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.912+0000 7f5f69636740 1 -- 172.21.15.114:0/1539687406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7a14d0 msgr2=0x56078bb796d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.912+0000 7f5f69636740 1 --2- 172.21.15.114:0/1539687406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7a14d0 0x56078bb796d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0710d0 tx=0x7f5f4c04c000 comp rx=0 tx=0).stop 2024-05-27T07:57:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.915+0000 7f5f69636740 1 -- 172.21.15.114:0/1539687406 shutdown_connections 2024-05-27T07:57:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.915+0000 7f5f69636740 1 --2- 172.21.15.114:0/1539687406 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7c21b0 0x56078bb79c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.915+0000 7f5f69636740 1 --2- 172.21.15.114:0/1539687406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7a14d0 0x56078bb796d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.915+0000 7f5f69636740 1 --2- 172.21.15.114:0/1539687406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7992f0 0x56078bb5fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.915+0000 7f5f69636740 1 -- 172.21.15.114:0/1539687406 >> 172.21.15.114:0/1539687406 conn(0x56078b7b6300 msgr2=0x56078bb5f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.924+0000 7f5f69636740 1 -- 172.21.15.114:0/1539687406 shutdown_connections 2024-05-27T07:57:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.925+0000 7f5f69636740 1 -- 172.21.15.114:0/1539687406 wait complete. 2024-05-27T07:57:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.926+0000 7f5f69636740 1 -- start start 2024-05-27T07:57:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.926+0000 7f5f69636740 1 Processor -- start 2024-05-27T07:57:04.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.926+0000 7f5f69636740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7992f0 0x56078bb70880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.926+0000 7f5f69636740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7c21b0 0x56078bb70dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.926+0000 7f5f69636740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7561c0 0x56078bb71300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.926+0000 7f5f69636740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4800e620 con 0x56078b7c21b0 2024-05-27T07:57:04.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.926+0000 7f5f69636740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4800e060 con 0x56078b7561c0 2024-05-27T07:57:04.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.926+0000 7f5f69636740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56078bb647b0 con 0x56078b7992f0 2024-05-27T07:57:04.928 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.926+0000 7f5f69636740 10 client.0 _ll_register_callbacks cb 0x7f5f500a5470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.927+0000 7f5f5b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7c21b0 0x56078bb70dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.927+0000 7f5f5b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7c21b0 0x56078bb70dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33786/0 (socket says 172.21.15.114:33786) 2024-05-27T07:57:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.927+0000 7f5f5b7fe640 1 -- 172.21.15.114:0/1481370807 learned_addr learned my addr 172.21.15.114:0/1481370807 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.927+0000 7f5f5bfff640 1 --2- 172.21.15.114:0/1481370807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7992f0 0x56078bb70880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.927+0000 7f5f60d21640 1 --2- 172.21.15.114:0/1481370807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7561c0 0x56078bb71300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.928+0000 7f5f5b7fe640 1 -- 172.21.15.114:0/1481370807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7992f0 msgr2=0x56078bb70880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.928+0000 7f5f5b7fe640 1 --2- 172.21.15.114:0/1481370807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7992f0 0x56078bb70880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.928+0000 7f5f5b7fe640 1 -- 172.21.15.114:0/1481370807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7561c0 msgr2=0x56078bb71300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.928+0000 7f5f5b7fe640 1 --2- 172.21.15.114:0/1481370807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7561c0 0x56078bb71300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.928+0000 7f5f5b7fe640 1 -- 172.21.15.114:0/1481370807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56078bbf7790 con 0x56078b7c21b0 2024-05-27T07:57:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.929+0000 7f5f5b7fe640 1 --2- 172.21.15.114:0/1481370807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7c21b0 0x56078bb70dc0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c003610 tx=0x7f5f4c003640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce957c91b5f7daf3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.929+0000 7f5f3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:57:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.929+0000 7f5f3b7fe640 1 -- 172.21.15.114:0/1481370807 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c067060 con 0x56078b7c21b0 2024-05-27T07:57:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.929+0000 7f5f69636740 1 -- 172.21.15.114:0/1481370807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56078bad7b80 con 0x56078b7c21b0 2024-05-27T07:57:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.930+0000 7f5f3b7fe640 1 -- 172.21.15.114:0/1481370807 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7f5f4c040a70 con 0x56078b7c21b0 2024-05-27T07:57:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.930+0000 7f5f3b7fe640 1 -- 172.21.15.114:0/1481370807 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f5f4c05e040 con 0x56078b7c21b0 2024-05-27T07:57:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.930+0000 7f5f69636740 10 client.19365 fetch_fsmap learned FSMap version 96 2024-05-27T07:57:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.930+0000 7f5f69636740 1 -- 172.21.15.114:0/1481370807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x56078bbf8a70 con 0x56078b7c21b0 2024-05-27T07:57:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.930+0000 7f5f3b7fe640 1 -- 172.21.15.114:0/1481370807 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56078bad7b80 con 0x56078b7c21b0 2024-05-27T07:57:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.931+0000 7f5f3b7fe640 1 -- 172.21.15.114:0/1481370807 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 96) ==== 4095+0+0 (secure 0 0 0) 0x7f5f4c04a030 con 0x56078b7c21b0 2024-05-27T07:57:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.931+0000 7f5f69636740 10 client.19365 fetch_fsmap finished waiting for FSMap version 96 2024-05-27T07:57:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.931+0000 7f5f69636740 10 client.19365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.931+0000 7f5f69636740 20 client.19365 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.931+0000 7f5f69636740 1 --2- 172.21.15.114:0/1481370807 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56078bbf8f10 0x56078bbfb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.931+0000 7f5f69636740 4 client.19365 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:57:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.931+0000 7f5f69636740 1 -- 172.21.15.114:0/1481370807 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56078bad1360 con 0x56078bbf8f10 2024-05-27T07:57:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.931+0000 7f5f5bfff640 1 --2- 172.21.15.114:0/1481370807 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56078bbf8f10 0x56078bbfb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.932+0000 7f5f5bfff640 1 --2- 172.21.15.114:0/1481370807 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56078bbf8f10 0x56078bbfb330 crc :-1 s=READY pgs=465 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f46db054c8f4446e server_cookie=df6849c671e95c87 in_seq=0 out_seq=0 2024-05-27T07:57:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:04.932+0000 7f5f3b7fe640 10 client.19365 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:05.605 INFO:tasks.workunit.client.0.smithi114.stdout: 10 48820 12.92 MB/sec execute 77 sec latency 1945.779 ms 2024-05-27T07:57:06.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:05 smithi114 ceph-mon[32038]: pgmap v593: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 355 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.8 MiB/s wr, 347 op/s 2024-05-27T07:57:06.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:05 smithi163 ceph-mon[36451]: pgmap v593: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 355 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.8 MiB/s wr, 347 op/s 2024-05-27T07:57:06.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:05 smithi186 ceph-mon[36647]: pgmap v593: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 355 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.8 MiB/s wr, 347 op/s 2024-05-27T07:57:06.605 INFO:tasks.workunit.client.0.smithi114.stdout: 10 48841 12.77 MB/sec execute 78 sec latency 2945.830 ms 2024-05-27T07:57:07.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.755+0000 7f225352b700 1 -- 172.21.15.114:0/3041751241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c10b270 msgr2=0x7f224c11d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:07.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.755+0000 7f225352b700 1 --2- 172.21.15.114:0/3041751241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c10b270 0x7f224c11d3e0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f224401c0d0 tx=0x7f224401c3e0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.755+0000 7f225352b700 1 -- 172.21.15.114:0/3041751241 shutdown_connections 2024-05-27T07:57:07.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.755+0000 7f225352b700 1 --2- 172.21.15.114:0/3041751241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c11d920 0x7f224c11fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.755+0000 7f225352b700 1 --2- 172.21.15.114:0/3041751241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f224c10a920 0x7f224c10ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.755+0000 7f225352b700 1 --2- 172.21.15.114:0/3041751241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c10b270 0x7f224c11d3e0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.756+0000 7f225352b700 1 -- 172.21.15.114:0/3041751241 >> 172.21.15.114:0/3041751241 conn(0x7f224c06f4c0 msgr2=0x7f224c06f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:07.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.760+0000 7f225352b700 1 -- 172.21.15.114:0/3041751241 shutdown_connections 2024-05-27T07:57:07.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.761+0000 7f225352b700 1 -- 172.21.15.114:0/3041751241 wait complete. 2024-05-27T07:57:07.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.762+0000 7f225352b700 1 Processor -- start 2024-05-27T07:57:07.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.764+0000 7f225352b700 1 -- start start 2024-05-27T07:57:07.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.764+0000 7f225352b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c10a920 0x7f224c11d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.764+0000 7f225352b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c10b270 0x7f224c119430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.764+0000 7f225352b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f224c11d920 0x7f224c119990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.764+0000 7f225352b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f224c1154a0 con 0x7f224c10a920 2024-05-27T07:57:07.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.764+0000 7f225352b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f224c115320 con 0x7f224c11d920 2024-05-27T07:57:07.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.764+0000 7f225352b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f224c115620 con 0x7f224c10b270 2024-05-27T07:57:07.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.764+0000 7f2250ac6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c10b270 0x7f224c119430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:07.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.764+0000 7f2250ac6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c10b270 0x7f224c119430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:53648/0 (socket says 172.21.15.114:53648) 2024-05-27T07:57:07.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.764+0000 7f2250ac6700 1 -- 172.21.15.114:0/389314345 learned_addr learned my addr 172.21.15.114:0/389314345 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:07.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.765+0000 7f2250ac6700 1 -- 172.21.15.114:0/389314345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f224c11d920 msgr2=0x7f224c119990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:57:07.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.765+0000 7f2250ac6700 1 --2- 172.21.15.114:0/389314345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f224c11d920 0x7f224c119990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.765+0000 7f2250ac6700 1 -- 172.21.15.114:0/389314345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c10a920 msgr2=0x7f224c11d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:57:07.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.765+0000 7f2250ac6700 1 --2- 172.21.15.114:0/389314345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c10a920 0x7f224c11d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.765+0000 7f2250ac6700 1 -- 172.21.15.114:0/389314345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f224c11a1c0 con 0x7f224c10b270 2024-05-27T07:57:07.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.765+0000 7f2250ac6700 1 --2- 172.21.15.114:0/389314345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c10b270 0x7f224c119430 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2244004fb0 tx=0x7f224401cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:07.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.766+0000 7f22427fc700 1 -- 172.21.15.114:0/389314345 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2244031020 con 0x7f224c10b270 2024-05-27T07:57:07.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.766+0000 7f22427fc700 1 -- 172.21.15.114:0/389314345 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2244018c30 con 0x7f224c10b270 2024-05-27T07:57:07.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.766+0000 7f225352b700 1 -- 172.21.15.114:0/389314345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f224c1ad910 con 0x7f224c10b270 2024-05-27T07:57:07.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.767+0000 7f225352b700 1 -- 172.21.15.114:0/389314345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f224c1ade20 con 0x7f224c10b270 2024-05-27T07:57:07.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.768+0000 7f22427fc700 1 -- 172.21.15.114:0/389314345 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224402dc40 con 0x7f224c10b270 2024-05-27T07:57:07.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.774+0000 7f2261f40b80 1 Processor -- start 2024-05-27T07:57:07.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.779+0000 7f2261f40b80 1 -- start start 2024-05-27T07:57:07.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.779+0000 7f2261f40b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c177d5e40 0x562c1795ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.780+0000 7f2261f40b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c1795b070 0x562c17954be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.780+0000 7f2261f40b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c17955120 0x562c17955530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.780+0000 7f2261f40b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562c17882d70 con 0x562c17955120 2024-05-27T07:57:07.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.780+0000 7f2261f40b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562c17872910 con 0x562c177d5e40 2024-05-27T07:57:07.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.780+0000 7f2261f40b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562c17876950 con 0x562c1795b070 2024-05-27T07:57:07.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.780+0000 7f22427fc700 1 -- 172.21.15.114:0/389314345 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f2244034020 con 0x7f224c10b270 2024-05-27T07:57:07.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.781+0000 7f22427fc700 1 --2- 172.21.15.114:0/389314345 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f22380b2ad0 0x7f22380b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.782+0000 7f22427fc700 1 -- 172.21.15.114:0/389314345 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7f2244020070 con 0x7f224c10b270 2024-05-27T07:57:07.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.785+0000 7f2250ac6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c17955120 0x562c17955530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:07.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.785+0000 7f2250ac6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c17955120 0x562c17955530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33810/0 (socket says 172.21.15.114:33810) 2024-05-27T07:57:07.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.785+0000 7f2250ac6700 1 -- 172.21.15.114:0/3106562734 learned_addr learned my addr 172.21.15.114:0/3106562734 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:07.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.785+0000 7f2250ac6700 1 -- 172.21.15.114:0/3106562734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c1795b070 msgr2=0x562c17954be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:07.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.785+0000 7f2250ac6700 1 --2- 172.21.15.114:0/3106562734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c1795b070 0x562c17954be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.785+0000 7f2250ac6700 1 -- 172.21.15.114:0/3106562734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c177d5e40 msgr2=0x562c1795ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:07.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.785+0000 7f2250ac6700 1 --2- 172.21.15.114:0/3106562734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c177d5e40 0x562c1795ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.785+0000 7f2250ac6700 1 -- 172.21.15.114:0/3106562734 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c1781bfb0 con 0x562c17955120 2024-05-27T07:57:07.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.787+0000 7f2251ac8700 1 --2- 172.21.15.114:0/389314345 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f22380b2ad0 0x7f22380b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:07.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.789+0000 7f2250ac6700 1 --2- 172.21.15.114:0/3106562734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c17955120 0x562c17955530 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f22440b7230 tx=0x7f224401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629279cc72ac4652 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:07.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.790+0000 7f22377fe700 1 -- 172.21.15.114:0/3106562734 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22440bb8b0 con 0x562c17955120 2024-05-27T07:57:07.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.790+0000 7f22377fe700 1 -- 172.21.15.114:0/3106562734 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2244018de0 con 0x562c17955120 2024-05-27T07:57:07.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.790+0000 7f22377fe700 1 -- 172.21.15.114:0/3106562734 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22440f1810 con 0x562c17955120 2024-05-27T07:57:07.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.792+0000 7f2261f40b80 1 -- 172.21.15.114:0/3106562734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c17955120 msgr2=0x562c17955530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:07.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.792+0000 7f2261f40b80 1 --2- 172.21.15.114:0/3106562734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c17955120 0x562c17955530 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f22440b7230 tx=0x7f224401c3e0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.792+0000 7f2251ac8700 1 --2- 172.21.15.114:0/389314345 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f22380b2ad0 0x7f22380b4fe0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f224c112b90 tx=0x7f224800b250 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:07.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.792+0000 7f2261f40b80 1 -- 172.21.15.114:0/3106562734 shutdown_connections 2024-05-27T07:57:07.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.792+0000 7f2261f40b80 1 --2- 172.21.15.114:0/3106562734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c17955120 0x562c17955530 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.792+0000 7f2261f40b80 1 --2- 172.21.15.114:0/3106562734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c177d5e40 0x562c1795ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.792+0000 7f2261f40b80 1 --2- 172.21.15.114:0/3106562734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c1795b070 0x562c17954be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.792+0000 7f2261f40b80 1 -- 172.21.15.114:0/3106562734 >> 172.21.15.114:0/3106562734 conn(0x562c177d8770 msgr2=0x562c1794fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:07.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.797+0000 7f2261f40b80 1 -- 172.21.15.114:0/3106562734 shutdown_connections 2024-05-27T07:57:07.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.797+0000 7f2261f40b80 1 -- 172.21.15.114:0/3106562734 wait complete. 2024-05-27T07:57:07.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.798+0000 7f2261f40b80 1 -- start start 2024-05-27T07:57:07.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.798+0000 7f2261f40b80 1 Processor -- start 2024-05-27T07:57:07.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.799+0000 7f2261f40b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c177d5e40 0x562c1795ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.799+0000 7f2261f40b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c17955120 0x562c1795ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.799+0000 7f2261f40b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c1795b070 0x562c1795f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.799+0000 7f2261f40b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562c17778fa0 con 0x562c1795b070 2024-05-27T07:57:07.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.799+0000 7f2261f40b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562c177dada0 con 0x562c17955120 2024-05-27T07:57:07.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.799+0000 7f2261f40b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562c17961920 con 0x562c177d5e40 2024-05-27T07:57:07.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.799+0000 7f2250ac6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c17955120 0x562c1795ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:07.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.799+0000 7f2250ac6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c17955120 0x562c1795ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:54660/0 (socket says 172.21.15.114:54660) 2024-05-27T07:57:07.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.799+0000 7f2250ac6700 1 -- 172.21.15.114:0/3121299087 learned_addr learned my addr 172.21.15.114:0/3121299087 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:07.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.799+0000 7f22512c7700 1 --2- 172.21.15.114:0/3121299087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c177d5e40 0x562c1795ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:07.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.799+0000 7f2261f40b80 10 client.0 _ll_register_callbacks cb 0x562c178095b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:07.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.800+0000 7f2250ac6700 1 -- 172.21.15.114:0/3121299087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c177d5e40 msgr2=0x562c1795ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:07.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.800+0000 7f2250ac6700 1 --2- 172.21.15.114:0/3121299087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c177d5e40 0x562c1795ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.800+0000 7f2250ac6700 1 -- 172.21.15.114:0/3121299087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c1795b070 msgr2=0x562c1795f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:07.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.800+0000 7f2250ac6700 1 --2- 172.21.15.114:0/3121299087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c1795b070 0x562c1795f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.800+0000 7f2250ac6700 1 -- 172.21.15.114:0/3121299087 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c177c28d0 con 0x562c17955120 2024-05-27T07:57:07.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.800+0000 7f2250ac6700 1 --2- 172.21.15.114:0/3121299087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c17955120 0x562c1795ef50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f22440b8420 tx=0x7f224402d670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8495544e3d509c16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:07.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.801+0000 7f2234ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:57:07.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.806+0000 7f2234ff9700 1 -- 172.21.15.114:0/3121299087 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22440bf560 con 0x562c17955120 2024-05-27T07:57:07.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.806+0000 7f2234ff9700 1 -- 172.21.15.114:0/3121299087 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7f22440bb410 con 0x562c17955120 2024-05-27T07:57:07.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.806+0000 7f2261f40b80 1 -- 172.21.15.114:0/3121299087 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2244018de0 con 0x562c17955120 2024-05-27T07:57:07.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.810+0000 7f2234ff9700 1 -- 172.21.15.114:0/3121299087 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f22440bb8b0 con 0x562c17955120 2024-05-27T07:57:07.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.810+0000 7f2261f40b80 10 client.28718 fetch_fsmap learned FSMap version 96 2024-05-27T07:57:07.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.810+0000 7f2261f40b80 1 -- 172.21.15.114:0/3121299087 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x562c1782c760 con 0x562c17955120 2024-05-27T07:57:07.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.816+0000 7f2234ff9700 1 -- 172.21.15.114:0/3121299087 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2244018de0 con 0x562c17955120 2024-05-27T07:57:07.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.816+0000 7f2234ff9700 1 -- 172.21.15.114:0/3121299087 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 96) ==== 4095+0+0 (secure 0 0 0) 0x7f22440f1ce0 con 0x562c17955120 2024-05-27T07:57:07.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.816+0000 7f2261f40b80 10 client.28718 fetch_fsmap finished waiting for FSMap version 96 2024-05-27T07:57:07.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.816+0000 7f2261f40b80 10 client.28718 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:07.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.816+0000 7f2261f40b80 20 client.28718 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:07.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.816+0000 7f2261f40b80 1 --2- 172.21.15.114:0/3121299087 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562c1796e850 0x562c17970c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.816+0000 7f2261f40b80 4 client.28718 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:57:07.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.817+0000 7f2261f40b80 1 -- 172.21.15.114:0/3121299087 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c17884cc0 con 0x562c1796e850 2024-05-27T07:57:07.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.818+0000 7f22512c7700 1 --2- 172.21.15.114:0/3121299087 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562c1796e850 0x562c17970c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:07.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.821+0000 7f22512c7700 1 --2- 172.21.15.114:0/3121299087 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562c1796e850 0x562c17970c90 crc :-1 s=READY pgs=466 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf646405d23760f5 server_cookie=19693b8c31e477d5 in_seq=0 out_seq=0 2024-05-27T07:57:07.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.823+0000 7f2234ff9700 10 client.28718 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:07.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.823+0000 7f2234ff9700 1 -- 172.21.15.114:0/3121299087 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562c17884cc0 con 0x562c1796e850 2024-05-27T07:57:07.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.823+0000 7f2234ff9700 10 client.28718 handle_command_reply: tid=0 2024-05-27T07:57:07.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.823+0000 7f2261f40b80 1 client.28718 shutdown 2024-05-27T07:57:07.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.824+0000 7f2261f40b80 1 -- 172.21.15.114:0/3121299087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c17955120 msgr2=0x562c1795ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:07.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.824+0000 7f2261f40b80 1 --2- 172.21.15.114:0/3121299087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c17955120 0x562c1795ef50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f22440b8420 tx=0x7f224402d670 comp rx=0 tx=0).stop 2024-05-27T07:57:07.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.824+0000 7f2261f40b80 1 -- 172.21.15.114:0/3121299087 shutdown_connections 2024-05-27T07:57:07.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.824+0000 7f2261f40b80 1 --2- 172.21.15.114:0/3121299087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c1795b070 0x562c1795f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.824+0000 7f2261f40b80 1 --2- 172.21.15.114:0/3121299087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c177d5e40 0x562c1795ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.824+0000 7f2261f40b80 1 --2- 172.21.15.114:0/3121299087 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562c1796e850 0x562c17970c90 crc :-1 s=READY pgs=466 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.824+0000 7f2261f40b80 1 --2- 172.21.15.114:0/3121299087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c17955120 0x562c1795ef50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.824+0000 7f2261f40b80 1 -- 172.21.15.114:0/3121299087 >> 172.21.15.114:0/3121299087 conn(0x562c177d8770 msgr2=0x562c17953bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:07.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.824+0000 7f2234ff9700 0 client.28718 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:07.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.828+0000 7f2261f40b80 1 -- 172.21.15.114:0/3121299087 shutdown_connections 2024-05-27T07:57:07.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.828+0000 7f2261f40b80 1 -- 172.21.15.114:0/3121299087 wait complete. 2024-05-27T07:57:07.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.828+0000 7f2261f40b80 20 client.28718 trim_cache size 0 max 16384 2024-05-27T07:57:07.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.856+0000 7f2261f40b80 1 Processor -- start 2024-05-27T07:57:07.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.856+0000 7f2261f40b80 1 -- start start 2024-05-27T07:57:07.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.857+0000 7f2261f40b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c177d5e40 0x562c1795d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.857+0000 7f2261f40b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c1795b070 0x562c17960e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.857+0000 7f2261f40b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c1796a980 0x562c17958ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.857+0000 7f2250ac6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c1795b070 0x562c17960e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:07.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.858+0000 7f2250ac6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c1795b070 0x562c17960e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:53680/0 (socket says 172.21.15.114:53680) 2024-05-27T07:57:07.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.858+0000 7f2250ac6700 1 -- 172.21.15.114:0/4270798162 learned_addr learned my addr 172.21.15.114:0/4270798162 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:07.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.858+0000 7f2261f40b80 1 -- 172.21.15.114:0/4270798162 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f223000ed20 con 0x562c1796a980 2024-05-27T07:57:07.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.858+0000 7f2261f40b80 1 -- 172.21.15.114:0/4270798162 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f223000f3f0 con 0x562c177d5e40 2024-05-27T07:57:07.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.858+0000 7f2261f40b80 1 -- 172.21.15.114:0/4270798162 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f223000dca0 con 0x562c1795b070 2024-05-27T07:57:07.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.859+0000 7f2251ac8700 1 --2- 172.21.15.114:0/4270798162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c1796a980 0x562c17958ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:07.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.859+0000 7f22512c7700 1 --2- 172.21.15.114:0/4270798162 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c177d5e40 0x562c1795d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:07.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.861+0000 7f2250ac6700 1 -- 172.21.15.114:0/4270798162 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c177d5e40 msgr2=0x562c1795d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:07.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.861+0000 7f2250ac6700 1 --2- 172.21.15.114:0/4270798162 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c177d5e40 0x562c1795d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.861+0000 7f2250ac6700 1 -- 172.21.15.114:0/4270798162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c1796a980 msgr2=0x562c17958ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:07.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.861+0000 7f2250ac6700 1 --2- 172.21.15.114:0/4270798162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c1796a980 0x562c17958ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.861+0000 7f2250ac6700 1 -- 172.21.15.114:0/4270798162 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c1795cf80 con 0x562c1795b070 2024-05-27T07:57:07.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.863+0000 7f2251ac8700 1 --2- 172.21.15.114:0/4270798162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c1796a980 0x562c17958ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:57:07.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.865+0000 7f2250ac6700 1 --2- 172.21.15.114:0/4270798162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c1795b070 0x562c17960e80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f22440c3950 tx=0x7f224401c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf680ab591910c23 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:07.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.872+0000 7f2236ffd700 1 -- 172.21.15.114:0/4270798162 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22440bf560 con 0x562c1795b070 2024-05-27T07:57:07.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.872+0000 7f2236ffd700 1 -- 172.21.15.114:0/4270798162 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22440bb1b0 con 0x562c1795b070 2024-05-27T07:57:07.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.872+0000 7f2236ffd700 1 -- 172.21.15.114:0/4270798162 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22440bb410 con 0x562c1795b070 2024-05-27T07:57:07.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.874+0000 7f2261f40b80 1 -- 172.21.15.114:0/4270798162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c1795b070 msgr2=0x562c17960e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:07.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.874+0000 7f2261f40b80 1 --2- 172.21.15.114:0/4270798162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c1795b070 0x562c17960e80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f22440c3950 tx=0x7f224401c3e0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.881+0000 7f2261f40b80 1 -- 172.21.15.114:0/4270798162 shutdown_connections 2024-05-27T07:57:07.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.881+0000 7f2261f40b80 1 --2- 172.21.15.114:0/4270798162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c1796a980 0x562c17958ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.881+0000 7f2261f40b80 1 --2- 172.21.15.114:0/4270798162 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c177d5e40 0x562c1795d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.881+0000 7f2261f40b80 1 --2- 172.21.15.114:0/4270798162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c1795b070 0x562c17960e80 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.881+0000 7f2261f40b80 1 -- 172.21.15.114:0/4270798162 >> 172.21.15.114:0/4270798162 conn(0x562c177d8770 msgr2=0x562c17958490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:07.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.884+0000 7f2261f40b80 1 -- 172.21.15.114:0/4270798162 shutdown_connections 2024-05-27T07:57:07.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.884+0000 7f2261f40b80 1 -- 172.21.15.114:0/4270798162 wait complete. 2024-05-27T07:57:07.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.884+0000 7f2261f40b80 1 -- start start 2024-05-27T07:57:07.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.884+0000 7f2261f40b80 1 Processor -- start 2024-05-27T07:57:07.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.885+0000 7f2261f40b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c177d5e40 0x562c17966b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.885+0000 7f2261f40b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c1795b070 0x562c179670b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.885+0000 7f2261f40b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c17975480 0x562c17975890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.885+0000 7f2261f40b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f223000d5d0 con 0x562c17975480 2024-05-27T07:57:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.885+0000 7f2261f40b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f223000cf80 con 0x562c1795b070 2024-05-27T07:57:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.885+0000 7f2261f40b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562c1794c650 con 0x562c177d5e40 2024-05-27T07:57:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.885+0000 7f2261f40b80 10 client.0 _ll_register_callbacks cb 0x562c178095b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.886+0000 7f2250ac6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c1795b070 0x562c179670b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.886+0000 7f2251ac8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c17975480 0x562c17975890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.886+0000 7f2251ac8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c17975480 0x562c17975890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33832/0 (socket says 172.21.15.114:33832) 2024-05-27T07:57:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.886+0000 7f2251ac8700 1 -- 172.21.15.114:0/2588886938 learned_addr learned my addr 172.21.15.114:0/2588886938 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.887+0000 7f2250ac6700 1 -- 172.21.15.114:0/2588886938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c177d5e40 msgr2=0x562c17966b70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:57:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.887+0000 7f2250ac6700 1 --2- 172.21.15.114:0/2588886938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c177d5e40 0x562c17966b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.887+0000 7f2250ac6700 1 -- 172.21.15.114:0/2588886938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c17975480 msgr2=0x562c17975890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.887+0000 7f2250ac6700 1 --2- 172.21.15.114:0/2588886938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c17975480 0x562c17975890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.887+0000 7f2250ac6700 1 -- 172.21.15.114:0/2588886938 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c17976ea0 con 0x562c1795b070 2024-05-27T07:57:07.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.895+0000 7f2250ac6700 1 --2- 172.21.15.114:0/2588886938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c1795b070 0x562c179670b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f22440f1b90 tx=0x7f22440bb640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67ed94d31dd6b0d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:07.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.898+0000 7f2234ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:57:07.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.900+0000 7f2234ff9700 1 -- 172.21.15.114:0/2588886938 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22440ba920 con 0x562c1795b070 2024-05-27T07:57:07.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.900+0000 7f2261f40b80 1 -- 172.21.15.114:0/2588886938 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22440bb1b0 con 0x562c1795b070 2024-05-27T07:57:07.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.900+0000 7f2234ff9700 1 -- 172.21.15.114:0/2588886938 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7f22440b77a0 con 0x562c1795b070 2024-05-27T07:57:07.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.900+0000 7f2234ff9700 1 -- 172.21.15.114:0/2588886938 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f22440bf560 con 0x562c1795b070 2024-05-27T07:57:07.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.900+0000 7f2261f40b80 10 client.28724 fetch_fsmap learned FSMap version 96 2024-05-27T07:57:07.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.900+0000 7f2261f40b80 1 -- 172.21.15.114:0/2588886938 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x562c17976ea0 con 0x562c1795b070 2024-05-27T07:57:07.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.900+0000 7f2234ff9700 1 -- 172.21.15.114:0/2588886938 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22440bb1b0 con 0x562c1795b070 2024-05-27T07:57:07.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.901+0000 7f2234ff9700 1 -- 172.21.15.114:0/2588886938 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 96) ==== 4095+0+0 (secure 0 0 0) 0x7f22440f1ce0 con 0x562c1795b070 2024-05-27T07:57:07.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.901+0000 7f2261f40b80 10 client.28724 fetch_fsmap finished waiting for FSMap version 96 2024-05-27T07:57:07.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.901+0000 7f2261f40b80 10 client.28724 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:07.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.901+0000 7f2261f40b80 20 client.28724 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:07.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.901+0000 7f2261f40b80 1 --2- 172.21.15.114:0/2588886938 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562c179d8c70 0x562c179db090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:07.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.901+0000 7f2261f40b80 4 client.28724 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:57:07.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.901+0000 7f2261f40b80 1 -- 172.21.15.114:0/2588886938 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562c1787e4a0 con 0x562c179d8c70 2024-05-27T07:57:07.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.906+0000 7f22512c7700 1 --2- 172.21.15.114:0/2588886938 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562c179d8c70 0x562c179db090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:07.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.917+0000 7f22512c7700 1 --2- 172.21.15.114:0/2588886938 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562c179d8c70 0x562c179db090 crc :-1 s=READY pgs=467 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d4a97a6e32c1409 server_cookie=833499ee57fb9aec in_seq=0 out_seq=0 2024-05-27T07:57:07.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.918+0000 7f2234ff9700 10 client.28724 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:07.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.918+0000 7f2234ff9700 1 -- 172.21.15.114:0/2588886938 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562c1787e4a0 con 0x562c179d8c70 2024-05-27T07:57:07.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.918+0000 7f2234ff9700 10 client.28724 handle_command_reply: tid=0 2024-05-27T07:57:07.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.918+0000 7f2261f40b80 1 client.28724 shutdown 2024-05-27T07:57:07.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.923+0000 7f2261f40b80 1 -- 172.21.15.114:0/2588886938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c1795b070 msgr2=0x562c179670b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:07.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.923+0000 7f2261f40b80 1 --2- 172.21.15.114:0/2588886938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c1795b070 0x562c179670b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f22440f1b90 tx=0x7f22440bb640 comp rx=0 tx=0).stop 2024-05-27T07:57:07.042 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:57:07.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.923+0000 7f2261f40b80 1 -- 172.21.15.114:0/2588886938 shutdown_connections 2024-05-27T07:57:07.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.923+0000 7f2261f40b80 1 --2- 172.21.15.114:0/2588886938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562c17975480 0x562c17975890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.923+0000 7f2261f40b80 1 --2- 172.21.15.114:0/2588886938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c177d5e40 0x562c17966b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.923+0000 7f2261f40b80 1 --2- 172.21.15.114:0/2588886938 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562c179d8c70 0x562c179db090 crc :-1 s=READY pgs=467 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.923+0000 7f2261f40b80 1 --2- 172.21.15.114:0/2588886938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c1795b070 0x562c179670b0 secure :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f22440f1b90 tx=0x7f22440bb640 comp rx=0 tx=0).stop 2024-05-27T07:57:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.923+0000 7f2261f40b80 1 -- 172.21.15.114:0/2588886938 >> 172.21.15.114:0/2588886938 conn(0x562c177d8770 msgr2=0x562c1795a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.924+0000 7f2234ff9700 0 client.28724 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.924+0000 7f2261f40b80 1 -- 172.21.15.114:0/2588886938 shutdown_connections 2024-05-27T07:57:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.924+0000 7f2261f40b80 1 -- 172.21.15.114:0/2588886938 wait complete. 2024-05-27T07:57:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.924+0000 7f2261f40b80 20 client.28724 trim_cache size 0 max 16384 2024-05-27T07:57:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.933+0000 7f225352b700 1 -- 172.21.15.114:0/389314345 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f22380b2ad0 msgr2=0x7f22380b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.933+0000 7f225352b700 1 --2- 172.21.15.114:0/389314345 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f22380b2ad0 0x7f22380b4fe0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f224c112b90 tx=0x7f224800b250 comp rx=0 tx=0).stop 2024-05-27T07:57:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.933+0000 7f225352b700 1 -- 172.21.15.114:0/389314345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c10b270 msgr2=0x7f224c119430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:07.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.933+0000 7f225352b700 1 --2- 172.21.15.114:0/389314345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c10b270 0x7f224c119430 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2244004fb0 tx=0x7f224401cef0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.933+0000 7f225352b700 1 -- 172.21.15.114:0/389314345 shutdown_connections 2024-05-27T07:57:07.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.933+0000 7f225352b700 1 --2- 172.21.15.114:0/389314345 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f22380b2ad0 0x7f22380b4fe0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.933+0000 7f225352b700 1 --2- 172.21.15.114:0/389314345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f224c11d920 0x7f224c119990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.933+0000 7f225352b700 1 --2- 172.21.15.114:0/389314345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c10a920 0x7f224c11d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.933+0000 7f225352b700 1 --2- 172.21.15.114:0/389314345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c10b270 0x7f224c119430 secure :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2244004fb0 tx=0x7f224401cef0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.933+0000 7f225352b700 1 -- 172.21.15.114:0/389314345 >> 172.21.15.114:0/389314345 conn(0x7f224c06f4c0 msgr2=0x7f224c11ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:07.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.934+0000 7f225352b700 1 -- 172.21.15.114:0/389314345 shutdown_connections 2024-05-27T07:57:07.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:06.934+0000 7f225352b700 1 -- 172.21.15.114:0/389314345 wait complete. 2024-05-27T07:57:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f3b7fe640 1 -- 172.21.15.114:0/1481370807 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56078bad1360 con 0x56078bbf8f10 2024-05-27T07:57:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f3b7fe640 10 client.19365 handle_command_reply: tid=0 2024-05-27T07:57:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f69636740 1 client.19365 shutdown 2024-05-27T07:57:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f69636740 1 -- 172.21.15.114:0/1481370807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7c21b0 msgr2=0x56078bb70dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f69636740 1 --2- 172.21.15.114:0/1481370807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7c21b0 0x56078bb70dc0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c003610 tx=0x7f5f4c003640 comp rx=0 tx=0).stop 2024-05-27T07:57:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f69636740 1 -- 172.21.15.114:0/1481370807 shutdown_connections 2024-05-27T07:57:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f69636740 1 --2- 172.21.15.114:0/1481370807 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56078bbf8f10 0x56078bbfb330 crc :-1 s=READY pgs=465 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f69636740 1 --2- 172.21.15.114:0/1481370807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56078b7561c0 0x56078bb71300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f69636740 1 --2- 172.21.15.114:0/1481370807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56078b7c21b0 0x56078bb70dc0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f69636740 1 --2- 172.21.15.114:0/1481370807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56078b7992f0 0x56078bb70880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f69636740 1 -- 172.21.15.114:0/1481370807 >> 172.21.15.114:0/1481370807 conn(0x56078b7b6300 msgr2=0x56078bb5f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f3b7fe640 0 client.19365 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f69636740 1 -- 172.21.15.114:0/1481370807 shutdown_connections 2024-05-27T07:57:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f69636740 1 -- 172.21.15.114:0/1481370807 wait complete. 2024-05-27T07:57:07.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.205+0000 7f5f69636740 20 client.19365 trim_cache size 0 max 16384 2024-05-27T07:57:07.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:57:07.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:57:07.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:57:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.209+0000 7f5f62784640 1 -- 172.21.15.114:0/3488042198 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5f440b2b10 msgr2=0x7f5f440b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.209+0000 7f5f62784640 1 --2- 172.21.15.114:0/3488042198 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5f440b2b10 0x7f5f440b5020 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x56078bb60540 tx=0x7f5f4c000bc0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.209+0000 7f5f62784640 1 -- 172.21.15.114:0/3488042198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f5c109030 msgr2=0x7f5f5c06fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.209+0000 7f5f62784640 1 --2- 172.21.15.114:0/3488042198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f5c109030 0x7f5f5c06fe50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000d680 tx=0x7f5f5000db40 comp rx=0 tx=0).stop 2024-05-27T07:57:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.209+0000 7f5f62784640 1 -- 172.21.15.114:0/3488042198 shutdown_connections 2024-05-27T07:57:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.209+0000 7f5f62784640 1 --2- 172.21.15.114:0/3488042198 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5f440b2b10 0x7f5f440b5020 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.209+0000 7f5f62784640 1 --2- 172.21.15.114:0/3488042198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f5c109030 0x7f5f5c06fe50 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.209+0000 7f5f62784640 1 --2- 172.21.15.114:0/3488042198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f5c108b90 0x7f5f5c06f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.209+0000 7f5f62784640 1 --2- 172.21.15.114:0/3488042198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5c1032e0 0x7f5f5c0747f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.209+0000 7f5f62784640 1 -- 172.21.15.114:0/3488042198 >> 172.21.15.114:0/3488042198 conn(0x7f5f5c0782d0 msgr2=0x7f5f5c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.209+0000 7f5f62784640 1 -- 172.21.15.114:0/3488042198 shutdown_connections 2024-05-27T07:57:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:07.209+0000 7f5f62784640 1 -- 172.21.15.114:0/3488042198 wait complete. 2024-05-27T07:57:07.605 INFO:tasks.workunit.client.0.smithi114.stdout: 10 48885 12.66 MB/sec execute 79 sec latency 3579.955 ms 2024-05-27T07:57:08.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:07 smithi163 ceph-mon[36451]: pgmap v594: 129 pgs: 7 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 109 active+clean; 357 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.9 MiB/s wr, 390 op/s 2024-05-27T07:57:08.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:07 smithi186 ceph-mon[36647]: pgmap v594: 129 pgs: 7 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 109 active+clean; 357 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.9 MiB/s wr, 390 op/s 2024-05-27T07:57:08.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:07 smithi114 ceph-mon[32038]: pgmap v594: 129 pgs: 7 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 109 active+clean; 357 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.9 MiB/s wr, 390 op/s 2024-05-27T07:57:08.605 INFO:tasks.workunit.client.0.smithi114.stdout: 10 48947 12.54 MB/sec execute 80 sec latency 3538.268 ms 2024-05-27T07:57:08.881 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:57:08.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:57:09.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:08 smithi163 ceph-mon[36451]: pgmap v595: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 356 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.0 MiB/s wr, 528 op/s 2024-05-27T07:57:09.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:08 smithi186 ceph-mon[36647]: pgmap v595: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 356 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.0 MiB/s wr, 528 op/s 2024-05-27T07:57:09.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:08 smithi114 ceph-mon[32038]: pgmap v595: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 356 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.0 MiB/s wr, 528 op/s 2024-05-27T07:57:09.396 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:57:09.605 INFO:tasks.workunit.client.0.smithi114.stdout: 10 49023 12.42 MB/sec execute 81 sec latency 4318.544 ms 2024-05-27T07:57:10.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 49080 12.31 MB/sec execute 82 sec latency 1850.606 ms 2024-05-27T07:57:10.986 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:10 smithi114 ceph-mon[32038]: pgmap v596: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 351 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.0 MiB/s wr, 553 op/s 2024-05-27T07:57:11.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:10 smithi163 ceph-mon[36451]: pgmap v596: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 351 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.0 MiB/s wr, 553 op/s 2024-05-27T07:57:11.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:10 smithi186 ceph-mon[36647]: pgmap v596: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 351 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.0 MiB/s wr, 553 op/s 2024-05-27T07:57:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.487+0000 7f4ec6c0c700 1 -- 172.21.15.114:0/3416741602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ec010b270 msgr2=0x7f4ec011d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.487+0000 7f4ec6c0c700 1 --2- 172.21.15.114:0/3416741602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ec010b270 0x7f4ec011d3e0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f4eb401a9a0 tx=0x7f4eb401acb0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.490+0000 7f4ec6c0c700 1 -- 172.21.15.114:0/3416741602 shutdown_connections 2024-05-27T07:57:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.490+0000 7f4ec6c0c700 1 --2- 172.21.15.114:0/3416741602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ec011d920 0x7f4ec011fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.490+0000 7f4ec6c0c700 1 --2- 172.21.15.114:0/3416741602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec010a920 0x7f4ec010ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.490+0000 7f4ec6c0c700 1 --2- 172.21.15.114:0/3416741602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ec010b270 0x7f4ec011d3e0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.490+0000 7f4ec6c0c700 1 -- 172.21.15.114:0/3416741602 >> 172.21.15.114:0/3416741602 conn(0x7f4ec006f4c0 msgr2=0x7f4ec006f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:11.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.495+0000 7f4ec6c0c700 1 -- 172.21.15.114:0/3416741602 shutdown_connections 2024-05-27T07:57:11.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.498+0000 7f4ec6c0c700 1 -- 172.21.15.114:0/3416741602 wait complete. 2024-05-27T07:57:11.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.500+0000 7f4ec6c0c700 1 Processor -- start 2024-05-27T07:57:11.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.504+0000 7f4ec6c0c700 1 -- start start 2024-05-27T07:57:11.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.504+0000 7f4ec6c0c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ec010a920 0x7f4ec011d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.504+0000 7f4ec6c0c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec010b270 0x7f4ec0119480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.504+0000 7f4ec6c0c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ec011d920 0x7f4ec0119a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.504+0000 7f4ec6c0c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec01154a0 con 0x7f4ec011d920 2024-05-27T07:57:11.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.504+0000 7f4ec6c0c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec0115320 con 0x7f4ec010a920 2024-05-27T07:57:11.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.504+0000 7f4ec6c0c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec0115620 con 0x7f4ec010b270 2024-05-27T07:57:11.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.506+0000 7f4ec51a9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ec011d920 0x7f4ec0119a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.506+0000 7f4ec51a9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ec011d920 0x7f4ec0119a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33848/0 (socket says 172.21.15.114:33848) 2024-05-27T07:57:11.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.506+0000 7f4ec51a9700 1 -- 172.21.15.114:0/3130836762 learned_addr learned my addr 172.21.15.114:0/3130836762 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:11.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.506+0000 7f4ec49a8700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ec010a920 0x7f4ec011d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.506+0000 7f4ebffff700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec010b270 0x7f4ec0119480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.507+0000 7f4ebffff700 1 -- 172.21.15.114:0/3130836762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ec010a920 msgr2=0x7f4ec011d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:11.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.507+0000 7f4ebffff700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ec010a920 0x7f4ec011d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.507+0000 7f4ebffff700 1 -- 172.21.15.114:0/3130836762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ec011d920 msgr2=0x7f4ec0119a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:11.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.507+0000 7f4ebffff700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ec011d920 0x7f4ec0119a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.507+0000 7f4ebffff700 1 -- 172.21.15.114:0/3130836762 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec011a2b0 con 0x7f4ec010b270 2024-05-27T07:57:11.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.508+0000 7f4ebffff700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec010b270 0x7f4ec0119480 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4000c00 tx=0x7f4eb4003bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:11.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.509+0000 7f4ebdffb700 1 -- 172.21.15.114:0/3130836762 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4003f60 con 0x7f4ec010b270 2024-05-27T07:57:11.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.510+0000 7f4ec6c0c700 1 -- 172.21.15.114:0/3130836762 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec01ad970 con 0x7f4ec010b270 2024-05-27T07:57:11.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.510+0000 7f4ec6c0c700 1 -- 172.21.15.114:0/3130836762 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec01ade50 con 0x7f4ec010b270 2024-05-27T07:57:11.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.510+0000 7f4ebdffb700 1 -- 172.21.15.114:0/3130836762 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4eb4004100 con 0x7f4ec010b270 2024-05-27T07:57:11.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.510+0000 7f4ebdffb700 1 -- 172.21.15.114:0/3130836762 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb401f3f0 con 0x7f4ec010b270 2024-05-27T07:57:11.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.516+0000 7f4ebdffb700 1 -- 172.21.15.114:0/3130836762 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f4eb403b440 con 0x7f4ec010b270 2024-05-27T07:57:11.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.521+0000 7f4ebdffb700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4ea80b2c00 0x7f4ea80b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.521+0000 7f4ec49a8700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4ea80b2c00 0x7f4ea80b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.521+0000 7f4ebdffb700 1 -- 172.21.15.114:0/3130836762 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7f4eb40f55c0 con 0x7f4ec010b270 2024-05-27T07:57:11.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.522+0000 7f4ed5621b80 1 Processor -- start 2024-05-27T07:57:11.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.523+0000 7f4ec49a8700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4ea80b2c00 0x7f4ea80b5110 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000a850 tx=0x7f4eb0008040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:11.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.524+0000 7f4ed5621b80 1 -- start start 2024-05-27T07:57:11.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.524+0000 7f4ed5621b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e493ec0 0x55562e618bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.524+0000 7f4ed5621b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e612c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.524+0000 7f4ed5621b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6131a0 0x55562e6135b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.524+0000 7f4ed5621b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55562e540df0 con 0x55562e6131a0 2024-05-27T07:57:11.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.524+0000 7f4ed5621b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55562e530990 con 0x55562e493ec0 2024-05-27T07:57:11.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.524+0000 7f4ed5621b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55562e5349d0 con 0x55562e6190f0 2024-05-27T07:57:11.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.526+0000 7f4ec49a8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e493ec0 0x55562e618bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.526+0000 7f4ec49a8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e493ec0 0x55562e618bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:54706/0 (socket says 172.21.15.114:54706) 2024-05-27T07:57:11.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.526+0000 7f4ec49a8700 1 -- 172.21.15.114:0/4049012172 learned_addr learned my addr 172.21.15.114:0/4049012172 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:11.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.527+0000 7f4ec51a9700 1 --2- 172.21.15.114:0/4049012172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6131a0 0x55562e6135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.528+0000 7f4ebffff700 1 --2- 172.21.15.114:0/4049012172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e612c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.531+0000 7f4ebffff700 1 -- 172.21.15.114:0/4049012172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e493ec0 msgr2=0x55562e618bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:11.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.531+0000 7f4ebffff700 1 --2- 172.21.15.114:0/4049012172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e493ec0 0x55562e618bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.531+0000 7f4ebffff700 1 -- 172.21.15.114:0/4049012172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6131a0 msgr2=0x55562e6135b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:11.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.531+0000 7f4ebffff700 1 --2- 172.21.15.114:0/4049012172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6131a0 0x55562e6135b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.531+0000 7f4ebffff700 1 -- 172.21.15.114:0/4049012172 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55562e4da030 con 0x55562e6190f0 2024-05-27T07:57:11.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.533+0000 7f4ebffff700 1 --2- 172.21.15.114:0/4049012172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e612c60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40f2850 tx=0x7f4eb40bbc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ff42bae8ef0448d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:11.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.538+0000 7f4ea6ffd700 1 -- 172.21.15.114:0/4049012172 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb40bcb70 con 0x55562e6190f0 2024-05-27T07:57:11.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.538+0000 7f4ea6ffd700 1 -- 172.21.15.114:0/4049012172 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4eb40bc470 con 0x55562e6190f0 2024-05-27T07:57:11.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.539+0000 7f4ed5621b80 1 -- 172.21.15.114:0/4049012172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 msgr2=0x55562e612c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:11.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.539+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/4049012172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e612c60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40f2850 tx=0x7f4eb40bbc60 comp rx=0 tx=0).stop 2024-05-27T07:57:11.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.546+0000 7f4ed5621b80 1 -- 172.21.15.114:0/4049012172 shutdown_connections 2024-05-27T07:57:11.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.546+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/4049012172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6131a0 0x55562e6135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.546+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/4049012172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e493ec0 0x55562e618bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.546+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/4049012172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e612c60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.546+0000 7f4ed5621b80 1 -- 172.21.15.114:0/4049012172 >> 172.21.15.114:0/4049012172 conn(0x55562e4967f0 msgr2=0x55562e60de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:11.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.550+0000 7f4ed5621b80 1 -- 172.21.15.114:0/4049012172 shutdown_connections 2024-05-27T07:57:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.552+0000 7f4ed5621b80 1 -- 172.21.15.114:0/4049012172 wait complete. 2024-05-27T07:57:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.552+0000 7f4ed5621b80 1 -- start start 2024-05-27T07:57:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.553+0000 7f4ed5621b80 1 Processor -- start 2024-05-27T07:57:11.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.553+0000 7f4ed5621b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e493ec0 0x55562e61c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.553+0000 7f4ed5621b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e6131a0 0x55562e61cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.553+0000 7f4ed5621b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e61d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.553+0000 7f4ed5621b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55562e437020 con 0x55562e493ec0 2024-05-27T07:57:11.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.553+0000 7f4ed5621b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55562e498e20 con 0x55562e6131a0 2024-05-27T07:57:11.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.553+0000 7f4ed5621b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55562e61f9a0 con 0x55562e6190f0 2024-05-27T07:57:11.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.553+0000 7f4ed5621b80 10 client.0 _ll_register_callbacks cb 0x55562e4c7630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:11.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.554+0000 7f4ec51a9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e61d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.554+0000 7f4ec51a9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e61d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:53718/0 (socket says 172.21.15.114:53718) 2024-05-27T07:57:11.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.554+0000 7f4ec51a9700 1 -- 172.21.15.114:0/1039409468 learned_addr learned my addr 172.21.15.114:0/1039409468 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:11.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.557+0000 7f4ec49a8700 1 --2- 172.21.15.114:0/1039409468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e493ec0 0x55562e61c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.558+0000 7f4ebffff700 1 --2- 172.21.15.114:0/1039409468 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e6131a0 0x55562e61cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.558+0000 7f4ec51a9700 1 -- 172.21.15.114:0/1039409468 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e6131a0 msgr2=0x55562e61cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:11.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.558+0000 7f4ec51a9700 1 --2- 172.21.15.114:0/1039409468 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e6131a0 0x55562e61cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.558+0000 7f4ec51a9700 1 -- 172.21.15.114:0/1039409468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e493ec0 msgr2=0x55562e61c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:11.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.558+0000 7f4ec51a9700 1 --2- 172.21.15.114:0/1039409468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e493ec0 0x55562e61c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.558+0000 7f4ec51a9700 1 -- 172.21.15.114:0/1039409468 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55562e480950 con 0x55562e6190f0 2024-05-27T07:57:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.560+0000 7f4ec51a9700 1 --2- 172.21.15.114:0/1039409468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e61d290 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8007e00 tx=0x7f4eb8010440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1501b7bf7386730e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.560+0000 7f4e9ffff700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:57:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.560+0000 7f4e9ffff700 1 -- 172.21.15.114:0/1039409468 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb80109d0 con 0x55562e6190f0 2024-05-27T07:57:11.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.560+0000 7f4ed5621b80 1 -- 172.21.15.114:0/1039409468 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4eb40bc470 con 0x55562e6190f0 2024-05-27T07:57:11.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.560+0000 7f4e9ffff700 1 -- 172.21.15.114:0/1039409468 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7f4eb8010b70 con 0x55562e6190f0 2024-05-27T07:57:11.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.560+0000 7f4e9ffff700 1 -- 172.21.15.114:0/1039409468 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f4eb8010d70 con 0x55562e6190f0 2024-05-27T07:57:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.564+0000 7f4ed5621b80 10 client.28879 fetch_fsmap learned FSMap version 96 2024-05-27T07:57:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.564+0000 7f4ed5621b80 1 -- 172.21.15.114:0/1039409468 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x55562e4ea7e0 con 0x55562e6190f0 2024-05-27T07:57:11.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.571+0000 7f4e9ffff700 1 -- 172.21.15.114:0/1039409468 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4eb40bc470 con 0x55562e6190f0 2024-05-27T07:57:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.572+0000 7f4e9ffff700 1 -- 172.21.15.114:0/1039409468 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 96) ==== 4095+0+0 (secure 0 0 0) 0x7f4eb8016070 con 0x55562e6190f0 2024-05-27T07:57:11.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.573+0000 7f4ed5621b80 10 client.28879 fetch_fsmap finished waiting for FSMap version 96 2024-05-27T07:57:11.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.573+0000 7f4ed5621b80 10 client.28879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:11.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.573+0000 7f4ed5621b80 20 client.28879 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:11.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.573+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/1039409468 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55562e62cc00 0x55562e62f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.573+0000 7f4ed5621b80 4 client.28879 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:57:11.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.573+0000 7f4ed5621b80 1 -- 172.21.15.114:0/1039409468 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4eb40f53b0 con 0x55562e62cc00 2024-05-27T07:57:11.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.578+0000 7f4ec49a8700 1 --2- 172.21.15.114:0/1039409468 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55562e62cc00 0x55562e62f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.589+0000 7f4ec49a8700 1 --2- 172.21.15.114:0/1039409468 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55562e62cc00 0x55562e62f090 crc :-1 s=READY pgs=468 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a718dadd8531551e server_cookie=dcaf98bda5f1645 in_seq=0 out_seq=0 2024-05-27T07:57:11.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.589+0000 7f4e9ffff700 10 client.28879 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:11.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.594+0000 7f4e9ffff700 1 -- 172.21.15.114:0/1039409468 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f4eb40f53b0 con 0x55562e62cc00 2024-05-27T07:57:11.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.594+0000 7f4e9ffff700 10 client.28879 handle_command_reply: tid=0 2024-05-27T07:57:11.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.595+0000 7f4ed5621b80 1 client.28879 shutdown 2024-05-27T07:57:11.605 INFO:tasks.workunit.client.0.smithi114.stdout: 10 49213 12.26 MB/sec execute 83 sec latency 1963.757 ms 2024-05-27T07:57:11.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.605+0000 7f4ed5621b80 1 -- 172.21.15.114:0/1039409468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 msgr2=0x55562e61d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:11.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.605+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/1039409468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e61d290 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8007e00 tx=0x7f4eb8010440 comp rx=0 tx=0).stop 2024-05-27T07:57:11.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.605+0000 7f4ed5621b80 1 -- 172.21.15.114:0/1039409468 shutdown_connections 2024-05-27T07:57:11.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.605+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/1039409468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e61d290 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.605+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/1039409468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e493ec0 0x55562e61c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.605+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/1039409468 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55562e62cc00 0x55562e62f090 crc :-1 s=READY pgs=468 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.605+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/1039409468 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e6131a0 0x55562e61cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.605+0000 7f4ed5621b80 1 -- 172.21.15.114:0/1039409468 >> 172.21.15.114:0/1039409468 conn(0x55562e4967f0 msgr2=0x55562e611c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:11.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.606+0000 7f4e9ffff700 0 client.28879 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:11.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.606+0000 7f4ed5621b80 1 -- 172.21.15.114:0/1039409468 shutdown_connections 2024-05-27T07:57:11.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.606+0000 7f4ed5621b80 1 -- 172.21.15.114:0/1039409468 wait complete. 2024-05-27T07:57:11.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.606+0000 7f4ed5621b80 20 client.28879 trim_cache size 0 max 16384 2024-05-27T07:57:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.646+0000 7f4ed5621b80 1 Processor -- start 2024-05-27T07:57:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.647+0000 7f4ed5621b80 1 -- start start 2024-05-27T07:57:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.647+0000 7f4ed5621b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e493ec0 0x55562e617f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.647+0000 7f4ed5621b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6131a0 0x55562e628a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.647+0000 7f4ed5621b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e634890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.647+0000 7f4ed5621b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac00eb40 con 0x55562e6131a0 2024-05-27T07:57:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.647+0000 7f4ed5621b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac00f210 con 0x55562e493ec0 2024-05-27T07:57:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.647+0000 7f4ed5621b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac00dac0 con 0x55562e6190f0 2024-05-27T07:57:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.649+0000 7f4ec51a9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e634890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.649+0000 7f4ec51a9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e634890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:53730/0 (socket says 172.21.15.114:53730) 2024-05-27T07:57:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.649+0000 7f4ec51a9700 1 -- 172.21.15.114:0/3145459212 learned_addr learned my addr 172.21.15.114:0/3145459212 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.650+0000 7f4ec51a9700 1 -- 172.21.15.114:0/3145459212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e493ec0 msgr2=0x55562e617f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.650+0000 7f4ec51a9700 1 --2- 172.21.15.114:0/3145459212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e493ec0 0x55562e617f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.650+0000 7f4ec51a9700 1 -- 172.21.15.114:0/3145459212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6131a0 msgr2=0x55562e628a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.650+0000 7f4ec51a9700 1 --2- 172.21.15.114:0/3145459212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6131a0 0x55562e628a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.650+0000 7f4ec51a9700 1 -- 172.21.15.114:0/3145459212 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55562e61b7f0 con 0x55562e6190f0 2024-05-27T07:57:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.651+0000 7f4ec51a9700 1 --2- 172.21.15.114:0/3145459212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e634890 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4eb80022f0 tx=0x7f4eb8010f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2ea8e28be8ed791 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.651+0000 7f4ea67fc700 1 -- 172.21.15.114:0/3145459212 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb802e070 con 0x55562e6190f0 2024-05-27T07:57:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.651+0000 7f4ea67fc700 1 -- 172.21.15.114:0/3145459212 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4eb801c070 con 0x55562e6190f0 2024-05-27T07:57:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.653+0000 7f4ed5621b80 1 -- 172.21.15.114:0/3145459212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 msgr2=0x55562e634890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.653+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/3145459212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e634890 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4eb80022f0 tx=0x7f4eb8010f80 comp rx=0 tx=0).stop 2024-05-27T07:57:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.654+0000 7f4ed5621b80 1 -- 172.21.15.114:0/3145459212 shutdown_connections 2024-05-27T07:57:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.654+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/3145459212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e6190f0 0x55562e634890 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.654+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/3145459212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e493ec0 0x55562e617f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.654+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/3145459212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6131a0 0x55562e628a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.654+0000 7f4ed5621b80 1 -- 172.21.15.114:0/3145459212 >> 172.21.15.114:0/3145459212 conn(0x55562e4967f0 msgr2=0x55562e617820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:11.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.657+0000 7f4ed5621b80 1 -- 172.21.15.114:0/3145459212 shutdown_connections 2024-05-27T07:57:11.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.657+0000 7f4ed5621b80 1 -- 172.21.15.114:0/3145459212 wait complete. 2024-05-27T07:57:11.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.658+0000 7f4ed5621b80 1 -- start start 2024-05-27T07:57:11.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.658+0000 7f4ed5621b80 1 Processor -- start 2024-05-27T07:57:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.659+0000 7f4ed5621b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6190f0 0x55562e625170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.659+0000 7f4ed5621b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e621bc0 0x55562e621fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.659+0000 7f4ed5621b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e622510 0x55562e697770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.659+0000 7f4ed5621b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac00d3f0 con 0x55562e6190f0 2024-05-27T07:57:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.659+0000 7f4ed5621b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac00ce30 con 0x55562e621bc0 2024-05-27T07:57:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.659+0000 7f4ed5621b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55562e437020 con 0x55562e622510 2024-05-27T07:57:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.659+0000 7f4ed5621b80 10 client.0 _ll_register_callbacks cb 0x55562e4c7630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:11.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.661+0000 7f4ec49a8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6190f0 0x55562e625170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.662+0000 7f4ec49a8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6190f0 0x55562e625170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33876/0 (socket says 172.21.15.114:33876) 2024-05-27T07:57:11.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.662+0000 7f4ec49a8700 1 -- 172.21.15.114:0/3819390097 learned_addr learned my addr 172.21.15.114:0/3819390097 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:11.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.667+0000 7f4ebffff700 1 --2- 172.21.15.114:0/3819390097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e621bc0 0x55562e621fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.667+0000 7f4ec51a9700 1 --2- 172.21.15.114:0/3819390097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e622510 0x55562e697770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.674+0000 7f4ec49a8700 1 -- 172.21.15.114:0/3819390097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e622510 msgr2=0x55562e697770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:11.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.674+0000 7f4ec49a8700 1 --2- 172.21.15.114:0/3819390097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e622510 0x55562e697770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.674+0000 7f4ec49a8700 1 -- 172.21.15.114:0/3819390097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e621bc0 msgr2=0x55562e621fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:11.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.674+0000 7f4ec49a8700 1 --2- 172.21.15.114:0/3819390097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e621bc0 0x55562e621fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.674+0000 7f4ec49a8700 1 -- 172.21.15.114:0/3819390097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55562e625800 con 0x55562e6190f0 2024-05-27T07:57:11.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.691+0000 7f4ec49a8700 1 --2- 172.21.15.114:0/3819390097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6190f0 0x55562e625170 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0006ea0 tx=0x7f4eb0004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ae69d22491b6a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:11.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.692+0000 7f4e9ffff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:57:11.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.696+0000 7f4e9ffff700 1 -- 172.21.15.114:0/3819390097 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb002e060 con 0x55562e6190f0 2024-05-27T07:57:11.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.696+0000 7f4ed5621b80 1 -- 172.21.15.114:0/3819390097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4eb801c070 con 0x55562e6190f0 2024-05-27T07:57:11.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.696+0000 7f4e9ffff700 1 -- 172.21.15.114:0/3819390097 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7f4eb0027070 con 0x55562e6190f0 2024-05-27T07:57:11.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.704+0000 7f4e9ffff700 1 -- 172.21.15.114:0/3819390097 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f4eb0035050 con 0x55562e6190f0 2024-05-27T07:57:11.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.704+0000 7f4ed5621b80 10 client.19389 fetch_fsmap learned FSMap version 96 2024-05-27T07:57:11.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.704+0000 7f4ed5621b80 1 -- 172.21.15.114:0/3819390097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x55562e698e00 con 0x55562e6190f0 2024-05-27T07:57:11.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.705+0000 7f4e9ffff700 1 -- 172.21.15.114:0/3819390097 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4eb801c070 con 0x55562e6190f0 2024-05-27T07:57:11.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.706+0000 7f4e9ffff700 1 -- 172.21.15.114:0/3819390097 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 96) ==== 4095+0+0 (secure 0 0 0) 0x7f4eb0028ad0 con 0x55562e6190f0 2024-05-27T07:57:11.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.706+0000 7f4ed5621b80 10 client.19389 fetch_fsmap finished waiting for FSMap version 96 2024-05-27T07:57:11.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.706+0000 7f4ed5621b80 10 client.19389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:11.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.706+0000 7f4ed5621b80 20 client.19389 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:11.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.706+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/3819390097 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55562e699520 0x55562e69b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:11.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.706+0000 7f4ed5621b80 4 client.19389 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:57:11.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.706+0000 7f4ed5621b80 1 -- 172.21.15.114:0/3819390097 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55562e542d40 con 0x55562e699520 2024-05-27T07:57:11.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.706+0000 7f4ebffff700 1 --2- 172.21.15.114:0/3819390097 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55562e699520 0x55562e69b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:11.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.707+0000 7f4e9ffff700 10 client.19389 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:11.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.708+0000 7f4ebffff700 1 --2- 172.21.15.114:0/3819390097 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55562e699520 0x55562e69b940 crc :-1 s=READY pgs=469 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=499b208b287fdb47 server_cookie=ae93740d6b0534aa in_seq=0 out_seq=0 2024-05-27T07:57:11.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.735+0000 7f4e9ffff700 1 -- 172.21.15.114:0/3819390097 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55562e542d40 con 0x55562e699520 2024-05-27T07:57:11.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.735+0000 7f4e9ffff700 10 client.19389 handle_command_reply: tid=0 2024-05-27T07:57:11.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.735+0000 7f4ed5621b80 1 client.19389 shutdown 2024-05-27T07:57:11.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.735+0000 7f4ed5621b80 1 -- 172.21.15.114:0/3819390097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6190f0 msgr2=0x55562e625170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:11.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.735+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/3819390097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6190f0 0x55562e625170 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0006ea0 tx=0x7f4eb0004770 comp rx=0 tx=0).stop 2024-05-27T07:57:11.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.736+0000 7f4ed5621b80 1 -- 172.21.15.114:0/3819390097 shutdown_connections 2024-05-27T07:57:11.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.736+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/3819390097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55562e622510 0x55562e697770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.736+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/3819390097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55562e6190f0 0x55562e625170 secure :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0006ea0 tx=0x7f4eb0004770 comp rx=0 tx=0).stop 2024-05-27T07:57:11.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.736+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/3819390097 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55562e699520 0x55562e69b940 crc :-1 s=READY pgs=469 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.736+0000 7f4ed5621b80 1 --2- 172.21.15.114:0/3819390097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55562e621bc0 0x55562e621fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.736+0000 7f4ed5621b80 1 -- 172.21.15.114:0/3819390097 >> 172.21.15.114:0/3819390097 conn(0x55562e4967f0 msgr2=0x55562e616510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:11.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.736+0000 7f4e9ffff700 0 client.19389 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:11.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.736+0000 7f4ed5621b80 1 -- 172.21.15.114:0/3819390097 shutdown_connections 2024-05-27T07:57:11.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.736+0000 7f4ed5621b80 1 -- 172.21.15.114:0/3819390097 wait complete. 2024-05-27T07:57:11.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.736+0000 7f4ed5621b80 20 client.19389 trim_cache size 0 max 16384 2024-05-27T07:57:11.741 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:57:11.741 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:57:11.741 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "c72748c7-2301-485b-9cf0-abc17644f1b3", 2024-05-27T07:57:11.741 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:57:11.741 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:57:11.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.746+0000 7f4ec6c0c700 1 -- 172.21.15.114:0/3130836762 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4ea80b2c00 msgr2=0x7f4ea80b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:11.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.746+0000 7f4ec6c0c700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4ea80b2c00 0x7f4ea80b5110 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000a850 tx=0x7f4eb0008040 comp rx=0 tx=0).stop 2024-05-27T07:57:11.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.746+0000 7f4ec6c0c700 1 -- 172.21.15.114:0/3130836762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec010b270 msgr2=0x7f4ec0119480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:11.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.746+0000 7f4ec6c0c700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec010b270 0x7f4ec0119480 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4000c00 tx=0x7f4eb4003bb0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.746+0000 7f4ec6c0c700 1 -- 172.21.15.114:0/3130836762 shutdown_connections 2024-05-27T07:57:11.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.746+0000 7f4ec6c0c700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ec011d920 0x7f4ec0119a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.746+0000 7f4ec6c0c700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4ea80b2c00 0x7f4ea80b5110 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.746+0000 7f4ec6c0c700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ec010a920 0x7f4ec011d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.746+0000 7f4ec6c0c700 1 --2- 172.21.15.114:0/3130836762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec010b270 0x7f4ec0119480 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:11.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.746+0000 7f4ec6c0c700 1 -- 172.21.15.114:0/3130836762 >> 172.21.15.114:0/3130836762 conn(0x7f4ec006f4c0 msgr2=0x7f4ec011df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:11.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.751+0000 7f4ec6c0c700 1 -- 172.21.15.114:0/3130836762 shutdown_connections 2024-05-27T07:57:11.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:11.755+0000 7f4ec6c0c700 1 -- 172.21.15.114:0/3130836762 wait complete. 2024-05-27T07:57:12.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 49328 12.27 MB/sec execute 84 sec latency 1587.821 ms 2024-05-27T07:57:12.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:57:13.143 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:57:13.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:12 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:57:13.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:12 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:57:13.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:12 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:57:13.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:12 smithi163 ceph-mon[36451]: pgmap v597: 129 pgs: 17 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 59 active+clean; 337 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 571 op/s 2024-05-27T07:57:13.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:12 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:57:13.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:12 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:57:13.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:12 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:57:13.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:12 smithi186 ceph-mon[36647]: pgmap v597: 129 pgs: 17 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 59 active+clean; 337 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 571 op/s 2024-05-27T07:57:13.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:12 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:57:13.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:12 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:57:13.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:12 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:57:13.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:12 smithi114 ceph-mon[32038]: pgmap v597: 129 pgs: 17 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 59 active+clean; 337 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 571 op/s 2024-05-27T07:57:13.605 INFO:tasks.workunit.client.0.smithi114.stdout: 10 49478 12.22 MB/sec execute 85 sec latency 1245.314 ms 2024-05-27T07:57:14.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 49608 12.18 MB/sec execute 86 sec latency 1560.191 ms 2024-05-27T07:57:14.909 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:14 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:57:14.909 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:14 smithi114 systemd[1]: Stopped Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:57:14.909 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:14 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Consumed 3.402s CPU time. 2024-05-27T07:57:14.910 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:14 smithi114 systemd[1]: Starting Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:57:15.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:14 smithi163 ceph-mon[36451]: pgmap v598: 129 pgs: 17 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 59 active+clean; 340 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 578 op/s 2024-05-27T07:57:15.249 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:14 smithi163 ceph-mon[36451]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:57:15.249 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:14 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 2 up:standby 2024-05-27T07:57:15.250 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:14 smithi186 ceph-mon[36647]: pgmap v598: 129 pgs: 17 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 59 active+clean; 340 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 578 op/s 2024-05-27T07:57:15.250 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:14 smithi186 ceph-mon[36647]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:57:15.250 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:14 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 2 up:standby 2024-05-27T07:57:15.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:14 smithi114 ceph-mon[32038]: pgmap v598: 129 pgs: 17 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 59 active+clean; 340 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 578 op/s 2024-05-27T07:57:15.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:14 smithi114 ceph-mon[32038]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:57:15.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:14 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 2 up:standby 2024-05-27T07:57:15.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb814e9a700 time 2024-05-27T07:57:14.996496+0000 2024-05-27T07:57:15.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:57:15.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:15.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb8276dfba6] 2024-05-27T07:57:15.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8276dfdd1] 2024-05-27T07:57:15.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b429c8823d] 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b429c73971] 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b429c71fa0] 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b429c72651] 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b429c07347] 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b42990387e] 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b429908f2f] 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b429913969] 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b429cac48c] 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b429cb59cb] 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 13: (Thread::entry_wrapper()+0x43) [0x7fb8276ae72d] 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb8276ae749] 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb8262521ca] 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 16: clone() 2024-05-27T07:57:15.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: *** Caught signal (Aborted) ** 2024-05-27T07:57:15.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: in thread 7fb814e9a700 thread_name:md_log_replay 2024-05-27T07:57:15.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:14.997+0000 7fb814e9a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb814e9a700 time 2024-05-27T07:57:14.996496+0000 2024-05-27T07:57:15.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:57:15.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:15.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:15.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb8276dfba6] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8276dfdd1] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b429c8823d] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b429c73971] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b429c71fa0] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b429c72651] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b429c07347] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b42990387e] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b429908f2f] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b429913969] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b429cac48c] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b429cb59cb] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 13: (Thread::entry_wrapper()+0x43) [0x7fb8276ae72d] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb8276ae749] 2024-05-27T07:57:15.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb8262521ca] 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 16: clone() 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb82625cd20] 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2: gsignal() 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 3: abort() 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb8276dfced] 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8276dfdd1] 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b429c8823d] 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b429c73971] 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b429c71fa0] 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b429c72651] 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b429c07347] 2024-05-27T07:57:15.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b42990387e] 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b429908f2f] 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b429913969] 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b429cac48c] 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b429cb59cb] 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 16: (Thread::entry_wrapper()+0x43) [0x7fb8276ae72d] 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb8276ae749] 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb8262521ca] 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 19: clone() 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:15.000+0000 7fb814e9a700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: in thread 7fb814e9a700 thread_name:md_log_replay 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb82625cd20] 2024-05-27T07:57:15.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2: gsignal() 2024-05-27T07:57:15.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 3: abort() 2024-05-27T07:57:15.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb8276dfced] 2024-05-27T07:57:15.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8276dfdd1] 2024-05-27T07:57:15.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b429c8823d] 2024-05-27T07:57:15.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b429c73971] 2024-05-27T07:57:15.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b429c71fa0] 2024-05-27T07:57:15.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b429c72651] 2024-05-27T07:57:15.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b429c07347] 2024-05-27T07:57:15.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b42990387e] 2024-05-27T07:57:15.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b429908f2f] 2024-05-27T07:57:15.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b429913969] 2024-05-27T07:57:15.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b429cac48c] 2024-05-27T07:57:15.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b429cb59cb] 2024-05-27T07:57:15.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 16: (Thread::entry_wrapper()+0x43) [0x7fb8276ae72d] 2024-05-27T07:57:15.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb8276ae749] 2024-05-27T07:57:15.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb8262521ca] 2024-05-27T07:57:15.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 19: clone() 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: -1> 2024-05-27T07:57:14.997+0000 7fb814e9a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb814e9a700 time 2024-05-27T07:57:14.996496+0000 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb8276dfba6] 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8276dfdd1] 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b429c8823d] 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b429c73971] 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b429c71fa0] 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b429c72651] 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b429c07347] 2024-05-27T07:57:15.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b42990387e] 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b429908f2f] 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b429913969] 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b429cac48c] 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b429cb59cb] 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 13: (Thread::entry_wrapper()+0x43) [0x7fb8276ae72d] 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb8276ae749] 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb8262521ca] 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 16: clone() 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 0> 2024-05-27T07:57:15.000+0000 7fb814e9a700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: in thread 7fb814e9a700 thread_name:md_log_replay 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb82625cd20] 2024-05-27T07:57:15.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2: gsignal() 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 3: abort() 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb8276dfced] 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8276dfdd1] 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b429c8823d] 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b429c73971] 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b429c71fa0] 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b429c72651] 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b429c07347] 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b42990387e] 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b429908f2f] 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b429913969] 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b429cac48c] 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b429cb59cb] 2024-05-27T07:57:15.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 16: (Thread::entry_wrapper()+0x43) [0x7fb8276ae72d] 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb8276ae749] 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb8262521ca] 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 19: clone() 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: -1> 2024-05-27T07:57:14.997+0000 7fb814e9a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb814e9a700 time 2024-05-27T07:57:14.996496+0000 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb8276dfba6] 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8276dfdd1] 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b429c8823d] 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b429c73971] 2024-05-27T07:57:15.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b429c71fa0] 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b429c72651] 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b429c07347] 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b42990387e] 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b429908f2f] 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b429913969] 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b429cac48c] 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b429cb59cb] 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 13: (Thread::entry_wrapper()+0x43) [0x7fb8276ae72d] 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb8276ae749] 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb8262521ca] 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 16: clone() 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 0> 2024-05-27T07:57:15.000+0000 7fb814e9a700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:57:15.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: in thread 7fb814e9a700 thread_name:md_log_replay 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb82625cd20] 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2: gsignal() 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 3: abort() 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb8276dfced] 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8276dfdd1] 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b429c8823d] 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b429c73971] 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b429c71fa0] 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b429c72651] 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b429c07347] 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b42990387e] 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b429908f2f] 2024-05-27T07:57:15.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b429913969] 2024-05-27T07:57:15.322 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b429cac48c] 2024-05-27T07:57:15.322 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b429cb59cb] 2024-05-27T07:57:15.322 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 16: (Thread::entry_wrapper()+0x43) [0x7fb8276ae72d] 2024-05-27T07:57:15.322 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb8276ae749] 2024-05-27T07:57:15.322 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb8262521ca] 2024-05-27T07:57:15.322 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 19: clone() 2024-05-27T07:57:15.322 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:57:15.322 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[70952]: 2024-05-27T07:57:15.610 INFO:tasks.workunit.client.0.smithi114.stdout: 10 49743 12.12 MB/sec execute 87 sec latency 957.418 ms 2024-05-27T07:57:16.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:15 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:57:16.058 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:15 smithi114 podman[128807]: 2024-05-27 07:57:15.745538383 +0000 UTC m=+0.677370475 container died 5542c738fdbb55c1542dbd44aab6cfe2e3d865fbeb89bfc73ab14b6138196adb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-27T07:57:16.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:15 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:57:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:15 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:57:16.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 49883 12.03 MB/sec execute 88 sec latency 340.265 ms 2024-05-27T07:57:16.791 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:16 smithi114 podman[128807]: 2024-05-27 07:57:16.483907944 +0000 UTC m=+1.415740032 container remove 5542c738fdbb55c1542dbd44aab6cfe2e3d865fbeb89bfc73ab14b6138196adb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2) 2024-05-27T07:57:16.791 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:16 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:57:17.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:16 smithi114 ceph-mon[32038]: pgmap v599: 129 pgs: 17 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 59 active+clean; 341 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.2 MiB/s wr, 523 op/s 2024-05-27T07:57:17.057 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:16 smithi114 podman[128866]: 2024-05-27 07:57:16.785686186 +0000 UTC m=+0.117777280 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:57:17.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:16 smithi114 podman[128866]: 2024-05-27 07:57:16.921466326 +0000 UTC m=+0.253557402 container create 459081d6e2d5570403951de332ca4c2ab38395490179130597c927795134064a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T07:57:17.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:16 smithi163 ceph-mon[36451]: pgmap v599: 129 pgs: 17 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 59 active+clean; 341 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.2 MiB/s wr, 523 op/s 2024-05-27T07:57:17.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:16 smithi186 ceph-mon[36647]: pgmap v599: 129 pgs: 17 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 59 active+clean; 341 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.2 MiB/s wr, 523 op/s 2024-05-27T07:57:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.386+0000 7fc6ad180700 1 -- 172.21.15.114:0/1069297886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6a811d920 msgr2=0x7fc6a811fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.386+0000 7fc6ad180700 1 --2- 172.21.15.114:0/1069297886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6a811d920 0x7fc6a811fd40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000b0a0 tx=0x7fc6a000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.386+0000 7fc6ad180700 1 -- 172.21.15.114:0/1069297886 shutdown_connections 2024-05-27T07:57:17.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.386+0000 7fc6ad180700 1 --2- 172.21.15.114:0/1069297886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6a811d920 0x7fc6a811fd40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.386+0000 7fc6ad180700 1 --2- 172.21.15.114:0/1069297886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a810a920 0x7fc6a810ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.386+0000 7fc6ad180700 1 --2- 172.21.15.114:0/1069297886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6a810b270 0x7fc6a811d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.386+0000 7fc6ad180700 1 -- 172.21.15.114:0/1069297886 >> 172.21.15.114:0/1069297886 conn(0x7fc6a806f4c0 msgr2=0x7fc6a806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:17.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.393+0000 7fc6ad180700 1 -- 172.21.15.114:0/1069297886 shutdown_connections 2024-05-27T07:57:17.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.397+0000 7fc6ad180700 1 -- 172.21.15.114:0/1069297886 wait complete. 2024-05-27T07:57:17.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.398+0000 7fc6ad180700 1 Processor -- start 2024-05-27T07:57:17.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.398+0000 7fc6ad180700 1 -- start start 2024-05-27T07:57:17.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.399+0000 7fc6ad180700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6a810b270 0x7fc6a81a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.399+0000 7fc6ad180700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a811d920 0x7fc6a81aa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.399+0000 7fc6ad180700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6a81ae560 0x7fc6a81ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.400+0000 7fc6ad180700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a81233e0 con 0x7fc6a811d920 2024-05-27T07:57:17.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.400+0000 7fc6ad180700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a8123260 con 0x7fc6a81ae560 2024-05-27T07:57:17.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.400+0000 7fc6ad180700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a8123560 con 0x7fc6a810b270 2024-05-27T07:57:17.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.400+0000 7fc69ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a811d920 0x7fc6a81aa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:17.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.400+0000 7fc69ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a811d920 0x7fc6a81aa2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49416/0 (socket says 172.21.15.114:49416) 2024-05-27T07:57:17.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.401+0000 7fc69ffff700 1 -- 172.21.15.114:0/300359907 learned_addr learned my addr 172.21.15.114:0/300359907 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:17.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.402+0000 7fc69ffff700 1 -- 172.21.15.114:0/300359907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6a810b270 msgr2=0x7fc6a81a9d30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:57:17.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.402+0000 7fc69ffff700 1 --2- 172.21.15.114:0/300359907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6a810b270 0x7fc6a81a9d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.402+0000 7fc69ffff700 1 -- 172.21.15.114:0/300359907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6a81ae560 msgr2=0x7fc6a81ae970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:57:17.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.402+0000 7fc69ffff700 1 --2- 172.21.15.114:0/300359907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6a81ae560 0x7fc6a81ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.402+0000 7fc69ffff700 1 -- 172.21.15.114:0/300359907 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a81af010 con 0x7fc6a811d920 2024-05-27T07:57:17.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.404+0000 7fc69ffff700 1 --2- 172.21.15.114:0/300359907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a811d920 0x7fc6a81aa2d0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fc69000c920 tx=0x7fc69000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:17.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.406+0000 7fc6a4d99700 1 -- 172.21.15.114:0/300359907 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690007b00 con 0x7fc6a811d920 2024-05-27T07:57:17.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.406+0000 7fc6a4d99700 1 -- 172.21.15.114:0/300359907 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc690004ce0 con 0x7fc6a811d920 2024-05-27T07:57:17.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.406+0000 7fc6a4d99700 1 -- 172.21.15.114:0/300359907 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690005650 con 0x7fc6a811d920 2024-05-27T07:57:17.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.406+0000 7fc6ad180700 1 -- 172.21.15.114:0/300359907 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6a81af300 con 0x7fc6a811d920 2024-05-27T07:57:17.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.406+0000 7fc6ad180700 1 -- 172.21.15.114:0/300359907 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6a81b6aa0 con 0x7fc6a811d920 2024-05-27T07:57:17.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.411+0000 7fc6bbb95b80 1 Processor -- start 2024-05-27T07:57:17.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.413+0000 7fc6bbb95b80 1 -- start start 2024-05-27T07:57:17.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.414+0000 7fc6a4d99700 1 -- 172.21.15.114:0/300359907 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fc690007ca0 con 0x7fc6a811d920 2024-05-27T07:57:17.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.416+0000 7fc6a4d99700 1 --2- 172.21.15.114:0/300359907 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc6940b2c00 0x7fc6940b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.416+0000 7fc6a4d99700 1 -- 172.21.15.114:0/300359907 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7fc6900e2630 con 0x7fc6a811d920 2024-05-27T07:57:17.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.416+0000 7fc6bbb95b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250b45e40 0x559250ccab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.416+0000 7fc6bbb95b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250ccb070 0x559250cc4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.417+0000 7fc6bbb95b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cc5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.417+0000 7fc6bbb95b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559250bf2d70 con 0x559250b45e40 2024-05-27T07:57:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.417+0000 7fc6bbb95b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559250be2910 con 0x559250cc5120 2024-05-27T07:57:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.417+0000 7fc6bbb95b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559250be6950 con 0x559250ccb070 2024-05-27T07:57:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.418+0000 7fc6a759e700 1 --2- 172.21.15.114:0/300359907 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc6940b2c00 0x7fc6940b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.418+0000 7fc6a759e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cc5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.418+0000 7fc6a759e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cc5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:38500/0 (socket says 172.21.15.114:38500) 2024-05-27T07:57:17.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.418+0000 7fc6a759e700 1 -- 172.21.15.114:0/376553651 learned_addr learned my addr 172.21.15.114:0/376553651 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:17.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.418+0000 7fc6a759e700 1 -- 172.21.15.114:0/376553651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250ccb070 msgr2=0x559250cc4be0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:57:17.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.418+0000 7fc6a759e700 1 --2- 172.21.15.114:0/376553651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250ccb070 0x559250cc4be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.419+0000 7fc6a759e700 1 -- 172.21.15.114:0/376553651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250b45e40 msgr2=0x559250ccab30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:57:17.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.419+0000 7fc6a759e700 1 --2- 172.21.15.114:0/376553651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250b45e40 0x559250ccab30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.419+0000 7fc6a759e700 1 -- 172.21.15.114:0/376553651 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559250b8bfb0 con 0x559250cc5120 2024-05-27T07:57:17.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.419+0000 7fc6a759e700 1 --2- 172.21.15.114:0/300359907 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc6940b2c00 0x7fc6940b5110 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000b0a0 tx=0x7fc6a00090d0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:17.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.419+0000 7fc6a759e700 1 --2- 172.21.15.114:0/376553651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cc5530 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0006070 tx=0x7fc6a0024000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=900400d0ad954d09 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:17.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.419+0000 7fc69d7fa700 1 -- 172.21.15.114:0/376553651 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0009260 con 0x559250cc5120 2024-05-27T07:57:17.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.420+0000 7fc69d7fa700 1 -- 172.21.15.114:0/376553651 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6a001f8f0 con 0x559250cc5120 2024-05-27T07:57:17.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.420+0000 7fc69d7fa700 1 -- 172.21.15.114:0/376553651 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a001c820 con 0x559250cc5120 2024-05-27T07:57:17.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.423+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/376553651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 msgr2=0x559250cc5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:17.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.423+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/376553651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cc5530 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0006070 tx=0x7fc6a0024000 comp rx=0 tx=0).stop 2024-05-27T07:57:17.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.424+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/376553651 shutdown_connections 2024-05-27T07:57:17.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.424+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/376553651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cc5530 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.424+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/376553651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250b45e40 0x559250ccab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.424+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/376553651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250ccb070 0x559250cc4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.424+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/376553651 >> 172.21.15.114:0/376553651 conn(0x559250b48770 msgr2=0x559250cbfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:17.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.429+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/376553651 shutdown_connections 2024-05-27T07:57:17.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.431+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/376553651 wait complete. 2024-05-27T07:57:17.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.431+0000 7fc6bbb95b80 1 -- start start 2024-05-27T07:57:17.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.431+0000 7fc6bbb95b80 1 Processor -- start 2024-05-27T07:57:17.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.432+0000 7fc6bbb95b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250b45e40 0x559250cce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.432+0000 7fc6bbb95b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.432+0000 7fc6bbb95b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250ccb070 0x559250ccf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.432+0000 7fc6bbb95b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559250ae8fa0 con 0x559250ccb070 2024-05-27T07:57:17.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.432+0000 7fc6bbb95b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559250b4ada0 con 0x559250cc5120 2024-05-27T07:57:17.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.432+0000 7fc6bbb95b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559250cd1b00 con 0x559250b45e40 2024-05-27T07:57:17.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.433+0000 7fc6bbb95b80 10 client.0 _ll_register_callbacks cb 0x559250b795b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:17.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.433+0000 7fc6a759e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250ccb070 0x559250ccf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:17.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.434+0000 7fc6a759e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250ccb070 0x559250ccf410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49428/0 (socket says 172.21.15.114:49428) 2024-05-27T07:57:17.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.434+0000 7fc6a759e700 1 -- 172.21.15.114:0/2740963204 learned_addr learned my addr 172.21.15.114:0/2740963204 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:17.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.438+0000 7fc69ffff700 1 --2- 172.21.15.114:0/2740963204 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:17.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.438+0000 7fc6a759e700 1 -- 172.21.15.114:0/2740963204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250b45e40 msgr2=0x559250cce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:17.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.438+0000 7fc6a6d9d700 1 --2- 172.21.15.114:0/2740963204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250b45e40 0x559250cce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:17.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.438+0000 7fc6a759e700 1 --2- 172.21.15.114:0/2740963204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250b45e40 0x559250cce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.438+0000 7fc6a759e700 1 -- 172.21.15.114:0/2740963204 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 msgr2=0x559250cceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:17.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.438+0000 7fc6a759e700 1 --2- 172.21.15.114:0/2740963204 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cceed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.438+0000 7fc6a759e700 1 -- 172.21.15.114:0/2740963204 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559250b328d0 con 0x559250ccb070 2024-05-27T07:57:17.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.439+0000 7fc6a759e700 1 --2- 172.21.15.114:0/2740963204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250ccb070 0x559250ccf410 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000d010 tx=0x7fc6a0009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c4267c2a66fa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:17.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.439+0000 7fc68e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:57:17.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.439+0000 7fc68e7fc700 1 -- 172.21.15.114:0/2740963204 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0030070 con 0x559250ccb070 2024-05-27T07:57:17.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.439+0000 7fc68e7fc700 1 -- 172.21.15.114:0/2740963204 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7fc6a001f650 con 0x559250ccb070 2024-05-27T07:57:17.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.440+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2740963204 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6a001f8f0 con 0x559250ccb070 2024-05-27T07:57:17.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.440+0000 7fc68e7fc700 1 -- 172.21.15.114:0/2740963204 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fc6a0003a00 con 0x559250ccb070 2024-05-27T07:57:17.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.440+0000 7fc6bbb95b80 10 client.19395 fetch_fsmap learned FSMap version 97 2024-05-27T07:57:17.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.440+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2740963204 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x559250b9c760 con 0x559250ccb070 2024-05-27T07:57:17.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.441+0000 7fc68e7fc700 1 -- 172.21.15.114:0/2740963204 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6a001f8f0 con 0x559250ccb070 2024-05-27T07:57:17.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.441+0000 7fc68e7fc700 1 -- 172.21.15.114:0/2740963204 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 97) ==== 3490+0+0 (secure 0 0 0) 0x7fc6a0011070 con 0x559250ccb070 2024-05-27T07:57:17.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.441+0000 7fc6bbb95b80 10 client.19395 fetch_fsmap finished waiting for FSMap version 97 2024-05-27T07:57:17.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.441+0000 7fc6bbb95b80 10 client.19395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:17.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.442+0000 7fc6bbb95b80 20 client.19395 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:17.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.442+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/2740963204 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559250cdeab0 0x559250ce0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.442+0000 7fc6bbb95b80 4 client.19395 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:57:17.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.442+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2740963204 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559250bf4cc0 con 0x559250cdeab0 2024-05-27T07:57:17.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.446+0000 7fc6a6d9d700 1 --2- 172.21.15.114:0/2740963204 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559250cdeab0 0x559250ce0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:17.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.455+0000 7fc6a6d9d700 1 --2- 172.21.15.114:0/2740963204 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559250cdeab0 0x559250ce0ed0 crc :-1 s=READY pgs=470 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c14a98998ba9ffa1 server_cookie=893517c35483ffa6 in_seq=0 out_seq=0 2024-05-27T07:57:17.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.456+0000 7fc68e7fc700 10 client.19395 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.462+0000 7fc68e7fc700 1 -- 172.21.15.114:0/2740963204 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559250bf4cc0 con 0x559250cdeab0 2024-05-27T07:57:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.462+0000 7fc68e7fc700 10 client.19395 handle_command_reply: tid=0 2024-05-27T07:57:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.462+0000 7fc6bbb95b80 1 client.19395 shutdown 2024-05-27T07:57:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.463+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2740963204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250ccb070 msgr2=0x559250ccf410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:17.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.463+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/2740963204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250ccb070 0x559250ccf410 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000d010 tx=0x7fc6a0009700 comp rx=0 tx=0).stop 2024-05-27T07:57:17.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.463+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2740963204 shutdown_connections 2024-05-27T07:57:17.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.463+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/2740963204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250ccb070 0x559250ccf410 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.463+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/2740963204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250b45e40 0x559250cce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.463+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/2740963204 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559250cdeab0 0x559250ce0ed0 crc :-1 s=READY pgs=470 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.463+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/2740963204 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.463+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2740963204 >> 172.21.15.114:0/2740963204 conn(0x559250b48770 msgr2=0x559250cc3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:17.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.463+0000 7fc68e7fc700 0 client.19395 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:17.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.465+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2740963204 shutdown_connections 2024-05-27T07:57:17.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.465+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2740963204 wait complete. 2024-05-27T07:57:17.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.465+0000 7fc6bbb95b80 20 client.19395 trim_cache size 0 max 16384 2024-05-27T07:57:17.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.502+0000 7fc6bbb95b80 1 Processor -- start 2024-05-27T07:57:17.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.503+0000 7fc6bbb95b80 1 -- start start 2024-05-27T07:57:17.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.504+0000 7fc6bbb95b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250b45e40 0x559250cc4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.504+0000 7fc6bbb95b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250cc5120 0x559250cc8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.504+0000 7fc6bbb95b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250ccb070 0x559250cd0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.504+0000 7fc6bbb95b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a81bf1f0 con 0x559250ccb070 2024-05-27T07:57:17.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.504+0000 7fc6bbb95b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a81bf8c0 con 0x559250b45e40 2024-05-27T07:57:17.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.504+0000 7fc6bbb95b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a81bdbf0 con 0x559250cc5120 2024-05-27T07:57:17.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.507+0000 7fc69ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250cc5120 0x559250cc8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:17.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.507+0000 7fc69ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250cc5120 0x559250cc8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:50868/0 (socket says 172.21.15.114:50868) 2024-05-27T07:57:17.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.507+0000 7fc69ffff700 1 -- 172.21.15.114:0/1275419182 learned_addr learned my addr 172.21.15.114:0/1275419182 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:17.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.510+0000 7fc69ffff700 1 -- 172.21.15.114:0/1275419182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250b45e40 msgr2=0x559250cc4be0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:57:17.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.510+0000 7fc69ffff700 1 --2- 172.21.15.114:0/1275419182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250b45e40 0x559250cc4be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.510+0000 7fc69ffff700 1 -- 172.21.15.114:0/1275419182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250ccb070 msgr2=0x559250cd0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:17.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.510+0000 7fc69ffff700 1 --2- 172.21.15.114:0/1275419182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250ccb070 0x559250cd0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.510+0000 7fc69ffff700 1 -- 172.21.15.114:0/1275419182 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559250cc1590 con 0x559250cc5120 2024-05-27T07:57:17.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.514+0000 7fc69ffff700 1 --2- 172.21.15.114:0/1275419182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250cc5120 0x559250cc8ec0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc6900a66c0 tx=0x7fc6900a8fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a3e8680b4e857de server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.518+0000 7fc69cff9700 1 -- 172.21.15.114:0/1275419182 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6900c3070 con 0x559250cc5120 2024-05-27T07:57:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.518+0000 7fc69cff9700 1 -- 172.21.15.114:0/1275419182 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6900ad800 con 0x559250cc5120 2024-05-27T07:57:17.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.519+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/1275419182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250cc5120 msgr2=0x559250cc8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:17.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.519+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/1275419182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250cc5120 0x559250cc8ec0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc6900a66c0 tx=0x7fc6900a8fe0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.520+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/1275419182 shutdown_connections 2024-05-27T07:57:17.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.520+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/1275419182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250ccb070 0x559250cd0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.520+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/1275419182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250b45e40 0x559250cc4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.520+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/1275419182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250cc5120 0x559250cc8ec0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.520+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/1275419182 >> 172.21.15.114:0/1275419182 conn(0x559250b48770 msgr2=0x559250cce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:17.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.520+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/1275419182 shutdown_connections 2024-05-27T07:57:17.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.520+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/1275419182 wait complete. 2024-05-27T07:57:17.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.521+0000 7fc6bbb95b80 1 -- start start 2024-05-27T07:57:17.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.521+0000 7fc6bbb95b80 1 Processor -- start 2024-05-27T07:57:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.521+0000 7fc6bbb95b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250b45e40 0x559250cd7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.521+0000 7fc6bbb95b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cd7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.521+0000 7fc6bbb95b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250ccb070 0x559250cd3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.521+0000 7fc6bbb95b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559250bf2d70 con 0x559250b45e40 2024-05-27T07:57:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.522+0000 7fc6bbb95b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a81be1f0 con 0x559250cc5120 2024-05-27T07:57:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.522+0000 7fc6bbb95b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559250cc5530 con 0x559250ccb070 2024-05-27T07:57:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.522+0000 7fc6bbb95b80 10 client.0 _ll_register_callbacks cb 0x559250b795b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.523+0000 7fc6a6d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250b45e40 0x559250cd7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.523+0000 7fc6a6d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250b45e40 0x559250cd7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49450/0 (socket says 172.21.15.114:49450) 2024-05-27T07:57:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.523+0000 7fc6a6d9d700 1 -- 172.21.15.114:0/2942828309 learned_addr learned my addr 172.21.15.114:0/2942828309 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.526+0000 7fc6a759e700 1 --2- 172.21.15.114:0/2942828309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250ccb070 0x559250cd3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.526+0000 7fc69ffff700 1 --2- 172.21.15.114:0/2942828309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cd7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.526+0000 7fc6a6d9d700 1 -- 172.21.15.114:0/2942828309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250ccb070 msgr2=0x559250cd3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.526+0000 7fc6a6d9d700 1 --2- 172.21.15.114:0/2942828309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250ccb070 0x559250cd3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.526+0000 7fc6a6d9d700 1 -- 172.21.15.114:0/2942828309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 msgr2=0x559250cd7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.526+0000 7fc6a6d9d700 1 --2- 172.21.15.114:0/2942828309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cd7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.526+0000 7fc6a6d9d700 1 -- 172.21.15.114:0/2942828309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559250cc9ff0 con 0x559250b45e40 2024-05-27T07:57:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.528+0000 7fc6a6d9d700 1 --2- 172.21.15.114:0/2942828309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250b45e40 0x559250cd7000 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fc698001eb0 tx=0x7fc69800abd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef617cd31511d5ac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.528+0000 7fc68e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:57:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.530+0000 7fc68e7fc700 1 -- 172.21.15.114:0/2942828309 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc698015820 con 0x559250b45e40 2024-05-27T07:57:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.530+0000 7fc68e7fc700 1 -- 172.21.15.114:0/2942828309 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7fc6980159c0 con 0x559250b45e40 2024-05-27T07:57:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.530+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2942828309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6900ad800 con 0x559250b45e40 2024-05-27T07:57:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.531+0000 7fc68e7fc700 1 -- 172.21.15.114:0/2942828309 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fc698015bc0 con 0x559250b45e40 2024-05-27T07:57:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.532+0000 7fc6bbb95b80 10 client.19398 fetch_fsmap learned FSMap version 97 2024-05-27T07:57:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.532+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2942828309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x559250d49dd0 con 0x559250b45e40 2024-05-27T07:57:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.533+0000 7fc68e7fc700 1 -- 172.21.15.114:0/2942828309 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6900ad800 con 0x559250b45e40 2024-05-27T07:57:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.533+0000 7fc68e7fc700 1 -- 172.21.15.114:0/2942828309 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 97) ==== 3490+0+0 (secure 0 0 0) 0x7fc69801d070 con 0x559250b45e40 2024-05-27T07:57:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.533+0000 7fc6bbb95b80 10 client.19398 fetch_fsmap finished waiting for FSMap version 97 2024-05-27T07:57:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.533+0000 7fc6bbb95b80 10 client.19398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.533+0000 7fc6bbb95b80 20 client.19398 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.533+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/2942828309 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559250d4a470 0x559250d4c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.533+0000 7fc6bbb95b80 4 client.19398 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:57:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.533+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2942828309 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559250bee4a0 con 0x559250d4a470 2024-05-27T07:57:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.533+0000 7fc69ffff700 1 --2- 172.21.15.114:0/2942828309 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559250d4a470 0x559250d4c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.537+0000 7fc69ffff700 1 --2- 172.21.15.114:0/2942828309 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559250d4a470 0x559250d4c890 crc :-1 s=READY pgs=471 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a94c62c506a216cb server_cookie=108cba474c00c1cc in_seq=0 out_seq=0 2024-05-27T07:57:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.538+0000 7fc68e7fc700 10 client.19398 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:17.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.543+0000 7fc68e7fc700 1 -- 172.21.15.114:0/2942828309 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559250bee4a0 con 0x559250d4a470 2024-05-27T07:57:17.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.543+0000 7fc68e7fc700 10 client.19398 handle_command_reply: tid=0 2024-05-27T07:57:17.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.543+0000 7fc6bbb95b80 1 client.19398 shutdown 2024-05-27T07:57:17.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.545+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2942828309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250b45e40 msgr2=0x559250cd7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:17.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.545+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/2942828309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250b45e40 0x559250cd7000 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fc698001eb0 tx=0x7fc69800abd0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.545+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2942828309 shutdown_connections 2024-05-27T07:57:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.545+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/2942828309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559250ccb070 0x559250cd3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.545+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/2942828309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559250b45e40 0x559250cd7000 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.545+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/2942828309 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x559250d4a470 0x559250d4c890 crc :-1 s=READY pgs=471 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.545+0000 7fc6bbb95b80 1 --2- 172.21.15.114:0/2942828309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559250cc5120 0x559250cd7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.545+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2942828309 >> 172.21.15.114:0/2942828309 conn(0x559250b48770 msgr2=0x559250cc8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.545+0000 7fc68e7fc700 0 client.19398 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.546+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2942828309 shutdown_connections 2024-05-27T07:57:17.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.550+0000 7fc6bbb95b80 1 -- 172.21.15.114:0/2942828309 wait complete. 2024-05-27T07:57:17.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.550+0000 7fc6bbb95b80 20 client.19398 trim_cache size 0 max 16384 2024-05-27T07:57:17.554 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:57:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:57:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:57:17.554 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:57:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.554+0000 7fc6ad180700 1 -- 172.21.15.114:0/300359907 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc6940b2c00 msgr2=0x7fc6940b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.554+0000 7fc6ad180700 1 --2- 172.21.15.114:0/300359907 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc6940b2c00 0x7fc6940b5110 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000b0a0 tx=0x7fc6a00090d0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.554+0000 7fc6ad180700 1 -- 172.21.15.114:0/300359907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a811d920 msgr2=0x7fc6a81aa2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.554+0000 7fc6ad180700 1 --2- 172.21.15.114:0/300359907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a811d920 0x7fc6a81aa2d0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fc69000c920 tx=0x7fc69000cce0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.557+0000 7fc6ad180700 1 -- 172.21.15.114:0/300359907 shutdown_connections 2024-05-27T07:57:17.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.557+0000 7fc6ad180700 1 --2- 172.21.15.114:0/300359907 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc6940b2c00 0x7fc6940b5110 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.557+0000 7fc6ad180700 1 --2- 172.21.15.114:0/300359907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6a81ae560 0x7fc6a81ae970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.557+0000 7fc6ad180700 1 --2- 172.21.15.114:0/300359907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6a810b270 0x7fc6a81a9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.557+0000 7fc6ad180700 1 --2- 172.21.15.114:0/300359907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a811d920 0x7fc6a81aa2d0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.557+0000 7fc6ad180700 1 -- 172.21.15.114:0/300359907 >> 172.21.15.114:0/300359907 conn(0x7fc6a806f4c0 msgr2=0x7fc6a811b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.559+0000 7fc6ad180700 1 -- 172.21.15.114:0/300359907 shutdown_connections 2024-05-27T07:57:17.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:17.560+0000 7fc6ad180700 1 -- 172.21.15.114:0/300359907 wait complete. 2024-05-27T07:57:17.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 50010 11.95 MB/sec execute 89 sec latency 178.450 ms 2024-05-27T07:57:17.663 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:17 smithi114 podman[128866]: 2024-05-27 07:57:17.579524579 +0000 UTC m=+0.911615662 container init 459081d6e2d5570403951de332ca4c2ab38395490179130597c927795134064a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-27T07:57:17.663 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:17 smithi114 podman[128866]: 2024-05-27 07:57:17.594359728 +0000 UTC m=+0.926450812 container start 459081d6e2d5570403951de332ca4c2ab38395490179130597c927795134064a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:57:17.931 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:17 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: starting mds.a at 2024-05-27T07:57:18.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:17 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:57:18.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:17 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:57:18.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:17 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:57:18.426 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:57:18.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:57:18.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 50177 11.89 MB/sec execute 90 sec latency 173.017 ms 2024-05-27T07:57:18.807 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:18 smithi114 bash[128866]: 459081d6e2d5570403951de332ca4c2ab38395490179130597c927795134064a 2024-05-27T07:57:18.807 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:18 smithi114 systemd[1]: Started Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:57:18.919 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:57:19.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:18 smithi163 ceph-mon[36451]: pgmap v600: 129 pgs: 11 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 94 active+clean; 339 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 527 op/s 2024-05-27T07:57:19.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:18 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6838/2267591646,v1:172.21.15.114:6839/2267591646] up:boot 2024-05-27T07:57:19.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:18 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:57:19.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:18 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:57:19.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:18 smithi186 ceph-mon[36647]: pgmap v600: 129 pgs: 11 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 94 active+clean; 339 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 527 op/s 2024-05-27T07:57:19.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:18 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6838/2267591646,v1:172.21.15.114:6839/2267591646] up:boot 2024-05-27T07:57:19.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:18 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:57:19.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:18 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:57:19.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:18 smithi114 ceph-mon[32038]: pgmap v600: 129 pgs: 11 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 94 active+clean; 339 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 527 op/s 2024-05-27T07:57:19.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:18 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6838/2267591646,v1:172.21.15.114:6839/2267591646] up:boot 2024-05-27T07:57:19.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:18 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:57:19.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:18 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:57:19.255 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:19 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:57:19.255 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:19 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.586s CPU time. 2024-05-27T07:57:19.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 50364 11.84 MB/sec execute 91 sec latency 127.002 ms 2024-05-27T07:57:20.378 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.376+0000 7fd73475e640 1 -- 172.21.15.114:0/3724338869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd72c11d240 msgr2=0x7fd72c11f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.377+0000 7fd7314f8640 1 -- 172.21.15.114:0/3724338869 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c035020 con 0x7fd72c11d240 2024-05-27T07:57:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.377+0000 7fd73475e640 1 --2- 172.21.15.114:0/3724338869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd72c11d240 0x7fd72c11f660 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fd71c009900 tx=0x7fd71c02a4d0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.382+0000 7fd73475e640 1 -- 172.21.15.114:0/3724338869 shutdown_connections 2024-05-27T07:57:20.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.382+0000 7fd73475e640 1 --2- 172.21.15.114:0/3724338869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd72c11d240 0x7fd72c11f660 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.382+0000 7fd73475e640 1 --2- 172.21.15.114:0/3724338869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd72c11a8e0 0x7fd72c11cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.382+0000 7fd73475e640 1 --2- 172.21.15.114:0/3724338869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd72c10a1f0 0x7fd72c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.382+0000 7fd73475e640 1 -- 172.21.15.114:0/3724338869 >> 172.21.15.114:0/3724338869 conn(0x7fd72c06e8f0 msgr2=0x7fd72c06ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:20.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.392+0000 7fd73475e640 1 -- 172.21.15.114:0/3724338869 shutdown_connections 2024-05-27T07:57:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.393+0000 7fd73475e640 1 -- 172.21.15.114:0/3724338869 wait complete. 2024-05-27T07:57:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.394+0000 7fd73475e640 1 Processor -- start 2024-05-27T07:57:20.401 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.398+0000 7fd73475e640 1 -- start start 2024-05-27T07:57:20.401 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.398+0000 7fd73475e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd72c10a1f0 0x7fd72c1134e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.401 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.398+0000 7fd73475e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd72c11a8e0 0x7fd72c113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.401 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.398+0000 7fd73475e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd72c11d240 0x7fd72c113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.401 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.398+0000 7fd73475e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c122d50 con 0x7fd72c11a8e0 2024-05-27T07:57:20.401 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.398+0000 7fd73475e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c122bd0 con 0x7fd72c10a1f0 2024-05-27T07:57:20.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.398+0000 7fd73475e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c122ed0 con 0x7fd72c11d240 2024-05-27T07:57:20.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.400+0000 7fd731cf9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd72c11a8e0 0x7fd72c113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.401+0000 7fd731cf9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd72c11a8e0 0x7fd72c113a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49478/0 (socket says 172.21.15.114:49478) 2024-05-27T07:57:20.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.401+0000 7fd731cf9640 1 -- 172.21.15.114:0/3881393476 learned_addr learned my addr 172.21.15.114:0/3881393476 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:20.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.401+0000 7fd7324fa640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd72c10a1f0 0x7fd72c1134e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.402+0000 7fd732cfb640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd72c11d240 0x7fd72c113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.407+0000 7fd7324fa640 1 -- 172.21.15.114:0/3881393476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd72c11d240 msgr2=0x7fd72c113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.407+0000 7fd7324fa640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd72c11d240 0x7fd72c113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.407+0000 7fd7324fa640 1 -- 172.21.15.114:0/3881393476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd72c11a8e0 msgr2=0x7fd72c113a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.407+0000 7fd7324fa640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd72c11a8e0 0x7fd72c113a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.407+0000 7fd7324fa640 1 -- 172.21.15.114:0/3881393476 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c117ee0 con 0x7fd72c10a1f0 2024-05-27T07:57:20.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.408+0000 7fd7324fa640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd72c10a1f0 0x7fd72c1134e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd728009fd0 tx=0x7fd72800b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:20.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.413+0000 7fd7237fe640 1 -- 172.21.15.114:0/3881393476 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728015020 con 0x7fd72c10a1f0 2024-05-27T07:57:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.413+0000 7fd73475e640 1 -- 172.21.15.114:0/3881393476 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72c1181d0 con 0x7fd72c10a1f0 2024-05-27T07:57:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.413+0000 7fd73475e640 1 -- 172.21.15.114:0/3881393476 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72c118720 con 0x7fd72c10a1f0 2024-05-27T07:57:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.416+0000 7fd73b60e740 1 Processor -- start 2024-05-27T07:57:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.416+0000 7fd7237fe640 1 -- 172.21.15.114:0/3881393476 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd728011ce0 con 0x7fd72c10a1f0 2024-05-27T07:57:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.417+0000 7fd7237fe640 1 -- 172.21.15.114:0/3881393476 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728010de0 con 0x7fd72c10a1f0 2024-05-27T07:57:20.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.419+0000 7fd73b60e740 1 -- start start 2024-05-27T07:57:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.420+0000 7fd73b60e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106f71b0 0x564110a983d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.420+0000 7fd73b60e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106ce2f0 0x564110a98910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.420+0000 7fd73b60e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106d64d0 0x564110a98e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.420+0000 7fd73b60e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564110a0ac30 con 0x5641106ce2f0 2024-05-27T07:57:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.420+0000 7fd73b60e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5641109fa7d0 con 0x5641106d64d0 2024-05-27T07:57:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.420+0000 7fd73b60e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5641109fe810 con 0x5641106f71b0 2024-05-27T07:57:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.421+0000 7fd731cf9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106d64d0 0x564110a98e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.421+0000 7fd731cf9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106d64d0 0x564110a98e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:38584/0 (socket says 172.21.15.114:38584) 2024-05-27T07:57:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.421+0000 7fd731cf9640 1 -- 172.21.15.114:0/1256563240 learned_addr learned my addr 172.21.15.114:0/1256563240 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.421+0000 7fd732cfb640 1 --2- 172.21.15.114:0/1256563240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106f71b0 0x564110a983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.421+0000 7fd731cf9640 1 -- 172.21.15.114:0/1256563240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106f71b0 msgr2=0x564110a983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.421+0000 7fd731cf9640 1 --2- 172.21.15.114:0/1256563240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106f71b0 0x564110a983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.421+0000 7fd731cf9640 1 -- 172.21.15.114:0/1256563240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106ce2f0 msgr2=0x564110a98910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:57:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.421+0000 7fd731cf9640 1 --2- 172.21.15.114:0/1256563240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106ce2f0 0x564110a98910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.421+0000 7fd731cf9640 1 -- 172.21.15.114:0/1256563240 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56411070f210 con 0x5641106d64d0 2024-05-27T07:57:20.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.423+0000 7fd7237fe640 1 -- 172.21.15.114:0/3881393476 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fd728018020 con 0x7fd72c10a1f0 2024-05-27T07:57:20.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.423+0000 7fd731cf9640 1 --2- 172.21.15.114:0/1256563240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106d64d0 0x564110a98e50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd72400e270 tx=0x7fd724009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a6921630e7b0503 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:20.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.424+0000 7fd7237fe640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7180b2b70 0x7fd7180b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.425+0000 7fd7237fe640 1 -- 172.21.15.114:0/3881393476 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7fd7280e1f50 con 0x7fd72c10a1f0 2024-05-27T07:57:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.427+0000 7fd720ff9640 1 -- 172.21.15.114:0/1256563240 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724018020 con 0x5641106d64d0 2024-05-27T07:57:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.427+0000 7fd720ff9640 1 -- 172.21.15.114:0/1256563240 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd724014d40 con 0x5641106d64d0 2024-05-27T07:57:20.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.428+0000 7fd73b60e740 1 -- 172.21.15.114:0/1256563240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106d64d0 msgr2=0x564110a98e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:20.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.428+0000 7fd73b60e740 1 --2- 172.21.15.114:0/1256563240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106d64d0 0x564110a98e50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd72400e270 tx=0x7fd724009790 comp rx=0 tx=0).stop 2024-05-27T07:57:20.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.429+0000 7fd732cfb640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7180b2b70 0x7fd7180b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.430+0000 7fd73b60e740 1 -- 172.21.15.114:0/1256563240 shutdown_connections 2024-05-27T07:57:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.430+0000 7fd73b60e740 1 --2- 172.21.15.114:0/1256563240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106d64d0 0x564110a98e50 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.430+0000 7fd73b60e740 1 --2- 172.21.15.114:0/1256563240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106ce2f0 0x564110a98910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.430+0000 7fd73b60e740 1 --2- 172.21.15.114:0/1256563240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106f71b0 0x564110a983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.430+0000 7fd73b60e740 1 -- 172.21.15.114:0/1256563240 >> 172.21.15.114:0/1256563240 conn(0x5641106eb300 msgr2=0x564110a94b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:20.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.433+0000 7fd73b60e740 1 -- 172.21.15.114:0/1256563240 shutdown_connections 2024-05-27T07:57:20.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.433+0000 7fd732cfb640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7180b2b70 0x7fd7180b5080 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd71c02aac0 tx=0x7fd71c005df0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.434+0000 7fd73b60e740 1 -- 172.21.15.114:0/1256563240 wait complete. 2024-05-27T07:57:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.435+0000 7fd73b60e740 1 -- start start 2024-05-27T07:57:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.435+0000 7fd73b60e740 1 Processor -- start 2024-05-27T07:57:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.435+0000 7fd73b60e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106ce2f0 0x564110aa29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.435+0000 7fd73b60e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aa2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.435+0000 7fd73b60e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106f71b0 0x564110aae8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.435+0000 7fd73b60e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5641106e0790 con 0x5641106ce2f0 2024-05-27T07:57:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.435+0000 7fd73b60e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5641109b6890 con 0x5641106f71b0 2024-05-27T07:57:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.435+0000 7fd73b60e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564110aa5550 con 0x5641106d64d0 2024-05-27T07:57:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.435+0000 7fd73b60e740 10 client.0 _ll_register_callbacks cb 0x56411070ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.436+0000 7fd731cf9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aa2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.436+0000 7fd731cf9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aa2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:50926/0 (socket says 172.21.15.114:50926) 2024-05-27T07:57:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.436+0000 7fd731cf9640 1 -- 172.21.15.114:0/87205572 learned_addr learned my addr 172.21.15.114:0/87205572 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.436+0000 7fd732cfb640 1 --2- 172.21.15.114:0/87205572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106f71b0 0x564110aae8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.436+0000 7fd731cf9640 1 -- 172.21.15.114:0/87205572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106f71b0 msgr2=0x564110aae8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.436+0000 7fd731cf9640 1 --2- 172.21.15.114:0/87205572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106f71b0 0x564110aae8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.436+0000 7fd731cf9640 1 -- 172.21.15.114:0/87205572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106ce2f0 msgr2=0x564110aa29f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:57:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.436+0000 7fd731cf9640 1 --2- 172.21.15.114:0/87205572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106ce2f0 0x564110aa29f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.436+0000 7fd731cf9640 1 -- 172.21.15.114:0/87205572 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641107db620 con 0x5641106d64d0 2024-05-27T07:57:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.437+0000 7fd731cf9640 1 --2- 172.21.15.114:0/87205572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aa2f30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd72400f010 tx=0x7fd724009e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22403868436e2c7e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.437+0000 7fd715ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:57:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.437+0000 7fd715ffb640 1 -- 172.21.15.114:0/87205572 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724018030 con 0x5641106d64d0 2024-05-27T07:57:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.437+0000 7fd73b60e740 1 -- 172.21.15.114:0/87205572 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd724014d40 con 0x5641106d64d0 2024-05-27T07:57:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.437+0000 7fd715ffb640 1 -- 172.21.15.114:0/87205572 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7fd724014690 con 0x5641106d64d0 2024-05-27T07:57:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.439+0000 7fd715ffb640 1 -- 172.21.15.114:0/87205572 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7fd724003e20 con 0x5641106d64d0 2024-05-27T07:57:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.441+0000 7fd73b60e740 10 client.28900 fetch_fsmap learned FSMap version 98 2024-05-27T07:57:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.441+0000 7fd73b60e740 1 -- 172.21.15.114:0/87205572 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=4+}) -- 0x564110ab1790 con 0x5641106d64d0 2024-05-27T07:57:20.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.442+0000 7fd715ffb640 1 -- 172.21.15.114:0/87205572 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd724014d40 con 0x5641106d64d0 2024-05-27T07:57:20.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.443+0000 7fd715ffb640 1 -- 172.21.15.114:0/87205572 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 98) ==== 4095+0+0 (secure 0 0 0) 0x7fd724027030 con 0x5641106d64d0 2024-05-27T07:57:20.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.443+0000 7fd73b60e740 10 client.28900 fetch_fsmap finished waiting for FSMap version 98 2024-05-27T07:57:20.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.443+0000 7fd73b60e740 10 client.28900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:20.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.443+0000 7fd73b60e740 20 client.28900 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:20.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.443+0000 7fd73b60e740 1 --2- 172.21.15.114:0/87205572 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5641106b5d80 0x564110ab5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.443+0000 7fd73b60e740 4 client.28900 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:57:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.443+0000 7fd73b60e740 1 -- 172.21.15.114:0/87205572 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564110a0cb80 con 0x5641106b5d80 2024-05-27T07:57:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.443+0000 7fd7324fa640 1 --2- 172.21.15.114:0/87205572 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5641106b5d80 0x564110ab5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.445+0000 7fd7324fa640 1 --2- 172.21.15.114:0/87205572 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5641106b5d80 0x564110ab5000 crc :-1 s=READY pgs=472 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ad0090594c2c605 server_cookie=ca6b92430ef3f07d in_seq=0 out_seq=0 2024-05-27T07:57:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.445+0000 7fd715ffb640 10 client.28900 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:20.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.451+0000 7fd715ffb640 1 -- 172.21.15.114:0/87205572 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564110a0cb80 con 0x5641106b5d80 2024-05-27T07:57:20.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.452+0000 7fd715ffb640 10 client.28900 handle_command_reply: tid=0 2024-05-27T07:57:20.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.452+0000 7fd73b60e740 1 client.28900 shutdown 2024-05-27T07:57:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.452+0000 7fd73b60e740 1 -- 172.21.15.114:0/87205572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 msgr2=0x564110aa2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.452+0000 7fd73b60e740 1 --2- 172.21.15.114:0/87205572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aa2f30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd72400f010 tx=0x7fd724009e90 comp rx=0 tx=0).stop 2024-05-27T07:57:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.452+0000 7fd73b60e740 1 -- 172.21.15.114:0/87205572 shutdown_connections 2024-05-27T07:57:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.452+0000 7fd73b60e740 1 --2- 172.21.15.114:0/87205572 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5641106b5d80 0x564110ab5000 crc :-1 s=READY pgs=472 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.452+0000 7fd73b60e740 1 --2- 172.21.15.114:0/87205572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106f71b0 0x564110aae8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.452+0000 7fd73b60e740 1 --2- 172.21.15.114:0/87205572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aa2f30 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.452+0000 7fd73b60e740 1 --2- 172.21.15.114:0/87205572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106ce2f0 0x564110aa29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.452+0000 7fd73b60e740 1 -- 172.21.15.114:0/87205572 >> 172.21.15.114:0/87205572 conn(0x5641106eb300 msgr2=0x564110a9a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.452+0000 7fd715ffb640 0 client.28900 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.455+0000 7fd73b60e740 1 -- 172.21.15.114:0/87205572 shutdown_connections 2024-05-27T07:57:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.455+0000 7fd73b60e740 1 -- 172.21.15.114:0/87205572 wait complete. 2024-05-27T07:57:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.457+0000 7fd73b60e740 20 client.28900 trim_cache size 0 max 16384 2024-05-27T07:57:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.487+0000 7fd73b60e740 1 Processor -- start 2024-05-27T07:57:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.487+0000 7fd73b60e740 1 -- start start 2024-05-27T07:57:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.489+0000 7fd73b60e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106ce2f0 0x564110a9c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.489+0000 7fd73b60e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aae640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.489+0000 7fd73b60e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106f71b0 0x564110aaeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.489+0000 7fd73b60e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd71000ff80 con 0x5641106ce2f0 2024-05-27T07:57:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.489+0000 7fd73b60e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd710010650 con 0x5641106f71b0 2024-05-27T07:57:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.489+0000 7fd73b60e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd71000ef20 con 0x5641106d64d0 2024-05-27T07:57:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.490+0000 7fd732cfb640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106f71b0 0x564110aaeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.490+0000 7fd731cf9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.490+0000 7fd731cf9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:50940/0 (socket says 172.21.15.114:50940) 2024-05-27T07:57:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.490+0000 7fd731cf9640 1 -- 172.21.15.114:0/3769722712 learned_addr learned my addr 172.21.15.114:0/3769722712 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.491+0000 7fd731cf9640 1 -- 172.21.15.114:0/3769722712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106f71b0 msgr2=0x564110aaeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.491+0000 7fd731cf9640 1 --2- 172.21.15.114:0/3769722712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106f71b0 0x564110aaeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.491+0000 7fd731cf9640 1 -- 172.21.15.114:0/3769722712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106ce2f0 msgr2=0x564110a9c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.491+0000 7fd731cf9640 1 --2- 172.21.15.114:0/3769722712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106ce2f0 0x564110a9c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.491+0000 7fd731cf9640 1 -- 172.21.15.114:0/3769722712 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564110a95aa0 con 0x5641106d64d0 2024-05-27T07:57:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.491+0000 7fd731cf9640 1 --2- 172.21.15.114:0/3769722712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aae640 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd724007030 tx=0x7fd724013e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=482b5dea0d993444 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.492+0000 7fd717fff640 1 -- 172.21.15.114:0/3769722712 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724018040 con 0x5641106d64d0 2024-05-27T07:57:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.492+0000 7fd717fff640 1 -- 172.21.15.114:0/3769722712 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd724009790 con 0x5641106d64d0 2024-05-27T07:57:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.492+0000 7fd717fff640 1 -- 172.21.15.114:0/3769722712 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724014a10 con 0x5641106d64d0 2024-05-27T07:57:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.493+0000 7fd73b60e740 1 -- 172.21.15.114:0/3769722712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 msgr2=0x564110aae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.493+0000 7fd73b60e740 1 --2- 172.21.15.114:0/3769722712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aae640 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd724007030 tx=0x7fd724013e90 comp rx=0 tx=0).stop 2024-05-27T07:57:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.494+0000 7fd73b60e740 1 -- 172.21.15.114:0/3769722712 shutdown_connections 2024-05-27T07:57:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.494+0000 7fd73b60e740 1 --2- 172.21.15.114:0/3769722712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106f71b0 0x564110aaeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.494+0000 7fd73b60e740 1 --2- 172.21.15.114:0/3769722712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aae640 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.494+0000 7fd73b60e740 1 --2- 172.21.15.114:0/3769722712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106ce2f0 0x564110a9c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.494+0000 7fd73b60e740 1 -- 172.21.15.114:0/3769722712 >> 172.21.15.114:0/3769722712 conn(0x5641106eb300 msgr2=0x564110a9be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.494+0000 7fd73b60e740 1 -- 172.21.15.114:0/3769722712 shutdown_connections 2024-05-27T07:57:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.494+0000 7fd73b60e740 1 -- 172.21.15.114:0/3769722712 wait complete. 2024-05-27T07:57:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.495+0000 7fd73b60e740 1 -- start start 2024-05-27T07:57:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.495+0000 7fd73b60e740 1 Processor -- start 2024-05-27T07:57:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.496+0000 7fd73b60e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106ce2f0 0x564110aab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.496+0000 7fd73b60e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aa6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.498 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.496+0000 7fd73b60e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106f71b0 0x564110aa6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.498 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.496+0000 7fd7324fa640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106ce2f0 0x564110aab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.498 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.496+0000 7fd7324fa640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106ce2f0 0x564110aab030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:38614/0 (socket says 172.21.15.114:38614) 2024-05-27T07:57:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.496+0000 7fd731cf9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aa6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.496+0000 7fd731cf9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aa6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:50942/0 (socket says 172.21.15.114:50942) 2024-05-27T07:57:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.497+0000 7fd7324fa640 1 -- 172.21.15.114:0/2404920745 learned_addr learned my addr 172.21.15.114:0/2404920745 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.497+0000 7fd732cfb640 1 --2- 172.21.15.114:0/2404920745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106f71b0 0x564110aa6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.497+0000 7fd73b60e740 1 -- 172.21.15.114:0/2404920745 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd71000e850 con 0x5641106f71b0 2024-05-27T07:57:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.497+0000 7fd73b60e740 1 -- 172.21.15.114:0/2404920745 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd71000e200 con 0x5641106ce2f0 2024-05-27T07:57:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.497+0000 7fd73b60e740 1 -- 172.21.15.114:0/2404920745 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564110aa0d30 con 0x5641106d64d0 2024-05-27T07:57:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.497+0000 7fd73b60e740 10 client.0 _ll_register_callbacks cb 0x7fd72400e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.497+0000 7fd7324fa640 1 -- 172.21.15.114:0/2404920745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 msgr2=0x564110aa6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.497+0000 7fd7324fa640 1 --2- 172.21.15.114:0/2404920745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aa6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.497+0000 7fd7324fa640 1 -- 172.21.15.114:0/2404920745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106f71b0 msgr2=0x564110aa6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.497+0000 7fd7324fa640 1 --2- 172.21.15.114:0/2404920745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106f71b0 0x564110aa6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.497+0000 7fd7324fa640 1 -- 172.21.15.114:0/2404920745 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564110aa7130 con 0x5641106ce2f0 2024-05-27T07:57:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.497+0000 7fd73b60e740 1 -- 172.21.15.114:0/2404920745 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564110a06360 con 0x5641106ce2f0 2024-05-27T07:57:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.498+0000 7fd7324fa640 1 --2- 172.21.15.114:0/2404920745 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106ce2f0 0x564110aab030 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd728010910 tx=0x7fd7280a8a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93a71b97b16f24f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.498+0000 7fd715ffb640 10 client.28745 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:57:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.498+0000 7fd715ffb640 1 -- 172.21.15.114:0/2404920745 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7280c7020 con 0x5641106ce2f0 2024-05-27T07:57:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.498+0000 7fd715ffb640 1 -- 172.21.15.114:0/2404920745 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7fd7280a5110 con 0x5641106ce2f0 2024-05-27T07:57:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.498+0000 7fd715ffb640 1 -- 172.21.15.114:0/2404920745 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7fd7280ad230 con 0x5641106ce2f0 2024-05-27T07:57:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.499+0000 7fd73b60e740 10 client.28745 fetch_fsmap learned FSMap version 98 2024-05-27T07:57:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.499+0000 7fd73b60e740 1 -- 172.21.15.114:0/2404920745 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({fsmap=98}) -- 0x564110b1d600 con 0x5641106ce2f0 2024-05-27T07:57:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.499+0000 7fd715ffb640 1 -- 172.21.15.114:0/2404920745 <== mon.2 v2:172.21.15.163:3300/0 4 ==== fsmap(e 98) ==== 4095+0+0 (secure 0 0 0) 0x7fd7280df460 con 0x5641106ce2f0 2024-05-27T07:57:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.499+0000 7fd73b60e740 10 client.28745 fetch_fsmap finished waiting for FSMap version 98 2024-05-27T07:57:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.499+0000 7fd73b60e740 10 client.28745 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.499+0000 7fd73b60e740 20 client.28745 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.499+0000 7fd73b60e740 1 --2- 172.21.15.114:0/2404920745 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56411068b1c0 0x564110b1fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.499+0000 7fd73b60e740 4 client.28745 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:57:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.499+0000 7fd73b60e740 1 -- 172.21.15.114:0/2404920745 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5641109805f0 con 0x56411068b1c0 2024-05-27T07:57:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.500+0000 7fd731cf9640 1 --2- 172.21.15.114:0/2404920745 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56411068b1c0 0x564110b1fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.501+0000 7fd715ffb640 10 client.28745 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:20.501+0000 7fd731cf9640 1 --2- 172.21.15.114:0/2404920745 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56411068b1c0 0x564110b1fd30 crc :-1 s=READY pgs=473 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e6ec278798947cb server_cookie=45e283df9cf6cb63 in_seq=0 out_seq=0 2024-05-27T07:57:20.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 50650 11.85 MB/sec execute 92 sec latency 164.910 ms 2024-05-27T07:57:21.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:20 smithi163 ceph-mon[36451]: pgmap v601: 129 pgs: 10 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 102 active+clean; 341 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 363 op/s 2024-05-27T07:57:21.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:20 smithi186 ceph-mon[36647]: pgmap v601: 129 pgs: 10 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 102 active+clean; 341 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 363 op/s 2024-05-27T07:57:21.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:20 smithi114 ceph-mon[32038]: pgmap v601: 129 pgs: 10 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 102 active+clean; 341 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 363 op/s 2024-05-27T07:57:21.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 50690 11.77 MB/sec execute 93 sec latency 903.039 ms 2024-05-27T07:57:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.885+0000 7fd715ffb640 1 -- 172.21.15.114:0/2404920745 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5641109805f0 con 0x56411068b1c0 2024-05-27T07:57:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.885+0000 7fd715ffb640 10 client.28745 handle_command_reply: tid=0 2024-05-27T07:57:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.885+0000 7fd73b60e740 1 client.28745 shutdown 2024-05-27T07:57:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.885+0000 7fd73b60e740 1 -- 172.21.15.114:0/2404920745 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106ce2f0 msgr2=0x564110aab030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.885+0000 7fd73b60e740 1 --2- 172.21.15.114:0/2404920745 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106ce2f0 0x564110aab030 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd728010910 tx=0x7fd7280a8a50 comp rx=0 tx=0).stop 2024-05-27T07:57:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.885+0000 7fd73b60e740 1 -- 172.21.15.114:0/2404920745 shutdown_connections 2024-05-27T07:57:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.886+0000 7fd73b60e740 1 --2- 172.21.15.114:0/2404920745 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56411068b1c0 0x564110b1fd30 crc :-1 s=READY pgs=473 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.886+0000 7fd715ffb640 0 client.28745 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.886+0000 7fd73b60e740 1 --2- 172.21.15.114:0/2404920745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5641106f71b0 0x564110aa6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.886+0000 7fd73b60e740 1 --2- 172.21.15.114:0/2404920745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641106d64d0 0x564110aa6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.886+0000 7fd73b60e740 1 --2- 172.21.15.114:0/2404920745 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5641106ce2f0 0x564110aab030 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.886+0000 7fd73b60e740 1 -- 172.21.15.114:0/2404920745 >> 172.21.15.114:0/2404920745 conn(0x5641106eb300 msgr2=0x564110a98160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.886+0000 7fd73b60e740 1 -- 172.21.15.114:0/2404920745 shutdown_connections 2024-05-27T07:57:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.886+0000 7fd73b60e740 1 -- 172.21.15.114:0/2404920745 wait complete. 2024-05-27T07:57:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.886+0000 7fd73b60e740 20 client.28745 trim_cache size 0 max 16384 2024-05-27T07:57:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:57:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:57:21.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:57:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.887+0000 7fd73475e640 1 -- 172.21.15.114:0/3881393476 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7180b2b70 msgr2=0x7fd7180b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.887+0000 7fd73475e640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7180b2b70 0x7fd7180b5080 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd71c02aac0 tx=0x7fd71c005df0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.887+0000 7fd73475e640 1 -- 172.21.15.114:0/3881393476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd72c10a1f0 msgr2=0x7fd72c1134e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.887+0000 7fd73475e640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd72c10a1f0 0x7fd72c1134e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd728009fd0 tx=0x7fd72800b870 comp rx=0 tx=0).stop 2024-05-27T07:57:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.887+0000 7fd73475e640 1 -- 172.21.15.114:0/3881393476 shutdown_connections 2024-05-27T07:57:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.887+0000 7fd73475e640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7180b2b70 0x7fd7180b5080 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.887+0000 7fd73475e640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd72c11d240 0x7fd72c113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.887+0000 7fd73475e640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd72c11a8e0 0x7fd72c113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.888+0000 7fd73475e640 1 --2- 172.21.15.114:0/3881393476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd72c10a1f0 0x7fd72c1134e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.888+0000 7fd73475e640 1 -- 172.21.15.114:0/3881393476 >> 172.21.15.114:0/3881393476 conn(0x7fd72c06e8f0 msgr2=0x7fd72c11acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.889+0000 7fd73475e640 1 -- 172.21.15.114:0/3881393476 shutdown_connections 2024-05-27T07:57:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:21.889+0000 7fd73475e640 1 -- 172.21.15.114:0/3881393476 wait complete. 2024-05-27T07:57:21.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.927+0000 7f9ba6b81700 1 -- 172.21.15.114:0/966218201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ba010b010 msgr2=0x7f9ba010b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:21.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.927+0000 7f9ba6b81700 1 --2- 172.21.15.114:0/966218201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ba010b010 0x7f9ba010b420 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009580 tx=0x7f9b94009890 comp rx=0 tx=0).stop 2024-05-27T07:57:21.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.927+0000 7f9ba6b81700 1 -- 172.21.15.114:0/966218201 shutdown_connections 2024-05-27T07:57:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.927+0000 7f9ba6b81700 1 --2- 172.21.15.114:0/966218201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ba0110290 0x7f9ba0117bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.927+0000 7f9ba6b81700 1 --2- 172.21.15.114:0/966218201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ba010b010 0x7f9ba010b420 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.927+0000 7f9ba6b81700 1 --2- 172.21.15.114:0/966218201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ba010f940 0x7f9ba010fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.927+0000 7f9ba6b81700 1 -- 172.21.15.114:0/966218201 >> 172.21.15.114:0/966218201 conn(0x7f9ba006f980 msgr2=0x7f9ba006fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.927+0000 7f9ba6b81700 1 -- 172.21.15.114:0/966218201 shutdown_connections 2024-05-27T07:57:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.927+0000 7f9ba6b81700 1 -- 172.21.15.114:0/966218201 wait complete. 2024-05-27T07:57:21.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.928+0000 7f9ba6b81700 1 Processor -- start 2024-05-27T07:57:21.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.929+0000 7f9ba6b81700 1 -- start start 2024-05-27T07:57:21.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.929+0000 7f9ba6b81700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ba010b010 0x7f9ba01a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:21.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.930+0000 7f9ba6b81700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ba010f940 0x7f9ba01a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:21.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.930+0000 7f9ba6b81700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ba0110290 0x7f9ba01a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:21.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.930+0000 7f9b9ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ba010f940 0x7f9ba01a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:21.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.930+0000 7f9b9ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ba010f940 0x7f9ba01a1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:38628/0 (socket says 172.21.15.114:38628) 2024-05-27T07:57:21.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.930+0000 7f9ba511e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ba0110290 0x7f9ba01a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:21.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.930+0000 7f9ba511e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ba0110290 0x7f9ba01a5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49510/0 (socket says 172.21.15.114:49510) 2024-05-27T07:57:21.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.930+0000 7f9ba511e700 1 -- 172.21.15.114:0/1692374141 learned_addr learned my addr 172.21.15.114:0/1692374141 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:21.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.930+0000 7f9ba6b81700 1 -- 172.21.15.114:0/1692374141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba011ac20 con 0x7f9ba0110290 2024-05-27T07:57:21.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.930+0000 7f9ba6b81700 1 -- 172.21.15.114:0/1692374141 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba011aaa0 con 0x7f9ba010f940 2024-05-27T07:57:21.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.930+0000 7f9ba6b81700 1 -- 172.21.15.114:0/1692374141 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba011ada0 con 0x7f9ba010b010 2024-05-27T07:57:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.931+0000 7f9ba491d700 1 --2- 172.21.15.114:0/1692374141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ba010b010 0x7f9ba01a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.931+0000 7f9ba511e700 1 -- 172.21.15.114:0/1692374141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ba010b010 msgr2=0x7f9ba01a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.931+0000 7f9ba511e700 1 --2- 172.21.15.114:0/1692374141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ba010b010 0x7f9ba01a1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.931+0000 7f9ba511e700 1 -- 172.21.15.114:0/1692374141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ba010f940 msgr2=0x7f9ba01a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.931+0000 7f9ba511e700 1 --2- 172.21.15.114:0/1692374141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ba010f940 0x7f9ba01a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.931+0000 7f9ba511e700 1 -- 172.21.15.114:0/1692374141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba01a6340 con 0x7f9ba0110290 2024-05-27T07:57:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.932+0000 7f9ba511e700 1 --2- 172.21.15.114:0/1692374141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ba0110290 0x7f9ba01a5ca0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800b780 tx=0x7f9b9800ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.933+0000 7f9b9dffb700 1 -- 172.21.15.114:0/1692374141 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98012af0 con 0x7f9ba0110290 2024-05-27T07:57:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.933+0000 7f9b9dffb700 1 -- 172.21.15.114:0/1692374141 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b98012c90 con 0x7f9ba0110290 2024-05-27T07:57:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.933+0000 7f9ba6b81700 1 -- 172.21.15.114:0/1692374141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba01a6630 con 0x7f9ba0110290 2024-05-27T07:57:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.933+0000 7f9b9dffb700 1 -- 172.21.15.114:0/1692374141 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b980075b0 con 0x7f9ba0110290 2024-05-27T07:57:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.933+0000 7f9ba6b81700 1 -- 172.21.15.114:0/1692374141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba01a6ac0 con 0x7f9ba0110290 2024-05-27T07:57:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.935+0000 7f9ba6b81700 1 -- 172.21.15.114:0/1692374141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba00512b0 con 0x7f9ba0110290 2024-05-27T07:57:21.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.938+0000 7f9b9dffb700 1 -- 172.21.15.114:0/1692374141 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f9b9801b940 con 0x7f9ba0110290 2024-05-27T07:57:21.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.939+0000 7f9b9dffb700 1 --2- 172.21.15.114:0/1692374141 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9b900b2aa0 0x7f9b900b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:21.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.939+0000 7f9b9dffb700 1 -- 172.21.15.114:0/1692374141 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9642+0+0 (secure 0 0 0) 0x7f9b980e6ad0 con 0x7f9ba0110290 2024-05-27T07:57:21.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.940+0000 7f9ba491d700 1 --2- 172.21.15.114:0/1692374141 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9b900b2aa0 0x7f9b900b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:21.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.941+0000 7f9ba491d700 1 --2- 172.21.15.114:0/1692374141 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9b900b2aa0 0x7f9b900b4fb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009b10 tx=0x7f9b9400c650 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:21.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:21.946+0000 7f9b9dffb700 1 -- 172.21.15.114:0/1692374141 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9b98029080 con 0x7f9ba0110290 2024-05-27T07:57:22.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.314+0000 7f9ba6b81700 1 -- 172.21.15.114:0/1692374141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ba01a27d0 con 0x7f9ba0110290 2024-05-27T07:57:22.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.315+0000 7f9b9dffb700 1 -- 172.21.15.114:0/1692374141 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) ==== 94+0+6036 (secure 0 0 0) 0x7f9ba01a27d0 con 0x7f9ba0110290 2024-05-27T07:57:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:57:22.317 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":19404,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2267591646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2267591646},{"type":"v1","addr":"172.21.15.114:6839","nonce":2267591646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:57:14.419576+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14574":{"gid":14574,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6839/2637229588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2637229588},{"type":"v1","addr":"172.21.15.114:6839","nonce":2637229588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:57:22.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.317+0000 7f9ba6b81700 1 -- 172.21.15.114:0/1692374141 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9b900b2aa0 msgr2=0x7f9b900b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:22.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.317+0000 7f9ba6b81700 1 --2- 172.21.15.114:0/1692374141 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9b900b2aa0 0x7f9b900b4fb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009b10 tx=0x7f9b9400c650 comp rx=0 tx=0).stop 2024-05-27T07:57:22.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.317+0000 7f9ba6b81700 1 -- 172.21.15.114:0/1692374141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ba0110290 msgr2=0x7f9ba01a5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:22.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.317+0000 7f9ba6b81700 1 --2- 172.21.15.114:0/1692374141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ba0110290 0x7f9ba01a5ca0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800b780 tx=0x7f9b9800ba90 comp rx=0 tx=0).stop 2024-05-27T07:57:22.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.317+0000 7f9ba6b81700 1 -- 172.21.15.114:0/1692374141 shutdown_connections 2024-05-27T07:57:22.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.317+0000 7f9ba6b81700 1 --2- 172.21.15.114:0/1692374141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ba0110290 0x7f9ba01a5ca0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:22.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.317+0000 7f9ba6b81700 1 --2- 172.21.15.114:0/1692374141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ba010b010 0x7f9ba01a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:22.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.317+0000 7f9ba6b81700 1 --2- 172.21.15.114:0/1692374141 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9b900b2aa0 0x7f9b900b4fb0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:22.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.317+0000 7f9ba6b81700 1 --2- 172.21.15.114:0/1692374141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ba010f940 0x7f9ba01a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:22.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.317+0000 7f9ba6b81700 1 -- 172.21.15.114:0/1692374141 >> 172.21.15.114:0/1692374141 conn(0x7f9ba006f980 msgr2=0x7f9ba010cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:22.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.318+0000 7f9ba6b81700 1 -- 172.21.15.114:0/1692374141 shutdown_connections 2024-05-27T07:57:22.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:22.318+0000 7f9ba6b81700 1 -- 172.21.15.114:0/1692374141 wait complete. 2024-05-27T07:57:22.320 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 98 2024-05-27T07:57:22.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 50903 11.76 MB/sec execute 94 sec latency 912.910 ms 2024-05-27T07:57:23.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:22 smithi163 ceph-mon[36451]: pgmap v602: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 351 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.5 MiB/s wr, 479 op/s 2024-05-27T07:57:23.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:22 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1692374141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:57:23.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:22 smithi186 ceph-mon[36647]: pgmap v602: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 351 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.5 MiB/s wr, 479 op/s 2024-05-27T07:57:23.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:22 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1692374141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:57:23.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:22 smithi114 ceph-mon[32038]: pgmap v602: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 351 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.5 MiB/s wr, 479 op/s 2024-05-27T07:57:23.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:22 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1692374141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:57:23.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:57:23.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 51255 11.79 MB/sec execute 95 sec latency 182.363 ms 2024-05-27T07:57:23.766 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:57:24.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 51734 11.87 MB/sec execute 96 sec latency 159.000 ms 2024-05-27T07:57:25.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:24 smithi163 ceph-mon[36451]: pgmap v603: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 445 op/s 2024-05-27T07:57:25.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:24 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:57:25.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:24 smithi163 ceph-mon[36451]: osdmap e152: 12 total, 12 up, 12 in 2024-05-27T07:57:25.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:24 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:57:25.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:24 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:57:25.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:24 smithi163 ceph-mon[36451]: pgmap v605: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.1 MiB/s wr, 481 op/s 2024-05-27T07:57:25.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:24 smithi163 ceph-mon[36451]: pgmap v606: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 421 op/s 2024-05-27T07:57:25.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:24 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:57:25.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:24 smithi186 ceph-mon[36647]: pgmap v603: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 445 op/s 2024-05-27T07:57:25.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:24 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:57:25.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:24 smithi186 ceph-mon[36647]: osdmap e152: 12 total, 12 up, 12 in 2024-05-27T07:57:25.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:24 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:57:25.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:24 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:57:25.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:24 smithi186 ceph-mon[36647]: pgmap v605: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.1 MiB/s wr, 481 op/s 2024-05-27T07:57:25.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:24 smithi186 ceph-mon[36647]: pgmap v606: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 421 op/s 2024-05-27T07:57:25.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:24 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:57:25.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:24 smithi114 ceph-mon[32038]: pgmap v603: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 445 op/s 2024-05-27T07:57:25.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:24 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:57:25.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:24 smithi114 ceph-mon[32038]: osdmap e152: 12 total, 12 up, 12 in 2024-05-27T07:57:25.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:24 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:57:25.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:24 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:57:25.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:24 smithi114 ceph-mon[32038]: pgmap v605: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.1 MiB/s wr, 481 op/s 2024-05-27T07:57:25.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:24 smithi114 ceph-mon[32038]: pgmap v606: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 421 op/s 2024-05-27T07:57:25.262 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:24 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:57:25.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 52185 12.11 MB/sec execute 97 sec latency 172.273 ms 2024-05-27T07:57:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.922+0000 7fdbe2ffd700 1 --2- 172.21.15.114:0/1035606226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbe410b000 0x7fdbe410b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:57:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.923+0000 7fdbe974c700 1 -- 172.21.15.114:0/1035606226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbe4110280 msgr2=0x7fdbe4117be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.923+0000 7fdbe974c700 1 --2- 172.21.15.114:0/1035606226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbe4110280 0x7fdbe4117be0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fdbd80095b0 tx=0x7fdbd80098c0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.925+0000 7fdbe974c700 1 -- 172.21.15.114:0/1035606226 shutdown_connections 2024-05-27T07:57:25.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.925+0000 7fdbe974c700 1 --2- 172.21.15.114:0/1035606226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbe4110280 0x7fdbe4117be0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.925+0000 7fdbe974c700 1 --2- 172.21.15.114:0/1035606226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbe410b000 0x7fdbe410b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.925+0000 7fdbe974c700 1 --2- 172.21.15.114:0/1035606226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbe410f930 0x7fdbe410fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.925+0000 7fdbe974c700 1 -- 172.21.15.114:0/1035606226 >> 172.21.15.114:0/1035606226 conn(0x7fdbe406f4c0 msgr2=0x7fdbe406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:25.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.927+0000 7fdbe974c700 1 -- 172.21.15.114:0/1035606226 shutdown_connections 2024-05-27T07:57:25.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.928+0000 7fdbe974c700 1 -- 172.21.15.114:0/1035606226 wait complete. 2024-05-27T07:57:25.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.929+0000 7fdbe974c700 1 Processor -- start 2024-05-27T07:57:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.929+0000 7fdbe974c700 1 -- start start 2024-05-27T07:57:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.930+0000 7fdbe974c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbe410b000 0x7fdbe41a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.930+0000 7fdbe974c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbe410f930 0x7fdbe41a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.930+0000 7fdbe974c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbe4110280 0x7fdbe41a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.930+0000 7fdbe2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbe410b000 0x7fdbe41a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.930+0000 7fdbe2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbe410b000 0x7fdbe41a13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51340/0 (socket says 172.21.15.114:51340) 2024-05-27T07:57:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.930+0000 7fdbe2ffd700 1 -- 172.21.15.114:0/662084414 learned_addr learned my addr 172.21.15.114:0/662084414 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.930+0000 7fdbe27fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbe410f930 0x7fdbe41a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.931+0000 7fdbe974c700 1 -- 172.21.15.114:0/662084414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe411ab60 con 0x7fdbe410f930 2024-05-27T07:57:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.931+0000 7fdbe974c700 1 -- 172.21.15.114:0/662084414 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe411a9e0 con 0x7fdbe410b000 2024-05-27T07:57:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.931+0000 7fdbe974c700 1 -- 172.21.15.114:0/662084414 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe411ace0 con 0x7fdbe4110280 2024-05-27T07:57:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.931+0000 7fdbe27fc700 1 -- 172.21.15.114:0/662084414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbe4110280 msgr2=0x7fdbe41a5be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:57:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.931+0000 7fdbe27fc700 1 --2- 172.21.15.114:0/662084414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbe4110280 0x7fdbe41a5be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.931+0000 7fdbe27fc700 1 -- 172.21.15.114:0/662084414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbe410b000 msgr2=0x7fdbe41a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:25.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.931+0000 7fdbe27fc700 1 --2- 172.21.15.114:0/662084414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbe410b000 0x7fdbe41a13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.931+0000 7fdbe27fc700 1 -- 172.21.15.114:0/662084414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe41a6280 con 0x7fdbe410f930 2024-05-27T07:57:25.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.932+0000 7fdbe27fc700 1 --2- 172.21.15.114:0/662084414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbe410f930 0x7fdbe41a1950 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400d720 tx=0x7fdbd400da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:25.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.933+0000 7fdbcbfff700 1 -- 172.21.15.114:0/662084414 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4017070 con 0x7fdbe410f930 2024-05-27T07:57:25.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.933+0000 7fdbe974c700 1 -- 172.21.15.114:0/662084414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbe41a6570 con 0x7fdbe410f930 2024-05-27T07:57:25.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.933+0000 7fdbe974c700 1 -- 172.21.15.114:0/662084414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbe41ade90 con 0x7fdbe410f930 2024-05-27T07:57:25.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.933+0000 7fdbcbfff700 1 -- 172.21.15.114:0/662084414 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbd4005020 con 0x7fdbe410f930 2024-05-27T07:57:25.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.933+0000 7fdbcbfff700 1 -- 172.21.15.114:0/662084414 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4020060 con 0x7fdbe410f930 2024-05-27T07:57:25.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.937+0000 7fdbcbfff700 1 -- 172.21.15.114:0/662084414 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fdbd40054b0 con 0x7fdbe410f930 2024-05-27T07:57:25.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.938+0000 7fdbcbfff700 1 --2- 172.21.15.114:0/662084414 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdbcc0b6e10 0x7fdbcc0b9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.938+0000 7fdbe2ffd700 1 --2- 172.21.15.114:0/662084414 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdbcc0b6e10 0x7fdbcc0b9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.938+0000 7fdbcbfff700 1 -- 172.21.15.114:0/662084414 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 9566+0+0 (secure 0 0 0) 0x7fdbd40e2200 con 0x7fdbe410f930 2024-05-27T07:57:25.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.939+0000 7fdbe2ffd700 1 --2- 172.21.15.114:0/662084414 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdbcc0b6e10 0x7fdbcc0b9320 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00aee0 tx=0x7fdbdc00c040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:25.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.940+0000 7fdbf8161b80 1 Processor -- start 2024-05-27T07:57:25.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.941+0000 7fdbf8161b80 1 -- start start 2024-05-27T07:57:25.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.941+0000 7fdbf8161b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffa03e40 0x558fffb88b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.942+0000 7fdbf8161b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb89070 0x558fffb82be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.942+0000 7fdbf8161b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb83120 0x558fffb83530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.942+0000 7fdbe27fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb89070 0x558fffb82be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.942+0000 7fdbe27fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb89070 0x558fffb82be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51350/0 (socket says 172.21.15.114:51350) 2024-05-27T07:57:25.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.942+0000 7fdbe2ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffa03e40 0x558fffb88b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.942+0000 7fdbe2ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffa03e40 0x558fffb88b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:46956/0 (socket says 172.21.15.114:46956) 2024-05-27T07:57:25.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.942+0000 7fdbe27fc700 1 -- 172.21.15.114:0/3195155709 learned_addr learned my addr 172.21.15.114:0/3195155709 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:25.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.943+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3195155709 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558fffab0d70 con 0x558fffb83120 2024-05-27T07:57:25.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.943+0000 7fdbe37fe700 1 --2- 172.21.15.114:0/3195155709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb83120 0x558fffb83530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.943+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3195155709 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558fffaa0910 con 0x558fffb89070 2024-05-27T07:57:25.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.943+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3195155709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558fffaa4950 con 0x558fffa03e40 2024-05-27T07:57:25.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.944+0000 7fdbe27fc700 1 -- 172.21.15.114:0/3195155709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffa03e40 msgr2=0x558fffb88b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:25.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.944+0000 7fdbe27fc700 1 --2- 172.21.15.114:0/3195155709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffa03e40 0x558fffb88b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.944+0000 7fdbe27fc700 1 -- 172.21.15.114:0/3195155709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb83120 msgr2=0x558fffb83530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:25.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.944+0000 7fdbe27fc700 1 --2- 172.21.15.114:0/3195155709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb83120 0x558fffb83530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.944+0000 7fdbe27fc700 1 -- 172.21.15.114:0/3195155709 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fffa49fb0 con 0x558fffb89070 2024-05-27T07:57:25.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.944+0000 7fdbe27fc700 1 --2- 172.21.15.114:0/3195155709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb89070 0x558fffb82be0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4005e10 tx=0x7fdbd40a90d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47c8217da8fe3701 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:25.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.944+0000 7fdbc97fa700 1 -- 172.21.15.114:0/3195155709 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40b1f30 con 0x558fffb89070 2024-05-27T07:57:25.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.944+0000 7fdbc97fa700 1 -- 172.21.15.114:0/3195155709 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbd40ad170 con 0x558fffb89070 2024-05-27T07:57:25.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.945+0000 7fdbc97fa700 1 -- 172.21.15.114:0/3195155709 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40b1d00 con 0x558fffb89070 2024-05-27T07:57:25.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.946+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3195155709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb89070 msgr2=0x558fffb82be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:25.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.946+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/3195155709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb89070 0x558fffb82be0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4005e10 tx=0x7fdbd40a90d0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.946+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3195155709 shutdown_connections 2024-05-27T07:57:25.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.946+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/3195155709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb83120 0x558fffb83530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.947+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/3195155709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffa03e40 0x558fffb88b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.947+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/3195155709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb89070 0x558fffb82be0 secure :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4005e10 tx=0x7fdbd40a90d0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.947+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3195155709 >> 172.21.15.114:0/3195155709 conn(0x558fffa06770 msgr2=0x558fffb7dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:25.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.947+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3195155709 shutdown_connections 2024-05-27T07:57:25.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.947+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3195155709 wait complete. 2024-05-27T07:57:25.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.947+0000 7fdbf8161b80 1 -- start start 2024-05-27T07:57:25.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.948+0000 7fdbf8161b80 1 Processor -- start 2024-05-27T07:57:25.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.952+0000 7fdbf8161b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffa03e40 0x558fffb8c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.952+0000 7fdbf8161b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb83120 0x558fffb8cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.952+0000 7fdbf8161b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb8d440 0x558fffba1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.952+0000 7fdbe27fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb83120 0x558fffb8cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.952+0000 7fdbe2ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffa03e40 0x558fffb8c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.952+0000 7fdbe2ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffa03e40 0x558fffb8c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:46964/0 (socket says 172.21.15.114:46964) 2024-05-27T07:57:25.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.952+0000 7fdbe27fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb83120 0x558fffb8cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59468/0 (socket says 172.21.15.114:59468) 2024-05-27T07:57:25.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.952+0000 7fdbe2ffd700 1 -- 172.21.15.114:0/3116072185 learned_addr learned my addr 172.21.15.114:0/3116072185 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:25.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.952+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3116072185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558fff9a6fa0 con 0x558fffb83120 2024-05-27T07:57:25.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.952+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3116072185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558fffa08da0 con 0x558fffb8d440 2024-05-27T07:57:25.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.952+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3116072185 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558fffb8fb00 con 0x558fffa03e40 2024-05-27T07:57:25.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.953+0000 7fdbf8161b80 10 client.0 _ll_register_callbacks cb 0x558fffa375b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:25.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.953+0000 7fdbe37fe700 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb8d440 0x558fffba1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.953+0000 7fdbe2ffd700 1 -- 172.21.15.114:0/3116072185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb8d440 msgr2=0x558fffba1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:25.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.953+0000 7fdbe2ffd700 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb8d440 0x558fffba1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.953+0000 7fdbe2ffd700 1 -- 172.21.15.114:0/3116072185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb83120 msgr2=0x558fffb8cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.953+0000 7fdbe2ffd700 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb83120 0x558fffb8cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.953+0000 7fdbe2ffd700 1 -- 172.21.15.114:0/3116072185 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fff9f08d0 con 0x558fffa03e40 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.954+0000 7fdbe37fe700 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb8d440 0x558fffba1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.954+0000 7fdbe27fc700 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb83120 0x558fffb8cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.954+0000 7fdbe2ffd700 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffa03e40 0x558fffb8c9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc017c50 tx=0x7fdbdc004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1efaec335b6702d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.954+0000 7fdbc6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.954+0000 7fdbc6ff5700 1 -- 172.21.15.114:0/3116072185 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc027070 con 0x558fffa03e40 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.954+0000 7fdbc6ff5700 1 -- 172.21.15.114:0/3116072185 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(152..152 src has 1..152) ==== 9566+0+0 (secure 0 0 0) 0x7fdbdc015dc0 con 0x558fffa03e40 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.954+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3116072185 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbd40ad170 con 0x558fffa03e40 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.955+0000 7fdbc6ff5700 1 -- 172.21.15.114:0/3116072185 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7fdbdc032050 con 0x558fffa03e40 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.955+0000 7fdbf8161b80 10 client.28918 fetch_fsmap learned FSMap version 98 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.955+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3116072185 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=4+}) -- 0x558fffa5a760 con 0x558fffa03e40 2024-05-27T07:57:25.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.955+0000 7fdbc6ff5700 1 -- 172.21.15.114:0/3116072185 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbd40ad170 con 0x558fffa03e40 2024-05-27T07:57:25.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.956+0000 7fdbc6ff5700 1 -- 172.21.15.114:0/3116072185 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 98) ==== 4095+0+0 (secure 0 0 0) 0x7fdbdc0046b0 con 0x558fffa03e40 2024-05-27T07:57:25.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.956+0000 7fdbf8161b80 10 client.28918 fetch_fsmap finished waiting for FSMap version 98 2024-05-27T07:57:25.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.956+0000 7fdbf8161b80 10 client.28918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:25.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.956+0000 7fdbf8161b80 20 client.28918 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:25.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.956+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558fffba5c90 0x558fffba8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.956+0000 7fdbf8161b80 4 client.28918 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:57:25.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.956+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3116072185 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fffab2cc0 con 0x558fffba5c90 2024-05-27T07:57:25.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.956+0000 7fdbe27fc700 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558fffba5c90 0x558fffba8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.956+0000 7fdbc6ff5700 10 client.28918 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:25.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.956+0000 7fdbe27fc700 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558fffba5c90 0x558fffba8160 crc :-1 s=READY pgs=474 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb955dfc48488028 server_cookie=32b225f8d0ee1c47 in_seq=0 out_seq=0 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.958+0000 7fdbc6ff5700 1 -- 172.21.15.114:0/3116072185 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558fffab2cc0 con 0x558fffba5c90 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.958+0000 7fdbc6ff5700 10 client.28918 handle_command_reply: tid=0 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.958+0000 7fdbf8161b80 1 client.28918 shutdown 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.959+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3116072185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffa03e40 msgr2=0x558fffb8c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.959+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffa03e40 0x558fffb8c9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc017c50 tx=0x7fdbdc004ad0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.959+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3116072185 shutdown_connections 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.959+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558fffba5c90 0x558fffba8160 crc :-1 s=READY pgs=474 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.959+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb8d440 0x558fffba1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.959+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffa03e40 0x558fffb8c9c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.959+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/3116072185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb83120 0x558fffb8cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.959+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3116072185 >> 172.21.15.114:0/3116072185 conn(0x558fffa06770 msgr2=0x558fffb81bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.959+0000 7fdbc6ff5700 0 client.28918 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.959+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3116072185 shutdown_connections 2024-05-27T07:57:25.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.959+0000 7fdbf8161b80 1 -- 172.21.15.114:0/3116072185 wait complete. 2024-05-27T07:57:25.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.959+0000 7fdbf8161b80 20 client.28918 trim_cache size 0 max 16384 2024-05-27T07:57:25.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.975+0000 7fdbf8161b80 1 Processor -- start 2024-05-27T07:57:25.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.976+0000 7fdbf8161b80 1 -- start start 2024-05-27T07:57:25.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.976+0000 7fdbf8161b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffa03e40 0x558fffb8b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.976+0000 7fdbf8161b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb83120 0x558fffbad920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.976+0000 7fdbf8161b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb8d440 0x558fffbade60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.976+0000 7fdbf8161b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe41b6a10 con 0x558fffb8d440 2024-05-27T07:57:25.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.976+0000 7fdbe27fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb83120 0x558fffbad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.976+0000 7fdbf8161b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe41b70e0 con 0x558fffa03e40 2024-05-27T07:57:25.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.976+0000 7fdbe27fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb83120 0x558fffbad920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:46978/0 (socket says 172.21.15.114:46978) 2024-05-27T07:57:25.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.976+0000 7fdbe27fc700 1 -- 172.21.15.114:0/821946077 learned_addr learned my addr 172.21.15.114:0/821946077 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:25.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.976+0000 7fdbe2ffd700 1 --2- 172.21.15.114:0/821946077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffa03e40 0x558fffb8b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.976+0000 7fdbf8161b80 1 -- 172.21.15.114:0/821946077 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe41b5990 con 0x558fffb83120 2024-05-27T07:57:25.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.976+0000 7fdbe37fe700 1 --2- 172.21.15.114:0/821946077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb8d440 0x558fffbade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.977+0000 7fdbe27fc700 1 -- 172.21.15.114:0/821946077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffa03e40 msgr2=0x558fffb8b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:25.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.977+0000 7fdbe27fc700 1 --2- 172.21.15.114:0/821946077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffa03e40 0x558fffb8b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.977+0000 7fdbe27fc700 1 -- 172.21.15.114:0/821946077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb8d440 msgr2=0x558fffbade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:25.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.977+0000 7fdbe27fc700 1 --2- 172.21.15.114:0/821946077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb8d440 0x558fffbade60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.977+0000 7fdbe27fc700 1 -- 172.21.15.114:0/821946077 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fffb7eab0 con 0x558fffb83120 2024-05-27T07:57:25.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.977+0000 7fdbe27fc700 1 --2- 172.21.15.114:0/821946077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb83120 0x558fffbad920 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40a8dd0 tx=0x7fdbd4005c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4342dd9da8bcf39e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:25.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.977+0000 7fdbc8ff9700 1 -- 172.21.15.114:0/821946077 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd406e070 con 0x558fffb83120 2024-05-27T07:57:25.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.977+0000 7fdbc8ff9700 1 -- 172.21.15.114:0/821946077 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbd40b1f30 con 0x558fffb83120 2024-05-27T07:57:25.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.977+0000 7fdbc8ff9700 1 -- 172.21.15.114:0/821946077 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40b1d00 con 0x558fffb83120 2024-05-27T07:57:25.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.978+0000 7fdbf8161b80 1 -- 172.21.15.114:0/821946077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb83120 msgr2=0x558fffbad920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:25.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.978+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/821946077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb83120 0x558fffbad920 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40a8dd0 tx=0x7fdbd4005c40 comp rx=0 tx=0).stop 2024-05-27T07:57:25.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.978+0000 7fdbf8161b80 1 -- 172.21.15.114:0/821946077 shutdown_connections 2024-05-27T07:57:25.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.978+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/821946077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffb8d440 0x558fffbade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.978+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/821946077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffa03e40 0x558fffb8b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.978+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/821946077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb83120 0x558fffbad920 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.978+0000 7fdbf8161b80 1 -- 172.21.15.114:0/821946077 >> 172.21.15.114:0/821946077 conn(0x558fffa06770 msgr2=0x558fffb8b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.978+0000 7fdbf8161b80 1 -- 172.21.15.114:0/821946077 shutdown_connections 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbf8161b80 1 -- 172.21.15.114:0/821946077 wait complete. 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbf8161b80 1 -- start start 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbf8161b80 1 Processor -- start 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbf8161b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffa03e40 0x558fffb9d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbf8161b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb83120 0x558fffb9db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbf8161b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb8d440 0x558fffba05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbf8161b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe41b52c0 con 0x558fffa03e40 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbf8161b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe41b4c50 con 0x558fffb83120 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbf8161b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558fffa08da0 con 0x558fffb8d440 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbf8161b80 10 client.0 _ll_register_callbacks cb 0x558fffa375b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbe2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffa03e40 0x558fffb9d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbe37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb8d440 0x558fffba05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbe27fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb83120 0x558fffb9db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbe2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffa03e40 0x558fffb9d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59480/0 (socket says 172.21.15.114:59480) 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbe27fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb83120 0x558fffb9db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51370/0 (socket says 172.21.15.114:51370) 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbe37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb8d440 0x558fffba05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:46984/0 (socket says 172.21.15.114:46984) 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.979+0000 7fdbe2ffd700 1 -- 172.21.15.114:0/1155519486 learned_addr learned my addr 172.21.15.114:0/1155519486 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.980+0000 7fdbe37fe700 1 -- 172.21.15.114:0/1155519486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb83120 msgr2=0x558fffb9db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.980+0000 7fdbe37fe700 1 --2- 172.21.15.114:0/1155519486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb83120 0x558fffb9db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.980+0000 7fdbe37fe700 1 -- 172.21.15.114:0/1155519486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffa03e40 msgr2=0x558fffb9d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.980+0000 7fdbe37fe700 1 --2- 172.21.15.114:0/1155519486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffa03e40 0x558fffb9d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.980+0000 7fdbe37fe700 1 -- 172.21.15.114:0/1155519486 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fffba0b20 con 0x558fffb8d440 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.980+0000 7fdbe2ffd700 1 --2- 172.21.15.114:0/1155519486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffa03e40 0x558fffb9d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.980+0000 7fdbe37fe700 1 --2- 172.21.15.114:0/1155519486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb8d440 0x558fffba05e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdbd80094d0 tx=0x7fdbd800bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b541256fbaffe97d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.980+0000 7fdbc6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.980+0000 7fdbc6ff5700 1 -- 172.21.15.114:0/1155519486 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8003ad0 con 0x558fffb8d440 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.980+0000 7fdbf8161b80 1 -- 172.21.15.114:0/1155519486 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbd40b1f30 con 0x558fffb8d440 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.980+0000 7fdbc6ff5700 1 -- 172.21.15.114:0/1155519486 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(152..152 src has 1..152) ==== 9566+0+0 (secure 0 0 0) 0x7fdbd8003c70 con 0x558fffb8d440 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.981+0000 7fdbc6ff5700 1 -- 172.21.15.114:0/1155519486 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7fdbd8005070 con 0x558fffb8d440 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.981+0000 7fdbf8161b80 10 client.28924 fetch_fsmap learned FSMap version 98 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.981+0000 7fdbf8161b80 1 -- 172.21.15.114:0/1155519486 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=4+}) -- 0x558fffc0fee0 con 0x558fffb8d440 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.981+0000 7fdbc6ff5700 1 -- 172.21.15.114:0/1155519486 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbd40b1f30 con 0x558fffb8d440 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.981+0000 7fdbc6ff5700 1 -- 172.21.15.114:0/1155519486 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 98) ==== 4095+0+0 (secure 0 0 0) 0x7fdbd800fce0 con 0x558fffb8d440 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.981+0000 7fdbf8161b80 10 client.28924 fetch_fsmap finished waiting for FSMap version 98 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.981+0000 7fdbf8161b80 10 client.28924 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.981+0000 7fdbf8161b80 20 client.28924 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.981+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/1155519486 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558fffc105f0 0x558fffc12a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.981+0000 7fdbf8161b80 4 client.28924 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:57:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.981+0000 7fdbf8161b80 1 -- 172.21.15.114:0/1155519486 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558fffaac4a0 con 0x558fffc105f0 2024-05-27T07:57:25.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.981+0000 7fdbe2ffd700 1 --2- 172.21.15.114:0/1155519486 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558fffc105f0 0x558fffc12a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:25.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.982+0000 7fdbc6ff5700 10 client.28924 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:25.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.982+0000 7fdbe2ffd700 1 --2- 172.21.15.114:0/1155519486 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558fffc105f0 0x558fffc12a10 crc :-1 s=READY pgs=475 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d94a5583d958c15 server_cookie=1f795400af0f2d3a in_seq=0 out_seq=0 2024-05-27T07:57:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.983+0000 7fdbc6ff5700 1 -- 172.21.15.114:0/1155519486 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558fffaac4a0 con 0x558fffc105f0 2024-05-27T07:57:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.983+0000 7fdbc6ff5700 10 client.28924 handle_command_reply: tid=0 2024-05-27T07:57:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.983+0000 7fdbf8161b80 1 client.28924 shutdown 2024-05-27T07:57:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.983+0000 7fdbf8161b80 1 -- 172.21.15.114:0/1155519486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb8d440 msgr2=0x558fffba05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.983+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/1155519486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb8d440 0x558fffba05e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdbd80094d0 tx=0x7fdbd800bf50 comp rx=0 tx=0).stop 2024-05-27T07:57:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbf8161b80 1 -- 172.21.15.114:0/1155519486 shutdown_connections 2024-05-27T07:57:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/1155519486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fffb8d440 0x558fffba05e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/1155519486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fffa03e40 0x558fffb9d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/1155519486 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558fffc105f0 0x558fffc12a10 crc :-1 s=READY pgs=475 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbf8161b80 1 --2- 172.21.15.114:0/1155519486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558fffb83120 0x558fffb9db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbf8161b80 1 -- 172.21.15.114:0/1155519486 >> 172.21.15.114:0/1155519486 conn(0x558fffa06770 msgr2=0x558fffb86490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbc6ff5700 0 client.28924 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbf8161b80 1 -- 172.21.15.114:0/1155519486 shutdown_connections 2024-05-27T07:57:25.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbf8161b80 1 -- 172.21.15.114:0/1155519486 wait complete. 2024-05-27T07:57:25.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbf8161b80 20 client.28924 trim_cache size 0 max 16384 2024-05-27T07:57:25.989 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:57:25.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbe974c700 1 -- 172.21.15.114:0/662084414 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdbcc0b6e10 msgr2=0x7fdbcc0b9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:25.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbe974c700 1 --2- 172.21.15.114:0/662084414 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdbcc0b6e10 0x7fdbcc0b9320 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00aee0 tx=0x7fdbdc00c040 comp rx=0 tx=0).stop 2024-05-27T07:57:25.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbe974c700 1 -- 172.21.15.114:0/662084414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbe410f930 msgr2=0x7fdbe41a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:25.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.984+0000 7fdbe974c700 1 --2- 172.21.15.114:0/662084414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbe410f930 0x7fdbe41a1950 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400d720 tx=0x7fdbd400da30 comp rx=0 tx=0).stop 2024-05-27T07:57:25.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.985+0000 7fdbe974c700 1 -- 172.21.15.114:0/662084414 shutdown_connections 2024-05-27T07:57:25.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.985+0000 7fdbe974c700 1 --2- 172.21.15.114:0/662084414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbe4110280 0x7fdbe41a5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.985+0000 7fdbe974c700 1 --2- 172.21.15.114:0/662084414 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdbcc0b6e10 0x7fdbcc0b9320 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.985+0000 7fdbe974c700 1 --2- 172.21.15.114:0/662084414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbe410b000 0x7fdbe41a13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.985+0000 7fdbe974c700 1 --2- 172.21.15.114:0/662084414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbe410f930 0x7fdbe41a1950 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.985+0000 7fdbe974c700 1 -- 172.21.15.114:0/662084414 >> 172.21.15.114:0/662084414 conn(0x7fdbe406f4c0 msgr2=0x7fdbe410e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.985+0000 7fdbe974c700 1 -- 172.21.15.114:0/662084414 shutdown_connections 2024-05-27T07:57:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:25.985+0000 7fdbe974c700 1 -- 172.21.15.114:0/662084414 wait complete. 2024-05-27T07:57:26.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 52431 12.10 MB/sec execute 98 sec latency 371.898 ms 2024-05-27T07:57:27.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 52744 12.27 MB/sec execute 99 sec latency 315.745 ms 2024-05-27T07:57:27.735 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:57:27.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:57:27.992 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:57:28.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:27 smithi114 ceph-mon[32038]: pgmap v607: 129 pgs: 129 active+clean; 343 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.0 MiB/s wr, 634 op/s 2024-05-27T07:57:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:27 smithi163 ceph-mon[36451]: pgmap v607: 129 pgs: 129 active+clean; 343 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.0 MiB/s wr, 634 op/s 2024-05-27T07:57:28.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:27 smithi186 ceph-mon[36647]: pgmap v607: 129 pgs: 129 active+clean; 343 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.0 MiB/s wr, 634 op/s 2024-05-27T07:57:28.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 53065 12.32 MB/sec execute 100 sec latency 163.833 ms 2024-05-27T07:57:29.307 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:29 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:57:29.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:29 smithi114 systemd[1]: Stopped Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:57:29.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:29 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.586s CPU time. 2024-05-27T07:57:29.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:29 smithi114 systemd[1]: Starting Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:57:29.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 53269 12.28 MB/sec execute 101 sec latency 217.973 ms 2024-05-27T07:57:30.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-mon[32038]: pgmap v608: 129 pgs: 129 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 472 op/s 2024-05-27T07:57:30.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-mon[32038]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:57:30.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 2 up:standby 2024-05-27T07:57:30.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-mon[32038]: osdmap e153: 12 total, 12 up, 12 in 2024-05-27T07:57:30.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2bfed31700 time 2024-05-27T07:57:29.608093+0000 2024-05-27T07:57:30.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:57:30.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:30.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2c11576ba6] 2024-05-27T07:57:30.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2c11576dd1] 2024-05-27T07:57:30.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5573392ff23d] 2024-05-27T07:57:30.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5573392ea971] 2024-05-27T07:57:30.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 5: (SnapRealm::check_cache() const+0x13e) [0x5573392e8fa0] 2024-05-27T07:57:30.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5573392e9651] 2024-05-27T07:57:30.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55733927e347] 2024-05-27T07:57:30.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557338f7a87e] 2024-05-27T07:57:30.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x557338f7ff2f] 2024-05-27T07:57:30.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557338f8a969] 2024-05-27T07:57:30.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 11: (MDLog::_replay_thread()+0x1f38) [0x55733932348c] 2024-05-27T07:57:30.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55733932c9cb] 2024-05-27T07:57:30.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 13: (Thread::entry_wrapper()+0x43) [0x7f2c1154572d] 2024-05-27T07:57:30.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2c11545749] 2024-05-27T07:57:30.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2c100e91ca] 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 16: clone() 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:29.612+0000 7f2bfed31700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2bfed31700 time 2024-05-27T07:57:29.608093+0000 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2c11576ba6] 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2c11576dd1] 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5573392ff23d] 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5573392ea971] 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 5: (SnapRealm::check_cache() const+0x13e) [0x5573392e8fa0] 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5573392e9651] 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55733927e347] 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557338f7a87e] 2024-05-27T07:57:30.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x557338f7ff2f] 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557338f8a969] 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 11: (MDLog::_replay_thread()+0x1f38) [0x55733932348c] 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55733932c9cb] 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 13: (Thread::entry_wrapper()+0x43) [0x7f2c1154572d] 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2c11545749] 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2c100e91ca] 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 16: clone() 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: *** Caught signal (Aborted) ** 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: in thread 7f2bfed31700 thread_name:md_log_replay 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2c100f3d20] 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2: gsignal() 2024-05-27T07:57:30.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 3: abort() 2024-05-27T07:57:30.062 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2c11576ced] 2024-05-27T07:57:30.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2c11576dd1] 2024-05-27T07:57:30.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5573392ff23d] 2024-05-27T07:57:30.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5573392ea971] 2024-05-27T07:57:30.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 8: (SnapRealm::check_cache() const+0x13e) [0x5573392e8fa0] 2024-05-27T07:57:30.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5573392e9651] 2024-05-27T07:57:30.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55733927e347] 2024-05-27T07:57:30.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557338f7a87e] 2024-05-27T07:57:30.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x557338f7ff2f] 2024-05-27T07:57:30.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557338f8a969] 2024-05-27T07:57:30.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 14: (MDLog::_replay_thread()+0x1f38) [0x55733932348c] 2024-05-27T07:57:30.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55733932c9cb] 2024-05-27T07:57:30.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 16: (Thread::entry_wrapper()+0x43) [0x7f2c1154572d] 2024-05-27T07:57:30.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2c11545749] 2024-05-27T07:57:30.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2c100e91ca] 2024-05-27T07:57:30.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 19: clone() 2024-05-27T07:57:30.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:29.615+0000 7f2bfed31700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:57:30.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: in thread 7f2bfed31700 thread_name:md_log_replay 2024-05-27T07:57:30.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:30.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:30.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2c100f3d20] 2024-05-27T07:57:30.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2: gsignal() 2024-05-27T07:57:30.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 3: abort() 2024-05-27T07:57:30.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2c11576ced] 2024-05-27T07:57:30.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2c11576dd1] 2024-05-27T07:57:30.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5573392ff23d] 2024-05-27T07:57:30.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5573392ea971] 2024-05-27T07:57:30.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 8: (SnapRealm::check_cache() const+0x13e) [0x5573392e8fa0] 2024-05-27T07:57:30.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5573392e9651] 2024-05-27T07:57:30.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55733927e347] 2024-05-27T07:57:30.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557338f7a87e] 2024-05-27T07:57:30.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x557338f7ff2f] 2024-05-27T07:57:30.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557338f8a969] 2024-05-27T07:57:30.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 14: (MDLog::_replay_thread()+0x1f38) [0x55733932348c] 2024-05-27T07:57:30.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55733932c9cb] 2024-05-27T07:57:30.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 16: (Thread::entry_wrapper()+0x43) [0x7f2c1154572d] 2024-05-27T07:57:30.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2c11545749] 2024-05-27T07:57:30.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2c100e91ca] 2024-05-27T07:57:30.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 19: clone() 2024-05-27T07:57:30.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:57:30.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:30.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: -1> 2024-05-27T07:57:29.612+0000 7f2bfed31700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2bfed31700 time 2024-05-27T07:57:29.608093+0000 2024-05-27T07:57:30.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:57:30.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:30.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:30.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2c11576ba6] 2024-05-27T07:57:30.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2c11576dd1] 2024-05-27T07:57:30.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5573392ff23d] 2024-05-27T07:57:30.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5573392ea971] 2024-05-27T07:57:30.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 5: (SnapRealm::check_cache() const+0x13e) [0x5573392e8fa0] 2024-05-27T07:57:30.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5573392e9651] 2024-05-27T07:57:30.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55733927e347] 2024-05-27T07:57:30.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557338f7a87e] 2024-05-27T07:57:30.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x557338f7ff2f] 2024-05-27T07:57:30.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557338f8a969] 2024-05-27T07:57:30.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 11: (MDLog::_replay_thread()+0x1f38) [0x55733932348c] 2024-05-27T07:57:30.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55733932c9cb] 2024-05-27T07:57:30.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 13: (Thread::entry_wrapper()+0x43) [0x7f2c1154572d] 2024-05-27T07:57:30.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2c11545749] 2024-05-27T07:57:30.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2c100e91ca] 2024-05-27T07:57:30.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 16: clone() 2024-05-27T07:57:30.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:30.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 0> 2024-05-27T07:57:29.615+0000 7f2bfed31700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:57:30.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: in thread 7f2bfed31700 thread_name:md_log_replay 2024-05-27T07:57:30.076 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:30.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:30.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2c100f3d20] 2024-05-27T07:57:30.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2: gsignal() 2024-05-27T07:57:30.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 3: abort() 2024-05-27T07:57:30.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2c11576ced] 2024-05-27T07:57:30.077 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2c11576dd1] 2024-05-27T07:57:30.078 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5573392ff23d] 2024-05-27T07:57:30.078 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5573392ea971] 2024-05-27T07:57:30.078 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 8: (SnapRealm::check_cache() const+0x13e) [0x5573392e8fa0] 2024-05-27T07:57:30.078 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5573392e9651] 2024-05-27T07:57:30.078 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55733927e347] 2024-05-27T07:57:30.079 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557338f7a87e] 2024-05-27T07:57:30.079 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x557338f7ff2f] 2024-05-27T07:57:30.079 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557338f8a969] 2024-05-27T07:57:30.079 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 14: (MDLog::_replay_thread()+0x1f38) [0x55733932348c] 2024-05-27T07:57:30.079 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55733932c9cb] 2024-05-27T07:57:30.079 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 16: (Thread::entry_wrapper()+0x43) [0x7f2c1154572d] 2024-05-27T07:57:30.080 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2c11545749] 2024-05-27T07:57:30.080 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2c100e91ca] 2024-05-27T07:57:30.080 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 19: clone() 2024-05-27T07:57:30.080 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:57:30.080 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:30.080 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: -1> 2024-05-27T07:57:29.612+0000 7f2bfed31700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2bfed31700 time 2024-05-27T07:57:29.608093+0000 2024-05-27T07:57:30.080 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:57:30.081 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:30.081 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:30.081 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2c11576ba6] 2024-05-27T07:57:30.081 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2c11576dd1] 2024-05-27T07:57:30.081 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5573392ff23d] 2024-05-27T07:57:30.081 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5573392ea971] 2024-05-27T07:57:30.081 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 5: (SnapRealm::check_cache() const+0x13e) [0x5573392e8fa0] 2024-05-27T07:57:30.082 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5573392e9651] 2024-05-27T07:57:30.082 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55733927e347] 2024-05-27T07:57:30.082 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557338f7a87e] 2024-05-27T07:57:30.082 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x557338f7ff2f] 2024-05-27T07:57:30.082 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557338f8a969] 2024-05-27T07:57:30.082 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 11: (MDLog::_replay_thread()+0x1f38) [0x55733932348c] 2024-05-27T07:57:30.082 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55733932c9cb] 2024-05-27T07:57:30.083 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 13: (Thread::entry_wrapper()+0x43) [0x7f2c1154572d] 2024-05-27T07:57:30.083 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2c11545749] 2024-05-27T07:57:30.083 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2c100e91ca] 2024-05-27T07:57:30.083 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 16: clone() 2024-05-27T07:57:30.083 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:30.083 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 0> 2024-05-27T07:57:29.615+0000 7f2bfed31700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:57:30.083 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: in thread 7f2bfed31700 thread_name:md_log_replay 2024-05-27T07:57:30.083 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:30.083 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:57:30.083 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2c100f3d20] 2024-05-27T07:57:30.083 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2: gsignal() 2024-05-27T07:57:30.083 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 3: abort() 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2c11576ced] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2c11576dd1] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5573392ff23d] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5573392ea971] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 8: (SnapRealm::check_cache() const+0x13e) [0x5573392e8fa0] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5573392e9651] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55733927e347] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557338f7a87e] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x557338f7ff2f] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557338f8a969] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 14: (MDLog::_replay_thread()+0x1f38) [0x55733932348c] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55733932c9cb] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 16: (Thread::entry_wrapper()+0x43) [0x7f2c1154572d] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2c11545749] 2024-05-27T07:57:30.084 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2c100e91ca] 2024-05-27T07:57:30.085 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 19: clone() 2024-05-27T07:57:30.085 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:57:30.085 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[128889]: 2024-05-27T07:57:30.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:29 smithi163 ceph-mon[36451]: pgmap v608: 129 pgs: 129 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 472 op/s 2024-05-27T07:57:30.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:29 smithi163 ceph-mon[36451]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:57:30.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:29 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 2 up:standby 2024-05-27T07:57:30.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:29 smithi163 ceph-mon[36451]: osdmap e153: 12 total, 12 up, 12 in 2024-05-27T07:57:30.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:29 smithi186 ceph-mon[36647]: pgmap v608: 129 pgs: 129 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 472 op/s 2024-05-27T07:57:30.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:29 smithi186 ceph-mon[36647]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:57:30.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:29 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 2 up:standby 2024-05-27T07:57:30.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:29 smithi186 ceph-mon[36647]: osdmap e153: 12 total, 12 up, 12 in 2024-05-27T07:57:30.557 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:30 smithi114 podman[129620]: 2024-05-27 07:57:30.305973035 +0000 UTC m=+0.648127514 container died 459081d6e2d5570403951de332ca4c2ab38395490179130597c927795134064a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD) 2024-05-27T07:57:30.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 53494 12.25 MB/sec execute 102 sec latency 179.230 ms 2024-05-27T07:57:31.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:30 smithi163 ceph-mon[36451]: pgmap v610: 129 pgs: 129 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 392 op/s 2024-05-27T07:57:31.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:30 smithi186 ceph-mon[36647]: pgmap v610: 129 pgs: 129 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 392 op/s 2024-05-27T07:57:31.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:30 smithi114 ceph-mon[32038]: pgmap v610: 129 pgs: 129 active+clean; 346 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 392 op/s 2024-05-27T07:57:31.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 53623 12.19 MB/sec execute 103 sec latency 250.870 ms 2024-05-27T07:57:31.763 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:31 smithi114 podman[129620]: 2024-05-27 07:57:31.4209203 +0000 UTC m=+1.763074785 container remove 459081d6e2d5570403951de332ca4c2ab38395490179130597c927795134064a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:57:31.763 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:31 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:57:32.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.114+0000 7f77dddc6700 1 -- 172.21.15.114:0/3953612593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d811d920 msgr2=0x7f77d811fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:32.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.114+0000 7f77dddc6700 1 --2- 172.21.15.114:0/3953612593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d811d920 0x7f77d811fd40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009f90 tx=0x7f77d000b330 comp rx=0 tx=0).stop 2024-05-27T07:57:32.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.115+0000 7f77dddc6700 1 -- 172.21.15.114:0/3953612593 shutdown_connections 2024-05-27T07:57:32.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.115+0000 7f77dddc6700 1 --2- 172.21.15.114:0/3953612593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d811d920 0x7f77d811fd40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.115+0000 7f77dddc6700 1 --2- 172.21.15.114:0/3953612593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77d810a920 0x7f77d810ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.115+0000 7f77dddc6700 1 --2- 172.21.15.114:0/3953612593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d810b270 0x7f77d811d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.115+0000 7f77dddc6700 1 -- 172.21.15.114:0/3953612593 >> 172.21.15.114:0/3953612593 conn(0x7f77d806f4c0 msgr2=0x7f77d806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:32.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.118+0000 7f77dddc6700 1 -- 172.21.15.114:0/3953612593 shutdown_connections 2024-05-27T07:57:32.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.122+0000 7f77dddc6700 1 -- 172.21.15.114:0/3953612593 wait complete. 2024-05-27T07:57:32.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.123+0000 7f77dddc6700 1 Processor -- start 2024-05-27T07:57:32.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.123+0000 7f77dddc6700 1 -- start start 2024-05-27T07:57:32.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.123+0000 7f77dddc6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d811d920 0x7f77d811d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.123+0000 7f77dddc6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77d8119490 0x7f77d81198a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.123+0000 7f77dddc6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d8119e30 0x7f77d81ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.123+0000 7f77dddc6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f77d81154a0 con 0x7f77d8119490 2024-05-27T07:57:32.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.123+0000 7f77dddc6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f77d8115320 con 0x7f77d8119e30 2024-05-27T07:57:32.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.123+0000 7f77dddc6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f77d8115620 con 0x7f77d811d920 2024-05-27T07:57:32.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.124+0000 7f77d77fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d811d920 0x7f77d811d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.124+0000 7f77d77fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d811d920 0x7f77d811d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47004/0 (socket says 172.21.15.114:47004) 2024-05-27T07:57:32.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.124+0000 7f77d77fe700 1 -- 172.21.15.114:0/3105899522 learned_addr learned my addr 172.21.15.114:0/3105899522 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:32.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.124+0000 7f77d7fff700 1 --2- 172.21.15.114:0/3105899522 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d8119e30 0x7f77d81ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.126+0000 7f77d7fff700 1 -- 172.21.15.114:0/3105899522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d811d920 msgr2=0x7f77d811d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:32.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.126+0000 7f77d7fff700 1 --2- 172.21.15.114:0/3105899522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d811d920 0x7f77d811d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.126+0000 7f77d7fff700 1 -- 172.21.15.114:0/3105899522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77d8119490 msgr2=0x7f77d81198a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:32.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.126+0000 7f77d7fff700 1 --2- 172.21.15.114:0/3105899522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77d8119490 0x7f77d81198a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.126+0000 7f77d7fff700 1 -- 172.21.15.114:0/3105899522 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d81adfb0 con 0x7f77d8119e30 2024-05-27T07:57:32.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.126+0000 7f77d7fff700 1 --2- 172.21.15.114:0/3105899522 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d8119e30 0x7f77d81ad8b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009660 tx=0x7f77d0009440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:32.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.130+0000 7f77d4ff9700 1 -- 172.21.15.114:0/3105899522 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d0021070 con 0x7f77d8119e30 2024-05-27T07:57:32.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.130+0000 7f77d4ff9700 1 -- 172.21.15.114:0/3105899522 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77d0012d80 con 0x7f77d8119e30 2024-05-27T07:57:32.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.130+0000 7f77d4ff9700 1 -- 172.21.15.114:0/3105899522 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d001db30 con 0x7f77d8119e30 2024-05-27T07:57:32.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.130+0000 7f77dddc6700 1 -- 172.21.15.114:0/3105899522 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77d81ae240 con 0x7f77d8119e30 2024-05-27T07:57:32.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.130+0000 7f77dddc6700 1 -- 172.21.15.114:0/3105899522 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77d81ae730 con 0x7f77d8119e30 2024-05-27T07:57:32.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.135+0000 7f77ec7dbb80 1 Processor -- start 2024-05-27T07:57:32.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.136+0000 7f77ec7dbb80 1 -- start start 2024-05-27T07:57:32.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.138+0000 7f77d4ff9700 1 -- 172.21.15.114:0/3105899522 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f77d0024070 con 0x7f77d8119e30 2024-05-27T07:57:32.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.139+0000 7f77d4ff9700 1 --2- 172.21.15.114:0/3105899522 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f77c00b2bf0 0x7f77c00b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.140+0000 7f77d4ff9700 1 -- 172.21.15.114:0/3105899522 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 9538+0+0 (secure 0 0 0) 0x7f77d000f020 con 0x7f77d8119e30 2024-05-27T07:57:32.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.140+0000 7f77ec7dbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x555924873790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.140+0000 7f77ec7dbb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555924873cd0 0x55592486dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.140+0000 7f77ec7dbb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x55592486e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.140+0000 7f77ec7dbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55592479cd10 con 0x5559246efec0 2024-05-27T07:57:32.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.140+0000 7f77ec7dbb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55592478c8b0 con 0x555924873cd0 2024-05-27T07:57:32.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.140+0000 7f77ec7dbb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5559247908f0 con 0x55592486e000 2024-05-27T07:57:32.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.141+0000 7f77d7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x55592486e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.141+0000 7f77d7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x55592486e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:47012/0 (socket says 172.21.15.114:47012) 2024-05-27T07:57:32.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.141+0000 7f77d7fff700 1 -- 172.21.15.114:0/4095882442 learned_addr learned my addr 172.21.15.114:0/4095882442 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:32.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.141+0000 7f77d77fe700 1 --2- 172.21.15.114:0/4095882442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x555924873790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.143+0000 7f77d7fff700 1 -- 172.21.15.114:0/4095882442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555924873cd0 msgr2=0x55592486dac0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:57:32.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.143+0000 7f77d7fff700 1 --2- 172.21.15.114:0/4095882442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555924873cd0 0x55592486dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.143+0000 7f77d7fff700 1 -- 172.21.15.114:0/4095882442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 msgr2=0x555924873790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:32.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.143+0000 7f77d7fff700 1 --2- 172.21.15.114:0/4095882442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x555924873790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.143+0000 7f77d7fff700 1 -- 172.21.15.114:0/4095882442 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559246dc950 con 0x55592486e000 2024-05-27T07:57:32.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.143+0000 7f77d6ffd700 1 --2- 172.21.15.114:0/3105899522 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f77c00b2bf0 0x7f77c00b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.143+0000 7f77d7fff700 1 --2- 172.21.15.114:0/4095882442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x55592486e410 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f77d00aa550 tx=0x7f77d00a9fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8501915e400e052 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:32.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.147+0000 7f77bdffb700 1 -- 172.21.15.114:0/4095882442 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d00aacc0 con 0x55592486e000 2024-05-27T07:57:32.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.147+0000 7f77bdffb700 1 -- 172.21.15.114:0/4095882442 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77d00aa5c0 con 0x55592486e000 2024-05-27T07:57:32.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.147+0000 7f77bdffb700 1 -- 172.21.15.114:0/4095882442 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d00e3230 con 0x55592486e000 2024-05-27T07:57:32.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.149+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/4095882442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 msgr2=0x55592486e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:32.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.149+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/4095882442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x55592486e410 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f77d00aa550 tx=0x7f77d00a9fe0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.150+0000 7f77d6ffd700 1 --2- 172.21.15.114:0/3105899522 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f77c00b2bf0 0x7f77c00b5100 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x55592486a120 tx=0x7f77cc0058e0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:32.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.154+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/4095882442 shutdown_connections 2024-05-27T07:57:32.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.154+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/4095882442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x55592486e410 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.154+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/4095882442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x555924873790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.154+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/4095882442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555924873cd0 0x55592486dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.154+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/4095882442 >> 172.21.15.114:0/4095882442 conn(0x55592474ab70 msgr2=0x5559248686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:32.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.159+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/4095882442 shutdown_connections 2024-05-27T07:57:32.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.162+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/4095882442 wait complete. 2024-05-27T07:57:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.163+0000 7f77ec7dbb80 1 -- start start 2024-05-27T07:57:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.163+0000 7f77ec7dbb80 1 Processor -- start 2024-05-27T07:57:32.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.169+0000 7f77ec7dbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x5559248774c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.169+0000 7f77ec7dbb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x555924877a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.169+0000 7f77ec7dbb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555924873cd0 0x555924877f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.169+0000 7f77ec7dbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555924693020 con 0x5559246efec0 2024-05-27T07:57:32.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.169+0000 7f77ec7dbb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55592472e840 con 0x555924873cd0 2024-05-27T07:57:32.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.169+0000 7f77ec7dbb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55592487a580 con 0x55592486e000 2024-05-27T07:57:32.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.169+0000 7f77ec7dbb80 10 client.0 _ll_register_callbacks cb 0x5559247279a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:32.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.170+0000 7f77d77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x5559248774c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.170+0000 7f77d77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x5559248774c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59510/0 (socket says 172.21.15.114:59510) 2024-05-27T07:57:32.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.170+0000 7f77d77fe700 1 -- 172.21.15.114:0/2562217779 learned_addr learned my addr 172.21.15.114:0/2562217779 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:32.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.173+0000 7f77d6ffd700 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x555924877a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.173+0000 7f77d7fff700 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555924873cd0 0x555924877f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.176+0000 7f77d77fe700 1 -- 172.21.15.114:0/2562217779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 msgr2=0x555924877a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:32.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.176+0000 7f77d77fe700 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x555924877a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.176+0000 7f77d77fe700 1 -- 172.21.15.114:0/2562217779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555924873cd0 msgr2=0x555924877f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:32.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.176+0000 7f77d77fe700 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555924873cd0 0x555924877f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.176+0000 7f77d77fe700 1 -- 172.21.15.114:0/2562217779 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55592472b710 con 0x5559246efec0 2024-05-27T07:57:32.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.179+0000 7f77d6ffd700 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x555924877a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:57:32.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.179+0000 7f77d7fff700 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555924873cd0 0x555924877f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:57:32.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.188+0000 7f77d77fe700 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x5559248774c0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f77c80104b0 tx=0x7f77c80107c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d7f38d060e39ed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:32.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.188+0000 7f77b6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:57:32.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.189+0000 7f77b6ffd700 1 -- 172.21.15.114:0/2562217779 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c8018070 con 0x5559246efec0 2024-05-27T07:57:32.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.189+0000 7f77b6ffd700 1 -- 172.21.15.114:0/2562217779 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9538+0+0 (secure 0 0 0) 0x7f77c8014640 con 0x5559246efec0 2024-05-27T07:57:32.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.189+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/2562217779 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77d00aa5c0 con 0x5559246efec0 2024-05-27T07:57:32.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.191+0000 7f77b6ffd700 1 -- 172.21.15.114:0/2562217779 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f77c8022050 con 0x5559246efec0 2024-05-27T07:57:32.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.191+0000 7f77ec7dbb80 10 client.19440 fetch_fsmap learned FSMap version 99 2024-05-27T07:57:32.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.191+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/2562217779 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x5559248850d0 con 0x5559246efec0 2024-05-27T07:57:32.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.192+0000 7f77b6ffd700 1 -- 172.21.15.114:0/2562217779 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77d00aa5c0 con 0x5559246efec0 2024-05-27T07:57:32.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.192+0000 7f77b6ffd700 1 -- 172.21.15.114:0/2562217779 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 99) ==== 3490+0+0 (secure 0 0 0) 0x7f77c801d940 con 0x5559246efec0 2024-05-27T07:57:32.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.192+0000 7f77ec7dbb80 10 client.19440 fetch_fsmap finished waiting for FSMap version 99 2024-05-27T07:57:32.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.192+0000 7f77ec7dbb80 10 client.19440 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:32.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.192+0000 7f77ec7dbb80 20 client.19440 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:32.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.192+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555924887af0 0x555924889f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.192+0000 7f77ec7dbb80 4 client.19440 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:57:32.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.192+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/2562217779 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555924839a70 con 0x555924887af0 2024-05-27T07:57:32.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.197+0000 7f77d6ffd700 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555924887af0 0x555924889f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.206+0000 7f77b6ffd700 10 client.19440 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:32.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.206+0000 7f77d6ffd700 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555924887af0 0x555924889f80 crc :-1 s=READY pgs=476 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=404ebb08607870df server_cookie=81b7e9711ba09e44 in_seq=0 out_seq=0 2024-05-27T07:57:32.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.216+0000 7f77b6ffd700 1 -- 172.21.15.114:0/2562217779 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555924839a70 con 0x555924887af0 2024-05-27T07:57:32.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.216+0000 7f77b6ffd700 10 client.19440 handle_command_reply: tid=0 2024-05-27T07:57:32.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.216+0000 7f77ec7dbb80 1 client.19440 shutdown 2024-05-27T07:57:32.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.216+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/2562217779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 msgr2=0x5559248774c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:32.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.216+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x5559248774c0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f77c80104b0 tx=0x7f77c80107c0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.217+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/2562217779 shutdown_connections 2024-05-27T07:57:32.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.217+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555924887af0 0x555924889f80 crc :-1 s=READY pgs=476 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.217+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555924873cd0 0x555924877f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.217+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x5559248774c0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.217+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/2562217779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x555924877a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.217+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/2562217779 >> 172.21.15.114:0/2562217779 conn(0x55592474ab70 msgr2=0x55592486cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:32.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.217+0000 7f77b6ffd700 0 client.19440 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:32.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.217+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/2562217779 shutdown_connections 2024-05-27T07:57:32.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.217+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/2562217779 wait complete. 2024-05-27T07:57:32.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.217+0000 7f77ec7dbb80 20 client.19440 trim_cache size 0 max 16384 2024-05-27T07:57:32.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.255+0000 7f77ec7dbb80 1 Processor -- start 2024-05-27T07:57:32.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.257+0000 7f77ec7dbb80 1 -- start start 2024-05-27T07:57:32.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.257+0000 7f77ec7dbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x55592486dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.257+0000 7f77ec7dbb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55592486e000 0x555924872f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.257+0000 7f77d77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x55592486dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.257+0000 7f77d77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x55592486dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59522/0 (socket says 172.21.15.114:59522) 2024-05-27T07:57:32.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.257+0000 7f77ec7dbb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555924873cd0 0x555924877d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.257+0000 7f77d77fe700 1 -- 172.21.15.114:0/361467613 learned_addr learned my addr 172.21.15.114:0/361467613 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:32.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.258+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/361467613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f77d81b6c00 con 0x5559246efec0 2024-05-27T07:57:32.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.258+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/361467613 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f77d81b72d0 con 0x55592486e000 2024-05-27T07:57:32.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.258+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/361467613 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f77d81b55b0 con 0x555924873cd0 2024-05-27T07:57:32.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.258+0000 7f77d7fff700 1 --2- 172.21.15.114:0/361467613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555924873cd0 0x555924877d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.258+0000 7f77d77fe700 1 -- 172.21.15.114:0/361467613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555924873cd0 msgr2=0x555924877d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:32.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.258+0000 7f77d77fe700 1 --2- 172.21.15.114:0/361467613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555924873cd0 0x555924877d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.258+0000 7f77d77fe700 1 -- 172.21.15.114:0/361467613 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55592486e000 msgr2=0x555924872f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:32.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.258+0000 7f77d77fe700 1 --2- 172.21.15.114:0/361467613 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55592486e000 0x555924872f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.258+0000 7f77d77fe700 1 -- 172.21.15.114:0/361467613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55592487a2c0 con 0x5559246efec0 2024-05-27T07:57:32.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.259+0000 7f77d77fe700 1 --2- 172.21.15.114:0/361467613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x55592486dac0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f77c8007f70 tx=0x7f77c8007480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a92704d3df281 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:32.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.259+0000 7f77bd7fa700 1 -- 172.21.15.114:0/361467613 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c802b030 con 0x5559246efec0 2024-05-27T07:57:32.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.259+0000 7f77bd7fa700 1 -- 172.21.15.114:0/361467613 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77c801d8f0 con 0x5559246efec0 2024-05-27T07:57:32.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.259+0000 7f77bd7fa700 1 -- 172.21.15.114:0/361467613 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c8018070 con 0x5559246efec0 2024-05-27T07:57:32.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.261+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/361467613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 msgr2=0x55592486dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:32.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.261+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/361467613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x55592486dac0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f77c8007f70 tx=0x7f77c8007480 comp rx=0 tx=0).stop 2024-05-27T07:57:32.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.261+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/361467613 shutdown_connections 2024-05-27T07:57:32.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.261+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/361467613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555924873cd0 0x555924877d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.261+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/361467613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5559246efec0 0x55592486dac0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.261+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/361467613 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55592486e000 0x555924872f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.261+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/361467613 >> 172.21.15.114:0/361467613 conn(0x55592474ab70 msgr2=0x555924871090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:32.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.262+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/361467613 shutdown_connections 2024-05-27T07:57:32.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.262+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/361467613 wait complete. 2024-05-27T07:57:32.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.262+0000 7f77ec7dbb80 1 -- start start 2024-05-27T07:57:32.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.262+0000 7f77ec7dbb80 1 Processor -- start 2024-05-27T07:57:32.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.263+0000 7f77ec7dbb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5559246efec0 0x55592487f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.263+0000 7f77ec7dbb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x55592487ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.263+0000 7f77ec7dbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555924873cd0 0x55592487c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.263+0000 7f77ec7dbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555924878810 con 0x555924873cd0 2024-05-27T07:57:32.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.263+0000 7f77ec7dbb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f77d81b5c00 con 0x5559246efec0 2024-05-27T07:57:32.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.263+0000 7f77ec7dbb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5559248782c0 con 0x55592486e000 2024-05-27T07:57:32.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.263+0000 7f77ec7dbb80 10 client.0 _ll_register_callbacks cb 0x5559247279a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:32.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.263+0000 7f77d7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555924873cd0 0x55592487c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.263+0000 7f77d7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555924873cd0 0x55592487c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59526/0 (socket says 172.21.15.114:59526) 2024-05-27T07:57:32.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.263+0000 7f77d7fff700 1 -- 172.21.15.114:0/3067313737 learned_addr learned my addr 172.21.15.114:0/3067313737 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:32.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.264+0000 7f77d6ffd700 1 --2- 172.21.15.114:0/3067313737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x55592487ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.264+0000 7f77d7fff700 1 -- 172.21.15.114:0/3067313737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 msgr2=0x55592487ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:32.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.264+0000 7f77d7fff700 1 --2- 172.21.15.114:0/3067313737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x55592487ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.264+0000 7f77d7fff700 1 -- 172.21.15.114:0/3067313737 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5559246efec0 msgr2=0x55592487f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:32.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.264+0000 7f77d7fff700 1 --2- 172.21.15.114:0/3067313737 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5559246efec0 0x55592487f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.264+0000 7f77d7fff700 1 -- 172.21.15.114:0/3067313737 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55592487ced0 con 0x555924873cd0 2024-05-27T07:57:32.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.264+0000 7f77d7fff700 1 --2- 172.21.15.114:0/3067313737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555924873cd0 0x55592487c7a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f77d0004fd0 tx=0x7f77d0012730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9045e9f17aee93d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:32.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.264+0000 7f77b6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:57:32.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.264+0000 7f77b6ffd700 1 -- 172.21.15.114:0/3067313737 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d00aacc0 con 0x555924873cd0 2024-05-27T07:57:32.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.264+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/3067313737 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77c801d8f0 con 0x555924873cd0 2024-05-27T07:57:32.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.264+0000 7f77b6ffd700 1 -- 172.21.15.114:0/3067313737 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9538+0+0 (secure 0 0 0) 0x7f77d00a6bb0 con 0x555924873cd0 2024-05-27T07:57:32.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.265+0000 7f77b6ffd700 1 -- 172.21.15.114:0/3067313737 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f77d00b3390 con 0x555924873cd0 2024-05-27T07:57:32.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.265+0000 7f77ec7dbb80 10 client.19446 fetch_fsmap learned FSMap version 99 2024-05-27T07:57:32.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.265+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/3067313737 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x5559248f3020 con 0x555924873cd0 2024-05-27T07:57:32.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.265+0000 7f77b6ffd700 1 -- 172.21.15.114:0/3067313737 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77c801d8f0 con 0x555924873cd0 2024-05-27T07:57:32.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.265+0000 7f77b6ffd700 1 -- 172.21.15.114:0/3067313737 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 99) ==== 3490+0+0 (secure 0 0 0) 0x7f77d00a4ce0 con 0x555924873cd0 2024-05-27T07:57:32.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.265+0000 7f77ec7dbb80 10 client.19446 fetch_fsmap finished waiting for FSMap version 99 2024-05-27T07:57:32.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.265+0000 7f77ec7dbb80 10 client.19446 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:32.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.265+0000 7f77ec7dbb80 20 client.19446 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:32.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.265+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/3067313737 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5559248f3160 0x5559248f5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:32.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.265+0000 7f77ec7dbb80 4 client.19446 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:57:32.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.265+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/3067313737 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55592479ec60 con 0x5559248f3160 2024-05-27T07:57:32.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.266+0000 7f77d77fe700 1 --2- 172.21.15.114:0/3067313737 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5559248f3160 0x5559248f5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:32.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.267+0000 7f77b6ffd700 10 client.19446 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:32.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.267+0000 7f77d77fe700 1 --2- 172.21.15.114:0/3067313737 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5559248f3160 0x5559248f5580 crc :-1 s=READY pgs=477 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52169a794b962e6c server_cookie=3a0abcf1c5cb32ca in_seq=0 out_seq=0 2024-05-27T07:57:32.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.285+0000 7f77b6ffd700 1 -- 172.21.15.114:0/3067313737 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55592479ec60 con 0x5559248f3160 2024-05-27T07:57:32.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.285+0000 7f77b6ffd700 10 client.19446 handle_command_reply: tid=0 2024-05-27T07:57:32.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.285+0000 7f77ec7dbb80 1 client.19446 shutdown 2024-05-27T07:57:32.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.285+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/3067313737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555924873cd0 msgr2=0x55592487c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:32.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.285+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/3067313737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555924873cd0 0x55592487c7a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f77d0004fd0 tx=0x7f77d0012730 comp rx=0 tx=0).stop 2024-05-27T07:57:32.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.285+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/3067313737 shutdown_connections 2024-05-27T07:57:32.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.285+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/3067313737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555924873cd0 0x55592487c7a0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.285+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/3067313737 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5559248f3160 0x5559248f5580 crc :-1 s=READY pgs=477 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.285+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/3067313737 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5559246efec0 0x55592487f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.285+0000 7f77ec7dbb80 1 --2- 172.21.15.114:0/3067313737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55592486e000 0x55592487ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.285+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/3067313737 >> 172.21.15.114:0/3067313737 conn(0x55592474ab70 msgr2=0x555924872c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:32.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.285+0000 7f77b6ffd700 0 client.19446 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:32.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.287+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/3067313737 shutdown_connections 2024-05-27T07:57:32.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.287+0000 7f77ec7dbb80 1 -- 172.21.15.114:0/3067313737 wait complete. 2024-05-27T07:57:32.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.287+0000 7f77ec7dbb80 20 client.19446 trim_cache size 0 max 16384 2024-05-27T07:57:32.289 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:57:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:57:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "c1f56efe-7efd-4160-b28e-0e974e2347c2", 2024-05-27T07:57:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:57:32.289 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:57:32.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.289+0000 7f77dddc6700 1 -- 172.21.15.114:0/3105899522 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f77c00b2bf0 msgr2=0x7f77c00b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:32.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.289+0000 7f77dddc6700 1 --2- 172.21.15.114:0/3105899522 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f77c00b2bf0 0x7f77c00b5100 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x55592486a120 tx=0x7f77cc0058e0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.289+0000 7f77dddc6700 1 -- 172.21.15.114:0/3105899522 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d8119e30 msgr2=0x7f77d81ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:32.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.289+0000 7f77dddc6700 1 --2- 172.21.15.114:0/3105899522 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d8119e30 0x7f77d81ad8b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009660 tx=0x7f77d0009440 comp rx=0 tx=0).stop 2024-05-27T07:57:32.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.289+0000 7f77dddc6700 1 -- 172.21.15.114:0/3105899522 shutdown_connections 2024-05-27T07:57:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.289+0000 7f77dddc6700 1 --2- 172.21.15.114:0/3105899522 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f77c00b2bf0 0x7f77c00b5100 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.289+0000 7f77dddc6700 1 --2- 172.21.15.114:0/3105899522 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d8119e30 0x7f77d81ad8b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.289+0000 7f77dddc6700 1 --2- 172.21.15.114:0/3105899522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d811d920 0x7f77d811d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.289+0000 7f77dddc6700 1 --2- 172.21.15.114:0/3105899522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77d8119490 0x7f77d81198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.289+0000 7f77dddc6700 1 -- 172.21.15.114:0/3105899522 >> 172.21.15.114:0/3105899522 conn(0x7f77d806f4c0 msgr2=0x7f77d811e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.289+0000 7f77dddc6700 1 -- 172.21.15.114:0/3105899522 shutdown_connections 2024-05-27T07:57:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:32.289+0000 7f77dddc6700 1 -- 172.21.15.114:0/3105899522 wait complete. 2024-05-27T07:57:32.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:32 smithi114 podman[129691]: 2024-05-27 07:57:32.063333929 +0000 UTC m=+0.236469417 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:57:32.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:32 smithi114 podman[129691]: 2024-05-27 07:57:32.177118178 +0000 UTC m=+0.350253651 container create 96dacd30b5d5a415c620e33757e78e239b3982d41e3581ffa92c451f84923686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True) 2024-05-27T07:57:32.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 53807 12.14 MB/sec execute 104 sec latency 199.558 ms 2024-05-27T07:57:32.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:32 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:57:32.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:32 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:57:32.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:32 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:57:32.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:32 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:57:32.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:32 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:57:32.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:32 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:57:32.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:32 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:57:32.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:32 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:57:32.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:32 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:57:33.057 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:32 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:57:33.058 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:32 smithi114 podman[129691]: 2024-05-27 07:57:32.791787978 +0000 UTC m=+0.964923451 container init 96dacd30b5d5a415c620e33757e78e239b3982d41e3581ffa92c451f84923686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-27T07:57:33.058 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:32 smithi114 podman[129691]: 2024-05-27 07:57:32.794870851 +0000 UTC m=+0.968006324 container start 96dacd30b5d5a415c620e33757e78e239b3982d41e3581ffa92c451f84923686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-05-27T07:57:33.058 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:32 smithi114 ceph-mds[129779]: -- 172.21.15.114:0/1995103013 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5589d6b71a40 con 0x5589d6bb9180 2024-05-27T07:57:33.058 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:32 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: starting mds.g at 2024-05-27T07:57:33.058 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:32 smithi114 bash[129691]: 96dacd30b5d5a415c620e33757e78e239b3982d41e3581ffa92c451f84923686 2024-05-27T07:57:33.058 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:57:32 smithi114 systemd[1]: Started Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:57:33.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:57:33.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 54130 12.14 MB/sec execute 105 sec latency 93.961 ms 2024-05-27T07:57:33.701 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:57:33.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:33 smithi114 ceph-mon[32038]: pgmap v611: 129 pgs: 129 active+clean; 355 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 6.2 MiB/s wr, 447 op/s 2024-05-27T07:57:33.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:33 smithi163 ceph-mon[36451]: pgmap v611: 129 pgs: 129 active+clean; 355 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 6.2 MiB/s wr, 447 op/s 2024-05-27T07:57:33.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:33 smithi186 ceph-mon[36647]: pgmap v611: 129 pgs: 129 active+clean; 355 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 6.2 MiB/s wr, 447 op/s 2024-05-27T07:57:34.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 54575 12.35 MB/sec execute 106 sec latency 134.516 ms 2024-05-27T07:57:34.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:34 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6838/2945643320,v1:172.21.15.114:6839/2945643320] up:boot 2024-05-27T07:57:34.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:34 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:57:34.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:34 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:57:34.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:34 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:57:34.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:34 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6838/2945643320,v1:172.21.15.114:6839/2945643320] up:boot 2024-05-27T07:57:34.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:34 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:57:34.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:34 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:57:34.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:34 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:57:34.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:34 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6838/2945643320,v1:172.21.15.114:6839/2945643320] up:boot 2024-05-27T07:57:34.979 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:34 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:57:34.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:34 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:57:34.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:34 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:57:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.014+0000 7f42f5e84640 1 -- 172.21.15.114:0/3141867569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f01032f0 msgr2=0x7f42f010f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.014+0000 7f42f5e84640 1 --2- 172.21.15.114:0/3141867569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f01032f0 0x7f42f010f6c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f42e8009ca0 tx=0x7f42e802a8c0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.014+0000 7f42f5e84640 1 -- 172.21.15.114:0/3141867569 shutdown_connections 2024-05-27T07:57:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.014+0000 7f42f5e84640 1 --2- 172.21.15.114:0/3141867569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f01032f0 0x7f42f010f6c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.014+0000 7f42f5e84640 1 --2- 172.21.15.114:0/3141867569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42f0109040 0x7f42f0102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.014+0000 7f42f5e84640 1 --2- 172.21.15.114:0/3141867569 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f0108ba0 0x7f42f0102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.014+0000 7f42f5e84640 1 -- 172.21.15.114:0/3141867569 >> 172.21.15.114:0/3141867569 conn(0x7f42f0078270 msgr2=0x7f42f00786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.014+0000 7f42f5e84640 1 -- 172.21.15.114:0/3141867569 shutdown_connections 2024-05-27T07:57:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.014+0000 7f42f5e84640 1 -- 172.21.15.114:0/3141867569 wait complete. 2024-05-27T07:57:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.015+0000 7f42f5e84640 1 Processor -- start 2024-05-27T07:57:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.015+0000 7f42f5e84640 1 -- start start 2024-05-27T07:57:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.015+0000 7f42f5e84640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f01032f0 0x7f42f01a0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.015+0000 7f42f5e84640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f0108ba0 0x7f42f01a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.015+0000 7f42f5e84640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42f0109040 0x7f42f019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.015+0000 7f42f5e84640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f42f0112d90 con 0x7f42f0109040 2024-05-27T07:57:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.015+0000 7f42f5e84640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f42f0112c10 con 0x7f42f01032f0 2024-05-27T07:57:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.015+0000 7f42f5e84640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f42f0112f10 con 0x7f42f0108ba0 2024-05-27T07:57:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.015+0000 7f42effff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42f0109040 0x7f42f019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.015+0000 7f42e7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f0108ba0 0x7f42f01a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.015+0000 7f42effff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42f0109040 0x7f42f019a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41080/0 (socket says 172.21.15.114:41080) 2024-05-27T07:57:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.015+0000 7f42effff640 1 -- 172.21.15.114:0/3176415088 learned_addr learned my addr 172.21.15.114:0/3176415088 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42ef7fe640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f01032f0 0x7f42f01a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42effff640 1 -- 172.21.15.114:0/3176415088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f0108ba0 msgr2=0x7f42f01a0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42effff640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f0108ba0 0x7f42f01a0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42effff640 1 -- 172.21.15.114:0/3176415088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f01032f0 msgr2=0x7f42f01a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42effff640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f01032f0 0x7f42f01a0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42effff640 1 -- 172.21.15.114:0/3176415088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f019ae10 con 0x7f42f0109040 2024-05-27T07:57:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42e7fff640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f0108ba0 0x7f42f01a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:57:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42ef7fe640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f01032f0 0x7f42f01a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:57:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42effff640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42f0109040 0x7f42f019a6e0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f42e8005cc0 tx=0x7f42e8031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42ed7fa640 1 -- 172.21.15.114:0/3176415088 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8003b70 con 0x7f42f0109040 2024-05-27T07:57:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42f5e84640 1 -- 172.21.15.114:0/3176415088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f019b040 con 0x7f42f0109040 2024-05-27T07:57:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42ed7fa640 1 -- 172.21.15.114:0/3176415088 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42e80318f0 con 0x7f42f0109040 2024-05-27T07:57:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42ed7fa640 1 -- 172.21.15.114:0/3176415088 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e802f950 con 0x7f42f0109040 2024-05-27T07:57:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.016+0000 7f42f5e84640 1 -- 172.21.15.114:0/3176415088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f01a7300 con 0x7f42f0109040 2024-05-27T07:57:35.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.019+0000 7f42ed7fa640 1 -- 172.21.15.114:0/3176415088 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f42e802fbf0 con 0x7f42f0109040 2024-05-27T07:57:35.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.019+0000 7f42fce10740 1 Processor -- start 2024-05-27T07:57:35.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.019+0000 7f42ed7fa640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f42dc0b29d0 0x7f42dc0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.019+0000 7f42fce10740 1 -- start start 2024-05-27T07:57:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42ef7fe640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f42dc0b29d0 0x7f42dc0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42ed7fa640 1 -- 172.21.15.114:0/3176415088 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 9538+0+0 (secure 0 0 0) 0x7f42e8103bc0 con 0x7f42f0109040 2024-05-27T07:57:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42fce10740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf301b0 0x55a69c2d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42fce10740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 0x55a69c2d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42fce10740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2d1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42fce10740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a69c243c30 con 0x55a69bf072f0 2024-05-27T07:57:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42fce10740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a69c2337d0 con 0x55a69bf0f4d0 2024-05-27T07:57:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42fce10740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a69c237810 con 0x55a69bf301b0 2024-05-27T07:57:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 0x55a69c2d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 0x55a69c2d1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41090/0 (socket says 172.21.15.114:41090) 2024-05-27T07:57:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42e7fff640 1 -- 172.21.15.114:0/2238648278 learned_addr learned my addr 172.21.15.114:0/2238648278 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42ef7fe640 1 --2- 172.21.15.114:0/2238648278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf301b0 0x55a69c2d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42ef7fe640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f42dc0b29d0 0x7f42dc0b4ee0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f42f0103880 tx=0x7f42e0005f90 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42effff640 1 --2- 172.21.15.114:0/2238648278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2d1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42e7fff640 1 -- 172.21.15.114:0/2238648278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf301b0 msgr2=0x55a69c2d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42e7fff640 1 --2- 172.21.15.114:0/2238648278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf301b0 0x55a69c2d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42e7fff640 1 -- 172.21.15.114:0/2238648278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 msgr2=0x55a69c2d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42e7fff640 1 --2- 172.21.15.114:0/2238648278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2d1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42e7fff640 1 -- 172.21.15.114:0/2238648278 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a69bf48210 con 0x55a69bf072f0 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42e7fff640 1 --2- 172.21.15.114:0/2238648278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 0x55a69c2d1910 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f42d8009f50 tx=0x7f42d80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9ce3890dc98140 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.020+0000 7f42e5ffb640 1 -- 172.21.15.114:0/2238648278 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d801e070 con 0x55a69bf072f0 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.021+0000 7f42e5ffb640 1 -- 172.21.15.114:0/2238648278 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42d800f9f0 con 0x55a69bf072f0 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.021+0000 7f42e5ffb640 1 -- 172.21.15.114:0/2238648278 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d80199f0 con 0x55a69bf072f0 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.021+0000 7f42fce10740 1 -- 172.21.15.114:0/2238648278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 msgr2=0x55a69c2d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.021+0000 7f42fce10740 1 --2- 172.21.15.114:0/2238648278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 0x55a69c2d1910 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f42d8009f50 tx=0x7f42d80118d0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.021+0000 7f42fce10740 1 -- 172.21.15.114:0/2238648278 shutdown_connections 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.021+0000 7f42fce10740 1 --2- 172.21.15.114:0/2238648278 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.021+0000 7f42fce10740 1 --2- 172.21.15.114:0/2238648278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 0x55a69c2d1910 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.021+0000 7f42fce10740 1 --2- 172.21.15.114:0/2238648278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf301b0 0x55a69c2d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.021+0000 7f42fce10740 1 -- 172.21.15.114:0/2238648278 >> 172.21.15.114:0/2238648278 conn(0x55a69bf24300 msgr2=0x55a69c2cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.021+0000 7f42fce10740 1 -- 172.21.15.114:0/2238648278 shutdown_connections 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.021+0000 7f42fce10740 1 -- 172.21.15.114:0/2238648278 wait complete. 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.022+0000 7f42fce10740 1 -- start start 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.022+0000 7f42fce10740 1 Processor -- start 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.022+0000 7f42fce10740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf072f0 0x55a69c2dd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.022+0000 7f42fce10740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf0f4d0 0x55a69c2dda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.022+0000 7f42fce10740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf301b0 0x55a69c2db940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.022+0000 7f42fce10740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a69bf19790 con 0x55a69bf0f4d0 2024-05-27T07:57:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.022+0000 7f42fce10740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a69c1ef890 con 0x55a69bf072f0 2024-05-27T07:57:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.022+0000 7f42fce10740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a69c2de500 con 0x55a69bf301b0 2024-05-27T07:57:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42fce10740 10 client.0 _ll_register_callbacks cb 0x55a69bf47a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42effff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf301b0 0x55a69c2db940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf0f4d0 0x55a69c2dda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42effff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf301b0 0x55a69c2db940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35652/0 (socket says 172.21.15.114:35652) 2024-05-27T07:57:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42ef7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf072f0 0x55a69c2dd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42effff640 1 -- 172.21.15.114:0/1304132650 learned_addr learned my addr 172.21.15.114:0/1304132650 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf0f4d0 0x55a69c2dda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41100/0 (socket says 172.21.15.114:41100) 2024-05-27T07:57:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42e7fff640 1 -- 172.21.15.114:0/1304132650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf301b0 msgr2=0x55a69c2db940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42e7fff640 1 --2- 172.21.15.114:0/1304132650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf301b0 0x55a69c2db940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42e7fff640 1 -- 172.21.15.114:0/1304132650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf072f0 msgr2=0x55a69c2dd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42e7fff640 1 --2- 172.21.15.114:0/1304132650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf072f0 0x55a69c2dd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42e7fff640 1 -- 172.21.15.114:0/1304132650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a69c014620 con 0x55a69bf0f4d0 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42effff640 1 --2- 172.21.15.114:0/1304132650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf301b0 0x55a69c2db940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42ef7fe640 1 --2- 172.21.15.114:0/1304132650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf072f0 0x55a69c2dd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42e7fff640 1 --2- 172.21.15.114:0/1304132650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf0f4d0 0x55a69c2dda00 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x55a69c2d2780 tx=0x7f42d8009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d521d193edbaff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42ceffd640 1 -- 172.21.15.114:0/1304132650 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d801e040 con 0x55a69bf0f4d0 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42ceffd640 1 -- 172.21.15.114:0/1304132650 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9538+0+0 (secure 0 0 0) 0x7f42d800f6c0 con 0x55a69bf0f4d0 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42fce10740 1 -- 172.21.15.114:0/1304132650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42d800f9f0 con 0x55a69bf0f4d0 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42ceffd640 1 -- 172.21.15.114:0/1304132650 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f42d8031050 con 0x55a69bf0f4d0 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.023+0000 7f42fce10740 10 client.19464 fetch_fsmap learned FSMap version 100 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.024+0000 7f42fce10740 1 -- 172.21.15.114:0/1304132650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55a69c2ea860 con 0x55a69bf0f4d0 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.024+0000 7f42ceffd640 1 -- 172.21.15.114:0/1304132650 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42d800f9f0 con 0x55a69bf0f4d0 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.024+0000 7f42ceffd640 1 -- 172.21.15.114:0/1304132650 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 100) ==== 4095+0+0 (secure 0 0 0) 0x7f42d802c030 con 0x55a69bf0f4d0 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.024+0000 7f42fce10740 10 client.19464 fetch_fsmap finished waiting for FSMap version 100 2024-05-27T07:57:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.024+0000 7f42fce10740 10 client.19464 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.024+0000 7f42fce10740 20 client.19464 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.024+0000 7f42fce10740 1 --2- 172.21.15.114:0/1304132650 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a69beeed80 0x55a69c2ee080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.024+0000 7f42fce10740 4 client.19464 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:57:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.024+0000 7f42fce10740 1 -- 172.21.15.114:0/1304132650 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a69c245b80 con 0x55a69beeed80 2024-05-27T07:57:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.024+0000 7f42ef7fe640 1 --2- 172.21.15.114:0/1304132650 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a69beeed80 0x55a69c2ee080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.024+0000 7f42ceffd640 10 client.19464 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.025+0000 7f42ef7fe640 1 --2- 172.21.15.114:0/1304132650 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a69beeed80 0x55a69c2ee080 crc :-1 s=READY pgs=478 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e10dd4d9a1ee8c5b server_cookie=3bf4e81827906162 in_seq=0 out_seq=0 2024-05-27T07:57:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42ceffd640 1 -- 172.21.15.114:0/1304132650 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a69c245b80 con 0x55a69beeed80 2024-05-27T07:57:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42ceffd640 10 client.19464 handle_command_reply: tid=0 2024-05-27T07:57:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42fce10740 1 client.19464 shutdown 2024-05-27T07:57:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42fce10740 1 -- 172.21.15.114:0/1304132650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf0f4d0 msgr2=0x55a69c2dda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42fce10740 1 --2- 172.21.15.114:0/1304132650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf0f4d0 0x55a69c2dda00 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x55a69c2d2780 tx=0x7f42d8009ce0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42fce10740 1 -- 172.21.15.114:0/1304132650 shutdown_connections 2024-05-27T07:57:35.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42fce10740 1 --2- 172.21.15.114:0/1304132650 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a69beeed80 0x55a69c2ee080 crc :-1 s=READY pgs=478 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42fce10740 1 --2- 172.21.15.114:0/1304132650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf301b0 0x55a69c2db940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42ceffd640 0 client.19464 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:35.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42fce10740 1 --2- 172.21.15.114:0/1304132650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf0f4d0 0x55a69c2dda00 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42fce10740 1 --2- 172.21.15.114:0/1304132650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf072f0 0x55a69c2dd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42fce10740 1 -- 172.21.15.114:0/1304132650 >> 172.21.15.114:0/1304132650 conn(0x55a69bf24300 msgr2=0x55a69c2d1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:35.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42fce10740 1 -- 172.21.15.114:0/1304132650 shutdown_connections 2024-05-27T07:57:35.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42fce10740 1 -- 172.21.15.114:0/1304132650 wait complete. 2024-05-27T07:57:35.033 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.026+0000 7f42fce10740 20 client.19464 trim_cache size 0 max 16384 2024-05-27T07:57:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.043+0000 7f42fce10740 1 Processor -- start 2024-05-27T07:57:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.043+0000 7f42fce10740 1 -- start start 2024-05-27T07:57:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.043+0000 7f42fce10740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69beeed80 0x55a69c2cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.043+0000 7f42fce10740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf072f0 0x55a69c2f3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.043+0000 7f42fce10740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.043+0000 7f42fce10740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f42f01b0fd0 con 0x55a69beeed80 2024-05-27T07:57:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.043+0000 7f42fce10740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f42f01b16a0 con 0x55a69bf0f4d0 2024-05-27T07:57:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.043+0000 7f42fce10740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f42f01aff70 con 0x55a69bf072f0 2024-05-27T07:57:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.043+0000 7f42ef7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69beeed80 0x55a69c2cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.043+0000 7f42effff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.043+0000 7f42ef7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69beeed80 0x55a69c2cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41114/0 (socket says 172.21.15.114:41114) 2024-05-27T07:57:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.043+0000 7f42ef7fe640 1 -- 172.21.15.114:0/1309617123 learned_addr learned my addr 172.21.15.114:0/1309617123 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.043+0000 7f42e7fff640 1 --2- 172.21.15.114:0/1309617123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf072f0 0x55a69c2f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.044+0000 7f42effff640 1 -- 172.21.15.114:0/1309617123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf072f0 msgr2=0x55a69c2f3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.044+0000 7f42effff640 1 --2- 172.21.15.114:0/1309617123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf072f0 0x55a69c2f3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.044+0000 7f42effff640 1 -- 172.21.15.114:0/1309617123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69beeed80 msgr2=0x55a69c2cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.044+0000 7f42effff640 1 --2- 172.21.15.114:0/1309617123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69beeed80 0x55a69c2cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.044+0000 7f42effff640 1 -- 172.21.15.114:0/1309617123 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a69c2e87d0 con 0x55a69bf0f4d0 2024-05-27T07:57:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.044+0000 7f42ef7fe640 1 --2- 172.21.15.114:0/1309617123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69beeed80 0x55a69c2cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:57:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.044+0000 7f42effff640 1 --2- 172.21.15.114:0/1309617123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2f3d60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f42e80ca8b0 tx=0x7f42e8009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf81e33552d2df98 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.044+0000 7f42e57fa640 1 -- 172.21.15.114:0/1309617123 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e80ebe60 con 0x55a69bf0f4d0 2024-05-27T07:57:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.044+0000 7f42e57fa640 1 -- 172.21.15.114:0/1309617123 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42e8036040 con 0x55a69bf0f4d0 2024-05-27T07:57:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.044+0000 7f42e57fa640 1 -- 172.21.15.114:0/1309617123 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e80cef60 con 0x55a69bf0f4d0 2024-05-27T07:57:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.045+0000 7f42fce10740 1 -- 172.21.15.114:0/1309617123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 msgr2=0x55a69c2f3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.045+0000 7f42fce10740 1 --2- 172.21.15.114:0/1309617123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2f3d60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f42e80ca8b0 tx=0x7f42e8009900 comp rx=0 tx=0).stop 2024-05-27T07:57:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.045+0000 7f42fce10740 1 -- 172.21.15.114:0/1309617123 shutdown_connections 2024-05-27T07:57:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.045+0000 7f42fce10740 1 --2- 172.21.15.114:0/1309617123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2f3d60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.045+0000 7f42fce10740 1 --2- 172.21.15.114:0/1309617123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69bf072f0 0x55a69c2f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.045+0000 7f42fce10740 1 --2- 172.21.15.114:0/1309617123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69beeed80 0x55a69c2cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.045+0000 7f42fce10740 1 -- 172.21.15.114:0/1309617123 >> 172.21.15.114:0/1309617123 conn(0x55a69bf24300 msgr2=0x55a69c2cd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.045+0000 7f42fce10740 1 -- 172.21.15.114:0/1309617123 shutdown_connections 2024-05-27T07:57:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.045+0000 7f42fce10740 1 -- 172.21.15.114:0/1309617123 wait complete. 2024-05-27T07:57:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.045+0000 7f42fce10740 1 -- start start 2024-05-27T07:57:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.046+0000 7f42fce10740 1 Processor -- start 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.046+0000 7f42fce10740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69beeed80 0x55a69c2e0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.046+0000 7f42fce10740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 0x55a69c2e0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.046+0000 7f42fce10740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2e1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.046+0000 7f42fce10740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f42f01af8a0 con 0x55a69bf072f0 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.046+0000 7f42fce10740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f42f01af250 con 0x55a69bf0f4d0 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.046+0000 7f42fce10740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a69c3546f0 con 0x55a69beeed80 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.046+0000 7f42fce10740 10 client.0 _ll_register_callbacks cb 0x7f42d801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.046+0000 7f42e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 0x55a69c2e0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.046+0000 7f42e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 0x55a69c2e0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41128/0 (socket says 172.21.15.114:41128) 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.046+0000 7f42e7fff640 1 -- 172.21.15.114:0/1809185455 learned_addr learned my addr 172.21.15.114:0/1809185455 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.046+0000 7f42ef7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69beeed80 0x55a69c2e0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.046+0000 7f42effff640 1 --2- 172.21.15.114:0/1809185455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2e1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42e7fff640 1 -- 172.21.15.114:0/1809185455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69beeed80 msgr2=0x55a69c2e0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42e7fff640 1 --2- 172.21.15.114:0/1809185455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69beeed80 0x55a69c2e0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42e7fff640 1 -- 172.21.15.114:0/1809185455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 msgr2=0x55a69c2e1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42e7fff640 1 --2- 172.21.15.114:0/1809185455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2e1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42e7fff640 1 -- 172.21.15.114:0/1809185455 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a69c354870 con 0x55a69bf072f0 2024-05-27T07:57:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42effff640 1 --2- 172.21.15.114:0/1809185455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2e1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:57:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42e7fff640 1 --2- 172.21.15.114:0/1809185455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 0x55a69c2e0ac0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x55a69c2c7e60 tx=0x7f42d800fef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848efb35fbe39320 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:57:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42ceffd640 1 -- 172.21.15.114:0/1809185455 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d801e040 con 0x55a69bf072f0 2024-05-27T07:57:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42ceffd640 1 -- 172.21.15.114:0/1809185455 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9538+0+0 (secure 0 0 0) 0x7f42d800f950 con 0x55a69bf072f0 2024-05-27T07:57:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42fce10740 1 -- 172.21.15.114:0/1809185455 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42e8036040 con 0x55a69bf072f0 2024-05-27T07:57:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42ceffd640 1 -- 172.21.15.114:0/1809185455 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f42d8009e20 con 0x55a69bf072f0 2024-05-27T07:57:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42fce10740 10 client.19470 fetch_fsmap learned FSMap version 100 2024-05-27T07:57:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42fce10740 1 -- 172.21.15.114:0/1809185455 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55a69c355ae0 con 0x55a69bf072f0 2024-05-27T07:57:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42ceffd640 1 -- 172.21.15.114:0/1809185455 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42e8036040 con 0x55a69bf072f0 2024-05-27T07:57:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42ceffd640 1 -- 172.21.15.114:0/1809185455 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 100) ==== 4095+0+0 (secure 0 0 0) 0x7f42d802c070 con 0x55a69bf072f0 2024-05-27T07:57:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42fce10740 10 client.19470 fetch_fsmap finished waiting for FSMap version 100 2024-05-27T07:57:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42fce10740 10 client.19470 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42fce10740 20 client.19470 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42fce10740 1 --2- 172.21.15.114:0/1809185455 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a69bf301b0 0x55a69c358180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42fce10740 4 client.19470 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:57:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.047+0000 7f42fce10740 1 -- 172.21.15.114:0/1809185455 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a69c23f360 con 0x55a69bf301b0 2024-05-27T07:57:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.048+0000 7f42ef7fe640 1 --2- 172.21.15.114:0/1809185455 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a69bf301b0 0x55a69c358180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.048+0000 7f42ceffd640 10 client.19470 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:35.048+0000 7f42ef7fe640 1 --2- 172.21.15.114:0/1809185455 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a69bf301b0 0x55a69c358180 crc :-1 s=READY pgs=479 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2713d290f07bbc4c server_cookie=461216c902e4bf93 in_seq=0 out_seq=0 2024-05-27T07:57:35.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 54888 12.31 MB/sec execute 107 sec latency 457.283 ms 2024-05-27T07:57:35.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:35 smithi114 ceph-mon[32038]: pgmap v612: 129 pgs: 129 active+clean; 360 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.6 MiB/s wr, 400 op/s 2024-05-27T07:57:35.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.933+0000 7f7ee8e27700 1 -- 172.21.15.114:0/640703295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ee410a920 msgr2=0x7f7ee410ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:35.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.933+0000 7f7ee8e27700 1 --2- 172.21.15.114:0/640703295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ee410a920 0x7f7ee410ad30 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc009990 tx=0x7f7ecc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.936+0000 7f7ee8e27700 1 -- 172.21.15.114:0/640703295 shutdown_connections 2024-05-27T07:57:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.936+0000 7f7ee8e27700 1 --2- 172.21.15.114:0/640703295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7ee411d920 0x7f7ee411fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.936+0000 7f7ee8e27700 1 --2- 172.21.15.114:0/640703295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ee410a920 0x7f7ee410ad30 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.936+0000 7f7ee8e27700 1 --2- 172.21.15.114:0/640703295 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ee410b270 0x7f7ee411d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.937+0000 7f7ee8e27700 1 -- 172.21.15.114:0/640703295 >> 172.21.15.114:0/640703295 conn(0x7f7ee406f4c0 msgr2=0x7f7ee406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:35.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.945+0000 7f7ee8e27700 1 -- 172.21.15.114:0/640703295 shutdown_connections 2024-05-27T07:57:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.947+0000 7f7ee8e27700 1 -- 172.21.15.114:0/640703295 wait complete. 2024-05-27T07:57:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.948+0000 7f7ee8e27700 1 Processor -- start 2024-05-27T07:57:35.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.952+0000 7f7ee8e27700 1 -- start start 2024-05-27T07:57:35.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.952+0000 7f7ee8e27700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ee410a920 0x7f7ee411bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.952+0000 7f7ee8e27700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ee410b270 0x7f7ee4114e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.952+0000 7f7ee8e27700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7ee411d920 0x7f7ee4115480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.952+0000 7f7ee8e27700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee4123390 con 0x7f7ee410b270 2024-05-27T07:57:35.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.952+0000 7f7ee8e27700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee4123210 con 0x7f7ee411d920 2024-05-27T07:57:35.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.952+0000 7f7ee8e27700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee4123510 con 0x7f7ee410a920 2024-05-27T07:57:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.954+0000 7f7ee2d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7ee411d920 0x7f7ee4115480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.954+0000 7f7ee2d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7ee411d920 0x7f7ee4115480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:38292/0 (socket says 172.21.15.114:38292) 2024-05-27T07:57:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.954+0000 7f7ee2d9d700 1 -- 172.21.15.114:0/2300856744 learned_addr learned my addr 172.21.15.114:0/2300856744 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.957+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ee410b270 0x7f7ee4114e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.957+0000 7f7ee259c700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ee410a920 0x7f7ee411bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.959+0000 7f7ee2d9d700 1 -- 172.21.15.114:0/2300856744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ee410a920 msgr2=0x7f7ee411bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.959+0000 7f7ee2d9d700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ee410a920 0x7f7ee411bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.959+0000 7f7ee2d9d700 1 -- 172.21.15.114:0/2300856744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ee410b270 msgr2=0x7f7ee4114e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.959+0000 7f7ee2d9d700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ee410b270 0x7f7ee4114e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.959+0000 7f7ee2d9d700 1 -- 172.21.15.114:0/2300856744 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee4115cb0 con 0x7f7ee411d920 2024-05-27T07:57:35.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.966+0000 7f7ee2d9d700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7ee411d920 0x7f7ee4115480 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400b450 tx=0x7f7ed400b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:35.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.971+0000 7f7edb7fe700 1 -- 172.21.15.114:0/2300856744 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4015020 con 0x7f7ee411d920 2024-05-27T07:57:35.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.971+0000 7f7edb7fe700 1 -- 172.21.15.114:0/2300856744 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ed4011ce0 con 0x7f7ee411d920 2024-05-27T07:57:35.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.971+0000 7f7edb7fe700 1 -- 172.21.15.114:0/2300856744 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4010dd0 con 0x7f7ee411d920 2024-05-27T07:57:35.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.972+0000 7f7ee8e27700 1 -- 172.21.15.114:0/2300856744 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee41199a0 con 0x7f7ee411d920 2024-05-27T07:57:35.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.972+0000 7f7ee8e27700 1 -- 172.21.15.114:0/2300856744 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee4119e90 con 0x7f7ee411d920 2024-05-27T07:57:35.976 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:35 smithi163 ceph-mon[36451]: pgmap v612: 129 pgs: 129 active+clean; 360 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.6 MiB/s wr, 400 op/s 2024-05-27T07:57:35.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.978+0000 7f7ef783cb80 1 Processor -- start 2024-05-27T07:57:35.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.978+0000 7f7ef783cb80 1 -- start start 2024-05-27T07:57:35.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.978+0000 7f7ef783cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1a51e40 0x5602b1bd6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.979+0000 7f7ef783cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bd7070 0x5602b1bd0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.979+0000 7f7ef783cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bd1120 0x5602b1bd1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.979+0000 7f7ef783cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5602b1afed70 con 0x5602b1a51e40 2024-05-27T07:57:35.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.979+0000 7f7ef783cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5602b1aee910 con 0x5602b1bd1120 2024-05-27T07:57:35.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.979+0000 7f7ef783cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5602b1af2950 con 0x5602b1bd7070 2024-05-27T07:57:35.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:35 smithi186 ceph-mon[36647]: pgmap v612: 129 pgs: 129 active+clean; 360 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.6 MiB/s wr, 400 op/s 2024-05-27T07:57:35.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.983+0000 7f7ee259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bd7070 0x5602b1bd0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.983+0000 7f7ee1d9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bd1120 0x5602b1bd1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.983+0000 7f7ee1d9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bd1120 0x5602b1bd1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:38306/0 (socket says 172.21.15.114:38306) 2024-05-27T07:57:35.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.983+0000 7f7ee1d9b700 1 -- 172.21.15.114:0/417238786 learned_addr learned my addr 172.21.15.114:0/417238786 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:35.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.983+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/417238786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1a51e40 0x5602b1bd6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.988+0000 7f7edb7fe700 1 -- 172.21.15.114:0/2300856744 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f7ed4018020 con 0x7f7ee411d920 2024-05-27T07:57:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.989+0000 7f7edb7fe700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7ed00b2c00 0x7f7ed00b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.989+0000 7f7edb7fe700 1 -- 172.21.15.114:0/2300856744 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 9538+0+0 (secure 0 0 0) 0x7f7ed40e1cb0 con 0x7f7ee411d920 2024-05-27T07:57:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.990+0000 7f7ee1d9b700 1 -- 172.21.15.114:0/417238786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bd7070 msgr2=0x5602b1bd0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.990+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/417238786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bd7070 0x5602b1bd0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.990+0000 7f7ee1d9b700 1 -- 172.21.15.114:0/417238786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1a51e40 msgr2=0x5602b1bd6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.990+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/417238786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1a51e40 0x5602b1bd6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.990+0000 7f7ee1d9b700 1 -- 172.21.15.114:0/417238786 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602b1a97fb0 con 0x5602b1bd1120 2024-05-27T07:57:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.991+0000 7f7ee259c700 1 --2- 172.21.15.114:0/417238786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bd7070 0x5602b1bd0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:57:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.991+0000 7f7ee2d9d700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7ed00b2c00 0x7f7ed00b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:35.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.994+0000 7f7ee2d9d700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7ed00b2c00 0x7f7ed00b5110 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40df110 tx=0x7f7ed4004800 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:36.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:35.999+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/417238786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bd1120 0x5602b1bd1530 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7edc0078d0 tx=0x7f7edc009380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f3dd35d8b0a5000 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:36.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.006+0000 7f7ed8ff9700 1 -- 172.21.15.114:0/417238786 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc01d070 con 0x5602b1bd1120 2024-05-27T07:57:36.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.006+0000 7f7ed8ff9700 1 -- 172.21.15.114:0/417238786 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7edc016c20 con 0x5602b1bd1120 2024-05-27T07:57:36.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.006+0000 7f7ed8ff9700 1 -- 172.21.15.114:0/417238786 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc018a50 con 0x5602b1bd1120 2024-05-27T07:57:36.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.008+0000 7f7ef783cb80 1 -- 172.21.15.114:0/417238786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bd1120 msgr2=0x5602b1bd1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:36.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.008+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/417238786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bd1120 0x5602b1bd1530 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7edc0078d0 tx=0x7f7edc009380 comp rx=0 tx=0).stop 2024-05-27T07:57:36.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.008+0000 7f7ef783cb80 1 -- 172.21.15.114:0/417238786 shutdown_connections 2024-05-27T07:57:36.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.008+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/417238786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bd1120 0x5602b1bd1530 secure :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7edc0078d0 tx=0x7f7edc009380 comp rx=0 tx=0).stop 2024-05-27T07:57:36.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.008+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/417238786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1a51e40 0x5602b1bd6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.008+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/417238786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bd7070 0x5602b1bd0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.008+0000 7f7ef783cb80 1 -- 172.21.15.114:0/417238786 >> 172.21.15.114:0/417238786 conn(0x5602b1a54770 msgr2=0x5602b1bcbd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.014+0000 7f7ef783cb80 1 -- 172.21.15.114:0/417238786 shutdown_connections 2024-05-27T07:57:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.015+0000 7f7ef783cb80 1 -- 172.21.15.114:0/417238786 wait complete. 2024-05-27T07:57:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.015+0000 7f7ef783cb80 1 -- start start 2024-05-27T07:57:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.015+0000 7f7ef783cb80 1 Processor -- start 2024-05-27T07:57:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.016+0000 7f7ef783cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1a51e40 0x5602b1bda9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.016+0000 7f7ef783cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bd7070 0x5602b1bdaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.016+0000 7f7ef783cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1bdb420 0x5602b1be6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.016+0000 7f7ef783cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5602b19f4fa0 con 0x5602b1bdb420 2024-05-27T07:57:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.016+0000 7f7ef783cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5602b1a56da0 con 0x5602b1bd7070 2024-05-27T07:57:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.016+0000 7f7ef783cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5602b1bddb60 con 0x5602b1a51e40 2024-05-27T07:57:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.016+0000 7f7ef783cb80 10 client.0 _ll_register_callbacks cb 0x5602b1a855b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:36.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.017+0000 7f7ee1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1bdb420 0x5602b1be6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:36.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.017+0000 7f7ee1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1bdb420 0x5602b1be6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41172/0 (socket says 172.21.15.114:41172) 2024-05-27T07:57:36.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.017+0000 7f7ee1d9b700 1 -- 172.21.15.114:0/347514066 learned_addr learned my addr 172.21.15.114:0/347514066 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:36.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.017+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/347514066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1a51e40 0x5602b1bda9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:36.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.019+0000 7f7ee259c700 1 --2- 172.21.15.114:0/347514066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bd7070 0x5602b1bdaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:36.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.025+0000 7f7ee259c700 1 -- 172.21.15.114:0/347514066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1a51e40 msgr2=0x5602b1bda9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:36.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.025+0000 7f7ee259c700 1 --2- 172.21.15.114:0/347514066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1a51e40 0x5602b1bda9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.025+0000 7f7ee259c700 1 -- 172.21.15.114:0/347514066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1bdb420 msgr2=0x5602b1be6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:36.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.025+0000 7f7ee259c700 1 --2- 172.21.15.114:0/347514066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1bdb420 0x5602b1be6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.025+0000 7f7ee259c700 1 -- 172.21.15.114:0/347514066 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602b1a3e8d0 con 0x5602b1bd7070 2024-05-27T07:57:36.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.030+0000 7f7ee259c700 1 --2- 172.21.15.114:0/347514066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bd7070 0x5602b1bdaee0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc006010 tx=0x7f7ecc00bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ff91d91d825fd4e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:36.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.031+0000 7f7ec9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:57:36.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.039+0000 7f7ec9ffb700 1 -- 172.21.15.114:0/347514066 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc003af0 con 0x5602b1bd7070 2024-05-27T07:57:36.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.039+0000 7f7ec9ffb700 1 -- 172.21.15.114:0/347514066 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9538+0+0 (secure 0 0 0) 0x7f7ecc003c90 con 0x5602b1bd7070 2024-05-27T07:57:36.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.040+0000 7f7ef783cb80 1 -- 172.21.15.114:0/347514066 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7edc016c20 con 0x5602b1bd7070 2024-05-27T07:57:36.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.045+0000 7f7ec9ffb700 1 -- 172.21.15.114:0/347514066 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f7ecc005110 con 0x5602b1bd7070 2024-05-27T07:57:36.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.045+0000 7f7ef783cb80 10 client.28808 fetch_fsmap learned FSMap version 100 2024-05-27T07:57:36.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.045+0000 7f7ef783cb80 1 -- 172.21.15.114:0/347514066 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x5602b1aa8760 con 0x5602b1bd7070 2024-05-27T07:57:36.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.049+0000 7f7ec9ffb700 1 -- 172.21.15.114:0/347514066 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7edc016c20 con 0x5602b1bd7070 2024-05-27T07:57:36.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.049+0000 7f7ec9ffb700 1 -- 172.21.15.114:0/347514066 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 100) ==== 4095+0+0 (secure 0 0 0) 0x7f7ecc021020 con 0x5602b1bd7070 2024-05-27T07:57:36.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.049+0000 7f7ef783cb80 10 client.28808 fetch_fsmap finished waiting for FSMap version 100 2024-05-27T07:57:36.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.049+0000 7f7ef783cb80 10 client.28808 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:36.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.049+0000 7f7ef783cb80 20 client.28808 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:36.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.049+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/347514066 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602b1beb500 0x5602b1bed950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:36.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.049+0000 7f7ef783cb80 4 client.28808 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:57:36.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.049+0000 7f7ef783cb80 1 -- 172.21.15.114:0/347514066 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602b1b00cc0 con 0x5602b1beb500 2024-05-27T07:57:36.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.053+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/347514066 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602b1beb500 0x5602b1bed950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:36.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.060+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/347514066 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602b1beb500 0x5602b1bed950 crc :-1 s=READY pgs=480 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=747c0de426f96fa2 server_cookie=6cb4d3a3f493d128 in_seq=0 out_seq=0 2024-05-27T07:57:36.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.061+0000 7f7ec9ffb700 10 client.28808 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:36.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.065+0000 7f7ec9ffb700 1 -- 172.21.15.114:0/347514066 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5602b1b00cc0 con 0x5602b1beb500 2024-05-27T07:57:36.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.065+0000 7f7ec9ffb700 10 client.28808 handle_command_reply: tid=0 2024-05-27T07:57:36.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.065+0000 7f7ef783cb80 1 client.28808 shutdown 2024-05-27T07:57:36.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.066+0000 7f7ef783cb80 1 -- 172.21.15.114:0/347514066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bd7070 msgr2=0x5602b1bdaee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:36.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.066+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/347514066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bd7070 0x5602b1bdaee0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc006010 tx=0x7f7ecc00bf00 comp rx=0 tx=0).stop 2024-05-27T07:57:36.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.070+0000 7f7ef783cb80 1 -- 172.21.15.114:0/347514066 shutdown_connections 2024-05-27T07:57:36.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.070+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/347514066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1bdb420 0x5602b1be6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.070+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/347514066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1a51e40 0x5602b1bda9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.070+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/347514066 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602b1beb500 0x5602b1bed950 crc :-1 s=READY pgs=480 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.070+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/347514066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bd7070 0x5602b1bdaee0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.070+0000 7f7ef783cb80 1 -- 172.21.15.114:0/347514066 >> 172.21.15.114:0/347514066 conn(0x5602b1a54770 msgr2=0x5602b1bcfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:36.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.073+0000 7f7ec9ffb700 0 client.28808 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:36.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.073+0000 7f7ef783cb80 1 -- 172.21.15.114:0/347514066 shutdown_connections 2024-05-27T07:57:36.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.073+0000 7f7ef783cb80 1 -- 172.21.15.114:0/347514066 wait complete. 2024-05-27T07:57:36.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.073+0000 7f7ef783cb80 20 client.28808 trim_cache size 0 max 16384 2024-05-27T07:57:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.116+0000 7f7ef783cb80 1 Processor -- start 2024-05-27T07:57:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.121+0000 7f7ef783cb80 1 -- start start 2024-05-27T07:57:36.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.121+0000 7f7ef783cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1a51e40 0x5602b1bd0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:36.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.121+0000 7f7ef783cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1bd7070 0x5602b1be6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:36.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.122+0000 7f7ef783cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bdb420 0x5602b1bc5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:36.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.122+0000 7f7ef783cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec400ebf0 con 0x5602b1bd7070 2024-05-27T07:57:36.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.122+0000 7f7ef783cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec400f2c0 con 0x5602b1a51e40 2024-05-27T07:57:36.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.122+0000 7f7ef783cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec400db70 con 0x5602b1bdb420 2024-05-27T07:57:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.126+0000 7f7ee259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1bd7070 0x5602b1be6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:36.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.126+0000 7f7ee259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1bd7070 0x5602b1be6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41178/0 (socket says 172.21.15.114:41178) 2024-05-27T07:57:36.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.126+0000 7f7ee259c700 1 -- 172.21.15.114:0/539411000 learned_addr learned my addr 172.21.15.114:0/539411000 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:36.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.126+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/539411000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1a51e40 0x5602b1bd0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:36.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.129+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/539411000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bdb420 0x5602b1bc5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:36.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.131+0000 7f7ee259c700 1 -- 172.21.15.114:0/539411000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bdb420 msgr2=0x5602b1bc5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:36.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.131+0000 7f7ee259c700 1 --2- 172.21.15.114:0/539411000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bdb420 0x5602b1bc5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.131+0000 7f7ee259c700 1 -- 172.21.15.114:0/539411000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1a51e40 msgr2=0x5602b1bd0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:36.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.131+0000 7f7ee259c700 1 --2- 172.21.15.114:0/539411000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1a51e40 0x5602b1bd0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.131+0000 7f7ee259c700 1 -- 172.21.15.114:0/539411000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602b1bc7aa0 con 0x5602b1bd7070 2024-05-27T07:57:36.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.137+0000 7f7ee259c700 1 --2- 172.21.15.114:0/539411000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1bd7070 0x5602b1be6c60 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0098e0 tx=0x7f7ecc009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25cf0e8c624c828c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:36.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.142+0000 7f7ecbfff700 1 -- 172.21.15.114:0/539411000 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc0095d0 con 0x5602b1bd7070 2024-05-27T07:57:36.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.142+0000 7f7ecbfff700 1 -- 172.21.15.114:0/539411000 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ecc00f460 con 0x5602b1bd7070 2024-05-27T07:57:36.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.142+0000 7f7ecbfff700 1 -- 172.21.15.114:0/539411000 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc003da0 con 0x5602b1bd7070 2024-05-27T07:57:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.144+0000 7f7ef783cb80 1 -- 172.21.15.114:0/539411000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1bd7070 msgr2=0x5602b1be6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.144+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/539411000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1bd7070 0x5602b1be6c60 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0098e0 tx=0x7f7ecc009290 comp rx=0 tx=0).stop 2024-05-27T07:57:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.149+0000 7f7ef783cb80 1 -- 172.21.15.114:0/539411000 shutdown_connections 2024-05-27T07:57:36.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.149+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/539411000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bdb420 0x5602b1bc5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.149+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/539411000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1a51e40 0x5602b1bd0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.149+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/539411000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1bd7070 0x5602b1be6c60 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.149+0000 7f7ef783cb80 1 -- 172.21.15.114:0/539411000 >> 172.21.15.114:0/539411000 conn(0x5602b1a54770 msgr2=0x5602b1bc54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:36.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.153+0000 7f7ef783cb80 1 -- 172.21.15.114:0/539411000 shutdown_connections 2024-05-27T07:57:36.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.153+0000 7f7ef783cb80 1 -- 172.21.15.114:0/539411000 wait complete. 2024-05-27T07:57:36.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.153+0000 7f7ef783cb80 1 -- start start 2024-05-27T07:57:36.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.154+0000 7f7ef783cb80 1 Processor -- start 2024-05-27T07:57:36.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.159+0000 7f7ef783cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1a51e40 0x5602b1be3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:36.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.159+0000 7f7ef783cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bd7070 0x5602b1bf26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:36.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.160+0000 7f7ef783cb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bdb420 0x5602b1bf2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:36.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.160+0000 7f7ef783cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec400d4a0 con 0x5602b1a51e40 2024-05-27T07:57:36.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.160+0000 7f7ef783cb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec400ce50 con 0x5602b1bdb420 2024-05-27T07:57:36.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.160+0000 7f7ee259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bd7070 0x5602b1bf26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:36.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.160+0000 7f7ee259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bd7070 0x5602b1bf26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:35722/0 (socket says 172.21.15.114:35722) 2024-05-27T07:57:36.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.160+0000 7f7ee259c700 1 -- 172.21.15.114:0/502943125 learned_addr learned my addr 172.21.15.114:0/502943125 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:36.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.160+0000 7f7ef783cb80 1 -- 172.21.15.114:0/502943125 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5602b1b00cc0 con 0x5602b1bd7070 2024-05-27T07:57:36.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.160+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/502943125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bdb420 0x5602b1bf2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:36.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.160+0000 7f7ef783cb80 10 client.0 _ll_register_callbacks cb 0x5602b1a855b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:36.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.160+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/502943125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1a51e40 0x5602b1be3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:36.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.160+0000 7f7ee1d9b700 1 -- 172.21.15.114:0/502943125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bd7070 msgr2=0x5602b1bf26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:36.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.160+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/502943125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bd7070 0x5602b1bf26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.161+0000 7f7ee1d9b700 1 -- 172.21.15.114:0/502943125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1a51e40 msgr2=0x5602b1be3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:36.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.161+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/502943125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1a51e40 0x5602b1be3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.161+0000 7f7ee1d9b700 1 -- 172.21.15.114:0/502943125 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602b1bf3160 con 0x5602b1bdb420 2024-05-27T07:57:36.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.163+0000 7f7ee1d9b700 1 --2- 172.21.15.114:0/502943125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bdb420 0x5602b1bf2c20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7edc017010 tx=0x7f7edc000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b59bd9033f3d61b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:36.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.163+0000 7f7ec9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:57:36.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.163+0000 7f7ec9ffb700 1 -- 172.21.15.114:0/502943125 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc01d040 con 0x5602b1bdb420 2024-05-27T07:57:36.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.163+0000 7f7ef783cb80 1 -- 172.21.15.114:0/502943125 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ecc00f460 con 0x5602b1bdb420 2024-05-27T07:57:36.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.163+0000 7f7ec9ffb700 1 -- 172.21.15.114:0/502943125 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9538+0+0 (secure 0 0 0) 0x7f7edc016870 con 0x5602b1bdb420 2024-05-27T07:57:36.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.164+0000 7f7ec9ffb700 1 -- 172.21.15.114:0/502943125 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f7edc039050 con 0x5602b1bdb420 2024-05-27T07:57:36.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.167+0000 7f7ef783cb80 10 client.28814 fetch_fsmap learned FSMap version 100 2024-05-27T07:57:36.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.167+0000 7f7ef783cb80 1 -- 172.21.15.114:0/502943125 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x5602b1c55c30 con 0x5602b1bdb420 2024-05-27T07:57:36.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.170+0000 7f7ec9ffb700 1 -- 172.21.15.114:0/502943125 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7edc018bf0 con 0x5602b1bdb420 2024-05-27T07:57:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.170+0000 7f7ec9ffb700 1 -- 172.21.15.114:0/502943125 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 100) ==== 4095+0+0 (secure 0 0 0) 0x7f7edc023070 con 0x5602b1bdb420 2024-05-27T07:57:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.170+0000 7f7ef783cb80 10 client.28814 fetch_fsmap finished waiting for FSMap version 100 2024-05-27T07:57:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.170+0000 7f7ef783cb80 10 client.28814 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.170+0000 7f7ef783cb80 20 client.28814 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.170+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/502943125 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602b1c56200 0x5602b1c58620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.170+0000 7f7ef783cb80 4 client.28814 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:57:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.171+0000 7f7ef783cb80 1 -- 172.21.15.114:0/502943125 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f7ecc00f460 con 0x5602b1c56200 2024-05-27T07:57:36.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.175+0000 7f7ee259c700 1 --2- 172.21.15.114:0/502943125 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602b1c56200 0x5602b1c58620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:36.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.175+0000 7f7ee259c700 1 --2- 172.21.15.114:0/502943125 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602b1c56200 0x5602b1c58620 crc :-1 s=READY pgs=481 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27bcc1aa13aaa533 server_cookie=f1a4622a77c3e9c9 in_seq=0 out_seq=0 2024-05-27T07:57:36.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.176+0000 7f7ec9ffb700 10 client.28814 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:36.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.176+0000 7f7ec9ffb700 1 -- 172.21.15.114:0/502943125 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f7ecc00f460 con 0x5602b1c56200 2024-05-27T07:57:36.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.176+0000 7f7ec9ffb700 10 client.28814 handle_command_reply: tid=0 2024-05-27T07:57:36.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.176+0000 7f7ef783cb80 1 client.28814 shutdown 2024-05-27T07:57:36.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.177+0000 7f7ef783cb80 1 -- 172.21.15.114:0/502943125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bdb420 msgr2=0x5602b1bf2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:36.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.177+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/502943125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bdb420 0x5602b1bf2c20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7edc017010 tx=0x7f7edc000f40 comp rx=0 tx=0).stop 2024-05-27T07:57:36.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.178+0000 7f7ef783cb80 1 -- 172.21.15.114:0/502943125 shutdown_connections 2024-05-27T07:57:36.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.178+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/502943125 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5602b1c56200 0x5602b1c58620 crc :-1 s=READY pgs=481 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.178+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/502943125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602b1bdb420 0x5602b1bf2c20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.178+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/502943125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5602b1a51e40 0x5602b1be3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.178+0000 7f7ef783cb80 1 --2- 172.21.15.114:0/502943125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602b1bd7070 0x5602b1bf26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.178+0000 7f7ef783cb80 1 -- 172.21.15.114:0/502943125 >> 172.21.15.114:0/502943125 conn(0x5602b1a54770 msgr2=0x5602b1bc54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:36.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.178+0000 7f7ec9ffb700 0 client.28814 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:36.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.182+0000 7f7ef783cb80 1 -- 172.21.15.114:0/502943125 shutdown_connections 2024-05-27T07:57:36.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.182+0000 7f7ef783cb80 1 -- 172.21.15.114:0/502943125 wait complete. 2024-05-27T07:57:36.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.182+0000 7f7ef783cb80 20 client.28814 trim_cache size 0 max 16384 2024-05-27T07:57:36.183 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:57:36.183 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:57:36.183 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:57:36.183 INFO:teuthology.orchestra.run.smithi114.stdout: "c1f56efe-7efd-4160-b28e-0e974e2347c2": { 2024-05-27T07:57:36.183 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:57:36.183 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "c1f56efe-7efd-4160-b28e-0e974e2347c2", 2024-05-27T07:57:36.183 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:57:36.184 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:57:36.184 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:57:36.184 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:57:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.184+0000 7f7ee8e27700 1 -- 172.21.15.114:0/2300856744 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7ed00b2c00 msgr2=0x7f7ed00b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.184+0000 7f7ee8e27700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7ed00b2c00 0x7f7ed00b5110 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40df110 tx=0x7f7ed4004800 comp rx=0 tx=0).stop 2024-05-27T07:57:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.184+0000 7f7ee8e27700 1 -- 172.21.15.114:0/2300856744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7ee411d920 msgr2=0x7f7ee4115480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:36.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.184+0000 7f7ee8e27700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7ee411d920 0x7f7ee4115480 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400b450 tx=0x7f7ed400b810 comp rx=0 tx=0).stop 2024-05-27T07:57:36.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.188+0000 7f7ee8e27700 1 -- 172.21.15.114:0/2300856744 shutdown_connections 2024-05-27T07:57:36.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.188+0000 7f7ee8e27700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7ed00b2c00 0x7f7ed00b5110 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.188+0000 7f7ee8e27700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7ee411d920 0x7f7ee4115480 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.188+0000 7f7ee8e27700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ee410a920 0x7f7ee411bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.188+0000 7f7ee8e27700 1 --2- 172.21.15.114:0/2300856744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ee410b270 0x7f7ee4114e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.188+0000 7f7ee8e27700 1 -- 172.21.15.114:0/2300856744 >> 172.21.15.114:0/2300856744 conn(0x7f7ee406f4c0 msgr2=0x7f7ee4072780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:36.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.188+0000 7f7ee8e27700 1 -- 172.21.15.114:0/2300856744 shutdown_connections 2024-05-27T07:57:36.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:57:36.188+0000 7f7ee8e27700 1 -- 172.21.15.114:0/2300856744 wait complete. 2024-05-27T07:57:36.606 INFO:tasks.workunit.client.0.smithi114.stdout: 10 54981 12.21 MB/sec execute 108 sec latency 1222.491 ms 2024-05-27T07:57:36.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.631+0000 7f42ceffd640 1 -- 172.21.15.114:0/1809185455 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a69c23f360 con 0x55a69bf301b0 2024-05-27T07:57:36.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.631+0000 7f42ceffd640 10 client.19470 handle_command_reply: tid=0 2024-05-27T07:57:36.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.631+0000 7f42fce10740 1 client.19470 shutdown 2024-05-27T07:57:36.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42fce10740 1 -- 172.21.15.114:0/1809185455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 msgr2=0x55a69c2e0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:36.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42fce10740 1 --2- 172.21.15.114:0/1809185455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 0x55a69c2e0ac0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x55a69c2c7e60 tx=0x7f42d800fef0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42fce10740 1 -- 172.21.15.114:0/1809185455 shutdown_connections 2024-05-27T07:57:36.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42fce10740 1 --2- 172.21.15.114:0/1809185455 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55a69bf301b0 0x55a69c358180 crc :-1 s=READY pgs=479 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42fce10740 1 --2- 172.21.15.114:0/1809185455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a69bf0f4d0 0x55a69c2e1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42fce10740 1 --2- 172.21.15.114:0/1809185455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a69bf072f0 0x55a69c2e0ac0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42fce10740 1 --2- 172.21.15.114:0/1809185455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a69beeed80 0x55a69c2e0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42ceffd640 0 client.19470 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42fce10740 1 -- 172.21.15.114:0/1809185455 >> 172.21.15.114:0/1809185455 conn(0x55a69bf24300 msgr2=0x55a69c2d1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42fce10740 1 -- 172.21.15.114:0/1809185455 shutdown_connections 2024-05-27T07:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42fce10740 1 -- 172.21.15.114:0/1809185455 wait complete. 2024-05-27T07:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42fce10740 20 client.19470 trim_cache size 0 max 16384 2024-05-27T07:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:57:36.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42f5e84640 1 -- 172.21.15.114:0/3176415088 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f42dc0b29d0 msgr2=0x7f42dc0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42f5e84640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f42dc0b29d0 0x7f42dc0b4ee0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f42f0103880 tx=0x7f42e0005f90 comp rx=0 tx=0).stop 2024-05-27T07:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42f5e84640 1 -- 172.21.15.114:0/3176415088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42f0109040 msgr2=0x7f42f019a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.632+0000 7f42f5e84640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42f0109040 0x7f42f019a6e0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f42e8005cc0 tx=0x7f42e8031700 comp rx=0 tx=0).stop 2024-05-27T07:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.633+0000 7f42f5e84640 1 -- 172.21.15.114:0/3176415088 shutdown_connections 2024-05-27T07:57:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.633+0000 7f42f5e84640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f42dc0b29d0 0x7f42dc0b4ee0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.633+0000 7f42f5e84640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42f0109040 0x7f42f019a6e0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.633+0000 7f42f5e84640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f0108ba0 0x7f42f01a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.633+0000 7f42f5e84640 1 --2- 172.21.15.114:0/3176415088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f01032f0 0x7f42f01a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.633+0000 7f42f5e84640 1 -- 172.21.15.114:0/3176415088 >> 172.21.15.114:0/3176415088 conn(0x7f42f0078270 msgr2=0x7f42f006b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.633+0000 7f42f5e84640 1 -- 172.21.15.114:0/3176415088 shutdown_connections 2024-05-27T07:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:36.633+0000 7f42f5e84640 1 -- 172.21.15.114:0/3176415088 wait complete. 2024-05-27T07:57:37.178 INFO:tasks.cephfs.filesystem:scrub status for tag:c1f56efe-7efd-4160-b28e-0e974e2347c2 - {'path': '/', 'tag': 'c1f56efe-7efd-4160-b28e-0e974e2347c2', 'options': 'recursive,force'} 2024-05-27T07:57:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 55386 12.26 MB/sec execute 109 sec latency 226.199 ms 2024-05-27T07:57:38.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:37 smithi163 ceph-mon[36451]: pgmap v613: 129 pgs: 129 active+clean; 365 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 522 op/s 2024-05-27T07:57:38.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:37 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:57:38.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:37 smithi186 ceph-mon[36647]: pgmap v613: 129 pgs: 129 active+clean; 365 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 522 op/s 2024-05-27T07:57:38.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:37 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:57:38.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:37 smithi114 ceph-mon[32038]: pgmap v613: 129 pgs: 129 active+clean; 365 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 522 op/s 2024-05-27T07:57:38.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:37 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:57:38.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 55814 12.47 MB/sec execute 110 sec latency 165.550 ms 2024-05-27T07:57:39.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 56271 12.70 MB/sec execute 111 sec latency 92.212 ms 2024-05-27T07:57:40.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:39 smithi163 ceph-mon[36451]: pgmap v614: 129 pgs: 129 active+clean; 363 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 620 op/s 2024-05-27T07:57:40.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:39 smithi186 ceph-mon[36647]: pgmap v614: 129 pgs: 129 active+clean; 363 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 620 op/s 2024-05-27T07:57:40.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:39 smithi114 ceph-mon[32038]: pgmap v614: 129 pgs: 129 active+clean; 363 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 620 op/s 2024-05-27T07:57:40.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 56546 12.70 MB/sec execute 112 sec latency 118.245 ms 2024-05-27T07:57:41.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 56710 12.63 MB/sec execute 113 sec latency 220.289 ms 2024-05-27T07:57:42.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:41 smithi163 ceph-mon[36451]: pgmap v615: 129 pgs: 129 active+clean; 359 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 581 op/s 2024-05-27T07:57:42.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:41 smithi186 ceph-mon[36647]: pgmap v615: 129 pgs: 129 active+clean; 359 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 581 op/s 2024-05-27T07:57:42.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:41 smithi114 ceph-mon[32038]: pgmap v615: 129 pgs: 129 active+clean; 359 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 581 op/s 2024-05-27T07:57:42.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 56890 12.59 MB/sec execute 114 sec latency 198.238 ms 2024-05-27T07:57:43.327 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:43 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:57:43.327 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:43 smithi114 systemd[1]: Stopped Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:57:43.327 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:43 smithi114 systemd[1]: Starting Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:57:43.328 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:43 smithi114 podman[130073]: 2024-05-27 07:57:43.197834547 +0000 UTC m=+0.018179722 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:57:43.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 57135 12.54 MB/sec execute 115 sec latency 185.266 ms 2024-05-27T07:57:43.722 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:43 smithi114 podman[130073]: 2024-05-27 07:57:43.326822283 +0000 UTC m=+0.147167444 container create c8ee4cca6dcd10c24fa8690fcf55d848be184b751704c60e96044e1e10667ee3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:57:44.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:43 smithi114 ceph-mon[32038]: pgmap v616: 129 pgs: 129 active+clean; 353 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.8 MiB/s wr, 596 op/s 2024-05-27T07:57:44.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:43 smithi114 ceph-mon[32038]: daemon mds.a restarted 2024-05-27T07:57:44.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:43 smithi114 podman[130073]: 2024-05-27 07:57:43.722077958 +0000 UTC m=+0.542423124 container init c8ee4cca6dcd10c24fa8690fcf55d848be184b751704c60e96044e1e10667ee3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-27T07:57:44.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:43 smithi114 podman[130073]: 2024-05-27 07:57:43.726223514 +0000 UTC m=+0.546568678 container start c8ee4cca6dcd10c24fa8690fcf55d848be184b751704c60e96044e1e10667ee3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T07:57:44.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:43 smithi114 ceph-mds[130091]: -- 172.21.15.114:0/1688588226 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5615c4eb5a40 con 0x5615c4efda80 2024-05-27T07:57:44.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:43 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: starting mds.a at 2024-05-27T07:57:44.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:43 smithi114 bash[130073]: c8ee4cca6dcd10c24fa8690fcf55d848be184b751704c60e96044e1e10667ee3 2024-05-27T07:57:44.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:57:43 smithi114 systemd[1]: Started Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:57:44.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:43 smithi163 ceph-mon[36451]: pgmap v616: 129 pgs: 129 active+clean; 353 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.8 MiB/s wr, 596 op/s 2024-05-27T07:57:44.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:43 smithi163 ceph-mon[36451]: daemon mds.a restarted 2024-05-27T07:57:44.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:43 smithi186 ceph-mon[36647]: pgmap v616: 129 pgs: 129 active+clean; 353 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.8 MiB/s wr, 596 op/s 2024-05-27T07:57:44.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:43 smithi186 ceph-mon[36647]: daemon mds.a restarted 2024-05-27T07:57:44.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 57458 12.53 MB/sec execute 116 sec latency 224.825 ms 2024-05-27T07:57:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:44 smithi163 ceph-mon[36451]: osdmap e154: 12 total, 12 up, 12 in 2024-05-27T07:57:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:44 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6840/584614487,v1:172.21.15.114:6841/584614487] up:boot 2024-05-27T07:57:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:44 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:57:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:57:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:44 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:57:45.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:44 smithi186 ceph-mon[36647]: osdmap e154: 12 total, 12 up, 12 in 2024-05-27T07:57:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:44 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6840/584614487,v1:172.21.15.114:6841/584614487] up:boot 2024-05-27T07:57:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:44 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:57:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:57:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:44 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:57:45.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:44 smithi114 ceph-mon[32038]: osdmap e154: 12 total, 12 up, 12 in 2024-05-27T07:57:45.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:44 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6840/584614487,v1:172.21.15.114:6841/584614487] up:boot 2024-05-27T07:57:45.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:44 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:57:45.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:57:45.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:44 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:57:45.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 57927 12.70 MB/sec execute 117 sec latency 136.327 ms 2024-05-27T07:57:46.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:45 smithi114 ceph-mon[32038]: pgmap v618: 129 pgs: 129 active+clean; 354 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.4 MiB/s wr, 571 op/s 2024-05-27T07:57:46.320 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:45 smithi186 ceph-mon[36647]: pgmap v618: 129 pgs: 129 active+clean; 354 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.4 MiB/s wr, 571 op/s 2024-05-27T07:57:46.320 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:45 smithi163 ceph-mon[36451]: pgmap v618: 129 pgs: 129 active+clean; 354 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.4 MiB/s wr, 571 op/s 2024-05-27T07:57:46.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 58388 12.72 MB/sec execute 118 sec latency 339.961 ms 2024-05-27T07:57:47.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 58776 12.77 MB/sec execute 119 sec latency 188.261 ms 2024-05-27T07:57:48.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:47 smithi163 ceph-mon[36451]: pgmap v619: 129 pgs: 129 active+clean; 367 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 370 op/s 2024-05-27T07:57:48.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:47 smithi186 ceph-mon[36647]: pgmap v619: 129 pgs: 129 active+clean; 367 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 370 op/s 2024-05-27T07:57:48.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:47 smithi114 ceph-mon[32038]: pgmap v619: 129 pgs: 129 active+clean; 367 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 370 op/s 2024-05-27T07:57:48.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 59236 12.97 MB/sec execute 120 sec latency 208.565 ms 2024-05-27T07:57:49.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 59623 13.13 MB/sec execute 121 sec latency 187.929 ms 2024-05-27T07:57:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.750+0000 7f4e97889640 1 -- 172.21.15.114:0/579944615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e90109030 msgr2=0x7f4e90102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.750+0000 7f4e97889640 1 --2- 172.21.15.114:0/579944615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e90109030 0x7f4e90102c70 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f4e80009900 tx=0x7f4e8002a410 comp rx=0 tx=0).stop 2024-05-27T07:57:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.750+0000 7f4e97889640 1 -- 172.21.15.114:0/579944615 shutdown_connections 2024-05-27T07:57:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.750+0000 7f4e97889640 1 --2- 172.21.15.114:0/579944615 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e901032e0 0x7f4e9010f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.750+0000 7f4e97889640 1 --2- 172.21.15.114:0/579944615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e90109030 0x7f4e90102c70 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.750+0000 7f4e97889640 1 --2- 172.21.15.114:0/579944615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e90108b90 0x7f4e90102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.750+0000 7f4e97889640 1 -- 172.21.15.114:0/579944615 >> 172.21.15.114:0/579944615 conn(0x7f4e900782d0 msgr2=0x7f4e90078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.750+0000 7f4e97889640 1 -- 172.21.15.114:0/579944615 shutdown_connections 2024-05-27T07:57:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.750+0000 7f4e97889640 1 -- 172.21.15.114:0/579944615 wait complete. 2024-05-27T07:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.751+0000 7f4e97889640 1 Processor -- start 2024-05-27T07:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.751+0000 7f4e97889640 1 -- start start 2024-05-27T07:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.751+0000 7f4e97889640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e901032e0 0x7f4e9019b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.751+0000 7f4e97889640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e90108b90 0x7f4e9019bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.751+0000 7f4e97889640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e90109030 0x7f4e9019c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.751+0000 7f4e97889640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4e90112d80 con 0x7f4e90109030 2024-05-27T07:57:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.751+0000 7f4e97889640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4e90112c00 con 0x7f4e901032e0 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.751+0000 7f4e97889640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4e90112f00 con 0x7f4e90108b90 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.751+0000 7f4e95e26640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e90109030 0x7f4e9019c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.751+0000 7f4e95e26640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e90109030 0x7f4e9019c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44432/0 (socket says 172.21.15.114:44432) 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.751+0000 7f4e95625640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e901032e0 0x7f4e9019b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.751+0000 7f4e95e26640 1 -- 172.21.15.114:0/3971728735 learned_addr learned my addr 172.21.15.114:0/3971728735 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.752+0000 7f4e94e24640 1 --2- 172.21.15.114:0/3971728735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e90108b90 0x7f4e9019bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.752+0000 7f4e95e26640 1 -- 172.21.15.114:0/3971728735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e90108b90 msgr2=0x7f4e9019bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.752+0000 7f4e95e26640 1 --2- 172.21.15.114:0/3971728735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e90108b90 0x7f4e9019bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.752+0000 7f4e95e26640 1 -- 172.21.15.114:0/3971728735 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e901032e0 msgr2=0x7f4e9019b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.752+0000 7f4e95e26640 1 --2- 172.21.15.114:0/3971728735 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e901032e0 0x7f4e9019b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.752+0000 7f4e95e26640 1 -- 172.21.15.114:0/3971728735 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e901a6b10 con 0x7f4e90109030 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.752+0000 7f4e95e26640 1 --2- 172.21.15.114:0/3971728735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e90109030 0x7f4e9019c280 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f4e88009fd0 tx=0x7f4e8800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.752+0000 7f4e867fc640 1 -- 172.21.15.114:0/3971728735 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88017070 con 0x7f4e90109030 2024-05-27T07:57:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.752+0000 7f4e97889640 1 -- 172.21.15.114:0/3971728735 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e901a6e00 con 0x7f4e90109030 2024-05-27T07:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.752+0000 7f4e867fc640 1 -- 172.21.15.114:0/3971728735 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e88003e20 con 0x7f4e90109030 2024-05-27T07:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.752+0000 7f4e97889640 1 -- 172.21.15.114:0/3971728735 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e901a7240 con 0x7f4e90109030 2024-05-27T07:57:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.752+0000 7f4e867fc640 1 -- 172.21.15.114:0/3971728735 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88020060 con 0x7f4e90109030 2024-05-27T07:57:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.755+0000 7f4e9e81a740 1 Processor -- start 2024-05-27T07:57:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.755+0000 7f4e867fc640 1 -- 172.21.15.114:0/3971728735 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f4e88003970 con 0x7f4e90109030 2024-05-27T07:57:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.755+0000 7f4e9e81a740 1 -- start start 2024-05-27T07:57:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e9e81a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee0b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e9e81a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edce62f0 0x5622ee0b0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e9e81a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edcee4d0 0x5622ee0b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e9e81a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5622ee022c30 con 0x5622edcee4d0 2024-05-27T07:57:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e9e81a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5622ee0127d0 con 0x5622edd0f1b0 2024-05-27T07:57:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e9e81a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5622ee016810 con 0x5622edce62f0 2024-05-27T07:57:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e94e24640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edcee4d0 0x5622ee0b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e95625640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edce62f0 0x5622ee0b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e94e24640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee0b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e95625640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edce62f0 0x5622ee0b0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:37406/0 (socket says 172.21.15.114:37406) 2024-05-27T07:57:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e94e24640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee0b03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:34406/0 (socket says 172.21.15.114:34406) 2024-05-27T07:57:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e95625640 1 -- 172.21.15.114:0/740922378 learned_addr learned my addr 172.21.15.114:0/740922378 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e94e24640 1 -- 172.21.15.114:0/740922378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edce62f0 msgr2=0x5622ee0b0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e94e24640 1 --2- 172.21.15.114:0/740922378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edce62f0 0x5622ee0b0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e94e24640 1 -- 172.21.15.114:0/740922378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edcee4d0 msgr2=0x5622ee0b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e94e24640 1 --2- 172.21.15.114:0/740922378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edcee4d0 0x5622ee0b0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e94e24640 1 -- 172.21.15.114:0/740922378 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622edd27210 con 0x5622edd0f1b0 2024-05-27T07:57:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e867fc640 1 --2- 172.21.15.114:0/3971728735 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4e7c0b2950 0x7f4e7c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.756+0000 7f4e95625640 1 --2- 172.21.15.114:0/3971728735 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4e7c0b2950 0x7f4e7c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.757+0000 7f4e94e24640 1 --2- 172.21.15.114:0/740922378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee0b03d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4e8002a930 tx=0x7f4e80031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f928d624eb66a768 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.757+0000 7f4e867fc640 1 -- 172.21.15.114:0/3971728735 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 9466+0+0 (secure 0 0 0) 0x7f4e880e2680 con 0x7f4e90109030 2024-05-27T07:57:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.757+0000 7f4e7b7fe640 1 -- 172.21.15.114:0/740922378 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e80035070 con 0x5622edd0f1b0 2024-05-27T07:57:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.757+0000 7f4e7b7fe640 1 -- 172.21.15.114:0/740922378 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e80004030 con 0x5622edd0f1b0 2024-05-27T07:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.757+0000 7f4e7b7fe640 1 -- 172.21.15.114:0/740922378 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e800052a0 con 0x5622edd0f1b0 2024-05-27T07:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.758+0000 7f4e9e81a740 1 -- 172.21.15.114:0/740922378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 msgr2=0x5622ee0b03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.758+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/740922378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee0b03d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4e8002a930 tx=0x7f4e80031400 comp rx=0 tx=0).stop 2024-05-27T07:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.758+0000 7f4e95625640 1 --2- 172.21.15.114:0/3971728735 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4e7c0b2950 0x7f4e7c0b4e60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x5622ee0b1780 tx=0x7f4e8c005f90 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.758+0000 7f4e9e81a740 1 -- 172.21.15.114:0/740922378 shutdown_connections 2024-05-27T07:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.758+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/740922378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edcee4d0 0x5622ee0b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.758+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/740922378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edce62f0 0x5622ee0b0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.758+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/740922378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee0b03d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.758+0000 7f4e9e81a740 1 -- 172.21.15.114:0/740922378 >> 172.21.15.114:0/740922378 conn(0x5622edd03300 msgr2=0x5622ee0acb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.758+0000 7f4e9e81a740 1 -- 172.21.15.114:0/740922378 shutdown_connections 2024-05-27T07:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.758+0000 7f4e9e81a740 1 -- 172.21.15.114:0/740922378 wait complete. 2024-05-27T07:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.758+0000 7f4e9e81a740 1 -- start start 2024-05-27T07:57:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.758+0000 7f4e9e81a740 1 Processor -- start 2024-05-27T07:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e9e81a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edce62f0 0x5622ee0bc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e9e81a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 0x5622ee0bcae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e9e81a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edd0f1b0 0x5622ee0ba9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e9e81a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5622edcf8790 con 0x5622edd0f1b0 2024-05-27T07:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e9e81a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5622edfce890 con 0x5622edce62f0 2024-05-27T07:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e9e81a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5622ee0bd4a0 con 0x5622edcee4d0 2024-05-27T07:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e9e81a740 10 client.0 _ll_register_callbacks cb 0x5622edd26a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e95625640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edce62f0 0x5622ee0bc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e94e24640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 0x5622ee0bcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e95e26640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edd0f1b0 0x5622ee0ba9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e94e24640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 0x5622ee0bcae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:37414/0 (socket says 172.21.15.114:37414) 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e94e24640 1 -- 172.21.15.114:0/799236694 learned_addr learned my addr 172.21.15.114:0/799236694 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e95e26640 1 -- 172.21.15.114:0/799236694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 msgr2=0x5622ee0bcae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e95e26640 1 --2- 172.21.15.114:0/799236694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 0x5622ee0bcae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e95e26640 1 -- 172.21.15.114:0/799236694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edce62f0 msgr2=0x5622ee0bc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e95e26640 1 --2- 172.21.15.114:0/799236694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edce62f0 0x5622ee0bc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e95e26640 1 -- 172.21.15.114:0/799236694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622eddf3620 con 0x5622edd0f1b0 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e95625640 1 --2- 172.21.15.114:0/799236694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edce62f0 0x5622ee0bc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e95e26640 1 --2- 172.21.15.114:0/799236694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edd0f1b0 0x5622ee0ba9d0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f4e880dfa20 tx=0x7f4e880a9540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f9660d4353d567 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.759+0000 7f4e78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e78ff9640 1 -- 172.21.15.114:0/799236694 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e880e1070 con 0x5622edd0f1b0 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e78ff9640 1 -- 172.21.15.114:0/799236694 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(154..154 src has 1..154) ==== 9466+0+0 (secure 0 0 0) 0x7f4e880a58f0 con 0x5622edd0f1b0 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e9e81a740 1 -- 172.21.15.114:0/799236694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e80004030 con 0x5622edd0f1b0 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e78ff9640 1 -- 172.21.15.114:0/799236694 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f4e880ad960 con 0x5622edd0f1b0 2024-05-27T07:57:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e9e81a740 10 client.19506 fetch_fsmap learned FSMap version 102 2024-05-27T07:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e9e81a740 1 -- 172.21.15.114:0/799236694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x5622ee0cdd30 con 0x5622edd0f1b0 2024-05-27T07:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e78ff9640 1 -- 172.21.15.114:0/799236694 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e80004030 con 0x5622edd0f1b0 2024-05-27T07:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e78ff9640 1 -- 172.21.15.114:0/799236694 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 102) ==== 4095+0+0 (secure 0 0 0) 0x7f4e880b0ce0 con 0x5622edd0f1b0 2024-05-27T07:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e9e81a740 10 client.19506 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e9e81a740 10 client.19506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e9e81a740 20 client.19506 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/799236694 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622edccdd80 0x5622ee0d1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e9e81a740 4 client.19506 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e9e81a740 1 -- 172.21.15.114:0/799236694 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622ee024b80 con 0x5622edccdd80 2024-05-27T07:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.760+0000 7f4e95625640 1 --2- 172.21.15.114:0/799236694 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622edccdd80 0x5622ee0d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.761+0000 7f4e78ff9640 10 client.19506 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.761+0000 7f4e95625640 1 --2- 172.21.15.114:0/799236694 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622edccdd80 0x5622ee0d1660 crc :-1 s=READY pgs=482 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aee140e2d153ab31 server_cookie=ed3583a0f3d734fe in_seq=0 out_seq=0 2024-05-27T07:57:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e78ff9640 1 -- 172.21.15.114:0/799236694 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5622ee024b80 con 0x5622edccdd80 2024-05-27T07:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e78ff9640 10 client.19506 handle_command_reply: tid=0 2024-05-27T07:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e9e81a740 1 client.19506 shutdown 2024-05-27T07:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e9e81a740 1 -- 172.21.15.114:0/799236694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edd0f1b0 msgr2=0x5622ee0ba9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/799236694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edd0f1b0 0x5622ee0ba9d0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f4e880dfa20 tx=0x7f4e880a9540 comp rx=0 tx=0).stop 2024-05-27T07:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e9e81a740 1 -- 172.21.15.114:0/799236694 shutdown_connections 2024-05-27T07:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/799236694 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622edccdd80 0x5622ee0d1660 crc :-1 s=READY pgs=482 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/799236694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edd0f1b0 0x5622ee0ba9d0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/799236694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 0x5622ee0bcae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/799236694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edce62f0 0x5622ee0bc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e78ff9640 0 client.19506 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e9e81a740 1 -- 172.21.15.114:0/799236694 >> 172.21.15.114:0/799236694 conn(0x5622edd03300 msgr2=0x5622ee0b01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e9e81a740 1 -- 172.21.15.114:0/799236694 shutdown_connections 2024-05-27T07:57:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.766+0000 7f4e9e81a740 1 -- 172.21.15.114:0/799236694 wait complete. 2024-05-27T07:57:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.767+0000 7f4e9e81a740 20 client.19506 trim_cache size 0 max 16384 2024-05-27T07:57:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.782+0000 7f4e9e81a740 1 Processor -- start 2024-05-27T07:57:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.782+0000 7f4e9e81a740 1 -- start start 2024-05-27T07:57:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.782+0000 7f4e9e81a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 0x5622ee0c2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.782+0000 7f4e9e81a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 0x5622ee0c2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.782+0000 7f4e9e81a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee0b4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.782+0000 7f4e9e81a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4e901aec10 con 0x5622edce62f0 2024-05-27T07:57:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.782+0000 7f4e9e81a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4e901af2e0 con 0x5622edd0f1b0 2024-05-27T07:57:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.782+0000 7f4e9e81a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4e901adbb0 con 0x5622edcee4d0 2024-05-27T07:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e95625640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 0x5622ee0c2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e95e26640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee0b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e95625640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 0x5622ee0c2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44458/0 (socket says 172.21.15.114:44458) 2024-05-27T07:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e95625640 1 -- 172.21.15.114:0/1985043386 learned_addr learned my addr 172.21.15.114:0/1985043386 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e94e24640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 0x5622ee0c2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e95625640 1 -- 172.21.15.114:0/1985043386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 msgr2=0x5622ee0c2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e95625640 1 --2- 172.21.15.114:0/1985043386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 0x5622ee0c2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e95625640 1 -- 172.21.15.114:0/1985043386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 msgr2=0x5622ee0b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e95625640 1 --2- 172.21.15.114:0/1985043386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee0b4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e95625640 1 -- 172.21.15.114:0/1985043386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622ee0b17b0 con 0x5622edce62f0 2024-05-27T07:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e95e26640 1 --2- 172.21.15.114:0/1985043386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee0b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:57:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e95625640 1 --2- 172.21.15.114:0/1985043386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 0x5622ee0c2730 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x5622ee0adb10 tx=0x7f4e8c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3735a24ce78f0d30 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e7affd640 1 -- 172.21.15.114:0/1985043386 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c030070 con 0x5622edce62f0 2024-05-27T07:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e7affd640 1 -- 172.21.15.114:0/1985043386 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e8c003e50 con 0x5622edce62f0 2024-05-27T07:57:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.783+0000 7f4e7affd640 1 -- 172.21.15.114:0/1985043386 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c01f960 con 0x5622edce62f0 2024-05-27T07:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.785+0000 7f4e9e81a740 1 -- 172.21.15.114:0/1985043386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 msgr2=0x5622ee0c2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.785+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/1985043386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 0x5622ee0c2730 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x5622ee0adb10 tx=0x7f4e8c01b000 comp rx=0 tx=0).stop 2024-05-27T07:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.785+0000 7f4e9e81a740 1 -- 172.21.15.114:0/1985043386 shutdown_connections 2024-05-27T07:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.785+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/1985043386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee0b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.785+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/1985043386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 0x5622ee0c2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.785+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/1985043386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 0x5622ee0c2730 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.785+0000 7f4e9e81a740 1 -- 172.21.15.114:0/1985043386 >> 172.21.15.114:0/1985043386 conn(0x5622edd03300 msgr2=0x5622ee0b92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.785+0000 7f4e9e81a740 1 -- 172.21.15.114:0/1985043386 shutdown_connections 2024-05-27T07:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.785+0000 7f4e9e81a740 1 -- 172.21.15.114:0/1985043386 wait complete. 2024-05-27T07:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.786+0000 7f4e9e81a740 1 -- start start 2024-05-27T07:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.786+0000 7f4e9e81a740 1 Processor -- start 2024-05-27T07:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.786+0000 7f4e9e81a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 0x5622ee0c25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.786+0000 7f4e9e81a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 0x5622ee0c2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.786+0000 7f4e9e81a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee131320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.786+0000 7f4e9e81a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4e901ad4e0 con 0x5622edce62f0 2024-05-27T07:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.786+0000 7f4e9e81a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4e901ace70 con 0x5622edd0f1b0 2024-05-27T07:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.786+0000 7f4e9e81a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5622ee024b80 con 0x5622edcee4d0 2024-05-27T07:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e9e81a740 10 client.0 _ll_register_callbacks cb 0x7f4e88004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e95625640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 0x5622ee0c25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e94e24640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 0x5622ee0c2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e95625640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 0x5622ee0c25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44470/0 (socket says 172.21.15.114:44470) 2024-05-27T07:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e95625640 1 -- 172.21.15.114:0/2110619398 learned_addr learned my addr 172.21.15.114:0/2110619398 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e95e26640 1 --2- 172.21.15.114:0/2110619398 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee131320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e95625640 1 -- 172.21.15.114:0/2110619398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 msgr2=0x5622ee0c2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e95625640 1 --2- 172.21.15.114:0/2110619398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 0x5622ee0c2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e95625640 1 -- 172.21.15.114:0/2110619398 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 msgr2=0x5622ee131320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e95625640 1 --2- 172.21.15.114:0/2110619398 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee131320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e95625640 1 -- 172.21.15.114:0/2110619398 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622ee0a8340 con 0x5622edce62f0 2024-05-27T07:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e95625640 1 --2- 172.21.15.114:0/2110619398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 0x5622ee0c25e0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c019040 tx=0x7f4e8c005b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732c9ab11e0f088c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e78ff9640 1 -- 172.21.15.114:0/2110619398 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c030040 con 0x5622edce62f0 2024-05-27T07:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e78ff9640 1 -- 172.21.15.114:0/2110619398 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(154..154 src has 1..154) ==== 9466+0+0 (secure 0 0 0) 0x7f4e8c005380 con 0x5622edce62f0 2024-05-27T07:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.787+0000 7f4e9e81a740 1 -- 172.21.15.114:0/2110619398 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e8c003e50 con 0x5622edce62f0 2024-05-27T07:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.788+0000 7f4e78ff9640 1 -- 172.21.15.114:0/2110619398 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f4e8c044050 con 0x5622edce62f0 2024-05-27T07:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.788+0000 7f4e9e81a740 10 client.19512 fetch_fsmap learned FSMap version 102 2024-05-27T07:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.788+0000 7f4e9e81a740 1 -- 172.21.15.114:0/2110619398 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x5622ee139c40 con 0x5622edce62f0 2024-05-27T07:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.788+0000 7f4e78ff9640 1 -- 172.21.15.114:0/2110619398 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e8c003e50 con 0x5622edce62f0 2024-05-27T07:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.788+0000 7f4e78ff9640 1 -- 172.21.15.114:0/2110619398 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 102) ==== 4095+0+0 (secure 0 0 0) 0x7f4e8c033030 con 0x5622edce62f0 2024-05-27T07:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.788+0000 7f4e9e81a740 10 client.19512 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.788+0000 7f4e9e81a740 10 client.19512 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.788+0000 7f4e9e81a740 20 client.19512 populate_metadata read hostname 'smithi114' 2024-05-27T07:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.788+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/2110619398 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622edca31c0 0x5622ee13c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.788+0000 7f4e9e81a740 4 client.19512 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.788+0000 7f4e9e81a740 1 -- 172.21.15.114:0/2110619398 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5622ee01e360 con 0x5622edca31c0 2024-05-27T07:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.788+0000 7f4e94e24640 1 --2- 172.21.15.114:0/2110619398 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622edca31c0 0x5622ee13c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.788+0000 7f4e78ff9640 10 client.19512 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:49.789+0000 7f4e94e24640 1 --2- 172.21.15.114:0/2110619398 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622edca31c0 0x5622ee13c360 crc :-1 s=READY pgs=483 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fc1f587aeb75c9b server_cookie=6127bb4b6a5678e in_seq=0 out_seq=0 2024-05-27T07:57:50.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:49 smithi163 ceph-mon[36451]: pgmap v620: 129 pgs: 129 active+clean; 374 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 261 op/s 2024-05-27T07:57:50.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:49 smithi186 ceph-mon[36647]: pgmap v620: 129 pgs: 129 active+clean; 374 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 261 op/s 2024-05-27T07:57:50.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:49 smithi114 ceph-mon[32038]: pgmap v620: 129 pgs: 129 active+clean; 374 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 261 op/s 2024-05-27T07:57:50.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 59725 13.07 MB/sec execute 122 sec latency 622.675 ms 2024-05-27T07:57:51.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:50 smithi163 ceph-mon[36451]: pgmap v621: 129 pgs: 129 active+clean; 374 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 259 op/s 2024-05-27T07:57:51.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:50 smithi186 ceph-mon[36647]: pgmap v621: 129 pgs: 129 active+clean; 374 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 259 op/s 2024-05-27T07:57:51.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:50 smithi114 ceph-mon[32038]: pgmap v621: 129 pgs: 129 active+clean; 374 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 259 op/s 2024-05-27T07:57:51.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.406+0000 7f4e78ff9640 1 -- 172.21.15.114:0/2110619398 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5622ee01e360 con 0x5622edca31c0 2024-05-27T07:57:51.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.406+0000 7f4e78ff9640 10 client.19512 handle_command_reply: tid=0 2024-05-27T07:57:51.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.407+0000 7f4e9e81a740 1 client.19512 shutdown 2024-05-27T07:57:51.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.407+0000 7f4e9e81a740 1 -- 172.21.15.114:0/2110619398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 msgr2=0x5622ee0c25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:51.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.407+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/2110619398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 0x5622ee0c25e0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c019040 tx=0x7f4e8c005b80 comp rx=0 tx=0).stop 2024-05-27T07:57:51.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.407+0000 7f4e9e81a740 1 -- 172.21.15.114:0/2110619398 shutdown_connections 2024-05-27T07:57:51.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.407+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/2110619398 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5622edca31c0 0x5622ee13c360 crc :-1 s=READY pgs=483 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:51.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.407+0000 7f4e78ff9640 0 client.19512 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:57:51.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.407+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/2110619398 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5622edd0f1b0 0x5622ee131320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:51.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.407+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/2110619398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5622edcee4d0 0x5622ee0c2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:51.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.407+0000 7f4e9e81a740 1 --2- 172.21.15.114:0/2110619398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5622edce62f0 0x5622ee0c25e0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:51.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.407+0000 7f4e9e81a740 1 -- 172.21.15.114:0/2110619398 >> 172.21.15.114:0/2110619398 conn(0x5622edd03300 msgr2=0x5622ee0b9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:51.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.408+0000 7f4e9e81a740 1 -- 172.21.15.114:0/2110619398 shutdown_connections 2024-05-27T07:57:51.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.408+0000 7f4e9e81a740 1 -- 172.21.15.114:0/2110619398 wait complete. 2024-05-27T07:57:51.411 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.408+0000 7f4e9e81a740 20 client.19512 trim_cache size 0 max 16384 2024-05-27T07:57:51.411 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:57:51.411 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:57:51.411 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:57:51.411 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.409+0000 7f4e97889640 1 -- 172.21.15.114:0/3971728735 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4e7c0b2950 msgr2=0x7f4e7c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.409+0000 7f4e97889640 1 --2- 172.21.15.114:0/3971728735 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4e7c0b2950 0x7f4e7c0b4e60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x5622ee0b1780 tx=0x7f4e8c005f90 comp rx=0 tx=0).stop 2024-05-27T07:57:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.409+0000 7f4e97889640 1 -- 172.21.15.114:0/3971728735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e90109030 msgr2=0x7f4e9019c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:57:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.409+0000 7f4e97889640 1 --2- 172.21.15.114:0/3971728735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e90109030 0x7f4e9019c280 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f4e88009fd0 tx=0x7f4e8800b260 comp rx=0 tx=0).stop 2024-05-27T07:57:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.409+0000 7f4e97889640 1 -- 172.21.15.114:0/3971728735 shutdown_connections 2024-05-27T07:57:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.409+0000 7f4e97889640 1 --2- 172.21.15.114:0/3971728735 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4e7c0b2950 0x7f4e7c0b4e60 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.409+0000 7f4e97889640 1 --2- 172.21.15.114:0/3971728735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e90109030 0x7f4e9019c280 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.409+0000 7f4e97889640 1 --2- 172.21.15.114:0/3971728735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e90108b90 0x7f4e9019bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.409+0000 7f4e97889640 1 --2- 172.21.15.114:0/3971728735 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e901032e0 0x7f4e9019b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:57:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.409+0000 7f4e97889640 1 -- 172.21.15.114:0/3971728735 >> 172.21.15.114:0/3971728735 conn(0x7f4e900782d0 msgr2=0x7f4e901012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:57:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.409+0000 7f4e97889640 1 -- 172.21.15.114:0/3971728735 shutdown_connections 2024-05-27T07:57:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:57:51.409+0000 7f4e97889640 1 -- 172.21.15.114:0/3971728735 wait complete. 2024-05-27T07:57:51.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 59811 12.99 MB/sec execute 123 sec latency 1082.603 ms 2024-05-27T07:57:52.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 60045 12.95 MB/sec execute 124 sec latency 244.805 ms 2024-05-27T07:57:53.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 60283 12.93 MB/sec execute 125 sec latency 202.409 ms 2024-05-27T07:57:54.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:53 smithi163 ceph-mon[36451]: pgmap v622: 129 pgs: 129 active+clean; 365 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.6 MiB/s wr, 490 op/s 2024-05-27T07:57:54.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:53 smithi186 ceph-mon[36647]: pgmap v622: 129 pgs: 129 active+clean; 365 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.6 MiB/s wr, 490 op/s 2024-05-27T07:57:54.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:53 smithi114 ceph-mon[32038]: pgmap v622: 129 pgs: 129 active+clean; 365 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.6 MiB/s wr, 490 op/s 2024-05-27T07:57:54.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 60582 12.91 MB/sec execute 126 sec latency 64.424 ms 2024-05-27T07:57:55.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 60845 12.90 MB/sec execute 127 sec latency 110.346 ms 2024-05-27T07:57:56.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:55 smithi163 ceph-mon[36451]: pgmap v623: 129 pgs: 129 active+clean; 363 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.8 MiB/s wr, 518 op/s 2024-05-27T07:57:56.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:55 smithi186 ceph-mon[36647]: pgmap v623: 129 pgs: 129 active+clean; 363 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.8 MiB/s wr, 518 op/s 2024-05-27T07:57:56.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:55 smithi114 ceph-mon[32038]: pgmap v623: 129 pgs: 129 active+clean; 363 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.8 MiB/s wr, 518 op/s 2024-05-27T07:57:56.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 61179 12.99 MB/sec execute 128 sec latency 331.150 ms 2024-05-27T07:57:57.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 61478 13.02 MB/sec execute 129 sec latency 351.024 ms 2024-05-27T07:57:58.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:57 smithi163 ceph-mon[36451]: pgmap v624: 129 pgs: 129 active+clean; 353 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.0 MiB/s wr, 524 op/s 2024-05-27T07:57:58.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:57 smithi186 ceph-mon[36647]: pgmap v624: 129 pgs: 129 active+clean; 353 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.0 MiB/s wr, 524 op/s 2024-05-27T07:57:58.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:57 smithi114 ceph-mon[32038]: pgmap v624: 129 pgs: 129 active+clean; 353 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.0 MiB/s wr, 524 op/s 2024-05-27T07:57:58.608 INFO:tasks.workunit.client.0.smithi114.stdout: 10 61888 13.00 MB/sec execute 130 sec latency 174.402 ms 2024-05-27T07:57:59.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 62268 13.05 MB/sec execute 131 sec latency 160.751 ms 2024-05-27T07:58:00.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:57:59 smithi163 ceph-mon[36451]: pgmap v625: 129 pgs: 129 active+clean; 361 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 502 op/s 2024-05-27T07:58:00.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:57:59 smithi186 ceph-mon[36647]: pgmap v625: 129 pgs: 129 active+clean; 361 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 502 op/s 2024-05-27T07:58:00.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:57:59 smithi114 ceph-mon[32038]: pgmap v625: 129 pgs: 129 active+clean; 361 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 502 op/s 2024-05-27T07:58:00.607 INFO:tasks.workunit.client.0.smithi114.stdout: 10 62760 13.24 MB/sec execute 132 sec latency 57.432 ms 2024-05-27T07:58:01.608 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63038 13.29 MB/sec execute 133 sec latency 237.363 ms 2024-05-27T07:58:01.805 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f2ba60700 time 2024-05-27T07:58:01.365691+0000 2024-05-27T07:58:01.805 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:58:01.806 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:01.806 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f3e2a5ba6] 2024-05-27T07:58:01.806 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f3e2a5dd1] 2024-05-27T07:58:01.806 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5589d58ca23d] 2024-05-27T07:58:01.806 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5589d58b5971] 2024-05-27T07:58:01.806 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 5: (SnapRealm::check_cache() const+0x13e) [0x5589d58b3fa0] 2024-05-27T07:58:01.806 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5589d58b4651] 2024-05-27T07:58:01.806 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5589d5849347] 2024-05-27T07:58:01.806 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5589d554587e] 2024-05-27T07:58:01.806 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5589d554ae94] 2024-05-27T07:58:01.806 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5589d5555969] 2024-05-27T07:58:01.806 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 11: (MDLog::_replay_thread()+0x1f38) [0x5589d58ee48c] 2024-05-27T07:58:01.806 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5589d58f79cb] 2024-05-27T07:58:01.807 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f3e27472d] 2024-05-27T07:58:01.807 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f3e274749] 2024-05-27T07:58:01.807 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f3ce181ca] 2024-05-27T07:58:01.807 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 16: clone() 2024-05-27T07:58:01.807 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.368+0000 7f6f2ba60700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f2ba60700 time 2024-05-27T07:58:01.365691+0000 2024-05-27T07:58:01.807 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:58:01.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:01.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f3e2a5ba6] 2024-05-27T07:58:01.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f3e2a5dd1] 2024-05-27T07:58:01.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5589d58ca23d] 2024-05-27T07:58:01.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5589d58b5971] 2024-05-27T07:58:01.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 5: (SnapRealm::check_cache() const+0x13e) [0x5589d58b3fa0] 2024-05-27T07:58:01.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5589d58b4651] 2024-05-27T07:58:01.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5589d5849347] 2024-05-27T07:58:01.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5589d554587e] 2024-05-27T07:58:01.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5589d554ae94] 2024-05-27T07:58:01.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5589d5555969] 2024-05-27T07:58:01.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 11: (MDLog::_replay_thread()+0x1f38) [0x5589d58ee48c] 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5589d58f79cb] 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f3e27472d] 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f3e274749] 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f3ce181ca] 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 16: clone() 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: *** Caught signal (Aborted) ** 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: in thread 7f6f2ba60700 thread_name:md_log_replay 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f3ce22d20] 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2: gsignal() 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 3: abort() 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f3e2a5ced] 2024-05-27T07:58:01.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f3e2a5dd1] 2024-05-27T07:58:01.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5589d58ca23d] 2024-05-27T07:58:01.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5589d58b5971] 2024-05-27T07:58:01.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 8: (SnapRealm::check_cache() const+0x13e) [0x5589d58b3fa0] 2024-05-27T07:58:01.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5589d58b4651] 2024-05-27T07:58:01.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5589d5849347] 2024-05-27T07:58:01.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5589d554587e] 2024-05-27T07:58:01.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5589d554ae94] 2024-05-27T07:58:01.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5589d5555969] 2024-05-27T07:58:01.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 14: (MDLog::_replay_thread()+0x1f38) [0x5589d58ee48c] 2024-05-27T07:58:01.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5589d58f79cb] 2024-05-27T07:58:01.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f3e27472d] 2024-05-27T07:58:01.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f3e274749] 2024-05-27T07:58:01.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f3ce181ca] 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 19: clone() 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.371+0000 7f6f2ba60700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: in thread 7f6f2ba60700 thread_name:md_log_replay 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f3ce22d20] 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2: gsignal() 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 3: abort() 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f3e2a5ced] 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f3e2a5dd1] 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5589d58ca23d] 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5589d58b5971] 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 8: (SnapRealm::check_cache() const+0x13e) [0x5589d58b3fa0] 2024-05-27T07:58:01.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5589d58b4651] 2024-05-27T07:58:01.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5589d5849347] 2024-05-27T07:58:01.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5589d554587e] 2024-05-27T07:58:01.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5589d554ae94] 2024-05-27T07:58:01.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5589d5555969] 2024-05-27T07:58:01.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 14: (MDLog::_replay_thread()+0x1f38) [0x5589d58ee48c] 2024-05-27T07:58:01.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5589d58f79cb] 2024-05-27T07:58:01.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f3e27472d] 2024-05-27T07:58:01.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f3e274749] 2024-05-27T07:58:01.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f3ce181ca] 2024-05-27T07:58:01.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 19: clone() 2024-05-27T07:58:01.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:58:01.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: -1> 2024-05-27T07:58:01.368+0000 7f6f2ba60700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f2ba60700 time 2024-05-27T07:58:01.365691+0000 2024-05-27T07:58:01.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:58:01.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:01.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f3e2a5ba6] 2024-05-27T07:58:01.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f3e2a5dd1] 2024-05-27T07:58:01.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5589d58ca23d] 2024-05-27T07:58:01.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5589d58b5971] 2024-05-27T07:58:01.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 5: (SnapRealm::check_cache() const+0x13e) [0x5589d58b3fa0] 2024-05-27T07:58:01.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5589d58b4651] 2024-05-27T07:58:01.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5589d5849347] 2024-05-27T07:58:01.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5589d554587e] 2024-05-27T07:58:01.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5589d554ae94] 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5589d5555969] 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 11: (MDLog::_replay_thread()+0x1f38) [0x5589d58ee48c] 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5589d58f79cb] 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f3e27472d] 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f3e274749] 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f3ce181ca] 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 16: clone() 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 0> 2024-05-27T07:58:01.371+0000 7f6f2ba60700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: in thread 7f6f2ba60700 thread_name:md_log_replay 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f3ce22d20] 2024-05-27T07:58:01.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2: gsignal() 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 3: abort() 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f3e2a5ced] 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f3e2a5dd1] 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5589d58ca23d] 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5589d58b5971] 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 8: (SnapRealm::check_cache() const+0x13e) [0x5589d58b3fa0] 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5589d58b4651] 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5589d5849347] 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5589d554587e] 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5589d554ae94] 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5589d5555969] 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 14: (MDLog::_replay_thread()+0x1f38) [0x5589d58ee48c] 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5589d58f79cb] 2024-05-27T07:58:01.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f3e27472d] 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f3e274749] 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f3ce181ca] 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 19: clone() 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: -1> 2024-05-27T07:58:01.368+0000 7f6f2ba60700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f2ba60700 time 2024-05-27T07:58:01.365691+0000 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f3e2a5ba6] 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f3e2a5dd1] 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5589d58ca23d] 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5589d58b5971] 2024-05-27T07:58:01.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 5: (SnapRealm::check_cache() const+0x13e) [0x5589d58b3fa0] 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5589d58b4651] 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5589d5849347] 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5589d554587e] 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5589d554ae94] 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5589d5555969] 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 11: (MDLog::_replay_thread()+0x1f38) [0x5589d58ee48c] 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5589d58f79cb] 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f3e27472d] 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f3e274749] 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f3ce181ca] 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 16: clone() 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 0> 2024-05-27T07:58:01.371+0000 7f6f2ba60700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:58:01.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: in thread 7f6f2ba60700 thread_name:md_log_replay 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f3ce22d20] 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2: gsignal() 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 3: abort() 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f3e2a5ced] 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f3e2a5dd1] 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5589d58ca23d] 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5589d58b5971] 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 8: (SnapRealm::check_cache() const+0x13e) [0x5589d58b3fa0] 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5589d58b4651] 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5589d5849347] 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5589d554587e] 2024-05-27T07:58:01.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5589d554ae94] 2024-05-27T07:58:01.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5589d5555969] 2024-05-27T07:58:01.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 14: (MDLog::_replay_thread()+0x1f38) [0x5589d58ee48c] 2024-05-27T07:58:01.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5589d58f79cb] 2024-05-27T07:58:01.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f3e27472d] 2024-05-27T07:58:01.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f3e274749] 2024-05-27T07:58:01.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f3ce181ca] 2024-05-27T07:58:01.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 19: clone() 2024-05-27T07:58:01.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:58:01.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[129760]: 2024-05-27T07:58:01.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:01 smithi114 podman[130197]: 2024-05-27 07:58:01.498112945 +0000 UTC m=+0.057350950 container died 96dacd30b5d5a415c620e33757e78e239b3982d41e3581ffa92c451f84923686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:58:02.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:01 smithi114 ceph-mon[32038]: pgmap v626: 129 pgs: 129 active+clean; 361 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 461 op/s 2024-05-27T07:58:02.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:01 smithi163 ceph-mon[36451]: pgmap v626: 129 pgs: 129 active+clean; 361 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 461 op/s 2024-05-27T07:58:02.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:01 smithi186 ceph-mon[36647]: pgmap v626: 129 pgs: 129 active+clean; 361 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 461 op/s 2024-05-27T07:58:02.557 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:02 smithi114 podman[130197]: 2024-05-27 07:58:02.238039607 +0000 UTC m=+0.797277588 container remove 96dacd30b5d5a415c620e33757e78e239b3982d41e3581ffa92c451f84923686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS) 2024-05-27T07:58:02.557 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:02 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:58:02.608 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63149 13.23 MB/sec execute 134 sec latency 866.663 ms 2024-05-27T07:58:03.057 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:02 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:58:03.057 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:02 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Consumed 3.518s CPU time. 2024-05-27T07:58:03.608 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63222 13.14 MB/sec execute 135 sec latency 1095.497 ms 2024-05-27T07:58:04.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:03 smithi163 ceph-mon[36451]: pgmap v627: 129 pgs: 129 active+clean; 368 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.7 MiB/s wr, 565 op/s 2024-05-27T07:58:04.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:03 smithi163 ceph-mon[36451]: osdmap e155: 12 total, 12 up, 12 in 2024-05-27T07:58:04.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:03 smithi186 ceph-mon[36647]: pgmap v627: 129 pgs: 129 active+clean; 368 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.7 MiB/s wr, 565 op/s 2024-05-27T07:58:04.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:03 smithi186 ceph-mon[36647]: osdmap e155: 12 total, 12 up, 12 in 2024-05-27T07:58:04.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:03 smithi114 ceph-mon[32038]: pgmap v627: 129 pgs: 129 active+clean; 368 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.7 MiB/s wr, 565 op/s 2024-05-27T07:58:04.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:03 smithi114 ceph-mon[32038]: osdmap e155: 12 total, 12 up, 12 in 2024-05-27T07:58:04.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.583+0000 7f23308b6640 1 -- 172.21.15.114:0/2927549563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f232c11d240 msgr2=0x7f232c11f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:04.588 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.583+0000 7f23308b6640 1 --2- 172.21.15.114:0/2927549563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f232c11d240 0x7f232c11f660 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2324009ca0 tx=0x7f232402a960 comp rx=0 tx=0).stop 2024-05-27T07:58:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.589+0000 7f23308b6640 1 -- 172.21.15.114:0/2927549563 shutdown_connections 2024-05-27T07:58:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.589+0000 7f23308b6640 1 --2- 172.21.15.114:0/2927549563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f232c11d240 0x7f232c11f660 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.589+0000 7f23308b6640 1 --2- 172.21.15.114:0/2927549563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f232c11a8e0 0x7f232c11cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.589+0000 7f23308b6640 1 --2- 172.21.15.114:0/2927549563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f232c10a1f0 0x7f232c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.589+0000 7f23308b6640 1 -- 172.21.15.114:0/2927549563 >> 172.21.15.114:0/2927549563 conn(0x7f232c06e8f0 msgr2=0x7f232c06ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.597+0000 7f23308b6640 1 -- 172.21.15.114:0/2927549563 shutdown_connections 2024-05-27T07:58:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.598+0000 7f23308b6640 1 -- 172.21.15.114:0/2927549563 wait complete. 2024-05-27T07:58:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.599+0000 7f23308b6640 1 Processor -- start 2024-05-27T07:58:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.601+0000 7f23308b6640 1 -- start start 2024-05-27T07:58:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.601+0000 7f23308b6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f232c10a1f0 0x7f232c1a9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.601+0000 7f23308b6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f232c11a8e0 0x7f232c1a9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.601+0000 7f23308b6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f232c11d240 0x7f232c1ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.601+0000 7f23308b6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f232c122c40 con 0x7f232c11a8e0 2024-05-27T07:58:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.601+0000 7f23308b6640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f232c122ac0 con 0x7f232c10a1f0 2024-05-27T07:58:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.601+0000 7f23308b6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f232c122dc0 con 0x7f232c11d240 2024-05-27T07:58:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.602+0000 7f232ad9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f232c11d240 0x7f232c1ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.602+0000 7f232ad9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f232c11d240 0x7f232c1ade80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:48248/0 (socket says 172.21.15.114:48248) 2024-05-27T07:58:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.602+0000 7f232ad9d640 1 -- 172.21.15.114:0/1557526639 learned_addr learned my addr 172.21.15.114:0/1557526639 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.603+0000 7f232ad9d640 1 -- 172.21.15.114:0/1557526639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f232c10a1f0 msgr2=0x7f232c1a9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.603+0000 7f232ad9d640 1 --2- 172.21.15.114:0/1557526639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f232c10a1f0 0x7f232c1a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.603+0000 7f232ad9d640 1 -- 172.21.15.114:0/1557526639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f232c11a8e0 msgr2=0x7f232c1a9bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:58:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.603+0000 7f232ad9d640 1 --2- 172.21.15.114:0/1557526639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f232c11a8e0 0x7f232c1a9bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.603+0000 7f232ad9d640 1 -- 172.21.15.114:0/1557526639 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f232c1ae4c0 con 0x7f232c11d240 2024-05-27T07:58:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.604+0000 7f232ad9d640 1 --2- 172.21.15.114:0/1557526639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f232c11d240 0x7f232c1ade80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2324006010 tx=0x7f2324003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.604+0000 7f23237fe640 1 -- 172.21.15.114:0/1557526639 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2324034040 con 0x7f232c11d240 2024-05-27T07:58:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.604+0000 7f23308b6640 1 -- 172.21.15.114:0/1557526639 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f232c1ae750 con 0x7f232c11d240 2024-05-27T07:58:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.605+0000 7f23308b6640 1 -- 172.21.15.114:0/1557526639 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f232c1aec90 con 0x7f232c11d240 2024-05-27T07:58:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.605+0000 7f23237fe640 1 -- 172.21.15.114:0/1557526639 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f232403e070 con 0x7f232c11d240 2024-05-27T07:58:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.606+0000 7f23237fe640 1 -- 172.21.15.114:0/1557526639 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2324039450 con 0x7f232c11d240 2024-05-27T07:58:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.610+0000 7f2337769740 1 Processor -- start 2024-05-27T07:58:04.614 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63258 13.05 MB/sec execute 136 sec latency 1213.734 ms 2024-05-27T07:58:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.613+0000 7f2337769740 1 -- start start 2024-05-27T07:58:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.614+0000 7f2337769740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034751b0 0x55fc038163d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.614+0000 7f2337769740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc0344c2f0 0x55fc03816910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.614+0000 7f2337769740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc034544d0 0x55fc03816e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.614+0000 7f2337769740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fc03788c30 con 0x55fc034751b0 2024-05-27T07:58:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.614+0000 7f2337769740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fc037787d0 con 0x55fc0344c2f0 2024-05-27T07:58:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.614+0000 7f2337769740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fc0377c810 con 0x55fc034544d0 2024-05-27T07:58:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.614+0000 7f23237fe640 1 -- 172.21.15.114:0/1557526639 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f2324037020 con 0x7f232c11d240 2024-05-27T07:58:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.615+0000 7f23237fe640 1 --2- 172.21.15.114:0/1557526639 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f23180b2880 0x7f23180b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.616+0000 7f23237fe640 1 -- 172.21.15.114:0/1557526639 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 9642+0+0 (secure 0 0 0) 0x7f2324103f40 con 0x7f232c11d240 2024-05-27T07:58:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.616+0000 7f232a59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc0344c2f0 0x55fc03816910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.618+0000 7f232a59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc0344c2f0 0x55fc03816910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:35826/0 (socket says 172.21.15.114:35826) 2024-05-27T07:58:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.618+0000 7f232a59c640 1 -- 172.21.15.114:0/3197602358 learned_addr learned my addr 172.21.15.114:0/3197602358 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.620+0000 7f232ad9d640 1 --2- 172.21.15.114:0/1557526639 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f23180b2880 0x7f23180b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:04.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.624+0000 7f2323fff640 1 --2- 172.21.15.114:0/3197602358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034751b0 0x55fc038163d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:04.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.624+0000 7f232a59c640 1 -- 172.21.15.114:0/3197602358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc034544d0 msgr2=0x55fc03816e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:58:04.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.624+0000 7f232a59c640 1 --2- 172.21.15.114:0/3197602358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc034544d0 0x55fc03816e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.624+0000 7f232a59c640 1 -- 172.21.15.114:0/3197602358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034751b0 msgr2=0x55fc038163d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:04.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.624+0000 7f232a59c640 1 --2- 172.21.15.114:0/3197602358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034751b0 0x55fc038163d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.624+0000 7f232a59c640 1 -- 172.21.15.114:0/3197602358 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc0348d210 con 0x55fc0344c2f0 2024-05-27T07:58:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.624+0000 7f2323fff640 1 --2- 172.21.15.114:0/3197602358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034751b0 0x55fc038163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:58:04.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.630+0000 7f232a59c640 1 --2- 172.21.15.114:0/3197602358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc0344c2f0 0x55fc03816910 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f231c009820 tx=0x7f231c009ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f65af14b3fc73129 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.630+0000 7f232ad9d640 1 --2- 172.21.15.114:0/1557526639 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f23180b2880 0x7f23180b4d90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f23240c6ef0 tx=0x7f232403c040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:04.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.631+0000 7f2320ff9640 1 -- 172.21.15.114:0/3197602358 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c015020 con 0x55fc0344c2f0 2024-05-27T07:58:04.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.631+0000 7f2320ff9640 1 -- 172.21.15.114:0/3197602358 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f231c011420 con 0x55fc0344c2f0 2024-05-27T07:58:04.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.631+0000 7f2320ff9640 1 -- 172.21.15.114:0/3197602358 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c0105e0 con 0x55fc0344c2f0 2024-05-27T07:58:04.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.635+0000 7f2337769740 1 -- 172.21.15.114:0/3197602358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc0344c2f0 msgr2=0x55fc03816910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:04.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.635+0000 7f2337769740 1 --2- 172.21.15.114:0/3197602358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc0344c2f0 0x55fc03816910 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f231c009820 tx=0x7f231c009ce0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.640+0000 7f2337769740 1 -- 172.21.15.114:0/3197602358 shutdown_connections 2024-05-27T07:58:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.640+0000 7f2337769740 1 --2- 172.21.15.114:0/3197602358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc034544d0 0x55fc03816e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.640+0000 7f2337769740 1 --2- 172.21.15.114:0/3197602358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc0344c2f0 0x55fc03816910 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.640+0000 7f2337769740 1 --2- 172.21.15.114:0/3197602358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034751b0 0x55fc038163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.640+0000 7f2337769740 1 -- 172.21.15.114:0/3197602358 >> 172.21.15.114:0/3197602358 conn(0x55fc03469300 msgr2=0x55fc03812b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.644+0000 7f2337769740 1 -- 172.21.15.114:0/3197602358 shutdown_connections 2024-05-27T07:58:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.644+0000 7f2337769740 1 -- 172.21.15.114:0/3197602358 wait complete. 2024-05-27T07:58:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.645+0000 7f2337769740 1 -- start start 2024-05-27T07:58:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.645+0000 7f2337769740 1 Processor -- start 2024-05-27T07:58:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.646+0000 7f2337769740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc0344c2f0 0x55fc03822490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.646+0000 7f2337769740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034544d0 0x55fc038229d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.646+0000 7f2337769740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034751b0 0x55fc038209a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.646+0000 7f2337769740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fc0345e790 con 0x55fc034751b0 2024-05-27T07:58:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.646+0000 7f2337769740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fc03734890 con 0x55fc034544d0 2024-05-27T07:58:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.646+0000 7f2337769740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fc03823500 con 0x55fc0344c2f0 2024-05-27T07:58:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.646+0000 7f2337769740 10 client.0 _ll_register_callbacks cb 0x55fc0348ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.647+0000 7f232a59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034544d0 0x55fc038229d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.647+0000 7f232a59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034544d0 0x55fc038229d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:35828/0 (socket says 172.21.15.114:35828) 2024-05-27T07:58:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.647+0000 7f232a59c640 1 -- 172.21.15.114:0/1506184942 learned_addr learned my addr 172.21.15.114:0/1506184942 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.647+0000 7f2323fff640 1 --2- 172.21.15.114:0/1506184942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034751b0 0x55fc038209a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.647+0000 7f2323fff640 1 --2- 172.21.15.114:0/1506184942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc0344c2f0 0x55fc03822490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.647+0000 7f232a59c640 1 -- 172.21.15.114:0/1506184942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc0344c2f0 msgr2=0x55fc03822490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.647+0000 7f232a59c640 1 --2- 172.21.15.114:0/1506184942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc0344c2f0 0x55fc03822490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.647+0000 7f232a59c640 1 -- 172.21.15.114:0/1506184942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034751b0 msgr2=0x55fc038209a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.647+0000 7f232a59c640 1 --2- 172.21.15.114:0/1506184942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034751b0 0x55fc038209a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.648+0000 7f232a59c640 1 -- 172.21.15.114:0/1506184942 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc03559620 con 0x55fc034544d0 2024-05-27T07:58:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.651+0000 7f232a59c640 1 --2- 172.21.15.114:0/1506184942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034544d0 0x55fc038229d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f231c011e40 tx=0x7f231c011800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6262976663faa2a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.652+0000 7f2311ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:58:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.652+0000 7f2311ffb640 1 -- 172.21.15.114:0/1506184942 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c027060 con 0x55fc034544d0 2024-05-27T07:58:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.652+0000 7f2311ffb640 1 -- 172.21.15.114:0/1506184942 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9642+0+0 (secure 0 0 0) 0x7f231c00bae0 con 0x55fc034544d0 2024-05-27T07:58:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.653+0000 7f2337769740 1 -- 172.21.15.114:0/1506184942 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f231c011420 con 0x55fc034544d0 2024-05-27T07:58:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.659+0000 7f2311ffb640 1 -- 172.21.15.114:0/1506184942 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f231c0119f0 con 0x55fc034544d0 2024-05-27T07:58:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.661+0000 7f2337769740 10 client.28844 fetch_fsmap learned FSMap version 102 2024-05-27T07:58:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.661+0000 7f2337769740 1 -- 172.21.15.114:0/1506184942 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x55fc0382f8a0 con 0x55fc034544d0 2024-05-27T07:58:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.662+0000 7f2311ffb640 1 -- 172.21.15.114:0/1506184942 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f231c011420 con 0x55fc034544d0 2024-05-27T07:58:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.662+0000 7f2311ffb640 1 -- 172.21.15.114:0/1506184942 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 102) ==== 4095+0+0 (secure 0 0 0) 0x7f231c02d480 con 0x55fc034544d0 2024-05-27T07:58:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.662+0000 7f2337769740 10 client.28844 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:58:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.662+0000 7f2337769740 10 client.28844 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.662+0000 7f2337769740 20 client.28844 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.662+0000 7f2337769740 1 --2- 172.21.15.114:0/1506184942 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fc03433d80 0x55fc03832fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.662+0000 7f2337769740 4 client.28844 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:58:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.662+0000 7f2337769740 1 -- 172.21.15.114:0/1506184942 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc0378ab80 con 0x55fc03433d80 2024-05-27T07:58:04.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.673+0000 7f2323fff640 1 --2- 172.21.15.114:0/1506184942 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fc03433d80 0x55fc03832fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:04.684 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.679+0000 7f2323fff640 1 --2- 172.21.15.114:0/1506184942 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fc03433d80 0x55fc03832fe0 crc :-1 s=READY pgs=484 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=683c604786ba1e55 server_cookie=a3c20bcfa2254da5 in_seq=0 out_seq=0 2024-05-27T07:58:04.684 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.680+0000 7f2311ffb640 10 client.28844 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.684+0000 7f2311ffb640 1 -- 172.21.15.114:0/1506184942 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fc0378ab80 con 0x55fc03433d80 2024-05-27T07:58:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.684+0000 7f2311ffb640 10 client.28844 handle_command_reply: tid=0 2024-05-27T07:58:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.684+0000 7f2337769740 1 client.28844 shutdown 2024-05-27T07:58:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.687+0000 7f2337769740 1 -- 172.21.15.114:0/1506184942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034544d0 msgr2=0x55fc038229d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.687+0000 7f2337769740 1 --2- 172.21.15.114:0/1506184942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034544d0 0x55fc038229d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f231c011e40 tx=0x7f231c011800 comp rx=0 tx=0).stop 2024-05-27T07:58:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.688+0000 7f2337769740 1 -- 172.21.15.114:0/1506184942 shutdown_connections 2024-05-27T07:58:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.688+0000 7f2337769740 1 --2- 172.21.15.114:0/1506184942 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fc03433d80 0x55fc03832fe0 crc :-1 s=READY pgs=484 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.688+0000 7f2337769740 1 --2- 172.21.15.114:0/1506184942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034751b0 0x55fc038209a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.688+0000 7f2337769740 1 --2- 172.21.15.114:0/1506184942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034544d0 0x55fc038229d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.688+0000 7f2337769740 1 --2- 172.21.15.114:0/1506184942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc0344c2f0 0x55fc03822490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.688+0000 7f2337769740 1 -- 172.21.15.114:0/1506184942 >> 172.21.15.114:0/1506184942 conn(0x55fc03469300 msgr2=0x55fc038180f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.688+0000 7f2311ffb640 0 client.28844 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.694+0000 7f2337769740 1 -- 172.21.15.114:0/1506184942 shutdown_connections 2024-05-27T07:58:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.698+0000 7f2337769740 1 -- 172.21.15.114:0/1506184942 wait complete. 2024-05-27T07:58:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.698+0000 7f2337769740 20 client.28844 trim_cache size 0 max 16384 2024-05-27T07:58:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.726+0000 7f2337769740 1 Processor -- start 2024-05-27T07:58:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.726+0000 7f2337769740 1 -- start start 2024-05-27T07:58:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.726+0000 7f2337769740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc0344c2f0 0x55fc038205f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.726+0000 7f2337769740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034544d0 0x55fc038387a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.727+0000 7f2337769740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034751b0 0x55fc03838ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.727+0000 7f2337769740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f230c00ff00 con 0x55fc034544d0 2024-05-27T07:58:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.727+0000 7f2337769740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f230c0105d0 con 0x55fc034751b0 2024-05-27T07:58:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.727+0000 7f2337769740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f230c00eea0 con 0x55fc0344c2f0 2024-05-27T07:58:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.727+0000 7f232a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034544d0 0x55fc038387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.727+0000 7f232a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034544d0 0x55fc038387a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45266/0 (socket says 172.21.15.114:45266) 2024-05-27T07:58:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.727+0000 7f232a59c640 1 -- 172.21.15.114:0/2973287554 learned_addr learned my addr 172.21.15.114:0/2973287554 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.728+0000 7f232a59c640 1 -- 172.21.15.114:0/2973287554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc0344c2f0 msgr2=0x55fc038205f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:58:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.728+0000 7f232a59c640 1 --2- 172.21.15.114:0/2973287554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc0344c2f0 0x55fc038205f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.728+0000 7f232a59c640 1 -- 172.21.15.114:0/2973287554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034751b0 msgr2=0x55fc03838ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:58:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.728+0000 7f232a59c640 1 --2- 172.21.15.114:0/2973287554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034751b0 0x55fc03838ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.728+0000 7f232a59c640 1 -- 172.21.15.114:0/2973287554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc03823580 con 0x55fc034544d0 2024-05-27T07:58:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.728+0000 7f232a59c640 1 --2- 172.21.15.114:0/2973287554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034544d0 0x55fc038387a0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f231c011c00 tx=0x7f231c006dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f31c86b7e1c5b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.729+0000 7f2313fff640 1 -- 172.21.15.114:0/2973287554 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c015070 con 0x55fc034544d0 2024-05-27T07:58:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.729+0000 7f2313fff640 1 -- 172.21.15.114:0/2973287554 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f231c011420 con 0x55fc034544d0 2024-05-27T07:58:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.730+0000 7f2313fff640 1 -- 172.21.15.114:0/2973287554 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c027040 con 0x55fc034544d0 2024-05-27T07:58:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.730+0000 7f2337769740 1 -- 172.21.15.114:0/2973287554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034544d0 msgr2=0x55fc038387a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.730+0000 7f2337769740 1 --2- 172.21.15.114:0/2973287554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034544d0 0x55fc038387a0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f231c011c00 tx=0x7f231c006dd0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.731+0000 7f2337769740 1 -- 172.21.15.114:0/2973287554 shutdown_connections 2024-05-27T07:58:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.731+0000 7f2337769740 1 --2- 172.21.15.114:0/2973287554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034751b0 0x55fc03838ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.731+0000 7f2337769740 1 --2- 172.21.15.114:0/2973287554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc034544d0 0x55fc038387a0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.731+0000 7f2337769740 1 --2- 172.21.15.114:0/2973287554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc0344c2f0 0x55fc038205f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.731+0000 7f2337769740 1 -- 172.21.15.114:0/2973287554 >> 172.21.15.114:0/2973287554 conn(0x55fc03469300 msgr2=0x55fc0381fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:04.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.737+0000 7f2337769740 1 -- 172.21.15.114:0/2973287554 shutdown_connections 2024-05-27T07:58:04.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.739+0000 7f2337769740 1 -- 172.21.15.114:0/2973287554 wait complete. 2024-05-27T07:58:04.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.739+0000 7f2337769740 1 -- start start 2024-05-27T07:58:04.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.739+0000 7f2337769740 1 Processor -- start 2024-05-27T07:58:04.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.740+0000 7f2337769740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc0344c2f0 0x55fc03828d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.740+0000 7f2337769740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc034544d0 0x55fc038260a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.740+0000 7f2337769740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034751b0 0x55fc038265e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.740+0000 7f2337769740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f230c00e7d0 con 0x55fc0344c2f0 2024-05-27T07:58:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.740+0000 7f2337769740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f230c00e180 con 0x55fc034751b0 2024-05-27T07:58:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.740+0000 7f2337769740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fc0378ab80 con 0x55fc034544d0 2024-05-27T07:58:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.740+0000 7f2337769740 10 client.0 _ll_register_callbacks cb 0x7f231c009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:04.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.743+0000 7f232a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc034544d0 0x55fc038260a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.743+0000 7f232a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc034544d0 0x55fc038260a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:48274/0 (socket says 172.21.15.114:48274) 2024-05-27T07:58:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.743+0000 7f232a59c640 1 -- 172.21.15.114:0/276954747 learned_addr learned my addr 172.21.15.114:0/276954747 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:04.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.746+0000 7f232a59c640 1 -- 172.21.15.114:0/276954747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034751b0 msgr2=0x55fc038265e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:04.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.746+0000 7f232a59c640 1 --2- 172.21.15.114:0/276954747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034751b0 0x55fc038265e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.746+0000 7f232a59c640 1 -- 172.21.15.114:0/276954747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc0344c2f0 msgr2=0x55fc03828d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:04.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.746+0000 7f232a59c640 1 --2- 172.21.15.114:0/276954747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc0344c2f0 0x55fc03828d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:04.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.746+0000 7f232a59c640 1 -- 172.21.15.114:0/276954747 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc03826b20 con 0x55fc034544d0 2024-05-27T07:58:04.751 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.750+0000 7f232a59c640 1 --2- 172.21.15.114:0/276954747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc034544d0 0x55fc038260a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f231c009200 tx=0x7f231c02d660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80f117a2a4cb4c47 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:04.754 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.750+0000 7f2311ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:58:04.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.754+0000 7f2311ffb640 1 -- 172.21.15.114:0/276954747 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c015070 con 0x55fc034544d0 2024-05-27T07:58:04.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.754+0000 7f2337769740 1 -- 172.21.15.114:0/276954747 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f231c011420 con 0x55fc034544d0 2024-05-27T07:58:04.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.754+0000 7f2311ffb640 1 -- 172.21.15.114:0/276954747 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9642+0+0 (secure 0 0 0) 0x7f231c01a420 con 0x55fc034544d0 2024-05-27T07:58:04.758 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.755+0000 7f2311ffb640 1 -- 172.21.15.114:0/276954747 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f231c027040 con 0x55fc034544d0 2024-05-27T07:58:04.758 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.757+0000 7f2337769740 10 client.29011 fetch_fsmap learned FSMap version 102 2024-05-27T07:58:04.758 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.757+0000 7f2337769740 1 -- 172.21.15.114:0/276954747 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x55fc0389b380 con 0x55fc034544d0 2024-05-27T07:58:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.757+0000 7f2311ffb640 1 -- 172.21.15.114:0/276954747 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f231c011420 con 0x55fc034544d0 2024-05-27T07:58:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.757+0000 7f2311ffb640 1 -- 172.21.15.114:0/276954747 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 102) ==== 4095+0+0 (secure 0 0 0) 0x7f231c035410 con 0x55fc034544d0 2024-05-27T07:58:04.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.760+0000 7f2337769740 10 client.29011 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:58:04.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.760+0000 7f2337769740 10 client.29011 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:04.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.760+0000 7f2337769740 20 client.29011 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:04.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.760+0000 7f2337769740 1 --2- 172.21.15.114:0/276954747 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fc034091c0 0x55fc0389d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:04.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.760+0000 7f2337769740 4 client.29011 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:58:04.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.760+0000 7f2337769740 1 -- 172.21.15.114:0/276954747 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fc03784360 con 0x55fc034091c0 2024-05-27T07:58:04.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.766+0000 7f2323fff640 1 --2- 172.21.15.114:0/276954747 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fc034091c0 0x55fc0389d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:04.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.775+0000 7f2323fff640 1 --2- 172.21.15.114:0/276954747 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fc034091c0 0x55fc0389d820 crc :-1 s=READY pgs=485 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12b3a5a155afda05 server_cookie=6f5b25b2d4a53936 in_seq=0 out_seq=0 2024-05-27T07:58:04.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:04.777+0000 7f2311ffb640 10 client.29011 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:05.612 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63267 12.96 MB/sec execute 137 sec latency 1818.063 ms 2024-05-27T07:58:06.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:05 smithi163 ceph-mon[36451]: pgmap v629: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 368 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.7 MiB/s wr, 314 op/s 2024-05-27T07:58:06.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:05 smithi186 ceph-mon[36647]: pgmap v629: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 368 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.7 MiB/s wr, 314 op/s 2024-05-27T07:58:06.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:05 smithi114 ceph-mon[32038]: pgmap v629: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 368 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.7 MiB/s wr, 314 op/s 2024-05-27T07:58:06.612 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63269 12.87 MB/sec execute 138 sec latency 2179.124 ms 2024-05-27T07:58:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.765+0000 7f2311ffb640 1 -- 172.21.15.114:0/276954747 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55fc03784360 con 0x55fc034091c0 2024-05-27T07:58:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.765+0000 7f2311ffb640 10 client.29011 handle_command_reply: tid=0 2024-05-27T07:58:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.766+0000 7f2337769740 1 client.29011 shutdown 2024-05-27T07:58:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.766+0000 7f2337769740 1 -- 172.21.15.114:0/276954747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc034544d0 msgr2=0x55fc038260a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.766+0000 7f2337769740 1 --2- 172.21.15.114:0/276954747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc034544d0 0x55fc038260a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f231c009200 tx=0x7f231c02d660 comp rx=0 tx=0).stop 2024-05-27T07:58:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.766+0000 7f2337769740 1 -- 172.21.15.114:0/276954747 shutdown_connections 2024-05-27T07:58:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.766+0000 7f2337769740 1 --2- 172.21.15.114:0/276954747 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fc034091c0 0x55fc0389d820 crc :-1 s=READY pgs=485 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.766+0000 7f2337769740 1 --2- 172.21.15.114:0/276954747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc034751b0 0x55fc038265e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.766+0000 7f2337769740 1 --2- 172.21.15.114:0/276954747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc034544d0 0x55fc038260a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.766+0000 7f2337769740 1 --2- 172.21.15.114:0/276954747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fc0344c2f0 0x55fc03828d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.766+0000 7f2337769740 1 -- 172.21.15.114:0/276954747 >> 172.21.15.114:0/276954747 conn(0x55fc03469300 msgr2=0x55fc03822730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.766+0000 7f2311ffb640 0 client.29011 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.766+0000 7f2337769740 1 -- 172.21.15.114:0/276954747 shutdown_connections 2024-05-27T07:58:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.766+0000 7f2337769740 1 -- 172.21.15.114:0/276954747 wait complete. 2024-05-27T07:58:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.766+0000 7f2337769740 20 client.29011 trim_cache size 0 max 16384 2024-05-27T07:58:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:58:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:58:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:58:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.768+0000 7f23308b6640 1 -- 172.21.15.114:0/1557526639 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f23180b2880 msgr2=0x7f23180b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.768+0000 7f23308b6640 1 --2- 172.21.15.114:0/1557526639 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f23180b2880 0x7f23180b4d90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f23240c6ef0 tx=0x7f232403c040 comp rx=0 tx=0).stop 2024-05-27T07:58:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.768+0000 7f23308b6640 1 -- 172.21.15.114:0/1557526639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f232c11d240 msgr2=0x7f232c1ade80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.768+0000 7f23308b6640 1 --2- 172.21.15.114:0/1557526639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f232c11d240 0x7f232c1ade80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2324006010 tx=0x7f2324003ba0 comp rx=0 tx=0).stop 2024-05-27T07:58:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.768+0000 7f23308b6640 1 -- 172.21.15.114:0/1557526639 shutdown_connections 2024-05-27T07:58:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.768+0000 7f23308b6640 1 --2- 172.21.15.114:0/1557526639 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f23180b2880 0x7f23180b4d90 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.768+0000 7f23308b6640 1 --2- 172.21.15.114:0/1557526639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f232c11d240 0x7f232c1ade80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.768+0000 7f23308b6640 1 --2- 172.21.15.114:0/1557526639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f232c11a8e0 0x7f232c1a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.768+0000 7f23308b6640 1 --2- 172.21.15.114:0/1557526639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f232c10a1f0 0x7f232c1a9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.768+0000 7f23308b6640 1 -- 172.21.15.114:0/1557526639 >> 172.21.15.114:0/1557526639 conn(0x7f232c06e8f0 msgr2=0x7f232c11acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.768+0000 7f23308b6640 1 -- 172.21.15.114:0/1557526639 shutdown_connections 2024-05-27T07:58:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:06.768+0000 7f23308b6640 1 -- 172.21.15.114:0/1557526639 wait complete. 2024-05-27T07:58:07.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:58:07.613 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63270 12.78 MB/sec execute 139 sec latency 2814.501 ms 2024-05-27T07:58:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:58:08.115 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:07 smithi114 ceph-mon[32038]: pgmap v630: 129 pgs: 11 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 93 active+clean; 372 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.4 MiB/s wr, 499 op/s 2024-05-27T07:58:08.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:07 smithi163 ceph-mon[36451]: pgmap v630: 129 pgs: 11 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 93 active+clean; 372 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.4 MiB/s wr, 499 op/s 2024-05-27T07:58:08.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:07 smithi186 ceph-mon[36647]: pgmap v630: 129 pgs: 11 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 93 active+clean; 372 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.4 MiB/s wr, 499 op/s 2024-05-27T07:58:08.613 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63271 12.69 MB/sec execute 140 sec latency 3480.098 ms 2024-05-27T07:58:09.613 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63273 12.61 MB/sec execute 141 sec latency 1618.157 ms 2024-05-27T07:58:10.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.064+0000 7f9ab1f39700 1 -- 172.21.15.114:0/3949513930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9aac103070 msgr2=0x7f9aac103480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:10.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.064+0000 7f9ab1f39700 1 --2- 172.21.15.114:0/3949513930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9aac103070 0x7f9aac103480 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009990 tx=0x7f9a9c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:09 smithi114 ceph-mon[32038]: pgmap v631: 129 pgs: 19 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 61 active+clean; 359 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.4 MiB/s wr, 594 op/s 2024-05-27T07:58:10.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.067+0000 7f9ab1f39700 1 -- 172.21.15.114:0/3949513930 shutdown_connections 2024-05-27T07:58:10.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.067+0000 7f9ab1f39700 1 --2- 172.21.15.114:0/3949513930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac1084c0 0x7f9aac108990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.067+0000 7f9ab1f39700 1 --2- 172.21.15.114:0/3949513930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9aac103070 0x7f9aac103480 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.067+0000 7f9ab1f39700 1 --2- 172.21.15.114:0/3949513930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9aac1039c0 0x7f9aac107ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.067+0000 7f9ab1f39700 1 -- 172.21.15.114:0/3949513930 >> 172.21.15.114:0/3949513930 conn(0x7f9aac078ab0 msgr2=0x7f9aac078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:10.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.069+0000 7f9ab1f39700 1 -- 172.21.15.114:0/3949513930 shutdown_connections 2024-05-27T07:58:10.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.069+0000 7f9ab1f39700 1 -- 172.21.15.114:0/3949513930 wait complete. 2024-05-27T07:58:10.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.070+0000 7f9ab1f39700 1 Processor -- start 2024-05-27T07:58:10.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.070+0000 7f9ab1f39700 1 -- start start 2024-05-27T07:58:10.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.070+0000 7f9ab1f39700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9aac103070 0x7f9aac075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.070+0000 7f9ab1f39700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9aac1039c0 0x7f9aac06e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.070+0000 7f9ab1f39700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac1084c0 0x7f9aac06e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.070+0000 7f9ab1f39700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac11aa00 con 0x7f9aac1084c0 2024-05-27T07:58:10.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.070+0000 7f9ab1f39700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac11a880 con 0x7f9aac103070 2024-05-27T07:58:10.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.070+0000 7f9ab1f39700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac11ab80 con 0x7f9aac1039c0 2024-05-27T07:58:10.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.070+0000 7f9ab1738700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac1084c0 0x7f9aac06e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.071+0000 7f9ab1738700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac1084c0 0x7f9aac06e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45282/0 (socket says 172.21.15.114:45282) 2024-05-27T07:58:10.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.071+0000 7f9ab1738700 1 -- 172.21.15.114:0/552637907 learned_addr learned my addr 172.21.15.114:0/552637907 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:10.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.071+0000 7f9ab1738700 1 -- 172.21.15.114:0/552637907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9aac1039c0 msgr2=0x7f9aac06e440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:58:10.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.071+0000 7f9ab1738700 1 --2- 172.21.15.114:0/552637907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9aac1039c0 0x7f9aac06e440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.071+0000 7f9ab1738700 1 -- 172.21.15.114:0/552637907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9aac103070 msgr2=0x7f9aac075310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:58:10.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.071+0000 7f9ab1738700 1 --2- 172.21.15.114:0/552637907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9aac103070 0x7f9aac075310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.071+0000 7f9ab1738700 1 -- 172.21.15.114:0/552637907 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aac06f240 con 0x7f9aac1084c0 2024-05-27T07:58:10.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.071+0000 7f9ab1738700 1 --2- 172.21.15.114:0/552637907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac1084c0 0x7f9aac06e9e0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400b550 tx=0x7f9a9400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:10.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.073+0000 7f9aa9ffb700 1 -- 172.21.15.114:0/552637907 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94015020 con 0x7f9aac1084c0 2024-05-27T07:58:10.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.073+0000 7f9ab1f39700 1 -- 172.21.15.114:0/552637907 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aac072f60 con 0x7f9aac1084c0 2024-05-27T07:58:10.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.073+0000 7f9ab1f39700 1 -- 172.21.15.114:0/552637907 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aac0734a0 con 0x7f9aac1084c0 2024-05-27T07:58:10.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.073+0000 7f9aa9ffb700 1 -- 172.21.15.114:0/552637907 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a94011ce0 con 0x7f9aac1084c0 2024-05-27T07:58:10.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.073+0000 7f9aa9ffb700 1 -- 172.21.15.114:0/552637907 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94010d10 con 0x7f9aac1084c0 2024-05-27T07:58:10.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.081+0000 7f9aa9ffb700 1 -- 172.21.15.114:0/552637907 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f9a94018020 con 0x7f9aac1084c0 2024-05-27T07:58:10.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.084+0000 7f9ac1bb0b80 1 Processor -- start 2024-05-27T07:58:10.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.086+0000 7f9aa9ffb700 1 --2- 172.21.15.114:0/552637907 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a980b2c00 0x7f9a980b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.086+0000 7f9aa9ffb700 1 -- 172.21.15.114:0/552637907 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 9642+0+0 (secure 0 0 0) 0x7f9a940e2010 con 0x7f9aac1084c0 2024-05-27T07:58:10.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.088+0000 7f9aabfff700 1 --2- 172.21.15.114:0/552637907 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a980b2c00 0x7f9a980b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.088+0000 7f9ac1bb0b80 1 -- start start 2024-05-27T07:58:10.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.088+0000 7f9ac1bb0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf383e40 0x563bcf507880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.088+0000 7f9ac1bb0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf507dc0 0x563bcf501a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.088+0000 7f9ac1bb0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf501f80 0x563bcf502390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.089+0000 7f9ac1bb0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563bcf430c90 con 0x563bcf501f80 2024-05-27T07:58:10.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.089+0000 7f9ac1bb0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563bcf420830 con 0x563bcf507dc0 2024-05-27T07:58:10.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.089+0000 7f9ac1bb0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563bcf424870 con 0x563bcf383e40 2024-05-27T07:58:10.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.091+0000 7f9ab1738700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf501f80 0x563bcf502390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.091+0000 7f9ab1738700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf501f80 0x563bcf502390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45284/0 (socket says 172.21.15.114:45284) 2024-05-27T07:58:10.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.091+0000 7f9ab1738700 1 -- 172.21.15.114:0/1711708562 learned_addr learned my addr 172.21.15.114:0/1711708562 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:10.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.093+0000 7f9ab1738700 1 -- 172.21.15.114:0/1711708562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf383e40 msgr2=0x563bcf507880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:10.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.093+0000 7f9ab1738700 1 --2- 172.21.15.114:0/1711708562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf383e40 0x563bcf507880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.093+0000 7f9ab1738700 1 -- 172.21.15.114:0/1711708562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf507dc0 msgr2=0x563bcf501a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:58:10.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.093+0000 7f9ab1738700 1 --2- 172.21.15.114:0/1711708562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf507dc0 0x563bcf501a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.094+0000 7f9ab1738700 1 -- 172.21.15.114:0/1711708562 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bcf3708d0 con 0x563bcf501f80 2024-05-27T07:58:10.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.096+0000 7f9ab1738700 1 --2- 172.21.15.114:0/1711708562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf501f80 0x563bcf502390 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9a940a61b0 tx=0x7f9a940a8780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90401b6680ca368 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:10.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.097+0000 7f9aa2ffd700 1 -- 172.21.15.114:0/1711708562 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a940a8f40 con 0x563bcf501f80 2024-05-27T07:58:10.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.097+0000 7f9aa2ffd700 1 -- 172.21.15.114:0/1711708562 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a940ad2f0 con 0x563bcf501f80 2024-05-27T07:58:10.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.098+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/1711708562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf501f80 msgr2=0x563bcf502390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:10.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.098+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/1711708562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf501f80 0x563bcf502390 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9a940a61b0 tx=0x7f9a940a8780 comp rx=0 tx=0).stop 2024-05-27T07:58:10.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.101+0000 7f9aabfff700 1 --2- 172.21.15.114:0/552637907 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a980b2c00 0x7f9a980b5110 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9aac1038e0 tx=0x7f9aa4009250 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:10.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.102+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/1711708562 shutdown_connections 2024-05-27T07:58:10.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.102+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/1711708562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf501f80 0x563bcf502390 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.102+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/1711708562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf383e40 0x563bcf507880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.102+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/1711708562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf507dc0 0x563bcf501a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.102+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/1711708562 >> 172.21.15.114:0/1711708562 conn(0x563bcf3deaf0 msgr2=0x563bcf4fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:10.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.108+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/1711708562 shutdown_connections 2024-05-27T07:58:10.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.109+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/1711708562 wait complete. 2024-05-27T07:58:10.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.110+0000 7f9ac1bb0b80 1 -- start start 2024-05-27T07:58:10.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.110+0000 7f9ac1bb0b80 1 Processor -- start 2024-05-27T07:58:10.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.111+0000 7f9ac1bb0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf383e40 0x563bcf50b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.111+0000 7f9ac1bb0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf501f80 0x563bcf50ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.111+0000 7f9ac1bb0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf507dc0 0x563bcf50bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.111+0000 7f9ac1bb0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563bcf326fa0 con 0x563bcf507dc0 2024-05-27T07:58:10.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.111+0000 7f9ac1bb0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563bcf3c27c0 con 0x563bcf383e40 2024-05-27T07:58:10.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.111+0000 7f9ac1bb0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563bcf50e8e0 con 0x563bcf501f80 2024-05-27T07:58:10.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.111+0000 7f9ac1bb0b80 10 client.0 _ll_register_callbacks cb 0x563bcf3bb920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:10.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.112+0000 7f9ab1738700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf507dc0 0x563bcf50bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.112+0000 7f9ab1738700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf507dc0 0x563bcf50bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45294/0 (socket says 172.21.15.114:45294) 2024-05-27T07:58:10.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.112+0000 7f9ab1738700 1 -- 172.21.15.114:0/2062389780 learned_addr learned my addr 172.21.15.114:0/2062389780 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:10.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.117+0000 7f9ab0f37700 1 --2- 172.21.15.114:0/2062389780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf383e40 0x563bcf50b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.117+0000 7f9aabfff700 1 --2- 172.21.15.114:0/2062389780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf501f80 0x563bcf50ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.117+0000 7f9ab1738700 1 -- 172.21.15.114:0/2062389780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf501f80 msgr2=0x563bcf50ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:10.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.117+0000 7f9ab1738700 1 --2- 172.21.15.114:0/2062389780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf501f80 0x563bcf50ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.117+0000 7f9ab1738700 1 -- 172.21.15.114:0/2062389780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf383e40 msgr2=0x563bcf50b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:10.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.117+0000 7f9ab1738700 1 --2- 172.21.15.114:0/2062389780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf383e40 0x563bcf50b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.117+0000 7f9ab1738700 1 -- 172.21.15.114:0/2062389780 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bcf3bf690 con 0x563bcf507dc0 2024-05-27T07:58:10.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.119+0000 7f9ab1738700 1 --2- 172.21.15.114:0/2062389780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf507dc0 0x563bcf50bf60 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9a940a8c40 tx=0x7f9a940df350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c208cdb217d0e2f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:10.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.120+0000 7f9a8bfff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:58:10.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.124+0000 7f9a8bfff700 1 -- 172.21.15.114:0/2062389780 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a940c9070 con 0x563bcf507dc0 2024-05-27T07:58:10.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.124+0000 7f9a8bfff700 1 -- 172.21.15.114:0/2062389780 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9642+0+0 (secure 0 0 0) 0x7f9a940b1b20 con 0x563bcf507dc0 2024-05-27T07:58:10.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.124+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2062389780 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a940ad2f0 con 0x563bcf507dc0 2024-05-27T07:58:10.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.128+0000 7f9a8bfff700 1 -- 172.21.15.114:0/2062389780 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f9a94011830 con 0x563bcf507dc0 2024-05-27T07:58:10.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.130+0000 7f9ac1bb0b80 10 client.19527 fetch_fsmap learned FSMap version 102 2024-05-27T07:58:10.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.130+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2062389780 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x563bcf5193b0 con 0x563bcf507dc0 2024-05-27T07:58:10.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.130+0000 7f9a8bfff700 1 -- 172.21.15.114:0/2062389780 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a940ad2f0 con 0x563bcf507dc0 2024-05-27T07:58:10.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.130+0000 7f9a8bfff700 1 -- 172.21.15.114:0/2062389780 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 102) ==== 4095+0+0 (secure 0 0 0) 0x7f9a940a7c10 con 0x563bcf507dc0 2024-05-27T07:58:10.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.133+0000 7f9ac1bb0b80 10 client.19527 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:58:10.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.133+0000 7f9ac1bb0b80 10 client.19527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:10.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.133+0000 7f9ac1bb0b80 20 client.19527 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:10.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.133+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2062389780 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563bcf51b880 0x563bcf51dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.133+0000 7f9ac1bb0b80 4 client.19527 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:58:10.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.133+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2062389780 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563bcf4cd9f0 con 0x563bcf51b880 2024-05-27T07:58:10.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.134+0000 7f9ab0f37700 1 --2- 172.21.15.114:0/2062389780 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563bcf51b880 0x563bcf51dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.141+0000 7f9a8bfff700 10 client.19527 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:10.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.141+0000 7f9ab0f37700 1 --2- 172.21.15.114:0/2062389780 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563bcf51b880 0x563bcf51dcc0 crc :-1 s=READY pgs=486 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12d57ce5309f9e2a server_cookie=fa7b8fc9a24fc176 in_seq=0 out_seq=0 2024-05-27T07:58:10.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.146+0000 7f9a8bfff700 1 -- 172.21.15.114:0/2062389780 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563bcf4cd9f0 con 0x563bcf51b880 2024-05-27T07:58:10.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.146+0000 7f9a8bfff700 10 client.19527 handle_command_reply: tid=0 2024-05-27T07:58:10.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.147+0000 7f9ac1bb0b80 1 client.19527 shutdown 2024-05-27T07:58:10.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.155+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2062389780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf507dc0 msgr2=0x563bcf50bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:10.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.155+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2062389780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf507dc0 0x563bcf50bf60 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9a940a8c40 tx=0x7f9a940df350 comp rx=0 tx=0).stop 2024-05-27T07:58:10.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.155+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2062389780 shutdown_connections 2024-05-27T07:58:10.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.155+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2062389780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf507dc0 0x563bcf50bf60 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.155+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2062389780 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563bcf51b880 0x563bcf51dcc0 crc :-1 s=READY pgs=486 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.155+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2062389780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf383e40 0x563bcf50b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.155+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2062389780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf501f80 0x563bcf50ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.155+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2062389780 >> 172.21.15.114:0/2062389780 conn(0x563bcf3deaf0 msgr2=0x563bcf500a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:10.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.155+0000 7f9a8bfff700 0 client.19527 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:10.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.155+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2062389780 shutdown_connections 2024-05-27T07:58:10.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.155+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2062389780 wait complete. 2024-05-27T07:58:10.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.157+0000 7f9ac1bb0b80 20 client.19527 trim_cache size 0 max 16384 2024-05-27T07:58:10.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.194+0000 7f9ac1bb0b80 1 Processor -- start 2024-05-27T07:58:10.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.195+0000 7f9ac1bb0b80 1 -- start start 2024-05-27T07:58:10.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.195+0000 7f9ac1bb0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf383e40 0x563bcf4fc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.195+0000 7f9ac1bb0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf501f80 0x563bcf50d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.195+0000 7f9ac1bb0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf507dc0 0x563bcf50bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.195+0000 7f9ac1bb0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c00ed70 con 0x563bcf383e40 2024-05-27T07:58:10.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.195+0000 7f9ac1bb0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c00f440 con 0x563bcf501f80 2024-05-27T07:58:10.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.195+0000 7f9ac1bb0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c00dcf0 con 0x563bcf507dc0 2024-05-27T07:58:10.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.196+0000 7f9ab1738700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf507dc0 0x563bcf50bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.197+0000 7f9aabfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf501f80 0x563bcf50d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.200+0000 7f9ab1738700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf507dc0 0x563bcf50bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:48318/0 (socket says 172.21.15.114:48318) 2024-05-27T07:58:10.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.200+0000 7f9ab1738700 1 -- 172.21.15.114:0/2641445859 learned_addr learned my addr 172.21.15.114:0/2641445859 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:10.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.204+0000 7f9ab0f37700 1 --2- 172.21.15.114:0/2641445859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf383e40 0x563bcf4fc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.204+0000 7f9ab1738700 1 -- 172.21.15.114:0/2641445859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf501f80 msgr2=0x563bcf50d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:10.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.204+0000 7f9ab1738700 1 --2- 172.21.15.114:0/2641445859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf501f80 0x563bcf50d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.204+0000 7f9ab1738700 1 -- 172.21.15.114:0/2641445859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf383e40 msgr2=0x563bcf4fc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:10.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.204+0000 7f9ab1738700 1 --2- 172.21.15.114:0/2641445859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf383e40 0x563bcf4fc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.204+0000 7f9ab1738700 1 -- 172.21.15.114:0/2641445859 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bcf50c4f0 con 0x563bcf507dc0 2024-05-27T07:58:10.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.204+0000 7f9ab1738700 1 --2- 172.21.15.114:0/2641445859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf507dc0 0x563bcf50bda0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9a940a4fc0 tx=0x7f9a940107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d663a7a8f79d5e40 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:10.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.207+0000 7f9aa27fc700 1 -- 172.21.15.114:0/2641445859 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a940c9070 con 0x563bcf507dc0 2024-05-27T07:58:10.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.207+0000 7f9aa27fc700 1 -- 172.21.15.114:0/2641445859 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a940b1b20 con 0x563bcf507dc0 2024-05-27T07:58:10.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.207+0000 7f9aa27fc700 1 -- 172.21.15.114:0/2641445859 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a940a7c10 con 0x563bcf507dc0 2024-05-27T07:58:10.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.209+0000 7f9ab0f37700 1 --2- 172.21.15.114:0/2641445859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf383e40 0x563bcf4fc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:58:10.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.209+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2641445859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf507dc0 msgr2=0x563bcf50bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:10.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.209+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2641445859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf507dc0 0x563bcf50bda0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9a940a4fc0 tx=0x7f9a940107a0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.210+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2641445859 shutdown_connections 2024-05-27T07:58:10.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.210+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2641445859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf507dc0 0x563bcf50bda0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.210+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2641445859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf383e40 0x563bcf4fc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.210+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2641445859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf501f80 0x563bcf50d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.210+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2641445859 >> 172.21.15.114:0/2641445859 conn(0x563bcf3deaf0 msgr2=0x563bcf505830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:10.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.210+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2641445859 shutdown_connections 2024-05-27T07:58:10.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.210+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2641445859 wait complete. 2024-05-27T07:58:10.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.211+0000 7f9ac1bb0b80 1 -- start start 2024-05-27T07:58:10.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.211+0000 7f9ac1bb0b80 1 Processor -- start 2024-05-27T07:58:10.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.212+0000 7f9ac1bb0b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf383e40 0x563bcf513c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.212+0000 7f9ac1bb0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf501f80 0x563bcf514150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.212+0000 7f9ac1bb0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf507dc0 0x563bcf522510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.212+0000 7f9ac1bb0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c00d620 con 0x563bcf501f80 2024-05-27T07:58:10.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.212+0000 7f9ac1bb0b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c00cfd0 con 0x563bcf383e40 2024-05-27T07:58:10.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.212+0000 7f9ac1bb0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563bcf502720 con 0x563bcf507dc0 2024-05-27T07:58:10.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.212+0000 7f9ab0f37700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf383e40 0x563bcf513c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.212+0000 7f9ab0f37700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf383e40 0x563bcf513c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:35910/0 (socket says 172.21.15.114:35910) 2024-05-27T07:58:10.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.212+0000 7f9ab0f37700 1 -- 172.21.15.114:0/2755581105 learned_addr learned my addr 172.21.15.114:0/2755581105 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:10.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.212+0000 7f9ac1bb0b80 10 client.0 _ll_register_callbacks cb 0x563bcf3bb920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:10.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.213+0000 7f9aabfff700 1 --2- 172.21.15.114:0/2755581105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf501f80 0x563bcf514150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.213+0000 7f9ab1738700 1 --2- 172.21.15.114:0/2755581105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf507dc0 0x563bcf522510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.213+0000 7f9aabfff700 1 -- 172.21.15.114:0/2755581105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf507dc0 msgr2=0x563bcf522510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:10.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.213+0000 7f9aabfff700 1 --2- 172.21.15.114:0/2755581105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf507dc0 0x563bcf522510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.213+0000 7f9aabfff700 1 -- 172.21.15.114:0/2755581105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf383e40 msgr2=0x563bcf513c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:10.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.213+0000 7f9aabfff700 1 --2- 172.21.15.114:0/2755581105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf383e40 0x563bcf513c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.214+0000 7f9aabfff700 1 -- 172.21.15.114:0/2755581105 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bcf522a50 con 0x563bcf501f80 2024-05-27T07:58:10.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.214+0000 7f9aabfff700 1 --2- 172.21.15.114:0/2755581105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf501f80 0x563bcf514150 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x563bcf4fe6c0 tx=0x7f9aa4004b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c8a41417f9679e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:10.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.215+0000 7f9a8bfff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:58:10.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.217+0000 7f9a8bfff700 1 -- 172.21.15.114:0/2755581105 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa400ce50 con 0x563bcf501f80 2024-05-27T07:58:10.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.217+0000 7f9a8bfff700 1 -- 172.21.15.114:0/2755581105 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9642+0+0 (secure 0 0 0) 0x7f9aa4019800 con 0x563bcf501f80 2024-05-27T07:58:10.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.217+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2755581105 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a940b1b20 con 0x563bcf501f80 2024-05-27T07:58:10.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.218+0000 7f9a8bfff700 1 -- 172.21.15.114:0/2755581105 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f9aa4019a00 con 0x563bcf501f80 2024-05-27T07:58:10.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.219+0000 7f9ac1bb0b80 10 client.19530 fetch_fsmap learned FSMap version 102 2024-05-27T07:58:10.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.219+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2755581105 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x563bcf585a30 con 0x563bcf501f80 2024-05-27T07:58:10.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.220+0000 7f9a8bfff700 1 -- 172.21.15.114:0/2755581105 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a940b1b20 con 0x563bcf501f80 2024-05-27T07:58:10.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.220+0000 7f9a8bfff700 1 -- 172.21.15.114:0/2755581105 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 102) ==== 4095+0+0 (secure 0 0 0) 0x7f9aa4024420 con 0x563bcf501f80 2024-05-27T07:58:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.222+0000 7f9ac1bb0b80 10 client.19530 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:58:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.222+0000 7f9ac1bb0b80 10 client.19530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.222+0000 7f9ac1bb0b80 20 client.19530 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.222+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2755581105 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563bcf5860e0 0x563bcf588500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.222+0000 7f9ac1bb0b80 4 client.19530 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:58:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.222+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2755581105 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563bcf432be0 con 0x563bcf5860e0 2024-05-27T07:58:10.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.223+0000 7f9ab0f37700 1 --2- 172.21.15.114:0/2755581105 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563bcf5860e0 0x563bcf588500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:10.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:09 smithi163 ceph-mon[36451]: pgmap v631: 129 pgs: 19 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 61 active+clean; 359 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.4 MiB/s wr, 594 op/s 2024-05-27T07:58:10.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:09 smithi186 ceph-mon[36647]: pgmap v631: 129 pgs: 19 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 61 active+clean; 359 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.4 MiB/s wr, 594 op/s 2024-05-27T07:58:10.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.228+0000 7f9a8bfff700 10 client.19530 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:10.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.228+0000 7f9ab0f37700 1 --2- 172.21.15.114:0/2755581105 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563bcf5860e0 0x563bcf588500 crc :-1 s=READY pgs=487 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ea5b7ce88f72476 server_cookie=b1f4eba1456284ea in_seq=0 out_seq=0 2024-05-27T07:58:10.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.228+0000 7f9a8bfff700 1 -- 172.21.15.114:0/2755581105 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563bcf432be0 con 0x563bcf5860e0 2024-05-27T07:58:10.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.228+0000 7f9a8bfff700 10 client.19530 handle_command_reply: tid=0 2024-05-27T07:58:10.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.228+0000 7f9ac1bb0b80 1 client.19530 shutdown 2024-05-27T07:58:10.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.229+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2755581105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf501f80 msgr2=0x563bcf514150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:10.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.229+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2755581105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf501f80 0x563bcf514150 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x563bcf4fe6c0 tx=0x7f9aa4004b60 comp rx=0 tx=0).stop 2024-05-27T07:58:10.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.229+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2755581105 shutdown_connections 2024-05-27T07:58:10.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.229+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2755581105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bcf507dc0 0x563bcf522510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.229+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2755581105 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563bcf5860e0 0x563bcf588500 crc :-1 s=READY pgs=487 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.229+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2755581105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bcf383e40 0x563bcf513c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.229+0000 7f9ac1bb0b80 1 --2- 172.21.15.114:0/2755581105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bcf501f80 0x563bcf514150 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.229+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2755581105 >> 172.21.15.114:0/2755581105 conn(0x563bcf3deaf0 msgr2=0x563bcf50d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:10.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.229+0000 7f9a8bfff700 0 client.19530 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:10.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.230+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2755581105 shutdown_connections 2024-05-27T07:58:10.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.230+0000 7f9ac1bb0b80 1 -- 172.21.15.114:0/2755581105 wait complete. 2024-05-27T07:58:10.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.230+0000 7f9ac1bb0b80 20 client.19530 trim_cache size 0 max 16384 2024-05-27T07:58:10.234 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:58:10.234 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:58:10.234 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:58:10.234 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:58:10.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.233+0000 7f9ab1f39700 1 -- 172.21.15.114:0/552637907 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a980b2c00 msgr2=0x7f9a980b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:10.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.233+0000 7f9ab1f39700 1 --2- 172.21.15.114:0/552637907 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a980b2c00 0x7f9a980b5110 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9aac1038e0 tx=0x7f9aa4009250 comp rx=0 tx=0).stop 2024-05-27T07:58:10.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.233+0000 7f9ab1f39700 1 -- 172.21.15.114:0/552637907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac1084c0 msgr2=0x7f9aac06e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:10.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.233+0000 7f9ab1f39700 1 --2- 172.21.15.114:0/552637907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac1084c0 0x7f9aac06e9e0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400b550 tx=0x7f9a9400b910 comp rx=0 tx=0).stop 2024-05-27T07:58:10.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.234+0000 7f9ab1f39700 1 -- 172.21.15.114:0/552637907 shutdown_connections 2024-05-27T07:58:10.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.234+0000 7f9ab1f39700 1 --2- 172.21.15.114:0/552637907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac1084c0 0x7f9aac06e9e0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.234+0000 7f9ab1f39700 1 --2- 172.21.15.114:0/552637907 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f9a980b2c00 0x7f9a980b5110 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.234+0000 7f9ab1f39700 1 --2- 172.21.15.114:0/552637907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9aac103070 0x7f9aac075310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.234+0000 7f9ab1f39700 1 --2- 172.21.15.114:0/552637907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9aac1039c0 0x7f9aac06e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:10.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.234+0000 7f9ab1f39700 1 -- 172.21.15.114:0/552637907 >> 172.21.15.114:0/552637907 conn(0x7f9aac078ab0 msgr2=0x7f9aac105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:10.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.234+0000 7f9ab1f39700 1 -- 172.21.15.114:0/552637907 shutdown_connections 2024-05-27T07:58:10.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:10.234+0000 7f9ab1f39700 1 -- 172.21.15.114:0/552637907 wait complete. 2024-05-27T07:58:10.613 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63275 12.53 MB/sec execute 142 sec latency 1730.352 ms 2024-05-27T07:58:11.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:10 smithi114 ceph-mon[32038]: pgmap v632: 129 pgs: 19 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 61 active+clean; 359 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.3 MiB/s wr, 593 op/s 2024-05-27T07:58:11.178 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:58:11.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:58:11.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:10 smithi163 ceph-mon[36451]: pgmap v632: 129 pgs: 19 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 61 active+clean; 359 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.3 MiB/s wr, 593 op/s 2024-05-27T07:58:11.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:10 smithi186 ceph-mon[36647]: pgmap v632: 129 pgs: 19 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 61 active+clean; 359 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.3 MiB/s wr, 593 op/s 2024-05-27T07:58:11.613 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63284 12.44 MB/sec execute 143 sec latency 1604.218 ms 2024-05-27T07:58:11.692 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:58:12.613 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63315 12.36 MB/sec execute 144 sec latency 2604.260 ms 2024-05-27T07:58:12.867 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:12 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:58:12.867 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:12 smithi114 systemd[1]: Stopped Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:58:12.867 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:12 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Consumed 3.518s CPU time. 2024-05-27T07:58:12.867 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:12 smithi114 systemd[1]: Starting Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:58:13.613 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63377 12.31 MB/sec execute 145 sec latency 3604.299 ms 2024-05-27T07:58:14.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:13 smithi114 ceph-mon[32038]: pgmap v633: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 353 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 3.0 MiB/s wr, 533 op/s 2024-05-27T07:58:14.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:13 smithi163 ceph-mon[36451]: pgmap v633: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 353 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 3.0 MiB/s wr, 533 op/s 2024-05-27T07:58:14.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:13 smithi186 ceph-mon[36647]: pgmap v633: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 353 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 3.0 MiB/s wr, 533 op/s 2024-05-27T07:58:14.613 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63395 12.23 MB/sec execute 146 sec latency 4138.456 ms 2024-05-27T07:58:15.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.063+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/65323760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd141029b0 msgr2=0x7fbd14102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:15.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.063+0000 7fbd1ac0a700 1 --2- 172.21.15.114:0/65323760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd141029b0 0x7fbd14102dc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800d9e0 tx=0x7fbd0800dcf0 comp rx=0 tx=0).stop 2024-05-27T07:58:15.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.070+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/65323760 shutdown_connections 2024-05-27T07:58:15.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.071+0000 7fbd1ac0a700 1 --2- 172.21.15.114:0/65323760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd14103300 0x7fbd1410f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:15.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.071+0000 7fbd1ac0a700 1 --2- 172.21.15.114:0/65323760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd141089e0 0x7fbd14108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:15.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.071+0000 7fbd1ac0a700 1 --2- 172.21.15.114:0/65323760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd141029b0 0x7fbd14102dc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:15.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.071+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/65323760 >> 172.21.15.114:0/65323760 conn(0x7fbd140fe6e0 msgr2=0x7fbd14100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:15.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.073+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/65323760 shutdown_connections 2024-05-27T07:58:15.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.077+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/65323760 wait complete. 2024-05-27T07:58:15.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.078+0000 7fbd1ac0a700 1 Processor -- start 2024-05-27T07:58:15.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.080+0000 7fbd1ac0a700 1 -- start start 2024-05-27T07:58:15.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.080+0000 7fbd1ac0a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd141029b0 0x7fbd141a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:15.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.080+0000 7fbd1ac0a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd14103300 0x7fbd141a1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:15.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.080+0000 7fbd1ac0a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd141089e0 0x7fbd141a5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:15.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.081+0000 7fbd1ac0a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1411abe0 con 0x7fbd141029b0 2024-05-27T07:58:15.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.081+0000 7fbd1ac0a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1411aa60 con 0x7fbd141089e0 2024-05-27T07:58:15.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.081+0000 7fbd1ac0a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1411ad60 con 0x7fbd14103300 2024-05-27T07:58:15.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.082+0000 7fbd13fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd14103300 0x7fbd141a1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:15.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.085+0000 7fbd189a6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd141029b0 0x7fbd141a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:15.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.085+0000 7fbd189a6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd141029b0 0x7fbd141a1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36882/0 (socket says 172.21.15.114:36882) 2024-05-27T07:58:15.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.085+0000 7fbd189a6700 1 -- 172.21.15.114:0/4222351653 learned_addr learned my addr 172.21.15.114:0/4222351653 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.086+0000 7fbd13fff700 1 -- 172.21.15.114:0/4222351653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd141089e0 msgr2=0x7fbd141a5e60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:58:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.086+0000 7fbd13fff700 1 --2- 172.21.15.114:0/4222351653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd141089e0 0x7fbd141a5e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.086+0000 7fbd13fff700 1 -- 172.21.15.114:0/4222351653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd141029b0 msgr2=0x7fbd141a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.086+0000 7fbd13fff700 1 --2- 172.21.15.114:0/4222351653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd141029b0 0x7fbd141a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.086+0000 7fbd13fff700 1 -- 172.21.15.114:0/4222351653 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd141a6500 con 0x7fbd14103300 2024-05-27T07:58:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.089+0000 7fbd13fff700 1 --2- 172.21.15.114:0/4222351653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd14103300 0x7fbd141a1bd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbd08000c00 tx=0x7fbd08005280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:15.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.091+0000 7fbd11ffb700 1 -- 172.21.15.114:0/4222351653 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0801f070 con 0x7fbd14103300 2024-05-27T07:58:15.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.091+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/4222351653 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd141a6790 con 0x7fbd14103300 2024-05-27T07:58:15.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.092+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/4222351653 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd141a6c70 con 0x7fbd14103300 2024-05-27T07:58:15.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.092+0000 7fbd11ffb700 1 -- 172.21.15.114:0/4222351653 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd080056b0 con 0x7fbd14103300 2024-05-27T07:58:15.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.093+0000 7fbd11ffb700 1 -- 172.21.15.114:0/4222351653 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0801a6f0 con 0x7fbd14103300 2024-05-27T07:58:15.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.098+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/4222351653 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd00005330 con 0x7fbd14103300 2024-05-27T07:58:15.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.103+0000 7fbd11ffb700 1 -- 172.21.15.114:0/4222351653 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fbd08051d10 con 0x7fbd14103300 2024-05-27T07:58:15.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.104+0000 7fbd11ffb700 1 --2- 172.21.15.114:0/4222351653 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbcfc0b29d0 0x7fbcfc0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:15.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.104+0000 7fbd11ffb700 1 -- 172.21.15.114:0/4222351653 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 9642+0+0 (secure 0 0 0) 0x7fbd08013070 con 0x7fbd14103300 2024-05-27T07:58:15.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.104+0000 7fbd189a6700 1 --2- 172.21.15.114:0/4222351653 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbcfc0b29d0 0x7fbcfc0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:15.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.105+0000 7fbd189a6700 1 --2- 172.21.15.114:0/4222351653 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbcfc0b29d0 0x7fbcfc0b4ee0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009a90 tx=0x7fbd04009210 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:15.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.110+0000 7fbd11ffb700 1 -- 172.21.15.114:0/4222351653 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbd08029020 con 0x7fbd14103300 2024-05-27T07:58:15.112 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:15 smithi114 podman[130598]: 2024-05-27 07:58:15.001495418 +0000 UTC m=+0.252870335 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:58:15.387 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:15 smithi114 podman[130598]: 2024-05-27 07:58:15.112543063 +0000 UTC m=+0.363917950 container create 1ced8c6a38e3b0e778b9246070d06690ad9a6e8020c6770433c017aaafcd57ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-27T07:58:15.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.597+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/4222351653 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd00005620 con 0x7fbd14103300 2024-05-27T07:58:15.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.602+0000 7fbd11ffb700 1 -- 172.21.15.114:0/4222351653 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+6036 (secure 0 0 0) 0x7fbd08017070 con 0x7fbd14103300 2024-05-27T07:58:15.604 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:58:15.604 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":19494,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/584614487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":584614487},{"type":"v1","addr":"172.21.15.114:6841","nonce":584614487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":102,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:57:43.826922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_28781":{"gid":28781,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6839/2945643320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2945643320},{"type":"v1","addr":"172.21.15.114:6839","nonce":2945643320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:58:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.608+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/4222351653 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbcfc0b29d0 msgr2=0x7fbcfc0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.608+0000 7fbd1ac0a700 1 --2- 172.21.15.114:0/4222351653 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbcfc0b29d0 0x7fbcfc0b4ee0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009a90 tx=0x7fbd04009210 comp rx=0 tx=0).stop 2024-05-27T07:58:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.608+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/4222351653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd14103300 msgr2=0x7fbd141a1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.608+0000 7fbd1ac0a700 1 --2- 172.21.15.114:0/4222351653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd14103300 0x7fbd141a1bd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbd08000c00 tx=0x7fbd08005280 comp rx=0 tx=0).stop 2024-05-27T07:58:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.608+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/4222351653 shutdown_connections 2024-05-27T07:58:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.608+0000 7fbd1ac0a700 1 --2- 172.21.15.114:0/4222351653 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbcfc0b29d0 0x7fbcfc0b4ee0 secure :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009a90 tx=0x7fbd04009210 comp rx=0 tx=0).stop 2024-05-27T07:58:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.608+0000 7fbd1ac0a700 1 --2- 172.21.15.114:0/4222351653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd141089e0 0x7fbd141a5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:15.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.608+0000 7fbd1ac0a700 1 --2- 172.21.15.114:0/4222351653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd141029b0 0x7fbd141a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:15.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.608+0000 7fbd1ac0a700 1 --2- 172.21.15.114:0/4222351653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd14103300 0x7fbd141a1bd0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:15.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.608+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/4222351653 >> 172.21.15.114:0/4222351653 conn(0x7fbd140fe6e0 msgr2=0x7fbd14100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:15.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.609+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/4222351653 shutdown_connections 2024-05-27T07:58:15.613 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63473 12.18 MB/sec execute 147 sec latency 2519.472 ms 2024-05-27T07:58:15.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:15.612+0000 7fbd1ac0a700 1 -- 172.21.15.114:0/4222351653 wait complete. 2024-05-27T07:58:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 102 2024-05-27T07:58:15.643 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:15 smithi114 podman[130598]: 2024-05-27 07:58:15.386937225 +0000 UTC m=+0.638312129 container init 1ced8c6a38e3b0e778b9246070d06690ad9a6e8020c6770433c017aaafcd57ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-27T07:58:15.643 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:15 smithi114 podman[130598]: 2024-05-27 07:58:15.395911048 +0000 UTC m=+0.647285937 container start 1ced8c6a38e3b0e778b9246070d06690ad9a6e8020c6770433c017aaafcd57ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-27T07:58:15.643 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: starting mds.g at 2024-05-27T07:58:15.643 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:15 smithi114 bash[130598]: 1ced8c6a38e3b0e778b9246070d06690ad9a6e8020c6770433c017aaafcd57ba 2024-05-27T07:58:15.643 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:15 smithi114 systemd[1]: Started Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:58:16.033 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:15 smithi114 ceph-mon[32038]: pgmap v634: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 354 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 3.1 MiB/s wr, 469 op/s 2024-05-27T07:58:16.033 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:15 smithi114 ceph-mon[32038]: daemon mds.g restarted 2024-05-27T07:58:16.033 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:15 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/4222351653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:58:16.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:15 smithi163 ceph-mon[36451]: pgmap v634: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 354 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 3.1 MiB/s wr, 469 op/s 2024-05-27T07:58:16.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:15 smithi163 ceph-mon[36451]: daemon mds.g restarted 2024-05-27T07:58:16.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:15 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/4222351653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:58:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:15 smithi186 ceph-mon[36647]: pgmap v634: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 354 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 3.1 MiB/s wr, 469 op/s 2024-05-27T07:58:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:15 smithi186 ceph-mon[36647]: daemon mds.g restarted 2024-05-27T07:58:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:15 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/4222351653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:58:16.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f48425da700 time 2024-05-27T07:58:16.167139+0000 2024-05-27T07:58:16.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:58:16.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:16.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4854e1fba6] 2024-05-27T07:58:16.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4854e1fdd1] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591b77e623d] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5591b77d1971] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 5: (SnapRealm::check_cache() const+0x13e) [0x5591b77cffa0] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591b77d0651] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591b7765347] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591b746187e] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5591b7466f2f] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5591b7471969] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 11: (MDLog::_replay_thread()+0x1f38) [0x5591b780a48c] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5591b78139cb] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 13: (Thread::entry_wrapper()+0x43) [0x7f4854dee72d] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4854dee749] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f48539921ca] 2024-05-27T07:58:16.309 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 16: clone() 2024-05-27T07:58:16.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.170+0000 7f48425da700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f48425da700 time 2024-05-27T07:58:16.167139+0000 2024-05-27T07:58:16.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:58:16.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:16.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4854e1fba6] 2024-05-27T07:58:16.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4854e1fdd1] 2024-05-27T07:58:16.310 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591b77e623d] 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5591b77d1971] 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 5: (SnapRealm::check_cache() const+0x13e) [0x5591b77cffa0] 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591b77d0651] 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591b7765347] 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591b746187e] 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5591b7466f2f] 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5591b7471969] 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 11: (MDLog::_replay_thread()+0x1f38) [0x5591b780a48c] 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5591b78139cb] 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 13: (Thread::entry_wrapper()+0x43) [0x7f4854dee72d] 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4854dee749] 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f48539921ca] 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 16: clone() 2024-05-27T07:58:16.311 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: *** Caught signal (Aborted) ** 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: in thread 7f48425da700 thread_name:md_log_replay 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f485399cd20] 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2: gsignal() 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 3: abort() 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4854e1fced] 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4854e1fdd1] 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591b77e623d] 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5591b77d1971] 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 8: (SnapRealm::check_cache() const+0x13e) [0x5591b77cffa0] 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591b77d0651] 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591b7765347] 2024-05-27T07:58:16.312 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591b746187e] 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5591b7466f2f] 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5591b7471969] 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 14: (MDLog::_replay_thread()+0x1f38) [0x5591b780a48c] 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5591b78139cb] 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 16: (Thread::entry_wrapper()+0x43) [0x7f4854dee72d] 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4854dee749] 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f48539921ca] 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 19: clone() 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.173+0000 7f48425da700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: in thread 7f48425da700 thread_name:md_log_replay 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f485399cd20] 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2: gsignal() 2024-05-27T07:58:16.313 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 3: abort() 2024-05-27T07:58:16.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4854e1fced] 2024-05-27T07:58:16.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4854e1fdd1] 2024-05-27T07:58:16.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591b77e623d] 2024-05-27T07:58:16.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5591b77d1971] 2024-05-27T07:58:16.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 8: (SnapRealm::check_cache() const+0x13e) [0x5591b77cffa0] 2024-05-27T07:58:16.314 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591b77d0651] 2024-05-27T07:58:16.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591b7765347] 2024-05-27T07:58:16.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591b746187e] 2024-05-27T07:58:16.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5591b7466f2f] 2024-05-27T07:58:16.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5591b7471969] 2024-05-27T07:58:16.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 14: (MDLog::_replay_thread()+0x1f38) [0x5591b780a48c] 2024-05-27T07:58:16.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5591b78139cb] 2024-05-27T07:58:16.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 16: (Thread::entry_wrapper()+0x43) [0x7f4854dee72d] 2024-05-27T07:58:16.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4854dee749] 2024-05-27T07:58:16.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f48539921ca] 2024-05-27T07:58:16.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 19: clone() 2024-05-27T07:58:16.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:58:16.315 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: -1> 2024-05-27T07:58:16.170+0000 7f48425da700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f48425da700 time 2024-05-27T07:58:16.167139+0000 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4854e1fba6] 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4854e1fdd1] 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591b77e623d] 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5591b77d1971] 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 5: (SnapRealm::check_cache() const+0x13e) [0x5591b77cffa0] 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591b77d0651] 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591b7765347] 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591b746187e] 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5591b7466f2f] 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5591b7471969] 2024-05-27T07:58:16.316 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 11: (MDLog::_replay_thread()+0x1f38) [0x5591b780a48c] 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5591b78139cb] 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 13: (Thread::entry_wrapper()+0x43) [0x7f4854dee72d] 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4854dee749] 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f48539921ca] 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 16: clone() 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 0> 2024-05-27T07:58:16.173+0000 7f48425da700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: in thread 7f48425da700 thread_name:md_log_replay 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f485399cd20] 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2: gsignal() 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 3: abort() 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4854e1fced] 2024-05-27T07:58:16.317 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4854e1fdd1] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591b77e623d] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5591b77d1971] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 8: (SnapRealm::check_cache() const+0x13e) [0x5591b77cffa0] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591b77d0651] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591b7765347] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591b746187e] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5591b7466f2f] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5591b7471969] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 14: (MDLog::_replay_thread()+0x1f38) [0x5591b780a48c] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5591b78139cb] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 16: (Thread::entry_wrapper()+0x43) [0x7f4854dee72d] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4854dee749] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f48539921ca] 2024-05-27T07:58:16.318 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 19: clone() 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: -1> 2024-05-27T07:58:16.170+0000 7f48425da700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f48425da700 time 2024-05-27T07:58:16.167139+0000 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4854e1fba6] 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4854e1fdd1] 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591b77e623d] 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5591b77d1971] 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 5: (SnapRealm::check_cache() const+0x13e) [0x5591b77cffa0] 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591b77d0651] 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591b7765347] 2024-05-27T07:58:16.319 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591b746187e] 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5591b7466f2f] 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5591b7471969] 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 11: (MDLog::_replay_thread()+0x1f38) [0x5591b780a48c] 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5591b78139cb] 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 13: (Thread::entry_wrapper()+0x43) [0x7f4854dee72d] 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4854dee749] 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f48539921ca] 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 16: clone() 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 0> 2024-05-27T07:58:16.173+0000 7f48425da700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: in thread 7f48425da700 thread_name:md_log_replay 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f485399cd20] 2024-05-27T07:58:16.320 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2: gsignal() 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 3: abort() 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4854e1fced] 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4854e1fdd1] 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591b77e623d] 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5591b77d1971] 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 8: (SnapRealm::check_cache() const+0x13e) [0x5591b77cffa0] 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591b77d0651] 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591b7765347] 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591b746187e] 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5591b7466f2f] 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5591b7471969] 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 14: (MDLog::_replay_thread()+0x1f38) [0x5591b780a48c] 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5591b78139cb] 2024-05-27T07:58:16.321 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 16: (Thread::entry_wrapper()+0x43) [0x7f4854dee72d] 2024-05-27T07:58:16.322 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4854dee749] 2024-05-27T07:58:16.322 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f48539921ca] 2024-05-27T07:58:16.322 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 19: clone() 2024-05-27T07:58:16.322 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:58:16.322 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[130633]: 2024-05-27T07:58:16.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:58:16.613 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63592 12.11 MB/sec execute 148 sec latency 2219.675 ms 2024-05-27T07:58:16.775 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:16 smithi114 podman[130681]: 2024-05-27 07:58:16.528700869 +0000 UTC m=+0.280865888 container died 1ced8c6a38e3b0e778b9246070d06690ad9a6e8020c6770433c017aaafcd57ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:58:17.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:16 smithi114 ceph-mon[32038]: osdmap e156: 12 total, 12 up, 12 in 2024-05-27T07:58:17.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:16 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6838/3081028203,v1:172.21.15.114:6839/3081028203] up:boot 2024-05-27T07:58:17.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:16 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:58:17.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:16 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:58:17.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:16 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:58:17.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:16 smithi163 ceph-mon[36451]: osdmap e156: 12 total, 12 up, 12 in 2024-05-27T07:58:17.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:16 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6838/3081028203,v1:172.21.15.114:6839/3081028203] up:boot 2024-05-27T07:58:17.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:16 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:58:17.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:16 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:58:17.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:16 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:58:17.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:16 smithi186 ceph-mon[36647]: osdmap e156: 12 total, 12 up, 12 in 2024-05-27T07:58:17.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:16 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6838/3081028203,v1:172.21.15.114:6839/3081028203] up:boot 2024-05-27T07:58:17.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:16 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:58:17.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:16 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:58:17.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:16 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:58:17.573 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:58:17.615 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63698 12.06 MB/sec execute 149 sec latency 1740.720 ms 2024-05-27T07:58:17.733 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:17 smithi114 podman[130681]: 2024-05-27 07:58:17.326334619 +0000 UTC m=+1.078499638 container remove 1ced8c6a38e3b0e778b9246070d06690ad9a6e8020c6770433c017aaafcd57ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-05-27T07:58:17.734 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:17 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:58:17.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:17 smithi114 ceph-mon[32038]: pgmap v636: 129 pgs: 15 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 73 active+clean; 361 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 3.9 MiB/s wr, 403 op/s 2024-05-27T07:58:18.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:17 smithi163 ceph-mon[36451]: pgmap v636: 129 pgs: 15 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 73 active+clean; 361 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 3.9 MiB/s wr, 403 op/s 2024-05-27T07:58:18.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:17 smithi186 ceph-mon[36647]: pgmap v636: 129 pgs: 15 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 73 active+clean; 361 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 3.9 MiB/s wr, 403 op/s 2024-05-27T07:58:18.615 INFO:tasks.workunit.client.0.smithi114.stdout: 10 63822 12.03 MB/sec execute 150 sec latency 1153.384 ms 2024-05-27T07:58:18.807 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:18 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:58:18.807 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:18 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.210s CPU time. 2024-05-27T07:58:19.615 INFO:tasks.workunit.client.0.smithi114.stdout: 10 64042 12.02 MB/sec execute 151 sec latency 1610.065 ms 2024-05-27T07:58:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.933+0000 7f5ca1884640 1 -- 172.21.15.114:0/2945929834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c9c1075d0 msgr2=0x7f5c9c1079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.933+0000 7f5ca1884640 1 --2- 172.21.15.114:0/2945929834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c9c1075d0 0x7f5c9c1079e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c009900 tx=0x7f5c8c02a420 comp rx=0 tx=0).stop 2024-05-27T07:58:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.936+0000 7f5ca1884640 1 -- 172.21.15.114:0/2945929834 shutdown_connections 2024-05-27T07:58:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.936+0000 7f5ca1884640 1 --2- 172.21.15.114:0/2945929834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c9c108000 0x7f5c9c108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.936+0000 7f5ca1884640 1 --2- 172.21.15.114:0/2945929834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c9c1075d0 0x7f5c9c1079e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.936+0000 7f5ca1884640 1 --2- 172.21.15.114:0/2945929834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c9c102b80 0x7f5c9c102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.936+0000 7f5ca1884640 1 -- 172.21.15.114:0/2945929834 >> 172.21.15.114:0/2945929834 conn(0x7f5c9c0782d0 msgr2=0x7f5c9c078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.937+0000 7f5ca1884640 1 -- 172.21.15.114:0/2945929834 shutdown_connections 2024-05-27T07:58:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.937+0000 7f5ca1884640 1 -- 172.21.15.114:0/2945929834 wait complete. 2024-05-27T07:58:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.938+0000 7f5ca1884640 1 Processor -- start 2024-05-27T07:58:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.938+0000 7f5ca1884640 1 -- start start 2024-05-27T07:58:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.938+0000 7f5ca1884640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c9c102b80 0x7f5c9c10c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.938+0000 7f5ca1884640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c9c1075d0 0x7f5c9c10cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.938+0000 7f5ca1884640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c9c108000 0x7f5c9c073ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.938+0000 7f5ca1884640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c11a4a0 con 0x7f5c9c102b80 2024-05-27T07:58:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.938+0000 7f5ca1884640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c11a320 con 0x7f5c9c1075d0 2024-05-27T07:58:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.938+0000 7f5ca1884640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c11a620 con 0x7f5c9c108000 2024-05-27T07:58:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.941+0000 7f5c9affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c9c102b80 0x7f5c9c10c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.941+0000 7f5c9affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c9c102b80 0x7f5c9c10c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36890/0 (socket says 172.21.15.114:36890) 2024-05-27T07:58:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.941+0000 7f5c9affd640 1 -- 172.21.15.114:0/106195047 learned_addr learned my addr 172.21.15.114:0/106195047 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.941+0000 7f5c9b7fe640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c9c108000 0x7f5c9c073ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.941+0000 7f5c9a7fc640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c9c1075d0 0x7f5c9c10cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.941+0000 7f5c9affd640 1 -- 172.21.15.114:0/106195047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c9c108000 msgr2=0x7f5c9c073ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.941+0000 7f5c9affd640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c9c108000 0x7f5c9c073ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.941+0000 7f5c9affd640 1 -- 172.21.15.114:0/106195047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c9c1075d0 msgr2=0x7f5c9c10cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.941+0000 7f5c9affd640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c9c1075d0 0x7f5c9c10cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.941+0000 7f5c9affd640 1 -- 172.21.15.114:0/106195047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9c0742d0 con 0x7f5c9c102b80 2024-05-27T07:58:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.942+0000 7f5c9affd640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c9c102b80 0x7f5c9c10c580 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f5c9400cce0 tx=0x7f5c9400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.943+0000 7f5ca0882640 1 -- 172.21.15.114:0/106195047 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c94004b20 con 0x7f5c9c102b80 2024-05-27T07:58:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.943+0000 7f5ca0882640 1 -- 172.21.15.114:0/106195047 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c94004cc0 con 0x7f5c9c102b80 2024-05-27T07:58:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.943+0000 7f5ca0882640 1 -- 172.21.15.114:0/106195047 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c9400f740 con 0x7f5c9c102b80 2024-05-27T07:58:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.944+0000 7f5ca1884640 1 -- 172.21.15.114:0/106195047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c9c0745c0 con 0x7f5c9c102b80 2024-05-27T07:58:19.946 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.944+0000 7f5ca1884640 1 -- 172.21.15.114:0/106195047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c9c1142a0 con 0x7f5c9c102b80 2024-05-27T07:58:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.953+0000 7f5ca0882640 1 -- 172.21.15.114:0/106195047 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f5c94005560 con 0x7f5c9c102b80 2024-05-27T07:58:19.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.959+0000 7f5ca0882640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5c840b2b20 0x7f5c840b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:19.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.959+0000 7f5ca0882640 1 -- 172.21.15.114:0/106195047 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 9554+0+0 (secure 0 0 0) 0x7f5c940e2dd0 con 0x7f5c9c102b80 2024-05-27T07:58:19.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.961+0000 7f5ca8809740 1 Processor -- start 2024-05-27T07:58:19.963 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.962+0000 7f5c9a7fc640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5c840b2b20 0x7f5c840b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.964+0000 7f5ca8809740 1 -- start start 2024-05-27T07:58:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.964+0000 7f5ca8809740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 0x564db04db3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.964+0000 7f5ca8809740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db01112f0 0x564db04db910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.964+0000 7f5ca8809740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 0x564db04dbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.964+0000 7f5ca8809740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564db044dc30 con 0x564db01112f0 2024-05-27T07:58:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.964+0000 7f5ca8809740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564db043d7d0 con 0x564db013a1b0 2024-05-27T07:58:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.964+0000 7f5ca8809740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564db0441810 con 0x564db01194d0 2024-05-27T07:58:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.965+0000 7f5c9affd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 0x564db04db3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.965+0000 7f5c9a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db01112f0 0x564db04db910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.965+0000 7f5c9affd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 0x564db04db3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:48600/0 (socket says 172.21.15.114:48600) 2024-05-27T07:58:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.965+0000 7f5c9affd640 1 -- 172.21.15.114:0/637686941 learned_addr learned my addr 172.21.15.114:0/637686941 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.965+0000 7f5c9b7fe640 1 --2- 172.21.15.114:0/637686941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 0x564db04dbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.966+0000 7f5c9affd640 1 -- 172.21.15.114:0/637686941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 msgr2=0x564db04dbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.966+0000 7f5c9affd640 1 --2- 172.21.15.114:0/637686941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 0x564db04dbe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.966+0000 7f5c9affd640 1 -- 172.21.15.114:0/637686941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db01112f0 msgr2=0x564db04db910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.966+0000 7f5c9affd640 1 --2- 172.21.15.114:0/637686941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db01112f0 0x564db04db910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.966+0000 7f5c9affd640 1 -- 172.21.15.114:0/637686941 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564db0152210 con 0x564db013a1b0 2024-05-27T07:58:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.966+0000 7f5c9affd640 1 --2- 172.21.15.114:0/637686941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 0x564db04db3d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5c940e0050 tx=0x7f5c940a9560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34e549ca1f0a6bf2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.966+0000 7f5c81ffb640 1 -- 172.21.15.114:0/637686941 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c940a9cc0 con 0x564db013a1b0 2024-05-27T07:58:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.966+0000 7f5c81ffb640 1 -- 172.21.15.114:0/637686941 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c940ae070 con 0x564db013a1b0 2024-05-27T07:58:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.966+0000 7f5c81ffb640 1 -- 172.21.15.114:0/637686941 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c940a5890 con 0x564db013a1b0 2024-05-27T07:58:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.968+0000 7f5ca8809740 1 -- 172.21.15.114:0/637686941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 msgr2=0x564db04db3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.968+0000 7f5ca8809740 1 --2- 172.21.15.114:0/637686941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 0x564db04db3d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5c940e0050 tx=0x7f5c940a9560 comp rx=0 tx=0).stop 2024-05-27T07:58:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.969+0000 7f5c9a7fc640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5c840b2b20 0x7f5c840b5030 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c006010 tx=0x7f5c8c000bc0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.969+0000 7f5ca8809740 1 -- 172.21.15.114:0/637686941 shutdown_connections 2024-05-27T07:58:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.969+0000 7f5ca8809740 1 --2- 172.21.15.114:0/637686941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 0x564db04dbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.969+0000 7f5ca8809740 1 --2- 172.21.15.114:0/637686941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db01112f0 0x564db04db910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.969+0000 7f5ca8809740 1 --2- 172.21.15.114:0/637686941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 0x564db04db3d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.969+0000 7f5ca8809740 1 -- 172.21.15.114:0/637686941 >> 172.21.15.114:0/637686941 conn(0x564db012e300 msgr2=0x564db04d7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.973+0000 7f5ca8809740 1 -- 172.21.15.114:0/637686941 shutdown_connections 2024-05-27T07:58:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.975+0000 7f5ca8809740 1 -- 172.21.15.114:0/637686941 wait complete. 2024-05-27T07:58:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.976+0000 7f5ca8809740 1 -- start start 2024-05-27T07:58:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.976+0000 7f5ca8809740 1 Processor -- start 2024-05-27T07:58:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.979+0000 7f5ca8809740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01112f0 0x564db04e74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.979+0000 7f5ca8809740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db01194d0 0x564db04e7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.979+0000 7f5ca8809740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 0x564db04e5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.979+0000 7f5ca8809740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564db0123790 con 0x564db01194d0 2024-05-27T07:58:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.979+0000 7f5ca8809740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564db03f9890 con 0x564db013a1b0 2024-05-27T07:58:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.979+0000 7f5ca8809740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564db04e8500 con 0x564db01112f0 2024-05-27T07:58:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.979+0000 7f5ca8809740 10 client.0 _ll_register_callbacks cb 0x564db0151a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.979+0000 7f5c9b7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 0x564db04e5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.980+0000 7f5c9b7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 0x564db04e5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:48612/0 (socket says 172.21.15.114:48612) 2024-05-27T07:58:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.980+0000 7f5c9b7fe640 1 -- 172.21.15.114:0/2008746840 learned_addr learned my addr 172.21.15.114:0/2008746840 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.980+0000 7f5c9b7fe640 1 -- 172.21.15.114:0/2008746840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01112f0 msgr2=0x564db04e74f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:58:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.980+0000 7f5c9b7fe640 1 --2- 172.21.15.114:0/2008746840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01112f0 0x564db04e74f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.980+0000 7f5c9b7fe640 1 -- 172.21.15.114:0/2008746840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db01194d0 msgr2=0x564db04e7a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:58:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.980+0000 7f5c9b7fe640 1 --2- 172.21.15.114:0/2008746840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db01194d0 0x564db04e7a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.980+0000 7f5c9b7fe640 1 -- 172.21.15.114:0/2008746840 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564db021e620 con 0x564db013a1b0 2024-05-27T07:58:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.980+0000 7f5c9b7fe640 1 --2- 172.21.15.114:0/2008746840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 0x564db04e5a00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009f40 tx=0x7f5c9000e600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84cfd4bebdf34e57 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.980+0000 7f5c7affd640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:58:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.981+0000 7f5c7affd640 1 -- 172.21.15.114:0/2008746840 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c90017020 con 0x564db013a1b0 2024-05-27T07:58:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.981+0000 7f5ca8809740 1 -- 172.21.15.114:0/2008746840 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c940ae070 con 0x564db013a1b0 2024-05-27T07:58:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.981+0000 7f5c7affd640 1 -- 172.21.15.114:0/2008746840 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(156..156 src has 1..156) ==== 9554+0+0 (secure 0 0 0) 0x7f5c90013410 con 0x564db013a1b0 2024-05-27T07:58:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.981+0000 7f5c7affd640 1 -- 172.21.15.114:0/2008746840 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7f5c90022050 con 0x564db013a1b0 2024-05-27T07:58:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.981+0000 7f5ca8809740 10 client.28874 fetch_fsmap learned FSMap version 104 2024-05-27T07:58:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.981+0000 7f5ca8809740 1 -- 172.21.15.114:0/2008746840 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x564db04f4960 con 0x564db013a1b0 2024-05-27T07:58:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.982+0000 7f5c7affd640 1 -- 172.21.15.114:0/2008746840 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c940ae070 con 0x564db013a1b0 2024-05-27T07:58:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.982+0000 7f5c7affd640 1 -- 172.21.15.114:0/2008746840 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 104) ==== 4095+0+0 (secure 0 0 0) 0x7f5c90012050 con 0x564db013a1b0 2024-05-27T07:58:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.982+0000 7f5ca8809740 10 client.28874 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:58:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.982+0000 7f5ca8809740 10 client.28874 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.982+0000 7f5ca8809740 20 client.28874 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.982+0000 7f5ca8809740 1 --2- 172.21.15.114:0/2008746840 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564db00f8d80 0x564db04f8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.982+0000 7f5ca8809740 4 client.28874 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:58:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.982+0000 7f5ca8809740 1 -- 172.21.15.114:0/2008746840 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564db044fb80 con 0x564db00f8d80 2024-05-27T07:58:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.984+0000 7f5c9affd640 1 --2- 172.21.15.114:0/2008746840 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564db00f8d80 0x564db04f8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.989+0000 7f5c9affd640 1 --2- 172.21.15.114:0/2008746840 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564db00f8d80 0x564db04f8420 crc :-1 s=READY pgs=488 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac04cfa3afa76d66 server_cookie=f0b47e76f0e73c70 in_seq=0 out_seq=0 2024-05-27T07:58:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.990+0000 7f5c7affd640 10 client.28874 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:19.993 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.993+0000 7f5c7affd640 1 -- 172.21.15.114:0/2008746840 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564db044fb80 con 0x564db00f8d80 2024-05-27T07:58:19.994 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.993+0000 7f5c7affd640 10 client.28874 handle_command_reply: tid=0 2024-05-27T07:58:19.994 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.993+0000 7f5ca8809740 1 client.28874 shutdown 2024-05-27T07:58:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.996+0000 7f5ca8809740 1 -- 172.21.15.114:0/2008746840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 msgr2=0x564db04e5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:19.997 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.996+0000 7f5ca8809740 1 --2- 172.21.15.114:0/2008746840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 0x564db04e5a00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009f40 tx=0x7f5c9000e600 comp rx=0 tx=0).stop 2024-05-27T07:58:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.999+0000 7f5ca8809740 1 -- 172.21.15.114:0/2008746840 shutdown_connections 2024-05-27T07:58:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.999+0000 7f5ca8809740 1 --2- 172.21.15.114:0/2008746840 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564db00f8d80 0x564db04f8420 crc :-1 s=READY pgs=488 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.999+0000 7f5ca8809740 1 --2- 172.21.15.114:0/2008746840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db013a1b0 0x564db04e5a00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.999+0000 7f5ca8809740 1 --2- 172.21.15.114:0/2008746840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db01194d0 0x564db04e7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.999+0000 7f5ca8809740 1 --2- 172.21.15.114:0/2008746840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01112f0 0x564db04e74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:19.999+0000 7f5ca8809740 1 -- 172.21.15.114:0/2008746840 >> 172.21.15.114:0/2008746840 conn(0x564db012e300 msgr2=0x564db04db1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.000+0000 7f5c7affd640 0 client.28874 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.005+0000 7f5ca8809740 1 -- 172.21.15.114:0/2008746840 shutdown_connections 2024-05-27T07:58:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.007+0000 7f5ca8809740 1 -- 172.21.15.114:0/2008746840 wait complete. 2024-05-27T07:58:20.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.008+0000 7f5ca8809740 20 client.28874 trim_cache size 0 max 16384 2024-05-27T07:58:20.038 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.037+0000 7f5ca8809740 1 Processor -- start 2024-05-27T07:58:20.038 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.037+0000 7f5ca8809740 1 -- start start 2024-05-27T07:58:20.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.037+0000 7f5ca8809740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db01112f0 0x564db04d7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.038+0000 7f5ca8809740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 0x564db04db160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.038+0000 7f5ca8809740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db013a1b0 0x564db04db6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.038+0000 7f5c9affd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db01112f0 0x564db04d7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.038+0000 7f5c9affd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db01112f0 0x564db04d7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:48618/0 (socket says 172.21.15.114:48618) 2024-05-27T07:58:20.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.038+0000 7f5c9affd640 1 -- 172.21.15.114:0/3581960532 learned_addr learned my addr 172.21.15.114:0/3581960532 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:20.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.038+0000 7f5c9b7fe640 1 --2- 172.21.15.114:0/3581960532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db013a1b0 0x564db04db6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.038+0000 7f5ca8809740 1 -- 172.21.15.114:0/3581960532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564db04e6b90 con 0x564db013a1b0 2024-05-27T07:58:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.038+0000 7f5ca8809740 1 -- 172.21.15.114:0/3581960532 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c1c8810 con 0x564db01112f0 2024-05-27T07:58:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.038+0000 7f5ca8809740 1 -- 172.21.15.114:0/3581960532 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c1c8ee0 con 0x564db01194d0 2024-05-27T07:58:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.039+0000 7f5c9a7fc640 1 --2- 172.21.15.114:0/3581960532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 0x564db04db160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.039+0000 7f5c9b7fe640 1 -- 172.21.15.114:0/3581960532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 msgr2=0x564db04db160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.039+0000 7f5c9b7fe640 1 --2- 172.21.15.114:0/3581960532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 0x564db04db160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.039+0000 7f5c9b7fe640 1 -- 172.21.15.114:0/3581960532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db01112f0 msgr2=0x564db04d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.039+0000 7f5c9b7fe640 1 --2- 172.21.15.114:0/3581960532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db01112f0 0x564db04d7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.039+0000 7f5c9b7fe640 1 -- 172.21.15.114:0/3581960532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564db04f89c0 con 0x564db013a1b0 2024-05-27T07:58:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.039+0000 7f5c9affd640 1 --2- 172.21.15.114:0/3581960532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db01112f0 0x564db04d7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:58:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.040+0000 7f5c9b7fe640 1 --2- 172.21.15.114:0/3581960532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db013a1b0 0x564db04db6a0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f5c9001baf0 tx=0x7f5c9000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2bfed966214896 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.040+0000 7f5c817fa640 1 -- 172.21.15.114:0/3581960532 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c90017040 con 0x564db013a1b0 2024-05-27T07:58:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.041+0000 7f5c817fa640 1 -- 172.21.15.114:0/3581960532 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c9001bd70 con 0x564db013a1b0 2024-05-27T07:58:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.041+0000 7f5c817fa640 1 -- 172.21.15.114:0/3581960532 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c9000f040 con 0x564db013a1b0 2024-05-27T07:58:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.042+0000 7f5ca8809740 1 -- 172.21.15.114:0/3581960532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db013a1b0 msgr2=0x564db04db6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.042+0000 7f5ca8809740 1 --2- 172.21.15.114:0/3581960532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db013a1b0 0x564db04db6a0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f5c9001baf0 tx=0x7f5c9000ec30 comp rx=0 tx=0).stop 2024-05-27T07:58:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.043+0000 7f5ca8809740 1 -- 172.21.15.114:0/3581960532 shutdown_connections 2024-05-27T07:58:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.043+0000 7f5ca8809740 1 --2- 172.21.15.114:0/3581960532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db013a1b0 0x564db04db6a0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.043+0000 7f5ca8809740 1 --2- 172.21.15.114:0/3581960532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 0x564db04db160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.043+0000 7f5ca8809740 1 --2- 172.21.15.114:0/3581960532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db01112f0 0x564db04d7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.043+0000 7f5ca8809740 1 -- 172.21.15.114:0/3581960532 >> 172.21.15.114:0/3581960532 conn(0x564db012e300 msgr2=0x564db04d7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.046+0000 7f5ca8809740 1 -- 172.21.15.114:0/3581960532 shutdown_connections 2024-05-27T07:58:20.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.049+0000 7f5ca8809740 1 -- 172.21.15.114:0/3581960532 wait complete. 2024-05-27T07:58:20.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.049+0000 7f5ca8809740 1 -- start start 2024-05-27T07:58:20.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.050+0000 7f5ca8809740 1 Processor -- start 2024-05-27T07:58:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.050+0000 7f5ca8809740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db01112f0 0x564db04edb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.050+0000 7f5ca8809740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 0x564db04e9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.050+0000 7f5ca8809740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db013a1b0 0x564db04ea3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.050+0000 7f5ca8809740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c1c6a70 con 0x564db013a1b0 2024-05-27T07:58:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.050+0000 7f5ca8809740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c1c77b0 con 0x564db01112f0 2024-05-27T07:58:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.050+0000 7f5ca8809740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c1c70e0 con 0x564db01194d0 2024-05-27T07:58:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.050+0000 7f5ca8809740 10 client.0 _ll_register_callbacks cb 0x7f5c90006f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:20.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.051+0000 7f5c9affd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db01112f0 0x564db04edb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.051+0000 7f5c9affd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db01112f0 0x564db04edb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:48626/0 (socket says 172.21.15.114:48626) 2024-05-27T07:58:20.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.051+0000 7f5c9affd640 1 -- 172.21.15.114:0/628406020 learned_addr learned my addr 172.21.15.114:0/628406020 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.051+0000 7f5c9a7fc640 1 --2- 172.21.15.114:0/628406020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 0x564db04e9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.053+0000 7f5c9b7fe640 1 --2- 172.21.15.114:0/628406020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db013a1b0 0x564db04ea3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.053+0000 7f5c9b7fe640 1 -- 172.21.15.114:0/628406020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 msgr2=0x564db04e9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:20.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.053+0000 7f5c9b7fe640 1 --2- 172.21.15.114:0/628406020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 0x564db04e9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.054+0000 7f5c9b7fe640 1 -- 172.21.15.114:0/628406020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db01112f0 msgr2=0x564db04edb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:20.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.054+0000 7f5c9b7fe640 1 --2- 172.21.15.114:0/628406020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db01112f0 0x564db04edb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.054+0000 7f5c9b7fe640 1 -- 172.21.15.114:0/628406020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564db04ea920 con 0x564db013a1b0 2024-05-27T07:58:20.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.054+0000 7f5c9b7fe640 1 --2- 172.21.15.114:0/628406020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db013a1b0 0x564db04ea3e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009800 tx=0x7f5c90006be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d32e08ff9ca0e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:20.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.055+0000 7f5c7affd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:58:20.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.055+0000 7f5c7affd640 1 -- 172.21.15.114:0/628406020 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c90017040 con 0x564db013a1b0 2024-05-27T07:58:20.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.055+0000 7f5c7affd640 1 -- 172.21.15.114:0/628406020 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(156..156 src has 1..156) ==== 9554+0+0 (secure 0 0 0) 0x7f5c90013ca0 con 0x564db013a1b0 2024-05-27T07:58:20.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.055+0000 7f5ca8809740 1 -- 172.21.15.114:0/628406020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c9001bd70 con 0x564db013a1b0 2024-05-27T07:58:20.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.056+0000 7f5c7affd640 1 -- 172.21.15.114:0/628406020 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7f5c90013430 con 0x564db013a1b0 2024-05-27T07:58:20.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.058+0000 7f5ca8809740 10 client.19551 fetch_fsmap learned FSMap version 104 2024-05-27T07:58:20.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.058+0000 7f5ca8809740 1 -- 172.21.15.114:0/628406020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x564db055f680 con 0x564db013a1b0 2024-05-27T07:58:20.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.059+0000 7f5c7affd640 1 -- 172.21.15.114:0/628406020 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c9001bd70 con 0x564db013a1b0 2024-05-27T07:58:20.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.059+0000 7f5c7affd640 1 -- 172.21.15.114:0/628406020 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 104) ==== 4095+0+0 (secure 0 0 0) 0x7f5c90012070 con 0x564db013a1b0 2024-05-27T07:58:20.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.060+0000 7f5ca8809740 10 client.19551 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:58:20.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.060+0000 7f5ca8809740 10 client.19551 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:20.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.060+0000 7f5ca8809740 20 client.19551 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:20.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.060+0000 7f5ca8809740 1 --2- 172.21.15.114:0/628406020 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564db00ce1c0 0x564db0561c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.060+0000 7f5ca8809740 4 client.19551 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:58:20.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.060+0000 7f5ca8809740 1 -- 172.21.15.114:0/628406020 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564db0449360 con 0x564db00ce1c0 2024-05-27T07:58:20.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.065+0000 7f5c9affd640 1 --2- 172.21.15.114:0/628406020 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564db00ce1c0 0x564db0561c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.076+0000 7f5c9affd640 1 --2- 172.21.15.114:0/628406020 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564db00ce1c0 0x564db0561c50 crc :-1 s=READY pgs=489 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=922e2182dc7e1a5b server_cookie=9edb23798e821dc2 in_seq=0 out_seq=0 2024-05-27T07:58:20.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:20.076+0000 7f5c7affd640 10 client.19551 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:20.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:19 smithi163 ceph-mon[36451]: pgmap v637: 129 pgs: 11 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 90 active+clean; 357 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 4.1 MiB/s wr, 296 op/s 2024-05-27T07:58:20.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:19 smithi186 ceph-mon[36647]: pgmap v637: 129 pgs: 11 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 90 active+clean; 357 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 4.1 MiB/s wr, 296 op/s 2024-05-27T07:58:20.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:19 smithi114 ceph-mon[32038]: pgmap v637: 129 pgs: 11 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 90 active+clean; 357 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 4.1 MiB/s wr, 296 op/s 2024-05-27T07:58:20.569 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:58:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.271+0000 7f0d89ac0700 1 -- 172.21.15.114:0/3855412936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d84110290 msgr2=0x7f0d84117bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.271+0000 7f0d89ac0700 1 --2- 172.21.15.114:0/3855412936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d84110290 0x7f0d84117bf0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f0d7400afc0 tx=0x7f0d74008850 comp rx=0 tx=0).stop 2024-05-27T07:58:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.274+0000 7f0d89ac0700 1 -- 172.21.15.114:0/3855412936 shutdown_connections 2024-05-27T07:58:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.274+0000 7f0d89ac0700 1 --2- 172.21.15.114:0/3855412936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d84110290 0x7f0d84117bf0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.274+0000 7f0d89ac0700 1 --2- 172.21.15.114:0/3855412936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d8410b010 0x7f0d8410b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.274+0000 7f0d89ac0700 1 --2- 172.21.15.114:0/3855412936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d8410f940 0x7f0d8410fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.274+0000 7f0d89ac0700 1 -- 172.21.15.114:0/3855412936 >> 172.21.15.114:0/3855412936 conn(0x7f0d8406f980 msgr2=0x7f0d8406fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.276+0000 7f0d89ac0700 1 -- 172.21.15.114:0/3855412936 shutdown_connections 2024-05-27T07:58:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.277+0000 7f0d89ac0700 1 -- 172.21.15.114:0/3855412936 wait complete. 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.278+0000 7f0d89ac0700 1 Processor -- start 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.280+0000 7f0d89ac0700 1 -- start start 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.280+0000 7f0d89ac0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d8410b010 0x7f0d8411f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.281+0000 7f0d89ac0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d8410f940 0x7f0d8411f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.281+0000 7f0d89ac0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d84110290 0x7f0d8411a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.281+0000 7f0d89ac0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8412bb60 con 0x7f0d8410b010 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.281+0000 7f0d89ac0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8412b9e0 con 0x7f0d8410f940 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.281+0000 7f0d89ac0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8412bce0 con 0x7f0d84110290 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.281+0000 7f0d837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d8410b010 0x7f0d8411f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.281+0000 7f0d837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d8410b010 0x7f0d8411f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36930/0 (socket says 172.21.15.114:36930) 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.281+0000 7f0d837fe700 1 -- 172.21.15.114:0/3460656722 learned_addr learned my addr 172.21.15.114:0/3460656722 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.281+0000 7f0d82ffd700 1 --2- 172.21.15.114:0/3460656722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d8410f940 0x7f0d8411f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.282+0000 7f0d837fe700 1 -- 172.21.15.114:0/3460656722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d84110290 msgr2=0x7f0d8411a2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:58:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.282+0000 7f0d837fe700 1 --2- 172.21.15.114:0/3460656722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d84110290 0x7f0d8411a2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.282+0000 7f0d837fe700 1 -- 172.21.15.114:0/3460656722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d8410f940 msgr2=0x7f0d8411f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.282+0000 7f0d837fe700 1 --2- 172.21.15.114:0/3460656722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d8410f940 0x7f0d8411f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.282+0000 7f0d837fe700 1 -- 172.21.15.114:0/3460656722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d8411ab20 con 0x7f0d8410b010 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.282+0000 7f0d837fe700 1 --2- 172.21.15.114:0/3460656722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d8410b010 0x7f0d8411f090 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00b840 tx=0x7f0d7c00bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.283+0000 7f0d80ff9700 1 -- 172.21.15.114:0/3460656722 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c00d890 con 0x7f0d8410b010 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.283+0000 7f0d89ac0700 1 -- 172.21.15.114:0/3460656722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d8411ae10 con 0x7f0d8410b010 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.283+0000 7f0d89ac0700 1 -- 172.21.15.114:0/3460656722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d84126dc0 con 0x7f0d8410b010 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.283+0000 7f0d80ff9700 1 -- 172.21.15.114:0/3460656722 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d7c00e430 con 0x7f0d8410b010 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.283+0000 7f0d80ff9700 1 -- 172.21.15.114:0/3460656722 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c00c5e0 con 0x7f0d8410b010 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.294+0000 7f0d984d5b80 1 Processor -- start 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.296+0000 7f0d80ff9700 1 -- 172.21.15.114:0/3460656722 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f0d7c00c780 con 0x7f0d8410b010 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.297+0000 7f0d80ff9700 1 --2- 172.21.15.114:0/3460656722 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0d6c0b2c00 0x7f0d6c0b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.297+0000 7f0d80ff9700 1 -- 172.21.15.114:0/3460656722 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 9554+0+0 (secure 0 0 0) 0x7f0d7c0e29a0 con 0x7f0d8410b010 2024-05-27T07:58:20.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.297+0000 7f0d984d5b80 1 -- start start 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.298+0000 7f0d984d5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 0x55ebd333e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.298+0000 7f0d984d5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd333ec50 0x55ebd3338a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.298+0000 7f0d984d5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd3338f80 0x55ebd3339390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.298+0000 7f0d984d5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ebd3267c90 con 0x55ebd31bae40 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.298+0000 7f0d984d5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ebd3257830 con 0x55ebd333ec50 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.298+0000 7f0d984d5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ebd325b870 con 0x55ebd3338f80 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.300+0000 7f0d837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 0x55ebd333e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.300+0000 7f0d837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 0x55ebd333e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36940/0 (socket says 172.21.15.114:36940) 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.300+0000 7f0d837fe700 1 -- 172.21.15.114:0/1455324134 learned_addr learned my addr 172.21.15.114:0/1455324134 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.302+0000 7f0d83fff700 1 --2- 172.21.15.114:0/3460656722 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0d6c0b2c00 0x7f0d6c0b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.304+0000 7f0d83fff700 1 --2- 172.21.15.114:0/1455324134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd3338f80 0x55ebd3339390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.304+0000 7f0d837fe700 1 -- 172.21.15.114:0/1455324134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd3338f80 msgr2=0x55ebd3339390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.304+0000 7f0d837fe700 1 --2- 172.21.15.114:0/1455324134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd3338f80 0x55ebd3339390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.304+0000 7f0d837fe700 1 -- 172.21.15.114:0/1455324134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd333ec50 msgr2=0x55ebd3338a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:58:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.304+0000 7f0d837fe700 1 --2- 172.21.15.114:0/1455324134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd333ec50 0x55ebd3338a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.304+0000 7f0d837fe700 1 -- 172.21.15.114:0/1455324134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebd31a78d0 con 0x55ebd31bae40 2024-05-27T07:58:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.307+0000 7f0d837fe700 1 --2- 172.21.15.114:0/1455324134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 0x55ebd333e710 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0a5890 tx=0x7f0d7c0aa060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf3a88f63b1def5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.311+0000 7f0d83fff700 1 --2- 172.21.15.114:0/3460656722 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0d6c0b2c00 0x7f0d6c0b5110 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0d7401a040 tx=0x7f0d7400b320 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.314+0000 7f0d69ffb700 1 -- 172.21.15.114:0/1455324134 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c0a5aa0 con 0x55ebd31bae40 2024-05-27T07:58:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.314+0000 7f0d69ffb700 1 -- 172.21.15.114:0/1455324134 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d7c0dff00 con 0x55ebd31bae40 2024-05-27T07:58:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.314+0000 7f0d69ffb700 1 -- 172.21.15.114:0/1455324134 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c0b21e0 con 0x55ebd31bae40 2024-05-27T07:58:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.315+0000 7f0d984d5b80 1 -- 172.21.15.114:0/1455324134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 msgr2=0x55ebd333e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.315+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/1455324134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 0x55ebd333e710 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0a5890 tx=0x7f0d7c0aa060 comp rx=0 tx=0).stop 2024-05-27T07:58:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.316+0000 7f0d984d5b80 1 -- 172.21.15.114:0/1455324134 shutdown_connections 2024-05-27T07:58:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.316+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/1455324134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd3338f80 0x55ebd3339390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.316+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/1455324134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 0x55ebd333e710 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.316+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/1455324134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd333ec50 0x55ebd3338a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.316+0000 7f0d984d5b80 1 -- 172.21.15.114:0/1455324134 >> 172.21.15.114:0/1455324134 conn(0x55ebd3215af0 msgr2=0x55ebd3333640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.318+0000 7f0d984d5b80 1 -- 172.21.15.114:0/1455324134 shutdown_connections 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.318+0000 7f0d984d5b80 1 -- 172.21.15.114:0/1455324134 wait complete. 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.319+0000 7f0d984d5b80 1 -- start start 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.319+0000 7f0d984d5b80 1 Processor -- start 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.319+0000 7f0d984d5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd31bae40 0x55ebd3342370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.320+0000 7f0d984d5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd3338f80 0x55ebd33428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.320+0000 7f0d984d5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd333ec50 0x55ebd3342df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.320+0000 7f0d984d5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ebd315dfa0 con 0x55ebd333ec50 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.320+0000 7f0d984d5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ebd31f97c0 con 0x55ebd31bae40 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.320+0000 7f0d984d5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ebd3345740 con 0x55ebd3338f80 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.320+0000 7f0d984d5b80 10 client.0 _ll_register_callbacks cb 0x55ebd31f2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.320+0000 7f0d837fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd31bae40 0x55ebd3342370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.320+0000 7f0d837fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd31bae40 0x55ebd3342370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:48660/0 (socket says 172.21.15.114:48660) 2024-05-27T07:58:20.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.320+0000 7f0d837fe700 1 -- 172.21.15.114:0/2566001050 learned_addr learned my addr 172.21.15.114:0/2566001050 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.320+0000 7f0d82ffd700 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd3338f80 0x55ebd33428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.321+0000 7f0d83fff700 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd333ec50 0x55ebd3342df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.321+0000 7f0d837fe700 1 -- 172.21.15.114:0/2566001050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd3338f80 msgr2=0x55ebd33428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.321+0000 7f0d837fe700 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd3338f80 0x55ebd33428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.321+0000 7f0d837fe700 1 -- 172.21.15.114:0/2566001050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd333ec50 msgr2=0x55ebd3342df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.321+0000 7f0d837fe700 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd333ec50 0x55ebd3342df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.321+0000 7f0d837fe700 1 -- 172.21.15.114:0/2566001050 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebd31f6690 con 0x55ebd31bae40 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.321+0000 7f0d83fff700 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd333ec50 0x55ebd3342df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.321+0000 7f0d837fe700 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd31bae40 0x55ebd3342370 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0a9510 tx=0x7f0d7c00c2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16f71adf162e3aab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.321+0000 7f0d62ffd700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.322+0000 7f0d62ffd700 1 -- 172.21.15.114:0/2566001050 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c0c2070 con 0x55ebd31bae40 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.322+0000 7f0d62ffd700 1 -- 172.21.15.114:0/2566001050 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(156..156 src has 1..156) ==== 9554+0+0 (secure 0 0 0) 0x7f0d7c0e00b0 con 0x55ebd31bae40 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.323+0000 7f0d984d5b80 1 -- 172.21.15.114:0/2566001050 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d7c0dff00 con 0x55ebd31bae40 2024-05-27T07:58:20.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.324+0000 7f0d62ffd700 1 -- 172.21.15.114:0/2566001050 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7f0d7c0a9cc0 con 0x55ebd31bae40 2024-05-27T07:58:20.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.324+0000 7f0d984d5b80 10 client.28883 fetch_fsmap learned FSMap version 104 2024-05-27T07:58:20.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.324+0000 7f0d984d5b80 1 -- 172.21.15.114:0/2566001050 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x55ebd3350030 con 0x55ebd31bae40 2024-05-27T07:58:20.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.324+0000 7f0d62ffd700 1 -- 172.21.15.114:0/2566001050 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d7c0aad50 con 0x55ebd31bae40 2024-05-27T07:58:20.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.325+0000 7f0d62ffd700 1 -- 172.21.15.114:0/2566001050 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 104) ==== 4095+0+0 (secure 0 0 0) 0x7f0d7c0a5210 con 0x55ebd31bae40 2024-05-27T07:58:20.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.325+0000 7f0d984d5b80 10 client.28883 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:58:20.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.325+0000 7f0d984d5b80 10 client.28883 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:20.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.325+0000 7f0d984d5b80 20 client.28883 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:20.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.325+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ebd3352630 0x55ebd3354ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.325+0000 7f0d984d5b80 4 client.28883 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:58:20.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.325+0000 7f0d984d5b80 1 -- 172.21.15.114:0/2566001050 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0d7c0dff00 con 0x55ebd3352630 2024-05-27T07:58:20.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.325+0000 7f0d82ffd700 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ebd3352630 0x55ebd3354ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.326+0000 7f0d62ffd700 10 client.28883 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:20.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.326+0000 7f0d82ffd700 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ebd3352630 0x55ebd3354ac0 crc :-1 s=READY pgs=490 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=110bb180693663c8 server_cookie=a5835d0a1098c08 in_seq=0 out_seq=0 2024-05-27T07:58:20.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.341+0000 7f0d62ffd700 1 -- 172.21.15.114:0/2566001050 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f0d7c0dff00 con 0x55ebd3352630 2024-05-27T07:58:20.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.341+0000 7f0d62ffd700 10 client.28883 handle_command_reply: tid=0 2024-05-27T07:58:20.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.341+0000 7f0d984d5b80 1 client.28883 shutdown 2024-05-27T07:58:20.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.342+0000 7f0d984d5b80 1 -- 172.21.15.114:0/2566001050 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd31bae40 msgr2=0x55ebd3342370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:20.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.342+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd31bae40 0x55ebd3342370 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0a9510 tx=0x7f0d7c00c2c0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.342+0000 7f0d984d5b80 1 -- 172.21.15.114:0/2566001050 shutdown_connections 2024-05-27T07:58:20.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.342+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd333ec50 0x55ebd3342df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.342+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ebd3352630 0x55ebd3354ac0 crc :-1 s=READY pgs=490 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.342+0000 7f0d62ffd700 0 client.28883 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:20.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.342+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd31bae40 0x55ebd3342370 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.342+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/2566001050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd3338f80 0x55ebd33428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.342+0000 7f0d984d5b80 1 -- 172.21.15.114:0/2566001050 >> 172.21.15.114:0/2566001050 conn(0x55ebd3215af0 msgr2=0x55ebd3337a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:20.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.342+0000 7f0d984d5b80 1 -- 172.21.15.114:0/2566001050 shutdown_connections 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.342+0000 7f0d984d5b80 1 -- 172.21.15.114:0/2566001050 wait complete. 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.342+0000 7f0d984d5b80 20 client.28883 trim_cache size 0 max 16384 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.357+0000 7f0d984d5b80 1 Processor -- start 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.357+0000 7f0d984d5b80 1 -- start start 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.357+0000 7f0d984d5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 0x55ebd3338a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.357+0000 7f0d984d5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd3338f80 0x55ebd33350d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.357+0000 7f0d984d5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd333ec50 0x55ebd334e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.357+0000 7f0d984d5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7000ebc0 con 0x55ebd31bae40 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.357+0000 7f0d984d5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7000f290 con 0x55ebd3338f80 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.357+0000 7f0d984d5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7000db40 con 0x55ebd333ec50 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.357+0000 7f0d837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 0x55ebd3338a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.357+0000 7f0d837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 0x55ebd3338a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36960/0 (socket says 172.21.15.114:36960) 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.357+0000 7f0d837fe700 1 -- 172.21.15.114:0/3922131969 learned_addr learned my addr 172.21.15.114:0/3922131969 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.357+0000 7f0d83fff700 1 --2- 172.21.15.114:0/3922131969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd333ec50 0x55ebd334e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.357+0000 7f0d82ffd700 1 --2- 172.21.15.114:0/3922131969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd3338f80 0x55ebd33350d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.358+0000 7f0d837fe700 1 -- 172.21.15.114:0/3922131969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd333ec50 msgr2=0x55ebd334e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:20.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.358+0000 7f0d837fe700 1 --2- 172.21.15.114:0/3922131969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd333ec50 0x55ebd334e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.358+0000 7f0d837fe700 1 -- 172.21.15.114:0/3922131969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd3338f80 msgr2=0x55ebd33350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:20.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.358+0000 7f0d837fe700 1 --2- 172.21.15.114:0/3922131969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd3338f80 0x55ebd33350d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.358+0000 7f0d837fe700 1 -- 172.21.15.114:0/3922131969 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebd3333890 con 0x55ebd31bae40 2024-05-27T07:58:20.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.358+0000 7f0d83fff700 1 --2- 172.21.15.114:0/3922131969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd333ec50 0x55ebd334e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:58:20.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.358+0000 7f0d837fe700 1 --2- 172.21.15.114:0/3922131969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 0x55ebd3338a40 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0b2140 tx=0x7f0d7c0b27f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6339a4d92e30afc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:20.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.358+0000 7f0d697fa700 1 -- 172.21.15.114:0/3922131969 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c0c2070 con 0x55ebd31bae40 2024-05-27T07:58:20.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.358+0000 7f0d697fa700 1 -- 172.21.15.114:0/3922131969 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d7c0dff00 con 0x55ebd31bae40 2024-05-27T07:58:20.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.358+0000 7f0d697fa700 1 -- 172.21.15.114:0/3922131969 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c00ca70 con 0x55ebd31bae40 2024-05-27T07:58:20.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.359+0000 7f0d984d5b80 1 -- 172.21.15.114:0/3922131969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 msgr2=0x55ebd3338a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.359+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/3922131969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 0x55ebd3338a40 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0b2140 tx=0x7f0d7c0b27f0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.359+0000 7f0d984d5b80 1 -- 172.21.15.114:0/3922131969 shutdown_connections 2024-05-27T07:58:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.359+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/3922131969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd333ec50 0x55ebd334e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.359+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/3922131969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd31bae40 0x55ebd3338a40 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.359+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/3922131969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd3338f80 0x55ebd33350d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.359+0000 7f0d984d5b80 1 -- 172.21.15.114:0/3922131969 >> 172.21.15.114:0/3922131969 conn(0x55ebd3215af0 msgr2=0x55ebd3340c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.359+0000 7f0d984d5b80 1 -- 172.21.15.114:0/3922131969 shutdown_connections 2024-05-27T07:58:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.359+0000 7f0d984d5b80 1 -- 172.21.15.114:0/3922131969 wait complete. 2024-05-27T07:58:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.359+0000 7f0d984d5b80 1 -- start start 2024-05-27T07:58:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.360+0000 7f0d984d5b80 1 Processor -- start 2024-05-27T07:58:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.360+0000 7f0d984d5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd31bae40 0x55ebd334aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.360+0000 7f0d984d5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd3338f80 0x55ebd334b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.360+0000 7f0d984d5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd333ec50 0x55ebd33596d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.360+0000 7f0d984d5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7000d470 con 0x55ebd333ec50 2024-05-27T07:58:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.360+0000 7f0d984d5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7000ce00 con 0x55ebd3338f80 2024-05-27T07:58:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.360+0000 7f0d984d5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ebd3343480 con 0x55ebd31bae40 2024-05-27T07:58:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.360+0000 7f0d984d5b80 10 client.0 _ll_register_callbacks cb 0x55ebd31f2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.360+0000 7f0d83fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd333ec50 0x55ebd33596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.360+0000 7f0d83fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd333ec50 0x55ebd33596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36964/0 (socket says 172.21.15.114:36964) 2024-05-27T07:58:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.360+0000 7f0d83fff700 1 -- 172.21.15.114:0/1770701748 learned_addr learned my addr 172.21.15.114:0/1770701748 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.360+0000 7f0d82ffd700 1 --2- 172.21.15.114:0/1770701748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd3338f80 0x55ebd334b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.360+0000 7f0d837fe700 1 --2- 172.21.15.114:0/1770701748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd31bae40 0x55ebd334aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d83fff700 1 -- 172.21.15.114:0/1770701748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd31bae40 msgr2=0x55ebd334aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d83fff700 1 --2- 172.21.15.114:0/1770701748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd31bae40 0x55ebd334aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d83fff700 1 -- 172.21.15.114:0/1770701748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd3338f80 msgr2=0x55ebd334b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d83fff700 1 --2- 172.21.15.114:0/1770701748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd3338f80 0x55ebd334b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d83fff700 1 -- 172.21.15.114:0/1770701748 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebd3359e70 con 0x55ebd333ec50 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d83fff700 1 --2- 172.21.15.114:0/1770701748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd333ec50 0x55ebd33596d0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f0d8411fcc0 tx=0x7f0d7400acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1cfd8a62208c91 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d62ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d62ffd700 1 -- 172.21.15.114:0/1770701748 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7400b770 con 0x55ebd333ec50 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d62ffd700 1 -- 172.21.15.114:0/1770701748 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(156..156 src has 1..156) ==== 9554+0+0 (secure 0 0 0) 0x7f0d7400b910 con 0x55ebd333ec50 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d984d5b80 1 -- 172.21.15.114:0/1770701748 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d7c0dff00 con 0x55ebd333ec50 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d62ffd700 1 -- 172.21.15.114:0/1770701748 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7f0d7400bb10 con 0x55ebd333ec50 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d984d5b80 10 client.19566 fetch_fsmap learned FSMap version 104 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d984d5b80 1 -- 172.21.15.114:0/1770701748 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x55ebd33bcbd0 con 0x55ebd333ec50 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.361+0000 7f0d62ffd700 1 -- 172.21.15.114:0/1770701748 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d7c0dff00 con 0x55ebd333ec50 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.362+0000 7f0d62ffd700 1 -- 172.21.15.114:0/1770701748 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 104) ==== 4095+0+0 (secure 0 0 0) 0x7f0d7402a020 con 0x55ebd333ec50 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.362+0000 7f0d984d5b80 10 client.19566 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.362+0000 7f0d984d5b80 10 client.19566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:20.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.362+0000 7f0d984d5b80 20 client.19566 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.362+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/1770701748 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ebd33bd2e0 0x55ebd33bf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.362+0000 7f0d984d5b80 4 client.19566 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.362+0000 7f0d984d5b80 1 -- 172.21.15.114:0/1770701748 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ebd33049f0 con 0x55ebd33bd2e0 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.364+0000 7f0d837fe700 1 --2- 172.21.15.114:0/1770701748 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ebd33bd2e0 0x55ebd33bf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.365+0000 7f0d62ffd700 10 client.19566 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.365+0000 7f0d837fe700 1 --2- 172.21.15.114:0/1770701748 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ebd33bd2e0 0x55ebd33bf700 crc :-1 s=READY pgs=491 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f382c94050eab2b server_cookie=29260a995c6c4a61 in_seq=0 out_seq=0 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.377+0000 7f0d62ffd700 1 -- 172.21.15.114:0/1770701748 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ebd33049f0 con 0x55ebd33bd2e0 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.377+0000 7f0d62ffd700 10 client.19566 handle_command_reply: tid=0 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.378+0000 7f0d984d5b80 1 client.19566 shutdown 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.378+0000 7f0d984d5b80 1 -- 172.21.15.114:0/1770701748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd333ec50 msgr2=0x55ebd33596d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.378+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/1770701748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd333ec50 0x55ebd33596d0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f0d8411fcc0 tx=0x7f0d7400acb0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.378+0000 7f0d984d5b80 1 -- 172.21.15.114:0/1770701748 shutdown_connections 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.378+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/1770701748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ebd333ec50 0x55ebd33596d0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.378+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/1770701748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebd31bae40 0x55ebd334aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.378+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/1770701748 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55ebd33bd2e0 0x55ebd33bf700 crc :-1 s=READY pgs=491 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.378+0000 7f0d984d5b80 1 --2- 172.21.15.114:0/1770701748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ebd3338f80 0x55ebd334b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.378+0000 7f0d984d5b80 1 -- 172.21.15.114:0/1770701748 >> 172.21.15.114:0/1770701748 conn(0x55ebd3215af0 msgr2=0x55ebd333c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.378+0000 7f0d62ffd700 0 client.19566 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.378+0000 7f0d984d5b80 1 -- 172.21.15.114:0/1770701748 shutdown_connections 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.378+0000 7f0d984d5b80 1 -- 172.21.15.114:0/1770701748 wait complete. 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.378+0000 7f0d984d5b80 20 client.19566 trim_cache size 0 max 16384 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.379+0000 7f0d89ac0700 1 -- 172.21.15.114:0/3460656722 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0d6c0b2c00 msgr2=0x7f0d6c0b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.379+0000 7f0d89ac0700 1 --2- 172.21.15.114:0/3460656722 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0d6c0b2c00 0x7f0d6c0b5110 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0d7401a040 tx=0x7f0d7400b320 comp rx=0 tx=0).stop 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.379+0000 7f0d89ac0700 1 -- 172.21.15.114:0/3460656722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d8410b010 msgr2=0x7f0d8411f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.379+0000 7f0d89ac0700 1 --2- 172.21.15.114:0/3460656722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d8410b010 0x7f0d8411f090 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00b840 tx=0x7f0d7c00bc00 comp rx=0 tx=0).stop 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.379+0000 7f0d89ac0700 1 -- 172.21.15.114:0/3460656722 shutdown_connections 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.379+0000 7f0d89ac0700 1 --2- 172.21.15.114:0/3460656722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d84110290 0x7f0d8411a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.379+0000 7f0d89ac0700 1 --2- 172.21.15.114:0/3460656722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d8410b010 0x7f0d8411f090 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.379+0000 7f0d89ac0700 1 --2- 172.21.15.114:0/3460656722 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0d6c0b2c00 0x7f0d6c0b5110 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.379+0000 7f0d89ac0700 1 --2- 172.21.15.114:0/3460656722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d8410f940 0x7f0d8411f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:20.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.379+0000 7f0d89ac0700 1 -- 172.21.15.114:0/3460656722 >> 172.21.15.114:0/3460656722 conn(0x7f0d8406f980 msgr2=0x7f0d8410d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:20.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.379+0000 7f0d89ac0700 1 -- 172.21.15.114:0/3460656722 shutdown_connections 2024-05-27T07:58:20.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:20.379+0000 7f0d89ac0700 1 -- 172.21.15.114:0/3460656722 wait complete. 2024-05-27T07:58:20.615 INFO:tasks.workunit.client.0.smithi114.stdout: 10 64172 11.97 MB/sec execute 152 sec latency 1543.799 ms 2024-05-27T07:58:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.197+0000 7f5c7affd640 1 -- 172.21.15.114:0/628406020 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x564db0449360 con 0x564db00ce1c0 2024-05-27T07:58:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.198+0000 7f5c7affd640 10 client.19551 handle_command_reply: tid=0 2024-05-27T07:58:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.198+0000 7f5ca8809740 1 client.19551 shutdown 2024-05-27T07:58:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.198+0000 7f5ca8809740 1 -- 172.21.15.114:0/628406020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db013a1b0 msgr2=0x564db04ea3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.198+0000 7f5ca8809740 1 --2- 172.21.15.114:0/628406020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db013a1b0 0x564db04ea3e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009800 tx=0x7f5c90006be0 comp rx=0 tx=0).stop 2024-05-27T07:58:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.198+0000 7f5ca8809740 1 -- 172.21.15.114:0/628406020 shutdown_connections 2024-05-27T07:58:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.198+0000 7f5ca8809740 1 --2- 172.21.15.114:0/628406020 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564db00ce1c0 0x564db0561c50 crc :-1 s=READY pgs=489 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.198+0000 7f5ca8809740 1 --2- 172.21.15.114:0/628406020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564db013a1b0 0x564db04ea3e0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.198+0000 7f5ca8809740 1 --2- 172.21.15.114:0/628406020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db01194d0 0x564db04e9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.198+0000 7f5ca8809740 1 --2- 172.21.15.114:0/628406020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564db01112f0 0x564db04edb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.198+0000 7f5ca8809740 1 -- 172.21.15.114:0/628406020 >> 172.21.15.114:0/628406020 conn(0x564db012e300 msgr2=0x564db04df020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.198+0000 7f5c7affd640 0 client.19551 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.198+0000 7f5ca8809740 1 -- 172.21.15.114:0/628406020 shutdown_connections 2024-05-27T07:58:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.198+0000 7f5ca8809740 1 -- 172.21.15.114:0/628406020 wait complete. 2024-05-27T07:58:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.199+0000 7f5ca8809740 20 client.19551 trim_cache size 0 max 16384 2024-05-27T07:58:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:58:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:58:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:58:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.199+0000 7f5ca1884640 1 -- 172.21.15.114:0/106195047 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5c840b2b20 msgr2=0x7f5c840b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.199+0000 7f5ca1884640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5c840b2b20 0x7f5c840b5030 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c006010 tx=0x7f5c8c000bc0 comp rx=0 tx=0).stop 2024-05-27T07:58:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.199+0000 7f5ca1884640 1 -- 172.21.15.114:0/106195047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c9c102b80 msgr2=0x7f5c9c10c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.199+0000 7f5ca1884640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c9c102b80 0x7f5c9c10c580 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f5c9400cce0 tx=0x7f5c9400a620 comp rx=0 tx=0).stop 2024-05-27T07:58:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.200+0000 7f5ca1884640 1 -- 172.21.15.114:0/106195047 shutdown_connections 2024-05-27T07:58:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.200+0000 7f5ca1884640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5c840b2b20 0x7f5c840b5030 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.200+0000 7f5ca1884640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c9c108000 0x7f5c9c073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.200+0000 7f5ca1884640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c9c1075d0 0x7f5c9c10cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.200+0000 7f5ca1884640 1 --2- 172.21.15.114:0/106195047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c9c102b80 0x7f5c9c10c580 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.200+0000 7f5ca1884640 1 -- 172.21.15.114:0/106195047 >> 172.21.15.114:0/106195047 conn(0x7f5c9c0782d0 msgr2=0x7f5c9c105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.200+0000 7f5ca1884640 1 -- 172.21.15.114:0/106195047 shutdown_connections 2024-05-27T07:58:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:21.200+0000 7f5ca1884640 1 -- 172.21.15.114:0/106195047 wait complete. 2024-05-27T07:58:21.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:20 smithi163 ceph-mon[36451]: pgmap v638: 129 pgs: 11 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 90 active+clean; 357 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.1 MiB/s wr, 312 op/s 2024-05-27T07:58:21.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:20 smithi186 ceph-mon[36647]: pgmap v638: 129 pgs: 11 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 90 active+clean; 357 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.1 MiB/s wr, 312 op/s 2024-05-27T07:58:21.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:20 smithi114 ceph-mon[32038]: pgmap v638: 129 pgs: 11 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 90 active+clean; 357 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.1 MiB/s wr, 312 op/s 2024-05-27T07:58:21.615 INFO:tasks.workunit.client.0.smithi114.stdout: 10 64386 11.96 MB/sec execute 153 sec latency 1675.034 ms 2024-05-27T07:58:22.512 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:58:22.513 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:58:22.615 INFO:tasks.workunit.client.0.smithi114.stdout: 10 64544 11.97 MB/sec execute 154 sec latency 1041.834 ms 2024-05-27T07:58:22.905 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:58:23.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 64822 11.98 MB/sec execute 155 sec latency 660.887 ms 2024-05-27T07:58:24.086 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:23 smithi114 ceph-mon[32038]: pgmap v639: 129 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 102 active+clean; 363 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.1 MiB/s wr, 427 op/s 2024-05-27T07:58:24.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:23 smithi163 ceph-mon[36451]: pgmap v639: 129 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 102 active+clean; 363 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.1 MiB/s wr, 427 op/s 2024-05-27T07:58:24.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:23 smithi186 ceph-mon[36647]: pgmap v639: 129 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 102 active+clean; 363 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.1 MiB/s wr, 427 op/s 2024-05-27T07:58:24.615 INFO:tasks.workunit.client.0.smithi114.stdout: 10 65192 12.03 MB/sec execute 156 sec latency 143.570 ms 2024-05-27T07:58:24.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.877+0000 7fafd6b3c700 1 -- 172.21.15.114:0/3301511775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafd0107e00 msgr2=0x7fafd0108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.877+0000 7fafd6b3c700 1 --2- 172.21.15.114:0/3301511775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafd0107e00 0x7fafd0108210 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fafbc0099c0 tx=0x7fafbc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.877+0000 7fafd6b3c700 1 -- 172.21.15.114:0/3301511775 shutdown_connections 2024-05-27T07:58:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.877+0000 7fafd6b3c700 1 --2- 172.21.15.114:0/3301511775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafd0108750 0x7fafd01100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.877+0000 7fafd6b3c700 1 --2- 172.21.15.114:0/3301511775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafd01034d0 0x7fafd01038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.877+0000 7fafd6b3c700 1 --2- 172.21.15.114:0/3301511775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafd0107e00 0x7fafd0108210 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.877+0000 7fafd6b3c700 1 -- 172.21.15.114:0/3301511775 >> 172.21.15.114:0/3301511775 conn(0x7fafd00fef20 msgr2=0x7fafd0101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.877+0000 7fafd6b3c700 1 -- 172.21.15.114:0/3301511775 shutdown_connections 2024-05-27T07:58:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.877+0000 7fafd6b3c700 1 -- 172.21.15.114:0/3301511775 wait complete. 2024-05-27T07:58:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.878+0000 7fafd6b3c700 1 Processor -- start 2024-05-27T07:58:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.878+0000 7fafd6b3c700 1 -- start start 2024-05-27T07:58:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.878+0000 7fafd6b3c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafd01034d0 0x7fafd0199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.878+0000 7fafd6b3c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafd0107e00 0x7fafd019a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.878+0000 7fafd6b3c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafd0108750 0x7fafd019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.878+0000 7fafd6b3c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fafd0113380 con 0x7fafd0108750 2024-05-27T07:58:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.878+0000 7fafd6b3c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fafd0113200 con 0x7fafd01034d0 2024-05-27T07:58:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.878+0000 7fafd6b3c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fafd0113500 con 0x7fafd0107e00 2024-05-27T07:58:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafd50d9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafd0108750 0x7fafd019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafd48d8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafd01034d0 0x7fafd0199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafcffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafd0107e00 0x7fafd019a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafd50d9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafd0108750 0x7fafd019e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40634/0 (socket says 172.21.15.114:40634) 2024-05-27T07:58:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafd50d9700 1 -- 172.21.15.114:0/3081055939 learned_addr learned my addr 172.21.15.114:0/3081055939 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafd50d9700 1 -- 172.21.15.114:0/3081055939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafd0107e00 msgr2=0x7fafd019a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:24.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafd50d9700 1 --2- 172.21.15.114:0/3081055939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafd0107e00 0x7fafd019a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafd50d9700 1 -- 172.21.15.114:0/3081055939 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafd01034d0 msgr2=0x7fafd0199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:24.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafd50d9700 1 --2- 172.21.15.114:0/3081055939 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafd01034d0 0x7fafd0199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafd50d9700 1 -- 172.21.15.114:0/3081055939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd019eca0 con 0x7fafd0108750 2024-05-27T07:58:24.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafd48d8700 1 --2- 172.21.15.114:0/3081055939 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafd01034d0 0x7fafd0199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:24.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafd50d9700 1 --2- 172.21.15.114:0/3081055939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafd0108750 0x7fafd019e600 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fafc800ef30 tx=0x7fafc800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:24.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafcdffb700 1 -- 172.21.15.114:0/3081055939 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc80092e0 con 0x7fafd0108750 2024-05-27T07:58:24.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafd6b3c700 1 -- 172.21.15.114:0/3081055939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd019ef30 con 0x7fafd0108750 2024-05-27T07:58:24.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafcdffb700 1 -- 172.21.15.114:0/3081055939 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc8004bb0 con 0x7fafd0108750 2024-05-27T07:58:24.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.879+0000 7fafcdffb700 1 -- 172.21.15.114:0/3081055939 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8004020 con 0x7fafd0108750 2024-05-27T07:58:24.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.880+0000 7fafd6b3c700 1 -- 172.21.15.114:0/3081055939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd019f440 con 0x7fafd0108750 2024-05-27T07:58:24.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.882+0000 7fafcdffb700 1 -- 172.21.15.114:0/3081055939 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fafc801b030 con 0x7fafd0108750 2024-05-27T07:58:24.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.883+0000 7fafcdffb700 1 --2- 172.21.15.114:0/3081055939 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fafc00b29f0 0x7fafc00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.883+0000 7fafd48d8700 1 --2- 172.21.15.114:0/3081055939 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fafc00b29f0 0x7fafc00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.883+0000 7fafcdffb700 1 -- 172.21.15.114:0/3081055939 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 9554+0+0 (secure 0 0 0) 0x7fafc80e2450 con 0x7fafd0108750 2024-05-27T07:58:24.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafd48d8700 1 --2- 172.21.15.114:0/3081055939 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fafc00b29f0 0x7fafc00b4f00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fafc4011490 tx=0x7fafc4009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:24.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafe5551b80 1 Processor -- start 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafe5551b80 1 -- start start 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafe5551b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 0x55978a1e9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafe5551b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e9cd0 0x55978a1e3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafe5551b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 0x55978a1e4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafe5551b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55978a112d10 con 0x55978a065ec0 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafd48d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 0x55978a1e9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafe5551b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55978a1028b0 con 0x55978a1e9cd0 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafe5551b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55978a1068f0 con 0x55978a1e4000 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafd48d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 0x55978a1e9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40640/0 (socket says 172.21.15.114:40640) 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafd48d8700 1 -- 172.21.15.114:0/670974549 learned_addr learned my addr 172.21.15.114:0/670974549 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafd50d9700 1 --2- 172.21.15.114:0/670974549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 0x55978a1e4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.884+0000 7fafcffff700 1 --2- 172.21.15.114:0/670974549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e9cd0 0x55978a1e3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.885+0000 7fafd48d8700 1 -- 172.21.15.114:0/670974549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 msgr2=0x55978a1e4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.885+0000 7fafd48d8700 1 --2- 172.21.15.114:0/670974549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 0x55978a1e4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.885+0000 7fafd48d8700 1 -- 172.21.15.114:0/670974549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e9cd0 msgr2=0x55978a1e3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.885+0000 7fafd48d8700 1 --2- 172.21.15.114:0/670974549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e9cd0 0x55978a1e3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.885+0000 7fafd48d8700 1 -- 172.21.15.114:0/670974549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55978a052950 con 0x55978a065ec0 2024-05-27T07:58:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.885+0000 7fafcffff700 1 --2- 172.21.15.114:0/670974549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e9cd0 0x55978a1e3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:58:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.885+0000 7fafd48d8700 1 --2- 172.21.15.114:0/670974549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 0x55978a1e9790 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fafc40043b0 tx=0x7fafc4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39a7919d6a34cd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.885+0000 7fafbaffd700 1 -- 172.21.15.114:0/670974549 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4003d10 con 0x55978a065ec0 2024-05-27T07:58:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.885+0000 7fafbaffd700 1 -- 172.21.15.114:0/670974549 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc4011e50 con 0x55978a065ec0 2024-05-27T07:58:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.885+0000 7fafbaffd700 1 -- 172.21.15.114:0/670974549 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc400e610 con 0x55978a065ec0 2024-05-27T07:58:24.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.886+0000 7fafe5551b80 1 -- 172.21.15.114:0/670974549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 msgr2=0x55978a1e9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:24.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.886+0000 7fafe5551b80 1 --2- 172.21.15.114:0/670974549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 0x55978a1e9790 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fafc40043b0 tx=0x7fafc4014000 comp rx=0 tx=0).stop 2024-05-27T07:58:24.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.886+0000 7fafe5551b80 1 -- 172.21.15.114:0/670974549 shutdown_connections 2024-05-27T07:58:24.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.886+0000 7fafe5551b80 1 --2- 172.21.15.114:0/670974549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 0x55978a1e4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.886+0000 7fafe5551b80 1 --2- 172.21.15.114:0/670974549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 0x55978a1e9790 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.886+0000 7fafe5551b80 1 --2- 172.21.15.114:0/670974549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e9cd0 0x55978a1e3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.886+0000 7fafe5551b80 1 -- 172.21.15.114:0/670974549 >> 172.21.15.114:0/670974549 conn(0x55978a0c0b70 msgr2=0x55978a1de6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:24.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.886+0000 7fafe5551b80 1 -- 172.21.15.114:0/670974549 shutdown_connections 2024-05-27T07:58:24.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.886+0000 7fafe5551b80 1 -- 172.21.15.114:0/670974549 wait complete. 2024-05-27T07:58:24.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.887+0000 7fafe5551b80 1 -- start start 2024-05-27T07:58:24.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.887+0000 7fafe5551b80 1 Processor -- start 2024-05-27T07:58:24.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.887+0000 7fafe5551b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a065ec0 0x55978a1ed3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.887+0000 7fafe5551b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e4000 0x55978a1ed920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.887+0000 7fafe5551b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 0x55978a1ede60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.887+0000 7fafe5551b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55978a009020 con 0x55978a1e9cd0 2024-05-27T07:58:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.887+0000 7fafe5551b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55978a0a4840 con 0x55978a1e4000 2024-05-27T07:58:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.887+0000 7fafe5551b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55978a1f07c0 con 0x55978a065ec0 2024-05-27T07:58:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.887+0000 7fafd50d9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 0x55978a1ede60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.887+0000 7fafe5551b80 10 client.0 _ll_register_callbacks cb 0x55978a09d9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafd48d8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a065ec0 0x55978a1ed3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafd50d9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 0x55978a1ede60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40648/0 (socket says 172.21.15.114:40648) 2024-05-27T07:58:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafd50d9700 1 -- 172.21.15.114:0/4005325914 learned_addr learned my addr 172.21.15.114:0/4005325914 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafcffff700 1 --2- 172.21.15.114:0/4005325914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e4000 0x55978a1ed920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafd50d9700 1 -- 172.21.15.114:0/4005325914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a065ec0 msgr2=0x55978a1ed3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafd50d9700 1 --2- 172.21.15.114:0/4005325914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a065ec0 0x55978a1ed3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafd50d9700 1 -- 172.21.15.114:0/4005325914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e4000 msgr2=0x55978a1ed920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafd50d9700 1 --2- 172.21.15.114:0/4005325914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e4000 0x55978a1ed920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafd50d9700 1 -- 172.21.15.114:0/4005325914 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55978a0a1710 con 0x55978a1e9cd0 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafcffff700 1 --2- 172.21.15.114:0/4005325914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e4000 0x55978a1ed920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafd50d9700 1 --2- 172.21.15.114:0/4005325914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 0x55978a1ede60 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fafc800ef00 tx=0x7fafc80e23a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b804bd2a9e2e37bb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafaffff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafaffff700 1 -- 172.21.15.114:0/4005325914 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc80a9ab0 con 0x55978a1e9cd0 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafaffff700 1 -- 172.21.15.114:0/4005325914 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(156..156 src has 1..156) ==== 9554+0+0 (secure 0 0 0) 0x7fafc80b1f00 con 0x55978a1e9cd0 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafe5551b80 1 -- 172.21.15.114:0/4005325914 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafc4011e50 con 0x55978a1e9cd0 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafaffff700 1 -- 172.21.15.114:0/4005325914 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fafc80a93b0 con 0x55978a1e9cd0 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafe5551b80 10 client.19578 fetch_fsmap learned FSMap version 104 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.888+0000 7fafe5551b80 1 -- 172.21.15.114:0/4005325914 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x55978a1fb290 con 0x55978a1e9cd0 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.889+0000 7fafaffff700 1 -- 172.21.15.114:0/4005325914 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc4011e50 con 0x55978a1e9cd0 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.889+0000 7fafaffff700 1 -- 172.21.15.114:0/4005325914 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 104) ==== 4095+0+0 (secure 0 0 0) 0x7fafc80a3ad0 con 0x55978a1e9cd0 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.889+0000 7fafe5551b80 10 client.19578 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:58:24.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.889+0000 7fafe5551b80 10 client.19578 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:24.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.889+0000 7fafe5551b80 20 client.19578 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:24.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.889+0000 7fafe5551b80 1 --2- 172.21.15.114:0/4005325914 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55978a1fdcd0 0x55978a2000f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.889+0000 7fafe5551b80 4 client.19578 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:58:24.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.889+0000 7fafe5551b80 1 -- 172.21.15.114:0/4005325914 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55978a1afa70 con 0x55978a1fdcd0 2024-05-27T07:58:24.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.889+0000 7fafd48d8700 1 --2- 172.21.15.114:0/4005325914 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55978a1fdcd0 0x55978a2000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.889+0000 7fafaffff700 10 client.19578 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:24.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.890+0000 7fafd48d8700 1 --2- 172.21.15.114:0/4005325914 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55978a1fdcd0 0x55978a2000f0 crc :-1 s=READY pgs=492 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e469db96320eb6c server_cookie=93e7ded9e8125c46 in_seq=0 out_seq=0 2024-05-27T07:58:24.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.891+0000 7fafaffff700 1 -- 172.21.15.114:0/4005325914 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55978a1afa70 con 0x55978a1fdcd0 2024-05-27T07:58:24.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.891+0000 7fafaffff700 10 client.19578 handle_command_reply: tid=0 2024-05-27T07:58:24.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.891+0000 7fafe5551b80 1 client.19578 shutdown 2024-05-27T07:58:24.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.891+0000 7fafe5551b80 1 -- 172.21.15.114:0/4005325914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 msgr2=0x55978a1ede60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:24.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.891+0000 7fafe5551b80 1 --2- 172.21.15.114:0/4005325914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 0x55978a1ede60 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fafc800ef00 tx=0x7fafc80e23a0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.892+0000 7fafe5551b80 1 -- 172.21.15.114:0/4005325914 shutdown_connections 2024-05-27T07:58:24.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.892+0000 7fafe5551b80 1 --2- 172.21.15.114:0/4005325914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 0x55978a1ede60 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.892+0000 7fafe5551b80 1 --2- 172.21.15.114:0/4005325914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a065ec0 0x55978a1ed3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.892+0000 7fafe5551b80 1 --2- 172.21.15.114:0/4005325914 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55978a1fdcd0 0x55978a2000f0 crc :-1 s=READY pgs=492 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.892+0000 7fafe5551b80 1 --2- 172.21.15.114:0/4005325914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e4000 0x55978a1ed920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.892+0000 7fafaffff700 0 client.19578 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:24.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.892+0000 7fafe5551b80 1 -- 172.21.15.114:0/4005325914 >> 172.21.15.114:0/4005325914 conn(0x55978a0c0b70 msgr2=0x55978a1e2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:24.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.892+0000 7fafe5551b80 1 -- 172.21.15.114:0/4005325914 shutdown_connections 2024-05-27T07:58:24.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.892+0000 7fafe5551b80 1 -- 172.21.15.114:0/4005325914 wait complete. 2024-05-27T07:58:24.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.892+0000 7fafe5551b80 20 client.19578 trim_cache size 0 max 16384 2024-05-27T07:58:24.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.920+0000 7fafe5551b80 1 Processor -- start 2024-05-27T07:58:24.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.920+0000 7fafe5551b80 1 -- start start 2024-05-27T07:58:24.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.920+0000 7fafe5551b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a065ec0 0x55978a1e3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.920+0000 7fafe5551b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 0x55978a1e7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.920+0000 7fafe5551b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 0x55978a1f9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.920+0000 7fafe5551b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01ae260 con 0x55978a1e9cd0 2024-05-27T07:58:24.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.920+0000 7fafe5551b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01ae930 con 0x55978a065ec0 2024-05-27T07:58:24.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.920+0000 7fafe5551b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01ad1e0 con 0x55978a1e4000 2024-05-27T07:58:24.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.920+0000 7fafd50d9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 0x55978a1f9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.920+0000 7fafd50d9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 0x55978a1f9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40664/0 (socket says 172.21.15.114:40664) 2024-05-27T07:58:24.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.920+0000 7fafd50d9700 1 -- 172.21.15.114:0/1879077791 learned_addr learned my addr 172.21.15.114:0/1879077791 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:24.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.920+0000 7fafcffff700 1 --2- 172.21.15.114:0/1879077791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 0x55978a1e7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.920+0000 7fafd48d8700 1 --2- 172.21.15.114:0/1879077791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a065ec0 0x55978a1e3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.921+0000 7fafcffff700 1 -- 172.21.15.114:0/1879077791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a065ec0 msgr2=0x55978a1e3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:24.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.921+0000 7fafcffff700 1 --2- 172.21.15.114:0/1879077791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a065ec0 0x55978a1e3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.921+0000 7fafcffff700 1 -- 172.21.15.114:0/1879077791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 msgr2=0x55978a1f9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:24.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.921+0000 7fafcffff700 1 --2- 172.21.15.114:0/1879077791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 0x55978a1f9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.921+0000 7fafcffff700 1 -- 172.21.15.114:0/1879077791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55978a1ee4a0 con 0x55978a1e4000 2024-05-27T07:58:24.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.921+0000 7fafd50d9700 1 --2- 172.21.15.114:0/1879077791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 0x55978a1f9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:24.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.921+0000 7fafcffff700 1 --2- 172.21.15.114:0/1879077791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 0x55978a1e7ac0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00b580 tx=0x7fafbc005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66af365857766f02 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:24.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.921+0000 7fafba7fc700 1 -- 172.21.15.114:0/1879077791 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc02f070 con 0x55978a1e4000 2024-05-27T07:58:24.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.921+0000 7fafba7fc700 1 -- 172.21.15.114:0/1879077791 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafbc0055c0 con 0x55978a1e4000 2024-05-27T07:58:24.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.921+0000 7fafba7fc700 1 -- 172.21.15.114:0/1879077791 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc00be00 con 0x55978a1e4000 2024-05-27T07:58:24.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.922+0000 7fafe5551b80 1 -- 172.21.15.114:0/1879077791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 msgr2=0x55978a1e7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:24.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.922+0000 7fafe5551b80 1 --2- 172.21.15.114:0/1879077791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 0x55978a1e7ac0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00b580 tx=0x7fafbc005040 comp rx=0 tx=0).stop 2024-05-27T07:58:24.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.922+0000 7fafe5551b80 1 -- 172.21.15.114:0/1879077791 shutdown_connections 2024-05-27T07:58:24.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.922+0000 7fafe5551b80 1 --2- 172.21.15.114:0/1879077791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a1e9cd0 0x55978a1f9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.922+0000 7fafe5551b80 1 --2- 172.21.15.114:0/1879077791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a065ec0 0x55978a1e3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.922+0000 7fafe5551b80 1 --2- 172.21.15.114:0/1879077791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 0x55978a1e7ac0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.922+0000 7fafe5551b80 1 -- 172.21.15.114:0/1879077791 >> 172.21.15.114:0/1879077791 conn(0x55978a0c0b70 msgr2=0x55978a1ea2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:24.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.922+0000 7fafe5551b80 1 -- 172.21.15.114:0/1879077791 shutdown_connections 2024-05-27T07:58:24.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.922+0000 7fafe5551b80 1 -- 172.21.15.114:0/1879077791 wait complete. 2024-05-27T07:58:24.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.922+0000 7fafe5551b80 1 -- start start 2024-05-27T07:58:24.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.923+0000 7fafe5551b80 1 Processor -- start 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.923+0000 7fafe5551b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 0x55978a1f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.923+0000 7fafe5551b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 0x55978a1f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.923+0000 7fafe5551b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e9cd0 0x55978a1f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.923+0000 7fafe5551b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01acb10 con 0x55978a065ec0 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.923+0000 7fafe5551b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01ac4a0 con 0x55978a1e9cd0 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.923+0000 7fafe5551b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55978a200680 con 0x55978a1e4000 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.923+0000 7fafe5551b80 10 client.0 _ll_register_callbacks cb 0x55978a09d9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.923+0000 7fafd48d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 0x55978a1f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.923+0000 7fafd48d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 0x55978a1f5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40666/0 (socket says 172.21.15.114:40666) 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.923+0000 7fafd48d8700 1 -- 172.21.15.114:0/305013738 learned_addr learned my addr 172.21.15.114:0/305013738 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.923+0000 7fafcffff700 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 0x55978a1f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.923+0000 7fafd50d9700 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e9cd0 0x55978a1f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafd48d8700 1 -- 172.21.15.114:0/305013738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 msgr2=0x55978a1f5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafd48d8700 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 0x55978a1f5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafd48d8700 1 -- 172.21.15.114:0/305013738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e9cd0 msgr2=0x55978a1f2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafd48d8700 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e9cd0 0x55978a1f2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafd48d8700 1 -- 172.21.15.114:0/305013738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55978a1f2fb0 con 0x55978a065ec0 2024-05-27T07:58:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafd50d9700 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e9cd0 0x55978a1f2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafd48d8700 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 0x55978a1f5a50 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fafc4003960 tx=0x7fafc4003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9975c3669f0a99bb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafaffff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:58:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafaffff700 1 -- 172.21.15.114:0/305013738 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4012e60 con 0x55978a065ec0 2024-05-27T07:58:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafaffff700 1 -- 172.21.15.114:0/305013738 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(156..156 src has 1..156) ==== 9554+0+0 (secure 0 0 0) 0x7fafc4005450 con 0x55978a065ec0 2024-05-27T07:58:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafe5551b80 1 -- 172.21.15.114:0/305013738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafbc0055c0 con 0x55978a065ec0 2024-05-27T07:58:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafaffff700 1 -- 172.21.15.114:0/305013738 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fafc4045050 con 0x55978a065ec0 2024-05-27T07:58:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafe5551b80 10 client.19584 fetch_fsmap learned FSMap version 104 2024-05-27T07:58:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafe5551b80 1 -- 172.21.15.114:0/305013738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x55978a268fa0 con 0x55978a065ec0 2024-05-27T07:58:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.924+0000 7fafaffff700 1 -- 172.21.15.114:0/305013738 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafbc0055c0 con 0x55978a065ec0 2024-05-27T07:58:24.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.925+0000 7fafaffff700 1 -- 172.21.15.114:0/305013738 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 104) ==== 4095+0+0 (secure 0 0 0) 0x7fafc401e480 con 0x55978a065ec0 2024-05-27T07:58:24.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.925+0000 7fafe5551b80 10 client.19584 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:58:24.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.925+0000 7fafe5551b80 10 client.19584 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:24.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.925+0000 7fafe5551b80 20 client.19584 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:24.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.925+0000 7fafe5551b80 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55978a269660 0x55978a26ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:24.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.925+0000 7fafe5551b80 4 client.19584 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:58:24.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.925+0000 7fafe5551b80 1 -- 172.21.15.114:0/305013738 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55978a114c60 con 0x55978a269660 2024-05-27T07:58:24.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.925+0000 7fafcffff700 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55978a269660 0x55978a26ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:24.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.926+0000 7fafcffff700 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55978a269660 0x55978a26ba80 crc :-1 s=READY pgs=493 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fc65b4dd0110972 server_cookie=e941a42903fe3040 in_seq=0 out_seq=0 2024-05-27T07:58:24.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.926+0000 7fafaffff700 10 client.19584 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:24.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafaffff700 1 -- 172.21.15.114:0/305013738 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55978a114c60 con 0x55978a269660 2024-05-27T07:58:24.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafaffff700 10 client.19584 handle_command_reply: tid=0 2024-05-27T07:58:24.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafe5551b80 1 client.19584 shutdown 2024-05-27T07:58:24.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafe5551b80 1 -- 172.21.15.114:0/305013738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 msgr2=0x55978a1f5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:24.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafe5551b80 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 0x55978a1f5a50 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fafc4003960 tx=0x7fafc4003f80 comp rx=0 tx=0).stop 2024-05-27T07:58:24.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafe5551b80 1 -- 172.21.15.114:0/305013738 shutdown_connections 2024-05-27T07:58:24.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafe5551b80 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55978a269660 0x55978a26ba80 crc :-1 s=READY pgs=493 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafe5551b80 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55978a1e9cd0 0x55978a1f2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafe5551b80 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55978a065ec0 0x55978a1f5a50 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafe5551b80 1 --2- 172.21.15.114:0/305013738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55978a1e4000 0x55978a1f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafaffff700 0 client.19584 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:24.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafe5551b80 1 -- 172.21.15.114:0/305013738 >> 172.21.15.114:0/305013738 conn(0x55978a0c0b70 msgr2=0x55978a1e7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:24.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafe5551b80 1 -- 172.21.15.114:0/305013738 shutdown_connections 2024-05-27T07:58:24.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafe5551b80 1 -- 172.21.15.114:0/305013738 wait complete. 2024-05-27T07:58:24.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.933+0000 7fafe5551b80 20 client.19584 trim_cache size 0 max 16384 2024-05-27T07:58:24.935 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:58:24.935 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:58:24.935 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "2fd0f88a-865d-4963-b562-2f6b3726c1a6", 2024-05-27T07:58:24.935 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:58:24.935 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:58:24.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.934+0000 7fafd6b3c700 1 -- 172.21.15.114:0/3081055939 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fafc00b29f0 msgr2=0x7fafc00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:24.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.934+0000 7fafd6b3c700 1 --2- 172.21.15.114:0/3081055939 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fafc00b29f0 0x7fafc00b4f00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fafc4011490 tx=0x7fafc4009260 comp rx=0 tx=0).stop 2024-05-27T07:58:24.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.934+0000 7fafd6b3c700 1 -- 172.21.15.114:0/3081055939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafd0108750 msgr2=0x7fafd019e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:24.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.934+0000 7fafd6b3c700 1 --2- 172.21.15.114:0/3081055939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafd0108750 0x7fafd019e600 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fafc800ef30 tx=0x7fafc800c3a0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.934+0000 7fafd6b3c700 1 -- 172.21.15.114:0/3081055939 shutdown_connections 2024-05-27T07:58:24.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.934+0000 7fafd6b3c700 1 --2- 172.21.15.114:0/3081055939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafd0108750 0x7fafd019e600 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.934+0000 7fafd6b3c700 1 --2- 172.21.15.114:0/3081055939 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fafc00b29f0 0x7fafc00b4f00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.934+0000 7fafd6b3c700 1 --2- 172.21.15.114:0/3081055939 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafd01034d0 0x7fafd0199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.934+0000 7fafd6b3c700 1 --2- 172.21.15.114:0/3081055939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafd0107e00 0x7fafd019a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:24.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.934+0000 7fafd6b3c700 1 -- 172.21.15.114:0/3081055939 >> 172.21.15.114:0/3081055939 conn(0x7fafd00fef20 msgr2=0x7fafd01075c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:24.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.934+0000 7fafd6b3c700 1 -- 172.21.15.114:0/3081055939 shutdown_connections 2024-05-27T07:58:24.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:24.934+0000 7fafd6b3c700 1 -- 172.21.15.114:0/3081055939 wait complete. 2024-05-27T07:58:25.615 INFO:tasks.workunit.client.0.smithi114.stdout: 10 65500 12.04 MB/sec execute 157 sec latency 205.593 ms 2024-05-27T07:58:26.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:25 smithi163 ceph-mon[36451]: pgmap v640: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 363 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 452 op/s 2024-05-27T07:58:26.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:25 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:58:26.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:25 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:58:26.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:25 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:58:26.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:25 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:58:26.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:25 smithi186 ceph-mon[36647]: pgmap v640: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 363 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 452 op/s 2024-05-27T07:58:26.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:25 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:58:26.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:25 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:58:26.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:25 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:58:26.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:25 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:58:26.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:25 smithi114 ceph-mon[32038]: pgmap v640: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 363 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 452 op/s 2024-05-27T07:58:26.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:25 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:58:26.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:25 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:58:26.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:25 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:58:26.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:25 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:58:26.615 INFO:tasks.workunit.client.0.smithi114.stdout: 10 65786 12.07 MB/sec execute 158 sec latency 222.911 ms 2024-05-27T07:58:27.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:58:27.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:26 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:58:27.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:26 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:58:27.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:26 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:58:27.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:26 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:58:27.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:26 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:58:27.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:26 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:58:27.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:26 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:58:27.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:26 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:58:27.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:26 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:58:27.446 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:58:27.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 65921 12.03 MB/sec execute 159 sec latency 574.915 ms 2024-05-27T07:58:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:27 smithi163 ceph-mon[36451]: pgmap v641: 129 pgs: 129 active+clean; 365 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.4 MiB/s wr, 534 op/s 2024-05-27T07:58:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:27 smithi163 ceph-mon[36451]: pgmap v642: 129 pgs: 129 active+clean; 365 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 483 op/s 2024-05-27T07:58:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:27 smithi163 ceph-mon[36451]: pgmap v643: 129 pgs: 129 active+clean; 365 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.9 MiB/s wr, 535 op/s 2024-05-27T07:58:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:27 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:58:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:27 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:58:28.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:27 smithi186 ceph-mon[36647]: pgmap v641: 129 pgs: 129 active+clean; 365 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.4 MiB/s wr, 534 op/s 2024-05-27T07:58:28.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:27 smithi186 ceph-mon[36647]: pgmap v642: 129 pgs: 129 active+clean; 365 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 483 op/s 2024-05-27T07:58:28.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:27 smithi186 ceph-mon[36647]: pgmap v643: 129 pgs: 129 active+clean; 365 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.9 MiB/s wr, 535 op/s 2024-05-27T07:58:28.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:27 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:58:28.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:27 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:58:28.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:27 smithi114 ceph-mon[32038]: pgmap v641: 129 pgs: 129 active+clean; 365 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.4 MiB/s wr, 534 op/s 2024-05-27T07:58:28.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:27 smithi114 ceph-mon[32038]: pgmap v642: 129 pgs: 129 active+clean; 365 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 483 op/s 2024-05-27T07:58:28.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:27 smithi114 ceph-mon[32038]: pgmap v643: 129 pgs: 129 active+clean; 365 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.9 MiB/s wr, 535 op/s 2024-05-27T07:58:28.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:27 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:58:28.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:27 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:58:28.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 66239 12.10 MB/sec execute 160 sec latency 190.425 ms 2024-05-27T07:58:28.807 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:28 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:58:28.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:28 smithi114 systemd[1]: Stopped Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:58:28.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:28 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.210s CPU time. 2024-05-27T07:58:28.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:28 smithi114 systemd[1]: Starting Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:58:29.226 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:28 smithi114 ceph-mon[32038]: pgmap v644: 129 pgs: 129 active+clean; 368 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.9 MiB/s wr, 607 op/s 2024-05-27T07:58:29.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:28 smithi163 ceph-mon[36451]: pgmap v644: 129 pgs: 129 active+clean; 368 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.9 MiB/s wr, 607 op/s 2024-05-27T07:58:29.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:28 smithi186 ceph-mon[36647]: pgmap v644: 129 pgs: 129 active+clean; 368 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.9 MiB/s wr, 607 op/s 2024-05-27T07:58:29.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 66590 12.16 MB/sec execute 161 sec latency 192.170 ms 2024-05-27T07:58:30.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.003+0000 7f59ebe42700 1 -- 172.21.15.114:0/3244674196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4108f00 msgr2=0x7f59e4109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:30.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.003+0000 7f59ebe42700 1 --2- 172.21.15.114:0/3244674196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4108f00 0x7f59e4109310 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f59d40099c0 tx=0x7f59d4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.003+0000 7f59ebe42700 1 -- 172.21.15.114:0/3244674196 shutdown_connections 2024-05-27T07:58:30.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.003+0000 7f59ebe42700 1 --2- 172.21.15.114:0/3244674196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e4109850 0x7f59e41111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.003+0000 7f59ebe42700 1 --2- 172.21.15.114:0/3244674196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e41045d0 0x7f59e41049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.003+0000 7f59ebe42700 1 --2- 172.21.15.114:0/3244674196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4108f00 0x7f59e4109310 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.003+0000 7f59ebe42700 1 -- 172.21.15.114:0/3244674196 >> 172.21.15.114:0/3244674196 conn(0x7f59e40ffec0 msgr2=0x7f59e4102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:30.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.003+0000 7f59ebe42700 1 -- 172.21.15.114:0/3244674196 shutdown_connections 2024-05-27T07:58:30.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.003+0000 7f59ebe42700 1 -- 172.21.15.114:0/3244674196 wait complete. 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ebe42700 1 Processor -- start 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ebe42700 1 -- start start 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ebe42700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e41045d0 0x7f59e419ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ebe42700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e4108f00 0x7f59e419b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ebe42700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4109850 0x7f59e419f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ebe42700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f59e4114460 con 0x7f59e4109850 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ebe42700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41142e0 con 0x7f59e4108f00 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ebe42700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41145e0 con 0x7f59e41045d0 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59e9bde700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e41045d0 0x7f59e419ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59e93dd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e4108f00 0x7f59e419b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ea3df700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4109850 0x7f59e419f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59e9bde700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e41045d0 0x7f59e419ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:50194/0 (socket says 172.21.15.114:50194) 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59e93dd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e4108f00 0x7f59e419b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37428/0 (socket says 172.21.15.114:37428) 2024-05-27T07:58:30.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59e9bde700 1 -- 172.21.15.114:0/811317000 learned_addr learned my addr 172.21.15.114:0/811317000 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:30.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ea3df700 1 -- 172.21.15.114:0/811317000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e41045d0 msgr2=0x7f59e419ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ea3df700 1 --2- 172.21.15.114:0/811317000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e41045d0 0x7f59e419ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ea3df700 1 -- 172.21.15.114:0/811317000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e4108f00 msgr2=0x7f59e419b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ea3df700 1 --2- 172.21.15.114:0/811317000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e4108f00 0x7f59e419b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59ea3df700 1 -- 172.21.15.114:0/811317000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e419fd20 con 0x7f59e4109850 2024-05-27T07:58:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.004+0000 7f59e93dd700 1 --2- 172.21.15.114:0/811317000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e4108f00 0x7f59e419b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.005+0000 7f59ea3df700 1 --2- 172.21.15.114:0/811317000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4109850 0x7f59e419f680 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f59dc007f10 tx=0x7f59dc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.005+0000 7f59daffd700 1 -- 172.21.15.114:0/811317000 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc004db0 con 0x7f59e4109850 2024-05-27T07:58:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.005+0000 7f59daffd700 1 -- 172.21.15.114:0/811317000 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59dc003a00 con 0x7f59e4109850 2024-05-27T07:58:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.005+0000 7f59daffd700 1 -- 172.21.15.114:0/811317000 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc004580 con 0x7f59e4109850 2024-05-27T07:58:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.005+0000 7f59ebe42700 1 -- 172.21.15.114:0/811317000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e41a0010 con 0x7f59e4109850 2024-05-27T07:58:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.005+0000 7f59ebe42700 1 -- 172.21.15.114:0/811317000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e41a7930 con 0x7f59e4109850 2024-05-27T07:58:30.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.008+0000 7f59daffd700 1 -- 172.21.15.114:0/811317000 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f59dc01c020 con 0x7f59e4109850 2024-05-27T07:58:30.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.009+0000 7f59daffd700 1 --2- 172.21.15.114:0/811317000 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f59d00b2a70 0x7f59d00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.009+0000 7f59daffd700 1 -- 172.21.15.114:0/811317000 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7f59dc0e2050 con 0x7f59e4109850 2024-05-27T07:58:30.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.009+0000 7f59e9bde700 1 --2- 172.21.15.114:0/811317000 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f59d00b2a70 0x7f59d00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.009+0000 7f59fa857b80 1 Processor -- start 2024-05-27T07:58:30.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.009+0000 7f59fa857b80 1 -- start start 2024-05-27T07:58:30.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59fa857b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5644909a6e40 0x564490b2a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59fa857b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b2ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59fa857b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564490b2efb0 0x564490b2f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59fa857b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564490a53d70 con 0x564490b2efb0 2024-05-27T07:58:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59fa857b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564490a43910 con 0x564490b2a7d0 2024-05-27T07:58:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59fa857b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564490a47950 con 0x5644909a6e40 2024-05-27T07:58:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59e9bde700 1 --2- 172.21.15.114:0/811317000 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f59d00b2a70 0x7f59d00b4f80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f59e0009d20 tx=0x7f59e0009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59e93dd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b2ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59e93dd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b2ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37436/0 (socket says 172.21.15.114:37436) 2024-05-27T07:58:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59ea3df700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564490b2efb0 0x564490b2f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59e93dd700 1 -- 172.21.15.114:0/158725534 learned_addr learned my addr 172.21.15.114:0/158725534 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59e9bde700 1 --2- 172.21.15.114:0/158725534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5644909a6e40 0x564490b2a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59e93dd700 1 -- 172.21.15.114:0/158725534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5644909a6e40 msgr2=0x564490b2a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59e93dd700 1 --2- 172.21.15.114:0/158725534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5644909a6e40 0x564490b2a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59e93dd700 1 -- 172.21.15.114:0/158725534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564490b2efb0 msgr2=0x564490b2f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59e93dd700 1 --2- 172.21.15.114:0/158725534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564490b2efb0 0x564490b2f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59e93dd700 1 -- 172.21.15.114:0/158725534 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644909ecfb0 con 0x564490b2a7d0 2024-05-27T07:58:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.010+0000 7f59e93dd700 1 --2- 172.21.15.114:0/158725534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b2ea70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f59d40099c0 tx=0x7f59d4005fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de5af75252de29eb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.011+0000 7f59cffff700 1 -- 172.21.15.114:0/158725534 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4027070 con 0x564490b2a7d0 2024-05-27T07:58:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.011+0000 7f59cffff700 1 -- 172.21.15.114:0/158725534 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59d4005550 con 0x564490b2a7d0 2024-05-27T07:58:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.011+0000 7f59cffff700 1 -- 172.21.15.114:0/158725534 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4003db0 con 0x564490b2a7d0 2024-05-27T07:58:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.012+0000 7f59fa857b80 1 -- 172.21.15.114:0/158725534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 msgr2=0x564490b2ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.012+0000 7f59fa857b80 1 --2- 172.21.15.114:0/158725534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b2ea70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f59d40099c0 tx=0x7f59d4005fb0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.012+0000 7f59fa857b80 1 -- 172.21.15.114:0/158725534 shutdown_connections 2024-05-27T07:58:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.012+0000 7f59fa857b80 1 --2- 172.21.15.114:0/158725534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564490b2efb0 0x564490b2f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.012+0000 7f59fa857b80 1 --2- 172.21.15.114:0/158725534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5644909a6e40 0x564490b2a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.012+0000 7f59fa857b80 1 --2- 172.21.15.114:0/158725534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b2ea70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.012+0000 7f59fa857b80 1 -- 172.21.15.114:0/158725534 >> 172.21.15.114:0/158725534 conn(0x5644909a9770 msgr2=0x564490b20d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.012+0000 7f59fa857b80 1 -- 172.21.15.114:0/158725534 shutdown_connections 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.012+0000 7f59fa857b80 1 -- 172.21.15.114:0/158725534 wait complete. 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.012+0000 7f59fa857b80 1 -- start start 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.012+0000 7f59fa857b80 1 Processor -- start 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.013+0000 7f59fa857b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 0x564490b33cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.013+0000 7f59fa857b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b34200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.013+0000 7f59fa857b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564490b2efb0 0x564490b34740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.013+0000 7f59fa857b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564490949fa0 con 0x5644909a6e40 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.013+0000 7f59e9bde700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 0x564490b33cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.013+0000 7f59fa857b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5644909abda0 con 0x564490b2a7d0 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.013+0000 7f59fa857b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564490b36f60 con 0x564490b2efb0 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.013+0000 7f59e9bde700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 0x564490b33cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40688/0 (socket says 172.21.15.114:40688) 2024-05-27T07:58:30.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.013+0000 7f59e9bde700 1 -- 172.21.15.114:0/127448682 learned_addr learned my addr 172.21.15.114:0/127448682 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.013+0000 7f59fa857b80 10 client.0 _ll_register_callbacks cb 0x5644909da5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.013+0000 7f59e93dd700 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b34200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.013+0000 7f59ea3df700 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564490b2efb0 0x564490b34740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59e9bde700 1 -- 172.21.15.114:0/127448682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564490b2efb0 msgr2=0x564490b34740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59e9bde700 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564490b2efb0 0x564490b34740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59e9bde700 1 -- 172.21.15.114:0/127448682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 msgr2=0x564490b34200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59e9bde700 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b34200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59e9bde700 1 -- 172.21.15.114:0/127448682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644909938d0 con 0x5644909a6e40 2024-05-27T07:58:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59e93dd700 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b34200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59e9bde700 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 0x564490b33cc0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f59e001ec60 tx=0x7f59e00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1005d8b51f959568 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59cd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:58:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59cd7fa700 1 -- 172.21.15.114:0/127448682 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e000e840 con 0x5644909a6e40 2024-05-27T07:58:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59fa857b80 1 -- 172.21.15.114:0/127448682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59d4005550 con 0x5644909a6e40 2024-05-27T07:58:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59cd7fa700 1 -- 172.21.15.114:0/127448682 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7f59e000e9e0 con 0x5644909a6e40 2024-05-27T07:58:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59cd7fa700 1 -- 172.21.15.114:0/127448682 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7f59e000ebe0 con 0x5644909a6e40 2024-05-27T07:58:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59fa857b80 10 client.19596 fetch_fsmap learned FSMap version 104 2024-05-27T07:58:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.014+0000 7f59fa857b80 1 -- 172.21.15.114:0/127448682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x5644909fd760 con 0x5644909a6e40 2024-05-27T07:58:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.015+0000 7f59cd7fa700 1 -- 172.21.15.114:0/127448682 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59d4005550 con 0x5644909a6e40 2024-05-27T07:58:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.015+0000 7f59cd7fa700 1 -- 172.21.15.114:0/127448682 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 104) ==== 4095+0+0 (secure 0 0 0) 0x7f59e0025070 con 0x5644909a6e40 2024-05-27T07:58:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.015+0000 7f59fa857b80 10 client.19596 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:58:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.015+0000 7f59fa857b80 10 client.19596 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.015+0000 7f59fa857b80 20 client.19596 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.015+0000 7f59fa857b80 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564490b44330 0x564490b467e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.015+0000 7f59fa857b80 4 client.19596 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:58:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.015+0000 7f59fa857b80 1 -- 172.21.15.114:0/127448682 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564490a55cc0 con 0x564490b44330 2024-05-27T07:58:30.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.015+0000 7f59e93dd700 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564490b44330 0x564490b467e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.015+0000 7f59cd7fa700 10 client.19596 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:30.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.016+0000 7f59e93dd700 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564490b44330 0x564490b467e0 crc :-1 s=READY pgs=494 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd9d414447dec195 server_cookie=a2de95dafb09168f in_seq=0 out_seq=0 2024-05-27T07:58:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.018+0000 7f59cd7fa700 1 -- 172.21.15.114:0/127448682 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564490a55cc0 con 0x564490b44330 2024-05-27T07:58:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.018+0000 7f59cd7fa700 10 client.19596 handle_command_reply: tid=0 2024-05-27T07:58:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.018+0000 7f59fa857b80 1 client.19596 shutdown 2024-05-27T07:58:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.019+0000 7f59fa857b80 1 -- 172.21.15.114:0/127448682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 msgr2=0x564490b33cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.019+0000 7f59fa857b80 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 0x564490b33cc0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f59e001ec60 tx=0x7f59e00047c0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.019+0000 7f59fa857b80 1 -- 172.21.15.114:0/127448682 shutdown_connections 2024-05-27T07:58:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.019+0000 7f59fa857b80 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564490b2efb0 0x564490b34740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.019+0000 7f59fa857b80 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 0x564490b33cc0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.019+0000 7f59fa857b80 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564490b44330 0x564490b467e0 crc :-1 s=READY pgs=494 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.019+0000 7f59fa857b80 1 --2- 172.21.15.114:0/127448682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b34200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.019+0000 7f59cd7fa700 0 client.19596 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:30.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.019+0000 7f59fa857b80 1 -- 172.21.15.114:0/127448682 >> 172.21.15.114:0/127448682 conn(0x5644909a9770 msgr2=0x564490b24bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:30.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.019+0000 7f59fa857b80 1 -- 172.21.15.114:0/127448682 shutdown_connections 2024-05-27T07:58:30.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.019+0000 7f59fa857b80 1 -- 172.21.15.114:0/127448682 wait complete. 2024-05-27T07:58:30.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.019+0000 7f59fa857b80 20 client.19596 trim_cache size 0 max 16384 2024-05-27T07:58:30.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.041+0000 7f59fa857b80 1 Processor -- start 2024-05-27T07:58:30.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.041+0000 7f59fa857b80 1 -- start start 2024-05-27T07:58:30.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59fa857b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5644909a6e40 0x564490b25600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59fa857b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b34580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59fa857b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564490b2efb0 0x564490b2e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59fa857b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41ae090 con 0x564490b2efb0 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59ea3df700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564490b2efb0 0x564490b2e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59e9bde700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5644909a6e40 0x564490b25600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59ea3df700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564490b2efb0 0x564490b2e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40704/0 (socket says 172.21.15.114:40704) 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59ea3df700 1 -- 172.21.15.114:0/3490722723 learned_addr learned my addr 172.21.15.114:0/3490722723 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59fa857b80 1 -- 172.21.15.114:0/3490722723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41ae760 con 0x564490b2a7d0 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59fa857b80 1 -- 172.21.15.114:0/3490722723 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41ad010 con 0x5644909a6e40 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59e93dd700 1 --2- 172.21.15.114:0/3490722723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b34580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59ea3df700 1 -- 172.21.15.114:0/3490722723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5644909a6e40 msgr2=0x564490b25600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59ea3df700 1 --2- 172.21.15.114:0/3490722723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5644909a6e40 0x564490b25600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59ea3df700 1 -- 172.21.15.114:0/3490722723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 msgr2=0x564490b34580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59ea3df700 1 --2- 172.21.15.114:0/3490722723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b34580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59ea3df700 1 -- 172.21.15.114:0/3490722723 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564490b1caa0 con 0x564490b2efb0 2024-05-27T07:58:30.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.042+0000 7f59ea3df700 1 --2- 172.21.15.114:0/3490722723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564490b2efb0 0x564490b2e8b0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0df990 tx=0x7f59dc0096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8803680d0f3d11d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:30.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.043+0000 7f59cf7fe700 1 -- 172.21.15.114:0/3490722723 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc0c3070 con 0x564490b2efb0 2024-05-27T07:58:30.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.043+0000 7f59cf7fe700 1 -- 172.21.15.114:0/3490722723 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59dc0ad2d0 con 0x564490b2efb0 2024-05-27T07:58:30.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.043+0000 7f59cf7fe700 1 -- 172.21.15.114:0/3490722723 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc019070 con 0x564490b2efb0 2024-05-27T07:58:30.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.043+0000 7f59fa857b80 1 -- 172.21.15.114:0/3490722723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564490b2efb0 msgr2=0x564490b2e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:30.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.043+0000 7f59fa857b80 1 --2- 172.21.15.114:0/3490722723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564490b2efb0 0x564490b2e8b0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0df990 tx=0x7f59dc0096c0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.044+0000 7f59fa857b80 1 -- 172.21.15.114:0/3490722723 shutdown_connections 2024-05-27T07:58:30.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.044+0000 7f59fa857b80 1 --2- 172.21.15.114:0/3490722723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564490b2efb0 0x564490b2e8b0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.044+0000 7f59fa857b80 1 --2- 172.21.15.114:0/3490722723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5644909a6e40 0x564490b25600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.044+0000 7f59fa857b80 1 --2- 172.21.15.114:0/3490722723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2a7d0 0x564490b34580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.044+0000 7f59fa857b80 1 -- 172.21.15.114:0/3490722723 >> 172.21.15.114:0/3490722723 conn(0x5644909a9770 msgr2=0x564490b1a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:30.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.044+0000 7f59fa857b80 1 -- 172.21.15.114:0/3490722723 shutdown_connections 2024-05-27T07:58:30.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.044+0000 7f59fa857b80 1 -- 172.21.15.114:0/3490722723 wait complete. 2024-05-27T07:58:30.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.044+0000 7f59fa857b80 1 -- start start 2024-05-27T07:58:30.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.044+0000 7f59fa857b80 1 Processor -- start 2024-05-27T07:58:30.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59fa857b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 0x564490b3c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59fa857b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564490b2a7d0 0x564490b4b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59fa857b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2efb0 0x564490b4bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59fa857b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41ac960 con 0x5644909a6e40 2024-05-27T07:58:30.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59fa857b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41ac310 con 0x564490b2efb0 2024-05-27T07:58:30.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59fa857b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564490b2edf0 con 0x564490b2a7d0 2024-05-27T07:58:30.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59e9bde700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 0x564490b3c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59fa857b80 10 client.0 _ll_register_callbacks cb 0x5644909da5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59e9bde700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 0x564490b3c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40712/0 (socket says 172.21.15.114:40712) 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59e9bde700 1 -- 172.21.15.114:0/3733775840 learned_addr learned my addr 172.21.15.114:0/3733775840 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59e93dd700 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564490b2a7d0 0x564490b4b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59ea3df700 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2efb0 0x564490b4bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59e9bde700 1 -- 172.21.15.114:0/3733775840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564490b2a7d0 msgr2=0x564490b4b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59e9bde700 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564490b2a7d0 0x564490b4b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59e9bde700 1 -- 172.21.15.114:0/3733775840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2efb0 msgr2=0x564490b4bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59e9bde700 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2efb0 0x564490b4bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59e9bde700 1 -- 172.21.15.114:0/3733775840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564490b4c190 con 0x5644909a6e40 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59ea3df700 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2efb0 0x564490b4bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59e93dd700 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564490b2a7d0 0x564490b4b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59e9bde700 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 0x564490b3c740 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f59e001d510 tx=0x7f59e000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6280cdfa661b2c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:30.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.045+0000 7f59cd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:58:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.046+0000 7f59cd7fa700 1 -- 172.21.15.114:0/3733775840 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0039030 con 0x5644909a6e40 2024-05-27T07:58:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.046+0000 7f59cd7fa700 1 -- 172.21.15.114:0/3733775840 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7f59e001de00 con 0x5644909a6e40 2024-05-27T07:58:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.046+0000 7f59fa857b80 1 -- 172.21.15.114:0/3733775840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59dc0ad2d0 con 0x5644909a6e40 2024-05-27T07:58:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.046+0000 7f59cd7fa700 1 -- 172.21.15.114:0/3733775840 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7f59e0043050 con 0x5644909a6e40 2024-05-27T07:58:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.046+0000 7f59fa857b80 10 client.19602 fetch_fsmap learned FSMap version 104 2024-05-27T07:58:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.046+0000 7f59fa857b80 1 -- 172.21.15.114:0/3733775840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x564490baec10 con 0x5644909a6e40 2024-05-27T07:58:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.046+0000 7f59cd7fa700 1 -- 172.21.15.114:0/3733775840 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59dc0ad2d0 con 0x5644909a6e40 2024-05-27T07:58:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.046+0000 7f59cd7fa700 1 -- 172.21.15.114:0/3733775840 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 104) ==== 4095+0+0 (secure 0 0 0) 0x7f59e0025040 con 0x5644909a6e40 2024-05-27T07:58:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.046+0000 7f59fa857b80 10 client.19602 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:58:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.046+0000 7f59fa857b80 10 client.19602 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.046+0000 7f59fa857b80 20 client.19602 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.046+0000 7f59fa857b80 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564490baf310 0x564490bb1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.047+0000 7f59fa857b80 4 client.19602 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.047+0000 7f59fa857b80 1 -- 172.21.15.114:0/3733775840 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564490a4f4a0 con 0x564490baf310 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.047+0000 7f59e93dd700 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564490baf310 0x564490bb1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.047+0000 7f59cd7fa700 10 client.19602 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.047+0000 7f59e93dd700 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564490baf310 0x564490bb1730 crc :-1 s=READY pgs=495 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bc1ac492127d50f server_cookie=58d88495fa707717 in_seq=0 out_seq=0 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59cd7fa700 1 -- 172.21.15.114:0/3733775840 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564490a4f4a0 con 0x564490baf310 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59cd7fa700 10 client.19602 handle_command_reply: tid=0 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59fa857b80 1 client.19602 shutdown 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59fa857b80 1 -- 172.21.15.114:0/3733775840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 msgr2=0x564490b3c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59fa857b80 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 0x564490b3c740 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f59e001d510 tx=0x7f59e000ee00 comp rx=0 tx=0).stop 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59fa857b80 1 -- 172.21.15.114:0/3733775840 shutdown_connections 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59fa857b80 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x564490baf310 0x564490bb1730 crc :-1 s=READY pgs=495 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59fa857b80 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564490b2efb0 0x564490b4bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59fa857b80 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644909a6e40 0x564490b3c740 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59fa857b80 1 --2- 172.21.15.114:0/3733775840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564490b2a7d0 0x564490b4b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59fa857b80 1 -- 172.21.15.114:0/3733775840 >> 172.21.15.114:0/3733775840 conn(0x5644909a9770 msgr2=0x564490b321a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:30.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59cd7fa700 0 client.19602 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:30.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59fa857b80 1 -- 172.21.15.114:0/3733775840 shutdown_connections 2024-05-27T07:58:30.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59fa857b80 1 -- 172.21.15.114:0/3733775840 wait complete. 2024-05-27T07:58:30.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.049+0000 7f59fa857b80 20 client.19602 trim_cache size 0 max 16384 2024-05-27T07:58:30.056 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:58:30.056 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:58:30.056 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:58:30.056 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:58:30.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.050+0000 7f59ebe42700 1 -- 172.21.15.114:0/811317000 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f59d00b2a70 msgr2=0x7f59d00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:30.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.050+0000 7f59ebe42700 1 --2- 172.21.15.114:0/811317000 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f59d00b2a70 0x7f59d00b4f80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f59e0009d20 tx=0x7f59e0009260 comp rx=0 tx=0).stop 2024-05-27T07:58:30.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.050+0000 7f59ebe42700 1 -- 172.21.15.114:0/811317000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4109850 msgr2=0x7f59e419f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:30.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.050+0000 7f59ebe42700 1 --2- 172.21.15.114:0/811317000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4109850 0x7f59e419f680 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f59dc007f10 tx=0x7f59dc00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.050+0000 7f59ebe42700 1 -- 172.21.15.114:0/811317000 shutdown_connections 2024-05-27T07:58:30.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.050+0000 7f59ebe42700 1 --2- 172.21.15.114:0/811317000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4109850 0x7f59e419f680 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.050+0000 7f59ebe42700 1 --2- 172.21.15.114:0/811317000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e41045d0 0x7f59e419ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.050+0000 7f59ebe42700 1 --2- 172.21.15.114:0/811317000 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f59d00b2a70 0x7f59d00b4f80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.050+0000 7f59ebe42700 1 --2- 172.21.15.114:0/811317000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e4108f00 0x7f59e419b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:30.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.050+0000 7f59ebe42700 1 -- 172.21.15.114:0/811317000 >> 172.21.15.114:0/811317000 conn(0x7f59e40ffec0 msgr2=0x7f59e410a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:30.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.050+0000 7f59ebe42700 1 -- 172.21.15.114:0/811317000 shutdown_connections 2024-05-27T07:58:30.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:30.050+0000 7f59ebe42700 1 -- 172.21.15.114:0/811317000 wait complete. 2024-05-27T07:58:30.226 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:30 smithi114 podman[131398]: 2024-05-27 07:58:29.984169982 +0000 UTC m=+0.021068013 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:58:30.227 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:30 smithi114 podman[131398]: 2024-05-27 07:58:30.094276209 +0000 UTC m=+0.131174221 container create d9064388867dde7e37d9ad4432fcf11a73f6384ff76dc5a6a7a92f636609970d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=) 2024-05-27T07:58:30.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:30 smithi114 ceph-mon[32038]: osdmap e157: 12 total, 12 up, 12 in 2024-05-27T07:58:30.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 66933 12.16 MB/sec execute 162 sec latency 185.986 ms 2024-05-27T07:58:30.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:30 smithi163 ceph-mon[36451]: osdmap e157: 12 total, 12 up, 12 in 2024-05-27T07:58:30.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:30 smithi186 ceph-mon[36647]: osdmap e157: 12 total, 12 up, 12 in 2024-05-27T07:58:30.980 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:58:30.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:58:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:58:31.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 podman[131398]: 2024-05-27 07:58:31.086988708 +0000 UTC m=+1.123886729 container init d9064388867dde7e37d9ad4432fcf11a73f6384ff76dc5a6a7a92f636609970d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-27T07:58:31.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 podman[131398]: 2024-05-27 07:58:31.091351249 +0000 UTC m=+1.128249272 container start d9064388867dde7e37d9ad4432fcf11a73f6384ff76dc5a6a7a92f636609970d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:58:31.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-mds[131494]: -- 172.21.15.114:0/328802323 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c6c41aba40 con 0x55c6c41f3a80 2024-05-27T07:58:31.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: starting mds.g at 2024-05-27T07:58:31.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 bash[131398]: d9064388867dde7e37d9ad4432fcf11a73f6384ff76dc5a6a7a92f636609970d 2024-05-27T07:58:31.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 systemd[1]: Started Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:58:31.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 67232 12.21 MB/sec execute 163 sec latency 47.476 ms 2024-05-27T07:58:31.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:31 smithi163 ceph-mon[36451]: pgmap v646: 129 pgs: 129 active+clean; 368 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 377 op/s 2024-05-27T07:58:31.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:31 smithi163 ceph-mon[36451]: daemon mds.g restarted 2024-05-27T07:58:31.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:31 smithi186 ceph-mon[36647]: pgmap v646: 129 pgs: 129 active+clean; 368 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 377 op/s 2024-05-27T07:58:31.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:31 smithi186 ceph-mon[36647]: daemon mds.g restarted 2024-05-27T07:58:31.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:31 smithi114 ceph-mon[32038]: pgmap v646: 129 pgs: 129 active+clean; 368 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 377 op/s 2024-05-27T07:58:31.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:31 smithi114 ceph-mon[32038]: daemon mds.g restarted 2024-05-27T07:58:31.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7b25833700 time 2024-05-27T07:58:31.480730+0000 2024-05-27T07:58:31.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:58:31.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:31.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7b38078ba6] 2024-05-27T07:58:31.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7b38078dd1] 2024-05-27T07:58:31.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6c337823d] 2024-05-27T07:58:31.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c6c3363971] 2024-05-27T07:58:31.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c6c3361fa0] 2024-05-27T07:58:31.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6c3362651] 2024-05-27T07:58:31.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6c32f7347] 2024-05-27T07:58:31.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6c2ff387e] 2024-05-27T07:58:31.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6c2ff8f2f] 2024-05-27T07:58:31.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c6c3003969] 2024-05-27T07:58:31.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c6c339c48c] 2024-05-27T07:58:31.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c6c33a59cb] 2024-05-27T07:58:31.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 13: (Thread::entry_wrapper()+0x43) [0x7f7b3804772d] 2024-05-27T07:58:31.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7b38047749] 2024-05-27T07:58:31.809 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7b36beb1ca] 2024-05-27T07:58:31.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 16: clone() 2024-05-27T07:58:31.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.481+0000 7f7b25833700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7b25833700 time 2024-05-27T07:58:31.480730+0000 2024-05-27T07:58:31.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:58:31.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:31.810 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7b38078ba6] 2024-05-27T07:58:31.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7b38078dd1] 2024-05-27T07:58:31.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6c337823d] 2024-05-27T07:58:31.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c6c3363971] 2024-05-27T07:58:31.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c6c3361fa0] 2024-05-27T07:58:31.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6c3362651] 2024-05-27T07:58:31.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6c32f7347] 2024-05-27T07:58:31.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6c2ff387e] 2024-05-27T07:58:31.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6c2ff8f2f] 2024-05-27T07:58:31.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c6c3003969] 2024-05-27T07:58:31.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c6c339c48c] 2024-05-27T07:58:31.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c6c33a59cb] 2024-05-27T07:58:31.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 13: (Thread::entry_wrapper()+0x43) [0x7f7b3804772d] 2024-05-27T07:58:31.811 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7b38047749] 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7b36beb1ca] 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 16: clone() 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: *** Caught signal (Aborted) ** 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: in thread 7f7b25833700 thread_name:md_log_replay 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7b36bf5d20] 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2: gsignal() 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 3: abort() 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7b38078ced] 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7b38078dd1] 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6c337823d] 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c6c3363971] 2024-05-27T07:58:31.812 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c6c3361fa0] 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6c3362651] 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6c32f7347] 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6c2ff387e] 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6c2ff8f2f] 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c6c3003969] 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c6c339c48c] 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c6c33a59cb] 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 16: (Thread::entry_wrapper()+0x43) [0x7f7b3804772d] 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7b38047749] 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7b36beb1ca] 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 19: clone() 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.484+0000 7f7b25833700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: in thread 7f7b25833700 thread_name:md_log_replay 2024-05-27T07:58:31.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:31.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7b36bf5d20] 2024-05-27T07:58:31.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2: gsignal() 2024-05-27T07:58:31.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 3: abort() 2024-05-27T07:58:31.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7b38078ced] 2024-05-27T07:58:31.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7b38078dd1] 2024-05-27T07:58:31.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6c337823d] 2024-05-27T07:58:31.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c6c3363971] 2024-05-27T07:58:31.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c6c3361fa0] 2024-05-27T07:58:31.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6c3362651] 2024-05-27T07:58:31.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6c32f7347] 2024-05-27T07:58:31.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6c2ff387e] 2024-05-27T07:58:31.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6c2ff8f2f] 2024-05-27T07:58:31.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c6c3003969] 2024-05-27T07:58:31.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c6c339c48c] 2024-05-27T07:58:31.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c6c33a59cb] 2024-05-27T07:58:31.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 16: (Thread::entry_wrapper()+0x43) [0x7f7b3804772d] 2024-05-27T07:58:31.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7b38047749] 2024-05-27T07:58:31.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7b36beb1ca] 2024-05-27T07:58:31.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 19: clone() 2024-05-27T07:58:31.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:58:31.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: -1> 2024-05-27T07:58:31.481+0000 7f7b25833700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7b25833700 time 2024-05-27T07:58:31.480730+0000 2024-05-27T07:58:31.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:58:31.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:31.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7b38078ba6] 2024-05-27T07:58:31.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7b38078dd1] 2024-05-27T07:58:31.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6c337823d] 2024-05-27T07:58:31.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c6c3363971] 2024-05-27T07:58:31.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c6c3361fa0] 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6c3362651] 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6c32f7347] 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6c2ff387e] 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6c2ff8f2f] 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c6c3003969] 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c6c339c48c] 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c6c33a59cb] 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 13: (Thread::entry_wrapper()+0x43) [0x7f7b3804772d] 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7b38047749] 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7b36beb1ca] 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 16: clone() 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 0> 2024-05-27T07:58:31.484+0000 7f7b25833700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:58:31.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: in thread 7f7b25833700 thread_name:md_log_replay 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7b36bf5d20] 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2: gsignal() 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 3: abort() 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7b38078ced] 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7b38078dd1] 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6c337823d] 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c6c3363971] 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c6c3361fa0] 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6c3362651] 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6c32f7347] 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6c2ff387e] 2024-05-27T07:58:31.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6c2ff8f2f] 2024-05-27T07:58:31.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c6c3003969] 2024-05-27T07:58:31.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c6c339c48c] 2024-05-27T07:58:31.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c6c33a59cb] 2024-05-27T07:58:31.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 16: (Thread::entry_wrapper()+0x43) [0x7f7b3804772d] 2024-05-27T07:58:31.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7b38047749] 2024-05-27T07:58:31.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7b36beb1ca] 2024-05-27T07:58:31.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 19: clone() 2024-05-27T07:58:31.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:58:31.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: -1> 2024-05-27T07:58:31.481+0000 7f7b25833700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7b25833700 time 2024-05-27T07:58:31.480730+0000 2024-05-27T07:58:31.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:58:31.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:31.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7b38078ba6] 2024-05-27T07:58:31.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7b38078dd1] 2024-05-27T07:58:31.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6c337823d] 2024-05-27T07:58:31.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c6c3363971] 2024-05-27T07:58:31.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c6c3361fa0] 2024-05-27T07:58:31.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6c3362651] 2024-05-27T07:58:31.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6c32f7347] 2024-05-27T07:58:31.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6c2ff387e] 2024-05-27T07:58:31.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6c2ff8f2f] 2024-05-27T07:58:31.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c6c3003969] 2024-05-27T07:58:31.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c6c339c48c] 2024-05-27T07:58:31.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c6c33a59cb] 2024-05-27T07:58:31.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 13: (Thread::entry_wrapper()+0x43) [0x7f7b3804772d] 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7b38047749] 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7b36beb1ca] 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 16: clone() 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 0> 2024-05-27T07:58:31.484+0000 7f7b25833700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: in thread 7f7b25833700 thread_name:md_log_replay 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7b36bf5d20] 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2: gsignal() 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 3: abort() 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7b38078ced] 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7b38078dd1] 2024-05-27T07:58:31.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c6c337823d] 2024-05-27T07:58:31.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c6c3363971] 2024-05-27T07:58:31.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c6c3361fa0] 2024-05-27T07:58:31.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c6c3362651] 2024-05-27T07:58:31.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c6c32f7347] 2024-05-27T07:58:31.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c6c2ff387e] 2024-05-27T07:58:31.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c6c2ff8f2f] 2024-05-27T07:58:31.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c6c3003969] 2024-05-27T07:58:31.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c6c339c48c] 2024-05-27T07:58:31.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c6c33a59cb] 2024-05-27T07:58:31.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 16: (Thread::entry_wrapper()+0x43) [0x7f7b3804772d] 2024-05-27T07:58:31.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7b38047749] 2024-05-27T07:58:31.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7b36beb1ca] 2024-05-27T07:58:31.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 19: clone() 2024-05-27T07:58:31.823 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:58:31.823 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[131469]: 2024-05-27T07:58:31.823 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:31 smithi114 podman[131580]: 2024-05-27 07:58:31.531964923 +0000 UTC m=+0.031478708 container died d9064388867dde7e37d9ad4432fcf11a73f6384ff76dc5a6a7a92f636609970d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-27T07:58:32.456 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:32 smithi114 podman[131580]: 2024-05-27 07:58:32.164608988 +0000 UTC m=+0.664122765 container remove d9064388867dde7e37d9ad4432fcf11a73f6384ff76dc5a6a7a92f636609970d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:58:32.456 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:32 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:58:32.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 67394 12.19 MB/sec execute 164 sec latency 440.849 ms 2024-05-27T07:58:32.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:32 smithi163 ceph-mon[36451]: osdmap e158: 12 total, 12 up, 12 in 2024-05-27T07:58:32.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:32 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6838/925272910,v1:172.21.15.114:6839/925272910] up:boot 2024-05-27T07:58:32.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:32 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:58:32.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:58:32.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:32 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:58:32.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:32 smithi186 ceph-mon[36647]: osdmap e158: 12 total, 12 up, 12 in 2024-05-27T07:58:32.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:32 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6838/925272910,v1:172.21.15.114:6839/925272910] up:boot 2024-05-27T07:58:32.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:32 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:58:32.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:58:32.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:32 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:58:32.791 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:32 smithi114 ceph-mon[32038]: osdmap e158: 12 total, 12 up, 12 in 2024-05-27T07:58:32.791 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:32 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6838/925272910,v1:172.21.15.114:6839/925272910] up:boot 2024-05-27T07:58:32.791 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:32 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:58:32.791 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:58:32.791 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:32 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:58:33.402 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:33 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:58:33.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 67656 12.17 MB/sec execute 165 sec latency 525.374 ms 2024-05-27T07:58:33.660 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:33 smithi114 ceph-mon[32038]: pgmap v648: 129 pgs: 129 active+clean; 378 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.7 MiB/s wr, 326 op/s 2024-05-27T07:58:33.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.671+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3799737491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd748109850 msgr2=0x7fd7481111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:33.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.671+0000 7fd74fdbb700 1 --2- 172.21.15.114:0/3799737491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd748109850 0x7fd7481111b0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fd74000b0d0 tx=0x7fd74000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:58:33.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.671+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3799737491 shutdown_connections 2024-05-27T07:58:33.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.671+0000 7fd74fdbb700 1 --2- 172.21.15.114:0/3799737491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd748109850 0x7fd7481111b0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:33.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.671+0000 7fd74fdbb700 1 --2- 172.21.15.114:0/3799737491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7481045d0 0x7fd7481049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:33.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.671+0000 7fd74fdbb700 1 --2- 172.21.15.114:0/3799737491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd748108f00 0x7fd748109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:33.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.671+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3799737491 >> 172.21.15.114:0/3799737491 conn(0x7fd7480ffe60 msgr2=0x7fd7481022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:33.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.672+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3799737491 shutdown_connections 2024-05-27T07:58:33.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.672+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3799737491 wait complete. 2024-05-27T07:58:33.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.672+0000 7fd74fdbb700 1 Processor -- start 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.672+0000 7fd74fdbb700 1 -- start start 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74fdbb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7481045d0 0x7fd748075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74fdbb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd748108f00 0x7fd748075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74fdbb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd748109850 0x7fd748076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74fdbb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd748114480 con 0x7fd748109850 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74fdbb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd748114300 con 0x7fd748108f00 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74fdbb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd748114600 con 0x7fd7481045d0 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74db57700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7481045d0 0x7fd748075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74e358700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd748109850 0x7fd748076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74db57700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7481045d0 0x7fd748075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:50264/0 (socket says 172.21.15.114:50264) 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74db57700 1 -- 172.21.15.114:0/3982218335 learned_addr learned my addr 172.21.15.114:0/3982218335 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74d356700 1 --2- 172.21.15.114:0/3982218335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd748108f00 0x7fd748075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74db57700 1 -- 172.21.15.114:0/3982218335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd748108f00 msgr2=0x7fd748075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74db57700 1 --2- 172.21.15.114:0/3982218335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd748108f00 0x7fd748075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74db57700 1 -- 172.21.15.114:0/3982218335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd748109850 msgr2=0x7fd748076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:33.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74db57700 1 --2- 172.21.15.114:0/3982218335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd748109850 0x7fd748076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:33.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74db57700 1 -- 172.21.15.114:0/3982218335 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7481a9110 con 0x7fd7481045d0 2024-05-27T07:58:33.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74db57700 1 --2- 172.21.15.114:0/3982218335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7481045d0 0x7fd748075ab0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd74400e9a0 tx=0x7fd74400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:33.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd73effd700 1 -- 172.21.15.114:0/3982218335 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74400ca10 con 0x7fd7481045d0 2024-05-27T07:58:33.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3982218335 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7481a9400 con 0x7fd7481045d0 2024-05-27T07:58:33.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.673+0000 7fd73effd700 1 -- 172.21.15.114:0/3982218335 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd74400cbb0 con 0x7fd7481045d0 2024-05-27T07:58:33.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.674+0000 7fd73effd700 1 -- 172.21.15.114:0/3982218335 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd744018680 con 0x7fd7481045d0 2024-05-27T07:58:33.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.674+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3982218335 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7481a9940 con 0x7fd7481045d0 2024-05-27T07:58:33.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.674+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3982218335 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7480512b0 con 0x7fd7481045d0 2024-05-27T07:58:33.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.676+0000 7fd73effd700 1 -- 172.21.15.114:0/3982218335 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fd744010430 con 0x7fd7481045d0 2024-05-27T07:58:33.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.677+0000 7fd73effd700 1 --2- 172.21.15.114:0/3982218335 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7340b29f0 0x7fd7340b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:33.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.677+0000 7fd74d356700 1 --2- 172.21.15.114:0/3982218335 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7340b29f0 0x7fd7340b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:33.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.677+0000 7fd73effd700 1 -- 172.21.15.114:0/3982218335 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 9466+0+0 (secure 0 0 0) 0x7fd7440e2e80 con 0x7fd7481045d0 2024-05-27T07:58:33.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.677+0000 7fd74d356700 1 --2- 172.21.15.114:0/3982218335 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7340b29f0 0x7fd7340b4f00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd7481038b0 tx=0x7fd7380073d0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:33.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:33.680+0000 7fd73effd700 1 -- 172.21.15.114:0/3982218335 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd7440aa4b0 con 0x7fd7481045d0 2024-05-27T07:58:33.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:33 smithi163 ceph-mon[36451]: pgmap v648: 129 pgs: 129 active+clean; 378 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.7 MiB/s wr, 326 op/s 2024-05-27T07:58:33.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:33 smithi186 ceph-mon[36647]: pgmap v648: 129 pgs: 129 active+clean; 378 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.7 MiB/s wr, 326 op/s 2024-05-27T07:58:34.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.009+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3982218335 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd748076fe0 con 0x7fd7481045d0 2024-05-27T07:58:34.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.011+0000 7fd73effd700 1 -- 172.21.15.114:0/3982218335 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+6033 (secure 0 0 0) 0x7fd7440a9db0 con 0x7fd7481045d0 2024-05-27T07:58:34.013 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:58:34.013 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":19494,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/584614487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":584614487},{"type":"v1","addr":"172.21.15.114:6841","nonce":584614487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:58:31.454559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_19608":{"gid":19608,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6839/925272910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":925272910},{"type":"v1","addr":"172.21.15.114:6839","nonce":925272910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:58:34.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.016+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3982218335 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7340b29f0 msgr2=0x7fd7340b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:34.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.016+0000 7fd74fdbb700 1 --2- 172.21.15.114:0/3982218335 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7340b29f0 0x7fd7340b4f00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd7481038b0 tx=0x7fd7380073d0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.016+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3982218335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7481045d0 msgr2=0x7fd748075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:34.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.016+0000 7fd74fdbb700 1 --2- 172.21.15.114:0/3982218335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7481045d0 0x7fd748075ab0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd74400e9a0 tx=0x7fd74400ecb0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.019+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3982218335 shutdown_connections 2024-05-27T07:58:34.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.019+0000 7fd74fdbb700 1 --2- 172.21.15.114:0/3982218335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd748109850 0x7fd748076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.019+0000 7fd74fdbb700 1 --2- 172.21.15.114:0/3982218335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7481045d0 0x7fd748075ab0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.019+0000 7fd74fdbb700 1 --2- 172.21.15.114:0/3982218335 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd7340b29f0 0x7fd7340b4f00 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.019+0000 7fd74fdbb700 1 --2- 172.21.15.114:0/3982218335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd748108f00 0x7fd748075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.019+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3982218335 >> 172.21.15.114:0/3982218335 conn(0x7fd7480ffe60 msgr2=0x7fd748108620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:34.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.021+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3982218335 shutdown_connections 2024-05-27T07:58:34.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:34.021+0000 7fd74fdbb700 1 -- 172.21.15.114:0/3982218335 wait complete. 2024-05-27T07:58:34.028 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 106 2024-05-27T07:58:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.368+0000 7fb32c80c640 1 -- 172.21.15.114:0/2959385816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb32811a8e0 msgr2=0x7fb32811cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.368+0000 7fb32c80c640 1 --2- 172.21.15.114:0/2959385816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb32811a8e0 0x7fb32811cd00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009900 tx=0x7fb31c02a490 comp rx=0 tx=0).stop 2024-05-27T07:58:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.374+0000 7fb32c80c640 1 -- 172.21.15.114:0/2959385816 shutdown_connections 2024-05-27T07:58:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.374+0000 7fb32c80c640 1 --2- 172.21.15.114:0/2959385816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb32811d240 0x7fb32811f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.374+0000 7fb32c80c640 1 --2- 172.21.15.114:0/2959385816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb32811a8e0 0x7fb32811cd00 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.374+0000 7fb32c80c640 1 --2- 172.21.15.114:0/2959385816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32810a1f0 0x7fb32810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.374+0000 7fb32c80c640 1 -- 172.21.15.114:0/2959385816 >> 172.21.15.114:0/2959385816 conn(0x7fb32806e8f0 msgr2=0x7fb32806ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:34.382 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.380+0000 7fb32c80c640 1 -- 172.21.15.114:0/2959385816 shutdown_connections 2024-05-27T07:58:34.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.382+0000 7fb32c80c640 1 -- 172.21.15.114:0/2959385816 wait complete. 2024-05-27T07:58:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.383+0000 7fb32c80c640 1 Processor -- start 2024-05-27T07:58:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.385+0000 7fb32c80c640 1 -- start start 2024-05-27T07:58:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.385+0000 7fb32c80c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb32810a1f0 0x7fb328114580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.385+0000 7fb32c80c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb32811a8e0 0x7fb328114ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.385+0000 7fb32c80c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32811d240 0x7fb328115000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.385+0000 7fb32c80c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb3281129b0 con 0x7fb32811d240 2024-05-27T07:58:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.385+0000 7fb32c80c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb328112830 con 0x7fb32811a8e0 2024-05-27T07:58:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.385+0000 7fb32c80c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb328112b30 con 0x7fb32810a1f0 2024-05-27T07:58:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.386+0000 7fb326d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32811d240 0x7fb328115000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.386+0000 7fb326d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32811d240 0x7fb328115000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40784/0 (socket says 172.21.15.114:40784) 2024-05-27T07:58:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.386+0000 7fb326d9d640 1 -- 172.21.15.114:0/3669736169 learned_addr learned my addr 172.21.15.114:0/3669736169 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.387+0000 7fb32659c640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb32810a1f0 0x7fb328114580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.388+0000 7fb325d9b640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb32811a8e0 0x7fb328114ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.390+0000 7fb325d9b640 1 -- 172.21.15.114:0/3669736169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb32810a1f0 msgr2=0x7fb328114580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.390+0000 7fb325d9b640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb32810a1f0 0x7fb328114580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.390+0000 7fb325d9b640 1 -- 172.21.15.114:0/3669736169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32811d240 msgr2=0x7fb328115000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.390+0000 7fb325d9b640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32811d240 0x7fb328115000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.390+0000 7fb325d9b640 1 -- 172.21.15.114:0/3669736169 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3281be100 con 0x7fb32811a8e0 2024-05-27T07:58:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.390+0000 7fb326d9d640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32811d240 0x7fb328115000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:58:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.391+0000 7fb325d9b640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb32811a8e0 0x7fb328114ac0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb31c02aa80 tx=0x7fb31c003d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.393+0000 7fb3177fe640 1 -- 172.21.15.114:0/3669736169 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c03f070 con 0x7fb32811a8e0 2024-05-27T07:58:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.393+0000 7fb3177fe640 1 -- 172.21.15.114:0/3669736169 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb31c031d60 con 0x7fb32811a8e0 2024-05-27T07:58:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.393+0000 7fb3177fe640 1 -- 172.21.15.114:0/3669736169 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c035030 con 0x7fb32811a8e0 2024-05-27T07:58:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.393+0000 7fb32c80c640 1 -- 172.21.15.114:0/3669736169 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3281be390 con 0x7fb32811a8e0 2024-05-27T07:58:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.394+0000 7fb32c80c640 1 -- 172.21.15.114:0/3669736169 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3281be800 con 0x7fb32811a8e0 2024-05-27T07:58:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.401+0000 7fb333797740 1 Processor -- start 2024-05-27T07:58:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.404+0000 7fb3177fe640 1 -- 172.21.15.114:0/3669736169 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fb31c042070 con 0x7fb32811a8e0 2024-05-27T07:58:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.404+0000 7fb333797740 1 -- start start 2024-05-27T07:58:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.404+0000 7fb333797740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b851b0 0x563ac3f263d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.404+0000 7fb333797740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b5c2f0 0x563ac3f26910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.404+0000 7fb326d9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b851b0 0x563ac3f263d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.404+0000 7fb326d9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b851b0 0x563ac3f263d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37520/0 (socket says 172.21.15.114:37520) 2024-05-27T07:58:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.404+0000 7fb326d9d640 1 -- 172.21.15.114:0/2092343167 learned_addr learned my addr 172.21.15.114:0/2092343167 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.404+0000 7fb333797740 1 --2- 172.21.15.114:0/2092343167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 0x563ac3f26e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.404+0000 7fb333797740 1 -- 172.21.15.114:0/2092343167 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563ac3e98c30 con 0x563ac3b5c2f0 2024-05-27T07:58:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.404+0000 7fb333797740 1 -- 172.21.15.114:0/2092343167 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563ac3e887d0 con 0x563ac3b851b0 2024-05-27T07:58:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.404+0000 7fb333797740 1 -- 172.21.15.114:0/2092343167 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563ac3e8c810 con 0x563ac3b644d0 2024-05-27T07:58:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.408+0000 7fb32659c640 1 --2- 172.21.15.114:0/2092343167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b5c2f0 0x563ac3f26910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.408+0000 7fb326d9d640 1 -- 172.21.15.114:0/2092343167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 msgr2=0x563ac3f26e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.408+0000 7fb326d9d640 1 --2- 172.21.15.114:0/2092343167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 0x563ac3f26e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.408+0000 7fb326d9d640 1 -- 172.21.15.114:0/2092343167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b5c2f0 msgr2=0x563ac3f26910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.408+0000 7fb326d9d640 1 --2- 172.21.15.114:0/2092343167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b5c2f0 0x563ac3f26910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.408+0000 7fb326d9d640 1 -- 172.21.15.114:0/2092343167 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ac3b9d210 con 0x563ac3b851b0 2024-05-27T07:58:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.408+0000 7fb3177fe640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb3100b2b60 0x7fb3100b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.408+0000 7fb32659c640 1 --2- 172.21.15.114:0/2092343167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b5c2f0 0x563ac3f26910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:58:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.408+0000 7fb326d9d640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb3100b2b60 0x7fb3100b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.408+0000 7fb326d9d640 1 --2- 172.21.15.114:0/2092343167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b851b0 0x563ac3f263d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb318008c80 tx=0x7fb31800e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c925f702928942b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.408+0000 7fb3177fe640 1 -- 172.21.15.114:0/3669736169 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 9466+0+0 (secure 0 0 0) 0x7fb31c103080 con 0x7fb32811a8e0 2024-05-27T07:58:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.409+0000 7fb326d9d640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb3100b2b60 0x7fb3100b5070 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb31801d000 tx=0x7fb31800fc10 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.411+0000 7fb314ff9640 1 -- 172.21.15.114:0/2092343167 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb318020070 con 0x563ac3b851b0 2024-05-27T07:58:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.411+0000 7fb314ff9640 1 -- 172.21.15.114:0/2092343167 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb31801a410 con 0x563ac3b851b0 2024-05-27T07:58:34.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.412+0000 7fb333797740 1 -- 172.21.15.114:0/2092343167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b851b0 msgr2=0x563ac3f263d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:34.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.412+0000 7fb333797740 1 --2- 172.21.15.114:0/2092343167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b851b0 0x563ac3f263d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb318008c80 tx=0x7fb31800e720 comp rx=0 tx=0).stop 2024-05-27T07:58:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.420+0000 7fb333797740 1 -- 172.21.15.114:0/2092343167 shutdown_connections 2024-05-27T07:58:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.420+0000 7fb333797740 1 --2- 172.21.15.114:0/2092343167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 0x563ac3f26e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.420+0000 7fb333797740 1 --2- 172.21.15.114:0/2092343167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b5c2f0 0x563ac3f26910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.420+0000 7fb333797740 1 --2- 172.21.15.114:0/2092343167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b851b0 0x563ac3f263d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.420+0000 7fb333797740 1 -- 172.21.15.114:0/2092343167 >> 172.21.15.114:0/2092343167 conn(0x563ac3b79300 msgr2=0x563ac3f22b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.422+0000 7fb333797740 1 -- 172.21.15.114:0/2092343167 shutdown_connections 2024-05-27T07:58:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.422+0000 7fb333797740 1 -- 172.21.15.114:0/2092343167 wait complete. 2024-05-27T07:58:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.422+0000 7fb333797740 1 -- start start 2024-05-27T07:58:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.423+0000 7fb333797740 1 Processor -- start 2024-05-27T07:58:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.423+0000 7fb333797740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f30a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.423+0000 7fb333797740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 0x563ac3f30fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.423+0000 7fb333797740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b851b0 0x563ac3f36930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.423+0000 7fb333797740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563ac3b6e790 con 0x563ac3b851b0 2024-05-27T07:58:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.423+0000 7fb333797740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563ac3e44890 con 0x563ac3b5c2f0 2024-05-27T07:58:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.423+0000 7fb333797740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563ac3f33550 con 0x563ac3b644d0 2024-05-27T07:58:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.423+0000 7fb326d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b851b0 0x563ac3f36930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.423+0000 7fb326d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b851b0 0x563ac3f36930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40802/0 (socket says 172.21.15.114:40802) 2024-05-27T07:58:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.423+0000 7fb326d9d640 1 -- 172.21.15.114:0/617526092 learned_addr learned my addr 172.21.15.114:0/617526092 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.423+0000 7fb333797740 10 client.0 _ll_register_callbacks cb 0x563ac3b9ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.425+0000 7fb32659c640 1 --2- 172.21.15.114:0/617526092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f30a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.426+0000 7fb326d9d640 1 -- 172.21.15.114:0/617526092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 msgr2=0x563ac3f30fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:58:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.426+0000 7fb326d9d640 1 --2- 172.21.15.114:0/617526092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 0x563ac3f30fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.426+0000 7fb326d9d640 1 -- 172.21.15.114:0/617526092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 msgr2=0x563ac3f30a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.426+0000 7fb326d9d640 1 --2- 172.21.15.114:0/617526092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f30a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.426+0000 7fb326d9d640 1 -- 172.21.15.114:0/617526092 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ac3c69620 con 0x563ac3b851b0 2024-05-27T07:58:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.427+0000 7fb326d9d640 1 --2- 172.21.15.114:0/617526092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b851b0 0x563ac3f36930 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fb318008c50 tx=0x7fb318014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a3c1f9b79a0dad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.427+0000 7fb30dffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:58:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.427+0000 7fb30dffb640 1 -- 172.21.15.114:0/617526092 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb318034060 con 0x563ac3b851b0 2024-05-27T07:58:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.427+0000 7fb333797740 1 -- 172.21.15.114:0/617526092 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb31801a410 con 0x563ac3b851b0 2024-05-27T07:58:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.427+0000 7fb30dffb640 1 -- 172.21.15.114:0/617526092 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(158..158 src has 1..158) ==== 9466+0+0 (secure 0 0 0) 0x7fb318010750 con 0x563ac3b851b0 2024-05-27T07:58:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.428+0000 7fb30dffb640 1 -- 172.21.15.114:0/617526092 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7fb318020050 con 0x563ac3b851b0 2024-05-27T07:58:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.430+0000 7fb333797740 10 client.19620 fetch_fsmap learned FSMap version 106 2024-05-27T07:58:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.430+0000 7fb333797740 1 -- 172.21.15.114:0/617526092 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x563ac3f61840 con 0x563ac3b851b0 2024-05-27T07:58:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.430+0000 7fb30dffb640 1 -- 172.21.15.114:0/617526092 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb31801a410 con 0x563ac3b851b0 2024-05-27T07:58:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.432+0000 7fb30dffb640 1 -- 172.21.15.114:0/617526092 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 106) ==== 4095+0+0 (secure 0 0 0) 0x7fb318005200 con 0x563ac3b851b0 2024-05-27T07:58:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.432+0000 7fb333797740 10 client.19620 fetch_fsmap finished waiting for FSMap version 106 2024-05-27T07:58:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.432+0000 7fb333797740 10 client.19620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.432+0000 7fb333797740 20 client.19620 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.432+0000 7fb333797740 1 --2- 172.21.15.114:0/617526092 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563ac3b43d80 0x563ac3f65180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.432+0000 7fb333797740 4 client.19620 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:58:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.432+0000 7fb333797740 1 -- 172.21.15.114:0/617526092 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ac3e9ab80 con 0x563ac3b43d80 2024-05-27T07:58:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.434+0000 7fb32659c640 1 --2- 172.21.15.114:0/617526092 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563ac3b43d80 0x563ac3f65180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.436+0000 7fb32659c640 1 --2- 172.21.15.114:0/617526092 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563ac3b43d80 0x563ac3f65180 crc :-1 s=READY pgs=496 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c6c7881b906c0fb server_cookie=546865ced247b804 in_seq=0 out_seq=0 2024-05-27T07:58:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.437+0000 7fb30dffb640 10 client.19620 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.446+0000 7fb30dffb640 1 -- 172.21.15.114:0/617526092 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563ac3e9ab80 con 0x563ac3b43d80 2024-05-27T07:58:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.446+0000 7fb30dffb640 10 client.19620 handle_command_reply: tid=0 2024-05-27T07:58:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.447+0000 7fb333797740 1 client.19620 shutdown 2024-05-27T07:58:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.447+0000 7fb333797740 1 -- 172.21.15.114:0/617526092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b851b0 msgr2=0x563ac3f36930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.447+0000 7fb333797740 1 --2- 172.21.15.114:0/617526092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b851b0 0x563ac3f36930 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fb318008c50 tx=0x7fb318014000 comp rx=0 tx=0).stop 2024-05-27T07:58:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.447+0000 7fb333797740 1 -- 172.21.15.114:0/617526092 shutdown_connections 2024-05-27T07:58:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.447+0000 7fb333797740 1 --2- 172.21.15.114:0/617526092 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563ac3b43d80 0x563ac3f65180 crc :-1 s=READY pgs=496 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.447+0000 7fb333797740 1 --2- 172.21.15.114:0/617526092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b851b0 0x563ac3f36930 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.447+0000 7fb333797740 1 --2- 172.21.15.114:0/617526092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 0x563ac3f30fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.447+0000 7fb333797740 1 --2- 172.21.15.114:0/617526092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f30a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.447+0000 7fb333797740 1 -- 172.21.15.114:0/617526092 >> 172.21.15.114:0/617526092 conn(0x563ac3b79300 msgr2=0x563ac3f26200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.447+0000 7fb30dffb640 0 client.19620 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.447+0000 7fb333797740 1 -- 172.21.15.114:0/617526092 shutdown_connections 2024-05-27T07:58:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.447+0000 7fb333797740 1 -- 172.21.15.114:0/617526092 wait complete. 2024-05-27T07:58:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.447+0000 7fb333797740 20 client.19620 trim_cache size 0 max 16384 2024-05-27T07:58:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.475+0000 7fb333797740 1 Processor -- start 2024-05-27T07:58:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.475+0000 7fb333797740 1 -- start start 2024-05-27T07:58:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.476+0000 7fb333797740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f3d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.476+0000 7fb333797740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b644d0 0x563ac3f2a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.476+0000 7fb32659c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f3d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.476+0000 7fb32659c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f3d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:37542/0 (socket says 172.21.15.114:37542) 2024-05-27T07:58:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.476+0000 7fb325d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b644d0 0x563ac3f2a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.476+0000 7fb325d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b644d0 0x563ac3f2a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40808/0 (socket says 172.21.15.114:40808) 2024-05-27T07:58:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.476+0000 7fb32659c640 1 -- 172.21.15.114:0/1864380089 learned_addr learned my addr 172.21.15.114:0/1864380089 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.477+0000 7fb333797740 1 --2- 172.21.15.114:0/1864380089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b851b0 0x563ac3f6a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.478+0000 7fb333797740 1 -- 172.21.15.114:0/1864380089 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb30800fd00 con 0x563ac3b644d0 2024-05-27T07:58:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.478+0000 7fb333797740 1 -- 172.21.15.114:0/1864380089 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb3080103d0 con 0x563ac3b5c2f0 2024-05-27T07:58:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.478+0000 7fb333797740 1 -- 172.21.15.114:0/1864380089 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb30800eca0 con 0x563ac3b851b0 2024-05-27T07:58:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.478+0000 7fb326d9d640 1 --2- 172.21.15.114:0/1864380089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b851b0 0x563ac3f6a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.478+0000 7fb326d9d640 1 -- 172.21.15.114:0/1864380089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 msgr2=0x563ac3f3d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.478+0000 7fb326d9d640 1 --2- 172.21.15.114:0/1864380089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f3d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.478+0000 7fb326d9d640 1 -- 172.21.15.114:0/1864380089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b644d0 msgr2=0x563ac3f2a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.478+0000 7fb326d9d640 1 --2- 172.21.15.114:0/1864380089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b644d0 0x563ac3f2a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.478+0000 7fb326d9d640 1 -- 172.21.15.114:0/1864380089 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ac3f23710 con 0x563ac3b851b0 2024-05-27T07:58:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.478+0000 7fb325d9b640 1 --2- 172.21.15.114:0/1864380089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b644d0 0x563ac3f2a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.478+0000 7fb326d9d640 1 --2- 172.21.15.114:0/1864380089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b851b0 0x563ac3f6a940 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb3180055d0 tx=0x7fb318010940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c525542118db8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.480+0000 7fb30ffff640 1 -- 172.21.15.114:0/1864380089 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb318034040 con 0x563ac3b851b0 2024-05-27T07:58:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.480+0000 7fb30ffff640 1 -- 172.21.15.114:0/1864380089 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3180197c0 con 0x563ac3b851b0 2024-05-27T07:58:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.480+0000 7fb30ffff640 1 -- 172.21.15.114:0/1864380089 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31802f920 con 0x563ac3b851b0 2024-05-27T07:58:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.481+0000 7fb333797740 1 -- 172.21.15.114:0/1864380089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b851b0 msgr2=0x563ac3f6a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.481+0000 7fb333797740 1 --2- 172.21.15.114:0/1864380089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b851b0 0x563ac3f6a940 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb3180055d0 tx=0x7fb318010940 comp rx=0 tx=0).stop 2024-05-27T07:58:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.481+0000 7fb333797740 1 -- 172.21.15.114:0/1864380089 shutdown_connections 2024-05-27T07:58:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.481+0000 7fb333797740 1 --2- 172.21.15.114:0/1864380089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b851b0 0x563ac3f6a940 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.481+0000 7fb333797740 1 --2- 172.21.15.114:0/1864380089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b644d0 0x563ac3f2a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.481+0000 7fb333797740 1 --2- 172.21.15.114:0/1864380089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f3d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.481+0000 7fb333797740 1 -- 172.21.15.114:0/1864380089 >> 172.21.15.114:0/1864380089 conn(0x563ac3b79300 msgr2=0x563ac3f29f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.481+0000 7fb333797740 1 -- 172.21.15.114:0/1864380089 shutdown_connections 2024-05-27T07:58:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.481+0000 7fb333797740 1 -- 172.21.15.114:0/1864380089 wait complete. 2024-05-27T07:58:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb333797740 1 -- start start 2024-05-27T07:58:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb333797740 1 Processor -- start 2024-05-27T07:58:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb333797740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f39ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb333797740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 0x563ac3f3a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb333797740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b851b0 0x563ac3f3a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb333797740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb30800e5d0 con 0x563ac3b851b0 2024-05-27T07:58:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb333797740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb30800df60 con 0x563ac3b5c2f0 2024-05-27T07:58:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb333797740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563ac3e98c30 con 0x563ac3b644d0 2024-05-27T07:58:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb333797740 10 client.0 _ll_register_callbacks cb 0x7fb31801e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb326d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b851b0 0x563ac3f3a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb325d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 0x563ac3f3a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb326d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b851b0 0x563ac3f3a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40822/0 (socket says 172.21.15.114:40822) 2024-05-27T07:58:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb325d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 0x563ac3f3a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:50314/0 (socket says 172.21.15.114:50314) 2024-05-27T07:58:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb326d9d640 1 -- 172.21.15.114:0/1140724381 learned_addr learned my addr 172.21.15.114:0/1140724381 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb32659c640 1 --2- 172.21.15.114:0/1140724381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f39ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb325d9b640 1 -- 172.21.15.114:0/1140724381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 msgr2=0x563ac3f39ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.482+0000 7fb325d9b640 1 --2- 172.21.15.114:0/1140724381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f39ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb325d9b640 1 -- 172.21.15.114:0/1140724381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b851b0 msgr2=0x563ac3f3a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb325d9b640 1 --2- 172.21.15.114:0/1140724381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b851b0 0x563ac3f3a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb325d9b640 1 -- 172.21.15.114:0/1140724381 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ac3fcb810 con 0x563ac3b644d0 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb32659c640 1 --2- 172.21.15.114:0/1140724381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f39ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb325d9b640 1 --2- 172.21.15.114:0/1140724381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 0x563ac3f3a000 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb31c1003c0 tx=0x7fb31c0c9d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df48ad8b3bd40259 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb30dffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb30dffb640 1 -- 172.21.15.114:0/1140724381 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c0ca650 con 0x563ac3b644d0 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb30dffb640 1 -- 172.21.15.114:0/1140724381 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(158..158 src has 1..158) ==== 9466+0+0 (secure 0 0 0) 0x7fb31c0ca1b0 con 0x563ac3b644d0 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb333797740 1 -- 172.21.15.114:0/1140724381 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3180197c0 con 0x563ac3b644d0 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb30dffb640 1 -- 172.21.15.114:0/1140724381 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7fb31c0c9f50 con 0x563ac3b644d0 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb333797740 10 client.29104 fetch_fsmap learned FSMap version 106 2024-05-27T07:58:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb333797740 1 -- 172.21.15.114:0/1140724381 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x563ac3fcc9d0 con 0x563ac3b644d0 2024-05-27T07:58:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb30dffb640 1 -- 172.21.15.114:0/1140724381 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3180197c0 con 0x563ac3b644d0 2024-05-27T07:58:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb30dffb640 1 -- 172.21.15.114:0/1140724381 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 106) ==== 4095+0+0 (secure 0 0 0) 0x7fb31c0d29b0 con 0x563ac3b644d0 2024-05-27T07:58:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb333797740 10 client.29104 fetch_fsmap finished waiting for FSMap version 106 2024-05-27T07:58:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb333797740 10 client.29104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb333797740 20 client.29104 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb333797740 1 --2- 172.21.15.114:0/1140724381 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563ac3b191c0 0x563ac3fcf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb333797740 4 client.29104 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:58:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.483+0000 7fb333797740 1 -- 172.21.15.114:0/1140724381 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563ac3e94360 con 0x563ac3b191c0 2024-05-27T07:58:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.484+0000 7fb32659c640 1 --2- 172.21.15.114:0/1140724381 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563ac3b191c0 0x563ac3fcf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.484+0000 7fb30dffb640 10 client.29104 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:34.484+0000 7fb32659c640 1 --2- 172.21.15.114:0/1140724381 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563ac3b191c0 0x563ac3fcf040 crc :-1 s=READY pgs=497 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a51cd224f9e06440 server_cookie=591525c0926fcd1e in_seq=0 out_seq=0 2024-05-27T07:58:34.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 67866 12.18 MB/sec execute 166 sec latency 190.342 ms 2024-05-27T07:58:34.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:34 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3982218335' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:58:34.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:34 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3982218335' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:58:34.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:34 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3982218335' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:58:35.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 67921 12.13 MB/sec execute 167 sec latency 788.998 ms 2024-05-27T07:58:35.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:58:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.724+0000 7fb30dffb640 1 -- 172.21.15.114:0/1140724381 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x563ac3e94360 con 0x563ac3b191c0 2024-05-27T07:58:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.724+0000 7fb30dffb640 10 client.29104 handle_command_reply: tid=0 2024-05-27T07:58:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb333797740 1 client.29104 shutdown 2024-05-27T07:58:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb333797740 1 -- 172.21.15.114:0/1140724381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 msgr2=0x563ac3f3a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb333797740 1 --2- 172.21.15.114:0/1140724381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 0x563ac3f3a000 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb31c1003c0 tx=0x7fb31c0c9d90 comp rx=0 tx=0).stop 2024-05-27T07:58:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb333797740 1 -- 172.21.15.114:0/1140724381 shutdown_connections 2024-05-27T07:58:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb333797740 1 --2- 172.21.15.114:0/1140724381 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x563ac3b191c0 0x563ac3fcf040 crc :-1 s=READY pgs=497 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb333797740 1 --2- 172.21.15.114:0/1140724381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ac3b851b0 0x563ac3f3a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb333797740 1 --2- 172.21.15.114:0/1140724381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac3b644d0 0x563ac3f3a000 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb333797740 1 --2- 172.21.15.114:0/1140724381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563ac3b5c2f0 0x563ac3f39ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb30dffb640 0 client.29104 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb333797740 1 -- 172.21.15.114:0/1140724381 >> 172.21.15.114:0/1140724381 conn(0x563ac3b79300 msgr2=0x563ac3f26160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb333797740 1 -- 172.21.15.114:0/1140724381 shutdown_connections 2024-05-27T07:58:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb333797740 1 -- 172.21.15.114:0/1140724381 wait complete. 2024-05-27T07:58:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb333797740 20 client.29104 trim_cache size 0 max 16384 2024-05-27T07:58:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:58:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:58:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:58:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb32c80c640 1 -- 172.21.15.114:0/3669736169 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb3100b2b60 msgr2=0x7fb3100b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb32c80c640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb3100b2b60 0x7fb3100b5070 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb31801d000 tx=0x7fb31800fc10 comp rx=0 tx=0).stop 2024-05-27T07:58:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.725+0000 7fb32c80c640 1 -- 172.21.15.114:0/3669736169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb32811a8e0 msgr2=0x7fb328114ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.726+0000 7fb32c80c640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb32811a8e0 0x7fb328114ac0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb31c02aa80 tx=0x7fb31c003d60 comp rx=0 tx=0).stop 2024-05-27T07:58:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.726+0000 7fb32c80c640 1 -- 172.21.15.114:0/3669736169 shutdown_connections 2024-05-27T07:58:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.726+0000 7fb32c80c640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb3100b2b60 0x7fb3100b5070 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.726+0000 7fb32c80c640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32811d240 0x7fb328115000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.726+0000 7fb32c80c640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb32811a8e0 0x7fb328114ac0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.726+0000 7fb32c80c640 1 --2- 172.21.15.114:0/3669736169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb32810a1f0 0x7fb328114580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.726+0000 7fb32c80c640 1 -- 172.21.15.114:0/3669736169 >> 172.21.15.114:0/3669736169 conn(0x7fb32806e8f0 msgr2=0x7fb32811acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.726+0000 7fb32c80c640 1 -- 172.21.15.114:0/3669736169 shutdown_connections 2024-05-27T07:58:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:35.726+0000 7fb32c80c640 1 -- 172.21.15.114:0/3669736169 wait complete. 2024-05-27T07:58:35.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:35 smithi114 ceph-mon[32038]: pgmap v649: 129 pgs: 129 active+clean; 383 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 289 op/s 2024-05-27T07:58:35.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:35 smithi163 ceph-mon[36451]: pgmap v649: 129 pgs: 129 active+clean; 383 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 289 op/s 2024-05-27T07:58:35.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:35 smithi186 ceph-mon[36647]: pgmap v649: 129 pgs: 129 active+clean; 383 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 289 op/s 2024-05-27T07:58:36.187 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:58:36.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 68305 12.17 MB/sec execute 168 sec latency 568.252 ms 2024-05-27T07:58:37.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 68661 12.20 MB/sec execute 169 sec latency 216.519 ms 2024-05-27T07:58:37.907 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:37 smithi114 ceph-mon[32038]: pgmap v650: 129 pgs: 129 active+clean; 380 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.8 MiB/s wr, 501 op/s 2024-05-27T07:58:38.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.028+0000 7fc78b7df700 1 -- 172.21.15.114:0/2886044468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7841034b0 msgr2=0x7fc7841038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:38.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.028+0000 7fc78b7df700 1 --2- 172.21.15.114:0/2886044468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7841034b0 0x7fc7841038c0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fc7800099c0 tx=0x7fc780009cd0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.028+0000 7fc78b7df700 1 -- 172.21.15.114:0/2886044468 shutdown_connections 2024-05-27T07:58:38.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.028+0000 7fc78b7df700 1 --2- 172.21.15.114:0/2886044468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc784108730 0x7fc784110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.028+0000 7fc78b7df700 1 --2- 172.21.15.114:0/2886044468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7841034b0 0x7fc7841038c0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.028+0000 7fc78b7df700 1 --2- 172.21.15.114:0/2886044468 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc784107de0 0x7fc7841081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.028+0000 7fc78b7df700 1 -- 172.21.15.114:0/2886044468 >> 172.21.15.114:0/2886044468 conn(0x7fc7840fedb0 msgr2=0x7fc7841011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:38.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.028+0000 7fc78b7df700 1 -- 172.21.15.114:0/2886044468 shutdown_connections 2024-05-27T07:58:38.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.028+0000 7fc78b7df700 1 -- 172.21.15.114:0/2886044468 wait complete. 2024-05-27T07:58:38.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc78b7df700 1 Processor -- start 2024-05-27T07:58:38.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc78b7df700 1 -- start start 2024-05-27T07:58:38.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc78b7df700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7841034b0 0x7fc78419f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc78b7df700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc784107de0 0x7fc78419fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc78b7df700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc784108730 0x7fc784199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc78b7df700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc784113360 con 0x7fc784108730 2024-05-27T07:58:38.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc78b7df700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841131e0 con 0x7fc7841034b0 2024-05-27T07:58:38.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc78b7df700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841134e0 con 0x7fc784107de0 2024-05-27T07:58:38.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc789d7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc784108730 0x7fc784199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc788d7a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc784107de0 0x7fc78419fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc78957b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7841034b0 0x7fc78419f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc789d7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc784108730 0x7fc784199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40218/0 (socket says 172.21.15.114:40218) 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc788d7a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc784107de0 0x7fc78419fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:45554/0 (socket says 172.21.15.114:45554) 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc789d7c700 1 -- 172.21.15.114:0/1719435871 learned_addr learned my addr 172.21.15.114:0/1719435871 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.029+0000 7fc78957b700 1 -- 172.21.15.114:0/1719435871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc784107de0 msgr2=0x7fc78419fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.030+0000 7fc78957b700 1 --2- 172.21.15.114:0/1719435871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc784107de0 0x7fc78419fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.030+0000 7fc78957b700 1 -- 172.21.15.114:0/1719435871 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc784108730 msgr2=0x7fc784199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.030+0000 7fc78957b700 1 --2- 172.21.15.114:0/1719435871 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc784108730 0x7fc784199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.030+0000 7fc78957b700 1 -- 172.21.15.114:0/1719435871 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc78419a270 con 0x7fc7841034b0 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.030+0000 7fc789d7c700 1 --2- 172.21.15.114:0/1719435871 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc784108730 0x7fc784199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.030+0000 7fc78957b700 1 --2- 172.21.15.114:0/1719435871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7841034b0 0x7fc78419f9b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc7800098e0 tx=0x7fc780004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.030+0000 7fc77a7fc700 1 -- 172.21.15.114:0/1719435871 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc780026070 con 0x7fc7841034b0 2024-05-27T07:58:38.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.030+0000 7fc78b7df700 1 -- 172.21.15.114:0/1719435871 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc78419a4a0 con 0x7fc7841034b0 2024-05-27T07:58:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.030+0000 7fc77a7fc700 1 -- 172.21.15.114:0/1719435871 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc780003c70 con 0x7fc7841034b0 2024-05-27T07:58:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.030+0000 7fc77a7fc700 1 -- 172.21.15.114:0/1719435871 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78000f9c0 con 0x7fc7841034b0 2024-05-27T07:58:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.030+0000 7fc78b7df700 1 -- 172.21.15.114:0/1719435871 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc78406b600 con 0x7fc7841034b0 2024-05-27T07:58:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.033+0000 7fc77a7fc700 1 -- 172.21.15.114:0/1719435871 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fc780029070 con 0x7fc7841034b0 2024-05-27T07:58:38.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.034+0000 7fc77a7fc700 1 --2- 172.21.15.114:0/1719435871 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7700b29f0 0x7fc7700b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.034+0000 7fc788d7a700 1 --2- 172.21.15.114:0/1719435871 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7700b29f0 0x7fc7700b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.034+0000 7fc77a7fc700 1 -- 172.21.15.114:0/1719435871 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 9466+0+0 (secure 0 0 0) 0x7fc7800e3190 con 0x7fc7841034b0 2024-05-27T07:58:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.034+0000 7fc788d7a700 1 --2- 172.21.15.114:0/1719435871 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7700b29f0 0x7fc7700b4f00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc78419b270 tx=0x7fc774009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.034+0000 7fc79a1f4b80 1 Processor -- start 2024-05-27T07:58:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc79a1f4b80 1 -- start start 2024-05-27T07:58:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc79a1f4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b6514b1e40 0x55b651635710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc79a1f4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 0x55b65162fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc79a1f4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b65162ff80 0x55b651630390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc79a1f4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b65155ec90 con 0x55b651635c50 2024-05-27T07:58:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc79a1f4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b65154e830 con 0x55b6514b1e40 2024-05-27T07:58:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc79a1f4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b651552870 con 0x55b65162ff80 2024-05-27T07:58:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc788d7a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 0x55b65162fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc788d7a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 0x55b65162fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40226/0 (socket says 172.21.15.114:40226) 2024-05-27T07:58:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc788d7a700 1 -- 172.21.15.114:0/1810373351 learned_addr learned my addr 172.21.15.114:0/1810373351 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc789d7c700 1 --2- 172.21.15.114:0/1810373351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b65162ff80 0x55b651630390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc78957b700 1 --2- 172.21.15.114:0/1810373351 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b6514b1e40 0x55b651635710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc788d7a700 1 -- 172.21.15.114:0/1810373351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b65162ff80 msgr2=0x55b651630390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc788d7a700 1 --2- 172.21.15.114:0/1810373351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b65162ff80 0x55b651630390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc788d7a700 1 -- 172.21.15.114:0/1810373351 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b6514b1e40 msgr2=0x55b651635710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc788d7a700 1 --2- 172.21.15.114:0/1810373351 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b6514b1e40 0x55b651635710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc788d7a700 1 -- 172.21.15.114:0/1810373351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b65149e8d0 con 0x55b651635c50 2024-05-27T07:58:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.035+0000 7fc789d7c700 1 --2- 172.21.15.114:0/1810373351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b65162ff80 0x55b651630390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.036+0000 7fc788d7a700 1 --2- 172.21.15.114:0/1810373351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 0x55b65162fa40 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fc774011cf0 tx=0x7fc774014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db29921f9ff1721 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.036+0000 7fc76f7fe700 1 -- 172.21.15.114:0/1810373351 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774004220 con 0x55b651635c50 2024-05-27T07:58:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.036+0000 7fc76f7fe700 1 -- 172.21.15.114:0/1810373351 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7740043c0 con 0x55b651635c50 2024-05-27T07:58:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.036+0000 7fc76f7fe700 1 -- 172.21.15.114:0/1810373351 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774003dc0 con 0x55b651635c50 2024-05-27T07:58:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.037+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/1810373351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 msgr2=0x55b65162fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.037+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/1810373351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 0x55b65162fa40 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fc774011cf0 tx=0x7fc774014000 comp rx=0 tx=0).stop 2024-05-27T07:58:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.037+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/1810373351 shutdown_connections 2024-05-27T07:58:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.037+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/1810373351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b65162ff80 0x55b651630390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.037+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/1810373351 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b6514b1e40 0x55b651635710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.037+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/1810373351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 0x55b65162fa40 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.037+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/1810373351 >> 172.21.15.114:0/1810373351 conn(0x55b65150caf0 msgr2=0x55b65162a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.037+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/1810373351 shutdown_connections 2024-05-27T07:58:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.037+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/1810373351 wait complete. 2024-05-27T07:58:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.037+0000 7fc79a1f4b80 1 -- start start 2024-05-27T07:58:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.037+0000 7fc79a1f4b80 1 Processor -- start 2024-05-27T07:58:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc79a1f4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b6514b1e40 0x55b651639330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc79a1f4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b65162ff80 0x55b651639870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc79a1f4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b651635c50 0x55b651639db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc79a1f4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b651454fa0 con 0x55b65162ff80 2024-05-27T07:58:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc79a1f4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b6514f07c0 con 0x55b6514b1e40 2024-05-27T07:58:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc79a1f4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b65163c740 con 0x55b651635c50 2024-05-27T07:58:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc79a1f4b80 10 client.0 _ll_register_callbacks cb 0x55b6514e9920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc788d7a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b65162ff80 0x55b651639870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc789d7c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b651635c50 0x55b651639db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc788d7a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b65162ff80 0x55b651639870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40236/0 (socket says 172.21.15.114:40236) 2024-05-27T07:58:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc788d7a700 1 -- 172.21.15.114:0/2905089669 learned_addr learned my addr 172.21.15.114:0/2905089669 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc789d7c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b651635c50 0x55b651639db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:45562/0 (socket says 172.21.15.114:45562) 2024-05-27T07:58:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc78957b700 1 --2- 172.21.15.114:0/2905089669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b6514b1e40 0x55b651639330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc789d7c700 1 -- 172.21.15.114:0/2905089669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b6514b1e40 msgr2=0x55b651639330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc789d7c700 1 --2- 172.21.15.114:0/2905089669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b6514b1e40 0x55b651639330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc789d7c700 1 -- 172.21.15.114:0/2905089669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b65162ff80 msgr2=0x55b651639870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc789d7c700 1 --2- 172.21.15.114:0/2905089669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b65162ff80 0x55b651639870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc789d7c700 1 -- 172.21.15.114:0/2905089669 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6514ed690 con 0x55b651635c50 2024-05-27T07:58:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.038+0000 7fc788d7a700 1 --2- 172.21.15.114:0/2905089669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b65162ff80 0x55b651639870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc789d7c700 1 --2- 172.21.15.114:0/2905089669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b651635c50 0x55b651639db0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc77c007fb0 tx=0x7fc77c00e7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29a84e32aaab2020 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc76cff9700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc76cff9700 1 -- 172.21.15.114:0/2905089669 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c01a070 con 0x55b651635c50 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc76cff9700 1 -- 172.21.15.114:0/2905089669 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(158..158 src has 1..158) ==== 9466+0+0 (secure 0 0 0) 0x7fc77c004db0 con 0x55b651635c50 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/2905089669 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7740043c0 con 0x55b651635c50 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc76cff9700 1 -- 172.21.15.114:0/2905089669 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7fc77c026050 con 0x55b651635c50 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc79a1f4b80 10 client.29116 fetch_fsmap learned FSMap version 106 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/2905089669 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x55b6516471f0 con 0x55b651635c50 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc76cff9700 1 -- 172.21.15.114:0/2905089669 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7740043c0 con 0x55b651635c50 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc76cff9700 1 -- 172.21.15.114:0/2905089669 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 106) ==== 4095+0+0 (secure 0 0 0) 0x7fc77c01e5f0 con 0x55b651635c50 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc79a1f4b80 10 client.29116 fetch_fsmap finished waiting for FSMap version 106 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc79a1f4b80 10 client.29116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc79a1f4b80 20 client.29116 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/2905089669 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b651649c80 0x55b65164c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc79a1f4b80 4 client.29116 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:58:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.039+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/2905089669 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6515fb9f0 con 0x55b651649c80 2024-05-27T07:58:38.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.040+0000 7fc78957b700 1 --2- 172.21.15.114:0/2905089669 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b651649c80 0x55b65164c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.040+0000 7fc78957b700 1 --2- 172.21.15.114:0/2905089669 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b651649c80 0x55b65164c0d0 crc :-1 s=READY pgs=498 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbe1d9fc61d90cfe server_cookie=e9a4e0b0e8a747f6 in_seq=0 out_seq=0 2024-05-27T07:58:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.040+0000 7fc76cff9700 10 client.29116 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.042+0000 7fc76cff9700 1 -- 172.21.15.114:0/2905089669 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b6515fb9f0 con 0x55b651649c80 2024-05-27T07:58:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.042+0000 7fc76cff9700 10 client.29116 handle_command_reply: tid=0 2024-05-27T07:58:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.042+0000 7fc79a1f4b80 1 client.29116 shutdown 2024-05-27T07:58:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.043+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/2905089669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b651635c50 msgr2=0x55b651639db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.043+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/2905089669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b651635c50 0x55b651639db0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc77c007fb0 tx=0x7fc77c00e7d0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.043+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/2905089669 shutdown_connections 2024-05-27T07:58:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.043+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/2905089669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b651635c50 0x55b651639db0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.043+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/2905089669 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b651649c80 0x55b65164c0d0 crc :-1 s=READY pgs=498 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.043+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/2905089669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b6514b1e40 0x55b651639330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.043+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/2905089669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b65162ff80 0x55b651639870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.043+0000 7fc76cff9700 0 client.29116 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.043+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/2905089669 >> 172.21.15.114:0/2905089669 conn(0x55b65150caf0 msgr2=0x55b65162ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.043+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/2905089669 shutdown_connections 2024-05-27T07:58:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.043+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/2905089669 wait complete. 2024-05-27T07:58:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.043+0000 7fc79a1f4b80 20 client.29116 trim_cache size 0 max 16384 2024-05-27T07:58:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.059+0000 7fc79a1f4b80 1 Processor -- start 2024-05-27T07:58:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.059+0000 7fc79a1f4b80 1 -- start start 2024-05-27T07:58:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc79a1f4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6514b1e40 0x55b65162fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc79a1f4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b65162ff80 0x55b651639bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc79a1f4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b651635c50 0x55b6516457e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc79a1f4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841b00f0 con 0x55b6514b1e40 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc79a1f4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841b07c0 con 0x55b651635c50 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc79a1f4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841af070 con 0x55b65162ff80 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc78957b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6514b1e40 0x55b65162fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc78957b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6514b1e40 0x55b65162fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40246/0 (socket says 172.21.15.114:40246) 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc78957b700 1 -- 172.21.15.114:0/3279902217 learned_addr learned my addr 172.21.15.114:0/3279902217 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc788d7a700 1 --2- 172.21.15.114:0/3279902217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b65162ff80 0x55b651639bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc789d7c700 1 --2- 172.21.15.114:0/3279902217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b651635c50 0x55b6516457e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc78957b700 1 -- 172.21.15.114:0/3279902217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b65162ff80 msgr2=0x55b651639bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc78957b700 1 --2- 172.21.15.114:0/3279902217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b65162ff80 0x55b651639bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc78957b700 1 -- 172.21.15.114:0/3279902217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b651635c50 msgr2=0x55b6516457e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc78957b700 1 --2- 172.21.15.114:0/3279902217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b651635c50 0x55b6516457e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc78957b700 1 -- 172.21.15.114:0/3279902217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b65163a130 con 0x55b6514b1e40 2024-05-27T07:58:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc789d7c700 1 --2- 172.21.15.114:0/3279902217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b651635c50 0x55b6516457e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:58:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc78957b700 1 --2- 172.21.15.114:0/3279902217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6514b1e40 0x55b65162fa40 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc7800e0530 tx=0x7fc780009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74aea5831957e3a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.060+0000 7fc76effd700 1 -- 172.21.15.114:0/3279902217 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7800aa7f0 con 0x55b6514b1e40 2024-05-27T07:58:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.061+0000 7fc76effd700 1 -- 172.21.15.114:0/3279902217 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7800aa0f0 con 0x55b6514b1e40 2024-05-27T07:58:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.061+0000 7fc76effd700 1 -- 172.21.15.114:0/3279902217 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7800aa5b0 con 0x55b6514b1e40 2024-05-27T07:58:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.061+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/3279902217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6514b1e40 msgr2=0x55b65162fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.061+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/3279902217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6514b1e40 0x55b65162fa40 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc7800e0530 tx=0x7fc780009310 comp rx=0 tx=0).stop 2024-05-27T07:58:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.061+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/3279902217 shutdown_connections 2024-05-27T07:58:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.061+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/3279902217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b651635c50 0x55b6516457e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.061+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/3279902217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6514b1e40 0x55b65162fa40 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.061+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/3279902217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b65162ff80 0x55b651639bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.061+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/3279902217 >> 172.21.15.114:0/3279902217 conn(0x55b65150caf0 msgr2=0x55b65162a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.062+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/3279902217 shutdown_connections 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.062+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/3279902217 wait complete. 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.062+0000 7fc79a1f4b80 1 -- start start 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.062+0000 7fc79a1f4b80 1 Processor -- start 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.062+0000 7fc79a1f4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6514b1e40 0x55b651652120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.062+0000 7fc79a1f4b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b65162ff80 0x55b6516419d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.062+0000 7fc79a1f4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 0x55b651642020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.062+0000 7fc79a1f4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841ae9a0 con 0x55b651635c50 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.062+0000 7fc79a1f4b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841ae330 con 0x55b65162ff80 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.062+0000 7fc79a1f4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b651638650 con 0x55b6514b1e40 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.062+0000 7fc79a1f4b80 10 client.0 _ll_register_callbacks cb 0x55b6514e9920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc789d7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 0x55b651642020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc78957b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6514b1e40 0x55b651652120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc789d7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 0x55b651642020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40248/0 (socket says 172.21.15.114:40248) 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc789d7c700 1 -- 172.21.15.114:0/333400137 learned_addr learned my addr 172.21.15.114:0/333400137 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc788d7a700 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b65162ff80 0x55b6516419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc789d7c700 1 -- 172.21.15.114:0/333400137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6514b1e40 msgr2=0x55b651652120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc789d7c700 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6514b1e40 0x55b651652120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc789d7c700 1 -- 172.21.15.114:0/333400137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b65162ff80 msgr2=0x55b6516419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc789d7c700 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b65162ff80 0x55b6516419d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc789d7c700 1 -- 172.21.15.114:0/333400137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6516508f0 con 0x55b651635c50 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc78957b700 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6514b1e40 0x55b651652120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc788d7a700 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b65162ff80 0x55b6516419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc789d7c700 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 0x55b651642020 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fc77c01ea10 tx=0x7fc77c01e3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9462d35376ee153 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc76cff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:58:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc76cff9700 1 -- 172.21.15.114:0/333400137 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c02f060 con 0x55b651635c50 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc76cff9700 1 -- 172.21.15.114:0/333400137 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(158..158 src has 1..158) ==== 9466+0+0 (secure 0 0 0) 0x7fc77c003c00 con 0x55b651635c50 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.063+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/333400137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7800aa0f0 con 0x55b651635c50 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.064+0000 7fc76cff9700 1 -- 172.21.15.114:0/333400137 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7fc77c036050 con 0x55b651635c50 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.064+0000 7fc79a1f4b80 10 client.19644 fetch_fsmap learned FSMap version 106 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.064+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/333400137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x55b6516b3a20 con 0x55b651635c50 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.064+0000 7fc76cff9700 1 -- 172.21.15.114:0/333400137 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7800aa0f0 con 0x55b651635c50 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.064+0000 7fc76cff9700 1 -- 172.21.15.114:0/333400137 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 106) ==== 4095+0+0 (secure 0 0 0) 0x7fc77c02b490 con 0x55b651635c50 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.064+0000 7fc79a1f4b80 10 client.19644 fetch_fsmap finished waiting for FSMap version 106 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.064+0000 7fc79a1f4b80 10 client.19644 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.064+0000 7fc79a1f4b80 20 client.19644 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.064+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b6516b4100 0x55b6516b6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.064+0000 7fc79a1f4b80 4 client.19644 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.064+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/333400137 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b651560be0 con 0x55b6516b4100 2024-05-27T07:58:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.064+0000 7fc78957b700 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b6516b4100 0x55b6516b6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.065+0000 7fc76cff9700 10 client.19644 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.065+0000 7fc78957b700 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b6516b4100 0x55b6516b6520 crc :-1 s=READY pgs=499 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=407ff1279d6434ad server_cookie=d6bd86710e981c6c in_seq=0 out_seq=0 2024-05-27T07:58:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc76cff9700 1 -- 172.21.15.114:0/333400137 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b651560be0 con 0x55b6516b4100 2024-05-27T07:58:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc76cff9700 10 client.19644 handle_command_reply: tid=0 2024-05-27T07:58:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc79a1f4b80 1 client.19644 shutdown 2024-05-27T07:58:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/333400137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 msgr2=0x55b651642020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 0x55b651642020 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fc77c01ea10 tx=0x7fc77c01e3e0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/333400137 shutdown_connections 2024-05-27T07:58:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b651635c50 0x55b651642020 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6514b1e40 0x55b651652120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b6516b4100 0x55b6516b6520 crc :-1 s=READY pgs=499 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc79a1f4b80 1 --2- 172.21.15.114:0/333400137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b65162ff80 0x55b6516419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/333400137 >> 172.21.15.114:0/333400137 conn(0x55b65150caf0 msgr2=0x55b65162a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc76cff9700 0 client.19644 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/333400137 shutdown_connections 2024-05-27T07:58:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc79a1f4b80 1 -- 172.21.15.114:0/333400137 wait complete. 2024-05-27T07:58:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.070+0000 7fc79a1f4b80 20 client.19644 trim_cache size 0 max 16384 2024-05-27T07:58:38.073 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:58:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.071+0000 7fc78b7df700 1 -- 172.21.15.114:0/1719435871 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7700b29f0 msgr2=0x7fc7700b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.071+0000 7fc78b7df700 1 --2- 172.21.15.114:0/1719435871 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7700b29f0 0x7fc7700b4f00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc78419b270 tx=0x7fc774009260 comp rx=0 tx=0).stop 2024-05-27T07:58:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.071+0000 7fc78b7df700 1 -- 172.21.15.114:0/1719435871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7841034b0 msgr2=0x7fc78419f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.071+0000 7fc78b7df700 1 --2- 172.21.15.114:0/1719435871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7841034b0 0x7fc78419f9b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc7800098e0 tx=0x7fc780004270 comp rx=0 tx=0).stop 2024-05-27T07:58:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.071+0000 7fc78b7df700 1 -- 172.21.15.114:0/1719435871 shutdown_connections 2024-05-27T07:58:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.071+0000 7fc78b7df700 1 --2- 172.21.15.114:0/1719435871 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc784108730 0x7fc784199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.071+0000 7fc78b7df700 1 --2- 172.21.15.114:0/1719435871 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fc7700b29f0 0x7fc7700b4f00 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.071+0000 7fc78b7df700 1 --2- 172.21.15.114:0/1719435871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7841034b0 0x7fc78419f9b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.071+0000 7fc78b7df700 1 --2- 172.21.15.114:0/1719435871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc784107de0 0x7fc78419fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.071+0000 7fc78b7df700 1 -- 172.21.15.114:0/1719435871 >> 172.21.15.114:0/1719435871 conn(0x7fc7840fedb0 msgr2=0x7fc784109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.071+0000 7fc78b7df700 1 -- 172.21.15.114:0/1719435871 shutdown_connections 2024-05-27T07:58:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:38.071+0000 7fc78b7df700 1 -- 172.21.15.114:0/1719435871 wait complete. 2024-05-27T07:58:38.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:37 smithi163 ceph-mon[36451]: pgmap v650: 129 pgs: 129 active+clean; 380 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.8 MiB/s wr, 501 op/s 2024-05-27T07:58:38.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:37 smithi186 ceph-mon[36647]: pgmap v650: 129 pgs: 129 active+clean; 380 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.8 MiB/s wr, 501 op/s 2024-05-27T07:58:38.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 69099 12.29 MB/sec execute 170 sec latency 161.980 ms 2024-05-27T07:58:39.616 INFO:tasks.workunit.client.0.smithi114.stdout: 10 69385 12.28 MB/sec execute 171 sec latency 120.817 ms 2024-05-27T07:58:39.884 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:58:39.884 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:58:40.175 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:58:40.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:39 smithi163 ceph-mon[36451]: pgmap v651: 129 pgs: 129 active+clean; 376 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.3 MiB/s wr, 590 op/s 2024-05-27T07:58:40.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:39 smithi186 ceph-mon[36647]: pgmap v651: 129 pgs: 129 active+clean; 376 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.3 MiB/s wr, 590 op/s 2024-05-27T07:58:40.298 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:39 smithi114 ceph-mon[32038]: pgmap v651: 129 pgs: 129 active+clean; 376 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.3 MiB/s wr, 590 op/s 2024-05-27T07:58:40.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 69770 12.35 MB/sec execute 172 sec latency 139.975 ms 2024-05-27T07:58:41.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 70100 12.42 MB/sec execute 173 sec latency 147.751 ms 2024-05-27T07:58:42.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:41 smithi114 ceph-mon[32038]: pgmap v652: 129 pgs: 129 active+clean; 376 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.4 MiB/s wr, 573 op/s 2024-05-27T07:58:42.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:41 smithi163 ceph-mon[36451]: pgmap v652: 129 pgs: 129 active+clean; 376 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.4 MiB/s wr, 573 op/s 2024-05-27T07:58:42.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:41 smithi186 ceph-mon[36647]: pgmap v652: 129 pgs: 129 active+clean; 376 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.4 MiB/s wr, 573 op/s 2024-05-27T07:58:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.333+0000 7fa6cd93f700 1 -- 172.21.15.114:0/964988728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c8107de0 msgr2=0x7fa6c81081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.333+0000 7fa6cd93f700 1 --2- 172.21.15.114:0/964988728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c8107de0 0x7fa6c81081f0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009920 tx=0x7fa6bc009c30 comp rx=0 tx=0).stop 2024-05-27T07:58:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.333+0000 7fa6cd93f700 1 -- 172.21.15.114:0/964988728 shutdown_connections 2024-05-27T07:58:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.333+0000 7fa6cd93f700 1 --2- 172.21.15.114:0/964988728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6c8108730 0x7fa6c8110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.333+0000 7fa6cd93f700 1 --2- 172.21.15.114:0/964988728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6c81034b0 0x7fa6c81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.333+0000 7fa6cd93f700 1 --2- 172.21.15.114:0/964988728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c8107de0 0x7fa6c81081f0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.333+0000 7fa6cd93f700 1 -- 172.21.15.114:0/964988728 >> 172.21.15.114:0/964988728 conn(0x7fa6c80fedb0 msgr2=0x7fa6c81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:42.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.333+0000 7fa6cd93f700 1 -- 172.21.15.114:0/964988728 shutdown_connections 2024-05-27T07:58:42.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.333+0000 7fa6cd93f700 1 -- 172.21.15.114:0/964988728 wait complete. 2024-05-27T07:58:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.334+0000 7fa6cd93f700 1 Processor -- start 2024-05-27T07:58:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.334+0000 7fa6cd93f700 1 -- start start 2024-05-27T07:58:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.334+0000 7fa6cd93f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c81034b0 0x7fa6c819aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6cd93f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6c8107de0 0x7fa6c819b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6cd93f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6c8108730 0x7fa6c819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6cd93f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c8113340 con 0x7fa6c81034b0 2024-05-27T07:58:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6c6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c81034b0 0x7fa6c819aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6c6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c81034b0 0x7fa6c819aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40260/0 (socket says 172.21.15.114:40260) 2024-05-27T07:58:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6c6ffd700 1 -- 172.21.15.114:0/4250750534 learned_addr learned my addr 172.21.15.114:0/4250750534 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6c67fc700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6c8107de0 0x7fa6c819b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6cd93f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81131c0 con 0x7fa6c8108730 2024-05-27T07:58:42.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6c77fe700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6c8108730 0x7fa6c819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6cd93f700 1 -- 172.21.15.114:0/4250750534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81134c0 con 0x7fa6c8107de0 2024-05-27T07:58:42.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6c67fc700 1 -- 172.21.15.114:0/4250750534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6c8108730 msgr2=0x7fa6c819f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:42.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6c67fc700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6c8108730 0x7fa6c819f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6c67fc700 1 -- 172.21.15.114:0/4250750534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c81034b0 msgr2=0x7fa6c819aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:42.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6c67fc700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c81034b0 0x7fa6c819aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6c67fc700 1 -- 172.21.15.114:0/4250750534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c819fd70 con 0x7fa6c8107de0 2024-05-27T07:58:42.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.335+0000 7fa6c6ffd700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c81034b0 0x7fa6c819aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:42.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.336+0000 7fa6c67fc700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6c8107de0 0x7fa6c819b440 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc000c00 tx=0x7fa6bc00f950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:42.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.336+0000 7fa6cc93d700 1 -- 172.21.15.114:0/4250750534 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc005040 con 0x7fa6c8107de0 2024-05-27T07:58:42.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.336+0000 7fa6cc93d700 1 -- 172.21.15.114:0/4250750534 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6bc005690 con 0x7fa6c8107de0 2024-05-27T07:58:42.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.336+0000 7fa6cc93d700 1 -- 172.21.15.114:0/4250750534 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc003950 con 0x7fa6c8107de0 2024-05-27T07:58:42.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.336+0000 7fa6cd93f700 1 -- 172.21.15.114:0/4250750534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6c81a0000 con 0x7fa6c8107de0 2024-05-27T07:58:42.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.336+0000 7fa6cd93f700 1 -- 172.21.15.114:0/4250750534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6c81a0490 con 0x7fa6c8107de0 2024-05-27T07:58:42.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.339+0000 7fa6cc93d700 1 -- 172.21.15.114:0/4250750534 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa6bc003af0 con 0x7fa6c8107de0 2024-05-27T07:58:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6dc354b80 1 Processor -- start 2024-05-27T07:58:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6dc354b80 1 -- start start 2024-05-27T07:58:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6cc93d700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa6b00b29f0 0x7fa6b00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6c77fe700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa6b00b29f0 0x7fa6b00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6cc93d700 1 -- 172.21.15.114:0/4250750534 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 9466+0+0 (secure 0 0 0) 0x7fa6bc0e3840 con 0x7fa6c8107de0 2024-05-27T07:58:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6dc354b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb1c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6dc354b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb1ccd0 0x562a5eb16ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6c6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb1c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6c6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb1c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40276/0 (socket says 172.21.15.114:40276) 2024-05-27T07:58:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6c6ffd700 1 -- 172.21.15.114:0/4073761614 learned_addr learned my addr 172.21.15.114:0/4073761614 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/4073761614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb17000 0x562a5eb17410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6dc354b80 1 -- 172.21.15.114:0/4073761614 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562a5ea45d10 con 0x562a5e998ec0 2024-05-27T07:58:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6dc354b80 1 -- 172.21.15.114:0/4073761614 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562a5ea358b0 con 0x562a5eb1ccd0 2024-05-27T07:58:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.340+0000 7fa6dc354b80 1 -- 172.21.15.114:0/4073761614 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562a5ea398f0 con 0x562a5eb17000 2024-05-27T07:58:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.341+0000 7fa6c67fc700 1 --2- 172.21.15.114:0/4073761614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb1ccd0 0x562a5eb16ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.341+0000 7fa6c77fe700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa6b00b29f0 0x7fa6b00b4f00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa6c000f540 tx=0x7fa6c000b040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.341+0000 7fa6c77fe700 1 --2- 172.21.15.114:0/4073761614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb17000 0x562a5eb17410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.341+0000 7fa6c6ffd700 1 -- 172.21.15.114:0/4073761614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb17000 msgr2=0x562a5eb17410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.341+0000 7fa6c6ffd700 1 --2- 172.21.15.114:0/4073761614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb17000 0x562a5eb17410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.341+0000 7fa6c6ffd700 1 -- 172.21.15.114:0/4073761614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb1ccd0 msgr2=0x562a5eb16ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.341+0000 7fa6c6ffd700 1 --2- 172.21.15.114:0/4073761614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb1ccd0 0x562a5eb16ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.341+0000 7fa6c6ffd700 1 -- 172.21.15.114:0/4073761614 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a5e985950 con 0x562a5e998ec0 2024-05-27T07:58:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.341+0000 7fa6c6ffd700 1 --2- 172.21.15.114:0/4073761614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb1c790 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80117e0 tx=0x7fa6b8011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b479a3fe6d842e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.341+0000 7fa6adffb700 1 -- 172.21.15.114:0/4073761614 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b801e070 con 0x562a5e998ec0 2024-05-27T07:58:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.342+0000 7fa6adffb700 1 -- 172.21.15.114:0/4073761614 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6b800f930 con 0x562a5e998ec0 2024-05-27T07:58:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.342+0000 7fa6adffb700 1 -- 172.21.15.114:0/4073761614 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8019a70 con 0x562a5e998ec0 2024-05-27T07:58:42.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.342+0000 7fa6dc354b80 1 -- 172.21.15.114:0/4073761614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 msgr2=0x562a5eb1c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:42.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.342+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/4073761614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb1c790 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80117e0 tx=0x7fa6b8011af0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.343+0000 7fa6dc354b80 1 -- 172.21.15.114:0/4073761614 shutdown_connections 2024-05-27T07:58:42.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.343+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/4073761614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb17000 0x562a5eb17410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.343+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/4073761614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb1c790 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.343+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/4073761614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb1ccd0 0x562a5eb16ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.343+0000 7fa6dc354b80 1 -- 172.21.15.114:0/4073761614 >> 172.21.15.114:0/4073761614 conn(0x562a5e9f3b70 msgr2=0x562a5eb116c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:42.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.343+0000 7fa6dc354b80 1 -- 172.21.15.114:0/4073761614 shutdown_connections 2024-05-27T07:58:42.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.343+0000 7fa6dc354b80 1 -- 172.21.15.114:0/4073761614 wait complete. 2024-05-27T07:58:42.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.343+0000 7fa6dc354b80 1 -- start start 2024-05-27T07:58:42.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.343+0000 7fa6dc354b80 1 Processor -- start 2024-05-27T07:58:42.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6dc354b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb203f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6dc354b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb17000 0x562a5eb20930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6dc354b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb1ccd0 0x562a5eb20e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6dc354b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562a5e93c020 con 0x562a5e998ec0 2024-05-27T07:58:42.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6dc354b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562a5e9d7840 con 0x562a5eb17000 2024-05-27T07:58:42.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6dc354b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562a5eb27bd0 con 0x562a5eb1ccd0 2024-05-27T07:58:42.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6dc354b80 10 client.0 _ll_register_callbacks cb 0x562a5e9d09a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:42.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6c77fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb1ccd0 0x562a5eb20e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6c6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb203f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6c77fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb1ccd0 0x562a5eb20e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:45612/0 (socket says 172.21.15.114:45612) 2024-05-27T07:58:42.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6c77fe700 1 -- 172.21.15.114:0/1343063376 learned_addr learned my addr 172.21.15.114:0/1343063376 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:42.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6c67fc700 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb17000 0x562a5eb20930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6c77fe700 1 -- 172.21.15.114:0/1343063376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb17000 msgr2=0x562a5eb20930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6c77fe700 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb17000 0x562a5eb20930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6c77fe700 1 -- 172.21.15.114:0/1343063376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 msgr2=0x562a5eb203f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6c77fe700 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb203f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6c77fe700 1 -- 172.21.15.114:0/1343063376 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a5e9d4710 con 0x562a5eb1ccd0 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6c67fc700 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb17000 0x562a5eb20930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.344+0000 7fa6c6ffd700 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb203f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.345+0000 7fa6c77fe700 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb1ccd0 0x562a5eb20e70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0008b80 tx=0x7fa6c0004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=407cb3a9f3fcf43d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.345+0000 7fa6a6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.345+0000 7fa6a6ffd700 1 -- 172.21.15.114:0/1343063376 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0025070 con 0x562a5eb1ccd0 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.345+0000 7fa6dc354b80 1 -- 172.21.15.114:0/1343063376 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6b800f930 con 0x562a5eb1ccd0 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.345+0000 7fa6a6ffd700 1 -- 172.21.15.114:0/1343063376 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(158..158 src has 1..158) ==== 9466+0+0 (secure 0 0 0) 0x7fa6c0017ce0 con 0x562a5eb1ccd0 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.345+0000 7fa6a6ffd700 1 -- 172.21.15.114:0/1343063376 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7fa6c0030050 con 0x562a5eb1ccd0 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.345+0000 7fa6dc354b80 10 client.29134 fetch_fsmap learned FSMap version 106 2024-05-27T07:58:42.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.345+0000 7fa6dc354b80 1 -- 172.21.15.114:0/1343063376 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x562a5eb32720 con 0x562a5eb1ccd0 2024-05-27T07:58:42.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.345+0000 7fa6a6ffd700 1 -- 172.21.15.114:0/1343063376 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6b800f930 con 0x562a5eb1ccd0 2024-05-27T07:58:42.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.345+0000 7fa6a6ffd700 1 -- 172.21.15.114:0/1343063376 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 106) ==== 4095+0+0 (secure 0 0 0) 0x7fa6c0020040 con 0x562a5eb1ccd0 2024-05-27T07:58:42.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.345+0000 7fa6dc354b80 10 client.29134 fetch_fsmap finished waiting for FSMap version 106 2024-05-27T07:58:42.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.345+0000 7fa6dc354b80 10 client.29134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:42.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.346+0000 7fa6dc354b80 20 client.29134 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:42.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.346+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562a5eb35000 0x562a5eb374d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.346+0000 7fa6dc354b80 4 client.29134 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:58:42.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.346+0000 7fa6dc354b80 1 -- 172.21.15.114:0/1343063376 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a5eae2a70 con 0x562a5eb35000 2024-05-27T07:58:42.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.346+0000 7fa6c6ffd700 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562a5eb35000 0x562a5eb374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.346+0000 7fa6a6ffd700 10 client.29134 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:42.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.346+0000 7fa6c6ffd700 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562a5eb35000 0x562a5eb374d0 crc :-1 s=READY pgs=500 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2020b7aa07a5867 server_cookie=4e5ab9c3934a13fb in_seq=0 out_seq=0 2024-05-27T07:58:42.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6a6ffd700 1 -- 172.21.15.114:0/1343063376 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562a5eae2a70 con 0x562a5eb35000 2024-05-27T07:58:42.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6a6ffd700 10 client.29134 handle_command_reply: tid=0 2024-05-27T07:58:42.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6dc354b80 1 client.29134 shutdown 2024-05-27T07:58:42.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6dc354b80 1 -- 172.21.15.114:0/1343063376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb1ccd0 msgr2=0x562a5eb20e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:42.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb1ccd0 0x562a5eb20e70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0008b80 tx=0x7fa6c0004ad0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6dc354b80 1 -- 172.21.15.114:0/1343063376 shutdown_connections 2024-05-27T07:58:42.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb1ccd0 0x562a5eb20e70 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb203f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562a5eb35000 0x562a5eb374d0 crc :-1 s=READY pgs=500 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6a6ffd700 0 client.29134 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:42.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/1343063376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb17000 0x562a5eb20930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6dc354b80 1 -- 172.21.15.114:0/1343063376 >> 172.21.15.114:0/1343063376 conn(0x562a5e9f3b70 msgr2=0x562a5eb15ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:42.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6dc354b80 1 -- 172.21.15.114:0/1343063376 shutdown_connections 2024-05-27T07:58:42.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6dc354b80 1 -- 172.21.15.114:0/1343063376 wait complete. 2024-05-27T07:58:42.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.351+0000 7fa6dc354b80 20 client.29134 trim_cache size 0 max 16384 2024-05-27T07:58:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6dc354b80 1 Processor -- start 2024-05-27T07:58:42.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6dc354b80 1 -- start start 2024-05-27T07:58:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6dc354b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb1aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6dc354b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb17000 0x562a5eb30c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6dc354b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb1ccd0 0x562a5eb22060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6dc354b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81adfd0 con 0x562a5e998ec0 2024-05-27T07:58:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6dc354b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81ae6a0 con 0x562a5eb1ccd0 2024-05-27T07:58:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6dc354b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81acf50 con 0x562a5eb17000 2024-05-27T07:58:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6c6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb1aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6c67fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb17000 0x562a5eb30c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6c6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb1aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40288/0 (socket says 172.21.15.114:40288) 2024-05-27T07:58:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6c6ffd700 1 -- 172.21.15.114:0/3520091873 learned_addr learned my addr 172.21.15.114:0/3520091873 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6c77fe700 1 --2- 172.21.15.114:0/3520091873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb1ccd0 0x562a5eb22060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6c6ffd700 1 -- 172.21.15.114:0/3520091873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb17000 msgr2=0x562a5eb30c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6c6ffd700 1 --2- 172.21.15.114:0/3520091873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb17000 0x562a5eb30c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6c6ffd700 1 -- 172.21.15.114:0/3520091873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb1ccd0 msgr2=0x562a5eb22060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:42.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6c6ffd700 1 --2- 172.21.15.114:0/3520091873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb1ccd0 0x562a5eb22060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6c6ffd700 1 -- 172.21.15.114:0/3520091873 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a5eb1e980 con 0x562a5e998ec0 2024-05-27T07:58:42.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6c77fe700 1 --2- 172.21.15.114:0/3520091873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb1ccd0 0x562a5eb22060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:42.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.373+0000 7fa6c67fc700 1 --2- 172.21.15.114:0/3520091873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb17000 0x562a5eb30c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:42.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.374+0000 7fa6c6ffd700 1 --2- 172.21.15.114:0/3520091873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb1aac0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8011ef0 tx=0x7fa6b800fef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7230ea860a281a1c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:42.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.374+0000 7fa6ad7fa700 1 -- 172.21.15.114:0/3520091873 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b801e040 con 0x562a5e998ec0 2024-05-27T07:58:42.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.374+0000 7fa6ad7fa700 1 -- 172.21.15.114:0/3520091873 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6b800f8b0 con 0x562a5e998ec0 2024-05-27T07:58:42.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.374+0000 7fa6ad7fa700 1 -- 172.21.15.114:0/3520091873 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8003a70 con 0x562a5e998ec0 2024-05-27T07:58:42.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.375+0000 7fa6dc354b80 1 -- 172.21.15.114:0/3520091873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 msgr2=0x562a5eb1aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:42.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.375+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/3520091873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb1aac0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8011ef0 tx=0x7fa6b800fef0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.375+0000 7fa6dc354b80 1 -- 172.21.15.114:0/3520091873 shutdown_connections 2024-05-27T07:58:42.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.375+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/3520091873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb1ccd0 0x562a5eb22060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.375+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/3520091873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5e998ec0 0x562a5eb1aac0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.375+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/3520091873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5eb17000 0x562a5eb30c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.375+0000 7fa6dc354b80 1 -- 172.21.15.114:0/3520091873 >> 172.21.15.114:0/3520091873 conn(0x562a5e9f3b70 msgr2=0x562a5eb21950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.375+0000 7fa6dc354b80 1 -- 172.21.15.114:0/3520091873 shutdown_connections 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.375+0000 7fa6dc354b80 1 -- 172.21.15.114:0/3520091873 wait complete. 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.375+0000 7fa6dc354b80 1 -- start start 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.375+0000 7fa6dc354b80 1 Processor -- start 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6dc354b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5e998ec0 0x562a5eb2b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6dc354b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb17000 0x562a5eb2b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6dc354b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5eb1ccd0 0x562a5eb9ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6dc354b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81ac880 con 0x562a5eb1ccd0 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6dc354b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81ac210 con 0x562a5eb17000 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6dc354b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562a5eb0c450 con 0x562a5e998ec0 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6dc354b80 10 client.0 _ll_register_callbacks cb 0x562a5e9d09a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:42.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6c6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5e998ec0 0x562a5eb2b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6c77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5eb1ccd0 0x562a5eb9ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6c6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5e998ec0 0x562a5eb2b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:45632/0 (socket says 172.21.15.114:45632) 2024-05-27T07:58:42.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6c6ffd700 1 -- 172.21.15.114:0/3409059464 learned_addr learned my addr 172.21.15.114:0/3409059464 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:42.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6c67fc700 1 --2- 172.21.15.114:0/3409059464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb17000 0x562a5eb2b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6c77fe700 1 -- 172.21.15.114:0/3409059464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5e998ec0 msgr2=0x562a5eb2b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:42.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6c77fe700 1 --2- 172.21.15.114:0/3409059464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5e998ec0 0x562a5eb2b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6c77fe700 1 -- 172.21.15.114:0/3409059464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb17000 msgr2=0x562a5eb2b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:42.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6c77fe700 1 --2- 172.21.15.114:0/3409059464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb17000 0x562a5eb2b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6c77fe700 1 -- 172.21.15.114:0/3409059464 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a5eb9f0f0 con 0x562a5eb1ccd0 2024-05-27T07:58:42.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6c6ffd700 1 --2- 172.21.15.114:0/3409059464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5e998ec0 0x562a5eb2b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:42.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6c77fe700 1 --2- 172.21.15.114:0/3409059464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5eb1ccd0 0x562a5eb9ebb0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fa6c000fc90 tx=0x7fa6c0017790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b0a0e3bf66f7f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:42.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.376+0000 7fa6a6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:58:42.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.377+0000 7fa6a6ffd700 1 -- 172.21.15.114:0/3409059464 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0038020 con 0x562a5eb1ccd0 2024-05-27T07:58:42.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.377+0000 7fa6a6ffd700 1 -- 172.21.15.114:0/3409059464 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(158..158 src has 1..158) ==== 9466+0+0 (secure 0 0 0) 0x7fa6c0004b40 con 0x562a5eb1ccd0 2024-05-27T07:58:42.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.377+0000 7fa6dc354b80 1 -- 172.21.15.114:0/3409059464 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6b800f8b0 con 0x562a5eb1ccd0 2024-05-27T07:58:42.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.377+0000 7fa6a6ffd700 1 -- 172.21.15.114:0/3409059464 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7fa6c0043050 con 0x562a5eb1ccd0 2024-05-27T07:58:42.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.377+0000 7fa6dc354b80 10 client.19662 fetch_fsmap learned FSMap version 106 2024-05-27T07:58:42.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.377+0000 7fa6dc354b80 1 -- 172.21.15.114:0/3409059464 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x562a5eba0370 con 0x562a5eb1ccd0 2024-05-27T07:58:42.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.377+0000 7fa6a6ffd700 1 -- 172.21.15.114:0/3409059464 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6c001b070 con 0x562a5eb1ccd0 2024-05-27T07:58:42.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.377+0000 7fa6a6ffd700 1 -- 172.21.15.114:0/3409059464 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 106) ==== 4095+0+0 (secure 0 0 0) 0x7fa6c0020040 con 0x562a5eb1ccd0 2024-05-27T07:58:42.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.377+0000 7fa6dc354b80 10 client.19662 fetch_fsmap finished waiting for FSMap version 106 2024-05-27T07:58:42.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.377+0000 7fa6dc354b80 10 client.19662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:42.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.377+0000 7fa6dc354b80 20 client.19662 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:42.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.377+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/3409059464 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562a5eba0a70 0x562a5eba2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:42.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.378+0000 7fa6dc354b80 4 client.19662 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:58:42.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.378+0000 7fa6dc354b80 1 -- 172.21.15.114:0/3409059464 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fa6b800f8b0 con 0x562a5eba0a70 2024-05-27T07:58:42.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.378+0000 7fa6c6ffd700 1 --2- 172.21.15.114:0/3409059464 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562a5eba0a70 0x562a5eba2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:42.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.378+0000 7fa6a6ffd700 10 client.19662 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:42.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.378+0000 7fa6c6ffd700 1 --2- 172.21.15.114:0/3409059464 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562a5eba0a70 0x562a5eba2e90 crc :-1 s=READY pgs=501 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba0e7997306a7855 server_cookie=536aa68a300e1e0d in_seq=0 out_seq=0 2024-05-27T07:58:42.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.385+0000 7fa6a6ffd700 1 -- 172.21.15.114:0/3409059464 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fa6b800f8b0 con 0x562a5eba0a70 2024-05-27T07:58:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.385+0000 7fa6a6ffd700 10 client.19662 handle_command_reply: tid=0 2024-05-27T07:58:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.385+0000 7fa6dc354b80 1 client.19662 shutdown 2024-05-27T07:58:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.385+0000 7fa6dc354b80 1 -- 172.21.15.114:0/3409059464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5eb1ccd0 msgr2=0x562a5eb9ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.385+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/3409059464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5eb1ccd0 0x562a5eb9ebb0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fa6c000fc90 tx=0x7fa6c0017790 comp rx=0 tx=0).stop 2024-05-27T07:58:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.386+0000 7fa6dc354b80 1 -- 172.21.15.114:0/3409059464 shutdown_connections 2024-05-27T07:58:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.386+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/3409059464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562a5eb1ccd0 0x562a5eb9ebb0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.386+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/3409059464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a5e998ec0 0x562a5eb2b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.386+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/3409059464 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562a5eba0a70 0x562a5eba2e90 crc :-1 s=READY pgs=501 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.386+0000 7fa6dc354b80 1 --2- 172.21.15.114:0/3409059464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a5eb17000 0x562a5eb2b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.386+0000 7fa6dc354b80 1 -- 172.21.15.114:0/3409059464 >> 172.21.15.114:0/3409059464 conn(0x562a5e9f3b70 msgr2=0x562a5eb1a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.386+0000 7fa6a6ffd700 0 client.19662 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.386+0000 7fa6dc354b80 1 -- 172.21.15.114:0/3409059464 shutdown_connections 2024-05-27T07:58:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.386+0000 7fa6dc354b80 1 -- 172.21.15.114:0/3409059464 wait complete. 2024-05-27T07:58:42.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.387+0000 7fa6dc354b80 20 client.19662 trim_cache size 0 max 16384 2024-05-27T07:58:42.390 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:58:42.390 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:58:42.390 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "f7462d28-89c3-42b5-8438-f21f57975515", 2024-05-27T07:58:42.390 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:58:42.390 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:58:42.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.388+0000 7fa6cd93f700 1 -- 172.21.15.114:0/4250750534 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa6b00b29f0 msgr2=0x7fa6b00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:42.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.388+0000 7fa6cd93f700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa6b00b29f0 0x7fa6b00b4f00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa6c000f540 tx=0x7fa6c000b040 comp rx=0 tx=0).stop 2024-05-27T07:58:42.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.388+0000 7fa6cd93f700 1 -- 172.21.15.114:0/4250750534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6c8107de0 msgr2=0x7fa6c819b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:42.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.388+0000 7fa6cd93f700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6c8107de0 0x7fa6c819b440 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc000c00 tx=0x7fa6bc00f950 comp rx=0 tx=0).stop 2024-05-27T07:58:42.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.389+0000 7fa6cd93f700 1 -- 172.21.15.114:0/4250750534 shutdown_connections 2024-05-27T07:58:42.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.389+0000 7fa6cd93f700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa6b00b29f0 0x7fa6b00b4f00 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.389+0000 7fa6cd93f700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6c8108730 0x7fa6c819f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.389+0000 7fa6cd93f700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c81034b0 0x7fa6c819aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.389+0000 7fa6cd93f700 1 --2- 172.21.15.114:0/4250750534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6c8107de0 0x7fa6c819b440 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:42.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.389+0000 7fa6cd93f700 1 -- 172.21.15.114:0/4250750534 >> 172.21.15.114:0/4250750534 conn(0x7fa6c80fedb0 msgr2=0x7fa6c806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:42.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.389+0000 7fa6cd93f700 1 -- 172.21.15.114:0/4250750534 shutdown_connections 2024-05-27T07:58:42.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:42.389+0000 7fa6cd93f700 1 -- 172.21.15.114:0/4250750534 wait complete. 2024-05-27T07:58:42.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 70348 12.41 MB/sec execute 174 sec latency 205.111 ms 2024-05-27T07:58:43.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:42 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:58:43.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:42 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:58:43.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:42 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:58:43.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:42 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:58:43.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:42 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:58:43.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:42 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:58:43.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:42 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:58:43.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:42 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:58:43.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:42 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:58:43.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:58:43.455 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:43 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:58:43.456 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:43 smithi114 systemd[1]: Stopped Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:58:43.456 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:43 smithi114 systemd[1]: Starting Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:58:43.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 70549 12.40 MB/sec execute 175 sec latency 201.335 ms 2024-05-27T07:58:43.722 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:43 smithi114 podman[132166]: 2024-05-27 07:58:43.357103866 +0000 UTC m=+0.018885997 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:58:43.723 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:43 smithi114 podman[132166]: 2024-05-27 07:58:43.485364357 +0000 UTC m=+0.147146481 container create 88a389f76c0eaa1bf1b65f80eb02885c5530a5f79c6ceeaeb00e0876248bc553 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:58:43.723 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:43 smithi114 podman[132166]: 2024-05-27 07:58:43.721748232 +0000 UTC m=+0.383530356 container init 88a389f76c0eaa1bf1b65f80eb02885c5530a5f79c6ceeaeb00e0876248bc553 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-27T07:58:43.892 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:58:44.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:43 smithi114 ceph-mon[32038]: pgmap v653: 129 pgs: 129 active+clean; 383 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 488 op/s 2024-05-27T07:58:44.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:43 smithi114 ceph-mon[32038]: daemon mds.g restarted 2024-05-27T07:58:44.023 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:43 smithi114 podman[132166]: 2024-05-27 07:58:43.724941594 +0000 UTC m=+0.386723726 container start 88a389f76c0eaa1bf1b65f80eb02885c5530a5f79c6ceeaeb00e0876248bc553 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-05-27T07:58:44.023 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:43 smithi114 ceph-mds[132208]: -- 172.21.15.114:0/4196063109 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5618efcec4e0 con 0x5618efd37600 2024-05-27T07:58:44.023 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:43 smithi114 ceph-mds[132208]: -- 172.21.15.114:0/4196063109 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5618efcefa40 con 0x5618efd37600 2024-05-27T07:58:44.023 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:43 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: starting mds.g at 2024-05-27T07:58:44.023 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:43 smithi114 bash[132166]: 88a389f76c0eaa1bf1b65f80eb02885c5530a5f79c6ceeaeb00e0876248bc553 2024-05-27T07:58:44.023 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:58:43 smithi114 systemd[1]: Started Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:58:44.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:43 smithi163 ceph-mon[36451]: pgmap v653: 129 pgs: 129 active+clean; 383 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 488 op/s 2024-05-27T07:58:44.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:43 smithi163 ceph-mon[36451]: daemon mds.g restarted 2024-05-27T07:58:44.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:43 smithi186 ceph-mon[36647]: pgmap v653: 129 pgs: 129 active+clean; 383 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 488 op/s 2024-05-27T07:58:44.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:43 smithi186 ceph-mon[36647]: daemon mds.g restarted 2024-05-27T07:58:44.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 70858 12.38 MB/sec execute 176 sec latency 181.673 ms 2024-05-27T07:58:45.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:44 smithi114 ceph-mon[32038]: osdmap e159: 12 total, 12 up, 12 in 2024-05-27T07:58:45.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:44 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6838/3015532065,v1:172.21.15.114:6839/3015532065] up:boot 2024-05-27T07:58:45.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:44 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:58:45.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:44 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:58:45.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:44 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:58:45.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:44 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:58:45.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:44 smithi114 ceph-mon[32038]: pgmap v655: 129 pgs: 129 active+clean; 387 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 524 op/s 2024-05-27T07:58:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:44 smithi163 ceph-mon[36451]: osdmap e159: 12 total, 12 up, 12 in 2024-05-27T07:58:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:44 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6838/3015532065,v1:172.21.15.114:6839/3015532065] up:boot 2024-05-27T07:58:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:44 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:58:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:44 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:58:45.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:44 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:58:45.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:44 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:58:45.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:44 smithi163 ceph-mon[36451]: pgmap v655: 129 pgs: 129 active+clean; 387 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 524 op/s 2024-05-27T07:58:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:44 smithi186 ceph-mon[36647]: osdmap e159: 12 total, 12 up, 12 in 2024-05-27T07:58:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:44 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6838/3015532065,v1:172.21.15.114:6839/3015532065] up:boot 2024-05-27T07:58:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:44 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:58:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:44 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:58:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:44 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:58:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:44 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:58:45.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:44 smithi186 ceph-mon[36647]: pgmap v655: 129 pgs: 129 active+clean; 387 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 524 op/s 2024-05-27T07:58:45.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 71182 12.41 MB/sec execute 177 sec latency 223.143 ms 2024-05-27T07:58:45.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.932+0000 7f01dc5b7700 1 -- 172.21.15.114:0/2684902680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d41089e0 msgr2=0x7f01d4108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:45.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.932+0000 7f01dc5b7700 1 --2- 172.21.15.114:0/2684902680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d41089e0 0x7f01d4108df0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009f90 tx=0x7f01c400b330 comp rx=0 tx=0).stop 2024-05-27T07:58:45.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.932+0000 7f01dc5b7700 1 -- 172.21.15.114:0/2684902680 shutdown_connections 2024-05-27T07:58:45.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.932+0000 7f01dc5b7700 1 --2- 172.21.15.114:0/2684902680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01d4103300 0x7f01d410f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.932+0000 7f01dc5b7700 1 --2- 172.21.15.114:0/2684902680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d41089e0 0x7f01d4108df0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.932+0000 7f01dc5b7700 1 --2- 172.21.15.114:0/2684902680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01d41029b0 0x7f01d4102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.932+0000 7f01dc5b7700 1 -- 172.21.15.114:0/2684902680 >> 172.21.15.114:0/2684902680 conn(0x7f01d40fe510 msgr2=0x7f01d4100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:45.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.932+0000 7f01dc5b7700 1 -- 172.21.15.114:0/2684902680 shutdown_connections 2024-05-27T07:58:45.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.932+0000 7f01dc5b7700 1 -- 172.21.15.114:0/2684902680 wait complete. 2024-05-27T07:58:45.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.933+0000 7f01dc5b7700 1 Processor -- start 2024-05-27T07:58:45.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.933+0000 7f01dc5b7700 1 -- start start 2024-05-27T07:58:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.933+0000 7f01dc5b7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01d41029b0 0x7f01d419af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.933+0000 7f01dc5b7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01d4103300 0x7f01d419b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.933+0000 7f01dc5b7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d41089e0 0x7f01d419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.933+0000 7f01dc5b7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f01d4112330 con 0x7f01d41089e0 2024-05-27T07:58:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.933+0000 7f01dc5b7700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f01d41121b0 con 0x7f01d4103300 2024-05-27T07:58:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.933+0000 7f01dc5b7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f01d41124b0 con 0x7f01d41029b0 2024-05-27T07:58:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.933+0000 7f01dab54700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d41089e0 0x7f01d419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.933+0000 7f01da353700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01d41029b0 0x7f01d419af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.933+0000 7f01dab54700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d41089e0 0x7f01d419f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45892/0 (socket says 172.21.15.114:45892) 2024-05-27T07:58:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.933+0000 7f01dab54700 1 -- 172.21.15.114:0/2731705825 learned_addr learned my addr 172.21.15.114:0/2731705825 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.933+0000 7f01d9b52700 1 --2- 172.21.15.114:0/2731705825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01d4103300 0x7f01d419b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.934+0000 7f01dab54700 1 -- 172.21.15.114:0/2731705825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01d41029b0 msgr2=0x7f01d419af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:45.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.934+0000 7f01dab54700 1 --2- 172.21.15.114:0/2731705825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01d41029b0 0x7f01d419af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.934+0000 7f01dab54700 1 -- 172.21.15.114:0/2731705825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01d4103300 msgr2=0x7f01d419b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:45.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.934+0000 7f01dab54700 1 --2- 172.21.15.114:0/2731705825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01d4103300 0x7f01d419b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.934+0000 7f01dab54700 1 -- 172.21.15.114:0/2731705825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d419fdf0 con 0x7f01d41089e0 2024-05-27T07:58:45.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.934+0000 7f01dab54700 1 --2- 172.21.15.114:0/2731705825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d41089e0 0x7f01d419f750 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00eb20 tx=0x7f01cc00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:45.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.934+0000 7f01d37fe700 1 -- 172.21.15.114:0/2731705825 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc00cad0 con 0x7f01d41089e0 2024-05-27T07:58:45.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.934+0000 7f01dc5b7700 1 -- 172.21.15.114:0/2731705825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d41a00e0 con 0x7f01d41089e0 2024-05-27T07:58:45.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.934+0000 7f01d37fe700 1 -- 172.21.15.114:0/2731705825 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01cc0044d0 con 0x7f01d41089e0 2024-05-27T07:58:45.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.934+0000 7f01d37fe700 1 -- 172.21.15.114:0/2731705825 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc003e40 con 0x7f01d41089e0 2024-05-27T07:58:45.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.935+0000 7f01dc5b7700 1 -- 172.21.15.114:0/2731705825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d41a7a00 con 0x7f01d41089e0 2024-05-27T07:58:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.938+0000 7f01d37fe700 1 -- 172.21.15.114:0/2731705825 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f01cc01d060 con 0x7f01d41089e0 2024-05-27T07:58:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.939+0000 7f01d37fe700 1 --2- 172.21.15.114:0/2731705825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f01bc0b29f0 0x7f01bc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.939+0000 7f01da353700 1 --2- 172.21.15.114:0/2731705825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f01bc0b29f0 0x7f01bc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.939+0000 7f01d37fe700 1 -- 172.21.15.114:0/2731705825 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 9466+0+0 (secure 0 0 0) 0x7f01cc01b070 con 0x7f01d41089e0 2024-05-27T07:58:45.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.939+0000 7f01da353700 1 --2- 172.21.15.114:0/2731705825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f01bc0b29f0 0x7f01bc0b4f00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f01c400b6c0 tx=0x7f01c40096c0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:45.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.940+0000 7f01eafccb80 1 Processor -- start 2024-05-27T07:58:45.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.940+0000 7f01eafccb80 1 -- start start 2024-05-27T07:58:45.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.940+0000 7f01eafccb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c8125b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.940+0000 7f01eafccb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125b7d0 0x555c8125fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.940+0000 7f01eafccb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125ffb0 0x555c812603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.940+0000 7f01eafccb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555c81184d70 con 0x555c810d7e40 2024-05-27T07:58:45.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.940+0000 7f01eafccb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555c81174910 con 0x555c8125ffb0 2024-05-27T07:58:45.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.940+0000 7f01eafccb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555c81178950 con 0x555c8125b7d0 2024-05-27T07:58:45.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.940+0000 7f01da353700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c8125b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.940+0000 7f01da353700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c8125b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45902/0 (socket says 172.21.15.114:45902) 2024-05-27T07:58:45.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.940+0000 7f01da353700 1 -- 172.21.15.114:0/3282949009 learned_addr learned my addr 172.21.15.114:0/3282949009 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:45.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.940+0000 7f01d9b52700 1 --2- 172.21.15.114:0/3282949009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125b7d0 0x555c8125fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.940+0000 7f01dab54700 1 --2- 172.21.15.114:0/3282949009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125ffb0 0x555c812603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.941+0000 7f01da353700 1 -- 172.21.15.114:0/3282949009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125b7d0 msgr2=0x555c8125fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:45.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.941+0000 7f01da353700 1 --2- 172.21.15.114:0/3282949009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125b7d0 0x555c8125fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.941+0000 7f01da353700 1 -- 172.21.15.114:0/3282949009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125ffb0 msgr2=0x555c812603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:45.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.941+0000 7f01da353700 1 --2- 172.21.15.114:0/3282949009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125ffb0 0x555c812603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.941+0000 7f01da353700 1 -- 172.21.15.114:0/3282949009 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c8111dfb0 con 0x555c810d7e40 2024-05-27T07:58:45.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.941+0000 7f01da353700 1 --2- 172.21.15.114:0/3282949009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c8125b290 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f01c4015000 tx=0x7f01c401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede8b8c2dbd4b5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:45.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.941+0000 7f01d0ff9700 1 -- 172.21.15.114:0/3282949009 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c400bb70 con 0x555c810d7e40 2024-05-27T07:58:45.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.941+0000 7f01d0ff9700 1 -- 172.21.15.114:0/3282949009 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01c402d070 con 0x555c810d7e40 2024-05-27T07:58:45.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.941+0000 7f01d0ff9700 1 -- 172.21.15.114:0/3282949009 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c4028780 con 0x555c810d7e40 2024-05-27T07:58:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.942+0000 7f01eafccb80 1 -- 172.21.15.114:0/3282949009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 msgr2=0x555c8125b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.942+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3282949009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c8125b290 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f01c4015000 tx=0x7f01c401d000 comp rx=0 tx=0).stop 2024-05-27T07:58:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.942+0000 7f01eafccb80 1 -- 172.21.15.114:0/3282949009 shutdown_connections 2024-05-27T07:58:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.942+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3282949009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125ffb0 0x555c812603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.942+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3282949009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c8125b290 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.942+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3282949009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125b7d0 0x555c8125fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.942+0000 7f01eafccb80 1 -- 172.21.15.114:0/3282949009 >> 172.21.15.114:0/3282949009 conn(0x555c810da770 msgr2=0x555c81251d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.942+0000 7f01eafccb80 1 -- 172.21.15.114:0/3282949009 shutdown_connections 2024-05-27T07:58:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.942+0000 7f01eafccb80 1 -- 172.21.15.114:0/3282949009 wait complete. 2024-05-27T07:58:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.942+0000 7f01eafccb80 1 -- start start 2024-05-27T07:58:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01eafccb80 1 Processor -- start 2024-05-27T07:58:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01eafccb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c81264c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01eafccb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c81265170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01eafccb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 0x555c812656b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01eafccb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555c8107afa0 con 0x555c810d7e40 2024-05-27T07:58:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01eafccb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555c810dcda0 con 0x555c8125b7d0 2024-05-27T07:58:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01eafccb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555c81267f60 con 0x555c8125ffb0 2024-05-27T07:58:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01eafccb80 10 client.0 _ll_register_callbacks cb 0x555c8110b5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01da353700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c81264c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01da353700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c81264c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45908/0 (socket says 172.21.15.114:45908) 2024-05-27T07:58:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01da353700 1 -- 172.21.15.114:0/250902386 learned_addr learned my addr 172.21.15.114:0/250902386 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01dab54700 1 --2- 172.21.15.114:0/250902386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 0x555c812656b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01d9b52700 1 --2- 172.21.15.114:0/250902386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c81265170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01da353700 1 -- 172.21.15.114:0/250902386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 msgr2=0x555c812656b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01da353700 1 --2- 172.21.15.114:0/250902386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 0x555c812656b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01da353700 1 -- 172.21.15.114:0/250902386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 msgr2=0x555c81265170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01da353700 1 --2- 172.21.15.114:0/250902386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c81265170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.943+0000 7f01da353700 1 -- 172.21.15.114:0/250902386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c810c48d0 con 0x555c810d7e40 2024-05-27T07:58:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01da353700 1 --2- 172.21.15.114:0/250902386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c81264c30 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009fd0 tx=0x7f01c4009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73f251ce841ee40 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01b9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:58:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01b9ffb700 1 -- 172.21.15.114:0/250902386 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c4028670 con 0x555c810d7e40 2024-05-27T07:58:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01b9ffb700 1 -- 172.21.15.114:0/250902386 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 9466+0+0 (secure 0 0 0) 0x7f01c4028810 con 0x555c810d7e40 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01eafccb80 1 -- 172.21.15.114:0/250902386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01c402d070 con 0x555c810d7e40 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01b9ffb700 1 -- 172.21.15.114:0/250902386 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f01c4028a10 con 0x555c810d7e40 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01eafccb80 10 client.19680 fetch_fsmap learned FSMap version 108 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01eafccb80 1 -- 172.21.15.114:0/250902386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x555c8112e760 con 0x555c810d7e40 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01b9ffb700 1 -- 172.21.15.114:0/250902386 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01c402d070 con 0x555c810d7e40 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01b9ffb700 1 -- 172.21.15.114:0/250902386 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 108) ==== 4095+0+0 (secure 0 0 0) 0x7f01c4011070 con 0x555c810d7e40 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01eafccb80 10 client.19680 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01eafccb80 10 client.19680 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01eafccb80 20 client.19680 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01eafccb80 1 --2- 172.21.15.114:0/250902386 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555c81274d90 0x555c81277210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01eafccb80 4 client.19680 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.944+0000 7f01eafccb80 1 -- 172.21.15.114:0/250902386 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c81186cc0 con 0x555c81274d90 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.945+0000 7f01d9b52700 1 --2- 172.21.15.114:0/250902386 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555c81274d90 0x555c81277210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.945+0000 7f01d9b52700 1 --2- 172.21.15.114:0/250902386 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555c81274d90 0x555c81277210 crc :-1 s=READY pgs=502 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a633ae037cfecf2f server_cookie=5673b5fab3936ad5 in_seq=0 out_seq=0 2024-05-27T07:58:45.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.945+0000 7f01b9ffb700 10 client.19680 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:45.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.947+0000 7f01b9ffb700 1 -- 172.21.15.114:0/250902386 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555c81186cc0 con 0x555c81274d90 2024-05-27T07:58:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.947+0000 7f01b9ffb700 10 client.19680 handle_command_reply: tid=0 2024-05-27T07:58:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.947+0000 7f01eafccb80 1 client.19680 shutdown 2024-05-27T07:58:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.947+0000 7f01eafccb80 1 -- 172.21.15.114:0/250902386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 msgr2=0x555c81264c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.947+0000 7f01eafccb80 1 --2- 172.21.15.114:0/250902386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c81264c30 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009fd0 tx=0x7f01c4009ac0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.948+0000 7f01eafccb80 1 -- 172.21.15.114:0/250902386 shutdown_connections 2024-05-27T07:58:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.948+0000 7f01eafccb80 1 --2- 172.21.15.114:0/250902386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 0x555c812656b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.948+0000 7f01eafccb80 1 --2- 172.21.15.114:0/250902386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c81264c30 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.948+0000 7f01eafccb80 1 --2- 172.21.15.114:0/250902386 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555c81274d90 0x555c81277210 crc :-1 s=READY pgs=502 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.948+0000 7f01eafccb80 1 --2- 172.21.15.114:0/250902386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c81265170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.948+0000 7f01eafccb80 1 -- 172.21.15.114:0/250902386 >> 172.21.15.114:0/250902386 conn(0x555c810da770 msgr2=0x555c81255bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.948+0000 7f01b9ffb700 0 client.19680 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.948+0000 7f01eafccb80 1 -- 172.21.15.114:0/250902386 shutdown_connections 2024-05-27T07:58:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.948+0000 7f01eafccb80 1 -- 172.21.15.114:0/250902386 wait complete. 2024-05-27T07:58:45.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.948+0000 7f01eafccb80 20 client.19680 trim_cache size 0 max 16384 2024-05-27T07:58:45.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.974+0000 7f01eafccb80 1 Processor -- start 2024-05-27T07:58:45.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01eafccb80 1 -- start start 2024-05-27T07:58:45.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01eafccb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c81256600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01eafccb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c81270fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01eafccb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 0x555c8127ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01eafccb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555c8125f4a0 con 0x555c810d7e40 2024-05-27T07:58:45.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01eafccb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555c81277630 con 0x555c8125b7d0 2024-05-27T07:58:45.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01eafccb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f01d41adf20 con 0x555c8125ffb0 2024-05-27T07:58:45.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01da353700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c81256600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01da353700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c81256600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45920/0 (socket says 172.21.15.114:45920) 2024-05-27T07:58:45.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01da353700 1 -- 172.21.15.114:0/3654634818 learned_addr learned my addr 172.21.15.114:0/3654634818 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:45.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01d9b52700 1 --2- 172.21.15.114:0/3654634818 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c81270fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01dab54700 1 --2- 172.21.15.114:0/3654634818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 0x555c8127ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01d9b52700 1 -- 172.21.15.114:0/3654634818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 msgr2=0x555c8127ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:45.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01d9b52700 1 --2- 172.21.15.114:0/3654634818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 0x555c8127ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01d9b52700 1 -- 172.21.15.114:0/3654634818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 msgr2=0x555c81256600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:45.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01d9b52700 1 --2- 172.21.15.114:0/3654634818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c81256600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01d9b52700 1 -- 172.21.15.114:0/3654634818 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c812777b0 con 0x555c8125b7d0 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.975+0000 7f01da353700 1 --2- 172.21.15.114:0/3654634818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c81256600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.976+0000 7f01d9b52700 1 --2- 172.21.15.114:0/3654634818 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c81270fc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x555c8124e790 tx=0x7f01c8009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0ea79d50021e8d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.976+0000 7f01bbfff700 1 -- 172.21.15.114:0/3654634818 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c8004830 con 0x555c8125b7d0 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.976+0000 7f01bbfff700 1 -- 172.21.15.114:0/3654634818 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01c8005020 con 0x555c8125b7d0 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.976+0000 7f01bbfff700 1 -- 172.21.15.114:0/3654634818 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c801e740 con 0x555c8125b7d0 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.977+0000 7f01eafccb80 1 -- 172.21.15.114:0/3654634818 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 msgr2=0x555c81270fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.977+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3654634818 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c81270fc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x555c8124e790 tx=0x7f01c8009310 comp rx=0 tx=0).stop 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.977+0000 7f01eafccb80 1 -- 172.21.15.114:0/3654634818 shutdown_connections 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.977+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3654634818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 0x555c8127ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.977+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3654634818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c81256600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.977+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3654634818 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c81270fc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.977+0000 7f01eafccb80 1 -- 172.21.15.114:0/3654634818 >> 172.21.15.114:0/3654634818 conn(0x555c810da770 msgr2=0x555c81263b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:45.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.977+0000 7f01eafccb80 1 -- 172.21.15.114:0/3654634818 shutdown_connections 2024-05-27T07:58:45.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.977+0000 7f01eafccb80 1 -- 172.21.15.114:0/3654634818 wait complete. 2024-05-27T07:58:45.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.977+0000 7f01eafccb80 1 -- start start 2024-05-27T07:58:45.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.978+0000 7f01eafccb80 1 Processor -- start 2024-05-27T07:58:45.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.978+0000 7f01eafccb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c8126d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.978+0000 7f01eafccb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c8126ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.978+0000 7f01eafccb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 0x555c8126b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.978+0000 7f01eafccb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f01d41acea0 con 0x555c810d7e40 2024-05-27T07:58:45.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.978+0000 7f01eafccb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f01d41ae5f0 con 0x555c8125b7d0 2024-05-27T07:58:45.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.978+0000 7f01eafccb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f01d41ac160 con 0x555c8125ffb0 2024-05-27T07:58:45.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.978+0000 7f01eafccb80 10 client.0 _ll_register_callbacks cb 0x555c8110b5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:45.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.978+0000 7f01da353700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c8126d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.978+0000 7f01d9b52700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c8126ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.978+0000 7f01d9b52700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c8126ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51160/0 (socket says 172.21.15.114:51160) 2024-05-27T07:58:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.978+0000 7f01d9b52700 1 -- 172.21.15.114:0/3969447590 learned_addr learned my addr 172.21.15.114:0/3969447590 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.978+0000 7f01dab54700 1 --2- 172.21.15.114:0/3969447590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 0x555c8126b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.979+0000 7f01d9b52700 1 -- 172.21.15.114:0/3969447590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 msgr2=0x555c8126b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.979+0000 7f01d9b52700 1 --2- 172.21.15.114:0/3969447590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 0x555c8126b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.979+0000 7f01d9b52700 1 -- 172.21.15.114:0/3969447590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 msgr2=0x555c8126d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.979+0000 7f01d9b52700 1 --2- 172.21.15.114:0/3969447590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c8126d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.979+0000 7f01d9b52700 1 -- 172.21.15.114:0/3969447590 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c8126b740 con 0x555c8125b7d0 2024-05-27T07:58:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.979+0000 7f01d9b52700 1 --2- 172.21.15.114:0/3969447590 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c8126ab60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f01c8000c00 tx=0x7f01c8009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d43e4333368be3f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.979+0000 7f01b9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:58:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.979+0000 7f01b9ffb700 1 -- 172.21.15.114:0/3969447590 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c8005350 con 0x555c8125b7d0 2024-05-27T07:58:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.979+0000 7f01b9ffb700 1 -- 172.21.15.114:0/3969447590 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 9466+0+0 (secure 0 0 0) 0x7f01c80047c0 con 0x555c8125b7d0 2024-05-27T07:58:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.979+0000 7f01eafccb80 1 -- 172.21.15.114:0/3969447590 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01c8005020 con 0x555c8125b7d0 2024-05-27T07:58:45.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.979+0000 7f01b9ffb700 1 -- 172.21.15.114:0/3969447590 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f01c80054f0 con 0x555c8125b7d0 2024-05-27T07:58:45.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.979+0000 7f01eafccb80 10 client.29012 fetch_fsmap learned FSMap version 108 2024-05-27T07:58:45.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.979+0000 7f01eafccb80 1 -- 172.21.15.114:0/3969447590 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x555c812df730 con 0x555c8125b7d0 2024-05-27T07:58:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.980+0000 7f01b9ffb700 1 -- 172.21.15.114:0/3969447590 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01c8005020 con 0x555c8125b7d0 2024-05-27T07:58:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.980+0000 7f01b9ffb700 1 -- 172.21.15.114:0/3969447590 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 108) ==== 4095+0+0 (secure 0 0 0) 0x7f01c801ec50 con 0x555c8125b7d0 2024-05-27T07:58:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.980+0000 7f01eafccb80 10 client.29012 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:58:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.980+0000 7f01eafccb80 10 client.29012 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.980+0000 7f01eafccb80 20 client.29012 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.980+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3969447590 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555c812dfdb0 0x555c812e21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.980+0000 7f01eafccb80 4 client.29012 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:58:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.980+0000 7f01eafccb80 1 -- 172.21.15.114:0/3969447590 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555c811804a0 con 0x555c812dfdb0 2024-05-27T07:58:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.980+0000 7f01da353700 1 --2- 172.21.15.114:0/3969447590 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555c812dfdb0 0x555c812e21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.981+0000 7f01b9ffb700 10 client.29012 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.981+0000 7f01da353700 1 --2- 172.21.15.114:0/3969447590 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555c812dfdb0 0x555c812e21d0 crc :-1 s=READY pgs=503 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ff6ee48b951b58c server_cookie=7f89161b83f28329 in_seq=0 out_seq=0 2024-05-27T07:58:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01b9ffb700 1 -- 172.21.15.114:0/3969447590 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555c811804a0 con 0x555c812dfdb0 2024-05-27T07:58:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01b9ffb700 10 client.29012 handle_command_reply: tid=0 2024-05-27T07:58:45.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01eafccb80 1 client.29012 shutdown 2024-05-27T07:58:45.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01eafccb80 1 -- 172.21.15.114:0/3969447590 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 msgr2=0x555c8126ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:45.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3969447590 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c8126ab60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f01c8000c00 tx=0x7f01c8009ba0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01eafccb80 1 -- 172.21.15.114:0/3969447590 shutdown_connections 2024-05-27T07:58:45.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3969447590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c8125ffb0 0x555c8126b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3969447590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555c810d7e40 0x555c8126d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3969447590 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x555c812dfdb0 0x555c812e21d0 crc :-1 s=READY pgs=503 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01eafccb80 1 --2- 172.21.15.114:0/3969447590 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c8125b7d0 0x555c8126ab60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01b9ffb700 0 client.29012 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:45.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01eafccb80 1 -- 172.21.15.114:0/3969447590 >> 172.21.15.114:0/3969447590 conn(0x555c810da770 msgr2=0x555c81256360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:45.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01eafccb80 1 -- 172.21.15.114:0/3969447590 shutdown_connections 2024-05-27T07:58:45.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01eafccb80 1 -- 172.21.15.114:0/3969447590 wait complete. 2024-05-27T07:58:45.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.985+0000 7f01eafccb80 20 client.29012 trim_cache size 0 max 16384 2024-05-27T07:58:45.990 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:58:45.990 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:58:45.990 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T07:58:45.990 INFO:teuthology.orchestra.run.smithi114.stdout: "f7462d28-89c3-42b5-8438-f21f57975515": { 2024-05-27T07:58:45.990 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T07:58:45.990 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "f7462d28-89c3-42b5-8438-f21f57975515", 2024-05-27T07:58:45.990 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T07:58:45.990 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:58:45.990 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T07:58:45.990 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:58:45.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.986+0000 7f01dc5b7700 1 -- 172.21.15.114:0/2731705825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f01bc0b29f0 msgr2=0x7f01bc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:45.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.986+0000 7f01dc5b7700 1 --2- 172.21.15.114:0/2731705825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f01bc0b29f0 0x7f01bc0b4f00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f01c400b6c0 tx=0x7f01c40096c0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.986+0000 7f01dc5b7700 1 -- 172.21.15.114:0/2731705825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d41089e0 msgr2=0x7f01d419f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:45.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.986+0000 7f01dc5b7700 1 --2- 172.21.15.114:0/2731705825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d41089e0 0x7f01d419f750 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00eb20 tx=0x7f01cc00ee30 comp rx=0 tx=0).stop 2024-05-27T07:58:45.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.986+0000 7f01dc5b7700 1 -- 172.21.15.114:0/2731705825 shutdown_connections 2024-05-27T07:58:45.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.986+0000 7f01dc5b7700 1 --2- 172.21.15.114:0/2731705825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d41089e0 0x7f01d419f750 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.986+0000 7f01dc5b7700 1 --2- 172.21.15.114:0/2731705825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01d41029b0 0x7f01d419af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.986+0000 7f01dc5b7700 1 --2- 172.21.15.114:0/2731705825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f01bc0b29f0 0x7f01bc0b4f00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.986+0000 7f01dc5b7700 1 --2- 172.21.15.114:0/2731705825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01d4103300 0x7f01d419b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:45.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.986+0000 7f01dc5b7700 1 -- 172.21.15.114:0/2731705825 >> 172.21.15.114:0/2731705825 conn(0x7f01d40fe510 msgr2=0x7f01d4106b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:45.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.986+0000 7f01dc5b7700 1 -- 172.21.15.114:0/2731705825 shutdown_connections 2024-05-27T07:58:45.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:58:45.986+0000 7f01dc5b7700 1 -- 172.21.15.114:0/2731705825 wait complete. 2024-05-27T07:58:46.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 71628 12.44 MB/sec execute 178 sec latency 172.637 ms 2024-05-27T07:58:46.848 INFO:tasks.cephfs.filesystem:scrub status for tag:f7462d28-89c3-42b5-8438-f21f57975515 - {'path': '/', 'tag': 'f7462d28-89c3-42b5-8438-f21f57975515', 'options': 'recursive,force'} 2024-05-27T07:58:47.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:46 smithi114 ceph-mon[32038]: pgmap v656: 129 pgs: 129 active+clean; 397 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 375 op/s 2024-05-27T07:58:47.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:46 smithi163 ceph-mon[36451]: pgmap v656: 129 pgs: 129 active+clean; 397 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 375 op/s 2024-05-27T07:58:47.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:46 smithi186 ceph-mon[36647]: pgmap v656: 129 pgs: 129 active+clean; 397 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 375 op/s 2024-05-27T07:58:47.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 72002 12.51 MB/sec execute 179 sec latency 187.988 ms 2024-05-27T07:58:48.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:47 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:58:48.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:47 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:58:48.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:47 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:58:48.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 72433 12.57 MB/sec execute 180 sec latency 173.849 ms 2024-05-27T07:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.870+0000 7fbe14338640 1 -- 172.21.15.114:0/5682896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe0c109050 msgr2=0x7fbe0c102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.870+0000 7fbe14338640 1 --2- 172.21.15.114:0/5682896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe0c109050 0x7fbe0c102c90 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009900 tx=0x7fbdfc02a410 comp rx=0 tx=0).stop 2024-05-27T07:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.870+0000 7fbe14338640 1 -- 172.21.15.114:0/5682896 shutdown_connections 2024-05-27T07:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.870+0000 7fbe14338640 1 --2- 172.21.15.114:0/5682896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe0c103300 0x7fbe0c10f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.870+0000 7fbe14338640 1 --2- 172.21.15.114:0/5682896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe0c109050 0x7fbe0c102c90 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.870+0000 7fbe14338640 1 --2- 172.21.15.114:0/5682896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe0c108bb0 0x7fbe0c102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.870+0000 7fbe14338640 1 -- 172.21.15.114:0/5682896 >> 172.21.15.114:0/5682896 conn(0x7fbe0c0781a0 msgr2=0x7fbe0c078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.870+0000 7fbe14338640 1 -- 172.21.15.114:0/5682896 shutdown_connections 2024-05-27T07:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.870+0000 7fbe14338640 1 -- 172.21.15.114:0/5682896 wait complete. 2024-05-27T07:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.871+0000 7fbe14338640 1 Processor -- start 2024-05-27T07:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.871+0000 7fbe14338640 1 -- start start 2024-05-27T07:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe14338640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe0c103300 0x7fbe0c19b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe14338640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe0c108bb0 0x7fbe0c19bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe14338640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe0c109050 0x7fbe0c19c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe14338640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c112da0 con 0x7fbe0c109050 2024-05-27T07:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe14338640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c112c20 con 0x7fbe0c108bb0 2024-05-27T07:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe14338640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c112f20 con 0x7fbe0c103300 2024-05-27T07:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe128d5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe0c109050 0x7fbe0c19c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe128d5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe0c109050 0x7fbe0c19c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45964/0 (socket says 172.21.15.114:45964) 2024-05-27T07:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe120d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe0c103300 0x7fbe0c19b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe128d5640 1 -- 172.21.15.114:0/2044125741 learned_addr learned my addr 172.21.15.114:0/2044125741 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe118d3640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe0c108bb0 0x7fbe0c19bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe120d4640 1 -- 172.21.15.114:0/2044125741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe0c108bb0 msgr2=0x7fbe0c19bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe120d4640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe0c108bb0 0x7fbe0c19bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe120d4640 1 -- 172.21.15.114:0/2044125741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe0c109050 msgr2=0x7fbe0c19c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe120d4640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe0c109050 0x7fbe0c19c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe120d4640 1 -- 172.21.15.114:0/2044125741 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0c1a6ac0 con 0x7fbe0c103300 2024-05-27T07:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe128d5640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe0c109050 0x7fbe0c19c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.872+0000 7fbe120d4640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe0c103300 0x7fbe0c19b780 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbe0800e9c0 tx=0x7fbe0800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.873+0000 7fbe037fe640 1 -- 172.21.15.114:0/2044125741 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0800ccc0 con 0x7fbe0c103300 2024-05-27T07:58:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.873+0000 7fbe14338640 1 -- 172.21.15.114:0/2044125741 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0c1a6db0 con 0x7fbe0c103300 2024-05-27T07:58:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.873+0000 7fbe037fe640 1 -- 172.21.15.114:0/2044125741 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe080108e0 con 0x7fbe0c103300 2024-05-27T07:58:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.873+0000 7fbe037fe640 1 -- 172.21.15.114:0/2044125741 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080187a0 con 0x7fbe0c103300 2024-05-27T07:58:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.873+0000 7fbe14338640 1 -- 172.21.15.114:0/2044125741 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe0c1a7270 con 0x7fbe0c103300 2024-05-27T07:58:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe037fe640 1 -- 172.21.15.114:0/2044125741 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fbe08018940 con 0x7fbe0c103300 2024-05-27T07:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe1b1ef740 1 Processor -- start 2024-05-27T07:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe1b1ef740 1 -- start start 2024-05-27T07:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe1b1ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280f091b0 0x5622812ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe1b1ef740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee02f0 0x5622812aed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe1b1ef740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee84d0 0x5622812af260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe1b1ef740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56228121cc30 con 0x562280ee02f0 2024-05-27T07:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe1b1ef740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56228120c7d0 con 0x562280ee84d0 2024-05-27T07:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe1b1ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562281210810 con 0x562280f091b0 2024-05-27T07:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe120d4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee02f0 0x5622812aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe118d3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee84d0 0x5622812af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe128d5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280f091b0 0x5622812ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe120d4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee02f0 0x5622812aed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45974/0 (socket says 172.21.15.114:45974) 2024-05-27T07:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe128d5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280f091b0 0x5622812ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36644/0 (socket says 172.21.15.114:36644) 2024-05-27T07:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe118d3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee84d0 0x5622812af260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51190/0 (socket says 172.21.15.114:51190) 2024-05-27T07:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.876+0000 7fbe120d4640 1 -- 172.21.15.114:0/287302983 learned_addr learned my addr 172.21.15.114:0/287302983 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.877+0000 7fbe037fe640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbdf80b2950 0x7fbdf80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.877+0000 7fbe118d3640 1 -- 172.21.15.114:0/287302983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280f091b0 msgr2=0x5622812ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.877+0000 7fbe118d3640 1 --2- 172.21.15.114:0/287302983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280f091b0 0x5622812ae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.877+0000 7fbe118d3640 1 -- 172.21.15.114:0/287302983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee02f0 msgr2=0x5622812aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.877+0000 7fbe118d3640 1 --2- 172.21.15.114:0/287302983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee02f0 0x5622812aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.877+0000 7fbe118d3640 1 -- 172.21.15.114:0/287302983 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562280f21210 con 0x562280ee84d0 2024-05-27T07:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.877+0000 7fbe128d5640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbdf80b2950 0x7fbdf80b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.877+0000 7fbe037fe640 1 -- 172.21.15.114:0/2044125741 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 9466+0+0 (secure 0 0 0) 0x7fbe080e2cd0 con 0x7fbe0c103300 2024-05-27T07:58:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.877+0000 7fbe128d5640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbdf80b2950 0x7fbdf80b4e60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x5622812ab950 tx=0x7fbe04004a40 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.877+0000 7fbe118d3640 1 --2- 172.21.15.114:0/287302983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee84d0 0x5622812af260 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc02a930 tx=0x7fbdfc000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7169d232ef289852 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.878+0000 7fbe00bfe640 1 -- 172.21.15.114:0/287302983 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc040070 con 0x562280ee84d0 2024-05-27T07:58:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.878+0000 7fbe00bfe640 1 -- 172.21.15.114:0/287302983 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdfc004030 con 0x562280ee84d0 2024-05-27T07:58:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.879+0000 7fbe00bfe640 1 -- 172.21.15.114:0/287302983 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc035040 con 0x562280ee84d0 2024-05-27T07:58:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.879+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/287302983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee84d0 msgr2=0x5622812af260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.879+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/287302983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee84d0 0x5622812af260 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc02a930 tx=0x7fbdfc000bc0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.879+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/287302983 shutdown_connections 2024-05-27T07:58:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.879+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/287302983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee84d0 0x5622812af260 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.879+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/287302983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee02f0 0x5622812aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.879+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/287302983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280f091b0 0x5622812ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.879+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/287302983 >> 172.21.15.114:0/287302983 conn(0x562280efd300 msgr2=0x5622812aaf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.879+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/287302983 shutdown_connections 2024-05-27T07:58:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.879+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/287302983 wait complete. 2024-05-27T07:58:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.879+0000 7fbe1b1ef740 1 -- start start 2024-05-27T07:58:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.879+0000 7fbe1b1ef740 1 Processor -- start 2024-05-27T07:58:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe1b1ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee02f0 0x5622812b8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe1b1ef740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee84d0 0x5622812b93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe1b1ef740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280f091b0 0x5622812becc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe1b1ef740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562280ef2790 con 0x562280ee84d0 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe1b1ef740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5622811c8890 con 0x562280f091b0 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe1b1ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5622812bb900 con 0x562280ee02f0 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe118d3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee84d0 0x5622812b93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe1b1ef740 10 client.0 _ll_register_callbacks cb 0x562280f20a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe120d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee02f0 0x5622812b8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe118d3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee84d0 0x5622812b93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45984/0 (socket says 172.21.15.114:45984) 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe118d3640 1 -- 172.21.15.114:0/508655680 learned_addr learned my addr 172.21.15.114:0/508655680 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe128d5640 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280f091b0 0x5622812becc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe120d4640 1 -- 172.21.15.114:0/508655680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280f091b0 msgr2=0x5622812becc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe120d4640 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280f091b0 0x5622812becc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe120d4640 1 -- 172.21.15.114:0/508655680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee84d0 msgr2=0x5622812b93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe120d4640 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee84d0 0x5622812b93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe120d4640 1 -- 172.21.15.114:0/508655680 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562280fed620 con 0x562280ee02f0 2024-05-27T07:58:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe128d5640 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280f091b0 0x5622812becc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe118d3640 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee84d0 0x5622812b93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.880+0000 7fbe120d4640 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee02f0 0x5622812b8e90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbe080e0670 tx=0x7fbe08009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b323c565965cc4c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbdf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:58:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbdf5ffb640 1 -- 172.21.15.114:0/508655680 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080a9420 con 0x562280ee02f0 2024-05-27T07:58:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/508655680 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdfc004030 con 0x562280ee02f0 2024-05-27T07:58:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbdf5ffb640 1 -- 172.21.15.114:0/508655680 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 9466+0+0 (secure 0 0 0) 0x7fbe080a62a0 con 0x562280ee02f0 2024-05-27T07:58:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbdf5ffb640 1 -- 172.21.15.114:0/508655680 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fbe080a5440 con 0x562280ee02f0 2024-05-27T07:58:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbe1b1ef740 10 client.29170 fetch_fsmap learned FSMap version 108 2024-05-27T07:58:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/508655680 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5622812c7ca0 con 0x562280ee02f0 2024-05-27T07:58:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbdf5ffb640 1 -- 172.21.15.114:0/508655680 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdfc004030 con 0x562280ee02f0 2024-05-27T07:58:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbdf5ffb640 1 -- 172.21.15.114:0/508655680 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 108) ==== 4095+0+0 (secure 0 0 0) 0x7fbe080a8f10 con 0x562280ee02f0 2024-05-27T07:58:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbe1b1ef740 10 client.29170 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:58:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbe1b1ef740 10 client.29170 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbe1b1ef740 20 client.29170 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562280ec7d80 0x5622812cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbe1b1ef740 4 client.29170 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:58:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.881+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/508655680 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56228121eb80 con 0x562280ec7d80 2024-05-27T07:58:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.882+0000 7fbe118d3640 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562280ec7d80 0x5622812cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.882+0000 7fbdf5ffb640 10 client.29170 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.882+0000 7fbe118d3640 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562280ec7d80 0x5622812cb5f0 crc :-1 s=READY pgs=504 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7c401902b78d315 server_cookie=563b90a5683ddf64 in_seq=0 out_seq=0 2024-05-27T07:58:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbdf5ffb640 1 -- 172.21.15.114:0/508655680 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56228121eb80 con 0x562280ec7d80 2024-05-27T07:58:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbdf5ffb640 10 client.29170 handle_command_reply: tid=0 2024-05-27T07:58:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbe1b1ef740 1 client.29170 shutdown 2024-05-27T07:58:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/508655680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee02f0 msgr2=0x5622812b8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee02f0 0x5622812b8e90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbe080e0670 tx=0x7fbe08009700 comp rx=0 tx=0).stop 2024-05-27T07:58:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/508655680 shutdown_connections 2024-05-27T07:58:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562280ec7d80 0x5622812cb5f0 crc :-1 s=READY pgs=504 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280f091b0 0x5622812becc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee84d0 0x5622812b93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/508655680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee02f0 0x5622812b8e90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/508655680 >> 172.21.15.114:0/508655680 conn(0x562280efd300 msgr2=0x5622812b0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbdf5ffb640 0 client.29170 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/508655680 shutdown_connections 2024-05-27T07:58:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/508655680 wait complete. 2024-05-27T07:58:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.886+0000 7fbe1b1ef740 20 client.29170 trim_cache size 0 max 16384 2024-05-27T07:58:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.908+0000 7fbe1b1ef740 1 Processor -- start 2024-05-27T07:58:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.908+0000 7fbe1b1ef740 1 -- start start 2024-05-27T07:58:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.908+0000 7fbe1b1ef740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee02f0 0x5622812af580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.908+0000 7fbe1b1ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee84d0 0x5622812b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.908+0000 7fbe1b1ef740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280f091b0 0x5622812b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.908+0000 7fbe1b1ef740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c1aedd0 con 0x562280ee02f0 2024-05-27T07:58:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.908+0000 7fbe1b1ef740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c1af4a0 con 0x562280f091b0 2024-05-27T07:58:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.908+0000 7fbe1b1ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c1add70 con 0x562280ee84d0 2024-05-27T07:58:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.908+0000 7fbe120d4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee02f0 0x5622812af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.908+0000 7fbe118d3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee84d0 0x5622812b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.908+0000 7fbe120d4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee02f0 0x5622812af580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45990/0 (socket says 172.21.15.114:45990) 2024-05-27T07:58:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.908+0000 7fbe120d4640 1 -- 172.21.15.114:0/894814556 learned_addr learned my addr 172.21.15.114:0/894814556 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.908+0000 7fbe128d5640 1 --2- 172.21.15.114:0/894814556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280f091b0 0x5622812b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.909+0000 7fbe118d3640 1 -- 172.21.15.114:0/894814556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280f091b0 msgr2=0x5622812b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.909+0000 7fbe118d3640 1 --2- 172.21.15.114:0/894814556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280f091b0 0x5622812b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.909+0000 7fbe118d3640 1 -- 172.21.15.114:0/894814556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee02f0 msgr2=0x5622812af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.909+0000 7fbe118d3640 1 --2- 172.21.15.114:0/894814556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee02f0 0x5622812af580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.909+0000 7fbe118d3640 1 -- 172.21.15.114:0/894814556 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562280ec7fb0 con 0x562280ee84d0 2024-05-27T07:58:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.909+0000 7fbe120d4640 1 --2- 172.21.15.114:0/894814556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee02f0 0x5622812af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:58:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.909+0000 7fbe118d3640 1 --2- 172.21.15.114:0/894814556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee84d0 0x5622812b2560 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x5622812cbd30 tx=0x7fbdfc005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d39fa1b7dc0badb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:48.913 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.909+0000 7fbdf7fff640 1 -- 172.21.15.114:0/894814556 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc035040 con 0x562280ee84d0 2024-05-27T07:58:48.913 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.909+0000 7fbdf7fff640 1 -- 172.21.15.114:0/894814556 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdfc055070 con 0x562280ee84d0 2024-05-27T07:58:48.913 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.909+0000 7fbdf7fff640 1 -- 172.21.15.114:0/894814556 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc040040 con 0x562280ee84d0 2024-05-27T07:58:48.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.910+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/894814556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee84d0 msgr2=0x5622812b2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:48.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.910+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/894814556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee84d0 0x5622812b2560 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x5622812cbd30 tx=0x7fbdfc005fd0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.910+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/894814556 shutdown_connections 2024-05-27T07:58:48.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.910+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/894814556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280f091b0 0x5622812b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.910+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/894814556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee84d0 0x5622812b2560 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.910+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/894814556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280ee02f0 0x5622812af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.910+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/894814556 >> 172.21.15.114:0/894814556 conn(0x562280efd300 msgr2=0x5622812a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:48.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.911+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/894814556 shutdown_connections 2024-05-27T07:58:48.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.911+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/894814556 wait complete. 2024-05-27T07:58:48.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.911+0000 7fbe1b1ef740 1 -- start start 2024-05-27T07:58:48.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.911+0000 7fbe1b1ef740 1 Processor -- start 2024-05-27T07:58:48.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.911+0000 7fbe1b1ef740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee02f0 0x5622812bf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe1b1ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee84d0 0x5622812bdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe1b1ef740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280f091b0 0x5622812be0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe120d4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee02f0 0x5622812bf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe118d3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee84d0 0x5622812bdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe120d4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee02f0 0x5622812bf0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51220/0 (socket says 172.21.15.114:51220) 2024-05-27T07:58:48.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe118d3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee84d0 0x5622812bdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:36668/0 (socket says 172.21.15.114:36668) 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe128d5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280f091b0 0x5622812be0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe128d5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280f091b0 0x5622812be0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45994/0 (socket says 172.21.15.114:45994) 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe120d4640 1 -- 172.21.15.114:0/3488307238 learned_addr learned my addr 172.21.15.114:0/3488307238 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/3488307238 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c1ad6a0 con 0x562280f091b0 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/3488307238 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c1ad030 con 0x562280ee02f0 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/3488307238 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5622812ab980 con 0x562280ee84d0 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe1b1ef740 10 client.0 _ll_register_callbacks cb 0x7fbe080a5b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe120d4640 1 -- 172.21.15.114:0/3488307238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee84d0 msgr2=0x5622812bdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe120d4640 1 --2- 172.21.15.114:0/3488307238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee84d0 0x5622812bdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe120d4640 1 -- 172.21.15.114:0/3488307238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280f091b0 msgr2=0x5622812be0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe120d4640 1 --2- 172.21.15.114:0/3488307238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280f091b0 0x5622812be0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.912+0000 7fbe120d4640 1 -- 172.21.15.114:0/3488307238 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622812be630 con 0x562280ee02f0 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.913+0000 7fbe120d4640 1 --2- 172.21.15.114:0/3488307238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee02f0 0x5622812bf0a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbe080e0670 tx=0x7fbe0801f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87b82be779b67e2d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:58:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.913+0000 7fbdf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:58:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.913+0000 7fbdf5ffb640 1 -- 172.21.15.114:0/3488307238 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080c42d0 con 0x562280ee02f0 2024-05-27T07:58:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.913+0000 7fbdf5ffb640 1 -- 172.21.15.114:0/3488307238 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 9466+0+0 (secure 0 0 0) 0x7fbe08091040 con 0x562280ee02f0 2024-05-27T07:58:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.913+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/3488307238 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdfc055070 con 0x562280ee02f0 2024-05-27T07:58:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.918+0000 7fbdf5ffb640 1 -- 172.21.15.114:0/3488307238 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fbe080a97c0 con 0x562280ee02f0 2024-05-27T07:58:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.918+0000 7fbe1b1ef740 10 client.29030 fetch_fsmap learned FSMap version 108 2024-05-27T07:58:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.918+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/3488307238 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5622812c4830 con 0x562280ee02f0 2024-05-27T07:58:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.918+0000 7fbdf5ffb640 1 -- 172.21.15.114:0/3488307238 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdfc055070 con 0x562280ee02f0 2024-05-27T07:58:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.918+0000 7fbdf5ffb640 1 -- 172.21.15.114:0/3488307238 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 108) ==== 4095+0+0 (secure 0 0 0) 0x7fbe080af140 con 0x562280ee02f0 2024-05-27T07:58:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.918+0000 7fbe1b1ef740 10 client.29030 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:58:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.918+0000 7fbe1b1ef740 10 client.29030 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:58:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.918+0000 7fbe1b1ef740 20 client.29030 populate_metadata read hostname 'smithi114' 2024-05-27T07:58:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.919+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/3488307238 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562280e9d1c0 0x5622812c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:58:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.919+0000 7fbe1b1ef740 4 client.29030 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:58:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.919+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/3488307238 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562281218360 con 0x562280e9d1c0 2024-05-27T07:58:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.919+0000 7fbe118d3640 1 --2- 172.21.15.114:0/3488307238 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562280e9d1c0 0x5622812c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:58:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.919+0000 7fbdf5ffb640 10 client.29030 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:48.919+0000 7fbe118d3640 1 --2- 172.21.15.114:0/3488307238 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562280e9d1c0 0x5622812c4fc0 crc :-1 s=READY pgs=505 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53f00d079a0ff75d server_cookie=9693120fc8c3ce9f in_seq=0 out_seq=0 2024-05-27T07:58:49.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:48 smithi163 ceph-mon[36451]: pgmap v657: 129 pgs: 129 active+clean; 400 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.5 MiB/s wr, 282 op/s 2024-05-27T07:58:49.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:48 smithi186 ceph-mon[36647]: pgmap v657: 129 pgs: 129 active+clean; 400 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.5 MiB/s wr, 282 op/s 2024-05-27T07:58:49.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:48 smithi114 ceph-mon[32038]: pgmap v657: 129 pgs: 129 active+clean; 400 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.5 MiB/s wr, 282 op/s 2024-05-27T07:58:49.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 72600 12.54 MB/sec execute 181 sec latency 591.699 ms 2024-05-27T07:58:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.430+0000 7fbdf5ffb640 1 -- 172.21.15.114:0/3488307238 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562281218360 con 0x562280e9d1c0 2024-05-27T07:58:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.430+0000 7fbdf5ffb640 10 client.29030 handle_command_reply: tid=0 2024-05-27T07:58:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.430+0000 7fbe1b1ef740 1 client.29030 shutdown 2024-05-27T07:58:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.431+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/3488307238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee02f0 msgr2=0x5622812bf0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.431+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/3488307238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee02f0 0x5622812bf0a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbe080e0670 tx=0x7fbe0801f000 comp rx=0 tx=0).stop 2024-05-27T07:58:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.431+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/3488307238 shutdown_connections 2024-05-27T07:58:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.431+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/3488307238 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x562280e9d1c0 0x5622812c4fc0 crc :-1 s=READY pgs=505 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.431+0000 7fbdf5ffb640 0 client.29030 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:58:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.431+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/3488307238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562280f091b0 0x5622812be0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.431+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/3488307238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562280ee84d0 0x5622812bdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.431+0000 7fbe1b1ef740 1 --2- 172.21.15.114:0/3488307238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562280ee02f0 0x5622812bf0a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.431+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/3488307238 >> 172.21.15.114:0/3488307238 conn(0x562280efd300 msgr2=0x5622812b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.431+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/3488307238 shutdown_connections 2024-05-27T07:58:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.431+0000 7fbe1b1ef740 1 -- 172.21.15.114:0/3488307238 wait complete. 2024-05-27T07:58:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.431+0000 7fbe1b1ef740 20 client.29030 trim_cache size 0 max 16384 2024-05-27T07:58:50.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:58:50.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:58:50.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:58:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.432+0000 7fbe14338640 1 -- 172.21.15.114:0/2044125741 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbdf80b2950 msgr2=0x7fbdf80b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.432+0000 7fbe14338640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbdf80b2950 0x7fbdf80b4e60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x5622812ab950 tx=0x7fbe04004a40 comp rx=0 tx=0).stop 2024-05-27T07:58:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.432+0000 7fbe14338640 1 -- 172.21.15.114:0/2044125741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe0c103300 msgr2=0x7fbe0c19b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:58:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.432+0000 7fbe14338640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe0c103300 0x7fbe0c19b780 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbe0800e9c0 tx=0x7fbe0800ee80 comp rx=0 tx=0).stop 2024-05-27T07:58:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.432+0000 7fbe14338640 1 -- 172.21.15.114:0/2044125741 shutdown_connections 2024-05-27T07:58:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.432+0000 7fbe14338640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fbdf80b2950 0x7fbdf80b4e60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.432+0000 7fbe14338640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe0c109050 0x7fbe0c19c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.432+0000 7fbe14338640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe0c108bb0 0x7fbe0c19bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.432+0000 7fbe14338640 1 --2- 172.21.15.114:0/2044125741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe0c103300 0x7fbe0c19b780 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:58:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.432+0000 7fbe14338640 1 -- 172.21.15.114:0/2044125741 >> 172.21.15.114:0/2044125741 conn(0x7fbe0c0781a0 msgr2=0x7fbe0c1012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:58:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.432+0000 7fbe14338640 1 -- 172.21.15.114:0/2044125741 shutdown_connections 2024-05-27T07:58:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:58:50.432+0000 7fbe14338640 1 -- 172.21.15.114:0/2044125741 wait complete. 2024-05-27T07:58:50.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 72854 12.58 MB/sec execute 182 sec latency 723.346 ms 2024-05-27T07:58:51.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 73178 12.62 MB/sec execute 183 sec latency 99.720 ms 2024-05-27T07:58:52.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:51 smithi163 ceph-mon[36451]: pgmap v658: 129 pgs: 129 active+clean; 398 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.4 MiB/s wr, 303 op/s 2024-05-27T07:58:52.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:51 smithi186 ceph-mon[36647]: pgmap v658: 129 pgs: 129 active+clean; 398 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.4 MiB/s wr, 303 op/s 2024-05-27T07:58:52.310 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:51 smithi114 ceph-mon[32038]: pgmap v658: 129 pgs: 129 active+clean; 398 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.4 MiB/s wr, 303 op/s 2024-05-27T07:58:52.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 73495 12.66 MB/sec execute 184 sec latency 75.486 ms 2024-05-27T07:58:53.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 73726 12.67 MB/sec execute 185 sec latency 173.419 ms 2024-05-27T07:58:54.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:53 smithi163 ceph-mon[36451]: pgmap v659: 129 pgs: 129 active+clean; 383 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.4 MiB/s wr, 514 op/s 2024-05-27T07:58:54.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:53 smithi186 ceph-mon[36647]: pgmap v659: 129 pgs: 129 active+clean; 383 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.4 MiB/s wr, 514 op/s 2024-05-27T07:58:54.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:53 smithi114 ceph-mon[32038]: pgmap v659: 129 pgs: 129 active+clean; 383 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.4 MiB/s wr, 514 op/s 2024-05-27T07:58:54.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 73900 12.64 MB/sec execute 186 sec latency 236.627 ms 2024-05-27T07:58:55.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:54 smithi163 ceph-mon[36451]: pgmap v660: 129 pgs: 129 active+clean; 379 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.8 MiB/s wr, 509 op/s 2024-05-27T07:58:55.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:54 smithi186 ceph-mon[36647]: pgmap v660: 129 pgs: 129 active+clean; 379 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.8 MiB/s wr, 509 op/s 2024-05-27T07:58:55.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:54 smithi114 ceph-mon[32038]: pgmap v660: 129 pgs: 129 active+clean; 379 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.8 MiB/s wr, 509 op/s 2024-05-27T07:58:55.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 74210 12.63 MB/sec execute 187 sec latency 240.441 ms 2024-05-27T07:58:56.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 74574 12.67 MB/sec execute 188 sec latency 200.503 ms 2024-05-27T07:58:57.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 74999 12.70 MB/sec execute 189 sec latency 174.057 ms 2024-05-27T07:58:58.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:57 smithi163 ceph-mon[36451]: pgmap v661: 129 pgs: 129 active+clean; 390 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.2 MiB/s wr, 544 op/s 2024-05-27T07:58:58.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:57 smithi186 ceph-mon[36647]: pgmap v661: 129 pgs: 129 active+clean; 390 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.2 MiB/s wr, 544 op/s 2024-05-27T07:58:58.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:57 smithi114 ceph-mon[32038]: pgmap v661: 129 pgs: 129 active+clean; 390 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.2 MiB/s wr, 544 op/s 2024-05-27T07:58:58.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 75247 12.72 MB/sec execute 190 sec latency 396.635 ms 2024-05-27T07:58:59.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:58:58 smithi163 ceph-mon[36451]: pgmap v662: 129 pgs: 129 active+clean; 390 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.6 MiB/s wr, 490 op/s 2024-05-27T07:58:59.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:58:58 smithi186 ceph-mon[36647]: pgmap v662: 129 pgs: 129 active+clean; 390 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.6 MiB/s wr, 490 op/s 2024-05-27T07:58:59.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:58:58 smithi114 ceph-mon[32038]: pgmap v662: 129 pgs: 129 active+clean; 390 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.6 MiB/s wr, 490 op/s 2024-05-27T07:58:59.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 75559 12.74 MB/sec execute 191 sec latency 212.001 ms 2024-05-27T07:59:00.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 75956 12.78 MB/sec execute 192 sec latency 205.873 ms 2024-05-27T07:59:01.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76275 12.80 MB/sec execute 193 sec latency 188.109 ms 2024-05-27T07:59:01.932 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:01 smithi114 ceph-mon[32038]: pgmap v663: 129 pgs: 129 active+clean; 394 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 466 op/s 2024-05-27T07:59:01.933 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd8e80db700 time 2024-05-27T07:59:01.616319+0000 2024-05-27T07:59:01.933 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:01.933 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:01.933 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd8fa920ba6] 2024-05-27T07:59:01.933 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8fa920dd1] 2024-05-27T07:59:01.933 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5618ee4f823d] 2024-05-27T07:59:01.933 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5618ee4e3971] 2024-05-27T07:59:01.933 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 5: (SnapRealm::check_cache() const+0x13e) [0x5618ee4e1fa0] 2024-05-27T07:59:01.933 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5618ee4e2651] 2024-05-27T07:59:01.934 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5618ee477347] 2024-05-27T07:59:01.934 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5618ee17387e] 2024-05-27T07:59:01.934 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5618ee178e94] 2024-05-27T07:59:01.934 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5618ee183969] 2024-05-27T07:59:01.934 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 11: (MDLog::_replay_thread()+0x1f38) [0x5618ee51c48c] 2024-05-27T07:59:01.934 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5618ee5259cb] 2024-05-27T07:59:01.934 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 13: (Thread::entry_wrapper()+0x43) [0x7fd8fa8ef72d] 2024-05-27T07:59:01.934 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd8fa8ef749] 2024-05-27T07:59:01.934 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd8f94931ca] 2024-05-27T07:59:01.934 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 16: clone() 2024-05-27T07:59:01.934 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.618+0000 7fd8e80db700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd8e80db700 time 2024-05-27T07:59:01.616319+0000 2024-05-27T07:59:01.935 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:01.935 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.935 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:01.935 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd8fa920ba6] 2024-05-27T07:59:01.935 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8fa920dd1] 2024-05-27T07:59:01.935 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5618ee4f823d] 2024-05-27T07:59:01.935 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5618ee4e3971] 2024-05-27T07:59:01.936 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 5: (SnapRealm::check_cache() const+0x13e) [0x5618ee4e1fa0] 2024-05-27T07:59:01.936 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5618ee4e2651] 2024-05-27T07:59:01.936 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5618ee477347] 2024-05-27T07:59:01.936 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5618ee17387e] 2024-05-27T07:59:01.936 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5618ee178e94] 2024-05-27T07:59:01.936 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5618ee183969] 2024-05-27T07:59:01.936 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 11: (MDLog::_replay_thread()+0x1f38) [0x5618ee51c48c] 2024-05-27T07:59:01.936 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5618ee5259cb] 2024-05-27T07:59:01.936 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 13: (Thread::entry_wrapper()+0x43) [0x7fd8fa8ef72d] 2024-05-27T07:59:01.936 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd8fa8ef749] 2024-05-27T07:59:01.936 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd8f94931ca] 2024-05-27T07:59:01.936 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 16: clone() 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: *** Caught signal (Aborted) ** 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: in thread 7fd8e80db700 thread_name:md_log_replay 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd8f949dd20] 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2: gsignal() 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 3: abort() 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd8fa920ced] 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8fa920dd1] 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5618ee4f823d] 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5618ee4e3971] 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 8: (SnapRealm::check_cache() const+0x13e) [0x5618ee4e1fa0] 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5618ee4e2651] 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5618ee477347] 2024-05-27T07:59:01.937 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5618ee17387e] 2024-05-27T07:59:01.938 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5618ee178e94] 2024-05-27T07:59:01.938 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5618ee183969] 2024-05-27T07:59:01.938 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 14: (MDLog::_replay_thread()+0x1f38) [0x5618ee51c48c] 2024-05-27T07:59:01.938 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5618ee5259cb] 2024-05-27T07:59:01.938 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 16: (Thread::entry_wrapper()+0x43) [0x7fd8fa8ef72d] 2024-05-27T07:59:01.938 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd8fa8ef749] 2024-05-27T07:59:01.938 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd8f94931ca] 2024-05-27T07:59:01.938 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 19: clone() 2024-05-27T07:59:01.938 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.621+0000 7fd8e80db700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:59:01.938 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: in thread 7fd8e80db700 thread_name:md_log_replay 2024-05-27T07:59:01.938 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.938 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:01.938 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd8f949dd20] 2024-05-27T07:59:01.939 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2: gsignal() 2024-05-27T07:59:01.939 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 3: abort() 2024-05-27T07:59:01.939 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd8fa920ced] 2024-05-27T07:59:01.939 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8fa920dd1] 2024-05-27T07:59:01.939 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5618ee4f823d] 2024-05-27T07:59:01.939 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5618ee4e3971] 2024-05-27T07:59:01.939 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 8: (SnapRealm::check_cache() const+0x13e) [0x5618ee4e1fa0] 2024-05-27T07:59:01.939 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5618ee4e2651] 2024-05-27T07:59:01.940 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5618ee477347] 2024-05-27T07:59:01.940 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5618ee17387e] 2024-05-27T07:59:01.940 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5618ee178e94] 2024-05-27T07:59:01.940 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5618ee183969] 2024-05-27T07:59:01.940 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 14: (MDLog::_replay_thread()+0x1f38) [0x5618ee51c48c] 2024-05-27T07:59:01.940 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5618ee5259cb] 2024-05-27T07:59:01.940 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 16: (Thread::entry_wrapper()+0x43) [0x7fd8fa8ef72d] 2024-05-27T07:59:01.940 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd8fa8ef749] 2024-05-27T07:59:01.941 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd8f94931ca] 2024-05-27T07:59:01.941 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 19: clone() 2024-05-27T07:59:01.941 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:59:01.941 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.941 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: -1> 2024-05-27T07:59:01.618+0000 7fd8e80db700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd8e80db700 time 2024-05-27T07:59:01.616319+0000 2024-05-27T07:59:01.941 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:01.941 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.941 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:01.941 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd8fa920ba6] 2024-05-27T07:59:01.941 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8fa920dd1] 2024-05-27T07:59:01.941 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5618ee4f823d] 2024-05-27T07:59:01.941 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5618ee4e3971] 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 5: (SnapRealm::check_cache() const+0x13e) [0x5618ee4e1fa0] 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5618ee4e2651] 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5618ee477347] 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5618ee17387e] 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5618ee178e94] 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5618ee183969] 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 11: (MDLog::_replay_thread()+0x1f38) [0x5618ee51c48c] 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5618ee5259cb] 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 13: (Thread::entry_wrapper()+0x43) [0x7fd8fa8ef72d] 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd8fa8ef749] 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd8f94931ca] 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 16: clone() 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.942 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 0> 2024-05-27T07:59:01.621+0000 7fd8e80db700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: in thread 7fd8e80db700 thread_name:md_log_replay 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd8f949dd20] 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2: gsignal() 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 3: abort() 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd8fa920ced] 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8fa920dd1] 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5618ee4f823d] 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5618ee4e3971] 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 8: (SnapRealm::check_cache() const+0x13e) [0x5618ee4e1fa0] 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5618ee4e2651] 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5618ee477347] 2024-05-27T07:59:01.943 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5618ee17387e] 2024-05-27T07:59:01.944 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5618ee178e94] 2024-05-27T07:59:01.944 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5618ee183969] 2024-05-27T07:59:01.944 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 14: (MDLog::_replay_thread()+0x1f38) [0x5618ee51c48c] 2024-05-27T07:59:01.944 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5618ee5259cb] 2024-05-27T07:59:01.944 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 16: (Thread::entry_wrapper()+0x43) [0x7fd8fa8ef72d] 2024-05-27T07:59:01.944 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd8fa8ef749] 2024-05-27T07:59:01.944 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd8f94931ca] 2024-05-27T07:59:01.944 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 19: clone() 2024-05-27T07:59:01.944 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:59:01.944 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.944 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: -1> 2024-05-27T07:59:01.618+0000 7fd8e80db700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd8e80db700 time 2024-05-27T07:59:01.616319+0000 2024-05-27T07:59:01.944 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd8fa920ba6] 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8fa920dd1] 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5618ee4f823d] 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5618ee4e3971] 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 5: (SnapRealm::check_cache() const+0x13e) [0x5618ee4e1fa0] 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5618ee4e2651] 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5618ee477347] 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5618ee17387e] 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5618ee178e94] 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5618ee183969] 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 11: (MDLog::_replay_thread()+0x1f38) [0x5618ee51c48c] 2024-05-27T07:59:01.945 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5618ee5259cb] 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 13: (Thread::entry_wrapper()+0x43) [0x7fd8fa8ef72d] 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd8fa8ef749] 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd8f94931ca] 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 16: clone() 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 0> 2024-05-27T07:59:01.621+0000 7fd8e80db700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: in thread 7fd8e80db700 thread_name:md_log_replay 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd8f949dd20] 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2: gsignal() 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 3: abort() 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd8fa920ced] 2024-05-27T07:59:01.946 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd8fa920dd1] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5618ee4f823d] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5618ee4e3971] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 8: (SnapRealm::check_cache() const+0x13e) [0x5618ee4e1fa0] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5618ee4e2651] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5618ee477347] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5618ee17387e] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5618ee178e94] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5618ee183969] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 14: (MDLog::_replay_thread()+0x1f38) [0x5618ee51c48c] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5618ee5259cb] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 16: (Thread::entry_wrapper()+0x43) [0x7fd8fa8ef72d] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd8fa8ef749] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd8f94931ca] 2024-05-27T07:59:01.947 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 19: clone() 2024-05-27T07:59:01.948 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:59:01.948 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132192]: 2024-05-27T07:59:01.948 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:01 smithi114 podman[132436]: 2024-05-27 07:59:01.710665621 +0000 UTC m=+0.031394284 container died 88a389f76c0eaa1bf1b65f80eb02885c5530a5f79c6ceeaeb00e0876248bc553 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-27T07:59:02.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:01 smithi163 ceph-mon[36451]: pgmap v663: 129 pgs: 129 active+clean; 394 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 466 op/s 2024-05-27T07:59:02.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:01 smithi186 ceph-mon[36647]: pgmap v663: 129 pgs: 129 active+clean; 394 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 466 op/s 2024-05-27T07:59:02.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76370 12.75 MB/sec execute 194 sec latency 527.990 ms 2024-05-27T07:59:02.807 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:02 smithi114 podman[132436]: 2024-05-27 07:59:02.535095854 +0000 UTC m=+0.855824505 container remove 88a389f76c0eaa1bf1b65f80eb02885c5530a5f79c6ceeaeb00e0876248bc553 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T07:59:02.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:02 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:59:03.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:02 smithi163 ceph-mon[36451]: pgmap v664: 129 pgs: 129 active+clean; 402 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.0 MiB/s wr, 538 op/s 2024-05-27T07:59:03.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:02 smithi163 ceph-mon[36451]: osdmap e160: 12 total, 12 up, 12 in 2024-05-27T07:59:03.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:02 smithi186 ceph-mon[36647]: pgmap v664: 129 pgs: 129 active+clean; 402 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.0 MiB/s wr, 538 op/s 2024-05-27T07:59:03.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:02 smithi186 ceph-mon[36647]: osdmap e160: 12 total, 12 up, 12 in 2024-05-27T07:59:03.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:02 smithi114 ceph-mon[32038]: pgmap v664: 129 pgs: 129 active+clean; 402 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.0 MiB/s wr, 538 op/s 2024-05-27T07:59:03.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:02 smithi114 ceph-mon[32038]: osdmap e160: 12 total, 12 up, 12 in 2024-05-27T07:59:03.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:02 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:59:03.308 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:02 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Consumed 3.702s CPU time. 2024-05-27T07:59:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.606+0000 7f6da3738640 1 -- 172.21.15.114:0/1076667789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d9c108010 msgr2=0x7f6d9c1084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.606+0000 7f6da3738640 1 --2- 172.21.15.114:0/1076667789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d9c108010 0x7f6d9c1084a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009be0 tx=0x7f6d9402a710 comp rx=0 tx=0).stop 2024-05-27T07:59:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.614+0000 7f6da3738640 1 -- 172.21.15.114:0/1076667789 shutdown_connections 2024-05-27T07:59:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.614+0000 7f6da3738640 1 --2- 172.21.15.114:0/1076667789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d9c108010 0x7f6d9c1084a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.614+0000 7f6da3738640 1 --2- 172.21.15.114:0/1076667789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d9c1075e0 0x7f6d9c1079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.614+0000 7f6da3738640 1 --2- 172.21.15.114:0/1076667789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d9c102b90 0x7f6d9c102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.614+0000 7f6da3738640 1 -- 172.21.15.114:0/1076667789 >> 172.21.15.114:0/1076667789 conn(0x7f6d9c078270 msgr2=0x7f6d9c0786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:03.617 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76398 12.70 MB/sec execute 195 sec latency 1037.477 ms 2024-05-27T07:59:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.617+0000 7f6da3738640 1 -- 172.21.15.114:0/1076667789 shutdown_connections 2024-05-27T07:59:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.620+0000 7f6da3738640 1 -- 172.21.15.114:0/1076667789 wait complete. 2024-05-27T07:59:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.621+0000 7f6da3738640 1 Processor -- start 2024-05-27T07:59:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.622+0000 7f6da3738640 1 -- start start 2024-05-27T07:59:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.622+0000 7f6da3738640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d9c102b90 0x7f6d9c074820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.623+0000 7f6da3738640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d9c1075e0 0x7f6d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.623+0000 7f6da3738640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d9c108010 0x7f6d9c06df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.623+0000 7f6da3738640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c11a4e0 con 0x7f6d9c108010 2024-05-27T07:59:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.623+0000 7f6da3738640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c11a360 con 0x7f6d9c102b90 2024-05-27T07:59:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.623+0000 7f6da3738640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c11a660 con 0x7f6d9c1075e0 2024-05-27T07:59:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.626+0000 7f6da14d4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d9c102b90 0x7f6d9c074820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.628+0000 7f6da1cd5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d9c108010 0x7f6d9c06df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.628+0000 7f6da1cd5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d9c108010 0x7f6d9c06df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41228/0 (socket says 172.21.15.114:41228) 2024-05-27T07:59:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.628+0000 7f6da1cd5640 1 -- 172.21.15.114:0/55586633 learned_addr learned my addr 172.21.15.114:0/55586633 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.629+0000 7f6da0cd3640 1 --2- 172.21.15.114:0/55586633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d9c1075e0 0x7f6d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.630+0000 7f6da1cd5640 1 -- 172.21.15.114:0/55586633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d9c1075e0 msgr2=0x7f6d9c06d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.630+0000 7f6da1cd5640 1 --2- 172.21.15.114:0/55586633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d9c1075e0 0x7f6d9c06d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.630+0000 7f6da1cd5640 1 -- 172.21.15.114:0/55586633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d9c102b90 msgr2=0x7f6d9c074820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.630+0000 7f6da1cd5640 1 --2- 172.21.15.114:0/55586633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d9c102b90 0x7f6d9c074820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.630+0000 7f6da1cd5640 1 -- 172.21.15.114:0/55586633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d9c06e730 con 0x7f6d9c108010 2024-05-27T07:59:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.631+0000 7f6da1cd5640 1 --2- 172.21.15.114:0/55586633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d9c108010 0x7f6d9c06df10 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f6d94034040 tx=0x7f6d94003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:03.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.632+0000 7f6d927fc640 1 -- 172.21.15.114:0/55586633 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94041020 con 0x7f6d9c108010 2024-05-27T07:59:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.633+0000 7f6da3738640 1 -- 172.21.15.114:0/55586633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d9c0723a0 con 0x7f6d9c108010 2024-05-27T07:59:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.633+0000 7f6da3738640 1 -- 172.21.15.114:0/55586633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d9c072890 con 0x7f6d9c108010 2024-05-27T07:59:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.633+0000 7f6d927fc640 1 -- 172.21.15.114:0/55586633 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d940318f0 con 0x7f6d9c108010 2024-05-27T07:59:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.633+0000 7f6d927fc640 1 -- 172.21.15.114:0/55586633 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94036040 con 0x7f6d9c108010 2024-05-27T07:59:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.643+0000 7f6daa5ee740 1 Processor -- start 2024-05-27T07:59:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.645+0000 7f6daa5ee740 1 -- start start 2024-05-27T07:59:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.646+0000 7f6daa5ee740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 0x5651f397b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.646+0000 7f6daa5ee740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 0x5651f397b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.646+0000 7f6daa5ee740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b94d0 0x5651f397be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.646+0000 7f6daa5ee740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5651f38edc30 con 0x5651f35da1b0 2024-05-27T07:59:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.646+0000 7f6daa5ee740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5651f38dd7d0 con 0x5651f35b94d0 2024-05-27T07:59:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.646+0000 7f6daa5ee740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5651f38e1810 con 0x5651f35b12f0 2024-05-27T07:59:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.646+0000 7f6d927fc640 1 -- 172.21.15.114:0/55586633 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6d94044020 con 0x7f6d9c108010 2024-05-27T07:59:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.647+0000 7f6d927fc640 1 --2- 172.21.15.114:0/55586633 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6d880b2b70 0x7f6d880b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.648+0000 7f6d927fc640 1 -- 172.21.15.114:0/55586633 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 9642+0+0 (secure 0 0 0) 0x7f6d94103320 con 0x7f6d9c108010 2024-05-27T07:59:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.649+0000 7f6da14d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 0x5651f397b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.649+0000 7f6da14d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 0x5651f397b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40882/0 (socket says 172.21.15.114:40882) 2024-05-27T07:59:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.649+0000 7f6da14d4640 1 -- 172.21.15.114:0/2659242926 learned_addr learned my addr 172.21.15.114:0/2659242926 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.650+0000 7f6da0cd3640 1 --2- 172.21.15.114:0/2659242926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b94d0 0x5651f397be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.650+0000 7f6da14d4640 1 -- 172.21.15.114:0/2659242926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b94d0 msgr2=0x5651f397be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.650+0000 7f6da14d4640 1 --2- 172.21.15.114:0/2659242926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b94d0 0x5651f397be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.650+0000 7f6da14d4640 1 -- 172.21.15.114:0/2659242926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 msgr2=0x5651f397b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.650+0000 7f6da14d4640 1 --2- 172.21.15.114:0/2659242926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 0x5651f397b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.650+0000 7f6da14d4640 1 -- 172.21.15.114:0/2659242926 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651f35f2210 con 0x5651f35b12f0 2024-05-27T07:59:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.651+0000 7f6da1cd5640 1 --2- 172.21.15.114:0/55586633 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6d880b2b70 0x7f6d880b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.651+0000 7f6da14d4640 1 --2- 172.21.15.114:0/2659242926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 0x5651f397b910 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c00dc80 tx=0x7f6d8c00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec197274c4f0777d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.656+0000 7f6da1cd5640 1 --2- 172.21.15.114:0/55586633 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6d880b2b70 0x7f6d880b5080 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6d940c73b0 tx=0x7f6d9402a710 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.658+0000 7f6d877fe640 1 -- 172.21.15.114:0/2659242926 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c011440 con 0x5651f35b12f0 2024-05-27T07:59:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.658+0000 7f6d877fe640 1 -- 172.21.15.114:0/2659242926 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d8c011a90 con 0x5651f35b12f0 2024-05-27T07:59:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.659+0000 7f6daa5ee740 1 -- 172.21.15.114:0/2659242926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 msgr2=0x5651f397b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.659+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/2659242926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 0x5651f397b910 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c00dc80 tx=0x7f6d8c00f730 comp rx=0 tx=0).stop 2024-05-27T07:59:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.660+0000 7f6daa5ee740 1 -- 172.21.15.114:0/2659242926 shutdown_connections 2024-05-27T07:59:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.660+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/2659242926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b94d0 0x5651f397be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.660+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/2659242926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 0x5651f397b910 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.660+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/2659242926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 0x5651f397b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.660+0000 7f6daa5ee740 1 -- 172.21.15.114:0/2659242926 >> 172.21.15.114:0/2659242926 conn(0x5651f35ce300 msgr2=0x5651f3977b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.665+0000 7f6daa5ee740 1 -- 172.21.15.114:0/2659242926 shutdown_connections 2024-05-27T07:59:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.670+0000 7f6daa5ee740 1 -- 172.21.15.114:0/2659242926 wait complete. 2024-05-27T07:59:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.670+0000 7f6daa5ee740 1 -- start start 2024-05-27T07:59:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.671+0000 7f6daa5ee740 1 Processor -- start 2024-05-27T07:59:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.674+0000 7f6daa5ee740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b12f0 0x5651f3985a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.674+0000 7f6daa5ee740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b94d0 0x5651f3985fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.674+0000 7f6daa5ee740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 0x5651f398b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.674+0000 7f6daa5ee740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5651f35c3790 con 0x5651f35da1b0 2024-05-27T07:59:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.674+0000 7f6daa5ee740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5651f3899890 con 0x5651f35b12f0 2024-05-27T07:59:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.674+0000 7f6daa5ee740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5651f3988550 con 0x5651f35b94d0 2024-05-27T07:59:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.674+0000 7f6daa5ee740 10 client.0 _ll_register_callbacks cb 0x5651f35f1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.678+0000 7f6da14d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b94d0 0x5651f3985fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.678+0000 7f6da14d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b94d0 0x5651f3985fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40886/0 (socket says 172.21.15.114:40886) 2024-05-27T07:59:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.678+0000 7f6da14d4640 1 -- 172.21.15.114:0/3400399397 learned_addr learned my addr 172.21.15.114:0/3400399397 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.678+0000 7f6da0cd3640 1 --2- 172.21.15.114:0/3400399397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 0x5651f398b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.678+0000 7f6da0cd3640 1 --2- 172.21.15.114:0/3400399397 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b12f0 0x5651f3985a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.685+0000 7f6da14d4640 1 -- 172.21.15.114:0/3400399397 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b12f0 msgr2=0x5651f3985a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:03.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.685+0000 7f6da14d4640 1 --2- 172.21.15.114:0/3400399397 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b12f0 0x5651f3985a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.685+0000 7f6da14d4640 1 -- 172.21.15.114:0/3400399397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 msgr2=0x5651f398b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:03.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.685+0000 7f6da14d4640 1 --2- 172.21.15.114:0/3400399397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 0x5651f398b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.685+0000 7f6da14d4640 1 -- 172.21.15.114:0/3400399397 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651f36be620 con 0x5651f35b94d0 2024-05-27T07:59:03.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.694+0000 7f6da14d4640 1 --2- 172.21.15.114:0/3400399397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b94d0 0x5651f3985fc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c010510 tx=0x7f6d8c010f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d01e17598ec5695 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:03.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.698+0000 7f6d84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:59:03.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.699+0000 7f6d84ff9640 1 -- 172.21.15.114:0/3400399397 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c029060 con 0x5651f35b94d0 2024-05-27T07:59:03.699 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.699+0000 7f6daa5ee740 1 -- 172.21.15.114:0/3400399397 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d8c011a90 con 0x5651f35b94d0 2024-05-27T07:59:03.700 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.699+0000 7f6d84ff9640 1 -- 172.21.15.114:0/3400399397 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(160..160 src has 1..160) ==== 9642+0+0 (secure 0 0 0) 0x7f6d8c011650 con 0x5651f35b94d0 2024-05-27T07:59:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.705+0000 7f6d84ff9640 1 -- 172.21.15.114:0/3400399397 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f6d8c00d880 con 0x5651f35b94d0 2024-05-27T07:59:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.707+0000 7f6daa5ee740 10 client.29185 fetch_fsmap learned FSMap version 108 2024-05-27T07:59:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.707+0000 7f6daa5ee740 1 -- 172.21.15.114:0/3400399397 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5651f3994720 con 0x5651f35b94d0 2024-05-27T07:59:03.712 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.712+0000 7f6d84ff9640 1 -- 172.21.15.114:0/3400399397 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d8c011a90 con 0x5651f35b94d0 2024-05-27T07:59:03.713 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.713+0000 7f6d84ff9640 1 -- 172.21.15.114:0/3400399397 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 108) ==== 4095+0+0 (secure 0 0 0) 0x7f6d8c003aa0 con 0x5651f35b94d0 2024-05-27T07:59:03.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.713+0000 7f6daa5ee740 10 client.29185 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:59:03.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.713+0000 7f6daa5ee740 10 client.29185 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:03.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.713+0000 7f6daa5ee740 20 client.29185 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:03.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.713+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/3400399397 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5651f3598d80 0x5651f3997ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.713+0000 7f6daa5ee740 4 client.29185 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:59:03.714 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.713+0000 7f6daa5ee740 1 -- 172.21.15.114:0/3400399397 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651f38efb80 con 0x5651f3598d80 2024-05-27T07:59:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.717+0000 7f6da0cd3640 1 --2- 172.21.15.114:0/3400399397 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5651f3598d80 0x5651f3997ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.724+0000 7f6da0cd3640 1 --2- 172.21.15.114:0/3400399397 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5651f3598d80 0x5651f3997ec0 crc :-1 s=READY pgs=506 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d1baa9af24df77f server_cookie=4c8f3c2b6d32fc34 in_seq=0 out_seq=0 2024-05-27T07:59:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.724+0000 7f6d84ff9640 10 client.29185 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.725+0000 7f6d84ff9640 1 -- 172.21.15.114:0/3400399397 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5651f38efb80 con 0x5651f3598d80 2024-05-27T07:59:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.725+0000 7f6d84ff9640 10 client.29185 handle_command_reply: tid=0 2024-05-27T07:59:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.726+0000 7f6daa5ee740 1 client.29185 shutdown 2024-05-27T07:59:03.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.732+0000 7f6daa5ee740 1 -- 172.21.15.114:0/3400399397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b94d0 msgr2=0x5651f3985fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:03.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.732+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/3400399397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b94d0 0x5651f3985fc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c010510 tx=0x7f6d8c010f00 comp rx=0 tx=0).stop 2024-05-27T07:59:03.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.735+0000 7f6daa5ee740 1 -- 172.21.15.114:0/3400399397 shutdown_connections 2024-05-27T07:59:03.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.735+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/3400399397 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5651f3598d80 0x5651f3997ec0 crc :-1 s=READY pgs=506 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.735+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/3400399397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 0x5651f398b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.735+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/3400399397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b94d0 0x5651f3985fc0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.735+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/3400399397 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b12f0 0x5651f3985a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.735+0000 7f6daa5ee740 1 -- 172.21.15.114:0/3400399397 >> 172.21.15.114:0/3400399397 conn(0x5651f35ce300 msgr2=0x5651f397d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:03.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.736+0000 7f6d84ff9640 0 client.29185 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:03.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.737+0000 7f6daa5ee740 1 -- 172.21.15.114:0/3400399397 shutdown_connections 2024-05-27T07:59:03.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.739+0000 7f6daa5ee740 1 -- 172.21.15.114:0/3400399397 wait complete. 2024-05-27T07:59:03.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.739+0000 7f6daa5ee740 20 client.29185 trim_cache size 0 max 16384 2024-05-27T07:59:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.768+0000 7f6daa5ee740 1 Processor -- start 2024-05-27T07:59:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.768+0000 7f6daa5ee740 1 -- start start 2024-05-27T07:59:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.769+0000 7f6daa5ee740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b12f0 0x5651f39847b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.769+0000 7f6daa5ee740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b94d0 0x5651f3985f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.769+0000 7f6daa5ee740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 0x5651f398b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.769+0000 7f6daa5ee740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5651f3981480 con 0x5651f35da1b0 2024-05-27T07:59:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.769+0000 7f6daa5ee740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8000fd90 con 0x5651f35b12f0 2024-05-27T07:59:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.769+0000 7f6daa5ee740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d80010460 con 0x5651f35b94d0 2024-05-27T07:59:03.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.771+0000 7f6da0cd3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 0x5651f398b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.771+0000 7f6da0cd3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 0x5651f398b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41252/0 (socket says 172.21.15.114:41252) 2024-05-27T07:59:03.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.771+0000 7f6da0cd3640 1 -- 172.21.15.114:0/1969541275 learned_addr learned my addr 172.21.15.114:0/1969541275 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:03.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.771+0000 7f6da0cd3640 1 --2- 172.21.15.114:0/1969541275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b12f0 0x5651f39847b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.772+0000 7f6da0cd3640 1 -- 172.21.15.114:0/1969541275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b94d0 msgr2=0x5651f3985f40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:59:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.772+0000 7f6da0cd3640 1 --2- 172.21.15.114:0/1969541275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b94d0 0x5651f3985f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.772+0000 7f6da0cd3640 1 -- 172.21.15.114:0/1969541275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 msgr2=0x5651f398b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.772+0000 7f6da0cd3640 1 --2- 172.21.15.114:0/1969541275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 0x5651f398b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.772+0000 7f6da0cd3640 1 -- 172.21.15.114:0/1969541275 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651f3983dc0 con 0x5651f35b12f0 2024-05-27T07:59:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.772+0000 7f6da0cd3640 1 --2- 172.21.15.114:0/1969541275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b12f0 0x5651f39847b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6d98000c00 tx=0x7f6d980047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9cab92f916035b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.776+0000 7f6d86ffd640 1 -- 172.21.15.114:0/1969541275 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d98015930 con 0x5651f35b12f0 2024-05-27T07:59:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.776+0000 7f6d86ffd640 1 -- 172.21.15.114:0/1969541275 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d98018450 con 0x5651f35b12f0 2024-05-27T07:59:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.777+0000 7f6daa5ee740 1 -- 172.21.15.114:0/1969541275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b12f0 msgr2=0x5651f39847b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.777+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/1969541275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b12f0 0x5651f39847b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6d98000c00 tx=0x7f6d980047d0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.782+0000 7f6daa5ee740 1 -- 172.21.15.114:0/1969541275 shutdown_connections 2024-05-27T07:59:03.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.782+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/1969541275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35da1b0 0x5651f398b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.782+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/1969541275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b94d0 0x5651f3985f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.782+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/1969541275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35b12f0 0x5651f39847b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.782+0000 7f6daa5ee740 1 -- 172.21.15.114:0/1969541275 >> 172.21.15.114:0/1969541275 conn(0x5651f35ce300 msgr2=0x5651f3985830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:03.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.789+0000 7f6daa5ee740 1 -- 172.21.15.114:0/1969541275 shutdown_connections 2024-05-27T07:59:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.791+0000 7f6daa5ee740 1 -- 172.21.15.114:0/1969541275 wait complete. 2024-05-27T07:59:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.791+0000 7f6daa5ee740 1 -- start start 2024-05-27T07:59:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.791+0000 7f6daa5ee740 1 Processor -- start 2024-05-27T07:59:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.792+0000 7f6daa5ee740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 0x5651f398cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.792+0000 7f6daa5ee740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35b94d0 0x5651f398d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.792+0000 7f6daa5ee740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35da1b0 0x5651f39fe560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.792+0000 7f6daa5ee740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8000e0a0 con 0x5651f35b94d0 2024-05-27T07:59:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.792+0000 7f6daa5ee740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8000ed30 con 0x5651f35da1b0 2024-05-27T07:59:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.792+0000 7f6daa5ee740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8000e660 con 0x5651f35b12f0 2024-05-27T07:59:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.792+0000 7f6daa5ee740 10 client.0 _ll_register_callbacks cb 0x7f6d8c015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.796+0000 7f6da0cd3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 0x5651f398cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.796+0000 7f6da0cd3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35da1b0 0x5651f39fe560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.796+0000 7f6da0cd3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 0x5651f398cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:40906/0 (socket says 172.21.15.114:40906) 2024-05-27T07:59:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.796+0000 7f6da0cd3640 1 -- 172.21.15.114:0/871825912 learned_addr learned my addr 172.21.15.114:0/871825912 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.797+0000 7f6da14d4640 1 --2- 172.21.15.114:0/871825912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35b94d0 0x5651f398d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.800+0000 7f6da0cd3640 1 -- 172.21.15.114:0/871825912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35da1b0 msgr2=0x5651f39fe560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.800+0000 7f6da0cd3640 1 --2- 172.21.15.114:0/871825912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35da1b0 0x5651f39fe560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.800+0000 7f6da0cd3640 1 -- 172.21.15.114:0/871825912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35b94d0 msgr2=0x5651f398d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.800+0000 7f6da0cd3640 1 --2- 172.21.15.114:0/871825912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35b94d0 0x5651f398d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.800+0000 7f6da0cd3640 1 -- 172.21.15.114:0/871825912 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651f398dac0 con 0x5651f35b12f0 2024-05-27T07:59:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.802+0000 7f6da14d4640 1 --2- 172.21.15.114:0/871825912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35b94d0 0x5651f398d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:59:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.807+0000 7f6da0cd3640 1 --2- 172.21.15.114:0/871825912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 0x5651f398cda0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6d980163f0 tx=0x7f6d98018d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2bb5d738afee901 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.807+0000 7f6d84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:59:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.811+0000 7f6d84ff9640 1 -- 172.21.15.114:0/871825912 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d98037060 con 0x5651f35b12f0 2024-05-27T07:59:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.811+0000 7f6d84ff9640 1 -- 172.21.15.114:0/871825912 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(160..160 src has 1..160) ==== 9642+0+0 (secure 0 0 0) 0x7f6d980188d0 con 0x5651f35b12f0 2024-05-27T07:59:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.814+0000 7f6daa5ee740 1 -- 172.21.15.114:0/871825912 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d98018450 con 0x5651f35b12f0 2024-05-27T07:59:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.817+0000 7f6d84ff9640 1 -- 172.21.15.114:0/871825912 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f6d98018ad0 con 0x5651f35b12f0 2024-05-27T07:59:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.817+0000 7f6daa5ee740 10 client.29188 fetch_fsmap learned FSMap version 108 2024-05-27T07:59:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.817+0000 7f6daa5ee740 1 -- 172.21.15.114:0/871825912 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5651f39ffa40 con 0x5651f35b12f0 2024-05-27T07:59:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.821+0000 7f6d84ff9640 1 -- 172.21.15.114:0/871825912 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d98018450 con 0x5651f35b12f0 2024-05-27T07:59:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.821+0000 7f6d84ff9640 1 -- 172.21.15.114:0/871825912 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 108) ==== 4095+0+0 (secure 0 0 0) 0x7f6d98021050 con 0x5651f35b12f0 2024-05-27T07:59:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.821+0000 7f6daa5ee740 10 client.29188 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:59:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.821+0000 7f6daa5ee740 10 client.29188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.821+0000 7f6daa5ee740 20 client.29188 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.821+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/871825912 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5651f356e1c0 0x5651f3a02220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.821+0000 7f6daa5ee740 4 client.29188 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:59:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.821+0000 7f6daa5ee740 1 -- 172.21.15.114:0/871825912 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5651f38e9360 con 0x5651f356e1c0 2024-05-27T07:59:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.823+0000 7f6da14d4640 1 --2- 172.21.15.114:0/871825912 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5651f356e1c0 0x5651f3a02220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.830+0000 7f6da14d4640 1 --2- 172.21.15.114:0/871825912 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5651f356e1c0 0x5651f3a02220 crc :-1 s=READY pgs=507 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5ad5fb6bc1c1e0a server_cookie=8406a2ad8f2b9e6 in_seq=0 out_seq=0 2024-05-27T07:59:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:03.830+0000 7f6d84ff9640 10 client.29188 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:04.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76404 12.63 MB/sec execute 196 sec latency 1860.687 ms 2024-05-27T07:59:05.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76409 12.57 MB/sec execute 197 sec latency 2860.729 ms 2024-05-27T07:59:06.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:05 smithi163 ceph-mon[36451]: pgmap v666: 129 pgs: 4 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 114 active+clean; 407 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.7 MiB/s wr, 340 op/s 2024-05-27T07:59:06.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:05 smithi186 ceph-mon[36647]: pgmap v666: 129 pgs: 4 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 114 active+clean; 407 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.7 MiB/s wr, 340 op/s 2024-05-27T07:59:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.241+0000 7f6d84ff9640 1 -- 172.21.15.114:0/871825912 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5651f38e9360 con 0x5651f356e1c0 2024-05-27T07:59:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.241+0000 7f6d84ff9640 10 client.29188 handle_command_reply: tid=0 2024-05-27T07:59:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.241+0000 7f6daa5ee740 1 client.29188 shutdown 2024-05-27T07:59:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.241+0000 7f6daa5ee740 1 -- 172.21.15.114:0/871825912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 msgr2=0x5651f398cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.241+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/871825912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 0x5651f398cda0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6d980163f0 tx=0x7f6d98018d90 comp rx=0 tx=0).stop 2024-05-27T07:59:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.242+0000 7f6daa5ee740 1 -- 172.21.15.114:0/871825912 shutdown_connections 2024-05-27T07:59:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.242+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/871825912 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5651f356e1c0 0x5651f3a02220 crc :-1 s=READY pgs=507 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.242+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/871825912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651f35da1b0 0x5651f39fe560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.242+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/871825912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651f35b94d0 0x5651f398d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.242+0000 7f6daa5ee740 1 --2- 172.21.15.114:0/871825912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651f35b12f0 0x5651f398cda0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.242+0000 7f6daa5ee740 1 -- 172.21.15.114:0/871825912 >> 172.21.15.114:0/871825912 conn(0x5651f35ce300 msgr2=0x5651f397b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:06.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.242+0000 7f6d84ff9640 0 client.29188 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:06.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.242+0000 7f6daa5ee740 1 -- 172.21.15.114:0/871825912 shutdown_connections 2024-05-27T07:59:06.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.242+0000 7f6daa5ee740 1 -- 172.21.15.114:0/871825912 wait complete. 2024-05-27T07:59:06.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.242+0000 7f6daa5ee740 20 client.29188 trim_cache size 0 max 16384 2024-05-27T07:59:06.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:59:06.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:59:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:59:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.243+0000 7f6da3738640 1 -- 172.21.15.114:0/55586633 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6d880b2b70 msgr2=0x7f6d880b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.243+0000 7f6da3738640 1 --2- 172.21.15.114:0/55586633 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6d880b2b70 0x7f6d880b5080 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6d940c73b0 tx=0x7f6d9402a710 comp rx=0 tx=0).stop 2024-05-27T07:59:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.243+0000 7f6da3738640 1 -- 172.21.15.114:0/55586633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d9c108010 msgr2=0x7f6d9c06df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.243+0000 7f6da3738640 1 --2- 172.21.15.114:0/55586633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d9c108010 0x7f6d9c06df10 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f6d94034040 tx=0x7f6d94003ba0 comp rx=0 tx=0).stop 2024-05-27T07:59:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.243+0000 7f6da3738640 1 -- 172.21.15.114:0/55586633 shutdown_connections 2024-05-27T07:59:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.243+0000 7f6da3738640 1 --2- 172.21.15.114:0/55586633 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6d880b2b70 0x7f6d880b5080 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.243+0000 7f6da3738640 1 --2- 172.21.15.114:0/55586633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d9c108010 0x7f6d9c06df10 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.243+0000 7f6da3738640 1 --2- 172.21.15.114:0/55586633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d9c1075e0 0x7f6d9c06d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.243+0000 7f6da3738640 1 --2- 172.21.15.114:0/55586633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d9c102b90 0x7f6d9c074820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.243+0000 7f6da3738640 1 -- 172.21.15.114:0/55586633 >> 172.21.15.114:0/55586633 conn(0x7f6d9c078270 msgr2=0x7f6d9c105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.243+0000 7f6da3738640 1 -- 172.21.15.114:0/55586633 shutdown_connections 2024-05-27T07:59:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:06.243+0000 7f6da3738640 1 -- 172.21.15.114:0/55586633 wait complete. 2024-05-27T07:59:06.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:05 smithi114 ceph-mon[32038]: pgmap v666: 129 pgs: 4 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 114 active+clean; 407 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.7 MiB/s wr, 340 op/s 2024-05-27T07:59:06.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76428 12.51 MB/sec execute 198 sec latency 3100.795 ms 2024-05-27T07:59:07.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76444 12.45 MB/sec execute 199 sec latency 1815.933 ms 2024-05-27T07:59:08.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:07 smithi163 ceph-mon[36451]: pgmap v667: 129 pgs: 13 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 86 active+clean; 401 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 477 op/s 2024-05-27T07:59:08.229 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:07 smithi186 ceph-mon[36647]: pgmap v667: 129 pgs: 13 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 86 active+clean; 401 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 477 op/s 2024-05-27T07:59:08.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:07 smithi114 ceph-mon[32038]: pgmap v667: 129 pgs: 13 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 86 active+clean; 401 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 477 op/s 2024-05-27T07:59:08.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76457 12.40 MB/sec execute 200 sec latency 1641.488 ms 2024-05-27T07:59:09.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76472 12.34 MB/sec execute 201 sec latency 2130.991 ms 2024-05-27T07:59:10.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:09 smithi163 ceph-mon[36451]: pgmap v668: 129 pgs: 19 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 63 active+clean; 400 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.3 MiB/s wr, 608 op/s 2024-05-27T07:59:10.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:09 smithi186 ceph-mon[36647]: pgmap v668: 129 pgs: 19 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 63 active+clean; 400 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.3 MiB/s wr, 608 op/s 2024-05-27T07:59:10.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:09 smithi114 ceph-mon[32038]: pgmap v668: 129 pgs: 19 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 63 active+clean; 400 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.3 MiB/s wr, 608 op/s 2024-05-27T07:59:10.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76501 12.30 MB/sec execute 202 sec latency 2730.812 ms 2024-05-27T07:59:11.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:10 smithi163 ceph-mon[36451]: pgmap v669: 129 pgs: 19 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 63 active+clean; 387 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 588 op/s 2024-05-27T07:59:11.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:10 smithi186 ceph-mon[36647]: pgmap v669: 129 pgs: 19 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 63 active+clean; 387 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 588 op/s 2024-05-27T07:59:11.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:10 smithi114 ceph-mon[32038]: pgmap v669: 129 pgs: 19 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 63 active+clean; 387 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 588 op/s 2024-05-27T07:59:11.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76539 12.24 MB/sec execute 203 sec latency 1644.294 ms 2024-05-27T07:59:12.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76590 12.19 MB/sec execute 204 sec latency 2268.365 ms 2024-05-27T07:59:13.497 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:13 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:59:13.497 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:13 smithi114 systemd[1]: Stopped Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:59:13.497 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:13 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Consumed 3.702s CPU time. 2024-05-27T07:59:13.497 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:13 smithi114 systemd[1]: Starting Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:59:13.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76624 12.14 MB/sec execute 205 sec latency 1906.147 ms 2024-05-27T07:59:13.807 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:13 smithi114 podman[132564]: 2024-05-27 07:59:13.402778455 +0000 UTC m=+0.080349172 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:59:13.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:13 smithi114 podman[132564]: 2024-05-27 07:59:13.531380288 +0000 UTC m=+0.208951010 container create 3d6d74ef03281ea5b65bc0ff501822a45f57bf7d5c93658cb7a59197a88deb78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-05-27T07:59:14.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:13 smithi163 ceph-mon[36451]: pgmap v670: 129 pgs: 19 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 63 active+clean; 375 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 3.8 MiB/s wr, 543 op/s 2024-05-27T07:59:14.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:13 smithi186 ceph-mon[36647]: pgmap v670: 129 pgs: 19 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 63 active+clean; 375 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 3.8 MiB/s wr, 543 op/s 2024-05-27T07:59:14.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:13 smithi114 ceph-mon[32038]: pgmap v670: 129 pgs: 19 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 63 active+clean; 375 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 3.8 MiB/s wr, 543 op/s 2024-05-27T07:59:14.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76671 12.10 MB/sec execute 206 sec latency 2132.059 ms 2024-05-27T07:59:14.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:14 smithi114 podman[132564]: 2024-05-27 07:59:14.406812223 +0000 UTC m=+1.084382941 container init 3d6d74ef03281ea5b65bc0ff501822a45f57bf7d5c93658cb7a59197a88deb78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:59:14.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:14 smithi114 podman[132564]: 2024-05-27 07:59:14.421352628 +0000 UTC m=+1.098923330 container start 3d6d74ef03281ea5b65bc0ff501822a45f57bf7d5c93658cb7a59197a88deb78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-27T07:59:14.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: starting mds.g at 2024-05-27T07:59:14.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:14 smithi114 bash[132564]: 3d6d74ef03281ea5b65bc0ff501822a45f57bf7d5c93658cb7a59197a88deb78 2024-05-27T07:59:14.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:14 smithi114 systemd[1]: Started Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:59:15.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:14 smithi163 ceph-mon[36451]: daemon mds.g restarted 2024-05-27T07:59:15.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:14 smithi186 ceph-mon[36647]: daemon mds.g restarted 2024-05-27T07:59:15.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:14 smithi114 ceph-mon[32038]: daemon mds.g restarted 2024-05-27T07:59:15.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76744 12.07 MB/sec execute 207 sec latency 3128.234 ms 2024-05-27T07:59:15.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff88584c700 time 2024-05-27T07:59:15.497970+0000 2024-05-27T07:59:15.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:15.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:15.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff898091ba6] 2024-05-27T07:59:15.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff898091dd1] 2024-05-27T07:59:15.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fc192923d] 2024-05-27T07:59:15.813 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563fc1914971] 2024-05-27T07:59:15.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 5: (SnapRealm::check_cache() const+0x13e) [0x563fc1912fa0] 2024-05-27T07:59:15.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fc1913651] 2024-05-27T07:59:15.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fc18a8347] 2024-05-27T07:59:15.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fc15a487e] 2024-05-27T07:59:15.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563fc15a9f2f] 2024-05-27T07:59:15.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563fc15b4969] 2024-05-27T07:59:15.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 11: (MDLog::_replay_thread()+0x1f38) [0x563fc194d48c] 2024-05-27T07:59:15.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563fc19569cb] 2024-05-27T07:59:15.814 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 13: (Thread::entry_wrapper()+0x43) [0x7ff89806072d] 2024-05-27T07:59:15.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff898060749] 2024-05-27T07:59:15.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff896c041ca] 2024-05-27T07:59:15.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 16: clone() 2024-05-27T07:59:15.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: *** Caught signal (Aborted) ** 2024-05-27T07:59:15.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: in thread 7ff88584c700 thread_name:md_log_replay 2024-05-27T07:59:15.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.501+0000 7ff88584c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff88584c700 time 2024-05-27T07:59:15.497970+0000 2024-05-27T07:59:15.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:15.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:15.815 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff898091ba6] 2024-05-27T07:59:15.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff898091dd1] 2024-05-27T07:59:15.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fc192923d] 2024-05-27T07:59:15.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563fc1914971] 2024-05-27T07:59:15.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 5: (SnapRealm::check_cache() const+0x13e) [0x563fc1912fa0] 2024-05-27T07:59:15.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fc1913651] 2024-05-27T07:59:15.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fc18a8347] 2024-05-27T07:59:15.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fc15a487e] 2024-05-27T07:59:15.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563fc15a9f2f] 2024-05-27T07:59:15.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563fc15b4969] 2024-05-27T07:59:15.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 11: (MDLog::_replay_thread()+0x1f38) [0x563fc194d48c] 2024-05-27T07:59:15.816 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563fc19569cb] 2024-05-27T07:59:15.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 13: (Thread::entry_wrapper()+0x43) [0x7ff89806072d] 2024-05-27T07:59:15.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff898060749] 2024-05-27T07:59:15.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff896c041ca] 2024-05-27T07:59:15.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 16: clone() 2024-05-27T07:59:15.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:15.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff896c0ed20] 2024-05-27T07:59:15.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2: gsignal() 2024-05-27T07:59:15.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 3: abort() 2024-05-27T07:59:15.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff898091ced] 2024-05-27T07:59:15.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff898091dd1] 2024-05-27T07:59:15.817 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fc192923d] 2024-05-27T07:59:15.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563fc1914971] 2024-05-27T07:59:15.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 8: (SnapRealm::check_cache() const+0x13e) [0x563fc1912fa0] 2024-05-27T07:59:15.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fc1913651] 2024-05-27T07:59:15.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fc18a8347] 2024-05-27T07:59:15.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fc15a487e] 2024-05-27T07:59:15.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563fc15a9f2f] 2024-05-27T07:59:15.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563fc15b4969] 2024-05-27T07:59:15.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 14: (MDLog::_replay_thread()+0x1f38) [0x563fc194d48c] 2024-05-27T07:59:15.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563fc19569cb] 2024-05-27T07:59:15.818 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 16: (Thread::entry_wrapper()+0x43) [0x7ff89806072d] 2024-05-27T07:59:15.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff898060749] 2024-05-27T07:59:15.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff896c041ca] 2024-05-27T07:59:15.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 19: clone() 2024-05-27T07:59:15.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.507+0000 7ff88584c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:59:15.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: in thread 7ff88584c700 thread_name:md_log_replay 2024-05-27T07:59:15.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:15.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff896c0ed20] 2024-05-27T07:59:15.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2: gsignal() 2024-05-27T07:59:15.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 3: abort() 2024-05-27T07:59:15.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff898091ced] 2024-05-27T07:59:15.819 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff898091dd1] 2024-05-27T07:59:15.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fc192923d] 2024-05-27T07:59:15.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563fc1914971] 2024-05-27T07:59:15.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 8: (SnapRealm::check_cache() const+0x13e) [0x563fc1912fa0] 2024-05-27T07:59:15.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fc1913651] 2024-05-27T07:59:15.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fc18a8347] 2024-05-27T07:59:15.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fc15a487e] 2024-05-27T07:59:15.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563fc15a9f2f] 2024-05-27T07:59:15.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563fc15b4969] 2024-05-27T07:59:15.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 14: (MDLog::_replay_thread()+0x1f38) [0x563fc194d48c] 2024-05-27T07:59:15.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563fc19569cb] 2024-05-27T07:59:15.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 16: (Thread::entry_wrapper()+0x43) [0x7ff89806072d] 2024-05-27T07:59:15.820 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff898060749] 2024-05-27T07:59:15.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff896c041ca] 2024-05-27T07:59:15.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 19: clone() 2024-05-27T07:59:15.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:59:15.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: -1> 2024-05-27T07:59:15.501+0000 7ff88584c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff88584c700 time 2024-05-27T07:59:15.497970+0000 2024-05-27T07:59:15.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:15.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:15.821 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff898091ba6] 2024-05-27T07:59:15.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff898091dd1] 2024-05-27T07:59:15.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fc192923d] 2024-05-27T07:59:15.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563fc1914971] 2024-05-27T07:59:15.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 5: (SnapRealm::check_cache() const+0x13e) [0x563fc1912fa0] 2024-05-27T07:59:15.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fc1913651] 2024-05-27T07:59:15.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fc18a8347] 2024-05-27T07:59:15.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fc15a487e] 2024-05-27T07:59:15.822 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563fc15a9f2f] 2024-05-27T07:59:15.823 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563fc15b4969] 2024-05-27T07:59:15.823 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 11: (MDLog::_replay_thread()+0x1f38) [0x563fc194d48c] 2024-05-27T07:59:15.823 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563fc19569cb] 2024-05-27T07:59:15.823 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 13: (Thread::entry_wrapper()+0x43) [0x7ff89806072d] 2024-05-27T07:59:15.823 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff898060749] 2024-05-27T07:59:15.823 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff896c041ca] 2024-05-27T07:59:15.823 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 16: clone() 2024-05-27T07:59:15.823 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.823 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 0> 2024-05-27T07:59:15.507+0000 7ff88584c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:59:15.824 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: in thread 7ff88584c700 thread_name:md_log_replay 2024-05-27T07:59:15.824 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.824 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:15.824 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff896c0ed20] 2024-05-27T07:59:15.824 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2: gsignal() 2024-05-27T07:59:15.824 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 3: abort() 2024-05-27T07:59:15.824 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff898091ced] 2024-05-27T07:59:15.824 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff898091dd1] 2024-05-27T07:59:15.824 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fc192923d] 2024-05-27T07:59:15.825 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563fc1914971] 2024-05-27T07:59:15.825 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 8: (SnapRealm::check_cache() const+0x13e) [0x563fc1912fa0] 2024-05-27T07:59:15.825 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fc1913651] 2024-05-27T07:59:15.825 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fc18a8347] 2024-05-27T07:59:15.825 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fc15a487e] 2024-05-27T07:59:15.825 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563fc15a9f2f] 2024-05-27T07:59:15.825 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563fc15b4969] 2024-05-27T07:59:15.825 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 14: (MDLog::_replay_thread()+0x1f38) [0x563fc194d48c] 2024-05-27T07:59:15.825 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563fc19569cb] 2024-05-27T07:59:15.826 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 16: (Thread::entry_wrapper()+0x43) [0x7ff89806072d] 2024-05-27T07:59:15.826 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff898060749] 2024-05-27T07:59:15.827 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff896c041ca] 2024-05-27T07:59:15.827 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 19: clone() 2024-05-27T07:59:15.827 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:59:15.827 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.827 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: -1> 2024-05-27T07:59:15.501+0000 7ff88584c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff88584c700 time 2024-05-27T07:59:15.497970+0000 2024-05-27T07:59:15.827 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:15.827 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.827 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:15.828 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff898091ba6] 2024-05-27T07:59:15.828 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff898091dd1] 2024-05-27T07:59:15.828 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fc192923d] 2024-05-27T07:59:15.828 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563fc1914971] 2024-05-27T07:59:15.828 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 5: (SnapRealm::check_cache() const+0x13e) [0x563fc1912fa0] 2024-05-27T07:59:15.828 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fc1913651] 2024-05-27T07:59:15.828 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fc18a8347] 2024-05-27T07:59:15.828 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fc15a487e] 2024-05-27T07:59:15.828 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563fc15a9f2f] 2024-05-27T07:59:15.828 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563fc15b4969] 2024-05-27T07:59:15.829 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 11: (MDLog::_replay_thread()+0x1f38) [0x563fc194d48c] 2024-05-27T07:59:15.829 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563fc19569cb] 2024-05-27T07:59:15.829 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 13: (Thread::entry_wrapper()+0x43) [0x7ff89806072d] 2024-05-27T07:59:15.829 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff898060749] 2024-05-27T07:59:15.829 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff896c041ca] 2024-05-27T07:59:15.829 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 16: clone() 2024-05-27T07:59:15.829 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.829 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 0> 2024-05-27T07:59:15.507+0000 7ff88584c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:59:15.829 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: in thread 7ff88584c700 thread_name:md_log_replay 2024-05-27T07:59:15.830 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.830 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:15.830 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff896c0ed20] 2024-05-27T07:59:15.830 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2: gsignal() 2024-05-27T07:59:15.830 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 3: abort() 2024-05-27T07:59:15.830 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff898091ced] 2024-05-27T07:59:15.830 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff898091dd1] 2024-05-27T07:59:15.830 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fc192923d] 2024-05-27T07:59:15.830 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563fc1914971] 2024-05-27T07:59:15.831 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 8: (SnapRealm::check_cache() const+0x13e) [0x563fc1912fa0] 2024-05-27T07:59:15.831 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fc1913651] 2024-05-27T07:59:15.831 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fc18a8347] 2024-05-27T07:59:15.831 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fc15a487e] 2024-05-27T07:59:15.831 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x563fc15a9f2f] 2024-05-27T07:59:15.831 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563fc15b4969] 2024-05-27T07:59:15.831 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 14: (MDLog::_replay_thread()+0x1f38) [0x563fc194d48c] 2024-05-27T07:59:15.831 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563fc19569cb] 2024-05-27T07:59:15.831 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 16: (Thread::entry_wrapper()+0x43) [0x7ff89806072d] 2024-05-27T07:59:15.832 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff898060749] 2024-05-27T07:59:15.832 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff896c041ca] 2024-05-27T07:59:15.832 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 19: clone() 2024-05-27T07:59:15.832 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:59:15.832 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g[132577]: 2024-05-27T07:59:15.832 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:15 smithi114 podman[132612]: 2024-05-27 07:59:15.644915979 +0000 UTC m=+0.050185767 container died 3d6d74ef03281ea5b65bc0ff501822a45f57bf7d5c93658cb7a59197a88deb78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T07:59:16.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:15 smithi163 ceph-mon[36451]: pgmap v671: 129 pgs: 19 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 63 active+clean; 376 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 3.7 MiB/s wr, 485 op/s 2024-05-27T07:59:16.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:15 smithi163 ceph-mon[36451]: osdmap e161: 12 total, 12 up, 12 in 2024-05-27T07:59:16.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:15 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6838/3476461298,v1:172.21.15.114:6839/3476461298] up:boot 2024-05-27T07:59:16.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:15 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:59:16.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:15 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:59:16.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:15 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:59:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:15 smithi186 ceph-mon[36647]: pgmap v671: 129 pgs: 19 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 63 active+clean; 376 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 3.7 MiB/s wr, 485 op/s 2024-05-27T07:59:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:15 smithi186 ceph-mon[36647]: osdmap e161: 12 total, 12 up, 12 in 2024-05-27T07:59:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:15 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6838/3476461298,v1:172.21.15.114:6839/3476461298] up:boot 2024-05-27T07:59:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:15 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:59:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:15 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:59:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:15 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:59:16.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:15 smithi114 ceph-mon[32038]: pgmap v671: 129 pgs: 19 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 63 active+clean; 376 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 3.7 MiB/s wr, 485 op/s 2024-05-27T07:59:16.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:15 smithi114 ceph-mon[32038]: osdmap e161: 12 total, 12 up, 12 in 2024-05-27T07:59:16.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:15 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6838/3476461298,v1:172.21.15.114:6839/3476461298] up:boot 2024-05-27T07:59:16.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:15 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 4 up:standby 2024-05-27T07:59:16.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:15 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:59:16.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:15 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 3 up:standby 2024-05-27T07:59:16.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76833 12.04 MB/sec execute 208 sec latency 1360.889 ms 2024-05-27T07:59:16.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:59:17.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:16 smithi114 ceph-mon[32038]: pgmap v673: 129 pgs: 15 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 80 active+clean; 375 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.6 MiB/s wr, 368 op/s 2024-05-27T07:59:17.058 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:16 smithi114 podman[132612]: 2024-05-27 07:59:16.688398722 +0000 UTC m=+1.093668473 container remove 3d6d74ef03281ea5b65bc0ff501822a45f57bf7d5c93658cb7a59197a88deb78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-g, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T07:59:17.058 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:16 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:59:17.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:16 smithi163 ceph-mon[36451]: pgmap v673: 129 pgs: 15 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 80 active+clean; 375 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.6 MiB/s wr, 368 op/s 2024-05-27T07:59:17.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:16 smithi186 ceph-mon[36647]: pgmap v673: 129 pgs: 15 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 80 active+clean; 375 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.6 MiB/s wr, 368 op/s 2024-05-27T07:59:17.358 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:59:17.557 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:17 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:59:17.558 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:17 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.332s CPU time. 2024-05-27T07:59:17.618 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76901 12.00 MB/sec execute 209 sec latency 1160.298 ms 2024-05-27T07:59:18.619 INFO:tasks.workunit.client.0.smithi114.stdout: 10 76964 11.96 MB/sec execute 210 sec latency 1843.696 ms 2024-05-27T07:59:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.431+0000 7ff273aaf640 1 -- 172.21.15.114:0/1215159821 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff26c11d240 msgr2=0x7ff26c11f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.431+0000 7ff273aaf640 1 --2- 172.21.15.114:0/1215159821 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff26c11d240 0x7ff26c11f660 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff264009900 tx=0x7ff26402a4d0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.431+0000 7ff273aaf640 1 -- 172.21.15.114:0/1215159821 shutdown_connections 2024-05-27T07:59:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.431+0000 7ff273aaf640 1 --2- 172.21.15.114:0/1215159821 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff26c11d240 0x7ff26c11f660 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.431+0000 7ff273aaf640 1 --2- 172.21.15.114:0/1215159821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff26c11a8e0 0x7ff26c11cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.431+0000 7ff273aaf640 1 --2- 172.21.15.114:0/1215159821 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff26c10a1f0 0x7ff26c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.431+0000 7ff273aaf640 1 -- 172.21.15.114:0/1215159821 >> 172.21.15.114:0/1215159821 conn(0x7ff26c06e8f0 msgr2=0x7ff26c06ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.436+0000 7ff273aaf640 1 -- 172.21.15.114:0/1215159821 shutdown_connections 2024-05-27T07:59:19.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.439+0000 7ff273aaf640 1 -- 172.21.15.114:0/1215159821 wait complete. 2024-05-27T07:59:19.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.440+0000 7ff273aaf640 1 Processor -- start 2024-05-27T07:59:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.446+0000 7ff273aaf640 1 -- start start 2024-05-27T07:59:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.446+0000 7ff273aaf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff26c10a1f0 0x7ff26c1a9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.446+0000 7ff273aaf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff26c11a8e0 0x7ff26c1a9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.446+0000 7ff273aaf640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff26c11d240 0x7ff26c1adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.446+0000 7ff273aaf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c122c60 con 0x7ff26c10a1f0 2024-05-27T07:59:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.446+0000 7ff273aaf640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c122ae0 con 0x7ff26c11d240 2024-05-27T07:59:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.446+0000 7ff273aaf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c122de0 con 0x7ff26c11a8e0 2024-05-27T07:59:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.450+0000 7ff27104a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff26c11a8e0 0x7ff26c1a9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.450+0000 7ff27184b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff26c10a1f0 0x7ff26c1a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.450+0000 7ff27184b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff26c10a1f0 0x7ff26c1a9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44594/0 (socket says 172.21.15.114:44594) 2024-05-27T07:59:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.451+0000 7ff27184b640 1 -- 172.21.15.114:0/3192641170 learned_addr learned my addr 172.21.15.114:0/3192641170 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.451+0000 7ff27204c640 1 --2- 172.21.15.114:0/3192641170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff26c11d240 0x7ff26c1adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.455+0000 7ff27184b640 1 -- 172.21.15.114:0/3192641170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff26c11a8e0 msgr2=0x7ff26c1a9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.455+0000 7ff27184b640 1 --2- 172.21.15.114:0/3192641170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff26c11a8e0 0x7ff26c1a9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.455+0000 7ff27184b640 1 -- 172.21.15.114:0/3192641170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff26c11d240 msgr2=0x7ff26c1adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.455+0000 7ff27184b640 1 --2- 172.21.15.114:0/3192641170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff26c11d240 0x7ff26c1adea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.455+0000 7ff27184b640 1 -- 172.21.15.114:0/3192641170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff26c1ae4e0 con 0x7ff26c10a1f0 2024-05-27T07:59:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.455+0000 7ff27184b640 1 --2- 172.21.15.114:0/3192641170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff26c10a1f0 0x7ff26c1a9680 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7ff26000eab0 tx=0x7ff26000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.456+0000 7ff25effd640 1 -- 172.21.15.114:0/3192641170 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26000ce40 con 0x7ff26c10a1f0 2024-05-27T07:59:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.456+0000 7ff25effd640 1 -- 172.21.15.114:0/3192641170 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff260011920 con 0x7ff26c10a1f0 2024-05-27T07:59:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.456+0000 7ff273aaf640 1 -- 172.21.15.114:0/3192641170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff26c1ae7d0 con 0x7ff26c10a1f0 2024-05-27T07:59:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.456+0000 7ff25effd640 1 -- 172.21.15.114:0/3192641170 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260010a30 con 0x7ff26c10a1f0 2024-05-27T07:59:19.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.460+0000 7ff25effd640 1 -- 172.21.15.114:0/3192641170 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ff260010bd0 con 0x7ff26c10a1f0 2024-05-27T07:59:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.462+0000 7ff273aaf640 1 -- 172.21.15.114:0/3192641170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff26c1120d0 con 0x7ff26c10a1f0 2024-05-27T07:59:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.462+0000 7ff25effd640 1 --2- 172.21.15.114:0/3192641170 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2580b2ae0 0x7ff2580b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.462+0000 7ff25effd640 1 -- 172.21.15.114:0/3192641170 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 9554+0+0 (secure 0 0 0) 0x7ff2600e6060 con 0x7ff26c10a1f0 2024-05-27T07:59:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.464+0000 7ff27104a640 1 --2- 172.21.15.114:0/3192641170 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2580b2ae0 0x7ff2580b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.470+0000 7ff27104a640 1 --2- 172.21.15.114:0/3192641170 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2580b2ae0 0x7ff2580b4ff0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff26c11c640 tx=0x7ff2680090e0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:19.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.471+0000 7ff5dfeb4700 1 -- 172.21.15.114:0/1457846692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d810b270 msgr2=0x7ff5d811d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.471+0000 7ff5dfeb4700 1 --2- 172.21.15.114:0/1457846692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d810b270 0x7ff5d811d3e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009990 tx=0x7ff5d4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.472+0000 7ff5dfeb4700 1 -- 172.21.15.114:0/1457846692 shutdown_connections 2024-05-27T07:59:19.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.472+0000 7ff5dfeb4700 1 --2- 172.21.15.114:0/1457846692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5d811d920 0x7ff5d811fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.472+0000 7ff5dfeb4700 1 --2- 172.21.15.114:0/1457846692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d810a920 0x7ff5d810ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.472+0000 7ff5dfeb4700 1 --2- 172.21.15.114:0/1457846692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d810b270 0x7ff5d811d3e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.472+0000 7ff5dfeb4700 1 -- 172.21.15.114:0/1457846692 >> 172.21.15.114:0/1457846692 conn(0x7ff5d806f4c0 msgr2=0x7ff5d806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:19.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.474+0000 7ff5dfeb4700 1 -- 172.21.15.114:0/1457846692 shutdown_connections 2024-05-27T07:59:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.479+0000 7ff27a967740 1 Processor -- start 2024-05-27T07:59:19.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.476+0000 7ff5dfeb4700 1 -- 172.21.15.114:0/1457846692 wait complete. 2024-05-27T07:59:19.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.477+0000 7ff5dfeb4700 1 Processor -- start 2024-05-27T07:59:19.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.477+0000 7ff5dfeb4700 1 -- start start 2024-05-27T07:59:19.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.477+0000 7ff5dfeb4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d810a920 0x7ff5d811bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.477+0000 7ff5dfeb4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5d810b270 0x7ff5d8114eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.477+0000 7ff5dfeb4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d811d920 0x7ff5d81154b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.477+0000 7ff5dfeb4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d81233a0 con 0x7ff5d810b270 2024-05-27T07:59:19.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.477+0000 7ff5dfeb4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d8123220 con 0x7ff5d811d920 2024-05-27T07:59:19.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.477+0000 7ff5dfeb4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d8123520 con 0x7ff5d810a920 2024-05-27T07:59:19.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.479+0000 7ff5de451700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d811d920 0x7ff5d81154b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.479+0000 7ff5ddc50700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d810a920 0x7ff5d811bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.480+0000 7ff5de451700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d811d920 0x7ff5d81154b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:53652/0 (socket says 172.21.15.114:53652) 2024-05-27T07:59:19.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.480+0000 7ff5de451700 1 -- 172.21.15.114:0/4001986427 learned_addr learned my addr 172.21.15.114:0/4001986427 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:19.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.480+0000 7ff5de451700 1 -- 172.21.15.114:0/4001986427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d810a920 msgr2=0x7ff5d811bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.480+0000 7ff5de451700 1 --2- 172.21.15.114:0/4001986427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d810a920 0x7ff5d811bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.480+0000 7ff5de451700 1 -- 172.21.15.114:0/4001986427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5d810b270 msgr2=0x7ff5d8114eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.480+0000 7ff5de451700 1 --2- 172.21.15.114:0/4001986427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5d810b270 0x7ff5d8114eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.480+0000 7ff5de451700 1 -- 172.21.15.114:0/4001986427 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d81198b0 con 0x7ff5d811d920 2024-05-27T07:59:19.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.481+0000 7ff27a967740 1 -- start start 2024-05-27T07:59:19.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.481+0000 7ff27a967740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c341b0 0x558698fd53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.481+0000 7ff27a967740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c0b2f0 0x558698fd5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.481+0000 7ff27a967740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698c134d0 0x558698fd5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.481+0000 7ff27a967740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558698f47c30 con 0x558698c0b2f0 2024-05-27T07:59:19.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.481+0000 7ff27a967740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558698f377d0 con 0x558698c134d0 2024-05-27T07:59:19.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.481+0000 7ff27a967740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558698f3b810 con 0x558698c341b0 2024-05-27T07:59:19.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.481+0000 7ff5de451700 1 --2- 172.21.15.114:0/4001986427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d811d920 0x7ff5d81154b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800b480 tx=0x7ff5c800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:19.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.481+0000 7ff5ceffd700 1 -- 172.21.15.114:0/4001986427 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8015020 con 0x7ff5d811d920 2024-05-27T07:59:19.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.481+0000 7ff5dfeb4700 1 -- 172.21.15.114:0/4001986427 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d8119ba0 con 0x7ff5d811d920 2024-05-27T07:59:19.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.481+0000 7ff5dfeb4700 1 -- 172.21.15.114:0/4001986427 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d811a0e0 con 0x7ff5d811d920 2024-05-27T07:59:19.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.481+0000 7ff5ceffd700 1 -- 172.21.15.114:0/4001986427 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5c8011da0 con 0x7ff5d811d920 2024-05-27T07:59:19.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.481+0000 7ff5ceffd700 1 -- 172.21.15.114:0/4001986427 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c801a400 con 0x7ff5d811d920 2024-05-27T07:59:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.483+0000 7ff27104a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c0b2f0 0x558698fd5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.483+0000 7ff27104a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c0b2f0 0x558698fd5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44626/0 (socket says 172.21.15.114:44626) 2024-05-27T07:59:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.483+0000 7ff27104a640 1 -- 172.21.15.114:0/2590119008 learned_addr learned my addr 172.21.15.114:0/2590119008 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.483+0000 7ff27184b640 1 --2- 172.21.15.114:0/2590119008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c341b0 0x558698fd53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.487+0000 7ff27104a640 1 -- 172.21.15.114:0/2590119008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c341b0 msgr2=0x558698fd53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.487+0000 7ff27104a640 1 --2- 172.21.15.114:0/2590119008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c341b0 0x558698fd53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.487+0000 7ff27104a640 1 -- 172.21.15.114:0/2590119008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698c134d0 msgr2=0x558698fd5e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:59:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.487+0000 7ff27104a640 1 --2- 172.21.15.114:0/2590119008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698c134d0 0x558698fd5e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.487+0000 7ff27104a640 1 -- 172.21.15.114:0/2590119008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558698c4c210 con 0x558698c0b2f0 2024-05-27T07:59:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.488+0000 7ff27104a640 1 --2- 172.21.15.114:0/2590119008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c0b2f0 0x558698fd5910 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7ff268009fd0 tx=0x7ff268011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834a24e2e450e32f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:19.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.488+0000 7ff5ceffd700 1 -- 172.21.15.114:0/4001986427 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ff5c8018020 con 0x7ff5d811d920 2024-05-27T07:59:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.489+0000 7ff257fff640 1 -- 172.21.15.114:0/2590119008 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26801b040 con 0x558698c0b2f0 2024-05-27T07:59:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.489+0000 7ff257fff640 1 -- 172.21.15.114:0/2590119008 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff268024070 con 0x558698c0b2f0 2024-05-27T07:59:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.490+0000 7ff27a967740 1 -- 172.21.15.114:0/2590119008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c0b2f0 msgr2=0x558698fd5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.490+0000 7ff27a967740 1 --2- 172.21.15.114:0/2590119008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c0b2f0 0x558698fd5910 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7ff268009fd0 tx=0x7ff268011000 comp rx=0 tx=0).stop 2024-05-27T07:59:19.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.492+0000 7ff5ee8c9b80 1 Processor -- start 2024-05-27T07:59:19.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.492+0000 7ff5ee8c9b80 1 -- start start 2024-05-27T07:59:19.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.492+0000 7ff5ee8c9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f156e40 0x55e41f2dbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.492+0000 7ff5ee8c9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f2dc070 0x55e41f2d5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.492+0000 7ff5ee8c9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2d6120 0x55e41f2d6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.492+0000 7ff5ee8c9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e41f203d70 con 0x55e41f2dc070 2024-05-27T07:59:19.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.492+0000 7ff5ee8c9b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e41f1f3910 con 0x55e41f2d6120 2024-05-27T07:59:19.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.492+0000 7ff5ee8c9b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e41f1f7950 con 0x55e41f156e40 2024-05-27T07:59:19.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.493+0000 7ff5ceffd700 1 --2- 172.21.15.114:0/4001986427 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff5c40b2b20 0x7ff5c40b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.493+0000 7ff5ddc50700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f2dc070 0x55e41f2d5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.493+0000 7ff5ddc50700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f2dc070 0x55e41f2d5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44642/0 (socket says 172.21.15.114:44642) 2024-05-27T07:59:19.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.493+0000 7ff5ddc50700 1 -- 172.21.15.114:0/1427947586 learned_addr learned my addr 172.21.15.114:0/1427947586 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.497+0000 7ff27a967740 1 -- 172.21.15.114:0/2590119008 shutdown_connections 2024-05-27T07:59:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.497+0000 7ff27a967740 1 --2- 172.21.15.114:0/2590119008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698c134d0 0x558698fd5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.497+0000 7ff27a967740 1 --2- 172.21.15.114:0/2590119008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c0b2f0 0x558698fd5910 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.497+0000 7ff27a967740 1 --2- 172.21.15.114:0/2590119008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c341b0 0x558698fd53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.497+0000 7ff27a967740 1 -- 172.21.15.114:0/2590119008 >> 172.21.15.114:0/2590119008 conn(0x558698c28300 msgr2=0x558698fd1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:19.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.494+0000 7ff5ceffd700 1 -- 172.21.15.114:0/4001986427 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 9554+0+0 (secure 0 0 0) 0x7ff5c80e3110 con 0x7ff5d811d920 2024-05-27T07:59:19.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.495+0000 7ff5dd44f700 1 --2- 172.21.15.114:0/1427947586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2d6120 0x55e41f2d6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.496+0000 7ff5de451700 1 --2- 172.21.15.114:0/4001986427 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff5c40b2b20 0x7ff5c40b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.496+0000 7ff5ddc50700 1 -- 172.21.15.114:0/1427947586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f156e40 msgr2=0x55e41f2dbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:19.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.496+0000 7ff5ddc50700 1 --2- 172.21.15.114:0/1427947586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f156e40 0x55e41f2dbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.496+0000 7ff5ddc50700 1 -- 172.21.15.114:0/1427947586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2d6120 msgr2=0x55e41f2d6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:19.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.496+0000 7ff5ddc50700 1 --2- 172.21.15.114:0/1427947586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2d6120 0x55e41f2d6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.496+0000 7ff5ddc50700 1 -- 172.21.15.114:0/1427947586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e41f19cfb0 con 0x55e41f2dc070 2024-05-27T07:59:19.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.497+0000 7ff5ddc50700 1 --2- 172.21.15.114:0/1427947586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f2dc070 0x55e41f2d5be0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0007550 tx=0x7ff5d0007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2b5ef82022fce8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:19.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.497+0000 7ff5c3fff700 1 -- 172.21.15.114:0/1427947586 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0018070 con 0x55e41f2dc070 2024-05-27T07:59:19.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.497+0000 7ff5c3fff700 1 -- 172.21.15.114:0/1427947586 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d0011bc0 con 0x55e41f2dc070 2024-05-27T07:59:19.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.497+0000 7ff5c3fff700 1 -- 172.21.15.114:0/1427947586 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0013be0 con 0x55e41f2dc070 2024-05-27T07:59:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.499+0000 7ff27a967740 1 -- 172.21.15.114:0/2590119008 shutdown_connections 2024-05-27T07:59:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.500+0000 7ff27a967740 1 -- 172.21.15.114:0/2590119008 wait complete. 2024-05-27T07:59:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.501+0000 7ff27a967740 1 -- start start 2024-05-27T07:59:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.501+0000 7ff27a967740 1 Processor -- start 2024-05-27T07:59:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.502+0000 7ff27a967740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c0b2f0 0x558698fe1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.502+0000 7ff27a967740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c134d0 0x558698fe1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.502+0000 7ff27a967740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698c341b0 0x558698fdf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.502+0000 7ff27a967740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558698c1d790 con 0x558698c134d0 2024-05-27T07:59:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.502+0000 7ff27a967740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558698ef3890 con 0x558698c341b0 2024-05-27T07:59:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.502+0000 7ff27a967740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558698fe2550 con 0x558698c0b2f0 2024-05-27T07:59:19.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.502+0000 7ff27a967740 10 client.0 _ll_register_callbacks cb 0x558698c4ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.504+0000 7ff27184b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c0b2f0 0x558698fe1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.504+0000 7ff27184b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c0b2f0 0x558698fe1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:52820/0 (socket says 172.21.15.114:52820) 2024-05-27T07:59:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.504+0000 7ff27184b640 1 -- 172.21.15.114:0/492289186 learned_addr learned my addr 172.21.15.114:0/492289186 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:19.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.500+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/1427947586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f2dc070 msgr2=0x55e41f2d5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.500+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/1427947586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f2dc070 0x55e41f2d5be0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0007550 tx=0x7ff5d0007860 comp rx=0 tx=0).stop 2024-05-27T07:59:19.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.504+0000 7ff5de451700 1 --2- 172.21.15.114:0/4001986427 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff5c40b2b20 0x7ff5c40b5030 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff5c80a6110 tx=0x7ff5c8004860 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:19.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.504+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/1427947586 shutdown_connections 2024-05-27T07:59:19.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.504+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/1427947586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2d6120 0x55e41f2d6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.504+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/1427947586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f156e40 0x55e41f2dbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.504+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/1427947586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f2dc070 0x55e41f2d5be0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.504+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/1427947586 >> 172.21.15.114:0/1427947586 conn(0x55e41f159770 msgr2=0x55e41f2d0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.504+0000 7ff27104a640 1 --2- 172.21.15.114:0/492289186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c134d0 0x558698fe1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.508+0000 7ff27104a640 1 -- 172.21.15.114:0/492289186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c0b2f0 msgr2=0x558698fe1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.508+0000 7ff27104a640 1 --2- 172.21.15.114:0/492289186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c0b2f0 0x558698fe1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.508+0000 7ff27104a640 1 -- 172.21.15.114:0/492289186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698c341b0 msgr2=0x558698fdf9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:59:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.508+0000 7ff27104a640 1 --2- 172.21.15.114:0/492289186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698c341b0 0x558698fdf9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.508+0000 7ff27104a640 1 -- 172.21.15.114:0/492289186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558698d18620 con 0x558698c134d0 2024-05-27T07:59:19.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.507+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/1427947586 shutdown_connections 2024-05-27T07:59:19.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.508+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/1427947586 wait complete. 2024-05-27T07:59:19.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.508+0000 7ff5ee8c9b80 1 -- start start 2024-05-27T07:59:19.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.508+0000 7ff5ee8c9b80 1 Processor -- start 2024-05-27T07:59:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.510+0000 7ff27104a640 1 --2- 172.21.15.114:0/492289186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c134d0 0x558698fe1a60 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x558698fd6780 tx=0x7ff268013e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=977bef9eb57407e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.510+0000 7ff2557fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:59:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.510+0000 7ff2557fa640 1 -- 172.21.15.114:0/492289186 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff268004450 con 0x558698c134d0 2024-05-27T07:59:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.511+0000 7ff27a967740 1 -- 172.21.15.114:0/492289186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558698f49b80 con 0x558698c134d0 2024-05-27T07:59:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.511+0000 7ff2557fa640 1 -- 172.21.15.114:0/492289186 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9554+0+0 (secure 0 0 0) 0x7ff268013450 con 0x558698c134d0 2024-05-27T07:59:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.511+0000 7ff2557fa640 1 -- 172.21.15.114:0/492289186 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7ff268039050 con 0x558698c134d0 2024-05-27T07:59:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.511+0000 7ff27a967740 10 client.19734 fetch_fsmap learned FSMap version 110 2024-05-27T07:59:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.511+0000 7ff27a967740 1 -- 172.21.15.114:0/492289186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=4+}) -- 0x558698feea00 con 0x558698c134d0 2024-05-27T07:59:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.512+0000 7ff2557fa640 1 -- 172.21.15.114:0/492289186 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558698f49b80 con 0x558698c134d0 2024-05-27T07:59:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.512+0000 7ff2557fa640 1 -- 172.21.15.114:0/492289186 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 110) ==== 4095+0+0 (secure 0 0 0) 0x7ff26801e030 con 0x558698c134d0 2024-05-27T07:59:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.512+0000 7ff27a967740 10 client.19734 fetch_fsmap finished waiting for FSMap version 110 2024-05-27T07:59:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.512+0000 7ff27a967740 10 client.19734 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.512+0000 7ff27a967740 20 client.19734 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.512+0000 7ff27a967740 1 --2- 172.21.15.114:0/492289186 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558698bf2d80 0x558698ff2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.512+0000 7ff27a967740 4 client.19734 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:59:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.512+0000 7ff27a967740 1 -- 172.21.15.114:0/492289186 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558698f43360 con 0x558698bf2d80 2024-05-27T07:59:19.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.509+0000 7ff5ee8c9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f156e40 0x55e41f2df840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.509+0000 7ff5ee8c9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f2d6120 0x55e41f2dfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.509+0000 7ff5ee8c9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2dc070 0x55e41f2e02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.509+0000 7ff5ee8c9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e41f0f9fa0 con 0x55e41f156e40 2024-05-27T07:59:19.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.509+0000 7ff5ee8c9b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e41f15bda0 con 0x55e41f2dc070 2024-05-27T07:59:19.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.509+0000 7ff5ee8c9b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e41f2e2b60 con 0x55e41f2d6120 2024-05-27T07:59:19.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.509+0000 7ff5ee8c9b80 10 client.0 _ll_register_callbacks cb 0x55e41f18a5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:19.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.510+0000 7ff5ddc50700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f2d6120 0x55e41f2dfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.510+0000 7ff5ddc50700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f2d6120 0x55e41f2dfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:52828/0 (socket says 172.21.15.114:52828) 2024-05-27T07:59:19.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.510+0000 7ff5ddc50700 1 -- 172.21.15.114:0/2665234865 learned_addr learned my addr 172.21.15.114:0/2665234865 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:19.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.510+0000 7ff5ddc50700 1 -- 172.21.15.114:0/2665234865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2dc070 msgr2=0x55e41f2e02c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:59:19.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.510+0000 7ff5ddc50700 1 --2- 172.21.15.114:0/2665234865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2dc070 0x55e41f2e02c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.510+0000 7ff5ddc50700 1 -- 172.21.15.114:0/2665234865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f156e40 msgr2=0x55e41f2df840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:19.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.510+0000 7ff5ddc50700 1 --2- 172.21.15.114:0/2665234865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f156e40 0x55e41f2df840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.510+0000 7ff5ddc50700 1 -- 172.21.15.114:0/2665234865 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e41f1438d0 con 0x55e41f2d6120 2024-05-27T07:59:19.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.511+0000 7ff5ddc50700 1 --2- 172.21.15.114:0/2665234865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f2d6120 0x55e41f2dfd80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0007520 tx=0x7ff5d0013400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4fb492ae717f4db server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:19.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.511+0000 7ff5c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:59:19.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.514+0000 7ff5c17fa700 1 -- 172.21.15.114:0/2665234865 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0021020 con 0x55e41f2d6120 2024-05-27T07:59:19.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.514+0000 7ff5c17fa700 1 -- 172.21.15.114:0/2665234865 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9554+0+0 (secure 0 0 0) 0x7ff5d0013860 con 0x55e41f2d6120 2024-05-27T07:59:19.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.514+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/2665234865 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5d0011bc0 con 0x55e41f2d6120 2024-05-27T07:59:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.518+0000 7ff27184b640 1 --2- 172.21.15.114:0/492289186 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558698bf2d80 0x558698ff2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.522+0000 7ff5c17fa700 1 -- 172.21.15.114:0/2665234865 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7ff5d0018040 con 0x55e41f2d6120 2024-05-27T07:59:19.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.523+0000 7ff5ee8c9b80 10 client.29209 fetch_fsmap learned FSMap version 110 2024-05-27T07:59:19.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.523+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/2665234865 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=4+}) -- 0x55e41f1ad760 con 0x55e41f2d6120 2024-05-27T07:59:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.528+0000 7ff27184b640 1 --2- 172.21.15.114:0/492289186 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558698bf2d80 0x558698ff2260 crc :-1 s=READY pgs=508 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b04fc3d7c848e67b server_cookie=50e8094d1fd670d8 in_seq=0 out_seq=0 2024-05-27T07:59:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.528+0000 7ff2557fa640 10 client.19734 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:19.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.528+0000 7ff5c17fa700 1 -- 172.21.15.114:0/2665234865 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d0011bc0 con 0x55e41f2d6120 2024-05-27T07:59:19.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.528+0000 7ff5c17fa700 1 -- 172.21.15.114:0/2665234865 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 110) ==== 4095+0+0 (secure 0 0 0) 0x7ff5d001db40 con 0x55e41f2d6120 2024-05-27T07:59:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.530+0000 7ff2557fa640 1 -- 172.21.15.114:0/492289186 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558698f43360 con 0x558698bf2d80 2024-05-27T07:59:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.530+0000 7ff2557fa640 10 client.19734 handle_command_reply: tid=0 2024-05-27T07:59:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.530+0000 7ff27a967740 1 client.19734 shutdown 2024-05-27T07:59:19.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.531+0000 7ff5ee8c9b80 10 client.29209 fetch_fsmap finished waiting for FSMap version 110 2024-05-27T07:59:19.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.531+0000 7ff5ee8c9b80 10 client.29209 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:19.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.531+0000 7ff5ee8c9b80 20 client.29209 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:19.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.531+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/2665234865 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e41f2efd50 0x55e41f2f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.531+0000 7ff5ee8c9b80 4 client.29209 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:59:19.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.531+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/2665234865 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e41f205cc0 con 0x55e41f2efd50 2024-05-27T07:59:19.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.532+0000 7ff5dd44f700 1 --2- 172.21.15.114:0/2665234865 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e41f2efd50 0x55e41f2f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.536+0000 7ff27a967740 1 -- 172.21.15.114:0/492289186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c134d0 msgr2=0x558698fe1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.536+0000 7ff27a967740 1 --2- 172.21.15.114:0/492289186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c134d0 0x558698fe1a60 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x558698fd6780 tx=0x7ff268013e90 comp rx=0 tx=0).stop 2024-05-27T07:59:19.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.539+0000 7ff27a967740 1 -- 172.21.15.114:0/492289186 shutdown_connections 2024-05-27T07:59:19.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.539+0000 7ff27a967740 1 --2- 172.21.15.114:0/492289186 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x558698bf2d80 0x558698ff2260 crc :-1 s=READY pgs=508 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.540 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.539+0000 7ff27a967740 1 --2- 172.21.15.114:0/492289186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698c341b0 0x558698fdf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.539+0000 7ff27a967740 1 --2- 172.21.15.114:0/492289186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c134d0 0x558698fe1a60 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.539+0000 7ff27a967740 1 --2- 172.21.15.114:0/492289186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c0b2f0 0x558698fe1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.539+0000 7ff27a967740 1 -- 172.21.15.114:0/492289186 >> 172.21.15.114:0/492289186 conn(0x558698c28300 msgr2=0x558698fd70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.539+0000 7ff2557fa640 0 client.19734 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:19.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.535+0000 7ff5dd44f700 1 --2- 172.21.15.114:0/2665234865 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e41f2efd50 0x55e41f2f21e0 crc :-1 s=READY pgs=509 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bd925d8ab2359f2 server_cookie=23b5e4e0e6b3b252 in_seq=0 out_seq=0 2024-05-27T07:59:19.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.535+0000 7ff5c17fa700 10 client.29209 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:19.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.540+0000 7ff5c17fa700 1 -- 172.21.15.114:0/2665234865 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e41f205cc0 con 0x55e41f2efd50 2024-05-27T07:59:19.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.540+0000 7ff5c17fa700 10 client.29209 handle_command_reply: tid=0 2024-05-27T07:59:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.545+0000 7ff27a967740 1 -- 172.21.15.114:0/492289186 shutdown_connections 2024-05-27T07:59:19.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.541+0000 7ff5ee8c9b80 1 client.29209 shutdown 2024-05-27T07:59:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.547+0000 7ff27a967740 1 -- 172.21.15.114:0/492289186 wait complete. 2024-05-27T07:59:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.547+0000 7ff27a967740 20 client.19734 trim_cache size 0 max 16384 2024-05-27T07:59:19.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.546+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/2665234865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f2d6120 msgr2=0x55e41f2dfd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.546+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/2665234865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f2d6120 0x55e41f2dfd80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0007520 tx=0x7ff5d0013400 comp rx=0 tx=0).stop 2024-05-27T07:59:19.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.547+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/2665234865 shutdown_connections 2024-05-27T07:59:19.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.547+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/2665234865 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e41f2efd50 0x55e41f2f21e0 crc :-1 s=READY pgs=509 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.547+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/2665234865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2dc070 0x55e41f2e02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.547+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/2665234865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f156e40 0x55e41f2df840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.547+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/2665234865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f2d6120 0x55e41f2dfd80 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.547+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/2665234865 >> 172.21.15.114:0/2665234865 conn(0x55e41f159770 msgr2=0x55e41f2d4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:19.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.551+0000 7ff5c17fa700 0 client.29209 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:19.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.551+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/2665234865 shutdown_connections 2024-05-27T07:59:19.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.556+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/2665234865 wait complete. 2024-05-27T07:59:19.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.560+0000 7ff5ee8c9b80 20 client.29209 trim_cache size 0 max 16384 2024-05-27T07:59:19.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.582+0000 7ff27a967740 1 Processor -- start 2024-05-27T07:59:19.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.583+0000 7ff27a967740 1 -- start start 2024-05-27T07:59:19.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.583+0000 7ff27a967740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c0b2f0 0x558698fd6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.583+0000 7ff27a967740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c134d0 0x558698fd7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.584+0000 7ff27184b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c0b2f0 0x558698fd6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.584+0000 7ff27184b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c0b2f0 0x558698fd6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:52830/0 (socket says 172.21.15.114:52830) 2024-05-27T07:59:19.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.584+0000 7ff27104a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c134d0 0x558698fd7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.584+0000 7ff27104a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c134d0 0x558698fd7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44670/0 (socket says 172.21.15.114:44670) 2024-05-27T07:59:19.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.584+0000 7ff27184b640 1 -- 172.21.15.114:0/2577437245 learned_addr learned my addr 172.21.15.114:0/2577437245 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:19.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.584+0000 7ff27a967740 1 --2- 172.21.15.114:0/2577437245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698c341b0 0x558698fd9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.586 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.584+0000 7ff27a967740 1 -- 172.21.15.114:0/2577437245 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c1c8890 con 0x558698c134d0 2024-05-27T07:59:19.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.584+0000 7ff27a967740 1 -- 172.21.15.114:0/2577437245 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c1c8f60 con 0x558698c341b0 2024-05-27T07:59:19.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.584+0000 7ff27a967740 1 -- 172.21.15.114:0/2577437245 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c1c7830 con 0x558698c0b2f0 2024-05-27T07:59:19.587 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.584+0000 7ff27204c640 1 --2- 172.21.15.114:0/2577437245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698c341b0 0x558698fd9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.588+0000 7ff27184b640 1 -- 172.21.15.114:0/2577437245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698c341b0 msgr2=0x558698fd9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.588+0000 7ff27184b640 1 --2- 172.21.15.114:0/2577437245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698c341b0 0x558698fd9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.588+0000 7ff27184b640 1 -- 172.21.15.114:0/2577437245 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c134d0 msgr2=0x558698fd7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.588+0000 7ff27184b640 1 --2- 172.21.15.114:0/2577437245 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c134d0 0x558698fd7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.588+0000 7ff27184b640 1 -- 172.21.15.114:0/2577437245 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558698fd2aa0 con 0x558698c0b2f0 2024-05-27T07:59:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.594+0000 7ff27184b640 1 --2- 172.21.15.114:0/2577437245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c0b2f0 0x558698fd6170 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff2600a9100 tx=0x7ff2600e4000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78dd3fcd279c5e20 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:19.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.596+0000 7ff2577fe640 1 -- 172.21.15.114:0/2577437245 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2600b1fd0 con 0x558698c0b2f0 2024-05-27T07:59:19.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.596+0000 7ff2577fe640 1 -- 172.21.15.114:0/2577437245 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2600c2030 con 0x558698c0b2f0 2024-05-27T07:59:19.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.596+0000 7ff2577fe640 1 -- 172.21.15.114:0/2577437245 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2600b1b40 con 0x558698c0b2f0 2024-05-27T07:59:19.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.597+0000 7ff27a967740 1 -- 172.21.15.114:0/2577437245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c0b2f0 msgr2=0x558698fd6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.597+0000 7ff27a967740 1 --2- 172.21.15.114:0/2577437245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c0b2f0 0x558698fd6170 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff2600a9100 tx=0x7ff2600e4000 comp rx=0 tx=0).stop 2024-05-27T07:59:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.597+0000 7ff27a967740 1 -- 172.21.15.114:0/2577437245 shutdown_connections 2024-05-27T07:59:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.597+0000 7ff27a967740 1 --2- 172.21.15.114:0/2577437245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698c341b0 0x558698fd9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.598+0000 7ff27a967740 1 --2- 172.21.15.114:0/2577437245 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c134d0 0x558698fd7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.598+0000 7ff27a967740 1 --2- 172.21.15.114:0/2577437245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c0b2f0 0x558698fd6170 secure :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff2600a9100 tx=0x7ff2600e4000 comp rx=0 tx=0).stop 2024-05-27T07:59:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.598+0000 7ff27a967740 1 -- 172.21.15.114:0/2577437245 >> 172.21.15.114:0/2577437245 conn(0x558698c28300 msgr2=0x558698fd7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:19.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.603+0000 7ff27a967740 1 -- 172.21.15.114:0/2577437245 shutdown_connections 2024-05-27T07:59:19.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.603+0000 7ff27a967740 1 -- 172.21.15.114:0/2577437245 wait complete. 2024-05-27T07:59:19.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.603+0000 7ff5ee8c9b80 1 Processor -- start 2024-05-27T07:59:19.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.603+0000 7ff5ee8c9b80 1 -- start start 2024-05-27T07:59:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.604+0000 7ff27a967740 1 -- start start 2024-05-27T07:59:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.604+0000 7ff27a967740 1 Processor -- start 2024-05-27T07:59:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.604+0000 7ff27a967740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c134d0 0x558698fe4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.605+0000 7ff27a967740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c341b0 0x558698fe4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.605+0000 7ff27a967740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698bc81c0 0x558699058900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.605+0000 7ff27a967740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c1c7200 con 0x558698c341b0 2024-05-27T07:59:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.605+0000 7ff27a967740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c10b560 con 0x558698bc81c0 2024-05-27T07:59:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.605+0000 7ff27a967740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558698f43360 con 0x558698c134d0 2024-05-27T07:59:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.605+0000 7ff27a967740 10 client.0 _ll_register_callbacks cb 0x7ff26800ea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.605+0000 7ff27204c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698bc81c0 0x558699058900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.605+0000 7ff27204c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698bc81c0 0x558699058900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:53702/0 (socket says 172.21.15.114:53702) 2024-05-27T07:59:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.605+0000 7ff27204c640 1 -- 172.21.15.114:0/3509085459 learned_addr learned my addr 172.21.15.114:0/3509085459 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.605+0000 7ff27184b640 1 --2- 172.21.15.114:0/3509085459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c134d0 0x558698fe4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.606+0000 7ff27204c640 1 -- 172.21.15.114:0/3509085459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c134d0 msgr2=0x558698fe4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.606+0000 7ff27204c640 1 --2- 172.21.15.114:0/3509085459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c134d0 0x558698fe4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.606+0000 7ff27204c640 1 -- 172.21.15.114:0/3509085459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c341b0 msgr2=0x558698fe4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.606+0000 7ff27204c640 1 --2- 172.21.15.114:0/3509085459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c341b0 0x558698fe4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.606+0000 7ff27204c640 1 -- 172.21.15.114:0/3509085459 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558698fe5640 con 0x558698bc81c0 2024-05-27T07:59:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.609+0000 7ff27204c640 1 --2- 172.21.15.114:0/3509085459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698bc81c0 0x558699058900 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff264009900 tx=0x7ff264004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b4845dc9ea508fe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:19.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.604+0000 7ff5ee8c9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f156e40 0x55e41f2d5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.604+0000 7ff5ee8c9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2d6120 0x55e41f2ebc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.604+0000 7ff5ee8c9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f2dc070 0x55e41f2cac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.604+0000 7ff5ee8c9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e41f2de9e0 con 0x55e41f156e40 2024-05-27T07:59:19.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.604+0000 7ff5ee8c9b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc00ec00 con 0x55e41f2d6120 2024-05-27T07:59:19.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.604+0000 7ff5ee8c9b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc00f2d0 con 0x55e41f2dc070 2024-05-27T07:59:19.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.606+0000 7ff5ddc50700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2d6120 0x55e41f2ebc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.606+0000 7ff5ddc50700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2d6120 0x55e41f2ebc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:53700/0 (socket says 172.21.15.114:53700) 2024-05-27T07:59:19.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.606+0000 7ff5ddc50700 1 -- 172.21.15.114:0/773371053 learned_addr learned my addr 172.21.15.114:0/773371053 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:19.614 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.610+0000 7ff2557fa640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:59:19.614 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.610+0000 7ff2557fa640 1 -- 172.21.15.114:0/3509085459 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff264035040 con 0x558698bc81c0 2024-05-27T07:59:19.615 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.610+0000 7ff27a967740 1 -- 172.21.15.114:0/3509085459 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2600c2030 con 0x558698bc81c0 2024-05-27T07:59:19.615 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.611+0000 7ff2557fa640 1 -- 172.21.15.114:0/3509085459 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9554+0+0 (secure 0 0 0) 0x7ff264005590 con 0x558698bc81c0 2024-05-27T07:59:19.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.608+0000 7ff5ddc50700 1 -- 172.21.15.114:0/773371053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f2dc070 msgr2=0x55e41f2cac00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:59:19.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.609+0000 7ff5ddc50700 1 --2- 172.21.15.114:0/773371053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f2dc070 0x55e41f2cac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.609+0000 7ff5ddc50700 1 -- 172.21.15.114:0/773371053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f156e40 msgr2=0x55e41f2d5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:19.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.609+0000 7ff5ddc50700 1 --2- 172.21.15.114:0/773371053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f156e40 0x55e41f2d5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.609+0000 7ff5ddc50700 1 -- 172.21.15.114:0/773371053 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e41f2df610 con 0x55e41f2d6120 2024-05-27T07:59:19.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.609+0000 7ff5ddc50700 1 --2- 172.21.15.114:0/773371053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2d6120 0x55e41f2ebc60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0028040 tx=0x7ff5d0027570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db6b91627880df88 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:19.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.611+0000 7ff2557fa640 1 -- 172.21.15.114:0/3509085459 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7ff2640046c0 con 0x558698bc81c0 2024-05-27T07:59:19.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.613+0000 7ff27a967740 10 client.29063 fetch_fsmap learned FSMap version 110 2024-05-27T07:59:19.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.613+0000 7ff27a967740 1 -- 172.21.15.114:0/3509085459 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=4+}) -- 0x558699059de0 con 0x558698bc81c0 2024-05-27T07:59:19.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.614+0000 7ff2557fa640 1 -- 172.21.15.114:0/3509085459 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2600c2030 con 0x558698bc81c0 2024-05-27T07:59:19.617 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.614+0000 7ff2557fa640 1 -- 172.21.15.114:0/3509085459 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 110) ==== 4095+0+0 (secure 0 0 0) 0x7ff264045470 con 0x558698bc81c0 2024-05-27T07:59:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.614+0000 7ff27a967740 10 client.29063 fetch_fsmap finished waiting for FSMap version 110 2024-05-27T07:59:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.614+0000 7ff27a967740 10 client.29063 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.614+0000 7ff27a967740 20 client.29063 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.614+0000 7ff27a967740 1 --2- 172.21.15.114:0/3509085459 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55869905a540 0x55869905c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.614+0000 7ff27a967740 4 client.29063 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:59:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.614+0000 7ff27a967740 1 -- 172.21.15.114:0/3509085459 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558698ebd5f0 con 0x55869905a540 2024-05-27T07:59:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.616+0000 7ff27184b640 1 --2- 172.21.15.114:0/3509085459 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55869905a540 0x55869905c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.616+0000 7ff5c37fe700 1 -- 172.21.15.114:0/773371053 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0021040 con 0x55e41f2d6120 2024-05-27T07:59:19.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.616+0000 7ff5c37fe700 1 -- 172.21.15.114:0/773371053 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d00083f0 con 0x55e41f2d6120 2024-05-27T07:59:19.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.616+0000 7ff5c37fe700 1 -- 172.21.15.114:0/773371053 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d00139a0 con 0x55e41f2d6120 2024-05-27T07:59:19.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.618+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/773371053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2d6120 msgr2=0x55e41f2ebc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.618+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/773371053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2d6120 0x55e41f2ebc60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0028040 tx=0x7ff5d0027570 comp rx=0 tx=0).stop 2024-05-27T07:59:19.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 77047 11.93 MB/sec execute 211 sec latency 1887.924 ms 2024-05-27T07:59:19.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.621+0000 7ff27184b640 1 --2- 172.21.15.114:0/3509085459 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55869905a540 0x55869905c9d0 crc :-1 s=READY pgs=510 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e87d968f7864661 server_cookie=324af14ae325e4d in_seq=0 out_seq=0 2024-05-27T07:59:19.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:19.622+0000 7ff2557fa640 10 client.29063 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:19.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.619+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/773371053 shutdown_connections 2024-05-27T07:59:19.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.619+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/773371053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f2dc070 0x55e41f2cac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.619+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/773371053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f156e40 0x55e41f2d5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.619+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/773371053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2d6120 0x55e41f2ebc60 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.619+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/773371053 >> 172.21.15.114:0/773371053 conn(0x55e41f159770 msgr2=0x55e41f2ca4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:19.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.626+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/773371053 shutdown_connections 2024-05-27T07:59:19.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.626+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/773371053 wait complete. 2024-05-27T07:59:19.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.627+0000 7ff5ee8c9b80 1 -- start start 2024-05-27T07:59:19.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.627+0000 7ff5ee8c9b80 1 Processor -- start 2024-05-27T07:59:19.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.631+0000 7ff5ee8c9b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f156e40 0x55e41f2e8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.631+0000 7ff5ee8c9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f2d6120 0x55e41f2e8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.631+0000 7ff5ee8c9b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2dc070 0x55e41f2f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.631+0000 7ff5ee8c9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc00ce40 con 0x55e41f2d6120 2024-05-27T07:59:19.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.631+0000 7ff5ee8c9b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc00db80 con 0x55e41f2dc070 2024-05-27T07:59:19.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.631+0000 7ff5ee8c9b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc00d4b0 con 0x55e41f156e40 2024-05-27T07:59:19.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.631+0000 7ff5ee8c9b80 10 client.0 _ll_register_callbacks cb 0x55e41f18a5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:19.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.632+0000 7ff5dd44f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f156e40 0x55e41f2e8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.632+0000 7ff5dd44f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2dc070 0x55e41f2f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.632+0000 7ff5dd44f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f156e40 0x55e41f2e8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:52850/0 (socket says 172.21.15.114:52850) 2024-05-27T07:59:19.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.632+0000 7ff5dd44f700 1 -- 172.21.15.114:0/4171721023 learned_addr learned my addr 172.21.15.114:0/4171721023 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:19.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.637+0000 7ff5dd44f700 1 -- 172.21.15.114:0/4171721023 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2dc070 msgr2=0x55e41f2f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:19.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.637+0000 7ff5dd44f700 1 --2- 172.21.15.114:0/4171721023 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2dc070 0x55e41f2f6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.637+0000 7ff5dd44f700 1 -- 172.21.15.114:0/4171721023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f2d6120 msgr2=0x55e41f2e8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:19.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.637+0000 7ff5dd44f700 1 --2- 172.21.15.114:0/4171721023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f2d6120 0x55e41f2e8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.637+0000 7ff5dd44f700 1 -- 172.21.15.114:0/4171721023 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e41f2f7470 con 0x55e41f156e40 2024-05-27T07:59:19.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.645+0000 7ff5dd44f700 1 --2- 172.21.15.114:0/4171721023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f156e40 0x55e41f2e8050 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009750 tx=0x7ff5d4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3927245f8cee2007 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:19.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.645+0000 7ff5c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:59:19.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.646+0000 7ff5c17fa700 1 -- 172.21.15.114:0/4171721023 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d40052b0 con 0x55e41f156e40 2024-05-27T07:59:19.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.647+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/4171721023 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5d00083f0 con 0x55e41f156e40 2024-05-27T07:59:19.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.647+0000 7ff5c17fa700 1 -- 172.21.15.114:0/4171721023 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9554+0+0 (secure 0 0 0) 0x7ff5d4005450 con 0x55e41f156e40 2024-05-27T07:59:19.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.648+0000 7ff5c17fa700 1 -- 172.21.15.114:0/4171721023 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7ff5d4034050 con 0x55e41f156e40 2024-05-27T07:59:19.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.648+0000 7ff5ee8c9b80 10 client.29218 fetch_fsmap learned FSMap version 110 2024-05-27T07:59:19.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.648+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/4171721023 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=4+}) -- 0x55e41f35a2a0 con 0x55e41f156e40 2024-05-27T07:59:19.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.650+0000 7ff5c17fa700 1 -- 172.21.15.114:0/4171721023 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d00083f0 con 0x55e41f156e40 2024-05-27T07:59:19.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.650+0000 7ff5c17fa700 1 -- 172.21.15.114:0/4171721023 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 110) ==== 4095+0+0 (secure 0 0 0) 0x7ff5d40244d0 con 0x55e41f156e40 2024-05-27T07:59:19.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.650+0000 7ff5ee8c9b80 10 client.29218 fetch_fsmap finished waiting for FSMap version 110 2024-05-27T07:59:19.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.650+0000 7ff5ee8c9b80 10 client.29218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:19.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.650+0000 7ff5ee8c9b80 20 client.29218 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:19.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.650+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/4171721023 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e41f35a990 0x55e41f35cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:19.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.650+0000 7ff5ee8c9b80 4 client.29218 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:59:19.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.650+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/4171721023 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e41f1ff4a0 con 0x55e41f35a990 2024-05-27T07:59:19.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.652+0000 7ff5ddc50700 1 --2- 172.21.15.114:0/4171721023 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e41f35a990 0x55e41f35cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:19.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.657+0000 7ff5ddc50700 1 --2- 172.21.15.114:0/4171721023 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e41f35a990 0x55e41f35cdb0 crc :-1 s=READY pgs=511 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6aa7d3ddb5ef4c2c server_cookie=da1d8ac6aa9162bf in_seq=0 out_seq=0 2024-05-27T07:59:19.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.657+0000 7ff5c17fa700 10 client.29218 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:19.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.664+0000 7ff5c17fa700 1 -- 172.21.15.114:0/4171721023 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e41f1ff4a0 con 0x55e41f35a990 2024-05-27T07:59:19.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.664+0000 7ff5c17fa700 10 client.29218 handle_command_reply: tid=0 2024-05-27T07:59:19.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.665+0000 7ff5ee8c9b80 1 client.29218 shutdown 2024-05-27T07:59:19.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.672+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/4171721023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f156e40 msgr2=0x55e41f2e8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.672+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/4171721023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f156e40 0x55e41f2e8050 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009750 tx=0x7ff5d4000bc0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.673+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/4171721023 shutdown_connections 2024-05-27T07:59:19.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.673+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/4171721023 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e41f35a990 0x55e41f35cdb0 crc :-1 s=READY pgs=511 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.673+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/4171721023 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e41f2dc070 0x55e41f2f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.673+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/4171721023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e41f156e40 0x55e41f2e8050 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.673+0000 7ff5ee8c9b80 1 --2- 172.21.15.114:0/4171721023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e41f2d6120 0x55e41f2e8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.673+0000 7ff5c17fa700 0 client.29218 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:19.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.673+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/4171721023 >> 172.21.15.114:0/4171721023 conn(0x55e41f159770 msgr2=0x55e41f2ddd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:19.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.677+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/4171721023 shutdown_connections 2024-05-27T07:59:19.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.677+0000 7ff5ee8c9b80 1 -- 172.21.15.114:0/4171721023 wait complete. 2024-05-27T07:59:19.679 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:59:19.679 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:59:19.679 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:59:19.679 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:59:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.677+0000 7ff5ee8c9b80 20 client.29218 trim_cache size 0 max 16384 2024-05-27T07:59:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.678+0000 7ff5dfeb4700 1 -- 172.21.15.114:0/4001986427 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff5c40b2b20 msgr2=0x7ff5c40b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.678+0000 7ff5dfeb4700 1 --2- 172.21.15.114:0/4001986427 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff5c40b2b20 0x7ff5c40b5030 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff5c80a6110 tx=0x7ff5c8004860 comp rx=0 tx=0).stop 2024-05-27T07:59:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.678+0000 7ff5dfeb4700 1 -- 172.21.15.114:0/4001986427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d811d920 msgr2=0x7ff5d81154b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.678+0000 7ff5dfeb4700 1 --2- 172.21.15.114:0/4001986427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d811d920 0x7ff5d81154b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800b480 tx=0x7ff5c800b840 comp rx=0 tx=0).stop 2024-05-27T07:59:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.678+0000 7ff5dfeb4700 1 -- 172.21.15.114:0/4001986427 shutdown_connections 2024-05-27T07:59:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.678+0000 7ff5dfeb4700 1 --2- 172.21.15.114:0/4001986427 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff5c40b2b20 0x7ff5c40b5030 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.678+0000 7ff5dfeb4700 1 --2- 172.21.15.114:0/4001986427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d811d920 0x7ff5d81154b0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.678+0000 7ff5dfeb4700 1 --2- 172.21.15.114:0/4001986427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d810a920 0x7ff5d811bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.678+0000 7ff5dfeb4700 1 --2- 172.21.15.114:0/4001986427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5d810b270 0x7ff5d8114eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:19.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.678+0000 7ff5dfeb4700 1 -- 172.21.15.114:0/4001986427 >> 172.21.15.114:0/4001986427 conn(0x7ff5d806f4c0 msgr2=0x7ff5d8072780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:19.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.680+0000 7ff5dfeb4700 1 -- 172.21.15.114:0/4001986427 shutdown_connections 2024-05-27T07:59:19.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:19.680+0000 7ff5dfeb4700 1 -- 172.21.15.114:0/4001986427 wait complete. 2024-05-27T07:59:20.153 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:19 smithi114 ceph-mon[32038]: pgmap v674: 129 pgs: 11 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 101 active+clean; 375 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 4.0 MiB/s wr, 277 op/s 2024-05-27T07:59:20.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:19 smithi163 ceph-mon[36451]: pgmap v674: 129 pgs: 11 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 101 active+clean; 375 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 4.0 MiB/s wr, 277 op/s 2024-05-27T07:59:20.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:19 smithi186 ceph-mon[36647]: pgmap v674: 129 pgs: 11 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 101 active+clean; 375 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 4.0 MiB/s wr, 277 op/s 2024-05-27T07:59:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.489+0000 7ff2557fa640 1 -- 172.21.15.114:0/3509085459 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x558698ebd5f0 con 0x55869905a540 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.489+0000 7ff2557fa640 10 client.29063 handle_command_reply: tid=0 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.489+0000 7ff27a967740 1 client.29063 shutdown 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.489+0000 7ff27a967740 1 -- 172.21.15.114:0/3509085459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698bc81c0 msgr2=0x558699058900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.489+0000 7ff27a967740 1 --2- 172.21.15.114:0/3509085459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698bc81c0 0x558699058900 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff264009900 tx=0x7ff264004610 comp rx=0 tx=0).stop 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.489+0000 7ff27a967740 1 -- 172.21.15.114:0/3509085459 shutdown_connections 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.489+0000 7ff27a967740 1 --2- 172.21.15.114:0/3509085459 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55869905a540 0x55869905c9d0 crc :-1 s=READY pgs=510 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.489+0000 7ff27a967740 1 --2- 172.21.15.114:0/3509085459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558698bc81c0 0x558699058900 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.489+0000 7ff27a967740 1 --2- 172.21.15.114:0/3509085459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558698c341b0 0x558698fe4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.489+0000 7ff27a967740 1 --2- 172.21.15.114:0/3509085459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558698c134d0 0x558698fe4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.489+0000 7ff27a967740 1 -- 172.21.15.114:0/3509085459 >> 172.21.15.114:0/3509085459 conn(0x558698c28300 msgr2=0x558698fde290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.489+0000 7ff2557fa640 0 client.29063 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.490+0000 7ff27a967740 1 -- 172.21.15.114:0/3509085459 shutdown_connections 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.490+0000 7ff27a967740 1 -- 172.21.15.114:0/3509085459 wait complete. 2024-05-27T07:59:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.490+0000 7ff27a967740 20 client.29063 trim_cache size 0 max 16384 2024-05-27T07:59:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:59:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:59:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:59:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.491+0000 7ff273aaf640 1 -- 172.21.15.114:0/3192641170 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2580b2ae0 msgr2=0x7ff2580b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.491+0000 7ff273aaf640 1 --2- 172.21.15.114:0/3192641170 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2580b2ae0 0x7ff2580b4ff0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff26c11c640 tx=0x7ff2680090e0 comp rx=0 tx=0).stop 2024-05-27T07:59:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.491+0000 7ff273aaf640 1 -- 172.21.15.114:0/3192641170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff26c10a1f0 msgr2=0x7ff26c1a9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.491+0000 7ff273aaf640 1 --2- 172.21.15.114:0/3192641170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff26c10a1f0 0x7ff26c1a9680 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7ff26000eab0 tx=0x7ff26000ef70 comp rx=0 tx=0).stop 2024-05-27T07:59:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.491+0000 7ff273aaf640 1 -- 172.21.15.114:0/3192641170 shutdown_connections 2024-05-27T07:59:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.491+0000 7ff273aaf640 1 --2- 172.21.15.114:0/3192641170 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff2580b2ae0 0x7ff2580b4ff0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.491+0000 7ff273aaf640 1 --2- 172.21.15.114:0/3192641170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff26c11d240 0x7ff26c1adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.491+0000 7ff273aaf640 1 --2- 172.21.15.114:0/3192641170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff26c11a8e0 0x7ff26c1a9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.491+0000 7ff273aaf640 1 --2- 172.21.15.114:0/3192641170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff26c10a1f0 0x7ff26c1a9680 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.491+0000 7ff273aaf640 1 -- 172.21.15.114:0/3192641170 >> 172.21.15.114:0/3192641170 conn(0x7ff26c06e8f0 msgr2=0x7ff26c11acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.491+0000 7ff273aaf640 1 -- 172.21.15.114:0/3192641170 shutdown_connections 2024-05-27T07:59:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:20.491+0000 7ff273aaf640 1 -- 172.21.15.114:0/3192641170 wait complete. 2024-05-27T07:59:20.525 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:59:20.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:59:20.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 77134 11.90 MB/sec execute 212 sec latency 1690.484 ms 2024-05-27T07:59:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:59:21.620 INFO:tasks.workunit.client.0.smithi114.stdout: 10 77298 11.90 MB/sec execute 213 sec latency 1710.591 ms 2024-05-27T07:59:22.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:21 smithi163 ceph-mon[36451]: pgmap v675: 129 pgs: 11 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 101 active+clean; 375 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 4.1 MiB/s wr, 286 op/s 2024-05-27T07:59:22.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:21 smithi186 ceph-mon[36647]: pgmap v675: 129 pgs: 11 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 101 active+clean; 375 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 4.1 MiB/s wr, 286 op/s 2024-05-27T07:59:22.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:21 smithi114 ceph-mon[32038]: pgmap v675: 129 pgs: 11 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 101 active+clean; 375 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 4.1 MiB/s wr, 286 op/s 2024-05-27T07:59:22.620 INFO:tasks.workunit.client.0.smithi114.stdout: 10 77495 11.88 MB/sec execute 214 sec latency 273.150 ms 2024-05-27T07:59:23.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 77751 11.86 MB/sec execute 215 sec latency 276.756 ms 2024-05-27T07:59:24.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:23 smithi163 ceph-mon[36451]: pgmap v676: 129 pgs: 9 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 103 active+clean; 371 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 338 op/s 2024-05-27T07:59:24.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:23 smithi186 ceph-mon[36647]: pgmap v676: 129 pgs: 9 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 103 active+clean; 371 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 338 op/s 2024-05-27T07:59:24.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:23 smithi114 ceph-mon[32038]: pgmap v676: 129 pgs: 9 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 103 active+clean; 371 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 338 op/s 2024-05-27T07:59:24.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 77943 11.83 MB/sec execute 216 sec latency 489.612 ms 2024-05-27T07:59:25.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 78255 11.88 MB/sec execute 217 sec latency 186.145 ms 2024-05-27T07:59:25.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.661+0000 7f4b28f1e700 1 -- 172.21.15.114:0/1227894478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b2410b010 msgr2=0x7f4b2410b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:25.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.661+0000 7f4b28f1e700 1 --2- 172.21.15.114:0/1227894478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b2410b010 0x7f4b2410b420 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c00b110 tx=0x7f4b1c00aa40 comp rx=0 tx=0).stop 2024-05-27T07:59:25.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.663+0000 7f4b28f1e700 1 -- 172.21.15.114:0/1227894478 shutdown_connections 2024-05-27T07:59:25.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.663+0000 7f4b28f1e700 1 --2- 172.21.15.114:0/1227894478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b24110290 0x7f4b24117bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:25.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.663+0000 7f4b28f1e700 1 --2- 172.21.15.114:0/1227894478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b2410b010 0x7f4b2410b420 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:25.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.663+0000 7f4b28f1e700 1 --2- 172.21.15.114:0/1227894478 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b2410f940 0x7f4b2410fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:25.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.663+0000 7f4b28f1e700 1 -- 172.21.15.114:0/1227894478 >> 172.21.15.114:0/1227894478 conn(0x7f4b2406f980 msgr2=0x7f4b2406fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:25.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.664+0000 7f4b28f1e700 1 -- 172.21.15.114:0/1227894478 shutdown_connections 2024-05-27T07:59:25.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.664+0000 7f4b28f1e700 1 -- 172.21.15.114:0/1227894478 wait complete. 2024-05-27T07:59:25.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.665+0000 7f4b28f1e700 1 Processor -- start 2024-05-27T07:59:25.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.665+0000 7f4b28f1e700 1 -- start start 2024-05-27T07:59:25.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.666+0000 7f4b28f1e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b2410b010 0x7f4b241af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:25.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.666+0000 7f4b28f1e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b2410f940 0x7f4b241afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:25.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.666+0000 7f4b28f1e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b24110290 0x7f4b241a9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:25.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.666+0000 7f4b28f1e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2411aa50 con 0x7f4b24110290 2024-05-27T07:59:25.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.666+0000 7f4b28f1e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2411a8d0 con 0x7f4b2410f940 2024-05-27T07:59:25.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.666+0000 7f4b28f1e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2411abd0 con 0x7f4b2410b010 2024-05-27T07:59:25.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.666+0000 7f4b22d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b24110290 0x7f4b241a9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:25.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.666+0000 7f4b22d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b24110290 0x7f4b241a9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47516/0 (socket says 172.21.15.114:47516) 2024-05-27T07:59:25.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.666+0000 7f4b22d9d700 1 -- 172.21.15.114:0/4191337281 learned_addr learned my addr 172.21.15.114:0/4191337281 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:25.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.666+0000 7f4b2259c700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b2410b010 0x7f4b241af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:25.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.666+0000 7f4b21d9b700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b2410f940 0x7f4b241afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:25.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.667+0000 7f4b22d9d700 1 -- 172.21.15.114:0/4191337281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b2410b010 msgr2=0x7f4b241af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:25.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.667+0000 7f4b22d9d700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b2410b010 0x7f4b241af860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:25.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.667+0000 7f4b22d9d700 1 -- 172.21.15.114:0/4191337281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b2410f940 msgr2=0x7f4b241afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:25.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.667+0000 7f4b22d9d700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b2410f940 0x7f4b241afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:25.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.667+0000 7f4b22d9d700 1 -- 172.21.15.114:0/4191337281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b241aa130 con 0x7f4b24110290 2024-05-27T07:59:25.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.667+0000 7f4b2259c700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b2410b010 0x7f4b241af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:25.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.667+0000 7f4b22d9d700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b24110290 0x7f4b241a9a80 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f4b18007b50 tx=0x7f4b18007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:25.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.667+0000 7f4b137fe700 1 -- 172.21.15.114:0/4191337281 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b18019030 con 0x7f4b24110290 2024-05-27T07:59:25.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.667+0000 7f4b28f1e700 1 -- 172.21.15.114:0/4191337281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b241aa420 con 0x7f4b24110290 2024-05-27T07:59:25.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.667+0000 7f4b137fe700 1 -- 172.21.15.114:0/4191337281 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b18004ce0 con 0x7f4b24110290 2024-05-27T07:59:25.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.667+0000 7f4b137fe700 1 -- 172.21.15.114:0/4191337281 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1800fa30 con 0x7f4b24110290 2024-05-27T07:59:25.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.668+0000 7f4b28f1e700 1 -- 172.21.15.114:0/4191337281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b241b66f0 con 0x7f4b24110290 2024-05-27T07:59:25.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.669+0000 7f4b28f1e700 1 -- 172.21.15.114:0/4191337281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b240512b0 con 0x7f4b24110290 2024-05-27T07:59:25.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.671+0000 7f4b137fe700 1 -- 172.21.15.114:0/4191337281 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f4b1801c070 con 0x7f4b24110290 2024-05-27T07:59:25.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.672+0000 7f4b137fe700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b0c0b2af0 0x7f4b0c0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:25.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.672+0000 7f4b2259c700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b0c0b2af0 0x7f4b0c0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:25.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.672+0000 7f4b137fe700 1 -- 172.21.15.114:0/4191337281 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 9554+0+0 (secure 0 0 0) 0x7f4b180e2ca0 con 0x7f4b24110290 2024-05-27T07:59:25.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.673+0000 7f4b2259c700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b0c0b2af0 0x7f4b0c0b5000 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4b241aabc0 tx=0x7f4b1c00a9d0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:25.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:25.676+0000 7f4b137fe700 1 -- 172.21.15.114:0/4191337281 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4b180aa210 con 0x7f4b24110290 2024-05-27T07:59:26.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.043+0000 7f4b28f1e700 1 -- 172.21.15.114:0/4191337281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b2402e5e0 con 0x7f4b24110290 2024-05-27T07:59:26.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.046+0000 7f4b137fe700 1 -- 172.21.15.114:0/4191337281 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6036 (secure 0 0 0) 0x7f4b180a9b10 con 0x7f4b24110290 2024-05-27T07:59:26.049 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:59:26.049 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":19494,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/584614487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":584614487},{"type":"v1","addr":"172.21.15.114:6841","nonce":584614487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":110,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:59:14.889673+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_29048":{"gid":29048,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6839/3476461298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3476461298},{"type":"v1","addr":"172.21.15.114:6839","nonce":3476461298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:59:26.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.057+0000 7f4b28f1e700 1 -- 172.21.15.114:0/4191337281 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b0c0b2af0 msgr2=0x7f4b0c0b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:26.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.057+0000 7f4b28f1e700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b0c0b2af0 0x7f4b0c0b5000 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4b241aabc0 tx=0x7f4b1c00a9d0 comp rx=0 tx=0).stop 2024-05-27T07:59:26.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.057+0000 7f4b28f1e700 1 -- 172.21.15.114:0/4191337281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b24110290 msgr2=0x7f4b241a9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:26.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.057+0000 7f4b28f1e700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b24110290 0x7f4b241a9a80 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f4b18007b50 tx=0x7f4b18007e60 comp rx=0 tx=0).stop 2024-05-27T07:59:26.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.057+0000 7f4b28f1e700 1 -- 172.21.15.114:0/4191337281 shutdown_connections 2024-05-27T07:59:26.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.057+0000 7f4b28f1e700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b24110290 0x7f4b241a9a80 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:26.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.057+0000 7f4b28f1e700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b2410b010 0x7f4b241af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:26.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.057+0000 7f4b28f1e700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4b0c0b2af0 0x7f4b0c0b5000 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:26.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.057+0000 7f4b28f1e700 1 --2- 172.21.15.114:0/4191337281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b2410f940 0x7f4b241afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:26.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.057+0000 7f4b28f1e700 1 -- 172.21.15.114:0/4191337281 >> 172.21.15.114:0/4191337281 conn(0x7f4b2406f980 msgr2=0x7f4b2410d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:26.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.058+0000 7f4b28f1e700 1 -- 172.21.15.114:0/4191337281 shutdown_connections 2024-05-27T07:59:26.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:26.058+0000 7f4b28f1e700 1 -- 172.21.15.114:0/4191337281 wait complete. 2024-05-27T07:59:26.061 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 110 2024-05-27T07:59:26.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:25 smithi163 ceph-mon[36451]: pgmap v677: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 362 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 394 op/s 2024-05-27T07:59:26.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:25 smithi186 ceph-mon[36647]: pgmap v677: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 362 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 394 op/s 2024-05-27T07:59:26.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:25 smithi114 ceph-mon[32038]: pgmap v677: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 362 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 394 op/s 2024-05-27T07:59:26.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 78619 11.93 MB/sec execute 218 sec latency 162.288 ms 2024-05-27T07:59:27.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:26 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/4191337281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:59:27.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:26 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/4191337281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:59:27.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:26 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/4191337281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:59:27.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:59:27.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 78915 11.96 MB/sec execute 219 sec latency 248.615 ms 2024-05-27T07:59:27.807 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:27 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:59:27.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:27 smithi114 systemd[1]: Stopped Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:59:27.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:27 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.332s CPU time. 2024-05-27T07:59:27.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:27 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Start request repeated too quickly. 2024-05-27T07:59:27.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:27 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:59:27.808 INFO:journalctl@ceph.mds.g.smithi114.stdout:May 27 07:59:27 smithi114 systemd[1]: Failed to start Ceph mds.g for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:59:27.821 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:59:28.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:27 smithi114 ceph-mon[32038]: pgmap v678: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 368 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 390 op/s 2024-05-27T07:59:28.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:27 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:59:28.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:27 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:59:28.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:27 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:59:28.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:27 smithi114 ceph-mon[32038]: pgmap v679: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 372 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.1 MiB/s wr, 389 op/s 2024-05-27T07:59:28.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:27 smithi114 ceph-mon[32038]: pgmap v680: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 372 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 389 op/s 2024-05-27T07:59:28.138 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:27 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:59:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:27 smithi163 ceph-mon[36451]: pgmap v678: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 368 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 390 op/s 2024-05-27T07:59:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:27 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:59:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:27 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:59:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:27 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:59:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:27 smithi163 ceph-mon[36451]: pgmap v679: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 372 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.1 MiB/s wr, 389 op/s 2024-05-27T07:59:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:27 smithi163 ceph-mon[36451]: pgmap v680: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 372 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 389 op/s 2024-05-27T07:59:28.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:27 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:59:28.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:27 smithi186 ceph-mon[36647]: pgmap v678: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 368 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 390 op/s 2024-05-27T07:59:28.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:27 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:59:28.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:27 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:59:28.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:27 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:59:28.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:27 smithi186 ceph-mon[36647]: pgmap v679: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 372 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.1 MiB/s wr, 389 op/s 2024-05-27T07:59:28.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:27 smithi186 ceph-mon[36647]: pgmap v680: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 372 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 389 op/s 2024-05-27T07:59:28.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:27 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T07:59:28.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 79214 11.98 MB/sec execute 220 sec latency 175.158 ms 2024-05-27T07:59:29.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 79357 11.96 MB/sec execute 221 sec latency 453.540 ms 2024-05-27T07:59:29.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.695+0000 7fa9526c8700 1 -- 172.21.15.114:0/1097474675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa94c1032a0 msgr2=0x7fa94c1036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:29.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.695+0000 7fa9526c8700 1 --2- 172.21.15.114:0/1097474675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa94c1032a0 0x7fa94c1036b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa948009990 tx=0x7fa948009ca0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.698+0000 7fa9526c8700 1 -- 172.21.15.114:0/1097474675 shutdown_connections 2024-05-27T07:59:29.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.698+0000 7fa9526c8700 1 --2- 172.21.15.114:0/1097474675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa94c108520 0x7fa94c10fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.698+0000 7fa9526c8700 1 --2- 172.21.15.114:0/1097474675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa94c1032a0 0x7fa94c1036b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.698+0000 7fa9526c8700 1 --2- 172.21.15.114:0/1097474675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa94c107bd0 0x7fa94c107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.698+0000 7fa9526c8700 1 -- 172.21.15.114:0/1097474675 >> 172.21.15.114:0/1097474675 conn(0x7fa94c078b70 msgr2=0x7fa94c078fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.700+0000 7fa9526c8700 1 -- 172.21.15.114:0/1097474675 shutdown_connections 2024-05-27T07:59:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.703+0000 7fa9526c8700 1 -- 172.21.15.114:0/1097474675 wait complete. 2024-05-27T07:59:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.704+0000 7fa9526c8700 1 Processor -- start 2024-05-27T07:59:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.704+0000 7fa9526c8700 1 -- start start 2024-05-27T07:59:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.705+0000 7fa9526c8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa94c1032a0 0x7fa94c1126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.705+0000 7fa9526c8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa94c107bd0 0x7fa94c112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.705+0000 7fa9526c8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa94c108520 0x7fa94c113120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.705+0000 7fa9526c8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1121c0 con 0x7fa94c107bd0 2024-05-27T07:59:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.705+0000 7fa9526c8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c112040 con 0x7fa94c1032a0 2024-05-27T07:59:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.705+0000 7fa9526c8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c112340 con 0x7fa94c108520 2024-05-27T07:59:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.705+0000 7fa950ec5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa94c107bd0 0x7fa94c112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.705+0000 7fa950ec5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa94c107bd0 0x7fa94c112be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47524/0 (socket says 172.21.15.114:47524) 2024-05-27T07:59:29.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.705+0000 7fa950ec5700 1 -- 172.21.15.114:0/581596422 learned_addr learned my addr 172.21.15.114:0/581596422 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:29.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.706+0000 7fa951ec7700 1 --2- 172.21.15.114:0/581596422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa94c108520 0x7fa94c113120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.707+0000 7fa951ec7700 1 -- 172.21.15.114:0/581596422 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa94c1032a0 msgr2=0x7fa94c1126a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:59:29.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.707+0000 7fa951ec7700 1 --2- 172.21.15.114:0/581596422 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa94c1032a0 0x7fa94c1126a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.707+0000 7fa951ec7700 1 -- 172.21.15.114:0/581596422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa94c107bd0 msgr2=0x7fa94c112be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:29.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.707+0000 7fa951ec7700 1 --2- 172.21.15.114:0/581596422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa94c107bd0 0x7fa94c112be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.707+0000 7fa951ec7700 1 -- 172.21.15.114:0/581596422 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c0744b0 con 0x7fa94c108520 2024-05-27T07:59:29.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.708+0000 7fa951ec7700 1 --2- 172.21.15.114:0/581596422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa94c108520 0x7fa94c113120 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa94400be20 tx=0x7fa94400bf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:29.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.710+0000 7fa9427fc700 1 -- 172.21.15.114:0/581596422 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944019070 con 0x7fa94c108520 2024-05-27T07:59:29.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.710+0000 7fa9427fc700 1 -- 172.21.15.114:0/581596422 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa94400cd10 con 0x7fa94c108520 2024-05-27T07:59:29.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.710+0000 7fa9427fc700 1 -- 172.21.15.114:0/581596422 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9440127a0 con 0x7fa94c108520 2024-05-27T07:59:29.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.710+0000 7fa9526c8700 1 -- 172.21.15.114:0/581596422 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c0747a0 con 0x7fa94c108520 2024-05-27T07:59:29.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.711+0000 7fa9526c8700 1 -- 172.21.15.114:0/581596422 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94c074c90 con 0x7fa94c108520 2024-05-27T07:59:29.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.716+0000 7fa96233fb80 1 Processor -- start 2024-05-27T07:59:29.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.718+0000 7fa9427fc700 1 -- 172.21.15.114:0/581596422 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa94401c070 con 0x7fa94c108520 2024-05-27T07:59:29.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.718+0000 7fa96233fb80 1 -- start start 2024-05-27T07:59:29.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.719+0000 7fa96233fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d122e40 0x55e73d2a1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.719+0000 7fa96233fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d2a1550 0x55e73d2a1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.719+0000 7fa96233fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2abde0 0x55e73d2ae200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.719+0000 7fa96233fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e73d1cfd70 con 0x55e73d2a1550 2024-05-27T07:59:29.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.719+0000 7fa96233fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e73d1bf910 con 0x55e73d122e40 2024-05-27T07:59:29.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.719+0000 7fa96233fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e73d1c3950 con 0x55e73d2abde0 2024-05-27T07:59:29.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.719+0000 7fa9516c6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d2a1550 0x55e73d2a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.719+0000 7fa9516c6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d2a1550 0x55e73d2a1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47536/0 (socket says 172.21.15.114:47536) 2024-05-27T07:59:29.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.719+0000 7fa950ec5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d122e40 0x55e73d2a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.719+0000 7fa9516c6700 1 -- 172.21.15.114:0/2329355985 learned_addr learned my addr 172.21.15.114:0/2329355985 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:29.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.721+0000 7fa950ec5700 1 --2- 172.21.15.114:0/2329355985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2abde0 0x55e73d2ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.723+0000 7fa9516c6700 1 -- 172.21.15.114:0/2329355985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2abde0 msgr2=0x55e73d2ae200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:29.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.723+0000 7fa9427fc700 1 --2- 172.21.15.114:0/581596422 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa9380b2aa0 0x7fa9380b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.724+0000 7fa9427fc700 1 -- 172.21.15.114:0/581596422 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 9554+0+0 (secure 0 0 0) 0x7fa9440e2450 con 0x7fa94c108520 2024-05-27T07:59:29.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.724+0000 7fa9516c6700 1 --2- 172.21.15.114:0/2329355985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2abde0 0x55e73d2ae200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.724+0000 7fa9516c6700 1 -- 172.21.15.114:0/2329355985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d122e40 msgr2=0x55e73d2a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:29.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.724+0000 7fa9516c6700 1 --2- 172.21.15.114:0/2329355985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d122e40 0x55e73d2a1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.724+0000 7fa9516c6700 1 -- 172.21.15.114:0/2329355985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e73d168fb0 con 0x55e73d2a1550 2024-05-27T07:59:29.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.724+0000 7fa950ec5700 1 --2- 172.21.15.114:0/2329355985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2abde0 0x55e73d2ae200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:59:29.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.725+0000 7fa951ec7700 1 --2- 172.21.15.114:0/581596422 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa9380b2aa0 0x7fa9380b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.725+0000 7fa951ec7700 1 --2- 172.21.15.114:0/581596422 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa9380b2aa0 0x7fa9380b4fb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa9440a5450 tx=0x7fa944015fc0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:29.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.729+0000 7fa9516c6700 1 --2- 172.21.15.114:0/2329355985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d2a1550 0x55e73d2a1960 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fa94800b5c0 tx=0x7fa948003810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c10e547b2362b312 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:29.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.730+0000 7fa9377fe700 1 -- 172.21.15.114:0/2329355985 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9480038f0 con 0x55e73d2a1550 2024-05-27T07:59:29.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.730+0000 7fa9377fe700 1 -- 172.21.15.114:0/2329355985 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa948005070 con 0x55e73d2a1550 2024-05-27T07:59:29.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.732+0000 7fa96233fb80 1 -- 172.21.15.114:0/2329355985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d2a1550 msgr2=0x55e73d2a1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:29.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.732+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2329355985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d2a1550 0x55e73d2a1960 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fa94800b5c0 tx=0x7fa948003810 comp rx=0 tx=0).stop 2024-05-27T07:59:29.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.733+0000 7fa96233fb80 1 -- 172.21.15.114:0/2329355985 shutdown_connections 2024-05-27T07:59:29.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.733+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2329355985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2abde0 0x55e73d2ae200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.733+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2329355985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d122e40 0x55e73d2a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.733+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2329355985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d2a1550 0x55e73d2a1960 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.733+0000 7fa96233fb80 1 -- 172.21.15.114:0/2329355985 >> 172.21.15.114:0/2329355985 conn(0x55e73d125770 msgr2=0x55e73d29d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:29.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.741+0000 7fa96233fb80 1 -- 172.21.15.114:0/2329355985 shutdown_connections 2024-05-27T07:59:29.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.746+0000 7fa96233fb80 1 -- 172.21.15.114:0/2329355985 wait complete. 2024-05-27T07:59:29.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.746+0000 7fa96233fb80 1 -- start start 2024-05-27T07:59:29.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.746+0000 7fa96233fb80 1 Processor -- start 2024-05-27T07:59:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.749+0000 7fa96233fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 0x55e73d2a8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.749+0000 7fa96233fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 0x55e73d2a91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.749+0000 7fa96233fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d2abde0 0x55e73d2a9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.749+0000 7fa96233fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e73d0c5fa0 con 0x55e73d122e40 2024-05-27T07:59:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.749+0000 7fa96233fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e73d127da0 con 0x55e73d2abde0 2024-05-27T07:59:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.749+0000 7fa96233fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e73d2aecf0 con 0x55e73d2a1550 2024-05-27T07:59:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.749+0000 7fa96233fb80 10 client.0 _ll_register_callbacks cb 0x55e73d1565b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.749+0000 7fa950ec5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 0x55e73d2a8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.749+0000 7fa950ec5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 0x55e73d2a8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47542/0 (socket says 172.21.15.114:47542) 2024-05-27T07:59:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.749+0000 7fa950ec5700 1 -- 172.21.15.114:0/2140613341 learned_addr learned my addr 172.21.15.114:0/2140613341 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.749+0000 7fa950ec5700 1 --2- 172.21.15.114:0/2140613341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d2abde0 0x55e73d2a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.750+0000 7fa950ec5700 1 -- 172.21.15.114:0/2140613341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 msgr2=0x55e73d2a91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:29.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.750+0000 7fa950ec5700 1 --2- 172.21.15.114:0/2140613341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 0x55e73d2a91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.750+0000 7fa950ec5700 1 -- 172.21.15.114:0/2140613341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d2abde0 msgr2=0x55e73d2a9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:29.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.750+0000 7fa950ec5700 1 --2- 172.21.15.114:0/2140613341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d2abde0 0x55e73d2a9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.750+0000 7fa950ec5700 1 -- 172.21.15.114:0/2140613341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e73d10f8d0 con 0x55e73d122e40 2024-05-27T07:59:29.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.750+0000 7fa950ec5700 1 --2- 172.21.15.114:0/2140613341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 0x55e73d2a8cb0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fa93c018e90 tx=0x7fa93c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac745e99660feb4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:29.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.750+0000 7fa934ff9700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:59:29.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.751+0000 7fa934ff9700 1 -- 172.21.15.114:0/2140613341 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c03f020 con 0x55e73d122e40 2024-05-27T07:59:29.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.751+0000 7fa96233fb80 1 -- 172.21.15.114:0/2140613341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa948005070 con 0x55e73d122e40 2024-05-27T07:59:29.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.751+0000 7fa934ff9700 1 -- 172.21.15.114:0/2140613341 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(162..162 src has 1..162) ==== 9554+0+0 (secure 0 0 0) 0x7fa93c016bc0 con 0x55e73d122e40 2024-05-27T07:59:29.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.752+0000 7fa934ff9700 1 -- 172.21.15.114:0/2140613341 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7fa93c04a050 con 0x55e73d122e40 2024-05-27T07:59:29.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.755+0000 7fa96233fb80 10 client.19752 fetch_fsmap learned FSMap version 110 2024-05-27T07:59:29.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.755+0000 7fa96233fb80 1 -- 172.21.15.114:0/2140613341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=4+}) -- 0x55e73d179760 con 0x55e73d122e40 2024-05-27T07:59:29.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.756+0000 7fa934ff9700 1 -- 172.21.15.114:0/2140613341 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa948005070 con 0x55e73d122e40 2024-05-27T07:59:29.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.756+0000 7fa934ff9700 1 -- 172.21.15.114:0/2140613341 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 110) ==== 4095+0+0 (secure 0 0 0) 0x7fa93c011030 con 0x55e73d122e40 2024-05-27T07:59:29.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.756+0000 7fa96233fb80 10 client.19752 fetch_fsmap finished waiting for FSMap version 110 2024-05-27T07:59:29.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.756+0000 7fa96233fb80 10 client.19752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:29.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.756+0000 7fa96233fb80 20 client.19752 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:29.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.756+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2140613341 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e73d2bbfe0 0x55e73d2be490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.756+0000 7fa96233fb80 4 client.19752 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:59:29.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.756+0000 7fa96233fb80 1 -- 172.21.15.114:0/2140613341 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e73d1d1cc0 con 0x55e73d2bbfe0 2024-05-27T07:59:29.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.760+0000 7fa9516c6700 1 --2- 172.21.15.114:0/2140613341 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e73d2bbfe0 0x55e73d2be490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.768+0000 7fa9516c6700 1 --2- 172.21.15.114:0/2140613341 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e73d2bbfe0 0x55e73d2be490 crc :-1 s=READY pgs=512 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db090220ecb4f1b4 server_cookie=d8eedbc56f850699 in_seq=0 out_seq=0 2024-05-27T07:59:29.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.769+0000 7fa934ff9700 10 client.19752 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:29.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.772+0000 7fa934ff9700 1 -- 172.21.15.114:0/2140613341 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e73d1d1cc0 con 0x55e73d2bbfe0 2024-05-27T07:59:29.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.772+0000 7fa934ff9700 10 client.19752 handle_command_reply: tid=0 2024-05-27T07:59:29.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.772+0000 7fa96233fb80 1 client.19752 shutdown 2024-05-27T07:59:29.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.774+0000 7fa96233fb80 1 -- 172.21.15.114:0/2140613341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 msgr2=0x55e73d2a8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:29.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.774+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2140613341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 0x55e73d2a8cb0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fa93c018e90 tx=0x7fa93c008040 comp rx=0 tx=0).stop 2024-05-27T07:59:29.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.775+0000 7fa96233fb80 1 -- 172.21.15.114:0/2140613341 shutdown_connections 2024-05-27T07:59:29.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.776+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2140613341 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e73d2bbfe0 0x55e73d2be490 crc :-1 s=READY pgs=512 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.776+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2140613341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d2abde0 0x55e73d2a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.776+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2140613341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 0x55e73d2a8cb0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.776+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2140613341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 0x55e73d2a91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.776+0000 7fa934ff9700 0 client.19752 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:29.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.776+0000 7fa96233fb80 1 -- 172.21.15.114:0/2140613341 >> 172.21.15.114:0/2140613341 conn(0x55e73d125770 msgr2=0x55e73d29b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:29.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.776+0000 7fa96233fb80 1 -- 172.21.15.114:0/2140613341 shutdown_connections 2024-05-27T07:59:29.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.776+0000 7fa96233fb80 1 -- 172.21.15.114:0/2140613341 wait complete. 2024-05-27T07:59:29.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.776+0000 7fa96233fb80 20 client.19752 trim_cache size 0 max 16384 2024-05-27T07:59:29.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.805+0000 7fa96233fb80 1 Processor -- start 2024-05-27T07:59:29.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.805+0000 7fa96233fb80 1 -- start start 2024-05-27T07:59:29.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa96233fb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d122e40 0x55e73d2b7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa96233fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 0x55e73d2c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa96233fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d2abde0 0x55e73d2c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa96233fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa93000ec00 con 0x55e73d2abde0 2024-05-27T07:59:29.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa96233fb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa93000f2d0 con 0x55e73d122e40 2024-05-27T07:59:29.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa96233fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa93000db80 con 0x55e73d2a1550 2024-05-27T07:59:29.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa9516c6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 0x55e73d2c3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa9516c6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 0x55e73d2c3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54206/0 (socket says 172.21.15.114:54206) 2024-05-27T07:59:29.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa9516c6700 1 -- 172.21.15.114:0/2049720666 learned_addr learned my addr 172.21.15.114:0/2049720666 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:29.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa950ec5700 1 --2- 172.21.15.114:0/2049720666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d2abde0 0x55e73d2c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa950ec5700 1 --2- 172.21.15.114:0/2049720666 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d122e40 0x55e73d2b7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa9516c6700 1 -- 172.21.15.114:0/2049720666 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d122e40 msgr2=0x55e73d2b7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:29.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa9516c6700 1 --2- 172.21.15.114:0/2049720666 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d122e40 0x55e73d2b7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa9516c6700 1 -- 172.21.15.114:0/2049720666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d2abde0 msgr2=0x55e73d2c41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:29.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa9516c6700 1 --2- 172.21.15.114:0/2049720666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d2abde0 0x55e73d2c41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa9516c6700 1 -- 172.21.15.114:0/2049720666 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e73d2abc10 con 0x55e73d2a1550 2024-05-27T07:59:29.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.806+0000 7fa9516c6700 1 --2- 172.21.15.114:0/2049720666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 0x55e73d2c3c90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa948005b40 tx=0x7fa948009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afb7bcf0d0ceb219 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:29.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.807+0000 7fa936ffd700 1 -- 172.21.15.114:0/2049720666 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa948027040 con 0x55e73d2a1550 2024-05-27T07:59:29.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.807+0000 7fa936ffd700 1 -- 172.21.15.114:0/2049720666 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa948003da0 con 0x55e73d2a1550 2024-05-27T07:59:29.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.807+0000 7fa936ffd700 1 -- 172.21.15.114:0/2049720666 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa948005670 con 0x55e73d2a1550 2024-05-27T07:59:29.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.808+0000 7fa96233fb80 1 -- 172.21.15.114:0/2049720666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 msgr2=0x55e73d2c3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:29.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.808+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2049720666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 0x55e73d2c3c90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa948005b40 tx=0x7fa948009ee0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.808+0000 7fa96233fb80 1 -- 172.21.15.114:0/2049720666 shutdown_connections 2024-05-27T07:59:29.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.808+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2049720666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d2abde0 0x55e73d2c41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.808+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2049720666 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d122e40 0x55e73d2b7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.808+0000 7fa96233fb80 1 --2- 172.21.15.114:0/2049720666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 0x55e73d2c3c90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.808+0000 7fa96233fb80 1 -- 172.21.15.114:0/2049720666 >> 172.21.15.114:0/2049720666 conn(0x55e73d125770 msgr2=0x55e73d2a8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:29.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.808+0000 7fa96233fb80 1 -- 172.21.15.114:0/2049720666 shutdown_connections 2024-05-27T07:59:29.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.808+0000 7fa96233fb80 1 -- 172.21.15.114:0/2049720666 wait complete. 2024-05-27T07:59:29.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.809+0000 7fa96233fb80 1 -- start start 2024-05-27T07:59:29.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.809+0000 7fa96233fb80 1 Processor -- start 2024-05-27T07:59:29.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.809+0000 7fa96233fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 0x55e73d2b3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa96233fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 0x55e73d2b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa950ec5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 0x55e73d2b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa950ec5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 0x55e73d2b3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47560/0 (socket says 172.21.15.114:47560) 2024-05-27T07:59:29.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa950ec5700 1 -- 172.21.15.114:0/3718448353 learned_addr learned my addr 172.21.15.114:0/3718448353 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:29.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa96233fb80 1 --2- 172.21.15.114:0/3718448353 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d2abde0 0x55e73d31ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa96233fb80 1 -- 172.21.15.114:0/3718448353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa93000d4b0 con 0x55e73d122e40 2024-05-27T07:59:29.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa96233fb80 1 -- 172.21.15.114:0/3718448353 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa93000ce60 con 0x55e73d2abde0 2024-05-27T07:59:29.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa96233fb80 1 -- 172.21.15.114:0/3718448353 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e73d1d1cc0 con 0x55e73d2a1550 2024-05-27T07:59:29.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa9516c6700 1 --2- 172.21.15.114:0/3718448353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 0x55e73d2b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa950ec5700 1 -- 172.21.15.114:0/3718448353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 msgr2=0x55e73d2b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:29.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa950ec5700 1 --2- 172.21.15.114:0/3718448353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 0x55e73d2b3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa950ec5700 1 -- 172.21.15.114:0/3718448353 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d2abde0 msgr2=0x55e73d31ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa950ec5700 1 --2- 172.21.15.114:0/3718448353 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d2abde0 0x55e73d31ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa950ec5700 1 -- 172.21.15.114:0/3718448353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e73d31f1a0 con 0x55e73d122e40 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa96233fb80 10 client.0 _ll_register_callbacks cb 0x55e73d1565b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa96233fb80 1 -- 172.21.15.114:0/3718448353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa948003da0 con 0x55e73d122e40 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa950ec5700 1 --2- 172.21.15.114:0/3718448353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 0x55e73d2b3780 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00c240 tx=0x7fa93c00ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2694c4f54803636 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa934ff9700 10 client.19758 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa934ff9700 1 -- 172.21.15.114:0/3718448353 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c054020 con 0x55e73d122e40 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.810+0000 7fa934ff9700 1 -- 172.21.15.114:0/3718448353 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(162..162 src has 1..162) ==== 9554+0+0 (secure 0 0 0) 0x7fa93c045040 con 0x55e73d122e40 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.811+0000 7fa934ff9700 1 -- 172.21.15.114:0/3718448353 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7fa93c03f040 con 0x55e73d122e40 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.811+0000 7fa96233fb80 10 client.19758 fetch_fsmap learned FSMap version 110 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.811+0000 7fa96233fb80 1 -- 172.21.15.114:0/3718448353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({fsmap=110}) -- 0x55e73d326370 con 0x55e73d122e40 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.811+0000 7fa934ff9700 1 -- 172.21.15.114:0/3718448353 <== mon.0 v2:172.21.15.114:3300/0 4 ==== fsmap(e 110) ==== 4095+0+0 (secure 0 0 0) 0x7fa93c011040 con 0x55e73d122e40 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.811+0000 7fa96233fb80 10 client.19758 fetch_fsmap finished waiting for FSMap version 110 2024-05-27T07:59:29.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.811+0000 7fa96233fb80 10 client.19758 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:29.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.811+0000 7fa96233fb80 20 client.19758 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:29.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.811+0000 7fa96233fb80 1 --2- 172.21.15.114:0/3718448353 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e73d326980 0x55e73d328dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:29.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.811+0000 7fa96233fb80 4 client.19758 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:59:29.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.811+0000 7fa96233fb80 1 -- 172.21.15.114:0/3718448353 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e73d1cb4a0 con 0x55e73d326980 2024-05-27T07:59:29.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.811+0000 7fa9516c6700 1 --2- 172.21.15.114:0/3718448353 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e73d326980 0x55e73d328dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:29.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.812+0000 7fa934ff9700 10 client.19758 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:29.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.812+0000 7fa9516c6700 1 --2- 172.21.15.114:0/3718448353 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e73d326980 0x55e73d328dd0 crc :-1 s=READY pgs=513 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=deabdbab4f19b6f8 server_cookie=4fd04bec29110eaf in_seq=0 out_seq=0 2024-05-27T07:59:29.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa934ff9700 1 -- 172.21.15.114:0/3718448353 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e73d1cb4a0 con 0x55e73d326980 2024-05-27T07:59:29.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa934ff9700 10 client.19758 handle_command_reply: tid=0 2024-05-27T07:59:29.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa96233fb80 1 client.19758 shutdown 2024-05-27T07:59:29.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa96233fb80 1 -- 172.21.15.114:0/3718448353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 msgr2=0x55e73d2b3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:29.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa96233fb80 1 --2- 172.21.15.114:0/3718448353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 0x55e73d2b3780 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00c240 tx=0x7fa93c00ca70 comp rx=0 tx=0).stop 2024-05-27T07:59:29.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa96233fb80 1 -- 172.21.15.114:0/3718448353 shutdown_connections 2024-05-27T07:59:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa96233fb80 1 --2- 172.21.15.114:0/3718448353 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55e73d326980 0x55e73d328dd0 crc :-1 s=READY pgs=513 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa96233fb80 1 --2- 172.21.15.114:0/3718448353 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e73d2abde0 0x55e73d31ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa96233fb80 1 --2- 172.21.15.114:0/3718448353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e73d122e40 0x55e73d2b3780 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa934ff9700 0 client.19758 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa96233fb80 1 --2- 172.21.15.114:0/3718448353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e73d2a1550 0x55e73d2b3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa96233fb80 1 -- 172.21.15.114:0/3718448353 >> 172.21.15.114:0/3718448353 conn(0x55e73d125770 msgr2=0x55e73d2a8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa96233fb80 1 -- 172.21.15.114:0/3718448353 shutdown_connections 2024-05-27T07:59:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa96233fb80 1 -- 172.21.15.114:0/3718448353 wait complete. 2024-05-27T07:59:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.817+0000 7fa96233fb80 20 client.19758 trim_cache size 0 max 16384 2024-05-27T07:59:29.821 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:59:29.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.818+0000 7fa9526c8700 1 -- 172.21.15.114:0/581596422 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa9380b2aa0 msgr2=0x7fa9380b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:29.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.818+0000 7fa9526c8700 1 --2- 172.21.15.114:0/581596422 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa9380b2aa0 0x7fa9380b4fb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa9440a5450 tx=0x7fa944015fc0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.818+0000 7fa9526c8700 1 -- 172.21.15.114:0/581596422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa94c108520 msgr2=0x7fa94c113120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:29.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.818+0000 7fa9526c8700 1 --2- 172.21.15.114:0/581596422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa94c108520 0x7fa94c113120 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa94400be20 tx=0x7fa94400bf00 comp rx=0 tx=0).stop 2024-05-27T07:59:29.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.818+0000 7fa9526c8700 1 -- 172.21.15.114:0/581596422 shutdown_connections 2024-05-27T07:59:29.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.818+0000 7fa9526c8700 1 --2- 172.21.15.114:0/581596422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa94c108520 0x7fa94c113120 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.818+0000 7fa9526c8700 1 --2- 172.21.15.114:0/581596422 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa9380b2aa0 0x7fa9380b4fb0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.818+0000 7fa9526c8700 1 --2- 172.21.15.114:0/581596422 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa94c1032a0 0x7fa94c1126a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.818+0000 7fa9526c8700 1 --2- 172.21.15.114:0/581596422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa94c107bd0 0x7fa94c112be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:29.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.818+0000 7fa9526c8700 1 -- 172.21.15.114:0/581596422 >> 172.21.15.114:0/581596422 conn(0x7fa94c078b70 msgr2=0x7fa94c1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:29.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.818+0000 7fa9526c8700 1 -- 172.21.15.114:0/581596422 shutdown_connections 2024-05-27T07:59:29.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:29.818+0000 7fa9526c8700 1 -- 172.21.15.114:0/581596422 wait complete. 2024-05-27T07:59:30.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 79614 11.96 MB/sec execute 222 sec latency 217.139 ms 2024-05-27T07:59:30.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:30 smithi163 ceph-mon[36451]: osdmap e162: 12 total, 12 up, 12 in 2024-05-27T07:59:30.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:30 smithi163 ceph-mon[36451]: pgmap v682: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 376 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.3 MiB/s wr, 355 op/s 2024-05-27T07:59:30.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:30 smithi186 ceph-mon[36647]: osdmap e162: 12 total, 12 up, 12 in 2024-05-27T07:59:30.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:30 smithi186 ceph-mon[36647]: pgmap v682: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 376 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.3 MiB/s wr, 355 op/s 2024-05-27T07:59:30.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:30 smithi114 ceph-mon[32038]: osdmap e162: 12 total, 12 up, 12 in 2024-05-27T07:59:30.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:30 smithi114 ceph-mon[32038]: pgmap v682: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 376 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.3 MiB/s wr, 355 op/s 2024-05-27T07:59:31.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 79897 11.98 MB/sec execute 223 sec latency 182.635 ms 2024-05-27T07:59:32.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:31 smithi163 ceph-mon[36451]: pgmap v683: 129 pgs: 129 active+clean; 379 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.7 MiB/s wr, 295 op/s 2024-05-27T07:59:32.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:31 smithi186 ceph-mon[36647]: pgmap v683: 129 pgs: 129 active+clean; 379 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.7 MiB/s wr, 295 op/s 2024-05-27T07:59:32.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:31 smithi114 ceph-mon[32038]: pgmap v683: 129 pgs: 129 active+clean; 379 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.7 MiB/s wr, 295 op/s 2024-05-27T07:59:32.436 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:59:32.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:59:32.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 80198 11.97 MB/sec execute 224 sec latency 242.729 ms 2024-05-27T07:59:32.742 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:59:33.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 80571 12.02 MB/sec execute 225 sec latency 214.437 ms 2024-05-27T07:59:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.731+0000 7f0fb97ce640 1 -- 172.21.15.114:0/683333032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb4109030 msgr2=0x7f0fb4102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.731+0000 7f0fb97ce640 1 --2- 172.21.15.114:0/683333032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb4109030 0x7f0fb4102c70 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009900 tx=0x7f0fa802a400 comp rx=0 tx=0).stop 2024-05-27T07:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.735+0000 7f0fb97ce640 1 -- 172.21.15.114:0/683333032 shutdown_connections 2024-05-27T07:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.735+0000 7f0fb97ce640 1 --2- 172.21.15.114:0/683333032 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fb41032e0 0x7f0fb410f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.735+0000 7f0fb97ce640 1 --2- 172.21.15.114:0/683333032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb4109030 0x7f0fb4102c70 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.735+0000 7f0fb97ce640 1 --2- 172.21.15.114:0/683333032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb4108b90 0x7f0fb4102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.735+0000 7f0fb97ce640 1 -- 172.21.15.114:0/683333032 >> 172.21.15.114:0/683333032 conn(0x7f0fb40782d0 msgr2=0x7f0fb4078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.741+0000 7f0fb97ce640 1 -- 172.21.15.114:0/683333032 shutdown_connections 2024-05-27T07:59:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.742+0000 7f0fb97ce640 1 -- 172.21.15.114:0/683333032 wait complete. 2024-05-27T07:59:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.743+0000 7f0fb97ce640 1 Processor -- start 2024-05-27T07:59:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.744+0000 7f0fb97ce640 1 -- start start 2024-05-27T07:59:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.744+0000 7f0fb97ce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb41032e0 0x7f0fb419d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.744+0000 7f0fb97ce640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb4108b90 0x7f0fb419d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.744+0000 7f0fb97ce640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fb4109030 0x7f0fb4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.744+0000 7f0fb97ce640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb4111cc0 con 0x7f0fb41032e0 2024-05-27T07:59:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.744+0000 7f0fb97ce640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb4111b40 con 0x7f0fb4109030 2024-05-27T07:59:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.744+0000 7f0fb97ce640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb4111e40 con 0x7f0fb4108b90 2024-05-27T07:59:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.744+0000 7f0fb37fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fb4109030 0x7f0fb4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.746+0000 7f0fb37fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fb4109030 0x7f0fb4198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:55592/0 (socket says 172.21.15.114:55592) 2024-05-27T07:59:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.746+0000 7f0fb37fe640 1 -- 172.21.15.114:0/1072046194 learned_addr learned my addr 172.21.15.114:0/1072046194 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.748+0000 7f0fb27fc640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb4108b90 0x7f0fb419d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.748+0000 7f0fb2ffd640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb41032e0 0x7f0fb419d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.752+0000 7f0fb37fe640 1 -- 172.21.15.114:0/1072046194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb4108b90 msgr2=0x7f0fb419d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.752+0000 7f0fb37fe640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb4108b90 0x7f0fb419d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.752+0000 7f0fb37fe640 1 -- 172.21.15.114:0/1072046194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb41032e0 msgr2=0x7f0fb419d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.752+0000 7f0fb37fe640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb41032e0 0x7f0fb419d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.752+0000 7f0fb37fe640 1 -- 172.21.15.114:0/1072046194 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb4198d30 con 0x7f0fb4109030 2024-05-27T07:59:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.759+0000 7f0fb37fe640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fb4109030 0x7f0fb4198530 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400eb10 tx=0x7f0fa400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.760+0000 7f0f9bfff640 1 -- 172.21.15.114:0/1072046194 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4004b20 con 0x7f0fb4109030 2024-05-27T07:59:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.760+0000 7f0f9bfff640 1 -- 172.21.15.114:0/1072046194 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fa4004cc0 con 0x7f0fb4109030 2024-05-27T07:59:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.760+0000 7f0f9bfff640 1 -- 172.21.15.114:0/1072046194 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4010670 con 0x7f0fb4109030 2024-05-27T07:59:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.760+0000 7f0fb97ce640 1 -- 172.21.15.114:0/1072046194 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb4199020 con 0x7f0fb4109030 2024-05-27T07:59:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.760+0000 7f0fb97ce640 1 -- 172.21.15.114:0/1072046194 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb41a50a0 con 0x7f0fb4109030 2024-05-27T07:59:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.765+0000 7f0fc0683740 1 Processor -- start 2024-05-27T07:59:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.765+0000 7f0fc0683740 1 -- start start 2024-05-27T07:59:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.765+0000 7f0fc0683740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803ea1b0 0x55fd8078b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.765+0000 7f0fc0683740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c12f0 0x55fd8078b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.765+0000 7f0fc0683740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 0x55fd8078bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.765+0000 7f0fc0683740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fd806fdc30 con 0x55fd803c12f0 2024-05-27T07:59:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.765+0000 7f0fc0683740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fd806ed7d0 con 0x55fd803ea1b0 2024-05-27T07:59:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.765+0000 7f0fc0683740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fd806f1810 con 0x55fd803c94d0 2024-05-27T07:59:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.766+0000 7f0fb27fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803ea1b0 0x55fd8078b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.766+0000 7f0fb27fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803ea1b0 0x55fd8078b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:55602/0 (socket says 172.21.15.114:55602) 2024-05-27T07:59:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.766+0000 7f0fb27fc640 1 -- 172.21.15.114:0/4045107573 learned_addr learned my addr 172.21.15.114:0/4045107573 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.767+0000 7f0fb2ffd640 1 --2- 172.21.15.114:0/4045107573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c12f0 0x55fd8078b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.770+0000 7f0fb27fc640 1 -- 172.21.15.114:0/4045107573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 msgr2=0x55fd8078bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.770+0000 7f0fb27fc640 1 --2- 172.21.15.114:0/4045107573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 0x55fd8078bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.770+0000 7f0fb27fc640 1 -- 172.21.15.114:0/4045107573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c12f0 msgr2=0x55fd8078b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.770+0000 7f0fb27fc640 1 --2- 172.21.15.114:0/4045107573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c12f0 0x55fd8078b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.770+0000 7f0fb27fc640 1 -- 172.21.15.114:0/4045107573 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd80402210 con 0x55fd803ea1b0 2024-05-27T07:59:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.770+0000 7f0f9bfff640 1 -- 172.21.15.114:0/1072046194 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f0fa4005020 con 0x7f0fb4109030 2024-05-27T07:59:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.771+0000 7f0f9bfff640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0f9c0b2b60 0x7f0f9c0b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.771+0000 7f0f9bfff640 1 -- 172.21.15.114:0/1072046194 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 9554+0+0 (secure 0 0 0) 0x7f0fa40e2fc0 con 0x7f0fb4109030 2024-05-27T07:59:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.771+0000 7f0fb2ffd640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0f9c0b2b60 0x7f0f9c0b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.773+0000 7f0fb27fc640 1 --2- 172.21.15.114:0/4045107573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803ea1b0 0x55fd8078b260 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8005ab0 tx=0x7f0fa80095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76de5ece26d06c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.778+0000 7f0f997fa640 1 -- 172.21.15.114:0/4045107573 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8035030 con 0x55fd803ea1b0 2024-05-27T07:59:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.778+0000 7f0f997fa640 1 -- 172.21.15.114:0/4045107573 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fa8062030 con 0x55fd803ea1b0 2024-05-27T07:59:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.778+0000 7f0f997fa640 1 -- 172.21.15.114:0/4045107573 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8004e40 con 0x55fd803ea1b0 2024-05-27T07:59:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.780+0000 7f0fc0683740 1 -- 172.21.15.114:0/4045107573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803ea1b0 msgr2=0x55fd8078b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.780+0000 7f0fc0683740 1 --2- 172.21.15.114:0/4045107573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803ea1b0 0x55fd8078b260 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8005ab0 tx=0x7f0fa80095e0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.780+0000 7f0fb2ffd640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0f9c0b2b60 0x7f0f9c0b5070 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x55fd80788c30 tx=0x7f0fac005f90 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:33.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.781+0000 7f0fc0683740 1 -- 172.21.15.114:0/4045107573 shutdown_connections 2024-05-27T07:59:33.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.781+0000 7f0fc0683740 1 --2- 172.21.15.114:0/4045107573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 0x55fd8078bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.781+0000 7f0fc0683740 1 --2- 172.21.15.114:0/4045107573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c12f0 0x55fd8078b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.781+0000 7f0fc0683740 1 --2- 172.21.15.114:0/4045107573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803ea1b0 0x55fd8078b260 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.781+0000 7f0fc0683740 1 -- 172.21.15.114:0/4045107573 >> 172.21.15.114:0/4045107573 conn(0x55fd803de300 msgr2=0x55fd80787b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.785+0000 7f0fc0683740 1 -- 172.21.15.114:0/4045107573 shutdown_connections 2024-05-27T07:59:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.786+0000 7f0fc0683740 1 -- 172.21.15.114:0/4045107573 wait complete. 2024-05-27T07:59:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.786+0000 7f0fc0683740 1 -- start start 2024-05-27T07:59:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.787+0000 7f0fc0683740 1 Processor -- start 2024-05-27T07:59:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.787+0000 7f0fc0683740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803c12f0 0x55fd8079aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.787+0000 7f0fc0683740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c94d0 0x55fd8079afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.787+0000 7f0fc0683740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803ea1b0 0x55fd80795e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.787+0000 7f0fc0683740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fd803d3790 con 0x55fd803c94d0 2024-05-27T07:59:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.787+0000 7f0fc0683740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fd806a9890 con 0x55fd803c12f0 2024-05-27T07:59:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.787+0000 7f0fc0683740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fd8078e3d0 con 0x55fd803ea1b0 2024-05-27T07:59:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.787+0000 7f0fc0683740 10 client.0 _ll_register_callbacks cb 0x55fd80401a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.788+0000 7f0fb37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803ea1b0 0x55fd80795e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.788+0000 7f0fb37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803ea1b0 0x55fd80795e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:54252/0 (socket says 172.21.15.114:54252) 2024-05-27T07:59:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.788+0000 7f0fb37fe640 1 -- 172.21.15.114:0/4124347014 learned_addr learned my addr 172.21.15.114:0/4124347014 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.788+0000 7f0fb2ffd640 1 --2- 172.21.15.114:0/4124347014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803c12f0 0x55fd8079aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.788+0000 7f0fb27fc640 1 --2- 172.21.15.114:0/4124347014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c94d0 0x55fd8079afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.788+0000 7f0fb37fe640 1 -- 172.21.15.114:0/4124347014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803c12f0 msgr2=0x55fd8079aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.788+0000 7f0fb37fe640 1 --2- 172.21.15.114:0/4124347014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803c12f0 0x55fd8079aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.788+0000 7f0fb37fe640 1 -- 172.21.15.114:0/4124347014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c94d0 msgr2=0x55fd8079afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.788+0000 7f0fb37fe640 1 --2- 172.21.15.114:0/4124347014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c94d0 0x55fd8079afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.788+0000 7f0fb37fe640 1 -- 172.21.15.114:0/4124347014 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd804ce620 con 0x55fd803ea1b0 2024-05-27T07:59:33.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.792+0000 7f0fb37fe640 1 --2- 172.21.15.114:0/4124347014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803ea1b0 0x55fd80795e50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40a9b30 tx=0x7f0fa40a98b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=428125a4cf9959dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:33.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.792+0000 7f0f96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:59:33.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.792+0000 7f0f96ff5640 1 -- 172.21.15.114:0/4124347014 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa40a9e30 con 0x55fd803ea1b0 2024-05-27T07:59:33.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.792+0000 7f0fc0683740 1 -- 172.21.15.114:0/4124347014 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fa8062030 con 0x55fd803ea1b0 2024-05-27T07:59:33.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.792+0000 7f0f96ff5640 1 -- 172.21.15.114:0/4124347014 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(162..162 src has 1..162) ==== 9554+0+0 (secure 0 0 0) 0x7f0fa40aa2a0 con 0x55fd803ea1b0 2024-05-27T07:59:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.793+0000 7f0f96ff5640 1 -- 172.21.15.114:0/4124347014 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f0fa40ae1e0 con 0x55fd803ea1b0 2024-05-27T07:59:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.793+0000 7f0fc0683740 10 client.29245 fetch_fsmap learned FSMap version 110 2024-05-27T07:59:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.793+0000 7f0fc0683740 1 -- 172.21.15.114:0/4124347014 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=4+}) -- 0x55fd807a48c0 con 0x55fd803ea1b0 2024-05-27T07:59:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.795+0000 7f0f96ff5640 1 -- 172.21.15.114:0/4124347014 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fa8062030 con 0x55fd803ea1b0 2024-05-27T07:59:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.796+0000 7f0f96ff5640 1 -- 172.21.15.114:0/4124347014 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 110) ==== 4095+0+0 (secure 0 0 0) 0x7f0fa40a8b00 con 0x55fd803ea1b0 2024-05-27T07:59:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.796+0000 7f0fc0683740 10 client.29245 fetch_fsmap finished waiting for FSMap version 110 2024-05-27T07:59:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.796+0000 7f0fc0683740 10 client.29245 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.796+0000 7f0fc0683740 20 client.29245 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.796+0000 7f0fc0683740 1 --2- 172.21.15.114:0/4124347014 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fd803a8d80 0x55fd807a8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.796+0000 7f0fc0683740 4 client.29245 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:59:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.796+0000 7f0fc0683740 1 -- 172.21.15.114:0/4124347014 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd806ffb80 con 0x55fd803a8d80 2024-05-27T07:59:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.796+0000 7f0fb2ffd640 1 --2- 172.21.15.114:0/4124347014 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fd803a8d80 0x55fd807a8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.797+0000 7f0fb2ffd640 1 --2- 172.21.15.114:0/4124347014 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fd803a8d80 0x55fd807a8210 crc :-1 s=READY pgs=514 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=469c2e5e54e0d032 server_cookie=99215957d45e440e in_seq=0 out_seq=0 2024-05-27T07:59:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.797+0000 7f0f96ff5640 10 client.29245 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.799+0000 7f0f96ff5640 1 -- 172.21.15.114:0/4124347014 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fd806ffb80 con 0x55fd803a8d80 2024-05-27T07:59:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.799+0000 7f0f96ff5640 10 client.29245 handle_command_reply: tid=0 2024-05-27T07:59:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.799+0000 7f0fc0683740 1 client.29245 shutdown 2024-05-27T07:59:33.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.809+0000 7f0fc0683740 1 -- 172.21.15.114:0/4124347014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803ea1b0 msgr2=0x55fd80795e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:33.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.809+0000 7f0fc0683740 1 --2- 172.21.15.114:0/4124347014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803ea1b0 0x55fd80795e50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40a9b30 tx=0x7f0fa40a98b0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.809+0000 7f0fc0683740 1 -- 172.21.15.114:0/4124347014 shutdown_connections 2024-05-27T07:59:33.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.809+0000 7f0fc0683740 1 --2- 172.21.15.114:0/4124347014 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fd803a8d80 0x55fd807a8210 crc :-1 s=READY pgs=514 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.809+0000 7f0fc0683740 1 --2- 172.21.15.114:0/4124347014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803ea1b0 0x55fd80795e50 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.809+0000 7f0fc0683740 1 --2- 172.21.15.114:0/4124347014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c94d0 0x55fd8079afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.809+0000 7f0fc0683740 1 --2- 172.21.15.114:0/4124347014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803c12f0 0x55fd8079aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.809+0000 7f0fc0683740 1 -- 172.21.15.114:0/4124347014 >> 172.21.15.114:0/4124347014 conn(0x55fd803de300 msgr2=0x55fd8078bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:33.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.810+0000 7f0f96ff5640 0 client.29245 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.815+0000 7f0fc0683740 1 -- 172.21.15.114:0/4124347014 shutdown_connections 2024-05-27T07:59:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.816+0000 7f0fc0683740 1 -- 172.21.15.114:0/4124347014 wait complete. 2024-05-27T07:59:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.816+0000 7f0fc0683740 20 client.29245 trim_cache size 0 max 16384 2024-05-27T07:59:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.840+0000 7f0fc0683740 1 Processor -- start 2024-05-27T07:59:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.840+0000 7f0fc0683740 1 -- start start 2024-05-27T07:59:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.840+0000 7f0fc0683740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c12f0 0x55fd8079a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.840+0000 7f0fc0683740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 0x55fd8078b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.840+0000 7f0fc0683740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803ea1b0 0x55fd8078b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.840+0000 7f0fc0683740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb41a6590 con 0x55fd803c12f0 2024-05-27T07:59:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.840+0000 7f0fc0683740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb41a6c60 con 0x55fd803ea1b0 2024-05-27T07:59:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.840+0000 7f0fc0683740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb41a55b0 con 0x55fd803c94d0 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.840+0000 7f0fb2ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c12f0 0x55fd8079a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.840+0000 7f0fb2ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c12f0 0x55fd8079a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47616/0 (socket says 172.21.15.114:47616) 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.840+0000 7f0fb2ffd640 1 -- 172.21.15.114:0/2770961186 learned_addr learned my addr 172.21.15.114:0/2770961186 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.840+0000 7f0fb27fc640 1 --2- 172.21.15.114:0/2770961186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 0x55fd8078b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.841+0000 7f0fb37fe640 1 --2- 172.21.15.114:0/2770961186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803ea1b0 0x55fd8078b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.841+0000 7f0fb2ffd640 1 -- 172.21.15.114:0/2770961186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 msgr2=0x55fd8078b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.841+0000 7f0fb2ffd640 1 --2- 172.21.15.114:0/2770961186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 0x55fd8078b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.841+0000 7f0fb2ffd640 1 -- 172.21.15.114:0/2770961186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803ea1b0 msgr2=0x55fd8078b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.841+0000 7f0fb2ffd640 1 --2- 172.21.15.114:0/2770961186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803ea1b0 0x55fd8078b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.841+0000 7f0fb2ffd640 1 -- 172.21.15.114:0/2770961186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd803a8fb0 con 0x55fd803c12f0 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.841+0000 7f0fb2ffd640 1 --2- 172.21.15.114:0/2770961186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c12f0 0x55fd8079a440 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x55fd80788ec0 tx=0x7f0fac016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37ed270485d389 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.841+0000 7f0f98ff9640 1 -- 172.21.15.114:0/2770961186 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac004220 con 0x55fd803c12f0 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.841+0000 7f0f98ff9640 1 -- 172.21.15.114:0/2770961186 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fac0043c0 con 0x55fd803c12f0 2024-05-27T07:59:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.841+0000 7f0f98ff9640 1 -- 172.21.15.114:0/2770961186 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac003a80 con 0x55fd803c12f0 2024-05-27T07:59:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.842+0000 7f0fc0683740 1 -- 172.21.15.114:0/2770961186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c12f0 msgr2=0x55fd8079a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.842+0000 7f0fc0683740 1 --2- 172.21.15.114:0/2770961186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c12f0 0x55fd8079a440 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x55fd80788ec0 tx=0x7f0fac016000 comp rx=0 tx=0).stop 2024-05-27T07:59:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.842+0000 7f0fc0683740 1 -- 172.21.15.114:0/2770961186 shutdown_connections 2024-05-27T07:59:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.842+0000 7f0fc0683740 1 --2- 172.21.15.114:0/2770961186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803ea1b0 0x55fd8078b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.842+0000 7f0fc0683740 1 --2- 172.21.15.114:0/2770961186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 0x55fd8078b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.842+0000 7f0fc0683740 1 --2- 172.21.15.114:0/2770961186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803c12f0 0x55fd8079a440 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.842+0000 7f0fc0683740 1 -- 172.21.15.114:0/2770961186 >> 172.21.15.114:0/2770961186 conn(0x55fd803de300 msgr2=0x55fd80794720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.842+0000 7f0fc0683740 1 -- 172.21.15.114:0/2770961186 shutdown_connections 2024-05-27T07:59:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.842+0000 7f0fc0683740 1 -- 172.21.15.114:0/2770961186 wait complete. 2024-05-27T07:59:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.842+0000 7f0fc0683740 1 -- start start 2024-05-27T07:59:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fc0683740 1 Processor -- start 2024-05-27T07:59:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fc0683740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803c12f0 0x55fd807ae940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fc0683740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 0x55fd807a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fc0683740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803ea1b0 0x55fd807a9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fc0683740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb4074c00 con 0x55fd803ea1b0 2024-05-27T07:59:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fb37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803ea1b0 0x55fd807a9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fb2ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803c12f0 0x55fd807ae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fb27fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 0x55fd807a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fb37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803ea1b0 0x55fd807a9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47618/0 (socket says 172.21.15.114:47618) 2024-05-27T07:59:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fb37fe640 1 -- 172.21.15.114:0/1673813389 learned_addr learned my addr 172.21.15.114:0/1673813389 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fc0683740 1 -- 172.21.15.114:0/1673813389 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb406a230 con 0x55fd803c12f0 2024-05-27T07:59:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fc0683740 1 -- 172.21.15.114:0/1673813389 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fd80791a30 con 0x55fd803c94d0 2024-05-27T07:59:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fc0683740 10 client.0 _ll_register_callbacks cb 0x7f0fa4004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fb37fe640 1 -- 172.21.15.114:0/1673813389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 msgr2=0x55fd807a92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fb37fe640 1 --2- 172.21.15.114:0/1673813389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 0x55fd807a92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fb37fe640 1 -- 172.21.15.114:0/1673813389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803c12f0 msgr2=0x55fd807ae940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fb37fe640 1 --2- 172.21.15.114:0/1673813389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803c12f0 0x55fd807ae940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.843+0000 7f0fb37fe640 1 -- 172.21.15.114:0/1673813389 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd807a9e50 con 0x55fd803ea1b0 2024-05-27T07:59:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0fb37fe640 1 --2- 172.21.15.114:0/1673813389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803ea1b0 0x55fd807a9810 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40a8e70 tx=0x7f0fa4009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56c030376c714eb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0f96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:59:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0f96ff5640 1 -- 172.21.15.114:0/1673813389 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa40a9c00 con 0x55fd803ea1b0 2024-05-27T07:59:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0f96ff5640 1 -- 172.21.15.114:0/1673813389 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(162..162 src has 1..162) ==== 9554+0+0 (secure 0 0 0) 0x7f0fa40b2800 con 0x55fd803ea1b0 2024-05-27T07:59:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0fc0683740 1 -- 172.21.15.114:0/1673813389 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fac0043c0 con 0x55fd803ea1b0 2024-05-27T07:59:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0f96ff5640 1 -- 172.21.15.114:0/1673813389 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f0fa408c050 con 0x55fd803ea1b0 2024-05-27T07:59:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0fc0683740 10 client.19776 fetch_fsmap learned FSMap version 110 2024-05-27T07:59:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0fc0683740 1 -- 172.21.15.114:0/1673813389 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=4+}) -- 0x55fd8080ffa0 con 0x55fd803ea1b0 2024-05-27T07:59:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0f96ff5640 1 -- 172.21.15.114:0/1673813389 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fa40a9e30 con 0x55fd803ea1b0 2024-05-27T07:59:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0f96ff5640 1 -- 172.21.15.114:0/1673813389 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 110) ==== 4095+0+0 (secure 0 0 0) 0x7f0fa40c40e0 con 0x55fd803ea1b0 2024-05-27T07:59:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0fc0683740 10 client.19776 fetch_fsmap finished waiting for FSMap version 110 2024-05-27T07:59:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0fc0683740 10 client.19776 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0fc0683740 20 client.19776 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.844+0000 7f0fc0683740 1 --2- 172.21.15.114:0/1673813389 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fd8037e1c0 0x55fd80812480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.845+0000 7f0fc0683740 4 client.19776 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:59:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.845+0000 7f0fc0683740 1 -- 172.21.15.114:0/1673813389 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f0fac0043c0 con 0x55fd8037e1c0 2024-05-27T07:59:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.845+0000 7f0fb2ffd640 1 --2- 172.21.15.114:0/1673813389 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fd8037e1c0 0x55fd80812480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.845+0000 7f0f96ff5640 10 client.19776 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:33.845+0000 7f0fb2ffd640 1 --2- 172.21.15.114:0/1673813389 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fd8037e1c0 0x55fd80812480 crc :-1 s=READY pgs=515 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c47dff3fc60ca0e5 server_cookie=93e87ab8b80287bd in_seq=0 out_seq=0 2024-05-27T07:59:34.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:33 smithi163 ceph-mon[36451]: pgmap v684: 129 pgs: 129 active+clean; 392 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 335 op/s 2024-05-27T07:59:34.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:33 smithi186 ceph-mon[36647]: pgmap v684: 129 pgs: 129 active+clean; 392 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 335 op/s 2024-05-27T07:59:34.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:33 smithi114 ceph-mon[32038]: pgmap v684: 129 pgs: 129 active+clean; 392 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 335 op/s 2024-05-27T07:59:34.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 80638 11.97 MB/sec execute 226 sec latency 607.237 ms 2024-05-27T07:59:35.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:34 smithi114 ceph-mon[32038]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:59:35.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:34 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 2 up:standby 2024-05-27T07:59:35.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:34 smithi114 ceph-mon[32038]: osdmap e163: 12 total, 12 up, 12 in 2024-05-27T07:59:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.224+0000 7f0f96ff5640 1 -- 172.21.15.114:0/1673813389 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7f0fac0043c0 con 0x55fd8037e1c0 2024-05-27T07:59:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.224+0000 7f0f96ff5640 10 client.19776 handle_command_reply: tid=0 2024-05-27T07:59:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.224+0000 7f0fc0683740 1 client.19776 shutdown 2024-05-27T07:59:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.224+0000 7f0fc0683740 1 -- 172.21.15.114:0/1673813389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803ea1b0 msgr2=0x55fd807a9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.224+0000 7f0fc0683740 1 --2- 172.21.15.114:0/1673813389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803ea1b0 0x55fd807a9810 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40a8e70 tx=0x7f0fa4009200 comp rx=0 tx=0).stop 2024-05-27T07:59:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.224+0000 7f0fc0683740 1 -- 172.21.15.114:0/1673813389 shutdown_connections 2024-05-27T07:59:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.224+0000 7f0fc0683740 1 --2- 172.21.15.114:0/1673813389 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55fd8037e1c0 0x55fd80812480 crc :-1 s=READY pgs=515 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.224+0000 7f0fc0683740 1 --2- 172.21.15.114:0/1673813389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd803ea1b0 0x55fd807a9810 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.224+0000 7f0fc0683740 1 --2- 172.21.15.114:0/1673813389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fd803c94d0 0x55fd807a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.224+0000 7f0fc0683740 1 --2- 172.21.15.114:0/1673813389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd803c12f0 0x55fd807ae940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.225+0000 7f0fc0683740 1 -- 172.21.15.114:0/1673813389 >> 172.21.15.114:0/1673813389 conn(0x55fd803de300 msgr2=0x55fd80795390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.225+0000 7f0f96ff5640 0 client.19776 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.225+0000 7f0fc0683740 1 -- 172.21.15.114:0/1673813389 shutdown_connections 2024-05-27T07:59:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.225+0000 7f0fc0683740 1 -- 172.21.15.114:0/1673813389 wait complete. 2024-05-27T07:59:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.225+0000 7f0fc0683740 20 client.19776 trim_cache size 0 max 16384 2024-05-27T07:59:35.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:34 smithi163 ceph-mon[36451]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:59:35.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:34 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 2 up:standby 2024-05-27T07:59:35.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:34 smithi163 ceph-mon[36451]: osdmap e163: 12 total, 12 up, 12 in 2024-05-27T07:59:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:59:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:59:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:59:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.226+0000 7f0fb97ce640 1 -- 172.21.15.114:0/1072046194 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0f9c0b2b60 msgr2=0x7f0f9c0b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:35.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.226+0000 7f0fb97ce640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0f9c0b2b60 0x7f0f9c0b5070 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x55fd80788c30 tx=0x7f0fac005f90 comp rx=0 tx=0).stop 2024-05-27T07:59:35.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.226+0000 7f0fb97ce640 1 -- 172.21.15.114:0/1072046194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fb4109030 msgr2=0x7f0fb4198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:35.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.226+0000 7f0fb97ce640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fb4109030 0x7f0fb4198530 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400eb10 tx=0x7f0fa400c2d0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.226+0000 7f0fb97ce640 1 -- 172.21.15.114:0/1072046194 shutdown_connections 2024-05-27T07:59:35.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.226+0000 7f0fb97ce640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f0f9c0b2b60 0x7f0f9c0b5070 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.226+0000 7f0fb97ce640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fb4109030 0x7f0fb4198530 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.226+0000 7f0fb97ce640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb4108b90 0x7f0fb419d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.226+0000 7f0fb97ce640 1 --2- 172.21.15.114:0/1072046194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb41032e0 0x7f0fb419d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.226+0000 7f0fb97ce640 1 -- 172.21.15.114:0/1072046194 >> 172.21.15.114:0/1072046194 conn(0x7f0fb40782d0 msgr2=0x7f0fb41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:35.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.226+0000 7f0fb97ce640 1 -- 172.21.15.114:0/1072046194 shutdown_connections 2024-05-27T07:59:35.230 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:35.226+0000 7f0fb97ce640 1 -- 172.21.15.114:0/1072046194 wait complete. 2024-05-27T07:59:35.231 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:34 smithi186 ceph-mon[36647]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:59:35.231 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:34 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 2 up:standby 2024-05-27T07:59:35.231 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:34 smithi186 ceph-mon[36647]: osdmap e163: 12 total, 12 up, 12 in 2024-05-27T07:59:35.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.444+0000 7f75d00d3700 1 -- 172.21.15.114:0/1678046730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c8107e20 msgr2=0x7f75c8108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:35.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.444+0000 7f75d00d3700 1 --2- 172.21.15.114:0/1678046730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c8107e20 0x7f75c8108230 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009920 tx=0x7f75b8009c30 comp rx=0 tx=0).stop 2024-05-27T07:59:35.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.444+0000 7f75d00d3700 1 -- 172.21.15.114:0/1678046730 shutdown_connections 2024-05-27T07:59:35.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.444+0000 7f75d00d3700 1 --2- 172.21.15.114:0/1678046730 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c8108770 0x7f75c81100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.444+0000 7f75d00d3700 1 --2- 172.21.15.114:0/1678046730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c8103540 0x7f75c8103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.444+0000 7f75d00d3700 1 --2- 172.21.15.114:0/1678046730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c8107e20 0x7f75c8108230 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.444+0000 7f75d00d3700 1 -- 172.21.15.114:0/1678046730 >> 172.21.15.114:0/1678046730 conn(0x7f75c8078df0 msgr2=0x7f75c8079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:35.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.444+0000 7f75d00d3700 1 -- 172.21.15.114:0/1678046730 shutdown_connections 2024-05-27T07:59:35.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.445+0000 7f75d00d3700 1 -- 172.21.15.114:0/1678046730 wait complete. 2024-05-27T07:59:35.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.446+0000 7f75d00d3700 1 Processor -- start 2024-05-27T07:59:35.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.446+0000 7f75d00d3700 1 -- start start 2024-05-27T07:59:35.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.446+0000 7f75d00d3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c8103540 0x7f75c819af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.446+0000 7f75d00d3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c8107e20 0x7f75c819b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.446+0000 7f75d00d3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c8108770 0x7f75c819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.446+0000 7f75d00d3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f75c8113380 con 0x7f75c8107e20 2024-05-27T07:59:35.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.446+0000 7f75d00d3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f75c8113200 con 0x7f75c8103540 2024-05-27T07:59:35.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.446+0000 7f75d00d3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75c8113500 con 0x7f75c8108770 2024-05-27T07:59:35.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.446+0000 7f75cde6f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c8103540 0x7f75c819af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.446+0000 7f75cde6f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c8103540 0x7f75c819af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:42538/0 (socket says 172.21.15.114:42538) 2024-05-27T07:59:35.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.446+0000 7f75ce670700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c8108770 0x7f75c819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.446+0000 7f75cde6f700 1 -- 172.21.15.114:0/648422646 learned_addr learned my addr 172.21.15.114:0/648422646 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:35.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.447+0000 7f75cd66e700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c8107e20 0x7f75c819b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.447+0000 7f75ce670700 1 -- 172.21.15.114:0/648422646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c8103540 msgr2=0x7f75c819af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:35.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.447+0000 7f75ce670700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c8103540 0x7f75c819af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.447+0000 7f75ce670700 1 -- 172.21.15.114:0/648422646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c8107e20 msgr2=0x7f75c819b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:35.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.447+0000 7f75ce670700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c8107e20 0x7f75c819b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.447+0000 7f75ce670700 1 -- 172.21.15.114:0/648422646 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c819fe30 con 0x7f75c8108770 2024-05-27T07:59:35.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.447+0000 7f75cde6f700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c8103540 0x7f75c819af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:35.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.447+0000 7f75ce670700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c8108770 0x7f75c819f790 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f75c000efc0 tx=0x7f75c000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:35.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.447+0000 7f75beffd700 1 -- 172.21.15.114:0/648422646 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c00092e0 con 0x7f75c8108770 2024-05-27T07:59:35.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.447+0000 7f75d00d3700 1 -- 172.21.15.114:0/648422646 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c81a00c0 con 0x7f75c8108770 2024-05-27T07:59:35.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.447+0000 7f75beffd700 1 -- 172.21.15.114:0/648422646 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75c0004bb0 con 0x7f75c8108770 2024-05-27T07:59:35.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.447+0000 7f75beffd700 1 -- 172.21.15.114:0/648422646 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c0004020 con 0x7f75c8108770 2024-05-27T07:59:35.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.447+0000 7f75d00d3700 1 -- 172.21.15.114:0/648422646 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c8104df0 con 0x7f75c8108770 2024-05-27T07:59:35.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.451+0000 7f75beffd700 1 -- 172.21.15.114:0/648422646 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f75c001b030 con 0x7f75c8108770 2024-05-27T07:59:35.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.451+0000 7f75deae8b80 1 Processor -- start 2024-05-27T07:59:35.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.451+0000 7f75deae8b80 1 -- start start 2024-05-27T07:59:35.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.451+0000 7f75beffd700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75b40b29f0 0x7f75b40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75deae8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 0x5604c6797790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75cd66e700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75b40b29f0 0x7f75b40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75deae8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6797cd0 0x5604c6791ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75deae8b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6792000 0x5604c6792410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75beffd700 1 -- 172.21.15.114:0/648422646 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9510+0+0 (secure 0 0 0) 0x7f75c00e2400 con 0x7f75c8108770 2024-05-27T07:59:35.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75deae8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5604c66c0d10 con 0x5604c6613ec0 2024-05-27T07:59:35.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75deae8b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5604c66b08b0 con 0x5604c6792000 2024-05-27T07:59:35.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75deae8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5604c66b48f0 con 0x5604c6797cd0 2024-05-27T07:59:35.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75cde6f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 0x5604c6797790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75cde6f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 0x5604c6797790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46574/0 (socket says 172.21.15.114:46574) 2024-05-27T07:59:35.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75cde6f700 1 -- 172.21.15.114:0/3084900626 learned_addr learned my addr 172.21.15.114:0/3084900626 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:35.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75cde6f700 1 -- 172.21.15.114:0/3084900626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6797cd0 msgr2=0x5604c6791ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:35.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75cde6f700 1 --2- 172.21.15.114:0/3084900626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6797cd0 0x5604c6791ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75ce670700 1 --2- 172.21.15.114:0/3084900626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6792000 0x5604c6792410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75cde6f700 1 -- 172.21.15.114:0/3084900626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6792000 msgr2=0x5604c6792410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:35.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75cde6f700 1 --2- 172.21.15.114:0/3084900626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6792000 0x5604c6792410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75cde6f700 1 -- 172.21.15.114:0/3084900626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604c6600950 con 0x5604c6613ec0 2024-05-27T07:59:35.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75cd66e700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75b40b29f0 0x7f75b40b4f00 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f75b80098f0 tx=0x7f75b800bfa0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:35.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.452+0000 7f75cde6f700 1 --2- 172.21.15.114:0/3084900626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 0x5604c6797790 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f75c40115d0 tx=0x7f75c40118e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1124fd7ae269d8b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:35.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.453+0000 7f75b3fff700 1 -- 172.21.15.114:0/3084900626 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c401e070 con 0x5604c6613ec0 2024-05-27T07:59:35.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.453+0000 7f75b3fff700 1 -- 172.21.15.114:0/3084900626 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75c400fb30 con 0x5604c6613ec0 2024-05-27T07:59:35.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.453+0000 7f75b3fff700 1 -- 172.21.15.114:0/3084900626 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c4019cd0 con 0x5604c6613ec0 2024-05-27T07:59:35.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.454+0000 7f75deae8b80 1 -- 172.21.15.114:0/3084900626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 msgr2=0x5604c6797790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:35.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.454+0000 7f75deae8b80 1 --2- 172.21.15.114:0/3084900626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 0x5604c6797790 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f75c40115d0 tx=0x7f75c40118e0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.454+0000 7f75deae8b80 1 -- 172.21.15.114:0/3084900626 shutdown_connections 2024-05-27T07:59:35.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.454+0000 7f75deae8b80 1 --2- 172.21.15.114:0/3084900626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6792000 0x5604c6792410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.454+0000 7f75deae8b80 1 --2- 172.21.15.114:0/3084900626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 0x5604c6797790 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.454+0000 7f75deae8b80 1 --2- 172.21.15.114:0/3084900626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6797cd0 0x5604c6791ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.454+0000 7f75deae8b80 1 -- 172.21.15.114:0/3084900626 >> 172.21.15.114:0/3084900626 conn(0x5604c666eb70 msgr2=0x5604c678c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:35.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.454+0000 7f75deae8b80 1 -- 172.21.15.114:0/3084900626 shutdown_connections 2024-05-27T07:59:35.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.454+0000 7f75deae8b80 1 -- 172.21.15.114:0/3084900626 wait complete. 2024-05-27T07:59:35.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75deae8b80 1 -- start start 2024-05-27T07:59:35.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75deae8b80 1 Processor -- start 2024-05-27T07:59:35.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75deae8b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6613ec0 0x5604c679b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75deae8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6792000 0x5604c679b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75deae8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6797cd0 0x5604c679be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75deae8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5604c65b7020 con 0x5604c6792000 2024-05-27T07:59:35.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75deae8b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5604c6652840 con 0x5604c6613ec0 2024-05-27T07:59:35.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75deae8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5604c679e7c0 con 0x5604c6797cd0 2024-05-27T07:59:35.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75deae8b80 10 client.0 _ll_register_callbacks cb 0x5604c664b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:35.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75cde6f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6613ec0 0x5604c679b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75cde6f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6613ec0 0x5604c679b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:42560/0 (socket says 172.21.15.114:42560) 2024-05-27T07:59:35.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75cde6f700 1 -- 172.21.15.114:0/158962120 learned_addr learned my addr 172.21.15.114:0/158962120 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:35.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75cd66e700 1 --2- 172.21.15.114:0/158962120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6792000 0x5604c679b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.455+0000 7f75ce670700 1 --2- 172.21.15.114:0/158962120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6797cd0 0x5604c679be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.456+0000 7f75cde6f700 1 -- 172.21.15.114:0/158962120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6797cd0 msgr2=0x5604c679be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:35.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.456+0000 7f75cde6f700 1 --2- 172.21.15.114:0/158962120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6797cd0 0x5604c679be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.456+0000 7f75cde6f700 1 -- 172.21.15.114:0/158962120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6792000 msgr2=0x5604c679b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:35.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.456+0000 7f75cde6f700 1 --2- 172.21.15.114:0/158962120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6792000 0x5604c679b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.456+0000 7f75cde6f700 1 -- 172.21.15.114:0/158962120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604c664f710 con 0x5604c6613ec0 2024-05-27T07:59:35.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.456+0000 7f75cde6f700 1 --2- 172.21.15.114:0/158962120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6613ec0 0x5604c679b390 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f75c4005950 tx=0x7f75c400fe60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d9860e6a7a2a832 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.456+0000 7f75b17fa700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.456+0000 7f75b17fa700 1 -- 172.21.15.114:0/158962120 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c4026020 con 0x5604c6613ec0 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.456+0000 7f75deae8b80 1 -- 172.21.15.114:0/158962120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75c400fb30 con 0x5604c6613ec0 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.456+0000 7f75b17fa700 1 -- 172.21.15.114:0/158962120 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9510+0+0 (secure 0 0 0) 0x7f75c40198d0 con 0x5604c6613ec0 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.457+0000 7f75b17fa700 1 -- 172.21.15.114:0/158962120 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7f75c4031050 con 0x5604c6613ec0 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.457+0000 7f75deae8b80 10 client.29108 fetch_fsmap learned FSMap version 111 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.457+0000 7f75deae8b80 1 -- 172.21.15.114:0/158962120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=4+}) -- 0x5604c67a91f0 con 0x5604c6613ec0 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.457+0000 7f75b17fa700 1 -- 172.21.15.114:0/158962120 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75c400fb30 con 0x5604c6613ec0 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.457+0000 7f75b17fa700 1 -- 172.21.15.114:0/158962120 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 111) ==== 3490+0+0 (secure 0 0 0) 0x7f75c4004750 con 0x5604c6613ec0 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.457+0000 7f75deae8b80 10 client.29108 fetch_fsmap finished waiting for FSMap version 111 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.457+0000 7f75deae8b80 10 client.29108 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.457+0000 7f75deae8b80 20 client.29108 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.457+0000 7f75deae8b80 1 --2- 172.21.15.114:0/158962120 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5604c67abb40 0x5604c67adff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.457+0000 7f75deae8b80 4 client.29108 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:59:35.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.457+0000 7f75deae8b80 1 -- 172.21.15.114:0/158962120 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604c675da70 con 0x5604c67abb40 2024-05-27T07:59:35.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.457+0000 7f75cd66e700 1 --2- 172.21.15.114:0/158962120 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5604c67abb40 0x5604c67adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.458+0000 7f75b17fa700 10 client.29108 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:35.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.458+0000 7f75cd66e700 1 --2- 172.21.15.114:0/158962120 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5604c67abb40 0x5604c67adff0 crc :-1 s=READY pgs=516 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=195dfe9eee7752a2 server_cookie=af08c9ec17e0e2f3 in_seq=0 out_seq=0 2024-05-27T07:59:35.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.461+0000 7f75b17fa700 1 -- 172.21.15.114:0/158962120 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5604c675da70 con 0x5604c67abb40 2024-05-27T07:59:35.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.461+0000 7f75b17fa700 10 client.29108 handle_command_reply: tid=0 2024-05-27T07:59:35.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.461+0000 7f75deae8b80 1 client.29108 shutdown 2024-05-27T07:59:35.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.461+0000 7f75deae8b80 1 -- 172.21.15.114:0/158962120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6613ec0 msgr2=0x5604c679b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:35.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.461+0000 7f75deae8b80 1 --2- 172.21.15.114:0/158962120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6613ec0 0x5604c679b390 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f75c4005950 tx=0x7f75c400fe60 comp rx=0 tx=0).stop 2024-05-27T07:59:35.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.462+0000 7f75deae8b80 1 -- 172.21.15.114:0/158962120 shutdown_connections 2024-05-27T07:59:35.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.462+0000 7f75deae8b80 1 --2- 172.21.15.114:0/158962120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6797cd0 0x5604c679be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.462+0000 7f75deae8b80 1 --2- 172.21.15.114:0/158962120 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5604c67abb40 0x5604c67adff0 crc :-1 s=READY pgs=516 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.462+0000 7f75deae8b80 1 --2- 172.21.15.114:0/158962120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6613ec0 0x5604c679b390 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.462+0000 7f75deae8b80 1 --2- 172.21.15.114:0/158962120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6792000 0x5604c679b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.462+0000 7f75b17fa700 0 client.29108 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:35.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.462+0000 7f75deae8b80 1 -- 172.21.15.114:0/158962120 >> 172.21.15.114:0/158962120 conn(0x5604c666eb70 msgr2=0x5604c6790ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:35.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.462+0000 7f75deae8b80 1 -- 172.21.15.114:0/158962120 shutdown_connections 2024-05-27T07:59:35.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.462+0000 7f75deae8b80 1 -- 172.21.15.114:0/158962120 wait complete. 2024-05-27T07:59:35.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.462+0000 7f75deae8b80 20 client.29108 trim_cache size 0 max 16384 2024-05-27T07:59:35.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.484+0000 7f75deae8b80 1 Processor -- start 2024-05-27T07:59:35.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.484+0000 7f75deae8b80 1 -- start start 2024-05-27T07:59:35.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.484+0000 7f75deae8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 0x5604c6791ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.484+0000 7f75deae8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6792000 0x5604c6795950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.484+0000 7f75deae8b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6797cd0 0x5604c679bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.484+0000 7f75deae8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f75c81b1ea0 con 0x5604c6613ec0 2024-05-27T07:59:35.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.484+0000 7f75deae8b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f75c81b2570 con 0x5604c6797cd0 2024-05-27T07:59:35.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.484+0000 7f75deae8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75c81b0870 con 0x5604c6792000 2024-05-27T07:59:35.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75cde6f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 0x5604c6791ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75cde6f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 0x5604c6791ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46586/0 (socket says 172.21.15.114:46586) 2024-05-27T07:59:35.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75cde6f700 1 -- 172.21.15.114:0/143538687 learned_addr learned my addr 172.21.15.114:0/143538687 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:35.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75cd66e700 1 --2- 172.21.15.114:0/143538687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6792000 0x5604c6795950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75ce670700 1 --2- 172.21.15.114:0/143538687 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6797cd0 0x5604c679bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75cde6f700 1 -- 172.21.15.114:0/143538687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6792000 msgr2=0x5604c6795950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:35.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75cde6f700 1 --2- 172.21.15.114:0/143538687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6792000 0x5604c6795950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75cde6f700 1 -- 172.21.15.114:0/143538687 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6797cd0 msgr2=0x5604c679bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:35.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75cde6f700 1 --2- 172.21.15.114:0/143538687 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6797cd0 0x5604c679bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75cde6f700 1 -- 172.21.15.114:0/143538687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604c679c450 con 0x5604c6613ec0 2024-05-27T07:59:35.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75ce670700 1 --2- 172.21.15.114:0/143538687 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6797cd0 0x5604c679bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:59:35.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75cde6f700 1 --2- 172.21.15.114:0/143538687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 0x5604c6791ac0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f75c4011920 tx=0x7f75c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba1c075e23fbac2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:35.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75b37fe700 1 -- 172.21.15.114:0/143538687 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c403a070 con 0x5604c6613ec0 2024-05-27T07:59:35.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75b37fe700 1 -- 172.21.15.114:0/143538687 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75c4019d50 con 0x5604c6613ec0 2024-05-27T07:59:35.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.485+0000 7f75b37fe700 1 -- 172.21.15.114:0/143538687 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c4026070 con 0x5604c6613ec0 2024-05-27T07:59:35.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.486+0000 7f75deae8b80 1 -- 172.21.15.114:0/143538687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 msgr2=0x5604c6791ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:35.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.486+0000 7f75deae8b80 1 --2- 172.21.15.114:0/143538687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 0x5604c6791ac0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f75c4011920 tx=0x7f75c40057f0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.486+0000 7f75deae8b80 1 -- 172.21.15.114:0/143538687 shutdown_connections 2024-05-27T07:59:35.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.486+0000 7f75deae8b80 1 --2- 172.21.15.114:0/143538687 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6797cd0 0x5604c679bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.486+0000 7f75deae8b80 1 --2- 172.21.15.114:0/143538687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6613ec0 0x5604c6791ac0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.486+0000 7f75deae8b80 1 --2- 172.21.15.114:0/143538687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6792000 0x5604c6795950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.486+0000 7f75deae8b80 1 -- 172.21.15.114:0/143538687 >> 172.21.15.114:0/143538687 conn(0x5604c666eb70 msgr2=0x5604c679ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:35.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.486+0000 7f75deae8b80 1 -- 172.21.15.114:0/143538687 shutdown_connections 2024-05-27T07:59:35.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.486+0000 7f75deae8b80 1 -- 172.21.15.114:0/143538687 wait complete. 2024-05-27T07:59:35.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.487+0000 7f75deae8b80 1 -- start start 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.487+0000 7f75deae8b80 1 Processor -- start 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.487+0000 7f75deae8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6613ec0 0x5604c67b3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.487+0000 7f75deae8b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6792000 0x5604c67b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.487+0000 7f75deae8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6797cd0 0x5604c67b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.487+0000 7f75deae8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f75c81b0ec0 con 0x5604c6797cd0 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.487+0000 7f75deae8b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5604c6815680 con 0x5604c6792000 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.487+0000 7f75deae8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5604c6815800 con 0x5604c6613ec0 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75deae8b80 10 client.0 _ll_register_callbacks cb 0x5604c664b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75ce670700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6797cd0 0x5604c67b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75ce670700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6797cd0 0x5604c67b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46600/0 (socket says 172.21.15.114:46600) 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75ce670700 1 -- 172.21.15.114:0/177128127 learned_addr learned my addr 172.21.15.114:0/177128127 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75cde6f700 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6613ec0 0x5604c67b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75cd66e700 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6792000 0x5604c67b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75ce670700 1 -- 172.21.15.114:0/177128127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6613ec0 msgr2=0x5604c67b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:35.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75ce670700 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6613ec0 0x5604c67b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75ce670700 1 -- 172.21.15.114:0/177128127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6792000 msgr2=0x5604c67b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:35.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75ce670700 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6792000 0x5604c67b38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75ce670700 1 -- 172.21.15.114:0/177128127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604c6815980 con 0x5604c6797cd0 2024-05-27T07:59:35.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75cde6f700 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6613ec0 0x5604c67b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:35.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75ce670700 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6797cd0 0x5604c67b3de0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f75c00910d0 tx=0x7f75c00a8b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8f432986ca7413 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:35.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75b17fa700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:59:35.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75b17fa700 1 -- 172.21.15.114:0/177128127 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c00a9a90 con 0x5604c6797cd0 2024-05-27T07:59:35.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75b17fa700 1 -- 172.21.15.114:0/177128127 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9510+0+0 (secure 0 0 0) 0x7f75c00a5980 con 0x5604c6797cd0 2024-05-27T07:59:35.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75deae8b80 1 -- 172.21.15.114:0/177128127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75c4019d50 con 0x5604c6797cd0 2024-05-27T07:59:35.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75b17fa700 1 -- 172.21.15.114:0/177128127 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7f75c00a9390 con 0x5604c6797cd0 2024-05-27T07:59:35.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75deae8b80 10 client.19794 fetch_fsmap learned FSMap version 111 2024-05-27T07:59:35.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.488+0000 7f75deae8b80 1 -- 172.21.15.114:0/177128127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=4+}) -- 0x5604c6816bd0 con 0x5604c6797cd0 2024-05-27T07:59:35.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.489+0000 7f75b17fa700 1 -- 172.21.15.114:0/177128127 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75c4019d50 con 0x5604c6797cd0 2024-05-27T07:59:35.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.489+0000 7f75b17fa700 1 -- 172.21.15.114:0/177128127 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 111) ==== 3490+0+0 (secure 0 0 0) 0x7f75c00a3ab0 con 0x5604c6797cd0 2024-05-27T07:59:35.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.489+0000 7f75deae8b80 10 client.19794 fetch_fsmap finished waiting for FSMap version 111 2024-05-27T07:59:35.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.489+0000 7f75deae8b80 10 client.19794 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:35.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.489+0000 7f75deae8b80 20 client.19794 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:35.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.489+0000 7f75deae8b80 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5604c6817260 0x5604c6819680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:35.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.489+0000 7f75deae8b80 4 client.19794 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:59:35.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.489+0000 7f75deae8b80 1 -- 172.21.15.114:0/177128127 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5604c66c2c60 con 0x5604c6817260 2024-05-27T07:59:35.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.489+0000 7f75cde6f700 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5604c6817260 0x5604c6819680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:35.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.489+0000 7f75b17fa700 10 client.19794 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:35.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.490+0000 7f75cde6f700 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5604c6817260 0x5604c6819680 crc :-1 s=READY pgs=517 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa1cf9247fe06dbc server_cookie=e4f22ecb1bf724a8 in_seq=0 out_seq=0 2024-05-27T07:59:35.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75b17fa700 1 -- 172.21.15.114:0/177128127 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5604c66c2c60 con 0x5604c6817260 2024-05-27T07:59:35.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75b17fa700 10 client.19794 handle_command_reply: tid=0 2024-05-27T07:59:35.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75deae8b80 1 client.19794 shutdown 2024-05-27T07:59:35.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75deae8b80 1 -- 172.21.15.114:0/177128127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6797cd0 msgr2=0x5604c67b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:35.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75deae8b80 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6797cd0 0x5604c67b3de0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f75c00910d0 tx=0x7f75c00a8b00 comp rx=0 tx=0).stop 2024-05-27T07:59:35.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75deae8b80 1 -- 172.21.15.114:0/177128127 shutdown_connections 2024-05-27T07:59:35.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75deae8b80 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5604c6797cd0 0x5604c67b3de0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75deae8b80 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604c6613ec0 0x5604c67b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75deae8b80 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5604c6817260 0x5604c6819680 crc :-1 s=READY pgs=517 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75deae8b80 1 --2- 172.21.15.114:0/177128127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5604c6792000 0x5604c67b38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75b17fa700 0 client.19794 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:35.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75deae8b80 1 -- 172.21.15.114:0/177128127 >> 172.21.15.114:0/177128127 conn(0x5604c666eb70 msgr2=0x5604c6786f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:35.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75deae8b80 1 -- 172.21.15.114:0/177128127 shutdown_connections 2024-05-27T07:59:35.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75deae8b80 1 -- 172.21.15.114:0/177128127 wait complete. 2024-05-27T07:59:35.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.494+0000 7f75deae8b80 20 client.19794 trim_cache size 0 max 16384 2024-05-27T07:59:35.498 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:59:35.498 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:59:35.498 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "0435257c-5960-491f-9d70-05244a311902", 2024-05-27T07:59:35.498 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:59:35.498 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:59:35.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.495+0000 7f75d00d3700 1 -- 172.21.15.114:0/648422646 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75b40b29f0 msgr2=0x7f75b40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:35.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.495+0000 7f75d00d3700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75b40b29f0 0x7f75b40b4f00 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f75b80098f0 tx=0x7f75b800bfa0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.495+0000 7f75d00d3700 1 -- 172.21.15.114:0/648422646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c8108770 msgr2=0x7f75c819f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:35.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.495+0000 7f75d00d3700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c8108770 0x7f75c819f790 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f75c000efc0 tx=0x7f75c000c3a0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.495+0000 7f75d00d3700 1 -- 172.21.15.114:0/648422646 shutdown_connections 2024-05-27T07:59:35.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.495+0000 7f75d00d3700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c8108770 0x7f75c819f790 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.495+0000 7f75d00d3700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f75b40b29f0 0x7f75b40b4f00 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.495+0000 7f75d00d3700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75c8103540 0x7f75c819af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.495+0000 7f75d00d3700 1 --2- 172.21.15.114:0/648422646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75c8107e20 0x7f75c819b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:35.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.495+0000 7f75d00d3700 1 -- 172.21.15.114:0/648422646 >> 172.21.15.114:0/648422646 conn(0x7f75c8078df0 msgr2=0x7f75c8075e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:35.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.495+0000 7f75d00d3700 1 -- 172.21.15.114:0/648422646 shutdown_connections 2024-05-27T07:59:35.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:35.495+0000 7f75d00d3700 1 -- 172.21.15.114:0/648422646 wait complete. 2024-05-27T07:59:35.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 80845 11.98 MB/sec execute 227 sec latency 995.311 ms 2024-05-27T07:59:35.734 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5f5cf79700 time 2024-05-27T07:59:35.319669+0000 2024-05-27T07:59:35.734 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:35.734 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:35.734 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5f6f7beba6] 2024-05-27T07:59:35.734 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5f6f7bedd1] 2024-05-27T07:59:35.734 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5615c284423d] 2024-05-27T07:59:35.734 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5615c282f971] 2024-05-27T07:59:35.734 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 5: (SnapRealm::check_cache() const+0x13e) [0x5615c282dfa0] 2024-05-27T07:59:35.734 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5615c282e651] 2024-05-27T07:59:35.734 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5615c27c3347] 2024-05-27T07:59:35.734 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5615c24bf87e] 2024-05-27T07:59:35.734 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5615c24c4f2f] 2024-05-27T07:59:35.735 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5615c24cf969] 2024-05-27T07:59:35.735 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 11: (MDLog::_replay_thread()+0x1f38) [0x5615c286848c] 2024-05-27T07:59:35.735 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5615c28719cb] 2024-05-27T07:59:35.735 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 13: (Thread::entry_wrapper()+0x43) [0x7f5f6f78d72d] 2024-05-27T07:59:35.735 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5f6f78d749] 2024-05-27T07:59:35.735 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5f6e3311ca] 2024-05-27T07:59:35.735 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 16: clone() 2024-05-27T07:59:35.735 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: *** Caught signal (Aborted) ** 2024-05-27T07:59:35.735 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: in thread 7f5f5cf79700 thread_name:md_log_replay 2024-05-27T07:59:35.736 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.320+0000 7f5f5cf79700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5f5cf79700 time 2024-05-27T07:59:35.319669+0000 2024-05-27T07:59:35.736 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:35.736 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.736 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:35.737 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5f6f7beba6] 2024-05-27T07:59:35.737 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5f6f7bedd1] 2024-05-27T07:59:35.737 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5615c284423d] 2024-05-27T07:59:35.737 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5615c282f971] 2024-05-27T07:59:35.737 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 5: (SnapRealm::check_cache() const+0x13e) [0x5615c282dfa0] 2024-05-27T07:59:35.737 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5615c282e651] 2024-05-27T07:59:35.737 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5615c27c3347] 2024-05-27T07:59:35.737 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5615c24bf87e] 2024-05-27T07:59:35.737 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5615c24c4f2f] 2024-05-27T07:59:35.737 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5615c24cf969] 2024-05-27T07:59:35.737 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 11: (MDLog::_replay_thread()+0x1f38) [0x5615c286848c] 2024-05-27T07:59:35.737 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5615c28719cb] 2024-05-27T07:59:35.737 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 13: (Thread::entry_wrapper()+0x43) [0x7f5f6f78d72d] 2024-05-27T07:59:35.738 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5f6f78d749] 2024-05-27T07:59:35.738 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5f6e3311ca] 2024-05-27T07:59:35.738 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 16: clone() 2024-05-27T07:59:35.738 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.738 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:35.738 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5f6e33bd20] 2024-05-27T07:59:35.738 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2: gsignal() 2024-05-27T07:59:35.738 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 3: abort() 2024-05-27T07:59:35.738 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5f6f7beced] 2024-05-27T07:59:35.738 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5f6f7bedd1] 2024-05-27T07:59:35.738 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5615c284423d] 2024-05-27T07:59:35.738 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5615c282f971] 2024-05-27T07:59:35.738 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 8: (SnapRealm::check_cache() const+0x13e) [0x5615c282dfa0] 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5615c282e651] 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5615c27c3347] 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5615c24bf87e] 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5615c24c4f2f] 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5615c24cf969] 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 14: (MDLog::_replay_thread()+0x1f38) [0x5615c286848c] 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5615c28719cb] 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 16: (Thread::entry_wrapper()+0x43) [0x7f5f6f78d72d] 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5f6f78d749] 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5f6e3311ca] 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 19: clone() 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.323+0000 7f5f5cf79700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: in thread 7f5f5cf79700 thread_name:md_log_replay 2024-05-27T07:59:35.739 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.740 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:35.740 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5f6e33bd20] 2024-05-27T07:59:35.740 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2: gsignal() 2024-05-27T07:59:35.740 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 3: abort() 2024-05-27T07:59:35.740 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5f6f7beced] 2024-05-27T07:59:35.740 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5f6f7bedd1] 2024-05-27T07:59:35.740 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5615c284423d] 2024-05-27T07:59:35.740 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5615c282f971] 2024-05-27T07:59:35.740 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 8: (SnapRealm::check_cache() const+0x13e) [0x5615c282dfa0] 2024-05-27T07:59:35.740 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5615c282e651] 2024-05-27T07:59:35.740 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5615c27c3347] 2024-05-27T07:59:35.740 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5615c24bf87e] 2024-05-27T07:59:35.740 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5615c24c4f2f] 2024-05-27T07:59:35.741 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5615c24cf969] 2024-05-27T07:59:35.741 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 14: (MDLog::_replay_thread()+0x1f38) [0x5615c286848c] 2024-05-27T07:59:35.741 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5615c28719cb] 2024-05-27T07:59:35.741 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 16: (Thread::entry_wrapper()+0x43) [0x7f5f6f78d72d] 2024-05-27T07:59:35.741 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5f6f78d749] 2024-05-27T07:59:35.741 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5f6e3311ca] 2024-05-27T07:59:35.741 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 19: clone() 2024-05-27T07:59:35.741 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:59:35.741 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.741 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: -1> 2024-05-27T07:59:35.320+0000 7f5f5cf79700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5f5cf79700 time 2024-05-27T07:59:35.319669+0000 2024-05-27T07:59:35.741 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:35.741 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.741 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5f6f7beba6] 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5f6f7bedd1] 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5615c284423d] 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5615c282f971] 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 5: (SnapRealm::check_cache() const+0x13e) [0x5615c282dfa0] 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5615c282e651] 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5615c27c3347] 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5615c24bf87e] 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5615c24c4f2f] 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5615c24cf969] 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 11: (MDLog::_replay_thread()+0x1f38) [0x5615c286848c] 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5615c28719cb] 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 13: (Thread::entry_wrapper()+0x43) [0x7f5f6f78d72d] 2024-05-27T07:59:35.742 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5f6f78d749] 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5f6e3311ca] 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 16: clone() 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 0> 2024-05-27T07:59:35.323+0000 7f5f5cf79700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: in thread 7f5f5cf79700 thread_name:md_log_replay 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5f6e33bd20] 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2: gsignal() 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 3: abort() 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5f6f7beced] 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5f6f7bedd1] 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5615c284423d] 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5615c282f971] 2024-05-27T07:59:35.743 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 8: (SnapRealm::check_cache() const+0x13e) [0x5615c282dfa0] 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5615c282e651] 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5615c27c3347] 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5615c24bf87e] 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5615c24c4f2f] 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5615c24cf969] 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 14: (MDLog::_replay_thread()+0x1f38) [0x5615c286848c] 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5615c28719cb] 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 16: (Thread::entry_wrapper()+0x43) [0x7f5f6f78d72d] 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5f6f78d749] 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5f6e3311ca] 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 19: clone() 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.744 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: -1> 2024-05-27T07:59:35.320+0000 7f5f5cf79700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5f5cf79700 time 2024-05-27T07:59:35.319669+0000 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5f6f7beba6] 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5f6f7bedd1] 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5615c284423d] 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5615c282f971] 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 5: (SnapRealm::check_cache() const+0x13e) [0x5615c282dfa0] 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5615c282e651] 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5615c27c3347] 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5615c24bf87e] 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5615c24c4f2f] 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5615c24cf969] 2024-05-27T07:59:35.745 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 11: (MDLog::_replay_thread()+0x1f38) [0x5615c286848c] 2024-05-27T07:59:35.746 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5615c28719cb] 2024-05-27T07:59:35.746 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 13: (Thread::entry_wrapper()+0x43) [0x7f5f6f78d72d] 2024-05-27T07:59:35.746 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5f6f78d749] 2024-05-27T07:59:35.746 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5f6e3311ca] 2024-05-27T07:59:35.746 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 16: clone() 2024-05-27T07:59:35.746 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.746 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 0> 2024-05-27T07:59:35.323+0000 7f5f5cf79700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:59:35.746 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: in thread 7f5f5cf79700 thread_name:md_log_replay 2024-05-27T07:59:35.746 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.746 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:35.746 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5f6e33bd20] 2024-05-27T07:59:35.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2: gsignal() 2024-05-27T07:59:35.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 3: abort() 2024-05-27T07:59:35.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5f6f7beced] 2024-05-27T07:59:35.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5f6f7bedd1] 2024-05-27T07:59:35.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5615c284423d] 2024-05-27T07:59:35.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5615c282f971] 2024-05-27T07:59:35.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 8: (SnapRealm::check_cache() const+0x13e) [0x5615c282dfa0] 2024-05-27T07:59:35.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5615c282e651] 2024-05-27T07:59:35.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5615c27c3347] 2024-05-27T07:59:35.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5615c24bf87e] 2024-05-27T07:59:35.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5615c24c4f2f] 2024-05-27T07:59:35.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5615c24cf969] 2024-05-27T07:59:35.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 14: (MDLog::_replay_thread()+0x1f38) [0x5615c286848c] 2024-05-27T07:59:35.748 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5615c28719cb] 2024-05-27T07:59:35.748 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 16: (Thread::entry_wrapper()+0x43) [0x7f5f6f78d72d] 2024-05-27T07:59:35.748 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5f6f78d749] 2024-05-27T07:59:35.748 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5f6e3311ca] 2024-05-27T07:59:35.748 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 19: clone() 2024-05-27T07:59:35.748 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:59:35.748 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[130087]: 2024-05-27T07:59:35.748 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:35 smithi114 podman[133487]: 2024-05-27 07:59:35.447215173 +0000 UTC m=+0.078521194 container died c8ee4cca6dcd10c24fa8690fcf55d848be184b751704c60e96044e1e10667ee3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:59:36.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:59:36.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:59:36.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:59:36.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:35 smithi114 ceph-mon[32038]: pgmap v686: 129 pgs: 129 active+clean; 392 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.3 MiB/s wr, 292 op/s 2024-05-27T07:59:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:35 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:59:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:35 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:59:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:35 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:59:36.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:35 smithi163 ceph-mon[36451]: pgmap v686: 129 pgs: 129 active+clean; 392 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.3 MiB/s wr, 292 op/s 2024-05-27T07:59:36.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:35 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:59:36.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:35 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:59:36.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:35 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:59:36.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:35 smithi186 ceph-mon[36647]: pgmap v686: 129 pgs: 129 active+clean; 392 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.3 MiB/s wr, 292 op/s 2024-05-27T07:59:36.465 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:36 smithi114 podman[133487]: 2024-05-27 07:59:36.118492111 +0000 UTC m=+0.749798124 container remove c8ee4cca6dcd10c24fa8690fcf55d848be184b751704c60e96044e1e10667ee3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-27T07:59:36.466 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:36 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:59:36.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 81152 12.00 MB/sec execute 228 sec latency 299.506 ms 2024-05-27T07:59:36.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:59:37.057 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:37 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:59:37.147 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:59:37.621 INFO:tasks.workunit.client.0.smithi114.stdout: 10 81412 12.02 MB/sec execute 229 sec latency 292.714 ms 2024-05-27T07:59:37.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:37 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:59:37.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:37 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:59:37.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:37 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:59:37.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:37 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:59:38.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:37 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:59:38.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:37 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:59:38.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 81726 12.05 MB/sec execute 230 sec latency 184.081 ms 2024-05-27T07:59:38.778 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:38 smithi114 ceph-mon[32038]: pgmap v687: 129 pgs: 129 active+clean; 380 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 484 op/s 2024-05-27T07:59:38.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:38 smithi163 ceph-mon[36451]: pgmap v687: 129 pgs: 129 active+clean; 380 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 484 op/s 2024-05-27T07:59:38.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:38 smithi186 ceph-mon[36647]: pgmap v687: 129 pgs: 129 active+clean; 380 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 484 op/s 2024-05-27T07:59:39.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.392+0000 7f87d8905700 1 -- 172.21.15.114:0/3073417020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87d41045c0 msgr2=0x7f87d41049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:39.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.392+0000 7f87d8905700 1 --2- 172.21.15.114:0/3073417020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87d41045c0 0x7f87d41049d0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f87c40099c0 tx=0x7f87c4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.392+0000 7f87d8905700 1 -- 172.21.15.114:0/3073417020 shutdown_connections 2024-05-27T07:59:39.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.392+0000 7f87d8905700 1 --2- 172.21.15.114:0/3073417020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87d4109840 0x7f87d41111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.392+0000 7f87d8905700 1 --2- 172.21.15.114:0/3073417020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87d41045c0 0x7f87d41049d0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.392+0000 7f87d8905700 1 --2- 172.21.15.114:0/3073417020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87d4108ef0 0x7f87d4109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.392+0000 7f87d8905700 1 -- 172.21.15.114:0/3073417020 >> 172.21.15.114:0/3073417020 conn(0x7f87d40ffe90 msgr2=0x7f87d41022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:39.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.392+0000 7f87d8905700 1 -- 172.21.15.114:0/3073417020 shutdown_connections 2024-05-27T07:59:39.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.392+0000 7f87d8905700 1 -- 172.21.15.114:0/3073417020 wait complete. 2024-05-27T07:59:39.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.393+0000 7f87d8905700 1 Processor -- start 2024-05-27T07:59:39.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.393+0000 7f87d8905700 1 -- start start 2024-05-27T07:59:39.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d8905700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87d41045c0 0x7f87d4076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d8905700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87d4108ef0 0x7f87d4077100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d8905700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87d4109840 0x7f87d4077640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d8905700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f87d4114470 con 0x7f87d4109840 2024-05-27T07:59:39.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d8905700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41142f0 con 0x7f87d4108ef0 2024-05-27T07:59:39.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d8905700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41145f0 con 0x7f87d41045c0 2024-05-27T07:59:39.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87d4109840 0x7f87d4077640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87d4109840 0x7f87d4077640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46620/0 (socket says 172.21.15.114:46620) 2024-05-27T07:59:39.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d2d9d700 1 -- 172.21.15.114:0/746382906 learned_addr learned my addr 172.21.15.114:0/746382906 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:39.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87d41045c0 0x7f87d4076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d1d9b700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87d4108ef0 0x7f87d4077100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d2d9d700 1 -- 172.21.15.114:0/746382906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87d41045c0 msgr2=0x7f87d4076bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:39.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d2d9d700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87d41045c0 0x7f87d4076bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d2d9d700 1 -- 172.21.15.114:0/746382906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87d4108ef0 msgr2=0x7f87d4077100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:39.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d2d9d700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87d4108ef0 0x7f87d4077100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d2d9d700 1 -- 172.21.15.114:0/746382906 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d41a9140 con 0x7f87d4109840 2024-05-27T07:59:39.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d1d9b700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87d4108ef0 0x7f87d4077100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:39.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.394+0000 7f87d2d9d700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87d4109840 0x7f87d4077640 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00ef30 tx=0x7f87cc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:39.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.395+0000 7f87c37fe700 1 -- 172.21.15.114:0/746382906 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc0092e0 con 0x7f87d4109840 2024-05-27T07:59:39.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.395+0000 7f87c37fe700 1 -- 172.21.15.114:0/746382906 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87cc004bb0 con 0x7f87d4109840 2024-05-27T07:59:39.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.395+0000 7f87c37fe700 1 -- 172.21.15.114:0/746382906 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc004020 con 0x7f87d4109840 2024-05-27T07:59:39.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.395+0000 7f87d8905700 1 -- 172.21.15.114:0/746382906 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d41a93d0 con 0x7f87d4109840 2024-05-27T07:59:39.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.395+0000 7f87d8905700 1 -- 172.21.15.114:0/746382906 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d41a98c0 con 0x7f87d4109840 2024-05-27T07:59:39.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.398+0000 7f87c37fe700 1 -- 172.21.15.114:0/746382906 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f87cc01b030 con 0x7f87d4109840 2024-05-27T07:59:39.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.399+0000 7f87c37fe700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87bc0b29f0 0x7f87bc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.399+0000 7f87d259c700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87bc0b29f0 0x7f87bc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.399+0000 7f87c37fe700 1 -- 172.21.15.114:0/746382906 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9510+0+0 (secure 0 0 0) 0x7f87cc0e2440 con 0x7f87d4109840 2024-05-27T07:59:39.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87e731ab80 1 Processor -- start 2024-05-27T07:59:39.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87d259c700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87bc0b29f0 0x7f87bc0b4f00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f87c4000c00 tx=0x7f87c401a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:39.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87e731ab80 1 -- start start 2024-05-27T07:59:39.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87e731ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273a65e40 0x561273beab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87e731ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 0x561273be4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87e731ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273be5120 0x561273be5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87e731ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561273b12d70 con 0x561273be5120 2024-05-27T07:59:39.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87e731ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561273b02910 con 0x561273a65e40 2024-05-27T07:59:39.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87e731ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561273b06950 con 0x561273beb070 2024-05-27T07:59:39.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87d2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273be5120 0x561273be5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87d2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273be5120 0x561273be5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46636/0 (socket says 172.21.15.114:46636) 2024-05-27T07:59:39.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87d2d9d700 1 -- 172.21.15.114:0/3981265455 learned_addr learned my addr 172.21.15.114:0/3981265455 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:39.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87d1d9b700 1 --2- 172.21.15.114:0/3981265455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 0x561273be4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.400+0000 7f87d259c700 1 --2- 172.21.15.114:0/3981265455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273a65e40 0x561273beab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.401+0000 7f87d2d9d700 1 -- 172.21.15.114:0/3981265455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 msgr2=0x561273be4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:39.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.401+0000 7f87d2d9d700 1 --2- 172.21.15.114:0/3981265455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 0x561273be4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.401+0000 7f87d2d9d700 1 -- 172.21.15.114:0/3981265455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273a65e40 msgr2=0x561273beab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:39.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.401+0000 7f87d2d9d700 1 --2- 172.21.15.114:0/3981265455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273a65e40 0x561273beab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.401+0000 7f87d2d9d700 1 -- 172.21.15.114:0/3981265455 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561273aabfb0 con 0x561273be5120 2024-05-27T07:59:39.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.401+0000 7f87d259c700 1 --2- 172.21.15.114:0/3981265455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273a65e40 0x561273beab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:39.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.401+0000 7f87d1d9b700 1 --2- 172.21.15.114:0/3981265455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 0x561273be4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:39.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.401+0000 7f87d2d9d700 1 --2- 172.21.15.114:0/3981265455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273be5120 0x561273be5530 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0a90d0 tx=0x7f87cc0a8ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1e75c2ed12376e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:39.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.401+0000 7f87c0ff9700 1 -- 172.21.15.114:0/3981265455 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc0a93d0 con 0x561273be5120 2024-05-27T07:59:39.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.401+0000 7f87c0ff9700 1 -- 172.21.15.114:0/3981265455 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87cc0ad780 con 0x561273be5120 2024-05-27T07:59:39.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.401+0000 7f87c0ff9700 1 -- 172.21.15.114:0/3981265455 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc0b1f70 con 0x561273be5120 2024-05-27T07:59:39.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.402+0000 7f87e731ab80 1 -- 172.21.15.114:0/3981265455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273be5120 msgr2=0x561273be5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:39.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.402+0000 7f87e731ab80 1 --2- 172.21.15.114:0/3981265455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273be5120 0x561273be5530 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0a90d0 tx=0x7f87cc0a8ee0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.402+0000 7f87e731ab80 1 -- 172.21.15.114:0/3981265455 shutdown_connections 2024-05-27T07:59:39.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.402+0000 7f87e731ab80 1 --2- 172.21.15.114:0/3981265455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273be5120 0x561273be5530 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.402+0000 7f87e731ab80 1 --2- 172.21.15.114:0/3981265455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273a65e40 0x561273beab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.402+0000 7f87e731ab80 1 --2- 172.21.15.114:0/3981265455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 0x561273be4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.402+0000 7f87e731ab80 1 -- 172.21.15.114:0/3981265455 >> 172.21.15.114:0/3981265455 conn(0x561273a68770 msgr2=0x561273bdfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:39.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.402+0000 7f87e731ab80 1 -- 172.21.15.114:0/3981265455 shutdown_connections 2024-05-27T07:59:39.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.402+0000 7f87e731ab80 1 -- 172.21.15.114:0/3981265455 wait complete. 2024-05-27T07:59:39.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87e731ab80 1 -- start start 2024-05-27T07:59:39.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87e731ab80 1 Processor -- start 2024-05-27T07:59:39.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87e731ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 0x561273bee910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87e731ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273be5120 0x561273beee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87e731ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 0x561273bef390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87e731ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561273a08fa0 con 0x561273a65e40 2024-05-27T07:59:39.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87e731ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561273a6ada0 con 0x561273be5120 2024-05-27T07:59:39.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87e731ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561273bf1b60 con 0x561273beb070 2024-05-27T07:59:39.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87d259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 0x561273bee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87e731ab80 10 client.0 _ll_register_callbacks cb 0x561273a995b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:39.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87d259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 0x561273bee910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46648/0 (socket says 172.21.15.114:46648) 2024-05-27T07:59:39.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87d259c700 1 -- 172.21.15.114:0/3296711097 learned_addr learned my addr 172.21.15.114:0/3296711097 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:39.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87d1d9b700 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273be5120 0x561273beee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.403+0000 7f87d2d9d700 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 0x561273bef390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87d259c700 1 -- 172.21.15.114:0/3296711097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 msgr2=0x561273bef390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:39.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87d259c700 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 0x561273bef390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87d259c700 1 -- 172.21.15.114:0/3296711097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273be5120 msgr2=0x561273beee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:39.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87d259c700 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273be5120 0x561273beee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87d259c700 1 -- 172.21.15.114:0/3296711097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561273a528d0 con 0x561273a65e40 2024-05-27T07:59:39.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87d1d9b700 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273be5120 0x561273beee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:39.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87d259c700 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 0x561273bee910 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f87c40099c0 tx=0x7f87c4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c951599dd262df04 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:39.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87b9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:59:39.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87b9ffb700 1 -- 172.21.15.114:0/3296711097 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c400bd70 con 0x561273a65e40 2024-05-27T07:59:39.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87b9ffb700 1 -- 172.21.15.114:0/3296711097 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9510+0+0 (secure 0 0 0) 0x7f87c4025470 con 0x561273a65e40 2024-05-27T07:59:39.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87e731ab80 1 -- 172.21.15.114:0/3296711097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87cc0ad780 con 0x561273a65e40 2024-05-27T07:59:39.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87b9ffb700 1 -- 172.21.15.114:0/3296711097 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7f87c4025670 con 0x561273a65e40 2024-05-27T07:59:39.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87e731ab80 10 client.19806 fetch_fsmap learned FSMap version 111 2024-05-27T07:59:39.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87e731ab80 1 -- 172.21.15.114:0/3296711097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=4+}) -- 0x561273abc760 con 0x561273a65e40 2024-05-27T07:59:39.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.404+0000 7f87b9ffb700 1 -- 172.21.15.114:0/3296711097 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87cc0ad780 con 0x561273a65e40 2024-05-27T07:59:39.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.405+0000 7f87b9ffb700 1 -- 172.21.15.114:0/3296711097 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 111) ==== 3490+0+0 (secure 0 0 0) 0x7f87c4021070 con 0x561273a65e40 2024-05-27T07:59:39.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.405+0000 7f87e731ab80 10 client.19806 fetch_fsmap finished waiting for FSMap version 111 2024-05-27T07:59:39.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.405+0000 7f87e731ab80 10 client.19806 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:39.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.405+0000 7f87e731ab80 20 client.19806 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:39.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.405+0000 7f87e731ab80 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561273bfeda0 0x561273c01210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.405+0000 7f87e731ab80 4 client.19806 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:59:39.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.405+0000 7f87e731ab80 1 -- 172.21.15.114:0/3296711097 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561273b14cc0 con 0x561273bfeda0 2024-05-27T07:59:39.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.405+0000 7f87d1d9b700 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561273bfeda0 0x561273c01210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.405+0000 7f87d1d9b700 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561273bfeda0 0x561273c01210 crc :-1 s=READY pgs=518 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2346aaa9dbc2b8fa server_cookie=dcc76f3bdf954e4c in_seq=0 out_seq=0 2024-05-27T07:59:39.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.405+0000 7f87b9ffb700 10 client.19806 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:39.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87b9ffb700 1 -- 172.21.15.114:0/3296711097 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561273b14cc0 con 0x561273bfeda0 2024-05-27T07:59:39.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87b9ffb700 10 client.19806 handle_command_reply: tid=0 2024-05-27T07:59:39.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87e731ab80 1 client.19806 shutdown 2024-05-27T07:59:39.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87e731ab80 1 -- 172.21.15.114:0/3296711097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 msgr2=0x561273bee910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:39.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87e731ab80 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 0x561273bee910 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f87c40099c0 tx=0x7f87c4009310 comp rx=0 tx=0).stop 2024-05-27T07:59:39.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87e731ab80 1 -- 172.21.15.114:0/3296711097 shutdown_connections 2024-05-27T07:59:39.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87e731ab80 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 0x561273bef390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87e731ab80 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 0x561273bee910 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87e731ab80 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561273bfeda0 0x561273c01210 crc :-1 s=READY pgs=518 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87e731ab80 1 --2- 172.21.15.114:0/3296711097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273be5120 0x561273beee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87e731ab80 1 -- 172.21.15.114:0/3296711097 >> 172.21.15.114:0/3296711097 conn(0x561273a68770 msgr2=0x561273be3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:39.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87b9ffb700 0 client.19806 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:39.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87e731ab80 1 -- 172.21.15.114:0/3296711097 shutdown_connections 2024-05-27T07:59:39.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87e731ab80 1 -- 172.21.15.114:0/3296711097 wait complete. 2024-05-27T07:59:39.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.410+0000 7f87e731ab80 20 client.19806 trim_cache size 0 max 16384 2024-05-27T07:59:39.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87e731ab80 1 Processor -- start 2024-05-27T07:59:39.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87e731ab80 1 -- start start 2024-05-27T07:59:39.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87e731ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 0x561273be4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87e731ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273be5120 0x561273c01420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87e731ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 0x561273be8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87e731ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41b1cd0 con 0x561273a65e40 2024-05-27T07:59:39.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87e731ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41b23a0 con 0x561273be5120 2024-05-27T07:59:39.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87e731ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41b0680 con 0x561273beb070 2024-05-27T07:59:39.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87d259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 0x561273be4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87d259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 0x561273be4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46660/0 (socket says 172.21.15.114:46660) 2024-05-27T07:59:39.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87d259c700 1 -- 172.21.15.114:0/641123959 learned_addr learned my addr 172.21.15.114:0/641123959 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:39.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87d2d9d700 1 --2- 172.21.15.114:0/641123959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 0x561273be8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87d1d9b700 1 --2- 172.21.15.114:0/641123959 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273be5120 0x561273c01420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87d259c700 1 -- 172.21.15.114:0/641123959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 msgr2=0x561273be8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:39.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87d259c700 1 --2- 172.21.15.114:0/641123959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 0x561273be8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87d259c700 1 -- 172.21.15.114:0/641123959 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273be5120 msgr2=0x561273c01420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:39.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87d259c700 1 --2- 172.21.15.114:0/641123959 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273be5120 0x561273c01420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87d259c700 1 -- 172.21.15.114:0/641123959 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561273bea2a0 con 0x561273a65e40 2024-05-27T07:59:39.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.432+0000 7f87d1d9b700 1 --2- 172.21.15.114:0/641123959 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273be5120 0x561273c01420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:59:39.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.433+0000 7f87d259c700 1 --2- 172.21.15.114:0/641123959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 0x561273be4be0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f87c400f9e0 tx=0x7f87c4025600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c5fb7442e9b674 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:39.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.433+0000 7f87bbfff700 1 -- 172.21.15.114:0/641123959 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c400f660 con 0x561273a65e40 2024-05-27T07:59:39.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.433+0000 7f87bbfff700 1 -- 172.21.15.114:0/641123959 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87c4025810 con 0x561273a65e40 2024-05-27T07:59:39.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.433+0000 7f87bbfff700 1 -- 172.21.15.114:0/641123959 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c403f6e0 con 0x561273a65e40 2024-05-27T07:59:39.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.434+0000 7f87e731ab80 1 -- 172.21.15.114:0/641123959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 msgr2=0x561273be4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:39.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.434+0000 7f87e731ab80 1 --2- 172.21.15.114:0/641123959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 0x561273be4be0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f87c400f9e0 tx=0x7f87c4025600 comp rx=0 tx=0).stop 2024-05-27T07:59:39.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.434+0000 7f87e731ab80 1 -- 172.21.15.114:0/641123959 shutdown_connections 2024-05-27T07:59:39.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.434+0000 7f87e731ab80 1 --2- 172.21.15.114:0/641123959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273beb070 0x561273be8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.434+0000 7f87e731ab80 1 --2- 172.21.15.114:0/641123959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273a65e40 0x561273be4be0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.434+0000 7f87e731ab80 1 --2- 172.21.15.114:0/641123959 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273be5120 0x561273c01420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.434+0000 7f87e731ab80 1 -- 172.21.15.114:0/641123959 >> 172.21.15.114:0/641123959 conn(0x561273a68770 msgr2=0x561273be8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:39.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.434+0000 7f87e731ab80 1 -- 172.21.15.114:0/641123959 shutdown_connections 2024-05-27T07:59:39.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.434+0000 7f87e731ab80 1 -- 172.21.15.114:0/641123959 wait complete. 2024-05-27T07:59:39.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.434+0000 7f87e731ab80 1 -- start start 2024-05-27T07:59:39.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.435+0000 7f87e731ab80 1 Processor -- start 2024-05-27T07:59:39.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.435+0000 7f87e731ab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273be5120 0x561273bf7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.435+0000 7f87e731ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273beb070 0x561273bf76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.435+0000 7f87e731ab80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273c06060 0x561273c06530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.435+0000 7f87e731ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561273bed570 con 0x561273beb070 2024-05-27T07:59:39.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.435+0000 7f87e731ab80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41b0cd0 con 0x561273c06060 2024-05-27T07:59:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.435+0000 7f87e731ab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561273bed6f0 con 0x561273be5120 2024-05-27T07:59:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.435+0000 7f87e731ab80 10 client.0 _ll_register_callbacks cb 0x561273a995b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.435+0000 7f87d259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273be5120 0x561273bf7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.435+0000 7f87d259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273be5120 0x561273bf7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:34074/0 (socket says 172.21.15.114:34074) 2024-05-27T07:59:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.435+0000 7f87d259c700 1 -- 172.21.15.114:0/2745728401 learned_addr learned my addr 172.21.15.114:0/2745728401 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.435+0000 7f87d1d9b700 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273beb070 0x561273bf76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.435+0000 7f87d2d9d700 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273c06060 0x561273c06530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87d2d9d700 1 -- 172.21.15.114:0/2745728401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273be5120 msgr2=0x561273bf7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87d2d9d700 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273be5120 0x561273bf7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87d2d9d700 1 -- 172.21.15.114:0/2745728401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273beb070 msgr2=0x561273bf76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87d2d9d700 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273beb070 0x561273bf76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87d2d9d700 1 -- 172.21.15.114:0/2745728401 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561273c06cd0 con 0x561273c06060 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87d259c700 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273be5120 0x561273bf7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87d2d9d700 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273c06060 0x561273c06530 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0dfdb0 tx=0x7f87cc0b1f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=778c695ae881c82d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87b9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87b9ffb700 1 -- 172.21.15.114:0/2745728401 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc0ad780 con 0x561273c06060 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87b9ffb700 1 -- 172.21.15.114:0/2745728401 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9510+0+0 (secure 0 0 0) 0x7f87cc0a8840 con 0x561273c06060 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87e731ab80 1 -- 172.21.15.114:0/2745728401 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87c4025810 con 0x561273c06060 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87b9ffb700 1 -- 172.21.15.114:0/2745728401 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7f87cc0a93d0 con 0x561273c06060 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87e731ab80 10 client.29132 fetch_fsmap learned FSMap version 111 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.436+0000 7f87e731ab80 1 -- 172.21.15.114:0/2745728401 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=4+}) -- 0x561273c69830 con 0x561273c06060 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.437+0000 7f87b9ffb700 1 -- 172.21.15.114:0/2745728401 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87c4025810 con 0x561273c06060 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.437+0000 7f87b9ffb700 1 -- 172.21.15.114:0/2745728401 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 111) ==== 3490+0+0 (secure 0 0 0) 0x7f87cc0a4f40 con 0x561273c06060 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.437+0000 7f87e731ab80 10 client.29132 fetch_fsmap finished waiting for FSMap version 111 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.437+0000 7f87e731ab80 10 client.29132 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.437+0000 7f87e731ab80 20 client.29132 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:39.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.437+0000 7f87e731ab80 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561273c69d20 0x561273c6c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:39.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.437+0000 7f87e731ab80 4 client.29132 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:59:39.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.437+0000 7f87e731ab80 1 -- 172.21.15.114:0/2745728401 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561273b0e4a0 con 0x561273c69d20 2024-05-27T07:59:39.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.437+0000 7f87d259c700 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561273c69d20 0x561273c6c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:39.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.437+0000 7f87b9ffb700 10 client.29132 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:39.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.438+0000 7f87d259c700 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561273c69d20 0x561273c6c140 crc :-1 s=READY pgs=519 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e14c30d89d08a151 server_cookie=b2dea9f62558c7c6 in_seq=0 out_seq=0 2024-05-27T07:59:39.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87b9ffb700 1 -- 172.21.15.114:0/2745728401 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561273b0e4a0 con 0x561273c69d20 2024-05-27T07:59:39.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87b9ffb700 10 client.29132 handle_command_reply: tid=0 2024-05-27T07:59:39.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87e731ab80 1 client.29132 shutdown 2024-05-27T07:59:39.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87e731ab80 1 -- 172.21.15.114:0/2745728401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273c06060 msgr2=0x561273c06530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:39.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87e731ab80 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273c06060 0x561273c06530 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0dfdb0 tx=0x7f87cc0b1f40 comp rx=0 tx=0).stop 2024-05-27T07:59:39.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87e731ab80 1 -- 172.21.15.114:0/2745728401 shutdown_connections 2024-05-27T07:59:39.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87e731ab80 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x561273c69d20 0x561273c6c140 crc :-1 s=READY pgs=519 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87e731ab80 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561273c06060 0x561273c06530 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87e731ab80 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561273be5120 0x561273bf7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87b9ffb700 0 client.29132 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:39.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87e731ab80 1 --2- 172.21.15.114:0/2745728401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561273beb070 0x561273bf76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87e731ab80 1 -- 172.21.15.114:0/2745728401 >> 172.21.15.114:0/2745728401 conn(0x561273a68770 msgr2=0x561273bd94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:39.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87e731ab80 1 -- 172.21.15.114:0/2745728401 shutdown_connections 2024-05-27T07:59:39.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87e731ab80 1 -- 172.21.15.114:0/2745728401 wait complete. 2024-05-27T07:59:39.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.440+0000 7f87e731ab80 20 client.29132 trim_cache size 0 max 16384 2024-05-27T07:59:39.445 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:59:39.445 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:59:39.446 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:59:39.446 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:59:39.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.441+0000 7f87d8905700 1 -- 172.21.15.114:0/746382906 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87bc0b29f0 msgr2=0x7f87bc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:39.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.441+0000 7f87d8905700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87bc0b29f0 0x7f87bc0b4f00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f87c4000c00 tx=0x7f87c401a040 comp rx=0 tx=0).stop 2024-05-27T07:59:39.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.441+0000 7f87d8905700 1 -- 172.21.15.114:0/746382906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87d4109840 msgr2=0x7f87d4077640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:39.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.441+0000 7f87d8905700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87d4109840 0x7f87d4077640 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00ef30 tx=0x7f87cc00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.441+0000 7f87d8905700 1 -- 172.21.15.114:0/746382906 shutdown_connections 2024-05-27T07:59:39.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.441+0000 7f87d8905700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87d4109840 0x7f87d4077640 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.441+0000 7f87d8905700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87d41045c0 0x7f87d4076bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.441+0000 7f87d8905700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f87bc0b29f0 0x7f87bc0b4f00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.441+0000 7f87d8905700 1 --2- 172.21.15.114:0/746382906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87d4108ef0 0x7f87d4077100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:39.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.441+0000 7f87d8905700 1 -- 172.21.15.114:0/746382906 >> 172.21.15.114:0/746382906 conn(0x7f87d40ffe90 msgr2=0x7f87d410a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:39.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.441+0000 7f87d8905700 1 -- 172.21.15.114:0/746382906 shutdown_connections 2024-05-27T07:59:39.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:39.441+0000 7f87d8905700 1 -- 172.21.15.114:0/746382906 wait complete. 2024-05-27T07:59:39.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 82047 12.06 MB/sec execute 231 sec latency 201.451 ms 2024-05-27T07:59:40.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:39 smithi114 ceph-mon[32038]: pgmap v688: 129 pgs: 129 active+clean; 378 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.2 MiB/s wr, 566 op/s 2024-05-27T07:59:40.198 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:59:40.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:59:40.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:39 smithi163 ceph-mon[36451]: pgmap v688: 129 pgs: 129 active+clean; 378 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.2 MiB/s wr, 566 op/s 2024-05-27T07:59:40.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:39 smithi186 ceph-mon[36647]: pgmap v688: 129 pgs: 129 active+clean; 378 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.2 MiB/s wr, 566 op/s 2024-05-27T07:59:40.462 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:59:40.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 82333 12.06 MB/sec execute 232 sec latency 270.507 ms 2024-05-27T07:59:41.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 82721 12.10 MB/sec execute 233 sec latency 187.247 ms 2024-05-27T07:59:42.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:41 smithi114 ceph-mon[32038]: pgmap v689: 129 pgs: 129 active+clean; 387 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.9 MiB/s wr, 608 op/s 2024-05-27T07:59:42.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:41 smithi163 ceph-mon[36451]: pgmap v689: 129 pgs: 129 active+clean; 387 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.9 MiB/s wr, 608 op/s 2024-05-27T07:59:42.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:41 smithi186 ceph-mon[36647]: pgmap v689: 129 pgs: 129 active+clean; 387 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.9 MiB/s wr, 608 op/s 2024-05-27T07:59:42.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.306+0000 7f50c6b1c700 1 -- 172.21.15.114:0/3369395515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c010b270 msgr2=0x7f50c011d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:42.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.306+0000 7f50c6b1c700 1 --2- 172.21.15.114:0/3369395515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c010b270 0x7f50c011d3e0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f50b40098f0 tx=0x7f50b4009c00 comp rx=0 tx=0).stop 2024-05-27T07:59:42.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.306+0000 7f50c6b1c700 1 -- 172.21.15.114:0/3369395515 shutdown_connections 2024-05-27T07:59:42.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.306+0000 7f50c6b1c700 1 --2- 172.21.15.114:0/3369395515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50c011d920 0x7f50c011fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:42.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.306+0000 7f50c6b1c700 1 --2- 172.21.15.114:0/3369395515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50c010a920 0x7f50c010ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:42.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.307+0000 7f50c6b1c700 1 --2- 172.21.15.114:0/3369395515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c010b270 0x7f50c011d3e0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:42.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.307+0000 7f50c6b1c700 1 -- 172.21.15.114:0/3369395515 >> 172.21.15.114:0/3369395515 conn(0x7f50c006f4c0 msgr2=0x7f50c006f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:42.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.307+0000 7f50c6b1c700 1 -- 172.21.15.114:0/3369395515 shutdown_connections 2024-05-27T07:59:42.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.307+0000 7f50c6b1c700 1 -- 172.21.15.114:0/3369395515 wait complete. 2024-05-27T07:59:42.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.308+0000 7f50c6b1c700 1 Processor -- start 2024-05-27T07:59:42.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.308+0000 7f50c6b1c700 1 -- start start 2024-05-27T07:59:42.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.308+0000 7f50c6b1c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50c010a920 0x7f50c011d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:42.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.308+0000 7f50c6b1c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c010b270 0x7f50c01193a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:42.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.308+0000 7f50c6b1c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50c011d920 0x7f50c01198e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:42.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.308+0000 7f50c6b1c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f50c0115340 con 0x7f50c010b270 2024-05-27T07:59:42.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.308+0000 7f50c6b1c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f50c01151c0 con 0x7f50c011d920 2024-05-27T07:59:42.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.308+0000 7f50c6b1c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f50c01154c0 con 0x7f50c010a920 2024-05-27T07:59:42.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.308+0000 7f50bffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c010b270 0x7f50c01193a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:42.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.308+0000 7f50bffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c010b270 0x7f50c01193a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46670/0 (socket says 172.21.15.114:46670) 2024-05-27T07:59:42.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.308+0000 7f50bffff700 1 -- 172.21.15.114:0/2486832650 learned_addr learned my addr 172.21.15.114:0/2486832650 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:42.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.308+0000 7f50c48b8700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50c010a920 0x7f50c011d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:42.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.309+0000 7f50c50b9700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50c011d920 0x7f50c01198e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:42.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.309+0000 7f50bffff700 1 -- 172.21.15.114:0/2486832650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50c010a920 msgr2=0x7f50c011d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:42.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.309+0000 7f50bffff700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50c010a920 0x7f50c011d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:42.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.309+0000 7f50bffff700 1 -- 172.21.15.114:0/2486832650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50c011d920 msgr2=0x7f50c01198e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:42.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.309+0000 7f50bffff700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50c011d920 0x7f50c01198e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:42.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.309+0000 7f50bffff700 1 -- 172.21.15.114:0/2486832650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c011a0f0 con 0x7f50c010b270 2024-05-27T07:59:42.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.309+0000 7f50bffff700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c010b270 0x7f50c01193a0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f50b4005910 tx=0x7f50b4005940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:42.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.310+0000 7f50bdffb700 1 -- 172.21.15.114:0/2486832650 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b4025070 con 0x7f50c010b270 2024-05-27T07:59:42.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.311+0000 7f50bdffb700 1 -- 172.21.15.114:0/2486832650 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50b4003680 con 0x7f50c010b270 2024-05-27T07:59:42.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.311+0000 7f50c6b1c700 1 -- 172.21.15.114:0/2486832650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50c01ad8b0 con 0x7f50c010b270 2024-05-27T07:59:42.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.311+0000 7f50bdffb700 1 -- 172.21.15.114:0/2486832650 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b400b760 con 0x7f50c010b270 2024-05-27T07:59:42.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.311+0000 7f50c6b1c700 1 -- 172.21.15.114:0/2486832650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50c01add60 con 0x7f50c010b270 2024-05-27T07:59:42.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.320+0000 7f50bdffb700 1 -- 172.21.15.114:0/2486832650 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f50b4028070 con 0x7f50c010b270 2024-05-27T07:59:42.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.322+0000 7f50bdffb700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f50a80b2bc0 0x7f50a80b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:42.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.322+0000 7f50bdffb700 1 -- 172.21.15.114:0/2486832650 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9510+0+0 (secure 0 0 0) 0x7f50b40e34b0 con 0x7f50c010b270 2024-05-27T07:59:42.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.323+0000 7f50c48b8700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f50a80b2bc0 0x7f50a80b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:42.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.323+0000 7f50c48b8700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f50a80b2bc0 0x7f50a80b50d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009c40 tx=0x7f50b0009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:42.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.327+0000 7f50a77fe700 1 -- 172.21.15.114:0/2486832650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50c010c060 con 0x7f50c010b270 2024-05-27T07:59:42.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.335+0000 7f50bdffb700 1 -- 172.21.15.114:0/2486832650 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f50b401d070 con 0x7f50c010b270 2024-05-27T07:59:42.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 83027 12.14 MB/sec execute 234 sec latency 227.849 ms 2024-05-27T07:59:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.627+0000 7f50a77fe700 1 -- 172.21.15.114:0/2486832650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50c00512b0 con 0x7f50c010b270 2024-05-27T07:59:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.628+0000 7f50bdffb700 1 -- 172.21.15.114:0/2486832650 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5206 (secure 0 0 0) 0x7f50b4013030 con 0x7f50c010b270 2024-05-27T07:59:42.630 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:59:42.630 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":111,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:59:34.461413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_19494":{"gid":19494,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6841/584614487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":584614487},{"type":"v1","addr":"172.21.15.114:6841","nonce":584614487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:59:42.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.630+0000 7f50a77fe700 1 -- 172.21.15.114:0/2486832650 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f50a80b2bc0 msgr2=0x7f50a80b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.630+0000 7f50a77fe700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f50a80b2bc0 0x7f50a80b50d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009c40 tx=0x7f50b0009260 comp rx=0 tx=0).stop 2024-05-27T07:59:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.630+0000 7f50a77fe700 1 -- 172.21.15.114:0/2486832650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c010b270 msgr2=0x7f50c01193a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.630+0000 7f50a77fe700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c010b270 0x7f50c01193a0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f50b4005910 tx=0x7f50b4005940 comp rx=0 tx=0).stop 2024-05-27T07:59:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.630+0000 7f50a77fe700 1 -- 172.21.15.114:0/2486832650 shutdown_connections 2024-05-27T07:59:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.630+0000 7f50a77fe700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f50a80b2bc0 0x7f50a80b50d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.630+0000 7f50a77fe700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50c011d920 0x7f50c01198e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.630+0000 7f50a77fe700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50c010a920 0x7f50c011d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.630+0000 7f50a77fe700 1 --2- 172.21.15.114:0/2486832650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c010b270 0x7f50c01193a0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.630+0000 7f50a77fe700 1 -- 172.21.15.114:0/2486832650 >> 172.21.15.114:0/2486832650 conn(0x7f50c006f4c0 msgr2=0x7f50c011e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.631+0000 7f50a77fe700 1 -- 172.21.15.114:0/2486832650 shutdown_connections 2024-05-27T07:59:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:42.631+0000 7f50a77fe700 1 -- 172.21.15.114:0/2486832650 wait complete. 2024-05-27T07:59:42.634 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 111 2024-05-27T07:59:43.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:42 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2486832650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:59:43.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:42 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2486832650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:59:43.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:42 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2486832650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:59:43.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:59:43.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 83338 12.18 MB/sec execute 235 sec latency 162.548 ms 2024-05-27T07:59:43.683 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:59:44.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:43 smithi114 ceph-mon[32038]: pgmap v690: 129 pgs: 129 active+clean; 388 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.3 MiB/s wr, 519 op/s 2024-05-27T07:59:44.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:43 smithi163 ceph-mon[36451]: pgmap v690: 129 pgs: 129 active+clean; 388 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.3 MiB/s wr, 519 op/s 2024-05-27T07:59:44.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:43 smithi186 ceph-mon[36647]: pgmap v690: 129 pgs: 129 active+clean; 388 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.3 MiB/s wr, 519 op/s 2024-05-27T07:59:44.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 83595 12.17 MB/sec execute 236 sec latency 217.976 ms 2024-05-27T07:59:45.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.535+0000 7f4e0fe16700 1 -- 172.21.15.114:0/4005791448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e08105470 msgr2=0x7f4e08105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:45.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.535+0000 7f4e0fe16700 1 --2- 172.21.15.114:0/4005791448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e08105470 0x7f4e08105880 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f4e04009990 tx=0x7f4e04009ca0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.535+0000 7f4e0fe16700 1 -- 172.21.15.114:0/4005791448 shutdown_connections 2024-05-27T07:59:45.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.535+0000 7f4e0fe16700 1 --2- 172.21.15.114:0/4005791448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e0810e8e0 0x7f4e08110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.535+0000 7f4e0fe16700 1 --2- 172.21.15.114:0/4005791448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e08105470 0x7f4e08105880 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.535+0000 7f4e0fe16700 1 --2- 172.21.15.114:0/4005791448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e08105dc0 0x7f4e0810e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.535+0000 7f4e0fe16700 1 -- 172.21.15.114:0/4005791448 >> 172.21.15.114:0/4005791448 conn(0x7f4e080fef20 msgr2=0x7f4e08101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:45.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.535+0000 7f4e0fe16700 1 -- 172.21.15.114:0/4005791448 shutdown_connections 2024-05-27T07:59:45.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.536+0000 7f4e0fe16700 1 -- 172.21.15.114:0/4005791448 wait complete. 2024-05-27T07:59:45.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0fe16700 1 Processor -- start 2024-05-27T07:59:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0fe16700 1 -- start start 2024-05-27T07:59:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0fe16700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e08105470 0x7f4e0819ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0fe16700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e08105dc0 0x7f4e0819b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0fe16700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e0810e8e0 0x7f4e0819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0fe16700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4e08114430 con 0x7f4e08105470 2024-05-27T07:59:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0fe16700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4e081142b0 con 0x7f4e0810e8e0 2024-05-27T07:59:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0fe16700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4e081145b0 con 0x7f4e08105dc0 2024-05-27T07:59:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0dbb2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e08105470 0x7f4e0819ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0dbb2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e08105470 0x7f4e0819ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58334/0 (socket says 172.21.15.114:58334) 2024-05-27T07:59:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0dbb2700 1 -- 172.21.15.114:0/2444083829 learned_addr learned my addr 172.21.15.114:0/2444083829 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0d3b1700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e08105dc0 0x7f4e0819b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0e3b3700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e0810e8e0 0x7f4e0819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0dbb2700 1 -- 172.21.15.114:0/2444083829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e08105dc0 msgr2=0x7f4e0819b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:45.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e08105dc0 0x7f4e0819b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0dbb2700 1 -- 172.21.15.114:0/2444083829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e0810e8e0 msgr2=0x7f4e0819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:45.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.537+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e0810e8e0 0x7f4e0819f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.538+0000 7f4e0dbb2700 1 -- 172.21.15.114:0/2444083829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0819fcf0 con 0x7f4e08105470 2024-05-27T07:59:45.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.538+0000 7f4e0e3b3700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e0810e8e0 0x7f4e0819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:59:45.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.538+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e08105470 0x7f4e0819ae20 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000c00 tx=0x7f4e04009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:45.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.538+0000 7f4dfeffd700 1 -- 172.21.15.114:0/2444083829 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0401ec90 con 0x7f4e08105470 2024-05-27T07:59:45.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.538+0000 7f4e0fe16700 1 -- 172.21.15.114:0/2444083829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0819ff80 con 0x7f4e08105470 2024-05-27T07:59:45.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.538+0000 7f4dfeffd700 1 -- 172.21.15.114:0/2444083829 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e0401ee30 con 0x7f4e08105470 2024-05-27T07:59:45.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.538+0000 7f4dfeffd700 1 -- 172.21.15.114:0/2444083829 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04028920 con 0x7f4e08105470 2024-05-27T07:59:45.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.538+0000 7f4e0fe16700 1 -- 172.21.15.114:0/2444083829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e081a0400 con 0x7f4e08105470 2024-05-27T07:59:45.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.541+0000 7f4dfeffd700 1 -- 172.21.15.114:0/2444083829 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f4e04009d10 con 0x7f4e08105470 2024-05-27T07:59:45.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.542+0000 7f4dfeffd700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4df40b29f0 0x7f4df40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.542+0000 7f4e0d3b1700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4df40b29f0 0x7f4df40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.542+0000 7f4dfeffd700 1 -- 172.21.15.114:0/2444083829 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9510+0+0 (secure 0 0 0) 0x7f4e040e7e10 con 0x7f4e08105470 2024-05-27T07:59:45.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.543+0000 7f4e1e82bb80 1 Processor -- start 2024-05-27T07:59:45.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.543+0000 7f4e0d3b1700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4df40b29f0 0x7f4df40b4f00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4e08102400 tx=0x7f4df8009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:45.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.543+0000 7f4e1e82bb80 1 -- start start 2024-05-27T07:59:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.543+0000 7f4e1e82bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.543+0000 7f4e1e82bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5bfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.543+0000 7f4e1e82bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5bff80 0x55daba5c0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.543+0000 7f4e1e82bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55daba4eec90 con 0x55daba5c5c50 2024-05-27T07:59:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.543+0000 7f4e1e82bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55daba4de830 con 0x55daba5bff80 2024-05-27T07:59:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.543+0000 7f4e1e82bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55daba4e2870 con 0x55daba441e40 2024-05-27T07:59:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.543+0000 7f4e0dbb2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.543+0000 7f4e0d3b1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5bfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.543+0000 7f4e0dbb2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58758/0 (socket says 172.21.15.114:58758) 2024-05-27T07:59:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.543+0000 7f4e0dbb2700 1 -- 172.21.15.114:0/2680569888 learned_addr learned my addr 172.21.15.114:0/2680569888 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.544+0000 7f4e0e3b3700 1 --2- 172.21.15.114:0/2680569888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5bff80 0x55daba5c0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.544+0000 7f4e0dbb2700 1 -- 172.21.15.114:0/2680569888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5bff80 msgr2=0x55daba5c0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.544+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/2680569888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5bff80 0x55daba5c0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.544+0000 7f4e0dbb2700 1 -- 172.21.15.114:0/2680569888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 msgr2=0x55daba5bfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.544+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/2680569888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5bfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.544+0000 7f4e0dbb2700 1 -- 172.21.15.114:0/2680569888 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daba42e8d0 con 0x55daba441e40 2024-05-27T07:59:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.544+0000 7f4e0d3b1700 1 --2- 172.21.15.114:0/2680569888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5bfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.544+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/2680569888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5c5710 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4e040aed30 tx=0x7f4e04009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=851807aec4e2d801 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.544+0000 7f4df3fff700 1 -- 172.21.15.114:0/2680569888 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e040af4a0 con 0x55daba441e40 2024-05-27T07:59:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.544+0000 7f4df3fff700 1 -- 172.21.15.114:0/2680569888 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e040aeda0 con 0x55daba441e40 2024-05-27T07:59:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.544+0000 7f4df3fff700 1 -- 172.21.15.114:0/2680569888 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e040b7940 con 0x55daba441e40 2024-05-27T07:59:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.545+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/2680569888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 msgr2=0x55daba5c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.545+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/2680569888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5c5710 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4e040aed30 tx=0x7f4e04009310 comp rx=0 tx=0).stop 2024-05-27T07:59:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.545+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/2680569888 shutdown_connections 2024-05-27T07:59:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.545+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/2680569888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5bff80 0x55daba5c0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.545+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/2680569888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5c5710 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.545+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/2680569888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5bfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.545+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/2680569888 >> 172.21.15.114:0/2680569888 conn(0x55daba49caf0 msgr2=0x55daba5ba640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.545+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/2680569888 shutdown_connections 2024-05-27T07:59:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.546+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/2680569888 wait complete. 2024-05-27T07:59:45.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.546+0000 7f4e1e82bb80 1 -- start start 2024-05-27T07:59:45.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.546+0000 7f4e1e82bb80 1 Processor -- start 2024-05-27T07:59:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e1e82bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e1e82bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5bff80 0x55daba5c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e1e82bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e1e82bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55daba3e4fa0 con 0x55daba5c5c50 2024-05-27T07:59:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e1e82bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55daba4807c0 con 0x55daba5bff80 2024-05-27T07:59:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e1e82bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55daba4e2870 con 0x55daba441e40 2024-05-27T07:59:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e1e82bb80 10 client.0 _ll_register_callbacks cb 0x55daba479920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e0e3b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e0e3b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5c9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58356/0 (socket says 172.21.15.114:58356) 2024-05-27T07:59:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e0e3b3700 1 -- 172.21.15.114:0/3959318948 learned_addr learned my addr 172.21.15.114:0/3959318948 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e0d3b1700 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5bff80 0x55daba5c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e0d3b1700 1 -- 172.21.15.114:0/3959318948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 msgr2=0x55daba5c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e0d3b1700 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e0d3b1700 1 -- 172.21.15.114:0/3959318948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 msgr2=0x55daba5c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e0d3b1700 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e0d3b1700 1 -- 172.21.15.114:0/3959318948 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daba47d690 con 0x55daba5bff80 2024-05-27T07:59:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.547+0000 7f4e0e3b3700 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.548+0000 7f4e0d3b1700 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5bff80 0x55daba5c9a20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4df80097c0 tx=0x7f4df80047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a273ddce400b7d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.548+0000 7f4df17fa700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:59:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.548+0000 7f4df17fa700 1 -- 172.21.15.114:0/3959318948 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df80044f0 con 0x55daba5bff80 2024-05-27T07:59:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.548+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3959318948 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e040aeda0 con 0x55daba5bff80 2024-05-27T07:59:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.548+0000 7f4df17fa700 1 -- 172.21.15.114:0/3959318948 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9510+0+0 (secure 0 0 0) 0x7f4df801d410 con 0x55daba5bff80 2024-05-27T07:59:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.548+0000 7f4df17fa700 1 -- 172.21.15.114:0/3959318948 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7f4df802c690 con 0x55daba5bff80 2024-05-27T07:59:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.548+0000 7f4e1e82bb80 10 client.29147 fetch_fsmap learned FSMap version 111 2024-05-27T07:59:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.548+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3959318948 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=4+}) -- 0x55daba5d6f20 con 0x55daba5bff80 2024-05-27T07:59:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.548+0000 7f4df17fa700 1 -- 172.21.15.114:0/3959318948 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e040aeda0 con 0x55daba5bff80 2024-05-27T07:59:45.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.549+0000 7f4df17fa700 1 -- 172.21.15.114:0/3959318948 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 111) ==== 3490+0+0 (secure 0 0 0) 0x7f4df801d610 con 0x55daba5bff80 2024-05-27T07:59:45.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.549+0000 7f4e1e82bb80 10 client.29147 fetch_fsmap finished waiting for FSMap version 111 2024-05-27T07:59:45.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.549+0000 7f4e1e82bb80 10 client.29147 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:45.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.549+0000 7f4e1e82bb80 20 client.29147 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:45.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.549+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55daba5d9900 0x55daba5dbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.549+0000 7f4e1e82bb80 4 client.29147 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:59:45.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.549+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3959318948 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55daba58b9f0 con 0x55daba5d9900 2024-05-27T07:59:45.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.549+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55daba5d9900 0x55daba5dbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.549+0000 7f4df17fa700 10 client.29147 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:45.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.550+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55daba5d9900 0x55daba5dbd50 crc :-1 s=READY pgs=520 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34a3f95b20005e99 server_cookie=81cfb843d1581eb5 in_seq=0 out_seq=0 2024-05-27T07:59:45.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.552+0000 7f4df17fa700 1 -- 172.21.15.114:0/3959318948 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55daba58b9f0 con 0x55daba5d9900 2024-05-27T07:59:45.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.552+0000 7f4df17fa700 10 client.29147 handle_command_reply: tid=0 2024-05-27T07:59:45.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.553+0000 7f4e1e82bb80 1 client.29147 shutdown 2024-05-27T07:59:45.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.553+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3959318948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5bff80 msgr2=0x55daba5c9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:45.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.553+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5bff80 0x55daba5c9a20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4df80097c0 tx=0x7f4df80047c0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.553+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3959318948 shutdown_connections 2024-05-27T07:59:45.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.553+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.553+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.553+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55daba5d9900 0x55daba5dbd50 crc :-1 s=READY pgs=520 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.553+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/3959318948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5bff80 0x55daba5c9a20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.553+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3959318948 >> 172.21.15.114:0/3959318948 conn(0x55daba49caf0 msgr2=0x55daba5bea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:45.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.553+0000 7f4df17fa700 0 client.29147 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:45.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.553+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3959318948 shutdown_connections 2024-05-27T07:59:45.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.553+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3959318948 wait complete. 2024-05-27T07:59:45.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.553+0000 7f4e1e82bb80 20 client.29147 trim_cache size 0 max 16384 2024-05-27T07:59:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.572+0000 7f4e1e82bb80 1 Processor -- start 2024-05-27T07:59:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.572+0000 7f4e1e82bb80 1 -- start start 2024-05-27T07:59:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.572+0000 7f4e1e82bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5bfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.572+0000 7f4e1e82bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5bff80 0x55daba5c7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e1e82bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5c5c50 0x55daba5c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e1e82bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4e081ad980 con 0x55daba5bff80 2024-05-27T07:59:45.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e1e82bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4e081ae050 con 0x55daba5c5c50 2024-05-27T07:59:45.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e1e82bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4e081ac330 con 0x55daba441e40 2024-05-27T07:59:45.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e0dbb2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5bfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e0d3b1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5bff80 0x55daba5c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e0dbb2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5bfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58770/0 (socket says 172.21.15.114:58770) 2024-05-27T07:59:45.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e0dbb2700 1 -- 172.21.15.114:0/1357088790 learned_addr learned my addr 172.21.15.114:0/1357088790 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:45.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e0d3b1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5bff80 0x55daba5c7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58358/0 (socket says 172.21.15.114:58358) 2024-05-27T07:59:45.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e0e3b3700 1 --2- 172.21.15.114:0/1357088790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5c5c50 0x55daba5c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e0dbb2700 1 -- 172.21.15.114:0/1357088790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5c5c50 msgr2=0x55daba5c9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/1357088790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5c5c50 0x55daba5c9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e0dbb2700 1 -- 172.21.15.114:0/1357088790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5bff80 msgr2=0x55daba5c7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/1357088790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5bff80 0x55daba5c7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e0dbb2700 1 -- 172.21.15.114:0/1357088790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daba5c8800 con 0x55daba441e40 2024-05-27T07:59:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e0d3b1700 1 --2- 172.21.15.114:0/1357088790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5bff80 0x55daba5c7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.573+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/1357088790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5bfa40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4e04026000 tx=0x7f4e04021000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d97e02435998a198 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.574+0000 7f4df37fe700 1 -- 172.21.15.114:0/1357088790 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04074070 con 0x55daba441e40 2024-05-27T07:59:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.574+0000 7f4df37fe700 1 -- 172.21.15.114:0/1357088790 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e040b3150 con 0x55daba441e40 2024-05-27T07:59:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.574+0000 7f4df37fe700 1 -- 172.21.15.114:0/1357088790 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e040b2040 con 0x55daba441e40 2024-05-27T07:59:45.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.574+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/1357088790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 msgr2=0x55daba5bfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:45.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.574+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/1357088790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5bfa40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4e04026000 tx=0x7f4e04021000 comp rx=0 tx=0).stop 2024-05-27T07:59:45.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.575+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/1357088790 shutdown_connections 2024-05-27T07:59:45.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.575+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/1357088790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba5c5c50 0x55daba5c9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.575+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/1357088790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba441e40 0x55daba5bfa40 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.575+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/1357088790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5bff80 0x55daba5c7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.575+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/1357088790 >> 172.21.15.114:0/1357088790 conn(0x55daba49caf0 msgr2=0x55daba5ba1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:45.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.575+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/1357088790 shutdown_connections 2024-05-27T07:59:45.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.575+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/1357088790 wait complete. 2024-05-27T07:59:45.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.575+0000 7f4e1e82bb80 1 -- start start 2024-05-27T07:59:45.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.575+0000 7f4e1e82bb80 1 Processor -- start 2024-05-27T07:59:45.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e1e82bb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba441e40 0x55daba5e1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e1e82bb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba5bff80 0x55daba5e1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e1e82bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5d1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e1e82bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55daba5bbc80 con 0x55daba5c5c50 2024-05-27T07:59:45.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e1e82bb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4e081ac980 con 0x55daba441e40 2024-05-27T07:59:45.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e1e82bb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55daba5ca2e0 con 0x55daba5bff80 2024-05-27T07:59:45.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e1e82bb80 10 client.0 _ll_register_callbacks cb 0x55daba479920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:45.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e0d3b1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba5bff80 0x55daba5e1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e0d3b1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba5bff80 0x55daba5e1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58778/0 (socket says 172.21.15.114:58778) 2024-05-27T07:59:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e0e3b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5d1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e0d3b1700 1 -- 172.21.15.114:0/3944376160 learned_addr learned my addr 172.21.15.114:0/3944376160 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/3944376160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba441e40 0x55daba5e1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e0d3b1700 1 -- 172.21.15.114:0/3944376160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba441e40 msgr2=0x55daba5e1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e0d3b1700 1 --2- 172.21.15.114:0/3944376160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba441e40 0x55daba5e1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e0d3b1700 1 -- 172.21.15.114:0/3944376160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 msgr2=0x55daba5d1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e0d3b1700 1 --2- 172.21.15.114:0/3944376160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5d1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e0d3b1700 1 -- 172.21.15.114:0/3944376160 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daba5d1d30 con 0x55daba5bff80 2024-05-27T07:59:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/3944376160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba441e40 0x55daba5e1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:59:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4e0d3b1700 1 --2- 172.21.15.114:0/3944376160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba5bff80 0x55daba5e1d50 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x55daba5b2b40 tx=0x7f4df8004720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1dbcc9cc340f48c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.576+0000 7f4df17fa700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T07:59:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4df17fa700 1 -- 172.21.15.114:0/3944376160 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df802c4f0 con 0x55daba5bff80 2024-05-27T07:59:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4df17fa700 1 -- 172.21.15.114:0/3944376160 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9510+0+0 (secure 0 0 0) 0x7f4df801d620 con 0x55daba5bff80 2024-05-27T07:59:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3944376160 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55daba4f0be0 con 0x55daba5bff80 2024-05-27T07:59:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4df17fa700 1 -- 172.21.15.114:0/3944376160 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7f4df801d820 con 0x55daba5bff80 2024-05-27T07:59:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4e1e82bb80 10 client.29308 fetch_fsmap learned FSMap version 111 2024-05-27T07:59:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3944376160 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=4+}) -- 0x55daba5e1230 con 0x55daba5bff80 2024-05-27T07:59:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4df17fa700 1 -- 172.21.15.114:0/3944376160 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55daba4f0be0 con 0x55daba5bff80 2024-05-27T07:59:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4df17fa700 1 -- 172.21.15.114:0/3944376160 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 111) ==== 3490+0+0 (secure 0 0 0) 0x7f4df80358a0 con 0x55daba5bff80 2024-05-27T07:59:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4e1e82bb80 10 client.29308 fetch_fsmap finished waiting for FSMap version 111 2024-05-27T07:59:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4e1e82bb80 10 client.29308 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4e1e82bb80 20 client.29308 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/3944376160 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55daba643850 0x55daba645c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4e1e82bb80 4 client.29308 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:59:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.577+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3944376160 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55daba4ea3c0 con 0x55daba643850 2024-05-27T07:59:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.578+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/3944376160 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55daba643850 0x55daba645c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.578+0000 7f4df17fa700 10 client.29308 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.578+0000 7f4e0dbb2700 1 --2- 172.21.15.114:0/3944376160 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55daba643850 0x55daba645c70 crc :-1 s=READY pgs=521 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b82bf019b8b0384 server_cookie=55b9f8a6f200b3ce in_seq=0 out_seq=0 2024-05-27T07:59:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.579+0000 7f4df17fa700 1 -- 172.21.15.114:0/3944376160 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55daba4ea3c0 con 0x55daba643850 2024-05-27T07:59:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.579+0000 7f4df17fa700 10 client.29308 handle_command_reply: tid=0 2024-05-27T07:59:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.580+0000 7f4e1e82bb80 1 client.29308 shutdown 2024-05-27T07:59:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.580+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3944376160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba5bff80 msgr2=0x55daba5e1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.580+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/3944376160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba5bff80 0x55daba5e1d50 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x55daba5b2b40 tx=0x7f4df8004720 comp rx=0 tx=0).stop 2024-05-27T07:59:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.580+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3944376160 shutdown_connections 2024-05-27T07:59:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.580+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/3944376160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55daba5c5c50 0x55daba5d1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.580+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/3944376160 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55daba643850 0x55daba645c70 crc :-1 s=READY pgs=521 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.580+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/3944376160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daba441e40 0x55daba5e1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.580+0000 7f4e1e82bb80 1 --2- 172.21.15.114:0/3944376160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55daba5bff80 0x55daba5e1d50 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.580+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3944376160 >> 172.21.15.114:0/3944376160 conn(0x55daba49caf0 msgr2=0x55daba5c79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:45.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.580+0000 7f4df17fa700 0 client.29308 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:45.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.580+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3944376160 shutdown_connections 2024-05-27T07:59:45.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.580+0000 7f4e1e82bb80 1 -- 172.21.15.114:0/3944376160 wait complete. 2024-05-27T07:59:45.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.580+0000 7f4e1e82bb80 20 client.29308 trim_cache size 0 max 16384 2024-05-27T07:59:45.588 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T07:59:45.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.581+0000 7f4e0fe16700 1 -- 172.21.15.114:0/2444083829 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4df40b29f0 msgr2=0x7f4df40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:45.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.581+0000 7f4e0fe16700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4df40b29f0 0x7f4df40b4f00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4e08102400 tx=0x7f4df8009260 comp rx=0 tx=0).stop 2024-05-27T07:59:45.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.581+0000 7f4e0fe16700 1 -- 172.21.15.114:0/2444083829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e08105470 msgr2=0x7f4e0819ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:45.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.581+0000 7f4e0fe16700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e08105470 0x7f4e0819ae20 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000c00 tx=0x7f4e04009580 comp rx=0 tx=0).stop 2024-05-27T07:59:45.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.581+0000 7f4e0fe16700 1 -- 172.21.15.114:0/2444083829 shutdown_connections 2024-05-27T07:59:45.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.581+0000 7f4e0fe16700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f4df40b29f0 0x7f4df40b4f00 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.581+0000 7f4e0fe16700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e0810e8e0 0x7f4e0819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.581+0000 7f4e0fe16700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e08105470 0x7f4e0819ae20 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.581+0000 7f4e0fe16700 1 --2- 172.21.15.114:0/2444083829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e08105dc0 0x7f4e0819b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:45.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.581+0000 7f4e0fe16700 1 -- 172.21.15.114:0/2444083829 >> 172.21.15.114:0/2444083829 conn(0x7f4e080fef20 msgr2=0x7f4e08100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:45.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.581+0000 7f4e0fe16700 1 -- 172.21.15.114:0/2444083829 shutdown_connections 2024-05-27T07:59:45.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:45.581+0000 7f4e0fe16700 1 -- 172.21.15.114:0/2444083829 wait complete. 2024-05-27T07:59:45.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 83886 12.15 MB/sec execute 237 sec latency 193.444 ms 2024-05-27T07:59:46.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:45 smithi163 ceph-mon[36451]: pgmap v691: 129 pgs: 129 active+clean; 392 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.3 MiB/s wr, 475 op/s 2024-05-27T07:59:46.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:45 smithi186 ceph-mon[36647]: pgmap v691: 129 pgs: 129 active+clean; 392 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.3 MiB/s wr, 475 op/s 2024-05-27T07:59:46.272 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:45 smithi114 ceph-mon[32038]: pgmap v691: 129 pgs: 129 active+clean; 392 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.3 MiB/s wr, 475 op/s 2024-05-27T07:59:46.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 84200 12.16 MB/sec execute 238 sec latency 170.745 ms 2024-05-27T07:59:47.305 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:59:47.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:59:47.351 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:59:47.351 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 systemd[1]: Stopped Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:59:47.351 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 systemd[1]: Starting Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T07:59:47.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 84530 12.21 MB/sec execute 239 sec latency 161.517 ms 2024-05-27T07:59:47.710 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 podman[134159]: 2024-05-27 07:59:47.270427869 +0000 UTC m=+0.061172118 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:59:47.710 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 podman[134159]: 2024-05-27 07:59:47.455946555 +0000 UTC m=+0.246690782 container create aff38ba4d6ea90b61c26f853ab2ff0dce78166479b5ca8ecac3913436142cbb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:59:48.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-mon[32038]: daemon mds.a restarted 2024-05-27T07:59:48.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 podman[134159]: 2024-05-27 07:59:47.709198208 +0000 UTC m=+0.499942446 container init aff38ba4d6ea90b61c26f853ab2ff0dce78166479b5ca8ecac3913436142cbb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-27T07:59:48.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 podman[134159]: 2024-05-27 07:59:47.716131797 +0000 UTC m=+0.506876026 container start aff38ba4d6ea90b61c26f853ab2ff0dce78166479b5ca8ecac3913436142cbb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T07:59:48.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-mds[134221]: -- 172.21.15.114:0/2409091594 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5627474dfa40 con 0x562747527180 2024-05-27T07:59:48.058 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: starting mds.a at 2024-05-27T07:59:48.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 bash[134159]: aff38ba4d6ea90b61c26f853ab2ff0dce78166479b5ca8ecac3913436142cbb8 2024-05-27T07:59:48.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 systemd[1]: Started Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T07:59:48.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff510074700 time 2024-05-27T07:59:47.870026+0000 2024-05-27T07:59:48.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:48.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:48.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff5228b9ba6] 2024-05-27T07:59:48.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff5228b9dd1] 2024-05-27T07:59:48.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56274683e23d] 2024-05-27T07:59:48.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562746829971] 2024-05-27T07:59:48.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 5: (SnapRealm::check_cache() const+0x13e) [0x562746827fa0] 2024-05-27T07:59:48.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562746828651] 2024-05-27T07:59:48.059 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627467bd347] 2024-05-27T07:59:48.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627464b987e] 2024-05-27T07:59:48.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5627464bef2f] 2024-05-27T07:59:48.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5627464c9969] 2024-05-27T07:59:48.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 11: (MDLog::_replay_thread()+0x1f38) [0x56274686248c] 2024-05-27T07:59:48.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56274686b9cb] 2024-05-27T07:59:48.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 13: (Thread::entry_wrapper()+0x43) [0x7ff52288872d] 2024-05-27T07:59:48.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff522888749] 2024-05-27T07:59:48.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff52142c1ca] 2024-05-27T07:59:48.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 16: clone() 2024-05-27T07:59:48.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:47.871+0000 7ff510074700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff510074700 time 2024-05-27T07:59:47.870026+0000 2024-05-27T07:59:48.060 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:48.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:48.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:48.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff5228b9ba6] 2024-05-27T07:59:48.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff5228b9dd1] 2024-05-27T07:59:48.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56274683e23d] 2024-05-27T07:59:48.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562746829971] 2024-05-27T07:59:48.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 5: (SnapRealm::check_cache() const+0x13e) [0x562746827fa0] 2024-05-27T07:59:48.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562746828651] 2024-05-27T07:59:48.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627467bd347] 2024-05-27T07:59:48.061 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627464b987e] 2024-05-27T07:59:48.062 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5627464bef2f] 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5627464c9969] 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 11: (MDLog::_replay_thread()+0x1f38) [0x56274686248c] 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56274686b9cb] 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 13: (Thread::entry_wrapper()+0x43) [0x7ff52288872d] 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff522888749] 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff52142c1ca] 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 16: clone() 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: *** Caught signal (Aborted) ** 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: in thread 7ff510074700 thread_name:md_log_replay 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff521436d20] 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2: gsignal() 2024-05-27T07:59:48.063 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 3: abort() 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff5228b9ced] 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff5228b9dd1] 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56274683e23d] 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562746829971] 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 8: (SnapRealm::check_cache() const+0x13e) [0x562746827fa0] 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562746828651] 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627467bd347] 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627464b987e] 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5627464bef2f] 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5627464c9969] 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 14: (MDLog::_replay_thread()+0x1f38) [0x56274686248c] 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56274686b9cb] 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 16: (Thread::entry_wrapper()+0x43) [0x7ff52288872d] 2024-05-27T07:59:48.064 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff522888749] 2024-05-27T07:59:48.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff52142c1ca] 2024-05-27T07:59:48.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 19: clone() 2024-05-27T07:59:48.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:47.873+0000 7ff510074700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:59:48.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: in thread 7ff510074700 thread_name:md_log_replay 2024-05-27T07:59:48.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:48.065 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff521436d20] 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2: gsignal() 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 3: abort() 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff5228b9ced] 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff5228b9dd1] 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56274683e23d] 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562746829971] 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 8: (SnapRealm::check_cache() const+0x13e) [0x562746827fa0] 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562746828651] 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627467bd347] 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627464b987e] 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5627464bef2f] 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5627464c9969] 2024-05-27T07:59:48.066 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 14: (MDLog::_replay_thread()+0x1f38) [0x56274686248c] 2024-05-27T07:59:48.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56274686b9cb] 2024-05-27T07:59:48.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 16: (Thread::entry_wrapper()+0x43) [0x7ff52288872d] 2024-05-27T07:59:48.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff522888749] 2024-05-27T07:59:48.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff52142c1ca] 2024-05-27T07:59:48.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 19: clone() 2024-05-27T07:59:48.067 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:59:48.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:48.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: -1> 2024-05-27T07:59:47.871+0000 7ff510074700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff510074700 time 2024-05-27T07:59:47.870026+0000 2024-05-27T07:59:48.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:48.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:48.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:48.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff5228b9ba6] 2024-05-27T07:59:48.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff5228b9dd1] 2024-05-27T07:59:48.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56274683e23d] 2024-05-27T07:59:48.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562746829971] 2024-05-27T07:59:48.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 5: (SnapRealm::check_cache() const+0x13e) [0x562746827fa0] 2024-05-27T07:59:48.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562746828651] 2024-05-27T07:59:48.068 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627467bd347] 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627464b987e] 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5627464bef2f] 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5627464c9969] 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 11: (MDLog::_replay_thread()+0x1f38) [0x56274686248c] 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56274686b9cb] 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 13: (Thread::entry_wrapper()+0x43) [0x7ff52288872d] 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff522888749] 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff52142c1ca] 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 16: clone() 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 0> 2024-05-27T07:59:47.873+0000 7ff510074700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: in thread 7ff510074700 thread_name:md_log_replay 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:48.069 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff521436d20] 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2: gsignal() 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 3: abort() 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff5228b9ced] 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff5228b9dd1] 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56274683e23d] 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562746829971] 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 8: (SnapRealm::check_cache() const+0x13e) [0x562746827fa0] 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562746828651] 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627467bd347] 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627464b987e] 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5627464bef2f] 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5627464c9969] 2024-05-27T07:59:48.070 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 14: (MDLog::_replay_thread()+0x1f38) [0x56274686248c] 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56274686b9cb] 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 16: (Thread::entry_wrapper()+0x43) [0x7ff52288872d] 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff522888749] 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff52142c1ca] 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 19: clone() 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: -2> 2024-05-27T07:59:47.871+0000 7ff510074700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff510074700 time 2024-05-27T07:59:47.870026+0000 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff5228b9ba6] 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff5228b9dd1] 2024-05-27T07:59:48.071 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56274683e23d] 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562746829971] 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 5: (SnapRealm::check_cache() const+0x13e) [0x562746827fa0] 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562746828651] 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627467bd347] 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627464b987e] 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5627464bef2f] 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5627464c9969] 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 11: (MDLog::_replay_thread()+0x1f38) [0x56274686248c] 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56274686b9cb] 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 13: (Thread::entry_wrapper()+0x43) [0x7ff52288872d] 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff522888749] 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff52142c1ca] 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 16: clone() 2024-05-27T07:59:48.072 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:48.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: -1> 2024-05-27T07:59:47.873+0000 7ff510074700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:59:48.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: in thread 7ff510074700 thread_name:md_log_replay 2024-05-27T07:59:48.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:48.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:59:48.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff521436d20] 2024-05-27T07:59:48.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2: gsignal() 2024-05-27T07:59:48.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 3: abort() 2024-05-27T07:59:48.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff5228b9ced] 2024-05-27T07:59:48.073 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff5228b9dd1] 2024-05-27T07:59:48.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56274683e23d] 2024-05-27T07:59:48.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562746829971] 2024-05-27T07:59:48.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 8: (SnapRealm::check_cache() const+0x13e) [0x562746827fa0] 2024-05-27T07:59:48.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562746828651] 2024-05-27T07:59:48.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627467bd347] 2024-05-27T07:59:48.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627464b987e] 2024-05-27T07:59:48.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5627464bef2f] 2024-05-27T07:59:48.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5627464c9969] 2024-05-27T07:59:48.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 14: (MDLog::_replay_thread()+0x1f38) [0x56274686248c] 2024-05-27T07:59:48.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56274686b9cb] 2024-05-27T07:59:48.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 16: (Thread::entry_wrapper()+0x43) [0x7ff52288872d] 2024-05-27T07:59:48.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff522888749] 2024-05-27T07:59:48.074 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff52142c1ca] 2024-05-27T07:59:48.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 19: clone() 2024-05-27T07:59:48.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:59:48.075 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a[134211]: 2024-05-27T07:59:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:59:48.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:47 smithi163 ceph-mon[36451]: daemon mds.a restarted 2024-05-27T07:59:48.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:47 smithi186 ceph-mon[36647]: daemon mds.a restarted 2024-05-27T07:59:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.356+0000 7ff1f4813640 1 -- 172.21.15.114:0/630571464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1f010e200 msgr2=0x7ff1f01106f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.356+0000 7ff1f4813640 1 --2- 172.21.15.114:0/630571464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1f010e200 0x7ff1f01106f0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8009ca0 tx=0x7ff1e802a9e0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.356+0000 7ff1f4813640 1 -- 172.21.15.114:0/630571464 shutdown_connections 2024-05-27T07:59:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.356+0000 7ff1f4813640 1 --2- 172.21.15.114:0/630571464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1f010e200 0x7ff1f01106f0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.356+0000 7ff1f4813640 1 --2- 172.21.15.114:0/630571464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1f010b850 0x7ff1f010dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.356+0000 7ff1f4813640 1 --2- 172.21.15.114:0/630571464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f0104df0 0x7ff1f0105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.356+0000 7ff1f4813640 1 -- 172.21.15.114:0/630571464 >> 172.21.15.114:0/630571464 conn(0x7ff1f00782d0 msgr2=0x7ff1f0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.356+0000 7ff1f4813640 1 -- 172.21.15.114:0/630571464 shutdown_connections 2024-05-27T07:59:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.356+0000 7ff1f4813640 1 -- 172.21.15.114:0/630571464 wait complete. 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.356+0000 7ff1f4813640 1 Processor -- start 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.356+0000 7ff1f4813640 1 -- start start 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1f4813640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f0104df0 0x7ff1f01a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1f4813640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1f010b850 0x7ff1f01a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1f4813640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1f010e200 0x7ff1f019a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1f4813640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f0113eb0 con 0x7ff1f010b850 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1f4813640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f0113d30 con 0x7ff1f010e200 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1f4813640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f0114030 con 0x7ff1f0104df0 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1edd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1f010b850 0x7ff1f01a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1edd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1f010b850 0x7ff1f01a0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58426/0 (socket says 172.21.15.114:58426) 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1edd9b640 1 -- 172.21.15.114:0/1431642682 learned_addr learned my addr 172.21.15.114:0/1431642682 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1ee59c640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f0104df0 0x7ff1f01a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1eed9d640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1f010e200 0x7ff1f019a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1edd9b640 1 -- 172.21.15.114:0/1431642682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f0104df0 msgr2=0x7ff1f01a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1edd9b640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f0104df0 0x7ff1f01a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1edd9b640 1 -- 172.21.15.114:0/1431642682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1f010e200 msgr2=0x7ff1f019a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1edd9b640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1f010e200 0x7ff1f019a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1edd9b640 1 -- 172.21.15.114:0/1431642682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f019ad80 con 0x7ff1f010b850 2024-05-27T07:59:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1eed9d640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1f010e200 0x7ff1f019a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:59:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1edd9b640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1f010b850 0x7ff1f01a0a10 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400e9f0 tx=0x7ff1e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1df7fe640 1 -- 172.21.15.114:0/1431642682 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e400cd80 con 0x7ff1f010b850 2024-05-27T07:59:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1df7fe640 1 -- 172.21.15.114:0/1431642682 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1e40108e0 con 0x7ff1f010b850 2024-05-27T07:59:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1f4813640 1 -- 172.21.15.114:0/1431642682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1f019b010 con 0x7ff1f010b850 2024-05-27T07:59:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.357+0000 7ff1df7fe640 1 -- 172.21.15.114:0/1431642682 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4018980 con 0x7ff1f010b850 2024-05-27T07:59:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.358+0000 7ff1f4813640 1 -- 172.21.15.114:0/1431642682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1f01025e0 con 0x7ff1f010b850 2024-05-27T07:59:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.360+0000 7ff1df7fe640 1 -- 172.21.15.114:0/1431642682 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ff1e4010430 con 0x7ff1f010b850 2024-05-27T07:59:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.360+0000 7ff1fb79e740 1 Processor -- start 2024-05-27T07:59:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.360+0000 7ff1fb79e740 1 -- start start 2024-05-27T07:59:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.360+0000 7ff1fb79e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd4063d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.360+0000 7ff1fb79e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd406910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.360+0000 7ff1fb79e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 0x5563dd406e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1fb79e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5563dd378c30 con 0x5563dd03c2f0 2024-05-27T07:59:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1fb79e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5563dd3687d0 con 0x5563dd0444d0 2024-05-27T07:59:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1fb79e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5563dd36c810 con 0x5563dd0651b0 2024-05-27T07:59:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1eed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd4063d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1ee59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd406910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1eed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd4063d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58840/0 (socket says 172.21.15.114:58840) 2024-05-27T07:59:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1eed9d640 1 -- 172.21.15.114:0/3092775538 learned_addr learned my addr 172.21.15.114:0/3092775538 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1edd9b640 1 --2- 172.21.15.114:0/3092775538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 0x5563dd406e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1ee59c640 1 -- 172.21.15.114:0/3092775538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 msgr2=0x5563dd4063d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1ee59c640 1 --2- 172.21.15.114:0/3092775538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd4063d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1ee59c640 1 -- 172.21.15.114:0/3092775538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 msgr2=0x5563dd406e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1ee59c640 1 --2- 172.21.15.114:0/3092775538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 0x5563dd406e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1ee59c640 1 -- 172.21.15.114:0/3092775538 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563dd07d210 con 0x5563dd03c2f0 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1df7fe640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff1d80b2950 0x7ff1d80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1eed9d640 1 --2- 172.21.15.114:0/3092775538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd4063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1eed9d640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff1d80b2950 0x7ff1d80b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1ee59c640 1 --2- 172.21.15.114:0/3092775538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd406910 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009c00 tx=0x7ff1e0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1b9e0955ddff90 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1df7fe640 1 -- 172.21.15.114:0/1431642682 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7ff1e40e2d10 con 0x7ff1f010b850 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.361+0000 7ff1dcff9640 1 -- 172.21.15.114:0/3092775538 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0004230 con 0x5563dd03c2f0 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.362+0000 7ff1dcff9640 1 -- 172.21.15.114:0/3092775538 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1e00043d0 con 0x5563dd03c2f0 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.362+0000 7ff1dcff9640 1 -- 172.21.15.114:0/3092775538 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0004c30 con 0x5563dd03c2f0 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.362+0000 7ff1fb79e740 1 -- 172.21.15.114:0/3092775538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 msgr2=0x5563dd406910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.362+0000 7ff1eed9d640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff1d80b2950 0x7ff1d80b4e60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff1e80094e0 tx=0x7ff1e8009360 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.362+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/3092775538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd406910 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009c00 tx=0x7ff1e0011810 comp rx=0 tx=0).stop 2024-05-27T07:59:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 -- 172.21.15.114:0/3092775538 shutdown_connections 2024-05-27T07:59:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/3092775538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 0x5563dd406e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/3092775538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd406910 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/3092775538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd4063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 -- 172.21.15.114:0/3092775538 >> 172.21.15.114:0/3092775538 conn(0x5563dd059300 msgr2=0x5563dd402b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 -- 172.21.15.114:0/3092775538 shutdown_connections 2024-05-27T07:59:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 -- 172.21.15.114:0/3092775538 wait complete. 2024-05-27T07:59:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 -- start start 2024-05-27T07:59:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 Processor -- start 2024-05-27T07:59:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd03c2f0 0x5563dd410a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0444d0 0x5563dd410fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd0651b0 0x5563dd4168a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5563dd04e790 con 0x5563dd0651b0 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5563dd324890 con 0x5563dd03c2f0 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5563dd413500 con 0x5563dd0444d0 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.363+0000 7ff1fb79e740 10 client.0 _ll_register_callbacks cb 0x5563dd07ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1eed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd0651b0 0x5563dd4168a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1edd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0444d0 0x5563dd410fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1eed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd0651b0 0x5563dd4168a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58438/0 (socket says 172.21.15.114:58438) 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1ee59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd03c2f0 0x5563dd410a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1eed9d640 1 -- 172.21.15.114:0/3389094219 learned_addr learned my addr 172.21.15.114:0/3389094219 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1eed9d640 1 -- 172.21.15.114:0/3389094219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0444d0 msgr2=0x5563dd410fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1eed9d640 1 --2- 172.21.15.114:0/3389094219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0444d0 0x5563dd410fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1eed9d640 1 -- 172.21.15.114:0/3389094219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd03c2f0 msgr2=0x5563dd410a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1eed9d640 1 --2- 172.21.15.114:0/3389094219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd03c2f0 0x5563dd410a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1eed9d640 1 -- 172.21.15.114:0/3389094219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563dd149620 con 0x5563dd0651b0 2024-05-27T07:59:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1ee59c640 1 --2- 172.21.15.114:0/3389094219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd03c2f0 0x5563dd410a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1eed9d640 1 --2- 172.21.15.114:0/3389094219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd0651b0 0x5563dd4168a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8009d90 tx=0x7ff1e80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a17891215fc739 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1d5ffb640 1 -- 172.21.15.114:0/3389094219 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8033040 con 0x5563dd0651b0 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1fb79e740 1 -- 172.21.15.114:0/3389094219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1e00043d0 con 0x5563dd0651b0 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1d5ffb640 1 -- 172.21.15.114:0/3389094219 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7ff1e8003e00 con 0x5563dd0651b0 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1d5ffb640 1 -- 172.21.15.114:0/3389094219 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7ff1e805a050 con 0x5563dd0651b0 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.364+0000 7ff1fb79e740 10 client.19854 fetch_fsmap learned FSMap version 113 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.365+0000 7ff1fb79e740 1 -- 172.21.15.114:0/3389094219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=4+}) -- 0x5563dd41f830 con 0x5563dd0651b0 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.365+0000 7ff1d5ffb640 1 -- 172.21.15.114:0/3389094219 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1e00043d0 con 0x5563dd0651b0 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.365+0000 7ff1d5ffb640 1 -- 172.21.15.114:0/3389094219 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 113) ==== 3490+0+0 (secure 0 0 0) 0x7ff1e8038450 con 0x5563dd0651b0 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.365+0000 7ff1fb79e740 10 client.19854 fetch_fsmap finished waiting for FSMap version 113 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.365+0000 7ff1fb79e740 10 client.19854 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.365+0000 7ff1fb79e740 20 client.19854 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.365+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/3389094219 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5563dd023d80 0x5563dd4231c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.365+0000 7ff1fb79e740 4 client.19854 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:59:48.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.365+0000 7ff1fb79e740 1 -- 172.21.15.114:0/3389094219 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563dd37ab80 con 0x5563dd023d80 2024-05-27T07:59:48.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.365+0000 7ff1ee59c640 1 --2- 172.21.15.114:0/3389094219 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5563dd023d80 0x5563dd4231c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.366+0000 7ff1ee59c640 1 --2- 172.21.15.114:0/3389094219 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5563dd023d80 0x5563dd4231c0 crc :-1 s=READY pgs=522 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0b5d1ce1eeed897 server_cookie=7f4ad16114c73580 in_seq=0 out_seq=0 2024-05-27T07:59:48.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.366+0000 7ff1d5ffb640 10 client.19854 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1d5ffb640 1 -- 172.21.15.114:0/3389094219 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5563dd37ab80 con 0x5563dd023d80 2024-05-27T07:59:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1d5ffb640 10 client.19854 handle_command_reply: tid=0 2024-05-27T07:59:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1fb79e740 1 client.19854 shutdown 2024-05-27T07:59:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1fb79e740 1 -- 172.21.15.114:0/3389094219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd0651b0 msgr2=0x5563dd4168a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/3389094219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd0651b0 0x5563dd4168a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8009d90 tx=0x7ff1e80095c0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1fb79e740 1 -- 172.21.15.114:0/3389094219 shutdown_connections 2024-05-27T07:59:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/3389094219 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5563dd023d80 0x5563dd4231c0 crc :-1 s=READY pgs=522 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/3389094219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd0651b0 0x5563dd4168a0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/3389094219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0444d0 0x5563dd410fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1d5ffb640 0 client.19854 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/3389094219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd03c2f0 0x5563dd410a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1fb79e740 1 -- 172.21.15.114:0/3389094219 >> 172.21.15.114:0/3389094219 conn(0x5563dd059300 msgr2=0x5563dd4080f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1fb79e740 1 -- 172.21.15.114:0/3389094219 shutdown_connections 2024-05-27T07:59:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1fb79e740 1 -- 172.21.15.114:0/3389094219 wait complete. 2024-05-27T07:59:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.367+0000 7ff1fb79e740 20 client.19854 trim_cache size 0 max 16384 2024-05-27T07:59:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.384+0000 7ff1fb79e740 1 Processor -- start 2024-05-27T07:59:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.384+0000 7ff1fb79e740 1 -- start start 2024-05-27T07:59:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.384+0000 7ff1fb79e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd40a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.384+0000 7ff1fb79e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 0x5563dd4167a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.384+0000 7ff1fb79e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd416ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.384+0000 7ff1fb79e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f01b2a50 con 0x5563dd03c2f0 2024-05-27T07:59:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.384+0000 7ff1fb79e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f01b3120 con 0x5563dd0444d0 2024-05-27T07:59:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.384+0000 7ff1fb79e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f01b1450 con 0x5563dd0651b0 2024-05-27T07:59:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.384+0000 7ff1ee59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd40a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.384+0000 7ff1ee59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd40a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58444/0 (socket says 172.21.15.114:58444) 2024-05-27T07:59:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.384+0000 7ff1ee59c640 1 -- 172.21.15.114:0/1638932834 learned_addr learned my addr 172.21.15.114:0/1638932834 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.384+0000 7ff1eed9d640 1 --2- 172.21.15.114:0/1638932834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd416ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.384+0000 7ff1edd9b640 1 --2- 172.21.15.114:0/1638932834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 0x5563dd4167a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.385+0000 7ff1ee59c640 1 -- 172.21.15.114:0/1638932834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 msgr2=0x5563dd416ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.385+0000 7ff1ee59c640 1 --2- 172.21.15.114:0/1638932834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd416ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.385+0000 7ff1ee59c640 1 -- 172.21.15.114:0/1638932834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 msgr2=0x5563dd4167a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.385+0000 7ff1ee59c640 1 --2- 172.21.15.114:0/1638932834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 0x5563dd4167a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.385+0000 7ff1ee59c640 1 -- 172.21.15.114:0/1638932834 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563dd417220 con 0x5563dd03c2f0 2024-05-27T07:59:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.385+0000 7ff1ee59c640 1 --2- 172.21.15.114:0/1638932834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd40a700 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0011d10 tx=0x7ff1e0009fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6463b21308f2e1b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.385+0000 7ff1d7fff640 1 -- 172.21.15.114:0/1638932834 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e001e040 con 0x5563dd03c2f0 2024-05-27T07:59:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.385+0000 7ff1d7fff640 1 -- 172.21.15.114:0/1638932834 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1e0032070 con 0x5563dd03c2f0 2024-05-27T07:59:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.385+0000 7ff1d7fff640 1 -- 172.21.15.114:0/1638932834 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e002c3d0 con 0x5563dd03c2f0 2024-05-27T07:59:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.386+0000 7ff1fb79e740 1 -- 172.21.15.114:0/1638932834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 msgr2=0x5563dd40a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.386+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/1638932834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd40a700 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0011d10 tx=0x7ff1e0009fc0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.386+0000 7ff1fb79e740 1 -- 172.21.15.114:0/1638932834 shutdown_connections 2024-05-27T07:59:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.386+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/1638932834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd416ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.386+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/1638932834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 0x5563dd4167a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.386+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/1638932834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd40a700 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.386+0000 7ff1fb79e740 1 -- 172.21.15.114:0/1638932834 >> 172.21.15.114:0/1638932834 conn(0x5563dd059300 msgr2=0x5563dd40f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.386+0000 7ff1fb79e740 1 -- 172.21.15.114:0/1638932834 shutdown_connections 2024-05-27T07:59:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.386+0000 7ff1fb79e740 1 -- 172.21.15.114:0/1638932834 wait complete. 2024-05-27T07:59:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.386+0000 7ff1fb79e740 1 -- start start 2024-05-27T07:59:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.386+0000 7ff1fb79e740 1 Processor -- start 2024-05-27T07:59:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1fb79e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd41d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1fb79e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 0x5563dd417810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1fb79e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd417dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1fb79e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5563dd4077b0 con 0x5563dd03c2f0 2024-05-27T07:59:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1fb79e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f01b1ac0 con 0x5563dd0444d0 2024-05-27T07:59:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1fb79e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5563dd40edc0 con 0x5563dd0651b0 2024-05-27T07:59:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1fb79e740 10 client.0 _ll_register_callbacks cb 0x7ff1e0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1ee59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd41d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1eed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd417dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1ee59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd41d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58456/0 (socket says 172.21.15.114:58456) 2024-05-27T07:59:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1ee59c640 1 -- 172.21.15.114:0/574923733 learned_addr learned my addr 172.21.15.114:0/574923733 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1edd9b640 1 --2- 172.21.15.114:0/574923733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 0x5563dd417810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1ee59c640 1 -- 172.21.15.114:0/574923733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 msgr2=0x5563dd417dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1ee59c640 1 --2- 172.21.15.114:0/574923733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd417dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1ee59c640 1 -- 172.21.15.114:0/574923733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 msgr2=0x5563dd417810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1ee59c640 1 --2- 172.21.15.114:0/574923733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 0x5563dd417810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1ee59c640 1 -- 172.21.15.114:0/574923733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563dd418560 con 0x5563dd03c2f0 2024-05-27T07:59:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1eed9d640 1 --2- 172.21.15.114:0/574923733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd417dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1ee59c640 1 --2- 172.21.15.114:0/574923733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd41d3b0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009990 tx=0x7ff1e00096b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30059f0241fef2c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.387+0000 7ff1d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1d5ffb640 1 -- 172.21.15.114:0/574923733 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0019b10 con 0x5563dd03c2f0 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1d5ffb640 1 -- 172.21.15.114:0/574923733 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7ff1e0019cb0 con 0x5563dd03c2f0 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1fb79e740 1 -- 172.21.15.114:0/574923733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1e0032070 con 0x5563dd03c2f0 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1d5ffb640 1 -- 172.21.15.114:0/574923733 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7ff1e0046050 con 0x5563dd03c2f0 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1fb79e740 10 client.19860 fetch_fsmap learned FSMap version 113 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1fb79e740 1 -- 172.21.15.114:0/574923733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=4+}) -- 0x5563dd48a850 con 0x5563dd03c2f0 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1d5ffb640 1 -- 172.21.15.114:0/574923733 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1e0032070 con 0x5563dd03c2f0 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1d5ffb640 1 -- 172.21.15.114:0/574923733 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 113) ==== 3490+0+0 (secure 0 0 0) 0x7ff1e0024040 con 0x5563dd03c2f0 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1fb79e740 10 client.19860 fetch_fsmap finished waiting for FSMap version 113 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1fb79e740 10 client.19860 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1fb79e740 20 client.19860 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/574923733 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5563dcff91c0 0x5563dd48ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1fb79e740 4 client.19860 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1fb79e740 1 -- 172.21.15.114:0/574923733 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5563dd374360 con 0x5563dcff91c0 2024-05-27T07:59:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.388+0000 7ff1edd9b640 1 --2- 172.21.15.114:0/574923733 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5563dcff91c0 0x5563dd48ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:48.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.389+0000 7ff1d5ffb640 10 client.19860 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:48.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:48.389+0000 7ff1edd9b640 1 --2- 172.21.15.114:0/574923733 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5563dcff91c0 0x5563dd48ccc0 crc :-1 s=READY pgs=523 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1664ce140ecb2e8 server_cookie=b9c573a5bd442bbb in_seq=0 out_seq=0 2024-05-27T07:59:48.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 84855 12.24 MB/sec execute 240 sec latency 302.730 ms 2024-05-27T07:59:48.725 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:48 smithi114 podman[134263]: 2024-05-27 07:59:48.540394366 +0000 UTC m=+0.650345771 container remove aff38ba4d6ea90b61c26f853ab2ff0dce78166479b5ca8ecac3913436142cbb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-a, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-27T07:59:48.726 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:48 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:59:49.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:48 smithi114 ceph-mon[32038]: osdmap e164: 12 total, 12 up, 12 in 2024-05-27T07:59:49.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:48 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6838/1653686179,v1:172.21.15.114:6839/1653686179] up:boot 2024-05-27T07:59:49.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:48 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 3 up:standby 2024-05-27T07:59:49.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:48 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:59:49.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:48 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 2 up:standby 2024-05-27T07:59:49.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:48 smithi114 ceph-mon[32038]: pgmap v693: 129 pgs: 129 active+clean; 400 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 342 op/s 2024-05-27T07:59:49.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:48 smithi163 ceph-mon[36451]: osdmap e164: 12 total, 12 up, 12 in 2024-05-27T07:59:49.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:48 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6838/1653686179,v1:172.21.15.114:6839/1653686179] up:boot 2024-05-27T07:59:49.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:48 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 3 up:standby 2024-05-27T07:59:49.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:48 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:59:49.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:48 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 2 up:standby 2024-05-27T07:59:49.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:48 smithi163 ceph-mon[36451]: pgmap v693: 129 pgs: 129 active+clean; 400 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 342 op/s 2024-05-27T07:59:49.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:48 smithi186 ceph-mon[36647]: osdmap e164: 12 total, 12 up, 12 in 2024-05-27T07:59:49.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:48 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6838/1653686179,v1:172.21.15.114:6839/1653686179] up:boot 2024-05-27T07:59:49.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:48 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 3 up:standby 2024-05-27T07:59:49.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:48 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:59:49.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:48 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 2 up:standby 2024-05-27T07:59:49.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:48 smithi186 ceph-mon[36647]: pgmap v693: 129 pgs: 129 active+clean; 400 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 342 op/s 2024-05-27T07:59:49.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 84932 12.21 MB/sec execute 241 sec latency 635.828 ms 2024-05-27T07:59:49.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.909+0000 7ff1d5ffb640 1 -- 172.21.15.114:0/574923733 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5563dd374360 con 0x5563dcff91c0 2024-05-27T07:59:49.910 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.909+0000 7ff1d5ffb640 10 client.19860 handle_command_reply: tid=0 2024-05-27T07:59:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.909+0000 7ff1fb79e740 1 client.19860 shutdown 2024-05-27T07:59:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.909+0000 7ff1fb79e740 1 -- 172.21.15.114:0/574923733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 msgr2=0x5563dd41d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.909+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/574923733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd41d3b0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009990 tx=0x7ff1e00096b0 comp rx=0 tx=0).stop 2024-05-27T07:59:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.909+0000 7ff1fb79e740 1 -- 172.21.15.114:0/574923733 shutdown_connections 2024-05-27T07:59:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.909+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/574923733 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5563dcff91c0 0x5563dd48ccc0 crc :-1 s=READY pgs=523 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.909+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/574923733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563dd0651b0 0x5563dd417dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.909+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/574923733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563dd0444d0 0x5563dd417810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.909+0000 7ff1fb79e740 1 --2- 172.21.15.114:0/574923733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563dd03c2f0 0x5563dd41d3b0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.909+0000 7ff1fb79e740 1 -- 172.21.15.114:0/574923733 >> 172.21.15.114:0/574923733 conn(0x5563dd059300 msgr2=0x5563dd411e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.909+0000 7ff1d5ffb640 0 client.19860 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.910+0000 7ff1fb79e740 1 -- 172.21.15.114:0/574923733 shutdown_connections 2024-05-27T07:59:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.910+0000 7ff1fb79e740 1 -- 172.21.15.114:0/574923733 wait complete. 2024-05-27T07:59:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.910+0000 7ff1fb79e740 20 client.19860 trim_cache size 0 max 16384 2024-05-27T07:59:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-05-27T07:59:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-05-27T07:59:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-05-27T07:59:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.911+0000 7ff1f4813640 1 -- 172.21.15.114:0/1431642682 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff1d80b2950 msgr2=0x7ff1d80b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.911+0000 7ff1f4813640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff1d80b2950 0x7ff1d80b4e60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff1e80094e0 tx=0x7ff1e8009360 comp rx=0 tx=0).stop 2024-05-27T07:59:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.911+0000 7ff1f4813640 1 -- 172.21.15.114:0/1431642682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1f010b850 msgr2=0x7ff1f01a0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.911+0000 7ff1f4813640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1f010b850 0x7ff1f01a0a10 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400e9f0 tx=0x7ff1e400eeb0 comp rx=0 tx=0).stop 2024-05-27T07:59:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.912+0000 7ff1f4813640 1 -- 172.21.15.114:0/1431642682 shutdown_connections 2024-05-27T07:59:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.912+0000 7ff1f4813640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff1d80b2950 0x7ff1d80b4e60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:49.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.912+0000 7ff1f4813640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1f010e200 0x7ff1f019a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:49.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.912+0000 7ff1f4813640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1f010b850 0x7ff1f01a0a10 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:49.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.912+0000 7ff1f4813640 1 --2- 172.21.15.114:0/1431642682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f0104df0 0x7ff1f01a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:49.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.912+0000 7ff1f4813640 1 -- 172.21.15.114:0/1431642682 >> 172.21.15.114:0/1431642682 conn(0x7ff1f00782d0 msgr2=0x7ff1f01012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:49.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.912+0000 7ff1f4813640 1 -- 172.21.15.114:0/1431642682 shutdown_connections 2024-05-27T07:59:49.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-05-27T07:59:49.912+0000 7ff1f4813640 1 -- 172.21.15.114:0/1431642682 wait complete. 2024-05-27T07:59:50.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.460+0000 7ffbbdfac700 1 -- 172.21.15.114:0/408089640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbb81031c0 msgr2=0x7ffbb8111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:50.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.460+0000 7ffbbdfac700 1 --2- 172.21.15.114:0/408089640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbb81031c0 0x7ffbb8111160 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7ffbb000b0a0 tx=0x7ffbb000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.460+0000 7ffbbdfac700 1 -- 172.21.15.114:0/408089640 shutdown_connections 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.460+0000 7ffbbdfac700 1 --2- 172.21.15.114:0/408089640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbb81031c0 0x7ffbb8111160 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.460+0000 7ffbbdfac700 1 --2- 172.21.15.114:0/408089640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbb81092f0 0x7ffbb8109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.460+0000 7ffbbdfac700 1 --2- 172.21.15.114:0/408089640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffbb8102870 0x7ffbb8102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.460+0000 7ffbbdfac700 1 -- 172.21.15.114:0/408089640 >> 172.21.15.114:0/408089640 conn(0x7ffbb80fe6e0 msgr2=0x7ffbb8100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.460+0000 7ffbbdfac700 1 -- 172.21.15.114:0/408089640 shutdown_connections 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.460+0000 7ffbbdfac700 1 -- 172.21.15.114:0/408089640 wait complete. 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.461+0000 7ffbbdfac700 1 Processor -- start 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.461+0000 7ffbbdfac700 1 -- start start 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.461+0000 7ffbbdfac700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbb8102870 0x7ffbb819aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.461+0000 7ffbbdfac700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbb81031c0 0x7ffbb819b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.461+0000 7ffbbdfac700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffbb81092f0 0x7ffbb819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.461+0000 7ffbbdfac700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb81145b0 con 0x7ffbb8102870 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.461+0000 7ffbbdfac700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb8114430 con 0x7ffbb81092f0 2024-05-27T07:59:50.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.461+0000 7ffbbdfac700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb8114730 con 0x7ffbb81031c0 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.461+0000 7ffbb77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbb8102870 0x7ffbb819aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.461+0000 7ffbb6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbb81031c0 0x7ffbb819b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.461+0000 7ffbb77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbb8102870 0x7ffbb819aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58482/0 (socket says 172.21.15.114:58482) 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.461+0000 7ffbb77fe700 1 -- 172.21.15.114:0/2921870328 learned_addr learned my addr 172.21.15.114:0/2921870328 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.461+0000 7ffbb7fff700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffbb81092f0 0x7ffbb819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.462+0000 7ffbb7fff700 1 -- 172.21.15.114:0/2921870328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbb81031c0 msgr2=0x7ffbb819b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.462+0000 7ffbb7fff700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbb81031c0 0x7ffbb819b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.462+0000 7ffbb7fff700 1 -- 172.21.15.114:0/2921870328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbb8102870 msgr2=0x7ffbb819aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.462+0000 7ffbb7fff700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbb8102870 0x7ffbb819aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.462+0000 7ffbb7fff700 1 -- 172.21.15.114:0/2921870328 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb819fd70 con 0x7ffbb81092f0 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.462+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbb8102870 0x7ffbb819aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.462+0000 7ffbb6ffd700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbb81031c0 0x7ffbb819b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.462+0000 7ffbb7fff700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffbb81092f0 0x7ffbb819f6d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ffbb000b070 tx=0x7ffbb0004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:50.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.462+0000 7ffbb4ff9700 1 -- 172.21.15.114:0/2921870328 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0021070 con 0x7ffbb81092f0 2024-05-27T07:59:50.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.462+0000 7ffbbdfac700 1 -- 172.21.15.114:0/2921870328 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb81a0000 con 0x7ffbb81092f0 2024-05-27T07:59:50.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.462+0000 7ffbb4ff9700 1 -- 172.21.15.114:0/2921870328 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb0007e30 con 0x7ffbb81092f0 2024-05-27T07:59:50.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.462+0000 7ffbb4ff9700 1 -- 172.21.15.114:0/2921870328 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00045d0 con 0x7ffbb81092f0 2024-05-27T07:59:50.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.462+0000 7ffbbdfac700 1 -- 172.21.15.114:0/2921870328 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb81a7980 con 0x7ffbb81092f0 2024-05-27T07:59:50.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.465+0000 7ffbb4ff9700 1 -- 172.21.15.114:0/2921870328 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ffbb001d450 con 0x7ffbb81092f0 2024-05-27T07:59:50.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.465+0000 7ffbb4ff9700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffba00b29f0 0x7ffba00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.466+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffba00b29f0 0x7ffba00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.466+0000 7ffbb4ff9700 1 -- 172.21.15.114:0/2921870328 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7ffbb000f020 con 0x7ffbb81092f0 2024-05-27T07:59:50.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.466+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffba00b29f0 0x7ffba00b4f00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ffba8011550 tx=0x7ffba8009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:50.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.466+0000 7ffbcc9c1b80 1 Processor -- start 2024-05-27T07:59:50.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.466+0000 7ffbcc9c1b80 1 -- start start 2024-05-27T07:59:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.466+0000 7ffbcc9c1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bfb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.466+0000 7ffbcc9c1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bf5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbcc9c1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bf6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbcc9c1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b537b24d10 con 0x55b537bfbcd0 2024-05-27T07:59:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbcc9c1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b537b148b0 con 0x55b537a77ec0 2024-05-27T07:59:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbcc9c1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b537b188f0 con 0x55b537bf6000 2024-05-27T07:59:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bf6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bf5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bf6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58900/0 (socket says 172.21.15.114:58900) 2024-05-27T07:59:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb7fff700 1 -- 172.21.15.114:0/2856081009 learned_addr learned my addr 172.21.15.114:0/2856081009 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bf5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58484/0 (socket says 172.21.15.114:58484) 2024-05-27T07:59:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/2856081009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bfb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb7fff700 1 -- 172.21.15.114:0/2856081009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 msgr2=0x55b537bfb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb7fff700 1 --2- 172.21.15.114:0/2856081009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bfb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb7fff700 1 -- 172.21.15.114:0/2856081009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 msgr2=0x55b537bf5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb7fff700 1 --2- 172.21.15.114:0/2856081009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bf5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb7fff700 1 -- 172.21.15.114:0/2856081009 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b537a64950 con 0x55b537bf6000 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb6ffd700 1 --2- 172.21.15.114:0/2856081009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bf5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/2856081009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bfb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffbb7fff700 1 --2- 172.21.15.114:0/2856081009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bf6410 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00a6cf0 tx=0x7ffbb00a6480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efbb10dd85b80091 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffb9dffb700 1 -- 172.21.15.114:0/2856081009 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00b3390 con 0x55b537bf6000 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.467+0000 7ffb9dffb700 1 -- 172.21.15.114:0/2856081009 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb00aac70 con 0x55b537bf6000 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.468+0000 7ffb9dffb700 1 -- 172.21.15.114:0/2856081009 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0007980 con 0x55b537bf6000 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.468+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/2856081009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 msgr2=0x55b537bf6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.468+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/2856081009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bf6410 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00a6cf0 tx=0x7ffbb00a6480 comp rx=0 tx=0).stop 2024-05-27T07:59:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.468+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/2856081009 shutdown_connections 2024-05-27T07:59:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.468+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/2856081009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bf6410 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.468+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/2856081009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bfb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.468+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/2856081009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bf5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.468+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/2856081009 >> 172.21.15.114:0/2856081009 conn(0x55b537ad2b70 msgr2=0x55b537bf06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.468+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/2856081009 shutdown_connections 2024-05-27T07:59:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.468+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/2856081009 wait complete. 2024-05-27T07:59:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.469+0000 7ffbcc9c1b80 1 -- start start 2024-05-27T07:59:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.469+0000 7ffbcc9c1b80 1 Processor -- start 2024-05-27T07:59:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbcc9c1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bff470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbcc9c1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bff9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbcc9c1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bffef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbcc9c1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b537a1b020 con 0x55b537bfbcd0 2024-05-27T07:59:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbcc9c1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b537ab6840 con 0x55b537a77ec0 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbcc9c1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b537c06990 con 0x55b537bf6000 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbcc9c1b80 10 client.0 _ll_register_callbacks cb 0x55b537aaf9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbb77fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bff470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbb7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bffef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbb77fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bff470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:35062/0 (socket says 172.21.15.114:35062) 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbb77fe700 1 -- 172.21.15.114:0/1570504924 learned_addr learned my addr 172.21.15.114:0/1570504924 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbb6ffd700 1 --2- 172.21.15.114:0/1570504924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bff9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbb77fe700 1 -- 172.21.15.114:0/1570504924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 msgr2=0x55b537bff9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/1570504924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bff9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbb77fe700 1 -- 172.21.15.114:0/1570504924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 msgr2=0x55b537bffef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/1570504924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bffef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbb77fe700 1 -- 172.21.15.114:0/1570504924 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b537ab3710 con 0x55b537a77ec0 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/1570504924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bff470 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffba8003a90 tx=0x7ffba80047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d38f2682e819945d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:50.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.470+0000 7ffb96ffd700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:59:50.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.473+0000 7ffb96ffd700 1 -- 172.21.15.114:0/1570504924 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba800ea40 con 0x55b537a77ec0 2024-05-27T07:59:50.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.474+0000 7ffb96ffd700 1 -- 172.21.15.114:0/1570504924 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7ffba800ebe0 con 0x55b537a77ec0 2024-05-27T07:59:50.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.474+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/1570504924 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbb00aac70 con 0x55b537a77ec0 2024-05-27T07:59:50.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.474+0000 7ffb96ffd700 1 -- 172.21.15.114:0/1570504924 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7ffba800ede0 con 0x55b537a77ec0 2024-05-27T07:59:50.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.474+0000 7ffbcc9c1b80 10 client.29183 fetch_fsmap learned FSMap version 113 2024-05-27T07:59:50.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.474+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/1570504924 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=4+}) -- 0x55b537c11400 con 0x55b537a77ec0 2024-05-27T07:59:50.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.474+0000 7ffb96ffd700 1 -- 172.21.15.114:0/1570504924 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffba8023070 con 0x55b537a77ec0 2024-05-27T07:59:50.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.474+0000 7ffb96ffd700 1 -- 172.21.15.114:0/1570504924 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 113) ==== 3490+0+0 (secure 0 0 0) 0x7ffba801f600 con 0x55b537a77ec0 2024-05-27T07:59:50.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.474+0000 7ffbcc9c1b80 10 client.29183 fetch_fsmap finished waiting for FSMap version 113 2024-05-27T07:59:50.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.474+0000 7ffbcc9c1b80 10 client.29183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:50.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.474+0000 7ffbcc9c1b80 20 client.29183 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:50.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.474+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/1570504924 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537c13e70 0x55b537c162e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.474+0000 7ffbcc9c1b80 4 client.29183 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:59:50.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.474+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/1570504924 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ffbb00aac70 con 0x55b537c13e70 2024-05-27T07:59:50.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.475+0000 7ffbb6ffd700 1 --2- 172.21.15.114:0/1570504924 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537c13e70 0x55b537c162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.475+0000 7ffb96ffd700 10 client.29183 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:50.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.476+0000 7ffbb6ffd700 1 --2- 172.21.15.114:0/1570504924 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537c13e70 0x55b537c162e0 crc :-1 s=READY pgs=524 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9044939d4b5ea38e server_cookie=1cf9ed014486a64f in_seq=0 out_seq=0 2024-05-27T07:59:50.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.479+0000 7ffb96ffd700 1 -- 172.21.15.114:0/1570504924 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7ffbb00aac70 con 0x55b537c13e70 2024-05-27T07:59:50.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.479+0000 7ffb96ffd700 10 client.29183 handle_command_reply: tid=0 2024-05-27T07:59:50.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.479+0000 7ffbcc9c1b80 1 client.29183 shutdown 2024-05-27T07:59:50.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.479+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/1570504924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 msgr2=0x55b537bff470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:50.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.479+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/1570504924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bff470 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffba8003a90 tx=0x7ffba80047c0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.480+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/1570504924 shutdown_connections 2024-05-27T07:59:50.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.480+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/1570504924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.480+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/1570504924 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537c13e70 0x55b537c162e0 crc :-1 s=READY pgs=524 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.480+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/1570504924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bff470 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.480+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/1570504924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bff9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.480+0000 7ffb96ffd700 0 client.29183 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:50.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.480+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/1570504924 >> 172.21.15.114:0/1570504924 conn(0x55b537ad2b70 msgr2=0x55b537bf4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:50.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.480+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/1570504924 shutdown_connections 2024-05-27T07:59:50.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.480+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/1570504924 wait complete. 2024-05-27T07:59:50.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.480+0000 7ffbcc9c1b80 20 client.29183 trim_cache size 0 max 16384 2024-05-27T07:59:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.502+0000 7ffbcc9c1b80 1 Processor -- start 2024-05-27T07:59:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.502+0000 7ffbcc9c1b80 1 -- start start 2024-05-27T07:59:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.502+0000 7ffbcc9c1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bf5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.502+0000 7ffbcc9c1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bf9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.502+0000 7ffbcc9c1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bfade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.502+0000 7ffbcc9c1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b537c16710 con 0x55b537bfbcd0 2024-05-27T07:59:50.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.502+0000 7ffbcc9c1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b537c16890 con 0x55b537a77ec0 2024-05-27T07:59:50.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.502+0000 7ffbcc9c1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb81ad8a0 con 0x55b537bf6000 2024-05-27T07:59:50.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.502+0000 7ffbb7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bfade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.502+0000 7ffbb7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bfade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58504/0 (socket says 172.21.15.114:58504) 2024-05-27T07:59:50.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.502+0000 7ffbb7fff700 1 -- 172.21.15.114:0/3813318274 learned_addr learned my addr 172.21.15.114:0/3813318274 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:50.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.502+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/3813318274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bf5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.502+0000 7ffbb6ffd700 1 --2- 172.21.15.114:0/3813318274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bf9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.503+0000 7ffbb77fe700 1 -- 172.21.15.114:0/3813318274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 msgr2=0x55b537bf9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:50.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.503+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/3813318274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bf9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.503+0000 7ffbb77fe700 1 -- 172.21.15.114:0/3813318274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 msgr2=0x55b537bfade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:50.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.503+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/3813318274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bfade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.503+0000 7ffbb77fe700 1 -- 172.21.15.114:0/3813318274 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b537bf1190 con 0x55b537a77ec0 2024-05-27T07:59:50.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.503+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/3813318274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bf5ac0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ffba8003a60 tx=0x7ffba8011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7a9bf6e540958f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:50.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.503+0000 7ffb9d7fa700 1 -- 172.21.15.114:0/3813318274 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba8039020 con 0x55b537a77ec0 2024-05-27T07:59:50.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.503+0000 7ffb9d7fa700 1 -- 172.21.15.114:0/3813318274 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffba8023070 con 0x55b537a77ec0 2024-05-27T07:59:50.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.503+0000 7ffb9d7fa700 1 -- 172.21.15.114:0/3813318274 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba80356e0 con 0x55b537a77ec0 2024-05-27T07:59:50.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.504+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/3813318274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 msgr2=0x55b537bf5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:50.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.504+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/3813318274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bf5ac0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ffba8003a60 tx=0x7ffba8011ca0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.504+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/3813318274 shutdown_connections 2024-05-27T07:59:50.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.504+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/3813318274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bfbcd0 0x55b537bfade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.504+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/3813318274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537a77ec0 0x55b537bf5ac0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.504+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/3813318274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537bf6000 0x55b537bf9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.504+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/3813318274 >> 172.21.15.114:0/3813318274 conn(0x55b537ad2b70 msgr2=0x55b537bfc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:50.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.504+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/3813318274 shutdown_connections 2024-05-27T07:59:50.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.504+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/3813318274 wait complete. 2024-05-27T07:59:50.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.504+0000 7ffbcc9c1b80 1 -- start start 2024-05-27T07:59:50.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbcc9c1b80 1 Processor -- start 2024-05-27T07:59:50.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbcc9c1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537a77ec0 0x55b537c0c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbcc9c1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bf6000 0x55b537c1b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbcc9c1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537bfbcd0 0x55b537c1ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbcc9c1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb81ac230 con 0x55b537bf6000 2024-05-27T07:59:50.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbcc9c1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb81adf70 con 0x55b537bfbcd0 2024-05-27T07:59:50.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbcc9c1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb81ac8a0 con 0x55b537a77ec0 2024-05-27T07:59:50.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbcc9c1b80 10 client.0 _ll_register_callbacks cb 0x55b537aaf9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:50.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbb6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bf6000 0x55b537c1b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbb77fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537a77ec0 0x55b537c0c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbb6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bf6000 0x55b537c1b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58514/0 (socket says 172.21.15.114:58514) 2024-05-27T07:59:50.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbb77fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537a77ec0 0x55b537c0c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58918/0 (socket says 172.21.15.114:58918) 2024-05-27T07:59:50.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbb6ffd700 1 -- 172.21.15.114:0/3750192471 learned_addr learned my addr 172.21.15.114:0/3750192471 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:50.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.505+0000 7ffbb7fff700 1 --2- 172.21.15.114:0/3750192471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537bfbcd0 0x55b537c1ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffbb6ffd700 1 -- 172.21.15.114:0/3750192471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537a77ec0 msgr2=0x55b537c0c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:50.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffbb6ffd700 1 --2- 172.21.15.114:0/3750192471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537a77ec0 0x55b537c0c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffbb6ffd700 1 -- 172.21.15.114:0/3750192471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537bfbcd0 msgr2=0x55b537c1ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:50.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffbb6ffd700 1 --2- 172.21.15.114:0/3750192471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537bfbcd0 0x55b537c1ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffbb6ffd700 1 -- 172.21.15.114:0/3750192471 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b537c1c230 con 0x55b537bf6000 2024-05-27T07:59:50.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffbb7fff700 1 --2- 172.21.15.114:0/3750192471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537bfbcd0 0x55b537c1ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:50.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffbb6ffd700 1 --2- 172.21.15.114:0/3750192471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bf6000 0x55b537c1b490 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x55b537bf2120 tx=0x7ffbac0123b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e05cc584559fab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:50.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffb96ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T07:59:50.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffb96ffd700 1 -- 172.21.15.114:0/3750192471 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac01f7b0 con 0x55b537bf6000 2024-05-27T07:59:50.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffb96ffd700 1 -- 172.21.15.114:0/3750192471 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7ffbac01f950 con 0x55b537bf6000 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/3750192471 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffba8023070 con 0x55b537bf6000 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffb96ffd700 1 -- 172.21.15.114:0/3750192471 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7ffbac031050 con 0x55b537bf6000 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffbcc9c1b80 10 client.19878 fetch_fsmap learned FSMap version 113 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/3750192471 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=4+}) -- 0x55b537c7ea00 con 0x55b537bf6000 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.506+0000 7ffb96ffd700 1 -- 172.21.15.114:0/3750192471 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffba8023070 con 0x55b537bf6000 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.507+0000 7ffb96ffd700 1 -- 172.21.15.114:0/3750192471 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 113) ==== 3490+0+0 (secure 0 0 0) 0x7ffbac010430 con 0x55b537bf6000 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.507+0000 7ffbcc9c1b80 10 client.19878 fetch_fsmap finished waiting for FSMap version 113 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.507+0000 7ffbcc9c1b80 10 client.19878 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.507+0000 7ffbcc9c1b80 20 client.19878 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.507+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/3750192471 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537c7f140 0x55b537c81560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.507+0000 7ffbcc9c1b80 4 client.19878 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.507+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/3750192471 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b537bc1a70 con 0x55b537c7f140 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.507+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/3750192471 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537c7f140 0x55b537c81560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:50.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.507+0000 7ffb96ffd700 10 client.19878 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:50.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.507+0000 7ffbb77fe700 1 --2- 172.21.15.114:0/3750192471 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537c7f140 0x55b537c81560 crc :-1 s=READY pgs=525 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2306c17814f4fbea server_cookie=fd6a44dda1f95840 in_seq=0 out_seq=0 2024-05-27T07:59:50.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.512+0000 7ffb96ffd700 1 -- 172.21.15.114:0/3750192471 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b537bc1a70 con 0x55b537c7f140 2024-05-27T07:59:50.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.512+0000 7ffb96ffd700 10 client.19878 handle_command_reply: tid=0 2024-05-27T07:59:50.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.512+0000 7ffbcc9c1b80 1 client.19878 shutdown 2024-05-27T07:59:50.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/3750192471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bf6000 msgr2=0x55b537c1b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:50.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/3750192471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bf6000 0x55b537c1b490 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x55b537bf2120 tx=0x7ffbac0123b0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/3750192471 shutdown_connections 2024-05-27T07:59:50.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/3750192471 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55b537c7f140 0x55b537c81560 crc :-1 s=READY pgs=525 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/3750192471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b537bfbcd0 0x55b537c1ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/3750192471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b537a77ec0 0x55b537c0c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffbcc9c1b80 1 --2- 172.21.15.114:0/3750192471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b537bf6000 0x55b537c1b490 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/3750192471 >> 172.21.15.114:0/3750192471 conn(0x55b537ad2b70 msgr2=0x55b537bf9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:50.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffb96ffd700 0 client.19878 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:50.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/3750192471 shutdown_connections 2024-05-27T07:59:50.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffbcc9c1b80 1 -- 172.21.15.114:0/3750192471 wait complete. 2024-05-27T07:59:50.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffbcc9c1b80 20 client.19878 trim_cache size 0 max 16384 2024-05-27T07:59:50.515 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:59:50.515 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T07:59:50.515 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "c7d7ada6-7ecd-4e10-a435-a187edb5a8ba", 2024-05-27T07:59:50.515 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T07:59:50.516 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:59:50.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffbbdfac700 1 -- 172.21.15.114:0/2921870328 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffba00b29f0 msgr2=0x7ffba00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:50.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.513+0000 7ffbbdfac700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffba00b29f0 0x7ffba00b4f00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ffba8011550 tx=0x7ffba8009260 comp rx=0 tx=0).stop 2024-05-27T07:59:50.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.514+0000 7ffbbdfac700 1 -- 172.21.15.114:0/2921870328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffbb81092f0 msgr2=0x7ffbb819f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.514+0000 7ffbbdfac700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffbb81092f0 0x7ffbb819f6d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ffbb000b070 tx=0x7ffbb0004d00 comp rx=0 tx=0).stop 2024-05-27T07:59:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.514+0000 7ffbbdfac700 1 -- 172.21.15.114:0/2921870328 shutdown_connections 2024-05-27T07:59:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.514+0000 7ffbbdfac700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ffba00b29f0 0x7ffba00b4f00 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.514+0000 7ffbbdfac700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffbb81092f0 0x7ffbb819f6d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.514+0000 7ffbbdfac700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbb8102870 0x7ffbb819aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.514+0000 7ffbbdfac700 1 --2- 172.21.15.114:0/2921870328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbb81031c0 0x7ffbb819b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.514+0000 7ffbbdfac700 1 -- 172.21.15.114:0/2921870328 >> 172.21.15.114:0/2921870328 conn(0x7ffbb80fe6e0 msgr2=0x7ffbb8108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.514+0000 7ffbbdfac700 1 -- 172.21.15.114:0/2921870328 shutdown_connections 2024-05-27T07:59:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:50.514+0000 7ffbbdfac700 1 -- 172.21.15.114:0/2921870328 wait complete. 2024-05-27T07:59:50.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 85246 12.23 MB/sec execute 242 sec latency 259.908 ms 2024-05-27T07:59:50.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:50 smithi163 ceph-mon[36451]: pgmap v694: 129 pgs: 129 active+clean; 406 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 259 op/s 2024-05-27T07:59:50.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:50 smithi186 ceph-mon[36647]: pgmap v694: 129 pgs: 129 active+clean; 406 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 259 op/s 2024-05-27T07:59:50.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:50 smithi114 ceph-mon[32038]: pgmap v694: 129 pgs: 129 active+clean; 406 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 259 op/s 2024-05-27T07:59:50.808 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 07:59:50 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:59:51.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:59:51.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 85494 12.23 MB/sec execute 243 sec latency 227.341 ms 2024-05-27T07:59:51.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:51 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T07:59:51.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:51 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:59:51.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:51 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T07:59:51.733 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:59:51.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:51 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T07:59:51.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:51 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:59:51.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:51 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T07:59:51.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:51 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T07:59:51.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:51 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:59:51.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:51 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T07:59:52.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 85775 12.23 MB/sec execute 244 sec latency 180.856 ms 2024-05-27T07:59:52.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:52 smithi114 ceph-mon[32038]: pgmap v695: 129 pgs: 129 active+clean; 394 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.5 MiB/s wr, 397 op/s 2024-05-27T07:59:52.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:52 smithi163 ceph-mon[36451]: pgmap v695: 129 pgs: 129 active+clean; 394 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.5 MiB/s wr, 397 op/s 2024-05-27T07:59:52.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:52 smithi186 ceph-mon[36647]: pgmap v695: 129 pgs: 129 active+clean; 394 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.5 MiB/s wr, 397 op/s 2024-05-27T07:59:53.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:53 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T07:59:53.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:53 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T07:59:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.527+0000 7ff5932e0700 1 -- 172.21.15.114:0/2841897946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff58c1034b0 msgr2=0x7ff58c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.527+0000 7ff5932e0700 1 --2- 172.21.15.114:0/2841897946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff58c1034b0 0x7ff58c1038c0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7ff5880099c0 tx=0x7ff588009cd0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.527+0000 7ff5932e0700 1 -- 172.21.15.114:0/2841897946 shutdown_connections 2024-05-27T07:59:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.527+0000 7ff5932e0700 1 --2- 172.21.15.114:0/2841897946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff58c108730 0x7ff58c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.527+0000 7ff5932e0700 1 --2- 172.21.15.114:0/2841897946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff58c1034b0 0x7ff58c1038c0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.527+0000 7ff5932e0700 1 --2- 172.21.15.114:0/2841897946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff58c107de0 0x7ff58c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.527+0000 7ff5932e0700 1 -- 172.21.15.114:0/2841897946 >> 172.21.15.114:0/2841897946 conn(0x7ff58c0fedb0 msgr2=0x7ff58c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:53.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.527+0000 7ff5932e0700 1 -- 172.21.15.114:0/2841897946 shutdown_connections 2024-05-27T07:59:53.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.527+0000 7ff5932e0700 1 -- 172.21.15.114:0/2841897946 wait complete. 2024-05-27T07:59:53.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.528+0000 7ff5932e0700 1 Processor -- start 2024-05-27T07:59:53.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.528+0000 7ff5932e0700 1 -- start start 2024-05-27T07:59:53.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.528+0000 7ff5932e0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff58c1034b0 0x7ff58c19ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.528+0000 7ff5932e0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff58c107de0 0x7ff58c19b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.528+0000 7ff5932e0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff58c108730 0x7ff58c19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.528+0000 7ff5932e0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff58c113360 con 0x7ff58c107de0 2024-05-27T07:59:53.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.528+0000 7ff5932e0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff58c1131e0 con 0x7ff58c1034b0 2024-05-27T07:59:53.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.528+0000 7ff5932e0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff58c1134e0 con 0x7ff58c108730 2024-05-27T07:59:53.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff59087b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff58c107de0 0x7ff58c19b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff59107c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff58c1034b0 0x7ff58c19ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff59087b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff58c107de0 0x7ff58c19b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58546/0 (socket says 172.21.15.114:58546) 2024-05-27T07:59:53.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff59087b700 1 -- 172.21.15.114:0/1715094930 learned_addr learned my addr 172.21.15.114:0/1715094930 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:53.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff59187d700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff58c108730 0x7ff58c19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff59107c700 1 -- 172.21.15.114:0/1715094930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff58c108730 msgr2=0x7ff58c19f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:53.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff59107c700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff58c108730 0x7ff58c19f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff59107c700 1 -- 172.21.15.114:0/1715094930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff58c107de0 msgr2=0x7ff58c19b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff59107c700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff58c107de0 0x7ff58c19b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff59107c700 1 -- 172.21.15.114:0/1715094930 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff58c19fcb0 con 0x7ff58c1034b0 2024-05-27T07:59:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff59187d700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff58c108730 0x7ff58c19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff59087b700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff58c107de0 0x7ff58c19b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff59107c700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff58c1034b0 0x7ff58c19ade0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff5880098e0 tx=0x7ff588004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff5827fc700 1 -- 172.21.15.114:0/1715094930 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff588026070 con 0x7ff58c1034b0 2024-05-27T07:59:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff5932e0700 1 -- 172.21.15.114:0/1715094930 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff58c19ff40 con 0x7ff58c1034b0 2024-05-27T07:59:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff5827fc700 1 -- 172.21.15.114:0/1715094930 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff588003c70 con 0x7ff58c1034b0 2024-05-27T07:59:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.529+0000 7ff5827fc700 1 -- 172.21.15.114:0/1715094930 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58800f930 con 0x7ff58c1034b0 2024-05-27T07:59:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.530+0000 7ff5932e0700 1 -- 172.21.15.114:0/1715094930 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff58c0ff1c0 con 0x7ff58c1034b0 2024-05-27T07:59:53.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.533+0000 7ff5827fc700 1 -- 172.21.15.114:0/1715094930 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7ff588029070 con 0x7ff58c1034b0 2024-05-27T07:59:53.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.534+0000 7ff5a1cf5b80 1 Processor -- start 2024-05-27T07:59:53.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.534+0000 7ff5a1cf5b80 1 -- start start 2024-05-27T07:59:53.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.534+0000 7ff5827fc700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff5780b29f0 0x7ff5780b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff59187d700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff5780b29f0 0x7ff5780b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff5827fc700 1 -- 172.21.15.114:0/1715094930 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7ff5880e30a0 con 0x7ff58c1034b0 2024-05-27T07:59:53.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff5a1cf5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef76710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff5a1cf5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef76c50 0x55d15ef70a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff5a1cf5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef70f80 0x55d15ef71390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff5a1cf5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d15ee9fc90 con 0x55d15ef70f80 2024-05-27T07:59:53.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff5a1cf5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d15ee8f830 con 0x55d15ef76c50 2024-05-27T07:59:53.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff59107c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef76710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff59107c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef76710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:58954/0 (socket says 172.21.15.114:58954) 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff5a1cf5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d15ee93870 con 0x55d15edf2e40 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff59107c700 1 -- 172.21.15.114:0/1694919061 learned_addr learned my addr 172.21.15.114:0/1694919061 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff59187d700 1 --2- 172.21.15.114:0/1694919061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef70f80 0x55d15ef71390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff59087b700 1 --2- 172.21.15.114:0/1694919061 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef76c50 0x55d15ef70a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff59107c700 1 -- 172.21.15.114:0/1694919061 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef76c50 msgr2=0x55d15ef70a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff59107c700 1 --2- 172.21.15.114:0/1694919061 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef76c50 0x55d15ef70a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff59107c700 1 -- 172.21.15.114:0/1694919061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef70f80 msgr2=0x55d15ef71390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff59107c700 1 --2- 172.21.15.114:0/1694919061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef70f80 0x55d15ef71390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff59107c700 1 -- 172.21.15.114:0/1694919061 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d15eddf8d0 con 0x55d15edf2e40 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.535+0000 7ff59187d700 1 --2- 172.21.15.114:0/1694919061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef70f80 0x55d15ef71390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.536+0000 7ff59187d700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff5780b29f0 0x7ff5780b4f00 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff584010fd0 tx=0x7ff58400b040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.536+0000 7ff59107c700 1 --2- 172.21.15.114:0/1694919061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef76710 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff5880a9d60 tx=0x7ff5880a98d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e9204ddaeb37398 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.536+0000 7ff5777fe700 1 -- 172.21.15.114:0/1694919061 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5880a9b70 con 0x55d15edf2e40 2024-05-27T07:59:53.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.536+0000 7ff5777fe700 1 -- 172.21.15.114:0/1694919061 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5880aa760 con 0x55d15edf2e40 2024-05-27T07:59:53.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.536+0000 7ff5777fe700 1 -- 172.21.15.114:0/1694919061 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5880aa520 con 0x55d15edf2e40 2024-05-27T07:59:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.537+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/1694919061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 msgr2=0x55d15ef76710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.537+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/1694919061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef76710 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff5880a9d60 tx=0x7ff5880a98d0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.537+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/1694919061 shutdown_connections 2024-05-27T07:59:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.537+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/1694919061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef70f80 0x55d15ef71390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.537+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/1694919061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef76710 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.537+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/1694919061 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef76c50 0x55d15ef70a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.537+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/1694919061 >> 172.21.15.114:0/1694919061 conn(0x55d15ee4daf0 msgr2=0x55d15ef6b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.537+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/1694919061 shutdown_connections 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.537+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/1694919061 wait complete. 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.538+0000 7ff5a1cf5b80 1 -- start start 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.538+0000 7ff5a1cf5b80 1 Processor -- start 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff5a1cf5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15edf2e40 0x55d15ef7a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff5a1cf5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef70f80 0x55d15ef7a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff59087b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef70f80 0x55d15ef7a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff5a1cf5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15ef76c50 0x55d15ef7ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff5a1cf5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d15ed95fa0 con 0x55d15ef70f80 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff5a1cf5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d15ee317c0 con 0x55d15edf2e40 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff59107c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15edf2e40 0x55d15ef7a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff5a1cf5b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d15ef7d500 con 0x55d15ef76c50 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff59107c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15edf2e40 0x55d15ef7a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:35112/0 (socket says 172.21.15.114:35112) 2024-05-27T07:59:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff59107c700 1 -- 172.21.15.114:0/3967107419 learned_addr learned my addr 172.21.15.114:0/3967107419 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff5a1cf5b80 10 client.0 _ll_register_callbacks cb 0x55d15ee2a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff59187d700 1 --2- 172.21.15.114:0/3967107419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15ef76c50 0x55d15ef7ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff59087b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef70f80 0x55d15ef7a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58560/0 (socket says 172.21.15.114:58560) 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff59107c700 1 -- 172.21.15.114:0/3967107419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15ef76c50 msgr2=0x55d15ef7ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff59107c700 1 --2- 172.21.15.114:0/3967107419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15ef76c50 0x55d15ef7ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff59107c700 1 -- 172.21.15.114:0/3967107419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef70f80 msgr2=0x55d15ef7a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff59107c700 1 --2- 172.21.15.114:0/3967107419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef70f80 0x55d15ef7a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.539+0000 7ff59107c700 1 -- 172.21.15.114:0/3967107419 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d15ee2e690 con 0x55d15edf2e40 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.540+0000 7ff59107c700 1 --2- 172.21.15.114:0/3967107419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15edf2e40 0x55d15ef7a3b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff588009d70 tx=0x7ff5880b2d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7a2b64ed856effc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.540+0000 7ff574ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.540+0000 7ff574ff9700 1 -- 172.21.15.114:0/3967107419 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5880c3060 con 0x55d15edf2e40 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.540+0000 7ff574ff9700 1 -- 172.21.15.114:0/3967107419 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7ff5880aa2c0 con 0x55d15edf2e40 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.540+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/3967107419 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5880aa760 con 0x55d15edf2e40 2024-05-27T07:59:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.540+0000 7ff574ff9700 1 -- 172.21.15.114:0/3967107419 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7ff5880aa060 con 0x55d15edf2e40 2024-05-27T07:59:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.540+0000 7ff5a1cf5b80 10 client.29201 fetch_fsmap learned FSMap version 113 2024-05-27T07:59:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.540+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/3967107419 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=4+}) -- 0x55d15ef87f90 con 0x55d15edf2e40 2024-05-27T07:59:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.540+0000 7ff574ff9700 1 -- 172.21.15.114:0/3967107419 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5880b2e30 con 0x55d15edf2e40 2024-05-27T07:59:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.540+0000 7ff574ff9700 1 -- 172.21.15.114:0/3967107419 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 113) ==== 3490+0+0 (secure 0 0 0) 0x7ff5880a59d0 con 0x55d15edf2e40 2024-05-27T07:59:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.540+0000 7ff5a1cf5b80 10 client.29201 fetch_fsmap finished waiting for FSMap version 113 2024-05-27T07:59:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.540+0000 7ff5a1cf5b80 10 client.29201 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.541+0000 7ff5a1cf5b80 20 client.29201 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.541+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/3967107419 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d15ef8a4b0 0x55d15ef8c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.541+0000 7ff5a1cf5b80 4 client.29201 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:59:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.541+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/3967107419 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff5880aa760 con 0x55d15ef8a4b0 2024-05-27T07:59:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.541+0000 7ff59087b700 1 --2- 172.21.15.114:0/3967107419 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d15ef8a4b0 0x55d15ef8c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.541+0000 7ff574ff9700 10 client.29201 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.541+0000 7ff59087b700 1 --2- 172.21.15.114:0/3967107419 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d15ef8a4b0 0x55d15ef8c8d0 crc :-1 s=READY pgs=526 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39a9621e582cb6c3 server_cookie=437916a3f797c337 in_seq=0 out_seq=0 2024-05-27T07:59:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff574ff9700 1 -- 172.21.15.114:0/3967107419 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7ff5880aa760 con 0x55d15ef8a4b0 2024-05-27T07:59:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff574ff9700 10 client.29201 handle_command_reply: tid=0 2024-05-27T07:59:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff5a1cf5b80 1 client.29201 shutdown 2024-05-27T07:59:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/3967107419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15edf2e40 msgr2=0x55d15ef7a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/3967107419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15edf2e40 0x55d15ef7a3b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff588009d70 tx=0x7ff5880b2d30 comp rx=0 tx=0).stop 2024-05-27T07:59:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/3967107419 shutdown_connections 2024-05-27T07:59:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/3967107419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15ef76c50 0x55d15ef7ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/3967107419 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d15ef8a4b0 0x55d15ef8c8d0 crc :-1 s=READY pgs=526 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/3967107419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15edf2e40 0x55d15ef7a3b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/3967107419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef70f80 0x55d15ef7a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff574ff9700 0 client.29201 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/3967107419 >> 172.21.15.114:0/3967107419 conn(0x55d15ee4daf0 msgr2=0x55d15ef6fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/3967107419 shutdown_connections 2024-05-27T07:59:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.545+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/3967107419 wait complete. 2024-05-27T07:59:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.546+0000 7ff5a1cf5b80 20 client.29201 trim_cache size 0 max 16384 2024-05-27T07:59:53.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.577+0000 7ff5a1cf5b80 1 Processor -- start 2024-05-27T07:59:53.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.577+0000 7ff5a1cf5b80 1 -- start start 2024-05-27T07:59:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.578+0000 7ff5a1cf5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef78070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.578+0000 7ff5a1cf5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef70f80 0x55d15ef86560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.578+0000 7ff5a1cf5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef76c50 0x55d15ef920d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.578+0000 7ff5a1cf5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff58c1ad7d0 con 0x55d15ef76c50 2024-05-27T07:59:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.578+0000 7ff5a1cf5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff58c1adea0 con 0x55d15ef70f80 2024-05-27T07:59:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.578+0000 7ff59187d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef76c50 0x55d15ef920d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.578+0000 7ff59187d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef76c50 0x55d15ef920d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58572/0 (socket says 172.21.15.114:58572) 2024-05-27T07:59:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.578+0000 7ff59187d700 1 -- 172.21.15.114:0/595950911 learned_addr learned my addr 172.21.15.114:0/595950911 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.578+0000 7ff59087b700 1 --2- 172.21.15.114:0/595950911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef70f80 0x55d15ef86560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.578+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/595950911 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff58c1ac180 con 0x55d15edf2e40 2024-05-27T07:59:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.578+0000 7ff59107c700 1 --2- 172.21.15.114:0/595950911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef78070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.579+0000 7ff59187d700 1 -- 172.21.15.114:0/595950911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 msgr2=0x55d15ef78070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.579+0000 7ff59187d700 1 --2- 172.21.15.114:0/595950911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef78070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.579+0000 7ff59187d700 1 -- 172.21.15.114:0/595950911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef70f80 msgr2=0x55d15ef86560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.579+0000 7ff59187d700 1 --2- 172.21.15.114:0/595950911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef70f80 0x55d15ef86560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.579+0000 7ff59187d700 1 -- 172.21.15.114:0/595950911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d15ef65fa0 con 0x55d15ef76c50 2024-05-27T07:59:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.579+0000 7ff59087b700 1 --2- 172.21.15.114:0/595950911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef70f80 0x55d15ef86560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.579+0000 7ff59187d700 1 --2- 172.21.15.114:0/595950911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef76c50 0x55d15ef920d0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7ff584009320 tx=0x7ff584015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3dc7cb0918dc653 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.579+0000 7ff576ffd700 1 -- 172.21.15.114:0/595950911 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff584009350 con 0x55d15ef76c50 2024-05-27T07:59:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.579+0000 7ff576ffd700 1 -- 172.21.15.114:0/595950911 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff584030070 con 0x55d15ef76c50 2024-05-27T07:59:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.579+0000 7ff576ffd700 1 -- 172.21.15.114:0/595950911 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58402b850 con 0x55d15ef76c50 2024-05-27T07:59:53.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.580+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/595950911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef76c50 msgr2=0x55d15ef920d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:53.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.580+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/595950911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef76c50 0x55d15ef920d0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7ff584009320 tx=0x7ff584015000 comp rx=0 tx=0).stop 2024-05-27T07:59:53.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.580+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/595950911 shutdown_connections 2024-05-27T07:59:53.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.581+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/595950911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef76c50 0x55d15ef920d0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.581+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/595950911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef78070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.581+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/595950911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef70f80 0x55d15ef86560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.581+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/595950911 >> 172.21.15.114:0/595950911 conn(0x55d15ee4daf0 msgr2=0x55d15ef6b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:53.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.581+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/595950911 shutdown_connections 2024-05-27T07:59:53.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.581+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/595950911 wait complete. 2024-05-27T07:59:53.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.581+0000 7ff5a1cf5b80 1 -- start start 2024-05-27T07:59:53.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.581+0000 7ff5a1cf5b80 1 Processor -- start 2024-05-27T07:59:53.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.582+0000 7ff5a1cf5b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef82a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.582+0000 7ff5a1cf5b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef70f80 0x55d15ef82f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.582+0000 7ff5a1cf5b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef76c50 0x55d15ef914c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.582+0000 7ff5a1cf5b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d15ef796f0 con 0x55d15ef76c50 2024-05-27T07:59:53.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.582+0000 7ff5a1cf5b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff58c1ac7d0 con 0x55d15ef70f80 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.582+0000 7ff59087b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef70f80 0x55d15ef82f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.582+0000 7ff59087b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef70f80 0x55d15ef82f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:35126/0 (socket says 172.21.15.114:35126) 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.582+0000 7ff59087b700 1 -- 172.21.15.114:0/2830664065 learned_addr learned my addr 172.21.15.114:0/2830664065 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.582+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/2830664065 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d15ef76230 con 0x55d15edf2e40 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.582+0000 7ff59107c700 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef82a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.582+0000 7ff59187d700 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef76c50 0x55d15ef914c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.583+0000 7ff5a1cf5b80 10 client.0 _ll_register_callbacks cb 0x55d15ee2a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.583+0000 7ff59087b700 1 -- 172.21.15.114:0/2830664065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 msgr2=0x55d15ef82a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.583+0000 7ff59087b700 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef82a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.583+0000 7ff59087b700 1 -- 172.21.15.114:0/2830664065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef76c50 msgr2=0x55d15ef914c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.583+0000 7ff59087b700 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef76c50 0x55d15ef914c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.583+0000 7ff59087b700 1 -- 172.21.15.114:0/2830664065 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d15ef91c40 con 0x55d15ef70f80 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.583+0000 7ff59107c700 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef82a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:53.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.583+0000 7ff59087b700 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef70f80 0x55d15ef82f40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009690 tx=0x7ff57c013dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d06a518026737545 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:53.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.583+0000 7ff574ff9700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T07:59:53.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.584+0000 7ff574ff9700 1 -- 172.21.15.114:0/2830664065 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57c004830 con 0x55d15ef70f80 2024-05-27T07:59:53.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.584+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/2830664065 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff584030070 con 0x55d15ef70f80 2024-05-27T07:59:53.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.584+0000 7ff574ff9700 1 -- 172.21.15.114:0/2830664065 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7ff57c0049d0 con 0x55d15ef70f80 2024-05-27T07:59:53.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.584+0000 7ff574ff9700 1 -- 172.21.15.114:0/2830664065 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7ff57c004bd0 con 0x55d15ef70f80 2024-05-27T07:59:53.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.584+0000 7ff5a1cf5b80 10 client.29207 fetch_fsmap learned FSMap version 113 2024-05-27T07:59:53.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.584+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/2830664065 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=4+}) -- 0x55d15eff4a20 con 0x55d15ef70f80 2024-05-27T07:59:53.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.584+0000 7ff574ff9700 1 -- 172.21.15.114:0/2830664065 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff584030070 con 0x55d15ef70f80 2024-05-27T07:59:53.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.585+0000 7ff574ff9700 1 -- 172.21.15.114:0/2830664065 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 113) ==== 3490+0+0 (secure 0 0 0) 0x7ff57c004f30 con 0x55d15ef70f80 2024-05-27T07:59:53.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.585+0000 7ff5a1cf5b80 10 client.29207 fetch_fsmap finished waiting for FSMap version 113 2024-05-27T07:59:53.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.585+0000 7ff5a1cf5b80 10 client.29207 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T07:59:53.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.585+0000 7ff5a1cf5b80 20 client.29207 populate_metadata read hostname 'smithi114' 2024-05-27T07:59:53.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.585+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d15eff5110 0x55d15eff7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:53.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.585+0000 7ff5a1cf5b80 4 client.29207 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:59:53.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.585+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/2830664065 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d15ef3c9f0 con 0x55d15eff5110 2024-05-27T07:59:53.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.585+0000 7ff59107c700 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d15eff5110 0x55d15eff7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:53.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.586+0000 7ff59107c700 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d15eff5110 0x55d15eff7530 crc :-1 s=READY pgs=527 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c7c887255d35da7 server_cookie=30815ded3876228f in_seq=0 out_seq=0 2024-05-27T07:59:53.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.586+0000 7ff574ff9700 10 client.29207 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:53.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff574ff9700 1 -- 172.21.15.114:0/2830664065 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d15ef3c9f0 con 0x55d15eff5110 2024-05-27T07:59:53.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff574ff9700 10 client.29207 handle_command_reply: tid=0 2024-05-27T07:59:53.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff5a1cf5b80 1 client.29207 shutdown 2024-05-27T07:59:53.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/2830664065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef70f80 msgr2=0x55d15ef82f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:53.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef70f80 0x55d15ef82f40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009690 tx=0x7ff57c013dc0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/2830664065 shutdown_connections 2024-05-27T07:59:53.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d15ef76c50 0x55d15ef914c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d15edf2e40 0x55d15ef82a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55d15eff5110 0x55d15eff7530 crc :-1 s=READY pgs=527 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff5a1cf5b80 1 --2- 172.21.15.114:0/2830664065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d15ef70f80 0x55d15ef82f40 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff574ff9700 0 client.29207 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T07:59:53.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/2830664065 >> 172.21.15.114:0/2830664065 conn(0x55d15ee4daf0 msgr2=0x55d15ef753f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:53.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/2830664065 shutdown_connections 2024-05-27T07:59:53.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff5a1cf5b80 1 -- 172.21.15.114:0/2830664065 wait complete. 2024-05-27T07:59:53.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.591+0000 7ff5a1cf5b80 20 client.29207 trim_cache size 0 max 16384 2024-05-27T07:59:53.593 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T07:59:53.594 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T07:59:53.594 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T07:59:53.594 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T07:59:53.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.592+0000 7ff5932e0700 1 -- 172.21.15.114:0/1715094930 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff5780b29f0 msgr2=0x7ff5780b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:53.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.592+0000 7ff5932e0700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff5780b29f0 0x7ff5780b4f00 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff584010fd0 tx=0x7ff58400b040 comp rx=0 tx=0).stop 2024-05-27T07:59:53.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.592+0000 7ff5932e0700 1 -- 172.21.15.114:0/1715094930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff58c1034b0 msgr2=0x7ff58c19ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:53.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.592+0000 7ff5932e0700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff58c1034b0 0x7ff58c19ade0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff5880098e0 tx=0x7ff588004270 comp rx=0 tx=0).stop 2024-05-27T07:59:53.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.592+0000 7ff5932e0700 1 -- 172.21.15.114:0/1715094930 shutdown_connections 2024-05-27T07:59:53.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.592+0000 7ff5932e0700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff58c108730 0x7ff58c19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.592+0000 7ff5932e0700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7ff5780b29f0 0x7ff5780b4f00 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.592+0000 7ff5932e0700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff58c1034b0 0x7ff58c19ade0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.592+0000 7ff5932e0700 1 --2- 172.21.15.114:0/1715094930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff58c107de0 0x7ff58c19b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:53.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.592+0000 7ff5932e0700 1 -- 172.21.15.114:0/1715094930 >> 172.21.15.114:0/1715094930 conn(0x7ff58c0fedb0 msgr2=0x7ff58c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:53.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.592+0000 7ff5932e0700 1 -- 172.21.15.114:0/1715094930 shutdown_connections 2024-05-27T07:59:53.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:53.592+0000 7ff5932e0700 1 -- 172.21.15.114:0/1715094930 wait complete. 2024-05-27T07:59:53.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 86084 12.25 MB/sec execute 245 sec latency 169.927 ms 2024-05-27T07:59:53.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:53 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T07:59:53.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:53 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T07:59:53.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:53 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T07:59:53.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:53 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T07:59:54.333 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:59:54.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:59:54.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 86320 12.25 MB/sec execute 246 sec latency 423.123 ms 2024-05-27T07:59:54.633 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:59:54.768 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:54 smithi114 ceph-mon[32038]: pgmap v696: 129 pgs: 129 active+clean; 387 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.5 MiB/s wr, 481 op/s 2024-05-27T07:59:54.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:54 smithi163 ceph-mon[36451]: pgmap v696: 129 pgs: 129 active+clean; 387 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.5 MiB/s wr, 481 op/s 2024-05-27T07:59:54.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:54 smithi186 ceph-mon[36647]: pgmap v696: 129 pgs: 129 active+clean; 387 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.5 MiB/s wr, 481 op/s 2024-05-27T07:59:55.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 86668 12.26 MB/sec execute 247 sec latency 191.691 ms 2024-05-27T07:59:56.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:55 smithi163 ceph-mon[36451]: pgmap v697: 129 pgs: 129 active+clean; 384 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.9 MiB/s wr, 553 op/s 2024-05-27T07:59:56.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:55 smithi186 ceph-mon[36647]: pgmap v697: 129 pgs: 129 active+clean; 384 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.9 MiB/s wr, 553 op/s 2024-05-27T07:59:56.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:55 smithi114 ceph-mon[32038]: pgmap v697: 129 pgs: 129 active+clean; 384 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.9 MiB/s wr, 553 op/s 2024-05-27T07:59:56.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 87056 12.32 MB/sec execute 248 sec latency 264.394 ms 2024-05-27T07:59:56.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.994+0000 7f3caf75b700 1 -- 172.21.15.114:0/2141289416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca8108730 msgr2=0x7f3ca8110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:56.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.994+0000 7f3caf75b700 1 --2- 172.21.15.114:0/2141289416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca8108730 0x7f3ca8110090 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f3ca000b0d0 tx=0x7f3ca000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:59:56.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.994+0000 7f3caf75b700 1 -- 172.21.15.114:0/2141289416 shutdown_connections 2024-05-27T07:59:56.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.994+0000 7f3caf75b700 1 --2- 172.21.15.114:0/2141289416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca8108730 0x7f3ca8110090 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:56.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.994+0000 7f3caf75b700 1 --2- 172.21.15.114:0/2141289416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ca81034b0 0x7f3ca81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:56.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.994+0000 7f3caf75b700 1 --2- 172.21.15.114:0/2141289416 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ca8107de0 0x7f3ca81081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:56.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.994+0000 7f3caf75b700 1 -- 172.21.15.114:0/2141289416 >> 172.21.15.114:0/2141289416 conn(0x7f3ca80fedb0 msgr2=0x7f3ca81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:56.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.994+0000 7f3caf75b700 1 -- 172.21.15.114:0/2141289416 shutdown_connections 2024-05-27T07:59:56.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.994+0000 7f3caf75b700 1 -- 172.21.15.114:0/2141289416 wait complete. 2024-05-27T07:59:56.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.995+0000 7f3caf75b700 1 Processor -- start 2024-05-27T07:59:56.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.995+0000 7f3caf75b700 1 -- start start 2024-05-27T07:59:56.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.995+0000 7f3caf75b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca81034b0 0x7f3ca8100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:56.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.995+0000 7f3caf75b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ca8107de0 0x7f3ca80ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:56.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.995+0000 7f3caf75b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ca8108730 0x7f3ca80ff870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:56.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.995+0000 7f3caf75b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca8113360 con 0x7f3ca81034b0 2024-05-27T07:59:56.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.995+0000 7f3caf75b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca81131e0 con 0x7f3ca8107de0 2024-05-27T07:59:56.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.995+0000 7f3caf75b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca81134e0 con 0x7f3ca8108730 2024-05-27T07:59:56.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.995+0000 7f3cad4f7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca81034b0 0x7f3ca8100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.995+0000 7f3caccf6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ca8107de0 0x7f3ca80ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.995+0000 7f3caccf6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ca8107de0 0x7f3ca80ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:47522/0 (socket says 172.21.15.114:47522) 2024-05-27T07:59:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.995+0000 7f3cadcf8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ca8108730 0x7f3ca80ff870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.995+0000 7f3caccf6700 1 -- 172.21.15.114:0/3341189513 learned_addr learned my addr 172.21.15.114:0/3341189513 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T07:59:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.996+0000 7f3caccf6700 1 -- 172.21.15.114:0/3341189513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ca8108730 msgr2=0x7f3ca80ff870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.996+0000 7f3caccf6700 1 --2- 172.21.15.114:0/3341189513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ca8108730 0x7f3ca80ff870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.996+0000 7f3caccf6700 1 -- 172.21.15.114:0/3341189513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca81034b0 msgr2=0x7f3ca8100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.996+0000 7f3caccf6700 1 --2- 172.21.15.114:0/3341189513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca81034b0 0x7f3ca8100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.996+0000 7f3caccf6700 1 -- 172.21.15.114:0/3341189513 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca81000a0 con 0x7f3ca8107de0 2024-05-27T07:59:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.996+0000 7f3cadcf8700 1 --2- 172.21.15.114:0/3341189513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ca8108730 0x7f3ca80ff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:59:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.996+0000 7f3caccf6700 1 --2- 172.21.15.114:0/3341189513 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ca8107de0 0x7f3ca80ff290 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800ea30 tx=0x7f3c9800ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.996+0000 7f3c9e7fc700 1 -- 172.21.15.114:0/3341189513 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9800cbd0 con 0x7f3ca8107de0 2024-05-27T07:59:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.996+0000 7f3caf75b700 1 -- 172.21.15.114:0/3341189513 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ca81ad890 con 0x7f3ca8107de0 2024-05-27T07:59:57.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.996+0000 7f3c9e7fc700 1 -- 172.21.15.114:0/3341189513 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c980044d0 con 0x7f3ca8107de0 2024-05-27T07:59:57.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.996+0000 7f3c9e7fc700 1 -- 172.21.15.114:0/3341189513 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98003dc0 con 0x7f3ca8107de0 2024-05-27T07:59:57.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.997+0000 7f3caf75b700 1 -- 172.21.15.114:0/3341189513 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ca81adda0 con 0x7f3ca8107de0 2024-05-27T07:59:57.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.998+0000 7f3caf75b700 1 -- 172.21.15.114:0/3341189513 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ca8104960 con 0x7f3ca8107de0 2024-05-27T07:59:57.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:56.999+0000 7f3c9e7fc700 1 -- 172.21.15.114:0/3341189513 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f3c98004000 con 0x7f3ca8107de0 2024-05-27T07:59:57.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.000+0000 7f3c9e7fc700 1 --2- 172.21.15.114:0/3341189513 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3c940b29f0 0x7f3c940b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:59:57.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.000+0000 7f3cad4f7700 1 --2- 172.21.15.114:0/3341189513 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3c940b29f0 0x7f3c940b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:59:57.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.000+0000 7f3c9e7fc700 1 -- 172.21.15.114:0/3341189513 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7f3c980e2b20 con 0x7f3ca8107de0 2024-05-27T07:59:57.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.001+0000 7f3cad4f7700 1 --2- 172.21.15.114:0/3341189513 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3c940b29f0 0x7f3c940b4f00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009680 tx=0x7f3ca4009260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:59:57.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.004+0000 7f3c9e7fc700 1 -- 172.21.15.114:0/3341189513 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3c980042f0 con 0x7f3ca8107de0 2024-05-27T07:59:57.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.391+0000 7f3caf75b700 1 -- 172.21.15.114:0/3341189513 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ca802e5e0 con 0x7f3ca8107de0 2024-05-27T07:59:57.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.391+0000 7f3c9e7fc700 1 -- 172.21.15.114:0/3341189513 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) ==== 95+0+5209 (secure 0 0 0) 0x7f3c980aa1e0 con 0x7f3ca8107de0 2024-05-27T07:59:57.392 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T07:59:57.393 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14523,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1221779710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1221779710},{"type":"v1","addr":"172.21.15.114:6837","nonce":1221779710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":113,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T07:59:47.824622+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_19842":{"gid":19842,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6839/1653686179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":1653686179},{"type":"v1","addr":"172.21.15.114:6839","nonce":1653686179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T07:59:57.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.394+0000 7f3caf75b700 1 -- 172.21.15.114:0/3341189513 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3c940b29f0 msgr2=0x7f3c940b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:57.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.394+0000 7f3caf75b700 1 --2- 172.21.15.114:0/3341189513 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3c940b29f0 0x7f3c940b4f00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009680 tx=0x7f3ca4009260 comp rx=0 tx=0).stop 2024-05-27T07:59:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.394+0000 7f3caf75b700 1 -- 172.21.15.114:0/3341189513 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ca8107de0 msgr2=0x7f3ca80ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:59:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.394+0000 7f3caf75b700 1 --2- 172.21.15.114:0/3341189513 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ca8107de0 0x7f3ca80ff290 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800ea30 tx=0x7f3c9800ed40 comp rx=0 tx=0).stop 2024-05-27T07:59:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.394+0000 7f3caf75b700 1 -- 172.21.15.114:0/3341189513 shutdown_connections 2024-05-27T07:59:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.394+0000 7f3caf75b700 1 --2- 172.21.15.114:0/3341189513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ca8108730 0x7f3ca80ff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.394+0000 7f3caf75b700 1 --2- 172.21.15.114:0/3341189513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca81034b0 0x7f3ca8100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.394+0000 7f3caf75b700 1 --2- 172.21.15.114:0/3341189513 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f3c940b29f0 0x7f3c940b4f00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.394+0000 7f3caf75b700 1 --2- 172.21.15.114:0/3341189513 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ca8107de0 0x7f3ca80ff290 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:59:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.394+0000 7f3caf75b700 1 -- 172.21.15.114:0/3341189513 >> 172.21.15.114:0/3341189513 conn(0x7f3ca80fedb0 msgr2=0x7f3ca806b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:59:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.394+0000 7f3caf75b700 1 -- 172.21.15.114:0/3341189513 shutdown_connections 2024-05-27T07:59:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T07:59:57.394+0000 7f3caf75b700 1 -- 172.21.15.114:0/3341189513 wait complete. 2024-05-27T07:59:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 113 2024-05-27T07:59:57.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 87353 12.35 MB/sec execute 249 sec latency 188.342 ms 2024-05-27T07:59:57.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:57 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3341189513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:59:57.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:57 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3341189513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:59:57.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:57 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3341189513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:59:58.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:59:58.382 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:59:58.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 87731 12.39 MB/sec execute 250 sec latency 170.684 ms 2024-05-27T07:59:58.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:58 smithi163 ceph-mon[36451]: pgmap v698: 129 pgs: 129 active+clean; 389 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 570 op/s 2024-05-27T07:59:58.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:58 smithi186 ceph-mon[36647]: pgmap v698: 129 pgs: 129 active+clean; 389 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 570 op/s 2024-05-27T07:59:58.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:58 smithi114 ceph-mon[32038]: pgmap v698: 129 pgs: 129 active+clean; 389 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 570 op/s 2024-05-27T07:59:59.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 87944 12.37 MB/sec execute 251 sec latency 378.490 ms 2024-05-27T08:00:00.152 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 07:59:59 smithi114 ceph-mon[32038]: pgmap v699: 129 pgs: 129 active+clean; 400 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 533 op/s 2024-05-27T08:00:00.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 07:59:59 smithi163 ceph-mon[36451]: pgmap v699: 129 pgs: 129 active+clean; 400 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 533 op/s 2024-05-27T08:00:00.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 07:59:59 smithi186 ceph-mon[36647]: pgmap v699: 129 pgs: 129 active+clean; 400 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 533 op/s 2024-05-27T08:00:00.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.539+0000 7fcda6c31700 1 -- 172.21.15.114:0/4104339034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcda010a980 msgr2=0x7fcda010ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:00.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.539+0000 7fcda6c31700 1 --2- 172.21.15.114:0/4104339034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcda010a980 0x7fcda010ad90 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009990 tx=0x7fcd90009ca0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.541+0000 7fcda6c31700 1 -- 172.21.15.114:0/4104339034 shutdown_connections 2024-05-27T08:00:00.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.541+0000 7fcda6c31700 1 --2- 172.21.15.114:0/4104339034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcda010b2d0 0x7fcda0117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.541+0000 7fcda6c31700 1 --2- 172.21.15.114:0/4104339034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcda0110960 0x7fcda0110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.541+0000 7fcda6c31700 1 --2- 172.21.15.114:0/4104339034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcda010a980 0x7fcda010ad90 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.541+0000 7fcda6c31700 1 -- 172.21.15.114:0/4104339034 >> 172.21.15.114:0/4104339034 conn(0x7fcda0077c70 msgr2=0x7fcda00780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:00.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.542+0000 7fcda6c31700 1 -- 172.21.15.114:0/4104339034 shutdown_connections 2024-05-27T08:00:00.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.543+0000 7fcda6c31700 1 -- 172.21.15.114:0/4104339034 wait complete. 2024-05-27T08:00:00.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.544+0000 7fcda6c31700 1 Processor -- start 2024-05-27T08:00:00.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.544+0000 7fcda6c31700 1 -- start start 2024-05-27T08:00:00.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.544+0000 7fcda6c31700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcda010a980 0x7fcda0085190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.544+0000 7fcda6c31700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcda010b2d0 0x7fcda007d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.544+0000 7fcda6c31700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcda0110960 0x7fcda007d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.544+0000 7fcda6c31700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcda011ac00 con 0x7fcda010b2d0 2024-05-27T08:00:00.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.544+0000 7fcda6c31700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcda011aa80 con 0x7fcda010a980 2024-05-27T08:00:00.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.544+0000 7fcda6c31700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcda011ad80 con 0x7fcda0110960 2024-05-27T08:00:00.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.546+0000 7fcda51ce700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcda0110960 0x7fcda007d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.546+0000 7fcda51ce700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcda0110960 0x7fcda007d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:49966/0 (socket says 172.21.15.114:49966) 2024-05-27T08:00:00.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.546+0000 7fcda51ce700 1 -- 172.21.15.114:0/893444101 learned_addr learned my addr 172.21.15.114:0/893444101 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:00.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.546+0000 7fcda49cd700 1 --2- 172.21.15.114:0/893444101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcda010a980 0x7fcda0085190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.546+0000 7fcda51ce700 1 -- 172.21.15.114:0/893444101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcda010a980 msgr2=0x7fcda0085190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:00.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.546+0000 7fcda51ce700 1 --2- 172.21.15.114:0/893444101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcda010a980 0x7fcda0085190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.546+0000 7fcda51ce700 1 -- 172.21.15.114:0/893444101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcda010b2d0 msgr2=0x7fcda007d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:00.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.546+0000 7fcda51ce700 1 --2- 172.21.15.114:0/893444101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcda010b2d0 0x7fcda007d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.547+0000 7fcda51ce700 1 -- 172.21.15.114:0/893444101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda007de80 con 0x7fcda0110960 2024-05-27T08:00:00.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.551+0000 7fcda51ce700 1 --2- 172.21.15.114:0/893444101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcda0110960 0x7fcda007d740 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fcd9802d970 tx=0x7fcd9802dc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:00.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.554+0000 7fcd9dffb700 1 -- 172.21.15.114:0/893444101 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9803b070 con 0x7fcda0110960 2024-05-27T08:00:00.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.554+0000 7fcda6c31700 1 -- 172.21.15.114:0/893444101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcda01a1330 con 0x7fcda0110960 2024-05-27T08:00:00.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.554+0000 7fcda6c31700 1 -- 172.21.15.114:0/893444101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcda01a1870 con 0x7fcda0110960 2024-05-27T08:00:00.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.555+0000 7fcd9dffb700 1 -- 172.21.15.114:0/893444101 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd98028e50 con 0x7fcda0110960 2024-05-27T08:00:00.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.555+0000 7fcd9dffb700 1 -- 172.21.15.114:0/893444101 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd98036d40 con 0x7fcda0110960 2024-05-27T08:00:00.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.565+0000 7fcd9dffb700 1 -- 172.21.15.114:0/893444101 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fcd9803e070 con 0x7fcda0110960 2024-05-27T08:00:00.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.566+0000 7fcdb5646b80 1 Processor -- start 2024-05-27T08:00:00.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.566+0000 7fcdb5646b80 1 -- start start 2024-05-27T08:00:00.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.566+0000 7fcdb5646b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c3fbe40 0x56505c57f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.566+0000 7fcdb5646b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c57fc50 0x56505c579a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.566+0000 7fcdb5646b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 0x56505c57a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.566+0000 7fcdb5646b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56505c4a8c90 con 0x56505c579f80 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.566+0000 7fcdb5646b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56505c498830 con 0x56505c57fc50 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.566+0000 7fcdb5646b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56505c49c870 con 0x56505c3fbe40 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.566+0000 7fcd9ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 0x56505c57a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.566+0000 7fcd9ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 0x56505c57a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54444/0 (socket says 172.21.15.114:54444) 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.566+0000 7fcd9ffff700 1 -- 172.21.15.114:0/3592960768 learned_addr learned my addr 172.21.15.114:0/3592960768 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.566+0000 7fcd9ffff700 1 --2- 172.21.15.114:0/3592960768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c3fbe40 0x56505c57f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.566+0000 7fcda49cd700 1 --2- 172.21.15.114:0/3592960768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c57fc50 0x56505c579a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.567+0000 7fcd9dffb700 1 --2- 172.21.15.114:0/893444101 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcd880b2c00 0x7fcd880b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.568+0000 7fcd9dffb700 1 -- 172.21.15.114:0/893444101 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7fcd981038f0 con 0x7fcda0110960 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.568+0000 7fcda49cd700 1 -- 172.21.15.114:0/3592960768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c3fbe40 msgr2=0x56505c57f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.568+0000 7fcda49cd700 1 --2- 172.21.15.114:0/3592960768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c3fbe40 0x56505c57f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.568+0000 7fcda49cd700 1 -- 172.21.15.114:0/3592960768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 msgr2=0x56505c57a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.568+0000 7fcda49cd700 1 --2- 172.21.15.114:0/3592960768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 0x56505c57a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.568+0000 7fcda49cd700 1 -- 172.21.15.114:0/3592960768 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56505c3e88d0 con 0x56505c57fc50 2024-05-27T08:00:00.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.570+0000 7fcda49cd700 1 --2- 172.21.15.114:0/3592960768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c57fc50 0x56505c579a40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcd940099f0 tx=0x7fcd94009d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6a00b16aff2c22b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:00.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.571+0000 7fcda51ce700 1 --2- 172.21.15.114:0/893444101 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcd880b2c00 0x7fcd880b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.571+0000 7fcd86ffd700 1 -- 172.21.15.114:0/3592960768 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd94017070 con 0x56505c57fc50 2024-05-27T08:00:00.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.571+0000 7fcd86ffd700 1 -- 172.21.15.114:0/3592960768 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd9400ed80 con 0x56505c57fc50 2024-05-27T08:00:00.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.572+0000 7fcdb5646b80 1 -- 172.21.15.114:0/3592960768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c57fc50 msgr2=0x56505c579a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:00.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.572+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/3592960768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c57fc50 0x56505c579a40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcd940099f0 tx=0x7fcd94009d00 comp rx=0 tx=0).stop 2024-05-27T08:00:00.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.576+0000 7fcdb5646b80 1 -- 172.21.15.114:0/3592960768 shutdown_connections 2024-05-27T08:00:00.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.576+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/3592960768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 0x56505c57a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.576+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/3592960768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c3fbe40 0x56505c57f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.576+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/3592960768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c57fc50 0x56505c579a40 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.576+0000 7fcdb5646b80 1 -- 172.21.15.114:0/3592960768 >> 172.21.15.114:0/3592960768 conn(0x56505c456af0 msgr2=0x56505c574640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:00.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.576+0000 7fcdb5646b80 1 -- 172.21.15.114:0/3592960768 shutdown_connections 2024-05-27T08:00:00.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.577+0000 7fcda51ce700 1 --2- 172.21.15.114:0/893444101 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcd880b2c00 0x7fcd880b5110 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fcd9802e430 tx=0x7fcd980ed040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:00.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.577+0000 7fcdb5646b80 1 -- 172.21.15.114:0/3592960768 wait complete. 2024-05-27T08:00:00.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.577+0000 7fcdb5646b80 1 -- start start 2024-05-27T08:00:00.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.578+0000 7fcdb5646b80 1 Processor -- start 2024-05-27T08:00:00.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.578+0000 7fcdb5646b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c3fbe40 0x56505c585e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.578+0000 7fcdb5646b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 0x56505c5831e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.578+0000 7fcdb5646b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c57fc50 0x56505c583880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.578+0000 7fcdb5646b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56505c39efa0 con 0x56505c579f80 2024-05-27T08:00:00.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.578+0000 7fcdb5646b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56505c43a7c0 con 0x56505c57fc50 2024-05-27T08:00:00.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.578+0000 7fcdb5646b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56505c586790 con 0x56505c3fbe40 2024-05-27T08:00:00.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.578+0000 7fcdb5646b80 10 client.0 _ll_register_callbacks cb 0x56505c433920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T08:00:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.579+0000 7fcd9ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c57fc50 0x56505c583880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.579+0000 7fcd9ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c57fc50 0x56505c583880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:47574/0 (socket says 172.21.15.114:47574) 2024-05-27T08:00:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.579+0000 7fcd9ffff700 1 -- 172.21.15.114:0/2491681308 learned_addr learned my addr 172.21.15.114:0/2491681308 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.579+0000 7fcd9ffff700 1 --2- 172.21.15.114:0/2491681308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c3fbe40 0x56505c585e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.579+0000 7fcda49cd700 1 --2- 172.21.15.114:0/2491681308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 0x56505c5831e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.579+0000 7fcd9ffff700 1 -- 172.21.15.114:0/2491681308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c3fbe40 msgr2=0x56505c585e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.579+0000 7fcd9ffff700 1 --2- 172.21.15.114:0/2491681308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c3fbe40 0x56505c585e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.579+0000 7fcd9ffff700 1 -- 172.21.15.114:0/2491681308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 msgr2=0x56505c5831e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.579+0000 7fcd9ffff700 1 --2- 172.21.15.114:0/2491681308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 0x56505c5831e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.579+0000 7fcd9ffff700 1 -- 172.21.15.114:0/2491681308 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56505c437690 con 0x56505c57fc50 2024-05-27T08:00:00.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.587+0000 7fcd9ffff700 1 --2- 172.21.15.114:0/2491681308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c57fc50 0x56505c583880 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009fd0 tx=0x7fcd90003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba6b0c9a2a678092 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:00.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.587+0000 7fcd7ffff700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T08:00:00.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.591+0000 7fcd7ffff700 1 -- 172.21.15.114:0/2491681308 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9002e060 con 0x56505c57fc50 2024-05-27T08:00:00.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.591+0000 7fcd7ffff700 1 -- 172.21.15.114:0/2491681308 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7fcd90003af0 con 0x56505c57fc50 2024-05-27T08:00:00.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.591+0000 7fcdb5646b80 1 -- 172.21.15.114:0/2491681308 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd9400ed80 con 0x56505c57fc50 2024-05-27T08:00:00.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.592+0000 7fcd7ffff700 1 -- 172.21.15.114:0/2491681308 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7fcd90035050 con 0x56505c57fc50 2024-05-27T08:00:00.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.593+0000 7fcdb5646b80 10 client.29222 fetch_fsmap learned FSMap version 113 2024-05-27T08:00:00.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.593+0000 7fcdb5646b80 1 -- 172.21.15.114:0/2491681308 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=4+}) -- 0x56505c5a2180 con 0x56505c57fc50 2024-05-27T08:00:00.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.594+0000 7fcd7ffff700 1 -- 172.21.15.114:0/2491681308 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd9400ed80 con 0x56505c57fc50 2024-05-27T08:00:00.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.594+0000 7fcd7ffff700 1 -- 172.21.15.114:0/2491681308 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 113) ==== 3490+0+0 (secure 0 0 0) 0x7fcd90010840 con 0x56505c57fc50 2024-05-27T08:00:00.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.594+0000 7fcdb5646b80 10 client.29222 fetch_fsmap finished waiting for FSMap version 113 2024-05-27T08:00:00.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.594+0000 7fcdb5646b80 10 client.29222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T08:00:00.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.594+0000 7fcdb5646b80 20 client.29222 populate_metadata read hostname 'smithi114' 2024-05-27T08:00:00.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.594+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/2491681308 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56505c5a4b10 0x56505c5a6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.594+0000 7fcdb5646b80 4 client.29222 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T08:00:00.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.594+0000 7fcdb5646b80 1 -- 172.21.15.114:0/2491681308 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56505c5459f0 con 0x56505c5a4b10 2024-05-27T08:00:00.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.596+0000 7fcda49cd700 1 --2- 172.21.15.114:0/2491681308 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56505c5a4b10 0x56505c5a6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.601+0000 7fcda49cd700 1 --2- 172.21.15.114:0/2491681308 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56505c5a4b10 0x56505c5a6fa0 crc :-1 s=READY pgs=528 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12ace5b2df85538d server_cookie=928d27a09816f7c3 in_seq=0 out_seq=0 2024-05-27T08:00:00.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.601+0000 7fcd7ffff700 10 client.29222 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:00.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.603+0000 7fcd7ffff700 1 -- 172.21.15.114:0/2491681308 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56505c5459f0 con 0x56505c5a4b10 2024-05-27T08:00:00.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.603+0000 7fcd7ffff700 10 client.29222 handle_command_reply: tid=0 2024-05-27T08:00:00.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.604+0000 7fcdb5646b80 1 client.29222 shutdown 2024-05-27T08:00:00.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.604+0000 7fcdb5646b80 1 -- 172.21.15.114:0/2491681308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c57fc50 msgr2=0x56505c583880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:00.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.604+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/2491681308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c57fc50 0x56505c583880 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009fd0 tx=0x7fcd90003680 comp rx=0 tx=0).stop 2024-05-27T08:00:00.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.604+0000 7fcdb5646b80 1 -- 172.21.15.114:0/2491681308 shutdown_connections 2024-05-27T08:00:00.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.604+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/2491681308 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56505c5a4b10 0x56505c5a6fa0 crc :-1 s=READY pgs=528 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.604+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/2491681308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c57fc50 0x56505c583880 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.604+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/2491681308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c3fbe40 0x56505c585e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.604+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/2491681308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 0x56505c5831e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.604+0000 7fcdb5646b80 1 -- 172.21.15.114:0/2491681308 >> 172.21.15.114:0/2491681308 conn(0x56505c456af0 msgr2=0x56505c578a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:00.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.608+0000 7fcd7ffff700 0 client.29222 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:00.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.612+0000 7fcdb5646b80 1 -- 172.21.15.114:0/2491681308 shutdown_connections 2024-05-27T08:00:00.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.612+0000 7fcdb5646b80 1 -- 172.21.15.114:0/2491681308 wait complete. 2024-05-27T08:00:00.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.613+0000 7fcdb5646b80 20 client.29222 trim_cache size 0 max 16384 2024-05-27T08:00:00.622 INFO:tasks.workunit.client.0.smithi114.stdout: 10 88129 12.34 MB/sec execute 252 sec latency 566.682 ms 2024-05-27T08:00:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.643+0000 7fcdb5646b80 1 Processor -- start 2024-05-27T08:00:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.643+0000 7fcdb5646b80 1 -- start start 2024-05-27T08:00:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.644+0000 7fcdb5646b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c3fbe40 0x56505c579a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.644+0000 7fcdb5646b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c579f80 0x56505c5a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.644+0000 7fcdb5646b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c57fc50 0x56505c5ac7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.644+0000 7fcdb5646b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8c00e3e0 con 0x56505c3fbe40 2024-05-27T08:00:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.644+0000 7fcdb5646b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8c00eab0 con 0x56505c579f80 2024-05-27T08:00:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.644+0000 7fcdb5646b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8c00cd70 con 0x56505c57fc50 2024-05-27T08:00:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.644+0000 7fcd9ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c3fbe40 0x56505c579a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.645+0000 7fcd9ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c3fbe40 0x56505c579a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54472/0 (socket says 172.21.15.114:54472) 2024-05-27T08:00:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.645+0000 7fcd9ffff700 1 -- 172.21.15.114:0/2586863088 learned_addr learned my addr 172.21.15.114:0/2586863088 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.645+0000 7fcd9ffff700 1 --2- 172.21.15.114:0/2586863088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c57fc50 0x56505c5ac7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.649+0000 7fcda49cd700 1 --2- 172.21.15.114:0/2586863088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c579f80 0x56505c5a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.652+0000 7fcd9ffff700 1 -- 172.21.15.114:0/2586863088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c579f80 msgr2=0x56505c5a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:00.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.652+0000 7fcd9ffff700 1 --2- 172.21.15.114:0/2586863088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c579f80 0x56505c5a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.652+0000 7fcd9ffff700 1 -- 172.21.15.114:0/2586863088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c3fbe40 msgr2=0x56505c579a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:00.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.652+0000 7fcd9ffff700 1 --2- 172.21.15.114:0/2586863088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c3fbe40 0x56505c579a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.652+0000 7fcd9ffff700 1 -- 172.21.15.114:0/2586863088 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56505c57f260 con 0x56505c57fc50 2024-05-27T08:00:00.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.658+0000 7fcd9ffff700 1 --2- 172.21.15.114:0/2586863088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c57fc50 0x56505c5ac7a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcd9000c010 tx=0x7fcd90000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=296aff84b9af5795 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:00.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.662+0000 7fcd867fc700 1 -- 172.21.15.114:0/2586863088 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9002e070 con 0x56505c57fc50 2024-05-27T08:00:00.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.662+0000 7fcd867fc700 1 -- 172.21.15.114:0/2586863088 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd900038f0 con 0x56505c57fc50 2024-05-27T08:00:00.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.662+0000 7fcd867fc700 1 -- 172.21.15.114:0/2586863088 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd90020040 con 0x56505c57fc50 2024-05-27T08:00:00.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.663+0000 7fcdb5646b80 1 -- 172.21.15.114:0/2586863088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c57fc50 msgr2=0x56505c5ac7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:00.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.663+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/2586863088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c57fc50 0x56505c5ac7a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcd9000c010 tx=0x7fcd90000bc0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.665+0000 7fcdb5646b80 1 -- 172.21.15.114:0/2586863088 shutdown_connections 2024-05-27T08:00:00.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.665+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/2586863088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c57fc50 0x56505c5ac7a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.665+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/2586863088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c3fbe40 0x56505c579a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.665+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/2586863088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c579f80 0x56505c5a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.665+0000 7fcdb5646b80 1 -- 172.21.15.114:0/2586863088 >> 172.21.15.114:0/2586863088 conn(0x56505c456af0 msgr2=0x56505c5a6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:00.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.673+0000 7fcdb5646b80 1 -- 172.21.15.114:0/2586863088 shutdown_connections 2024-05-27T08:00:00.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.674+0000 7fcdb5646b80 1 -- 172.21.15.114:0/2586863088 wait complete. 2024-05-27T08:00:00.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.674+0000 7fcdb5646b80 1 -- start start 2024-05-27T08:00:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.674+0000 7fcdb5646b80 1 Processor -- start 2024-05-27T08:00:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.675+0000 7fcdb5646b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c3fbe40 0x56505c59cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.675+0000 7fcdb5646b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 0x56505c59d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.675+0000 7fcdb5646b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c57fc50 0x56505c59f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.675+0000 7fcdb5646b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56505c586870 con 0x56505c579f80 2024-05-27T08:00:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.675+0000 7fcdb5646b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8c00d3e0 con 0x56505c3fbe40 2024-05-27T08:00:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.675+0000 7fcdb5646b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56505c5857c0 con 0x56505c57fc50 2024-05-27T08:00:00.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.675+0000 7fcdb5646b80 10 client.0 _ll_register_callbacks cb 0x56505c433920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T08:00:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.682+0000 7fcd9ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c3fbe40 0x56505c59cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.682+0000 7fcd9ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c57fc50 0x56505c59f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.683+0000 7fcda49cd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 0x56505c59d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.683+0000 7fcd9ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c3fbe40 0x56505c59cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:47592/0 (socket says 172.21.15.114:47592) 2024-05-27T08:00:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.683+0000 7fcd9ffff700 1 -- 172.21.15.114:0/3230253681 learned_addr learned my addr 172.21.15.114:0/3230253681 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.683+0000 7fcd9ffff700 1 -- 172.21.15.114:0/3230253681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c3fbe40 msgr2=0x56505c59cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.683+0000 7fcd9ffff700 1 --2- 172.21.15.114:0/3230253681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c3fbe40 0x56505c59cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.683+0000 7fcd9ffff700 1 -- 172.21.15.114:0/3230253681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 msgr2=0x56505c59d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:00.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.683+0000 7fcd9ffff700 1 --2- 172.21.15.114:0/3230253681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 0x56505c59d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.683+0000 7fcd9ffff700 1 -- 172.21.15.114:0/3230253681 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56505c5a0070 con 0x56505c57fc50 2024-05-27T08:00:00.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.687+0000 7fcd9ffff700 1 --2- 172.21.15.114:0/3230253681 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c57fc50 0x56505c59f990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009fd0 tx=0x7fcd90005960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=321a40d7eac5fd21 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:00.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.687+0000 7fcd7ffff700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-27T08:00:00.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.691+0000 7fcd7ffff700 1 -- 172.21.15.114:0/3230253681 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9003e040 con 0x56505c57fc50 2024-05-27T08:00:00.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.691+0000 7fcdb5646b80 1 -- 172.21.15.114:0/3230253681 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd900038f0 con 0x56505c57fc50 2024-05-27T08:00:00.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.691+0000 7fcd7ffff700 1 -- 172.21.15.114:0/3230253681 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7fcd90003aa0 con 0x56505c57fc50 2024-05-27T08:00:00.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.697+0000 7fcd7ffff700 1 -- 172.21.15.114:0/3230253681 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7fcd9002e040 con 0x56505c57fc50 2024-05-27T08:00:00.701 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 08:00:00 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-05-27T08:00:00.701 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 08:00:00 smithi114 systemd[1]: Stopped Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:00:00.701 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 08:00:00 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Start request repeated too quickly. 2024-05-27T08:00:00.701 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 08:00:00 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T08:00:00.701 INFO:journalctl@ceph.mds.a.smithi114.stdout:May 27 08:00:00 smithi114 systemd[1]: Failed to start Ceph mds.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:00:00.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.702+0000 7fcdb5646b80 10 client.29389 fetch_fsmap learned FSMap version 113 2024-05-27T08:00:00.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.702+0000 7fcdb5646b80 1 -- 172.21.15.114:0/3230253681 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=4+}) -- 0x56505c60efb0 con 0x56505c57fc50 2024-05-27T08:00:00.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.705+0000 7fcd7ffff700 1 -- 172.21.15.114:0/3230253681 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd900038f0 con 0x56505c57fc50 2024-05-27T08:00:00.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.705+0000 7fcd7ffff700 1 -- 172.21.15.114:0/3230253681 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 113) ==== 3490+0+0 (secure 0 0 0) 0x7fcd900285e0 con 0x56505c57fc50 2024-05-27T08:00:00.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.705+0000 7fcdb5646b80 10 client.29389 fetch_fsmap finished waiting for FSMap version 113 2024-05-27T08:00:00.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.705+0000 7fcdb5646b80 10 client.29389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T08:00:00.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.705+0000 7fcdb5646b80 20 client.29389 populate_metadata read hostname 'smithi114' 2024-05-27T08:00:00.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.705+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/3230253681 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56505c60f2f0 0x56505c611710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:00.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.705+0000 7fcdb5646b80 4 client.29389 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T08:00:00.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.705+0000 7fcdb5646b80 1 -- 172.21.15.114:0/3230253681 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56505c4aabe0 con 0x56505c60f2f0 2024-05-27T08:00:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.713+0000 7fcda49cd700 1 --2- 172.21.15.114:0/3230253681 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56505c60f2f0 0x56505c611710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:00.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.723+0000 7fcd7ffff700 10 client.29389 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:00.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.723+0000 7fcda49cd700 1 --2- 172.21.15.114:0/3230253681 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56505c60f2f0 0x56505c611710 crc :-1 s=READY pgs=529 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aede4cb04d833d15 server_cookie=d540f36de15b1308 in_seq=0 out_seq=0 2024-05-27T08:00:00.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.730+0000 7fcd7ffff700 1 -- 172.21.15.114:0/3230253681 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56505c4aabe0 con 0x56505c60f2f0 2024-05-27T08:00:00.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.730+0000 7fcd7ffff700 10 client.29389 handle_command_reply: tid=0 2024-05-27T08:00:00.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.730+0000 7fcdb5646b80 1 client.29389 shutdown 2024-05-27T08:00:00.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.730+0000 7fcdb5646b80 1 -- 172.21.15.114:0/3230253681 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c57fc50 msgr2=0x56505c59f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:00.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.730+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/3230253681 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c57fc50 0x56505c59f990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009fd0 tx=0x7fcd90005960 comp rx=0 tx=0).stop 2024-05-27T08:00:00.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.730+0000 7fcdb5646b80 1 -- 172.21.15.114:0/3230253681 shutdown_connections 2024-05-27T08:00:00.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.730+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/3230253681 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56505c57fc50 0x56505c59f990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.730+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/3230253681 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x56505c60f2f0 0x56505c611710 crc :-1 s=READY pgs=529 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.730+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/3230253681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56505c3fbe40 0x56505c59cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.730+0000 7fcdb5646b80 1 --2- 172.21.15.114:0/3230253681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56505c579f80 0x56505c59d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.730+0000 7fcdb5646b80 1 -- 172.21.15.114:0/3230253681 >> 172.21.15.114:0/3230253681 conn(0x56505c456af0 msgr2=0x56505c5a6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:00.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.730+0000 7fcd7ffff700 0 client.29389 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:00.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.731+0000 7fcdb5646b80 1 -- 172.21.15.114:0/3230253681 shutdown_connections 2024-05-27T08:00:00.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.731+0000 7fcdb5646b80 1 -- 172.21.15.114:0/3230253681 wait complete. 2024-05-27T08:00:00.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.732+0000 7fcdb5646b80 20 client.29389 trim_cache size 0 max 16384 2024-05-27T08:00:00.734 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T08:00:00.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.733+0000 7fcda6c31700 1 -- 172.21.15.114:0/893444101 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcd880b2c00 msgr2=0x7fcd880b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:00.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.733+0000 7fcda6c31700 1 --2- 172.21.15.114:0/893444101 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcd880b2c00 0x7fcd880b5110 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fcd9802e430 tx=0x7fcd980ed040 comp rx=0 tx=0).stop 2024-05-27T08:00:00.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.733+0000 7fcda6c31700 1 -- 172.21.15.114:0/893444101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcda0110960 msgr2=0x7fcda007d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:00.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.733+0000 7fcda6c31700 1 --2- 172.21.15.114:0/893444101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcda0110960 0x7fcda007d740 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fcd9802d970 tx=0x7fcd9802dc80 comp rx=0 tx=0).stop 2024-05-27T08:00:00.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.734+0000 7fcda6c31700 1 -- 172.21.15.114:0/893444101 shutdown_connections 2024-05-27T08:00:00.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.734+0000 7fcda6c31700 1 --2- 172.21.15.114:0/893444101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcda0110960 0x7fcda007d740 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.734+0000 7fcda6c31700 1 --2- 172.21.15.114:0/893444101 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fcd880b2c00 0x7fcd880b5110 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.734+0000 7fcda6c31700 1 --2- 172.21.15.114:0/893444101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcda010a980 0x7fcda0085190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.734+0000 7fcda6c31700 1 --2- 172.21.15.114:0/893444101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcda010b2d0 0x7fcda007d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:00.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.734+0000 7fcda6c31700 1 -- 172.21.15.114:0/893444101 >> 172.21.15.114:0/893444101 conn(0x7fcda0077c70 msgr2=0x7fcda007b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:00.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.735+0000 7fcda6c31700 1 -- 172.21.15.114:0/893444101 shutdown_connections 2024-05-27T08:00:00.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:00.735+0000 7fcda6c31700 1 -- 172.21.15.114:0/893444101 wait complete. 2024-05-27T08:00:01.056 INFO:tasks.workunit.client.0.smithi114.stdout:[87280] write failed on handle 11579 (Resource temporarily unavailable) 2024-05-27T08:00:01.056 INFO:tasks.workunit.client.0.smithi114.stdout:Child failed with status 1 2024-05-27T08:00:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:00:01.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:00 smithi114 ceph-mon[32038]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s) 2024-05-27T08:00:01.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:00 smithi114 ceph-mon[32038]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-05-27T08:00:01.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:00 smithi114 ceph-mon[32038]: daemon mds.h on smithi163 is in error state 2024-05-27T08:00:01.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:00 smithi114 ceph-mon[32038]: daemon mds.b on smithi163 is in error state 2024-05-27T08:00:01.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:00 smithi114 ceph-mon[32038]: daemon mds.f on smithi186 is in error state 2024-05-27T08:00:01.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:00 smithi114 ceph-mon[32038]: daemon mds.c on smithi186 is in unknown state 2024-05-27T08:00:01.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:00 smithi163 ceph-mon[36451]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s) 2024-05-27T08:00:01.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:00 smithi163 ceph-mon[36451]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-05-27T08:00:01.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:00 smithi163 ceph-mon[36451]: daemon mds.h on smithi163 is in error state 2024-05-27T08:00:01.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:00 smithi163 ceph-mon[36451]: daemon mds.b on smithi163 is in error state 2024-05-27T08:00:01.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:00 smithi163 ceph-mon[36451]: daemon mds.f on smithi186 is in error state 2024-05-27T08:00:01.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:00 smithi163 ceph-mon[36451]: daemon mds.c on smithi186 is in unknown state 2024-05-27T08:00:01.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:00 smithi186 ceph-mon[36647]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s) 2024-05-27T08:00:01.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:00 smithi186 ceph-mon[36647]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-05-27T08:00:01.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:00 smithi186 ceph-mon[36647]: daemon mds.h on smithi163 is in error state 2024-05-27T08:00:01.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:00 smithi186 ceph-mon[36647]: daemon mds.b on smithi163 is in error state 2024-05-27T08:00:01.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:00 smithi186 ceph-mon[36647]: daemon mds.f on smithi186 is in error state 2024-05-27T08:00:01.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:00 smithi186 ceph-mon[36647]: daemon mds.c on smithi186 is in unknown state 2024-05-27T08:00:01.263 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-05-27T08:00:01.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-27T08:00:01.981 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi114 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-05-27T08:00:02.243 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=43b24159ea6f4cf5b69a0c12e17fd3a0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi114 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-05-27T08:00:02.246 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-05-27T08:00:02.257 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi114.front.sepia.ceph.com 2024-05-27T08:00:02.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:01 smithi186 ceph-mon[36647]: pgmap v700: 129 pgs: 129 active+clean; 404 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.7 MiB/s wr, 524 op/s 2024-05-27T08:00:02.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:01 smithi163 ceph-mon[36451]: pgmap v700: 129 pgs: 129 active+clean; 404 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.7 MiB/s wr, 524 op/s 2024-05-27T08:00:02.270 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:01 smithi114 ceph-mon[32038]: pgmap v700: 129 pgs: 129 active+clean; 404 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.7 MiB/s wr, 524 op/s 2024-05-27T08:00:02.270 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-05-27T08:00:02.280 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-05-27T08:00:02.280 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T08:00:02.280 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-05-27T08:00:02.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-05-27T08:00:02.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-05-27T08:00:02.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-27T08:00:02.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.563+0000 7fdb96196640 1 -- 172.21.15.114:0/4268662036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb90109260 msgr2=0x7fdb901033c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:02.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.563+0000 7fdb96196640 1 --2- 172.21.15.114:0/4268662036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb90109260 0x7fdb901033c0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009900 tx=0x7fdb8402a410 comp rx=0 tx=0).stop 2024-05-27T08:00:02.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.563+0000 7fdb96196640 1 -- 172.21.15.114:0/4268662036 shutdown_connections 2024-05-27T08:00:02.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.563+0000 7fdb96196640 1 --2- 172.21.15.114:0/4268662036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb90103a30 0x7fdb9010f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.563+0000 7fdb96196640 1 --2- 172.21.15.114:0/4268662036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb90109260 0x7fdb901033c0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.563+0000 7fdb96196640 1 --2- 172.21.15.114:0/4268662036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb90108dc0 0x7fdb90102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.563+0000 7fdb96196640 1 -- 172.21.15.114:0/4268662036 >> 172.21.15.114:0/4268662036 conn(0x7fdb900fd9c0 msgr2=0x7fdb900ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:02.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.563+0000 7fdb96196640 1 -- 172.21.15.114:0/4268662036 shutdown_connections 2024-05-27T08:00:02.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.563+0000 7fdb96196640 1 -- 172.21.15.114:0/4268662036 wait complete. 2024-05-27T08:00:02.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.564+0000 7fdb96196640 1 Processor -- start 2024-05-27T08:00:02.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.564+0000 7fdb96196640 1 -- start start 2024-05-27T08:00:02.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.564+0000 7fdb96196640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb90103a30 0x7fdb9019f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:02.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.564+0000 7fdb96196640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb90108dc0 0x7fdb9019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:02.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.564+0000 7fdb96196640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb90109260 0x7fdb90199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:02.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.564+0000 7fdb96196640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90112fb0 con 0x7fdb90108dc0 2024-05-27T08:00:02.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.564+0000 7fdb96196640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90112e30 con 0x7fdb90109260 2024-05-27T08:00:02.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.564+0000 7fdb96196640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90113130 con 0x7fdb90103a30 2024-05-27T08:00:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.564+0000 7fdb8effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb90108dc0 0x7fdb9019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.564+0000 7fdb8effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb90108dc0 0x7fdb9019fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54510/0 (socket says 172.21.15.114:54510) 2024-05-27T08:00:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.564+0000 7fdb8effd640 1 -- 172.21.15.114:0/2166046388 learned_addr learned my addr 172.21.15.114:0/2166046388 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.564+0000 7fdb8f7fe640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb90103a30 0x7fdb9019f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.565+0000 7fdb8ffff640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb90109260 0x7fdb90199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.565+0000 7fdb8effd640 1 -- 172.21.15.114:0/2166046388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb90103a30 msgr2=0x7fdb9019f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.565+0000 7fdb8effd640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb90103a30 0x7fdb9019f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.565+0000 7fdb8effd640 1 -- 172.21.15.114:0/2166046388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb90109260 msgr2=0x7fdb90199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:02.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.565+0000 7fdb8effd640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb90109260 0x7fdb90199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.565+0000 7fdb8effd640 1 -- 172.21.15.114:0/2166046388 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb90199e60 con 0x7fdb90108dc0 2024-05-27T08:00:02.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.565+0000 7fdb8ffff640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb90109260 0x7fdb90199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T08:00:02.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.565+0000 7fdb8effd640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb90108dc0 0x7fdb9019fba0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fdb8402a930 tx=0x7fdb84004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:02.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.565+0000 7fdb8cff9640 1 -- 172.21.15.114:0/2166046388 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8403f030 con 0x7fdb90108dc0 2024-05-27T08:00:02.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.565+0000 7fdb8cff9640 1 -- 172.21.15.114:0/2166046388 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb84003aa0 con 0x7fdb90108dc0 2024-05-27T08:00:02.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.565+0000 7fdb96196640 1 -- 172.21.15.114:0/2166046388 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb9019a0f0 con 0x7fdb90108dc0 2024-05-27T08:00:02.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.565+0000 7fdb8cff9640 1 -- 172.21.15.114:0/2166046388 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb84035070 con 0x7fdb90108dc0 2024-05-27T08:00:02.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.565+0000 7fdb96196640 1 -- 172.21.15.114:0/2166046388 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb9019a5e0 con 0x7fdb90108dc0 2024-05-27T08:00:02.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.566+0000 7fdb96196640 1 -- 172.21.15.114:0/2166046388 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb90050860 con 0x7fdb90108dc0 2024-05-27T08:00:02.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.567+0000 7fdb8cff9640 1 -- 172.21.15.114:0/2166046388 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fdb84042030 con 0x7fdb90108dc0 2024-05-27T08:00:02.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.568+0000 7fdb8cff9640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdb780b2950 0x7fdb780b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:02.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.568+0000 7fdb8f7fe640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdb780b2950 0x7fdb780b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:02.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.568+0000 7fdb8cff9640 1 -- 172.21.15.114:0/2166046388 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7fdb84103500 con 0x7fdb90108dc0 2024-05-27T08:00:02.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.569+0000 7fdb8f7fe640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdb780b2950 0x7fdb780b4e60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdb9019a9b0 tx=0x7fdb8000b390 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:02.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.572+0000 7fdb8cff9640 1 -- 172.21.15.114:0/2166046388 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdb840cab30 con 0x7fdb90108dc0 2024-05-27T08:00:02.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.778+0000 7fdb96196640 1 -- 172.21.15.114:0/2166046388 --> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fdb9006b660 con 0x7fdb780b2950 2024-05-27T08:00:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.789+0000 7fdb8cff9640 1 -- 172.21.15.114:0/2166046388 <== mgr.15138 v2:172.21.15.114:6800/3311646739 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7fdb9006b660 con 0x7fdb780b2950 2024-05-27T08:00:02.791 INFO:teuthology.orchestra.run.smithi114.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-27T00:00:00", "created": "2024-05-27T07:39:54", "first": "2024-05-27T07:40:00", "last": "2024-05-27T08:00:00", "last_pruned": "2024-05-27T08:00:00", "created_count": 21, "pruned_count": 15, "active": true} 2024-05-27T08:00:02.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.791+0000 7fdb96196640 1 -- 172.21.15.114:0/2166046388 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdb780b2950 msgr2=0x7fdb780b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:02.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.791+0000 7fdb96196640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdb780b2950 0x7fdb780b4e60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdb9019a9b0 tx=0x7fdb8000b390 comp rx=0 tx=0).stop 2024-05-27T08:00:02.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.791+0000 7fdb96196640 1 -- 172.21.15.114:0/2166046388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb90108dc0 msgr2=0x7fdb9019fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:02.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.791+0000 7fdb96196640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb90108dc0 0x7fdb9019fba0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fdb8402a930 tx=0x7fdb84004270 comp rx=0 tx=0).stop 2024-05-27T08:00:02.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.791+0000 7fdb96196640 1 -- 172.21.15.114:0/2166046388 shutdown_connections 2024-05-27T08:00:02.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.791+0000 7fdb96196640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fdb780b2950 0x7fdb780b4e60 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.791+0000 7fdb96196640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb90109260 0x7fdb90199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.791+0000 7fdb96196640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb90108dc0 0x7fdb9019fba0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.791+0000 7fdb96196640 1 --2- 172.21.15.114:0/2166046388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb90103a30 0x7fdb9019f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.791+0000 7fdb96196640 1 -- 172.21.15.114:0/2166046388 >> 172.21.15.114:0/2166046388 conn(0x7fdb900fd9c0 msgr2=0x7fdb9010e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:02.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.792+0000 7fdb96196640 1 -- 172.21.15.114:0/2166046388 shutdown_connections 2024-05-27T08:00:02.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.792+0000 7fdb96196640 1 -- 172.21.15.114:0/2166046388 wait complete. 2024-05-27T08:00:02.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-27T08:00:02.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.915+0000 7f5ed7c29640 1 -- 172.21.15.114:0/1866266488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed0109d20 msgr2=0x7f5ed0101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:02.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.915+0000 7f5ed7c29640 1 --2- 172.21.15.114:0/1866266488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed0109d20 0x7f5ed0101c90 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc009900 tx=0x7f5ecc02a410 comp rx=0 tx=0).stop 2024-05-27T08:00:02.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.915+0000 7f5ed7c29640 1 -- 172.21.15.114:0/1866266488 shutdown_connections 2024-05-27T08:00:02.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.915+0000 7f5ed7c29640 1 --2- 172.21.15.114:0/1866266488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5ed010e6d0 0x7f5ed0110af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.915+0000 7f5ed7c29640 1 --2- 172.21.15.114:0/1866266488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ed01021d0 0x7f5ed01025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.915+0000 7f5ed7c29640 1 --2- 172.21.15.114:0/1866266488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed0109d20 0x7f5ed0101c90 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.915+0000 7f5ed7c29640 1 -- 172.21.15.114:0/1866266488 >> 172.21.15.114:0/1866266488 conn(0x7f5ed00fd9c0 msgr2=0x7f5ed00ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:02.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.916+0000 7f5ed7c29640 1 -- 172.21.15.114:0/1866266488 shutdown_connections 2024-05-27T08:00:02.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.916+0000 7f5ed7c29640 1 -- 172.21.15.114:0/1866266488 wait complete. 2024-05-27T08:00:02.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.916+0000 7f5ed7c29640 1 Processor -- start 2024-05-27T08:00:02.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.916+0000 7f5ed7c29640 1 -- start start 2024-05-27T08:00:02.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.916+0000 7f5ed7c29640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed01021d0 0x7f5ed0102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:02.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.916+0000 7f5ed7c29640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5ed0109d20 0x7f5ed01033b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:02.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.916+0000 7f5ed7c29640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ed010e6d0 0x7f5ed01038f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:02.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.916+0000 7f5ed7c29640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed0113e90 con 0x7f5ed01021d0 2024-05-27T08:00:02.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.916+0000 7f5ed7c29640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed0113d10 con 0x7f5ed0109d20 2024-05-27T08:00:02.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.916+0000 7f5ed7c29640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed0114010 con 0x7f5ed010e6d0 2024-05-27T08:00:02.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ed59c5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed01021d0 0x7f5ed0102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:02.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ed59c5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed01021d0 0x7f5ed0102e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54532/0 (socket says 172.21.15.114:54532) 2024-05-27T08:00:02.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ed59c5640 1 -- 172.21.15.114:0/485828775 learned_addr learned my addr 172.21.15.114:0/485828775 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:02.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ed61c6640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ed010e6d0 0x7f5ed01038f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:02.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ed51c4640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5ed0109d20 0x7f5ed01033b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:02.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ed59c5640 1 -- 172.21.15.114:0/485828775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ed010e6d0 msgr2=0x7f5ed01038f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:02.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ed59c5640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ed010e6d0 0x7f5ed01038f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ed59c5640 1 -- 172.21.15.114:0/485828775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5ed0109d20 msgr2=0x7f5ed01033b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:02.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ed59c5640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5ed0109d20 0x7f5ed01033b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:02.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ed59c5640 1 -- 172.21.15.114:0/485828775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed01a8bf0 con 0x7f5ed01021d0 2024-05-27T08:00:02.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ed59c5640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed01021d0 0x7f5ed0102e70 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc006010 tx=0x7f5ecc004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:02.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ec6ffd640 1 -- 172.21.15.114:0/485828775 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc03f030 con 0x7f5ed01021d0 2024-05-27T08:00:02.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ec6ffd640 1 -- 172.21.15.114:0/485828775 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ecc003aa0 con 0x7f5ed01021d0 2024-05-27T08:00:02.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ed7c29640 1 -- 172.21.15.114:0/485828775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed01a8e80 con 0x7f5ed01021d0 2024-05-27T08:00:02.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ec6ffd640 1 -- 172.21.15.114:0/485828775 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc035070 con 0x7f5ed01021d0 2024-05-27T08:00:02.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.917+0000 7f5ed7c29640 1 -- 172.21.15.114:0/485828775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ed01a92f0 con 0x7f5ed01021d0 2024-05-27T08:00:02.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.918+0000 7f5ed7c29640 1 -- 172.21.15.114:0/485828775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ed0050860 con 0x7f5ed01021d0 2024-05-27T08:00:02.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.920+0000 7f5ec6ffd640 1 -- 172.21.15.114:0/485828775 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f5ecc042030 con 0x7f5ed01021d0 2024-05-27T08:00:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.920+0000 7f5ec6ffd640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5ebc0b29d0 0x7f5ebc0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.920+0000 7f5ed51c4640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5ebc0b29d0 0x7f5ebc0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.921+0000 7f5ec6ffd640 1 -- 172.21.15.114:0/485828775 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 9466+0+0 (secure 0 0 0) 0x7f5ecc103280 con 0x7f5ed01021d0 2024-05-27T08:00:02.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.921+0000 7f5ed51c4640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5ebc0b29d0 0x7f5ebc0b4ee0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0005fd0 tx=0x7f5ec0005850 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:02.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:02.924+0000 7f5ec6ffd640 1 -- 172.21.15.114:0/485828775 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5ecc0ca8b0 con 0x7f5ed01021d0 2024-05-27T08:00:03.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:02 smithi114 ceph-mon[32038]: from='client.19923 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T08:00:03.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:02 smithi163 ceph-mon[36451]: from='client.19923 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T08:00:03.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:02 smithi186 ceph-mon[36647]: from='client.19923 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T08:00:03.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.313+0000 7f5ed7c29640 1 -- 172.21.15.114:0/485828775 --> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f5ed00fe4e0 con 0x7f5ebc0b29d0 2024-05-27T08:00:03.322 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T08:00:03.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T08:00:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.467+0000 7f5ec6ffd640 1 -- 172.21.15.114:0/485828775 <== mgr.15138 v2:172.21.15.114:6800/3311646739 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f5ed00fe4e0 con 0x7f5ebc0b29d0 2024-05-27T08:00:03.470 INFO:teuthology.orchestra.run.smithi114.stdout:/ 1m 6m3h 2024-05-27T08:00:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.473+0000 7f5ec4ff9640 1 -- 172.21.15.114:0/485828775 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5ebc0b29d0 msgr2=0x7f5ebc0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.473+0000 7f5ec4ff9640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5ebc0b29d0 0x7f5ebc0b4ee0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0005fd0 tx=0x7f5ec0005850 comp rx=0 tx=0).stop 2024-05-27T08:00:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.473+0000 7f5ec4ff9640 1 -- 172.21.15.114:0/485828775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed01021d0 msgr2=0x7f5ed0102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.473+0000 7f5ec4ff9640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed01021d0 0x7f5ed0102e70 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc006010 tx=0x7f5ecc004060 comp rx=0 tx=0).stop 2024-05-27T08:00:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.475+0000 7f5ec4ff9640 1 -- 172.21.15.114:0/485828775 shutdown_connections 2024-05-27T08:00:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.475+0000 7f5ec4ff9640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f5ebc0b29d0 0x7f5ebc0b4ee0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.475+0000 7f5ec4ff9640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ed010e6d0 0x7f5ed01038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.475+0000 7f5ec4ff9640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5ed0109d20 0x7f5ed01033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.475+0000 7f5ec4ff9640 1 --2- 172.21.15.114:0/485828775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed01021d0 0x7f5ed0102e70 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.475+0000 7f5ec4ff9640 1 -- 172.21.15.114:0/485828775 >> 172.21.15.114:0/485828775 conn(0x7f5ed00fd9c0 msgr2=0x7f5ed00fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.476+0000 7f5ec4ff9640 1 -- 172.21.15.114:0/485828775 shutdown_connections 2024-05-27T08:00:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:03.476+0000 7f5ec4ff9640 1 -- 172.21.15.114:0/485828775 wait complete. 2024-05-27T08:00:03.505 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-05-27T08:00:03.516 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-05-27T08:00:03.846 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:00:04.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:03 smithi163 ceph-mon[36451]: osdmap e165: 12 total, 12 up, 12 in 2024-05-27T08:00:04.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:03 smithi163 ceph-mon[36451]: from='client.19929 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T08:00:04.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:03 smithi163 ceph-mon[36451]: pgmap v702: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 403 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.9 MiB/s wr, 345 op/s 2024-05-27T08:00:04.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:03 smithi186 ceph-mon[36647]: osdmap e165: 12 total, 12 up, 12 in 2024-05-27T08:00:04.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:03 smithi186 ceph-mon[36647]: from='client.19929 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T08:00:04.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:03 smithi186 ceph-mon[36647]: pgmap v702: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 403 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.9 MiB/s wr, 345 op/s 2024-05-27T08:00:04.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:03 smithi114 ceph-mon[32038]: osdmap e165: 12 total, 12 up, 12 in 2024-05-27T08:00:04.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:03 smithi114 ceph-mon[32038]: from='client.19929 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T08:00:04.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:03 smithi114 ceph-mon[32038]: pgmap v702: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 403 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.9 MiB/s wr, 345 op/s 2024-05-27T08:00:05.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:04 smithi163 ceph-mon[36451]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T08:00:05.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:04 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:05.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:04 smithi186 ceph-mon[36647]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T08:00:05.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:04 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:05.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:04 smithi114 ceph-mon[32038]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T08:00:05.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:04 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:06.000 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:05 smithi114 ceph-mon[32038]: pgmap v703: 129 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 102 active+clean; 399 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 273 op/s 2024-05-27T08:00:06.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:05 smithi163 ceph-mon[36451]: pgmap v703: 129 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 102 active+clean; 399 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 273 op/s 2024-05-27T08:00:06.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:05 smithi186 ceph-mon[36647]: pgmap v703: 129 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 102 active+clean; 399 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 273 op/s 2024-05-27T08:00:06.315 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f85ef9b9700 time 2024-05-27T08:00:06.160744+0000 2024-05-27T08:00:06.315 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:06.315 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:06.315 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f86021feba6] 2024-05-27T08:00:06.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86021fedd1] 2024-05-27T08:00:06.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55aa765f423d] 2024-05-27T08:00:06.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55aa765df971] 2024-05-27T08:00:06.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 5: (SnapRealm::check_cache() const+0x13e) [0x55aa765ddfa0] 2024-05-27T08:00:06.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55aa765de651] 2024-05-27T08:00:06.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55aa76573347] 2024-05-27T08:00:06.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55aa7626f87e] 2024-05-27T08:00:06.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55aa76274e94] 2024-05-27T08:00:06.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55aa7627f969] 2024-05-27T08:00:06.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 11: (MDLog::_replay_thread()+0x1f38) [0x55aa7661848c] 2024-05-27T08:00:06.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55aa766219cb] 2024-05-27T08:00:06.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 13: (Thread::entry_wrapper()+0x43) [0x7f86021cd72d] 2024-05-27T08:00:06.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 14: (Thread::_entry_func(void*)+0xd) [0x7f86021cd749] 2024-05-27T08:00:06.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8600d711ca] 2024-05-27T08:00:06.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 16: clone() 2024-05-27T08:00:06.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.163+0000 7f85ef9b9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f85ef9b9700 time 2024-05-27T08:00:06.160744+0000 2024-05-27T08:00:06.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:06.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:06.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f86021feba6] 2024-05-27T08:00:06.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86021fedd1] 2024-05-27T08:00:06.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55aa765f423d] 2024-05-27T08:00:06.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55aa765df971] 2024-05-27T08:00:06.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 5: (SnapRealm::check_cache() const+0x13e) [0x55aa765ddfa0] 2024-05-27T08:00:06.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55aa765de651] 2024-05-27T08:00:06.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55aa76573347] 2024-05-27T08:00:06.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55aa7626f87e] 2024-05-27T08:00:06.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55aa76274e94] 2024-05-27T08:00:06.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55aa7627f969] 2024-05-27T08:00:06.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 11: (MDLog::_replay_thread()+0x1f38) [0x55aa7661848c] 2024-05-27T08:00:06.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55aa766219cb] 2024-05-27T08:00:06.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 13: (Thread::entry_wrapper()+0x43) [0x7f86021cd72d] 2024-05-27T08:00:06.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 14: (Thread::_entry_func(void*)+0xd) [0x7f86021cd749] 2024-05-27T08:00:06.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8600d711ca] 2024-05-27T08:00:06.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 16: clone() 2024-05-27T08:00:06.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: *** Caught signal (Aborted) ** 2024-05-27T08:00:06.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: in thread 7f85ef9b9700 thread_name:md_log_replay 2024-05-27T08:00:06.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:06.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8600d7bd20] 2024-05-27T08:00:06.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2: gsignal() 2024-05-27T08:00:06.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 3: abort() 2024-05-27T08:00:06.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f86021feced] 2024-05-27T08:00:06.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86021fedd1] 2024-05-27T08:00:06.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55aa765f423d] 2024-05-27T08:00:06.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55aa765df971] 2024-05-27T08:00:06.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 8: (SnapRealm::check_cache() const+0x13e) [0x55aa765ddfa0] 2024-05-27T08:00:06.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55aa765de651] 2024-05-27T08:00:06.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55aa76573347] 2024-05-27T08:00:06.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55aa7626f87e] 2024-05-27T08:00:06.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55aa76274e94] 2024-05-27T08:00:06.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55aa7627f969] 2024-05-27T08:00:06.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 14: (MDLog::_replay_thread()+0x1f38) [0x55aa7661848c] 2024-05-27T08:00:06.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55aa766219cb] 2024-05-27T08:00:06.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 16: (Thread::entry_wrapper()+0x43) [0x7f86021cd72d] 2024-05-27T08:00:06.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 17: (Thread::_entry_func(void*)+0xd) [0x7f86021cd749] 2024-05-27T08:00:06.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8600d711ca] 2024-05-27T08:00:06.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 19: clone() 2024-05-27T08:00:06.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.167+0000 7f85ef9b9700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:00:06.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: in thread 7f85ef9b9700 thread_name:md_log_replay 2024-05-27T08:00:06.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:06.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8600d7bd20] 2024-05-27T08:00:06.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2: gsignal() 2024-05-27T08:00:06.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 3: abort() 2024-05-27T08:00:06.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f86021feced] 2024-05-27T08:00:06.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86021fedd1] 2024-05-27T08:00:06.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55aa765f423d] 2024-05-27T08:00:06.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55aa765df971] 2024-05-27T08:00:06.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 8: (SnapRealm::check_cache() const+0x13e) [0x55aa765ddfa0] 2024-05-27T08:00:06.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55aa765de651] 2024-05-27T08:00:06.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55aa76573347] 2024-05-27T08:00:06.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55aa7626f87e] 2024-05-27T08:00:06.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55aa76274e94] 2024-05-27T08:00:06.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55aa7627f969] 2024-05-27T08:00:06.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 14: (MDLog::_replay_thread()+0x1f38) [0x55aa7661848c] 2024-05-27T08:00:06.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55aa766219cb] 2024-05-27T08:00:06.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 16: (Thread::entry_wrapper()+0x43) [0x7f86021cd72d] 2024-05-27T08:00:06.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 17: (Thread::_entry_func(void*)+0xd) [0x7f86021cd749] 2024-05-27T08:00:06.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8600d711ca] 2024-05-27T08:00:06.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 19: clone() 2024-05-27T08:00:06.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:00:06.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: -1> 2024-05-27T08:00:06.163+0000 7f85ef9b9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f85ef9b9700 time 2024-05-27T08:00:06.160744+0000 2024-05-27T08:00:06.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:06.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:06.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f86021feba6] 2024-05-27T08:00:06.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86021fedd1] 2024-05-27T08:00:06.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55aa765f423d] 2024-05-27T08:00:06.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55aa765df971] 2024-05-27T08:00:06.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 5: (SnapRealm::check_cache() const+0x13e) [0x55aa765ddfa0] 2024-05-27T08:00:06.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55aa765de651] 2024-05-27T08:00:06.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55aa76573347] 2024-05-27T08:00:06.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55aa7626f87e] 2024-05-27T08:00:06.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55aa76274e94] 2024-05-27T08:00:06.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55aa7627f969] 2024-05-27T08:00:06.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 11: (MDLog::_replay_thread()+0x1f38) [0x55aa7661848c] 2024-05-27T08:00:06.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55aa766219cb] 2024-05-27T08:00:06.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 13: (Thread::entry_wrapper()+0x43) [0x7f86021cd72d] 2024-05-27T08:00:06.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 14: (Thread::_entry_func(void*)+0xd) [0x7f86021cd749] 2024-05-27T08:00:06.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8600d711ca] 2024-05-27T08:00:06.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 16: clone() 2024-05-27T08:00:06.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 0> 2024-05-27T08:00:06.167+0000 7f85ef9b9700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:00:06.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: in thread 7f85ef9b9700 thread_name:md_log_replay 2024-05-27T08:00:06.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:06.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8600d7bd20] 2024-05-27T08:00:06.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2: gsignal() 2024-05-27T08:00:06.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 3: abort() 2024-05-27T08:00:06.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f86021feced] 2024-05-27T08:00:06.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86021fedd1] 2024-05-27T08:00:06.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55aa765f423d] 2024-05-27T08:00:06.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55aa765df971] 2024-05-27T08:00:06.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 8: (SnapRealm::check_cache() const+0x13e) [0x55aa765ddfa0] 2024-05-27T08:00:06.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55aa765de651] 2024-05-27T08:00:06.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55aa76573347] 2024-05-27T08:00:06.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55aa7626f87e] 2024-05-27T08:00:06.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55aa76274e94] 2024-05-27T08:00:06.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55aa7627f969] 2024-05-27T08:00:06.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 14: (MDLog::_replay_thread()+0x1f38) [0x55aa7661848c] 2024-05-27T08:00:06.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55aa766219cb] 2024-05-27T08:00:06.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 16: (Thread::entry_wrapper()+0x43) [0x7f86021cd72d] 2024-05-27T08:00:06.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 17: (Thread::_entry_func(void*)+0xd) [0x7f86021cd749] 2024-05-27T08:00:06.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8600d711ca] 2024-05-27T08:00:06.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 19: clone() 2024-05-27T08:00:06.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:00:06.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: -1> 2024-05-27T08:00:06.163+0000 7f85ef9b9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f85ef9b9700 time 2024-05-27T08:00:06.160744+0000 2024-05-27T08:00:06.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:06.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:06.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f86021feba6] 2024-05-27T08:00:06.332 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86021fedd1] 2024-05-27T08:00:06.332 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55aa765f423d] 2024-05-27T08:00:06.332 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55aa765df971] 2024-05-27T08:00:06.332 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 5: (SnapRealm::check_cache() const+0x13e) [0x55aa765ddfa0] 2024-05-27T08:00:06.332 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55aa765de651] 2024-05-27T08:00:06.332 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55aa76573347] 2024-05-27T08:00:06.332 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55aa7626f87e] 2024-05-27T08:00:06.332 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55aa76274e94] 2024-05-27T08:00:06.332 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55aa7627f969] 2024-05-27T08:00:06.332 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 11: (MDLog::_replay_thread()+0x1f38) [0x55aa7661848c] 2024-05-27T08:00:06.333 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55aa766219cb] 2024-05-27T08:00:06.333 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 13: (Thread::entry_wrapper()+0x43) [0x7f86021cd72d] 2024-05-27T08:00:06.333 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 14: (Thread::_entry_func(void*)+0xd) [0x7f86021cd749] 2024-05-27T08:00:06.333 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8600d711ca] 2024-05-27T08:00:06.333 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 16: clone() 2024-05-27T08:00:06.333 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.333 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 0> 2024-05-27T08:00:06.167+0000 7f85ef9b9700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:00:06.333 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: in thread 7f85ef9b9700 thread_name:md_log_replay 2024-05-27T08:00:06.334 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.334 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:06.334 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8600d7bd20] 2024-05-27T08:00:06.334 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2: gsignal() 2024-05-27T08:00:06.334 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 3: abort() 2024-05-27T08:00:06.334 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f86021feced] 2024-05-27T08:00:06.334 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86021fedd1] 2024-05-27T08:00:06.334 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55aa765f423d] 2024-05-27T08:00:06.335 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55aa765df971] 2024-05-27T08:00:06.335 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 8: (SnapRealm::check_cache() const+0x13e) [0x55aa765ddfa0] 2024-05-27T08:00:06.335 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55aa765de651] 2024-05-27T08:00:06.335 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55aa76573347] 2024-05-27T08:00:06.335 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55aa7626f87e] 2024-05-27T08:00:06.335 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55aa76274e94] 2024-05-27T08:00:06.335 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55aa7627f969] 2024-05-27T08:00:06.335 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 14: (MDLog::_replay_thread()+0x1f38) [0x55aa7661848c] 2024-05-27T08:00:06.336 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55aa766219cb] 2024-05-27T08:00:06.336 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 16: (Thread::entry_wrapper()+0x43) [0x7f86021cd72d] 2024-05-27T08:00:06.336 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 17: (Thread::_entry_func(void*)+0xd) [0x7f86021cd749] 2024-05-27T08:00:06.336 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8600d711ca] 2024-05-27T08:00:06.336 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 19: clone() 2024-05-27T08:00:06.336 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:00:06.336 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[70122]: 2024-05-27T08:00:06.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.532+0000 7fd70dba1700 1 -- 172.21.15.114:0/1360218263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd70811d920 msgr2=0x7fd70811fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:06.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.532+0000 7fd70dba1700 1 --2- 172.21.15.114:0/1360218263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd70811d920 0x7fd70811fd40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80098f0 tx=0x7fd6f8009c00 comp rx=0 tx=0).stop 2024-05-27T08:00:06.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.533+0000 7fd70dba1700 1 -- 172.21.15.114:0/1360218263 shutdown_connections 2024-05-27T08:00:06.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.533+0000 7fd70dba1700 1 --2- 172.21.15.114:0/1360218263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd70811d920 0x7fd70811fd40 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.533+0000 7fd70dba1700 1 --2- 172.21.15.114:0/1360218263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd70810a7b0 0x7fd70810abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.533+0000 7fd70dba1700 1 --2- 172.21.15.114:0/1360218263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd70810b100 0x7fd70811d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.533+0000 7fd70dba1700 1 -- 172.21.15.114:0/1360218263 >> 172.21.15.114:0/1360218263 conn(0x7fd70806f280 msgr2=0x7fd70806f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:06.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.541+0000 7fd70dba1700 1 -- 172.21.15.114:0/1360218263 shutdown_connections 2024-05-27T08:00:06.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.541+0000 7fd70dba1700 1 -- 172.21.15.114:0/1360218263 wait complete. 2024-05-27T08:00:06.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.542+0000 7fd70dba1700 1 Processor -- start 2024-05-27T08:00:06.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.542+0000 7fd70dba1700 1 -- start start 2024-05-27T08:00:06.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.543+0000 7fd70dba1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd70810a7b0 0x7fd70811d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.543+0000 7fd70dba1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd70810b100 0x7fd708119160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.543+0000 7fd70dba1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd70811d920 0x7fd7081196a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.543+0000 7fd70dba1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd708115390 con 0x7fd70810b100 2024-05-27T08:00:06.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.543+0000 7fd70dba1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd708115210 con 0x7fd70811d920 2024-05-27T08:00:06.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.543+0000 7fd70dba1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd708115510 con 0x7fd70810a7b0 2024-05-27T08:00:06.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.543+0000 7fd70d3a0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd70811d920 0x7fd7081196a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.543+0000 7fd707fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd70810b100 0x7fd708119160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.543+0000 7fd707fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd70810b100 0x7fd708119160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55866/0 (socket says 172.21.15.114:55866) 2024-05-27T08:00:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.543+0000 7fd707fff700 1 -- 172.21.15.114:0/990385746 learned_addr learned my addr 172.21.15.114:0/990385746 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.544+0000 7fd70d3a0700 1 -- 172.21.15.114:0/990385746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd70810a7b0 msgr2=0x7fd70811d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T08:00:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.544+0000 7fd70d3a0700 1 --2- 172.21.15.114:0/990385746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd70810a7b0 0x7fd70811d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.544+0000 7fd70d3a0700 1 -- 172.21.15.114:0/990385746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd70810b100 msgr2=0x7fd708119160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.544+0000 7fd70d3a0700 1 --2- 172.21.15.114:0/990385746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd70810b100 0x7fd708119160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.544+0000 7fd70d3a0700 1 -- 172.21.15.114:0/990385746 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd708119c90 con 0x7fd70811d920 2024-05-27T08:00:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.544+0000 7fd70d3a0700 1 --2- 172.21.15.114:0/990385746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd70811d920 0x7fd7081196a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80098f0 tx=0x7fd6f800b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:06.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.549+0000 7fd705ffb700 1 -- 172.21.15.114:0/990385746 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f800bac0 con 0x7fd70811d920 2024-05-27T08:00:06.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.549+0000 7fd705ffb700 1 -- 172.21.15.114:0/990385746 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6f80054e0 con 0x7fd70811d920 2024-05-27T08:00:06.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.549+0000 7fd705ffb700 1 -- 172.21.15.114:0/990385746 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f8017400 con 0x7fd70811d920 2024-05-27T08:00:06.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.550+0000 7fd70dba1700 1 -- 172.21.15.114:0/990385746 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd708119f20 con 0x7fd70811d920 2024-05-27T08:00:06.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.550+0000 7fd70dba1700 1 -- 172.21.15.114:0/990385746 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7081cf880 con 0x7fd70811d920 2024-05-27T08:00:06.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.569+0000 7fd71d818b80 1 Processor -- start 2024-05-27T08:00:06.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.569+0000 7fd71d818b80 1 -- start start 2024-05-27T08:00:06.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.570+0000 7fd71d818b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49ac7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.570+0000 7fd71d818b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ac7e40 0x560a49ac1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.570+0000 7fd71d818b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac2000 0x560a49ac2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.570+0000 7fd71d818b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560a499f0d10 con 0x560a49ac7e40 2024-05-27T08:00:06.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.570+0000 7fd71d818b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560a499e08b0 con 0x560a49ac2000 2024-05-27T08:00:06.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.570+0000 7fd71d818b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560a499e48f0 con 0x560a49943ec0 2024-05-27T08:00:06.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.570+0000 7fd705ffb700 1 -- 172.21.15.114:0/990385746 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fd6f80046d0 con 0x7fd70811d920 2024-05-27T08:00:06.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.572+0000 7fd705ffb700 1 --2- 172.21.15.114:0/990385746 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd6f40b2c00 0x7fd6f40b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.572+0000 7fd70d3a0700 1 --2- 172.21.15.114:0/990385746 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd6f40b2c00 0x7fd6f40b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.573+0000 7fd70d3a0700 1 --2- 172.21.15.114:0/990385746 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd6f40b2c00 0x7fd6f40b5110 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80a6d00 tx=0x7fd6f80ad040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:06.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.573+0000 7fd705ffb700 1 -- 172.21.15.114:0/990385746 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 9642+0+0 (secure 0 0 0) 0x7fd6f80e3d10 con 0x7fd70811d920 2024-05-27T08:00:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.581+0000 7fd707fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49ac7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.581+0000 7fd707fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49ac7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55388/0 (socket says 172.21.15.114:55388) 2024-05-27T08:00:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.581+0000 7fd707fff700 1 -- 172.21.15.114:0/34752623 learned_addr learned my addr 172.21.15.114:0/34752623 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.581+0000 7fd707fff700 1 --2- 172.21.15.114:0/34752623 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac2000 0x560a49ac2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.585+0000 7fd707fff700 1 -- 172.21.15.114:0/34752623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 msgr2=0x560a49ac7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:06.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.585+0000 7fd707fff700 1 --2- 172.21.15.114:0/34752623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49ac7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.585+0000 7fd707fff700 1 -- 172.21.15.114:0/34752623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ac7e40 msgr2=0x560a49ac1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:06.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.585+0000 7fd707fff700 1 --2- 172.21.15.114:0/34752623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ac7e40 0x560a49ac1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.585+0000 7fd707fff700 1 -- 172.21.15.114:0/34752623 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a49930950 con 0x560a49ac2000 2024-05-27T08:00:06.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.587+0000 7fd707fff700 1 --2- 172.21.15.114:0/34752623 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac2000 0x560a49ac2410 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd700007ad0 tx=0x7fd7000092b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a46947ea663e3d79 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:06.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.587+0000 7fd6feffd700 1 -- 172.21.15.114:0/34752623 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700004bb0 con 0x560a49ac2000 2024-05-27T08:00:06.590 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:06 smithi114 podman[135233]: 2024-05-27 08:00:06.532014075 +0000 UTC m=+0.269776620 container died b20fb979dcf785fe8c4694ba1c525a1f2a180548501c24dd0755f2b028cf205a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-27T08:00:06.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.587+0000 7fd6feffd700 1 -- 172.21.15.114:0/34752623 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd700004d50 con 0x560a49ac2000 2024-05-27T08:00:06.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.588+0000 7fd71d818b80 1 -- 172.21.15.114:0/34752623 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac2000 msgr2=0x560a49ac2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:06.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.588+0000 7fd71d818b80 1 --2- 172.21.15.114:0/34752623 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac2000 0x560a49ac2410 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd700007ad0 tx=0x7fd7000092b0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.596+0000 7fd71d818b80 1 -- 172.21.15.114:0/34752623 shutdown_connections 2024-05-27T08:00:06.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.596+0000 7fd71d818b80 1 --2- 172.21.15.114:0/34752623 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac2000 0x560a49ac2410 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.596+0000 7fd71d818b80 1 --2- 172.21.15.114:0/34752623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49ac7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.596+0000 7fd71d818b80 1 --2- 172.21.15.114:0/34752623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ac7e40 0x560a49ac1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.596+0000 7fd71d818b80 1 -- 172.21.15.114:0/34752623 >> 172.21.15.114:0/34752623 conn(0x560a4999eb70 msgr2=0x560a49abc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:06.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.599+0000 7fd71d818b80 1 -- 172.21.15.114:0/34752623 shutdown_connections 2024-05-27T08:00:06.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.601+0000 7fd71d818b80 1 -- 172.21.15.114:0/34752623 wait complete. 2024-05-27T08:00:06.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.601+0000 7fd71d818b80 1 -- start start 2024-05-27T08:00:06.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.602+0000 7fd71d818b80 1 Processor -- start 2024-05-27T08:00:06.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.603+0000 7fd71d818b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49acb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.603+0000 7fd71d818b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ac2000 0x560a49acba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.603+0000 7fd71d818b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 0x560a49acbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.603+0000 7fd71d818b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560a498e7020 con 0x560a49ac2000 2024-05-27T08:00:06.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.603+0000 7fd71d818b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560a49982840 con 0x560a49ac7e40 2024-05-27T08:00:06.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.603+0000 7fd71d818b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560a49ace900 con 0x560a49943ec0 2024-05-27T08:00:06.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.603+0000 7fd71d818b80 10 client.0 _ll_register_callbacks cb 0x560a4997b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T08:00:06.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.604+0000 7fd70cb9f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ac2000 0x560a49acba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.604+0000 7fd70cb9f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ac2000 0x560a49acba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55878/0 (socket says 172.21.15.114:55878) 2024-05-27T08:00:06.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.604+0000 7fd70cb9f700 1 -- 172.21.15.114:0/2884036827 learned_addr learned my addr 172.21.15.114:0/2884036827 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:06.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.605+0000 7fd70cb9f700 1 -- 172.21.15.114:0/2884036827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 msgr2=0x560a49acb4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T08:00:06.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.605+0000 7fd70cb9f700 1 --2- 172.21.15.114:0/2884036827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49acb4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.605+0000 7fd70cb9f700 1 -- 172.21.15.114:0/2884036827 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 msgr2=0x560a49acbf40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T08:00:06.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.605+0000 7fd70cb9f700 1 --2- 172.21.15.114:0/2884036827 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 0x560a49acbf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.605+0000 7fd70cb9f700 1 -- 172.21.15.114:0/2884036827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a4997f710 con 0x560a49ac2000 2024-05-27T08:00:06.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.605+0000 7fd70cb9f700 1 --2- 172.21.15.114:0/2884036827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ac2000 0x560a49acba00 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fd6f001c550 tx=0x7fd6f001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=468a3d99a17ca9d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:06.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.606+0000 7fd6e7fff700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T08:00:06.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.606+0000 7fd6e7fff700 1 -- 172.21.15.114:0/2884036827 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0013020 con 0x560a49ac2000 2024-05-27T08:00:06.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.606+0000 7fd71d818b80 1 -- 172.21.15.114:0/2884036827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd700004d50 con 0x560a49ac2000 2024-05-27T08:00:06.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.607+0000 7fd6e7fff700 1 -- 172.21.15.114:0/2884036827 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(165..165 src has 1..165) ==== 9642+0+0 (secure 0 0 0) 0x7fd6f000f890 con 0x560a49ac2000 2024-05-27T08:00:06.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.607+0000 7fd6e7fff700 1 -- 172.21.15.114:0/2884036827 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) ==== 24+0+0 (secure 0 0 0) 0x7fd6f000fa90 con 0x560a49ac2000 2024-05-27T08:00:06.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.607+0000 7fd71d818b80 10 client.19938 fetch_fsmap learned FSMap version 114 2024-05-27T08:00:06.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.607+0000 7fd71d818b80 1 -- 172.21.15.114:0/2884036827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=4+}) -- 0x560a49ad91d0 con 0x560a49ac2000 2024-05-27T08:00:06.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.607+0000 7fd6e7fff700 1 -- 172.21.15.114:0/2884036827 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd700004d50 con 0x560a49ac2000 2024-05-27T08:00:06.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.607+0000 7fd6e7fff700 1 -- 172.21.15.114:0/2884036827 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 114) ==== 2885+0+0 (secure 0 0 0) 0x7fd6f000d340 con 0x560a49ac2000 2024-05-27T08:00:06.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.607+0000 7fd71d818b80 10 client.19938 fetch_fsmap finished waiting for FSMap version 114 2024-05-27T08:00:06.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.607+0000 7fd71d818b80 10 client.19938 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T08:00:06.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.607+0000 7fd71d818b80 20 client.19938 populate_metadata read hostname 'smithi114' 2024-05-27T08:00:06.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.607+0000 7fd71d818b80 1 --2- 172.21.15.114:0/2884036827 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a49adbc50 0x560a49ade0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.607+0000 7fd71d818b80 4 client.19938 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T08:00:06.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.607+0000 7fd71d818b80 1 -- 172.21.15.114:0/2884036827 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a49a8da70 con 0x560a49adbc50 2024-05-27T08:00:06.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.611+0000 7fd707fff700 1 --2- 172.21.15.114:0/2884036827 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a49adbc50 0x560a49ade0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.617+0000 7fd707fff700 1 --2- 172.21.15.114:0/2884036827 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a49adbc50 0x560a49ade0c0 crc :-1 s=READY pgs=530 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=465b9e2fb86f56ff server_cookie=2ff759e5f08162fc in_seq=0 out_seq=0 2024-05-27T08:00:06.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.618+0000 7fd6e7fff700 10 client.19938 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:06.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.620+0000 7fd6e7fff700 1 -- 172.21.15.114:0/2884036827 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560a49a8da70 con 0x560a49adbc50 2024-05-27T08:00:06.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.620+0000 7fd6e7fff700 10 client.19938 handle_command_reply: tid=0 2024-05-27T08:00:06.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.621+0000 7fd71d818b80 1 client.19938 shutdown 2024-05-27T08:00:06.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.624+0000 7fd71d818b80 1 -- 172.21.15.114:0/2884036827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ac2000 msgr2=0x560a49acba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:06.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.624+0000 7fd71d818b80 1 --2- 172.21.15.114:0/2884036827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ac2000 0x560a49acba00 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fd6f001c550 tx=0x7fd6f001c910 comp rx=0 tx=0).stop 2024-05-27T08:00:06.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.624+0000 7fd71d818b80 1 -- 172.21.15.114:0/2884036827 shutdown_connections 2024-05-27T08:00:06.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.624+0000 7fd71d818b80 1 --2- 172.21.15.114:0/2884036827 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a49adbc50 0x560a49ade0c0 crc :-1 s=READY pgs=530 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.624+0000 7fd71d818b80 1 --2- 172.21.15.114:0/2884036827 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 0x560a49acbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.624+0000 7fd71d818b80 1 --2- 172.21.15.114:0/2884036827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49acb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.624+0000 7fd71d818b80 1 --2- 172.21.15.114:0/2884036827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ac2000 0x560a49acba00 secure :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fd6f001c550 tx=0x7fd6f001c910 comp rx=0 tx=0).stop 2024-05-27T08:00:06.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.624+0000 7fd6e7fff700 0 client.19938 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:06.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.624+0000 7fd71d818b80 1 -- 172.21.15.114:0/2884036827 >> 172.21.15.114:0/2884036827 conn(0x560a4999eb70 msgr2=0x560a49ac0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:06.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.625+0000 7fd71d818b80 1 -- 172.21.15.114:0/2884036827 shutdown_connections 2024-05-27T08:00:06.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.626+0000 7fd71d818b80 1 -- 172.21.15.114:0/2884036827 wait complete. 2024-05-27T08:00:06.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.626+0000 7fd71d818b80 20 client.19938 trim_cache size 0 max 16384 2024-05-27T08:00:06.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.666+0000 7fd71d818b80 1 Processor -- start 2024-05-27T08:00:06.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.667+0000 7fd71d818b80 1 -- start start 2024-05-27T08:00:06.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.667+0000 7fd71d818b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49acd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.667+0000 7fd71d818b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 0x560a49acbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.667+0000 7fd71d818b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ad7a40 0x560a49ab6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.667+0000 7fd707fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49acd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.667+0000 7fd707fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49acd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55396/0 (socket says 172.21.15.114:55396) 2024-05-27T08:00:06.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.667+0000 7fd707fff700 1 -- 172.21.15.114:0/1858120998 learned_addr learned my addr 172.21.15.114:0/1858120998 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:06.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.668+0000 7fd71d818b80 1 -- 172.21.15.114:0/1858120998 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd7081d7830 con 0x560a49ad7a40 2024-05-27T08:00:06.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.668+0000 7fd71d818b80 1 -- 172.21.15.114:0/1858120998 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd7081d7f00 con 0x560a49ac7e40 2024-05-27T08:00:06.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.668+0000 7fd70cb9f700 1 --2- 172.21.15.114:0/1858120998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 0x560a49acbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.668+0000 7fd71d818b80 1 -- 172.21.15.114:0/1858120998 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7081d68b0 con 0x560a49943ec0 2024-05-27T08:00:06.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.668+0000 7fd707fff700 1 --2- 172.21.15.114:0/1858120998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ad7a40 0x560a49ab6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.668+0000 7fd707fff700 1 -- 172.21.15.114:0/1858120998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 msgr2=0x560a49acbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:06.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.668+0000 7fd707fff700 1 --2- 172.21.15.114:0/1858120998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 0x560a49acbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.668+0000 7fd707fff700 1 -- 172.21.15.114:0/1858120998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ad7a40 msgr2=0x560a49ab6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:06.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.668+0000 7fd707fff700 1 --2- 172.21.15.114:0/1858120998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ad7a40 0x560a49ab6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.668+0000 7fd707fff700 1 -- 172.21.15.114:0/1858120998 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a49ab49f0 con 0x560a49943ec0 2024-05-27T08:00:06.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.669+0000 7fd707fff700 1 --2- 172.21.15.114:0/1858120998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49acd5a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd700002230 tx=0x7fd7000177a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ada8cd3393d5d3f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:06.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.673+0000 7fd6fe7fc700 1 -- 172.21.15.114:0/1858120998 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700004b40 con 0x560a49943ec0 2024-05-27T08:00:06.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.673+0000 7fd6fe7fc700 1 -- 172.21.15.114:0/1858120998 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd700004ce0 con 0x560a49943ec0 2024-05-27T08:00:06.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.673+0000 7fd6fe7fc700 1 -- 172.21.15.114:0/1858120998 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70001b570 con 0x560a49943ec0 2024-05-27T08:00:06.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.674+0000 7fd71d818b80 1 -- 172.21.15.114:0/1858120998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 msgr2=0x560a49acd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:06.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.674+0000 7fd71d818b80 1 --2- 172.21.15.114:0/1858120998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49acd5a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd700002230 tx=0x7fd7000177a0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.676+0000 7fd71d818b80 1 -- 172.21.15.114:0/1858120998 shutdown_connections 2024-05-27T08:00:06.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.676+0000 7fd71d818b80 1 --2- 172.21.15.114:0/1858120998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ad7a40 0x560a49ab6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.676+0000 7fd71d818b80 1 --2- 172.21.15.114:0/1858120998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49acd5a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.676+0000 7fd71d818b80 1 --2- 172.21.15.114:0/1858120998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 0x560a49acbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.676+0000 7fd71d818b80 1 -- 172.21.15.114:0/1858120998 >> 172.21.15.114:0/1858120998 conn(0x560a4999eb70 msgr2=0x560a49ac6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:06.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.682+0000 7fd71d818b80 1 -- 172.21.15.114:0/1858120998 shutdown_connections 2024-05-27T08:00:06.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.682+0000 7fd71d818b80 1 -- 172.21.15.114:0/1858120998 wait complete. 2024-05-27T08:00:06.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.683+0000 7fd71d818b80 1 -- start start 2024-05-27T08:00:06.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.683+0000 7fd71d818b80 1 Processor -- start 2024-05-27T08:00:06.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.683+0000 7fd71d818b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49ad4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.684+0000 7fd71d818b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 0x560a49ad0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.684+0000 7fd71d818b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ad7a40 0x560a49ad1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.684+0000 7fd71d818b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560a49acdae0 con 0x560a49ad7a40 2024-05-27T08:00:06.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.684+0000 7fd71d818b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560a49982840 con 0x560a49ac7e40 2024-05-27T08:00:06.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.684+0000 7fd71d818b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560a49ad1760 con 0x560a49943ec0 2024-05-27T08:00:06.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.684+0000 7fd71d818b80 10 client.0 _ll_register_callbacks cb 0x560a4997b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T08:00:06.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.685+0000 7fd70cb9f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 0x560a49ad0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.685+0000 7fd707fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49ad4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.685+0000 7fd707fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49ad4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55398/0 (socket says 172.21.15.114:55398) 2024-05-27T08:00:06.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.685+0000 7fd707fff700 1 -- 172.21.15.114:0/224048535 learned_addr learned my addr 172.21.15.114:0/224048535 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:06.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.685+0000 7fd707fff700 1 --2- 172.21.15.114:0/224048535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ad7a40 0x560a49ad1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.686+0000 7fd70cb9f700 1 -- 172.21.15.114:0/224048535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 msgr2=0x560a49ad4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:06.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.686+0000 7fd70cb9f700 1 --2- 172.21.15.114:0/224048535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49ad4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.686+0000 7fd70cb9f700 1 -- 172.21.15.114:0/224048535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ad7a40 msgr2=0x560a49ad1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:06.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.686+0000 7fd70cb9f700 1 --2- 172.21.15.114:0/224048535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ad7a40 0x560a49ad1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.686+0000 7fd70cb9f700 1 -- 172.21.15.114:0/224048535 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a49ad1960 con 0x560a49ac7e40 2024-05-27T08:00:06.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.687+0000 7fd70cb9f700 1 --2- 172.21.15.114:0/224048535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 0x560a49ad0c00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x560a49acca90 tx=0x7fd6f000df60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a54ca58c00e13537 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:06.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.689+0000 7fd6e7fff700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T08:00:06.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.689+0000 7fd6e7fff700 1 -- 172.21.15.114:0/224048535 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0013040 con 0x560a49ac7e40 2024-05-27T08:00:06.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.689+0000 7fd6e7fff700 1 -- 172.21.15.114:0/224048535 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(165..165 src has 1..165) ==== 9642+0+0 (secure 0 0 0) 0x7fd6f000fc30 con 0x560a49ac7e40 2024-05-27T08:00:06.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.693+0000 7fd71d818b80 1 -- 172.21.15.114:0/224048535 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd700004ce0 con 0x560a49ac7e40 2024-05-27T08:00:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.698+0000 7fd6e7fff700 1 -- 172.21.15.114:0/224048535 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) ==== 24+0+0 (secure 0 0 0) 0x7fd6f000d980 con 0x560a49ac7e40 2024-05-27T08:00:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.699+0000 7fd71d818b80 10 client.29249 fetch_fsmap learned FSMap version 114 2024-05-27T08:00:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.699+0000 7fd71d818b80 1 -- 172.21.15.114:0/224048535 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=4+}) -- 0x560a49b47790 con 0x560a49ac7e40 2024-05-27T08:00:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.702+0000 7fd6e7fff700 1 -- 172.21.15.114:0/224048535 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd700004ce0 con 0x560a49ac7e40 2024-05-27T08:00:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.702+0000 7fd6e7fff700 1 -- 172.21.15.114:0/224048535 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 114) ==== 2885+0+0 (secure 0 0 0) 0x7fd6f0005570 con 0x560a49ac7e40 2024-05-27T08:00:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.703+0000 7fd71d818b80 10 client.29249 fetch_fsmap finished waiting for FSMap version 114 2024-05-27T08:00:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.703+0000 7fd71d818b80 10 client.29249 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T08:00:06.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.703+0000 7fd71d818b80 20 client.29249 populate_metadata read hostname 'smithi114' 2024-05-27T08:00:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.703+0000 7fd71d818b80 1 --2- 172.21.15.114:0/224048535 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a49b47de0 0x560a49b4a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.703+0000 7fd71d818b80 4 client.29249 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T08:00:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.703+0000 7fd71d818b80 1 -- 172.21.15.114:0/224048535 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560a49ac5f50 con 0x560a49b47de0 2024-05-27T08:00:06.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.704+0000 7fd707fff700 1 --2- 172.21.15.114:0/224048535 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a49b47de0 0x560a49b4a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:06.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.708+0000 7fd707fff700 1 --2- 172.21.15.114:0/224048535 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a49b47de0 0x560a49b4a200 crc :-1 s=READY pgs=531 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d04a7e945466c02 server_cookie=6171c0a1e075ff61 in_seq=0 out_seq=0 2024-05-27T08:00:06.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.709+0000 7fd6e7fff700 10 client.29249 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.709+0000 7fd6e7fff700 1 -- 172.21.15.114:0/224048535 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560a49ac5f50 con 0x560a49b47de0 2024-05-27T08:00:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.710+0000 7fd6e7fff700 10 client.29249 handle_command_reply: tid=0 2024-05-27T08:00:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.710+0000 7fd71d818b80 1 client.29249 shutdown 2024-05-27T08:00:06.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.711+0000 7fd71d818b80 1 -- 172.21.15.114:0/224048535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 msgr2=0x560a49ad0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:06.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.711+0000 7fd71d818b80 1 --2- 172.21.15.114:0/224048535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 0x560a49ad0c00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x560a49acca90 tx=0x7fd6f000df60 comp rx=0 tx=0).stop 2024-05-27T08:00:06.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.717+0000 7fd71d818b80 1 -- 172.21.15.114:0/224048535 shutdown_connections 2024-05-27T08:00:06.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.717+0000 7fd71d818b80 1 --2- 172.21.15.114:0/224048535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a49ad7a40 0x560a49ad1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.717+0000 7fd71d818b80 1 --2- 172.21.15.114:0/224048535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a49943ec0 0x560a49ad4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.717+0000 7fd71d818b80 1 --2- 172.21.15.114:0/224048535 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560a49b47de0 0x560a49b4a200 crc :-1 s=READY pgs=531 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.717+0000 7fd71d818b80 1 --2- 172.21.15.114:0/224048535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560a49ac7e40 0x560a49ad0c00 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.717+0000 7fd71d818b80 1 -- 172.21.15.114:0/224048535 >> 172.21.15.114:0/224048535 conn(0x560a4999eb70 msgr2=0x560a49accac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:06.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.717+0000 7fd6e7fff700 0 client.29249 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:06.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.721+0000 7fd71d818b80 1 -- 172.21.15.114:0/224048535 shutdown_connections 2024-05-27T08:00:06.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.724+0000 7fd71d818b80 1 -- 172.21.15.114:0/224048535 wait complete. 2024-05-27T08:00:06.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.724+0000 7fd71d818b80 20 client.29249 trim_cache size 0 max 16384 2024-05-27T08:00:06.727 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T08:00:06.727 INFO:teuthology.orchestra.run.smithi114.stdout: "return_code": 0, 2024-05-27T08:00:06.727 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_tag": "ceb8123e-d526-4710-b91b-b4ed0f98b5e0", 2024-05-27T08:00:06.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": "asynchronous" 2024-05-27T08:00:06.727 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T08:00:06.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.729+0000 7fd70dba1700 1 -- 172.21.15.114:0/990385746 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd6f40b2c00 msgr2=0x7fd6f40b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:06.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.729+0000 7fd70dba1700 1 --2- 172.21.15.114:0/990385746 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd6f40b2c00 0x7fd6f40b5110 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80a6d00 tx=0x7fd6f80ad040 comp rx=0 tx=0).stop 2024-05-27T08:00:06.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.729+0000 7fd70dba1700 1 -- 172.21.15.114:0/990385746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd70811d920 msgr2=0x7fd7081196a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:06.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.729+0000 7fd70dba1700 1 --2- 172.21.15.114:0/990385746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd70811d920 0x7fd7081196a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80098f0 tx=0x7fd6f800b790 comp rx=0 tx=0).stop 2024-05-27T08:00:06.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.729+0000 7fd70dba1700 1 -- 172.21.15.114:0/990385746 shutdown_connections 2024-05-27T08:00:06.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.729+0000 7fd70dba1700 1 --2- 172.21.15.114:0/990385746 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fd6f40b2c00 0x7fd6f40b5110 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.729+0000 7fd70dba1700 1 --2- 172.21.15.114:0/990385746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd70811d920 0x7fd7081196a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.729+0000 7fd70dba1700 1 --2- 172.21.15.114:0/990385746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd70810a7b0 0x7fd70811d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.729+0000 7fd70dba1700 1 --2- 172.21.15.114:0/990385746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd70810b100 0x7fd708119160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:06.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.729+0000 7fd70dba1700 1 -- 172.21.15.114:0/990385746 >> 172.21.15.114:0/990385746 conn(0x7fd70806f280 msgr2=0x7fd70810f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.731+0000 7fd70dba1700 1 -- 172.21.15.114:0/990385746 shutdown_connections 2024-05-27T08:00:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:06.731+0000 7fd70dba1700 1 -- 172.21.15.114:0/990385746 wait complete. 2024-05-27T08:00:07.593 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:07 smithi114 podman[135233]: 2024-05-27 08:00:07.477311936 +0000 UTC m=+1.215074457 container remove b20fb979dcf785fe8c4694ba1c525a1f2a180548501c24dd0755f2b028cf205a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T08:00:07.593 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:07 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T08:00:07.941 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:07 smithi114 ceph-mon[32038]: scrub queued for path: / 2024-05-27T08:00:07.941 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:07 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T08:00:07.941 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:07 smithi114 ceph-mon[32038]: scrub summary: active paths [/] 2024-05-27T08:00:07.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:07 smithi163 ceph-mon[36451]: scrub queued for path: / 2024-05-27T08:00:07.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:07 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T08:00:07.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:07 smithi163 ceph-mon[36451]: scrub summary: active paths [/] 2024-05-27T08:00:07.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:07 smithi186 ceph-mon[36647]: scrub queued for path: / 2024-05-27T08:00:07.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:07 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T08:00:07.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:07 smithi186 ceph-mon[36647]: scrub summary: active paths [/] 2024-05-27T08:00:07.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T08:00:08.567 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:00:08.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:08 smithi114 ceph-mon[32038]: pgmap v704: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 399 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.1 MiB/s wr, 221 op/s 2024-05-27T08:00:08.808 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:08 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T08:00:08.808 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:08 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Consumed 2.630s CPU time. 2024-05-27T08:00:08.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:08 smithi163 ceph-mon[36451]: pgmap v704: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 399 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.1 MiB/s wr, 221 op/s 2024-05-27T08:00:08.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:08 smithi186 ceph-mon[36647]: pgmap v704: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 399 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.1 MiB/s wr, 221 op/s 2024-05-27T08:00:09.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:09 smithi163 ceph-mon[36451]: scrub summary: idle+waiting paths [/] 2024-05-27T08:00:09.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:09 smithi186 ceph-mon[36647]: scrub summary: idle+waiting paths [/] 2024-05-27T08:00:10.034 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:09 smithi114 ceph-mon[32038]: scrub summary: idle+waiting paths [/] 2024-05-27T08:00:10.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.801+0000 7f7c99ccc700 1 -- 172.21.15.114:0/2433950536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c9410a920 msgr2=0x7f7c9410ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:10.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.801+0000 7f7c99ccc700 1 --2- 172.21.15.114:0/2433950536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c9410a920 0x7f7c9410ad30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009990 tx=0x7f7c84009ca0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.803+0000 7f7c99ccc700 1 -- 172.21.15.114:0/2433950536 shutdown_connections 2024-05-27T08:00:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.803+0000 7f7c99ccc700 1 --2- 172.21.15.114:0/2433950536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c9411d920 0x7f7c9411fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.803+0000 7f7c99ccc700 1 --2- 172.21.15.114:0/2433950536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c9410a920 0x7f7c9410ad30 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.803+0000 7f7c99ccc700 1 --2- 172.21.15.114:0/2433950536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c9410b270 0x7f7c9411d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.803+0000 7f7c99ccc700 1 -- 172.21.15.114:0/2433950536 >> 172.21.15.114:0/2433950536 conn(0x7f7c9406f4c0 msgr2=0x7f7c9406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:10.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.807+0000 7f7c99ccc700 1 -- 172.21.15.114:0/2433950536 shutdown_connections 2024-05-27T08:00:10.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.812+0000 7f7c99ccc700 1 -- 172.21.15.114:0/2433950536 wait complete. 2024-05-27T08:00:10.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.812+0000 7f7c99ccc700 1 Processor -- start 2024-05-27T08:00:10.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.813+0000 7f7c99ccc700 1 -- start start 2024-05-27T08:00:10.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.813+0000 7f7c99ccc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c9410a920 0x7f7c941a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.813+0000 7f7c99ccc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c9410b270 0x7f7c941aa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.813+0000 7f7c99ccc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c9411d920 0x7f7c941ae560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.813+0000 7f7c99ccc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7c941233c0 con 0x7f7c9410b270 2024-05-27T08:00:10.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.813+0000 7f7c99ccc700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7c94123240 con 0x7f7c9411d920 2024-05-27T08:00:10.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.813+0000 7f7c99ccc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c94123540 con 0x7f7c9410a920 2024-05-27T08:00:10.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.815+0000 7f7c92ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c9410b270 0x7f7c941aa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.815+0000 7f7c92ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c9410b270 0x7f7c941aa2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55906/0 (socket says 172.21.15.114:55906) 2024-05-27T08:00:10.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.815+0000 7f7c92ffd700 1 -- 172.21.15.114:0/3144531997 learned_addr learned my addr 172.21.15.114:0/3144531997 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:10.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.816+0000 7f7c937fe700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c9410a920 0x7f7c941a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.816+0000 7f7c93fff700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c9411d920 0x7f7c941ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.818+0000 7f7c93fff700 1 -- 172.21.15.114:0/3144531997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c9410a920 msgr2=0x7f7c941a9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:10.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.818+0000 7f7c93fff700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c9410a920 0x7f7c941a9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.818+0000 7f7c93fff700 1 -- 172.21.15.114:0/3144531997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c9410b270 msgr2=0x7f7c941aa2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:10.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.818+0000 7f7c93fff700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c9410b270 0x7f7c941aa2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.818+0000 7f7c93fff700 1 -- 172.21.15.114:0/3144531997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c941aec00 con 0x7f7c9411d920 2024-05-27T08:00:10.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.823+0000 7f7c93fff700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c9411d920 0x7f7c941ae560 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c007fd0 tx=0x7f7c8c00aed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:10.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.827+0000 7f7c90ff9700 1 -- 172.21.15.114:0/3144531997 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c017020 con 0x7f7c9411d920 2024-05-27T08:00:10.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.827+0000 7f7c90ff9700 1 -- 172.21.15.114:0/3144531997 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c8c004bb0 con 0x7f7c9411d920 2024-05-27T08:00:10.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.827+0000 7f7c90ff9700 1 -- 172.21.15.114:0/3144531997 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c017020 con 0x7f7c9411d920 2024-05-27T08:00:10.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.832+0000 7f7c99ccc700 1 -- 172.21.15.114:0/3144531997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c941aeef0 con 0x7f7c9411d920 2024-05-27T08:00:10.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.832+0000 7f7c99ccc700 1 -- 172.21.15.114:0/3144531997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c94112bc0 con 0x7f7c9411d920 2024-05-27T08:00:10.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.840+0000 7f7c90ff9700 1 -- 172.21.15.114:0/3144531997 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f7c8c01a070 con 0x7f7c9411d920 2024-05-27T08:00:10.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.841+0000 7f7c90ff9700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7c7c0b2e40 0x7f7c7c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.841+0000 7f7c90ff9700 1 -- 172.21.15.114:0/3144531997 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 9642+0+0 (secure 0 0 0) 0x7f7c8c010050 con 0x7f7c9411d920 2024-05-27T08:00:10.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.842+0000 7f7ca86e1b80 1 Processor -- start 2024-05-27T08:00:10.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.845+0000 7f7c937fe700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7c7c0b2e40 0x7f7c7c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.846+0000 7f7ca86e1b80 1 -- start start 2024-05-27T08:00:10.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.846+0000 7f7ca86e1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf8963b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.846+0000 7f7ca86e1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf8964070 0x560bf895dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.846+0000 7f7ca86e1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 0x560bf895e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.846+0000 7f7ca86e1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560bf888bd70 con 0x560bf8964070 2024-05-27T08:00:10.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.846+0000 7f7ca86e1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560bf887b910 con 0x560bf87dee40 2024-05-27T08:00:10.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.846+0000 7f7ca86e1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560bf887f950 con 0x560bf895e120 2024-05-27T08:00:10.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:10 smithi114 ceph-mon[32038]: pgmap v705: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 395 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 196 op/s 2024-05-27T08:00:10.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.848+0000 7f7c937fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf8963b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.848+0000 7f7c937fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf8963b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51498/0 (socket says 172.21.15.114:51498) 2024-05-27T08:00:10.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.848+0000 7f7c937fe700 1 -- 172.21.15.114:0/2062749579 learned_addr learned my addr 172.21.15.114:0/2062749579 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:10.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.848+0000 7f7c93fff700 1 --2- 172.21.15.114:0/2062749579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 0x560bf895e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.849+0000 7f7c92ffd700 1 --2- 172.21.15.114:0/2062749579 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf8964070 0x560bf895dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.851+0000 7f7c937fe700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7c7c0b2e40 0x7f7c7c0b5350 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009750 tx=0x7f7c8400b8f0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:10.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.851+0000 7f7c937fe700 1 -- 172.21.15.114:0/2062749579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 msgr2=0x560bf895e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:10.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.851+0000 7f7c937fe700 1 --2- 172.21.15.114:0/2062749579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 0x560bf895e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.851+0000 7f7c937fe700 1 -- 172.21.15.114:0/2062749579 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf8964070 msgr2=0x560bf895dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:10.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.851+0000 7f7c937fe700 1 --2- 172.21.15.114:0/2062749579 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf8964070 0x560bf895dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.851+0000 7f7c937fe700 1 -- 172.21.15.114:0/2062749579 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bf8824fb0 con 0x560bf87dee40 2024-05-27T08:00:10.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.857+0000 7f7c937fe700 1 --2- 172.21.15.114:0/2062749579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf8963b30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009990 tx=0x7f7c84027000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d167bf529ee7eca2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:10.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.860+0000 7f7c79ffb700 1 -- 172.21.15.114:0/2062749579 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8400f840 con 0x560bf87dee40 2024-05-27T08:00:10.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.860+0000 7f7c79ffb700 1 -- 172.21.15.114:0/2062749579 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c84027430 con 0x560bf87dee40 2024-05-27T08:00:10.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.860+0000 7f7c79ffb700 1 -- 172.21.15.114:0/2062749579 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8402f660 con 0x560bf87dee40 2024-05-27T08:00:10.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.862+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2062749579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 msgr2=0x560bf8963b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:10.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.862+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/2062749579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf8963b30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009990 tx=0x7f7c84027000 comp rx=0 tx=0).stop 2024-05-27T08:00:10.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.866+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2062749579 shutdown_connections 2024-05-27T08:00:10.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.866+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/2062749579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 0x560bf895e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.866+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/2062749579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf8963b30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.866+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/2062749579 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf8964070 0x560bf895dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.866+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2062749579 >> 172.21.15.114:0/2062749579 conn(0x560bf87e1770 msgr2=0x560bf8958d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:10.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.868+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2062749579 shutdown_connections 2024-05-27T08:00:10.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.871+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2062749579 wait complete. 2024-05-27T08:00:10.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.872+0000 7f7ca86e1b80 1 -- start start 2024-05-27T08:00:10.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.872+0000 7f7ca86e1b80 1 Processor -- start 2024-05-27T08:00:10.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.876+0000 7f7ca86e1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf87dee40 0x560bf8967930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.876+0000 7f7ca86e1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 0x560bf8967e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.876+0000 7f7ca86e1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf8964070 0x560bf89683b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.876+0000 7f7ca86e1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560bf8781fa0 con 0x560bf87dee40 2024-05-27T08:00:10.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.876+0000 7f7ca86e1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560bf87e3da0 con 0x560bf8964070 2024-05-27T08:00:10.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.876+0000 7f7ca86e1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560bf896ab00 con 0x560bf895e120 2024-05-27T08:00:10.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.876+0000 7f7ca86e1b80 10 client.0 _ll_register_callbacks cb 0x560bf88125b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T08:00:10.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.877+0000 7f7c92ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 0x560bf8967e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.882+0000 7f7c93fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf8964070 0x560bf89683b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.882+0000 7f7c93fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf8964070 0x560bf89683b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51502/0 (socket says 172.21.15.114:51502) 2024-05-27T08:00:10.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.882+0000 7f7c93fff700 1 -- 172.21.15.114:0/1962551630 learned_addr learned my addr 172.21.15.114:0/1962551630 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:10.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.883+0000 7f7c937fe700 1 --2- 172.21.15.114:0/1962551630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf87dee40 0x560bf8967930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.882+0000 7f7c92ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 0x560bf8967e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:55428/0 (socket says 172.21.15.114:55428) 2024-05-27T08:00:10.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.884+0000 7f7c937fe700 1 -- 172.21.15.114:0/1962551630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 msgr2=0x560bf8967e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:10.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.884+0000 7f7c937fe700 1 --2- 172.21.15.114:0/1962551630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 0x560bf8967e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.884+0000 7f7c937fe700 1 -- 172.21.15.114:0/1962551630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf8964070 msgr2=0x560bf89683b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:10.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.884+0000 7f7c937fe700 1 --2- 172.21.15.114:0/1962551630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf8964070 0x560bf89683b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.884+0000 7f7c937fe700 1 -- 172.21.15.114:0/1962551630 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bf87cb8d0 con 0x560bf87dee40 2024-05-27T08:00:10.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.897+0000 7f7c937fe700 1 --2- 172.21.15.114:0/1962551630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf87dee40 0x560bf8967930 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f7c840098e0 tx=0x7f7c84027810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de4e7fce6e33f1a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:10.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.897+0000 7f7c72ffd700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T08:00:10.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.901+0000 7f7c72ffd700 1 -- 172.21.15.114:0/1962551630 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84027e20 con 0x560bf87dee40 2024-05-27T08:00:10.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.901+0000 7f7c72ffd700 1 -- 172.21.15.114:0/1962551630 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(165..165 src has 1..165) ==== 9642+0+0 (secure 0 0 0) 0x7f7c84021040 con 0x560bf87dee40 2024-05-27T08:00:10.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.901+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/1962551630 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c84027430 con 0x560bf87dee40 2024-05-27T08:00:10.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.904+0000 7f7c72ffd700 1 -- 172.21.15.114:0/1962551630 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) ==== 24+0+0 (secure 0 0 0) 0x7f7c8400f840 con 0x560bf87dee40 2024-05-27T08:00:10.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.905+0000 7f7ca86e1b80 10 client.19953 fetch_fsmap learned FSMap version 114 2024-05-27T08:00:10.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.905+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/1962551630 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=4+}) -- 0x560bf8835760 con 0x560bf87dee40 2024-05-27T08:00:10.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.905+0000 7f7c72ffd700 1 -- 172.21.15.114:0/1962551630 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c8402f400 con 0x560bf87dee40 2024-05-27T08:00:10.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.905+0000 7f7c72ffd700 1 -- 172.21.15.114:0/1962551630 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 114) ==== 2885+0+0 (secure 0 0 0) 0x7f7c8403a3e0 con 0x560bf87dee40 2024-05-27T08:00:10.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.905+0000 7f7ca86e1b80 10 client.19953 fetch_fsmap finished waiting for FSMap version 114 2024-05-27T08:00:10.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.905+0000 7f7ca86e1b80 10 client.19953 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T08:00:10.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.905+0000 7f7ca86e1b80 20 client.19953 populate_metadata read hostname 'smithi114' 2024-05-27T08:00:10.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.905+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/1962551630 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560bf8977aa0 0x560bf8979f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.905+0000 7f7ca86e1b80 4 client.19953 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T08:00:10.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.905+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/1962551630 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7c84027430 con 0x560bf8977aa0 2024-05-27T08:00:10.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.911+0000 7f7c92ffd700 1 --2- 172.21.15.114:0/1962551630 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560bf8977aa0 0x560bf8979f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.915+0000 7f7c72ffd700 10 client.19953 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:10.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.915+0000 7f7c92ffd700 1 --2- 172.21.15.114:0/1962551630 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560bf8977aa0 0x560bf8979f30 crc :-1 s=READY pgs=532 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4e38c795aace8 server_cookie=45b79fb49dced5c9 in_seq=0 out_seq=0 2024-05-27T08:00:10.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.917+0000 7f7c72ffd700 1 -- 172.21.15.114:0/1962551630 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f7c84027430 con 0x560bf8977aa0 2024-05-27T08:00:10.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.917+0000 7f7c72ffd700 10 client.19953 handle_command_reply: tid=0 2024-05-27T08:00:10.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.917+0000 7f7ca86e1b80 1 client.19953 shutdown 2024-05-27T08:00:10.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.926+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/1962551630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf87dee40 msgr2=0x560bf8967930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:10.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.926+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/1962551630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf87dee40 0x560bf8967930 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f7c840098e0 tx=0x7f7c84027810 comp rx=0 tx=0).stop 2024-05-27T08:00:10.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.926+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/1962551630 shutdown_connections 2024-05-27T08:00:10.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.926+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/1962551630 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560bf8977aa0 0x560bf8979f30 crc :-1 s=READY pgs=532 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.926+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/1962551630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf8964070 0x560bf89683b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.926+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/1962551630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf87dee40 0x560bf8967930 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.926+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/1962551630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 0x560bf8967e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.926+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/1962551630 >> 172.21.15.114:0/1962551630 conn(0x560bf87e1770 msgr2=0x560bf895cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:10.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.926+0000 7f7c72ffd700 0 client.19953 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:10.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.927+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/1962551630 shutdown_connections 2024-05-27T08:00:10.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.927+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/1962551630 wait complete. 2024-05-27T08:00:10.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.927+0000 7f7ca86e1b80 20 client.19953 trim_cache size 0 max 16384 2024-05-27T08:00:10.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.967+0000 7f7ca86e1b80 1 Processor -- start 2024-05-27T08:00:10.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.967+0000 7f7ca86e1b80 1 -- start start 2024-05-27T08:00:10.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.967+0000 7f7ca86e1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf895dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.967+0000 7f7ca86e1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 0x560bf8968320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.967+0000 7f7ca86e1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf8964070 0x560bf8973ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.967+0000 7f7ca86e1b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9411a8d0 con 0x560bf8964070 2024-05-27T08:00:10.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.967+0000 7f7ca86e1b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9411afa0 con 0x560bf87dee40 2024-05-27T08:00:10.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.967+0000 7f7ca86e1b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c94119950 con 0x560bf895e120 2024-05-27T08:00:10.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.970+0000 7f7c937fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf895dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.970+0000 7f7c92ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 0x560bf8968320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.970+0000 7f7c937fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf895dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51506/0 (socket says 172.21.15.114:51506) 2024-05-27T08:00:10.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.970+0000 7f7c937fe700 1 -- 172.21.15.114:0/283603408 learned_addr learned my addr 172.21.15.114:0/283603408 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:10.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.971+0000 7f7c93fff700 1 --2- 172.21.15.114:0/283603408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf8964070 0x560bf8973ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:10 smithi163 ceph-mon[36451]: pgmap v705: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 395 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 196 op/s 2024-05-27T08:00:10.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.976+0000 7f7c937fe700 1 -- 172.21.15.114:0/283603408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 msgr2=0x560bf8968320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:10.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.976+0000 7f7c937fe700 1 --2- 172.21.15.114:0/283603408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 0x560bf8968320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.976+0000 7f7c937fe700 1 -- 172.21.15.114:0/283603408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf8964070 msgr2=0x560bf8973ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:10.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.976+0000 7f7c937fe700 1 --2- 172.21.15.114:0/283603408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf8964070 0x560bf8973ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.976+0000 7f7c937fe700 1 -- 172.21.15.114:0/283603408 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bf8965ec0 con 0x560bf87dee40 2024-05-27T08:00:10.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:10 smithi186 ceph-mon[36647]: pgmap v705: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 395 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 196 op/s 2024-05-27T08:00:10.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.979+0000 7f7c937fe700 1 --2- 172.21.15.114:0/283603408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf895dbe0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7c8400bab0 tx=0x7f7c84027c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c271c7e66e31d26c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:10.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.981+0000 7f7c797fa700 1 -- 172.21.15.114:0/283603408 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84027430 con 0x560bf87dee40 2024-05-27T08:00:10.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.981+0000 7f7c797fa700 1 -- 172.21.15.114:0/283603408 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c84005940 con 0x560bf87dee40 2024-05-27T08:00:10.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.981+0000 7f7c797fa700 1 -- 172.21.15.114:0/283603408 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8403b430 con 0x560bf87dee40 2024-05-27T08:00:10.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.983+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/283603408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 msgr2=0x560bf895dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:10.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.983+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/283603408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf895dbe0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7c8400bab0 tx=0x7f7c84027c40 comp rx=0 tx=0).stop 2024-05-27T08:00:10.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.985+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/283603408 shutdown_connections 2024-05-27T08:00:10.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.985+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/283603408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf8964070 0x560bf8973ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.985+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/283603408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf895dbe0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.985+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/283603408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf895e120 0x560bf8968320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.985+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/283603408 >> 172.21.15.114:0/283603408 conn(0x560bf87e1770 msgr2=0x560bf8966240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:10.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.986+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/283603408 shutdown_connections 2024-05-27T08:00:10.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.986+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/283603408 wait complete. 2024-05-27T08:00:10.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.987+0000 7f7ca86e1b80 1 -- start start 2024-05-27T08:00:10.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.987+0000 7f7ca86e1b80 1 Processor -- start 2024-05-27T08:00:10.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.987+0000 7f7ca86e1b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf89705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.987+0000 7f7ca86e1b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf895e120 0x560bf897f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.988+0000 7f7c937fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf89705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.988+0000 7f7ca86e1b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf8964070 0x560bf897f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:10.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.988+0000 7f7c937fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf89705a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:51508/0 (socket says 172.21.15.114:51508) 2024-05-27T08:00:10.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.988+0000 7f7c937fe700 1 -- 172.21.15.114:0/2074982078 learned_addr learned my addr 172.21.15.114:0/2074982078 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:10.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.988+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2074982078 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560bf8966950 con 0x560bf895e120 2024-05-27T08:00:10.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.988+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2074982078 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560bf888bd70 con 0x560bf87dee40 2024-05-27T08:00:10.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.988+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2074982078 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560bf8963270 con 0x560bf8964070 2024-05-27T08:00:10.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.988+0000 7f7ca86e1b80 10 client.0 _ll_register_callbacks cb 0x560bf88125b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T08:00:10.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.989+0000 7f7c93fff700 1 --2- 172.21.15.114:0/2074982078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf8964070 0x560bf897f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:10.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.990+0000 7f7c937fe700 1 -- 172.21.15.114:0/2074982078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf8964070 msgr2=0x560bf897f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:10.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.990+0000 7f7c937fe700 1 --2- 172.21.15.114:0/2074982078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf8964070 0x560bf897f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.990+0000 7f7c937fe700 1 -- 172.21.15.114:0/2074982078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf895e120 msgr2=0x560bf897f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:10.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.990+0000 7f7c937fe700 1 --2- 172.21.15.114:0/2074982078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf895e120 0x560bf897f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:10.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:10.990+0000 7f7c937fe700 1 -- 172.21.15.114:0/2074982078 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bf897fe90 con 0x560bf87dee40 2024-05-27T08:00:11.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.002+0000 7f7c937fe700 1 --2- 172.21.15.114:0/2074982078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf89705a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7c840098e0 tx=0x7f7c8403a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b822ff34a2b8ae64 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:11.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.002+0000 7f7c72ffd700 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-27T08:00:11.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.006+0000 7f7c72ffd700 1 -- 172.21.15.114:0/2074982078 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8404f060 con 0x560bf87dee40 2024-05-27T08:00:11.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.006+0000 7f7c72ffd700 1 -- 172.21.15.114:0/2074982078 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(165..165 src has 1..165) ==== 9642+0+0 (secure 0 0 0) 0x7f7c84021040 con 0x560bf87dee40 2024-05-27T08:00:11.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.006+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2074982078 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c84005940 con 0x560bf87dee40 2024-05-27T08:00:11.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.012+0000 7f7c72ffd700 1 -- 172.21.15.114:0/2074982078 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) ==== 24+0+0 (secure 0 0 0) 0x7f7c84027430 con 0x560bf87dee40 2024-05-27T08:00:11.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.014+0000 7f7ca86e1b80 10 client.29267 fetch_fsmap learned FSMap version 114 2024-05-27T08:00:11.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.014+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2074982078 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=4+}) -- 0x560bf89e2630 con 0x560bf87dee40 2024-05-27T08:00:11.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.014+0000 7f7c72ffd700 1 -- 172.21.15.114:0/2074982078 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c84005940 con 0x560bf87dee40 2024-05-27T08:00:11.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.014+0000 7f7c72ffd700 1 -- 172.21.15.114:0/2074982078 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 114) ==== 2885+0+0 (secure 0 0 0) 0x7f7c8403bb60 con 0x560bf87dee40 2024-05-27T08:00:11.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.014+0000 7f7ca86e1b80 10 client.29267 fetch_fsmap finished waiting for FSMap version 114 2024-05-27T08:00:11.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.014+0000 7f7ca86e1b80 10 client.29267 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T08:00:11.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.014+0000 7f7ca86e1b80 20 client.29267 populate_metadata read hostname 'smithi114' 2024-05-27T08:00:11.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.014+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/2074982078 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560bf89e28a0 0x560bf89e4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:11.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.014+0000 7f7ca86e1b80 4 client.29267 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T08:00:11.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.014+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2074982078 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560bf888dcc0 con 0x560bf89e28a0 2024-05-27T08:00:11.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.016+0000 7f7c92ffd700 1 --2- 172.21.15.114:0/2074982078 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560bf89e28a0 0x560bf89e4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:11.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.020+0000 7f7c72ffd700 10 client.29267 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:11.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.020+0000 7f7c92ffd700 1 --2- 172.21.15.114:0/2074982078 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560bf89e28a0 0x560bf89e4cc0 crc :-1 s=READY pgs=533 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=370d871456094532 server_cookie=c8184220c0dd1ca1 in_seq=0 out_seq=0 2024-05-27T08:00:11.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.020+0000 7f7c72ffd700 1 -- 172.21.15.114:0/2074982078 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560bf888dcc0 con 0x560bf89e28a0 2024-05-27T08:00:11.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.020+0000 7f7c72ffd700 10 client.29267 handle_command_reply: tid=0 2024-05-27T08:00:11.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.020+0000 7f7ca86e1b80 1 client.29267 shutdown 2024-05-27T08:00:11.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.021+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2074982078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 msgr2=0x560bf89705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:11.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.021+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/2074982078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf89705a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7c840098e0 tx=0x7f7c8403a6a0 comp rx=0 tx=0).stop 2024-05-27T08:00:11.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.021+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2074982078 shutdown_connections 2024-05-27T08:00:11.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.021+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/2074982078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560bf8964070 0x560bf897f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:11.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.021+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/2074982078 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x560bf89e28a0 0x560bf89e4cc0 crc :-1 s=READY pgs=533 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:11.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.021+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/2074982078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560bf87dee40 0x560bf89705a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:11.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.021+0000 7f7ca86e1b80 1 --2- 172.21.15.114:0/2074982078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bf895e120 0x560bf897f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:11.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.021+0000 7f7c72ffd700 0 client.29267 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:11.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.021+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2074982078 >> 172.21.15.114:0/2074982078 conn(0x560bf87e1770 msgr2=0x560bf89524f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:11.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.022+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2074982078 shutdown_connections 2024-05-27T08:00:11.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.022+0000 7f7ca86e1b80 1 -- 172.21.15.114:0/2074982078 wait complete. 2024-05-27T08:00:11.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.022+0000 7f7ca86e1b80 20 client.29267 trim_cache size 0 max 16384 2024-05-27T08:00:11.026 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T08:00:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T08:00:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": { 2024-05-27T08:00:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "ceb8123e-d526-4710-b91b-b4ed0f98b5e0": { 2024-05-27T08:00:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "path": "/", 2024-05-27T08:00:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "tag": "ceb8123e-d526-4710-b91b-b4ed0f98b5e0", 2024-05-27T08:00:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "options": "recursive,force" 2024-05-27T08:00:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:11.028 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T08:00:11.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.023+0000 7f7c99ccc700 1 -- 172.21.15.114:0/3144531997 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7c7c0b2e40 msgr2=0x7f7c7c0b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:11.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.023+0000 7f7c99ccc700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7c7c0b2e40 0x7f7c7c0b5350 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009750 tx=0x7f7c8400b8f0 comp rx=0 tx=0).stop 2024-05-27T08:00:11.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.023+0000 7f7c99ccc700 1 -- 172.21.15.114:0/3144531997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c9411d920 msgr2=0x7f7c941ae560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:11.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.023+0000 7f7c99ccc700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c9411d920 0x7f7c941ae560 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c007fd0 tx=0x7f7c8c00aed0 comp rx=0 tx=0).stop 2024-05-27T08:00:11.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.023+0000 7f7c99ccc700 1 -- 172.21.15.114:0/3144531997 shutdown_connections 2024-05-27T08:00:11.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.023+0000 7f7c99ccc700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7c7c0b2e40 0x7f7c7c0b5350 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:11.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.023+0000 7f7c99ccc700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c9411d920 0x7f7c941ae560 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:11.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.023+0000 7f7c99ccc700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c9410a920 0x7f7c941a9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:11.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.023+0000 7f7c99ccc700 1 --2- 172.21.15.114:0/3144531997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c9410b270 0x7f7c941aa2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:11.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.023+0000 7f7c99ccc700 1 -- 172.21.15.114:0/3144531997 >> 172.21.15.114:0/3144531997 conn(0x7f7c9406f4c0 msgr2=0x7f7c9411b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:11.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.023+0000 7f7c99ccc700 1 -- 172.21.15.114:0/3144531997 shutdown_connections 2024-05-27T08:00:11.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:11.023+0000 7f7c99ccc700 1 -- 172.21.15.114:0/3144531997 wait complete. 2024-05-27T08:00:11.825 INFO:tasks.cephfs.filesystem:scrub status for tag:ceb8123e-d526-4710-b91b-b4ed0f98b5e0 - {'path': '/', 'tag': 'ceb8123e-d526-4710-b91b-b4ed0f98b5e0', 'options': 'recursive,force'} 2024-05-27T08:00:12.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:11 smithi114 ceph-mon[32038]: pgmap v706: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 392 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-05-27T08:00:12.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:11 smithi163 ceph-mon[36451]: pgmap v706: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 392 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-05-27T08:00:12.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:11 smithi186 ceph-mon[36647]: pgmap v706: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 392 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-05-27T08:00:13.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:12 smithi163 ceph-mon[36451]: scrub summary: idle 2024-05-27T08:00:13.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:12 smithi186 ceph-mon[36647]: scrub summary: idle 2024-05-27T08:00:13.309 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:12 smithi114 ceph-mon[32038]: scrub summary: idle 2024-05-27T08:00:14.209 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:13 smithi114 ceph-mon[32038]: pgmap v707: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 390 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 191 op/s 2024-05-27T08:00:14.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:13 smithi163 ceph-mon[36451]: pgmap v707: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 390 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 191 op/s 2024-05-27T08:00:14.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:13 smithi186 ceph-mon[36647]: pgmap v707: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 390 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 191 op/s 2024-05-27T08:00:16.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:15 smithi163 ceph-mon[36451]: pgmap v708: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 389 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 163 op/s 2024-05-27T08:00:16.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:15 smithi186 ceph-mon[36647]: pgmap v708: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 389 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 163 op/s 2024-05-27T08:00:16.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:15 smithi114 ceph-mon[32038]: pgmap v708: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 389 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 163 op/s 2024-05-27T08:00:18.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:17 smithi163 ceph-mon[36451]: pgmap v709: 129 pgs: 13 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 76 active+clean; 387 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 174 op/s 2024-05-27T08:00:18.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:17 smithi186 ceph-mon[36647]: pgmap v709: 129 pgs: 13 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 76 active+clean; 387 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 174 op/s 2024-05-27T08:00:18.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:17 smithi114 ceph-mon[32038]: pgmap v709: 129 pgs: 13 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 76 active+clean; 387 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 174 op/s 2024-05-27T08:00:18.972 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:18 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-05-27T08:00:18.972 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:18 smithi114 systemd[1]: Stopped Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:00:18.972 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:18 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Consumed 2.630s CPU time. 2024-05-27T08:00:18.972 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:18 smithi114 systemd[1]: Starting Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:00:19.283 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:18 smithi114 podman[135539]: 2024-05-27 08:00:18.877190452 +0000 UTC m=+0.047715937 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:00:19.284 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:19 smithi114 podman[135539]: 2024-05-27 08:00:19.02901844 +0000 UTC m=+0.199543951 container create 6b2ba67db47bde9050853160f89e1629d5d25332057ee8e2010763fa287460f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T08:00:19.284 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:19 smithi114 podman[135539]: 2024-05-27 08:00:19.282236699 +0000 UTC m=+0.452762181 container init 6b2ba67db47bde9050853160f89e1629d5d25332057ee8e2010763fa287460f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T08:00:19.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:19 smithi114 ceph-mon[32038]: daemon mds.d restarted 2024-05-27T08:00:19.558 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:19 smithi114 podman[135539]: 2024-05-27 08:00:19.287192852 +0000 UTC m=+0.457718323 container start 6b2ba67db47bde9050853160f89e1629d5d25332057ee8e2010763fa287460f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-05-27T08:00:19.558 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:19 smithi114 ceph-mds[135556]: -- 172.21.15.114:0/1022966517 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5642e9717a40 con 0x5642e975fa80 2024-05-27T08:00:19.558 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:19 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: starting mds.d at 2024-05-27T08:00:19.558 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:19 smithi114 bash[135539]: 6b2ba67db47bde9050853160f89e1629d5d25332057ee8e2010763fa287460f0 2024-05-27T08:00:19.559 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:19 smithi114 systemd[1]: Started Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:00:19.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:19 smithi163 ceph-mon[36451]: daemon mds.d restarted 2024-05-27T08:00:19.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:19 smithi186 ceph-mon[36647]: daemon mds.d restarted 2024-05-27T08:00:20.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:20 smithi114 ceph-mon[32038]: osdmap e166: 12 total, 12 up, 12 in 2024-05-27T08:00:20.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:20 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6836/298251137,v1:172.21.15.114:6837/298251137] up:boot 2024-05-27T08:00:20.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:20 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 2 up:standby 2024-05-27T08:00:20.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:20 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:00:20.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:20 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:20.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:20 smithi114 ceph-mon[32038]: pgmap v711: 129 pgs: 7 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 104 active+clean; 379 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.8 MiB/s wr, 168 op/s 2024-05-27T08:00:20.559 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6e931f5700 time 2024-05-27T08:00:20.303715+0000 2024-05-27T08:00:20.559 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:20.559 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:20.559 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6ea5a3aba6] 2024-05-27T08:00:20.560 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6ea5a3add1] 2024-05-27T08:00:20.560 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5642e847723d] 2024-05-27T08:00:20.560 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5642e8462971] 2024-05-27T08:00:20.560 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 5: (SnapRealm::check_cache() const+0x13e) [0x5642e8460fa0] 2024-05-27T08:00:20.560 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5642e8461651] 2024-05-27T08:00:20.560 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5642e83f6347] 2024-05-27T08:00:20.560 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5642e80f287e] 2024-05-27T08:00:20.560 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5642e80f7e94] 2024-05-27T08:00:20.561 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5642e8102969] 2024-05-27T08:00:20.561 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 11: (MDLog::_replay_thread()+0x1f38) [0x5642e849b48c] 2024-05-27T08:00:20.561 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5642e84a49cb] 2024-05-27T08:00:20.561 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 13: (Thread::entry_wrapper()+0x43) [0x7f6ea5a0972d] 2024-05-27T08:00:20.561 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6ea5a09749] 2024-05-27T08:00:20.561 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6ea45ad1ca] 2024-05-27T08:00:20.561 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 16: clone() 2024-05-27T08:00:20.561 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.305+0000 7f6e931f5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6e931f5700 time 2024-05-27T08:00:20.303715+0000 2024-05-27T08:00:20.562 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:20.562 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.562 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:20.562 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6ea5a3aba6] 2024-05-27T08:00:20.562 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6ea5a3add1] 2024-05-27T08:00:20.562 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5642e847723d] 2024-05-27T08:00:20.562 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5642e8462971] 2024-05-27T08:00:20.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 5: (SnapRealm::check_cache() const+0x13e) [0x5642e8460fa0] 2024-05-27T08:00:20.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5642e8461651] 2024-05-27T08:00:20.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5642e83f6347] 2024-05-27T08:00:20.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5642e80f287e] 2024-05-27T08:00:20.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5642e80f7e94] 2024-05-27T08:00:20.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5642e8102969] 2024-05-27T08:00:20.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 11: (MDLog::_replay_thread()+0x1f38) [0x5642e849b48c] 2024-05-27T08:00:20.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5642e84a49cb] 2024-05-27T08:00:20.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 13: (Thread::entry_wrapper()+0x43) [0x7f6ea5a0972d] 2024-05-27T08:00:20.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6ea5a09749] 2024-05-27T08:00:20.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6ea45ad1ca] 2024-05-27T08:00:20.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 16: clone() 2024-05-27T08:00:20.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: *** Caught signal (Aborted) ** 2024-05-27T08:00:20.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: in thread 7f6e931f5700 thread_name:md_log_replay 2024-05-27T08:00:20.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:20.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6ea45b7d20] 2024-05-27T08:00:20.565 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2: gsignal() 2024-05-27T08:00:20.565 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 3: abort() 2024-05-27T08:00:20.565 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6ea5a3aced] 2024-05-27T08:00:20.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6ea5a3add1] 2024-05-27T08:00:20.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5642e847723d] 2024-05-27T08:00:20.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5642e8462971] 2024-05-27T08:00:20.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 8: (SnapRealm::check_cache() const+0x13e) [0x5642e8460fa0] 2024-05-27T08:00:20.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5642e8461651] 2024-05-27T08:00:20.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5642e83f6347] 2024-05-27T08:00:20.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5642e80f287e] 2024-05-27T08:00:20.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5642e80f7e94] 2024-05-27T08:00:20.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5642e8102969] 2024-05-27T08:00:20.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 14: (MDLog::_replay_thread()+0x1f38) [0x5642e849b48c] 2024-05-27T08:00:20.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5642e84a49cb] 2024-05-27T08:00:20.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 16: (Thread::entry_wrapper()+0x43) [0x7f6ea5a0972d] 2024-05-27T08:00:20.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6ea5a09749] 2024-05-27T08:00:20.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6ea45ad1ca] 2024-05-27T08:00:20.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 19: clone() 2024-05-27T08:00:20.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.308+0000 7f6e931f5700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:00:20.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: in thread 7f6e931f5700 thread_name:md_log_replay 2024-05-27T08:00:20.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:20.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6ea45b7d20] 2024-05-27T08:00:20.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2: gsignal() 2024-05-27T08:00:20.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 3: abort() 2024-05-27T08:00:20.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6ea5a3aced] 2024-05-27T08:00:20.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6ea5a3add1] 2024-05-27T08:00:20.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5642e847723d] 2024-05-27T08:00:20.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5642e8462971] 2024-05-27T08:00:20.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 8: (SnapRealm::check_cache() const+0x13e) [0x5642e8460fa0] 2024-05-27T08:00:20.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5642e8461651] 2024-05-27T08:00:20.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5642e83f6347] 2024-05-27T08:00:20.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5642e80f287e] 2024-05-27T08:00:20.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5642e80f7e94] 2024-05-27T08:00:20.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5642e8102969] 2024-05-27T08:00:20.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 14: (MDLog::_replay_thread()+0x1f38) [0x5642e849b48c] 2024-05-27T08:00:20.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5642e84a49cb] 2024-05-27T08:00:20.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 16: (Thread::entry_wrapper()+0x43) [0x7f6ea5a0972d] 2024-05-27T08:00:20.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6ea5a09749] 2024-05-27T08:00:20.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6ea45ad1ca] 2024-05-27T08:00:20.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 19: clone() 2024-05-27T08:00:20.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:00:20.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: -1> 2024-05-27T08:00:20.305+0000 7f6e931f5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6e931f5700 time 2024-05-27T08:00:20.303715+0000 2024-05-27T08:00:20.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:20.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:20.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6ea5a3aba6] 2024-05-27T08:00:20.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6ea5a3add1] 2024-05-27T08:00:20.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5642e847723d] 2024-05-27T08:00:20.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5642e8462971] 2024-05-27T08:00:20.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 5: (SnapRealm::check_cache() const+0x13e) [0x5642e8460fa0] 2024-05-27T08:00:20.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5642e8461651] 2024-05-27T08:00:20.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5642e83f6347] 2024-05-27T08:00:20.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5642e80f287e] 2024-05-27T08:00:20.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5642e80f7e94] 2024-05-27T08:00:20.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5642e8102969] 2024-05-27T08:00:20.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 11: (MDLog::_replay_thread()+0x1f38) [0x5642e849b48c] 2024-05-27T08:00:20.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5642e84a49cb] 2024-05-27T08:00:20.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 13: (Thread::entry_wrapper()+0x43) [0x7f6ea5a0972d] 2024-05-27T08:00:20.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6ea5a09749] 2024-05-27T08:00:20.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6ea45ad1ca] 2024-05-27T08:00:20.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 16: clone() 2024-05-27T08:00:20.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 0> 2024-05-27T08:00:20.308+0000 7f6e931f5700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:00:20.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: in thread 7f6e931f5700 thread_name:md_log_replay 2024-05-27T08:00:20.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:20.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6ea45b7d20] 2024-05-27T08:00:20.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2: gsignal() 2024-05-27T08:00:20.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 3: abort() 2024-05-27T08:00:20.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6ea5a3aced] 2024-05-27T08:00:20.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6ea5a3add1] 2024-05-27T08:00:20.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5642e847723d] 2024-05-27T08:00:20.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5642e8462971] 2024-05-27T08:00:20.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 8: (SnapRealm::check_cache() const+0x13e) [0x5642e8460fa0] 2024-05-27T08:00:20.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5642e8461651] 2024-05-27T08:00:20.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5642e83f6347] 2024-05-27T08:00:20.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5642e80f287e] 2024-05-27T08:00:20.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5642e80f7e94] 2024-05-27T08:00:20.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5642e8102969] 2024-05-27T08:00:20.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 14: (MDLog::_replay_thread()+0x1f38) [0x5642e849b48c] 2024-05-27T08:00:20.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5642e84a49cb] 2024-05-27T08:00:20.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 16: (Thread::entry_wrapper()+0x43) [0x7f6ea5a0972d] 2024-05-27T08:00:20.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6ea5a09749] 2024-05-27T08:00:20.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6ea45ad1ca] 2024-05-27T08:00:20.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 19: clone() 2024-05-27T08:00:20.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:00:20.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: -1> 2024-05-27T08:00:20.305+0000 7f6e931f5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6e931f5700 time 2024-05-27T08:00:20.303715+0000 2024-05-27T08:00:20.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:20.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:20.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6ea5a3aba6] 2024-05-27T08:00:20.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6ea5a3add1] 2024-05-27T08:00:20.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5642e847723d] 2024-05-27T08:00:20.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5642e8462971] 2024-05-27T08:00:20.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 5: (SnapRealm::check_cache() const+0x13e) [0x5642e8460fa0] 2024-05-27T08:00:20.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5642e8461651] 2024-05-27T08:00:20.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5642e83f6347] 2024-05-27T08:00:20.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5642e80f287e] 2024-05-27T08:00:20.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5642e80f7e94] 2024-05-27T08:00:20.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5642e8102969] 2024-05-27T08:00:20.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 11: (MDLog::_replay_thread()+0x1f38) [0x5642e849b48c] 2024-05-27T08:00:20.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5642e84a49cb] 2024-05-27T08:00:20.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 13: (Thread::entry_wrapper()+0x43) [0x7f6ea5a0972d] 2024-05-27T08:00:20.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6ea5a09749] 2024-05-27T08:00:20.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6ea45ad1ca] 2024-05-27T08:00:20.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 16: clone() 2024-05-27T08:00:20.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 0> 2024-05-27T08:00:20.308+0000 7f6e931f5700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:00:20.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: in thread 7f6e931f5700 thread_name:md_log_replay 2024-05-27T08:00:20.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:20.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6ea45b7d20] 2024-05-27T08:00:20.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2: gsignal() 2024-05-27T08:00:20.580 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 3: abort() 2024-05-27T08:00:20.580 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6ea5a3aced] 2024-05-27T08:00:20.580 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6ea5a3add1] 2024-05-27T08:00:20.580 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5642e847723d] 2024-05-27T08:00:20.580 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5642e8462971] 2024-05-27T08:00:20.580 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 8: (SnapRealm::check_cache() const+0x13e) [0x5642e8460fa0] 2024-05-27T08:00:20.580 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5642e8461651] 2024-05-27T08:00:20.580 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5642e83f6347] 2024-05-27T08:00:20.581 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5642e80f287e] 2024-05-27T08:00:20.581 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5642e80f7e94] 2024-05-27T08:00:20.581 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5642e8102969] 2024-05-27T08:00:20.581 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 14: (MDLog::_replay_thread()+0x1f38) [0x5642e849b48c] 2024-05-27T08:00:20.581 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5642e84a49cb] 2024-05-27T08:00:20.581 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 16: (Thread::entry_wrapper()+0x43) [0x7f6ea5a0972d] 2024-05-27T08:00:20.581 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6ea5a09749] 2024-05-27T08:00:20.581 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6ea45ad1ca] 2024-05-27T08:00:20.582 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 19: clone() 2024-05-27T08:00:20.582 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:00:20.582 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[135552]: 2024-05-27T08:00:20.582 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:20 smithi114 podman[135587]: 2024-05-27 08:00:20.388021164 +0000 UTC m=+0.022036014 container died 6b2ba67db47bde9050853160f89e1629d5d25332057ee8e2010763fa287460f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:00:20.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:20 smithi163 ceph-mon[36451]: osdmap e166: 12 total, 12 up, 12 in 2024-05-27T08:00:20.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:20 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6836/298251137,v1:172.21.15.114:6837/298251137] up:boot 2024-05-27T08:00:20.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:20 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 2 up:standby 2024-05-27T08:00:20.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:20 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:00:20.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:20 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:20.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:20 smithi163 ceph-mon[36451]: pgmap v711: 129 pgs: 7 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 104 active+clean; 379 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.8 MiB/s wr, 168 op/s 2024-05-27T08:00:20.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:20 smithi186 ceph-mon[36647]: osdmap e166: 12 total, 12 up, 12 in 2024-05-27T08:00:20.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:20 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6836/298251137,v1:172.21.15.114:6837/298251137] up:boot 2024-05-27T08:00:20.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:20 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 2 up:standby 2024-05-27T08:00:20.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:20 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:00:20.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:20 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:20.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:20 smithi186 ceph-mon[36647]: pgmap v711: 129 pgs: 7 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 104 active+clean; 379 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.8 MiB/s wr, 168 op/s 2024-05-27T08:00:21.557 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:21 smithi114 podman[135587]: 2024-05-27 08:00:21.240043733 +0000 UTC m=+0.874058574 container remove 6b2ba67db47bde9050853160f89e1629d5d25332057ee8e2010763fa287460f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:00:21.558 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:21 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T08:00:21.942 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:21 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T08:00:21.942 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:21 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Consumed 2.038s CPU time. 2024-05-27T08:00:22.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:21 smithi163 ceph-mon[36451]: pgmap v712: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 369 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 143 op/s 2024-05-27T08:00:22.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:21 smithi186 ceph-mon[36647]: pgmap v712: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 369 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 143 op/s 2024-05-27T08:00:22.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:21 smithi114 ceph-mon[32038]: pgmap v712: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 369 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 143 op/s 2024-05-27T08:00:24.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:23 smithi163 ceph-mon[36451]: pgmap v713: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 368 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 143 op/s 2024-05-27T08:00:24.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:23 smithi186 ceph-mon[36647]: pgmap v713: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 368 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 143 op/s 2024-05-27T08:00:24.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:23 smithi114 ceph-mon[32038]: pgmap v713: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 368 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 143 op/s 2024-05-27T08:00:25.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:25 smithi114 ceph-mon[32038]: osdmap e167: 12 total, 12 up, 12 in 2024-05-27T08:00:25.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:25 smithi163 ceph-mon[36451]: osdmap e167: 12 total, 12 up, 12 in 2024-05-27T08:00:25.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:25 smithi186 ceph-mon[36647]: osdmap e167: 12 total, 12 up, 12 in 2024-05-27T08:00:26.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:26 smithi114 ceph-mon[32038]: pgmap v715: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 369 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.4 MiB/s wr, 132 op/s 2024-05-27T08:00:26.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:26 smithi163 ceph-mon[36451]: pgmap v715: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 369 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.4 MiB/s wr, 132 op/s 2024-05-27T08:00:26.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:26 smithi186 ceph-mon[36647]: pgmap v715: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 369 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.4 MiB/s wr, 132 op/s 2024-05-27T08:00:28.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:27 smithi163 ceph-mon[36451]: pgmap v716: 129 pgs: 129 active+clean; 368 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.4 MiB/s wr, 140 op/s 2024-05-27T08:00:28.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:27 smithi186 ceph-mon[36647]: pgmap v716: 129 pgs: 129 active+clean; 368 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.4 MiB/s wr, 140 op/s 2024-05-27T08:00:28.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:27 smithi114 ceph-mon[32038]: pgmap v716: 129 pgs: 129 active+clean; 368 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.4 MiB/s wr, 140 op/s 2024-05-27T08:00:29.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:28 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T08:00:29.227 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:28 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:28 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:28 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:28 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:28 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:28 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:28 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T08:00:29.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:28 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:28 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:28 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:28 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.231 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:28 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.231 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:28 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:28 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T08:00:29.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:28 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:28 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:28 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:28 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:28 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:29.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:28 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:30.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:30 smithi114 ceph-mon[32038]: osdmap e168: 12 total, 12 up, 12 in 2024-05-27T08:00:30.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:30 smithi114 ceph-mon[32038]: pgmap v718: 129 pgs: 129 active+clean; 368 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.0 MiB/s wr, 91 op/s 2024-05-27T08:00:30.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:30 smithi163 ceph-mon[36451]: osdmap e168: 12 total, 12 up, 12 in 2024-05-27T08:00:30.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:30 smithi163 ceph-mon[36451]: pgmap v718: 129 pgs: 129 active+clean; 368 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.0 MiB/s wr, 91 op/s 2024-05-27T08:00:30.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:30 smithi186 ceph-mon[36647]: osdmap e168: 12 total, 12 up, 12 in 2024-05-27T08:00:30.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:30 smithi186 ceph-mon[36647]: pgmap v718: 129 pgs: 129 active+clean; 368 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.0 MiB/s wr, 91 op/s 2024-05-27T08:00:32.082 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:31 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-05-27T08:00:32.082 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:31 smithi114 systemd[1]: Stopped Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:00:32.082 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:31 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Consumed 2.038s CPU time. 2024-05-27T08:00:32.082 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:31 smithi114 systemd[1]: Starting Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:00:32.082 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:32 smithi114 podman[136200]: 2024-05-27 08:00:31.947272083 +0000 UTC m=+0.018621962 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:00:32.364 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:32 smithi114 podman[136200]: 2024-05-27 08:00:32.080633414 +0000 UTC m=+0.151983265 container create de8c1dc4081cf9ea5d53f0da28ff20f504d84d5e5f51187d81c8aac38547efcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-27T08:00:32.365 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:32 smithi114 podman[136200]: 2024-05-27 08:00:32.325975331 +0000 UTC m=+0.397325215 container init de8c1dc4081cf9ea5d53f0da28ff20f504d84d5e5f51187d81c8aac38547efcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-05-27T08:00:32.365 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:32 smithi114 podman[136200]: 2024-05-27 08:00:32.32922878 +0000 UTC m=+0.400578643 container start de8c1dc4081cf9ea5d53f0da28ff20f504d84d5e5f51187d81c8aac38547efcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T08:00:32.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: pgmap v719: 129 pgs: 129 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 MiB/s wr, 111 op/s 2024-05-27T08:00:32.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:32 smithi163 ceph-mon[36451]: daemon mds.d restarted 2024-05-27T08:00:32.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: pgmap v719: 129 pgs: 129 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 MiB/s wr, 111 op/s 2024-05-27T08:00:32.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:32 smithi186 ceph-mon[36647]: daemon mds.d restarted 2024-05-27T08:00:32.808 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mds[136250]: -- 172.21.15.114:0/81108133 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563b767b3a40 con 0x563b767fb180 2024-05-27T08:00:32.808 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:32 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: starting mds.d at 2024-05-27T08:00:32.808 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:32 smithi114 bash[136200]: de8c1dc4081cf9ea5d53f0da28ff20f504d84d5e5f51187d81c8aac38547efcc 2024-05-27T08:00:32.808 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:32 smithi114 systemd[1]: Started Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:00:32.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:32.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: pgmap v719: 129 pgs: 129 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 MiB/s wr, 111 op/s 2024-05-27T08:00:32.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:32.811 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:32 smithi114 ceph-mon[32038]: daemon mds.d restarted 2024-05-27T08:00:33.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-27T08:00:33.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: Unable to set osd_memory_target on smithi163 to 184781004: error parsing value: Value '184781004' is below minimum 939524096 2024-05-27T08:00:33.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-27T08:00:33.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T08:00:33.559 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: osdmap e169: 12 total, 12 up, 12 in 2024-05-27T08:00:33.559 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:33.559 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:33.559 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:33.559 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:33.559 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: Adjusting osd_memory_target on smithi114 to 176.1M 2024-05-27T08:00:33.559 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: Unable to set osd_memory_target on smithi114 to 184753049: error parsing value: Value '184753049' is below minimum 939524096 2024-05-27T08:00:33.560 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T08:00:33.560 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T08:00:33.560 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: pgmap v721: 129 pgs: 129 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.9 MiB/s wr, 113 op/s 2024-05-27T08:00:33.560 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: pgmap v722: 129 pgs: 129 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.4 MiB/s wr, 106 op/s 2024-05-27T08:00:33.560 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:33.560 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: Health check update: 10 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T08:00:33.560 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6836/3202601367,v1:172.21.15.114:6837/3202601367] up:boot 2024-05-27T08:00:33.560 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 2 up:standby 2024-05-27T08:00:33.561 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:00:33.561 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:33 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:33.562 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f382fde5700 time 2024-05-27T08:00:33.210776+0000 2024-05-27T08:00:33.562 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:33.562 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:33.562 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f384262aba6] 2024-05-27T08:00:33.562 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f384262add1] 2024-05-27T08:00:33.562 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563b74a9c23d] 2024-05-27T08:00:33.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563b74a87971] 2024-05-27T08:00:33.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 5: (SnapRealm::check_cache() const+0x13e) [0x563b74a85fa0] 2024-05-27T08:00:33.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563b74a86651] 2024-05-27T08:00:33.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563b74a1b347] 2024-05-27T08:00:33.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563b7471787e] 2024-05-27T08:00:33.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563b7471ce94] 2024-05-27T08:00:33.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563b74727969] 2024-05-27T08:00:33.563 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 11: (MDLog::_replay_thread()+0x1f38) [0x563b74ac048c] 2024-05-27T08:00:33.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563b74ac99cb] 2024-05-27T08:00:33.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 13: (Thread::entry_wrapper()+0x43) [0x7f38425f972d] 2024-05-27T08:00:33.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 14: (Thread::_entry_func(void*)+0xd) [0x7f38425f9749] 2024-05-27T08:00:33.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f384119d1ca] 2024-05-27T08:00:33.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 16: clone() 2024-05-27T08:00:33.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: *** Caught signal (Aborted) ** 2024-05-27T08:00:33.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: in thread 7f382fde5700 thread_name:md_log_replay 2024-05-27T08:00:33.564 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.211+0000 7f382fde5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f382fde5700 time 2024-05-27T08:00:33.210776+0000 2024-05-27T08:00:33.565 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:33.565 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.565 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:33.565 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f384262aba6] 2024-05-27T08:00:33.565 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f384262add1] 2024-05-27T08:00:33.565 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563b74a9c23d] 2024-05-27T08:00:33.565 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563b74a87971] 2024-05-27T08:00:33.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 5: (SnapRealm::check_cache() const+0x13e) [0x563b74a85fa0] 2024-05-27T08:00:33.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563b74a86651] 2024-05-27T08:00:33.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563b74a1b347] 2024-05-27T08:00:33.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563b7471787e] 2024-05-27T08:00:33.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563b7471ce94] 2024-05-27T08:00:33.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563b74727969] 2024-05-27T08:00:33.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 11: (MDLog::_replay_thread()+0x1f38) [0x563b74ac048c] 2024-05-27T08:00:33.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563b74ac99cb] 2024-05-27T08:00:33.566 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 13: (Thread::entry_wrapper()+0x43) [0x7f38425f972d] 2024-05-27T08:00:33.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 14: (Thread::_entry_func(void*)+0xd) [0x7f38425f9749] 2024-05-27T08:00:33.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f384119d1ca] 2024-05-27T08:00:33.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 16: clone() 2024-05-27T08:00:33.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:33.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38411a7d20] 2024-05-27T08:00:33.567 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2: gsignal() 2024-05-27T08:00:33.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 3: abort() 2024-05-27T08:00:33.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f384262aced] 2024-05-27T08:00:33.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f384262add1] 2024-05-27T08:00:33.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563b74a9c23d] 2024-05-27T08:00:33.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563b74a87971] 2024-05-27T08:00:33.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 8: (SnapRealm::check_cache() const+0x13e) [0x563b74a85fa0] 2024-05-27T08:00:33.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563b74a86651] 2024-05-27T08:00:33.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563b74a1b347] 2024-05-27T08:00:33.568 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563b7471787e] 2024-05-27T08:00:33.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563b7471ce94] 2024-05-27T08:00:33.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563b74727969] 2024-05-27T08:00:33.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 14: (MDLog::_replay_thread()+0x1f38) [0x563b74ac048c] 2024-05-27T08:00:33.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563b74ac99cb] 2024-05-27T08:00:33.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 16: (Thread::entry_wrapper()+0x43) [0x7f38425f972d] 2024-05-27T08:00:33.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 17: (Thread::_entry_func(void*)+0xd) [0x7f38425f9749] 2024-05-27T08:00:33.569 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f384119d1ca] 2024-05-27T08:00:33.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 19: clone() 2024-05-27T08:00:33.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.214+0000 7f382fde5700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:00:33.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: in thread 7f382fde5700 thread_name:md_log_replay 2024-05-27T08:00:33.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:33.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38411a7d20] 2024-05-27T08:00:33.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2: gsignal() 2024-05-27T08:00:33.570 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 3: abort() 2024-05-27T08:00:33.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f384262aced] 2024-05-27T08:00:33.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f384262add1] 2024-05-27T08:00:33.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563b74a9c23d] 2024-05-27T08:00:33.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563b74a87971] 2024-05-27T08:00:33.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 8: (SnapRealm::check_cache() const+0x13e) [0x563b74a85fa0] 2024-05-27T08:00:33.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563b74a86651] 2024-05-27T08:00:33.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563b74a1b347] 2024-05-27T08:00:33.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563b7471787e] 2024-05-27T08:00:33.571 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563b7471ce94] 2024-05-27T08:00:33.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563b74727969] 2024-05-27T08:00:33.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 14: (MDLog::_replay_thread()+0x1f38) [0x563b74ac048c] 2024-05-27T08:00:33.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563b74ac99cb] 2024-05-27T08:00:33.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 16: (Thread::entry_wrapper()+0x43) [0x7f38425f972d] 2024-05-27T08:00:33.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 17: (Thread::_entry_func(void*)+0xd) [0x7f38425f9749] 2024-05-27T08:00:33.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f384119d1ca] 2024-05-27T08:00:33.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 19: clone() 2024-05-27T08:00:33.572 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:00:33.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: -1> 2024-05-27T08:00:33.211+0000 7f382fde5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f382fde5700 time 2024-05-27T08:00:33.210776+0000 2024-05-27T08:00:33.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:33.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:33.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f384262aba6] 2024-05-27T08:00:33.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f384262add1] 2024-05-27T08:00:33.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563b74a9c23d] 2024-05-27T08:00:33.573 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563b74a87971] 2024-05-27T08:00:33.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 5: (SnapRealm::check_cache() const+0x13e) [0x563b74a85fa0] 2024-05-27T08:00:33.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563b74a86651] 2024-05-27T08:00:33.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563b74a1b347] 2024-05-27T08:00:33.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563b7471787e] 2024-05-27T08:00:33.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563b7471ce94] 2024-05-27T08:00:33.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563b74727969] 2024-05-27T08:00:33.574 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 11: (MDLog::_replay_thread()+0x1f38) [0x563b74ac048c] 2024-05-27T08:00:33.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563b74ac99cb] 2024-05-27T08:00:33.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 13: (Thread::entry_wrapper()+0x43) [0x7f38425f972d] 2024-05-27T08:00:33.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 14: (Thread::_entry_func(void*)+0xd) [0x7f38425f9749] 2024-05-27T08:00:33.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f384119d1ca] 2024-05-27T08:00:33.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 16: clone() 2024-05-27T08:00:33.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 0> 2024-05-27T08:00:33.214+0000 7f382fde5700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:00:33.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: in thread 7f382fde5700 thread_name:md_log_replay 2024-05-27T08:00:33.575 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:33.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38411a7d20] 2024-05-27T08:00:33.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2: gsignal() 2024-05-27T08:00:33.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 3: abort() 2024-05-27T08:00:33.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f384262aced] 2024-05-27T08:00:33.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f384262add1] 2024-05-27T08:00:33.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563b74a9c23d] 2024-05-27T08:00:33.576 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563b74a87971] 2024-05-27T08:00:33.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 8: (SnapRealm::check_cache() const+0x13e) [0x563b74a85fa0] 2024-05-27T08:00:33.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563b74a86651] 2024-05-27T08:00:33.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563b74a1b347] 2024-05-27T08:00:33.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563b7471787e] 2024-05-27T08:00:33.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563b7471ce94] 2024-05-27T08:00:33.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563b74727969] 2024-05-27T08:00:33.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 14: (MDLog::_replay_thread()+0x1f38) [0x563b74ac048c] 2024-05-27T08:00:33.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563b74ac99cb] 2024-05-27T08:00:33.577 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 16: (Thread::entry_wrapper()+0x43) [0x7f38425f972d] 2024-05-27T08:00:33.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 17: (Thread::_entry_func(void*)+0xd) [0x7f38425f9749] 2024-05-27T08:00:33.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f384119d1ca] 2024-05-27T08:00:33.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 19: clone() 2024-05-27T08:00:33.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:00:33.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: -1> 2024-05-27T08:00:33.211+0000 7f382fde5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f382fde5700 time 2024-05-27T08:00:33.210776+0000 2024-05-27T08:00:33.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:33.578 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:33.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f384262aba6] 2024-05-27T08:00:33.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f384262add1] 2024-05-27T08:00:33.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563b74a9c23d] 2024-05-27T08:00:33.579 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563b74a87971] 2024-05-27T08:00:33.581 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 5: (SnapRealm::check_cache() const+0x13e) [0x563b74a85fa0] 2024-05-27T08:00:33.581 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563b74a86651] 2024-05-27T08:00:33.581 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563b74a1b347] 2024-05-27T08:00:33.581 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563b7471787e] 2024-05-27T08:00:33.582 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563b7471ce94] 2024-05-27T08:00:33.582 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563b74727969] 2024-05-27T08:00:33.582 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 11: (MDLog::_replay_thread()+0x1f38) [0x563b74ac048c] 2024-05-27T08:00:33.582 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563b74ac99cb] 2024-05-27T08:00:33.582 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 13: (Thread::entry_wrapper()+0x43) [0x7f38425f972d] 2024-05-27T08:00:33.582 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 14: (Thread::_entry_func(void*)+0xd) [0x7f38425f9749] 2024-05-27T08:00:33.582 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f384119d1ca] 2024-05-27T08:00:33.582 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 16: clone() 2024-05-27T08:00:33.582 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.583 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 0> 2024-05-27T08:00:33.214+0000 7f382fde5700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:00:33.583 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: in thread 7f382fde5700 thread_name:md_log_replay 2024-05-27T08:00:33.583 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.583 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:33.583 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38411a7d20] 2024-05-27T08:00:33.583 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2: gsignal() 2024-05-27T08:00:33.583 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 3: abort() 2024-05-27T08:00:33.583 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f384262aced] 2024-05-27T08:00:33.583 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f384262add1] 2024-05-27T08:00:33.584 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563b74a9c23d] 2024-05-27T08:00:33.584 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563b74a87971] 2024-05-27T08:00:33.584 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 8: (SnapRealm::check_cache() const+0x13e) [0x563b74a85fa0] 2024-05-27T08:00:33.584 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563b74a86651] 2024-05-27T08:00:33.584 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563b74a1b347] 2024-05-27T08:00:33.584 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563b7471787e] 2024-05-27T08:00:33.584 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563b7471ce94] 2024-05-27T08:00:33.584 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563b74727969] 2024-05-27T08:00:33.584 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 14: (MDLog::_replay_thread()+0x1f38) [0x563b74ac048c] 2024-05-27T08:00:33.585 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563b74ac99cb] 2024-05-27T08:00:33.585 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 16: (Thread::entry_wrapper()+0x43) [0x7f38425f972d] 2024-05-27T08:00:33.585 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 17: (Thread::_entry_func(void*)+0xd) [0x7f38425f9749] 2024-05-27T08:00:33.585 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f384119d1ca] 2024-05-27T08:00:33.585 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 19: clone() 2024-05-27T08:00:33.585 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:00:33.585 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136227]: 2024-05-27T08:00:33.585 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:33 smithi114 podman[136313]: 2024-05-27 08:00:33.271168682 +0000 UTC m=+0.018419045 container died de8c1dc4081cf9ea5d53f0da28ff20f504d84d5e5f51187d81c8aac38547efcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-27T08:00:33.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-27T08:00:33.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: Unable to set osd_memory_target on smithi163 to 184781004: error parsing value: Value '184781004' is below minimum 939524096 2024-05-27T08:00:33.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-27T08:00:33.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T08:00:33.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: osdmap e169: 12 total, 12 up, 12 in 2024-05-27T08:00:33.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:33.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:33.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:33.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:33.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: Adjusting osd_memory_target on smithi114 to 176.1M 2024-05-27T08:00:33.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: Unable to set osd_memory_target on smithi114 to 184753049: error parsing value: Value '184753049' is below minimum 939524096 2024-05-27T08:00:33.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T08:00:33.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T08:00:33.730 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: pgmap v721: 129 pgs: 129 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.9 MiB/s wr, 113 op/s 2024-05-27T08:00:33.730 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: pgmap v722: 129 pgs: 129 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.4 MiB/s wr, 106 op/s 2024-05-27T08:00:33.730 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:33.730 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: Health check update: 10 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T08:00:33.730 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6836/3202601367,v1:172.21.15.114:6837/3202601367] up:boot 2024-05-27T08:00:33.730 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 2 up:standby 2024-05-27T08:00:33.730 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:00:33.730 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:33 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:33.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-27T08:00:33.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: Unable to set osd_memory_target on smithi163 to 184781004: error parsing value: Value '184781004' is below minimum 939524096 2024-05-27T08:00:33.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-27T08:00:33.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T08:00:33.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: osdmap e169: 12 total, 12 up, 12 in 2024-05-27T08:00:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T08:00:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: Adjusting osd_memory_target on smithi114 to 176.1M 2024-05-27T08:00:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: Unable to set osd_memory_target on smithi114 to 184753049: error parsing value: Value '184753049' is below minimum 939524096 2024-05-27T08:00:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T08:00:33.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T08:00:33.733 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: pgmap v721: 129 pgs: 129 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.9 MiB/s wr, 113 op/s 2024-05-27T08:00:33.733 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: pgmap v722: 129 pgs: 129 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.4 MiB/s wr, 106 op/s 2024-05-27T08:00:33.733 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:00:33.733 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: Health check update: 10 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T08:00:33.733 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6836/3202601367,v1:172.21.15.114:6837/3202601367] up:boot 2024-05-27T08:00:33.733 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 2 up:standby 2024-05-27T08:00:33.733 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:00:33.733 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:33 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:34.528 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:34 smithi114 podman[136313]: 2024-05-27 08:00:34.276145877 +0000 UTC m=+1.023396252 container remove de8c1dc4081cf9ea5d53f0da28ff20f504d84d5e5f51187d81c8aac38547efcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521) 2024-05-27T08:00:34.529 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:34 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T08:00:34.808 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:34 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T08:00:34.808 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:34 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.511s CPU time. 2024-05-27T08:00:35.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:35 smithi163 ceph-mon[36451]: pgmap v723: 129 pgs: 129 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.1 MiB/s wr, 104 op/s 2024-05-27T08:00:35.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:35 smithi186 ceph-mon[36647]: pgmap v723: 129 pgs: 129 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.1 MiB/s wr, 104 op/s 2024-05-27T08:00:35.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:35 smithi114 ceph-mon[32038]: pgmap v723: 129 pgs: 129 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.1 MiB/s wr, 104 op/s 2024-05-27T08:00:37.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:37 smithi163 ceph-mon[36451]: pgmap v724: 129 pgs: 129 active+clean; 371 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.7 MiB/s wr, 91 op/s 2024-05-27T08:00:37.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:37 smithi186 ceph-mon[36647]: pgmap v724: 129 pgs: 129 active+clean; 371 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.7 MiB/s wr, 91 op/s 2024-05-27T08:00:37.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:37 smithi114 ceph-mon[32038]: pgmap v724: 129 pgs: 129 active+clean; 371 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.7 MiB/s wr, 91 op/s 2024-05-27T08:00:39.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:39 smithi163 ceph-mon[36451]: pgmap v725: 129 pgs: 129 active+clean; 371 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.4 MiB/s wr, 46 op/s 2024-05-27T08:00:39.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:39 smithi186 ceph-mon[36647]: pgmap v725: 129 pgs: 129 active+clean; 371 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.4 MiB/s wr, 46 op/s 2024-05-27T08:00:39.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:39 smithi114 ceph-mon[32038]: pgmap v725: 129 pgs: 129 active+clean; 371 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.4 MiB/s wr, 46 op/s 2024-05-27T08:00:41.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:41 smithi186 ceph-mon[36647]: pgmap v726: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.3 MiB/s wr, 42 op/s 2024-05-27T08:00:41.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:41 smithi114 ceph-mon[32038]: pgmap v726: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.3 MiB/s wr, 42 op/s 2024-05-27T08:00:41.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T08:00:41.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:41 smithi163 ceph-mon[36451]: pgmap v726: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.3 MiB/s wr, 42 op/s 2024-05-27T08:00:42.079 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:00:43.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:43 smithi114 ceph-mon[32038]: pgmap v727: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.1 MiB/s wr, 34 op/s 2024-05-27T08:00:43.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.819+0000 7fb71ce98700 1 -- 172.21.15.114:0/4029761532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb718107e20 msgr2=0x7fb718108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:43.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.819+0000 7fb71ce98700 1 --2- 172.21.15.114:0/4029761532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb718107e20 0x7fb718108230 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fb7000099c0 tx=0x7fb700009cd0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.819+0000 7fb71ce98700 1 -- 172.21.15.114:0/4029761532 shutdown_connections 2024-05-27T08:00:43.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.819+0000 7fb71ce98700 1 --2- 172.21.15.114:0/4029761532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb718108770 0x7fb7181100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.819+0000 7fb71ce98700 1 --2- 172.21.15.114:0/4029761532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb718103540 0x7fb718103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.819+0000 7fb71ce98700 1 --2- 172.21.15.114:0/4029761532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb718107e20 0x7fb718108230 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.819+0000 7fb71ce98700 1 -- 172.21.15.114:0/4029761532 >> 172.21.15.114:0/4029761532 conn(0x7fb718078df0 msgr2=0x7fb718079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:43.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.819+0000 7fb71ce98700 1 -- 172.21.15.114:0/4029761532 shutdown_connections 2024-05-27T08:00:43.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.819+0000 7fb71ce98700 1 -- 172.21.15.114:0/4029761532 wait complete. 2024-05-27T08:00:43.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.820+0000 7fb71ce98700 1 Processor -- start 2024-05-27T08:00:43.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.820+0000 7fb71ce98700 1 -- start start 2024-05-27T08:00:43.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb71ce98700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb718103540 0x7fb718199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb71ce98700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb718107e20 0x7fb71819a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb71ce98700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb718108770 0x7fb71819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb71ce98700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb718113380 con 0x7fb718107e20 2024-05-27T08:00:43.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb71ce98700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb718113200 con 0x7fb718103540 2024-05-27T08:00:43.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb71ce98700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb718113500 con 0x7fb718108770 2024-05-27T08:00:43.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb70dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb718107e20 0x7fb71819a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb70dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb718107e20 0x7fb71819a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42742/0 (socket says 172.21.15.114:42742) 2024-05-27T08:00:43.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb70dd9b700 1 -- 172.21.15.114:0/1527221917 learned_addr learned my addr 172.21.15.114:0/1527221917 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:43.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb716d9d700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb718108770 0x7fb71819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb71659c700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb718103540 0x7fb718199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb70dd9b700 1 -- 172.21.15.114:0/1527221917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb718108770 msgr2=0x7fb71819e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:43.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb70dd9b700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb718108770 0x7fb71819e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb70dd9b700 1 -- 172.21.15.114:0/1527221917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb718103540 msgr2=0x7fb718199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:43.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb70dd9b700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb718103540 0x7fb718199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb70dd9b700 1 -- 172.21.15.114:0/1527221917 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71819eca0 con 0x7fb718107e20 2024-05-27T08:00:43.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb71659c700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb718103540 0x7fb718199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T08:00:43.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.821+0000 7fb70dd9b700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb718107e20 0x7fb71819a370 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fb7000045e0 tx=0x7fb700004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:43.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.822+0000 7fb70ffff700 1 -- 172.21.15.114:0/1527221917 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700028070 con 0x7fb718107e20 2024-05-27T08:00:43.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.822+0000 7fb71ce98700 1 -- 172.21.15.114:0/1527221917 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb71819ef30 con 0x7fb718107e20 2024-05-27T08:00:43.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.822+0000 7fb70ffff700 1 -- 172.21.15.114:0/1527221917 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb700019600 con 0x7fb718107e20 2024-05-27T08:00:43.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.822+0000 7fb70ffff700 1 -- 172.21.15.114:0/1527221917 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700009e60 con 0x7fb718107e20 2024-05-27T08:00:43.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.822+0000 7fb71ce98700 1 -- 172.21.15.114:0/1527221917 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7180759e0 con 0x7fb718107e20 2024-05-27T08:00:43.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.824+0000 7fb70ffff700 1 -- 172.21.15.114:0/1527221917 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fb70002b070 con 0x7fb718107e20 2024-05-27T08:00:43.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.825+0000 7fb70ffff700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb7040b29f0 0x7fb7040b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.825+0000 7fb71659c700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb7040b29f0 0x7fb7040b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.825+0000 7fb70ffff700 1 -- 172.21.15.114:0/1527221917 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 9466+0+0 (secure 0 0 0) 0x7fb7000e5590 con 0x7fb718107e20 2024-05-27T08:00:43.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.825+0000 7fb72b8adb80 1 Processor -- start 2024-05-27T08:00:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.825+0000 7fb71659c700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb7040b29f0 0x7fb7040b4f00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb708005950 tx=0x7fb7080058e0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.825+0000 7fb72b8adb80 1 -- start start 2024-05-27T08:00:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb72b8adb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6b08e40 0x5591c6c8db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb72b8adb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6c8e070 0x5591c6c87be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb72b8adb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c88530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb72b8adb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5591c6bb5d70 con 0x5591c6c88120 2024-05-27T08:00:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb72b8adb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5591c6ba5910 con 0x5591c6c8e070 2024-05-27T08:00:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb72b8adb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5591c6ba9950 con 0x5591c6b08e40 2024-05-27T08:00:43.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb716d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c88530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb716d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c88530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42744/0 (socket says 172.21.15.114:42744) 2024-05-27T08:00:43.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb716d9d700 1 -- 172.21.15.114:0/3609563701 learned_addr learned my addr 172.21.15.114:0/3609563701 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:43.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb70dd9b700 1 --2- 172.21.15.114:0/3609563701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6c8e070 0x5591c6c87be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb71659c700 1 --2- 172.21.15.114:0/3609563701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6b08e40 0x5591c6c8db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb716d9d700 1 -- 172.21.15.114:0/3609563701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6b08e40 msgr2=0x5591c6c8db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:43.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb716d9d700 1 --2- 172.21.15.114:0/3609563701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6b08e40 0x5591c6c8db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb716d9d700 1 -- 172.21.15.114:0/3609563701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6c8e070 msgr2=0x5591c6c87be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:43.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb716d9d700 1 --2- 172.21.15.114:0/3609563701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6c8e070 0x5591c6c87be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb716d9d700 1 -- 172.21.15.114:0/3609563701 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591c6b4efb0 con 0x5591c6c88120 2024-05-27T08:00:43.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb71659c700 1 --2- 172.21.15.114:0/3609563701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6b08e40 0x5591c6c8db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T08:00:43.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.826+0000 7fb716d9d700 1 --2- 172.21.15.114:0/3609563701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c88530 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fb710011680 tx=0x7fb710009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746a0e17ebe01a98 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:43.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.827+0000 7fb70cd99700 1 -- 172.21.15.114:0/3609563701 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb710019030 con 0x5591c6c88120 2024-05-27T08:00:43.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.827+0000 7fb70cd99700 1 -- 172.21.15.114:0/3609563701 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7100092e0 con 0x5591c6c88120 2024-05-27T08:00:43.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.827+0000 7fb70cd99700 1 -- 172.21.15.114:0/3609563701 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb710007500 con 0x5591c6c88120 2024-05-27T08:00:43.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.828+0000 7fb72b8adb80 1 -- 172.21.15.114:0/3609563701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 msgr2=0x5591c6c88530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.828+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/3609563701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c88530 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fb710011680 tx=0x7fb710009920 comp rx=0 tx=0).stop 2024-05-27T08:00:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.828+0000 7fb72b8adb80 1 -- 172.21.15.114:0/3609563701 shutdown_connections 2024-05-27T08:00:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.828+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/3609563701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c88530 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.828+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/3609563701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6b08e40 0x5591c6c8db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.828+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/3609563701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6c8e070 0x5591c6c87be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.828+0000 7fb72b8adb80 1 -- 172.21.15.114:0/3609563701 >> 172.21.15.114:0/3609563701 conn(0x5591c6b0b770 msgr2=0x5591c6c82d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.828+0000 7fb72b8adb80 1 -- 172.21.15.114:0/3609563701 shutdown_connections 2024-05-27T08:00:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.828+0000 7fb72b8adb80 1 -- 172.21.15.114:0/3609563701 wait complete. 2024-05-27T08:00:43.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.828+0000 7fb72b8adb80 1 -- start start 2024-05-27T08:00:43.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.828+0000 7fb72b8adb80 1 Processor -- start 2024-05-27T08:00:43.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb72b8adb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6b08e40 0x5591c6c91810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb72b8adb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c91d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb72b8adb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c8e070 0x5591c6c92290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb72b8adb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5591c6aabfa0 con 0x5591c6c88120 2024-05-27T08:00:43.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb72b8adb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5591c6b0dda0 con 0x5591c6b08e40 2024-05-27T08:00:43.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb72b8adb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5591c6c94b60 con 0x5591c6c8e070 2024-05-27T08:00:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb72b8adb80 10 client.0 _ll_register_callbacks cb 0x5591c6b3c5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T08:00:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb70dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c91d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb70dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c91d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42754/0 (socket says 172.21.15.114:42754) 2024-05-27T08:00:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb70dd9b700 1 -- 172.21.15.114:0/1829244501 learned_addr learned my addr 172.21.15.114:0/1829244501 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb716d9d700 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c8e070 0x5591c6c92290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb71659c700 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6b08e40 0x5591c6c91810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb70dd9b700 1 -- 172.21.15.114:0/1829244501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c8e070 msgr2=0x5591c6c92290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb70dd9b700 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c8e070 0x5591c6c92290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb70dd9b700 1 -- 172.21.15.114:0/1829244501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6b08e40 msgr2=0x5591c6c91810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb70dd9b700 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6b08e40 0x5591c6c91810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb70dd9b700 1 -- 172.21.15.114:0/1829244501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591c6af58d0 con 0x5591c6c88120 2024-05-27T08:00:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb716d9d700 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c8e070 0x5591c6c92290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T08:00:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb70dd9b700 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c91d50 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fb7000a97c0 tx=0x7fb7000abfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f1d4ea03db9fc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb6fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T08:00:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb6fdffb700 1 -- 172.21.15.114:0/1829244501 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7000ac550 con 0x5591c6c88120 2024-05-27T08:00:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb6fdffb700 1 -- 172.21.15.114:0/1829244501 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(169..169 src has 1..169) ==== 9466+0+0 (secure 0 0 0) 0x7fb7000ac9c0 con 0x5591c6c88120 2024-05-27T08:00:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.829+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1829244501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7100092e0 con 0x5591c6c88120 2024-05-27T08:00:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.830+0000 7fb6fdffb700 1 -- 172.21.15.114:0/1829244501 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7fb7000b0900 con 0x5591c6c88120 2024-05-27T08:00:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.830+0000 7fb72b8adb80 10 client.19980 fetch_fsmap learned FSMap version 118 2024-05-27T08:00:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.830+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1829244501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=4+}) -- 0x5591c6b5f760 con 0x5591c6c88120 2024-05-27T08:00:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.830+0000 7fb6fdffb700 1 -- 172.21.15.114:0/1829244501 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7100092e0 con 0x5591c6c88120 2024-05-27T08:00:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.830+0000 7fb6fdffb700 1 -- 172.21.15.114:0/1829244501 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 118) ==== 2885+0+0 (secure 0 0 0) 0x7fb7000ab7b0 con 0x5591c6c88120 2024-05-27T08:00:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.830+0000 7fb72b8adb80 10 client.19980 fetch_fsmap finished waiting for FSMap version 118 2024-05-27T08:00:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.830+0000 7fb72b8adb80 10 client.19980 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T08:00:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.830+0000 7fb72b8adb80 20 client.19980 populate_metadata read hostname 'smithi114' 2024-05-27T08:00:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.830+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591c6ca1b50 0x5591c6ca3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.830+0000 7fb72b8adb80 4 client.19980 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T08:00:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.830+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1829244501 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591c6bb7cc0 con 0x5591c6ca1b50 2024-05-27T08:00:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.830+0000 7fb71659c700 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591c6ca1b50 0x5591c6ca3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.831+0000 7fb6fdffb700 10 client.19980 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.831+0000 7fb71659c700 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591c6ca1b50 0x5591c6ca3fa0 crc :-1 s=READY pgs=534 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f1c56603e68b49 server_cookie=9901164448b7e5ca in_seq=0 out_seq=0 2024-05-27T08:00:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb6fdffb700 1 -- 172.21.15.114:0/1829244501 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5591c6bb7cc0 con 0x5591c6ca1b50 2024-05-27T08:00:43.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb6fdffb700 10 client.19980 handle_command_reply: tid=0 2024-05-27T08:00:43.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb72b8adb80 1 client.19980 shutdown 2024-05-27T08:00:43.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1829244501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 msgr2=0x5591c6c91d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:43.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c91d50 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fb7000a97c0 tx=0x7fb7000abfc0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1829244501 shutdown_connections 2024-05-27T08:00:43.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c8e070 0x5591c6c92290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591c6ca1b50 0x5591c6ca3fa0 crc :-1 s=READY pgs=534 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6b08e40 0x5591c6c91810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/1829244501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c91d50 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1829244501 >> 172.21.15.114:0/1829244501 conn(0x5591c6b0b770 msgr2=0x5591c6c86bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:43.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb6fdffb700 0 client.19980 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:43.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1829244501 shutdown_connections 2024-05-27T08:00:43.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1829244501 wait complete. 2024-05-27T08:00:43.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.832+0000 7fb72b8adb80 20 client.19980 trim_cache size 0 max 16384 2024-05-27T08:00:43.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.852+0000 7fb72b8adb80 1 Processor -- start 2024-05-27T08:00:43.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.852+0000 7fb72b8adb80 1 -- start start 2024-05-27T08:00:43.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb72b8adb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6b08e40 0x5591c6c87be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb72b8adb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c90b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb72b8adb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c8e070 0x5591c6ca3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb72b8adb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb7181ad160 con 0x5591c6c88120 2024-05-27T08:00:43.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb72b8adb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb7181ad830 con 0x5591c6b08e40 2024-05-27T08:00:43.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb72b8adb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb7181ac1c0 con 0x5591c6c8e070 2024-05-27T08:00:43.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb70dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c90b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb70dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c90b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42760/0 (socket says 172.21.15.114:42760) 2024-05-27T08:00:43.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb70dd9b700 1 -- 172.21.15.114:0/3760565736 learned_addr learned my addr 172.21.15.114:0/3760565736 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:43.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb716d9d700 1 --2- 172.21.15.114:0/3760565736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c8e070 0x5591c6ca3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb71659c700 1 --2- 172.21.15.114:0/3760565736 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6b08e40 0x5591c6c87be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb70dd9b700 1 -- 172.21.15.114:0/3760565736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c8e070 msgr2=0x5591c6ca3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:43.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb70dd9b700 1 --2- 172.21.15.114:0/3760565736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c8e070 0x5591c6ca3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb70dd9b700 1 -- 172.21.15.114:0/3760565736 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6b08e40 msgr2=0x5591c6c87be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:43.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb70dd9b700 1 --2- 172.21.15.114:0/3760565736 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6b08e40 0x5591c6c87be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb70dd9b700 1 -- 172.21.15.114:0/3760565736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591c6c8fda0 con 0x5591c6c88120 2024-05-27T08:00:43.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb71659c700 1 --2- 172.21.15.114:0/3760565736 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6b08e40 0x5591c6c87be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T08:00:43.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb70dd9b700 1 --2- 172.21.15.114:0/3760565736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c90b50 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fb7000a85d0 tx=0x7fb7000a8500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94bc2a69481aa31b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:43.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb6fffff700 1 -- 172.21.15.114:0/3760565736 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7000ac550 con 0x5591c6c88120 2024-05-27T08:00:43.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb6fffff700 1 -- 172.21.15.114:0/3760565736 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7000b5320 con 0x5591c6c88120 2024-05-27T08:00:43.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.853+0000 7fb6fffff700 1 -- 172.21.15.114:0/3760565736 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7000e5180 con 0x5591c6c88120 2024-05-27T08:00:43.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.854+0000 7fb72b8adb80 1 -- 172.21.15.114:0/3760565736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 msgr2=0x5591c6c90b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:43.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.854+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/3760565736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c90b50 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fb7000a85d0 tx=0x7fb7000a8500 comp rx=0 tx=0).stop 2024-05-27T08:00:43.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.854+0000 7fb72b8adb80 1 -- 172.21.15.114:0/3760565736 shutdown_connections 2024-05-27T08:00:43.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.854+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/3760565736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c8e070 0x5591c6ca3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.854+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/3760565736 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6b08e40 0x5591c6c87be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.854+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/3760565736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6c88120 0x5591c6c90b50 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.854+0000 7fb72b8adb80 1 -- 172.21.15.114:0/3760565736 >> 172.21.15.114:0/3760565736 conn(0x5591c6b0b770 msgr2=0x5591c6c7c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:43.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.854+0000 7fb72b8adb80 1 -- 172.21.15.114:0/3760565736 shutdown_connections 2024-05-27T08:00:43.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.854+0000 7fb72b8adb80 1 -- 172.21.15.114:0/3760565736 wait complete. 2024-05-27T08:00:43.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.855+0000 7fb72b8adb80 1 -- start start 2024-05-27T08:00:43.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.855+0000 7fb72b8adb80 1 Processor -- start 2024-05-27T08:00:43.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.855+0000 7fb72b8adb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6b08e40 0x5591c6c99df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.855+0000 7fb72b8adb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c88120 0x5591c6c9a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.855+0000 7fb72b8adb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6c8e070 0x5591c6c96dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.855+0000 7fb72b8adb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5591c6c8ce40 con 0x5591c6b08e40 2024-05-27T08:00:43.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.855+0000 7fb72b8adb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5591c6c8bb00 con 0x5591c6c8e070 2024-05-27T08:00:43.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.855+0000 7fb72b8adb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5591c6c8cfc0 con 0x5591c6c88120 2024-05-27T08:00:43.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.855+0000 7fb72b8adb80 10 client.0 _ll_register_callbacks cb 0x5591c6b3c5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T08:00:43.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.855+0000 7fb71659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6b08e40 0x5591c6c99df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.855+0000 7fb71659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6b08e40 0x5591c6c99df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42766/0 (socket says 172.21.15.114:42766) 2024-05-27T08:00:43.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.855+0000 7fb71659c700 1 -- 172.21.15.114:0/1210023234 learned_addr learned my addr 172.21.15.114:0/1210023234 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:43.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb716d9d700 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6c8e070 0x5591c6c96dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb70dd9b700 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c88120 0x5591c6c9a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb71659c700 1 -- 172.21.15.114:0/1210023234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c88120 msgr2=0x5591c6c9a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:43.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb71659c700 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c88120 0x5591c6c9a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb71659c700 1 -- 172.21.15.114:0/1210023234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6c8e070 msgr2=0x5591c6c96dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:43.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb71659c700 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6c8e070 0x5591c6c96dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb71659c700 1 -- 172.21.15.114:0/1210023234 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591c6c9aa10 con 0x5591c6b08e40 2024-05-27T08:00:43.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb716d9d700 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6c8e070 0x5591c6c96dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T08:00:43.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb71659c700 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6b08e40 0x5591c6c99df0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x5591c6c91090 tx=0x7fb708012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1fd126673be974 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:43.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb6fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T08:00:43.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb6fdffb700 1 -- 172.21.15.114:0/1210023234 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708003c30 con 0x5591c6b08e40 2024-05-27T08:00:43.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb6fdffb700 1 -- 172.21.15.114:0/1210023234 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(169..169 src has 1..169) ==== 9466+0+0 (secure 0 0 0) 0x7fb708003dd0 con 0x5591c6b08e40 2024-05-27T08:00:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1210023234 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7000b5320 con 0x5591c6b08e40 2024-05-27T08:00:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb6fdffb700 1 -- 172.21.15.114:0/1210023234 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7fb708039050 con 0x5591c6b08e40 2024-05-27T08:00:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb72b8adb80 10 client.19986 fetch_fsmap learned FSMap version 118 2024-05-27T08:00:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.856+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1210023234 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=4+}) -- 0x5591c6d0cbc0 con 0x5591c6b08e40 2024-05-27T08:00:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.857+0000 7fb6fdffb700 1 -- 172.21.15.114:0/1210023234 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7000b5320 con 0x5591c6b08e40 2024-05-27T08:00:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.857+0000 7fb6fdffb700 1 -- 172.21.15.114:0/1210023234 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 118) ==== 2885+0+0 (secure 0 0 0) 0x7fb7080315b0 con 0x5591c6b08e40 2024-05-27T08:00:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.857+0000 7fb72b8adb80 10 client.19986 fetch_fsmap finished waiting for FSMap version 118 2024-05-27T08:00:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.857+0000 7fb72b8adb80 10 client.19986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T08:00:43.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.857+0000 7fb72b8adb80 20 client.19986 populate_metadata read hostname 'smithi114' 2024-05-27T08:00:43.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.857+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591c6d0d2b0 0x5591c6d0f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:43.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.857+0000 7fb72b8adb80 4 client.19986 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T08:00:43.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.857+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1210023234 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5591c6bb14a0 con 0x5591c6d0d2b0 2024-05-27T08:00:43.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.857+0000 7fb70dd9b700 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591c6d0d2b0 0x5591c6d0f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:43.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb6fdffb700 10 client.19986 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:43.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb70dd9b700 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591c6d0d2b0 0x5591c6d0f6d0 crc :-1 s=READY pgs=535 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=244ec0144b0e6a52 server_cookie=bca7785a3b01e85d in_seq=0 out_seq=0 2024-05-27T08:00:43.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb6fdffb700 1 -- 172.21.15.114:0/1210023234 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5591c6bb14a0 con 0x5591c6d0d2b0 2024-05-27T08:00:43.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb6fdffb700 10 client.19986 handle_command_reply: tid=0 2024-05-27T08:00:43.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb72b8adb80 1 client.19986 shutdown 2024-05-27T08:00:43.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1210023234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6b08e40 msgr2=0x5591c6c99df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:43.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6b08e40 0x5591c6c99df0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x5591c6c91090 tx=0x7fb708012000 comp rx=0 tx=0).stop 2024-05-27T08:00:43.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1210023234 shutdown_connections 2024-05-27T08:00:43.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x5591c6d0d2b0 0x5591c6d0f6d0 crc :-1 s=READY pgs=535 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5591c6c8e070 0x5591c6c96dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5591c6b08e40 0x5591c6c99df0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb72b8adb80 1 --2- 172.21.15.114:0/1210023234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591c6c88120 0x5591c6c9a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb6fdffb700 0 client.19986 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:43.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1210023234 >> 172.21.15.114:0/1210023234 conn(0x5591c6b0b770 msgr2=0x5591c6c908b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:43.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1210023234 shutdown_connections 2024-05-27T08:00:43.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb72b8adb80 1 -- 172.21.15.114:0/1210023234 wait complete. 2024-05-27T08:00:43.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.858+0000 7fb72b8adb80 20 client.19986 trim_cache size 0 max 16384 2024-05-27T08:00:43.868 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T08:00:43.869 INFO:teuthology.orchestra.run.smithi114.stdout: "status": "no active scrubs running", 2024-05-27T08:00:43.869 INFO:teuthology.orchestra.run.smithi114.stdout: "scrubs": {} 2024-05-27T08:00:43.869 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T08:00:43.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.859+0000 7fb71ce98700 1 -- 172.21.15.114:0/1527221917 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb7040b29f0 msgr2=0x7fb7040b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:43.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.859+0000 7fb71ce98700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb7040b29f0 0x7fb7040b4f00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb708005950 tx=0x7fb7080058e0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.859+0000 7fb71ce98700 1 -- 172.21.15.114:0/1527221917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb718107e20 msgr2=0x7fb71819a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:43.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.859+0000 7fb71ce98700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb718107e20 0x7fb71819a370 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fb7000045e0 tx=0x7fb700004610 comp rx=0 tx=0).stop 2024-05-27T08:00:43.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.859+0000 7fb71ce98700 1 -- 172.21.15.114:0/1527221917 shutdown_connections 2024-05-27T08:00:43.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.859+0000 7fb71ce98700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb718108770 0x7fb71819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.859+0000 7fb71ce98700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb7040b29f0 0x7fb7040b4f00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.859+0000 7fb71ce98700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb718103540 0x7fb718199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.859+0000 7fb71ce98700 1 --2- 172.21.15.114:0/1527221917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb718107e20 0x7fb71819a370 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:43.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.859+0000 7fb71ce98700 1 -- 172.21.15.114:0/1527221917 >> 172.21.15.114:0/1527221917 conn(0x7fb718078df0 msgr2=0x7fb718100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:43.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.859+0000 7fb71ce98700 1 -- 172.21.15.114:0/1527221917 shutdown_connections 2024-05-27T08:00:43.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:43.859+0000 7fb71ce98700 1 -- 172.21.15.114:0/1527221917 wait complete. 2024-05-27T08:00:43.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:43 smithi163 ceph-mon[36451]: pgmap v727: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.1 MiB/s wr, 34 op/s 2024-05-27T08:00:43.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:43 smithi186 ceph-mon[36647]: pgmap v727: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.1 MiB/s wr, 34 op/s 2024-05-27T08:00:44.584 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T08:00:44.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T08:00:44.804 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:44 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-05-27T08:00:44.805 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:44 smithi114 systemd[1]: Stopped Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:00:44.805 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:44 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.511s CPU time. 2024-05-27T08:00:44.805 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:44 smithi114 systemd[1]: Starting Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:00:45.058 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:44 smithi114 podman[136603]: 2024-05-27 08:00:44.705971487 +0000 UTC m=+0.018618443 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:00:45.058 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:44 smithi114 podman[136603]: 2024-05-27 08:00:44.836951839 +0000 UTC m=+0.149598784 container create c26124751fd7d3e11983b8da66c67ab74132ca23e24f26397434c2208ebfd2cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-05-27T08:00:45.263 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:00:45.386 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:45 smithi114 podman[136603]: 2024-05-27 08:00:45.098206275 +0000 UTC m=+0.410853233 container init c26124751fd7d3e11983b8da66c67ab74132ca23e24f26397434c2208ebfd2cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T08:00:45.387 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:45 smithi114 podman[136603]: 2024-05-27 08:00:45.101066514 +0000 UTC m=+0.413713457 container start c26124751fd7d3e11983b8da66c67ab74132ca23e24f26397434c2208ebfd2cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-27T08:00:45.387 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:45 smithi114 ceph-mds[136649]: -- 172.21.15.114:0/2562531771 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56117a599a40 con 0x56117a5e1a80 2024-05-27T08:00:45.387 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:45 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: starting mds.d at 2024-05-27T08:00:45.387 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:45 smithi114 bash[136603]: c26124751fd7d3e11983b8da66c67ab74132ca23e24f26397434c2208ebfd2cb 2024-05-27T08:00:45.387 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:45 smithi114 systemd[1]: Started Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:00:45.679 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:45 smithi114 ceph-mon[32038]: pgmap v728: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 902 KiB/s wr, 28 op/s 2024-05-27T08:00:45.679 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:45 smithi114 ceph-mon[32038]: daemon mds.d restarted 2024-05-27T08:00:45.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:45 smithi163 ceph-mon[36451]: pgmap v728: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 902 KiB/s wr, 28 op/s 2024-05-27T08:00:45.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:45 smithi163 ceph-mon[36451]: daemon mds.d restarted 2024-05-27T08:00:45.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:45 smithi186 ceph-mon[36647]: pgmap v728: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 902 KiB/s wr, 28 op/s 2024-05-27T08:00:45.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:45 smithi186 ceph-mon[36647]: daemon mds.d restarted 2024-05-27T08:00:46.429 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb0b096c700 time 2024-05-27T08:00:46.088712+0000 2024-05-27T08:00:46.429 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:46.430 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:46.430 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb0c31b1ba6] 2024-05-27T08:00:46.430 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0c31b1dd1] 2024-05-27T08:00:46.430 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56117933023d] 2024-05-27T08:00:46.430 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56117931b971] 2024-05-27T08:00:46.430 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 5: (SnapRealm::check_cache() const+0x13e) [0x561179319fa0] 2024-05-27T08:00:46.430 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56117931a651] 2024-05-27T08:00:46.431 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5611792af347] 2024-05-27T08:00:46.431 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561178fab87e] 2024-05-27T08:00:46.431 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561178fb0e94] 2024-05-27T08:00:46.431 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561178fbb969] 2024-05-27T08:00:46.431 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 11: (MDLog::_replay_thread()+0x1f38) [0x56117935448c] 2024-05-27T08:00:46.431 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56117935d9cb] 2024-05-27T08:00:46.431 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 13: (Thread::entry_wrapper()+0x43) [0x7fb0c318072d] 2024-05-27T08:00:46.431 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb0c3180749] 2024-05-27T08:00:46.432 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb0c1d241ca] 2024-05-27T08:00:46.432 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 16: clone() 2024-05-27T08:00:46.432 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.089+0000 7fb0b096c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb0b096c700 time 2024-05-27T08:00:46.088712+0000 2024-05-27T08:00:46.433 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:46.433 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.433 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:46.433 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb0c31b1ba6] 2024-05-27T08:00:46.433 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0c31b1dd1] 2024-05-27T08:00:46.433 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56117933023d] 2024-05-27T08:00:46.433 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56117931b971] 2024-05-27T08:00:46.433 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 5: (SnapRealm::check_cache() const+0x13e) [0x561179319fa0] 2024-05-27T08:00:46.434 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56117931a651] 2024-05-27T08:00:46.434 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5611792af347] 2024-05-27T08:00:46.434 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561178fab87e] 2024-05-27T08:00:46.434 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561178fb0e94] 2024-05-27T08:00:46.434 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561178fbb969] 2024-05-27T08:00:46.434 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 11: (MDLog::_replay_thread()+0x1f38) [0x56117935448c] 2024-05-27T08:00:46.434 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56117935d9cb] 2024-05-27T08:00:46.434 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 13: (Thread::entry_wrapper()+0x43) [0x7fb0c318072d] 2024-05-27T08:00:46.435 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb0c3180749] 2024-05-27T08:00:46.435 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb0c1d241ca] 2024-05-27T08:00:46.435 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 16: clone() 2024-05-27T08:00:46.435 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.435 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: *** Caught signal (Aborted) ** 2024-05-27T08:00:46.435 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: in thread 7fb0b096c700 thread_name:md_log_replay 2024-05-27T08:00:46.435 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:46.435 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb0c1d2ed20] 2024-05-27T08:00:46.436 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2: gsignal() 2024-05-27T08:00:46.436 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 3: abort() 2024-05-27T08:00:46.436 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb0c31b1ced] 2024-05-27T08:00:46.436 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0c31b1dd1] 2024-05-27T08:00:46.436 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56117933023d] 2024-05-27T08:00:46.436 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56117931b971] 2024-05-27T08:00:46.436 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 8: (SnapRealm::check_cache() const+0x13e) [0x561179319fa0] 2024-05-27T08:00:46.436 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56117931a651] 2024-05-27T08:00:46.437 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5611792af347] 2024-05-27T08:00:46.437 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561178fab87e] 2024-05-27T08:00:46.437 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561178fb0e94] 2024-05-27T08:00:46.437 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561178fbb969] 2024-05-27T08:00:46.437 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 14: (MDLog::_replay_thread()+0x1f38) [0x56117935448c] 2024-05-27T08:00:46.437 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56117935d9cb] 2024-05-27T08:00:46.437 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 16: (Thread::entry_wrapper()+0x43) [0x7fb0c318072d] 2024-05-27T08:00:46.437 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb0c3180749] 2024-05-27T08:00:46.437 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb0c1d241ca] 2024-05-27T08:00:46.438 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 19: clone() 2024-05-27T08:00:46.438 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.092+0000 7fb0b096c700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:00:46.438 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: in thread 7fb0b096c700 thread_name:md_log_replay 2024-05-27T08:00:46.438 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.438 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:46.438 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb0c1d2ed20] 2024-05-27T08:00:46.438 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2: gsignal() 2024-05-27T08:00:46.439 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 3: abort() 2024-05-27T08:00:46.439 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb0c31b1ced] 2024-05-27T08:00:46.439 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0c31b1dd1] 2024-05-27T08:00:46.439 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56117933023d] 2024-05-27T08:00:46.439 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56117931b971] 2024-05-27T08:00:46.439 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 8: (SnapRealm::check_cache() const+0x13e) [0x561179319fa0] 2024-05-27T08:00:46.439 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56117931a651] 2024-05-27T08:00:46.439 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5611792af347] 2024-05-27T08:00:46.440 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561178fab87e] 2024-05-27T08:00:46.440 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561178fb0e94] 2024-05-27T08:00:46.440 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561178fbb969] 2024-05-27T08:00:46.440 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 14: (MDLog::_replay_thread()+0x1f38) [0x56117935448c] 2024-05-27T08:00:46.440 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56117935d9cb] 2024-05-27T08:00:46.440 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 16: (Thread::entry_wrapper()+0x43) [0x7fb0c318072d] 2024-05-27T08:00:46.440 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb0c3180749] 2024-05-27T08:00:46.440 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb0c1d241ca] 2024-05-27T08:00:46.441 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 19: clone() 2024-05-27T08:00:46.441 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:00:46.441 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.441 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: -1> 2024-05-27T08:00:46.089+0000 7fb0b096c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb0b096c700 time 2024-05-27T08:00:46.088712+0000 2024-05-27T08:00:46.441 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:46.441 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.441 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:46.441 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb0c31b1ba6] 2024-05-27T08:00:46.442 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0c31b1dd1] 2024-05-27T08:00:46.442 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56117933023d] 2024-05-27T08:00:46.442 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56117931b971] 2024-05-27T08:00:46.442 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 5: (SnapRealm::check_cache() const+0x13e) [0x561179319fa0] 2024-05-27T08:00:46.442 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56117931a651] 2024-05-27T08:00:46.442 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5611792af347] 2024-05-27T08:00:46.442 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561178fab87e] 2024-05-27T08:00:46.443 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561178fb0e94] 2024-05-27T08:00:46.443 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561178fbb969] 2024-05-27T08:00:46.443 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 11: (MDLog::_replay_thread()+0x1f38) [0x56117935448c] 2024-05-27T08:00:46.443 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56117935d9cb] 2024-05-27T08:00:46.444 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 13: (Thread::entry_wrapper()+0x43) [0x7fb0c318072d] 2024-05-27T08:00:46.444 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb0c3180749] 2024-05-27T08:00:46.444 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb0c1d241ca] 2024-05-27T08:00:46.444 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 16: clone() 2024-05-27T08:00:46.444 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.444 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 0> 2024-05-27T08:00:46.092+0000 7fb0b096c700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:00:46.444 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: in thread 7fb0b096c700 thread_name:md_log_replay 2024-05-27T08:00:46.444 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.445 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:46.445 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb0c1d2ed20] 2024-05-27T08:00:46.445 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2: gsignal() 2024-05-27T08:00:46.445 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 3: abort() 2024-05-27T08:00:46.445 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb0c31b1ced] 2024-05-27T08:00:46.445 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0c31b1dd1] 2024-05-27T08:00:46.445 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56117933023d] 2024-05-27T08:00:46.445 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56117931b971] 2024-05-27T08:00:46.446 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 8: (SnapRealm::check_cache() const+0x13e) [0x561179319fa0] 2024-05-27T08:00:46.446 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56117931a651] 2024-05-27T08:00:46.446 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5611792af347] 2024-05-27T08:00:46.446 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561178fab87e] 2024-05-27T08:00:46.446 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561178fb0e94] 2024-05-27T08:00:46.446 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561178fbb969] 2024-05-27T08:00:46.446 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 14: (MDLog::_replay_thread()+0x1f38) [0x56117935448c] 2024-05-27T08:00:46.446 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56117935d9cb] 2024-05-27T08:00:46.447 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 16: (Thread::entry_wrapper()+0x43) [0x7fb0c318072d] 2024-05-27T08:00:46.447 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb0c3180749] 2024-05-27T08:00:46.447 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb0c1d241ca] 2024-05-27T08:00:46.447 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 19: clone() 2024-05-27T08:00:46.447 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:00:46.447 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.447 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: -1> 2024-05-27T08:00:46.089+0000 7fb0b096c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb0b096c700 time 2024-05-27T08:00:46.088712+0000 2024-05-27T08:00:46.447 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:00:46.448 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.448 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:46.448 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb0c31b1ba6] 2024-05-27T08:00:46.448 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0c31b1dd1] 2024-05-27T08:00:46.448 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56117933023d] 2024-05-27T08:00:46.448 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56117931b971] 2024-05-27T08:00:46.448 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 5: (SnapRealm::check_cache() const+0x13e) [0x561179319fa0] 2024-05-27T08:00:46.448 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56117931a651] 2024-05-27T08:00:46.449 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5611792af347] 2024-05-27T08:00:46.449 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561178fab87e] 2024-05-27T08:00:46.449 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561178fb0e94] 2024-05-27T08:00:46.449 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561178fbb969] 2024-05-27T08:00:46.449 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 11: (MDLog::_replay_thread()+0x1f38) [0x56117935448c] 2024-05-27T08:00:46.449 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56117935d9cb] 2024-05-27T08:00:46.449 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 13: (Thread::entry_wrapper()+0x43) [0x7fb0c318072d] 2024-05-27T08:00:46.449 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb0c3180749] 2024-05-27T08:00:46.449 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb0c1d241ca] 2024-05-27T08:00:46.450 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 16: clone() 2024-05-27T08:00:46.450 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.450 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 0> 2024-05-27T08:00:46.092+0000 7fb0b096c700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:00:46.450 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: in thread 7fb0b096c700 thread_name:md_log_replay 2024-05-27T08:00:46.450 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.450 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:00:46.450 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb0c1d2ed20] 2024-05-27T08:00:46.451 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2: gsignal() 2024-05-27T08:00:46.451 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 3: abort() 2024-05-27T08:00:46.451 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb0c31b1ced] 2024-05-27T08:00:46.451 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0c31b1dd1] 2024-05-27T08:00:46.451 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56117933023d] 2024-05-27T08:00:46.451 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56117931b971] 2024-05-27T08:00:46.451 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 8: (SnapRealm::check_cache() const+0x13e) [0x561179319fa0] 2024-05-27T08:00:46.451 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56117931a651] 2024-05-27T08:00:46.451 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5611792af347] 2024-05-27T08:00:46.452 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561178fab87e] 2024-05-27T08:00:46.452 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561178fb0e94] 2024-05-27T08:00:46.452 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561178fbb969] 2024-05-27T08:00:46.452 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 14: (MDLog::_replay_thread()+0x1f38) [0x56117935448c] 2024-05-27T08:00:46.452 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56117935d9cb] 2024-05-27T08:00:46.452 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 16: (Thread::entry_wrapper()+0x43) [0x7fb0c318072d] 2024-05-27T08:00:46.452 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb0c3180749] 2024-05-27T08:00:46.453 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb0c1d241ca] 2024-05-27T08:00:46.453 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 19: clone() 2024-05-27T08:00:46.453 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:00:46.453 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[136632]: 2024-05-27T08:00:46.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:46 smithi114 ceph-mon[32038]: osdmap e170: 12 total, 12 up, 12 in 2024-05-27T08:00:46.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:46 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6836/4188137421,v1:172.21.15.114:6837/4188137421] up:boot 2024-05-27T08:00:46.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:46 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 2 up:standby 2024-05-27T08:00:46.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:46 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:00:46.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:46 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:46.809 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:46 smithi114 podman[136738]: 2024-05-27 08:00:46.489204196 +0000 UTC m=+0.355526579 container died c26124751fd7d3e11983b8da66c67ab74132ca23e24f26397434c2208ebfd2cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:00:46.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:46 smithi163 ceph-mon[36451]: osdmap e170: 12 total, 12 up, 12 in 2024-05-27T08:00:46.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:46 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6836/4188137421,v1:172.21.15.114:6837/4188137421] up:boot 2024-05-27T08:00:46.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:46 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 2 up:standby 2024-05-27T08:00:46.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:46 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:00:46.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:46 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:46.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:46 smithi186 ceph-mon[36647]: osdmap e170: 12 total, 12 up, 12 in 2024-05-27T08:00:46.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:46 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6836/4188137421,v1:172.21.15.114:6837/4188137421] up:boot 2024-05-27T08:00:46.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:46 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 2 up:standby 2024-05-27T08:00:46.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:46 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:00:46.981 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:46 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:47.457 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:47 smithi114 podman[136738]: 2024-05-27 08:00:47.146819368 +0000 UTC m=+1.013141740 container remove c26124751fd7d3e11983b8da66c67ab74132ca23e24f26397434c2208ebfd2cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2) 2024-05-27T08:00:47.457 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:47 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T08:00:47.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.681+0000 7fb0b6e91700 1 -- 172.21.15.114:0/82550236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b0108750 msgr2=0x7fb0b01100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:47.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.681+0000 7fb0b6e91700 1 --2- 172.21.15.114:0/82550236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b0108750 0x7fb0b01100b0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800b0d0 tx=0x7fb0a800b3e0 comp rx=0 tx=0).stop 2024-05-27T08:00:47.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.681+0000 7fb0b6e91700 1 -- 172.21.15.114:0/82550236 shutdown_connections 2024-05-27T08:00:47.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.681+0000 7fb0b6e91700 1 --2- 172.21.15.114:0/82550236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b0108750 0x7fb0b01100b0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:47.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.681+0000 7fb0b6e91700 1 --2- 172.21.15.114:0/82550236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0b01034d0 0x7fb0b01038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:47.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.681+0000 7fb0b6e91700 1 --2- 172.21.15.114:0/82550236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0b0107e00 0x7fb0b0108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:47.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.681+0000 7fb0b6e91700 1 -- 172.21.15.114:0/82550236 >> 172.21.15.114:0/82550236 conn(0x7fb0b00fef20 msgr2=0x7fb0b0101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:47.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.681+0000 7fb0b6e91700 1 -- 172.21.15.114:0/82550236 shutdown_connections 2024-05-27T08:00:47.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.681+0000 7fb0b6e91700 1 -- 172.21.15.114:0/82550236 wait complete. 2024-05-27T08:00:47.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.682+0000 7fb0b6e91700 1 Processor -- start 2024-05-27T08:00:47.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.682+0000 7fb0b6e91700 1 -- start start 2024-05-27T08:00:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.682+0000 7fb0b6e91700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b01034d0 0x7fb0b019f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.682+0000 7fb0b6e91700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0b0107e00 0x7fb0b019fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.682+0000 7fb0b6e91700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0b0108750 0x7fb0b0199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:47.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.682+0000 7fb0b6e91700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b01133a0 con 0x7fb0b01034d0 2024-05-27T08:00:47.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.682+0000 7fb0b6e91700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b0113220 con 0x7fb0b0107e00 2024-05-27T08:00:47.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.682+0000 7fb0b6e91700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b0113520 con 0x7fb0b0108750 2024-05-27T08:00:47.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.682+0000 7fb0b4c2d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b01034d0 0x7fb0b019f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:47.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.682+0000 7fb0b4c2d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b01034d0 0x7fb0b019f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35848/0 (socket says 172.21.15.114:35848) 2024-05-27T08:00:47.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.682+0000 7fb0b4c2d700 1 -- 172.21.15.114:0/830082880 learned_addr learned my addr 172.21.15.114:0/830082880 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:47.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.683+0000 7fb0b542e700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0b0108750 0x7fb0b0199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:47.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.683+0000 7fb0affff700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0b0107e00 0x7fb0b019fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:47.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.683+0000 7fb0b4c2d700 1 -- 172.21.15.114:0/830082880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0b0108750 msgr2=0x7fb0b0199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:47.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.683+0000 7fb0b4c2d700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0b0108750 0x7fb0b0199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:47.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.683+0000 7fb0b4c2d700 1 -- 172.21.15.114:0/830082880 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0b0107e00 msgr2=0x7fb0b019fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:47.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.683+0000 7fb0b4c2d700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0b0107e00 0x7fb0b019fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:47.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.683+0000 7fb0b4c2d700 1 -- 172.21.15.114:0/830082880 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b019a150 con 0x7fb0b01034d0 2024-05-27T08:00:47.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.683+0000 7fb0b542e700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0b0108750 0x7fb0b0199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T08:00:47.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.683+0000 7fb0affff700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0b0107e00 0x7fb0b019fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T08:00:47.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.683+0000 7fb0b4c2d700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b01034d0 0x7fb0b019f880 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400d980 tx=0x7fb0a400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:47.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.683+0000 7fb0adffb700 1 -- 172.21.15.114:0/830082880 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4017070 con 0x7fb0b01034d0 2024-05-27T08:00:47.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.683+0000 7fb0b6e91700 1 -- 172.21.15.114:0/830082880 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b019a440 con 0x7fb0b01034d0 2024-05-27T08:00:47.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.683+0000 7fb0adffb700 1 -- 172.21.15.114:0/830082880 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0a4004830 con 0x7fb0b01034d0 2024-05-27T08:00:47.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.684+0000 7fb0adffb700 1 -- 172.21.15.114:0/830082880 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4005440 con 0x7fb0b01034d0 2024-05-27T08:00:47.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.684+0000 7fb0b6e91700 1 -- 172.21.15.114:0/830082880 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b006c030 con 0x7fb0b01034d0 2024-05-27T08:00:47.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.685+0000 7fb0b6e91700 1 -- 172.21.15.114:0/830082880 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b0104980 con 0x7fb0b01034d0 2024-05-27T08:00:47.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.686+0000 7fb0adffb700 1 -- 172.21.15.114:0/830082880 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fb0a40055e0 con 0x7fb0b01034d0 2024-05-27T08:00:47.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.687+0000 7fb0adffb700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0a00b29f0 0x7fb0a00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:47.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.687+0000 7fb0affff700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0a00b29f0 0x7fb0a00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:47.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.688+0000 7fb0adffb700 1 -- 172.21.15.114:0/830082880 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 9466+0+0 (secure 0 0 0) 0x7fb0a40e2db0 con 0x7fb0b01034d0 2024-05-27T08:00:47.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.688+0000 7fb0affff700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0a00b29f0 0x7fb0a00b4f00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb09c005fd0 tx=0x7fb09c005d40 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:47.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.691+0000 7fb0adffb700 1 -- 172.21.15.114:0/830082880 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb0a4020020 con 0x7fb0b01034d0 2024-05-27T08:00:47.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:47 smithi163 ceph-mon[36451]: pgmap v730: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 286 KiB/s wr, 11 op/s 2024-05-27T08:00:47.784 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:47 smithi114 ceph-mon[32038]: pgmap v730: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 286 KiB/s wr, 11 op/s 2024-05-27T08:00:47.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.947+0000 7fb0b6e91700 1 -- 172.21.15.114:0/830082880 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0b019afd0 con 0x7fb0b01034d0 2024-05-27T08:00:47.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.948+0000 7fb0adffb700 1 -- 172.21.15.114:0/830082880 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) ==== 95+0+4381 (secure 0 0 0) 0x7fb0a40aa380 con 0x7fb0b01034d0 2024-05-27T08:00:47.950 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T08:00:47.950 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":120,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T08:00:45.487795+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_19992":{"gid":19992,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6837/4188137421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":4188137421},{"type":"v1","addr":"172.21.15.114:6837","nonce":4188137421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T08:00:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.950+0000 7fb0b6e91700 1 -- 172.21.15.114:0/830082880 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0a00b29f0 msgr2=0x7fb0a00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.950+0000 7fb0b6e91700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0a00b29f0 0x7fb0a00b4f00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb09c005fd0 tx=0x7fb09c005d40 comp rx=0 tx=0).stop 2024-05-27T08:00:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.950+0000 7fb0b6e91700 1 -- 172.21.15.114:0/830082880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b01034d0 msgr2=0x7fb0b019f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.950+0000 7fb0b6e91700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b01034d0 0x7fb0b019f880 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400d980 tx=0x7fb0a400dd40 comp rx=0 tx=0).stop 2024-05-27T08:00:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.950+0000 7fb0b6e91700 1 -- 172.21.15.114:0/830082880 shutdown_connections 2024-05-27T08:00:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.950+0000 7fb0b6e91700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0b0108750 0x7fb0b0199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.950+0000 7fb0b6e91700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b01034d0 0x7fb0b019f880 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.950+0000 7fb0b6e91700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fb0a00b29f0 0x7fb0a00b4f00 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.950+0000 7fb0b6e91700 1 --2- 172.21.15.114:0/830082880 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0b0107e00 0x7fb0b019fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.950+0000 7fb0b6e91700 1 -- 172.21.15.114:0/830082880 >> 172.21.15.114:0/830082880 conn(0x7fb0b00fef20 msgr2=0x7fb0b0107540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.950+0000 7fb0b6e91700 1 -- 172.21.15.114:0/830082880 shutdown_connections 2024-05-27T08:00:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:47.950+0000 7fb0b6e91700 1 -- 172.21.15.114:0/830082880 wait complete. 2024-05-27T08:00:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 120 2024-05-27T08:00:47.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:47 smithi186 ceph-mon[36647]: pgmap v730: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 286 KiB/s wr, 11 op/s 2024-05-27T08:00:48.057 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:47 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T08:00:48.058 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:47 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.651s CPU time. 2024-05-27T08:00:48.757 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:48 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/830082880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T08:00:48.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T08:00:48.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:48 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/830082880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T08:00:48.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:48 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/830082880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T08:00:49.024 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:00:49.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:49 smithi114 ceph-mon[32038]: pgmap v731: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 162 KiB/s wr, 6 op/s 2024-05-27T08:00:49.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:49 smithi163 ceph-mon[36451]: pgmap v731: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 162 KiB/s wr, 6 op/s 2024-05-27T08:00:49.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:49 smithi186 ceph-mon[36647]: pgmap v731: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 162 KiB/s wr, 6 op/s 2024-05-27T08:00:50.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.731+0000 7f57b1349700 1 -- 172.21.15.114:0/2391868841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac108750 msgr2=0x7f57ac1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:50.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.731+0000 7f57b1349700 1 --2- 172.21.15.114:0/2391868841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac108750 0x7f57ac1100b0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f57a400b0d0 tx=0x7f57a400b3e0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.731+0000 7f57b1349700 1 -- 172.21.15.114:0/2391868841 shutdown_connections 2024-05-27T08:00:50.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.731+0000 7f57b1349700 1 --2- 172.21.15.114:0/2391868841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac108750 0x7f57ac1100b0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.731+0000 7f57b1349700 1 --2- 172.21.15.114:0/2391868841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57ac1034d0 0x7f57ac1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.731+0000 7f57b1349700 1 --2- 172.21.15.114:0/2391868841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57ac107e00 0x7f57ac108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.731+0000 7f57b1349700 1 -- 172.21.15.114:0/2391868841 >> 172.21.15.114:0/2391868841 conn(0x7f57ac0fef20 msgr2=0x7f57ac101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:50.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.731+0000 7f57b1349700 1 -- 172.21.15.114:0/2391868841 shutdown_connections 2024-05-27T08:00:50.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.731+0000 7f57b1349700 1 -- 172.21.15.114:0/2391868841 wait complete. 2024-05-27T08:00:50.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.732+0000 7f57b1349700 1 Processor -- start 2024-05-27T08:00:50.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.732+0000 7f57b1349700 1 -- start start 2024-05-27T08:00:50.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57b1349700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57ac1034d0 0x7f57ac19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57b1349700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac107e00 0x7f57ac19fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57b1349700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57ac108750 0x7f57ac199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57b1349700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac113380 con 0x7f57ac107e00 2024-05-27T08:00:50.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57b1349700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac113200 con 0x7f57ac1034d0 2024-05-27T08:00:50.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57b1349700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac113500 con 0x7f57ac108750 2024-05-27T08:00:50.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57aa7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac107e00 0x7f57ac19fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57aa7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac107e00 0x7f57ac19fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35868/0 (socket says 172.21.15.114:35868) 2024-05-27T08:00:50.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57aa7fc700 1 -- 172.21.15.114:0/4171371825 learned_addr learned my addr 172.21.15.114:0/4171371825 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:50.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57ab7fe700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57ac108750 0x7f57ac199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57aaffd700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57ac1034d0 0x7f57ac19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57aa7fc700 1 -- 172.21.15.114:0/4171371825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57ac108750 msgr2=0x7f57ac199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:50.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57aa7fc700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57ac108750 0x7f57ac199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57aa7fc700 1 -- 172.21.15.114:0/4171371825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57ac1034d0 msgr2=0x7f57ac19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:50.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57aa7fc700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57ac1034d0 0x7f57ac19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.733+0000 7f57aa7fc700 1 -- 172.21.15.114:0/4171371825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57ac19a0c0 con 0x7f57ac107e00 2024-05-27T08:00:50.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.734+0000 7f57aa7fc700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac107e00 0x7f57ac19fda0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f57a000c9e0 tx=0x7f57a000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:50.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.734+0000 7f5793fff700 1 -- 172.21.15.114:0/4171371825 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0007aa0 con 0x7f57ac107e00 2024-05-27T08:00:50.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.734+0000 7f57b1349700 1 -- 172.21.15.114:0/4171371825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57ac19a3b0 con 0x7f57ac107e00 2024-05-27T08:00:50.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.734+0000 7f5793fff700 1 -- 172.21.15.114:0/4171371825 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a0007c40 con 0x7f57ac107e00 2024-05-27T08:00:50.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.734+0000 7f5793fff700 1 -- 172.21.15.114:0/4171371825 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0018660 con 0x7f57ac107e00 2024-05-27T08:00:50.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.734+0000 7f57b1349700 1 -- 172.21.15.114:0/4171371825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57ac1a66f0 con 0x7f57ac107e00 2024-05-27T08:00:50.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.737+0000 7f5793fff700 1 -- 172.21.15.114:0/4171371825 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f57a0018800 con 0x7f57ac107e00 2024-05-27T08:00:50.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.737+0000 7f5793fff700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f57940b29f0 0x7f57940b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.737+0000 7f57aaffd700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f57940b29f0 0x7f57940b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.737+0000 7f5793fff700 1 -- 172.21.15.114:0/4171371825 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 9466+0+0 (secure 0 0 0) 0x7f57a00e2910 con 0x7f57ac107e00 2024-05-27T08:00:50.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57bfd5eb80 1 Processor -- start 2024-05-27T08:00:50.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57aaffd700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f57940b29f0 0x7f57940b4f00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f579c005950 tx=0x7f579c005ed0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:50.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57bfd5eb80 1 -- start start 2024-05-27T08:00:50.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57bfd5eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc42f7be40 0x55bc430fabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57bfd5eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc430fb120 0x55bc430fb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57bfd5eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 0x55bc4310b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57bfd5eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bc43028d70 con 0x55bc43109260 2024-05-27T08:00:50.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57bfd5eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bc43018910 con 0x55bc42f7be40 2024-05-27T08:00:50.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57bfd5eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bc4301c950 con 0x55bc430fb120 2024-05-27T08:00:50.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57ab7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 0x55bc4310b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57ab7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 0x55bc4310b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35884/0 (socket says 172.21.15.114:35884) 2024-05-27T08:00:50.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57ab7fe700 1 -- 172.21.15.114:0/3713926384 learned_addr learned my addr 172.21.15.114:0/3713926384 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:50.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57aa7fc700 1 --2- 172.21.15.114:0/3713926384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc430fb120 0x55bc430fb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.738+0000 7f57aaffd700 1 --2- 172.21.15.114:0/3713926384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc42f7be40 0x55bc430fabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.739+0000 7f57ab7fe700 1 -- 172.21.15.114:0/3713926384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc430fb120 msgr2=0x55bc430fb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:50.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.739+0000 7f57ab7fe700 1 --2- 172.21.15.114:0/3713926384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc430fb120 0x55bc430fb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.739+0000 7f57ab7fe700 1 -- 172.21.15.114:0/3713926384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc42f7be40 msgr2=0x55bc430fabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:50.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.739+0000 7f57ab7fe700 1 --2- 172.21.15.114:0/3713926384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc42f7be40 0x55bc430fabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.739+0000 7f57ab7fe700 1 -- 172.21.15.114:0/3713926384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc42fc1fb0 con 0x55bc43109260 2024-05-27T08:00:50.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.739+0000 7f57aa7fc700 1 --2- 172.21.15.114:0/3713926384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc430fb120 0x55bc430fb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T08:00:50.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.739+0000 7f57aaffd700 1 --2- 172.21.15.114:0/3713926384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc42f7be40 0x55bc430fabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T08:00:50.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.739+0000 7f57ab7fe700 1 --2- 172.21.15.114:0/3713926384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 0x55bc4310b680 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f57a4005b80 tx=0x7f57a40090d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0e6953a6118246 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:50.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.739+0000 7f57917fa700 1 -- 172.21.15.114:0/3713926384 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a4003a00 con 0x55bc43109260 2024-05-27T08:00:50.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.739+0000 7f57917fa700 1 -- 172.21.15.114:0/3713926384 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a4003ba0 con 0x55bc43109260 2024-05-27T08:00:50.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.739+0000 7f57917fa700 1 -- 172.21.15.114:0/3713926384 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a40053f0 con 0x55bc43109260 2024-05-27T08:00:50.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.740+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/3713926384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 msgr2=0x55bc4310b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:50.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.740+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/3713926384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 0x55bc4310b680 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f57a4005b80 tx=0x7f57a40090d0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.740+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/3713926384 shutdown_connections 2024-05-27T08:00:50.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.740+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/3713926384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 0x55bc4310b680 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.740+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/3713926384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc42f7be40 0x55bc430fabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.740+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/3713926384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc430fb120 0x55bc430fb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.740+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/3713926384 >> 172.21.15.114:0/3713926384 conn(0x55bc42f7e770 msgr2=0x55bc430f5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:50.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.740+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/3713926384 shutdown_connections 2024-05-27T08:00:50.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.740+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/3713926384 wait complete. 2024-05-27T08:00:50.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.740+0000 7f57bfd5eb80 1 -- start start 2024-05-27T08:00:50.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57bfd5eb80 1 Processor -- start 2024-05-27T08:00:50.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57bfd5eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc42f7be40 0x55bc431062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57bfd5eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc430fb120 0x55bc431067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57bfd5eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc43109260 0x55bc43106d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57bfd5eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bc42f1efa0 con 0x55bc42f7be40 2024-05-27T08:00:50.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57bfd5eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bc42f80da0 con 0x55bc430fb120 2024-05-27T08:00:50.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57bfd5eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bc4310bf10 con 0x55bc43109260 2024-05-27T08:00:50.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57bfd5eb80 10 client.0 _ll_register_callbacks cb 0x55bc42faf5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T08:00:50.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57aaffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc42f7be40 0x55bc431062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57aaffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc42f7be40 0x55bc431062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35888/0 (socket says 172.21.15.114:35888) 2024-05-27T08:00:50.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57aaffd700 1 -- 172.21.15.114:0/1983610967 learned_addr learned my addr 172.21.15.114:0/1983610967 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:50.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57ab7fe700 1 --2- 172.21.15.114:0/1983610967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc43109260 0x55bc43106d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57aa7fc700 1 --2- 172.21.15.114:0/1983610967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc430fb120 0x55bc431067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57aaffd700 1 -- 172.21.15.114:0/1983610967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc43109260 msgr2=0x55bc43106d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:50.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57aaffd700 1 --2- 172.21.15.114:0/1983610967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc43109260 0x55bc43106d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57aaffd700 1 -- 172.21.15.114:0/1983610967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc430fb120 msgr2=0x55bc431067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:50.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57aaffd700 1 --2- 172.21.15.114:0/1983610967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc430fb120 0x55bc431067f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.741+0000 7f57aaffd700 1 -- 172.21.15.114:0/1983610967 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc42f688d0 con 0x55bc42f7be40 2024-05-27T08:00:50.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f57aaffd700 1 --2- 172.21.15.114:0/1983610967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc42f7be40 0x55bc431062b0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f579c010820 tx=0x7f579c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30db09593519f423 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:50.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f578eff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T08:00:50.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f578eff5700 1 -- 172.21.15.114:0/1983610967 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c00f850 con 0x55bc42f7be40 2024-05-27T08:00:50.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f578eff5700 1 -- 172.21.15.114:0/1983610967 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(170..170 src has 1..170) ==== 9466+0+0 (secure 0 0 0) 0x7f579c00f9f0 con 0x55bc42f7be40 2024-05-27T08:00:50.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/1983610967 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57a4003ba0 con 0x55bc42f7be40 2024-05-27T08:00:50.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f578eff5700 1 -- 172.21.15.114:0/1983610967 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=120) ==== 24+0+0 (secure 0 0 0) 0x7f579c02f050 con 0x55bc42f7be40 2024-05-27T08:00:50.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f57bfd5eb80 10 client.20010 fetch_fsmap learned FSMap version 120 2024-05-27T08:00:50.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/1983610967 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=120,monmap=4+}) -- 0x55bc42fd2760 con 0x55bc42f7be40 2024-05-27T08:00:50.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f578eff5700 1 -- 172.21.15.114:0/1983610967 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a4003ba0 con 0x55bc42f7be40 2024-05-27T08:00:50.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f578eff5700 1 -- 172.21.15.114:0/1983610967 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 120) ==== 2885+0+0 (secure 0 0 0) 0x7f579c01d3c0 con 0x55bc42f7be40 2024-05-27T08:00:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f57bfd5eb80 10 client.20010 fetch_fsmap finished waiting for FSMap version 120 2024-05-27T08:00:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f57bfd5eb80 10 client.20010 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T08:00:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f57bfd5eb80 20 client.20010 populate_metadata read hostname 'smithi114' 2024-05-27T08:00:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/1983610967 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bc43119330 0x55bc4311b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f57bfd5eb80 4 client.20010 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T08:00:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.742+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/1983610967 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc4302acc0 con 0x55bc43119330 2024-05-27T08:00:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.743+0000 7f57aa7fc700 1 --2- 172.21.15.114:0/1983610967 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bc43119330 0x55bc4311b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.743+0000 7f57aa7fc700 1 --2- 172.21.15.114:0/1983610967 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bc43119330 0x55bc4311b7e0 crc :-1 s=READY pgs=536 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68be35a41c702117 server_cookie=f8ee5689eb31735e in_seq=0 out_seq=0 2024-05-27T08:00:50.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.743+0000 7f578eff5700 10 client.20010 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:50.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f578eff5700 1 -- 172.21.15.114:0/1983610967 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bc4302acc0 con 0x55bc43119330 2024-05-27T08:00:50.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f578eff5700 10 client.20010 handle_command_reply: tid=0 2024-05-27T08:00:50.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f57bfd5eb80 1 client.20010 shutdown 2024-05-27T08:00:50.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/1983610967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc42f7be40 msgr2=0x55bc431062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:50.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/1983610967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc42f7be40 0x55bc431062b0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f579c010820 tx=0x7f579c004750 comp rx=0 tx=0).stop 2024-05-27T08:00:50.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/1983610967 shutdown_connections 2024-05-27T08:00:50.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/1983610967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc43109260 0x55bc43106d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/1983610967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc42f7be40 0x55bc431062b0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/1983610967 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bc43119330 0x55bc4311b7e0 crc :-1 s=READY pgs=536 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/1983610967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc430fb120 0x55bc431067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f578eff5700 0 client.20010 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/1983610967 >> 172.21.15.114:0/1983610967 conn(0x55bc42f7e770 msgr2=0x55bc430f7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/1983610967 shutdown_connections 2024-05-27T08:00:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/1983610967 wait complete. 2024-05-27T08:00:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.744+0000 7f57bfd5eb80 20 client.20010 trim_cache size 0 max 16384 2024-05-27T08:00:50.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.759+0000 7f57bfd5eb80 1 Processor -- start 2024-05-27T08:00:50.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.759+0000 7f57bfd5eb80 1 -- start start 2024-05-27T08:00:50.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.759+0000 7f57bfd5eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc42f7be40 0x55bc430fabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.759+0000 7f57bfd5eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc430fb120 0x55bc43105210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.759+0000 7f57bfd5eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 0x55bc43106b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.759+0000 7f57bfd5eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac1acff0 con 0x55bc43109260 2024-05-27T08:00:50.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.759+0000 7f57bfd5eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac1ad6c0 con 0x55bc430fb120 2024-05-27T08:00:50.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.759+0000 7f57bfd5eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac1ac050 con 0x55bc42f7be40 2024-05-27T08:00:50.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.759+0000 7f57ab7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 0x55bc43106b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.759+0000 7f57ab7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 0x55bc43106b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35896/0 (socket says 172.21.15.114:35896) 2024-05-27T08:00:50.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.759+0000 7f57ab7fe700 1 -- 172.21.15.114:0/3211683322 learned_addr learned my addr 172.21.15.114:0/3211683322 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:50.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.760+0000 7f57aaffd700 1 --2- 172.21.15.114:0/3211683322 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc42f7be40 0x55bc430fabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.760+0000 7f57aa7fc700 1 --2- 172.21.15.114:0/3211683322 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc430fb120 0x55bc43105210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.760+0000 7f57ab7fe700 1 -- 172.21.15.114:0/3211683322 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc42f7be40 msgr2=0x55bc430fabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:50.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.760+0000 7f57ab7fe700 1 --2- 172.21.15.114:0/3211683322 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc42f7be40 0x55bc430fabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.760+0000 7f57ab7fe700 1 -- 172.21.15.114:0/3211683322 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc430fb120 msgr2=0x55bc43105210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:50.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.760+0000 7f57ab7fe700 1 --2- 172.21.15.114:0/3211683322 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc430fb120 0x55bc43105210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.760+0000 7f57ab7fe700 1 -- 172.21.15.114:0/3211683322 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc431070b0 con 0x55bc43109260 2024-05-27T08:00:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.760+0000 7f57ab7fe700 1 --2- 172.21.15.114:0/3211683322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 0x55bc43106b70 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f57a4007f90 tx=0x7f57a4009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77f6d66ffd4c24f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.760+0000 7f5790ff9700 1 -- 172.21.15.114:0/3211683322 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a402b070 con 0x55bc43109260 2024-05-27T08:00:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.760+0000 7f5790ff9700 1 -- 172.21.15.114:0/3211683322 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a4007c70 con 0x55bc43109260 2024-05-27T08:00:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.760+0000 7f5790ff9700 1 -- 172.21.15.114:0/3211683322 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a40058c0 con 0x55bc43109260 2024-05-27T08:00:50.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.761+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/3211683322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 msgr2=0x55bc43106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:50.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.761+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/3211683322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 0x55bc43106b70 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f57a4007f90 tx=0x7f57a4009860 comp rx=0 tx=0).stop 2024-05-27T08:00:50.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.761+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/3211683322 shutdown_connections 2024-05-27T08:00:50.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.761+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/3211683322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc43109260 0x55bc43106b70 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.761+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/3211683322 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc42f7be40 0x55bc430fabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.761+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/3211683322 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc430fb120 0x55bc43105210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.761+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/3211683322 >> 172.21.15.114:0/3211683322 conn(0x55bc42f7e770 msgr2=0x55bc43107960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.761+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/3211683322 shutdown_connections 2024-05-27T08:00:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.761+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/3211683322 wait complete. 2024-05-27T08:00:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.761+0000 7f57bfd5eb80 1 -- start start 2024-05-27T08:00:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57bfd5eb80 1 Processor -- start 2024-05-27T08:00:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57bfd5eb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc42f7be40 0x55bc4310c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57bfd5eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc430fb120 0x55bc4310cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57bfd5eb80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc43109260 0x55bc43110ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57bfd5eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bc4302acc0 con 0x55bc430fb120 2024-05-27T08:00:50.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57bfd5eb80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bc430f58c0 con 0x55bc43109260 2024-05-27T08:00:50.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57bfd5eb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bc431090b0 con 0x55bc42f7be40 2024-05-27T08:00:50.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57bfd5eb80 10 client.0 _ll_register_callbacks cb 0x55bc42faf5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T08:00:50.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57aa7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc430fb120 0x55bc4310cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57aa7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc430fb120 0x55bc4310cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35898/0 (socket says 172.21.15.114:35898) 2024-05-27T08:00:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57aa7fc700 1 -- 172.21.15.114:0/2458486575 learned_addr learned my addr 172.21.15.114:0/2458486575 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57aaffd700 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc42f7be40 0x55bc4310c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57ab7fe700 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc43109260 0x55bc43110ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57aa7fc700 1 -- 172.21.15.114:0/2458486575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc42f7be40 msgr2=0x55bc4310c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57aa7fc700 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc42f7be40 0x55bc4310c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57aa7fc700 1 -- 172.21.15.114:0/2458486575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc43109260 msgr2=0x55bc43110ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T08:00:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57aa7fc700 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc43109260 0x55bc43110ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57aa7fc700 1 -- 172.21.15.114:0/2458486575 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc43111460 con 0x55bc430fb120 2024-05-27T08:00:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57ab7fe700 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc43109260 0x55bc43110ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T08:00:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f57aa7fc700 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc430fb120 0x55bc4310cec0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f57a00dfd60 tx=0x7f57a00cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d1feabdc05b81a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.762+0000 7f578eff5700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-05-27T08:00:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f578eff5700 1 -- 172.21.15.114:0/2458486575 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0007df0 con 0x55bc430fb120 2024-05-27T08:00:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f578eff5700 1 -- 172.21.15.114:0/2458486575 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(170..170 src has 1..170) ==== 9466+0+0 (secure 0 0 0) 0x7f57a00a9cb0 con 0x55bc430fb120 2024-05-27T08:00:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/2458486575 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57a4007c70 con 0x55bc430fb120 2024-05-27T08:00:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f578eff5700 1 -- 172.21.15.114:0/2458486575 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=120) ==== 24+0+0 (secure 0 0 0) 0x7f57a00a9840 con 0x55bc430fb120 2024-05-27T08:00:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f57bfd5eb80 10 client.20016 fetch_fsmap learned FSMap version 120 2024-05-27T08:00:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/2458486575 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=120,monmap=4+}) -- 0x55bc431838e0 con 0x55bc430fb120 2024-05-27T08:00:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f578eff5700 1 -- 172.21.15.114:0/2458486575 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a00c6080 con 0x55bc430fb120 2024-05-27T08:00:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f578eff5700 1 -- 172.21.15.114:0/2458486575 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 120) ==== 2885+0+0 (secure 0 0 0) 0x7f57a006b020 con 0x55bc430fb120 2024-05-27T08:00:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f57bfd5eb80 10 client.20016 fetch_fsmap finished waiting for FSMap version 120 2024-05-27T08:00:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f57bfd5eb80 10 client.20016 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-27T08:00:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f57bfd5eb80 20 client.20016 populate_metadata read hostname 'smithi114' 2024-05-27T08:00:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bc43183ff0 0x55bc43186410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f57bfd5eb80 4 client.20016 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T08:00:50.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.763+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/2458486575 --> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bc430244a0 con 0x55bc43183ff0 2024-05-27T08:00:50.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.764+0000 7f57aaffd700 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bc43183ff0 0x55bc43186410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:50.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.764+0000 7f578eff5700 10 client.20016 ms_handle_connect on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:50.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.764+0000 7f57aaffd700 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bc43183ff0 0x55bc43186410 crc :-1 s=READY pgs=537 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbe47a1ada8a7e70 server_cookie=10dc3f7614240d0e in_seq=0 out_seq=0 2024-05-27T08:00:50.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.764+0000 7f578eff5700 1 -- 172.21.15.114:0/2458486575 <== mds.0 v2:172.21.15.186:6838/4115872608 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bc430244a0 con 0x55bc43183ff0 2024-05-27T08:00:50.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.764+0000 7f578eff5700 10 client.20016 handle_command_reply: tid=0 2024-05-27T08:00:50.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.764+0000 7f57bfd5eb80 1 client.20016 shutdown 2024-05-27T08:00:50.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.764+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/2458486575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc430fb120 msgr2=0x55bc4310cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:50.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.764+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc430fb120 0x55bc4310cec0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f57a00dfd60 tx=0x7f57a00cc040 comp rx=0 tx=0).stop 2024-05-27T08:00:50.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.764+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/2458486575 shutdown_connections 2024-05-27T08:00:50.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.764+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.186:6838/4115872608,v1:172.21.15.186:6839/4115872608] conn(0x55bc43183ff0 0x55bc43186410 crc :-1 s=READY pgs=537 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bc43109260 0x55bc43110ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bc42f7be40 0x55bc4310c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f578eff5700 0 client.20016 ms_handle_reset on v2:172.21.15.186:6838/4115872608 2024-05-27T08:00:50.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57bfd5eb80 1 --2- 172.21.15.114:0/2458486575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc430fb120 0x55bc4310cec0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/2458486575 >> 172.21.15.114:0/2458486575 conn(0x55bc42f7e770 msgr2=0x55bc431078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:50.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/2458486575 shutdown_connections 2024-05-27T08:00:50.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57bfd5eb80 1 -- 172.21.15.114:0/2458486575 wait complete. 2024-05-27T08:00:50.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57bfd5eb80 20 client.20016 trim_cache size 0 max 16384 2024-05-27T08:00:50.778 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2024-05-27T08:00:50.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57b1349700 1 -- 172.21.15.114:0/4171371825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f57940b29f0 msgr2=0x7f57940b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:50.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57b1349700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f57940b29f0 0x7f57940b4f00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f579c005950 tx=0x7f579c005ed0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57b1349700 1 -- 172.21.15.114:0/4171371825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac107e00 msgr2=0x7f57ac19fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:50.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57b1349700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac107e00 0x7f57ac19fda0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f57a000c9e0 tx=0x7f57a000ccf0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57b1349700 1 -- 172.21.15.114:0/4171371825 shutdown_connections 2024-05-27T08:00:50.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57b1349700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57ac108750 0x7f57ac199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57b1349700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f57940b29f0 0x7f57940b4f00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57b1349700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57ac1034d0 0x7f57ac19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57b1349700 1 --2- 172.21.15.114:0/4171371825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac107e00 0x7f57ac19fda0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:50.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57b1349700 1 -- 172.21.15.114:0/4171371825 >> 172.21.15.114:0/4171371825 conn(0x7f57ac0fef20 msgr2=0x7f57ac107540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:50.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.765+0000 7f57b1349700 1 -- 172.21.15.114:0/4171371825 shutdown_connections 2024-05-27T08:00:50.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:50.766+0000 7f57b1349700 1 -- 172.21.15.114:0/4171371825 wait complete. 2024-05-27T08:00:51.570 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-05-27T08:00:51.579 INFO:tasks.fwd_scrub:done joining 2024-05-27T08:00:51.579 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-05-27T08:00:51.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T08:00:51.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.703+0000 7f567de61640 1 -- 172.21.15.114:0/3041477334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5678102180 msgr2=0x7f5678102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:51.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.703+0000 7f567de61640 1 --2- 172.21.15.114:0/3041477334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5678102180 0x7f5678102590 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f566c009900 tx=0x7f566c02a410 comp rx=0 tx=0).stop 2024-05-27T08:00:51.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.703+0000 7f567de61640 1 -- 172.21.15.114:0/3041477334 shutdown_connections 2024-05-27T08:00:51.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.703+0000 7f567de61640 1 --2- 172.21.15.114:0/3041477334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567810e680 0x7f5678110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:51.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.703+0000 7f567de61640 1 --2- 172.21.15.114:0/3041477334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5678102180 0x7f5678102590 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:51.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.703+0000 7f567de61640 1 --2- 172.21.15.114:0/3041477334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5678109cd0 0x7f5678101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:51.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.703+0000 7f567de61640 1 -- 172.21.15.114:0/3041477334 >> 172.21.15.114:0/3041477334 conn(0x7f56780fd970 msgr2=0x7f56780ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:51.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.703+0000 7f567de61640 1 -- 172.21.15.114:0/3041477334 shutdown_connections 2024-05-27T08:00:51.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.703+0000 7f567de61640 1 -- 172.21.15.114:0/3041477334 wait complete. 2024-05-27T08:00:51.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.704+0000 7f567de61640 1 Processor -- start 2024-05-27T08:00:51.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.704+0000 7f567de61640 1 -- start start 2024-05-27T08:00:51.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.704+0000 7f567de61640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5678102180 0x7f56781a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:51.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.704+0000 7f567de61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5678109cd0 0x7f56781a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:51.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.704+0000 7f567de61640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567810e680 0x7f567819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:51.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.704+0000 7f567de61640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5678113e40 con 0x7f567810e680 2024-05-27T08:00:51.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.704+0000 7f567de61640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5678113cc0 con 0x7f5678102180 2024-05-27T08:00:51.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.704+0000 7f567de61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5678113fc0 con 0x7f5678109cd0 2024-05-27T08:00:51.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f5677fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567810e680 0x7f567819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:51.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f5677fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567810e680 0x7f567819a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35918/0 (socket says 172.21.15.114:35918) 2024-05-27T08:00:51.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f5677fff640 1 -- 172.21.15.114:0/1060218384 learned_addr learned my addr 172.21.15.114:0/1060218384 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:51.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f5676ffd640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5678109cd0 0x7f56781a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:51.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f56777fe640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5678102180 0x7f56781a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:51.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f5677fff640 1 -- 172.21.15.114:0/1060218384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5678109cd0 msgr2=0x7f56781a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:51.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f5677fff640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5678109cd0 0x7f56781a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:51.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f5677fff640 1 -- 172.21.15.114:0/1060218384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5678102180 msgr2=0x7f56781a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:51.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f5677fff640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5678102180 0x7f56781a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:51.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f5677fff640 1 -- 172.21.15.114:0/1060218384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567819ad90 con 0x7f567810e680 2024-05-27T08:00:51.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f5677fff640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567810e680 0x7f567819a660 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f567000e970 tx=0x7f567000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:51.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f5674ff9640 1 -- 172.21.15.114:0/1060218384 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567000ccc0 con 0x7f567810e680 2024-05-27T08:00:51.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f5674ff9640 1 -- 172.21.15.114:0/1060218384 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56700054d0 con 0x7f567810e680 2024-05-27T08:00:51.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f567de61640 1 -- 172.21.15.114:0/1060218384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f567819b020 con 0x7f567810e680 2024-05-27T08:00:51.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f5674ff9640 1 -- 172.21.15.114:0/1060218384 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5670010430 con 0x7f567810e680 2024-05-27T08:00:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.705+0000 7f567de61640 1 -- 172.21.15.114:0/1060218384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56781a7280 con 0x7f567810e680 2024-05-27T08:00:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.706+0000 7f567de61640 1 -- 172.21.15.114:0/1060218384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5678104f60 con 0x7f567810e680 2024-05-27T08:00:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.708+0000 7f5674ff9640 1 -- 172.21.15.114:0/1060218384 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f5670004fe0 con 0x7f567810e680 2024-05-27T08:00:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.709+0000 7f5674ff9640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f56600b2960 0x7f56600b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.709+0000 7f56777fe640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f56600b2960 0x7f56600b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.709+0000 7f5674ff9640 1 -- 172.21.15.114:0/1060218384 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 9466+0+0 (secure 0 0 0) 0x7f56700e2f90 con 0x7f567810e680 2024-05-27T08:00:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.709+0000 7f56777fe640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f56600b2960 0x7f56600b4e70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f567806be30 tx=0x7f5668009870 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:51.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.712+0000 7f5674ff9640 1 -- 172.21.15.114:0/1060218384 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5670054df0 con 0x7f567810e680 2024-05-27T08:00:51.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:51 smithi114 ceph-mon[32038]: pgmap v732: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 34 KiB/s wr, 4 op/s 2024-05-27T08:00:51.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.898+0000 7f567de61640 1 -- 172.21.15.114:0/1060218384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f56781a7450 con 0x7f567810e680 2024-05-27T08:00:51.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.900+0000 7f5674ff9640 1 -- 172.21.15.114:0/1060218384 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) ==== 74+0+25358 (secure 0 0 0) 0x7f56700aa5c0 con 0x7f567810e680 2024-05-27T08:00:51.902 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T08:00:51.902 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":170,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T08:00:45.139773+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":37,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":37,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/1628485461":"2024-05-28T07:39:44.126253+0000","172.21.15.114:0/1761274168":"2024-05-28T07:39:44.126253+0000","172.21.15.114:0/2574023732":"2024-05-28T07:39:44.126253+0000","172.21.15.114:6800/2291864634":"2024-05-28T07:39:44.126253+0000","172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2381281099":"2024-05-28T07:39:44.126253+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/2291864634":"2024-05-28T07:39:44.126253+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T08:00:51.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.902+0000 7f567de61640 1 -- 172.21.15.114:0/1060218384 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f56600b2960 msgr2=0x7f56600b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:51.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.902+0000 7f567de61640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f56600b2960 0x7f56600b4e70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f567806be30 tx=0x7f5668009870 comp rx=0 tx=0).stop 2024-05-27T08:00:51.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.902+0000 7f567de61640 1 -- 172.21.15.114:0/1060218384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567810e680 msgr2=0x7f567819a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:51.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.902+0000 7f567de61640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567810e680 0x7f567819a660 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f567000e970 tx=0x7f567000ee30 comp rx=0 tx=0).stop 2024-05-27T08:00:51.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.902+0000 7f567de61640 1 -- 172.21.15.114:0/1060218384 shutdown_connections 2024-05-27T08:00:51.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.902+0000 7f567de61640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f56600b2960 0x7f56600b4e70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:51.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.902+0000 7f567de61640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567810e680 0x7f567819a660 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:51.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.902+0000 7f567de61640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5678109cd0 0x7f56781a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:51.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.902+0000 7f567de61640 1 --2- 172.21.15.114:0/1060218384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5678102180 0x7f56781a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:51.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.902+0000 7f567de61640 1 -- 172.21.15.114:0/1060218384 >> 172.21.15.114:0/1060218384 conn(0x7f56780fd970 msgr2=0x7f56780fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:51.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.902+0000 7f567de61640 1 -- 172.21.15.114:0/1060218384 shutdown_connections 2024-05-27T08:00:51.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:51.902+0000 7f567de61640 1 -- 172.21.15.114:0/1060218384 wait complete. 2024-05-27T08:00:51.916 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T07:31:09.365695+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T07:35:59.962762+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T07:36:03.708207+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '165', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 37, 'snap_epoch': 165, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-27T07:36:13.010229+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '165', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 37, 'snap_epoch': 165, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-27T08:00:51.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T08:00:51.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:51 smithi163 ceph-mon[36451]: pgmap v732: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 34 KiB/s wr, 4 op/s 2024-05-27T08:00:51.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:51 smithi186 ceph-mon[36647]: pgmap v732: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 34 KiB/s wr, 4 op/s 2024-05-27T08:00:52.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.028+0000 7f34b3c49640 1 -- 172.21.15.114:0/2185582085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34ac104380 msgr2=0x7f34ac110750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.028+0000 7f34b3c49640 1 --2- 172.21.15.114:0/2185582085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34ac104380 0x7f34ac110750 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f34a4009ca0 tx=0x7f34a402a8c0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.028+0000 7f34b3c49640 1 -- 172.21.15.114:0/2185582085 shutdown_connections 2024-05-27T08:00:52.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.028+0000 7f34b3c49640 1 --2- 172.21.15.114:0/2185582085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34ac104380 0x7f34ac110750 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.028+0000 7f34b3c49640 1 --2- 172.21.15.114:0/2185582085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f34ac10a0d0 0x7f34ac103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.028+0000 7f34b3c49640 1 --2- 172.21.15.114:0/2185582085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34ac109c30 0x7f34ac1037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.028+0000 7f34b3c49640 1 -- 172.21.15.114:0/2185582085 >> 172.21.15.114:0/2185582085 conn(0x7f34ac0ff330 msgr2=0x7f34ac101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:52.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.028+0000 7f34b3c49640 1 -- 172.21.15.114:0/2185582085 shutdown_connections 2024-05-27T08:00:52.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.028+0000 7f34b3c49640 1 -- 172.21.15.114:0/2185582085 wait complete. 2024-05-27T08:00:52.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.029+0000 7f34b3c49640 1 Processor -- start 2024-05-27T08:00:52.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.029+0000 7f34b3c49640 1 -- start start 2024-05-27T08:00:52.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.029+0000 7f34b3c49640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f34ac104380 0x7f34ac19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.029+0000 7f34b3c49640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34ac109c30 0x7f34ac19ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.029+0000 7f34b3c49640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34ac10a0d0 0x7f34ac19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.029+0000 7f34b3c49640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac113e20 con 0x7f34ac10a0d0 2024-05-27T08:00:52.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.029+0000 7f34b3c49640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac113ca0 con 0x7f34ac104380 2024-05-27T08:00:52.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.029+0000 7f34b3c49640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac113fa0 con 0x7f34ac109c30 2024-05-27T08:00:52.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.029+0000 7f34b21e6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34ac10a0d0 0x7f34ac19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34b21e6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34ac10a0d0 0x7f34ac19f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35934/0 (socket says 172.21.15.114:35934) 2024-05-27T08:00:52.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34b21e6640 1 -- 172.21.15.114:0/447662196 learned_addr learned my addr 172.21.15.114:0/447662196 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:52.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34b11e4640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34ac109c30 0x7f34ac19ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34b19e5640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f34ac104380 0x7f34ac19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34b21e6640 1 -- 172.21.15.114:0/447662196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34ac109c30 msgr2=0x7f34ac19ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34b21e6640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34ac109c30 0x7f34ac19ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34b21e6640 1 -- 172.21.15.114:0/447662196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f34ac104380 msgr2=0x7f34ac19a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34b21e6640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f34ac104380 0x7f34ac19a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34b21e6640 1 -- 172.21.15.114:0/447662196 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ac19f660 con 0x7f34ac10a0d0 2024-05-27T08:00:52.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34b21e6640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34ac10a0d0 0x7f34ac19f020 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f34a4005cc0 tx=0x7f34a402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:52.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34a2ffd640 1 -- 172.21.15.114:0/447662196 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a4003bf0 con 0x7f34ac10a0d0 2024-05-27T08:00:52.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34b3c49640 1 -- 172.21.15.114:0/447662196 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34ac19f8f0 con 0x7f34ac10a0d0 2024-05-27T08:00:52.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34a2ffd640 1 -- 172.21.15.114:0/447662196 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34a4003d90 con 0x7f34ac10a0d0 2024-05-27T08:00:52.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34a2ffd640 1 -- 172.21.15.114:0/447662196 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a402fbc0 con 0x7f34ac10a0d0 2024-05-27T08:00:52.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.030+0000 7f34b3c49640 1 -- 172.21.15.114:0/447662196 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34ac19fe40 con 0x7f34ac10a0d0 2024-05-27T08:00:52.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.031+0000 7f34b3c49640 1 -- 172.21.15.114:0/447662196 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34ac06a280 con 0x7f34ac10a0d0 2024-05-27T08:00:52.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.033+0000 7f34a2ffd640 1 -- 172.21.15.114:0/447662196 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f34a402fd60 con 0x7f34ac10a0d0 2024-05-27T08:00:52.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.033+0000 7f34a2ffd640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f34980b2960 0x7f34980b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.034+0000 7f34b19e5640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f34980b2960 0x7f34980b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.034+0000 7f34a2ffd640 1 -- 172.21.15.114:0/447662196 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 9466+0+0 (secure 0 0 0) 0x7f34a41040f0 con 0x7f34ac10a0d0 2024-05-27T08:00:52.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.034+0000 7f34b19e5640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f34980b2960 0x7f34980b4e70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f34ac104910 tx=0x7f34a800b390 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:52.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.037+0000 7f34a2ffd640 1 -- 172.21.15.114:0/447662196 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f34a40cb830 con 0x7f34ac10a0d0 2024-05-27T08:00:52.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.218+0000 7f34b3c49640 1 -- 172.21.15.114:0/447662196 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f34ac10bd70 con 0x7f34ac10a0d0 2024-05-27T08:00:52.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.218+0000 7f34a2ffd640 1 -- 172.21.15.114:0/447662196 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v170) ==== 93+0+10 (secure 0 0 0) 0x7f34a40cb130 con 0x7f34ac10a0d0 2024-05-27T08:00:52.221 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-05-27T08:00:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.220+0000 7f34b3c49640 1 -- 172.21.15.114:0/447662196 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f34980b2960 msgr2=0x7f34980b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.220+0000 7f34b3c49640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f34980b2960 0x7f34980b4e70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f34ac104910 tx=0x7f34a800b390 comp rx=0 tx=0).stop 2024-05-27T08:00:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.220+0000 7f34b3c49640 1 -- 172.21.15.114:0/447662196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34ac10a0d0 msgr2=0x7f34ac19f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.220+0000 7f34b3c49640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34ac10a0d0 0x7f34ac19f020 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f34a4005cc0 tx=0x7f34a402f380 comp rx=0 tx=0).stop 2024-05-27T08:00:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.220+0000 7f34b3c49640 1 -- 172.21.15.114:0/447662196 shutdown_connections 2024-05-27T08:00:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.220+0000 7f34b3c49640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f34980b2960 0x7f34980b4e70 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.220+0000 7f34b3c49640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34ac10a0d0 0x7f34ac19f020 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.220+0000 7f34b3c49640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34ac109c30 0x7f34ac19ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.220+0000 7f34b3c49640 1 --2- 172.21.15.114:0/447662196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f34ac104380 0x7f34ac19a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.220+0000 7f34b3c49640 1 -- 172.21.15.114:0/447662196 >> 172.21.15.114:0/447662196 conn(0x7f34ac0ff330 msgr2=0x7f34ac106660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.221+0000 7f34b3c49640 1 -- 172.21.15.114:0/447662196 shutdown_connections 2024-05-27T08:00:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.221+0000 7f34b3c49640 1 -- 172.21.15.114:0/447662196 wait complete. 2024-05-27T08:00:52.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T08:00:52.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.348+0000 7f43ae71f640 1 -- 172.21.15.114:0/2021586342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43a810b7d0 msgr2=0x7f43a810dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.348+0000 7f43ae71f640 1 --2- 172.21.15.114:0/2021586342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43a810b7d0 0x7f43a810dbf0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f4390009900 tx=0x7f439002a510 comp rx=0 tx=0).stop 2024-05-27T08:00:52.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.348+0000 7f43ae71f640 1 -- 172.21.15.114:0/2021586342 shutdown_connections 2024-05-27T08:00:52.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.348+0000 7f43ae71f640 1 --2- 172.21.15.114:0/2021586342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43a810e180 0x7f43a8110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.348+0000 7f43ae71f640 1 --2- 172.21.15.114:0/2021586342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43a810b7d0 0x7f43a810dbf0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.348+0000 7f43ae71f640 1 --2- 172.21.15.114:0/2021586342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a8104d70 0x7f43a8105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.348+0000 7f43ae71f640 1 -- 172.21.15.114:0/2021586342 >> 172.21.15.114:0/2021586342 conn(0x7f43a8078220 msgr2=0x7f43a8078660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:52.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.348+0000 7f43ae71f640 1 -- 172.21.15.114:0/2021586342 shutdown_connections 2024-05-27T08:00:52.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.348+0000 7f43ae71f640 1 -- 172.21.15.114:0/2021586342 wait complete. 2024-05-27T08:00:52.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.349+0000 7f43ae71f640 1 Processor -- start 2024-05-27T08:00:52.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.349+0000 7f43ae71f640 1 -- start start 2024-05-27T08:00:52.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.349+0000 7f43ae71f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a8104d70 0x7f43a8102630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.349+0000 7f43ae71f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43a810b7d0 0x7f43a8102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.349+0000 7f43ae71f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43a810e180 0x7f43a81030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.349+0000 7f43ae71f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f43a8113e20 con 0x7f43a810e180 2024-05-27T08:00:52.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.349+0000 7f43ae71f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f43a8113ca0 con 0x7f43a8104d70 2024-05-27T08:00:52.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.349+0000 7f43ae71f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f43a8113fa0 con 0x7f43a810b7d0 2024-05-27T08:00:52.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.349+0000 7f43accbc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43a810e180 0x7f43a81030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.349+0000 7f43accbc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43a810e180 0x7f43a81030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35954/0 (socket says 172.21.15.114:35954) 2024-05-27T08:00:52.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.349+0000 7f43accbc640 1 -- 172.21.15.114:0/2057093489 learned_addr learned my addr 172.21.15.114:0/2057093489 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:52.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.349+0000 7f43a7fff640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a8104d70 0x7f43a8102630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.350+0000 7f439ffff640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43a810b7d0 0x7f43a8102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.350+0000 7f43accbc640 1 -- 172.21.15.114:0/2057093489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43a810b7d0 msgr2=0x7f43a8102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.350+0000 7f43accbc640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43a810b7d0 0x7f43a8102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.350+0000 7f43accbc640 1 -- 172.21.15.114:0/2057093489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a8104d70 msgr2=0x7f43a8102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.350+0000 7f43accbc640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a8104d70 0x7f43a8102630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.350+0000 7f43accbc640 1 -- 172.21.15.114:0/2057093489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a81a8c00 con 0x7f43a810e180 2024-05-27T08:00:52.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.350+0000 7f43accbc640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43a810e180 0x7f43a81030b0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f43a000b970 tx=0x7f43a000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:52.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.350+0000 7f43a5ffb640 1 -- 172.21.15.114:0/2057093489 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0019070 con 0x7f43a810e180 2024-05-27T08:00:52.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.350+0000 7f43ae71f640 1 -- 172.21.15.114:0/2057093489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a81a8ef0 con 0x7f43a810e180 2024-05-27T08:00:52.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.350+0000 7f43a5ffb640 1 -- 172.21.15.114:0/2057093489 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43a0014440 con 0x7f43a810e180 2024-05-27T08:00:52.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.350+0000 7f43a5ffb640 1 -- 172.21.15.114:0/2057093489 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a00075d0 con 0x7f43a810e180 2024-05-27T08:00:52.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.350+0000 7f43ae71f640 1 -- 172.21.15.114:0/2057093489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a81a9350 con 0x7f43a810e180 2024-05-27T08:00:52.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.351+0000 7f43ae71f640 1 -- 172.21.15.114:0/2057093489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43a806b0c0 con 0x7f43a810e180 2024-05-27T08:00:52.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.353+0000 7f43a5ffb640 1 -- 172.21.15.114:0/2057093489 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f43a001c070 con 0x7f43a810e180 2024-05-27T08:00:52.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.353+0000 7f43a5ffb640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f43940b2960 0x7f43940b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.353+0000 7f43a7fff640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f43940b2960 0x7f43940b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.354+0000 7f43a5ffb640 1 -- 172.21.15.114:0/2057093489 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 9466+0+0 (secure 0 0 0) 0x7f43a00e2330 con 0x7f43a810e180 2024-05-27T08:00:52.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.354+0000 7f43a7fff640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f43940b2960 0x7f43940b4e70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f43980058c0 tx=0x7f439800a2d0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:52.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.357+0000 7f43a5ffb640 1 -- 172.21.15.114:0/2057093489 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f43a00a9420 con 0x7f43a810e180 2024-05-27T08:00:52.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.541+0000 7f43ae71f640 1 -- 172.21.15.114:0/2057093489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f43a81a96f0 con 0x7f43a810e180 2024-05-27T08:00:52.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.541+0000 7f43a5ffb640 1 -- 172.21.15.114:0/2057093489 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v170) ==== 104+0+11 (secure 0 0 0) 0x7f43a00ad720 con 0x7f43a810e180 2024-05-27T08:00:52.544 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-05-27T08:00:52.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.543+0000 7f43ae71f640 1 -- 172.21.15.114:0/2057093489 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f43940b2960 msgr2=0x7f43940b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.543+0000 7f43ae71f640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f43940b2960 0x7f43940b4e70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f43980058c0 tx=0x7f439800a2d0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.543+0000 7f43ae71f640 1 -- 172.21.15.114:0/2057093489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43a810e180 msgr2=0x7f43a81030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.543+0000 7f43ae71f640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43a810e180 0x7f43a81030b0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f43a000b970 tx=0x7f43a000be30 comp rx=0 tx=0).stop 2024-05-27T08:00:52.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.543+0000 7f43ae71f640 1 -- 172.21.15.114:0/2057093489 shutdown_connections 2024-05-27T08:00:52.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.543+0000 7f43ae71f640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f43940b2960 0x7f43940b4e70 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.543+0000 7f43ae71f640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43a810e180 0x7f43a81030b0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.543+0000 7f43ae71f640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43a810b7d0 0x7f43a8102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.543+0000 7f43ae71f640 1 --2- 172.21.15.114:0/2057093489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a8104d70 0x7f43a8102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.543+0000 7f43ae71f640 1 -- 172.21.15.114:0/2057093489 >> 172.21.15.114:0/2057093489 conn(0x7f43a8078220 msgr2=0x7f43a81012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:52.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.543+0000 7f43ae71f640 1 -- 172.21.15.114:0/2057093489 shutdown_connections 2024-05-27T08:00:52.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.543+0000 7f43ae71f640 1 -- 172.21.15.114:0/2057093489 wait complete. 2024-05-27T08:00:52.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T08:00:52.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.670+0000 7f037b889640 1 -- 172.21.15.114:0/3811292775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f037410dbe0 msgr2=0x7f0374110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.670+0000 7f037b889640 1 --2- 172.21.15.114:0/3811292775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f037410dbe0 0x7f0374110000 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f0364009900 tx=0x7f036402a410 comp rx=0 tx=0).stop 2024-05-27T08:00:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.670+0000 7f037b889640 1 -- 172.21.15.114:0/3811292775 shutdown_connections 2024-05-27T08:00:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.670+0000 7f037b889640 1 --2- 172.21.15.114:0/3811292775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0374110540 0x7f0374112960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.670+0000 7f037b889640 1 --2- 172.21.15.114:0/3811292775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f037410dbe0 0x7f0374110000 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.670+0000 7f037b889640 1 --2- 172.21.15.114:0/3811292775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03741029a0 0x7f0374102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.670+0000 7f037b889640 1 -- 172.21.15.114:0/3811292775 >> 172.21.15.114:0/3811292775 conn(0x7f0374078230 msgr2=0x7f0374078670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.670+0000 7f037b889640 1 -- 172.21.15.114:0/3811292775 shutdown_connections 2024-05-27T08:00:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.670+0000 7f037b889640 1 -- 172.21.15.114:0/3811292775 wait complete. 2024-05-27T08:00:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.670+0000 7f037b889640 1 Processor -- start 2024-05-27T08:00:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.670+0000 7f037b889640 1 -- start start 2024-05-27T08:00:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f037b889640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03741029a0 0x7f03741a26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f037b889640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f037410dbe0 0x7f03741a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f037b889640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0374110540 0x7f037419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f037b889640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0374115fd0 con 0x7f03741029a0 2024-05-27T08:00:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f037b889640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0374115e50 con 0x7f0374110540 2024-05-27T08:00:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f037b889640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0374116150 con 0x7f037410dbe0 2024-05-27T08:00:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f0379625640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03741029a0 0x7f03741a26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f0379625640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03741029a0 0x7f03741a26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35976/0 (socket says 172.21.15.114:35976) 2024-05-27T08:00:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f0379625640 1 -- 172.21.15.114:0/4127074315 learned_addr learned my addr 172.21.15.114:0/4127074315 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f0379e26640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0374110540 0x7f037419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f0378e24640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f037410dbe0 0x7f03741a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f0379625640 1 -- 172.21.15.114:0/4127074315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f037410dbe0 msgr2=0x7f03741a2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f0379625640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f037410dbe0 0x7f03741a2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f0379625640 1 -- 172.21.15.114:0/4127074315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0374110540 msgr2=0x7f037419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f0379625640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0374110540 0x7f037419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f0379625640 1 -- 172.21.15.114:0/4127074315 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037419cec0 con 0x7f03741029a0 2024-05-27T08:00:52.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f0379625640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03741029a0 0x7f03741a26c0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f037000d950 tx=0x7f037000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:52.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f036a7fc640 1 -- 172.21.15.114:0/4127074315 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0370017070 con 0x7f03741029a0 2024-05-27T08:00:52.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f036a7fc640 1 -- 172.21.15.114:0/4127074315 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0370009c30 con 0x7f03741029a0 2024-05-27T08:00:52.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f037b889640 1 -- 172.21.15.114:0/4127074315 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f037419d1b0 con 0x7f03741029a0 2024-05-27T08:00:52.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.671+0000 7f036a7fc640 1 -- 172.21.15.114:0/4127074315 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037000f620 con 0x7f03741029a0 2024-05-27T08:00:52.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.672+0000 7f037b889640 1 -- 172.21.15.114:0/4127074315 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03741a9540 con 0x7f03741029a0 2024-05-27T08:00:52.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.672+0000 7f037b889640 1 -- 172.21.15.114:0/4127074315 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f037406a280 con 0x7f03741029a0 2024-05-27T08:00:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.674+0000 7f036a7fc640 1 -- 172.21.15.114:0/4127074315 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f0370010460 con 0x7f03741029a0 2024-05-27T08:00:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.675+0000 7f036a7fc640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03600b29e0 0x7f03600b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.675+0000 7f0378e24640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03600b29e0 0x7f03600b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.675+0000 7f036a7fc640 1 -- 172.21.15.114:0/4127074315 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 9466+0+0 (secure 0 0 0) 0x7f03700e2610 con 0x7f03741029a0 2024-05-27T08:00:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.675+0000 7f0378e24640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03600b29e0 0x7f03600b4ef0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f036402a930 tx=0x7f0364000bc0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:52.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.678+0000 7f036a7fc640 1 -- 172.21.15.114:0/4127074315 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f03700a9cc0 con 0x7f03741029a0 2024-05-27T08:00:52.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:52 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1060218384' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T08:00:52.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:52 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/447662196' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T08:00:52.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.861+0000 7f037b889640 1 -- 172.21.15.114:0/4127074315 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f03741a9710 con 0x7f03741029a0 2024-05-27T08:00:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.861+0000 7f036a7fc640 1 -- 172.21.15.114:0/4127074315 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v170) ==== 100+0+11 (secure 0 0 0) 0x7f0370009de0 con 0x7f03741029a0 2024-05-27T08:00:52.863 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-05-27T08:00:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.863+0000 7f037b889640 1 -- 172.21.15.114:0/4127074315 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03600b29e0 msgr2=0x7f03600b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.863+0000 7f037b889640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03600b29e0 0x7f03600b4ef0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f036402a930 tx=0x7f0364000bc0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.863+0000 7f037b889640 1 -- 172.21.15.114:0/4127074315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03741029a0 msgr2=0x7f03741a26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.863+0000 7f037b889640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03741029a0 0x7f03741a26c0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f037000d950 tx=0x7f037000de10 comp rx=0 tx=0).stop 2024-05-27T08:00:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.863+0000 7f037b889640 1 -- 172.21.15.114:0/4127074315 shutdown_connections 2024-05-27T08:00:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.863+0000 7f037b889640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f03600b29e0 0x7f03600b4ef0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.863+0000 7f037b889640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0374110540 0x7f037419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.863+0000 7f037b889640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f037410dbe0 0x7f03741a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.863+0000 7f037b889640 1 --2- 172.21.15.114:0/4127074315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03741029a0 0x7f03741a26c0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.863+0000 7f037b889640 1 -- 172.21.15.114:0/4127074315 >> 172.21.15.114:0/4127074315 conn(0x7f0374078230 msgr2=0x7f03741012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.863+0000 7f037b889640 1 -- 172.21.15.114:0/4127074315 shutdown_connections 2024-05-27T08:00:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.863+0000 7f037b889640 1 -- 172.21.15.114:0/4127074315 wait complete. 2024-05-27T08:00:52.877 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-27T08:00:52.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:52 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1060218384' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T08:00:52.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:52 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/447662196' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T08:00:52.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:52 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1060218384' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T08:00:52.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:52 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/447662196' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T08:00:52.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.989+0000 7f641dbb7640 1 -- 172.21.15.114:0/1385280851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6418108b20 msgr2=0x7f64181026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.989+0000 7f641dbb7640 1 --2- 172.21.15.114:0/1385280851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6418108b20 0x7f64181026c0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f6408009900 tx=0x7f640802a410 comp rx=0 tx=0).stop 2024-05-27T08:00:52.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.989+0000 7f641dbb7640 1 -- 172.21.15.114:0/1385280851 shutdown_connections 2024-05-27T08:00:52.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.989+0000 7f641dbb7640 1 --2- 172.21.15.114:0/1385280851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6418103270 0x7f641810f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.989+0000 7f641dbb7640 1 --2- 172.21.15.114:0/1385280851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6418108fc0 0x7f6418102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.989+0000 7f641dbb7640 1 --2- 172.21.15.114:0/1385280851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6418108b20 0x7f64181026c0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.989+0000 7f641dbb7640 1 -- 172.21.15.114:0/1385280851 >> 172.21.15.114:0/1385280851 conn(0x7f64180fe380 msgr2=0x7f64181007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.989+0000 7f641dbb7640 1 -- 172.21.15.114:0/1385280851 shutdown_connections 2024-05-27T08:00:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.989+0000 7f641dbb7640 1 -- 172.21.15.114:0/1385280851 wait complete. 2024-05-27T08:00:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.990+0000 7f641dbb7640 1 Processor -- start 2024-05-27T08:00:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.990+0000 7f641dbb7640 1 -- start start 2024-05-27T08:00:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.990+0000 7f641dbb7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6418103270 0x7f641819b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.990+0000 7f641dbb7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6418108b20 0x7f641819bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.990+0000 7f641dbb7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6418108fc0 0x7f641819c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:52.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.990+0000 7f641dbb7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6418112d10 con 0x7f6418108fc0 2024-05-27T08:00:52.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.990+0000 7f641dbb7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6418112b90 con 0x7f6418108b20 2024-05-27T08:00:52.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.990+0000 7f641dbb7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6418112e90 con 0x7f6418103270 2024-05-27T08:00:52.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f6417fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6418108fc0 0x7f641819c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f6417fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6418108fc0 0x7f641819c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35990/0 (socket says 172.21.15.114:35990) 2024-05-27T08:00:52.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f6417fff640 1 -- 172.21.15.114:0/2136444503 learned_addr learned my addr 172.21.15.114:0/2136444503 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:52.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f64177fe640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6418103270 0x7f641819b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f6416ffd640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6418108b20 0x7f641819bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:52.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f6417fff640 1 -- 172.21.15.114:0/2136444503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6418103270 msgr2=0x7f641819b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f6417fff640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6418103270 0x7f641819b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f6417fff640 1 -- 172.21.15.114:0/2136444503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6418108b20 msgr2=0x7f641819bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:52.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f6417fff640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6418108b20 0x7f641819bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:52.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f6417fff640 1 -- 172.21.15.114:0/2136444503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64181a6a10 con 0x7f6418108fc0 2024-05-27T08:00:52.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f6417fff640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6418108fc0 0x7f641819c1d0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f641000eaa0 tx=0x7f641000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f6414ff9640 1 -- 172.21.15.114:0/2136444503 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641000cdb0 con 0x7f6418108fc0 2024-05-27T08:00:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f6414ff9640 1 -- 172.21.15.114:0/2136444503 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6410005020 con 0x7f6418108fc0 2024-05-27T08:00:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f641dbb7640 1 -- 172.21.15.114:0/2136444503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64181a6d00 con 0x7f6418108fc0 2024-05-27T08:00:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f6414ff9640 1 -- 172.21.15.114:0/2136444503 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64100116d0 con 0x7f6418108fc0 2024-05-27T08:00:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.991+0000 7f641dbb7640 1 -- 172.21.15.114:0/2136444503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64181a71f0 con 0x7f6418108fc0 2024-05-27T08:00:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.992+0000 7f641dbb7640 1 -- 172.21.15.114:0/2136444503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f641806a280 con 0x7f6418108fc0 2024-05-27T08:00:53.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.995+0000 7f6414ff9640 1 -- 172.21.15.114:0/2136444503 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f64100051d0 con 0x7f6418108fc0 2024-05-27T08:00:53.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.996+0000 7f6414ff9640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64000b2960 0x7f64000b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:53.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.996+0000 7f64177fe640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64000b2960 0x7f64000b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:53.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.996+0000 7f6414ff9640 1 -- 172.21.15.114:0/2136444503 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 9466+0+0 (secure 0 0 0) 0x7f64100e2f40 con 0x7f6418108fc0 2024-05-27T08:00:53.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.996+0000 7f64177fe640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64000b2960 0x7f64000b4e70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6408000c00 tx=0x7f6408031400 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:53.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:52.999+0000 7f6414ff9640 1 -- 172.21.15.114:0/2136444503 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6410016030 con 0x7f6418108fc0 2024-05-27T08:00:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.184+0000 7f641dbb7640 1 -- 172.21.15.114:0/2136444503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f64181a7550 con 0x7f6418108fc0 2024-05-27T08:00:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.184+0000 7f6414ff9640 1 -- 172.21.15.114:0/2136444503 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v170) ==== 103+0+11 (secure 0 0 0) 0x7f64100aa570 con 0x7f6418108fc0 2024-05-27T08:00:53.186 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-05-27T08:00:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.186+0000 7f641dbb7640 1 -- 172.21.15.114:0/2136444503 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64000b2960 msgr2=0x7f64000b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.186+0000 7f641dbb7640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64000b2960 0x7f64000b4e70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6408000c00 tx=0x7f6408031400 comp rx=0 tx=0).stop 2024-05-27T08:00:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.186+0000 7f641dbb7640 1 -- 172.21.15.114:0/2136444503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6418108fc0 msgr2=0x7f641819c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.186+0000 7f641dbb7640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6418108fc0 0x7f641819c1d0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f641000eaa0 tx=0x7f641000ef60 comp rx=0 tx=0).stop 2024-05-27T08:00:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.186+0000 7f641dbb7640 1 -- 172.21.15.114:0/2136444503 shutdown_connections 2024-05-27T08:00:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.186+0000 7f641dbb7640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f64000b2960 0x7f64000b4e70 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.186+0000 7f641dbb7640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6418108fc0 0x7f641819c1d0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.186+0000 7f641dbb7640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6418108b20 0x7f641819bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.186+0000 7f641dbb7640 1 --2- 172.21.15.114:0/2136444503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6418103270 0x7f641819b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.186+0000 7f641dbb7640 1 -- 172.21.15.114:0/2136444503 >> 172.21.15.114:0/2136444503 conn(0x7f64180fe380 msgr2=0x7f64180fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.186+0000 7f641dbb7640 1 -- 172.21.15.114:0/2136444503 shutdown_connections 2024-05-27T08:00:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.186+0000 7f641dbb7640 1 -- 172.21.15.114:0/2136444503 wait complete. 2024-05-27T08:00:53.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-05-27T08:00:53.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.310+0000 7f1987fff640 1 -- 172.21.15.114:0/2075912149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1988109ef0 msgr2=0x7f1988103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:53.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.310+0000 7f1987fff640 1 --2- 172.21.15.114:0/2075912149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1988109ef0 0x7f1988103b40 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f1978009ca0 tx=0x7f197802a8c0 comp rx=0 tx=0).stop 2024-05-27T08:00:53.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.310+0000 7f1987fff640 1 -- 172.21.15.114:0/2075912149 shutdown_connections 2024-05-27T08:00:53.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.310+0000 7f1987fff640 1 --2- 172.21.15.114:0/2075912149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19881041b0 0x7f19881104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:53.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.310+0000 7f1987fff640 1 --2- 172.21.15.114:0/2075912149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1988109ef0 0x7f1988103b40 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:53.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.310+0000 7f1987fff640 1 --2- 172.21.15.114:0/2075912149 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1988109a50 0x7f1988103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:53.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.310+0000 7f1987fff640 1 -- 172.21.15.114:0/2075912149 >> 172.21.15.114:0/2075912149 conn(0x7f19880ff0e0 msgr2=0x7f1988101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:53.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.310+0000 7f1987fff640 1 -- 172.21.15.114:0/2075912149 shutdown_connections 2024-05-27T08:00:53.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.310+0000 7f1987fff640 1 -- 172.21.15.114:0/2075912149 wait complete. 2024-05-27T08:00:53.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.310+0000 7f1987fff640 1 Processor -- start 2024-05-27T08:00:53.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1987fff640 1 -- start start 2024-05-27T08:00:53.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1987fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19881041b0 0x7f1988074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:53.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1987fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1988109a50 0x7f1988075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:53.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1987fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1988109ef0 0x7f1988075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:53.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1987fff640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1988113c70 con 0x7f19881041b0 2024-05-27T08:00:53.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1987fff640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1988113af0 con 0x7f1988109ef0 2024-05-27T08:00:53.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1987fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1988113df0 con 0x7f1988109a50 2024-05-27T08:00:53.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1986ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19881041b0 0x7f1988074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:53.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1986ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19881041b0 0x7f1988074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36010/0 (socket says 172.21.15.114:36010) 2024-05-27T08:00:53.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1986ffd640 1 -- 172.21.15.114:0/1321248586 learned_addr learned my addr 172.21.15.114:0/1321248586 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:00:53.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f19867fc640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1988109a50 0x7f1988075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:53.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f19877fe640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1988109ef0 0x7f1988075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:53.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1986ffd640 1 -- 172.21.15.114:0/1321248586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1988109a50 msgr2=0x7f1988075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:53.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1986ffd640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1988109a50 0x7f1988075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1986ffd640 1 -- 172.21.15.114:0/1321248586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1988109ef0 msgr2=0x7f1988075a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1986ffd640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1988109ef0 0x7f1988075a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.311+0000 7f1986ffd640 1 -- 172.21.15.114:0/1321248586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19881a89b0 con 0x7f19881041b0 2024-05-27T08:00:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.312+0000 7f1986ffd640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19881041b0 0x7f1988074fc0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f197c00e9f0 tx=0x7f197c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.312+0000 7f198c9fd640 1 -- 172.21.15.114:0/1321248586 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c00ccc0 con 0x7f19881041b0 2024-05-27T08:00:53.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.312+0000 7f1987fff640 1 -- 172.21.15.114:0/1321248586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19881a8ca0 con 0x7f19881041b0 2024-05-27T08:00:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.312+0000 7f198c9fd640 1 -- 172.21.15.114:0/1321248586 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f197c0108e0 con 0x7f19881041b0 2024-05-27T08:00:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.312+0000 7f198c9fd640 1 -- 172.21.15.114:0/1321248586 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c018860 con 0x7f19881041b0 2024-05-27T08:00:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.312+0000 7f1987fff640 1 -- 172.21.15.114:0/1321248586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19881a91c0 con 0x7f19881041b0 2024-05-27T08:00:53.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.313+0000 7f1987fff640 1 -- 172.21.15.114:0/1321248586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1988069d00 con 0x7f19881041b0 2024-05-27T08:00:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.314+0000 7f198c9fd640 1 -- 172.21.15.114:0/1321248586 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f197c010430 con 0x7f19881041b0 2024-05-27T08:00:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.315+0000 7f198c9fd640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f19700b2950 0x7f19700b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:00:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.315+0000 7f19867fc640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f19700b2950 0x7f19700b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:00:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.315+0000 7f198c9fd640 1 -- 172.21.15.114:0/1321248586 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 9466+0+0 (secure 0 0 0) 0x7f197c0e3230 con 0x7f19881041b0 2024-05-27T08:00:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.316+0000 7f19867fc640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f19700b2950 0x7f19700b4e60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1978000f80 tx=0x7f1978009360 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:00:53.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.319+0000 7f198c9fd640 1 -- 172.21.15.114:0/1321248586 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f197c0aa860 con 0x7f19881041b0 2024-05-27T08:00:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.553+0000 7f1987fff640 1 -- 172.21.15.114:0/1321248586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f19881a9460 con 0x7f19881041b0 2024-05-27T08:00:53.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.562+0000 7f198c9fd640 1 -- 172.21.15.114:0/1321248586 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+816180 (secure 0 0 0) 0x7f197c0aa160 con 0x7f19881041b0 2024-05-27T08:00:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T08:00:53.576 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-05-27T08:00:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 25, 2024-05-27T08:00:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: "active_gid": 15138, 2024-05-27T08:00:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: "active_name": "x", 2024-05-27T08:00:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: "active_addrs": { 2024-05-27T08:00:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-05-27T08:00:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-05-27T08:00:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6800", 2024-05-27T08:00:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3311646739 2024-05-27T08:00:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-05-27T08:00:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6801", 2024-05-27T08:00:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3311646739 2024-05-27T08:00:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-05-27T08:00:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: "active_addr": "172.21.15.114:6801/3311646739", 2024-05-27T08:00:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: "active_change": "2024-05-27T07:39:44.126518+0000", 2024-05-27T08:00:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: "active_mgr_features": 4540701547738038271, 2024-05-27T08:00:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: "available": true, 2024-05-27T08:00:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: "standbys": [ 2024-05-27T08:00:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: "gid": 24824, 2024-05-27T08:00:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "y", 2024-05-27T08:00:53.580 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_features": 4540701547738038271, 2024-05-27T08:00:53.580 INFO:teuthology.orchestra.run.smithi114.stdout: "available_modules": [ 2024-05-27T08:00:53.580 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:53.580 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "alerts", 2024-05-27T08:00:53.580 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:53.580 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:53.580 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:53.580 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:53.580 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:53.581 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.581 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.581 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.581 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:53.581 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.581 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.581 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.581 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T08:00:53.582 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.582 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.582 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.582 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.582 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:53.582 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:53.582 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.583 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.583 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.583 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.583 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.583 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.583 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.583 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.584 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.584 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.584 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.584 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.584 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.584 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.584 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.585 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.585 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.585 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.585 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.585 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:53.585 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:53.585 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.585 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.586 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.586 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.586 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.586 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.586 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.586 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.586 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.586 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.587 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.587 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.587 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:53.587 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:53.587 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.587 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.587 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.587 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:53.588 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.588 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.588 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.588 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.588 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.588 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.588 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.589 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.589 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.589 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.589 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.589 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.589 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.589 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.590 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.590 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:53.590 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:53.590 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.590 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.590 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.591 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.591 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.591 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.591 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.591 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.591 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.591 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.591 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.592 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.592 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_destination": { 2024-05-27T08:00:53.592 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_destination", 2024-05-27T08:00:53.592 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.592 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.592 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.592 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.593 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.593 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.593 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.593 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Email address to send alerts to", 2024-05-27T08:00:53.593 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.593 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.593 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.594 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.594 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_from_name": { 2024-05-27T08:00:53.594 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_from_name", 2024-05-27T08:00:53.594 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.594 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.594 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.594 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "Ceph", 2024-05-27T08:00:53.594 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.595 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.595 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.595 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Email From: name", 2024-05-27T08:00:53.595 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.595 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.595 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.596 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_host": { 2024-05-27T08:00:53.596 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_host", 2024-05-27T08:00:53.596 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.596 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.596 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.596 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.596 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.597 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.597 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.597 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP server", 2024-05-27T08:00:53.597 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.597 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.597 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.597 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_password": { 2024-05-27T08:00:53.598 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_password", 2024-05-27T08:00:53.598 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.598 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.598 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.598 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.598 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.598 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.598 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.599 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Password to authenticate with", 2024-05-27T08:00:53.599 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.599 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.599 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.599 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_port": { 2024-05-27T08:00:53.599 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_port", 2024-05-27T08:00:53.600 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.600 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.600 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.600 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "465", 2024-05-27T08:00:53.600 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.600 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.600 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.600 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP port", 2024-05-27T08:00:53.601 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.601 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.601 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.601 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_sender": { 2024-05-27T08:00:53.601 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_sender", 2024-05-27T08:00:53.601 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.601 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.602 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.602 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.602 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.602 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.602 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.602 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP envelope sender", 2024-05-27T08:00:53.602 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.602 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.603 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.603 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_ssl": { 2024-05-27T08:00:53.603 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_ssl", 2024-05-27T08:00:53.603 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.603 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.603 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.604 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.604 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.604 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.604 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.604 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T08:00:53.604 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.604 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.604 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.604 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.605 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_user": { 2024-05-27T08:00:53.605 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_user", 2024-05-27T08:00:53.605 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.605 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.605 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.605 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.605 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.605 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.606 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.606 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "User to authenticate as", 2024-05-27T08:00:53.606 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.606 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.606 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.606 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.606 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:53.607 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:53.607 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.607 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:53.607 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.607 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:53.607 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.607 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.607 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.608 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.608 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.608 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.608 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.608 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.608 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.608 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.608 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:53.609 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "balancer", 2024-05-27T08:00:53.609 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:53.609 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:53.609 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:53.609 INFO:teuthology.orchestra.run.smithi114.stdout: "active": { 2024-05-27T08:00:53.609 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "active", 2024-05-27T08:00:53.609 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.610 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.610 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.610 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.610 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.610 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.610 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.610 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T08:00:53.610 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.611 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.611 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.611 INFO:teuthology.orchestra.run.smithi114.stdout: "begin_time": { 2024-05-27T08:00:53.611 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "begin_time", 2024-05-27T08:00:53.611 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.611 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.611 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.612 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0000", 2024-05-27T08:00:53.612 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.612 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.612 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.612 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T08:00:53.612 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T08:00:53.612 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.612 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.613 INFO:teuthology.orchestra.run.smithi114.stdout: "begin_weekday": { 2024-05-27T08:00:53.613 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "begin_weekday", 2024-05-27T08:00:53.613 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:53.613 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.613 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.613 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:53.613 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0", 2024-05-27T08:00:53.613 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "6", 2024-05-27T08:00:53.613 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.614 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T08:00:53.614 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T08:00:53.614 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.614 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.614 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.614 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_max_iterations": { 2024-05-27T08:00:53.618 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_max_iterations", 2024-05-27T08:00:53.618 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:53.619 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.619 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.619 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "25", 2024-05-27T08:00:53.619 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-05-27T08:00:53.620 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "250", 2024-05-27T08:00:53.620 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.620 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T08:00:53.620 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.621 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.621 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.621 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.621 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_metrics": { 2024-05-27T08:00:53.622 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_metrics", 2024-05-27T08:00:53.622 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.622 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.623 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.623 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T08:00:53.623 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.623 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.624 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.624 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T08:00:53.624 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T08:00:53.625 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.625 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.625 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.625 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_step": { 2024-05-27T08:00:53.626 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_step", 2024-05-27T08:00:53.626 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:53.626 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.626 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.627 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0.5", 2024-05-27T08:00:53.627 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0.001", 2024-05-27T08:00:53.627 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "0.999", 2024-05-27T08:00:53.628 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.628 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "aggressiveness of optimization", 2024-05-27T08:00:53.628 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T08:00:53.628 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.629 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.629 INFO:teuthology.orchestra.run.smithi114.stdout: "end_time": { 2024-05-27T08:00:53.629 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "end_time", 2024-05-27T08:00:53.630 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.630 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.630 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.630 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "2359", 2024-05-27T08:00:53.631 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.631 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.631 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.632 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T08:00:53.632 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T08:00:53.632 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.632 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.633 INFO:teuthology.orchestra.run.smithi114.stdout: "end_weekday": { 2024-05-27T08:00:53.633 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "end_weekday", 2024-05-27T08:00:53.633 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:53.634 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.634 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.634 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:53.634 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0", 2024-05-27T08:00:53.634 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "6", 2024-05-27T08:00:53.635 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.635 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T08:00:53.635 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T08:00:53.635 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.635 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.635 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.635 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:53.635 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:53.636 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.636 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.636 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.636 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.636 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.636 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.636 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.636 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.637 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.637 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.637 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.637 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.637 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.637 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.638 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.638 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.638 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.638 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.638 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:53.638 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:53.639 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.639 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.639 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.639 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.639 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.639 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.639 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.640 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.640 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.640 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.640 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.640 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.640 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:53.640 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:53.640 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.641 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.641 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.641 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:53.641 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.641 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.641 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.641 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.641 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.641 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.642 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.642 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.642 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.642 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.642 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.642 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.642 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.642 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.643 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:53.643 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:53.643 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.643 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.643 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.644 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.644 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.644 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.644 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.644 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.644 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.644 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.644 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.645 INFO:teuthology.orchestra.run.smithi114.stdout: "min_score": { 2024-05-27T08:00:53.645 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "min_score", 2024-05-27T08:00:53.645 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:53.645 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.645 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.645 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:53.646 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.646 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.646 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.646 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T08:00:53.646 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.646 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.646 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.647 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": { 2024-05-27T08:00:53.647 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mode", 2024-05-27T08:00:53.647 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.647 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.647 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.647 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "upmap", 2024-05-27T08:00:53.647 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.647 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.647 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: "crush-compat", 2024-05-27T08:00:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: "none", 2024-05-27T08:00:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: "read", 2024-05-27T08:00:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap", 2024-05-27T08:00:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap-read" 2024-05-27T08:00:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Balancer mode", 2024-05-27T08:00:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_ids": { 2024-05-27T08:00:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pool_ids", 2024-05-27T08:00:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.650 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.650 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.650 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.650 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.650 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.650 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T08:00:53.650 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.650 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.651 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.651 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:53.651 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:53.651 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.651 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.651 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.651 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T08:00:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.653 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:53.653 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:53.653 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.653 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:53.653 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.653 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:53.653 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.654 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.654 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.654 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.654 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.654 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.654 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.655 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap_max_deviation": { 2024-05-27T08:00:53.655 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "upmap_max_deviation", 2024-05-27T08:00:53.655 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.655 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.655 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.655 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-05-27T08:00:53.655 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-05-27T08:00:53.656 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.656 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.656 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T08:00:53.656 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T08:00:53.656 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.656 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.656 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap_max_optimizations": { 2024-05-27T08:00:53.657 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "upmap_max_optimizations", 2024-05-27T08:00:53.657 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:53.657 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.657 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.657 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:53.657 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.657 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.657 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.657 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T08:00:53.658 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.658 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.658 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.658 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.658 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:53.658 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephadm", 2024-05-27T08:00:53.659 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:53.659 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:53.659 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:53.659 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_down_multiplier": { 2024-05-27T08:00:53.659 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_down_multiplier", 2024-05-27T08:00:53.659 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:53.659 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.659 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.660 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3.0", 2024-05-27T08:00:53.660 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.660 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.660 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.660 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T08:00:53.660 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.660 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.660 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.661 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_refresh_rate": { 2024-05-27T08:00:53.661 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_refresh_rate", 2024-05-27T08:00:53.661 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.661 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.661 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.661 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "20", 2024-05-27T08:00:53.662 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.662 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.662 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.662 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T08:00:53.662 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.662 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.662 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.663 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_starting_port": { 2024-05-27T08:00:53.663 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_starting_port", 2024-05-27T08:00:53.663 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.663 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.663 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.663 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "4721", 2024-05-27T08:00:53.664 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.664 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.664 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.664 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T08:00:53.664 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.664 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.664 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.665 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_ptrace": { 2024-05-27T08:00:53.665 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_ptrace", 2024-05-27T08:00:53.665 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.665 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.665 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.665 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.665 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.665 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.666 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.666 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T08:00:53.666 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T08:00:53.666 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.666 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.666 INFO:teuthology.orchestra.run.smithi114.stdout: "autotune_interval": { 2024-05-27T08:00:53.667 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "autotune_interval", 2024-05-27T08:00:53.667 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.667 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.667 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.667 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:53.667 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.667 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.667 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.668 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T08:00:53.668 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.668 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.668 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.668 INFO:teuthology.orchestra.run.smithi114.stdout: "autotune_memory_target_ratio": { 2024-05-27T08:00:53.668 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T08:00:53.668 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:53.669 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.669 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.669 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0.7", 2024-05-27T08:00:53.669 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.670 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.670 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.670 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T08:00:53.670 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.671 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.671 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.671 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm_log_destination": { 2024-05-27T08:00:53.672 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephadm_log_destination", 2024-05-27T08:00:53.672 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.672 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.672 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.672 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.673 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.673 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.673 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.673 INFO:teuthology.orchestra.run.smithi114.stdout: "file", 2024-05-27T08:00:53.673 INFO:teuthology.orchestra.run.smithi114.stdout: "file,syslog", 2024-05-27T08:00:53.674 INFO:teuthology.orchestra.run.smithi114.stdout: "syslog" 2024-05-27T08:00:53.674 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.674 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T08:00:53.674 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.675 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.675 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.675 INFO:teuthology.orchestra.run.smithi114.stdout: "cgroups_split": { 2024-05-27T08:00:53.676 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cgroups_split", 2024-05-27T08:00:53.676 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.676 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.676 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.677 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.677 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.677 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.677 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.677 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T08:00:53.677 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.677 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.678 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.678 INFO:teuthology.orchestra.run.smithi114.stdout: "config_checks_enabled": { 2024-05-27T08:00:53.678 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_checks_enabled", 2024-05-27T08:00:53.678 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.678 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.678 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.679 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.679 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.679 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.679 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.679 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T08:00:53.679 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.679 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.679 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.680 INFO:teuthology.orchestra.run.smithi114.stdout: "config_dashboard": { 2024-05-27T08:00:53.680 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_dashboard", 2024-05-27T08:00:53.680 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.680 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.680 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.680 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.680 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.680 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.681 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.681 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T08:00:53.681 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.681 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.681 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.681 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_alertmanager": { 2024-05-27T08:00:53.681 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_alertmanager", 2024-05-27T08:00:53.682 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.682 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.682 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.682 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T08:00:53.682 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.682 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.682 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.682 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-05-27T08:00:53.683 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.683 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.683 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.683 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_base": { 2024-05-27T08:00:53.683 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_base", 2024-05-27T08:00:53.683 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.683 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.683 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.684 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T08:00:53.684 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.684 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.684 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.684 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Container image name, without the tag", 2024-05-27T08:00:53.684 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.684 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.684 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.685 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_elasticsearch": { 2024-05-27T08:00:53.685 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_elasticsearch", 2024-05-27T08:00:53.685 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.685 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.685 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.685 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T08:00:53.685 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.685 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.686 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.686 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "elasticsearch container image", 2024-05-27T08:00:53.686 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.686 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.686 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.686 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_grafana": { 2024-05-27T08:00:53.687 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_grafana", 2024-05-27T08:00:53.687 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.687 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.687 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.687 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T08:00:53.687 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.687 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.687 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.688 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-05-27T08:00:53.688 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.688 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.688 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.688 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.688 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_haproxy": { 2024-05-27T08:00:53.688 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_haproxy", 2024-05-27T08:00:53.688 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.688 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.689 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.689 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T08:00:53.689 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.689 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.689 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.689 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "HAproxy container image", 2024-05-27T08:00:53.689 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.690 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.690 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.690 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_agent": { 2024-05-27T08:00:53.690 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_agent", 2024-05-27T08:00:53.690 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.690 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.691 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.691 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T08:00:53.691 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.691 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.691 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.692 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger agent container image", 2024-05-27T08:00:53.692 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.692 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.692 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.692 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_collector": { 2024-05-27T08:00:53.693 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_collector", 2024-05-27T08:00:53.693 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.693 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.693 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.694 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T08:00:53.694 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.694 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.694 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.694 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger collector container image", 2024-05-27T08:00:53.694 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.694 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.695 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.695 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_query": { 2024-05-27T08:00:53.695 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_query", 2024-05-27T08:00:53.695 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.695 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.695 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.696 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T08:00:53.696 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.696 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.696 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.696 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger query container image", 2024-05-27T08:00:53.696 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.696 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.696 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.697 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_keepalived": { 2024-05-27T08:00:53.697 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_keepalived", 2024-05-27T08:00:53.697 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.697 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.697 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.698 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T08:00:53.698 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.698 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.698 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.698 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Keepalived container image", 2024-05-27T08:00:53.698 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.698 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.699 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.699 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_loki": { 2024-05-27T08:00:53.699 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_loki", 2024-05-27T08:00:53.699 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.699 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.699 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.699 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T08:00:53.699 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.700 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.700 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.700 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Loki container image", 2024-05-27T08:00:53.700 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.700 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.700 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.700 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_node_exporter": { 2024-05-27T08:00:53.701 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_node_exporter", 2024-05-27T08:00:53.701 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.701 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.701 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.701 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T08:00:53.701 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.702 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.702 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.702 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-05-27T08:00:53.702 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.702 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.702 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.702 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_nvmeof": { 2024-05-27T08:00:53.703 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_nvmeof", 2024-05-27T08:00:53.703 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.703 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.703 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.703 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T08:00:53.703 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.703 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.704 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.704 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Nvme-of container image", 2024-05-27T08:00:53.704 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.704 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.704 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.704 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_prometheus": { 2024-05-27T08:00:53.704 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_prometheus", 2024-05-27T08:00:53.705 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.705 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.705 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.705 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T08:00:53.705 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.705 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.705 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.705 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-05-27T08:00:53.705 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.706 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.706 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.706 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_promtail": { 2024-05-27T08:00:53.706 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_promtail", 2024-05-27T08:00:53.706 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.706 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.706 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.707 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T08:00:53.707 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.707 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.707 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.707 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Promtail container image", 2024-05-27T08:00:53.707 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.707 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.707 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.708 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_samba": { 2024-05-27T08:00:53.708 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_samba", 2024-05-27T08:00:53.708 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.708 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.708 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.708 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T08:00:53.708 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.708 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.709 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.709 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Samba/SMB container image", 2024-05-27T08:00:53.709 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.709 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.709 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.709 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_snmp_gateway": { 2024-05-27T08:00:53.709 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_snmp_gateway", 2024-05-27T08:00:53.709 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.710 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.710 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.710 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T08:00:53.710 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.710 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.710 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.710 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SNMP Gateway container image", 2024-05-27T08:00:53.710 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.711 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.711 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.711 INFO:teuthology.orchestra.run.smithi114.stdout: "container_init": { 2024-05-27T08:00:53.711 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_init", 2024-05-27T08:00:53.711 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.711 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.711 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.712 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.712 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.712 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.712 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.712 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T08:00:53.713 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.713 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.713 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.714 INFO:teuthology.orchestra.run.smithi114.stdout: "daemon_cache_timeout": { 2024-05-27T08:00:53.714 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "daemon_cache_timeout", 2024-05-27T08:00:53.714 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.714 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.715 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.715 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:53.715 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.716 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.716 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.716 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T08:00:53.716 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.717 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.717 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.717 INFO:teuthology.orchestra.run.smithi114.stdout: "default_cephadm_command_timeout": { 2024-05-27T08:00:53.717 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T08:00:53.718 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.718 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.718 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.718 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "900", 2024-05-27T08:00:53.719 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.719 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.719 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.720 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T08:00:53.720 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.720 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.720 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.721 INFO:teuthology.orchestra.run.smithi114.stdout: "default_registry": { 2024-05-27T08:00:53.721 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "default_registry", 2024-05-27T08:00:53.721 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.722 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.722 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.722 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io", 2024-05-27T08:00:53.722 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.723 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.723 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.723 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T08:00:53.724 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.724 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.724 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.724 INFO:teuthology.orchestra.run.smithi114.stdout: "device_cache_timeout": { 2024-05-27T08:00:53.725 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_cache_timeout", 2024-05-27T08:00:53.725 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.725 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.726 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.726 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1800", 2024-05-27T08:00:53.726 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.726 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.727 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.727 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache device inventory", 2024-05-27T08:00:53.727 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.727 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.727 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.728 INFO:teuthology.orchestra.run.smithi114.stdout: "device_enhanced_scan": { 2024-05-27T08:00:53.728 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_enhanced_scan", 2024-05-27T08:00:53.728 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.729 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.729 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.729 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.729 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.730 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.730 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.730 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T08:00:53.730 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.731 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.731 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.731 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.731 INFO:teuthology.orchestra.run.smithi114.stdout: "facts_cache_timeout": { 2024-05-27T08:00:53.732 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "facts_cache_timeout", 2024-05-27T08:00:53.732 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.732 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.733 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.733 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:53.733 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.733 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache host facts data", 2024-05-27T08:00:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "grafana_dashboards_path": { 2024-05-27T08:00:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "grafana_dashboards_path", 2024-05-27T08:00:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T08:00:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.738 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T08:00:53.738 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.738 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.738 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: "host_check_interval": { 2024-05-27T08:00:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "host_check_interval", 2024-05-27T08:00:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to perform a host check", 2024-05-27T08:00:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: "hw_monitoring": { 2024-05-27T08:00:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "hw_monitoring", 2024-05-27T08:00:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T08:00:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: "inventory_list_all": { 2024-05-27T08:00:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "inventory_list_all", 2024-05-27T08:00:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.747 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.747 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.747 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.747 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.747 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.748 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T08:00:53.748 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.748 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.748 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.749 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.749 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:53.749 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:53.750 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.750 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.750 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.750 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.751 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.751 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.751 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.751 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.752 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.752 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.752 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.752 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.753 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.753 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.753 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.753 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.753 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.754 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.754 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.754 INFO:teuthology.orchestra.run.smithi114.stdout: "log_refresh_metadata": { 2024-05-27T08:00:53.755 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_refresh_metadata", 2024-05-27T08:00:53.755 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.755 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.755 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.756 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.756 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.756 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.756 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.757 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T08:00:53.757 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.757 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.758 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.758 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.758 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:53.758 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:53.758 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.758 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.758 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.759 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.759 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.759 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.759 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.759 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T08:00:53.759 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.759 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.759 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.760 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.760 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:53.760 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:53.760 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.760 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.760 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.760 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:53.760 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.761 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.761 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.761 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.761 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.761 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.761 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.761 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.761 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.762 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.762 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.762 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.762 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.762 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.762 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.762 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:53.762 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:53.763 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.763 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.763 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.763 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.763 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.763 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.763 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.763 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.763 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.764 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.764 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.764 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.764 INFO:teuthology.orchestra.run.smithi114.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T08:00:53.764 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T08:00:53.764 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.764 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.764 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.765 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.765 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.765 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.765 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.765 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T08:00:53.765 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.765 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.765 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.765 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.766 INFO:teuthology.orchestra.run.smithi114.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T08:00:53.766 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T08:00:53.766 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.766 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.766 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.766 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "*", 2024-05-27T08:00:53.766 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.766 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.767 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.767 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T08:00:53.767 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.767 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.767 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.767 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.767 INFO:teuthology.orchestra.run.smithi114.stdout: "max_count_per_host": { 2024-05-27T08:00:53.767 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_count_per_host", 2024-05-27T08:00:53.768 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.768 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.768 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.768 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:53.768 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.768 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.768 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.768 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "max number of daemons per service per host", 2024-05-27T08:00:53.768 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.769 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.769 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.769 INFO:teuthology.orchestra.run.smithi114.stdout: "max_osd_draining_count": { 2024-05-27T08:00:53.769 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_osd_draining_count", 2024-05-27T08:00:53.769 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.769 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.769 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.769 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:53.770 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.770 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.770 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.770 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T08:00:53.770 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.770 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.770 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.771 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.771 INFO:teuthology.orchestra.run.smithi114.stdout: "migration_current": { 2024-05-27T08:00:53.771 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "migration_current", 2024-05-27T08:00:53.771 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.771 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.771 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.771 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.771 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.772 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.772 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.772 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "internal - do not modify", 2024-05-27T08:00:53.772 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.772 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.772 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.772 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.772 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": { 2024-05-27T08:00:53.773 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mode", 2024-05-27T08:00:53.773 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.773 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.773 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.773 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "root", 2024-05-27T08:00:53.773 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.773 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.773 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.774 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm-package", 2024-05-27T08:00:53.774 INFO:teuthology.orchestra.run.smithi114.stdout: "root" 2024-05-27T08:00:53.774 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.774 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T08:00:53.774 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.774 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.774 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.775 INFO:teuthology.orchestra.run.smithi114.stdout: "oob_default_addr": { 2024-05-27T08:00:53.775 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "oob_default_addr", 2024-05-27T08:00:53.775 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.775 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.775 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.775 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "169.254.1.1", 2024-05-27T08:00:53.775 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.775 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.776 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.776 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T08:00:53.776 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.776 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.776 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.776 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.776 INFO:teuthology.orchestra.run.smithi114.stdout: "prometheus_alerts_path": { 2024-05-27T08:00:53.776 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prometheus_alerts_path", 2024-05-27T08:00:53.776 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.777 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.777 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.777 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T08:00:53.777 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.777 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.777 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.777 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T08:00:53.777 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.778 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.778 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.778 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.778 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_insecure": { 2024-05-27T08:00:53.778 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_insecure", 2024-05-27T08:00:53.778 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.778 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.778 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.779 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.779 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.779 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.779 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.779 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T08:00:53.779 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.779 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.779 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.780 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.780 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_password": { 2024-05-27T08:00:53.780 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_password", 2024-05-27T08:00:53.780 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.780 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.780 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.780 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.780 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.780 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.781 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.781 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T08:00:53.781 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.781 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.781 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.781 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.781 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_url": { 2024-05-27T08:00:53.781 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_url", 2024-05-27T08:00:53.781 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.782 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.782 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.782 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.782 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.782 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.782 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.782 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T08:00:53.782 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.782 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.783 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.783 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.783 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_username": { 2024-05-27T08:00:53.783 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_username", 2024-05-27T08:00:53.783 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.783 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.783 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.783 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.783 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.784 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.784 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.784 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T08:00:53.784 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.784 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.784 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.784 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.784 INFO:teuthology.orchestra.run.smithi114.stdout: "secure_monitoring_stack": { 2024-05-27T08:00:53.784 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "secure_monitoring_stack", 2024-05-27T08:00:53.785 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.785 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.785 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.785 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.785 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.785 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.785 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.785 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T08:00:53.785 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.785 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.786 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.786 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.786 INFO:teuthology.orchestra.run.smithi114.stdout: "service_discovery_port": { 2024-05-27T08:00:53.786 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "service_discovery_port", 2024-05-27T08:00:53.786 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.786 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.786 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.786 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8765", 2024-05-27T08:00:53.786 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.787 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.787 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.787 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "cephadm service discovery port", 2024-05-27T08:00:53.787 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.787 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.787 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.787 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.787 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:53.787 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:53.788 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.788 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:53.788 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.788 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:53.788 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.788 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.788 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.788 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.789 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.789 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.789 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.789 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.789 INFO:teuthology.orchestra.run.smithi114.stdout: "ssh_config_file": { 2024-05-27T08:00:53.789 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssh_config_file", 2024-05-27T08:00:53.789 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.789 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.789 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.789 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.790 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.790 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.790 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.790 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T08:00:53.790 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.790 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.790 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.791 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.791 INFO:teuthology.orchestra.run.smithi114.stdout: "use_agent": { 2024-05-27T08:00:53.791 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "use_agent", 2024-05-27T08:00:53.791 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.791 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.791 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.791 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.791 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.792 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.792 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.792 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T08:00:53.792 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.792 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.792 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.792 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.792 INFO:teuthology.orchestra.run.smithi114.stdout: "use_repo_digest": { 2024-05-27T08:00:53.793 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "use_repo_digest", 2024-05-27T08:00:53.793 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.793 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.793 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.793 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.793 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.793 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.793 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.793 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T08:00:53.794 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.794 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.794 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.794 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_failed_host_check": { 2024-05-27T08:00:53.794 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_failed_host_check", 2024-05-27T08:00:53.794 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.795 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.795 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.795 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.795 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.795 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.795 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.795 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T08:00:53.795 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.796 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.796 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.796 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.796 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_stray_daemons": { 2024-05-27T08:00:53.796 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_stray_daemons", 2024-05-27T08:00:53.796 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.796 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.796 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.797 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.797 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.797 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.797 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.797 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T08:00:53.797 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.797 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.797 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.798 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_stray_hosts": { 2024-05-27T08:00:53.798 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_stray_hosts", 2024-05-27T08:00:53.798 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.798 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.798 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.798 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.798 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.799 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.799 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.799 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T08:00:53.799 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.799 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.799 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.799 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.799 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.800 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:53.800 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crash", 2024-05-27T08:00:53.800 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:53.800 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:53.800 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:53.800 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:53.800 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:53.801 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.801 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.801 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.801 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.801 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.801 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.801 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.801 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.802 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.802 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.802 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.802 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.802 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.802 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.802 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.803 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.803 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.803 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.803 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.803 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:53.803 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:53.803 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.803 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.804 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.804 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.804 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.804 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.804 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.804 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.804 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.804 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.805 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.805 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:53.805 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:53.805 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.805 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.805 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.805 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:53.805 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.806 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.806 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.806 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.806 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.806 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.806 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.806 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.807 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.807 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.807 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.807 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.807 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.807 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.807 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.807 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:53.808 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:53.808 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.808 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.808 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.808 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.808 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.808 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.808 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.809 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.809 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.809 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.809 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.809 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.809 INFO:teuthology.orchestra.run.smithi114.stdout: "retain_interval": { 2024-05-27T08:00:53.809 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "retain_interval", 2024-05-27T08:00:53.809 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.810 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.810 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.810 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "31536000", 2024-05-27T08:00:53.810 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.810 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.810 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.810 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T08:00:53.810 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.811 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.811 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.811 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:53.811 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:53.811 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.811 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:53.811 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.812 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:53.812 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.812 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.812 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.812 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.812 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.812 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.813 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.813 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_recent_interval": { 2024-05-27T08:00:53.813 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_recent_interval", 2024-05-27T08:00:53.813 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.813 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.813 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.813 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1209600", 2024-05-27T08:00:53.813 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.813 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.814 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.814 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T08:00:53.814 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.814 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.814 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.814 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.814 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.815 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:53.815 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "dashboard", 2024-05-27T08:00:53.815 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:53.815 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:53.815 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:53.815 INFO:teuthology.orchestra.run.smithi114.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T08:00:53.815 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T08:00:53.815 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.815 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.816 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.816 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:53.816 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.816 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.816 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.816 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.816 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.816 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.816 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.817 INFO:teuthology.orchestra.run.smithi114.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T08:00:53.817 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T08:00:53.817 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.817 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.817 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.817 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.817 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.817 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.818 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.818 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.818 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.818 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.818 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.818 INFO:teuthology.orchestra.run.smithi114.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T08:00:53.819 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T08:00:53.819 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.819 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.819 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.819 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.819 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.819 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.819 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.819 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.820 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.822 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.822 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.822 INFO:teuthology.orchestra.run.smithi114.stdout: "AUDIT_API_ENABLED": { 2024-05-27T08:00:53.822 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T08:00:53.822 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.822 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.823 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.823 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.823 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.823 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.823 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.823 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.823 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.823 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.824 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.824 INFO:teuthology.orchestra.run.smithi114.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T08:00:53.824 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T08:00:53.824 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.824 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.824 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.824 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.825 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.825 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.825 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.825 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.825 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.825 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.825 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.825 INFO:teuthology.orchestra.run.smithi114.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T08:00:53.826 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T08:00:53.826 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.826 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.826 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.826 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.826 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.826 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.826 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.827 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.827 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.827 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.827 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.827 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T08:00:53.827 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T08:00:53.827 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.827 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.828 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.828 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.828 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.828 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.828 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.828 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.828 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.828 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.829 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.829 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T08:00:53.829 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T08:00:53.829 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.829 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.829 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.829 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.830 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.830 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.830 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.830 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.830 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.830 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.830 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.830 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T08:00:53.831 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T08:00:53.831 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.831 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.831 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.831 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.831 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.831 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.831 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.832 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.832 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.832 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.832 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.832 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T08:00:53.832 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T08:00:53.832 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.832 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.833 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.833 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.833 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.833 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.833 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.833 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.833 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.833 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.834 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.834 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.834 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T08:00:53.834 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T08:00:53.834 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.834 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.834 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.834 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.835 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.835 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.835 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.835 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.835 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.835 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.835 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.835 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T08:00:53.836 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T08:00:53.836 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.836 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.836 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.836 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.836 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.836 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.836 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.837 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.837 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.837 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.837 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.837 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T08:00:53.837 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T08:00:53.837 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.837 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.838 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.838 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.838 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.838 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.838 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.838 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.838 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.838 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.839 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.839 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.839 INFO:teuthology.orchestra.run.smithi114.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T08:00:53.839 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T08:00:53.839 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.839 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.839 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.839 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.840 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.840 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.840 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.840 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.840 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.840 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.840 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.841 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T08:00:53.841 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T08:00:53.841 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.841 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.841 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.841 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-05-27T08:00:53.841 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.841 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.841 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.842 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.842 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.842 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.842 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.842 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T08:00:53.842 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T08:00:53.843 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.843 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.843 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.843 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.843 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.843 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.843 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.843 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.844 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.844 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.844 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.844 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_URL": { 2024-05-27T08:00:53.844 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_URL", 2024-05-27T08:00:53.844 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.844 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.845 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.845 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.845 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.845 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.845 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.845 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.845 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.845 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.845 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.846 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T08:00:53.846 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T08:00:53.846 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.846 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.846 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.846 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-05-27T08:00:53.846 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.847 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.847 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.847 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.847 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.847 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.847 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.847 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T08:00:53.847 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T08:00:53.848 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.848 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.848 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.848 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.848 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.848 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.848 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.848 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.849 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.849 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.849 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.849 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T08:00:53.849 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T08:00:53.849 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.849 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.849 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.850 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.850 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.850 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.850 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.850 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.850 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.850 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.850 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.851 INFO:teuthology.orchestra.run.smithi114.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T08:00:53.851 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T08:00:53.851 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.851 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.851 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.851 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.851 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.852 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.852 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.852 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.852 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.852 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.852 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.852 INFO:teuthology.orchestra.run.smithi114.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T08:00:53.852 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T08:00:53.853 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.853 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.853 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.853 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.853 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.853 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.853 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.853 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.853 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.854 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.854 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.854 INFO:teuthology.orchestra.run.smithi114.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T08:00:53.854 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T08:00:53.854 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.854 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.854 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.855 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "[]", 2024-05-27T08:00:53.855 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.855 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.855 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.855 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.855 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.855 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.855 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.856 INFO:teuthology.orchestra.run.smithi114.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T08:00:53.856 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T08:00:53.856 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.856 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.856 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.856 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "{}", 2024-05-27T08:00:53.856 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.856 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.857 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.857 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.857 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.857 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.857 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.857 INFO:teuthology.orchestra.run.smithi114.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T08:00:53.857 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T08:00:53.857 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.858 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.858 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.858 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.858 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.858 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.858 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.858 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.858 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.859 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.859 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.859 INFO:teuthology.orchestra.run.smithi114.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T08:00:53.859 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T08:00:53.859 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.859 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.859 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.860 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.860 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.860 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.860 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.860 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.860 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.860 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.860 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.861 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.861 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T08:00:53.861 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T08:00:53.861 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.861 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.861 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.861 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.861 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.862 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.862 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.862 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.862 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.862 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.862 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.862 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T08:00:53.862 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T08:00:53.863 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.863 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.863 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.863 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.863 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.863 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.863 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.863 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.864 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.864 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.864 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.864 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T08:00:53.864 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T08:00:53.864 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.864 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.865 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.865 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.865 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.865 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.865 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.865 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.865 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.865 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.866 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.866 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T08:00:53.866 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T08:00:53.866 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.866 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.866 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.866 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.867 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.867 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.867 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.867 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.867 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.867 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.867 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.868 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T08:00:53.868 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T08:00:53.868 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.868 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.868 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.868 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.868 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.868 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.868 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.869 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.869 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.869 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.869 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.869 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T08:00:53.869 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T08:00:53.869 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.869 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.870 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.870 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.870 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.870 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.870 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.870 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.870 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.870 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.871 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.871 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.871 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T08:00:53.871 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T08:00:53.871 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.871 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.871 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.871 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.872 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.872 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.872 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.872 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.872 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.872 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.872 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.873 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T08:00:53.873 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T08:00:53.873 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.873 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.873 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.873 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.873 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.873 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.873 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.874 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.874 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.874 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.874 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.874 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T08:00:53.874 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T08:00:53.874 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.875 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.875 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.875 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T08:00:53.875 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.875 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.875 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.875 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.875 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.875 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.876 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.876 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T08:00:53.876 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T08:00:53.876 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.876 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.876 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.876 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:53.877 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.877 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.877 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.877 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.877 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.877 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.877 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.878 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T08:00:53.878 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T08:00:53.878 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.878 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.878 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.878 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8", 2024-05-27T08:00:53.878 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.878 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.879 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.879 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.879 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.879 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.879 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.879 INFO:teuthology.orchestra.run.smithi114.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T08:00:53.879 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T08:00:53.879 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.880 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.880 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.880 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "45", 2024-05-27T08:00:53.880 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.880 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.880 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.880 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.880 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.881 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.881 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.881 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T08:00:53.881 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T08:00:53.881 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.881 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.881 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.881 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.882 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.882 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.882 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.882 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.882 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.882 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.882 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.883 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T08:00:53.883 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T08:00:53.883 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.883 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.883 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.883 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-05-27T08:00:53.883 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.884 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.884 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.884 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.884 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.884 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.884 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.885 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T08:00:53.885 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T08:00:53.885 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.885 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.885 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.885 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.885 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.885 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.886 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.886 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.886 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.886 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.886 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.886 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.886 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T08:00:53.887 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T08:00:53.887 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.887 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.887 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.887 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.887 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.887 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.888 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.888 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.888 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.888 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.888 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.888 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.888 INFO:teuthology.orchestra.run.smithi114.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T08:00:53.889 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T08:00:53.889 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.889 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.889 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.889 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.889 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.889 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.890 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.890 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.890 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.890 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.890 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.890 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T08:00:53.890 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T08:00:53.891 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.891 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.891 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.891 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:53.891 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.891 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.891 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.892 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.892 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.892 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.892 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.892 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.892 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T08:00:53.892 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T08:00:53.893 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.893 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.893 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.893 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:53.893 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.893 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.893 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.893 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.894 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.894 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.894 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.894 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T08:00:53.894 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T08:00:53.895 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.895 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.895 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.895 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-05-27T08:00:53.895 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.895 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.895 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.895 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.896 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.896 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.896 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.896 INFO:teuthology.orchestra.run.smithi114.stdout: "cross_origin_url": { 2024-05-27T08:00:53.896 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cross_origin_url", 2024-05-27T08:00:53.896 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.897 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.897 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.897 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.897 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.897 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.897 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.897 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.897 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.898 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.898 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.898 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.898 INFO:teuthology.orchestra.run.smithi114.stdout: "crt_file": { 2024-05-27T08:00:53.898 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crt_file", 2024-05-27T08:00:53.898 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.898 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.899 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.899 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.899 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.899 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.899 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.899 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.899 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.900 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.900 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.900 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.900 INFO:teuthology.orchestra.run.smithi114.stdout: "debug": { 2024-05-27T08:00:53.900 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "debug", 2024-05-27T08:00:53.900 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.900 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.900 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.900 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.901 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.901 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.901 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.901 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable/disable debug options", 2024-05-27T08:00:53.901 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.901 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.901 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.901 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.902 INFO:teuthology.orchestra.run.smithi114.stdout: "jwt_token_ttl": { 2024-05-27T08:00:53.902 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "jwt_token_ttl", 2024-05-27T08:00:53.902 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.902 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.902 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.902 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "28800", 2024-05-27T08:00:53.902 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.902 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.903 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.903 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.903 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.903 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.903 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.903 INFO:teuthology.orchestra.run.smithi114.stdout: "key_file": { 2024-05-27T08:00:53.903 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "key_file", 2024-05-27T08:00:53.903 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.904 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.904 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.904 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.904 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.904 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.904 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.904 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.904 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.905 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.905 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.905 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.905 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:53.905 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:53.905 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.905 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.905 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.905 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.906 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.906 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.906 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.906 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.906 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.906 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.906 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.906 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.907 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.907 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.907 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.907 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.907 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.907 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.908 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:53.908 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:53.908 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.908 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.908 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.908 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.908 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.909 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.909 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.909 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.909 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.909 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.909 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.909 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:53.909 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:53.910 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.910 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.910 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.910 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:53.910 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.910 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.910 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.910 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.911 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.911 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.911 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.911 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.911 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.911 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.911 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.912 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.912 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.912 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.912 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:53.912 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:53.912 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.913 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.913 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.913 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.913 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.913 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.913 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.913 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.913 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.913 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.914 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.914 INFO:teuthology.orchestra.run.smithi114.stdout: "motd": { 2024-05-27T08:00:53.914 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "motd", 2024-05-27T08:00:53.914 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.914 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.914 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.915 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.915 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.915 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.915 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.915 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "The message of the day", 2024-05-27T08:00:53.915 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.915 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.915 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.916 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.916 INFO:teuthology.orchestra.run.smithi114.stdout: "redirect_resolve_ip_addr": { 2024-05-27T08:00:53.916 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T08:00:53.916 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.916 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.916 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.916 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.916 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.917 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.917 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.917 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.917 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.917 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.917 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.918 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.918 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-05-27T08:00:53.918 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-05-27T08:00:53.918 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.918 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.918 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.918 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "::", 2024-05-27T08:00:53.918 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.919 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.919 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.919 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.919 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.919 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.919 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.919 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.920 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-05-27T08:00:53.920 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-05-27T08:00:53.920 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.920 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.920 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.920 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8080", 2024-05-27T08:00:53.920 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.921 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.921 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.921 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.921 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.921 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.921 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.922 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:53.922 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:53.922 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.922 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:53.922 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.922 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:53.923 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.923 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.923 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.923 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.923 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.923 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.923 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.924 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl": { 2024-05-27T08:00:53.924 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl", 2024-05-27T08:00:53.924 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.924 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.924 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.924 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.924 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.924 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.925 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.925 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.925 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.925 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.925 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.925 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.925 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl_server_port": { 2024-05-27T08:00:53.925 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl_server_port", 2024-05-27T08:00:53.926 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.926 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.926 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.926 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8443", 2024-05-27T08:00:53.926 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.926 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.926 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.926 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.927 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.927 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.927 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.927 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_behaviour": { 2024-05-27T08:00:53.927 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_behaviour", 2024-05-27T08:00:53.927 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.927 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.927 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.928 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "redirect", 2024-05-27T08:00:53.928 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.928 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.928 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.928 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.928 INFO:teuthology.orchestra.run.smithi114.stdout: "redirect" 2024-05-27T08:00:53.928 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.928 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.929 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.929 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.929 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.929 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_error_status_code": { 2024-05-27T08:00:53.929 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_error_status_code", 2024-05-27T08:00:53.929 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.929 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.930 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.930 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "500", 2024-05-27T08:00:53.930 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "400", 2024-05-27T08:00:53.930 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "599", 2024-05-27T08:00:53.930 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.930 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.930 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.930 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.931 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.931 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.931 INFO:teuthology.orchestra.run.smithi114.stdout: "url_prefix": { 2024-05-27T08:00:53.931 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "url_prefix", 2024-05-27T08:00:53.931 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.931 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.931 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.932 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.932 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.932 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.932 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.932 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.932 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.932 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.932 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.933 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.933 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.933 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:53.933 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "devicehealth", 2024-05-27T08:00:53.933 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:53.933 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:53.933 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:53.933 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_monitoring": { 2024-05-27T08:00:53.934 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enable_monitoring", 2024-05-27T08:00:53.934 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.934 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.934 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.934 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.934 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.934 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.934 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.935 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "monitor device health metrics", 2024-05-27T08:00:53.935 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.935 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.935 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.935 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:53.935 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:53.935 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.936 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.936 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.936 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.936 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.936 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.936 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.936 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.937 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.937 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.937 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.937 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.937 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.937 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.937 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.937 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.938 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.938 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.938 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:53.938 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:53.938 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.938 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.938 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.939 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.939 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.939 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.939 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.939 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.939 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.939 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.939 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.940 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:53.940 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:53.940 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.940 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.940 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.940 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:53.940 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.941 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.941 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.941 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.941 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.941 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.941 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.941 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.941 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.942 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.942 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.942 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.942 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.942 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.942 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:53.942 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:53.943 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.943 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.943 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.943 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.943 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.943 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.943 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.943 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.944 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.944 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.944 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.944 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.944 INFO:teuthology.orchestra.run.smithi114.stdout: "mark_out_threshold": { 2024-05-27T08:00:53.944 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mark_out_threshold", 2024-05-27T08:00:53.944 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.944 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.945 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.945 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "2419200", 2024-05-27T08:00:53.945 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.945 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.945 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.945 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T08:00:53.945 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.945 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.945 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.946 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.946 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": { 2024-05-27T08:00:53.946 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pool_name", 2024-05-27T08:00:53.946 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.946 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.946 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.946 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "device_health_metrics", 2024-05-27T08:00:53.947 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.947 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.947 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.947 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T08:00:53.947 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.947 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.947 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.947 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.948 INFO:teuthology.orchestra.run.smithi114.stdout: "retention_period": { 2024-05-27T08:00:53.948 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "retention_period", 2024-05-27T08:00:53.948 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.948 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.948 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.948 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15552000", 2024-05-27T08:00:53.948 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.948 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.949 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.949 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long to retain device health metrics", 2024-05-27T08:00:53.949 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.949 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.949 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.949 INFO:teuthology.orchestra.run.smithi114.stdout: "scrape_frequency": { 2024-05-27T08:00:53.949 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "scrape_frequency", 2024-05-27T08:00:53.950 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.950 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.950 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.950 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "86400", 2024-05-27T08:00:53.950 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.950 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.950 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.950 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T08:00:53.951 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.951 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.951 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.951 INFO:teuthology.orchestra.run.smithi114.stdout: "self_heal": { 2024-05-27T08:00:53.951 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "self_heal", 2024-05-27T08:00:53.951 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.951 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T08:00:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T08:00:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_threshold": { 2024-05-27T08:00:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_threshold", 2024-05-27T08:00:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "7257600", 2024-05-27T08:00:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T08:00:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "diskprediction_local", 2024-05-27T08:00:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "predict_interval": { 2024-05-27T08:00:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "predict_interval", 2024-05-27T08:00:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "86400", 2024-05-27T08:00:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "predictor_model": { 2024-05-27T08:00:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "predictor_model", 2024-05-27T08:00:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.970 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.970 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.970 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "prophetstor", 2024-05-27T08:00:53.970 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.970 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.970 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.970 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.970 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.971 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.971 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.971 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:53.971 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:53.971 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.971 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.971 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.971 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:53.972 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.972 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.972 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.972 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.972 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.972 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.972 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.973 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:53.973 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:53.973 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.973 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:53.973 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.973 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:53.973 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.973 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.973 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.974 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.974 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.974 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.974 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.974 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.974 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:53.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:53.975 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "influx", 2024-05-27T08:00:53.975 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": false, 2024-05-27T08:00:53.975 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "influxdb python module not found", 2024-05-27T08:00:53.975 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:53.975 INFO:teuthology.orchestra.run.smithi114.stdout: "batch_size": { 2024-05-27T08:00:53.975 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "batch_size", 2024-05-27T08:00:53.975 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.975 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.976 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.976 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5000", 2024-05-27T08:00:53.976 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.976 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.976 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.976 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T08:00:53.976 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.976 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.976 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.977 INFO:teuthology.orchestra.run.smithi114.stdout: "database": { 2024-05-27T08:00:53.977 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "database", 2024-05-27T08:00:53.977 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.977 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.977 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.977 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "ceph", 2024-05-27T08:00:53.977 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.978 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.978 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.978 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T08:00:53.978 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.978 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.978 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.978 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.978 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": { 2024-05-27T08:00:53.979 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "hostname", 2024-05-27T08:00:53.979 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.979 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.979 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.979 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.979 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.979 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.979 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.980 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB server hostname", 2024-05-27T08:00:53.980 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.980 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.980 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.980 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:53.980 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:53.980 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:53.981 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.981 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.981 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "30", 2024-05-27T08:00:53.981 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "5", 2024-05-27T08:00:53.981 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.981 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.981 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T08:00:53.981 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.982 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.982 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.982 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:53.982 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:53.982 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.982 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.983 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.983 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.983 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.983 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.983 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.983 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.983 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.984 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.984 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.984 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.984 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.984 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.984 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.984 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.984 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.985 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.985 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.985 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:53.985 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:53.985 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.985 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.985 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.985 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.986 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.986 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.986 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.986 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.986 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.986 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.986 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.987 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:53.987 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:53.987 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.987 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.987 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.987 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:53.987 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.987 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.987 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:53.988 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:53.988 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:53.988 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:53.988 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:53.988 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:53.988 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:53.988 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:53.988 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.989 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.989 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.989 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.989 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:53.989 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:53.989 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:53.989 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.990 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.990 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:53.990 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.990 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.990 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.990 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.990 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.990 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.991 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.991 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.991 INFO:teuthology.orchestra.run.smithi114.stdout: "password": { 2024-05-27T08:00:53.991 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "password", 2024-05-27T08:00:53.991 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.991 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.991 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.991 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:53.992 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.992 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.992 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.992 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "password of InfluxDB server user", 2024-05-27T08:00:53.992 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.992 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.992 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.992 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.993 INFO:teuthology.orchestra.run.smithi114.stdout: "port": { 2024-05-27T08:00:53.993 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "port", 2024-05-27T08:00:53.993 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.993 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.993 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.993 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8086", 2024-05-27T08:00:53.993 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.993 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.994 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.994 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB server port", 2024-05-27T08:00:53.994 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.994 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.994 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.994 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:53.994 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:53.995 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.995 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:53.995 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:53.995 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:53.995 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.995 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.995 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.995 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:53.996 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.996 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.996 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.996 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl": { 2024-05-27T08:00:53.996 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl", 2024-05-27T08:00:53.996 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:53.996 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.996 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.997 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "false", 2024-05-27T08:00:53.997 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:53.997 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:53.997 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.997 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T08:00:53.997 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.997 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.997 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.998 INFO:teuthology.orchestra.run.smithi114.stdout: "threads": { 2024-05-27T08:00:53.998 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "threads", 2024-05-27T08:00:53.998 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:53.998 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:53.998 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:53.998 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-05-27T08:00:53.998 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-05-27T08:00:53.998 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "32", 2024-05-27T08:00:53.999 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:53.999 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T08:00:53.999 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:53.999 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:53.999 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:53.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:53.999 INFO:teuthology.orchestra.run.smithi114.stdout: "username": { 2024-05-27T08:00:53.999 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "username", 2024-05-27T08:00:54.000 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.000 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.000 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.000 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.000 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.000 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.000 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.000 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "username of InfluxDB server user", 2024-05-27T08:00:54.001 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.001 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.001 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.001 INFO:teuthology.orchestra.run.smithi114.stdout: "verify_ssl": { 2024-05-27T08:00:54.001 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "verify_ssl", 2024-05-27T08:00:54.001 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.001 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.001 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.002 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "true", 2024-05-27T08:00:54.002 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.002 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.002 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.002 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T08:00:54.002 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.002 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.003 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.003 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.003 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.003 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "insights", 2024-05-27T08:00:54.003 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.003 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.004 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.004 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.004 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.004 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.004 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.004 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.004 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.004 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.005 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.005 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.005 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.005 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.005 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.005 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.005 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.005 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.005 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.006 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.006 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.006 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.006 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.006 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.006 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.007 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.007 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.007 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.007 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.007 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.007 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.007 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.007 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.007 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.008 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.008 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.008 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.008 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.008 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.008 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.008 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.008 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.009 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.009 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.009 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.009 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.009 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.009 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.009 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.009 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.010 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.010 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.010 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.010 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.010 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.010 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.010 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.011 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.011 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.011 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.011 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.011 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.011 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.011 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.011 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.011 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.012 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.012 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.012 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.012 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.012 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.012 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.012 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.013 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.013 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.013 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.013 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.013 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.013 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.013 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.013 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.014 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.014 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.014 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.014 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.014 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "iostat", 2024-05-27T08:00:54.014 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.015 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.015 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.015 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.015 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.015 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.015 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.015 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.015 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.015 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.016 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.016 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.016 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.016 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.016 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.016 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.016 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.016 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.016 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.017 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.017 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.017 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.017 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.017 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.017 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.017 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.018 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.018 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.018 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.018 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.018 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.018 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.018 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.018 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.019 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.019 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.019 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.019 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.019 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.019 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.019 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.019 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.020 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.020 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.020 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.020 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.020 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.020 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.020 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.020 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.021 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.021 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.021 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.021 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.021 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.021 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.021 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.021 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.022 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.022 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.022 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.022 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.022 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.022 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.022 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.022 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.023 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.023 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.023 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.023 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.023 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.023 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.024 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.024 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.024 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.024 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.024 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.024 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.024 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.024 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.024 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.025 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.025 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.025 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.025 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "k8sevents", 2024-05-27T08:00:54.025 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.025 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.026 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.026 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_event_retention_days": { 2024-05-27T08:00:54.026 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ceph_event_retention_days", 2024-05-27T08:00:54.026 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.026 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.026 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.026 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "7", 2024-05-27T08:00:54.026 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.027 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.027 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.027 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T08:00:54.027 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.027 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.027 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.027 INFO:teuthology.orchestra.run.smithi114.stdout: "config_check_secs": { 2024-05-27T08:00:54.028 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_check_secs", 2024-05-27T08:00:54.028 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.028 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.028 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.028 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:54.028 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "10", 2024-05-27T08:00:54.028 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.028 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.028 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T08:00:54.029 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.029 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.029 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.029 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.029 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.029 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.030 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.030 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.030 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.030 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.030 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.030 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.030 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.030 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.031 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.031 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.031 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.031 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.031 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.031 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.031 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.031 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.031 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.032 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.032 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.032 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.032 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.032 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.032 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.032 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.033 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.033 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.033 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.033 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.033 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.033 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.033 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.033 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.034 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.034 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.034 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.034 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.034 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.034 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.034 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.034 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.035 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.035 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.035 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.035 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.035 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.035 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.035 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.035 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.036 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.036 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.036 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.036 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.036 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.036 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.036 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.037 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.037 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.037 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.037 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.037 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.037 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.037 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.038 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.038 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.038 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.038 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.038 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.038 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.038 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.038 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.039 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.039 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.039 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.039 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.039 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.039 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.039 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.039 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.040 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "localpool", 2024-05-27T08:00:54.040 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.040 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.040 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.040 INFO:teuthology.orchestra.run.smithi114.stdout: "failure_domain": { 2024-05-27T08:00:54.040 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "failure_domain", 2024-05-27T08:00:54.040 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.041 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.041 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.041 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "host", 2024-05-27T08:00:54.041 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.041 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.041 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.041 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "failure domain for any created local pool", 2024-05-27T08:00:54.041 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T08:00:54.042 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.042 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.042 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.042 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.042 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.042 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.042 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.043 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.043 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.043 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.043 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.043 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.043 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.043 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.043 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.043 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.044 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.044 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.044 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.044 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.044 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.044 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.044 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.045 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.045 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.045 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.045 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.045 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.045 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.045 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.046 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.046 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.046 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.046 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.046 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.046 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.046 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.047 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.047 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.047 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.047 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.047 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.047 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.047 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.048 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.048 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.048 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.048 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.048 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.048 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.048 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.049 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.049 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.049 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.049 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.049 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.049 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.049 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.050 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.050 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.050 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.050 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.050 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.050 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.050 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.050 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.051 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.051 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.051 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": { 2024-05-27T08:00:54.051 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "min_size", 2024-05-27T08:00:54.051 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.051 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.051 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.052 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.052 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.052 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.052 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.052 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default min_size for any created local pool", 2024-05-27T08:00:54.052 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T08:00:54.052 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.053 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.053 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rep": { 2024-05-27T08:00:54.053 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "num_rep", 2024-05-27T08:00:54.053 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.053 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.053 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.054 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3", 2024-05-27T08:00:54.054 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.054 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.054 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.054 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default replica count for any created local pool", 2024-05-27T08:00:54.054 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.054 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.055 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.055 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": { 2024-05-27T08:00:54.055 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pg_num", 2024-05-27T08:00:54.055 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.055 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.055 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.055 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "128", 2024-05-27T08:00:54.056 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.056 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.056 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.056 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T08:00:54.056 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.056 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.056 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.057 INFO:teuthology.orchestra.run.smithi114.stdout: "prefix": { 2024-05-27T08:00:54.057 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prefix", 2024-05-27T08:00:54.057 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.057 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.057 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.057 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.057 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.057 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.058 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.058 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "name prefix for any created local pool", 2024-05-27T08:00:54.058 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.058 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.058 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.058 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.058 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.059 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.059 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.059 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.059 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.059 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.059 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.059 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.059 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.060 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.060 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.060 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.060 INFO:teuthology.orchestra.run.smithi114.stdout: "subtree": { 2024-05-27T08:00:54.060 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "subtree", 2024-05-27T08:00:54.060 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.060 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.060 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.061 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "rack", 2024-05-27T08:00:54.061 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.061 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.061 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.061 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T08:00:54.061 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T08:00:54.061 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.061 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.062 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.062 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.062 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mds_autoscaler", 2024-05-27T08:00:54.062 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.062 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.062 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.063 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.063 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.063 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.063 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.063 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.063 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.063 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.063 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.064 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.064 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.064 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.064 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.064 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.064 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.064 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.064 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.065 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.065 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.065 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.065 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.065 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.065 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.065 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.066 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.066 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.066 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.066 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.066 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.066 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.066 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.066 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.066 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.067 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.067 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.067 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.067 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.067 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.067 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.067 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.068 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.068 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.068 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.068 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.068 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.068 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.068 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.068 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.068 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.069 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.069 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.069 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.069 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.069 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.069 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.069 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.070 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.070 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.070 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.070 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.070 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.070 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.070 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.070 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.070 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.071 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.071 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.071 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.071 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.071 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.071 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.071 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.072 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.072 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.072 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.072 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.072 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.072 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.072 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.072 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.072 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.073 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.073 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.073 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mirroring", 2024-05-27T08:00:54.073 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.073 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.073 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.073 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.074 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.074 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.074 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.074 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.074 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.074 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.074 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.074 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.075 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.075 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.075 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.075 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.075 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.075 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.075 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.076 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.076 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.076 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.076 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.076 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.076 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.077 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.077 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.077 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.077 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.077 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.077 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.077 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.078 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.078 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.078 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.078 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.078 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.078 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.079 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.079 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.079 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.079 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.079 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.079 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.079 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.079 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.080 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.080 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.080 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.080 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.080 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.080 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.080 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.080 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.081 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.081 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.081 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.081 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.081 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.081 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.081 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.082 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.082 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.082 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.082 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.082 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.082 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.082 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.082 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.083 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.083 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.083 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.083 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.083 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.083 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.083 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.083 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.084 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.084 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.084 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.084 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.084 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.084 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.084 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.085 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "nfs", 2024-05-27T08:00:54.085 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.085 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.085 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.085 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.085 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.085 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.086 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.086 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.086 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.086 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.086 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.086 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.086 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.086 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.087 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.087 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.087 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.087 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.087 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.087 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.087 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.087 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.087 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.088 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.088 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.088 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.088 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.088 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.088 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.088 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.089 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.089 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.089 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.089 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.089 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.089 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.089 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.090 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.090 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.090 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.090 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.090 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.090 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.090 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.090 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.091 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.091 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.091 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.091 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.091 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.091 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.091 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.092 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.092 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.092 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.092 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.092 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.093 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.093 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.093 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.093 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.093 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.093 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.093 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.093 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.094 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.094 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.094 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.094 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.095 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.095 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.095 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.095 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.095 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.095 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.095 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.096 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.096 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.096 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.096 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.096 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.096 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.096 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.096 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.097 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "orchestrator", 2024-05-27T08:00:54.097 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.097 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.097 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.097 INFO:teuthology.orchestra.run.smithi114.stdout: "fail_fs": { 2024-05-27T08:00:54.097 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "fail_fs", 2024-05-27T08:00:54.097 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.098 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.098 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.098 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.098 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.098 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.098 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.098 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T08:00:54.098 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.099 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.099 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.099 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.099 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.099 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.099 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.099 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.099 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.100 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.100 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.100 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.100 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.100 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.100 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.100 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.100 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.101 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.101 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.101 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.101 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.101 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.101 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.101 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.102 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.102 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.102 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.102 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.102 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.102 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.102 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.103 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.103 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.103 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.103 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.103 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.103 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.103 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.103 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.104 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.104 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.104 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.104 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.104 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.104 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.104 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.104 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.105 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.105 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.105 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.105 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.105 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.105 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.105 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.105 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.105 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.106 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.106 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.106 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.106 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.106 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.106 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.106 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.107 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.107 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.107 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.107 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.107 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.107 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.107 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator": { 2024-05-27T08:00:54.108 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "orchestrator", 2024-05-27T08:00:54.108 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.108 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.108 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.108 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.108 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.108 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.108 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.109 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm", 2024-05-27T08:00:54.109 INFO:teuthology.orchestra.run.smithi114.stdout: "rook", 2024-05-27T08:00:54.109 INFO:teuthology.orchestra.run.smithi114.stdout: "test_orchestrator" 2024-05-27T08:00:54.109 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.109 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Orchestrator backend", 2024-05-27T08:00:54.109 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.109 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.109 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.110 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.110 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.110 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.110 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.110 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.110 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.110 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.111 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.111 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.111 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.111 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.111 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.111 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.111 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.111 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.112 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "osd_perf_query", 2024-05-27T08:00:54.112 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.112 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.112 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.112 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.112 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.112 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.113 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.113 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.113 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.113 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.113 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.113 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.113 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.113 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.113 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.114 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.114 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.114 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.114 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.114 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.114 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.114 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.114 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.115 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.115 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.115 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.115 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.115 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.115 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.115 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.116 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.116 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.116 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.116 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.116 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.116 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.116 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.117 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.117 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.117 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.117 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.117 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.117 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.117 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.117 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.117 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.118 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.118 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.118 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.118 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.118 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.118 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.118 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.119 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.119 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.119 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.119 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.119 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.119 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.119 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.119 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.120 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.120 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.120 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.120 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.120 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.120 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.120 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.121 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.121 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.121 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.121 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.121 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.121 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.121 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.122 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.122 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.122 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.122 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.122 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.122 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.122 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.122 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.123 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.123 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "osd_support", 2024-05-27T08:00:54.123 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.123 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.123 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.123 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.124 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.124 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.124 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.124 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.124 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.124 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.124 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.124 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.124 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.125 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.125 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.125 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.125 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.125 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.125 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.125 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.125 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.126 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.126 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.126 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.126 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.126 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.126 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.127 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.127 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.127 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.127 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.127 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.127 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.127 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.127 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.128 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.128 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.128 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.128 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.128 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.128 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.128 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.129 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.129 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.129 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.129 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.129 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.129 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.129 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.129 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.130 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.130 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.130 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.130 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.130 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.130 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.130 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.131 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.131 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.131 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.131 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.131 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.131 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.131 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.131 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.131 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.132 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.132 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.132 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.132 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.132 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.132 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.132 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.133 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.133 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.133 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.133 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.133 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.133 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.133 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.133 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.133 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.134 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.134 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.134 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pg_autoscaler", 2024-05-27T08:00:54.134 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.134 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.134 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.135 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.135 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.135 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.135 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.135 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.135 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.135 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.135 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.135 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.136 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.136 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.136 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.136 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.136 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.136 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.136 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.136 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.137 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.137 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.137 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.137 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.137 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.137 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.137 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.138 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.138 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.138 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.138 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.138 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.138 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.138 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.138 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.139 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.139 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.139 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.139 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.139 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.139 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.139 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.140 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.140 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.140 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.140 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.140 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.140 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.140 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.140 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.141 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.141 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.141 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.141 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.141 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.141 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.141 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.141 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.142 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.142 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.142 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.142 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.142 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.142 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.142 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.142 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.143 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.143 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.143 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.143 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:54.143 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:54.143 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.143 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.144 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.144 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:54.144 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.144 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.144 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.144 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.144 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.144 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.144 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.145 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.145 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.145 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.145 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.145 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.145 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.145 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.146 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.146 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.146 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.146 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.146 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.146 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.146 INFO:teuthology.orchestra.run.smithi114.stdout: "threshold": { 2024-05-27T08:00:54.147 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "threshold", 2024-05-27T08:00:54.147 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:54.147 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.147 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.147 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3.0", 2024-05-27T08:00:54.147 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1.0", 2024-05-27T08:00:54.147 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.147 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.148 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "scaling threshold", 2024-05-27T08:00:54.148 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T08:00:54.148 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.148 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.148 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.148 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.148 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "progress", 2024-05-27T08:00:54.149 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.149 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.149 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.149 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_pg_recovery_event": { 2024-05-27T08:00:54.149 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_pg_recovery_event", 2024-05-27T08:00:54.149 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.149 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.149 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.150 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.150 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.150 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.150 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.150 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T08:00:54.150 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.150 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.150 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.151 INFO:teuthology.orchestra.run.smithi114.stdout: "enabled": { 2024-05-27T08:00:54.151 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enabled", 2024-05-27T08:00:54.151 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.151 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.151 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.151 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.151 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.151 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.152 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.152 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.152 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.152 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.152 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.152 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.152 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.153 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.153 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.153 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.153 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.153 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.153 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.153 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.153 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.153 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.154 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.154 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.154 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.154 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.154 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.154 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.154 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.154 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.155 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.155 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.155 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.155 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.155 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.155 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.155 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.156 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.156 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.156 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.156 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.156 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.156 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.156 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.157 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.157 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.157 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.157 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.157 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.157 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.157 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.157 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.158 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.158 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.158 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.158 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.158 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.158 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.158 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.158 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.159 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.159 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.159 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.159 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.159 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.159 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.159 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.160 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.160 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.160 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.160 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.160 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.160 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.160 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.160 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.160 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.161 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.161 INFO:teuthology.orchestra.run.smithi114.stdout: "max_completed_events": { 2024-05-27T08:00:54.161 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_completed_events", 2024-05-27T08:00:54.161 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.161 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.161 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.161 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "50", 2024-05-27T08:00:54.162 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.162 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.162 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.162 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "number of past completed events to remember", 2024-05-27T08:00:54.162 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.162 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.162 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.163 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:54.163 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:54.163 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.163 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.163 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.163 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-05-27T08:00:54.163 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.163 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long the module is going to sleep", 2024-05-27T08:00:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.165 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.165 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.165 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.165 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.165 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.165 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.165 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.165 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.166 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.166 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.166 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.166 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.166 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.166 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prometheus", 2024-05-27T08:00:54.167 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.167 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.167 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.167 INFO:teuthology.orchestra.run.smithi114.stdout: "cache": { 2024-05-27T08:00:54.167 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cache", 2024-05-27T08:00:54.167 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.167 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.168 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.168 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.168 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.168 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.168 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.168 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.168 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.169 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.169 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.169 INFO:teuthology.orchestra.run.smithi114.stdout: "exclude_perf_counters": { 2024-05-27T08:00:54.169 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "exclude_perf_counters", 2024-05-27T08:00:54.169 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.169 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.170 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.170 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.170 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.170 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.170 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.170 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T08:00:54.170 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T08:00:54.170 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.171 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.171 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.171 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.171 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.171 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.171 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.171 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.172 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.172 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.172 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.172 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.172 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.172 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.172 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.173 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.173 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.173 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.173 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.173 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.173 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.175 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.175 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.175 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.175 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.175 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.175 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.175 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.176 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.176 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.176 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.176 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.176 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.176 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.176 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.176 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.177 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.177 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.177 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.177 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.177 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.177 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.177 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.177 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.178 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.178 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.178 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.178 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.178 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.178 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.178 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.179 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.179 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.179 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.179 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.179 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.179 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.179 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.179 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.179 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.180 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.180 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_stats_pools": { 2024-05-27T08:00:54.180 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_stats_pools", 2024-05-27T08:00:54.180 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.180 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.180 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.180 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.181 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.181 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.181 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.181 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.181 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.181 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.181 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.181 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T08:00:54.181 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T08:00:54.182 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.182 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.182 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.182 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "300", 2024-05-27T08:00:54.182 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.182 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.182 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.182 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.183 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.183 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.183 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.183 INFO:teuthology.orchestra.run.smithi114.stdout: "scrape_interval": { 2024-05-27T08:00:54.183 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "scrape_interval", 2024-05-27T08:00:54.183 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:54.183 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.184 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.184 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15.0", 2024-05-27T08:00:54.184 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.184 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.184 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.184 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.184 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.184 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.185 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.185 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-05-27T08:00:54.185 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-05-27T08:00:54.185 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.185 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.185 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.185 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "::", 2024-05-27T08:00:54.186 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.186 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.186 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.186 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T08:00:54.186 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.186 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.186 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.186 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.187 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-05-27T08:00:54.187 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-05-27T08:00:54.187 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.187 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.187 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.187 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "9283", 2024-05-27T08:00:54.187 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.187 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.188 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.188 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T08:00:54.188 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.188 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.188 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.188 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.188 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.189 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.189 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.189 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.189 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.189 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.189 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.189 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.189 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.189 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.190 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.190 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.190 INFO:teuthology.orchestra.run.smithi114.stdout: "stale_cache_strategy": { 2024-05-27T08:00:54.190 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "stale_cache_strategy", 2024-05-27T08:00:54.190 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.190 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.190 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.191 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "log", 2024-05-27T08:00:54.191 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.191 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.191 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.191 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.191 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.191 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.191 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.192 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_behaviour": { 2024-05-27T08:00:54.192 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_behaviour", 2024-05-27T08:00:54.192 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.192 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.192 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.192 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "default", 2024-05-27T08:00:54.192 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.192 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.193 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.193 INFO:teuthology.orchestra.run.smithi114.stdout: "default", 2024-05-27T08:00:54.193 INFO:teuthology.orchestra.run.smithi114.stdout: "error" 2024-05-27T08:00:54.193 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.193 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.193 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.193 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.193 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.194 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.194 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_error_status_code": { 2024-05-27T08:00:54.194 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_error_status_code", 2024-05-27T08:00:54.194 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.194 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.194 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.194 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "500", 2024-05-27T08:00:54.194 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "400", 2024-05-27T08:00:54.194 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "599", 2024-05-27T08:00:54.195 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.195 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.195 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.195 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.195 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.195 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.195 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.196 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_support", 2024-05-27T08:00:54.196 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.196 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.196 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.196 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.196 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.196 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.197 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.197 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.197 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.197 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.197 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.197 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.197 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.197 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.198 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.198 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.198 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.198 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.198 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.198 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.198 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.198 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.199 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.199 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.199 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.199 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.199 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.199 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.199 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.200 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.200 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.200 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.200 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.200 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.200 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.200 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.200 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.201 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.201 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.201 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.201 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.201 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.201 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.201 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.201 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.202 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.202 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.202 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.202 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.202 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.202 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.202 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.202 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.203 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.203 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.203 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.203 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.203 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.203 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.203 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.203 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.203 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.204 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.204 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.204 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.204 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.204 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.204 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.204 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.204 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.205 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.205 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.205 INFO:teuthology.orchestra.run.smithi114.stdout: "max_concurrent_snap_create": { 2024-05-27T08:00:54.205 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_concurrent_snap_create", 2024-05-27T08:00:54.205 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.205 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.205 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.205 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:54.205 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.206 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.206 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.206 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.206 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.206 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.206 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.206 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.207 INFO:teuthology.orchestra.run.smithi114.stdout: "mirror_snapshot_schedule": { 2024-05-27T08:00:54.207 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T08:00:54.207 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.207 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.207 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.207 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.207 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.207 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.207 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.208 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.208 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.208 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.208 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.208 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.208 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.208 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.208 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.208 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.209 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.209 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.209 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.209 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.209 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.209 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.209 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.209 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.210 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.210 INFO:teuthology.orchestra.run.smithi114.stdout: "trash_purge_schedule": { 2024-05-27T08:00:54.210 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "trash_purge_schedule", 2024-05-27T08:00:54.210 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.210 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.210 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.210 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.210 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.211 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.211 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.211 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.211 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.211 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.211 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.211 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.211 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.212 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.212 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "restful", 2024-05-27T08:00:54.212 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.212 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.212 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.212 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_auth": { 2024-05-27T08:00:54.212 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enable_auth", 2024-05-27T08:00:54.212 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.212 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.213 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.213 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.213 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.213 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.213 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.213 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.213 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.213 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.213 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.214 INFO:teuthology.orchestra.run.smithi114.stdout: "key_file": { 2024-05-27T08:00:54.214 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "key_file", 2024-05-27T08:00:54.214 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.214 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.214 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.214 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.214 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.215 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.215 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.215 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.215 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.215 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.215 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.215 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.215 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.215 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.216 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.216 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.216 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.216 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.216 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.216 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.216 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.216 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.217 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.217 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.217 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.217 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.217 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.217 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.217 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.217 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.218 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.218 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.218 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.218 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.218 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.218 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.219 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.219 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.219 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.219 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.219 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.219 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.220 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.220 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.220 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.220 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.220 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.220 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.221 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.221 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.221 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.221 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.221 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.221 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.221 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.222 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.222 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.222 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.222 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.222 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.222 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.222 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.222 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.223 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.223 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.223 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.223 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.223 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.223 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.223 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.224 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.224 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.224 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.224 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.224 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.224 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.225 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.225 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-05-27T08:00:54.225 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-05-27T08:00:54.225 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.225 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.225 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.225 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.225 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.226 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.226 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.226 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.226 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.226 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.226 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.227 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-05-27T08:00:54.227 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-05-27T08:00:54.227 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.227 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.227 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.227 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.227 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.228 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.228 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.228 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.228 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.228 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.228 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.228 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.229 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.229 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.229 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.229 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.229 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.229 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.229 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.229 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.230 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.230 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.230 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.230 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.230 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.230 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rgw", 2024-05-27T08:00:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.239 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.239 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.239 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.239 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.239 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.239 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.239 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.239 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.240 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.240 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.240 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.240 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.240 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.240 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.240 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.240 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.241 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.241 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.241 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.241 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.241 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.241 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.241 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.242 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.242 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.242 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.242 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.242 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.242 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.242 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.242 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.243 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.243 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.243 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.243 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.243 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.243 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.243 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.243 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.244 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.244 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.244 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.244 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.244 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.244 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.244 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.244 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.245 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.245 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.245 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.245 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rook", 2024-05-27T08:00:54.245 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.245 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.245 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.246 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.246 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.246 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.246 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.246 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.246 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.246 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.246 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.247 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.247 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.247 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.247 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.247 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.247 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.247 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.247 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.248 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.248 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.248 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.248 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.248 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.248 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.248 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.248 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.248 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.249 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.249 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.249 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.249 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.249 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.249 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.249 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.249 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.249 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.250 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.250 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.250 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.250 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.250 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.250 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.250 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.251 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.251 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.251 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.251 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.251 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.251 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.251 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.251 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.251 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.252 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.252 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.252 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.252 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.252 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.252 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.252 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.252 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.253 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.253 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.253 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.253 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.253 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.253 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.253 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.253 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.254 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.254 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.254 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.254 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.254 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.254 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.255 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.255 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.255 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.255 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.255 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.255 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.255 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.256 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.256 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.256 INFO:teuthology.orchestra.run.smithi114.stdout: "storage_class": { 2024-05-27T08:00:54.256 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "storage_class", 2024-05-27T08:00:54.256 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.256 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.256 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.256 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "local", 2024-05-27T08:00:54.257 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.257 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.257 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.257 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T08:00:54.257 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.257 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.258 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.258 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.258 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.258 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.258 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.259 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "selftest", 2024-05-27T08:00:54.259 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.259 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.259 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.259 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.259 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.259 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.260 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.260 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.260 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.260 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.260 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.260 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.260 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.260 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.261 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.261 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.261 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.261 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.261 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.261 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.261 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.261 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.262 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.262 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.262 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.262 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.262 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.262 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.262 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.263 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.263 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.263 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.263 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.263 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.263 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.263 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.263 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.263 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.264 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.264 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.264 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.264 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.264 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.264 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.264 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.264 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.264 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.265 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.265 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.265 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.265 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.265 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.265 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.265 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.265 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.266 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.266 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.266 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.266 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.266 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.266 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.266 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.266 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.267 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.267 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.267 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.267 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.267 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.267 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.267 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.267 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.268 INFO:teuthology.orchestra.run.smithi114.stdout: "roption1": { 2024-05-27T08:00:54.268 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "roption1", 2024-05-27T08:00:54.268 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.268 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.268 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.268 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.268 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.268 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.269 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.269 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.269 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.269 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.269 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.269 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.269 INFO:teuthology.orchestra.run.smithi114.stdout: "roption2": { 2024-05-27T08:00:54.269 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "roption2", 2024-05-27T08:00:54.270 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.270 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.270 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.270 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "xyz", 2024-05-27T08:00:54.270 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.270 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.270 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.270 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.271 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.271 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.271 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.271 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.271 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption1": { 2024-05-27T08:00:54.271 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption1", 2024-05-27T08:00:54.271 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.271 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.272 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.272 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.272 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.272 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.272 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.272 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.272 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.273 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.273 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.273 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.273 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption2": { 2024-05-27T08:00:54.273 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption2", 2024-05-27T08:00:54.273 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.273 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.273 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.274 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.274 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.274 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.274 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.274 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.274 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.274 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.274 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.275 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.275 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption3": { 2024-05-27T08:00:54.275 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption3", 2024-05-27T08:00:54.275 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:54.275 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.275 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.275 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.275 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.276 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.276 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.276 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.276 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.276 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.276 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.276 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.276 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption4": { 2024-05-27T08:00:54.277 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption4", 2024-05-27T08:00:54.277 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.277 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.277 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.277 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.277 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.277 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.277 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.277 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.278 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.278 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.278 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.278 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.278 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption5": { 2024-05-27T08:00:54.278 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption5", 2024-05-27T08:00:54.278 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.278 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.279 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.279 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.279 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.279 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.279 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.279 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.279 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.279 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.279 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.280 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.280 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption6": { 2024-05-27T08:00:54.280 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption6", 2024-05-27T08:00:54.280 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.280 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.280 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.280 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.280 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.280 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.281 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.281 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.281 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.281 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.281 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.281 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.281 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption7": { 2024-05-27T08:00:54.281 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption7", 2024-05-27T08:00:54.282 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.282 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.282 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.282 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.282 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-05-27T08:00:54.282 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "42", 2024-05-27T08:00:54.282 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.282 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.282 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.283 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.283 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.283 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.283 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.283 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.283 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.283 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.283 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.283 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.284 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.284 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.284 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.284 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.284 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.284 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.284 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.284 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.284 INFO:teuthology.orchestra.run.smithi114.stdout: "testkey": { 2024-05-27T08:00:54.284 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testkey", 2024-05-27T08:00:54.285 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.285 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.285 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.285 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.285 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.285 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.285 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.285 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.285 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.286 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.286 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.286 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.286 INFO:teuthology.orchestra.run.smithi114.stdout: "testlkey": { 2024-05-27T08:00:54.286 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testlkey", 2024-05-27T08:00:54.286 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.286 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.286 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.287 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.287 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.287 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.287 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.287 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.287 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.287 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.287 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.288 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.288 INFO:teuthology.orchestra.run.smithi114.stdout: "testnewline": { 2024-05-27T08:00:54.288 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testnewline", 2024-05-27T08:00:54.288 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.288 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.288 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.288 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.288 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.289 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.289 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.289 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.289 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.289 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.289 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.289 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.289 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.290 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.290 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.290 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smb", 2024-05-27T08:00:54.290 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.290 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.290 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.290 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.290 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.291 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.291 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.291 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.291 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.291 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.291 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.291 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.291 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.292 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.292 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.292 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.292 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.292 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.292 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.292 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.292 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.293 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.293 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.293 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.293 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.293 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.293 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.293 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.293 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.294 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.294 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.294 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.294 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.294 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.294 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.294 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.295 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.295 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.295 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.295 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.295 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.295 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.295 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.295 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.295 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.296 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.296 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.296 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.296 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.296 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.296 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.296 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.296 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.297 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.297 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.297 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.297 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.297 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.297 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.297 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.297 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.298 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.298 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.298 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.298 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.298 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.298 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.298 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.298 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.299 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.299 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.299 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.299 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.299 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.299 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.299 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.299 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.300 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.300 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.300 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.300 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.300 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.300 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.300 INFO:teuthology.orchestra.run.smithi114.stdout: "update_orchestration": { 2024-05-27T08:00:54.300 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "update_orchestration", 2024-05-27T08:00:54.301 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.301 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.301 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.301 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.301 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.301 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.301 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.301 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T08:00:54.301 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.302 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.302 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.302 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.302 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.302 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.302 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.302 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snap_schedule", 2024-05-27T08:00:54.302 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.302 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.303 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.303 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_m_granularity": { 2024-05-27T08:00:54.303 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_m_granularity", 2024-05-27T08:00:54.303 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.303 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.303 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.303 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.303 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.303 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.304 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.304 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T08:00:54.304 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.304 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.304 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.304 INFO:teuthology.orchestra.run.smithi114.stdout: "dump_on_update": { 2024-05-27T08:00:54.304 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "dump_on_update", 2024-05-27T08:00:54.304 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.305 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.305 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.305 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.305 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.305 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.305 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.305 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "dump database to debug log on update", 2024-05-27T08:00:54.305 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.306 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.306 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.306 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.306 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.306 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.306 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.306 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.307 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.307 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.307 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.307 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.307 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.307 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.307 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.307 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.307 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.308 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.308 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.308 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.308 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.308 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.308 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.308 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.309 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.309 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.309 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.309 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.309 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.309 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.309 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.309 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.309 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.309 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.310 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.310 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.310 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.310 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.310 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.310 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.311 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.311 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.311 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.311 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.311 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.311 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.311 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.311 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.311 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.312 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.312 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.312 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.312 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.312 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.312 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.312 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.313 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.313 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.313 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.313 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.313 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.313 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.313 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.313 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.313 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.314 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.314 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.314 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.314 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.314 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.314 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.314 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.315 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.315 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.315 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.315 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.315 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.315 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.315 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.316 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.316 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.316 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.316 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.316 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.316 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.316 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.316 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "stats", 2024-05-27T08:00:54.316 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.317 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.317 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.317 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.317 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.317 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.317 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.317 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.317 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.317 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.318 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.318 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.318 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.318 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.318 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.318 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.318 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.318 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.319 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.319 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.319 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.319 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.319 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.319 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.319 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.320 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.320 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.320 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.320 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.320 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.320 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.320 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.320 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.320 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.320 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.321 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.321 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.321 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.321 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.321 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.321 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.321 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.321 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.322 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.322 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.322 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.322 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.322 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.322 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.322 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.322 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.323 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.323 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.323 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.323 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.323 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.323 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.323 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.324 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.324 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.324 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.324 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.324 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.324 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.324 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.324 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.325 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.325 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.325 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.325 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.325 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.325 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.325 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.325 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.325 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "status", 2024-05-27T08:00:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "telegraf", 2024-05-27T08:00:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "address": { 2024-05-27T08:00:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "address", 2024-05-27T08:00:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T08:00:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15", 2024-05-27T08:00:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "telemetry", 2024-05-27T08:00:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_basic": { 2024-05-27T08:00:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_basic", 2024-05-27T08:00:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T08:00:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_crash": { 2024-05-27T08:00:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_crash", 2024-05-27T08:00:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T08:00:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_device": { 2024-05-27T08:00:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_device", 2024-05-27T08:00:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T08:00:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_ident": { 2024-05-27T08:00:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_ident", 2024-05-27T08:00:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T08:00:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_perf": { 2024-05-27T08:00:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_perf", 2024-05-27T08:00:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T08:00:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "contact": { 2024-05-27T08:00:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "contact", 2024-05-27T08:00:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "description": { 2024-05-27T08:00:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "description", 2024-05-27T08:00:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: "device_url": { 2024-05-27T08:00:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_url", 2024-05-27T08:00:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T08:00:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: "enabled": { 2024-05-27T08:00:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enabled", 2024-05-27T08:00:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "24", 2024-05-27T08:00:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "8", 2024-05-27T08:00:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "last_opt_revision": { 2024-05-27T08:00:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "last_opt_revision", 2024-05-27T08:00:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1", 2024-05-27T08:00:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "leaderboard": { 2024-05-27T08:00:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "leaderboard", 2024-05-27T08:00:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "leaderboard_description": { 2024-05-27T08:00:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "leaderboard_description", 2024-05-27T08:00:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "organization": { 2024-05-27T08:00:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "organization", 2024-05-27T08:00:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "proxy": { 2024-05-27T08:00:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "proxy", 2024-05-27T08:00:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "url": { 2024-05-27T08:00:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "url", 2024-05-27T08:00:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T08:00:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "test_orchestrator", 2024-05-27T08:00:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "volumes", 2024-05-27T08:00:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: "max_concurrent_clones": { 2024-05-27T08:00:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_concurrent_clones", 2024-05-27T08:00:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "4", 2024-05-27T08:00:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T08:00:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: "periodic_async_work": { 2024-05-27T08:00:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "periodic_async_work", 2024-05-27T08:00:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Periodically check for async work", 2024-05-27T08:00:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: "snapshot_clone_delay": { 2024-05-27T08:00:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snapshot_clone_delay", 2024-05-27T08:00:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T08:00:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "snapshot_clone_no_wait": { 2024-05-27T08:00:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T08:00:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T08:00:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix", 2024-05-27T08:00:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "discovery_interval": { 2024-05-27T08:00:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "discovery_interval", 2024-05-27T08:00:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "100", 2024-05-27T08:00:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.423 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.423 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.423 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.423 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.423 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.423 INFO:teuthology.orchestra.run.smithi114.stdout: "identifier": { 2024-05-27T08:00:54.423 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "identifier", 2024-05-27T08:00:54.423 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.424 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.424 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.424 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.424 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.424 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.424 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.424 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.424 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.424 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.425 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.425 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:54.425 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:54.425 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.425 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.425 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.425 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_host": { 2024-05-27T08:00:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_host", 2024-05-27T08:00:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_port": { 2024-05-27T08:00:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_port", 2024-05-27T08:00:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10051", 2024-05-27T08:00:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_sender": { 2024-05-27T08:00:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_sender", 2024-05-27T08:00:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T08:00:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-05-27T08:00:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "gid": 24838, 2024-05-27T08:00:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "z", 2024-05-27T08:00:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_features": 4540701547738038271, 2024-05-27T08:00:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "available_modules": [ 2024-05-27T08:00:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "alerts", 2024-05-27T08:00:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T08:00:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_destination": { 2024-05-27T08:00:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_destination", 2024-05-27T08:00:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Email address to send alerts to", 2024-05-27T08:00:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_from_name": { 2024-05-27T08:00:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_from_name", 2024-05-27T08:00:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "Ceph", 2024-05-27T08:00:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Email From: name", 2024-05-27T08:00:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_host": { 2024-05-27T08:00:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_host", 2024-05-27T08:00:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP server", 2024-05-27T08:00:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_password": { 2024-05-27T08:00:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_password", 2024-05-27T08:00:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Password to authenticate with", 2024-05-27T08:00:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_port": { 2024-05-27T08:00:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_port", 2024-05-27T08:00:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "465", 2024-05-27T08:00:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP port", 2024-05-27T08:00:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_sender": { 2024-05-27T08:00:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_sender", 2024-05-27T08:00:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP envelope sender", 2024-05-27T08:00:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_ssl": { 2024-05-27T08:00:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_ssl", 2024-05-27T08:00:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T08:00:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_user": { 2024-05-27T08:00:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_user", 2024-05-27T08:00:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "User to authenticate as", 2024-05-27T08:00:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.469 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.469 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.469 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.469 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.469 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.469 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.469 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.470 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.470 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.470 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "balancer", 2024-05-27T08:00:54.470 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.470 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.470 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.470 INFO:teuthology.orchestra.run.smithi114.stdout: "active": { 2024-05-27T08:00:54.471 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "active", 2024-05-27T08:00:54.471 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.471 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.471 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.471 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.471 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.471 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.471 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.471 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T08:00:54.472 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.472 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.472 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.472 INFO:teuthology.orchestra.run.smithi114.stdout: "begin_time": { 2024-05-27T08:00:54.472 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "begin_time", 2024-05-27T08:00:54.472 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.472 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.472 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.473 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0000", 2024-05-27T08:00:54.473 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.473 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.473 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.473 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T08:00:54.473 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T08:00:54.473 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.473 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.474 INFO:teuthology.orchestra.run.smithi114.stdout: "begin_weekday": { 2024-05-27T08:00:54.474 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "begin_weekday", 2024-05-27T08:00:54.474 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:54.474 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.474 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.474 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.474 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0", 2024-05-27T08:00:54.475 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "6", 2024-05-27T08:00:54.475 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.475 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T08:00:54.475 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T08:00:54.475 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.475 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.475 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_max_iterations": { 2024-05-27T08:00:54.476 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_max_iterations", 2024-05-27T08:00:54.476 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:54.476 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.476 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.476 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "25", 2024-05-27T08:00:54.476 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-05-27T08:00:54.476 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "250", 2024-05-27T08:00:54.476 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.476 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T08:00:54.477 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.477 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.477 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.477 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_metrics": { 2024-05-27T08:00:54.477 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_metrics", 2024-05-27T08:00:54.477 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.477 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.477 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.478 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T08:00:54.478 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.478 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.478 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.478 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T08:00:54.478 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T08:00:54.478 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.478 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.479 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_step": { 2024-05-27T08:00:54.479 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_step", 2024-05-27T08:00:54.479 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:54.479 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.479 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.479 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0.5", 2024-05-27T08:00:54.479 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0.001", 2024-05-27T08:00:54.480 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "0.999", 2024-05-27T08:00:54.480 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.480 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "aggressiveness of optimization", 2024-05-27T08:00:54.480 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T08:00:54.480 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.480 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.480 INFO:teuthology.orchestra.run.smithi114.stdout: "end_time": { 2024-05-27T08:00:54.481 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "end_time", 2024-05-27T08:00:54.481 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.481 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.481 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.481 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "2359", 2024-05-27T08:00:54.481 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.481 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.481 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.482 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T08:00:54.482 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T08:00:54.482 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.482 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.482 INFO:teuthology.orchestra.run.smithi114.stdout: "end_weekday": { 2024-05-27T08:00:54.482 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "end_weekday", 2024-05-27T08:00:54.482 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:54.483 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.483 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.483 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.483 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0", 2024-05-27T08:00:54.483 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "6", 2024-05-27T08:00:54.483 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.483 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T08:00:54.483 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T08:00:54.483 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.484 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.484 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.484 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.484 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.484 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.484 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.485 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.485 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.485 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.485 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.485 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.485 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.485 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.485 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.485 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.486 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.486 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.486 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.486 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.486 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.486 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.486 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.487 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.487 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.487 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.487 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.487 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.487 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.487 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.487 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.488 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.488 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.488 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.488 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.488 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.488 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.488 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.489 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.489 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.489 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.489 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.489 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.489 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.489 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.489 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.490 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.490 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.490 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.490 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.490 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.490 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.490 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.490 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.491 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.491 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.491 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.491 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.491 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.491 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.491 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.492 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.492 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.492 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.492 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.492 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.492 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.492 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.492 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.492 INFO:teuthology.orchestra.run.smithi114.stdout: "min_score": { 2024-05-27T08:00:54.493 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "min_score", 2024-05-27T08:00:54.493 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:54.493 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.493 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.493 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.493 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.493 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.493 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.494 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T08:00:54.494 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.494 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.494 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.494 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.494 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": { 2024-05-27T08:00:54.494 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mode", 2024-05-27T08:00:54.494 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.494 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.495 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.495 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "upmap", 2024-05-27T08:00:54.495 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.495 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.495 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.495 INFO:teuthology.orchestra.run.smithi114.stdout: "crush-compat", 2024-05-27T08:00:54.495 INFO:teuthology.orchestra.run.smithi114.stdout: "none", 2024-05-27T08:00:54.495 INFO:teuthology.orchestra.run.smithi114.stdout: "read", 2024-05-27T08:00:54.496 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap", 2024-05-27T08:00:54.496 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap-read" 2024-05-27T08:00:54.496 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.496 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Balancer mode", 2024-05-27T08:00:54.496 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.496 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.496 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.497 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_ids": { 2024-05-27T08:00:54.497 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pool_ids", 2024-05-27T08:00:54.497 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.497 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.497 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.497 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.497 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.497 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.498 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.498 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T08:00:54.498 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.498 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.498 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.498 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:54.498 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:54.499 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.499 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.499 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.499 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:54.499 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.499 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.499 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.499 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T08:00:54.500 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.500 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.500 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.500 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.500 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.500 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.500 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.500 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.501 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.501 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.501 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.501 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.501 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.501 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.501 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.501 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.502 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.502 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap_max_deviation": { 2024-05-27T08:00:54.502 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "upmap_max_deviation", 2024-05-27T08:00:54.502 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.502 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.502 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.502 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-05-27T08:00:54.502 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-05-27T08:00:54.502 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.503 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.503 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T08:00:54.503 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T08:00:54.503 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.503 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.503 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap_max_optimizations": { 2024-05-27T08:00:54.504 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "upmap_max_optimizations", 2024-05-27T08:00:54.504 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:54.504 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.504 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.504 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:54.504 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.504 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.504 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.505 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T08:00:54.505 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.505 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.505 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.505 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.505 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.505 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephadm", 2024-05-27T08:00:54.506 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.506 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.506 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.506 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_down_multiplier": { 2024-05-27T08:00:54.506 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_down_multiplier", 2024-05-27T08:00:54.506 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:54.506 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.506 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.507 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3.0", 2024-05-27T08:00:54.507 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.507 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.507 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.507 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T08:00:54.507 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.507 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.507 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.508 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_refresh_rate": { 2024-05-27T08:00:54.508 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_refresh_rate", 2024-05-27T08:00:54.508 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.508 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.508 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.508 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "20", 2024-05-27T08:00:54.508 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.508 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.509 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.509 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T08:00:54.509 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.509 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.509 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.509 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_starting_port": { 2024-05-27T08:00:54.509 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_starting_port", 2024-05-27T08:00:54.510 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.510 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.510 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.510 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "4721", 2024-05-27T08:00:54.510 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.510 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.510 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.510 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T08:00:54.511 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.511 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.511 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.511 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.511 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_ptrace": { 2024-05-27T08:00:54.511 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_ptrace", 2024-05-27T08:00:54.511 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.511 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.512 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.512 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.512 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.512 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.512 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.512 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T08:00:54.512 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T08:00:54.512 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.513 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.513 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.513 INFO:teuthology.orchestra.run.smithi114.stdout: "autotune_interval": { 2024-05-27T08:00:54.513 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "autotune_interval", 2024-05-27T08:00:54.513 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.513 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.513 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.513 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:54.514 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.514 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.514 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.514 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T08:00:54.514 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.514 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.514 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.514 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.515 INFO:teuthology.orchestra.run.smithi114.stdout: "autotune_memory_target_ratio": { 2024-05-27T08:00:54.515 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T08:00:54.515 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:54.515 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.515 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.515 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0.7", 2024-05-27T08:00:54.515 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.515 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.516 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.516 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T08:00:54.516 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.516 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.516 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.516 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.516 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm_log_destination": { 2024-05-27T08:00:54.516 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephadm_log_destination", 2024-05-27T08:00:54.517 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.517 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.517 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.517 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.517 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.517 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.517 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.517 INFO:teuthology.orchestra.run.smithi114.stdout: "file", 2024-05-27T08:00:54.517 INFO:teuthology.orchestra.run.smithi114.stdout: "file,syslog", 2024-05-27T08:00:54.518 INFO:teuthology.orchestra.run.smithi114.stdout: "syslog" 2024-05-27T08:00:54.518 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.518 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T08:00:54.518 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.518 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.518 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.518 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.518 INFO:teuthology.orchestra.run.smithi114.stdout: "cgroups_split": { 2024-05-27T08:00:54.519 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cgroups_split", 2024-05-27T08:00:54.519 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.519 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.519 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.519 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.519 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.519 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.519 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.519 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T08:00:54.520 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.520 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.520 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.520 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.520 INFO:teuthology.orchestra.run.smithi114.stdout: "config_checks_enabled": { 2024-05-27T08:00:54.520 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_checks_enabled", 2024-05-27T08:00:54.520 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.520 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.521 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.521 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.521 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.521 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.521 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.521 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T08:00:54.521 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.521 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.521 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.522 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.522 INFO:teuthology.orchestra.run.smithi114.stdout: "config_dashboard": { 2024-05-27T08:00:54.522 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_dashboard", 2024-05-27T08:00:54.522 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.522 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.522 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.522 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.522 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.523 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.523 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.523 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T08:00:54.523 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.523 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.523 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.523 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.523 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_alertmanager": { 2024-05-27T08:00:54.524 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_alertmanager", 2024-05-27T08:00:54.524 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.524 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.524 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.524 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T08:00:54.524 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.524 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.525 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.525 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-05-27T08:00:54.525 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.525 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.525 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.525 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.525 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_base": { 2024-05-27T08:00:54.525 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_base", 2024-05-27T08:00:54.525 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.526 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.526 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.526 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T08:00:54.526 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.526 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.526 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.526 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Container image name, without the tag", 2024-05-27T08:00:54.526 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.527 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.527 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.527 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_elasticsearch": { 2024-05-27T08:00:54.527 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_elasticsearch", 2024-05-27T08:00:54.527 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.527 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.528 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.528 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T08:00:54.528 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.528 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.528 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.528 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "elasticsearch container image", 2024-05-27T08:00:54.528 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.528 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.529 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.529 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_grafana": { 2024-05-27T08:00:54.529 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_grafana", 2024-05-27T08:00:54.529 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.529 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.529 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.529 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T08:00:54.530 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.530 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.530 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.530 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-05-27T08:00:54.530 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.530 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.530 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.530 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.531 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_haproxy": { 2024-05-27T08:00:54.531 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_haproxy", 2024-05-27T08:00:54.531 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.531 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.531 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.531 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T08:00:54.531 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.531 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.532 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.532 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "HAproxy container image", 2024-05-27T08:00:54.532 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.532 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.532 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.532 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.532 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_agent": { 2024-05-27T08:00:54.532 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_agent", 2024-05-27T08:00:54.533 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.533 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.533 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.533 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T08:00:54.533 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.533 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.533 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.533 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger agent container image", 2024-05-27T08:00:54.534 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.534 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.534 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.534 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.534 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_collector": { 2024-05-27T08:00:54.534 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_collector", 2024-05-27T08:00:54.534 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.534 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.535 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.535 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T08:00:54.535 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.535 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.535 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.535 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger collector container image", 2024-05-27T08:00:54.535 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.535 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.535 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.536 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.536 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_query": { 2024-05-27T08:00:54.536 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_query", 2024-05-27T08:00:54.536 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.536 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.536 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.536 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T08:00:54.536 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.537 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.537 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.537 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger query container image", 2024-05-27T08:00:54.537 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.537 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.537 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.537 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.537 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_keepalived": { 2024-05-27T08:00:54.538 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_keepalived", 2024-05-27T08:00:54.538 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.538 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.538 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.538 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T08:00:54.538 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.538 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.538 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.538 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Keepalived container image", 2024-05-27T08:00:54.539 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.539 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.539 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.539 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.539 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_loki": { 2024-05-27T08:00:54.539 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_loki", 2024-05-27T08:00:54.539 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.539 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.539 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.540 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T08:00:54.540 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.540 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.540 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.540 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Loki container image", 2024-05-27T08:00:54.540 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.540 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.540 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.541 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.541 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_node_exporter": { 2024-05-27T08:00:54.541 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_node_exporter", 2024-05-27T08:00:54.541 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.541 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.541 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.541 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T08:00:54.541 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.541 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.542 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.542 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-05-27T08:00:54.542 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.542 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.542 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.542 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_nvmeof": { 2024-05-27T08:00:54.542 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_nvmeof", 2024-05-27T08:00:54.543 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.543 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.543 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.543 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T08:00:54.543 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.543 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.543 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.543 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Nvme-of container image", 2024-05-27T08:00:54.544 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.544 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.544 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.544 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.544 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_prometheus": { 2024-05-27T08:00:54.544 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_prometheus", 2024-05-27T08:00:54.544 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.544 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T08:00:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-05-27T08:00:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_promtail": { 2024-05-27T08:00:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_promtail", 2024-05-27T08:00:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T08:00:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Promtail container image", 2024-05-27T08:00:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_samba": { 2024-05-27T08:00:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_samba", 2024-05-27T08:00:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T08:00:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Samba/SMB container image", 2024-05-27T08:00:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_snmp_gateway": { 2024-05-27T08:00:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_snmp_gateway", 2024-05-27T08:00:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T08:00:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SNMP Gateway container image", 2024-05-27T08:00:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: "container_init": { 2024-05-27T08:00:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_init", 2024-05-27T08:00:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T08:00:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: "daemon_cache_timeout": { 2024-05-27T08:00:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "daemon_cache_timeout", 2024-05-27T08:00:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T08:00:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: "default_cephadm_command_timeout": { 2024-05-27T08:00:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T08:00:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "900", 2024-05-27T08:00:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T08:00:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.556 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.556 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.556 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.556 INFO:teuthology.orchestra.run.smithi114.stdout: "default_registry": { 2024-05-27T08:00:54.556 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "default_registry", 2024-05-27T08:00:54.556 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.556 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.556 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.557 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io", 2024-05-27T08:00:54.557 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.557 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.557 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.557 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T08:00:54.557 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.557 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.557 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.557 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.558 INFO:teuthology.orchestra.run.smithi114.stdout: "device_cache_timeout": { 2024-05-27T08:00:54.558 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_cache_timeout", 2024-05-27T08:00:54.558 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.558 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.558 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.558 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1800", 2024-05-27T08:00:54.558 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.558 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.559 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.559 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache device inventory", 2024-05-27T08:00:54.559 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.559 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.559 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.559 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.559 INFO:teuthology.orchestra.run.smithi114.stdout: "device_enhanced_scan": { 2024-05-27T08:00:54.559 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_enhanced_scan", 2024-05-27T08:00:54.560 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.560 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.560 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.560 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.560 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.560 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.560 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.560 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T08:00:54.560 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.561 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.561 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.561 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.561 INFO:teuthology.orchestra.run.smithi114.stdout: "facts_cache_timeout": { 2024-05-27T08:00:54.561 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "facts_cache_timeout", 2024-05-27T08:00:54.561 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.561 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.561 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.562 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:54.562 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.562 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.562 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.562 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache host facts data", 2024-05-27T08:00:54.562 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.562 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.562 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.563 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.563 INFO:teuthology.orchestra.run.smithi114.stdout: "grafana_dashboards_path": { 2024-05-27T08:00:54.563 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "grafana_dashboards_path", 2024-05-27T08:00:54.563 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.563 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.563 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.563 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T08:00:54.563 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.564 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.564 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.564 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T08:00:54.564 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.564 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.564 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.564 INFO:teuthology.orchestra.run.smithi114.stdout: "host_check_interval": { 2024-05-27T08:00:54.564 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "host_check_interval", 2024-05-27T08:00:54.565 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.565 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.565 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.565 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:54.565 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.565 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.565 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.566 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to perform a host check", 2024-05-27T08:00:54.566 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.566 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.566 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.566 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.566 INFO:teuthology.orchestra.run.smithi114.stdout: "hw_monitoring": { 2024-05-27T08:00:54.566 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "hw_monitoring", 2024-05-27T08:00:54.566 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.567 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.567 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.567 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.567 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.567 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.567 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.567 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T08:00:54.567 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.568 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.568 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.568 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.568 INFO:teuthology.orchestra.run.smithi114.stdout: "inventory_list_all": { 2024-05-27T08:00:54.568 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "inventory_list_all", 2024-05-27T08:00:54.568 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.568 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.568 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.568 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.569 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.569 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.569 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.569 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T08:00:54.569 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.569 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.569 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.569 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.570 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.570 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.570 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.570 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.570 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.570 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.570 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.570 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.571 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.571 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.571 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.571 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.571 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.571 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.571 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.571 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.572 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.572 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.572 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.572 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.572 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.572 INFO:teuthology.orchestra.run.smithi114.stdout: "log_refresh_metadata": { 2024-05-27T08:00:54.572 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_refresh_metadata", 2024-05-27T08:00:54.572 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.573 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.573 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.573 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.573 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.573 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.573 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.573 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T08:00:54.573 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.574 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.574 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.574 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.574 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.574 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.574 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.574 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.574 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.575 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.575 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.575 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.575 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.575 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T08:00:54.575 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.575 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.575 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.576 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.576 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.576 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.576 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.576 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.576 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.576 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.577 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.577 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.577 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.577 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.577 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.577 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.577 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.577 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.578 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.578 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.578 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.578 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.578 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.578 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.578 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.578 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.579 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.579 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.579 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.579 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.579 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.579 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.579 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.579 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.580 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.580 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.580 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.580 INFO:teuthology.orchestra.run.smithi114.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T08:00:54.580 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T08:00:54.580 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.580 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.581 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.581 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.581 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.581 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.581 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.581 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T08:00:54.581 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.581 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.582 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.582 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.582 INFO:teuthology.orchestra.run.smithi114.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T08:00:54.582 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T08:00:54.582 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.582 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.582 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.583 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "*", 2024-05-27T08:00:54.583 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.583 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.583 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.583 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T08:00:54.583 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.583 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.583 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.584 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.584 INFO:teuthology.orchestra.run.smithi114.stdout: "max_count_per_host": { 2024-05-27T08:00:54.584 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_count_per_host", 2024-05-27T08:00:54.584 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.584 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.584 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.584 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:54.584 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.584 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.585 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.585 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "max number of daemons per service per host", 2024-05-27T08:00:54.585 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.585 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.585 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.585 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.585 INFO:teuthology.orchestra.run.smithi114.stdout: "max_osd_draining_count": { 2024-05-27T08:00:54.585 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_osd_draining_count", 2024-05-27T08:00:54.586 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.586 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.586 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.586 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:54.586 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.586 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.586 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.586 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T08:00:54.587 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.587 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.587 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.587 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.587 INFO:teuthology.orchestra.run.smithi114.stdout: "migration_current": { 2024-05-27T08:00:54.587 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "migration_current", 2024-05-27T08:00:54.587 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.587 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.588 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.588 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.588 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.588 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.588 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.588 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "internal - do not modify", 2024-05-27T08:00:54.588 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.588 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.589 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.589 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.589 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": { 2024-05-27T08:00:54.589 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mode", 2024-05-27T08:00:54.589 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.589 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.589 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.589 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "root", 2024-05-27T08:00:54.589 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.590 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.590 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.590 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm-package", 2024-05-27T08:00:54.590 INFO:teuthology.orchestra.run.smithi114.stdout: "root" 2024-05-27T08:00:54.590 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.590 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T08:00:54.590 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.591 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.591 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.591 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.591 INFO:teuthology.orchestra.run.smithi114.stdout: "oob_default_addr": { 2024-05-27T08:00:54.591 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "oob_default_addr", 2024-05-27T08:00:54.591 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.591 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.591 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.591 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "169.254.1.1", 2024-05-27T08:00:54.592 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.592 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.592 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.592 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T08:00:54.592 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.592 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.592 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.592 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: "prometheus_alerts_path": { 2024-05-27T08:00:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prometheus_alerts_path", 2024-05-27T08:00:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T08:00:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.594 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.594 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T08:00:54.594 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.594 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.594 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.594 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.594 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_insecure": { 2024-05-27T08:00:54.594 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_insecure", 2024-05-27T08:00:54.595 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.595 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.595 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.595 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.595 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.595 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.595 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.595 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T08:00:54.595 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.596 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.596 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.596 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_password": { 2024-05-27T08:00:54.596 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_password", 2024-05-27T08:00:54.596 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.596 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.597 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.597 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.597 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.597 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.597 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.597 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T08:00:54.597 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.597 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.597 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.598 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_url": { 2024-05-27T08:00:54.598 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_url", 2024-05-27T08:00:54.598 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.598 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.598 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.598 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.598 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.599 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.599 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.599 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T08:00:54.599 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.599 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.599 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.600 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_username": { 2024-05-27T08:00:54.600 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_username", 2024-05-27T08:00:54.600 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.600 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.600 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.600 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.600 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.600 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.600 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.601 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T08:00:54.601 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.601 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.601 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.601 INFO:teuthology.orchestra.run.smithi114.stdout: "secure_monitoring_stack": { 2024-05-27T08:00:54.601 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "secure_monitoring_stack", 2024-05-27T08:00:54.602 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.602 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.602 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.602 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.602 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.602 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.602 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.602 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T08:00:54.603 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.603 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.603 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.603 INFO:teuthology.orchestra.run.smithi114.stdout: "service_discovery_port": { 2024-05-27T08:00:54.603 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "service_discovery_port", 2024-05-27T08:00:54.603 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.603 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.603 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.604 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8765", 2024-05-27T08:00:54.604 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.604 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.604 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.604 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "cephadm service discovery port", 2024-05-27T08:00:54.604 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.604 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.604 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.605 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.605 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.605 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.605 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.605 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.605 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.605 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.605 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.606 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.606 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.606 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.606 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.606 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.606 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.606 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.606 INFO:teuthology.orchestra.run.smithi114.stdout: "ssh_config_file": { 2024-05-27T08:00:54.606 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssh_config_file", 2024-05-27T08:00:54.607 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.607 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.607 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.607 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.607 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.607 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.607 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.607 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T08:00:54.608 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.608 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.608 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.608 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.608 INFO:teuthology.orchestra.run.smithi114.stdout: "use_agent": { 2024-05-27T08:00:54.608 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "use_agent", 2024-05-27T08:00:54.608 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.608 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.609 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.609 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.609 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.609 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.609 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.609 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T08:00:54.609 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.609 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.610 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.610 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.610 INFO:teuthology.orchestra.run.smithi114.stdout: "use_repo_digest": { 2024-05-27T08:00:54.610 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "use_repo_digest", 2024-05-27T08:00:54.610 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.610 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.610 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.610 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.611 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.611 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.611 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.611 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T08:00:54.611 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.611 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.611 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.612 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_failed_host_check": { 2024-05-27T08:00:54.612 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_failed_host_check", 2024-05-27T08:00:54.612 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.612 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.612 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.612 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.612 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.612 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.613 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.613 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T08:00:54.613 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.613 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.613 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.613 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_stray_daemons": { 2024-05-27T08:00:54.613 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_stray_daemons", 2024-05-27T08:00:54.613 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.614 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.614 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.614 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.614 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.614 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.614 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.614 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T08:00:54.615 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.615 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.615 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.615 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.615 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_stray_hosts": { 2024-05-27T08:00:54.615 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_stray_hosts", 2024-05-27T08:00:54.615 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.615 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.616 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.616 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.616 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.616 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.616 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.616 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T08:00:54.616 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.616 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.616 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.617 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.617 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.617 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.617 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crash", 2024-05-27T08:00:54.617 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.617 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.617 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.618 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.618 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.618 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.618 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.618 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.618 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.618 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.618 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.619 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.619 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.619 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.619 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.619 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.619 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.619 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.619 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.622 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.622 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.622 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.622 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.622 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.622 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.622 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.623 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.623 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.623 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.623 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.623 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.623 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.623 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.623 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.623 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.624 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.624 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.624 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.624 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.624 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.624 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.624 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.624 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.624 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.625 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.625 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.625 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.625 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.625 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.625 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.625 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.625 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.626 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.626 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.626 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.626 INFO:teuthology.orchestra.run.smithi114.stdout: "retain_interval": { 2024-05-27T08:00:54.626 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "retain_interval", 2024-05-27T08:00:54.626 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.626 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.627 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.627 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "31536000", 2024-05-27T08:00:54.627 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.627 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.627 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.627 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T08:00:54.627 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.627 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.628 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.628 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.628 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.628 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.628 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.628 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.628 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.628 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.628 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.629 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.629 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.629 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.629 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.629 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.629 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.629 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_recent_interval": { 2024-05-27T08:00:54.630 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_recent_interval", 2024-05-27T08:00:54.630 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.630 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.630 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.630 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1209600", 2024-05-27T08:00:54.630 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.630 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.630 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.631 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T08:00:54.631 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.631 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.631 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.631 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.631 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.631 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.632 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "dashboard", 2024-05-27T08:00:54.632 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.632 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.632 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.632 INFO:teuthology.orchestra.run.smithi114.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T08:00:54.632 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T08:00:54.632 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.632 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.633 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.633 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:54.633 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.633 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.633 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.633 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.633 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.633 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.634 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.634 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.634 INFO:teuthology.orchestra.run.smithi114.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T08:00:54.634 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T08:00:54.634 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.634 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.634 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.634 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.635 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.635 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.635 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.635 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.635 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.635 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.635 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.635 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.636 INFO:teuthology.orchestra.run.smithi114.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T08:00:54.636 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T08:00:54.636 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.636 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.636 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.636 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.636 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.636 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.636 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.637 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.637 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.637 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.637 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.637 INFO:teuthology.orchestra.run.smithi114.stdout: "AUDIT_API_ENABLED": { 2024-05-27T08:00:54.637 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T08:00:54.637 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.638 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.638 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.638 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.638 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.638 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.638 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.638 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.638 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.639 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.639 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.643 INFO:teuthology.orchestra.run.smithi114.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T08:00:54.643 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T08:00:54.643 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.643 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.643 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.643 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.643 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.644 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.644 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.644 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.644 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.644 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.644 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.644 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.644 INFO:teuthology.orchestra.run.smithi114.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T08:00:54.644 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T08:00:54.645 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.645 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.645 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.645 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.645 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.645 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.645 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.645 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.646 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.646 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.646 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.646 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T08:00:54.646 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T08:00:54.646 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.646 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.647 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.647 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.647 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.647 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.647 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.647 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.647 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.647 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.648 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.648 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T08:00:54.648 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T08:00:54.648 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.648 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.648 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.648 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.649 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.649 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.649 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.649 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.649 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.649 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.649 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.649 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T08:00:54.650 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T08:00:54.650 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.650 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.650 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.650 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.650 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.650 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.650 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.651 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.651 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.651 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.651 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.651 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T08:00:54.651 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T08:00:54.651 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.652 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.652 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.652 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.652 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.652 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.652 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.652 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.652 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.652 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.653 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.653 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T08:00:54.653 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T08:00:54.653 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.653 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.653 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.653 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.653 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.654 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.654 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.654 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.654 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.654 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.654 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.654 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T08:00:54.655 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T08:00:54.655 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.655 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.655 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.655 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.655 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.655 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.655 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.656 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.656 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.656 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.656 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.656 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T08:00:54.656 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T08:00:54.656 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.657 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.657 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.657 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.657 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.657 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.657 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.657 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.657 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.658 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.658 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.658 INFO:teuthology.orchestra.run.smithi114.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T08:00:54.658 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T08:00:54.658 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.658 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.658 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.659 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.659 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.659 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.659 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.659 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.659 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.659 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.659 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.660 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T08:00:54.660 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T08:00:54.660 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.660 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.660 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.660 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-05-27T08:00:54.660 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.661 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.661 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.661 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.661 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.661 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.661 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.661 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T08:00:54.662 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T08:00:54.662 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.662 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.662 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.662 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.662 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.662 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.662 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.663 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.663 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.663 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.663 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.663 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.663 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_URL": { 2024-05-27T08:00:54.663 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_URL", 2024-05-27T08:00:54.663 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.663 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.664 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.664 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.664 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.664 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.664 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.664 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.664 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.664 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.665 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.665 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T08:00:54.665 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T08:00:54.665 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.665 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.665 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.665 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-05-27T08:00:54.666 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.666 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.666 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.666 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.666 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.666 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.666 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.667 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T08:00:54.667 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T08:00:54.667 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.667 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.667 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.667 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.667 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.667 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.668 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.668 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.668 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.668 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.668 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.668 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T08:00:54.668 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T08:00:54.669 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.669 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.669 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.669 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.669 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.669 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.669 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.669 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.669 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.670 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.670 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.670 INFO:teuthology.orchestra.run.smithi114.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T08:00:54.670 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T08:00:54.670 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.670 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.670 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.671 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.671 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.671 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.671 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.671 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.671 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.671 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.671 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.672 INFO:teuthology.orchestra.run.smithi114.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T08:00:54.672 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T08:00:54.672 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.672 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.672 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.672 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.672 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.673 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.673 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.673 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.673 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.673 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.673 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.673 INFO:teuthology.orchestra.run.smithi114.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T08:00:54.674 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T08:00:54.674 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.674 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.674 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.674 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "[]", 2024-05-27T08:00:54.674 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.674 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.674 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.675 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.675 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.675 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.675 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.675 INFO:teuthology.orchestra.run.smithi114.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T08:00:54.675 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T08:00:54.675 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.676 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.676 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.676 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "{}", 2024-05-27T08:00:54.676 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.676 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.676 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.676 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.676 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.677 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.677 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.677 INFO:teuthology.orchestra.run.smithi114.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T08:00:54.677 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T08:00:54.677 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.677 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.677 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.678 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.678 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.678 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.678 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.678 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.678 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.678 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.678 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.679 INFO:teuthology.orchestra.run.smithi114.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T08:00:54.679 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T08:00:54.679 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.679 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.679 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.679 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.679 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.679 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.680 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.680 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.680 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.680 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.680 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.680 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T08:00:54.681 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T08:00:54.681 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.681 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.681 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.681 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.681 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.681 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.681 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.681 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.682 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.682 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.682 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.682 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T08:00:54.682 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T08:00:54.682 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.682 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.683 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.683 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.683 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.683 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.683 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.683 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.683 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.683 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.683 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.684 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T08:00:54.684 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T08:00:54.684 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.684 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.684 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.684 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.684 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.685 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.685 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.685 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.685 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.685 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.685 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.685 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T08:00:54.686 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T08:00:54.686 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.686 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.686 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.686 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.686 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.686 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.686 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.687 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.687 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.687 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.687 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.687 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T08:00:54.687 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T08:00:54.687 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.688 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.688 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.688 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.688 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.688 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.688 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.688 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.688 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.688 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.689 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.689 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T08:00:54.689 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T08:00:54.689 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.689 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.689 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.689 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.689 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.690 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.690 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.690 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.690 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.690 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.690 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.690 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T08:00:54.691 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T08:00:54.691 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.691 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.691 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.691 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.691 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.691 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.691 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.692 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.692 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.692 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.692 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.692 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T08:00:54.692 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T08:00:54.692 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.693 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.693 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.693 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.693 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.693 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.693 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.693 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.694 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.694 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.694 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.694 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T08:00:54.694 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T08:00:54.694 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.694 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.695 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.695 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T08:00:54.695 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.695 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.695 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.695 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.695 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.695 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.695 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.696 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T08:00:54.696 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T08:00:54.696 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.696 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.696 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.696 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:54.696 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.697 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.697 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.697 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.697 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.697 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.697 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.697 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T08:00:54.698 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T08:00:54.698 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.698 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.698 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.698 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8", 2024-05-27T08:00:54.698 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.698 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.698 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.699 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.699 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.699 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.699 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.699 INFO:teuthology.orchestra.run.smithi114.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T08:00:54.699 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T08:00:54.699 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.700 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.700 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.700 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "45", 2024-05-27T08:00:54.700 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.700 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.700 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.700 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.700 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.701 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.701 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.701 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T08:00:54.701 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T08:00:54.701 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.701 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.701 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.701 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.702 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.702 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.702 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.702 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.702 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.702 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.702 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.703 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T08:00:54.703 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T08:00:54.703 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.703 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.703 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.703 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-05-27T08:00:54.703 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.704 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.704 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.704 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.704 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.704 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.704 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.704 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T08:00:54.704 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T08:00:54.705 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.705 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.705 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.705 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.705 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.705 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.705 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.705 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.706 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.706 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.706 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.706 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T08:00:54.706 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T08:00:54.706 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.706 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.707 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.707 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.707 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.707 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.707 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.707 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.707 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.707 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.707 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.708 INFO:teuthology.orchestra.run.smithi114.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T08:00:54.708 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T08:00:54.708 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.708 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.708 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.708 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.708 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.709 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.709 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.709 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.709 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.709 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.709 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.709 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T08:00:54.709 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T08:00:54.710 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.710 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.710 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.710 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.710 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.710 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.710 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.710 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.710 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.711 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.711 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.711 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T08:00:54.711 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T08:00:54.711 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.711 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.711 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.712 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:54.712 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.712 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.712 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.712 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.712 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.712 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.712 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.713 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T08:00:54.713 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T08:00:54.713 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.713 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.713 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.713 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-05-27T08:00:54.713 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.713 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.714 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.714 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.714 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.714 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.714 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.714 INFO:teuthology.orchestra.run.smithi114.stdout: "cross_origin_url": { 2024-05-27T08:00:54.714 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cross_origin_url", 2024-05-27T08:00:54.715 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.715 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.715 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.715 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.715 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.715 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.715 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.715 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.716 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.716 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.716 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.716 INFO:teuthology.orchestra.run.smithi114.stdout: "crt_file": { 2024-05-27T08:00:54.716 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crt_file", 2024-05-27T08:00:54.716 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.716 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.716 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.717 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.717 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.717 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.717 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.717 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.717 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.717 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.717 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.718 INFO:teuthology.orchestra.run.smithi114.stdout: "debug": { 2024-05-27T08:00:54.718 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "debug", 2024-05-27T08:00:54.718 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.718 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.718 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.718 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.718 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.719 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.719 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.719 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable/disable debug options", 2024-05-27T08:00:54.719 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.719 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.719 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.719 INFO:teuthology.orchestra.run.smithi114.stdout: "jwt_token_ttl": { 2024-05-27T08:00:54.720 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "jwt_token_ttl", 2024-05-27T08:00:54.720 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.720 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.720 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.720 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "28800", 2024-05-27T08:00:54.720 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.720 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.720 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.721 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.721 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.721 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.721 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.721 INFO:teuthology.orchestra.run.smithi114.stdout: "key_file": { 2024-05-27T08:00:54.721 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "key_file", 2024-05-27T08:00:54.721 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.721 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.722 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.722 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.722 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.722 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.722 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.722 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.722 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.722 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.723 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.723 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.723 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.723 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.723 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.723 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.723 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.723 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.724 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.724 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.724 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.724 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.724 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.724 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.724 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.724 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.725 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.725 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.725 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.725 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.725 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.725 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.725 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.726 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.726 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.726 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.726 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.726 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.726 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.726 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.726 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.727 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.727 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.727 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.727 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.727 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.727 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.727 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.727 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.728 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.728 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.728 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.728 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.728 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.728 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.728 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.728 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.729 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.729 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.729 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.729 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.729 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.729 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.729 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.729 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.730 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.730 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.730 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.730 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.730 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.730 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.730 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.731 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.731 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.731 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.731 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.731 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.731 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.731 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.731 INFO:teuthology.orchestra.run.smithi114.stdout: "motd": { 2024-05-27T08:00:54.731 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "motd", 2024-05-27T08:00:54.732 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.732 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.732 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.732 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.732 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.732 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.732 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.732 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "The message of the day", 2024-05-27T08:00:54.733 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.733 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.733 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.733 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.733 INFO:teuthology.orchestra.run.smithi114.stdout: "redirect_resolve_ip_addr": { 2024-05-27T08:00:54.733 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T08:00:54.733 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.733 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.733 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.734 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.734 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.734 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.734 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.734 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.734 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.734 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.734 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.735 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.735 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-05-27T08:00:54.735 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-05-27T08:00:54.735 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.735 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.735 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.735 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "::", 2024-05-27T08:00:54.735 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.736 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.736 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.736 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.736 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.736 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.736 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.736 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.736 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-05-27T08:00:54.737 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-05-27T08:00:54.737 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.737 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.737 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.737 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8080", 2024-05-27T08:00:54.737 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.737 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.737 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.738 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.738 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.738 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.738 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.738 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.738 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.738 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.738 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.739 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.739 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.739 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.739 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.739 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.739 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.739 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.739 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.740 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.740 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.740 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl": { 2024-05-27T08:00:54.740 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl", 2024-05-27T08:00:54.740 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.740 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.740 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.740 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.741 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.741 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.741 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.741 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.741 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.741 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.741 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.741 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.742 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl_server_port": { 2024-05-27T08:00:54.742 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl_server_port", 2024-05-27T08:00:54.742 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.742 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.742 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.742 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8443", 2024-05-27T08:00:54.742 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.742 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.743 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.743 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.743 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.743 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.743 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.743 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.743 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_behaviour": { 2024-05-27T08:00:54.743 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_behaviour", 2024-05-27T08:00:54.744 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.744 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.744 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.744 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "redirect", 2024-05-27T08:00:54.744 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.744 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.744 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.744 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.745 INFO:teuthology.orchestra.run.smithi114.stdout: "redirect" 2024-05-27T08:00:54.745 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.745 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.745 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.745 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.745 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.745 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.745 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_error_status_code": { 2024-05-27T08:00:54.745 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_error_status_code", 2024-05-27T08:00:54.746 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.746 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.746 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.746 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "500", 2024-05-27T08:00:54.746 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "400", 2024-05-27T08:00:54.746 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "599", 2024-05-27T08:00:54.746 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.746 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.747 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.747 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.747 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.747 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.747 INFO:teuthology.orchestra.run.smithi114.stdout: "url_prefix": { 2024-05-27T08:00:54.747 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "url_prefix", 2024-05-27T08:00:54.747 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.747 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.748 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.748 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.748 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.748 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.748 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.748 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.748 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.748 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.749 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.749 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.749 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.749 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.749 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.749 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "devicehealth", 2024-05-27T08:00:54.749 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.749 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.750 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.750 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_monitoring": { 2024-05-27T08:00:54.750 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enable_monitoring", 2024-05-27T08:00:54.750 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.750 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.750 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.750 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.750 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.751 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.751 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.751 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "monitor device health metrics", 2024-05-27T08:00:54.751 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.751 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.751 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.751 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.751 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.751 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.752 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.752 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.752 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.752 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.752 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.752 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.752 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.752 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.753 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.753 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.753 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.753 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.753 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.753 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.753 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.753 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.754 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.754 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.754 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.754 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.754 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.754 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.754 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.754 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.755 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.755 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.755 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.755 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.755 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.755 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.755 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.755 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.756 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.756 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.756 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.756 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.756 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.756 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.756 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.756 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.756 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.757 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.757 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.757 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.757 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.757 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.757 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.757 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.757 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.758 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.758 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.758 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.758 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.758 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.758 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.758 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.758 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.759 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.759 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.759 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.759 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.759 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.759 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.759 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.759 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.759 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.760 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.760 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.760 INFO:teuthology.orchestra.run.smithi114.stdout: "mark_out_threshold": { 2024-05-27T08:00:54.760 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mark_out_threshold", 2024-05-27T08:00:54.760 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.760 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.760 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.760 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "2419200", 2024-05-27T08:00:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T08:00:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": { 2024-05-27T08:00:54.762 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pool_name", 2024-05-27T08:00:54.762 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.762 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.762 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.762 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "device_health_metrics", 2024-05-27T08:00:54.762 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.762 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.762 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.763 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T08:00:54.763 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.763 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.763 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.763 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.763 INFO:teuthology.orchestra.run.smithi114.stdout: "retention_period": { 2024-05-27T08:00:54.763 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "retention_period", 2024-05-27T08:00:54.763 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.764 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.764 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.764 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15552000", 2024-05-27T08:00:54.764 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.764 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.764 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.764 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long to retain device health metrics", 2024-05-27T08:00:54.764 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "scrape_frequency": { 2024-05-27T08:00:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "scrape_frequency", 2024-05-27T08:00:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "86400", 2024-05-27T08:00:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T08:00:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "self_heal": { 2024-05-27T08:00:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "self_heal", 2024-05-27T08:00:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T08:00:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T08:00:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_threshold": { 2024-05-27T08:00:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_threshold", 2024-05-27T08:00:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "7257600", 2024-05-27T08:00:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T08:00:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "diskprediction_local", 2024-05-27T08:00:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.776 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.776 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.776 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.776 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.776 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.776 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.776 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.776 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.777 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.777 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.777 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.777 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.777 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.777 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.777 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.777 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.777 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.778 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.778 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.778 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.778 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.778 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.778 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.778 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.778 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.779 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.779 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.779 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.779 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.779 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.779 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.779 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.779 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.780 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.780 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.780 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.780 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.780 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.780 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.780 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.780 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.780 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.781 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.781 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.781 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.781 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.781 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.781 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.781 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.781 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.781 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.782 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.782 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.782 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.782 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.782 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.782 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.782 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.782 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.783 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.783 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.783 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.783 INFO:teuthology.orchestra.run.smithi114.stdout: "predict_interval": { 2024-05-27T08:00:54.783 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "predict_interval", 2024-05-27T08:00:54.783 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.783 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.783 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.784 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "86400", 2024-05-27T08:00:54.784 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.784 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.784 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.784 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.784 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.784 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.784 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.784 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.785 INFO:teuthology.orchestra.run.smithi114.stdout: "predictor_model": { 2024-05-27T08:00:54.785 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "predictor_model", 2024-05-27T08:00:54.785 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.785 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.785 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.785 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "prophetstor", 2024-05-27T08:00:54.785 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.785 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.786 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.786 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.786 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.786 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.786 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.786 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.786 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:54.786 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:54.787 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.787 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.787 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.787 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:54.787 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.787 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.787 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.787 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.787 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.788 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.788 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.788 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.788 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.788 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.788 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.788 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.788 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.789 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.789 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.789 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.789 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.789 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.789 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.789 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.789 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.789 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.790 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.790 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.790 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.790 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "influx", 2024-05-27T08:00:54.790 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": false, 2024-05-27T08:00:54.790 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "influxdb python module not found", 2024-05-27T08:00:54.790 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.790 INFO:teuthology.orchestra.run.smithi114.stdout: "batch_size": { 2024-05-27T08:00:54.791 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "batch_size", 2024-05-27T08:00:54.791 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.791 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.791 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.791 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5000", 2024-05-27T08:00:54.791 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.791 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.791 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.792 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T08:00:54.792 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.792 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.792 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.792 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.792 INFO:teuthology.orchestra.run.smithi114.stdout: "database": { 2024-05-27T08:00:54.792 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "database", 2024-05-27T08:00:54.792 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.793 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.793 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.793 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "ceph", 2024-05-27T08:00:54.793 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.793 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.793 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.793 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T08:00:54.794 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.794 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.794 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.794 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": { 2024-05-27T08:00:54.794 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "hostname", 2024-05-27T08:00:54.794 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.794 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.795 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.795 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.795 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.795 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.795 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.795 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB server hostname", 2024-05-27T08:00:54.795 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.795 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.796 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.796 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.796 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:54.796 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:54.796 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.796 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.796 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.796 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "30", 2024-05-27T08:00:54.796 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "5", 2024-05-27T08:00:54.797 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.797 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.797 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T08:00:54.797 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.797 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.797 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.797 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.797 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.798 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.798 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.798 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.798 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.798 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.798 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.798 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.798 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.799 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.799 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.799 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.799 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.799 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.799 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.799 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.800 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.800 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.800 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.800 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.800 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.800 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.800 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.801 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.801 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.801 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.801 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.801 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.801 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.801 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.802 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.802 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.802 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.802 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.802 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.802 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.802 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.802 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.803 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.803 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.803 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.803 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.803 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.803 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.803 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.803 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.804 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.804 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.804 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.804 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.804 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.804 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.804 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.804 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.805 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.805 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.805 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.805 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.805 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.805 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.805 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.805 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.806 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.806 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.806 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.806 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.806 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.806 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.806 INFO:teuthology.orchestra.run.smithi114.stdout: "password": { 2024-05-27T08:00:54.806 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "password", 2024-05-27T08:00:54.807 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.807 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.807 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.807 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.807 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.807 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.807 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.807 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "password of InfluxDB server user", 2024-05-27T08:00:54.808 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.808 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.808 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.808 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.808 INFO:teuthology.orchestra.run.smithi114.stdout: "port": { 2024-05-27T08:00:54.808 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "port", 2024-05-27T08:00:54.808 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.808 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.808 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.809 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8086", 2024-05-27T08:00:54.809 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.809 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.809 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.809 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB server port", 2024-05-27T08:00:54.809 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.809 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.809 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.810 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.810 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.810 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.810 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.810 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.810 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.810 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.810 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.810 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.811 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.811 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.811 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.811 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.811 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.811 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl": { 2024-05-27T08:00:54.811 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl", 2024-05-27T08:00:54.812 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.812 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.812 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.812 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "false", 2024-05-27T08:00:54.812 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.812 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.812 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.812 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T08:00:54.813 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.813 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.813 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.813 INFO:teuthology.orchestra.run.smithi114.stdout: "threads": { 2024-05-27T08:00:54.813 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "threads", 2024-05-27T08:00:54.813 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.813 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.814 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.814 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-05-27T08:00:54.814 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-05-27T08:00:54.814 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "32", 2024-05-27T08:00:54.814 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.814 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T08:00:54.814 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.815 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.815 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.815 INFO:teuthology.orchestra.run.smithi114.stdout: "username": { 2024-05-27T08:00:54.815 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "username", 2024-05-27T08:00:54.815 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.815 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.815 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.816 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.816 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.816 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.816 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.816 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "username of InfluxDB server user", 2024-05-27T08:00:54.816 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.816 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.816 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.817 INFO:teuthology.orchestra.run.smithi114.stdout: "verify_ssl": { 2024-05-27T08:00:54.817 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "verify_ssl", 2024-05-27T08:00:54.817 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.817 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.817 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.817 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "true", 2024-05-27T08:00:54.817 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.818 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.818 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.818 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T08:00:54.818 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.818 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.818 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.818 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.819 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.819 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.819 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.819 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "insights", 2024-05-27T08:00:54.819 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.819 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.819 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.819 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.820 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.820 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.820 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.820 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.820 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.820 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.820 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.820 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.821 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.821 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.821 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.821 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.821 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.821 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.821 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.821 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.822 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.822 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.822 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.822 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.822 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.822 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.822 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.823 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.823 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.823 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.823 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.823 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.823 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.823 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.823 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.824 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.824 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.824 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.824 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.824 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.824 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.824 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.825 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.825 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.825 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.825 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.825 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.825 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.825 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.825 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.826 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.826 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.826 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.826 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.826 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.826 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.827 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.827 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.827 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.827 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.827 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.827 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.827 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.827 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.827 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.828 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.828 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.828 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.828 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.828 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.828 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.828 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.829 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.829 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.829 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.829 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.829 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.829 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.829 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.829 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.830 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.830 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.830 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.830 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.830 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "iostat", 2024-05-27T08:00:54.830 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.831 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.831 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.831 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.831 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.831 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.831 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.831 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.832 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.832 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.832 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.832 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.832 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.832 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.832 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.832 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.832 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.833 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.833 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.833 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.833 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.833 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.833 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.833 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.834 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.834 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.834 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.834 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.834 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.834 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.834 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.834 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.835 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.835 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.835 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.835 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.835 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.835 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.835 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.836 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.836 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.836 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.836 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.836 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.836 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.836 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.836 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.837 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.837 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.837 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.837 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.837 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.837 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.837 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.837 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.838 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.838 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.838 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.838 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.838 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.838 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.838 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.839 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.839 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.839 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.839 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.839 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.839 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.839 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.840 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.840 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.840 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.840 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.840 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.840 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.840 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.841 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.841 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.841 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.841 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.841 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.841 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.841 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.841 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.842 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "k8sevents", 2024-05-27T08:00:54.842 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.842 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.842 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.847 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_event_retention_days": { 2024-05-27T08:00:54.847 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ceph_event_retention_days", 2024-05-27T08:00:54.847 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.848 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.848 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.848 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "7", 2024-05-27T08:00:54.848 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.848 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.848 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.848 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T08:00:54.848 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.849 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.849 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.849 INFO:teuthology.orchestra.run.smithi114.stdout: "config_check_secs": { 2024-05-27T08:00:54.849 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_check_secs", 2024-05-27T08:00:54.849 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.849 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.849 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.850 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:54.850 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "10", 2024-05-27T08:00:54.850 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.850 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.850 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T08:00:54.850 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.850 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.850 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.851 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.851 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.851 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.851 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.851 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.851 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.851 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.851 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.852 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.852 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.852 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.852 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.852 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.852 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.852 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.852 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.853 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.853 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.853 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.853 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.853 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.853 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.853 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.853 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.854 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.854 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.854 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.854 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.854 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.854 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.854 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.854 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.855 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.855 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.855 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.855 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.855 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.855 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.855 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.856 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.856 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.856 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.856 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.856 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.856 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.856 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.856 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.856 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.857 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.857 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.857 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.857 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.857 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.857 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.857 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.858 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.858 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.858 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.858 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.858 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.858 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.858 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.858 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.859 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.859 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.859 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.859 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.859 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.859 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.859 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.860 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.860 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.860 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.860 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.860 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.860 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.860 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.860 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.860 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.861 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.861 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.861 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.861 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.861 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "localpool", 2024-05-27T08:00:54.861 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.861 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.861 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.862 INFO:teuthology.orchestra.run.smithi114.stdout: "failure_domain": { 2024-05-27T08:00:54.862 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "failure_domain", 2024-05-27T08:00:54.862 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.862 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.862 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.862 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "host", 2024-05-27T08:00:54.862 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.862 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.863 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.863 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "failure domain for any created local pool", 2024-05-27T08:00:54.863 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T08:00:54.863 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.863 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.863 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.863 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.863 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.863 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.864 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.864 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.864 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.864 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.864 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.864 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.864 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.864 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.865 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.865 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.865 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.865 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.865 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.865 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.865 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.865 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.866 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.866 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.866 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.866 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.866 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.866 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.866 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.867 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.867 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.867 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.867 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.867 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.867 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.867 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.868 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.868 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.868 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.868 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.868 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.868 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.868 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.868 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.869 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.869 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.869 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.869 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.869 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.869 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.869 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.869 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.870 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.870 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.870 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.870 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.870 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.870 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.871 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.871 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.871 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.871 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.871 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.871 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.871 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.871 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.872 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.872 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.872 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.872 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": { 2024-05-27T08:00:54.872 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "min_size", 2024-05-27T08:00:54.872 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.872 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.873 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.873 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.873 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.873 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.873 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.873 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default min_size for any created local pool", 2024-05-27T08:00:54.873 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T08:00:54.873 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.873 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.874 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rep": { 2024-05-27T08:00:54.874 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "num_rep", 2024-05-27T08:00:54.874 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.874 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.874 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.874 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3", 2024-05-27T08:00:54.874 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.875 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.875 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.875 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default replica count for any created local pool", 2024-05-27T08:00:54.875 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.875 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.875 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.875 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": { 2024-05-27T08:00:54.876 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pg_num", 2024-05-27T08:00:54.876 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.876 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.876 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.876 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "128", 2024-05-27T08:00:54.876 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.876 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.876 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.877 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T08:00:54.877 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.877 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.877 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.877 INFO:teuthology.orchestra.run.smithi114.stdout: "prefix": { 2024-05-27T08:00:54.877 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prefix", 2024-05-27T08:00:54.877 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.877 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.878 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.878 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.878 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.878 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.878 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.878 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "name prefix for any created local pool", 2024-05-27T08:00:54.878 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.879 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.879 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.879 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.879 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.879 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.879 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.879 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.880 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.880 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.880 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.880 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.880 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.880 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.880 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.880 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.881 INFO:teuthology.orchestra.run.smithi114.stdout: "subtree": { 2024-05-27T08:00:54.881 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "subtree", 2024-05-27T08:00:54.881 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.881 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.881 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.881 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "rack", 2024-05-27T08:00:54.881 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.881 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.882 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.882 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T08:00:54.882 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T08:00:54.882 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.882 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.882 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.882 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.883 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.883 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mds_autoscaler", 2024-05-27T08:00:54.883 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.883 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.883 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.883 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.883 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.883 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.884 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.884 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.884 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.884 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.884 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.884 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.884 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.884 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.885 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.885 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.885 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.885 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.885 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.885 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.885 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.885 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.886 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.886 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.886 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.886 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.886 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.886 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.886 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.886 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.887 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.887 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.887 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.887 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.887 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.887 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.887 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.888 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.888 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.888 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.888 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.888 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.888 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.888 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.888 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.889 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.889 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.889 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.889 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.889 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.889 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.889 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.890 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.890 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.890 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.890 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.890 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.890 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.890 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.891 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.891 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.891 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.891 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.891 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.891 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.891 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.892 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.892 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.892 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.892 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.892 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.892 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.892 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.893 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.893 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.893 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.893 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.893 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.893 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.893 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.893 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.894 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.894 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.894 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.894 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.894 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.894 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.894 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.895 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mirroring", 2024-05-27T08:00:54.895 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.895 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.895 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.895 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.895 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.895 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.895 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.896 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.896 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.896 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.896 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.896 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.896 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.896 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.896 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.897 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.897 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.897 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.897 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.897 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.897 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.897 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.897 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.897 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.898 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.898 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.898 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.898 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.898 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.898 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.898 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.898 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.899 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.899 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.899 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.899 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.899 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.899 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.899 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.899 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.900 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.900 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.900 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.900 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.900 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.900 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.900 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.900 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.900 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.901 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.901 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.901 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.901 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.901 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.901 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.901 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.901 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.902 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.902 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.902 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.902 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.902 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.902 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.902 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.902 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.903 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.903 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.903 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.903 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.903 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.903 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.903 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.904 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.904 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.904 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.904 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.904 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.904 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.904 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.904 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.904 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.905 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.905 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.905 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.905 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.905 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.905 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.905 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.905 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.906 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "nfs", 2024-05-27T08:00:54.906 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.906 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.906 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.906 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.906 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.906 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.906 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.906 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.907 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.907 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.907 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.907 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.907 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.907 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.907 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.908 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.908 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.908 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.908 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.908 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.908 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.908 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.908 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.909 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.909 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.909 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.909 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.909 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.909 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.909 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.909 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.909 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.910 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.910 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.910 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.910 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.910 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.910 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.910 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.911 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.911 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.911 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.911 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.911 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.911 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.911 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.911 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.912 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.912 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.912 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.912 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.912 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.912 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.912 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.912 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.913 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.913 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.913 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.913 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.913 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.913 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.913 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.913 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.914 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.914 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.914 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.914 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.914 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.914 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.915 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.915 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.915 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.915 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.915 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.915 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.915 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.915 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.915 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.916 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.916 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.916 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.916 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.916 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.916 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.916 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.916 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.917 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "orchestrator", 2024-05-27T08:00:54.917 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.917 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.917 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.917 INFO:teuthology.orchestra.run.smithi114.stdout: "fail_fs": { 2024-05-27T08:00:54.917 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "fail_fs", 2024-05-27T08:00:54.917 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.918 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.918 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.918 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.918 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.918 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.918 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.918 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T08:00:54.918 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.919 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.919 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.919 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.919 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.919 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.919 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.919 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.919 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator": { 2024-05-27T08:00:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "orchestrator", 2024-05-27T08:00:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm", 2024-05-27T08:00:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "rook", 2024-05-27T08:00:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "test_orchestrator" 2024-05-27T08:00:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Orchestrator backend", 2024-05-27T08:00:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "osd_perf_query", 2024-05-27T08:00:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.933 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.933 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.933 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.933 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.933 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.933 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.933 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.934 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.934 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.934 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.934 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.934 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.934 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.934 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.934 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.935 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.935 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.935 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.935 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.935 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.935 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.935 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.936 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.936 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.936 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.936 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.936 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.936 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.936 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.936 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.937 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.937 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.937 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.937 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.937 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.937 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.937 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.938 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.938 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.938 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.938 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.938 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.938 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.938 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.938 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.939 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.939 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.939 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.939 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.939 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.939 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.939 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.940 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.940 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.940 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.940 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.940 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.940 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.940 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.940 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.940 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.941 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.941 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.941 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.941 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.941 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.941 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.941 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.942 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.942 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.942 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.942 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.942 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.942 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.942 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.943 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.943 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.943 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.943 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.943 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "osd_support", 2024-05-27T08:00:54.943 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.943 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.944 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.944 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.944 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.944 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.944 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.944 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.944 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.944 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.945 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.945 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.945 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.945 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.945 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.945 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.945 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.946 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.946 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.946 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.946 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.946 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.946 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.946 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.947 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.947 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.947 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.947 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.947 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.947 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.947 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.947 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.948 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.948 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.948 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.948 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.948 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.948 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.948 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.949 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.949 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.949 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.949 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.949 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.949 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.949 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.949 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.949 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.950 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.950 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.950 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.950 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.950 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.950 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.950 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.951 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.951 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.951 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.951 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.951 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.951 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.951 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.951 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.952 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.952 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.952 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.952 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.952 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.952 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.952 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.953 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.953 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.953 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.953 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.953 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.953 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.953 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.953 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.954 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.954 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.954 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.954 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.954 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.954 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.954 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.955 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.955 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pg_autoscaler", 2024-05-27T08:00:54.955 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.955 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.955 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.955 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.955 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.955 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.956 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.956 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.956 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.956 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.956 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.956 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.956 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.956 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.956 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.957 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.957 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.957 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.957 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.957 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.957 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.957 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.957 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.958 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.958 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.958 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.958 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.958 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.958 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.958 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.959 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.959 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.959 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.959 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.959 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.959 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.959 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.960 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.960 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.960 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.960 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.960 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.960 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.960 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.960 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.960 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.961 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.961 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.961 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.961 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.961 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.961 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.961 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.961 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.962 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.962 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.962 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.962 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.962 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.962 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.962 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.962 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.963 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.963 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.963 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.963 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.963 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.963 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.963 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.963 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.964 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:54.964 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:54.964 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.964 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.964 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.964 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:54.964 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.964 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.965 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.965 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.965 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.965 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.965 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.965 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.965 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.966 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.966 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.966 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.966 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.966 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.966 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.966 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.966 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.966 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.967 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.967 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.967 INFO:teuthology.orchestra.run.smithi114.stdout: "threshold": { 2024-05-27T08:00:54.967 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "threshold", 2024-05-27T08:00:54.967 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:54.967 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.967 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.968 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3.0", 2024-05-27T08:00:54.968 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1.0", 2024-05-27T08:00:54.968 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.968 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.968 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "scaling threshold", 2024-05-27T08:00:54.968 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T08:00:54.968 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.968 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.969 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.969 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.969 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "progress", 2024-05-27T08:00:54.969 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.969 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.969 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.970 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_pg_recovery_event": { 2024-05-27T08:00:54.970 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_pg_recovery_event", 2024-05-27T08:00:54.970 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.970 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.970 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.970 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.970 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.970 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.971 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.971 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T08:00:54.971 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.971 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.971 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.971 INFO:teuthology.orchestra.run.smithi114.stdout: "enabled": { 2024-05-27T08:00:54.971 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enabled", 2024-05-27T08:00:54.972 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.972 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.972 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.972 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.972 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.972 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.972 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.972 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.972 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.973 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.973 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.973 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.973 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.973 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.973 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.973 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.973 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.974 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.974 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.974 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.974 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.974 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.974 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.974 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.974 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.975 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.975 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.975 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.975 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.975 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.975 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.975 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.975 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.976 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.976 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.976 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.976 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.976 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.976 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.976 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.976 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.977 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.977 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.977 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.977 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.977 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.977 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.977 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.978 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.978 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.978 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.978 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.978 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.978 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.978 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.978 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.979 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.979 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.979 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.979 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.979 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.979 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.979 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.979 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.980 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.980 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.980 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.980 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.980 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.980 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.980 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.981 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.981 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.981 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.981 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.981 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.981 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.981 INFO:teuthology.orchestra.run.smithi114.stdout: "max_completed_events": { 2024-05-27T08:00:54.981 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_completed_events", 2024-05-27T08:00:54.982 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.982 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.982 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.982 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "50", 2024-05-27T08:00:54.982 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.982 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.982 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.982 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "number of past completed events to remember", 2024-05-27T08:00:54.983 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.983 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.983 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.983 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.983 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:54.983 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:54.983 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:54.983 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.983 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.984 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-05-27T08:00:54.984 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.984 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.984 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.984 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long the module is going to sleep", 2024-05-27T08:00:54.984 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.984 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.984 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.985 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:54.985 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:54.985 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:54.985 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:54.985 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.985 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:54.985 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.985 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.985 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.986 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.986 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.986 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.986 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.986 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.986 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:54.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.986 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:54.987 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prometheus", 2024-05-27T08:00:54.987 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:54.987 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:54.987 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:54.987 INFO:teuthology.orchestra.run.smithi114.stdout: "cache": { 2024-05-27T08:00:54.987 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cache", 2024-05-27T08:00:54.987 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.987 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.988 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:54.988 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.988 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.988 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.988 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.988 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.988 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.988 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.989 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.989 INFO:teuthology.orchestra.run.smithi114.stdout: "exclude_perf_counters": { 2024-05-27T08:00:54.989 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "exclude_perf_counters", 2024-05-27T08:00:54.989 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.989 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.989 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.989 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:54.990 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.990 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.990 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.990 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T08:00:54.990 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T08:00:54.990 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.990 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.991 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:54.991 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:54.991 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.991 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.991 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.991 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:54.991 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.991 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.992 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.992 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.992 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.992 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.992 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.992 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.992 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.992 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.993 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.993 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.993 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.993 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.993 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:54.993 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:54.993 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.993 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.994 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.994 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.994 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.994 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.994 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.994 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.994 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.994 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.995 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.995 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:54.995 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:54.995 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.995 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.995 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.995 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:54.995 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.996 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.996 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:54.996 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:54.996 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:54.996 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:54.996 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:54.996 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:54.996 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:54.997 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:54.997 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.997 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.997 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.997 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.997 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:54.997 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:54.998 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:54.998 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:54.998 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:54.998 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:54.998 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:54.998 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:54.998 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:54.998 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:54.999 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:54.999 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:54.999 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:54.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:54.999 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_stats_pools": { 2024-05-27T08:00:54.999 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_stats_pools", 2024-05-27T08:00:54.999 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:54.999 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.000 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.000 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.000 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.000 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.000 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.000 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.000 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.001 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.001 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.001 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T08:00:55.001 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T08:00:55.001 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.001 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.001 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.002 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "300", 2024-05-27T08:00:55.002 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.002 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.002 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.002 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.002 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.002 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.002 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.003 INFO:teuthology.orchestra.run.smithi114.stdout: "scrape_interval": { 2024-05-27T08:00:55.003 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "scrape_interval", 2024-05-27T08:00:55.003 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:55.003 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.003 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.003 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15.0", 2024-05-27T08:00:55.003 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.003 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.004 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.004 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.004 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.004 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.004 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.004 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.004 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-05-27T08:00:55.004 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-05-27T08:00:55.004 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.005 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.005 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.005 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "::", 2024-05-27T08:00:55.005 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.005 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.005 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.005 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T08:00:55.005 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.006 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.006 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.006 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-05-27T08:00:55.006 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-05-27T08:00:55.006 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.006 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.006 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.007 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "9283", 2024-05-27T08:00:55.007 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.007 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.007 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.007 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T08:00:55.007 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.007 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.007 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.008 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.008 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.008 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.008 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.008 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.008 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.008 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.008 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.009 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.009 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.009 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.009 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.009 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.009 INFO:teuthology.orchestra.run.smithi114.stdout: "stale_cache_strategy": { 2024-05-27T08:00:55.009 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "stale_cache_strategy", 2024-05-27T08:00:55.010 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.010 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.010 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.010 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "log", 2024-05-27T08:00:55.010 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.010 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.010 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.010 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.011 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.011 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.011 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.011 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_behaviour": { 2024-05-27T08:00:55.011 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_behaviour", 2024-05-27T08:00:55.011 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.011 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.012 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.012 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "default", 2024-05-27T08:00:55.012 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.012 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.012 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.012 INFO:teuthology.orchestra.run.smithi114.stdout: "default", 2024-05-27T08:00:55.012 INFO:teuthology.orchestra.run.smithi114.stdout: "error" 2024-05-27T08:00:55.012 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.013 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.013 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.013 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.013 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.013 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_error_status_code": { 2024-05-27T08:00:55.013 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_error_status_code", 2024-05-27T08:00:55.013 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.014 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.014 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.014 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "500", 2024-05-27T08:00:55.014 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "400", 2024-05-27T08:00:55.014 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "599", 2024-05-27T08:00:55.014 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.014 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.014 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.015 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.015 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.015 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.015 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.015 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.015 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_support", 2024-05-27T08:00:55.015 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.016 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.016 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.016 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.016 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.016 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.016 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.016 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.016 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.017 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.017 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.017 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.017 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.017 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.017 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.017 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.017 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.018 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.018 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.018 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.018 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.018 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.018 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.018 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.019 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.019 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.019 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.019 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.019 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.019 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.019 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.019 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.019 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.020 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.020 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.020 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.020 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.020 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.020 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.020 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.020 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.021 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.021 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.021 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.021 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.021 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.021 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.021 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.021 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.022 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.022 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.022 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.022 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.022 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.022 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.022 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.023 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.023 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.023 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.023 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.023 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.023 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.023 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.023 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.024 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.024 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.024 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.024 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.024 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.024 INFO:teuthology.orchestra.run.smithi114.stdout: "max_concurrent_snap_create": { 2024-05-27T08:00:55.025 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_concurrent_snap_create", 2024-05-27T08:00:55.025 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.025 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.025 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.025 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:55.025 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.025 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.025 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.026 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.026 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.026 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.026 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.026 INFO:teuthology.orchestra.run.smithi114.stdout: "mirror_snapshot_schedule": { 2024-05-27T08:00:55.026 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T08:00:55.026 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.027 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.027 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.027 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.027 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.027 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.027 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.027 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.027 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.028 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.028 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.028 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.028 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.028 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.028 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.029 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.029 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.029 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.029 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.029 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.029 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.029 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.029 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.030 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.030 INFO:teuthology.orchestra.run.smithi114.stdout: "trash_purge_schedule": { 2024-05-27T08:00:55.030 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "trash_purge_schedule", 2024-05-27T08:00:55.030 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.030 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.030 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.030 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.031 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.031 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.031 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.031 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.031 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.031 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.031 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.031 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.031 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.032 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "restful", 2024-05-27T08:00:55.032 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.032 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.032 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.032 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_auth": { 2024-05-27T08:00:55.032 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enable_auth", 2024-05-27T08:00:55.033 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.033 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.033 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.033 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.033 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.033 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.033 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.033 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.033 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.034 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.034 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.034 INFO:teuthology.orchestra.run.smithi114.stdout: "key_file": { 2024-05-27T08:00:55.034 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "key_file", 2024-05-27T08:00:55.034 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.034 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.035 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.035 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.035 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.035 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.035 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.035 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.035 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.035 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.035 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.036 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.036 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.036 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.036 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.036 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.036 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.036 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.036 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.037 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.037 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.037 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.037 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.037 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.037 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.037 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.037 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.037 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.037 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.038 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.038 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.038 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.038 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.038 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.038 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.038 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.039 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.039 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.039 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.039 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.039 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.039 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.039 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.039 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.040 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.040 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.040 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.040 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.040 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.040 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.040 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.040 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.040 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.041 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.041 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.041 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.041 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.041 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.041 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.041 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.041 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.041 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.042 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.042 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.042 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.042 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.042 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.042 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.042 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.043 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.043 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.043 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.043 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.043 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.043 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.043 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.043 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.043 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-05-27T08:00:55.044 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-05-27T08:00:55.044 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.044 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.044 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.044 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.044 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.044 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.044 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.045 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.045 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.045 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.045 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.045 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-05-27T08:00:55.045 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-05-27T08:00:55.045 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.045 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.046 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.046 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.046 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.046 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.046 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.046 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.050 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.050 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.050 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.051 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.051 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.051 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.051 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.051 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.051 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.051 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.052 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.052 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.052 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.052 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.052 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.052 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.052 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.052 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.053 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rgw", 2024-05-27T08:00:55.053 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.053 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.053 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.053 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.053 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.053 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.053 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.054 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.054 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.054 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.054 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.054 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.054 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.054 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.054 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.055 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.055 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.055 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.055 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.055 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.055 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.055 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.055 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.056 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.056 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.056 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.056 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.056 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.056 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.056 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.057 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.057 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.057 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.057 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.057 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.057 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.057 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.057 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.058 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.058 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.058 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.058 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.058 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.058 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.058 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.058 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.059 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.059 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.059 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.059 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.059 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.059 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.059 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.059 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.060 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.060 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.060 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.060 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.060 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.060 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.060 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.060 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.061 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.061 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.061 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.061 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.061 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.061 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.061 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.062 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.062 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.062 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.062 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.062 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.062 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.062 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.062 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.063 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.063 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.063 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.063 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.063 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.063 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.063 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.064 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rook", 2024-05-27T08:00:55.064 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.064 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.064 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.064 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.064 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.064 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.064 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.065 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.065 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.065 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.065 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.065 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.065 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.065 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.065 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.065 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.066 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.066 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.066 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.066 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.066 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.066 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.066 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.067 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.067 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.067 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.067 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.067 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.067 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.067 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.067 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.068 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.068 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.068 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.068 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.068 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.068 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.068 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.069 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.069 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.069 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.069 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.069 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.069 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.069 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.069 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.070 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.070 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.070 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.070 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.070 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.070 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.070 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.070 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.071 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.071 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.071 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.071 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.071 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.071 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.071 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.072 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.072 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.072 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.072 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.072 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.072 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.072 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.072 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.073 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.073 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.073 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.073 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.073 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.073 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.073 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.073 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.074 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.074 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.074 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.074 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.074 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.074 INFO:teuthology.orchestra.run.smithi114.stdout: "storage_class": { 2024-05-27T08:00:55.074 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "storage_class", 2024-05-27T08:00:55.074 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.075 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.075 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.075 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "local", 2024-05-27T08:00:55.075 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.075 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.075 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.075 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T08:00:55.075 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.076 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.076 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.076 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.076 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.076 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "selftest", 2024-05-27T08:00:55.076 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.076 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.077 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.077 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.077 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.077 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.077 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.077 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.077 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.077 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.078 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.078 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.078 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.078 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.078 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.078 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.078 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.078 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.079 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.079 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.079 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.079 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.079 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.079 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.079 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.079 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.080 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.080 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.080 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.080 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.080 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.080 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.080 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.080 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.080 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "roption1": { 2024-05-27T08:00:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "roption1", 2024-05-27T08:00:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "roption2": { 2024-05-27T08:00:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "roption2", 2024-05-27T08:00:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "xyz", 2024-05-27T08:00:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption1": { 2024-05-27T08:00:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption1", 2024-05-27T08:00:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption2": { 2024-05-27T08:00:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption2", 2024-05-27T08:00:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption3": { 2024-05-27T08:00:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption3", 2024-05-27T08:00:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption4": { 2024-05-27T08:00:55.094 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption4", 2024-05-27T08:00:55.094 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.094 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.094 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.094 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.094 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.094 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.094 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.095 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.095 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.095 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.095 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.095 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption5": { 2024-05-27T08:00:55.095 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption5", 2024-05-27T08:00:55.095 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.095 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.096 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.096 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.096 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.096 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.096 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.096 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.096 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.096 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.097 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.097 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption6": { 2024-05-27T08:00:55.097 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption6", 2024-05-27T08:00:55.097 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.097 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.097 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.097 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.097 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.098 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.098 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.098 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.098 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.098 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.098 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.098 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption7": { 2024-05-27T08:00:55.099 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption7", 2024-05-27T08:00:55.099 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.099 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.099 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.099 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.099 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-05-27T08:00:55.099 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "42", 2024-05-27T08:00:55.099 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.099 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.100 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.100 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.100 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.100 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.100 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.100 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.100 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.101 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.101 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.101 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.101 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.101 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.101 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.101 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.101 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.101 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.102 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.102 INFO:teuthology.orchestra.run.smithi114.stdout: "testkey": { 2024-05-27T08:00:55.102 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testkey", 2024-05-27T08:00:55.102 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.102 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.102 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.102 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.102 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.103 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.103 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.103 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.103 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.103 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.103 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.103 INFO:teuthology.orchestra.run.smithi114.stdout: "testlkey": { 2024-05-27T08:00:55.104 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testlkey", 2024-05-27T08:00:55.104 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.104 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.104 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.104 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.104 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.104 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.104 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.104 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.105 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.105 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.105 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.105 INFO:teuthology.orchestra.run.smithi114.stdout: "testnewline": { 2024-05-27T08:00:55.105 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testnewline", 2024-05-27T08:00:55.105 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.105 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.105 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.106 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.106 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.106 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.106 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.106 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.106 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.106 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.106 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.107 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.107 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.107 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smb", 2024-05-27T08:00:55.107 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.107 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.107 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.107 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.108 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.108 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.108 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.108 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.108 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.108 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.108 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.108 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.109 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.109 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.109 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.109 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.109 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.109 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.109 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.109 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.110 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.110 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.110 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.110 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.110 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.110 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.110 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.111 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.111 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.111 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.111 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.111 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.111 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.111 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.111 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.112 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.112 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.112 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.112 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.112 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.112 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.112 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.113 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.113 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.113 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.113 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.113 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.113 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.113 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.113 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.113 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.114 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.114 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.114 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.114 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.114 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.114 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.114 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.115 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.115 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.115 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.115 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.115 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.115 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.115 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.115 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.115 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.116 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.116 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.116 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.116 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.116 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.116 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.116 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.117 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.117 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.117 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.117 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.117 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.117 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.117 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.117 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.118 INFO:teuthology.orchestra.run.smithi114.stdout: "update_orchestration": { 2024-05-27T08:00:55.118 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "update_orchestration", 2024-05-27T08:00:55.118 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.118 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.118 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.118 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.118 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.118 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.119 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.119 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T08:00:55.119 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.119 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.119 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.119 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.119 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.120 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snap_schedule", 2024-05-27T08:00:55.120 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.120 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.120 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.120 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_m_granularity": { 2024-05-27T08:00:55.120 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_m_granularity", 2024-05-27T08:00:55.120 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.120 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.121 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.121 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.121 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.121 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.121 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.121 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T08:00:55.121 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.121 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.122 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.122 INFO:teuthology.orchestra.run.smithi114.stdout: "dump_on_update": { 2024-05-27T08:00:55.122 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "dump_on_update", 2024-05-27T08:00:55.122 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.122 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.122 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.122 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.123 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.123 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.123 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.123 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "dump database to debug log on update", 2024-05-27T08:00:55.123 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.123 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.123 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.123 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.124 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.124 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.124 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.124 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.124 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.124 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.124 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.124 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.125 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.125 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.125 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.125 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.125 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.125 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.125 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.125 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.125 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.126 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.126 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.126 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.126 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.126 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.126 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.127 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.127 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.127 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.127 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.127 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.127 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.127 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.127 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.127 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.128 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.128 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.128 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.128 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.128 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.128 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.129 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.129 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.129 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.129 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.129 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.129 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.129 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.129 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.130 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.130 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.130 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.130 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.130 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.130 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.130 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.131 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.131 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.131 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.131 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.131 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.131 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.131 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.132 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.132 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.132 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.132 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.132 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.132 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.132 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.132 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.133 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.133 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.133 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.133 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.133 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.133 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.133 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.133 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.134 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.134 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.134 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.134 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.134 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "stats", 2024-05-27T08:00:55.134 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.135 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.135 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.135 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.135 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.135 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.135 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.135 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.135 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.136 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.136 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.136 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.136 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.136 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.136 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.136 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.136 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.137 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.137 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.137 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.137 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.137 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.137 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.137 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.137 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.138 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.138 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.138 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.138 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.138 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.138 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.138 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.138 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.139 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.139 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.139 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.139 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.139 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.139 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.139 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.140 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.140 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.140 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.140 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.140 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.140 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.140 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.140 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.141 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.141 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.141 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.141 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.141 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.141 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.141 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.141 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.142 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.142 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.142 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.142 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.142 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.142 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.142 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.142 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.143 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.143 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.143 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.143 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.143 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.143 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.143 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.144 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.144 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.144 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.144 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.144 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.144 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.144 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.144 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.145 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.145 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.145 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.145 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.145 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.145 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "status", 2024-05-27T08:00:55.145 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.145 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.146 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.146 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.146 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.146 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.146 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.146 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.146 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.146 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.147 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.147 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.147 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.147 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.147 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.147 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.147 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.147 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.148 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.148 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.148 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.148 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.148 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.148 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.148 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.148 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.149 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.149 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.149 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.149 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.149 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.149 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.149 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.149 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.150 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.150 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.150 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.150 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.150 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.150 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.150 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.151 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.151 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.151 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.151 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.151 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.151 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.151 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.151 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.152 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.152 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.152 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.152 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.152 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.152 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.152 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.153 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.153 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.153 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.153 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.153 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.153 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.153 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.154 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.154 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.154 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.154 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.154 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.154 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.154 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.155 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.155 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.155 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.155 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.155 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.155 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.155 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.155 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.156 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.156 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.156 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.156 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.156 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.156 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.156 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "telegraf", 2024-05-27T08:00:55.157 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.157 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.157 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.157 INFO:teuthology.orchestra.run.smithi114.stdout: "address": { 2024-05-27T08:00:55.157 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "address", 2024-05-27T08:00:55.157 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.157 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.157 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.158 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T08:00:55.158 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.158 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.158 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.158 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.158 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.158 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.158 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.159 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:55.159 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:55.159 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.159 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.159 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.159 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15", 2024-05-27T08:00:55.159 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.159 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.160 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.160 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.160 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.160 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.160 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.160 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.160 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.161 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.161 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.161 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.161 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.161 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.161 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.161 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.161 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.162 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.162 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.162 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.162 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.162 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.162 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.162 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.162 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.163 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.163 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.163 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.163 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.163 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.163 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.163 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.163 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.164 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.164 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.164 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.164 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.164 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.164 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.164 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.165 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.165 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.165 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.165 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.165 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.165 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.165 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.165 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.165 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.166 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.166 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.166 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.166 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.166 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.166 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.166 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.166 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.167 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.167 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.167 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.167 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.167 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.167 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.167 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.168 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.168 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.168 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.168 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.168 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.168 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.168 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.168 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.170 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.170 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.170 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.170 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.170 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.170 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.170 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.170 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.170 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.171 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "telemetry", 2024-05-27T08:00:55.171 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.171 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.171 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.171 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_basic": { 2024-05-27T08:00:55.171 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_basic", 2024-05-27T08:00:55.172 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.172 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.172 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.172 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.172 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.172 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.172 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.172 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T08:00:55.173 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.173 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.173 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.173 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_crash": { 2024-05-27T08:00:55.173 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_crash", 2024-05-27T08:00:55.173 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.173 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.173 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.174 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.174 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.174 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.174 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.174 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T08:00:55.174 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.174 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.175 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.175 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_device": { 2024-05-27T08:00:55.175 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_device", 2024-05-27T08:00:55.175 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.175 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.175 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.175 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.176 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.176 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.176 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.176 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T08:00:55.176 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.176 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.176 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.177 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_ident": { 2024-05-27T08:00:55.177 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_ident", 2024-05-27T08:00:55.177 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.177 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.177 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.177 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.177 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.177 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.178 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.178 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T08:00:55.178 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.178 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.178 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.178 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_perf": { 2024-05-27T08:00:55.178 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_perf", 2024-05-27T08:00:55.179 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.179 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.179 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.179 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.179 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.179 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.179 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.179 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T08:00:55.180 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.180 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.180 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.180 INFO:teuthology.orchestra.run.smithi114.stdout: "contact": { 2024-05-27T08:00:55.180 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "contact", 2024-05-27T08:00:55.180 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.180 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.180 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.181 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.181 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.181 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.181 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.181 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.181 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.181 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.181 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.182 INFO:teuthology.orchestra.run.smithi114.stdout: "description": { 2024-05-27T08:00:55.182 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "description", 2024-05-27T08:00:55.182 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.182 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.182 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.182 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.182 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.183 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.183 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.183 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.183 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.183 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.183 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.183 INFO:teuthology.orchestra.run.smithi114.stdout: "device_url": { 2024-05-27T08:00:55.184 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_url", 2024-05-27T08:00:55.184 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.184 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.184 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.184 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T08:00:55.184 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.184 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.184 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.184 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.185 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.185 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.185 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.185 INFO:teuthology.orchestra.run.smithi114.stdout: "enabled": { 2024-05-27T08:00:55.185 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enabled", 2024-05-27T08:00:55.185 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.185 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.186 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.186 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.186 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.186 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.186 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.186 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.186 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.186 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.187 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.187 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:55.187 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:55.187 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.187 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.187 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.187 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "24", 2024-05-27T08:00:55.188 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "8", 2024-05-27T08:00:55.188 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.188 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.188 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.188 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.188 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.188 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.189 INFO:teuthology.orchestra.run.smithi114.stdout: "last_opt_revision": { 2024-05-27T08:00:55.189 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "last_opt_revision", 2024-05-27T08:00:55.189 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.189 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.189 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.189 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1", 2024-05-27T08:00:55.189 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.189 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.190 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.190 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.190 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.190 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.190 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.190 INFO:teuthology.orchestra.run.smithi114.stdout: "leaderboard": { 2024-05-27T08:00:55.190 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "leaderboard", 2024-05-27T08:00:55.191 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.191 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.191 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.191 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.191 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.191 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.191 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.191 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.192 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.192 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.192 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.192 INFO:teuthology.orchestra.run.smithi114.stdout: "leaderboard_description": { 2024-05-27T08:00:55.192 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "leaderboard_description", 2024-05-27T08:00:55.192 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.192 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.193 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.193 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.193 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.193 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.193 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.193 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.193 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.193 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.193 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.194 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.194 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.194 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.194 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.194 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.194 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.194 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.194 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.195 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.195 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.195 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.195 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.195 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.195 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.195 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.195 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.196 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.196 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.196 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.196 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.196 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.196 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.196 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.196 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.197 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.197 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.197 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.197 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.197 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.197 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.197 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.197 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.197 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.198 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.198 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.198 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.198 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.198 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.198 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.198 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.199 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.199 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.199 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.199 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.199 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.199 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.199 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.199 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.199 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.200 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.200 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.200 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.200 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.200 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.200 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.200 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.200 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.201 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.201 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.201 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.201 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.201 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.201 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.201 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.201 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.202 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.202 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.202 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.202 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.202 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.202 INFO:teuthology.orchestra.run.smithi114.stdout: "organization": { 2024-05-27T08:00:55.202 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "organization", 2024-05-27T08:00:55.203 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.203 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.203 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.203 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.203 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.203 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.203 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.203 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.204 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.204 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.204 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.204 INFO:teuthology.orchestra.run.smithi114.stdout: "proxy": { 2024-05-27T08:00:55.204 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "proxy", 2024-05-27T08:00:55.204 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.204 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.204 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.205 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.205 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.205 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.205 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.205 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.205 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.205 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.205 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.206 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.206 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.206 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.206 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.206 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.206 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.206 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.206 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.207 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.207 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.207 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.207 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.207 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.207 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.207 INFO:teuthology.orchestra.run.smithi114.stdout: "url": { 2024-05-27T08:00:55.208 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "url", 2024-05-27T08:00:55.208 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.208 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.208 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.208 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T08:00:55.208 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.208 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.208 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.209 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.209 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.209 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.209 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.209 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.209 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.209 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.210 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "test_orchestrator", 2024-05-27T08:00:55.210 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.210 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.210 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.210 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.210 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.210 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.210 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.211 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.211 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.211 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.211 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.211 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.211 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.211 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.211 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.211 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.212 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.212 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.212 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.212 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.212 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.212 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.212 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.213 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.213 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.213 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.213 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.213 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.213 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.213 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.213 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.214 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.214 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.214 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.214 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.214 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.214 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.214 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.214 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.215 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.215 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.215 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.215 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.215 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.215 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.215 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.215 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.216 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.216 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.216 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.216 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.216 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.216 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.216 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.216 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.217 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.217 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.217 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.217 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.217 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.217 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.217 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.218 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.218 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.218 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.218 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.218 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.218 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.218 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.219 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.219 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.219 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.219 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.219 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.219 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.219 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.219 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.219 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.220 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.220 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.220 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.220 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.220 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.220 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.220 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "volumes", 2024-05-27T08:00:55.221 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.221 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.221 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.221 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.221 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.221 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.221 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.221 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.222 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.222 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.222 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.222 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.222 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.222 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.222 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.222 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.223 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.223 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.223 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.223 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.223 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.223 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.223 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.223 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.224 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.224 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.224 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.224 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.224 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.224 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.224 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.224 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.225 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.225 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.225 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.225 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.225 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.225 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.225 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.226 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.226 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.226 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.226 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.226 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.226 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.226 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.226 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.227 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.227 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.227 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.227 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.227 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.227 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.227 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.227 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.228 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.228 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.228 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.228 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.228 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.228 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.228 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.229 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.229 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.229 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.229 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.229 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.229 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.229 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.230 INFO:teuthology.orchestra.run.smithi114.stdout: "max_concurrent_clones": { 2024-05-27T08:00:55.230 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_concurrent_clones", 2024-05-27T08:00:55.230 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.230 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.230 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.230 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "4", 2024-05-27T08:00:55.230 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.230 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.231 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.231 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T08:00:55.231 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.231 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.231 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.231 INFO:teuthology.orchestra.run.smithi114.stdout: "periodic_async_work": { 2024-05-27T08:00:55.231 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "periodic_async_work", 2024-05-27T08:00:55.232 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.232 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.232 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.232 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.232 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.232 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.232 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.232 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Periodically check for async work", 2024-05-27T08:00:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: "snapshot_clone_delay": { 2024-05-27T08:00:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snapshot_clone_delay", 2024-05-27T08:00:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T08:00:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.235 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.235 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.235 INFO:teuthology.orchestra.run.smithi114.stdout: "snapshot_clone_no_wait": { 2024-05-27T08:00:55.235 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T08:00:55.235 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.235 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.235 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.235 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T08:00:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.237 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.237 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.237 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.237 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.237 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.237 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.237 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.237 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.237 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.238 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.238 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.238 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.238 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.238 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.238 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.239 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix", 2024-05-27T08:00:55.239 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.239 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.239 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.239 INFO:teuthology.orchestra.run.smithi114.stdout: "discovery_interval": { 2024-05-27T08:00:55.239 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "discovery_interval", 2024-05-27T08:00:55.239 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:55.239 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.240 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.240 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "100", 2024-05-27T08:00:55.240 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.240 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.240 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.240 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.240 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.240 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.241 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.241 INFO:teuthology.orchestra.run.smithi114.stdout: "identifier": { 2024-05-27T08:00:55.241 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "identifier", 2024-05-27T08:00:55.241 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.241 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.241 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.241 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.242 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.242 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.242 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.242 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.242 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.242 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.242 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.243 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:55.243 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:55.243 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.243 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.243 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.243 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:55.243 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.243 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.244 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.244 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.244 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.244 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.244 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.244 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.244 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.245 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.245 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.245 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.245 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.245 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.245 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.245 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.245 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.246 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.246 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.246 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.246 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.246 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.246 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.246 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.246 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.247 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.247 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.247 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.247 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.247 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.247 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.247 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.247 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.248 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.248 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.248 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.248 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.248 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.248 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.248 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.248 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.249 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.249 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.249 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.249 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.249 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.249 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.249 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.249 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.250 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.250 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.250 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.250 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.250 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.250 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.250 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.250 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.251 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.251 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.251 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.251 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.251 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.251 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.251 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.252 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.252 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.252 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.252 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.252 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.252 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.252 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.252 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.252 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.253 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.253 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.253 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.253 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.253 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.253 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.253 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.253 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.254 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.254 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.254 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.254 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.254 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.254 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.254 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_host": { 2024-05-27T08:00:55.255 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_host", 2024-05-27T08:00:55.255 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.255 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.255 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.255 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.255 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.255 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.255 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.255 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.256 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.256 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.256 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.256 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.256 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_port": { 2024-05-27T08:00:55.256 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_port", 2024-05-27T08:00:55.256 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.256 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.256 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.257 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10051", 2024-05-27T08:00:55.257 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.257 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.257 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.257 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.257 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.257 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.257 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.258 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.258 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_sender": { 2024-05-27T08:00:55.258 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_sender", 2024-05-27T08:00:55.258 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.258 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.258 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.258 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T08:00:55.259 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.259 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.259 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.259 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.259 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.259 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.259 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.259 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.260 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.260 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.260 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-05-27T08:00:55.260 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.260 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.260 INFO:teuthology.orchestra.run.smithi114.stdout: "modules": [ 2024-05-27T08:00:55.260 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm", 2024-05-27T08:00:55.260 INFO:teuthology.orchestra.run.smithi114.stdout: "dashboard", 2024-05-27T08:00:55.261 INFO:teuthology.orchestra.run.smithi114.stdout: "iostat", 2024-05-27T08:00:55.261 INFO:teuthology.orchestra.run.smithi114.stdout: "nfs", 2024-05-27T08:00:55.261 INFO:teuthology.orchestra.run.smithi114.stdout: "restful", 2024-05-27T08:00:55.261 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_schedule" 2024-05-27T08:00:55.261 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.261 INFO:teuthology.orchestra.run.smithi114.stdout: "available_modules": [ 2024-05-27T08:00:55.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.261 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "alerts", 2024-05-27T08:00:55.261 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.262 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.262 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.262 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:55.262 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:55.262 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.262 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.262 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.262 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:55.263 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.263 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.263 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.263 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T08:00:55.263 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.263 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.263 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.263 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.264 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.264 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.264 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.264 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.264 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.264 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.264 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.264 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.265 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.265 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.265 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.265 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.265 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.265 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.265 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.265 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.266 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.266 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.266 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.266 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.266 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.266 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.266 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.266 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.266 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.267 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.267 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.267 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.267 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.267 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.267 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.267 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.267 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.268 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.268 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.268 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.268 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.268 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.268 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.268 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.268 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.269 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.269 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.269 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.269 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.269 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.269 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.269 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.269 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.270 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.270 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.270 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.270 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.270 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.270 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.270 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.270 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.271 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.271 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.271 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.271 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.271 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.271 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.271 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.271 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.272 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.272 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.272 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.272 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.272 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_destination": { 2024-05-27T08:00:55.272 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_destination", 2024-05-27T08:00:55.272 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.272 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.273 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.273 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.273 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.273 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.273 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.273 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Email address to send alerts to", 2024-05-27T08:00:55.273 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.273 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.274 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.274 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_from_name": { 2024-05-27T08:00:55.274 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_from_name", 2024-05-27T08:00:55.274 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.274 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.274 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.274 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "Ceph", 2024-05-27T08:00:55.275 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.275 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.275 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.275 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Email From: name", 2024-05-27T08:00:55.275 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.275 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.275 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.275 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.276 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_host": { 2024-05-27T08:00:55.276 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_host", 2024-05-27T08:00:55.276 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.276 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.276 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.276 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.276 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.277 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.277 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.277 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP server", 2024-05-27T08:00:55.277 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.277 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.277 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.277 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_password": { 2024-05-27T08:00:55.278 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_password", 2024-05-27T08:00:55.278 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.278 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.278 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.278 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.278 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.278 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.278 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.279 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Password to authenticate with", 2024-05-27T08:00:55.279 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.279 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.279 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.279 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.279 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_port": { 2024-05-27T08:00:55.279 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_port", 2024-05-27T08:00:55.279 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.279 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.280 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.280 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "465", 2024-05-27T08:00:55.280 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.280 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.280 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.280 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP port", 2024-05-27T08:00:55.280 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.280 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.280 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.281 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.281 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_sender": { 2024-05-27T08:00:55.281 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_sender", 2024-05-27T08:00:55.281 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.281 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.281 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.281 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.281 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.282 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.282 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.282 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP envelope sender", 2024-05-27T08:00:55.282 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.282 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.282 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.282 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.282 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_ssl": { 2024-05-27T08:00:55.283 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_ssl", 2024-05-27T08:00:55.283 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.283 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.283 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.283 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.283 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.283 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.283 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.283 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T08:00:55.284 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.284 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.284 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.284 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.284 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_user": { 2024-05-27T08:00:55.284 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_user", 2024-05-27T08:00:55.284 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.284 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.285 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.285 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.285 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.285 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.285 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.285 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "User to authenticate as", 2024-05-27T08:00:55.285 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.285 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.286 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.286 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.286 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.286 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.286 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.286 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.286 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.286 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.287 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.287 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.287 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.287 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.287 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.287 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.287 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.287 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.288 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.288 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.288 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.288 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "balancer", 2024-05-27T08:00:55.288 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.288 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.288 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.288 INFO:teuthology.orchestra.run.smithi114.stdout: "active": { 2024-05-27T08:00:55.289 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "active", 2024-05-27T08:00:55.289 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.289 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.289 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.289 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.289 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.289 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.289 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.290 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T08:00:55.290 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.290 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.290 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.290 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.290 INFO:teuthology.orchestra.run.smithi114.stdout: "begin_time": { 2024-05-27T08:00:55.290 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "begin_time", 2024-05-27T08:00:55.290 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.291 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.291 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.291 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0000", 2024-05-27T08:00:55.291 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.291 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.291 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.291 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T08:00:55.291 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T08:00:55.292 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.292 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.292 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.292 INFO:teuthology.orchestra.run.smithi114.stdout: "begin_weekday": { 2024-05-27T08:00:55.292 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "begin_weekday", 2024-05-27T08:00:55.292 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:55.292 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.292 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.293 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.293 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0", 2024-05-27T08:00:55.293 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "6", 2024-05-27T08:00:55.293 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.293 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T08:00:55.293 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T08:00:55.293 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.293 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.294 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.294 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_max_iterations": { 2024-05-27T08:00:55.294 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_max_iterations", 2024-05-27T08:00:55.294 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:55.294 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.294 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.294 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "25", 2024-05-27T08:00:55.295 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-05-27T08:00:55.295 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "250", 2024-05-27T08:00:55.295 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.295 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T08:00:55.295 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.295 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.295 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.296 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_metrics": { 2024-05-27T08:00:55.296 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_metrics", 2024-05-27T08:00:55.296 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.296 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.296 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.296 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T08:00:55.296 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.297 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.297 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.297 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T08:00:55.297 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T08:00:55.297 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.297 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.297 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_step": { 2024-05-27T08:00:55.298 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_step", 2024-05-27T08:00:55.298 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:55.298 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.298 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.298 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0.5", 2024-05-27T08:00:55.298 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0.001", 2024-05-27T08:00:55.298 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "0.999", 2024-05-27T08:00:55.298 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.299 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "aggressiveness of optimization", 2024-05-27T08:00:55.299 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T08:00:55.299 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.299 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.299 INFO:teuthology.orchestra.run.smithi114.stdout: "end_time": { 2024-05-27T08:00:55.299 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "end_time", 2024-05-27T08:00:55.300 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.300 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.300 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.300 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "2359", 2024-05-27T08:00:55.300 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.300 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.300 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.300 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T08:00:55.300 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T08:00:55.301 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.301 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.301 INFO:teuthology.orchestra.run.smithi114.stdout: "end_weekday": { 2024-05-27T08:00:55.301 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "end_weekday", 2024-05-27T08:00:55.301 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:55.301 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.302 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.302 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.302 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0", 2024-05-27T08:00:55.302 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "6", 2024-05-27T08:00:55.302 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.302 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T08:00:55.302 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T08:00:55.303 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.303 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.303 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.303 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.303 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.303 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.303 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.304 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.304 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.304 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.304 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.304 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.304 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.304 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.304 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.305 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.305 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.305 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.305 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.305 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.305 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.305 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.306 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.306 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.306 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.306 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.306 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.306 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.306 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.307 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.307 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.307 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.307 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.307 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.307 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.307 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.308 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.308 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.308 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.308 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.308 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.308 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.308 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.308 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.309 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.309 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.309 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.309 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.309 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.309 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.309 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.309 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.310 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.310 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.310 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.310 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.310 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.310 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.310 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.311 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.311 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.311 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.311 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.311 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.311 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.311 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.311 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.312 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.312 INFO:teuthology.orchestra.run.smithi114.stdout: "min_score": { 2024-05-27T08:00:55.312 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "min_score", 2024-05-27T08:00:55.312 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:55.312 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.312 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.312 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.312 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.313 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.313 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.313 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T08:00:55.313 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.313 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.313 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.314 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": { 2024-05-27T08:00:55.314 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mode", 2024-05-27T08:00:55.314 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.314 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.314 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.314 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "upmap", 2024-05-27T08:00:55.314 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.315 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.315 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.315 INFO:teuthology.orchestra.run.smithi114.stdout: "crush-compat", 2024-05-27T08:00:55.315 INFO:teuthology.orchestra.run.smithi114.stdout: "none", 2024-05-27T08:00:55.315 INFO:teuthology.orchestra.run.smithi114.stdout: "read", 2024-05-27T08:00:55.315 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap", 2024-05-27T08:00:55.315 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap-read" 2024-05-27T08:00:55.315 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.316 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Balancer mode", 2024-05-27T08:00:55.316 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.316 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.316 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.316 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.316 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_ids": { 2024-05-27T08:00:55.316 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pool_ids", 2024-05-27T08:00:55.316 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.317 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.317 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.317 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.317 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.317 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.317 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.317 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T08:00:55.317 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.317 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.318 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.318 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:55.318 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:55.318 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.318 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.318 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.318 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:55.319 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.319 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.319 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.319 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T08:00:55.319 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.319 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.319 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.320 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.320 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.320 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.320 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.320 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.320 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.320 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.320 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.321 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.321 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.321 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.321 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.321 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.321 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.321 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap_max_deviation": { 2024-05-27T08:00:55.321 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "upmap_max_deviation", 2024-05-27T08:00:55.322 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.322 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.322 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.322 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-05-27T08:00:55.322 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-05-27T08:00:55.322 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.322 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.322 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T08:00:55.323 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T08:00:55.323 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.323 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.323 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap_max_optimizations": { 2024-05-27T08:00:55.323 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "upmap_max_optimizations", 2024-05-27T08:00:55.323 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:55.323 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.324 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.324 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:55.324 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.324 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.324 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.324 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T08:00:55.324 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.325 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.325 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.325 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.325 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.325 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.325 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.325 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephadm", 2024-05-27T08:00:55.325 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.326 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.326 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.326 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_down_multiplier": { 2024-05-27T08:00:55.326 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_down_multiplier", 2024-05-27T08:00:55.326 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:55.326 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.326 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.326 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3.0", 2024-05-27T08:00:55.327 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.327 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.327 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.327 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T08:00:55.327 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.327 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.327 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.328 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_refresh_rate": { 2024-05-27T08:00:55.328 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_refresh_rate", 2024-05-27T08:00:55.328 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.328 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.328 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.328 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "20", 2024-05-27T08:00:55.328 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.328 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.329 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.329 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T08:00:55.329 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.329 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.329 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.329 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.329 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_starting_port": { 2024-05-27T08:00:55.329 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_starting_port", 2024-05-27T08:00:55.329 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.330 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.330 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.330 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "4721", 2024-05-27T08:00:55.330 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.330 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.330 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.330 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T08:00:55.331 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.331 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.331 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.331 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_ptrace": { 2024-05-27T08:00:55.331 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_ptrace", 2024-05-27T08:00:55.331 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.331 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.331 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.332 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.332 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.332 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.332 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.332 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T08:00:55.332 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T08:00:55.332 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.332 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.333 INFO:teuthology.orchestra.run.smithi114.stdout: "autotune_interval": { 2024-05-27T08:00:55.333 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "autotune_interval", 2024-05-27T08:00:55.333 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.333 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.333 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.333 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:55.333 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.334 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.334 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.334 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T08:00:55.334 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.334 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.334 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.334 INFO:teuthology.orchestra.run.smithi114.stdout: "autotune_memory_target_ratio": { 2024-05-27T08:00:55.335 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T08:00:55.335 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:55.335 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.335 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.335 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0.7", 2024-05-27T08:00:55.335 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.335 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.335 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.335 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T08:00:55.336 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.336 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.336 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.336 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm_log_destination": { 2024-05-27T08:00:55.336 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephadm_log_destination", 2024-05-27T08:00:55.336 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.336 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.337 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.337 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.337 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.337 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.337 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.337 INFO:teuthology.orchestra.run.smithi114.stdout: "file", 2024-05-27T08:00:55.337 INFO:teuthology.orchestra.run.smithi114.stdout: "file,syslog", 2024-05-27T08:00:55.337 INFO:teuthology.orchestra.run.smithi114.stdout: "syslog" 2024-05-27T08:00:55.337 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.338 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T08:00:55.338 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.338 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.338 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.338 INFO:teuthology.orchestra.run.smithi114.stdout: "cgroups_split": { 2024-05-27T08:00:55.338 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cgroups_split", 2024-05-27T08:00:55.338 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.339 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.339 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.339 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.339 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.339 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.339 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.339 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T08:00:55.339 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.339 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.340 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.340 INFO:teuthology.orchestra.run.smithi114.stdout: "config_checks_enabled": { 2024-05-27T08:00:55.340 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_checks_enabled", 2024-05-27T08:00:55.340 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.340 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.340 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.340 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.340 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.340 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.341 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.341 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T08:00:55.341 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.341 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.341 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.341 INFO:teuthology.orchestra.run.smithi114.stdout: "config_dashboard": { 2024-05-27T08:00:55.341 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_dashboard", 2024-05-27T08:00:55.342 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.342 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.342 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.342 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.342 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.342 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.342 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.342 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T08:00:55.343 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.343 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.343 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.343 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_alertmanager": { 2024-05-27T08:00:55.343 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_alertmanager", 2024-05-27T08:00:55.343 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.343 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.343 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.344 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T08:00:55.344 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.344 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.344 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.344 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-05-27T08:00:55.344 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.344 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.344 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.345 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_base": { 2024-05-27T08:00:55.345 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_base", 2024-05-27T08:00:55.345 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.345 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.345 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.345 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T08:00:55.345 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.345 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.346 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.346 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Container image name, without the tag", 2024-05-27T08:00:55.346 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.346 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.346 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.346 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_elasticsearch": { 2024-05-27T08:00:55.346 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_elasticsearch", 2024-05-27T08:00:55.347 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.347 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.347 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.347 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T08:00:55.347 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.347 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.347 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.347 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "elasticsearch container image", 2024-05-27T08:00:55.348 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.348 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.348 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.348 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_grafana": { 2024-05-27T08:00:55.348 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_grafana", 2024-05-27T08:00:55.348 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.348 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.348 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.349 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T08:00:55.349 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.349 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.349 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.349 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-05-27T08:00:55.349 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.349 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.349 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.350 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_haproxy": { 2024-05-27T08:00:55.350 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_haproxy", 2024-05-27T08:00:55.350 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.350 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.350 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.350 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T08:00:55.350 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.350 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.351 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.351 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "HAproxy container image", 2024-05-27T08:00:55.351 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.351 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.351 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.351 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_agent": { 2024-05-27T08:00:55.351 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_agent", 2024-05-27T08:00:55.352 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.352 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.352 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.352 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T08:00:55.352 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.352 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.352 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.352 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger agent container image", 2024-05-27T08:00:55.352 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.353 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.353 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.353 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_collector": { 2024-05-27T08:00:55.353 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_collector", 2024-05-27T08:00:55.353 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.353 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.353 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.354 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T08:00:55.354 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.354 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.354 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.354 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger collector container image", 2024-05-27T08:00:55.354 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.354 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.355 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.355 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_query": { 2024-05-27T08:00:55.355 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_query", 2024-05-27T08:00:55.355 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.355 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.355 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.355 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T08:00:55.355 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.356 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.356 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.356 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger query container image", 2024-05-27T08:00:55.356 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.356 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.356 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.356 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_keepalived": { 2024-05-27T08:00:55.357 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_keepalived", 2024-05-27T08:00:55.357 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.357 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.357 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.357 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T08:00:55.357 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.357 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.357 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.358 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Keepalived container image", 2024-05-27T08:00:55.358 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.358 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.358 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.358 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_loki": { 2024-05-27T08:00:55.358 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_loki", 2024-05-27T08:00:55.358 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.359 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.359 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.359 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T08:00:55.359 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.359 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.359 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.359 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Loki container image", 2024-05-27T08:00:55.359 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.360 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.360 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.360 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_node_exporter": { 2024-05-27T08:00:55.360 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_node_exporter", 2024-05-27T08:00:55.360 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.360 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.360 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.361 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T08:00:55.361 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.361 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.361 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.361 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-05-27T08:00:55.361 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.361 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.361 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.362 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_nvmeof": { 2024-05-27T08:00:55.362 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_nvmeof", 2024-05-27T08:00:55.362 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.362 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.362 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.362 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T08:00:55.362 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.363 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.363 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.363 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Nvme-of container image", 2024-05-27T08:00:55.363 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.363 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.363 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.363 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_prometheus": { 2024-05-27T08:00:55.363 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_prometheus", 2024-05-27T08:00:55.364 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.364 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.364 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.364 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T08:00:55.364 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.364 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.364 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.364 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-05-27T08:00:55.365 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.365 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.365 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.365 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_promtail": { 2024-05-27T08:00:55.365 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_promtail", 2024-05-27T08:00:55.365 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.365 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.365 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.366 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T08:00:55.366 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.366 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.366 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.366 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Promtail container image", 2024-05-27T08:00:55.366 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.366 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.366 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.367 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_samba": { 2024-05-27T08:00:55.367 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_samba", 2024-05-27T08:00:55.367 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.367 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.367 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.367 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T08:00:55.367 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.367 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.368 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.368 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Samba/SMB container image", 2024-05-27T08:00:55.368 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.368 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.368 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.368 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_snmp_gateway": { 2024-05-27T08:00:55.368 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_snmp_gateway", 2024-05-27T08:00:55.369 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.369 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.369 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.369 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T08:00:55.369 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.369 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.369 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.369 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SNMP Gateway container image", 2024-05-27T08:00:55.370 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.370 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.370 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.370 INFO:teuthology.orchestra.run.smithi114.stdout: "container_init": { 2024-05-27T08:00:55.370 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_init", 2024-05-27T08:00:55.370 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.370 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.371 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.371 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.371 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.371 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.371 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.371 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T08:00:55.371 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.371 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.372 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.372 INFO:teuthology.orchestra.run.smithi114.stdout: "daemon_cache_timeout": { 2024-05-27T08:00:55.372 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "daemon_cache_timeout", 2024-05-27T08:00:55.372 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.372 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.372 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.372 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:55.373 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.373 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.373 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.373 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T08:00:55.373 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.373 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.373 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.373 INFO:teuthology.orchestra.run.smithi114.stdout: "default_cephadm_command_timeout": { 2024-05-27T08:00:55.374 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T08:00:55.374 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.374 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.374 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.374 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "900", 2024-05-27T08:00:55.374 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.374 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.374 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.375 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T08:00:55.375 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.375 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.375 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.375 INFO:teuthology.orchestra.run.smithi114.stdout: "default_registry": { 2024-05-27T08:00:55.375 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "default_registry", 2024-05-27T08:00:55.375 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.376 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.376 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.376 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io", 2024-05-27T08:00:55.376 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.376 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.376 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.376 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T08:00:55.376 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.376 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.377 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.377 INFO:teuthology.orchestra.run.smithi114.stdout: "device_cache_timeout": { 2024-05-27T08:00:55.377 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_cache_timeout", 2024-05-27T08:00:55.377 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.377 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.377 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.377 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1800", 2024-05-27T08:00:55.378 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.378 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.378 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.378 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache device inventory", 2024-05-27T08:00:55.378 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.378 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.378 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.379 INFO:teuthology.orchestra.run.smithi114.stdout: "device_enhanced_scan": { 2024-05-27T08:00:55.379 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_enhanced_scan", 2024-05-27T08:00:55.379 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.379 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.379 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.379 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.379 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.379 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.380 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.380 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T08:00:55.380 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.380 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.380 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.380 INFO:teuthology.orchestra.run.smithi114.stdout: "facts_cache_timeout": { 2024-05-27T08:00:55.380 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "facts_cache_timeout", 2024-05-27T08:00:55.380 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.381 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.381 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.381 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:55.381 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.381 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.381 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.381 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache host facts data", 2024-05-27T08:00:55.381 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.382 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.382 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.382 INFO:teuthology.orchestra.run.smithi114.stdout: "grafana_dashboards_path": { 2024-05-27T08:00:55.382 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "grafana_dashboards_path", 2024-05-27T08:00:55.382 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.382 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.382 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.383 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T08:00:55.383 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.383 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.383 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.383 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T08:00:55.383 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.383 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.383 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.384 INFO:teuthology.orchestra.run.smithi114.stdout: "host_check_interval": { 2024-05-27T08:00:55.384 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "host_check_interval", 2024-05-27T08:00:55.384 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.384 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.384 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.384 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:55.384 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.384 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.384 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.385 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to perform a host check", 2024-05-27T08:00:55.385 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.385 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.385 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.385 INFO:teuthology.orchestra.run.smithi114.stdout: "hw_monitoring": { 2024-05-27T08:00:55.385 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "hw_monitoring", 2024-05-27T08:00:55.385 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.385 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.386 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.386 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.386 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.386 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.386 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.386 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T08:00:55.386 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.386 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.386 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.387 INFO:teuthology.orchestra.run.smithi114.stdout: "inventory_list_all": { 2024-05-27T08:00:55.387 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "inventory_list_all", 2024-05-27T08:00:55.387 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.387 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.387 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.387 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.387 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.387 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.388 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.388 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T08:00:55.388 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.388 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.388 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.388 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.388 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.388 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.389 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.389 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.389 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.389 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.389 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.389 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.389 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.389 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.390 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.390 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.390 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.390 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.390 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.390 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.390 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.390 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.391 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.391 INFO:teuthology.orchestra.run.smithi114.stdout: "log_refresh_metadata": { 2024-05-27T08:00:55.391 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_refresh_metadata", 2024-05-27T08:00:55.391 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.391 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.391 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.391 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.391 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.392 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.392 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.392 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T08:00:55.392 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.392 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.392 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.392 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.393 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.393 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.393 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.393 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.393 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.393 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.393 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.393 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.393 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T08:00:55.394 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.394 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.394 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.394 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.394 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.394 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.394 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.395 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.395 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.395 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.395 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.395 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.395 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.395 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.395 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.395 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.395 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.396 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.396 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.396 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.396 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.396 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.396 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.396 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.396 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.397 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.397 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.397 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.397 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.397 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.397 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.397 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.397 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.398 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.398 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.398 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.398 INFO:teuthology.orchestra.run.smithi114.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T08:00:55.398 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T08:00:55.398 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.398 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.399 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.399 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.399 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.399 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.399 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.399 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T08:00:55.399 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.399 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.400 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.400 INFO:teuthology.orchestra.run.smithi114.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T08:00:55.400 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T08:00:55.400 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.400 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.400 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.400 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "*", 2024-05-27T08:00:55.401 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.401 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.401 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.401 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T08:00:55.401 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.401 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.401 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.402 INFO:teuthology.orchestra.run.smithi114.stdout: "max_count_per_host": { 2024-05-27T08:00:55.402 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_count_per_host", 2024-05-27T08:00:55.402 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.402 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.402 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.402 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:55.402 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.402 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.402 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.403 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "max number of daemons per service per host", 2024-05-27T08:00:55.403 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.403 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.403 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.403 INFO:teuthology.orchestra.run.smithi114.stdout: "max_osd_draining_count": { 2024-05-27T08:00:55.404 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_osd_draining_count", 2024-05-27T08:00:55.404 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.404 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.404 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.404 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:55.404 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.404 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.404 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.405 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T08:00:55.405 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.405 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.405 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.405 INFO:teuthology.orchestra.run.smithi114.stdout: "migration_current": { 2024-05-27T08:00:55.405 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "migration_current", 2024-05-27T08:00:55.405 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.405 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.406 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.406 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.406 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.406 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.406 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.406 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "internal - do not modify", 2024-05-27T08:00:55.406 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.407 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.407 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.407 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": { 2024-05-27T08:00:55.407 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mode", 2024-05-27T08:00:55.407 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.407 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.407 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.407 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "root", 2024-05-27T08:00:55.408 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.408 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.408 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.408 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm-package", 2024-05-27T08:00:55.408 INFO:teuthology.orchestra.run.smithi114.stdout: "root" 2024-05-27T08:00:55.408 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.408 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T08:00:55.408 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.409 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.409 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.409 INFO:teuthology.orchestra.run.smithi114.stdout: "oob_default_addr": { 2024-05-27T08:00:55.409 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "oob_default_addr", 2024-05-27T08:00:55.409 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.409 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.409 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.409 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "169.254.1.1", 2024-05-27T08:00:55.410 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.410 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.410 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.410 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T08:00:55.410 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.410 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.410 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.411 INFO:teuthology.orchestra.run.smithi114.stdout: "prometheus_alerts_path": { 2024-05-27T08:00:55.411 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prometheus_alerts_path", 2024-05-27T08:00:55.411 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.411 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.411 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.411 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T08:00:55.411 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.411 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.412 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.412 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T08:00:55.412 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.412 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.412 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.412 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_insecure": { 2024-05-27T08:00:55.412 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_insecure", 2024-05-27T08:00:55.412 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.413 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.413 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.413 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.413 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.413 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.413 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.413 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T08:00:55.414 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.414 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.414 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.414 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_password": { 2024-05-27T08:00:55.414 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_password", 2024-05-27T08:00:55.414 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.414 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.415 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.415 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.415 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.415 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.415 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.415 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T08:00:55.415 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.415 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.416 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.416 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_url": { 2024-05-27T08:00:55.416 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_url", 2024-05-27T08:00:55.416 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.416 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.416 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.416 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.416 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.417 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.417 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.417 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T08:00:55.417 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.417 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.417 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.417 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_username": { 2024-05-27T08:00:55.418 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_username", 2024-05-27T08:00:55.418 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.418 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.418 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.418 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.418 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.418 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.418 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.419 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T08:00:55.419 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.419 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.419 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.419 INFO:teuthology.orchestra.run.smithi114.stdout: "secure_monitoring_stack": { 2024-05-27T08:00:55.419 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "secure_monitoring_stack", 2024-05-27T08:00:55.419 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.420 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.420 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.420 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.420 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.420 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.420 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.420 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T08:00:55.420 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.421 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.421 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.421 INFO:teuthology.orchestra.run.smithi114.stdout: "service_discovery_port": { 2024-05-27T08:00:55.421 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "service_discovery_port", 2024-05-27T08:00:55.421 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.421 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.421 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.422 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8765", 2024-05-27T08:00:55.422 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.422 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.422 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.422 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "cephadm service discovery port", 2024-05-27T08:00:55.422 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.422 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.422 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.423 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.423 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.423 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.423 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.423 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.423 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.423 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.423 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.424 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.424 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.424 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.424 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.424 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.424 INFO:teuthology.orchestra.run.smithi114.stdout: "ssh_config_file": { 2024-05-27T08:00:55.424 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssh_config_file", 2024-05-27T08:00:55.425 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.425 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.425 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.425 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.425 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.425 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.425 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.425 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T08:00:55.425 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.426 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.426 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.426 INFO:teuthology.orchestra.run.smithi114.stdout: "use_agent": { 2024-05-27T08:00:55.426 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "use_agent", 2024-05-27T08:00:55.426 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.426 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.426 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.427 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.427 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.427 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.427 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.427 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T08:00:55.427 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.427 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.427 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.428 INFO:teuthology.orchestra.run.smithi114.stdout: "use_repo_digest": { 2024-05-27T08:00:55.428 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "use_repo_digest", 2024-05-27T08:00:55.428 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.428 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.428 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.428 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.428 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.428 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.429 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.429 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T08:00:55.429 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.429 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.429 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.429 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_failed_host_check": { 2024-05-27T08:00:55.429 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_failed_host_check", 2024-05-27T08:00:55.430 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.430 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.430 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.430 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.430 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.430 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.430 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.431 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T08:00:55.431 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.431 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.431 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.431 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_stray_daemons": { 2024-05-27T08:00:55.431 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_stray_daemons", 2024-05-27T08:00:55.431 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.432 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.432 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.432 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.432 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.432 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.432 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.432 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T08:00:55.432 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.433 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.433 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.433 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_stray_hosts": { 2024-05-27T08:00:55.433 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_stray_hosts", 2024-05-27T08:00:55.433 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.433 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.433 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.433 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.434 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.434 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.434 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.434 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T08:00:55.434 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.434 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.434 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.434 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.435 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.435 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crash", 2024-05-27T08:00:55.435 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.435 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.435 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.435 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.436 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.436 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.436 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.436 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.436 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.436 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.436 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.436 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.436 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.437 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.437 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.437 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.437 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.437 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.437 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.437 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.437 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.438 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.438 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.438 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.438 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.438 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.438 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.438 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.439 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.439 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.439 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.439 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.439 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.439 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.439 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.439 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.440 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.440 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.440 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.440 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.440 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.440 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.440 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.440 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.441 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.441 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.441 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.441 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.441 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.441 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.441 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.441 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.441 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.442 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.442 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.442 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.442 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.442 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.442 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.442 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.443 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.443 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.443 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.443 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.443 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.443 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.443 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.443 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.444 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.444 INFO:teuthology.orchestra.run.smithi114.stdout: "retain_interval": { 2024-05-27T08:00:55.444 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "retain_interval", 2024-05-27T08:00:55.444 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.444 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.444 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.444 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "31536000", 2024-05-27T08:00:55.444 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.445 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.445 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.445 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T08:00:55.445 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.445 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.445 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.445 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.446 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.446 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.446 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.446 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.446 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.446 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.446 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.446 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.447 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.447 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.447 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.447 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.447 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_recent_interval": { 2024-05-27T08:00:55.447 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_recent_interval", 2024-05-27T08:00:55.447 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.447 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.447 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.448 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1209600", 2024-05-27T08:00:55.448 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.448 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.448 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.448 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T08:00:55.448 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.448 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.448 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.448 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.449 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.449 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "dashboard", 2024-05-27T08:00:55.449 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.449 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.449 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.449 INFO:teuthology.orchestra.run.smithi114.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T08:00:55.450 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T08:00:55.450 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.450 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.450 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.450 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:55.450 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.450 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.450 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.451 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.451 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.451 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.451 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.451 INFO:teuthology.orchestra.run.smithi114.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T08:00:55.451 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T08:00:55.451 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.452 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.452 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.452 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.452 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.452 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.452 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.452 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.452 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.452 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.453 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.453 INFO:teuthology.orchestra.run.smithi114.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T08:00:55.453 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T08:00:55.453 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.453 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.453 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.453 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.453 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.454 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.454 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.454 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.454 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.454 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.454 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.454 INFO:teuthology.orchestra.run.smithi114.stdout: "AUDIT_API_ENABLED": { 2024-05-27T08:00:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T08:00:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T08:00:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T08:00:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T08:00:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T08:00:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.460 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T08:00:55.460 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T08:00:55.460 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.460 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.460 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.460 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.460 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.460 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.461 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.461 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.461 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.461 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.461 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.461 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T08:00:55.461 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T08:00:55.461 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.462 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.462 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.462 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.462 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.462 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.462 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.462 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.463 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.463 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.463 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.463 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T08:00:55.463 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T08:00:55.463 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.463 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.464 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.464 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.464 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.464 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.464 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.464 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.464 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.464 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.465 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.465 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T08:00:55.465 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T08:00:55.465 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.465 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.465 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.465 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.466 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.466 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.466 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.466 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.466 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.466 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.466 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.467 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T08:00:55.467 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T08:00:55.467 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.467 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.467 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.467 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.467 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.468 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.468 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.468 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.468 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.468 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.468 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.468 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.468 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T08:00:55.469 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T08:00:55.469 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.469 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.469 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.469 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.469 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.469 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.469 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.470 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.470 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.470 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.470 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.470 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T08:00:55.470 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T08:00:55.470 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.471 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.471 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.471 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.471 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.471 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.471 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.471 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.471 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.472 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.472 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.472 INFO:teuthology.orchestra.run.smithi114.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T08:00:55.472 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T08:00:55.472 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.472 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.472 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.473 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.473 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.473 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.473 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.473 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.473 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.473 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.473 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.474 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T08:00:55.474 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T08:00:55.474 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.474 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.474 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.474 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-05-27T08:00:55.475 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.475 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.475 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.475 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.475 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.475 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.475 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.476 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T08:00:55.476 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T08:00:55.476 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.476 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.476 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.476 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.476 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.476 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.477 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.477 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.477 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.477 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.477 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.477 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_URL": { 2024-05-27T08:00:55.477 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_URL", 2024-05-27T08:00:55.477 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.478 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.478 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.478 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.478 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.478 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.478 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.478 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.479 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.479 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.479 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.479 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T08:00:55.479 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T08:00:55.479 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.479 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.480 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.480 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-05-27T08:00:55.480 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.480 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.480 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.480 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.480 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.480 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.480 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.481 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.481 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T08:00:55.481 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T08:00:55.481 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.481 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.481 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.481 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.481 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.482 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.482 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.482 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.482 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.482 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.482 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.482 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T08:00:55.483 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T08:00:55.483 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.483 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.483 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.483 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.483 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.483 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.483 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.483 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.484 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.484 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.484 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.484 INFO:teuthology.orchestra.run.smithi114.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T08:00:55.484 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T08:00:55.484 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.484 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.485 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.485 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.485 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.485 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.485 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.485 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.485 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.485 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.486 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.486 INFO:teuthology.orchestra.run.smithi114.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T08:00:55.486 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T08:00:55.486 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.486 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.486 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.487 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.487 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.487 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.487 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.487 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.487 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.487 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.487 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.488 INFO:teuthology.orchestra.run.smithi114.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T08:00:55.488 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T08:00:55.488 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.488 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.488 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.488 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "[]", 2024-05-27T08:00:55.488 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.488 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.489 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.489 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.489 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.489 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.489 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.489 INFO:teuthology.orchestra.run.smithi114.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T08:00:55.489 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T08:00:55.489 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.490 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.490 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.490 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "{}", 2024-05-27T08:00:55.490 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.490 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.490 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.490 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.491 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.491 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.491 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.491 INFO:teuthology.orchestra.run.smithi114.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T08:00:55.491 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T08:00:55.491 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.491 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.492 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.492 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.492 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.492 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.492 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.492 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.492 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.492 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.492 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.493 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.493 INFO:teuthology.orchestra.run.smithi114.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T08:00:55.493 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T08:00:55.493 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.493 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.493 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.493 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.494 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.494 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.494 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.494 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.494 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.494 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.494 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.494 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.495 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T08:00:55.495 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T08:00:55.495 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.495 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.495 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.495 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.495 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.495 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.496 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.496 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.496 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.496 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.496 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.496 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T08:00:55.496 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T08:00:55.496 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.497 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.497 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.497 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.497 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.497 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.497 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.497 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.497 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.498 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.498 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.498 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T08:00:55.498 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T08:00:55.498 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.498 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.498 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.499 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.499 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.499 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.499 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.499 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.499 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.499 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.499 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.500 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T08:00:55.500 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T08:00:55.500 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.500 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.500 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.500 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.500 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.500 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.501 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.501 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.501 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.501 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.501 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.501 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T08:00:55.502 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T08:00:55.502 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.502 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.502 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.502 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.502 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.502 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.502 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.503 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.503 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.503 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.503 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.503 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T08:00:55.503 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T08:00:55.503 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.503 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.504 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.504 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.504 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.504 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.504 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.504 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.504 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.504 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.505 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.505 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T08:00:55.505 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T08:00:55.505 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.505 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.505 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.505 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.506 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.506 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.506 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.506 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.506 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.506 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.506 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.507 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T08:00:55.507 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T08:00:55.507 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.507 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.507 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.507 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.507 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.507 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.507 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.508 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.508 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.508 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.508 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.508 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T08:00:55.508 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T08:00:55.508 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.509 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.509 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.509 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T08:00:55.509 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.509 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.509 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.509 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.509 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.509 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.510 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.510 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.510 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T08:00:55.510 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T08:00:55.510 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.510 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.510 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.510 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:55.511 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.511 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.511 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.511 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.511 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.511 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.511 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.512 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.512 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T08:00:55.512 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T08:00:55.512 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.512 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.512 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.512 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8", 2024-05-27T08:00:55.512 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.512 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.513 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.513 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.513 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.513 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.513 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.513 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.513 INFO:teuthology.orchestra.run.smithi114.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T08:00:55.513 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T08:00:55.514 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.514 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.514 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.514 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "45", 2024-05-27T08:00:55.514 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.514 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.514 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.514 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.515 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.515 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.515 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.515 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.515 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T08:00:55.515 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T08:00:55.515 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.515 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.516 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.516 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.516 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.516 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.516 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.516 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.516 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.516 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.517 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.517 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.517 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T08:00:55.517 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T08:00:55.517 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.517 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.517 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.517 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-05-27T08:00:55.518 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.518 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.518 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.518 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.518 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.518 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.518 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.518 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.519 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T08:00:55.519 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T08:00:55.519 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.519 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.519 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.519 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.519 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.520 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.520 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.520 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.520 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.520 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.520 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.520 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.520 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T08:00:55.520 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T08:00:55.521 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.521 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.521 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.521 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.521 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.521 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.521 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.521 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.522 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.522 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.522 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.522 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.522 INFO:teuthology.orchestra.run.smithi114.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T08:00:55.522 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T08:00:55.522 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.523 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.523 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.523 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.523 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.523 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.523 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.523 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.523 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.524 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.524 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.524 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.524 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T08:00:55.524 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T08:00:55.524 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.524 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.524 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.525 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.525 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.525 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.525 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.525 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.525 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.525 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.525 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.526 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.526 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T08:00:55.526 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T08:00:55.526 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.526 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.526 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.526 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:55.526 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.527 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.527 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.527 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.527 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.527 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.527 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.527 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T08:00:55.528 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T08:00:55.528 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.528 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.528 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.528 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-05-27T08:00:55.528 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.528 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.528 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.529 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.529 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.529 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.529 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.529 INFO:teuthology.orchestra.run.smithi114.stdout: "cross_origin_url": { 2024-05-27T08:00:55.529 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cross_origin_url", 2024-05-27T08:00:55.529 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.530 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.530 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.530 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.530 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.530 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.530 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.530 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.530 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.531 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.531 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.531 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.531 INFO:teuthology.orchestra.run.smithi114.stdout: "crt_file": { 2024-05-27T08:00:55.531 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crt_file", 2024-05-27T08:00:55.531 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.531 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.531 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.532 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.532 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.532 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.532 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.532 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.532 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.532 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.532 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.533 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.533 INFO:teuthology.orchestra.run.smithi114.stdout: "debug": { 2024-05-27T08:00:55.533 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "debug", 2024-05-27T08:00:55.533 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.533 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.533 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.533 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.533 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.534 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.534 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.534 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable/disable debug options", 2024-05-27T08:00:55.534 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.534 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.534 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.534 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.534 INFO:teuthology.orchestra.run.smithi114.stdout: "jwt_token_ttl": { 2024-05-27T08:00:55.535 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "jwt_token_ttl", 2024-05-27T08:00:55.535 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.535 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.535 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.535 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "28800", 2024-05-27T08:00:55.535 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.535 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.535 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.536 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.536 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.536 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.536 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.536 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.536 INFO:teuthology.orchestra.run.smithi114.stdout: "key_file": { 2024-05-27T08:00:55.536 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "key_file", 2024-05-27T08:00:55.536 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.537 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.537 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.537 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.537 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.537 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.537 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.537 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.537 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.538 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.538 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.538 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.538 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.538 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.538 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.538 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.538 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.539 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.539 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.539 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.539 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.539 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.539 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.539 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.539 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.540 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.540 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.540 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.540 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.540 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.540 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.540 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.540 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.541 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.541 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.541 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.541 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.541 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.541 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.541 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.541 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.542 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.542 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.542 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.542 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.542 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.542 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.542 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.542 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.543 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.543 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.543 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.543 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.543 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.543 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.543 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.543 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.544 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.544 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.544 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.544 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.544 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.544 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.544 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.544 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.544 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.545 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.545 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.545 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.545 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.545 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.545 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.545 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.545 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.546 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.546 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.546 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.546 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.546 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.546 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.546 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.546 INFO:teuthology.orchestra.run.smithi114.stdout: "motd": { 2024-05-27T08:00:55.547 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "motd", 2024-05-27T08:00:55.547 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.547 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.547 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.547 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.547 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.547 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.547 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.547 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "The message of the day", 2024-05-27T08:00:55.548 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.548 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.548 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.548 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.548 INFO:teuthology.orchestra.run.smithi114.stdout: "redirect_resolve_ip_addr": { 2024-05-27T08:00:55.548 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T08:00:55.548 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.548 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.549 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.549 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.549 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.549 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.549 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.549 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.549 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.549 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.550 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.550 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.550 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-05-27T08:00:55.550 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-05-27T08:00:55.550 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.550 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.550 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.550 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "::", 2024-05-27T08:00:55.551 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.551 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.551 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.551 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.551 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.551 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.551 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.551 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.551 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-05-27T08:00:55.552 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-05-27T08:00:55.552 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.552 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.552 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.552 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8080", 2024-05-27T08:00:55.552 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.552 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.552 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.553 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.553 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.553 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.553 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.553 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.553 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.553 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.553 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.553 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.554 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.554 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.554 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.554 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.554 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.554 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.554 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.554 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.555 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.555 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.555 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl": { 2024-05-27T08:00:55.555 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl", 2024-05-27T08:00:55.555 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.555 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.555 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.555 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.556 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.556 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.556 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.556 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.556 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.556 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.556 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.556 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.556 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl_server_port": { 2024-05-27T08:00:55.557 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl_server_port", 2024-05-27T08:00:55.557 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.557 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.557 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.557 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8443", 2024-05-27T08:00:55.557 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.557 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.557 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.557 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.558 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.558 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.558 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.558 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.558 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_behaviour": { 2024-05-27T08:00:55.558 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_behaviour", 2024-05-27T08:00:55.558 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.558 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.559 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.559 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "redirect", 2024-05-27T08:00:55.559 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.559 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.559 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.559 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.559 INFO:teuthology.orchestra.run.smithi114.stdout: "redirect" 2024-05-27T08:00:55.560 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.560 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.560 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.560 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.560 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.560 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.560 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_error_status_code": { 2024-05-27T08:00:55.560 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_error_status_code", 2024-05-27T08:00:55.561 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.561 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.561 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.561 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "500", 2024-05-27T08:00:55.561 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "400", 2024-05-27T08:00:55.561 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "599", 2024-05-27T08:00:55.561 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.561 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.562 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.562 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.562 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.562 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.562 INFO:teuthology.orchestra.run.smithi114.stdout: "url_prefix": { 2024-05-27T08:00:55.562 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "url_prefix", 2024-05-27T08:00:55.562 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.562 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.563 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.563 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.563 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.563 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.563 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.563 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.563 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.563 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.564 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.564 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.564 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.564 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.564 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "devicehealth", 2024-05-27T08:00:55.564 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.564 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.564 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.565 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_monitoring": { 2024-05-27T08:00:55.565 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enable_monitoring", 2024-05-27T08:00:55.565 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.565 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.565 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.565 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.565 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.565 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.566 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.566 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "monitor device health metrics", 2024-05-27T08:00:55.566 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.566 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.566 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.566 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.566 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.566 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.567 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.567 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.567 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.567 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.567 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.567 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.567 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.567 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.568 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.568 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.568 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.568 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.568 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.568 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.568 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.568 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.569 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.569 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.569 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.569 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.569 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.569 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.569 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.569 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.570 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.570 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.570 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.570 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.570 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.570 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.570 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.570 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.571 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.571 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.571 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.571 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.571 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.571 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.571 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.572 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.572 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.572 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.572 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.572 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.572 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.572 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.572 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.573 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.573 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.573 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.573 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.573 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.573 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.573 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.573 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.573 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.574 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.574 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.574 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.574 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.574 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.574 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.574 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.574 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.574 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.575 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.575 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.575 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.575 INFO:teuthology.orchestra.run.smithi114.stdout: "mark_out_threshold": { 2024-05-27T08:00:55.575 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mark_out_threshold", 2024-05-27T08:00:55.575 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.575 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.575 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.575 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "2419200", 2024-05-27T08:00:55.576 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.576 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.576 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.576 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T08:00:55.576 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.576 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.576 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.577 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": { 2024-05-27T08:00:55.577 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pool_name", 2024-05-27T08:00:55.577 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.577 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.577 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.577 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "device_health_metrics", 2024-05-27T08:00:55.577 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.577 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.578 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.578 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T08:00:55.578 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.578 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.578 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.578 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.578 INFO:teuthology.orchestra.run.smithi114.stdout: "retention_period": { 2024-05-27T08:00:55.578 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "retention_period", 2024-05-27T08:00:55.579 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.579 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.579 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.579 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15552000", 2024-05-27T08:00:55.579 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.579 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.579 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.579 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long to retain device health metrics", 2024-05-27T08:00:55.580 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.580 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.580 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.580 INFO:teuthology.orchestra.run.smithi114.stdout: "scrape_frequency": { 2024-05-27T08:00:55.580 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "scrape_frequency", 2024-05-27T08:00:55.580 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.580 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.580 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.581 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "86400", 2024-05-27T08:00:55.581 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.581 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.581 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.581 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T08:00:55.581 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.581 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.581 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.581 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.582 INFO:teuthology.orchestra.run.smithi114.stdout: "self_heal": { 2024-05-27T08:00:55.582 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "self_heal", 2024-05-27T08:00:55.582 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.582 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.582 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.582 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.582 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.582 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.582 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.583 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T08:00:55.583 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.583 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.583 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.583 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.583 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:55.583 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:55.584 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.584 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.584 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.584 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:55.584 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.584 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.584 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.584 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T08:00:55.584 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.585 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.585 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.585 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.585 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.585 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.585 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.585 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.585 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.586 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.586 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.586 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.586 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.586 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.586 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.586 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.587 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.587 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.587 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_threshold": { 2024-05-27T08:00:55.587 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_threshold", 2024-05-27T08:00:55.587 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.587 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.587 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.587 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "7257600", 2024-05-27T08:00:55.588 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.588 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.588 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.588 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T08:00:55.588 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.588 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.588 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.588 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.588 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.589 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.589 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.589 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "diskprediction_local", 2024-05-27T08:00:55.589 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.589 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.589 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.589 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.589 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.590 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.590 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.590 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.590 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.590 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.590 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.590 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.590 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.591 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.591 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.591 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.591 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.591 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.591 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.591 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.591 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.592 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.592 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.592 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.592 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.592 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.592 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.592 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.592 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.593 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.593 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.593 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.593 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.593 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.593 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.593 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.593 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.593 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.594 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.594 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.594 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.594 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.594 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.594 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.594 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.594 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.595 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.595 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.595 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.595 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.595 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.595 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.595 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.595 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.596 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.596 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.596 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.596 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.596 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.596 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.596 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.596 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.597 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.597 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.597 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.597 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.597 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.597 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.597 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.597 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.598 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.598 INFO:teuthology.orchestra.run.smithi114.stdout: "predict_interval": { 2024-05-27T08:00:55.598 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "predict_interval", 2024-05-27T08:00:55.598 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.598 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.598 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.598 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "86400", 2024-05-27T08:00:55.599 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.599 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.599 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.599 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.599 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.599 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.599 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.600 INFO:teuthology.orchestra.run.smithi114.stdout: "predictor_model": { 2024-05-27T08:00:55.600 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "predictor_model", 2024-05-27T08:00:55.600 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.600 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.600 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.600 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "prophetstor", 2024-05-27T08:00:55.600 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.600 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.600 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.601 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.601 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.601 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.601 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.601 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:55.601 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:55.601 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.602 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.602 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.602 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-05-27T08:00:55.602 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.602 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.602 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.602 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.603 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.603 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.603 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.603 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.603 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.603 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.603 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.604 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.604 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.604 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.604 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.604 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.604 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.604 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.604 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.605 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.605 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.605 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.605 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.605 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.605 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "influx", 2024-05-27T08:00:55.605 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": false, 2024-05-27T08:00:55.605 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "influxdb python module not found", 2024-05-27T08:00:55.605 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.606 INFO:teuthology.orchestra.run.smithi114.stdout: "batch_size": { 2024-05-27T08:00:55.606 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "batch_size", 2024-05-27T08:00:55.606 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.606 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.606 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.606 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5000", 2024-05-27T08:00:55.606 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.606 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.607 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.607 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T08:00:55.607 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.607 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.607 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.607 INFO:teuthology.orchestra.run.smithi114.stdout: "database": { 2024-05-27T08:00:55.607 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "database", 2024-05-27T08:00:55.608 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.608 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.608 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.608 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "ceph", 2024-05-27T08:00:55.608 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.608 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.608 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.608 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T08:00:55.609 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.609 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.609 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.609 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": { 2024-05-27T08:00:55.609 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "hostname", 2024-05-27T08:00:55.609 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.609 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.610 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.610 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.610 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.610 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.610 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.610 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB server hostname", 2024-05-27T08:00:55.610 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.611 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.611 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.611 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:55.611 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:55.611 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.611 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.611 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.611 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "30", 2024-05-27T08:00:55.612 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "5", 2024-05-27T08:00:55.612 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.612 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.612 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T08:00:55.612 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.612 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.612 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.613 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.613 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.613 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.613 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.613 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.613 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.613 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.613 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.613 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.614 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.614 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.614 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.614 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.614 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.614 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.614 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.615 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.615 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.615 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.615 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.615 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.615 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.615 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.615 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.615 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.616 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.616 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.616 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.616 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.616 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.616 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.616 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.616 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.617 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.617 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.617 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.617 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.617 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.617 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.617 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.617 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.618 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.618 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.618 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.618 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.618 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.618 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.618 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.619 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.619 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.619 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.619 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.619 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.619 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.619 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.620 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.620 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.620 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.620 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.620 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.620 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.620 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.620 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.620 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.621 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.621 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.621 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.621 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.621 INFO:teuthology.orchestra.run.smithi114.stdout: "password": { 2024-05-27T08:00:55.621 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "password", 2024-05-27T08:00:55.621 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.621 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.621 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.622 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.622 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.622 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.622 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.622 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "password of InfluxDB server user", 2024-05-27T08:00:55.622 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.622 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.622 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.623 INFO:teuthology.orchestra.run.smithi114.stdout: "port": { 2024-05-27T08:00:55.623 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "port", 2024-05-27T08:00:55.623 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.623 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.623 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.623 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8086", 2024-05-27T08:00:55.623 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.623 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.624 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.624 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB server port", 2024-05-27T08:00:55.624 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.624 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.624 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.624 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.624 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.624 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.625 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.625 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.625 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.625 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.625 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.625 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.625 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.625 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.626 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.626 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.626 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.626 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl": { 2024-05-27T08:00:55.626 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl", 2024-05-27T08:00:55.626 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.627 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.627 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.627 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "false", 2024-05-27T08:00:55.627 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.627 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.627 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.627 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T08:00:55.627 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.628 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.628 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.628 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.628 INFO:teuthology.orchestra.run.smithi114.stdout: "threads": { 2024-05-27T08:00:55.628 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "threads", 2024-05-27T08:00:55.628 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.628 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.628 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.629 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-05-27T08:00:55.629 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-05-27T08:00:55.629 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "32", 2024-05-27T08:00:55.629 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.629 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T08:00:55.629 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.629 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.629 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.630 INFO:teuthology.orchestra.run.smithi114.stdout: "username": { 2024-05-27T08:00:55.630 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "username", 2024-05-27T08:00:55.630 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.630 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.630 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.630 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.630 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.631 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.631 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.631 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "username of InfluxDB server user", 2024-05-27T08:00:55.631 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.631 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.631 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.631 INFO:teuthology.orchestra.run.smithi114.stdout: "verify_ssl": { 2024-05-27T08:00:55.632 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "verify_ssl", 2024-05-27T08:00:55.632 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.632 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.632 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.632 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "true", 2024-05-27T08:00:55.632 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.632 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.632 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.633 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T08:00:55.633 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.633 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.633 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.633 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.633 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.633 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.634 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "insights", 2024-05-27T08:00:55.634 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.634 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.634 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.634 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.634 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.634 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.635 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.635 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.635 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.635 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.635 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.635 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.635 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.635 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.635 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.636 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.636 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.636 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.636 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.636 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.636 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.636 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.636 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.637 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.637 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.637 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.637 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.637 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.637 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.637 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.638 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.638 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.638 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.638 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.638 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.638 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.638 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.638 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.639 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.639 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.639 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.639 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.639 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.639 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.639 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.639 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.640 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.640 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.640 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.640 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.640 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.640 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.640 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.640 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.641 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.641 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.641 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.641 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.641 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.641 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.641 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.642 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.642 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.642 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.642 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.642 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.642 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.642 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.642 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.643 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.643 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.643 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.643 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.643 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.643 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.643 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.644 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.644 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.644 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.644 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.644 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.644 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.644 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.644 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.645 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.645 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "iostat", 2024-05-27T08:00:55.645 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.645 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.645 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.645 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.645 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.646 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.646 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.646 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.646 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.646 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.646 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.646 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.646 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.647 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.647 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.647 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.647 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.647 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.647 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.647 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.647 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.648 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.648 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.648 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.648 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.648 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.648 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.648 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.649 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.649 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.649 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.649 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.649 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.649 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.649 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.649 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.650 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.650 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.650 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.650 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.650 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.650 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.650 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.650 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.651 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.651 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.651 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.651 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.651 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.651 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.651 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.651 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.652 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.652 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.652 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.652 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.652 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.652 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.652 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.653 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.653 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.653 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.653 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.653 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.653 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.653 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.653 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.654 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.654 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.654 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.654 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.654 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.654 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.654 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.655 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.655 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.655 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.655 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.655 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.655 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.655 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.655 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.656 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.656 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.656 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.656 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "k8sevents", 2024-05-27T08:00:55.656 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.656 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.656 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.657 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_event_retention_days": { 2024-05-27T08:00:55.657 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ceph_event_retention_days", 2024-05-27T08:00:55.657 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.657 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.657 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.657 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "7", 2024-05-27T08:00:55.657 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.657 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.657 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.658 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T08:00:55.658 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.658 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.658 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.658 INFO:teuthology.orchestra.run.smithi114.stdout: "config_check_secs": { 2024-05-27T08:00:55.658 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_check_secs", 2024-05-27T08:00:55.658 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.659 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.659 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.659 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:55.659 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "10", 2024-05-27T08:00:55.659 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.659 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.659 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T08:00:55.659 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.659 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.660 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.660 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.660 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.660 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.660 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.660 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.660 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.660 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.661 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.661 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.661 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.661 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.661 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.661 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.661 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.661 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.661 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.662 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.662 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.662 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.662 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.662 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.666 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.666 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.666 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.666 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.666 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.666 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.666 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.667 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.667 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.667 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.667 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.667 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.667 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.667 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.668 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.668 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.668 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.668 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.668 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.668 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.668 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.668 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.669 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.669 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.669 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.669 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.669 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.669 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.669 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.669 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "localpool", 2024-05-27T08:00:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "failure_domain": { 2024-05-27T08:00:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "failure_domain", 2024-05-27T08:00:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "host", 2024-05-27T08:00:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "failure domain for any created local pool", 2024-05-27T08:00:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T08:00:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": { 2024-05-27T08:00:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "min_size", 2024-05-27T08:00:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default min_size for any created local pool", 2024-05-27T08:00:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T08:00:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rep": { 2024-05-27T08:00:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "num_rep", 2024-05-27T08:00:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3", 2024-05-27T08:00:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default replica count for any created local pool", 2024-05-27T08:00:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": { 2024-05-27T08:00:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pg_num", 2024-05-27T08:00:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "128", 2024-05-27T08:00:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T08:00:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "prefix": { 2024-05-27T08:00:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prefix", 2024-05-27T08:00:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "name prefix for any created local pool", 2024-05-27T08:00:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "subtree": { 2024-05-27T08:00:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "subtree", 2024-05-27T08:00:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "rack", 2024-05-27T08:00:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T08:00:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T08:00:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mds_autoscaler", 2024-05-27T08:00:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mirroring", 2024-05-27T08:00:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "nfs", 2024-05-27T08:00:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "orchestrator", 2024-05-27T08:00:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "fail_fs": { 2024-05-27T08:00:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "fail_fs", 2024-05-27T08:00:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T08:00:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator": { 2024-05-27T08:00:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "orchestrator", 2024-05-27T08:00:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm", 2024-05-27T08:00:55.740 INFO:teuthology.orchestra.run.smithi114.stdout: "rook", 2024-05-27T08:00:55.740 INFO:teuthology.orchestra.run.smithi114.stdout: "test_orchestrator" 2024-05-27T08:00:55.740 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.740 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Orchestrator backend", 2024-05-27T08:00:55.740 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.740 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.740 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.740 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "osd_perf_query", 2024-05-27T08:00:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.751 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.751 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.751 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.751 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.751 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.751 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.751 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.751 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "osd_support", 2024-05-27T08:00:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pg_autoscaler", 2024-05-27T08:00:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "threshold": { 2024-05-27T08:00:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "threshold", 2024-05-27T08:00:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3.0", 2024-05-27T08:00:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1.0", 2024-05-27T08:00:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "scaling threshold", 2024-05-27T08:00:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T08:00:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "progress", 2024-05-27T08:00:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_pg_recovery_event": { 2024-05-27T08:00:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_pg_recovery_event", 2024-05-27T08:00:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T08:00:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "enabled": { 2024-05-27T08:00:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enabled", 2024-05-27T08:00:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "max_completed_events": { 2024-05-27T08:00:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_completed_events", 2024-05-27T08:00:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "50", 2024-05-27T08:00:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "number of past completed events to remember", 2024-05-27T08:00:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-05-27T08:00:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-05-27T08:00:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-05-27T08:00:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long the module is going to sleep", 2024-05-27T08:00:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prometheus", 2024-05-27T08:00:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "cache": { 2024-05-27T08:00:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cache", 2024-05-27T08:00:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "exclude_perf_counters": { 2024-05-27T08:00:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "exclude_perf_counters", 2024-05-27T08:00:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T08:00:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T08:00:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_stats_pools": { 2024-05-27T08:00:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_stats_pools", 2024-05-27T08:00:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T08:00:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T08:00:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "300", 2024-05-27T08:00:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "scrape_interval": { 2024-05-27T08:00:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "scrape_interval", 2024-05-27T08:00:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15.0", 2024-05-27T08:00:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-05-27T08:00:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-05-27T08:00:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "::", 2024-05-27T08:00:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T08:00:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-05-27T08:00:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-05-27T08:00:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "9283", 2024-05-27T08:00:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T08:00:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "stale_cache_strategy": { 2024-05-27T08:00:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "stale_cache_strategy", 2024-05-27T08:00:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "log", 2024-05-27T08:00:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_behaviour": { 2024-05-27T08:00:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_behaviour", 2024-05-27T08:00:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "default", 2024-05-27T08:00:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "default", 2024-05-27T08:00:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "error" 2024-05-27T08:00:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_error_status_code": { 2024-05-27T08:00:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_error_status_code", 2024-05-27T08:00:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "500", 2024-05-27T08:00:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "400", 2024-05-27T08:00:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "599", 2024-05-27T08:00:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_support", 2024-05-27T08:00:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "max_concurrent_snap_create": { 2024-05-27T08:00:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_concurrent_snap_create", 2024-05-27T08:00:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-05-27T08:00:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "mirror_snapshot_schedule": { 2024-05-27T08:00:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T08:00:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "trash_purge_schedule": { 2024-05-27T08:00:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "trash_purge_schedule", 2024-05-27T08:00:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.838 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.838 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "restful", 2024-05-27T08:00:55.838 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.838 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.838 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.838 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_auth": { 2024-05-27T08:00:55.838 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enable_auth", 2024-05-27T08:00:55.838 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.839 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.839 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.839 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.839 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.839 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.839 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.839 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.839 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.840 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.840 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.840 INFO:teuthology.orchestra.run.smithi114.stdout: "key_file": { 2024-05-27T08:00:55.840 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "key_file", 2024-05-27T08:00:55.840 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.840 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.840 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.840 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.841 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.841 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.841 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.841 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.841 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.841 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.841 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.841 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.842 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.842 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.842 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.842 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.842 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.842 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.842 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.842 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.843 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.843 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.843 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.843 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.843 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.843 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.843 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.843 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.843 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.844 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.844 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.844 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.844 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.844 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.844 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.844 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.844 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.845 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.845 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.845 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.845 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.845 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.845 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.845 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.845 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.846 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.846 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.846 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.846 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.846 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.846 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.846 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.846 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.847 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.847 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.847 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.847 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.847 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.847 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.847 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.847 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.847 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.847 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.848 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.848 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.848 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.848 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.848 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.848 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.848 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.848 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.849 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.849 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.849 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.849 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.849 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.849 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.849 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-05-27T08:00:55.850 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-05-27T08:00:55.850 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.850 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.850 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.850 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.850 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.850 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.850 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.851 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.851 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.851 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.851 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.851 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-05-27T08:00:55.851 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-05-27T08:00:55.851 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.852 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.852 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.852 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.852 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.852 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.852 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.852 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.852 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.853 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.853 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.853 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.853 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.853 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.853 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.853 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.854 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.854 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.854 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.854 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.854 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.854 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.854 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.854 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.855 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.855 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.855 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.855 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rgw", 2024-05-27T08:00:55.855 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.855 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.855 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.856 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.856 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.856 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.856 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.856 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.856 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.856 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.856 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.856 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.857 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.857 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.857 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.857 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.857 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.857 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.857 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.857 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.857 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.858 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.858 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.858 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.858 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.858 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.858 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.858 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.859 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.859 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.859 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.859 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.859 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.859 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.859 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.859 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.860 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.860 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.860 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.860 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.860 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.860 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.860 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.860 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.861 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.861 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.861 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.861 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.861 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.861 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.861 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.861 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.861 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.862 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.862 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.862 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.862 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.862 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.862 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.862 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.863 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.863 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.863 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.863 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.863 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.863 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.863 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.863 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.863 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.864 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.864 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.864 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.864 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.864 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.864 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.864 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.864 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.865 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.865 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.865 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.865 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.865 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.865 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.865 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.865 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.865 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.866 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rook", 2024-05-27T08:00:55.866 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.866 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.866 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.866 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.866 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.870 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.870 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.870 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.870 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.870 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.870 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.871 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.871 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.871 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.871 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.871 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.871 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.871 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.871 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.871 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.872 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.872 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.872 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.872 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.872 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.872 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.872 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.873 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.873 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.873 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.873 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.873 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.873 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.873 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.873 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.873 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.874 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.874 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.874 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.874 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.874 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.874 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.874 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.875 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.875 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.875 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.875 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.875 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.875 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.875 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.875 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.875 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.876 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.876 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.876 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.876 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.876 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.876 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.876 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.876 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.877 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.877 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.877 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.877 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.877 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.877 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.877 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.877 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.878 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.878 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.878 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.878 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.878 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.878 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.878 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.878 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.879 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.879 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.879 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.879 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.879 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.879 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.879 INFO:teuthology.orchestra.run.smithi114.stdout: "storage_class": { 2024-05-27T08:00:55.880 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "storage_class", 2024-05-27T08:00:55.880 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.880 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.880 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.880 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "local", 2024-05-27T08:00:55.880 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.880 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.880 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.880 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T08:00:55.881 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.881 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.881 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.881 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.881 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.881 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "selftest", 2024-05-27T08:00:55.881 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.882 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.882 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.882 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.882 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.882 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.882 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.882 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.882 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.882 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.883 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.883 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.883 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.883 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.883 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.883 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.883 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.883 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.884 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.884 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.884 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.884 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.884 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.884 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.884 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.885 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.885 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.885 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.885 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.885 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.885 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.885 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.886 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.886 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.886 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.886 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.886 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.886 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.886 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.886 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.887 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.887 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.887 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.887 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.887 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.887 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.887 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.887 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.888 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.888 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.888 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.888 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.888 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.888 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.888 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.888 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.889 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.889 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.889 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.889 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.889 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.889 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.889 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.890 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.890 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.890 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.890 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.890 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.890 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.890 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.891 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.891 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.891 INFO:teuthology.orchestra.run.smithi114.stdout: "roption1": { 2024-05-27T08:00:55.891 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "roption1", 2024-05-27T08:00:55.891 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.891 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.891 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.891 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.892 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.892 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.892 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.892 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.892 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.892 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.892 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.893 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.893 INFO:teuthology.orchestra.run.smithi114.stdout: "roption2": { 2024-05-27T08:00:55.893 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "roption2", 2024-05-27T08:00:55.893 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.893 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.893 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.893 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "xyz", 2024-05-27T08:00:55.894 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.894 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.894 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.894 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.894 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.894 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.894 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.895 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption1": { 2024-05-27T08:00:55.895 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption1", 2024-05-27T08:00:55.895 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.895 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.895 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.895 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.895 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.896 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.896 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.896 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.896 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.896 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.896 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.896 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption2": { 2024-05-27T08:00:55.897 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption2", 2024-05-27T08:00:55.897 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.897 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.897 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.897 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.897 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.897 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.897 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.897 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.898 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.898 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.898 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.898 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.898 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption3": { 2024-05-27T08:00:55.898 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption3", 2024-05-27T08:00:55.898 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-05-27T08:00:55.898 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.899 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.899 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.899 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.899 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.899 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.899 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.899 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.899 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.899 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.900 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.900 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption4": { 2024-05-27T08:00:55.900 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption4", 2024-05-27T08:00:55.900 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.900 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.900 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.900 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.900 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.901 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.901 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.901 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.901 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.901 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.901 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.901 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.901 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption5": { 2024-05-27T08:00:55.902 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption5", 2024-05-27T08:00:55.902 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.902 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.902 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.902 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.902 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.902 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.902 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.903 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.903 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.903 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.903 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.903 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption6": { 2024-05-27T08:00:55.903 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption6", 2024-05-27T08:00:55.903 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.903 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.904 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.904 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.904 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.904 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.904 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.904 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.904 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.904 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.905 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.905 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.905 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption7": { 2024-05-27T08:00:55.905 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption7", 2024-05-27T08:00:55.905 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.905 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.905 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.905 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.905 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-05-27T08:00:55.905 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "42", 2024-05-27T08:00:55.906 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.906 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.906 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.906 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.906 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.906 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.906 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.907 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.907 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.907 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.907 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.907 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.907 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.907 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.907 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.907 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.908 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.908 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.908 INFO:teuthology.orchestra.run.smithi114.stdout: "testkey": { 2024-05-27T08:00:55.908 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testkey", 2024-05-27T08:00:55.908 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.908 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.908 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.909 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.909 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.909 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.909 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.909 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.909 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.909 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.909 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.910 INFO:teuthology.orchestra.run.smithi114.stdout: "testlkey": { 2024-05-27T08:00:55.910 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testlkey", 2024-05-27T08:00:55.910 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.910 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.910 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.910 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.911 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.911 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.911 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.911 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.911 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.911 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.911 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.911 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.912 INFO:teuthology.orchestra.run.smithi114.stdout: "testnewline": { 2024-05-27T08:00:55.912 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testnewline", 2024-05-27T08:00:55.912 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.912 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.912 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.912 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.912 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.912 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.912 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.912 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.913 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.913 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.913 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.913 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.913 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.913 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smb", 2024-05-27T08:00:55.914 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.914 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.914 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.914 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.914 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.914 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.914 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.914 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.915 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.915 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.915 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.915 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.915 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.915 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.915 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.915 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.916 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.916 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.916 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.916 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.916 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.916 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.916 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.916 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.917 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.917 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.917 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.917 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.917 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.917 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.917 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.917 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.918 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.918 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.918 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.918 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.918 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.918 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.918 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.918 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.919 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.919 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.919 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.919 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.919 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.919 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.919 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.919 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.920 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.920 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.920 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.920 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.920 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.920 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.920 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.920 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.921 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.921 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.921 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.921 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.921 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.921 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.921 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.922 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.922 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.922 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.922 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.922 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.922 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.922 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.922 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.923 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.923 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.923 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.923 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.923 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.923 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.923 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.924 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.924 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.924 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.924 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.924 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.924 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.924 INFO:teuthology.orchestra.run.smithi114.stdout: "update_orchestration": { 2024-05-27T08:00:55.925 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "update_orchestration", 2024-05-27T08:00:55.925 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.925 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.925 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.925 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.925 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.925 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.925 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.926 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T08:00:55.926 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.926 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.926 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.926 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.926 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.926 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.926 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.927 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snap_schedule", 2024-05-27T08:00:55.927 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.927 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.927 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.927 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_m_granularity": { 2024-05-27T08:00:55.927 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_m_granularity", 2024-05-27T08:00:55.927 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.927 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.928 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.928 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.928 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.928 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.928 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.928 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T08:00:55.928 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.928 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.929 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.929 INFO:teuthology.orchestra.run.smithi114.stdout: "dump_on_update": { 2024-05-27T08:00:55.929 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "dump_on_update", 2024-05-27T08:00:55.929 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.929 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.929 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.929 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.930 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.930 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.930 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.930 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "dump database to debug log on update", 2024-05-27T08:00:55.930 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.930 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.930 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.931 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.931 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.931 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.931 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.931 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.931 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.931 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.931 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.931 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.932 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.932 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.932 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.932 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.932 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.932 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.932 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.932 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.932 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.933 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.933 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.933 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.933 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.933 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.933 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.933 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.934 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.934 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.934 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.934 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.934 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.934 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.934 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.934 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.935 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.935 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.935 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.935 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.935 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.935 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.935 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.935 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.936 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.936 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.936 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.936 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.936 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.936 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.936 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.936 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.936 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.937 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.937 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.937 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.937 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.937 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.937 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.937 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.937 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.938 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.938 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.938 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.938 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.938 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.938 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.938 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.938 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.939 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.939 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.939 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.939 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.939 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.939 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.939 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.940 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.940 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.940 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.940 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.940 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.940 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.940 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.940 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.941 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "stats", 2024-05-27T08:00:55.941 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.941 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.941 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.941 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.941 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.941 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.942 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.942 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.942 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.942 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.942 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.942 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.942 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.942 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.943 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.943 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.943 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.943 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.943 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.943 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.943 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.943 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.943 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.944 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.944 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.944 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.944 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.944 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.944 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.944 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.944 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.945 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.945 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.945 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.945 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.945 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.945 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.945 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.946 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.946 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.946 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.946 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.946 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.946 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.946 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.947 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.947 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.947 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.947 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.947 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.947 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.947 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.947 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.948 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.948 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.948 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.948 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.948 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.948 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.948 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.949 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.949 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.949 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.949 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.949 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.949 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.949 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.949 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.950 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.950 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.950 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.950 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.950 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.950 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.950 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.951 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.951 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.951 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.951 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.951 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.951 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.951 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.951 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.952 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.952 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.952 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.952 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "status", 2024-05-27T08:00:55.952 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.952 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.952 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.953 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.953 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.953 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.953 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.953 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.953 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.953 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.953 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.954 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.954 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.954 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.954 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.954 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.954 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.954 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.954 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.955 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.955 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.955 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.955 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.955 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.955 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.955 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.956 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.956 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.956 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.956 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.956 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.956 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.956 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.956 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.957 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.957 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.957 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.957 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.957 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.957 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.957 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.957 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.958 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.958 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.958 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.958 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.958 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.958 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.958 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.959 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.959 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.959 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.959 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.959 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.959 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.959 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.959 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.960 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.960 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.960 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.960 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.960 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.960 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.960 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.960 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.961 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.961 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.961 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.961 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.961 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.961 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.961 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.961 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.962 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.962 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.962 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.962 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.962 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.962 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.962 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.962 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.963 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.963 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.963 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.963 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.963 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "telegraf", 2024-05-27T08:00:55.963 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.963 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.963 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.963 INFO:teuthology.orchestra.run.smithi114.stdout: "address": { 2024-05-27T08:00:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "address", 2024-05-27T08:00:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T08:00:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.966 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15", 2024-05-27T08:00:55.966 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.966 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.966 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.966 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.966 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.966 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.966 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.967 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.967 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.967 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.967 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.967 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.967 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.967 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.967 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.967 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.968 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.968 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.968 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.968 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.968 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.968 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.968 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.968 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.968 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.969 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.969 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.969 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:55.969 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:55.969 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.969 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.969 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.969 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.970 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.970 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.970 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.970 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.970 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.970 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.970 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.971 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:55.971 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:55.971 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.971 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.971 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.971 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:55.971 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.971 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.972 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.972 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:55.972 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:55.972 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:55.972 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:55.972 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:55.972 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:55.972 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:55.972 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.973 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.973 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.973 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.973 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:55.973 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:55.973 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.973 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.973 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.974 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.974 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.974 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.974 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.974 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.974 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.974 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.974 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.975 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:55.975 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:55.975 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.975 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:55.975 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.975 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:55.975 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.975 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.976 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.976 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.976 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.976 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.976 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.976 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.976 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:55.976 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.976 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:55.977 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "telemetry", 2024-05-27T08:00:55.977 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:55.977 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:55.977 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:55.977 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_basic": { 2024-05-27T08:00:55.977 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_basic", 2024-05-27T08:00:55.977 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.977 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.977 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.978 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.978 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.978 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.978 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.978 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T08:00:55.978 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.978 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.979 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.979 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_crash": { 2024-05-27T08:00:55.979 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_crash", 2024-05-27T08:00:55.979 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.979 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.979 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.979 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.979 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.980 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.980 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.980 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T08:00:55.980 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.980 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.980 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.980 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_device": { 2024-05-27T08:00:55.981 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_device", 2024-05-27T08:00:55.981 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.981 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.981 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.981 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:55.981 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.981 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.981 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.981 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T08:00:55.982 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.982 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.982 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.982 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_ident": { 2024-05-27T08:00:55.982 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_ident", 2024-05-27T08:00:55.982 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.982 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.982 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.983 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.983 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.983 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.983 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.983 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T08:00:55.983 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.983 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.983 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.983 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.984 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_perf": { 2024-05-27T08:00:55.984 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_perf", 2024-05-27T08:00:55.984 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.984 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.984 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.984 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.984 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.984 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.984 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.985 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T08:00:55.985 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.985 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.985 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.985 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.985 INFO:teuthology.orchestra.run.smithi114.stdout: "contact": { 2024-05-27T08:00:55.985 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "contact", 2024-05-27T08:00:55.985 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.986 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.986 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.986 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.986 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.986 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.986 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.986 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.986 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.987 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.987 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.987 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.987 INFO:teuthology.orchestra.run.smithi114.stdout: "description": { 2024-05-27T08:00:55.987 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "description", 2024-05-27T08:00:55.987 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.987 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.987 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.987 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.988 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.988 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.988 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.988 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.988 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.988 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.988 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.988 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.988 INFO:teuthology.orchestra.run.smithi114.stdout: "device_url": { 2024-05-27T08:00:55.988 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_url", 2024-05-27T08:00:55.989 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.989 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.989 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.989 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T08:00:55.989 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.989 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.989 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.989 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.989 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.990 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.990 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.990 INFO:teuthology.orchestra.run.smithi114.stdout: "enabled": { 2024-05-27T08:00:55.990 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enabled", 2024-05-27T08:00:55.990 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.990 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.990 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.991 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.991 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.991 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.991 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.991 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.991 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.991 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.991 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.992 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.992 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:55.992 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:55.992 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.992 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.992 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.992 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "24", 2024-05-27T08:00:55.992 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "8", 2024-05-27T08:00:55.993 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.993 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.993 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.993 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.993 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.993 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.993 INFO:teuthology.orchestra.run.smithi114.stdout: "last_opt_revision": { 2024-05-27T08:00:55.994 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "last_opt_revision", 2024-05-27T08:00:55.994 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:55.994 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.994 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.994 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1", 2024-05-27T08:00:55.994 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.994 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.994 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.995 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.995 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.995 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.995 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.995 INFO:teuthology.orchestra.run.smithi114.stdout: "leaderboard": { 2024-05-27T08:00:55.995 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "leaderboard", 2024-05-27T08:00:55.995 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:55.996 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.996 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.996 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:55.996 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.996 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.996 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.996 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.996 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.996 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.997 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.997 INFO:teuthology.orchestra.run.smithi114.stdout: "leaderboard_description": { 2024-05-27T08:00:55.997 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "leaderboard_description", 2024-05-27T08:00:55.997 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.997 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.997 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:55.997 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.997 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.998 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.998 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:55.998 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:55.998 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:55.998 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:55.998 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:55.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:55.998 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:55.999 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:55.999 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:55.999 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:55.999 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:55.999 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:55.999 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:55.999 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:55.999 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:55.999 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:56.000 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:56.000 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:56.000 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:56.000 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:56.000 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:56.000 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:56.000 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.000 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.001 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.001 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.001 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:56.001 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:56.001 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:56.001 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.001 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.002 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:56.002 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.002 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.002 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.002 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.002 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.002 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.002 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.003 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:56.003 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:56.003 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:56.003 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.003 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.003 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:56.003 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.004 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.004 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:56.004 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:56.004 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:56.004 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:56.004 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:56.004 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:56.004 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:56.005 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:56.005 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.005 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.005 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.005 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.005 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:56.005 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:56.005 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:56.006 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.006 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.006 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:56.006 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.006 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.006 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.006 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.006 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.007 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.007 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.007 INFO:teuthology.orchestra.run.smithi114.stdout: "organization": { 2024-05-27T08:00:56.007 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "organization", 2024-05-27T08:00:56.007 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:56.007 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.007 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:56.008 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:56.008 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.008 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.008 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.008 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.008 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.008 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.008 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.009 INFO:teuthology.orchestra.run.smithi114.stdout: "proxy": { 2024-05-27T08:00:56.009 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "proxy", 2024-05-27T08:00:56.009 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:56.009 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.009 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:56.009 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:56.009 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.009 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.010 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.010 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.010 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.010 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.010 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.010 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:56.010 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:56.011 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:56.011 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:56.011 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.011 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:56.011 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.011 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.011 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.011 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.012 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.012 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.012 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.012 INFO:teuthology.orchestra.run.smithi114.stdout: "url": { 2024-05-27T08:00:56.012 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "url", 2024-05-27T08:00:56.012 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:56.012 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.012 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:56.013 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T08:00:56.013 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.013 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.013 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.013 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.013 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.013 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.013 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.014 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.014 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.014 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "test_orchestrator", 2024-05-27T08:00:56.014 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:56.014 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:56.014 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:56.015 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:56.015 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:56.015 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:56.015 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.015 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.015 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:56.015 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.015 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.016 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:56.016 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:56.016 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:56.016 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:56.016 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:56.016 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:56.016 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:56.016 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:56.017 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.017 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.017 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.017 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.017 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:56.017 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:56.017 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:56.018 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.018 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.018 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:56.018 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.018 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.018 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.018 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.018 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.018 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.019 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.019 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:56.019 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:56.019 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:56.019 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.019 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.020 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:56.020 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.020 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.020 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:56.020 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:56.020 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:56.020 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:56.020 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:56.020 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:56.021 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:56.021 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:56.021 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.021 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.021 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.021 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.021 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:56.021 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:56.022 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:56.022 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.022 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.022 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:56.022 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.022 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.022 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.022 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.023 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.023 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.023 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.023 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:56.023 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:56.023 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:56.023 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:56.023 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.024 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:56.024 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.024 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.024 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.024 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.024 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.024 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.024 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.024 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.025 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.025 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "volumes", 2024-05-27T08:00:56.025 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:56.025 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:56.025 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:56.025 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:56.025 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:56.026 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:56.026 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.026 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.026 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:56.026 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.026 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.026 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:56.026 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:56.026 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:56.027 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:56.027 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:56.027 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:56.027 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:56.027 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:56.027 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.027 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.027 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.027 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.028 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:56.028 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:56.028 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:56.028 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.028 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.028 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:56.028 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.029 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.029 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.029 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.029 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.029 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.029 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.029 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:56.029 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:56.030 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:56.030 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.030 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.030 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:56.030 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.030 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.030 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:56.030 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:56.031 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:56.031 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:56.031 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:56.031 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:56.031 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:56.031 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:56.031 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.031 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.031 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.032 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.032 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:56.032 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:56.032 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:56.032 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.032 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.032 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:56.033 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.033 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.033 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.033 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.033 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.033 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.033 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.033 INFO:teuthology.orchestra.run.smithi114.stdout: "max_concurrent_clones": { 2024-05-27T08:00:56.034 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_concurrent_clones", 2024-05-27T08:00:56.034 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:56.034 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.034 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:56.034 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "4", 2024-05-27T08:00:56.034 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.034 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.035 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.035 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T08:00:56.035 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.035 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.035 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.035 INFO:teuthology.orchestra.run.smithi114.stdout: "periodic_async_work": { 2024-05-27T08:00:56.035 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "periodic_async_work", 2024-05-27T08:00:56.035 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:56.036 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.036 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:56.036 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:56.036 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.036 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.036 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.036 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Periodically check for async work", 2024-05-27T08:00:56.036 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.036 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.037 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.037 INFO:teuthology.orchestra.run.smithi114.stdout: "snapshot_clone_delay": { 2024-05-27T08:00:56.037 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snapshot_clone_delay", 2024-05-27T08:00:56.037 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:56.037 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.037 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:56.037 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:56.038 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.038 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.038 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.038 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T08:00:56.038 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.038 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.038 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.039 INFO:teuthology.orchestra.run.smithi114.stdout: "snapshot_clone_no_wait": { 2024-05-27T08:00:56.039 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T08:00:56.039 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:56.039 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.039 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:56.039 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-05-27T08:00:56.039 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.039 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.040 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.040 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T08:00:56.040 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.040 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.040 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.040 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:56.040 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:56.041 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:56.041 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:56.041 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.041 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:56.041 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.041 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.041 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.041 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.042 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.042 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.042 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.042 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.042 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.042 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix", 2024-05-27T08:00:56.043 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-05-27T08:00:56.043 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-05-27T08:00:56.043 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-05-27T08:00:56.043 INFO:teuthology.orchestra.run.smithi114.stdout: "discovery_interval": { 2024-05-27T08:00:56.043 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "discovery_interval", 2024-05-27T08:00:56.043 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-05-27T08:00:56.043 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.043 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:56.043 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "100", 2024-05-27T08:00:56.044 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.044 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.044 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.044 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.044 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.044 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.044 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.045 INFO:teuthology.orchestra.run.smithi114.stdout: "identifier": { 2024-05-27T08:00:56.045 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "identifier", 2024-05-27T08:00:56.045 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:56.045 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.045 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:56.045 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:56.045 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.045 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.046 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.046 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.046 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.046 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.046 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.046 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-05-27T08:00:56.046 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-05-27T08:00:56.047 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-05-27T08:00:56.047 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.047 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:56.047 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-05-27T08:00:56.047 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.047 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.047 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.047 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.047 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-05-27T08:00:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-05-27T08:00:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-05-27T08:00:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-05-27T08:00:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-05-27T08:00:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-05-27T08:00:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-05-27T08:00:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-05-27T08:00:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-05-27T08:00:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-05-27T08:00:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-05-27T08:00:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-05-27T08:00:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-05-27T08:00:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-05-27T08:00:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-05-27T08:00:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-05-27T08:00:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-05-27T08:00:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-05-27T08:00:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: "sqlite3_killpoint": { 2024-05-27T08:00:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sqlite3_killpoint", 2024-05-27T08:00:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "dev", 2024-05-27T08:00:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-05-27T08:00:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-05-27T08:00:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_host": { 2024-05-27T08:00:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_host", 2024-05-27T08:00:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-05-27T08:00:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.060 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.060 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.060 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_port": { 2024-05-27T08:00:56.060 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_port", 2024-05-27T08:00:56.060 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-05-27T08:00:56.060 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.060 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:56.061 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10051", 2024-05-27T08:00:56.061 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.061 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.061 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.061 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.061 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.061 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.061 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.062 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_sender": { 2024-05-27T08:00:56.062 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_sender", 2024-05-27T08:00:56.062 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-05-27T08:00:56.062 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-05-27T08:00:56.062 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-05-27T08:00:56.062 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T08:00:56.062 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-05-27T08:00:56.063 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-05-27T08:00:56.063 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-05-27T08:00:56.063 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-05-27T08:00:56.063 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-05-27T08:00:56.063 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-05-27T08:00:56.063 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-05-27T08:00:56.063 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.063 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.064 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.064 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:56.064 INFO:teuthology.orchestra.run.smithi114.stdout: "services": { 2024-05-27T08:00:56.064 INFO:teuthology.orchestra.run.smithi114.stdout: "dashboard": "https://172.21.15.114:8443/" 2024-05-27T08:00:56.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.064 INFO:teuthology.orchestra.run.smithi114.stdout: "always_on_modules": { 2024-05-27T08:00:56.064 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus": [ 2024-05-27T08:00:56.064 INFO:teuthology.orchestra.run.smithi114.stdout: "balancer", 2024-05-27T08:00:56.064 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-05-27T08:00:56.065 INFO:teuthology.orchestra.run.smithi114.stdout: "devicehealth", 2024-05-27T08:00:56.065 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator", 2024-05-27T08:00:56.065 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscaler", 2024-05-27T08:00:56.065 INFO:teuthology.orchestra.run.smithi114.stdout: "progress", 2024-05-27T08:00:56.065 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_support", 2024-05-27T08:00:56.065 INFO:teuthology.orchestra.run.smithi114.stdout: "status", 2024-05-27T08:00:56.065 INFO:teuthology.orchestra.run.smithi114.stdout: "telemetry", 2024-05-27T08:00:56.065 INFO:teuthology.orchestra.run.smithi114.stdout: "volumes" 2024-05-27T08:00:56.065 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:56.066 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific": [ 2024-05-27T08:00:56.066 INFO:teuthology.orchestra.run.smithi114.stdout: "balancer", 2024-05-27T08:00:56.066 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-05-27T08:00:56.066 INFO:teuthology.orchestra.run.smithi114.stdout: "devicehealth", 2024-05-27T08:00:56.066 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator", 2024-05-27T08:00:56.066 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscaler", 2024-05-27T08:00:56.066 INFO:teuthology.orchestra.run.smithi114.stdout: "progress", 2024-05-27T08:00:56.066 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_support", 2024-05-27T08:00:56.067 INFO:teuthology.orchestra.run.smithi114.stdout: "status", 2024-05-27T08:00:56.067 INFO:teuthology.orchestra.run.smithi114.stdout: "telemetry", 2024-05-27T08:00:56.067 INFO:teuthology.orchestra.run.smithi114.stdout: "volumes" 2024-05-27T08:00:56.067 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:56.067 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy": [ 2024-05-27T08:00:56.067 INFO:teuthology.orchestra.run.smithi114.stdout: "balancer", 2024-05-27T08:00:56.067 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-05-27T08:00:56.067 INFO:teuthology.orchestra.run.smithi114.stdout: "devicehealth", 2024-05-27T08:00:56.067 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator", 2024-05-27T08:00:56.068 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscaler", 2024-05-27T08:00:56.068 INFO:teuthology.orchestra.run.smithi114.stdout: "progress", 2024-05-27T08:00:56.068 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_support", 2024-05-27T08:00:56.068 INFO:teuthology.orchestra.run.smithi114.stdout: "status", 2024-05-27T08:00:56.068 INFO:teuthology.orchestra.run.smithi114.stdout: "telemetry", 2024-05-27T08:00:56.068 INFO:teuthology.orchestra.run.smithi114.stdout: "volumes" 2024-05-27T08:00:56.068 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:56.068 INFO:teuthology.orchestra.run.smithi114.stdout: "reef": [ 2024-05-27T08:00:56.068 INFO:teuthology.orchestra.run.smithi114.stdout: "balancer", 2024-05-27T08:00:56.069 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-05-27T08:00:56.069 INFO:teuthology.orchestra.run.smithi114.stdout: "devicehealth", 2024-05-27T08:00:56.069 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator", 2024-05-27T08:00:56.069 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscaler", 2024-05-27T08:00:56.069 INFO:teuthology.orchestra.run.smithi114.stdout: "progress", 2024-05-27T08:00:56.069 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_support", 2024-05-27T08:00:56.069 INFO:teuthology.orchestra.run.smithi114.stdout: "status", 2024-05-27T08:00:56.069 INFO:teuthology.orchestra.run.smithi114.stdout: "telemetry", 2024-05-27T08:00:56.069 INFO:teuthology.orchestra.run.smithi114.stdout: "volumes" 2024-05-27T08:00:56.070 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-05-27T08:00:56.070 INFO:teuthology.orchestra.run.smithi114.stdout: "squid": [ 2024-05-27T08:00:56.070 INFO:teuthology.orchestra.run.smithi114.stdout: "balancer", 2024-05-27T08:00:56.070 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-05-27T08:00:56.070 INFO:teuthology.orchestra.run.smithi114.stdout: "devicehealth", 2024-05-27T08:00:56.070 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator", 2024-05-27T08:00:56.070 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscaler", 2024-05-27T08:00:56.070 INFO:teuthology.orchestra.run.smithi114.stdout: "progress", 2024-05-27T08:00:56.071 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_support", 2024-05-27T08:00:56.071 INFO:teuthology.orchestra.run.smithi114.stdout: "status", 2024-05-27T08:00:56.071 INFO:teuthology.orchestra.run.smithi114.stdout: "telemetry", 2024-05-27T08:00:56.074 INFO:teuthology.orchestra.run.smithi114.stdout: "volumes" 2024-05-27T08:00:56.074 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-05-27T08:00:56.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.074 INFO:teuthology.orchestra.run.smithi114.stdout: "last_failure_osd_epoch": 90, 2024-05-27T08:00:56.074 INFO:teuthology.orchestra.run.smithi114.stdout: "active_clients": [ 2024-05-27T08:00:56.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.074 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "devicehealth", 2024-05-27T08:00:56.075 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-05-27T08:00:56.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.075 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-05-27T08:00:56.075 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:0", 2024-05-27T08:00:56.075 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 918710886 2024-05-27T08:00:56.075 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.075 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-05-27T08:00:56.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.076 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "libcephsqlite", 2024-05-27T08:00:56.076 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-05-27T08:00:56.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.076 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-05-27T08:00:56.076 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:0", 2024-05-27T08:00:56.076 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 186989419 2024-05-27T08:00:56.076 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.076 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-05-27T08:00:56.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.077 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_support", 2024-05-27T08:00:56.077 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-05-27T08:00:56.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.077 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-05-27T08:00:56.077 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:0", 2024-05-27T08:00:56.077 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 425543839 2024-05-27T08:00:56.077 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-05-27T08:00:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snap_schedule", 2024-05-27T08:00:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-05-27T08:00:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-05-27T08:00:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:0", 2024-05-27T08:00:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 622758261 2024-05-27T08:00:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-05-27T08:00:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snap_schedule", 2024-05-27T08:00:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-05-27T08:00:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-05-27T08:00:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:0", 2024-05-27T08:00:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 144964737 2024-05-27T08:00:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-05-27T08:00:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-05-27T08:00:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "volumes", 2024-05-27T08:00:56.081 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-05-27T08:00:56.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-05-27T08:00:56.081 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-05-27T08:00:56.081 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:0", 2024-05-27T08:00:56.081 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3663457551 2024-05-27T08:00:56.081 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.081 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-05-27T08:00:56.081 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-05-27T08:00:56.081 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-05-27T08:00:56.082 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-05-27T08:00:56.083 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:53 smithi186 ceph-mon[36647]: pgmap v733: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 34 KiB/s wr, 3 op/s 2024-05-27T08:00:56.084 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:53 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2057093489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T08:00:56.084 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:53 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/4127074315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T08:00:56.084 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:53 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2136444503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T08:00:56.084 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:54 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1321248586' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T08:00:56.084 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:55 smithi186 ceph-mon[36647]: pgmap v734: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 34 KiB/s wr, 3 op/s 2024-05-27T08:00:56.084 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:53 smithi163 ceph-mon[36451]: pgmap v733: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 34 KiB/s wr, 3 op/s 2024-05-27T08:00:56.084 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:53 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2057093489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T08:00:56.085 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:53 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/4127074315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T08:00:56.085 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:53 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2136444503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T08:00:56.085 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:54 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1321248586' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T08:00:56.085 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:55 smithi163 ceph-mon[36451]: pgmap v734: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 34 KiB/s wr, 3 op/s 2024-05-27T08:00:56.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.568+0000 7f1987fff640 1 -- 172.21.15.114:0/1321248586 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f19700b2950 msgr2=0x7f19700b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:56.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.568+0000 7f1987fff640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f19700b2950 0x7f19700b4e60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1978000f80 tx=0x7f1978009360 comp rx=0 tx=0).stop 2024-05-27T08:00:56.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.568+0000 7f1987fff640 1 -- 172.21.15.114:0/1321248586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19881041b0 msgr2=0x7f1988074fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:00:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.568+0000 7f1987fff640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19881041b0 0x7f1988074fc0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f197c00e9f0 tx=0x7f197c00eeb0 comp rx=0 tx=0).stop 2024-05-27T08:00:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.568+0000 7f1987fff640 1 -- 172.21.15.114:0/1321248586 shutdown_connections 2024-05-27T08:00:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.568+0000 7f1987fff640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f19700b2950 0x7f19700b4e60 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.568+0000 7f1987fff640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1988109ef0 0x7f1988075a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.568+0000 7f1987fff640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1988109a50 0x7f1988075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.568+0000 7f1987fff640 1 --2- 172.21.15.114:0/1321248586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19881041b0 0x7f1988074fc0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:00:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.568+0000 7f1987fff640 1 -- 172.21.15.114:0/1321248586 >> 172.21.15.114:0/1321248586 conn(0x7f19880ff0e0 msgr2=0x7f198810f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:00:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.568+0000 7f1987fff640 1 -- 172.21.15.114:0/1321248586 shutdown_connections 2024-05-27T08:00:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:00:53.568+0000 7f1987fff640 1 -- 172.21.15.114:0/1321248586 wait complete. 2024-05-27T08:00:56.096 DEBUG:tasks.check_counter:Getting stats from a 2024-05-27T08:00:56.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump 2024-05-27T08:00:56.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:53 smithi114 ceph-mon[32038]: pgmap v733: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 34 KiB/s wr, 3 op/s 2024-05-27T08:00:56.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:53 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2057093489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T08:00:56.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:53 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/4127074315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T08:00:56.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:53 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2136444503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T08:00:56.098 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:54 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1321248586' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T08:00:56.098 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:55 smithi114 ceph-mon[32038]: pgmap v734: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 34 KiB/s wr, 3 op/s 2024-05-27T08:00:56.346 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:00:57.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:57 smithi114 ceph-mon[32038]: pgmap v735: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 36 KiB/s wr, 3 op/s 2024-05-27T08:00:57.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:57 smithi163 ceph-mon[36451]: pgmap v735: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 36 KiB/s wr, 3 op/s 2024-05-27T08:00:57.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:57 smithi186 ceph-mon[36647]: pgmap v735: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 36 KiB/s wr, 3 op/s 2024-05-27T08:00:58.064 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:57 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-05-27T08:00:58.065 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:57 smithi114 systemd[1]: Stopped Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:00:58.065 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:57 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.651s CPU time. 2024-05-27T08:00:58.065 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:57 smithi114 systemd[1]: Starting Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:00:58.085 INFO:teuthology.orchestra.run.smithi114.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-05-27T08:00:58.813 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-27T08:00:58.814 ERROR:teuthology.run_tasks:Manager failed: check-counter Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/check_counter.py", line 84, in end proc = manager.admin_socket(daemon_type, daemon_id, ["perf", "dump"]) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1893, in admin_socket return shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi114 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump' 2024-05-27T08:00:58.823 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-27T08:00:58.834 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-27T08:00:58.834 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T08:00:58.834 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T08:00:58.862 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T08:00:58.862 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T08:00:58.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T08:00:59.055 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:59 smithi114 podman[137485]: 2024-05-27 08:00:58.933824407 +0000 UTC m=+0.017390082 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:00:59.308 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:59 smithi114 podman[137485]: 2024-05-27 08:00:59.053757384 +0000 UTC m=+0.137323046 container create cf7e4f572f6d7ab27b2837942afab27e4be0cf96973ef6d34c951659deb185b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-27T08:00:59.493 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:00:59.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:59 smithi114 ceph-mon[32038]: pgmap v736: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 33 KiB/s wr, 2 op/s 2024-05-27T08:00:59.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:59 smithi114 ceph-mon[32038]: daemon mds.d restarted 2024-05-27T08:00:59.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:59 smithi114 ceph-mon[32038]: osdmap e171: 12 total, 12 up, 12 in 2024-05-27T08:00:59.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:59 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6836/4052324855,v1:172.21.15.114:6837/4052324855] up:boot 2024-05-27T08:00:59.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:59 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:59.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:00:59 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:00:59.619 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:59 smithi114 podman[137485]: 2024-05-27 08:00:59.356691305 +0000 UTC m=+0.440256981 container init cf7e4f572f6d7ab27b2837942afab27e4be0cf96973ef6d34c951659deb185b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-27T08:00:59.620 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:59 smithi114 podman[137485]: 2024-05-27 08:00:59.35946246 +0000 UTC m=+0.443028129 container start cf7e4f572f6d7ab27b2837942afab27e4be0cf96973ef6d34c951659deb185b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , ceph=True) 2024-05-27T08:00:59.620 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:59 smithi114 ceph-mds[137555]: -- 172.21.15.114:0/1899941626 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bfd28eda40 con 0x55bfd2935600 2024-05-27T08:00:59.620 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:59 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: starting mds.d at 2024-05-27T08:00:59.620 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:59 smithi114 bash[137485]: cf7e4f572f6d7ab27b2837942afab27e4be0cf96973ef6d34c951659deb185b6 2024-05-27T08:00:59.620 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:00:59 smithi114 systemd[1]: Started Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:00:59.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:59 smithi163 ceph-mon[36451]: pgmap v736: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 33 KiB/s wr, 2 op/s 2024-05-27T08:00:59.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:59 smithi163 ceph-mon[36451]: daemon mds.d restarted 2024-05-27T08:00:59.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:59 smithi163 ceph-mon[36451]: osdmap e171: 12 total, 12 up, 12 in 2024-05-27T08:00:59.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:59 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6836/4052324855,v1:172.21.15.114:6837/4052324855] up:boot 2024-05-27T08:00:59.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:59 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:59.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:00:59 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:00:59.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:59 smithi186 ceph-mon[36647]: pgmap v736: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 33 KiB/s wr, 2 op/s 2024-05-27T08:00:59.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:59 smithi186 ceph-mon[36647]: daemon mds.d restarted 2024-05-27T08:00:59.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:59 smithi186 ceph-mon[36647]: osdmap e171: 12 total, 12 up, 12 in 2024-05-27T08:00:59.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:59 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6836/4052324855,v1:172.21.15.114:6837/4052324855] up:boot 2024-05-27T08:00:59.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:59 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:00:59.981 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:00:59 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:01:00.308 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fdce7df6700 time 2024-05-27T08:01:00.056766+0000 2024-05-27T08:01:00.309 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:01:00.309 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:00.309 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fdcfa63bba6] 2024-05-27T08:01:00.309 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fdcfa63bdd1] 2024-05-27T08:01:00.309 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bfd001823d] 2024-05-27T08:01:00.309 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bfd0003971] 2024-05-27T08:01:00.310 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bfd0001fa0] 2024-05-27T08:01:00.310 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bfd0002651] 2024-05-27T08:01:00.310 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bfcff97347] 2024-05-27T08:01:00.310 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bfcfc9387e] 2024-05-27T08:01:00.310 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bfcfc98e94] 2024-05-27T08:01:00.310 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bfcfca3969] 2024-05-27T08:01:00.311 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bfd003c48c] 2024-05-27T08:01:00.311 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bfd00459cb] 2024-05-27T08:01:00.311 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 13: (Thread::entry_wrapper()+0x43) [0x7fdcfa60a72d] 2024-05-27T08:01:00.311 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 14: (Thread::_entry_func(void*)+0xd) [0x7fdcfa60a749] 2024-05-27T08:01:00.311 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdcf91ae1ca] 2024-05-27T08:01:00.311 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 16: clone() 2024-05-27T08:01:00.311 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: *** Caught signal (Aborted) ** 2024-05-27T08:01:00.311 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: in thread 7fdce7df6700 thread_name:md_log_replay 2024-05-27T08:01:00.312 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.057+0000 7fdce7df6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fdce7df6700 time 2024-05-27T08:01:00.056766+0000 2024-05-27T08:01:00.312 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:01:00.312 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.312 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:00.312 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fdcfa63bba6] 2024-05-27T08:01:00.312 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fdcfa63bdd1] 2024-05-27T08:01:00.312 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bfd001823d] 2024-05-27T08:01:00.312 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bfd0003971] 2024-05-27T08:01:00.313 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bfd0001fa0] 2024-05-27T08:01:00.313 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bfd0002651] 2024-05-27T08:01:00.313 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bfcff97347] 2024-05-27T08:01:00.313 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bfcfc9387e] 2024-05-27T08:01:00.313 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bfcfc98e94] 2024-05-27T08:01:00.313 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bfcfca3969] 2024-05-27T08:01:00.313 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bfd003c48c] 2024-05-27T08:01:00.313 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bfd00459cb] 2024-05-27T08:01:00.314 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 13: (Thread::entry_wrapper()+0x43) [0x7fdcfa60a72d] 2024-05-27T08:01:00.314 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 14: (Thread::_entry_func(void*)+0xd) [0x7fdcfa60a749] 2024-05-27T08:01:00.314 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdcf91ae1ca] 2024-05-27T08:01:00.314 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 16: clone() 2024-05-27T08:01:00.314 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.314 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:00.314 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdcf91b8d20] 2024-05-27T08:01:00.315 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2: gsignal() 2024-05-27T08:01:00.315 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 3: abort() 2024-05-27T08:01:00.315 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fdcfa63bced] 2024-05-27T08:01:00.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fdcfa63bdd1] 2024-05-27T08:01:00.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bfd001823d] 2024-05-27T08:01:00.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bfd0003971] 2024-05-27T08:01:00.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bfd0001fa0] 2024-05-27T08:01:00.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bfd0002651] 2024-05-27T08:01:00.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bfcff97347] 2024-05-27T08:01:00.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bfcfc9387e] 2024-05-27T08:01:00.316 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bfcfc98e94] 2024-05-27T08:01:00.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bfcfca3969] 2024-05-27T08:01:00.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bfd003c48c] 2024-05-27T08:01:00.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bfd00459cb] 2024-05-27T08:01:00.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 16: (Thread::entry_wrapper()+0x43) [0x7fdcfa60a72d] 2024-05-27T08:01:00.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 17: (Thread::_entry_func(void*)+0xd) [0x7fdcfa60a749] 2024-05-27T08:01:00.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdcf91ae1ca] 2024-05-27T08:01:00.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 19: clone() 2024-05-27T08:01:00.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.060+0000 7fdce7df6700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:01:00.317 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: in thread 7fdce7df6700 thread_name:md_log_replay 2024-05-27T08:01:00.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:00.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdcf91b8d20] 2024-05-27T08:01:00.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2: gsignal() 2024-05-27T08:01:00.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 3: abort() 2024-05-27T08:01:00.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fdcfa63bced] 2024-05-27T08:01:00.318 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fdcfa63bdd1] 2024-05-27T08:01:00.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bfd001823d] 2024-05-27T08:01:00.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bfd0003971] 2024-05-27T08:01:00.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bfd0001fa0] 2024-05-27T08:01:00.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bfd0002651] 2024-05-27T08:01:00.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bfcff97347] 2024-05-27T08:01:00.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bfcfc9387e] 2024-05-27T08:01:00.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bfcfc98e94] 2024-05-27T08:01:00.319 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bfcfca3969] 2024-05-27T08:01:00.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bfd003c48c] 2024-05-27T08:01:00.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bfd00459cb] 2024-05-27T08:01:00.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 16: (Thread::entry_wrapper()+0x43) [0x7fdcfa60a72d] 2024-05-27T08:01:00.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 17: (Thread::_entry_func(void*)+0xd) [0x7fdcfa60a749] 2024-05-27T08:01:00.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdcf91ae1ca] 2024-05-27T08:01:00.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 19: clone() 2024-05-27T08:01:00.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:01:00.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.320 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: -1> 2024-05-27T08:01:00.057+0000 7fdce7df6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fdce7df6700 time 2024-05-27T08:01:00.056766+0000 2024-05-27T08:01:00.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:01:00.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:00.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fdcfa63bba6] 2024-05-27T08:01:00.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fdcfa63bdd1] 2024-05-27T08:01:00.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bfd001823d] 2024-05-27T08:01:00.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bfd0003971] 2024-05-27T08:01:00.321 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bfd0001fa0] 2024-05-27T08:01:00.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bfd0002651] 2024-05-27T08:01:00.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bfcff97347] 2024-05-27T08:01:00.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bfcfc9387e] 2024-05-27T08:01:00.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bfcfc98e94] 2024-05-27T08:01:00.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bfcfca3969] 2024-05-27T08:01:00.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bfd003c48c] 2024-05-27T08:01:00.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bfd00459cb] 2024-05-27T08:01:00.322 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 13: (Thread::entry_wrapper()+0x43) [0x7fdcfa60a72d] 2024-05-27T08:01:00.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 14: (Thread::_entry_func(void*)+0xd) [0x7fdcfa60a749] 2024-05-27T08:01:00.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdcf91ae1ca] 2024-05-27T08:01:00.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 16: clone() 2024-05-27T08:01:00.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 0> 2024-05-27T08:01:00.060+0000 7fdce7df6700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:01:00.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: in thread 7fdce7df6700 thread_name:md_log_replay 2024-05-27T08:01:00.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.323 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:00.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdcf91b8d20] 2024-05-27T08:01:00.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2: gsignal() 2024-05-27T08:01:00.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 3: abort() 2024-05-27T08:01:00.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fdcfa63bced] 2024-05-27T08:01:00.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fdcfa63bdd1] 2024-05-27T08:01:00.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bfd001823d] 2024-05-27T08:01:00.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bfd0003971] 2024-05-27T08:01:00.324 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bfd0001fa0] 2024-05-27T08:01:00.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bfd0002651] 2024-05-27T08:01:00.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bfcff97347] 2024-05-27T08:01:00.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bfcfc9387e] 2024-05-27T08:01:00.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bfcfc98e94] 2024-05-27T08:01:00.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bfcfca3969] 2024-05-27T08:01:00.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bfd003c48c] 2024-05-27T08:01:00.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bfd00459cb] 2024-05-27T08:01:00.325 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 16: (Thread::entry_wrapper()+0x43) [0x7fdcfa60a72d] 2024-05-27T08:01:00.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 17: (Thread::_entry_func(void*)+0xd) [0x7fdcfa60a749] 2024-05-27T08:01:00.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdcf91ae1ca] 2024-05-27T08:01:00.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 19: clone() 2024-05-27T08:01:00.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:01:00.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: -1> 2024-05-27T08:01:00.057+0000 7fdce7df6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fdce7df6700 time 2024-05-27T08:01:00.056766+0000 2024-05-27T08:01:00.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:01:00.326 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:00.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fdcfa63bba6] 2024-05-27T08:01:00.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fdcfa63bdd1] 2024-05-27T08:01:00.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bfd001823d] 2024-05-27T08:01:00.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bfd0003971] 2024-05-27T08:01:00.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bfd0001fa0] 2024-05-27T08:01:00.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bfd0002651] 2024-05-27T08:01:00.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bfcff97347] 2024-05-27T08:01:00.327 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bfcfc9387e] 2024-05-27T08:01:00.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bfcfc98e94] 2024-05-27T08:01:00.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bfcfca3969] 2024-05-27T08:01:00.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bfd003c48c] 2024-05-27T08:01:00.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bfd00459cb] 2024-05-27T08:01:00.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 13: (Thread::entry_wrapper()+0x43) [0x7fdcfa60a72d] 2024-05-27T08:01:00.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 14: (Thread::_entry_func(void*)+0xd) [0x7fdcfa60a749] 2024-05-27T08:01:00.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdcf91ae1ca] 2024-05-27T08:01:00.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 16: clone() 2024-05-27T08:01:00.328 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 0> 2024-05-27T08:01:00.060+0000 7fdce7df6700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:01:00.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: in thread 7fdce7df6700 thread_name:md_log_replay 2024-05-27T08:01:00.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:00.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdcf91b8d20] 2024-05-27T08:01:00.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2: gsignal() 2024-05-27T08:01:00.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 3: abort() 2024-05-27T08:01:00.329 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fdcfa63bced] 2024-05-27T08:01:00.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fdcfa63bdd1] 2024-05-27T08:01:00.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bfd001823d] 2024-05-27T08:01:00.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bfd0003971] 2024-05-27T08:01:00.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bfd0001fa0] 2024-05-27T08:01:00.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bfd0002651] 2024-05-27T08:01:00.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bfcff97347] 2024-05-27T08:01:00.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bfcfc9387e] 2024-05-27T08:01:00.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bfcfc98e94] 2024-05-27T08:01:00.330 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bfcfca3969] 2024-05-27T08:01:00.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bfd003c48c] 2024-05-27T08:01:00.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bfd00459cb] 2024-05-27T08:01:00.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 16: (Thread::entry_wrapper()+0x43) [0x7fdcfa60a72d] 2024-05-27T08:01:00.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 17: (Thread::_entry_func(void*)+0xd) [0x7fdcfa60a749] 2024-05-27T08:01:00.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdcf91ae1ca] 2024-05-27T08:01:00.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 19: clone() 2024-05-27T08:01:00.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:01:00.331 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[137537]: 2024-05-27T08:01:00.663 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 podman[137632]: 2024-05-27 08:01:00.349397867 +0000 UTC m=+0.019045401 container died cf7e4f572f6d7ab27b2837942afab27e4be0cf96973ef6d34c951659deb185b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True) 2024-05-27T08:01:01.308 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 podman[137632]: 2024-05-27 08:01:00.992958675 +0000 UTC m=+0.662606191 container remove cf7e4f572f6d7ab27b2837942afab27e4be0cf96973ef6d34c951659deb185b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T08:01:01.308 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:00 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T08:01:01.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:01 smithi163 ceph-mon[36451]: pgmap v738: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 28 KiB/s wr, 2 op/s 2024-05-27T08:01:01.795 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:01 smithi114 ceph-mon[32038]: pgmap v738: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 28 KiB/s wr, 2 op/s 2024-05-27T08:01:01.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:01 smithi186 ceph-mon[36647]: pgmap v738: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 28 KiB/s wr, 2 op/s 2024-05-27T08:01:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.082+0000 7f45d56a3700 1 -- 172.21.15.114:0/2323827527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45d0108ef0 msgr2=0x7f45d0109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.082+0000 7f45d56a3700 1 --2- 172.21.15.114:0/2323827527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45d0108ef0 0x7f45d0109300 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f45b80099c0 tx=0x7f45b8009cd0 comp rx=0 tx=0).stop 2024-05-27T08:01:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.082+0000 7f45d56a3700 1 -- 172.21.15.114:0/2323827527 shutdown_connections 2024-05-27T08:01:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.082+0000 7f45d56a3700 1 --2- 172.21.15.114:0/2323827527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d0109840 0x7f45d01111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.082+0000 7f45d56a3700 1 --2- 172.21.15.114:0/2323827527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d01045c0 0x7f45d01049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.082+0000 7f45d56a3700 1 --2- 172.21.15.114:0/2323827527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45d0108ef0 0x7f45d0109300 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.082+0000 7f45d56a3700 1 -- 172.21.15.114:0/2323827527 >> 172.21.15.114:0/2323827527 conn(0x7f45d00ffe90 msgr2=0x7f45d01022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.082+0000 7f45d56a3700 1 -- 172.21.15.114:0/2323827527 shutdown_connections 2024-05-27T08:01:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.082+0000 7f45d56a3700 1 -- 172.21.15.114:0/2323827527 wait complete. 2024-05-27T08:01:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.083+0000 7f45d56a3700 1 Processor -- start 2024-05-27T08:01:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.083+0000 7f45d56a3700 1 -- start start 2024-05-27T08:01:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.083+0000 7f45d56a3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d01045c0 0x7f45d0076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.083+0000 7f45d56a3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d0108ef0 0x7f45d0077100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.083+0000 7f45d56a3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45d0109840 0x7f45d0077640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.083+0000 7f45d56a3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f45d0114450 con 0x7f45d0109840 2024-05-27T08:01:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.083+0000 7f45d56a3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01142d0 con 0x7f45d01045c0 2024-05-27T08:01:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.083+0000 7f45d56a3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01145d0 con 0x7f45d0108ef0 2024-05-27T08:01:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45c7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d0108ef0 0x7f45d0077100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45cf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45d0109840 0x7f45d0077640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45c7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d0108ef0 0x7f45d0077100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:56508/0 (socket says 172.21.15.114:56508) 2024-05-27T08:01:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45c7fff700 1 -- 172.21.15.114:0/1473276363 learned_addr learned my addr 172.21.15.114:0/1473276363 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:01:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45ceffd700 1 --2- 172.21.15.114:0/1473276363 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d01045c0 0x7f45d0076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45cf7fe700 1 -- 172.21.15.114:0/1473276363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d0108ef0 msgr2=0x7f45d0077100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45cf7fe700 1 --2- 172.21.15.114:0/1473276363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d0108ef0 0x7f45d0077100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45cf7fe700 1 -- 172.21.15.114:0/1473276363 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d01045c0 msgr2=0x7f45d0076bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45cf7fe700 1 --2- 172.21.15.114:0/1473276363 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d01045c0 0x7f45d0076bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45cf7fe700 1 -- 172.21.15.114:0/1473276363 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d01a91d0 con 0x7f45d0109840 2024-05-27T08:01:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45cf7fe700 1 --2- 172.21.15.114:0/1473276363 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45d0109840 0x7f45d0077640 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f45c800cc50 tx=0x7f45c800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45ccff9700 1 -- 172.21.15.114:0/1473276363 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c8016020 con 0x7f45d0109840 2024-05-27T08:01:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45d56a3700 1 -- 172.21.15.114:0/1473276363 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d01a94c0 con 0x7f45d0109840 2024-05-27T08:01:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45ccff9700 1 -- 172.21.15.114:0/1473276363 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45c8012420 con 0x7f45d0109840 2024-05-27T08:01:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45ccff9700 1 -- 172.21.15.114:0/1473276363 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c80115a0 con 0x7f45d0109840 2024-05-27T08:01:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.084+0000 7f45d56a3700 1 -- 172.21.15.114:0/1473276363 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d01a9a00 con 0x7f45d0109840 2024-05-27T08:01:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.086+0000 7f45d56a3700 1 -- 172.21.15.114:0/1473276363 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d00512b0 con 0x7f45d0109840 2024-05-27T08:01:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.088+0000 7f45ccff9700 1 -- 172.21.15.114:0/1473276363 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f45c8019020 con 0x7f45d0109840 2024-05-27T08:01:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.089+0000 7f45ccff9700 1 --2- 172.21.15.114:0/1473276363 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f45bc0b29f0 0x7f45bc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.089+0000 7f45ceffd700 1 --2- 172.21.15.114:0/1473276363 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f45bc0b29f0 0x7f45bc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.089+0000 7f45ccff9700 1 -- 172.21.15.114:0/1473276363 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(171..171 src has 1..171) ==== 9466+0+0 (secure 0 0 0) 0x7f45c80e2af0 con 0x7f45d0109840 2024-05-27T08:01:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.089+0000 7f45ceffd700 1 --2- 172.21.15.114:0/1473276363 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f45bc0b29f0 0x7f45bc0b4f00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f45c0007be0 tx=0x7f45c00073d0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.093+0000 7f45ccff9700 1 -- 172.21.15.114:0/1473276363 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f45c8020020 con 0x7f45d0109840 2024-05-27T08:01:02.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.323+0000 7f45d56a3700 1 -- 172.21.15.114:0/1473276363 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f45d00780f0 con 0x7f45d0109840 2024-05-27T08:01:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.323+0000 7f45ccff9700 1 -- 172.21.15.114:0/1473276363 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v171) ==== 81+0+960 (secure 0 0 0) 0x7f45c80aa1d0 con 0x7f45d0109840 2024-05-27T08:01:02.326 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/1628485461 2024-05-28T07:39:44.126253+0000 2024-05-27T08:01:02.326 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/1761274168 2024-05-28T07:39:44.126253+0000 2024-05-27T08:01:02.326 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2574023732 2024-05-28T07:39:44.126253+0000 2024-05-27T08:01:02.326 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/2291864634 2024-05-28T07:39:44.126253+0000 2024-05-27T08:01:02.327 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/3782422776 2024-05-28T07:28:43.271481+0000 2024-05-27T08:01:02.327 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/3323915605 2024-05-28T07:28:43.271481+0000 2024-05-27T08:01:02.327 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2381281099 2024-05-28T07:39:44.126253+0000 2024-05-27T08:01:02.327 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/752776783 2024-05-28T07:28:43.271481+0000 2024-05-27T08:01:02.327 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2604585922 2024-05-28T07:28:16.331472+0000 2024-05-27T08:01:02.327 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2303561217 2024-05-28T07:28:16.331472+0000 2024-05-27T08:01:02.327 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/1568684596 2024-05-28T07:28:16.331472+0000 2024-05-27T08:01:02.327 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/2291864634 2024-05-28T07:39:44.126253+0000 2024-05-27T08:01:02.327 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/1305604861 2024-05-28T07:28:43.271481+0000 2024-05-27T08:01:02.328 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/1568684596 2024-05-28T07:28:16.331472+0000 2024-05-27T08:01:02.328 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/752776783 2024-05-28T07:28:43.271481+0000 2024-05-27T08:01:02.328 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2442312517 2024-05-28T07:28:16.331472+0000 2024-05-27T08:01:02.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.325+0000 7f45d56a3700 1 -- 172.21.15.114:0/1473276363 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f45bc0b29f0 msgr2=0x7f45bc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:02.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.325+0000 7f45d56a3700 1 --2- 172.21.15.114:0/1473276363 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f45bc0b29f0 0x7f45bc0b4f00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f45c0007be0 tx=0x7f45c00073d0 comp rx=0 tx=0).stop 2024-05-27T08:01:02.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.325+0000 7f45d56a3700 1 -- 172.21.15.114:0/1473276363 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45d0109840 msgr2=0x7f45d0077640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:02.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.325+0000 7f45d56a3700 1 --2- 172.21.15.114:0/1473276363 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45d0109840 0x7f45d0077640 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f45c800cc50 tx=0x7f45c800cf60 comp rx=0 tx=0).stop 2024-05-27T08:01:02.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.326+0000 7f45d56a3700 1 -- 172.21.15.114:0/1473276363 shutdown_connections 2024-05-27T08:01:02.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.326+0000 7f45d56a3700 1 --2- 172.21.15.114:0/1473276363 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45d0109840 0x7f45d0077640 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.326+0000 7f45d56a3700 1 --2- 172.21.15.114:0/1473276363 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f45bc0b29f0 0x7f45bc0b4f00 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.326+0000 7f45d56a3700 1 --2- 172.21.15.114:0/1473276363 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d01045c0 0x7f45d0076bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.326+0000 7f45d56a3700 1 --2- 172.21.15.114:0/1473276363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d0108ef0 0x7f45d0077100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.326+0000 7f45d56a3700 1 -- 172.21.15.114:0/1473276363 >> 172.21.15.114:0/1473276363 conn(0x7f45d00ffe90 msgr2=0x7f45d01022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.326+0000 7f45d56a3700 1 -- 172.21.15.114:0/1473276363 shutdown_connections 2024-05-27T08:01:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:02.326+0000 7f45d56a3700 1 -- 172.21.15.114:0/1473276363 wait complete. 2024-05-27T08:01:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:listed 16 entries 2024-05-27T08:01:02.340 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:02 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T08:01:02.340 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:02 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.613s CPU time. 2024-05-27T08:01:02.682 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:02 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1473276363' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T08:01:02.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:02 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1473276363' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T08:01:02.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:02 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1473276363' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T08:01:03.010 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-27T08:01:03.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T08:01:03.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-27T08:01:03.292 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-05-27T08:01:03.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T08:01:03.292 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-27T08:01:03.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T08:01:03.307 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-27T08:01:03.384 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-27T08:01:03.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T08:01:03.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T08:01:03.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T08:01:03.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-05-27T08:01:03.538 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-27T08:01:03.538 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-27T08:01:03.538 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T08:01:03.555 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056323,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=388 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248824k,nr_inodes=812206,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/94a76e44ce53f523d4fe8fad40ad2ac27ed4c2cc608ce5fe5b3a2995dc8f8432/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YC6M55CF2MGSQRTZWHZQRKQ62A:/var/lib/containers/storage/overlay/l/4IMGWJTEJYL2ATLQTC56C4XWQI,upperdir=/var/lib/containers/storage/overlay/94a76e44ce53f523d4fe8fad40ad2ac27ed4c2cc608ce5fe5b3a2995dc8f8432/diff,workdir=/var/lib/containers/storage/overlay/94a76e44ce53f523d4fe8fad40ad2ac27ed4c2cc608ce5fe5b3a2995dc8f8432/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/ee480c929f8bd9822f52aae45df3aa587b7cd5ed42aef098965da117c7a916c9/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/YC6M55CF2MGSQRTZWHZQRKQ62A:/var/lib/containers/storage/overlay/l/4IMGWJTEJYL2ATLQTC56C4XWQI,upperdir=/var/lib/containers/storage/overlay/ee480c929f8bd9822f52aae45df3aa587b7cd5ed42aef098965da117c7a916c9/diff,workdir=/var/lib/containers/storage/overlay/ee480c929f8bd9822f52aae45df3aa587b7cd5ed42aef098965da117c7a916c9/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248824k,nr_inodes=812206,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/673d5a0837d86807d06484440d3afe658f01c7941f261b9775444d243dfdd2e9/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YC6M55CF2MGSQRTZWHZQRKQ62A:/var/lib/containers/storage/overlay/l/4IMGWJTEJYL2ATLQTC56C4XWQI,upperdir=/var/lib/containers/storage/overlay/673d5a0837d86807d06484440d3afe658f01c7941f261b9775444d243dfdd2e9/diff,workdir=/var/lib/containers/storage/overlay/673d5a0837d86807d06484440d3afe658f01c7941f261b9775444d243dfdd2e9/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d927e3fc95c4122297cd265b8a5a04fb0962353c0313eeff6ec02547a8b2e34e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YC6M55CF2MGSQRTZWHZQRKQ62A:/var/lib/containers/storage/overlay/l/4IMGWJTEJYL2ATLQTC56C4XWQI,upperdir=/var/lib/containers/storage/overlay/d927e3fc95c4122297cd265b8a5a04fb0962353c0313eeff6ec02547a8b2e34e/diff,workdir=/var/lib/containers/storage/overlay/d927e3fc95c4122297cd265b8a5a04fb0962353c0313eeff6ec02547a8b2e34e/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/fa72f966b5c746878ccfc28b3538a12da909f4674f6b0a1016c10bfed8bbb628/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YC6M55CF2MGSQRTZWHZQRKQ62A:/var/lib/containers/storage/overlay/l/4IMGWJTEJYL2ATLQTC56C4XWQI,upperdir=/var/lib/containers/storage/overlay/fa72f966b5c746878ccfc28b3538a12da909f4674f6b0a1016c10bfed8bbb628/diff,workdir=/var/lib/containers/storage/overlay/fa72f966b5c746878ccfc28b3538a12da909f4674f6b0a1016c10bfed8bbb628/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/536e7f1f97386e3c5090df963fb0ae0fd0348dd0eadc2c2fcaa2e23cb4e5b53a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YC6M55CF2MGSQRTZWHZQRKQ62A:/var/lib/containers/storage/overlay/l/4IMGWJTEJYL2ATLQTC56C4XWQI,upperdir=/var/lib/containers/storage/overlay/536e7f1f97386e3c5090df963fb0ae0fd0348dd0eadc2c2fcaa2e23cb4e5b53a/diff,workdir=/var/lib/containers/storage/overlay/536e7f1f97386e3c5090df963fb0ae0fd0348dd0eadc2c2fcaa2e23cb4e5b53a/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/956275fd9eef98faf21ac8e53dd7b69d02e7d64518f82a276aef72e3fc9ee902/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/YC6M55CF2MGSQRTZWHZQRKQ62A:/var/lib/containers/storage/overlay/l/4IMGWJTEJYL2ATLQTC56C4XWQI,upperdir=/var/lib/containers/storage/overlay/956275fd9eef98faf21ac8e53dd7b69d02e7d64518f82a276aef72e3fc9ee902/diff,workdir=/var/lib/containers/storage/overlay/956275fd9eef98faf21ac8e53dd7b69d02e7d64518f82a276aef72e3fc9ee902/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 2024-05-27T08:01:03.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T08:01:03.556 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-27T08:01:03.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T08:01:03.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-05-27T08:01:03.712 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2024-05-27T08:01:03.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:01:03.714 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-27T08:01:03.725 INFO:tasks.cephadm:Teardown begin 2024-05-27T08:01:03.726 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-27T08:01:03.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T08:01:03.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:03 smithi114 ceph-mon[32038]: pgmap v739: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 28 KiB/s wr, 3 op/s 2024-05-27T08:01:03.966 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:01:03.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:03 smithi163 ceph-mon[36451]: pgmap v739: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 28 KiB/s wr, 3 op/s 2024-05-27T08:01:03.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:03 smithi186 ceph-mon[36647]: pgmap v739: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 28 KiB/s wr, 3 op/s 2024-05-27T08:01:05.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:05 smithi114 ceph-mon[32038]: pgmap v740: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 54 KiB/s wr, 20 op/s 2024-05-27T08:01:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.725+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/1543555155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf0108010 msgr2=0x7f6cf0108420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.725+0000 7f6cf4a4d700 1 --2- 172.21.15.114:0/1543555155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf0108010 0x7f6cf0108420 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40099c0 tx=0x7f6ce4009cd0 comp rx=0 tx=0).stop 2024-05-27T08:01:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.725+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/1543555155 shutdown_connections 2024-05-27T08:01:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.725+0000 7f6cf4a4d700 1 --2- 172.21.15.114:0/1543555155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf0108960 0x7f6cf01102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.725+0000 7f6cf4a4d700 1 --2- 172.21.15.114:0/1543555155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf0103c00 0x7f6cf0104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.725+0000 7f6cf4a4d700 1 --2- 172.21.15.114:0/1543555155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf0108010 0x7f6cf0108420 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:05.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.725+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/1543555155 >> 172.21.15.114:0/1543555155 conn(0x7f6cf00fe510 msgr2=0x7f6cf0100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:05.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.725+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/1543555155 shutdown_connections 2024-05-27T08:01:05.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.725+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/1543555155 wait complete. 2024-05-27T08:01:05.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.726+0000 7f6cf4a4d700 1 Processor -- start 2024-05-27T08:01:05.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.726+0000 7f6cf4a4d700 1 -- start start 2024-05-27T08:01:05.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cf4a4d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf0103c00 0x7f6cf019bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:05.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cf4a4d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf0108010 0x7f6cf019c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:05.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cf4a4d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf0108960 0x7f6cf019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:05.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cf4a4d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf0113590 con 0x7f6cf0103c00 2024-05-27T08:01:05.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cf4a4d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf0113410 con 0x7f6cf0108010 2024-05-27T08:01:05.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cf4a4d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf0113710 con 0x7f6cf0108960 2024-05-27T08:01:05.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cee59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf0103c00 0x7f6cf019bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:05.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cee59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf0103c00 0x7f6cf019bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42868/0 (socket says 172.21.15.114:42868) 2024-05-27T08:01:05.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cee59c700 1 -- 172.21.15.114:0/3011675065 learned_addr learned my addr 172.21.15.114:0/3011675065 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:01:05.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cedd9b700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf0108010 0x7f6cf019c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:05.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cee59c700 1 -- 172.21.15.114:0/3011675065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf0108960 msgr2=0x7f6cf019c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:05.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6ceed9d700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf0108960 0x7f6cf019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:05.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cee59c700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf0108960 0x7f6cf019c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:05.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cee59c700 1 -- 172.21.15.114:0/3011675065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf0108010 msgr2=0x7f6cf019c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:05.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cee59c700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf0108010 0x7f6cf019c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:05.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cee59c700 1 -- 172.21.15.114:0/3011675065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf01a7220 con 0x7f6cf0103c00 2024-05-27T08:01:05.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cedd9b700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf0108010 0x7f6cf019c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T08:01:05.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6ceed9d700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf0108960 0x7f6cf019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T08:01:05.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.727+0000 7f6cee59c700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf0103c00 0x7f6cf019bed0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8007f10 tx=0x7f6ce800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:05.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.728+0000 7f6cdf7fe700 1 -- 172.21.15.114:0/3011675065 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce800dc90 con 0x7f6cf0103c00 2024-05-27T08:01:05.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.728+0000 7f6cdf7fe700 1 -- 172.21.15.114:0/3011675065 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ce800de30 con 0x7f6cf0103c00 2024-05-27T08:01:05.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.728+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/3011675065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf01a7510 con 0x7f6cf0103c00 2024-05-27T08:01:05.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.728+0000 7f6cdf7fe700 1 -- 172.21.15.114:0/3011675065 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce80133f0 con 0x7f6cf0103c00 2024-05-27T08:01:05.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.728+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/3011675065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf01a7a50 con 0x7f6cf0103c00 2024-05-27T08:01:05.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.729+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/3011675065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf00512b0 con 0x7f6cf0103c00 2024-05-27T08:01:05.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.731+0000 7f6cdf7fe700 1 -- 172.21.15.114:0/3011675065 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f6ce801a020 con 0x7f6cf0103c00 2024-05-27T08:01:05.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.732+0000 7f6cdf7fe700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6cd80b2a70 0x7f6cd80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:05.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.732+0000 7f6cedd9b700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6cd80b2a70 0x7f6cd80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:05.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.732+0000 7f6cdf7fe700 1 -- 172.21.15.114:0/3011675065 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(171..171 src has 1..171) ==== 9466+0+0 (secure 0 0 0) 0x7f6ce80e2820 con 0x7f6cf0103c00 2024-05-27T08:01:05.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.732+0000 7f6cedd9b700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6cd80b2a70 0x7f6cd80b4f80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4006010 tx=0x7f6ce4005e80 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:05.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.735+0000 7f6cdf7fe700 1 -- 172.21.15.114:0/3011675065 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6ce8010070 con 0x7f6cf0103c00 2024-05-27T08:01:05.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:05 smithi163 ceph-mon[36451]: pgmap v740: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 54 KiB/s wr, 20 op/s 2024-05-27T08:01:05.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:05 smithi186 ceph-mon[36647]: pgmap v740: 129 pgs: 129 active+clean; 372 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 54 KiB/s wr, 20 op/s 2024-05-27T08:01:05.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.989+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/3011675065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6cf0102e50 con 0x7f6cf0103c00 2024-05-27T08:01:05.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.990+0000 7f6cdf7fe700 1 -- 172.21.15.114:0/3011675065 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+4381 (secure 0 0 0) 0x7f6ce80a9e50 con 0x7f6cf0103c00 2024-05-27T08:01:05.993 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T08:01:05.993 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":121,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T08:00:59.492292+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_20058":{"gid":20058,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6837/4052324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":4052324855},{"type":"v1","addr":"172.21.15.114:6837","nonce":4052324855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T08:01:05.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.992+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/3011675065 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6cd80b2a70 msgr2=0x7f6cd80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:05.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.992+0000 7f6cf4a4d700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6cd80b2a70 0x7f6cd80b4f80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4006010 tx=0x7f6ce4005e80 comp rx=0 tx=0).stop 2024-05-27T08:01:05.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.992+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/3011675065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf0103c00 msgr2=0x7f6cf019bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:05.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.992+0000 7f6cf4a4d700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf0103c00 0x7f6cf019bed0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8007f10 tx=0x7f6ce800d3b0 comp rx=0 tx=0).stop 2024-05-27T08:01:05.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.992+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/3011675065 shutdown_connections 2024-05-27T08:01:05.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.992+0000 7f6cf4a4d700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf0108960 0x7f6cf019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:05.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.992+0000 7f6cf4a4d700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf0103c00 0x7f6cf019bed0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:05.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.992+0000 7f6cf4a4d700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f6cd80b2a70 0x7f6cd80b4f80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:05.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.992+0000 7f6cf4a4d700 1 --2- 172.21.15.114:0/3011675065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf0108010 0x7f6cf019c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:05.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.992+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/3011675065 >> 172.21.15.114:0/3011675065 conn(0x7f6cf00fe510 msgr2=0x7f6cf01098c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:05.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.992+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/3011675065 shutdown_connections 2024-05-27T08:01:05.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:05.992+0000 7f6cf4a4d700 1 -- 172.21.15.114:0/3011675065 wait complete. 2024-05-27T08:01:05.999 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 121 2024-05-27T08:01:06.637 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:06 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3011675065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T08:01:06.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:06 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3011675065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T08:01:06.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T08:01:06.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:06 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3011675065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T08:01:07.101 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:01:07.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:07 smithi114 ceph-mon[32038]: pgmap v741: 129 pgs: 129 active+clean; 375 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 339 KiB/s wr, 24 op/s 2024-05-27T08:01:07.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:07 smithi163 ceph-mon[36451]: pgmap v741: 129 pgs: 129 active+clean; 375 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 339 KiB/s wr, 24 op/s 2024-05-27T08:01:07.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:07 smithi186 ceph-mon[36647]: pgmap v741: 129 pgs: 129 active+clean; 375 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 339 KiB/s wr, 24 op/s 2024-05-27T08:01:08.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:08 smithi114 ceph-mon[32038]: osdmap e172: 12 total, 12 up, 12 in 2024-05-27T08:01:08.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:08 smithi163 ceph-mon[36451]: osdmap e172: 12 total, 12 up, 12 in 2024-05-27T08:01:08.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:08 smithi186 ceph-mon[36647]: osdmap e172: 12 total, 12 up, 12 in 2024-05-27T08:01:09.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.259+0000 7eff03f1f700 1 -- 172.21.15.114:0/2739398477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efefc10a920 msgr2=0x7efefc10ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:09.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.259+0000 7eff03f1f700 1 --2- 172.21.15.114:0/2739398477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efefc10a920 0x7efefc10ad30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7efef400b0a0 tx=0x7efef400b3b0 comp rx=0 tx=0).stop 2024-05-27T08:01:09.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.261+0000 7eff03f1f700 1 -- 172.21.15.114:0/2739398477 shutdown_connections 2024-05-27T08:01:09.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.261+0000 7eff03f1f700 1 --2- 172.21.15.114:0/2739398477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc11d920 0x7efefc11fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:09.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.261+0000 7eff03f1f700 1 --2- 172.21.15.114:0/2739398477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efefc10a920 0x7efefc10ad30 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:09.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.261+0000 7eff03f1f700 1 --2- 172.21.15.114:0/2739398477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efefc10b270 0x7efefc11d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:09.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.261+0000 7eff03f1f700 1 -- 172.21.15.114:0/2739398477 >> 172.21.15.114:0/2739398477 conn(0x7efefc06f4c0 msgr2=0x7efefc06f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:09.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.268+0000 7eff03f1f700 1 -- 172.21.15.114:0/2739398477 shutdown_connections 2024-05-27T08:01:09.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.269+0000 7eff03f1f700 1 -- 172.21.15.114:0/2739398477 wait complete. 2024-05-27T08:01:09.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.270+0000 7eff03f1f700 1 Processor -- start 2024-05-27T08:01:09.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.270+0000 7eff03f1f700 1 -- start start 2024-05-27T08:01:09.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.270+0000 7eff03f1f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efefc10a920 0x7efefc11d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:09.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.270+0000 7eff03f1f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc10b270 0x7efefc1174c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:09.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.270+0000 7eff03f1f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efefc11d920 0x7efefc117b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:09.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.270+0000 7eff03f1f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1134c0 con 0x7efefc10b270 2024-05-27T08:01:09.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.270+0000 7eff03f1f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efefc113340 con 0x7efefc10a920 2024-05-27T08:01:09.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.270+0000 7eff03f1f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efefc113640 con 0x7efefc11d920 2024-05-27T08:01:09.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.274+0000 7eff01cbb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efefc10a920 0x7efefc11d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:09.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.274+0000 7eff01cbb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efefc10a920 0x7efefc11d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.114:60250/0 (socket says 172.21.15.114:60250) 2024-05-27T08:01:09.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.274+0000 7eff01cbb700 1 -- 172.21.15.114:0/2696581936 learned_addr learned my addr 172.21.15.114:0/2696581936 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:01:09.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.274+0000 7eff024bc700 1 --2- 172.21.15.114:0/2696581936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efefc11d920 0x7efefc117b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:09.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.277+0000 7eff01cbb700 1 -- 172.21.15.114:0/2696581936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efefc11d920 msgr2=0x7efefc117b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:09.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.277+0000 7eff01cbb700 1 --2- 172.21.15.114:0/2696581936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efefc11d920 0x7efefc117b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:09.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.277+0000 7eff01cbb700 1 -- 172.21.15.114:0/2696581936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc10b270 msgr2=0x7efefc1174c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:09.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.277+0000 7eff01cbb700 1 --2- 172.21.15.114:0/2696581936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc10b270 0x7efefc1174c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:09.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.277+0000 7eff01cbb700 1 -- 172.21.15.114:0/2696581936 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efefc1be7e0 con 0x7efefc10a920 2024-05-27T08:01:09.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.285+0000 7eff01cbb700 1 --2- 172.21.15.114:0/2696581936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efefc10a920 0x7efefc11d3a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efef4009bd0 tx=0x7efef400ce50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:09.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.286+0000 7efef2ffd700 1 -- 172.21.15.114:0/2696581936 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4021070 con 0x7efefc10a920 2024-05-27T08:01:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.286+0000 7efef2ffd700 1 -- 172.21.15.114:0/2696581936 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efef4007c80 con 0x7efefc10a920 2024-05-27T08:01:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.286+0000 7efef2ffd700 1 -- 172.21.15.114:0/2696581936 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef401d450 con 0x7efefc10a920 2024-05-27T08:01:09.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.287+0000 7eff03f1f700 1 -- 172.21.15.114:0/2696581936 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efefc1be9b0 con 0x7efefc10a920 2024-05-27T08:01:09.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.289+0000 7eff03f1f700 1 -- 172.21.15.114:0/2696581936 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efefc1beea0 con 0x7efefc10a920 2024-05-27T08:01:09.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.295+0000 7efef0ff9700 1 -- 172.21.15.114:0/2696581936 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee4005330 con 0x7efefc10a920 2024-05-27T08:01:09.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.301+0000 7efef2ffd700 1 -- 172.21.15.114:0/2696581936 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7efef4003a00 con 0x7efefc10a920 2024-05-27T08:01:09.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.302+0000 7efef2ffd700 1 --2- 172.21.15.114:0/2696581936 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efee80b2c00 0x7efee80b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:09.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.302+0000 7efef2ffd700 1 -- 172.21.15.114:0/2696581936 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 9642+0+0 (secure 0 0 0) 0x7efef400f020 con 0x7efefc10a920 2024-05-27T08:01:09.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.309+0000 7efef2ffd700 1 -- 172.21.15.114:0/2696581936 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7efef402a020 con 0x7efefc10a920 2024-05-27T08:01:09.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.309+0000 7eff014ba700 1 --2- 172.21.15.114:0/2696581936 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efee80b2c00 0x7efee80b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:09.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.326+0000 7eff014ba700 1 --2- 172.21.15.114:0/2696581936 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efee80b2c00 0x7efee80b5110 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7efefc118c80 tx=0x7efef8005c80 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:09.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.801+0000 7efef0ff9700 1 -- 172.21.15.114:0/2696581936 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efee4005620 con 0x7efefc10a920 2024-05-27T08:01:09.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.804+0000 7efef2ffd700 1 -- 172.21.15.114:0/2696581936 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+4381 (secure 0 0 0) 0x7efef40abb60 con 0x7efefc10a920 2024-05-27T08:01:09.807 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T08:01:09.807 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14511,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/807487035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":807487035},{"type":"v1","addr":"172.21.15.114:6835","nonce":807487035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":121,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:36:06.848027+0000","modified":"2024-05-27T08:00:59.492292+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_20058":{"gid":20058,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6837/4052324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":4052324855},{"type":"v1","addr":"172.21.15.114:6837","nonce":4052324855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6839/4115872608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":4115872608},{"type":"v1","addr":"172.21.15.186:6839","nonce":4115872608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24463,"qdb_cluster":[24463]},"id":1}]} 2024-05-27T08:01:09.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.810+0000 7efef0ff9700 1 -- 172.21.15.114:0/2696581936 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efee80b2c00 msgr2=0x7efee80b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:09.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.810+0000 7efef0ff9700 1 --2- 172.21.15.114:0/2696581936 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efee80b2c00 0x7efee80b5110 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7efefc118c80 tx=0x7efef8005c80 comp rx=0 tx=0).stop 2024-05-27T08:01:09.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.811+0000 7efef0ff9700 1 -- 172.21.15.114:0/2696581936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efefc10a920 msgr2=0x7efefc11d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:09.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.811+0000 7efef0ff9700 1 --2- 172.21.15.114:0/2696581936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efefc10a920 0x7efefc11d3a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efef4009bd0 tx=0x7efef400ce50 comp rx=0 tx=0).stop 2024-05-27T08:01:09.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.813+0000 7efef0ff9700 1 -- 172.21.15.114:0/2696581936 shutdown_connections 2024-05-27T08:01:09.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.813+0000 7efef0ff9700 1 --2- 172.21.15.114:0/2696581936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efefc11d920 0x7efefc117b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:09.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.813+0000 7efef0ff9700 1 --2- 172.21.15.114:0/2696581936 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7efee80b2c00 0x7efee80b5110 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:09.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.813+0000 7efef0ff9700 1 --2- 172.21.15.114:0/2696581936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efefc10a920 0x7efefc11d3a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:09.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.813+0000 7efef0ff9700 1 --2- 172.21.15.114:0/2696581936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc10b270 0x7efefc1174c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:09.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.813+0000 7efef0ff9700 1 -- 172.21.15.114:0/2696581936 >> 172.21.15.114:0/2696581936 conn(0x7efefc06f4c0 msgr2=0x7efefc11dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:09.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.820+0000 7efef0ff9700 1 -- 172.21.15.114:0/2696581936 shutdown_connections 2024-05-27T08:01:09.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:09.821+0000 7efef0ff9700 1 -- 172.21.15.114:0/2696581936 wait complete. 2024-05-27T08:01:09.833 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 121 2024-05-27T08:01:09.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:09 smithi163 ceph-mon[36451]: pgmap v743: 129 pgs: 4 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 110 active+clean; 375 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 379 KiB/s wr, 26 op/s 2024-05-27T08:01:09.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:09 smithi186 ceph-mon[36647]: pgmap v743: 129 pgs: 4 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 110 active+clean; 375 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 379 KiB/s wr, 26 op/s 2024-05-27T08:01:10.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:09 smithi114 ceph-mon[32038]: pgmap v743: 129 pgs: 4 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 110 active+clean; 375 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 379 KiB/s wr, 26 op/s 2024-05-27T08:01:10.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T08:01:10.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:10 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/2696581936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T08:01:10.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:10 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/2696581936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T08:01:11.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:10 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/2696581936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T08:01:11.244 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:01:11.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:11 smithi114 ceph-mon[32038]: pgmap v744: 129 pgs: 10 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 84 active+clean; 374 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 337 KiB/s wr, 24 op/s 2024-05-27T08:01:11.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:11 smithi163 ceph-mon[36451]: pgmap v744: 129 pgs: 10 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 84 active+clean; 374 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 337 KiB/s wr, 24 op/s 2024-05-27T08:01:11.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:11 smithi186 ceph-mon[36647]: pgmap v744: 129 pgs: 10 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 84 active+clean; 374 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 337 KiB/s wr, 24 op/s 2024-05-27T08:01:12.653 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:12 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-05-27T08:01:12.653 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:12 smithi114 systemd[1]: Stopped Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:01:12.653 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:12 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.613s CPU time. 2024-05-27T08:01:12.653 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:12 smithi114 systemd[1]: Starting Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:01:13.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.463+0000 7fad7e278700 1 -- 172.21.15.114:0/698675372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad78105450 msgr2=0x7fad78105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:13.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.463+0000 7fad76ffd700 1 -- 172.21.15.114:0/698675372 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c004230 con 0x7fad78105450 2024-05-27T08:01:13.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.463+0000 7fad7e278700 1 --2- 172.21.15.114:0/698675372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad78105450 0x7fad78105860 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009990 tx=0x7fad6c009ca0 comp rx=0 tx=0).stop 2024-05-27T08:01:13.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.464+0000 7fad7e278700 1 -- 172.21.15.114:0/698675372 shutdown_connections 2024-05-27T08:01:13.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.464+0000 7fad7e278700 1 --2- 172.21.15.114:0/698675372 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad7810e8c0 0x7fad78110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:13.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.464+0000 7fad7e278700 1 --2- 172.21.15.114:0/698675372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad78105450 0x7fad78105860 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:13.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.465+0000 7fad7e278700 1 --2- 172.21.15.114:0/698675372 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad78105da0 0x7fad7810e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:13.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.465+0000 7fad7e278700 1 -- 172.21.15.114:0/698675372 >> 172.21.15.114:0/698675372 conn(0x7fad780fedb0 msgr2=0x7fad781011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:13.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.465+0000 7fad7e278700 1 -- 172.21.15.114:0/698675372 shutdown_connections 2024-05-27T08:01:13.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.466+0000 7fad7e278700 1 -- 172.21.15.114:0/698675372 wait complete. 2024-05-27T08:01:13.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.467+0000 7fad7e278700 1 Processor -- start 2024-05-27T08:01:13.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.467+0000 7fad7e278700 1 -- start start 2024-05-27T08:01:13.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.468+0000 7fad7e278700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad78105450 0x7fad7819af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:13.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.468+0000 7fad7e278700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad78105da0 0x7fad7819b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:13.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.468+0000 7fad7e278700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad7810e8c0 0x7fad7819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:13.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.468+0000 7fad7e278700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fad781145f0 con 0x7fad7810e8c0 2024-05-27T08:01:13.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.468+0000 7fad7e278700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fad78114470 con 0x7fad78105da0 2024-05-27T08:01:13.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.468+0000 7fad7e278700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fad78114770 con 0x7fad78105450 2024-05-27T08:01:13.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.469+0000 7fad77fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad78105450 0x7fad7819af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:13.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.469+0000 7fad77fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad78105450 0x7fad7819af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.114:57980/0 (socket says 172.21.15.114:57980) 2024-05-27T08:01:13.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.469+0000 7fad77fff700 1 -- 172.21.15.114:0/1604010329 learned_addr learned my addr 172.21.15.114:0/1604010329 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:01:13.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.469+0000 7fad7c815700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad7810e8c0 0x7fad7819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:13.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.470+0000 7fad777fe700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad78105da0 0x7fad7819b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:13.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.470+0000 7fad7c815700 1 -- 172.21.15.114:0/1604010329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad78105450 msgr2=0x7fad7819af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:13.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.470+0000 7fad7c815700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad78105450 0x7fad7819af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:13.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.471+0000 7fad7c815700 1 -- 172.21.15.114:0/1604010329 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad78105da0 msgr2=0x7fad7819b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:13.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.471+0000 7fad7c815700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad78105da0 0x7fad7819b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:13.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.471+0000 7fad7c815700 1 -- 172.21.15.114:0/1604010329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7819fe10 con 0x7fad7810e8c0 2024-05-27T08:01:13.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.471+0000 7fad7c815700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad7810e8c0 0x7fad7819f770 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fad7000bb00 tx=0x7fad7000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:13.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.473+0000 7fad757fa700 1 -- 172.21.15.114:0/1604010329 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70019070 con 0x7fad7810e8c0 2024-05-27T08:01:13.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.474+0000 7fad757fa700 1 -- 172.21.15.114:0/1604010329 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad70014440 con 0x7fad7810e8c0 2024-05-27T08:01:13.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.474+0000 7fad757fa700 1 -- 172.21.15.114:0/1604010329 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70012560 con 0x7fad7810e8c0 2024-05-27T08:01:13.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.477+0000 7fad7e278700 1 -- 172.21.15.114:0/1604010329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad781a0100 con 0x7fad7810e8c0 2024-05-27T08:01:13.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.477+0000 7fad7e278700 1 -- 172.21.15.114:0/1604010329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad781a7a20 con 0x7fad7810e8c0 2024-05-27T08:01:13.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.479+0000 7fad7e278700 1 -- 172.21.15.114:0/1604010329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad780512b0 con 0x7fad7810e8c0 2024-05-27T08:01:13.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.481+0000 7fad757fa700 1 -- 172.21.15.114:0/1604010329 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fad700145f0 con 0x7fad7810e8c0 2024-05-27T08:01:13.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.483+0000 7fad757fa700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fad600b2bf0 0x7fad600b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:13.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.483+0000 7fad757fa700 1 -- 172.21.15.114:0/1604010329 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 9642+0+0 (secure 0 0 0) 0x7fad700e1cb0 con 0x7fad7810e8c0 2024-05-27T08:01:13.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.484+0000 7fad77fff700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fad600b2bf0 0x7fad600b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:13.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.493+0000 7fad757fa700 1 -- 172.21.15.114:0/1604010329 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fad700a92e0 con 0x7fad7810e8c0 2024-05-27T08:01:13.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.494+0000 7fad77fff700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fad600b2bf0 0x7fad600b5100 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009750 tx=0x7fad6c01a040 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:13.904 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:13 smithi114 ceph-mon[32038]: pgmap v745: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 370 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 376 KiB/s wr, 28 op/s 2024-05-27T08:01:13.904 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:13 smithi114 podman[138356]: 2024-05-27 08:01:13.484640782 +0000 UTC m=+0.046017077 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:01:13.905 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:13 smithi114 podman[138356]: 2024-05-27 08:01:13.62461514 +0000 UTC m=+0.185991427 container create 1aa32cf604446340554f5dd0d561abb8628c49207fb58b4a812d7354c5ae3142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-27T08:01:13.905 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:13 smithi114 podman[138356]: 2024-05-27 08:01:13.903145792 +0000 UTC m=+0.464522096 container init 1aa32cf604446340554f5dd0d561abb8628c49207fb58b4a812d7354c5ae3142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-27T08:01:13.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.921+0000 7fad7e278700 1 -- 172.21.15.114:0/1604010329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad7819c2d0 con 0x7fad7810e8c0 2024-05-27T08:01:13.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.925+0000 7fad757fa700 1 -- 172.21.15.114:0/1604010329 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) ==== 74+0+25632 (secure 0 0 0) 0x7fad7819c2d0 con 0x7fad7810e8c0 2024-05-27T08:01:13.928 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-27T08:01:13.928 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":172,"fsid":"755b3728-1bfa-11ef-bc9b-c7b262605968","created":"2024-05-27T07:27:47.693918+0000","modified":"2024-05-27T08:01:07.568843+0000","last_up_change":"2024-05-27T07:35:03.880786+0000","last_in_change":"2024-05-27T07:34:43.570570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:31:09.365695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:35:59.962762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:36:03.708207+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":172,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:36:13.010229+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":172,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"24622a6e-31e4-46ce-89d3-163303255c27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6803","nonce":2663646464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6805","nonce":2663646464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6809","nonce":2663646464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2663646464},{"type":"v1","addr":"172.21.15.114:6807","nonce":2663646464}]},"public_addr":"172.21.15.114:6803/2663646464","cluster_addr":"172.21.15.114:6805/2663646464","heartbeat_back_addr":"172.21.15.114:6809/2663646464","heartbeat_front_addr":"172.21.15.114:6807/2663646464","state":["exists","up"]},{"osd":1,"uuid":"e291f805-5a90-4c10-b6d6-3481a96f3f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6801","nonce":3356982241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6803","nonce":3356982241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6807","nonce":3356982241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3356982241},{"type":"v1","addr":"172.21.15.163:6805","nonce":3356982241}]},"public_addr":"172.21.15.163:6801/3356982241","cluster_addr":"172.21.15.163:6803/3356982241","heartbeat_back_addr":"172.21.15.163:6807/3356982241","heartbeat_front_addr":"172.21.15.163:6805/3356982241","state":["exists","up"]},{"osd":2,"uuid":"5bde05b7-5b12-46fe-a8f3-6992ed96f603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6801","nonce":249211108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6803","nonce":249211108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6807","nonce":249211108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":249211108},{"type":"v1","addr":"172.21.15.186:6805","nonce":249211108}]},"public_addr":"172.21.15.186:6801/249211108","cluster_addr":"172.21.15.186:6803/249211108","heartbeat_back_addr":"172.21.15.186:6807/249211108","heartbeat_front_addr":"172.21.15.186:6805/249211108","state":["exists","up"]},{"osd":3,"uuid":"d810071c-e2dd-442a-b5c5-080b81828549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6811","nonce":1973916170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6813","nonce":1973916170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6817","nonce":1973916170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1973916170},{"type":"v1","addr":"172.21.15.114:6815","nonce":1973916170}]},"public_addr":"172.21.15.114:6811/1973916170","cluster_addr":"172.21.15.114:6813/1973916170","heartbeat_back_addr":"172.21.15.114:6817/1973916170","heartbeat_front_addr":"172.21.15.114:6815/1973916170","state":["exists","up"]},{"osd":4,"uuid":"f013c785-92e2-493f-884a-3c71bb47c24c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6809","nonce":4093855927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6811","nonce":4093855927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6815","nonce":4093855927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4093855927},{"type":"v1","addr":"172.21.15.163:6813","nonce":4093855927}]},"public_addr":"172.21.15.163:6809/4093855927","cluster_addr":"172.21.15.163:6811/4093855927","heartbeat_back_addr":"172.21.15.163:6815/4093855927","heartbeat_front_addr":"172.21.15.163:6813/4093855927","state":["exists","up"]},{"osd":5,"uuid":"1cd4e0ab-5b13-4c1f-87d8-70db1827c7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6809","nonce":3483853241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6811","nonce":3483853241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6815","nonce":3483853241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3483853241},{"type":"v1","addr":"172.21.15.186:6813","nonce":3483853241}]},"public_addr":"172.21.15.186:6809/3483853241","cluster_addr":"172.21.15.186:6811/3483853241","heartbeat_back_addr":"172.21.15.186:6815/3483853241","heartbeat_front_addr":"172.21.15.186:6813/3483853241","state":["exists","up"]},{"osd":6,"uuid":"21fe0537-2b7a-4588-82fc-a0df295b078c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6819","nonce":2237458644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6821","nonce":2237458644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6825","nonce":2237458644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2237458644},{"type":"v1","addr":"172.21.15.114:6823","nonce":2237458644}]},"public_addr":"172.21.15.114:6819/2237458644","cluster_addr":"172.21.15.114:6821/2237458644","heartbeat_back_addr":"172.21.15.114:6825/2237458644","heartbeat_front_addr":"172.21.15.114:6823/2237458644","state":["exists","up"]},{"osd":7,"uuid":"a0aeb4d1-9292-4159-a1d5-64cb4fb3f81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6817","nonce":1909415962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6819","nonce":1909415962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6823","nonce":1909415962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":1909415962},{"type":"v1","addr":"172.21.15.163:6821","nonce":1909415962}]},"public_addr":"172.21.15.163:6817/1909415962","cluster_addr":"172.21.15.163:6819/1909415962","heartbeat_back_addr":"172.21.15.163:6823/1909415962","heartbeat_front_addr":"172.21.15.163:6821/1909415962","state":["exists","up"]},{"osd":8,"uuid":"5acc04eb-36af-4a65-9840-dba989b888a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6817","nonce":3389603353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6819","nonce":3389603353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6823","nonce":3389603353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3389603353},{"type":"v1","addr":"172.21.15.186:6821","nonce":3389603353}]},"public_addr":"172.21.15.186:6817/3389603353","cluster_addr":"172.21.15.186:6819/3389603353","heartbeat_back_addr":"172.21.15.186:6823/3389603353","heartbeat_front_addr":"172.21.15.186:6821/3389603353","state":["exists","up"]},{"osd":9,"uuid":"56bba69d-42d5-4bc9-8e80-1761b4fb12cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6827","nonce":3900437915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6829","nonce":3900437915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6833","nonce":3900437915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3900437915},{"type":"v1","addr":"172.21.15.114:6831","nonce":3900437915}]},"public_addr":"172.21.15.114:6827/3900437915","cluster_addr":"172.21.15.114:6829/3900437915","heartbeat_back_addr":"172.21.15.114:6833/3900437915","heartbeat_front_addr":"172.21.15.114:6831/3900437915","state":["exists","up"]},{"osd":10,"uuid":"825ee9ad-c4c8-4199-98df-76444163633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6825","nonce":1908338938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6827","nonce":1908338938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6831","nonce":1908338938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":1908338938},{"type":"v1","addr":"172.21.15.163:6829","nonce":1908338938}]},"public_addr":"172.21.15.163:6825/1908338938","cluster_addr":"172.21.15.163:6827/1908338938","heartbeat_back_addr":"172.21.15.163:6831/1908338938","heartbeat_front_addr":"172.21.15.163:6829/1908338938","state":["exists","up"]},{"osd":11,"uuid":"920e5b32-7159-48c0-9957-d7cb7db2ac38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6825","nonce":2406495501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6827","nonce":2406495501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6831","nonce":2406495501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2406495501},{"type":"v1","addr":"172.21.15.186:6829","nonce":2406495501}]},"public_addr":"172.21.15.186:6825/2406495501","cluster_addr":"172.21.15.186:6827/2406495501","heartbeat_back_addr":"172.21.15.186:6831/2406495501","heartbeat_front_addr":"172.21.15.186:6829/2406495501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:16.969845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:30:41.519505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:05.766911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:30.583910+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:31:56.362676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:22.471861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:32:48.840178+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:15.154271+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:33:41.256384+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:08.489286+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:34:34.771799+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:35:02.040713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/1628485461":"2024-05-28T07:39:44.126253+0000","172.21.15.114:0/1761274168":"2024-05-28T07:39:44.126253+0000","172.21.15.114:0/2574023732":"2024-05-28T07:39:44.126253+0000","172.21.15.114:6800/2291864634":"2024-05-28T07:39:44.126253+0000","172.21.15.114:0/3782422776":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/3323915605":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2381281099":"2024-05-28T07:39:44.126253+0000","172.21.15.114:6801/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2604585922":"2024-05-28T07:28:16.331472+0000","172.21.15.114:0/2303561217":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6801/2291864634":"2024-05-28T07:39:44.126253+0000","172.21.15.114:0/1305604861":"2024-05-28T07:28:43.271481+0000","172.21.15.114:6800/1568684596":"2024-05-28T07:28:16.331472+0000","172.21.15.114:6800/752776783":"2024-05-28T07:28:43.271481+0000","172.21.15.114:0/2442312517":"2024-05-28T07:28:16.331472+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":26,"length":1},{"begin":39,"length":1}]},{"pool":4,"snaps":[{"begin":26,"length":1},{"begin":39,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":26,"length":1},{"begin":39,"length":1}]},{"pool":4,"snaps":[{"begin":26,"length":1},{"begin":39,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T08:01:13.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.930+0000 7fad7e278700 1 -- 172.21.15.114:0/1604010329 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fad600b2bf0 msgr2=0x7fad600b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:13.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.930+0000 7fad7e278700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fad600b2bf0 0x7fad600b5100 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009750 tx=0x7fad6c01a040 comp rx=0 tx=0).stop 2024-05-27T08:01:13.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.930+0000 7fad7e278700 1 -- 172.21.15.114:0/1604010329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad7810e8c0 msgr2=0x7fad7819f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:13.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.930+0000 7fad7e278700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad7810e8c0 0x7fad7819f770 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fad7000bb00 tx=0x7fad7000be10 comp rx=0 tx=0).stop 2024-05-27T08:01:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.936+0000 7fad7e278700 1 -- 172.21.15.114:0/1604010329 shutdown_connections 2024-05-27T08:01:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.936+0000 7fad7e278700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad7810e8c0 0x7fad7819f770 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.936+0000 7fad7e278700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad78105450 0x7fad7819af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.936+0000 7fad7e278700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fad600b2bf0 0x7fad600b5100 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.936+0000 7fad7e278700 1 --2- 172.21.15.114:0/1604010329 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad78105da0 0x7fad7819b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.936+0000 7fad7e278700 1 -- 172.21.15.114:0/1604010329 >> 172.21.15.114:0/1604010329 conn(0x7fad780fedb0 msgr2=0x7fad780ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.941+0000 7fad7e278700 1 -- 172.21.15.114:0/1604010329 shutdown_connections 2024-05-27T08:01:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:13.945+0000 7fad7e278700 1 -- 172.21.15.114:0/1604010329 wait complete. 2024-05-27T08:01:13.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:13 smithi163 ceph-mon[36451]: pgmap v745: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 370 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 376 KiB/s wr, 28 op/s 2024-05-27T08:01:13.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:13 smithi186 ceph-mon[36647]: pgmap v745: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 370 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 376 KiB/s wr, 28 op/s 2024-05-27T08:01:14.308 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:13 smithi114 podman[138356]: 2024-05-27 08:01:13.911331087 +0000 UTC m=+0.472707381 container start 1aa32cf604446340554f5dd0d561abb8628c49207fb58b4a812d7354c5ae3142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-05-27T08:01:14.308 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: starting mds.d at 2024-05-27T08:01:14.308 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:14 smithi114 bash[138356]: 1aa32cf604446340554f5dd0d561abb8628c49207fb58b4a812d7354c5ae3142 2024-05-27T08:01:14.308 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:14 smithi114 systemd[1]: Started Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:01:14.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:14 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1604010329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T08:01:14.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:14 smithi163 ceph-mon[36451]: daemon mds.d restarted 2024-05-27T08:01:14.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:14 smithi163 ceph-mon[36451]: osdmap e173: 12 total, 12 up, 12 in 2024-05-27T08:01:14.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:14 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.114:6836/3613238529,v1:172.21.15.114:6837/3613238529] up:boot 2024-05-27T08:01:14.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:14 smithi163 ceph-mon[36451]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:01:14.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:14 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:01:14.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:14 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1604010329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T08:01:14.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:14 smithi186 ceph-mon[36647]: daemon mds.d restarted 2024-05-27T08:01:14.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:14 smithi186 ceph-mon[36647]: osdmap e173: 12 total, 12 up, 12 in 2024-05-27T08:01:14.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:14 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.114:6836/3613238529,v1:172.21.15.114:6837/3613238529] up:boot 2024-05-27T08:01:14.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:14 smithi186 ceph-mon[36647]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:01:14.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:14 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:01:15.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:14 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1604010329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T08:01:15.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:14 smithi114 ceph-mon[32038]: daemon mds.d restarted 2024-05-27T08:01:15.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:14 smithi114 ceph-mon[32038]: osdmap e173: 12 total, 12 up, 12 in 2024-05-27T08:01:15.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:14 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.114:6836/3613238529,v1:172.21.15.114:6837/3613238529] up:boot 2024-05-27T08:01:15.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:14 smithi114 ceph-mon[32038]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T08:01:15.059 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:14 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T08:01:15.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs fail cephfs 2024-05-27T08:01:15.929 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:15 smithi114 ceph-mon[32038]: pgmap v746: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 368 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 351 KiB/s wr, 10 op/s 2024-05-27T08:01:15.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:15 smithi163 ceph-mon[36451]: pgmap v746: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 368 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 351 KiB/s wr, 10 op/s 2024-05-27T08:01:15.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:15 smithi186 ceph-mon[36647]: pgmap v746: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 368 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 351 KiB/s wr, 10 op/s 2024-05-27T08:01:16.383 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:01:16.814 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe6ff08f700 time 2024-05-27T08:01:16.642204+0000 2024-05-27T08:01:16.815 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:01:16.815 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:16.815 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe7118d4ba6] 2024-05-27T08:01:16.815 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe7118d4dd1] 2024-05-27T08:01:16.815 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a57453723d] 2024-05-27T08:01:16.815 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a574522971] 2024-05-27T08:01:16.816 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a574520fa0] 2024-05-27T08:01:16.816 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a574521651] 2024-05-27T08:01:16.816 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5744b6347] 2024-05-27T08:01:16.816 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5741b287e] 2024-05-27T08:01:16.816 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5741b7e94] 2024-05-27T08:01:16.816 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a5741c2969] 2024-05-27T08:01:16.816 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a57455b48c] 2024-05-27T08:01:16.816 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a5745649cb] 2024-05-27T08:01:16.817 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 13: (Thread::entry_wrapper()+0x43) [0x7fe7118a372d] 2024-05-27T08:01:16.817 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe7118a3749] 2024-05-27T08:01:16.817 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe7104471ca] 2024-05-27T08:01:16.817 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 16: clone() 2024-05-27T08:01:16.817 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: *** Caught signal (Aborted) ** 2024-05-27T08:01:16.817 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: in thread 7fe6ff08f700 thread_name:md_log_replay 2024-05-27T08:01:16.817 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:16.644+0000 7fe6ff08f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe6ff08f700 time 2024-05-27T08:01:16.642204+0000 2024-05-27T08:01:16.817 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:01:16.818 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:16.818 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:16.818 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe7118d4ba6] 2024-05-27T08:01:16.818 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe7118d4dd1] 2024-05-27T08:01:16.818 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a57453723d] 2024-05-27T08:01:16.818 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a574522971] 2024-05-27T08:01:16.818 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a574520fa0] 2024-05-27T08:01:16.818 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a574521651] 2024-05-27T08:01:16.819 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5744b6347] 2024-05-27T08:01:16.819 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5741b287e] 2024-05-27T08:01:16.819 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5741b7e94] 2024-05-27T08:01:16.819 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a5741c2969] 2024-05-27T08:01:16.819 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a57455b48c] 2024-05-27T08:01:16.819 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a5745649cb] 2024-05-27T08:01:16.819 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 13: (Thread::entry_wrapper()+0x43) [0x7fe7118a372d] 2024-05-27T08:01:16.819 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe7118a3749] 2024-05-27T08:01:16.820 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe7104471ca] 2024-05-27T08:01:16.820 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 16: clone() 2024-05-27T08:01:16.820 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:16.820 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:16.820 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe710451d20] 2024-05-27T08:01:16.820 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2: gsignal() 2024-05-27T08:01:16.820 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 3: abort() 2024-05-27T08:01:16.820 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe7118d4ced] 2024-05-27T08:01:16.821 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe7118d4dd1] 2024-05-27T08:01:16.821 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a57453723d] 2024-05-27T08:01:16.821 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a574522971] 2024-05-27T08:01:16.821 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a574520fa0] 2024-05-27T08:01:16.821 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a574521651] 2024-05-27T08:01:16.821 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5744b6347] 2024-05-27T08:01:16.821 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5741b287e] 2024-05-27T08:01:16.821 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5741b7e94] 2024-05-27T08:01:16.822 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a5741c2969] 2024-05-27T08:01:16.822 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a57455b48c] 2024-05-27T08:01:16.822 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a5745649cb] 2024-05-27T08:01:16.822 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 16: (Thread::entry_wrapper()+0x43) [0x7fe7118a372d] 2024-05-27T08:01:16.822 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe7118a3749] 2024-05-27T08:01:16.822 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe7104471ca] 2024-05-27T08:01:16.822 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 19: clone() 2024-05-27T08:01:16.822 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:16.647+0000 7fe6ff08f700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:01:16.823 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: in thread 7fe6ff08f700 thread_name:md_log_replay 2024-05-27T08:01:16.823 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:16.823 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:16.823 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe710451d20] 2024-05-27T08:01:16.823 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2: gsignal() 2024-05-27T08:01:16.823 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 3: abort() 2024-05-27T08:01:16.823 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe7118d4ced] 2024-05-27T08:01:16.823 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe7118d4dd1] 2024-05-27T08:01:16.824 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a57453723d] 2024-05-27T08:01:16.824 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a574522971] 2024-05-27T08:01:16.824 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a574520fa0] 2024-05-27T08:01:16.824 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a574521651] 2024-05-27T08:01:16.824 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5744b6347] 2024-05-27T08:01:16.824 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5741b287e] 2024-05-27T08:01:16.825 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5741b7e94] 2024-05-27T08:01:16.825 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a5741c2969] 2024-05-27T08:01:16.825 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a57455b48c] 2024-05-27T08:01:16.825 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a5745649cb] 2024-05-27T08:01:16.825 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 16: (Thread::entry_wrapper()+0x43) [0x7fe7118a372d] 2024-05-27T08:01:16.825 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe7118a3749] 2024-05-27T08:01:16.825 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe7104471ca] 2024-05-27T08:01:16.825 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 19: clone() 2024-05-27T08:01:16.825 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:01:16.826 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:16.826 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: -1> 2024-05-27T08:01:16.644+0000 7fe6ff08f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe6ff08f700 time 2024-05-27T08:01:16.642204+0000 2024-05-27T08:01:16.826 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:01:16.826 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:16.826 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:16.826 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe7118d4ba6] 2024-05-27T08:01:16.826 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe7118d4dd1] 2024-05-27T08:01:16.827 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a57453723d] 2024-05-27T08:01:16.827 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a574522971] 2024-05-27T08:01:16.827 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a574520fa0] 2024-05-27T08:01:16.827 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a574521651] 2024-05-27T08:01:16.827 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5744b6347] 2024-05-27T08:01:16.827 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5741b287e] 2024-05-27T08:01:16.827 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5741b7e94] 2024-05-27T08:01:16.827 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a5741c2969] 2024-05-27T08:01:16.827 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a57455b48c] 2024-05-27T08:01:16.828 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a5745649cb] 2024-05-27T08:01:16.828 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 13: (Thread::entry_wrapper()+0x43) [0x7fe7118a372d] 2024-05-27T08:01:16.828 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe7118a3749] 2024-05-27T08:01:16.828 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe7104471ca] 2024-05-27T08:01:16.828 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 16: clone() 2024-05-27T08:01:16.828 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:16.828 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 0> 2024-05-27T08:01:16.647+0000 7fe6ff08f700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:01:16.828 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: in thread 7fe6ff08f700 thread_name:md_log_replay 2024-05-27T08:01:16.829 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:16.829 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:16.829 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe710451d20] 2024-05-27T08:01:16.829 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2: gsignal() 2024-05-27T08:01:16.829 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 3: abort() 2024-05-27T08:01:16.829 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe7118d4ced] 2024-05-27T08:01:16.829 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe7118d4dd1] 2024-05-27T08:01:16.829 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a57453723d] 2024-05-27T08:01:16.830 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a574522971] 2024-05-27T08:01:16.830 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a574520fa0] 2024-05-27T08:01:16.830 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a574521651] 2024-05-27T08:01:16.830 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5744b6347] 2024-05-27T08:01:16.830 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5741b287e] 2024-05-27T08:01:16.830 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5741b7e94] 2024-05-27T08:01:16.830 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a5741c2969] 2024-05-27T08:01:16.830 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a57455b48c] 2024-05-27T08:01:16.830 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a5745649cb] 2024-05-27T08:01:16.831 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 16: (Thread::entry_wrapper()+0x43) [0x7fe7118a372d] 2024-05-27T08:01:16.831 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe7118a3749] 2024-05-27T08:01:16.831 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe7104471ca] 2024-05-27T08:01:16.831 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 19: clone() 2024-05-27T08:01:16.831 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:01:16.831 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:16.831 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: -1> 2024-05-27T08:01:16.644+0000 7fe6ff08f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe6ff08f700 time 2024-05-27T08:01:16.642204+0000 2024-05-27T08:01:16.831 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T08:01:16.832 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:16.832 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:16.832 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe7118d4ba6] 2024-05-27T08:01:16.832 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe7118d4dd1] 2024-05-27T08:01:16.832 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a57453723d] 2024-05-27T08:01:16.832 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a574522971] 2024-05-27T08:01:16.832 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a574520fa0] 2024-05-27T08:01:16.832 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a574521651] 2024-05-27T08:01:16.833 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5744b6347] 2024-05-27T08:01:16.833 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5741b287e] 2024-05-27T08:01:16.833 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5741b7e94] 2024-05-27T08:01:16.833 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a5741c2969] 2024-05-27T08:01:16.833 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a57455b48c] 2024-05-27T08:01:16.833 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a5745649cb] 2024-05-27T08:01:16.833 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 13: (Thread::entry_wrapper()+0x43) [0x7fe7118a372d] 2024-05-27T08:01:16.833 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe7118a3749] 2024-05-27T08:01:16.834 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe7104471ca] 2024-05-27T08:01:16.834 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 16: clone() 2024-05-27T08:01:16.834 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:16.834 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 0> 2024-05-27T08:01:16.647+0000 7fe6ff08f700 -1 *** Caught signal (Aborted) ** 2024-05-27T08:01:16.834 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: in thread 7fe6ff08f700 thread_name:md_log_replay 2024-05-27T08:01:16.834 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:16.834 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T08:01:16.834 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe710451d20] 2024-05-27T08:01:16.835 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2: gsignal() 2024-05-27T08:01:16.835 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 3: abort() 2024-05-27T08:01:16.835 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe7118d4ced] 2024-05-27T08:01:16.835 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe7118d4dd1] 2024-05-27T08:01:16.835 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a57453723d] 2024-05-27T08:01:16.835 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a574522971] 2024-05-27T08:01:16.835 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a574520fa0] 2024-05-27T08:01:16.835 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a574521651] 2024-05-27T08:01:16.836 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5744b6347] 2024-05-27T08:01:16.836 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5741b287e] 2024-05-27T08:01:16.836 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5741b7e94] 2024-05-27T08:01:16.836 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a5741c2969] 2024-05-27T08:01:16.836 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a57455b48c] 2024-05-27T08:01:16.836 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a5745649cb] 2024-05-27T08:01:16.836 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 16: (Thread::entry_wrapper()+0x43) [0x7fe7118a372d] 2024-05-27T08:01:16.836 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe7118a3749] 2024-05-27T08:01:16.836 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe7104471ca] 2024-05-27T08:01:16.837 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 19: clone() 2024-05-27T08:01:16.837 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T08:01:16.837 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d[138387]: 2024-05-27T08:01:17.265 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:17 smithi114 podman[138506]: 2024-05-27 08:01:17.034034711 +0000 UTC m=+0.298463107 container died 1aa32cf604446340554f5dd0d561abb8628c49207fb58b4a812d7354c5ae3142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T08:01:17.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:17 smithi163 ceph-mon[36451]: pgmap v748: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 68 KiB/s wr, 11 op/s 2024-05-27T08:01:17.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:17 smithi186 ceph-mon[36647]: pgmap v748: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 68 KiB/s wr, 11 op/s 2024-05-27T08:01:17.982 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:17 smithi114 ceph-mon[32038]: pgmap v748: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 366 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 68 KiB/s wr, 11 op/s 2024-05-27T08:01:17.983 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:17 smithi114 podman[138506]: 2024-05-27 08:01:17.733485891 +0000 UTC m=+0.997914288 container remove 1aa32cf604446340554f5dd0d561abb8628c49207fb58b4a812d7354c5ae3142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-d, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T08:01:17.983 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:17 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T08:01:19.713 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:19 smithi114 ceph-mon[32038]: pgmap v749: 129 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 77 active+clean; 365 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 60 KiB/s wr, 10 op/s 2024-05-27T08:01:19.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.884+0000 7fe6392e0700 1 -- 172.21.15.114:0/1912498149 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62400bb40 con 0x7fe63410a920 2024-05-27T08:01:19.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.884+0000 7fe63c546700 1 -- 172.21.15.114:0/1912498149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe63410a920 msgr2=0x7fe63410ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:19.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.884+0000 7fe63c546700 1 --2- 172.21.15.114:0/1912498149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe63410a920 0x7fe63410ad30 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fe624009990 tx=0x7fe624009ca0 comp rx=0 tx=0).stop 2024-05-27T08:01:19.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.884+0000 7fe63c546700 1 -- 172.21.15.114:0/1912498149 shutdown_connections 2024-05-27T08:01:19.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.884+0000 7fe63c546700 1 --2- 172.21.15.114:0/1912498149 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe63411d920 0x7fe63411fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:19.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.885+0000 7fe63c546700 1 --2- 172.21.15.114:0/1912498149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe63410a920 0x7fe63410ad30 secure :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fe624009990 tx=0x7fe624009ca0 comp rx=0 tx=0).stop 2024-05-27T08:01:19.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.885+0000 7fe63c546700 1 --2- 172.21.15.114:0/1912498149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe63410b270 0x7fe63411d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:19.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.885+0000 7fe63c546700 1 -- 172.21.15.114:0/1912498149 >> 172.21.15.114:0/1912498149 conn(0x7fe63406f4c0 msgr2=0x7fe63406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:19.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.885+0000 7fe63c546700 1 -- 172.21.15.114:0/1912498149 shutdown_connections 2024-05-27T08:01:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.885+0000 7fe63c546700 1 -- 172.21.15.114:0/1912498149 wait complete. 2024-05-27T08:01:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.886+0000 7fe63c546700 1 Processor -- start 2024-05-27T08:01:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.886+0000 7fe63c546700 1 -- start start 2024-05-27T08:01:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.886+0000 7fe63c546700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe63410b270 0x7fe6341a9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.886+0000 7fe63c546700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe63411d920 0x7fe6341aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.886+0000 7fe63c546700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6341ae6c0 0x7fe6341aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.886+0000 7fe63c546700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6341233e0 con 0x7fe6341ae6c0 2024-05-27T08:01:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.886+0000 7fe63c546700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe634123260 con 0x7fe63410b270 2024-05-27T08:01:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.886+0000 7fe63c546700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe634123560 con 0x7fe63411d920 2024-05-27T08:01:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.887+0000 7fe63aae3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6341ae6c0 0x7fe6341aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.887+0000 7fe63aae3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6341ae6c0 0x7fe6341aead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56920/0 (socket says 172.21.15.114:56920) 2024-05-27T08:01:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.887+0000 7fe63aae3700 1 -- 172.21.15.114:0/1205466379 learned_addr learned my addr 172.21.15.114:0/1205466379 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:01:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.887+0000 7fe639ae1700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe63411d920 0x7fe6341aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.887+0000 7fe63a2e2700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe63410b270 0x7fe6341a9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.887+0000 7fe63aae3700 1 -- 172.21.15.114:0/1205466379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe63411d920 msgr2=0x7fe6341aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.887+0000 7fe63aae3700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe63411d920 0x7fe6341aa4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:19.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.887+0000 7fe63aae3700 1 -- 172.21.15.114:0/1205466379 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe63410b270 msgr2=0x7fe6341a9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:19.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.887+0000 7fe63aae3700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe63410b270 0x7fe6341a9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:19.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.887+0000 7fe63aae3700 1 -- 172.21.15.114:0/1205466379 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6341af210 con 0x7fe6341ae6c0 2024-05-27T08:01:19.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.888+0000 7fe63aae3700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6341ae6c0 0x7fe6341aead0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00ef90 tx=0x7fe62c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:19.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.888+0000 7fe62b7fe700 1 -- 172.21.15.114:0/1205466379 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c0092e0 con 0x7fe6341ae6c0 2024-05-27T08:01:19.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.888+0000 7fe62b7fe700 1 -- 172.21.15.114:0/1205466379 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe62c004bb0 con 0x7fe6341ae6c0 2024-05-27T08:01:19.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.888+0000 7fe62b7fe700 1 -- 172.21.15.114:0/1205466379 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c004280 con 0x7fe6341ae6c0 2024-05-27T08:01:19.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.888+0000 7fe63c546700 1 -- 172.21.15.114:0/1205466379 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6341af4a0 con 0x7fe6341ae6c0 2024-05-27T08:01:19.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.888+0000 7fe63c546700 1 -- 172.21.15.114:0/1205466379 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6341b6c30 con 0x7fe6341ae6c0 2024-05-27T08:01:19.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.890+0000 7fe63c546700 1 -- 172.21.15.114:0/1205466379 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6340512b0 con 0x7fe6341ae6c0 2024-05-27T08:01:19.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.892+0000 7fe62b7fe700 1 -- 172.21.15.114:0/1205466379 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fe62c01b030 con 0x7fe6341ae6c0 2024-05-27T08:01:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.893+0000 7fe62b7fe700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe6200b2a70 0x7fe6200b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.893+0000 7fe62b7fe700 1 -- 172.21.15.114:0/1205466379 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 9554+0+0 (secure 0 0 0) 0x7fe62c0e25a0 con 0x7fe6341ae6c0 2024-05-27T08:01:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.894+0000 7fe63a2e2700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe6200b2a70 0x7fe6200b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:19.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.895+0000 7fe63a2e2700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe6200b2a70 0x7fe6200b4f80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe624006010 tx=0x7fe62400b260 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:19.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:19.900+0000 7fe62b7fe700 1 -- 172.21.15.114:0/1205466379 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe62c022020 con 0x7fe6341ae6c0 2024-05-27T08:01:19.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:19 smithi163 ceph-mon[36451]: pgmap v749: 129 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 77 active+clean; 365 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 60 KiB/s wr, 10 op/s 2024-05-27T08:01:19.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:19 smithi186 ceph-mon[36647]: pgmap v749: 129 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 77 active+clean; 365 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 60 KiB/s wr, 10 op/s 2024-05-27T08:01:20.058 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:19 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T08:01:20.058 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:19 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Consumed 2.687s CPU time. 2024-05-27T08:01:20.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.344+0000 7fe63c546700 1 -- 172.21.15.114:0/1205466379 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe6341b6ef0 con 0x7fe6341ae6c0 2024-05-27T08:01:20.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.349+0000 7fe62b7fe700 1 -- 172.21.15.114:0/1205466379 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v123) ==== 161+0+0 (secure 0 0 0) 0x7fe62c010040 con 0x7fe6341ae6c0 2024-05-27T08:01:20.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.353+0000 7fe63c546700 1 -- 172.21.15.114:0/1205466379 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe6200b2a70 msgr2=0x7fe6200b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:20.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.353+0000 7fe63c546700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe6200b2a70 0x7fe6200b4f80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe624006010 tx=0x7fe62400b260 comp rx=0 tx=0).stop 2024-05-27T08:01:20.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.353+0000 7fe63c546700 1 -- 172.21.15.114:0/1205466379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6341ae6c0 msgr2=0x7fe6341aead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:20.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.353+0000 7fe63c546700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6341ae6c0 0x7fe6341aead0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00ef90 tx=0x7fe62c00c3a0 comp rx=0 tx=0).stop 2024-05-27T08:01:20.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.357+0000 7fe63c546700 1 -- 172.21.15.114:0/1205466379 shutdown_connections 2024-05-27T08:01:20.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.357+0000 7fe63c546700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6341ae6c0 0x7fe6341aead0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:20.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.357+0000 7fe63c546700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fe6200b2a70 0x7fe6200b4f80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:20.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.357+0000 7fe63c546700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe63410b270 0x7fe6341a9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:20.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.357+0000 7fe63c546700 1 --2- 172.21.15.114:0/1205466379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe63411d920 0x7fe6341aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:20.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.357+0000 7fe63c546700 1 -- 172.21.15.114:0/1205466379 >> 172.21.15.114:0/1205466379 conn(0x7fe63406f4c0 msgr2=0x7fe63411b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:20.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.359+0000 7fe63c546700 1 -- 172.21.15.114:0/1205466379 shutdown_connections 2024-05-27T08:01:20.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:20.360+0000 7fe63c546700 1 -- 172.21.15.114:0/1205466379 wait complete. 2024-05-27T08:01:20.366 INFO:teuthology.orchestra.run.smithi114.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-27T08:01:20.646 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a[32034]: 2024-05-27T08:01:20.345+0000 7f0d487a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-27T08:01:20.646 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:20 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1205466379' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-27T08:01:20.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:20 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1205466379' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-27T08:01:20.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:20 smithi186 ceph-mon[36647]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-27T08:01:20.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:20 smithi186 ceph-mon[36647]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-27T08:01:20.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:20 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1205466379' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-27T08:01:20.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:20 smithi186 ceph-mon[36647]: osdmap e174: 12 total, 12 up, 12 in 2024-05-27T08:01:20.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:20 smithi186 ceph-mon[36647]: fsmap cephfs:0/1 1 up:standby, 1 failed 2024-05-27T08:01:20.731 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 08:01:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-i[67994]: ignoring --setuser ceph since I am not root 2024-05-27T08:01:20.731 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 08:01:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-i[67994]: ignoring --setgroup ceph since I am not root 2024-05-27T08:01:20.732 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 08:01:20 smithi186 ceph-mds[67998]: -- 172.21.15.186:0/1046163353 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c18282fa40 con 0x55c182877600 2024-05-27T08:01:20.732 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 08:01:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-i[67994]: starting mds.i at 2024-05-27T08:01:20.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:20 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1205466379' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-27T08:01:20.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:20 smithi163 ceph-mon[36451]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-27T08:01:20.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:20 smithi163 ceph-mon[36451]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-27T08:01:20.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:20 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1205466379' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-27T08:01:20.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:20 smithi163 ceph-mon[36451]: osdmap e174: 12 total, 12 up, 12 in 2024-05-27T08:01:20.979 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:20 smithi163 ceph-mon[36451]: fsmap cephfs:0/1 1 up:standby, 1 failed 2024-05-27T08:01:21.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:20 smithi114 ceph-mon[32038]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-27T08:01:21.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:20 smithi114 ceph-mon[32038]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-27T08:01:21.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:20 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1205466379' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-27T08:01:21.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:20 smithi114 ceph-mon[32038]: osdmap e174: 12 total, 12 up, 12 in 2024-05-27T08:01:21.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:20 smithi114 ceph-mon[32038]: fsmap cephfs:0/1 1 up:standby, 1 failed 2024-05-27T08:01:21.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:21 smithi114 ceph-mon[32038]: pgmap v751: 129 pgs: 9 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 94 active+clean; 364 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 18 KiB/s wr, 5 op/s 2024-05-27T08:01:21.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:21 smithi114 ceph-mon[32038]: mds.? [v2:172.21.15.186:6832/2712397635,v1:172.21.15.186:6833/2712397635] up:boot 2024-05-27T08:01:21.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:21 smithi114 ceph-mon[32038]: fsmap cephfs:0/1 2 up:standby, 1 failed 2024-05-27T08:01:21.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:21 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T08:01:21.892 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-05-27T08:01:21.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:21 smithi163 ceph-mon[36451]: pgmap v751: 129 pgs: 9 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 94 active+clean; 364 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 18 KiB/s wr, 5 op/s 2024-05-27T08:01:21.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:21 smithi163 ceph-mon[36451]: mds.? [v2:172.21.15.186:6832/2712397635,v1:172.21.15.186:6833/2712397635] up:boot 2024-05-27T08:01:21.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:21 smithi163 ceph-mon[36451]: fsmap cephfs:0/1 2 up:standby, 1 failed 2024-05-27T08:01:21.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:21 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T08:01:21.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:21 smithi186 ceph-mon[36647]: pgmap v751: 129 pgs: 9 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 94 active+clean; 364 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 18 KiB/s wr, 5 op/s 2024-05-27T08:01:21.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:21 smithi186 ceph-mon[36647]: mds.? [v2:172.21.15.186:6832/2712397635,v1:172.21.15.186:6833/2712397635] up:boot 2024-05-27T08:01:21.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:21 smithi186 ceph-mon[36647]: fsmap cephfs:0/1 2 up:standby, 1 failed 2024-05-27T08:01:21.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:21 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T08:01:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:01:23.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:23 smithi163 ceph-mon[36451]: pgmap v752: 129 pgs: 5 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 112 active+clean; 360 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 17 KiB/s wr, 5 op/s 2024-05-27T08:01:23.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:23 smithi186 ceph-mon[36647]: pgmap v752: 129 pgs: 5 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 112 active+clean; 360 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 17 KiB/s wr, 5 op/s 2024-05-27T08:01:24.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:23 smithi114 ceph-mon[32038]: pgmap v752: 129 pgs: 5 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 112 active+clean; 360 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 17 KiB/s wr, 5 op/s 2024-05-27T08:01:24.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.989+0000 7f703da9c700 1 -- 172.21.15.114:0/2085219205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7038107df0 msgr2=0x7f7038108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:24.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.989+0000 7f703da9c700 1 --2- 172.21.15.114:0/2085219205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7038107df0 0x7f7038108200 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f70280099c0 tx=0x7f7028009cd0 comp rx=0 tx=0).stop 2024-05-27T08:01:24.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.989+0000 7f703da9c700 1 -- 172.21.15.114:0/2085219205 shutdown_connections 2024-05-27T08:01:24.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.989+0000 7f703da9c700 1 --2- 172.21.15.114:0/2085219205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7038108740 0x7f70381100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:24.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.989+0000 7f703da9c700 1 --2- 172.21.15.114:0/2085219205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7038103c50 0x7f7038104060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:24.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.989+0000 7f703da9c700 1 --2- 172.21.15.114:0/2085219205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7038107df0 0x7f7038108200 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:24.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.989+0000 7f703da9c700 1 -- 172.21.15.114:0/2085219205 >> 172.21.15.114:0/2085219205 conn(0x7f70380fe6e0 msgr2=0x7f7038100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:24.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.990+0000 7f703da9c700 1 -- 172.21.15.114:0/2085219205 shutdown_connections 2024-05-27T08:01:24.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.990+0000 7f703da9c700 1 -- 172.21.15.114:0/2085219205 wait complete. 2024-05-27T08:01:24.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.990+0000 7f703da9c700 1 Processor -- start 2024-05-27T08:01:24.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f703da9c700 1 -- start start 2024-05-27T08:01:24.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f703da9c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7038103c50 0x7f703819bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:24.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f703da9c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7038107df0 0x7f703819c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:24.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f703da9c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7038108740 0x7f703819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:24.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f703da9c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7038113370 con 0x7f7038103c50 2024-05-27T08:01:24.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f703da9c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f70381131f0 con 0x7f7038108740 2024-05-27T08:01:24.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f703da9c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f70381134f0 con 0x7f7038107df0 2024-05-27T08:01:24.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f70377fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7038103c50 0x7f703819bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:24.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f70377fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7038103c50 0x7f703819bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48882/0 (socket says 172.21.15.114:48882) 2024-05-27T08:01:24.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f70377fe700 1 -- 172.21.15.114:0/3619031410 learned_addr learned my addr 172.21.15.114:0/3619031410 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:01:24.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f7036ffd700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7038107df0 0x7f703819c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:24.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f7037fff700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7038108740 0x7f703819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:24.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f70377fe700 1 -- 172.21.15.114:0/3619031410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7038107df0 msgr2=0x7f703819c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:24.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f70377fe700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7038107df0 0x7f703819c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:24.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f70377fe700 1 -- 172.21.15.114:0/3619031410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7038108740 msgr2=0x7f703819c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:24.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f70377fe700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7038108740 0x7f703819c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:24.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f70377fe700 1 -- 172.21.15.114:0/3619031410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70381a7060 con 0x7f7038103c50 2024-05-27T08:01:24.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.991+0000 7f7037fff700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7038108740 0x7f703819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T08:01:24.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.992+0000 7f70377fe700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7038103c50 0x7f703819bd10 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f702c00c970 tx=0x7f702c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:24.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.992+0000 7f7034ff9700 1 -- 172.21.15.114:0/3619031410 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c007aa0 con 0x7f7038103c50 2024-05-27T08:01:24.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.992+0000 7f703da9c700 1 -- 172.21.15.114:0/3619031410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70381a7350 con 0x7f7038103c50 2024-05-27T08:01:24.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.992+0000 7f703da9c700 1 -- 172.21.15.114:0/3619031410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70381a7860 con 0x7f7038103c50 2024-05-27T08:01:24.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.992+0000 7f7034ff9700 1 -- 172.21.15.114:0/3619031410 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f702c004ce0 con 0x7f7038103c50 2024-05-27T08:01:24.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.992+0000 7f7034ff9700 1 -- 172.21.15.114:0/3619031410 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c005510 con 0x7f7038103c50 2024-05-27T08:01:24.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.993+0000 7f703da9c700 1 -- 172.21.15.114:0/3619031410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70381049d0 con 0x7f7038103c50 2024-05-27T08:01:24.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.995+0000 7f7034ff9700 1 -- 172.21.15.114:0/3619031410 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f702c007c40 con 0x7f7038103c50 2024-05-27T08:01:24.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.995+0000 7f7034ff9700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f70200b2a70 0x7f70200b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:24.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.995+0000 7f7036ffd700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f70200b2a70 0x7f70200b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:24.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.996+0000 7f7034ff9700 1 -- 172.21.15.114:0/3619031410 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 9640+0+0 (secure 0 0 0) 0x7f702c0e2a20 con 0x7f7038103c50 2024-05-27T08:01:24.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.996+0000 7f7036ffd700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f70200b2a70 0x7f70200b4f80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7028006010 tx=0x7f7028000bc0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:25.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:24.999+0000 7f7034ff9700 1 -- 172.21.15.114:0/3619031410 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f702c0aa0c0 con 0x7f7038103c50 2024-05-27T08:01:25.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.255+0000 7f703da9c700 1 -- 172.21.15.114:0/3619031410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f703802e5e0 con 0x7f7038103c50 2024-05-27T08:01:25.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.650+0000 7f7034ff9700 1 -- 172.21.15.114:0/3619031410 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v125) ==== 104+0+0 (secure 0 0 0) 0x7f702c0a99c0 con 0x7f7038103c50 2024-05-27T08:01:25.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.653+0000 7f703da9c700 1 -- 172.21.15.114:0/3619031410 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f70200b2a70 msgr2=0x7f70200b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:25.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.653+0000 7f703da9c700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f70200b2a70 0x7f70200b4f80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7028006010 tx=0x7f7028000bc0 comp rx=0 tx=0).stop 2024-05-27T08:01:25.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.653+0000 7f703da9c700 1 -- 172.21.15.114:0/3619031410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7038103c50 msgr2=0x7f703819bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:25.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.653+0000 7f703da9c700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7038103c50 0x7f703819bd10 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f702c00c970 tx=0x7f702c00cc80 comp rx=0 tx=0).stop 2024-05-27T08:01:25.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.653+0000 7f703da9c700 1 -- 172.21.15.114:0/3619031410 shutdown_connections 2024-05-27T08:01:25.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.653+0000 7f703da9c700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f70200b2a70 0x7f70200b4f80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:25.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.653+0000 7f703da9c700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7038108740 0x7f703819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:25.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.653+0000 7f703da9c700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7038103c50 0x7f703819bd10 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:25.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.653+0000 7f703da9c700 1 --2- 172.21.15.114:0/3619031410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7038107df0 0x7f703819c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:25.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.653+0000 7f703da9c700 1 -- 172.21.15.114:0/3619031410 >> 172.21.15.114:0/3619031410 conn(0x7f70380fe6e0 msgr2=0x7f70381096a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:25.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.653+0000 7f703da9c700 1 -- 172.21.15.114:0/3619031410 shutdown_connections 2024-05-27T08:01:25.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:25.653+0000 7f703da9c700 1 -- 172.21.15.114:0/3619031410 wait complete. 2024-05-27T08:01:25.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:25 smithi163 ceph-mon[36451]: pgmap v753: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 359 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 14 KiB/s wr, 4 op/s 2024-05-27T08:01:25.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:25 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3619031410' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T08:01:25.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:25 smithi186 ceph-mon[36647]: pgmap v753: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 359 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 14 KiB/s wr, 4 op/s 2024-05-27T08:01:25.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:25 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3619031410' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T08:01:26.037 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:25 smithi114 ceph-mon[32038]: pgmap v753: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 359 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 14 KiB/s wr, 4 op/s 2024-05-27T08:01:26.037 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:25 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3619031410' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T08:01:26.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-27T08:01:26.625 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:01:26.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:26 smithi114 ceph-mon[32038]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-27T08:01:26.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:26 smithi114 ceph-mon[32038]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-27T08:01:26.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:26 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/3619031410' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-27T08:01:26.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:26 smithi114 ceph-mon[32038]: fsmap 2 up:standby 2024-05-27T08:01:26.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:26 smithi163 ceph-mon[36451]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-27T08:01:26.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:26 smithi163 ceph-mon[36451]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-27T08:01:26.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:26 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/3619031410' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-27T08:01:26.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:26 smithi163 ceph-mon[36451]: fsmap 2 up:standby 2024-05-27T08:01:26.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:26 smithi186 ceph-mon[36647]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-27T08:01:26.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:26 smithi186 ceph-mon[36647]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-27T08:01:26.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:26 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/3619031410' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-27T08:01:26.981 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:26 smithi186 ceph-mon[36647]: fsmap 2 up:standby 2024-05-27T08:01:27.925 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:27 smithi114 ceph-mon[32038]: pgmap v754: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 358 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 102 B/s wr, 0 op/s 2024-05-27T08:01:27.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:27 smithi163 ceph-mon[36451]: pgmap v754: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 358 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 102 B/s wr, 0 op/s 2024-05-27T08:01:27.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:27 smithi186 ceph-mon[36647]: pgmap v754: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 358 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 102 B/s wr, 0 op/s 2024-05-27T08:01:28.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.366+0000 7f90f3143700 1 -- 172.21.15.114:0/1125598318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90ec105e90 msgr2=0x7f90ec10e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:28.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.366+0000 7f90f3143700 1 --2- 172.21.15.114:0/1125598318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90ec105e90 0x7f90ec10e3e0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f90d8009990 tx=0x7f90d8009ca0 comp rx=0 tx=0).stop 2024-05-27T08:01:28.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.366+0000 7f90f3143700 1 -- 172.21.15.114:0/1125598318 shutdown_connections 2024-05-27T08:01:28.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.366+0000 7f90f3143700 1 --2- 172.21.15.114:0/1125598318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90ec10e920 0x7f90ec110e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:28.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.366+0000 7f90f3143700 1 --2- 172.21.15.114:0/1125598318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90ec105540 0x7f90ec105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:28.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.366+0000 7f90f3143700 1 --2- 172.21.15.114:0/1125598318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90ec105e90 0x7f90ec10e3e0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:28.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.366+0000 7f90f3143700 1 -- 172.21.15.114:0/1125598318 >> 172.21.15.114:0/1125598318 conn(0x7f90ec078df0 msgr2=0x7f90ec079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:28.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.366+0000 7f90f3143700 1 -- 172.21.15.114:0/1125598318 shutdown_connections 2024-05-27T08:01:28.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.366+0000 7f90f3143700 1 -- 172.21.15.114:0/1125598318 wait complete. 2024-05-27T08:01:28.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.367+0000 7f90f3143700 1 Processor -- start 2024-05-27T08:01:28.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.367+0000 7f90f3143700 1 -- start start 2024-05-27T08:01:28.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.367+0000 7f90f3143700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90ec105540 0x7f90ec19ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:28.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.367+0000 7f90f3143700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90ec105e90 0x7f90ec19b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:28.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.367+0000 7f90f3143700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90ec10e920 0x7f90ec19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:28.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.367+0000 7f90f3143700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f90ec114470 con 0x7f90ec10e920 2024-05-27T08:01:28.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.367+0000 7f90f3143700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f90ec1142f0 con 0x7f90ec105e90 2024-05-27T08:01:28.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.367+0000 7f90f3143700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f90ec1145f0 con 0x7f90ec105540 2024-05-27T08:01:28.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.367+0000 7f90f16e0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90ec10e920 0x7f90ec19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:28.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.367+0000 7f90f16e0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90ec10e920 0x7f90ec19f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48890/0 (socket says 172.21.15.114:48890) 2024-05-27T08:01:28.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.367+0000 7f90f16e0700 1 -- 172.21.15.114:0/1004510189 learned_addr learned my addr 172.21.15.114:0/1004510189 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:01:28.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.367+0000 7f90f0edf700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90ec105540 0x7f90ec19ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:28.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.367+0000 7f90ebfff700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90ec105e90 0x7f90ec19b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:28.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.368+0000 7f90f16e0700 1 -- 172.21.15.114:0/1004510189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90ec105540 msgr2=0x7f90ec19ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:28.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.368+0000 7f90f16e0700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90ec105540 0x7f90ec19ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:28.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.368+0000 7f90f16e0700 1 -- 172.21.15.114:0/1004510189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90ec105e90 msgr2=0x7f90ec19b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:28.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.368+0000 7f90f16e0700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90ec105e90 0x7f90ec19b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:28.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.368+0000 7f90f16e0700 1 -- 172.21.15.114:0/1004510189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90ec19fcb0 con 0x7f90ec10e920 2024-05-27T08:01:28.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.368+0000 7f90f16e0700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90ec10e920 0x7f90ec19f610 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f90e4007ee0 tx=0x7f90e400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:28.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.368+0000 7f90e9ffb700 1 -- 172.21.15.114:0/1004510189 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4004bb0 con 0x7f90ec10e920 2024-05-27T08:01:28.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.368+0000 7f90e9ffb700 1 -- 172.21.15.114:0/1004510189 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90e4004d50 con 0x7f90ec10e920 2024-05-27T08:01:28.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.368+0000 7f90f3143700 1 -- 172.21.15.114:0/1004510189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90ec19ffa0 con 0x7f90ec10e920 2024-05-27T08:01:28.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.368+0000 7f90e9ffb700 1 -- 172.21.15.114:0/1004510189 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4005920 con 0x7f90ec10e920 2024-05-27T08:01:28.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.368+0000 7f90f3143700 1 -- 172.21.15.114:0/1004510189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90ec1a78c0 con 0x7f90ec10e920 2024-05-27T08:01:28.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.369+0000 7f90f3143700 1 -- 172.21.15.114:0/1004510189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90ec076210 con 0x7f90ec10e920 2024-05-27T08:01:28.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.371+0000 7f90e9ffb700 1 -- 172.21.15.114:0/1004510189 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f90e401a020 con 0x7f90ec10e920 2024-05-27T08:01:28.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.372+0000 7f90e9ffb700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f90dc0b29f0 0x7f90dc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:28.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.372+0000 7f90f0edf700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f90dc0b29f0 0x7f90dc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:28.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.372+0000 7f90e9ffb700 1 -- 172.21.15.114:0/1004510189 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 9640+0+0 (secure 0 0 0) 0x7f90e40e20a0 con 0x7f90ec10e920 2024-05-27T08:01:28.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.372+0000 7f90f0edf700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f90dc0b29f0 0x7f90dc0b4f00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f90e0005950 tx=0x7f90e000b410 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:28.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.377+0000 7f90e9ffb700 1 -- 172.21.15.114:0/1004510189 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f90e4010070 con 0x7f90ec10e920 2024-05-27T08:01:28.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.582+0000 7f90f3143700 1 -- 172.21.15.114:0/1004510189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f90ec02e5e0 con 0x7f90ec10e920 2024-05-27T08:01:28.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.677+0000 7f90e9ffb700 1 -- 172.21.15.114:0/1004510189 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v175) ==== 181+0+0 (secure 0 0 0) 0x7f90e40a9620 con 0x7f90ec10e920 2024-05-27T08:01:28.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.679+0000 7f90f3143700 1 -- 172.21.15.114:0/1004510189 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f90dc0b29f0 msgr2=0x7f90dc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:28.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.679+0000 7f90f3143700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f90dc0b29f0 0x7f90dc0b4f00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f90e0005950 tx=0x7f90e000b410 comp rx=0 tx=0).stop 2024-05-27T08:01:28.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.679+0000 7f90f3143700 1 -- 172.21.15.114:0/1004510189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90ec10e920 msgr2=0x7f90ec19f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:28.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.679+0000 7f90f3143700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90ec10e920 0x7f90ec19f610 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f90e4007ee0 tx=0x7f90e400d3b0 comp rx=0 tx=0).stop 2024-05-27T08:01:28.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.680+0000 7f90f3143700 1 -- 172.21.15.114:0/1004510189 shutdown_connections 2024-05-27T08:01:28.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.680+0000 7f90f3143700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90ec10e920 0x7f90ec19f610 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:28.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.680+0000 7f90f3143700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90ec105540 0x7f90ec19ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:28.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.680+0000 7f90f3143700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f90dc0b29f0 0x7f90dc0b4f00 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:28.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.680+0000 7f90f3143700 1 --2- 172.21.15.114:0/1004510189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90ec105e90 0x7f90ec19b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:28.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.680+0000 7f90f3143700 1 -- 172.21.15.114:0/1004510189 >> 172.21.15.114:0/1004510189 conn(0x7f90ec078df0 msgr2=0x7f90ec100030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:28.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.680+0000 7f90f3143700 1 -- 172.21.15.114:0/1004510189 shutdown_connections 2024-05-27T08:01:28.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:28.680+0000 7f90f3143700 1 -- 172.21.15.114:0/1004510189 wait complete. 2024-05-27T08:01:28.683 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' removed 2024-05-27T08:01:28.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:28 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1004510189' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T08:01:28.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:28 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1004510189' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T08:01:29.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:28 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1004510189' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T08:01:29.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-27T08:01:29.747 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:01:29.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:29 smithi114 ceph-mon[32038]: pgmap v755: 129 pgs: 129 active+clean; 357 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 102 B/s wr, 0 op/s 2024-05-27T08:01:29.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:29 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1004510189' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T08:01:29.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:29 smithi114 ceph-mon[32038]: osdmap e175: 12 total, 12 up, 12 in 2024-05-27T08:01:29.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:29 smithi114 ceph-mon[32038]: osdmap e176: 12 total, 12 up, 12 in 2024-05-27T08:01:29.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:29 smithi163 ceph-mon[36451]: pgmap v755: 129 pgs: 129 active+clean; 357 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 102 B/s wr, 0 op/s 2024-05-27T08:01:29.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:29 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1004510189' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T08:01:29.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:29 smithi163 ceph-mon[36451]: osdmap e175: 12 total, 12 up, 12 in 2024-05-27T08:01:29.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:29 smithi163 ceph-mon[36451]: osdmap e176: 12 total, 12 up, 12 in 2024-05-27T08:01:29.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:29 smithi186 ceph-mon[36647]: pgmap v755: 129 pgs: 129 active+clean; 357 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 102 B/s wr, 0 op/s 2024-05-27T08:01:29.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:29 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1004510189' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T08:01:29.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:29 smithi186 ceph-mon[36647]: osdmap e175: 12 total, 12 up, 12 in 2024-05-27T08:01:29.981 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:29 smithi186 ceph-mon[36647]: osdmap e176: 12 total, 12 up, 12 in 2024-05-27T08:01:30.145 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:29 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-05-27T08:01:30.145 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:29 smithi114 systemd[1]: Stopped Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:01:30.145 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:29 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Consumed 2.687s CPU time. 2024-05-27T08:01:30.145 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:29 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Start request repeated too quickly. 2024-05-27T08:01:30.146 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:29 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T08:01:30.146 INFO:journalctl@ceph.mds.d.smithi114.stdout:May 27 08:01:29 smithi114 systemd[1]: Failed to start Ceph mds.d for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:01:31.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.519+0000 7fa995b0c700 1 -- 172.21.15.114:0/213353321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990108f10 msgr2=0x7fa990109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:31.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.519+0000 7fa995b0c700 1 --2- 172.21.15.114:0/213353321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990108f10 0x7fa990109320 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fa9840099c0 tx=0x7fa984009cd0 comp rx=0 tx=0).stop 2024-05-27T08:01:31.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.520+0000 7fa995b0c700 1 -- 172.21.15.114:0/213353321 shutdown_connections 2024-05-27T08:01:31.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.520+0000 7fa995b0c700 1 --2- 172.21.15.114:0/213353321 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa990109860 0x7fa9901111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:31.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.520+0000 7fa995b0c700 1 --2- 172.21.15.114:0/213353321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9901045e0 0x7fa9901049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:31.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.520+0000 7fa995b0c700 1 --2- 172.21.15.114:0/213353321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990108f10 0x7fa990109320 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:31.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.520+0000 7fa995b0c700 1 -- 172.21.15.114:0/213353321 >> 172.21.15.114:0/213353321 conn(0x7fa9900fffd0 msgr2=0x7fa9901023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:31.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.520+0000 7fa995b0c700 1 -- 172.21.15.114:0/213353321 shutdown_connections 2024-05-27T08:01:31.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.520+0000 7fa995b0c700 1 -- 172.21.15.114:0/213353321 wait complete. 2024-05-27T08:01:31.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.520+0000 7fa995b0c700 1 Processor -- start 2024-05-27T08:01:31.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.520+0000 7fa995b0c700 1 -- start start 2024-05-27T08:01:31.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa995b0c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9901045e0 0x7fa99019aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:31.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa995b0c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990108f10 0x7fa99019b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:31.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa995b0c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa990109860 0x7fa99019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:31.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa995b0c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa990114560 con 0x7fa990108f10 2024-05-27T08:01:31.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa995b0c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9901143e0 con 0x7fa9901045e0 2024-05-27T08:01:31.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa995b0c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa9901146e0 con 0x7fa990109860 2024-05-27T08:01:31.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa98effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990108f10 0x7fa99019b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:31.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa98effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990108f10 0x7fa99019b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48908/0 (socket says 172.21.15.114:48908) 2024-05-27T08:01:31.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa98effd700 1 -- 172.21.15.114:0/202313101 learned_addr learned my addr 172.21.15.114:0/202313101 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:01:31.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa98f7fe700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9901045e0 0x7fa99019aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:31.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa98ffff700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa990109860 0x7fa99019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:31.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa98effd700 1 -- 172.21.15.114:0/202313101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa990109860 msgr2=0x7fa99019f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:31.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa98effd700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa990109860 0x7fa99019f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:31.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa98effd700 1 -- 172.21.15.114:0/202313101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9901045e0 msgr2=0x7fa99019aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:31.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa98effd700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9901045e0 0x7fa99019aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:31.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa98effd700 1 -- 172.21.15.114:0/202313101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99019fdc0 con 0x7fa990108f10 2024-05-27T08:01:31.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.521+0000 7fa98effd700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990108f10 0x7fa99019b490 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa984009f50 tx=0x7fa9840037c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:31.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.522+0000 7fa98cff9700 1 -- 172.21.15.114:0/202313101 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa984027070 con 0x7fa990108f10 2024-05-27T08:01:31.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.522+0000 7fa98cff9700 1 -- 172.21.15.114:0/202313101 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa984004090 con 0x7fa990108f10 2024-05-27T08:01:31.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.522+0000 7fa995b0c700 1 -- 172.21.15.114:0/202313101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9901a0050 con 0x7fa990108f10 2024-05-27T08:01:31.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.522+0000 7fa98cff9700 1 -- 172.21.15.114:0/202313101 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa984022e60 con 0x7fa990108f10 2024-05-27T08:01:31.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.522+0000 7fa995b0c700 1 -- 172.21.15.114:0/202313101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9901a0530 con 0x7fa990108f10 2024-05-27T08:01:31.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.523+0000 7fa995b0c700 1 -- 172.21.15.114:0/202313101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9900512b0 con 0x7fa990108f10 2024-05-27T08:01:31.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.524+0000 7fa98cff9700 1 -- 172.21.15.114:0/202313101 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7fa98402a070 con 0x7fa990108f10 2024-05-27T08:01:31.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.525+0000 7fa98cff9700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa9780b29f0 0x7fa9780b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:31.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.525+0000 7fa98f7fe700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa9780b29f0 0x7fa9780b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:31.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.525+0000 7fa98cff9700 1 -- 172.21.15.114:0/202313101 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 9086+0+0 (secure 0 0 0) 0x7fa9840e3860 con 0x7fa990108f10 2024-05-27T08:01:31.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.526+0000 7fa98f7fe700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa9780b29f0 0x7fa9780b4f00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa980009a10 tx=0x7fa980009190 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:31.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.529+0000 7fa98cff9700 1 -- 172.21.15.114:0/202313101 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa984013070 con 0x7fa990108f10 2024-05-27T08:01:31.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:31.735+0000 7fa995b0c700 1 -- 172.21.15.114:0/202313101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa99002e5e0 con 0x7fa990108f10 2024-05-27T08:01:31.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:31 smithi114 ceph-mon[32038]: pgmap v758: 97 pgs: 97 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:31.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:31 smithi163 ceph-mon[36451]: pgmap v758: 97 pgs: 97 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:31.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:31 smithi186 ceph-mon[36647]: pgmap v758: 97 pgs: 97 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:32.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:32.691+0000 7fa98cff9700 1 -- 172.21.15.114:0/202313101 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v177) ==== 169+0+0 (secure 0 0 0) 0x7fa9840ab0a0 con 0x7fa990108f10 2024-05-27T08:01:32.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:32.694+0000 7fa995b0c700 1 -- 172.21.15.114:0/202313101 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa9780b29f0 msgr2=0x7fa9780b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:32.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:32.694+0000 7fa995b0c700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa9780b29f0 0x7fa9780b4f00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa980009a10 tx=0x7fa980009190 comp rx=0 tx=0).stop 2024-05-27T08:01:32.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:32.694+0000 7fa995b0c700 1 -- 172.21.15.114:0/202313101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990108f10 msgr2=0x7fa99019b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:32.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:32.694+0000 7fa995b0c700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990108f10 0x7fa99019b490 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa984009f50 tx=0x7fa9840037c0 comp rx=0 tx=0).stop 2024-05-27T08:01:32.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:32.694+0000 7fa995b0c700 1 -- 172.21.15.114:0/202313101 shutdown_connections 2024-05-27T08:01:32.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:32.694+0000 7fa995b0c700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa990109860 0x7fa99019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:32.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:32.694+0000 7fa995b0c700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7fa9780b29f0 0x7fa9780b4f00 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:32.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:32.694+0000 7fa995b0c700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9901045e0 0x7fa99019aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:32.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:32.694+0000 7fa995b0c700 1 --2- 172.21.15.114:0/202313101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990108f10 0x7fa99019b490 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:32.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:32.694+0000 7fa995b0c700 1 -- 172.21.15.114:0/202313101 >> 172.21.15.114:0/202313101 conn(0x7fa9900fffd0 msgr2=0x7fa990107000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:32.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:32.694+0000 7fa995b0c700 1 -- 172.21.15.114:0/202313101 shutdown_connections 2024-05-27T08:01:32.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:32.694+0000 7fa995b0c700 1 -- 172.21.15.114:0/202313101 wait complete. 2024-05-27T08:01:32.700 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' removed 2024-05-27T08:01:32.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:32 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/202313101' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T08:01:32.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:32 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T08:01:32.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:32 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/202313101' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T08:01:32.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:32 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T08:01:33.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:32 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/202313101' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T08:01:33.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:32 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T08:01:33.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-05-27T08:01:33.670 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T08:01:33.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:33 smithi163 ceph-mon[36451]: pgmap v759: 97 pgs: 97 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:33.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:33 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/202313101' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T08:01:33.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:33 smithi163 ceph-mon[36451]: osdmap e177: 12 total, 12 up, 12 in 2024-05-27T08:01:33.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T08:01:33.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T08:01:33.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:33 smithi163 ceph-mon[36451]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:01:33.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:33 smithi186 ceph-mon[36647]: pgmap v759: 97 pgs: 97 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:33.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:33 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/202313101' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T08:01:33.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:33 smithi186 ceph-mon[36647]: osdmap e177: 12 total, 12 up, 12 in 2024-05-27T08:01:33.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T08:01:33.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T08:01:33.981 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:33 smithi186 ceph-mon[36647]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:01:34.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:33 smithi114 ceph-mon[32038]: pgmap v759: 97 pgs: 97 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:34.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:33 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/202313101' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T08:01:34.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:33 smithi114 ceph-mon[32038]: osdmap e177: 12 total, 12 up, 12 in 2024-05-27T08:01:34.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T08:01:34.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T08:01:34.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:33 smithi114 ceph-mon[32038]: from='mgr.15138 172.21.15.114:0/926979124' entity='mgr.x' 2024-05-27T08:01:34.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:34 smithi114 ceph-mon[32038]: pgmap v761: 33 pgs: 33 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:34.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:34 smithi114 ceph-mon[32038]: pgmap v762: 33 pgs: 33 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:34.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:34 smithi163 ceph-mon[36451]: pgmap v761: 33 pgs: 33 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:34.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:34 smithi163 ceph-mon[36451]: pgmap v762: 33 pgs: 33 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:34.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:34 smithi186 ceph-mon[36647]: pgmap v761: 33 pgs: 33 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:34.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:34 smithi186 ceph-mon[36647]: pgmap v762: 33 pgs: 33 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:35.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.364+0000 7f7f6b5df700 1 -- 172.21.15.114:0/598428364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f64107e10 msgr2=0x7f7f64108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:35.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.364+0000 7f7f6b5df700 1 --2- 172.21.15.114:0/598428364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f64107e10 0x7f7f64108220 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f7f540099c0 tx=0x7f7f54009cd0 comp rx=0 tx=0).stop 2024-05-27T08:01:35.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.364+0000 7f7f6b5df700 1 -- 172.21.15.114:0/598428364 shutdown_connections 2024-05-27T08:01:35.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.364+0000 7f7f6b5df700 1 --2- 172.21.15.114:0/598428364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f64108760 0x7f7f641100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:35.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.364+0000 7f7f6b5df700 1 --2- 172.21.15.114:0/598428364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f64103530 0x7f7f64103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:35.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.364+0000 7f7f6b5df700 1 --2- 172.21.15.114:0/598428364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f64107e10 0x7f7f64108220 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:35.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.364+0000 7f7f6b5df700 1 -- 172.21.15.114:0/598428364 >> 172.21.15.114:0/598428364 conn(0x7f7f64078e40 msgr2=0x7f7f64079280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:35.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.364+0000 7f7f6b5df700 1 -- 172.21.15.114:0/598428364 shutdown_connections 2024-05-27T08:01:35.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.364+0000 7f7f6b5df700 1 -- 172.21.15.114:0/598428364 wait complete. 2024-05-27T08:01:35.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.365+0000 7f7f6b5df700 1 Processor -- start 2024-05-27T08:01:35.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.365+0000 7f7f6b5df700 1 -- start start 2024-05-27T08:01:35.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.365+0000 7f7f6b5df700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f64103530 0x7f7f6419bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:35.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.365+0000 7f7f6b5df700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f64107e10 0x7f7f6419c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:35.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.365+0000 7f7f6b5df700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f64108760 0x7f7f6419c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:35.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.365+0000 7f7f6b5df700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7f64113370 con 0x7f7f64108760 2024-05-27T08:01:35.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.365+0000 7f7f6b5df700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7f641131f0 con 0x7f7f64103530 2024-05-27T08:01:35.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.365+0000 7f7f6b5df700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7f641134f0 con 0x7f7f64107e10 2024-05-27T08:01:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.365+0000 7f7f69b7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f64108760 0x7f7f6419c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.365+0000 7f7f69b7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f64108760 0x7f7f6419c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54548/0 (socket says 172.21.15.114:54548) 2024-05-27T08:01:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.365+0000 7f7f69b7c700 1 -- 172.21.15.114:0/1029091527 learned_addr learned my addr 172.21.15.114:0/1029091527 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-27T08:01:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.365+0000 7f7f68b7a700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f64107e10 0x7f7f6419c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.365+0000 7f7f6937b700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f64103530 0x7f7f6419bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.366+0000 7f7f69b7c700 1 -- 172.21.15.114:0/1029091527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f64107e10 msgr2=0x7f7f6419c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:35.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.366+0000 7f7f69b7c700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f64107e10 0x7f7f6419c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.366+0000 7f7f69b7c700 1 -- 172.21.15.114:0/1029091527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f64103530 msgr2=0x7f7f6419bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.366+0000 7f7f69b7c700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f64103530 0x7f7f6419bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.366+0000 7f7f69b7c700 1 -- 172.21.15.114:0/1029091527 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f641a7000 con 0x7f7f64108760 2024-05-27T08:01:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.366+0000 7f7f6937b700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f64103530 0x7f7f6419bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T08:01:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.366+0000 7f7f69b7c700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f64108760 0x7f7f6419c750 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c007ee0 tx=0x7f7f5c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.366+0000 7f7f5a7fc700 1 -- 172.21.15.114:0/1029091527 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c00ddd0 con 0x7f7f64108760 2024-05-27T08:01:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.366+0000 7f7f5a7fc700 1 -- 172.21.15.114:0/1029091527 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f5c004020 con 0x7f7f64108760 2024-05-27T08:01:35.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.366+0000 7f7f6b5df700 1 -- 172.21.15.114:0/1029091527 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f641a72f0 con 0x7f7f64108760 2024-05-27T08:01:35.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.367+0000 7f7f5a7fc700 1 -- 172.21.15.114:0/1029091527 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c015660 con 0x7f7f64108760 2024-05-27T08:01:35.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.367+0000 7f7f6b5df700 1 -- 172.21.15.114:0/1029091527 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f641a77b0 con 0x7f7f64108760 2024-05-27T08:01:35.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.368+0000 7f7f6b5df700 1 -- 172.21.15.114:0/1029091527 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f64104990 con 0x7f7f64108760 2024-05-27T08:01:35.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.369+0000 7f7f5a7fc700 1 -- 172.21.15.114:0/1029091527 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 149784+0+0 (secure 0 0 0) 0x7f7f5c003a00 con 0x7f7f64108760 2024-05-27T08:01:35.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.370+0000 7f7f5a7fc700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7f500b28d0 0x7f7f500b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T08:01:35.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.370+0000 7f7f6937b700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7f500b28d0 0x7f7f500b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T08:01:35.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.370+0000 7f7f5a7fc700 1 -- 172.21.15.114:0/1029091527 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 8410+0+0 (secure 0 0 0) 0x7f7f5c0e1f70 con 0x7f7f64108760 2024-05-27T08:01:35.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.370+0000 7f7f6937b700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7f500b28d0 0x7f7f500b4de0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7f60005950 tx=0x7f7f600058e0 comp rx=0 tx=0).ready entity=mgr.15138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T08:01:35.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.375+0000 7f7f5a7fc700 1 -- 172.21.15.114:0/1029091527 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7f5c010070 con 0x7f7f64108760 2024-05-27T08:01:35.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.583+0000 7f7f6b5df700 1 -- 172.21.15.114:0/1029091527 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7f640512b0 con 0x7f7f64108760 2024-05-27T08:01:35.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.707+0000 7f7f5a7fc700 1 -- 172.21.15.114:0/1029091527 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v178) ==== 178+0+0 (secure 0 0 0) 0x7f7f5c0a99c0 con 0x7f7f64108760 2024-05-27T08:01:35.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.709+0000 7f7f6b5df700 1 -- 172.21.15.114:0/1029091527 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7f500b28d0 msgr2=0x7f7f500b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:35.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.709+0000 7f7f6b5df700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7f500b28d0 0x7f7f500b4de0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7f60005950 tx=0x7f7f600058e0 comp rx=0 tx=0).stop 2024-05-27T08:01:35.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.709+0000 7f7f6b5df700 1 -- 172.21.15.114:0/1029091527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f64108760 msgr2=0x7f7f6419c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T08:01:35.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.709+0000 7f7f6b5df700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f64108760 0x7f7f6419c750 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c007ee0 tx=0x7f7f5c00d3b0 comp rx=0 tx=0).stop 2024-05-27T08:01:35.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.709+0000 7f7f6b5df700 1 -- 172.21.15.114:0/1029091527 shutdown_connections 2024-05-27T08:01:35.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.709+0000 7f7f6b5df700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f64108760 0x7f7f6419c750 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:35.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.709+0000 7f7f6b5df700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.114:6800/3311646739,v1:172.21.15.114:6801/3311646739] conn(0x7f7f500b28d0 0x7f7f500b4de0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:35.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.709+0000 7f7f6b5df700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f64103530 0x7f7f6419bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:35.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.709+0000 7f7f6b5df700 1 --2- 172.21.15.114:0/1029091527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f64107e10 0x7f7f6419c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T08:01:35.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.709+0000 7f7f6b5df700 1 -- 172.21.15.114:0/1029091527 >> 172.21.15.114:0/1029091527 conn(0x7f7f64078e40 msgr2=0x7f7f64100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T08:01:35.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.709+0000 7f7f6b5df700 1 -- 172.21.15.114:0/1029091527 shutdown_connections 2024-05-27T08:01:35.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-27T08:01:35.710+0000 7f7f6b5df700 1 -- 172.21.15.114:0/1029091527 wait complete. 2024-05-27T08:01:35.713 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data_ec' removed 2024-05-27T08:01:35.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:35 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1029091527' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T08:01:35.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:35 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1029091527' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T08:01:36.019 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:35 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1029091527' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T08:01:36.346 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T08:01:36.374 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T08:01:36.400 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T08:01:36.470 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-27T08:01:36.470 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T08:01:36.497 DEBUG:teuthology.orchestra.run.smithi163:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T08:01:36.516 DEBUG:teuthology.orchestra.run.smithi186:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T08:01:36.540 INFO:tasks.cephadm:Stopping all daemons... 2024-05-27T08:01:36.541 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-27T08:01:36.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.a 2024-05-27T08:01:36.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:36 smithi163 ceph-mon[36451]: pgmap v763: 33 pgs: 33 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:36.978 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:36 smithi163 ceph-mon[36451]: from='client.? 172.21.15.114:0/1029091527' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T08:01:36.979 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:36 smithi163 ceph-mon[36451]: osdmap e178: 12 total, 12 up, 12 in 2024-05-27T08:01:36.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:36 smithi186 ceph-mon[36647]: pgmap v763: 33 pgs: 33 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:36.981 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:36 smithi186 ceph-mon[36647]: from='client.? 172.21.15.114:0/1029091527' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T08:01:36.981 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:36 smithi186 ceph-mon[36647]: osdmap e178: 12 total, 12 up, 12 in 2024-05-27T08:01:37.037 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:36 smithi114 systemd[1]: Stopping Ceph mon.a for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:01:37.038 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:36 smithi114 ceph-mon[32038]: pgmap v763: 33 pgs: 33 active+clean; 320 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T08:01:37.038 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:36 smithi114 ceph-mon[32038]: from='client.? 172.21.15.114:0/1029091527' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T08:01:37.038 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:36 smithi114 ceph-mon[32038]: osdmap e178: 12 total, 12 up, 12 in 2024-05-27T08:01:37.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:37 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a[32034]: 2024-05-27T08:01:37.034+0000 7f0d4dfb1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:01:37.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:37 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a[32034]: 2024-05-27T08:01:37.034+0000 7f0d4dfb1700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-27T08:01:37.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:37 smithi114 podman[139465]: 2024-05-27 08:01:37.10695828 +0000 UTC m=+0.218179847 container died b13d09b4408fae57d961203d4b3dc4b83a85fd2fb43b7045e0d030c234ec1f09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T08:01:37.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:37 smithi114 podman[139465]: 2024-05-27 08:01:37.490736959 +0000 UTC m=+0.601958512 container cleanup b13d09b4408fae57d961203d4b3dc4b83a85fd2fb43b7045e0d030c234ec1f09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T08:01:37.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:37 smithi114 bash[139465]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a 2024-05-27T08:01:37.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.a.service' 2024-05-27T08:01:38.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:37 smithi114 podman[139476]: 2024-05-27 08:01:37.800796452 +0000 UTC m=+0.681264418 container remove b13d09b4408fae57d961203d4b3dc4b83a85fd2fb43b7045e0d030c234ec1f09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T08:01:38.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:37 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-27T08:01:38.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:37 smithi114 systemd[1]: Stopped Ceph mon.a for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:01:38.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:May 27 08:01:37 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.a.service: Consumed 30.982s CPU time. 2024-05-27T08:01:38.071 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:01:38.072 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-27T08:01:38.072 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-27T08:01:38.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.b 2024-05-27T08:01:38.405 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:38 smithi163 systemd[1]: Stopping Ceph mon.b for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:01:38.669 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:38 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-b[36447]: 2024-05-27T08:01:38.402+0000 7fb51458d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:01:38.670 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:38 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-b[36447]: 2024-05-27T08:01:38.402+0000 7fb51458d700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-27T08:01:38.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:38 smithi163 podman[76674]: 2024-05-27 08:01:38.66848146 +0000 UTC m=+0.401353820 container died 61a1565b03aed0cadf900de1da0cfe8ea0dddc3559430fac53d3abf409abab75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD) 2024-05-27T08:01:39.477 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:39 smithi163 podman[76674]: 2024-05-27 08:01:39.187995946 +0000 UTC m=+0.920868327 container cleanup 61a1565b03aed0cadf900de1da0cfe8ea0dddc3559430fac53d3abf409abab75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-27T08:01:39.478 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:39 smithi163 bash[76674]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-b 2024-05-27T08:01:39.786 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.b.service' 2024-05-27T08:01:39.808 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:39 smithi163 podman[76686]: 2024-05-27 08:01:39.548346317 +0000 UTC m=+0.880637469 container remove 61a1565b03aed0cadf900de1da0cfe8ea0dddc3559430fac53d3abf409abab75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-27T08:01:39.808 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:39 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.b.service: Deactivated successfully. 2024-05-27T08:01:39.808 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:39 smithi163 systemd[1]: Stopped Ceph mon.b for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:01:39.809 INFO:journalctl@ceph.mon.b.smithi163.stdout:May 27 08:01:39 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.b.service: Consumed 19.710s CPU time. 2024-05-27T08:01:39.828 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:01:39.828 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-27T08:01:39.829 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-27T08:01:39.829 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.c 2024-05-27T08:01:40.216 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:39 smithi186 systemd[1]: Stopping Ceph mon.c for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:01:40.481 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:40 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-c[36621]: 2024-05-27T08:01:40.213+0000 7fe908549700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:01:40.481 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:40 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-c[36621]: 2024-05-27T08:01:40.213+0000 7fe908549700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-27T08:01:40.481 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:40 smithi186 podman[77784]: 2024-05-27 08:01:40.378248018 +0000 UTC m=+0.354570748 container died 497b6738db89c9e8eb4b680467c2da43ef047f64336f6763c68bbac0501fe068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-05-27T08:01:40.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:40 smithi186 podman[77784]: 2024-05-27 08:01:40.923981399 +0000 UTC m=+0.900304175 container cleanup 497b6738db89c9e8eb4b680467c2da43ef047f64336f6763c68bbac0501fe068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=) 2024-05-27T08:01:40.980 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 27 08:01:40 smithi186 bash[77784]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mon-c 2024-05-27T08:01:41.418 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mon.c.service' 2024-05-27T08:01:41.462 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:01:41.462 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-27T08:01:41.463 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-27T08:01:41.463 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.x 2024-05-27T08:01:41.782 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 08:01:41 smithi114 systemd[1]: Stopping Ceph mgr.x for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:01:42.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 08:01:41 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x[32267]: 2024-05-27T08:01:41.779+0000 7f55c9dfe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:01:42.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 08:01:41 smithi114 podman[139581]: 2024-05-27 08:01:41.820999298 +0000 UTC m=+0.162151586 container died 4289f83cbe60b2bdd7a150e823213e8e43ba27b27ec11fc271c5327e61966d1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-27T08:01:42.466 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 08:01:42 smithi114 podman[139581]: 2024-05-27 08:01:42.237913311 +0000 UTC m=+0.579065599 container cleanup 4289f83cbe60b2bdd7a150e823213e8e43ba27b27ec11fc271c5327e61966d1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T08:01:42.466 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 08:01:42 smithi114 bash[139581]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x 2024-05-27T08:01:42.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.x.service' 2024-05-27T08:01:42.729 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 08:01:42 smithi114 podman[139593]: 2024-05-27 08:01:42.464990125 +0000 UTC m=+0.646673084 container remove 4289f83cbe60b2bdd7a150e823213e8e43ba27b27ec11fc271c5327e61966d1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T08:01:42.729 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 08:01:42 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.x.service: Deactivated successfully. 2024-05-27T08:01:42.729 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 08:01:42 smithi114 systemd[1]: Stopped Ceph mgr.x for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:01:42.729 INFO:journalctl@ceph.mgr.x.smithi114.stdout:May 27 08:01:42 smithi114 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.x.service: Consumed 1min 40.631s CPU time. 2024-05-27T08:01:42.743 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:01:42.743 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-27T08:01:42.743 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-27T08:01:42.744 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.y 2024-05-27T08:01:43.083 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 08:01:42 smithi163 systemd[1]: Stopping Ceph mgr.y for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:01:43.383 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 08:01:43 smithi163 podman[76797]: 2024-05-27 08:01:43.082410985 +0000 UTC m=+0.137963245 container died a3f682a56a5e28985ed87b9e022ceb5c44bcb26c5cf7bd834cb4b5293457893b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-27T08:01:43.727 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 08:01:43 smithi163 podman[76797]: 2024-05-27 08:01:43.523336887 +0000 UTC m=+0.578889147 container cleanup a3f682a56a5e28985ed87b9e022ceb5c44bcb26c5cf7bd834cb4b5293457893b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T08:01:43.728 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 08:01:43 smithi163 bash[76797]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y 2024-05-27T08:01:44.009 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.y.service' 2024-05-27T08:01:44.030 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 08:01:43 smithi163 podman[76811]: 2024-05-27 08:01:43.766988275 +0000 UTC m=+0.682546427 container remove a3f682a56a5e28985ed87b9e022ceb5c44bcb26c5cf7bd834cb4b5293457893b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS) 2024-05-27T08:01:44.030 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 08:01:43 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-05-27T08:01:44.030 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 08:01:44 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-05-27T08:01:44.030 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 08:01:44 smithi163 systemd[1]: Stopped Ceph mgr.y for 755b3728-1bfa-11ef-bc9b-c7b262605968. 2024-05-27T08:01:44.030 INFO:journalctl@ceph.mgr.y.smithi163.stdout:May 27 08:01:44 smithi163 systemd[1]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.y.service: Consumed 23.949s CPU time. 2024-05-27T08:01:44.042 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:01:44.042 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-27T08:01:44.042 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-27T08:01:44.042 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.z 2024-05-27T08:01:44.470 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 08:01:44 smithi186 systemd[1]: Stopping Ceph mgr.z for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:01:44.730 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 08:01:44 smithi186 podman[77904]: 2024-05-27 08:01:44.468699844 +0000 UTC m=+0.236499925 container died 32c8c30f71f14326c96fddcba2848a053d17e18843c4bf1c11d9868d958bc019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True) 2024-05-27T08:01:45.375 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 08:01:45 smithi186 podman[77904]: 2024-05-27 08:01:45.063857328 +0000 UTC m=+0.831657414 container cleanup 32c8c30f71f14326c96fddcba2848a053d17e18843c4bf1c11d9868d958bc019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD) 2024-05-27T08:01:45.375 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 27 08:01:45 smithi186 bash[77904]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mgr-z 2024-05-27T08:01:45.552 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mgr.z.service' 2024-05-27T08:01:45.584 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:01:45.585 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-27T08:01:45.585 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-27T08:01:45.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.0 2024-05-27T08:01:45.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:45 smithi114 systemd[1]: Stopping Ceph osd.0 for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:01:46.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:45 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0[44060]: 2024-05-27T08:01:45.909+0000 7ff89c7cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:01:46.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:45 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0[44060]: 2024-05-27T08:01:45.909+0000 7ff89c7cc700 -1 osd.0 178 *** Got signal Terminated *** 2024-05-27T08:01:46.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:45 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0[44060]: 2024-05-27T08:01:45.909+0000 7ff89c7cc700 -1 osd.0 178 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T08:01:51.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:51 smithi114 podman[139704]: 2024-05-27 08:01:51.348925605 +0000 UTC m=+5.560235374 container died 64f415f677c1b2a4631c66d0c47fb09596f1adf6e2f646d1a0125af44167ae15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-27T08:01:52.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:51 smithi114 podman[139704]: 2024-05-27 08:01:51.843665472 +0000 UTC m=+6.054975243 container cleanup 64f415f677c1b2a4631c66d0c47fb09596f1adf6e2f646d1a0125af44167ae15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True) 2024-05-27T08:01:52.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:51 smithi114 bash[139704]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0 2024-05-27T08:01:52.472 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:52 smithi114 podman[139733]: 2024-05-27 08:01:52.12941168 +0000 UTC m=+0.773991998 container remove 64f415f677c1b2a4631c66d0c47fb09596f1adf6e2f646d1a0125af44167ae15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T08:01:52.751 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:52 smithi114 podman[139823]: 2024-05-27 08:01:52.374369477 +0000 UTC m=+0.017058447 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:01:52.751 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:52 smithi114 podman[139823]: 2024-05-27 08:01:52.505417831 +0000 UTC m=+0.148106798 container create fe2309037c6002f24aca1f7f1a2ae9b2f95786d4c9589d45736c02cd07657812 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-27T08:01:52.751 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:52 smithi114 podman[139823]: 2024-05-27 08:01:52.749921857 +0000 UTC m=+0.392610833 container init fe2309037c6002f24aca1f7f1a2ae9b2f95786d4c9589d45736c02cd07657812 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T08:01:53.058 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:52 smithi114 podman[139823]: 2024-05-27 08:01:52.753435747 +0000 UTC m=+0.396124711 container start fe2309037c6002f24aca1f7f1a2ae9b2f95786d4c9589d45736c02cd07657812 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-27T08:01:53.058 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:52 smithi114 podman[139823]: 2024-05-27 08:01:52.866632735 +0000 UTC m=+0.509321706 container attach fe2309037c6002f24aca1f7f1a2ae9b2f95786d4c9589d45736c02cd07657812 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:01:53.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:May 27 08:01:53 smithi114 podman[139823]: 2024-05-27 08:01:53.081641334 +0000 UTC m=+0.724330308 container died fe2309037c6002f24aca1f7f1a2ae9b2f95786d4c9589d45736c02cd07657812 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-27T08:01:53.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.0.service' 2024-05-27T08:01:53.809 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:01:53.809 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-27T08:01:53.809 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-27T08:01:53.809 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.1 2024-05-27T08:01:54.138 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:01:53 smithi163 systemd[1]: Stopping Ceph osd.1 for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:01:54.477 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:01:54 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1[40993]: 2024-05-27T08:01:54.136+0000 7f08e9bd9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:01:54.478 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:01:54 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1[40993]: 2024-05-27T08:01:54.136+0000 7f08e9bd9700 -1 osd.1 178 *** Got signal Terminated *** 2024-05-27T08:01:54.478 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:01:54 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1[40993]: 2024-05-27T08:01:54.136+0000 7f08e9bd9700 -1 osd.1 178 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T08:01:59.930 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:01:59 smithi163 podman[76919]: 2024-05-27 08:01:59.638115765 +0000 UTC m=+5.621666793 container died 89eeef1de66c10245ab91a0e0ffbc0c8aa1d09d1241ef818e2fbc7247219de5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:02:00.227 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:02:00 smithi163 podman[76919]: 2024-05-27 08:02:00.028207627 +0000 UTC m=+6.011758649 container cleanup 89eeef1de66c10245ab91a0e0ffbc0c8aa1d09d1241ef818e2fbc7247219de5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS) 2024-05-27T08:02:00.228 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:02:00 smithi163 bash[76919]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1 2024-05-27T08:02:00.696 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:02:00 smithi163 podman[76937]: 2024-05-27 08:02:00.262329511 +0000 UTC m=+0.627185643 container remove 89eeef1de66c10245ab91a0e0ffbc0c8aa1d09d1241ef818e2fbc7247219de5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5) 2024-05-27T08:02:00.977 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:02:00 smithi163 podman[77026]: 2024-05-27 08:02:00.598030266 +0000 UTC m=+0.017182051 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:02:00.978 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:02:00 smithi163 podman[77026]: 2024-05-27 08:02:00.713266801 +0000 UTC m=+0.132418563 container create 2c69676d8c1db124f5ac5b9c17c299a1d6e0ad0fff68e8d9f182038c2c769ceb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521) 2024-05-27T08:02:00.978 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:02:00 smithi163 podman[77026]: 2024-05-27 08:02:00.941157817 +0000 UTC m=+0.360309584 container init 2c69676d8c1db124f5ac5b9c17c299a1d6e0ad0fff68e8d9f182038c2c769ceb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T08:02:00.978 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:02:00 smithi163 podman[77026]: 2024-05-27 08:02:00.944433194 +0000 UTC m=+0.363584961 container start 2c69676d8c1db124f5ac5b9c17c299a1d6e0ad0fff68e8d9f182038c2c769ceb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T08:02:01.275 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:02:01 smithi163 podman[77026]: 2024-05-27 08:02:01.041310533 +0000 UTC m=+0.460462301 container attach 2c69676d8c1db124f5ac5b9c17c299a1d6e0ad0fff68e8d9f182038c2c769ceb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-27T08:02:01.275 INFO:journalctl@ceph.osd.1.smithi163.stdout:May 27 08:02:01 smithi163 podman[77026]: 2024-05-27 08:02:01.274016595 +0000 UTC m=+0.693168375 container died 2c69676d8c1db124f5ac5b9c17c299a1d6e0ad0fff68e8d9f182038c2c769ceb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T08:02:02.017 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.1.service' 2024-05-27T08:02:02.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:02:02.055 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-27T08:02:02.056 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-27T08:02:02.056 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.2 2024-05-27T08:02:02.395 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:02 smithi186 systemd[1]: Stopping Ceph osd.2 for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:02:02.730 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:02 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2[41791]: 2024-05-27T08:02:02.392+0000 7f87543b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:02:02.731 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:02 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2[41791]: 2024-05-27T08:02:02.392+0000 7f87543b1700 -1 osd.2 178 *** Got signal Terminated *** 2024-05-27T08:02:02.731 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:02 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2[41791]: 2024-05-27T08:02:02.392+0000 7f87543b1700 -1 osd.2 178 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T08:02:08.204 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:07 smithi186 podman[78029]: 2024-05-27 08:02:07.900596119 +0000 UTC m=+5.650937154 container died 9c78ea804f682be487a566c03d603e991475afe8de9dd58f9252f59497c13d4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T08:02:08.480 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:08 smithi186 podman[78029]: 2024-05-27 08:02:08.293691796 +0000 UTC m=+6.044032814 container cleanup 9c78ea804f682be487a566c03d603e991475afe8de9dd58f9252f59497c13d4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T08:02:08.481 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:08 smithi186 bash[78029]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2 2024-05-27T08:02:08.911 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:08 smithi186 podman[78046]: 2024-05-27 08:02:08.561966454 +0000 UTC m=+0.664754686 container remove 9c78ea804f682be487a566c03d603e991475afe8de9dd58f9252f59497c13d4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:02:09.216 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:08 smithi186 podman[78132]: 2024-05-27 08:02:08.812817037 +0000 UTC m=+0.017031913 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:02:09.216 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:09 smithi186 podman[78132]: 2024-05-27 08:02:09.055031411 +0000 UTC m=+0.259246266 container create 4bae0c2c30008436a2bc28f85930a4c00767a0c3ed6a5e80a7defa2d0e08081a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-27T08:02:09.481 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:09 smithi186 podman[78132]: 2024-05-27 08:02:09.316299131 +0000 UTC m=+0.520513996 container init 4bae0c2c30008436a2bc28f85930a4c00767a0c3ed6a5e80a7defa2d0e08081a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-27T08:02:09.481 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:09 smithi186 podman[78132]: 2024-05-27 08:02:09.329852835 +0000 UTC m=+0.534067698 container start 4bae0c2c30008436a2bc28f85930a4c00767a0c3ed6a5e80a7defa2d0e08081a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2) 2024-05-27T08:02:09.481 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:09 smithi186 podman[78132]: 2024-05-27 08:02:09.433108809 +0000 UTC m=+0.637323699 container attach 4bae0c2c30008436a2bc28f85930a4c00767a0c3ed6a5e80a7defa2d0e08081a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-27T08:02:10.138 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 27 08:02:09 smithi186 podman[78132]: 2024-05-27 08:02:09.832679791 +0000 UTC m=+1.036894654 container died 4bae0c2c30008436a2bc28f85930a4c00767a0c3ed6a5e80a7defa2d0e08081a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T08:02:10.464 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.2.service' 2024-05-27T08:02:10.503 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:02:10.503 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-27T08:02:10.503 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-27T08:02:10.503 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.3 2024-05-27T08:02:10.808 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:10 smithi114 systemd[1]: Stopping Ceph osd.3 for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:02:11.308 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:10 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3[50652]: 2024-05-27T08:02:10.831+0000 7fde58240700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:02:11.308 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:10 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3[50652]: 2024-05-27T08:02:10.831+0000 7fde58240700 -1 osd.3 178 *** Got signal Terminated *** 2024-05-27T08:02:11.309 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:10 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3[50652]: 2024-05-27T08:02:10.831+0000 7fde58240700 -1 osd.3 178 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T08:02:12.231 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:11 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:11.960+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:12.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:12 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:12.505+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:13.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:12 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:12.997+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:13.522 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:13 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3[50652]: 2024-05-27T08:02:13.228+0000 7fde507ee700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.258071+0000 front 2024-05-27T08:01:49.258031+0000 (oldest deadline 2024-05-27T08:02:12.758150+0000) 2024-05-27T08:02:13.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:13 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:13.521+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:14.227 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:13.808+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:14.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:13 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:13.983+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:14.523 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3[50652]: 2024-05-27T08:02:14.229+0000 7fde507ee700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.258071+0000 front 2024-05-27T08:01:49.258031+0000 (oldest deadline 2024-05-27T08:02:12.758150+0000) 2024-05-27T08:02:14.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:14 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:14.522+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:15.227 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:14 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:14.815+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:15.245 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:14 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:14.987+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:15.246 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:15.195+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:15.558 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3[50652]: 2024-05-27T08:02:15.261+0000 7fde507ee700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.258071+0000 front 2024-05-27T08:01:49.258031+0000 (oldest deadline 2024-05-27T08:02:12.758150+0000) 2024-05-27T08:02:15.730 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:15.244+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:15.828 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:15 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:15.570+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:16.227 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:15 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:15.850+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:16.230 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:16.151+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:16.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:16.195+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:16.231 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:15.967+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:16.539 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:16 smithi114 podman[139929]: 2024-05-27 08:02:16.328910921 +0000 UTC m=+5.628187038 container died b429a7d9ff37492a10a8c6e71f1030c3fc8c3fbf472b6085544906e4f24795f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521) 2024-05-27T08:02:16.539 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:16.233+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:16.794 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:16 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:16.537+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:17.058 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:16 smithi114 podman[139929]: 2024-05-27 08:02:16.865722995 +0000 UTC m=+6.164999107 container cleanup b429a7d9ff37492a10a8c6e71f1030c3fc8c3fbf472b6085544906e4f24795f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-27T08:02:17.058 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:16 smithi114 bash[139929]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3 2024-05-27T08:02:17.185 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:16.930+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:17.227 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:16 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:16.845+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:17.480 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:17.183+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:17.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:17.192+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:17.533 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:17 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:17.188+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:17.533 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:17 smithi114 podman[139955]: 2024-05-27 08:02:17.242741194 +0000 UTC m=+0.911768048 container remove b429a7d9ff37492a10a8c6e71f1030c3fc8c3fbf472b6085544906e4f24795f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2) 2024-05-27T08:02:17.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:17 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:17.531+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:17.808 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:17 smithi114 podman[140047]: 2024-05-27 08:02:17.631603873 +0000 UTC m=+0.017302460 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:02:17.808 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:17 smithi114 podman[140047]: 2024-05-27 08:02:17.760372004 +0000 UTC m=+0.146070591 container create 5e6fafd299c63287d703a1c0c6967717c34d603c243bc27cb41276295d49f343 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-27T08:02:18.170 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:18.146+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:18.170 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:17.918+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:18.199 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:18 smithi114 podman[140047]: 2024-05-27 08:02:18.013219104 +0000 UTC m=+0.398917701 container init 5e6fafd299c63287d703a1c0c6967717c34d603c243bc27cb41276295d49f343 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-27T08:02:18.199 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:18 smithi114 podman[140047]: 2024-05-27 08:02:18.016912538 +0000 UTC m=+0.402611129 container start 5e6fafd299c63287d703a1c0c6967717c34d603c243bc27cb41276295d49f343 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T08:02:18.199 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:18 smithi114 podman[140047]: 2024-05-27 08:02:18.096157714 +0000 UTC m=+0.481856303 container attach 5e6fafd299c63287d703a1c0c6967717c34d603c243bc27cb41276295d49f343 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-05-27T08:02:18.227 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:17 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:17.889+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:18.480 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:18.168+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:18.524 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:18 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:18.197+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:18.525 INFO:journalctl@ceph.osd.3.smithi114.stdout:May 27 08:02:18 smithi114 podman[140047]: 2024-05-27 08:02:18.30587327 +0000 UTC m=+0.691571861 container died 5e6fafd299c63287d703a1c0c6967717c34d603c243bc27cb41276295d49f343 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T08:02:18.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:18 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:18.523+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:19.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.3.service' 2024-05-27T08:02:19.169 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:02:19.169 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-27T08:02:19.169 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-27T08:02:19.169 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.4 2024-05-27T08:02:19.196 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:18 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:18.870+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:19.230 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:18.939+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:19.230 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:19.172+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:19.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:19.171+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:19.477 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:19 smithi163 systemd[1]: Stopping Ceph osd.4 for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:02:19.499 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:19 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:19.206+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:19.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:19 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:19.497+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:19.860 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4[47519]: 2024-05-27T08:02:19.497+0000 7fc772bd3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:02:19.860 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4[47519]: 2024-05-27T08:02:19.497+0000 7fc772bd3700 -1 osd.4 178 *** Got signal Terminated *** 2024-05-27T08:02:19.860 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4[47519]: 2024-05-27T08:02:19.497+0000 7fc772bd3700 -1 osd.4 178 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T08:02:20.227 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:19.859+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:20.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:19 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:19.859+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:20.230 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:19.923+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:20.230 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:20.161+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:20.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:20.167+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:20.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:20.167+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:20.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:20.482+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:20.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:20.241+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:20.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:20 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:20.241+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:21.218 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:21 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:21.129+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:21.218 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:20.960+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:21.227 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:20 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:20.894+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:21.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:20 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:20.894+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:21.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:21 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:21.216+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:21.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:21 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:21.216+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:21.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:21.462+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:21.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:21.260+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:21.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:21 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:21.260+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:21.884 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:21 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4[47519]: 2024-05-27T08:02:21.528+0000 7fc76f189700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.100961+0000 front 2024-05-27T08:01:57.100940+0000 (oldest deadline 2024-05-27T08:02:21.201027+0000) 2024-05-27T08:02:22.160 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:21 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:21.949+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:22.227 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:21 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:21.882+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:22.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:21 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:21.882+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:22.480 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:22 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:22.158+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:22.480 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:22 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:22.158+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.678070+0000 front 2024-05-27T08:01:57.678100+0000 (oldest deadline 2024-05-27T08:02:21.178073+0000) 2024-05-27T08:02:22.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:22 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:22.265+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:22.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:22 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:22.265+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:22.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:22 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:22.418+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:22.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:22 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:22.288+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:22.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:22 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:22.288+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:22.780 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:22 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4[47519]: 2024-05-27T08:02:22.479+0000 7fc76f189700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.100961+0000 front 2024-05-27T08:01:57.100940+0000 (oldest deadline 2024-05-27T08:02:21.201027+0000) 2024-05-27T08:02:23.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:22 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:22.887+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:23.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:22 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:22.887+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:23.230 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:22 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:22.977+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:23.231 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:23 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:23.129+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:23.231 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:23 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:23.129+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.678070+0000 front 2024-05-27T08:01:57.678100+0000 (oldest deadline 2024-05-27T08:02:21.178073+0000) 2024-05-27T08:02:23.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:23 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:23.218+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:23.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:23 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:23.218+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:23.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:23 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:23.436+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:23.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:23 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:23.262+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:23.559 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:23 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:23.262+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:23.727 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:23 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4[47519]: 2024-05-27T08:02:23.526+0000 7fc76f189700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.100961+0000 front 2024-05-27T08:01:57.100940+0000 (oldest deadline 2024-05-27T08:02:21.201027+0000) 2024-05-27T08:02:24.126 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:23 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:23.839+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:24.126 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:23 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:23.839+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:24.477 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:24 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:24.123+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:24.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:24 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:24.001+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:24.481 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:24 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:24.104+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:24.481 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:24 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:24.104+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.678070+0000 front 2024-05-27T08:01:57.678100+0000 (oldest deadline 2024-05-27T08:02:21.178073+0000) 2024-05-27T08:02:24.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:24 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:24.192+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:24.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:24 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:24.192+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:24.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:24 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:24.245+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:24.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:24 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:24.245+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:24.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:24 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:24.433+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:24.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:24 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:24.433+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:24.864 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:24 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4[47519]: 2024-05-27T08:02:24.480+0000 7fc76f189700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.100961+0000 front 2024-05-27T08:01:57.100940+0000 (oldest deadline 2024-05-27T08:02:21.201027+0000) 2024-05-27T08:02:25.120 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:25 smithi163 podman[77148]: 2024-05-27 08:02:25.051873836 +0000 UTC m=+5.690613185 container stop e332dc16ccba212df009160232c007554d9dc2fc0af07d7f8c9609ff57f9e74e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS) 2024-05-27T08:02:25.121 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:25 smithi163 podman[77148]: 2024-05-27 08:02:25.077462292 +0000 UTC m=+5.716201650 container died e332dc16ccba212df009160232c007554d9dc2fc0af07d7f8c9609ff57f9e74e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T08:02:25.121 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:24 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:24.863+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:25.121 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:24 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:24.863+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:25.376 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:25 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:25.118+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:25.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:24 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:24.980+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:25.481 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:25 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:25.120+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:25.481 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:25 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:25.120+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.678070+0000 front 2024-05-27T08:01:57.678100+0000 (oldest deadline 2024-05-27T08:02:21.178073+0000) 2024-05-27T08:02:25.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:25 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:25.171+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:25.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:25 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:25.171+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:25.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:25 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:25.263+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:25.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:25 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:25.263+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:25.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:25 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:25.451+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:25.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:25 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:25.451+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:25.693 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:25 smithi163 podman[77148]: 2024-05-27 08:02:25.473480488 +0000 UTC m=+6.112219846 container cleanup e332dc16ccba212df009160232c007554d9dc2fc0af07d7f8c9609ff57f9e74e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T08:02:25.693 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:25 smithi163 bash[77148]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4 2024-05-27T08:02:25.977 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:25 smithi163 podman[77165]: 2024-05-27 08:02:25.691825158 +0000 UTC m=+0.633843254 container remove e332dc16ccba212df009160232c007554d9dc2fc0af07d7f8c9609ff57f9e74e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-27T08:02:25.978 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:25 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:25.895+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:25.978 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:25 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:25.895+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:26.388 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:26.115+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:26.388 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:26 smithi163 podman[77254]: 2024-05-27 08:02:26.037638043 +0000 UTC m=+0.017028152 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:02:26.388 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:26 smithi163 podman[77254]: 2024-05-27 08:02:26.151074855 +0000 UTC m=+0.130464949 container create 81374a77ab395eb0e11e76cd9320e38ccb66b95f4c7c75d91a4caea6191a13b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-27T08:02:26.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:25 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:25.981+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:26.481 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:26 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:26.112+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:26.481 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:26 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:26.112+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.678070+0000 front 2024-05-27T08:01:57.678100+0000 (oldest deadline 2024-05-27T08:02:21.178073+0000) 2024-05-27T08:02:26.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:26 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:26.128+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:26.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:26 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:26.128+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:26.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:26 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:26.247+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:26.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:26 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:26.247+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:26.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:26 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:26.460+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:26.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:26 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:26.460+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:26.684 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:26 smithi163 podman[77254]: 2024-05-27 08:02:26.387020518 +0000 UTC m=+0.366410618 container init 81374a77ab395eb0e11e76cd9320e38ccb66b95f4c7c75d91a4caea6191a13b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD) 2024-05-27T08:02:26.684 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:26 smithi163 podman[77254]: 2024-05-27 08:02:26.390344684 +0000 UTC m=+0.369734787 container start 81374a77ab395eb0e11e76cd9320e38ccb66b95f4c7c75d91a4caea6191a13b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-27T08:02:26.684 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:26 smithi163 podman[77254]: 2024-05-27 08:02:26.50395691 +0000 UTC m=+0.483347013 container attach 81374a77ab395eb0e11e76cd9320e38ccb66b95f4c7c75d91a4caea6191a13b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:02:26.939 INFO:journalctl@ceph.osd.4.smithi163.stdout:May 27 08:02:26 smithi163 podman[77254]: 2024-05-27 08:02:26.683077688 +0000 UTC m=+0.662467798 container died 81374a77ab395eb0e11e76cd9320e38ccb66b95f4c7c75d91a4caea6191a13b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-27T08:02:27.227 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:27.114+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:27.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:26.936+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:27.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:26 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:26.936+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:27.230 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:26 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:26.971+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:27.231 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:27 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:27.105+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:27.231 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:27 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:27.105+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.678070+0000 front 2024-05-27T08:01:57.678100+0000 (oldest deadline 2024-05-27T08:02:21.178073+0000) 2024-05-27T08:02:27.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:27 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:27.104+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:27.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:27 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:27.104+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:27.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.4.service' 2024-05-27T08:02:27.452 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:02:27.452 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-27T08:02:27.452 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-27T08:02:27.453 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.5 2024-05-27T08:02:27.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:27 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:27.437+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:27.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:27 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:27.437+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:27.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:27 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:27.267+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:27.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:27 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:27.267+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:27.730 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:27 smithi186 systemd[1]: Stopping Ceph osd.5 for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:02:27.731 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:27 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:27.723+0000 7f1e8fd19700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:02:27.731 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:27 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:27.723+0000 7f1e8fd19700 -1 osd.5 178 *** Got signal Terminated *** 2024-05-27T08:02:27.731 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:27 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:27.723+0000 7f1e8fd19700 -1 osd.5 178 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T08:02:28.230 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:27 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:27.923+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:28.231 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:28 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:28.113+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:28.231 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:28 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:28.113+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.678070+0000 front 2024-05-27T08:01:57.678100+0000 (oldest deadline 2024-05-27T08:02:21.178073+0000) 2024-05-27T08:02:28.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:28 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:28.141+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:28.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:28 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:28.141+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:28.477 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:28 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:28.092+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:28.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:27.981+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:28.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:27 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:27.981+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:28.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:28 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:28.426+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:28.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:28 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:28.426+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:28.559 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:28 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:28.297+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:28.559 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:28 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:28.297+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:29.230 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:28 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:28.924+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:29.231 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:29 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:29.073+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:29.231 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:29 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:29.073+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.678070+0000 front 2024-05-27T08:01:57.678100+0000 (oldest deadline 2024-05-27T08:02:21.178073+0000) 2024-05-27T08:02:29.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:29 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:29.121+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:29.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:29 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:29.121+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:29.232 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:29 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:29.121+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:29.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:29 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:29.119+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:29.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:29 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:29.119+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:29.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:29 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:29.005+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:29.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:29 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:29.005+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:29.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:29 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:29.005+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:29.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:29.381+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:29.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:29.381+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:29.808 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:29.325+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:29.809 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:29 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:29.325+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:30.161 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:29 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:29.879+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:30.162 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:30 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:30.057+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:30.162 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:30 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:30.057+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.678070+0000 front 2024-05-27T08:01:57.678100+0000 (oldest deadline 2024-05-27T08:02:21.178073+0000) 2024-05-27T08:02:30.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:30 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:30.147+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:30.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:30 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:30.147+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:30.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:30 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:30.049+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:30.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:30 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:30.049+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:30.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:30 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:30.049+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:30.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:30 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:30.160+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:30.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:30 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:30.160+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:30.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:30 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:30.160+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:30.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:30 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:30.281+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:30.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:30 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:30.281+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:30.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:30 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:30.409+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:30.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:30 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:30.409+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:30.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:30 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:30.409+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:31.117 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:31 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:31.017+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:31.117 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:31 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:31.017+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.678070+0000 front 2024-05-27T08:01:57.678100+0000 (oldest deadline 2024-05-27T08:02:21.178073+0000) 2024-05-27T08:02:31.118 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:30 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:30.840+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:31.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:31 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:31.108+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:31.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:31 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:31.108+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:31.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:31 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:31.011+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:31.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:31 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:31.011+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:31.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:31 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:31.011+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:31.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:31 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:31.114+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:31.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:31 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:31.115+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:31.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:31 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:31.115+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:31.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:31.281+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:31.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:31.281+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:31.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:31.420+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:31.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:31.420+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:31.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:31 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:31.420+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:32.130 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:32 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:32.055+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.076964+0000 front 2024-05-27T08:01:50.076926+0000 (oldest deadline 2024-05-27T08:02:14.776922+0000) 2024-05-27T08:02:32.130 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:32 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5[48708]: 2024-05-27T08:02:32.055+0000 7f1e8c2cf700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:57.678070+0000 front 2024-05-27T08:01:57.678100+0000 (oldest deadline 2024-05-27T08:02:21.178073+0000) 2024-05-27T08:02:32.130 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:31 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:31.869+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:32.130 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:31 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:31.869+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:32.477 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:32 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:32.146+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:32.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:32 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:32.146+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:32.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:32 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:32.008+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:32.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:32 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:32.008+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:32.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:32 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:32.008+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:32.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:32 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:32.128+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:32.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:32 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:32.128+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:32.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:32 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:32.128+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:32.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:32 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:32.448+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:32.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:32 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:32.448+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:32.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:32 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:32.448+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:32.559 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:32 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:32.259+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:32.559 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:32 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:32.259+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:32.559 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:32 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:32.259+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:07.217823+0000 front 2024-05-27T08:02:07.218011+0000 (oldest deadline 2024-05-27T08:02:31.317980+0000) 2024-05-27T08:02:33.167 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:32 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:32.858+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:33.167 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:32 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:32.859+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:33.451 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:33 smithi186 podman[78251]: 2024-05-27 08:02:33.177382483 +0000 UTC m=+5.528853615 container died dd2e496b59ee52cbf17930d7215d655ab2ca3be4dcd7500bd025aea61284df06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-27T08:02:33.451 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:33.165+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:33.451 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:33.165+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:33.452 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:33.165+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:33.477 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:33 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:33.141+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:33.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:33 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:33.141+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:33.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:33 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:33.004+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:33.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:33 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:33.004+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:33.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:33 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:33.004+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:33.493 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:33.216+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:33.493 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:33.216+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:33.493 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:33.216+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:07.217823+0000 front 2024-05-27T08:02:07.218011+0000 (oldest deadline 2024-05-27T08:02:31.317980+0000) 2024-05-27T08:02:33.730 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:33 smithi186 podman[78251]: 2024-05-27 08:02:33.557504167 +0000 UTC m=+5.908975291 container cleanup dd2e496b59ee52cbf17930d7215d655ab2ca3be4dcd7500bd025aea61284df06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T08:02:33.730 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:33 smithi186 bash[78251]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5 2024-05-27T08:02:33.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:33.491+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:33.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:33.491+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:33.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:33 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:33.491+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:34.170 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:33 smithi186 podman[78266]: 2024-05-27 08:02:33.842430822 +0000 UTC m=+0.661922222 container remove dd2e496b59ee52cbf17930d7215d655ab2ca3be4dcd7500bd025aea61284df06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-27T08:02:34.171 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:33.898+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:34.171 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:33 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:33.898+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:34.439 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:34.193+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:34.440 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:34.193+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:34.440 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:34.193+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:34.440 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:34 smithi186 podman[78354]: 2024-05-27 08:02:34.072161356 +0000 UTC m=+0.016973802 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:02:34.440 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:34 smithi186 podman[78354]: 2024-05-27 08:02:34.176527045 +0000 UTC m=+0.121339501 container create a8d61fad919d305d261452503fdb89b407290a28f0045e281ff26886dc9b9d47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T08:02:34.441 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:34 smithi186 podman[78354]: 2024-05-27 08:02:34.438286521 +0000 UTC m=+0.383098961 container init a8d61fad919d305d261452503fdb89b407290a28f0045e281ff26886dc9b9d47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-05-27T08:02:34.467 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:34 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:34.212+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:34.468 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:34 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:34.212+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:34.468 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:34 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:34.212+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:07.217823+0000 front 2024-05-27T08:02:07.218011+0000 (oldest deadline 2024-05-27T08:02:31.317980+0000) 2024-05-27T08:02:34.477 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:34.116+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:34.477 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:34.116+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:34.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:34.003+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:34.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:34.003+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:34.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:34.003+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:34.730 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:34 smithi186 podman[78354]: 2024-05-27 08:02:34.441786135 +0000 UTC m=+0.386598575 container start a8d61fad919d305d261452503fdb89b407290a28f0045e281ff26886dc9b9d47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T08:02:34.731 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:34 smithi186 podman[78354]: 2024-05-27 08:02:34.529597501 +0000 UTC m=+0.474409935 container attach a8d61fad919d305d261452503fdb89b407290a28f0045e281ff26886dc9b9d47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-05-27T08:02:34.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:34 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:34.466+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:34.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:34 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:34.466+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:34.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:34 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:34.466+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:35.050 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 27 08:02:34 smithi186 podman[78354]: 2024-05-27 08:02:34.740849049 +0000 UTC m=+0.685661485 container died a8d61fad919d305d261452503fdb89b407290a28f0045e281ff26886dc9b9d47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-5-deactivate, ceph=True, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T08:02:35.051 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:34.945+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:35.051 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:34 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:34.945+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:35.368 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:35 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:35.203+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:35.368 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:35 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:35.203+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:35.368 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:35 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:35.203+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:35.396 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.5.service' 2024-05-27T08:02:35.434 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:02:35.434 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-27T08:02:35.435 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-27T08:02:35.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.6 2024-05-27T08:02:35.477 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:35 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:35.118+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:35.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:35 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:35.118+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:35.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:34.978+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:35.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:34.978+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:35.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:34 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:34.978+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:35.487 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:35.453+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:35.487 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:35.453+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:35.488 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:35.453+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:35.488 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:35.233+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:35.488 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:35.233+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:35.488 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:35.233+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:07.217823+0000 front 2024-05-27T08:02:07.218011+0000 (oldest deadline 2024-05-27T08:02:31.317980+0000) 2024-05-27T08:02:35.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:35 smithi114 systemd[1]: Stopping Ceph osd.6 for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:02:36.058 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:35.752+0000 7fa82f887700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:02:36.058 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:35.752+0000 7fa82f887700 -1 osd.6 178 *** Got signal Terminated *** 2024-05-27T08:02:36.058 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:35 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:35.752+0000 7fa82f887700 -1 osd.6 178 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T08:02:36.183 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:35 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:35.901+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:36.184 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:35 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:35.901+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:36.477 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:36 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:36.094+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:36.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:36 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:36.094+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:36.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:35 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:35.982+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:36.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:35 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:35.982+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:36.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:35 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:35.982+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:36.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:36 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:36.181+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:36.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:36 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:36.181+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:36.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:36 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:36.181+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:36.487 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:36.231+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:36.488 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:36.231+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:36.488 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:36.231+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:07.217823+0000 front 2024-05-27T08:02:07.218011+0000 (oldest deadline 2024-05-27T08:02:31.317980+0000) 2024-05-27T08:02:36.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:36.485+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:36.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:36.485+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:36.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:36 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:36.485+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:37.186 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:36 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:36.903+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:37.187 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:36 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:36.903+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:37.227 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:36 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:36.960+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:37.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:36 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:36.960+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:37.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:36 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:36.960+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:37.228 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:37 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:37.091+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:37.228 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:37 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:37.091+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:37.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:37.184+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:37.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:37.184+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:37.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:37.184+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:37.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:37.184+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:37.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:37 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:37.201+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:37.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:37 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:37.201+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:37.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:37 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:37.201+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:07.217823+0000 front 2024-05-27T08:02:07.218011+0000 (oldest deadline 2024-05-27T08:02:31.317980+0000) 2024-05-27T08:02:37.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:37 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:37.449+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:37.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:37 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:37.449+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:37.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:37 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:37.449+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:38.221 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:37.881+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:38.221 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:37 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:37.881+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:38.227 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:38 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:38.140+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:38.227 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:38 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:38.140+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:38.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:37 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:37.961+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:38.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:37 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:37.961+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:38.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:37 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:37.961+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:38.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:38 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:38.218+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:38.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:38 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:38.218+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:38.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:38 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:38.218+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:38.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:38 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:38.218+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:38.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:38 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:38.468+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:38.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:38 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:38.468+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:38.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:38 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:38.468+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:38.559 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:38 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:38.235+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:38.559 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:38 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:38.235+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:38.559 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:38 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:38.235+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:07.217823+0000 front 2024-05-27T08:02:07.218011+0000 (oldest deadline 2024-05-27T08:02:31.317980+0000) 2024-05-27T08:02:39.184 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:38 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:38.896+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:39.184 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:38 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:38.896+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:39.436 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:39 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:39.181+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:39.436 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:39 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:39.181+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:39.436 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:39 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:39.181+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:39.436 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:39 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:39.181+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:39.477 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:38 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:38.980+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:39.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:38 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:38.980+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:39.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:38 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:38.980+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:39.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:39.167+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:39.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:39.167+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:39.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:39.259+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:39.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:39.259+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:39.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:39.259+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:07.217823+0000 front 2024-05-27T08:02:07.218011+0000 (oldest deadline 2024-05-27T08:02:31.317980+0000) 2024-05-27T08:02:39.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:39.433+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:39.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:39.433+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:39.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:39.433+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:39.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:39 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:39.433+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:40.227 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:39.946+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:40.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:39.946+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:40.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:39.946+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:40.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:39 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:39.946+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:40.228 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:40 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:40.125+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:40.228 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:40 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:40.125+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:40.229 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:40 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:40.125+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.760381+0000 front 2024-05-27T08:02:15.760393+0000 (oldest deadline 2024-05-27T08:02:39.860262+0000) 2024-05-27T08:02:40.230 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:39 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:39.938+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:40.231 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:39 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:39.938+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:40.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:40 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:40.177+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:40.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:40 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:40.177+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:40.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:40 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:40.177+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:40.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:40 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:40.177+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:40.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:40 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:40.259+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:50.314293+0000 front 2024-05-27T08:01:50.314327+0000 (oldest deadline 2024-05-27T08:02:15.614502+0000) 2024-05-27T08:02:40.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:40 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:40.259+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.115915+0000 front 2024-05-27T08:01:59.115989+0000 (oldest deadline 2024-05-27T08:02:20.215949+0000) 2024-05-27T08:02:40.559 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:40 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6[57283]: 2024-05-27T08:02:40.259+0000 7fa82be3d700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:07.217823+0000 front 2024-05-27T08:02:07.218011+0000 (oldest deadline 2024-05-27T08:02:31.317980+0000) 2024-05-27T08:02:40.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:40 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:40.409+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:40.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:40 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:40.409+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:40.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:40 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:40.409+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:40.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:40 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:40.409+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:41.227 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:40 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:40.909+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:41.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:40 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:40.909+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:41.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:40 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:40.909+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:41.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:40 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:40.909+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:41.228 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:41 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:41.156+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:41.228 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:41 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:41.156+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:41.229 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:41 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:41.156+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.760381+0000 front 2024-05-27T08:02:15.760393+0000 (oldest deadline 2024-05-27T08:02:39.860262+0000) 2024-05-27T08:02:41.230 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:40 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:40.971+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:41.230 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:40 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:40.971+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:41.230 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:40 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:40.971+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:41.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:41.128+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:41.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:41.128+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:41.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:41.128+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:41.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:41.128+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:41.663 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:41 smithi114 podman[140153]: 2024-05-27 08:02:41.391058847 +0000 UTC m=+5.756868413 container died a6b2ac4595dda117b698c9f8479763076cb82a3130fce7c14d62f5d0913bc03f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2) 2024-05-27T08:02:41.663 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:41 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:41.431+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:41.664 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:41 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:41.431+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:41.664 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:41 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:41.431+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:41.664 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:41 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:41.431+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:42.029 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:41 smithi114 podman[140153]: 2024-05-27 08:02:41.828194733 +0000 UTC m=+6.194004341 container cleanup a6b2ac4595dda117b698c9f8479763076cb82a3130fce7c14d62f5d0913bc03f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-27T08:02:42.029 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:41 smithi114 bash[140153]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6 2024-05-27T08:02:42.179 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:41 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:41.916+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:42.179 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:41 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:41.916+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:42.179 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:41 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:41.916+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:42.180 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:41 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:41.916+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:42.231 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:41.962+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:42.231 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:41.962+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:42.231 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:41 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:41.962+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:42.232 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:42 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:42.150+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:42.232 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:42 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:42.150+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:42.232 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:42 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:42.150+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:42.232 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:42 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:42.150+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:42.308 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:42 smithi114 podman[140169]: 2024-05-27 08:02:42.027645994 +0000 UTC m=+0.635198857 container remove a6b2ac4595dda117b698c9f8479763076cb82a3130fce7c14d62f5d0913bc03f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-27T08:02:42.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:42.177+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:42.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:42.177+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:42.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:42.177+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.760381+0000 front 2024-05-27T08:02:15.760393+0000 (oldest deadline 2024-05-27T08:02:39.860262+0000) 2024-05-27T08:02:42.709 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:42 smithi114 podman[140258]: 2024-05-27 08:02:42.343632847 +0000 UTC m=+0.017214189 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:02:42.710 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:42 smithi114 podman[140258]: 2024-05-27 08:02:42.463846857 +0000 UTC m=+0.137428193 container create 2396df335c05948520c6f81efa21f904b7b1d0c5836070766e7933ca07d71d50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T08:02:42.710 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:42 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:42.392+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:42.710 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:42 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:42.392+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:42.710 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:42 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:42.392+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:42.710 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:42 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:42.392+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:43.011 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:42 smithi114 podman[140258]: 2024-05-27 08:02:42.708561749 +0000 UTC m=+0.382143098 container init 2396df335c05948520c6f81efa21f904b7b1d0c5836070766e7933ca07d71d50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T08:02:43.012 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:42 smithi114 podman[140258]: 2024-05-27 08:02:42.712163095 +0000 UTC m=+0.385744437 container start 2396df335c05948520c6f81efa21f904b7b1d0c5836070766e7933ca07d71d50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-27T08:02:43.012 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:42 smithi114 podman[140258]: 2024-05-27 08:02:42.816713031 +0000 UTC m=+0.490294374 container attach 2396df335c05948520c6f81efa21f904b7b1d0c5836070766e7933ca07d71d50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-27T08:02:43.222 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:42.956+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:43.222 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:42.956+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:43.222 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:42.956+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:43.223 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:42 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:42.956+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:43.308 INFO:journalctl@ceph.osd.6.smithi114.stdout:May 27 08:02:43 smithi114 podman[140258]: 2024-05-27 08:02:43.010716579 +0000 UTC m=+0.684297924 container died 2396df335c05948520c6f81efa21f904b7b1d0c5836070766e7933ca07d71d50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:02:43.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:43 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:43.220+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:43.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:43 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:43.220+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:43.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:43 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:43.220+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.760381+0000 front 2024-05-27T08:02:15.760393+0000 (oldest deadline 2024-05-27T08:02:39.860262+0000) 2024-05-27T08:02:43.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:42 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:42.995+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:43.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:42 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:42.995+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:43.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:42 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:42.995+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:43.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:43 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:43.126+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:43.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:43 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:43.126+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:43.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:43 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:43.126+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:43.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:43 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:43.126+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:43.671 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:43 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:43.375+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:43.672 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:43 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:43.375+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:43.672 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:43 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:43.375+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:43.672 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:43 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:43.375+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:43.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.6.service' 2024-05-27T08:02:43.734 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:02:43.734 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-27T08:02:43.734 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-27T08:02:43.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.7 2024-05-27T08:02:44.053 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:43 smithi163 systemd[1]: Stopping Ceph osd.7 for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:02:44.054 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:44 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:44.051+0000 7f21a2252700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:02:44.054 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:44 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:44.051+0000 7f21a2252700 -1 osd.7 178 *** Got signal Terminated *** 2024-05-27T08:02:44.054 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:44 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:44.051+0000 7f21a2252700 -1 osd.7 178 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T08:02:44.054 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:43 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:43.964+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:44.055 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:43 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:43.964+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:44.055 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:43 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:43.964+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:44.055 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:43 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:43.964+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:44.477 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:44 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:44.251+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:44.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:44 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:44.251+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:44.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:44 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:44.251+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.760381+0000 front 2024-05-27T08:02:15.760393+0000 (oldest deadline 2024-05-27T08:02:39.860262+0000) 2024-05-27T08:02:44.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:44 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:44.020+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:44.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:44 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:44.020+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:44.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:44 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:44.020+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:44.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:44 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:44.091+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:44.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:44 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:44.091+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:44.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:44 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:44.091+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:44.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:44 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:44.091+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:44.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:44.353+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:44.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:44.353+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:44.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:44.353+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:44.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:44 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:44.353+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:45.262 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:44 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:44.983+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:45.263 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:44 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:44.983+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:45.263 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:44 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:44.983+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:45.263 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:44 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:44.983+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:45.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:44 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:44.997+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:45.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:44 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:44.997+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:45.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:44 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:44.997+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:45.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:45 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:45.052+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:45.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:45 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:45.052+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:45.482 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:45 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:45.052+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:45.482 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:45 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:45.052+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:45.710 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:45 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:45.345+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:45.710 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:45 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:45.345+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:45.711 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:45 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:45.345+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:45.711 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:45 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:45.345+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:45.727 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:45 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:45.260+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:45.728 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:45 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:45.260+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:45.728 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:45 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:45.260+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.760381+0000 front 2024-05-27T08:02:15.760393+0000 (oldest deadline 2024-05-27T08:02:39.860262+0000) 2024-05-27T08:02:46.303 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:45 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:45.976+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:46.304 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:45 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:45.976+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:46.304 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:45 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:45.976+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:46.304 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:45 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:45.976+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:46.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:46 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:46.067+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:46.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:46 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:46.067+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:46.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:46 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:46.067+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:46.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:46 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:46.067+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:46.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:45 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:45.991+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:46.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:45 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:45.991+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:46.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:45 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:45.991+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:46.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:45 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:45.991+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:46.727 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:46.301+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:46.728 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:46.301+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:46.728 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:46.301+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.760381+0000 front 2024-05-27T08:02:15.760393+0000 (oldest deadline 2024-05-27T08:02:39.860262+0000) 2024-05-27T08:02:46.728 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:46.342+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:46.728 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:46.342+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:46.729 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:46.342+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:46.729 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:46 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:46.342+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:47.292 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:46.988+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:47.293 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:46.988+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:47.293 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:46.988+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:47.293 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:46 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:46.988+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:47.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:47 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:47.080+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:47.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:47 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:47.080+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:47.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:47 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:47.080+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:47.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:47 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:47.080+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:47.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:47 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:47.035+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:47.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:47 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:47.035+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:47.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:47 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:47.035+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:47.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:47 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:47.035+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:47.727 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:47.290+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:47.728 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:47.290+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:47.728 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:47.290+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.760381+0000 front 2024-05-27T08:02:15.760393+0000 (oldest deadline 2024-05-27T08:02:39.860262+0000) 2024-05-27T08:02:47.728 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:47 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:47.290+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.161156+0000 front 2024-05-27T08:02:22.161290+0000 (oldest deadline 2024-05-27T08:02:46.861325+0000) 2024-05-27T08:02:47.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:47.324+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:47.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:47.324+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:47.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:47.324+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:47.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:47.324+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:47.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:47 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:47.324+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:02:48.302 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:48 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:48.014+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:48.302 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:48 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:48.014+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:48.302 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:48 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:48.014+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:48.303 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:48 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:48.014+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:48.303 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:48 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:48.014+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:02:48.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:48.033+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:48.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:48.033+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:48.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:48.033+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:48.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:48.033+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:48.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:48.067+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:48.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:48.067+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:48.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:48.067+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:48.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:48 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:48.067+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:48.727 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:48 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:48.300+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:59.056210+0000 front 2024-05-27T08:01:59.056320+0000 (oldest deadline 2024-05-27T08:02:23.156333+0000) 2024-05-27T08:02:48.728 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:48 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:48.300+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:03.157126+0000 front 2024-05-27T08:02:03.157084+0000 (oldest deadline 2024-05-27T08:02:28.357633+0000) 2024-05-27T08:02:48.728 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:48 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:48.300+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.760381+0000 front 2024-05-27T08:02:15.760393+0000 (oldest deadline 2024-05-27T08:02:39.860262+0000) 2024-05-27T08:02:48.728 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:48 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7[54060]: 2024-05-27T08:02:48.300+0000 7f219a800700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.161156+0000 front 2024-05-27T08:02:22.161290+0000 (oldest deadline 2024-05-27T08:02:46.861325+0000) 2024-05-27T08:02:48.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:48 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:48.366+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:48.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:48 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:48.366+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:48.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:48 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:48.366+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:48.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:48 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:48.366+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:48.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:48 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:48.366+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:02:49.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:49 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:49.034+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:49.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:49 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:49.034+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:49.231 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:49 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:49.034+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:49.232 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:49 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:49.034+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:49.232 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:49 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:49.089+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:49.232 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:49 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:49.089+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:49.232 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:49 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:49.089+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:49.233 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:49 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:49.089+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:49.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:49.046+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:49.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:49.046+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:49.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:49.046+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:49.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:49.046+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:49.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:49 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:49.046+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:02:49.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:49 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:49.375+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:49.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:49 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:49.375+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:49.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:49 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:49.375+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:49.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:49 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:49.375+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:49.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:49 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:49.375+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:02:49.978 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:49 smithi163 podman[77372]: 2024-05-27 08:02:49.720539095 +0000 UTC m=+5.802389487 container died ab3fd55197072a436d8c8acf28b021ceedef216078fcdf9dea7255e252cafc20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T08:02:50.321 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:50 smithi163 podman[77372]: 2024-05-27 08:02:50.09351273 +0000 UTC m=+6.175363116 container cleanup ab3fd55197072a436d8c8acf28b021ceedef216078fcdf9dea7255e252cafc20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD) 2024-05-27T08:02:50.322 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:50 smithi163 bash[77372]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7 2024-05-27T08:02:50.322 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:50 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:50.055+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:50.322 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:50 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:50.055+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:50.322 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:50 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:50.055+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:50.323 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:50 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:50.055+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:50.323 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:50 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:50.055+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:02:50.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:50 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:50.033+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:50.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:50 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:50.033+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:50.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:50 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:50.033+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:50.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:50 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:50.033+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:50.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:50 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:50.060+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:50.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:50 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:50.060+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:50.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:50 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:50.060+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:50.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:50 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:50.060+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:50.727 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:50 smithi163 podman[77389]: 2024-05-27 08:02:50.320070909 +0000 UTC m=+0.594995898 container remove ab3fd55197072a436d8c8acf28b021ceedef216078fcdf9dea7255e252cafc20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:02:50.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:50 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:50.359+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:50.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:50 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:50.359+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:50.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:50 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:50.359+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:50.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:50 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:50.359+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:50.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:50 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:50.359+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:02:51.062 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:50 smithi163 podman[77477]: 2024-05-27 08:02:50.656636658 +0000 UTC m=+0.016908184 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:02:51.062 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:50 smithi163 podman[77477]: 2024-05-27 08:02:50.754257886 +0000 UTC m=+0.114529388 container create aba82ff01eb1b38b2c26f7966f5da7cef0bb691852224915cd3f6f03dfef2291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-27T08:02:51.062 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:50 smithi163 podman[77477]: 2024-05-27 08:02:50.998529617 +0000 UTC m=+0.358801128 container init aba82ff01eb1b38b2c26f7966f5da7cef0bb691852224915cd3f6f03dfef2291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T08:02:51.062 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:51 smithi163 podman[77477]: 2024-05-27 08:02:51.001505594 +0000 UTC m=+0.361777113 container start aba82ff01eb1b38b2c26f7966f5da7cef0bb691852224915cd3f6f03dfef2291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-27T08:02:51.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:51 smithi163 podman[77477]: 2024-05-27 08:02:51.068747123 +0000 UTC m=+0.429018646 container attach aba82ff01eb1b38b2c26f7966f5da7cef0bb691852224915cd3f6f03dfef2291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T08:02:51.478 INFO:journalctl@ceph.osd.7.smithi163.stdout:May 27 08:02:51 smithi163 podman[77477]: 2024-05-27 08:02:51.309423933 +0000 UTC m=+0.669695447 container died aba82ff01eb1b38b2c26f7966f5da7cef0bb691852224915cd3f6f03dfef2291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T08:02:51.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:51 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:51.059+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:51.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:51 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:51.059+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:51.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:51 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:51.059+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:51.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:51 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:51.059+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:51.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:51 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:51.059+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:02:51.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:51 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:51.044+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:51.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:51 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:51.044+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:51.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:51 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:51.044+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:51.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:51 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:51.044+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:51.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:51 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:51.039+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:51.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:51 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:51.039+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:51.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:51 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:51.039+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:51.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:51 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:51.039+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:51.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:51 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:51.315+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:51.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:51 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:51.315+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:51.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:51 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:51.315+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:51.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:51 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:51.315+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:51.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:51 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:51.315+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:02:51.998 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.7.service' 2024-05-27T08:02:52.039 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:02:52.039 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-27T08:02:52.039 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-27T08:02:52.040 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.8 2024-05-27T08:02:52.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:52 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:52.064+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:52.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:52 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:52.064+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:52.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:52 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:52.064+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:52.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:52 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:52.064+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:52.228 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:52 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:52.064+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:02:52.371 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:52.057+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:52.372 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:52.057+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:52.372 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:52.057+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:52.372 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:52.057+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:52.372 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:52.077+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:52.372 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:52.077+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:52.373 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:52.077+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:52.373 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:52.077+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:52.373 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:52 smithi186 systemd[1]: Stopping Ceph osd.8 for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:02:52.373 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:52.369+0000 7f8bf2f81700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:02:52.373 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:52.369+0000 7f8bf2f81700 -1 osd.8 178 *** Got signal Terminated *** 2024-05-27T08:02:52.373 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:52 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:52.369+0000 7f8bf2f81700 -1 osd.8 178 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T08:02:52.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:52 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:52.294+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:52.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:52 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:52.294+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:52.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:52 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:52.294+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:52.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:52 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:52.294+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:52.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:52 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:52.294+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:02:53.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:53 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:53.112+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:53.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:53 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:53.112+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:53.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:53 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:53.112+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:53.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:53 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:53.112+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:53.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:53 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:53.112+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:02:53.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:53 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:53.087+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:53.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:53 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:53.087+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:53.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:53 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:53.087+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:53.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:53 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:53.087+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:53.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:53 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:53.102+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:53.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:53 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:53.102+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:53.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:53 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:53.102+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:53.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:53 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:53.102+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:53.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:53 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:53.281+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:53.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:53 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:53.281+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:53.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:53 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:53.281+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:53.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:53 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:53.281+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:53.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:53 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:53.281+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:02:54.477 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:54 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:54.138+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:54.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:54 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:54.138+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:54.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:54 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:54.138+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:54.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:54 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:54.138+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:54.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:54 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:54.138+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:02:54.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:54 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:54.113+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:54.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:54 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:54.113+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:54.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:54 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:54.113+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:54.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:54 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:54.113+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:54.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:54 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:54.136+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:54.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:54 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:54.136+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:54.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:54 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:54.136+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:54.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:54 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:54.136+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:54.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:54 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:54.325+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:54.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:54 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:54.325+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:54.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:54 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:54.325+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:54.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:54 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:54.325+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:54.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:54 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:54.325+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:02:55.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:55 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:55.181+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:55.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:55 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:55.181+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:55.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:55 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:55.181+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:55.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:55 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:55.181+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:55.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:55 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:55.181+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:02:55.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:55 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:55.132+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:55.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:55 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:55.132+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:55.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:55 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:55.132+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:55.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:55 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:55.132+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:55.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:55 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:55.135+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:55.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:55 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:55.135+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:55.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:55 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:55.135+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:55.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:55 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:55.135+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:55.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:55 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:55.135+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:02:55.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:55 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:55.284+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:55.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:55 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:55.284+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:55.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:55 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:55.284+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:55.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:55 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:55.284+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:55.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:55 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:55.284+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:02:56.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:56 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:56.164+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:56.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:56 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:56.164+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:56.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:56 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:56.164+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:56.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:56 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:56.164+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:56.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:56 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:56.164+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:02:56.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:56 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:56.131+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:56.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:56 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:56.131+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:56.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:56 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:56.131+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:56.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:56 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:56.131+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:56.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:56 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:56.131+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:32.161465+0000 front 2024-05-27T08:02:32.161518+0000 (oldest deadline 2024-05-27T08:02:55.661693+0000) 2024-05-27T08:02:56.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:56 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:56.115+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:56.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:56 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:56.115+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:56.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:56 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:56.115+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:56.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:56 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:56.115+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:56.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:56 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:56.115+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:02:56.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:56 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:56.288+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:56.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:56 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:56.288+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:56.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:56 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:56.288+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:56.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:56 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:56.288+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:56.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:56 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:56.288+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:02:56.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:56 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:56.288+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.231009+0000 front 2024-05-27T08:02:31.230801+0000 (oldest deadline 2024-05-27T08:02:55.930680+0000) 2024-05-27T08:02:57.477 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:57 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:57.158+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:57.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:57 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:57.158+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:57.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:57 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:57.158+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:57.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:57 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:57.158+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:57.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:57 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:57.158+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:02:57.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:57 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:57.158+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:02:57.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:57.103+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.454111+0000 front 2024-05-27T08:01:48.454062+0000 (oldest deadline 2024-05-27T08:02:14.354107+0000) 2024-05-27T08:02:57.480 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:57.103+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.455241+0000 front 2024-05-27T08:01:58.455426+0000 (oldest deadline 2024-05-27T08:02:20.155195+0000) 2024-05-27T08:02:57.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:57.103+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:05.456158+0000 front 2024-05-27T08:02:05.456244+0000 (oldest deadline 2024-05-27T08:02:28.956318+0000) 2024-05-27T08:02:57.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:57.103+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.458646+0000 front 2024-05-27T08:02:13.458573+0000 (oldest deadline 2024-05-27T08:02:36.958393+0000) 2024-05-27T08:02:57.481 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8[55691]: 2024-05-27T08:02:57.103+0000 7f8bef537700 -1 osd.8 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:32.161465+0000 front 2024-05-27T08:02:32.161518+0000 (oldest deadline 2024-05-27T08:02:55.661693+0000) 2024-05-27T08:02:57.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:57.113+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:57.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:57.113+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:57.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:57.113+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:57.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:57.113+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:57.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:57 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:57.113+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:02:57.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:57 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:57.294+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:57.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:57 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:57.294+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:57.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:57 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:57.294+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:57.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:57 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:57.294+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:57.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:57 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:57.294+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:02:57.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:57 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:57.294+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.231009+0000 front 2024-05-27T08:02:31.230801+0000 (oldest deadline 2024-05-27T08:02:55.930680+0000) 2024-05-27T08:02:58.096 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:57 smithi186 podman[78477]: 2024-05-27 08:02:57.796708726 +0000 UTC m=+5.549654836 container died fb489a33d5bbaa3d60b6584c1de39291664b98b495c4e32164a3cf42813c13f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-27T08:02:58.472 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:58 smithi186 podman[78477]: 2024-05-27 08:02:58.169935345 +0000 UTC m=+5.922881447 container cleanup fb489a33d5bbaa3d60b6584c1de39291664b98b495c4e32164a3cf42813c13f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-27T08:02:58.473 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:58 smithi186 bash[78477]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8 2024-05-27T08:02:58.473 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:58 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:58.094+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:58.473 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:58 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:58.094+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:58.473 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:58 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:58.094+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:58.473 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:58 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:58.094+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:58.474 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:58 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:58.094+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:02:58.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:58 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:58.203+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:58.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:58 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:58.203+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:58.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:58 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:58.203+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:58.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:58 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:58.203+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:58.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:58 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:58.203+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:02:58.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:58 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:58.203+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:02:58.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:58 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:58.273+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:58.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:58 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:58.273+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:58.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:58 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:58.273+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:58.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:58 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:58.273+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:58.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:58 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:58.273+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:02:58.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:58 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:58.273+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.231009+0000 front 2024-05-27T08:02:31.230801+0000 (oldest deadline 2024-05-27T08:02:55.930680+0000) 2024-05-27T08:02:58.730 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:58 smithi186 podman[78494]: 2024-05-27 08:02:58.471289464 +0000 UTC m=+0.667975975 container remove fb489a33d5bbaa3d60b6584c1de39291664b98b495c4e32164a3cf42813c13f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:02:59.037 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:58 smithi186 podman[78583]: 2024-05-27 08:02:58.686765683 +0000 UTC m=+0.017348947 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:02:59.037 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:58 smithi186 podman[78583]: 2024-05-27 08:02:58.797239328 +0000 UTC m=+0.127822583 container create f774f8964b523d3f9ed57babae19fb79c92f90cafd11bc2e090647e571b3b4b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-27T08:02:59.038 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:59 smithi186 podman[78583]: 2024-05-27 08:02:59.033149049 +0000 UTC m=+0.363732302 container init f774f8964b523d3f9ed57babae19fb79c92f90cafd11bc2e090647e571b3b4b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T08:02:59.038 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:59 smithi186 podman[78583]: 2024-05-27 08:02:59.036161363 +0000 UTC m=+0.366744626 container start f774f8964b523d3f9ed57babae19fb79c92f90cafd11bc2e090647e571b3b4b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T08:02:59.308 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:59 smithi186 podman[78583]: 2024-05-27 08:02:59.141826261 +0000 UTC m=+0.472409546 container attach f774f8964b523d3f9ed57babae19fb79c92f90cafd11bc2e090647e571b3b4b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T08:02:59.308 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:59 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:59.058+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:02:59.309 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:59 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:59.058+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:02:59.309 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:59 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:59.058+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:02:59.309 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:59 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:59.058+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:02:59.309 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:02:59 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:02:59.058+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:02:59.477 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:59.165+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:02:59.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:59.165+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:02:59.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:59.165+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:02:59.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:59.165+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:02:59.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:59.165+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:02:59.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:02:59 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:02:59.165+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:02:59.638 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 27 08:02:59 smithi186 podman[78583]: 2024-05-27 08:02:59.341734619 +0000 UTC m=+0.672317888 container died f774f8964b523d3f9ed57babae19fb79c92f90cafd11bc2e090647e571b3b4b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T08:02:59.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:59 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:59.309+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:02:59.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:59 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:59.309+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:02:59.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:59 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:59.309+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:02:59.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:59 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:59.309+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:02:59.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:59 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:59.309+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:02:59.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:02:59 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:02:59.309+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.231009+0000 front 2024-05-27T08:02:31.230801+0000 (oldest deadline 2024-05-27T08:02:55.930680+0000) 2024-05-27T08:02:59.967 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.8.service' 2024-05-27T08:03:00.003 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:00.003 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-27T08:03:00.004 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-27T08:03:00.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.9 2024-05-27T08:03:00.230 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:00.066+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:00.231 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:00.066+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:00.231 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:00.066+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:00.231 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:00.066+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:00.231 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:00 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:00.066+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:00.308 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:00 smithi114 systemd[1]: Stopping Ceph osd.9 for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:03:00.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:00.157+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:00.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:00.157+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:00.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:00.157+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:00.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:00.157+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:00.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:00.157+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:00.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:00 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:00.157+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:00.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:00.340+0000 7f0d8eff5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:03:00.808 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:00.340+0000 7f0d8eff5700 -1 osd.9 178 *** Got signal Terminated *** 2024-05-27T08:03:00.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:00.340+0000 7f0d8eff5700 -1 osd.9 178 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T08:03:00.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:00.345+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:03:00.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:00.345+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:03:00.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:00.345+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:03:00.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:00.345+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:03:00.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:00.345+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:03:00.809 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:00 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:00.345+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.231009+0000 front 2024-05-27T08:02:31.230801+0000 (oldest deadline 2024-05-27T08:02:55.930680+0000) 2024-05-27T08:03:01.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:01.193+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:01.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:01.193+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:01.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:01.193+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:01.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:01.193+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:01.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:01.193+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:01.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:01 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:01.193+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:01.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:01.067+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:01.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:01.067+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:01.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:01.067+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:01.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:01.067+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:01.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:01 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:01.067+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:01.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:01.296+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:03:01.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:01.296+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:03:01.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:01.296+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:03:01.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:01.296+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:03:01.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:01.296+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:03:01.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:01 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:01.296+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.231009+0000 front 2024-05-27T08:02:31.230801+0000 (oldest deadline 2024-05-27T08:02:55.930680+0000) 2024-05-27T08:03:02.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:02 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:02.151+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:02.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:02 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:02.151+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:02.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:02 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:02.151+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:02.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:02 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:02.151+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:02.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:02 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:02.151+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:02.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:02 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:02.151+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:02.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:02 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:02.093+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:02.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:02 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:02.093+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:02.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:02 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:02.093+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:02.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:02 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:02.093+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:02.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:02 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:02.093+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:02.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:02 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:02.298+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:03:02.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:02 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:02.298+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:03:02.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:02 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:02.298+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:03:02.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:02 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:02.298+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:03:02.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:02 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:02.298+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:03:02.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:02 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:02.298+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.231009+0000 front 2024-05-27T08:02:31.230801+0000 (oldest deadline 2024-05-27T08:02:55.930680+0000) 2024-05-27T08:03:03.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:03.197+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:03.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:03.197+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:03.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:03.197+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:03.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:03.197+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:03.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:03.197+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:03.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:03.197+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:03.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:03 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:03.197+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:39.398612+0000 front 2024-05-27T08:02:39.398717+0000 (oldest deadline 2024-05-27T08:03:02.298510+0000) 2024-05-27T08:03:03.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:03 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:03.050+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:03.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:03 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:03.050+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:03.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:03 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:03.050+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:03.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:03 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:03.050+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:03.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:03 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:03.050+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:03.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:03 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:03.254+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:03:03.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:03 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:03.254+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:03:03.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:03 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:03.254+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:03:03.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:03 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:03.254+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:03:03.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:03 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:03.254+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:03:03.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:03 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:03.254+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.231009+0000 front 2024-05-27T08:02:31.230801+0000 (oldest deadline 2024-05-27T08:02:55.930680+0000) 2024-05-27T08:03:04.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:04.043+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:04.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:04.043+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:04.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:04.043+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:04.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:04.043+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:04.483 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:04.043+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:04.483 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:04 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:04.043+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:04.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:04 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:04.295+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:48.424485+0000 front 2024-05-27T08:01:48.424510+0000 (oldest deadline 2024-05-27T08:02:11.924696+0000) 2024-05-27T08:03:04.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:04 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:04.295+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:58.326594+0000 front 2024-05-27T08:01:58.326390+0000 (oldest deadline 2024-05-27T08:02:24.226605+0000) 2024-05-27T08:03:04.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:04 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:04.295+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:04.227413+0000 front 2024-05-27T08:02:04.227340+0000 (oldest deadline 2024-05-27T08:02:29.527316+0000) 2024-05-27T08:03:04.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:04 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:04.295+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:13.628322+0000 front 2024-05-27T08:02:13.628346+0000 (oldest deadline 2024-05-27T08:02:38.928550+0000) 2024-05-27T08:03:04.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:04 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:04.295+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:23.629826+0000 front 2024-05-27T08:02:23.629653+0000 (oldest deadline 2024-05-27T08:02:46.529778+0000) 2024-05-27T08:03:04.559 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:04 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9[64126]: 2024-05-27T08:03:04.295+0000 7f0d875a3700 -1 osd.9 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.231009+0000 front 2024-05-27T08:02:31.230801+0000 (oldest deadline 2024-05-27T08:02:55.930680+0000) 2024-05-27T08:03:04.728 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:04 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:04.229+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:04.728 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:04 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:04.229+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:04.728 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:04 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:04.229+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:04.729 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:04 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:04.229+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:04.729 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:04 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:04.229+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:04.729 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:04 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:04.229+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:04.729 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:04 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:04.229+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:39.398612+0000 front 2024-05-27T08:02:39.398717+0000 (oldest deadline 2024-05-27T08:03:02.298510+0000) 2024-05-27T08:03:05.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:05 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:05.187+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:05.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:05 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:05.187+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:05.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:05 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:05.187+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:05.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:05 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:05.187+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:05.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:05 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:05.187+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:05.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:05 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:05.187+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:05.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:05 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:05.187+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:39.398612+0000 front 2024-05-27T08:02:39.398717+0000 (oldest deadline 2024-05-27T08:03:02.298510+0000) 2024-05-27T08:03:05.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:05.031+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:05.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:05.031+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:05.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:05.031+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:05.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:05.031+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:05.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:05.031+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:05.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:05 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:05.031+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:06.257 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:05 smithi114 podman[140376]: 2024-05-27 08:03:05.972655003 +0000 UTC m=+5.771642051 container died 227a5431062e8fd9a4ef4d04b9a771b7c30c6e6f7c9fc83658a9077bf38e04eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-27T08:03:06.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:06 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:06.153+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:06.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:06 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:06.153+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:06.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:06 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:06.153+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:06.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:06 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:06.153+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:06.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:06 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:06.153+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:06.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:06 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:06.153+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:06.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:06 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:06.153+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:39.398612+0000 front 2024-05-27T08:02:39.398717+0000 (oldest deadline 2024-05-27T08:03:02.298510+0000) 2024-05-27T08:03:06.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:06.073+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:06.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:06.073+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:06.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:06.073+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:06.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:06.073+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:06.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:06.073+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:06.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:06 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:06.073+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:06.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:06 smithi114 podman[140376]: 2024-05-27 08:03:06.439358308 +0000 UTC m=+6.238345375 container cleanup 227a5431062e8fd9a4ef4d04b9a771b7c30c6e6f7c9fc83658a9077bf38e04eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5) 2024-05-27T08:03:06.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:06 smithi114 bash[140376]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9 2024-05-27T08:03:06.984 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:06 smithi114 podman[140395]: 2024-05-27 08:03:06.682390969 +0000 UTC m=+0.712423778 container remove 227a5431062e8fd9a4ef4d04b9a771b7c30c6e6f7c9fc83658a9077bf38e04eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T08:03:07.303 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:06 smithi114 podman[140482]: 2024-05-27 08:03:06.98299044 +0000 UTC m=+0.102385724 container create 1e9588342323bfa52fd649cd9a9ef79362d1a6a6e405d8d417b8857d2b25d598 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-27T08:03:07.303 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:06 smithi114 podman[140482]: 2024-05-27 08:03:06.897913257 +0000 UTC m=+0.017308546 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:03:07.303 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:07 smithi114 podman[140482]: 2024-05-27 08:03:07.201904654 +0000 UTC m=+0.321299933 container init 1e9588342323bfa52fd649cd9a9ef79362d1a6a6e405d8d417b8857d2b25d598 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-27T08:03:07.303 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:07 smithi114 podman[140482]: 2024-05-27 08:03:07.205176421 +0000 UTC m=+0.324571712 container start 1e9588342323bfa52fd649cd9a9ef79362d1a6a6e405d8d417b8857d2b25d598 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T08:03:07.304 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:07 smithi114 podman[140482]: 2024-05-27 08:03:07.301834842 +0000 UTC m=+0.421230136 container attach 1e9588342323bfa52fd649cd9a9ef79362d1a6a6e405d8d417b8857d2b25d598 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD) 2024-05-27T08:03:07.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:07 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:07.195+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:07.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:07 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:07.195+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:07.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:07 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:07.195+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:07.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:07 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:07.195+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:07.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:07 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:07.195+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:07.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:07 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:07.195+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:07.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:07 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:07.195+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:39.398612+0000 front 2024-05-27T08:02:39.398717+0000 (oldest deadline 2024-05-27T08:03:02.298510+0000) 2024-05-27T08:03:07.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:07 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:07.108+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:07.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:07 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:07.108+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:07.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:07 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:07.108+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:07.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:07 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:07.108+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:07.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:07 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:07.108+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:07.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:07 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:07.108+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:07.558 INFO:journalctl@ceph.osd.9.smithi114.stdout:May 27 08:03:07 smithi114 podman[140482]: 2024-05-27 08:03:07.496676829 +0000 UTC m=+0.616072117 container died 1e9588342323bfa52fd649cd9a9ef79362d1a6a6e405d8d417b8857d2b25d598 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:03:08.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.9.service' 2024-05-27T08:03:08.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:08.118 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-27T08:03:08.119 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-27T08:03:08.119 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.10 2024-05-27T08:03:08.382 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:08 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:08.101+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:08.382 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:08 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:08.101+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:08.382 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:08 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:08.101+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:08.383 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:08 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:08.101+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:08.383 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:08 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:08.101+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:08.383 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:08 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:08.101+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:08.448 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:08 smithi163 systemd[1]: Stopping Ceph osd.10 for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:03:08.448 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:08 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:08.226+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:08.448 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:08 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:08.226+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:08.448 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:08 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:08.226+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:08.448 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:08 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:08.226+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:08.449 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:08 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:08.226+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:08.449 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:08 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:08.226+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:08.449 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:08 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:08.226+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:39.398612+0000 front 2024-05-27T08:02:39.398717+0000 (oldest deadline 2024-05-27T08:03:02.298510+0000) 2024-05-27T08:03:08.449 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:08 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:08.445+0000 7f391a4fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:03:08.449 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:08 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:08.445+0000 7f391a4fe700 -1 osd.10 178 *** Got signal Terminated *** 2024-05-27T08:03:08.449 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:08 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:08.445+0000 7f391a4fe700 -1 osd.10 178 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T08:03:09.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:09 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:09.202+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:09.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:09 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:09.202+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:09.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:09 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:09.202+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:09.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:09 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:09.202+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:09.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:09 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:09.202+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:09.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:09 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:09.202+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:09.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:09 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:09.202+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:39.398612+0000 front 2024-05-27T08:02:39.398717+0000 (oldest deadline 2024-05-27T08:03:02.298510+0000) 2024-05-27T08:03:09.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:09 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:09.071+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:09.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:09 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:09.071+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:09.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:09 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:09.071+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:09.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:09 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:09.071+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:09.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:09 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:09.071+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:09.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:09 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:09.071+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:10.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:10 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:10.203+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:10.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:10 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:10.203+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:10.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:10 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:10.203+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:10.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:10 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:10.203+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:10.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:10 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:10.203+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:10.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:10 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:10.203+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:10.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:10 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:10.203+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:39.398612+0000 front 2024-05-27T08:02:39.398717+0000 (oldest deadline 2024-05-27T08:03:02.298510+0000) 2024-05-27T08:03:10.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:10 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:10.053+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:10.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:10 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:10.053+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:10.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:10 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:10.053+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:10.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:10 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:10.053+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:10.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:10 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:10.053+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:10.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:10 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:10.053+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:11.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:11 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:11.200+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:11.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:11 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:11.200+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:11.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:11 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:11.200+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:11.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:11 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:11.200+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:11.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:11 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:11.200+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:11.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:11 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:11.200+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:11.479 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:11 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:11.200+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:39.398612+0000 front 2024-05-27T08:02:39.398717+0000 (oldest deadline 2024-05-27T08:03:02.298510+0000) 2024-05-27T08:03:11.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:11 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:11.058+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:11.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:11 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:11.058+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:11.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:11 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:11.058+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:11.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:11 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:11.058+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:11.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:11 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:11.058+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:11.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:11 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:11.058+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:12.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:12 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:12.090+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:12.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:12 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:12.090+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:12.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:12 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:12.090+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:12.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:12 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:12.090+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:12.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:12 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:12.090+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:12.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:12 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:12.090+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:12.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:12 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:12.090+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-05-27T08:02:44.211242+0000 front 2024-05-27T08:02:44.211274+0000 (oldest deadline 2024-05-27T08:03:11.811864+0000) 2024-05-27T08:03:12.653 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:12 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:12.247+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:12.653 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:12 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:12.247+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:12.653 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:12 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:12.247+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:12.653 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:12 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:12.247+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:12.653 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:12 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:12.247+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:12.653 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:12 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:12.247+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:12.654 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:12 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:12.247+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:39.398612+0000 front 2024-05-27T08:02:39.398717+0000 (oldest deadline 2024-05-27T08:03:02.298510+0000) 2024-05-27T08:03:13.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:13 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:13.059+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:13.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:13 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:13.059+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:13.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:13 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:13.059+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:13.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:13 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:13.059+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:13.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:13 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:13.059+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:13.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:13 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:13.059+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:13.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:13 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:13.059+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-05-27T08:02:44.211242+0000 front 2024-05-27T08:02:44.211274+0000 (oldest deadline 2024-05-27T08:03:11.811864+0000) 2024-05-27T08:03:13.577 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:13.290+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.390617+0000 front 2024-05-27T08:01:49.390547+0000 (oldest deadline 2024-05-27T08:02:13.490513+0000) 2024-05-27T08:03:13.577 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:13.290+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-05-27T08:01:56.891984+0000 front 2024-05-27T08:01:56.892051+0000 (oldest deadline 2024-05-27T08:02:19.791990+0000) 2024-05-27T08:03:13.577 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:13.290+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:02.092890+0000 front 2024-05-27T08:02:02.093071+0000 (oldest deadline 2024-05-27T08:02:27.992806+0000) 2024-05-27T08:03:13.577 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:13.290+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:14.295367+0000 front 2024-05-27T08:02:14.295186+0000 (oldest deadline 2024-05-27T08:02:38.995169+0000) 2024-05-27T08:03:13.577 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:13.290+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:22.396529+0000 front 2024-05-27T08:02:22.396559+0000 (oldest deadline 2024-05-27T08:02:47.096591+0000) 2024-05-27T08:03:13.577 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:13.290+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:31.197924+0000 front 2024-05-27T08:02:31.197853+0000 (oldest deadline 2024-05-27T08:02:57.097557+0000) 2024-05-27T08:03:13.578 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:13 smithi163 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10[60952]: 2024-05-27T08:03:13.290+0000 7f3916ab4700 -1 osd.10 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:39.398612+0000 front 2024-05-27T08:02:39.398717+0000 (oldest deadline 2024-05-27T08:03:02.298510+0000) 2024-05-27T08:03:14.289 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:13 smithi163 podman[77584]: 2024-05-27 08:03:13.996374634 +0000 UTC m=+5.678502804 container died b2bfae1778138bed2fd7e566bbf7d0681b0eac7d044c4019e0d62fff129f87ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0) 2024-05-27T08:03:14.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:14 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:14.056+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:14.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:14 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:14.056+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:14.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:14 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:14.056+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:14.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:14 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:14.056+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:14.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:14 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:14.056+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:14.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:14 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:14.056+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:14.483 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:14 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:14.056+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-05-27T08:02:44.211242+0000 front 2024-05-27T08:02:44.211274+0000 (oldest deadline 2024-05-27T08:03:11.811864+0000) 2024-05-27T08:03:14.615 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:14 smithi163 podman[77584]: 2024-05-27 08:03:14.387883477 +0000 UTC m=+6.070011646 container cleanup b2bfae1778138bed2fd7e566bbf7d0681b0eac7d044c4019e0d62fff129f87ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-27T08:03:14.615 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:14 smithi163 bash[77584]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10 2024-05-27T08:03:14.917 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:14 smithi163 podman[77612]: 2024-05-27 08:03:14.614022537 +0000 UTC m=+0.614629538 container remove b2bfae1778138bed2fd7e566bbf7d0681b0eac7d044c4019e0d62fff129f87ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=) 2024-05-27T08:03:15.210 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:14 smithi163 podman[77700]: 2024-05-27 08:03:14.914903766 +0000 UTC m=+0.104465797 container create cb0feabb0a4a66901c47b0247ac13cd4db454a405c987e24fd0e109a86a1476b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-27T08:03:15.210 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:14 smithi163 podman[77700]: 2024-05-27 08:03:14.827774221 +0000 UTC m=+0.017336252 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:03:15.211 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:15 smithi163 podman[77700]: 2024-05-27 08:03:15.133996445 +0000 UTC m=+0.323558478 container init cb0feabb0a4a66901c47b0247ac13cd4db454a405c987e24fd0e109a86a1476b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T08:03:15.211 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:15 smithi163 podman[77700]: 2024-05-27 08:03:15.137620286 +0000 UTC m=+0.327182318 container start cb0feabb0a4a66901c47b0247ac13cd4db454a405c987e24fd0e109a86a1476b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T08:03:15.211 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:15 smithi163 podman[77700]: 2024-05-27 08:03:15.20899412 +0000 UTC m=+0.398556152 container attach cb0feabb0a4a66901c47b0247ac13cd4db454a405c987e24fd0e109a86a1476b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True) 2024-05-27T08:03:15.478 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:15 smithi163 podman[77700]: 2024-05-27 08:03:15.42944027 +0000 UTC m=+0.619002310 container died cb0feabb0a4a66901c47b0247ac13cd4db454a405c987e24fd0e109a86a1476b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-05-27T08:03:15.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:15.047+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:15.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:15.047+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:15.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:15.047+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:15.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:15.047+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:15.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:15.047+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:15.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:15.047+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:15.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:15 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:15.047+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-05-27T08:02:44.211242+0000 front 2024-05-27T08:02:44.211274+0000 (oldest deadline 2024-05-27T08:03:11.811864+0000) 2024-05-27T08:03:15.978 INFO:journalctl@ceph.osd.10.smithi163.stdout:May 27 08:03:15 smithi163 podman[77726]: 2024-05-27 08:03:15.962153078 +0000 UTC m=+0.523072911 container remove cb0feabb0a4a66901c47b0247ac13cd4db454a405c987e24fd0e109a86a1476b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:03:15.984 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.10.service' 2024-05-27T08:03:16.019 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:16.020 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-27T08:03:16.020 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-27T08:03:16.020 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.11 2024-05-27T08:03:16.333 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:16 smithi186 systemd[1]: Stopping Ceph osd.11 for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:03:16.333 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:16.070+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:16.334 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:16.070+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:16.334 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:16.070+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:16.334 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:16.070+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:16.334 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:16.070+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:16.335 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:16.070+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:16.335 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:16.070+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-05-27T08:02:44.211242+0000 front 2024-05-27T08:02:44.211274+0000 (oldest deadline 2024-05-27T08:03:11.811864+0000) 2024-05-27T08:03:16.335 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:16.330+0000 7f827d14a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:03:16.335 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:16.330+0000 7f827d14a700 -1 osd.11 178 *** Got signal Terminated *** 2024-05-27T08:03:16.336 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:16 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:16.330+0000 7f827d14a700 -1 osd.11 178 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T08:03:17.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:17.025+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:17.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:17.025+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:17.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:17.025+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:17.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:17.025+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:17.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:17.025+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:17.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:17.025+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:17.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:17 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:17.025+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-05-27T08:02:44.211242+0000 front 2024-05-27T08:02:44.211274+0000 (oldest deadline 2024-05-27T08:03:11.811864+0000) 2024-05-27T08:03:18.441 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:18.034+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:18.441 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:18.034+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:18.442 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:18.034+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:18.442 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:18.034+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:18.442 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:18.034+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:18.442 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:18.034+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:18.442 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:18 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:18.034+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-05-27T08:02:44.211242+0000 front 2024-05-27T08:02:44.211274+0000 (oldest deadline 2024-05-27T08:03:11.811864+0000) 2024-05-27T08:03:19.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:19.076+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:19.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:19.076+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:19.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:19.076+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:19.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:19.076+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:19.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:19.076+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:19.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:19.076+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:19.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:19.076+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-05-27T08:02:44.211242+0000 front 2024-05-27T08:02:44.211274+0000 (oldest deadline 2024-05-27T08:03:11.811864+0000) 2024-05-27T08:03:19.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:19 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:19.076+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6820 osd.8 since back 2024-05-27T08:02:55.813073+0000 front 2024-05-27T08:02:55.813105+0000 (oldest deadline 2024-05-27T08:03:18.713094+0000) 2024-05-27T08:03:20.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:20.051+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:20.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:20.051+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:20.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:20.051+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:20.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:20.051+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:20.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:20.051+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:20.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:20.051+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:20.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:20.051+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-05-27T08:02:44.211242+0000 front 2024-05-27T08:02:44.211274+0000 (oldest deadline 2024-05-27T08:03:11.811864+0000) 2024-05-27T08:03:20.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:20 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:20.051+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6820 osd.8 since back 2024-05-27T08:02:55.813073+0000 front 2024-05-27T08:02:55.813105+0000 (oldest deadline 2024-05-27T08:03:18.713094+0000) 2024-05-27T08:03:21.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:21 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:21.002+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-05-27T08:01:49.902173+0000 front 2024-05-27T08:01:49.902232+0000 (oldest deadline 2024-05-27T08:02:11.602214+0000) 2024-05-27T08:03:21.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:21 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:21.002+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-27T08:02:06.205252+0000 front 2024-05-27T08:02:06.205283+0000 (oldest deadline 2024-05-27T08:02:30.905533+0000) 2024-05-27T08:03:21.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:21 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:21.002+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-05-27T08:02:15.006835+0000 front 2024-05-27T08:02:15.006624+0000 (oldest deadline 2024-05-27T08:02:40.906772+0000) 2024-05-27T08:03:21.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:21 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:21.002+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-05-27T08:02:20.907406+0000 front 2024-05-27T08:02:20.907268+0000 (oldest deadline 2024-05-27T08:02:45.607402+0000) 2024-05-27T08:03:21.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:21 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:21.002+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-27T08:02:29.108436+0000 front 2024-05-27T08:02:29.108330+0000 (oldest deadline 2024-05-27T08:02:55.008415+0000) 2024-05-27T08:03:21.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:21 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:21.002+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-05-27T08:02:37.309733+0000 front 2024-05-27T08:02:37.309681+0000 (oldest deadline 2024-05-27T08:03:03.209546+0000) 2024-05-27T08:03:21.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:21 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:21.002+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-05-27T08:02:44.211242+0000 front 2024-05-27T08:02:44.211274+0000 (oldest deadline 2024-05-27T08:03:11.811864+0000) 2024-05-27T08:03:21.482 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:21 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11[62262]: 2024-05-27T08:03:21.002+0000 7f82756f8700 -1 osd.11 178 heartbeat_check: no reply from 172.21.15.186:6820 osd.8 since back 2024-05-27T08:02:55.813073+0000 front 2024-05-27T08:02:55.813105+0000 (oldest deadline 2024-05-27T08:03:18.713094+0000) 2024-05-27T08:03:22.141 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:21 smithi186 podman[78702]: 2024-05-27 08:03:21.868114733 +0000 UTC m=+5.658877821 container died 21c0463cf20f9485259dcb6edd1d2155982cd7735e820dea59d1b48aeeb13aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-27T08:03:22.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:22 smithi186 podman[78702]: 2024-05-27 08:03:22.231222532 +0000 UTC m=+6.021985602 container cleanup 21c0463cf20f9485259dcb6edd1d2155982cd7735e820dea59d1b48aeeb13aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-27T08:03:22.480 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:22 smithi186 bash[78702]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11 2024-05-27T08:03:22.842 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:22 smithi186 podman[78720]: 2024-05-27 08:03:22.507523753 +0000 UTC m=+0.639322014 container remove 21c0463cf20f9485259dcb6edd1d2155982cd7735e820dea59d1b48aeeb13aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-05-27T08:03:23.180 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:22 smithi186 podman[78809]: 2024-05-27 08:03:22.841580493 +0000 UTC m=+0.102552217 container create 7ace96af6488906ec958cffed6ab0988a6d2c5468feed292a2a2ad2127bbaeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:03:23.180 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:22 smithi186 podman[78809]: 2024-05-27 08:03:22.755885844 +0000 UTC m=+0.016857575 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T08:03:23.180 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:23 smithi186 podman[78809]: 2024-05-27 08:03:23.052585326 +0000 UTC m=+0.313557059 container init 7ace96af6488906ec958cffed6ab0988a6d2c5468feed292a2a2ad2127bbaeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T08:03:23.180 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:23 smithi186 podman[78809]: 2024-05-27 08:03:23.05593121 +0000 UTC m=+0.316902944 container start 7ace96af6488906ec958cffed6ab0988a6d2c5468feed292a2a2ad2127bbaeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-27T08:03:23.181 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:23 smithi186 podman[78809]: 2024-05-27 08:03:23.179012441 +0000 UTC m=+0.439984185 container attach 7ace96af6488906ec958cffed6ab0988a6d2c5468feed292a2a2ad2127bbaeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T08:03:23.481 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:23 smithi186 podman[78809]: 2024-05-27 08:03:23.36244393 +0000 UTC m=+0.623415669 container died 7ace96af6488906ec958cffed6ab0988a6d2c5468feed292a2a2ad2127bbaeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-27T08:03:23.925 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 27 08:03:23 smithi186 podman[78835]: 2024-05-27 08:03:23.907868732 +0000 UTC m=+0.539948558 container remove 7ace96af6488906ec958cffed6ab0988a6d2c5468feed292a2a2ad2127bbaeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2) 2024-05-27T08:03:23.929 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@osd.11.service' 2024-05-27T08:03:23.967 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:23.968 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-27T08:03:23.968 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-27T08:03:23.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a 2024-05-27T08:03:24.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.a.service' 2024-05-27T08:03:24.084 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:24.085 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-27T08:03:24.085 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-27T08:03:24.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d 2024-05-27T08:03:24.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.d.service' 2024-05-27T08:03:24.231 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:24.231 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-27T08:03:24.232 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-27T08:03:24.232 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g 2024-05-27T08:03:24.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.g.service' 2024-05-27T08:03:24.374 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:24.374 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-27T08:03:24.375 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-27T08:03:24.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.j 2024-05-27T08:03:24.558 INFO:journalctl@ceph.mds.j.smithi114.stdout:May 27 08:03:24 smithi114 systemd[1]: Stopping Ceph mds.j for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:03:25.058 INFO:journalctl@ceph.mds.j.smithi114.stdout:May 27 08:03:24 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-j[69703]: 2024-05-27T08:03:24.699+0000 7f5467869700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:03:25.058 INFO:journalctl@ceph.mds.j.smithi114.stdout:May 27 08:03:24 smithi114 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-j[69703]: 2024-05-27T08:03:24.699+0000 7f5467869700 -1 mds.j *** got signal Terminated *** 2024-05-27T08:03:28.175 INFO:journalctl@ceph.mds.j.smithi114.stdout:May 27 08:03:27 smithi114 podman[140709]: 2024-05-27 08:03:27.905877688 +0000 UTC m=+3.313321655 container died 737e349e4d437e8a4b77dfaa5bc3aad33309ab2fb36247cb0d2bce751b7f5fbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-j, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-27T08:03:28.558 INFO:journalctl@ceph.mds.j.smithi114.stdout:May 27 08:03:28 smithi114 podman[140709]: 2024-05-27 08:03:28.365932664 +0000 UTC m=+3.773376653 container cleanup 737e349e4d437e8a4b77dfaa5bc3aad33309ab2fb36247cb0d2bce751b7f5fbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-j, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-27T08:03:28.558 INFO:journalctl@ceph.mds.j.smithi114.stdout:May 27 08:03:28 smithi114 bash[140709]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-j 2024-05-27T08:03:28.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.j.service' 2024-05-27T08:03:28.839 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:28.840 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-27T08:03:28.840 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-27T08:03:28.840 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b 2024-05-27T08:03:28.876 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.b.service' 2024-05-27T08:03:28.954 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:28.955 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-27T08:03:28.955 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-27T08:03:28.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e 2024-05-27T08:03:29.026 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.e.service' 2024-05-27T08:03:29.099 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:29.100 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-27T08:03:29.100 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-27T08:03:29.100 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h 2024-05-27T08:03:29.170 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.h.service' 2024-05-27T08:03:29.243 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:29.243 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-27T08:03:29.244 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-27T08:03:29.244 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k 2024-05-27T08:03:29.313 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.k.service' 2024-05-27T08:03:29.383 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:29.383 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-27T08:03:29.383 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-27T08:03:29.384 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c 2024-05-27T08:03:29.419 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.c.service' 2024-05-27T08:03:29.495 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:29.495 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-27T08:03:29.496 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-27T08:03:29.496 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f 2024-05-27T08:03:29.565 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.f.service' 2024-05-27T08:03:29.638 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:29.639 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-27T08:03:29.639 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-27T08:03:29.639 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.i 2024-05-27T08:03:29.970 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 08:03:29 smithi186 systemd[1]: Stopping Ceph mds.i for 755b3728-1bfa-11ef-bc9b-c7b262605968... 2024-05-27T08:03:30.230 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 08:03:29 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-i[67994]: 2024-05-27T08:03:29.968+0000 7fb154c01700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T08:03:30.231 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 08:03:29 smithi186 ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-i[67994]: 2024-05-27T08:03:29.968+0000 7fb154c01700 -1 mds.i *** got signal Terminated *** 2024-05-27T08:03:32.753 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 08:03:32 smithi186 podman[78987]: 2024-05-27 08:03:32.500197336 +0000 UTC m=+2.644053872 container died a502f02b07474e87e2c9b1a76eaa4c55a6a6caa72b9fc9185d0e9f645df63997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-i, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-27T08:03:33.037 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 08:03:32 smithi186 podman[78987]: 2024-05-27 08:03:32.843639411 +0000 UTC m=+2.987495955 container cleanup a502f02b07474e87e2c9b1a76eaa4c55a6a6caa72b9fc9185d0e9f645df63997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-i, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-05-27T08:03:33.038 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 27 08:03:32 smithi186 bash[78987]: ceph-755b3728-1bfa-11ef-bc9b-c7b262605968-mds-i 2024-05-27T08:03:33.245 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.i.service' 2024-05-27T08:03:33.278 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:33.278 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-27T08:03:33.278 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-27T08:03:33.279 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l 2024-05-27T08:03:33.347 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-755b3728-1bfa-11ef-bc9b-c7b262605968@mds.l.service' 2024-05-27T08:03:33.418 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T08:03:33.418 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-27T08:03:33.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T08:03:33.607 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T08:03:35.983 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T08:03:36.172 INFO:teuthology.orchestra.run.smithi163.stdout:Deleting cluster with fsid: 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T08:03:38.504 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T08:03:38.987 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T08:03:41.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T08:03:41.605 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T08:03:41.636 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T08:03:41.663 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-27T08:03:41.665 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664/remote/smithi114/crash 2024-05-27T08:03:41.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T08:03:41.707 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T08:03:41.708 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now 2024-05-27T08:03:41.709 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664/remote/smithi163/crash 2024-05-27T08:03:41.710 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T08:03:41.757 INFO:teuthology.orchestra.run.smithi163.stderr:tar: /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T08:03:41.757 INFO:teuthology.orchestra.run.smithi163.stderr:tar: Error is not recoverable: exiting now 2024-05-27T08:03:41.759 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664/remote/smithi186/crash 2024-05-27T08:03:41.760 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T08:03:41.802 INFO:teuthology.orchestra.run.smithi186.stderr:tar: /var/lib/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T08:03:41.803 INFO:teuthology.orchestra.run.smithi186.stderr:tar: Error is not recoverable: exiting now 2024-05-27T08:03:41.804 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-27T08:03:41.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-27T08:03:41.857 INFO:teuthology.orchestra.run.smithi114.stdout:2024-05-27T07:50:16.652009+0000 mon.a (mon.0) 1452 : cluster [WRN] Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T08:03:41.858 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-05-27T08:03:41.858 INFO:tasks.cephadm:Compressing logs... 2024-05-27T08:03:41.859 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T08:03:41.861 DEBUG:teuthology.orchestra.run.smithi163:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T08:03:41.864 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T08:03:41.886 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T08:03:41.886 INFO:teuthology.orchestra.run.smithi114.stderr:: No such file or directory 2024-05-27T08:03:41.887 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T08:03:41.887 INFO:teuthology.orchestra.run.smithi114.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T08:03:41.887 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.g.log 2024-05-27T08:03:41.887 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.j.log 2024-05-27T08:03:41.887 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-05-27T08:03:41.887 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.g.log: /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-05-27T08:03:41.895 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mgr.x.log 2024-05-27T08:03:41.896 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mon.a.log 2024-05-27T08:03:41.896 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.6.log 2024-05-27T08:03:41.896 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T08:03:41.896 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.a.log 2024-05-27T08:03:41.896 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T08:03:41.896 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.d.log 2024-05-27T08:03:41.896 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.9.log 2024-05-27T08:03:41.896 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-05-27T08:03:41.897 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.9.log: 87.1% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T08:03:41.897 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-05-27T08:03:41.899 INFO:teuthology.orchestra.run.smithi163.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T08:03:41.899 INFO:teuthology.orchestra.run.smithi163.stderr:: No such file or directory 2024-05-27T08:03:41.899 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T08:03:41.899 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T08:03:41.899 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.7.log 2024-05-27T08:03:41.900 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.b.log 2024-05-27T08:03:41.900 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-05-27T08:03:41.900 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-05-27T08:03:41.900 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mgr.y.log 2024-05-27T08:03:41.900 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.h.log 2024-05-27T08:03:41.900 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.10.log 2024-05-27T08:03:41.900 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T08:03:41.900 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.e.log 2024-05-27T08:03:41.901 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T08:03:41.901 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mon.b.log 2024-05-27T08:03:41.901 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.k.log 2024-05-27T08:03:41.903 INFO:teuthology.orchestra.run.smithi186.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-27T08:03:41.903 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T08:03:41.903 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T08:03:41.903 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-05-27T08:03:41.903 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.f.log 2024-05-27T08:03:41.903 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T08:03:41.903 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-05-27T08:03:41.903 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.c.log 2024-05-27T08:03:41.904 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.audit.log: /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mon.c.log 2024-05-27T08:03:41.904 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mgr.z.log 2024-05-27T08:03:41.904 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.i.log 2024-05-27T08:03:41.904 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mon.c.log: /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T08:03:41.904 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.11.log 2024-05-27T08:03:41.904 INFO:teuthology.orchestra.run.smithi186.stderr:gzip/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.cephadm.log: -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.8.log 2024-05-27T08:03:41.904 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T08:03:41.904 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.l.log 2024-05-27T08:03:41.921 INFO:teuthology.orchestra.run.smithi114.stderr: 84.6% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T08:03:41.958 INFO:teuthology.orchestra.run.smithi114.stderr: 90.3% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T08:03:41.976 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-volume.log: /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.l.log: 84.4% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T08:03:42.011 INFO:teuthology.orchestra.run.smithi114.stderr: 94.6% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.j.log.gz 2024-05-27T08:03:42.045 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mon.b.log: /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.k.log: 92.5% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mgr.y.log.gz 2024-05-27T08:03:42.088 INFO:teuthology.orchestra.run.smithi114.stderr: 94.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T08:03:42.103 INFO:teuthology.orchestra.run.smithi163.stderr: 91.0% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T08:03:42.113 INFO:teuthology.orchestra.run.smithi186.stderr: 92.5% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mgr.z.log.gz 2024-05-27T08:03:42.146 INFO:teuthology.orchestra.run.smithi186.stderr: 91.0% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T08:03:42.159 INFO:teuthology.orchestra.run.smithi114.stderr: 91.0% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T08:03:42.204 INFO:teuthology.orchestra.run.smithi186.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T08:03:42.222 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T08:03:42.246 INFO:teuthology.orchestra.run.smithi163.stderr: 84.4% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T08:03:42.250 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T08:03:42.251 INFO:teuthology.orchestra.run.smithi163.stderr: 90.5% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T08:03:42.256 INFO:teuthology.orchestra.run.smithi163.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T08:03:42.902 INFO:teuthology.orchestra.run.smithi163.stderr: 91.5% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mon.b.log.gz 2024-05-27T08:03:42.942 INFO:teuthology.orchestra.run.smithi186.stderr: 87.1% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T08:03:43.101 INFO:teuthology.orchestra.run.smithi114.stderr: 90.3% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mgr.x.log.gz 2024-05-27T08:03:45.150 INFO:teuthology.orchestra.run.smithi114.stderr: 90.1% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mon.a.log.gz 2024-05-27T08:03:50.443 INFO:teuthology.orchestra.run.smithi163.stderr: 94.7% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.e.log.gz 2024-05-27T08:03:52.897 INFO:teuthology.orchestra.run.smithi163.stderr: 95.1% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.h.log.gz 2024-05-27T08:03:54.833 INFO:teuthology.orchestra.run.smithi163.stderr: 93.7% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.k.log.gz 2024-05-27T08:03:56.856 INFO:teuthology.orchestra.run.smithi163.stderr: 95.4% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.b.log.gz 2024-05-27T08:03:58.518 INFO:teuthology.orchestra.run.smithi114.stderr: 93.7% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.a.log.gz 2024-05-27T08:04:00.198 INFO:teuthology.orchestra.run.smithi114.stderr: 93.7% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.d.log.gz 2024-05-27T08:04:19.157 INFO:teuthology.orchestra.run.smithi163.stderr: 91.5% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.10.log.gz 2024-05-27T08:04:20.556 INFO:teuthology.orchestra.run.smithi114.stderr: 93.8% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.g.log.gz 2024-05-27T08:04:30.721 INFO:teuthology.orchestra.run.smithi186.stderr: 92.8% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mon.c.log.gz 2024-05-27T08:04:36.968 INFO:teuthology.orchestra.run.smithi163.stderr: 91.6% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-05-27T08:04:49.839 INFO:teuthology.orchestra.run.smithi163.stderr: 91.3% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.7.log.gz 2024-05-27T08:04:52.527 INFO:teuthology.orchestra.run.smithi186.stderr: 95.3% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.c.log.gz 2024-05-27T08:04:53.399 INFO:teuthology.orchestra.run.smithi186.stderr: 95.2% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.f.log.gz 2024-05-27T08:05:04.231 INFO:teuthology.orchestra.run.smithi163.stderr: 91.4% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-05-27T08:05:04.233 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-05-27T08:05:04.233 INFO:teuthology.orchestra.run.smithi163.stderr:real 1m22.355s 2024-05-27T08:05:04.233 INFO:teuthology.orchestra.run.smithi163.stderr:user 3m23.914s 2024-05-27T08:05:04.233 INFO:teuthology.orchestra.run.smithi163.stderr:sys 0m6.702s 2024-05-27T08:05:15.161 INFO:teuthology.orchestra.run.smithi114.stderr: 91.5% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-05-27T08:05:19.599 INFO:teuthology.orchestra.run.smithi186.stderr: 95.3% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.l.log.gz 2024-05-27T08:05:22.690 INFO:teuthology.orchestra.run.smithi114.stderr: 91.3% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-05-27T08:05:25.151 INFO:teuthology.orchestra.run.smithi114.stderr: 91.2% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.6.log.gz 2024-05-27T08:05:34.632 INFO:teuthology.orchestra.run.smithi114.stderr: 91.3% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.9.log.gz 2024-05-27T08:05:34.635 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-05-27T08:05:34.635 INFO:teuthology.orchestra.run.smithi114.stderr:real 1m52.760s 2024-05-27T08:05:34.636 INFO:teuthology.orchestra.run.smithi114.stderr:user 3m48.091s 2024-05-27T08:05:34.636 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m7.916s 2024-05-27T08:11:05.672 INFO:teuthology.orchestra.run.smithi186.stderr: 91.2% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.8.log.gz 2024-05-27T08:11:27.194 INFO:teuthology.orchestra.run.smithi186.stderr: 91.4% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.11.log.gz 2024-05-27T08:14:17.571 INFO:teuthology.orchestra.run.smithi186.stderr: 91.1% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-05-27T08:14:55.781 INFO:teuthology.orchestra.run.smithi186.stderr: 91.5% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-05-27T08:21:16.693 INFO:teuthology.orchestra.run.smithi186.stderr: 93.3% -- replaced with /var/log/ceph/755b3728-1bfa-11ef-bc9b-c7b262605968/ceph-mds.i.log.gz 2024-05-27T08:21:16.706 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-27T08:21:16.706 INFO:teuthology.orchestra.run.smithi186.stderr:real 17m34.829s 2024-05-27T08:21:16.706 INFO:teuthology.orchestra.run.smithi186.stderr:user 10m9.003s 2024-05-27T08:21:16.706 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m25.596s 2024-05-27T08:21:16.707 INFO:tasks.cephadm:Archiving logs... 2024-05-27T08:21:16.707 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664/remote/smithi114/log 2024-05-27T08:21:16.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T08:21:55.856 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664/remote/smithi163/log 2024-05-27T08:21:55.857 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T08:22:30.604 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664/remote/smithi186/log 2024-05-27T08:22:30.606 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T08:24:07.015 INFO:tasks.cephadm:Removing cluster... 2024-05-27T08:24:07.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 --force 2024-05-27T08:24:07.210 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T08:24:07.782 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 --force 2024-05-27T08:24:07.974 INFO:teuthology.orchestra.run.smithi163.stdout:Deleting cluster with fsid: 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T08:24:08.565 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 755b3728-1bfa-11ef-bc9b-c7b262605968 --force 2024-05-27T08:24:09.148 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: 755b3728-1bfa-11ef-bc9b-c7b262605968 2024-05-27T08:24:09.875 INFO:tasks.cephadm:Removing cephadm ... 2024-05-27T08:24:09.875 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T08:24:09.895 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T08:24:09.911 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T08:24:09.927 INFO:tasks.cephadm:Teardown complete 2024-05-27T08:24:09.928 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-27T08:24:09.939 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-27T08:24:09.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T08:24:09.942 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T08:24:09.954 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T08:24:10.082 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-27T08:24:10.100 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-27T08:24:10.100 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T08:24:10.103 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T08:24:10.106 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T08:24:10.117 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-05-27T08:24:10.118 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-05-27T08:24:10.119 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-05-27T08:24:10.121 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T08:24:10.121 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-05-27T08:24:10.121 INFO:teuthology.orchestra.run.smithi163.stdout:^* hv01.front.sepia.ceph.com 3 6 377 8 +695us[ +637us] +/- 60ms 2024-05-27T08:24:10.121 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 9 -667us[ -725us] +/- 50ms 2024-05-27T08:24:10.121 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +28us[ -30us] +/- 74ms 2024-05-27T08:24:10.122 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T08:24:10.152 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T08:24:10.153 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-05-27T08:24:10.153 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv01.front.sepia.ceph.com 3 6 377 12 +707us[ +636us] +/- 60ms 2024-05-27T08:24:10.153 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 13 -654us[ -725us] +/- 50ms 2024-05-27T08:24:10.153 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +36us[ +36us] +/- 74ms 2024-05-27T08:24:10.153 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T08:24:10.223 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T08:24:10.224 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-05-27T08:24:10.224 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv01.front.sepia.ceph.com 3 6 377 8 +665us[ +658us] +/- 60ms 2024-05-27T08:24:10.224 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 -697us[ -697us] +/- 50ms 2024-05-27T08:24:10.224 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 -16us[ -23us] +/- 74ms 2024-05-27T08:24:10.224 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T08:24:10.225 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-27T08:24:10.239 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-27T08:24:10.239 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-27T08:24:10.303 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T08:24:10.459 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T08:24:10.551 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T08:24:10.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T08:24:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:24:10.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T08:24:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:24:10.702 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T08:24:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:24:10.729 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-27T08:24:10.741 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-27T08:24:10.751 INFO:teuthology.task.internal:Duration was 3869.738178 seconds 2024-05-27T08:24:10.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-27T08:24:10.762 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-27T08:24:10.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T08:24:10.765 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T08:24:10.767 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T08:24:10.805 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T08:24:10.811 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T08:24:10.847 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T08:24:11.271 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-27T08:24:11.271 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-05-27T08:24:11.272 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T08:24:11.294 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2024-05-27T08:24:11.295 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T08:24:11.337 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-05-27T08:24:11.338 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T08:24:11.372 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-27T08:24:11.372 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T08:24:11.375 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T08:24:11.380 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T08:24:11.440 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-27T08:24:11.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T08:24:11.445 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T08:24:11.447 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T08:24:13.023 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-27T08:24:13.093 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-27T08:24:13.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T08:24:13.119 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T08:24:13.153 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T08:24:13.186 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-27T08:24:13.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T08:24:13.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T08:24:13.392 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T08:24:13.417 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2024-05-27T08:24:13.425 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-05-27T08:24:13.600 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-05-27T08:24:13.621 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T08:24:13.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:24:13.638 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T08:24:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:24:13.659 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T08:24:13.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:24:13.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-27T08:24:13.730 INFO:teuthology.task.internal:Transferring archived files... 2024-05-27T08:24:13.731 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664/remote/smithi114 2024-05-27T08:24:13.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T08:24:14.341 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664/remote/smithi163 2024-05-27T08:24:14.341 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T08:24:14.399 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727664/remote/smithi186 2024-05-27T08:24:14.400 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T08:24:14.446 INFO:teuthology.task.internal:Removing archive directory... 2024-05-27T08:24:14.446 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T08:24:14.450 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T08:24:14.454 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T08:24:14.494 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-27T08:24:14.509 INFO:teuthology.task.internal:Not uploading archives. 2024-05-27T08:24:14.509 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-27T08:24:14.545 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-27T08:24:14.545 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T08:24:14.549 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T08:24:14.553 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T08:24:14.563 INFO:teuthology.orchestra.run.smithi114.stdout: 265911 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 08:24 /home/ubuntu/cephtest 2024-05-27T08:24:14.567 INFO:teuthology.orchestra.run.smithi163.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 08:24 /home/ubuntu/cephtest 2024-05-27T08:24:14.571 INFO:teuthology.orchestra.run.smithi186.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 08:24 /home/ubuntu/cephtest 2024-05-27T08:24:14.574 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-27T08:24:14.587 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-27T08:24:14.652 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/no 6-workunit/suites/dbench}} duration: 3869.738178253174 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi114 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=43b24159ea6f4cf5b69a0c12e17fd3a0 status: fail success: false 2024-05-27T08:24:14.652 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T08:24:14.742 INFO:teuthology.run:FAIL